47a183aa65bdf17b7c8d601f7efe97cbde515e2a
353 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
47a183aa65 |
feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup. - Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic. - Updated `.env.example` to document the new environment variable and its behavior. - Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges. - Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration. - Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management. |
||
|
|
e6a469631b |
feat(agent): add intent prompt customization in agent editor
Allow per-intent system prompt overrides for non-retrieval intents in normal mode. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
|
500c821817 |
feat(builtin-models): validate YAML entries and align ID length with schema
Three correctness fixes that the lifecycle PR deliberately deferred:
1. ID length / struct-tag drift
- models.id is varchar(64) on both PG and SQLite (per the init
migrations), but Model.ID's GORM tag said varchar(36) — a remnant
from when the field only held UUIDs. The mismatch is harmless under
golang-migrate (struct tag is ignored), but misleading on AutoMigrate
paths and in IDE tooltips. Tag now matches the real column width.
- New ModelIDMaxLen constant (=64) is the single source of truth for
anyone accepting user-provided ids. The YAML loader uses it to
reject too-long ids up front with a clear message instead of letting
the INSERT explode with a generic "value too long for type" error.
2. Field validation in the YAML loader
- Type, Source, and Status are typed strings but YAML can supply any
value. Misspellings (e.g. `type: knowledgeqa` lowercase, `type: LLM`)
were previously persisted as-is and produced rows that looked fine
in the table but failed at provider-factory lookup time, which is
hard to debug.
- validateBuiltinModelEntry now checks: empty id, id length, empty
type, type ∈ {KnowledgeQA, Embedding, Rerank, VLLM, ASR}, and
status ∈ {active, downloading, download_failed, empty}. Source is
intentionally NOT validated because the provider matrix in
internal/models/* keeps growing and a strict allow-list here would
force changes in two places per new provider.
- Invalid entries are warned + skipped (not aborting the whole load),
and excluded from the keep-set so the drift sweep does not delete
existing matching rows on the strength of a typo'd YAML retry.
3. Magic number cleanup
- DefaultBuiltinModelTenantID (=10000) replaces the hard-coded `10000`
literal in toModel(). The invariant lives in three places already
(PG migration, SQLite migration, this constant); naming it makes
the cross-reference explicit and grep-able.
Tests:
- New TestLoadBuiltinModelsConfig_RejectsInvalidEntries with five
sub-cases (id-too-long, missing-type, lowercase-type, unknown-type,
unknown-status) asserts the table stays empty after each.
- All 11 existing tests still pass.
|
||
|
|
4bce10d1ea |
refactor(builtin-models): unify logging via stdlib log with stable prefix
Original PR #1453 used fmt.Printf, which lands as unstructured noise in release/JSON log pipelines. The natural fix is logger.Infof/Warnf, but internal/logger itself imports internal/types — using it from here would create an import cycle (this is the same constraint that forces model.go's crypto error logs onto stdlib log). Switch all loader output to log.Printf with a stable "[builtin-models]" prefix (mirroring the "[crypto]" convention already established in model.go). The prefix gives operators a grep handle even though the lines stay unstructured. Levels are encoded inline as "WARN:" for warnings so log shippers and humans can still discriminate. |
||
|
|
7b192e546f |
feat(builtin-models): reconcile YAML lifecycle with drift sweep
Extend the builtin_models.yaml loader so the YAML file becomes a complete
source of truth for the rows it owns. Builds on the previous commit's
managed_by column.
Lifecycle contract:
- Every UPSERTed entry is tagged managed_by="yaml".
- The DoUpdates list now includes deleted_at, so an entry that was
soft-deleted (e.g. via UI/API) is automatically resurrected when it
reappears in the file. Closes the "ghost row that exists but is
invisible" failure mode.
- After all UPSERTs, the loader soft-deletes rows where
managed_by='yaml' AND id NOT IN (current YAML id set). Removing an
entry from YAML is now the supported way to retire a built-in model —
no manual SQL needed.
- Rows tagged managed_by='' (UI/API/SQL-seeded built-ins) are invisible
to the reconcile path and never touched.
- When a YAML entry sets is_default=true, the loader first clears
is_default on any other rows in the same (tenant_id, type) bucket,
mirroring the invariant enforced by the API path
(repository.UnsetDefaultModel).
Failure handling stays defensive:
- File missing / not a regular file / parse error: warn and skip; the
drift sweep is NOT executed so a malformed file cannot wipe rows.
- Per-entry UPSERT error: warn, drop the id from the keep-set so the
sweep also leaves the existing row alone ("leave alone on failure").
Tests cover: file-missing, parse-error, basic upsert + defaults,
idempotency, ${ENV} interpolation (set vs unset), drift sweep removing
YAML rows, drift sweep ignoring manual rows, soft-delete resurrection,
is_default cleanup across tenant+type, explicit empty list sweeping all
yaml-managed rows, and a regression guard ensuring BeforeCreate does not
overwrite YAML-supplied stable ids.
Docs are rewritten so operators see "delete from YAML and restart" as
the supported removal path; SQL is retained only for the legacy
managed_by='' slice.
|
||
|
|
fdc22fd7a5 |
feat(builtin-models): add managed_by column to models table
Introduce a `managed_by` varchar column on `models` so future declarative
loaders can claim ownership of a subset of rows without disturbing entries
created via the UI/API or seeded by hand-written SQL.
- versioned/000052_models_managed_by.{up,down}.sql add the column with a
default of '' and a partial index on non-empty values to keep startup
reconciliation cheap.
- sqlite/000000_init.up.sql is updated in place (the Lite init migration
is a single file per project convention) so fresh SQLite databases get
the column from the start.
- Model.ManagedBy mirrors the column. Existing rows default to '' which
the YAML loader treats as "manually managed, never touch".
Schema half of the YAML-driven built-in-model lifecycle work that follows
up on #1453; the reconciler that uses the column lands in the next commit.
|
||
|
|
d439b3ae07 |
feat(builtin-models): add YAML-based declarative config with ${ENV} interpolation
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.
Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.
The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
|
||
|
|
caeb6a44d8 |
refactor(chunk): remove VideoInfo field from Chunk struct
- Removed the VideoInfo field from the Chunk struct in chunk.go, streamlining the data model. - This change reflects a shift in focus away from video information storage within the Chunk type. |
||
|
|
29820e4cac |
docs(chat): clarify cached-token semantics for explicit-cache providers
`cached_tokens` is reported by every OpenAI-compatible provider that
supports prompt caching, but how it becomes non-zero differs by mode:
- Implicit caching (OpenAI, Azure OpenAI, DeepSeek, …) populates the
field automatically whenever a prompt prefix matches a previous
request within the provider's cache TTL. No client-side opt-in.
- Explicit caching (Qwen on Aliyun, Anthropic Claude, …) only
populates the field after the caller attaches `cache_control:
{"type": "ephemeral"}` to the relevant message / content block.
Until that opt-in is applied upstream of the request, the field
stays zero even when the prefix is otherwise byte-stable.
Without this distinction documented, the previous commit reads as if
`TokenUsage.CachedTokens` will show non-zero values for Qwen / Claude
once this PR lands — which is not the case. The plumbing here is a
prerequisite (stable prefix via sorted tools) and a meter (visibility
of the field), but the explicit-cache opt-in itself is out of scope
and lives elsewhere.
Document this on `TokenUsage.CachedTokens` and the `cachedTokens`
helper so callers do not mistake observability for activation.
|
||
|
|
582f7b3056 |
feat(chat): surface cached prompt tokens from upstream usage
OpenAI-compatible providers (Qwen, DeepSeek, OpenAI, Azure, etc.) report
prompt-cache hits in `usage.prompt_tokens_details.cached_tokens`. This
value was being read by go-openai but dropped at the WeKnora boundary,
so there was no way to tell whether prompt caching was actually working.
This change plumbs the field end-to-end:
- `types.TokenUsage.CachedTokens` (json:"cached_tokens,omitempty") —
zero-values are omitted so payloads stay quiet for providers that
never report cache hits.
- `cachedTokens` helper in remote_api.go guards against nil
`PromptTokensDetails` (Ollama and older OpenAI-compat backends omit
the details block entirely).
- All three response-parsing paths populate it:
* `parseCompletionResponse` (non-streaming)
* `processStream` (SDK streaming)
* `processRawHTTPStream` (raw-HTTP streaming, used when callers
need to inject custom fields like `cache_control`)
- The five `[LLM Usage]` log lines now print `cached_tokens=%d` so
cache hit rate is visible in `journalctl` / log tail without going
through metrics.
Together with the deterministic tool ordering from the previous commit,
this makes Qwen explicit caching observable: a warmed prefix should
show `cached_tokens` ≈ system + tools token count (typically several
thousand) on subsequent requests within the 5-minute TTL.
Tests:
- `TestCachedTokensHelper` — nil safety + round-trip
- `TestParseCompletionResponse_CachedTokens` — populated + missing
details paths through `parseCompletionResponse`
- `TestTokenUsage_CachedTokensJSONOmitempty` — zero is omitted,
non-zero is emitted
|
||
|
|
744a367f16 |
feat(retriever): add OpenSearch type prep ahead of Phase 3 driver
Lays type-system groundwork for the upcoming OpenSearch k-NN driver (Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate: this PR ships only inert constants, schema extensions, and an unreachable normalizer case. No path activates an OpenSearch VectorStore yet — creation continues to fail with "not a valid engine type" until the activation switch lands in Phase 3 PR 3. Changes: - OpenSearchRetrieverEngineType constant ("opensearch") in internal/types/retriever.go. Not added to validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping yet (gated). - ConnectionConfig.InsecureSkipVerify (bool, default false) in internal/types/vectorstore.go, placed inside the // Common section because it is a cross-driver TLS option. Distinct from the Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify only controls verification of an already-TLS HTTPS connection. AES-GCM Value/Scan round-trips the field as plaintext (it travels alongside the encrypted Password / APIKey but is not sensitive itself). - VectorStoreFieldInfo gains four optional fields: Immutable (bool), Min/Max (*float64), Enum ([]string). All omitempty so existing UI schema entries serialize identically. The fields will drive the UI in Phase 3 PR 3 (read-only on Edit + range/enum constraints). - Six new AuditAction constants under vector_store.* and opensearch.* namespaces. Definitions only — emission lands in Phase 3 PR 3. - EngineAwareNormalizer.Normalize is restructured to group engines by the effective score range observed by the normalizer (not the theoretical raw cosine range): Range [-1, 1] (raw cosine, mapped via (score + 1) / 2): - Milvus (COSINE metric mode). Milvus docs explicitly state the COSINE metric range is [-1, 1]. Range [0, 1] (passthrough — already on the target scale): - Elasticsearch v8 / ElasticFaiss. The driver issues a cosineSimilarity(...) script_score script, and Lucene rejects negative final scores ("Final relevance scores from the script_score query cannot be negative" — ES docs); the effective range observed by the normalizer is therefore [0, 1] for IR-normalized embeddings. ES was previously grouped with Milvus and over-corrected via (score + 1) / 2, which inflated every ES result by 50% in mixed-engine RRF fusion. - OpenSearch. The k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation maps the underlying Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1] before the score reaches us. Source: github.com/opensearch-project/k-NN at src/main/java/org/opensearch/knn/index/SpaceType.java (COSINESIMIL enum, scoreTranslation method). - Weaviate. The driver requests `certainty`, defined by Weaviate as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1]. - Postgres pgvector / SQLite sqlite-vec / Qdrant / TencentVectorDB / Doris. The driver computes (1 - cosine_distance) or normalized inner_product, whose theoretical range is [-1, 1]. The IR-normalized positive-component unit vectors WeKnora targets (BGE / OpenAI text-embedding-3 / Cohere / sentence-transformers) keep the observed range in [0, 1]; negative-cosine embedding models would silently clamp to 0 downstream — explicitly documented as the IR-normalization caveat in the struct godoc. Dead enum references (ElasticFaiss, Infinity) are flagged in the godoc with a pointer to internal/types/vectorstore.go's existing "legacy/experimental, no standalone deployable instance" annotation. Their case labels are kept for switch exhaustiveness. Test coverage: - OpenSearch constant wire value + collision check against existing 10 engines + gated invariant (NOT in validEngineTypes). - ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON field deserializes as false) + round-trip + AES-GCM coexistence with encrypted Password / APIKey. - VectorStoreFieldInfo omitempty preservation + new-field serialization + *float64 pointer distinction (min=0 vs nil). - AuditAction dot-namespace convention enforcement, prefix invariants for vector_store.* and opensearch.*, no wire-string collisions, exact wire-value pins for the six new constants. - EngineAwareNormalizer: - CosineRange retains its (score + 1) / 2 coverage but now only for Milvus. - UnitInterval now covers the full passthrough group (ES / ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant / Infinity / TencentVectorDB / Doris). - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an explicit regression guard for the score-range correction: cos=0.5 maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed). - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift (1.0001) + defensive negative + ±Inf / NaN edges + keyword passthrough + nil-ctx safety. Backward compatibility is preserved at every layer: - All new struct fields are omitempty / pointer-tagged so existing rows and existing wire formats remain unchanged. - Normalizer's new OpenSearch case is unreachable until the driver lands. The ES regrouping changes the post-normalization value for every ES vector search result (a correctness fix, not a feature) — ES vector retrieval is currently the only production path affected. - AuditAction constants emit no audit_log rows in this PR. - engine_type=opensearch VectorStore creation still rejected (gated). |
||
|
|
8b7d00b260 |
feat: expose KB ↔ vector store binding in list, editor, and detail UI
Backend ------- ListKnowledgeBases now enriches each row with the resolved vector_store metadata (name / source / engine_type / status) via a new buildKBListResponse helper. Store views are batch-resolved once per request through BatchResolveStoreView so an N-KB list costs one vector-store service call rather than N — closing the N+1 limitation flagged in #1372's known-limitations section. Cross-tenant shared KBs continue to render via SharedStoreDisplay so the owning tenant's store inventory cannot be correlated across rows; the underlying vector_store_id UUID is stripped from those responses. Resolver failures degrade gracefully: bound KBs render as unavailable instead of breaking the list. Test coverage pins the env / bound / shared distinction, the batch-call-count invariant, and the graceful-failure path. Frontend -------- KB editor modal gains a new "Vector Store" section. Create mode shows a dropdown that combines the system default (env store) and the tenant's configured user stores, fetched once at mount via the existing listVectorStores API. Edit mode shows the bound store read-only via a new VectorStoreBadge component with an explicit immutability hint — matching the backend's `<-:create` GORM tag and the service-layer UpdateKnowledgeBaseRequest DTO that already omit the field. KB list cards surface a small engine-type badge for own-tenant bound KBs, and a warning badge when the bound store is unavailable. Env-bound and shared KBs render no badge (visual noise control). KB detail header shows the bound store via the same VectorStoreBadge component; shared KBs fall through to the badge's internal "shared" branch with no name / engine / id rendered. The KB editor's create-time error handler translates the typed ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201) into localized messages and jumps the user back to the VectorStore section so they can pick a different store or fall back to the system default. The KB row type gains five optional fields (vector_store_id / name / engine_type / source / status). i18n: 18 new keys added to en-US, ko-KR, zh-CN; ru-RU receives English placeholders pending translation (consistent with prior PRs in this locale). Part of #993 (Phase 2: Per-KB VectorStore Binding). Phase 2 roadmap item: PR 5 (KB binding UI + list-response enrichment). Depends on #994, #1310, #1372, #1386 (all backend in the Phase 2 series). |
||
|
|
513e589494 |
feat: add vLLM server URL configuration for MinerU
Add support for configuring vLLM server URL when using vlm-http-client or hybrid-http-client backend in MinerU. Changes: - internal/types/tenant.go: Add MinerUVLMServerURL field to ParserEngineConfig - internal/infrastructure/docparser/mineru_converter.go: Pass server_url to MinerU API when backend is vlm-http-client or hybrid-http-client - frontend/src/api/system/index.ts: Add TypeScript type definition - frontend/src/views/settings/ParserEngineSettings.vue: Add vLLM server URL input field - frontend/src/i18n/locales/*.ts: Add translations (zh-CN, en-US, ko-KR) |
||
|
|
7bb38f81b6 |
feat(session): persist last request state for UI restoration
Sessions now record the input-bar state used for the most recent QA request (agent, model, KB scope, web search). The chat UI hydrates those settings on session reopen so users see the same configuration they used last time, instead of the global default. The state is stored in the existing sessions.agent_config JSONB column to avoid a new migration. Frontend snapshots the user's global defaults on session enter and restores them on session leave, so opening an old session does not pollute new-chat defaults. |
||
|
|
0cdb922d84 |
feat(tenant): remember user's last-active workspace across logins
Until now a fresh login (new device, expired refresh token, cleared
browser) always dropped the user into their home tenant, even if they
spend most of their time in a peer workspace. The session-local
X-Tenant-ID override in localStorage gave a "same browser sticky"
effect, but never crossed devices or new sessions.
This adds a small server-side preference, `users.preferences.
last_active_tenant_id`, persisted in the existing jsonb column (no
new migration), and threads it through:
* Backend
* `UserPreferences` gains `LastActiveTenantID *uint64` with sentinel
semantics (`*0` from the PATCH endpoint = clear preference).
* `resolveLoginTenantID` validates the stored id (tenant still
exists + active membership, or CanAccessAllTenants) and falls
back to home on any failure, best-effort clearing the stale
preference so subsequent logins don't pay for it again.
* `Login` and `LoginWithOIDC` resolve once and use the result for
both the JWT `tenant_id` claim and the returned `active_tenant`,
keeping the two in sync. `RefreshToken` rides through
`GenerateTokens` so refresh rotations also land the user back in
their preferred tenant instead of bouncing to home.
* `UpdateUserPreferences` learns to merge the new key.
* `PUT /auth/me/preferences` accepts the new field.
* Frontend
* `Login.vue` now writes the user's HOME tenant id into
`user.tenant_id` (matching the field's documented semantics) and
expresses any active-vs-home divergence via `setSelectedTenant`,
so `useHomeTenant` and the "current"/"home" badges stay correct
after the backend honours a remembered preference. `App.vue`'s
OIDC sync does the same reconciliation.
* `TenantSelector`, `UserMenu` and the post-tenant-create handlers
fire `persistLastActiveTenantPreference` after every successful
user-initiated switch (switching to home sends `0` to clear).
The call is raced against the existing reload-grace window so
most writes finish before the page tears down; lost writes are
recoverable on the next switch.
No new UI. Users will simply notice that, after re-logging in on
another device, they land back in the workspace they were last
using rather than always in their home tenant.
Note: `make docs` is unrelated-broken on `main` (audit_log.go
references `errors.AppError` which swag can't resolve), so the
Swagger artifacts under docs/ are intentionally not regenerated
in this PR. The handler code is the source of truth.
|
||
|
|
b3dbf86bf5 | feat(obs): 支持华为云obs存储 | ||
|
|
1506c604e1 | refactor(knowledgebase): remove TogglePinKnowledgeBase method from KnowledgeBaseRepository interface | ||
|
|
7528175a74 |
feat(knowledge-base): implement per-user pinning for knowledge bases
Refactored the pinning mechanism for knowledge bases to be user-specific, allowing any user with read access to pin knowledge bases independently. This change includes the introduction of a new `user_kb_pins` table to manage pin states, replacing the previous tenant-wide pinning system. Updated relevant API endpoints and service logic to support this new functionality, ensuring that the UI reflects the user's personal pin status. Additionally, adjusted the knowledge base listing to sort by user-specific pin status, enhancing the user experience by prioritizing pinned items. Refs: #1303 |
||
|
|
68ac7fb973 |
feat(user-preferences): implement user preferences management with backend support
Added functionality for managing user preferences, including the ability to update preferences via a new API endpoint. The preferences are now stored server-side, allowing for synchronization across devices. Updated relevant components to handle user preferences, including the UserMenu and settings views. Enhanced internationalization support for error messages related to preference updates. |
||
|
|
8d4c58c5fa |
feat(creator_name): add creator_name field to agents and knowledge bases for enhanced visibility
Introduced a new optional field, creator_name, in both CustomAgent and KnowledgeBase types to allow the front end to display the creator's name. This enhancement enables better differentiation between resources created by the current user and those created by other members of the same tenant. Updated relevant API handlers to populate this field during list operations, ensuring accurate representation in the UI. Additionally, modified the ResourceOriginBadge component to utilize the creator_name for improved context in resource listings. Refs: #1303 |
||
|
|
884ac44283 |
feat(invitations): implement pending invitations feature with polling and dialog
Added functionality for managing pending invitations, including a polling mechanism to update the invitation count in real-time. Introduced a new dialog component for users to view and respond to their invitations without navigating away from their current context. Updated API endpoints for fetching and managing invitations, and enhanced the user interface with relevant internationalization support. Refs: #1303 |
||
|
|
8332a0dab6 |
feat(favorites): implement user favorites and recents functionality
Introduced a new API client for managing user favorites, allowing users to list, add, and remove starred resources. Enhanced the sidebar to display favorites and recents, improving user navigation. Updated internationalization files to include relevant labels and tooltips for the new features across multiple languages. Refs: #1303 |
||
|
|
7d030a6f6a |
feat(tenant): implement tenant creation limit and error handling
Added a new configuration option to limit the number of tenants a non-superuser can create via self-service. Introduced a new error type for handling cases where users exceed this limit, returning a 429 status code. Updated the tenant creation and update handlers to enforce this limit and provide appropriate feedback to users. Additionally, refactored the tenant update request structure to ensure only mutable fields are allowed. Refs: #1303 |
||
|
|
dc8d033b4d |
feat(organization): transition membership model to tenant-level and enhance invite functionality
Updated the organization membership model to operate at the tenant level, aligning with Plan 3. This change introduces new interfaces and modifies existing ones to accommodate tenant-centric operations, including the `InviteMemberRequest` and `TenantInviteCandidate`. The search functionality for inviting members has been updated to reflect this shift, allowing for tenant-based searches instead of user-based. Additionally, internationalization files have been updated to ensure consistent messaging across languages regarding tenant membership and roles. Refs: #1303 |
||
|
|
ff1f03ce0d |
feat(rbac): daily retention sweep for audit_logs (default 90 days)
The audit_logs table grows monotonically until something purges it. PR 6 explicitly listed retention as out of scope for v1; this picks that up. Adds an `audit.retention_days` config (default 90, 0 disables) and a small background goroutine `AuditLogRetentionRunner` that fires once ~10 minutes after boot and then every 24h, calling `AuditLogService.Purge` which DELETEs rows older than the cutoff in a single statement. The dependency surface is intentionally minimal — no robfig/cron, no asynq, no migration — because retention has no wall-clock alignment requirement. Wiring is the same shape as the existing data source scheduler: container `Provide(NewAuditLogRetentionRunner)`, an `Invoke` that calls `Start`, and a `ResourceCleaner.RegisterWithName` for graceful shutdown. Defaults preserve operator intent: - `audit:` section omitted from YAML -> retention_days=90. - explicit `audit.retention_days: 0` in YAML -> purge disabled. - `WEKNORA_AUDIT_RETENTION_DAYS=N` env override (incl. N=0). - `retention_days < 0` is rejected by ValidateConfig. Tests cover the service Purge contract (no-op when disabled, cutoff math, error propagation) and the runner lifecycle (Start no-op when disabled, idempotent Start/Stop, ticker cadence, runOnce swallows errors). 10 new tests, all green. docs/rbac.md updated with the new YAML / env / behaviour. Refs: #1303 |
||
|
|
cf6c28fe7e |
fix(rbac-ui): identify org owner row by tenant_id, not user_id
Plan 3 (#1303) lifted Org membership from per-user to per-tenant, and 40fdb978 pinned the org's owner tenant in DB as organizations.owner_tenant_id. The OrganizationSettingsModal members list, however, still decided "which row is the owner" by comparing member.user_id against orgInfo.owner_id. After Plan 3 the members list is keyed on tenant_id, so the user-id check is structurally wrong: the representative user of the owner tenant happens to be the owner today, but if that user is moved to another tenant — or a different user of the owner tenant becomes the representative — the modal would either lose the "(owner)" tag, or re-tag a non-owner row, and would offer the role-select / remove-button affordances on what is in fact the owner tenant row. This change: - Adds OwnerTenantID to OrganizationResponse and populates it in toOrgResponse from the persisted column. IsOwner now compares the caller's current tenant against owner_tenant_id (with a user-id fallback for legacy rows where owner_tenant_id == 0). - Adds owner_tenant_id to the frontend Organization type and a small isOwnerMember(member) helper in OrganizationSettingsModal, which tenant-matches by default and falls back to user-id only on pre-000046 rows. The role <select>, role tag suffix, and the delete button on member rows now all use the helper. The "is-me" CSS class and the me-tag in the member name still match on user_id — that one really is a per-user marker ("this row's user is me, the signed-in user"). |
||
|
|
a16fe83034 |
fix(rbac): pin organization owner tenant in DB instead of deriving at runtime
Plan 3's isOwnerTenant resolved "is this tenant the owner tenant?" by looking up the owner user's CURRENT tenant via userRepo. That breaks silently the moment the owner user is moved to another tenant: the formerly-protected owning tenant becomes removable (can be kicked from OTM, role-downgraded), and the new tenant the owner moved to is treated as untouchable even though it isn't even in OTM. Promote owner-tenant to a first-class column on `organizations`: - New migration 000046 adds owner_tenant_id BIGINT NOT NULL with index. Backfill: (1) owner user's current tenant when known, (2) earliest Admin tenant in OTM as fallback for orphans, then (3) RAISE EXCEPTION on anything still NULL — refuse to silently ship a bogus value. SQLite init schema mirrors the column. - CreateOrganization now sets OwnerTenantID at create time so the pinned value is correct for all new orgs. - isOwnerTenant becomes a pure equality check on the persisted column — no userRepo lookup, no error tuple. Fail-closed on a zero column value (legacy data not yet backfilled or unit tests bypassing the migration) so the worst case is "membership table frozen for that org" rather than "owner tenant is removable", matching the prior fail-closed-on-error semantics. - RemoveTenantMember / UpdateTenantMemberRole simplify to a single bool check. |
||
|
|
928f5abe3f |
feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303)
Before this change, the `organization_members` table keyed on user_id — joining an org as a Viewer in tenant T meant only that one user got visibility on shared KBs/agents, while the rest of T (including its Owner) saw nothing. The mismatch surfaced as confusing UX bugs: "my colleague shared this with us, why can't I see it?" This change replaces `organization_members` with `organization_tenant_members`. Membership and Org-RBAC role now key on (org_id, tenant_id); `representative_user_id` stays as a display-only label of the user who first brought the tenant in. Permission resolution gains a 3-D cap so tenant-RBAC stays honest under cross-tenant sharing: effective = min(share.Permission, tenant_org_role, tenant_role_cap) where tenant_role_cap pins tenant Viewers to OrgRoleViewer regardless of the org-level grant. A Viewer in their own tenant cannot edit a shared KB even if the org granted Editor — preserves the "Viewer = read-only everywhere" promise. Migration 000045 backfills `organization_tenant_members` from the old table by collapsing each (org, tenant) group to its max role, then renames the old table to `organization_members_pre_plan3` for safe rollback. Sqlite mirror (Lite mode) is fresh-bootstrap: the init schema replaces the old table directly. Routes: member-management URLs change from `:user_id` to `:tenant_id`. Org create / join / leave now require Admin+ in the caller's tenant — joining an Org affects everyone in T, so the decision can't sit with a Viewer. Frontend: OrganizationSettingsModal + store update to address members by tenant_id. |
||
|
|
1e8b5c8d40 |
refactor(api): drop deprecated tenant ConversationConfig field and KV write path
The Tenant.ConversationConfig field was marked deprecated when
CustomAgent (builtin-quick-answer) replaced the per-tenant defaults
path, but the JSONB column, GET/PUT KV handlers, write helpers and
read fall-throughs all stayed wired. This had two concrete failure
modes:
- Input-field.vue silently PUT-ed per-user model picks back into
/tenants/kv/conversation-config, which requires Admin+ — Viewers
and Contributors got a 403 every time they switched chat models.
- The same write also overwrote the tenant-shared default for every
other user, turning a per-user UX into a per-tenant footgun.
Remove the field, the struct, the validate helper, the PUT case and
the two dead Summary helpers in session/helpers.go. KnowledgeSearch
loses its tenant-conversation fallback (one chunk-tool reader); the
global config + hardcoded defaults still cover it. session_agent_qa
already required agents to declare their own rerank_model_id, so no
behaviour change there.
Frontend follows:
- Input-field.vue: per-user 'last selected chat model' moves to
localStorage, no API call.
- AgentEditorModal: defaults for builtin-quick-answer creation now
come from prompt-templates (system/context/rewrite/fallback) and
retrieval-config (top_k / threshold), the two endpoints that
actually own those settings now.
- AgentSettings.vue: orphan settings page with no router/import
referencing it; deleted.
- api/system: getConversationConfig / updateConversationConfig and
the ConversationConfig interface go away.
Net: 137 insertions, 2761 deletions.
|
||
|
|
766a6c482f |
feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303)
PR 1-5 of the multi-tenant RBAC series enforce the role matrix; what they don't do is keep a durable record of who did what. Today the RBAC middleware logs "[rbac] role insufficient ..." to stderr - perishable, not searchable. Operators cannot answer "who removed Alice from tenant T-foo on Tuesday" or "did we 403 a probing user 200 times last hour" after the fact. This PR adds a generic per-tenant audit log that: - Captures mutating RBAC actions (member add/remove/role-change/leave) durably from the tenant_member service after the repo write succeeds. - Captures enforcement rejections (RequireRole / RequireOwnershipOrRole denies under EnableRBAC=true) via a gin-context-injected hook. - Sliding-window dedup (1 minute) on (tenant, actor, action, path) for denied events so a probing client cannot flood audit_logs. - Default-on - no extra env flag. Schema is generic enough for future KB / agent / datasource events to reuse via new action namespaces without another migration. Schema (migration 000044) mirrors wiki_log_entries' shape: a single audit_logs table with action, target_type/id, request path/method, outcome, and a JSONB details column. Three indexes: (tenant_id, id DESC) for the per-tenant cursor feed, actor_user_id for "what did Alice do", (tenant_id, action) for both the action filter and the dedup lookup. Sqlite mirror added for Lite mode. Wiring is nil-safe: the audit field on tenantMemberService and the gin-context lookup in rbac.go both tolerate a nil service so existing tests construct without a stub and dormant deployments degrade gracefully if the provider is misconfigured. The query API is GET /tenants/:id/audit-log, gated by Admin+ on top of the existing PathTenantMatch (denial histories should not surface to ordinary members). Cursor-paginated by descending id (sidesteps the duplicate-timestamp tie-breaking that BIGSERIAL trivially solves). Frontend: TenantMembers.vue gets a t-tabs wrapper with Members (existing) and Audit log (new) tabs; the audit tab is gated to Admin+ to mirror the server, lazy-loads on first open, paginates via the next_cursor pattern, and renders coloured chips per action/outcome so an operator can scan for anomalies. i18n keys added in all four locales. Tests: 14 new tests across service (6), middleware (4), and handler (4). Service tests pin the dedup window behaviour (writes again after expiry, dedup is per-actor-per-path, degrades on lookup error). Middleware tests pin both reject sites' audit hook and the "dormant mode does not fire" + "nil audit does not panic" invariants. Handler tests pin the cursor envelope and that an empty page returns next_cursor=0. Out of scope: per-action ACL on the audit log, retention/TTL job, KB/agent/datasource event wiring (schema supports it - just no emitters yet), async writes, frontend filter UI. |
||
|
|
d42a41b784 |
feat(rbac): per-KB ownership for knowledge / chunks / wiki pages (#1303)
PR 1–4 of the multi-tenant RBAC series gated KB-level and Agent-level
mutations behind a "creator OR Admin+" check via OwnedKBOrAdmin /
OwnedAgentOrAdmin. The layer underneath — knowledge entries, chunks,
wiki pages — was still gated only by Contributor, which is wider than
intended: any tenant Contributor could edit/delete documents in a
private KB they didn't own. PR 2 / wiki author left explicit follow-up
breadcrumbs at the three call sites in router.go acknowledging this gap.
This PR closes those breadcrumbs by reusing the existing scaffolding:
- KnowledgeService.GetOwningKBCreatorID walks knowledge_id -> kb_id ->
KB.CreatorID, tenant-scoped through GetKnowledgeByID. The chain lives
in the service layer so the handler-side lookups stay pure adapters.
- Three new CreatorLookup closures (KBCreatorLookupFromKnowledgeID,
KBCreatorLookupFromKnowledgeIDParam, KBCreatorLookupFromKBPath) are
added in handler/rbac_lookups.go alongside the existing
KBCreatorLookup / AgentCreatorLookup. Each translates the repository
sentinels into middleware.ErrResourceNotFound and re-checks tenant
scope as defence-in-depth, mirroring KBCreatorLookup exactly.
- router/rbac.go gains three guard factories — OwnedKnowledgeKBOrAdmin,
OwnedChunkKBOrAdmin, OwnedWikiKBOrAdmin — wired through the same
rbacGuards bundle.
- Mutating routes in RegisterKnowledgeRoutes / RegisterChunkRoutes /
RegisterWikiPageRoutes switch from g.Contributor() to the new
ownership guards. Cross-KB batch operations (POST /knowledge/move,
POST /knowledge/batch-delete, PUT /knowledge/tags) stay
Contributor-gated since there's no single owning KB to check
against. KB-scoped uploads (POST /knowledge-bases/:id/knowledge/...)
reuse the existing OwnedKBOrAdmin because the URL :id is the KB id
directly. The chunks.DELETE("/by-id/:id/questions") route addresses
chunks by their own id with no knowledge id in the URL, so it stays
Contributor-gated as a documented carve-out.
No schema changes: documents inside a shared KB are KB-owned content,
not personal uploads — that's how the product is positioned and matches
the original "per-KB-ownership granularity" wording from the PR 2
breadcrumbs. Legacy/built-in KBs with empty CreatorID surface as
tenant-owned (Admin+ only), same rule as built-in agents.
Tests:
- 9 new test cases in handler/rbac_lookups_test.go covering the three
new lookups: NotFoundMapsToSentinel (each repository sentinel),
CrossTenantIsHiddenAsNotFound, OwnerMatchReturnsCreatorID,
MissingTenantContext (fail-closed), and ChunkRouteShape (param-name
regression guard).
- 5 new test cases in service/knowledge_owner_chain_test.go covering
the chain helper: knowledge-not-found, kb-not-found, nil-kb-without-
err, happy path, and legacy KB with empty CreatorID.
Build, vet, and the targeted ./internal/middleware/...
./internal/handler/... ./internal/router/...
./internal/application/service test runs are clean. Pre-existing
TestCreateStore_* / TestOssEnsureBucket_CreateFails failures on main
are unrelated and reproduce on a clean checkout.
|
||
|
|
543e9709be |
fix(rbac): address PR 3 + leave-endpoint review feedback
Security - Reject URL :id when it does not match the caller's active tenant on every /tenants/:id/members/* and /tenants/:id/leave endpoint. Without this an Owner of tenant A could drive operations on tenant B simply by changing the URL, because RequireRole resolves the caller's role against the active tenant from context, not the URL param. The cross-tenant superuser carve-out mirrors middleware/rbac.go (CanAccessAllTenants + EnableCrossTenantAccess required). - AddMember no longer records the synthetic "system-<tenantID>" caller (X-API-Key auth path) as invited_by, matching the same filter KB.CreatorID got in PR 2. - AuthHandler now logs a loud one-shot error when constructed with a nil/incomplete config so a wiring bug that silently disables the invite_only gate is visible at boot. Race fixes - New repo helpers DemoteOwnerAtomically / RemoveOwnerAtomically wrap the count-other-owners + write step in a transaction that holds SELECT … FOR UPDATE locks on every other active Owner row. The service routes Owner demotion / Owner removal through them, closing the TOCTOU window where two concurrent demotes of two different Owners could each observe count=2 and both commit, leaving the tenant ownerless. Performance - ListMembers hydration now batches user lookups via a new GetUsersByIDs interface method (one SELECT … WHERE id IN (…) instead of N round-trips). Failure degrades to "no email/username" rather than dropping rows. Frontend - canManage also fires for cross-tenant superusers (authStore.canAccessAllTenants), mirroring the backend carve-out so the UI controls match what the server actually allows. Tests - Cross-tenant URL guard regression cases for List / Add / Update / Remove / Leave. - Superuser bypass tests (flag on -> service reached, flag off -> 403). - Batched lookup test ensures GetUsersByIDs is called exactly once and GetUserByID is never used as a fallback. - Synthetic caller invited_by filter regression test. - Service-layer atomic last-Owner regression tests for both UpdateRole and RemoveMember. |
||
|
|
3f7150cb5c |
feat(rbac): tenant member management + invite_only gate (PR 3 of 3, #1303)
Wire the human-facing surface for the tenant-RBAC system that PR 1 introduced and PR 2 enforced: REST endpoints + UI for managing tenant members, plus the long-pending invite_only registration gate. Backend - internal/handler/auth.go: Register now rejects with 403 when auth.registration_mode=invite_only. PR 1 already exposes the mode via /auth/config so the frontend hides the sign-up link; this is the server-side enforcement for direct API calls. - internal/handler/tenant_member.go: new TenantMemberHandler with ListMembers / AddMember / UpdateMemberRole / RemoveMember. Maps service-layer sentinels onto HTTP semantics (ErrLastOwner -> 409, ErrMembershipAlreadyExists -> 409, ErrUserNotFound on AddMember -> 404, ErrInvalidTenantRole -> 400). AddMember requires the invitee to already have an account; SMTP-driven email invites are deferred. - internal/types/tenant_member.go: new TenantMemberResponse projection carrying the fields the management UI renders (email, username, avatar, role, joined_at). - internal/router/router.go: four routes under /tenants/:id/members gated per the matrix (Viewer+ for list, Owner+ for every mutation). - internal/container/container.go: dig provider for the new handler. Tests - handler/tenant_member_test.go: 12 table-driven cases covering happy path, dangling-user tolerance, ErrLastOwner -> 409, ErrMembershipNotFound -> 404, ErrMembershipAlreadyExists -> 409, unknown email -> 404, invalid role rejected before user lookup, bad tenant id -> 400. - handler/auth_register_invite_only_test.go: invite_only -> 403, self_serve -> 201, nil Auth config falls through. Frontend - src/api/tenant/members.ts: typed client (TenantRole/TenantMember/...) for the four endpoints. - src/views/settings/TenantMembers.vue: members table with add dialog, inline role select, confirm-remove modal. All mutation widgets gated behind currentTenantRole === 'owner'; server still enforces. - src/views/settings/Settings.vue: register the Members tab; show only when currentTenantRole is loaded (avoids tab flash on cold nav). - i18n in en-US/zh-CN/ko-KR/ru-RU: new tenantMember namespace covering all UI strings and structured error messages (lastOwner / userNotFound / alreadyMember / invalidRole etc). This completes the #1303 RBAC rollout. With cfg.Tenant.EnableRBAC=true operators now have a full management surface; with EnableRBAC=false the routes still exist but mutations log-and-pass per PR 2 contract. |
||
|
|
8b71def60b |
fix(rbac): address PR 2 review feedback
Middleware - RequireOwnershipOrRole now skips the creator lookup entirely when EnableRBAC is off, eliminating a hidden per-request DB roundtrip in the rollout window. - New ErrResourceNotFound sentinel lets lookups signal "not visible to this tenant"; the middleware passes the request through so handlers can return a real 404 instead of a misleading 403. - Transient lookup failures now surface as 503 (was 403) so monitoring can tell denied-by-policy from cannot-verify-right-now. - Cross-tenant superusers (CanAccessAllTenants) bypass tenant role gates, restoring DELETE /tenants/:id and ResetAPIKey for org operators that resolve to Admin in foreign tenants. - Construction-time warning fires once when cfg is nil/incomplete, so a wiring bug that disables enforcement globally is visible at boot. Route coverage - Add role guards to FAQ, KnowledgeTag, WikiPage, Skill, WebSearch, WebSearchProvider, WeKnoraCloud, Session, Chat, and Message route groups so the matrix actually matches the PR description. Service layer - KB and agent creators are no longer recorded as the synthetic system-<tenantID> user used by the X-API-Key path; resources stay tenant-owned. New types.IsSyntheticUserID helper centralises the prefix check. - KBCreatorLookup and AgentCreatorLookup re-check tenant scope so a same-ID resource in another tenant cannot piggyback through the ownership shortcut. Docs - CustomAgent.RunnableByViewer comment now matches enforcement: it only blocks Viewer-driven agent-mode runs, not plain RAG/Wiki QA. Tests - New cases for the 503/404 paths, lookup-skip behaviour, cross-tenant superuser bypass, the lookup helpers (not-found, cross-tenant, built-in agent, missing tenant context), and IsSyntheticUserID. |
||
|
|
af23483433 |
fix(auth): honour JWT tenant_id claim in middleware
POST /auth/switch-tenant mints an access token whose `tenant_id` claim
points to the target tenant, but the middleware previously discarded
that claim entirely:
user, err := userService.ValidateToken(ctx, token) // returned only *User
targetTenantID := user.TenantID // always the home tenant
The session therefore stayed scoped to the user's home tenant despite
the supposedly fresh token, making the new endpoint a no-op once
the request hit any tenant-scoped query.
This change widens the contract:
- UserService.ValidateToken now returns (user, activeTenantID, error)
where activeTenantID is the JWT's `tenant_id` claim, falling back to
user.TenantID when the claim is absent (backward compatibility with
pre-RBAC tokens still in the wild).
- middleware/auth.go uses activeTenantID as the default target before
applying any X-Tenant-ID header override, and recomputes
crossTenantSwitch from `activeTenantID != user.TenantID` so the
resolveTenantRole branches stay correct for switch-tenant sessions.
- The non-middleware /auth/validate handler in handler/auth.go absorbs
the new return value (it only cares about user identity).
Tests:
- internal/application/service/user_jwt_test.go pins the JWT->tenant-id
projection (`tenantIDFromClaims`) — claim present beats fallback,
zero/negative/string claims fall back, int64/uint64 are accepted for
test-built tokens.
Required for PR 2 multi-tenant integration tests, where a single user
needs to switch between Owner-in-A and Viewer-in-B sessions and observe
distinct authority on each side.
|
||
|
|
f3d1210577 |
refactor(rbac): address PR 1 review feedback
Tighten correctness, security and observability of the tenant RBAC foundation introduced in 3c4cdd70 in response to internal review. Bugfixes - custom_agents.runnable_by_viewer: drop the GORM `default:true` tag. GORM treats a plain bool's zero value as "unspecified" and silently rewrites it to the column default on insert, which would prevent admins from ever creating an agent with viewer-runtime disabled. The column-level DEFAULT TRUE on the database side still backstops rows inserted by raw SQL / migrations. - TenantMemberRepository.HasAnyMembers: replace `Limit(1).Count(&n)` with `Take(&probe)`. GORM's Count ignores Limit and scans the whole table, which matters because this runs on the auth middleware's hot path for users without a cached membership. - service/user.LoginWithOIDC: include User, Tenant and Memberships in the OIDCCallbackResponse so the OIDC flow returns the same shape as password login (previously User and friends were left nil). Security - middleware/auth.resolveTenantRole: restrict orphan-tenant auto-promotion to the caller's home tenant. Cross-tenant superuser visits and JWTs minted for foreign tenants no longer trigger silent Owner grants. Two regression tests pin the new behaviour. - service/user.synthFallbackMembership: default the synthesised role to Viewer instead of Owner. The login response only feeds UI; the backend re-derives the real role from tenant_members on every request, so showing a Viewer UI is preferable to a misleading Owner UI that would surface admin controls the backend will then 403. - migrations/versioned/000043_tenant_rbac.down.sql: gate the destructive drop behind a `weknora.allow_destructive_migration` GUC so an accidental `migrate down` cannot wipe role data in production. - frontend/src/stores/auth.ts: comment currentTenantRole as rendering-only so future PRs cannot mistake the localStorage-backed value for an authorisation signal. API contract - Memberships in LoginResponse and OIDCCallbackResponse now serialises without omitempty, and buildMembershipsForUser / synthFallbackMembership always return a non-nil slice — frontend can rely on JSON `[]`. - handler/auth.GetCurrentUser includes memberships so /auth/me refreshes recover currentTenantRole after a page reload. Performance - Add TenantService.GetTenantsByIDs (with the matching repo method) and use it inside buildMembershipsForUser to batch tenant-name lookups instead of issuing one query per membership. Observability - Replace the remaining `log.Printf` calls in middleware/auth with contextual logger.Warnf / logger.Infof so role resolution events ride the standard request-id / Langfuse trace plumbing. The successful auto-promote line is tagged `[audit]` for log mining. Tests - internal/application/service/tenant_member_test.go: hand-rolled fakeTenantMemberRepo plus 11 cases covering AddMember validation, EnsureOwner idempotence, "cannot demote/remove the last Owner", same-role no-op, ErrInvalidTenantRole / ErrMembershipNotFound, and the TenantRole.HasPermission lattice. - internal/middleware/auth_role_test.go: fakeMemberService plus 8 cases pinning every branch of resolveTenantRole, including the two H1 regressions (cross-tenant superuser must not auto-promote; orphan auto-promote requires the home tenant). |
||
|
|
7f7a48880f |
feat(rbac): add tenant_members schema and auth wiring (PR 1 of 3, #1303)
Lay the foundation for tenant-level RBAC without enforcing it yet. Schema, types, middleware lookup, and login-flow shape changes ship now; RequireRole/route annotations and member-management endpoints follow in PR 2 and PR 3. Highlights: - New tenant_members table (Postgres + SQLite) with (user_id, tenant_id) unique key and (owner|admin|contributor|viewer) role values. Migration 000043 backfills one row per existing user, marking the earliest-created active user in each tenant as Owner and the rest as Contributor. - KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so PR 2 can wire ownership-based and Viewer-runtime checks. - Auth middleware loads the active TenantMember row, sets a new TenantRoleContextKey, and falls back via three escape hatches: cross-tenant superusers (CanAccessAllTenants) get Admin in the target tenant; orphan tenants (zero members) auto-promote the first authenticating human to Owner; and when tenant.enable_rbac is false (the v1 default) unmatched lookups fail open with Admin to preserve current behaviour. - Register / OIDC provisioning now insert an Owner membership for the registrant via TenantMemberService.EnsureOwner. - LoginResponse renamed tenant -> active_tenant and gained a memberships array. POST /auth/switch-tenant issues a fresh token pair scoped to a target membership; GET /auth/config exposes auth.registration_mode for the frontend register-tab gating. - New config keys: auth.registration_mode (self_serve | invite_only, defaults to self_serve) and tenant.enable_rbac (defaults to false). Both have WEKNORA_* env overrides. - Frontend store persists memberships and exposes currentTenantRole; Login.vue calls /auth/config and hides the Register tab in invite_only mode. Go SDK types updated to match the new LoginResponse shape. |
||
|
|
0e8de6192c |
feat(knowledge-base): validate vector store bindings on create, copy, and delete
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory into the user-facing knowledge-base lifecycle: - POST /knowledge-bases validates the requested vector_store_id against the caller's tenant scope and the engine registry. New error codes ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201) distinguish the typed branches without echoing UUIDs to the client. - GET / POST / PUT / PUT-pin responses embed the bound store's display metadata (name, source, engine_type, status) without exposing any connection credentials. Cross-tenant shared KBs receive a suppressed payload (vector_store_id stripped, source="shared") so operator-chosen store names cannot be enumerated across tenants. - POST /knowledge-bases/copy synchronously rejects clones whose target has a different embedding model or vector store, before the async clone task is enqueued. The async clone worker re-applies the same checks for defense in depth. - DELETE /vector-stores/:id refuses to remove a store with bound KBs, inside a transaction that row-locks the store on PostgreSQL and serializes via WAL on SQLite. unregister-from-registry is wrapped in defer/recover so a panic surfaces as a structured warning instead of silently leaking a stale engine. - vector_store_id is immutable after creation. The GORM <-:create tag blocks every ORM update path; the service-layer DTO omits the field entirely; a reflection-based regression test catches any future maintainer who adds it back to either layer. - Empty-string vector_store_id is normalized to nil at both the create path and inside SharesStoreWith, so rows persisted by callers that did not run Normalize first cannot trip false same-store comparisons. Part of #993. Depends on #994 and #1310. |
||
|
|
dc634e863a |
feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling
Two encryption gaps and one fail-loud regression, fixed together because
they are inseparable in practice.
At-rest encryption gaps
-----------------------
Before this change, only Tenant.APIKey, Tenant.Credentials.WeKnoraCloud.AppSecret,
Model.Parameters.APIKey/AppSecret, and WebSearchProviderParameters.APIKey
were encrypted with SYSTEM_AES_KEY. MCP and DataSource were missed:
- MCPAuthConfig.Value/Scan went through bare json.Marshal/Unmarshal, so
api_key and token were written to mcp_services.auth_config (json) in
plaintext regardless of whether SYSTEM_AES_KEY was set.
- DataSource.Config (types.JSON) is a byte passthrough, so connector
secrets (Notion token, GitHub PAT, Feishu app_secret, ...) inside the
Credentials map were stored in plaintext too.
Now both follow the ModelParameters / WebSearchProviderParameters pattern.
Encryption operates on a local copy to avoid mutating the caller's
in-memory struct (otherwise the next read of the same *MCPAuthConfig
would surface ciphertext as if it were the API key). For DataSource the
ToJSON ↔ ParseConfig pair is the only application boundary that
materializes credentials, so encrypting there is sufficient and avoids
having to fork the generic types.JSON.
Backward compatibility
----------------------
DecryptStoredSecret returns rows without the enc:v1: prefix verbatim, so
historical plaintext records keep working. Next write re-saves them
encrypted. No migration step.
Graceful Scan-time degradation
------------------------------
The strict "fail loud on key rotation" behaviour of DecryptStoredSecret
turned into a footgun once it ran from GORM Scan: a single broken row
caused ListModels / ListMCPServices / ListDataSources to return zero
results, hiding all other rows and giving the operator no way to find
which credential needed re-entering.
New utils.DecryptStoredSecretLenient: same logic, but on failure returns
("", false). The strict variant is kept for hot paths that are about to
actually USE the credential (so a rotated key surfaces loudly there
rather than silently degrading to "missing api key").
All Scan paths switch to lenient + per-failure log line:
ModelParameters.Scan, WebSearchProviderParameters.Scan,
MCPAuthConfig.Scan, DataSource.ParseConfig, CredentialsConfig.Scan.
On lenient failure the field blanks out (never ciphertext), the row
still loads, the DTO reports configured=false so the UI shows "not
configured", and any upstream call attempted with the empty value fails
with a clear "missing api key" rather than sending ciphertext as the key.
Tenant.AfterFind (Tenant.APIKey) intentionally stays strict — it is the
primary HTTP auth credential and a silent empty value would change auth
semantics ambiguously.
|
||
|
|
7643dd457e |
refactor(credentials): introduce /credentials subresource pattern
Replace the legacy "redacted placeholder + Clear* boolean" pattern with
dedicated per-resource credential subresources across MCP services,
Models, Web search providers, and Data sources.
Why
---
The previous design had three problems:
1. Main PUT body carried secret fields. The frontend echoed back a
redacted "***" placeholder, and a fragile MergeUpdate / IsRedactedOrEmpty
defense in the service layer tried to detect "user did not change this"
vs "user wants to clear this". A regression in that defense (or a new
frontend forgetting it) silently overwrites the stored secret with the
placeholder.
2. The "remove this credential" UX was a red checkbox under a pre-filled
password input. Three intents (preserve / replace / clear) collapsed
onto one field, and credential changes were bundled with unrelated
config edits in the same submit. Users wiped working keys by mistake.
3. Secret presence was inferred from "did the response come back with a
'***' placeholder", which couples the contract to a magic string.
Design
------
Each of the four resources now exposes:
PUT /{resource}/{id}/credentials # write one or more fields
DELETE /{resource}/{id}/credentials/{field} # clear a single field
"Is this configured?" metadata travels on the main resource response as
a typed map (dto.MCPServiceResponse.Credentials etc.) — no separate GET
endpoint. The frontend reads the configured boolean from the main GET
and never sees secret values at all.
Main PUT endpoints now ignore any secret fields in the request body and
log a deprecation warning if they appear, so legacy clients fail loudly
rather than overwriting silently.
Frontend
--------
- New reusable <CredentialResource> component renders a three-state card
(unconfigured / configured / editing) and drives the dedicated
endpoints. Used by MCP, Model, Web search; DataSource has a bespoke
card with the same behaviour because its credentials are a single
atomic per-connector map.
- Cancel from edit mode now restores state synchronously from the meta
prop. The previous async refresh() was a no-op while state was still
'editing', leaving the input frozen open.
- Remove is single-click + toast. The danger-themed button is the
deterrent; a modal confirm adds friction without adding safety (the
plaintext is irrecoverable client-side either way — recovery means
re-typing).
DTOs (internal/handler/dto/) are deliberately separate from the GORM
models so "no secret in response" is a compile-time invariant: a future
contributor cannot leak a secret without explicitly adding the field to
the DTO, which is review-able in a single diff.
Storage is unchanged — credentials still live in the existing jsonb /
parameters columns. No schema migration.
Cleanup
-------
- types.MCPService / types.Model / types.WebSearchProviderEntity /
types.DataSourceConfig: drop ClearAPIKey / ClearToken / ClearAppSecret
/ ClearCredentials boolean fields, MergeUpdate(), RedactSensitiveData().
- utils/types/secret.go: drop PreserveIfRedacted / IsRedactedOrEmpty.
RedactedSecretPlaceholder constant is retained because VectorStore
still uses the old pattern and is out of scope here.
- Frontend hasExistingApiKey / clearApiKey / convertToLegacyFormat
redaction handling removed; i18n keys renamed secret -> credential.
|
||
|
|
d869cc1ff6 |
fix: redact sensitive fields and prevent credential loss on edit
Closes #988 Unify the "write-only secrets" pattern across MCP services, Models, WebSearch providers, and DataSources. Fix two related defects: 1. GET responses leaked plaintext credentials — GetMCPServiceByID while List masked, all Model / WebSearchProvider / DataSource GET responses unconditionally. 2. When the frontend echoed a masked placeholder back on save, the update path overwrote the stored secret with the mask — silently wiping the real credential. MCP surfaced this most visibly: ListMCPServices masked Token to "first4****last4" and McpServiceDialog round-tripped that masked string back into the DB. - internal/types/secret.go introduces the shared sentinel RedactedSecretPlaceholder = "***" and two helpers: - IsRedactedOrEmpty(s): true for "" or the sentinel - PreserveIfRedacted(incoming, existing): merge predicate - Each resource adds a RedactSensitiveData (mutates-in-place) or Redacted (returns-a-copy) method and a MergeUpdate contract: empty or placeholder preserves, any other value replaces, and a write-only Clear* bool flag (gorm:"-", json omitempty) wipes with precedence over preserve / replace. Clear* never persists. - Every GET / List / Create / Update response path now redacts. VectorStore's pre-existing "***" literal is refactored onto the shared constant for consistency. All four dialogs — McpServiceDialog, ModelEditorDialog, WebSearchSettings, DataSourceEditorDialog — never pre-fill secret inputs. The stored-state signal is collapsed into the input placeholder: "•••••••• (Enter new value to replace)" when the server reports a stored value, falling back to the connector-specific placeholder otherwise. The input itself carries the "something is there" cue, matching how password dots render. An opt-in "Remove this credential" checkbox appears in edit mode when a value is stored; checking it disables the input and triggers a danger-themed confirmation dialog before submit. DataSource rolls the clear up to a connector-level "Remove all credentials" flag since its credentials are per-connector atomic bundles (e.g. Feishu app_id + app_secret). i18n updates across en-US, zh-CN, ko-KR, ru-RU: secret.storedPlaceholder / clearHint / confirmClearTitle / confirmClearBody. - MCP update handler's manual map-to-struct projection was silently dropping clear_api_key / clear_token / custom_headers on partial updates. Pull them off the map explicitly. - MCP configChanged detection now snapshots URL / StdioConfig / TransportType before the merge so connection recycling still triggers on real transport changes (not just whenever AuthConfig is present). - MCP CustomHeaders: nil preserves existing, non-nil replaces. Was always overwriting, silently wiping headers on any auth-bearing update. - GetMCPServiceByID now redacts (was plaintext while List masked). - WebSearchProvider update preserves Name / Description when empty instead of clobbering them. - WebSearchProvider TestProviderRaw rejects the "***" placeholder up front with an actionable message rather than forwarding it to the upstream provider API as a literal key. - Model builtin hideSensitiveInfo now preserves non-secret metadata (SupportsVision, Provider, ExtraConfig, InterfaceType, ParameterSize, EmbeddingParameters) that were previously zeroed out for builtin model cards. - Replace local equalStringSlices / equalStringMaps helpers with stdlib slices.Equal / maps.Equal (Go 1.21+). - internal/types/{secret,mcp,model,web_search_provider,datasource}_test.go: table-driven coverage of merge, redact, clear, and write-only flag invariants for every resource. - internal/application/service/mcp_service_test.go: full service layer with a fake repo; covers ListMCPServices / GetMCPServiceByID redaction, UpdateMCPService partial-update honoring of Clear*, CustomHeaders preserve-on-nil / replace-on-non-nil, and clear-flag precedence over submitted values. Additive only. No changes to request or response JSON shape beyond the write-only Clear* flags (omitted by default via omitempty). No DB schema changes. No SDK struct field removals. Existing clients that don't know about the new flags continue to function unchanged. - DataSource.Config jsonb at-rest encryption. Pre-existing residual risk; tracked as a follow-up. - Renaming VectorStore.MaskSensitiveFields to match the new RedactSensitiveData / Redacted naming convention. Left alone to minimize diff; a follow-up rename PR can unify. |
||
|
|
8f4e5a459f | fix(session): scope session access by user | ||
|
|
7478e3cddb |
refactor(retriever): introduce factory for KB-scoped retrieve engine resolution
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across seven services into two factory functions with tenant-ownership verification: CreateRetrieveEngineForKB (synchronous) and CreateRetrieveEngineFromPayload (async task handlers). Promotes GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry. Extends KBDeletePayload / IndexDeletePayload with an omitempty VectorStoreID snapshot. Design highlights - Factory verifies tenant ownership of the resolved store (defense-in-depth) so a gap in PR3 validation or a tampered Asynq payload cannot cross tenants. Cross-tenant attempts return ErrVectorStoreForbidden with a structured log entry. - Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound, ErrVectorStoreForbidden) let async handlers classify non-retryable failures as asynq.SkipRetry. - nil and empty-string pointers for vectorStoreID normalize to "unbound" so callers never send an empty UUID to GetByStoreID. - resolveBoundEngine constructs engineInfos directly (with slices.Clone of Support()) instead of going through NewCompositeRetrieveEngine. The latter only reads from the byEngineType map and cannot reach DB stores in byStoreID -- multiple stores can share the same engine type. Unbound fallbacks still reuse NewCompositeRetrieveEngine so the tenant effective-engines path is byte-for-byte unchanged. - DeleteKnowledgeBase now reads the KB before soft-delete so the enqueued KBDeletePayload carries a VectorStoreID snapshot; once the row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT per KB-delete request (a rare admin operation). - enqueueIndexDeleteTask gains a vectorStoreID *string parameter that the single caller (tag.go) populates from the owning KB. The worker (ProcessIndexDelete) validates ownership via the factory. - cleanupKnowledgeResources loads the KB so bound-store cleanup routes correctly. If the load fails it warns and falls back to tenant effective engines; orphan vectors become observable in logs instead of disappearing silently. - DeleteKnowledges (batch) stays on the tenant effective-engines path -- batches can span stores and multi-store fan-out is PR4 scope. Noted inline. Merge safety All knowledge bases have VectorStoreID = NULL at merge time (introduced nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()), which is the existing code path for these KBs. Async payloads from before this change decode to VectorStoreID = nil (omitempty) and take the same fallback. Testing - go build ./... - go vet ./... - go test -race -count=1 ./internal/application/service/retriever/... covers unbound (nil/empty), bound, cross-tenant, unregistered store, ownership infra error, legacy payload shapes (missing field/null/ empty string/nil), tampered payload, and parallel invocation. - grep -r NewCompositeRetrieveEngine internal/application/service returns zero results outside factory.go/composite.go. Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994. |
||
|
|
b00bc84f35 |
fix(agent): pass reasoning_content back to providers that require it
MiMo and DeepSeek V3.2/V4 reject multi-turn requests in thinking mode when the prior assistant message lacks reasoning_content with HTTP 400: "The reasoning_content in the thinking mode must be passed back to the API." Agent's ReAct loop is the worst-case scenario — every round produces tool calls, exactly the case DeepSeek's docs specify reasoning_content MUST participate in subsequent context. Plumb reasoning_content through the full assistant-message round-trip: - chat.Message / types.ChatResponse / types.AgentStep gain a reasoning_content field (AgentStep persists via the existing Message.AgentSteps jsonb column, no migration needed). - streamLLMToEventBus accumulates reasoning chunks into result.ReasoningContent and surfaces it on the round's ChatResponse. - engine.runReActIteration writes it onto AgentStep so cross-turn replay preserves it. - observe.appendToolResults attaches it to the same-turn assistant message; agent_history.buildAssistantHistoryMessages does the same on cross-turn replay. - RemoteAPIChat.ConvertMessages forwards it on assistant turns to openai.ChatCompletionMessage.ReasoningContent (already supported by go-openai); providers that don't recognize the field ignore it. Tests cover the three boundaries: ConvertMessages serializes it for upstream, appendToolResults preserves it within the same turn, and buildAssistantHistoryMessages replays it across turns. Scope is intentionally limited to Agent mode — KnowledgeQA's chat pipeline and Anthropic's signed thinking_blocks are separate fixes that require schema changes (rendered_content / thinking_blocks columns). Fixes #1302 |
||
|
|
64b20a2d87 |
feat(agent): support dedicated model for query understanding step
Quick-answer (RAG) agents can now configure a separate chat model for the query-understanding stage (rewrite + intent classification), decoupling it from the main conversation model so users can route the lightweight rewrite call to a cheaper / faster model. - Add CustomAgentConfig.QueryUnderstandModelID and plumb it through PipelineRequest and ChatManage.Clone. - query_understand plugin prefers QueryUnderstandModelID on the text-only path; falls back to ChatModelID if the configured model cannot be resolved (with a warn log). Multimodal path is unchanged to keep vision-capable model selection intact. - AgentEditorModal exposes a ModelSelector under the existing Query Rewrite block; empty means reuse the main chat model. - Add i18n strings (queryUnderstandModel / placeholder / desc) for zh-CN, en-US, ko-KR, ru-RU. - Extend Go SDK AgentConfig with the new field. |
||
|
|
3f9b09e306 |
fix(pipeline): make data-analysis stage opt-in per agent (#1244)
The legacy in-pipeline DuckDB SQL data-analysis stage used to run on every quick-answer RAG request whose retrieved chunks included a CSV/Excel file, adding one extra LLM round-trip (~3s) to generate a SQL query that most plain Q&A users never wanted. There was no way to disable it. Introduce a per-agent DataAnalysisEnabled flag (default off), wire it through PipelineRequest, and gate the DATA_ANALYSIS stage on it. Surface the toggle in the agent editor for quick-answer agents with at least one knowledge base attached. Refs: https://github.com/Tencent/WeKnora/issues/1244 |
||
|
|
cacca049d9 |
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base detail page — parse status, source/channel, and updated time range — and rework multi-select to no longer cause the card title to jitter on hover. Backend - Introduce types.KnowledgeListFilter to aggregate optional filter dimensions (tag, keyword, file_type, parse_status, source, updated_from/to) and switch ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept it instead of a growing positional parameter list. - The ListKnowledge HTTP handler accepts new parse_status, source, start_time and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS" and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type column to stay consistent with file_type semantics; other source values match the channel column. - Update the four other callers (agent_service, initialization) to pass an empty filter struct, preserving prior behavior. Frontend - Add three controls in the doc-filter-bar (status select, source select, date-range picker with future-date disabled) wired through getKnowled / listKnowledgeFiles into the new backend params. - Replace the hover-triggered card checkbox with an explicit "批量管理" mode (mirrors the session list UX): in card view the checkbox only renders while batch mode is on, entered via the per-card "..." menu; the list view keeps its leading checkbox column. Switching from list to grid auto-enables batch mode when something is already selected, so the selection stays visible. - DocumentBatchBar now stays open whenever batch mode or selection > 0, and its "取消选择" button both clears the selection and exits batch mode. API surface sync - Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml). - Update docs/api/knowledge.md with the new query parameters. - Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the Go SDK; the existing ListKnowledge keeps its signature. i18n - New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing menu.batchManage / batchManage.cancel for the multi-select strings. |
||
|
|
1f5970b67e |
refactor(agent): rebuild multi-turn history from DB, drop llmcontext layer
The Agent ran a parallel context cache (Redis/in-memory) on top of the messages table to feed multi-turn history into the LLM. That dual-write caused subtle drift (e.g. compression diverging from DB, system-prompt swaps lost on restart) and required a separate ClearContext path on session/IM clear. Make the messages table the single source of truth: - Add service.LoadAgentHistory: rebuilds chat.Message history per turn from the persisted messages, expanding AgentSteps into proper OpenAI assistant_with_tool_calls + tool messages and replaying the canonical final answer (with <think> blocks stripped). final_answer tool calls are filtered to avoid duplicating the trailing answer. - Make AgentEngine stateless across turns: drop ContextManager / sessionID cache plumbing from the engine, agent service, and CreateAgentEngine signature. The engine only uses sessionID for logging/event emission. - Wire AgentQA to load history from DB on demand using HistoryTurns (default 5) when MultiTurnEnabled, otherwise run with empty history. - Delete the llmcontext package (ContextManager interface, Redis/memory storage, factory) and the SessionService.ClearContext API path; IM /clear and session message clear no longer need to invalidate cache. Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with empty AgentSteps just produce the single canonical assistant message), and Agent-mode turns now consistently see prior tool calls and results. |
||
|
|
1110615300 |
feat(web-search): add SearXNG provider (#1166)
支持对接自建/公共 SearXNG 实例作为网络搜索引擎,缓解免费搜索引擎在国内 网络环境下访问受限的问题。 - types: 新增 WebSearchProviderTypeSearxng 与 BaseURL 参数字段; 类型元数据新增 RequiresBaseURL,前端可动态渲染 Instance URL 表单。 - infrastructure/web_search/searxng.go: 调用 /search?format=json,强制 utils.ValidateURLForSSRF 校验 base_url,可选 api_key 透传给反代鉴权。 - service: isValidProviderType 与参数校验接入 searxng。 - container: 注册 NewSearxngProvider 工厂。 - frontend: WebSearchSettings 表单根据 requires_base_url 渲染 Instance URL 输入框;编辑回填、free 判定同步更新。 - docker: 新增可选 searxng 服务(profile=searxng/full),附带最小化 settings.yml(启用 JSON 格式、关闭 limiter、关闭遥测), docker-compose 默认 SSRF_WHITELIST 包含 searxng 容器名。 - .env.example: 补充 SEARXNG_PORT / SEARXNG_SECRET 说明。 Closes #1166 |
||
|
|
5510ea8f5a |
feat(agent): human-in-the-loop approval for MCP tool calls (#1173)
Add an opt-in human approval gate so Agent runs pause before executing MCP tools that operators flag as dangerous, surface an approval card in the chat UI, and only resume after the user approves (optionally with edited args) or rejects. Backend - New mcp_tool_approvals table + repo/service to mark per-tool approval required (PG migration 000042 + sqlite init). - approval.Gate coordinates RequestAndWait / Resolve with sync.Once delivery, configurable timeout, and Redis Pub/Sub fan-out so multi- replica deployments work without sticky sessions. - MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx (without the per-tool 60s timeout) for the wait, and re-derives a fresh 60s exec ctx after approval so CallTool keeps a full window. - New SSE response types (tool_approval_required / _resolved) and EventBus events plumb approval state to AgentStreamDisplay. - REST: list/set per-tool approval flag, resolve pending approval. - Configurable via agent.tool_approval_timeout_seconds (yaml) or WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go duration). Frontend - MCP settings: per-tool "require approval" switch on the test panel. - Chat: ToolApprovalCard renders the pause point with editable JSON args, validation feedback, mm:ss countdown that turns warning/danger near deadline, and a resolved state that retains context. - i18n strings added for zh-CN / en-US / ko-KR / ru-RU. Docs - docs/zh/mcp-approval.md covering behavior, config, API, deployment considerations (Redis cross-instance, restart limitations). |