dc8d033b4d68639f6b45d1bcd647d5aa4e4121e6
550 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wizardchen
|
dc8d033b4d |
feat(organization): transition membership model to tenant-level and enhance invite functionality
Updated the organization membership model to operate at the tenant level, aligning with Plan 3. This change introduces new interfaces and modifies existing ones to accommodate tenant-centric operations, including the `InviteMemberRequest` and `TenantInviteCandidate`. The search functionality for inviting members has been updated to reflect this shift, allowing for tenant-based searches instead of user-based. Additionally, internationalization files have been updated to ensure consistent messaging across languages regarding tenant membership and roles. Refs: #1303 |
||
|
wizardchen
|
ff1f03ce0d |
feat(rbac): daily retention sweep for audit_logs (default 90 days)
The audit_logs table grows monotonically until something purges it. PR 6 explicitly listed retention as out of scope for v1; this picks that up. Adds an `audit.retention_days` config (default 90, 0 disables) and a small background goroutine `AuditLogRetentionRunner` that fires once ~10 minutes after boot and then every 24h, calling `AuditLogService.Purge` which DELETEs rows older than the cutoff in a single statement. The dependency surface is intentionally minimal — no robfig/cron, no asynq, no migration — because retention has no wall-clock alignment requirement. Wiring is the same shape as the existing data source scheduler: container `Provide(NewAuditLogRetentionRunner)`, an `Invoke` that calls `Start`, and a `ResourceCleaner.RegisterWithName` for graceful shutdown. Defaults preserve operator intent: - `audit:` section omitted from YAML -> retention_days=90. - explicit `audit.retention_days: 0` in YAML -> purge disabled. - `WEKNORA_AUDIT_RETENTION_DAYS=N` env override (incl. N=0). - `retention_days < 0` is rejected by ValidateConfig. Tests cover the service Purge contract (no-op when disabled, cutoff math, error propagation) and the runner lifecycle (Start no-op when disabled, idempotent Start/Stop, ticker cadence, runOnce swallows errors). 10 new tests, all green. docs/rbac.md updated with the new YAML / env / behaviour. Refs: #1303 |
||
|
wizardchen
|
a16fe83034 |
fix(rbac): pin organization owner tenant in DB instead of deriving at runtime
Plan 3's isOwnerTenant resolved "is this tenant the owner tenant?" by looking up the owner user's CURRENT tenant via userRepo. That breaks silently the moment the owner user is moved to another tenant: the formerly-protected owning tenant becomes removable (can be kicked from OTM, role-downgraded), and the new tenant the owner moved to is treated as untouchable even though it isn't even in OTM. Promote owner-tenant to a first-class column on `organizations`: - New migration 000046 adds owner_tenant_id BIGINT NOT NULL with index. Backfill: (1) owner user's current tenant when known, (2) earliest Admin tenant in OTM as fallback for orphans, then (3) RAISE EXCEPTION on anything still NULL — refuse to silently ship a bogus value. SQLite init schema mirrors the column. - CreateOrganization now sets OwnerTenantID at create time so the pinned value is correct for all new orgs. - isOwnerTenant becomes a pure equality check on the persisted column — no userRepo lookup, no error tuple. Fail-closed on a zero column value (legacy data not yet backfilled or unit tests bypassing the migration) so the worst case is "membership table frozen for that org" rather than "owner tenant is removable", matching the prior fail-closed-on-error semantics. - RemoveTenantMember / UpdateTenantMemberRole simplify to a single bool check. |
||
|
wizardchen
|
f6684f226b |
fix(rbac): allow source-tenant Admin+ to manage their cross-tenant shares
Plan 3 of #1303 lifts share ownership to the tenant level, but RemoveShare / UpdateSharePermission still gated on the original sharer user. If that user left the tenant or moved on, the source tenant lost the ability to revoke its own share — only the target org's admin could clean up, which is governance theatre rather than ownership. Extend the authz envelope on KB-share and agent-share mutations: (1) original sharer (unchanged); (2) source-tenant Admin+ — new, encodes "ownership lives in the tenant" by reading callerTenantRole from ctx; (3) target-org admin (unchanged). Factored out as kbShareService.callerCanManageShare so Update and Remove cannot drift apart. |
||
|
wizardchen
|
eda6b1011a |
fix(rbac): close org-tenant gate gaps from Plan 3 review
Three follow-ups from #1350 review: 1. DELETE /organizations/:id/members/:tenant_id was un-gated, so a tenant Viewer could remove their own tenant from an org via the members endpoint and bypass the Admin+ requirement that POST /:id/leave already enforces. Gate it with g.Admin() — symmetric with /leave. 2. isOwnerTenant() returned (false, err) when the owner user lookup failed, and the two call sites only checked the bool. A deleted / unreadable owner user would silently fall through and let admins remove or re-role the would-be owner tenant. Make the helper fail-closed: any lookup error now returns (true, err), and both call sites refuse the destructive op on error. 3. POST /:id/request-upgrade was un-gated. Approval of the request changes the whole tenant's org role, so it is in the same trust tier as join/leave/create. Add g.Admin(). |
||
|
wizardchen
|
928f5abe3f |
feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303)
Before this change, the `organization_members` table keyed on user_id — joining an org as a Viewer in tenant T meant only that one user got visibility on shared KBs/agents, while the rest of T (including its Owner) saw nothing. The mismatch surfaced as confusing UX bugs: "my colleague shared this with us, why can't I see it?" This change replaces `organization_members` with `organization_tenant_members`. Membership and Org-RBAC role now key on (org_id, tenant_id); `representative_user_id` stays as a display-only label of the user who first brought the tenant in. Permission resolution gains a 3-D cap so tenant-RBAC stays honest under cross-tenant sharing: effective = min(share.Permission, tenant_org_role, tenant_role_cap) where tenant_role_cap pins tenant Viewers to OrgRoleViewer regardless of the org-level grant. A Viewer in their own tenant cannot edit a shared KB even if the org granted Editor — preserves the "Viewer = read-only everywhere" promise. Migration 000045 backfills `organization_tenant_members` from the old table by collapsing each (org, tenant) group to its max role, then renames the old table to `organization_members_pre_plan3` for safe rollback. Sqlite mirror (Lite mode) is fresh-bootstrap: the init schema replaces the old table directly. Routes: member-management URLs change from `:user_id` to `:tenant_id`. Org create / join / leave now require Admin+ in the caller's tenant — joining an Org affects everyone in T, so the decision can't sit with a Viewer. Frontend: OrganizationSettingsModal + store update to address members by tenant_id. |
||
|
wizardchen
|
1e8b5c8d40 |
refactor(api): drop deprecated tenant ConversationConfig field and KV write path
The Tenant.ConversationConfig field was marked deprecated when
CustomAgent (builtin-quick-answer) replaced the per-tenant defaults
path, but the JSONB column, GET/PUT KV handlers, write helpers and
read fall-throughs all stayed wired. This had two concrete failure
modes:
- Input-field.vue silently PUT-ed per-user model picks back into
/tenants/kv/conversation-config, which requires Admin+ — Viewers
and Contributors got a 403 every time they switched chat models.
- The same write also overwrote the tenant-shared default for every
other user, turning a per-user UX into a per-tenant footgun.
Remove the field, the struct, the validate helper, the PUT case and
the two dead Summary helpers in session/helpers.go. KnowledgeSearch
loses its tenant-conversation fallback (one chunk-tool reader); the
global config + hardcoded defaults still cover it. session_agent_qa
already required agents to declare their own rerank_model_id, so no
behaviour change there.
Frontend follows:
- Input-field.vue: per-user 'last selected chat model' moves to
localStorage, no API call.
- AgentEditorModal: defaults for builtin-quick-answer creation now
come from prompt-templates (system/context/rewrite/fallback) and
retrieval-config (top_k / threshold), the two endpoints that
actually own those settings now.
- AgentSettings.vue: orphan settings page with no router/import
referencing it; deleted.
- api/system: getConversationConfig / updateConversationConfig and
the ConversationConfig interface go away.
Net: 137 insertions, 2761 deletions.
|
||
|
wizardchen
|
766a6c482f |
feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303)
PR 1-5 of the multi-tenant RBAC series enforce the role matrix; what they don't do is keep a durable record of who did what. Today the RBAC middleware logs "[rbac] role insufficient ..." to stderr - perishable, not searchable. Operators cannot answer "who removed Alice from tenant T-foo on Tuesday" or "did we 403 a probing user 200 times last hour" after the fact. This PR adds a generic per-tenant audit log that: - Captures mutating RBAC actions (member add/remove/role-change/leave) durably from the tenant_member service after the repo write succeeds. - Captures enforcement rejections (RequireRole / RequireOwnershipOrRole denies under EnableRBAC=true) via a gin-context-injected hook. - Sliding-window dedup (1 minute) on (tenant, actor, action, path) for denied events so a probing client cannot flood audit_logs. - Default-on - no extra env flag. Schema is generic enough for future KB / agent / datasource events to reuse via new action namespaces without another migration. Schema (migration 000044) mirrors wiki_log_entries' shape: a single audit_logs table with action, target_type/id, request path/method, outcome, and a JSONB details column. Three indexes: (tenant_id, id DESC) for the per-tenant cursor feed, actor_user_id for "what did Alice do", (tenant_id, action) for both the action filter and the dedup lookup. Sqlite mirror added for Lite mode. Wiring is nil-safe: the audit field on tenantMemberService and the gin-context lookup in rbac.go both tolerate a nil service so existing tests construct without a stub and dormant deployments degrade gracefully if the provider is misconfigured. The query API is GET /tenants/:id/audit-log, gated by Admin+ on top of the existing PathTenantMatch (denial histories should not surface to ordinary members). Cursor-paginated by descending id (sidesteps the duplicate-timestamp tie-breaking that BIGSERIAL trivially solves). Frontend: TenantMembers.vue gets a t-tabs wrapper with Members (existing) and Audit log (new) tabs; the audit tab is gated to Admin+ to mirror the server, lazy-loads on first open, paginates via the next_cursor pattern, and renders coloured chips per action/outcome so an operator can scan for anomalies. i18n keys added in all four locales. Tests: 14 new tests across service (6), middleware (4), and handler (4). Service tests pin the dedup window behaviour (writes again after expiry, dedup is per-actor-per-path, degrades on lookup error). Middleware tests pin both reject sites' audit hook and the "dormant mode does not fire" + "nil audit does not panic" invariants. Handler tests pin the cursor envelope and that an empty page returns next_cursor=0. Out of scope: per-action ACL on the audit log, retention/TTL job, KB/agent/datasource event wiring (schema supports it - just no emitters yet), async writes, frontend filter UI. |
||
|
wizardchen
|
d42a41b784 |
feat(rbac): per-KB ownership for knowledge / chunks / wiki pages (#1303)
PR 1–4 of the multi-tenant RBAC series gated KB-level and Agent-level
mutations behind a "creator OR Admin+" check via OwnedKBOrAdmin /
OwnedAgentOrAdmin. The layer underneath — knowledge entries, chunks,
wiki pages — was still gated only by Contributor, which is wider than
intended: any tenant Contributor could edit/delete documents in a
private KB they didn't own. PR 2 / wiki author left explicit follow-up
breadcrumbs at the three call sites in router.go acknowledging this gap.
This PR closes those breadcrumbs by reusing the existing scaffolding:
- KnowledgeService.GetOwningKBCreatorID walks knowledge_id -> kb_id ->
KB.CreatorID, tenant-scoped through GetKnowledgeByID. The chain lives
in the service layer so the handler-side lookups stay pure adapters.
- Three new CreatorLookup closures (KBCreatorLookupFromKnowledgeID,
KBCreatorLookupFromKnowledgeIDParam, KBCreatorLookupFromKBPath) are
added in handler/rbac_lookups.go alongside the existing
KBCreatorLookup / AgentCreatorLookup. Each translates the repository
sentinels into middleware.ErrResourceNotFound and re-checks tenant
scope as defence-in-depth, mirroring KBCreatorLookup exactly.
- router/rbac.go gains three guard factories — OwnedKnowledgeKBOrAdmin,
OwnedChunkKBOrAdmin, OwnedWikiKBOrAdmin — wired through the same
rbacGuards bundle.
- Mutating routes in RegisterKnowledgeRoutes / RegisterChunkRoutes /
RegisterWikiPageRoutes switch from g.Contributor() to the new
ownership guards. Cross-KB batch operations (POST /knowledge/move,
POST /knowledge/batch-delete, PUT /knowledge/tags) stay
Contributor-gated since there's no single owning KB to check
against. KB-scoped uploads (POST /knowledge-bases/:id/knowledge/...)
reuse the existing OwnedKBOrAdmin because the URL :id is the KB id
directly. The chunks.DELETE("/by-id/:id/questions") route addresses
chunks by their own id with no knowledge id in the URL, so it stays
Contributor-gated as a documented carve-out.
No schema changes: documents inside a shared KB are KB-owned content,
not personal uploads — that's how the product is positioned and matches
the original "per-KB-ownership granularity" wording from the PR 2
breadcrumbs. Legacy/built-in KBs with empty CreatorID surface as
tenant-owned (Admin+ only), same rule as built-in agents.
Tests:
- 9 new test cases in handler/rbac_lookups_test.go covering the three
new lookups: NotFoundMapsToSentinel (each repository sentinel),
CrossTenantIsHiddenAsNotFound, OwnerMatchReturnsCreatorID,
MissingTenantContext (fail-closed), and ChunkRouteShape (param-name
regression guard).
- 5 new test cases in service/knowledge_owner_chain_test.go covering
the chain helper: knowledge-not-found, kb-not-found, nil-kb-without-
err, happy path, and legacy KB with empty CreatorID.
Build, vet, and the targeted ./internal/middleware/...
./internal/handler/... ./internal/router/...
./internal/application/service test runs are clean. Pre-existing
TestCreateStore_* / TestOssEnsureBucket_CreateFails failures on main
are unrelated and reproduce on a clean checkout.
|
||
|
wizardchen
|
543e9709be |
fix(rbac): address PR 3 + leave-endpoint review feedback
Security - Reject URL :id when it does not match the caller's active tenant on every /tenants/:id/members/* and /tenants/:id/leave endpoint. Without this an Owner of tenant A could drive operations on tenant B simply by changing the URL, because RequireRole resolves the caller's role against the active tenant from context, not the URL param. The cross-tenant superuser carve-out mirrors middleware/rbac.go (CanAccessAllTenants + EnableCrossTenantAccess required). - AddMember no longer records the synthetic "system-<tenantID>" caller (X-API-Key auth path) as invited_by, matching the same filter KB.CreatorID got in PR 2. - AuthHandler now logs a loud one-shot error when constructed with a nil/incomplete config so a wiring bug that silently disables the invite_only gate is visible at boot. Race fixes - New repo helpers DemoteOwnerAtomically / RemoveOwnerAtomically wrap the count-other-owners + write step in a transaction that holds SELECT … FOR UPDATE locks on every other active Owner row. The service routes Owner demotion / Owner removal through them, closing the TOCTOU window where two concurrent demotes of two different Owners could each observe count=2 and both commit, leaving the tenant ownerless. Performance - ListMembers hydration now batches user lookups via a new GetUsersByIDs interface method (one SELECT … WHERE id IN (…) instead of N round-trips). Failure degrades to "no email/username" rather than dropping rows. Frontend - canManage also fires for cross-tenant superusers (authStore.canAccessAllTenants), mirroring the backend carve-out so the UI controls match what the server actually allows. Tests - Cross-tenant URL guard regression cases for List / Add / Update / Remove / Leave. - Superuser bypass tests (flag on -> service reached, flag off -> 403). - Batched lookup test ensures GetUsersByIDs is called exactly once and GetUserByID is never used as a fallback. - Synthetic caller invited_by filter regression test. - Service-layer atomic last-Owner regression tests for both UpdateRole and RemoveMember. |
||
|
wizardchen
|
8b71def60b |
fix(rbac): address PR 2 review feedback
Middleware - RequireOwnershipOrRole now skips the creator lookup entirely when EnableRBAC is off, eliminating a hidden per-request DB roundtrip in the rollout window. - New ErrResourceNotFound sentinel lets lookups signal "not visible to this tenant"; the middleware passes the request through so handlers can return a real 404 instead of a misleading 403. - Transient lookup failures now surface as 503 (was 403) so monitoring can tell denied-by-policy from cannot-verify-right-now. - Cross-tenant superusers (CanAccessAllTenants) bypass tenant role gates, restoring DELETE /tenants/:id and ResetAPIKey for org operators that resolve to Admin in foreign tenants. - Construction-time warning fires once when cfg is nil/incomplete, so a wiring bug that disables enforcement globally is visible at boot. Route coverage - Add role guards to FAQ, KnowledgeTag, WikiPage, Skill, WebSearch, WebSearchProvider, WeKnoraCloud, Session, Chat, and Message route groups so the matrix actually matches the PR description. Service layer - KB and agent creators are no longer recorded as the synthetic system-<tenantID> user used by the X-API-Key path; resources stay tenant-owned. New types.IsSyntheticUserID helper centralises the prefix check. - KBCreatorLookup and AgentCreatorLookup re-check tenant scope so a same-ID resource in another tenant cannot piggyback through the ownership shortcut. Docs - CustomAgent.RunnableByViewer comment now matches enforcement: it only blocks Viewer-driven agent-mode runs, not plain RAG/Wiki QA. Tests - New cases for the 503/404 paths, lookup-skip behaviour, cross-tenant superuser bypass, the lookup helpers (not-found, cross-tenant, built-in agent, missing tenant context), and IsSyntheticUserID. |
||
|
wizardchen
|
9e3c92ca95 |
feat(rbac): enforce tenant role matrix on routes (PR 2 of 3, #1303)
Build on PR 1's tenant_members schema by enforcing the role matrix on
every authenticated route and at agent run time. Enforcement is gated
behind cfg.Tenant.EnableRBAC so the change ships dormant — the same
code paths log the would-be rejection during the rollout window and
start blocking once operators flip the flag.
Middleware (internal/middleware/rbac.go):
- RequireRole(min, cfg): role-only gate using TenantRole.HasPermission.
- RequireOwnershipOrRole(min, lookup, cfg): allows the original creator
through even when their role is below min, otherwise falls back to
the role check. Lookup errors fail closed; an empty creator_id is
treated as tenant-owned and requires the role.
- Both honour cfg.Tenant.EnableRBAC for the rollout window.
Creator lookups (internal/handler/rbac_lookups.go):
- KBCreatorLookup resolves :id -> KnowledgeBase.CreatorID.
- AgentCreatorLookup resolves :id -> CustomAgent.CreatedBy; built-in
agents (IsBuiltin=true) return ("", nil) so they remain Admin+ only.
- Compile-time guards keep the methods aligned with CreatorLookup.
Route guards (internal/router/rbac.go + router.go):
- rbacGuards bundles the matrix into Viewer/Contributor/Admin/Owner
helpers plus OwnedKBOrAdmin / OwnedAgentOrAdmin. Each Register*
function takes *rbacGuards so route lines remain readable.
- KB / Agent / Knowledge / Chunk / Tenant / Model / MCPService /
VectorStore / DataSource / IMChannel routes annotated per the PR 1
design doc matrix. Tenant-wide infra (models, vector stores, MCP,
data sources, IM channels) reads as Viewer+ and writes as Admin+;
per-resource mutations on KB / Agent route through the
ownership-or-Admin guard.
Resource ownership (services):
- CreateKnowledgeBase / CopyKnowledgeBase populate CreatorID from the
authenticated user, so the OwnedKBOrAdmin guard has a creator to
match.
- CreateAgent / CopyAgent populate CreatedBy similarly. Built-in
agents stay tenant-owned.
Agent run-time gate (internal/handler/session/qa.go):
- AgentQA refuses to launch the agent engine for Viewers when the
target agent has RunnableByViewer=false. Same EnableRBAC gating as
the middleware so it's a no-op until enforcement is turned on.
Tests:
- 12 unit tests cover RequireRole / RequireOwnershipOrRole, including
the fail-open and fail-closed branches and the lookup-error path.
|
||
|
wizardchen
|
af23483433 |
fix(auth): honour JWT tenant_id claim in middleware
POST /auth/switch-tenant mints an access token whose `tenant_id` claim
points to the target tenant, but the middleware previously discarded
that claim entirely:
user, err := userService.ValidateToken(ctx, token) // returned only *User
targetTenantID := user.TenantID // always the home tenant
The session therefore stayed scoped to the user's home tenant despite
the supposedly fresh token, making the new endpoint a no-op once
the request hit any tenant-scoped query.
This change widens the contract:
- UserService.ValidateToken now returns (user, activeTenantID, error)
where activeTenantID is the JWT's `tenant_id` claim, falling back to
user.TenantID when the claim is absent (backward compatibility with
pre-RBAC tokens still in the wild).
- middleware/auth.go uses activeTenantID as the default target before
applying any X-Tenant-ID header override, and recomputes
crossTenantSwitch from `activeTenantID != user.TenantID` so the
resolveTenantRole branches stay correct for switch-tenant sessions.
- The non-middleware /auth/validate handler in handler/auth.go absorbs
the new return value (it only cares about user identity).
Tests:
- internal/application/service/user_jwt_test.go pins the JWT->tenant-id
projection (`tenantIDFromClaims`) — claim present beats fallback,
zero/negative/string claims fall back, int64/uint64 are accepted for
test-built tokens.
Required for PR 2 multi-tenant integration tests, where a single user
needs to switch between Owner-in-A and Viewer-in-B sessions and observe
distinct authority on each side.
|
||
|
wizardchen
|
f3d1210577 |
refactor(rbac): address PR 1 review feedback
Tighten correctness, security and observability of the tenant RBAC foundation introduced in 3c4cdd70 in response to internal review. Bugfixes - custom_agents.runnable_by_viewer: drop the GORM `default:true` tag. GORM treats a plain bool's zero value as "unspecified" and silently rewrites it to the column default on insert, which would prevent admins from ever creating an agent with viewer-runtime disabled. The column-level DEFAULT TRUE on the database side still backstops rows inserted by raw SQL / migrations. - TenantMemberRepository.HasAnyMembers: replace `Limit(1).Count(&n)` with `Take(&probe)`. GORM's Count ignores Limit and scans the whole table, which matters because this runs on the auth middleware's hot path for users without a cached membership. - service/user.LoginWithOIDC: include User, Tenant and Memberships in the OIDCCallbackResponse so the OIDC flow returns the same shape as password login (previously User and friends were left nil). Security - middleware/auth.resolveTenantRole: restrict orphan-tenant auto-promotion to the caller's home tenant. Cross-tenant superuser visits and JWTs minted for foreign tenants no longer trigger silent Owner grants. Two regression tests pin the new behaviour. - service/user.synthFallbackMembership: default the synthesised role to Viewer instead of Owner. The login response only feeds UI; the backend re-derives the real role from tenant_members on every request, so showing a Viewer UI is preferable to a misleading Owner UI that would surface admin controls the backend will then 403. - migrations/versioned/000043_tenant_rbac.down.sql: gate the destructive drop behind a `weknora.allow_destructive_migration` GUC so an accidental `migrate down` cannot wipe role data in production. - frontend/src/stores/auth.ts: comment currentTenantRole as rendering-only so future PRs cannot mistake the localStorage-backed value for an authorisation signal. API contract - Memberships in LoginResponse and OIDCCallbackResponse now serialises without omitempty, and buildMembershipsForUser / synthFallbackMembership always return a non-nil slice — frontend can rely on JSON `[]`. - handler/auth.GetCurrentUser includes memberships so /auth/me refreshes recover currentTenantRole after a page reload. Performance - Add TenantService.GetTenantsByIDs (with the matching repo method) and use it inside buildMembershipsForUser to batch tenant-name lookups instead of issuing one query per membership. Observability - Replace the remaining `log.Printf` calls in middleware/auth with contextual logger.Warnf / logger.Infof so role resolution events ride the standard request-id / Langfuse trace plumbing. The successful auto-promote line is tagged `[audit]` for log mining. Tests - internal/application/service/tenant_member_test.go: hand-rolled fakeTenantMemberRepo plus 11 cases covering AddMember validation, EnsureOwner idempotence, "cannot demote/remove the last Owner", same-role no-op, ErrInvalidTenantRole / ErrMembershipNotFound, and the TenantRole.HasPermission lattice. - internal/middleware/auth_role_test.go: fakeMemberService plus 8 cases pinning every branch of resolveTenantRole, including the two H1 regressions (cross-tenant superuser must not auto-promote; orphan auto-promote requires the home tenant). |
||
|
wizardchen
|
7f7a48880f |
feat(rbac): add tenant_members schema and auth wiring (PR 1 of 3, #1303)
Lay the foundation for tenant-level RBAC without enforcing it yet. Schema, types, middleware lookup, and login-flow shape changes ship now; RequireRole/route annotations and member-management endpoints follow in PR 2 and PR 3. Highlights: - New tenant_members table (Postgres + SQLite) with (user_id, tenant_id) unique key and (owner|admin|contributor|viewer) role values. Migration 000043 backfills one row per existing user, marking the earliest-created active user in each tenant as Owner and the rest as Contributor. - KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so PR 2 can wire ownership-based and Viewer-runtime checks. - Auth middleware loads the active TenantMember row, sets a new TenantRoleContextKey, and falls back via three escape hatches: cross-tenant superusers (CanAccessAllTenants) get Admin in the target tenant; orphan tenants (zero members) auto-promote the first authenticating human to Owner; and when tenant.enable_rbac is false (the v1 default) unmatched lookups fail open with Admin to preserve current behaviour. - Register / OIDC provisioning now insert an Owner membership for the registrant via TenantMemberService.EnsureOwner. - LoginResponse renamed tenant -> active_tenant and gained a memberships array. POST /auth/switch-tenant issues a fresh token pair scoped to a target membership; GET /auth/config exposes auth.registration_mode for the frontend register-tab gating. - New config keys: auth.registration_mode (self_serve | invite_only, defaults to self_serve) and tenant.enable_rbac (defaults to false). Both have WEKNORA_* env overrides. - Frontend store persists memberships and exposes currentTenantRole; Login.vue calls /auth/config and hides the Register tab in invite_only mode. Go SDK types updated to match the new LoginResponse shape. |
||
|
wolfkill
|
3214e3d952 | fix(agent): exclude processing docs from prompt | ||
|
ochan.kwon
|
0e8de6192c |
feat(knowledge-base): validate vector store bindings on create, copy, and delete
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory into the user-facing knowledge-base lifecycle: - POST /knowledge-bases validates the requested vector_store_id against the caller's tenant scope and the engine registry. New error codes ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201) distinguish the typed branches without echoing UUIDs to the client. - GET / POST / PUT / PUT-pin responses embed the bound store's display metadata (name, source, engine_type, status) without exposing any connection credentials. Cross-tenant shared KBs receive a suppressed payload (vector_store_id stripped, source="shared") so operator-chosen store names cannot be enumerated across tenants. - POST /knowledge-bases/copy synchronously rejects clones whose target has a different embedding model or vector store, before the async clone task is enqueued. The async clone worker re-applies the same checks for defense in depth. - DELETE /vector-stores/:id refuses to remove a store with bound KBs, inside a transaction that row-locks the store on PostgreSQL and serializes via WAL on SQLite. unregister-from-registry is wrapped in defer/recover so a panic surfaces as a structured warning instead of silently leaking a stale engine. - vector_store_id is immutable after creation. The GORM <-:create tag blocks every ORM update path; the service-layer DTO omits the field entirely; a reflection-based regression test catches any future maintainer who adds it back to either layer. - Empty-string vector_store_id is normalized to nil at both the create path and inside SharesStoreWith, so rows persisted by callers that did not run Normalize first cannot trip false same-store comparisons. Part of #993. Depends on #994 and #1310. |
||
|
taoyang
|
b3004f73d4 |
test(repo): add wiki_config + indexing_strategy to sqlite test DDL
The in-memory SQLite DDL in `knowledgebase_sqlite_test.go` mirrors the
`knowledge_bases` columns the tests need to insert against. It missed two
columns that the `KnowledgeBase` struct has been carrying for a while:
- `wiki_config` (gorm:"column:wiki_config;type:json")
- `indexing_strategy` (gorm:"column:indexing_strategy;type:json")
Without those columns the `INSERT INTO knowledge_bases (...)` statement
GORM emits fails with:
table knowledge_bases has no column named wiki_config
…which cascades to every TestKnowledgeBase_VectorStoreID_* test
failing on a fresh `go test ./internal/application/repository/`.
Reproduce on main (before this commit):
go test ./internal/application/repository/ -count=1
...
--- FAIL: TestKnowledgeBase_VectorStoreID_Save_Immutable (0.00s)
knowledgebase_sqlite_test.go:91:
Received unexpected error:
table knowledge_bases has no column named wiki_config
FAIL github.com/Tencent/WeKnora/internal/application/repository
After this commit:
ok github.com/Tencent/WeKnora/internal/application/repository
The fix is two `TEXT NULL` column additions to the inline DDL. No
production code, no migration, no schema change — strictly aligning the
test scaffold with the model fields that already exist.
|
||
|
wizardchen
|
7643dd457e |
refactor(credentials): introduce /credentials subresource pattern
Replace the legacy "redacted placeholder + Clear* boolean" pattern with
dedicated per-resource credential subresources across MCP services,
Models, Web search providers, and Data sources.
Why
---
The previous design had three problems:
1. Main PUT body carried secret fields. The frontend echoed back a
redacted "***" placeholder, and a fragile MergeUpdate / IsRedactedOrEmpty
defense in the service layer tried to detect "user did not change this"
vs "user wants to clear this". A regression in that defense (or a new
frontend forgetting it) silently overwrites the stored secret with the
placeholder.
2. The "remove this credential" UX was a red checkbox under a pre-filled
password input. Three intents (preserve / replace / clear) collapsed
onto one field, and credential changes were bundled with unrelated
config edits in the same submit. Users wiped working keys by mistake.
3. Secret presence was inferred from "did the response come back with a
'***' placeholder", which couples the contract to a magic string.
Design
------
Each of the four resources now exposes:
PUT /{resource}/{id}/credentials # write one or more fields
DELETE /{resource}/{id}/credentials/{field} # clear a single field
"Is this configured?" metadata travels on the main resource response as
a typed map (dto.MCPServiceResponse.Credentials etc.) — no separate GET
endpoint. The frontend reads the configured boolean from the main GET
and never sees secret values at all.
Main PUT endpoints now ignore any secret fields in the request body and
log a deprecation warning if they appear, so legacy clients fail loudly
rather than overwriting silently.
Frontend
--------
- New reusable <CredentialResource> component renders a three-state card
(unconfigured / configured / editing) and drives the dedicated
endpoints. Used by MCP, Model, Web search; DataSource has a bespoke
card with the same behaviour because its credentials are a single
atomic per-connector map.
- Cancel from edit mode now restores state synchronously from the meta
prop. The previous async refresh() was a no-op while state was still
'editing', leaving the input frozen open.
- Remove is single-click + toast. The danger-themed button is the
deterrent; a modal confirm adds friction without adding safety (the
plaintext is irrecoverable client-side either way — recovery means
re-typing).
DTOs (internal/handler/dto/) are deliberately separate from the GORM
models so "no secret in response" is a compile-time invariant: a future
contributor cannot leak a secret without explicitly adding the field to
the DTO, which is review-able in a single diff.
Storage is unchanged — credentials still live in the existing jsonb /
parameters columns. No schema migration.
Cleanup
-------
- types.MCPService / types.Model / types.WebSearchProviderEntity /
types.DataSourceConfig: drop ClearAPIKey / ClearToken / ClearAppSecret
/ ClearCredentials boolean fields, MergeUpdate(), RedactSensitiveData().
- utils/types/secret.go: drop PreserveIfRedacted / IsRedactedOrEmpty.
RedactedSecretPlaceholder constant is retained because VectorStore
still uses the old pattern and is out of scope here.
- Frontend hasExistingApiKey / clearApiKey / convertToLegacyFormat
redaction handling removed; i18n keys renamed secret -> credential.
|
||
|
ochan.kwon
|
d869cc1ff6 |
fix: redact sensitive fields and prevent credential loss on edit
Closes #988 Unify the "write-only secrets" pattern across MCP services, Models, WebSearch providers, and DataSources. Fix two related defects: 1. GET responses leaked plaintext credentials — GetMCPServiceByID while List masked, all Model / WebSearchProvider / DataSource GET responses unconditionally. 2. When the frontend echoed a masked placeholder back on save, the update path overwrote the stored secret with the mask — silently wiping the real credential. MCP surfaced this most visibly: ListMCPServices masked Token to "first4****last4" and McpServiceDialog round-tripped that masked string back into the DB. - internal/types/secret.go introduces the shared sentinel RedactedSecretPlaceholder = "***" and two helpers: - IsRedactedOrEmpty(s): true for "" or the sentinel - PreserveIfRedacted(incoming, existing): merge predicate - Each resource adds a RedactSensitiveData (mutates-in-place) or Redacted (returns-a-copy) method and a MergeUpdate contract: empty or placeholder preserves, any other value replaces, and a write-only Clear* bool flag (gorm:"-", json omitempty) wipes with precedence over preserve / replace. Clear* never persists. - Every GET / List / Create / Update response path now redacts. VectorStore's pre-existing "***" literal is refactored onto the shared constant for consistency. All four dialogs — McpServiceDialog, ModelEditorDialog, WebSearchSettings, DataSourceEditorDialog — never pre-fill secret inputs. The stored-state signal is collapsed into the input placeholder: "•••••••• (Enter new value to replace)" when the server reports a stored value, falling back to the connector-specific placeholder otherwise. The input itself carries the "something is there" cue, matching how password dots render. An opt-in "Remove this credential" checkbox appears in edit mode when a value is stored; checking it disables the input and triggers a danger-themed confirmation dialog before submit. DataSource rolls the clear up to a connector-level "Remove all credentials" flag since its credentials are per-connector atomic bundles (e.g. Feishu app_id + app_secret). i18n updates across en-US, zh-CN, ko-KR, ru-RU: secret.storedPlaceholder / clearHint / confirmClearTitle / confirmClearBody. - MCP update handler's manual map-to-struct projection was silently dropping clear_api_key / clear_token / custom_headers on partial updates. Pull them off the map explicitly. - MCP configChanged detection now snapshots URL / StdioConfig / TransportType before the merge so connection recycling still triggers on real transport changes (not just whenever AuthConfig is present). - MCP CustomHeaders: nil preserves existing, non-nil replaces. Was always overwriting, silently wiping headers on any auth-bearing update. - GetMCPServiceByID now redacts (was plaintext while List masked). - WebSearchProvider update preserves Name / Description when empty instead of clobbering them. - WebSearchProvider TestProviderRaw rejects the "***" placeholder up front with an actionable message rather than forwarding it to the upstream provider API as a literal key. - Model builtin hideSensitiveInfo now preserves non-secret metadata (SupportsVision, Provider, ExtraConfig, InterfaceType, ParameterSize, EmbeddingParameters) that were previously zeroed out for builtin model cards. - Replace local equalStringSlices / equalStringMaps helpers with stdlib slices.Equal / maps.Equal (Go 1.21+). - internal/types/{secret,mcp,model,web_search_provider,datasource}_test.go: table-driven coverage of merge, redact, clear, and write-only flag invariants for every resource. - internal/application/service/mcp_service_test.go: full service layer with a fake repo; covers ListMCPServices / GetMCPServiceByID redaction, UpdateMCPService partial-update honoring of Clear*, CustomHeaders preserve-on-nil / replace-on-non-nil, and clear-flag precedence over submitted values. Additive only. No changes to request or response JSON shape beyond the write-only Clear* flags (omitted by default via omitempty). No DB schema changes. No SDK struct field removals. Existing clients that don't know about the new flags continue to function unchanged. - DataSource.Config jsonb at-rest encryption. Pre-existing residual risk; tracked as a follow-up. - Renaming VectorStore.MaskSensitiveFields to match the new RedactSensitiveData / Redacted naming convention. Left alone to minimize diff; a follow-up rename PR can unify. |
||
|
hjz
|
06feee0fee | feat: modify the skills registration directory | ||
|
wolfkill
|
8f4e5a459f | fix(session): scope session access by user | ||
|
ochan.kwon
|
7478e3cddb |
refactor(retriever): introduce factory for KB-scoped retrieve engine resolution
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across seven services into two factory functions with tenant-ownership verification: CreateRetrieveEngineForKB (synchronous) and CreateRetrieveEngineFromPayload (async task handlers). Promotes GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry. Extends KBDeletePayload / IndexDeletePayload with an omitempty VectorStoreID snapshot. Design highlights - Factory verifies tenant ownership of the resolved store (defense-in-depth) so a gap in PR3 validation or a tampered Asynq payload cannot cross tenants. Cross-tenant attempts return ErrVectorStoreForbidden with a structured log entry. - Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound, ErrVectorStoreForbidden) let async handlers classify non-retryable failures as asynq.SkipRetry. - nil and empty-string pointers for vectorStoreID normalize to "unbound" so callers never send an empty UUID to GetByStoreID. - resolveBoundEngine constructs engineInfos directly (with slices.Clone of Support()) instead of going through NewCompositeRetrieveEngine. The latter only reads from the byEngineType map and cannot reach DB stores in byStoreID -- multiple stores can share the same engine type. Unbound fallbacks still reuse NewCompositeRetrieveEngine so the tenant effective-engines path is byte-for-byte unchanged. - DeleteKnowledgeBase now reads the KB before soft-delete so the enqueued KBDeletePayload carries a VectorStoreID snapshot; once the row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT per KB-delete request (a rare admin operation). - enqueueIndexDeleteTask gains a vectorStoreID *string parameter that the single caller (tag.go) populates from the owning KB. The worker (ProcessIndexDelete) validates ownership via the factory. - cleanupKnowledgeResources loads the KB so bound-store cleanup routes correctly. If the load fails it warns and falls back to tenant effective engines; orphan vectors become observable in logs instead of disappearing silently. - DeleteKnowledges (batch) stays on the tenant effective-engines path -- batches can span stores and multi-store fan-out is PR4 scope. Noted inline. Merge safety All knowledge bases have VectorStoreID = NULL at merge time (introduced nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()), which is the existing code path for these KBs. Async payloads from before this change decode to VectorStoreID = nil (omitempty) and take the same fallback. Testing - go build ./... - go vet ./... - go test -race -count=1 ./internal/application/service/retriever/... covers unbound (nil/empty), bound, cross-tenant, unregistered store, ownership infra error, legacy payload shapes (missing field/null/ empty string/nil), tampered payload, and parallel invocation. - grep -r NewCompositeRetrieveEngine internal/application/service returns zero results outside factory.go/composite.go. Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994. |
||
|
Andy Yang
|
6e6d61d945 |
fix(org): searchable join bypasses invite code expiry
JoinByOrganizationID no longer delegates to JoinByInviteCode for direct join, matching the product copy that searchable spaces need no invite code. Extract joinAsViewerWithChecks for shared member-limit logic. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
wizardchen
|
b00bc84f35 |
fix(agent): pass reasoning_content back to providers that require it
MiMo and DeepSeek V3.2/V4 reject multi-turn requests in thinking mode when the prior assistant message lacks reasoning_content with HTTP 400: "The reasoning_content in the thinking mode must be passed back to the API." Agent's ReAct loop is the worst-case scenario — every round produces tool calls, exactly the case DeepSeek's docs specify reasoning_content MUST participate in subsequent context. Plumb reasoning_content through the full assistant-message round-trip: - chat.Message / types.ChatResponse / types.AgentStep gain a reasoning_content field (AgentStep persists via the existing Message.AgentSteps jsonb column, no migration needed). - streamLLMToEventBus accumulates reasoning chunks into result.ReasoningContent and surfaces it on the round's ChatResponse. - engine.runReActIteration writes it onto AgentStep so cross-turn replay preserves it. - observe.appendToolResults attaches it to the same-turn assistant message; agent_history.buildAssistantHistoryMessages does the same on cross-turn replay. - RemoteAPIChat.ConvertMessages forwards it on assistant turns to openai.ChatCompletionMessage.ReasoningContent (already supported by go-openai); providers that don't recognize the field ignore it. Tests cover the three boundaries: ConvertMessages serializes it for upstream, appendToolResults preserves it within the same turn, and buildAssistantHistoryMessages replays it across turns. Scope is intentionally limited to Agent mode — KnowledgeQA's chat pipeline and Anthropic's signed thinking_blocks are separate fixes that require schema changes (rendered_content / thinking_blocks columns). Fixes #1302 |
||
|
wizardchen
|
f740e7cecf |
feat(observability): expand Langfuse spans across chat pipeline
Previously the Langfuse timeline for a knowledge-chat request only showed generations (chat.completion, embedding.embed, rerank), so the work happening between them — query setup, vector/keyword search, result merging, prompt assembly — appeared as unexplained gaps. Add spans around the pieces that fill those gaps: * qa.setup wraps request-time KB / model / search-target resolution before the pipeline event loop starts, accounting for the visible delay before the first chat.completion. * pipeline.<event> wraps each pipeline stage's eventManager.Trigger call so generations inside a stage nest under it. CHAT_COMPLETION_ STREAM is intentionally skipped because its OnEvent returns as soon as the streaming goroutine starts; a stage span would always finish before the chat.completion.stream generation it nominally parents. * retrieve wraps the actual vector + keyword retrieve call inside HybridSearch, exposing the DB round-trip that previously sat invisibly between embedding generations and rerank. * web_search wraps the external web-search HTTP call when enabled. |
||
|
wizardchen
|
64b20a2d87 |
feat(agent): support dedicated model for query understanding step
Quick-answer (RAG) agents can now configure a separate chat model for the query-understanding stage (rewrite + intent classification), decoupling it from the main conversation model so users can route the lightweight rewrite call to a cheaper / faster model. - Add CustomAgentConfig.QueryUnderstandModelID and plumb it through PipelineRequest and ChatManage.Clone. - query_understand plugin prefers QueryUnderstandModelID on the text-only path; falls back to ChatModelID if the configured model cannot be resolved (with a warn log). Multimodal path is unchanged to keep vision-capable model selection intact. - AgentEditorModal exposes a ModelSelector under the existing Query Rewrite block; empty means reuse the main chat model. - Add i18n strings (queryUnderstandModel / placeholder / desc) for zh-CN, en-US, ko-KR, ru-RU. - Extend Go SDK AgentConfig with the new field. |
||
|
wizardchen
|
3f9b09e306 |
fix(pipeline): make data-analysis stage opt-in per agent (#1244)
The legacy in-pipeline DuckDB SQL data-analysis stage used to run on every quick-answer RAG request whose retrieved chunks included a CSV/Excel file, adding one extra LLM round-trip (~3s) to generate a SQL query that most plain Q&A users never wanted. There was no way to disable it. Introduce a per-agent DataAnalysisEnabled flag (default off), wire it through PipelineRequest, and gate the DATA_ANALYSIS stage on it. Surface the toggle in the agent editor for quick-answer agents with at least one knowledge base attached. Refs: https://github.com/Tencent/WeKnora/issues/1244 |
||
|
wizardchen
|
cacca049d9 |
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base detail page — parse status, source/channel, and updated time range — and rework multi-select to no longer cause the card title to jitter on hover. Backend - Introduce types.KnowledgeListFilter to aggregate optional filter dimensions (tag, keyword, file_type, parse_status, source, updated_from/to) and switch ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept it instead of a growing positional parameter list. - The ListKnowledge HTTP handler accepts new parse_status, source, start_time and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS" and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type column to stay consistent with file_type semantics; other source values match the channel column. - Update the four other callers (agent_service, initialization) to pass an empty filter struct, preserving prior behavior. Frontend - Add three controls in the doc-filter-bar (status select, source select, date-range picker with future-date disabled) wired through getKnowled / listKnowledgeFiles into the new backend params. - Replace the hover-triggered card checkbox with an explicit "批量管理" mode (mirrors the session list UX): in card view the checkbox only renders while batch mode is on, entered via the per-card "..." menu; the list view keeps its leading checkbox column. Switching from list to grid auto-enables batch mode when something is already selected, so the selection stays visible. - DocumentBatchBar now stays open whenever batch mode or selection > 0, and its "取消选择" button both clears the selection and exits batch mode. API surface sync - Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml). - Update docs/api/knowledge.md with the new query parameters. - Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the Go SDK; the existing ListKnowledge keeps its signature. i18n - New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing menu.batchManage / batchManage.cancel for the multi-select strings. |
||
|
wizardchen
|
8b2a36d759 |
fix(multimodal): unblock processing when provider:// image read fails
When image bytes for a multimodal task cannot be read via FileService (e.g. tenant.StorageEngineConfig.MinIO is empty while the image was saved using the global MINIO_* env vars), the previous code fell back to the HTTP downloader, which rejected the provider:// URL with "unsupported URL scheme". The asynq handler then returned an error, asynq retried until exhaustion, and the per-knowledge "pending images" counter was never decremented — leaving the document stuck in "processing" indefinitely (issue #1282). Changes: - ImageMultimodalService now holds a default FileService and resolveFileServiceForPayload falls back to it when the tenant-scoped storage config cannot produce a usable service, mirroring the write-side fallback in knowledgeService.resolveFileService. - Extract readImageBytes: provider:// URLs are read exclusively via FileService and never handed to the HTTP downloader. - On unrecoverable read failure for a single image, log and skip that image but still call checkAndFinalizeAllImages so the parent knowledge can progress to post-processing. Fixes #1282 |
||
|
wizardchen
|
5949be739f |
fix(agent): relax rerank model requirement for custom agents
Previously the custom agent editor hid the rerank model field when no RAG-type knowledge base existed in the configured scope, but the not-ready check and runtime hard-failed whenever knowledge_search was in allowed_tools. Users with wiki-only or empty scopes saw "Rerank Model required" warnings they could not resolve, and "All knowledge bases" agents broke later if a RAG-type KB was added. - Backend: when knowledge_search is enabled, fall back to the tenant default rerank model (ConversationConfig.RerankModelID) before erroring out, matching the built-in agent behaviour. - Editor: always show the rerank field once a KB scope is selected; only mark it required (red *) when the scope contains a RAG KB, with a hint explaining the tenant-default fallback. - Editor: only render rerank top_k / threshold sliders when a rerank model is actually selected. - Input field: drop the eager "missing rerank" not-ready reason; the backend is now the single source of truth for rerank availability. - i18n: add agent.editor.rerankModelOptionalHint across all locales. |
||
|
nullkey
|
4ed7b90eac |
fix(faq): correct PostgreSQL cast precedence in FAQ search
`metadata->'similar_questions'::text` parses as
`metadata -> ('similar_questions'::text)` because `::` binds tighter
than `->`, so the expression yields jsonb instead of text. ILIKE on
jsonb then fails with "operator does not exist: jsonb ~~* unknown",
returning 1007 Internal server error for `search_field=similar_questions`
and `search_field=answers`. The `standard_question` branch was unaffected
because it uses `->>` (text) directly.
Wrap the json access in parens so the cast applies to the extracted
value: `(metadata->'similar_questions')::text ILIKE ?`. The MySQL
branch uses JSON_EXTRACT and is unchanged.
Fixes #1264
|
||
|
wizardchen
|
d199628701 |
fix(agent): exclude wiki-only KBs from quick-answer (RAG) mode
Quick-answer agent mode retrieves purely through vector/keyword chunk search and ships with no `allowed_tools`, so the existing capability filter (which only reads from `allowed_tools`) let wiki-only KBs through in every entry point. End result: users could @-mention, select, and receive suggested questions from wiki-only KBs in quick-answer mode, but the underlying retrieval always returned empty. Treat "RAG-only" as an implicit property of `agent_mode = quick-answer` and union it with the tool-derived filter. The same predicate is now used everywhere the user can pick or be steered toward a KB: Backend - `tools.DeriveKBFilterForAgent` / `KBSatisfiesAgentRequirements` layer the implicit quick-answer requirement on top of tool derivation. - `ListKnowledgeBases`, `SearchKnowledge` (shared-agent `@file`), `resolveKnowledgeBasesFromAgent` (chat runtime), `/search` IM command, and `GetSuggestedQuestions` now all use the agent-mode-aware variant. - `GetSuggestedQuestions` also skips the wiki-page fallback for quick-answer agents to cover the `selected` / explicit-kb-ids paths where a wiki-only KB could still slip through. Frontend - `deriveKbFilterForAgent` / `kbSatisfiesAgentRequirements` mirror the Go helpers. - `@` mention dropdown (`Input-field.vue`) uses the new helper. - Agent editor's "specified KB" picker (`AgentEditorModal.vue`) grays out wiki-only KBs for quick-answer agents with a tooltip, and the pre-save warning fires for quick-answer mode too. - i18n: add `agentEditor.agentType.kbMismatch.quickAnswer` across all four locales. |
||
|
wizardchen
|
a082c04d28 |
chore(deps): update dependencies in /docreader and adjust dependabot configuration
- Updated `pydantic` from 2.12.3 to 2.13.4 and `pypdfium2` from 5.0.0 to 5.8.0 in the `docreader` requirements. - Modified the dependabot configuration to set `open-pull-requests-limit` to 0 and added an `ignore` rule for version updates across all ecosystems, allowing only security updates. - Adjusted settings for `server-security`, `client-security`, `frontend-security`, and `miniprogram-security` groups to streamline security update handling. This change aims to enhance dependency management and maintain security while reducing noise from version update PRs. |
||
|
wizardchen
|
9ad8e7ca78 |
fix(agent): replay attachments in multi-turn history
Agent mode does not persist `rendered_content` for user messages, so when the next turn's history was rebuilt from DB, attachments uploaded in prior turns disappeared — the model only saw the raw query plus the prior assistant reply, breaking follow-up questions that referenced the file (e.g. "what is in there?"). Reconstruct the attachment prompt from the stored `Attachments` column when `RenderedContent` is empty, mirroring how image captions are already replayed. KnowledgeQA turns (which do persist `RenderedContent`) are unaffected and won't get attachments injected twice. Refs #1237 |
||
|
wizardchen
|
1f5970b67e |
refactor(agent): rebuild multi-turn history from DB, drop llmcontext layer
The Agent ran a parallel context cache (Redis/in-memory) on top of the messages table to feed multi-turn history into the LLM. That dual-write caused subtle drift (e.g. compression diverging from DB, system-prompt swaps lost on restart) and required a separate ClearContext path on session/IM clear. Make the messages table the single source of truth: - Add service.LoadAgentHistory: rebuilds chat.Message history per turn from the persisted messages, expanding AgentSteps into proper OpenAI assistant_with_tool_calls + tool messages and replaying the canonical final answer (with <think> blocks stripped). final_answer tool calls are filtered to avoid duplicating the trailing answer. - Make AgentEngine stateless across turns: drop ContextManager / sessionID cache plumbing from the engine, agent service, and CreateAgentEngine signature. The engine only uses sessionID for logging/event emission. - Wire AgentQA to load history from DB on demand using HistoryTurns (default 5) when MultiTurnEnabled, otherwise run with empty history. - Delete the llmcontext package (ContextManager interface, Redis/memory storage, factory) and the SessionService.ClearContext API path; IM /clear and session message clear no longer need to invalidate cache. Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with empty AgentSteps just produce the single canonical assistant message), and Agent-mode turns now consistently see prior tool calls and results. |
||
|
wizardchen
|
d2a1006beb |
fix(web-search): address review fixups for SearXNG provider
- utils: 合并 SSRF_WHITELIST 与 SSRF_WHITELIST_EXTRA，避免部署侧默认值 （如 docker-compose 注入的 searxng 主机名）被用户的 SSRF_WHITELIST 自定义值覆盖。 - docker-compose.yml: 把 searxng 默认值挪到 SSRF_WHITELIST_EXTRA。 - searxng: 抽出导出函数 ValidateSearxngBaseURL，让 service 层保存校验 和 provider 构造校验完全一致；service 改为调用同一函数。 - searxng: language 由非法的 "auto" 改为 "all"；移除强制 safesearch=1 让实例 settings.yml 决定。 - searxng: publishedDate 增加多格式 fallback（RFC3339Nano/无时区/ 日期-only/RFC1123 等），无法解析时 debug 日志记录。 - searxng: 解析 unresponsive_engines；结果为空时打 warn 日志，便于排查 "配置正确却搜不到结果"的情况。 - frontend: WebSearchSettings 的 Instance URL 标签/占位符走 i18n， zh-CN / en-US / ru-RU / ko-KR 四个 locale 补齐 baseUrlLabel / baseUrlPlaceholder。 |
||
|
wizardchen
|
1110615300 |
feat(web-search): add SearXNG provider (#1166)
支持对接自建/公共 SearXNG 实例作为网络搜索引擎，缓解免费搜索引擎在国内 网络环境下访问受限的问题。 - types: 新增 WebSearchProviderTypeSearxng 与 BaseURL 参数字段； 类型元数据新增 RequiresBaseURL，前端可动态渲染 Instance URL 表单。 - infrastructure/web_search/searxng.go: 调用 /search?format=json，强制 utils.ValidateURLForSSRF 校验 base_url，可选 api_key 透传给反代鉴权。 - service: isValidProviderType 与参数校验接入 searxng。 - container: 注册 NewSearxngProvider 工厂。 - frontend: WebSearchSettings 表单根据 requires_base_url 渲染 Instance URL 输入框；编辑回填、free 判定同步更新。 - docker: 新增可选 searxng 服务（profile=searxng/full），附带最小化 settings.yml（启用 JSON 格式、关闭 limiter、关闭遥测）， docker-compose 默认 SSRF_WHITELIST 包含 searxng 容器名。 - .env.example: 补充 SEARXNG_PORT / SEARXNG_SECRET 说明。 Closes #1166 |
||
|
wizardchen
|
6d0f09d75f |
fix(agent): 加固 MCP 工具人审批的跨实例与并发语义
- 跨实例 Resolve 通过 Redis reply channel 接收 owning 实例的 ack （ok / not_found / tenant_mismatch / user_mismatch / already_resolved）， HTTP 不再静默 200，超时退化为 NotFound。 - PendingRequest 增加 UserID，Resolve 校验会话所有者，避免同租户 其他用户越权批准/篡改参数。 - 新增 ErrAlreadyResolved，timer/ctx 命中后再次 Resolve 返回 400 而 不是静默成功；waiter.deliver 暴露胜出语义。 - NeedsApproval 默认 fail-close（DB 故障要求审批），可经 WEKNORA_AGENT_TOOL_APPROVAL_FAIL_OPEN=true 回退。 - Pub/Sub 增加 OriginID 自过滤、退避重连、namespace 后缀，避免自回 环噪声与多部署串扰。 - 工具执行超时常量统一通过 ToolExecContext.ExecTimeout 注入， mcp_tool.go 不再硬编码 60s。 - handler 修正 path 二次解码、错误用 errors.Is、modified_args=null 的 静默清空、ListMCPToolApprovals 的 404/500 误判。 - repository 用 clause.OnConflict 做原子 upsert，消除并发 500。 - 增加 NotFound / TenantMismatch / UserMismatch / AlreadyResolved / Race 等用例。 |
||
|
wizardchen
|
5510ea8f5a |
feat(agent): human-in-the-loop approval for MCP tool calls (#1173)
Add an opt-in human approval gate so Agent runs pause before executing MCP tools that operators flag as dangerous, surface an approval card in the chat UI, and only resume after the user approves (optionally with edited args) or rejects. Backend - New mcp_tool_approvals table + repo/service to mark per-tool approval required (PG migration 000042 + sqlite init). - approval.Gate coordinates RequestAndWait / Resolve with sync.Once delivery, configurable timeout, and Redis Pub/Sub fan-out so multi- replica deployments work without sticky sessions. - MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx (without the per-tool 60s timeout) for the wait, and re-derives a fresh 60s exec ctx after approval so CallTool keeps a full window. - New SSE response types (tool_approval_required / _resolved) and EventBus events plumb approval state to AgentStreamDisplay. - REST: list/set per-tool approval flag, resolve pending approval. - Configurable via agent.tool_approval_timeout_seconds (yaml) or WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go duration). Frontend - MCP settings: per-tool "require approval" switch on the test panel. - Chat: ToolApprovalCard renders the pause point with editable JSON args, validation feedback, mm:ss countdown that turns warning/danger near deadline, and a resolved state that retains context. - i18n strings added for zh-CN / en-US / ko-KR / ru-RU. Docs - docs/zh/mcp-approval.md covering behavior, config, API, deployment considerations (Redis cross-instance, restart limitations). |
||
|
langcaiye
|
02be05c095 | feat: support Tencent VectorDB keyword retrieval | ||
|
wizardchen
|
82947c726e |
fix(wiki-ingest,asynqdl): review fixups for PR #1241
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
items" had two args), and harden validMerge against un-prefixed
slugs whose strings.Index returned -1 and silently passed the type
check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
letter row instead of a hard-coded 0; tighten payloadProbe to the
set of field names with consistent semantics across payloads
(drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
outside an asynq worker ctx so the semantics stay pinned.
|
||
|
wizardchen
|
04e8105728 |
perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
|
||
|
langcaiye
|
74b1342440 | feat: add Tencent VectorDB retriever backend | ||
|
wizardchen
|
b7388f10e1 |
fix(wiki-ingest): reconcile summary links and log feed when reduce LLM fails
When a doc's entity/concept page generation hit a transient LLM error in
the reduce phase, the page never got written, but the doc's summary page
(generated in parallel with the slug list baked into its content) was
already persisted with [[entity/foo|name]] links pointing at the missing
page. The wiki log feed also surfaced the failed slugs as clickable
entries that 404'd. cleanDeadLinks only ran for retract batches, so the
debris persisted indefinitely.
reduceSlugUpdates now reports addition-path failures back to the batch
driver, which collects them and:
- rewrites this batch's summary pages, replacing dead [[slug|display]]
refs with plain display text (or a humanized slug tail);
- filters dead slugs out of docResults[].Pages before the wiki log
entries are flushed;
- broadens the cleanDeadLinks trigger to fire whenever any page was
touched or any slug failed, as a long-tail safety net.
Pure text replacement; no extra LLM calls. Failed slugs are picked up
naturally on the next ingest of the same document via the existing slug-
continuity rules, so the system self-heals without manual reingest.
|
||
|
wizardchen
|
fc6f160eff |
fix(retriever/doris): code review cleanup
针对 4cce6f2e（接入 Apache Doris）的 code review 修复，主要修正若干阻断性 问题与可读性问题，并剔除不应进入主仓的本地工作流文件。 阻断性修复： - docker-compose: Doris 镜像由 2.1.0 升至 4.1.0。原 2.1.0 不支持 HNSW ANN、cosine_distance_approximate 与 Stream Load partial_columns， 按当前 DDL 一启动就会失败。 - DSN 字面量拼接改用 mysql.Config.FormatDSN()。原 fmt.Sprintf 在用户名/ 密码包含 `@`/`:`/`/` 等字符时会跑偏。覆盖 health check 与 engine factory 两处。 健壮性修复： - 新增 validateEmbedding，写入与查询前拒绝 NaN/±Inf；strconv.FormatFloat 对非有限值会输出 "NaN"/"+Inf" 拼成无效 SQL。 - waitANNReady 改为后台 goroutine + 独立 context，避免新维度首次写入路径 阻塞最长 30s；ANN 未就绪时 Doris 会自动退化为 brute-force。 清理： - annIndexReady 移除最终两个分支都 return true 的冗余写法。 - Save 移除冗余的双重 toDorisVectorEmbedding。 - testDorisConnection 把 "5.7.99 Doris-4.1.0" 解析为裸 "4.1.0"，与 Postgres/ES 的版本格式对齐。 剔除（不应合入主仓）： - docs/wiki/集成扩展/Doris改动与上游同步.md：纯 fork 维护工作流文档。 - scripts/e2e-doris.sh：作者本地 E2E 验证清单。 测试： - repository_test 用 require.Eventually 适配 ANN 异步轮询。 - 现有 doris 单测全部通过。 |
||
|
issunion
|
4cce6f2e99 |
feat(retriever): 接入 Apache Doris 4.1 作为向量数据库
为 RetrieveEngine 体系新增 Doris 后端，与现有 Qdrant/Milvus/Weaviate
等保持完整能力对齐：向量检索、关键词检索、健康检查、环境变量与多实例
DB 配置、前端类型注册、单元测试、Docker Compose 模板。
实现要点：
- 协议分工：主链路用 MySQL 协议（database/sql + go-sql-driver/mysql）
做 DDL / 查询 / 删除；批量更新走 Stream Load HTTP API，并启用
partial_update=true、merge_type=APPEND，自动按 1MiB 切分批次并处理
307 重定向。
- 表结构：UNIQUE KEY(id) + enable_unique_key_merge_on_write=true 以
支持 upsert/部分列更新；按维度分表（<base>_<dim>），每张表上建
HNSW ANN 索引（metric_type=cosine_distance）和 INVERTED 索引
（parser=chinese）。
- 分数语义：使用 cosine_distance_approximate，再以 1 - dist 转换为
"越大越相似"，与现有 KVHybridRetrieveEngine 约定一致。
- 异步索引：ANN 索引为后台构建，ensureTable 通过轮询 SHOW INDEX 等
待索引就绪后再放行写入，避免首次检索召回为空。
- ARRAY<FLOAT> 序列化：go-sql-driver/mysql 不支持数组占位符，
embeddingLiteral 将 []float32 转成 SQL 字面量字符串再拼接。
新增文件：
- internal/application/repository/retriever/doris/{structs,schema,
query,repository,streamload,repository_test}.go
- scripts/e2e-doris.sh：E2E 验证清单
- docs/wiki/集成扩展/Doris改动与上游同步.md：fork-and-rebase 工作流
与改动清单
修改文件（接线 + 文档）：
- internal/types/{retriever,tenant,vectorstore}.go：新增
DorisRetrieverEngineType、env 解析、表单 schema 与索引参数校验
- internal/container/{container,engine_factory}.go：环境变量驱动
与 VectorStore 配置驱动两条路径都支持 Doris
- internal/application/service/vectorstore{,_healthcheck}.go：连接
校验 + Ping/Version 健康检查
- docker-compose.yml：新增 doris-fe / doris-be 服务（profile=doris）
- .env.example：DORIS_* 环境变量与示例
- docs/{使用其他向量数据库,wiki/集成扩展/集成向量数据库}.md：
使用说明与索引/分数行为说明
依赖：go.mod/go.sum 新增 github.com/go-sql-driver/mysql（运行时）和
github.com/DATA-DOG/go-sqlmock（测试）。
测试：repository 层 SQL 形状、Stream Load HTTP 行为、whereBuilder
逻辑、embeddingLiteral 往返、健康检查错误路径均有单测覆盖。
|
||
|
wizardchen
|
e66c42343b |
fix(wiki-ingest): harden conflict retry and right-size retry budget
Avoid treating Redis pending-length read failures as empty queues, reset stale fail-count keys on fresh ingest/drop paths, and reduce wiki ingest MaxRetry to a moderate shared constant to limit queue churn. |
||
|
wizardchen
|
d0144f3586 |
perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
toy0116
|
bf12868923 |
fix(wiki): cap requeueFailedOps retry count to prevent queue pile-up
Root cause: requeueFailedOps unconditionally RPush'd every failed op back to
wiki:pending, causing unbounded growth when a document consistently triggers
LLM timeouts. Observed: 553 entries in wiki:pending:08134644 (one KB), mostly
duplicates for ~5 unique documents that each timed out on every batch cycle.
Fix:
- Add wikiFailCountKeyPrefix ("wiki:failcount:") and wikiMaxFailRetries (5)
constants in wiki_ingest.go
- requeueFailedOps now atomically Incr(wiki:failcount:{kbID}:{knowledgeID})
before each RPush; skips re-queue (drops op permanently) if count > 5
- Expire the fail-count key at wikiPendingTTL (24h) on every update
- wiki_ingest_batch.go: Del the fail-count key at ingestSucceeded++ so
transient errors don't permanently burn through a document's retry budget
Evidence of problem severity: 5 persistently-failing docs × ~110 requeue
cycles each = 553 pending entries; had to use Lua atomic dedup to recover.
Behaviour after this fix:
- First 5 failures → normal retry (re-queued, follow-up batch scheduled)
- 6th failure → logged as WARN "dropping op … after 6 failures (limit 5)",
not re-queued; prevents the feedback loop
- If document is fixed upstream and re-ingested via EnqueueWikiIngest, the
op is a fresh entry with no existing fail-count key → full 5-retry budget
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|