3226cb97ff0daa71c668a362bef3410d37796b7d
361 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wizardchen
|
8d4c58c5fa |
feat(creator_name): add creator_name field to agents and knowledge bases for enhanced visibility
Introduced a new optional field, creator_name, in both CustomAgent and KnowledgeBase types to allow the front end to display the creator's name. This enhancement enables better differentiation between resources created by the current user and those created by other members of the same tenant. Updated relevant API handlers to populate this field during list operations, ensuring accurate representation in the UI. Additionally, modified the ResourceOriginBadge component to utilize the creator_name for improved context in resource listings. Refs: #1303 |
||
|
wizardchen
|
884ac44283 |
feat(invitations): implement pending invitations feature with polling and dialog
Added functionality for managing pending invitations, including a polling mechanism to update the invitation count in real-time. Introduced a new dialog component for users to view and respond to their invitations without navigating away from their current context. Updated API endpoints for fetching and managing invitations, and enhanced the user interface with relevant internationalization support. Refs: #1303 |
||
|
wizardchen
|
3d30a9d842 |
feat(organization): implement tenant-based access control for organization visibility
Enhanced the GetOrganization and ListMembers methods to enforce tenant-based access control. Users can now only access organization details if their tenant is a member or if the organization is marked as searchable. This change prevents unauthorized enumeration of organizations and ensures sensitive member information is only accessible to authorized tenants. Refs: #1303 |
||
|
wizardchen
|
8332a0dab6 |
feat(favorites): implement user favorites and recents functionality
Introduced a new API client for managing user favorites, allowing users to list, add, and remove starred resources. Enhanced the sidebar to display favorites and recents, improving user navigation. Updated internationalization files to include relevant labels and tooltips for the new features across multiple languages. Refs: #1303 |
||
|
wizardchen
|
7d030a6f6a |
feat(tenant): implement tenant creation limit and error handling
Added a new configuration option to limit the number of tenants a non-superuser can create via self-service. Introduced a new error type for handling cases where users exceed this limit, returning a 429 status code. Updated the tenant creation and update handlers to enforce this limit and provide appropriate feedback to users. Additionally, refactored the tenant update request structure to ensure only mutable fields are allowed. Refs: #1303 |
||
|
wizardchen
|
29dc99ae79 |
feat(tenant): implement self-service tenant creation functionality
Added a new API endpoint for creating tenants, allowing any logged-in user to create their own workspace. The backend automatically assigns the creator as the Owner of the new tenant. A new CreateTenantDialog component was introduced in the frontend, enabling users to input the workspace name and an optional description. Internationalization files were updated to include relevant messages for this feature across multiple languages. Refs: #1303 |
||
|
wizardchen
|
dc8d033b4d |
feat(organization): transition membership model to tenant-level and enhance invite functionality
Updated the organization membership model to operate at the tenant level, aligning with Plan 3. This change introduces new interfaces and modifies existing ones to accommodate tenant-centric operations, including the `InviteMemberRequest` and `TenantInviteCandidate`. The search functionality for inviting members has been updated to reflect this shift, allowing for tenant-based searches instead of user-based. Additionally, internationalization files have been updated to ensure consistent messaging across languages regarding tenant membership and roles. Refs: #1303 |
||
|
wizardchen
|
cf6c28fe7e |
fix(rbac-ui): identify org owner row by tenant_id, not user_id
Plan 3 (#1303) lifted Org membership from per-user to per-tenant, and 40fdb978 pinned the org's owner tenant in DB as organizations.owner_tenant_id. The OrganizationSettingsModal members list, however, still decided "which row is the owner" by comparing member.user_id against orgInfo.owner_id. After Plan 3 the members list is keyed on tenant_id, so the user-id check is structurally wrong: the representative user of the owner tenant happens to be the owner today, but if that user is moved to another tenant — or a different user of the owner tenant becomes the representative — the modal would either lose the "(owner)" tag, or re-tag a non-owner row, and would offer the role-select / remove-button affordances on what is in fact the owner tenant row. This change: - Adds OwnerTenantID to OrganizationResponse and populates it in toOrgResponse from the persisted column. IsOwner now compares the caller's current tenant against owner_tenant_id (with a user-id fallback for legacy rows where owner_tenant_id == 0). - Adds owner_tenant_id to the frontend Organization type and a small isOwnerMember(member) helper in OrganizationSettingsModal, which tenant-matches by default and falls back to user-id only on pre-000046 rows. The role <select>, role tag suffix, and the delete button on member rows now all use the helper. The "is-me" CSS class and the me-tag in the member name still match on user_id — that one really is a per-user marker ("this row's user is me, the signed-in user"). |
||
|
wizardchen
|
ef01f99e69 |
refactor(rbac): extend KB-access guard to chunk / knowledge / knowledgebase routes
Follow-up to the FAQ + Tag refactor in this same PR. Three additional
handler areas now run through the route-level g.KBAccess* guard:
- chunk.go : full refactor — helper effectiveCtxForKnowledge and
validateAndGetChunk's access check are gone; the
handler's chunk-fetch + ownership-mismatch check stay
as defence-in-depth (a same-tenant attacker can't
pass mismatched knowledge_id + chunk_id).
- knowledge.go and knowledgebase.go : routes now run the guard
BEFORE the handler. The in-handler
validateKnowledgeBaseAccess /
resolveKnowledgeAndValidateKBAccess helpers stay for
now because they branch on the agent_id query param
(specific-agent check vs. any-agent fallback) — that
handler-specific branch can collapse into the guard
in a follow-up. The duplicated DB roundtrip is
negligible compared to the handler's main work.
New middleware exports:
- KBIDFromChunkIDParam : walks chunk_id → KnowledgeBaseID via the
chunk's denormalised column.
- g.KBAccessReadFromChunkIDParam / g.KBAccessWriteFromChunkIDParam
: convenience methods on rbacGuards.
Route changes are purely additive (existing role / ownership guards
stay in place; the new KB-access guard runs after them and rewrites
the request's tenant context to the source tenant for shared KBs).
No new dependencies on the kbShareService / agentShareService at the
handler level for chunk.go — those move to the route layer.
Tests pass; behaviour unchanged.
|
||
|
wizardchen
|
1109cb7bff |
refactor(rbac): KB-access guard at the route layer (no helper in handlers)
Five handler files used to carry near-identical 30-line helpers
(effectiveCtxForKB / validateAndGetKnowledgeBase) that did the same
own-or-shared resolution before every KB-scoped operation:
1. KB belongs to caller's tenant -> grant own access
2. Org-shared KB -> grant min(share, role) cap
3. Shared agent carries the KB -> grant Viewer (read-only)
A bug found in any one of them had to be fixed in five places, and
several already drifted apart (different log levels, slightly
different error messages, one path missing the share-cap check).
Move the resolution to a route-level gin.HandlerFunc:
middleware.RequireKBAccess, exposed on rbacGuards as
g.KBAccessRead("id") / g.KBAccessWrite("id"). The guard runs after
the role/ownership checks, then on success:
- stashes the resolved (KB + effective tenant id + permission)
on c.Keys under middleware.KBAccessContextKey, and
- rewrites c.Request.Context() to carry the effective tenant id
so handlers downstream just read tenant the way they always did
(types.MustTenantIDFromContext) without a per-handler helper.
Refactor FAQ and Tag handlers to drop their helper methods; their
service constructors lose the kbShareService / agentShareService
dependencies (now lifted to the route layer). Routes:
faq.GET("/entries", g.Viewer(), g.KBAccessRead("id"), ...)
faq.POST("/entries", g.OwnedKBOrAdmin(), g.KBAccessWrite("id"), ...)
kbTags.GET("", g.Viewer(), g.KBAccessRead("id"), ...)
kbTags.POST("", g.OwnedKBOrAdmin(), g.KBAccessWrite("id"), ...)
The middleware also exports KBIDFromKnowledgeIDParam so chunk routes
(URL :knowledge_id) can adopt the same pattern in a follow-up PR
without reshaping the helper.
Five new tests cover own-KB, not-found, shared-with-sufficient-perm,
shared-but-below-min, and missing-tenant-context paths. Handler
tests still pass (no behaviour change).
|
||
|
wizardchen
|
928f5abe3f |
feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303)
Before this change, the `organization_members` table keyed on user_id — joining an org as a Viewer in tenant T meant only that one user got visibility on shared KBs/agents, while the rest of T (including its Owner) saw nothing. The mismatch surfaced as confusing UX bugs: "my colleague shared this with us, why can't I see it?" This change replaces `organization_members` with `organization_tenant_members`. Membership and Org-RBAC role now key on (org_id, tenant_id); `representative_user_id` stays as a display-only label of the user who first brought the tenant in. Permission resolution gains a 3-D cap so tenant-RBAC stays honest under cross-tenant sharing: effective = min(share.Permission, tenant_org_role, tenant_role_cap) where tenant_role_cap pins tenant Viewers to OrgRoleViewer regardless of the org-level grant. A Viewer in their own tenant cannot edit a shared KB even if the org granted Editor — preserves the "Viewer = read-only everywhere" promise. Migration 000045 backfills `organization_tenant_members` from the old table by collapsing each (org, tenant) group to its max role, then renames the old table to `organization_members_pre_plan3` for safe rollback. Sqlite mirror (Lite mode) is fresh-bootstrap: the init schema replaces the old table directly. Routes: member-management URLs change from `:user_id` to `:tenant_id`. Org create / join / leave now require Admin+ in the caller's tenant — joining an Org affects everyone in T, so the decision can't sit with a Viewer. Frontend: OrganizationSettingsModal + store update to address members by tenant_id. |
||
|
wizardchen
|
8c0510f2e1 |
fix(api): reject /agent-chat early when agent_id is missing
AgentQA used to trust request.AgentEnabled even when resolveAgent() returned nil (no agent_id, or unresolvable id) — the request reached the async goroutine, AgentQA service errored with 'custom agent configuration is required', and the user saw a broken SSE stream with no actionable error. Most likely root cause is a stale frontend settings store where isAgentEnabled stayed true after selectedAgentId got blanked (typically a cross-tenant switch where the previously selected agent isn't visible in the new tenant). The frontend fix lives in selectAgent() syncing both flags, but the handler shouldn't trust that contract. Add an early 400 with a clear message before executeQA dispatches. Logs the actual request.AgentID (sanitized) so future stuck-state reports are diagnosable from a single log line. |
||
|
wizardchen
|
619e14a2eb |
fix(rbac): wire chunk-id ownership chain for generated-question delete
DELETE /chunks/by-id/:id/questions used to be gated at flat
Contributor+ as a deliberate carve-out: the URL only carries the
chunk's own id, so the chunk -> knowledge -> KB ownership chain
wasn't reachable from request params alone. The carve-out had two
problems:
- Inconsistent with every other chunk mutation. PUT
/chunks/:knowledge_id/:id is OwnedKBOrAdmin (KB creator OR
Admin+); a Contributor with no KB ownership can't update a
chunk but COULD delete a generated question on the same chunk.
- User-visible: a Contributor in tenant A who never created any
KB still saw the 'delete question' button work, while every
other chunk control 403'd.
Wire the missing hop:
KBCreatorLookupFromChunkIDParam (handler) reads :id, fetches the
chunk via the existing GetChunkByIDOnly (unscoped repo call),
re-checks chunk.TenantID against the caller's tenant context to
block cross-tenant chunk-id probing, then defers to the existing
resolveKBCreatorByKnowledgeID helper for the rest of the chain.
OwnedChunkKBOrAdminFromChunkID (router) wraps it in
RequireOwnershipOrRole(Admin, ...) — same matrix as every other
chunk/knowledge/wiki mutation guard.
chunks.DELETE switches from g.Contributor() to the new guard.
4 unit tests cover happy path, chunk-not-found, cross-tenant
chunk-id probing, and KB-not-found upstream — same edge-case shape
as the existing KBCreatorLookup_* tests so future refactors get
caught loudly.
|
||
|
wizardchen
|
1e8b5c8d40 |
refactor(api): drop deprecated tenant ConversationConfig field and KV write path
The Tenant.ConversationConfig field was marked deprecated when
CustomAgent (builtin-quick-answer) replaced the per-tenant defaults
path, but the JSONB column, GET/PUT KV handlers, write helpers and
read fall-throughs all stayed wired. This had two concrete failure
modes:
- Input-field.vue silently PUT-ed per-user model picks back into
/tenants/kv/conversation-config, which requires Admin+ — Viewers
and Contributors got a 403 every time they switched chat models.
- The same write also overwrote the tenant-shared default for every
other user, turning a per-user UX into a per-tenant footgun.
Remove the field, the struct, the validate helper, the PUT case and
the two dead Summary helpers in session/helpers.go. KnowledgeSearch
loses its tenant-conversation fallback (one chunk-tool reader); the
global config + hardcoded defaults still cover it. session_agent_qa
already required agents to declare their own rerank_model_id, so no
behaviour change there.
Frontend follows:
- Input-field.vue: per-user 'last selected chat model' moves to
localStorage, no API call.
- AgentEditorModal: defaults for builtin-quick-answer creation now
come from prompt-templates (system/context/rewrite/fallback) and
retrieval-config (top_k / threshold), the two endpoints that
actually own those settings now.
- AgentSettings.vue: orphan settings page with no router/import
referencing it; deleted.
- api/system: getConversationConfig / updateConversationConfig and
the ConversationConfig interface go away.
Net: 137 insertions, 2761 deletions.
|
||
|
wizardchen
|
0a44077498 |
fix(rbac): plug ungated mutating routes; log RBAC state at startup
Sweep over the route table found four classes of mutating endpoints
that had no role guard:
- POST /tenants — creating a brand-new tenant. Now g.CrossTenant():
only org-level superusers may create tenants, matching /tenants/all
and /tenants/search.
- POST /initialization/initialize/:kbId, PUT /initialization/config/:kbId
— these change a KB's embedding/parser/storage configuration, which
is at least as sensitive as PUT /knowledge-bases/:id. New
KBCreatorLookupFromKbIDParam (sister of KBCreatorLookup but reading
:kbId) lets us reuse the OwnedKBOrAdmin matrix; routes now use the
new g.OwnedKBOrAdminFromKbIDParam() guard.
- POST /initialization/{ollama,remote,embedding,rerank,asr,multimodal,
extract}/* — system-level model probes / downloads / extraction
pipelines. These trigger LLM calls and outbound network fanout
with tenant credentials; gated to Admin+. The corresponding GETs
(status / list / progress) drop to Viewer+ since they're read-only
observability.
- POST /system/{parser-engines,docreader,storage-engine}/* — same
pattern: GETs that report state are Viewer+, the *-check /
/reconnect actions are Admin+.
POST /evaluation/ runs the eval harness which fans out LLM calls
across the tenant — gated to Admin+.
Operational improvement: log the resolved tenant RBAC state at
LoadConfig completion. air's hot-reload only rebuilds the binary on
Go-source changes; it does NOT re-source .env, so flipping
WEKNORA_TENANT_ENABLE_RBAC=true while the dev loop is already
running has no effect until the parent dev-app process restarts.
This was a real diagnostic loop in PR review — printing the value at
startup makes the "I edited .env but the gates still aren't firing"
trap obvious from the first console line.
The Printf includes both the resolved cfg.Tenant.EnableRBAC value
and the raw env var, so a mismatch between "what the .env file
says" and "what this process actually inherited" is one glance.
|
||
|
wizardchen
|
1a46fcf8ca |
fix(rbac): role-aware UI gates leaked write affordances after tenant switch
When an Owner of tenant A invited user B as a Viewer, B could log in,
switch the active tenant from B's home to A via the user menu, and
still see every write button — Create KB, Upload Document, Edit/Delete
agent — because the role-gate plumbing on the client was wired to the
wrong tenant. The server's RBAC enforcement (PR 2 / PR 5) still blocked
the actual mutations with 403, so no data leaked, but the UI presented
authority the user did not have, which is a confusing and embarrassing
defect.
Five distinct root causes, all on the client / server boundary:
1. `currentTenantRole` (frontend/src/stores/auth.ts) read
`tenant.value.id` — the user's *home* tenant — instead of the active
tenant id. After switching to A, the helper still returned B's home
role (Owner), so every `hasRole(...)` gate in the codebase silently
short-circuited. Fix: read `selectedTenantId` first, fall back to
`tenant.id`.
2. `setMemberships` was only called once, on /auth/login. Hard reloads
(which the tenant switcher triggers) re-ran `getCurrentUser()` via
`App.vue` and `router/index.ts`, but those callers discarded the
`memberships` field of the response, so the auth store kept the
login-time snapshot forever. Role changes between sessions stayed
invisible. Fix: extract memberships in both /auth/me consumers and
feed them into `setMemberships`.
3. `KnowledgeBase.vue`'s `isOwner` compared `kb.tenant_id` to the
active tenant id, so any KB inside the current tenant was treated
as "owned by the caller" — bypassing every role gate below. Fix:
compare against `kb.creator_id` (the field PR 5 introduced) and
require an actual creator match. `canEdit` / `canManage` now layer
`hasRole('admin')` on top so a non-creator with sufficient role can
still manage.
4. `KnowledgeBaseList.vue` and `AgentList.vue` rendered the per-card
more-menu (Settings, Delete, Edit) unconditionally. With the
ownership column from PR 5 this is straightforward to gate on the
client: Settings/Delete now require creator-match or `hasRole('admin')`
per the server's `OwnedKBOrAdmin` / `OwnedAgentOrAdmin` matrix.
5. The axios interceptor (frontend/src/utils/request.ts) had a
"selectedTenantId === defaultTenantId → don't attach X-Tenant-ID"
short-circuit. Any code that wrote the active tenant into
`weknora_tenant` (UserMenu's loadUserInfo, OIDC sync, router
hydrate) made the two values equal, after which the header silently
stopped attaching and subsequent navigation requests landed on the
home tenant. Fix: drop the short-circuit; always attach when
selectedTenantId is set.
Server-side, /auth/me read `user.TenantID` (the immutable home tenant
id stamped at signup) when fetching the tenant for the response, so
the `tenant` field always pointed at home even after the auth
middleware had already resolved an active tenant from X-Tenant-ID.
The frontend then re-keyed `authStore.tenant` to the home tenant on
every refresh, undoing the switch. Fix: read the active tenant id from
context (`types.TenantIDFromContext(ctx)`) and only fall back to
user.TenantID when context is unset.
The server-side enforcement was correct throughout — every fix here is
purely about not lying to the user with the rendered UI.
|
||
|
wizardchen
|
766a6c482f |
feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303)
PR 1-5 of the multi-tenant RBAC series enforce the role matrix; what they don't do is keep a durable record of who did what. Today the RBAC middleware logs "[rbac] role insufficient ..." to stderr - perishable, not searchable. Operators cannot answer "who removed Alice from tenant T-foo on Tuesday" or "did we 403 a probing user 200 times last hour" after the fact. This PR adds a generic per-tenant audit log that: - Captures mutating RBAC actions (member add/remove/role-change/leave) durably from the tenant_member service after the repo write succeeds. - Captures enforcement rejections (RequireRole / RequireOwnershipOrRole denies under EnableRBAC=true) via a gin-context-injected hook. - Sliding-window dedup (1 minute) on (tenant, actor, action, path) for denied events so a probing client cannot flood audit_logs. - Default-on - no extra env flag. Schema is generic enough for future KB / agent / datasource events to reuse via new action namespaces without another migration. Schema (migration 000044) mirrors wiki_log_entries' shape: a single audit_logs table with action, target_type/id, request path/method, outcome, and a JSONB details column. Three indexes: (tenant_id, id DESC) for the per-tenant cursor feed, actor_user_id for "what did Alice do", (tenant_id, action) for both the action filter and the dedup lookup. Sqlite mirror added for Lite mode. Wiring is nil-safe: the audit field on tenantMemberService and the gin-context lookup in rbac.go both tolerate a nil service so existing tests construct without a stub and dormant deployments degrade gracefully if the provider is misconfigured. The query API is GET /tenants/:id/audit-log, gated by Admin+ on top of the existing PathTenantMatch (denial histories should not surface to ordinary members). Cursor-paginated by descending id (sidesteps the duplicate-timestamp tie-breaking that BIGSERIAL trivially solves). Frontend: TenantMembers.vue gets a t-tabs wrapper with Members (existing) and Audit log (new) tabs; the audit tab is gated to Admin+ to mirror the server, lazy-loads on first open, paginates via the next_cursor pattern, and renders coloured chips per action/outcome so an operator can scan for anomalies. i18n keys added in all four locales. Tests: 14 new tests across service (6), middleware (4), and handler (4). Service tests pin the dedup window behaviour (writes again after expiry, dedup is per-actor-per-path, degrades on lookup error). Middleware tests pin both reject sites' audit hook and the "dormant mode does not fire" + "nil audit does not panic" invariants. Handler tests pin the cursor envelope and that an empty page returns next_cursor=0. Out of scope: per-action ACL on the audit log, retention/TTL job, KB/agent/datasource event wiring (schema supports it - just no emitters yet), async writes, frontend filter UI. |
||
|
wizardchen
|
d42a41b784 |
feat(rbac): per-KB ownership for knowledge / chunks / wiki pages (#1303)
PR 1–4 of the multi-tenant RBAC series gated KB-level and Agent-level
mutations behind a "creator OR Admin+" check via OwnedKBOrAdmin /
OwnedAgentOrAdmin. The layer underneath — knowledge entries, chunks,
wiki pages — was still gated only by Contributor, which is wider than
intended: any tenant Contributor could edit/delete documents in a
private KB they didn't own. PR 2 / wiki author left explicit follow-up
breadcrumbs at the three call sites in router.go acknowledging this gap.
This PR closes those breadcrumbs by reusing the existing scaffolding:
- KnowledgeService.GetOwningKBCreatorID walks knowledge_id -> kb_id ->
KB.CreatorID, tenant-scoped through GetKnowledgeByID. The chain lives
in the service layer so the handler-side lookups stay pure adapters.
- Three new CreatorLookup closures (KBCreatorLookupFromKnowledgeID,
KBCreatorLookupFromKnowledgeIDParam, KBCreatorLookupFromKBPath) are
added in handler/rbac_lookups.go alongside the existing
KBCreatorLookup / AgentCreatorLookup. Each translates the repository
sentinels into middleware.ErrResourceNotFound and re-checks tenant
scope as defence-in-depth, mirroring KBCreatorLookup exactly.
- router/rbac.go gains three guard factories — OwnedKnowledgeKBOrAdmin,
OwnedChunkKBOrAdmin, OwnedWikiKBOrAdmin — wired through the same
rbacGuards bundle.
- Mutating routes in RegisterKnowledgeRoutes / RegisterChunkRoutes /
RegisterWikiPageRoutes switch from g.Contributor() to the new
ownership guards. Cross-KB batch operations (POST /knowledge/move,
POST /knowledge/batch-delete, PUT /knowledge/tags) stay
Contributor-gated since there's no single owning KB to check
against. KB-scoped uploads (POST /knowledge-bases/:id/knowledge/...)
reuse the existing OwnedKBOrAdmin because the URL :id is the KB id
directly. The chunks.DELETE("/by-id/:id/questions") route addresses
chunks by their own id with no knowledge id in the URL, so it stays
Contributor-gated as a documented carve-out.
No schema changes: documents inside a shared KB are KB-owned content,
not personal uploads — that's how the product is positioned and matches
the original "per-KB-ownership granularity" wording from the PR 2
breadcrumbs. Legacy/built-in KBs with empty CreatorID surface as
tenant-owned (Admin+ only), same rule as built-in agents.
Tests:
- 9 new test cases in handler/rbac_lookups_test.go covering the three
new lookups: NotFoundMapsToSentinel (each repository sentinel),
CrossTenantIsHiddenAsNotFound, OwnerMatchReturnsCreatorID,
MissingTenantContext (fail-closed), and ChunkRouteShape (param-name
regression guard).
- 5 new test cases in service/knowledge_owner_chain_test.go covering
the chain helper: knowledge-not-found, kb-not-found, nil-kb-without-
err, happy path, and legacy KB with empty CreatorID.
Build, vet, and the targeted ./internal/middleware/...
./internal/handler/... ./internal/router/...
./internal/application/service test runs are clean. Pre-existing
TestCreateStore_* / TestOssEnsureBucket_CreateFails failures on main
are unrelated and reproduce on a clean checkout.
|
||
|
wizardchen
|
9e1c2c2546 |
refactor(tenant_member): simplify tenant ID resolution in handlers
Removed the resolveTenantIDFromPath method from TenantMemberHandler, consolidating tenant ID parsing directly into the ListMembers, AddMember, UpdateMemberRole, and RemoveMember methods. This change streamlines the code by utilizing the existing parseTenantIDFromPath function, enhancing readability and reducing redundancy. No functional changes were introduced, maintaining the existing behavior of tenant ID resolution. |
||
|
wizardchen
|
2082b63951 |
refactor(rbac): centralise tenant-access guards in middleware
Tenant-access decisions previously lived in four places with subtly different shapes: canAccessTenant in middleware/auth.go, an unexported isCrossTenantSuperuser in middleware/rbac.go, authorizeTenantAccess + two opening if-blocks in handler/tenant.go, and resolveTenantIDFromPath in handler/tenant_member.go. Each variant had its own logging, its own treatment of the cluster-wide EnableCrossTenantAccess flag, and its own subtle difference in what "URL :id matches active tenant" meant. Collapse all four into middleware/access.go: - IsCrossTenantSuperuser(ctx, cfg): exported, flag-aware. Both the per-user CanAccessAllTenants attribute AND cfg.Tenant.EnableCrossTenantAccess must be true; either alone rejects, so flipping the cluster flag off operationally revokes every superuser without a token re-issue. - IsTenantAccessible(ctx, user, target, memberService, cfg): home tenant + superuser path + active tenant_members row. Replaces the old superuser-only canAccessTenant; the X-Tenant-ID gate now also honours ordinary multi-tenant memberships. - RequireCrossTenantAccess(cfg): route guard for /tenants/all and /tenants/search, replacing the 12-line if-blocks at the top of ListAllTenants and SearchTenants. - RequirePathTenantMatch(cfg): route guard for /tenants/:id/..., replacing authorizeTenantAccess (4 callers in tenant.go) and resolveTenantIDFromPath's superuser carve-out (5 endpoints in tenant_member.go). The router declares it once on the /tenants/:id group so the rule lives next to the route definitions. router/rbac.go grows two factory methods (CrossTenant, PathTenantMatch) so RegisterTenantRoutes stays declarative and reads the same as the existing g.Viewer() / g.Owner() / g.OwnedKBOrAdmin() guards. The /tenants/:id endpoints now group under one PathTenantMatch() middleware instead of each handler re-deriving the check. Handler cleanups follow naturally: tenant.go drops authorizeTenantAccess and the cross-tenant if-blocks; tenant_member.go drops the cfg field and shrinks resolveTenantIDFromPath to a thin parser. NewTenantMemberHandler loses its *config.Config argument; container/container.go's dig wiring picks up the simpler signature automatically. middleware/access_test.go covers all four exports with truth-table cases (flag/attribute/membership combinations, lookup errors, missing context). tenant_member_test.go's existing handler-level cross-tenant assertions still pass because memberTestRouter now mounts RequirePathTenantMatch the same way the production router does. Net -41 lines across 8 changed files + 2 new files; no behaviour change other than the X-Tenant-ID gate accepting active memberships (which was the original PR 4 goal that motivated this refactor). |
||
|
wizardchen
|
543e9709be |
fix(rbac): address PR 3 + leave-endpoint review feedback
Security - Reject URL :id when it does not match the caller's active tenant on every /tenants/:id/members/* and /tenants/:id/leave endpoint. Without this an Owner of tenant A could drive operations on tenant B simply by changing the URL, because RequireRole resolves the caller's role against the active tenant from context, not the URL param. The cross-tenant superuser carve-out mirrors middleware/rbac.go (CanAccessAllTenants + EnableCrossTenantAccess required). - AddMember no longer records the synthetic "system-<tenantID>" caller (X-API-Key auth path) as invited_by, matching the same filter KB.CreatorID got in PR 2. - AuthHandler now logs a loud one-shot error when constructed with a nil/incomplete config so a wiring bug that silently disables the invite_only gate is visible at boot. Race fixes - New repo helpers DemoteOwnerAtomically / RemoveOwnerAtomically wrap the count-other-owners + write step in a transaction that holds SELECT … FOR UPDATE locks on every other active Owner row. The service routes Owner demotion / Owner removal through them, closing the TOCTOU window where two concurrent demotes of two different Owners could each observe count=2 and both commit, leaving the tenant ownerless. Performance - ListMembers hydration now batches user lookups via a new GetUsersByIDs interface method (one SELECT … WHERE id IN (…) instead of N round-trips). Failure degrades to "no email/username" rather than dropping rows. Frontend - canManage also fires for cross-tenant superusers (authStore.canAccessAllTenants), mirroring the backend carve-out so the UI controls match what the server actually allows. Tests - Cross-tenant URL guard regression cases for List / Add / Update / Remove / Leave. - Superuser bypass tests (flag on -> service reached, flag off -> 403). - Batched lookup test ensures GetUsersByIDs is called exactly once and GetUserByID is never used as a fallback. - Synthetic caller invited_by filter regression test. - Service-layer atomic last-Owner regression tests for both UpdateRole and RemoveMember. |
||
|
wizardchen
|
ae7cd02cd6 |
fix(rbac): polish tenant member UI; add /leave endpoint
Self-review of PR 3's TenantMembers.vue surfaced a handful of real bugs
and a few rough edges. This commit fixes them and adds the self-leave
endpoint that PR 3's design called out as a follow-up.
Frontend bug fixes
- Read err.status (not err.response.status). The axios interceptor in
utils/request.ts already flattens errors into { status, message,
... }, so the original branch-by-status code never matched and every
4xx fell through to the generic toast. Fixes 404/409 surfacing as
"userNotFound" / "alreadyMember" / "lastOwner" instead of "Something
went wrong".
- t-dialog @confirm replaces :on-confirm so the dialog stays open on
validation failure and on API error. Users can fix and retry without
retyping the email.
- Drop the t-select revert hack; with :model-value the rendered value
is already source-of-truth-on-row.role, so no setTimeout dance is
needed.
- Don't toast "no active tenant" on cold mount; watch the activeTenantId
ref and load once the auth store hydrates.
Frontend UX additions
- Empty state via t-empty (also covers the search-no-results case).
- Compact role-permissions matrix above the table, mirroring
OrganizationSettingsModal so Owners can see what each role grants
before promoting/inviting.
- Search box filtering by username/email (client-side, sufficient for
expected per-tenant member counts).
- "Leave Tenant" button — visible to any active member except the
last-Owner case (server still enforces).
Backend
- New POST /tenants/:id/leave route. Calls TenantMemberService.RemoveMember
with the caller's own user id so non-Owners can quit on their own.
Gated Viewer+ in the matrix; the existing last-Owner invariant in the
service still rejects last-Owner self-leave with 409.
- 3 new handler tests cover the happy path, the last-Owner 409 path,
and the missing-caller-id 401 path (defensive — auth middleware
always attaches a user, but the test pins the contract).
i18n
- Add tenantMember.permissions / search / leave / empty strings in all
four locales (en/zh/ko/ru).
|
||
|
wizardchen
|
3f7150cb5c |
feat(rbac): tenant member management + invite_only gate (PR 3 of 3, #1303)
Wire the human-facing surface for the tenant-RBAC system that PR 1 introduced and PR 2 enforced: REST endpoints + UI for managing tenant members, plus the long-pending invite_only registration gate. Backend - internal/handler/auth.go: Register now rejects with 403 when auth.registration_mode=invite_only. PR 1 already exposes the mode via /auth/config so the frontend hides the sign-up link; this is the server-side enforcement for direct API calls. - internal/handler/tenant_member.go: new TenantMemberHandler with ListMembers / AddMember / UpdateMemberRole / RemoveMember. Maps service-layer sentinels onto HTTP semantics (ErrLastOwner -> 409, ErrMembershipAlreadyExists -> 409, ErrUserNotFound on AddMember -> 404, ErrInvalidTenantRole -> 400). AddMember requires the invitee to already have an account; SMTP-driven email invites are deferred. - internal/types/tenant_member.go: new TenantMemberResponse projection carrying the fields the management UI renders (email, username, avatar, role, joined_at). - internal/router/router.go: four routes under /tenants/:id/members gated per the matrix (Viewer+ for list, Owner+ for every mutation). - internal/container/container.go: dig provider for the new handler. Tests - handler/tenant_member_test.go: 12 table-driven cases covering happy path, dangling-user tolerance, ErrLastOwner -> 409, ErrMembershipNotFound -> 404, ErrMembershipAlreadyExists -> 409, unknown email -> 404, invalid role rejected before user lookup, bad tenant id -> 400. - handler/auth_register_invite_only_test.go: invite_only -> 403, self_serve -> 201, nil Auth config falls through. Frontend - src/api/tenant/members.ts: typed client (TenantRole/TenantMember/...) for the four endpoints. - src/views/settings/TenantMembers.vue: members table with add dialog, inline role select, confirm-remove modal. All mutation widgets gated behind currentTenantRole === 'owner'; server still enforces. - src/views/settings/Settings.vue: register the Members tab; show only when currentTenantRole is loaded (avoids tab flash on cold nav). - i18n in en-US/zh-CN/ko-KR/ru-RU: new tenantMember namespace covering all UI strings and structured error messages (lastOwner / userNotFound / alreadyMember / invalidRole etc). This completes the #1303 RBAC rollout. With cfg.Tenant.EnableRBAC=true operators now have a full management surface; with EnableRBAC=false the routes still exist but mutations log-and-pass per PR 2 contract. |
||
|
wizardchen
|
0f0f7e4339 |
test(rbac): cover review gaps from PR 2 self-review
- Add TestRequireRole_CrossTenantSuperuserBypass: pin the CanAccessAllTenants -> Owner-gate bypass behaviour for RequireRole, matching the existing coverage on RequireOwnershipOrRole. Catches reorderings of the fast paths that would re-lock org-level superusers out of foreign-tenant Owner-only routes. - Add TestAgentCreatorLookup_CrossTenantIsHiddenAsNotFound: lock down the defence-in-depth tenant re-check inside AgentCreatorLookup so a future refactor that loosens the service-layer scope (or wires a cross-tenant variant by mistake) cannot leak access via the ownership shortcut. - Reword RequireOwnershipOrRole step 7-8 doc comment so "role check already failed" is unambiguous (it refers to step 1 above). - Annotate the synthetic-system-user construction in middleware/auth.go with a back-pointer to types.IsSyntheticUserID so anyone changing the prefix or numeric suffix knows to update the helper too. |
||
|
wizardchen
|
8b71def60b |
fix(rbac): address PR 2 review feedback
Middleware - RequireOwnershipOrRole now skips the creator lookup entirely when EnableRBAC is off, eliminating a hidden per-request DB roundtrip in the rollout window. - New ErrResourceNotFound sentinel lets lookups signal "not visible to this tenant"; the middleware passes the request through so handlers can return a real 404 instead of a misleading 403. - Transient lookup failures now surface as 503 (was 403) so monitoring can tell denied-by-policy from cannot-verify-right-now. - Cross-tenant superusers (CanAccessAllTenants) bypass tenant role gates, restoring DELETE /tenants/:id and ResetAPIKey for org operators that resolve to Admin in foreign tenants. - Construction-time warning fires once when cfg is nil/incomplete, so a wiring bug that disables enforcement globally is visible at boot. Route coverage - Add role guards to FAQ, KnowledgeTag, WikiPage, Skill, WebSearch, WebSearchProvider, WeKnoraCloud, Session, Chat, and Message route groups so the matrix actually matches the PR description. Service layer - KB and agent creators are no longer recorded as the synthetic system-<tenantID> user used by the X-API-Key path; resources stay tenant-owned. New types.IsSyntheticUserID helper centralises the prefix check. - KBCreatorLookup and AgentCreatorLookup re-check tenant scope so a same-ID resource in another tenant cannot piggyback through the ownership shortcut. Docs - CustomAgent.RunnableByViewer comment now matches enforcement: it only blocks Viewer-driven agent-mode runs, not plain RAG/Wiki QA. Tests - New cases for the 503/404 paths, lookup-skip behaviour, cross-tenant superuser bypass, the lookup helpers (not-found, cross-tenant, built-in agent, missing tenant context), and IsSyntheticUserID. |
||
|
wizardchen
|
9e3c92ca95 |
feat(rbac): enforce tenant role matrix on routes (PR 2 of 3, #1303)
Build on PR 1's tenant_members schema by enforcing the role matrix on
every authenticated route and at agent run time. Enforcement is gated
behind cfg.Tenant.EnableRBAC so the change ships dormant — the same
code paths log the would-be rejection during the rollout window and
start blocking once operators flip the flag.
Middleware (internal/middleware/rbac.go):
- RequireRole(min, cfg): role-only gate using TenantRole.HasPermission.
- RequireOwnershipOrRole(min, lookup, cfg): allows the original creator
through even when their role is below min, otherwise falls back to
the role check. Lookup errors fail closed; an empty creator_id is
treated as tenant-owned and requires the role.
- Both honour cfg.Tenant.EnableRBAC for the rollout window.
Creator lookups (internal/handler/rbac_lookups.go):
- KBCreatorLookup resolves :id -> KnowledgeBase.CreatorID.
- AgentCreatorLookup resolves :id -> CustomAgent.CreatedBy; built-in
agents (IsBuiltin=true) return ("", nil) so they remain Admin+ only.
- Compile-time guards keep the methods aligned with CreatorLookup.
Route guards (internal/router/rbac.go + router.go):
- rbacGuards bundles the matrix into Viewer/Contributor/Admin/Owner
helpers plus OwnedKBOrAdmin / OwnedAgentOrAdmin. Each Register*
function takes *rbacGuards so route lines remain readable.
- KB / Agent / Knowledge / Chunk / Tenant / Model / MCPService /
VectorStore / DataSource / IMChannel routes annotated per the PR 1
design doc matrix. Tenant-wide infra (models, vector stores, MCP,
data sources, IM channels) reads as Viewer+ and writes as Admin+;
per-resource mutations on KB / Agent route through the
ownership-or-Admin guard.
Resource ownership (services):
- CreateKnowledgeBase / CopyKnowledgeBase populate CreatorID from the
authenticated user, so the OwnedKBOrAdmin guard has a creator to
match.
- CreateAgent / CopyAgent populate CreatedBy similarly. Built-in
agents stay tenant-owned.
Agent run-time gate (internal/handler/session/qa.go):
- AgentQA refuses to launch the agent engine for Viewers when the
target agent has RunnableByViewer=false. Same EnableRBAC gating as
the middleware so it's a no-op until enforcement is turned on.
Tests:
- 12 unit tests cover RequireRole / RequireOwnershipOrRole, including
the fail-open and fail-closed branches and the lookup-error path.
|
||
|
wizardchen
|
af23483433 |
fix(auth): honour JWT tenant_id claim in middleware
POST /auth/switch-tenant mints an access token whose `tenant_id` claim
points to the target tenant, but the middleware previously discarded
that claim entirely:
user, err := userService.ValidateToken(ctx, token) // returned only *User
targetTenantID := user.TenantID // always the home tenant
The session therefore stayed scoped to the user's home tenant despite
the supposedly fresh token, making the new endpoint a no-op once
the request hit any tenant-scoped query.
This change widens the contract:
- UserService.ValidateToken now returns (user, activeTenantID, error)
where activeTenantID is the JWT's `tenant_id` claim, falling back to
user.TenantID when the claim is absent (backward compatibility with
pre-RBAC tokens still in the wild).
- middleware/auth.go uses activeTenantID as the default target before
applying any X-Tenant-ID header override, and recomputes
crossTenantSwitch from `activeTenantID != user.TenantID` so the
resolveTenantRole branches stay correct for switch-tenant sessions.
- The non-middleware /auth/validate handler in handler/auth.go absorbs
the new return value (it only cares about user identity).
Tests:
- internal/application/service/user_jwt_test.go pins the JWT->tenant-id
projection (`tenantIDFromClaims`) — claim present beats fallback,
zero/negative/string claims fall back, int64/uint64 are accepted for
test-built tokens.
Required for PR 2 multi-tenant integration tests, where a single user
needs to switch between Owner-in-A and Viewer-in-B sessions and observe
distinct authority on each side.
|
||
|
wizardchen
|
f3d1210577 |
refactor(rbac): address PR 1 review feedback
Tighten correctness, security and observability of the tenant RBAC foundation introduced in 3c4cdd70 in response to internal review. Bugfixes - custom_agents.runnable_by_viewer: drop the GORM `default:true` tag. GORM treats a plain bool's zero value as "unspecified" and silently rewrites it to the column default on insert, which would prevent admins from ever creating an agent with viewer-runtime disabled. The column-level DEFAULT TRUE on the database side still backstops rows inserted by raw SQL / migrations. - TenantMemberRepository.HasAnyMembers: replace `Limit(1).Count(&n)` with `Take(&probe)`. GORM's Count ignores Limit and scans the whole table, which matters because this runs on the auth middleware's hot path for users without a cached membership. - service/user.LoginWithOIDC: include User, Tenant and Memberships in the OIDCCallbackResponse so the OIDC flow returns the same shape as password login (previously User and friends were left nil). Security - middleware/auth.resolveTenantRole: restrict orphan-tenant auto-promotion to the caller's home tenant. Cross-tenant superuser visits and JWTs minted for foreign tenants no longer trigger silent Owner grants. Two regression tests pin the new behaviour. - service/user.synthFallbackMembership: default the synthesised role to Viewer instead of Owner. The login response only feeds UI; the backend re-derives the real role from tenant_members on every request, so showing a Viewer UI is preferable to a misleading Owner UI that would surface admin controls the backend will then 403. - migrations/versioned/000043_tenant_rbac.down.sql: gate the destructive drop behind a `weknora.allow_destructive_migration` GUC so an accidental `migrate down` cannot wipe role data in production. - frontend/src/stores/auth.ts: comment currentTenantRole as rendering-only so future PRs cannot mistake the localStorage-backed value for an authorisation signal. API contract - Memberships in LoginResponse and OIDCCallbackResponse now serialises without omitempty, and buildMembershipsForUser / synthFallbackMembership always return a non-nil slice — frontend can rely on JSON `[]`. - handler/auth.GetCurrentUser includes memberships so /auth/me refreshes recover currentTenantRole after a page reload. Performance - Add TenantService.GetTenantsByIDs (with the matching repo method) and use it inside buildMembershipsForUser to batch tenant-name lookups instead of issuing one query per membership. Observability - Replace the remaining `log.Printf` calls in middleware/auth with contextual logger.Warnf / logger.Infof so role resolution events ride the standard request-id / Langfuse trace plumbing. The successful auto-promote line is tagged `[audit]` for log mining. Tests - internal/application/service/tenant_member_test.go: hand-rolled fakeTenantMemberRepo plus 11 cases covering AddMember validation, EnsureOwner idempotence, "cannot demote/remove the last Owner", same-role no-op, ErrInvalidTenantRole / ErrMembershipNotFound, and the TenantRole.HasPermission lattice. - internal/middleware/auth_role_test.go: fakeMemberService plus 8 cases pinning every branch of resolveTenantRole, including the two H1 regressions (cross-tenant superuser must not auto-promote; orphan auto-promote requires the home tenant). |
||
|
wizardchen
|
7f7a48880f |
feat(rbac): add tenant_members schema and auth wiring (PR 1 of 3, #1303)
Lay the foundation for tenant-level RBAC without enforcing it yet. Schema, types, middleware lookup, and login-flow shape changes ship now; RequireRole/route annotations and member-management endpoints follow in PR 2 and PR 3. Highlights: - New tenant_members table (Postgres + SQLite) with (user_id, tenant_id) unique key and (owner|admin|contributor|viewer) role values. Migration 000043 backfills one row per existing user, marking the earliest-created active user in each tenant as Owner and the rest as Contributor. - KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so PR 2 can wire ownership-based and Viewer-runtime checks. - Auth middleware loads the active TenantMember row, sets a new TenantRoleContextKey, and falls back via three escape hatches: cross-tenant superusers (CanAccessAllTenants) get Admin in the target tenant; orphan tenants (zero members) auto-promote the first authenticating human to Owner; and when tenant.enable_rbac is false (the v1 default) unmatched lookups fail open with Admin to preserve current behaviour. - Register / OIDC provisioning now insert an Owner membership for the registrant via TenantMemberService.EnsureOwner. - LoginResponse renamed tenant -> active_tenant and gained a memberships array. POST /auth/switch-tenant issues a fresh token pair scoped to a target membership; GET /auth/config exposes auth.registration_mode for the frontend register-tab gating. - New config keys: auth.registration_mode (self_serve | invite_only, defaults to self_serve) and tenant.enable_rbac (defaults to false). Both have WEKNORA_* env overrides. - Frontend store persists memberships and exposes currentTenantRole; Login.vue calls /auth/config and hides the Register tab in invite_only mode. Go SDK types updated to match the new LoginResponse shape. |
||
|
ochan.kwon
|
0e8de6192c |
feat(knowledge-base): validate vector store bindings on create, copy, and delete
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory into the user-facing knowledge-base lifecycle: - POST /knowledge-bases validates the requested vector_store_id against the caller's tenant scope and the engine registry. New error codes ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201) distinguish the typed branches without echoing UUIDs to the client. - GET / POST / PUT / PUT-pin responses embed the bound store's display metadata (name, source, engine_type, status) without exposing any connection credentials. Cross-tenant shared KBs receive a suppressed payload (vector_store_id stripped, source="shared") so operator-chosen store names cannot be enumerated across tenants. - POST /knowledge-bases/copy synchronously rejects clones whose target has a different embedding model or vector store, before the async clone task is enqueued. The async clone worker re-applies the same checks for defense in depth. - DELETE /vector-stores/:id refuses to remove a store with bound KBs, inside a transaction that row-locks the store on PostgreSQL and serializes via WAL on SQLite. unregister-from-registry is wrapped in defer/recover so a panic surfaces as a structured warning instead of silently leaking a stale engine. - vector_store_id is immutable after creation. The GORM <-:create tag blocks every ORM update path; the service-layer DTO omits the field entirely; a reflection-based regression test catches any future maintainer who adds it back to either layer. - Empty-string vector_store_id is normalized to nil at both the create path and inside SharesStoreWith, so rows persisted by callers that did not run Normalize first cannot trip false same-store comparisons. Part of #993. Depends on #994 and #1310. |
||
|
wizardchen
|
ce8969ab14 |
fix(initialization): surface upstream and SSRF errors verbatim in test-connection responses
Test-connection's "连接失败" was nearly useless for debugging because two layers conspired to swallow the actual upstream error: Backend swallow --------------- checkChatModelConnection and CheckASRModel pattern-matched on the upstream error string (401 / 404 / timeout / ...) and replaced it with a fixed Chinese hint. The original SDK message (request URL, response body, provider's error code description) was discarded. Refactor: extract classifyConnectionError(errMsg) — returns just the short Chinese hint. Callers combine "<hint>：<raw err>" so the hint indicates "where to start looking" and the raw error shows "what actually happened". The status-400 special-case (endpoint reachable, parameter mismatch only) still returns success. Frontend swallow ---------------- ModelEditorDialog.testConnection ignored result.message on success paths (fine), but also overwrote it with a generic i18n string on failure paths and 4xx rejections — even though axios's interceptor already lifts the backend's error.message to error.message. Failure branch now uses error?.message || result.message as the toast content; i18n fallback only when neither is present. SSRF allowlist hint ------------------- SSRF validation failures previously returned "Base URL 未通过安全校验: <reason>" with no mention of how to recover (the allowlist is server-side, not configurable in the UI). Operators hit it and were stuck. New utils.FormatSSRFError(label, rawURL, err) wraps the error with guidance to add the host to SSRF_WHITELIST_EXTRA (the operator-facing append-only escape hatch — SSRF_WHITELIST itself is the project's baseline). The hint echoes the failed host so it can be copy-pasted straight into the env var: Base URL 未通过安全校验：DNS resolution failed for hostname test.local.com ... 如该地址确实可信，请联系运维在服务端环境变量 SSRF_WHITELIST_EXTRA 中加入该主机（支持精确域名 / *.example.com 通配 / IP / CIDR），示例：SSRF_WHITELIST_EXTRA=test.local.com, *.example.com,10.0.0.0/8 All SSRF call sites switch to the helper: initialization (CheckRemoteModel / TestEmbedding / CheckRerank / CheckASR / TestMultimodal / batch validation), model CRUD, mcp_service CRUD, knowledge URL ingest, system ReconnectDocReader. |
||
|
wizardchen
|
a9d37970ae |
fix(initialization): fall back to stored apiKey when test-connecting an existing model
Direct regression from the previous credential subresource refactor: the edit-mode ModelEditorDialog stopped binding apiKey into formData (the field now lives in <CredentialResource>), so "Test connection" sent apiKey="" and every upstream call (DeepSeek / lkeap / OpenAI etc.) 401'd. Fix in the backend so the frontend stays simple: - ModelTestRequest gains an optional ModelID field. - New fillSecretsFromStoredModel helper: when ModelID is set AND a secret field on the request is empty, copy it from the stored (decrypted) model. Non-empty request values still win, so users typing a new key to verify it before saving behave the same as before. - The four test entrypoints (chat / embedding / rerank / asr) call the helper right after ShouldBindJSON. Frontend: ModelEditorDialog.testConnection forwards modelId when isEdit is true. The four API client signatures accept the optional field. WebSearch and MCP are unaffected: WebSearch already has a TestProviderByID ID-based path, MCP has no standalone raw test endpoint. |
||
|
wizardchen
|
7643dd457e |
refactor(credentials): introduce /credentials subresource pattern
Replace the legacy "redacted placeholder + Clear* boolean" pattern with
dedicated per-resource credential subresources across MCP services,
Models, Web search providers, and Data sources.
Why
---
The previous design had three problems:
1. Main PUT body carried secret fields. The frontend echoed back a
redacted "***" placeholder, and a fragile MergeUpdate / IsRedactedOrEmpty
defense in the service layer tried to detect "user did not change this"
vs "user wants to clear this". A regression in that defense (or a new
frontend forgetting it) silently overwrites the stored secret with the
placeholder.
2. The "remove this credential" UX was a red checkbox under a pre-filled
password input. Three intents (preserve / replace / clear) collapsed
onto one field, and credential changes were bundled with unrelated
config edits in the same submit. Users wiped working keys by mistake.
3. Secret presence was inferred from "did the response come back with a
'***' placeholder", which couples the contract to a magic string.
Design
------
Each of the four resources now exposes:
PUT /{resource}/{id}/credentials # write one or more fields
DELETE /{resource}/{id}/credentials/{field} # clear a single field
"Is this configured?" metadata travels on the main resource response as
a typed map (dto.MCPServiceResponse.Credentials etc.) — no separate GET
endpoint. The frontend reads the configured boolean from the main GET
and never sees secret values at all.
Main PUT endpoints now ignore any secret fields in the request body and
log a deprecation warning if they appear, so legacy clients fail loudly
rather than overwriting silently.
Frontend
--------
- New reusable <CredentialResource> component renders a three-state card
(unconfigured / configured / editing) and drives the dedicated
endpoints. Used by MCP, Model, Web search; DataSource has a bespoke
card with the same behaviour because its credentials are a single
atomic per-connector map.
- Cancel from edit mode now restores state synchronously from the meta
prop. The previous async refresh() was a no-op while state was still
'editing', leaving the input frozen open.
- Remove is single-click + toast. The danger-themed button is the
deterrent; a modal confirm adds friction without adding safety (the
plaintext is irrecoverable client-side either way — recovery means
re-typing).
DTOs (internal/handler/dto/) are deliberately separate from the GORM
models so "no secret in response" is a compile-time invariant: a future
contributor cannot leak a secret without explicitly adding the field to
the DTO, which is review-able in a single diff.
Storage is unchanged — credentials still live in the existing jsonb /
parameters columns. No schema migration.
Cleanup
-------
- types.MCPService / types.Model / types.WebSearchProviderEntity /
types.DataSourceConfig: drop ClearAPIKey / ClearToken / ClearAppSecret
/ ClearCredentials boolean fields, MergeUpdate(), RedactSensitiveData().
- utils/types/secret.go: drop PreserveIfRedacted / IsRedactedOrEmpty.
RedactedSecretPlaceholder constant is retained because VectorStore
still uses the old pattern and is out of scope here.
- Frontend hasExistingApiKey / clearApiKey / convertToLegacyFormat
redaction handling removed; i18n keys renamed secret -> credential.
|
||
|
ochan.kwon
|
d869cc1ff6 |
fix: redact sensitive fields and prevent credential loss on edit
Closes #988 Unify the "write-only secrets" pattern across MCP services, Models, WebSearch providers, and DataSources. Fix two related defects: 1. GET responses leaked plaintext credentials — GetMCPServiceByID while List masked, all Model / WebSearchProvider / DataSource GET responses unconditionally. 2. When the frontend echoed a masked placeholder back on save, the update path overwrote the stored secret with the mask — silently wiping the real credential. MCP surfaced this most visibly: ListMCPServices masked Token to "first4****last4" and McpServiceDialog round-tripped that masked string back into the DB. - internal/types/secret.go introduces the shared sentinel RedactedSecretPlaceholder = "***" and two helpers: - IsRedactedOrEmpty(s): true for "" or the sentinel - PreserveIfRedacted(incoming, existing): merge predicate - Each resource adds a RedactSensitiveData (mutates-in-place) or Redacted (returns-a-copy) method and a MergeUpdate contract: empty or placeholder preserves, any other value replaces, and a write-only Clear* bool flag (gorm:"-", json omitempty) wipes with precedence over preserve / replace. Clear* never persists. - Every GET / List / Create / Update response path now redacts. VectorStore's pre-existing "***" literal is refactored onto the shared constant for consistency. All four dialogs — McpServiceDialog, ModelEditorDialog, WebSearchSettings, DataSourceEditorDialog — never pre-fill secret inputs. The stored-state signal is collapsed into the input placeholder: "•••••••• (Enter new value to replace)" when the server reports a stored value, falling back to the connector-specific placeholder otherwise. The input itself carries the "something is there" cue, matching how password dots render. An opt-in "Remove this credential" checkbox appears in edit mode when a value is stored; checking it disables the input and triggers a danger-themed confirmation dialog before submit. DataSource rolls the clear up to a connector-level "Remove all credentials" flag since its credentials are per-connector atomic bundles (e.g. Feishu app_id + app_secret). i18n updates across en-US, zh-CN, ko-KR, ru-RU: secret.storedPlaceholder / clearHint / confirmClearTitle / confirmClearBody. - MCP update handler's manual map-to-struct projection was silently dropping clear_api_key / clear_token / custom_headers on partial updates. Pull them off the map explicitly. - MCP configChanged detection now snapshots URL / StdioConfig / TransportType before the merge so connection recycling still triggers on real transport changes (not just whenever AuthConfig is present). - MCP CustomHeaders: nil preserves existing, non-nil replaces. Was always overwriting, silently wiping headers on any auth-bearing update. - GetMCPServiceByID now redacts (was plaintext while List masked). - WebSearchProvider update preserves Name / Description when empty instead of clobbering them. - WebSearchProvider TestProviderRaw rejects the "***" placeholder up front with an actionable message rather than forwarding it to the upstream provider API as a literal key. - Model builtin hideSensitiveInfo now preserves non-secret metadata (SupportsVision, Provider, ExtraConfig, InterfaceType, ParameterSize, EmbeddingParameters) that were previously zeroed out for builtin model cards. - Replace local equalStringSlices / equalStringMaps helpers with stdlib slices.Equal / maps.Equal (Go 1.21+). - internal/types/{secret,mcp,model,web_search_provider,datasource}_test.go: table-driven coverage of merge, redact, clear, and write-only flag invariants for every resource. - internal/application/service/mcp_service_test.go: full service layer with a fake repo; covers ListMCPServices / GetMCPServiceByID redaction, UpdateMCPService partial-update honoring of Clear*, CustomHeaders preserve-on-nil / replace-on-non-nil, and clear-flag precedence over submitted values. Additive only. No changes to request or response JSON shape beyond the write-only Clear* flags (omitted by default via omitempty). No DB schema changes. No SDK struct field removals. Existing clients that don't know about the new flags continue to function unchanged. - DataSource.Config jsonb at-rest encryption. Pre-existing residual risk; tracked as a follow-up. - Renaming VectorStore.MaskSensitiveFields to match the new RedactSensitiveData / Redacted naming convention. Left alone to minimize diff; a follow-up rename PR can unify. |
||
|
wizardchen
|
4fb089d4d7 |
fix(kb): map ErrKnowledgeBaseNotFound to 404 across handler helpers
Five handler helpers (validateAndGetKnowledgeBase in knowledgebase.go,
validateKnowledgeBaseAccessWithKBID in knowledge.go, the kbService
guards in faq.go and tag.go, and getKnowledgeBaseForInitialization +
the per-kb config getter in initialization.go) wrapped every
GetKnowledgeBaseByID error — including the well-known
repository.ErrKnowledgeBaseNotFound sentinel — as
NewInternalServerError. The result was that every probe of a stale or
cross-tenant kb id surfaced as a 500 instead of the 404 it should have
been, both confusing clients ("real 5xx vs. wrong URL") and burning
ops attention on monitoring alerts.
The mapping pattern is the same as PR #1336 for sessions: detect the
sentinel via stderrors.Is and emit NewNotFoundError; everything else
still surfaces as a 500 so genuine DB / repo failures keep firing the
alerts that matter. Caught during the RBAC e2e smoke run on
feat/rbac, where a deliberate cross-tenant kb-id probe produced
HTTP 500 + body "knowledge base not found" — the smoking gun.
Tests: new internal/handler/knowledgebase_not_found_test.go covers
three cases — bare sentinel, fmt.Errorf("%w") wrapped sentinel
(regression guard against a future revert to `==`), and a non-sentinel
infrastructure error that must still 500. All three pass.
The full handler test package is green.
|
||
|
wizardchen
|
d7b5835519 |
fix(session): map ErrSessionNotFound to 404 across all handlers
PR #1309 plumbed user-scope into the session/message service layer so non-owner / wrong-tenant lookups now surface as ErrSessionNotFound at every entry point that goes through sessionRepo.Get. The handler-side mapping, however, only existed in 4 of the 8 affected sites: - handler/session/handler.go:GetSession ✓ already mapped (with `==`) - handler/session/handler.go:UpdateSession ✓ already mapped (with `==`) - handler/session/handler.go:DeleteSession ✓ already mapped (with `==`) - handler/session/handler.go:BatchDeleteSessions ✓ already mapped (with `==`) - handler/session/handler.go:ClearSessionMessages ✗ always 500 - handler/session/stream.go:ContinueStream session lookup ✓ (with `==`) - handler/session/stream.go:ContinueStream message lookup ✗ always 500 - handler/message.go:LoadMessages (recent + before-time) ✗ always 500 - handler/message.go:DeleteMessage ✗ always 500 Result: a Contributor in tenant A asking for a Contributor B's session messages got a 500 instead of 404 — the wire response leaked "something is broken" rather than "you can't see this URL", which breaks SDK / frontend axios interceptor error handling that keys on HTTP status. Two changes: 1. Add ErrSessionNotFound → 404 mapping to the 4 sites that lacked it (ClearSessionMessages, ContinueStream's GetMessage, LoadMessages's two paths, DeleteMessage). 2. Replace `err == errors.ErrSessionNotFound` with `stderrors.Is(err, errors.ErrSessionNotFound)` everywhere. Sentinel error comparison must use errors.Is so wrapped errors (`fmt.Errorf("...: %w", ErrSessionNotFound)`) still match. Today no service-layer caller wraps, but a future refactor that does would silently turn 404s into 500s with no test catching it. Tests: - internal/handler/message_session_not_found_test.go covers the four new mapping sites in handler/message.go (LoadMessages-recent, LoadMessages-before-time, DeleteMessage) and pins the wrapped-error behaviour with a dedicated regression test that fmt.Errorf("%w") a sentinel must still be detected as ErrSessionNotFound. The bare handler is hit through gin.Engine + middleware.ErrorHandler so the response shape matches production exactly. - PR #1309's existing service-level tests (session_user_scope_test, session_test on the repository) still pass — this PR is a strictly handler-layer follow-up. |
||
|
wizardchen
|
6f95c75ed2 |
feat(system-info): surface DB migration errors with troubleshooting links
When a startup database migration fails (e.g. issue #1319: pg_trgm not available so 000041 cannot build its trigram index), the application intentionally keeps booting so operators can reach the UI to diagnose. However, before this change the system info page silently dropped the "DB Version" row because the value was empty: - migration.go only cached the version after a successful m.Up(); the error path returned early and left migrationVersionSet=false. - system.go used CachedMigrationVersion's ok=false to skip emitting db_version, and the JSON tag was already omitempty. - SystemInfo.vue gated the entire row on v-if="systemInfo?.db_version". The end result was the most useful diagnostic surface vanishing in the exact failure mode that needs it most — Wiki ingest and KG features would silently produce nothing with no UI hint. Changes: - migration.go: replace sync.Once-based setter with an RWMutex-guarded state struct holding {version, dirty, err}. Every failure path now calls captureMigrationFailure(m, err), which best-effort reads m.Version() so the cached value still reflects the partial state. - system.go (handler): always emit db_version (falling back to "unknown" when no version could be read), append " (failed)" when an error is recorded, and add db_migration_error to the response. - swagger / client SDK: keep the API contract in sync with the new response field. - SystemInfo.vue: render the DB version row whenever either field is present, show a "Migration failed" danger tag, and add a full-width alert below the row carrying the error message plus two links: 1. View troubleshooting guide -> new docs/migration-troubleshooting.md 2. Report an issue -> github.com/Tencent/WeKnora/issues/new, prefilled with the captured error and environment metadata. - docs/migration-troubleshooting.md: new self-service guide covering the common failure modes (missing extension, dirty state, privileges, out of disk, schema drift) with concrete psql / make commands. - i18n: add the new keys to zh-CN, en-US, ko-KR, ru-RU. Refs #1319. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
wolfkill
|
8f4e5a459f | fix(session): scope session access by user | ||
|
wizardchen
|
55caeefc3d |
feat(observability): log end-to-end TTFB on both ends of chat stream
To diagnose where latency lives between "user hits send" and "first token appears" we need a single number that can be matched across the browser console and the server log. Add correlated TTFB markers keyed by X-Request-ID: Frontend (streame.ts) * Generate the X-Request-ID once and reuse it for logging. * Log request:start when fetchEventSource is invoked, response:headers when onopen fires, and response:first_answer the first time an SSE payload with response_type === 'answer' arrives. Filtering by event type avoids treating session_title / references / tool_call as the "first token". Backend (session handler) * parseQARequest records the wall-clock entry time and logs TTFB:start with the same X-Request-ID. * qaRequestContext carries receivedAt through to the stream handler. * AgentStreamHandler emits a one-shot TTFB:first_answer_chunk log the first time it observes an AgentFinalAnswerData chunk, so the delta against TTFB:start is the server's request-in → first-token-out budget. The frontend delta minus the backend delta is then attributable to network + gin middleware, which were previously invisible. |
||
|
wizardchen
|
cacca049d9 |
feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base detail page — parse status, source/channel, and updated time range — and rework multi-select to no longer cause the card title to jitter on hover. Backend - Introduce types.KnowledgeListFilter to aggregate optional filter dimensions (tag, keyword, file_type, parse_status, source, updated_from/to) and switch ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept it instead of a growing positional parameter list. - The ListKnowledge HTTP handler accepts new parse_status, source, start_time and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS" and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type column to stay consistent with file_type semantics; other source values match the channel column. - Update the four other callers (agent_service, initialization) to pass an empty filter struct, preserving prior behavior. Frontend - Add three controls in the doc-filter-bar (status select, source select, date-range picker with future-date disabled) wired through getKnowled / listKnowledgeFiles into the new backend params. - Replace the hover-triggered card checkbox with an explicit "批量管理" mode (mirrors the session list UX): in card view the checkbox only renders while batch mode is on, entered via the per-card "..." menu; the list view keeps its leading checkbox column. Switching from list to grid auto-enables batch mode when something is already selected, so the selection stays visible. - DocumentBatchBar now stays open whenever batch mode or selection > 0, and its "取消选择" button both clears the selection and exits batch mode. API surface sync - Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml). - Update docs/api/knowledge.md with the new query parameters. - Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the Go SDK; the existing ListKnowledge keeps its signature. i18n - New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing menu.batchManage / batchManage.cancel for the multi-select strings. |
||
|
wizardchen
|
d199628701 |
fix(agent): exclude wiki-only KBs from quick-answer (RAG) mode
Quick-answer agent mode retrieves purely through vector/keyword chunk search and ships with no `allowed_tools`, so the existing capability filter (which only reads from `allowed_tools`) let wiki-only KBs through in every entry point. End result: users could @-mention, select, and receive suggested questions from wiki-only KBs in quick-answer mode, but the underlying retrieval always returned empty. Treat "RAG-only" as an implicit property of `agent_mode = quick-answer` and union it with the tool-derived filter. The same predicate is now used everywhere the user can pick or be steered toward a KB: Backend - `tools.DeriveKBFilterForAgent` / `KBSatisfiesAgentRequirements` layer the implicit quick-answer requirement on top of tool derivation. - `ListKnowledgeBases`, `SearchKnowledge` (shared-agent `@file`), `resolveKnowledgeBasesFromAgent` (chat runtime), `/search` IM command, and `GetSuggestedQuestions` now all use the agent-mode-aware variant. - `GetSuggestedQuestions` also skips the wiki-page fallback for quick-answer agents to cover the `selected` / explicit-kb-ids paths where a wiki-only KB could still slip through. Frontend - `deriveKbFilterForAgent` / `kbSatisfiesAgentRequirements` mirror the Go helpers. - `@` mention dropdown (`Input-field.vue`) uses the new helper. - Agent editor's "specified KB" picker (`AgentEditorModal.vue`) grays out wiki-only KBs for quick-answer agents with a tooltip, and the pre-save warning fires for quick-answer mode too. - i18n: add `agentEditor.agentType.kbMismatch.quickAnswer` across all four locales. |
||
|
nullkey
|
c753c67608 |
docs(api): add and fix swag annotations on handlers
将 swag 注解覆盖率从 ~88% 提升到 100%，并修复审计中发现的若干既有
注解 bug。
新增注解（24 个）：
- im.go: 4 (UpdateIMChannel/DeleteIMChannel/ToggleIMChannel/IMCallback)
其中 IMCallback 同时注册 GET 和 POST，两个方法都加了 @Router 声明。
- web_search.go: 1 (GetProviders)
- web_search_provider.go: 6
- wechat_qrcode.go: 2
- weknoracloud.go: 2
- knowledge.go: 2 (MoveKnowledge/GetKnowledgeMoveProgress)
- knowledgebase.go: 1 (ListMoveTargets)
- organization.go: 3 (ListSharedAgents/ListOrgAgentShares/RemoveAgentShare)
- mcp_service.go: 2 (SetMCPToolApproval/ResolveToolApproval)
- chunker_debug.go: 1 (PreviewChunking)
修复既有注解 bug（12 处）：
- initialization.go: 10 个 @Router 路径与真实路由不符
(e.g. /initialization/kb/{kbId}/config → /initialization/config/{kbId}
/initialization/fabri/tag GET → /initialization/extract/fabri-tag POST)
- session/stream.go: ContinueStream 路径
/sessions/{session_id}/continue → /sessions/continue-stream/{session_id}
- mcp_service.go: ResolveToolApproval 请求体 {approve: bool}
→ {decision: "approve"|"reject", reason?, modified_args?}
收敛响应 schema 到已有命名 struct（之前用 map[string]interface{}）：
- knowledge.go MoveKnowledge → handler.MoveKnowledgeResponse
- knowledge.go GetKnowledgeMoveProgress → types.KnowledgeMoveProgress
- web_search_provider.go GetProvider/UpdateProvider → types.WebSearchProviderEntity
- initialization.go InitializeByKB → handler.InitializationRequest
- initialization.go 三个 model-test endpoint → handler.ModelTestRequest
修正 organization.go 中 errors.AppError 应使用 apperrors 别名（该文件
stdlib errors 未别名化）。
不补 /files 和 /api/v1/files/presigned 内联闭包路由（文件服务，非业务 REST）。
Refs #890 #1049 #1168
|
||
|
wizardchen
|
1f5970b67e |
refactor(agent): rebuild multi-turn history from DB, drop llmcontext layer
The Agent ran a parallel context cache (Redis/in-memory) on top of the messages table to feed multi-turn history into the LLM. That dual-write caused subtle drift (e.g. compression diverging from DB, system-prompt swaps lost on restart) and required a separate ClearContext path on session/IM clear. Make the messages table the single source of truth: - Add service.LoadAgentHistory: rebuilds chat.Message history per turn from the persisted messages, expanding AgentSteps into proper OpenAI assistant_with_tool_calls + tool messages and replaying the canonical final answer (with <think> blocks stripped). final_answer tool calls are filtered to avoid duplicating the trailing answer. - Make AgentEngine stateless across turns: drop ContextManager / sessionID cache plumbing from the engine, agent service, and CreateAgentEngine signature. The engine only uses sessionID for logging/event emission. - Wire AgentQA to load history from DB on demand using HistoryTurns (default 5) when MultiTurnEnabled, otherwise run with empty history. - Delete the llmcontext package (ContextManager interface, Redis/memory storage, factory) and the SessionService.ClearContext API path; IM /clear and session message clear no longer need to invalidate cache. Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with empty AgentSteps just produce the single canonical assistant message), and Agent-mode turns now consistently see prior tool calls and results. |
||
|
wizardchen
|
7c0964bb95 |
fix(agent): enhance user authorization and concurrency handling in tool approval
- Introduced a new test to ensure that empty user IDs are rejected when a waiter has a user, preventing unauthorized approvals. - Added RequestNonce to resolve messages to uniquely identify Resolve calls, ensuring concurrent requests do not interfere with each other. - Updated waiter structure to use atomic operations for the resolved state, improving thread safety. - Enhanced error handling in the Resolve method to properly manage user mismatches and ensure accurate acknowledgment delivery across instances. - Updated MCP service handler to reject unauthenticated requests early, providing clearer feedback on authorization requirements. |
||
|
wizardchen
|
6d0f09d75f |
fix(agent): 加固 MCP 工具人审批的跨实例与并发语义
- 跨实例 Resolve 通过 Redis reply channel 接收 owning 实例的 ack （ok / not_found / tenant_mismatch / user_mismatch / already_resolved）， HTTP 不再静默 200，超时退化为 NotFound。 - PendingRequest 增加 UserID，Resolve 校验会话所有者，避免同租户 其他用户越权批准/篡改参数。 - 新增 ErrAlreadyResolved，timer/ctx 命中后再次 Resolve 返回 400 而 不是静默成功；waiter.deliver 暴露胜出语义。 - NeedsApproval 默认 fail-close（DB 故障要求审批），可经 WEKNORA_AGENT_TOOL_APPROVAL_FAIL_OPEN=true 回退。 - Pub/Sub 增加 OriginID 自过滤、退避重连、namespace 后缀，避免自回 环噪声与多部署串扰。 - 工具执行超时常量统一通过 ToolExecContext.ExecTimeout 注入， mcp_tool.go 不再硬编码 60s。 - handler 修正 path 二次解码、错误用 errors.Is、modified_args=null 的 静默清空、ListMCPToolApprovals 的 404/500 误判。 - repository 用 clause.OnConflict 做原子 upsert，消除并发 500。 - 增加 NotFound / TenantMismatch / UserMismatch / AlreadyResolved / Race 等用例。 |
||
|
wizardchen
|
5510ea8f5a |
feat(agent): human-in-the-loop approval for MCP tool calls (#1173)
Add an opt-in human approval gate so Agent runs pause before executing MCP tools that operators flag as dangerous, surface an approval card in the chat UI, and only resume after the user approves (optionally with edited args) or rejects. Backend - New mcp_tool_approvals table + repo/service to mark per-tool approval required (PG migration 000042 + sqlite init). - approval.Gate coordinates RequestAndWait / Resolve with sync.Once delivery, configurable timeout, and Redis Pub/Sub fan-out so multi- replica deployments work without sticky sessions. - MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx (without the per-tool 60s timeout) for the wait, and re-derives a fresh 60s exec ctx after approval so CallTool keeps a full window. - New SSE response types (tool_approval_required / _resolved) and EventBus events plumb approval state to AgentStreamDisplay. - REST: list/set per-tool approval flag, resolve pending approval. - Configurable via agent.tool_approval_timeout_seconds (yaml) or WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go duration). Frontend - MCP settings: per-tool "require approval" switch on the test panel. - Chat: ToolApprovalCard renders the pause point with editable JSON args, validation feedback, mm:ss countdown that turns warning/danger near deadline, and a resolved state that retains context. - i18n strings added for zh-CN / en-US / ko-KR / ru-RU. Docs - docs/zh/mcp-approval.md covering behavior, config, API, deployment considerations (Redis cross-instance, restart limitations). |
||
|
wizardchen
|
d0144f3586 |
perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
wizardchen
|
df1e9a1bfd |
perf(wiki): subgraph API and interactive exploration for large graphs
Wikis with tens of thousands of pages used to crash the browser trying to render the entire link graph at once (30MB+ JSON, 100k+ SVG elements). This change moves the graph viewer from "fetch everything, render everything" to "fetch a slice, expand on demand" with several matching UX improvements. Backend * GET /wiki/graph now accepts `mode` (overview | ego), `center`, `depth`, `types`, `limit` query params. Default is overview top-500 by link_count, capped at 2000. Response includes a Meta object so the frontend can render a truncation hint and drive the UI. * Pure helper `computeGraphSubset` extracted for testability; six unit tests cover overview truncation / type filter / ego BFS / missing center error. * WikiLintService passes Limit=0 (uncapped) so link integrity checks still walk every page. * Repo `Search` adds CASE-based relevance ranking (title 4, slug 3, summary 2, content 1) so full-text results put the most obvious matches first instead of whatever was updated most recently. Frontend * Graph viewer fetches overview on entry; ego pivot on double-click, search, wiki-link click, URL ?slug=, global issues jump. * Shift+click or hover ⊕ button blooms neighbors onto the current canvas additively. Bloom tracks generations and LRU-evicts oldest when total exceeds 1500 nodes; ego center / selected node / latest anchor are always protected. * "Grow frontier (N)" legend action expands every dashed-ring node in one click (6-way concurrency) while skipping Index/Log super-nodes that would otherwise dump the whole wiki onto the canvas. * Node dashed expansion ring + drawer "X/Y neighbors shown" hint tell users which nodes still have neighbors to load. * Search dropdown uses remote full-text search (debounced, sequence- numbered to drop stale responses) and falls back to the overview top-500 snapshot when the input is empty. * Type filters now round-trip to the server so top-N is always computed from the user's active type set rather than hiding nodes client-side and shrinking the view. * Status card replaces the cramped "centered on X · N hops · M nodes" line with a structured focus/overview summary, resolving slugs to page titles. * Help popover in the legend lists every canvas shortcut. i18n entries added for zh-CN / en-US / ko-KR / ru-RU. |
||
|
Claude
|
8388a50c23 |
fix(chunker): post-review fixes for preview endpoint robustness
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.
Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
not accept a context.Context, so when previewTimeout fires the
worker keeps running. Bounding input size keeps worst-case CPU
per request well under a second on commodity hardware. Sized so
10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
follow-up: real cancellation needs the splitter to take a ctx.
Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
reuse a single rune-count per chunk for stats + size + token
estimation. Eliminates the previous triple []rune allocation per
chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
instead of []chunker.Chunk; sumSq computed in float64 to avoid
the int*int overflow at l > ~46k.
Correctness / UX
- Preview panel sends strategy / token_limit / languages
unconditionally, mirroring the buildSubmitData convention so the
preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
gin's cryptic 'Field validation failed on the required tag'.
Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
made the original name a misnomer; the test still asserts the
right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
cases (empty / single / varying / no-variance underflow) plus
PreviewChunking httptest scenarios (auto path, legacy strategy,
empty-text rejection, oversize rejection, chunk truncation with
full-set stats).
Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
obsolete since the heading and heuristic splitters shipped.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
|
||
|
Claude
|
beb7d005e9 |
fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness
Resolves the issues surfaced by the post-Phase-3 self-review. High severity - Strategy / TokenLimit / Languages can now be reset. The KB-config handler uses pointer DTOs so the absence of a field in the payload means "no change" while an explicit empty string / 0 / [] clears the value. The frontend now sends the fields unconditionally so the user can revert to defaults via the clear icon — previously these fields were write-once. - DocProfile is no longer computed twice in the preview endpoint. Diagnostics carries the profile that drove tier selection (when auto-strategy ran); the handler only re-profiles for explicit strategies that bypass the profiler. - Chunk-size stats (avg / min / max / stddev) are now computed over the FULL chunk set before the response is truncated to previewMaxChunks. Avoids systematically misleading metrics for large documents. - SplitWithDiagnostics defaults SelectedTier to TierLegacy so an empty-text request never leaves the diag with a blank tier string. Medium severity - Split() reverts to its own loop instead of delegating to SplitWithDiagnostics. Diagnostics struct allocation is reserved for callers that actually want the trace (preview endpoint). Matters in SplitParentChild where Split runs once per parent. - Preview endpoint runs the splitter on a goroutine wrapped in a 5s context.WithTimeout. CJK input at the 256k-rune ceiling could otherwise tie up the worker for several seconds. Low severity - KBChunkingSettings emits arrays via spread copies so parent-side mutation cannot drift the form's reactive state. - KBChunkingDebug font-family includes a monospace fallback so the chunk content stays monospaced even when the TDesign CSS variable is missing. Documentation - DocProfile JSON shape is now annotated as part of the public preview API; field renames would be breaking. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k |