Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.
Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.
Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:
* auto (default/recommended): probe Doris server on first use to detect available vector
functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
(older builds lacking inner_product_approximate)
* legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
builds without inner_product_approximate support)
* inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
(for modern Doris 4.0+ and current SelectDB with normalized embeddings)
Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level
Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
Sessions now record the input-bar state used for the most recent QA
request (agent, model, KB scope, web search). The chat UI hydrates
those settings on session reopen so users see the same configuration
they used last time, instead of the global default.
The state is stored in the existing sessions.agent_config JSONB column
to avoid a new migration. Frontend snapshots the user's global defaults
on session enter and restores them on session leave, so opening an old
session does not pollute new-chat defaults.
Refactored the pinning mechanism for knowledge bases to be user-specific, allowing any user with read access to pin knowledge bases independently. This change includes the introduction of a new `user_kb_pins` table to manage pin states, replacing the previous tenant-wide pinning system. Updated relevant API endpoints and service logic to support this new functionality, ensuring that the UI reflects the user's personal pin status.
Additionally, adjusted the knowledge base listing to sort by user-specific pin status, enhancing the user experience by prioritizing pinned items.
Refs: #1303
Added functionality for managing pending invitations, including a polling mechanism to update the invitation count in real-time. Introduced a new dialog component for users to view and respond to their invitations without navigating away from their current context. Updated API endpoints for fetching and managing invitations, and enhanced the user interface with relevant internationalization support.
Refs: #1303
Introduced a new API client for managing user favorites, allowing users to list, add, and remove starred resources. Enhanced the sidebar to display favorites and recents, improving user navigation. Updated internationalization files to include relevant labels and tooltips for the new features across multiple languages.
Refs: #1303
The audit_logs table grows monotonically until something purges it.
PR 6 explicitly listed retention as out of scope for v1; this picks
that up.
Adds an `audit.retention_days` config (default 90, 0 disables) and a
small background goroutine `AuditLogRetentionRunner` that fires once
~10 minutes after boot and then every 24h, calling
`AuditLogService.Purge` which DELETEs rows older than the cutoff in a
single statement. The dependency surface is intentionally minimal —
no robfig/cron, no asynq, no migration — because retention has no
wall-clock alignment requirement.
Wiring is the same shape as the existing data source scheduler:
container `Provide(NewAuditLogRetentionRunner)`, an `Invoke` that
calls `Start`, and a `ResourceCleaner.RegisterWithName` for graceful
shutdown.
Defaults preserve operator intent:
- `audit:` section omitted from YAML -> retention_days=90.
- explicit `audit.retention_days: 0` in YAML -> purge disabled.
- `WEKNORA_AUDIT_RETENTION_DAYS=N` env override (incl. N=0).
- `retention_days < 0` is rejected by ValidateConfig.
Tests cover the service Purge contract (no-op when disabled, cutoff
math, error propagation) and the runner lifecycle (Start no-op when
disabled, idempotent Start/Stop, ticker cadence, runOnce swallows
errors). 10 new tests, all green.
docs/rbac.md updated with the new YAML / env / behaviour.
Refs: #1303
Before this change, the `organization_members` table keyed on user_id —
joining an org as a Viewer in tenant T meant only that one user got
visibility on shared KBs/agents, while the rest of T (including its
Owner) saw nothing. The mismatch surfaced as confusing UX bugs:
"my colleague shared this with us, why can't I see it?"
This change replaces `organization_members` with
`organization_tenant_members`. Membership and Org-RBAC role now key on
(org_id, tenant_id); `representative_user_id` stays as a display-only
label of the user who first brought the tenant in.
Permission resolution gains a 3-D cap so tenant-RBAC stays honest under
cross-tenant sharing:
effective = min(share.Permission, tenant_org_role, tenant_role_cap)
where tenant_role_cap pins tenant Viewers to OrgRoleViewer regardless
of the org-level grant. A Viewer in their own tenant cannot edit a
shared KB even if the org granted Editor — preserves the
"Viewer = read-only everywhere" promise.
Migration 000045 backfills `organization_tenant_members` from the old
table by collapsing each (org, tenant) group to its max role, then
renames the old table to `organization_members_pre_plan3` for safe
rollback. Sqlite mirror (Lite mode) is fresh-bootstrap: the init
schema replaces the old table directly.
Routes: member-management URLs change from `:user_id` to `:tenant_id`.
Org create / join / leave now require Admin+ in the caller's tenant —
joining an Org affects everyone in T, so the decision can't sit with
a Viewer.
Frontend: OrganizationSettingsModal + store update to address members
by tenant_id.
PR 1-5 of the multi-tenant RBAC series enforce the role matrix; what
they don't do is keep a durable record of who did what. Today the
RBAC middleware logs "[rbac] role insufficient ..." to stderr -
perishable, not searchable. Operators cannot answer "who removed
Alice from tenant T-foo on Tuesday" or "did we 403 a probing user
200 times last hour" after the fact.
This PR adds a generic per-tenant audit log that:
- Captures mutating RBAC actions (member add/remove/role-change/leave)
durably from the tenant_member service after the repo write succeeds.
- Captures enforcement rejections (RequireRole / RequireOwnershipOrRole
denies under EnableRBAC=true) via a gin-context-injected hook.
- Sliding-window dedup (1 minute) on (tenant, actor, action, path)
for denied events so a probing client cannot flood audit_logs.
- Default-on - no extra env flag. Schema is generic enough for future
KB / agent / datasource events to reuse via new action namespaces
without another migration.
Schema (migration 000044) mirrors wiki_log_entries' shape: a single
audit_logs table with action, target_type/id, request path/method,
outcome, and a JSONB details column. Three indexes: (tenant_id, id
DESC) for the per-tenant cursor feed, actor_user_id for "what did
Alice do", (tenant_id, action) for both the action filter and the
dedup lookup. Sqlite mirror added for Lite mode.
Wiring is nil-safe: the audit field on tenantMemberService and the
gin-context lookup in rbac.go both tolerate a nil service so existing
tests construct without a stub and dormant deployments degrade
gracefully if the provider is misconfigured.
The query API is GET /tenants/:id/audit-log, gated by Admin+ on top
of the existing PathTenantMatch (denial histories should not surface
to ordinary members). Cursor-paginated by descending id (sidesteps
the duplicate-timestamp tie-breaking that BIGSERIAL trivially solves).
Frontend: TenantMembers.vue gets a t-tabs wrapper with Members
(existing) and Audit log (new) tabs; the audit tab is gated to
Admin+ to mirror the server, lazy-loads on first open, paginates via
the next_cursor pattern, and renders coloured chips per
action/outcome so an operator can scan for anomalies. i18n keys
added in all four locales.
Tests: 14 new tests across service (6), middleware (4), and handler
(4). Service tests pin the dedup window behaviour (writes again
after expiry, dedup is per-actor-per-path, degrades on lookup
error). Middleware tests pin both reject sites' audit hook and the
"dormant mode does not fire" + "nil audit does not panic"
invariants. Handler tests pin the cursor envelope and that an empty
page returns next_cursor=0.
Out of scope: per-action ACL on the audit log, retention/TTL job,
KB/agent/datasource event wiring (schema supports it - just no
emitters yet), async writes, frontend filter UI.
Security
- Reject URL :id when it does not match the caller's active tenant on
every /tenants/:id/members/* and /tenants/:id/leave endpoint. Without
this an Owner of tenant A could drive operations on tenant B simply
by changing the URL, because RequireRole resolves the caller's role
against the active tenant from context, not the URL param. The
cross-tenant superuser carve-out mirrors middleware/rbac.go
(CanAccessAllTenants + EnableCrossTenantAccess required).
- AddMember no longer records the synthetic "system-<tenantID>" caller
(X-API-Key auth path) as invited_by, matching the same filter
KB.CreatorID got in PR 2.
- AuthHandler now logs a loud one-shot error when constructed with a
nil/incomplete config so a wiring bug that silently disables the
invite_only gate is visible at boot.
Race fixes
- New repo helpers DemoteOwnerAtomically / RemoveOwnerAtomically wrap
the count-other-owners + write step in a transaction that holds
SELECT … FOR UPDATE locks on every other active Owner row. The
service routes Owner demotion / Owner removal through them, closing
the TOCTOU window where two concurrent demotes of two different
Owners could each observe count=2 and both commit, leaving the
tenant ownerless.
Performance
- ListMembers hydration now batches user lookups via a new
GetUsersByIDs interface method (one SELECT … WHERE id IN (…)
instead of N round-trips). Failure degrades to "no email/username"
rather than dropping rows.
Frontend
- canManage also fires for cross-tenant superusers
(authStore.canAccessAllTenants), mirroring the backend carve-out so
the UI controls match what the server actually allows.
Tests
- Cross-tenant URL guard regression cases for List / Add / Update /
Remove / Leave.
- Superuser bypass tests (flag on -> service reached, flag off -> 403).
- Batched lookup test ensures GetUsersByIDs is called exactly once and
GetUserByID is never used as a fallback.
- Synthetic caller invited_by filter regression test.
- Service-layer atomic last-Owner regression tests for both
UpdateRole and RemoveMember.
Tighten correctness, security and observability of the tenant RBAC
foundation introduced in 3c4cdd70 in response to internal review.
Bugfixes
- custom_agents.runnable_by_viewer: drop the GORM `default:true` tag.
GORM treats a plain bool's zero value as "unspecified" and silently
rewrites it to the column default on insert, which would prevent
admins from ever creating an agent with viewer-runtime disabled. The
column-level DEFAULT TRUE on the database side still backstops rows
inserted by raw SQL / migrations.
- TenantMemberRepository.HasAnyMembers: replace `Limit(1).Count(&n)`
with `Take(&probe)`. GORM's Count ignores Limit and scans the whole
table, which matters because this runs on the auth middleware's
hot path for users without a cached membership.
- service/user.LoginWithOIDC: include User, Tenant and Memberships in
the OIDCCallbackResponse so the OIDC flow returns the same shape as
password login (previously User and friends were left nil).
Security
- middleware/auth.resolveTenantRole: restrict orphan-tenant
auto-promotion to the caller's home tenant. Cross-tenant superuser
visits and JWTs minted for foreign tenants no longer trigger silent
Owner grants. Two regression tests pin the new behaviour.
- service/user.synthFallbackMembership: default the synthesised role
to Viewer instead of Owner. The login response only feeds UI; the
backend re-derives the real role from tenant_members on every
request, so showing a Viewer UI is preferable to a misleading Owner
UI that would surface admin controls the backend will then 403.
- migrations/versioned/000043_tenant_rbac.down.sql: gate the destructive
drop behind a `weknora.allow_destructive_migration` GUC so an
accidental `migrate down` cannot wipe role data in production.
- frontend/src/stores/auth.ts: comment currentTenantRole as
rendering-only so future PRs cannot mistake the localStorage-backed
value for an authorisation signal.
API contract
- Memberships in LoginResponse and OIDCCallbackResponse now serialises
without omitempty, and buildMembershipsForUser / synthFallbackMembership
always return a non-nil slice — frontend can rely on JSON `[]`.
- handler/auth.GetCurrentUser includes memberships so /auth/me refreshes
recover currentTenantRole after a page reload.
Performance
- Add TenantService.GetTenantsByIDs (with the matching repo method) and
use it inside buildMembershipsForUser to batch tenant-name lookups
instead of issuing one query per membership.
Observability
- Replace the remaining `log.Printf` calls in middleware/auth with
contextual logger.Warnf / logger.Infof so role resolution events ride
the standard request-id / Langfuse trace plumbing. The successful
auto-promote line is tagged `[audit]` for log mining.
Tests
- internal/application/service/tenant_member_test.go: hand-rolled
fakeTenantMemberRepo plus 11 cases covering AddMember validation,
EnsureOwner idempotence, "cannot demote/remove the last Owner",
same-role no-op, ErrInvalidTenantRole / ErrMembershipNotFound, and
the TenantRole.HasPermission lattice.
- internal/middleware/auth_role_test.go: fakeMemberService plus 8 cases
pinning every branch of resolveTenantRole, including the two H1
regressions (cross-tenant superuser must not auto-promote; orphan
auto-promote requires the home tenant).
Lay the foundation for tenant-level RBAC without enforcing it yet.
Schema, types, middleware lookup, and login-flow shape changes ship now;
RequireRole/route annotations and member-management endpoints follow in
PR 2 and PR 3.
Highlights:
- New tenant_members table (Postgres + SQLite) with (user_id, tenant_id)
unique key and (owner|admin|contributor|viewer) role values. Migration
000043 backfills one row per existing user, marking the earliest-created
active user in each tenant as Owner and the rest as Contributor.
- KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so
PR 2 can wire ownership-based and Viewer-runtime checks.
- Auth middleware loads the active TenantMember row, sets a new
TenantRoleContextKey, and falls back via three escape hatches:
cross-tenant superusers (CanAccessAllTenants) get Admin in the target
tenant; orphan tenants (zero members) auto-promote the first authenticating
human to Owner; and when tenant.enable_rbac is false (the v1 default)
unmatched lookups fail open with Admin to preserve current behaviour.
- Register / OIDC provisioning now insert an Owner membership for the
registrant via TenantMemberService.EnsureOwner.
- LoginResponse renamed tenant -> active_tenant and gained a memberships
array. POST /auth/switch-tenant issues a fresh token pair scoped to a
target membership; GET /auth/config exposes auth.registration_mode for
the frontend register-tab gating.
- New config keys: auth.registration_mode (self_serve | invite_only,
defaults to self_serve) and tenant.enable_rbac (defaults to false).
Both have WEKNORA_* env overrides.
- Frontend store persists memberships and exposes currentTenantRole;
Login.vue calls /auth/config and hides the Register tab in invite_only
mode. Go SDK types updated to match the new LoginResponse shape.
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory
into the user-facing knowledge-base lifecycle:
- POST /knowledge-bases validates the requested vector_store_id against
the caller's tenant scope and the engine registry. New error codes
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201)
distinguish the typed branches without echoing UUIDs to the client.
- GET / POST / PUT / PUT-pin responses embed the bound store's display
metadata (name, source, engine_type, status) without exposing any
connection credentials. Cross-tenant shared KBs receive a suppressed
payload (vector_store_id stripped, source="shared") so operator-chosen
store names cannot be enumerated across tenants.
- POST /knowledge-bases/copy synchronously rejects clones whose target
has a different embedding model or vector store, before the async
clone task is enqueued. The async clone worker re-applies the same
checks for defense in depth.
- DELETE /vector-stores/:id refuses to remove a store with bound KBs,
inside a transaction that row-locks the store on PostgreSQL and
serializes via WAL on SQLite. unregister-from-registry is wrapped in
defer/recover so a panic surfaces as a structured warning instead of
silently leaking a stale engine.
- vector_store_id is immutable after creation. The GORM <-:create tag
blocks every ORM update path; the service-layer DTO omits the field
entirely; a reflection-based regression test catches any future
maintainer who adds it back to either layer.
- Empty-string vector_store_id is normalized to nil at both the create
path and inside SharesStoreWith, so rows persisted by callers that
did not run Normalize first cannot trip false same-store comparisons.
Part of #993. Depends on #994 and #1310.
The in-memory SQLite DDL in `knowledgebase_sqlite_test.go` mirrors the
`knowledge_bases` columns the tests need to insert against. It missed two
columns that the `KnowledgeBase` struct has been carrying for a while:
- `wiki_config` (gorm:"column:wiki_config;type:json")
- `indexing_strategy` (gorm:"column:indexing_strategy;type:json")
Without those columns the `INSERT INTO knowledge_bases (...)` statement
GORM emits fails with:
table knowledge_bases has no column named wiki_config
…which cascades to every TestKnowledgeBase_VectorStoreID_* test
failing on a fresh `go test ./internal/application/repository/`.
Reproduce on main (before this commit):
go test ./internal/application/repository/ -count=1
...
--- FAIL: TestKnowledgeBase_VectorStoreID_Save_Immutable (0.00s)
knowledgebase_sqlite_test.go:91:
Received unexpected error:
table knowledge_bases has no column named wiki_config
FAIL github.com/Tencent/WeKnora/internal/application/repository
After this commit:
ok github.com/Tencent/WeKnora/internal/application/repository
The fix is two `TEXT NULL` column additions to the inline DDL. No
production code, no migration, no schema change — strictly aligning the
test scaffold with the model fields that already exist.
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
`metadata->'similar_questions'::text` parses as
`metadata -> ('similar_questions'::text)` because `::` binds tighter
than `->`, so the expression yields jsonb instead of text. ILIKE on
jsonb then fails with "operator does not exist: jsonb ~~* unknown",
returning 1007 Internal server error for `search_field=similar_questions`
and `search_field=answers`. The `standard_question` branch was unaffected
because it uses `->>` (text) directly.
Wrap the json access in parens so the cast applies to the extracted
value: `(metadata->'similar_questions')::text ILIKE ?`. The MySQL
branch uses JSON_EXTRACT and is unchanged.
Fixes#1264
- Updated `pydantic` from 2.12.3 to 2.13.4 and `pypdfium2` from 5.0.0 to 5.8.0 in the `docreader` requirements.
- Modified the dependabot configuration to set `open-pull-requests-limit` to 0 and added an `ignore` rule for version updates across all ecosystems, allowing only security updates.
- Adjusted settings for `server-security`, `client-security`, `frontend-security`, and `miniprogram-security` groups to streamline security update handling.
This change aims to enhance dependency management and maintain security while reducing noise from version update PRs.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
items" had two args), and harden validMerge against un-prefixed
slugs whose strings.Index returned -1 and silently passed the type
check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
letter row instead of a hard-coded 0; tighten payloadProbe to the
set of field names with consistent semantics across payloads
(drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
outside an asynq worker ctx so the semantics stay pinned.
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wikis with tens of thousands of pages used to crash the browser trying
to render the entire link graph at once (30MB+ JSON, 100k+ SVG elements).
This change moves the graph viewer from "fetch everything, render
everything" to "fetch a slice, expand on demand" with several matching
UX improvements.
Backend
* GET /wiki/graph now accepts `mode` (overview | ego), `center`, `depth`,
`types`, `limit` query params. Default is overview top-500 by
link_count, capped at 2000. Response includes a Meta object so the
frontend can render a truncation hint and drive the UI.
* Pure helper `computeGraphSubset` extracted for testability; six unit
tests cover overview truncation / type filter / ego BFS / missing
center error.
* WikiLintService passes Limit=0 (uncapped) so link integrity checks
still walk every page.
* Repo `Search` adds CASE-based relevance ranking (title 4, slug 3,
summary 2, content 1) so full-text results put the most obvious
matches first instead of whatever was updated most recently.
Frontend
* Graph viewer fetches overview on entry; ego pivot on double-click,
search, wiki-link click, URL ?slug=, global issues jump.
* Shift+click or hover ⊕ button blooms neighbors onto the current
canvas additively. Bloom tracks generations and LRU-evicts oldest
when total exceeds 1500 nodes; ego center / selected node / latest
anchor are always protected.
* "Grow frontier (N)" legend action expands every dashed-ring node in
one click (6-way concurrency) while skipping Index/Log super-nodes
that would otherwise dump the whole wiki onto the canvas.
* Node dashed expansion ring + drawer "X/Y neighbors shown" hint tell
users which nodes still have neighbors to load.
* Search dropdown uses remote full-text search (debounced, sequence-
numbered to drop stale responses) and falls back to the overview
top-500 snapshot when the input is empty.
* Type filters now round-trip to the server so top-N is always
computed from the user's active type set rather than hiding nodes
client-side and shrinking the view.
* Status card replaces the cramped "centered on X · N hops · M nodes"
line with a structured focus/overview summary, resolving slugs to
page titles.
* Help popover in the legend lists every canvas shortcut.
i18n entries added for zh-CN / en-US / ko-KR / ru-RU.
- Apply the session-list additions to the SQLite init migration (Lite build)
and make QueryPaged dialect-aware: LOWER(..) LIKE on SQLite, ILIKE on
Postgres, drop NULLS LAST on SQLite. Escape LIKE wildcards in the keyword
via the existing escapeLikeKeyword helper.
- SetPinned now returns rowsAffected so the handler can respond 404 on
unknown / unauthorized session IDs instead of a misleading 200.
- GET /sessions always returns the enriched shape (pin state + IM origin
fields) so the frontend never needs a second roundtrip; the dual-path
legacy branch in the handler is gone.
- Register pin routes with the wildcard name that matches each verb's
existing radix tree (POST :session_id, DELETE :id) and accept either
param name in the handler; avoids gin's "wildcard conflicts" panic.
- Drop the redundant [platform] prefix from IM session titles now that
the list renders a platform icon alongside the title; add unit tests
for shortID / buildUserSessionTitle / buildThreadSessionTitle.
- Frontend: remove the submenu search input and its i18n/keyword wiring
(search lives elsewhere in the app); pin icon uses the TDesign `pin`
glyph and inherits color so the active session turns green; optimistic
pin moves the item to the top of the list so it shows up at the top
of the Pinned group; IM text badge replaced by the platform SVG icons
from assets/img/im, desaturated by default and full-color on hover
or when the session is active.
Sessions today are a flat list ordered by updated_at. Two gaps showed up in
practice:
- Users cannot find specific chats as the list grows beyond a screen.
- IM-created sessions (WeCom/Feishu/Slack/...) are indistinguishable:
every title was "IM-<platform>" or "IM-<platform>-<username>" and the
list API hid the underlying im_channel_sessions mapping, so admins had
no way to tell which Feishu group a session came from.
Backend
- Migration 000039 adds sessions.user_id (owner), is_pinned, pinned_at
plus a composite index for the list query. Existing rows keep user_id
NULL and stay visible at the tenant level for backward compatibility.
- CreateSession now writes the caller's user_id from auth context.
- GET /sessions accepts keyword / source / agent_id. When any filter is
set, the response switches to enriched items that LEFT JOIN
im_channel_sessions and expose im_platform / im_chat_id / im_thread_id
/ im_user_id / im_agent_id / im_channel_id. No filters => legacy shape,
existing clients unaffected.
- Ordering: is_pinned DESC, pinned_at DESC NULLS LAST, updated_at DESC.
- POST/DELETE /sessions/:id/pin for user-scoped pin/unpin.
- IM session titles: "[platform] <user|chat|thread>" with short ID
suffixes so group/DM/thread sessions are visually distinct without
needing a round-trip to fetch a display name from the IM adapter.
Frontend
- Search input debounced at 300ms drives the keyword filter.
- Pinned chats render in a dedicated group above the time-based groups,
with a pin icon and a pin/unpin entry in the per-chat dropdown.
- IM chats get a short [platform] badge in the list.
- Pin toggle is optimistic and guards against double-clicks.
- zh/en/ko/ru i18n keys added for the new strings.
- Introduced UpdateAutoLinkedContent method to persist changes from machine-only link decorators without incrementing the version number, ensuring user-facing revisions remain consistent.
- Updated existing methods to utilize the new auto-linked content update logic, enhancing the handling of cross-link injection and dead-link cleanup.
- Enhanced the KnowledgeBase and WikiBrowser components to improve user experience during wiki operations, including status tracking and indexing indicators.
- Added scheduling for wiki status probes to provide timely feedback on indexing processes after user-triggered actions.
- Enhanced the UpdatePage and UpdateMeta methods to differentiate between user-visible content changes and bookkeeping updates, ensuring the version number is only incremented for actual content modifications.
- Updated the WikiPage struct and associated interfaces to reflect the new versioning policy, improving clarity on when the version is modified.
- Improved documentation for methods to clarify their intended use and behavior regarding versioning and metadata updates.
- Updated the SuggestedQuestion interface to include 'wiki' as a new source type, allowing for more diverse question origins.
- Implemented ListRecentForSuggestions method in the wikiPageRepository to retrieve recent wiki pages for fallback suggestions, improving the agent's ability to generate relevant questions when other sources are unavailable.
- Enhanced the customAgentService to utilize the new wiki page suggestions, ensuring a broader range of questions can be generated from wiki content.
These changes significantly improve the agent's suggestion capabilities by incorporating wiki content, enhancing user interaction and information retrieval.
- Introduced `encodeSlugPath` function to safely encode wiki slugs for API requests, preserving hierarchical routing.
- Updated API functions to utilize the new slug encoding for `getWikiPage`, `updateWikiPage`, and `deleteWikiPage`.
- Enhanced `ListBySourceRef` method in the repository to safely handle source knowledge IDs with JSON marshalling, improving query security.
- Added `escapeLikePattern` function to ensure safe concatenation of LIKE patterns in SQL queries, preventing unintended matches.
- Updated `KnowledgePostProcessService` to include Redis client for task management, improving performance in handling wiki ingest tasks.
- Enhanced `WikiLintIssue` structure to include `TargetSlug` for better issue tracking and auto-fixing capabilities.
These changes significantly improve the robustness and security of the wiki management system, enhancing user experience and data integrity.
- Added a new tool, `wiki_flag_issue`, enabling users to report factual errors, mixed entities, or outdated information on wiki pages.
- Updated the agent system prompt and tool definitions to include the new tool, enhancing the agent's capabilities for maintaining wiki accuracy.
- Implemented backend functionality for creating, listing, and updating the status of flagged issues, ensuring effective tracking and resolution.
These changes significantly improve the agent's ability to manage and report issues within the wiki, fostering a more reliable knowledge base.
- Updated the agent system prompt to clarify the workflow for knowledge retrieval, emphasizing the use of `wiki_search` and `wiki_read_page` tools for specific queries and general overviews.
- Modified the `wikiReadPageTool` to support reading multiple wiki pages simultaneously, improving efficiency in fetching content.
- Enhanced the `wikiSearchTool` to utilize PostgreSQL POSIX regular expressions for more effective search queries, allowing for complex pattern matching.
- Adjusted the `wikiPageRepository` to replace `ILIKE` with regex matching for search queries, increasing the precision of search results.
These changes significantly improve the agent's ability to retrieve and synthesize information from the wiki, enhancing user interactions and response accuracy.
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.
These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
- Updated the search logic in the WikiPage repository to use ILIKE for case-insensitive matching on title, content, summary, and slug fields, improving search accuracy.
- Added a watcher in the WikiBrowser component to reload all pages when the search query is cleared, enhancing user experience by ensuring relevant content is displayed immediately.
These changes collectively improve the search capabilities and responsiveness of the wiki interface.
Implemented new features in the WikiBrowser component to allow users to toggle the visibility of arrows on graph edges. This includes:
1. **Localization Updates**: Added translations for "Show arrows" and "Hide arrows" in English, Korean, Russian, and Chinese.
2. **Graph Rendering Enhancements**: Introduced logic to dynamically update SVG line attributes based on the toggle state, improving visual clarity in graph representations.
3. **State Management**: Added reactive state management for arrow visibility, ensuring seamless user interaction.
These changes enhance the user experience by providing more control over graph visualizations.
Added functionality to enable updates for the `wiki_config` field in the `KnowledgeBaseConfig` struct. This includes:
1. **Type Definition Update**: Added `WikiConfig` field to `KnowledgeBaseConfig` to allow JSON unmarshaling during update requests.
2. **Service Layer Enhancement**: Updated the `UpdateKnowledgeBase` method to persist changes to `wiki_config` when provided in the request.
3. **Frontend TypeScript Adjustments**: Enhanced type definitions in the frontend to support `wiki_config`, improving IDE autocomplete and type checking.
This fix addresses the critical issue where `wiki_config` was not being updated due to its absence in the `KnowledgeBaseConfig` struct, ensuring proper handling of wiki configurations during knowledge base updates.
Added a new utility function, WithDeadlockRetry, to handle MySQL deadlock errors during chunk creation. This function retries the operation up to three times with exponential backoff. Updated the CreateChunks method to utilize this new function, ensuring robust handling of concurrent write scenarios and preventing potential deadlocks. Additionally, improved documentation for CreateChunks to clarify the use of Omit for avoiding auto-increment column issues.
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes:
**Backend Changes:**
- Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log)
- Database schema: wiki_pages table with full migration support
- WikiPageService: CRUD operations and page management
- WikiPageRepository: GORM-based persistence layer
- Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents
* Summary page generation using LLM
* Entity and concept extraction in a single LLM call
* Synthesis opportunity detection
* Index page rebuilding
* Log page maintenance
- Wiki boost feature: Enhance chat retrieval with wiki context
- Wiki linting: Maintenance and validation utilities
- Agent wiki tools: Enable agents to query and interact with wiki pages
- Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks
- Language support: Reuse existing middleware language infrastructure for LLM prompts
**Frontend Changes:**
- Wiki browser UI: View all wiki pages with filtering and search
- Wiki API client: Knowledge base wiki management endpoints
- Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model)
- i18n updates: Support for English, Korean, Russian, and Chinese interfaces
**Configuration:**
- Container DI: Wire up all wiki services
- Router: Register wiki API endpoints
- Task handling: Support async wiki ingest tasks
**Testing:**
- Unit tests for wiki page types
- Service layer tests
- Endpoint tests for wiki operations
- Integration tests with LLM mocking
**Documentation:**
- Language refactoring analysis and guides
- Implementation completion reports
- Quick reference guides for developers
**Key Features:**
✅ LLM-powered wiki page generation from documents
✅ Multi-language support (9+ languages)
✅ Automatic extraction of entities and concepts
✅ Synthesis opportunity detection
✅ Index and log page maintenance
✅ Progressive wiki building across multiple documents
✅ Agent-based wiki interaction
✅ Chat retrieval enhancement with wiki context
✅ Full frontend UI for wiki browsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Adjusted the expanded TopK limit to a maximum of 200 candidates to enhance HNSW efficiency.
- Updated the vector retrieval query to ensure the ORDER BY expression matches the indexed expression, improving index usage.
- Implemented local settings for HNSW's ef_search and iterative_scan within a transaction to optimize candidate retrieval.
- Added fallback logic to handle unsupported GUCs gracefully, ensuring query execution continues even with older pgvector versions.
These changes improve the performance and reliability of vector retrieval in the PostgreSQL implementation.