Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
- Added a LocalImageResolver function to resolve local storage URLs to their byte data using the tenant's storage configuration.
- Integrated the resolver into the container setup to support reading local images in multimodal chat scenarios.
- Enhanced the handling of local storage paths to correctly parse tenant IDs and retrieve files based on the configured storage settings.
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.
Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
buildEnvStoreForDriver — every gated surface now recognises
"opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
in initRetrieveEngineRegistry; NewEngineFactory now closes over the
AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
testOpenSearchConnection case so CreateStore's connectivity probe
accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
NewOpenSearchClient is exported so the factory and env path can build
the TLS-hardened client; healthcheck.go reuses the unexported
probeVersion / probeKNNPlugin for the service-layer probe.
Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.
Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
search → BatchSave with the source_id remap for generated questions —
so dim/keyword routing and the source_id contract come from BatchSave
for free. embeddingMap is keyed by the *target* SourceID because
OpenSearch's BatchSave looks up embeddings by SourceID
(lookupEmbedding), not by chunk_id (the ES driver's convention).
Pagination is from/size; copies larger than max_result_window
(default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
group the input by target value and issue one _update_by_query per
group over the cross-dim <base>_* pattern. Caller values flow through
bound script params only — never string-interpolated into the Painless
source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
full failure reason goes to the debug log only; the returned error
carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
shape as DeleteByQuery's quirk), so refresh=wait_for is not
expressible; we use refresh=true.
Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
functional option) lets the driver emit opensearch.index_created and
opensearch.reindex_executed events without importing any service
package — the service layer implements the interface. NewRepository
takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
When the context carries no tenant (the env-path registration ctx
during boot, for example) the adapter skips the emit with a warning
rather than silently writing tenant_id=0, which would collide with the
system-scope sentinel.
Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
immutable. VectorStoreSettings.vue is now schema-driven: a closed
`enum` renders a t-select; number inputs use the schema's `:min`/`:max`
and fall back to the legacy replica-vs-shard heuristic only when the
schema does not pin them; a danger-coloured warning fires when
insecure_skip_verify is toggled on (the switch and warning are wrapped
in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
knn_engine / insecure_skip_verify plus the warning copy in en-US,
ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
security plugin disabled for dev only). OpenSearch Dashboards lives in a
separate, opt-in opensearch-ui profile so the heavy UI container is not
forced up alongside the cluster (the driver e2e is fully curl-verifiable
against :9200). The new docs/dev/opensearch-integration-test.md covers the
end-to-end exercise and the single-node guidance (set replicas=0 to keep
the cluster Green).
Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
their positive counterparts in this PR. The test suite was the
activation checklist; the activation flip is its diff.
Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
other engines' serialised config is byte-identical. Existing
Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
stores are untouched. No migrations.
Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
endpoint), the docreader gRPC tests, and the doris SQL-shape tests
fail; all three are pre-existing on upstream/main and untouched by
this PR.
- New tests across internal/types, opensearch, service and container —
including a full end-to-end env-path test that exercises
initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
httptest cluster.
Split knowledge list/update queries to avoid GORM UPDATE...FROM
duplicate-table errors after Find, and use sync_logs started_at/
finished_at column names instead of start_time/end_time.
This commit introduces the CancelOpenSpansByName method in the KnowledgeSpanRepository, allowing for the cancellation of open spans by their name for a specific knowledge ID and attempt. This functionality is crucial for managing spans during retries or server restarts, preventing duplicate entries in the trace tree. Additionally, a new test case, TestKnowledgeSpanRepo_CancelOpenSpansByName, has been added to ensure the correct behavior of this method, verifying that only the intended spans are cancelled while others remain unaffected. This enhancement improves the robustness of span management in the application.
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
Adds a five-segment progress model for the document parsing pipeline so
the UI (PR③) can render a timeline showing where each document is
(DocReader → Chunking → Embedding → Multimodal → PostProcess) and
which stage failed with what error code.
- New table `knowledge_processing_stages` (migration 000052) with one
row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an
attempt counter so re-parses don't lose history.
- StageTracker service exposes Begin/Done/Fail/Skip; all calls are
best-effort and never break the pipeline if persistence fails.
- Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT /
VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized
remediation hints.
- Tracker call sites added at the four meaningful failure points:
convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding),
enqueueImageMultimodalTasks (Multimodal start),
KnowledgePostProcess.Handle (Multimodal close + PostProcess).
- New endpoint `GET /api/v1/knowledge/:id/stages` returns the five
canonical stages — missing rows are synthesized as "pending" so
the timeline always renders five segments. Includes current_stage
and last_error block.
Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.
- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
(WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.
- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
(default 30m). Without this, a hung docreader pinned a worker for the
full DocumentProcessTimeout window.
- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
failing single image no longer strands the parent — the asynq retry
is allowed to run, but on the final attempt we count the image
regardless of outcome. Redis DECR errors fall back to enqueuing the
post-process task instead of returning silently.
- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
ManualProcess exhausts retries, immediately mark the corresponding
Knowledge as failed with the last error. This surfaces the failure
in the UI without waiting for the housekeeping sweep.
- HousekeepingService: 5-minute cron that flips knowledge rows stuck
in "processing" past DocumentProcessTimeout + 10m to failed, plus
summary rows stuck > 1h. Catches anything the other safety nets
miss (worker SIGKILL mid-handler, etc.). Disable with
WEKNORA_HOUSEKEEPING_ENABLED=false.
- Distributed startup recovery: previously the post-restart sweep was
skipped whenever REDIS_ADDR was set, even though Asynq does not
reschedule the task that was actively running on the dead instance.
Now the sweep runs in distributed mode too, but only against rows
older than 30 minutes to avoid racing peer instances.
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
Problem:
Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing
VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime
with SQL parse errors.
Root cause:
1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in
MySQL/PostgreSQL but not in Doris's SQL dialect)
2. Doris MySQL driver was not configured to interpolate parameters, preventing
automatic client-side placeholder replacement
Fix:
1. Enable parameter interpolation in Doris connection DSN:
- Add &interpolateParams=true to mysql.Open() DSN in container.go
- This enables driver-level parameter substitution at query time
2. Inline LIMIT/OFFSET as string literals in Doris queries:
- VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK)
- KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK)
- CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset)
- Remove inlined values from args slice
Result:
Queries are now built with literal LIMIT/OFFSET values instead of placeholders,
compatible with Doris's SQL parser.
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.
Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.
The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
Added functionality for managing pending invitations, including a polling mechanism to update the invitation count in real-time. Introduced a new dialog component for users to view and respond to their invitations without navigating away from their current context. Updated API endpoints for fetching and managing invitations, and enhanced the user interface with relevant internationalization support.
Refs: #1303
Introduced a new API client for managing user favorites, allowing users to list, add, and remove starred resources. Enhanced the sidebar to display favorites and recents, improving user navigation. Updated internationalization files to include relevant labels and tooltips for the new features across multiple languages.
Refs: #1303
The audit_logs table grows monotonically until something purges it.
PR 6 explicitly listed retention as out of scope for v1; this picks
that up.
Adds an `audit.retention_days` config (default 90, 0 disables) and a
small background goroutine `AuditLogRetentionRunner` that fires once
~10 minutes after boot and then every 24h, calling
`AuditLogService.Purge` which DELETEs rows older than the cutoff in a
single statement. The dependency surface is intentionally minimal —
no robfig/cron, no asynq, no migration — because retention has no
wall-clock alignment requirement.
Wiring is the same shape as the existing data source scheduler:
container `Provide(NewAuditLogRetentionRunner)`, an `Invoke` that
calls `Start`, and a `ResourceCleaner.RegisterWithName` for graceful
shutdown.
Defaults preserve operator intent:
- `audit:` section omitted from YAML -> retention_days=90.
- explicit `audit.retention_days: 0` in YAML -> purge disabled.
- `WEKNORA_AUDIT_RETENTION_DAYS=N` env override (incl. N=0).
- `retention_days < 0` is rejected by ValidateConfig.
Tests cover the service Purge contract (no-op when disabled, cutoff
math, error propagation) and the runner lifecycle (Start no-op when
disabled, idempotent Start/Stop, ticker cadence, runOnce swallows
errors). 10 new tests, all green.
docs/rbac.md updated with the new YAML / env / behaviour.
Refs: #1303
PR 1-5 of the multi-tenant RBAC series enforce the role matrix; what
they don't do is keep a durable record of who did what. Today the
RBAC middleware logs "[rbac] role insufficient ..." to stderr -
perishable, not searchable. Operators cannot answer "who removed
Alice from tenant T-foo on Tuesday" or "did we 403 a probing user
200 times last hour" after the fact.
This PR adds a generic per-tenant audit log that:
- Captures mutating RBAC actions (member add/remove/role-change/leave)
durably from the tenant_member service after the repo write succeeds.
- Captures enforcement rejections (RequireRole / RequireOwnershipOrRole
denies under EnableRBAC=true) via a gin-context-injected hook.
- Sliding-window dedup (1 minute) on (tenant, actor, action, path)
for denied events so a probing client cannot flood audit_logs.
- Default-on - no extra env flag. Schema is generic enough for future
KB / agent / datasource events to reuse via new action namespaces
without another migration.
Schema (migration 000044) mirrors wiki_log_entries' shape: a single
audit_logs table with action, target_type/id, request path/method,
outcome, and a JSONB details column. Three indexes: (tenant_id, id
DESC) for the per-tenant cursor feed, actor_user_id for "what did
Alice do", (tenant_id, action) for both the action filter and the
dedup lookup. Sqlite mirror added for Lite mode.
Wiring is nil-safe: the audit field on tenantMemberService and the
gin-context lookup in rbac.go both tolerate a nil service so existing
tests construct without a stub and dormant deployments degrade
gracefully if the provider is misconfigured.
The query API is GET /tenants/:id/audit-log, gated by Admin+ on top
of the existing PathTenantMatch (denial histories should not surface
to ordinary members). Cursor-paginated by descending id (sidesteps
the duplicate-timestamp tie-breaking that BIGSERIAL trivially solves).
Frontend: TenantMembers.vue gets a t-tabs wrapper with Members
(existing) and Audit log (new) tabs; the audit tab is gated to
Admin+ to mirror the server, lazy-loads on first open, paginates via
the next_cursor pattern, and renders coloured chips per
action/outcome so an operator can scan for anomalies. i18n keys
added in all four locales.
Tests: 14 new tests across service (6), middleware (4), and handler
(4). Service tests pin the dedup window behaviour (writes again
after expiry, dedup is per-actor-per-path, degrades on lookup
error). Middleware tests pin both reject sites' audit hook and the
"dormant mode does not fire" + "nil audit does not panic"
invariants. Handler tests pin the cursor envelope and that an empty
page returns next_cursor=0.
Out of scope: per-action ACL on the audit log, retention/TTL job,
KB/agent/datasource event wiring (schema supports it - just no
emitters yet), async writes, frontend filter UI.
Wire the human-facing surface for the tenant-RBAC system that PR 1
introduced and PR 2 enforced: REST endpoints + UI for managing tenant
members, plus the long-pending invite_only registration gate.
Backend
- internal/handler/auth.go: Register now rejects with 403 when
auth.registration_mode=invite_only. PR 1 already exposes the mode via
/auth/config so the frontend hides the sign-up link; this is the
server-side enforcement for direct API calls.
- internal/handler/tenant_member.go: new TenantMemberHandler with
ListMembers / AddMember / UpdateMemberRole / RemoveMember. Maps
service-layer sentinels onto HTTP semantics (ErrLastOwner -> 409,
ErrMembershipAlreadyExists -> 409, ErrUserNotFound on AddMember -> 404,
ErrInvalidTenantRole -> 400). AddMember requires the invitee to
already have an account; SMTP-driven email invites are deferred.
- internal/types/tenant_member.go: new TenantMemberResponse projection
carrying the fields the management UI renders (email, username,
avatar, role, joined_at).
- internal/router/router.go: four routes under /tenants/:id/members
gated per the matrix (Viewer+ for list, Owner+ for every mutation).
- internal/container/container.go: dig provider for the new handler.
Tests
- handler/tenant_member_test.go: 12 table-driven cases covering happy
path, dangling-user tolerance, ErrLastOwner -> 409, ErrMembershipNotFound
-> 404, ErrMembershipAlreadyExists -> 409, unknown email -> 404,
invalid role rejected before user lookup, bad tenant id -> 400.
- handler/auth_register_invite_only_test.go: invite_only -> 403,
self_serve -> 201, nil Auth config falls through.
Frontend
- src/api/tenant/members.ts: typed client (TenantRole/TenantMember/...)
for the four endpoints.
- src/views/settings/TenantMembers.vue: members table with add dialog,
inline role select, confirm-remove modal. All mutation widgets gated
behind currentTenantRole === 'owner'; server still enforces.
- src/views/settings/Settings.vue: register the Members tab; show only
when currentTenantRole is loaded (avoids tab flash on cold nav).
- i18n in en-US/zh-CN/ko-KR/ru-RU: new tenantMember namespace covering
all UI strings and structured error messages (lastOwner / userNotFound
/ alreadyMember / invalidRole etc).
This completes the #1303 RBAC rollout. With cfg.Tenant.EnableRBAC=true
operators now have a full management surface; with EnableRBAC=false
the routes still exist but mutations log-and-pass per PR 2 contract.
Lay the foundation for tenant-level RBAC without enforcing it yet.
Schema, types, middleware lookup, and login-flow shape changes ship now;
RequireRole/route annotations and member-management endpoints follow in
PR 2 and PR 3.
Highlights:
- New tenant_members table (Postgres + SQLite) with (user_id, tenant_id)
unique key and (owner|admin|contributor|viewer) role values. Migration
000043 backfills one row per existing user, marking the earliest-created
active user in each tenant as Owner and the rest as Contributor.
- KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so
PR 2 can wire ownership-based and Viewer-runtime checks.
- Auth middleware loads the active TenantMember row, sets a new
TenantRoleContextKey, and falls back via three escape hatches:
cross-tenant superusers (CanAccessAllTenants) get Admin in the target
tenant; orphan tenants (zero members) auto-promote the first authenticating
human to Owner; and when tenant.enable_rbac is false (the v1 default)
unmatched lookups fail open with Admin to preserve current behaviour.
- Register / OIDC provisioning now insert an Owner membership for the
registrant via TenantMemberService.EnsureOwner.
- LoginResponse renamed tenant -> active_tenant and gained a memberships
array. POST /auth/switch-tenant issues a fresh token pair scoped to a
target membership; GET /auth/config exposes auth.registration_mode for
the frontend register-tab gating.
- New config keys: auth.registration_mode (self_serve | invite_only,
defaults to self_serve) and tenant.enable_rbac (defaults to false).
Both have WEKNORA_* env overrides.
- Frontend store persists memberships and exposes currentTenantRole;
Login.vue calls /auth/config and hides the Register tab in invite_only
mode. Go SDK types updated to match the new LoginResponse shape.
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory
into the user-facing knowledge-base lifecycle:
- POST /knowledge-bases validates the requested vector_store_id against
the caller's tenant scope and the engine registry. New error codes
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201)
distinguish the typed branches without echoing UUIDs to the client.
- GET / POST / PUT / PUT-pin responses embed the bound store's display
metadata (name, source, engine_type, status) without exposing any
connection credentials. Cross-tenant shared KBs receive a suppressed
payload (vector_store_id stripped, source="shared") so operator-chosen
store names cannot be enumerated across tenants.
- POST /knowledge-bases/copy synchronously rejects clones whose target
has a different embedding model or vector store, before the async
clone task is enqueued. The async clone worker re-applies the same
checks for defense in depth.
- DELETE /vector-stores/:id refuses to remove a store with bound KBs,
inside a transaction that row-locks the store on PostgreSQL and
serializes via WAL on SQLite. unregister-from-registry is wrapped in
defer/recover so a panic surfaces as a structured warning instead of
silently leaking a stale engine.
- vector_store_id is immutable after creation. The GORM <-:create tag
blocks every ORM update path; the service-layer DTO omits the field
entirely; a reflection-based regression test catches any future
maintainer who adds it back to either layer.
- Empty-string vector_store_id is normalized to nil at both the create
path and inside SharesStoreWith, so rows persisted by callers that
did not run Normalize first cannot trip false same-store comparisons.
Part of #993. Depends on #994 and #1310.
Replace the legacy "redacted placeholder + Clear* boolean" pattern with
dedicated per-resource credential subresources across MCP services,
Models, Web search providers, and Data sources.
Why
---
The previous design had three problems:
1. Main PUT body carried secret fields. The frontend echoed back a
redacted "***" placeholder, and a fragile MergeUpdate / IsRedactedOrEmpty
defense in the service layer tried to detect "user did not change this"
vs "user wants to clear this". A regression in that defense (or a new
frontend forgetting it) silently overwrites the stored secret with the
placeholder.
2. The "remove this credential" UX was a red checkbox under a pre-filled
password input. Three intents (preserve / replace / clear) collapsed
onto one field, and credential changes were bundled with unrelated
config edits in the same submit. Users wiped working keys by mistake.
3. Secret presence was inferred from "did the response come back with a
'***' placeholder", which couples the contract to a magic string.
Design
------
Each of the four resources now exposes:
PUT /{resource}/{id}/credentials # write one or more fields
DELETE /{resource}/{id}/credentials/{field} # clear a single field
"Is this configured?" metadata travels on the main resource response as
a typed map (dto.MCPServiceResponse.Credentials etc.) — no separate GET
endpoint. The frontend reads the configured boolean from the main GET
and never sees secret values at all.
Main PUT endpoints now ignore any secret fields in the request body and
log a deprecation warning if they appear, so legacy clients fail loudly
rather than overwriting silently.
Frontend
--------
- New reusable <CredentialResource> component renders a three-state card
(unconfigured / configured / editing) and drives the dedicated
endpoints. Used by MCP, Model, Web search; DataSource has a bespoke
card with the same behaviour because its credentials are a single
atomic per-connector map.
- Cancel from edit mode now restores state synchronously from the meta
prop. The previous async refresh() was a no-op while state was still
'editing', leaving the input frozen open.
- Remove is single-click + toast. The danger-themed button is the
deterrent; a modal confirm adds friction without adding safety (the
plaintext is irrecoverable client-side either way — recovery means
re-typing).
DTOs (internal/handler/dto/) are deliberately separate from the GORM
models so "no secret in response" is a compile-time invariant: a future
contributor cannot leak a secret without explicitly adding the field to
the DTO, which is review-able in a single diff.
Storage is unchanged — credentials still live in the existing jsonb /
parameters columns. No schema migration.
Cleanup
-------
- types.MCPService / types.Model / types.WebSearchProviderEntity /
types.DataSourceConfig: drop ClearAPIKey / ClearToken / ClearAppSecret
/ ClearCredentials boolean fields, MergeUpdate(), RedactSensitiveData().
- utils/types/secret.go: drop PreserveIfRedacted / IsRedactedOrEmpty.
RedactedSecretPlaceholder constant is retained because VectorStore
still uses the old pattern and is out of scope here.
- Frontend hasExistingApiKey / clearApiKey / convertToLegacyFormat
redaction handling removed; i18n keys renamed secret -> credential.
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across
seven services into two factory functions with tenant-ownership
verification: CreateRetrieveEngineForKB (synchronous) and
CreateRetrieveEngineFromPayload (async task handlers). Promotes
GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry.
Extends KBDeletePayload / IndexDeletePayload with an omitempty
VectorStoreID snapshot.
Design highlights
- Factory verifies tenant ownership of the resolved store
(defense-in-depth) so a gap in PR3 validation or a tampered Asynq
payload cannot cross tenants. Cross-tenant attempts return
ErrVectorStoreForbidden with a structured log entry.
- Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound,
ErrVectorStoreForbidden) let async handlers classify non-retryable
failures as asynq.SkipRetry.
- nil and empty-string pointers for vectorStoreID normalize to
"unbound" so callers never send an empty UUID to GetByStoreID.
- resolveBoundEngine constructs engineInfos directly (with
slices.Clone of Support()) instead of going through
NewCompositeRetrieveEngine. The latter only reads from the
byEngineType map and cannot reach DB stores in byStoreID -- multiple
stores can share the same engine type. Unbound fallbacks still reuse
NewCompositeRetrieveEngine so the tenant effective-engines path is
byte-for-byte unchanged.
- DeleteKnowledgeBase now reads the KB before soft-delete so the
enqueued KBDeletePayload carries a VectorStoreID snapshot; once the
row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT
per KB-delete request (a rare admin operation).
- enqueueIndexDeleteTask gains a vectorStoreID *string parameter that
the single caller (tag.go) populates from the owning KB. The worker
(ProcessIndexDelete) validates ownership via the factory.
- cleanupKnowledgeResources loads the KB so bound-store cleanup routes
correctly. If the load fails it warns and falls back to tenant
effective engines; orphan vectors become observable in logs instead
of disappearing silently.
- DeleteKnowledges (batch) stays on the tenant effective-engines path
-- batches can span stores and multi-store fan-out is PR4 scope.
Noted inline.
Merge safety
All knowledge bases have VectorStoreID = NULL at merge time (introduced
nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to
NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()),
which is the existing code path for these KBs. Async payloads from
before this change decode to VectorStoreID = nil (omitempty) and take
the same fallback.
Testing
- go build ./...
- go vet ./...
- go test -race -count=1 ./internal/application/service/retriever/...
covers unbound (nil/empty), bound, cross-tenant, unregistered store,
ownership infra error, legacy payload shapes (missing field/null/
empty string/nil), tampered payload, and parallel invocation.
- grep -r NewCompositeRetrieveEngine internal/application/service
returns zero results outside factory.go/composite.go.
Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994.
The Agent ran a parallel context cache (Redis/in-memory) on top of the
messages table to feed multi-turn history into the LLM. That dual-write
caused subtle drift (e.g. compression diverging from DB, system-prompt
swaps lost on restart) and required a separate ClearContext path on
session/IM clear.
Make the messages table the single source of truth:
- Add service.LoadAgentHistory: rebuilds chat.Message history per turn
from the persisted messages, expanding AgentSteps into proper OpenAI
assistant_with_tool_calls + tool messages and replaying the canonical
final answer (with <think> blocks stripped). final_answer tool calls
are filtered to avoid duplicating the trailing answer.
- Make AgentEngine stateless across turns: drop ContextManager / sessionID
cache plumbing from the engine, agent service, and CreateAgentEngine
signature. The engine only uses sessionID for logging/event emission.
- Wire AgentQA to load history from DB on demand using HistoryTurns
(default 5) when MultiTurnEnabled, otherwise run with empty history.
- Delete the llmcontext package (ContextManager interface, Redis/memory
storage, factory) and the SessionService.ClearContext API path; IM
/clear and session message clear no longer need to invalidate cache.
Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with
empty AgentSteps just produce the single canonical assistant message),
and Agent-mode turns now consistently see prior tool calls and results.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
IM platforms (Feishu, Slack, Telegram, DingTalk, Mattermost, WeCom) cannot
render provider:// URLs (local://, minio://, s3://, etc.) that appear in
LLM answers containing knowledge base images. The web frontend handles
these via the authenticated /files endpoint, but IM clients need publicly
resolvable HTTP URLs.
Changes:
- Add HMAC-SHA256 presigned URL utility (internal/utils/presign.go) for
generating time-limited, signature-verified file access URLs
- Add GET /api/v1/files/presigned endpoint that serves files without
session auth, verified by HMAC signature and expiry
- Update localFileService.GetFileURL() to return presigned HTTP URLs
when APP_EXTERNAL_URL is configured (cloud backends already return
presigned URLs via their SDKs)
- Add IM content rewriting pipeline: strip <image> XML tags, strip
citation tags, rewrite storage URLs to HTTP — applied at all IM
output points (streaming flush, non-streaming reply, fallback)
- Add holdback buffer in streaming flush to prevent URL/tag splitting
across chunk boundaries
Closes#1058
registerIMAdapterFactories in internal/container/container.go had grown
to ~290 lines of platform-specific config parsing and runtime start/stop
logic across 7 IM platforms, plus three private credential helpers. The
container package is meant to wire dependencies, not own each platform's
credentials schema and websocket lifecycle.
Extract each platform's factory into its own subpackage file and expose
the credential helpers on the im package:
- New im.ParseCredentials / im.GetString / im.GetBool (formerly the
unexported parseCredentials / getString / credentialBool in container).
- New im.ResolveMode helper collapses the six identical "default to
websocket when Mode is empty" blocks.
- New {wecom,feishu,slack,telegram,dingtalk,mattermost,wechat}.NewFactory
returns an im.AdapterFactory; container.go now calls each one in a
single line.
Behavior is preserved: adapter construction, goroutine start/stop,
log messages, default modes (mattermost still defaults to "webhook";
others to "websocket"), and error strings are unchanged. The WeCom
corp_agent_id float64/int switch and the WeChat long-poll-only path
are preserved as-is.
Net: container.go drops ~320 lines and no longer needs to be touched
when adding or tuning an IM platform.
DuckDB's st_read (spatial) only reads the first layer/sheet of a .xlsx
workbook, so every sheet beyond Sheet1 was silently dropped from the
DuckDB table the Data Analysis tool builds. Users trying to analyse
multi-sheet workbooks could only see the first sheet.
Switch to DuckDB's dedicated 'excel' extension (read_xlsx) for the
actual data load, enumerate sheets via the spatial extension's
st_read_meta, and UNION ALL BY NAME the rows of every sheet into one
table. A synthetic __sheet_name column records the source so the LLM
can still filter/aggregate per sheet; schema drift between sheets is
tolerated via UNION BY NAME. If enumeration fails (older DuckDB, local
filesystem errors, …) we fall back to reading the first sheet so the
tool stays usable.
- Install & LOAD the 'excel' extension alongside 'spatial' both at
startup (internal/container) and in the offline prefetch binary
(cmd/download/duckdb).
- Harden sheet/path handling against single quotes.
- Update the tool description so the agent knows about __sheet_name.
- Add unit tests for the CREATE TABLE SQL builder covering 0 / 1 / N
sheets and quote escaping.
Refs: https://github.com/Tencent/WeKnora/issues/1007
Closes#620#497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes:
**Backend Changes:**
- Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log)
- Database schema: wiki_pages table with full migration support
- WikiPageService: CRUD operations and page management
- WikiPageRepository: GORM-based persistence layer
- Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents
* Summary page generation using LLM
* Entity and concept extraction in a single LLM call
* Synthesis opportunity detection
* Index page rebuilding
* Log page maintenance
- Wiki boost feature: Enhance chat retrieval with wiki context
- Wiki linting: Maintenance and validation utilities
- Agent wiki tools: Enable agents to query and interact with wiki pages
- Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks
- Language support: Reuse existing middleware language infrastructure for LLM prompts
**Frontend Changes:**
- Wiki browser UI: View all wiki pages with filtering and search
- Wiki API client: Knowledge base wiki management endpoints
- Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model)
- i18n updates: Support for English, Korean, Russian, and Chinese interfaces
**Configuration:**
- Container DI: Wire up all wiki services
- Router: Register wiki API endpoints
- Task handling: Support async wiki ingest tasks
**Testing:**
- Unit tests for wiki page types
- Service layer tests
- Endpoint tests for wiki operations
- Integration tests with LLM mocking
**Documentation:**
- Language refactoring analysis and guides
- Implementation completion reports
- Quick reference guides for developers
**Key Features:**
✅ LLM-powered wiki page generation from documents
✅ Multi-language support (9+ languages)
✅ Automatic extraction of entities and concepts
✅ Synthesis opportunity detection
✅ Index and log page maintenance
✅ Progressive wiki building across multiple documents
✅ Agent-based wiki interaction
✅ Chat retrieval enhancement with wiki context
✅ Full frontend UI for wiki browsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Syncs personal and team Yuque knowledge bases into WeKnora as Markdown,
with full and incremental sync + deletion detection. Mirrors the Feishu
connector pattern; fetches V2Doc.body directly (no async export step).
Config: api_token (required) + base_url (optional, defaults to public cloud).
401/403 wraps ErrInvalidCredentials; 429 honors Retry-After; filters to
status=1 Doc-type at list stage to conserve API quota. Body is trusted as
Markdown for format=markdown|lake — empirically verified against the v2
API: Lake XML lives separately in body_lake, body itself is the Markdown
view. Other formats (e.g. html) are skipped with a placeholder item so
they surface in sync logs instead of silently feeding HTML to the parser.
v2Doc.status accepts both int and string shapes — Yuque's OpenAPI spec
declares string but the live API returns int. sanitizeFileName truncates
at a UTF-8 rune boundary so multi-byte titles don't emit invalid UTF-8
that downstream filename validation rejects.
Also adds logger.SetOutput test hook, used by the token-redaction test.
- Extend RetrieveEngineRegistry with byStoreID map for DB store engines
- Add StoreRegistry interface and EngineFactory function type
- Create engine factory with per-driver SDK client construction
- Load VectorStore records from DB into registry at startup
- Wire Create/Delete to dynamically register/unregister engines
- Add comprehensive unit tests for registry and service
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
Introduce KnowledgePostProcess task to decouple image multimodal
processing from downstream operations like summary, question
generation, and knowledge graph extraction.
This allows unifying the processing logic across documents and
images and ensures all text content, including from images,
is properly extracted before post-processing.
- Removed support for video file uploads across the application, including updates to the knowledge base and document processing services.
- Updated internationalization files to reflect the removal of video-related terminology, changing labels and descriptions to focus solely on audio processing.
- Adjusted file type handling to exclude video formats, ensuring users receive accurate feedback regarding supported file types.
- Enhanced error handling to provide clear messages when users attempt to upload unsupported video files.
This update streamlines the application by focusing on audio processing capabilities and improving user experience through clearer messaging.
Wire VectorStoreService to HTTP with 8 endpoints: types metadata, CRUD
(create/list/get/update/delete), and connection testing (raw + by ID).
Register routes, DI container bindings, and add API documentation.
Allow configuring custom WebSocket and API base URLs for WeCom
via optional credentials fields (ws_endpoint, api_base_url), enabling
integration with privately deployed WeCom instances.
Changes:
- LongConnClient and WebhookAdapter accept custom endpoint URLs
- Endpoint URLs validated at construction (scheme + SSRF check)
- Private deployment hostnames added to file download SSRF allowlist
- Frontend forms added for both WebSocket and Webhook modes
- i18n support for zh-CN, en-US, ru-RU, ko-KR
Uses default public cloud endpoints when not specified.
- Remove infrastructure/crypto package (CryptoService, config, PBKDF2-based encryption)
- Replace with lightweight utils.GetAESKey/EncryptAESGCM/DecryptAESGCM helpers
- Remove cryptoSvc dependency from modelService, tenantService, weKnoraCloudService
- Remove crypto state persistence logic from container initialization
- Add TruncatePromptTokens field to WeKnoraCloud embed request
- Update frontend hint text for WeKnoraCloud credential errors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
wizardchen