Enhance error messaging for storage connectivity issues by sanitizing raw errors to prevent leaking sensitive information. Introduce a new utility function for consistent error handling across storage backend operations. Additionally, implement logic to support multi-instance storage backends, allowing for more flexible workspace configurations and improved reporting of active storage providers.
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
Extract storage allowlist into internal/storageallowlist so KB creation
uses FirstAllowed when tenant default_provider is empty or disallowed.
Share pickUsableStorageProvider via editorResources for consistent UI
selection, and add regression tests for OBS status and allow-list defaults.
Stop failing distributed-mode knowledge rows on startup; let housekeeping
decide using span activity and Asynq queue state. In Lite mode, resume
wiki work from durable task_pending_ops and surface SERVER_RESTART errors
in the processing timeline with i18n support.
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
Allow platform system administrators to update builtin model parameters
and credentials from the management UI while keeping secrets masked in API
responses. UI edits clear YAML ownership so startup reconciliation no
longer overwrites runtime overrides.
Added new tests for the ImageMultimodalService to verify the handling of orphan tasks and the finalization of pending counters. Improved runtime task inspection with better error handling and task cancellation logic, ensuring robust management of orphan tasks in the system.
buildParentChildConfigs only copied ChunkSize/ChunkOverlap/Separators into
the parent and child SplitterConfigs, dropping base.Strategy. Since an empty
Strategy resolves to the legacy tier (resolveChainWithProfile treats "" as
legacy for backwards compatibility), parent-child chunking always ran the
legacy recursive splitter regardless of the configured strategy: no heading
alignment and no ContextHeader breadcrumbs in child embeddings, silently.
The flat path (buildSplitterConfigFromChunking) already propagates Strategy;
parent-child predates the strategy system (b3334990, 2026-03) and was not
updated when the 3-tier strategy landed (ae6fde23, 2026-05). SplitParentChild's
docstring ("Child splitting honours childCfg.Strategy") and mergeBreadcrumbs
both assume the strategy reaches the parent-child path.
Copy base.Strategy into both configs and add a regression test.
Enable agents to configure starter prompts and async follow-up questions after completed answers, with durable caching, analytics events, and chat/embed UI integration.
Let system administrators reset another user's local password from the settings UI, revoke all active sessions, and record the action in the audit log.
Drop the standalone OIDC_AUTH_DEFAULT_TENANT_MODE / oidc_auth.default_tenant_mode
knob so OIDC auto-provisioning and local password registration share the single
auth.default_tenant_mode policy (resolved by the callback handler). Repair
tenantless sessions by falling back to the first active membership on the login
happy path and stale-preference paths, allow tenantless PUT /auth/me/preferences,
clear the tenant snapshot on /auth/me responses without a tenant, and default the
frontend canCreateTenant capability to fail-closed until /auth/me confirms it.
Allow deployments to register users without auto-creating a personal tenant, expose creation policy via /auth/me, and guide tenantless accounts through a workspace onboarding flow.
Enforce tenant ownership in ListSharesByAgent so full-access API keys cannot enumerate cross-tenant shares, and allow full-access keys to create KBs and manage share routes consistently with other tenant-wide operations.
Wiki ingest prompts previously encouraged inline [cNNN] citations that
leak internal chunk metadata into reader-facing Markdown. Strip legacy
aliases on read/write paths and update the modify prompt to keep prose
clean while source links remain in ChunkRefs.
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
Follow-up to the knowledge span name column expansion: truncate span
names by rune count to match PostgreSQL VARCHAR(255), apply the same
normalization in LookupSpanByName, and add regression tests for mid-length
wiki slugs and CJK truncation boundaries.
Wiki ingest builds postprocess.wiki.page[<slug>] subspan names that can exceed
varchar(64); widen the column to 255 and hash-suffix truncation so long slugs
still persist distinctly.
Co-authored-by: Cursor <cursoragent@cursor.com>
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
Expose live asynq queue depths and worker-pool concurrency via a new SystemAdmin API and settings UI, so operators can monitor parsing/wiki backlog without Redis CLI access.
Add a tenant.auto_create_api_key setting (env
WEKNORA_TENANT_AUTO_CREATE_API_KEY, default false) that restores the
pre-break-change behaviour of minting a full_access API key when a
tenant is created and returning its plaintext token in the create
response (data.api_key).
- system_setting: register tenant.auto_create_api_key bool spec
- handler: gate auto key creation behind the setting; embed the token
via tenantWithAPIKey without re-adding a persisted api_key column;
key-creation failures log a warning and do not fail tenant creation
- client: add optional Tenant.APIKey field populated only on create
- docs/env: document the switch (.env.example, docker-compose,
tenant.md, regenerated swagger)
Increase the default value of WEKNORA_MODEL_MAX_CONCURRENCY from 8 to 32 in .env.example, docker-compose.yml, and related application code. This change enhances the concurrency limit for background tasks, ensuring better performance and resource management across the system.
Add a new `max_concurrency` setting to the model configuration, allowing for fine-tuned control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This update includes localization support across multiple languages and integrates the setting into the system's concurrency governor, enabling immediate application of changes without requiring a service restart.
- Renamed TestTaskPendingOps_ClaimBatch_NewSiblingRowNotDoubleClaimed to TestTaskPendingOps_ClaimBatch_LateSiblingBlockedByFreshClaim for clarity on behavior.
- Added new test TestTaskPendingOps_ClaimBatch_LateSiblingClaimableAfterStale to verify claim behavior when a holder crashes.
- Updated ClaimBatch logic to ensure that fresh claims block late siblings entirely, maintaining serialization of same-document operations.
- Improved comments in task_queue.go to clarify the handling of concurrent claims and the implications of the claimed_at field.
- Added crash safety mechanism in ProcessWikiIngest to release claims on abnormal exits, ensuring timely re-claims of pending operations.
- claim/peekPendingList now surface DB errors so a transient load
failure retries via asynq instead of acking as a false "no pending
ops" success that strands the queue.
- no_pending_ops now arms a coalesced stale-claim recheck (TaskID
keyed, delayed past wikiClaimStaleAfter) so rows held by a crashed
batch's fresh claim are reclaimed automatically instead of sitting
unclaimable and never re-triggered.
- ProcessWikiFinalize fails CLOSED on a SetNX error (returns an error
to retry) instead of running unlocked, avoiding two finalize runs
double-rebuilding the index page.
Two correctness fixes on the Phase 3 concurrent claim-based ingest path:
- Slug-lock timeout / reduce failure no longer silently drops work. When a
per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
an error, the page's read-modify-write never lands. Previously the owning
document was still counted as successful and its pending row was deleted at
trim time, permanently losing the contribution (finalize only rebuilds the
index / cross-links, it does not re-run reduce). Such documents are now
folded into failedOps: their rows stay queued, their finalizing slot is
held, and requeueFailedOps' fail_count budget retries them (dead-lettering
a permanently hot/broken slug).
- ClaimBatch now claims by dedup_key affinity so a document is never split
across concurrent batches. limit counts distinct dedup_keys (documents),
and ALL eligible rows sharing a key are claimed together. On Postgres the
per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
concurrent claimers disjoint key sets. This restores in-document ordering
(e.g. ingest-then-retract collapses correctly) that the per-batch dedup
could not guarantee once the exclusive per-KB lock was removed, preventing
a stale ingest from resurrecting a retracted document.
Adds repository tests for same-key affinity and in-flight sibling rows.
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.
- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
concurrency, default 16), split from the parse pool (NewParseAsynqServer).
Both run the same mux over disjoint queue sets. Replaces the old
weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
batches per KB; Map/Reduce fan-out sized within a batch. New config knob
+ system setting + swagger docs.
- docreader: serialize all pdfium access behind a global lock. pypdfium2
is process-global and not thread-safe, so concurrent PDF uploads could
corrupt its shared state and deadlock the whole process (requests stuck
in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
Drop client-supplied target_id, generate IDs server-side, localize duplicate
names from request language, document the endpoint, and expand handler/client tests.
Default missing KB metadata to document tag relation lookup, propagate tag resolution failures instead of returning empty results, and reject ambiguous bare tag_ids when multiple knowledge bases are in scope.
Emit search_source metadata from RAG progress events so quick-answer steps,
summaries, and IM labels no longer mislabel web-only results as knowledge base search.
Add Keenable Search alongside the existing DuckDuckGo / Bing / Google /
Tavily / Ollama / Baidu / SearXNG providers.
- internal/infrastructure/web_search/keenable.go: KeenableProvider
implementing the WebSearchProvider interface, following the Tavily
provider pattern. Keyless by default: with no API key it calls the
public endpoint (/v1/search/public, rate-limited); a key switches to
/v1/search and lifts the cap. Base URL is hardcoded (SSRF-safe), proxy
is honored.
- Registered "keenable" in the container registry.
- Added the keenable provider type + type metadata (RequiresAPIKey=false)
and service validation (no required key, like DuckDuckGo).
- Provider-specific empty-results diagnostic (keyless rate-limit hint).
- Frontend: extend the provider union type with 'keenable'.
- Unit tests: keyless vs keyed endpoint, attribution header, maxResults
trim, result mapping, error status, name.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enhance the tenant API key service by introducing asynchronous throttling for the last_used_at updates during API key authentication. This change reduces database write frequency under high request rates, improving performance. Additionally, update tests to validate the throttling behavior, ensuring that last_used_at is only updated once per key within the defined interval. This implementation aligns with the goal of optimizing API key management and maintaining system efficiency.
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.
- Add APIKeyGate as the single authority for API-key authorization: routes
declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
routes are denied by default. JWT principals short-circuit the gate and the
JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
is a pure downstream data filter); owner is a full-access tenant tier that
unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
writes on API-key encryption failure, deny KB-restricted keys on /files, and
backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
gin paths before comparison; add regression tests.
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
Make document keyword search case-insensitive (LOWER(...) LIKE LOWER(?)) in the
list filter and SearchKnowledge / SearchKnowledgeInScopes paths so a query and
its differently-cased variant return the same results.
Classify not-found consistently as HTTP 404 (not 500) so clients read a
permanent resource.not_found instead of a retryable 5xx: map the document /
chunk not-found sentinels in the KB-access middleware (typed ErrChunkNotFound
via errors.Is), and return 404 for a missing message on the continue-stream and
message-delete endpoints.
Reject refresh attempts when the DB token type is not refresh_token,
and add service-layer tests for bearer refresh rejection, logout
revocation, and expired-token logout handling.
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
wizardchen