Implement a comprehensive API for managing storage backends, including endpoints for listing, creating, updating, and deleting storage instances. Enhance the UI to support multi-instance configurations, allowing users to manage multiple storage types effectively. Update localization files for new storage backend features and improve error handling for connectivity tests. This update also includes detailed documentation for the storage backend API.
Enhance error messaging for storage connectivity issues by sanitizing raw errors to prevent leaking sensitive information. Introduce a new utility function for consistent error handling across storage backend operations. Additionally, implement logic to support multi-instance storage backends, allowing for more flexible workspace configurations and improved reporting of active storage providers.
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
Extract storage allowlist into internal/storageallowlist so KB creation
uses FirstAllowed when tenant default_provider is empty or disallowed.
Share pickUsableStorageProvider via editorResources for consistent UI
selection, and add regression tests for OBS status and allow-list defaults.
Update the manage_kbs capability to cover the full lifecycle of knowledge bases, including create, copy, duplicate, update, and delete operations. Adjust related API routes and tests to ensure proper access control and policy requirements for knowledge base management.
Stop failing distributed-mode knowledge rows on startup; let housekeeping
decide using span activity and Asynq queue state. In Lite mode, resume
wiki work from durable task_pending_ops and surface SERVER_RESTART errors
in the processing timeline with i18n support.
Replace offset-based page numbers with opaque Redis-backed cursors so
scheduled/retry tasks sort by next run time and live lists stay stable
when queue state changes between requests.
Refactor the error handling logic in the GetKnowledgeSpans function to utilize a new helper function, knowledgeSpansLastError, which consolidates the logic for determining the last error from span failures and knowledge row failures. This improves clarity and maintainability of the code. Additionally, update test cases to cover new scenarios for last error handling.
Latest attempt tabs and header now trust parse_status over a closed root span, surface recovery failures via last_error, and keep attempt hover styles from washing out the active tab.
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
Allow platform system administrators to update builtin model parameters
and credentials from the management UI while keeping secrets masked in API
responses. UI edits clear YAML ownership so startup reconciliation no
longer overwrites runtime overrides.
Added new tests for the ImageMultimodalService to verify the handling of orphan tasks and the finalization of pending counters. Improved runtime task inspection with better error handling and task cancellation logic, ensuring robust management of orphan tasks in the system.
buildParentChildConfigs only copied ChunkSize/ChunkOverlap/Separators into
the parent and child SplitterConfigs, dropping base.Strategy. Since an empty
Strategy resolves to the legacy tier (resolveChainWithProfile treats "" as
legacy for backwards compatibility), parent-child chunking always ran the
legacy recursive splitter regardless of the configured strategy: no heading
alignment and no ContextHeader breadcrumbs in child embeddings, silently.
The flat path (buildSplitterConfigFromChunking) already propagates Strategy;
parent-child predates the strategy system (b3334990, 2026-03) and was not
updated when the 3-tier strategy landed (ae6fde23, 2026-05). SplitParentChild's
docstring ("Child splitting honours childCfg.Strategy") and mergeBreadcrumbs
both assume the strategy reaches the parent-child path.
Copy base.Strategy into both configs and add a regression test.
Enable agents to configure starter prompts and async follow-up questions after completed answers, with durable caching, analytics events, and chat/embed UI integration.
Updated the logger to include regex patterns for sanitizing "new_password" and "old_password" fields in JSON logs. Added a corresponding test case to verify the sanitization functionality.
Let system administrators reset another user's local password from the settings UI, revoke all active sessions, and record the action in the audit log.
Drop the standalone OIDC_AUTH_DEFAULT_TENANT_MODE / oidc_auth.default_tenant_mode
knob so OIDC auto-provisioning and local password registration share the single
auth.default_tenant_mode policy (resolved by the callback handler). Repair
tenantless sessions by falling back to the first active membership on the login
happy path and stale-preference paths, allow tenantless PUT /auth/me/preferences,
clear the tenant snapshot on /auth/me responses without a tenant, and default the
frontend canCreateTenant capability to fail-closed until /auth/me confirms it.
Allow deployments to register users without auto-creating a personal tenant, expose creation policy via /auth/me, and guide tenantless accounts through a workspace onboarding flow.
Enforce tenant ownership in ListSharesByAgent so full-access API keys cannot enumerate cross-tenant shares, and allow full-access keys to create KBs and manage share routes consistently with other tenant-wide operations.
Requesty is an OpenAI-compatible LLM gateway that uses the same provider/model
naming convention as OpenRouter, so this mirrors the existing OpenRouter provider.
- internal/models/provider/requesty.go: RequestyProvider mirroring openrouter.go
(base URL https://router.requesty.ai/v1, REQUESTY_API_KEY, RequiresAuth)
- provider.go: ProviderRequesty const + AllProviders() entry + DetectProvider case
- internal/types/model.go + client/model.go: ModelSourceRequesty
- provider_test.go: TestRequestyProviderValidation + DetectProvider case
- frontend model editor + i18n (en/zh/ko/ru) + docs/BUILTIN_MODELS.md provider list
- docs/swagger.{json,yaml} + docs.go: requesty enum value (hand-edited to match; these
are normally swag-generated)
Verified: go build/vet/test pass; live call to https://router.requesty.ai/v1 succeeds.
Signed-off-by: Thibault Jaigu <thibault.jaigu@gmail.com>
Wiki ingest prompts previously encouraged inline [cNNN] citations that
leak internal chunk metadata into reader-facing Markdown. Strip legacy
aliases on read/write paths and update the modify prompt to keep prose
clean while source links remain in ChunkRefs.
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
Borrowers with shared-KB read access could previously request any
owner-tenant storage path via /knowledge-bases/:id/files. Limit the
proxy to exports/ embedded images; raw knowledge uploads continue to
use /knowledge/:id/download.
Org-shared and agent-visible knowledge bases store images under the owner
tenant's storage path, which the tenant-scoped /files route rejects for
borrowing tenants. Add GET /api/v1/knowledge-bases/:id/files gated by
RequireKBAccess and route frontend image hydration through it when kbId
is known.
Register the FAQ import progress route through apiKeyRoute with retrieve
or ingest capability so scoped keys that start imports can poll task
status without hitting APIKeyGate fail-closed 403.
Follow-up to the knowledge span name column expansion: truncate span
names by rune count to match PostgreSQL VARCHAR(255), apply the same
normalization in LookupSpanByName, and add regression tests for mid-length
wiki slugs and CJK truncation boundaries.
Wiki ingest builds postprocess.wiki.page[<slug>] subspan names that can exceed
varchar(64); widen the column to 255 and hash-suffix truncation so long slugs
still persist distinctly.
Co-authored-by: Cursor <cursoragent@cursor.com>
Surface per-model active/waiting/limit from the background task limiter so admins can distinguish task scheduling from model-service throttling, with full i18n coverage.
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
Expose live asynq queue depths and worker-pool concurrency via a new SystemAdmin API and settings UI, so operators can monitor parsing/wiki backlog without Redis CLI access.
Previously the Feishu adapter sent replies with the "send message" API
(POST /im/v1/messages) without a root_id. In topic-enabled groups this
spawned a new topic for every reply instead of attaching to the thread
the user @-mentioned the bot from, fragmenting the conversation.
Switch SendReply (text) and sendCardByCardID (streaming card) to the
"reply message" API (POST /im/v1/messages/:message_id/reply), using
incoming.MessageID as the path parameter. Feishu automatically replies
in thread form when the replied-to message is itself a thread message,
so no reply_in_thread flag or session-mode plumbing is needed.
Add sendWithFallback to retry via the plain send-message API on
fallback-eligible error codes (230019 topic gone, 230054 unsupported
message type, 230071 group does not support reply-in-thread), and to
skip straight to the fallback when incoming.MessageID is empty. This
keeps replies deliverable even when the reply API cannot be used.
Also add a debug log in convertEvent (longconn.go) that prints every
raw received event (msg_type/chat_id/msg_id/root_id/parent_id/thread_id/
content) to aid diagnosing intermittent event-delivery issues.
No interface signature changes; other platform adapters untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a tenant.auto_create_api_key setting (env
WEKNORA_TENANT_AUTO_CREATE_API_KEY, default false) that restores the
pre-break-change behaviour of minting a full_access API key when a
tenant is created and returning its plaintext token in the create
response (data.api_key).
- system_setting: register tenant.auto_create_api_key bool spec
- handler: gate auto key creation behind the setting; embed the token
via tenantWithAPIKey without re-adding a persisted api_key column;
key-creation failures log a warning and do not fail tenant creation
- client: add optional Tenant.APIKey field populated only on create
- docs/env: document the switch (.env.example, docker-compose,
tenant.md, regenerated swagger)
Increase the default value of WEKNORA_MODEL_MAX_CONCURRENCY from 8 to 32 in .env.example, docker-compose.yml, and related application code. This change enhances the concurrency limit for background tasks, ensuring better performance and resource management across the system.
Add a new `max_concurrency` setting to the model configuration, allowing for fine-tuned control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This update includes localization support across multiple languages and integrates the setting into the system's concurrency governor, enabling immediate application of changes without requiring a service restart.
Introduce a new `max_concurrency` parameter to the model configuration, allowing control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This feature is applicable to chat, embedding, and VLM models, with a value of 0 or undefined falling back to the global default. Update UI components and localization files to support this new setting, ensuring a seamless user experience.
Extend the per-model background concurrency governor to embedding, the
highest-volume background model call (document ingestion vectorises every
chunk, so a batch upload can burst the whole worker pool against one provider).
The wrapper sits innermost (below debug/langfuse) because BatchEmbedWithPool
fans a batch out into per-sub-batch BatchEmbed calls via the pooler, and the
pooler invokes BatchEmbed on whichever Embedder is threaded down as `model`;
sitting innermost routes those per-sub-batch provider round-trips through Gate
so the semaphore bounds real concurrent calls rather than one coarse per-doc
unit. Only background (asynq) calls are throttled; interactive query embedding
is never gated.
Follow-up hardening on the per-model background concurrency governor:
- limiter: heartbeat now refreshes the ZSET key's PEXPIRE alongside the
lease score, so a semaphore that stays saturated with no slot turnover
can't let the whole key expire (which would drop every lease and admit
over the limit); reuse one poll timer instead of allocating time.After
each iteration under contention.
- governor: extract the process-wide governor + Gate helper into the
limiter package so chat and vlm share one limiter/limit without
importing each other; add an in-process localLimiter for Lite mode.
- vlm: gate background multimodal (VLM) calls through the same governor;
previously only chat was throttled.
- chat: ChatStream releases its slot when the consumer abandons the
stream and cancels ctx (select on ctx.Done, drain inner), fixing a
slot/goroutine leak.
- container/sync_task: install the local governor in Lite mode and tag
SyncTaskExecutor runs as background; honour model.max_concurrency=0 as
an explicit disable instead of falling back to the default.
- tests: cover the governor gating, local limiter, and stream release.
Background ingestion/enrichment stages (summary, question, graph,
multimodal) all target the same chat model, and asynq queue weights are
scheduling priority rather than throttling. A large upload could therefore
burst up to the entire worker pool of concurrent chat calls against a
single provider, unbounded across replicas, with no model-side rate limit.
Govern concurrency at the chat-client layer (the one place that sees every
task type), keyed by model ID:
- internal/models/limiter: Redis ZSET + Lua distributed semaphore with a
lease heartbeat, expired-lease reclaim for crashed holders, and fail-open
on any backend error or context cancellation.
- chat: concurrencyChat decorator (outermost in NewChat) acquires a
per-model slot only for background tasks; interactive chat is never gated.
- router/task: asynq middleware tags every worker execution as background.
- container: wire the Redis limiter with WEKNORA_MODEL_MAX_CONCURRENCY
(model.max_concurrency), default 8; enabled only when Redis is present.
Tested with miniredis: concurrency cap, expired-lease reclaim, per-key
isolation, and all fail-open paths (nil client, backend down, cancelled ctx).
- Renamed TestTaskPendingOps_ClaimBatch_NewSiblingRowNotDoubleClaimed to TestTaskPendingOps_ClaimBatch_LateSiblingBlockedByFreshClaim for clarity on behavior.
- Added new test TestTaskPendingOps_ClaimBatch_LateSiblingClaimableAfterStale to verify claim behavior when a holder crashes.
- Updated ClaimBatch logic to ensure that fresh claims block late siblings entirely, maintaining serialization of same-document operations.
- Improved comments in task_queue.go to clarify the handling of concurrent claims and the implications of the claimed_at field.
- Added crash safety mechanism in ProcessWikiIngest to release claims on abnormal exits, ensuring timely re-claims of pending operations.
- claim/peekPendingList now surface DB errors so a transient load
failure retries via asynq instead of acking as a false "no pending
ops" success that strands the queue.
- no_pending_ops now arms a coalesced stale-claim recheck (TaskID
keyed, delayed past wikiClaimStaleAfter) so rows held by a crashed
batch's fresh claim are reclaimed automatically instead of sitting
unclaimable and never re-triggered.
- ProcessWikiFinalize fails CLOSED on a SetNX error (returns an error
to retry) instead of running unlocked, avoiding two finalize runs
double-rebuilding the index page.
Two correctness fixes on the Phase 3 concurrent claim-based ingest path:
- Slug-lock timeout / reduce failure no longer silently drops work. When a
per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
an error, the page's read-modify-write never lands. Previously the owning
document was still counted as successful and its pending row was deleted at
trim time, permanently losing the contribution (finalize only rebuilds the
index / cross-links, it does not re-run reduce). Such documents are now
folded into failedOps: their rows stay queued, their finalizing slot is
held, and requeueFailedOps' fail_count budget retries them (dead-lettering
a permanently hot/broken slug).
- ClaimBatch now claims by dedup_key affinity so a document is never split
across concurrent batches. limit counts distinct dedup_keys (documents),
and ALL eligible rows sharing a key are claimed together. On Postgres the
per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
concurrent claimers disjoint key sets. This restores in-document ordering
(e.g. ingest-then-retract collapses correctly) that the per-batch dedup
could not guarantee once the exclusive per-KB lock was removed, preventing
a stale ingest from resurrecting a retracted document.
Adds repository tests for same-key affinity and in-flight sibling rows.
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.
- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
concurrency, default 16), split from the parse pool (NewParseAsynqServer).
Both run the same mux over disjoint queue sets. Replaces the old
weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
batches per KB; Map/Reduce fan-out sized within a batch. New config knob
+ system setting + swagger docs.
Validate temporary download URLs against allowed DingTalk CDN hosts before fetching,
and accept pictureDownloadCode when downloadCode is absent so image messages ingest reliably.
The DingTalk adapter only ever classified incoming messages as text and
did not implement FileDownloader, so files sent to the bot were never
saved to the channel's configured knowledge base — unlike WeCom, which
worked. Users saw the file received and read, but never stored.
Parse file/picture msgtypes in both the webhook and stream paths (set
MessageTypeFile/Image, FileName, FileKey=downloadCode, robot_code), and
implement DownloadFile: exchange the downloadCode for a temporary URL via
robot/messageFiles/download, then fetch it. robotCode comes from the
callback (webhook) or the app client ID (stream). Pictures get a
MsgID-derived name, mirroring the WeCom adapter.