Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
Operators can clear all finally-failed tasks in one queue from the runtime
drawer, with audit logging and scoped API access. Also localize the
chat_attachment queue label in the runtime queue details table.
Prevent concurrent refresh-token rotation across instances, distinguish
expired-but-refreshable tokens from active authorization, and bind OAuth
popup polling to a per-attempt ID so stale tokens cannot satisfy new consent.
- Updated the listFAQEntries function to support multiple tag IDs for filtering.
- Modified the frontend components to accommodate the new tag_ids parameter.
- Enhanced the KbTagManageDrawer and FAQEntryManager components to reflect changes in tag handling.
- Improved API documentation to clarify the new tag filtering capabilities.
This commit enhances the flexibility of FAQ entry retrieval, allowing users to filter by multiple tags for more precise results.
- Updated the exportFAQEntries function to support exporting FAQ entries in both CSV and JSON formats.
- Modified the frontend to include options for selecting the export format.
- Enhanced localization files to reflect the new export options in English, Russian, and Chinese.
- Implemented new tests to validate the export functionality for both formats.
This commit improves the flexibility of FAQ exports, allowing users to choose their preferred format for better integration with other tools.
- Added new fields for tracking added, merged, and partially failed entries during FAQ imports.
- Updated the import result statistics to include detailed counts for added and merged entries.
- Enhanced the UI to display progress hints and improved import result summaries.
- Implemented a new method for saving chunks in a single transaction to improve performance.
This commit improves the visibility and accuracy of FAQ import results, providing better user feedback and tracking capabilities.
- Added new actions for FAQ import lifecycle: `faq.import_started`, `faq.import_completed`, and `faq.import_failed`.
- Updated activity details to include success, failure, and skipped counts for FAQ imports.
- Enhanced UI to display additional activity information, including filtering options and clearer summaries.
- Implemented tests to validate the new FAQ import outcomes and activity details.
This commit improves the tracking and visibility of FAQ import activities within the knowledge base, ensuring better audit trails and user feedback.
Automatically remove wiki folders that become empty when knowledge is
retracted, using the durable finalize lane so pruning waits until ingest
drains. Repository DeleteFolder now atomically checks emptiness to avoid
races with concurrent page or child-folder writes.
- Added AminusB method to the knowledgeRepository, which returns IDs of knowledge in A that have no counterpart in B, comparing by file_hash as a multiset.
- This implementation addresses issues with handling duplicate entries and NULL file_hash values, ensuring accurate synchronization between knowledge bases.
- Updated the KnowledgeRepository interface to reflect the new functionality and improved documentation for clarity.
Introduce platform API keys with capability-based access to tenant management,
system settings, runtime queues, and audit logs, plus admin UI for key lifecycle
and system audit log viewing.
Large Feishu wikis stopped syncing after the first batch (Tencent/WeKnora#2136:
5000+ docs, only ~150 ingested; #1262: a child-list failure aborted the run).
Three compounding causes, fixed in layers:
1. Rate-limit / transient failures had no retry. Feishu's wiki/drive/export
APIs are aggressively rate limited; a thousand-doc sync issues tens of
thousands of calls, and a single 429/5xx burst failed whole swathes of docs.
Add exponential backoff honouring Retry-After to doRequest (JSON) and
downloadRawBytes (files): 429 up to 3x, 5xx once, transport errors back off.
2. The batch path fetched the entire tree into memory before ingesting, so a
sync that hit the 2h task timeout lost all progress and restarted from
scratch, and a transient per-document export failure advanced the cursor and
skipped that doc permanently. Add an optional StreamingConnector interface,
implemented for Feishu: FetchStream interleaves fetch -> ingest -> checkpoint,
bounding memory to one item and persisting a per-node edit-time cursor at
page boundaries. The service dispatches to it via type assertion; connectors
without it fall back to FetchAll / FetchIncremental unchanged (non-breaking).
A resumed/incremental run skips nodes already recorded at their current edit
time (converges), a failed fetch does not advance the cursor (retried next
run instead of dropped), and partial listing failures emit error items and
carry prior edit times forward instead of aborting the sync.
3. Failures and skips were invisible. Unsupported node types (mindnote/slides/…)
were skipped with no item, error or log ("13 docs synced only 3"). FetchAll
now tallies discovered/fetched/failed/skipped per resource with a per-type
skip breakdown and a progress line every 100 nodes. The backend already
persisted per-item errors in SyncLog.result.errors but the frontend type
omitted the field; declare `result` on SyncLog and render up to 50 failed
documents in the sync-log drawer (#1262). i18n for zh-CN/en-US/ko-KR/ru-RU.
result.errors is a jsonb column shipped in every list response, so it is
capped at 100 samples server-side (recordSyncError); the exact failure total
comes from the bounded items_failed counter, which the drawer uses for its
count and "N more" line so a sync that fails thousands of docs stays bounded.
Each sample is a structured, localisable SyncItemError instead of a raw
"title: ...api error: status=500 body={json}" dump that repeated the title
and leaked status/body/log_id. The connector (feishuFailure) classifies the
error into a stable i18n code (+ the feishu code as a param); the service
builds {title, code, params, message}; the drawer localises via vue-i18n to
the viewer's language (zh/en/ko/ru), falling back to message. A custom
UnmarshalJSON keeps old string-form logs readable. The banner is localised
from items_failed too. Raw errors stay in the server logs. Follows the
two-tier norm of Airbyte/Fivetran/Onyx: human, actionable, localised
summaries in the UI; raw detail in logs. Transient errors say "will retry"
(the cursor is retained); auth/permission errors point at the fix instead.
Tests: client_retry_test (backoff), tally_test (accounting), connector_stream /
datasource_stream (streaming dispatch + checkpoint), connector_convergence
(timeout + transient-failure resume converges to full coverage; proven to catch
#2136 via bug injection), datasource_result_cap (error-sample cap + structured
localisable SyncItemError, no raw body), connector_error_reason (error→i18n-code
classification), sync_item_error (legacy-string + structured back-compat) — all
in the normal suite. connector_realapi / connector_seed are an opt-in real-Feishu
harness gated by the feishu_integration build tag AND env vars, never run in CI.
Multiple GORM tag options must be separated by ';', not ','. The comma was
parsed as part of the type value, producing malformed DDL that causes
AutoMigrate(&Message{}) to fail on SQLite with a syntax error.
Restrict channel-managed sessions (API keys, IM, embed) to tenant admin
in the Web console for list/read paths, hide channel sidebar buckets from
non-admins, and add CountSessionsBySource for aggregate stats without
exposing session rows to viewers.
Add GetOwnedSession method to enforce strict ownership checks for session mutations, ensuring tenant admins can read but not modify API-key sessions. Update relevant handlers to utilize this method for operations that alter session data, maintaining security and integrity of API-key sessions.
Let tenant admins list and open API-key sessions from the Web console while
keeping per-key isolation for everyone else. Pin the session source filter so
it no longer jumps when switching buckets, and resolve session origin on read
so deep links stay in sync with the active folder after refresh.
Migrate WeKnora's Langfuse tracing from the legacy /api/public/ingestion
batch API to the OpenTelemetry Go SDK emitting OTLP/HTTP to
/api/public/otel/v1/traces — the Langfuse v3+ standard ingestion protocol.
Why: Langfuse v3 deprecated the legacy ingestion API in favor of OTLP, and
OTel-only backends (e.g. the LiteFuse Doris fork) dropped it entirely, so the
legacy client was incompatible. Migrating to the OTel SDK also adds W3C
traceparent cross-service propagation: an upstream caller's trace id is
inherited so its run and the WeKnora call land under one trace in the UI.
What:
- Manager owns an OTel TracerProvider backed by an otlptracehttp exporter
(Basic auth + the x-langfuse-ingestion-version gate header). Spans are
buffered by BatchSpanProcessor and exported complete on End, so a long
trace no longer fragments across flush batches.
- *Trace/*Span/*Generation wrap OTel spans; the public API (StartTrace/
Span/Generation, handles, InjectTracing, asynq/gin middleware, Config,
TokenUsage) is unchanged, so callers (agent/chat/handler/scheduler/asynqdl)
need no edits.
- W3C traceparent: GinMiddleware extracts traceparent from incoming requests;
AsynqMiddleware + InjectTracing propagate it across the asynq payload.
types.TracingContext gains a LangfuseTraceparent field.
- logger.CloneContext preserves the active OTel span across context rebuilds
so child spans (agent engine, summary generation) stitch to the HTTP trace
instead of orphaning into separate traces.
- StartSpan/StartGeneration re-establish the parent from the *Trace handle
when the ctx lost the active OTel span, covering background/derived contexts.
Compatibility: works with Langfuse v3+ (cloud / self-hosted / the bundled
docker-compose profile) and LiteFuse. Drops Langfuse v2 (legacy-only) support.
Uses the OTel Go SDK already (indirectly) in go.mod, promoting
otlptracehttp to direct; no third-party langfuse-go library.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chat attachment parser rules belong on agents; drop the duplicate workspace settings section while preserving legacy tenant rules when engine config is updated without that field.
Lark and Feishu are the same product deployed on two isolated clouds
(open.larksuite.com and open.feishu.cn). WeKnora touches both from two
places — the IM bot and the Wiki data-source connector — and this brings
Lark to parity across both. Neither API surface differs, so rather than
duplicate either implementation, each is parametrized by a Region and
registered twice.
IM (internal/im/feishu):
Every API call routes through a.api(), which prefixes the region's
host, and the WebSocket client passes larkws.WithDomain — the SDK
defaults to open.feishu.cn, so Lark long connections would otherwise
fail to authenticate.
Data source (internal/datasource/connector/feishu):
Lark was half-supported: the client honoured a base_url override and a
LarkBaseURL constant existed, but the connector registered only as
"feishu" and the UI never exposed base_url, so Lark wikis could not be
configured at all. Lark is now its own connector type. base_url is kept
as an explicit override so data sources that pointed a "feishu"
connector at open.larksuite.com keep working.
Bugs found and fixed along the way:
- Wiki resource links were hardcoded to https://feishu.cn/wiki/{token} in
four places, so a Lark wiki linked into the wrong cloud entirely.
- Bot identity is now platform-prefixed (feishu:cli_x vs lark:cli_x).
BotIdentity carries a unique index, so without this a Lark channel
would be rejected as a duplicate of an unrelated Feishu bot sharing an
app_id.
- The card image cache is now keyed by app_id. image_keys are issued per
app on one cloud; a Feishu key handed to a Lark app fails the card
update with code=200570.
- The datasource doc claimed Lark "adapts automatically" — it could not,
since the UI had no way to reach base_url.
Permissions are where the clouds genuinely diverge. Identical APIs do not
mean identical permission catalogues: aily:file:read, aily:file:write and
corehr:file:download exist on Feishu but not on Lark, so pasting Feishu's
scope JSON into Lark fails to import. That JSON is also over-broad (it
covers products WeKnora never calls) and simultaneously incomplete (it
omits the drive:*/docx:* scopes the connector needs). Lark therefore gets
its own list, derived from the 17 endpoints the code actually calls and
checked identifier-by-identifier against Lark's own API reference, plus a
full Feishu→Lark mapping table marking each scope verified, unverified,
or absent. Both features use tenant_access_token only — no user_access_
token exists anywhere in the repo — so the user dimension stays empty.
Known limitation: Region.ThinkingText makes Lark's initial card
placeholder English, but once streaming starts the think-block headers
come from the shared MarkdownThinkStyle and are Chinese on every platform
(Lark, Slack, Telegram, Mattermost alike). Fixing that means threading a
locale through FormatIMDisplayContent, which has no platform context — a
cross-platform change, not a Lark one. Documented as such.
The channel-creation 400 message is now derived from validIMPlatforms so
the list cannot drift as platforms are added.
Tests: region invariants and per-region URL construction on both sides,
the Feishu/Lark bot-identity split, platform-to-channel mapping, base_url
override precedence, an end-to-end SendReply against a fake Open Platform
(token fetch, reply, thread-unsupported fallback, hard-error surfacing),
and a ListResources drive asserting wiki links never leak the other
cloud's host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the SuggestedQuestionsRequest to include TagScopes for better filtering of knowledge items based on their parent knowledge bases. Modify related API documentation and frontend interfaces to accommodate the new structure. Ensure backend services handle the new tag scope logic correctly, including tests for the updated functionality.
Move citation protocol injection to the system layer so custom prompts no
longer own citation syntax. Agents can disable inline citations while source
aliases still encode and decode tool arguments correctly.
Implement a maximum limit of 5 for the total number of images and attachments that can be uploaded in a chat session. Introduce a warning message for users when the limit is exceeded. Additionally, add functionality to clear pending uploads when switching sessions to prevent errors related to stale document IDs. Update localization files to include new warning messages in English, Korean, Russian, and Chinese.
Add functionality to normalize temporary attachment IDs, ensuring deduplication and enforcing a maximum limit of 5 attachments per message. Update related handlers and tests to support this new behavior, enhancing the robustness of attachment management in chat sessions.
Let users upload documents in chat without ingesting them into a knowledge base: files are parsed in the background, referenced by ID on send, and expire after a configurable TTL.
Introduce resource://<handle> as the stable application identity for stored
objects, decoupling API responses, persisted rich text, and LLM prompts from
physical storage paths. Adds a resource catalog service (register/resolve/bind,
revocable expiring access grants), a request-local alias registry that compacts
resource references while in the LLM context and restores them (including across
streaming chunks), plus router, IM, chat-pipeline, and multimodal integration.
Includes schema migrations (resources, resource_bindings, resource_access_grants)
for sqlite, and hardens URL/output security handling.
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
Replace offset-based page numbers with opaque Redis-backed cursors so
scheduled/retry tasks sort by next run time and live lists stay stable
when queue state changes between requests.
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
Allow platform system administrators to update builtin model parameters
and credentials from the management UI while keeping secrets masked in API
responses. UI edits clear YAML ownership so startup reconciliation no
longer overwrites runtime overrides.
Enable agents to configure starter prompts and async follow-up questions after completed answers, with durable caching, analytics events, and chat/embed UI integration.
Let system administrators reset another user's local password from the settings UI, revoke all active sessions, and record the action in the audit log.
Drop the standalone OIDC_AUTH_DEFAULT_TENANT_MODE / oidc_auth.default_tenant_mode
knob so OIDC auto-provisioning and local password registration share the single
auth.default_tenant_mode policy (resolved by the callback handler). Repair
tenantless sessions by falling back to the first active membership on the login
happy path and stale-preference paths, allow tenantless PUT /auth/me/preferences,
clear the tenant snapshot on /auth/me responses without a tenant, and default the
frontend canCreateTenant capability to fail-closed until /auth/me confirms it.
Allow deployments to register users without auto-creating a personal tenant, expose creation policy via /auth/me, and guide tenantless accounts through a workspace onboarding flow.
Enforce tenant ownership in ListSharesByAgent so full-access API keys cannot enumerate cross-tenant shares, and allow full-access keys to create KBs and manage share routes consistently with other tenant-wide operations.
Requesty is an OpenAI-compatible LLM gateway that uses the same provider/model
naming convention as OpenRouter, so this mirrors the existing OpenRouter provider.
- internal/models/provider/requesty.go: RequestyProvider mirroring openrouter.go
(base URL https://router.requesty.ai/v1, REQUESTY_API_KEY, RequiresAuth)
- provider.go: ProviderRequesty const + AllProviders() entry + DetectProvider case
- internal/types/model.go + client/model.go: ModelSourceRequesty
- provider_test.go: TestRequestyProviderValidation + DetectProvider case
- frontend model editor + i18n (en/zh/ko/ru) + docs/BUILTIN_MODELS.md provider list
- docs/swagger.{json,yaml} + docs.go: requesty enum value (hand-edited to match; these
are normally swag-generated)
Verified: go build/vet/test pass; live call to https://router.requesty.ai/v1 succeeds.
Signed-off-by: Thibault Jaigu <thibault.jaigu@gmail.com>
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
Wiki ingest builds postprocess.wiki.page[<slug>] subspan names that can exceed
varchar(64); widen the column to 255 and hash-suffix truncation so long slugs
still persist distinctly.
Co-authored-by: Cursor <cursoragent@cursor.com>
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
Expose live asynq queue depths and worker-pool concurrency via a new SystemAdmin API and settings UI, so operators can monitor parsing/wiki backlog without Redis CLI access.
Introduce a new `max_concurrency` parameter to the model configuration, allowing control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This feature is applicable to chat, embedding, and VLM models, with a value of 0 or undefined falling back to the global default. Update UI components and localization files to support this new setting, ensuring a seamless user experience.
Background ingestion/enrichment stages (summary, question, graph,
multimodal) all target the same chat model, and asynq queue weights are
scheduling priority rather than throttling. A large upload could therefore
burst up to the entire worker pool of concurrent chat calls against a
single provider, unbounded across replicas, with no model-side rate limit.
Govern concurrency at the chat-client layer (the one place that sees every
task type), keyed by model ID:
- internal/models/limiter: Redis ZSET + Lua distributed semaphore with a
lease heartbeat, expired-lease reclaim for crashed holders, and fail-open
on any backend error or context cancellation.
- chat: concurrencyChat decorator (outermost in NewChat) acquires a
per-model slot only for background tasks; interactive chat is never gated.
- router/task: asynq middleware tags every worker execution as background.
- container: wire the Redis limiter with WEKNORA_MODEL_MAX_CONCURRENCY
(model.max_concurrency), default 8; enabled only when Redis is present.
Tested with miniredis: concurrency cap, expired-lease reclaim, per-key
isolation, and all fail-open paths (nil client, backend down, cancelled ctx).
wizardchen