Two correctness fixes on the Phase 3 concurrent claim-based ingest path:
- Slug-lock timeout / reduce failure no longer silently drops work. When a
per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
an error, the page's read-modify-write never lands. Previously the owning
document was still counted as successful and its pending row was deleted at
trim time, permanently losing the contribution (finalize only rebuilds the
index / cross-links, it does not re-run reduce). Such documents are now
folded into failedOps: their rows stay queued, their finalizing slot is
held, and requeueFailedOps' fail_count budget retries them (dead-lettering
a permanently hot/broken slug).
- ClaimBatch now claims by dedup_key affinity so a document is never split
across concurrent batches. limit counts distinct dedup_keys (documents),
and ALL eligible rows sharing a key are claimed together. On Postgres the
per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
concurrent claimers disjoint key sets. This restores in-document ordering
(e.g. ingest-then-retract collapses correctly) that the per-batch dedup
could not guarantee once the exclusive per-KB lock was removed, preventing
a stale ingest from resurrecting a retracted document.
Adds repository tests for same-key affinity and in-flight sibling rows.
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.
- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
concurrency, default 16), split from the parse pool (NewParseAsynqServer).
Both run the same mux over disjoint queue sets. Replaces the old
weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
batches per KB; Map/Reduce fan-out sized within a batch. New config knob
+ system setting + swagger docs.
Drop client-supplied target_id, generate IDs server-side, localize duplicate
names from request language, document the endpoint, and expand handler/client tests.
Expose optional API key in settings UI, map longer API snippets into Content
for RAG, align maxResults fallback with other providers, sync Swagger enum,
and add follow-up unit tests.
Add Keenable Search alongside the existing DuckDuckGo / Bing / Google /
Tavily / Ollama / Baidu / SearXNG providers.
- internal/infrastructure/web_search/keenable.go: KeenableProvider
implementing the WebSearchProvider interface, following the Tavily
provider pattern. Keyless by default: with no API key it calls the
public endpoint (/v1/search/public, rate-limited); a key switches to
/v1/search and lifts the cap. Base URL is hardcoded (SSRF-safe), proxy
is honored.
- Registered "keenable" in the container registry.
- Added the keenable provider type + type metadata (RequiresAPIKey=false)
and service validation (no required key, like DuckDuckGo).
- Provider-specific empty-results diagnostic (keyless rate-limit hint).
- Frontend: extend the provider union type with 'keenable'.
- Unit tests: keyless vs keyed endpoint, attribution header, maxResults
trim, result mapping, error status, name.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce unit tests for the tenant API key validation logic, ensuring that scoped keys require capabilities, while full-access keys are validated without them. The tests cover various scenarios, reinforcing the expected behavior of the API key validation function.
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.
- Add APIKeyGate as the single authority for API-key authorization: routes
declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
routes are denied by default. JWT principals short-circuit the gate and the
JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
is a pure downstream data filter); owner is a full-access tenant tier that
unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
writes on API-key encryption failure, deny KB-restricted keys on /files, and
backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
gin paths before comparison; add regression tests.
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
Prevent low-privileged tenant members from reading raw API keys and
infrastructure credentials via tenant profile, KV config, model/MCP/web
search listings, and KB initialization config responses.
Restore logger config via t.Cleanup after t.Setenv so Yuque token tests
do not leak global state, add unit tests for MCP OAuth non-interactive
context helpers and IM auth notice formatting, and fix EmbedChatCore
indentation.
OAuth authorization was only handled in MCPTool.Execute, so unauthenticated
services failed silently during RegisterMCPTools before any tool reached the
agent. Wire EventBus/session metadata into registration and reuse the same
getOrCreateMCPClientWithOAuthRetry path used at execution time.
Extract MCP OAuth session/wait/connect logic into mcp_oauth.go and remove the
duplicate waitForOAuthAndRetry + RegisterMCPContext layering in mcp_tool.go.
When an agent invokes an OAuth-enabled MCP service that the current user has
not authorized yet, the tool call now pauses and surfaces an "Authorize" card
in the chat instead of failing with an opaque error. After the user completes
the per-user OAuth popup, the backend verifies a token exists and resumes the
paused tool call automatically.
Backend:
- Add mcp_oauth_required / mcp_oauth_resolved events and response types.
- Add OAuthPendingRequest + Gate.RequestOAuthAndWait, reusing the existing
approval waiter, Resolve, Redis cross-instance fan-out and timeout.
- MCPTool.Execute pauses on an authorization-required error for OAuth services
and retries once (with a fresh ctx and recycled connection) after the user
authorizes.
- Forward the new events to the SSE stream.
- Add ResolveMCPOAuth handler + route; it verifies authorization before
unblocking so a premature resolve cannot resume into another failure.
Frontend:
- Add McpOAuthCard with an Authorize button reusing the popup + status-poll
flow, then resolve the paused run.
- Handle the new stream events (dedup multiple pending cards per service).
- Add resolveMCPOAuth API client and i18n keys (zh/en/ko/ru).
Select the auth header for the configured strategy only, instead of
emitting both X-API-Key and Authorization whenever the fields happened
to be set (which double-authed after switching strategies). AuthType=""
keeps the legacy infer-from-fields behavior for pre-existing rows.
Add a configurable api_key_header (non-secret, defaults to X-API-Key) so
services expecting the key in a different header work without plaintext
custom headers.
On connection test, upgrade a bare 401 that advertises RFC 9728
protected-resource metadata into an explicit OAuthRequired signal so the
UI can guide the user to switch to OAuth 2.0. Surface the server's
description from the initialize handshake in the test result.
Wire per-turn @mentions for skills, MCP services, tags, and files through
QA handlers into agent config, runtime_context/must_use user blocks, and
tool registration. Fix tag-only scopes dropping KB tools, refresh suggested
questions on mention changes, and strengthen MCP must_use so @selected
services are invoked before local KB search.
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
Move batch reparse work to an asynq task to avoid HTTP timeouts, add i18n
for all locales, guard in-flight documents, and prevent concurrent delete/reparse.
Default unset agent thinking to explicit false so thinking_type models
receive disabled/enabled wire params. Render reasoning inside the quick-answer
pipeline timeline only when thinking events arrive, avoiding a flash of the
thinking step when thinking mode is off.
The runnable_by_viewer field was never populated for built-in agents,
so TenantRoleViewer users always hit 403 on agent-chat. Drop the field
and RBAC check so agent-chat matches knowledge-chat access for viewers.
Scope auth headers to feed fetches, use feed-scoped external IDs and
content-based incremental fingerprints, tolerate partial feed failures,
store feed URLs in settings, and fix credential configured detection.
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.
- Persist per-user access/refresh tokens and per-service DCR client
credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
authorization status/actions, and render the connection test result
inline in the editor drawer instead of a stacked centered dialog
When editing a data source whose tree loads one level at a time (Feishu
wiki), a saved deep-node selection was invisible: only top-level spaces
were listed, so the selected node could not be seen and its ancestor
space rendered as unchecked.
- Add Connector.ResolveResourceAncestors: given selected resource IDs,
return the ancestor IDs that must be expanded to reveal them. Feishu
walks up via GetWikiNode (O(depth) per selection, no full-tree walk);
Notion/Yuque return empty.
- Fix wikiNode.ParentNodeID JSON tag: the Feishu API field is
parent_node_token, not parent_node_id, so GetWikiNode can return the
parent and the ancestor walk works in production.
- Expose POST /datasource/{id}/resource-ancestors (service + handler +
route) and a matching frontend API client.
- Edit-mode picker reveals hidden selections by expanding and lazily
loading each ancestor level, so selections are visible and checked.
- Gate child fetching on treeFullyLoaded instead of a partial-children
heuristic, avoiding a redundant request for full-tree connectors.
Listing resources for a Feishu data source eagerly walked the entire wiki
tree (one API call per node across every space), which exceeded the request
timeout for large knowledge bases. ListResources now takes a parentID and
returns a single level at a time: empty lists the wiki spaces, a space/node
ID lists only its direct children. The frontend fetches children on demand
when a node is expanded. Notion (full tree in one call) and Yuque (flat)
keep their existing behavior by ignoring non-empty parentID.
Refs: https://github.com/Tencent/WeKnora/issues/1672
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.
Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.
Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.
Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.