Commit Graph

441 Commits

Author SHA1 Message Date
Thibault Jaigu 35b92f73aa feat(provider): add Requesty as an OpenAI-compatible model provider
Requesty is an OpenAI-compatible LLM gateway that uses the same provider/model
naming convention as OpenRouter, so this mirrors the existing OpenRouter provider.

- internal/models/provider/requesty.go: RequestyProvider mirroring openrouter.go
  (base URL https://router.requesty.ai/v1, REQUESTY_API_KEY, RequiresAuth)
- provider.go: ProviderRequesty const + AllProviders() entry + DetectProvider case
- internal/types/model.go + client/model.go: ModelSourceRequesty
- provider_test.go: TestRequestyProviderValidation + DetectProvider case
- frontend model editor + i18n (en/zh/ko/ru) + docs/BUILTIN_MODELS.md provider list
- docs/swagger.{json,yaml} + docs.go: requesty enum value (hand-edited to match; these
  are normally swag-generated)

Verified: go build/vet/test pass; live call to https://router.requesty.ai/v1 succeeds.
Signed-off-by: Thibault Jaigu <thibault.jaigu@gmail.com>
2026-07-13 17:03:23 +08:00
wizardchen 07ba01ec93 feat(worker): replace aggregate asynq concurrency with per-pool governance
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
2026-07-13 15:51:06 +08:00
wizardchen 5eefa70e6f Enhance knowledge base configuration by normalizing and validating custom instruction fields. Implemented length checks for prompt instructions and added tests for validation logic. Updated GraphSettings to retain custom instructions when extraction is disabled. 2026-07-13 12:16:19 +08:00
wizardchen c1cddb4d4d Add instruction fields to knowledge base configuration 2026-07-13 12:16:19 +08:00
wizardchen 26f8afa3db fix(spans): expand knowledge span name column and truncate long wiki names
Wiki ingest builds postprocess.wiki.page[<slug>] subspan names that can exceed
varchar(64); widen the column to 255 and hash-suffix truncation so long slugs
still persist distinctly.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-13 10:35:44 +08:00
wizardchen d496d0fc73 feat(task-queue): split upstream workers into isolated core/enrichment/maintenance pools
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
2026-07-10 23:51:29 +08:00
wizardchen fe47ac1f96 feat(system): add runtime task queue observability dashboard for system admins
Expose live asynq queue depths and worker-pool concurrency via a new SystemAdmin API and settings UI, so operators can monitor parsing/wiki backlog without Redis CLI access.
2026-07-10 23:51:29 +08:00
wizardchen 20256287ec feat(models): add max concurrency setting for background tasks
Introduce a new `max_concurrency` parameter to the model configuration, allowing control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This feature is applicable to chat, embedding, and VLM models, with a value of 0 or undefined falling back to the global default. Update UI components and localization files to support this new setting, ensuring a seamless user experience.
2026-07-09 20:40:51 +08:00
wizardchen 63198c5d9f feat(models): cap background LLM concurrency per model
Background ingestion/enrichment stages (summary, question, graph,
multimodal) all target the same chat model, and asynq queue weights are
scheduling priority rather than throttling. A large upload could therefore
burst up to the entire worker pool of concurrent chat calls against a
single provider, unbounded across replicas, with no model-side rate limit.

Govern concurrency at the chat-client layer (the one place that sees every
task type), keyed by model ID:

- internal/models/limiter: Redis ZSET + Lua distributed semaphore with a
  lease heartbeat, expired-lease reclaim for crashed holders, and fail-open
  on any backend error or context cancellation.
- chat: concurrencyChat decorator (outermost in NewChat) acquires a
  per-model slot only for background tasks; interactive chat is never gated.
- router/task: asynq middleware tags every worker execution as background.
- container: wire the Redis limiter with WEKNORA_MODEL_MAX_CONCURRENCY
  (model.max_concurrency), default 8; enabled only when Redis is present.

Tested with miniredis: concurrency cap, expired-lease reclaim, per-key
isolation, and all fail-open paths (nil client, backend down, cancelled ctx).
2026-07-09 20:40:51 +08:00
wizardchen c8a4fd5ba3 fix(wiki): prevent silent work loss and cross-batch doc races
Two correctness fixes on the Phase 3 concurrent claim-based ingest path:

- Slug-lock timeout / reduce failure no longer silently drops work. When a
  per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
  an error, the page's read-modify-write never lands. Previously the owning
  document was still counted as successful and its pending row was deleted at
  trim time, permanently losing the contribution (finalize only rebuilds the
  index / cross-links, it does not re-run reduce). Such documents are now
  folded into failedOps: their rows stay queued, their finalizing slot is
  held, and requeueFailedOps' fail_count budget retries them (dead-lettering
  a permanently hot/broken slug).

- ClaimBatch now claims by dedup_key affinity so a document is never split
  across concurrent batches. limit counts distinct dedup_keys (documents),
  and ALL eligible rows sharing a key are claimed together. On Postgres the
  per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
  concurrent claimers disjoint key sets. This restores in-document ordering
  (e.g. ingest-then-retract collapses correctly) that the per-batch dedup
  could not guarantee once the exclusive per-KB lock was removed, preventing
  a stale ingest from resurrecting a retracted document.

Adds repository tests for same-key affinity and in-flight sibling rows.
2026-07-09 17:41:07 +08:00
wizardchen 70e2e85fa6 feat(wiki): dedicated worker pool with concurrent claim-based batching
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.

- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
  own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
  concurrency, default 16), split from the parse pool (NewParseAsynqServer).
  Both run the same mux over disjoint queue sets. Replaces the old
  weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
  and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
  without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
  dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
  batches per KB; Map/Reduce fan-out sized within a batch. New config knob
  + system setting + swagger docs.
2026-07-09 17:41:07 +08:00
wizardchen 6ac1703198 Simplify KB duplicate API and improve naming/docs coverage.
Drop client-supplied target_id, generate IDs server-side, localize duplicate
names from request language, document the endpoint, and expand handler/client tests.
2026-07-08 20:55:54 +08:00
wizardchen 58b79876ae Add knowledge base duplicate flow 2026-07-08 20:55:54 +08:00
wizardchen 77cf6a5789 fix(web-search): polish Keenable provider integration after #1946
Expose optional API key in settings UI, map longer API snippets into Content
for RAG, align maxResults fallback with other providers, sync Swagger enum,
and add follow-up unit tests.
2026-07-07 17:14:39 +08:00
Ilya Bogin 8cf9f1c882 feat: add Keenable as a configurable web search provider
Add Keenable Search alongside the existing DuckDuckGo / Bing / Google /
Tavily / Ollama / Baidu / SearXNG providers.

- internal/infrastructure/web_search/keenable.go: KeenableProvider
  implementing the WebSearchProvider interface, following the Tavily
  provider pattern. Keyless by default: with no API key it calls the
  public endpoint (/v1/search/public, rate-limited); a key switches to
  /v1/search and lifts the cap. Base URL is hardcoded (SSRF-safe), proxy
  is honored.
- Registered "keenable" in the container registry.
- Added the keenable provider type + type metadata (RequiresAPIKey=false)
  and service validation (no required key, like DuckDuckGo).
- Provider-specific empty-results diagnostic (keyless rate-limit hint).
- Frontend: extend the provider union type with 'keenable'.
- Unit tests: keyless vs keyed endpoint, attribution header, maxResults
  trim, result mapping, error status, name.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:58:26 +08:00
wizardchen 974231190c Add API key member and space capabilities 2026-07-07 12:00:13 +08:00
wizardchen d60614475e test(api-keys): add unit tests for tenant API key validation
Introduce unit tests for the tenant API key validation logic, ensuring that scoped keys require capabilities, while full-access keys are validated without them. The tests cover various scenarios, reinforcing the expected behavior of the API key validation function.
2026-07-07 12:00:13 +08:00
wizardchen 2c6f082062 Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
wizardchen 97c36f634a refactor(api-keys): remove API key reset endpoint and update documentation
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
2026-07-07 12:00:13 +08:00
wizardchen d2ffc1b973 refactor(api-keys): simplify authorization to a single-role gate
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.

- Add APIKeyGate as the single authority for API-key authorization: routes
  declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
  routes are denied by default. JWT principals short-circuit the gate and the
  JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
  is a pure downstream data filter); owner is a full-access tenant tier that
  unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
  writes on API-key encryption failure, deny KB-restricted keys on /files, and
  backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
  role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
  gin paths before comparison; add regression tests.
2026-07-07 12:00:13 +08:00
wizardchen 9a76a24ce6 refactor(api-keys): use role field and route-level API key guards
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
2026-07-07 12:00:13 +08:00
wizardchen 3f516d7f31 fix(api-keys): harden scoped key auth after code review
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
2026-07-07 12:00:13 +08:00
wizardchen 505bc7ddec feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
langcaiye 1708d6895a fix(search): apply tag filters to knowledge search 2026-07-06 23:30:33 +08:00
wizardchen 040504447e fix(security): block refresh tokens as bearer and sandbox MCP uploads
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
2026-07-02 17:08:17 +08:00
wizardchen f23c674bad fix(security): close six open GHSA findings from security triage
Harden share-link listing, password rotation, OIDC callback binding,
org invite search, knowledge batch mutations, and connector redirects.
2026-07-02 15:03:53 +08:00
wizardchen 1a264e09c1 fix(security): harden secret redaction follow-ups for PR #1884
Restore PreserveIfRedacted merge on tenant KV PUT so Admin settings saves
no longer overwrite stored secrets with placeholder values. Redact cross-
tenant tenant listings, tighten Viewer MCP/model responses, and align init
config frontend types with credentials metadata.
2026-07-02 14:41:42 +08:00
wizardchen 0b73080bb8 fix(security): redact integration secrets from viewer-accessible endpoints
Prevent low-privileged tenant members from reading raw API keys and
infrastructure credentials via tenant profile, KV config, model/MCP/web
search listings, and KB initialization config responses.
2026-07-02 14:41:42 +08:00
wizardchen c694fcf6cd test: address PR review follow-ups for OAuth and logger cleanup
Restore logger config via t.Cleanup after t.Setenv so Yuque token tests
do not leak global state, add unit tests for MCP OAuth non-interactive
context helpers and IM auth notice formatting, and fix EmbedChatCore
indentation.
2026-07-01 17:31:42 +08:00
wizardchen 8d40959f88 fix lint and test failures 2026-07-01 17:31:42 +08:00
wizardchen 4c962a9403 feat(agent): add configurable MCP OAuth wait timeout per agent
Allow agents to set how long in-conversation OAuth prompts wait before
auto-skipping, with UI in the agent editor and bounded gate behavior.
2026-07-01 14:37:37 +08:00
wizardchen 5074344c16 feat(embed): scope OAuth and sessions per visitor principal
Add embed visitor principals (X-Embed-Visitor), widen sessions.user_id,
embed MCP OAuth/approval routes, OAuth callback context detachment, and
CloneContext principal preservation for async QA streams.
2026-06-30 20:01:05 +08:00
wizardchen 97bca74a31 fix: harden API principal config after review
Add require_direct_header toggle, external user ID validation, JWT TTL
limits, SaveToken error handling, ru-RU i18n, API docs, and Go SDK methods.
2026-06-30 20:01:05 +08:00
wizardchen cb99736d72 feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
wizardchen 505195865a fix(mcp): trigger OAuth prompt during tool registration and dedupe helpers
OAuth authorization was only handled in MCPTool.Execute, so unauthenticated
services failed silently during RegisterMCPTools before any tool reached the
agent. Wire EventBus/session metadata into registration and reuse the same
getOrCreateMCPClientWithOAuthRetry path used at execution time.

Extract MCP OAuth session/wait/connect logic into mcp_oauth.go and remove the
duplicate waitForOAuthAndRetry + RegisterMCPContext layering in mcp_tool.go.
2026-06-29 23:22:28 +08:00
wizardchen 9fd7a7f522 feat(mcp): prompt and resume OAuth authorization mid-conversation
When an agent invokes an OAuth-enabled MCP service that the current user has
not authorized yet, the tool call now pauses and surfaces an "Authorize" card
in the chat instead of failing with an opaque error. After the user completes
the per-user OAuth popup, the backend verifies a token exists and resumes the
paused tool call automatically.

Backend:
- Add mcp_oauth_required / mcp_oauth_resolved events and response types.
- Add OAuthPendingRequest + Gate.RequestOAuthAndWait, reusing the existing
  approval waiter, Resolve, Redis cross-instance fan-out and timeout.
- MCPTool.Execute pauses on an authorization-required error for OAuth services
  and retries once (with a fresh ctx and recycled connection) after the user
  authorizes.
- Forward the new events to the SSE stream.
- Add ResolveMCPOAuth handler + route; it verifies authorization before
  unblocking so a premature resolve cannot resume into another failure.

Frontend:
- Add McpOAuthCard with an Authorize button reusing the popup + status-poll
  flow, then resolve the paused run.
- Handle the new stream events (dedup multiple pending cards per service).
- Add resolveMCPOAuth API client and i18n keys (zh/en/ko/ru).
2026-06-29 23:22:28 +08:00
wizardchen f2686e9e39 feat(mcp): drive auth by AuthType and auto-detect OAuth on test
Select the auth header for the configured strategy only, instead of
emitting both X-API-Key and Authorization whenever the fields happened
to be set (which double-authed after switching strategies). AuthType=""
keeps the legacy infer-from-fields behavior for pre-existing rows.

Add a configurable api_key_header (non-secret, defaults to X-API-Key) so
services expecting the key in a different header work without plaintext
custom headers.

On connection test, upgrade a bare 401 that advertises RFC 9728
protected-resource metadata into an explicit OAuthRequired signal so the
UI can guide the user to switch to OAuth 2.0. Surface the server's
description from the initialize handshake in the test result.
2026-06-29 22:37:52 +08:00
wizardchen 049f85139d feat(chat): add @Skill/@MCP mentions with scoped agent runtime
Wire per-turn @mentions for skills, MCP services, tags, and files through
QA handlers into agent config, runtime_context/must_use user blocks, and
tool registration. Fix tag-only scopes dropping KB tools, refresh suggested
questions on mention changes, and strengthen MCP must_use so @selected
services are invoked before local KB search.
2026-06-29 18:09:44 +08:00
Anky c007a897cf feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge

- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)

* feat(knowledge): tag edit dialog with dynamic overflow and inline creation

- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder

* fix: exclude soft-deleted knowledge from tag reference counts

CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.

* fix: refresh tag counts after async knowledge deletion completes

Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.

---------

Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-26 00:18:48 +08:00
wizardchen 97b7bb9890 fix(knowledge): harden batch reparse UX and async processing
Move batch reparse work to an asynq task to avoid HTTP timeouts, add i18n
for all locales, guard in-flight documents, and prevent concurrent delete/reparse.
2026-06-25 23:57:54 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
wizardchen 7855f36799 fix(model): block deletion when model is referenced by KB or agent
Reject model delete with HTTP 400 when the current tenant still has
knowledge bases or custom agents bound to the model ID.
2026-06-23 20:36:16 +08:00
wizardchen bc9ad3aca0 fix(qa): wire agent thinking toggle and show reasoning in RAG timeline
Default unset agent thinking to explicit false so thinking_type models
receive disabled/enabled wire params. Render reasoning inside the quick-answer
pipeline timeline only when thinking events arrive, avoiding a flash of the
thinking step when thinking mode is off.
2026-06-23 20:17:19 +08:00
wizardchen be26563f86 fix(agent): remove unused RunnableByViewer gate blocking viewer chat
The runnable_by_viewer field was never populated for built-in agents,
so TenantRoleViewer users always hit 403 on agent-chat. Drop the field
and RBAC check so agent-chat matches knowledge-chat access for viewers.
2026-06-23 18:21:23 +08:00
wizardchen 3cb0e906a7 fix(datasource): harden RSS connector and editor UX after review
Scope auth headers to feed fetches, use feed-scoped external IDs and
content-based incremental fingerprints, tolerate partial feed failures,
store feed URLs in settings, and fix credential configured detection.
2026-06-23 17:30:38 +08:00
wizardchen e40ad802d7 feat(datasource): add RSS feed connector with full-text ingestion
Enable private RSS feeds as a knowledge base data source with custom
headers, article body fetching, and Markdown output for incremental sync.
2026-06-23 17:30:38 +08:00
langcaiye 3eb49da746 feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
wizardchen 7f3f219f58 feat(mcp): add OAuth2 authorization for MCP services
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.

- Persist per-user access/refresh tokens and per-service DCR client
  credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
  in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
  the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
  hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
  authorization status/actions, and render the connection test result
  inline in the editor drawer instead of a stacked centered dialog
2026-06-22 20:19:31 +08:00
wizardchen 5d0d317a36 feat(datasource): reveal pre-existing selections in lazily-loaded picker
When editing a data source whose tree loads one level at a time (Feishu
wiki), a saved deep-node selection was invisible: only top-level spaces
were listed, so the selected node could not be seen and its ancestor
space rendered as unchecked.

- Add Connector.ResolveResourceAncestors: given selected resource IDs,
  return the ancestor IDs that must be expanded to reveal them. Feishu
  walks up via GetWikiNode (O(depth) per selection, no full-tree walk);
  Notion/Yuque return empty.
- Fix wikiNode.ParentNodeID JSON tag: the Feishu API field is
  parent_node_token, not parent_node_id, so GetWikiNode can return the
  parent and the ancestor walk works in production.
- Expose POST /datasource/{id}/resource-ancestors (service + handler +
  route) and a matching frontend API client.
- Edit-mode picker reveals hidden selections by expanding and lazily
  loading each ancestor level, so selections are visible and checked.
- Gate child fetching on treeFullyLoaded instead of a partial-children
  heuristic, avoiding a redundant request for full-tree connectors.
2026-06-22 19:24:08 +08:00
wizardchen d8f13ada4a fix(datasource): lazy-load Feishu wiki resources to avoid listing timeout (#1672)
Listing resources for a Feishu data source eagerly walked the entire wiki
tree (one API call per node across every space), which exceeded the request
timeout for large knowledge bases. ListResources now takes a parentID and
returns a single level at a time: empty lists the wiki spaces, a space/node
ID lists only its direct children. The frontend fetches children on demand
when a node is expanded. Notion (full tree in one call) and Yuque (flat)
keep their existing behavior by ignoring non-empty parentID.

Refs: https://github.com/Tencent/WeKnora/issues/1672
2026-06-22 17:53:30 +08:00