Commit Graph

192 Commits

Author SHA1 Message Date
wizardchen 70e2e85fa6 feat(wiki): dedicated worker pool with concurrent claim-based batching
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.

- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
  own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
  concurrency, default 16), split from the parse pool (NewParseAsynqServer).
  Both run the same mux over disjoint queue sets. Replaces the old
  weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
  and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
  without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
  dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
  batches per KB; Map/Reduce fan-out sized within a batch. New config knob
  + system setting + swagger docs.
2026-07-09 17:41:07 +08:00
wizardchen 58b79876ae Add knowledge base duplicate flow 2026-07-08 20:55:54 +08:00
wizardchen 974231190c Add API key member and space capabilities 2026-07-07 12:00:13 +08:00
wizardchen d60614475e test(api-keys): add unit tests for tenant API key validation
Introduce unit tests for the tenant API key validation logic, ensuring that scoped keys require capabilities, while full-access keys are validated without them. The tests cover various scenarios, reinforcing the expected behavior of the API key validation function.
2026-07-07 12:00:13 +08:00
wizardchen 2c6f082062 Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
wizardchen b5cc7d2a28 feat(api-keys): implement throttled last_used_at updates for API key authentication
Enhance the tenant API key service by introducing asynchronous throttling for the last_used_at updates during API key authentication. This change reduces database write frequency under high request rates, improving performance. Additionally, update tests to validate the throttling behavior, ensuring that last_used_at is only updated once per key within the defined interval. This implementation aligns with the goal of optimizing API key management and maintaining system efficiency.
2026-07-07 12:00:13 +08:00
wizardchen 97c36f634a refactor(api-keys): remove API key reset endpoint and update documentation
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
2026-07-07 12:00:13 +08:00
wizardchen 1d1e1e3d25 refactor(api-keys): update role handling and enhance API key restrictions
Change the default role for tenant API keys from 'admin' to 'owner' to align with the new authorization model. Update related tests to reflect this change, ensuring that API keys with the owner role are correctly denied access to management paths and system-admin routes. Introduce new tests to validate that API keys are properly restricted from accessing sensitive endpoints, reinforcing the security measures in place.
2026-07-07 12:00:13 +08:00
wizardchen a6759d61b5 feat(api-keys): implement DenyAPIKeyPrincipal middleware and related tests
Add DenyAPIKeyPrincipal middleware to explicitly block API-key principals from accessing certain routes, ensuring security for endpoints outside the /api/v1 group. Introduce tests to validate that API keys are correctly denied access while allowing JWT sessions to pass through. This enhances the overall authorization model by reinforcing the default-deny approach for API keys.
2026-07-07 12:00:13 +08:00
wizardchen d2ffc1b973 refactor(api-keys): simplify authorization to a single-role gate
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.

- Add APIKeyGate as the single authority for API-key authorization: routes
  declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
  routes are denied by default. JWT principals short-circuit the gate and the
  JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
  is a pure downstream data filter); owner is a full-access tenant tier that
  unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
  writes on API-key encryption failure, deny KB-restricted keys on /files, and
  backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
  role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
  gin paths before comparison; add regression tests.
2026-07-07 12:00:13 +08:00
wizardchen 9a76a24ce6 refactor(api-keys): use role field and route-level API key guards
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
2026-07-07 12:00:13 +08:00
wizardchen 3f516d7f31 fix(api-keys): harden scoped key auth after code review
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
2026-07-07 12:00:13 +08:00
wizardchen 505bc7ddec feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
qilifan d5b8d2f886 feat(redis): support TLS for Redis connections (#1930)
[Feature]: Support TLS for Redis connections
2026-07-06 23:02:02 +08:00
wizardchen 663d13eb58 fix(security): harden outbound requests and file previews 2026-07-02 19:15:46 +08:00
wizardchen a17a2250ed fix(security): enforce KB access on wiki reads and related IDOR paths
Gate wiki GET routes and initialization config reads with KBAccessRead,
and bind async task progress endpoints to the caller tenant via task ID
parsing. Closes cross-tenant read paths reported in GHSA-ph8f-4qwx-89rr
and related initialization/progress leaks.
2026-07-02 12:06:22 +08:00
wizardchen ee1014f87c feat(frontend): add API integration playground 2026-07-01 11:14:51 +08:00
wizardchen 5074344c16 feat(embed): scope OAuth and sessions per visitor principal
Add embed visitor principals (X-Embed-Visitor), widen sessions.user_id,
embed MCP OAuth/approval routes, OAuth callback context detachment, and
CloneContext principal preservation for async QA streams.
2026-06-30 20:01:05 +08:00
wizardchen cb99736d72 feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
wizardchen b5a1c88f75 fix(mcp): add OAuth skip endpoint and streamline in-chat interaction cards
Add a dedicated cancel route so users can skip mid-conversation OAuth without
token verification, and restyle OAuth/approval cards to match AgentStreamDisplay
with compact inline actions.
2026-06-29 23:22:28 +08:00
wizardchen 9fd7a7f522 feat(mcp): prompt and resume OAuth authorization mid-conversation
When an agent invokes an OAuth-enabled MCP service that the current user has
not authorized yet, the tool call now pauses and surfaces an "Authorize" card
in the chat instead of failing with an opaque error. After the user completes
the per-user OAuth popup, the backend verifies a token exists and resumes the
paused tool call automatically.

Backend:
- Add mcp_oauth_required / mcp_oauth_resolved events and response types.
- Add OAuthPendingRequest + Gate.RequestOAuthAndWait, reusing the existing
  approval waiter, Resolve, Redis cross-instance fan-out and timeout.
- MCPTool.Execute pauses on an authorization-required error for OAuth services
  and retries once (with a fresh ctx and recycled connection) after the user
  authorizes.
- Forward the new events to the SSE stream.
- Add ResolveMCPOAuth handler + route; it verifies authorization before
  unblocking so a premature resolve cannot resume into another failure.

Frontend:
- Add McpOAuthCard with an Authorize button reusing the popup + status-poll
  flow, then resolve the paused run.
- Handle the new stream events (dedup multiple pending cards per service).
- Add resolveMCPOAuth API client and i18n keys (zh/en/ko/ru).
2026-06-29 23:22:28 +08:00
wizardchen 6932c3a85d feat(embed): always show channel key via detail API
Fetch publish tokens when opening an embed channel drawer so deploy snippets
stay available after browser restarts, while list endpoints still omit secrets.
2026-06-26 14:55:39 +08:00
wizardchen 97b7bb9890 fix(knowledge): harden batch reparse UX and async processing
Move batch reparse work to an asynq task to avoid HTTP timeouts, add i18n
for all locales, guard in-flight documents, and prevent concurrent delete/reparse.
2026-06-25 23:57:54 +08:00
jw_mac a14c1d87aa feat(knowledge): add batch reparse capability
This commit introduces the Batch Reparse feature for the Knowledge Base.
Users can now select multiple documents and trigger a re-parsing pipeline in batches, streamlining the recovery from parse failures.
2026-06-25 23:43:27 +08:00
wizardchen 2c44a41f2a feat(models): add model test debugger for saved model configs
Let admins send real requests against configured chat, embedding, rerank,
VLM, and ASR models from settings, with thinking controls shown only when
the model supports them.
2026-06-25 17:05:33 +08:00
wizardchen 7f3f219f58 feat(mcp): add OAuth2 authorization for MCP services
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.

- Persist per-user access/refresh tokens and per-service DCR client
  credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
  in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
  the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
  hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
  authorization status/actions, and render the connection test result
  inline in the editor drawer instead of a stacked centered dialog
2026-06-22 20:19:31 +08:00
wizardchen 5d0d317a36 feat(datasource): reveal pre-existing selections in lazily-loaded picker
When editing a data source whose tree loads one level at a time (Feishu
wiki), a saved deep-node selection was invisible: only top-level spaces
were listed, so the selected node could not be seen and its ancestor
space rendered as unchecked.

- Add Connector.ResolveResourceAncestors: given selected resource IDs,
  return the ancestor IDs that must be expanded to reveal them. Feishu
  walks up via GetWikiNode (O(depth) per selection, no full-tree walk);
  Notion/Yuque return empty.
- Fix wikiNode.ParentNodeID JSON tag: the Feishu API field is
  parent_node_token, not parent_node_id, so GetWikiNode can return the
  parent and the ancestor walk works in production.
- Expose POST /datasource/{id}/resource-ancestors (service + handler +
  route) and a matching frontend API client.
- Edit-mode picker reveals hidden selections by expanding and lazily
  loading each ancestor level, so selections are visible and checked.
- Gate child fetching on treeFullyLoaded instead of a partial-children
  heuristic, avoiding a redundant request for full-tree connectors.
2026-06-22 19:24:08 +08:00
wizardchen abf6a99b90 fix(api): add POST for hybrid-search while keeping GET compatibility (#1727)
GET with a JSON body was non-standard and confused integrators. Register
POST as the preferred method and keep GET for existing clients; update SDK,
MCP server, and API docs accordingly.

Refs: https://github.com/Tencent/WeKnora/issues/1727
2026-06-22 17:22:35 +08:00
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
wizardchen ed9802ff2b feat(embed): add backend embed channels with auth and rate limiting (#1657)
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
2026-06-12 23:10:39 +08:00
wizardchen 1778c5ac7b refactor(tracing): remove Jaeger integration and related components
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
2026-06-08 21:01:04 +08:00
wizardchen 035093dc35 feat(configuration): update asynq concurrency settings and enhance FAQ handling
- Increased the default worker pool size for the asynq server from 16 to 32 to improve performance during concurrent uploads.
- Updated the .env.example and docker-compose.yml files to reflect the new default concurrency setting.
- Enhanced the handling of FAQ entries in the tool results, ensuring distinct identification through standard questions, improving user experience and clarity in the output.
- Refactored related components to utilize the new FAQ attributes, ensuring consistent display and interaction across the application.
2026-06-04 15:48:04 +08:00
ochan.kwon e9980c6011 fix: deep-copy stored files and images when cloning a knowledge base
Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.

Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.

Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.

Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
2026-06-03 14:45:59 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen bb202a203a feat: implement tenant validation for file access and enhance related tests
This commit introduces a new validation mechanism to ensure that file access paths include the correct tenant segment, preventing cross-tenant access. The `ValidateStoragePathTenant` function has been added to enforce this rule, and the `serveFiles` function has been updated to return a forbidden status for invalid paths. Additionally, new tests have been added to verify the behavior of the file service under various tenant scenarios, ensuring robust handling of file access permissions.
2026-05-29 13:47:32 +08:00
wizardchen 44d6175559 feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.

State machine

  pending -> processing -> finalizing -> completed
                              |
                              +-> cancelled (any of the three
                                            in-flight states)
                              +-> failed
                              +-> deleting

`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.

Backend

- New ParseStatusFinalizing + pending_subtasks_count column with
  migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
  conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
  counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
  count up front, flip to finalizing (or completed when no
  enrichment is enabled), and only then fan out subtasks. Subtask
  handlers (summary, question, graph extract) defer-decrement on
  terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
  pending / processing / finalizing. Marks the row cancelled,
  zeroes the counter, best-effort dequeues asynq tasks via a new
  TaskInspector abstraction (asynq-mode walks pending/scheduled/
  retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
  for the attempt via a new repo.CancelAllOpenSpans helper so the
  trace viewer's striped bars all flip to cancelled, even leaf
  generations whose parent stage already EndSpan'd (multimodal
  fan-out pattern). knowledge_post_process closes its postSpan
  via SkipSpan on the cancel/deleting entry guard so a worker
  that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
  identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
  downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
  expensive enrichment (graph extract) bails immediately when the
  parent knowledge is aborted.

Frontend

- New cancelKnowledgeParse API client + "Stop parsing" entry in
  both list view and card view more menus, gated on
  pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
  that recognises `finalizing` (previously the doc list silently
  stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
  new isHardTerminal short-circuits LIVE for cancelled/failed/
  completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
  ("增强中") from completed and shows the previous "生成摘要中"
  copy when summary_status is still pending under finalizing.
  Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
  across zh-CN, en-US, ko-KR, ru-RU.

Docs / SDK

- docs/api/knowledge.md: documents the new finalizing state,
  cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
  the cancellable statuses.
2026-05-28 20:16:02 +08:00
wizardchen 4db6e69b4e fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.

- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
  closes the root row idempotently. Re-closing a terminal root no-ops
  so success / cascade-fail / dead-letter paths can fire without
  coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
  on the success path. Async downstream work (summary/question/wiki/
  graph) still records its own spans; their completion extends the
  trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
  (docreader / chunking / embedding / multimodal / postprocess).
  Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
  DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
  document-related task exhausts retries. The probe payload now
  extracts the Attempt field that Document/Manual/PostProcess
  payloads already thread through.

Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
              output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
            output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
             output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph

i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-28 15:14:45 +08:00
wizardchen c82b098f44 fix(knowledge): address span-tree review feedback
- Collapse migrations 000052 (flat stages) + 000053 (span tree) into a
  single 000052_knowledge_processing_spans migration; the flat stages
  table never escaped this branch and the create-then-drop sequence had
  no value.
- BeginStage: detect an existing (kid, attempt, stage) row before
  inserting and reuse its span_id with reset state, so re-entry from
  asynq retries or adjacent code paths no longer produces duplicate
  timeline segments.
- FailSpan: when sibling-stage cascade flips a dependent stage to
  cancelled, also CancelDescendants on its subtree so already-running
  subspans (embedding.batch[i] etc.) cannot remain as orphan running
  rows under a cancelled parent.
- Dead-letter callback: replace two sequential UpdateKnowledgeColumn
  writes with a single UpdateKnowledgeColumns map update so we cannot
  end up with parse_status=failed and stale error_message (or vice
  versa) when one of the two writes fails.
- touchKnowledgeHeartbeat: skip subspan/generation transitions; only
  root and stage transitions poke knowledge.updated_at. Spans-table
  MAX(updated_at) already covers subspan progress for housekeeping, so
  this avoids 2*N+ UPDATE bursts on the same hot row when a multimodal
  stage fans out to many images.
- Add regression tests for the BeginStage idempotency contract and the
  cascade-into-subspans behaviour.
2026-05-28 15:14:45 +08:00
wizardchen c9941a6688 refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:

  1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
     are independent of each other, both downstream of Chunking, both
     upstream of PostProcess. The flat (knowledge_id, stage) table
     couldn't represent that, so a Chunking failure left dependents
     stranded as "pending" forever instead of being marked as
     impossible-to-run.

  2. No history across attempts. A reparse erased the previous run's
     status before the new run started, leaving operators with no way
     to investigate "why did this fail twice?".

  3. Stages had only status + duration. Operators want to know how big
     the work was — pages parsed, chunks created, tokens embedded, VLM
     calls made — to distinguish "slow because the file is huge" from
     "slow because the docreader is wedged".

  4. Multimodal fans out N image tasks; Embedding fans out M batches;
     PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
     is interesting on its own (Langfuse already captures this for
     LLM calls). The flat model couldn't express it.

The redesign mirrors Langfuse's trace/span/generation hierarchy:

  * Migration 000053 supersedes 000052: knowledge_processing_spans
    table with (knowledge_id, attempt, span_id) primary key, plus
    parent_span_id, kind ∈ {root, stage, subspan, generation},
    status ∈ {pending,running,done,failed,skipped,cancelled}, and
    JSONB input/output/metadata fields.

  * SpanTracker (replacing StageTracker) exposes OpenAttempt /
    BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
    LookupStage. Cross-process workers (image_multimodal) get the
    parent's attempt + span via payload + LookupStage so subspans
    attach correctly.

  * StageDependencies declares the DAG; FailSpan now cascades —
    descendants of the failed span and dependent stages are flipped
    to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
    blast radius instead of orphan spinners.

  * Reparse now calls OpenAttempt up front so the timeline reflects
    "new attempt, all pending" instead of letting the previous run's
    status linger until the worker picks up the task.

  * Image_multimodal records each image as a generation subspan with
    its own success/failure on the parent attempt's multimodal stage.
    The finalize-on-last-attempt counter logic is preserved unchanged.

  * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
    a tree shape with synthesized pending placeholders so the
    frontend always renders five timeline segments. ?attempt=N
    enables history navigation.
2026-05-28 15:14:45 +08:00
wizardchen 04f56f9cda feat(knowledge): track per-stage parsing progress with /stages API
Adds a five-segment progress model for the document parsing pipeline so
the UI (PR③) can render a timeline showing where each document is
(DocReader → Chunking → Embedding → Multimodal → PostProcess) and
which stage failed with what error code.

- New table `knowledge_processing_stages` (migration 000052) with one
  row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an
  attempt counter so re-parses don't lose history.

- StageTracker service exposes Begin/Done/Fail/Skip; all calls are
  best-effort and never break the pipeline if persistence fails.

- Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT /
  VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized
  remediation hints.

- Tracker call sites added at the four meaningful failure points:
  convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding),
  enqueueImageMultimodalTasks (Multimodal start),
  KnowledgePostProcess.Handle (Multimodal close + PostProcess).

- New endpoint `GET /api/v1/knowledge/:id/stages` returns the five
  canonical stages — missing rows are synthesized as "pending" so
  the timeline always renders five segments. Includes current_stage
  and last_error block.
2026-05-28 15:14:45 +08:00
wizardchen 3ae3ea97c5 fix(knowledge): prevent documents from getting stuck in "processing"
Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.

- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
  WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
  CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
  (WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.

- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
  (default 30m). Without this, a hung docreader pinned a worker for the
  full DocumentProcessTimeout window.

- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
  failing single image no longer strands the parent — the asynq retry
  is allowed to run, but on the final attempt we count the image
  regardless of outcome. Redis DECR errors fall back to enqueuing the
  post-process task instead of returning silently.

- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
  ManualProcess exhausts retries, immediately mark the corresponding
  Knowledge as failed with the last error. This surfaces the failure
  in the UI without waiting for the housekeeping sweep.

- HousekeepingService: 5-minute cron that flips knowledge rows stuck
  in "processing" past DocumentProcessTimeout + 10m to failed, plus
  summary rows stuck > 1h. Catches anything the other safety nets
  miss (worker SIGKILL mid-handler, etc.). Disable with
  WEKNORA_HOUSEKEEPING_ENABLED=false.

- Distributed startup recovery: previously the post-restart sweep was
  skipped whenever REDIS_ADDR was set, even though Asynq does not
  reschedule the task that was actively running on the dead instance.
  Now the sweep runs in distributed mode too, but only against rows
  older than 30 minutes to avoid racing peer instances.
2026-05-28 15:14:45 +08:00
wizardchen ae4ec0cf06 fix(im): make presigned URL flow diagnosable end-to-end
When IM image rendering breaks, operators previously had no log line to
correlate against the IM platform's fetch attempt. Add observability
hooks on both ends and unblock HEAD probes so common IM previews work
at all.

- log rewriteStorageURLs success/failure/no-op with the full signed URL
  (operators can copy it from logs and verify public reachability)
- log presigned handler 4xx with client_ip + UA + tenant_id + file_path
  so failures correlate against IM platform fetch logs; use the request
  context so trace IDs are preserved
- accept HEAD on /api/v1/files/presigned: IM platforms (Feishu, Slack
  etc.) probe with HEAD before GET when rendering image previews, and a
  401 there is enough to break the inline image even when the GET would
  have succeeded
- add Admin-only GET /api/v1/files/presigned-preview that returns the
  exact URL an IM channel would embed for the calling tenant, for
  self-service verification without sending a real IM message
- clarify APP_EXTERNAL_URL and MINIO_ENDPOINT public-reachability rules
  in .env.example; misconfigured endpoints are the most common cause of
  "image broken in IM" reports
2026-05-28 08:03:57 +08:00
lyingbug 3611a8607d feat(rbac): add multi-use share-link invitations for invite_only mode (#1504)
* feat(rbac): add multi-use share-link invitations for invite_only mode

When the auth.registration_mode toggle flips to invite_only, the public
/auth/register endpoint returns 403 — but until now there was no
channel for unregistered users to actually join a tenant. The existing
invitation table required a registered invitee, so new emails could
not enter the system at all.

This change introduces a share-link model:

- Owner generates a multi-use registration link with a per-tenant role
- Recipient opens the link, registers with their own email, and is
  added to the tenant on the same request
- Token is stored in plaintext (not hashed) so the management UI can
  re-display the URL on demand without a "copy now or revoke" trap.
  Threat model is bounded by 7-day TTL, revocability, and the fact
  that all the link grants is membership in one tenant.
- accepted_count tracks how many users joined through the link so the
  Owner can tell whether a link is fresh or has already been spread

Per-user invitations (registered email -> in-app inbox accept) are
unchanged. Share-link rows use empty invitee_user_id as discriminator;
the partial unique index on (tenant_id, invitee_user_id) was relaxed
to skip empty values so multiple share links can coexist per tenant.

Frontend reuses Login.vue for /register?token=xxx instead of a
parallel page, and adds icon-only revoke / remove actions plus a
distinct "active" status tag for share-link rows.

Migration 000054 adds the token column, the accepted_count column,
and the relaxed pending unique index in a single step.

* feat(auth): update invitation token handling and add rate limiting

- Refactor the invitation token retrieval to use POST instead of GET, enhancing security by preventing the plaintext token from appearing in logs and browser history.
- Update the API endpoint from `/auth/invitations/:token` to `/auth/invitations/lookup` to reflect the new method.
- Introduce a rate limiter for unauthenticated share-link endpoints to mitigate brute-force attacks and abuse, ensuring a maximum of 30 requests per minute per IP.
- Adjust the registration flow to maintain user experience while securing the invitation process.
2026-05-27 19:42:01 +08:00
wizardchen acdc0a526f feat(system): expose platform audit log + polish audit drawers
The system_settings update, system admin promote/revoke, and
apply-default-storage-quota routes have been writing audit rows since
the prior commits, but with TenantID=0 (system-scope). The per-tenant
GET /tenants/:id/audit-log endpoint filters by tenant_id and never
returns them, so until now those rows existed only in the DB with no UI
surface. This commit closes the loop:

Backend:
- GET /api/v1/system/admin/audit-log: new SystemAdmin-gated endpoint
  reusing AuditLogService.List with tenant_id=0. Same cursor-paged
  shape and query params (after_id / limit / action / outcome / actor)
  as the per-tenant feed, so the frontend reuses the same client logic.
- Wired through RegisterSystemAdminRoutes (mounted on the existing
  adminRoutes group so it inherits the SystemAdmin() guard). The
  handler dependency is optional: nil auditLogHandler skips the route,
  mirroring RegisterTenantRoutes' /audit-log handling.

Frontend — new platform audit drawer in SystemSettings.vue:
- "审计日志" entry button in the section header opens a side drawer
  (880px) listing system-scope events. Lazy-loaded on first open;
  refresh is explicit via a button inside the drawer.
- Table columns: stacked date/time (so 50 events in the same minute
  remain distinguishable), stacked actor/role, action tag, structured
  target (subject key + diff line), outcome. The dead request column
  (system actions don't go through middleware path capture) is dropped
  in favour of richer target rendering.
- Per-action target formatters:
    * system.setting_changed: subject = registry key, diff = `old → new`
      (JSON-encoded, 80-char truncation). Reset shows `old → (空)`.
    * tenant_storage_quota bulk apply: subject = "批量同步", diff =
      "applied to N tenants (X GB)".
    * system.admin_promoted / revoked: subject = "name (email)", diff
      annotates idempotent / noop branches so an audit reader can tell
      a real grant from a probe.
- Click-to-expand row reveals the full audit context: actor UUID,
  target_user_id / target_type / target_id, and raw details JSON in
  monospaced scroll-capped block. No psql round-trip needed for
  forensic spot checks.
- Sticky thead pinned to the scroll container so column labels survive
  long scrolls. Cells vertical-aligned middle to keep single-line tag
  cells visually balanced against multi-line target cells. No zebra
  stripes — the stacked content already provides row separation, and
  stripes on top read as noise.

Frontend — same polish back-ported to TenantMembers.vue audit drawer:
- Same drawer width, stacked time / actor cells, structured target +
  diff layout, expandable raw-details row, sticky thead, vertical-
  align middle, no stripes. Refresh button reformulated as a text
  button with label (was an outlined square icon-only).
- request_path column kept (rbac.access_denied carries meaningful
  paths) but empty values render as a placeholder dash so they don't
  read as broken.
- Diff line now covers rbac.invitation_sent / invitation_revoked role
  in addition to the existing role_changed / access_denied details.

API:
- frontend/src/api/system/index.ts: listSystemAuditLog() reuses the
  AuditLog / ListAuditLogParams types from @/api/tenant/audit-log
  (re-exported) so consumers don't need to cross-import.

i18n (zh-CN / en-US / ko-KR / ru-RU):
- system.globalSettings.audit.*: full drawer copy + per-action labels
  (system.setting_changed / admin_promoted / admin_revoked) + target
  diff templates + expanded-row labels.
- tenantMember.audit.expanded.*: expanded-row labels added so the
  shared drawer treatment renders cleanly under tenant scope.
2026-05-26 21:13:56 +08:00
wizardchen 9cce0c8e5e feat(system): consolidate system admin and settings into one Settings panel
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.

Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
  key so the 3-tier resolver falls back to ENV / built-in default.
  Idempotent (resetting a never-persisted key returns nil); emits an
  audit row only on real deletions, invalidates the local cache, and
  publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
  storage_quota in one statement. Powers POST /system/admin/tenants/
  apply-default-storage-quota; bypasses the per-tenant whitelist on
  PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
  emitAdminAudit() in SystemHandler — promote / revoke now leave an
  audit trail with TenantID=0 and {target_email,target_username,
  idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
  (username, email fallback) so the UI shows "wizardchen" instead of
  a UUID prefix without storing a denormalised column.

Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
  auto-persisting controls (switch/select @change, input/inputnumber
  @blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
  roster). auth.registration_mode change goes through an inline
  popconfirm; cancel rolls back. Reset / bulk-apply also inline
  popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
  so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
  admins are now compatibility redirects to /platform/settings?section=
  system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
  so the wider tables (members, system settings) breathe; <1100px
  viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
  empty / badges / priority hint / per-key labels / reset / bulkApply /
  admins (label, placeholder, save messages, popconfirm copy) across
  zh-CN, en-US, ko-KR, ru-RU.

Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
  deploy-time-only knob (nginx + docreader + frontend each cache the
  env at startup); a SystemAdmin UI override would mislead operators
  because nginx would still 413. Until all four layers can hot-reload
  the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
  off SystemSettingService for live updates; ENV remains the fallback
  for never-overridden deployments.
2026-05-26 21:13:56 +08:00
wizardchen 47a183aa65 feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
2026-05-26 21:13:56 +08:00
wizardchen 7528175a74 feat(knowledge-base): implement per-user pinning for knowledge bases
Refactored the pinning mechanism for knowledge bases to be user-specific, allowing any user with read access to pin knowledge bases independently. This change includes the introduction of a new `user_kb_pins` table to manage pin states, replacing the previous tenant-wide pinning system. Updated relevant API endpoints and service logic to support this new functionality, ensuring that the UI reflects the user's personal pin status.

Additionally, adjusted the knowledge base listing to sort by user-specific pin status, enhancing the user experience by prioritizing pinned items.

Refs: #1303
2026-05-18 17:28:58 +08:00