Commit Graph

13 Commits

Author SHA1 Message Date
wizardchen 6c794f64cf feat(runtime): add bulk purge for archived queue tasks and chat_attachment i18n
Operators can clear all finally-failed tasks in one queue from the runtime
drawer, with audit logging and scoped API access. Also localize the
chat_attachment queue label in the runtime queue details table.
2026-07-23 16:52:35 +08:00
lhy c74b0caa35 fix(task-inspector): handle lazy queue not found errors 2026-07-23 14:24:21 +08:00
zhangzherui 37de0afad0 fix(tasks): dequeue work for deleted knowledge bases 2026-07-23 12:17:14 +08:00
wizardchen 01f1011511 feat(system): add cursor pagination for runtime queue tasks
Replace offset-based page numbers with opaque Redis-backed cursors so
scheduled/retry tasks sort by next run time and live lists stay stable
when queue state changes between requests.
2026-07-14 16:53:07 +08:00
wizardchen 4ae48d6acf feat(system): expand runtime queue task inspection and safe actions
Replace failed-only task views with state-aware runtime task listing,
operator-safe cancel/run-now/delete actions, orphan detection, and
multimodal queue cleanup for deleted knowledge rows.
2026-07-14 15:38:06 +08:00
wizardchen 6171962e56 feat(system): add failed-task inspection and manual retry for runtime queues
Give system admins visibility into asynq tasks that exhausted retries, with safe retry/delete actions and operator-facing UI in Runtime Queues.
2026-07-14 13:11:06 +08:00
wizardchen 07ba01ec93 feat(worker): replace aggregate asynq concurrency with per-pool governance
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
2026-07-13 15:51:06 +08:00
wizardchen d496d0fc73 feat(task-queue): split upstream workers into isolated core/enrichment/maintenance pools
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
2026-07-10 23:51:29 +08:00
wizardchen fe47ac1f96 feat(system): add runtime task queue observability dashboard for system admins
Expose live asynq queue depths and worker-pool concurrency via a new SystemAdmin API and settings UI, so operators can monitor parsing/wiki backlog without Redis CLI access.
2026-07-10 23:51:29 +08:00
wizardchen 70e2e85fa6 feat(wiki): dedicated worker pool with concurrent claim-based batching
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.

- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
  own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
  concurrency, default 16), split from the parse pool (NewParseAsynqServer).
  Both run the same mux over disjoint queue sets. Replaces the old
  weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
  and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
  without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
  dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
  batches per KB; Map/Reduce fan-out sized within a batch. New config knob
  + system setting + swagger docs.
2026-07-09 17:41:07 +08:00
wizardchen 035093dc35 feat(configuration): update asynq concurrency settings and enhance FAQ handling
- Increased the default worker pool size for the asynq server from 16 to 32 to improve performance during concurrent uploads.
- Updated the .env.example and docker-compose.yml files to reflect the new default concurrency setting.
- Enhanced the handling of FAQ entries in the tool results, ensuring distinct identification through standard questions, improving user experience and clarity in the output.
- Refactored related components to utilize the new FAQ attributes, ensuring consistent display and interaction across the application.
2026-06-04 15:48:04 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen 44d6175559 feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.

State machine

  pending -> processing -> finalizing -> completed
                              |
                              +-> cancelled (any of the three
                                            in-flight states)
                              +-> failed
                              +-> deleting

`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.

Backend

- New ParseStatusFinalizing + pending_subtasks_count column with
  migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
  conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
  counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
  count up front, flip to finalizing (or completed when no
  enrichment is enabled), and only then fan out subtasks. Subtask
  handlers (summary, question, graph extract) defer-decrement on
  terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
  pending / processing / finalizing. Marks the row cancelled,
  zeroes the counter, best-effort dequeues asynq tasks via a new
  TaskInspector abstraction (asynq-mode walks pending/scheduled/
  retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
  for the attempt via a new repo.CancelAllOpenSpans helper so the
  trace viewer's striped bars all flip to cancelled, even leaf
  generations whose parent stage already EndSpan'd (multimodal
  fan-out pattern). knowledge_post_process closes its postSpan
  via SkipSpan on the cancel/deleting entry guard so a worker
  that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
  identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
  downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
  expensive enrichment (graph extract) bails immediately when the
  parent knowledge is aborted.

Frontend

- New cancelKnowledgeParse API client + "Stop parsing" entry in
  both list view and card view more menus, gated on
  pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
  that recognises `finalizing` (previously the doc list silently
  stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
  new isHardTerminal short-circuits LIVE for cancelled/failed/
  completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
  ("增强中") from completed and shows the previous "生成摘要中"
  copy when summary_status is still pending under finalizing.
  Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
  across zh-CN, en-US, ko-KR, ru-RU.

Docs / SDK

- docs/api/knowledge.md: documents the new finalizing state,
  cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
  the cancellable statuses.
2026-05-28 20:16:02 +08:00