feat(wiki): dedicated worker pool with concurrent claim-based batching

Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.

- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
  own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
  concurrency, default 16), split from the parse pool (NewParseAsynqServer).
  Both run the same mux over disjoint queue sets. Replaces the old
  weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
  and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
  without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
  dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
  batches per KB; Map/Reduce fan-out sized within a batch. New config knob
  + system setting + swagger docs.
This commit is contained in:
wizardchen
2026-07-09 13:27:47 +08:00
committed by lyingbug
parent 7642117b70
commit 70e2e85fa6
17 changed files with 1282 additions and 377 deletions
+15 -5
View File
@@ -708,13 +708,23 @@ DOCREADER_TRANSPORT=grpc
# WEKNORA_DOCUMENT_PROCESS_TIMEOUT=2h
# --- Async pipeline tuning (optional) -----------------------------------------
# Worker pool size for the asynq server. Default 32 — raise it on machines
# that handle many concurrent uploads (default Go runtime.NumCPU() under-
# provisions for the I/O-bound document pipeline). Can also be set in the
# management UI under Settings → System settings (asynq.concurrency);
# UI changes require a process restart.
# Worker pool size for the PARSE asynq server (document parse / post-process /
# embedding / multimodal / graph / question — everything except wiki). Default
# 32 — raise it on machines that handle many concurrent uploads (default Go
# runtime.NumCPU() under-provisions for the I/O-bound document pipeline). Can
# also be set in the management UI under Settings → System settings
# (asynq.concurrency); UI changes require a process restart.
# WEKNORA_ASYNQ_CONCURRENCY=32
# Worker pool size for the DEDICATED wiki asynq server. The wiki generation
# pipeline runs on its own pool with its own concurrency budget, hard-isolated
# from the parse pool above: heavy document parsing can never starve wiki
# generation during upload storms, and a wiki burst can never starve
# user-facing parsing. Default 16. Wiki work is synthesis-LLM-bound, so size
# this to your LLM provider's concurrency budget. Also settable in the UI
# (asynq.wiki_concurrency); requires a process restart.
# WEKNORA_WIKI_ASYNQ_CONCURRENCY=16
# Read/write timeout (in milliseconds) the asynq client uses against Redis.
# Default 500ms (writes scale to 1000ms). Bump if you see "i/o timeout"
# errors during heavy multimodal counter DECRs or large batch uploads.