feat(wiki): dedicated worker pool with concurrent claim-based batching
Isolate the wiki generation pipeline from the document parse pipeline so the two can no longer starve each other during concurrent uploads. - Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_ concurrency, default 16), split from the parse pool (NewParseAsynqServer). Both run the same mux over disjoint queue sets. Replaces the old weighted-lottery approach where wiki sat on the low queue (weight 1). - Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at) and ReleaseByIDs so multiple concurrent consumers pull disjoint rows without an external per-KB lock; PeekBatch retained for Lite mode. - New TypeWikiFinalize debounced KB-global finalize task (index rebuild / dead-link cleanup / cross-linking), registered on both Redis and Lite paths. - Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent batches per KB; Map/Reduce fan-out sized within a batch. New config knob + system setting + swagger docs.
This commit is contained in:
+15
-5
@@ -708,13 +708,23 @@ DOCREADER_TRANSPORT=grpc
|
||||
# WEKNORA_DOCUMENT_PROCESS_TIMEOUT=2h
|
||||
|
||||
# --- Async pipeline tuning (optional) -----------------------------------------
|
||||
# Worker pool size for the asynq server. Default 32 — raise it on machines
|
||||
# that handle many concurrent uploads (default Go runtime.NumCPU() under-
|
||||
# provisions for the I/O-bound document pipeline). Can also be set in the
|
||||
# management UI under Settings → System settings (asynq.concurrency);
|
||||
# UI changes require a process restart.
|
||||
# Worker pool size for the PARSE asynq server (document parse / post-process /
|
||||
# embedding / multimodal / graph / question — everything except wiki). Default
|
||||
# 32 — raise it on machines that handle many concurrent uploads (default Go
|
||||
# runtime.NumCPU() under-provisions for the I/O-bound document pipeline). Can
|
||||
# also be set in the management UI under Settings → System settings
|
||||
# (asynq.concurrency); UI changes require a process restart.
|
||||
# WEKNORA_ASYNQ_CONCURRENCY=32
|
||||
|
||||
# Worker pool size for the DEDICATED wiki asynq server. The wiki generation
|
||||
# pipeline runs on its own pool with its own concurrency budget, hard-isolated
|
||||
# from the parse pool above: heavy document parsing can never starve wiki
|
||||
# generation during upload storms, and a wiki burst can never starve
|
||||
# user-facing parsing. Default 16. Wiki work is synthesis-LLM-bound, so size
|
||||
# this to your LLM provider's concurrency budget. Also settable in the UI
|
||||
# (asynq.wiki_concurrency); requires a process restart.
|
||||
# WEKNORA_WIKI_ASYNQ_CONCURRENCY=16
|
||||
|
||||
# Read/write timeout (in milliseconds) the asynq client uses against Redis.
|
||||
# Default 500ms (writes scale to 1000ms). Bump if you see "i/o timeout"
|
||||
# errors during heavy multimodal counter DECRs or large batch uploads.
|
||||
|
||||
Reference in New Issue
Block a user