feat(worker): replace aggregate asynq concurrency with per-pool governance
Split the monolithic upstream worker budget into explicit core, post-process, enrichment, maintenance, and shared elastic pools, expose live cluster capacity on the runtime dashboard, and retire the legacy asynq.concurrency setting.
This commit is contained in:
+22
-9
@@ -716,22 +716,35 @@ DOCREADER_TRANSPORT=grpc
|
||||
# WEKNORA_DOCUMENT_PROCESS_TIMEOUT=2h
|
||||
|
||||
# --- Async pipeline tuning (optional) -----------------------------------------
|
||||
# Worker pool size for the PARSE asynq server (document parse / post-process /
|
||||
# embedding / multimodal / graph / question — everything except wiki). Default
|
||||
# 32 — raise it on machines that handle many concurrent uploads (default Go
|
||||
# runtime.NumCPU() under-provisions for the I/O-bound document pipeline). Can
|
||||
# also be set in the management UI under Settings → System settings
|
||||
# (asynq.concurrency); UI changes require a process restart.
|
||||
# WEKNORA_ASYNQ_CONCURRENCY=32
|
||||
# Per-instance guaranteed worker capacity for document parse / manual reparse.
|
||||
# Core and enrichment can additionally borrow from the shared elastic pool.
|
||||
# WEKNORA_ASYNQ_CORE_CONCURRENCY=8
|
||||
|
||||
# Dedicated lightweight post-process/fan-out workers. Keeping this separate
|
||||
# prevents completed parses from waiting behind long DocReader jobs.
|
||||
# WEKNORA_ASYNQ_POSTPROCESS_CONCURRENCY=2
|
||||
|
||||
# Per-instance guaranteed capacity for summary, image, graph, and question
|
||||
# enrichment tasks.
|
||||
# WEKNORA_ASYNQ_ENRICHMENT_CONCURRENCY=12
|
||||
|
||||
# Per-instance capacity for datasource sync, batch operations, and cleanup.
|
||||
# Maintenance is not eligible for shared burst capacity because its tasks can
|
||||
# be long-running.
|
||||
# WEKNORA_ASYNQ_MAINTENANCE_CONCURRENCY=4
|
||||
|
||||
# Elastic capacity shared by core parse and enrichment queues. Queue dequeue is
|
||||
# atomic, so these workers safely supplement both dedicated pools.
|
||||
# WEKNORA_ASYNQ_SHARED_CONCURRENCY=6
|
||||
|
||||
# Worker pool size for the DEDICATED wiki asynq server. The wiki generation
|
||||
# pipeline runs on its own pool with its own concurrency budget, hard-isolated
|
||||
# from the parse pool above: heavy document parsing can never starve wiki
|
||||
# generation during upload storms, and a wiki burst can never starve
|
||||
# user-facing parsing. Default 16. Wiki work is synthesis-LLM-bound, so size
|
||||
# user-facing parsing. Default 8. Wiki work is synthesis-LLM-bound, so size
|
||||
# this to your LLM provider's concurrency budget. Also settable in the UI
|
||||
# (asynq.wiki_concurrency); requires a process restart.
|
||||
# WEKNORA_WIKI_ASYNQ_CONCURRENCY=16
|
||||
# WEKNORA_WIKI_ASYNQ_CONCURRENCY=8
|
||||
|
||||
# Default per-model concurrency cap for background tasks (document ingest /
|
||||
# enrichment). Shared across replicas, keyed by model ID. Only affects
|
||||
|
||||
Reference in New Issue
Block a user