Surface per-model active/waiting/limit from the background task limiter so admins can distinguish task scheduling from model-service throttling, with full i18n coverage.
Add a new `max_concurrency` setting to the model configuration, allowing for fine-tuned control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This update includes localization support across multiple languages and integrates the setting into the system's concurrency governor, enabling immediate application of changes without requiring a service restart.
Introduce a new `max_concurrency` parameter to the model configuration, allowing control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This feature is applicable to chat, embedding, and VLM models, with a value of 0 or undefined falling back to the global default. Update UI components and localization files to support this new setting, ensuring a seamless user experience.
Follow-up hardening on the per-model background concurrency governor:
- limiter: heartbeat now refreshes the ZSET key's PEXPIRE alongside the
lease score, so a semaphore that stays saturated with no slot turnover
can't let the whole key expire (which would drop every lease and admit
over the limit); reuse one poll timer instead of allocating time.After
each iteration under contention.
- governor: extract the process-wide governor + Gate helper into the
limiter package so chat and vlm share one limiter/limit without
importing each other; add an in-process localLimiter for Lite mode.
- vlm: gate background multimodal (VLM) calls through the same governor;
previously only chat was throttled.
- chat: ChatStream releases its slot when the consumer abandons the
stream and cancels ctx (select on ctx.Done, drain inner), fixing a
slot/goroutine leak.
- container/sync_task: install the local governor in Lite mode and tag
SyncTaskExecutor runs as background; honour model.max_concurrency=0 as
an explicit disable instead of falling back to the default.
- tests: cover the governor gating, local limiter, and stream release.
Background ingestion/enrichment stages (summary, question, graph,
multimodal) all target the same chat model, and asynq queue weights are
scheduling priority rather than throttling. A large upload could therefore
burst up to the entire worker pool of concurrent chat calls against a
single provider, unbounded across replicas, with no model-side rate limit.
Govern concurrency at the chat-client layer (the one place that sees every
task type), keyed by model ID:
- internal/models/limiter: Redis ZSET + Lua distributed semaphore with a
lease heartbeat, expired-lease reclaim for crashed holders, and fail-open
on any backend error or context cancellation.
- chat: concurrencyChat decorator (outermost in NewChat) acquires a
per-model slot only for background tasks; interactive chat is never gated.
- router/task: asynq middleware tags every worker execution as background.
- container: wire the Redis limiter with WEKNORA_MODEL_MAX_CONCURRENCY
(model.max_concurrency), default 8; enabled only when Redis is present.
Tested with miniredis: concurrency cap, expired-lease reclaim, per-key
isolation, and all fail-open paths (nil client, backend down, cancelled ctx).