98ad41cda4
Follow-up hardening on the per-model background concurrency governor: - limiter: heartbeat now refreshes the ZSET key's PEXPIRE alongside the lease score, so a semaphore that stays saturated with no slot turnover can't let the whole key expire (which would drop every lease and admit over the limit); reuse one poll timer instead of allocating time.After each iteration under contention. - governor: extract the process-wide governor + Gate helper into the limiter package so chat and vlm share one limiter/limit without importing each other; add an in-process localLimiter for Lite mode. - vlm: gate background multimodal (VLM) calls through the same governor; previously only chat was throttled. - chat: ChatStream releases its slot when the consumer abandons the stream and cancels ctx (select on ctx.Done, drain inner), fixing a slot/goroutine leak. - container/sync_task: install the local governor in Lite mode and tag SyncTaskExecutor runs as background; honour model.max_concurrency=0 as an explicit disable instead of falling back to the default. - tests: cover the governor gating, local limiter, and stream release.
36 lines
1.2 KiB
Go
36 lines
1.2 KiB
Go
package vlm
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/Tencent/WeKnora/internal/models/limiter"
|
|
)
|
|
|
|
// Multimodal enrichment (image OCR / caption) is a high-volume, slow background
|
|
// stage that hits the same provider budget as chat. Like chat, it must be
|
|
// governed at the client layer so an image-heavy ingestion storm can't burst
|
|
// the whole worker pool against one VLM provider. Only background (asynq
|
|
// worker) calls are throttled — see limiter.Gate / types.IsBackgroundTask.
|
|
type concurrencyVLM struct {
|
|
inner VLM
|
|
}
|
|
|
|
func (w *concurrencyVLM) GetModelName() string { return w.inner.GetModelName() }
|
|
func (w *concurrencyVLM) GetModelID() string { return w.inner.GetModelID() }
|
|
|
|
func (w *concurrencyVLM) Predict(ctx context.Context, imgBytes [][]byte, prompt string) (string, error) {
|
|
release := limiter.Gate(ctx, w.inner.GetModelID())
|
|
defer release()
|
|
return w.inner.Predict(ctx, imgBytes, prompt)
|
|
}
|
|
|
|
// wrapVLMConcurrency installs the background concurrency governor as the
|
|
// outermost VLM decorator. Always applied; a cheap passthrough when no limiter
|
|
// is installed or the call is interactive.
|
|
func wrapVLMConcurrency(v VLM, err error) (VLM, error) {
|
|
if err != nil || v == nil {
|
|
return v, err
|
|
}
|
|
return &concurrencyVLM{inner: v}, nil
|
|
}
|