feat(wiki): dedicated worker pool with concurrent claim-based batching

Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.

- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
  own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
  concurrency, default 16), split from the parse pool (NewParseAsynqServer).
  Both run the same mux over disjoint queue sets. Replaces the old
  weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
  and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
  without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
  dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
  batches per KB; Map/Reduce fan-out sized within a batch. New config knob
  + system setting + swagger docs.
This commit is contained in:
wizardchen
2026-07-09 13:27:47 +08:00
committed by lyingbug
parent 7642117b70
commit 70e2e85fa6
17 changed files with 1282 additions and 377 deletions
+15 -5
View File
@@ -708,13 +708,23 @@ DOCREADER_TRANSPORT=grpc
# WEKNORA_DOCUMENT_PROCESS_TIMEOUT=2h
# --- Async pipeline tuning (optional) -----------------------------------------
# Worker pool size for the asynq server. Default 32 — raise it on machines
# that handle many concurrent uploads (default Go runtime.NumCPU() under-
# provisions for the I/O-bound document pipeline). Can also be set in the
# management UI under Settings → System settings (asynq.concurrency);
# UI changes require a process restart.
# Worker pool size for the PARSE asynq server (document parse / post-process /
# embedding / multimodal / graph / question — everything except wiki). Default
# 32 — raise it on machines that handle many concurrent uploads (default Go
# runtime.NumCPU() under-provisions for the I/O-bound document pipeline). Can
# also be set in the management UI under Settings → System settings
# (asynq.concurrency); UI changes require a process restart.
# WEKNORA_ASYNQ_CONCURRENCY=32
# Worker pool size for the DEDICATED wiki asynq server. The wiki generation
# pipeline runs on its own pool with its own concurrency budget, hard-isolated
# from the parse pool above: heavy document parsing can never starve wiki
# generation during upload storms, and a wiki burst can never starve
# user-facing parsing. Default 16. Wiki work is synthesis-LLM-bound, so size
# this to your LLM provider's concurrency budget. Also settable in the UI
# (asynq.wiki_concurrency); requires a process restart.
# WEKNORA_WIKI_ASYNQ_CONCURRENCY=16
# Read/write timeout (in milliseconds) the asynq client uses against Redis.
# Default 500ms (writes scale to 1000ms). Bump if you see "i/o timeout"
# errors during heavy multimodal counter DECRs or large batch uploads.
+1
View File
@@ -193,6 +193,7 @@ services:
- WEKNORA_TENANT_ENABLE_RBAC=${WEKNORA_TENANT_ENABLE_RBAC:-}
- WEKNORA_TENANT_MAX_OWNED_PER_USER=${WEKNORA_TENANT_MAX_OWNED_PER_USER:-}
- WEKNORA_ASYNQ_CONCURRENCY=${WEKNORA_ASYNQ_CONCURRENCY:-32}
- WEKNORA_WIKI_ASYNQ_CONCURRENCY=${WEKNORA_WIKI_ASYNQ_CONCURRENCY:-16}
- APK_MIRROR_ARG=${APK_MIRROR_ARG:-}
- WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL=${WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL:-}
depends_on:
+7 -3
View File
@@ -18084,15 +18084,19 @@ const docTemplate = `{
]
},
"ingest_batch_size": {
"description": "IngestBatchSize controls how many pending ops a single batch\nprocesses before scheduling a follow-up. 0 falls back to the\nhard-coded default (5). Operators on large KBs (4w+ docs) can\nraise this to 10–20 to amortize the lock-acquire / index-rebuild\noverhead across more documents per round.",
"description": "IngestBatchSize controls how many pending ops a single batch claims and\nprocesses before scheduling a follow-up. 0 falls back to the hard-coded\ndefault (5). Larger batches amortize per-batch setup and let more docs\nshare the batch-internal Map/Reduce fan-out; smaller batches spread a\nKB's backlog across more concurrent batches (finer scheduling grain).",
"type": "integer"
},
"ingest_map_parallel": {
"description": "IngestMapParallel sets the errgroup limit for the Map phase\n(per-document extraction + summary + chunk citation). 0 falls\nback to 10. Bound by the LLM provider's concurrency limit and\nthe worker's outbound HTTP pool.",
"description": "IngestMapParallel sets the errgroup limit for the Map phase\n(per-document extraction + summary + chunk citation) WITHIN one batch.\n0 falls back to 10. Bound by the LLM provider's concurrency limit and\nthe worker's outbound HTTP pool. Remember it multiplies with the number\nof concurrent batches (IngestMaxInflight).",
"type": "integer"
},
"ingest_max_inflight": {
"description": "IngestMaxInflight caps how many ingest batches for THIS KB may run\nconcurrently in the shared wiki worker pool (standard/Redis mode\nonly). 0 falls back to the hard-coded default (4). Since Phase 3\nremoved the exclusive per-KB lock, one KB's backlog could otherwise\nmonopolize the whole pool during a bulk import and starve other KBs;\nthis knob trades a single KB's peak throughput for cross-KB fairness.\nSet it \u003e= the wiki pool size to effectively disable the cap.",
"type": "integer"
},
"ingest_reduce_parallel": {
"description": "IngestReduceParallel sets the errgroup limit for the Reduce phase\n(per-slug page write). 0 falls back to 10. Bound by the same\nLLM concurrency / HTTP pool considerations as the Map phase,\nplus DB connection pool size.",
"description": "IngestReduceParallel sets the errgroup limit for the Reduce phase\n(per-slug page write) WITHIN one batch. 0 falls back to 10. Bound by the\nsame LLM concurrency / HTTP pool considerations as the Map phase, plus\nDB connection pool size. Same multiplier caveat as IngestMapParallel.",
"type": "integer"
},
"max_pages_per_ingest": {
+7 -3
View File
@@ -18077,15 +18077,19 @@
]
},
"ingest_batch_size": {
"description": "IngestBatchSize controls how many pending ops a single batch\nprocesses before scheduling a follow-up. 0 falls back to the\nhard-coded default (5). Operators on large KBs (4w+ docs) can\nraise this to 10–20 to amortize the lock-acquire / index-rebuild\noverhead across more documents per round.",
"description": "IngestBatchSize controls how many pending ops a single batch claims and\nprocesses before scheduling a follow-up. 0 falls back to the hard-coded\ndefault (5). Larger batches amortize per-batch setup and let more docs\nshare the batch-internal Map/Reduce fan-out; smaller batches spread a\nKB's backlog across more concurrent batches (finer scheduling grain).",
"type": "integer"
},
"ingest_map_parallel": {
"description": "IngestMapParallel sets the errgroup limit for the Map phase\n(per-document extraction + summary + chunk citation). 0 falls\nback to 10. Bound by the LLM provider's concurrency limit and\nthe worker's outbound HTTP pool.",
"description": "IngestMapParallel sets the errgroup limit for the Map phase\n(per-document extraction + summary + chunk citation) WITHIN one batch.\n0 falls back to 10. Bound by the LLM provider's concurrency limit and\nthe worker's outbound HTTP pool. Remember it multiplies with the number\nof concurrent batches (IngestMaxInflight).",
"type": "integer"
},
"ingest_max_inflight": {
"description": "IngestMaxInflight caps how many ingest batches for THIS KB may run\nconcurrently in the shared wiki worker pool (standard/Redis mode\nonly). 0 falls back to the hard-coded default (4). Since Phase 3\nremoved the exclusive per-KB lock, one KB's backlog could otherwise\nmonopolize the whole pool during a bulk import and starve other KBs;\nthis knob trades a single KB's peak throughput for cross-KB fairness.\nSet it \u003e= the wiki pool size to effectively disable the cap.",
"type": "integer"
},
"ingest_reduce_parallel": {
"description": "IngestReduceParallel sets the errgroup limit for the Reduce phase\n(per-slug page write). 0 falls back to 10. Bound by the same\nLLM concurrency / HTTP pool considerations as the Map phase,\nplus DB connection pool size.",
"description": "IngestReduceParallel sets the errgroup limit for the Reduce phase\n(per-slug page write) WITHIN one batch. 0 falls back to 10. Bound by the\nsame LLM concurrency / HTTP pool considerations as the Map phase, plus\nDB connection pool size. Same multiplier caveat as IngestMapParallel.",
"type": "integer"
},
"max_pages_per_ingest": {
+22 -11
View File
@@ -3480,25 +3480,36 @@ definitions:
per document. Empty / unknown value is treated as WikiExtractionStandard.
ingest_batch_size:
description: |-
IngestBatchSize controls how many pending ops a single batch
processes before scheduling a follow-up. 0 falls back to the
hard-coded default (5). Operators on large KBs (4w+ docs) can
raise this to 10–20 to amortize the lock-acquire / index-rebuild
overhead across more documents per round.
IngestBatchSize controls how many pending ops a single batch claims and
processes before scheduling a follow-up. 0 falls back to the hard-coded
default (5). Larger batches amortize per-batch setup and let more docs
share the batch-internal Map/Reduce fan-out; smaller batches spread a
KB's backlog across more concurrent batches (finer scheduling grain).
type: integer
ingest_map_parallel:
description: |-
IngestMapParallel sets the errgroup limit for the Map phase
(per-document extraction + summary + chunk citation). 0 falls
back to 10. Bound by the LLM provider's concurrency limit and
the worker's outbound HTTP pool.
(per-document extraction + summary + chunk citation) WITHIN one batch.
0 falls back to 10. Bound by the LLM provider's concurrency limit and
the worker's outbound HTTP pool. Remember it multiplies with the number
of concurrent batches (IngestMaxInflight).
type: integer
ingest_max_inflight:
description: |-
IngestMaxInflight caps how many ingest batches for THIS KB may run
concurrently in the shared wiki worker pool (standard/Redis mode
only). 0 falls back to the hard-coded default (4). Since Phase 3
removed the exclusive per-KB lock, one KB's backlog could otherwise
monopolize the whole pool during a bulk import and starve other KBs;
this knob trades a single KB's peak throughput for cross-KB fairness.
Set it >= the wiki pool size to effectively disable the cap.
type: integer
ingest_reduce_parallel:
description: |-
IngestReduceParallel sets the errgroup limit for the Reduce phase
(per-slug page write). 0 falls back to 10. Bound by the same
LLM concurrency / HTTP pool considerations as the Map phase,
plus DB connection pool size.
(per-slug page write) WITHIN one batch. 0 falls back to 10. Bound by the
same LLM concurrency / HTTP pool considerations as the Map phase, plus
DB connection pool size. Same multiplier caveat as IngestMapParallel.
type: integer
max_pages_per_ingest:
description: MaxPagesPerIngest limits pages created/updated per ingest operation
@@ -5,10 +5,12 @@ import (
"errors"
"fmt"
"strconv"
"time"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// taskPendingOpsRepository implements interfaces.TaskPendingOpsRepository.
@@ -71,6 +73,74 @@ func (r *taskPendingOpsRepository) PeekBatch(
return ops, nil
}
// ClaimBatch atomically claims up to `limit` eligible rows for the tuple.
// Eligibility = unclaimed (claimed_at IS NULL) OR stale claim
// (claimed_at < staleBefore). The whole thing runs in one transaction:
// we first lock the candidate ids (FOR UPDATE SKIP LOCKED on Postgres so
// concurrent claimers slide past each other's rows), stamp claimed_at,
// then read the claimed rows back. SKIP LOCKED is Postgres-only; on other
// dialects (SQLite, used by unit tests / Lite) writes are serialized by
// the engine so the plain SELECT+UPDATE is already race-free within the
// single-writer model.
func (r *taskPendingOpsRepository) ClaimBatch(
ctx context.Context,
taskType, scope, scopeID string,
limit int,
staleBefore time.Time,
) ([]*types.TaskPendingOp, error) {
if limit <= 0 {
limit = 1
}
if limit > 1000 {
limit = 1000
}
now := time.Now()
var claimed []*types.TaskPendingOp
err := r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
sel := tx.Model(&types.TaskPendingOp{}).
Where("task_type = ? AND scope = ? AND scope_id = ?", taskType, scope, scopeID).
// Parentheses are explicit so the OR binds tightly and doesn't
// leak past the tuple filter (GORM ANDs Where clauses but does
// not parenthesize raw fragments).
Where("(claimed_at IS NULL OR claimed_at < ?)", staleBefore).
Order("id ASC").
Limit(limit)
if tx.Dialector.Name() == "postgres" {
sel = sel.Clauses(clause.Locking{Strength: "UPDATE", Options: "SKIP LOCKED"})
}
var ids []int64
if err := sel.Pluck("id", &ids).Error; err != nil {
return err
}
if len(ids) == 0 {
return nil
}
if err := tx.Model(&types.TaskPendingOp{}).
Where("id IN ?", ids).
Update("claimed_at", now).Error; err != nil {
return err
}
return tx.Where("id IN ?", ids).Order("id ASC").Find(&claimed).Error
})
if err != nil {
return nil, err
}
return claimed, nil
}
// ReleaseByIDs clears claimed_at for the given rows, returning them to the
// unclaimed pool. Empty input is a no-op. Setting claimed_at back to NULL
// on a row that was never claimed is harmless.
func (r *taskPendingOpsRepository) ReleaseByIDs(ctx context.Context, ids []int64) error {
if len(ids) == 0 {
return nil
}
return r.db.WithContext(ctx).
Model(&types.TaskPendingOp{}).
Where("id IN ?", ids).
Update("claimed_at", nil).Error
}
// DeleteByIDs removes the given rows in one statement. Empty input is a
// no-op so the caller can invoke unconditionally at the end of a batch.
func (r *taskPendingOpsRepository) DeleteByIDs(ctx context.Context, ids []int64) error {
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"testing"
"time"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/assert"
@@ -270,6 +271,97 @@ func TestTaskPendingOps_DeleteByDedupKey_Filters(t *testing.T) {
assert.Equal(t, "k2", rows[0].DedupKey)
}
// TestTaskPendingOps_ClaimBatch_MarksAndReturnsDisjoint verifies that
// ClaimBatch returns rows in FIFO order, stamps claimed_at, and that a
// second claim does NOT re-return the already-claimed rows (the disjoint
// property concurrent consumers rely on).
func TestTaskPendingOps_ClaimBatch_MarksAndReturnsDisjoint(t *testing.T) {
db := setupTaskQueueTestDB(t)
repo := NewTaskPendingOpsRepository(db)
ctx := context.Background()
for _, k := range []string{"k1", "k2", "k3"} {
require.NoError(t, repo.Enqueue(ctx, makePendingOp("wiki:ingest", "knowledge_base", "kb", "ingest", k, nil)))
}
// Unrelated tuple must never be claimed.
require.NoError(t, repo.Enqueue(ctx, makePendingOp("wiki:ingest", "knowledge_base", "other", "ingest", "z", nil)))
stale := time.Now().Add(-time.Hour)
first, err := repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 2, stale)
require.NoError(t, err)
require.Len(t, first, 2)
assert.Equal(t, "k1", first[0].DedupKey)
assert.Equal(t, "k2", first[1].DedupKey)
assert.NotNil(t, first[0].ClaimedAt, "claimed_at should be stamped")
// Second claim skips the two already-claimed rows and returns the last.
second, err := repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, stale)
require.NoError(t, err)
require.Len(t, second, 1)
assert.Equal(t, "k3", second[0].DedupKey)
// Nothing left to claim.
third, err := repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, stale)
require.NoError(t, err)
assert.Len(t, third, 0)
}
// TestTaskPendingOps_ClaimBatch_ReclaimsStale verifies a claim older than
// staleBefore is re-claimable (crash recovery), while a fresh claim is not.
func TestTaskPendingOps_ClaimBatch_ReclaimsStale(t *testing.T) {
db := setupTaskQueueTestDB(t)
repo := NewTaskPendingOpsRepository(db)
ctx := context.Background()
require.NoError(t, repo.Enqueue(ctx, makePendingOp("wiki:ingest", "knowledge_base", "kb", "ingest", "k1", nil)))
// Claim it now.
got, err := repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, time.Now().Add(-time.Hour))
require.NoError(t, err)
require.Len(t, got, 1)
// A stale threshold in the past — the fresh claim is NOT stale yet.
got, err = repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, time.Now().Add(-time.Hour))
require.NoError(t, err)
assert.Len(t, got, 0, "fresh claim must not be re-claimed")
// A stale threshold in the future makes every existing claim stale.
got, err = repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, time.Now().Add(time.Hour))
require.NoError(t, err)
require.Len(t, got, 1, "stale claim must be recoverable")
assert.Equal(t, "k1", got[0].DedupKey)
}
// TestTaskPendingOps_ReleaseByIDs_ReturnsToPool verifies a released row
// becomes immediately claimable again without waiting for staleness.
func TestTaskPendingOps_ReleaseByIDs_ReturnsToPool(t *testing.T) {
db := setupTaskQueueTestDB(t)
repo := NewTaskPendingOpsRepository(db)
ctx := context.Background()
op := makePendingOp("wiki:ingest", "knowledge_base", "kb", "ingest", "k1", nil)
require.NoError(t, repo.Enqueue(ctx, op))
stale := time.Now().Add(-time.Hour)
got, err := repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, stale)
require.NoError(t, err)
require.Len(t, got, 1)
// Claimed → not re-claimable.
got, err = repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, stale)
require.NoError(t, err)
require.Len(t, got, 0)
// Release → immediately claimable again.
require.NoError(t, repo.ReleaseByIDs(ctx, nil)) // no-op tolerated
require.NoError(t, repo.ReleaseByIDs(ctx, []int64{op.ID}))
got, err = repo.ClaimBatch(ctx, "wiki:ingest", "knowledge_base", "kb", 10, stale)
require.NoError(t, err)
require.Len(t, got, 1)
assert.Equal(t, "k1", got[0].DedupKey)
}
// ---------------- TaskDeadLetterRepository ----------------
func makeDeadLetter(taskType, scope, scopeID, relatedID, lastErr string) *types.TaskDeadLetter {
+15 -1
View File
@@ -175,10 +175,24 @@ var registry = map[string]settingSpec{
Default: int64(32),
Category: "worker",
RequiresRestart: true,
Description: "异步任务 worker 并发数(asynq 线程池大小)。" +
Description: "文档解析池的 worker 并发数(asynq 线程池大小,不含 Wiki)。" +
"文档解析、嵌入等任务多为 I/O 等待,适当提高可缩短批量上传排队时间。" +
"修改后需重启服务进程方可生效。",
},
// asynq.wiki_concurrency is the size of the DEDICATED wiki worker pool,
// separate from asynq.concurrency. Read once when the wiki asynq server
// starts — changing it in the UI requires a process restart. Mirrors
// WEKNORA_WIKI_ASYNQ_CONCURRENCY (default 16).
"asynq.wiki_concurrency": {
Type: "int",
EnvName: "WEKNORA_WIKI_ASYNQ_CONCURRENCY",
Default: int64(16),
Category: "worker",
RequiresRestart: true,
Description: "Wiki 生成专用池的 worker 并发数(与文档解析池相互隔离)。" +
"Wiki 生成以合成大模型调用为主,独立并发预算可避免上传高峰期被解析任务饿死," +
"同时不会因 Wiki 洪峰拖慢用户面解析。修改后需重启服务进程方可生效。",
},
}
// systemSettingService wires the repository, audit log, and (P2)
+407 -30
View File
@@ -20,31 +20,97 @@ import (
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/google/uuid"
"github.com/hibiken/asynq"
"github.com/redis/go-redis/v9"
)
// ErrWikiIngestConcurrent is returned by the wiki ingest handler when another
// batch is already running for the same KB (i.e. the `wiki:active:<kbID>`
// Redis lock is held). The asynq server's RetryDelayFunc uses errors.Is on
// this sentinel to apply a short, fixed retry delay instead of asynq's default
// exponential backoff — otherwise a freshly orphaned lock (e.g. from a crash
// or restart) would force newcomers to wait minutes even after the lock
// naturally expires.
// ErrWikiIngestConcurrent is returned by the wiki ingest handler in Lite mode
// when another batch is already running for the same KB (the in-process
// liteLocks guard is held). The asynq/sync executor's RetryDelayFunc uses
// errors.Is on this sentinel to apply a short, fixed retry delay instead of
// exponential backoff, so the deferred batch retries promptly once the active
// one releases. Standard (Redis) mode no longer takes an exclusive per-KB
// lock (Phase 3) and never returns this.
var ErrWikiIngestConcurrent = errors.New("concurrent wiki task active")
const (
// maxContentForWiki limits the document content sent to LLM for wiki generation
maxContentForWiki = 32768
// wikiActiveKeyPrefix is the Redis key for the "batch in progress" flag.
// Key format: wiki:active:{kbID} → "1" with TTL. Prevents concurrent batches.
wikiActiveKeyPrefix = "wiki:active:"
// --- Phase 3: concurrent per-KB batches (standard/Redis mode) ---------
//
// Phase 3 removed the exclusive per-KB Redis "batch in progress" lock
// (wiki:active:<kbID>). Standard mode now allows concurrent batches for
// one KB, kept safe by row claiming (claimPendingList) plus per-slug
// reduce locks (withSlugLock). Lite mode still serializes per KB via the
// in-process liteLocks map.
//
// wikiClaimStaleAfter is how long a claimed-but-undrained ingest row
// waits before another worker may re-claim it. It MUST exceed the asynq
// task Timeout for wiki:ingest (60m) so a still-running batch's rows are
// never stolen mid-flight — only genuinely crashed/abandoned claims are
// recovered. This preserves the pre-claim crash behaviour where a dead
// batch's undeleted rows were simply re-peeked by the next trigger.
wikiClaimStaleAfter = 90 * time.Minute
// wikiSlugLockPrefix guards read-modify-write on a single shared wiki
// page (entity/concept/summary/index) so two concurrent batches for the
// same KB can't lost-update the same slug. Key: wiki:slug:{kbID}:{slug}.
wikiSlugLockPrefix = "wiki:slug:"
// wikiSlugLockTTL bounds the per-slug lock so a crashed reducer can't
// wedge a hot page forever. Comfortably longer than a single reduce
// (one LLM modify call).
wikiSlugLockTTL = 5 * time.Minute
// wikiSlugLockWait / Poll bound how long a reduce goroutine blocks
// waiting for a contended slug before falling back to a best-effort
// skip (matching the pre-existing reduce-failure semantics).
wikiSlugLockWait = 2 * time.Minute
wikiSlugLockPoll = 50 * time.Millisecond
// --- Phase 4: per-KB in-flight cap (standard/Redis mode) --------------
//
// Since Phase 3 lets many batches for one KB run at once, a single KB's
// bulk import could otherwise occupy the entire wiki pool and starve
// other KBs. Each running batch reserves a slot in a per-KB Redis sorted
// set (score = expiry). ZREMRANGEBYSCORE purges slots left behind by a
// crashed worker, so the cap is self-healing without an explicit lock.
//
// wikiInflightPrefix keys the per-KB slot set: wiki:inflight:{kbID}.
wikiInflightPrefix = "wiki:inflight:"
// wikiInflightDefault is the fallback cap when WikiConfig.IngestMaxInflight
// is unset. 4 leaves headroom in a default 16-worker wiki pool for other
// KBs while still giving one KB solid parallelism.
wikiInflightDefault = 4
// wikiInflightTTL is how long a reserved slot survives without renewal.
// MUST exceed wikiInflightRenew by a comfortable margin so a single
// missed renew (GC pause, Redis blip) doesn't drop a live batch's slot.
wikiInflightTTL = 90 * time.Second
// wikiInflightRenew is how often a running batch bumps its slot's expiry.
wikiInflightRenew = 30 * time.Second
// wikiInflightBackoff delays the follow-up trigger when a batch is turned
// away by the cap, so it retries after a slot plausibly frees.
wikiInflightBackoff = 10 * time.Second
// wikiIngestDelay is how long to wait after a document is added before
// the batch task fires. Debounces rapid uploads.
wikiIngestDelay = 30 * time.Second
// wikiFollowUpDelay is the normal light debounce before a follow-up
// batch drains the remaining pending rows. Standard mode already fans a
// KB's backlog across concurrent claiming batches, so this is just a
// short breather, not a lock-release wait.
wikiFollowUpDelay = 5 * time.Second
// wikiRateLimitBackoff is the (much longer) follow-up delay used when a
// batch's failures were caused by upstream rate limiting (HTTP 429 /
// quota). Retrying a rate-limited document at the normal 5s cadence just
// throws more requests at an already-saturated rpm budget — every retry
// re-issues extract + classify + summary calls for the doc — which keeps
// the limiter tripped and drags the whole KB's ingest out. Backing off
// gives the per-minute window time to reset before we try again.
wikiRateLimitBackoff = 60 * time.Second
// wikiMaxDocsPerBatch limits how many documents a single batch processes.
// Prevents unbounded execution time. Remaining ops stay in
// task_pending_ops and are picked up by the follow-up task.
@@ -75,20 +141,6 @@ const (
// exceed the longest plausible ingest run (LLM extraction + reduce).
wikiDeletedTTL = 1 * time.Hour
// wikiActiveLockTTL is the TTL for the per-KB "batch in progress" flag.
// Kept short (relative to total batch runtime) so that if the owning
// process crashes without running its `defer Del`, the orphaned lock
// expires quickly and newcomers aren't blocked. A periodic renew
// (wikiActiveLockRenew) keeps the lock alive while the handler is
// genuinely still running.
wikiActiveLockTTL = 60 * time.Second
// wikiActiveLockRenew is how often the in-flight handler bumps the TTL.
// Must be comfortably shorter than wikiActiveLockTTL so a single missed
// tick (GC pause, Redis blip) doesn't let the lock slip out from under a
// live handler.
wikiActiveLockRenew = 20 * time.Second
// wikiLLMMaxAttempts is the total attempt count (initial + retries) for
// every LLM call routed through generateWithTemplate. 3 was chosen to
// absorb transient 504/timeouts from upstream gateways without
@@ -108,8 +160,65 @@ const (
// wikiTaskScope is the scope used by both pending ops and dead letters.
// Wiki ingest is per-KB, so every op is scoped to a knowledge_base.
wikiTaskScope = types.TaskScopeKnowledgeBase
// --- Finalize lane (Phase 1: debounced KB-global convergence) ---------
//
// The KB-global convergence steps (index-intro rebuild, dead-link
// cleanup, cross-link injection) used to run at the tail of EVERY
// 5-doc ingest batch. On a bulk import that meant O(docs/batchSize)
// index rebuilds — each an LLM call plus an index-page write. They are
// now deferred to a debounced per-KB wiki:finalize task: each ingest
// batch records what changed into a separate lane of task_pending_ops
// (task_type=wiki:finalize) and schedules a coalesced trigger, so a
// burst of N documents rebuilds the index ONCE at the end.
// wikiFinalizeTaskType is the task_pending_ops lane for finalize work.
// Keyed by (task_type, scope, scope_id) like the ingest lane, but with a
// distinct task_type so PeekBatch never mixes the two.
wikiFinalizeTaskType = "wiki:finalize"
// wikiFinalizeOpSlug rows carry one affected page slug (+ its fresh title
// when this batch wrote it) for the dead-link / cross-link passes.
wikiFinalizeOpSlug = "slug"
// wikiFinalizeOpChange rows carry a doc-level add/remove change entry for
// the index-intro change description.
wikiFinalizeOpChange = "change"
wikiFinalizeAdded = "added"
wikiFinalizeRemoved = "removed"
// wikiFinalizeDelay debounces the finalize trigger so multiple ingest
// batches within the window coalesce into a single index rebuild.
wikiFinalizeDelay = 20 * time.Second
// wikiFinalizeMaxRows caps how many finalize rows one run drains. A run
// that hits the cap reschedules itself for the remainder.
wikiFinalizeMaxRows = 5000
// wikiFinalizeLockTTL / Renew guard against two finalize runs for the
// same KB writing the index page concurrently (belt-and-suspenders on
// top of the asynq.TaskID coalescing).
wikiFinalizeLockTTL = 60 * time.Second
wikiFinalizeLockRenew = 20 * time.Second
)
// wikiFinalizeChange is a doc-level add/remove entry for the index-intro
// change description, persisted as a finalize-lane row payload.
type wikiFinalizeChange struct {
Action string `json:"action"` // wikiFinalizeAdded | wikiFinalizeRemoved
DocTitle string `json:"doc_title,omitempty"`
DocSummary string `json:"doc_summary,omitempty"`
}
// wikiFinalizeRow is the JSON payload of a task_pending_ops row in the
// finalize lane. Exactly one of {Slug, Change} is set, distinguished by the
// row's Op column (wikiFinalizeOpSlug / wikiFinalizeOpChange).
type wikiFinalizeRow struct {
Slug string `json:"slug,omitempty"`
Title string `json:"title,omitempty"`
Change *wikiFinalizeChange `json:"change,omitempty"`
}
// WikiDeletedTombstoneKey returns the Redis key used to mark a knowledge as
// recently deleted, so wiki_ingest tasks in flight can short-circuit. Exposed
// so knowledgeService.cleanupWikiOnKnowledgeDelete can write the same key
@@ -212,6 +321,10 @@ type wikiIngestService struct {
// liteLocks provides per-KB mutual exclusion in Lite mode (no Redis).
// Keys are kbID strings; values are unused (presence = locked).
liteLocks sync.Map
// liteFinalizeLocks is the Lite-mode counterpart of the Redis
// wiki:finalize:active:<kbID> lock, keeping two finalize runs for the
// same KB from overlapping when there is no Redis.
liteFinalizeLocks sync.Map
}
// NewWikiIngestService creates a new wiki ingest service
@@ -343,7 +456,7 @@ func EnqueueWikiIngest(
triggerBytes, _ := json.Marshal(trigger)
t := asynq.NewTask(types.TypeWikiIngest, triggerBytes,
asynq.Queue("low"),
asynq.Queue(types.QueueWiki),
asynq.MaxRetry(wikiIngestMaxRetry),
asynq.Timeout(60*time.Minute),
asynq.ProcessIn(wikiIngestDelay),
@@ -401,7 +514,7 @@ func EnqueueWikiRetract(
langfuse.InjectTracing(ctx, &trigger)
triggerBytes, _ := json.Marshal(trigger)
t := asynq.NewTask(types.TypeWikiIngest, triggerBytes,
asynq.Queue("low"),
asynq.Queue(types.QueueWiki),
asynq.MaxRetry(wikiIngestMaxRetry),
asynq.Timeout(60*time.Minute),
asynq.ProcessIn(5*time.Second), // Retract can trigger the batch quickly
@@ -411,11 +524,95 @@ func EnqueueWikiRetract(
}
}
// Handle implements interfaces.TaskHandler for asynq task processing.
// Wiki ingest tasks are debounced via asynq.Unique + ProcessIn, so at most
// one ingest task runs per KB at a time. No distributed lock needed.
// Handle implements interfaces.TaskHandler for asynq task processing. The
// wiki service owns two task types, dispatched here by type so a single
// registered handler covers both the ingest batch and the debounced
// KB-global finalize pass.
func (s *wikiIngestService) Handle(ctx context.Context, t *asynq.Task) error {
return s.ProcessWikiIngest(ctx, t)
switch t.Type() {
case types.TypeWikiFinalize:
return s.ProcessWikiFinalize(ctx, t)
default:
return s.ProcessWikiIngest(ctx, t)
}
}
// enqueueFinalize persists this batch's KB-global convergence work into the
// finalize lane of task_pending_ops and schedules a debounced trigger. One
// "slug" row per affected page (carrying its fresh title when this batch wrote
// it, for cross-linking) plus one "change" row per doc added/removed (for the
// index-intro change description).
func (s *wikiIngestService) enqueueFinalize(
ctx context.Context,
payload WikiIngestPayload,
affectedSlugs []string,
freshTitleBySlug map[string]string,
changes []wikiFinalizeChange,
) {
if s.pendingRepo == nil {
return
}
for _, slug := range affectedSlugs {
row := wikiFinalizeRow{Slug: slug, Title: freshTitleBySlug[slug]}
b, err := json.Marshal(row)
if err != nil {
continue
}
if err := s.pendingRepo.Enqueue(ctx, &types.TaskPendingOp{
TenantID: payload.TenantID,
TaskType: wikiFinalizeTaskType,
Scope: wikiTaskScope,
ScopeID: payload.KnowledgeBaseID,
Op: wikiFinalizeOpSlug,
DedupKey: slug,
Payload: b,
}); err != nil {
logger.Warnf(ctx, "wiki finalize: enqueue slug row for %s failed: %v", slug, err)
}
}
for i := range changes {
row := wikiFinalizeRow{Change: &changes[i]}
b, err := json.Marshal(row)
if err != nil {
continue
}
if err := s.pendingRepo.Enqueue(ctx, &types.TaskPendingOp{
TenantID: payload.TenantID,
TaskType: wikiFinalizeTaskType,
Scope: wikiTaskScope,
ScopeID: payload.KnowledgeBaseID,
Op: wikiFinalizeOpChange,
DedupKey: "",
Payload: b,
}); err != nil {
logger.Warnf(ctx, "wiki finalize: enqueue change row failed: %v", err)
}
}
s.scheduleFinalize(ctx, payload)
}
// scheduleFinalize enqueues a debounced, coalesced KB-global finalize trigger.
// asynq.TaskID ("wiki-finalize-<kbID>") makes concurrent schedules within the
// debounce window collapse into one pending task; the conflict error is the
// expected coalescing signal, not a failure. In Lite mode the sync executor
// ignores TaskID, so finalize simply runs once per batch (acceptable at the
// small scale Lite mode targets).
func (s *wikiIngestService) scheduleFinalize(ctx context.Context, payload WikiIngestPayload) {
langfuse.InjectTracing(ctx, &payload)
b, _ := json.Marshal(payload)
t := asynq.NewTask(types.TypeWikiFinalize, b,
asynq.Queue(types.QueueWiki),
asynq.MaxRetry(wikiIngestMaxRetry),
asynq.Timeout(30*time.Minute),
asynq.ProcessIn(wikiFinalizeDelay),
asynq.TaskID("wiki-finalize-"+payload.KnowledgeBaseID),
)
if _, err := s.task.Enqueue(t); err != nil {
if errors.Is(err, asynq.ErrTaskIDConflict) || errors.Is(err, asynq.ErrDuplicateTask) {
return // a finalize is already scheduled/running for this KB — coalesced
}
logger.Warnf(ctx, "wiki finalize: schedule trigger failed: %v", err)
}
}
// peekPendingList loads up to `limit` ops from task_pending_ops for
@@ -441,6 +638,178 @@ func (s *wikiIngestService) peekPendingList(ctx context.Context, kbID string, li
logger.Warnf(ctx, "wiki ingest: failed to peek pending list: %v", err)
return nil, nil
}
return s.decodePendingRows(ctx, rows)
}
// claimPendingList is the standard-mode (Redis) counterpart of
// peekPendingList: it atomically CLAIMS up to `limit` ops (marks
// claimed_at) so concurrent batches for the same KB — allowed since Phase 3
// removed the exclusive per-KB lock — pull DISJOINT documents instead of
// double-processing. Stale claims (older than wikiClaimStaleAfter, i.e. from
// a crashed worker) are recovered. Dedup / peekedIDs semantics match
// peekPendingList; the returned peekedIDs are the claimed rows that the
// caller must DeleteByIDs on success or ReleaseByIDs to retry.
func (s *wikiIngestService) claimPendingList(ctx context.Context, kbID string, limit int) (ops []WikiPendingOp, peekedIDs []int64) {
if s.pendingRepo == nil {
return nil, nil
}
if limit <= 0 {
limit = wikiMaxDocsPerBatch
}
rows, err := s.pendingRepo.ClaimBatch(ctx, wikiTaskType, wikiTaskScope, kbID, limit,
time.Now().Add(-wikiClaimStaleAfter))
if err != nil {
logger.Warnf(ctx, "wiki ingest: failed to claim pending list: %v", err)
return nil, nil
}
return s.decodePendingRows(ctx, rows)
}
// withSlugLock serializes read-modify-write on one shared wiki page across
// concurrent batches. Phase 3 removed the exclusive per-KB batch lock, so
// two batches for the same KB can both produce updates for a shared
// entity/concept slug; without this lock their GetPageBySlug→UpdatePage
// cycles would race and lose one contribution.
//
// Redis mode only: in Lite mode a single process runs one batch per KB at a
// time (liteLocks), so same-slug contention cannot occur and fn runs
// directly. Returns (false, nil) if the lock could not be acquired within
// wikiSlugLockWait — the caller then treats the slug like a best-effort
// reduce miss (matching pre-existing reduce-failure handling). On a Redis
// error we fail OPEN (run fn unlocked): a rare lost-update on a shared page
// is tolerated by the finalize/dead-link passes, whereas silently dropping
// the update is strictly worse.
func (s *wikiIngestService) withSlugLock(ctx context.Context, kbID, slug string, fn func() error) (bool, error) {
if s.redisClient == nil {
return true, fn()
}
key := wikiSlugLockPrefix + kbID + ":" + slug
deadline := time.Now().Add(wikiSlugLockWait)
for {
ok, rerr := s.redisClient.SetNX(ctx, key, "1", wikiSlugLockTTL).Result()
if rerr != nil {
logger.Warnf(ctx, "wiki reduce: slug lock SetNX failed for %s: %v (running unlocked)", slug, rerr)
return true, fn()
}
if ok {
break
}
if time.Now().After(deadline) {
return false, nil
}
select {
case <-ctx.Done():
return false, ctx.Err()
case <-time.After(wikiSlugLockPoll):
}
}
defer s.redisClient.Del(context.Background(), key)
return true, fn()
}
// wikiInflightReserveScript atomically enforces the per-KB in-flight cap.
// It purges expired slots (crash recovery), counts live slots, and adds the
// caller's slot iff under the cap. Returning 1 = reserved, 0 = at cap.
// Keeping purge+count+add in one Lua call closes the check-then-act race two
// concurrent reservers would otherwise have.
const wikiInflightReserveScript = `
local now = tonumber(ARGV[1])
local expiry = tonumber(ARGV[2])
local maxInflight = tonumber(ARGV[3])
local token = ARGV[4]
local ttl = tonumber(ARGV[5])
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now)
if redis.call('ZCARD', KEYS[1]) >= maxInflight then
return 0
end
redis.call('ZADD', KEYS[1], expiry, token)
redis.call('PEXPIRE', KEYS[1], ttl)
return 1
`
// reserveInflightSlot claims one of KB's concurrent-batch slots (Phase 4,
// standard/Redis mode). Returns (release, true) when granted — release() MUST
// run when the batch finishes; (nil, false) when the KB is already at
// maxInflight, so the caller should reschedule and bail. A background renew
// keeps the slot alive for the batch's duration; a crashed batch's slot simply
// expires (wikiInflightTTL) and is purged by the next reserver. Lite mode has
// no shared-pool contention (liteLocks already serialize per KB), so it always
// grants a no-op slot. Fails OPEN on a Redis error: a blip must not halt wiki
// generation, and the pool size still bounds total work.
func (s *wikiIngestService) reserveInflightSlot(ctx context.Context, kbID string, maxInflight int) (func(), bool) {
if s.redisClient == nil || maxInflight <= 0 {
return func() {}, true
}
key := wikiInflightPrefix + kbID
token := uuid.New().String()
now := time.Now()
res, err := s.redisClient.Eval(ctx, wikiInflightReserveScript,
[]string{key},
now.UnixMilli(),
now.Add(wikiInflightTTL).UnixMilli(),
maxInflight,
token,
wikiInflightTTL.Milliseconds(),
).Int()
if err != nil {
logger.Warnf(ctx, "wiki ingest: inflight reserve failed for KB %s: %v (running uncapped)", kbID, err)
return func() {}, true
}
if res == 0 {
return nil, false
}
renewCtx, cancel := context.WithCancel(context.Background())
go func() {
ticker := time.NewTicker(wikiInflightRenew)
defer ticker.Stop()
for {
select {
case <-renewCtx.Done():
return
case <-ticker.C:
exp := float64(time.Now().Add(wikiInflightTTL).UnixMilli())
s.redisClient.ZAdd(context.Background(), key, redis.Z{Score: exp, Member: token})
s.redisClient.PExpire(context.Background(), key, wikiInflightTTL)
}
}
}()
return func() {
cancel()
s.redisClient.ZRem(context.Background(), key, token)
}, true
}
// scheduleCappedRetry enqueues a single coalesced follow-up trigger after a
// batch was turned away by the in-flight cap. asynq.TaskID collapses all
// turned-away triggers for one KB into a single pending retry (no thundering
// herd), and the running batches that hold the slots also chain their own
// follow-ups on completion, so the turned-away rows are guaranteed to drain
// once a slot frees.
func (s *wikiIngestService) scheduleCappedRetry(ctx context.Context, payload WikiIngestPayload) {
langfuse.InjectTracing(ctx, &payload)
b, _ := json.Marshal(payload)
t := asynq.NewTask(types.TypeWikiIngest, b,
asynq.Queue(types.QueueWiki),
asynq.MaxRetry(wikiIngestMaxRetry),
asynq.Timeout(60*time.Minute),
asynq.ProcessIn(wikiInflightBackoff),
asynq.TaskID("wiki-ingest-capped-"+payload.KnowledgeBaseID),
)
if _, err := s.task.Enqueue(t); err != nil {
if errors.Is(err, asynq.ErrTaskIDConflict) || errors.Is(err, asynq.ErrDuplicateTask) {
return // a capped retry is already pending for this KB — coalesced
}
logger.Warnf(ctx, "wiki ingest: capped-retry enqueue failed: %v", err)
}
}
// decodePendingRows converts raw task_pending_ops rows into WikiPendingOps,
// applying the last-write-wins dedup by knowledge_id. peekedIDs carries the
// db ids of EVERY row (including dedup-collapsed ones) so the caller can
// drain them all at trim time. Shared by peekPendingList (no claim) and
// claimPendingList (claimed rows).
func (s *wikiIngestService) decodePendingRows(ctx context.Context, rows []*types.TaskPendingOp) (ops []WikiPendingOp, peekedIDs []int64) {
if len(rows) == 0 {
return nil, nil
}
@@ -562,6 +931,14 @@ func (s *wikiIngestService) requeueFailedOps(ctx context.Context, payload WikiIn
continue
}
if count <= wikiMaxFailRetries {
// Release the claim so the row is immediately eligible for the
// next trigger's ClaimBatch instead of waiting out
// wikiClaimStaleAfter. No-op in Lite mode (row was peeked, never
// claimed). ReleaseByIDs preserves fail_count, so the retry
// budget still counts down.
if err := s.pendingRepo.ReleaseByIDs(ctx, []int64{op.dbID}); err != nil {
logger.Warnf(ctx, "wiki ingest: failed to release claim for retry id=%d: %v", op.dbID, err)
}
logger.Infof(ctx, "wiki ingest: re-queued failed op %s (%s) for retry (attempt %d/%d)", op.KnowledgeID, op.DocTitle, count, wikiMaxFailRetries)
continue
}
+470 -278
View File
@@ -23,10 +23,19 @@ import (
// still pending ops in task_pending_ops for this KB. Returns true when
// a follow-up was scheduled.
//
// We use a short ProcessIn (5s) so the active-batch lock has time to
// release before the next worker tries to acquire it; otherwise we'd
// just bounce on ErrWikiIngestConcurrent and burn an asynq retry slot.
func (s *wikiIngestService) scheduleFollowUp(ctx context.Context, payload WikiIngestPayload) bool {
// Post-Phase-3 this only backstops the case where a batch drained its
// claimed window but more rows remain and no other trigger is pending
// (e.g. steady trickle of uploads). Standard mode already fans a KB's
// backlog across concurrent claiming batches, so the short delay is
// normally just a light debounce rather than a lock-release wait.
//
// `delay` is the ProcessIn before the follow-up fires. Callers pass
// wikiFollowUpDelay for the normal case and wikiRateLimitBackoff when the
// batch tripped an upstream rate limit — the released failed rows are
// eligible immediately, but nothing claims them until a trigger fires, so
// stretching the follow-up interval is what actually paces retries down
// during a 429 storm.
func (s *wikiIngestService) scheduleFollowUp(ctx context.Context, payload WikiIngestPayload, delay time.Duration) bool {
if s.pendingRepo == nil {
return false
}
@@ -35,15 +44,15 @@ func (s *wikiIngestService) scheduleFollowUp(ctx context.Context, payload WikiIn
return false
}
logger.Infof(ctx, "wiki ingest: %d more documents pending for KB %s, scheduling follow-up", count, payload.KnowledgeBaseID)
logger.Infof(ctx, "wiki ingest: %d more documents pending for KB %s, scheduling follow-up in %s", count, payload.KnowledgeBaseID, delay)
langfuse.InjectTracing(ctx, &payload)
payloadBytes, _ := json.Marshal(payload)
t := asynq.NewTask(types.TypeWikiIngest, payloadBytes,
asynq.Queue("low"),
asynq.Queue(types.QueueWiki),
asynq.MaxRetry(wikiIngestMaxRetry),
asynq.Timeout(60*time.Minute),
asynq.ProcessIn(5*time.Second), // short delay — active flag will be released by then
asynq.ProcessIn(delay), // debounce (or rate-limit backoff) before draining the remainder
)
if _, err := s.task.Enqueue(t); err != nil {
logger.Warnf(ctx, "wiki ingest: follow-up enqueue failed: %v", err)
@@ -52,205 +61,17 @@ func (s *wikiIngestService) scheduleFollowUp(ctx context.Context, payload WikiIn
return true
}
func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task) error {
taskStartedAt := time.Now()
retryCount, _ := asynq.GetRetryCount(ctx)
maxRetry, _ := asynq.GetMaxRetry(ctx)
var payload WikiIngestPayload
exitStatus := "success"
mode := "redis"
lockAcquired := false
pendingOpsCount := 0
ingestOps := 0
retractOps := 0
ingestSucceeded := 0
ingestFailed := 0
retractHandled := 0
indexRebuildAttempted := false
indexRebuildSucceeded := false
followUpScheduled := false
totalPagesAffected := 0
docPreview := make([]string, 0, 6)
// Tunables resolved from KB.WikiConfig once we've loaded the KB.
// Captured up here so the deferred stats log can observe them
// regardless of which exit path we took.
loggedBatchSize := 0
loggedMapPar := 0
loggedReducePar := 0
defer func() {
logger.Infof(
ctx,
"wiki ingest stats: kb=%s tenant=%d retry=%d/%d status=%s elapsed=%s mode=%s lock_acquired=%v pending_ops=%d ops(ingest=%d,retract=%d) ingest(success=%d,failed=%d) retract_handled=%d pages(total=%d) index(rebuild_attempted=%v,rebuild_succeeded=%v) followup=%v tunables(batch=%d,map_par=%d,reduce_par=%d) preview=%s",
payload.KnowledgeBaseID,
payload.TenantID,
retryCount,
maxRetry,
exitStatus,
time.Since(taskStartedAt).Round(time.Millisecond),
mode,
lockAcquired,
pendingOpsCount,
ingestOps,
retractOps,
ingestSucceeded,
ingestFailed,
retractHandled,
totalPagesAffected,
indexRebuildAttempted,
indexRebuildSucceeded,
followUpScheduled,
loggedBatchSize,
loggedMapPar,
loggedReducePar,
previewStringSlice(docPreview, 6),
)
}()
if err := json.Unmarshal(t.Payload(), &payload); err != nil {
exitStatus = "invalid_payload"
return fmt.Errorf("wiki ingest: unmarshal payload: %w", err)
}
// Inject context
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
// Try to acquire the "active batch" flag (non-blocking).
//
// TTL is intentionally short (wikiActiveLockTTL ≈ 60s) so that if the
// owning process dies without releasing the lock (crash, kill -9,
// container restart), the orphaned key expires within ~1 minute and new
// tasks aren't starved. A renew goroutine keeps the lock alive while
// the handler is genuinely running.
if s.redisClient != nil {
activeKey := wikiActiveKeyPrefix + payload.KnowledgeBaseID
acquired, err := s.redisClient.SetNX(ctx, activeKey, "1", wikiActiveLockTTL).Result()
if err != nil {
logger.Warnf(ctx, "wiki ingest: redis SetNX failed: %v", err)
} else if !acquired {
exitStatus = "active_lock_conflict"
// If task_pending_ops is already empty for this KB, the active
// batch will drain whatever was queued. Returning nil avoids
// burning through the retry budget on tasks that would be
// no-ops when they eventually acquire the lock. If rows still
// remain, retry so we don't miss them in case the active
// batch drained its peek before our op landed.
n, nErr := s.pendingRepo.PendingCount(ctx, wikiTaskType, wikiTaskScope, payload.KnowledgeBaseID)
if nErr != nil {
logger.Warnf(ctx, "wiki ingest: failed to read pending count during lock conflict for KB %s: %v", payload.KnowledgeBaseID, nErr)
logger.Infof(ctx, "wiki ingest: another batch active for KB %s, deferring to asynq retry", payload.KnowledgeBaseID)
return ErrWikiIngestConcurrent
}
if n == 0 {
exitStatus = "active_lock_conflict_empty"
logger.Infof(ctx, "wiki ingest: concurrent batch active for KB %s, pending queue empty — skipping", payload.KnowledgeBaseID)
return nil
}
logger.Infof(ctx, "wiki ingest: another batch active for KB %s, deferring to asynq retry", payload.KnowledgeBaseID)
return ErrWikiIngestConcurrent
}
lockAcquired = acquired
lockCtx, cancelLock := context.WithCancel(context.Background())
defer func() {
cancelLock()
s.redisClient.Del(context.Background(), activeKey)
}()
go func() {
ticker := time.NewTicker(wikiActiveLockRenew)
defer ticker.Stop()
for {
select {
case <-lockCtx.Done():
return
case <-ticker.C:
s.redisClient.Expire(context.Background(), activeKey, wikiActiveLockTTL)
}
}
}()
} else {
mode = "lite"
// In-process mutual exclusion: mirrors the Redis SetNX lock above.
if _, loaded := s.liteLocks.LoadOrStore(payload.KnowledgeBaseID, struct{}{}); loaded {
exitStatus = "active_lock_conflict"
logger.Infof(ctx, "wiki ingest: another batch active for KB %s (lite lock), deferring to asynq retry", payload.KnowledgeBaseID)
return ErrWikiIngestConcurrent
}
lockAcquired = true
defer s.liteLocks.Delete(payload.KnowledgeBaseID)
}
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
if err != nil {
exitStatus = "get_kb_failed"
return fmt.Errorf("wiki ingest: get KB: %w", err)
}
if !kb.IsWikiEnabled() {
exitStatus = "kb_not_wiki_enabled"
return fmt.Errorf("wiki ingest: KB %s is not wiki type", kb.ID)
}
var synthesisModelID string
if kb.WikiConfig != nil {
synthesisModelID = kb.WikiConfig.SynthesisModelID
}
if synthesisModelID == "" {
synthesisModelID = kb.SummaryModelID
}
if synthesisModelID == "" {
exitStatus = "missing_synthesis_model"
return fmt.Errorf("wiki ingest: no synthesis model configured for KB %s", kb.ID)
}
chatModel, err := s.modelService.GetChatModel(ctx, synthesisModelID)
if err != nil {
exitStatus = "get_chat_model_failed"
return fmt.Errorf("wiki ingest: get chat model: %w", err)
}
// Resolve per-KB tunables once. WikiConfig.IngestBatchSize /
// IngestMapParallel / IngestReduceParallel let operators on
// 4w-document KBs raise the throughput knob (more docs per batch +
// more concurrent LLM calls) without a code deploy. Zero falls back
// to the historical defaults so existing KBs see no behaviour
// change until they opt in.
batchSize := kb.WikiConfig.IngestBatchSizeOrDefault(wikiMaxDocsPerBatch)
mapParallel := kb.WikiConfig.IngestMapParallelOrDefault(10)
reduceParallel := kb.WikiConfig.IngestReduceParallelOrDefault(10)
loggedBatchSize = batchSize
loggedMapPar = mapParallel
loggedReducePar = reduceParallel
lang := types.LanguageNameFromContext(ctx)
pendingOps, peekedIDs := s.peekPendingList(ctx, payload.KnowledgeBaseID, batchSize)
pendingOpsCount = len(pendingOps)
if len(pendingOps) == 0 {
exitStatus = "no_pending_ops"
logger.Infof(ctx, "wiki ingest: no pending operations for KB %s", payload.KnowledgeBaseID)
return nil
}
logger.Infof(ctx, "wiki ingest: batch processing %d ops for KB %s", len(pendingOps), payload.KnowledgeBaseID)
// Resolve extraction granularity once per batch. Historical rows with
// empty/unknown values fall back to Standard via Normalize(). Failures
// to load the KB (unlikely since we're already acting on it) also
// degrade gracefully to Standard.
granularity := types.WikiExtractionStandard
if kb, kbErr := s.kbService.GetKnowledgeBaseByID(ctx, payload.KnowledgeBaseID); kbErr == nil && kb != nil && kb.WikiConfig != nil {
granularity = kb.WikiConfig.ExtractionGranularity.Normalize()
}
// Build the per-batch lazy fetchers. These replace the legacy
// pre-batch ListAllPages dump: instead of pulling ~100MB of rows
// up front (and walking them several more times during the batch),
// callers pay only for the slugs / knowledge ids they actually
// reach for. Cache hits keep repeat lookups within the batch free.
// newWikiBatchContext builds the per-run lazy fetchers used by both the ingest
// batch and the debounced finalize task. These replace the legacy pre-batch
// ListAllPages dump: instead of pulling ~100MB of rows up front (and walking
// them several more times), callers pay only for the slugs / knowledge ids they
// actually reach for. Cache hits keep repeat lookups within a single run free.
// The cache is per-call (goroutine-safe via fetchMu), so each task gets a fresh,
// isolated view.
func (s *wikiIngestService) newWikiBatchContext(
kbID string,
granularity types.WikiExtractionGranularity,
) *WikiBatchContext {
var (
fetchMu sync.Mutex
slugTitleCache = make(map[string]string) // slug -> title; "" = known-missing
@@ -270,7 +91,7 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
fetchMu.Unlock()
if len(need) > 0 {
pages, err := s.wikiService.ListBySlugs(ctx, payload.KnowledgeBaseID, need)
pages, err := s.wikiService.ListBySlugs(ctx, kbID, need)
if err != nil {
logger.Warnf(ctx, "wiki ingest: ListBySlugs(%d slugs) failed: %v", len(need), err)
}
@@ -318,7 +139,7 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
fetchMu.Unlock()
if len(need) > 0 {
contents, err := s.wikiService.ListSummariesByKnowledgeIDs(ctx, payload.KnowledgeBaseID, need)
contents, err := s.wikiService.ListSummariesByKnowledgeIDs(ctx, kbID, need)
if err != nil {
logger.Warnf(ctx, "wiki ingest: ListSummariesByKnowledgeIDs(%d kids) failed: %v", len(need), err)
}
@@ -344,7 +165,7 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
return out
}
batchCtx := &WikiBatchContext{
return &WikiBatchContext{
SlugTitle: func(ctx context.Context, slug string) string {
m := resolveSlugs(ctx, []string{slug})
return m[slug]
@@ -356,13 +177,194 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
},
ExtractionGranularity: granularity,
}
}
func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task) error {
taskStartedAt := time.Now()
retryCount, _ := asynq.GetRetryCount(ctx)
maxRetry, _ := asynq.GetMaxRetry(ctx)
var payload WikiIngestPayload
exitStatus := "success"
mode := "redis"
pendingOpsCount := 0
ingestOps := 0
retractOps := 0
ingestSucceeded := 0
ingestFailed := 0
retractHandled := 0
followUpScheduled := false
totalPagesAffected := 0
docPreview := make([]string, 0, 6)
// Tunables resolved from KB.WikiConfig once we've loaded the KB.
// Captured up here so the deferred stats log can observe them
// regardless of which exit path we took. (Index-intro rebuild moved to
// the debounced wiki:finalize task in Phase 1, so it's logged there, not
// here; the exclusive per-KB lock was removed in Phase 3.)
loggedBatchSize := 0
loggedMapPar := 0
loggedReducePar := 0
loggedMaxInflight := 0
defer func() {
logger.Infof(
ctx,
"wiki ingest stats: kb=%s tenant=%d retry=%d/%d status=%s elapsed=%s mode=%s pending_ops=%d ops(ingest=%d,retract=%d) ingest(success=%d,failed=%d) retract_handled=%d pages(total=%d) followup=%v tunables(batch=%d,map_par=%d,reduce_par=%d,max_inflight=%d) preview=%s",
payload.KnowledgeBaseID,
payload.TenantID,
retryCount,
maxRetry,
exitStatus,
time.Since(taskStartedAt).Round(time.Millisecond),
mode,
pendingOpsCount,
ingestOps,
retractOps,
ingestSucceeded,
ingestFailed,
retractHandled,
totalPagesAffected,
followUpScheduled,
loggedBatchSize,
loggedMapPar,
loggedReducePar,
loggedMaxInflight,
previewStringSlice(docPreview, 6),
)
}()
if err := json.Unmarshal(t.Payload(), &payload); err != nil {
exitStatus = "invalid_payload"
return fmt.Errorf("wiki ingest: unmarshal payload: %w", err)
}
// Inject context
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
// Concurrency model (Phase 3):
//
// - Standard (Redis) mode: NO exclusive per-KB lock. Multiple batches
// for the same KB may run at once, each claiming a DISJOINT set of
// rows via claimPendingList (below). This lets one KB's backlog fan
// out across the whole wiki worker pool instead of draining one
// 5-doc batch at a time. Same-slug reduce safety is provided by
// withSlugLock, not by a batch-wide lock.
//
// - Lite mode (no Redis, single process): keep the in-process
// liteLocks guard so exactly one batch per KB runs at a time. Lite
// targets small/local scale where serial-per-KB is simplest and the
// claiming machinery (which needs FOR UPDATE SKIP LOCKED) buys
// nothing.
if s.redisClient == nil {
mode = "lite"
if _, loaded := s.liteLocks.LoadOrStore(payload.KnowledgeBaseID, struct{}{}); loaded {
exitStatus = "active_lock_conflict"
logger.Infof(ctx, "wiki ingest: another batch active for KB %s (lite lock), deferring to asynq retry", payload.KnowledgeBaseID)
return ErrWikiIngestConcurrent
}
defer s.liteLocks.Delete(payload.KnowledgeBaseID)
}
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
if err != nil {
exitStatus = "get_kb_failed"
return fmt.Errorf("wiki ingest: get KB: %w", err)
}
if !kb.IsWikiEnabled() {
exitStatus = "kb_not_wiki_enabled"
return fmt.Errorf("wiki ingest: KB %s is not wiki type", kb.ID)
}
var synthesisModelID string
if kb.WikiConfig != nil {
synthesisModelID = kb.WikiConfig.SynthesisModelID
}
if synthesisModelID == "" {
synthesisModelID = kb.SummaryModelID
}
if synthesisModelID == "" {
exitStatus = "missing_synthesis_model"
return fmt.Errorf("wiki ingest: no synthesis model configured for KB %s", kb.ID)
}
chatModel, err := s.modelService.GetChatModel(ctx, synthesisModelID)
if err != nil {
exitStatus = "get_chat_model_failed"
return fmt.Errorf("wiki ingest: get chat model: %w", err)
}
// Resolve per-KB tunables once. WikiConfig.IngestBatchSize /
// IngestMapParallel / IngestReduceParallel let operators on
// 4w-document KBs raise the throughput knob (more docs per batch +
// more concurrent LLM calls) without a code deploy. Zero falls back
// to the historical defaults so existing KBs see no behaviour
// change until they opt in.
batchSize := kb.WikiConfig.IngestBatchSizeOrDefault(wikiMaxDocsPerBatch)
mapParallel := kb.WikiConfig.IngestMapParallelOrDefault(10)
reduceParallel := kb.WikiConfig.IngestReduceParallelOrDefault(10)
loggedBatchSize = batchSize
loggedMapPar = mapParallel
loggedReducePar = reduceParallel
lang := types.LanguageNameFromContext(ctx)
// Per-KB in-flight cap (Phase 4, standard mode): keep one KB's bulk
// import from monopolizing the shared wiki pool. If the KB is already at
// its cap we schedule a coalesced retry and bail without claiming, so the
// rows stay available for whichever running batch frees a slot first.
maxInflight := kb.WikiConfig.IngestMaxInflightOrDefault(wikiInflightDefault)
loggedMaxInflight = maxInflight
releaseSlot, gotSlot := s.reserveInflightSlot(ctx, payload.KnowledgeBaseID, maxInflight)
if !gotSlot {
exitStatus = "inflight_cap"
logger.Infof(ctx, "wiki ingest: KB %s at in-flight cap (%d), rescheduling", payload.KnowledgeBaseID, maxInflight)
s.scheduleCappedRetry(ctx, payload)
return nil
}
defer releaseSlot()
// Standard mode claims rows (marks claimed_at, disjoint across concurrent
// batches); Lite mode peeks under its in-process lock.
var pendingOps []WikiPendingOp
var peekedIDs []int64
if s.redisClient != nil {
pendingOps, peekedIDs = s.claimPendingList(ctx, payload.KnowledgeBaseID, batchSize)
} else {
pendingOps, peekedIDs = s.peekPendingList(ctx, payload.KnowledgeBaseID, batchSize)
}
pendingOpsCount = len(pendingOps)
if len(pendingOps) == 0 {
exitStatus = "no_pending_ops"
logger.Infof(ctx, "wiki ingest: no pending operations for KB %s", payload.KnowledgeBaseID)
return nil
}
logger.Infof(ctx, "wiki ingest: batch processing %d ops for KB %s", len(pendingOps), payload.KnowledgeBaseID)
// Resolve extraction granularity once per batch. Historical rows with
// empty/unknown values fall back to Standard via Normalize(). Failures
// to load the KB (unlikely since we're already acting on it) also
// degrade gracefully to Standard.
granularity := types.WikiExtractionStandard
if kb, kbErr := s.kbService.GetKnowledgeBaseByID(ctx, payload.KnowledgeBaseID); kbErr == nil && kb != nil && kb.WikiConfig != nil {
granularity = kb.WikiConfig.ExtractionGranularity.Normalize()
}
batchCtx := s.newWikiBatchContext(payload.KnowledgeBaseID, granularity)
// 1. MAP PHASE (Parallel extraction and generation of updates)
var mapMu sync.Mutex
var failedOps []WikiPendingOp
slugUpdates := make(map[string][]SlugUpdate)
var docResults []*docIngestResult
var retractChangeDesc strings.Builder
// rateLimited flips true when any map/reduce LLM failure looks like an
// upstream 429/quota trip. It steers the follow-up scheduler onto the
// longer wikiRateLimitBackoff so retries don't keep hammering an already
// saturated rpm budget. Guarded by mapMu (map phase) / reduceMu (reduce
// phase) — both are held when written.
rateLimited := false
eg, mapCtx := errgroup.WithContext(ctx)
eg.SetLimit(mapParallel) // Map phase limit (configurable via WikiConfig)
@@ -416,7 +418,6 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
retractOps++
retractHandled++
docPreview = append(docPreview, fmt.Sprintf("retract[%s]: %s (%d slugs)", previewText(op.KnowledgeID, 24), previewText(op.DocTitle, 48), len(slugSet)))
fmt.Fprintf(&retractChangeDesc, "<document_removed>\n<title>%s</title>\n<summary>%s</summary>\n</document_removed>\n\n", op.DocTitle, op.DocSummary)
for slug := range slugSet {
slugUpdates[slug] = append(slugUpdates[slug], SlugUpdate{
@@ -443,6 +444,9 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
mapMu.Lock()
ingestFailed++
failedOps = append(failedOps, op)
if isLikelyRateLimitError(err) {
rateLimited = true
}
mapMu.Unlock()
logger.Warnf(mapCtx, "wiki ingest: failed to map knowledge %s: %v", op.KnowledgeID, err)
return nil // Don't fail the whole batch
@@ -524,9 +528,37 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
slug := slug
updates := updates
egReduce.Go(func() error {
changed, affectedType, additionFailed, err := s.reduceSlugUpdates(reduceCtx, chatModel, payload.KnowledgeBaseID, slug, updates, payload.TenantID, batchCtx, kidToWikiSpan)
if err != nil {
logger.Warnf(reduceCtx, "wiki ingest: reduce failed for slug %s: %v", slug, err)
var (
changed bool
affectedType string
additionFailed bool
reduceErr error
)
// Serialize same-slug read-modify-write across concurrent batches
// (standard mode). runs fn directly in Lite mode.
acquired, lockErr := s.withSlugLock(reduceCtx, payload.KnowledgeBaseID, slug, func() error {
changed, affectedType, additionFailed, reduceErr = s.reduceSlugUpdates(
reduceCtx, chatModel, payload.KnowledgeBaseID, slug, updates, payload.TenantID, batchCtx, kidToWikiSpan)
return reduceErr
})
if lockErr != nil {
// ctx cancelled (batch timeout / shutdown) — stop quietly.
return nil
}
if !acquired {
// Contended slug we couldn't get in time. Best-effort skip,
// matching reduce-failure semantics: the page keeps its prior
// content and the finalize/dead-link passes reconcile later.
logger.Warnf(reduceCtx, "wiki ingest: slug %s busy > %s, deferring update", slug, wikiSlugLockWait)
return nil
}
if reduceErr != nil {
logger.Warnf(reduceCtx, "wiki ingest: reduce failed for slug %s: %v", slug, reduceErr)
if isLikelyRateLimitError(reduceErr) {
reduceMu.Lock()
rateLimited = true
reduceMu.Unlock()
}
}
if changed {
reduceMu.Lock()
@@ -612,74 +644,55 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
}
}
// Build change description for the Index Intro LLM prompt
var changeDesc strings.Builder
if len(docResults) > 0 {
for _, r := range docResults {
fmt.Fprintf(&changeDesc, "<document_added>\n<title>%s</title>\n<summary>%s</summary>\n</document_added>\n\n", r.DocTitle, r.Summary)
}
}
if retractChangeDesc.Len() > 0 {
changeDesc.WriteString(retractChangeDesc.String())
}
// Rebuild index page
if changeDesc.Len() > 0 {
indexRebuildAttempted = true
logger.Infof(ctx, "wiki ingest: rebuilding index page")
if err := s.rebuildIndexPage(ctx, chatModel, payload, changeDesc.String(), lang); err != nil {
logger.Warnf(ctx, "wiki ingest: rebuild index failed: %v", err)
docPreview = append(docPreview, fmt.Sprintf("index_change=%s", previewText(changeDesc.String(), 160)))
} else {
indexRebuildSucceeded = true
docPreview = append(docPreview, fmt.Sprintf("index_change=%s", previewText(changeDesc.String(), 160)))
}
}
// Clean dead [[slug]] references whenever ANY page was touched this
// batch (not just retracts). Reduce-phase failures can leave stale
// references in pages we just rewrote (e.g. summary pages cite
// failed entity slugs); sanitizeDeadSummaryLinks above handles the
// well-known summary case, and this pass is the safety net for the
// long tail (cross-doc citations, prior batches' lingering refs).
// Dead-link cleanup: scoped to this batch's affected pages so the
// pass scales with batch size, not with KB size. The lint
// AutoFix path takes care of long-tail cleanup across the whole
// KB out-of-band.
// Publish freshly-generated pages immediately (NOT deferred to finalize):
// users should see a document's wiki pages as soon as their content is
// written, not after the debounce window. This is a cheap status flip.
if len(allPagesAffected) > 0 {
logger.Infof(ctx, "wiki ingest: cleaning dead links")
s.cleanDeadLinks(ctx, payload.KnowledgeBaseID, allPagesAffected, batchCtx)
}
if len(allPagesAffected) > 0 {
// Build the freshRefs set: every (slug, title) pair this batch
// successfully wrote, minus any that landed in failedAdditionSlugs.
// These are the "newly-mentionable" pages — links to them will
// not have appeared in older content yet, so injectCrossLinks
// targets exactly the affected pages with this fresh ref set.
freshRefs := make([]linkRef, 0, len(docResults)*4)
for _, dr := range docResults {
if dr == nil {
continue
}
for _, p := range dr.Pages {
if p.Slug == "" || p.Title == "" {
continue
}
if _, bad := failedAdditionSlugs[p.Slug]; bad {
continue
}
freshRefs = append(freshRefs, linkRef{slug: p.Slug, matchText: p.Title})
}
}
logger.Infof(ctx, "wiki ingest: injecting cross links")
s.injectCrossLinks(ctx, payload.KnowledgeBaseID, allPagesAffected, freshRefs, batchCtx)
logger.Infof(ctx, "wiki ingest: publishing draft pages")
s.publishDraftPages(ctx, payload.KnowledgeBaseID, allPagesAffected)
}
// Defer KB-global convergence (index-intro rebuild + dead-link cleanup +
// cross-link injection) to a debounced per-KB wiki:finalize task instead of
// running it at the tail of every 5-doc batch. We record what changed into
// the finalize lane and schedule a coalesced trigger; a burst of N documents
// then rebuilds the index ONCE. See ProcessWikiFinalize.
//
// freshTitleBySlug carries the (slug → title) pairs this batch successfully
// wrote (minus reduce-phase failures) so finalize's cross-link pass can
// linkify mentions of the new pages.
freshTitleBySlug := make(map[string]string, len(docResults)*4)
for _, dr := range docResults {
if dr == nil {
continue
}
for _, p := range dr.Pages {
if p.Slug == "" || p.Title == "" {
continue
}
if _, bad := failedAdditionSlugs[p.Slug]; bad {
continue
}
freshTitleBySlug[p.Slug] = p.Title
}
}
if len(allPagesAffected) > 0 || len(docResults) > 0 {
changes := make([]wikiFinalizeChange, 0, len(docResults)+len(pendingOps))
for _, r := range docResults {
changes = append(changes, wikiFinalizeChange{
Action: wikiFinalizeAdded, DocTitle: r.DocTitle, DocSummary: r.Summary,
})
}
for _, op := range pendingOps {
if op.Op == WikiOpRetract {
changes = append(changes, wikiFinalizeChange{
Action: wikiFinalizeRemoved, DocTitle: op.DocTitle, DocSummary: op.DocSummary,
})
}
}
s.enqueueFinalize(ctx, payload, allPagesAffected, freshTitleBySlug, changes)
}
// Close postprocess.wiki spans for every successfully-mapped doc.
// Span duration now spans map + reduce + index rebuild + cleanup +
// cross-link injection + publish, matching the wall-clock window
@@ -765,7 +778,186 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
logger.Infof(ctx, "wiki ingest: batch completed for KB %s, %d ops, %d pages affected", payload.KnowledgeBaseID, len(pendingOps), len(allPagesAffected))
followUpScheduled = s.scheduleFollowUp(ctx, payload)
// Pace the follow-up: on a rate-limit trip, back off so the per-minute
// window can reset instead of retrying the failed docs immediately.
followUpDelay := wikiFollowUpDelay
if rateLimited {
followUpDelay = wikiRateLimitBackoff
logger.Warnf(ctx, "wiki ingest: KB %s hit upstream rate limiting, backing off follow-up to %s", payload.KnowledgeBaseID, followUpDelay)
}
followUpScheduled = s.scheduleFollowUp(ctx, payload, followUpDelay)
return nil
}
// ProcessWikiFinalize runs the debounced, per-KB KB-global convergence pass:
// index-intro rebuild, dead-link cleanup, and cross-link injection. It drains
// the finalize lane of task_pending_ops (written by ProcessWikiIngest via
// enqueueFinalize) so a burst of documents rebuilds the index ONCE instead of
// once per 5-doc batch.
func (s *wikiIngestService) ProcessWikiFinalize(ctx context.Context, t *asynq.Task) error {
startedAt := time.Now()
var payload WikiIngestPayload
if err := json.Unmarshal(t.Payload(), &payload); err != nil {
return fmt.Errorf("wiki finalize: unmarshal payload: %w", err)
}
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
if s.pendingRepo == nil {
return nil
}
// Per-KB finalize lock, separate from the ingest active lock so finalize
// and ingest batches never block each other. Coalescing via asynq.TaskID
// already keeps at most one finalize pending per KB; this guards the
// reschedule-overlap window against a concurrent index-page write.
if s.redisClient != nil {
key := "wiki:finalize:active:" + payload.KnowledgeBaseID
acquired, err := s.redisClient.SetNX(ctx, key, "1", wikiFinalizeLockTTL).Result()
if err != nil {
logger.Warnf(ctx, "wiki finalize: SetNX failed: %v", err)
} else if !acquired {
// Another finalize is running; it will drain the lane and
// reschedule if rows remain. Safe to no-op.
return nil
}
lockCtx, cancelLock := context.WithCancel(context.Background())
defer func() {
cancelLock()
s.redisClient.Del(context.Background(), key)
}()
go func() {
ticker := time.NewTicker(wikiFinalizeLockRenew)
defer ticker.Stop()
for {
select {
case <-lockCtx.Done():
return
case <-ticker.C:
s.redisClient.Expire(context.Background(), key, wikiFinalizeLockTTL)
}
}
}()
} else {
if _, loaded := s.liteFinalizeLocks.LoadOrStore(payload.KnowledgeBaseID, struct{}{}); loaded {
return nil
}
defer s.liteFinalizeLocks.Delete(payload.KnowledgeBaseID)
}
rows, err := s.pendingRepo.PeekBatch(ctx, wikiFinalizeTaskType, wikiTaskScope, payload.KnowledgeBaseID, wikiFinalizeMaxRows)
if err != nil {
return fmt.Errorf("wiki finalize: peek: %w", err)
}
if len(rows) == 0 {
return nil
}
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
if err != nil {
return fmt.Errorf("wiki finalize: get KB: %w", err)
}
// Aggregate the drained rows into: affected slugs (dedup), fresh cross-link
// refs, and the index-intro change description. We collect ids up front so
// we can drain the lane even on the KB-disabled short-circuit below.
ids := make([]int64, 0, len(rows))
affectedSet := make(map[string]struct{}, len(rows))
var affectedSlugs []string
var freshRefs []linkRef
var changeDesc strings.Builder
for _, r := range rows {
ids = append(ids, r.ID)
if len(r.Payload) == 0 {
continue
}
var row wikiFinalizeRow
if err := json.Unmarshal(r.Payload, &row); err != nil {
logger.Warnf(ctx, "wiki finalize: unmarshal row id=%d failed: %v", r.ID, err)
continue
}
if row.Change != nil {
if row.Change.Action == wikiFinalizeRemoved {
fmt.Fprintf(&changeDesc, "<document_removed>\n<title>%s</title>\n<summary>%s</summary>\n</document_removed>\n\n", row.Change.DocTitle, row.Change.DocSummary)
} else {
fmt.Fprintf(&changeDesc, "<document_added>\n<title>%s</title>\n<summary>%s</summary>\n</document_added>\n\n", row.Change.DocTitle, row.Change.DocSummary)
}
continue
}
if row.Slug != "" {
if _, ok := affectedSet[row.Slug]; !ok {
affectedSet[row.Slug] = struct{}{}
affectedSlugs = append(affectedSlugs, row.Slug)
}
if row.Title != "" {
freshRefs = append(freshRefs, linkRef{slug: row.Slug, matchText: row.Title})
}
}
}
// KB flipped away from wiki (deleted / type change) — drain the lane so the
// rows don't accumulate, then stop.
if !kb.IsWikiEnabled() {
s.trimPendingList(ctx, ids)
return nil
}
synthesisModelID := ""
if kb.WikiConfig != nil {
synthesisModelID = kb.WikiConfig.SynthesisModelID
}
if synthesisModelID == "" {
synthesisModelID = kb.SummaryModelID
}
if synthesisModelID == "" {
// No model to rebuild the index with; still run the pure-text passes,
// then drain. Missing model is a config gap, not a transient error.
logger.Warnf(ctx, "wiki finalize: no synthesis model for KB %s, skipping index rebuild", payload.KnowledgeBaseID)
}
granularity := types.WikiExtractionStandard
if kb.WikiConfig != nil {
granularity = kb.WikiConfig.ExtractionGranularity.Normalize()
}
batchCtx := s.newWikiBatchContext(payload.KnowledgeBaseID, granularity)
lang := types.LanguageNameFromContext(ctx)
indexRebuilt := false
if changeDesc.Len() > 0 && synthesisModelID != "" {
chatModel, mErr := s.modelService.GetChatModel(ctx, synthesisModelID)
if mErr != nil {
logger.Warnf(ctx, "wiki finalize: get chat model failed: %v", mErr)
} else if err := s.rebuildIndexPage(ctx, chatModel, payload, changeDesc.String(), lang); err != nil {
logger.Warnf(ctx, "wiki finalize: rebuild index failed: %v", err)
} else {
indexRebuilt = true
}
}
if len(affectedSlugs) > 0 {
s.cleanDeadLinks(ctx, payload.KnowledgeBaseID, affectedSlugs, batchCtx)
s.injectCrossLinks(ctx, payload.KnowledgeBaseID, affectedSlugs, freshRefs, batchCtx)
}
// Drain the processed rows. Best-effort convergence mirrors the legacy
// in-batch behaviour: a failed index rebuild is logged (not retried),
// so we delete regardless to avoid re-doing the whole pass forever.
s.trimPendingList(ctx, ids)
// If more finalize rows landed while we were working, reschedule so they
// get their own convergence pass.
rescheduled := false
if n, cErr := s.pendingRepo.PendingCount(ctx, wikiFinalizeTaskType, wikiTaskScope, payload.KnowledgeBaseID); cErr == nil && n > 0 {
s.scheduleFinalize(ctx, payload)
rescheduled = true
}
logger.Infof(ctx,
"wiki finalize: kb=%s rows=%d affected_slugs=%d index_rebuilt=%v rescheduled=%v elapsed=%s",
payload.KnowledgeBaseID, len(rows), len(affectedSlugs), indexRebuilt, rescheduled,
time.Since(startedAt).Round(time.Millisecond),
)
return nil
}
+7 -1
View File
@@ -263,7 +263,13 @@ func BuildContainer(container *dig.Container) *dig.Container {
redisAvailable := os.Getenv("REDIS_ADDR") != ""
if redisAvailable {
must(container.Provide(router.NewAsyncqClient, dig.As(new(interfaces.TaskEnqueuer))))
must(container.Provide(router.NewAsynqServer))
// Two independent asynq worker pools with separate concurrency budgets:
// the parse pool (all queues except QueueWiki) and the wiki pool
// (QueueWiki only). Hard capacity isolation so upstream parsing and the
// downstream wiki pipeline can't starve each other during concurrent
// uploads. Both run the same mux (see router.RunAsynqServer).
must(container.Provide(router.NewParseAsynqServer, dig.Name("parseAsynqServer")))
must(container.Provide(router.NewWikiAsynqServer, dig.Name("wikiAsynqServer")))
// Asynq inspector for cancel-by-knowledge-id (best-effort
// dequeue of pending/scheduled/retry tasks + active-task cancel).
must(container.Provide(router.NewAsynqInspector))
+1
View File
@@ -146,5 +146,6 @@ func RegisterSyncHandlers(params SyncTaskParams) {
params.Executor.RegisterHandler(types.TypeKnowledgePostProcess, params.KnowledgePostProcess.Handle)
params.Executor.RegisterHandler(types.TypeDataSourceSync, params.DataSourceService.ProcessSync)
params.Executor.RegisterHandler(types.TypeWikiIngest, params.WikiIngest.Handle)
params.Executor.RegisterHandler(types.TypeWikiFinalize, params.WikiIngest.Handle)
logger.Infof(context.Background(), "[SyncTask] All task handlers registered (Lite mode, no Redis)")
}
+84 -26
View File
@@ -24,7 +24,15 @@ import (
type AsynqTaskParams struct {
dig.In
Server *asynq.Server
// ParseServer drains every queue EXCEPT QueueWiki: the user-facing
// document parsing / post-processing / enrichment pipeline. WikiServer
// drains QueueWiki only. Two independent worker pools give hard capacity
// isolation between the upstream parse pipeline and the downstream wiki
// pipeline (see NewParseAsynqServer / NewWikiAsynqServer). The same mux is
// run on both; asynq routes by the queue each server subscribes to, and
// queue↔task-type is 1:1 by convention, so no task is processed twice.
ParseServer *asynq.Server `name:"parseAsynqServer"`
WikiServer *asynq.Server `name:"wikiAsynqServer"`
KnowledgeService interfaces.KnowledgeService
KnowledgeBaseService interfaces.KnowledgeBaseService
TagService interfaces.KnowledgeTagService
@@ -122,8 +130,36 @@ func asynqRetryDelayFunc(n int, e error, t *asynq.Task) time.Duration {
// not on local CPU).
const defaultAsynqConcurrency = 32
func NewAsynqServer(svc interfaces.SystemSettingService) *asynq.Server {
// defaultWikiAsynqConcurrency is the worker pool size for the dedicated wiki
// pool (QueueWiki) when WEKNORA_WIKI_ASYNQ_CONCURRENCY is unset. Kept
// independent from the parse pool so the two can be sized separately for the
// host / LLM-provider budget. The wiki pipeline is LLM-bound (synthesis model
// calls dominate), so a moderate default avoids overrunning upstream provider
// concurrency while still letting many documents' wiki generation run in
// parallel across knowledge bases.
const defaultWikiAsynqConcurrency = 16
// newAsynqServer builds an asynq server bound to a specific queue set and
// concurrency. Shared by the parse and wiki pools so both inherit the same
// Redis options and retry-delay policy.
func newAsynqServer(concurrency int, queues map[string]int) *asynq.Server {
opt := getAsynqRedisClientOpt()
return asynq.NewServer(
opt,
asynq.Config{
Concurrency: concurrency,
Queues: queues,
RetryDelayFunc: asynqRetryDelayFunc,
},
)
}
// NewParseAsynqServer builds the upstream pool: every queue EXCEPT QueueWiki.
// This is the user-facing document parse / post-process / enrichment pipeline.
// Concurrency comes from WEKNORA_ASYNQ_CONCURRENCY (default 32), unchanged from
// the historical single-pool behaviour so existing deployments see the same
// parse throughput.
func NewParseAsynqServer(svc interfaces.SystemSettingService) *asynq.Server {
concurrency := defaultAsynqConcurrency
if svc != nil {
n := svc.GetInt(context.Background(), "asynq.concurrency", "WEKNORA_ASYNQ_CONCURRENCY", defaultAsynqConcurrency)
@@ -131,24 +167,35 @@ func NewAsynqServer(svc interfaces.SystemSettingService) *asynq.Server {
concurrency = int(n)
}
}
log.Printf("asynq server starting with concurrency=%d redis_op_timeout=%dms",
log.Printf("asynq parse-pool server starting with concurrency=%d redis_op_timeout=%dms",
concurrency, readRedisOpTimeoutMs())
srv := asynq.NewServer(
opt,
asynq.Config{
Concurrency: concurrency,
Queues: map[string]int{
types.QueueCritical: 6, // Highest priority queue
types.QueueDefault: 3, // Default priority queue
types.QueueLow: 1, // Lowest priority queue
types.QueueMultimodal: 1, // Isolated lane for high-volume slow VLM image tasks
types.QueueGraph: 1, // Isolated lane for high-volume slow graph-extraction tasks
types.QueueQuestion: 1, // Isolated lane for high-volume slow question-generation tasks
},
RetryDelayFunc: asynqRetryDelayFunc,
},
)
return srv
return newAsynqServer(concurrency, map[string]int{
types.QueueCritical: 6, // Highest priority queue
types.QueueDefault: 3, // Default priority queue
types.QueueLow: 1, // Lowest priority queue
types.QueueMultimodal: 1, // Isolated lane for high-volume slow VLM image tasks
types.QueueGraph: 1, // Isolated lane for high-volume slow graph-extraction tasks
types.QueueQuestion: 1, // Isolated lane for high-volume slow question-generation tasks
})
}
// NewWikiAsynqServer builds the dedicated wiki pool: QueueWiki only. Runs the
// same mux as the parse pool but only ever pulls wiki tasks, so its concurrency
// budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY, default 16) is spent exclusively on
// wiki generation. This is the hard capacity isolation that prevents the parse
// pipeline from starving wiki (and vice-versa) during concurrent uploads.
func NewWikiAsynqServer(svc interfaces.SystemSettingService) *asynq.Server {
concurrency := defaultWikiAsynqConcurrency
if svc != nil {
n := svc.GetInt(context.Background(), "asynq.wiki_concurrency", "WEKNORA_WIKI_ASYNQ_CONCURRENCY", defaultWikiAsynqConcurrency)
if n > 0 {
concurrency = int(n)
}
}
log.Printf("asynq wiki-pool server starting with concurrency=%d", concurrency)
return newAsynqServer(concurrency, map[string]int{
types.QueueWiki: 1,
})
}
func RunAsynqServer(params AsynqTaskParams) *asynq.ServeMux {
@@ -225,15 +272,26 @@ func RunAsynqServer(params AsynqTaskParams) *asynq.ServeMux {
// Register data source sync handler
mux.HandleFunc(types.TypeDataSourceSync, params.DataSourceService.ProcessSync)
// Register wiki ingest handler
// Register wiki ingest handler + the debounced KB-global finalize handler.
// Both route to the same dispatch (WikiIngest.Handle switches on task type)
// and both land on QueueWiki, so the dedicated wiki pool serves them.
mux.HandleFunc(types.TypeWikiIngest, params.WikiIngest.Handle)
mux.HandleFunc(types.TypeWikiFinalize, params.WikiIngest.Handle)
go func() {
// Start the server
if err := params.Server.Run(mux); err != nil {
log.Fatalf("could not run server: %v", err)
}
}()
// Run the same mux on both pools. Each server subscribes to a disjoint set
// of queues (parse pool = everything except QueueWiki; wiki pool =
// QueueWiki only), so a given task is dequeued by exactly one pool. This is
// how the two pools get independent concurrency budgets while sharing one
// handler registration.
runPool := func(name string, srv *asynq.Server) {
go func() {
if err := srv.Run(mux); err != nil {
log.Fatalf("could not run %s asynq server: %v", name, err)
}
}()
}
runPool("parse-pool", params.ParseServer)
runPool("wiki-pool", params.WikiServer)
return mux
}
+1
View File
@@ -57,6 +57,7 @@ var queuesScanned = []string{
types.QueueMultimodal,
types.QueueGraph,
types.QueueQuestion,
types.QueueWiki,
}
// taskTypesForKnowledgeCancel lists every asynq task type that carries
+30 -5
View File
@@ -2,6 +2,7 @@ package interfaces
import (
"context"
"time"
"github.com/Tencent/WeKnora/internal/types"
)
@@ -13,11 +14,15 @@ import (
// tuple is the only routing primitive the repository understands;
// deduplication, batching, and retry policy live in the consumer.
//
// Concurrency model in this revision: PeekBatch does NOT take row
// locks. Consumers are expected to enforce per-(scope_id) serialization
// out-of-band (wiki ingest does this via Redis SetNX wiki:active:<kbID>).
// The reserved `claimed_at` column lets a future revision adopt
// pessimistic locking without a schema change.
// Concurrency model: two consumption primitives coexist.
// - PeekBatch does NOT take row locks; consumers enforce per-(scope_id)
// serialization out-of-band (e.g. an external Redis lock). This is the
// original primitive and is still used by single-process (Lite) mode.
// - ClaimBatch DOES take row locks (SELECT ... FOR UPDATE SKIP LOCKED on
// Postgres) and marks rows with claimed_at, so multiple concurrent
// consumers of the same tuple pull DISJOINT rows without an external
// lock. This is what lets the wiki pipeline drop its exclusive per-KB
// batch lock and spread one KB's backlog across the whole worker pool.
type TaskPendingOpsRepository interface {
// Enqueue inserts a single op. The caller fills in TenantID, TaskType,
// Scope, ScopeID, Op, DedupKey, Payload; ID, FailCount, EnqueuedAt
@@ -30,6 +35,26 @@ type TaskPendingOpsRepository interface {
// IncrFailCount and leave them for the next pass).
PeekBatch(ctx context.Context, taskType, scope, scopeID string, limit int) ([]*types.TaskPendingOp, error)
// ClaimBatch atomically claims up to `limit` rows for the tuple,
// ordered by id ASC (FIFO), and returns them with claimed_at set to
// now. A row is eligible when it is unclaimed (claimed_at IS NULL) or
// its claim is stale (claimed_at < staleBefore) — the latter recovers
// rows abandoned by a crashed worker. On Postgres the selection uses
// FOR UPDATE SKIP LOCKED so concurrent claimers never block on or
// double-claim each other's rows.
//
// Claimed rows are NOT removed: the consumer must DeleteByIDs on
// success, or ReleaseByIDs to hand a still-retryable row back to the
// pool (otherwise it stays claimed until staleBefore elapses).
ClaimBatch(ctx context.Context, taskType, scope, scopeID string, limit int, staleBefore time.Time) ([]*types.TaskPendingOp, error)
// ReleaseByIDs clears claimed_at (back to NULL) for the given rows so
// a claimed-but-not-consumed row becomes immediately eligible for the
// next ClaimBatch. No-op for empty input. Used by the wiki retry path
// to re-queue a transiently-failed op without waiting for its claim to
// go stale. Harmless on rows that were never claimed (Lite mode).
ReleaseByIDs(ctx context.Context, ids []int64) error
// DeleteByIDs removes the given rows. No-op for empty input. Used
// to consume a successfully-processed batch, and to drop ops that
// have been moved to task_dead_letters.
+14 -3
View File
@@ -1,8 +1,8 @@
package types
// Asynq queue names. MUST stay in sync with the Queues weight map in
// router.NewAsynqServer — a task enqueued to a queue that the server does not
// list will never be consumed.
// Asynq queue names. MUST stay in sync with the Queues weight maps in
// router.NewParseAsynqServer / router.NewWikiAsynqServer — a task enqueued to a
// queue that no server subscribes to will never be consumed.
const (
QueueCritical = "critical"
QueueDefault = "default"
@@ -21,6 +21,16 @@ const (
// question batches from starving the lightweight tasks in the low queue
// (summary, deletes, wiki ingest).
QueueQuestion = "question"
// QueueWiki is the dedicated lane for the Wiki ingest pipeline. Unlike the
// other queues above (which all share a single asynq worker pool), this
// queue is consumed by a SEPARATE asynq server (router.NewWikiAsynqServer)
// with its own concurrency budget. Hard capacity isolation: heavy document
// parsing in the upstream pool can never starve wiki generation, and a
// wiki-generation burst can never starve user-facing parsing. Sizing the
// two pools independently replaces the old weighted-lottery approach where
// wiki sat on the low queue (weight 1) and got ~1/13 of the schedule
// during upload storms.
QueueWiki = "wiki"
)
const (
@@ -41,6 +51,7 @@ const (
TypeManualProcess = "manual:process" // 手工知识更新任务(cleanup + 重新索引)
TypeDataSourceSync = "datasource:sync" // 数据源同步任务
TypeWikiIngest = "wiki:ingest" // Wiki 页面同步任务
TypeWikiFinalize = "wiki:finalize" // Wiki KB 级收尾任务(防抖:索引重建/死链清理/交叉链接)
)
// ExtractChunkPayload represents the extract chunk task payload
+39 -11
View File
@@ -348,24 +348,43 @@ type WikiConfig struct {
// per document. Empty / unknown value is treated as WikiExtractionStandard.
ExtractionGranularity WikiExtractionGranularity `yaml:"extraction_granularity" json:"extraction_granularity,omitempty"`
// IngestBatchSize controls how many pending ops a single batch
// processes before scheduling a follow-up. 0 falls back to the
// hard-coded default (5). Operators on large KBs (4w+ docs) can
// raise this to 10–20 to amortize the lock-acquire / index-rebuild
// overhead across more documents per round.
// Wiki ingest concurrency is two-level:
// 1. batch-level: multiple batches per KB run concurrently in the wiki
// worker pool, capped by IngestMaxInflight (below).
// 2. batch-internal: within one batch, Map and Reduce fan out with
// errgroups sized by IngestMapParallel / IngestReduceParallel.
// Effective peak in-flight LLM calls for one KB ≈
// IngestMaxInflight × max(IngestMapParallel, IngestReduceParallel),
// further bounded by the wiki pool size (WEKNORA_WIKI_ASYNQ_CONCURRENCY).
// IngestBatchSize controls how many pending ops a single batch claims and
// processes before scheduling a follow-up. 0 falls back to the hard-coded
// default (5). Larger batches amortize per-batch setup and let more docs
// share the batch-internal Map/Reduce fan-out; smaller batches spread a
// KB's backlog across more concurrent batches (finer scheduling grain).
IngestBatchSize int `yaml:"ingest_batch_size" json:"ingest_batch_size,omitempty"`
// IngestMapParallel sets the errgroup limit for the Map phase
// (per-document extraction + summary + chunk citation). 0 falls
// back to 10. Bound by the LLM provider's concurrency limit and
// the worker's outbound HTTP pool.
// (per-document extraction + summary + chunk citation) WITHIN one batch.
// 0 falls back to 10. Bound by the LLM provider's concurrency limit and
// the worker's outbound HTTP pool. Remember it multiplies with the number
// of concurrent batches (IngestMaxInflight).
IngestMapParallel int `yaml:"ingest_map_parallel" json:"ingest_map_parallel,omitempty"`
// IngestReduceParallel sets the errgroup limit for the Reduce phase
// (per-slug page write). 0 falls back to 10. Bound by the same
// LLM concurrency / HTTP pool considerations as the Map phase,
// plus DB connection pool size.
// (per-slug page write) WITHIN one batch. 0 falls back to 10. Bound by the
// same LLM concurrency / HTTP pool considerations as the Map phase, plus
// DB connection pool size. Same multiplier caveat as IngestMapParallel.
IngestReduceParallel int `yaml:"ingest_reduce_parallel" json:"ingest_reduce_parallel,omitempty"`
// IngestMaxInflight caps how many ingest batches for THIS KB may run
// concurrently in the shared wiki worker pool (standard/Redis mode
// only). 0 falls back to the hard-coded default (4). Since Phase 3
// removed the exclusive per-KB lock, one KB's backlog could otherwise
// monopolize the whole pool during a bulk import and starve other KBs;
// this knob trades a single KB's peak throughput for cross-KB fairness.
// Set it >= the wiki pool size to effectively disable the cap.
IngestMaxInflight int `yaml:"ingest_max_inflight" json:"ingest_max_inflight,omitempty"`
}
// IngestBatchSizeOrDefault returns IngestBatchSize when set (> 0),
@@ -396,6 +415,15 @@ func (c *WikiConfig) IngestReduceParallelOrDefault(fallback int) int {
return c.IngestReduceParallel
}
// IngestMaxInflightOrDefault returns IngestMaxInflight when set,
// otherwise the hard-coded fallback.
func (c *WikiConfig) IngestMaxInflightOrDefault(fallback int) int {
if c == nil || c.IngestMaxInflight <= 0 {
return fallback
}
return c.IngestMaxInflight
}
// Value implements the driver.Valuer interface
func (c WikiConfig) Value() (driver.Value, error) {
return json.Marshal(c)