Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.
- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
(WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.
- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
(default 30m). Without this, a hung docreader pinned a worker for the
full DocumentProcessTimeout window.
- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
failing single image no longer strands the parent — the asynq retry
is allowed to run, but on the final attempt we count the image
regardless of outcome. Redis DECR errors fall back to enqueuing the
post-process task instead of returning silently.
- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
ManualProcess exhausts retries, immediately mark the corresponding
Knowledge as failed with the last error. This surfaces the failure
in the UI without waiting for the housekeeping sweep.
- HousekeepingService: 5-minute cron that flips knowledge rows stuck
in "processing" past DocumentProcessTimeout + 10m to failed, plus
summary rows stuck > 1h. Catches anything the other safety nets
miss (worker SIGKILL mid-handler, etc.). Disable with
WEKNORA_HOUSEKEEPING_ENABLED=false.
- Distributed startup recovery: previously the post-restart sweep was
skipped whenever REDIS_ADDR was set, even though Asynq does not
reschedule the task that was actively running on the dead instance.
Now the sweep runs in distributed mode too, but only against rows
older than 30 minutes to avoid racing peer instances.
Documents whose only payload is an embedded image (e.g. a docx with a
single picture) intermittently produced the refusal line "No textual
content was extractable from this document." even though the vision
model had successfully extracted a caption.
Three coordinated fixes:
- Clarify the summary prompt that text inside `<image_caption>` and
`<image_ocr>` is first-class extracted content, not an image
reference, so the model only triggers the empty-content branch when
the body is genuinely textless.
- For image-dominated documents (real text < 200 runes after stripping
image markup) include OCR alongside captions so screenshots and
scanned figures contribute their actual content; text-heavy
documents continue to use caption-only enrichment to avoid OCR
noise from incidental figures.
- Add `EnrichContentCaptionAndOCR` which embeds caption + OCR text
inline next to the original Markdown image link, deliberately
omitting the `<image url=...>` and `<image_original>` wrapper
blocks. Those wrappers carry only opaque export hashes that consume
tokens and have been observed to retrigger the LLM's "image
reference with no extracted text" heuristic.
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across
seven services into two factory functions with tenant-ownership
verification: CreateRetrieveEngineForKB (synchronous) and
CreateRetrieveEngineFromPayload (async task handlers). Promotes
GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry.
Extends KBDeletePayload / IndexDeletePayload with an omitempty
VectorStoreID snapshot.
Design highlights
- Factory verifies tenant ownership of the resolved store
(defense-in-depth) so a gap in PR3 validation or a tampered Asynq
payload cannot cross tenants. Cross-tenant attempts return
ErrVectorStoreForbidden with a structured log entry.
- Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound,
ErrVectorStoreForbidden) let async handlers classify non-retryable
failures as asynq.SkipRetry.
- nil and empty-string pointers for vectorStoreID normalize to
"unbound" so callers never send an empty UUID to GetByStoreID.
- resolveBoundEngine constructs engineInfos directly (with
slices.Clone of Support()) instead of going through
NewCompositeRetrieveEngine. The latter only reads from the
byEngineType map and cannot reach DB stores in byStoreID -- multiple
stores can share the same engine type. Unbound fallbacks still reuse
NewCompositeRetrieveEngine so the tenant effective-engines path is
byte-for-byte unchanged.
- DeleteKnowledgeBase now reads the KB before soft-delete so the
enqueued KBDeletePayload carries a VectorStoreID snapshot; once the
row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT
per KB-delete request (a rare admin operation).
- enqueueIndexDeleteTask gains a vectorStoreID *string parameter that
the single caller (tag.go) populates from the owning KB. The worker
(ProcessIndexDelete) validates ownership via the factory.
- cleanupKnowledgeResources loads the KB so bound-store cleanup routes
correctly. If the load fails it warns and falls back to tenant
effective engines; orphan vectors become observable in logs instead
of disappearing silently.
- DeleteKnowledges (batch) stays on the tenant effective-engines path
-- batches can span stores and multi-store fan-out is PR4 scope.
Noted inline.
Merge safety
All knowledge bases have VectorStoreID = NULL at merge time (introduced
nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to
NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()),
which is the existing code path for these KBs. Async payloads from
before this change decode to VectorStoreID = nil (omitempty) and take
the same fallback.
Testing
- go build ./...
- go vet ./...
- go test -race -count=1 ./internal/application/service/retriever/...
covers unbound (nil/empty), bound, cross-tenant, unregistered store,
ownership infra error, legacy payload shapes (missing field/null/
empty string/nil), tampered payload, and parallel invocation.
- grep -r NewCompositeRetrieveEngine internal/application/service
returns zero results outside factory.go/composite.go.
Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994.
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.