feat(knowledge): track per-stage parsing progress with /stages API
Adds a five-segment progress model for the document parsing pipeline so the UI (PR③) can render a timeline showing where each document is (DocReader → Chunking → Embedding → Multimodal → PostProcess) and which stage failed with what error code. - New table `knowledge_processing_stages` (migration 000052) with one row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an attempt counter so re-parses don't lose history. - StageTracker service exposes Begin/Done/Fail/Skip; all calls are best-effort and never break the pipeline if persistence fails. - Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT / VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized remediation hints. - Tracker call sites added at the four meaningful failure points: convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding), enqueueImageMultimodalTasks (Multimodal start), KnowledgePostProcess.Handle (Multimodal close + PostProcess). - New endpoint `GET /api/v1/knowledge/:id/stages` returns the five canonical stages — missing rows are synthesized as "pending" so the timeline always renders five segments. Includes current_stage and last_error block.
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
// Package errors: parse stage error codes.
|
||||
//
|
||||
// These constants are the stable wire format the frontend uses to map a
|
||||
// failure to a localized "what to do about it" message. Adding a new code
|
||||
// is non-breaking; renaming an existing one requires a coordinated
|
||||
// frontend release because the i18n keys are looked up by code.
|
||||
//
|
||||
// Codes are SCREAMING_SNAKE so they survive JSON case transforms and look
|
||||
// distinct from Go identifiers in logs/dashboards.
|
||||
package errors
|
||||
|
||||
const (
|
||||
// ErrCodeDocReaderTimeout — DocReader RPC exceeded
|
||||
// WEKNORA_DOCREADER_CALL_TIMEOUT (default 30m). Suggest splitting
|
||||
// large files or checking docreader load.
|
||||
ErrCodeDocReaderTimeout = "DOCREADER_TIMEOUT"
|
||||
|
||||
// ErrCodeDocReaderUnavailable — no DocReader configured for the
|
||||
// requested file type / engine, or the service refused connection.
|
||||
ErrCodeDocReaderUnavailable = "DOCREADER_UNAVAILABLE"
|
||||
|
||||
// ErrCodeDocReaderParseFailed — DocReader returned an explicit parse
|
||||
// error (encoding, corrupted file, OCR engine crash, ...).
|
||||
ErrCodeDocReaderParseFailed = "DOCREADER_PARSE_FAILED"
|
||||
|
||||
// ErrCodeChunkingFailed — text chunking step itself failed (rare;
|
||||
// usually only on extreme-size inputs).
|
||||
ErrCodeChunkingFailed = "CHUNKING_FAILED"
|
||||
|
||||
// ErrCodeEmbeddingRateLimit — embedding provider returned 429 or
|
||||
// equivalent. Suggest retrying later or scaling out.
|
||||
ErrCodeEmbeddingRateLimit = "EMBEDDING_RATE_LIMIT"
|
||||
|
||||
// ErrCodeEmbeddingProviderFail — non-rate-limit embedding error
|
||||
// (auth, model not found, bad input). Usually permanent without
|
||||
// config changes.
|
||||
ErrCodeEmbeddingProviderFail = "EMBEDDING_PROVIDER_FAIL"
|
||||
|
||||
// ErrCodeVectorStoreWriteFailed — chunks embedded fine but the
|
||||
// vector DB write failed (quota, connectivity, schema mismatch).
|
||||
ErrCodeVectorStoreWriteFailed = "VECTORSTORE_WRITE_FAILED"
|
||||
|
||||
// ErrCodeMultimodalVLMFailed — a single image's OCR or caption call
|
||||
// failed. Note: per-image failures don't fail the whole parent;
|
||||
// see image_multimodal.go finalize-on-last-attempt.
|
||||
ErrCodeMultimodalVLMFailed = "MULTIMODAL_VLM_FAILED"
|
||||
|
||||
// ErrCodeMultimodalAllFailed — every image task hit dead-letter.
|
||||
// The parent still completes (caption / OCR is optional content),
|
||||
// but stage status is marked failed so the UI can warn.
|
||||
ErrCodeMultimodalAllFailed = "MULTIMODAL_ALL_FAILED"
|
||||
|
||||
// ErrCodeTaskTimeout — asynq retry budget exhausted. Used by the
|
||||
// dead-letter callback when promoting a task failure into a stage
|
||||
// failure. Distinct from DocReaderTimeout: this is the asynq-level
|
||||
// timeout (whole task), not the docreader-call-level timeout.
|
||||
ErrCodeTaskTimeout = "TASK_TIMEOUT"
|
||||
|
||||
// ErrCodeUnknown — fallback when a wrapped error doesn't classify.
|
||||
// The full message is still recorded in error_detail so operators
|
||||
// can debug; the UI shows a generic "see admin" hint.
|
||||
ErrCodeUnknown = "UNKNOWN"
|
||||
)
|
||||
Reference in New Issue
Block a user