c9941a6688
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
34 lines
1.7 KiB
SQL
34 lines
1.7 KiB
SQL
-- Migration: 000053_knowledge_processing_spans (rollback)
|
|
-- Restores the 000052 flat-stage table so the system is operational on a
|
|
-- back-rev. We do NOT migrate data back — the spans table carries strict
|
|
-- supersets of the stages table's information, so a forward-and-back round
|
|
-- trip silently drops the new tree shape (acceptable; rollback is a rare
|
|
-- emergency lever).
|
|
DO $$ BEGIN RAISE NOTICE '[Migration 000053 rollback] Dropping spans, restoring stages table'; END $$;
|
|
|
|
DROP INDEX IF EXISTS idx_kpspan_parent;
|
|
DROP INDEX IF EXISTS idx_kpspan_status_started;
|
|
DROP INDEX IF EXISTS idx_kpspan_knowledge_attempt;
|
|
DROP TABLE IF EXISTS knowledge_processing_spans;
|
|
|
|
CREATE TABLE IF NOT EXISTS knowledge_processing_stages (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
knowledge_id VARCHAR(64) NOT NULL,
|
|
stage VARCHAR(32) NOT NULL,
|
|
status VARCHAR(16) NOT NULL,
|
|
started_at TIMESTAMP WITH TIME ZONE,
|
|
finished_at TIMESTAMP WITH TIME ZONE,
|
|
duration_ms BIGINT,
|
|
error_code VARCHAR(64),
|
|
error_message TEXT,
|
|
error_detail TEXT,
|
|
attempt INT NOT NULL DEFAULT 1,
|
|
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
CONSTRAINT uq_kps_knowledge_stage UNIQUE (knowledge_id, stage)
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_kps_knowledge ON knowledge_processing_stages (knowledge_id);
|
|
CREATE INDEX IF NOT EXISTS idx_kps_status_started ON knowledge_processing_stages (status, started_at);
|
|
|
|
DO $$ BEGIN RAISE NOTICE '[Migration 000053 rollback] complete'; END $$;
|