Files
Enterprise-Knowledge-Base/internal/application/repository/knowledge_span_repo.go
T
wizardchen 44d6175559 feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.

State machine

  pending -> processing -> finalizing -> completed
                              |
                              +-> cancelled (any of the three
                                            in-flight states)
                              +-> failed
                              +-> deleting

`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.

Backend

- New ParseStatusFinalizing + pending_subtasks_count column with
  migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
  conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
  counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
  count up front, flip to finalizing (or completed when no
  enrichment is enabled), and only then fan out subtasks. Subtask
  handlers (summary, question, graph extract) defer-decrement on
  terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
  pending / processing / finalizing. Marks the row cancelled,
  zeroes the counter, best-effort dequeues asynq tasks via a new
  TaskInspector abstraction (asynq-mode walks pending/scheduled/
  retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
  for the attempt via a new repo.CancelAllOpenSpans helper so the
  trace viewer's striped bars all flip to cancelled, even leaf
  generations whose parent stage already EndSpan'd (multimodal
  fan-out pattern). knowledge_post_process closes its postSpan
  via SkipSpan on the cancel/deleting entry guard so a worker
  that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
  identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
  downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
  expensive enrichment (graph extract) bails immediately when the
  parent knowledge is aborted.

Frontend

- New cancelKnowledgeParse API client + "Stop parsing" entry in
  both list view and card view more menus, gated on
  pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
  that recognises `finalizing` (previously the doc list silently
  stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
  new isHardTerminal short-circuits LIVE for cancelled/failed/
  completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
  ("增强中") from completed and shows the previous "生成摘要中"
  copy when summary_status is still pending under finalizing.
  Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
  across zh-CN, en-US, ko-KR, ru-RU.

Docs / SDK

- docs/api/knowledge.md: documents the new finalizing state,
  cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
  the cancellable statuses.
2026-05-28 20:16:02 +08:00

231 lines
8.7 KiB
Go

package repository
import (
"context"
"errors"
"time"
"github.com/Tencent/WeKnora/internal/types"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// KnowledgeSpanRepository persists the per-attempt span tree used by the
// processing pipeline. Operations are deliberately narrow:
//
// - Upsert covers Begin/End/Fail/Skip — every state transition routes
// through the same write so the row stays internally consistent.
// - NextAttempt allocates a new attempt for re-parses without touching
// historical rows. Old attempts stay queryable for post-mortem.
// - ListByAttempt is the only read path; the handler builds the tree
// in memory rather than recursing through the DB.
type KnowledgeSpanRepository interface {
Upsert(ctx context.Context, row *types.KnowledgeProcessingSpan) error
NextAttempt(ctx context.Context, knowledgeID string) (int, error)
LatestAttempt(ctx context.Context, knowledgeID string) (int, error)
ListByAttempt(ctx context.Context, knowledgeID string, attempt int) ([]types.KnowledgeProcessingSpan, error)
GetSpan(ctx context.Context, knowledgeID string, attempt int, spanID string) (*types.KnowledgeProcessingSpan, error)
// CancelDescendants marks every descendant of a parent span as
// "cancelled" with the given reason. Used by the tracker to
// cascade an upstream failure across a stage's downstream subtree
// without iterating in Go memory.
CancelDescendants(ctx context.Context, knowledgeID string, attempt int, parentSpanID, reason string) (int64, error)
// CancelAllOpenSpans flips every non-terminal (pending/running) span
// for (knowledgeID, attempt) to "cancelled" in one statement,
// regardless of tree position. Used by the user-cancel path where
// fan-out stages (e.g. "多模态识别") flip themselves to done as soon
// as they finish dispatching, while their async children are still
// running — a tree walk that stops at terminal parents would miss
// those orphan leaves.
CancelAllOpenSpans(ctx context.Context, knowledgeID string, attempt int, errorCode, reason string) (int64, error)
}
type knowledgeSpanRepository struct {
db *gorm.DB
}
// NewKnowledgeSpanRepository wires the GORM-backed implementation.
func NewKnowledgeSpanRepository(db *gorm.DB) KnowledgeSpanRepository {
return &knowledgeSpanRepository{db: db}
}
func (r *knowledgeSpanRepository) Upsert(ctx context.Context, row *types.KnowledgeProcessingSpan) error {
if row == nil || row.KnowledgeID == "" || row.SpanID == "" {
return errors.New("knowledgeSpanRepository.Upsert: knowledge_id and span_id required")
}
if row.Attempt == 0 {
row.Attempt = 1
}
// We let GORM populate created_at/updated_at via the autoCreate /
// autoUpdate tags. ON CONFLICT updates only the fields that may
// transition between calls — name/kind/parent are immutable once
// set so we don't list them in DoUpdates (saves a few bytes per
// write, and any mismatch indicates a programming error).
//
// CRITICAL: input / output / metadata are CONTENT fields that
// individual call sites only fill when they have something to set.
// EndSpan e.g. only sets `output`; if we always listed `input` in
// DoUpdates, the End call would clobber the input set by Begin with
// NULL. Same for metadata. Build the DoUpdates list dynamically and
// skip these three columns when the incoming row has nothing to
// write — so "no value" preserves the existing column instead of
// nuking it.
cols := []string{
"status",
"error_code",
"error_message",
"error_detail",
"started_at",
"finished_at",
"duration_ms",
"updated_at",
}
if row.Input != nil {
cols = append(cols, "input")
}
if row.Output != nil {
cols = append(cols, "output")
}
if row.Metadata != nil {
cols = append(cols, "metadata")
}
return r.db.WithContext(ctx).Clauses(clause.OnConflict{
Columns: []clause.Column{
{Name: "knowledge_id"},
{Name: "attempt"},
{Name: "span_id"},
},
DoUpdates: clause.AssignmentColumns(cols),
}).Create(row).Error
}
func (r *knowledgeSpanRepository) NextAttempt(ctx context.Context, knowledgeID string) (int, error) {
var max int
err := r.db.WithContext(ctx).Model(&types.KnowledgeProcessingSpan{}).
Where("knowledge_id = ?", knowledgeID).
Select("COALESCE(MAX(attempt), 0)").
Row().Scan(&max)
if err != nil {
return 0, err
}
return max + 1, nil
}
func (r *knowledgeSpanRepository) LatestAttempt(ctx context.Context, knowledgeID string) (int, error) {
var max int
err := r.db.WithContext(ctx).Model(&types.KnowledgeProcessingSpan{}).
Where("knowledge_id = ?", knowledgeID).
Select("COALESCE(MAX(attempt), 0)").
Row().Scan(&max)
return max, err
}
func (r *knowledgeSpanRepository) ListByAttempt(ctx context.Context, knowledgeID string, attempt int) ([]types.KnowledgeProcessingSpan, error) {
if knowledgeID == "" {
return nil, nil
}
var rows []types.KnowledgeProcessingSpan
q := r.db.WithContext(ctx).Where("knowledge_id = ?", knowledgeID)
if attempt > 0 {
q = q.Where("attempt = ?", attempt)
}
// id ASC keeps the natural insertion order — useful for stable
// rendering of fan-out subspans (e.g. multimodal.image[0..N] in
// the order they were enqueued).
err := q.Order("id ASC").Find(&rows).Error
return rows, err
}
func (r *knowledgeSpanRepository) GetSpan(ctx context.Context, knowledgeID string, attempt int, spanID string) (*types.KnowledgeProcessingSpan, error) {
var row types.KnowledgeProcessingSpan
err := r.db.WithContext(ctx).
Where("knowledge_id = ? AND attempt = ? AND span_id = ?", knowledgeID, attempt, spanID).
Take(&row).Error
if err != nil {
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil, nil
}
return nil, err
}
return &row, nil
}
// CancelDescendants performs an iterative SQL walk: each level we update
// every row whose parent_span_id is in the previous level's span_id set,
// flipping pending/running rows to cancelled. We bail when a level adds
// zero rows (fixed point reached) or after a generous depth bound.
//
// Postgres-specific WITH RECURSIVE would be denser but harder to test on
// the SQLite Lite backend. The iterative path stays portable.
func (r *knowledgeSpanRepository) CancelDescendants(ctx context.Context, knowledgeID string, attempt int, parentSpanID, reason string) (int64, error) {
frontier := []string{parentSpanID}
var totalAffected int64
for depth := 0; depth < 16 && len(frontier) > 0; depth++ {
var nextFrontier []string
// Find children of every span currently on the frontier
// that are still in a non-terminal state — terminal rows
// (done/failed/skipped/cancelled) are left as-is so the UI
// can still see their original outcome.
var children []types.KnowledgeProcessingSpan
err := r.db.WithContext(ctx).
Where("knowledge_id = ? AND attempt = ? AND parent_span_id IN ? AND status IN ?",
knowledgeID, attempt, frontier,
[]string{types.SpanStatusPending, types.SpanStatusRunning}).
Find(&children).Error
if err != nil {
return totalAffected, err
}
if len(children) == 0 {
break
}
ids := make([]string, 0, len(children))
for _, c := range children {
ids = append(ids, c.SpanID)
nextFrontier = append(nextFrontier, c.SpanID)
}
res := r.db.WithContext(ctx).Model(&types.KnowledgeProcessingSpan{}).
Where("knowledge_id = ? AND attempt = ? AND span_id IN ?", knowledgeID, attempt, ids).
Updates(map[string]any{
"status": types.SpanStatusCancelled,
"error_code": "UPSTREAM_FAILED",
"error_message": reason,
})
if res.Error != nil {
return totalAffected, res.Error
}
totalAffected += res.RowsAffected
frontier = nextFrontier
}
return totalAffected, nil
}
// CancelAllOpenSpans is the "abort the attempt" counterpart to
// CancelDescendants. It avoids the BFS entirely so spans whose parent
// is already terminal (typical for stage fan-outs that EndSpan as soon
// as they finish dispatching async work) still get flipped to cancelled.
// We deliberately do NOT touch finished_at / duration_ms here — the
// span row remains observable in the trace tree with its original
// start time and gets a cancelled status + reason, which is enough
// for the UI to drop the running-bar styling.
func (r *knowledgeSpanRepository) CancelAllOpenSpans(
ctx context.Context, knowledgeID string, attempt int, errorCode, reason string,
) (int64, error) {
now := time.Now()
updates := map[string]any{
"status": types.SpanStatusCancelled,
"error_code": errorCode,
"error_message": reason,
"finished_at": now,
"updated_at": now,
}
res := r.db.WithContext(ctx).Model(&types.KnowledgeProcessingSpan{}).
Where("knowledge_id = ? AND attempt = ? AND status IN ?",
knowledgeID, attempt,
[]string{types.SpanStatusPending, types.SpanStatusRunning}).
Updates(updates)
if res.Error != nil {
return 0, res.Error
}
return res.RowsAffected, nil
}