44d6175559
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
255 lines
8.1 KiB
Go
255 lines
8.1 KiB
Go
package router
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
|
|
"github.com/Tencent/WeKnora/internal/logger"
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
|
"github.com/hibiken/asynq"
|
|
)
|
|
|
|
// NewAsynqInspector constructs an *asynq.Inspector pointed at the same
|
|
// Redis used by the asynq client. Only registered in asynq mode.
|
|
func NewAsynqInspector() *asynq.Inspector {
|
|
return asynq.NewInspector(getAsynqRedisClientOpt())
|
|
}
|
|
|
|
// asynqTaskInspector implements interfaces.TaskInspector backed by an
|
|
// *asynq.Inspector. Scans the queues we actually use ("default",
|
|
// "critical", "low") and matches tasks whose payload carries the given
|
|
// knowledge_id. Best-effort: any scan/delete error is logged and
|
|
// swallowed so the cancel API still returns success even when Redis is
|
|
// flaky.
|
|
type asynqTaskInspector struct {
|
|
inspector *asynq.Inspector
|
|
}
|
|
|
|
// NewAsynqTaskInspector returns a TaskInspector wrapping the given
|
|
// *asynq.Inspector. nil-safe: a nil inspector degrades to a no-op so
|
|
// the cancel path remains usable when the inspector failed to init.
|
|
func NewAsynqTaskInspector(inspector *asynq.Inspector) interfaces.TaskInspector {
|
|
if inspector == nil {
|
|
return noopTaskInspector{}
|
|
}
|
|
return &asynqTaskInspector{inspector: inspector}
|
|
}
|
|
|
|
// knowledgeIDProbe is the minimal payload shape we need to filter
|
|
// tasks. All pipeline payload types embed a json:"knowledge_id" field,
|
|
// so a single struct covers Document / ImageMultimodal / PostProcess /
|
|
// Question / Summary / Extract / Manual.
|
|
type knowledgeIDProbe struct {
|
|
KnowledgeID string `json:"knowledge_id,omitempty"`
|
|
}
|
|
|
|
// queuesScanned is the fixed set of queue names this codebase enqueues
|
|
// into. Kept tight on purpose — we never scan user-defined queues.
|
|
var queuesScanned = []string{"default", "critical", "low"}
|
|
|
|
// taskTypesForKnowledgeCancel lists every asynq task type that carries
|
|
// a knowledge_id in its payload and should be cancelable. The set is
|
|
// deliberately narrow: we don't touch FAQ import / KB-level tasks
|
|
// because the cancel API is per-knowledge.
|
|
var taskTypesForKnowledgeCancel = map[string]struct{}{
|
|
types.TypeDocumentProcess: {},
|
|
types.TypeManualProcess: {},
|
|
types.TypeImageMultimodal: {},
|
|
types.TypeKnowledgePostProcess: {},
|
|
types.TypeQuestionGeneration: {},
|
|
types.TypeSummaryGeneration: {},
|
|
types.TypeChunkExtract: {},
|
|
}
|
|
|
|
// listPageSize caps each Redis LIST call. Asynq pages tasks, so we
|
|
// loop until a short page comes back. 100 matches asynq's default.
|
|
const listPageSize = 100
|
|
|
|
// CancelTasksForKnowledge removes queued tasks whose payload references
|
|
// the given knowledge_id and signals active workers running such tasks
|
|
// to stop.
|
|
func (a *asynqTaskInspector) CancelTasksForKnowledge(
|
|
ctx context.Context, knowledgeID string,
|
|
) (int, int, error) {
|
|
if a == nil || a.inspector == nil || knowledgeID == "" {
|
|
return 0, 0, nil
|
|
}
|
|
deleted := 0
|
|
cancelled := 0
|
|
|
|
for _, queue := range queuesScanned {
|
|
// Pending / Scheduled / Retry can all be deleted by task ID.
|
|
// Archived tasks are NOT touched: dead-letter rows are
|
|
// already final and should remain visible to operators.
|
|
deleted += a.deletePendingMatches(ctx, queue, knowledgeID)
|
|
deleted += a.deleteScheduledMatches(ctx, queue, knowledgeID)
|
|
deleted += a.deleteRetryMatches(ctx, queue, knowledgeID)
|
|
cancelled += a.cancelActiveMatches(ctx, queue, knowledgeID)
|
|
}
|
|
|
|
logger.Infof(ctx,
|
|
"[TaskInspector] knowledge=%s cancel summary: deleted_from_queue=%d active_cancel_signaled=%d",
|
|
knowledgeID, deleted, cancelled,
|
|
)
|
|
return deleted, cancelled, nil
|
|
}
|
|
|
|
// matchesKnowledge returns true when the task type is one we cancel
|
|
// AND its payload references the target knowledge ID.
|
|
func matchesKnowledge(taskType string, payload []byte, knowledgeID string) bool {
|
|
if _, ok := taskTypesForKnowledgeCancel[taskType]; !ok {
|
|
return false
|
|
}
|
|
var probe knowledgeIDProbe
|
|
if err := json.Unmarshal(payload, &probe); err != nil {
|
|
return false
|
|
}
|
|
return probe.KnowledgeID == knowledgeID
|
|
}
|
|
|
|
func (a *asynqTaskInspector) deletePendingMatches(ctx context.Context, queue, knowledgeID string) int {
|
|
deleted := 0
|
|
page := 1
|
|
for {
|
|
tasks, err := a.inspector.ListPendingTasks(queue, asynq.PageSize(listPageSize), asynq.Page(page))
|
|
if err != nil {
|
|
if !errors.Is(err, asynq.ErrQueueNotFound) {
|
|
logger.Warnf(ctx, "[TaskInspector] list pending queue=%s page=%d: %v", queue, page, err)
|
|
}
|
|
return deleted
|
|
}
|
|
if len(tasks) == 0 {
|
|
return deleted
|
|
}
|
|
for _, t := range tasks {
|
|
if !matchesKnowledge(t.Type, t.Payload, knowledgeID) {
|
|
continue
|
|
}
|
|
if err := a.inspector.DeleteTask(queue, t.ID); err != nil {
|
|
logger.Warnf(ctx, "[TaskInspector] delete pending type=%s id=%s: %v", t.Type, t.ID, err)
|
|
continue
|
|
}
|
|
deleted++
|
|
}
|
|
if len(tasks) < listPageSize {
|
|
return deleted
|
|
}
|
|
page++
|
|
}
|
|
}
|
|
|
|
func (a *asynqTaskInspector) deleteScheduledMatches(ctx context.Context, queue, knowledgeID string) int {
|
|
deleted := 0
|
|
page := 1
|
|
for {
|
|
tasks, err := a.inspector.ListScheduledTasks(queue, asynq.PageSize(listPageSize), asynq.Page(page))
|
|
if err != nil {
|
|
if !errors.Is(err, asynq.ErrQueueNotFound) {
|
|
logger.Warnf(ctx, "[TaskInspector] list scheduled queue=%s page=%d: %v", queue, page, err)
|
|
}
|
|
return deleted
|
|
}
|
|
if len(tasks) == 0 {
|
|
return deleted
|
|
}
|
|
for _, t := range tasks {
|
|
if !matchesKnowledge(t.Type, t.Payload, knowledgeID) {
|
|
continue
|
|
}
|
|
if err := a.inspector.DeleteTask(queue, t.ID); err != nil {
|
|
logger.Warnf(ctx, "[TaskInspector] delete scheduled type=%s id=%s: %v", t.Type, t.ID, err)
|
|
continue
|
|
}
|
|
deleted++
|
|
}
|
|
if len(tasks) < listPageSize {
|
|
return deleted
|
|
}
|
|
page++
|
|
}
|
|
}
|
|
|
|
func (a *asynqTaskInspector) deleteRetryMatches(ctx context.Context, queue, knowledgeID string) int {
|
|
deleted := 0
|
|
page := 1
|
|
for {
|
|
tasks, err := a.inspector.ListRetryTasks(queue, asynq.PageSize(listPageSize), asynq.Page(page))
|
|
if err != nil {
|
|
if !errors.Is(err, asynq.ErrQueueNotFound) {
|
|
logger.Warnf(ctx, "[TaskInspector] list retry queue=%s page=%d: %v", queue, page, err)
|
|
}
|
|
return deleted
|
|
}
|
|
if len(tasks) == 0 {
|
|
return deleted
|
|
}
|
|
for _, t := range tasks {
|
|
if !matchesKnowledge(t.Type, t.Payload, knowledgeID) {
|
|
continue
|
|
}
|
|
if err := a.inspector.DeleteTask(queue, t.ID); err != nil {
|
|
logger.Warnf(ctx, "[TaskInspector] delete retry type=%s id=%s: %v", t.Type, t.ID, err)
|
|
continue
|
|
}
|
|
deleted++
|
|
}
|
|
if len(tasks) < listPageSize {
|
|
return deleted
|
|
}
|
|
page++
|
|
}
|
|
}
|
|
|
|
// cancelActiveMatches signals active workers to abort via
|
|
// Inspector.CancelProcessing. The worker's ctx becomes Done() so the
|
|
// next blocking call (or our checkpoint reads) bails. The DB-level
|
|
// abort flag (parse_status=cancelled) remains the durable signal —
|
|
// this is a latency optimization, not the correctness mechanism.
|
|
func (a *asynqTaskInspector) cancelActiveMatches(ctx context.Context, queue, knowledgeID string) int {
|
|
cancelled := 0
|
|
page := 1
|
|
for {
|
|
tasks, err := a.inspector.ListActiveTasks(queue, asynq.PageSize(listPageSize), asynq.Page(page))
|
|
if err != nil {
|
|
if !errors.Is(err, asynq.ErrQueueNotFound) {
|
|
logger.Warnf(ctx, "[TaskInspector] list active queue=%s page=%d: %v", queue, page, err)
|
|
}
|
|
return cancelled
|
|
}
|
|
if len(tasks) == 0 {
|
|
return cancelled
|
|
}
|
|
for _, t := range tasks {
|
|
if !matchesKnowledge(t.Type, t.Payload, knowledgeID) {
|
|
continue
|
|
}
|
|
if err := a.inspector.CancelProcessing(t.ID); err != nil {
|
|
logger.Warnf(ctx, "[TaskInspector] cancel active type=%s id=%s: %v", t.Type, t.ID, err)
|
|
continue
|
|
}
|
|
cancelled++
|
|
}
|
|
if len(tasks) < listPageSize {
|
|
return cancelled
|
|
}
|
|
page++
|
|
}
|
|
}
|
|
|
|
// noopTaskInspector is the Lite-mode (no Redis) inspector. Inline
|
|
// goroutines spawned by SyncTaskExecutor cannot be dequeued before
|
|
// they start; the checkpoint-based abort in worker code is the only
|
|
// stop signal in that mode.
|
|
type noopTaskInspector struct{}
|
|
|
|
// NewNoopTaskInspector returns a no-op TaskInspector for Lite mode.
|
|
func NewNoopTaskInspector() interfaces.TaskInspector { return noopTaskInspector{} }
|
|
|
|
func (noopTaskInspector) CancelTasksForKnowledge(
|
|
ctx context.Context, knowledgeID string,
|
|
) (int, int, error) {
|
|
return 0, 0, nil
|
|
}
|