Let users upload documents in chat without ingesting them into a knowledge base: files are parsed in the background, referenced by ID on send, and expire after a configurable TTL.
Introduce resource://<handle> as the stable application identity for stored
objects, decoupling API responses, persisted rich text, and LLM prompts from
physical storage paths. Adds a resource catalog service (register/resolve/bind,
revocable expiring access grants), a request-local alias registry that compacts
resource references while in the LLM context and restores them (including across
streaming chunks), plus router, IM, chat-pipeline, and multimodal integration.
Includes schema migrations (resources, resource_bindings, resource_access_grants)
for sqlite, and hardens URL/output security handling.
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
Enable agents to configure starter prompts and async follow-up questions after completed answers, with durable caching, analytics events, and chat/embed UI integration.
Allow deployments to register users without auto-creating a personal tenant, expose creation policy via /auth/me, and guide tenantless accounts through a workspace onboarding flow.
Wiki ingest builds postprocess.wiki.page[<slug>] subspan names that can exceed
varchar(64); widen the column to 255 and hash-suffix truncation so long slugs
still persist distinctly.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Renamed TestTaskPendingOps_ClaimBatch_NewSiblingRowNotDoubleClaimed to TestTaskPendingOps_ClaimBatch_LateSiblingBlockedByFreshClaim for clarity on behavior.
- Added new test TestTaskPendingOps_ClaimBatch_LateSiblingClaimableAfterStale to verify claim behavior when a holder crashes.
- Updated ClaimBatch logic to ensure that fresh claims block late siblings entirely, maintaining serialization of same-document operations.
- Improved comments in task_queue.go to clarify the handling of concurrent claims and the implications of the claimed_at field.
- Added crash safety mechanism in ProcessWikiIngest to release claims on abnormal exits, ensuring timely re-claims of pending operations.
Two correctness fixes on the Phase 3 concurrent claim-based ingest path:
- Slug-lock timeout / reduce failure no longer silently drops work. When a
per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
an error, the page's read-modify-write never lands. Previously the owning
document was still counted as successful and its pending row was deleted at
trim time, permanently losing the contribution (finalize only rebuilds the
index / cross-links, it does not re-run reduce). Such documents are now
folded into failedOps: their rows stay queued, their finalizing slot is
held, and requeueFailedOps' fail_count budget retries them (dead-lettering
a permanently hot/broken slug).
- ClaimBatch now claims by dedup_key affinity so a document is never split
across concurrent batches. limit counts distinct dedup_keys (documents),
and ALL eligible rows sharing a key are claimed together. On Postgres the
per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
concurrent claimers disjoint key sets. This restores in-document ordering
(e.g. ingest-then-retract collapses correctly) that the per-batch dedup
could not guarantee once the exclusive per-KB lock was removed, preventing
a stale ingest from resurrecting a retracted document.
Adds repository tests for same-key affinity and in-flight sibling rows.
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.
- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
concurrency, default 16), split from the parse pool (NewParseAsynqServer).
Both run the same mux over disjoint queue sets. Replaces the old
weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
batches per KB; Map/Reduce fan-out sized within a batch. New config knob
+ system setting + swagger docs.
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.
- Add APIKeyGate as the single authority for API-key authorization: routes
declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
routes are denied by default. JWT principals short-circuit the gate and the
JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
is a pure downstream data filter); owner is a full-access tenant tier that
unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
writes on API-key encryption failure, deny KB-restricted keys on /files, and
backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
gin paths before comparison; add regression tests.
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
Make document keyword search case-insensitive (LOWER(...) LIKE LOWER(?)) in the
list filter and SearchKnowledge / SearchKnowledgeInScopes paths so a query and
its differently-cased variant return the same results.
Classify not-found consistently as HTTP 404 (not 500) so clients read a
permanent resource.not_found instead of a retryable 5xx: map the document /
chunk not-found sentinels in the KB-access middleware (typed ErrChunkNotFound
via errors.Is), and return 404 for a missing message on the continue-stream and
message-delete endpoints.
Wire per-turn @mentions for skills, MCP services, tags, and files through
QA handlers into agent config, runtime_context/must_use user blocks, and
tool registration. Fix tag-only scopes dropping KB tools, refresh suggested
questions on mention changes, and strengthen MCP must_use so @selected
services are invoked before local KB search.
Two pre-existing test failures on main, unrelated to any feature change:
- Doris VectorRetrieve/KeywordsRetrieve inline LIMIT as a literal (Doris
does not support placeholders for LIMIT/OFFSET), but the SQL-shape tests
still expected `LIMIT ?` with TopK as a bound arg. Match the literal
LIMIT and drop the extra argument.
- TestOssEnsureBucket_CreateFails used the bucket name "test-bucket",
which already exists globally on OSS, so IsBucketExist short-circuited
and the create path never ran. Use a non-existent bucket name so the
create-with-invalid-credentials path is actually exercised.
Clear tag relations on move/delete, validate tags on create, batch-load tags,
restore KB-scoped reference counts, dedupe SetKnowledgeTags, fix filter
double-fetch and ResizeObserver leaks, and regenerate Swagger for tag_ids.
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.
- Persist per-user access/refresh tokens and per-service DCR client
credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
authorization status/actions, and render the connection test result
inline in the editor drawer instead of a stacked centered dialog
Cascade soft-delete tenant_members when a workspace is removed so the
tenant switcher no longer shows ghost entries as "#<id>". Also filter
orphaned memberships in /auth/me and switch to another workspace after
delete when one remains.
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.
Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.
Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.
Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
Three related improvements to the IM-session experience:
- Auto-generated titles: a new IM session starts untitled and gets a
content-based title from its first message via the same GenerateTitleAsync
flow web sessions use, instead of an identity title like "user · dm". Falls
back to the identity title when the first message has no text. All platforms.
- In-flight reply recovery (web chat): opening a session whose reply is still
being generated on an IM channel made the web client attach via
continue-stream and fail (the IM answer never streams through this server),
flashing a "stream failed" toast and forcing a manual refresh. Treat a failed
attach to an IM reply (channel === 'im') as recoverable — poll until it
completes, then reload the thread; on timeout surface the standard error.
Web/api replies keep the original behaviour, untouched.
- Source-filter classification: the sidebar filter treats "web" as sessions
with no IM mapping, but the list join excluded soft-deleted mappings, so
/clear'd IM sessions leaked into the user's own web chats. Include soft-deleted
mappings so a session ever bound to an IM channel stays under its platform.
Adds a repository regression test.
Introduced a new test file for the knowledge repository, implementing various scenarios to ensure the reliability of the finalizing subtask counter. Key tests include:
- Concurrent promotion of subtasks to verify atomic behavior.
- Handling of partial decrements to ensure the row remains in "finalizing" state.
- Safety checks to prevent underflow of the pending subtask count.
- Regression tests to confirm that updates do not overwrite the pending counter.
These tests address previous issues with the finalizing process and enhance overall stability.
The finalizing subtask counter (pending_subtasks_count) could get stuck at a
non-zero value, leaving knowledge permanently in "finalizing" and never
promoted to "completed". Root cause and related hardening:
- UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the
omit list, so any concurrent enrichment subtask that loaded the row, did slow
work (LLM call), then saved an unrelated field wrote back the STALE counter,
clobbering decrements other subtasks performed in between. Add
PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers
(SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it.
- FinalizeSubtask no longer gates the finalizing->completed promote on a
separate re-read of the counter. Every caller unconditionally runs the
guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single
authoritative atomic check, so a racy/stale read can't strand the row.
- Decrements run on a context detached from the caller's cancellation
(context.WithoutCancel + timeout) so graceful shutdown / preemption can't
silently skip a decrement.
- Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph)
and release any shortfall so un-enqueued planned slots (e.g. graph with
NEO4J off) don't strand the row.
- Reparse paths reset pending_subtasks_count via an explicit column write since
full-row Save now omits it.
Question generation is also moved off the synchronous single-task path onto a
batched async fan-out (windows of text chunks), each batch independently
queued / retried / traced, grouped under a postprocess.question span.
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.
Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
buildEnvStoreForDriver — every gated surface now recognises
"opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
in initRetrieveEngineRegistry; NewEngineFactory now closes over the
AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
testOpenSearchConnection case so CreateStore's connectivity probe
accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
NewOpenSearchClient is exported so the factory and env path can build
the TLS-hardened client; healthcheck.go reuses the unexported
probeVersion / probeKNNPlugin for the service-layer probe.
Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.
Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
search → BatchSave with the source_id remap for generated questions —
so dim/keyword routing and the source_id contract come from BatchSave
for free. embeddingMap is keyed by the *target* SourceID because
OpenSearch's BatchSave looks up embeddings by SourceID
(lookupEmbedding), not by chunk_id (the ES driver's convention).
Pagination is from/size; copies larger than max_result_window
(default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
group the input by target value and issue one _update_by_query per
group over the cross-dim <base>_* pattern. Caller values flow through
bound script params only — never string-interpolated into the Painless
source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
full failure reason goes to the debug log only; the returned error
carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
shape as DeleteByQuery's quirk), so refresh=wait_for is not
expressible; we use refresh=true.
Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
functional option) lets the driver emit opensearch.index_created and
opensearch.reindex_executed events without importing any service
package — the service layer implements the interface. NewRepository
takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
When the context carries no tenant (the env-path registration ctx
during boot, for example) the adapter skips the emit with a warning
rather than silently writing tenant_id=0, which would collide with the
system-scope sentinel.
Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
immutable. VectorStoreSettings.vue is now schema-driven: a closed
`enum` renders a t-select; number inputs use the schema's `:min`/`:max`
and fall back to the legacy replica-vs-shard heuristic only when the
schema does not pin them; a danger-coloured warning fires when
insecure_skip_verify is toggled on (the switch and warning are wrapped
in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
knn_engine / insecure_skip_verify plus the warning copy in en-US,
ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
security plugin disabled for dev only). OpenSearch Dashboards lives in a
separate, opt-in opensearch-ui profile so the heavy UI container is not
forced up alongside the cluster (the driver e2e is fully curl-verifiable
against :9200). The new docs/dev/opensearch-integration-test.md covers the
end-to-end exercise and the single-node guidance (set replicas=0 to keep
the cluster Green).
Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
their positive counterparts in this PR. The test suite was the
activation checklist; the activation flip is its diff.
Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
other engines' serialised config is byte-identical. Existing
Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
stores are untouched. No migrations.
Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
endpoint), the docreader gRPC tests, and the doris SQL-shape tests
fail; all three are pre-existing on upstream/main and untouched by
this PR.
- New tests across internal/types, opensearch, service and container —
including a full end-to-end env-path test that exercises
initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
httptest cluster.
- Updated the loadTags function to prevent unnecessary calls when tags are already loading, enhancing performance and user experience.
- Modified tag loading calls in various tag-related functions to ensure the reset parameter is consistently set to true, ensuring the tag list is refreshed correctly after operations like create, edit, and delete.
- Improved the FAQEntryManager component to handle tag loading more efficiently during scrolling and batch operations.
This commit introduces the CancelOpenSpansByName method in the KnowledgeSpanRepository, allowing for the cancellation of open spans by their name for a specific knowledge ID and attempt. This functionality is crucial for managing spans during retries or server restarts, preventing duplicate entries in the trace tree. Additionally, a new test case, TestKnowledgeSpanRepo_CancelOpenSpansByName, has been added to ensure the correct behavior of this method, verifying that only the intended spans are cancelled while others remain unaffected. This enhancement improves the robustness of span management in the application.
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
Three fixes in response to user feedback:
1. Span input disappearing on End/Fail
The Upsert's DoUpdates always listed input/output/metadata, so calls
that only set output (EndSpan) or only set error_* (FailSpan) wrote
NULL into input/metadata, clobbering whatever Begin had recorded.
Build the column list dynamically: skip input/output/metadata when
the incoming row's value is nil. nil now means "preserve existing"
(matches user's intuition "Begin recorded it, End shouldn't erase it").
2. Subspans not auto-expanded
Stages with children (multimodal.image[*], postprocess.summary,
postprocess.question, postprocess.graph.chunk[*]) required a click
on the ▸ caret to surface — easy to miss. On the FIRST successful
fetch per (knowledgeId × attempt), auto-expand any stage that has
children. Subsequent polls honor whatever the user has collapsed,
so manual collapse mid-parse stays collapsed.
3. Auto-poll still not firing
Force-arm the polling interval in onMounted regardless of state.
The per-tick callback decides whether to actually fetch based on
current parse_status — so the loop can never get stranded waiting
on a status transition that already happened. Added a console.debug
when the interval arms, so we can verify from DevTools console that
polling is actually running.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Collapse migrations 000052 (flat stages) + 000053 (span tree) into a
single 000052_knowledge_processing_spans migration; the flat stages
table never escaped this branch and the create-then-drop sequence had
no value.
- BeginStage: detect an existing (kid, attempt, stage) row before
inserting and reuse its span_id with reset state, so re-entry from
asynq retries or adjacent code paths no longer produces duplicate
timeline segments.
- FailSpan: when sibling-stage cascade flips a dependent stage to
cancelled, also CancelDescendants on its subtree so already-running
subspans (embedding.batch[i] etc.) cannot remain as orphan running
rows under a cancelled parent.
- Dead-letter callback: replace two sequential UpdateKnowledgeColumn
writes with a single UpdateKnowledgeColumns map update so we cannot
end up with parse_status=failed and stale error_message (or vice
versa) when one of the two writes fails.
- touchKnowledgeHeartbeat: skip subspan/generation transitions; only
root and stage transitions poke knowledge.updated_at. Spans-table
MAX(updated_at) already covers subspan progress for housekeeping, so
this avoids 2*N+ UPDATE bursts on the same hot row when a multimodal
stage fans out to many images.
- Add regression tests for the BeginStage idempotency contract and the
cascade-into-subspans behaviour.
Addresses three review concerns from the prior PR:
1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5).
This commit adds coverage for the four most regression-prone
surfaces: span repo upsert/cascade/attempt isolation, span
tracker cascade-cancel and cross-process LookupStage,
housekeeping false-kill protection, and handler tree assembly.
2. Housekeeping was using only knowledge.updated_at as its staleness
signal, but knowledge.updated_at advances only at parse_status
transitions — a long DocReader call (or large embedding batch)
can run for an hour with no updated_at change, so a tight
DocumentProcessTimeout setting would falsely flip an actively
running parse to "failed".
The sweep now does a two-stage check: candidates by knowledge
updated_at, then filtered by MAX(spans.updated_at). Every
SpanTracker Begin/End/Fail/Skip now also pokes
knowledge.updated_at as a side-channel heartbeat, so the
filter sees recent activity even when no parse_status
transition has fired.
3. parseHeartbeatTime accepts the timestamp formats both Postgres
and SQLite emit for an aggregated MAX() column (the SQLite
driver doesn't auto-cast aggregates to time.Time the way
Postgres does), so the same code path works in Lite mode.
The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression
test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at
combined with a 2-minute-fresh span row must NOT be killed.
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
Adds a five-segment progress model for the document parsing pipeline so
the UI (PR③) can render a timeline showing where each document is
(DocReader → Chunking → Embedding → Multimodal → PostProcess) and
which stage failed with what error code.
- New table `knowledge_processing_stages` (migration 000052) with one
row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an
attempt counter so re-parses don't lose history.
- StageTracker service exposes Begin/Done/Fail/Skip; all calls are
best-effort and never break the pipeline if persistence fails.
- Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT /
VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized
remediation hints.
- Tracker call sites added at the four meaningful failure points:
convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding),
enqueueImageMultimodalTasks (Multimodal start),
KnowledgePostProcess.Handle (Multimodal close + PostProcess).
- New endpoint `GET /api/v1/knowledge/:id/stages` returns the five
canonical stages — missing rows are synthesized as "pending" so
the timeline always renders five segments. Includes current_stage
and last_error block.
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the
PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their
real implementations, drops the stubs from `stubs.go`, and adds the
27 corresponding test cases that were intentionally held back. The
driver remains gated dead code (no registry / factory / env path
mentions OpenSearch — PR 3 ships the activation switch); the
behaviour added here only fires once PR 3 lands.
Depends on PR 2a of #1440 (driver skeleton).
What lands in PR 2b
-------------------
Three new production files (+762 LoC):
- `query.go` (+154) — typed retrieveFilters struct as the only entry
to query construction (closes the map[string]any injection surface;
hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN,
applies min_score when Threshold > 0) and buildKeywordQuery (BM25);
native hybrid intentionally omitted (out of scope for Phase 3 —
service-layer RRF stays in charge). Filter struct carries
ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode
use cases.
- `retrieve.go` (+204) — Retrieve resolves the per-dim alias from
(1) params.AdditionalParams["dim"], (2) len(params.Embedding), or
(3) the cross-dim wildcard <base>_* for BM25-only callers. Always
returns exactly one *RetrieveResult (the Phase 2 fan-out contract).
Hit parsing caps response body at 16 MB and warns if _id !=
_source.chunk_id. Score passes through raw in [0, 1] — the k-NN
plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped
(1 + cos) / 2 before the client sees it.
- `crud.go` (+405) — Save / BatchSave with _id = chunk_id for
idempotency, dim-aware pre-marshal NDJSON size cap (rejects with
ErrBatchTooLarge before allocating a 50 MB body for a too-big
batch), and per-item error inspection that does NOT leak cluster-
side Reason strings into wrapped public errors. additionalParams
contract verified against elasticsearch/structs.go
ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding
is map[string][]float32 keyed by SourceID, chunk_enabled is
map[string]bool keyed by ChunkID. Missing embedding routes Save /
BatchSave to the dim-less <base>_keywords index. Sync DeleteBy*
paths capped at 1000 IDs per call with ErrBatchTooLarge.
toDoc writes the is_recommended boolean from IndexInfo. The
matching mapping entry was added in PR 2a alongside is_enabled —
the pair stays in sync between the vector and the keyword-only
index so the field round-trips through CopyIndices (PR 3) and any
future FAQ-priority filtering.
Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve /
DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList
no longer return ErrFeatureNotEnabled — they delegate to the
production implementations above. The remaining stubs (CopyIndices,
BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion,
EstimateStorageSize lower-bound) stay until PR 3.
Tests (+490 net to repository_test.go, 28 new cases):
- retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs,
IncludeDisabled skip semantics, default is_enabled=true pin.
- buildKNNQuery: min_score applied when Threshold > 0; omitted at
threshold zero.
- buildKeywordQuery: content match appended after typed filters.
- toDoc: embedding field omitted when empty, present when set,
source_type serialised as integer, **is_recommended round-trips
both true and false**.
- lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key
contract; nil-params and wrong-type degrade paths.
- extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings
yield dim=0.
- estimateBulkBodyBytes: size estimator boundary.
- effectiveTopK: clamping when caller leaves TopK at the magic-10
default.
- resolveDim: AdditionalParams takes precedence; Embedding fallback;
zero-zero triggers cross-dim multi-index search.
- inspectBulkResponse: all-succeeded fast path; total-count + leak-
guard for cluster-side Reason strings.
- BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal);
empty list is a no-op.
- DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty
list is a no-op.
SDK quirk addressed in PR 2b
----------------------------
DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level
"wait_for" value documented in the OpenSearch REST API is not
directly expressible. deleteByTerms uses Refresh:&true which forces
immediate segment flush (read-your-writes guaranteed but more
expensive than wait_for). A follow-up PR can drop to
*opensearchapi.Client.Transport.Perform for wait_for if telemetry
shows the cost is material.
Backward compatibility
----------------------
- Additive within the opensearch package only. No file outside the
package is touched.
- Driver still unreachable: PR 3 ships the activation switch.
- The PR 1 normalizer case for OpenSearch remains unreachable here
(no driver instance produces a result yet).
Test plan
---------
- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
shows only PR 1's normalizer case + this driver's EngineType()
and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.