2a54bb2d45dd41b9569aa2dad753f9cecc8d36cc
605 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wolfkill
|
2a54bb2d45 | fix: avoid stale knowledge records on upload failure | ||
|
ochan.kwon
|
40b74e2efa |
feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3)
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482) laid the type prep + driver skeleton + read/write paths as gated dead code; this PR wires every activation surface so opensearch becomes a registerable VectorStore engine. Activation wiring - internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping / buildEnvStoreForDriver — every gated surface now recognises "opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM / HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines' serialised config byte-identical. - internal/container: createOpenSearchEngine + the switch case in createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path in initRetrieveEngineRegistry; NewEngineFactory now closes over the AuditLogService (the EngineFactory type itself is unchanged). - internal/application/service/vectorstore_healthcheck.go: a testOpenSearchConnection case so CreateStore's connectivity probe accepts opensearch instead of returning 400. - internal/application/repository/retriever/opensearch/transport.go: NewOpenSearchClient is exported so the factory and env path can build the TLS-hardened client; healthcheck.go reuses the unexported probeVersion / probeKNNPlugin for the service-layer probe. Service-layer validation - validateOpenSearchIndexConfig validates the HNSW caps (m 2-100, ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss). Shards/replicas continue to be enforced by the flat ValidateIndexConfig. Create-only: UpdateStore mutates the name only. - validateConnectionConfig requires addr for opensearch. Sync implementations (stubs.go shrinks) - CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern — search → BatchSave with the source_id remap for generated questions — so dim/keyword routing and the source_id contract come from BatchSave for free. embeddingMap is keyed by the *target* SourceID because OpenSearch's BatchSave looks up embeddings by SourceID (lookupEmbedding), not by chunk_id (the ES driver's convention). Pagination is from/size; copies larger than max_result_window (default 10000) need the scroll-based async path that lands later. - BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go) group the input by target value and issue one _update_by_query per group over the cross-dim <base>_* pattern. Caller values flow through bound script params only — never string-interpolated into the Painless source — closing the script-injection surface. - inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the full failure reason goes to the debug log only; the returned error carries the bounded id + type. - UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same shape as DeleteByQuery's quirk), so refresh=wait_for is not expressible; we use refresh=true. Driver-owned audit (DIP) - A new opensearch.AuditSink interface (with nopSink + WithAuditSink functional option) lets the driver emit opensearch.index_created and opensearch.reindex_executed events without importing any service package — the service layer implements the interface. NewRepository takes opts, so existing 4-arg test call sites keep compiling unchanged. - internal/container/audit_sink.go bridges AuditSink to AuditLogService. When the context carries no tenant (the env-path registration ctx during boot, for example) the adapter skips the emit with a warning rather than silently writing tenant_id=0, which would collide with the system-scope sentinel. Frontend + polish - FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/ immutable. VectorStoreSettings.vue is now schema-driven: a closed `enum` renders a t-select; number inputs use the schema's `:min`/`:max` and fall back to the legacy replica-vs-shard heuristic only when the schema does not pin them; a danger-coloured warning fires when insecure_skip_verify is toggled on (the switch and warning are wrapped in a vertical stack so the warning sits on its own row below the switch). - i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search / knn_engine / insecure_skip_verify plus the warning copy in en-US, ko-KR, zh-CN, ru-RU. - docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with security plugin disabled for dev only). OpenSearch Dashboards lives in a separate, opt-in opensearch-ui profile so the heavy UI container is not forced up alongside the cluster (the driver e2e is fully curl-verifiable against :9200). The new docs/dev/opensearch-integration-test.md covers the end-to-end exercise and the single-node guidance (set replicas=0 to keep the cluster Green). Gating-guard tests flipped - The "OpenSearch is NOT in validEngineTypes / mapping / types list / env builder / stubs" guard tests from PR 1 / PR 2 are replaced by their positive counterparts in this PR. The test suite was the activation checklist; the activation flip is its diff. Backward compatibility - Additive everywhere. IndexConfig's new HNSW fields are omitempty so other engines' serialised config is byte-identical. Existing Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB stores are untouched. No migrations. Test plan - go build ./... clean - go vet ./... clean - gofmt -l clean on touched files - go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS endpoint), the docreader gRPC tests, and the doris SQL-shape tests fail; all three are pre-existing on upstream/main and untouched by this PR. - New tests across internal/types, opensearch, service and container — including a full end-to-end env-path test that exercises initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an httptest cluster. |
||
|
wizardchen
|
7d1dfc78b7 |
fix(knowledgeBase): improve tag loading logic and ensure consistent behavior
- Updated the loadTags function to prevent unnecessary calls when tags are already loading, enhancing performance and user experience. - Modified tag loading calls in various tag-related functions to ensure the reset parameter is consistently set to true, ensuring the tag list is refreshed correctly after operations like create, edit, and delete. - Improved the FAQEntryManager component to handle tag loading more efficiently during scrolling and batch operations. |
||
|
wizardchen
|
48381fbaf5 |
feat: implement tenant default storage provider handling in Knowledge Base creation
This commit introduces functionality to utilize the tenant's default storage provider when creating a Knowledge Base. It includes updates to the frontend to load the default provider from settings and apply it during Knowledge Base initialization. Additionally, the backend has been enhanced to ensure that the storage provider is set correctly based on tenant configuration, improving consistency across the application. Tests have been added to verify the correct application of the default storage provider in various scenarios. |
||
|
wizardchen
|
e0823eff31 |
feat: enhance IM file service tests and refactor storage URL handling
This commit introduces a new test suite for the IM file service, including a stub implementation for testing purposes. It adds tests for resolving file services based on storage providers and ensures proper fallback mechanisms for MinIO URLs. Additionally, the `rewriteStorageURLs` and `cleanIMContent` functions have been refactored to utilize a resolver for improved caching and efficiency. These changes enhance the robustness of file service handling and improve test coverage for various storage scenarios. |
||
|
wizardchen
|
5d02404fdd |
feat: add unit tests for COS object name parsing and enhance error handling
This commit introduces unit tests for the `parseCosObjectName` method in the `cosFileService`, ensuring it correctly rejects local scheme URLs and properly parses COS scheme URLs. Additionally, the `parseCosObjectName` method has been updated to return an error for unsupported schemes, improving error handling in the `GetFile` and `DeleteFile` methods. This enhancement ensures more robust handling of file paths in the application. |
||
|
wizardchen
|
0e1282c2da |
test: add document process task options tests
Centralize DefaultDocumentProcessTimeout in the config package and reuse DocumentProcessTimeout() from documentProcessTaskOptions. |
||
|
HelloWeit
|
edfabf3135 | bugfix | ||
|
wizardchen
|
f1a27e0e18 |
feat: implement CancelOpenSpansByName method and related tests
This commit introduces the CancelOpenSpansByName method in the KnowledgeSpanRepository, allowing for the cancellation of open spans by their name for a specific knowledge ID and attempt. This functionality is crucial for managing spans during retries or server restarts, preventing duplicate entries in the trace tree. Additionally, a new test case, TestKnowledgeSpanRepo_CancelOpenSpansByName, has been added to ensure the correct behavior of this method, verifying that only the intended spans are cancelled while others remain unaffected. This enhancement improves the robustness of span management in the application. |
||
|
wizardchen
|
44d6175559 |
feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
|
||
|
wizardchen
|
797f55c567 |
feat(timeline): enhance knowledge processing spans and UI elements
This commit introduces several improvements to the knowledge processing timeline and related components. Key changes include: 1. Added a `gracePoll` prop to the `KnowledgeProcessingTimeline` component to manage polling behavior more effectively. 2. Enhanced the UI by displaying the document title in the drawer, improving user visibility of the current document context. 3. Implemented new CSS classes for better styling of the drawer title bar, ensuring a more polished appearance. 4. Updated the backend to support the new `WikiSpan` tracking, allowing for detailed monitoring of document processing stages. These changes aim to improve user experience and provide better insights into the document processing workflow. |
||
|
wizardchen
|
e697eee07f |
feat(trace): enable direct access to trace drawer from card menu
This commit introduces a new method to open the trace drawer directly from the card menu, enhancing user experience by allowing immediate access to trace details without navigating through the document detail drawer. The implementation includes updates to the `handleViewTrace` function to ensure the correct knowledge ID and parse status are set before opening the trace drawer. Additionally, minor adjustments were made to the UI for better consistency and clarity. |
||
|
wizardchen
|
2a4d6f9019 |
fix: preserve span input/metadata, auto-expand subspans, force-arm poll
Three fixes in response to user feedback: 1. Span input disappearing on End/Fail The Upsert's DoUpdates always listed input/output/metadata, so calls that only set output (EndSpan) or only set error_* (FailSpan) wrote NULL into input/metadata, clobbering whatever Begin had recorded. Build the column list dynamically: skip input/output/metadata when the incoming row's value is nil. nil now means "preserve existing" (matches user's intuition "Begin recorded it, End shouldn't erase it"). 2. Subspans not auto-expanded Stages with children (multimodal.image[*], postprocess.summary, postprocess.question, postprocess.graph.chunk[*]) required a click on the ▸ caret to surface — easy to miss. On the FIRST successful fetch per (knowledgeId × attempt), auto-expand any stage that has children. Subsequent polls honor whatever the user has collapsed, so manual collapse mid-parse stays collapsed. 3. Auto-poll still not firing Force-arm the polling interval in onMounted regardless of state. The per-tick callback decides whether to actually fetch based on current parse_status — so the loop can never get stranded waiting on a status transition that already happened. Added a console.debug when the interval arms, so we can verify from DevTools console that polling is actually running. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
wizardchen
|
5dc0a49a9b |
feat(timeline): enrich per-image multimodal subspan output
The per-image multimodal subspan only captured image_url / enable_ocr / enable_caption on input and chunk_id on output, so the trace viewer could not answer "what did THIS image actually produce?" without joining back to the chunks table. Adds to the per-image span output: - vlm_model_id (or "legacy_inline" for inline-config KBs) - image_bytes (read size) - ocr_prompt: "default" | "scanned_pdf" - ocr_chars + ocr_preview (sanitized text, capped at 200 runes) - caption_chars + caption_preview - chunks_created (count of OCR/caption child chunks) - indexed (true after BatchIndex completes) - per-step error fields (read_error / ocr_error / caption_error / skipped reason) when something fails Also adds parent_chunk_id to the span input so the trace links back to the text chunk this image hangs off — useful when a doc has hundreds of inline images and you need to know WHERE in the text this one came from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
wizardchen
|
f99bcb79e7 |
feat(timeline): scale axis to async tail, enrich span payloads, polish UI
The post-process stage closes in ~9ms (just enqueue work) but its async subspans (postprocess.summary, postprocess.question, postprocess.graph) keep producing rows for tens of seconds AFTER the root finalizes. The old timeline used trace.duration_ms as the time axis maximum, which clipped those subspan bars past the right edge. Timeline: - totalMs now always takes max(trace.duration_ms, observed-tail), so the axis stretches to fit the latest descendant end regardless of parse_status. - Render a faint dashed wrapping outline behind a parent span when its descendants extend past its own finished_at, so the postprocess stage row visibly spans the full window without overloading the 9ms self-time bar. - Tree expand/collapse caret bumped from 10 to 14px in a 16x16 hit area; copy icons in detail panel bumped from 11/14 to 14/18px; .kp-kv-copy button grown from 18 to 22px. - Short input/output payloads (<= 8 entries / <= 600 bytes JSON) auto-expand inline so users see the actual data without an extra click; longer payloads keep the click-to-expand summary. Span payloads (subspans only - root keeps the canonical identity, no duplicate knowledge_id/kb_id/tenant_id on every child): - extract.go: graph subspan output gains chunk_chars, chunk_preview, sample_nodes, sample_relations. - summary subspan output gains model_id, summary_preview. - question subspan output gains model_id and a sample_question captured from the first non-empty LLM response. i18n: new key knowledgeStages.detail.includingChildren for the wrapping-bar tooltip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
wizardchen
|
bd029b6d19 |
feat(knowledge): instrument postprocess subspans + polish timeline UI
Backend: summary, question, and graph-extract async tasks now record real processing time as subspans hanging off the (closed) postprocess stage, so the trace viewer no longer caps the postprocess row at the ~10 ms enqueue duration. Carries Attempt through SummaryGeneration / QuestionGeneration / ExtractChunk payloads so cross-process workers can resolve the right parent attempt. Frontend: drawer now uses attach="body" so the secondary 820px detail drawer escapes the 654px container of the main drawer; timeline timestamps include date prefix (MM-DD, or YYYY-MM-DD across years); "updated X ago" caption only shows during live polling. Tests: 4 new cases covering postprocess subspan attaching under a closed parent, missing-parent fallthrough, and Attempt JSON round-trip on the three task payloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
wizardchen
|
4db6e69b4e |
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
|
||
|
wizardchen
|
c82b098f44 |
fix(knowledge): address span-tree review feedback
- Collapse migrations 000052 (flat stages) + 000053 (span tree) into a single 000052_knowledge_processing_spans migration; the flat stages table never escaped this branch and the create-then-drop sequence had no value. - BeginStage: detect an existing (kid, attempt, stage) row before inserting and reuse its span_id with reset state, so re-entry from asynq retries or adjacent code paths no longer produces duplicate timeline segments. - FailSpan: when sibling-stage cascade flips a dependent stage to cancelled, also CancelDescendants on its subtree so already-running subspans (embedding.batch[i] etc.) cannot remain as orphan running rows under a cancelled parent. - Dead-letter callback: replace two sequential UpdateKnowledgeColumn writes with a single UpdateKnowledgeColumns map update so we cannot end up with parse_status=failed and stale error_message (or vice versa) when one of the two writes fails. - touchKnowledgeHeartbeat: skip subspan/generation transitions; only root and stage transitions poke knowledge.updated_at. Spans-table MAX(updated_at) already covers subspan progress for housekeeping, so this avoids 2*N+ UPDATE bursts on the same hot row when a multimodal stage fans out to many images. - Add regression tests for the BeginStage idempotency contract and the cascade-into-subspans behaviour. |
||
|
wizardchen
|
414749681f |
test(knowledge): add coverage and protect housekeeping from false-killing long stages
Addresses three review concerns from the prior PR:
1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5).
This commit adds coverage for the four most regression-prone
surfaces: span repo upsert/cascade/attempt isolation, span
tracker cascade-cancel and cross-process LookupStage,
housekeeping false-kill protection, and handler tree assembly.
2. Housekeeping was using only knowledge.updated_at as its staleness
signal, but knowledge.updated_at advances only at parse_status
transitions — a long DocReader call (or large embedding batch)
can run for an hour with no updated_at change, so a tight
DocumentProcessTimeout setting would falsely flip an actively
running parse to "failed".
The sweep now does a two-stage check: candidates by knowledge
updated_at, then filtered by MAX(spans.updated_at). Every
SpanTracker Begin/End/Fail/Skip now also pokes
knowledge.updated_at as a side-channel heartbeat, so the
filter sees recent activity even when no parse_status
transition has fired.
3. parseHeartbeatTime accepts the timestamp formats both Postgres
and SQLite emit for an aggregated MAX() column (the SQLite
driver doesn't auto-cast aggregates to time.Time the way
Postgres does), so the same code path works in Lite mode.
The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression
test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at
combined with a 2-minute-fresh span row must NOT be killed.
|
||
|
wizardchen
|
c9941a6688 |
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
|
||
|
wizardchen
|
04f56f9cda |
feat(knowledge): track per-stage parsing progress with /stages API
Adds a five-segment progress model for the document parsing pipeline so the UI (PR③) can render a timeline showing where each document is (DocReader → Chunking → Embedding → Multimodal → PostProcess) and which stage failed with what error code. - New table `knowledge_processing_stages` (migration 000052) with one row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an attempt counter so re-parses don't lose history. - StageTracker service exposes Begin/Done/Fail/Skip; all calls are best-effort and never break the pipeline if persistence fails. - Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT / VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized remediation hints. - Tracker call sites added at the four meaningful failure points: convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding), enqueueImageMultimodalTasks (Multimodal start), KnowledgePostProcess.Handle (Multimodal close + PostProcess). - New endpoint `GET /api/v1/knowledge/:id/stages` returns the five canonical stages — missing rows are synthesized as "pending" so the timeline always renders five segments. Includes current_stage and last_error block. |
||
|
wizardchen
|
3ae3ea97c5 |
fix(knowledge): prevent documents from getting stuck in "processing"
Several failure modes left Knowledge.parse_status pinned at "processing" forever, with no signal to users beyond a permanent spinner. This commit addresses the root causes and adds a safety net. - Asynq worker pool: explicit Concurrency (default 16, env-tunable via WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms (WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops. - DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT (default 30m). Without this, a hung docreader pinned a worker for the full DocumentProcessTimeout window. - ImageMultimodal: finalize-on-last-attempt semantics. A permanently failing single image no longer strands the parent — the asynq retry is allowed to run, but on the final attempt we count the image regardless of outcome. Redis DECR errors fall back to enqueuing the post-process task instead of returning silently. - Dead-letter callback: when DocumentProcess / KnowledgePostProcess / ManualProcess exhausts retries, immediately mark the corresponding Knowledge as failed with the last error. This surfaces the failure in the UI without waiting for the housekeeping sweep. - HousekeepingService: 5-minute cron that flips knowledge rows stuck in "processing" past DocumentProcessTimeout + 10m to failed, plus summary rows stuck > 1h. Catches anything the other safety nets miss (worker SIGKILL mid-handler, etc.). Disable with WEKNORA_HOUSEKEEPING_ENABLED=false. - Distributed startup recovery: previously the post-restart sweep was skipped whenever REDIS_ADDR was set, even though Asynq does not reschedule the task that was actively running on the dead instance. Now the sweep runs in distributed mode too, but only against rows older than 30 minutes to avoid racing peer instances. |
||
|
Ochan Kwon
|
f5820c13ac |
feat(retriever): implement OpenSearch driver read/write paths (PR 2b of 3) (#1482)
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their real implementations, drops the stubs from `stubs.go`, and adds the 27 corresponding test cases that were intentionally held back. The driver remains gated dead code (no registry / factory / env path mentions OpenSearch — PR 3 ships the activation switch); the behaviour added here only fires once PR 3 lands. Depends on PR 2a of #1440 (driver skeleton). What lands in PR 2b ------------------- Three new production files (+762 LoC): - `query.go` (+154) — typed retrieveFilters struct as the only entry to query construction (closes the map[string]any injection surface; hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN, applies min_score when Threshold > 0) and buildKeywordQuery (BM25); native hybrid intentionally omitted (out of scope for Phase 3 — service-layer RRF stays in charge). Filter struct carries ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode use cases. - `retrieve.go` (+204) — Retrieve resolves the per-dim alias from (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or (3) the cross-dim wildcard <base>_* for BM25-only callers. Always returns exactly one *RetrieveResult (the Phase 2 fan-out contract). Hit parsing caps response body at 16 MB and warns if _id != _source.chunk_id. Score passes through raw in [0, 1] — the k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped (1 + cos) / 2 before the client sees it. - `crud.go` (+405) — Save / BatchSave with _id = chunk_id for idempotency, dim-aware pre-marshal NDJSON size cap (rejects with ErrBatchTooLarge before allocating a 50 MB body for a too-big batch), and per-item error inspection that does NOT leak cluster- side Reason strings into wrapped public errors. additionalParams contract verified against elasticsearch/structs.go ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding is map[string][]float32 keyed by SourceID, chunk_enabled is map[string]bool keyed by ChunkID. Missing embedding routes Save / BatchSave to the dim-less <base>_keywords index. Sync DeleteBy* paths capped at 1000 IDs per call with ErrBatchTooLarge. toDoc writes the is_recommended boolean from IndexInfo. The matching mapping entry was added in PR 2a alongside is_enabled — the pair stays in sync between the vector and the keyword-only index so the field round-trips through CopyIndices (PR 3) and any future FAQ-priority filtering. Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve / DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList no longer return ErrFeatureNotEnabled — they delegate to the production implementations above. The remaining stubs (CopyIndices, BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion, EstimateStorageSize lower-bound) stay until PR 3. Tests (+490 net to repository_test.go, 28 new cases): - retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs, IncludeDisabled skip semantics, default is_enabled=true pin. - buildKNNQuery: min_score applied when Threshold > 0; omitted at threshold zero. - buildKeywordQuery: content match appended after typed filters. - toDoc: embedding field omitted when empty, present when set, source_type serialised as integer, **is_recommended round-trips both true and false**. - lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key contract; nil-params and wrong-type degrade paths. - extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings yield dim=0. - estimateBulkBodyBytes: size estimator boundary. - effectiveTopK: clamping when caller leaves TopK at the magic-10 default. - resolveDim: AdditionalParams takes precedence; Embedding fallback; zero-zero triggers cross-dim multi-index search. - inspectBulkResponse: all-succeeded fast path; total-count + leak- guard for cluster-side Reason strings. - BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal); empty list is a no-op. - DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty list is a no-op. SDK quirk addressed in PR 2b ---------------------------- DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level "wait_for" value documented in the OpenSearch REST API is not directly expressible. deleteByTerms uses Refresh:&true which forces immediate segment flush (read-your-writes guaranteed but more expensive than wait_for). A follow-up PR can drop to *opensearchapi.Client.Transport.Perform for wait_for if telemetry shows the cost is material. Backward compatibility ---------------------- - Additive within the opensearch package only. No file outside the package is touched. - Driver still unreachable: PR 3 ships the activation switch. - The PR 1 normalizer case for OpenSearch remains unreachable here (no driver instance produces a result yet). Test plan --------- - [x] go build ./... clean - [x] go vet ./... clean - [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes - [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/ shows only PR 1's normalizer case + this driver's EngineType() and tests — no activation path. - [x] grep -r "case \"opensearch\"" internal/ shows no hits. |
||
|
Sheep44044
|
3645477dba | refactor(qdrant): optimize batch save, map iteration, and error wrapping (#1508) | ||
|
lyingbug
|
3611a8607d |
feat(rbac): add multi-use share-link invitations for invite_only mode (#1504)
* feat(rbac): add multi-use share-link invitations for invite_only mode When the auth.registration_mode toggle flips to invite_only, the public /auth/register endpoint returns 403 — but until now there was no channel for unregistered users to actually join a tenant. The existing invitation table required a registered invitee, so new emails could not enter the system at all. This change introduces a share-link model: - Owner generates a multi-use registration link with a per-tenant role - Recipient opens the link, registers with their own email, and is added to the tenant on the same request - Token is stored in plaintext (not hashed) so the management UI can re-display the URL on demand without a "copy now or revoke" trap. Threat model is bounded by 7-day TTL, revocability, and the fact that all the link grants is membership in one tenant. - accepted_count tracks how many users joined through the link so the Owner can tell whether a link is fresh or has already been spread Per-user invitations (registered email -> in-app inbox accept) are unchanged. Share-link rows use empty invitee_user_id as discriminator; the partial unique index on (tenant_id, invitee_user_id) was relaxed to skip empty values so multiple share links can coexist per tenant. Frontend reuses Login.vue for /register?token=xxx instead of a parallel page, and adds icon-only revoke / remove actions plus a distinct "active" status tag for share-link rows. Migration 000054 adds the token column, the accepted_count column, and the relaxed pending unique index in a single step. * feat(auth): update invitation token handling and add rate limiting - Refactor the invitation token retrieval to use POST instead of GET, enhancing security by preventing the plaintext token from appearing in logs and browser history. - Update the API endpoint from `/auth/invitations/:token` to `/auth/invitations/lookup` to reflect the new method. - Introduce a rate limiter for unauthenticated share-link endpoints to mitigate brute-force attacks and abuse, ensuring a maximum of 30 requests per minute per IP. - Adjust the registration flow to maintain user experience while securing the invitation process. |
||
|
wizardchen
|
9cce0c8e5e |
feat(system): consolidate system admin and settings into one Settings panel
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.
Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
key so the 3-tier resolver falls back to ENV / built-in default.
Idempotent (resetting a never-persisted key returns nil); emits an
audit row only on real deletions, invalidates the local cache, and
publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
storage_quota in one statement. Powers POST /system/admin/tenants/
apply-default-storage-quota; bypasses the per-tenant whitelist on
PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
emitAdminAudit() in SystemHandler — promote / revoke now leave an
audit trail with TenantID=0 and {target_email,target_username,
idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
(username, email fallback) so the UI shows "wizardchen" instead of
a UUID prefix without storing a denormalised column.
Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
auto-persisting controls (switch/select @change, input/inputnumber
@blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
roster). auth.registration_mode change goes through an inline
popconfirm; cancel rolls back. Reset / bulk-apply also inline
popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
admins are now compatibility redirects to /platform/settings?section=
system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
so the wider tables (members, system settings) breathe; <1100px
viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
empty / badges / priority hint / per-key labels / reset / bulkApply /
admins (label, placeholder, save messages, popconfirm copy) across
zh-CN, en-US, ko-KR, ru-RU.
Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
deploy-time-only knob (nginx + docreader + frontend each cache the
env at startup); a SystemAdmin UI override would mislead operators
because nginx would still 413. Until all four layers can hot-reload
the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
off SystemSettingService for live updates; ENV remains the fallback
for never-overridden deployments.
|
||
|
wizardchen
|
d074dc067a |
feat(system-admin): implement revocation of system admin privileges with safeguards
- Added RevokeSystemAdmin functionality to the user service and repository, ensuring atomic checks for self-revoke and last admin scenarios. - Updated the system handler to utilize the new revocation method, improving error handling for various edge cases. - Enhanced the bootstrap process to prevent unintended promotions when system admins already exist. - Refactored related comments and documentation for clarity on the new behavior and safeguards in place. |
||
|
wizardchen
|
47a183aa65 |
feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup. - Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic. - Updated `.env.example` to document the new environment variable and its behavior. - Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges. - Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration. - Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management. |
||
|
wizardchen
|
e6ee87759d |
fix(agent): preserve agent intent prompt whitespace and add tests
The intent prompt override logic in query_understand applied strings.TrimSpace to the value before assigning it as the system prompt override, which silently stripped trailing newlines and intentional formatting from agent-supplied prompts. Use TrimSpace only to detect emptiness (so whitespace-only strings still fall back to the global default) while passing the raw string through verbatim. Extract the resolution into applyIntentPromptOverride and add unit tests covering agent-wins, whitespace preservation, blank fallback, no-config, and global-only paths. |
||
|
ochan.kwon
|
11c3236e52 |
feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1 (#1445) by way of #1440. PR 2a ships a hollow, interface-compliant shell of the `internal/application/repository/retriever/opensearch/` package — every behavioural method (Save / BatchSave / DeleteBy* / Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* / EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled` or a conservative sentinel value. PR 2b lands the real read/write implementations in dedicated files (`query.go` + `retrieve.go` + `crud.go`) and replaces the stubs accordingly. Strict feature-gate (unchanged from PR 1): no entry is added to validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping / BuildEnvVectorStores / container env path / engine factory switch, so the driver remains unreachable. Attempting to register an `engine_type=opensearch` VectorStore continues to fail with the existing "not a valid engine type" error. What lands in PR 2a ------------------- Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC): - `repository.go` — Repository struct + NewRepository constructor that validates cluster reachability + OS version (2.4+ / 3.x; primary tested 3.3.2) + k-NN plugin presence on every cluster node. sync.Once-guarded ensureReady(ctx, dim) for lazy per- dimension index creation, with transient errors not cached so a momentary cluster blip does not permanently poison a dim. sanitizeIndexName enforces a strict OS-compatible name spec. probeVersion uses robust strings.Split/Atoi parsing for pre-release suffixes and missing-patch versions. EngineType returns the PR 1 constant; Support returns [keywords, vector]. - `transport.go` — newOpenSearchClient ships TLS posture (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy- only cipher list) and transport tuning for the driver. Caller exists only in PR 3 (container.go + engine_factory.go); PR 2a remains gated dead code. - `mapping.go` — buildIndexMapping(cfg, dim) produces the full knn_vector + HNSW + content-analyzer mapping with every *_id field as an explicit keyword and source_type as integer. buildKeywordsMapping ships the dim-less keyword-only index mapping used by the no-embedding save path. createIndexAndAlias creates <alias>_v1 and aliases <alias> to it, with best-effort orphan cleanup and mapping-drift detection. - `config.go` — internalCfg (value type) applying OpenSearch defaults (hnsw_m=16, ef_construction=100, ef_search=100, shards=4, replicas=1, engine=lucene). - `errors.go` — nine sentinels (ErrIndexNotFound, ErrDimensionMismatch, ErrAuth, ErrTransport, ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled, ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports apperrors; PR 3's engine factory wraps these to typed AppError 2200/2201. - `stubs.go` — every behavioural method returns ErrFeatureNotEnabled. EstimateStorageSize returns a conservative HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard fails-closed for non-empty KBs. Tests (~1115 LoC, 50 cases): - `repository_test.go` — interface satisfaction, sentinel mapping, sanitizeIndexName positive/negative matrix, semver parsing (pre-release / missing-patch), buildIndexMapping JSON shape pin (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 / 2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin multi-node coverage, ensureReady concurrency + per-dim isolation + transient retry, NewRepository storeID validation, all 11 stubs (CopyIndices, BatchUpdate*, EstimateStorageSize, SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*), wrapTransport sentinel mapping + leak guard, isNotFound / isAlreadyExistsError, drainAndClose / limitedDecode helpers. - `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify / TLS 1.2 pinning / cipher list / transport tuning. Single dependency addition: github.com/opensearch-project/ opensearch-go/v4 v4.6.0 in go.mod/go.sum. SDK quirks discovered (opensearch-go v4.6.0) -------------------------------------------- PR 2a includes the workarounds for two of three SDK limitations that landed during full implementation (the third, Refresh:*bool, only affects the delete path that ships in PR 2b): - AliasExists method passes dataPointer=nil to its internal do(), which means non-2xx responses come back as a plain *errors.errorString ("status: 404 Not Found") rather than as *opensearch.StructError. aliasExists therefore inspects resp.StatusCode directly (resp is returned even when err is non-nil) and only falls back to wrapTransport for the "no response at all" case. - sync.OnceReset is not in the standard library; the keyword-only index uses a mutex + ready/err flag pattern so transient failures can be retried by the next caller. The per-dimension path uses the `once map[int]*sync.Once` delete-and-recreate trick. Test fixes folded in -------------------- While doing a full `go test ./...` against PR 1-merged main, two deterministic regressions surfaced that block a clean run-everything signal. Both are unrelated to the driver and are folded into PR 2a so the PR's own CI run is green: (1) Follow-up to #1445 — fanout test missed the new normalizer policy (internal/application/service/knowledgebase_search_fanout_test.go, +46 / -6). #1445 changed EngineAwareNormalizer for ES / ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01 passthrough (those engines surface non-negative cosine to the normalizer per Lucene script_score non-negative invariant for ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for OpenSearch, engine-internal or IR-normalized conversions for the rest). Milvus is now the only engine that still surfaces raw signed cosine in [-1, 1]. TestRetrieveFromStores_MixedEngine_Normalizes still asserted the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3) which under passthrough now becomes clamp01(-0.4) = 0. The normalizer's own _test.go was updated at #1445 time, but this fan-out integration test was not. Fix: rewrite the godoc to spell out the two engine groups; restate sub-case 2 as ES passthrough on a production-possible mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3 pinning the cosine-shift branch via Milvus -0.4 → 0.3. (2) Pre-existing — SSRF whitelist singleton race surfaced by this run (internal/utils/security.go + internal/utils/security_test.go + internal/infrastructure/web_search/searxng_test.go, +33 / -9). loadSSRFWhitelist in internal/utils/security.go is cached via sync.Once on first call. The internal reset helper resetSSRFWhitelistForTest was unexported, so tests in other packages could not reset and saw whatever whitelist was cached by the first sync.Once.Do() in the same test binary. In internal/infrastructure/web_search/, TestValidateProxyURL runs before TestValidateSearxngBaseURL alphabetically and exercises ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty whitelist; the later setenv in searxng_test then has no effect and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted". Pre-existing on main; surfaced now because this PR was the first to do a full `go test ./...` run on top of #1445. Fix: capitalize the helper to ResetSSRFWhitelistForTest (the ForTest suffix is the test-only contract); update in-package callers; in web_search/searxng_test.go import internal/utils and call ResetSSRFWhitelistForTest around the env mutation in both TestValidateSearxngBaseURL and TestSearxngProvider_Search. No production code path changes. Roadmap ------- - PR 2b (next, depends on this PR) — read/write implementations: query.go + retrieve.go + crud.go land their real bodies; stubs for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are removed; corresponding CRUD/retrieve/filter test cases (~430 LoC) join repository_test.go. - PR 3 — activation switch + async paths (CopyIndices, BatchUpdate*, large-batch async deletes) + i18n + docker-compose dev profile. After PR 3 merges, the OpenSearch driver becomes reachable via either `engine_type=opensearch` VectorStore or `RETRIEVE_DRIVER=opensearch` env. Backward compatibility ---------------------- - New package — additive only. No existing file modified except go.mod / go.sum, the two test files in (1)/(2), and the test-only export rename in utils/security.go. - Driver is unreachable: no registry path activates it. - No SQL migration. - The PR 1 normalizer case for OpenSearch remains unreachable here (no driver instance produces a result yet). Test plan --------- - [x] go build ./... clean - [x] go vet ./... clean - [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes - [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/ shows only PR 1's normalizer case + this driver's EngineType() and tests — no activation path. - [x] grep -r "case \"opensearch\"" internal/ shows no hits. |
||
|
ChenRussell
|
80a69ae6f3 |
fix(pipeline): fallback to raw retrieval results when rerank API fails
When the rerank model returns an error (e.g. 401 Unauthorized), the pipeline previously discarded all retrieved candidates and returned empty results to the caller. Now p.rerank returns ([]RankResult, error) to distinguish API failure from threshold-filtered empty results. On API error, the pipeline falls back to the original retrieval candidates (directLoad + candidatesToRerank) and continues to CHUNK_MERGE/FILTER_TOP_K, so users still get results even when the rerank model is misconfigured. |
||
|
helloandyzhang
|
e6a469631b |
feat(agent): add intent prompt customization in agent editor
Allow per-intent system prompt overrides for non-retrieval intents in normal mode. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Miles Lai
|
09892ef763 |
fix(doris): inline LIMIT/OFFSET as literals and enable parameter interpolation
Problem: Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime with SQL parse errors. Root cause: 1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in MySQL/PostgreSQL but not in Doris's SQL dialect) 2. Doris MySQL driver was not configured to interpolate parameters, preventing automatic client-side placeholder replacement Fix: 1. Enable parameter interpolation in Doris connection DSN: - Add &interpolateParams=true to mysql.Open() DSN in container.go - This enables driver-level parameter substitution at query time 2. Inline LIMIT/OFFSET as string literals in Doris queries: - VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK) - KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK) - CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset) - Remove inlined values from args slice Result: Queries are now built with literal LIMIT/OFFSET values instead of placeholders, compatible with Doris's SQL parser. |
||
|
wizardchen
|
e9be53e830 |
fix: respect multi-turn-disabled flag in KnowledgeQA pipeline
When a custom agent has `MultiTurnEnabled=false`, `applyAgentOverridesToChatManage`
sets `chatManage.MaxRounds = 0` to signal "no history". Two pipeline plugins
mistreated this zero value as "use the global default" and silently re-loaded
session history into the LLM context:
- `PluginLoadHistory` fell back to `Conversation.MaxRounds` when
`chatManage.MaxRounds == 0`.
- `PluginQueryUnderstand.loadHistory` had the same fallback, and even when
`LOAD_HISTORY` was skipped it would re-populate `chatManage.History`,
leaking previous turns into rewrite, image analysis, and the final answer.
The RAG branch in `session_knowledge_qa` also added `LOAD_HISTORY`
unconditionally, unlike the pure-chat branch which guarded it with `hasHistory`.
This change:
- Treats `chatManage.MaxRounds <= 0` as an explicit disable in both plugins;
no fallback to global config.
- Makes the RAG pipeline consistent with the pure-chat path by gating
`LOAD_HISTORY` on `hasHistory`.
- Removes the duplicated `Current Time: {{current_time}}` line from
`agent_system_prompt.yaml`. The agent already receives a fresh
`<runtime_context><current_time>` block with each turn from
`observe.buildRuntimeContextBlock`, so the static placeholder was
redundant.
The ReAct agent path (`session_agent_qa`) already checked `MultiTurnEnabled`
directly and is not affected.
Closes #1479
|
||
|
wizardchen
|
31471f5efa |
refactor(knowledgeService): update checkStorageEngineConfigured method for improved clarity and functionality
- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability. - Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling. - Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance. |
||
|
wolfkill
|
6331cce23d | fix: skip empty milvus enabled status groups | ||
|
ochan.kwon
|
744a367f16 |
feat(retriever): add OpenSearch type prep ahead of Phase 3 driver
Lays type-system groundwork for the upcoming OpenSearch k-NN driver (Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate: this PR ships only inert constants, schema extensions, and an unreachable normalizer case. No path activates an OpenSearch VectorStore yet — creation continues to fail with "not a valid engine type" until the activation switch lands in Phase 3 PR 3. Changes: - OpenSearchRetrieverEngineType constant ("opensearch") in internal/types/retriever.go. Not added to validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping yet (gated). - ConnectionConfig.InsecureSkipVerify (bool, default false) in internal/types/vectorstore.go, placed inside the // Common section because it is a cross-driver TLS option. Distinct from the Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify only controls verification of an already-TLS HTTPS connection. AES-GCM Value/Scan round-trips the field as plaintext (it travels alongside the encrypted Password / APIKey but is not sensitive itself). - VectorStoreFieldInfo gains four optional fields: Immutable (bool), Min/Max (*float64), Enum ([]string). All omitempty so existing UI schema entries serialize identically. The fields will drive the UI in Phase 3 PR 3 (read-only on Edit + range/enum constraints). - Six new AuditAction constants under vector_store.* and opensearch.* namespaces. Definitions only — emission lands in Phase 3 PR 3. - EngineAwareNormalizer.Normalize is restructured to group engines by the effective score range observed by the normalizer (not the theoretical raw cosine range): Range [-1, 1] (raw cosine, mapped via (score + 1) / 2): - Milvus (COSINE metric mode). Milvus docs explicitly state the COSINE metric range is [-1, 1]. Range [0, 1] (passthrough — already on the target scale): - Elasticsearch v8 / ElasticFaiss. The driver issues a cosineSimilarity(...) script_score script, and Lucene rejects negative final scores ("Final relevance scores from the script_score query cannot be negative" — ES docs); the effective range observed by the normalizer is therefore [0, 1] for IR-normalized embeddings. ES was previously grouped with Milvus and over-corrected via (score + 1) / 2, which inflated every ES result by 50% in mixed-engine RRF fusion. - OpenSearch. The k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation maps the underlying Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1] before the score reaches us. Source: github.com/opensearch-project/k-NN at src/main/java/org/opensearch/knn/index/SpaceType.java (COSINESIMIL enum, scoreTranslation method). - Weaviate. The driver requests `certainty`, defined by Weaviate as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1]. - Postgres pgvector / SQLite sqlite-vec / Qdrant / TencentVectorDB / Doris. The driver computes (1 - cosine_distance) or normalized inner_product, whose theoretical range is [-1, 1]. The IR-normalized positive-component unit vectors WeKnora targets (BGE / OpenAI text-embedding-3 / Cohere / sentence-transformers) keep the observed range in [0, 1]; negative-cosine embedding models would silently clamp to 0 downstream — explicitly documented as the IR-normalization caveat in the struct godoc. Dead enum references (ElasticFaiss, Infinity) are flagged in the godoc with a pointer to internal/types/vectorstore.go's existing "legacy/experimental, no standalone deployable instance" annotation. Their case labels are kept for switch exhaustiveness. Test coverage: - OpenSearch constant wire value + collision check against existing 10 engines + gated invariant (NOT in validEngineTypes). - ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON field deserializes as false) + round-trip + AES-GCM coexistence with encrypted Password / APIKey. - VectorStoreFieldInfo omitempty preservation + new-field serialization + *float64 pointer distinction (min=0 vs nil). - AuditAction dot-namespace convention enforcement, prefix invariants for vector_store.* and opensearch.*, no wire-string collisions, exact wire-value pins for the six new constants. - EngineAwareNormalizer: - CosineRange retains its (score + 1) / 2 coverage but now only for Milvus. - UnitInterval now covers the full passthrough group (ES / ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant / Infinity / TencentVectorDB / Doris). - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an explicit regression guard for the score-range correction: cos=0.5 maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed). - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift (1.0001) + defensive negative + ±Inf / NaN edges + keyword passthrough + nil-ctx safety. Backward compatibility is preserved at every layer: - All new struct fields are omitempty / pointer-tagged so existing rows and existing wire formats remain unchanged. - Normalizer's new OpenSearch case is unreachable until the driver lands. The ES regrouping changes the post-normalization value for every ES vector search result (a correctness fix, not a feature) — ES vector retrieval is currently the only production path affected. - AuditAction constants emit no audit_log rows in this PR. - engine_type=opensearch VectorStore creation still rejected (gated). |
||
|
ochan.kwon
|
8b7d00b260 |
feat: expose KB ↔ vector store binding in list, editor, and detail UI
Backend ------- ListKnowledgeBases now enriches each row with the resolved vector_store metadata (name / source / engine_type / status) via a new buildKBListResponse helper. Store views are batch-resolved once per request through BatchResolveStoreView so an N-KB list costs one vector-store service call rather than N — closing the N+1 limitation flagged in #1372's known-limitations section. Cross-tenant shared KBs continue to render via SharedStoreDisplay so the owning tenant's store inventory cannot be correlated across rows; the underlying vector_store_id UUID is stripped from those responses. Resolver failures degrade gracefully: bound KBs render as unavailable instead of breaking the list. Test coverage pins the env / bound / shared distinction, the batch-call-count invariant, and the graceful-failure path. Frontend -------- KB editor modal gains a new "Vector Store" section. Create mode shows a dropdown that combines the system default (env store) and the tenant's configured user stores, fetched once at mount via the existing listVectorStores API. Edit mode shows the bound store read-only via a new VectorStoreBadge component with an explicit immutability hint — matching the backend's `<-:create` GORM tag and the service-layer UpdateKnowledgeBaseRequest DTO that already omit the field. KB list cards surface a small engine-type badge for own-tenant bound KBs, and a warning badge when the bound store is unavailable. Env-bound and shared KBs render no badge (visual noise control). KB detail header shows the bound store via the same VectorStoreBadge component; shared KBs fall through to the badge's internal "shared" branch with no name / engine / id rendered. The KB editor's create-time error handler translates the typed ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201) into localized messages and jumps the user back to the VectorStore section so they can pick a different store or fall back to the system default. The KB row type gains five optional fields (vector_store_id / name / engine_type / source / status). i18n: 18 new keys added to en-US, ko-KR, zh-CN; ru-RU receives English placeholders pending translation (consistent with prior PRs in this locale). Part of #993 (Phase 2: Per-KB VectorStore Binding). Phase 2 roadmap item: PR 5 (KB binding UI + list-response enrichment). Depends on #994, #1310, #1372, #1386 (all backend in the Phase 2 series). |
||
|
wizardchen
|
c0e4a1d2f1 |
fix(summary): preserve image caption/OCR text in document summaries
Documents whose only payload is an embedded image (e.g. a docx with a single picture) intermittently produced the refusal line "No textual content was extractable from this document." even though the vision model had successfully extracted a caption. Three coordinated fixes: - Clarify the summary prompt that text inside `<image_caption>` and `<image_ocr>` is first-class extracted content, not an image reference, so the model only triggers the empty-content branch when the body is genuinely textless. - For image-dominated documents (real text < 200 runes after stripping image markup) include OCR alongside captions so screenshots and scanned figures contribute their actual content; text-heavy documents continue to use caption-only enrichment to avoid OCR noise from incidental figures. - Add `EnrichContentCaptionAndOCR` which embeds caption + OCR text inline next to the original Markdown image link, deliberately omitting the `<image url=...>` and `<image_original>` wrapper blocks. Those wrappers carry only opaque export hashes that consume tokens and have been observed to retrigger the LLM's "image reference with no extracted text" heuristic. |
||
|
ChenRussell
|
d3832edea4 | fix(repository): qualify tenant_id with table name to resolve ambiguous column reference | ||
|
MarionMa
|
0a9a920460 | perf(repository): exclude embedding field from Elasticsearch search response to reduce size | ||
|
Thinh Tran
|
e1ce4270d0 | fix(repository): correct query syntax to get correctly built-in models | ||
|
wolfkill
|
bb95488ac8 | fix(knowledge): complete indexed documents immediately | ||
|
ochan.kwon
|
fd3d9f547c |
feat(search): fan-out KB retrieval across bound vector stores
Multi-KB hybrid search now groups KBs by their bound VectorStore (partition key (storeID, owner_tenant_id)), retrieves in parallel via errgroup with a SetLimit(4) cap and a per-group timeout (MULTI_STORE_RETRIEVE_TIMEOUT_SEC, default 30s), and merges results. When the collected results span more than one engine type, an EngineAwareNormalizer rescales vector scores to [0, 1]; keyword (BM25) scores pass through to the existing RRF fusion. Single-group calls take the fast path with zero fan-out overhead, preserving today's behavior for deployments where every KB has vector_store_id = NULL. Embedding-model consistency is now enforced explicitly via ResolveEmbeddingModelKeys. Multi-KB searches across KBs whose resolved model identities differ return BadRequest instead of silently producing incomparable scores. Cross-tenant Organization-shared KBs are preserved by partitioning on KB.TenantID so the factory's ownership lookup runs against the source tenant. Foreign-tenant KB UUIDs injected via the request body are rejected via kbShareService.HasTenantKBPermission (Plan 3 of #1303, 3-D capped) before any retrieval; rejected scopes surface as 404 to avoid leaking foreign KB existence. Service-layer typed AppErrors (ErrVectorStoreBindingInvalid 2200 / ErrVectorStoreUnavailable 2201) are mapped from PR2 sentinel hierarchy and preserved end-to-end: the iterative FAQ path returns them rather than swallowing, and the HybridSearch handler routes typed AppErrors to the client unchanged instead of downgrading to 500. Part of #993 (Phase 2: Per-KB VectorStore Binding). Phase 2 roadmap item: PR 4 (Multi-store fan-out search). Depends on #994, #1310, #1372. |
||
|
Miles Lai
|
e62c0563aa |
doris: add configurable compatibility modes and guard mode switches
Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.
Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.
Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:
* auto (default/recommended): probe Doris server on first use to detect available vector
functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
(older builds lacking inner_product_approximate)
* legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
builds without inner_product_approximate support)
* inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
(for modern Doris 4.0+ and current SelectDB with normalized embeddings)
Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level
Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
|
||
|
wizardchen
|
7bb38f81b6 |
feat(session): persist last request state for UI restoration
Sessions now record the input-bar state used for the most recent QA request (agent, model, KB scope, web search). The chat UI hydrates those settings on session reopen so users see the same configuration they used last time, instead of the global default. The state is stored in the existing sessions.agent_config JSONB column to avoid a new migration. Frontend snapshots the user's global defaults on session enter and restores them on session leave, so opening an old session does not pollute new-chat defaults. |
||
|
wizardchen
|
0cdb922d84 |
feat(tenant): remember user's last-active workspace across logins
Until now a fresh login (new device, expired refresh token, cleared
browser) always dropped the user into their home tenant, even if they
spend most of their time in a peer workspace. The session-local
X-Tenant-ID override in localStorage gave a "same browser sticky"
effect, but never crossed devices or new sessions.
This adds a small server-side preference, `users.preferences.
last_active_tenant_id`, persisted in the existing jsonb column (no
new migration), and threads it through:
* Backend
* `UserPreferences` gains `LastActiveTenantID *uint64` with sentinel
semantics (`*0` from the PATCH endpoint = clear preference).
* `resolveLoginTenantID` validates the stored id (tenant still
exists + active membership, or CanAccessAllTenants) and falls
back to home on any failure, best-effort clearing the stale
preference so subsequent logins don't pay for it again.
* `Login` and `LoginWithOIDC` resolve once and use the result for
both the JWT `tenant_id` claim and the returned `active_tenant`,
keeping the two in sync. `RefreshToken` rides through
`GenerateTokens` so refresh rotations also land the user back in
their preferred tenant instead of bouncing to home.
* `UpdateUserPreferences` learns to merge the new key.
* `PUT /auth/me/preferences` accepts the new field.
* Frontend
* `Login.vue` now writes the user's HOME tenant id into
`user.tenant_id` (matching the field's documented semantics) and
expresses any active-vs-home divergence via `setSelectedTenant`,
so `useHomeTenant` and the "current"/"home" badges stay correct
after the backend honours a remembered preference. `App.vue`'s
OIDC sync does the same reconciliation.
* `TenantSelector`, `UserMenu` and the post-tenant-create handlers
fire `persistLastActiveTenantPreference` after every successful
user-initiated switch (switching to home sends `0` to clear).
The call is raced against the existing reload-grace window so
most writes finish before the page tears down; lost writes are
recoverable on the next switch.
No new UI. Users will simply notice that, after re-logging in on
another device, they land back in the workspace they were last
using rather than always in their home tenant.
Note: `make docs` is unrelated-broken on `main` (audit_log.go
references `errors.AppError` which swag can't resolve), so the
Swagger artifacts under docs/ are intentionally not regenerated
in this PR. The handler code is the source of truth.
|
||
|
Li Xianggang
|
b3dbf86bf5 | feat(obs): 支持华为云obs存储 | ||
|
wizardchen
|
bbcedbee63 |
feat(tests): enhance vector store tests with fake Elasticsearch server and add user pinning methods to knowledge base repository
Introduced a new helper function to create a fake Elasticsearch server for testing, improving the reliability of connection probes in vector store tests. Additionally, added methods for setting and listing user-specific knowledge base pins in the knowledge base repository, aligning with recent user pinning features. Refs: #1303 |
||
|
wizardchen
|
7528175a74 |
feat(knowledge-base): implement per-user pinning for knowledge bases
Refactored the pinning mechanism for knowledge bases to be user-specific, allowing any user with read access to pin knowledge bases independently. This change includes the introduction of a new `user_kb_pins` table to manage pin states, replacing the previous tenant-wide pinning system. Updated relevant API endpoints and service logic to support this new functionality, ensuring that the UI reflects the user's personal pin status. Additionally, adjusted the knowledge base listing to sort by user-specific pin status, enhancing the user experience by prioritizing pinned items. Refs: #1303 |