Commit Graph

1102 Commits

Author SHA1 Message Date
Yeongpil Yoon c6fc315042 fix(chat): preserve provider tool-call metadata 2026-06-09 20:55:20 +08:00
wizardchen 2aadd61fcc feat: implement reparse functionality with process configuration support
- Enhanced the `reparseKnowledge` function to accept optional process configuration overrides, allowing users to adjust parsing settings during reparse operations.
- Updated the frontend components to integrate reparse capabilities, including the `UploadConfirmDialog` and `KnowledgeBase` views, providing a user-friendly interface for confirming reparse actions.
- Added new localization strings for reparse confirmation and process configuration details in multiple languages.
- Modified the backend API to handle reparse requests with optional configuration, ensuring proper validation and persistence of overrides.
- Introduced new types and interfaces to manage reparse-specific data structures in the store, improving type safety and clarity in the codebase.
2026-06-09 20:38:24 +08:00
wizardchen d8a26f7c23 refactor: remove enable_multimodal and adjust parent-child configuration
- Removed the `enable_multimodal` field from the `ChunkingConfigOverride` interface and related components to streamline configuration.
- Updated the logic in `UploadConfirmDialog.vue` to default `enableParentChild` to false when initializing from knowledge base info.
- Modified the `mergeChunkingConfig` function to ensure that `EnableParentChild` is authoritative, allowing explicit control over its state.
- Added a new test case in `knowledge_process_config_test.go` to verify the behavior of parent-child toggles in process configuration resolution.
2026-06-09 20:38:24 +08:00
wizardchen 76dc23226d fix: correct graph_enabled logic in process configuration
- Updated the logic for `graph_enabled` in `UploadConfirmDialog.vue` to require both `nodeExtractConfig.enabled` and `graphEnabled` to be true.
- Added a new test case in `knowledge_process_config_test.go` to ensure that `GraphEnabled` is false when `ExtractConfig.Enabled` is false.
- Adjusted the `ResolveProcessConfig` function to reflect the updated dependency between graph and extract configurations, ensuring proper behavior in the process configuration resolution.
2026-06-09 20:38:24 +08:00
wizardchen ed9e780c8c feat(client): enhance knowledge file upload with process configuration support
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides.
- Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations.
- Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads.
- Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process.
- Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic.
- Enhanced user experience by providing detailed configuration options during the upload process.
2026-06-09 20:38:24 +08:00
wizardchen 1778c5ac7b refactor(tracing): remove Jaeger integration and related components
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
2026-06-08 21:01:04 +08:00
langcaiye 521a663b0c fix: support custom jieba dictionary directory 2026-06-08 20:10:37 +08:00
langcaiye 99f35f8dea fix(duckdb): avoid extension install during startup 2026-06-08 15:29:42 +08:00
wizardchen 28de82a92e refactor(chat): improve model name sanitization and path validation
- Updated the sanitization logic in `newStreamPacketDumper` to exclude the '.' character, enhancing security against path traversal vulnerabilities.
- Added checks to ensure the cleaned file path remains within the specified directory, preventing unauthorized access to the filesystem.
- Improved overall robustness of file handling by enforcing stricter path validation rules.
2026-06-05 19:35:02 +08:00
wizardchen b820e690f0 fix(auth): allow anonymous access to share-link endpoints (#1617)
The global Auth middleware allowlist (noAuthAPI) was missing the
share-link routes, so an unauthenticated invitee opening /register?token=
hit a 401 on /auth/invitations/lookup and the frontend bounced them to
/login instead of showing the register page. Add both share-link
endpoints to the allowlist and stop the request interceptor from treating
their 401s as a session-expiry redirect.

Refs: https://github.com/Tencent/WeKnora/issues/1617
2026-06-05 18:51:32 +08:00
wizardchen 946ff5557a feat: release v0.6.1 with new document parsing trace timeline, OpenSearch vector store driver, YAML-based built-in model configuration, and system admin settings. Includes UI redesigns, new-user onboarding guide, and various improvements and bug fixes. See CHANGELOG.md for details. 2026-06-05 16:36:13 +08:00
langcaiye 9e898ae72c fix(schema): expand knowledge source length 2026-06-05 15:12:25 +08:00
ruihao.li a12b6ff61b feat: support Milvus database selection for vector stores 2026-06-05 15:11:37 +08:00
ochan.kwon db7baad84c fix: reject cross-store reuse_vectors knowledge move
Moving knowledge with mode=reuse_vectors copies vector index entries
directly between knowledge bases, which only works when both KBs are
bound to the same vector store (CopyIndices is routed through the source
store). Nothing rejected a cross-store reuse_vectors move, so it would
copy target-KB rows into the source store and then delete the source
indices — corrupting vector data. The move endpoint validated only the
embedding model, and ListMoveTargets surfaces same-model KBs regardless
of their store binding, so a cross-store target is reachable.

Reject reuse_vectors moves whose source and target KBs are bound to
different vector stores:

- MoveKnowledge handler: synchronous 400 before enqueueing, with a
  message pointing the caller at reparse mode.
- moveOneKnowledge: re-check before marking the knowledge as processing,
  so a rejected move leaves it untouched rather than stranded.
- moveKnowledgeReuseVectors: a top-of-function guard as defense in depth
  for any directly enqueued task, satisfying the pre-existing contract
  noted in the comment ("callers must reject cross-store moves").

reparse mode is unaffected: it re-indexes into the target store from
scratch and is the supported way to move across stores.
2026-06-05 15:00:41 +08:00
toy0116 e76210a695 fix(handler): /knowledge/search · accept ?query= and reject empty keyword
Silent-failure footgun in SearchKnowledge handler:

  keyword := c.Query("keyword")   // only "keyword" recognized
  // empty keyword → service falls back to default listing (created_at DESC)
  // → caller sees "all queries return the same N newest knowledges"

This is exactly what happened to a downstream agent (Support GPT) — it was
sending ?query=... (the parameter name every MCP / semantic-search tool
uses) and getting a constant 5-card listing back regardless of input. The
caller diagnosed it as "WeKnora embedding has degraded · vector channel
returns constant rankings", but actually the parameter was never read, the
search service got keyword="", and the SQL came back as a plain
list-by-created_at.

The handler stays a filename/title LIKE search (its real purpose is the
document-management UI). Two tweaks:

1. Accept ?query= as an alias for ?keyword= · this is what MCP / agent
   integrations actually send. Cheap compatibility win, zero downside.
2. Empty keyword/query → 400 with a clear hint. Refusing the request is
   strictly better than silently returning a default listing that looks
   like "search results" but isn't query-driven.

Separately worth noting: callers who actually want semantic recall should
use POST /api/v1/knowledge-bases/{kb_id}/hybrid-search · which is the
real vector + BM25 endpoint. This handler was never the right tool for
RAG queries · the silent failure just made it look like one.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-05 14:58:20 +08:00
wizardchen e2a41ecf0a feat(thinkingControl): implement thinking mode configuration in model editor
- Added a new thinking mode control field in the ModelEditorDialog, allowing users to specify how the agent's thinking mode is formatted for API requests.
- Introduced localization strings for thinking mode labels and descriptions in English, Korean, Russian, and Chinese, enhancing user experience across multiple languages.
- Updated the model form data structure to include thinking control settings, ensuring proper handling during model edits and saves.
- Implemented utility functions for resolving and defaulting thinking control values based on provider and model, improving backend integration.
- Added tests for the new thinking control functionality to ensure reliability and correctness in handling user configurations.
2026-06-05 14:56:48 +08:00
wizardchen 063809896a refactor(chat): improve handling of <think> tags in chat messages
- Updated logic to manage <think> and </think> tags in chat message content, ensuring accurate representation of thinking states.
- Introduced a new function to handle the closing of thinking events, enhancing clarity and reducing redundancy in event emissions.
- Adjusted event handling for reasoning content to support historical message reconstruction, improving user experience during chat interactions.
2026-06-05 14:56:48 +08:00
wizardchen 4bb8906520 feat(faq): enhance FAQ handling and localization support
- Updated the agent system prompt to clarify mandatory deep read requirements for FAQ entries, ensuring accurate retrieval and response generation.
- Added new localization strings for FAQ-related labels in English and Chinese, improving user experience and consistency across the application.
- Refactored the document and grep results components to support distinct FAQ identification, enhancing clarity in the output.
- Introduced new tests for FAQ metadata handling, ensuring robust functionality and correctness in the FAQ processing logic.
2026-06-04 21:43:16 +08:00
langcaiye ba8c96d83a fix(datasource): fail sync when all fetched items fail 2026-06-04 17:39:55 +08:00
lyingbug 4369d156e1 Revert "fix: preserve agent system prompt for greetings"
This reverts commit b22303cec8.
2026-06-04 17:39:38 +08:00
wolfkill b22303cec8 fix: preserve agent system prompt for greetings 2026-06-04 17:39:01 +08:00
wizardchen dda16436d0 feat(lkeap): integrate Tencent Cloud LKEAP Rerank support
- Added support for Tencent Cloud LKEAP Rerank, including new configuration options for SecretId and SecretKey.
- Updated frontend components to handle LKEAP-specific fields and enhance user experience with appropriate labels and placeholders.
- Enhanced backend logic to manage LKEAP credentials and integrate with the new rerank API.
- Improved localization for LKEAP-related terms across multiple languages, ensuring clarity and consistency in user-facing messages.
- Introduced tests for LKEAP Reranker functionality to ensure reliability and correctness in handling requests.
2026-06-04 17:22:18 +08:00
wizardchen 12ea490d89 fix(im): inject synthetic identity so IM channels can use shared KBs
IM callback routes bypass the auth middleware because IM platforms verify
their own signatures, so the downstream QA pipeline never received a
UserID/TenantRole. The Organization-shared knowledge base resolution gates
on a non-empty UserID, so IM-channel questions silently skipped shared KBs
while the Web path (which sets UserID via JWT auth) worked.

Inject a synthetic "system-<tenantID>" identity (mirroring the API-key path,
recognised by types.IsSyntheticUserID) plus a Viewer role when building the
IM session context. Viewer is the least privilege sufficient to retrieve
shared KBs.

Closes #1600
2026-06-04 16:43:54 +08:00
wizardchen 035093dc35 feat(configuration): update asynq concurrency settings and enhance FAQ handling
- Increased the default worker pool size for the asynq server from 16 to 32 to improve performance during concurrent uploads.
- Updated the .env.example and docker-compose.yml files to reflect the new default concurrency setting.
- Enhanced the handling of FAQ entries in the tool results, ensuring distinct identification through standard questions, improving user experience and clarity in the output.
- Refactored related components to utilize the new FAQ attributes, ensuring consistent display and interaction across the application.
2026-06-04 15:48:04 +08:00
wizardchen d38a6efd87 refactor(agent): remove final_answer tool references and update related logic
- Eliminated all instances of the final_answer tool from the agent's configuration and logic, transitioning to a model where the agent concludes responses with plain text instead.
- Updated event handling and response analysis to reflect this change, ensuring that the agent's output is now directly delivered as text without relying on a dedicated final answer tool.
- Adjusted localization strings across multiple languages to remove references to final_answer, enhancing clarity and consistency in user-facing messages.
- Improved the overall structure of the agent's response handling to streamline the process and reduce potential confusion regarding tool usage.

refactor(agent): streamline answer handling and preamble logic

- Updated the agent's response handling to treat any text streamed before a tool call as a preamble, ensuring it is not included in the final answer.
- Removed the superseded marker logic from the event emission process, simplifying the flow of answer content and enhancing clarity in the UI.
- Adjusted the event handling to ensure that preamble segments are correctly retracted and relocated into the steps tree, improving the overall user experience.
- Enhanced the agent's internal logic to better manage answer segments and their states during streaming, leading to a more coherent response structure.
2026-06-04 15:48:04 +08:00
wizardchen b9dde2abe8 feat(agent): improve content filtering in streaming event handling
- Added logic to suppress whitespace-only content emitted before the actual answer starts, addressing issues with spurious empty events interleaved with tool call events.
- Enhanced the event emission process to ensure that only meaningful content is routed to the final answer area, improving the clarity and relevance of streamed responses.
2026-06-04 15:48:04 +08:00
wizardchen 876fd0276f feat(agent): enhance event routing for reasoning and answers in streaming
- Introduced new tests to ensure correct routing of reasoning and answer content during streaming, addressing UX issues where answers appeared under the Thinking phase before transitioning.
- Implemented a ThinkStreamSplitter to separate inline reasoning from answer text, allowing for real-time updates to the UI without duplication.
- Updated the event emission logic to prevent re-emission of answers that were already streamed live, ensuring a seamless user experience.
- Enhanced the ChatResponse struct to include flags for streamed answers, improving the handling of final answer events.
- Added comprehensive unit tests for the ThinkStreamSplitter to validate its functionality across various scenarios.
2026-06-04 15:48:04 +08:00
wizardchen 710d759b8e feat(ui): update SVG assets and enhance localization for knowledge framework
- Redesigned SVG assets for screenshots to improve visual consistency and clarity, including updates to gradients, titles, and layout.
- Added a new screenshot asset to showcase the Wiki Knowledge Base feature.
- Enhanced localization strings across multiple languages to reflect updated terminology and descriptions for the knowledge framework, including agentic reasoning and hybrid search strategies.
- Improved the user interface in the login view to include new tags for agent and wiki features, enhancing user awareness of available functionalities.
2026-06-04 15:48:04 +08:00
wizardchen 226a37330c feat(system): add server startup time and uptime tracking
- Introduced `started_at` and `uptime_seconds` fields to the `SystemInfo` structure for tracking server boot time and uptime.
- Updated the `GetSystemInfo` handler to populate these new fields based on server runtime.
- Enhanced the frontend to display server startup time and uptime in the system information view, improving visibility for users.
- Added localization support for new labels and descriptions related to server uptime and startup time in multiple languages.
2026-06-04 15:48:04 +08:00
wizardchen 0c45cc07df feat(knowledge): refactor retrieval parameters for multi-KB search
- Updated the `resolveStoreGroups` and `buildRetrievalParams` functions to handle multiple knowledge bases more effectively.
- Changed the structure of `buckets` to store pointers to `KnowledgeBase` objects instead of just IDs, allowing for richer data handling.
- Enhanced the retrieval parameter construction to differentiate between FAQ and document-type knowledge bases, ensuring correct routing for vector and keyword retrieval.
- Improved comments for clarity on the logic behind retrieval parameter construction based on knowledge base types.
2026-06-04 15:48:04 +08:00
wizardchen aa4513d4f8 feat(observe): enhance runtime context with communication and final answer instructions
- Added detailed communication and final answer instructions to the runtime context block, guiding the model on user-facing text and tool usage.
- Updated the XML structure to reflect these changes, ensuring clarity on metadata and instructions for each turn.
- Improved the overall safety and clarity of the model's output by preventing the exposure of internal tool names and IDs.
2026-06-04 15:48:04 +08:00
wizardchen 1c9c17a7cf feat(tests): add unit tests for Langfuse wrapper functions
- Introduced a new test file for the Langfuse wrapper, implementing tests for the `buildLangfuseGenerationOutput` and `buildLangfuseMessages` functions.
- Validated the output structure for various scenarios, ensuring correct handling of content, tool calls, finish reasons, and reasoning content.
- Enhanced test coverage for the Langfuse functionality, improving reliability and maintainability of the codebase.
2026-06-04 15:48:04 +08:00
wizardchen 362bc7f6e5 feat(session): enhance stop watcher with timeout for improved resource management
- Introduced a constant `stopWatcherMaxDuration` to limit the lifetime of the stop watcher, preventing potential resource leaks in pathological stream scenarios.
- Updated the `startStopWatcher` function to use a context with a timeout, ensuring the watcher self-terminates after the specified duration.
- Refactored the handling of stop events to improve clarity and maintainability, ensuring the watcher exits appropriately on terminal stream events or user requests.
- Removed the `stopWatchCancel` from the `sseStreamContext` as the watcher now manages its own lifecycle more effectively.
2026-06-04 15:48:04 +08:00
wizardchen b6337cd970 fix(session): handle user-requested stop events gracefully in QA execution
- Updated the QA execution logic to differentiate between user-requested cancellations and actual errors, preventing unnecessary error events from being emitted when a stop is requested.
- Enhanced logging to provide clear feedback when a QA session is cancelled by the user, improving the user experience and reducing confusion.
- Added context cancellation checks to ensure proper handling of stopped generation scenarios, maintaining the integrity of the session flow.
2026-06-04 15:48:04 +08:00
gordonlu b1f240f1b5 fix(event): add panic recovery to async goroutines
Add defer recover() to both Emit (async mode) and EmitAndWait
goroutines to prevent handler panics from crashing the process.

Fixes #1580
2026-06-04 15:37:08 +08:00
langcaiye c8e9adbaa1 fix(datasource): align Feishu connector capabilities 2026-06-04 15:33:03 +08:00
langcaiye 822aaf5cbc fix: preserve feishu wiki node parents 2026-06-04 15:32:06 +08:00
langcaiye 3c6a8b718e feat: support feishu wiki node resources 2026-06-04 15:32:06 +08:00
ochan.kwon 7b2bc8a6b8 fix: validate vector store connection addresses against SSRF policy
Vector store create and raw connection-test accepted user-supplied
connection addresses without SSRF validation, while every other
admin-entered endpoint (e.g. model BaseURL) already runs
ValidateURLForSSRF. An admin could point an engine address at an
internal host and use the connection probe as an SSRF oracle.

Add validateConnectionAddrSSRF, applied at the two user-input
boundaries only:

- CreateStore: validated before any network dial.
- TestStoreRaw: a new TestRawConnection wrapper runs an engine-type
  allowlist, required-field validation, and SSRF validation before
  delegating to TestConnection. TestConnection stays validation-free
  for trusted callers (env stores, stored configs already validated
  at create time).

Validation covers every address-bearing field a driver dials
(Addr, qdrant Host+Port, weaviate Host and GrpcAddress) and
fails closed for unmapped engine types. The Elasticsearch probe no
longer follows redirects, closing a 302-to-internal bypass.

docker-compose whitelists the bundled vector-store service hostnames
(qdrant, milvus, weaviate, doris-fe) via SSRF_WHITELIST_EXTRA so the
out-of-box experience is preserved; postgres is intentionally not
whitelisted. .env.example and the OpenSearch dev guide document how
to whitelist external or local-dev addresses.
2026-06-04 15:26:56 +08:00
ochan.kwon e9980c6011 fix: deep-copy stored files and images when cloning a knowledge base
Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.

Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.

Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.

Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
2026-06-03 14:45:59 +08:00
wizardchen bbd3f6324a refactor(parser): reorganize Markdown parser and enhance gRPC document reading
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
2026-06-03 12:29:13 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
octo-patch f22995f835 feat: add MiniMax-M3 to provider model list
- Add MiniMax-M3 to the MiniMax provider description across i18n locales
- Remove deprecated MiniMax-M2.5 reference
- Retain MiniMax-M2.7 and MiniMax-M2.7-highspeed as supported models

MiniMax-M3 is the latest flagship model from MiniMax with extended context
and stronger reasoning capabilities. Users can now see it in the provider
selection UI when configuring a new Minimax model.
2026-06-02 21:38:30 +08:00
Yeongpil Yoon 8f8a276120 fix(embedding): support native Gemini embeddings 2026-06-02 21:37:58 +08:00
wizardchen 482686d17e feat(chat): implement local image resolver for multimodal chat
- Added a LocalImageResolver function to resolve local storage URLs to their byte data using the tenant's storage configuration.
- Integrated the resolver into the container setup to support reading local images in multimodal chat scenarios.
- Enhanced the handling of local storage paths to correctly parse tenant IDs and retrieve files based on the configured storage settings.
2026-06-01 20:54:37 +08:00
wizardchen f92302cc64 fix(knowledge): stitch chunks by text overlap, not by position
Reconstructing full document content from chunks relied on position math
(offset = content length - (EndAt - lastEndAt)) that assumes
len([]rune(Content)) == EndAt - StartAt. Two upstream behaviors break that
invariant and caused the merged "全文" view to drop or duplicate text:

- The parent-child chunker prepends a synthetic table header to split
  tables; that header is zero-width in position space (start == end), so
  Content is longer than EndAt - StartAt.
- Chunk Content can retain HTML entities (&#34;, &gt; ...), inflating the
  rune count relative to the source span.

Replace position-based trimming with text-overlap matching: find where the
accumulated text's suffix reappears at the head of the next chunk and join
there. Positions are only used to size the search window. This naturally
skips the prepended header and is immune to entity length drift.

Extract the logic into searchutil.AppendWithOverlap / MergeTextChunks and
reuse it across the three backend stitching sites (wiki_ingest
reconstructContent, graph mergeChunkContents, chat_pipeline
mergeOverlappingChunks). Mirror the same algorithm in the frontend
doc-content merge. Also fix the markdown hr/heading styles (thin solid
divider, primary heading color).
2026-05-30 17:53:47 +08:00
jackson.jia c8645843c3 fix(knowledge): wrap locale code with LanguageLocaleName in wiki ingest prompts
The wiki ingest pipeline was passing raw locale codes (e.g. "zh-CN",
"en-US") directly into LLM prompts where a human-readable language name
is expected. The model would see "zh-CN" instead of "Chinese (Simplified)"
in its instructions, which could degrade output quality.

Two call sites were affected:

- SlugUpdate.Language (ProcessWikiIngest, retract path): consumed by the
  reduce phase to tell the page editor model what language to write in.
- lang variable (mapOneDocument): consumed by entity/concept extraction,
  chunk citation classification, and index page rebuild — all LLM prompts.

Wrap both with types.LanguageLocaleName() which maps locale codes to
descriptive names (e.g. "zh-CN" → "Chinese (Simplified)"). Unknown
locales fall through unchanged so no information is lost.
2026-05-30 16:56:32 +08:00
wizardchen cea6ef0ce3 test(knowledge): add comprehensive tests for finalizing subtask behavior
Introduced a new test file for the knowledge repository, implementing various scenarios to ensure the reliability of the finalizing subtask counter. Key tests include:

- Concurrent promotion of subtasks to verify atomic behavior.
- Handling of partial decrements to ensure the row remains in "finalizing" state.
- Safety checks to prevent underflow of the pending subtask count.
- Regression tests to confirm that updates do not overwrite the pending counter.

These tests address previous issues with the finalizing process and enhance overall stability.
2026-05-30 01:00:15 +08:00
wizardchen 84bd95275d fix(knowledge): make finalizing subtask counter reliable + async question fan-out
The finalizing subtask counter (pending_subtasks_count) could get stuck at a
non-zero value, leaving knowledge permanently in "finalizing" and never
promoted to "completed". Root cause and related hardening:

- UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the
  omit list, so any concurrent enrichment subtask that loaded the row, did slow
  work (LLM call), then saved an unrelated field wrote back the STALE counter,
  clobbering decrements other subtasks performed in between. Add
  PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers
  (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it.
- FinalizeSubtask no longer gates the finalizing->completed promote on a
  separate re-read of the counter. Every caller unconditionally runs the
  guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single
  authoritative atomic check, so a racy/stale read can't strand the row.
- Decrements run on a context detached from the caller's cancellation
  (context.WithoutCancel + timeout) so graceful shutdown / preemption can't
  silently skip a decrement.
- Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph)
  and release any shortfall so un-enqueued planned slots (e.g. graph with
  NEO4J off) don't strand the row.
- Reparse paths reset pending_subtasks_count via an explicit column write since
  full-row Save now omits it.

Question generation is also moved off the synchronous single-task path onto a
batched async fan-out (windows of text chunks), each batch independently
queued / retried / traced, grouped under a postprocess.question span.
2026-05-30 01:00:15 +08:00