Commit Graph

1124 Commits

Author SHA1 Message Date
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
adawang ec66a3dac7 perf(common): compile LLM-JSON code-fence regexp once
ParseLLMJsonResponse recompiled its ```json fence-extraction regexp on
every call that fell through the direct-unmarshal fast path — i.e. every
fenced LLM response on the graph-extraction path. Hoist it to a
package-level var.

Adds a test (none existed) covering direct, fenced, no-lang and
prose-wrapped inputs, plus a benchmark.

Benchmark, fenced payload (-benchmem):
  before: 3205 ns/op, 5712 B/op, 54 allocs/op
  after:  1315 ns/op,  693 B/op, 15 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:52:00 +08:00
adawang 8f4b3333f2 perf(sandbox): compile command-substitution regexps once
hasCommandSubstitution recompiled its three regexps with regexp.MustCompile
on every call, and ValidateArgs calls it once per argument — so validating
an N-arg command compiled 3N patterns. Hoist them to a package-level slice
compiled once at init.

Behavior is unchanged (same patterns); covered by the existing
TestScriptValidator_ValidateArgs / ValidateAll tests.

Benchmark, ValidateArgs with 8 args (-benchmem):
  before: 16374 ns/op, 40142 B/op, 505 allocs/op
  after:   2092 ns/op,    32 B/op,   1 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:51:32 +08:00
adawang 1c735ebb9e feat(wiki): front-load static rules in chunk-citation prompt for prefix cache
WikiChunkCitationPrompt placed the per-batch <chunks> block before the
static rules, output schema and the per-document-stable <candidate_slugs>.
Within one document only ChunksXML changes between batches, so the stable
content trailing the chunks could not form a long common prefix and
providers re-billed the static rules on every citation batch (issue #1687
measured ~6.4% prefix-cache hit rate on wiki synthesis).

Reorder to [rules | output schema | candidate_slugs | chunks] so every
batch after the first shares the long [rules | candidate_slugs] prefix.
No instruction text was dropped; only block order and the inline
above/below directional references changed. Verified the cacheable prefix
grows from 281 to 2366 bytes and is byte-identical across batches.

Addresses #1687

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:58:12 +08:00
adawang 8d3030d339 fix(chunker): carry deep sub-heading context into large-section chunks
When the dominant heading level repeats and becomes the split backbone,
a long section is sub-split via SplitText and every sub-chunk inherited
the section-level breadcrumb. Deep `###`/`####` headings that define a
sub-chunk's meaning were dropped from Chunk.ContextHeader, so queries
about content nested far below a shallow heading failed to retrieve it.

Track the heading hierarchy across the section and assign each sub-chunk
the breadcrumb in effect at its start offset, so the deepest active
heading reaches the breadcrumb. Adds a regression test.

Fixes #1674

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:56:57 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen efe8b6f41d feat(agent): add tool result persistence trimming for large outputs
Introduce display_type-based output stripping for SSE/DB storage and
add knowledge_chunks_list display type for list_knowledge_chunks.
2026-06-16 01:09:23 +08:00
wizardchen 16f2a46465 feat(agent): rerank all knowledge search hits with threshold filtering
Remove FAQ bypass in agent rerank, apply composite model scores, and
filter low-confidence results using configurable rerank_threshold.
2026-06-16 01:09:23 +08:00
wizardchen de525a4f02 feat(observability): enhance Langfuse retrieve and rerank tracing
Add shared retrieval observation helpers, expand rerank generation output
with previews and score stats, and enrich retrieve/rerank pipeline spans.
2026-06-16 01:09:23 +08:00
wizardchen 9e916839ef feat(backend): add RAG pipeline progress events
Emit query-understanding and knowledge-search progress over SSE, emit
references before stream completion, and wire progress into session QA.
2026-06-16 01:09:07 +08:00
wizardchen 2906c6656e feat(backend): improve parent-child chunk merge and imageinfo matching
Scope image info to matched text children during parent resolution and
add range/URL filtering utilities with unit tests.
2026-06-16 01:09:07 +08:00
wizardchen 53e0333a53 Add embedding dimension override support 2026-06-15 19:18:19 +08:00
MaoMeng cf0a64b88e fix:dimensions not sent in embedding API requests (#1654)
fix:dimensions not sent in embedding API requests
2026-06-15 17:32:23 +08:00
FFFFFFpy a997441392 fix(wiki): mask image URLs before LLM to prevent UUID/path corruption
Wiki summary/entity/concept/page-modify prompts instructed the LLM to
reproduce ![caption](url) verbatim, so high-entropy image URLs passed
through generation and got corrupted (dropped UUID segments, duplicated
or mangled path prefixes, even CJK chars inserted into the bucket name),
producing object keys that don't exist -> 404.

Mask all image URLs to opaque placeholders at the generateWithTemplate
chokepoint and deterministically rehydrate after generation; drop
corrupted/hallucinated placeholders so broken links never reach storage.
2026-06-15 17:27:56 +08:00
wizardchen ed9802ff2b feat(embed): add backend embed channels with auth and rate limiting (#1657)
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
2026-06-12 23:10:39 +08:00
langcaiye a0786b6fe7 fix(wiki): clear stale citation chunks on deletion 2026-06-12 23:08:20 +08:00
langcaiye 903b014992 fix(tcvdb): keep default replica number at one 2026-06-12 22:54:44 +08:00
langcaiye d07a62af06 fix(tcvdb): improve collection compatibility 2026-06-12 22:54:44 +08:00
kshitijshresth 1e98739dbf Validate sync log limit parameter 2026-06-12 22:52:25 +08:00
nullkey 5851c64ba1 feat(im): IM session improvements — titles, in-flight recovery, classification
Three related improvements to the IM-session experience:

- Auto-generated titles: a new IM session starts untitled and gets a
  content-based title from its first message via the same GenerateTitleAsync
  flow web sessions use, instead of an identity title like "user · dm". Falls
  back to the identity title when the first message has no text. All platforms.

- In-flight reply recovery (web chat): opening a session whose reply is still
  being generated on an IM channel made the web client attach via
  continue-stream and fail (the IM answer never streams through this server),
  flashing a "stream failed" toast and forcing a manual refresh. Treat a failed
  attach to an IM reply (channel === 'im') as recoverable — poll until it
  completes, then reload the thread; on timeout surface the standard error.
  Web/api replies keep the original behaviour, untouched.

- Source-filter classification: the sidebar filter treats "web" as sessions
  with no IM mapping, but the list join excluded soft-deleted mappings, so
  /clear'd IM sessions leaked into the user's own web chats. Include soft-deleted
  mappings so a session ever bound to an IM channel stays under its platform.
  Adds a repository regression test.
2026-06-12 17:01:28 +08:00
wizardchen aeb02b0345 refactor(get_document_info): update schema definition and input parameters
- Removed dependency on utils for schema generation, replacing it with a hardcoded JSON schema for GetDocumentInfoInput.
- Updated the input struct to use omitempty for JSON fields, clarifying that either knowledge_ids or faq_ids may be provided, but both are optional.
- Enhanced documentation for GetDocumentInfoInput to specify the usage of knowledge_ids and faq_ids.
2026-06-10 12:21:02 +08:00
Yeongpil Yoon c6fc315042 fix(chat): preserve provider tool-call metadata 2026-06-09 20:55:20 +08:00
wizardchen 2aadd61fcc feat: implement reparse functionality with process configuration support
- Enhanced the `reparseKnowledge` function to accept optional process configuration overrides, allowing users to adjust parsing settings during reparse operations.
- Updated the frontend components to integrate reparse capabilities, including the `UploadConfirmDialog` and `KnowledgeBase` views, providing a user-friendly interface for confirming reparse actions.
- Added new localization strings for reparse confirmation and process configuration details in multiple languages.
- Modified the backend API to handle reparse requests with optional configuration, ensuring proper validation and persistence of overrides.
- Introduced new types and interfaces to manage reparse-specific data structures in the store, improving type safety and clarity in the codebase.
2026-06-09 20:38:24 +08:00
wizardchen d8a26f7c23 refactor: remove enable_multimodal and adjust parent-child configuration
- Removed the `enable_multimodal` field from the `ChunkingConfigOverride` interface and related components to streamline configuration.
- Updated the logic in `UploadConfirmDialog.vue` to default `enableParentChild` to false when initializing from knowledge base info.
- Modified the `mergeChunkingConfig` function to ensure that `EnableParentChild` is authoritative, allowing explicit control over its state.
- Added a new test case in `knowledge_process_config_test.go` to verify the behavior of parent-child toggles in process configuration resolution.
2026-06-09 20:38:24 +08:00
wizardchen 76dc23226d fix: correct graph_enabled logic in process configuration
- Updated the logic for `graph_enabled` in `UploadConfirmDialog.vue` to require both `nodeExtractConfig.enabled` and `graphEnabled` to be true.
- Added a new test case in `knowledge_process_config_test.go` to ensure that `GraphEnabled` is false when `ExtractConfig.Enabled` is false.
- Adjusted the `ResolveProcessConfig` function to reflect the updated dependency between graph and extract configurations, ensuring proper behavior in the process configuration resolution.
2026-06-09 20:38:24 +08:00
wizardchen ed9e780c8c feat(client): enhance knowledge file upload with process configuration support
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides.
- Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations.
- Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads.
- Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process.
- Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic.
- Enhanced user experience by providing detailed configuration options during the upload process.
2026-06-09 20:38:24 +08:00
wizardchen 1778c5ac7b refactor(tracing): remove Jaeger integration and related components
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
2026-06-08 21:01:04 +08:00
langcaiye 521a663b0c fix: support custom jieba dictionary directory 2026-06-08 20:10:37 +08:00
langcaiye 99f35f8dea fix(duckdb): avoid extension install during startup 2026-06-08 15:29:42 +08:00
wizardchen 28de82a92e refactor(chat): improve model name sanitization and path validation
- Updated the sanitization logic in `newStreamPacketDumper` to exclude the '.' character, enhancing security against path traversal vulnerabilities.
- Added checks to ensure the cleaned file path remains within the specified directory, preventing unauthorized access to the filesystem.
- Improved overall robustness of file handling by enforcing stricter path validation rules.
2026-06-05 19:35:02 +08:00
wizardchen b820e690f0 fix(auth): allow anonymous access to share-link endpoints (#1617)
The global Auth middleware allowlist (noAuthAPI) was missing the
share-link routes, so an unauthenticated invitee opening /register?token=
hit a 401 on /auth/invitations/lookup and the frontend bounced them to
/login instead of showing the register page. Add both share-link
endpoints to the allowlist and stop the request interceptor from treating
their 401s as a session-expiry redirect.

Refs: https://github.com/Tencent/WeKnora/issues/1617
2026-06-05 18:51:32 +08:00
wizardchen 946ff5557a feat: release v0.6.1 with new document parsing trace timeline, OpenSearch vector store driver, YAML-based built-in model configuration, and system admin settings. Includes UI redesigns, new-user onboarding guide, and various improvements and bug fixes. See CHANGELOG.md for details. 2026-06-05 16:36:13 +08:00
langcaiye 9e898ae72c fix(schema): expand knowledge source length 2026-06-05 15:12:25 +08:00
ruihao.li a12b6ff61b feat: support Milvus database selection for vector stores 2026-06-05 15:11:37 +08:00
ochan.kwon db7baad84c fix: reject cross-store reuse_vectors knowledge move
Moving knowledge with mode=reuse_vectors copies vector index entries
directly between knowledge bases, which only works when both KBs are
bound to the same vector store (CopyIndices is routed through the source
store). Nothing rejected a cross-store reuse_vectors move, so it would
copy target-KB rows into the source store and then delete the source
indices — corrupting vector data. The move endpoint validated only the
embedding model, and ListMoveTargets surfaces same-model KBs regardless
of their store binding, so a cross-store target is reachable.

Reject reuse_vectors moves whose source and target KBs are bound to
different vector stores:

- MoveKnowledge handler: synchronous 400 before enqueueing, with a
  message pointing the caller at reparse mode.
- moveOneKnowledge: re-check before marking the knowledge as processing,
  so a rejected move leaves it untouched rather than stranded.
- moveKnowledgeReuseVectors: a top-of-function guard as defense in depth
  for any directly enqueued task, satisfying the pre-existing contract
  noted in the comment ("callers must reject cross-store moves").

reparse mode is unaffected: it re-indexes into the target store from
scratch and is the supported way to move across stores.
2026-06-05 15:00:41 +08:00
toy0116 e76210a695 fix(handler): /knowledge/search · accept ?query= and reject empty keyword
Silent-failure footgun in SearchKnowledge handler:

  keyword := c.Query("keyword")   // only "keyword" recognized
  // empty keyword → service falls back to default listing (created_at DESC)
  // → caller sees "all queries return the same N newest knowledges"

This is exactly what happened to a downstream agent (Support GPT) — it was
sending ?query=... (the parameter name every MCP / semantic-search tool
uses) and getting a constant 5-card listing back regardless of input. The
caller diagnosed it as "WeKnora embedding has degraded · vector channel
returns constant rankings", but actually the parameter was never read, the
search service got keyword="", and the SQL came back as a plain
list-by-created_at.

The handler stays a filename/title LIKE search (its real purpose is the
document-management UI). Two tweaks:

1. Accept ?query= as an alias for ?keyword= · this is what MCP / agent
   integrations actually send. Cheap compatibility win, zero downside.
2. Empty keyword/query → 400 with a clear hint. Refusing the request is
   strictly better than silently returning a default listing that looks
   like "search results" but isn't query-driven.

Separately worth noting: callers who actually want semantic recall should
use POST /api/v1/knowledge-bases/{kb_id}/hybrid-search · which is the
real vector + BM25 endpoint. This handler was never the right tool for
RAG queries · the silent failure just made it look like one.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-05 14:58:20 +08:00
wizardchen e2a41ecf0a feat(thinkingControl): implement thinking mode configuration in model editor
- Added a new thinking mode control field in the ModelEditorDialog, allowing users to specify how the agent's thinking mode is formatted for API requests.
- Introduced localization strings for thinking mode labels and descriptions in English, Korean, Russian, and Chinese, enhancing user experience across multiple languages.
- Updated the model form data structure to include thinking control settings, ensuring proper handling during model edits and saves.
- Implemented utility functions for resolving and defaulting thinking control values based on provider and model, improving backend integration.
- Added tests for the new thinking control functionality to ensure reliability and correctness in handling user configurations.
2026-06-05 14:56:48 +08:00
wizardchen 063809896a refactor(chat): improve handling of <think> tags in chat messages
- Updated logic to manage <think> and </think> tags in chat message content, ensuring accurate representation of thinking states.
- Introduced a new function to handle the closing of thinking events, enhancing clarity and reducing redundancy in event emissions.
- Adjusted event handling for reasoning content to support historical message reconstruction, improving user experience during chat interactions.
2026-06-05 14:56:48 +08:00
wizardchen 4bb8906520 feat(faq): enhance FAQ handling and localization support
- Updated the agent system prompt to clarify mandatory deep read requirements for FAQ entries, ensuring accurate retrieval and response generation.
- Added new localization strings for FAQ-related labels in English and Chinese, improving user experience and consistency across the application.
- Refactored the document and grep results components to support distinct FAQ identification, enhancing clarity in the output.
- Introduced new tests for FAQ metadata handling, ensuring robust functionality and correctness in the FAQ processing logic.
2026-06-04 21:43:16 +08:00
langcaiye ba8c96d83a fix(datasource): fail sync when all fetched items fail 2026-06-04 17:39:55 +08:00
lyingbug 4369d156e1 Revert "fix: preserve agent system prompt for greetings"
This reverts commit b22303cec8.
2026-06-04 17:39:38 +08:00
wolfkill b22303cec8 fix: preserve agent system prompt for greetings 2026-06-04 17:39:01 +08:00
wizardchen dda16436d0 feat(lkeap): integrate Tencent Cloud LKEAP Rerank support
- Added support for Tencent Cloud LKEAP Rerank, including new configuration options for SecretId and SecretKey.
- Updated frontend components to handle LKEAP-specific fields and enhance user experience with appropriate labels and placeholders.
- Enhanced backend logic to manage LKEAP credentials and integrate with the new rerank API.
- Improved localization for LKEAP-related terms across multiple languages, ensuring clarity and consistency in user-facing messages.
- Introduced tests for LKEAP Reranker functionality to ensure reliability and correctness in handling requests.
2026-06-04 17:22:18 +08:00
wizardchen 12ea490d89 fix(im): inject synthetic identity so IM channels can use shared KBs
IM callback routes bypass the auth middleware because IM platforms verify
their own signatures, so the downstream QA pipeline never received a
UserID/TenantRole. The Organization-shared knowledge base resolution gates
on a non-empty UserID, so IM-channel questions silently skipped shared KBs
while the Web path (which sets UserID via JWT auth) worked.

Inject a synthetic "system-<tenantID>" identity (mirroring the API-key path,
recognised by types.IsSyntheticUserID) plus a Viewer role when building the
IM session context. Viewer is the least privilege sufficient to retrieve
shared KBs.

Closes #1600
2026-06-04 16:43:54 +08:00
wizardchen 035093dc35 feat(configuration): update asynq concurrency settings and enhance FAQ handling
- Increased the default worker pool size for the asynq server from 16 to 32 to improve performance during concurrent uploads.
- Updated the .env.example and docker-compose.yml files to reflect the new default concurrency setting.
- Enhanced the handling of FAQ entries in the tool results, ensuring distinct identification through standard questions, improving user experience and clarity in the output.
- Refactored related components to utilize the new FAQ attributes, ensuring consistent display and interaction across the application.
2026-06-04 15:48:04 +08:00
wizardchen d38a6efd87 refactor(agent): remove final_answer tool references and update related logic
- Eliminated all instances of the final_answer tool from the agent's configuration and logic, transitioning to a model where the agent concludes responses with plain text instead.
- Updated event handling and response analysis to reflect this change, ensuring that the agent's output is now directly delivered as text without relying on a dedicated final answer tool.
- Adjusted localization strings across multiple languages to remove references to final_answer, enhancing clarity and consistency in user-facing messages.
- Improved the overall structure of the agent's response handling to streamline the process and reduce potential confusion regarding tool usage.

refactor(agent): streamline answer handling and preamble logic

- Updated the agent's response handling to treat any text streamed before a tool call as a preamble, ensuring it is not included in the final answer.
- Removed the superseded marker logic from the event emission process, simplifying the flow of answer content and enhancing clarity in the UI.
- Adjusted the event handling to ensure that preamble segments are correctly retracted and relocated into the steps tree, improving the overall user experience.
- Enhanced the agent's internal logic to better manage answer segments and their states during streaming, leading to a more coherent response structure.
2026-06-04 15:48:04 +08:00
wizardchen b9dde2abe8 feat(agent): improve content filtering in streaming event handling
- Added logic to suppress whitespace-only content emitted before the actual answer starts, addressing issues with spurious empty events interleaved with tool call events.
- Enhanced the event emission process to ensure that only meaningful content is routed to the final answer area, improving the clarity and relevance of streamed responses.
2026-06-04 15:48:04 +08:00
wizardchen 876fd0276f feat(agent): enhance event routing for reasoning and answers in streaming
- Introduced new tests to ensure correct routing of reasoning and answer content during streaming, addressing UX issues where answers appeared under the Thinking phase before transitioning.
- Implemented a ThinkStreamSplitter to separate inline reasoning from answer text, allowing for real-time updates to the UI without duplication.
- Updated the event emission logic to prevent re-emission of answers that were already streamed live, ensuring a seamless user experience.
- Enhanced the ChatResponse struct to include flags for streamed answers, improving the handling of final answer events.
- Added comprehensive unit tests for the ThinkStreamSplitter to validate its functionality across various scenarios.
2026-06-04 15:48:04 +08:00
wizardchen 710d759b8e feat(ui): update SVG assets and enhance localization for knowledge framework
- Redesigned SVG assets for screenshots to improve visual consistency and clarity, including updates to gradients, titles, and layout.
- Added a new screenshot asset to showcase the Wiki Knowledge Base feature.
- Enhanced localization strings across multiple languages to reflect updated terminology and descriptions for the knowledge framework, including agentic reasoning and hybrid search strategies.
- Improved the user interface in the login view to include new tags for agent and wiki features, enhancing user awareness of available functionalities.
2026-06-04 15:48:04 +08:00