Commit Graph

978 Commits

Author SHA1 Message Date
wizardchen 3daaabfc4a refactor(elasticsearch): remove ".keyword" suffix from ID fields in Elasticsearch queries
- Updated Elasticsearch repository methods to use ID fields without the ".keyword" suffix, aligning with the new index mapping requirements.
- Enhanced documentation for the VectorEmbedding struct to clarify expected index mapping and usage guidelines for ID fields.
- Ensured consistency across all repository methods that interact with ID fields, improving query performance and maintainability.
2026-03-25 22:08:29 +08:00
wizardchen 164c39444d feat(knowledge): add optional channel parameter to CreateKnowledgeFromFile
- Updated the CreateKnowledgeFromFile function to include a new optional `channel` parameter, allowing for better tracking of knowledge entry sources.
- Enhanced documentation to reflect the addition of the channel parameter, improving clarity for future developers.
2026-03-25 22:08:29 +08:00
wizardchen 5245475e0f feat(knowledge): add channel support to knowledge entries and related components
- Introduced a new `channel` field in the Knowledge struct and associated request types to track the source channel (e.g., "web", "api", "browser_extension").
- Updated various frontend components to display channel information and enhance user experience with channel labels.
- Enhanced localization files to support channel labels in English and Chinese.
- Modified backend services and database migrations to accommodate the new channel feature, ensuring consistent tracking across knowledge entries.
- Refactored related functions to integrate channel handling, improving overall knowledge management and context.
2026-03-25 22:08:29 +08:00
wizardchen aedaea274d feat(agent): improve logging of LLM call messages to enhance clarity and reduce redundancy
- Updated logging in the callLLMWithRetry function to summarize message details, limiting the output to the last few messages to avoid repetition.
- Introduced a constant to define the maximum number of detailed messages logged, improving the readability of logs during LLM calls.
2026-03-25 22:08:29 +08:00
wizardchen 5b2ebd1f44 feat(channel): add channel support across various components and locales
- Introduced a new `channel` field in multiple request and message structures to track the source channel (e.g., "web", "api", "im").
- Updated frontend API calls and chat components to include the `channel` parameter, ensuring consistent channel tracking in user messages.
- Enhanced localization files to support channel labels in English, Korean, Russian, and Chinese.
- Added database migration scripts to incorporate the `channel` column in the messages table, facilitating the storage of channel information.
- Refactored related functions and components to accommodate the new channel feature, improving overall message context and tracking.
2026-03-25 22:08:29 +08:00
wizardchen de0169d7c0 feat(chat): add ParallelToolCalls option to ChatOptions and update related functionality
- Introduced a new ParallelToolCalls field in ChatOptions to control parallel execution of tool calls.
- Updated the BuildChatCompletionRequest method to handle the new ParallelToolCalls option, ensuring it is correctly propagated in requests.
- Enhanced the streamThinkingToEventBus function to utilize the ParallelToolCalls setting.
- Added comprehensive tests to validate the behavior of ParallelToolCalls in various scenarios, ensuring correct propagation and default handling.
2026-03-25 22:08:29 +08:00
wizardchen af2956fdf1 feat(tests): add comprehensive tests for MCPTool functionality
- Introduced a new test file for MCPTool, covering various aspects such as name sanitization, name generation based on service and tool names, and description handling.
- Implemented tests to ensure consistent tool name generation across different UUIDs and validate the maximum length constraints for tool names.
- Added tests for tool registration behavior, ensuring that the first registered tool wins in case of name collisions.
- Enhanced parameter handling tests to verify schema retrieval and default behavior when no schema is provided.
2026-03-25 22:08:29 +08:00
wizardchen a8924e301e feat(agent): enhance message consolidation and context management
- Improved the message consolidation logic to preserve the current turn's user query and all subsequent assistant/tool messages, ensuring better context retention.
- Updated the CompressContext function to reflect the new consolidation strategy, maintaining the system prompt and relevant recent messages.
- Refactored the context manager to support optional message repository for improved context rebuilding from persistent storage.
- Added comprehensive tests to validate the new consolidation behavior and ensure correct message handling across various scenarios.
2026-03-25 22:08:29 +08:00
wizardchen b980d4619d feat(knowledge): add ClearKnowledgeBaseContents API to delete all entries in a knowledge base
- Implemented ClearKnowledgeBaseContents function in the Client to asynchronously delete all knowledge entries while preserving the knowledge base.
- Added corresponding ClearKnowledgeBaseContents handler in the KnowledgeHandler to validate access and enqueue deletion tasks.
- Updated router to include a new DELETE endpoint for clearing knowledge base contents.
2026-03-25 22:08:29 +08:00
wizardchen 2b56c4fbac feat(agent): enhance streaming diagnostics and event tracking
- Added diagnostics for streaming events in the AgentEngine, including chunk counts and response type distributions.
- Implemented logging for emitted event types during the thinking process to aid in debugging and performance analysis.
- Updated the session handler to provide more informative warnings when fallback answers are emitted due to non-streaming behavior.
- Introduced a dedicated HTTP client for Ollama to improve connection handling and timeout management during streaming calls.
2026-03-25 22:08:29 +08:00
wizardchen 1c9503b063 feat(docparser): enhance image resolution capabilities in markdown and HTML
- Updated regex patterns in MarkdownImageUtil to support alt text containing brackets and handle MIME types with hyphens.
- Implemented new functions in ImageResolver for resolving HTML <img> tags with data URIs and bare base64 content, improving image handling in markdown.
- Added comprehensive tests for various image scenarios, ensuring robust handling of data URIs and base64 images.
2026-03-25 22:08:29 +08:00
wizardchen 2b4e614dc2 refactor(agent): enhanced the checkFAQQuestionDuplicate function in knowledge.go by adding structured comments to outline the validation steps for FAQ questions, improving readability and maintainability. 2026-03-25 22:08:29 +08:00
wizardchen 4fb08f3ec8 feat(docparser): enhance sortedKeys function for numeric string sorting
- Modified the sortedKeys function to sort keys numerically when all keys are numeric strings, ensuring correct order for keys like "2" and "10".
- Updated the NeedsKBRetrieval method to include IntentSummarize as a valid case for knowledge base retrieval.
- Added a new test data file for chat import to support testing of document parsing and processing functionalities.
2026-03-25 22:08:29 +08:00
wizardchen e7c21c0733 feat(chat): add Volcengine support and refactor thinking configuration
- Introduced Volcengine as a new provider in the chat provider specifications.
- Refactored the thinking configuration to a more generic format, applicable to both LKEAP and Volcengine.
- Updated request customizers to handle the new thinking configuration structure for both LKEAP and Volcengine providers.
2026-03-25 22:08:29 +08:00
wizardchen 3e3ffa4b0b feat(i18n): add JSON file type support across multiple locales
- Introduced support for JSON file types in English, Korean, Russian, and Chinese locale files.
- Updated the file type acceptance criteria in the document upload component to include JSON.
- Enhanced the knowledge base parser settings to recognize JSON as a valid file type.
- Implemented JSON conversion functionality in the document parser, allowing for JSON content to be processed and converted to markdown.
- Added comprehensive tests for JSON to markdown conversion to ensure functionality and reliability.
2026-03-25 22:08:29 +08:00
wizardchen 649c9d3dc9 refactor(chat_pipeline): improve regex patterns for table parsing
- Updated regex patterns for table separator and row matching to use [ \t] instead of \s, preventing newline consumption across rows.
- Enhanced comments for clarity on the changes made to the regex expressions.
2026-03-25 22:08:29 +08:00
wizardchen 47397464ef refactor(chat_pipeline): enhance merging logic and add partial overlap removal
- Updated the merging process in the chat pipeline to include a new step for re-merging overlapping ranges introduced by context expansion.
- Implemented a new function, removePartialOverlaps, to eliminate chunks that are largely contained within higher-scored chunks, improving deduplication accuracy.
- Enhanced normalization and overlap checking utilities to support the new deduplication logic.
2026-03-25 22:08:29 +08:00
wizardchen 15ebfe0b4c refactor(model): streamline model filtering and SSRF validation
- Simplified model filtering logic by removing the deduplication function and directly filtering models based on type.
- Updated SSRF validation to preserve backend-managed fields when parameters are not provided by the frontend.
- Clarified comments and improved code readability in the ModelEditorDialog and ModelSettings components.
2026-03-25 22:08:29 +08:00
wizardchen 68dd6aa778 refactor(knowledge): increase page size for tag retrieval in buildTagMap function
- Updated the page size from 100 to 1000 in the buildTagMap function to enhance performance and efficiency when retrieving tags from the knowledge base.
2026-03-25 22:08:29 +08:00
wizardchen a6c38123b6 refactor(knowledge): optimize tag retrieval in buildTagMap function
- Changed the tag retrieval process to support pagination, allowing for more efficient handling of large sets of tags.
- Adjusted the page size to 100 for better performance and reduced memory usage.
- Ensured that the function continues fetching tags until all are retrieved, improving the completeness of the tag map.
2026-03-25 22:08:29 +08:00
wizardchen f368b987bb feat(agent): refactor agent engine creation and enhance MCP tool registration
- Improved the CreateAgentEngine method by breaking down the process into distinct steps for better readability and maintainability.
- Added a new registerMCPTools method to streamline the registration of MCP tools based on the agent configuration.
- Enhanced error handling and logging for MCP service registration, ensuring clearer feedback during the process.
- Introduced resolveKBAndDocInfos to load knowledge base metadata and selected document information for prompt generation.
2026-03-25 22:08:29 +08:00
wizardchen b02e3f1dbb feat(knowledge): implement knowledge item rebuild confirmation dialog and enhance chunk processing options
- Added a confirmation dialog for rebuilding knowledge items in the KnowledgeBase component.
- Introduced new state variables to manage the rebuild process and item selection.
- Enhanced the processChunks function to support question generation options based on configuration.
- Updated the knowledge service to handle question generation settings during document processing.
2026-03-25 22:08:29 +08:00
wizardchen 0e01a8ca64 feat(server): enhance server startup and shutdown process
- Added support for SO_REUSEPORT in the listenWithRetry function to improve port binding during hot-reloads.
- Implemented graceful shutdown by closing the listener immediately upon receiving a shutdown signal, allowing for quicker port release.
- Updated logging to provide clearer feedback during server shutdown and error handling.
2026-03-25 22:08:29 +08:00
wizardchen 39816f2756 feat(security): add validation for file path to prevent path traversal attacks
- Implemented a check to reject file paths containing "..", enhancing security against path traversal vulnerabilities in the file serving functionality.
2026-03-25 22:08:29 +08:00
wizardchen b2c009e61c feat(security): implement SSRF and path traversal protections in image handling and file downloads
- Added SSRF protection by stripping client-supplied URL and Caption fields from image attachments in the QA request handler.
- Introduced a validation function to ensure Feishu API path parameters contain only safe characters, preventing path traversal attacks.
- Enhanced the WeCom webhook adapter to reject internal/private URLs unless they are on an allowlist, improving security during file downloads.
- Implemented a mechanism to bypass SSRF checks for trusted IM platform API hosts.
2026-03-25 22:08:29 +08:00
wizardchen 052be53c42 feat(prompts): update assistant descriptions and add intent-specific templates
- Updated the assistant descriptions in various prompt templates to specify that they are developed by Tencent.
- Added new intent-specific system prompt templates for greeting, chitchat, follow-up, image analysis, summarization, and web search unavailability scenarios.
- Enhanced citation rules in the system prompts to ensure proper inline citation formatting.
- Refactored the rewrite prompt to classify user intent more accurately and output structured JSON responses.
2026-03-25 22:08:29 +08:00
wizardchen 109393dd16 fix(server): reduce kill delay and implement port retry mechanism
- Changed the kill delay from 30 seconds to 2 seconds in the configuration.
- Refactored the server startup process to include a retry mechanism for binding to the port, addressing potential issues during hot-reload scenarios.
- Added a new function, listenWithRetry, to handle port binding with exponential backoff, improving server reliability during restarts.
2026-03-25 22:08:29 +08:00
wizardchen 0bb1a581de feat(agent): enhance token management and context handling in AgentEngine
- Added token usage tracking to the AgentEngine, allowing for better estimation of current context token counts.
- Implemented a new method to estimate current tokens based on previous usage and newly appended messages, improving context management.
- Updated context window management to utilize estimated token counts for more efficient message consolidation.
- Enhanced streaming methods to include token usage information, providing better insights into LLM interactions.
- Refactored the token estimator to support BPE tokenization, ensuring accurate token counts for messages.
- Added unit tests to validate new token estimation and context management functionalities.
2026-03-25 22:08:29 +08:00
wizardchen 0d4ceb6c96 feat(image): enhance image resolution handling for data URIs and remote images
- Added functionality to resolve embedded data:image;base64 images in markdown, storing them and replacing references with stable URLs.
- Updated the knowledge service to process data URIs before handling remote images, ensuring all images are resolved consistently.
- Improved logging for resolved images, providing better insights during manual processing.
- Added unit tests to validate the new data URI resolution functionality and ensure reliability.
2026-03-25 22:08:29 +08:00
wizardchen 9650188a35 feat(chunk): add FindFAQChunkWithDuplicateQuestion method for enhanced duplicate question checking
- Implemented a new method in the chunk repository to find FAQ chunks with overlapping standard or similar questions.
- Updated the knowledge service to utilize this method for checking duplicates during FAQ question creation.
- Improved error handling for duplicate questions, providing clearer feedback on conflicts.
2026-03-25 22:08:29 +08:00
wizardchen 3a8bd36d8a feat(env): add SSRF whitelist configuration to .env.example and docker-compose.yml 2026-03-25 22:08:29 +08:00
wizardchen d7fd3b5eee feat(types): add language field to QuestionGenerationPayload for locale support 2026-03-25 22:08:29 +08:00
wizardchen 59d2f6b54c feat(graph): enhance graph extraction prompt rendering and language context handling
- Added a new method to render graph extraction prompts with shared placeholders for language.
- Updated entity and relationship extraction methods to utilize the new rendering function, improving prompt customization.
- Enhanced question generation task to include language context, ensuring prompts are generated in the correct language.
- Improved error handling for empty prompt configurations in question generation, enhancing robustness.
2026-03-25 22:08:29 +08:00
wizardchen 08ac3937ac fix(agent): update tool hint comment for clarity in UI progress display 2026-03-25 22:08:29 +08:00
wizardchen a167886aac chore(docker): update PostgreSQL image version to v0.22.2-pg17 in development and production configurations 2026-03-25 22:08:29 +08:00
wizardchen e138e0c81a refactor(agent): remove query display from DatabaseQuery component and update tool result structure
- Removed the query display section from the DatabaseQuery component to streamline the UI.
- Updated the DatabaseQueryData interface to eliminate the query field, reflecting changes in tool result handling.
- Enhanced the DatabaseQuery tool execution to focus on returning structured results without exposing raw SQL queries.
- Added a new utility to strip <think> blocks from LLM outputs, improving content clarity and user experience.
2026-03-25 22:08:29 +08:00
wizardchen 637376428e feat(agent): implement core agent functionalities and context management
- Introduced the AgentEngine with methods for executing tool calls, managing context windows, and handling final answer generation.
- Added functionality to stream LLM responses and tool results through the EventBus, enhancing real-time interaction.
- Implemented context management to consolidate messages and prevent exceeding token limits, ensuring efficient use of context.
- Enhanced error handling and logging for tool execution and response analysis, improving robustness and traceability.
- Added unit tests to validate the new functionalities and ensure reliability in agent operations.
2026-03-25 22:08:29 +08:00
wizardchen 2f7e3a8ac1 feat(agent): enhance tool execution and message sanitization
- Introduced a default timeout for tool execution to prevent indefinite hangs during long-running tasks.
- Implemented context cancellation handling in the execution loop to gracefully manage timeouts and user cancellations.
- Added a JSON repair utility to fix common malformations in tool arguments, improving robustness in argument parsing.
- Enhanced message sanitization to merge consecutive user messages and handle orphaned tool results, ensuring compatibility with LLM providers.
- Added unit tests for the new JSON repair and message sanitization functionalities to validate their effectiveness.
2026-03-25 22:08:29 +08:00
wizardchen 0851d6d656 feat(agent): implement memory consolidation and context management
- Introduced a memory consolidator to summarize older messages when the context window exceeds a specified limit, preserving key facts and tool results.
- Enhanced the AgentEngine to utilize the new token estimator and memory consolidator for improved context window management.
- Implemented logic to compress message history when approaching token limits, ensuring efficient use of context.
- Added unit tests for the memory consolidator and token estimator to validate functionality and performance.
- Updated tool execution to validate parameters against JSON schema before execution, improving error handling and robustness.
2026-03-25 22:08:29 +08:00
wizardchen 9f4fb39e10 feat(chat): introduce provider-specific request customizers and specs
- Added a new `ProviderSpec` struct to define provider-specific behaviors for chat completions.
- Implemented request and endpoint customizers for various providers including Qwen3, LKEAP, DeepSeek, and Generic.
- Refactored the `NewRemoteChat` function to utilize the new provider specs for customizing requests based on the provider and model name.
- Removed legacy chat implementations for LKEAP, NVIDIA, DeepSeek, and Qwen, consolidating their logic into the new structure.
2026-03-25 22:08:29 +08:00
wizardchen e936e0b347 feat(agent): implement LLM call timeout and transient error handling
- Introduced a configurable LLM call timeout with a default value, allowing for better control over LLM call durations.
- Added logic to retry transient errors (e.g., timeouts, rate limits) up to a specified maximum number of retries, improving robustness in error handling.
- Implemented parameter casting for tool arguments to ensure correct types are used, addressing common LLM quirks.
- Enhanced tool execution error messages with guidance for retrying with different approaches.
- Added validation for configuration values to prevent runtime errors.
2026-03-25 22:08:29 +08:00
lyingbug 00ac14aabb fix: add LOAD_HISTORY to rag_stream pipeline for independent history
Previously, rag_stream relied entirely on the rewrite plugin's
internal loadHistory() to load conversation history. This meant that
when rewrite was disabled (enable_rewrite=false), RAG mode had NO
conversation history at all — multi-turn conversations silently
degraded to single-turn.

Changes:
1. Add LOAD_HISTORY as the first step in rag_stream pipeline,
   ensuring history is always available regardless of rewrite config.

2. Make rewrite plugin reuse chatManage.History if already populated
   by the upstream LOAD_HISTORY step, avoiding duplicate DB queries.

This brings rag_stream in line with chat_history_stream which already
has LOAD_HISTORY as an independent step.
2026-03-25 22:08:29 +08:00
lyingbug 03fda55261 fix(frontend): replace v-html with text interpolation in deepThink
Replace v-html + DOMPurify sanitization with plain Vue text
interpolation ({{ }}) and CSS white-space: pre-wrap for the
thinking content display.

The thinking content is raw model output that should never be
interpreted as HTML. Using v-html here created an unnecessary XSS
attack surface — even with DOMPurify, the \n→<br/> replacement
happened before sanitization, and future changes to sanitizer config
could introduce vulnerabilities.

Plain text rendering with pre-wrap:
- Eliminates XSS risk entirely (no HTML parsing)
- Preserves line breaks natively via CSS
- Removes the sanitizeHTML import and safeProcessThinkContent function
- Simpler code with identical visual output
2026-03-25 22:08:29 +08:00
lyingbug 34d711d15f fix(frontend): unify think tag parsing between stream and history
Two consistency fixes for <think> tag parsing in index.vue:

1. Streaming path (line 579): Changed indexOf to lastIndexOf when
   splitting thinking content from answer, matching the history
   loading path. Prevents incorrect splitting if </think> appears
   multiple times (e.g., model discusses think tags in answer).

2. History loading path (line 274-281): Removed redundant split()
   call, now uses only lastIndexOf consistently. The split was
   creating an unnecessary intermediate array that could produce
   wrong results with multiple </think> occurrences.

Both paths now use identical logic: lastIndexOf('</think>') to find
the boundary, substring(0, index) for thinkContent, substring(index+8)
for answer content.
2026-03-25 22:08:29 +08:00
lyingbug e5c46642e9 fix: handle thinking stream edge cases (unclosed tags + missing done)
Two related fixes for thinking mode streaming:

1. chat_completion_stream.go: When the response channel closes
   unexpectedly (context cancelled, upstream error) while thinking
   is in progress, the <think> tag was left unclosed. This caused
   the stored message to have an incomplete tag, making the frontend
   show a perpetual "thinking..." state on history reload.
   Now we emit a closing </think> tag when the channel ends with
   thinkingStarted=true but thinkingEnded=false.

2. remote_api.go: The thinking Done=true event was only sent when
   the first answer content chunk arrived. If the model produced
   only reasoning with no answer (e.g., finish_reason set with
   empty content), the thinking-done signal was never emitted.
   Now we also check isDone && state.hasThinking at the end of
   processStreamDelta to ensure the done event is always sent.
2026-03-25 22:08:29 +08:00
lyingbug 1c5013e559 fix: strip thinking content before indexing to chat history KB
The assistant message content may contain <think>...</think> tags from
extended thinking mode. Previously this raw thinking content was
indexed into the chat history knowledge base, polluting embeddings
and degrading RAG retrieval quality with intermediate reasoning text.

Now we strip thinking blocks before building the passage for
IndexMessageToKB, consistent with how load_history and rewrite
plugins already strip thinking from conversation history.
2026-03-25 22:08:29 +08:00
lyingbug cbee01f5e9 fix: improve markdown table cleaning and filter empty passages in rerank (#752)
Enhanced cleanPassageForRerank() to convert markdown table data rows
(| col1 | col2 |) into plain text ("col1, col2") instead of leaving
them intact. Added filtering to skip empty/whitespace-only passages
before sending to the rerank API, preventing Baidu Qianfan 400 errors
caused by empty document fragments after markdown cleaning.
2026-03-25 22:08:29 +08:00
ochan.kwon b291381cc2 feat: auto-describe MCP tool result images via VLM
When an MCP tool returns image content items, the agent engine now
automatically generates text descriptions using the configured VLM
model and appends them to the tool result message. This makes image
content accessible to LLMs that receive tool results as text.

Images are not passed directly to the LLM because Chat Completions API
does not reliably support images in tool role messages across providers
(tested: gpt-5.4 silently ignores them, Qwen3.5 processes them).

Changes:
- Add Images field to ToolResult for preserving base64 data URIs
- Add extractContentAndImages() with MIME whitelist, size/count limits
- Add ImageDescriberFunc injection to AgentEngine (no vlm package dep)
- Add describeImages() with graceful error handling and ctx cancellation
- Add VLM model injection in CreateAgentEngine (same pattern as
  SetAppConfig/SetSkillsManager)
- Redact image base64 from Data map to prevent log/SSE exposure
2026-03-25 21:38:15 +08:00
Manx98 9d2228a826 fix: upgrade marked dependency to version 17.0.5 to fix code block rendering 2026-03-25 21:25:54 +08:00
ochan.kwon 8d8cc051be fix(tenant): prevent api_key encryption loss on tenant settings update
When a tenant setting is updated via PUT /tenants/kv/{key} (e.g.,
retrieval-config, parser-engine-config), the GORM AfterFind-decrypted
plaintext api_key was silently written back to the database because
db.Updates() does not trigger the BeforeSave hook.

Guard the repository's UpdateTenant to skip writing plaintext api_key
back to the database. Pre-encrypted values (enc:v1:… from CreateTenant
/ UpdateAPIKey) are written as-is; AfterFind-decrypted plaintext is
blanked so GORM skips the column entirely.

Closes #798
2026-03-25 21:25:21 +08:00