Commit Graph

423 Commits

Author SHA1 Message Date
ChenRussell 166f106b7f fix: ensure builtin engine handles simple formats correctly
Builtin engine was incorrectly falling back to simple engine for md/txt/csv files. Added explicit builtin engine check in selection logic.

Fixes #755
2026-03-15 23:58:49 +08:00
Manx98 447fac4b54 fix: NVIDIA VLM model API default BaseURL 2026-03-13 18:14:08 +08:00
Manx98 a355ac889e feat: NVIDIA chat model API support custom endpoint 2026-03-13 18:14:08 +08:00
Manx98 5afc81d00b feat: support NVIDIA model API 2026-03-13 18:14:08 +08:00
wizardchen 125d3e5e4a feat: implement knowledge move functionality
- Added API endpoints for moving knowledge items between knowledge bases, including validation for source and target compatibility.
- Implemented asynchronous task handling for knowledge moves, tracking progress and status updates.
- Introduced new localization strings for move-related UI elements in English, Korean, Russian, and Chinese.
- Enhanced the frontend to support knowledge move operations, including target selection and progress monitoring.

These changes significantly improve the application's capabilities for managing knowledge across different bases, providing users with a seamless experience in organizing their knowledge assets.
2026-03-09 13:23:16 +08:00
wizardchen 33ccc41bea refactor: update storage size calculations for vector embeddings
- Revised the storage size calculation logic for Milvus, Qdrant, and Weaviate repositories to provide more accurate estimates based on the actual data structure and indexing methods.
- Updated comments to clarify the relationship between vector dimensions and index storage requirements, enhancing code readability and maintainability.
2026-03-09 13:23:16 +08:00
wizardchen 3f340771e9 feat(storage): integrate AWS S3 support in storage settings
- Added support for AWS S3 as a storage engine, including configuration options for endpoint, region, access key, secret key, bucket name, and path prefix.
- Updated the StorageEngineConfig interface to include S3 configuration.
- Enhanced the UI components to allow users to configure S3 settings and test connections.
- Updated localization files to include new S3-related labels and descriptions in multiple languages.

These changes significantly enhance the storage capabilities of the application, providing users with more options for managing their data.
2026-03-09 11:46:56 +08:00
wizardchen 1fb2bedd2b feat: implement chat history and retrieval settings
- Added new API endpoints for managing chat history configuration and retrieval settings, allowing tenants to enable message indexing and configure search parameters.
- Introduced new Vue components for ChatHistorySettings and RetrievalSettings, providing a user-friendly interface for managing these configurations.
- Updated localization files to include new settings descriptions and labels in multiple languages.
- Enhanced the KnowledgeSearch view to support searching across both knowledge bases and chat history, improving the overall search functionality.

These changes significantly enhance the application's capabilities in managing and retrieving chat history, contributing to a more robust user experience.
2026-03-09 11:46:56 +08:00
wizardchen a4b255a456 feat: implement final_answer tool and enhance agent duration tracking
- Introduced the final_answer tool to ensure agents submit their complete responses as the last action, improving response accuracy and consistency.
- Updated the agent engine to log detailed information about tool calls and responses, including the total duration of agent execution.
- Enhanced the chat message model to include agent_duration_ms, tracking the total execution time from query start to answer delivery.
- Implemented JSON field extraction for streaming responses, allowing for more efficient handling of tool call outputs.

These changes significantly improve the agent's response handling and ensure that final answers are consistently delivered, enhancing overall user experience.
2026-03-09 11:46:56 +08:00
DaWesen e309e0bed8 feat(storage): 集成S3存储适配器
添加对AWS S3及兼容存储服务的支持:
- 实现完整的S3FileService接口
- 支持文件上传、下载、删除功能
- 添加配置支持和环境变量检查
- 实现连接测试功能
- 遵循与其他存储适配器相同的代码风格
2026-03-09 10:39:46 +08:00
MaoMengww e898d62141 refactor: change Tokenization Method 2026-03-09 10:36:16 +08:00
MaoMengww c4a5a4d99b feat: support weaviate vectordb for knowledge retrieve 2026-03-09 10:36:16 +08:00
ochan.kwon 5583bf0eb4 fix(i18n): enable multilingual support for Knowledge Graph settings
- Replace hardcoded tag options with English equivalents (Content,
  Culture, Person, etc.) so random tag generation works across locales
- Replace default extract example with English text (Romeo and Juliet)
  matching the existing config.yaml extract_graph examples
- Update graph database status string to use locale-independent value
  and align frontend comparison accordingly
- Convert error messages in fabri text handler to English
2026-03-09 10:35:41 +08:00
AndyYang 6c69de2df1 feat(security): add AES-256-GCM encryption for API keys at rest
- Add crypto utility (internal/utils/crypto.go) with AES-256-GCM encrypt/decrypt
  using SYSTEM_AES_KEY env var, with "enc:v1:" prefix for versioned ciphertext
- Encrypt tenant API key via GORM BeforeSave/AfterFind hooks and manual
  encryption in CreateTenant/UpdateAPIKey (db.Updates bypasses hooks)
- Encrypt model API key in ModelParameters Value/Scan (driver.Valuer)
- Widen api_key column from varchar(64) to varchar(256) across all DB dialects
  (MySQL, ParadeDB, SQLite) and add versioned migration 000018
- Propagate SYSTEM_AES_KEY through docker-compose, Helm secrets and values
- Fix migration 000017 PL/pgSQL dollar-quoting syntax ($ -> $$)
2026-03-09 10:35:07 +08:00
ochan.kwon edcf7116cd feat: use language-neutral English prompts for multilingual LLM responses
Convert all LLM-facing prompts, templates, and tool output labels to
language-neutral English so the model responds in the user's input
language. Update frontend relevance/match-type mappings to match.

Output language strategy:
- User-facing: reply in user's input language
- Vector-indexed: match source document language
- LLM-internal: fixed English
2026-03-06 21:08:12 +08:00
DaWesen 65743da754 fix: 修复 Milvus retriever 相关问题
- 修正 envMilvusCollection 字段名拼写错误
- 将 log.Errorf 中的 %w 改为 %v,修复日志格式化问题
- 保留错误包装功能,确保错误处理的完整性
2026-03-06 10:33:09 +08:00
AndyYang 6dde1330c2 feat: add built-in MCP service support 2026-03-05 21:57:01 +08:00
wizardchen 927e5d4750 feat: add ListChunksByParentIDs method and enhance chunk merging logic
- Implemented ListChunksByParentIDs in chunkRepository to retrieve chunks based on multiple parent IDs, improving data retrieval efficiency.
- Enhanced the chunk merging logic in PluginMerge to better handle overlapping chunks and collect merged ImageInfo for parent chunks.
- Updated the ChunkRepository interface to include the new ListChunksByParentIDs method, ensuring consistency across the application.

These changes optimize chunk management and improve the handling of image information during merges, enhancing overall application functionality.
2026-03-05 18:14:14 +08:00
wizardchen 4ddc468fea feat: add GetUserByTenantID functionality to user repository and service
- Implemented GetUserByTenantID method in userRepository to retrieve the first user associated with a given tenant ID.
- Added corresponding GetUserByTenantID method in userService to facilitate user retrieval by tenant ID.
- Updated UserService and UserRepository interfaces to include the new method, ensuring consistency across the application.
- Enhanced auth middleware to store user information in context based on tenant ID, improving user management.

These changes enhance user retrieval capabilities by allowing access to user data based on tenant associations, improving overall application functionality.
2026-03-05 18:14:14 +08:00
wizardchen b697d4945e feat: update chunk size configurations for knowledge base processing
- Increased default sizes for parent and child chunks in the knowledge base editor and settings, enhancing data processing capabilities.
- Adjusted slider settings in the UI to reflect new minimum and maximum values for chunk sizes, improving user control over configurations.
- Updated backend logic to ensure consistent handling of chunk sizes across the application.

These changes optimize the chunking process, allowing for more efficient data handling and improved performance in knowledge base operations.
2026-03-05 18:14:14 +08:00
wizardchen 00bf0460cc feat: add pinning functionality for knowledge bases
- Introduced a new API endpoint to toggle the pin status of knowledge bases, allowing users to pin important entries to the top of the list.
- Updated the frontend to include UI elements for pinning and unpinning knowledge bases, with corresponding success and error messages.
- Enhanced localization support by adding translations for pinning actions in English, Korean, Russian, and Chinese.
- Modified the database schema to include `is_pinned` and `pinned_at` fields for knowledge bases, enabling persistent pinning status.

These changes improve the organization and accessibility of knowledge bases, enhancing user experience by allowing quick access to prioritized entries.
2026-03-05 18:14:14 +08:00
wizardchen a1c0682aa3 feat: enforce maximum length for tool names in MCPTool
- Introduced a constant `maxFunctionNameLength` set to 64 characters to comply with OpenAI API constraints on tool names.
- Updated the `Name` method in `MCPTool` to truncate tool names exceeding the maximum length, ensuring unique identification while preventing potential collisions.
- Enhanced the logic to handle long service IDs by using only the first 8 characters when necessary, maintaining the integrity of tool naming conventions.

These changes improve compliance with API requirements and enhance the robustness of tool name generation.
2026-03-05 18:14:14 +08:00
wizardchen 44f38ca6c3 feat: add fallback response handling and UI indicators
- Introduced a new `is_fallback` property to track when responses are generated without knowledge base matches.
- Updated the chat components to display fallback hints and indicators in the UI, enhancing user awareness of fallback responses.
- Added translations for fallback hints in multiple languages, improving accessibility for diverse users.
- Modified backend logic to emit fallback status in response events, ensuring consistent handling across the application.

These changes improve the user experience by providing clear feedback on fallback responses, enhancing the overall interaction with the chat system.
2026-03-05 18:14:14 +08:00
wizardchen 76a5b329f3 feat: enhance storage engine connectivity check with auto-creation of buckets
- Updated the StorageCheckResponse to include a new field `bucket_created` to indicate if a bucket was automatically created during the connectivity check.
- Modified the MinIO connectivity check logic to attempt auto-creation of the bucket if it does not exist, setting a public-read policy if successful.
- Adjusted the frontend to reflect the new response structure, updating the UI to display a 'created' status when a bucket is auto-created.

These changes improve the user experience by providing clear feedback on bucket creation during storage checks, enhancing the overall functionality of the storage engine settings.
2026-03-05 18:14:14 +08:00
wizardchen fd742fea42 feat: enhance MinerU response handling for document parsing
- Added support for parsing both `results.document` and `results.files` in the MinerU API response, allowing for more flexible handling of markdown content and images.
- Implemented fallback logic to prefer `results.document` when available, improving the robustness of document processing.
- Added debug and warning logging to track response path usage and potential issues with missing content.

These changes improve the reliability and clarity of document parsing from the MinerU API, ensuring better handling of varying response formats.
2026-03-05 18:14:14 +08:00
wizardchen 45eba325e2 feat: add image icon detection and filtering functionality
- Introduced a new function `isIconImage` to identify small images and decorative elements based on pixel dimensions and file size.
- Updated the image resolution logic to filter out identified icon images from markdown content, enhancing the quality of stored images.
- Added unit tests for icon image detection to ensure accurate functionality across various image sizes.

These changes improve image handling by effectively filtering out non-essential images, leading to cleaner document processing.
2026-03-05 18:14:14 +08:00
wizardchen 1b8d7bd4de feat: add passage cleaning functionality for rerank model
- Introduced a new function `cleanPassageForRerank` to strip markdown and structural noise from text, enhancing semantic clarity for rerank models.
- Implemented comprehensive regex patterns to remove images, links, code blocks, LaTeX, HTML tags, and other formatting elements.
- Added unit tests to validate the cleaning process across various input scenarios, ensuring expected output consistency.
- Updated the `getEnrichedPassage` function to utilize the new cleaning method, improving the quality of text passed to the rerank model.

These changes significantly enhance the preprocessing of text data, leading to better performance in reranking tasks.
2026-03-05 18:14:14 +08:00
wizardchen b333499078 feat: implement parent-child chunking strategy for enhanced context management
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.

These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
2026-03-05 18:14:14 +08:00
hylaz 8399527022 fix: cleanupCtx 在启动时创建,可能在 shutdown 前过期 2026-03-05 11:17:51 +08:00
wizardchen 58783ec96c feat: implement automatic soft-delete filtering in database queries
- Added automatic filtering for `deleted_at IS NULL` in database queries to ensure only active records are returned.
- Updated the `database_query.go` documentation to reflect the new soft-delete feature and its implications for query construction.
- Enhanced the SQL validation process to include soft-delete conditions, improving data integrity and security.

These changes enhance the robustness of database interactions by enforcing soft-delete logic across relevant queries.
2026-03-03 22:23:23 +08:00
wizardchen e2234e4960 feat: implement dynamic file service resolution for knowledge extraction
- Added a new method `resolveFileServiceForKnowledge` to dynamically resolve the appropriate file service based on the tenant's storage configuration and the knowledge file's provider.
- Updated the `extractionResources` struct to include a reference to the tenant, enabling the new resolution logic.
- Modified the `processTableData` method to utilize the resolved file service, improving the handling of file operations during data extraction.

These changes enhance the flexibility and reliability of file service management in the data extraction process.
2026-03-03 22:23:23 +08:00
wizardchen 4192cfd072 refactor: simplify image handling in _resolve_images function
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.

These changes improve the clarity and efficiency of image processing and API interactions within the application.
2026-03-03 22:23:23 +08:00
wizardchen 894d2c060a refactor: remove mineru_api_base_url from ParserEngineConfig and related components
- Eliminated the `mineru_api_base_url` field from the `ParserEngineConfig` interface and its usage across the application, simplifying the configuration structure.
- Updated the `ParserEngineSettings` view to remove the corresponding input field, enhancing the user interface.
- Adjusted the `PingMinerUCloud` function to no longer require the base URL parameter, defaulting to a predefined value instead.
- Modified database migration comments to reflect the removal of the base URL from the configuration.

These changes streamline the configuration process for the MinerU Cloud integration, improving clarity and reducing potential misconfigurations.
2026-03-03 13:24:20 +08:00
wizardchen 931bd38138 feat(repository): add TagID and IsRecommended fields to vector embeddings
- Introduced `TagID` and `IsRecommended` fields across various repository implementations (Elasticsearch, Milvus, Postgres, Qdrant, SQLite) to enhance document categorization and recommendation capabilities.
- Updated the logic for handling `IsEnabled` to ensure it reflects the actual state from the source data.
- Adjusted the knowledge service to incorporate the new `IsRecommended` field, improving the indexing process for knowledge chunks.

These changes enhance the flexibility and functionality of document retrieval and indexing across the application.
2026-03-02 21:43:30 +08:00
wizardchen 7b8a78aa2a feat(repository): enhance document indexing with isEnabled and isRecommended fields
- Added `IsEnabled` and `IsRecommended` fields to the document indexing process in both ElasticFaiss and Elasticsearch repositories, improving the flexibility and functionality of document retrieval.
- Updated the data extraction process to include these new fields, ensuring they are correctly populated from the source data.
- Adjusted the indexing logic in the knowledge service to reflect these changes, maintaining consistency across the application.
2026-03-02 21:43:30 +08:00
wizardchen b1fa96c175 refactor: enhance SSRF safety checks in MinerUCloudReader
- Removed hardcoded IP address from blocked hosts in `system.go`.
- Integrated SSRF safety checks in `MinerUCloudReader` for base URL and zip URL validation.
- Updated HTTP client initialization to use a new utility function for SSRF-safe HTTP clients, improving security during API requests.

These changes enhance the security of the MinerUCloudReader by ensuring that potentially unsafe URLs are properly validated before use.
2026-03-02 21:21:49 +08:00
wizardchen 284d6c623e feat: add deleteAllSessions functionality and update batch delete logic
- Introduced a new API function `deleteAllSessions` to allow deletion of all sessions for a tenant.
- Updated the `BatchManageDialog` component to support a confirmation dialog for deleting all sessions.
- Enhanced the menu component to reflect the new delete all functionality with appropriate confirmation messages.
- Added internationalization support for the new delete all confirmation message in multiple languages.

These changes improve session management by providing users with the ability to delete all sessions at once, enhancing the overall user experience.
2026-03-02 21:21:49 +08:00
wizardchen 4222b9dc21 fix: handle empty metadata case in Knowledge struct
- Updated the GetMetadata method to return an empty map when the Metadata field is empty, preventing potential errors during metadata retrieval.

This change improves the robustness of the Knowledge struct by ensuring it handles empty metadata gracefully.
2026-03-02 21:21:49 +08:00
wizardchen e595e9345e feat: implement tenant access authorization in TenantHandler
- Added `authorizeTenantAccess` method to check user permissions for accessing specific tenants.
- Integrated authorization checks in `GetTenant`, `UpdateTenant`, and `DeleteTenant` methods to enforce access control.
- Updated `ListTenants` method to ensure proper authentication before returning tenant data.

This update enhances security by ensuring that only authorized users can access or modify tenant information.
2026-03-02 21:21:49 +08:00
wizardchen aefa1c6fe8 feat: enhance system information display with database version
- Added `db_version` field to the `SystemInfo` interface to expose the current database migration version.
- Updated the system information response to include the database version, reflecting its state during application runtime.
- Enhanced the UI in the SystemInfo component to display the database version with appropriate labels and descriptions in multiple languages.

This update improves transparency regarding the database state within the system information settings.
2026-03-02 21:21:49 +08:00
wizardchen dbab7b0405 feat: introduce storage provider configuration and enhance backward compatibility
- Added a new `StorageProviderConfig` struct to the `KnowledgeBase` for improved storage provider management, separating provider selection from storage credentials.
- Updated the `UnmarshalJSON` method to auto-populate `StorageProviderConfig` from the legacy `StorageConfig` for backward compatibility.
- Refactored various service methods to utilize the new storage provider configuration, ensuring a smooth transition from the legacy system.
- Implemented database migrations to add the `storage_provider_config` column and migrate existing data, maintaining compatibility with historical knowledge bases.

This update enhances the clarity and flexibility of storage configurations while ensuring compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 565660dc46 feat: improve document chunk loading and error handling in doc-content component
- Added state management for pending document requests to enhance user experience during chunk loading.
- Implemented error handling for chunk loading failures, providing user feedback through warnings.
- Adjusted scroll handling logic to ensure smoother pagination and prevent multiple concurrent requests.

This update enhances the reliability of document loading and improves user interaction during scrolling.
2026-03-02 21:21:49 +08:00
wizardchen 4a9977b11f feat: enhance document loading and state management in doc-content component
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.

This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
2026-03-02 21:21:49 +08:00
wizardchen 393ebf9f92 feat: standardize storage configuration and enhance backward compatibility
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.

This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 469f320d10 feat: enhance storage engine configuration and UI updates
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.

This update enhances the flexibility of storage options and improves user experience with better UI interactions.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
wizardchen 5241dbc39e feat(security): implement path and filename validation utilities
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.

This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
2026-03-02 21:21:49 +08:00
ChenRussell 48cf6aeaea fix(swagger): resolve interface documentation generation error and regenerate docs
- Fix swagger documentation generation failure issue
- Regenerate swagger interface documentation
2026-02-27 09:52:07 +08:00
joeyczheng 1b56e99cac feat: support milvus vectordb for knowledge retrieve
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-27 09:51:17 +08:00
begoniezhao daf7dee4d2 refactor: Unify /url route and file URL handling 2026-02-26 19:46:08 +08:00