Commit Graph

276 Commits

Author SHA1 Message Date
ChenRussell 166f106b7f fix: ensure builtin engine handles simple formats correctly
Builtin engine was incorrectly falling back to simple engine for md/txt/csv files. Added explicit builtin engine check in selection logic.

Fixes #755
2026-03-15 23:58:49 +08:00
Manx98 5afc81d00b feat: support NVIDIA model API 2026-03-13 18:14:08 +08:00
wizardchen 125d3e5e4a feat: implement knowledge move functionality
- Added API endpoints for moving knowledge items between knowledge bases, including validation for source and target compatibility.
- Implemented asynchronous task handling for knowledge moves, tracking progress and status updates.
- Introduced new localization strings for move-related UI elements in English, Korean, Russian, and Chinese.
- Enhanced the frontend to support knowledge move operations, including target selection and progress monitoring.

These changes significantly improve the application's capabilities for managing knowledge across different bases, providing users with a seamless experience in organizing their knowledge assets.
2026-03-09 13:23:16 +08:00
wizardchen 33ccc41bea refactor: update storage size calculations for vector embeddings
- Revised the storage size calculation logic for Milvus, Qdrant, and Weaviate repositories to provide more accurate estimates based on the actual data structure and indexing methods.
- Updated comments to clarify the relationship between vector dimensions and index storage requirements, enhancing code readability and maintainability.
2026-03-09 13:23:16 +08:00
wizardchen 1fb2bedd2b feat: implement chat history and retrieval settings
- Added new API endpoints for managing chat history configuration and retrieval settings, allowing tenants to enable message indexing and configure search parameters.
- Introduced new Vue components for ChatHistorySettings and RetrievalSettings, providing a user-friendly interface for managing these configurations.
- Updated localization files to include new settings descriptions and labels in multiple languages.
- Enhanced the KnowledgeSearch view to support searching across both knowledge bases and chat history, improving the overall search functionality.

These changes significantly enhance the application's capabilities in managing and retrieving chat history, contributing to a more robust user experience.
2026-03-09 11:46:56 +08:00
wizardchen a4b255a456 feat: implement final_answer tool and enhance agent duration tracking
- Introduced the final_answer tool to ensure agents submit their complete responses as the last action, improving response accuracy and consistency.
- Updated the agent engine to log detailed information about tool calls and responses, including the total duration of agent execution.
- Enhanced the chat message model to include agent_duration_ms, tracking the total execution time from query start to answer delivery.
- Implemented JSON field extraction for streaming responses, allowing for more efficient handling of tool call outputs.

These changes significantly improve the agent's response handling and ensure that final answers are consistently delivered, enhancing overall user experience.
2026-03-09 11:46:56 +08:00
DaWesen e309e0bed8 feat(storage): 集成S3存储适配器
添加对AWS S3及兼容存储服务的支持:
- 实现完整的S3FileService接口
- 支持文件上传、下载、删除功能
- 添加配置支持和环境变量检查
- 实现连接测试功能
- 遵循与其他存储适配器相同的代码风格
2026-03-09 10:39:46 +08:00
MaoMengww e898d62141 refactor: change Tokenization Method 2026-03-09 10:36:16 +08:00
MaoMengww c4a5a4d99b feat: support weaviate vectordb for knowledge retrieve 2026-03-09 10:36:16 +08:00
AndyYang 6c69de2df1 feat(security): add AES-256-GCM encryption for API keys at rest
- Add crypto utility (internal/utils/crypto.go) with AES-256-GCM encrypt/decrypt
  using SYSTEM_AES_KEY env var, with "enc:v1:" prefix for versioned ciphertext
- Encrypt tenant API key via GORM BeforeSave/AfterFind hooks and manual
  encryption in CreateTenant/UpdateAPIKey (db.Updates bypasses hooks)
- Encrypt model API key in ModelParameters Value/Scan (driver.Valuer)
- Widen api_key column from varchar(64) to varchar(256) across all DB dialects
  (MySQL, ParadeDB, SQLite) and add versioned migration 000018
- Propagate SYSTEM_AES_KEY through docker-compose, Helm secrets and values
- Fix migration 000017 PL/pgSQL dollar-quoting syntax ($ -> $$)
2026-03-09 10:35:07 +08:00
ochan.kwon edcf7116cd feat: use language-neutral English prompts for multilingual LLM responses
Convert all LLM-facing prompts, templates, and tool output labels to
language-neutral English so the model responds in the user's input
language. Update frontend relevance/match-type mappings to match.

Output language strategy:
- User-facing: reply in user's input language
- Vector-indexed: match source document language
- LLM-internal: fixed English
2026-03-06 21:08:12 +08:00
DaWesen 65743da754 fix: 修复 Milvus retriever 相关问题
- 修正 envMilvusCollection 字段名拼写错误
- 将 log.Errorf 中的 %w 改为 %v,修复日志格式化问题
- 保留错误包装功能,确保错误处理的完整性
2026-03-06 10:33:09 +08:00
AndyYang 6dde1330c2 feat: add built-in MCP service support 2026-03-05 21:57:01 +08:00
wizardchen 927e5d4750 feat: add ListChunksByParentIDs method and enhance chunk merging logic
- Implemented ListChunksByParentIDs in chunkRepository to retrieve chunks based on multiple parent IDs, improving data retrieval efficiency.
- Enhanced the chunk merging logic in PluginMerge to better handle overlapping chunks and collect merged ImageInfo for parent chunks.
- Updated the ChunkRepository interface to include the new ListChunksByParentIDs method, ensuring consistency across the application.

These changes optimize chunk management and improve the handling of image information during merges, enhancing overall application functionality.
2026-03-05 18:14:14 +08:00
wizardchen 4ddc468fea feat: add GetUserByTenantID functionality to user repository and service
- Implemented GetUserByTenantID method in userRepository to retrieve the first user associated with a given tenant ID.
- Added corresponding GetUserByTenantID method in userService to facilitate user retrieval by tenant ID.
- Updated UserService and UserRepository interfaces to include the new method, ensuring consistency across the application.
- Enhanced auth middleware to store user information in context based on tenant ID, improving user management.

These changes enhance user retrieval capabilities by allowing access to user data based on tenant associations, improving overall application functionality.
2026-03-05 18:14:14 +08:00
wizardchen b697d4945e feat: update chunk size configurations for knowledge base processing
- Increased default sizes for parent and child chunks in the knowledge base editor and settings, enhancing data processing capabilities.
- Adjusted slider settings in the UI to reflect new minimum and maximum values for chunk sizes, improving user control over configurations.
- Updated backend logic to ensure consistent handling of chunk sizes across the application.

These changes optimize the chunking process, allowing for more efficient data handling and improved performance in knowledge base operations.
2026-03-05 18:14:14 +08:00
wizardchen 00bf0460cc feat: add pinning functionality for knowledge bases
- Introduced a new API endpoint to toggle the pin status of knowledge bases, allowing users to pin important entries to the top of the list.
- Updated the frontend to include UI elements for pinning and unpinning knowledge bases, with corresponding success and error messages.
- Enhanced localization support by adding translations for pinning actions in English, Korean, Russian, and Chinese.
- Modified the database schema to include `is_pinned` and `pinned_at` fields for knowledge bases, enabling persistent pinning status.

These changes improve the organization and accessibility of knowledge bases, enhancing user experience by allowing quick access to prioritized entries.
2026-03-05 18:14:14 +08:00
wizardchen 44f38ca6c3 feat: add fallback response handling and UI indicators
- Introduced a new `is_fallback` property to track when responses are generated without knowledge base matches.
- Updated the chat components to display fallback hints and indicators in the UI, enhancing user awareness of fallback responses.
- Added translations for fallback hints in multiple languages, improving accessibility for diverse users.
- Modified backend logic to emit fallback status in response events, ensuring consistent handling across the application.

These changes improve the user experience by providing clear feedback on fallback responses, enhancing the overall interaction with the chat system.
2026-03-05 18:14:14 +08:00
wizardchen 1b8d7bd4de feat: add passage cleaning functionality for rerank model
- Introduced a new function `cleanPassageForRerank` to strip markdown and structural noise from text, enhancing semantic clarity for rerank models.
- Implemented comprehensive regex patterns to remove images, links, code blocks, LaTeX, HTML tags, and other formatting elements.
- Added unit tests to validate the cleaning process across various input scenarios, ensuring expected output consistency.
- Updated the `getEnrichedPassage` function to utilize the new cleaning method, improving the quality of text passed to the rerank model.

These changes significantly enhance the preprocessing of text data, leading to better performance in reranking tasks.
2026-03-05 18:14:14 +08:00
wizardchen b333499078 feat: implement parent-child chunking strategy for enhanced context management
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.

These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
2026-03-05 18:14:14 +08:00
wizardchen e2234e4960 feat: implement dynamic file service resolution for knowledge extraction
- Added a new method `resolveFileServiceForKnowledge` to dynamically resolve the appropriate file service based on the tenant's storage configuration and the knowledge file's provider.
- Updated the `extractionResources` struct to include a reference to the tenant, enabling the new resolution logic.
- Modified the `processTableData` method to utilize the resolved file service, improving the handling of file operations during data extraction.

These changes enhance the flexibility and reliability of file service management in the data extraction process.
2026-03-03 22:23:23 +08:00
wizardchen 4192cfd072 refactor: simplify image handling in _resolve_images function
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.

These changes improve the clarity and efficiency of image processing and API interactions within the application.
2026-03-03 22:23:23 +08:00
wizardchen 931bd38138 feat(repository): add TagID and IsRecommended fields to vector embeddings
- Introduced `TagID` and `IsRecommended` fields across various repository implementations (Elasticsearch, Milvus, Postgres, Qdrant, SQLite) to enhance document categorization and recommendation capabilities.
- Updated the logic for handling `IsEnabled` to ensure it reflects the actual state from the source data.
- Adjusted the knowledge service to incorporate the new `IsRecommended` field, improving the indexing process for knowledge chunks.

These changes enhance the flexibility and functionality of document retrieval and indexing across the application.
2026-03-02 21:43:30 +08:00
wizardchen 7b8a78aa2a feat(repository): enhance document indexing with isEnabled and isRecommended fields
- Added `IsEnabled` and `IsRecommended` fields to the document indexing process in both ElasticFaiss and Elasticsearch repositories, improving the flexibility and functionality of document retrieval.
- Updated the data extraction process to include these new fields, ensuring they are correctly populated from the source data.
- Adjusted the indexing logic in the knowledge service to reflect these changes, maintaining consistency across the application.
2026-03-02 21:43:30 +08:00
wizardchen 284d6c623e feat: add deleteAllSessions functionality and update batch delete logic
- Introduced a new API function `deleteAllSessions` to allow deletion of all sessions for a tenant.
- Updated the `BatchManageDialog` component to support a confirmation dialog for deleting all sessions.
- Enhanced the menu component to reflect the new delete all functionality with appropriate confirmation messages.
- Added internationalization support for the new delete all confirmation message in multiple languages.

These changes improve session management by providing users with the ability to delete all sessions at once, enhancing the overall user experience.
2026-03-02 21:21:49 +08:00
wizardchen dbab7b0405 feat: introduce storage provider configuration and enhance backward compatibility
- Added a new `StorageProviderConfig` struct to the `KnowledgeBase` for improved storage provider management, separating provider selection from storage credentials.
- Updated the `UnmarshalJSON` method to auto-populate `StorageProviderConfig` from the legacy `StorageConfig` for backward compatibility.
- Refactored various service methods to utilize the new storage provider configuration, ensuring a smooth transition from the legacy system.
- Implemented database migrations to add the `storage_provider_config` column and migrate existing data, maintaining compatibility with historical knowledge bases.

This update enhances the clarity and flexibility of storage configurations while ensuring compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 4a9977b11f feat: enhance document loading and state management in doc-content component
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.

This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
2026-03-02 21:21:49 +08:00
wizardchen 393ebf9f92 feat: standardize storage configuration and enhance backward compatibility
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.

This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 469f320d10 feat: enhance storage engine configuration and UI updates
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.

This update enhances the flexibility of storage options and improves user experience with better UI interactions.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
wizardchen 5241dbc39e feat(security): implement path and filename validation utilities
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.

This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
2026-03-02 21:21:49 +08:00
joeyczheng 1b56e99cac feat: support milvus vectordb for knowledge retrieve
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-27 09:51:17 +08:00
begoniezhao daf7dee4d2 refactor: Unify /url route and file URL handling 2026-02-26 19:46:08 +08:00
begoniezhao e24bfb05a2 feat: Add knowledge creation support from remote file URLs 2026-02-26 19:46:08 +08:00
joeyczheng 12baa885d2 feat: implement preview of user level memory graph
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-25 14:29:54 +08:00
Dounx 6adecdb30e feat: add volcengine tos support 2026-02-25 14:24:50 +08:00
AndyYang 172bf797ee feat(chat): add batch management for conversations 2026-02-25 14:23:05 +08:00
begoniezhao bce968833e feat: Add async knowledge re-parse API and examples 2026-02-11 12:27:55 +08:00
Dounx 1b84fd5849 fix(data-analysis): load knowledge files via presigned URLs 2026-02-10 11:44:25 +08:00
wizardchen cbe66aba56 feat(api): integrate per-organization resource counts into organization listing
- Added `resource_counts` to the `ListOrganizationsResponse` to include per-organization knowledge base and agent counts, enhancing the sidebar display.
- Removed the deprecated `getMeResourceCounts` API endpoint, consolidating resource count retrieval into the organization listing.
- Updated the organization store to manage and utilize the new resource counts effectively.
- Adjusted UI components to leverage the new resource counts structure, improving data retrieval and display for agents and knowledge bases.

This update streamlines resource count management and enhances the user experience by providing relevant data in a single API call.
2026-02-09 11:40:25 +08:00
wizardchen 9a073d62bf feat(api): enhance organization shared knowledge base and agent functionalities
- Introduced new interfaces for handling shared knowledge bases and agents, including visibility and selection modes for agents.
- Added API functions to list shared knowledge bases and agents within an organization, improving data retrieval for UI components.
- Updated existing components to utilize the new API endpoints and interfaces, ensuring accurate representation of shared resources.
- Enhanced error handling in API calls to provide clearer feedback on failures.

This update improves the functionality and user experience by providing better access to shared resources and enhancing the overall data structure within the application.
2026-02-09 11:40:25 +08:00
wizardchen 4f44c12c41 feat(kbshare): enhance knowledge base share listing with tenant ownership 2026-02-09 11:40:25 +08:00
wizardchen ab1f15c97d feat(input-field): enhance knowledge base selection and UI components
- Updated the logic for cleaning invalid knowledge base IDs to include checks for shared agent knowledge bases, ensuring accurate selection retention.
- Refactored the mention chip component styles and structure for improved clarity and usability, including updated class names and enhanced visual elements.
- Introduced a new context key for session tenant ID to streamline tenant context handling in message and session services, improving the accuracy of operations in shared agent scenarios.

This update enhances the user experience by ensuring correct knowledge base selections and improving the visual representation of selected items.
2026-02-09 11:40:25 +08:00
wizardchen 03005d71fa refactor(session): streamline tenant context handling for async operations
- Updated session title generation and message completion to utilize the session's tenant ID directly, ensuring correct context is used in background operations.
- Removed unnecessary context value extraction and background context creation, simplifying the code and improving clarity.
- Enhanced the handling of stop events and message updates to consistently apply the session's tenant ID, improving the accuracy of operations in shared agent scenarios.

This update improves the reliability of async operations by ensuring the correct tenant context is consistently applied.
2026-02-09 11:40:25 +08:00
wizardchen c806eaf7f9 feat(agent): enhance agent and skill management features
- Updated the API to include disabled agent IDs for better management of user-specific agent visibility.
- Introduced a new function to set the disabled state for shared agents, allowing users to manage their shared agent preferences.
- Added a new Skills API to retrieve available skills, enhancing the overall agent capabilities.
- Improved the AgentSelector and related components to filter out disabled agents, ensuring a cleaner user experience.
- Updated localization strings to reflect new features and improve clarity in agent and skill management.

This update enhances the user experience by providing better control over agent visibility and introducing new skill management functionalities.
2026-02-09 11:40:25 +08:00
wizardchen 5ab20a9148 feat(i18n): update localization for knowledge bases and organization features
- Added new localization strings for 'Mine' in English and Chinese.
- Enhanced organization subtitles to include agents in shared spaces.
- Updated permission descriptions to clarify roles related to knowledge bases and agents.
- Improved UI components to reflect new labels and permissions for better user experience.

This update strengthens the localization support, ensuring clarity in knowledge base and organization functionalities across multiple languages.
2026-02-09 11:40:25 +08:00
wizardchen dfaf862bc0 feat(agent): enhance agent sharing and selection features
- Updated the AgentSelector component to improve agent selection logic, ensuring accurate representation of selected agents.
- Introduced new AgentShareSettings component for managing agent sharing with organizations, including UI for selecting organizations and displaying shared agents.
- Enhanced API interactions for sharing agents, including new methods for listing and managing agent shares.
- Improved localization support for agent sharing features across multiple languages.
- Updated various components to reflect changes in agent capabilities and sharing status, enhancing user experience and clarity.

This update strengthens the agent management features, providing users with better tools for sharing and selecting agents within organizations.
2026-02-09 11:40:25 +08:00
wizardchen fdb7985cde feat(organization): update default member limit to 200
- Changed the default maximum number of members per organization from 50 to 200 in both the documentation and the application service layer.
- This update allows for greater flexibility in organization membership management, accommodating larger teams.
2026-02-09 11:40:25 +08:00
wizardchen ffbb8e1b02 feat(organization): enhance organization selection and sharing UI
- Updated organization selection dropdowns to include new SVG icons for better visual representation.
- Improved organization option layout with additional metadata display, including member count and share count.
- Enhanced share item display with organization icons and improved action buttons for better user interaction.
- Adjusted styles for organization-related components to improve usability and aesthetics.

This update enhances the user experience by providing clearer organization visuals and more intuitive sharing options.
2026-02-09 11:40:25 +08:00
wizardchen b3d743a0a9 feat(kbshare): enhance sharing functionality with soft-delete handling and shared access
- Updated ListByOrganization and ListSharedKBsForUser methods to exclude soft-deleted knowledge bases and organizations, improving data integrity.
- Modified knowledge fetching methods to include documents from shared knowledge bases the user has access to, enhancing user experience.
- Improved comments for clarity on functionality and permission checks in UpdateSharePermission method.

This update strengthens the sharing capabilities and ensures users only see relevant knowledge bases, enhancing overall application usability.
2026-02-09 11:40:25 +08:00