Commit Graph

258 Commits

Author SHA1 Message Date
wizardchen 1b8d7bd4de feat: add passage cleaning functionality for rerank model
- Introduced a new function `cleanPassageForRerank` to strip markdown and structural noise from text, enhancing semantic clarity for rerank models.
- Implemented comprehensive regex patterns to remove images, links, code blocks, LaTeX, HTML tags, and other formatting elements.
- Added unit tests to validate the cleaning process across various input scenarios, ensuring expected output consistency.
- Updated the `getEnrichedPassage` function to utilize the new cleaning method, improving the quality of text passed to the rerank model.

These changes significantly enhance the preprocessing of text data, leading to better performance in reranking tasks.
2026-03-05 18:14:14 +08:00
wizardchen b333499078 feat: implement parent-child chunking strategy for enhanced context management
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.

These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
2026-03-05 18:14:14 +08:00
wizardchen e2234e4960 feat: implement dynamic file service resolution for knowledge extraction
- Added a new method `resolveFileServiceForKnowledge` to dynamically resolve the appropriate file service based on the tenant's storage configuration and the knowledge file's provider.
- Updated the `extractionResources` struct to include a reference to the tenant, enabling the new resolution logic.
- Modified the `processTableData` method to utilize the resolved file service, improving the handling of file operations during data extraction.

These changes enhance the flexibility and reliability of file service management in the data extraction process.
2026-03-03 22:23:23 +08:00
wizardchen 4192cfd072 refactor: simplify image handling in _resolve_images function
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.

These changes improve the clarity and efficiency of image processing and API interactions within the application.
2026-03-03 22:23:23 +08:00
wizardchen 931bd38138 feat(repository): add TagID and IsRecommended fields to vector embeddings
- Introduced `TagID` and `IsRecommended` fields across various repository implementations (Elasticsearch, Milvus, Postgres, Qdrant, SQLite) to enhance document categorization and recommendation capabilities.
- Updated the logic for handling `IsEnabled` to ensure it reflects the actual state from the source data.
- Adjusted the knowledge service to incorporate the new `IsRecommended` field, improving the indexing process for knowledge chunks.

These changes enhance the flexibility and functionality of document retrieval and indexing across the application.
2026-03-02 21:43:30 +08:00
wizardchen 7b8a78aa2a feat(repository): enhance document indexing with isEnabled and isRecommended fields
- Added `IsEnabled` and `IsRecommended` fields to the document indexing process in both ElasticFaiss and Elasticsearch repositories, improving the flexibility and functionality of document retrieval.
- Updated the data extraction process to include these new fields, ensuring they are correctly populated from the source data.
- Adjusted the indexing logic in the knowledge service to reflect these changes, maintaining consistency across the application.
2026-03-02 21:43:30 +08:00
wizardchen 284d6c623e feat: add deleteAllSessions functionality and update batch delete logic
- Introduced a new API function `deleteAllSessions` to allow deletion of all sessions for a tenant.
- Updated the `BatchManageDialog` component to support a confirmation dialog for deleting all sessions.
- Enhanced the menu component to reflect the new delete all functionality with appropriate confirmation messages.
- Added internationalization support for the new delete all confirmation message in multiple languages.

These changes improve session management by providing users with the ability to delete all sessions at once, enhancing the overall user experience.
2026-03-02 21:21:49 +08:00
wizardchen dbab7b0405 feat: introduce storage provider configuration and enhance backward compatibility
- Added a new `StorageProviderConfig` struct to the `KnowledgeBase` for improved storage provider management, separating provider selection from storage credentials.
- Updated the `UnmarshalJSON` method to auto-populate `StorageProviderConfig` from the legacy `StorageConfig` for backward compatibility.
- Refactored various service methods to utilize the new storage provider configuration, ensuring a smooth transition from the legacy system.
- Implemented database migrations to add the `storage_provider_config` column and migrate existing data, maintaining compatibility with historical knowledge bases.

This update enhances the clarity and flexibility of storage configurations while ensuring compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 4a9977b11f feat: enhance document loading and state management in doc-content component
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.

This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
2026-03-02 21:21:49 +08:00
wizardchen 393ebf9f92 feat: standardize storage configuration and enhance backward compatibility
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.

This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 469f320d10 feat: enhance storage engine configuration and UI updates
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.

This update enhances the flexibility of storage options and improves user experience with better UI interactions.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
wizardchen 5241dbc39e feat(security): implement path and filename validation utilities
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.

This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
2026-03-02 21:21:49 +08:00
joeyczheng 1b56e99cac feat: support milvus vectordb for knowledge retrieve
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-27 09:51:17 +08:00
begoniezhao daf7dee4d2 refactor: Unify /url route and file URL handling 2026-02-26 19:46:08 +08:00
begoniezhao e24bfb05a2 feat: Add knowledge creation support from remote file URLs 2026-02-26 19:46:08 +08:00
joeyczheng 12baa885d2 feat: implement preview of user level memory graph
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-25 14:29:54 +08:00
Dounx 6adecdb30e feat: add volcengine tos support 2026-02-25 14:24:50 +08:00
AndyYang 172bf797ee feat(chat): add batch management for conversations 2026-02-25 14:23:05 +08:00
begoniezhao bce968833e feat: Add async knowledge re-parse API and examples 2026-02-11 12:27:55 +08:00
Dounx 1b84fd5849 fix(data-analysis): load knowledge files via presigned URLs 2026-02-10 11:44:25 +08:00
wizardchen cbe66aba56 feat(api): integrate per-organization resource counts into organization listing
- Added `resource_counts` to the `ListOrganizationsResponse` to include per-organization knowledge base and agent counts, enhancing the sidebar display.
- Removed the deprecated `getMeResourceCounts` API endpoint, consolidating resource count retrieval into the organization listing.
- Updated the organization store to manage and utilize the new resource counts effectively.
- Adjusted UI components to leverage the new resource counts structure, improving data retrieval and display for agents and knowledge bases.

This update streamlines resource count management and enhances the user experience by providing relevant data in a single API call.
2026-02-09 11:40:25 +08:00
wizardchen 9a073d62bf feat(api): enhance organization shared knowledge base and agent functionalities
- Introduced new interfaces for handling shared knowledge bases and agents, including visibility and selection modes for agents.
- Added API functions to list shared knowledge bases and agents within an organization, improving data retrieval for UI components.
- Updated existing components to utilize the new API endpoints and interfaces, ensuring accurate representation of shared resources.
- Enhanced error handling in API calls to provide clearer feedback on failures.

This update improves the functionality and user experience by providing better access to shared resources and enhancing the overall data structure within the application.
2026-02-09 11:40:25 +08:00
wizardchen 4f44c12c41 feat(kbshare): enhance knowledge base share listing with tenant ownership 2026-02-09 11:40:25 +08:00
wizardchen ab1f15c97d feat(input-field): enhance knowledge base selection and UI components
- Updated the logic for cleaning invalid knowledge base IDs to include checks for shared agent knowledge bases, ensuring accurate selection retention.
- Refactored the mention chip component styles and structure for improved clarity and usability, including updated class names and enhanced visual elements.
- Introduced a new context key for session tenant ID to streamline tenant context handling in message and session services, improving the accuracy of operations in shared agent scenarios.

This update enhances the user experience by ensuring correct knowledge base selections and improving the visual representation of selected items.
2026-02-09 11:40:25 +08:00
wizardchen 03005d71fa refactor(session): streamline tenant context handling for async operations
- Updated session title generation and message completion to utilize the session's tenant ID directly, ensuring correct context is used in background operations.
- Removed unnecessary context value extraction and background context creation, simplifying the code and improving clarity.
- Enhanced the handling of stop events and message updates to consistently apply the session's tenant ID, improving the accuracy of operations in shared agent scenarios.

This update improves the reliability of async operations by ensuring the correct tenant context is consistently applied.
2026-02-09 11:40:25 +08:00
wizardchen c806eaf7f9 feat(agent): enhance agent and skill management features
- Updated the API to include disabled agent IDs for better management of user-specific agent visibility.
- Introduced a new function to set the disabled state for shared agents, allowing users to manage their shared agent preferences.
- Added a new Skills API to retrieve available skills, enhancing the overall agent capabilities.
- Improved the AgentSelector and related components to filter out disabled agents, ensuring a cleaner user experience.
- Updated localization strings to reflect new features and improve clarity in agent and skill management.

This update enhances the user experience by providing better control over agent visibility and introducing new skill management functionalities.
2026-02-09 11:40:25 +08:00
wizardchen 5ab20a9148 feat(i18n): update localization for knowledge bases and organization features
- Added new localization strings for 'Mine' in English and Chinese.
- Enhanced organization subtitles to include agents in shared spaces.
- Updated permission descriptions to clarify roles related to knowledge bases and agents.
- Improved UI components to reflect new labels and permissions for better user experience.

This update strengthens the localization support, ensuring clarity in knowledge base and organization functionalities across multiple languages.
2026-02-09 11:40:25 +08:00
wizardchen dfaf862bc0 feat(agent): enhance agent sharing and selection features
- Updated the AgentSelector component to improve agent selection logic, ensuring accurate representation of selected agents.
- Introduced new AgentShareSettings component for managing agent sharing with organizations, including UI for selecting organizations and displaying shared agents.
- Enhanced API interactions for sharing agents, including new methods for listing and managing agent shares.
- Improved localization support for agent sharing features across multiple languages.
- Updated various components to reflect changes in agent capabilities and sharing status, enhancing user experience and clarity.

This update strengthens the agent management features, providing users with better tools for sharing and selecting agents within organizations.
2026-02-09 11:40:25 +08:00
wizardchen fdb7985cde feat(organization): update default member limit to 200
- Changed the default maximum number of members per organization from 50 to 200 in both the documentation and the application service layer.
- This update allows for greater flexibility in organization membership management, accommodating larger teams.
2026-02-09 11:40:25 +08:00
wizardchen ffbb8e1b02 feat(organization): enhance organization selection and sharing UI
- Updated organization selection dropdowns to include new SVG icons for better visual representation.
- Improved organization option layout with additional metadata display, including member count and share count.
- Enhanced share item display with organization icons and improved action buttons for better user interaction.
- Adjusted styles for organization-related components to improve usability and aesthetics.

This update enhances the user experience by providing clearer organization visuals and more intuitive sharing options.
2026-02-09 11:40:25 +08:00
wizardchen b3d743a0a9 feat(kbshare): enhance sharing functionality with soft-delete handling and shared access
- Updated ListByOrganization and ListSharedKBsForUser methods to exclude soft-deleted knowledge bases and organizations, improving data integrity.
- Modified knowledge fetching methods to include documents from shared knowledge bases the user has access to, enhancing user experience.
- Improved comments for clarity on functionality and permission checks in UpdateSharePermission method.

This update strengthens the sharing capabilities and ensures users only see relevant knowledge bases, enhancing overall application usability.
2026-02-09 11:40:25 +08:00
wizardchen 342e56735f feat(avatar): enhance SpaceAvatar component with emoji support
- Updated SpaceAvatar component to allow emoji-based avatars, enabling users to select emojis as their space avatar.
- Added new props and computed properties to handle emoji logic and display.
- Enhanced UI to include an emoji picker and clear option for avatars.
- Updated localization strings in English and Chinese for new avatar features.
- Adjusted styles for emoji display and interaction.

This update improves user personalization options for avatars, enhancing the overall user experience in the application.
2026-02-09 11:40:25 +08:00
wizardchen 72173432c9 feat(organization): enhance searchable organizations with member limit and search by ID
- Added a member limit field to the SearchableOrganizationItem interface, allowing organizations to specify a maximum number of members.
- Updated API and service layers to support searching organizations by name, description, or ID.
- Enhanced UI components to display member limits and updated localization strings in English and Chinese for new features.
- Included migration files to add the searchable column to the organizations table.

This update improves the organization search functionality and membership management, enhancing user experience and administrative control.
2026-02-09 11:40:25 +08:00
wizardchen ca1b3da1df feat(organization): add member limit feature
- Introduced a new member limit feature for organizations, allowing a maximum number of members to be set (default is 50; 0 means unlimited).
- Updated API endpoints to handle member limit during organization creation and updates.
- Implemented validation to prevent adding members when the limit is reached.
- Enhanced UI components to display and manage member limits, including localization updates for English and Chinese.
- Added migration files to support the new member limit column in the organizations table.

This update improves organization management by allowing administrators to control membership size.
2026-02-09 11:40:25 +08:00
wizardchen 2818a4fe8f feat(organization): implement searchable organizations feature
- Added a 'searchable' flag to organizations, allowing them to be discoverable and joinable by ID without an invite code.
- Introduced API endpoints for searching and joining searchable organizations.
- Updated UI components to support searching for organizations and displaying results.
- Enhanced localization for new search-related features in both English and Chinese.

This update improves user experience by enabling easier discovery and joining of organizations.
2026-02-09 11:40:25 +08:00
wizardchen 4158773849 feat: add organization management features
- Introduced a new API for organization management, including creation, updating, and deletion of organizations.
- Implemented organization sharing functionality for knowledge bases, allowing users to share knowledge bases with specific organizations and set permissions.
- Added UI components for organization selection and sharing dialogs.
- Enhanced existing components to support organization-related features, including shared knowledge base handling and organization role management.

This update enhances collaboration capabilities within the application by allowing users to manage and share knowledge bases across different organizations.
2026-02-09 11:40:25 +08:00
wizardchen 602086d884 feat: implement tenant isolation for knowledge base retrieval
- Added GetKnowledgeBaseByIDAndTenant method to enforce tenant isolation when retrieving knowledge bases by ID.
- Updated CopyKnowledgeBase method to use tenant-scoped retrieval, preventing cross-tenant cloning.
- Enhanced error handling in the KnowledgeBaseHandler to validate tenant ownership for source and target knowledge bases during copy operations.
2026-02-06 19:55:29 +08:00
wizardchen 2d6efec84f feat: enhance agent skills sandbox configuration and availability
- Updated .env.example to set default sandbox mode to 'docker' and added timeout and docker image variables.
- Modified docker-compose files to include a sandbox service for building and pulling the sandbox image.
- Adjusted frontend API to reflect sandbox availability for skills, ensuring UI elements are conditionally displayed based on sandbox status.
- Implemented backend logic to disable skills when the sandbox is not enabled, improving error handling and user experience.
2026-02-04 20:58:24 +08:00
nullkey 03446f35d3 feat: adjust sandbox setting of agentConfig 2026-02-04 20:08:49 +08:00
nullkey 43a2c647ea feat: optimize security and deployment of agent skills 2026-02-04 20:08:49 +08:00
nullkey 3f21345546 feat: adapt preloaded skills to smart-reasoning agent 2026-02-04 20:08:49 +08:00
nullkey 33f7e7da66 feat: support agent skills
feat:support agent skills
2026-02-04 20:08:49 +08:00
joeyczheng 9c57e11f63 feat: add bing search provider
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-02 20:53:22 +08:00
begoniezhao 5f7c23a1b9 refactor: Refactor rune handling for correct chunk merging 2026-02-02 16:47:35 +08:00
wizardchen 14ad38fd83 fix: tighten SSRF protection for URL imports 2026-01-27 11:09:53 +08:00
wizardchen faf1c9a37b feat: add matched question field to FAQ search results 2026-01-26 16:32:18 +08:00
wizardchen 7d272cc92c feat: add retrieve KB only when mentioned option 2026-01-26 10:44:09 +08:00
nullkey de06bb24d3 fix: cleanup conversation context stored in Redis 2026-01-23 10:33:01 +08:00
wizardchen 2a99e2458d feat: add SSRF protection for URL fetching 2026-01-21 19:13:14 +08:00