Commit Graph

397 Commits

Author SHA1 Message Date
wizardchen 1b8d7bd4de feat: add passage cleaning functionality for rerank model
- Introduced a new function `cleanPassageForRerank` to strip markdown and structural noise from text, enhancing semantic clarity for rerank models.
- Implemented comprehensive regex patterns to remove images, links, code blocks, LaTeX, HTML tags, and other formatting elements.
- Added unit tests to validate the cleaning process across various input scenarios, ensuring expected output consistency.
- Updated the `getEnrichedPassage` function to utilize the new cleaning method, improving the quality of text passed to the rerank model.

These changes significantly enhance the preprocessing of text data, leading to better performance in reranking tasks.
2026-03-05 18:14:14 +08:00
wizardchen b333499078 feat: implement parent-child chunking strategy for enhanced context management
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.

These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
2026-03-05 18:14:14 +08:00
hylaz 8399527022 fix: cleanupCtx 在启动时创建,可能在 shutdown 前过期 2026-03-05 11:17:51 +08:00
wizardchen 58783ec96c feat: implement automatic soft-delete filtering in database queries
- Added automatic filtering for `deleted_at IS NULL` in database queries to ensure only active records are returned.
- Updated the `database_query.go` documentation to reflect the new soft-delete feature and its implications for query construction.
- Enhanced the SQL validation process to include soft-delete conditions, improving data integrity and security.

These changes enhance the robustness of database interactions by enforcing soft-delete logic across relevant queries.
2026-03-03 22:23:23 +08:00
wizardchen e2234e4960 feat: implement dynamic file service resolution for knowledge extraction
- Added a new method `resolveFileServiceForKnowledge` to dynamically resolve the appropriate file service based on the tenant's storage configuration and the knowledge file's provider.
- Updated the `extractionResources` struct to include a reference to the tenant, enabling the new resolution logic.
- Modified the `processTableData` method to utilize the resolved file service, improving the handling of file operations during data extraction.

These changes enhance the flexibility and reliability of file service management in the data extraction process.
2026-03-03 22:23:23 +08:00
wizardchen 4192cfd072 refactor: simplify image handling in _resolve_images function
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.

These changes improve the clarity and efficiency of image processing and API interactions within the application.
2026-03-03 22:23:23 +08:00
wizardchen 894d2c060a refactor: remove mineru_api_base_url from ParserEngineConfig and related components
- Eliminated the `mineru_api_base_url` field from the `ParserEngineConfig` interface and its usage across the application, simplifying the configuration structure.
- Updated the `ParserEngineSettings` view to remove the corresponding input field, enhancing the user interface.
- Adjusted the `PingMinerUCloud` function to no longer require the base URL parameter, defaulting to a predefined value instead.
- Modified database migration comments to reflect the removal of the base URL from the configuration.

These changes streamline the configuration process for the MinerU Cloud integration, improving clarity and reducing potential misconfigurations.
2026-03-03 13:24:20 +08:00
wizardchen 931bd38138 feat(repository): add TagID and IsRecommended fields to vector embeddings
- Introduced `TagID` and `IsRecommended` fields across various repository implementations (Elasticsearch, Milvus, Postgres, Qdrant, SQLite) to enhance document categorization and recommendation capabilities.
- Updated the logic for handling `IsEnabled` to ensure it reflects the actual state from the source data.
- Adjusted the knowledge service to incorporate the new `IsRecommended` field, improving the indexing process for knowledge chunks.

These changes enhance the flexibility and functionality of document retrieval and indexing across the application.
2026-03-02 21:43:30 +08:00
wizardchen 7b8a78aa2a feat(repository): enhance document indexing with isEnabled and isRecommended fields
- Added `IsEnabled` and `IsRecommended` fields to the document indexing process in both ElasticFaiss and Elasticsearch repositories, improving the flexibility and functionality of document retrieval.
- Updated the data extraction process to include these new fields, ensuring they are correctly populated from the source data.
- Adjusted the indexing logic in the knowledge service to reflect these changes, maintaining consistency across the application.
2026-03-02 21:43:30 +08:00
wizardchen b1fa96c175 refactor: enhance SSRF safety checks in MinerUCloudReader
- Removed hardcoded IP address from blocked hosts in `system.go`.
- Integrated SSRF safety checks in `MinerUCloudReader` for base URL and zip URL validation.
- Updated HTTP client initialization to use a new utility function for SSRF-safe HTTP clients, improving security during API requests.

These changes enhance the security of the MinerUCloudReader by ensuring that potentially unsafe URLs are properly validated before use.
2026-03-02 21:21:49 +08:00
wizardchen 284d6c623e feat: add deleteAllSessions functionality and update batch delete logic
- Introduced a new API function `deleteAllSessions` to allow deletion of all sessions for a tenant.
- Updated the `BatchManageDialog` component to support a confirmation dialog for deleting all sessions.
- Enhanced the menu component to reflect the new delete all functionality with appropriate confirmation messages.
- Added internationalization support for the new delete all confirmation message in multiple languages.

These changes improve session management by providing users with the ability to delete all sessions at once, enhancing the overall user experience.
2026-03-02 21:21:49 +08:00
wizardchen 4222b9dc21 fix: handle empty metadata case in Knowledge struct
- Updated the GetMetadata method to return an empty map when the Metadata field is empty, preventing potential errors during metadata retrieval.

This change improves the robustness of the Knowledge struct by ensuring it handles empty metadata gracefully.
2026-03-02 21:21:49 +08:00
wizardchen e595e9345e feat: implement tenant access authorization in TenantHandler
- Added `authorizeTenantAccess` method to check user permissions for accessing specific tenants.
- Integrated authorization checks in `GetTenant`, `UpdateTenant`, and `DeleteTenant` methods to enforce access control.
- Updated `ListTenants` method to ensure proper authentication before returning tenant data.

This update enhances security by ensuring that only authorized users can access or modify tenant information.
2026-03-02 21:21:49 +08:00
wizardchen aefa1c6fe8 feat: enhance system information display with database version
- Added `db_version` field to the `SystemInfo` interface to expose the current database migration version.
- Updated the system information response to include the database version, reflecting its state during application runtime.
- Enhanced the UI in the SystemInfo component to display the database version with appropriate labels and descriptions in multiple languages.

This update improves transparency regarding the database state within the system information settings.
2026-03-02 21:21:49 +08:00
wizardchen dbab7b0405 feat: introduce storage provider configuration and enhance backward compatibility
- Added a new `StorageProviderConfig` struct to the `KnowledgeBase` for improved storage provider management, separating provider selection from storage credentials.
- Updated the `UnmarshalJSON` method to auto-populate `StorageProviderConfig` from the legacy `StorageConfig` for backward compatibility.
- Refactored various service methods to utilize the new storage provider configuration, ensuring a smooth transition from the legacy system.
- Implemented database migrations to add the `storage_provider_config` column and migrate existing data, maintaining compatibility with historical knowledge bases.

This update enhances the clarity and flexibility of storage configurations while ensuring compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 565660dc46 feat: improve document chunk loading and error handling in doc-content component
- Added state management for pending document requests to enhance user experience during chunk loading.
- Implemented error handling for chunk loading failures, providing user feedback through warnings.
- Adjusted scroll handling logic to ensure smoother pagination and prevent multiple concurrent requests.

This update enhances the reliability of document loading and improves user interaction during scrolling.
2026-03-02 21:21:49 +08:00
wizardchen 4a9977b11f feat: enhance document loading and state management in doc-content component
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.

This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
2026-03-02 21:21:49 +08:00
wizardchen 393ebf9f92 feat: standardize storage configuration and enhance backward compatibility
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.

This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 469f320d10 feat: enhance storage engine configuration and UI updates
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.

This update enhances the flexibility of storage options and improves user experience with better UI interactions.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
wizardchen 5241dbc39e feat(security): implement path and filename validation utilities
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.

This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
2026-03-02 21:21:49 +08:00
ChenRussell 48cf6aeaea fix(swagger): resolve interface documentation generation error and regenerate docs
- Fix swagger documentation generation failure issue
- Regenerate swagger interface documentation
2026-02-27 09:52:07 +08:00
joeyczheng 1b56e99cac feat: support milvus vectordb for knowledge retrieve
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-27 09:51:17 +08:00
begoniezhao daf7dee4d2 refactor: Unify /url route and file URL handling 2026-02-26 19:46:08 +08:00
begoniezhao e24bfb05a2 feat: Add knowledge creation support from remote file URLs 2026-02-26 19:46:08 +08:00
joeyczheng 12baa885d2 feat: implement preview of user level memory graph
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-25 14:29:54 +08:00
Dounx 6adecdb30e feat: add volcengine tos support 2026-02-25 14:24:50 +08:00
AndyYang 172bf797ee feat(chat): add batch management for conversations 2026-02-25 14:23:05 +08:00
xiaoen b536e86cd0 fix: handle thinking content in Ollama chat responses
Ollama API returns reasoning model output in Message.Thinking field,
but OllamaChat only reads Message.Content. Add thinking content
handling for both streaming and non-streaming, consistent with
RemoteAPIChat behavior.
2026-02-25 11:11:24 +08:00
begoniezhao bce968833e feat: Add async knowledge re-parse API and examples 2026-02-11 12:27:55 +08:00
Dounx bd76bc592e fix: duckdb spatial extension not found error 2026-02-10 11:45:54 +08:00
Dounx 1b84fd5849 fix(data-analysis): load knowledge files via presigned URLs 2026-02-10 11:44:25 +08:00
CodeMarker 6a4a191318 perf: Use swap deletion 2026-02-09 14:13:29 +08:00
CodeMarker 6beb9685fc perf: cache tokens instead of recomputing each time 2026-02-09 14:13:29 +08:00
wizardchen cbe66aba56 feat(api): integrate per-organization resource counts into organization listing
- Added `resource_counts` to the `ListOrganizationsResponse` to include per-organization knowledge base and agent counts, enhancing the sidebar display.
- Removed the deprecated `getMeResourceCounts` API endpoint, consolidating resource count retrieval into the organization listing.
- Updated the organization store to manage and utilize the new resource counts effectively.
- Adjusted UI components to leverage the new resource counts structure, improving data retrieval and display for agents and knowledge bases.

This update streamlines resource count management and enhances the user experience by providing relevant data in a single API call.
2026-02-09 11:40:25 +08:00
wizardchen 9a073d62bf feat(api): enhance organization shared knowledge base and agent functionalities
- Introduced new interfaces for handling shared knowledge bases and agents, including visibility and selection modes for agents.
- Added API functions to list shared knowledge bases and agents within an organization, improving data retrieval for UI components.
- Updated existing components to utilize the new API endpoints and interfaces, ensuring accurate representation of shared resources.
- Enhanced error handling in API calls to provide clearer feedback on failures.

This update improves the functionality and user experience by providing better access to shared resources and enhancing the overall data structure within the application.
2026-02-09 11:40:25 +08:00
wizardchen 1bdd9a4693 feat(ui): unify card styles and improve layout consistency across agent, knowledge base, and organization components
- Standardized card dimensions and padding across AgentList, KnowledgeBaseList, and OrganizationList for a cohesive appearance.
- Adjusted font sizes and line heights in card descriptions to enhance readability and maintain uniformity.
- Enhanced card content layout with flex properties to improve visual structure and spacing.

This update improves the user interface by ensuring consistent styling and layout across multiple components, enhancing the overall user experience.
2026-02-09 11:40:25 +08:00
wizardchen 4f44c12c41 feat(kbshare): enhance knowledge base share listing with tenant ownership 2026-02-09 11:40:25 +08:00
wizardchen ab1f15c97d feat(input-field): enhance knowledge base selection and UI components
- Updated the logic for cleaning invalid knowledge base IDs to include checks for shared agent knowledge bases, ensuring accurate selection retention.
- Refactored the mention chip component styles and structure for improved clarity and usability, including updated class names and enhanced visual elements.
- Introduced a new context key for session tenant ID to streamline tenant context handling in message and session services, improving the accuracy of operations in shared agent scenarios.

This update enhances the user experience by ensuring correct knowledge base selections and improving the visual representation of selected items.
2026-02-09 11:40:25 +08:00
wizardchen 03005d71fa refactor(session): streamline tenant context handling for async operations
- Updated session title generation and message completion to utilize the session's tenant ID directly, ensuring correct context is used in background operations.
- Removed unnecessary context value extraction and background context creation, simplifying the code and improving clarity.
- Enhanced the handling of stop events and message updates to consistently apply the session's tenant ID, improving the accuracy of operations in shared agent scenarios.

This update improves the reliability of async operations by ensuring the correct tenant context is consistently applied.
2026-02-09 11:40:25 +08:00
wizardchen c806eaf7f9 feat(agent): enhance agent and skill management features
- Updated the API to include disabled agent IDs for better management of user-specific agent visibility.
- Introduced a new function to set the disabled state for shared agents, allowing users to manage their shared agent preferences.
- Added a new Skills API to retrieve available skills, enhancing the overall agent capabilities.
- Improved the AgentSelector and related components to filter out disabled agents, ensuring a cleaner user experience.
- Updated localization strings to reflect new features and improve clarity in agent and skill management.

This update enhances the user experience by providing better control over agent visibility and introducing new skill management functionalities.
2026-02-09 11:40:25 +08:00
wizardchen 5ab20a9148 feat(i18n): update localization for knowledge bases and organization features
- Added new localization strings for 'Mine' in English and Chinese.
- Enhanced organization subtitles to include agents in shared spaces.
- Updated permission descriptions to clarify roles related to knowledge bases and agents.
- Improved UI components to reflect new labels and permissions for better user experience.

This update strengthens the localization support, ensuring clarity in knowledge base and organization functionalities across multiple languages.
2026-02-09 11:40:25 +08:00
wizardchen dfaf862bc0 feat(agent): enhance agent sharing and selection features
- Updated the AgentSelector component to improve agent selection logic, ensuring accurate representation of selected agents.
- Introduced new AgentShareSettings component for managing agent sharing with organizations, including UI for selecting organizations and displaying shared agents.
- Enhanced API interactions for sharing agents, including new methods for listing and managing agent shares.
- Improved localization support for agent sharing features across multiple languages.
- Updated various components to reflect changes in agent capabilities and sharing status, enhancing user experience and clarity.

This update strengthens the agent management features, providing users with better tools for sharing and selecting agents within organizations.
2026-02-09 11:40:25 +08:00
wizardchen fdb7985cde feat(organization): update default member limit to 200
- Changed the default maximum number of members per organization from 50 to 200 in both the documentation and the application service layer.
- This update allows for greater flexibility in organization membership management, accommodating larger teams.
2026-02-09 11:40:25 +08:00
wizardchen ffbb8e1b02 feat(organization): enhance organization selection and sharing UI
- Updated organization selection dropdowns to include new SVG icons for better visual representation.
- Improved organization option layout with additional metadata display, including member count and share count.
- Enhanced share item display with organization icons and improved action buttons for better user interaction.
- Adjusted styles for organization-related components to improve usability and aesthetics.

This update enhances the user experience by providing clearer organization visuals and more intuitive sharing options.
2026-02-09 11:40:25 +08:00
wizardchen b3d743a0a9 feat(kbshare): enhance sharing functionality with soft-delete handling and shared access
- Updated ListByOrganization and ListSharedKBsForUser methods to exclude soft-deleted knowledge bases and organizations, improving data integrity.
- Modified knowledge fetching methods to include documents from shared knowledge bases the user has access to, enhancing user experience.
- Improved comments for clarity on functionality and permission checks in UpdateSharePermission method.

This update strengthens the sharing capabilities and ensures users only see relevant knowledge bases, enhancing overall application usability.
2026-02-09 11:40:25 +08:00
wizardchen 342e56735f feat(avatar): enhance SpaceAvatar component with emoji support
- Updated SpaceAvatar component to allow emoji-based avatars, enabling users to select emojis as their space avatar.
- Added new props and computed properties to handle emoji logic and display.
- Enhanced UI to include an emoji picker and clear option for avatars.
- Updated localization strings in English and Chinese for new avatar features.
- Adjusted styles for emoji display and interaction.

This update improves user personalization options for avatars, enhancing the overall user experience in the application.
2026-02-09 11:40:25 +08:00
wizardchen 72173432c9 feat(organization): enhance searchable organizations with member limit and search by ID
- Added a member limit field to the SearchableOrganizationItem interface, allowing organizations to specify a maximum number of members.
- Updated API and service layers to support searching organizations by name, description, or ID.
- Enhanced UI components to display member limits and updated localization strings in English and Chinese for new features.
- Included migration files to add the searchable column to the organizations table.

This update improves the organization search functionality and membership management, enhancing user experience and administrative control.
2026-02-09 11:40:25 +08:00
wizardchen ca1b3da1df feat(organization): add member limit feature
- Introduced a new member limit feature for organizations, allowing a maximum number of members to be set (default is 50; 0 means unlimited).
- Updated API endpoints to handle member limit during organization creation and updates.
- Implemented validation to prevent adding members when the limit is reached.
- Enhanced UI components to display and manage member limits, including localization updates for English and Chinese.
- Added migration files to support the new member limit column in the organizations table.

This update improves organization management by allowing administrators to control membership size.
2026-02-09 11:40:25 +08:00
wizardchen 2818a4fe8f feat(organization): implement searchable organizations feature
- Added a 'searchable' flag to organizations, allowing them to be discoverable and joinable by ID without an invite code.
- Introduced API endpoints for searching and joining searchable organizations.
- Updated UI components to support searching for organizations and displaying results.
- Enhanced localization for new search-related features in both English and Chinese.

This update improves user experience by enabling easier discovery and joining of organizations.
2026-02-09 11:40:25 +08:00