- Introduced a new function `cleanPassageForRerank` to strip markdown and structural noise from text, enhancing semantic clarity for rerank models.
- Implemented comprehensive regex patterns to remove images, links, code blocks, LaTeX, HTML tags, and other formatting elements.
- Added unit tests to validate the cleaning process across various input scenarios, ensuring expected output consistency.
- Updated the `getEnrichedPassage` function to utilize the new cleaning method, improving the quality of text passed to the rerank model.
These changes significantly enhance the preprocessing of text data, leading to better performance in reranking tasks.
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.
These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
- Added automatic filtering for `deleted_at IS NULL` in database queries to ensure only active records are returned.
- Updated the `database_query.go` documentation to reflect the new soft-delete feature and its implications for query construction.
- Enhanced the SQL validation process to include soft-delete conditions, improving data integrity and security.
These changes enhance the robustness of database interactions by enforcing soft-delete logic across relevant queries.
- Added a new method `resolveFileServiceForKnowledge` to dynamically resolve the appropriate file service based on the tenant's storage configuration and the knowledge file's provider.
- Updated the `extractionResources` struct to include a reference to the tenant, enabling the new resolution logic.
- Modified the `processTableData` method to utilize the resolved file service, improving the handling of file operations during data extraction.
These changes enhance the flexibility and reliability of file service management in the data extraction process.
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.
These changes improve the clarity and efficiency of image processing and API interactions within the application.
- Eliminated the `mineru_api_base_url` field from the `ParserEngineConfig` interface and its usage across the application, simplifying the configuration structure.
- Updated the `ParserEngineSettings` view to remove the corresponding input field, enhancing the user interface.
- Adjusted the `PingMinerUCloud` function to no longer require the base URL parameter, defaulting to a predefined value instead.
- Modified database migration comments to reflect the removal of the base URL from the configuration.
These changes streamline the configuration process for the MinerU Cloud integration, improving clarity and reducing potential misconfigurations.
- Introduced `TagID` and `IsRecommended` fields across various repository implementations (Elasticsearch, Milvus, Postgres, Qdrant, SQLite) to enhance document categorization and recommendation capabilities.
- Updated the logic for handling `IsEnabled` to ensure it reflects the actual state from the source data.
- Adjusted the knowledge service to incorporate the new `IsRecommended` field, improving the indexing process for knowledge chunks.
These changes enhance the flexibility and functionality of document retrieval and indexing across the application.
- Added `IsEnabled` and `IsRecommended` fields to the document indexing process in both ElasticFaiss and Elasticsearch repositories, improving the flexibility and functionality of document retrieval.
- Updated the data extraction process to include these new fields, ensuring they are correctly populated from the source data.
- Adjusted the indexing logic in the knowledge service to reflect these changes, maintaining consistency across the application.
- Removed hardcoded IP address from blocked hosts in `system.go`.
- Integrated SSRF safety checks in `MinerUCloudReader` for base URL and zip URL validation.
- Updated HTTP client initialization to use a new utility function for SSRF-safe HTTP clients, improving security during API requests.
These changes enhance the security of the MinerUCloudReader by ensuring that potentially unsafe URLs are properly validated before use.
- Introduced a new API function `deleteAllSessions` to allow deletion of all sessions for a tenant.
- Updated the `BatchManageDialog` component to support a confirmation dialog for deleting all sessions.
- Enhanced the menu component to reflect the new delete all functionality with appropriate confirmation messages.
- Added internationalization support for the new delete all confirmation message in multiple languages.
These changes improve session management by providing users with the ability to delete all sessions at once, enhancing the overall user experience.
- Updated the GetMetadata method to return an empty map when the Metadata field is empty, preventing potential errors during metadata retrieval.
This change improves the robustness of the Knowledge struct by ensuring it handles empty metadata gracefully.
- Added `authorizeTenantAccess` method to check user permissions for accessing specific tenants.
- Integrated authorization checks in `GetTenant`, `UpdateTenant`, and `DeleteTenant` methods to enforce access control.
- Updated `ListTenants` method to ensure proper authentication before returning tenant data.
This update enhances security by ensuring that only authorized users can access or modify tenant information.
- Added `db_version` field to the `SystemInfo` interface to expose the current database migration version.
- Updated the system information response to include the database version, reflecting its state during application runtime.
- Enhanced the UI in the SystemInfo component to display the database version with appropriate labels and descriptions in multiple languages.
This update improves transparency regarding the database state within the system information settings.
- Added a new `StorageProviderConfig` struct to the `KnowledgeBase` for improved storage provider management, separating provider selection from storage credentials.
- Updated the `UnmarshalJSON` method to auto-populate `StorageProviderConfig` from the legacy `StorageConfig` for backward compatibility.
- Refactored various service methods to utilize the new storage provider configuration, ensuring a smooth transition from the legacy system.
- Implemented database migrations to add the `storage_provider_config` column and migrate existing data, maintaining compatibility with historical knowledge bases.
This update enhances the clarity and flexibility of storage configurations while ensuring compatibility with existing data structures.
- Added state management for pending document requests to enhance user experience during chunk loading.
- Implemented error handling for chunk loading failures, providing user feedback through warnings.
- Adjusted scroll handling logic to ensure smoother pagination and prevent multiple concurrent requests.
This update enhances the reliability of document loading and improves user interaction during scrolling.
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.
This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.
This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.
This update enhances the flexibility of storage options and improves user experience with better UI interactions.
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.
This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.
This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
Ollama API returns reasoning model output in Message.Thinking field,
but OllamaChat only reads Message.Content. Add thinking content
handling for both streaming and non-streaming, consistent with
RemoteAPIChat behavior.
- Added `resource_counts` to the `ListOrganizationsResponse` to include per-organization knowledge base and agent counts, enhancing the sidebar display.
- Removed the deprecated `getMeResourceCounts` API endpoint, consolidating resource count retrieval into the organization listing.
- Updated the organization store to manage and utilize the new resource counts effectively.
- Adjusted UI components to leverage the new resource counts structure, improving data retrieval and display for agents and knowledge bases.
This update streamlines resource count management and enhances the user experience by providing relevant data in a single API call.
- Introduced new interfaces for handling shared knowledge bases and agents, including visibility and selection modes for agents.
- Added API functions to list shared knowledge bases and agents within an organization, improving data retrieval for UI components.
- Updated existing components to utilize the new API endpoints and interfaces, ensuring accurate representation of shared resources.
- Enhanced error handling in API calls to provide clearer feedback on failures.
This update improves the functionality and user experience by providing better access to shared resources and enhancing the overall data structure within the application.
- Standardized card dimensions and padding across AgentList, KnowledgeBaseList, and OrganizationList for a cohesive appearance.
- Adjusted font sizes and line heights in card descriptions to enhance readability and maintain uniformity.
- Enhanced card content layout with flex properties to improve visual structure and spacing.
This update improves the user interface by ensuring consistent styling and layout across multiple components, enhancing the overall user experience.
- Updated the logic for cleaning invalid knowledge base IDs to include checks for shared agent knowledge bases, ensuring accurate selection retention.
- Refactored the mention chip component styles and structure for improved clarity and usability, including updated class names and enhanced visual elements.
- Introduced a new context key for session tenant ID to streamline tenant context handling in message and session services, improving the accuracy of operations in shared agent scenarios.
This update enhances the user experience by ensuring correct knowledge base selections and improving the visual representation of selected items.
- Updated session title generation and message completion to utilize the session's tenant ID directly, ensuring correct context is used in background operations.
- Removed unnecessary context value extraction and background context creation, simplifying the code and improving clarity.
- Enhanced the handling of stop events and message updates to consistently apply the session's tenant ID, improving the accuracy of operations in shared agent scenarios.
This update improves the reliability of async operations by ensuring the correct tenant context is consistently applied.
- Updated the API to include disabled agent IDs for better management of user-specific agent visibility.
- Introduced a new function to set the disabled state for shared agents, allowing users to manage their shared agent preferences.
- Added a new Skills API to retrieve available skills, enhancing the overall agent capabilities.
- Improved the AgentSelector and related components to filter out disabled agents, ensuring a cleaner user experience.
- Updated localization strings to reflect new features and improve clarity in agent and skill management.
This update enhances the user experience by providing better control over agent visibility and introducing new skill management functionalities.
- Added new localization strings for 'Mine' in English and Chinese.
- Enhanced organization subtitles to include agents in shared spaces.
- Updated permission descriptions to clarify roles related to knowledge bases and agents.
- Improved UI components to reflect new labels and permissions for better user experience.
This update strengthens the localization support, ensuring clarity in knowledge base and organization functionalities across multiple languages.
- Updated the AgentSelector component to improve agent selection logic, ensuring accurate representation of selected agents.
- Introduced new AgentShareSettings component for managing agent sharing with organizations, including UI for selecting organizations and displaying shared agents.
- Enhanced API interactions for sharing agents, including new methods for listing and managing agent shares.
- Improved localization support for agent sharing features across multiple languages.
- Updated various components to reflect changes in agent capabilities and sharing status, enhancing user experience and clarity.
This update strengthens the agent management features, providing users with better tools for sharing and selecting agents within organizations.
- Changed the default maximum number of members per organization from 50 to 200 in both the documentation and the application service layer.
- This update allows for greater flexibility in organization membership management, accommodating larger teams.
- Updated organization selection dropdowns to include new SVG icons for better visual representation.
- Improved organization option layout with additional metadata display, including member count and share count.
- Enhanced share item display with organization icons and improved action buttons for better user interaction.
- Adjusted styles for organization-related components to improve usability and aesthetics.
This update enhances the user experience by providing clearer organization visuals and more intuitive sharing options.
- Updated ListByOrganization and ListSharedKBsForUser methods to exclude soft-deleted knowledge bases and organizations, improving data integrity.
- Modified knowledge fetching methods to include documents from shared knowledge bases the user has access to, enhancing user experience.
- Improved comments for clarity on functionality and permission checks in UpdateSharePermission method.
This update strengthens the sharing capabilities and ensures users only see relevant knowledge bases, enhancing overall application usability.
- Updated SpaceAvatar component to allow emoji-based avatars, enabling users to select emojis as their space avatar.
- Added new props and computed properties to handle emoji logic and display.
- Enhanced UI to include an emoji picker and clear option for avatars.
- Updated localization strings in English and Chinese for new avatar features.
- Adjusted styles for emoji display and interaction.
This update improves user personalization options for avatars, enhancing the overall user experience in the application.
- Added a member limit field to the SearchableOrganizationItem interface, allowing organizations to specify a maximum number of members.
- Updated API and service layers to support searching organizations by name, description, or ID.
- Enhanced UI components to display member limits and updated localization strings in English and Chinese for new features.
- Included migration files to add the searchable column to the organizations table.
This update improves the organization search functionality and membership management, enhancing user experience and administrative control.
- Introduced a new member limit feature for organizations, allowing a maximum number of members to be set (default is 50; 0 means unlimited).
- Updated API endpoints to handle member limit during organization creation and updates.
- Implemented validation to prevent adding members when the limit is reached.
- Enhanced UI components to display and manage member limits, including localization updates for English and Chinese.
- Added migration files to support the new member limit column in the organizations table.
This update improves organization management by allowing administrators to control membership size.
- Added a 'searchable' flag to organizations, allowing them to be discoverable and joinable by ID without an invite code.
- Introduced API endpoints for searching and joining searchable organizations.
- Updated UI components to support searching for organizations and displaying results.
- Enhanced localization for new search-related features in both English and Chinese.
This update improves user experience by enabling easier discovery and joining of organizations.