Added a new utility function, WithDeadlockRetry, to handle MySQL deadlock errors during chunk creation. This function retries the operation up to three times with exponential backoff. Updated the CreateChunks method to utilize this new function, ensuring robust handling of concurrent write scenarios and preventing potential deadlocks. Additionally, improved documentation for CreateChunks to clarify the use of Omit for avoiding auto-increment column issues.
- Updated the prompt template to clarify the task of understanding and rewriting user questions, improving the instructions for coreference resolution and ellipsis completion.
- Renamed sections for better clarity, changing "Rewriting Goals" to "Query Understanding."
- Added critical notes for intent classification regarding image and document attachments, ensuring accurate processing based on the presence of attachments.
- Improved examples to reflect the updated intent classification logic, enhancing the overall user experience in query handling.
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):
1. Redis nil pointer panic on FAQ import (knowledge.go)
- Add in-memory fallback (sync.Map) for FAQ import progress tracking
- Add nil guards for running task locks and progress queries
2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
- GORM autoIncrement on non-PK columns doesn't work in SQLite
- Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
- Add BeforeCreate hook for KnowledgeTag
- Use Unscoped() to include soft-deleted records in MAX(seq_id) query
3. NOW() function not available in SQLite (chunk.go)
- Replace raw SQL NOW() with time.Now() or datetime('now')
- Use dialector check for raw SQL that must differ between PG and SQLite
4. FAQ KB should reject file uploads (knowledge.go)
- Add kb.Type == "faq" check in CreateKnowledgeFromFile
5. Linux/macOS build compatibility (Makefile)
- Conditionally apply macOS-only linker flag based on uname
6. Add SQLite integration tests (chunk_sqlite_test.go)
- TestCreateChunks_SQLite_SeqIDAutoAssigned
- TestCreateChunks_SQLite_SeqIDContinuesFromExisting
- TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
- TestCreateChunks_SQLite_SeqIDAfterSoftDelete
- TestKnowledgeTag_SQLite_SeqIDAutoAssigned
- TestUpdateChunk_SQLite_NoNOWError
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously, only the original uploaded file was deleted via knowledge.FilePath.
Images extracted by the builtin parser (stored as individual S3/MinIO/local files
via ImageResolver) were never cleaned up, causing orphan files to accumulate.
- Add ListImageInfoByKnowledgeIDs repository method (no chunk_type filter)
- Add collectImageURLs utility with URL deduplication and JSON parse logging
- Add deleteExtractedImages helper (standalone, best-effort with error logging)
- Fix DeleteKnowledge: collect image URLs before errgroup, delete in file goroutine
- Fix DeleteKnowledgeList: bulk collect with KB-level fileSvc mapping
- Fix cleanupKnowledgeResources: collect before chunk deletion, delete after
- Fix ProcessKBDelete: collect before chunk deletion for KB-wide cleanup
- Updated context template to include runtime metadata such as current time and week, improving contextual awareness in user queries.
- Enhanced rewrite template with critical instructions for intent classification, ensuring that rewritten questions preserve essential entities and keywords.
- Refined intent classification logic to prioritize user intents more effectively, improving the accuracy of responses based on user queries.
- Added examples to clarify expected input and output formats for intent classification, enhancing usability for developers.
- Added a new API endpoint to fetch suggested questions based on the agent's associated knowledge bases.
- Implemented frontend components to display suggested questions in both chat and create chat views, enhancing user interaction.
- Updated localization files to include translations for suggested questions in English, Korean, Russian, and Chinese.
- Introduced debounce logic for fetching suggestions to optimize performance during knowledge base changes.
- Enhanced backend services to support fetching and returning suggested questions, improving the overall chat experience.
- Implemented a new method in the chunk repository to find FAQ chunks with overlapping standard or similar questions.
- Updated the knowledge service to utilize this method for checking duplicates during FAQ question creation.
- Improved error handling for duplicate questions, providing clearer feedback on conflicts.
- Updated DeleteChunks method to split IDs into batches of 5000 to prevent MySQL Error 1390.
- Added unit tests for CalculateFAQContentHash to ensure consistent hashing behavior across various scenarios, including normalization and case sensitivity.
- Added API endpoints for moving knowledge items between knowledge bases, including validation for source and target compatibility.
- Implemented asynchronous task handling for knowledge moves, tracking progress and status updates.
- Introduced new localization strings for move-related UI elements in English, Korean, Russian, and Chinese.
- Enhanced the frontend to support knowledge move operations, including target selection and progress monitoring.
These changes significantly improve the application's capabilities for managing knowledge across different bases, providing users with a seamless experience in organizing their knowledge assets.
- Implemented ListChunksByParentIDs in chunkRepository to retrieve chunks based on multiple parent IDs, improving data retrieval efficiency.
- Enhanced the chunk merging logic in PluginMerge to better handle overlapping chunks and collect merged ImageInfo for parent chunks.
- Updated the ChunkRepository interface to include the new ListChunksByParentIDs method, ensuring consistency across the application.
These changes optimize chunk management and improve the handling of image information during merges, enhancing overall application functionality.
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.
This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
- Introduced a new API for organization management, including creation, updating, and deletion of organizations.
- Implemented organization sharing functionality for knowledge bases, allowing users to share knowledge bases with specific organizations and set permissions.
- Added UI components for organization selection and sharing dialogs.
- Enhanced existing components to support organization-related features, including shared knowledge base handling and organization role management.
This update enhances collaboration capabilities within the application by allowing users to manage and share knowledge bases across different organizations.
- Updated chunk repository methods to support deletion of unindexed chunks and listing all FAQ chunks by knowledge ID.
- Introduced batch update functionality for chunks in the chunk service.
- Enhanced FAQ import logic to handle append and replace operations, improving efficiency and data integrity.
- Added content hash calculation for FAQ entries to facilitate quick matching and deduplication.
- Refactored related methods to streamline the import process and improve logging for better tracking of import tasks.
- Updated the `uploadKnowledgeFile` function to accept an optional progress callback, enabling real-time upload progress tracking.
- Refactored the document upload handling in the menu component to support multiple file uploads with individual progress events.
- Removed deprecated document processing state management from the KnowledgeBase view, streamlining the upload process.
- Improved user feedback during file uploads by dispatching events for upload start, progress, and completion.
- Added support for tracking the status of FAQ import tasks, including progress indicators for ongoing imports.
- Introduced new localization strings for processing tasks and batch operations in multiple languages.
- Updated the FAQ entry manager to display import progress and improved user feedback during the import process.
- Enhanced the KnowledgeBase component to reflect the processing state of knowledge items, improving user experience during document uploads.
- Implemented batch deletion functionality for chunks, streamlining the management of knowledge base content.
- Introduced new localization strings for document count, FAQ entries, and chunk count in English, Russian, and Chinese.
- Updated the KnowledgeBase and KnowledgeBaseList components to display knowledge and chunk counts.
- Implemented backend functions to count knowledge items and chunks associated with each knowledge base, improving data management and user insights.
- Added API functions for listing, creating, updating, and deleting knowledge base tags.
- Introduced batch update functionality for enabling/disabling FAQ entries and updating their tags.
- Enhanced the KnowledgeBase.vue and FAQEntryManager.vue components to support tag management and batch operations.
- Updated localization files to include new strings related to tag management and batch operations.
- Added support for creating and managing FAQ knowledge bases, including new API endpoints for listing, upserting, updating, and deleting FAQ entries.
- Enhanced the frontend to allow users to create FAQ knowledge bases and manage FAQ entries with a dedicated interface.
- Introduced new types and configurations for FAQ handling in the backend, ensuring proper indexing and retrieval of FAQ entries.
- Updated localization files to include new strings related to FAQ management in multiple languages.
- Refactored existing components to accommodate the new FAQ features, improving user experience and functionality.