Commit Graph

7 Commits

Author SHA1 Message Date
wizardchen 92b6b7507d refactor(chunk): improve parent-child chunk relationship handling in SplitTextParentChild
- Updated the SplitTextParentChild function to maintain a newParents slice, ensuring accurate parent indexing when splitting text into child chunks.
- This change enhances the integrity of parent-child relationships by correctly associating child chunks with their respective parents, improving overall chunk processing.
2026-04-16 18:29:33 +08:00
wizardchen 0fcab3a413 feat(video): update ASR terminology and file type labels for improved clarity
- Changed 'Audio Processing' to 'Audio & Video' in English, Korean, Russian, and Chinese translations to reflect expanded functionality.
- Updated ASR configuration titles and descriptions to include video processing capabilities, enhancing user understanding of the feature.
- Renamed file type labels from 'Audio Files' to 'Audio & Video' across multiple languages for consistency and clarity.
- Enhanced document upload handling in the KnowledgeBase to filter unsupported file types, providing user feedback for filtered images, videos, and audio files.

This update improves the internationalization of the application and clarifies the capabilities of the ASR feature.
2026-04-14 22:59:14 +08:00
wizardchen ce7df624fb feat(chunker): implement header tracking for Markdown tables during chunking
- Added headerTracker to preserve context of Markdown table headers across split chunks.
- Implemented logic to prepend headers to subsequent chunks after the first.
- Enhanced merging logic to ensure headers are not duplicated in overlapping content.
- Added comprehensive tests for header tracking and chunking behavior with Markdown tables.
2026-04-01 13:38:40 +08:00
wizardchen f3a3c72a4f fix(docparser): handle nested linked images and url parens 2026-03-18 21:56:34 +08:00
ChenRussell 0fe5d67b6a fix: enforce 7500 char limit in chunker to prevent embedding API errors
Prevents 400 Bad Request errors when uploading documents with large
tables or code blocks by adding hard limits in buildUnitsWithProtection
and mergeUnits functions.
2026-03-15 23:59:34 +08:00
wizardchen b333499078 feat: implement parent-child chunking strategy for enhanced context management
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.

These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
2026-03-05 18:14:14 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00