Commit Graph

26 Commits

Author SHA1 Message Date
wizardchen beca2b89a3 feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.

This change threads one trace end-to-end:

- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
  worker can attach to an upstream trace without emitting a duplicate
  trace-create; StartGeneration now auto-picks parentObservationId from
  ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
  payloads so trace_id / parent_obs_id / user_id / session_id serialise
  into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
  so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
  resumes the upstream trace or opens a standalone asynq.<type> trace
  for scheduled jobs, and wraps the handler in a SPAN with task metadata
  (id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
  FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
  ResumeTrace no-trace-create guarantee; asynq_test.go covers
  InjectTracing round-trip, middleware resume path, and standalone
  trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
  and documents the cross-process propagation model.

No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 13:16:47 +08:00
wizardchen da8ba98622 feat: Refactor knowledge base service and router for improved wiki handling
- Updated the knowledge base service to ensure WikiConfig is created when wiki indexing is enabled, enhancing backward compatibility.
- Removed legacy syncing of WikiConfig.Enabled to IndexingStrategy.WikiEnabled, streamlining the logic for wiki enablement.
- Introduced a custom retry delay function for asynq tasks to handle wiki ingest lock conflicts more effectively, improving task management.
- Cleaned up the container file by removing unnecessary whitespace, contributing to better code readability.
2026-04-22 21:18:32 +08:00
wizardchen 8daf6b368c feat: Add pending tasks and active status to WikiStats
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.

These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
2026-04-22 21:17:31 +08:00
wizardchen d4d005422d feat: Implement wiki_config updates in knowledge base management
Added functionality to enable updates for the `wiki_config` field in the `KnowledgeBaseConfig` struct. This includes:

1. **Type Definition Update**: Added `WikiConfig` field to `KnowledgeBaseConfig` to allow JSON unmarshaling during update requests.
2. **Service Layer Enhancement**: Updated the `UpdateKnowledgeBase` method to persist changes to `wiki_config` when provided in the request.
3. **Frontend TypeScript Adjustments**: Enhanced type definitions in the frontend to support `wiki_config`, improving IDE autocomplete and type checking.

This fix addresses the critical issue where `wiki_config` was not being updated due to its absence in the `KnowledgeBaseConfig` struct, ensuring proper handling of wiki configurations during knowledge base updates.
2026-04-22 21:17:21 +08:00
wizardchen e147209e77 feat: Implement comprehensive wiki feature for knowledge bases
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes:

**Backend Changes:**
- Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log)
- Database schema: wiki_pages table with full migration support
- WikiPageService: CRUD operations and page management
- WikiPageRepository: GORM-based persistence layer
- Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents
  * Summary page generation using LLM
  * Entity and concept extraction in a single LLM call
  * Synthesis opportunity detection
  * Index page rebuilding
  * Log page maintenance
- Wiki boost feature: Enhance chat retrieval with wiki context
- Wiki linting: Maintenance and validation utilities
- Agent wiki tools: Enable agents to query and interact with wiki pages
- Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks
- Language support: Reuse existing middleware language infrastructure for LLM prompts

**Frontend Changes:**
- Wiki browser UI: View all wiki pages with filtering and search
- Wiki API client: Knowledge base wiki management endpoints
- Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model)
- i18n updates: Support for English, Korean, Russian, and Chinese interfaces

**Configuration:**
- Container DI: Wire up all wiki services
- Router: Register wiki API endpoints
- Task handling: Support async wiki ingest tasks

**Testing:**
- Unit tests for wiki page types
- Service layer tests
- Endpoint tests for wiki operations
- Integration tests with LLM mocking

**Documentation:**
- Language refactoring analysis and guides
- Implementation completion reports
- Quick reference guides for developers

**Key Features:**
✅ LLM-powered wiki page generation from documents
✅ Multi-language support (9+ languages)
✅ Automatic extraction of entities and concepts
✅ Synthesis opportunity detection
✅ Index and log page maintenance
✅ Progressive wiki building across multiple documents
✅ Agent-based wiki interaction
✅ Chat retrieval enhancement with wiki context
✅ Full frontend UI for wiki browsing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:18 +08:00
wizardchen a1a07e6818 Refactor knowledge post-processing into orchestrator task
Introduce KnowledgePostProcess task to decouple image multimodal
processing from downstream operations like summary, question
generation, and knowledge graph extraction.

This allows unifying the processing logic across documents and
images and ensures all text content, including from images,
is properly extracted before post-processing.
2026-04-16 18:13:20 +08:00
wizardchen 86d26fa9bf feat(video): remove video upload support and update related terminology
- Removed support for video file uploads across the application, including updates to the knowledge base and document processing services.
- Updated internationalization files to reflect the removal of video-related terminology, changing labels and descriptions to focus solely on audio processing.
- Adjusted file type handling to exclude video formats, ensuring users receive accurate feedback regarding supported file types.
- Enhanced error handling to provide clear messages when users attempt to upload unsupported video files.

This update streamlines the application by focusing on audio processing capabilities and improving user experience through clearer messaging.
2026-04-14 22:59:14 +08:00
DaWesen b80bdb1079 feat: add video multimodal functionality 2026-04-09 19:19:42 +08:00
wizardchen adabf8e089 feat(datasource): implement data source management functionality
- Added a new API for managing data sources, including CRUD operations and connection validation.
- Introduced UI components for data source settings, sync logs, and editor dialog, enhancing user experience in managing external data sources.
- Updated localization files to support new data source features in multiple languages.
- Implemented cron expression humanization for better scheduling visibility.
- Enhanced knowledge base editor to include data source management options.
2026-03-30 20:46:07 +08:00
wizardchen 3450ed5e41 feat(knowledge): offload manual processing to async tasks 2026-03-18 21:56:34 +08:00
wizardchen 125d3e5e4a feat: implement knowledge move functionality
- Added API endpoints for moving knowledge items between knowledge bases, including validation for source and target compatibility.
- Implemented asynchronous task handling for knowledge moves, tracking progress and status updates.
- Introduced new localization strings for move-related UI elements in English, Korean, Russian, and Chinese.
- Enhanced the frontend to support knowledge move operations, including target selection and progress monitoring.

These changes significantly improve the application's capabilities for managing knowledge across different bases, providing users with a seamless experience in organizing their knowledge assets.
2026-03-09 13:23:16 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
Dounx 548450c170 feat: add REDIS_USERNAME support for asynq client 2026-02-06 21:47:34 +08:00
ChenRussell 0bf55dd8f7 chore: fix ChunkExtracter typo (ChunkExtractor) 2026-02-06 13:39:28 +08:00
wizardchen 0ea9acfc2f feat: 支持删除文档类型标签 2026-01-15 19:38:16 +08:00
wizardchen a63a2a8daf refactor: 合并 FAQ 导入和验证逻辑 2026-01-14 17:09:18 +08:00
wizardchen 100484f13d feat: 新增FAQ批量导入dry run功能 2026-01-14 17:08:49 +08:00
wizardchen c2c4ee5777 fix: asynq redis db 2025-12-30 14:35:10 +08:00
begoniezhao 907e9a5522 feat: Add DataSchema tool for retrieving schema information from CSV and Excel files 2025-12-29 20:03:51 +08:00
wizardchen 10f8830c01 feat: 优化知识库删除流程,引入异步任务处理重操作;新增ProcessKBDelete方法实现后台清理 2025-12-25 22:07:42 +08:00
wizardchen 4c0846f9c7 feat: 增强标签删除功能,支持排除指定ID;新增异步索引删除任务优化删除流程 2025-12-25 22:07:42 +08:00
wizardchen 509582a4d5 feat: 知识库异步复制功能支持进度追踪和增量同步 2025-12-17 22:04:18 +08:00
wizardchen 16562eebbb feat: 添加AI问题生成功能,支持配置生成数量并在前端展示生成的问题 2025-12-03 01:42:31 +08:00
wizardchen 4a2be2b878 feat: Enhance file upload functionality with progress tracking
- Updated the `uploadKnowledgeFile` function to accept an optional progress callback, enabling real-time upload progress tracking.
- Refactored the document upload handling in the menu component to support multiple file uploads with individual progress events.
- Removed deprecated document processing state management from the KnowledgeBase view, streamlining the upload process.
- Improved user feedback during file uploads by dispatching events for upload start, progress, and completion.
2025-11-24 00:21:42 +08:00
wizardchen 244aaab328 feat: Update web search providers and enhance knowledge base management
- Added DuckDuckGo as a new web search provider in the configuration.
- Removed deprecated web search providers (Baidu, Bing, Bocha, Exa, Google, Kuaisou, Searxng, Tavily, Zhipu) to streamline the codebase.
- Enhanced knowledge base management by introducing temporary knowledge base handling in the session service.
- Updated the web search service to support RAG-based compression for improved search result processing.
- Refactored session and knowledge service interfaces to accommodate new functionalities.
2025-11-10 22:45:27 +08:00
begoniezhao b6d8be634a feat: 新增异步任务提取服务 2025-10-16 17:48:21 +08:00