Commit Graph

1242 Commits

Author SHA1 Message Date
wizardchen 749e8a0a19 feat: Implement content filter handling in agent response analysis
Added logic to the analyzeResponse function to handle cases where the LLM response is blocked by the content filter. This includes emitting events for the final answer and stopping the agent loop to prevent infinite responses. The implementation ensures that users receive a clear message when content is filtered, enhancing the agent's response management.
2026-04-22 21:17:21 +08:00
wizardchen 4ac1860678 fix: Update intent prompts for knowledge assistant response clarity
Enhanced the system prompts for the knowledge assistant to better guide users when web search is disabled. The updates clarify the assistant's role in answering questions based on the user's personal knowledge base and emphasize the importance of indicating when information may be outdated or uncertain. This aims to improve user experience and ensure accurate communication of capabilities.
2026-04-22 21:17:20 +08:00
wizardchen ca113f3845 docs: Add WikiConfig fix completion report
Added comprehensive completion report documenting:

- All accomplishments across three phases (analysis, implementation, documentation)
- Implementation statistics (3 files, 43 lines added, 20 removed, +23 net)
- Two git commits with their purposes and changes
- Quality assurance checklist across code, testing, documentation, deployment
- Documentation deliverables organized by audience
- Deployment readiness checklist with all items marked complete
- Key learning points and architectural insights
- Business, technical, and operational impact analysis
- Next steps and timeline (5-7 days)
- All success criteria verified and met
- Sign-off confirming production readiness

This report serves as the final summary document for the WikiConfig update
fix project, confirming that all work is complete and ready for deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:20 +08:00
wizardchen 33c4626c14 docs: Add comprehensive WikiConfig update fix documentation
Added complete documentation suite for the WikiConfig update fix (commit b09893d5):

1. EXECUTIVE_SUMMARY_WIKI_CONFIG_FIX.md
   - High-level overview for stakeholders and project managers
   - Business impact, solution, and deployment plan
   - Risk assessment and success metrics

2. WIKI_CONFIG_FIX_COMPLETE.md
   - Complete technical implementation overview
   - Problem statement and root cause analysis
   - Solution details across all three system layers
   - Full data flow diagrams (before/after)
   - Testing scenarios and verification checklist
   - Deployment checklist and rollback plan

3. IMPLEMENTATION_SUMMARY.md
   - Detailed implementation guide for developers
   - Code changes with line numbers
   - Unit test and integration test examples
   - Data flow sequence diagrams
   - Backward compatibility analysis
   - Deployment considerations

4. WIKI_CONFIG_VERIFICATION.md
   - Testing and verification procedures
   - Four complete test cases with curl examples
   - Database level verification queries
   - Comprehensive verification checklist
   - Impact analysis matrix

5. README_WIKI_CONFIG_FIX.md
   - Documentation index organized by audience
   - Quick start guide for different roles
   - Troubleshooting section
   - Learning resources and related files

6. CHANGES_SUMMARY.txt
   - Concise summary of all changes
   - Code changes with exact line numbers
   - Statistics and impact analysis
   - Deployment steps and rollback procedure

Documentation is organized for multiple stakeholders:
- Project managers/leads: Start with EXECUTIVE_SUMMARY_WIKI_CONFIG_FIX.md
- Backend developers: Use WIKI_CONFIG_FIX_COMPLETE.md + IMPLEMENTATION_SUMMARY.md
- QA/Testers: Follow WIKI_CONFIG_VERIFICATION.md
- DevOps/Deployment: Review WIKI_CONFIG_FIX_COMPLETE.md deployment section

All documentation cross-references related files and provides complete
context for understanding, testing, and deploying the fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:20 +08:00
wizardchen e15adca0a6 fix: Enable wiki_config updates in knowledge base configuration
Implemented three critical fixes to enable wiki_config persistence during
KB creation and updates:

1. Add WikiConfig field to KnowledgeBaseConfig struct
   - File: internal/types/knowledgebase.go
   - Enables JSON unmarshaling of wiki_config from update requests
   - Previously the field was silently dropped during deserialization

2. Update UpdateKnowledgeBase service to persist wiki_config
   - File: internal/application/service/knowledgebase.go
   - Added logic to apply wiki_config changes when config parameter is provided
   - Mirrors existing logic for FAQConfig persistence

3. Update frontend TypeScript types for IDE completion
   - File: frontend/src/api/knowledge-base/index.ts
   - Added WikiConfig type to createKnowledgeBase function signature
   - Added structured WikiConfig type to updateKnowledgeBase config parameter
   - Enables TypeScript IDE autocomplete and type checking

Flow:
- Frontend: updateKnowledgeBase({name, description, config: {wiki_config}})
- Handler: Parses into UpdateKnowledgeBaseRequest via KnowledgeBaseConfig
- Service: Applies wiki_config to KB entity via UpdateKnowledgeBase
- Database: Persists wiki_config JSONB field to postgres

Root cause analysis:
Previously, the KnowledgeBaseConfig struct was missing the WikiConfig field,
causing JSON unmarshaling to silently drop any wiki_config data sent in
PUT requests. KB creation worked because it used the full KnowledgeBase struct.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:20 +08:00
wizardchen 16a0bff7bd feat: Enhance wiki functionality with recent activity log
This commit introduces a new "Recent Activity" section in the WikiBrowser component, allowing users to view the latest changes made to the wiki. The implementation includes:

- Addition of a toggleable recent activity log displaying the last five entries.
- Backend support for fetching the activity log from the knowledge base.
- Updates to the wiki ingest process to ensure draft pages are published post-ingest.
- Improvements to the deduplication process for entities and concepts during wiki page creation.

These enhancements aim to improve user engagement and provide better insights into wiki updates.
2026-04-22 21:17:19 +08:00
wizardchen 3386c0786b docs: Add project index and comprehensive navigation guide 2026-04-22 21:17:19 +08:00
wizardchen 00f8dff19a docs: Add executive summary for wiki feature project completion 2026-04-22 21:17:19 +08:00
wizardchen 103cc14d3e docs: Add comprehensive wiki feature README with API and deployment guide 2026-04-22 21:17:18 +08:00
wizardchen 94db0e0978 docs: Add comprehensive deployment checklist and guide 2026-04-22 21:17:18 +08:00
wizardchen 507c2ed2ca docs: Add final completion summary for wiki feature 2026-04-22 21:17:18 +08:00
wizardchen e147209e77 feat: Implement comprehensive wiki feature for knowledge bases
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes:

**Backend Changes:**
- Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log)
- Database schema: wiki_pages table with full migration support
- WikiPageService: CRUD operations and page management
- WikiPageRepository: GORM-based persistence layer
- Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents
  * Summary page generation using LLM
  * Entity and concept extraction in a single LLM call
  * Synthesis opportunity detection
  * Index page rebuilding
  * Log page maintenance
- Wiki boost feature: Enhance chat retrieval with wiki context
- Wiki linting: Maintenance and validation utilities
- Agent wiki tools: Enable agents to query and interact with wiki pages
- Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks
- Language support: Reuse existing middleware language infrastructure for LLM prompts

**Frontend Changes:**
- Wiki browser UI: View all wiki pages with filtering and search
- Wiki API client: Knowledge base wiki management endpoints
- Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model)
- i18n updates: Support for English, Korean, Russian, and Chinese interfaces

**Configuration:**
- Container DI: Wire up all wiki services
- Router: Register wiki API endpoints
- Task handling: Support async wiki ingest tasks

**Testing:**
- Unit tests for wiki page types
- Service layer tests
- Endpoint tests for wiki operations
- Integration tests with LLM mocking

**Documentation:**
- Language refactoring analysis and guides
- Implementation completion reports
- Quick reference guides for developers

**Key Features:**
✅ LLM-powered wiki page generation from documents
✅ Multi-language support (9+ languages)
✅ Automatic extraction of entities and concepts
✅ Synthesis opportunity detection
✅ Index and log page maintenance
✅ Progressive wiki building across multiple documents
✅ Agent-based wiki interaction
✅ Chat retrieval enhancement with wiki context
✅ Full frontend UI for wiki browsing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:18 +08:00
wizardchen e066f808b6 docs: Add quick reference card for language refactoring
- One-page summary of changes and metrics
- Testing and verification commands
- FAQ and impact summary
- Quick links to documentation and code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:18 +08:00
wizardchen 2570d8f740 docs: Add language refactoring README with quick start guide
- Quick reference guide for the language refactoring project
- Key achievements and changes summary
- Deployment checklist and verification commands
- Links to detailed documentation
- Commit hash and status information

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:17 +08:00
wizardchen 095ccaed10 refactor: Replace hardcoded language logic with middleware infrastructure
- Refactor wiki_ingest.go to use types.LanguageLocaleName() instead of hardcoded if-else
- Supports 9+ languages (Chinese, English, Korean, Japanese, Russian, French, German, Spanish, Portuguese)
- Reduces 7 lines to 4 lines (43% reduction in code)
- Eliminates code duplication by reusing existing middleware infrastructure
- Consistent naming with middleware layer ("Chinese (Simplified)" format)
- Add comprehensive unit tests for LanguageLocaleName() (30 test cases)
- Add implementation guide, analysis, and completion documentation

Benefits:
- Single source of truth for language mappings (types.LanguageLocaleName)
- Better code maintainability and consistency
- No breaking changes or schema modifications required
- Performance: ~4.3ns per call, no measurable impact
- All tests passing, build successful

Files modified:
- internal/application/service/wiki_ingest.go: Refactored language determination (line 139)
- internal/types/context_helpers_test.go: Added 30 unit test cases (NEW)
- Documentation: Added LANGUAGE_MIDDLEWARE_ANALYSIS.md, LANGUAGE_REFACTORING_COMPLETED.md, IMPLEMENTATION_GUIDE.md, REFACTORING_SUMMARY.txt

Refs: LANGUAGE_MIDDLEWARE_ANALYSIS.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:17 +08:00
nullkey 16792713c5 feat(frontend): enable Yuque datasource option with optional base_url
Flip the Yuque tile to available, add a base_url field (optional) for users
on private Yuque deployments, and populate the permission doc/page links.
Introduces an optional flag on ConnectorDef.fields so the Test Connection
validator skips empty optional fields; required fields get a visible * mark.

Also adds an optional hintKey on ConnectorDef.fields for per-field guidance
text, a Yuque-specific prereq guide, and a base_url hint explaining when to
leave it empty vs fill it for Yuque Enterprise / self-hosted deployments.
2026-04-22 16:39:55 +08:00
nullkey 4939011b0d fix(feishu): truncate sanitized filenames at UTF-8 rune boundary
Raw byte truncation at 200 bytes splits multi-byte codepoints (Chinese
characters are 3 bytes), producing invalid UTF-8 that downstream filename
validation (utf8.ValidString) rejects with "文件名包含非法字符". Peel
trailing bytes via DecodeLastRuneInString until the last rune is complete.

Same latent bug and fix as the Yuque connector's sanitizeFileName.
2026-04-22 16:39:55 +08:00
nullkey 656b07027a feat(datasource): add Yuque (语雀) connector
Syncs personal and team Yuque knowledge bases into WeKnora as Markdown,
with full and incremental sync + deletion detection. Mirrors the Feishu
connector pattern; fetches V2Doc.body directly (no async export step).

Config: api_token (required) + base_url (optional, defaults to public cloud).
401/403 wraps ErrInvalidCredentials; 429 honors Retry-After; filters to
status=1 Doc-type at list stage to conserve API quota. Body is trusted as
Markdown for format=markdown|lake — empirically verified against the v2
API: Lake XML lives separately in body_lake, body itself is the Markdown
view. Other formats (e.g. html) are skipped with a placeholder item so
they surface in sync logs instead of silently feeding HTML to the parser.

v2Doc.status accepts both int and string shapes — Yuque's OpenAPI spec
declares string but the live API returns int. sanitizeFileName truncates
at a UTF-8 rune boundary so multi-byte titles don't emit invalid UTF-8
that downstream filename validation rejects.

Also adds logger.SetOutput test hook, used by the token-redaction test.
2026-04-22 16:39:55 +08:00
yy 6accdc2a60 fix:修复make dev-app里面脚本问题 2026-04-22 16:38:50 +08:00
jw_mac a121a5d636 fix(frontend): 修复公式输出展示问题并引入 KaTeX 支持 (#912) 2026-04-22 16:38:24 +08:00
Liwx da8191cd57 fix(docker): 修复镜像构建兼容性问题 2026-04-21 23:29:41 +08:00
Mao Meng a54822d402 fix/air-toml-cgo-ldflags 2026-04-21 23:27:33 +08:00
draix 5591c84e7f fix: remove limit/offset from required in list_knowledge_chunks JSON Schema
limit and offset are optional parameters with default values (limit=20,
offset=0). Marking them as required in the JSON Schema forces LLM clients
to always supply them, breaking calls that rely on the defaults.

Fixes #1003
2026-04-21 23:27:14 +08:00
moineye 4ca0c7a644 fix: ReadEvent incompatible sse data format 2026-04-20 22:29:38 +08:00
jw_mac 09e1642335 fix(storage): auto-detect content-type to resolve image rendering issues
Fixes #995. Added mime type detection in SaveBytes for MinIO, OSS, S3, and TOS to prevent images from being incorrectly saved as text/csv.
2026-04-20 21:59:54 +08:00
bingxiang.cheng ede7dd08a2 fix: 修正 mcp-api-requester 为 mcp_api_requester 的引用 2026-04-20 19:54:54 +08:00
bingxiang.cheng 8f671817a3 refactor(openmaic-classroom): 规范文档并强化mcp-api-requester调用要求
- 重新排版和规范技能说明文档,提升可读性
- 明确强调必须使用WeKnora的mcp-api-requester MCP工具调用OpenMAIC API
- 取消curl命令降级方案,改为引导用户部署mcp-api-requester服务
- 增强本地模式Base URL处理,替换localhost和127.0.0.1为host.docker.internal
- 完善错误处理和任务状态查询流程说明
- 更新API参考文档,统一接口描述和响应格式
- 细化多文档生成课程集的调用和错误引导逻辑
- 提示单次生成任务时间及托管模式配额限制
2026-04-20 19:54:54 +08:00
bingxiang.cheng fff89d3abb feat(openmaic-classroom): 添加OpenMAIC互动课堂技能支持
- 新建openmaic-classroom技能文档,描述技能功能和使用流程
- 实现基于RAG检索结果和PDF生成互动课件的核心能力
- 支持托管模式和本地部署两种API调用方式
- 设计生成请求构建流程及任务状态轮询机制
- 实现多文档合成课程集的处理方案及错误处理说明
- 提供完整API接口参考,包括健康检查、生成任务、PDF解析等
- 添加需求构建指南,指导如何将检索结果转为课程需求
- 新增rag-to-requirement.py脚本,实现RAG结果向需求结构化转换
- 规范请求参数和返回格式,兼容多语言和教学深度设定
- 明确MCP调用工具的识别与降级策略,提升接口调用体验
2026-04-20 19:54:54 +08:00
ochan.kwon 127654ee78 feat: add vector_store_id column to knowledge_bases for per-KB vector store binding 2026-04-20 19:53:44 +08:00
Mao Meng e3b493e316 fix: use constant-time comparison for API key validation 2026-04-19 21:41:22 +08:00
ochan.kwon 8fd1d61333 feat: add VectorStore management UI settings page 2026-04-17 21:51:25 +08:00
ochan.kwon 927d3debff feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
hjz 7bd36bf954 feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
wizardchen 092044b68f refactor(postgres): optimize vector retrieval and HNSW settings
- Adjusted the expanded TopK limit to a maximum of 200 candidates to enhance HNSW efficiency.
- Updated the vector retrieval query to ensure the ORDER BY expression matches the indexed expression, improving index usage.
- Implemented local settings for HNSW's ef_search and iterative_scan within a transaction to optimize candidate retrieval.
- Added fallback logic to handle unsupported GUCs gracefully, ensuring query execution continues even with older pgvector versions.

These changes improve the performance and reliability of vector retrieval in the PostgreSQL implementation.
2026-04-17 12:11:03 +08:00
wizardchen d546406a15 feat(output): enhance output formatting to XML for various tools
- Updated the formatKnowledgeBaseList function to return knowledge base information as XML, improving data structure for integration.
- Modified the formatOutput function in GrepChunksTool to format search results as XML, enhancing readability and consistency.
- Adjusted the formatOutput method in KnowledgeSearchTool to output search results in XML format, providing a structured response.
- Refactored the buildOutput function in ListKnowledgeChunksTool to generate XML output, improving the clarity of chunk information.

These changes standardize output formats across multiple tools, facilitating better data handling and integration.
2026-04-17 12:11:03 +08:00
ochan.kwon c5fc05f3d0 feat: add registry dual-map and engine factory for VectorStore
- Extend RetrieveEngineRegistry with byStoreID map for DB store engines
- Add StoreRegistry interface and EngineFactory function type
- Create engine factory with per-driver SDK client construction
- Load VectorStore records from DB into registry at startup
- Wire Create/Delete to dynamically register/unregister engines
- Add comprehensive unit tests for registry and service
2026-04-17 00:15:37 +08:00
wizardchen 4f55994266 feat(prompt): enhance question rewriting and intent classification in prompt templates
- Updated the prompt template to clarify the task of understanding and rewriting user questions, improving the instructions for coreference resolution and ellipsis completion.
- Renamed sections for better clarity, changing "Rewriting Goals" to "Query Understanding."
- Added critical notes for intent classification regarding image and document attachments, ensuring accurate processing based on the presence of attachments.
- Improved examples to reflect the updated intent classification logic, enhancing the overall user experience in query handling.
2026-04-16 23:15:02 +08:00
wizardchen 53e8e17df4 feat(logger): implement LLM debug logging functionality
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
2026-04-16 23:15:02 +08:00
wizardchen 6235a82183 Fix logger format string errors in system handler 2026-04-16 18:39:30 +08:00
wizardchen dcb7fd05b1 Fix compile error in weaviate repository 2026-04-16 18:37:31 +08:00
wizardchen 92b6b7507d refactor(chunk): improve parent-child chunk relationship handling in SplitTextParentChild
- Updated the SplitTextParentChild function to maintain a newParents slice, ensuring accurate parent indexing when splitting text into child chunks.
- This change enhances the integrity of parent-child relationships by correctly associating child chunks with their respective parents, improving overall chunk processing.
2026-04-16 18:29:33 +08:00
wizardchen fdbd981868 feat(i18n): update connection test messages and labels in multiple languages 2026-04-16 18:13:21 +08:00
wizardchen 491c324197 feat(system): remove MinIO list buckets API and UI dependencies 2026-04-16 18:13:21 +08:00
wizardchen 3e61b91efd fix(storage): remove ListBucket permission from MinIO bucket policy 2026-04-16 18:13:21 +08:00
wizardchen a560497ee8 feat(tenant): add storage bucket uniqueness validation for tenant creation and updates
- Implemented a new method to validate the uniqueness of storage bucket names across tenants, ensuring no two tenants can use the same bucket name.
- Integrated this validation into the CreateTenant and UpdateTenant methods, enhancing data isolation and preventing potential conflicts.
- Improved error logging for better traceability when validation fails.

This update strengthens the integrity of tenant storage configurations.
2026-04-16 18:13:20 +08:00
wizardchen af58e9cff4 fix(repository): ensure expandedTopK is at least params.TopK in VectorRetrieve
- Added a check to set expandedTopK to params.TopK if it is lower, ensuring the subquery limit is not less than the final limit.
- Removed unnecessary type casting in the SQL query for embedding distance calculations, simplifying the query structure.

This update improves the accuracy of vector retrieval operations in the PostgreSQL repository.
2026-04-16 18:13:20 +08:00
wizardchen 1938094dcc feat(parser): add PDFScannedParser for handling scanned PDFs
- Introduced PDFScannedParser as a fallback parser for scanned PDFs that converts pages into images for OCR processing.
- Updated PDFParser to include PDFScannedParser in the parsing chain, enhancing the document parsing capabilities for scanned content.
- Improved logging for better error tracking during PDF parsing operations.
2026-04-16 18:13:20 +08:00
wizardchen fb714ea883 feat(document-preview): enhance security by sanitizing HTML and Markdown content
- Integrated sanitizeHTML and safeMarkdownToHTML functions to ensure that rendered HTML and Markdown content is safe from XSS vulnerabilities.
- Updated the rendering logic for Excel and Markdown to sanitize the output, improving overall security in document previews.
2026-04-16 18:13:20 +08:00
wizardchen a1a07e6818 Refactor knowledge post-processing into orchestrator task
Introduce KnowledgePostProcess task to decouple image multimodal
processing from downstream operations like summary, question
generation, and knowledge graph extraction.

This allows unifying the processing logic across documents and
images and ensures all text content, including from images,
is properly extracted before post-processing.
2026-04-16 18:13:20 +08:00
wizardchen 7c40fb6ac9 feat: enhance URL validation and timeout handling in knowledge base components
- Added isValidURL function to validate URLs before rendering links in the document content component, preventing invalid links from being clickable.
- Refactored timeout handling in KnowledgeBase.vue to use setTimeout instead of setInterval, improving performance and resource management.
- Ensured that timeouts are cleared appropriately to avoid memory leaks and unnecessary polling.

This update improves the robustness of URL handling and optimizes the component's performance.
2026-04-16 18:13:19 +08:00