Commit Graph

1199 Commits

Author SHA1 Message Date
wizardchen 3e61b91efd fix(storage): remove ListBucket permission from MinIO bucket policy 2026-04-16 18:13:21 +08:00
wizardchen a560497ee8 feat(tenant): add storage bucket uniqueness validation for tenant creation and updates
- Implemented a new method to validate the uniqueness of storage bucket names across tenants, ensuring no two tenants can use the same bucket name.
- Integrated this validation into the CreateTenant and UpdateTenant methods, enhancing data isolation and preventing potential conflicts.
- Improved error logging for better traceability when validation fails.

This update strengthens the integrity of tenant storage configurations.
2026-04-16 18:13:20 +08:00
wizardchen af58e9cff4 fix(repository): ensure expandedTopK is at least params.TopK in VectorRetrieve
- Added a check to set expandedTopK to params.TopK if it is lower, ensuring the subquery limit is not less than the final limit.
- Removed unnecessary type casting in the SQL query for embedding distance calculations, simplifying the query structure.

This update improves the accuracy of vector retrieval operations in the PostgreSQL repository.
2026-04-16 18:13:20 +08:00
wizardchen 1938094dcc feat(parser): add PDFScannedParser for handling scanned PDFs
- Introduced PDFScannedParser as a fallback parser for scanned PDFs that converts pages into images for OCR processing.
- Updated PDFParser to include PDFScannedParser in the parsing chain, enhancing the document parsing capabilities for scanned content.
- Improved logging for better error tracking during PDF parsing operations.
2026-04-16 18:13:20 +08:00
wizardchen fb714ea883 feat(document-preview): enhance security by sanitizing HTML and Markdown content
- Integrated sanitizeHTML and safeMarkdownToHTML functions to ensure that rendered HTML and Markdown content is safe from XSS vulnerabilities.
- Updated the rendering logic for Excel and Markdown to sanitize the output, improving overall security in document previews.
2026-04-16 18:13:20 +08:00
wizardchen a1a07e6818 Refactor knowledge post-processing into orchestrator task
Introduce KnowledgePostProcess task to decouple image multimodal
processing from downstream operations like summary, question
generation, and knowledge graph extraction.

This allows unifying the processing logic across documents and
images and ensures all text content, including from images,
is properly extracted before post-processing.
2026-04-16 18:13:20 +08:00
wizardchen 7c40fb6ac9 feat: enhance URL validation and timeout handling in knowledge base components
- Added isValidURL function to validate URLs before rendering links in the document content component, preventing invalid links from being clickable.
- Refactored timeout handling in KnowledgeBase.vue to use setTimeout instead of setInterval, improving performance and resource management.
- Ensured that timeouts are cleared appropriately to avoid memory leaks and unnecessary polling.

This update improves the robustness of URL handling and optimizes the component's performance.
2026-04-16 18:13:19 +08:00
wizardchen a78182cfc7 feat(handler): validate bucket name format and improve MinIO error handling
- Added validation for bucket name format to ensure it meets specified criteria, returning an error message for invalid formats.
- Simplified error messages for MinIO client creation and bucket auto-creation failures, enhancing clarity for users.
- Updated success messages to reflect the correct status of bucket creation without public-read policy settings.

This update improves the robustness of the MinIO integration and enhances user feedback during storage checks.
2026-04-16 18:13:19 +08:00
sn1p4am 0785a498e9 fix: use explicit localhost for docreader health probes 2026-04-16 17:41:15 +08:00
Liwx1014 2beb64c429 docs: add wiki documentation with cross-referenced links
- Create structured wiki from docs/ directory
- Add 17 markdown pages organized in 7 categories
- Include standard Markdown relative path links for navigation
- Add Mermaid knowledge graph visualization in Home.md
2026-04-15 20:38:31 +08:00
swim2sun d2c6814f22 fix: SQLite/Lite mode compatibility for FAQ import and chunk operations
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):

1. Redis nil pointer panic on FAQ import (knowledge.go)
   - Add in-memory fallback (sync.Map) for FAQ import progress tracking
   - Add nil guards for running task locks and progress queries

2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
   - GORM autoIncrement on non-PK columns doesn't work in SQLite
   - Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
   - Add BeforeCreate hook for KnowledgeTag
   - Use Unscoped() to include soft-deleted records in MAX(seq_id) query

3. NOW() function not available in SQLite (chunk.go)
   - Replace raw SQL NOW() with time.Now() or datetime('now')
   - Use dialector check for raw SQL that must differ between PG and SQLite

4. FAQ KB should reject file uploads (knowledge.go)
   - Add kb.Type == "faq" check in CreateKnowledgeFromFile

5. Linux/macOS build compatibility (Makefile)
   - Conditionally apply macOS-only linker flag based on uname

6. Add SQLite integration tests (chunk_sqlite_test.go)
   - TestCreateChunks_SQLite_SeqIDAutoAssigned
   - TestCreateChunks_SQLite_SeqIDContinuesFromExisting
   - TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
   - TestCreateChunks_SQLite_SeqIDAfterSoftDelete
   - TestKnowledgeTag_SQLite_SeqIDAutoAssigned
   - TestUpdateChunk_SQLite_NoNOWError

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 20:37:10 +08:00
moineye 5000e6e665 fix: missing deep copy entity 2026-04-15 20:35:48 +08:00
ugurtafrali 38077d6ccd Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
wizardchen 9bb4a12562 feat(repository): update keyword retrieval logic to use ParadeDB's matching operator
- Modified the SQL clause in the KeywordsRetrieve function to utilize ParadeDB's ||| operator for improved token matching.
- Enhanced the handling of retrieval results in the fuseOrDeduplicate function to maintain original scores for keyword-only results, ensuring better performance in FAQ scenarios.

This update optimizes the keyword retrieval process and improves the accuracy of search results.
2026-04-15 20:16:02 +08:00
wizardchen 8609c09a63 feat(request): enhance token refresh handling and public auth request logic
- Introduced a function to identify public authentication requests, preventing unnecessary token refresh attempts for specific endpoints.
- Simplified the redirection logic to the login page upon 401 errors, improving code readability and maintainability.
- Updated error handling to provide clearer messages for failed authentication attempts.

This update improves the request handling process, ensuring a smoother user experience during authentication flows.
2026-04-15 15:58:01 +08:00
wizardchen b0531900e9 feat(router): implement auto-setup failure handling for Lite edition
- Added logic to track auto-setup failures using localStorage, preventing repeated attempts if the setup fails.
- Updated the router's authentication flow to conditionally attempt auto-setup based on the failure status.
- Enhanced the login component to check for auto-setup failure before attempting the setup process.

This update improves the user experience by ensuring that failed auto-setup attempts do not lead to repeated failures during login.
2026-04-15 15:58:01 +08:00
wizardchen 9ba8582f70 feat(weknoracloud): implement WeKnoraCloud VLM integration
- Added WeKnoraCloudVLM struct to handle interactions with the WeKnoraCloud API, including image and text processing capabilities.
- Enhanced model configuration to include AppID and AppSecret for secure API access.
- Updated modelService to resolve WeKnoraCloud credentials and pass them to the VLM configuration.
- Implemented Predict method for sending requests to the WeKnoraCloud API and processing responses.

This update introduces a new integration with WeKnoraCloud, expanding the application's capabilities in handling multimedia content.
2026-04-15 15:43:08 +08:00
wizardchen 3fcfa89062 docs: update README files for version 0.4.0 with feature highlights and improved links 2026-04-15 11:37:24 +08:00
wizardchen 08d695d100 chore(changelog): update version 0.4.0 highlights and improve documentation
- Revised the changelog to reflect the new features in version 0.4.0, including the introduction of the Cloud Knowledge Assistant and updates to WeKnora Cloud integration.
- Enhanced descriptions for ClawHub Skill and data source import capabilities, clarifying functionality and usage.
- Updated README files in multiple languages to ensure consistency and accuracy in feature descriptions and links.

This update improves the clarity and accessibility of the project's documentation, ensuring users are well-informed about the latest features and integrations.
2026-04-15 11:37:24 +08:00
wizardchen 3f8201cd49 refactor(weknoracloud): update credential handling and configuration
- Replaced references to Docreader credentials with WeKnoraCloud credentials across multiple services and handlers.
- Introduced a new CredentialsConfig structure to manage third-party provider credentials, specifically for WeKnoraCloud.
- Updated database schema to include a credentials column for storing WeKnoraCloud AppID and AppSecret.
- Enhanced methods for retrieving and utilizing WeKnoraCloud credentials, ensuring proper encryption and decryption during storage and retrieval.

This update improves the management of WeKnoraCloud credentials, streamlining the integration and enhancing security measures.
2026-04-15 11:37:24 +08:00
wizardchen 3b23dccd5c feat(model_service): enhance WeKnoraCloud credential management
- Added tenantService to modelService for retrieving WeKnoraCloud credentials when not provided in model parameters.
- Implemented resolveWeKnoraCloudCredentials method to automatically fetch AppID and AppSecret from tenant configuration.
- Updated GetEmbeddingModel, GetEmbeddingModelForTenant, GetRerankModel, and GetChatModel methods to utilize the new credential resolution logic.
- Modified ModelParameters to encrypt and decrypt both APIKey and AppSecret for improved security during database operations.

This update streamlines credential management for WeKnoraCloud models, enhancing security and usability.
2026-04-15 11:37:24 +08:00
wizardchen 457e739978 feat(release): update to version 0.4.0 with new features, improvements, and bug fixes 2026-04-15 11:37:24 +08:00
wizardchen 86d26fa9bf feat(video): remove video upload support and update related terminology
- Removed support for video file uploads across the application, including updates to the knowledge base and document processing services.
- Updated internationalization files to reflect the removal of video-related terminology, changing labels and descriptions to focus solely on audio processing.
- Adjusted file type handling to exclude video formats, ensuring users receive accurate feedback regarding supported file types.
- Enhanced error handling to provide clear messages when users attempt to upload unsupported video files.

This update streamlines the application by focusing on audio processing capabilities and improving user experience through clearer messaging.
2026-04-14 22:59:14 +08:00
wizardchen 0fcab3a413 feat(video): update ASR terminology and file type labels for improved clarity
- Changed 'Audio Processing' to 'Audio & Video' in English, Korean, Russian, and Chinese translations to reflect expanded functionality.
- Updated ASR configuration titles and descriptions to include video processing capabilities, enhancing user understanding of the feature.
- Renamed file type labels from 'Audio Files' to 'Audio & Video' across multiple languages for consistency and clarity.
- Enhanced document upload handling in the KnowledgeBase to filter unsupported file types, providing user feedback for filtered images, videos, and audio files.

This update improves the internationalization of the application and clarifies the capabilities of the ASR feature.
2026-04-14 22:59:14 +08:00
wizardchen cb97489787 feat(attachment_processor): enhance error handling and content formatting for attachments
- Improved error handling in the attachment processing logic by providing detailed XML-like error messages for text, audio, and document processing failures.
- Updated the audio file content format to use XML tags for better structure, replacing the previous placeholder format.

This update enhances the clarity of error reporting and improves the overall structure of attachment content.
2026-04-14 18:21:12 +08:00
wizardchen bf3e913398 feat(chat_pipeline): enhance query handling with image and attachment metadata
- Updated the query content to include metadata for uploaded images and attachments, improving the context provided to the assistant.
- Modified the attachment prompt format to use XML-like tags for better structure and readability, including metadata for file type and size.
- Enhanced error handling for unsupported file content extraction, providing clearer feedback to users.

This update improves the assistant's ability to process and respond to queries involving multimedia content.
2026-04-14 18:21:12 +08:00
wizardchen 5a2f9f6a44 feat(prompt_templates): add document analysis prompt and enhance rewrite template descriptions
- Introduced a new prompt template for document analysis, allowing the assistant to provide detailed responses based on attached documents.
- Updated the rewrite template descriptions to clarify the handling of image and document attachments, ensuring better user understanding of intent classification.
- Added a new intent type for document-only queries to improve response accuracy.

This update enhances the assistant's capabilities in processing and analyzing documents while refining the user experience in intent classification.
2026-04-14 18:21:12 +08:00
wizardchen 1fc90f5772 refactor(weknoracloud): enhance internationalization and UI feedback for credential management
- Updated the WeKnoraCloud settings and related components to utilize i18n for improved localization.
- Replaced hardcoded strings with translation keys for better maintainability and user experience.
- Enhanced user feedback messages for credential status, including configuration, expiration, and unconfiguration states.
- Improved UI elements to guide users in managing WeKnoraCloud credentials effectively.

This update streamlines the user interface and ensures consistent messaging across different languages.
2026-04-14 17:36:01 +08:00
wizardchen 0eff63b308 feat(weknoracloud): implement credential management and status checks
- Introduced a new WeKnoraCloudSettings component for managing APPID and APPSECRET.
- Added functionality to save WeKnoraCloud credentials without automatically creating models.
- Implemented credential status checks across various components to provide user feedback on configuration status.
- Updated API endpoints to handle credential saving and status retrieval.
- Enhanced UI elements to guide users in configuring WeKnoraCloud settings.

This update improves the integration of WeKnoraCloud by allowing users to manage their credentials more effectively and receive real-time feedback on their configuration status.
2026-04-14 17:36:01 +08:00
lyingbug 0ce9dc056a Merge pull request #935 from MaoMengww/feat/attachment_support
feat: attachment_support
2026-04-14 17:35:31 +08:00
ochan.kwon 82b46b4b38 feat: add VectorStore CRUD API endpoints and documentation
Wire VectorStoreService to HTTP with 8 endpoints: types metadata, CRUD
(create/list/get/update/delete), and connection testing (raw + by ID).
Register routes, DI container bindings, and add API documentation.
2026-04-14 17:34:57 +08:00
bingxiang.cheng a2900d5540 chore(deps): 更新并新增go.mod依赖项
- 新增aliyun alibabacloud-oss-go-sdk-v2依赖
- 删除部分冗余依赖,调整依赖列表结构
- 添加jackc/pgx/v5作为直接依赖
- 统一和规范间接依赖声明顺序
- 移除go.opentelemetry.io/contrib的版本替换声明
2026-04-14 17:33:12 +08:00
begoniezhao 5082460cc0 fix(embedding): remove hardcoded TruncatePromptTokens in BatchEmbed
Remove the hardcoded TruncatePromptTokens: 512 limit from WeKnoraCloud
BatchEmbed to allow the API to use its default truncation behavior,
avoiding unexpected truncation of longer input texts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 16:29:49 +08:00
Mao Meng 84f70f0960 fix: rename sql file 2026-04-14 14:54:21 +08:00
Mao Meng 78e36d4ade Merge upstream main and resolve conflicts 2026-04-14 14:13:00 +08:00
Mao Meng 41aa35e346 refactor: change icon 2026-04-14 13:31:44 +08:00
lyingbug 1af90e87ab Merge pull request #949 from peterchan2020/feat_oss
Feat: Add Alibaba Cloud Storage Configuration
2026-04-14 11:08:56 +08:00
bingxiang.cheng 4054715d54 fix: add missing closing brace in checkOSS function 2026-04-14 10:03:45 +08:00
peterchan2020 8b9094f87c Merge branch 'main' into feat_oss 2026-04-14 00:41:29 +08:00
ochan.kwon 9edf0839ed feat: add VectorStore service layer with types and connection testing
Add VectorStoreService with CRUD validation, duplicate checking (DB + env
stores), and engine-specific health checks for 6 vector database types.
Include VectorStoreResponse DTO, env store builder, engine type metadata,
and comprehensive unit tests.
2026-04-13 22:57:17 +08:00
nullkey 30f349d06a feat(im): support custom endpoints for private WeCom deployments
Allow configuring custom WebSocket and API base URLs for WeCom
via optional credentials fields (ws_endpoint, api_base_url), enabling
integration with privately deployed WeCom instances.

Changes:
- LongConnClient and WebhookAdapter accept custom endpoint URLs
- Endpoint URLs validated at construction (scheme + SSRF check)
- Private deployment hostnames added to file download SSRF allowlist
- Frontend forms added for both WebSocket and Webhook modes
- i18n support for zh-CN, en-US, ru-RU, ko-KR

Uses default public cloud endpoints when not specified.
2026-04-13 22:54:42 +08:00
begoniezhao d4edd374df refactor: replace CryptoService with utils AES helpers and simplify encryption
- Remove infrastructure/crypto package (CryptoService, config, PBKDF2-based encryption)
- Replace with lightweight utils.GetAESKey/EncryptAESGCM/DecryptAESGCM helpers
- Remove cryptoSvc dependency from modelService, tenantService, weKnoraCloudService
- Remove crypto state persistence logic from container initialization
- Add TruncatePromptTokens field to WeKnoraCloud embed request
- Update frontend hint text for WeKnoraCloud credential errors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 20:55:39 +08:00
begoniezhao 2c0c1b8bab feat: integrate ChatBot provider and docreader 2026-04-13 17:47:24 +08:00
bingxiang.cheng 0164f4982d Merge branch 'feat_oss' of https://github.com/peterchan2020/WeKnora into feat_oss 2026-04-13 16:19:44 +08:00
bingxiang.cheng be65eb708f feat(oss): 添加阿里云OSS支持到文档和API说明中 2026-04-13 16:19:36 +08:00
peterchan2020 00ce568d02 Merge branch 'Tencent:main' into feat_oss 2026-04-13 15:58:46 +08:00
bingxiang.cheng dc8fc3d124 feat(i18n): 添加阿里云OSS的本地化支持到韩语和俄语翻译文件 2026-04-13 15:50:14 +08:00
bingxiang.cheng c21d1edd80 fix(oss): 更新OSS S3兼容模式的注释,修复路径样式描述并禁用自动校验和计算 2026-04-13 15:37:33 +08:00
bingxiang.cheng 847660e4fc feat(storage): 添加对阿里云OSS的支持,更新相关文档和界面文本 2026-04-13 15:37:33 +08:00
bingxiang.cheng 12001f23bf fix(oss): 禁用OSS S3兼容API的自动校验和计算,修复签名不匹配问题并调整路径样式设置 2026-04-13 15:37:33 +08:00