Commit Graph

393 Commits

Author SHA1 Message Date
wizardchen bf3e913398 feat(chat_pipeline): enhance query handling with image and attachment metadata
- Updated the query content to include metadata for uploaded images and attachments, improving the context provided to the assistant.
- Modified the attachment prompt format to use XML-like tags for better structure and readability, including metadata for file type and size.
- Enhanced error handling for unsupported file content extraction, providing clearer feedback to users.

This update improves the assistant's ability to process and respond to queries involving multimedia content.
2026-04-14 18:21:12 +08:00
wizardchen 0eff63b308 feat(weknoracloud): implement credential management and status checks
- Introduced a new WeKnoraCloudSettings component for managing APPID and APPSECRET.
- Added functionality to save WeKnoraCloud credentials without automatically creating models.
- Implemented credential status checks across various components to provide user feedback on configuration status.
- Updated API endpoints to handle credential saving and status retrieval.
- Enhanced UI elements to guide users in configuring WeKnoraCloud settings.

This update improves the integration of WeKnoraCloud by allowing users to manage their credentials more effectively and receive real-time feedback on their configuration status.
2026-04-14 17:36:01 +08:00
lyingbug 0ce9dc056a Merge pull request #935 from MaoMengww/feat/attachment_support
feat: attachment_support
2026-04-14 17:35:31 +08:00
bingxiang.cheng a2900d5540 chore(deps): 更新并新增go.mod依赖项
- 新增aliyun alibabacloud-oss-go-sdk-v2依赖
- 删除部分冗余依赖,调整依赖列表结构
- 添加jackc/pgx/v5作为直接依赖
- 统一和规范间接依赖声明顺序
- 移除go.opentelemetry.io/contrib的版本替换声明
2026-04-14 17:33:12 +08:00
Mao Meng 78e36d4ade Merge upstream main and resolve conflicts 2026-04-14 14:13:00 +08:00
peterchan2020 8b9094f87c Merge branch 'main' into feat_oss 2026-04-14 00:41:29 +08:00
ochan.kwon 9edf0839ed feat: add VectorStore service layer with types and connection testing
Add VectorStoreService with CRUD validation, duplicate checking (DB + env
stores), and engine-specific health checks for 6 vector database types.
Include VectorStoreResponse DTO, env store builder, engine type metadata,
and comprehensive unit tests.
2026-04-13 22:57:17 +08:00
begoniezhao d4edd374df refactor: replace CryptoService with utils AES helpers and simplify encryption
- Remove infrastructure/crypto package (CryptoService, config, PBKDF2-based encryption)
- Replace with lightweight utils.GetAESKey/EncryptAESGCM/DecryptAESGCM helpers
- Remove cryptoSvc dependency from modelService, tenantService, weKnoraCloudService
- Remove crypto state persistence logic from container initialization
- Add TruncatePromptTokens field to WeKnoraCloud embed request
- Update frontend hint text for WeKnoraCloud credential errors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 20:55:39 +08:00
begoniezhao 2c0c1b8bab feat: integrate ChatBot provider and docreader 2026-04-13 17:47:24 +08:00
bingxiang.cheng c21d1edd80 fix(oss): 更新OSS S3兼容模式的注释,修复路径样式描述并禁用自动校验和计算 2026-04-13 15:37:33 +08:00
bingxiang.cheng 12001f23bf fix(oss): 禁用OSS S3兼容API的自动校验和计算,修复签名不匹配问题并调整路径样式设置 2026-04-13 15:37:33 +08:00
bingxiang.cheng 00a09d6c56 fix(oss): 禁用OSS S3兼容API的自动校验和计算,以防止签名不匹配 2026-04-13 15:37:33 +08:00
bingxiang.cheng 3bd56c17a8 feat(oss): register OSS in factory and container initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:34:28 +08:00
bingxiang.cheng a584df052d feat(oss): implement OSS FileService using S3-compatible mode
Add oss.go with full FileService implementation (SaveFile, SaveBytes,
GetFile, DeleteFile, GetFileURL, CheckConnectivity) using aws-sdk-go-v2
with path-style addressing for Aliyun OSS. Includes temp bucket support
and unit tests for parseOssFilePath and client creation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:34:28 +08:00
Jhao 45ad36845e feat(web-search): add baidu web search provider support #907 2026-04-12 20:13:05 +08:00
wizardchen a655a25cf8 fix: revert to neo4j v6 and workaround Go 1.24 Windows compiler bug with -p=1
Go 1.24 has an internal compiler race condition on Windows that causes
'package without types' errors during parallel compilation. Setting
GOFLAGS="-p=1" serializes the build and avoids triggering the bug.
2026-04-11 22:08:02 +08:00
wizardchen 9525f08038 fix: downgrade neo4j driver from v6 to v5 to fix Go 1.24 Windows internal compiler bug 2026-04-11 21:59:03 +08:00
wizardchen 9a2b54536d fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
MaoMengww aaee5f044a feat: attachment_support 2026-04-10 12:49:20 +08:00
Jhao fd18264183 feat(web-search): add ollama web search provider support #907 2026-04-10 01:27:29 +08:00
DaWesen b80bdb1079 feat: add video multimodal functionality 2026-04-09 19:19:42 +08:00
liweixin 2eb2dea634 feat: add Azure OpenAI support for embedding models 2026-04-09 19:19:25 +08:00
liweixin 3cfd034eec feat: add Azure OpenAI support for VLM models 2026-04-09 19:19:25 +08:00
liweixin 22ec575727 feat: add Azure OpenAI support for chat models 2026-04-09 19:19:25 +08:00
ochan.kwon 69785ad0a3 feat: add VectorStore entity, repository, and migrations
Introduce the VectorStore domain model as a foundation for multi-store
vector DB architecture (Phase 1, PR 1/4).

- VectorStore entity with ConnectionConfig (AES-GCM encrypted) and IndexConfig
- VectorStoreRepository interface and GORM implementation (CRUD + duplicate check)
- PostgreSQL migration (000031) and SQLite migration (000001)
- Unit tests for types, encryption, validation, and helpers
2026-04-09 01:39:51 +08:00
nullkey 2cd979e1ea feat(notion): implement Connector interface, markdown renderer, and DI registration
- Implement Validate, ListResources, FetchAll, FetchIncremental
- Per-record database sync with individual knowledge items
- Incremental sync with record-level change detection for databases
- Tree-structured resource listing with parent-child relationships
- Markdown renderer for all Notion block types including nested content
- Register NotionConnector in DI container
- Add unit tests for connector and markdown
2026-04-08 16:31:25 +08:00
AndyYang d5ecc150e0 feat(agent): support customizable LLM call timeout and add docker-compose mapping 2026-04-07 11:26:52 +08:00
nullkey dc5046ad2a feat(pipeline): inject QuotedContext at LLM prompt stage 2026-04-03 11:34:22 +08:00
wizardchen 9bb776a91c feat(chat): refine parent-child chunk replacement logic in merge process
- Enhanced the logic for replacing child content with parent content to only apply to text chunks whose parent is a parent_text chunk.
- Added checks to skip non-text chunks and ensure that only appropriate parent chunks are used for replacement, improving content quality and integrity.
2026-04-02 21:42:47 +08:00
wizardchen c93f4e94ae feat(asr): add ASR model connection check and update related components
- Implemented a new function to check ASR model connectivity via the /v1/audio/transcriptions endpoint.
- Updated ModelEditorDialog.vue to integrate ASR model checks and adjust UI elements accordingly.
- Enhanced localization files to reflect changes in ASR terminology and descriptions.
- Added ASR configuration options to the knowledge base and updated related migration scripts.
- Refactored model handling to support ASR as a distinct model type in the application.
2026-04-02 21:27:27 +08:00
wizardchen 624e24ba49 feat(asr): integrate Automatic Speech Recognition (ASR) support
- Added ASR configuration options to the knowledge base and model settings.
- Implemented ASR model selection and transcription capabilities in the knowledge processing pipeline.
- Enhanced file upload support to include audio formats for ASR processing.
- Updated localization files to include ASR-related labels and descriptions.
- Refactored model handling to accommodate ASR as a new model type.
2026-04-02 21:27:27 +08:00
wizardchen cb36570cde feat(summary): enhance document summarization capabilities and UI
- Added max_input_chars configuration to limit input size for summary generation.
- Updated generate_summary.yaml to clarify summary generation steps and requirements.
- Introduced a new summary section in the document content view to display generated summaries or loading states.
- Refactored doc-content.vue to utilize computed properties for improved performance and readability.
- Enhanced knowledge base hooks to include description and summary status fields for better data handling.
- Updated internationalization files to include new summary-related labels in multiple languages.
2026-04-02 18:34:39 +08:00
wizardchen 9d021a29b7 feat(elasticsearch): enhance ID field handling with dynamic suffix detection
- Added logic to determine whether to append the ".keyword" suffix to ID fields based on index mapping.
- Introduced `detectFieldTypes` method to inspect index mappings and set the `useKeywordSuffix` flag accordingly.
- Updated methods to utilize the new `idField` function for consistent ID field querying across the repository.
- Improved logging for better visibility into mapping detection and suffix usage.
2026-04-02 15:26:34 +08:00
ochan.kwon 33e919cc80 fix(knowledge): delete extracted images from storage when knowledge is removed
Previously, only the original uploaded file was deleted via knowledge.FilePath.
Images extracted by the builtin parser (stored as individual S3/MinIO/local files
via ImageResolver) were never cleaned up, causing orphan files to accumulate.

- Add ListImageInfoByKnowledgeIDs repository method (no chunk_type filter)
- Add collectImageURLs utility with URL deduplication and JSON parse logging
- Add deleteExtractedImages helper (standalone, best-effort with error logging)
- Fix DeleteKnowledge: collect image URLs before errgroup, delete in file goroutine
- Fix DeleteKnowledgeList: bulk collect with KB-level fileSvc mapping
- Fix cleanupKnowledgeResources: collect before chunk deletion, delete after
- Fix ProcessKBDelete: collect before chunk deletion for KB-wide cleanup
2026-04-02 12:04:04 +08:00
ochan.kwon 8a375749f1 fix(i18n): use WEKNORA_LANGUAGE env for prompt language fallback instead of hardcoded zh-CN
- Add DefaultLanguage() that reads WEKNORA_LANGUAGE env with zh-CN fallback
- Replace hardcoded zh-CN in LanguageNameFromContext with DefaultLanguage()
- Unify middleware/language.go to use types.DefaultLanguage()
- Add Language field to DocumentProcessPayload (8 enqueue sites + worker restore)
- Add Language field to ImageMultimodalPayload (enqueue + worker restore)
- Propagate language from ImageMultimodalPayload to QuestionGenerationPayload
2026-04-02 12:01:25 +08:00
wizardchen 3756c7c796 feat(session): restrict @mentions to agent's allowed knowledge base scope
- Implemented logic to filter user-provided KB and knowledge IDs based on the shared agent's allowed scope.
- Added a new method, restrictMentionsToAgentScope, to validate and restrict @mentions, preventing unauthorized access to knowledge bases.
- Enhanced logging to provide visibility into blocked mentions and the filtering process.
2026-04-02 10:31:20 +08:00
wizardchen b1fe7abbe1 feat(knowledge): enhance knowledge tag batch update with authorization checks
- Updated UpdateKnowledgeTagBatch method to include an authorizedKBID parameter for restricting updates to specific knowledge items.
- Implemented validation to ensure that all requested knowledge IDs belong to the authorized knowledge base.
- Enhanced GetKnowledgeBatch and UpdateKnowledgeTagBatch methods to incorporate agent-specific knowledge base access restrictions.
- Added utility functions to resolve allowed knowledge base IDs for agents based on their configuration.
2026-04-02 10:31:20 +08:00
wizardchen cb504e35b8 refactor(knowledge): remove unnecessary URL resolution for image paths 2026-04-01 21:59:22 +08:00
wizardchen 8e1cfaccb7 refactor(prompt_templates): improve question generation guidelines and context handling
- Updated the question generation template to clarify the role of surrounding context and main content.
- Enhanced quality rules for generated questions to better align with user search intent.
- Revised output format and added explicit instructions on what not to generate.
- Improved logging and output in the web parser for better visibility of parsed content and metadata.
2026-04-01 21:59:22 +08:00
Windfarer 1b7070c1a2 simplify oidc callback result payload 2026-04-01 20:47:18 +08:00
ochan.kwon 258a786129 Revert "refactor(elasticsearch): remove ".keyword" suffix from ID fields in Elasticsearch queries"
This reverts commit 3daaabfc4a.
2026-04-01 20:41:24 +08:00
wizardchen c4f5db7e88 feat(metadata): enhance document processing to include metadata extraction and handling
- Updated DocReaderServicer to pass metadata in responses.
- Modified PipelineParser to accumulate and merge metadata from all parsers.
- Enhanced StdWebParser to extract and log the title from web page content.
- Implemented logic in knowledge service to update knowledge title based on extracted metadata.
2026-04-01 15:49:47 +08:00
nullkey 0b64ef9bd1 fix(security): support IPv6 in SSRF validation via whitelist mechanism
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
2026-03-31 20:45:15 +08:00
wizardchen a5ca9a2784 feat(web-search): refactor web search provider
- Updated relevant files to include provider registration, implementation, and metadata.
- Enhanced frontend components to support provider management and configuration.
- Added localization for new provider settings and messages.
- Implemented backend repository methods for CRUD operations on web search providers.
2026-03-31 20:22:34 +08:00
Tavily PR Agent b84ca0b72a feat: add Tavily as web search provider option 2026-03-31 12:24:39 +08:00
wizardchen 1f01183057 feat(database): enhance DatabaseQueryTool with search scope filtering
- Updated DatabaseQueryTool to accept search targets during initialization.
- Implemented search scope filtering in SQL validation to restrict queries to specified knowledge bases and documents.
- Enhanced the SQL validation process to include new filtering options for improved security and query accuracy.
2026-03-31 10:52:01 +08:00
wizardchen d2d1d6d59a feat(datasource): enhance DataSourceSettings and DataSourceSyncLogs functionality
- Implemented polling mechanism in DataSourceSettings for real-time updates on data source synchronization status.
- Improved loading state management and user feedback during data source operations.
- Added load more functionality in DataSourceSyncLogs for better log retrieval experience.
- Updated UI elements for consistency, including icon changes and styling adjustments for better visual clarity.
2026-03-30 20:46:07 +08:00
wizardchen 442c340e7f feat(datasource): enforce tenant isolation and knowledge base ownership
- Added checks to ensure that the tenant ID matches between data sources and knowledge bases during creation and updates.
- Implemented ownership verification for data sources and knowledge bases in the DataSourceHandler, enhancing security and access control.
- Refactored data source retrieval methods to centralize ownership validation, improving code maintainability and clarity.
- Updated error handling to provide more informative responses when access is denied or when knowledge bases are not found.
2026-03-30 20:46:07 +08:00
wizardchen 1a340a8374 refactor(datasource): remove WeCom document integration and related components
- Deleted WeCom document connector and callback handling code to streamline the datasource management.
- Removed associated localization entries and UI elements related to WeCom integration.
- Updated data source management logic to eliminate references to WeCom, enhancing maintainability and clarity.
- Adjusted error handling and logging to reflect the removal of WeCom-specific functionality.
2026-03-30 20:46:07 +08:00
wizardchen 53bd412cf6 fix(datasource): enhance error handling in ProcessSync method
- Improved logging for missing data sources, providing clearer warnings when a data source is not found.
- Updated sync log status to 'canceled' when a data source is deleted, ensuring accurate tracking of sync operations.
- Adjusted error handling to prevent unnecessary error returns, enhancing overall process stability.
2026-03-30 20:46:07 +08:00