Commit Graph

37 Commits

Author SHA1 Message Date
wizardchen ce8969ab14 fix(initialization): surface upstream and SSRF errors verbatim in test-connection responses
Test-connection's "连接失败" was nearly useless for debugging because
two layers conspired to swallow the actual upstream error:

Backend swallow
---------------
checkChatModelConnection and CheckASRModel pattern-matched on the
upstream error string (401 / 404 / timeout / ...) and replaced it with
a fixed Chinese hint. The original SDK message (request URL, response
body, provider's error code description) was discarded.

Refactor: extract classifyConnectionError(errMsg) — returns just the
short Chinese hint. Callers combine "<hint>:<raw err>" so the hint
indicates "where to start looking" and the raw error shows "what
actually happened". The status-400 special-case (endpoint reachable,
parameter mismatch only) still returns success.

Frontend swallow
----------------
ModelEditorDialog.testConnection ignored result.message on success
paths (fine), but also overwrote it with a generic i18n string on
failure paths and 4xx rejections — even though axios's interceptor
already lifts the backend's error.message to error.message. Failure
branch now uses error?.message || result.message as the toast content;
i18n fallback only when neither is present.

SSRF allowlist hint
-------------------
SSRF validation failures previously returned
"Base URL 未通过安全校验: <reason>" with no mention of how to recover
(the allowlist is server-side, not configurable in the UI). Operators
hit it and were stuck.

New utils.FormatSSRFError(label, rawURL, err) wraps the error with
guidance to add the host to SSRF_WHITELIST_EXTRA (the operator-facing
append-only escape hatch — SSRF_WHITELIST itself is the project's
baseline). The hint echoes the failed host so it can be copy-pasted
straight into the env var:

  Base URL 未通过安全校验:DNS resolution failed for hostname
  test.local.com ... 如该地址确实可信,请联系运维在服务端环境变量
  SSRF_WHITELIST_EXTRA 中加入该主机(支持精确域名 / *.example.com
  通配 / IP / CIDR),示例:SSRF_WHITELIST_EXTRA=test.local.com,
  *.example.com,10.0.0.0/8

All SSRF call sites switch to the helper:
  initialization (CheckRemoteModel / TestEmbedding / CheckRerank /
  CheckASR / TestMultimodal / batch validation), model CRUD,
  mcp_service CRUD, knowledge URL ingest, system ReconnectDocReader.
2026-05-17 15:27:52 +08:00
wizardchen 6f95c75ed2 feat(system-info): surface DB migration errors with troubleshooting links
When a startup database migration fails (e.g. issue #1319: pg_trgm not
available so 000041 cannot build its trigram index), the application
intentionally keeps booting so operators can reach the UI to diagnose.
However, before this change the system info page silently dropped the
"DB Version" row because the value was empty:

  - migration.go only cached the version after a successful m.Up(); the
    error path returned early and left migrationVersionSet=false.
  - system.go used CachedMigrationVersion's ok=false to skip emitting
    db_version, and the JSON tag was already omitempty.
  - SystemInfo.vue gated the entire row on v-if="systemInfo?.db_version".

The end result was the most useful diagnostic surface vanishing in the
exact failure mode that needs it most — Wiki ingest and KG features
would silently produce nothing with no UI hint.

Changes:

  - migration.go: replace sync.Once-based setter with an RWMutex-guarded
    state struct holding {version, dirty, err}. Every failure path now
    calls captureMigrationFailure(m, err), which best-effort reads
    m.Version() so the cached value still reflects the partial state.
  - system.go (handler): always emit db_version (falling back to
    "unknown" when no version could be read), append " (failed)" when an
    error is recorded, and add db_migration_error to the response.
  - swagger / client SDK: keep the API contract in sync with the new
    response field.
  - SystemInfo.vue: render the DB version row whenever either field is
    present, show a "Migration failed" danger tag, and add a full-width
    alert below the row carrying the error message plus two links:
      1. View troubleshooting guide -> new docs/migration-troubleshooting.md
      2. Report an issue -> github.com/Tencent/WeKnora/issues/new,
         prefilled with the captured error and environment metadata.
  - docs/migration-troubleshooting.md: new self-service guide covering
    the common failure modes (missing extension, dirty state, privileges,
    out of disk, schema drift) with concrete psql / make commands.
  - i18n: add the new keys to zh-CN, en-US, ko-KR, ru-RU.

Refs #1319.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-14 16:34:50 +08:00
voidborne-d 46cab52fb0 fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117)
knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.

- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
  StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
  Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
  SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
  so fall-through to the tenant-merge path is intentional rather than an
  unrecognised provider sliding past the switch. The legacy `cos_config` column
  predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
  handler/system.go StorageEngineStatusItem.Name comment now list all 7
  providers (matched against types.StorageEngineConfig and ParseProviderScheme).

Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
  population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
  regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
  StorageConfig data is only consumed for cos/minio and skipped for the
  others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.

Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
  a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
  the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
  fixing it requires API surface additions; can ship separately.

Closes #1117 (the buildStorageConfig switches; remaining items deferred).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 17:09:42 +08:00
ChenRussell 42348eb40d feat(ks3): add KS3 object storage support
Add Kingsoft Cloud KS3 as a supported storage provider, including
  backend service implementation, storage config UI, and provider
  scheme recognition throughout the codebase.
2026-05-05 00:22:27 +08:00
Windfarer 20be935fe0 Merge branch 'main' into storage-allow-list 2026-04-29 19:35:13 +08:00
Windfarer 6601a483be feat: add STORAGE_ALLOW_LIST env var 2026-04-29 18:58:09 +08:00
wizardchen f07561f6b6 fix: resolve failing test and build errors
Made-with: Cursor
2026-04-22 21:17:29 +08:00
wizardchen 6235a82183 Fix logger format string errors in system handler 2026-04-16 18:39:30 +08:00
wizardchen 491c324197 feat(system): remove MinIO list buckets API and UI dependencies 2026-04-16 18:13:21 +08:00
wizardchen a78182cfc7 feat(handler): validate bucket name format and improve MinIO error handling
- Added validation for bucket name format to ensure it meets specified criteria, returning an error message for invalid formats.
- Simplified error messages for MinIO client creation and bucket auto-creation failures, enhancing clarity for users.
- Updated success messages to reflect the correct status of bucket creation without public-read policy settings.

This update improves the robustness of the MinIO integration and enhances user feedback during storage checks.
2026-04-16 18:13:19 +08:00
wizardchen 3f8201cd49 refactor(weknoracloud): update credential handling and configuration
- Replaced references to Docreader credentials with WeKnoraCloud credentials across multiple services and handlers.
- Introduced a new CredentialsConfig structure to manage third-party provider credentials, specifically for WeKnoraCloud.
- Updated database schema to include a credentials column for storing WeKnoraCloud AppID and AppSecret.
- Enhanced methods for retrieving and utilizing WeKnoraCloud credentials, ensuring proper encryption and decryption during storage and retrieval.

This update improves the management of WeKnoraCloud credentials, streamlining the integration and enhancing security measures.
2026-04-15 11:37:24 +08:00
wizardchen 0eff63b308 feat(weknoracloud): implement credential management and status checks
- Introduced a new WeKnoraCloudSettings component for managing APPID and APPSECRET.
- Added functionality to save WeKnoraCloud credentials without automatically creating models.
- Implemented credential status checks across various components to provide user feedback on configuration status.
- Updated API endpoints to handle credential saving and status retrieval.
- Enhanced UI elements to guide users in configuring WeKnoraCloud settings.

This update improves the integration of WeKnoraCloud by allowing users to manage their credentials more effectively and receive real-time feedback on their configuration status.
2026-04-14 17:36:01 +08:00
bingxiang.cheng 4054715d54 fix: add missing closing brace in checkOSS function 2026-04-14 10:03:45 +08:00
peterchan2020 8b9094f87c Merge branch 'main' into feat_oss 2026-04-14 00:41:29 +08:00
begoniezhao 2c0c1b8bab feat: integrate ChatBot provider and docreader 2026-04-13 17:47:24 +08:00
bingxiang.cheng d11e043142 refactor(storage): 优化OSS存储初始化与URL格式,提升安全性和兼容性
- 添加判断oss2是否安装,未安装时打印错误日志
- 修改OSS桶不存在时不自动创建,避免误创建公共读桶
- OSS下载URL改为虚拟主机风格,提升兼容性
- OSS初始化错误、缺失配置时添加详细日志提示
- 其他存储类代码重排统一风格,提升可维护性
2026-04-13 15:34:28 +08:00
bingxiang.cheng f05298d934 feat(oss): add OSS status reporting and connectivity check handler 2026-04-13 15:34:28 +08:00
wizardchen a655a25cf8 fix: revert to neo4j v6 and workaround Go 1.24 Windows compiler bug with -p=1
Go 1.24 has an internal compiler race condition on Windows that causes
'package without types' errors during parallel compilation. Setting
GOFLAGS="-p=1" serializes the build and avoids triggering the bug.
2026-04-11 22:08:02 +08:00
wizardchen 9525f08038 fix: downgrade neo4j driver from v6 to v5 to fix Go 1.24 Windows internal compiler bug 2026-04-11 21:59:03 +08:00
nullkey 0b64ef9bd1 fix(security): support IPv6 in SSRF validation via whitelist mechanism
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
2026-03-31 20:45:15 +08:00
wizardchen 139a9c40ff feat: enhance localization and configuration support
- Added timezone and language settings to the environment configuration.
- Introduced built-in agent configurations with multilingual support for various agents.
- Updated Docker Compose to utilize new environment variables for timezone and language.
- Created new prompt templates for question generation, summary generation, and keywords extraction.
2026-03-17 22:23:24 +08:00
ChenRussell d89e5db0f6 git commit -m "fix: clarify MinIO endpoint format error message
Add specific error handling for 'Endpoint url cannot have fully qualified paths'
to guide users to remove http:// or https:// prefix from endpoint URL."
2026-03-16 21:50:07 +08:00
DaWesen e309e0bed8 feat(storage): 集成S3存储适配器
添加对AWS S3及兼容存储服务的支持:
- 实现完整的S3FileService接口
- 支持文件上传、下载、删除功能
- 添加配置支持和环境变量检查
- 实现连接测试功能
- 遵循与其他存储适配器相同的代码风格
2026-03-09 10:39:46 +08:00
ochan.kwon 5583bf0eb4 fix(i18n): enable multilingual support for Knowledge Graph settings
- Replace hardcoded tag options with English equivalents (Content,
  Culture, Person, etc.) so random tag generation works across locales
- Replace default extract example with English text (Romeo and Juliet)
  matching the existing config.yaml extract_graph examples
- Update graph database status string to use locale-independent value
  and align frontend comparison accordingly
- Convert error messages in fabri text handler to English
2026-03-09 10:35:41 +08:00
wizardchen 76a5b329f3 feat: enhance storage engine connectivity check with auto-creation of buckets
- Updated the StorageCheckResponse to include a new field `bucket_created` to indicate if a bucket was automatically created during the connectivity check.
- Modified the MinIO connectivity check logic to attempt auto-creation of the bucket if it does not exist, setting a public-read policy if successful.
- Adjusted the frontend to reflect the new response structure, updating the UI to display a 'created' status when a bucket is auto-created.

These changes improve the user experience by providing clear feedback on bucket creation during storage checks, enhancing the overall functionality of the storage engine settings.
2026-03-05 18:14:14 +08:00
wizardchen b1fa96c175 refactor: enhance SSRF safety checks in MinerUCloudReader
- Removed hardcoded IP address from blocked hosts in `system.go`.
- Integrated SSRF safety checks in `MinerUCloudReader` for base URL and zip URL validation.
- Updated HTTP client initialization to use a new utility function for SSRF-safe HTTP clients, improving security during API requests.

These changes enhance the security of the MinerUCloudReader by ensuring that potentially unsafe URLs are properly validated before use.
2026-03-02 21:21:49 +08:00
wizardchen aefa1c6fe8 feat: enhance system information display with database version
- Added `db_version` field to the `SystemInfo` interface to expose the current database migration version.
- Updated the system information response to include the database version, reflecting its state during application runtime.
- Enhanced the UI in the SystemInfo component to display the database version with appropriate labels and descriptions in multiple languages.

This update improves transparency regarding the database state within the system information settings.
2026-03-02 21:21:49 +08:00
wizardchen 469f320d10 feat: enhance storage engine configuration and UI updates
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.

This update enhances the flexibility of storage options and improves user experience with better UI interactions.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
begoniezhao 389900af7e refactor: Refactor retriever engine checks and expose mapping 2026-01-12 17:45:26 +08:00
wizardchen 8bb98636a7 refactor: 重构桶策略解析逻辑以支持复杂策略 2025-12-30 14:35:10 +08:00
wizardchen f9593fc050 feat: 支持列出MinIO 存储桶及权限 2025-12-30 14:35:10 +08:00
wizardchen a7df6900eb feat: 完善Swagger API文档生成功能 2025-12-17 22:04:18 +08:00
wizardchen a879bbdbb5 feat: Integrate Neo4j driver into SystemHandler for enhanced graph database support
- Updated SystemHandler to include a Neo4j driver, allowing for improved interaction with graph databases.
- Modified NewSystemHandler to accept the Neo4j driver as a parameter.
- Adjusted getGraphDatabaseEngine method to return the appropriate engine name based on Neo4j driver availability.
2025-11-20 16:35:07 +08:00
wizardchen da5f71dfd3 feat: Add MinIO and graph database status handling with localization updates
- Introduced a new boolean field for MinIO status in the system info API response.
- Updated localization files in English, Russian, and Chinese to include warnings and instructions related to MinIO and graph database configurations.
- Enhanced the KBAdvancedSettings component to display alerts and disable options based on the status of MinIO and graph database.
- Implemented logic to check system status on component mount and adjust settings accordingly.
2025-11-17 11:38:02 +08:00
wizardchen 667239a625 feat: Add system information fields and localization for new engines
- Introduced new fields for keyword index engine, vector store engine, and graph database engine in the system info API response.
- Updated localization files in English, Russian, and Chinese to include labels and descriptions for the new system information fields.
- Enhanced the SystemInfo.vue component to display the new engine information in the settings view.
2025-11-16 23:46:33 +08:00
wizardchen 98e47e83d2 feat(ui): Support multi knowledgebases operation 2025-09-17 16:02:08 +08:00