Commit Graph

37 Commits

Author SHA1 Message Date
wizardchen f8b3e5f148 feat(storage): add multi-instance storage backends per workspace
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
2026-07-15 14:54:39 +08:00
wizardchen 41975abcc5 refactor(terminology): rename user-facing tenant labels to workspace
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
2026-07-14 16:14:43 +08:00
wizardchen c1cddb4d4d Add instruction fields to knowledge base configuration 2026-07-13 12:16:19 +08:00
wizardchen 1a264e09c1 fix(security): harden secret redaction follow-ups for PR #1884
Restore PreserveIfRedacted merge on tenant KV PUT so Admin settings saves
no longer overwrite stored secrets with placeholder values. Redact cross-
tenant tenant listings, tighten Viewer MCP/model responses, and align init
config frontend types with credentials metadata.
2026-07-02 14:41:42 +08:00
wizardchen 53e0333a53 Add embedding dimension override support 2026-06-15 19:18:19 +08:00
wizardchen dda16436d0 feat(lkeap): integrate Tencent Cloud LKEAP Rerank support
- Added support for Tencent Cloud LKEAP Rerank, including new configuration options for SecretId and SecretKey.
- Updated frontend components to handle LKEAP-specific fields and enhance user experience with appropriate labels and placeholders.
- Enhanced backend logic to manage LKEAP credentials and integrate with the new rerank API.
- Improved localization for LKEAP-related terms across multiple languages, ensuring clarity and consistency in user-facing messages.
- Introduced tests for LKEAP Reranker functionality to ensure reliability and correctness in handling requests.
2026-06-04 17:22:18 +08:00
Li Xianggang b3dbf86bf5 feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
wizardchen b9a1ce7617 fix(rbac): drop X-Tenant-ID short-circuit in SSE / fetch paths
The previous PR removed the
"selectedTenantId === weknora_tenant.id → skip header" short-circuit
from the axios interceptor, but the same copy-pasted logic lived in
three more places that bypass axios:

- `api/chat/streame.ts` — `fetchEventSource` for the streaming chat
  endpoint. Without this fix the chat reply would 404 with
  "流式连接失败:HTTP 404" on the first message after switching to
  a non-home tenant: the SSE call landed on the home tenant where
  the freshly-created session did not exist.
- `api/initialization/index.ts` — native `fetch` for multimodal
  test (FormData) requests. Misroutes the test to the home tenant.
- `utils/security.ts` — `getProtectedFileRequestHeaders` used by the
  Markdown image proxy. Authenticated file fetches landed on the
  home tenant, breaking image rendering after a tenant switch.

All three now mirror the simplified rule: if `weknora_selected_tenant_id`
is set, attach `X-Tenant-ID` unconditionally. The server's
`IsTenantAccessible` already accepts a header pointing at the user's
home tenant, so always attaching is safe.

Same root cause as the axios fix: any code path that writes the
active tenant into `weknora_tenant` (UserMenu's `loadUserInfo`, OIDC
sync, router hydrate) made the two values equal, after which these
non-axios paths silently dropped the header and re-routed to home.
2026-05-18 17:28:58 +08:00
wizardchen a9d37970ae fix(initialization): fall back to stored apiKey when test-connecting an existing model
Direct regression from the previous credential subresource refactor: the
edit-mode ModelEditorDialog stopped binding apiKey into formData (the
field now lives in <CredentialResource>), so "Test connection" sent
apiKey="" and every upstream call (DeepSeek / lkeap / OpenAI etc.) 401'd.

Fix in the backend so the frontend stays simple:

- ModelTestRequest gains an optional ModelID field.
- New fillSecretsFromStoredModel helper: when ModelID is set AND a secret
  field on the request is empty, copy it from the stored (decrypted) model.
  Non-empty request values still win, so users typing a new key to verify
  it before saving behave the same as before.
- The four test entrypoints (chat / embedding / rerank / asr) call the
  helper right after ShouldBindJSON.

Frontend: ModelEditorDialog.testConnection forwards modelId when isEdit
is true. The four API client signatures accept the optional field.

WebSearch and MCP are unaffected: WebSearch already has a TestProviderByID
ID-based path, MCP has no standalone raw test endpoint.
2026-05-17 15:27:52 +08:00
Claude beb7d005e9 fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness
Resolves the issues surfaced by the post-Phase-3 self-review.

High severity
- Strategy / TokenLimit / Languages can now be reset. The KB-config
  handler uses pointer DTOs so the absence of a field in the payload
  means "no change" while an explicit empty string / 0 / [] clears
  the value. The frontend now sends the fields unconditionally so
  the user can revert to defaults via the clear icon — previously
  these fields were write-once.
- DocProfile is no longer computed twice in the preview endpoint.
  Diagnostics carries the profile that drove tier selection (when
  auto-strategy ran); the handler only re-profiles for explicit
  strategies that bypass the profiler.
- Chunk-size stats (avg / min / max / stddev) are now computed over
  the FULL chunk set before the response is truncated to
  previewMaxChunks. Avoids systematically misleading metrics for
  large documents.
- SplitWithDiagnostics defaults SelectedTier to TierLegacy so an
  empty-text request never leaves the diag with a blank tier string.

Medium severity
- Split() reverts to its own loop instead of delegating to
  SplitWithDiagnostics. Diagnostics struct allocation is reserved
  for callers that actually want the trace (preview endpoint).
  Matters in SplitParentChild where Split runs once per parent.
- Preview endpoint runs the splitter on a goroutine wrapped in a
  5s context.WithTimeout. CJK input at the 256k-rune ceiling
  could otherwise tie up the worker for several seconds.

Low severity
- KBChunkingSettings emits arrays via spread copies so parent-side
  mutation cannot drift the form's reactive state.
- KBChunkingDebug font-family includes a monospace fallback so the
  chunk content stays monospaced even when the TDesign CSS variable
  is missing.

Documentation
- DocProfile JSON shape is now annotated as part of the public
  preview API; field renames would be breaking.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 46d9c396c6 feat(frontend): expose chunking strategy in KB editor
Surfaces the adaptive 3-tier chunking controls in the existing
"Chunking" section of the KB editor modal. Users can now pick a
strategy from a dropdown without crafting JSON for the API:

- Automatic (recommended)  → backend profiler picks the tier
- Markdown-optimized       → forces heading-aware splitting
- Smart structure detection → forces heuristic boundary detection
- Classic                  → original recursive splitter (legacy)

The fifth backend value `recursive` is intentionally hidden from the
UI — it's functionally near `legacy` and would dilute the choice.

Advanced options (collapsible, disabled when strategy=legacy):
- TokenLimit (0–8192 approx tokens; 0 = char-based budget)
- Languages multi-select (DE/EN/ZH; empty = auto-detect)

Behaviour:
- New KBs default to strategy="auto"
- Existing KBs without the field render with empty strategy and keep
  classic behavior on save unless the user actively picks one
- Overlap warning when overlap >= chunkSize/2

Wired through:
- KBChunkingSettings.vue: strategy select + advanced collapsible +
  ChunkingConfig interface extended
- KnowledgeBaseEditorModal.vue: form-state defaults, kb-load mapping,
  buildSubmitData (chunking_config) + updateKBConfig (documentSplitting)
  pass strategy/token_limit/languages
- api/initialization/index.ts: types extended
- i18n: en-US / zh-CN / ko-KR / ru-RU all carry strategy + advanced
  keys (6 keys each)

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
wizardchen fd3e2992b1 feat(models): support custom HTTP headers across all remote model calls
Introduce per-model `custom_headers` config (similar to OpenAI Python SDK's
`extra_headers`) so users can inject gateway auth tokens, trace IDs, etc.
into every outbound API request for chat / embedding / rerank / VLLM / ASR
models. Reserved headers like Authorization / Content-Type are always
preserved to avoid breaking auth or signing flows.

Along the way, unify production and "test connection" paths onto a single
`ConfigFromModel(*types.Model, appID, appSecret)` constructor in each
model package. Both `service.modelService.GetXxxModel` and the four
`handler.initialization.Check*Model` / `TestEmbeddingModel` endpoints now
go through the same field mapping, so new parameters only need to be
added in one place.

Backend:
- Add `CustomHeaders map[string]string` to `types.ModelParameters` with
  JSON/YAML `omitempty` tags for forward compatibility.
- New `internal/utils/extraheaders.go` providing `ApplyCustomHeaders`
  (for hand-rolled HTTP paths) and `WrapHTTPClientWithHeaders` /
  `CustomHeadersRoundTripper` (for SDK paths like go-openai). Reserved
  headers (Authorization, api-key, Content-Type, Accept, Host,
  Content-Length, User-Agent) are filtered out.
- Inject headers in every remote model path: chat (SDK + raw HTTP),
  embedding (openai/aliyun/jina/volcengine/nvidia/azure_openai), rerank
  (remote_api/aliyun/jina/nvidia/zhipu — via shared `customHeaderSetter`
  interface), VLM and ASR (via http.Client wrapping).
- Add `ConfigFromModel` to each of chat/embedding/rerank/vlm/asr,
  consolidating field mapping (ExtraConfig, CustomHeaders, InterfaceType
  defaulting, WeKnoraCloud credentials, etc.) and cover it with
  dedicated unit tests per package.
- Refactor `service/model.go`: replace ~100 lines of hand-written Config
  literals with one-line `ConfigFromModel` calls; drop the now-unused
  `stringMapToAnyMap` helper.
- Refactor `handler/initialization.go` test-connection endpoints: merge
  four ad-hoc request structs into one `ModelTestRequest`, extract
  `buildTestModel` and `resolveTenantWeKnoraCloudCreds` helpers, and
  route all four endpoints through `ConfigFromModel` + `NewXxx` so the
  test path is now behaviorally identical to production.

Frontend:
- Add `custom_headers` to the `ModelConfig` API type.
- `ModelEditorDialog.vue`: new Key-Value header editor (add/remove rows),
  auto-converts map <-> array when loading/saving, and passes the
  current header set to the `Test Connection` button so the preview
  reflects exactly what production will send.
- `ModelSettings.vue`: serialize the header array back into a map for
  the backend (drops empty rows).
- i18n: add custom-header labels / descriptions / placeholders to
  zh-CN, en-US, ru-RU, ko-KR.

Tests:
- `internal/utils/extraheaders_test.go` covers reserved-header filtering
  and round-tripper wrapping (including nil-client fallback).
- New `config_from_model_test.go` in chat / embedding / rerank / vlm /
  asr verifies all fields (CustomHeaders, ExtraConfig, InterfaceType
  defaulting, AppID/AppSecret) are propagated end-to-end.
2026-04-23 23:36:40 +08:00
liweixin 4e4118d406 fix: pass provider to connection test so Azure OpenAI uses correct endpoint 2026-04-09 19:19:25 +08:00
wizardchen c93f4e94ae feat(asr): add ASR model connection check and update related components
- Implemented a new function to check ASR model connectivity via the /v1/audio/transcriptions endpoint.
- Updated ModelEditorDialog.vue to integrate ASR model checks and adjust UI elements accordingly.
- Enhanced localization files to reflect changes in ASR terminology and descriptions.
- Added ASR configuration options to the knowledge base and updated related migration scripts.
- Refactored model handling to support ASR as a distinct model type in the application.
2026-04-02 21:27:27 +08:00
wizardchen 624e24ba49 feat(asr): integrate Automatic Speech Recognition (ASR) support
- Added ASR configuration options to the knowledge base and model settings.
- Implemented ASR model selection and transcription capabilities in the knowledge processing pipeline.
- Enhanced file upload support to include audio formats for ASR processing.
- Updated localization files to include ASR-related labels and descriptions.
- Refactored model handling to accommodate ASR as a new model type.
2026-04-02 21:27:27 +08:00
wizardchen 5996d07aa6 refactor: update LLM configuration to model ID in API and frontend
- Replaced LLMConfig with model_id in TextRelationExtractionRequest and FabriTextRequest interfaces.
- Updated related API handlers to retrieve chat models based on model_id.
- Adjusted frontend components to pass model_id instead of LLMConfig, enhancing clarity and simplifying the configuration process.

These changes streamline the model configuration process across the application, improving maintainability and usability.
2026-03-13 19:34:35 +08:00
ochan.kwon db7cef5f17 feat(i18n): replace hardcoded Chinese strings with i18n calls across frontend
Replace all user-visible hardcoded Chinese strings in Vue/TS source files
with proper i18n function calls and add corresponding translation keys
to all 4 locale files (zh-CN, en-US, ko-KR, ru-RU).

Changes:
- Replace 80+ hardcoded Chinese strings across 30+ source files
- Add ~450 new translation keys per locale file
- Cover components, views, utils, hooks, and API modules
- Remove Chinese fallback patterns (|| '中文') from existing t() calls
- Add mermaid toolbar and ollama date format translations

Intentionally preserved (backend API values / backward compatibility):
- FAQ CSV column headers and parsing constants
- RelevanceLevel type and Record key mappings
- Graph database engine status comparison ('未启用')
- Language name labels ('简体中文')
2026-03-06 10:34:08 +08:00
wizardchen b333499078 feat: implement parent-child chunking strategy for enhanced context management
- Added support for parent-child chunking in the knowledge base, allowing for a two-level chunking strategy that improves context retrieval.
- Introduced new configuration options for enabling parent-child chunking, including parent and child chunk sizes.
- Updated relevant components and services to handle parent chunk data, ensuring proper linking and retrieval of child chunks.
- Enhanced user interface elements to reflect the new chunking options and provide better user guidance.

These changes significantly improve the knowledge extraction process by providing richer context through hierarchical chunking.
2026-03-05 18:14:14 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
nullkey e9a2a7b3e5 feat: 新增对多家平台的模型适配 2025-12-29 12:18:42 +08:00
wizardchen 26267f6e79 feat: Implement cross-tenant access headers for chat and multimodal test APIs 2025-12-03 11:39:10 +08:00
wizardchen 16562eebbb feat: 添加AI问题生成功能,支持配置生成数量并在前端展示生成的问题 2025-12-03 01:42:31 +08:00
begoniezhao 15603dbac7 refactor: Clean up and organize code structure in initialization and request modules for improved maintainability 2025-12-01 14:48:33 +08:00
wizardchen c1f46df0dd feat: Update VLM configuration in knowledge base and API documentation
- Replaced `vlm_model_id` with a structured `vlm_config` object in knowledge base configurations, including `enabled` flag and `model_id`.
- Updated API documentation to reflect changes in VLM configuration structure.
- Enhanced frontend interfaces to accommodate new VLM configuration format.
- Improved localization for new features related to knowledge graph and multimodal capabilities.
2025-11-20 20:08:24 +08:00
wizardchen 5965afa899 feat: 移除知识库中的重排序模型配置,改为全局会话设置 2025-11-20 01:03:19 +08:00
wizardchen 7a64371603 feat: Implement theme management and UI enhancements across the application
- Added theme initialization and management in the UI store.
- Enhanced dark mode styles for various components, including tables, dialogs, and inputs.
- Updated multiple components to use CSS variables for consistent theming.
- Introduced a new simplified API for updating knowledge base configurations.
- Improved overall UI consistency and responsiveness with new styles.
2025-11-06 02:45:44 +08:00
wizardchen 4fa3adbf3b feat: Add agent configuration and cleanup scripts for database migrations 2025-11-05 23:18:44 +08:00
begoniezhao b6d8be634a feat: 新增异步任务提取服务 2025-10-16 17:48:21 +08:00
wizardchen 98e47e83d2 feat(ui): Support multi knowledgebases operation 2025-09-17 16:02:08 +08:00
wizardchen 0292b3ce36 chore(ui): Update Setting page 2025-09-16 20:18:47 +08:00
wizardchen e08ccb286c fix(ui): Fix CSP error 2025-09-16 14:58:22 +08:00
wizardchen 81bd2e6c2c feat: Add Login Page 2025-09-16 02:47:39 +08:00
wizardchen 2b6cbee1b6 feat: add aliyun rerank 2025-09-10 01:22:25 +08:00
wizardchen 4214e6782b feat: add aliyun rerank model 2025-09-10 01:22:25 +08:00
wizardchen dacdad4dac feat: Get Embedding demension from backend 2025-08-15 10:37:59 +08:00
wizardchen 8b43931886 feat: support minio storage 2025-08-14 12:16:08 +08:00
wizardchen bdabed6bfa feat: Added web page for configuring model information 2025-08-10 17:11:07 +08:00