Commit Graph

1851 Commits

Author SHA1 Message Date
helloandyzhang af2f55c29f fix(frontend): apply default context template when switching to quick-answer mode
Start new agents with an empty context_template so mode-switch logic can
populate the system default instead of keeping the smart-reasoning placeholder.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-26 20:13:31 +08:00
wizardchen 500c821817 feat(builtin-models): validate YAML entries and align ID length with schema
Three correctness fixes that the lifecycle PR deliberately deferred:

1. ID length / struct-tag drift
   - models.id is varchar(64) on both PG and SQLite (per the init
     migrations), but Model.ID's GORM tag said varchar(36) — a remnant
     from when the field only held UUIDs. The mismatch is harmless under
     golang-migrate (struct tag is ignored), but misleading on AutoMigrate
     paths and in IDE tooltips. Tag now matches the real column width.
   - New ModelIDMaxLen constant (=64) is the single source of truth for
     anyone accepting user-provided ids. The YAML loader uses it to
     reject too-long ids up front with a clear message instead of letting
     the INSERT explode with a generic "value too long for type" error.

2. Field validation in the YAML loader
   - Type, Source, and Status are typed strings but YAML can supply any
     value. Misspellings (e.g. `type: knowledgeqa` lowercase, `type: LLM`)
     were previously persisted as-is and produced rows that looked fine
     in the table but failed at provider-factory lookup time, which is
     hard to debug.
   - validateBuiltinModelEntry now checks: empty id, id length, empty
     type, type ∈ {KnowledgeQA, Embedding, Rerank, VLLM, ASR}, and
     status ∈ {active, downloading, download_failed, empty}. Source is
     intentionally NOT validated because the provider matrix in
     internal/models/* keeps growing and a strict allow-list here would
     force changes in two places per new provider.
   - Invalid entries are warned + skipped (not aborting the whole load),
     and excluded from the keep-set so the drift sweep does not delete
     existing matching rows on the strength of a typo'd YAML retry.

3. Magic number cleanup
   - DefaultBuiltinModelTenantID (=10000) replaces the hard-coded `10000`
     literal in toModel(). The invariant lives in three places already
     (PG migration, SQLite migration, this constant); naming it makes
     the cross-reference explicit and grep-able.

Tests:
- New TestLoadBuiltinModelsConfig_RejectsInvalidEntries with five
  sub-cases (id-too-long, missing-type, lowercase-type, unknown-type,
  unknown-status) asserts the table stays empty after each.
- All 11 existing tests still pass.
2026-05-26 11:37:03 +08:00
wizardchen 4bce10d1ea refactor(builtin-models): unify logging via stdlib log with stable prefix
Original PR #1453 used fmt.Printf, which lands as unstructured noise in
release/JSON log pipelines. The natural fix is logger.Infof/Warnf, but
internal/logger itself imports internal/types — using it from here
would create an import cycle (this is the same constraint that forces
model.go's crypto error logs onto stdlib log).

Switch all loader output to log.Printf with a stable "[builtin-models]"
prefix (mirroring the "[crypto]" convention already established in
model.go). The prefix gives operators a grep handle even though the
lines stay unstructured.

Levels are encoded inline as "WARN:" for warnings so log shippers and
humans can still discriminate.
2026-05-26 11:37:03 +08:00
wizardchen ea47dce337 fix(compose): use env_file array form for builtin_models compatibility
PR #1453 introduced the map form of env_file:

  env_file:
    - path: .env
      required: false

which is only recognised by Docker Compose v2.24+ (Jan 2024). Older
deploys would refuse to parse docker-compose.yml even when builtin_models
YAML is not enabled — breaking the "no impact unless opted in" promise.

Switch to the array form (`env_file: [.env]`) which has been supported
since the earliest Compose schemas. The trade-off is that Compose now
errors when .env is absent, so:

- scripts/start_all.sh already calls check_env_file (cp .env.example .env
  if missing) before docker compose up; that path is unaffected.
- The bare `make docker-run` / `make docker-restart` targets in the
  Makefile are taught the same fallback: touch / copy .env.example before
  invoking docker-compose, so fresh clones keep working.

The docker-compose.yml comment block is updated to explain the version
trade-off so future maintainers don't re-introduce the map form.
2026-05-26 11:37:03 +08:00
wizardchen 7b192e546f feat(builtin-models): reconcile YAML lifecycle with drift sweep
Extend the builtin_models.yaml loader so the YAML file becomes a complete
source of truth for the rows it owns. Builds on the previous commit's
managed_by column.

Lifecycle contract:
- Every UPSERTed entry is tagged managed_by="yaml".
- The DoUpdates list now includes deleted_at, so an entry that was
  soft-deleted (e.g. via UI/API) is automatically resurrected when it
  reappears in the file. Closes the "ghost row that exists but is
  invisible" failure mode.
- After all UPSERTs, the loader soft-deletes rows where
  managed_by='yaml' AND id NOT IN (current YAML id set). Removing an
  entry from YAML is now the supported way to retire a built-in model —
  no manual SQL needed.
- Rows tagged managed_by='' (UI/API/SQL-seeded built-ins) are invisible
  to the reconcile path and never touched.
- When a YAML entry sets is_default=true, the loader first clears
  is_default on any other rows in the same (tenant_id, type) bucket,
  mirroring the invariant enforced by the API path
  (repository.UnsetDefaultModel).

Failure handling stays defensive:
- File missing / not a regular file / parse error: warn and skip; the
  drift sweep is NOT executed so a malformed file cannot wipe rows.
- Per-entry UPSERT error: warn, drop the id from the keep-set so the
  sweep also leaves the existing row alone ("leave alone on failure").

Tests cover: file-missing, parse-error, basic upsert + defaults,
idempotency, ${ENV} interpolation (set vs unset), drift sweep removing
YAML rows, drift sweep ignoring manual rows, soft-delete resurrection,
is_default cleanup across tenant+type, explicit empty list sweeping all
yaml-managed rows, and a regression guard ensuring BeforeCreate does not
overwrite YAML-supplied stable ids.

Docs are rewritten so operators see "delete from YAML and restart" as
the supported removal path; SQL is retained only for the legacy
managed_by='' slice.
2026-05-26 11:37:03 +08:00
wizardchen fdc22fd7a5 feat(builtin-models): add managed_by column to models table
Introduce a `managed_by` varchar column on `models` so future declarative
loaders can claim ownership of a subset of rows without disturbing entries
created via the UI/API or seeded by hand-written SQL.

- versioned/000052_models_managed_by.{up,down}.sql add the column with a
  default of '' and a partial index on non-empty values to keep startup
  reconciliation cheap.
- sqlite/000000_init.up.sql is updated in place (the Lite init migration
  is a single file per project convention) so fresh SQLite databases get
  the column from the start.
- Model.ManagedBy mirrors the column. Existing rows default to '' which
  the YAML loader treats as "manually managed, never touch".

Schema half of the YAML-driven built-in-model lifecycle work that follows
up on #1453; the reconciler that uses the column lands in the next commit.
2026-05-26 11:37:03 +08:00
jackson.jia d439b3ae07 feat(builtin-models): add YAML-based declarative config with ${ENV} interpolation
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.

Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.

The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
2026-05-26 11:31:01 +08:00
wizardchen e9be53e830 fix: respect multi-turn-disabled flag in KnowledgeQA pipeline
When a custom agent has `MultiTurnEnabled=false`, `applyAgentOverridesToChatManage`
sets `chatManage.MaxRounds = 0` to signal "no history". Two pipeline plugins
mistreated this zero value as "use the global default" and silently re-loaded
session history into the LLM context:

- `PluginLoadHistory` fell back to `Conversation.MaxRounds` when
  `chatManage.MaxRounds == 0`.
- `PluginQueryUnderstand.loadHistory` had the same fallback, and even when
  `LOAD_HISTORY` was skipped it would re-populate `chatManage.History`,
  leaking previous turns into rewrite, image analysis, and the final answer.

The RAG branch in `session_knowledge_qa` also added `LOAD_HISTORY`
unconditionally, unlike the pure-chat branch which guarded it with `hasHistory`.

This change:

- Treats `chatManage.MaxRounds <= 0` as an explicit disable in both plugins;
  no fallback to global config.
- Makes the RAG pipeline consistent with the pure-chat path by gating
  `LOAD_HISTORY` on `hasHistory`.
- Removes the duplicated `Current Time: {{current_time}}` line from
  `agent_system_prompt.yaml`. The agent already receives a fresh
  `<runtime_context><current_time>` block with each turn from
  `observe.buildRuntimeContextBlock`, so the static placeholder was
  redundant.

The ReAct agent path (`session_agent_qa`) already checked `MultiTurnEnabled`
directly and is not affected.

Closes #1479
2026-05-26 10:45:42 +08:00
wizardchen 91e4fe8b6b feat(frontend): introduce KBSwitcherDropdown component for improved knowledge base selection
Added a new KBSwitcherDropdown component to enhance the user experience when selecting knowledge bases. This component displays a list of knowledge bases with the current selection highlighted, allowing users to easily switch between them. Integrated the dropdown into both KnowledgeBase.vue and FAQEntryManager.vue, replacing the previous dropdown implementation for a more streamlined and consistent UI. The new component supports dynamic updates and maintains the order of knowledge bases while ensuring the current selection is always visible at the top.
2026-05-26 10:20:56 +08:00
wizardchen 257766545e style(frontend): refine KBInfoPopover button styles for improved visual hierarchy
Updated the styles for the info button in KBInfoPopover.vue to enhance its appearance and interaction. Adjusted dimensions, colors, and hover effects to create a clearer distinction between the info button and the settings button. Added a warning state for better visual feedback when necessary. These changes aim to improve user experience and maintain consistency in the UI design.
2026-05-26 10:20:56 +08:00
wizardchen baae7dd806 fix(frontend): update KBInfoPopover to correctly display FAQ and document counts
Refactor the logic in KBInfoPopover.vue to differentiate between FAQ and document knowledge bases. The component now uses `chunk_count` for FAQ KBs and retains `knowledge_count` for document KBs, ensuring accurate representation of entry totals. This change aligns the display logic with the intended user-facing metrics for each KB type.
2026-05-26 10:20:56 +08:00
wizardchen f97d588042 refactor(frontend): extract KBInfoPopover and reuse it on FAQ KBs
Move the info popover (intro'd in the previous commit) into a
dedicated `frontend/src/components/KBInfoPopover.vue` component so
both the document KB header and the FAQ KB header can share the
exact same metadata surface.

- KnowledgeBase.vue: replace the inline popover and the now-dead
  helper computeds (kbLastUpdated, kbHasDistinctUpdate,
  infoCardCapabilities, infoCardChunking, infoCardStats,
  chunkingStrategyLabel, supportedFileTypesList, accessRoleLabel,
  accessPermissionSummary) and their CSS with a single
  <KBInfoPopover> tag. The page still fetches parser engines and
  forwards the supported file types so the "Accepted formats" row
  renders for document KBs.
- FAQEntryManager.vue: drop the legacy inline meta strip
  (role tag · share source · last updated) along with the
  accessRoleLabel / accessPermissionSummary / kbLastUpdated /
  effectiveKBPermission computeds it only served, and wire the
  same <KBInfoPopover> into the actions cluster next to the
  settings button. FAQ KBs don't pass supportedFileTypes so the
  format row simply hides itself.
- KBInfoPopover.vue: self-contained — re-derives access role,
  shared-binding and chunking config from kbInfo + the org/auth
  stores. supported-file-types remains an opt-in prop so callers
  decide whether the parser-engine pipeline is relevant.

No user-visible regressions: both pages now render the same
Basic / Access / Capabilities / Chunking / Stats / Storage
Binding sections, with rows hiding when they don't apply.
2026-05-26 10:20:56 +08:00
wizardchen de8ee5ae16 feat(frontend): consolidate KB metadata into an info popover
Replace the inline meta strip on the knowledge base detail header
(role tag · share source · last updated · vector store badge) and
the vector-store engine badge on the list cards with a single
structured "info" popover next to the settings button.

The popover groups Basic, Access, Capabilities, Chunking, Stats
and Storage Binding into separate sections, only rendering those
with data. A warning dot on the info button surfaces unreachable
vector-store bindings without occupying header real estate. The
popover caps at min(70vh, 560px) with an internal scrolling body
so it never overflows the viewport.

Hide the last-updated row when it equals created_at, since GORM
only bumps KB.updated_at on metadata edits — never on document
uploads — so surfacing two identical timestamps next to "created
at" was misleading. Once the KB metadata is actually edited the
row reappears naturally.

i18n entries added for zh-CN, en-US, ko-KR, ru-RU.
2026-05-26 10:20:56 +08:00
wizardchen 31471f5efa refactor(knowledgeService): update checkStorageEngineConfigured method for improved clarity and functionality
- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability.
- Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling.
- Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance.
2026-05-25 19:16:08 +08:00
wizardchen caeb6a44d8 refactor(chunk): remove VideoInfo field from Chunk struct
- Removed the VideoInfo field from the Chunk struct in chunk.go, streamlining the data model.
- This change reflects a shift in focus away from video information storage within the Chunk type.
2026-05-25 19:15:39 +08:00
wizardchen 13301ca026 feat(parser): enhance web parser with improved image extension handling and XPath prioritization
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
2026-05-25 19:15:17 +08:00
begoniezhao 5c0243cd92 feat: Support filtering chunks by multiple chunk_type params 2026-05-25 19:14:46 +08:00
wizardchen 8dc50b3921 fix(frontend): close card popover before opening delete confirm dialog
The per-card "more actions" popover (edit / rebuild / move / delete ...)
stayed open when clicking "Delete document", so the popover floated above
the t-dialog and visually obscured the confirmation dialog.

Reset the card's isMore flag and moreIndex inside delCard before toggling
delDialog so the popover closes synchronously with the dialog opening.
2026-05-25 17:11:20 +08:00
wizardchen 15845f4e62 fix(frontend): poll knowledge list after single-item delete
DeleteKnowledge is now an async pipeline on the backend, so refreshing the
list immediately after a successful response can still show the deleted
entry. Mirror the polling loop already used for batch delete: re-list up to
~12s (30 polls * 400ms) until the deleted id disappears, then refresh tags.

Without this the UI shows "delete success" but the list visibly contains
the entry until the user manually refreshes.
2026-05-25 17:11:20 +08:00
wizardchen d12fb42e60 refactor(knowledge): route single delete through async pipeline
Reuse enqueueKnowledgeListDelete inside DeleteKnowledge so that single-item
delete shares the same hardening as BatchDeleteKnowledge / ClearKnowledgeBase:
asynq retries, business-aware queue routing, and marking-as-deleting inside
the worker.

The endpoint now returns 200 once the delete task has been enqueued; the
response body carries the asynq task_id and the message is updated to
"Delete task submitted". Swagger annotations, generated docs and the Go
client SDK comment are updated to reflect the new asynchronous semantics.

Note: this is a behavior change. Callers that previously assumed the
knowledge was already gone on a 200 response should poll the task status
or accept eventual consistency, matching the existing BatchDeleteKnowledge
contract.
2026-05-25 17:11:20 +08:00
chenwenhui af1f2b469a 增加sandbox对windows编译支持,现在默认是linux的实现,windows直接编译报错 2026-05-25 16:57:56 +08:00
wolfkill 6331cce23d fix: skip empty milvus enabled status groups 2026-05-25 16:54:56 +08:00
cn-kali-team fdb53ff92a fix type 2026-05-25 16:50:17 +08:00
young1lin 29820e4cac docs(chat): clarify cached-token semantics for explicit-cache providers
`cached_tokens` is reported by every OpenAI-compatible provider that
supports prompt caching, but how it becomes non-zero differs by mode:

- Implicit caching (OpenAI, Azure OpenAI, DeepSeek, …) populates the
  field automatically whenever a prompt prefix matches a previous
  request within the provider's cache TTL. No client-side opt-in.

- Explicit caching (Qwen on Aliyun, Anthropic Claude, …) only
  populates the field after the caller attaches `cache_control:
  {"type": "ephemeral"}` to the relevant message / content block.
  Until that opt-in is applied upstream of the request, the field
  stays zero even when the prefix is otherwise byte-stable.

Without this distinction documented, the previous commit reads as if
`TokenUsage.CachedTokens` will show non-zero values for Qwen / Claude
once this PR lands — which is not the case. The plumbing here is a
prerequisite (stable prefix via sorted tools) and a meter (visibility
of the field), but the explicit-cache opt-in itself is out of scope
and lives elsewhere.

Document this on `TokenUsage.CachedTokens` and the `cachedTokens`
helper so callers do not mistake observability for activation.
2026-05-25 16:47:14 +08:00
young1lin 582f7b3056 feat(chat): surface cached prompt tokens from upstream usage
OpenAI-compatible providers (Qwen, DeepSeek, OpenAI, Azure, etc.) report
prompt-cache hits in `usage.prompt_tokens_details.cached_tokens`. This
value was being read by go-openai but dropped at the WeKnora boundary,
so there was no way to tell whether prompt caching was actually working.

This change plumbs the field end-to-end:

- `types.TokenUsage.CachedTokens` (json:"cached_tokens,omitempty") —
  zero-values are omitted so payloads stay quiet for providers that
  never report cache hits.
- `cachedTokens` helper in remote_api.go guards against nil
  `PromptTokensDetails` (Ollama and older OpenAI-compat backends omit
  the details block entirely).
- All three response-parsing paths populate it:
    * `parseCompletionResponse` (non-streaming)
    * `processStream` (SDK streaming)
    * `processRawHTTPStream` (raw-HTTP streaming, used when callers
      need to inject custom fields like `cache_control`)
- The five `[LLM Usage]` log lines now print `cached_tokens=%d` so
  cache hit rate is visible in `journalctl` / log tail without going
  through metrics.

Together with the deterministic tool ordering from the previous commit,
this makes Qwen explicit caching observable: a warmed prefix should
show `cached_tokens` ≈ system + tools token count (typically several
thousand) on subsequent requests within the 5-minute TTL.

Tests:
- `TestCachedTokensHelper` — nil safety + round-trip
- `TestParseCompletionResponse_CachedTokens` — populated + missing
  details paths through `parseCompletionResponse`
- `TestTokenUsage_CachedTokensJSONOmitempty` — zero is omitted,
  non-zero is emitted
2026-05-25 16:47:14 +08:00
young1lin 29bec4204a fix(agent/tools): sort function definitions for deterministic ordering
`ToolRegistry.GetFunctionDefinitions` and `ListTools` previously ranged
over the internal map directly. Go map iteration is intentionally
randomized, so the resulting `tools` array reshuffled on every request.

That reshuffling silently breaks provider-side prompt caches that key on
a byte-level prefix match — most visibly Qwen explicit caching, which
requires the messages (system + tools + history) to be byte-identical up
to the `cache_control` marker. With random ordering the serialized tools
block changes every call, so the cache prefix never matches and the
hit rate stays at 0%.

Sort by tool name in both functions. Output is now byte-stable across
calls and `cache_control: ephemeral` can actually take effect.

Tests in registry_test.go cover:
- Deterministic ordering across 50 iterations
- JSON byte-stability across 20 iterations (the real motivation)
- Field projection (Name / Description / Parameters)
- Empty registry returns `[]` not `null`
- ListTools sorting
- First-wins duplicate registration policy (GHSA-67q9-58vj-32qx)
2026-05-25 16:47:14 +08:00
wizardchen 30332cca99 fix(frontend): correct floating UI positioning under root zoom
The user font-size preference applies CSS `zoom` on `<html>`
(see `composables/useFont.ts`). This breaks every popover that
anchors itself with `getBoundingClientRect()` + `position: fixed/absolute`,
because the rect is in visual pixels while CSS lengths are then
multiplied by zoom again — shifting popovers toward the lower-right
and miscomputing maxHeight / width / viewport-edge clamping.

PR #1459 patched only `AgentSelector.vue` (and left its width/maxHeight
unfixed). This PR introduces a shared helper `frontend/src/utils/zoom.ts`
(`getRootZoom`, `rectToCssPx`, `cssViewportSize`) and applies it to every
affected popover:

- `AgentSelector.vue` — refactored to use the helper; also normalizes
  `width` and `maxHeight` (previously still in visual px).
- `Input-field.vue` — model dropdown, mention popup (typed @ and button-
  triggered), and agent-mode dropdown.
- `KnowledgeBaseSelector.vue` — primary anchor path and fallback path.
- `UserMenu.vue` — tenant and IM floating submenus + viewport clamp.
- `FAQTagTooltip.vue` — fixed tooltip positioning + edge clamp.
- `AgentEditorModal.vue` — placeholder popups for system_prompt,
  context_template, and rewrite/fallback prompts.
- `AgentStreamDisplay.vue` — `kb-float-popup` (absolute under body, also
  under the zoom containing block).

Verified with `npm --prefix frontend run build`.
2026-05-24 23:03:26 +08:00
shendong a02d6d06b9 fix: correct agent selector position under zoom 2026-05-24 22:44:18 +08:00
wizardchen a469ee6a36 feat(wiki): add "View in Graph" entry on wiki page
Add a link in the wiki page header that navigates to the graph tab
with the page slug pre-selected, so users can jump from a wiki page
to its corresponding graph node in one click.
2026-05-22 21:02:01 +08:00
ochan.kwon 744a367f16 feat(retriever): add OpenSearch type prep ahead of Phase 3 driver
Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.

Changes:

- OpenSearchRetrieverEngineType constant ("opensearch") in
  internal/types/retriever.go. Not added to validEngineTypes /
  GetVectorStoreTypes / retrieverEngineMapping yet (gated).

- ConnectionConfig.InsecureSkipVerify (bool, default false) in
  internal/types/vectorstore.go, placed inside the // Common section
  because it is a cross-driver TLS option. Distinct from the
  Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
  only controls verification of an already-TLS HTTPS connection.
  AES-GCM Value/Scan round-trips the field as plaintext (it travels
  alongside the encrypted Password / APIKey but is not sensitive itself).

- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
  Min/Max (*float64), Enum ([]string). All omitempty so existing UI
  schema entries serialize identically. The fields will drive the UI
  in Phase 3 PR 3 (read-only on Edit + range/enum constraints).

- Six new AuditAction constants under vector_store.* and opensearch.*
  namespaces. Definitions only — emission lands in Phase 3 PR 3.

- EngineAwareNormalizer.Normalize is restructured to group engines by
  the effective score range observed by the normalizer (not the
  theoretical raw cosine range):

    Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
      - Milvus (COSINE metric mode). Milvus docs explicitly state the
        COSINE metric range is [-1, 1].

    Range [0, 1] (passthrough — already on the target scale):
      - Elasticsearch v8 / ElasticFaiss. The driver issues a
        cosineSimilarity(...) script_score script, and Lucene rejects
        negative final scores ("Final relevance scores from the
        script_score query cannot be negative" — ES docs); the
        effective range observed by the normalizer is therefore [0, 1]
        for IR-normalized embeddings. ES was previously grouped with
        Milvus and over-corrected via (score + 1) / 2, which inflated
        every ES result by 50% in mixed-engine RRF fusion.
      - OpenSearch. The k-NN plugin's
        SpaceType.COSINESIMIL.scoreTranslation maps the underlying
        Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
        before the score reaches us. Source:
        github.com/opensearch-project/k-NN at
        src/main/java/org/opensearch/knn/index/SpaceType.java
        (COSINESIMIL enum, scoreTranslation method).
      - Weaviate. The driver requests `certainty`, defined by Weaviate
        as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
      - Postgres pgvector / SQLite sqlite-vec / Qdrant /
        TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
        or normalized inner_product, whose theoretical range is [-1, 1].
        The IR-normalized positive-component unit vectors WeKnora
        targets (BGE / OpenAI text-embedding-3 / Cohere /
        sentence-transformers) keep the observed range in [0, 1];
        negative-cosine embedding models would silently clamp to 0
        downstream — explicitly documented as the IR-normalization
        caveat in the struct godoc.

  Dead enum references (ElasticFaiss, Infinity) are flagged in the
  godoc with a pointer to internal/types/vectorstore.go's existing
  "legacy/experimental, no standalone deployable instance" annotation.
  Their case labels are kept for switch exhaustiveness.

Test coverage:

- OpenSearch constant wire value + collision check against existing
  10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
  field deserializes as false) + round-trip + AES-GCM coexistence
  with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
  + *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
  for vector_store.* and opensearch.*, no wire-string collisions, exact
  wire-value pins for the six new constants.
- EngineAwareNormalizer:
  - CosineRange retains its (score + 1) / 2 coverage but now only for
    Milvus.
  - UnitInterval now covers the full passthrough group (ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
    Infinity / TencentVectorDB / Doris).
  - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
    explicit regression guard for the score-range correction: cos=0.5
    maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
  - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
    (1.0001) + defensive negative + ±Inf / NaN edges + keyword
    passthrough + nil-ctx safety.

Backward compatibility is preserved at every layer:

- All new struct fields are omitempty / pointer-tagged so existing
  rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
  lands. The ES regrouping changes the post-normalization value for
  every ES vector search result (a correctness fix, not a feature) —
  ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).
2026-05-22 20:43:50 +08:00
yuheng.huang 3d5b4c16fe refactor(logger): support LOG_FORMAT template and harden level coloring 2026-05-22 20:31:54 +08:00
yuheng.huang 39c9985c3b refactor(logger): support LOG_FORMAT template and harden level coloring
- Add CustomFormatter.Template driven by LOG_FORMAT env var with
  placeholders %d/%level/%thread/%logger/%traceId/%msg; default
  format unchanged for backward compatibility.
- Replace chained ReplaceAll with strings.NewReplacer for single-pass
  substitution, avoiding accidental re-substitution when a field value
  contains a literal placeholder string.
- Inject ANSI color at the %level substitution step; removes the old
  whole-line ReplaceAll(line, "INFO", ...) which would mis-color
  literal level tokens appearing inside messages.
- Cache threadNeeded on the formatter so runtime.Stack only runs when
  the template references %thread.
2026-05-22 20:31:54 +08:00
wizardchen bccc27b162 fix(frontend): keep X-Tenant-ID override when switching back to home
After re-login, the JWT is minted with the user's last-active tenant
(see userService.resolveLoginTenantID). Both TenantSelector and
UserMenu used to clear weknora_selected_tenant_id when "switching to
home", which made request.ts stop attaching X-Tenant-ID. Without the
header, the auth middleware fell back to the JWT-encoded tenant id —
i.e. the peer tenant the user just tried to leave — so the switch was
silently a no-op until the user manually wiped localStorage and
re-logged in.

Always write the active tenant id into selectedTenantId so the
"always attach X-Tenant-ID" invariant request.ts relies on stays
true. The server-side last_active_tenant_id preference is still
cleared when switching to home, so a clean re-login still lands the
user on home.

Also fix TenantSelector.defaultTenantId to read user.tenant_id (the
immutable home id) instead of authStore.tenant.id (the active tenant,
which is overwritten by /auth/me). This matches the contract spelled
out in useHomeTenant() and prevents switchingToHome from misfiring
when the active tenant differs from home.
2026-05-22 18:20:23 +08:00
ochan.kwon 8b7d00b260 feat: expose KB ↔ vector store binding in list, editor, and detail UI
Backend
-------
ListKnowledgeBases now enriches each row with the resolved vector_store
metadata (name / source / engine_type / status) via a new
buildKBListResponse helper. Store views are batch-resolved once per
request through BatchResolveStoreView so an N-KB list costs one
vector-store service call rather than N — closing the N+1 limitation
flagged in #1372's known-limitations section. Cross-tenant shared KBs
continue to render via SharedStoreDisplay so the owning tenant's store
inventory cannot be correlated across rows; the underlying vector_store_id
UUID is stripped from those responses.

Resolver failures degrade gracefully: bound KBs render as unavailable
instead of breaking the list. Test coverage pins the env / bound /
shared distinction, the batch-call-count invariant, and the
graceful-failure path.

Frontend
--------
KB editor modal gains a new "Vector Store" section. Create mode shows a
dropdown that combines the system default (env store) and the tenant's
configured user stores, fetched once at mount via the existing
listVectorStores API. Edit mode shows the bound store read-only via a
new VectorStoreBadge component with an explicit immutability hint —
matching the backend's `<-:create` GORM tag and the service-layer
UpdateKnowledgeBaseRequest DTO that already omit the field.

KB list cards surface a small engine-type badge for own-tenant bound
KBs, and a warning badge when the bound store is unavailable. Env-bound
and shared KBs render no badge (visual noise control). KB detail header
shows the bound store via the same VectorStoreBadge component; shared
KBs fall through to the badge's internal "shared" branch with no name /
engine / id rendered.

The KB editor's create-time error handler translates the typed
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable
(2201) into localized messages and jumps the user back to the
VectorStore section so they can pick a different store or fall back to
the system default.

The KB row type gains five optional fields (vector_store_id / name /
engine_type / source / status). i18n: 18 new keys added to en-US,
ko-KR, zh-CN; ru-RU receives English placeholders pending translation
(consistent with prior PRs in this locale).

Part of #993 (Phase 2: Per-KB VectorStore Binding).
Phase 2 roadmap item: PR 5 (KB binding UI + list-response enrichment).
Depends on #994, #1310, #1372, #1386 (all backend in the Phase 2 series).
2026-05-22 17:40:10 +08:00
Phuong Tran 0f57bf3ff8 fix (chat history): #1431 user's multilines query formatting was lost 2026-05-22 17:26:05 +08:00
wizardchen c0e4a1d2f1 fix(summary): preserve image caption/OCR text in document summaries
Documents whose only payload is an embedded image (e.g. a docx with a
single picture) intermittently produced the refusal line "No textual
content was extractable from this document." even though the vision
model had successfully extracted a caption.

Three coordinated fixes:

- Clarify the summary prompt that text inside `<image_caption>` and
  `<image_ocr>` is first-class extracted content, not an image
  reference, so the model only triggers the empty-content branch when
  the body is genuinely textless.
- For image-dominated documents (real text < 200 runes after stripping
  image markup) include OCR alongside captions so screenshots and
  scanned figures contribute their actual content; text-heavy
  documents continue to use caption-only enrichment to avoid OCR
  noise from incidental figures.
- Add `EnrichContentCaptionAndOCR` which embeds caption + OCR text
  inline next to the original Markdown image link, deliberately
  omitting the `<image url=...>` and `<image_original>` wrapper
  blocks. Those wrappers carry only opaque export hashes that consume
  tokens and have been observed to retrigger the LLM's "image
  reference with no extracted text" heuristic.
2026-05-22 17:25:39 +08:00
wizardchen 72e52f7258 fix(frontend): improve tenant settings and member list UX
Keep the member search input visible during empty/loading states, avoid layout jitter on search, simplify role select display, widen the role column, and use icon-only edit buttons for tenant name/description.
2026-05-21 23:28:48 +08:00
ChenRussell d3832edea4 fix(repository): qualify tenant_id with table name to resolve ambiguous column reference 2026-05-21 21:22:41 +08:00
ChenRussell 09f0f5b8a9 fix(swagger): Fix some Swagger API endpoints returning 404 errors and regenerate the Swagger documentation 2026-05-21 21:13:40 +08:00
MarionMa 0a9a920460 perf(repository): exclude embedding field from Elasticsearch search response to reduce size 2026-05-21 21:11:19 +08:00
wizardchen b0094ff479 docs(readme): flatten Latest Updates into single-line per-version list
Condense v0.6.0 highlights into one line and remove the collapsible
<details> wrapper so the full release history is visible by default
across all four READMEs (EN/CN/JA/KO).
2026-05-21 17:17:39 +08:00
wizardchen cdfc9ce23a chore(release): v0.6.0
Tenant RBAC headline release: 4-tier role matrix (Owner/Admin/
Contributor/Viewer), per-KB resource ownership, per-tenant audit
log, tenant member management, self-service workspaces.

Also: CLI v0.3/v0.4 GA, KB retrieval fan-out across vector stores,
AES-256-GCM credential at-rest, docreader gRPC TLS+Token, Zhipu
embedding, Huawei OBS, vLLM URL for MinerU, Apache Doris compat
modes, server-side user preferences, Go 1.26.0.

See CHANGELOG.md for the full list.

docs(rbac): wire RBAC screenshots into READMEs and RBAC guide

- README.md / README_CN.md / README_JA.md / README_KO.md: replace the
  single member-management thumbnail under the v0.6.0 RBAC highlight
  with a 2×2 showcase (member management, workspace switcher,
  self-service workspace creation, pending invitations).
- docs/RBAC说明.md: add the member-management screenshot to the
  existing 前端实际界面 showcase so the guide is self-contained
  and no longer cross-references README for it.

feat(rbac-ui): link tenant member page to RBAC guide

Add an inline doc-link in the Tenant Members settings page that
opens docs/RBAC说明.md on GitHub in a new tab, complementing the
existing in-app role-matrix popover. New i18n key
tenantMember.learnRbacGuide covered for zh-CN / en-US / ko-KR /
ru-RU.
2026-05-21 16:56:19 +08:00
Thinh Tran e1ce4270d0 fix(repository): correct query syntax to get correctly built-in models 2026-05-21 16:11:22 +08:00
Thinh Tran 3db5b0d126 fix(chat): adjust user message container to support pre-wrapped text 2026-05-21 16:11:22 +08:00
孙常熙 0dd6126826 fix(frontend): guard knowledge list against stale updates 2026-05-21 16:10:32 +08:00
wizardchen 7ebb29cd3e docs: add Chinese RBAC guide and link with shared space docs
Replace the English docs/rbac.md with a comprehensive Chinese
docs/RBAC说明.md and a wiki-style summary under docs/wiki/安全认证/.
Explain how tenant RBAC relates to the shared space feature (they
are orthogonal: tenant RBAC is the vertical defense, shared space
is the horizontal collaboration channel) and cross-link the two
docs in both the flat and wiki trees. Update inbound references in
.env.example, docker-compose.yml, and the auth legacy env test to
point at the new file name.
2026-05-21 12:28:31 +08:00
wizardchen 7aca1017db fix(docparser): address review feedback on PR #1404
Three follow-up fixes on top of the MinerU markdown preservation work:

- Stop applying normalizeMinerUMarkdown inside ResolveAndStore. The
  helper is already called by MinerUReader.Read, and ResolveAndStore is
  shared by every parser (docreader, session attachments, ...). Running
  the heading/image unescape regexes globally would silently rewrite
  content (including inside fenced code blocks) for non-MinerU sources.

- Recognize MinerU image references whose path contains spaces, e.g.
  "images/第 1 页.jpg". The previous regex used in
  extractImageRefsFromContent disallowed whitespace in the URL group,
  so such images were never matched and never persisted. Use a
  whitespace-tolerant pattern aligned with ResolveAndStore's own
  imgPattern.

- Deduplicate uploads when the same MinerU image is referenced under
  multiple path forms (e.g. "images/foo.png" vs "./images/foo.png").
  saveReferencedImage now caches by ref.Filename in addition to the
  raw ref path, so the second variant reuses the previously stored
  ServingURL instead of writing the same bytes to object storage
  again.

Tests added:
- TestProcessImagesMatchesPathsWithSpaces
- TestResolveAndStoreDedupsSameImageRefVariants
2026-05-21 11:42:56 +08:00
MidnightSun 6210f44fb6 fix(docparser): preserve MinerU markdown and persist relative images
MinerU already returns markdown with embedded HTML blocks, but the current\nreader runs the whole document back through html-to-markdown. That\nsecond conversion escapes valid headings and image syntax, so chunk\nprofiling sees plain text instead of markdown structure and relative\nimage references stop matching the storage pipeline.\n\nKeep MinerU output in its original markdown form and only apply narrow\ncompatibility normalization for the specific over-escaped patterns we\nactually need to recover. The converter now matches image refs by the\npaths that are really present in markdown or embedded HTML instead of\nassuming a single images/<name> form.\n\nExtend ImageResolver so relative HTML <img src=...> references share the\nsame storage rewrite path as markdown images, deduplicate repeated saves,\nand keep the frontend sanitizer compatible with MinerU's details/summary\nblocks. Add focused docparser tests that cover escaped markdown repair,\nvariant image path matching, and relative HTML image persistence.
2026-05-21 11:39:57 +08:00
wizardchen 9f6148784a fix(frontend): improve offline and legacy browser support 2026-05-21 11:19:44 +08:00
wizardchen 8f4626158d fix(chat): improve history rendering stability 2026-05-21 11:19:44 +08:00