- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
The intent prompt override logic in query_understand applied
strings.TrimSpace to the value before assigning it as the system prompt
override, which silently stripped trailing newlines and intentional
formatting from agent-supplied prompts. Use TrimSpace only to detect
emptiness (so whitespace-only strings still fall back to the global
default) while passing the raw string through verbatim.
Extract the resolution into applyIntentPromptOverride and add unit tests
covering agent-wins, whitespace preservation, blank fallback, no-config,
and global-only paths.
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.
Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.
What lands in PR 2a
-------------------
Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):
- `repository.go` — Repository struct + NewRepository constructor
that validates cluster reachability + OS version (2.4+ / 3.x;
primary tested 3.3.2) + k-NN plugin presence on every cluster
node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
dimension index creation, with transient errors not cached so a
momentary cluster blip does not permanently poison a dim.
sanitizeIndexName enforces a strict OS-compatible name spec.
probeVersion uses robust strings.Split/Atoi parsing for
pre-release suffixes and missing-patch versions. EngineType
returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
(MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
only cipher list) and transport tuning for the driver. Caller
exists only in PR 3 (container.go + engine_factory.go); PR 2a
remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
knn_vector + HNSW + content-analyzer mapping with every *_id
field as an explicit keyword and source_type as integer.
buildKeywordsMapping ships the dim-less keyword-only index
mapping used by the no-embedding save path. createIndexAndAlias
creates <alias>_v1 and aliases <alias> to it, with best-effort
orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
defaults (hnsw_m=16, ef_construction=100, ef_search=100,
shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
ErrDimensionMismatch, ErrAuth, ErrTransport,
ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
apperrors; PR 3's engine factory wraps these to typed AppError
2200/2201.
- `stubs.go` — every behavioural method returns
ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
fails-closed for non-empty KBs.
Tests (~1115 LoC, 50 cases):
- `repository_test.go` — interface satisfaction, sentinel mapping,
sanitizeIndexName positive/negative matrix, semver parsing
(pre-release / missing-patch), buildIndexMapping JSON shape pin
(Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
multi-node coverage, ensureReady concurrency + per-dim isolation
+ transient retry, NewRepository storeID validation, all 11
stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
wrapTransport sentinel mapping + leak guard, isNotFound /
isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
TLS 1.2 pinning / cipher list / transport tuning.
Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.
SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------
PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):
- AliasExists method passes dataPointer=nil to its internal do(),
which means non-2xx responses come back as a plain
*errors.errorString ("status: 404 Not Found") rather than as
*opensearch.StructError. aliasExists therefore inspects
resp.StatusCode directly (resp is returned even when err is
non-nil) and only falls back to wrapTransport for the "no
response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
index uses a mutex + ready/err flag pattern so transient failures
can be retried by the next caller. The per-dimension path uses
the `once map[int]*sync.Once` delete-and-recreate trick.
Test fixes folded in
--------------------
While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:
(1) Follow-up to #1445 — fanout test missed the new normalizer policy
(internal/application/service/knowledgebase_search_fanout_test.go,
+46 / -6). #1445 changed EngineAwareNormalizer for ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
passthrough (those engines surface non-negative cosine to the
normalizer per Lucene script_score non-negative invariant for
ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
OpenSearch, engine-internal or IR-normalized conversions for
the rest). Milvus is now the only engine that still surfaces
raw signed cosine in [-1, 1].
TestRetrieveFromStores_MixedEngine_Normalizes still asserted
the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
which under passthrough now becomes clamp01(-0.4) = 0. The
normalizer's own _test.go was updated at #1445 time, but this
fan-out integration test was not.
Fix: rewrite the godoc to spell out the two engine groups;
restate sub-case 2 as ES passthrough on a production-possible
mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
pinning the cosine-shift branch via Milvus -0.4 → 0.3.
(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
(internal/utils/security.go + internal/utils/security_test.go +
internal/infrastructure/web_search/searxng_test.go,
+33 / -9). loadSSRFWhitelist in internal/utils/security.go is
cached via sync.Once on first call. The internal reset helper
resetSSRFWhitelistForTest was unexported, so tests in other
packages could not reset and saw whatever whitelist was cached
by the first sync.Once.Do() in the same test binary. In
internal/infrastructure/web_search/, TestValidateProxyURL runs
before TestValidateSearxngBaseURL alphabetically and exercises
ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
whitelist; the later setenv in searxng_test then has no effect
and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
Pre-existing on main; surfaced now because this PR was the
first to do a full `go test ./...` run on top of #1445.
Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
ForTest suffix is the test-only contract); update in-package
callers; in web_search/searxng_test.go import internal/utils
and call ResetSSRFWhitelistForTest around the env mutation in
both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
No production code path changes.
Roadmap
-------
- PR 2b (next, depends on this PR) — read/write implementations:
query.go + retrieve.go + crud.go land their real bodies; stubs
for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
removed; corresponding CRUD/retrieve/filter test cases (~430
LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
BatchUpdate*, large-batch async deletes) + i18n + docker-compose
dev profile. After PR 3 merges, the OpenSearch driver becomes
reachable via either `engine_type=opensearch` VectorStore or
`RETRIEVE_DRIVER=opensearch` env.
Backward compatibility
----------------------
- New package — additive only. No existing file modified except
go.mod / go.sum, the two test files in (1)/(2), and the
test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
here (no driver instance produces a result yet).
Test plan
---------
- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
shows only PR 1's normalizer case + this driver's EngineType()
and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
- Add Type field to v2User struct to distinguish personal/team tokens
- Route team tokens directly to ListGroupRepos (skip ListUserGroups)
- Gracefully handle ListUserGroups 404 for personal tokens without teams
- Add rate-limiting delay between GetDocDetail calls to avoid API throttling
When the rerank model returns an error (e.g. 401 Unauthorized), the
pipeline previously discarded all retrieved candidates and returned
empty results to the caller.
Now p.rerank returns ([]RankResult, error) to distinguish API failure
from threshold-filtered empty results. On API error, the pipeline
falls back to the original retrieval candidates (directLoad +
candidatesToRerank) and continues to CHUNK_MERGE/FILTER_TOP_K,
so users still get results even when the rerank model is misconfigured.
Problem:
Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing
VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime
with SQL parse errors.
Root cause:
1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in
MySQL/PostgreSQL but not in Doris's SQL dialect)
2. Doris MySQL driver was not configured to interpolate parameters, preventing
automatic client-side placeholder replacement
Fix:
1. Enable parameter interpolation in Doris connection DSN:
- Add &interpolateParams=true to mysql.Open() DSN in container.go
- This enables driver-level parameter substitution at query time
2. Inline LIMIT/OFFSET as string literals in Doris queries:
- VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK)
- KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK)
- CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset)
- Remove inlined values from args slice
Result:
Queries are now built with literal LIMIT/OFFSET values instead of placeholders,
compatible with Doris's SQL parser.
When selecting an agent from the chat page's agent selector dropdown,
built-in agents other than `builtin-quick-answer` and
`builtin-smart-reasoning` trigger the wrong toast. "Switched to
Intelligent Reasoning" pops up regardless of which agent was actually
picked — Wiki Questioner, Data Analyst, Wiki Fixer, etc.
Root cause: the toast logic checks only `agent.is_builtin`, then
branches on `agent_mode === 'smart-reasoning'` to pick between the
"Switched to Intelligent Reasoning" / "Switched to Quick Q&A" strings.
Those strings were designed for the two original built-ins
(`builtin-quick-answer` and `builtin-smart-reasoning`), which the
dropdown re-brands as "Normal Mode" / "Agent Mode". They don't make
sense for any other built-in.
Narrow the special-case branch to those two IDs. Other built-ins fall
back to the generic `agentSelected` toast (the path custom agents
already take), so they get a toast naming the agent that was actually
selected.
Start new agents with an empty context_template so mode-switch logic can
populate the system default instead of keeping the smart-reasoning placeholder.
Co-authored-by: Cursor <cursoragent@cursor.com>
Three correctness fixes that the lifecycle PR deliberately deferred:
1. ID length / struct-tag drift
- models.id is varchar(64) on both PG and SQLite (per the init
migrations), but Model.ID's GORM tag said varchar(36) — a remnant
from when the field only held UUIDs. The mismatch is harmless under
golang-migrate (struct tag is ignored), but misleading on AutoMigrate
paths and in IDE tooltips. Tag now matches the real column width.
- New ModelIDMaxLen constant (=64) is the single source of truth for
anyone accepting user-provided ids. The YAML loader uses it to
reject too-long ids up front with a clear message instead of letting
the INSERT explode with a generic "value too long for type" error.
2. Field validation in the YAML loader
- Type, Source, and Status are typed strings but YAML can supply any
value. Misspellings (e.g. `type: knowledgeqa` lowercase, `type: LLM`)
were previously persisted as-is and produced rows that looked fine
in the table but failed at provider-factory lookup time, which is
hard to debug.
- validateBuiltinModelEntry now checks: empty id, id length, empty
type, type ∈ {KnowledgeQA, Embedding, Rerank, VLLM, ASR}, and
status ∈ {active, downloading, download_failed, empty}. Source is
intentionally NOT validated because the provider matrix in
internal/models/* keeps growing and a strict allow-list here would
force changes in two places per new provider.
- Invalid entries are warned + skipped (not aborting the whole load),
and excluded from the keep-set so the drift sweep does not delete
existing matching rows on the strength of a typo'd YAML retry.
3. Magic number cleanup
- DefaultBuiltinModelTenantID (=10000) replaces the hard-coded `10000`
literal in toModel(). The invariant lives in three places already
(PG migration, SQLite migration, this constant); naming it makes
the cross-reference explicit and grep-able.
Tests:
- New TestLoadBuiltinModelsConfig_RejectsInvalidEntries with five
sub-cases (id-too-long, missing-type, lowercase-type, unknown-type,
unknown-status) asserts the table stays empty after each.
- All 11 existing tests still pass.
Original PR #1453 used fmt.Printf, which lands as unstructured noise in
release/JSON log pipelines. The natural fix is logger.Infof/Warnf, but
internal/logger itself imports internal/types — using it from here
would create an import cycle (this is the same constraint that forces
model.go's crypto error logs onto stdlib log).
Switch all loader output to log.Printf with a stable "[builtin-models]"
prefix (mirroring the "[crypto]" convention already established in
model.go). The prefix gives operators a grep handle even though the
lines stay unstructured.
Levels are encoded inline as "WARN:" for warnings so log shippers and
humans can still discriminate.
PR #1453 introduced the map form of env_file:
env_file:
- path: .env
required: false
which is only recognised by Docker Compose v2.24+ (Jan 2024). Older
deploys would refuse to parse docker-compose.yml even when builtin_models
YAML is not enabled — breaking the "no impact unless opted in" promise.
Switch to the array form (`env_file: [.env]`) which has been supported
since the earliest Compose schemas. The trade-off is that Compose now
errors when .env is absent, so:
- scripts/start_all.sh already calls check_env_file (cp .env.example .env
if missing) before docker compose up; that path is unaffected.
- The bare `make docker-run` / `make docker-restart` targets in the
Makefile are taught the same fallback: touch / copy .env.example before
invoking docker-compose, so fresh clones keep working.
The docker-compose.yml comment block is updated to explain the version
trade-off so future maintainers don't re-introduce the map form.
Extend the builtin_models.yaml loader so the YAML file becomes a complete
source of truth for the rows it owns. Builds on the previous commit's
managed_by column.
Lifecycle contract:
- Every UPSERTed entry is tagged managed_by="yaml".
- The DoUpdates list now includes deleted_at, so an entry that was
soft-deleted (e.g. via UI/API) is automatically resurrected when it
reappears in the file. Closes the "ghost row that exists but is
invisible" failure mode.
- After all UPSERTs, the loader soft-deletes rows where
managed_by='yaml' AND id NOT IN (current YAML id set). Removing an
entry from YAML is now the supported way to retire a built-in model —
no manual SQL needed.
- Rows tagged managed_by='' (UI/API/SQL-seeded built-ins) are invisible
to the reconcile path and never touched.
- When a YAML entry sets is_default=true, the loader first clears
is_default on any other rows in the same (tenant_id, type) bucket,
mirroring the invariant enforced by the API path
(repository.UnsetDefaultModel).
Failure handling stays defensive:
- File missing / not a regular file / parse error: warn and skip; the
drift sweep is NOT executed so a malformed file cannot wipe rows.
- Per-entry UPSERT error: warn, drop the id from the keep-set so the
sweep also leaves the existing row alone ("leave alone on failure").
Tests cover: file-missing, parse-error, basic upsert + defaults,
idempotency, ${ENV} interpolation (set vs unset), drift sweep removing
YAML rows, drift sweep ignoring manual rows, soft-delete resurrection,
is_default cleanup across tenant+type, explicit empty list sweeping all
yaml-managed rows, and a regression guard ensuring BeforeCreate does not
overwrite YAML-supplied stable ids.
Docs are rewritten so operators see "delete from YAML and restart" as
the supported removal path; SQL is retained only for the legacy
managed_by='' slice.
Introduce a `managed_by` varchar column on `models` so future declarative
loaders can claim ownership of a subset of rows without disturbing entries
created via the UI/API or seeded by hand-written SQL.
- versioned/000052_models_managed_by.{up,down}.sql add the column with a
default of '' and a partial index on non-empty values to keep startup
reconciliation cheap.
- sqlite/000000_init.up.sql is updated in place (the Lite init migration
is a single file per project convention) so fresh SQLite databases get
the column from the start.
- Model.ManagedBy mirrors the column. Existing rows default to '' which
the YAML loader treats as "manually managed, never touch".
Schema half of the YAML-driven built-in-model lifecycle work that follows
up on #1453; the reconciler that uses the column lands in the next commit.
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.
Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.
The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
When a custom agent has `MultiTurnEnabled=false`, `applyAgentOverridesToChatManage`
sets `chatManage.MaxRounds = 0` to signal "no history". Two pipeline plugins
mistreated this zero value as "use the global default" and silently re-loaded
session history into the LLM context:
- `PluginLoadHistory` fell back to `Conversation.MaxRounds` when
`chatManage.MaxRounds == 0`.
- `PluginQueryUnderstand.loadHistory` had the same fallback, and even when
`LOAD_HISTORY` was skipped it would re-populate `chatManage.History`,
leaking previous turns into rewrite, image analysis, and the final answer.
The RAG branch in `session_knowledge_qa` also added `LOAD_HISTORY`
unconditionally, unlike the pure-chat branch which guarded it with `hasHistory`.
This change:
- Treats `chatManage.MaxRounds <= 0` as an explicit disable in both plugins;
no fallback to global config.
- Makes the RAG pipeline consistent with the pure-chat path by gating
`LOAD_HISTORY` on `hasHistory`.
- Removes the duplicated `Current Time: {{current_time}}` line from
`agent_system_prompt.yaml`. The agent already receives a fresh
`<runtime_context><current_time>` block with each turn from
`observe.buildRuntimeContextBlock`, so the static placeholder was
redundant.
The ReAct agent path (`session_agent_qa`) already checked `MultiTurnEnabled`
directly and is not affected.
Closes#1479
Added a new KBSwitcherDropdown component to enhance the user experience when selecting knowledge bases. This component displays a list of knowledge bases with the current selection highlighted, allowing users to easily switch between them. Integrated the dropdown into both KnowledgeBase.vue and FAQEntryManager.vue, replacing the previous dropdown implementation for a more streamlined and consistent UI. The new component supports dynamic updates and maintains the order of knowledge bases while ensuring the current selection is always visible at the top.
Updated the styles for the info button in KBInfoPopover.vue to enhance its appearance and interaction. Adjusted dimensions, colors, and hover effects to create a clearer distinction between the info button and the settings button. Added a warning state for better visual feedback when necessary. These changes aim to improve user experience and maintain consistency in the UI design.
Refactor the logic in KBInfoPopover.vue to differentiate between FAQ and document knowledge bases. The component now uses `chunk_count` for FAQ KBs and retains `knowledge_count` for document KBs, ensuring accurate representation of entry totals. This change aligns the display logic with the intended user-facing metrics for each KB type.
Move the info popover (intro'd in the previous commit) into a
dedicated `frontend/src/components/KBInfoPopover.vue` component so
both the document KB header and the FAQ KB header can share the
exact same metadata surface.
- KnowledgeBase.vue: replace the inline popover and the now-dead
helper computeds (kbLastUpdated, kbHasDistinctUpdate,
infoCardCapabilities, infoCardChunking, infoCardStats,
chunkingStrategyLabel, supportedFileTypesList, accessRoleLabel,
accessPermissionSummary) and their CSS with a single
<KBInfoPopover> tag. The page still fetches parser engines and
forwards the supported file types so the "Accepted formats" row
renders for document KBs.
- FAQEntryManager.vue: drop the legacy inline meta strip
(role tag · share source · last updated) along with the
accessRoleLabel / accessPermissionSummary / kbLastUpdated /
effectiveKBPermission computeds it only served, and wire the
same <KBInfoPopover> into the actions cluster next to the
settings button. FAQ KBs don't pass supportedFileTypes so the
format row simply hides itself.
- KBInfoPopover.vue: self-contained — re-derives access role,
shared-binding and chunking config from kbInfo + the org/auth
stores. supported-file-types remains an opt-in prop so callers
decide whether the parser-engine pipeline is relevant.
No user-visible regressions: both pages now render the same
Basic / Access / Capabilities / Chunking / Stats / Storage
Binding sections, with rows hiding when they don't apply.
Replace the inline meta strip on the knowledge base detail header
(role tag · share source · last updated · vector store badge) and
the vector-store engine badge on the list cards with a single
structured "info" popover next to the settings button.
The popover groups Basic, Access, Capabilities, Chunking, Stats
and Storage Binding into separate sections, only rendering those
with data. A warning dot on the info button surfaces unreachable
vector-store bindings without occupying header real estate. The
popover caps at min(70vh, 560px) with an internal scrolling body
so it never overflows the viewport.
Hide the last-updated row when it equals created_at, since GORM
only bumps KB.updated_at on metadata edits — never on document
uploads — so surfacing two identical timestamps next to "created
at" was misleading. Once the KB metadata is actually edited the
row reappears naturally.
i18n entries added for zh-CN, en-US, ko-KR, ru-RU.
- Refactored checkStorageEngineConfigured to be a method of knowledgeService, enhancing encapsulation and readability.
- Updated logic to allow fallback to global file service when no storage provider is configured at the KB or tenant level, improving error handling.
- Added detailed comments to clarify the method's behavior and internal logic, ensuring better understanding for future maintenance.
- Removed the VideoInfo field from the Chunk struct in chunk.go, streamlining the data model.
- This change reflects a shift in focus away from video information storage within the Chunk type.
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
The per-card "more actions" popover (edit / rebuild / move / delete ...)
stayed open when clicking "Delete document", so the popover floated above
the t-dialog and visually obscured the confirmation dialog.
Reset the card's isMore flag and moreIndex inside delCard before toggling
delDialog so the popover closes synchronously with the dialog opening.
DeleteKnowledge is now an async pipeline on the backend, so refreshing the
list immediately after a successful response can still show the deleted
entry. Mirror the polling loop already used for batch delete: re-list up to
~12s (30 polls * 400ms) until the deleted id disappears, then refresh tags.
Without this the UI shows "delete success" but the list visibly contains
the entry until the user manually refreshes.
Reuse enqueueKnowledgeListDelete inside DeleteKnowledge so that single-item
delete shares the same hardening as BatchDeleteKnowledge / ClearKnowledgeBase:
asynq retries, business-aware queue routing, and marking-as-deleting inside
the worker.
The endpoint now returns 200 once the delete task has been enqueued; the
response body carries the asynq task_id and the message is updated to
"Delete task submitted". Swagger annotations, generated docs and the Go
client SDK comment are updated to reflect the new asynchronous semantics.
Note: this is a behavior change. Callers that previously assumed the
knowledge was already gone on a 200 response should poll the task status
or accept eventual consistency, matching the existing BatchDeleteKnowledge
contract.
`cached_tokens` is reported by every OpenAI-compatible provider that
supports prompt caching, but how it becomes non-zero differs by mode:
- Implicit caching (OpenAI, Azure OpenAI, DeepSeek, …) populates the
field automatically whenever a prompt prefix matches a previous
request within the provider's cache TTL. No client-side opt-in.
- Explicit caching (Qwen on Aliyun, Anthropic Claude, …) only
populates the field after the caller attaches `cache_control:
{"type": "ephemeral"}` to the relevant message / content block.
Until that opt-in is applied upstream of the request, the field
stays zero even when the prefix is otherwise byte-stable.
Without this distinction documented, the previous commit reads as if
`TokenUsage.CachedTokens` will show non-zero values for Qwen / Claude
once this PR lands — which is not the case. The plumbing here is a
prerequisite (stable prefix via sorted tools) and a meter (visibility
of the field), but the explicit-cache opt-in itself is out of scope
and lives elsewhere.
Document this on `TokenUsage.CachedTokens` and the `cachedTokens`
helper so callers do not mistake observability for activation.
OpenAI-compatible providers (Qwen, DeepSeek, OpenAI, Azure, etc.) report
prompt-cache hits in `usage.prompt_tokens_details.cached_tokens`. This
value was being read by go-openai but dropped at the WeKnora boundary,
so there was no way to tell whether prompt caching was actually working.
This change plumbs the field end-to-end:
- `types.TokenUsage.CachedTokens` (json:"cached_tokens,omitempty") —
zero-values are omitted so payloads stay quiet for providers that
never report cache hits.
- `cachedTokens` helper in remote_api.go guards against nil
`PromptTokensDetails` (Ollama and older OpenAI-compat backends omit
the details block entirely).
- All three response-parsing paths populate it:
* `parseCompletionResponse` (non-streaming)
* `processStream` (SDK streaming)
* `processRawHTTPStream` (raw-HTTP streaming, used when callers
need to inject custom fields like `cache_control`)
- The five `[LLM Usage]` log lines now print `cached_tokens=%d` so
cache hit rate is visible in `journalctl` / log tail without going
through metrics.
Together with the deterministic tool ordering from the previous commit,
this makes Qwen explicit caching observable: a warmed prefix should
show `cached_tokens` ≈ system + tools token count (typically several
thousand) on subsequent requests within the 5-minute TTL.
Tests:
- `TestCachedTokensHelper` — nil safety + round-trip
- `TestParseCompletionResponse_CachedTokens` — populated + missing
details paths through `parseCompletionResponse`
- `TestTokenUsage_CachedTokensJSONOmitempty` — zero is omitted,
non-zero is emitted
`ToolRegistry.GetFunctionDefinitions` and `ListTools` previously ranged
over the internal map directly. Go map iteration is intentionally
randomized, so the resulting `tools` array reshuffled on every request.
That reshuffling silently breaks provider-side prompt caches that key on
a byte-level prefix match — most visibly Qwen explicit caching, which
requires the messages (system + tools + history) to be byte-identical up
to the `cache_control` marker. With random ordering the serialized tools
block changes every call, so the cache prefix never matches and the
hit rate stays at 0%.
Sort by tool name in both functions. Output is now byte-stable across
calls and `cache_control: ephemeral` can actually take effect.
Tests in registry_test.go cover:
- Deterministic ordering across 50 iterations
- JSON byte-stability across 20 iterations (the real motivation)
- Field projection (Name / Description / Parameters)
- Empty registry returns `[]` not `null`
- ListTools sorting
- First-wins duplicate registration policy (GHSA-67q9-58vj-32qx)
The user font-size preference applies CSS `zoom` on `<html>`
(see `composables/useFont.ts`). This breaks every popover that
anchors itself with `getBoundingClientRect()` + `position: fixed/absolute`,
because the rect is in visual pixels while CSS lengths are then
multiplied by zoom again — shifting popovers toward the lower-right
and miscomputing maxHeight / width / viewport-edge clamping.
PR #1459 patched only `AgentSelector.vue` (and left its width/maxHeight
unfixed). This PR introduces a shared helper `frontend/src/utils/zoom.ts`
(`getRootZoom`, `rectToCssPx`, `cssViewportSize`) and applies it to every
affected popover:
- `AgentSelector.vue` — refactored to use the helper; also normalizes
`width` and `maxHeight` (previously still in visual px).
- `Input-field.vue` — model dropdown, mention popup (typed @ and button-
triggered), and agent-mode dropdown.
- `KnowledgeBaseSelector.vue` — primary anchor path and fallback path.
- `UserMenu.vue` — tenant and IM floating submenus + viewport clamp.
- `FAQTagTooltip.vue` — fixed tooltip positioning + edge clamp.
- `AgentEditorModal.vue` — placeholder popups for system_prompt,
context_template, and rewrite/fallback prompts.
- `AgentStreamDisplay.vue` — `kb-float-popup` (absolute under body, also
under the zoom containing block).
Verified with `npm --prefix frontend run build`.
Add a link in the wiki page header that navigates to the graph tab
with the page slug pre-selected, so users can jump from a wiki page
to its corresponding graph node in one click.
Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.
Changes:
- OpenSearchRetrieverEngineType constant ("opensearch") in
internal/types/retriever.go. Not added to validEngineTypes /
GetVectorStoreTypes / retrieverEngineMapping yet (gated).
- ConnectionConfig.InsecureSkipVerify (bool, default false) in
internal/types/vectorstore.go, placed inside the // Common section
because it is a cross-driver TLS option. Distinct from the
Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
only controls verification of an already-TLS HTTPS connection.
AES-GCM Value/Scan round-trips the field as plaintext (it travels
alongside the encrypted Password / APIKey but is not sensitive itself).
- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
Min/Max (*float64), Enum ([]string). All omitempty so existing UI
schema entries serialize identically. The fields will drive the UI
in Phase 3 PR 3 (read-only on Edit + range/enum constraints).
- Six new AuditAction constants under vector_store.* and opensearch.*
namespaces. Definitions only — emission lands in Phase 3 PR 3.
- EngineAwareNormalizer.Normalize is restructured to group engines by
the effective score range observed by the normalizer (not the
theoretical raw cosine range):
Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
- Milvus (COSINE metric mode). Milvus docs explicitly state the
COSINE metric range is [-1, 1].
Range [0, 1] (passthrough — already on the target scale):
- Elasticsearch v8 / ElasticFaiss. The driver issues a
cosineSimilarity(...) script_score script, and Lucene rejects
negative final scores ("Final relevance scores from the
script_score query cannot be negative" — ES docs); the
effective range observed by the normalizer is therefore [0, 1]
for IR-normalized embeddings. ES was previously grouped with
Milvus and over-corrected via (score + 1) / 2, which inflated
every ES result by 50% in mixed-engine RRF fusion.
- OpenSearch. The k-NN plugin's
SpaceType.COSINESIMIL.scoreTranslation maps the underlying
Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
before the score reaches us. Source:
github.com/opensearch-project/k-NN at
src/main/java/org/opensearch/knn/index/SpaceType.java
(COSINESIMIL enum, scoreTranslation method).
- Weaviate. The driver requests `certainty`, defined by Weaviate
as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
- Postgres pgvector / SQLite sqlite-vec / Qdrant /
TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
or normalized inner_product, whose theoretical range is [-1, 1].
The IR-normalized positive-component unit vectors WeKnora
targets (BGE / OpenAI text-embedding-3 / Cohere /
sentence-transformers) keep the observed range in [0, 1];
negative-cosine embedding models would silently clamp to 0
downstream — explicitly documented as the IR-normalization
caveat in the struct godoc.
Dead enum references (ElasticFaiss, Infinity) are flagged in the
godoc with a pointer to internal/types/vectorstore.go's existing
"legacy/experimental, no standalone deployable instance" annotation.
Their case labels are kept for switch exhaustiveness.
Test coverage:
- OpenSearch constant wire value + collision check against existing
10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
field deserializes as false) + round-trip + AES-GCM coexistence
with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
+ *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
for vector_store.* and opensearch.*, no wire-string collisions, exact
wire-value pins for the six new constants.
- EngineAwareNormalizer:
- CosineRange retains its (score + 1) / 2 coverage but now only for
Milvus.
- UnitInterval now covers the full passthrough group (ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
Infinity / TencentVectorDB / Doris).
- New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
explicit regression guard for the score-range correction: cos=0.5
maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
- OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
(1.0001) + defensive negative + ±Inf / NaN edges + keyword
passthrough + nil-ctx safety.
Backward compatibility is preserved at every layer:
- All new struct fields are omitempty / pointer-tagged so existing
rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
lands. The ES regrouping changes the post-normalization value for
every ES vector search result (a correctness fix, not a feature) —
ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).
- Add CustomFormatter.Template driven by LOG_FORMAT env var with
placeholders %d/%level/%thread/%logger/%traceId/%msg; default
format unchanged for backward compatibility.
- Replace chained ReplaceAll with strings.NewReplacer for single-pass
substitution, avoiding accidental re-substitution when a field value
contains a literal placeholder string.
- Inject ANSI color at the %level substitution step; removes the old
whole-line ReplaceAll(line, "INFO", ...) which would mis-color
literal level tokens appearing inside messages.
- Cache threadNeeded on the formatter so runtime.Stack only runs when
the template references %thread.
After re-login, the JWT is minted with the user's last-active tenant
(see userService.resolveLoginTenantID). Both TenantSelector and
UserMenu used to clear weknora_selected_tenant_id when "switching to
home", which made request.ts stop attaching X-Tenant-ID. Without the
header, the auth middleware fell back to the JWT-encoded tenant id —
i.e. the peer tenant the user just tried to leave — so the switch was
silently a no-op until the user manually wiped localStorage and
re-logged in.
Always write the active tenant id into selectedTenantId so the
"always attach X-Tenant-ID" invariant request.ts relies on stays
true. The server-side last_active_tenant_id preference is still
cleared when switching to home, so a clean re-login still lands the
user on home.
Also fix TenantSelector.defaultTenantId to read user.tenant_id (the
immutable home id) instead of authStore.tenant.id (the active tenant,
which is overwritten by /auth/me). This matches the contract spelled
out in useHomeTenant() and prevents switchingToHome from misfiring
when the active tenant differs from home.
Backend
-------
ListKnowledgeBases now enriches each row with the resolved vector_store
metadata (name / source / engine_type / status) via a new
buildKBListResponse helper. Store views are batch-resolved once per
request through BatchResolveStoreView so an N-KB list costs one
vector-store service call rather than N — closing the N+1 limitation
flagged in #1372's known-limitations section. Cross-tenant shared KBs
continue to render via SharedStoreDisplay so the owning tenant's store
inventory cannot be correlated across rows; the underlying vector_store_id
UUID is stripped from those responses.
Resolver failures degrade gracefully: bound KBs render as unavailable
instead of breaking the list. Test coverage pins the env / bound /
shared distinction, the batch-call-count invariant, and the
graceful-failure path.
Frontend
--------
KB editor modal gains a new "Vector Store" section. Create mode shows a
dropdown that combines the system default (env store) and the tenant's
configured user stores, fetched once at mount via the existing
listVectorStores API. Edit mode shows the bound store read-only via a
new VectorStoreBadge component with an explicit immutability hint —
matching the backend's `<-:create` GORM tag and the service-layer
UpdateKnowledgeBaseRequest DTO that already omit the field.
KB list cards surface a small engine-type badge for own-tenant bound
KBs, and a warning badge when the bound store is unavailable. Env-bound
and shared KBs render no badge (visual noise control). KB detail header
shows the bound store via the same VectorStoreBadge component; shared
KBs fall through to the badge's internal "shared" branch with no name /
engine / id rendered.
The KB editor's create-time error handler translates the typed
ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable
(2201) into localized messages and jumps the user back to the
VectorStore section so they can pick a different store or fall back to
the system default.
The KB row type gains five optional fields (vector_store_id / name /
engine_type / source / status). i18n: 18 new keys added to en-US,
ko-KR, zh-CN; ru-RU receives English placeholders pending translation
(consistent with prior PRs in this locale).
Part of #993 (Phase 2: Per-KB VectorStore Binding).
Phase 2 roadmap item: PR 5 (KB binding UI + list-response enrichment).
Depends on #994, #1310, #1372, #1386 (all backend in the Phase 2 series).
Documents whose only payload is an embedded image (e.g. a docx with a
single picture) intermittently produced the refusal line "No textual
content was extractable from this document." even though the vision
model had successfully extracted a caption.
Three coordinated fixes:
- Clarify the summary prompt that text inside `<image_caption>` and
`<image_ocr>` is first-class extracted content, not an image
reference, so the model only triggers the empty-content branch when
the body is genuinely textless.
- For image-dominated documents (real text < 200 runes after stripping
image markup) include OCR alongside captions so screenshots and
scanned figures contribute their actual content; text-heavy
documents continue to use caption-only enrichment to avoid OCR
noise from incidental figures.
- Add `EnrichContentCaptionAndOCR` which embeds caption + OCR text
inline next to the original Markdown image link, deliberately
omitting the `<image url=...>` and `<image_original>` wrapper
blocks. Those wrappers carry only opaque export hashes that consume
tokens and have been observed to retrigger the LLM's "image
reference with no extracted text" heuristic.
Keep the member search input visible during empty/loading states, avoid layout jitter on search, simplify role select display, widen the role column, and use icon-only edit buttons for tenant name/description.
Condense v0.6.0 highlights into one line and remove the collapsible
<details> wrapper so the full release history is visible by default
across all four READMEs (EN/CN/JA/KO).
Tenant RBAC headline release: 4-tier role matrix (Owner/Admin/
Contributor/Viewer), per-KB resource ownership, per-tenant audit
log, tenant member management, self-service workspaces.
Also: CLI v0.3/v0.4 GA, KB retrieval fan-out across vector stores,
AES-256-GCM credential at-rest, docreader gRPC TLS+Token, Zhipu
embedding, Huawei OBS, vLLM URL for MinerU, Apache Doris compat
modes, server-side user preferences, Go 1.26.0.
See CHANGELOG.md for the full list.
docs(rbac): wire RBAC screenshots into READMEs and RBAC guide
- README.md / README_CN.md / README_JA.md / README_KO.md: replace the
single member-management thumbnail under the v0.6.0 RBAC highlight
with a 2×2 showcase (member management, workspace switcher,
self-service workspace creation, pending invitations).
- docs/RBAC说明.md: add the member-management screenshot to the
existing 前端实际界面 showcase so the guide is self-contained
and no longer cross-references README for it.
feat(rbac-ui): link tenant member page to RBAC guide
Add an inline doc-link in the Tenant Members settings page that
opens docs/RBAC说明.md on GitHub in a new tab, complementing the
existing in-app role-matrix popover. New i18n key
tenantMember.learnRbacGuide covered for zh-CN / en-US / ko-KR /
ru-RU.