Two encryption gaps and one fail-loud regression, fixed together because
they are inseparable in practice.
At-rest encryption gaps
-----------------------
Before this change, only Tenant.APIKey, Tenant.Credentials.WeKnoraCloud.AppSecret,
Model.Parameters.APIKey/AppSecret, and WebSearchProviderParameters.APIKey
were encrypted with SYSTEM_AES_KEY. MCP and DataSource were missed:
- MCPAuthConfig.Value/Scan went through bare json.Marshal/Unmarshal, so
api_key and token were written to mcp_services.auth_config (json) in
plaintext regardless of whether SYSTEM_AES_KEY was set.
- DataSource.Config (types.JSON) is a byte passthrough, so connector
secrets (Notion token, GitHub PAT, Feishu app_secret, ...) inside the
Credentials map were stored in plaintext too.
Now both follow the ModelParameters / WebSearchProviderParameters pattern.
Encryption operates on a local copy to avoid mutating the caller's
in-memory struct (otherwise the next read of the same *MCPAuthConfig
would surface ciphertext as if it were the API key). For DataSource the
ToJSON ↔ ParseConfig pair is the only application boundary that
materializes credentials, so encrypting there is sufficient and avoids
having to fork the generic types.JSON.
Backward compatibility
----------------------
DecryptStoredSecret returns rows without the enc:v1: prefix verbatim, so
historical plaintext records keep working. Next write re-saves them
encrypted. No migration step.
Graceful Scan-time degradation
------------------------------
The strict "fail loud on key rotation" behaviour of DecryptStoredSecret
turned into a footgun once it ran from GORM Scan: a single broken row
caused ListModels / ListMCPServices / ListDataSources to return zero
results, hiding all other rows and giving the operator no way to find
which credential needed re-entering.
New utils.DecryptStoredSecretLenient: same logic, but on failure returns
("", false). The strict variant is kept for hot paths that are about to
actually USE the credential (so a rotated key surfaces loudly there
rather than silently degrading to "missing api key").
All Scan paths switch to lenient + per-failure log line:
ModelParameters.Scan, WebSearchProviderParameters.Scan,
MCPAuthConfig.Scan, DataSource.ParseConfig, CredentialsConfig.Scan.
On lenient failure the field blanks out (never ciphertext), the row
still loads, the DTO reports configured=false so the UI shows "not
configured", and any upstream call attempted with the empty value fails
with a clear "missing api key" rather than sending ciphertext as the key.
Tenant.AfterFind (Tenant.APIKey) intentionally stays strict — it is the
primary HTTP auth credential and a silent empty value would change auth
semantics ambiguously.
Direct regression from the previous credential subresource refactor: the
edit-mode ModelEditorDialog stopped binding apiKey into formData (the
field now lives in <CredentialResource>), so "Test connection" sent
apiKey="" and every upstream call (DeepSeek / lkeap / OpenAI etc.) 401'd.
Fix in the backend so the frontend stays simple:
- ModelTestRequest gains an optional ModelID field.
- New fillSecretsFromStoredModel helper: when ModelID is set AND a secret
field on the request is empty, copy it from the stored (decrypted) model.
Non-empty request values still win, so users typing a new key to verify
it before saving behave the same as before.
- The four test entrypoints (chat / embedding / rerank / asr) call the
helper right after ShouldBindJSON.
Frontend: ModelEditorDialog.testConnection forwards modelId when isEdit
is true. The four API client signatures accept the optional field.
WebSearch and MCP are unaffected: WebSearch already has a TestProviderByID
ID-based path, MCP has no standalone raw test endpoint.
Replace the legacy "redacted placeholder + Clear* boolean" pattern with
dedicated per-resource credential subresources across MCP services,
Models, Web search providers, and Data sources.
Why
---
The previous design had three problems:
1. Main PUT body carried secret fields. The frontend echoed back a
redacted "***" placeholder, and a fragile MergeUpdate / IsRedactedOrEmpty
defense in the service layer tried to detect "user did not change this"
vs "user wants to clear this". A regression in that defense (or a new
frontend forgetting it) silently overwrites the stored secret with the
placeholder.
2. The "remove this credential" UX was a red checkbox under a pre-filled
password input. Three intents (preserve / replace / clear) collapsed
onto one field, and credential changes were bundled with unrelated
config edits in the same submit. Users wiped working keys by mistake.
3. Secret presence was inferred from "did the response come back with a
'***' placeholder", which couples the contract to a magic string.
Design
------
Each of the four resources now exposes:
PUT /{resource}/{id}/credentials # write one or more fields
DELETE /{resource}/{id}/credentials/{field} # clear a single field
"Is this configured?" metadata travels on the main resource response as
a typed map (dto.MCPServiceResponse.Credentials etc.) — no separate GET
endpoint. The frontend reads the configured boolean from the main GET
and never sees secret values at all.
Main PUT endpoints now ignore any secret fields in the request body and
log a deprecation warning if they appear, so legacy clients fail loudly
rather than overwriting silently.
Frontend
--------
- New reusable <CredentialResource> component renders a three-state card
(unconfigured / configured / editing) and drives the dedicated
endpoints. Used by MCP, Model, Web search; DataSource has a bespoke
card with the same behaviour because its credentials are a single
atomic per-connector map.
- Cancel from edit mode now restores state synchronously from the meta
prop. The previous async refresh() was a no-op while state was still
'editing', leaving the input frozen open.
- Remove is single-click + toast. The danger-themed button is the
deterrent; a modal confirm adds friction without adding safety (the
plaintext is irrecoverable client-side either way — recovery means
re-typing).
DTOs (internal/handler/dto/) are deliberately separate from the GORM
models so "no secret in response" is a compile-time invariant: a future
contributor cannot leak a secret without explicitly adding the field to
the DTO, which is review-able in a single diff.
Storage is unchanged — credentials still live in the existing jsonb /
parameters columns. No schema migration.
Cleanup
-------
- types.MCPService / types.Model / types.WebSearchProviderEntity /
types.DataSourceConfig: drop ClearAPIKey / ClearToken / ClearAppSecret
/ ClearCredentials boolean fields, MergeUpdate(), RedactSensitiveData().
- utils/types/secret.go: drop PreserveIfRedacted / IsRedactedOrEmpty.
RedactedSecretPlaceholder constant is retained because VectorStore
still uses the old pattern and is out of scope here.
- Frontend hasExistingApiKey / clearApiKey / convertToLegacyFormat
redaction handling removed; i18n keys renamed secret -> credential.
Closes#988
Unify the "write-only secrets" pattern across MCP services, Models,
WebSearch providers, and DataSources. Fix two related defects:
1. GET responses leaked plaintext credentials — GetMCPServiceByID
while List masked, all Model / WebSearchProvider / DataSource GET
responses unconditionally.
2. When the frontend echoed a masked placeholder back on save, the
update path overwrote the stored secret with the mask — silently
wiping the real credential. MCP surfaced this most visibly:
ListMCPServices masked Token to "first4****last4" and
McpServiceDialog round-tripped that masked string back into the DB.
- internal/types/secret.go introduces the shared sentinel
RedactedSecretPlaceholder = "***" and two helpers:
- IsRedactedOrEmpty(s): true for "" or the sentinel
- PreserveIfRedacted(incoming, existing): merge predicate
- Each resource adds a RedactSensitiveData (mutates-in-place) or
Redacted (returns-a-copy) method and a MergeUpdate contract:
empty or placeholder preserves, any other value replaces, and a
write-only Clear* bool flag (gorm:"-", json omitempty) wipes with
precedence over preserve / replace. Clear* never persists.
- Every GET / List / Create / Update response path now redacts.
VectorStore's pre-existing "***" literal is refactored onto the
shared constant for consistency.
All four dialogs — McpServiceDialog, ModelEditorDialog,
WebSearchSettings, DataSourceEditorDialog — never pre-fill secret
inputs. The stored-state signal is collapsed into the input
placeholder: "•••••••• (Enter new value to replace)" when the server
reports a stored value, falling back to the connector-specific
placeholder otherwise. The input itself carries the "something is
there" cue, matching how password dots render.
An opt-in "Remove this credential" checkbox appears in edit mode when
a value is stored; checking it disables the input and triggers a
danger-themed confirmation dialog before submit. DataSource rolls the
clear up to a connector-level "Remove all credentials" flag since its
credentials are per-connector atomic bundles (e.g. Feishu app_id +
app_secret).
i18n updates across en-US, zh-CN, ko-KR, ru-RU:
secret.storedPlaceholder / clearHint / confirmClearTitle /
confirmClearBody.
- MCP update handler's manual map-to-struct projection was silently
dropping clear_api_key / clear_token / custom_headers on partial
updates. Pull them off the map explicitly.
- MCP configChanged detection now snapshots URL / StdioConfig /
TransportType before the merge so connection recycling still
triggers on real transport changes (not just whenever AuthConfig
is present).
- MCP CustomHeaders: nil preserves existing, non-nil replaces. Was
always overwriting, silently wiping headers on any auth-bearing
update.
- GetMCPServiceByID now redacts (was plaintext while List masked).
- WebSearchProvider update preserves Name / Description when empty
instead of clobbering them.
- WebSearchProvider TestProviderRaw rejects the "***" placeholder up
front with an actionable message rather than forwarding it to the
upstream provider API as a literal key.
- Model builtin hideSensitiveInfo now preserves non-secret metadata
(SupportsVision, Provider, ExtraConfig, InterfaceType,
ParameterSize, EmbeddingParameters) that were previously zeroed
out for builtin model cards.
- Replace local equalStringSlices / equalStringMaps helpers with
stdlib slices.Equal / maps.Equal (Go 1.21+).
- internal/types/{secret,mcp,model,web_search_provider,datasource}_test.go:
table-driven coverage of merge, redact, clear, and write-only flag
invariants for every resource.
- internal/application/service/mcp_service_test.go: full service layer
with a fake repo; covers ListMCPServices / GetMCPServiceByID
redaction, UpdateMCPService partial-update honoring of Clear*,
CustomHeaders preserve-on-nil / replace-on-non-nil, and
clear-flag precedence over submitted values.
Additive only. No changes to request or response JSON shape beyond
the write-only Clear* flags (omitted by default via omitempty). No DB
schema changes. No SDK struct field removals. Existing clients that
don't know about the new flags continue to function unchanged.
- DataSource.Config jsonb at-rest encryption. Pre-existing residual
risk; tracked as a follow-up.
- Renaming VectorStore.MaskSensitiveFields to match the new
RedactSensitiveData / Redacted naming convention. Left alone to
minimize diff; a follow-up rename PR can unify.
Follow-up to #1359. Addresses a set of correctness and security gaps in
the initial docreader auth implementation.
- docker-compose: inject GRPC_TLS_*/GRPC_TLS_SERVER_NAME/GRPC_AUTH_TOKEN
into the WeKnora-app service. Without this the Go client never saw the
knobs, so enabling token auth on the server broke every RPC.
- client: bind tokenAuth.RequireTransportSecurity() to TLSEnabled so a
bearer token cannot be sent over an insecure channel once TLS is on.
- server: load_tls_credentials now raises TLSConfigError on misconfig
(cert/key missing, file unreadable, mTLS without CA); main.py exits 1
instead of silently downgrading to insecure.
- server: replace endswith("/Check"|"/Watch") health bypass with exact
match against /grpc.health.v1.Health/{Check,Watch}.
- server: compare tokens with hmac.compare_digest, warn on tokens < 16B.
- server: AuthInterceptor now returns an abort handler matching the
original RPC kind (unary/stream) and uses context.abort, so streaming
RPCs surface UNAUTHENTICATED instead of INTERNAL.
- internal/infrastructure/docparser/grpc_parser.go: drop the duplicated
TLS/tokenAuth block and reuse docreader/client.LoadAuthConfigFromEnv +
BuildDialOptions. Single source of truth for client-side auth.
- Add GRPC_TLS_SERVER_NAME (client SNI override) and
GRPC_MTLS_REQUIRE_CLIENT_CERT (server explicit mTLS toggle); document
the differing CA semantics between client and server in .env*.example.
- Reject half-configured client mTLS (cert XOR key) loudly.
- Fix missing trailing newline in .env.lite.example.
Verified locally: go build ./... and go vet ./... clean; auth.py
fail-fast / token paths smoke-tested.
Five handler helpers (validateAndGetKnowledgeBase in knowledgebase.go,
validateKnowledgeBaseAccessWithKBID in knowledge.go, the kbService
guards in faq.go and tag.go, and getKnowledgeBaseForInitialization +
the per-kb config getter in initialization.go) wrapped every
GetKnowledgeBaseByID error — including the well-known
repository.ErrKnowledgeBaseNotFound sentinel — as
NewInternalServerError. The result was that every probe of a stale or
cross-tenant kb id surfaced as a 500 instead of the 404 it should have
been, both confusing clients ("real 5xx vs. wrong URL") and burning
ops attention on monitoring alerts.
The mapping pattern is the same as PR #1336 for sessions: detect the
sentinel via stderrors.Is and emit NewNotFoundError; everything else
still surfaces as a 500 so genuine DB / repo failures keep firing the
alerts that matter. Caught during the RBAC e2e smoke run on
feat/rbac, where a deliberate cross-tenant kb-id probe produced
HTTP 500 + body "knowledge base not found" — the smoking gun.
Tests: new internal/handler/knowledgebase_not_found_test.go covers
three cases — bare sentinel, fmt.Errorf("%w") wrapped sentinel
(regression guard against a future revert to `==`), and a non-sentinel
infrastructure error that must still 500. All three pass.
The full handler test package is green.
PR #1309 plumbed user-scope into the session/message service layer so
non-owner / wrong-tenant lookups now surface as ErrSessionNotFound at
every entry point that goes through sessionRepo.Get. The handler-side
mapping, however, only existed in 4 of the 8 affected sites:
- handler/session/handler.go:GetSession ✓ already mapped (with `==`)
- handler/session/handler.go:UpdateSession ✓ already mapped (with `==`)
- handler/session/handler.go:DeleteSession ✓ already mapped (with `==`)
- handler/session/handler.go:BatchDeleteSessions ✓ already mapped (with `==`)
- handler/session/handler.go:ClearSessionMessages ✗ always 500
- handler/session/stream.go:ContinueStream session lookup ✓ (with `==`)
- handler/session/stream.go:ContinueStream message lookup ✗ always 500
- handler/message.go:LoadMessages (recent + before-time) ✗ always 500
- handler/message.go:DeleteMessage ✗ always 500
Result: a Contributor in tenant A asking for a Contributor B's session
messages got a 500 instead of 404 — the wire response leaked
"something is broken" rather than "you can't see this URL", which
breaks SDK / frontend axios interceptor error handling that keys on
HTTP status.
Two changes:
1. Add ErrSessionNotFound → 404 mapping to the 4 sites that lacked it
(ClearSessionMessages, ContinueStream's GetMessage, LoadMessages's
two paths, DeleteMessage).
2. Replace `err == errors.ErrSessionNotFound` with
`stderrors.Is(err, errors.ErrSessionNotFound)` everywhere. Sentinel
error comparison must use errors.Is so wrapped errors
(`fmt.Errorf("...: %w", ErrSessionNotFound)`) still match. Today no
service-layer caller wraps, but a future refactor that does would
silently turn 404s into 500s with no test catching it.
Tests:
- internal/handler/message_session_not_found_test.go covers the four
new mapping sites in handler/message.go (LoadMessages-recent,
LoadMessages-before-time, DeleteMessage) and pins the wrapped-error
behaviour with a dedicated regression test that fmt.Errorf("%w") a
sentinel must still be detected as ErrSessionNotFound. The bare
handler is hit through gin.Engine + middleware.ErrorHandler so the
response shape matches production exactly.
- PR #1309's existing service-level tests (session_user_scope_test,
session_test on the repository) still pass — this PR is a strictly
handler-layer follow-up.
When a startup database migration fails (e.g. issue #1319: pg_trgm not
available so 000041 cannot build its trigram index), the application
intentionally keeps booting so operators can reach the UI to diagnose.
However, before this change the system info page silently dropped the
"DB Version" row because the value was empty:
- migration.go only cached the version after a successful m.Up(); the
error path returned early and left migrationVersionSet=false.
- system.go used CachedMigrationVersion's ok=false to skip emitting
db_version, and the JSON tag was already omitempty.
- SystemInfo.vue gated the entire row on v-if="systemInfo?.db_version".
The end result was the most useful diagnostic surface vanishing in the
exact failure mode that needs it most — Wiki ingest and KG features
would silently produce nothing with no UI hint.
Changes:
- migration.go: replace sync.Once-based setter with an RWMutex-guarded
state struct holding {version, dirty, err}. Every failure path now
calls captureMigrationFailure(m, err), which best-effort reads
m.Version() so the cached value still reflects the partial state.
- system.go (handler): always emit db_version (falling back to
"unknown" when no version could be read), append " (failed)" when an
error is recorded, and add db_migration_error to the response.
- swagger / client SDK: keep the API contract in sync with the new
response field.
- SystemInfo.vue: render the DB version row whenever either field is
present, show a "Migration failed" danger tag, and add a full-width
alert below the row carrying the error message plus two links:
1. View troubleshooting guide -> new docs/migration-troubleshooting.md
2. Report an issue -> github.com/Tencent/WeKnora/issues/new,
prefilled with the captured error and environment metadata.
- docs/migration-troubleshooting.md: new self-service guide covering
the common failure modes (missing extension, dirty state, privileges,
out of disk, schema drift) with concrete psql / make commands.
- i18n: add the new keys to zh-CN, en-US, ko-KR, ru-RU.
Refs #1319.
Co-authored-by: Cursor <cursoragent@cursor.com>
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across
seven services into two factory functions with tenant-ownership
verification: CreateRetrieveEngineForKB (synchronous) and
CreateRetrieveEngineFromPayload (async task handlers). Promotes
GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry.
Extends KBDeletePayload / IndexDeletePayload with an omitempty
VectorStoreID snapshot.
Design highlights
- Factory verifies tenant ownership of the resolved store
(defense-in-depth) so a gap in PR3 validation or a tampered Asynq
payload cannot cross tenants. Cross-tenant attempts return
ErrVectorStoreForbidden with a structured log entry.
- Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound,
ErrVectorStoreForbidden) let async handlers classify non-retryable
failures as asynq.SkipRetry.
- nil and empty-string pointers for vectorStoreID normalize to
"unbound" so callers never send an empty UUID to GetByStoreID.
- resolveBoundEngine constructs engineInfos directly (with
slices.Clone of Support()) instead of going through
NewCompositeRetrieveEngine. The latter only reads from the
byEngineType map and cannot reach DB stores in byStoreID -- multiple
stores can share the same engine type. Unbound fallbacks still reuse
NewCompositeRetrieveEngine so the tenant effective-engines path is
byte-for-byte unchanged.
- DeleteKnowledgeBase now reads the KB before soft-delete so the
enqueued KBDeletePayload carries a VectorStoreID snapshot; once the
row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT
per KB-delete request (a rare admin operation).
- enqueueIndexDeleteTask gains a vectorStoreID *string parameter that
the single caller (tag.go) populates from the owning KB. The worker
(ProcessIndexDelete) validates ownership via the factory.
- cleanupKnowledgeResources loads the KB so bound-store cleanup routes
correctly. If the load fails it warns and falls back to tenant
effective engines; orphan vectors become observable in logs instead
of disappearing silently.
- DeleteKnowledges (batch) stays on the tenant effective-engines path
-- batches can span stores and multi-store fan-out is PR4 scope.
Noted inline.
Merge safety
All knowledge bases have VectorStoreID = NULL at merge time (introduced
nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to
NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()),
which is the existing code path for these KBs. Async payloads from
before this change decode to VectorStoreID = nil (omitempty) and take
the same fallback.
Testing
- go build ./...
- go vet ./...
- go test -race -count=1 ./internal/application/service/retriever/...
covers unbound (nil/empty), bound, cross-tenant, unregistered store,
ownership infra error, legacy payload shapes (missing field/null/
empty string/nil), tampered payload, and parallel invocation.
- grep -r NewCompositeRetrieveEngine internal/application/service
returns zero results outside factory.go/composite.go.
Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994.
JoinByOrganizationID no longer delegates to JoinByInviteCode for direct
join, matching the product copy that searchable spaces need no invite
code. Extract joinAsViewerWithChecks for shared member-limit logic.
Co-authored-by: Cursor <cursoragent@cursor.com>
MiMo and DeepSeek V3.2/V4 reject multi-turn requests in thinking mode
when the prior assistant message lacks reasoning_content with HTTP 400:
"The reasoning_content in the thinking mode must be passed back to the API."
Agent's ReAct loop is the worst-case scenario — every round produces tool
calls, exactly the case DeepSeek's docs specify reasoning_content MUST
participate in subsequent context.
Plumb reasoning_content through the full assistant-message round-trip:
- chat.Message / types.ChatResponse / types.AgentStep gain a
reasoning_content field (AgentStep persists via the existing
Message.AgentSteps jsonb column, no migration needed).
- streamLLMToEventBus accumulates reasoning chunks into
result.ReasoningContent and surfaces it on the round's ChatResponse.
- engine.runReActIteration writes it onto AgentStep so cross-turn replay
preserves it.
- observe.appendToolResults attaches it to the same-turn assistant
message; agent_history.buildAssistantHistoryMessages does the same on
cross-turn replay.
- RemoteAPIChat.ConvertMessages forwards it on assistant turns to
openai.ChatCompletionMessage.ReasoningContent (already supported by
go-openai); providers that don't recognize the field ignore it.
Tests cover the three boundaries: ConvertMessages serializes it for
upstream, appendToolResults preserves it within the same turn, and
buildAssistantHistoryMessages replays it across turns.
Scope is intentionally limited to Agent mode — KnowledgeQA's chat
pipeline and Anthropic's signed thinking_blocks are separate fixes that
require schema changes (rendered_content / thinking_blocks columns).
Fixes#1302
OpenAI's GPT-5 series and o-series reasoning models (o1/o3/o4-mini) no
longer accept `max_tokens` and reject non-default sampling params
(`temperature`, `top_p`, `frequency_penalty`, `presence_penalty`).
Azure OpenAI propagates the same constraint, returning HTTP 400 with
"this model is not supported MaxTokens, please use MaxCompletionTokens".
Lower the compatibility shim into `RemoteAPIChat.BuildChatCompletionRequest`
(the single OpenAI-protocol egress) so every internal caller keeps using
`MaxTokens` uniformly:
- Add `provider.IsOpenAIReasoningOrGPT5Model` with precise prefix matching
(covers `gpt-5*`, `o1`/`o1-*`, `o3`/`o3-*`, `o4`/`o4-*`; rejects
`olympus-1`, `openai-*`, `o3xtra`, etc.).
- When the provider is `openai`/`azure_openai` and the model matches, map
`MaxTokens` to `MaxCompletionTokens` (explicit `MaxCompletionTokens`
wins) and skip the unsupported sampling fields; `omitempty` keeps them
off the wire.
- Behavior is unchanged for gpt-4o, gpt-4, and all other providers.
Add unit tests for the matcher and for the request-building path
(Azure gpt-5.2, OpenAI gpt-5/o1-mini/o3/o4-mini, plus negative cases).
Fixes#1283
To diagnose where latency lives between "user hits send" and "first
token appears" we need a single number that can be matched across
the browser console and the server log. Add correlated TTFB markers
keyed by X-Request-ID:
Frontend (streame.ts)
* Generate the X-Request-ID once and reuse it for logging.
* Log request:start when fetchEventSource is invoked, response:headers
when onopen fires, and response:first_answer the first time an SSE
payload with response_type === 'answer' arrives. Filtering by event
type avoids treating session_title / references / tool_call as the
"first token".
Backend (session handler)
* parseQARequest records the wall-clock entry time and logs TTFB:start
with the same X-Request-ID.
* qaRequestContext carries receivedAt through to the stream handler.
* AgentStreamHandler emits a one-shot TTFB:first_answer_chunk log the
first time it observes an AgentFinalAnswerData chunk, so the delta
against TTFB:start is the server's request-in → first-token-out
budget.
The frontend delta minus the backend delta is then attributable to
network + gin middleware, which were previously invisible.
Previously the Langfuse timeline for a knowledge-chat request only
showed generations (chat.completion, embedding.embed, rerank), so the
work happening between them — query setup, vector/keyword search,
result merging, prompt assembly — appeared as unexplained gaps.
Add spans around the pieces that fill those gaps:
* qa.setup wraps request-time KB / model / search-target resolution
before the pipeline event loop starts, accounting for the visible
delay before the first chat.completion.
* pipeline.<event> wraps each pipeline stage's eventManager.Trigger
call so generations inside a stage nest under it. CHAT_COMPLETION_
STREAM is intentionally skipped because its OnEvent returns as soon
as the streaming goroutine starts; a stage span would always finish
before the chat.completion.stream generation it nominally parents.
* retrieve wraps the actual vector + keyword retrieve call inside
HybridSearch, exposing the DB round-trip that previously sat invisibly
between embedding generations and rerank.
* web_search wraps the external web-search HTTP call when enabled.
Quick-answer (RAG) agents can now configure a separate chat model for
the query-understanding stage (rewrite + intent classification),
decoupling it from the main conversation model so users can route the
lightweight rewrite call to a cheaper / faster model.
- Add CustomAgentConfig.QueryUnderstandModelID and plumb it through
PipelineRequest and ChatManage.Clone.
- query_understand plugin prefers QueryUnderstandModelID on the
text-only path; falls back to ChatModelID if the configured model
cannot be resolved (with a warn log). Multimodal path is unchanged
to keep vision-capable model selection intact.
- AgentEditorModal exposes a ModelSelector under the existing Query
Rewrite block; empty means reuse the main chat model.
- Add i18n strings (queryUnderstandModel / placeholder / desc) for
zh-CN, en-US, ko-KR, ru-RU.
- Extend Go SDK AgentConfig with the new field.
The legacy in-pipeline DuckDB SQL data-analysis stage used to run on every
quick-answer RAG request whose retrieved chunks included a CSV/Excel file,
adding one extra LLM round-trip (~3s) to generate a SQL query that most
plain Q&A users never wanted. There was no way to disable it.
Introduce a per-agent DataAnalysisEnabled flag (default off), wire it
through PipelineRequest, and gate the DATA_ANALYSIS stage on it. Surface
the toggle in the agent editor for quick-answer agents with at least one
knowledge base attached.
Refs: https://github.com/Tencent/WeKnora/issues/1244
The request logger's sanitizeBody only matched lowercase / snake_case
field names (api_key, apikey, access_token, ...), so values for the
camelCase JSON fields actually used by the API (apiKey, secretKey,
refreshToken, accessToken, ...) were written to logs in clear text.
Replace the per-field patterns with a single case-insensitive regex
that tolerates optional `_`/`-` separators, covering snake_case,
camelCase and PascalCase variants, and extend coverage to id_token,
client_secret, private_key, auth_token, api_secret and passwd. The
field name is preserved; only its value is replaced with "***".
Add unit tests for sanitizeBody covering the previously-leaking
camelCase fields and common variants.
Fixes#1287
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.
Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
(tag, keyword, file_type, parse_status, source, updated_from/to) and switch
ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
column to stay consistent with file_type semantics; other source values match
the channel column.
- Update the four other callers (agent_service, initialization) to pass an
empty filter struct, preserving prior behavior.
Frontend
- Add three controls in the doc-filter-bar (status select, source select,
date-range picker with future-date disabled) wired through getKnowled /
listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
(mirrors the session list UX): in card view the checkbox only renders while
batch mode is on, entered via the per-card "..." menu; the list view keeps
its leading checkbox column. Switching from list to grid auto-enables batch
mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
its "取消选择" button both clears the selection and exits batch mode.
API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
Go SDK; the existing ListKnowledge keeps its signature.
i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
menu.batchManage / batchManage.cancel for the multi-select strings.
When image bytes for a multimodal task cannot be read via FileService
(e.g. tenant.StorageEngineConfig.MinIO is empty while the image was
saved using the global MINIO_* env vars), the previous code fell back
to the HTTP downloader, which rejected the provider:// URL with
"unsupported URL scheme". The asynq handler then returned an error,
asynq retried until exhaustion, and the per-knowledge "pending images"
counter was never decremented — leaving the document stuck in
"processing" indefinitely (issue #1282).
Changes:
- ImageMultimodalService now holds a default FileService and
resolveFileServiceForPayload falls back to it when the tenant-scoped
storage config cannot produce a usable service, mirroring the
write-side fallback in knowledgeService.resolveFileService.
- Extract readImageBytes: provider:// URLs are read exclusively via
FileService and never handed to the HTTP downloader.
- On unrecoverable read failure for a single image, log and skip that
image but still call checkAndFinalizeAllImages so the parent
knowledge can progress to post-processing.
Fixes#1282
Previously the custom agent editor hid the rerank model field when no
RAG-type knowledge base existed in the configured scope, but the
not-ready check and runtime hard-failed whenever knowledge_search was
in allowed_tools. Users with wiki-only or empty scopes saw "Rerank
Model required" warnings they could not resolve, and "All knowledge
bases" agents broke later if a RAG-type KB was added.
- Backend: when knowledge_search is enabled, fall back to the tenant
default rerank model (ConversationConfig.RerankModelID) before
erroring out, matching the built-in agent behaviour.
- Editor: always show the rerank field once a KB scope is selected;
only mark it required (red *) when the scope contains a RAG KB, with
a hint explaining the tenant-default fallback.
- Editor: only render rerank top_k / threshold sliders when a rerank
model is actually selected.
- Input field: drop the eager "missing rerank" not-ready reason; the
backend is now the single source of truth for rerank availability.
- i18n: add agent.editor.rerankModelOptionalHint across all locales.
`metadata->'similar_questions'::text` parses as
`metadata -> ('similar_questions'::text)` because `::` binds tighter
than `->`, so the expression yields jsonb instead of text. ILIKE on
jsonb then fails with "operator does not exist: jsonb ~~* unknown",
returning 1007 Internal server error for `search_field=similar_questions`
and `search_field=answers`. The `standard_question` branch was unaffected
because it uses `->>` (text) directly.
Wrap the json access in parens so the cast applies to the extracted
value: `(metadata->'similar_questions')::text ILIKE ?`. The MySQL
branch uses JSON_EXTRACT and is unchanged.
Fixes#1264
Quick-answer agent mode retrieves purely through vector/keyword chunk
search and ships with no `allowed_tools`, so the existing capability
filter (which only reads from `allowed_tools`) let wiki-only KBs through
in every entry point. End result: users could @-mention, select, and
receive suggested questions from wiki-only KBs in quick-answer mode,
but the underlying retrieval always returned empty.
Treat "RAG-only" as an implicit property of `agent_mode = quick-answer`
and union it with the tool-derived filter. The same predicate is now
used everywhere the user can pick or be steered toward a KB:
Backend
- `tools.DeriveKBFilterForAgent` / `KBSatisfiesAgentRequirements`
layer the implicit quick-answer requirement on top of tool derivation.
- `ListKnowledgeBases`, `SearchKnowledge` (shared-agent `@file`),
`resolveKnowledgeBasesFromAgent` (chat runtime), `/search` IM command,
and `GetSuggestedQuestions` now all use the agent-mode-aware variant.
- `GetSuggestedQuestions` also skips the wiki-page fallback for
quick-answer agents to cover the `selected` / explicit-kb-ids paths
where a wiki-only KB could still slip through.
Frontend
- `deriveKbFilterForAgent` / `kbSatisfiesAgentRequirements` mirror the
Go helpers.
- `@` mention dropdown (`Input-field.vue`) uses the new helper.
- Agent editor's "specified KB" picker (`AgentEditorModal.vue`) grays
out wiki-only KBs for quick-answer agents with a tooltip, and the
pre-save warning fires for quick-answer mode too.
- i18n: add `agentEditor.agentType.kbMismatch.quickAnswer` across all
four locales.
- Updated `pydantic` from 2.12.3 to 2.13.4 and `pypdfium2` from 5.0.0 to 5.8.0 in the `docreader` requirements.
- Modified the dependabot configuration to set `open-pull-requests-limit` to 0 and added an `ignore` rule for version updates across all ecosystems, allowing only security updates.
- Adjusted settings for `server-security`, `client-security`, `frontend-security`, and `miniprogram-security` groups to streamline security update handling.
This change aims to enhance dependency management and maintain security while reducing noise from version update PRs.
Agent mode does not persist `rendered_content` for user messages, so
when the next turn's history was rebuilt from DB, attachments uploaded
in prior turns disappeared — the model only saw the raw query plus the
prior assistant reply, breaking follow-up questions that referenced
the file (e.g. "what is in there?").
Reconstruct the attachment prompt from the stored `Attachments` column
when `RenderedContent` is empty, mirroring how image captions are
already replayed. KnowledgeQA turns (which do persist
`RenderedContent`) are unaffected and won't get attachments injected
twice.
Refs #1237
The Agent ran a parallel context cache (Redis/in-memory) on top of the
messages table to feed multi-turn history into the LLM. That dual-write
caused subtle drift (e.g. compression diverging from DB, system-prompt
swaps lost on restart) and required a separate ClearContext path on
session/IM clear.
Make the messages table the single source of truth:
- Add service.LoadAgentHistory: rebuilds chat.Message history per turn
from the persisted messages, expanding AgentSteps into proper OpenAI
assistant_with_tool_calls + tool messages and replaying the canonical
final answer (with <think> blocks stripped). final_answer tool calls
are filtered to avoid duplicating the trailing answer.
- Make AgentEngine stateless across turns: drop ContextManager / sessionID
cache plumbing from the engine, agent service, and CreateAgentEngine
signature. The engine only uses sessionID for logging/event emission.
- Wire AgentQA to load history from DB on demand using HistoryTurns
(default 5) when MultiTurnEnabled, otherwise run with empty history.
- Delete the llmcontext package (ContextManager interface, Redis/memory
storage, factory) and the SessionService.ClearContext API path; IM
/clear and session message clear no longer need to invalidate cache.
Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with
empty AgentSteps just produce the single canonical assistant message),
and Agent-mode turns now consistently see prior tool calls and results.
The SearXNG provider forwarded `parameters.api_key` as
`Authorization: Bearer <key>` for reverse-proxy auth, but the frontend
never rendered the API key input for self-hosted providers
(`requires_api_key=false`), so the path was unreachable from the UI.
Rather than expose another UI knob for a niche reverse-proxy setup that
the project does not currently support, remove the Authorization header
branch and the `apiKey` field on `SearxngProvider`. Tenants who put
SearXNG behind an authenticating reverse proxy can still front it with
network-level auth (mTLS, IP allowlist) without changes here.
`WebSearchProviderParameters.APIKey` is generic and used by other
providers, so it stays.
- Adjusted the formatting of struct fields in the SSRF whitelist test cases for improved readability and consistency.
- No functional changes were made; this is purely a code style improvement to enhance maintainability.
- Updated .env.example to clarify SEARXNG_SECRET generation and added SSRF_WHITELIST_EXTRA for improved security.
- Modified docker-compose files to bind SearXNG to localhost by default and introduced a one-time initialization service to set up settings.yml correctly.
- Enhanced SearxngProvider with stricter URL validation, ensuring no query or fragment is present in the base URL.
- Added unit tests for SearXNG validation and date parsing to ensure robustness.
- Updated frontend WebSearchSettings to reflect changes in SearXNG instance URL handling.
This commit improves the security and usability of the SearXNG integration, addressing potential misconfigurations and enhancing the developer experience.
- Introduced a new test to ensure that empty user IDs are rejected when a waiter has a user, preventing unauthorized approvals.
- Added RequestNonce to resolve messages to uniquely identify Resolve calls, ensuring concurrent requests do not interfere with each other.
- Updated waiter structure to use atomic operations for the resolved state, improving thread safety.
- Enhanced error handling in the Resolve method to properly manage user mismatches and ensure accurate acknowledgment delivery across instances.
- Updated MCP service handler to reject unauthenticated requests early, providing clearer feedback on authorization requirements.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
items" had two args), and harden validMerge against un-prefixed
slugs whose strings.Index returned -1 and silently passed the type
check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
letter row instead of a hard-coded 0; tighten payloadProbe to the
set of field names with consistent semantics across payloads
(drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
outside an asynq worker ctx so the semantics stay pinned.
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
The ollama Go SDK changed ToolCallFunctionArguments from
map[string]any to a struct backed by an ordered map. Use ToMap() for
read access and NewToolCallFunctionArguments()+UnmarshalJSON for
construction so the build matches the upgraded SDK.