Gate wiki GET routes and initialization config reads with KBAccessRead,
and bind async task progress endpoints to the caller tenant via task ID
parsing. Closes cross-tenant read paths reported in GHSA-ph8f-4qwx-89rr
and related initialization/progress leaks.
Harden validateNode with default-deny semantics and PG17 SQL/JSON node
recursion so dangerous functions cannot hide inside JSON_SCALAR/JSON_VALUE.
Also recurse into FROM subqueries for DuckDB data_analysis and block
read_text/read_blob and related file-access functions.
checkSQLInjectionRisks and InjectAndConditions compiled their regexps
with regexp.MustCompile on every invocation. Both run on the datasource
SQL path — InjectAndConditions is called once per security filter
(tenant, soft-delete, hidden-KB, search-scope), so a single secured query
recompiled ~20 patterns. InjectAndConditions also compiled the identical
tail-clause pattern twice.
Hoist them to package-level vars (compiled once at init). Behavior is
unchanged — same patterns, covered by the existing TestValidateSQL_*
and TestInjectAndConditions tests.
Benchmarks (apple M, -benchmem):
InjectAndConditions 4347 ns, 72 allocs -> 1023 ns, 7 allocs
checkSQLInjectionRisks 36074 ns, 560 allocs -> 12984 ns, 12 allocs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit introduces a new validation mechanism to ensure that file access paths include the correct tenant segment, preventing cross-tenant access. The `ValidateStoragePathTenant` function has been added to enforce this rule, and the `serveFiles` function has been updated to return a forbidden status for invalid paths. Additionally, new tests have been added to verify the behavior of the file service under various tenant scenarios, ensuring robust handling of file access permissions.
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.
Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
key so the 3-tier resolver falls back to ENV / built-in default.
Idempotent (resetting a never-persisted key returns nil); emits an
audit row only on real deletions, invalidates the local cache, and
publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
storage_quota in one statement. Powers POST /system/admin/tenants/
apply-default-storage-quota; bypasses the per-tenant whitelist on
PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
emitAdminAudit() in SystemHandler — promote / revoke now leave an
audit trail with TenantID=0 and {target_email,target_username,
idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
(username, email fallback) so the UI shows "wizardchen" instead of
a UUID prefix without storing a denormalised column.
Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
auto-persisting controls (switch/select @change, input/inputnumber
@blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
roster). auth.registration_mode change goes through an inline
popconfirm; cancel rolls back. Reset / bulk-apply also inline
popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
admins are now compatibility redirects to /platform/settings?section=
system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
so the wider tables (members, system settings) breathe; <1100px
viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
empty / badges / priority hint / per-key labels / reset / bulkApply /
admins (label, placeholder, save messages, popconfirm copy) across
zh-CN, en-US, ko-KR, ru-RU.
Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
deploy-time-only knob (nginx + docreader + frontend each cache the
env at startup); a SystemAdmin UI override would mislead operators
because nginx would still 413. Until all four layers can hot-reload
the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
off SystemSettingService for live updates; ENV remains the fallback
for never-overridden deployments.
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.
Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.
What lands in PR 2a
-------------------
Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):
- `repository.go` — Repository struct + NewRepository constructor
that validates cluster reachability + OS version (2.4+ / 3.x;
primary tested 3.3.2) + k-NN plugin presence on every cluster
node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
dimension index creation, with transient errors not cached so a
momentary cluster blip does not permanently poison a dim.
sanitizeIndexName enforces a strict OS-compatible name spec.
probeVersion uses robust strings.Split/Atoi parsing for
pre-release suffixes and missing-patch versions. EngineType
returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
(MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
only cipher list) and transport tuning for the driver. Caller
exists only in PR 3 (container.go + engine_factory.go); PR 2a
remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
knn_vector + HNSW + content-analyzer mapping with every *_id
field as an explicit keyword and source_type as integer.
buildKeywordsMapping ships the dim-less keyword-only index
mapping used by the no-embedding save path. createIndexAndAlias
creates <alias>_v1 and aliases <alias> to it, with best-effort
orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
defaults (hnsw_m=16, ef_construction=100, ef_search=100,
shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
ErrDimensionMismatch, ErrAuth, ErrTransport,
ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
apperrors; PR 3's engine factory wraps these to typed AppError
2200/2201.
- `stubs.go` — every behavioural method returns
ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
fails-closed for non-empty KBs.
Tests (~1115 LoC, 50 cases):
- `repository_test.go` — interface satisfaction, sentinel mapping,
sanitizeIndexName positive/negative matrix, semver parsing
(pre-release / missing-patch), buildIndexMapping JSON shape pin
(Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
multi-node coverage, ensureReady concurrency + per-dim isolation
+ transient retry, NewRepository storeID validation, all 11
stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
wrapTransport sentinel mapping + leak guard, isNotFound /
isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
TLS 1.2 pinning / cipher list / transport tuning.
Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.
SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------
PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):
- AliasExists method passes dataPointer=nil to its internal do(),
which means non-2xx responses come back as a plain
*errors.errorString ("status: 404 Not Found") rather than as
*opensearch.StructError. aliasExists therefore inspects
resp.StatusCode directly (resp is returned even when err is
non-nil) and only falls back to wrapTransport for the "no
response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
index uses a mutex + ready/err flag pattern so transient failures
can be retried by the next caller. The per-dimension path uses
the `once map[int]*sync.Once` delete-and-recreate trick.
Test fixes folded in
--------------------
While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:
(1) Follow-up to #1445 — fanout test missed the new normalizer policy
(internal/application/service/knowledgebase_search_fanout_test.go,
+46 / -6). #1445 changed EngineAwareNormalizer for ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
passthrough (those engines surface non-negative cosine to the
normalizer per Lucene script_score non-negative invariant for
ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
OpenSearch, engine-internal or IR-normalized conversions for
the rest). Milvus is now the only engine that still surfaces
raw signed cosine in [-1, 1].
TestRetrieveFromStores_MixedEngine_Normalizes still asserted
the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
which under passthrough now becomes clamp01(-0.4) = 0. The
normalizer's own _test.go was updated at #1445 time, but this
fan-out integration test was not.
Fix: rewrite the godoc to spell out the two engine groups;
restate sub-case 2 as ES passthrough on a production-possible
mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
pinning the cosine-shift branch via Milvus -0.4 → 0.3.
(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
(internal/utils/security.go + internal/utils/security_test.go +
internal/infrastructure/web_search/searxng_test.go,
+33 / -9). loadSSRFWhitelist in internal/utils/security.go is
cached via sync.Once on first call. The internal reset helper
resetSSRFWhitelistForTest was unexported, so tests in other
packages could not reset and saw whatever whitelist was cached
by the first sync.Once.Do() in the same test binary. In
internal/infrastructure/web_search/, TestValidateProxyURL runs
before TestValidateSearxngBaseURL alphabetically and exercises
ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
whitelist; the later setenv in searxng_test then has no effect
and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
Pre-existing on main; surfaced now because this PR was the
first to do a full `go test ./...` run on top of #1445.
Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
ForTest suffix is the test-only contract); update in-package
callers; in web_search/searxng_test.go import internal/utils
and call ResetSSRFWhitelistForTest around the env mutation in
both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
No production code path changes.
Roadmap
-------
- PR 2b (next, depends on this PR) — read/write implementations:
query.go + retrieve.go + crud.go land their real bodies; stubs
for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
removed; corresponding CRUD/retrieve/filter test cases (~430
LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
BatchUpdate*, large-batch async deletes) + i18n + docker-compose
dev profile. After PR 3 merges, the OpenSearch driver becomes
reachable via either `engine_type=opensearch` VectorStore or
`RETRIEVE_DRIVER=opensearch` env.
Backward compatibility
----------------------
- New package — additive only. No existing file modified except
go.mod / go.sum, the two test files in (1)/(2), and the
test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
here (no driver instance produces a result yet).
Test plan
---------
- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
shows only PR 1's normalizer case + this driver's EngineType()
and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
Test-connection's "连接失败" was nearly useless for debugging because
two layers conspired to swallow the actual upstream error:
Backend swallow
---------------
checkChatModelConnection and CheckASRModel pattern-matched on the
upstream error string (401 / 404 / timeout / ...) and replaced it with
a fixed Chinese hint. The original SDK message (request URL, response
body, provider's error code description) was discarded.
Refactor: extract classifyConnectionError(errMsg) — returns just the
short Chinese hint. Callers combine "<hint>：<raw err>" so the hint
indicates "where to start looking" and the raw error shows "what
actually happened". The status-400 special-case (endpoint reachable,
parameter mismatch only) still returns success.
Frontend swallow
----------------
ModelEditorDialog.testConnection ignored result.message on success
paths (fine), but also overwrote it with a generic i18n string on
failure paths and 4xx rejections — even though axios's interceptor
already lifts the backend's error.message to error.message. Failure
branch now uses error?.message || result.message as the toast content;
i18n fallback only when neither is present.
SSRF allowlist hint
-------------------
SSRF validation failures previously returned
"Base URL 未通过安全校验: <reason>" with no mention of how to recover
(the allowlist is server-side, not configurable in the UI). Operators
hit it and were stuck.
New utils.FormatSSRFError(label, rawURL, err) wraps the error with
guidance to add the host to SSRF_WHITELIST_EXTRA (the operator-facing
append-only escape hatch — SSRF_WHITELIST itself is the project's
baseline). The hint echoes the failed host so it can be copy-pasted
straight into the env var:
Base URL 未通过安全校验：DNS resolution failed for hostname
test.local.com ... 如该地址确实可信，请联系运维在服务端环境变量
SSRF_WHITELIST_EXTRA 中加入该主机（支持精确域名 / *.example.com
通配 / IP / CIDR），示例：SSRF_WHITELIST_EXTRA=test.local.com,
*.example.com,10.0.0.0/8
All SSRF call sites switch to the helper:
initialization (CheckRemoteModel / TestEmbedding / CheckRerank /
CheckASR / TestMultimodal / batch validation), model CRUD,
mcp_service CRUD, knowledge URL ingest, system ReconnectDocReader.
Two encryption gaps and one fail-loud regression, fixed together because
they are inseparable in practice.
At-rest encryption gaps
-----------------------
Before this change, only Tenant.APIKey, Tenant.Credentials.WeKnoraCloud.AppSecret,
Model.Parameters.APIKey/AppSecret, and WebSearchProviderParameters.APIKey
were encrypted with SYSTEM_AES_KEY. MCP and DataSource were missed:
- MCPAuthConfig.Value/Scan went through bare json.Marshal/Unmarshal, so
api_key and token were written to mcp_services.auth_config (json) in
plaintext regardless of whether SYSTEM_AES_KEY was set.
- DataSource.Config (types.JSON) is a byte passthrough, so connector
secrets (Notion token, GitHub PAT, Feishu app_secret, ...) inside the
Credentials map were stored in plaintext too.
Now both follow the ModelParameters / WebSearchProviderParameters pattern.
Encryption operates on a local copy to avoid mutating the caller's
in-memory struct (otherwise the next read of the same *MCPAuthConfig
would surface ciphertext as if it were the API key). For DataSource the
ToJSON ↔ ParseConfig pair is the only application boundary that
materializes credentials, so encrypting there is sufficient and avoids
having to fork the generic types.JSON.
Backward compatibility
----------------------
DecryptStoredSecret returns rows without the enc:v1: prefix verbatim, so
historical plaintext records keep working. Next write re-saves them
encrypted. No migration step.
Graceful Scan-time degradation
------------------------------
The strict "fail loud on key rotation" behaviour of DecryptStoredSecret
turned into a footgun once it ran from GORM Scan: a single broken row
caused ListModels / ListMCPServices / ListDataSources to return zero
results, hiding all other rows and giving the operator no way to find
which credential needed re-entering.
New utils.DecryptStoredSecretLenient: same logic, but on failure returns
("", false). The strict variant is kept for hot paths that are about to
actually USE the credential (so a rotated key surfaces loudly there
rather than silently degrading to "missing api key").
All Scan paths switch to lenient + per-failure log line:
ModelParameters.Scan, WebSearchProviderParameters.Scan,
MCPAuthConfig.Scan, DataSource.ParseConfig, CredentialsConfig.Scan.
On lenient failure the field blanks out (never ciphertext), the row
still loads, the DTO reports configured=false so the UI shows "not
configured", and any upstream call attempted with the empty value fails
with a clear "missing api key" rather than sending ciphertext as the key.
Tenant.AfterFind (Tenant.APIKey) intentionally stays strict — it is the
primary HTTP auth credential and a silent empty value would change auth
semantics ambiguously.
- Adjusted the formatting of struct fields in the SSRF whitelist test cases for improved readability and consistency.
- No functional changes were made; this is purely a code style improvement to enhance maintainability.
- Updated .env.example to clarify SEARXNG_SECRET generation and added SSRF_WHITELIST_EXTRA for improved security.
- Modified docker-compose files to bind SearXNG to localhost by default and introduced a one-time initialization service to set up settings.yml correctly.
- Enhanced SearxngProvider with stricter URL validation, ensuring no query or fragment is present in the base URL.
- Added unit tests for SearXNG validation and date parsing to ensure robustness.
- Updated frontend WebSearchSettings to reflect changes in SearXNG instance URL handling.
This commit improves the security and usability of the SearXNG integration, addressing potential misconfigurations and enhancing the developer experience.
GORM Scan / AfterFind hooks for every AES-encrypted column followed
the same lenient pattern:
if decrypted, err := utils.DecryptAESGCM(c.APIKey, key); err == nil {
c.APIKey = decrypted
}
When SYSTEM_AES_KEY was missing, rotated, or the wrong length the
decryption error was swallowed and the in-memory struct kept the raw
"enc:v1:..." ciphertext. The application then happily forwarded the
ciphertext upstream as the actual API key / password, surfacing as
401/403/SignatureDoesNotMatch from third-party vendors. Worse, a
ciphertext snippet of a customer credential was leaking into the
external provider's request logs.
Introduce utils.DecryptStoredSecret that:
- returns "" / legacy plaintext untouched (no false positives for
pre-encryption rows);
- returns ErrEncryptedDataMissingKey when the value carries the
enc:v1: prefix but no AES key is configured;
- propagates any GCM auth-tag failure from a rotated key.
Wire it into the five Scan / AfterFind sites that currently swallow
the error:
- Tenant.AfterFind (tenants.api_key)
- CredentialsConfig.Scan (tenants.we_knora_cloud.app_secret)
- ModelParameters.Scan (models.parameters.api_key, .app_secret)
- ConnectionConfig.Scan (vector_store_connections.password, .api_key)
- WebSearchProviderParameters.Scan (web_search_providers.api_key)
The error is wrapped with the originating column so the failure is
diagnosable in the logs. Operators must restore the previous
SYSTEM_AES_KEY (or rotate the affected secrets) instead of receiving
silent vendor 401s.
Add table-driven tests covering empty input, legacy plaintext,
round-trip, missing key, wrong-length key, and rotated key. The
strict path explicitly asserts that ciphertext does NOT leak in the
returned plaintext on any error path.
Reverts the tenant-ID-from-context change in da5fcd3. The storage path
encodes the resource owner's tenant, and the presigned-URL verifier
uses that ID to look up the owning tenant's StorageEngineConfig. Using
the caller's tenant ID from context would break cross-tenant shared
resources — e.g. when tenant Y reads an image from a KB shared by
tenant X, signing with Y would cause the verifier to open Y's storage
backend and 404 on X's file.
Keep:
- presignDefaultTTL shortened from 24h to 2h (independent improvement).
- godoc note on ParseTenantIDFromStoragePath flagging the ambiguity
for cloud paths with numeric bucket/region names.
- Unit tests for path-based tenant extraction and the no-external-URL
backward-compat path.
Addresses review feedback on the IM storage URL rewrite:
- localFileService.GetFileURL now reads tenant ID from request context
first, falling back to ParseTenantIDFromStoragePath only when context
is absent. Fixes ambiguity for cloud providers whose paths embed
numeric bucket/region names before the tenant segment, which could
mint presigned URLs bound to the wrong tenant ID.
- Shorten presigned URL default TTL from 24h to 2h. A leaked HMAC key
authorizes cross-tenant file reads, so URLs should expire quickly;
IM clients fetch referenced images within seconds anyway.
- Document ParseTenantIDFromStoragePath as a best-effort fallback.
- Add unit tests covering context-first, path-fallback, and the
no-external-URL backward-compat path.
IM platforms (Feishu, Slack, Telegram, DingTalk, Mattermost, WeCom) cannot
render provider:// URLs (local://, minio://, s3://, etc.) that appear in
LLM answers containing knowledge base images. The web frontend handles
these via the authenticated /files endpoint, but IM clients need publicly
resolvable HTTP URLs.
Changes:
- Add HMAC-SHA256 presigned URL utility (internal/utils/presign.go) for
generating time-limited, signature-verified file access URLs
- Add GET /api/v1/files/presigned endpoint that serves files without
session auth, verified by HMAC signature and expiry
- Update localFileService.GetFileURL() to return presigned HTTP URLs
when APP_EXTERNAL_URL is configured (cloud backends already return
presigned URLs via their SDKs)
- Add IM content rewriting pipeline: strip <image> XML tags, strip
citation tags, rewrite storage URLs to HTTP — applied at all IM
output points (streaming flush, non-streaming reply, fallback)
- Add holdback buffer in streaming flush to prevent URL/tag splitting
across chunk boundaries
Closes#1058
Introduce per-model `custom_headers` config (similar to OpenAI Python SDK's
`extra_headers`) so users can inject gateway auth tokens, trace IDs, etc.
into every outbound API request for chat / embedding / rerank / VLLM / ASR
models. Reserved headers like Authorization / Content-Type are always
preserved to avoid breaking auth or signing flows.
Along the way, unify production and "test connection" paths onto a single
`ConfigFromModel(*types.Model, appID, appSecret)` constructor in each
model package. Both `service.modelService.GetXxxModel` and the four
`handler.initialization.Check*Model` / `TestEmbeddingModel` endpoints now
go through the same field mapping, so new parameters only need to be
added in one place.
Backend:
- Add `CustomHeaders map[string]string` to `types.ModelParameters` with
JSON/YAML `omitempty` tags for forward compatibility.
- New `internal/utils/extraheaders.go` providing `ApplyCustomHeaders`
(for hand-rolled HTTP paths) and `WrapHTTPClientWithHeaders` /
`CustomHeadersRoundTripper` (for SDK paths like go-openai). Reserved
headers (Authorization, api-key, Content-Type, Accept, Host,
Content-Length, User-Agent) are filtered out.
- Inject headers in every remote model path: chat (SDK + raw HTTP),
embedding (openai/aliyun/jina/volcengine/nvidia/azure_openai), rerank
(remote_api/aliyun/jina/nvidia/zhipu — via shared `customHeaderSetter`
interface), VLM and ASR (via http.Client wrapping).
- Add `ConfigFromModel` to each of chat/embedding/rerank/vlm/asr,
consolidating field mapping (ExtraConfig, CustomHeaders, InterfaceType
defaulting, WeKnoraCloud credentials, etc.) and cover it with
dedicated unit tests per package.
- Refactor `service/model.go`: replace ~100 lines of hand-written Config
literals with one-line `ConfigFromModel` calls; drop the now-unused
`stringMapToAnyMap` helper.
- Refactor `handler/initialization.go` test-connection endpoints: merge
four ad-hoc request structs into one `ModelTestRequest`, extract
`buildTestModel` and `resolveTenantWeKnoraCloudCreds` helpers, and
route all four endpoints through `ConfigFromModel` + `NewXxx` so the
test path is now behaviorally identical to production.
Frontend:
- Add `custom_headers` to the `ModelConfig` API type.
- `ModelEditorDialog.vue`: new Key-Value header editor (add/remove rows),
auto-converts map <-> array when loading/saving, and passes the
current header set to the `Test Connection` button so the preview
reflects exactly what production will send.
- `ModelSettings.vue`: serialize the header array back into a map for
the backend (drops empty rows).
- i18n: add custom-header labels / descriptions / placeholders to
zh-CN, en-US, ru-RU, ko-KR.
Tests:
- `internal/utils/extraheaders_test.go` covers reserved-header filtering
and round-tripper wrapping (including nil-client fallback).
- New `config_from_model_test.go` in chat / embedding / rerank / vlm /
asr verifies all fields (CustomHeaders, ExtraConfig, InterfaceType
defaulting, AppID/AppSecret) are propagated end-to-end.
- Enhanced the logging of the tenant access token in the Feishu client to handle variable lengths for prefix and suffix, ensuring accurate display even for shorter tokens.
- Updated the test cases in `connector_test.go` to reflect changes in the endpoint path for file downloads, improving test reliability.
- Adjusted example outputs in `example_test.go` for clarity and consistency.
- Modified image resolution tests to correct expected outcomes based on updated criteria for icon images.
- Added environment variable settings in image resolver tests to ensure localhost accessibility during testing.
These changes enhance the robustness of logging and testing mechanisms across the application.
- Added isValidURL function to validate URLs before rendering links in the document content component, preventing invalid links from being clickable.
- Refactored timeout handling in KnowledgeBase.vue to use setTimeout instead of setInterval, improving performance and resource management.
- Ensured that timeouts are cleared appropriately to avoid memory leaks and unnecessary polling.
This update improves the robustness of URL handling and optimizes the component's performance.
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
- Updated DatabaseQueryTool to accept search targets during initialization.
- Implemented search scope filtering in SQL validation to restrict queries to specified knowledge bases and documents.
- Enhanced the SQL validation process to include new filtering options for improved security and query accuracy.
- Replaced the default DialContext with SSRFSafeDialContext in the raw HTTP client to improve security against DNS rebinding attacks.
- Added SSRF validation for BaseURL and endpoint in NewRemoteAPIChat and chat methods, ensuring safer URL handling.
- Updated security utility functions to provide consistent SSRF checks across the application.
- Removed the query display section from the DatabaseQuery component to streamline the UI.
- Updated the DatabaseQueryData interface to eliminate the query field, reflecting changes in tool result handling.
- Enhanced the DatabaseQuery tool execution to focus on returning structured results without exposing raw SQL queries.
- Added a new utility to strip <think> blocks from LLM outputs, improving content clarity and user experience.
When a tenant setting is updated via PUT /tenants/kv/{key} (e.g.,
retrieval-config, parser-engine-config), the GORM AfterFind-decrypted
plaintext api_key was silently written back to the database because
db.Updates() does not trigger the BeforeSave hook.
Guard the repository's UpdateTenant to skip writing plaintext api_key
back to the database. Pre-encrypted values (enc:v1:… from CreateTenant
/ UpdateAPIKey) are written as-is; AfterFind-decrypted plaintext is
blanked so GORM skips the column entirely.
Closes#798
- Modified the session title generation template to focus on extracting only the intent of the user's question.
- Enhanced the session service to include language context when generating titles.
- Updated middleware to inject language into the request context.
- Added a new test suite for validating SSRF safety of URLs.
- Added timezone and language settings to the environment configuration.
- Introduced built-in agent configurations with multilingual support for various agents.
- Updated Docker Compose to utilize new environment variables for timezone and language.
- Created new prompt templates for question generation, summary generation, and keywords extraction.
- Added functionality for image uploads in chat, allowing users to attach images for multimodal Q&A.
- Enhanced the input field to handle image selection via drag-and-drop and paste, with validation for file types and sizes.
- Updated the backend to process images alongside text queries, including support for image analysis.
- Introduced new UI components for image previews and management, improving user interaction with uploaded content.
- Added localization strings for image upload features and error messages, enhancing accessibility for users in multiple languages.
These changes significantly improve the chat experience by enabling users to incorporate images into their queries, facilitating richer interactions and responses.
- Add crypto utility (internal/utils/crypto.go) with AES-256-GCM encrypt/decrypt
using SYSTEM_AES_KEY env var, with "enc:v1:" prefix for versioned ciphertext
- Encrypt tenant API key via GORM BeforeSave/AfterFind hooks and manual
encryption in CreateTenant/UpdateAPIKey (db.Updates bypasses hooks)
- Encrypt model API key in ModelParameters Value/Scan (driver.Valuer)
- Widen api_key column from varchar(64) to varchar(256) across all DB dialects
(MySQL, ParadeDB, SQLite) and add versioned migration 000018
- Propagate SYSTEM_AES_KEY through docker-compose, Helm secrets and values
- Fix migration 000017 PL/pgSQL dollar-quoting syntax ($ -> $$)
- Added automatic filtering for `deleted_at IS NULL` in database queries to ensure only active records are returned.
- Updated the `database_query.go` documentation to reflect the new soft-delete feature and its implications for query construction.
- Enhanced the SQL validation process to include soft-delete conditions, improving data integrity and security.
These changes enhance the robustness of database interactions by enforcing soft-delete logic across relevant queries.
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.
These changes improve the clarity and efficiency of image processing and API interactions within the application.
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.
This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.
This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
- Introduced a new validatedParams struct to hold validated input along with DNS-pinned host/IP for SSRF protection.
- Updated the validateParams function to validate and resolve the host to a single public IP, ensuring safe outbound fetches.
- Modified executeFetch and fetchHTMLContent methods to utilize the new validatedParams for improved security and clarity.
- Enhanced logging to reflect the display URL and provide better context during fetch operations.
This update strengthens the web fetch tool's security against SSRF attacks and improves the overall robustness of URL handling.
Add SSRF protection by validating redirect targets and resolved IPs,
blocking internal hostnames, restricted suffixes, and private IP ranges
including Docker and Kubernetes internal endpoints.
wizardchen