Extract NewSSRFSafeTransport and NewSSRFSafeHTTPClientWithTransport so
embedding and rerank can reuse one connection pool without allocating a
throwaway transport per client. SSRF dial and redirect policies are
unchanged; each client still keeps its own timeout.
Introduce resource://<handle> as the stable application identity for stored
objects, decoupling API responses, persisted rich text, and LLM prompts from
physical storage paths. Adds a resource catalog service (register/resolve/bind,
revocable expiring access grants), a request-local alias registry that compacts
resource references while in the LLM context and restores them (including across
streaming chunks), plus router, IM, chat-pipeline, and multimodal integration.
Includes schema migrations (resources, resource_bindings, resource_access_grants)
for sqlite, and hardens URL/output security handling.
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.
Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
key so the 3-tier resolver falls back to ENV / built-in default.
Idempotent (resetting a never-persisted key returns nil); emits an
audit row only on real deletions, invalidates the local cache, and
publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
storage_quota in one statement. Powers POST /system/admin/tenants/
apply-default-storage-quota; bypasses the per-tenant whitelist on
PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
emitAdminAudit() in SystemHandler — promote / revoke now leave an
audit trail with TenantID=0 and {target_email,target_username,
idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
(username, email fallback) so the UI shows "wizardchen" instead of
a UUID prefix without storing a denormalised column.
Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
auto-persisting controls (switch/select @change, input/inputnumber
@blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
roster). auth.registration_mode change goes through an inline
popconfirm; cancel rolls back. Reset / bulk-apply also inline
popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
admins are now compatibility redirects to /platform/settings?section=
system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
so the wider tables (members, system settings) breathe; <1100px
viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
empty / badges / priority hint / per-key labels / reset / bulkApply /
admins (label, placeholder, save messages, popconfirm copy) across
zh-CN, en-US, ko-KR, ru-RU.
Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
deploy-time-only knob (nginx + docreader + frontend each cache the
env at startup); a SystemAdmin UI override would mislead operators
because nginx would still 413. Until all four layers can hot-reload
the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
off SystemSettingService for live updates; ENV remains the fallback
for never-overridden deployments.
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.
Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.
What lands in PR 2a
-------------------
Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):
- `repository.go` — Repository struct + NewRepository constructor
that validates cluster reachability + OS version (2.4+ / 3.x;
primary tested 3.3.2) + k-NN plugin presence on every cluster
node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
dimension index creation, with transient errors not cached so a
momentary cluster blip does not permanently poison a dim.
sanitizeIndexName enforces a strict OS-compatible name spec.
probeVersion uses robust strings.Split/Atoi parsing for
pre-release suffixes and missing-patch versions. EngineType
returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
(MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
only cipher list) and transport tuning for the driver. Caller
exists only in PR 3 (container.go + engine_factory.go); PR 2a
remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
knn_vector + HNSW + content-analyzer mapping with every *_id
field as an explicit keyword and source_type as integer.
buildKeywordsMapping ships the dim-less keyword-only index
mapping used by the no-embedding save path. createIndexAndAlias
creates <alias>_v1 and aliases <alias> to it, with best-effort
orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
defaults (hnsw_m=16, ef_construction=100, ef_search=100,
shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
ErrDimensionMismatch, ErrAuth, ErrTransport,
ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
apperrors; PR 3's engine factory wraps these to typed AppError
2200/2201.
- `stubs.go` — every behavioural method returns
ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
fails-closed for non-empty KBs.
Tests (~1115 LoC, 50 cases):
- `repository_test.go` — interface satisfaction, sentinel mapping,
sanitizeIndexName positive/negative matrix, semver parsing
(pre-release / missing-patch), buildIndexMapping JSON shape pin
(Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
multi-node coverage, ensureReady concurrency + per-dim isolation
+ transient retry, NewRepository storeID validation, all 11
stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
wrapTransport sentinel mapping + leak guard, isNotFound /
isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
TLS 1.2 pinning / cipher list / transport tuning.
Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.
SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------
PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):
- AliasExists method passes dataPointer=nil to its internal do(),
which means non-2xx responses come back as a plain
*errors.errorString ("status: 404 Not Found") rather than as
*opensearch.StructError. aliasExists therefore inspects
resp.StatusCode directly (resp is returned even when err is
non-nil) and only falls back to wrapTransport for the "no
response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
index uses a mutex + ready/err flag pattern so transient failures
can be retried by the next caller. The per-dimension path uses
the `once map[int]*sync.Once` delete-and-recreate trick.
Test fixes folded in
--------------------
While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:
(1) Follow-up to #1445 — fanout test missed the new normalizer policy
(internal/application/service/knowledgebase_search_fanout_test.go,
+46 / -6). #1445 changed EngineAwareNormalizer for ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
passthrough (those engines surface non-negative cosine to the
normalizer per Lucene script_score non-negative invariant for
ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
OpenSearch, engine-internal or IR-normalized conversions for
the rest). Milvus is now the only engine that still surfaces
raw signed cosine in [-1, 1].
TestRetrieveFromStores_MixedEngine_Normalizes still asserted
the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
which under passthrough now becomes clamp01(-0.4) = 0. The
normalizer's own _test.go was updated at #1445 time, but this
fan-out integration test was not.
Fix: rewrite the godoc to spell out the two engine groups;
restate sub-case 2 as ES passthrough on a production-possible
mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
pinning the cosine-shift branch via Milvus -0.4 → 0.3.
(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
(internal/utils/security.go + internal/utils/security_test.go +
internal/infrastructure/web_search/searxng_test.go,
+33 / -9). loadSSRFWhitelist in internal/utils/security.go is
cached via sync.Once on first call. The internal reset helper
resetSSRFWhitelistForTest was unexported, so tests in other
packages could not reset and saw whatever whitelist was cached
by the first sync.Once.Do() in the same test binary. In
internal/infrastructure/web_search/, TestValidateProxyURL runs
before TestValidateSearxngBaseURL alphabetically and exercises
ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
whitelist; the later setenv in searxng_test then has no effect
and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
Pre-existing on main; surfaced now because this PR was the
first to do a full `go test ./...` run on top of #1445.
Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
ForTest suffix is the test-only contract); update in-package
callers; in web_search/searxng_test.go import internal/utils
and call ResetSSRFWhitelistForTest around the env mutation in
both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
No production code path changes.
Roadmap
-------
- PR 2b (next, depends on this PR) — read/write implementations:
query.go + retrieve.go + crud.go land their real bodies; stubs
for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
removed; corresponding CRUD/retrieve/filter test cases (~430
LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
BatchUpdate*, large-batch async deletes) + i18n + docker-compose
dev profile. After PR 3 merges, the OpenSearch driver becomes
reachable via either `engine_type=opensearch` VectorStore or
`RETRIEVE_DRIVER=opensearch` env.
Backward compatibility
----------------------
- New package — additive only. No existing file modified except
go.mod / go.sum, the two test files in (1)/(2), and the
test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
here (no driver instance produces a result yet).
Test plan
---------
- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
shows only PR 1's normalizer case + this driver's EngineType()
and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
Test-connection's "连接失败" was nearly useless for debugging because
two layers conspired to swallow the actual upstream error:
Backend swallow
---------------
checkChatModelConnection and CheckASRModel pattern-matched on the
upstream error string (401 / 404 / timeout / ...) and replaced it with
a fixed Chinese hint. The original SDK message (request URL, response
body, provider's error code description) was discarded.
Refactor: extract classifyConnectionError(errMsg) — returns just the
short Chinese hint. Callers combine "<hint>：<raw err>" so the hint
indicates "where to start looking" and the raw error shows "what
actually happened". The status-400 special-case (endpoint reachable,
parameter mismatch only) still returns success.
Frontend swallow
----------------
ModelEditorDialog.testConnection ignored result.message on success
paths (fine), but also overwrote it with a generic i18n string on
failure paths and 4xx rejections — even though axios's interceptor
already lifts the backend's error.message to error.message. Failure
branch now uses error?.message || result.message as the toast content;
i18n fallback only when neither is present.
SSRF allowlist hint
-------------------
SSRF validation failures previously returned
"Base URL 未通过安全校验: <reason>" with no mention of how to recover
(the allowlist is server-side, not configurable in the UI). Operators
hit it and were stuck.
New utils.FormatSSRFError(label, rawURL, err) wraps the error with
guidance to add the host to SSRF_WHITELIST_EXTRA (the operator-facing
append-only escape hatch — SSRF_WHITELIST itself is the project's
baseline). The hint echoes the failed host so it can be copy-pasted
straight into the env var:
Base URL 未通过安全校验：DNS resolution failed for hostname
test.local.com ... 如该地址确实可信，请联系运维在服务端环境变量
SSRF_WHITELIST_EXTRA 中加入该主机（支持精确域名 / *.example.com
通配 / IP / CIDR），示例：SSRF_WHITELIST_EXTRA=test.local.com,
*.example.com,10.0.0.0/8
All SSRF call sites switch to the helper:
initialization (CheckRemoteModel / TestEmbedding / CheckRerank /
CheckASR / TestMultimodal / batch validation), model CRUD,
mcp_service CRUD, knowledge URL ingest, system ReconnectDocReader.
- Added isValidURL function to validate URLs before rendering links in the document content component, preventing invalid links from being clickable.
- Refactored timeout handling in KnowledgeBase.vue to use setTimeout instead of setInterval, improving performance and resource management.
- Ensured that timeouts are cleared appropriately to avoid memory leaks and unnecessary polling.
This update improves the robustness of URL handling and optimizes the component's performance.
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
- Replaced the default DialContext with SSRFSafeDialContext in the raw HTTP client to improve security against DNS rebinding attacks.
- Added SSRF validation for BaseURL and endpoint in NewRemoteAPIChat and chat methods, ensuring safer URL handling.
- Updated security utility functions to provide consistent SSRF checks across the application.
- Added timezone and language settings to the environment configuration.
- Introduced built-in agent configurations with multilingual support for various agents.
- Updated Docker Compose to utilize new environment variables for timezone and language.
- Created new prompt templates for question generation, summary generation, and keywords extraction.
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.
These changes improve the clarity and efficiency of image processing and API interactions within the application.
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.
This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
- Introduced a new validatedParams struct to hold validated input along with DNS-pinned host/IP for SSRF protection.
- Updated the validateParams function to validate and resolve the host to a single public IP, ensuring safe outbound fetches.
- Modified executeFetch and fetchHTMLContent methods to utilize the new validatedParams for improved security and clarity.
- Enhanced logging to reflect the display URL and provide better context during fetch operations.
This update strengthens the web fetch tool's security against SSRF attacks and improves the overall robustness of URL handling.
Add SSRF protection by validating redirect targets and resolved IPs,
blocking internal hostnames, restricted suffixes, and private IP ranges
including Docker and Kubernetes internal endpoints.
wizardchen