Files
Enterprise-Knowledge-Base/internal/utils/security.go
T
ochan.kwon 11c3236e52 feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.

Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.

What lands in PR 2a
-------------------

Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):

- `repository.go` — Repository struct + NewRepository constructor
  that validates cluster reachability + OS version (2.4+ / 3.x;
  primary tested 3.3.2) + k-NN plugin presence on every cluster
  node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
  dimension index creation, with transient errors not cached so a
  momentary cluster blip does not permanently poison a dim.
  sanitizeIndexName enforces a strict OS-compatible name spec.
  probeVersion uses robust strings.Split/Atoi parsing for
  pre-release suffixes and missing-patch versions. EngineType
  returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
  (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
  only cipher list) and transport tuning for the driver. Caller
  exists only in PR 3 (container.go + engine_factory.go); PR 2a
  remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
  knn_vector + HNSW + content-analyzer mapping with every *_id
  field as an explicit keyword and source_type as integer.
  buildKeywordsMapping ships the dim-less keyword-only index
  mapping used by the no-embedding save path. createIndexAndAlias
  creates <alias>_v1 and aliases <alias> to it, with best-effort
  orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
  defaults (hnsw_m=16, ef_construction=100, ef_search=100,
  shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
  ErrDimensionMismatch, ErrAuth, ErrTransport,
  ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
  ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
  apperrors; PR 3's engine factory wraps these to typed AppError
  2200/2201.
- `stubs.go` — every behavioural method returns
  ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
  HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
  fails-closed for non-empty KBs.

Tests (~1115 LoC, 50 cases):

- `repository_test.go` — interface satisfaction, sentinel mapping,
  sanitizeIndexName positive/negative matrix, semver parsing
  (pre-release / missing-patch), buildIndexMapping JSON shape pin
  (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
  2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
  multi-node coverage, ensureReady concurrency + per-dim isolation
  + transient retry, NewRepository storeID validation, all 11
  stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
  SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
  wrapTransport sentinel mapping + leak guard, isNotFound /
  isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
  TLS 1.2 pinning / cipher list / transport tuning.

Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.

SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------

PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):

- AliasExists method passes dataPointer=nil to its internal do(),
  which means non-2xx responses come back as a plain
  *errors.errorString ("status: 404 Not Found") rather than as
  *opensearch.StructError. aliasExists therefore inspects
  resp.StatusCode directly (resp is returned even when err is
  non-nil) and only falls back to wrapTransport for the "no
  response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
  index uses a mutex + ready/err flag pattern so transient failures
  can be retried by the next caller. The per-dimension path uses
  the `once map[int]*sync.Once` delete-and-recreate trick.

Test fixes folded in
--------------------

While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:

(1) Follow-up to #1445 — fanout test missed the new normalizer policy
    (internal/application/service/knowledgebase_search_fanout_test.go,
    +46 / -6). #1445 changed EngineAwareNormalizer for ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
    Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
    passthrough (those engines surface non-negative cosine to the
    normalizer per Lucene script_score non-negative invariant for
    ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
    OpenSearch, engine-internal or IR-normalized conversions for
    the rest). Milvus is now the only engine that still surfaces
    raw signed cosine in [-1, 1].

    TestRetrieveFromStores_MixedEngine_Normalizes still asserted
    the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
    which under passthrough now becomes clamp01(-0.4) = 0. The
    normalizer's own _test.go was updated at #1445 time, but this
    fan-out integration test was not.

    Fix: rewrite the godoc to spell out the two engine groups;
    restate sub-case 2 as ES passthrough on a production-possible
    mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
    pinning the cosine-shift branch via Milvus -0.4 → 0.3.

(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
    (internal/utils/security.go + internal/utils/security_test.go +
    internal/infrastructure/web_search/searxng_test.go,
    +33 / -9). loadSSRFWhitelist in internal/utils/security.go is
    cached via sync.Once on first call. The internal reset helper
    resetSSRFWhitelistForTest was unexported, so tests in other
    packages could not reset and saw whatever whitelist was cached
    by the first sync.Once.Do() in the same test binary. In
    internal/infrastructure/web_search/, TestValidateProxyURL runs
    before TestValidateSearxngBaseURL alphabetically and exercises
    ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
    whitelist; the later setenv in searxng_test then has no effect
    and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
    Pre-existing on main; surfaced now because this PR was the
    first to do a full `go test ./...` run on top of #1445.

    Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
    ForTest suffix is the test-only contract); update in-package
    callers; in web_search/searxng_test.go import internal/utils
    and call ResetSSRFWhitelistForTest around the env mutation in
    both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
    No production code path changes.

Roadmap
-------

- PR 2b (next, depends on this PR) — read/write implementations:
  query.go + retrieve.go + crud.go land their real bodies; stubs
  for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
  removed; corresponding CRUD/retrieve/filter test cases (~430
  LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
  BatchUpdate*, large-batch async deletes) + i18n + docker-compose
  dev profile. After PR 3 merges, the OpenSearch driver becomes
  reachable via either `engine_type=opensearch` VectorStore or
  `RETRIEVE_DRIVER=opensearch` env.

Backward compatibility
----------------------

- New package — additive only. No existing file modified except
  go.mod / go.sum, the two test files in (1)/(2), and the
  test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
  here (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-26 20:54:58 +08:00

1088 lines
34 KiB
Go

package utils
import (
"context"
"fmt"
"html"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"sync"
"time"
"unicode/utf8"
"golang.org/x/net/http/httpproxy"
)
// XSS 防护相关正则表达式
var (
// 匹配潜在的 XSS 攻击模式
xssPatterns = []*regexp.Regexp{
regexp.MustCompile(`(?i)<script[^>]*>.*?</script>`),
regexp.MustCompile(`(?i)<iframe[^>]*>.*?</iframe>`),
regexp.MustCompile(`(?i)<object[^>]*>.*?</object>`),
regexp.MustCompile(`(?i)<embed[^>]*>.*?</embed>`),
regexp.MustCompile(`(?i)<embed[^>]*>`),
regexp.MustCompile(`(?i)<form[^>]*>.*?</form>`),
regexp.MustCompile(`(?i)<input[^>]*>`),
regexp.MustCompile(`(?i)<button[^>]*>.*?</button>`),
regexp.MustCompile(`(?i)javascript:`),
regexp.MustCompile(`(?i)vbscript:`),
regexp.MustCompile(`(?i)onload\s*=`),
regexp.MustCompile(`(?i)onerror\s*=`),
regexp.MustCompile(`(?i)onclick\s*=`),
regexp.MustCompile(`(?i)onmouseover\s*=`),
regexp.MustCompile(`(?i)onfocus\s*=`),
regexp.MustCompile(`(?i)onblur\s*=`),
}
)
// SanitizeHTML 清理 HTML 内容,防止 XSS 攻击
func SanitizeHTML(input string) string {
if input == "" {
return ""
}
// 检查输入长度
if len(input) > 10000 {
input = input[:10000]
}
// 检查是否包含潜在的 XSS 攻击
for _, pattern := range xssPatterns {
if pattern.MatchString(input) {
// 如果包含恶意内容,进行 HTML 转义
return html.EscapeString(input)
}
}
// 如果内容相对安全,返回原内容
return input
}
// EscapeHTML 转义 HTML 特殊字符
func EscapeHTML(input string) string {
if input == "" {
return ""
}
return html.EscapeString(input)
}
// ValidateInput 验证用户输入
func ValidateInput(input string) (string, bool) {
if input == "" {
return "", true
}
// 检查是否包含控制字符
for _, r := range input {
if r < 32 && r != 9 && r != 10 && r != 13 {
return "", false
}
}
// 检查 UTF-8 有效性
if !utf8.ValidString(input) {
return "", false
}
// 检查是否包含潜在的 XSS 攻击
for _, pattern := range xssPatterns {
if pattern.MatchString(input) {
return "", false
}
}
return strings.TrimSpace(input), true
}
// SafePathUnderBase 校验 filePath 是否落在 baseDir 下,防止路径遍历(如 ../../)。
// 返回规范化的绝对路径;若路径逃逸出 baseDir 则返回错误。
func SafePathUnderBase(baseDir, filePath string) (string, error) {
if baseDir == "" || filePath == "" {
return "", fmt.Errorf("baseDir and filePath cannot be empty")
}
absBase, err := filepath.Abs(filepath.Clean(baseDir))
if err != nil {
return "", fmt.Errorf("invalid base dir: %w", err)
}
absPath, err := filepath.Abs(filepath.Clean(filePath))
if err != nil {
return "", fmt.Errorf("invalid file path: %w", err)
}
sep := string(filepath.Separator)
if absPath != absBase && !strings.HasPrefix(absPath, absBase+sep) {
return "", fmt.Errorf("path traversal denied: path is outside base directory")
}
return absPath, nil
}
// SafeFileName 校验并返回安全的“仅文件名”部分,防止路径遍历。
// 仅保留最后一个路径成分,禁止 ".."、空名或仅含点,用于 SaveBytes 等场景。
func SafeFileName(fileName string) (string, error) {
if fileName == "" {
return "", fmt.Errorf("fileName cannot be empty")
}
base := filepath.Base(filepath.Clean(fileName))
if base == "" || base == "." || base == ".." {
return "", fmt.Errorf("invalid fileName: path traversal or empty name")
}
if strings.Contains(base, "..") {
return "", fmt.Errorf("invalid fileName: contains path traversal")
}
if len(base) > 255 {
return "", fmt.Errorf("fileName too long")
}
return base, nil
}
// SafeObjectKey 校验对象存储的 key(如 COS/MinIO objectName),禁止包含 ".." 等路径遍历
func SafeObjectKey(objectKey string) error {
if objectKey == "" {
return fmt.Errorf("object key cannot be empty")
}
if strings.Contains(objectKey, "..") {
return fmt.Errorf("object key contains path traversal")
}
return nil
}
// IsValidURL 验证 URL 是否安全
func IsValidURL(url string) bool {
if url == "" {
return false
}
// 检查长度
if len(url) > 2048 {
return false
}
// 检查协议, 只允许 http, https, local, minio, cos, tos, oss 协议
allowedProtocols := []string{"http://", "https://", "local://", "minio://", "cos://", "tos://", "oss://"}
isAllowed := false
for _, protocol := range allowedProtocols {
if strings.HasPrefix(strings.ToLower(url), protocol) {
isAllowed = true
break
}
}
if !isAllowed {
return false
}
// 检查是否包含恶意内容
for _, pattern := range xssPatterns {
if pattern.MatchString(url) {
return false
}
}
return true
}
// restrictedHostnames contains hostnames that are blocked for SSRF prevention
var restrictedHostnames = []string{
"localhost",
"127.0.0.1",
"::1",
"0.0.0.0",
"metadata.google.internal",
"metadata.tencentyun.com",
"metadata.aws.internal",
// Docker-specific internal hostnames
"host.docker.internal",
"gateway.docker.internal",
"kubernetes.docker.internal",
// Kubernetes internal hostnames
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster.local",
}
// restrictedHostSuffixes contains hostname suffixes that are blocked
var restrictedHostSuffixes = []string{
".local",
".localhost",
".internal",
".corp",
".lan",
".home",
".localdomain",
// Kubernetes internal suffixes
".svc.cluster.local",
".pod.cluster.local",
}
// restrictedIPv4Ranges contains CIDR ranges that should be blocked
// These are additional ranges not covered by Go's IsPrivate(), IsLoopback(), etc.
var restrictedIPv4Ranges = []*net.IPNet{
// 100.64.0.0/10 - Carrier-grade NAT (RFC 6598)
mustParseCIDR("100.64.0.0/10"),
// 198.18.0.0/15 - Network device benchmark testing (RFC 2544)
mustParseCIDR("198.18.0.0/15"),
// 198.51.100.0/24 - TEST-NET-2 for documentation (RFC 5737)
mustParseCIDR("198.51.100.0/24"),
// 203.0.113.0/24 - TEST-NET-3 for documentation (RFC 5737)
mustParseCIDR("203.0.113.0/24"),
// 192.0.0.0/24 - IETF Protocol Assignments (RFC 6890)
mustParseCIDR("192.0.0.0/24"),
// 192.0.2.0/24 - TEST-NET-1 for documentation (RFC 5737)
mustParseCIDR("192.0.2.0/24"),
// 0.0.0.0/8 - "This" network (RFC 1122)
mustParseCIDR("0.0.0.0/8"),
// 240.0.0.0/4 - Reserved for future use (RFC 1112)
mustParseCIDR("240.0.0.0/4"),
// 255.255.255.255/32 - Limited broadcast
mustParseCIDR("255.255.255.255/32"),
// Docker bridge network (default range)
mustParseCIDR("172.17.0.0/16"),
// Docker user-defined bridge networks (commonly used range)
mustParseCIDR("172.18.0.0/16"),
mustParseCIDR("172.19.0.0/16"),
mustParseCIDR("172.20.0.0/16"),
}
// mustParseCIDR parses a CIDR string and panics on error
func mustParseCIDR(s string) *net.IPNet {
_, ipNet, err := net.ParseCIDR(s)
if err != nil {
panic(fmt.Sprintf("invalid CIDR: %s", s))
}
return ipNet
}
// isRestrictedIP checks if an IP address falls within any restricted range
func isRestrictedIP(ip net.IP) (bool, string) {
// Check Go's built-in methods first
if ip.IsPrivate() {
return true, "private IP address"
}
if ip.IsLoopback() {
return true, "loopback address"
}
if ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() {
return true, "link-local address"
}
if ip.IsMulticast() {
return true, "multicast address"
}
if ip.IsUnspecified() {
return true, "unspecified address"
}
// Check IPv4-specific restricted ranges
if ip4 := ip.To4(); ip4 != nil {
for _, cidr := range restrictedIPv4Ranges {
if cidr.Contains(ip4) {
return true, fmt.Sprintf("restricted range %s", cidr.String())
}
}
}
// Check IPv6-specific restrictions
if ip.To4() == nil && len(ip) == 16 {
// Site-local (deprecated but still blocked): fec0::/10
if ip[0] == 0xfe && (ip[1]&0xc0) == 0xc0 {
return true, "site-local IPv6 address"
}
// Unique local address (ULA): fc00::/7 (already covered by IsPrivate for Go 1.17+)
if (ip[0] & 0xfe) == 0xfc {
return true, "unique local IPv6 address"
}
// IPv4-mapped IPv6 addresses: ::ffff:x.x.x.x
if isZeros(ip[0:10]) && ip[10] == 0xff && ip[11] == 0xff {
mappedIP := ip[12:16]
if restricted, reason := isRestrictedIP(net.IP(mappedIP)); restricted {
return true, fmt.Sprintf("IPv4-mapped %s", reason)
}
}
// Teredo tunneling addresses: 2001:0000::/32
// Embed arbitrary IPv4 in the payload; can reach internal hosts via relay.
if ip[0] == 0x20 && ip[1] == 0x01 && ip[2] == 0x00 && ip[3] == 0x00 {
return true, "Teredo tunneling address"
}
// 6to4 addresses: 2002::/16
// Bits 16-47 carry an IPv4 address; block when embedded IPv4 is restricted.
if ip[0] == 0x20 && ip[1] == 0x02 {
embeddedIP := net.IP(ip[2:6])
if restricted, reason := isRestrictedIP(embeddedIP); restricted {
return true, fmt.Sprintf("6to4 embedded %s", reason)
}
}
}
return false, ""
}
// IsPublicIP returns true if the IP is safe for outbound fetch (not private, loopback, link-local, etc.).
// Used for DNS pinning: after resolving a hostname we pick the first public IP and pin all requests to it.
func IsPublicIP(ip net.IP) bool {
restricted, _ := isRestrictedIP(ip)
return !restricted
}
// isZeros checks if a byte slice is all zeros
func isZeros(b []byte) bool {
for _, v := range b {
if v != 0 {
return false
}
}
return true
}
// ipLikePatterns contains regex patterns for detecting IP-like hostnames
// These catch various IP address obfuscation techniques
var ipLikePatterns = []*regexp.Regexp{
// Standard IPv4: 192.168.1.1
regexp.MustCompile(`^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$`),
// Decimal IP: 3232235777 (equivalent to 192.168.1.1)
regexp.MustCompile(`^\d{8,10}$`),
// Octal IP: 0300.0250.0001.0001 or 0177.0.0.1
regexp.MustCompile(`^0[0-7]+\.`),
// Hex IP: 0xC0.0xA8.0x01.0x01 or 0x7f.0.0.1
regexp.MustCompile(`(?i)^0x[0-9a-f]+\.`),
// Mixed formats with hex: 0xC0A80101
regexp.MustCompile(`(?i)^0x[0-9a-f]{6,8}$`),
// IPv6 patterns
regexp.MustCompile(`(?i)^[0-9a-f:]+::[0-9a-f:]*$`),
regexp.MustCompile(`(?i)^[0-9a-f]{1,4}(:[0-9a-f]{1,4}){7}$`),
// IPv4-mapped IPv6: ::ffff:192.168.1.1
regexp.MustCompile(`(?i)^::ffff:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$`),
// Bracketed IPv6: [::1]
regexp.MustCompile(`(?i)^\[[0-9a-f:]+\]$`),
}
// isIPLikeHostname checks if a hostname looks like an IP address in any format
// This catches obfuscation attempts like octal, hex, decimal, etc.
func isIPLikeHostname(hostname string) bool {
for _, pattern := range ipLikePatterns {
if pattern.MatchString(hostname) {
return true
}
}
return false
}
// isSSRFSafeURL validates a URL to prevent SSRF attacks
// It checks for:
// - Valid http/https protocol
// - Private IP addresses (10.x.x.x, 172.16-31.x.x, 192.168.x.x)
// - Loopback addresses (127.x.x.x, ::1)
// - Link-local addresses (169.254.x.x, fe80::)
// - Cloud metadata endpoints
// - Reserved hostnames (localhost, *.local, etc.)
func isSSRFSafeURL(rawURL string) (bool, string) {
if rawURL == "" {
return false, "URL is empty"
}
// Check URL length
if len(rawURL) > 2048 {
return false, "URL exceeds maximum length"
}
// Parse URL
parsed, err := url.Parse(rawURL)
if err != nil {
return false, fmt.Sprintf("invalid URL format: %v", err)
}
// Only allow http and https
scheme := strings.ToLower(parsed.Scheme)
if scheme != "http" && scheme != "https" {
return false, fmt.Sprintf("invalid scheme: %s (only http/https allowed)", scheme)
}
// Extract hostname
hostname := parsed.Hostname()
if hostname == "" {
return false, "URL has no hostname"
}
hostnameLower := strings.ToLower(hostname)
// Check against restricted hostnames
for _, restricted := range restrictedHostnames {
if hostnameLower == restricted {
return false, fmt.Sprintf("hostname %s is restricted", hostname)
}
}
// Check against restricted hostname suffixes
for _, suffix := range restrictedHostSuffixes {
if strings.HasSuffix(hostnameLower, suffix) {
return false, fmt.Sprintf("hostname suffix %s is restricted", suffix)
}
}
// STRICT MODE: Block all direct IP addresses in URLs (both IPv4 and IPv6).
// This prevents IP-based SSRF attacks including obfuscation, tunneling, and
// transition mechanism bypasses. Legitimate IPs should be whitelisted via
// SSRF_WHITELIST env var; the whitelist is checked by ValidateURLForSSRF
// before this function is called.
ip := net.ParseIP(hostname)
if ip != nil {
return false, "direct IP address access is not allowed, use domain name or add to SSRF_WHITELIST"
}
// Also check for IP addresses in various formats that ParseIP might not catch
// e.g., octal (0177.0.0.1), hex (0x7f.0.0.1), decimal (2130706433)
if isIPLikeHostname(hostname) {
return false, "IP-like hostname format is not allowed"
}
// Perform DNS resolution to check the resolved IP
// This prevents DNS rebinding attacks where a domain resolves to internal IPs
ips, err := net.LookupIP(hostname)
if err != nil {
return false, fmt.Sprintf("DNS resolution failed for hostname %s: cannot verify if it resolves to safe IP", hostname)
}
// Check if any resolved IP is restricted
for _, resolvedIP := range ips {
if restricted, reason := isRestrictedIP(resolvedIP); restricted {
return false, fmt.Sprintf("hostname %s resolves to restricted IP %s: %s", hostname, resolvedIP.String(), reason)
}
}
// Check for suspicious port numbers
port := parsed.Port()
if port != "" {
// Block common internal service ports
blockedPorts := map[string]bool{
"22": true, // SSH
"23": true, // Telnet
"25": true, // SMTP
"445": true, // SMB
"3389": true, // RDP
"5432": true, // PostgreSQL
"3306": true, // MySQL
"6379": true, // Redis
"27017": true, // MongoDB
"9200": true, // Elasticsearch
"2379": true, // etcd
"2380": true, // etcd
"8500": true, // Consul
"4001": true, // etcd (old)
}
if blockedPorts[port] {
return false, fmt.Sprintf("port %s is blocked for security reasons", port)
}
}
return true, ""
}
// IsValidImageURL 验证图片 URL 是否安全
func IsValidImageURL(url string) bool {
if !IsValidURL(url) {
return false
}
// 检查是否为图片文件
imageExtensions := []string{".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg", ".bmp", ".ico"}
lowerURL := strings.ToLower(url)
for _, ext := range imageExtensions {
if strings.Contains(lowerURL, ext) {
return true
}
}
return false
}
// CleanMarkdown 清理 Markdown 内容
func CleanMarkdown(input string) string {
if input == "" {
return ""
}
// 移除潜在的恶意脚本
cleaned := input
for _, pattern := range xssPatterns {
cleaned = pattern.ReplaceAllString(cleaned, "")
}
return cleaned
}
// SanitizeForDisplay 为显示清理内容
func SanitizeForDisplay(input string) string {
if input == "" {
return ""
}
// 首先清理 Markdown
cleaned := CleanMarkdown(input)
// 然后进行 HTML 转义
escaped := html.EscapeString(cleaned)
return escaped
}
// SanitizeForLog 清理日志输入,防止日志注入攻击
// 日志注入攻击是指攻击者通过在输入中插入换行符和其他控制字符,
// 伪造日志条目,可能导致日志分析工具误判或隐藏恶意活动
func SanitizeForLog(input string) string {
if input == "" {
return ""
}
// 替换换行符(LF, CR, CRLF)为空格,防止日志注入
sanitized := strings.ReplaceAll(input, "\n", " ")
sanitized = strings.ReplaceAll(sanitized, "\r", " ")
// 替换制表符为空格
sanitized = strings.ReplaceAll(sanitized, "\t", " ")
// 移除其他控制字符(ASCII 0-31,除了空格已处理的)
var builder strings.Builder
for _, r := range sanitized {
// 保留可打印字符和常用Unicode字符
if r >= 32 || r == ' ' {
builder.WriteRune(r)
}
}
sanitized = builder.String()
return sanitized
}
// SanitizeForLogArray 清理日志输入数组,防止日志注入攻击
func SanitizeForLogArray(input []string) []string {
if len(input) == 0 {
return []string{}
}
sanitized := make([]string, 0, len(input))
for _, item := range input {
sanitized = append(sanitized, SanitizeForLog(item))
}
return sanitized
}
// AllowedStdioCommands defines the whitelist of allowed commands for MCP stdio transport
// These are the standard MCP server launchers that are considered safe
var AllowedStdioCommands = map[string]bool{
"uvx": true, // Python package runner (uv)
"npx": true, // Node.js package runner
}
// DangerousArgPatterns contains patterns that indicate potentially dangerous arguments
var DangerousArgPatterns = []*regexp.Regexp{
regexp.MustCompile(`(?i)^-c$`), // Shell command execution flag
regexp.MustCompile(`(?i)^--command$`), // Shell command execution flag
regexp.MustCompile(`(?i)^-e$`), // Eval flag
regexp.MustCompile(`(?i)^--eval$`), // Eval flag
regexp.MustCompile(`(?i)[;&|]`), // Shell command chaining
regexp.MustCompile(`(?i)\$\(`), // Command substitution
regexp.MustCompile("(?i)`"), // Backtick command substitution
regexp.MustCompile(`(?i)>\s*[/~]`), // Output redirection to absolute/home path
regexp.MustCompile(`(?i)<\s*[/~]`), // Input redirection from absolute/home path
regexp.MustCompile(`(?i)^/bin/`), // Direct binary path
regexp.MustCompile(`(?i)^/usr/bin/`), // Direct binary path
regexp.MustCompile(`(?i)^/sbin/`), // Direct binary path
regexp.MustCompile(`(?i)^/usr/sbin/`), // Direct binary path
regexp.MustCompile(`(?i)^\.\./`), // Path traversal
regexp.MustCompile(`(?i)/\.\./`), // Path traversal in middle
regexp.MustCompile(`(?i)^(bash|sh|zsh|ksh|csh|tcsh|fish|dash)$`), // Shell interpreters as args
regexp.MustCompile(`(?i)^(curl|wget|nc|netcat|ncat)$`), // Network tools as args
regexp.MustCompile(`(?i)^(rm|dd|mkfs|fdisk)$`), // Destructive commands as args
}
// DangerousEnvVarPatterns contains patterns for dangerous environment variable names or values
var DangerousEnvVarPatterns = []*regexp.Regexp{
regexp.MustCompile(`(?i)^LD_PRELOAD$`), // Library injection
regexp.MustCompile(`(?i)^LD_LIBRARY_PATH$`), // Library path manipulation
regexp.MustCompile(`(?i)^DYLD_`), // macOS dynamic linker
regexp.MustCompile(`(?i)^PATH$`), // PATH manipulation
regexp.MustCompile(`(?i)^PYTHONPATH$`), // Python path manipulation
regexp.MustCompile(`(?i)^NODE_OPTIONS$`), // Node.js options injection
regexp.MustCompile(`(?i)^BASH_ENV$`), // Bash environment file
regexp.MustCompile(`(?i)^ENV$`), // Shell environment file
regexp.MustCompile(`(?i)^SHELL$`), // Shell override
}
// ValidateStdioCommand validates the command for MCP stdio transport
// Returns an error if the command is not in the whitelist or contains dangerous patterns
func ValidateStdioCommand(command string) error {
if command == "" {
return fmt.Errorf("command cannot be empty")
}
// Normalize command (extract base name if it's a path)
baseCommand := command
if strings.Contains(command, "/") {
parts := strings.Split(command, "/")
baseCommand = parts[len(parts)-1]
}
// Check against whitelist
if !AllowedStdioCommands[baseCommand] {
return fmt.Errorf("command '%s' is not in the allowed list. Allowed commands: uvx, npx, node, python, python3, deno, bun", baseCommand)
}
// Additional check: command should not contain path traversal
if strings.Contains(command, "..") {
return fmt.Errorf("command path contains invalid characters")
}
return nil
}
// ValidateStdioArgs validates the arguments for MCP stdio transport
// Returns an error if any argument contains dangerous patterns
func ValidateStdioArgs(args []string) error {
if len(args) == 0 {
return nil
}
for i, arg := range args {
// Check length
if len(arg) > 1024 {
return fmt.Errorf("argument %d exceeds maximum length (1024 characters)", i)
}
// Check against dangerous patterns
for _, pattern := range DangerousArgPatterns {
if pattern.MatchString(arg) {
return fmt.Errorf("argument %d contains potentially dangerous pattern: %s", i, SanitizeForLog(arg))
}
}
// Check for null bytes
if strings.Contains(arg, "\x00") {
return fmt.Errorf("argument %d contains null bytes", i)
}
}
return nil
}
// ValidateStdioEnvVars validates environment variables for MCP stdio transport
// Returns an error if any env var name or value is dangerous
func ValidateStdioEnvVars(envVars map[string]string) error {
if len(envVars) == 0 {
return nil
}
for key, value := range envVars {
// Check key against dangerous patterns
for _, pattern := range DangerousEnvVarPatterns {
if pattern.MatchString(key) {
return fmt.Errorf("environment variable '%s' is not allowed for security reasons", key)
}
}
// Check key length
if len(key) > 256 {
return fmt.Errorf("environment variable name '%s' exceeds maximum length", SanitizeForLog(key[:50]))
}
// Check value length
if len(value) > 4096 {
return fmt.Errorf("environment variable '%s' value exceeds maximum length", key)
}
// Check for null bytes in value
if strings.Contains(value, "\x00") {
return fmt.Errorf("environment variable '%s' value contains null bytes", key)
}
// Check value for shell injection patterns
for _, pattern := range DangerousArgPatterns {
if pattern.MatchString(value) {
return fmt.Errorf("environment variable '%s' value contains potentially dangerous pattern", key)
}
}
}
return nil
}
// ValidateStdioConfig performs comprehensive validation of stdio configuration
// This should be called before creating or executing any stdio-based MCP client
func ValidateStdioConfig(command string, args []string, envVars map[string]string) error {
// Validate command
if err := ValidateStdioCommand(command); err != nil {
return fmt.Errorf("invalid command: %w", err)
}
// Validate arguments
if err := ValidateStdioArgs(args); err != nil {
return fmt.Errorf("invalid arguments: %w", err)
}
// Validate environment variables
if err := ValidateStdioEnvVars(envVars); err != nil {
return fmt.Errorf("invalid environment variables: %w", err)
}
return nil
}
// SSRFSafeHTTPClientConfig contains configuration for the SSRF-safe HTTP client
type SSRFSafeHTTPClientConfig struct {
Timeout time.Duration
MaxRedirects int
DisableKeepAlives bool
DisableCompression bool
}
// DefaultSSRFSafeHTTPClientConfig returns the default configuration
func DefaultSSRFSafeHTTPClientConfig() SSRFSafeHTTPClientConfig {
return SSRFSafeHTTPClientConfig{
Timeout: 30 * time.Second,
MaxRedirects: 10,
DisableKeepAlives: false,
DisableCompression: false,
}
}
// ErrSSRFRedirectBlocked is returned when a redirect target is blocked due to SSRF protection
var ErrSSRFRedirectBlocked = fmt.Errorf("redirect blocked: target URL failed SSRF validation")
// NewSSRFSafeHTTPClient creates an HTTP client that validates redirect targets against SSRF protections.
// This prevents SSRF attacks via HTTP redirects where an attacker's server redirects to internal services.
func NewSSRFSafeHTTPClient(config SSRFSafeHTTPClientConfig) *http.Client {
transport := &http.Transport{
DisableKeepAlives: config.DisableKeepAlives,
DisableCompression: config.DisableCompression,
// Dial with SSRF protection - validates resolved IPs before connecting
DialContext: SSRFSafeDialContext,
}
return &http.Client{
Timeout: config.Timeout,
Transport: transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
// Check redirect count
if len(via) >= config.MaxRedirects {
return fmt.Errorf("stopped after %d redirects", config.MaxRedirects)
}
// Validate the redirect target URL for SSRF (whitelist-aware).
// Even whitelisted hosts must use http/https to prevent scheme-based attacks.
redirectScheme := strings.ToLower(req.URL.Scheme)
if redirectScheme != "http" && redirectScheme != "https" {
return fmt.Errorf("%w: invalid scheme %s", ErrSSRFRedirectBlocked, redirectScheme)
}
redirectHost := req.URL.Hostname()
if redirectHost != "" && IsSSRFWhitelisted(redirectHost) {
return nil
}
redirectURL := req.URL.String()
if safe, reason := isSSRFSafeURL(redirectURL); !safe {
return fmt.Errorf("%w: %s", ErrSSRFRedirectBlocked, reason)
}
return nil
},
}
}
// SSRFSafeDialContext is a custom dial function that validates the resolved IP addresses
// before establishing a connection. This provides an additional layer of SSRF protection
// against DNS rebinding attacks during the connection phase.
func SSRFSafeDialContext(ctx context.Context, network, addr string) (net.Conn, error) {
// Parse host and port
host, _, err := net.SplitHostPort(addr)
if err != nil {
return nil, fmt.Errorf("invalid address %s: %w", addr, err)
}
// Whitelisted hosts bypass all dial-time SSRF checks, consistent with
// ValidateURLForSSRF which skips isSSRFSafeURL for whitelisted hosts.
// NOTE: This intentionally relaxes DNS-rebinding protection for whitelisted
// hosts. Admins must ensure whitelisted domains are under their control.
if IsSystemProxy(addr) || IsSSRFWhitelisted(host) {
dialer := &net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
return dialer.DialContext(ctx, network, addr)
}
// Check if the host is a restricted hostname
hostLower := strings.ToLower(host)
for _, restricted := range restrictedHostnames {
if hostLower == restricted {
return nil, fmt.Errorf("connection blocked: hostname %s is restricted", host)
}
}
for _, suffix := range restrictedHostSuffixes {
if strings.HasSuffix(hostLower, suffix) {
return nil, fmt.Errorf("connection blocked: hostname suffix %s is restricted", suffix)
}
}
// Resolve the hostname to IP addresses
ips, err := net.DefaultResolver.LookupIPAddr(ctx, host)
if err != nil {
return nil, fmt.Errorf("DNS resolution failed for %s: %w", host, err)
}
// Validate all resolved IPs
for _, ipAddr := range ips {
if restricted, reason := isRestrictedIP(ipAddr.IP); restricted {
return nil, fmt.Errorf("connection blocked: %s resolves to restricted IP %s (%s)", host, ipAddr.IP.String(), reason)
}
}
// If we get here, all IPs are safe. Connect using the standard dialer.
// We dial the original address so that proper connection routing happens.
dialer := &net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
return dialer.DialContext(ctx, network, addr)
}
// ---------------------------------------------------------------------------
// SSRF Whitelist mechanism
// ---------------------------------------------------------------------------
//
// The environment variable SSRF_WHITELIST accepts a comma-separated list of
// allowed host patterns. Each entry can be:
// - An exact domain: "example.com"
// - A wildcard domain: "*.example.com" (matches all subdomains)
// - An IPv4 address: "203.0.113.5"
// - An IPv6 address: "2001:db8::1"
// - A CIDR range (v4 or v6): "10.0.0.0/8", "2001:db8::/32"
//
// Whitelisted entries bypass the normal SSRF checks performed by isSSRFSafeURL.
var (
ssrfWhitelistOnce sync.Once
ssrfWhitelist *ssrfWhitelistConfig
)
type ssrfWhitelistConfig struct {
exactHosts map[string]bool // lowercase exact hostnames / IPs
suffixHosts []string // suffix matches (from "*.example.com" → ".example.com")
cidrNets []*net.IPNet // CIDR ranges
}
// loadSSRFWhitelist parses the SSRF_WHITELIST environment variable once.
func loadSSRFWhitelist() *ssrfWhitelistConfig {
ssrfWhitelistOnce.Do(func() {
ssrfWhitelist = &ssrfWhitelistConfig{
exactHosts: make(map[string]bool),
}
raw := os.Getenv("SSRF_WHITELIST")
// SSRF_WHITELIST_EXTRA is merged in addition to SSRF_WHITELIST so that
// deployment-managed defaults (e.g. docker-compose injected sidecar host
// names like "searxng") aren't accidentally clobbered when an operator
// overrides SSRF_WHITELIST in their .env.
extra := os.Getenv("SSRF_WHITELIST_EXTRA")
if raw == "" && extra == "" {
return
}
if extra != "" {
if raw == "" {
raw = extra
} else {
raw = raw + "," + extra
}
}
for _, entry := range strings.Split(raw, ",") {
entry = strings.TrimSpace(entry)
if entry == "" {
continue
}
// CIDR range
if strings.Contains(entry, "/") {
_, ipNet, err := net.ParseCIDR(entry)
if err == nil {
ssrfWhitelist.cidrNets = append(ssrfWhitelist.cidrNets, ipNet)
continue
}
}
// Wildcard domain: *.example.com
if strings.HasPrefix(entry, "*.") {
suffix := strings.ToLower(entry[1:]) // ".example.com"
ssrfWhitelist.suffixHosts = append(ssrfWhitelist.suffixHosts, suffix)
continue
}
// Exact host or IP
ssrfWhitelist.exactHosts[strings.ToLower(entry)] = true
}
})
return ssrfWhitelist
}
// IsSSRFWhitelisted checks whether the given hostname (or IP string) is
// covered by the SSRF_WHITELIST environment variable.
func IsSSRFWhitelisted(hostname string) bool {
wl := loadSSRFWhitelist()
if wl == nil {
return false
}
lower := strings.ToLower(hostname)
// Exact match
if wl.exactHosts[lower] {
return true
}
// Suffix / wildcard match
for _, suffix := range wl.suffixHosts {
if strings.HasSuffix(lower, suffix) || lower == suffix[1:] {
return true
}
}
// CIDR match (only when hostname looks like an IP)
if ip := net.ParseIP(hostname); ip != nil {
for _, cidr := range wl.cidrNets {
if cidr.Contains(ip) {
return true
}
}
}
// Also resolve and check resolved IPs against CIDR whitelist
if net.ParseIP(hostname) == nil && len(wl.cidrNets) > 0 {
if ips, err := net.LookupIP(hostname); err == nil {
for _, ip := range ips {
for _, cidr := range wl.cidrNets {
if cidr.Contains(ip) {
return true
}
}
}
}
}
return false
}
// ResetSSRFWhitelistForTest resets the whitelist singleton so tests in any
// package can re-read the SSRF_WHITELIST environment variable after changing
// it. Exported (rather than unexported) because callers exist outside
// internal/utils — notably internal/infrastructure/web_search/searxng_test.go,
// whose tests would otherwise see whatever whitelist an alphabetically-
// earlier test in the same binary (e.g. proxy_test.go's TestValidateProxyURL)
// cached via the first sync.Once.Do(). NOT for production use — the ForTest
// suffix is the contract.
func ResetSSRFWhitelistForTest() {
ssrfWhitelistOnce = sync.Once{}
ssrfWhitelist = nil
}
// FormatSSRFError takes the error returned by ValidateURLForSSRF and wraps
// it with operator guidance — specifically how to add a host to the SSRF
// allow-list. Without this hint, users hit "Base URL 未通过安全校验" with
// no idea how to recover (the allowlist is configured server-side, not
// in the UI). The hint references SSRF_WHITELIST_EXTRA rather than
// SSRF_WHITELIST because the latter is the project's baseline list and
// EXTRA is the operator's append-only escape hatch.
//
// `label` is a short noun describing the URL field that failed, e.g.
// "Base URL" or "VLM Base URL". The function returns an empty string for
// a nil err so callers can use it inline without guarding.
func FormatSSRFError(label, rawURL string, err error) string {
if err == nil {
return ""
}
host := rawURL
if parsed, perr := parseHostForHint(rawURL); perr == nil && parsed != "" {
host = parsed
}
return fmt.Sprintf(
"%s 未通过安全校验:%v。如该地址确实可信,请联系运维在服务端环境变量 "+
"SSRF_WHITELIST_EXTRA 中加入该主机(支持精确域名 / *.example.com 通配 / IP / CIDR),"+
"示例:SSRF_WHITELIST_EXTRA=%s,*.example.com,10.0.0.0/8",
label, err, host,
)
}
// parseHostForHint extracts a hostname from rawURL purely so we can echo
// it back inside the SSRF hint. Best-effort — returns ("", err) for
// completely unparseable input and the caller falls back to the raw URL.
func parseHostForHint(rawURL string) (string, error) {
if rawURL == "" {
return "", fmt.Errorf("empty url")
}
norm := rawURL
if !strings.Contains(norm, "://") {
norm = "https://" + norm
}
u, err := url.Parse(norm)
if err != nil {
return "", err
}
return u.Hostname(), nil
}
// ValidateURLForSSRF is the centralised entry-point that all handlers should
// call to validate a user-supplied URL. It first checks the SSRF_WHITELIST;
// whitelisted hosts skip the full isSSRFSafeURL check.
//
// rawURL may be a full URL ("https://example.com/v1") or a bare host/host:port
// (for cases like ReconnectDocReader). If a scheme is missing the function
// prepends "https://" before parsing so that net/url can extract the host.
//
// Returns nil when the URL is safe, or an error describing the problem.
func ValidateURLForSSRF(rawURL string) error {
if rawURL == "" {
return nil // callers that require non-empty should validate separately
}
// Normalise: if no scheme, prepend https:// so url.Parse works correctly.
normalized := rawURL
if !strings.Contains(normalized, "://") {
normalized = "https://" + normalized
}
parsed, err := url.Parse(normalized)
if err != nil {
return fmt.Errorf("invalid URL: %w", err)
}
hostname := parsed.Hostname()
if hostname == "" {
return fmt.Errorf("URL has no hostname")
}
// If the host is whitelisted, skip the heavy checks.
if IsSSRFWhitelisted(hostname) {
return nil
}
// Delegate to the full SSRF validation (uses the normalised URL).
if safe, reason := isSSRFSafeURL(normalized); !safe {
return fmt.Errorf("SSRF validation failed: %s", reason)
}
return nil
}
// IsSystemProxy 判断是否为系统代理
func IsSystemProxy(host string) bool {
proxyCfg := httpproxy.FromEnvironment()
for _, proxyUrl := range []string{
proxyCfg.HTTPProxy,
proxyCfg.HTTPSProxy,
} {
if proxyUrl == "" {
continue
}
if parse, err := url.Parse(proxyUrl); err == nil {
if parse.Host == host {
return true
}
}
}
return false
}