Files
Enterprise-Knowledge-Base/internal/application/repository/retriever/opensearch/mapping.go
T
ochan.kwon 11c3236e52 feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.

Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.

What lands in PR 2a
-------------------

Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):

- `repository.go` — Repository struct + NewRepository constructor
  that validates cluster reachability + OS version (2.4+ / 3.x;
  primary tested 3.3.2) + k-NN plugin presence on every cluster
  node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
  dimension index creation, with transient errors not cached so a
  momentary cluster blip does not permanently poison a dim.
  sanitizeIndexName enforces a strict OS-compatible name spec.
  probeVersion uses robust strings.Split/Atoi parsing for
  pre-release suffixes and missing-patch versions. EngineType
  returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
  (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
  only cipher list) and transport tuning for the driver. Caller
  exists only in PR 3 (container.go + engine_factory.go); PR 2a
  remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
  knn_vector + HNSW + content-analyzer mapping with every *_id
  field as an explicit keyword and source_type as integer.
  buildKeywordsMapping ships the dim-less keyword-only index
  mapping used by the no-embedding save path. createIndexAndAlias
  creates <alias>_v1 and aliases <alias> to it, with best-effort
  orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
  defaults (hnsw_m=16, ef_construction=100, ef_search=100,
  shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
  ErrDimensionMismatch, ErrAuth, ErrTransport,
  ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
  ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
  apperrors; PR 3's engine factory wraps these to typed AppError
  2200/2201.
- `stubs.go` — every behavioural method returns
  ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
  HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
  fails-closed for non-empty KBs.

Tests (~1115 LoC, 50 cases):

- `repository_test.go` — interface satisfaction, sentinel mapping,
  sanitizeIndexName positive/negative matrix, semver parsing
  (pre-release / missing-patch), buildIndexMapping JSON shape pin
  (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
  2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
  multi-node coverage, ensureReady concurrency + per-dim isolation
  + transient retry, NewRepository storeID validation, all 11
  stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
  SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
  wrapTransport sentinel mapping + leak guard, isNotFound /
  isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
  TLS 1.2 pinning / cipher list / transport tuning.

Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.

SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------

PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):

- AliasExists method passes dataPointer=nil to its internal do(),
  which means non-2xx responses come back as a plain
  *errors.errorString ("status: 404 Not Found") rather than as
  *opensearch.StructError. aliasExists therefore inspects
  resp.StatusCode directly (resp is returned even when err is
  non-nil) and only falls back to wrapTransport for the "no
  response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
  index uses a mutex + ready/err flag pattern so transient failures
  can be retried by the next caller. The per-dimension path uses
  the `once map[int]*sync.Once` delete-and-recreate trick.

Test fixes folded in
--------------------

While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:

(1) Follow-up to #1445 — fanout test missed the new normalizer policy
    (internal/application/service/knowledgebase_search_fanout_test.go,
    +46 / -6). #1445 changed EngineAwareNormalizer for ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
    Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
    passthrough (those engines surface non-negative cosine to the
    normalizer per Lucene script_score non-negative invariant for
    ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
    OpenSearch, engine-internal or IR-normalized conversions for
    the rest). Milvus is now the only engine that still surfaces
    raw signed cosine in [-1, 1].

    TestRetrieveFromStores_MixedEngine_Normalizes still asserted
    the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
    which under passthrough now becomes clamp01(-0.4) = 0. The
    normalizer's own _test.go was updated at #1445 time, but this
    fan-out integration test was not.

    Fix: rewrite the godoc to spell out the two engine groups;
    restate sub-case 2 as ES passthrough on a production-possible
    mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
    pinning the cosine-shift branch via Milvus -0.4 → 0.3.

(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
    (internal/utils/security.go + internal/utils/security_test.go +
    internal/infrastructure/web_search/searxng_test.go,
    +33 / -9). loadSSRFWhitelist in internal/utils/security.go is
    cached via sync.Once on first call. The internal reset helper
    resetSSRFWhitelistForTest was unexported, so tests in other
    packages could not reset and saw whatever whitelist was cached
    by the first sync.Once.Do() in the same test binary. In
    internal/infrastructure/web_search/, TestValidateProxyURL runs
    before TestValidateSearxngBaseURL alphabetically and exercises
    ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
    whitelist; the later setenv in searxng_test then has no effect
    and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
    Pre-existing on main; surfaced now because this PR was the
    first to do a full `go test ./...` run on top of #1445.

    Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
    ForTest suffix is the test-only contract); update in-package
    callers; in web_search/searxng_test.go import internal/utils
    and call ResetSSRFWhitelistForTest around the env mutation in
    both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
    No production code path changes.

Roadmap
-------

- PR 2b (next, depends on this PR) — read/write implementations:
  query.go + retrieve.go + crud.go land their real bodies; stubs
  for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
  removed; corresponding CRUD/retrieve/filter test cases (~430
  LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
  BatchUpdate*, large-batch async deletes) + i18n + docker-compose
  dev profile. After PR 3 merges, the OpenSearch driver becomes
  reachable via either `engine_type=opensearch` VectorStore or
  `RETRIEVE_DRIVER=opensearch` env.

Backward compatibility
----------------------

- New package — additive only. No existing file modified except
  go.mod / go.sum, the two test files in (1)/(2), and the
  test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
  here (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-26 20:54:58 +08:00

409 lines
14 KiB
Go

package opensearch
import (
"bytes"
"context"
"encoding/json"
"fmt"
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
"github.com/Tencent/WeKnora/internal/logger"
)
// createIndexAndAlias creates "<alias>_v1" with the full k-NN mapping
// and then aliases "<alias>" → "<alias>_v1". Idempotent: returns nil if
// the alias already exists (so concurrent NewRepository calls in
// different processes don't race the PUT).
//
// Orphan cleanup: if aliasPut fails after a successful indicesCreate,
// the function attempts best-effort DELETE of the orphan _v1 index.
// Cleanup failure is logged but does not mask the original aliasPut
// error.
//
// Mapping-drift detection: if indicesCreate returns
// resource_already_exists_exception (cross-process race), the existing
// mapping's structural fingerprint is checked against the expected
// fingerprint. Drift returns ErrConfigInvalid wrapping a clear "manual
// reindex required" message instead of silently overlaying a foreign
// mapping.
func (r *Repository) createIndexAndAlias(ctx context.Context, dim int) error {
log := logger.GetLogger(ctx)
alias := r.indexAlias(dim)
realIndex := alias + "_v1"
exists, err := r.aliasExists(ctx, alias)
if err != nil {
return fmt.Errorf("alias check %s: %w", alias, err)
}
if exists {
return nil // another writer already set it up
}
body, err := buildIndexMapping(r.cfg, dim)
if err != nil {
return fmt.Errorf("build mapping for dim=%d: %w", dim, err)
}
indexCreated := false
if err := r.indicesCreate(ctx, realIndex, body); err != nil {
if isAlreadyExistsError(err) {
// Cross-process race or stale orphan. Verify the existing
// mapping matches what we expected; drift → ErrConfigInvalid.
if driftErr := r.verifyMappingMatches(ctx, realIndex, body); driftErr != nil {
return fmt.Errorf(
"mapping drift on %s: existing index has incompatible mapping; "+
"manual reindex required: %w",
realIndex, driftErr,
)
}
// Mapping matches → safe to proceed with alias setup.
} else {
return fmt.Errorf("create index %s: %w", realIndex, err)
}
} else {
indexCreated = true
}
if err := r.aliasPut(ctx, realIndex, alias); err != nil {
// Best-effort orphan cleanup so future retries find a clean slate.
if indexCreated {
if delErr := r.indicesDelete(ctx, realIndex); delErr != nil {
log.Warnf("[OpenSearch] orphan cleanup failed for %s: %v "+
"(operator must DELETE manually)", realIndex, delErr)
} else {
log.Infof("[OpenSearch] cleaned up orphan %s after aliasPut failure", realIndex)
}
}
return fmt.Errorf("put alias %s → %s: %w", alias, realIndex, err)
}
return nil
}
// buildIndexMapping returns the full mapping JSON for a new OpenSearch
// k-NN index. content uses the standard analyzer (BM25 default); all
// *_id fields are explicit keyword sub-fields so DeleteByQuery / terms
// filters hit them without the ES v8 "useKeywordSuffix" auto-detection
// dance.
//
// match_type is intentionally NOT in the mapping — it's a per-result
// classification derived from RetrieverType at parse time. source_type
// is stored as `integer` (not stringified) so the wire format remains
// stable across SourceType enum extension.
//
// Pure function — no IO, no logging. Easy to unit-test (byte-exact diff).
func buildIndexMapping(cfg internalCfg, dim int) ([]byte, error) {
type mapping struct {
Settings struct {
Index struct {
Knn bool `json:"knn"`
NumberOfShards int `json:"number_of_shards"`
NumberOfReplicas int `json:"number_of_replicas"`
RefreshInterval string `json:"refresh_interval"`
KnnAlgoParamEFSearch int `json:"knn.algo_param.ef_search"`
} `json:"index"`
} `json:"settings"`
Mappings struct {
Properties map[string]any `json:"properties"`
} `json:"mappings"`
}
var m mapping
m.Settings.Index.Knn = true
m.Settings.Index.NumberOfShards = cfg.shards
m.Settings.Index.NumberOfReplicas = cfg.replicas
m.Settings.Index.RefreshInterval = "1s"
m.Settings.Index.KnnAlgoParamEFSearch = cfg.efSearch
m.Mappings.Properties = map[string]any{
"embedding": map[string]any{
"type": "knn_vector",
"dimension": dim,
"method": map[string]any{
"name": "hnsw",
"space_type": "cosinesimil",
"engine": cfg.knnEngine,
"parameters": map[string]any{
"m": cfg.hnswM,
"ef_construction": cfg.hnswEFConstruction,
},
},
},
"content": map[string]any{"type": "text", "analyzer": "standard"},
"chunk_id": map[string]any{"type": "keyword"},
"knowledge_id": map[string]any{"type": "keyword"},
"knowledge_base_id": map[string]any{"type": "keyword"},
"tag_id": map[string]any{"type": "keyword"},
"source_id": map[string]any{"type": "keyword"},
"source_type": map[string]any{"type": "integer"},
"is_enabled": map[string]any{"type": "boolean"},
"is_recommended": map[string]any{"type": "boolean"},
}
out, err := json.Marshal(&m)
if err != nil {
// Cannot happen for our internal struct, but propagate for symmetry
// with the other JSON builders in the package.
return nil, fmt.Errorf("marshal mapping: %w", err)
}
return out, nil
}
// buildKeywordsMapping returns the mapping for the dim-less keyword-only
// index. Same as buildIndexMapping but with the embedding field omitted
// — used by the no-embedding save path (the service layer dispatches
// there when the knowledge base has vector indexing disabled).
func buildKeywordsMapping(cfg internalCfg) ([]byte, error) {
type mapping struct {
Settings struct {
Index struct {
NumberOfShards int `json:"number_of_shards"`
NumberOfReplicas int `json:"number_of_replicas"`
RefreshInterval string `json:"refresh_interval"`
} `json:"index"`
} `json:"settings"`
Mappings struct {
Properties map[string]any `json:"properties"`
} `json:"mappings"`
}
var m mapping
m.Settings.Index.NumberOfShards = cfg.shards
m.Settings.Index.NumberOfReplicas = cfg.replicas
m.Settings.Index.RefreshInterval = "1s"
m.Mappings.Properties = map[string]any{
"content": map[string]any{"type": "text", "analyzer": "standard"},
"chunk_id": map[string]any{"type": "keyword"},
"knowledge_id": map[string]any{"type": "keyword"},
"knowledge_base_id": map[string]any{"type": "keyword"},
"tag_id": map[string]any{"type": "keyword"},
"source_id": map[string]any{"type": "keyword"},
"source_type": map[string]any{"type": "integer"},
"is_enabled": map[string]any{"type": "boolean"},
"is_recommended": map[string]any{"type": "boolean"},
}
out, err := json.Marshal(&m)
if err != nil {
return nil, fmt.Errorf("marshal keywords mapping: %w", err)
}
return out, nil
}
// ensureKeywordsIndex creates the dim-less keyword-only index used by
// the no-embedding save path. mutex + flag pattern so transient
// failures can be retried by the next caller (sync.Once cannot be
// reset).
func (r *Repository) ensureKeywordsIndex(ctx context.Context) error {
r.keywordsMu.Lock()
defer r.keywordsMu.Unlock()
if r.keywordsReady {
return nil
}
if r.keywordsErr != nil && !isTransientErr(r.keywordsErr) {
// Permanent failure — don't retry, surface the same error.
return r.keywordsErr
}
name := r.keywordsIndex()
exists, err := r.aliasExists(ctx, name) // returns true if index exists too
if err != nil {
r.keywordsErr = err
return err
}
if exists {
r.keywordsReady = true
r.keywordsErr = nil
return nil
}
body, err := buildKeywordsMapping(r.cfg)
if err != nil {
r.keywordsErr = err
return err
}
if err := r.indicesCreate(ctx, name, body); err != nil {
if !isAlreadyExistsError(err) {
r.keywordsErr = err
return err
}
// resource_already_exists_exception — race with concurrent process,
// treat as success.
}
r.keywordsReady = true
r.keywordsErr = nil
return nil
}
// indicesCreate is the low-level wrapper around Indices.Create. Body is
// the JSON mapping produced by buildIndexMapping.
func (r *Repository) indicesCreate(ctx context.Context, name string, body []byte) error {
req := osapi.IndicesCreateReq{Index: name, Body: bytes.NewReader(body)}
resp, err := r.client.Indices.Create(ctx, req)
if err != nil {
return wrapTransport(err)
}
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
return nil
}
// indicesDelete is used by createIndexAndAlias's orphan cleanup path.
// Caller decides what to do with the error.
func (r *Repository) indicesDelete(ctx context.Context, name string) error {
req := osapi.IndicesDeleteReq{Indices: []string{name}}
resp, err := r.client.Indices.Delete(ctx, req)
if err != nil {
return wrapTransport(err)
}
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
return nil
}
// aliasExists uses HEAD /_alias/<name> (Indices.Alias.Exists) — returns
// true on 200, false on 404, error on 5xx / transport.
//
// SDK quirk (opensearch-go v4.6.0): the Exists method passes
// dataPointer=nil to its internal do(), which means non-2xx responses
// come back as a plain *errors.errorString ("status: 404 Not Found")
// rather than as a *opensearch.StructError. We therefore inspect
// resp.StatusCode directly (resp is always returned even when err is
// non-nil) and only fall back to wrapTransport for the "no response at
// all" case (true network failure).
func (r *Repository) aliasExists(ctx context.Context, alias string) (bool, error) {
req := osapi.AliasExistsReq{Alias: []string{alias}}
resp, err := r.client.Indices.Alias.Exists(ctx, req)
if resp != nil {
statusCode := resp.StatusCode
drainAndClose(resp.Body)
switch statusCode {
case 200:
return true, nil
case 404:
return false, nil
}
// Unexpected status (5xx) — fall through to the err path so
// wrapTransport can classify (and so the test handler's
// errBody is logged for diagnosis).
}
if err != nil {
return false, wrapTransport(err)
}
return false, nil
}
// aliasPut creates an alias pointing at index. Single add operation
// (rolling-reindex atomic swap is deferred to a follow-up PR — see
// stubs.go swapToVersion).
func (r *Repository) aliasPut(ctx context.Context, index, alias string) error {
req := osapi.AliasPutReq{
Indices: []string{index},
Alias: alias,
}
resp, err := r.client.Indices.Alias.Put(ctx, req)
if err != nil {
return wrapTransport(err)
}
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
return nil
}
// verifyMappingMatches GETs the existing index's mapping and compares
// the embedding field's structural fingerprint (dimension, HNSW
// parameters, engine) against the expected. A full field-by-field diff
// would be ideal but expensive; we check only the parts that matter for
// query correctness.
func (r *Repository) verifyMappingMatches(ctx context.Context, index string, expected []byte) error {
req := osapi.MappingGetReq{Indices: []string{index}}
resp, err := r.client.Indices.Mapping.Get(ctx, &req)
if err != nil {
return fmt.Errorf("get mapping %s: %w", index, wrapTransport(err))
}
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
// Both fingerprints are derived from the mapping's embedding-field
// HNSW parameters: dimension, m, ef_construction, engine, space_type.
// We parse the expected (our own JSON) and the actual (cluster
// response) and compare just those bits.
expectedFP, err := extractEmbeddingFingerprint(expected)
if err != nil {
return fmt.Errorf("parse expected fingerprint: %w", err)
}
actualFP, err := extractMappingFingerprintFromResp(resp, index)
if err != nil {
return fmt.Errorf("parse actual fingerprint: %w", err)
}
if expectedFP != actualFP {
return fmt.Errorf("mapping fingerprint mismatch: expected %+v, got %+v: %w",
expectedFP, actualFP, ErrConfigInvalid)
}
return nil
}
// embFingerprint is the minimal set of HNSW parameters that uniquely
// determine query semantics for a k-NN index. If any of these differ
// between create-time and an existing index, retrieval is incorrect.
type embFingerprint struct {
Dimension int
M int
EFConstruction int
Engine string
SpaceType string
}
// extractEmbeddingFingerprint parses our own mapping JSON (produced by
// buildIndexMapping) to pull out the HNSW parameters.
func extractEmbeddingFingerprint(body []byte) (embFingerprint, error) {
var m struct {
Mappings struct {
Properties map[string]any `json:"properties"`
} `json:"mappings"`
}
if err := json.Unmarshal(body, &m); err != nil {
return embFingerprint{}, err
}
return extractFingerprintFromProps(m.Mappings.Properties)
}
// extractMappingFingerprintFromResp parses the cluster's GET-mapping
// response and pulls out the embedding-field HNSW fingerprint for the
// given index.
func extractMappingFingerprintFromResp(resp *osapi.MappingGetResp, index string) (embFingerprint, error) {
indexMapping, ok := resp.Indices[index]
if !ok {
return embFingerprint{}, fmt.Errorf("index %s not in mapping response", index)
}
// MappingGetResp.Indices[index].Mappings is a json.RawMessage in
// the SDK — we decode the inner properties shape ourselves.
var m struct {
Properties map[string]any `json:"properties"`
}
if err := json.Unmarshal(indexMapping.Mappings, &m); err != nil {
return embFingerprint{}, err
}
return extractFingerprintFromProps(m.Properties)
}
func extractFingerprintFromProps(props map[string]any) (embFingerprint, error) {
emb, ok := props["embedding"].(map[string]any)
if !ok {
return embFingerprint{}, fmt.Errorf("embedding field missing or wrong type")
}
dim, _ := emb["dimension"].(float64) // json.Unmarshal produces float64 for numbers
method, _ := emb["method"].(map[string]any)
engine, _ := method["engine"].(string)
spaceType, _ := method["space_type"].(string)
params, _ := method["parameters"].(map[string]any)
m, _ := params["m"].(float64)
efc, _ := params["ef_construction"].(float64)
return embFingerprint{
Dimension: int(dim),
M: int(m),
EFConstruction: int(efc),
Engine: engine,
SpaceType: spaceType,
}, nil
}