Files
Enterprise-Knowledge-Base/internal/application/repository/retriever/opensearch/query.go
T
Ochan Kwon f5820c13ac feat(retriever): implement OpenSearch driver read/write paths (PR 2b of 3) (#1482)
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the
PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their
real implementations, drops the stubs from `stubs.go`, and adds the
27 corresponding test cases that were intentionally held back. The
driver remains gated dead code (no registry / factory / env path
mentions OpenSearch — PR 3 ships the activation switch); the
behaviour added here only fires once PR 3 lands.

Depends on PR 2a of #1440 (driver skeleton).

What lands in PR 2b
-------------------

Three new production files (+762 LoC):

- `query.go` (+154) — typed retrieveFilters struct as the only entry
  to query construction (closes the map[string]any injection surface;
  hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN,
  applies min_score when Threshold > 0) and buildKeywordQuery (BM25);
  native hybrid intentionally omitted (out of scope for Phase 3 —
  service-layer RRF stays in charge). Filter struct carries
  ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode
  use cases.
- `retrieve.go` (+204) — Retrieve resolves the per-dim alias from
  (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or
  (3) the cross-dim wildcard <base>_* for BM25-only callers. Always
  returns exactly one *RetrieveResult (the Phase 2 fan-out contract).
  Hit parsing caps response body at 16 MB and warns if _id !=
  _source.chunk_id. Score passes through raw in [0, 1] — the k-NN
  plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped
  (1 + cos) / 2 before the client sees it.
- `crud.go` (+405) — Save / BatchSave with _id = chunk_id for
  idempotency, dim-aware pre-marshal NDJSON size cap (rejects with
  ErrBatchTooLarge before allocating a 50 MB body for a too-big
  batch), and per-item error inspection that does NOT leak cluster-
  side Reason strings into wrapped public errors. additionalParams
  contract verified against elasticsearch/structs.go
  ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding
  is map[string][]float32 keyed by SourceID, chunk_enabled is
  map[string]bool keyed by ChunkID. Missing embedding routes Save /
  BatchSave to the dim-less <base>_keywords index. Sync DeleteBy*
  paths capped at 1000 IDs per call with ErrBatchTooLarge.

  toDoc writes the is_recommended boolean from IndexInfo. The
  matching mapping entry was added in PR 2a alongside is_enabled —
  the pair stays in sync between the vector and the keyword-only
  index so the field round-trips through CopyIndices (PR 3) and any
  future FAQ-priority filtering.

Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve /
DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList
no longer return ErrFeatureNotEnabled — they delegate to the
production implementations above. The remaining stubs (CopyIndices,
BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion,
EstimateStorageSize lower-bound) stay until PR 3.

Tests (+490 net to repository_test.go, 28 new cases):

- retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs,
  IncludeDisabled skip semantics, default is_enabled=true pin.
- buildKNNQuery: min_score applied when Threshold > 0; omitted at
  threshold zero.
- buildKeywordQuery: content match appended after typed filters.
- toDoc: embedding field omitted when empty, present when set,
  source_type serialised as integer, **is_recommended round-trips
  both true and false**.
- lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key
  contract; nil-params and wrong-type degrade paths.
- extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings
  yield dim=0.
- estimateBulkBodyBytes: size estimator boundary.
- effectiveTopK: clamping when caller leaves TopK at the magic-10
  default.
- resolveDim: AdditionalParams takes precedence; Embedding fallback;
  zero-zero triggers cross-dim multi-index search.
- inspectBulkResponse: all-succeeded fast path; total-count + leak-
  guard for cluster-side Reason strings.
- BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal);
  empty list is a no-op.
- DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty
  list is a no-op.

SDK quirk addressed in PR 2b
----------------------------

DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level
"wait_for" value documented in the OpenSearch REST API is not
directly expressible. deleteByTerms uses Refresh:&true which forces
immediate segment flush (read-your-writes guaranteed but more
expensive than wait_for). A follow-up PR can drop to
*opensearchapi.Client.Transport.Perform for wait_for if telemetry
shows the cost is material.

Backward compatibility
----------------------

- Additive within the opensearch package only. No file outside the
  package is touched.
- Driver still unreachable: PR 3 ships the activation switch.
- The PR 1 normalizer case for OpenSearch remains unreachable here
  (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-28 08:00:33 +08:00

155 lines
5.0 KiB
Go

package opensearch
import (
"encoding/json"
"fmt"
"github.com/Tencent/WeKnora/internal/types"
)
// retrieveFilters is the typed shape of OpenSearch query filter
// clauses. It is the ONLY way callers (Retrieve) can constrain a
// query — there is no JSON-fragment escape hatch. This prevents the
// "map[string]interface{} injection" failure mode where a hostile
// chunk_id like {"$or":[...]} would otherwise be inlined as JSON DSL.
//
// All slice fields are interpreted as IN-set membership (OS terms
// query). Empty slice = no constraint on that dimension.
//
// IncludeDisabled controls the implicit is_enabled = true clause:
// - false (default) — appends is_enabled term, hiding disabled chunks
// - true — omits the clause; admin / audit callers can
// observe disabled chunks. Service layer pins false for user-facing
// APIs.
type retrieveFilters struct {
KBIDs []string
KnowledgeIDs []string
TagIDs []string
ExcludeChunkIDs []string
ExcludeKnowledgeIDs []string
IncludeDisabled bool
}
// fromParams projects RetrieveParams to retrieveFilters, applying the
// multi-KB input from the service-layer dispatcher as terms.
// ExcludeKnowledgeIDs is included so the OpenSearch driver matches the
// behaviour of the Qdrant / pgvector drivers.
func fromParams(p types.RetrieveParams) *retrieveFilters {
return &retrieveFilters{
KBIDs: p.KnowledgeBaseIDs,
KnowledgeIDs: p.KnowledgeIDs,
TagIDs: p.TagIDs,
ExcludeChunkIDs: p.ExcludeChunkIDs,
ExcludeKnowledgeIDs: p.ExcludeKnowledgeIDs,
// IncludeDisabled stays false — set explicitly by admin callers
// only. Driver receives this from a typed field, not from
// AdditionalParams, so the contract is checked at compile time.
}
}
// toBoolMust emits the OpenSearch bool.must clause list for this
// filter. Returns a freshly allocated slice each call so callers can
// safely append additional clauses (e.g. the keyword match clause in
// buildKeywordQuery) without aliasing.
func (f *retrieveFilters) toBoolMust() []map[string]any {
var must []map[string]any
if len(f.KBIDs) > 0 {
must = append(must, map[string]any{
"terms": map[string]any{"knowledge_base_id": f.KBIDs},
})
}
if len(f.KnowledgeIDs) > 0 {
must = append(must, map[string]any{
"terms": map[string]any{"knowledge_id": f.KnowledgeIDs},
})
}
if len(f.TagIDs) > 0 {
must = append(must, map[string]any{
"terms": map[string]any{"tag_id": f.TagIDs},
})
}
if len(f.ExcludeChunkIDs) > 0 {
must = append(must, map[string]any{
"bool": map[string]any{
"must_not": map[string]any{
"terms": map[string]any{"chunk_id": f.ExcludeChunkIDs},
},
},
})
}
if len(f.ExcludeKnowledgeIDs) > 0 {
must = append(must, map[string]any{
"bool": map[string]any{
"must_not": map[string]any{
"terms": map[string]any{"knowledge_id": f.ExcludeKnowledgeIDs},
},
},
})
}
if !f.IncludeDisabled {
must = append(must, map[string]any{
"term": map[string]any{"is_enabled": true},
})
}
return must
}
// buildKNNQuery composes a pure k-NN ANN query with caller filters. No
// hybrid pipeline — native hybrid is out of scope for this PR; fusion
// stays in the service-layer RRF.
//
// Applies `min_score` when threshold > 0. The OpenSearch k-NN plugin's
// SpaceType.COSINESIMIL.scoreTranslation produces (1 + cos) / 2 ∈ [0, 1]
// at the client, so the caller's Threshold value passes through to
// min_score directly without further scaling.
//
// Signature returns ([]byte, error) so callers can propagate
// json.Marshal failures (reachable for NaN / +/-Inf in embedding).
func buildKNNQuery(embedding []float32, topK int, threshold float64, f *retrieveFilters) ([]byte, error) {
knn := map[string]any{
"embedding": map[string]any{
"vector": embedding,
"k": topK,
"filter": map[string]any{
"bool": map[string]any{"must": f.toBoolMust()},
},
},
}
body := map[string]any{
"size": topK,
"query": map[string]any{"knn": knn},
}
if threshold > 0 {
body["min_score"] = threshold
}
out, err := json.Marshal(body)
if err != nil {
return nil, fmt.Errorf("marshal knn query: %w", err)
}
return out, nil
}
// buildKeywordQuery composes a BM25 match query against the content
// field with caller filters. Score range is OpenSearch BM25 native (no
// upper bound) — the normalizer keyword-passthrough rule treats keyword
// retrievers as rank-based for RRF, so the score scale does not need
// normalization here. min_score still applied if requested.
func buildKeywordQuery(queryText string, topK int, threshold float64, f *retrieveFilters) ([]byte, error) {
must := f.toBoolMust()
must = append(must, map[string]any{
"match": map[string]any{"content": queryText},
})
body := map[string]any{
"size": topK,
"query": map[string]any{"bool": map[string]any{"must": must}},
}
if threshold > 0 {
body["min_score"] = threshold
}
out, err := json.Marshal(body)
if err != nil {
return nil, fmt.Errorf("marshal keyword query: %w", err)
}
return out, nil
}