Files
Enterprise-Knowledge-Base/internal/application/repository/retriever/opensearch/retrieve.go
T
Ochan Kwon f5820c13ac feat(retriever): implement OpenSearch driver read/write paths (PR 2b of 3) (#1482)
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the
PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their
real implementations, drops the stubs from `stubs.go`, and adds the
27 corresponding test cases that were intentionally held back. The
driver remains gated dead code (no registry / factory / env path
mentions OpenSearch — PR 3 ships the activation switch); the
behaviour added here only fires once PR 3 lands.

Depends on PR 2a of #1440 (driver skeleton).

What lands in PR 2b
-------------------

Three new production files (+762 LoC):

- `query.go` (+154) — typed retrieveFilters struct as the only entry
  to query construction (closes the map[string]any injection surface;
  hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN,
  applies min_score when Threshold > 0) and buildKeywordQuery (BM25);
  native hybrid intentionally omitted (out of scope for Phase 3 —
  service-layer RRF stays in charge). Filter struct carries
  ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode
  use cases.
- `retrieve.go` (+204) — Retrieve resolves the per-dim alias from
  (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or
  (3) the cross-dim wildcard <base>_* for BM25-only callers. Always
  returns exactly one *RetrieveResult (the Phase 2 fan-out contract).
  Hit parsing caps response body at 16 MB and warns if _id !=
  _source.chunk_id. Score passes through raw in [0, 1] — the k-NN
  plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped
  (1 + cos) / 2 before the client sees it.
- `crud.go` (+405) — Save / BatchSave with _id = chunk_id for
  idempotency, dim-aware pre-marshal NDJSON size cap (rejects with
  ErrBatchTooLarge before allocating a 50 MB body for a too-big
  batch), and per-item error inspection that does NOT leak cluster-
  side Reason strings into wrapped public errors. additionalParams
  contract verified against elasticsearch/structs.go
  ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding
  is map[string][]float32 keyed by SourceID, chunk_enabled is
  map[string]bool keyed by ChunkID. Missing embedding routes Save /
  BatchSave to the dim-less <base>_keywords index. Sync DeleteBy*
  paths capped at 1000 IDs per call with ErrBatchTooLarge.

  toDoc writes the is_recommended boolean from IndexInfo. The
  matching mapping entry was added in PR 2a alongside is_enabled —
  the pair stays in sync between the vector and the keyword-only
  index so the field round-trips through CopyIndices (PR 3) and any
  future FAQ-priority filtering.

Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve /
DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList
no longer return ErrFeatureNotEnabled — they delegate to the
production implementations above. The remaining stubs (CopyIndices,
BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion,
EstimateStorageSize lower-bound) stay until PR 3.

Tests (+490 net to repository_test.go, 28 new cases):

- retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs,
  IncludeDisabled skip semantics, default is_enabled=true pin.
- buildKNNQuery: min_score applied when Threshold > 0; omitted at
  threshold zero.
- buildKeywordQuery: content match appended after typed filters.
- toDoc: embedding field omitted when empty, present when set,
  source_type serialised as integer, **is_recommended round-trips
  both true and false**.
- lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key
  contract; nil-params and wrong-type degrade paths.
- extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings
  yield dim=0.
- estimateBulkBodyBytes: size estimator boundary.
- effectiveTopK: clamping when caller leaves TopK at the magic-10
  default.
- resolveDim: AdditionalParams takes precedence; Embedding fallback;
  zero-zero triggers cross-dim multi-index search.
- inspectBulkResponse: all-succeeded fast path; total-count + leak-
  guard for cluster-side Reason strings.
- BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal);
  empty list is a no-op.
- DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty
  list is a no-op.

SDK quirk addressed in PR 2b
----------------------------

DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level
"wait_for" value documented in the OpenSearch REST API is not
directly expressible. deleteByTerms uses Refresh:&true which forces
immediate segment flush (read-your-writes guaranteed but more
expensive than wait_for). A follow-up PR can drop to
*opensearchapi.Client.Transport.Perform for wait_for if telemetry
shows the cost is material.

Backward compatibility
----------------------

- Additive within the opensearch package only. No file outside the
  package is touched.
- Driver still unreachable: PR 3 ships the activation switch.
- The PR 1 normalizer case for OpenSearch remains unreachable here
  (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-28 08:00:33 +08:00

206 lines
6.9 KiB
Go

package opensearch
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
)
// Retrieve dispatches to vectorRetrieve or keywordsRetrieve based on
// params.RetrieverType. Returns exactly one *RetrieveResult — the
// service-layer multi-store dispatcher depends on this one-bundle-per-
// driver protocol. Score in result is RAW [0, 1] passthrough (the
// OpenSearch k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation has
// already mapped to (1 + cos) / 2 before the score reaches us).
//
// Dim resolution order:
// 1. params.AdditionalParams["dim"] if present and positive
// 2. len(params.Embedding) if positive
// 3. cross-dim multi-index search for keyword-only paths via
// "<base>_*" wildcard pattern
//
// Vector retrieve always requires a positive dim (it's intrinsic to k-NN).
func (r *Repository) Retrieve(
ctx context.Context,
params types.RetrieveParams,
) ([]*types.RetrieveResult, error) {
dim, multiIndex := resolveDim(params)
switch params.RetrieverType {
case types.VectorRetrieverType:
if dim == 0 {
return nil, fmt.Errorf(
"opensearch: vector retrieve requires embedding or AdditionalParams[\"dim\"]: %w",
ErrDimensionMismatch)
}
if err := r.ensureReady(ctx, dim); err != nil {
return nil, err
}
return r.vectorRetrieve(ctx, params, dim)
case types.KeywordsRetrieverType:
if multiIndex {
// BM25-only path without dim — search across "<base>_*"
// wildcard pattern. Slower than a single-index search but
// correct for admin / keyword-only callers.
return r.keywordsRetrieve(ctx, params, r.baseIndex+"_*")
}
if err := r.ensureReady(ctx, dim); err != nil {
return nil, err
}
return r.keywordsRetrieve(ctx, params, r.indexAlias(dim))
default:
return nil, fmt.Errorf("opensearch: unsupported retriever type %q", params.RetrieverType)
}
}
// resolveDim returns (dim, multiIndex). multiIndex=true means dim is
// unknown and the caller should use a wildcard alias pattern (keyword
// path only — vector retrieve cannot be dim-less).
func resolveDim(p types.RetrieveParams) (int, bool) {
if p.AdditionalParams != nil {
if v, ok := p.AdditionalParams["dim"].(int); ok && v > 0 {
return v, false
}
}
if len(p.Embedding) > 0 {
return len(p.Embedding), false
}
return 0, true // unknown — multi-index fallback for keyword
}
func (r *Repository) vectorRetrieve(
ctx context.Context, p types.RetrieveParams, dim int,
) ([]*types.RetrieveResult, error) {
body, err := buildKNNQuery(p.Embedding, effectiveTopK(ctx, p), p.Threshold, fromParams(p))
if err != nil {
return nil, err
}
hits, err := r.search(ctx, r.indexAlias(dim), body)
if err != nil {
return nil, err
}
return wrapResults(ctx, hits, types.VectorRetrieverType, types.MatchTypeEmbedding), nil
}
func (r *Repository) keywordsRetrieve(
ctx context.Context, p types.RetrieveParams, indexPattern string,
) ([]*types.RetrieveResult, error) {
body, err := buildKeywordQuery(p.Query, effectiveTopK(ctx, p), p.Threshold, fromParams(p))
if err != nil {
return nil, err
}
hits, err := r.search(ctx, indexPattern, body)
if err != nil {
return nil, err
}
return wrapResults(ctx, hits, types.KeywordsRetrieverType, types.MatchTypeKeywords), nil
}
// effectiveTopK returns the caller's TopK or 10 if unset. The 10 is a
// defensive default — every real caller sets TopK, so a TopK=10 in
// production logs indicates a caller bug. Caps at 10000 to stay below
// OpenSearch's index.max_result_window default.
func effectiveTopK(ctx context.Context, p types.RetrieveParams) int {
if p.TopK <= 0 {
logger.GetLogger(ctx).Warnf("[OpenSearch] Retrieve called with TopK<=0; defaulting to 10 (caller bug?)")
return 10
}
if p.TopK > 10000 {
return 10000
}
return p.TopK
}
func (r *Repository) search(ctx context.Context, indexPattern string, body []byte) ([]hit, error) {
req := osapi.SearchReq{Indices: []string{indexPattern}, Body: bytes.NewReader(body)}
resp, err := r.client.Search(ctx, &req)
if err != nil {
if isNotFound(err) {
return nil, fmt.Errorf("opensearch: index %s missing: %w", indexPattern, ErrIndexNotFound)
}
return nil, wrapTransport(err)
}
defer func() {
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
}()
// Cap response body at 16MB (search results bounded by max_result_window * doc_size).
return parseSearchHits(io.LimitReader(resp.Inspect().Response.Body, 16<<20))
}
// hit is the structural shape we pull from each OpenSearch search hit.
// Field-by-field decode (vs map[string]any) keeps the JSON shape
// pinned at compile time.
type hit struct {
ID string `json:"_id"` // equals chunk_id per the indexing invariant
Score float64 `json:"_score"`
Source struct {
Content string `json:"content"`
ChunkID string `json:"chunk_id"`
KnowledgeID string `json:"knowledge_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
SourceID string `json:"source_id"`
SourceType int `json:"source_type"` // integer, not stringified
TagID string `json:"tag_id"`
IsEnabled bool `json:"is_enabled"`
} `json:"_source"`
}
func parseSearchHits(body io.Reader) ([]hit, error) {
var resp struct {
Hits struct {
Hits []hit `json:"hits"`
} `json:"hits"`
}
if err := json.NewDecoder(body).Decode(&resp); err != nil {
return nil, fmt.Errorf("opensearch: parse search response: %w", ErrTransport)
}
return resp.Hits.Hits, nil
}
// wrapResults converts OpenSearch hits to the upstream IndexWithScore
// shape. Always returns exactly one *RetrieveResult — the service-layer
// multi-store dispatcher depends on this "one engine result per driver"
// contract.
//
// Defensive: warns if hit._id != _source.chunk_id (we set _id=chunk_id
// on every write, so a mismatch means external tooling has been writing
// into the index) but does not fail the query — the result is still
// usable for ranking, just suspicious.
func wrapResults(ctx context.Context, hits []hit, rt types.RetrieverType, mt types.MatchType) []*types.RetrieveResult {
log := logger.GetLogger(ctx)
out := make([]*types.IndexWithScore, 0, len(hits))
for _, h := range hits {
if h.ID != h.Source.ChunkID {
log.Warnf("[OpenSearch] hit._id=%q != _source.chunk_id=%q (D12 invariant violation)",
h.ID, h.Source.ChunkID)
}
out = append(out, &types.IndexWithScore{
ID: h.ID,
ChunkID: h.Source.ChunkID,
KnowledgeID: h.Source.KnowledgeID,
KnowledgeBaseID: h.Source.KnowledgeBaseID,
SourceID: h.Source.SourceID,
SourceType: types.SourceType(h.Source.SourceType),
TagID: h.Source.TagID,
Content: h.Source.Content,
Score: h.Score,
MatchType: mt,
IsEnabled: h.Source.IsEnabled,
})
}
return []*types.RetrieveResult{{
Results: out,
RetrieverEngineType: types.OpenSearchRetrieverEngineType,
RetrieverType: rt,
}}
}