f5820c13ac
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their real implementations, drops the stubs from `stubs.go`, and adds the 27 corresponding test cases that were intentionally held back. The driver remains gated dead code (no registry / factory / env path mentions OpenSearch — PR 3 ships the activation switch); the behaviour added here only fires once PR 3 lands. Depends on PR 2a of #1440 (driver skeleton). What lands in PR 2b ------------------- Three new production files (+762 LoC): - `query.go` (+154) — typed retrieveFilters struct as the only entry to query construction (closes the map[string]any injection surface; hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN, applies min_score when Threshold > 0) and buildKeywordQuery (BM25); native hybrid intentionally omitted (out of scope for Phase 3 — service-layer RRF stays in charge). Filter struct carries ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode use cases. - `retrieve.go` (+204) — Retrieve resolves the per-dim alias from (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or (3) the cross-dim wildcard <base>_* for BM25-only callers. Always returns exactly one *RetrieveResult (the Phase 2 fan-out contract). Hit parsing caps response body at 16 MB and warns if _id != _source.chunk_id. Score passes through raw in [0, 1] — the k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped (1 + cos) / 2 before the client sees it. - `crud.go` (+405) — Save / BatchSave with _id = chunk_id for idempotency, dim-aware pre-marshal NDJSON size cap (rejects with ErrBatchTooLarge before allocating a 50 MB body for a too-big batch), and per-item error inspection that does NOT leak cluster- side Reason strings into wrapped public errors. additionalParams contract verified against elasticsearch/structs.go ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding is map[string][]float32 keyed by SourceID, chunk_enabled is map[string]bool keyed by ChunkID. Missing embedding routes Save / BatchSave to the dim-less <base>_keywords index. Sync DeleteBy* paths capped at 1000 IDs per call with ErrBatchTooLarge. toDoc writes the is_recommended boolean from IndexInfo. The matching mapping entry was added in PR 2a alongside is_enabled — the pair stays in sync between the vector and the keyword-only index so the field round-trips through CopyIndices (PR 3) and any future FAQ-priority filtering. Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve / DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList no longer return ErrFeatureNotEnabled — they delegate to the production implementations above. The remaining stubs (CopyIndices, BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion, EstimateStorageSize lower-bound) stay until PR 3. Tests (+490 net to repository_test.go, 28 new cases): - retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs, IncludeDisabled skip semantics, default is_enabled=true pin. - buildKNNQuery: min_score applied when Threshold > 0; omitted at threshold zero. - buildKeywordQuery: content match appended after typed filters. - toDoc: embedding field omitted when empty, present when set, source_type serialised as integer, **is_recommended round-trips both true and false**. - lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key contract; nil-params and wrong-type degrade paths. - extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings yield dim=0. - estimateBulkBodyBytes: size estimator boundary. - effectiveTopK: clamping when caller leaves TopK at the magic-10 default. - resolveDim: AdditionalParams takes precedence; Embedding fallback; zero-zero triggers cross-dim multi-index search. - inspectBulkResponse: all-succeeded fast path; total-count + leak- guard for cluster-side Reason strings. - BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal); empty list is a no-op. - DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty list is a no-op. SDK quirk addressed in PR 2b ---------------------------- DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level "wait_for" value documented in the OpenSearch REST API is not directly expressible. deleteByTerms uses Refresh:&true which forces immediate segment flush (read-your-writes guaranteed but more expensive than wait_for). A follow-up PR can drop to *opensearchapi.Client.Transport.Perform for wait_for if telemetry shows the cost is material. Backward compatibility ---------------------- - Additive within the opensearch package only. No file outside the package is touched. - Driver still unreachable: PR 3 ships the activation switch. - The PR 1 normalizer case for OpenSearch remains unreachable here (no driver instance produces a result yet). Test plan --------- - [x] go build ./... clean - [x] go vet ./... clean - [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes - [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/ shows only PR 1's normalizer case + this driver's EngineType() and tests — no activation path. - [x] grep -r "case \"opensearch\"" internal/ shows no hits.
206 lines
6.9 KiB
Go
206 lines
6.9 KiB
Go
package opensearch
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
|
|
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
|
|
|
|
"github.com/Tencent/WeKnora/internal/logger"
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
)
|
|
|
|
// Retrieve dispatches to vectorRetrieve or keywordsRetrieve based on
|
|
// params.RetrieverType. Returns exactly one *RetrieveResult — the
|
|
// service-layer multi-store dispatcher depends on this one-bundle-per-
|
|
// driver protocol. Score in result is RAW [0, 1] passthrough (the
|
|
// OpenSearch k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation has
|
|
// already mapped to (1 + cos) / 2 before the score reaches us).
|
|
//
|
|
// Dim resolution order:
|
|
// 1. params.AdditionalParams["dim"] if present and positive
|
|
// 2. len(params.Embedding) if positive
|
|
// 3. cross-dim multi-index search for keyword-only paths via
|
|
// "<base>_*" wildcard pattern
|
|
//
|
|
// Vector retrieve always requires a positive dim (it's intrinsic to k-NN).
|
|
func (r *Repository) Retrieve(
|
|
ctx context.Context,
|
|
params types.RetrieveParams,
|
|
) ([]*types.RetrieveResult, error) {
|
|
dim, multiIndex := resolveDim(params)
|
|
|
|
switch params.RetrieverType {
|
|
case types.VectorRetrieverType:
|
|
if dim == 0 {
|
|
return nil, fmt.Errorf(
|
|
"opensearch: vector retrieve requires embedding or AdditionalParams[\"dim\"]: %w",
|
|
ErrDimensionMismatch)
|
|
}
|
|
if err := r.ensureReady(ctx, dim); err != nil {
|
|
return nil, err
|
|
}
|
|
return r.vectorRetrieve(ctx, params, dim)
|
|
case types.KeywordsRetrieverType:
|
|
if multiIndex {
|
|
// BM25-only path without dim — search across "<base>_*"
|
|
// wildcard pattern. Slower than a single-index search but
|
|
// correct for admin / keyword-only callers.
|
|
return r.keywordsRetrieve(ctx, params, r.baseIndex+"_*")
|
|
}
|
|
if err := r.ensureReady(ctx, dim); err != nil {
|
|
return nil, err
|
|
}
|
|
return r.keywordsRetrieve(ctx, params, r.indexAlias(dim))
|
|
default:
|
|
return nil, fmt.Errorf("opensearch: unsupported retriever type %q", params.RetrieverType)
|
|
}
|
|
}
|
|
|
|
// resolveDim returns (dim, multiIndex). multiIndex=true means dim is
|
|
// unknown and the caller should use a wildcard alias pattern (keyword
|
|
// path only — vector retrieve cannot be dim-less).
|
|
func resolveDim(p types.RetrieveParams) (int, bool) {
|
|
if p.AdditionalParams != nil {
|
|
if v, ok := p.AdditionalParams["dim"].(int); ok && v > 0 {
|
|
return v, false
|
|
}
|
|
}
|
|
if len(p.Embedding) > 0 {
|
|
return len(p.Embedding), false
|
|
}
|
|
return 0, true // unknown — multi-index fallback for keyword
|
|
}
|
|
|
|
func (r *Repository) vectorRetrieve(
|
|
ctx context.Context, p types.RetrieveParams, dim int,
|
|
) ([]*types.RetrieveResult, error) {
|
|
body, err := buildKNNQuery(p.Embedding, effectiveTopK(ctx, p), p.Threshold, fromParams(p))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
hits, err := r.search(ctx, r.indexAlias(dim), body)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return wrapResults(ctx, hits, types.VectorRetrieverType, types.MatchTypeEmbedding), nil
|
|
}
|
|
|
|
func (r *Repository) keywordsRetrieve(
|
|
ctx context.Context, p types.RetrieveParams, indexPattern string,
|
|
) ([]*types.RetrieveResult, error) {
|
|
body, err := buildKeywordQuery(p.Query, effectiveTopK(ctx, p), p.Threshold, fromParams(p))
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
hits, err := r.search(ctx, indexPattern, body)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return wrapResults(ctx, hits, types.KeywordsRetrieverType, types.MatchTypeKeywords), nil
|
|
}
|
|
|
|
// effectiveTopK returns the caller's TopK or 10 if unset. The 10 is a
|
|
// defensive default — every real caller sets TopK, so a TopK=10 in
|
|
// production logs indicates a caller bug. Caps at 10000 to stay below
|
|
// OpenSearch's index.max_result_window default.
|
|
func effectiveTopK(ctx context.Context, p types.RetrieveParams) int {
|
|
if p.TopK <= 0 {
|
|
logger.GetLogger(ctx).Warnf("[OpenSearch] Retrieve called with TopK<=0; defaulting to 10 (caller bug?)")
|
|
return 10
|
|
}
|
|
if p.TopK > 10000 {
|
|
return 10000
|
|
}
|
|
return p.TopK
|
|
}
|
|
|
|
func (r *Repository) search(ctx context.Context, indexPattern string, body []byte) ([]hit, error) {
|
|
req := osapi.SearchReq{Indices: []string{indexPattern}, Body: bytes.NewReader(body)}
|
|
resp, err := r.client.Search(ctx, &req)
|
|
if err != nil {
|
|
if isNotFound(err) {
|
|
return nil, fmt.Errorf("opensearch: index %s missing: %w", indexPattern, ErrIndexNotFound)
|
|
}
|
|
return nil, wrapTransport(err)
|
|
}
|
|
defer func() {
|
|
if resp != nil {
|
|
drainAndClose(resp.Inspect().Response.Body)
|
|
}
|
|
}()
|
|
// Cap response body at 16MB (search results bounded by max_result_window * doc_size).
|
|
return parseSearchHits(io.LimitReader(resp.Inspect().Response.Body, 16<<20))
|
|
}
|
|
|
|
// hit is the structural shape we pull from each OpenSearch search hit.
|
|
// Field-by-field decode (vs map[string]any) keeps the JSON shape
|
|
// pinned at compile time.
|
|
type hit struct {
|
|
ID string `json:"_id"` // equals chunk_id per the indexing invariant
|
|
Score float64 `json:"_score"`
|
|
Source struct {
|
|
Content string `json:"content"`
|
|
ChunkID string `json:"chunk_id"`
|
|
KnowledgeID string `json:"knowledge_id"`
|
|
KnowledgeBaseID string `json:"knowledge_base_id"`
|
|
SourceID string `json:"source_id"`
|
|
SourceType int `json:"source_type"` // integer, not stringified
|
|
TagID string `json:"tag_id"`
|
|
IsEnabled bool `json:"is_enabled"`
|
|
} `json:"_source"`
|
|
}
|
|
|
|
func parseSearchHits(body io.Reader) ([]hit, error) {
|
|
var resp struct {
|
|
Hits struct {
|
|
Hits []hit `json:"hits"`
|
|
} `json:"hits"`
|
|
}
|
|
if err := json.NewDecoder(body).Decode(&resp); err != nil {
|
|
return nil, fmt.Errorf("opensearch: parse search response: %w", ErrTransport)
|
|
}
|
|
return resp.Hits.Hits, nil
|
|
}
|
|
|
|
// wrapResults converts OpenSearch hits to the upstream IndexWithScore
|
|
// shape. Always returns exactly one *RetrieveResult — the service-layer
|
|
// multi-store dispatcher depends on this "one engine result per driver"
|
|
// contract.
|
|
//
|
|
// Defensive: warns if hit._id != _source.chunk_id (we set _id=chunk_id
|
|
// on every write, so a mismatch means external tooling has been writing
|
|
// into the index) but does not fail the query — the result is still
|
|
// usable for ranking, just suspicious.
|
|
func wrapResults(ctx context.Context, hits []hit, rt types.RetrieverType, mt types.MatchType) []*types.RetrieveResult {
|
|
log := logger.GetLogger(ctx)
|
|
out := make([]*types.IndexWithScore, 0, len(hits))
|
|
for _, h := range hits {
|
|
if h.ID != h.Source.ChunkID {
|
|
log.Warnf("[OpenSearch] hit._id=%q != _source.chunk_id=%q (D12 invariant violation)",
|
|
h.ID, h.Source.ChunkID)
|
|
}
|
|
out = append(out, &types.IndexWithScore{
|
|
ID: h.ID,
|
|
ChunkID: h.Source.ChunkID,
|
|
KnowledgeID: h.Source.KnowledgeID,
|
|
KnowledgeBaseID: h.Source.KnowledgeBaseID,
|
|
SourceID: h.Source.SourceID,
|
|
SourceType: types.SourceType(h.Source.SourceType),
|
|
TagID: h.Source.TagID,
|
|
Content: h.Source.Content,
|
|
Score: h.Score,
|
|
MatchType: mt,
|
|
IsEnabled: h.Source.IsEnabled,
|
|
})
|
|
}
|
|
return []*types.RetrieveResult{{
|
|
Results: out,
|
|
RetrieverEngineType: types.OpenSearchRetrieverEngineType,
|
|
RetrieverType: rt,
|
|
}}
|
|
}
|