f5820c13ac
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their real implementations, drops the stubs from `stubs.go`, and adds the 27 corresponding test cases that were intentionally held back. The driver remains gated dead code (no registry / factory / env path mentions OpenSearch — PR 3 ships the activation switch); the behaviour added here only fires once PR 3 lands. Depends on PR 2a of #1440 (driver skeleton). What lands in PR 2b ------------------- Three new production files (+762 LoC): - `query.go` (+154) — typed retrieveFilters struct as the only entry to query construction (closes the map[string]any injection surface; hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN, applies min_score when Threshold > 0) and buildKeywordQuery (BM25); native hybrid intentionally omitted (out of scope for Phase 3 — service-layer RRF stays in charge). Filter struct carries ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode use cases. - `retrieve.go` (+204) — Retrieve resolves the per-dim alias from (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or (3) the cross-dim wildcard <base>_* for BM25-only callers. Always returns exactly one *RetrieveResult (the Phase 2 fan-out contract). Hit parsing caps response body at 16 MB and warns if _id != _source.chunk_id. Score passes through raw in [0, 1] — the k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped (1 + cos) / 2 before the client sees it. - `crud.go` (+405) — Save / BatchSave with _id = chunk_id for idempotency, dim-aware pre-marshal NDJSON size cap (rejects with ErrBatchTooLarge before allocating a 50 MB body for a too-big batch), and per-item error inspection that does NOT leak cluster- side Reason strings into wrapped public errors. additionalParams contract verified against elasticsearch/structs.go ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding is map[string][]float32 keyed by SourceID, chunk_enabled is map[string]bool keyed by ChunkID. Missing embedding routes Save / BatchSave to the dim-less <base>_keywords index. Sync DeleteBy* paths capped at 1000 IDs per call with ErrBatchTooLarge. toDoc writes the is_recommended boolean from IndexInfo. The matching mapping entry was added in PR 2a alongside is_enabled — the pair stays in sync between the vector and the keyword-only index so the field round-trips through CopyIndices (PR 3) and any future FAQ-priority filtering. Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve / DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList no longer return ErrFeatureNotEnabled — they delegate to the production implementations above. The remaining stubs (CopyIndices, BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion, EstimateStorageSize lower-bound) stay until PR 3. Tests (+490 net to repository_test.go, 28 new cases): - retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs, IncludeDisabled skip semantics, default is_enabled=true pin. - buildKNNQuery: min_score applied when Threshold > 0; omitted at threshold zero. - buildKeywordQuery: content match appended after typed filters. - toDoc: embedding field omitted when empty, present when set, source_type serialised as integer, **is_recommended round-trips both true and false**. - lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key contract; nil-params and wrong-type degrade paths. - extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings yield dim=0. - estimateBulkBodyBytes: size estimator boundary. - effectiveTopK: clamping when caller leaves TopK at the magic-10 default. - resolveDim: AdditionalParams takes precedence; Embedding fallback; zero-zero triggers cross-dim multi-index search. - inspectBulkResponse: all-succeeded fast path; total-count + leak- guard for cluster-side Reason strings. - BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal); empty list is a no-op. - DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty list is a no-op. SDK quirk addressed in PR 2b ---------------------------- DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level "wait_for" value documented in the OpenSearch REST API is not directly expressible. deleteByTerms uses Refresh:&true which forces immediate segment flush (read-your-writes guaranteed but more expensive than wait_for). A follow-up PR can drop to *opensearchapi.Client.Transport.Perform for wait_for if telemetry shows the cost is material. Backward compatibility ---------------------- - Additive within the opensearch package only. No file outside the package is touched. - Driver still unreachable: PR 3 ships the activation switch. - The PR 1 normalizer case for OpenSearch remains unreachable here (no driver instance produces a result yet). Test plan --------- - [x] go build ./... clean - [x] go vet ./... clean - [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes - [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/ shows only PR 1's normalizer case + this driver's EngineType() and tests — no activation path. - [x] grep -r "case \"opensearch\"" internal/ shows no hits.
155 lines
5.0 KiB
Go
155 lines
5.0 KiB
Go
package opensearch
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
)
|
|
|
|
// retrieveFilters is the typed shape of OpenSearch query filter
|
|
// clauses. It is the ONLY way callers (Retrieve) can constrain a
|
|
// query — there is no JSON-fragment escape hatch. This prevents the
|
|
// "map[string]interface{} injection" failure mode where a hostile
|
|
// chunk_id like {"$or":[...]} would otherwise be inlined as JSON DSL.
|
|
//
|
|
// All slice fields are interpreted as IN-set membership (OS terms
|
|
// query). Empty slice = no constraint on that dimension.
|
|
//
|
|
// IncludeDisabled controls the implicit is_enabled = true clause:
|
|
// - false (default) — appends is_enabled term, hiding disabled chunks
|
|
// - true — omits the clause; admin / audit callers can
|
|
// observe disabled chunks. Service layer pins false for user-facing
|
|
// APIs.
|
|
type retrieveFilters struct {
|
|
KBIDs []string
|
|
KnowledgeIDs []string
|
|
TagIDs []string
|
|
ExcludeChunkIDs []string
|
|
ExcludeKnowledgeIDs []string
|
|
IncludeDisabled bool
|
|
}
|
|
|
|
// fromParams projects RetrieveParams to retrieveFilters, applying the
|
|
// multi-KB input from the service-layer dispatcher as terms.
|
|
// ExcludeKnowledgeIDs is included so the OpenSearch driver matches the
|
|
// behaviour of the Qdrant / pgvector drivers.
|
|
func fromParams(p types.RetrieveParams) *retrieveFilters {
|
|
return &retrieveFilters{
|
|
KBIDs: p.KnowledgeBaseIDs,
|
|
KnowledgeIDs: p.KnowledgeIDs,
|
|
TagIDs: p.TagIDs,
|
|
ExcludeChunkIDs: p.ExcludeChunkIDs,
|
|
ExcludeKnowledgeIDs: p.ExcludeKnowledgeIDs,
|
|
// IncludeDisabled stays false — set explicitly by admin callers
|
|
// only. Driver receives this from a typed field, not from
|
|
// AdditionalParams, so the contract is checked at compile time.
|
|
}
|
|
}
|
|
|
|
// toBoolMust emits the OpenSearch bool.must clause list for this
|
|
// filter. Returns a freshly allocated slice each call so callers can
|
|
// safely append additional clauses (e.g. the keyword match clause in
|
|
// buildKeywordQuery) without aliasing.
|
|
func (f *retrieveFilters) toBoolMust() []map[string]any {
|
|
var must []map[string]any
|
|
if len(f.KBIDs) > 0 {
|
|
must = append(must, map[string]any{
|
|
"terms": map[string]any{"knowledge_base_id": f.KBIDs},
|
|
})
|
|
}
|
|
if len(f.KnowledgeIDs) > 0 {
|
|
must = append(must, map[string]any{
|
|
"terms": map[string]any{"knowledge_id": f.KnowledgeIDs},
|
|
})
|
|
}
|
|
if len(f.TagIDs) > 0 {
|
|
must = append(must, map[string]any{
|
|
"terms": map[string]any{"tag_id": f.TagIDs},
|
|
})
|
|
}
|
|
if len(f.ExcludeChunkIDs) > 0 {
|
|
must = append(must, map[string]any{
|
|
"bool": map[string]any{
|
|
"must_not": map[string]any{
|
|
"terms": map[string]any{"chunk_id": f.ExcludeChunkIDs},
|
|
},
|
|
},
|
|
})
|
|
}
|
|
if len(f.ExcludeKnowledgeIDs) > 0 {
|
|
must = append(must, map[string]any{
|
|
"bool": map[string]any{
|
|
"must_not": map[string]any{
|
|
"terms": map[string]any{"knowledge_id": f.ExcludeKnowledgeIDs},
|
|
},
|
|
},
|
|
})
|
|
}
|
|
if !f.IncludeDisabled {
|
|
must = append(must, map[string]any{
|
|
"term": map[string]any{"is_enabled": true},
|
|
})
|
|
}
|
|
return must
|
|
}
|
|
|
|
// buildKNNQuery composes a pure k-NN ANN query with caller filters. No
|
|
// hybrid pipeline — native hybrid is out of scope for this PR; fusion
|
|
// stays in the service-layer RRF.
|
|
//
|
|
// Applies `min_score` when threshold > 0. The OpenSearch k-NN plugin's
|
|
// SpaceType.COSINESIMIL.scoreTranslation produces (1 + cos) / 2 ∈ [0, 1]
|
|
// at the client, so the caller's Threshold value passes through to
|
|
// min_score directly without further scaling.
|
|
//
|
|
// Signature returns ([]byte, error) so callers can propagate
|
|
// json.Marshal failures (reachable for NaN / +/-Inf in embedding).
|
|
func buildKNNQuery(embedding []float32, topK int, threshold float64, f *retrieveFilters) ([]byte, error) {
|
|
knn := map[string]any{
|
|
"embedding": map[string]any{
|
|
"vector": embedding,
|
|
"k": topK,
|
|
"filter": map[string]any{
|
|
"bool": map[string]any{"must": f.toBoolMust()},
|
|
},
|
|
},
|
|
}
|
|
body := map[string]any{
|
|
"size": topK,
|
|
"query": map[string]any{"knn": knn},
|
|
}
|
|
if threshold > 0 {
|
|
body["min_score"] = threshold
|
|
}
|
|
out, err := json.Marshal(body)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("marshal knn query: %w", err)
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
// buildKeywordQuery composes a BM25 match query against the content
|
|
// field with caller filters. Score range is OpenSearch BM25 native (no
|
|
// upper bound) — the normalizer keyword-passthrough rule treats keyword
|
|
// retrievers as rank-based for RRF, so the score scale does not need
|
|
// normalization here. min_score still applied if requested.
|
|
func buildKeywordQuery(queryText string, topK int, threshold float64, f *retrieveFilters) ([]byte, error) {
|
|
must := f.toBoolMust()
|
|
must = append(must, map[string]any{
|
|
"match": map[string]any{"content": queryText},
|
|
})
|
|
body := map[string]any{
|
|
"size": topK,
|
|
"query": map[string]any{"bool": map[string]any{"must": must}},
|
|
}
|
|
if threshold > 0 {
|
|
body["min_score"] = threshold
|
|
}
|
|
out, err := json.Marshal(body)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("marshal keyword query: %w", err)
|
|
}
|
|
return out, nil
|
|
}
|