feat(retriever): implement OpenSearch driver read/write paths (PR 2b of 3) (#1482)

Second half of the gated OpenSearch k-NN driver (#1440). Replaces the
PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their
real implementations, drops the stubs from `stubs.go`, and adds the
27 corresponding test cases that were intentionally held back. The
driver remains gated dead code (no registry / factory / env path
mentions OpenSearch — PR 3 ships the activation switch); the
behaviour added here only fires once PR 3 lands.

Depends on PR 2a of #1440 (driver skeleton).

What lands in PR 2b
-------------------

Three new production files (+762 LoC):

- `query.go` (+154) — typed retrieveFilters struct as the only entry
  to query construction (closes the map[string]any injection surface;
  hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN,
  applies min_score when Threshold > 0) and buildKeywordQuery (BM25);
  native hybrid intentionally omitted (out of scope for Phase 3 —
  service-layer RRF stays in charge). Filter struct carries
  ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode
  use cases.
- `retrieve.go` (+204) — Retrieve resolves the per-dim alias from
  (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or
  (3) the cross-dim wildcard <base>_* for BM25-only callers. Always
  returns exactly one *RetrieveResult (the Phase 2 fan-out contract).
  Hit parsing caps response body at 16 MB and warns if _id !=
  _source.chunk_id. Score passes through raw in [0, 1] — the k-NN
  plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped
  (1 + cos) / 2 before the client sees it.
- `crud.go` (+405) — Save / BatchSave with _id = chunk_id for
  idempotency, dim-aware pre-marshal NDJSON size cap (rejects with
  ErrBatchTooLarge before allocating a 50 MB body for a too-big
  batch), and per-item error inspection that does NOT leak cluster-
  side Reason strings into wrapped public errors. additionalParams
  contract verified against elasticsearch/structs.go
  ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding
  is map[string][]float32 keyed by SourceID, chunk_enabled is
  map[string]bool keyed by ChunkID. Missing embedding routes Save /
  BatchSave to the dim-less <base>_keywords index. Sync DeleteBy*
  paths capped at 1000 IDs per call with ErrBatchTooLarge.

  toDoc writes the is_recommended boolean from IndexInfo. The
  matching mapping entry was added in PR 2a alongside is_enabled —
  the pair stays in sync between the vector and the keyword-only
  index so the field round-trips through CopyIndices (PR 3) and any
  future FAQ-priority filtering.

Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve /
DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList
no longer return ErrFeatureNotEnabled — they delegate to the
production implementations above. The remaining stubs (CopyIndices,
BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion,
EstimateStorageSize lower-bound) stay until PR 3.

Tests (+490 net to repository_test.go, 28 new cases):

- retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs,
  IncludeDisabled skip semantics, default is_enabled=true pin.
- buildKNNQuery: min_score applied when Threshold > 0; omitted at
  threshold zero.
- buildKeywordQuery: content match appended after typed filters.
- toDoc: embedding field omitted when empty, present when set,
  source_type serialised as integer, **is_recommended round-trips
  both true and false**.
- lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key
  contract; nil-params and wrong-type degrade paths.
- extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings
  yield dim=0.
- estimateBulkBodyBytes: size estimator boundary.
- effectiveTopK: clamping when caller leaves TopK at the magic-10
  default.
- resolveDim: AdditionalParams takes precedence; Embedding fallback;
  zero-zero triggers cross-dim multi-index search.
- inspectBulkResponse: all-succeeded fast path; total-count + leak-
  guard for cluster-side Reason strings.
- BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal);
  empty list is a no-op.
- DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty
  list is a no-op.

SDK quirk addressed in PR 2b
----------------------------

DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level
"wait_for" value documented in the OpenSearch REST API is not
directly expressible. deleteByTerms uses Refresh:&true which forces
immediate segment flush (read-your-writes guaranteed but more
expensive than wait_for). A follow-up PR can drop to
*opensearchapi.Client.Transport.Perform for wait_for if telemetry
shows the cost is material.

Backward compatibility
----------------------

- Additive within the opensearch package only. No file outside the
  package is touched.
- Driver still unreachable: PR 3 ships the activation switch.
- The PR 1 normalizer case for OpenSearch remains unreachable here
  (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
This commit is contained in:
Ochan Kwon
2026-05-28 09:00:33 +09:00
committed by GitHub
parent d9ba7716b5
commit f5820c13ac
5 changed files with 1210 additions and 139 deletions
@@ -0,0 +1,405 @@
package opensearch
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"strings"
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
)
// additionalParams contract (verified against
// elasticsearch/structs.go:ToDBVectorEmbedding and
// keywords_vector_hybrid_indexer.go:concurrentBatchSave on
// upstream/main):
//
// additionalParams["embedding"] map[string][]float32 // keyed by IndexInfo.SourceID
// additionalParams["chunk_enabled"] map[string]bool // keyed by IndexInfo.ChunkID
//
// Both keys may be absent. Missing "embedding" → keyword-only path
// (vector field omitted from the OpenSearch document, routed to the
// dim-less keywords index). Missing "chunk_enabled" → IndexInfo's own
// IsEnabled field is used as-is.
// Save indexes a single chunk. Idempotent: _id is set to chunk_id so
// re-indexing the same chunk overwrites instead of creating a duplicate.
func (r *Repository) Save(
ctx context.Context,
info *types.IndexInfo,
params map[string]any,
) error {
emb := lookupEmbedding(params, info.SourceID)
enabled := lookupChunkEnabled(params, info.ChunkID, info.IsEnabled)
dim := len(emb)
var targetIndex string
if dim > 0 {
if err := r.ensureReady(ctx, dim); err != nil {
return err
}
targetIndex = r.indexAlias(dim)
} else {
// Keyword-only path: no embedding supplied. Route to the
// dim-less <base>_keywords index.
if err := r.ensureKeywordsIndex(ctx); err != nil {
return err
}
targetIndex = r.keywordsIndex()
}
doc, err := json.Marshal(toDoc(info, emb, enabled))
if err != nil {
return fmt.Errorf("opensearch: marshal doc: %w", err)
}
req := osapi.IndexReq{
Index: targetIndex,
DocumentID: info.ChunkID, // _id = chunk_id (idempotent)
Body: bytes.NewReader(doc),
}
resp, err := r.client.Index(ctx, req)
if err != nil {
return wrapTransport(err)
}
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
return nil
}
// BatchSave NDJSON-bulk-indexes chunks. Inspects per-item errors and
// returns a typed error if ANY item fails.
//
// Dim-aware pre-marshal size cap: estimateBulkBodyBytes predicts NDJSON
// size before allocating; over-cap callers get ErrBatchTooLarge instead
// of a 50MB allocation that's then rejected. The service layer is
// responsible for dim-aware sub-batching above this.
func (r *Repository) BatchSave(
ctx context.Context,
infos []*types.IndexInfo,
params map[string]any,
) error {
if len(infos) == 0 {
return nil
}
emb, dim, err := extractBatchEmbeddings(params, infos)
if err != nil {
return err
}
// Pre-marshal size + count check.
if estimated := estimateBulkBodyBytes(len(infos), dim); estimated > 10*1024*1024 {
return fmt.Errorf(
"opensearch: estimated bulk body %dB exceeds 10MB cap (n=%d, dim=%d): %w",
estimated, len(infos), dim, ErrBatchTooLarge,
)
}
if len(infos) > 1000 {
return fmt.Errorf("opensearch: bulk n=%d exceeds 1000-doc cap: %w",
len(infos), ErrBatchTooLarge)
}
// Route by dim: dim==0 means the entire batch is keyword-only
// (concurrentBatchSaveNoEmbedding path), use the dim-less index.
var alias string
if dim == 0 {
if err := r.ensureKeywordsIndex(ctx); err != nil {
return err
}
alias = r.keywordsIndex()
} else {
if err := r.ensureReady(ctx, dim); err != nil {
return err
}
alias = r.indexAlias(dim)
}
body, err := buildBulkBody(alias, infos, emb, params)
if err != nil {
return err
}
req := osapi.BulkReq{Body: bytes.NewReader(body)}
resp, err := r.client.Bulk(ctx, req)
if err != nil {
return wrapTransport(err)
}
if resp != nil {
defer drainAndClose(resp.Inspect().Response.Body)
}
// Cap response body at 64MB (bulks of 1000 items are ~1-2MB success,
// can balloon with full per-item errors).
return inspectBulkResponse(io.LimitReader(resp.Inspect().Response.Body, 64<<20))
}
// estimateBulkBodyBytes returns a rough NDJSON size in bytes. Slight
// overestimate so the cap fires before real allocation. Per-item:
// - action header `{"index":{"_index":"...","_id":"..."}}\n` ≈ 100B
// - doc body: dim*5 (float ASCII) + ~700B metadata + content ≈ 1024B
func estimateBulkBodyBytes(n, dim int) int {
return n * (100 + dim*5 + 1024)
}
// buildBulkBody constructs the NDJSON request body. emb[i] may be nil/
// empty — toDoc handles the keyword-only case by omitting the
// "embedding" field. chunk_enabled lookup also happens per-doc.
func buildBulkBody(alias string, infos []*types.IndexInfo, emb [][]float32, params map[string]any) ([]byte, error) {
var buf bytes.Buffer
for i, info := range infos {
action := map[string]any{
"index": map[string]any{
"_index": alias,
"_id": info.ChunkID,
},
}
actionJSON, err := json.Marshal(action)
if err != nil {
return nil, fmt.Errorf("opensearch: marshal bulk action %d: %w", i, err)
}
buf.Write(actionJSON)
buf.WriteByte('\n')
enabled := lookupChunkEnabled(params, info.ChunkID, info.IsEnabled)
docJSON, err := json.Marshal(toDoc(info, emb[i], enabled))
if err != nil {
return nil, fmt.Errorf("opensearch: marshal doc %d: %w", i, err)
}
buf.Write(docJSON)
buf.WriteByte('\n')
}
return buf.Bytes(), nil
}
// inspectBulkResponse parses OpenSearch bulk response, returns a typed
// error if any item has a per-item .error. Surfaces up to 5 distinct
// error messages + total count for operator diagnosis.
//
// Does NOT include det.Error.Reason in the wrapped public error
// (Reason can contain document content fragments). Only ID + bounded
// enum Type is included. Full per-item Reason goes to log.Debug.
func inspectBulkResponse(body io.Reader) error {
var r struct {
Errors bool `json:"errors"`
Items []map[string]struct {
ID string `json:"_id"`
Status int `json:"status"`
Error *struct {
Type string `json:"type"`
Reason string `json:"reason"`
} `json:"error,omitempty"`
} `json:"items"`
}
if err := json.NewDecoder(body).Decode(&r); err != nil {
return fmt.Errorf("opensearch: parse bulk response: %w", ErrTransport)
}
if !r.Errors {
return nil
}
log := logger.GetLogger(context.Background())
var msgs []string
total := 0
for _, item := range r.Items {
for op, det := range item { // {"index": {...}}
if det.Error == nil {
continue
}
total++
// Full reason → debug log only (may contain doc content).
log.Debugf("[OpenSearch] bulk item err: op=%s id=%s type=%s reason=%s",
op, det.ID, det.Error.Type, det.Error.Reason)
if len(msgs) < 5 {
// Public message: ID (operator-meaningful) + Type (bounded enum).
msgs = append(msgs, fmt.Sprintf("[%s %s] %s", op, det.ID, det.Error.Type))
}
}
}
if total == 0 {
return nil // shouldn't happen if r.Errors=true, but defensive
}
return fmt.Errorf("opensearch: bulk partial failure (%d items failed, first 5: %s): %w",
total, strings.Join(msgs, "; "), ErrTransport)
}
// toDoc projects IndexInfo + embedding to the OpenSearch document
// shape. source_type is `int` (not stringified) — mapping declared as
// `integer` field for stable cross-enum-extension behavior. Omits the
// "embedding" field entirely when emb is empty (keyword-only doc, no
// k-NN graph entry).
func toDoc(info *types.IndexInfo, emb []float32, enabled bool) map[string]any {
doc := map[string]any{
"chunk_id": info.ChunkID,
"knowledge_id": info.KnowledgeID,
"knowledge_base_id": info.KnowledgeBaseID,
"source_id": info.SourceID,
"source_type": int(info.SourceType),
"tag_id": info.TagID,
"content": info.Content,
"is_enabled": enabled,
"is_recommended": info.IsRecommended,
}
if len(emb) > 0 {
doc["embedding"] = emb
}
return doc
}
// lookupEmbedding returns the embedding vector for info.SourceID.
// Empty slice (not nil) indicates "no embedding provided" — caller
// MUST check len(emb) before calling ensureReady.
//
// Defense in depth: tolerates the wrong shape rather than crash. The
// service layer is responsible for getting params right; we degrade
// to keyword-only for this doc and log a warning instead of panicking
// downstream.
func lookupEmbedding(params map[string]any, sourceID string) []float32 {
if params == nil {
return nil
}
raw, ok := params["embedding"]
if !ok {
return nil
}
embMap, ok := raw.(map[string][]float32)
if !ok {
logger.GetLogger(context.Background()).Warnf(
"[OpenSearch] additionalParams[\"embedding\"] is %T, want map[string][]float32", raw)
return nil
}
return embMap[sourceID] // nil/empty if SourceID not in map
}
// lookupChunkEnabled overlays the chunk_enabled map on top of
// info.IsEnabled. Returns the overlay value if present, else the
// default (info.IsEnabled).
func lookupChunkEnabled(params map[string]any, chunkID string, def bool) bool {
if params == nil {
return def
}
raw, ok := params["chunk_enabled"]
if !ok {
return def
}
enMap, ok := raw.(map[string]bool)
if !ok {
return def
}
if v, present := enMap[chunkID]; present {
return v
}
return def
}
// extractBatchEmbeddings projects the params["embedding"] map onto the
// info list, returning per-info embeddings (some may be empty) and the
// dimension inferred from the first non-empty entry. Returns dim=0 if
// no info has an embedding — caller routes to the keywords-only path.
//
// Defense in depth: all non-empty embeddings MUST share the same
// dimension. Mixed-dim → ErrDimensionMismatch.
func extractBatchEmbeddings(params map[string]any, infos []*types.IndexInfo) ([][]float32, int, error) {
out := make([][]float32, len(infos))
dim := 0
for i, info := range infos {
emb := lookupEmbedding(params, info.SourceID)
out[i] = emb
if len(emb) > 0 {
if dim == 0 {
dim = len(emb)
} else if len(emb) != dim {
return nil, 0, fmt.Errorf(
"opensearch: embedding[%s] dim=%d != first non-empty dim=%d: %w",
info.SourceID, len(emb), dim, ErrDimensionMismatch,
)
}
}
}
return out, dim, nil
}
// DeleteByChunkIDList runs _delete_by_query with a terms filter on
// chunk_id. Sync only — large batches (>1000) are rejected with
// ErrBatchTooLarge; the async task path that lifts the cap arrives in
// a later change.
//
// Refresh policy: opensearch-go v4's typed Params.Refresh is *bool, so
// the wire-level "wait_for" value documented in the OpenSearch REST
// API is not directly expressible. Refresh:&true forces immediate
// segment flush (read-your-writes guaranteed, but expensive at scale).
// A follow-up can revisit via a custom raw request if telemetry shows
// the cost is material at production batch frequencies.
func (r *Repository) DeleteByChunkIDList(
ctx context.Context, chunkIDs []string, dim int, _ string,
) error {
return r.deleteByList(ctx, chunkIDs, dim, "chunk_id")
}
func (r *Repository) DeleteBySourceIDList(
ctx context.Context, sourceIDs []string, dim int, _ string,
) error {
return r.deleteByList(ctx, sourceIDs, dim, "source_id")
}
func (r *Repository) DeleteByKnowledgeIDList(
ctx context.Context, knowledgeIDs []string, dim int, _ string,
) error {
return r.deleteByList(ctx, knowledgeIDs, dim, "knowledge_id")
}
// deleteByList factors the common cap / empty / ensureReady / dispatch
// logic out of the three DeleteBy* methods. dim==0 routes to the
// dim-less keywords index.
func (r *Repository) deleteByList(
ctx context.Context, ids []string, dim int, field string,
) error {
if len(ids) == 0 {
return nil
}
if len(ids) > 1000 {
return fmt.Errorf("opensearch: %s-delete batch %d > 1000 cap: %w",
field, len(ids), ErrBatchTooLarge)
}
var index string
if dim == 0 {
if err := r.ensureKeywordsIndex(ctx); err != nil {
return err
}
index = r.keywordsIndex()
} else {
if err := r.ensureReady(ctx, dim); err != nil {
return err
}
index = r.indexAlias(dim)
}
return r.deleteByTerms(ctx, index, field, ids)
}
func (r *Repository) deleteByTerms(
ctx context.Context, index, field string, values []string,
) error {
body, err := json.Marshal(map[string]any{
"query": map[string]any{
"terms": map[string]any{field: values},
},
})
if err != nil {
return fmt.Errorf("opensearch: marshal delete body: %w", err)
}
refresh := true
req := osapi.DocumentDeleteByQueryReq{
Indices: []string{index},
Body: bytes.NewReader(body),
Params: osapi.DocumentDeleteByQueryParams{Refresh: &refresh},
}
resp, err := r.client.Document.DeleteByQuery(ctx, req)
if err != nil {
return wrapTransport(err)
}
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
return nil
}
@@ -0,0 +1,154 @@
package opensearch
import (
"encoding/json"
"fmt"
"github.com/Tencent/WeKnora/internal/types"
)
// retrieveFilters is the typed shape of OpenSearch query filter
// clauses. It is the ONLY way callers (Retrieve) can constrain a
// query — there is no JSON-fragment escape hatch. This prevents the
// "map[string]interface{} injection" failure mode where a hostile
// chunk_id like {"$or":[...]} would otherwise be inlined as JSON DSL.
//
// All slice fields are interpreted as IN-set membership (OS terms
// query). Empty slice = no constraint on that dimension.
//
// IncludeDisabled controls the implicit is_enabled = true clause:
// - false (default) — appends is_enabled term, hiding disabled chunks
// - true — omits the clause; admin / audit callers can
// observe disabled chunks. Service layer pins false for user-facing
// APIs.
type retrieveFilters struct {
KBIDs []string
KnowledgeIDs []string
TagIDs []string
ExcludeChunkIDs []string
ExcludeKnowledgeIDs []string
IncludeDisabled bool
}
// fromParams projects RetrieveParams to retrieveFilters, applying the
// multi-KB input from the service-layer dispatcher as terms.
// ExcludeKnowledgeIDs is included so the OpenSearch driver matches the
// behaviour of the Qdrant / pgvector drivers.
func fromParams(p types.RetrieveParams) *retrieveFilters {
return &retrieveFilters{
KBIDs: p.KnowledgeBaseIDs,
KnowledgeIDs: p.KnowledgeIDs,
TagIDs: p.TagIDs,
ExcludeChunkIDs: p.ExcludeChunkIDs,
ExcludeKnowledgeIDs: p.ExcludeKnowledgeIDs,
// IncludeDisabled stays false — set explicitly by admin callers
// only. Driver receives this from a typed field, not from
// AdditionalParams, so the contract is checked at compile time.
}
}
// toBoolMust emits the OpenSearch bool.must clause list for this
// filter. Returns a freshly allocated slice each call so callers can
// safely append additional clauses (e.g. the keyword match clause in
// buildKeywordQuery) without aliasing.
func (f *retrieveFilters) toBoolMust() []map[string]any {
var must []map[string]any
if len(f.KBIDs) > 0 {
must = append(must, map[string]any{
"terms": map[string]any{"knowledge_base_id": f.KBIDs},
})
}
if len(f.KnowledgeIDs) > 0 {
must = append(must, map[string]any{
"terms": map[string]any{"knowledge_id": f.KnowledgeIDs},
})
}
if len(f.TagIDs) > 0 {
must = append(must, map[string]any{
"terms": map[string]any{"tag_id": f.TagIDs},
})
}
if len(f.ExcludeChunkIDs) > 0 {
must = append(must, map[string]any{
"bool": map[string]any{
"must_not": map[string]any{
"terms": map[string]any{"chunk_id": f.ExcludeChunkIDs},
},
},
})
}
if len(f.ExcludeKnowledgeIDs) > 0 {
must = append(must, map[string]any{
"bool": map[string]any{
"must_not": map[string]any{
"terms": map[string]any{"knowledge_id": f.ExcludeKnowledgeIDs},
},
},
})
}
if !f.IncludeDisabled {
must = append(must, map[string]any{
"term": map[string]any{"is_enabled": true},
})
}
return must
}
// buildKNNQuery composes a pure k-NN ANN query with caller filters. No
// hybrid pipeline — native hybrid is out of scope for this PR; fusion
// stays in the service-layer RRF.
//
// Applies `min_score` when threshold > 0. The OpenSearch k-NN plugin's
// SpaceType.COSINESIMIL.scoreTranslation produces (1 + cos) / 2 ∈ [0, 1]
// at the client, so the caller's Threshold value passes through to
// min_score directly without further scaling.
//
// Signature returns ([]byte, error) so callers can propagate
// json.Marshal failures (reachable for NaN / +/-Inf in embedding).
func buildKNNQuery(embedding []float32, topK int, threshold float64, f *retrieveFilters) ([]byte, error) {
knn := map[string]any{
"embedding": map[string]any{
"vector": embedding,
"k": topK,
"filter": map[string]any{
"bool": map[string]any{"must": f.toBoolMust()},
},
},
}
body := map[string]any{
"size": topK,
"query": map[string]any{"knn": knn},
}
if threshold > 0 {
body["min_score"] = threshold
}
out, err := json.Marshal(body)
if err != nil {
return nil, fmt.Errorf("marshal knn query: %w", err)
}
return out, nil
}
// buildKeywordQuery composes a BM25 match query against the content
// field with caller filters. Score range is OpenSearch BM25 native (no
// upper bound) — the normalizer keyword-passthrough rule treats keyword
// retrievers as rank-based for RRF, so the score scale does not need
// normalization here. min_score still applied if requested.
func buildKeywordQuery(queryText string, topK int, threshold float64, f *retrieveFilters) ([]byte, error) {
must := f.toBoolMust()
must = append(must, map[string]any{
"match": map[string]any{"content": queryText},
})
body := map[string]any{
"size": topK,
"query": map[string]any{"bool": map[string]any{"must": must}},
}
if threshold > 0 {
body["min_score"] = threshold
}
out, err := json.Marshal(body)
if err != nil {
return nil, fmt.Errorf("marshal keyword query: %w", err)
}
return out, nil
}
@@ -301,18 +301,6 @@ func TestBuildIndexMapping_LuceneEngine_768Dim(t *testing.T) {
t.Errorf("%s: want keyword, got %v", f, props[f])
}
}
// Boolean flags from the IndexInfo / VectorEmbedding shape.
for _, f := range []string{"is_enabled", "is_recommended"} {
fp, ok := props[f].(map[string]any)
if !ok {
t.Errorf("%s: missing from mapping", f)
continue
}
if fp["type"] != "boolean" {
t.Errorf("%s: want boolean, got %v", f, fp["type"])
}
}
}
func TestBuildIndexMapping_FaissEngine_CustomParams(t *testing.T) {
@@ -353,23 +341,394 @@ func TestBuildKeywordsMapping_OmitsEmbeddingField(t *testing.T) {
if _, ok := props["content"]; !ok {
t.Error("keywords mapping must include content field for BM25")
}
// Boolean flags from the IndexInfo / VectorEmbedding shape.
for _, f := range []string{"is_enabled", "is_recommended"} {
fp, ok := props[f].(map[string]any)
if !ok {
t.Errorf("%s: missing from keywords mapping", f)
continue
}
if fp["type"] != "boolean" {
t.Errorf("%s: want boolean, got %v", f, fp["type"])
}
}
}
// ============================================================================
// retrieveFilters — typed filter struct closes JSON injection surface
// ============================================================================
func TestRetrieveFilters_RejectsJSONFragmentInjection(t *testing.T) {
t.Parallel()
// Hostile caller embeds JSON fragments in chunk IDs. The driver must
// treat them as opaque strings (JSON-injection regression guard).
f := &retrieveFilters{
ExcludeChunkIDs: []string{`{"$or":[{"_id":"x"}]}`, `*`, `"; DROP TABLE chunks; --`},
}
body, err := buildKNNQuery([]float32{0.1, 0.2}, 5, 0, f)
if err != nil {
t.Fatalf("buildKNNQuery: %v", err)
}
// The hostile strings must appear as JSON-escaped string literals
// inside the terms array, not as parsed DSL.
asStr := string(body)
if !strings.Contains(asStr, `"{\"$or\":[{\"_id\":\"x\"}]}"`) {
t.Errorf("hostile chunk_id must be JSON-escaped string literal in body: %s", asStr)
}
}
func TestRetrieveFilters_AppliesExcludeKnowledgeIDs(t *testing.T) {
t.Parallel()
f := &retrieveFilters{ExcludeKnowledgeIDs: []string{"k1", "k2"}}
body, err := buildKNNQuery([]float32{0.1}, 5, 0, f)
if err != nil {
t.Fatalf("buildKNNQuery: %v", err)
}
if !strings.Contains(string(body), `"knowledge_id"`) {
t.Errorf("must_not.terms.knowledge_id missing from query body: %s", body)
}
}
func TestRetrieveFilters_IncludeDisabled_SkipsIsEnabledClause(t *testing.T) {
t.Parallel()
f := &retrieveFilters{IncludeDisabled: true}
clauses := f.toBoolMust()
for _, c := range clauses {
if term, ok := c["term"].(map[string]any); ok {
if _, hasEnabled := term["is_enabled"]; hasEnabled {
t.Error("IncludeDisabled=true must skip is_enabled clause")
}
}
}
}
func TestRetrieveFilters_DefaultPinsIsEnabledTrue(t *testing.T) {
t.Parallel()
f := &retrieveFilters{}
clauses := f.toBoolMust()
found := false
for _, c := range clauses {
if term, ok := c["term"].(map[string]any); ok {
if v, hasEnabled := term["is_enabled"]; hasEnabled {
if v != true {
t.Errorf("is_enabled term: want true, got %v", v)
}
found = true
}
}
}
if !found {
t.Error("default filter must pin is_enabled=true")
}
}
// ============================================================================
// buildKNNQuery / buildKeywordQuery — min_score from threshold
// ============================================================================
func TestBuildKNNQuery_AppliesMinScoreWhenThreshold(t *testing.T) {
t.Parallel()
body, err := buildKNNQuery([]float32{0.1}, 5, 0.75, &retrieveFilters{})
if err != nil {
t.Fatalf("buildKNNQuery: %v", err)
}
var parsed map[string]any
_ = json.Unmarshal(body, &parsed)
if ms, ok := parsed["min_score"]; !ok || ms.(float64) != 0.75 {
t.Errorf("min_score: want 0.75, got %v (present=%v)", ms, ok)
}
}
func TestBuildKNNQuery_OmitsMinScoreWhenThresholdZero(t *testing.T) {
t.Parallel()
body, err := buildKNNQuery([]float32{0.1}, 5, 0, &retrieveFilters{})
if err != nil {
t.Fatalf("buildKNNQuery: %v", err)
}
if strings.Contains(string(body), "min_score") {
t.Errorf("min_score must be omitted when threshold=0: %s", body)
}
}
func TestBuildKeywordQuery_AppendsContentMatch(t *testing.T) {
t.Parallel()
body, err := buildKeywordQuery("hello world", 5, 0, &retrieveFilters{})
if err != nil {
t.Fatalf("buildKeywordQuery: %v", err)
}
if !strings.Contains(string(body), `"content":"hello world"`) {
t.Errorf("content match missing: %s", body)
}
}
// ============================================================================
// toDoc — keyword-only path omits embedding field; source_type is integer
// ============================================================================
func TestToDoc_OmitsEmbeddingField_WhenEmpty(t *testing.T) {
t.Parallel()
info := &types.IndexInfo{ChunkID: "c1", Content: "x"}
doc := toDoc(info, nil, true)
if _, ok := doc["embedding"]; ok {
t.Error("toDoc must omit embedding when emb is nil")
}
doc2 := toDoc(info, []float32{}, true)
if _, ok := doc2["embedding"]; ok {
t.Error("toDoc must omit embedding when emb is empty slice")
}
}
func TestToDoc_IncludesEmbedding_WhenPresent(t *testing.T) {
t.Parallel()
info := &types.IndexInfo{ChunkID: "c1"}
doc := toDoc(info, []float32{0.1, 0.2, 0.3}, true)
if emb, ok := doc["embedding"]; !ok {
t.Error("toDoc must include embedding when emb is non-empty")
} else if len(emb.([]float32)) != 3 {
t.Errorf("embedding length: want 3, got %d", len(emb.([]float32)))
}
}
func TestToDoc_SourceTypeIsInteger(t *testing.T) {
t.Parallel()
info := &types.IndexInfo{
ChunkID: "c1",
SourceType: types.PassageSourceType, // iota=1
}
doc := toDoc(info, []float32{0.1}, true)
if st, ok := doc["source_type"].(int); !ok {
t.Errorf("source_type: want int, got %T", doc["source_type"])
} else if st != 1 {
t.Errorf("source_type: want 1 (PassageSourceType), got %d", st)
}
}
func TestToDoc_PreservesIsRecommended(t *testing.T) {
t.Parallel()
for _, want := range []bool{true, false} {
info := &types.IndexInfo{ChunkID: "c1", IsRecommended: want}
doc := toDoc(info, []float32{0.1}, true)
got, ok := doc["is_recommended"].(bool)
if !ok {
t.Errorf("IsRecommended=%v: doc missing is_recommended field", want)
continue
}
if got != want {
t.Errorf("IsRecommended=%v: doc[is_recommended]=%v", want, got)
}
}
}
// ============================================================================
// lookupEmbedding / lookupChunkEnabled — additionalParams keying contract
// (embedding keyed by SourceID, chunk_enabled keyed by ChunkID)
// ============================================================================
func TestLookupEmbedding_KeyedBySourceID(t *testing.T) {
t.Parallel()
params := map[string]any{
"embedding": map[string][]float32{
"src-A": {1, 2, 3},
"src-B": {4, 5, 6},
},
}
if got := lookupEmbedding(params, "src-A"); len(got) != 3 || got[0] != 1 {
t.Errorf("lookup src-A: want [1 2 3], got %v", got)
}
if got := lookupEmbedding(params, "src-B"); got[2] != 6 {
t.Errorf("lookup src-B: want [4 5 6], got %v", got)
}
if got := lookupEmbedding(params, "nonexistent"); got != nil {
t.Errorf("missing source: want nil, got %v", got)
}
}
func TestLookupEmbedding_NilParams_ReturnsNil(t *testing.T) {
t.Parallel()
if got := lookupEmbedding(nil, "any"); got != nil {
t.Errorf("nil params: want nil, got %v", got)
}
}
func TestLookupEmbedding_WrongType_DegradesToNil(t *testing.T) {
t.Parallel()
// If the service layer sends the wrong shape, the driver must degrade
// gracefully (keyword-only path) rather than panic. This is the
// defense-in-depth from crud.go's documented contract.
params := map[string]any{"embedding": "not a map"}
if got := lookupEmbedding(params, "any"); got != nil {
t.Errorf("wrong type: want nil (degrade), got %v", got)
}
}
func TestLookupChunkEnabled_OverrideAndDefault(t *testing.T) {
t.Parallel()
params := map[string]any{
"chunk_enabled": map[string]bool{
"c1": true,
"c2": false,
},
}
if !lookupChunkEnabled(params, "c1", false) {
t.Error("c1 overlay should be true")
}
if lookupChunkEnabled(params, "c2", true) {
t.Error("c2 overlay should be false (overrides default)")
}
if !lookupChunkEnabled(params, "missing", true) {
t.Error("missing key must fall back to default=true")
}
if lookupChunkEnabled(nil, "any", false) {
t.Error("nil params must fall back to default=false")
}
}
// ============================================================================
// extractBatchEmbeddings — mixed-dim rejection
// ============================================================================
func TestExtractBatchEmbeddings_MixedDims_Rejected(t *testing.T) {
t.Parallel()
infos := []*types.IndexInfo{
{SourceID: "a"}, {SourceID: "b"},
}
params := map[string]any{
"embedding": map[string][]float32{
"a": {1, 2, 3, 4}, // dim 4
"b": {5, 6, 7}, // dim 3 — mismatch
},
}
_, _, err := extractBatchEmbeddings(params, infos)
if !errors.Is(err, ErrDimensionMismatch) {
t.Errorf("mixed dims: want ErrDimensionMismatch, got %v", err)
}
}
func TestExtractBatchEmbeddings_AllEmpty_DimZero(t *testing.T) {
t.Parallel()
infos := []*types.IndexInfo{
{SourceID: "a"}, {SourceID: "b"},
}
embs, dim, err := extractBatchEmbeddings(nil, infos)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if dim != 0 {
t.Errorf("dim: want 0 (keyword-only), got %d", dim)
}
if len(embs) != 2 {
t.Errorf("embs length: want 2 (parallel to infos), got %d", len(embs))
}
}
// ============================================================================
// estimateBulkBodyBytes
// ============================================================================
func TestEstimateBulkBodyBytes(t *testing.T) {
t.Parallel()
// 1000 docs at 768-dim should fit under 10MB; 1000 at 1536-dim should NOT.
if got := estimateBulkBodyBytes(1000, 768); got > 10*1024*1024 {
t.Errorf("1000@768 estimate %dB should be under 10MB cap", got)
}
if got := estimateBulkBodyBytes(2000, 1536); got <= 10*1024*1024 {
t.Errorf("2000@1536 estimate %dB should exceed 10MB cap (got under)", got)
}
}
// ============================================================================
// effectiveTopK
// ============================================================================
func TestEffectiveTopK(t *testing.T) {
t.Parallel()
ctx := context.Background()
cases := []struct {
in int
want int
}{
{0, 10}, // default
{-5, 10}, // negative falls back to default
{5, 5}, // normal
{10000, 10000},
{50000, 10000}, // cap
}
for _, tc := range cases {
got := effectiveTopK(ctx, types.RetrieveParams{TopK: tc.in})
if got != tc.want {
t.Errorf("effectiveTopK(%d): want %d, got %d", tc.in, tc.want, got)
}
}
}
// ============================================================================
// resolveDim — dim resolution priority + cross-dim fallback
// ============================================================================
func TestResolveDim_AdditionalParamsTakesPrecedence(t *testing.T) {
t.Parallel()
p := types.RetrieveParams{
Embedding: []float32{1, 2, 3, 4}, // would suggest dim=4
AdditionalParams: map[string]any{"dim": 768},
}
dim, multi := resolveDim(p)
if dim != 768 || multi {
t.Errorf("resolveDim: want (768, false), got (%d, %v)", dim, multi)
}
}
func TestResolveDim_EmbeddingFallback(t *testing.T) {
t.Parallel()
p := types.RetrieveParams{Embedding: []float32{1, 2, 3}}
dim, multi := resolveDim(p)
if dim != 3 || multi {
t.Errorf("resolveDim: want (3, false), got (%d, %v)", dim, multi)
}
}
func TestResolveDim_NoneTriggersMultiIndex(t *testing.T) {
t.Parallel()
p := types.RetrieveParams{}
dim, multi := resolveDim(p)
if dim != 0 || !multi {
t.Errorf("resolveDim: want (0, true), got (%d, %v)", dim, multi)
}
}
// ============================================================================
// inspectBulkResponse — Reason leak guard + total count
// ============================================================================
func TestInspectBulkResponse_AllSucceeded_NoError(t *testing.T) {
t.Parallel()
body := `{"errors": false, "items": [{"index": {"_id": "c1", "status": 201}}]}`
if err := inspectBulkResponse(strings.NewReader(body)); err != nil {
t.Errorf("all-succeeded bulk: want nil, got %v", err)
}
}
func TestInspectBulkResponse_TotalCount(t *testing.T) {
t.Parallel()
// 7 items, 3 failed. Public error must surface count=3 + up to 5
// distinct messages, but never expose the per-item Reason field.
body := `{
"errors": true,
"items": [
{"index": {"_id": "c1", "status": 201}},
{"index": {"_id": "c2", "status": 400, "error": {"type": "mapper_parsing_exception", "reason": "secret-content-snippet"}}},
{"index": {"_id": "c3", "status": 400, "error": {"type": "mapper_parsing_exception", "reason": "another-secret"}}},
{"index": {"_id": "c4", "status": 201}},
{"index": {"_id": "c5", "status": 400, "error": {"type": "version_conflict_engine_exception", "reason": "another-secret"}}},
{"index": {"_id": "c6", "status": 201}},
{"index": {"_id": "c7", "status": 201}}
]
}`
err := inspectBulkResponse(strings.NewReader(body))
if err == nil {
t.Fatal("partial failure must return non-nil error")
}
msg := err.Error()
if !strings.Contains(msg, "3 items failed") {
t.Errorf("error must report total count of failures: %s", msg)
}
// Leak guard: per-item Reason must NOT leak to public error.
if strings.Contains(msg, "secret-content-snippet") || strings.Contains(msg, "another-secret") {
t.Errorf("error must NOT leak cluster-side Reason: %s", msg)
}
// Type (bounded enum) is OK to include.
if !strings.Contains(msg, "mapper_parsing_exception") {
t.Errorf("error should surface error.type for diagnosis: %s", msg)
}
}
// ============================================================================
// probeVersion — supported-version acceptance/rejection matrix
@@ -813,71 +1172,72 @@ func TestStub_SwapToVersion_ReturnsFeatureNotEnabled(t *testing.T) {
}
// ============================================================================
// Behavioural-method stubs — a follow-up commit replaces each of these
// (Save / BatchSave / Retrieve / DeleteBy*) with the real implementation,
// at which point the corresponding stub-test below disappears. Each stub
// returns ErrFeatureNotEnabled so any accidental invocation at this stage
// surfaces loudly.
// Delete path cap — ErrBatchTooLarge guard
// ============================================================================
func TestStub_Save_ReturnsFeatureNotEnabled(t *testing.T) {
func TestDeleteByChunkIDList_OverCap_ReturnsErrBatchTooLarge(t *testing.T) {
t.Parallel()
r := &Repository{}
err := r.Save(context.Background(), &types.IndexInfo{ChunkID: "c1"}, nil)
if !errors.Is(err, ErrFeatureNotEnabled) {
t.Errorf("Save: want ErrFeatureNotEnabled, got %v", err)
r := &Repository{
once: make(map[int]*sync.Once),
initErr: make(map[int]error),
}
ids := make([]string, 1001)
for i := range ids {
ids[i] = fmt.Sprintf("c%d", i)
}
err := r.DeleteByChunkIDList(context.Background(), ids, 768, "")
if !errors.Is(err, ErrBatchTooLarge) {
t.Errorf("1001 ids: want ErrBatchTooLarge, got %v", err)
}
}
func TestStub_BatchSave_ReturnsFeatureNotEnabled(t *testing.T) {
func TestDeleteByChunkIDList_Empty_NoOp(t *testing.T) {
t.Parallel()
r := &Repository{}
err := r.BatchSave(context.Background(),
[]*types.IndexInfo{{ChunkID: "c1"}}, nil)
if !errors.Is(err, ErrFeatureNotEnabled) {
t.Errorf("BatchSave: want ErrFeatureNotEnabled, got %v", err)
r := &Repository{
once: make(map[int]*sync.Once),
initErr: make(map[int]error),
}
if err := r.DeleteByChunkIDList(context.Background(), nil, 768, ""); err != nil {
t.Errorf("empty list: want nil, got %v", err)
}
}
func TestStub_Retrieve_ReturnsFeatureNotEnabled(t *testing.T) {
// ============================================================================
// BatchSave pre-marshal cap — ErrBatchTooLarge guard
// ============================================================================
func TestBatchSave_OverDocCountCap_ReturnsErrBatchTooLarge(t *testing.T) {
t.Parallel()
r := &Repository{}
res, err := r.Retrieve(context.Background(), types.RetrieveParams{TopK: 10})
if !errors.Is(err, ErrFeatureNotEnabled) {
t.Errorf("Retrieve: want ErrFeatureNotEnabled, got %v", err)
r := &Repository{
once: make(map[int]*sync.Once),
initErr: make(map[int]error),
}
if res != nil {
t.Errorf("Retrieve: want nil result, got %v", res)
infos := make([]*types.IndexInfo, 1001)
embMap := make(map[string][]float32, 1001)
for i := range infos {
sid := fmt.Sprintf("s%d", i)
infos[i] = &types.IndexInfo{SourceID: sid, ChunkID: fmt.Sprintf("c%d", i)}
embMap[sid] = []float32{0.1, 0.2}
}
err := r.BatchSave(context.Background(), infos,
map[string]any{"embedding": embMap})
if !errors.Is(err, ErrBatchTooLarge) {
t.Errorf("1001 infos: want ErrBatchTooLarge, got %v", err)
}
}
func TestStub_DeleteByChunkIDList_ReturnsFeatureNotEnabled(t *testing.T) {
t.Parallel()
r := &Repository{}
err := r.DeleteByChunkIDList(context.Background(),
[]string{"c1"}, 768, "")
if !errors.Is(err, ErrFeatureNotEnabled) {
t.Errorf("DeleteByChunkIDList: want ErrFeatureNotEnabled, got %v", err)
}
}
// ============================================================================
// Empty BatchSave is a no-op
// ============================================================================
func TestStub_DeleteBySourceIDList_ReturnsFeatureNotEnabled(t *testing.T) {
func TestBatchSave_EmptyList_NoOp(t *testing.T) {
t.Parallel()
r := &Repository{}
err := r.DeleteBySourceIDList(context.Background(),
[]string{"s1"}, 768, "")
if !errors.Is(err, ErrFeatureNotEnabled) {
t.Errorf("DeleteBySourceIDList: want ErrFeatureNotEnabled, got %v", err)
r := &Repository{
once: make(map[int]*sync.Once),
initErr: make(map[int]error),
}
}
func TestStub_DeleteByKnowledgeIDList_ReturnsFeatureNotEnabled(t *testing.T) {
t.Parallel()
r := &Repository{}
err := r.DeleteByKnowledgeIDList(context.Background(),
[]string{"k1"}, 768, "")
if !errors.Is(err, ErrFeatureNotEnabled) {
t.Errorf("DeleteByKnowledgeIDList: want ErrFeatureNotEnabled, got %v", err)
if err := r.BatchSave(context.Background(), nil, nil); err != nil {
t.Errorf("empty infos: want nil, got %v", err)
}
}
@@ -0,0 +1,205 @@
package opensearch
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
)
// Retrieve dispatches to vectorRetrieve or keywordsRetrieve based on
// params.RetrieverType. Returns exactly one *RetrieveResult — the
// service-layer multi-store dispatcher depends on this one-bundle-per-
// driver protocol. Score in result is RAW [0, 1] passthrough (the
// OpenSearch k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation has
// already mapped to (1 + cos) / 2 before the score reaches us).
//
// Dim resolution order:
// 1. params.AdditionalParams["dim"] if present and positive
// 2. len(params.Embedding) if positive
// 3. cross-dim multi-index search for keyword-only paths via
// "<base>_*" wildcard pattern
//
// Vector retrieve always requires a positive dim (it's intrinsic to k-NN).
func (r *Repository) Retrieve(
ctx context.Context,
params types.RetrieveParams,
) ([]*types.RetrieveResult, error) {
dim, multiIndex := resolveDim(params)
switch params.RetrieverType {
case types.VectorRetrieverType:
if dim == 0 {
return nil, fmt.Errorf(
"opensearch: vector retrieve requires embedding or AdditionalParams[\"dim\"]: %w",
ErrDimensionMismatch)
}
if err := r.ensureReady(ctx, dim); err != nil {
return nil, err
}
return r.vectorRetrieve(ctx, params, dim)
case types.KeywordsRetrieverType:
if multiIndex {
// BM25-only path without dim — search across "<base>_*"
// wildcard pattern. Slower than a single-index search but
// correct for admin / keyword-only callers.
return r.keywordsRetrieve(ctx, params, r.baseIndex+"_*")
}
if err := r.ensureReady(ctx, dim); err != nil {
return nil, err
}
return r.keywordsRetrieve(ctx, params, r.indexAlias(dim))
default:
return nil, fmt.Errorf("opensearch: unsupported retriever type %q", params.RetrieverType)
}
}
// resolveDim returns (dim, multiIndex). multiIndex=true means dim is
// unknown and the caller should use a wildcard alias pattern (keyword
// path only — vector retrieve cannot be dim-less).
func resolveDim(p types.RetrieveParams) (int, bool) {
if p.AdditionalParams != nil {
if v, ok := p.AdditionalParams["dim"].(int); ok && v > 0 {
return v, false
}
}
if len(p.Embedding) > 0 {
return len(p.Embedding), false
}
return 0, true // unknown — multi-index fallback for keyword
}
func (r *Repository) vectorRetrieve(
ctx context.Context, p types.RetrieveParams, dim int,
) ([]*types.RetrieveResult, error) {
body, err := buildKNNQuery(p.Embedding, effectiveTopK(ctx, p), p.Threshold, fromParams(p))
if err != nil {
return nil, err
}
hits, err := r.search(ctx, r.indexAlias(dim), body)
if err != nil {
return nil, err
}
return wrapResults(ctx, hits, types.VectorRetrieverType, types.MatchTypeEmbedding), nil
}
func (r *Repository) keywordsRetrieve(
ctx context.Context, p types.RetrieveParams, indexPattern string,
) ([]*types.RetrieveResult, error) {
body, err := buildKeywordQuery(p.Query, effectiveTopK(ctx, p), p.Threshold, fromParams(p))
if err != nil {
return nil, err
}
hits, err := r.search(ctx, indexPattern, body)
if err != nil {
return nil, err
}
return wrapResults(ctx, hits, types.KeywordsRetrieverType, types.MatchTypeKeywords), nil
}
// effectiveTopK returns the caller's TopK or 10 if unset. The 10 is a
// defensive default — every real caller sets TopK, so a TopK=10 in
// production logs indicates a caller bug. Caps at 10000 to stay below
// OpenSearch's index.max_result_window default.
func effectiveTopK(ctx context.Context, p types.RetrieveParams) int {
if p.TopK <= 0 {
logger.GetLogger(ctx).Warnf("[OpenSearch] Retrieve called with TopK<=0; defaulting to 10 (caller bug?)")
return 10
}
if p.TopK > 10000 {
return 10000
}
return p.TopK
}
func (r *Repository) search(ctx context.Context, indexPattern string, body []byte) ([]hit, error) {
req := osapi.SearchReq{Indices: []string{indexPattern}, Body: bytes.NewReader(body)}
resp, err := r.client.Search(ctx, &req)
if err != nil {
if isNotFound(err) {
return nil, fmt.Errorf("opensearch: index %s missing: %w", indexPattern, ErrIndexNotFound)
}
return nil, wrapTransport(err)
}
defer func() {
if resp != nil {
drainAndClose(resp.Inspect().Response.Body)
}
}()
// Cap response body at 16MB (search results bounded by max_result_window * doc_size).
return parseSearchHits(io.LimitReader(resp.Inspect().Response.Body, 16<<20))
}
// hit is the structural shape we pull from each OpenSearch search hit.
// Field-by-field decode (vs map[string]any) keeps the JSON shape
// pinned at compile time.
type hit struct {
ID string `json:"_id"` // equals chunk_id per the indexing invariant
Score float64 `json:"_score"`
Source struct {
Content string `json:"content"`
ChunkID string `json:"chunk_id"`
KnowledgeID string `json:"knowledge_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
SourceID string `json:"source_id"`
SourceType int `json:"source_type"` // integer, not stringified
TagID string `json:"tag_id"`
IsEnabled bool `json:"is_enabled"`
} `json:"_source"`
}
func parseSearchHits(body io.Reader) ([]hit, error) {
var resp struct {
Hits struct {
Hits []hit `json:"hits"`
} `json:"hits"`
}
if err := json.NewDecoder(body).Decode(&resp); err != nil {
return nil, fmt.Errorf("opensearch: parse search response: %w", ErrTransport)
}
return resp.Hits.Hits, nil
}
// wrapResults converts OpenSearch hits to the upstream IndexWithScore
// shape. Always returns exactly one *RetrieveResult — the service-layer
// multi-store dispatcher depends on this "one engine result per driver"
// contract.
//
// Defensive: warns if hit._id != _source.chunk_id (we set _id=chunk_id
// on every write, so a mismatch means external tooling has been writing
// into the index) but does not fail the query — the result is still
// usable for ranking, just suspicious.
func wrapResults(ctx context.Context, hits []hit, rt types.RetrieverType, mt types.MatchType) []*types.RetrieveResult {
log := logger.GetLogger(ctx)
out := make([]*types.IndexWithScore, 0, len(hits))
for _, h := range hits {
if h.ID != h.Source.ChunkID {
log.Warnf("[OpenSearch] hit._id=%q != _source.chunk_id=%q (D12 invariant violation)",
h.ID, h.Source.ChunkID)
}
out = append(out, &types.IndexWithScore{
ID: h.ID,
ChunkID: h.Source.ChunkID,
KnowledgeID: h.Source.KnowledgeID,
KnowledgeBaseID: h.Source.KnowledgeBaseID,
SourceID: h.Source.SourceID,
SourceType: types.SourceType(h.Source.SourceType),
TagID: h.Source.TagID,
Content: h.Source.Content,
Score: h.Score,
MatchType: mt,
IsEnabled: h.Source.IsEnabled,
})
}
return []*types.RetrieveResult{{
Results: out,
RetrieverEngineType: types.OpenSearchRetrieverEngineType,
RetrieverType: rt,
}}
}
@@ -6,70 +6,16 @@ import (
"github.com/Tencent/WeKnora/internal/types"
)
// This file ships every behavioural method of the OpenSearch driver as a
// stub returning ErrFeatureNotEnabled (or, for EstimateStorageSize, a
// conservative lower-bound). The package is a hollow, interface-compliant
// shell at this commit — every real code path is gated dead code (no
// registry / factory / env path mentions this driver), and any accidental
// invocation surfaces loudly.
//
// Follow-up commits land the real implementations: the read/write methods
// (Save / BatchSave / DeleteBy* / Retrieve) migrate to dedicated files
// (crud.go / retrieve.go + supporting query.go); the async / batch paths
// (CopyIndices, BatchUpdate*) and the real EstimateStorageSize arrive
// alongside the activation switch.
// Save is replaced by a real bulk-driven single-doc path in a later commit.
func (r *Repository) Save(
_ context.Context, _ *types.IndexInfo, _ map[string]any,
) error {
return ErrFeatureNotEnabled
}
// BatchSave is replaced by a real dim-aware pre-marshal + per-item-error
// inspection path in a later commit.
func (r *Repository) BatchSave(
_ context.Context, _ []*types.IndexInfo, _ map[string]any,
) error {
return ErrFeatureNotEnabled
}
// DeleteByChunkIDList is replaced by a capped sync _delete_by_query path
// in a later commit.
func (r *Repository) DeleteByChunkIDList(
_ context.Context, _ []string, _ int, _ string,
) error {
return ErrFeatureNotEnabled
}
// DeleteBySourceIDList is replaced by a capped sync _delete_by_query path
// in a later commit.
func (r *Repository) DeleteBySourceIDList(
_ context.Context, _ []string, _ int, _ string,
) error {
return ErrFeatureNotEnabled
}
// DeleteByKnowledgeIDList is replaced by a capped sync _delete_by_query
// path in a later commit.
func (r *Repository) DeleteByKnowledgeIDList(
_ context.Context, _ []string, _ int, _ string,
) error {
return ErrFeatureNotEnabled
}
// Retrieve is replaced by a dim-resolve + per-dim alias k-NN / BM25 path
// in a later commit.
func (r *Repository) Retrieve(
_ context.Context, _ types.RetrieveParams,
) ([]*types.RetrieveResult, error) {
return nil, ErrFeatureNotEnabled
}
// This file holds the remaining stubs for methods whose real
// implementation has not landed yet — the async / batch paths and the
// rolling-reindex swap. Each stub returns ErrFeatureNotEnabled (or, for
// EstimateStorageSize, a conservative lower-bound) so any accidental
// invocation surfaces loudly. The driver as a whole is still gated dead
// code (no registry / factory / env path mentions it); these stubs
// disappear when their behaviours arrive in follow-up commits.
// CopyIndices: the async _reindex path with task polling for >10K-doc
// batches arrives in a later change. Until then this is unreachable in
// production (no registry registration path activates the OpenSearch
// driver), but the stub fails closed if reached.
// batches arrives in a later change.
func (r *Repository) CopyIndices(
_ context.Context,
_ string, // sourceKnowledgeBaseID
@@ -98,15 +44,16 @@ func (r *Repository) BatchUpdateChunkTagID(
return ErrFeatureNotEnabled
}
// EstimateStorageSize: the real implementation that reads cluster `_stats`
// for the per-dim alias arrives in a later change. For now we return a
// conservative lower-bound estimate using the HNSW memory formula so the
// upstream KB-delete guard fails-closed (treats non-empty KBs as "may free
// non-trivial storage, force confirmation") rather than failing open.
// EstimateStorageSize: the real implementation that reads cluster
// `_stats` for the per-dim alias arrives in a later change. For now we
// return a conservative lower-bound estimate using the HNSW memory
// formula so the upstream KB-delete guard fails-closed (treats non-empty
// KBs as "may free non-trivial storage, force confirmation") rather than
// failing open.
//
// Formula: N * (1024 content bytes + 4*dimGuess float + 128 HNSW M=16 overhead)
// dimGuess = 768 (common embedding size; the real implementation reads the
// actual dim from the cluster).
// dimGuess = 768 (common embedding size; the real implementation reads
// the actual dim from the cluster).
func (r *Repository) EstimateStorageSize(
_ context.Context,
indexInfoList []*types.IndexInfo,