From 744a367f16be14b42684e35805c2f297eacd09d8 Mon Sep 17 00:00:00 2001 From: "ochan.kwon" Date: Fri, 22 May 2026 15:19:16 +0900 Subject: [PATCH] feat(retriever): add OpenSearch type prep ahead of Phase 3 driver MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lays type-system groundwork for the upcoming OpenSearch k-NN driver (Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate: this PR ships only inert constants, schema extensions, and an unreachable normalizer case. No path activates an OpenSearch VectorStore yet — creation continues to fail with "not a valid engine type" until the activation switch lands in Phase 3 PR 3. Changes: - OpenSearchRetrieverEngineType constant ("opensearch") in internal/types/retriever.go. Not added to validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping yet (gated). - ConnectionConfig.InsecureSkipVerify (bool, default false) in internal/types/vectorstore.go, placed inside the // Common section because it is a cross-driver TLS option. Distinct from the Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify only controls verification of an already-TLS HTTPS connection. AES-GCM Value/Scan round-trips the field as plaintext (it travels alongside the encrypted Password / APIKey but is not sensitive itself). - VectorStoreFieldInfo gains four optional fields: Immutable (bool), Min/Max (*float64), Enum ([]string). All omitempty so existing UI schema entries serialize identically. The fields will drive the UI in Phase 3 PR 3 (read-only on Edit + range/enum constraints). - Six new AuditAction constants under vector_store.* and opensearch.* namespaces. Definitions only — emission lands in Phase 3 PR 3. - EngineAwareNormalizer.Normalize is restructured to group engines by the effective score range observed by the normalizer (not the theoretical raw cosine range): Range [-1, 1] (raw cosine, mapped via (score + 1) / 2): - Milvus (COSINE metric mode). Milvus docs explicitly state the COSINE metric range is [-1, 1]. Range [0, 1] (passthrough — already on the target scale): - Elasticsearch v8 / ElasticFaiss. The driver issues a cosineSimilarity(...) script_score script, and Lucene rejects negative final scores ("Final relevance scores from the script_score query cannot be negative" — ES docs); the effective range observed by the normalizer is therefore [0, 1] for IR-normalized embeddings. ES was previously grouped with Milvus and over-corrected via (score + 1) / 2, which inflated every ES result by 50% in mixed-engine RRF fusion. - OpenSearch. The k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation maps the underlying Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1] before the score reaches us. Source: github.com/opensearch-project/k-NN at src/main/java/org/opensearch/knn/index/SpaceType.java (COSINESIMIL enum, scoreTranslation method). - Weaviate. The driver requests `certainty`, defined by Weaviate as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1]. - Postgres pgvector / SQLite sqlite-vec / Qdrant / TencentVectorDB / Doris. The driver computes (1 - cosine_distance) or normalized inner_product, whose theoretical range is [-1, 1]. The IR-normalized positive-component unit vectors WeKnora targets (BGE / OpenAI text-embedding-3 / Cohere / sentence-transformers) keep the observed range in [0, 1]; negative-cosine embedding models would silently clamp to 0 downstream — explicitly documented as the IR-normalization caveat in the struct godoc. Dead enum references (ElasticFaiss, Infinity) are flagged in the godoc with a pointer to internal/types/vectorstore.go's existing "legacy/experimental, no standalone deployable instance" annotation. Their case labels are kept for switch exhaustiveness. Test coverage: - OpenSearch constant wire value + collision check against existing 10 engines + gated invariant (NOT in validEngineTypes). - ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON field deserializes as false) + round-trip + AES-GCM coexistence with encrypted Password / APIKey. - VectorStoreFieldInfo omitempty preservation + new-field serialization + *float64 pointer distinction (min=0 vs nil). - AuditAction dot-namespace convention enforcement, prefix invariants for vector_store.* and opensearch.*, no wire-string collisions, exact wire-value pins for the six new constants. - EngineAwareNormalizer: - CosineRange retains its (score + 1) / 2 coverage but now only for Milvus. - UnitInterval now covers the full passthrough group (ES / ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant / Infinity / TencentVectorDB / Doris). - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an explicit regression guard for the score-range correction: cos=0.5 maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed). - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift (1.0001) + defensive negative + ±Inf / NaN edges + keyword passthrough + nil-ctx safety. Backward compatibility is preserved at every layer: - All new struct fields are omitempty / pointer-tagged so existing rows and existing wire formats remain unchanged. - Normalizer's new OpenSearch case is unreachable until the driver lands. The ES regrouping changes the post-normalization value for every ES vector search result (a correctness fix, not a feature) — ES vector retrieval is currently the only production path affected. - AuditAction constants emit no audit_log rows in this PR. - engine_type=opensearch VectorStore creation still rejected (gated). --- .../service/retriever/normalizer.go | 103 +++++- .../service/retriever/normalizer_test.go | 152 ++++++++- internal/types/audit_log.go | 39 +++ internal/types/audit_log_test.go | 135 ++++++++ internal/types/retriever.go | 9 + internal/types/vectorstore.go | 34 +- internal/types/vectorstore_test.go | 314 +++++++++++++++++- 7 files changed, 762 insertions(+), 24 deletions(-) create mode 100644 internal/types/audit_log_test.go diff --git a/internal/application/service/retriever/normalizer.go b/internal/application/service/retriever/normalizer.go index 59088c7e..e1c3f58c 100644 --- a/internal/application/service/retriever/normalizer.go +++ b/internal/application/service/retriever/normalizer.go @@ -31,14 +31,73 @@ type ScoreNormalizer interface { // are semantically comparable across engines. // // Source formulas (verified against repository implementations on -// upstream/main 3214e3d9): +// upstream/main 3214e3d9, OpenSearch 2.17 / 3.5 docs, the k-NN plugin's +// SpaceType.java source, the Tencent VectorDB Go SDK base_document.go, +// pgvector / sqlite-vec / Qdrant / Weaviate docs, and the Lucene +// script_score non-negative invariant). Engines are grouped by the +// effective score range observed at the normalizer's input: // -// - Elasticsearch v8 / ElasticFaiss / Milvus (cosine mode): -// raw cosine in [-1, 1] -// - Postgres pgvector: (1 - cosine_distance) in [0, 1] -// - SQLite sqlite-vec: (1 - cosine_distance) in [0, 1] -// - Weaviate: raw score in [0, 1] -// - Qdrant / Infinity / TencentVectorDB / Doris: raw score in [0, 1] +// Range [-1, 1] (raw cosine, normalized via (score + 1) / 2): +// - Milvus (COSINE metric mode). Milvus docs explicitly state the +// COSINE metric "has a range of [-1, 1]". +// +// Range [0, 1] (passthrough — already on the target scale): +// - Elasticsearch v8 — driver issues `cosineSimilarity(query, 'embedding')` +// as a script_score script. Lucene rejects negative final scores +// ("Final relevance scores from the script_score query cannot be +// negative") so the effective range observed here is [0, 1] for +// IR-normalized embeddings (see IR-normalization note below). If +// a future driver swap inserts `1.0 + cosineSimilarity(...)` (the +// pattern recommended by ES docs) this group will need to move out. +// - OpenSearch — the k-NN plugin's SpaceType.COSINESIMIL.scoreTranslation +// applies Math.max((2.0F - rawScore) / 2.0F, 0.0F) where rawScore is +// the distance (1 - cosine) from Lucene/Faiss, so the score returned +// to the client is (1 + cosine) / 2 ∈ [0, 1]. The OpenSearch driver +// itself ships in a later Phase 3 PR (see #1440); this branch is +// unreachable from production retrieval paths until then, but the +// unit tests pin the formula in advance. Source: +// github.com/opensearch-project/k-NN at +// src/main/java/org/opensearch/knn/index/SpaceType.java (COSINESIMIL +// enum, scoreTranslation method). +// - Weaviate — driver requests `certainty`, defined by Weaviate as +// (2 - distance) / 2 = (1 + cosine) / 2, intrinsically ∈ [0, 1]. +// - Postgres pgvector — driver computes `(1 - distance) as score` +// where `<=>` is cosine distance ∈ [0, 2]; the theoretical range +// is therefore [-1, 1] but IR-normalized embeddings (see below) +// keep the observed range in [0, 1]. +// - SQLite sqlite-vec — driver computes `1 - v.Distance` where the +// vec0 `distance_metric=cosine` returns distance ∈ [0, 2] (per +// sqlite-vec docs: "0 for identical, 2 for opposite"). Same IR +// caveat as pgvector. +// - Qdrant — Distance.Cosine; Qdrant normalizes vectors at insert +// time and the score returned by ScoredPoint.score is the dot +// product of the normalized vectors = raw cosine ∈ [-1, 1] in +// theory, [0, 1] under the IR caveat. +// - TencentVectorDB — MetricType COSINE; per the SDK godoc +// (tcvectordb base_document.go): "COSINE: return when score >= +// radius, value range [-1, 1]." Same IR caveat. +// - Doris — driver runs `inner_product_approximate` against +// L2-normalized embeddings (or legacy `(1 - cosine_distance_approximate)`), +// which equals raw cosine ∈ [-1, 1]. Same IR caveat. +// +// IR-normalization caveat (applies to pgvector, sqlite-vec, Qdrant, +// TencentVectorDB, Doris): modern RAG embeddings are L2-normalized +// positive-component unit vectors (sentence-transformers, BGE, OpenAI +// text-embedding-3, Cohere, E5, etc.) that empirically keep cosine in +// [0, 1]. Theoretical [-1, 1] inputs would clamp to 0 below — silent +// information loss only for embedding models that produce negative +// cosines (rare in IR workloads but possible for centered word2vec, +// GloVe, or non-IR embeddings). If a future use case violates this, +// the affected engines should move into the [-1, 1] group. +// +// Dead enum references (kept as RetrieverEngineType constants but +// without a driver implementation — see internal/types/vectorstore.go +// near the GetVectorStoreTypes definition, where they are flagged +// "legacy/experimental, no standalone deployable instance"): +// - InfinityRetrieverEngineType +// - ElasticFaissRetrieverEngineType +// Their case labels below route to clamp01(score) defensively, but +// production code never returns these engine types. // // Unknown engines clamp to [0, 1]; the fan-out caller emits a single WARN // per request via warnIfUnknownEngine so Normalize itself stays lock-free @@ -62,21 +121,35 @@ func (EngineAwareNormalizer) Normalize( } switch engineType { + case types.MilvusRetrieverEngineType: + // Raw cosine in [-1, 1] → [0, 1]. Milvus is the only engine in + // this codebase whose driver surfaces the raw cosine signed range + // to the normalizer. Pass through clamp01 once more so that a + // misbehaving engine returning 1.0000002 does not leak past the + // [0, 1] envelope (caller sorts by score afterwards). + return clamp01((score + 1) / 2) case types.ElasticsearchRetrieverEngineType, types.ElasticFaissRetrieverEngineType, - types.MilvusRetrieverEngineType: - // Raw cosine in [-1, 1] → [0, 1]. Pass through clamp01 once more so - // that a misbehaving engine returning 1.0000002 does not leak past - // the [0, 1] envelope (caller sorts by score afterwards). - return clamp01((score + 1) / 2) - case types.PostgresRetrieverEngineType, - types.QdrantRetrieverEngineType, + types.OpenSearchRetrieverEngineType, types.WeaviateRetrieverEngineType, + types.PostgresRetrieverEngineType, types.SQLiteRetrieverEngineType, + types.QdrantRetrieverEngineType, types.InfinityRetrieverEngineType, types.TencentVectorDBRetrieverEngineType, types.DorisRetrieverEngineType: - // Already in [0, 1] by repository contract. + // Already in [0, 1] when the value reaches us. See struct godoc + // above for the per-engine derivation: Elasticsearch's Lucene + // script_score non-negative invariant; OpenSearch's k-NN plugin + // SpaceType.COSINESIMIL.scoreTranslation pre-translation; + // Weaviate's certainty intrinsic; and the IR-normalization + // caveat covering pgvector / sqlite-vec / Qdrant / + // TencentVectorDB / Doris (theoretical [-1, 1] but observed + // [0, 1] for L2-normalized positive-component IR embeddings). + // + // ElasticFaiss and Infinity are dead enum references — their + // case labels are kept so the switch is exhaustive over the + // declared engine constants, but no driver returns these. return clamp01(score) default: // Unknown engine. Clamp defensively; the caller emits WARN with ctx. diff --git a/internal/application/service/retriever/normalizer_test.go b/internal/application/service/retriever/normalizer_test.go index 0763a5a0..94e2299c 100644 --- a/internal/application/service/retriever/normalizer_test.go +++ b/internal/application/service/retriever/normalizer_test.go @@ -29,6 +29,14 @@ func TestEngineAwareNormalizer_KeywordPassthrough(t *testing.T) { func TestEngineAwareNormalizer_CosineRange(t *testing.T) { t.Parallel() n := EngineAwareNormalizer{} + // Only Milvus surfaces the raw cosine signed range [-1, 1] to the + // normalizer in this codebase. Elasticsearch v8 used to be in this + // group, but the driver issues a script_score cosineSimilarity script + // and Lucene rejects negative final scores ("Final relevance scores + // from the script_score query cannot be negative" — ES docs), so the + // score observed by the normalizer is already in [0, 1]; ES is now + // part of the passthrough group below. ElasticFaiss is a dead enum + // (no driver) and follows ES. cases := []struct { score float64 want float64 @@ -43,8 +51,6 @@ func TestEngineAwareNormalizer_CosineRange(t *testing.T) { {1.5, 1.0}, } for _, engine := range []types.RetrieverEngineType{ - types.ElasticsearchRetrieverEngineType, - types.ElasticFaissRetrieverEngineType, types.MilvusRetrieverEngineType, } { for _, tc := range cases { @@ -61,6 +67,22 @@ func TestEngineAwareNormalizer_CosineRange(t *testing.T) { func TestEngineAwareNormalizer_UnitInterval(t *testing.T) { t.Parallel() n := EngineAwareNormalizer{} + // All engines whose effective score arriving at the normalizer is + // already in [0, 1]: + // - Elasticsearch v8 / ElasticFaiss — Lucene script_score's + // non-negative invariant truncates the theoretical [-1, 1] + // cosineSimilarity output to [0, 1]. + // - OpenSearch — the k-NN plugin's + // SpaceType.COSINESIMIL.scoreTranslation pre-maps to (1 + cos)/2. + // - Weaviate — driver requests certainty, intrinsically in [0, 1]. + // - Postgres pgvector / SQLite sqlite-vec / Qdrant / + // TencentVectorDB / Doris — theoretically [-1, 1] but the + // IR-normalized embeddings WeKnora targets (BGE / OpenAI / + // Cohere / sentence-transformers) keep the observed range + // in [0, 1]; the silent floor at clamp01(<0) → 0 is the + // defense for embedding models that violate that assumption. + // - Infinity — dead enum reference; case label kept for switch + // exhaustiveness, never reached in production. cases := []struct { score float64 want float64 @@ -73,10 +95,13 @@ func TestEngineAwareNormalizer_UnitInterval(t *testing.T) { {1.5, 1}, } for _, engine := range []types.RetrieverEngineType{ - types.PostgresRetrieverEngineType, - types.QdrantRetrieverEngineType, + types.ElasticsearchRetrieverEngineType, + types.ElasticFaissRetrieverEngineType, + types.OpenSearchRetrieverEngineType, types.WeaviateRetrieverEngineType, + types.PostgresRetrieverEngineType, types.SQLiteRetrieverEngineType, + types.QdrantRetrieverEngineType, types.InfinityRetrieverEngineType, types.TencentVectorDBRetrieverEngineType, types.DorisRetrieverEngineType, @@ -92,6 +117,125 @@ func TestEngineAwareNormalizer_UnitInterval(t *testing.T) { } } +// TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an explicit +// regression guard for the score-range correction landed in this PR. ES's +// driver returns the raw output of the `cosineSimilarity(...)` script_score +// script. Lucene rejects negative final scores (per the ES documentation: +// "Final relevance scores from the script_score query cannot be negative. +// To support certain search optimizations, Lucene requires scores be +// positive or 0"), so for IR-normalized embeddings the effective range +// observed at the normalizer is already in [0, 1]. ES therefore belongs +// in the passthrough group and the cosine=0.5 case must map to 0.5 (not +// (0.5 + 1) / 2 = 0.75 as an earlier draft assumed). +func TestEngineAwareNormalizer_ElasticsearchCosinePassthrough(t *testing.T) { + t.Parallel() + n := EngineAwareNormalizer{} + cases := []struct { + name string + score float64 + want float64 + }{ + {"cos=0 orthogonal IR-clamped to 0", 0, 0}, + {"cos=0.25 IR-typical low passes through", 0.25, 0.25}, + {"cos=0.5 IR-typical mid passes through (not 0.75)", 0.5, 0.5}, + {"cos=0.75 IR-typical high passes through", 0.75, 0.75}, + {"cos=1 identical passes through as 1", 1, 1}, + // Lucene's non-negative invariant means the normalizer should + // not receive <0 from a sane driver, but the clamp guards the + // downstream sort comparator either way. + {"negative defensive clamps to 0", -0.5, 0}, + // Engine drift past 1 clamps to 1. + {"drift past 1 clamps to 1", 1.0001, 1}, + // Float-edge cases handled by clamp01. + {"+Inf clamps to 1", math.Inf(1), 1}, + {"-Inf clamps to 0", math.Inf(-1), 0}, + {"NaN clamps to 0", math.NaN(), 0}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := n.Normalize(context.Background(), tc.score, + types.VectorRetrieverType, types.ElasticsearchRetrieverEngineType) + if math.Abs(got-tc.want) > 1e-9 { + t.Fatalf("elasticsearch score=%v: want %v, got %v", + tc.score, tc.want, got) + } + }) + } +} + +func TestEngineAwareNormalizer_OpenSearchCosinePassthrough(t *testing.T) { + t.Parallel() + n := EngineAwareNormalizer{} + // OpenSearch k-NN cosinesimil scores arrive already mapped to [0, 1] + // by the plugin's SpaceType.COSINESIMIL.scoreTranslation function + // (rawScore = 1 - cosine; output = max((2 - rawScore) / 2, 0) = + // (1 + cosine) / 2). The normalizer therefore passes the value through + // (clamping only to guard against engine drift and NaN/Inf inputs that + // could otherwise break the slices.SortFunc strict-weak-ordering + // invariant downstream). + // Subtests run serially (no inner t.Parallel) — consistent with + // the other table-driven tests in this file. + cases := []struct { + name string + score float64 + want float64 + }{ + // Basic mapping (k-NN-translated _score in [0, 1]) + {"score 0 (cos=-1) passes through as 0", 0, 0}, + {"score 0.5 (cos=0, orthogonal) passes through", 0.5, 0.5}, + {"score 0.75 (cos=0.5, IR-typical mid) passes through", 0.75, 0.75}, + {"score 1 (cos=1, identical) passes through as 1", 1, 1}, + // Engine drift past [0, 1] envelope clamps. + {"score 1.0001 clamps to 1", 1.0001, 1}, + // Negative defensive: cosinesimil's translation has a max(., 0) + // floor so <0 should never reach us, but clamp guards the + // comparator-ordering invariant downstream. + {"score -0.5 clamps to 0", -0.5, 0}, + // Float-edge cases handled by clamp01. + {"+Inf clamps to 1", math.Inf(1), 1}, + {"-Inf clamps to 0", math.Inf(-1), 0}, + {"NaN clamps to 0", math.NaN(), 0}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := n.Normalize(context.Background(), tc.score, + types.VectorRetrieverType, types.OpenSearchRetrieverEngineType) + if math.Abs(got-tc.want) > 1e-9 { + t.Fatalf("opensearch score=%v: want %v, got %v", + tc.score, tc.want, got) + } + }) + } +} + +func TestEngineAwareNormalizer_OpenSearchKeywordPassthrough(t *testing.T) { + t.Parallel() + n := EngineAwareNormalizer{} + // BM25 keyword scores on OpenSearch are unbounded positive raw + // values — they MUST pass through unchanged so the service-layer + // RRF rank-based fusion handles the scale-mixed combination. + got := n.Normalize(context.Background(), 12.7, + types.KeywordsRetrieverType, types.OpenSearchRetrieverEngineType) + if got != 12.7 { + t.Fatalf("opensearch keyword passthrough: want 12.7, got %v", got) + } +} + +func TestEngineAwareNormalizer_OpenSearchNilCtx_DoesNotPanic(t *testing.T) { + // nil ctx must not panic. The OpenSearch case is IO-free by + // contract — it never reads from ctx and never logs. + t.Parallel() + defer func() { + if r := recover(); r != nil { + t.Fatalf("OpenSearch Normalize panicked on nil ctx: %v", r) + } + }() + _ = EngineAwareNormalizer{}.Normalize( + nil, 1.0, types.VectorRetrieverType, + types.OpenSearchRetrieverEngineType, + ) +} + func TestEngineAwareNormalizer_Unknown_ClampsAndDoesNotPanic(t *testing.T) { t.Parallel() n := EngineAwareNormalizer{} diff --git a/internal/types/audit_log.go b/internal/types/audit_log.go index 4432e7f7..407e8ae4 100644 --- a/internal/types/audit_log.go +++ b/internal/types/audit_log.go @@ -50,6 +50,45 @@ const ( // AuditActionInvitationExpired fires when the lazy sweep transitions // an overdue pending row to expired. Actor is empty (system). AuditActionInvitationExpired AuditAction = "rbac.invitation_expired" + + // VectorStore lifecycle actions. Emitted by VectorStoreService. + // Cover both env-store-derived (__env_*) and DB store create / + // update / delete paths. Details payload identifies the store_id + // and the changed key set; secret values (Password, APIKey, + // connection_config encrypted blob) MUST NOT appear in details. + + // AuditActionVectorStoreCreated fires when a new VectorStore row + // is committed to the DB (Phase 1 CRUD path). Actor is the tenant + // user; resource is the VectorStore. + AuditActionVectorStoreCreated AuditAction = "vector_store.created" + // AuditActionVectorStoreUpdated fires on UPDATE of any VectorStore + // mutable field. Details payload carries the changed key set + // (never the secret values themselves — only the field names). + AuditActionVectorStoreUpdated AuditAction = "vector_store.updated" + // AuditActionVectorStoreDeleted fires when a VectorStore is + // (soft-)deleted. Phase 2's delete guard already prevents deletion + // of stores with bound KBs; the audit row records the actor and + // the store_id for forensic traceability. + AuditActionVectorStoreDeleted AuditAction = "vector_store.deleted" + + // OpenSearch-specific actions emitted by the driver shipped in + // Phase 3. The OpenSearch index is a derived resource of the + // VectorStore — these events capture cluster-side side effects + // (PUT /, DELETE /, POST /_reindex) that operators + // may need to correlate with VectorStore lifecycle events. + + // AuditActionOpenSearchIndexCreated fires when the OpenSearch + // driver lazily creates a per-dimension index (the first time a + // KB with a given embedding dim binds to the store). Details + // payload: index name, alias name, dimension. + AuditActionOpenSearchIndexCreated AuditAction = "opensearch.index_created" + // AuditActionOpenSearchIndexDeleted fires when the OpenSearch + // driver drops an index (e.g. cascade from VectorStore delete). + AuditActionOpenSearchIndexDeleted AuditAction = "opensearch.index_deleted" + // AuditActionOpenSearchReindexExecuted fires when CopyIndices + // initiates a _reindex (sync or async). Details payload: source + // KB id, target KB id, sync-or-async, doc count if known. + AuditActionOpenSearchReindexExecuted AuditAction = "opensearch.reindex_executed" ) // AuditOutcome distinguishes successful mutations from middleware-level diff --git a/internal/types/audit_log_test.go b/internal/types/audit_log_test.go new file mode 100644 index 00000000..e0fad1c5 --- /dev/null +++ b/internal/types/audit_log_test.go @@ -0,0 +1,135 @@ +package types + +import ( + "strings" + "testing" + + "github.com/stretchr/testify/assert" +) + +// TestAuditAction_DotNamespaceConvention ensures every AuditAction +// constant follows the dot-namespaced `.` convention +// documented at the top of audit_log.go. Future PRs that add actions +// must keep this invariant so audit-log consumers can prefix-filter +// by area. +func TestAuditAction_DotNamespaceConvention(t *testing.T) { + all := []AuditAction{ + // RBAC namespace (Phase 2 PR 5 / #1427) + AuditActionMemberAdded, + AuditActionMemberRemoved, + AuditActionMemberRoleChanged, + AuditActionMemberLeft, + AuditActionAccessDenied, + AuditActionInvitationSent, + AuditActionInvitationAccepted, + AuditActionInvitationDeclined, + AuditActionInvitationRevoked, + AuditActionInvitationExpired, + // VectorStore namespace (Phase 3 PR 1 / #1440) + AuditActionVectorStoreCreated, + AuditActionVectorStoreUpdated, + AuditActionVectorStoreDeleted, + // OpenSearch namespace (Phase 3 PR 1 / #1440) + AuditActionOpenSearchIndexCreated, + AuditActionOpenSearchIndexDeleted, + AuditActionOpenSearchReindexExecuted, + } + for _, a := range all { + s := string(a) + area, event, ok := strings.Cut(s, ".") + assert.True(t, ok, + "action %q must contain exactly one dot separator", s) + assert.NotEmpty(t, area, "action %q has empty area", s) + assert.NotEmpty(t, event, "action %q has empty event", s) + } +} + +// TestAuditAction_VectorStoreNamespacePrefix pins the three +// vector_store.* actions added in Phase 3 PR 1 to their shared area +// prefix. The prefix is the contract by which an audit-log reader +// can subscribe to "all VectorStore lifecycle events" with a single +// filter. +func TestAuditAction_VectorStoreNamespacePrefix(t *testing.T) { + cases := []AuditAction{ + AuditActionVectorStoreCreated, + AuditActionVectorStoreUpdated, + AuditActionVectorStoreDeleted, + } + for _, a := range cases { + assert.True(t, + strings.HasPrefix(string(a), "vector_store."), + "expected %q to start with 'vector_store.'", a, + ) + } +} + +// TestAuditAction_OpenSearchNamespacePrefix pins the three +// opensearch.* actions added in Phase 3 PR 1 to their shared area +// prefix. OpenSearch index lifecycle is a driver-specific concern +// distinct from VectorStore lifecycle — both can co-occur for the +// same logical operation (e.g. CreateVectorStore + IndexCreated). +func TestAuditAction_OpenSearchNamespacePrefix(t *testing.T) { + cases := []AuditAction{ + AuditActionOpenSearchIndexCreated, + AuditActionOpenSearchIndexDeleted, + AuditActionOpenSearchReindexExecuted, + } + for _, a := range cases { + assert.True(t, + strings.HasPrefix(string(a), "opensearch."), + "expected %q to start with 'opensearch.'", a, + ) + } +} + +// TestAuditAction_NoCollisionsAcrossNamespaces ensures no two +// AuditAction constants share the same wire string. A duplicate would +// silently merge two logical events into one entry in the audit-log +// UI, defeating the dot-namespace convention. +func TestAuditAction_NoCollisionsAcrossNamespaces(t *testing.T) { + seen := make(map[AuditAction]string) + register := func(name string, a AuditAction) { + t.Helper() + if prev, exists := seen[a]; exists { + t.Fatalf("collision: %s and %s both map to %q", prev, name, a) + } + seen[a] = name + } + register("AuditActionMemberAdded", AuditActionMemberAdded) + register("AuditActionMemberRemoved", AuditActionMemberRemoved) + register("AuditActionMemberRoleChanged", AuditActionMemberRoleChanged) + register("AuditActionMemberLeft", AuditActionMemberLeft) + register("AuditActionAccessDenied", AuditActionAccessDenied) + register("AuditActionInvitationSent", AuditActionInvitationSent) + register("AuditActionInvitationAccepted", AuditActionInvitationAccepted) + register("AuditActionInvitationDeclined", AuditActionInvitationDeclined) + register("AuditActionInvitationRevoked", AuditActionInvitationRevoked) + register("AuditActionInvitationExpired", AuditActionInvitationExpired) + register("AuditActionVectorStoreCreated", AuditActionVectorStoreCreated) + register("AuditActionVectorStoreUpdated", AuditActionVectorStoreUpdated) + register("AuditActionVectorStoreDeleted", AuditActionVectorStoreDeleted) + register("AuditActionOpenSearchIndexCreated", AuditActionOpenSearchIndexCreated) + register("AuditActionOpenSearchIndexDeleted", AuditActionOpenSearchIndexDeleted) + register("AuditActionOpenSearchReindexExecuted", AuditActionOpenSearchReindexExecuted) +} + +// TestAuditAction_Phase3WireValues pins the exact wire strings for +// the six new Phase 3 actions. The wire strings are the public +// contract for audit-log consumers; changing them is a breaking +// change. +func TestAuditAction_Phase3WireValues(t *testing.T) { + cases := []struct { + constant AuditAction + wire string + }{ + {AuditActionVectorStoreCreated, "vector_store.created"}, + {AuditActionVectorStoreUpdated, "vector_store.updated"}, + {AuditActionVectorStoreDeleted, "vector_store.deleted"}, + {AuditActionOpenSearchIndexCreated, "opensearch.index_created"}, + {AuditActionOpenSearchIndexDeleted, "opensearch.index_deleted"}, + {AuditActionOpenSearchReindexExecuted, "opensearch.reindex_executed"}, + } + for _, c := range cases { + assert.Equal(t, c.wire, string(c.constant)) + } +} diff --git a/internal/types/retriever.go b/internal/types/retriever.go index 004b8066..afe5a424 100644 --- a/internal/types/retriever.go +++ b/internal/types/retriever.go @@ -15,6 +15,15 @@ const ( DorisRetrieverEngineType RetrieverEngineType = "doris" SQLiteRetrieverEngineType RetrieverEngineType = "sqlite" TencentVectorDBRetrieverEngineType RetrieverEngineType = "tencent_vectordb" + // OpenSearchRetrieverEngineType identifies the OpenSearch k-NN driver + // introduced in Phase 3 (see issue tracker). The driver itself ships + // in a subsequent PR; this constant exists so that + // EngineAwareNormalizer can register the OpenSearch case ahead of the + // driver landing and the AuditAction constants can reference it + // without forward dependencies. Wire value "opensearch" mirrors the + // official product name and matches the value used in + // retrieverEngineMapping / GetVectorStoreTypes once activation lands. + OpenSearchRetrieverEngineType RetrieverEngineType = "opensearch" ) // RetrieverType represents the type of retriever diff --git a/internal/types/vectorstore.go b/internal/types/vectorstore.go index 57c6bdc9..865d1615 100644 --- a/internal/types/vectorstore.go +++ b/internal/types/vectorstore.go @@ -119,6 +119,16 @@ type ConnectionConfig struct { Username string `yaml:"username" json:"username,omitempty"` Password string `yaml:"password" json:"password,omitempty"` // AES-GCM encrypted APIKey string `yaml:"api_key" json:"api_key,omitempty"` // AES-GCM encrypted + // InsecureSkipVerify disables TLS certificate verification when + // talking to the backing store over HTTPS. Defaults to false + // (secure). Set to true ONLY for self-signed development clusters; + // production deployments should provide trusted certificates via + // the system CA pool. Cross-driver applicable but currently only + // the OpenSearch driver (Phase 3) reads this field. Note: this + // differs from the Qdrant-specific UseTLS below, which *enables* + // TLS on gRPC connections — InsecureSkipVerify only controls + // *verification* of an already-TLS connection. + InsecureSkipVerify bool `yaml:"insecure_skip_verify" json:"insecure_skip_verify,omitempty"` // Qdrant Host string `yaml:"host" json:"host,omitempty"` Port int `yaml:"port" json:"port,omitempty"` @@ -590,7 +600,11 @@ type VectorStoreTypeInfo struct { IndexFields []VectorStoreFieldInfo `json:"index_fields,omitempty"` } -// VectorStoreFieldInfo describes a single configuration field. +// VectorStoreFieldInfo describes a single configuration field exposed +// by /api/v1/vector-stores/types for the registration UI. The optional +// validation hints (`Immutable`, `Min`, `Max`, `Enum`) are used by both +// the frontend (to disable / constrain inputs) and the backend +// (defense-in-depth validation in the service layer). type VectorStoreFieldInfo struct { Name string `json:"name"` Type string `json:"type"` // "string", "number", "boolean" @@ -598,6 +612,24 @@ type VectorStoreFieldInfo struct { Sensitive bool `json:"sensitive,omitempty"` Default any `json:"default,omitempty"` Description string `json:"description,omitempty"` + + // Immutable marks a field whose value cannot be changed after the + // VectorStore is first created. The UI shows the input as read-only + // in edit mode; the backend rejects modification attempts. Used by + // engines whose underlying index structure is fixed at create time + // (e.g. OpenSearch's HNSW engine / M / ef_construction). + Immutable bool `json:"immutable,omitempty"` + + // Min / Max set inclusive bounds for "number"-typed fields. nil + // means no bound on that side. Frontend uses these to constrain + // inputs; backend re-validates as defense-in-depth. + Min *float64 `json:"min,omitempty"` + Max *float64 `json:"max,omitempty"` + + // Enum constrains the allowed string values. Empty means no + // constraint. Used by fields whose value space is closed (e.g. + // OpenSearch's knn_engine ∈ {"lucene", "faiss"}). + Enum []string `json:"enum,omitempty"` } // GetVectorStoreTypes returns metadata for all supported engine types. diff --git a/internal/types/vectorstore_test.go b/internal/types/vectorstore_test.go index d4b01a48..5e018a15 100644 --- a/internal/types/vectorstore_test.go +++ b/internal/types/vectorstore_test.go @@ -600,6 +600,22 @@ func TestConnectionConfig_MaskSensitiveFields(t *testing.T) { assert.Equal(t, "secret-pass", c.Password) assert.Equal(t, "sk-api-key", c.APIKey) }) + + t.Run("preserves insecure_skip_verify as a visible operator-facing knob", func(t *testing.T) { + // InsecureSkipVerify is deliberately NOT redacted: operators + // must be able to see whether TLS verification is disabled. + // This pin prevents a well-meaning future change from quietly + // masking the flag and hiding a misconfiguration. + c := ConnectionConfig{ + Addr: "https://os:9200", + Password: "secret-pass", + InsecureSkipVerify: true, + } + masked := c.MaskSensitiveFields() + assert.Equal(t, RedactedSecretPlaceholder, masked.Password) + assert.True(t, masked.InsecureSkipVerify, + "InsecureSkipVerify must remain visible after masking") + }) } // --------------------------------------------------------------------------- @@ -1063,10 +1079,10 @@ func TestIndexConfig_ScalabilityFieldsRoundTrip(t *testing.T) { // read-path consistency for the two engines that are only meaningful as env // stores. They must: // -// 1. Be rejected by Validate() so POST /vector-stores returns a 4xx -// instead of silently persisting a row that has no separation effect. -// 2. Be absent from GetVectorStoreTypes() so the UI dropdown doesn't -// offer them as a choice. +// 1. Be rejected by Validate() so POST /vector-stores returns a 4xx +// instead of silently persisting a row that has no separation effect. +// 2. Be absent from GetVectorStoreTypes() so the UI dropdown doesn't +// offer them as a choice. // // Both checks live here so a future change that re-introduces one path // (e.g., adds Postgres back to validEngineTypes for some niche case) @@ -1105,3 +1121,293 @@ func TestVectorStore_PostgresSqliteNotRegisterable(t *testing.T) { "sqlite must not appear in the UI dropdown — env-store only") }) } + +// --------------------------------------------------------------------------- +// Phase 3 PR1 additions: ConnectionConfig.InsecureSkipVerify backward-compat +// and VectorStoreFieldInfo schema extensions (Immutable / Min / Max / Enum) +// --------------------------------------------------------------------------- + +// TestConnectionConfig_InsecureSkipVerify_BackwardCompat ensures that +// ConnectionConfigs persisted before Phase 3 deserialize cleanly: the +// missing JSON field maps to false (Go zero-value). This is the +// foundational backward-compat guarantee for the schema extension. +func TestConnectionConfig_InsecureSkipVerify_BackwardCompat(t *testing.T) { + t.Run("missing field defaults to false", func(t *testing.T) { + legacy := `{"addr":"https://es:9200","username":"u","password":"p"}` + var cfg ConnectionConfig + require.NoError(t, json.Unmarshal([]byte(legacy), &cfg)) + assert.False(t, cfg.InsecureSkipVerify) + }) + + t.Run("explicit false deserializes correctly", func(t *testing.T) { + raw := `{"addr":"https://es:9200","insecure_skip_verify":false}` + var cfg ConnectionConfig + require.NoError(t, json.Unmarshal([]byte(raw), &cfg)) + assert.False(t, cfg.InsecureSkipVerify) + }) + + t.Run("explicit true deserializes correctly", func(t *testing.T) { + raw := `{"addr":"https://os:9200","insecure_skip_verify":true}` + var cfg ConnectionConfig + require.NoError(t, json.Unmarshal([]byte(raw), &cfg)) + assert.True(t, cfg.InsecureSkipVerify) + }) +} + +// TestConnectionConfig_InsecureSkipVerify_RoundTrip verifies that the +// new field serializes correctly when set, and is omitted (omitempty) +// when false — so existing entries do not gain a new wire-format key +// after the Phase 3 schema extension lands. +func TestConnectionConfig_InsecureSkipVerify_RoundTrip(t *testing.T) { + t.Run("true serializes the field", func(t *testing.T) { + cfg := ConnectionConfig{Addr: "https://os:9200", InsecureSkipVerify: true} + out, err := json.Marshal(cfg) + require.NoError(t, err) + assert.Contains(t, string(out), `"insecure_skip_verify":true`) + }) + + t.Run("false omits the field via omitempty", func(t *testing.T) { + cfg := ConnectionConfig{Addr: "https://os:9200", InsecureSkipVerify: false} + out, err := json.Marshal(cfg) + require.NoError(t, err) + assert.NotContains(t, string(out), `"insecure_skip_verify"`) + }) +} + +// TestConnectionConfig_AESGCMRoundTrip_PreservesInsecureSkipVerify +// ensures the AES-GCM Value/Scan path round-trips the new field +// without corruption. The field is stored as plaintext (no AES-GCM) +// but travels alongside encrypted Password / APIKey. +func TestConnectionConfig_AESGCMRoundTrip_PreservesInsecureSkipVerify(t *testing.T) { + t.Setenv("SYSTEM_AES_KEY", testAESKey) + + original := ConnectionConfig{ + Addr: "https://os:9200", + Username: "admin", + Password: "secret-pass", + APIKey: "sk-api-key", + InsecureSkipVerify: true, + } + + // Value — encrypt + raw, err := original.Value() + require.NoError(t, err) + + // Verify the serialized JSON has the plaintext flag alongside the + // encrypted secret fields. + var intermediate map[string]interface{} + require.NoError(t, json.Unmarshal(raw.([]byte), &intermediate)) + assert.True(t, strings.HasPrefix(intermediate["password"].(string), "enc:v1:"), + "password should still be encrypted") + assert.Equal(t, true, intermediate["insecure_skip_verify"], + "insecure_skip_verify should be plaintext bool") + + // Scan — decrypt secrets and preserve the flag + var scanned ConnectionConfig + require.NoError(t, scanned.Scan(raw.([]byte))) + assert.Equal(t, "secret-pass", scanned.Password) + assert.Equal(t, "sk-api-key", scanned.APIKey) + assert.True(t, scanned.InsecureSkipVerify, + "InsecureSkipVerify should round-trip through Value/Scan") +} + +// TestVectorStoreFieldInfo_OmitemptyPreservesWire confirms that the +// four new optional fields (Immutable / Min / Max / Enum) added in +// Phase 3 PR 1 are omitted from JSON when unset, so existing +// VectorStoreFieldInfo entries (without these fields) serialize +// identically before and after the schema extension. +func TestVectorStoreFieldInfo_OmitemptyPreservesWire(t *testing.T) { + legacy := VectorStoreFieldInfo{ + Name: "addr", + Type: "string", + Required: true, + Description: "URL", + Default: "http://localhost:9200", + } + out, err := json.Marshal(legacy) + require.NoError(t, err) + // Phase 3 new fields must all be omitted from the wire format. + assert.NotContains(t, string(out), `"immutable"`) + assert.NotContains(t, string(out), `"min"`) + assert.NotContains(t, string(out), `"max"`) + assert.NotContains(t, string(out), `"enum"`) +} + +// TestVectorStoreFieldInfo_NewFieldsSerialize verifies the new fields +// appear in JSON when set — exercised by the OpenSearch IndexFields +// entry that lands in a later Phase 3 PR. +func TestVectorStoreFieldInfo_NewFieldsSerialize(t *testing.T) { + t.Run("immutable", func(t *testing.T) { + f := VectorStoreFieldInfo{Name: "knn_engine", Type: "string", Immutable: true} + out, err := json.Marshal(f) + require.NoError(t, err) + assert.Contains(t, string(out), `"immutable":true`) + }) + + t.Run("min and max", func(t *testing.T) { + minV, maxV := float64(4), float64(64) + f := VectorStoreFieldInfo{Name: "hnsw_m", Type: "number", Min: &minV, Max: &maxV} + out, err := json.Marshal(f) + require.NoError(t, err) + assert.Contains(t, string(out), `"min":4`) + assert.Contains(t, string(out), `"max":64`) + }) + + t.Run("enum", func(t *testing.T) { + f := VectorStoreFieldInfo{ + Name: "knn_engine", Type: "string", + Enum: []string{"lucene", "faiss"}, + } + out, err := json.Marshal(f) + require.NoError(t, err) + assert.Contains(t, string(out), `"enum":["lucene","faiss"]`) + }) + + t.Run("empty enum omits via omitempty", func(t *testing.T) { + f := VectorStoreFieldInfo{Name: "addr", Type: "string", Enum: []string{}} + out, err := json.Marshal(f) + require.NoError(t, err) + assert.NotContains(t, string(out), `"enum"`) + }) +} + +// TestVectorStoreFieldInfo_RoundTrip ensures deserialization back into +// the struct works for all new fields, including the *float64 pointer +// distinction (nil vs explicit 0). +func TestVectorStoreFieldInfo_RoundTrip(t *testing.T) { + t.Run("min=0 deserializes as &0, not nil", func(t *testing.T) { + raw := `{"name":"replicas","type":"number","min":0,"max":5}` + var f VectorStoreFieldInfo + require.NoError(t, json.Unmarshal([]byte(raw), &f)) + require.NotNil(t, f.Min) + assert.Equal(t, float64(0), *f.Min) + require.NotNil(t, f.Max) + assert.Equal(t, float64(5), *f.Max) + }) + + t.Run("missing min/max deserialize as nil", func(t *testing.T) { + raw := `{"name":"addr","type":"string"}` + var f VectorStoreFieldInfo + require.NoError(t, json.Unmarshal([]byte(raw), &f)) + assert.Nil(t, f.Min) + assert.Nil(t, f.Max) + }) + + t.Run("min and max are independent pointers", func(t *testing.T) { + // Setting only Min must leave Max nil (and vice versa). A + // refactor that accidentally aliases the two via a shared + // local would still pass the previous two subtests because + // they always set both ends. + raw := `{"name":"upper_only","type":"number","max":10}` + var f VectorStoreFieldInfo + require.NoError(t, json.Unmarshal([]byte(raw), &f)) + assert.Nil(t, f.Min) + require.NotNil(t, f.Max) + assert.Equal(t, float64(10), *f.Max) + }) + + t.Run("nil enum omits via omitempty", func(t *testing.T) { + // `nil` and `[]string{}` both round-trip through omitempty + // identically — pin both shapes so a future Go encoder + // behavior change cannot regress the wire format. + f := VectorStoreFieldInfo{Name: "addr", Type: "string"} // Enum nil + out, err := json.Marshal(f) + require.NoError(t, err) + assert.NotContains(t, string(out), `"enum"`) + }) +} + +// TestOpenSearchRetrieverEngineType_StringValue pins the wire string +// to the official product name. The constant exists in PR 1 so the +// EngineAwareNormalizer case and AuditAction constants can reference +// it; the driver itself lands in a later PR. +func TestOpenSearchRetrieverEngineType_StringValue(t *testing.T) { + assert.Equal(t, + RetrieverEngineType("opensearch"), + OpenSearchRetrieverEngineType, + ) +} + +// TestOpenSearchRetrieverEngineType_DistinctFromExisting ensures the +// new wire value does not collide with any of the 10 existing engine +// types. A collision would silently route requests to the wrong +// engine after the activation switch lands. +func TestOpenSearchRetrieverEngineType_DistinctFromExisting(t *testing.T) { + existing := []RetrieverEngineType{ + PostgresRetrieverEngineType, + ElasticsearchRetrieverEngineType, + InfinityRetrieverEngineType, + ElasticFaissRetrieverEngineType, + QdrantRetrieverEngineType, + MilvusRetrieverEngineType, + WeaviateRetrieverEngineType, + DorisRetrieverEngineType, + SQLiteRetrieverEngineType, + TencentVectorDBRetrieverEngineType, + } + for _, e := range existing { + assert.NotEqual(t, e, OpenSearchRetrieverEngineType, + "OpenSearch wire value must not collide with %s", e) + } +} + +// TestOpenSearchRetrieverEngineType_NotInValidEngineTypes verifies +// that PR 1 does NOT add the new engine type to validEngineTypes — +// this is the gate that keeps OpenSearch VectorStore registration +// rejected until activation lands in a later PR. +func TestOpenSearchRetrieverEngineType_NotInValidEngineTypes(t *testing.T) { + assert.False(t, IsValidEngineType(OpenSearchRetrieverEngineType), + "OpenSearch must remain invalid for VectorStore registration "+ + "until activation lands in a later PR (gated activation)") +} + +// The next three tests are defense-in-depth companions to +// TestOpenSearchRetrieverEngineType_NotInValidEngineTypes. Each pins +// a separate activation surface that must remain closed in PR 1 (and +// in PR 2). Activation lands together with a coordinated flip in +// PR 3, at which point each of these `assert.False` / `assert.NotContains` +// / `assert.Nil` lines flips to its positive counterpart in the same +// diff — the test suite becomes the activation checklist. + +// TestRetrieverEngineMapping_OpenSearchNotRegistered pins that +// `retrieverEngineMapping` does not have an `"opensearch"` key. Without +// this entry, setting `RETRIEVE_DRIVER=opensearch` is a silent no-op +// (GetDefaultRetrieverEngines drops the unknown driver from its loop +// at tenant.go). +func TestRetrieverEngineMapping_OpenSearchNotRegistered(t *testing.T) { + mapping := GetRetrieverEngineMapping() + _, ok := mapping["opensearch"] + assert.False(t, ok, + "retrieverEngineMapping must not register opensearch until "+ + "activation lands in a later PR (gated activation)") +} + +// TestGetVectorStoreTypes_OmitsOpenSearch pins that the +// /api/v1/vector-stores/types response does NOT list opensearch. +// Without this entry, the UI dropdown cannot offer OpenSearch as a +// store type even if a frontend renderer accidentally tries. +func TestGetVectorStoreTypes_OmitsOpenSearch(t *testing.T) { + listed := GetVectorStoreTypes() + for _, info := range listed { + assert.NotEqual(t, "opensearch", info.Type, + "GetVectorStoreTypes must not surface opensearch until "+ + "activation lands in a later PR (gated activation)") + } +} + +// TestBuildEnvVectorStores_OpenSearchSkipped pins that +// `BuildEnvVectorStores("opensearch", lookup)` returns nil — the +// `default:` arm of `buildEnvStoreForDriver`. Without an explicit +// `case "opensearch":` arm, container.go cannot synthesize an env +// store for the driver, completing the third lock on the activation +// chain. +func TestBuildEnvVectorStores_OpenSearchSkipped(t *testing.T) { + stores := BuildEnvVectorStores("opensearch", mockEnvLookup(map[string]string{ + "OPENSEARCH_ADDR": "https://os:9200", + "OPENSEARCH_USERNAME": "admin", + })) + assert.Empty(t, stores, + "BuildEnvVectorStores must not synthesize an env store for "+ + "opensearch until activation lands in a later PR (gated "+ + "activation)") +}