fd3d9f547c
Multi-KB hybrid search now groups KBs by their bound VectorStore (partition key (storeID, owner_tenant_id)), retrieves in parallel via errgroup with a SetLimit(4) cap and a per-group timeout (MULTI_STORE_RETRIEVE_TIMEOUT_SEC, default 30s), and merges results. When the collected results span more than one engine type, an EngineAwareNormalizer rescales vector scores to [0, 1]; keyword (BM25) scores pass through to the existing RRF fusion. Single-group calls take the fast path with zero fan-out overhead, preserving today's behavior for deployments where every KB has vector_store_id = NULL. Embedding-model consistency is now enforced explicitly via ResolveEmbeddingModelKeys. Multi-KB searches across KBs whose resolved model identities differ return BadRequest instead of silently producing incomparable scores. Cross-tenant Organization-shared KBs are preserved by partitioning on KB.TenantID so the factory's ownership lookup runs against the source tenant. Foreign-tenant KB UUIDs injected via the request body are rejected via kbShareService.HasTenantKBPermission (Plan 3 of #1303, 3-D capped) before any retrieval; rejected scopes surface as 404 to avoid leaking foreign KB existence. Service-layer typed AppErrors (ErrVectorStoreBindingInvalid 2200 / ErrVectorStoreUnavailable 2201) are mapped from PR2 sentinel hierarchy and preserved end-to-end: the iterative FAQ path returns them rather than swallowing, and the HybridSearch handler routes typed AppErrors to the client unchanged instead of downgrading to 500. Part of #993 (Phase 2: Per-KB VectorStore Binding). Phase 2 roadmap item: PR 4 (Multi-store fan-out search). Depends on #994, #1310, #1372.
102 lines
3.4 KiB
Go
102 lines
3.4 KiB
Go
package retriever
|
|
|
|
import (
|
|
"context"
|
|
"math"
|
|
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
)
|
|
|
|
// ScoreNormalizer maps raw retriever scores to a common [0, 1] scale so that
|
|
// vector scores produced by different engines can be compared in a single
|
|
// ranked list. Implementations MUST be safe for concurrent use and MUST be
|
|
// IO-free (Normalize is called inside a hot loop and may not log or block).
|
|
//
|
|
// Only vector scores are normalized. Keyword (BM25) scores have an unbounded
|
|
// positive range; rescaling them would collapse the long tail. Downstream
|
|
// RRF fusion is rank-based and immune to scale, so keyword scores pass
|
|
// through unchanged.
|
|
type ScoreNormalizer interface {
|
|
Normalize(
|
|
ctx context.Context,
|
|
score float64,
|
|
retrieverType types.RetrieverType,
|
|
engineType types.RetrieverEngineType,
|
|
) float64
|
|
}
|
|
|
|
// EngineAwareNormalizer applies the documented per-engine cosine-score
|
|
// formula. The caller (HybridSearch) enforces a same-embedding-model
|
|
// precondition via ResolveEmbeddingModelKeys, so post-normalization values
|
|
// are semantically comparable across engines.
|
|
//
|
|
// Source formulas (verified against repository implementations on
|
|
// upstream/main 3214e3d9):
|
|
//
|
|
// - Elasticsearch v8 / ElasticFaiss / Milvus (cosine mode):
|
|
// raw cosine in [-1, 1]
|
|
// - Postgres pgvector: (1 - cosine_distance) in [0, 1]
|
|
// - SQLite sqlite-vec: (1 - cosine_distance) in [0, 1]
|
|
// - Weaviate: raw score in [0, 1]
|
|
// - Qdrant / Infinity / TencentVectorDB / Doris: raw score in [0, 1]
|
|
//
|
|
// Unknown engines clamp to [0, 1]; the fan-out caller emits a single WARN
|
|
// per request via warnIfUnknownEngine so Normalize itself stays lock-free
|
|
// and panic-free even on nil ctx.
|
|
type EngineAwareNormalizer struct{}
|
|
|
|
// Compile-time interface satisfaction assertion.
|
|
var _ ScoreNormalizer = EngineAwareNormalizer{}
|
|
|
|
// Normalize implements ScoreNormalizer.
|
|
func (EngineAwareNormalizer) Normalize(
|
|
_ context.Context,
|
|
score float64,
|
|
retrieverType types.RetrieverType,
|
|
engineType types.RetrieverEngineType,
|
|
) float64 {
|
|
if retrieverType != types.VectorRetrieverType {
|
|
// BM25 and other non-vector retrievers: passthrough. RRF rank-based
|
|
// fusion handles scale-mixed input correctly.
|
|
return score
|
|
}
|
|
|
|
switch engineType {
|
|
case types.ElasticsearchRetrieverEngineType,
|
|
types.ElasticFaissRetrieverEngineType,
|
|
types.MilvusRetrieverEngineType:
|
|
// Raw cosine in [-1, 1] → [0, 1]. Pass through clamp01 once more so
|
|
// that a misbehaving engine returning 1.0000002 does not leak past
|
|
// the [0, 1] envelope (caller sorts by score afterwards).
|
|
return clamp01((score + 1) / 2)
|
|
case types.PostgresRetrieverEngineType,
|
|
types.QdrantRetrieverEngineType,
|
|
types.WeaviateRetrieverEngineType,
|
|
types.SQLiteRetrieverEngineType,
|
|
types.InfinityRetrieverEngineType,
|
|
types.TencentVectorDBRetrieverEngineType,
|
|
types.DorisRetrieverEngineType:
|
|
// Already in [0, 1] by repository contract.
|
|
return clamp01(score)
|
|
default:
|
|
// Unknown engine. Clamp defensively; the caller emits WARN with ctx.
|
|
return clamp01(score)
|
|
}
|
|
}
|
|
|
|
// clamp01 maps any float64 into [0, 1] safely, including NaN/Inf inputs that
|
|
// could otherwise break slices.SortFunc's strict-weak-ordering invariant
|
|
// downstream (NaN compares neither greater nor less than anything).
|
|
func clamp01(s float64) float64 {
|
|
if math.IsNaN(s) {
|
|
return 0
|
|
}
|
|
if s <= 0 || math.IsInf(s, -1) {
|
|
return 0
|
|
}
|
|
if s >= 1 || math.IsInf(s, 1) {
|
|
return 1
|
|
}
|
|
return s
|
|
}
|