Files
Enterprise-Knowledge-Base/internal/application/repository/retriever/doris/structs.go
T
Miles Lai e62c0563aa doris: add configurable compatibility modes and guard mode switches
Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.

Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.

Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:

  * auto (default/recommended): probe Doris server on first use to detect available vector
    functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
    (older builds lacking inner_product_approximate)

  * legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
    builds without inner_product_approximate support)

  * inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
    (for modern Doris 4.0+ and current SelectDB with normalized embeddings)

Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
  enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
  with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
  DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
  and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
  scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level

Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
2026-05-19 17:15:59 +08:00

70 lines
2.2 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package doris
import (
"database/sql"
"net/http"
"sync"
)
// dorisRepository 是 Apache Doris 4.1 的检索引擎仓储实现。
//
// 通信通道:
// - 读写主链路:MySQL 协议(database/sql + go-sql-driver/mysql),FE 默认 9030 端口。
// - Stream Load:HTTP(FE 默认 8030 端口),legacy 模式用于 partial update。
//
// 表结构按维度分表:<tableBaseName>_<dim>。
// 兼容模式由 DORIS_COMPAT_MODE 决定:
// - legacy:UNIQUE KEY(id) + cosine_distance ANN + Stream Load partial update
// - inner_product_duplicate:DUPLICATE KEY(id) + normalized inner product + delete/insert rewrite
// 该设置在 embedding 表创建后不可直接互换;切换模式前需要重建这些表。
//
// 与 Qdrant/Milvus/Weaviate 一样,initializedTables 缓存"已确保存在"的维度,
// 避免每次写入都打 SHOW TABLES。
type dorisRepository struct {
db *sql.DB
httpClient *http.Client
// fe HTTP base,例如 "http://doris-fe:8030"。Stream Load 路径
// 由 streamLoadURL(table) 拼接:<feHTTPBase>/api/<database>/<table>/_stream_load。
feHTTPBase string
username string
password string
database string
tableBaseName string
bucketsNum int // 0 -> default 10
replicationNum int // 0 -> default 1
compatModeRequested dorisCompatMode
compatModeResolved dorisCompatMode
compatResolveOnce sync.Once
compatResolveErr error
// 已经确保过 ensureTable 的维度集合:dim -> true。
initializedTables sync.Map
}
// DorisVectorEmbedding 是落到 Doris 表里的一行的领域模型。
//
// 字段顺序与 schema.go 中的 INSERT 列序保持一致,
// 调整时需要同时更新 createInsert 与 columns。
type DorisVectorEmbedding struct {
ID string
Content string
SourceID string
SourceType int
ChunkID string
KnowledgeID string
KnowledgeBaseID string
TagID string
IsEnabled bool
Embedding []float32
}
// DorisVectorEmbeddingWithScore 是检索结果的领域模型,
// Score 在向量检索时按当前 compat mode 计算,在关键词检索时统一赋 1.0。
type DorisVectorEmbeddingWithScore struct {
DorisVectorEmbedding
Score float64
}