e62c0563aa
Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.
Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.
Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:
* auto (default/recommended): probe Doris server on first use to detect available vector
functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
(older builds lacking inner_product_approximate)
* legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
builds without inner_product_approximate support)
* inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
(for modern Doris 4.0+ and current SelectDB with normalized embeddings)
Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level
Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
70 lines
2.2 KiB
Go
70 lines
2.2 KiB
Go
package doris
|
||
|
||
import (
|
||
"database/sql"
|
||
"net/http"
|
||
"sync"
|
||
)
|
||
|
||
// dorisRepository 是 Apache Doris 4.1 的检索引擎仓储实现。
|
||
//
|
||
// 通信通道:
|
||
// - 读写主链路:MySQL 协议(database/sql + go-sql-driver/mysql),FE 默认 9030 端口。
|
||
// - Stream Load:HTTP(FE 默认 8030 端口),legacy 模式用于 partial update。
|
||
//
|
||
// 表结构按维度分表:<tableBaseName>_<dim>。
|
||
// 兼容模式由 DORIS_COMPAT_MODE 决定:
|
||
// - legacy:UNIQUE KEY(id) + cosine_distance ANN + Stream Load partial update
|
||
// - inner_product_duplicate:DUPLICATE KEY(id) + normalized inner product + delete/insert rewrite
|
||
// 该设置在 embedding 表创建后不可直接互换;切换模式前需要重建这些表。
|
||
//
|
||
// 与 Qdrant/Milvus/Weaviate 一样,initializedTables 缓存"已确保存在"的维度,
|
||
// 避免每次写入都打 SHOW TABLES。
|
||
type dorisRepository struct {
|
||
db *sql.DB
|
||
|
||
httpClient *http.Client
|
||
// fe HTTP base,例如 "http://doris-fe:8030"。Stream Load 路径
|
||
// 由 streamLoadURL(table) 拼接:<feHTTPBase>/api/<database>/<table>/_stream_load。
|
||
feHTTPBase string
|
||
|
||
username string
|
||
password string
|
||
database string
|
||
|
||
tableBaseName string
|
||
bucketsNum int // 0 -> default 10
|
||
replicationNum int // 0 -> default 1
|
||
compatModeRequested dorisCompatMode
|
||
compatModeResolved dorisCompatMode
|
||
compatResolveOnce sync.Once
|
||
compatResolveErr error
|
||
|
||
// 已经确保过 ensureTable 的维度集合:dim -> true。
|
||
initializedTables sync.Map
|
||
}
|
||
|
||
// DorisVectorEmbedding 是落到 Doris 表里的一行的领域模型。
|
||
//
|
||
// 字段顺序与 schema.go 中的 INSERT 列序保持一致,
|
||
// 调整时需要同时更新 createInsert 与 columns。
|
||
type DorisVectorEmbedding struct {
|
||
ID string
|
||
Content string
|
||
SourceID string
|
||
SourceType int
|
||
ChunkID string
|
||
KnowledgeID string
|
||
KnowledgeBaseID string
|
||
TagID string
|
||
IsEnabled bool
|
||
Embedding []float32
|
||
}
|
||
|
||
// DorisVectorEmbeddingWithScore 是检索结果的领域模型,
|
||
// Score 在向量检索时按当前 compat mode 计算,在关键词检索时统一赋 1.0。
|
||
type DorisVectorEmbeddingWithScore struct {
|
||
DorisVectorEmbedding
|
||
Score float64
|
||
}
|