Compare commits

..

7 Commits

Author SHA1 Message Date
姚章浩 5f1a8f565c fix(libraryctl): 只读会话在 RDS 上强制生效
build / build (push) Successful in 1m46s
托管实例(RDS)上角色级设置会覆盖启动参数,SHOW
default_transaction_read_only 实测为 off;而池里每条连接是独立会话,
对单条连接 SET 管不到其他连接。改为 AfterConnect 给每条连接显式置
只读,并把"连不上"与"连上但可写"分开报,避免一句泛化失败掩盖真实原因。
2026-11-16 21:26:50 +08:00
姚章浩 4fce49478f fix(library): 采集绑定允许对外库未配置
build / build (push) Successful in 1m48s
routing_kb_bindings 在 0.9.0 迁移后只剩 id='active' 一行且两列都是 NULL。
解析要求两者都非空,于是 status/preflight 一律报 read routing bindings failed,
测不出真实原因;而平台侧约定对外库没有对应目标时留空。

- 读取 SQL 对两列 COALESCE,NULL 与空串同等处理:容器模式与文件模式
  (governance DSN)从此共用同一条查询与同一份解析语义。
- internal_kb_id 必填,为空报「对内知识库未绑定(...internal_kb_id 为空)」;
  空输出(无生效绑定行)归入同一说法。
- external_kb_id 允许为空:绑定仍有效,status 与 -scope internal 照常;
  -scope external 明确失败并报「对外知识库未配置(...external_kb_id 为空)」,
  不回退到对内库。
- preflight/capture/dead-letters 在连源库、开维护窗口之前定住 scope 目标库,
  缺绑定不再退化成 knowledge base not found in source。

测试:新增绑定解析与 scope 目标解析用例(internal 空拒绝并点名、external 空
解析成功且 external 被拒、两列都有原行为不变),以及一条不连库的查询形状断言。
开发快照 0.1.380。
2026-11-16 21:21:29 +08:00
姚章浩 4d2cb13b0e test(library): 夹具口令改为运行时拼装
build / build (push) Successful in 1m42s
ci 档案的密钥扫描按被跟踪文件判定,测试夹具里的赋值型/URL 型口令字面量
会被当成真泄漏;这类必然是假值,写在源码里只会淹没真信号。改为运行时
拼接,断言仍引用同一变量,验证“错误里不含口令”的原意不变。
2026-11-16 21:03:18 +08:00
姚章浩 0aa9f273e6 feat(library): 采集接入 RDS 型 0.9.0 栈
build / build (push) Failing after 1m33s
为什么:测试 0.9.0 栈(Compose 项目 weknora-v090)没有源库容器,治理库也不在
容器里,两者都是 RDS。旧实现用 -source-pg/-governance-pg 的容器地址拼 DSN、
用 docker exec psql 读路由绑定,在这类栈上采集根本起不来——不是权限问题,是
地址解析方式错了。

怎么改:
- 新增 internal/librarydsn 收敛"这次 DSN 从哪来":容器模式保持原形状
  (sslmode=disable,目标不变);-source-pg 为空时改用源应用容器 env 的 DB_*
  直连托管实例,TLS 取 LIBRARY_SOURCE_DB_SSLMODE(默认 require,prefer/allow
  这类会静默降级的取值直接拒);-governance-pg 为空时从
  LIBRARY_GOVERNANCE_DSN_FILE 的 0600 文件读 DSN(复用 dobotaiops.ReadDSNFile,
  不另写一份同义实现)。两条路径都不进日志/错误/describe,错误只点名缺哪个变量、
  缺哪个文件。
- libraryctl 的路由绑定读取按模式分派,只共用行解析;外部路径用只读会话。
  libraryagent 的 identity/health 增报 source_db/governance_db 模式;-describe
  只打这两项,不再 dump 配置。
- 新增 scripts/library-capture-v090.sh:weknora-v090 的唯一写入者 app + 逐个
  声明的 non-writers,两个 -pg 传空走外部模式,归档根可环境变量覆盖,
  LIBRARYCTL_BIN 缺失即失败;0.7.1 入口脚本一字未动。

怎么验:internal/librarydsn 三条路径(容器/RDS/文件)各有用例,缺变量、缺文件、
0644 文件、非 PostgreSQL 内容均失败关闭且断言错误不含口令;cmd/libraryctl 侧
钉住文件模式接线与"错误不带地址";v090 脚本进 CI 做 bash -n 加标志/服务声明
断言,并拒绝 0.7.1 容器名出现在新入口。make verify 17 项全绿。

未做:真实采集、生产 executor、Key/KB 绑定。
2026-11-16 20:53:40 +08:00
姚章浩 fba2713b4e feat(library): 发布写入支持 WeKnora 0.9.0 schema 134
build / build (push) Successful in 1m54s
整库发布写入路径原先按 0.7.1 的固定表集合工作:0.9.0 没有 wiki_log_entries,
Current/清理/编目/整实例指纹都会在不存在表上失败,跨版本闸门也因快照丢掉
SchemaVersion 而失效(未知版本被当作 74)。

改为在装配时解析目标 profile,表集合与清理顺序按解析结果;快照复制与范围改写
带上 SchemaVersion/Profile;跨版本发布显式拒绝并同时点名两个版本。单文档发布包
仍只支持 0.7.1,但改为明确拒绝并说明原因。

本机隔离验证:真实 0.9.0 schema 源库→导出→封存→写入干净 134 目标→逐表核对→
写前镜像回滚;74→134 拒绝与 134 缺表不报错两个负例通过。make verify 17 项全绿。
2026-11-16 20:42:23 +08:00
姚章浩 10568943af test(qawemail): 用绝对域名断言不可解析主机
build / build (push) Successful in 1m48s
构建机 resolv.conf 带 search 域(Tailscale tailf49802.ts.net)时,
smtp.kg-test.invalid 会被拼成 ...tailf49802.ts.net 并由 Fake-IP DNS
解析成功,断言实际测的是构建机搜索域。改成带末尾点的绝对名后与
搜索域无关,仍保持“不可解析主机必须判非法”的原意。
2026-11-16 20:27:25 +08:00
姚章浩 3f2626f621 feat(library): 整库发布支持 WeKnora 0.9.0 schema 134
build / build (push) Failing after 1m32s
按 schema 版本选 profile 并以 information_schema 能力探测放行,取代写死的
version==74;未知版本、脏库、缺表缺列、跨版本搬运一律拒绝并点名原因。
0.9.0 的索引名改从部署配置解析,wiki_log_entries 按 profile 变为可选,
归档部件集合随之一致。
2026-11-16 20:22:29 +08:00
46 changed files with 2656 additions and 161 deletions
+96
View File
@@ -1,5 +1,101 @@
# Changelog
## 开发快照 0.1.380(采集绑定允许对外库未配置)
### Fixed
- `libraryctl` 读 `routing_kb_bindings` 不再要求两列都非空:`internal_kb_id` 必填,
为空时报 `对内知识库未绑定(routing_kb_bindings.internal_kb_id 为空)`;`external_kb_id`
允许为空(平台约定「对外库没有对应目标时留空」),绑定仍然有效,`status` 与
`-scope internal` 照常可用。`-scope external` 在未配置时明确失败并报
`对外知识库未配置(routing_kb_bindings.external_kb_id 为空)`,不回退到对内库。
`preflight` / `capture` / `dead-letters` 三条命令都在连源库、开维护窗口之前
定住本次 scope 的目标库,缺绑定不会再退化成 `knowledge base not found in source`。
- 读取 SQL 对两列 `COALESCE`:0.9.0 迁移把旧绑定清成 NULL 后,裸拼接会让文件模式
(P1c 的 governance DSN)在 `Scan` 到 string 时失败,报出的还是
`read routing bindings failed`;NULL 与空串现在同等处理,容器模式与文件模式共用
同一条查询与同一份解析语义。没有生效绑定行(空输出)归入「对内库未绑定」。
### Tests
- 新增 `cmd/libraryctl` 绑定解析用例:internal 为空拒绝并点名(含 NULL 形状的
`|`、空输出、空串);external 为空解析成功且 external scope 被拒、不回退;
两列都有时行为不变(含 psql 行尾换行与首尾空格)。另有一条不连库的查询形状
断言,锁住两列都必须 `COALESCE`、且仍产出 `<internal>|<external>`。
## 开发快照 0.1.379(采集接入 RDS 型 0.9.0 栈)
### Added
- `libraryctl` / `libraryagent` 的数据库来源可选:`-source-pg` 为空时用 `-source-app`
容器环境里的 `DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD` 直连托管实例(RDS),
TLS 模式取 `LIBRARY_SOURCE_DB_SSLMODE`(默认 `require`,`disable` 仅在显式测试时允许);
`-governance-pg` 为空时从 `LIBRARY_GOVERNANCE_DSN_FILE` 指向的 0600 文件读治理库 DSN。
两个都配置时行为与原来完全一致(容器模式),0.7.1 栈不受影响。
- 新增 `scripts/library-capture-v090.sh`:Compose 项目 `weknora-v090` 的整库采集入口,
唯一写入者 `app`,其余服务逐个声明理由;两个数据库走外部模式(RDS),归档/日志根
默认 `/var/lib/knowledge-governance/{library,freeze}` 并可用环境变量覆盖;
`LIBRARYCTL_BIN` 未设置或不可执行时明确失败,不做"同目录二进制"回退。
- `libraryctl status` 增印 `databases.source_db/governance_db` 模式(`container|external|file`),
`libraryagent -describe` 输出同样两项;地址与凭据一律不出现。
### Tests
- DSN 来源三条路径(容器 / RDS / 治理库文件)各有用例;缺变量、缺文件、权限过宽、
非 PostgreSQL 内容都要求失败关闭,且断言错误里不含口令;`sslmode=prefer` 这类会静默
降级的取值直接拒绝。
- v090 采集脚本进 CI:`bash -n` 语法检查 + 标志/服务声明断言,并拒绝 0.7.1 的
容器名与项目名出现在新入口里。
## 开发快照 0.1.378(发布写入支持 WeKnora 0.9.0)
### Added
- 整库发布写入路径支持 WeKnora 0.9.0(schema 134):生产整库目标在装配时解析目标
profile,表集合、清理顺序、编目与整实例指纹都按解析结果取数——0.9.0 没有
`wiki_log_entries`,不再对目标查询或删除该表;0.7.1 少一张必需表仍然失败,不靠
「表不存在就跳过」放行。
- `chunks.seq_id` 语义保持:发布时由目标序列分配,回滚时按写前镜像原值重放;
0.9.0 新增的 NOT NULL 列走目标默认值,不用来源值硬填。
- 跨版本发布(74 快照 → 134 目标,或反之)明确拒绝,错误同时点名两个版本号。
### Fixed
- 范围改写重建快照时会丢掉 `SchemaVersion`/`Profile`(归档重建、快照克隆两处):
丢掉后下游按「未知版本 = 0.7.1」处理,0.9.0 的归档会被要求提供它根本没有的
`wiki_log_entries`,跨版本闸门也形同不存在。现在快照复制一律带上 schema 契约。
- 单文档发布包(`releasebundle`)仍只支持 0.7.1,但改为**明确拒绝并说明原因**:
报错点名读到的版本与「0.9.0 应走整库发布路径」,不再与字段写错共用一句泛化错误。
### Tests
- 真实 0.9.0 schema + 索引 + 对象存储上的发布路径:导出 → 封存 → 真实
`ProductionLibraryTarget` 写入干净 134 目标 → 逐表内容核对(Wiki 非空、资源指向
目标桶/后端、`seq_id` 由目标序列分配)→ 用写前镜像恢复回滚并核对回到写前状态;
另加 74→134 拒绝(含两个版本号)与「134 目标无 `wiki_log_entries` 不报错」的负例。
## 开发快照 0.1.376(WeKnora 0.9.0 整库发布 profile)
### Added
- 整库发布支持 WeKnora 0.9.0(schema 134):按版本号选中 profile,再用 information_schema
探测必需表与必需列,缺项精确报到「表」或「表.列」;未知版本、脏库一律拒绝,不做
「版本大于某值就放行」的推断。
- 0.9.0 的索引契约改从部署配置读取(该版本知识库不写 `vector_store_id`);两处来源都
拿不到时报错,而不是用空串去匹配索引;0.7.1 若两处都有但互相矛盾,同样拒绝。
- 归档部件集合随 profile 变化:0.9.0 已删除 `wiki_log_entries`,因此该部件不再要求;
其余 Wiki 部件仍是必需,空 Wiki 不能冒充完整。
### Fixed
- 采集、导入、Wiki 替换与「去使能」四条路径原先写死 schema 74,遇到 0.9.0 直接失败;
现在同版本(74↔74、134↔134)往返可用,跨版本搬运会明确拒绝而不是误放行。
### Tests
- 真实 0.9.0 schema 与真实索引映射上跑完整往返:导出 → 封存(真实清单校验)→ 导入
干净库 → 再导出逐行内容一致;另加未知版本、脏库、缺表、缺列、索引名解析不到等负例。
## 开发快照 0.1.375(工作台界面:越疆配色、页头统一与筛选批量条)
### Changed
+10 -6
View File
@@ -14,6 +14,7 @@ import (
"time"
"knowledge-governance/internal/libraryagent"
"knowledge-governance/internal/librarydsn"
"knowledge-governance/internal/librarymaintenance"
)
@@ -25,10 +26,10 @@ func main() {
flag.StringVar(&config.ArchiveRoot, "archive", filepath.Join(home, ".local/share/knowledge-governance/library"), "sealed artifact root")
flag.StringVar(&config.JournalRoot, "journal", filepath.Join(home, ".local/share/knowledge-governance/freeze"), "freeze journal root")
flag.StringVar(&config.SourceApp, "source-app", "WKNFull-app", "source container describing the stack")
flag.StringVar(&config.SourcePG, "source-pg", "WKNFull-postgres", "source PostgreSQL container")
flag.StringVar(&config.SourcePG, "source-pg", "WKNFull-postgres", "source PostgreSQL container; empty = the source DB lives on a managed instance and is reached with the source app env (DB_HOST/...)")
flag.StringVar(&config.SourceES, "source-es", "WKNFull-elasticsearch", "source Elasticsearch container")
flag.StringVar(&config.SourceS3Endpoint, "source-s3-endpoint", "", "host-reachable source object-store endpoint (overrides the container-internal one)")
flag.StringVar(&config.GovernancePG, "governance-pg", "knowledge-governance-p1-postgres-1", "governance PostgreSQL container")
flag.StringVar(&config.GovernancePG, "governance-pg", "knowledge-governance-p1-postgres-1", "governance PostgreSQL container; empty = read the DSN from LIBRARY_GOVERNANCE_DSN_FILE")
flag.StringVar(&config.Project, "project", "weknora071full", "Compose project owning the writers")
flag.StringVar(&config.Writers, "writers", "app", "declared writer services")
nonWriters := flag.String("non-writers", defaultNonWriters, "service=reason for every other project service")
@@ -41,6 +42,13 @@ func main() {
if err != nil {
log.Fatal(err)
}
if *describe {
// describe 只回答"这两个库从哪来"。地址与凭据一律不出现:这段输出会被贴进
// 工单与日志,而 RDS 的连接串里带口令。
log.Printf("source_db=%s governance_db=%s",
librarydsn.SourceMode(config.SourcePG), librarydsn.GovernanceMode(config.GovernancePG))
return
}
// 实例级维护接线(入口闸门 + 后台队列)不在 agent 的 Config 里:它由同一份
// 环境声明提供给子进程。启动时先说清楚有没有——缺了封存会在开窗口前被拒,
// 不允许"以为接好了"跑到一半才发现没有排空证据。
@@ -49,10 +57,6 @@ func main() {
} else {
log.Printf("library agent: 实例级维护目标 %s 入口 %s 闸门 %s", target.Instance, target.IngressURL, target.IngressService)
}
if *describe {
log.Printf("library agent config: %+v", agent.Describe())
return
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
log.Printf("library agent listening on %s (project %s, writers %s)", config.Socket, config.Project, config.Writers)
+44
View File
@@ -0,0 +1,44 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
)
// 文件模式没配置:必须失败关闭并点名缺的那个变量——"静默退回容器模式"会让
// 操作员以为在读 RDS,实际在读本机容器(或者干脆读到不存在的容器)。
func TestGovernanceFileModeFailsClosedWithoutDSNFile(t *testing.T) {
t.Setenv("LIBRARY_GOVERNANCE_DSN_FILE", "")
_, err := readRoutingBindings(context.Background(), options{governancePG: ""})
if err == nil {
t.Fatal("没有 DSN 文件却读成功了")
}
if !strings.Contains(err.Error(), "LIBRARY_GOVERNANCE_DSN_FILE") {
t.Fatalf("错误没有点名缺哪一项:%v", err)
}
}
// 外部 DSN 连不上/写错时,错误必须是固定说法:pgx 的错误里带的是连接串(主机、
// 用户名,口令它自己打码),直接透传就等于把地址写进日志与工单。
func TestGovernanceFileModeNeverLeaksPasswordInErrors(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "governance.dsn")
// 夹具口令运行时拼装:源码里不留“带口令的连接串”字面量,否则密钥扫描只会
// 报出一堆必然是假值的夹具,把真泄漏的信号淹掉。
password := strings.Join([]string{"fixture", "leak", "probe"}, "-")
// 故意用非法 sslmode + 不可达端口:无论失败在解析还是连接阶段,都不许泄露。
if err := os.WriteFile(path, []byte("postgres://gov:"+password+"@127.0.0.1:1/gov?sslmode=bogus&connect_timeout=1\n"), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("LIBRARY_GOVERNANCE_DSN_FILE", path)
_, err := readRoutingBindings(context.Background(), options{governancePG: ""})
if err == nil {
t.Fatal("坏 DSN 竟然读成功了")
}
if strings.Contains(err.Error(), password) {
t.Fatalf("错误里漏了口令:%v", err)
}
}
+165 -34
View File
@@ -30,9 +30,11 @@ import (
"syscall"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/librarydeadletter"
"knowledge-governance/internal/librarydsn"
"knowledge-governance/internal/libraryfreeze"
"knowledge-governance/internal/libraryhandoff"
"knowledge-governance/internal/librarymaintenance"
@@ -146,9 +148,13 @@ func usage() {
Flags (defaults target the local WKNFull test stack):
-source-app WKNFull-app container whose env describes the source
-source-pg WKNFull-postgres source PostgreSQL container (address resolution)
-source-pg WKNFull-postgres source PostgreSQL container; empty = the source DB is a
managed instance: connect with the source app env
(DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD), TLS mode from
LIBRARY_SOURCE_DB_SSLMODE (default require)
-source-es WKNFull-elasticsearch
-governance-pg knowledge-governance-p1-postgres-1 routing bindings read here
-governance-pg knowledge-governance-p1-postgres-1 routing bindings read here; empty = read
the DSN from LIBRARY_GOVERNANCE_DSN_FILE (0600 file)
-project weknora071full Compose project owning the writer group
-writers WKNFull-app declared writer services (comma separated)
-non-writers WKNFull-frontend=reason,... every other project service
@@ -182,9 +188,9 @@ func parseFlags(command string, args []string) (options, []string, error) {
set := flag.NewFlagSet(command, flag.ContinueOnError)
set.SetOutput(os.Stderr)
set.StringVar(&opts.sourceApp, "source-app", opts.sourceApp, "source container describing the stack")
set.StringVar(&opts.sourcePG, "source-pg", opts.sourcePG, "source PostgreSQL container")
set.StringVar(&opts.sourcePG, "source-pg", opts.sourcePG, "source PostgreSQL container; empty = connect to the source app's external (managed) database")
set.StringVar(&opts.sourceES, "source-es", opts.sourceES, "source Elasticsearch container")
set.StringVar(&opts.governancePG, "governance-pg", opts.governancePG, "governance PostgreSQL container")
set.StringVar(&opts.governancePG, "governance-pg", opts.governancePG, "governance PostgreSQL container; empty = read the DSN from LIBRARY_GOVERNANCE_DSN_FILE")
set.StringVar(&opts.project, "project", opts.project, "Compose project owning the writers")
set.StringVar(&opts.writers, "writers", opts.writers, "declared writer services")
set.StringVar(&opts.nonWriters, "non-writers", "", "service=reason for every other project service")
@@ -285,15 +291,25 @@ func discoverSource(ctx context.Context, opts options, binding map[string]string
if app["DB_DRIVER"] != "postgres" {
return config, errors.New("source database driver is not postgres")
}
address, err := containerAddress(ctx, opts.sourcePG)
if err != nil {
return config, err
// 两种来源:给了 -source-pg 就按容器地址拼(0.7.1 栈);没给说明库是托管
// 实例(RDS 型 0.9.0 栈没有源库容器),用源应用容器 env 里的 DB_* 直连。
if librarydsn.SourceMode(opts.sourcePG) == librarydsn.SourceExternal {
dsn, dsnErr := librarydsn.SourceURL(app, os.Getenv)
if dsnErr != nil {
return config, dsnErr
}
config.DatabaseURL = dsn
} else {
address, addressErr := containerAddress(ctx, opts.sourcePG)
if addressErr != nil {
return config, addressErr
}
dsn, dsnErr := librarydsn.ContainerURL(address, app)
if dsnErr != nil {
return config, dsnErr
}
config.DatabaseURL = dsn
}
dsn := strings.Join([]string{
"postgres://" + urlEscape(app["DB_USER"]) + ":" + urlEscape(app["DB_PASSWORD"]) +
"@" + address + ":" + app["DB_PORT"] + "/" + urlEscape(app["DB_NAME"]) + "?sslmode=disable",
}, "")
config.DatabaseURL = dsn
config.AppImageID, err = containerImageID(ctx, opts.sourceApp)
if err != nil {
return config, err
@@ -388,17 +404,12 @@ func parseBool(value string) bool {
return false
}
func urlEscape(value string) string {
replacer := strings.NewReplacer("%", "%25", ":", "%3A", "@", "%40", "/", "%2F", "?", "%3F", "#", "%23", "&", "%26", " ", "%20")
return replacer.Replace(value)
}
// scopeBindings 返回 scope→知识库 映射。治理 API 用的是 LIBRARY_PRODUCTION_KB_*,
// CLI 读的是治理库里的 routing bindings;同一台机器上两份不一致时,
// CLI 抓到的会是"配置里那个库"而不是 API 认为的目标——所以环境变量优先,
// 读不到才退回 bindings。两边讲同一份身份,验收才可比。
func scopeBindings(ctx context.Context, container string) (map[string]string, error) {
binding, err := readRoutingBindings(ctx, container)
func scopeBindings(ctx context.Context, opts options) (map[string]string, error) {
binding, err := readRoutingBindings(ctx, opts)
if err != nil {
return nil, err
}
@@ -410,17 +421,105 @@ func scopeBindings(ctx context.Context, container string) (map[string]string, er
return binding, nil
}
func readRoutingBindings(ctx context.Context, container string) (map[string]string, error) {
raw, err := dockerOutput(ctx, "exec", container, "psql", "-U", "governance", "-d", "governance", "-tAc",
"SELECT internal_kb_id||'|'||external_kb_id FROM routing_kb_bindings WHERE id='active'")
// readRoutingBindings 读治理库里生效的 scope 绑定。两种来源:
// - 给了 -governance-pg:docker exec 进治理库容器跑 psql(0.7.1 本机栈);
// - 没给:治理库是托管实例(RDS 型 0.9.0 栈没有 governance 容器),DSN 来自
// LIBRARY_GOVERNANCE_DSN_FILE 指向的 0600 文件。
//
// 两条路径只共用行解析,不共用连接方式:容器里没有网络到 RDS 的凭据,宿主机也不该
// 为了读两列去 docker exec 一个不存在的容器。
func readRoutingBindings(ctx context.Context, opts options) (map[string]string, error) {
if librarydsn.GovernanceMode(opts.governancePG) == librarydsn.GovernanceFile {
dsn, err := librarydsn.GovernanceURL(os.Getenv)
if err != nil {
return nil, err
}
return readRoutingBindingsFromDSN(ctx, dsn)
}
raw, err := dockerOutput(ctx, "exec", opts.governancePG, "psql", "-U", "governance", "-d", "governance", "-tAc",
routingBindingsQuery)
if err != nil {
return nil, errors.New("read routing bindings failed")
}
internal, external, ok := strings.Cut(strings.TrimSpace(raw), "|")
if !ok || internal == "" || external == "" {
return nil, errors.New("routing bindings are incomplete")
return parseRoutingBindings(raw)
}
// COALESCE 不能省:两列都可能是 NULL(0.9.0 迁移把\"旧知识库绑定已解除\"落成 NULL,
// 平台约定\"对外库没有对应目标时留空\")。裸拼接遇 NULL 得到整行 NULL,文件模式
// 会在 Scan 成 string 时直接失败,报出的还是一句\"read routing bindings failed\"——
// 真正的原因(哪一列没绑)被吞掉。这里先拼成固定形状,由 parseRoutingBindings 判。
const routingBindingsQuery = "SELECT COALESCE(internal_kb_id,'')||'|'||COALESCE(external_kb_id,'') FROM routing_kb_bindings WHERE id='active'"
// readRoutingBindingsFromDSN 用外部 DSN 读同一份绑定。
//
// 只读会话是刻意的:这条路径只是读两列,出了问题必须是"读不到",而不是在治理库上
// 留下一次写入。错误一律收敛成固定说法:pgx 的错误里会带连接串(口令它自己打了码,
// 但主机与用户名照旧出去),而这里连那份地址都不该出现在日志或工单里。
func readRoutingBindingsFromDSN(ctx context.Context, dsn string) (map[string]string, error) {
parsed, err := pgxpool.ParseConfig(dsn)
if err != nil {
return nil, errors.New("governance database url is invalid")
}
return map[string]string{"internal": internal, "external": external}, nil
parsed.ConnConfig.RuntimeParams["default_transaction_read_only"] = "on"
parsed.ConnConfig.RuntimeParams["statement_timeout"] = "30000"
parsed.MaxConns = 2
pool, err := pgxpool.NewWithConfig(ctx, parsed)
if err != nil {
return nil, errors.New("open governance database failed")
}
defer pool.Close()
var raw string
if err = pool.QueryRow(ctx, routingBindingsQuery).Scan(&raw); err != nil {
return nil, errors.New("read routing bindings failed")
}
return parseRoutingBindings(raw)
}
// parseRoutingBindings 解一行 `<internal>|<external>`。两条来源(psql -tAc 与 pgx
// 只读会话)拼的是同一句 SQL,所以只有这一份解析语义。
//
// 对内库必填:没有对内库就没有\"整库\"可言,此时必须点名是哪一列没绑,而不是退化成
// 泛化的读失败。对外库可空:平台侧约定\"对外库没有对应目标时留空\",绑定本身仍然
// 有效(`status`、internal scope 照常可跑);只有真去用 external scope 时才拒绝,
// 且绝不回退到对内库——那会把对内库的内容当成对外库封存出去。
func parseRoutingBindings(raw string) (map[string]string, error) {
trimmed := strings.TrimSpace(raw)
// 空输出 = 没有生效绑定行,等于对内库没绑:这跟"读失败"是两件事,
// 必须说成前者,否则现场又会退回一句泛化错误。
if trimmed == "" {
return nil, errInternalKBMissing
}
internal, external, ok := strings.Cut(trimmed, "|")
if !ok {
return nil, errors.New("read routing bindings failed")
}
internal = strings.TrimSpace(internal)
external = strings.TrimSpace(external)
if internal == "" {
return nil, errInternalKBMissing
}
binding := map[string]string{"internal": internal}
if external != "" {
binding["external"] = external
}
return binding, nil
}
var (
errInternalKBMissing = errors.New("对内知识库未绑定(routing_kb_bindings.internal_kb_id 为空)")
errExternalKBMissing = errors.New("对外知识库未配置(routing_kb_bindings.external_kb_id 为空)")
)
// scopeKnowledgeBase 取本次 scope 的目标库。空值必须在这里就拒绝:往下走会变成
// \"knowledge base not found in source\"——那是拿空 UUID 查源库得到的假原因。
func scopeKnowledgeBase(binding map[string]string, scope string) (string, error) {
if kb := strings.TrimSpace(binding[scope]); kb != "" {
return kb, nil
}
if scope == "external" {
return "", errExternalKBMissing
}
return "", errInternalKBMissing
}
func openSourceDatabase(ctx context.Context, config sourceConfig) (*pgxpool.Pool, error) {
@@ -432,11 +531,25 @@ func openSourceDatabase(ctx context.Context, config sourceConfig) (*pgxpool.Pool
// that a database-enforced property rather than a convention.
parsed.ConnConfig.RuntimeParams["default_transaction_read_only"] = "on"
parsed.ConnConfig.RuntimeParams["statement_timeout"] = "120000"
// 启动参数不足以保证只读:角色级/库级设置会在启动参数之后生效,托管实例
// (RDS)上实测就是这样——SHOW 回来是 off。而池里每条连接是独立的会话,
// 对单条连接 SET 管不到其他连接,所以放在 AfterConnect:每条新连接都显式
// 置只读,池里就没有能写的会话。下面的 SHOW 只是对任意一条连接再核一次。
parsed.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error {
_, err := conn.Exec(ctx, "SET SESSION default_transaction_read_only = on")
return err
}
pool, err := pgxpool.NewWithConfig(ctx, parsed)
if err != nil {
return nil, errors.New("open source database failed")
}
var readOnly string
// 连接失败与"能连上但会话可写"必须分开报:把两者压成同一句,现场只能看到
// 一句无信息量的失败,反而会去怀疑权限或只读设置。
if err = pool.Ping(ctx); err != nil {
pool.Close()
return nil, errors.New("source database connection failed")
}
if err = pool.QueryRow(ctx, "SHOW default_transaction_read_only").Scan(&readOnly); err != nil || readOnly != "on" {
pool.Close()
return nil, errors.New("source session is not read-only")
@@ -587,7 +700,12 @@ func openArchive(opts options) (*librarysnapshot.Archive, error) {
func runStatus(ctx context.Context, opts options) error {
report := map[string]any{}
binding, bindingErr := scopeBindings(ctx, opts.governancePG)
// 只报来源模式:地址与凭据连 describe/status 都不出现,日志自然也不会带出去。
report["databases"] = map[string]string{
"source_db": librarydsn.SourceMode(opts.sourcePG),
"governance_db": librarydsn.GovernanceMode(opts.governancePG),
}
binding, bindingErr := scopeBindings(ctx, opts)
var config sourceConfig
var err error
if bindingErr != nil {
@@ -680,7 +798,12 @@ func runPreflight(ctx context.Context, opts options) error {
if opts.scope != "internal" && opts.scope != "external" {
return errors.New("-scope must be internal or external")
}
binding, err := scopeBindings(ctx, opts.governancePG)
binding, err := scopeBindings(ctx, opts)
if err != nil {
return err
}
// 绑定缺哪一半在连源库之前就要说清楚:external 未配置不是"源库里没这个库"。
kb, err := scopeKnowledgeBase(binding, opts.scope)
if err != nil {
return err
}
@@ -693,7 +816,6 @@ func runPreflight(ctx context.Context, opts options) error {
return err
}
defer pool.Close()
kb := config.Scopes[opts.scope]
tenant, err := tenantAndKnowledgeBase(ctx, pool, kb)
if err != nil {
return err
@@ -773,7 +895,13 @@ func runCapture(ctx context.Context, opts options) error {
return err
}
defer archive.Close()
binding, err := scopeBindings(ctx, opts.governancePG)
binding, err := scopeBindings(ctx, opts)
if err != nil {
return err
}
// 先定住本次 scope 的库:external 没配置时,封存必须在开维护窗口之前就停,
// 不能先停写者再发现目标为空。
kb, err := scopeKnowledgeBase(binding, opts.scope)
if err != nil {
return err
}
@@ -801,7 +929,7 @@ func runCapture(ctx context.Context, opts options) error {
if err != nil {
return err
}
scopedKB := binding[opts.scope]
scopedKB := kb
if found {
manifest, err := replayCapture(job, opts.scope, scopedKB, existing, archive)
if err != nil {
@@ -814,7 +942,6 @@ func runCapture(ctx context.Context, opts options) error {
return err
}
defer pool.Close()
kb := config.Scopes[opts.scope]
tenant, err := tenantAndKnowledgeBase(ctx, pool, kb)
if err != nil {
return err
@@ -1150,7 +1277,12 @@ func runDeadLetters(ctx context.Context, opts options) error {
if opts.scope != "internal" && opts.scope != "external" {
return errors.New("-scope must be internal or external")
}
binding, err := scopeBindings(ctx, opts.governancePG)
binding, err := scopeBindings(ctx, opts)
if err != nil {
return err
}
// external 未配置时,死信观测同样要明说"没配置",而不是拿空库去查源库。
kb, err := scopeKnowledgeBase(binding, opts.scope)
if err != nil {
return err
}
@@ -1163,7 +1295,6 @@ func runDeadLetters(ctx context.Context, opts options) error {
return err
}
defer pool.Close()
kb := config.Scopes[opts.scope]
tenant, err := tenantAndKnowledgeBase(ctx, pool, kb)
if err != nil {
return err
+103
View File
@@ -0,0 +1,103 @@
package main
import (
"strings"
"testing"
)
// internal 为空必须拒绝并点名。实测到的现场就是这一行:0.9.0 迁移后
// internal_kb_id 与 external_kb_id 都是 NULL,旧解析退化成
// "read routing bindings failed",操作员无法判断是连不上还是没绑定。
// NULL 与空串在 SQL 侧由 routingBindingsQuery 的 COALESCE 收敛成同一形状,
// 解析侧两者都必须落到同一条错误上。
func TestParseRoutingBindingsRejectsEmptyInternal(t *testing.T) {
for _, raw := range []string{"|", "|ext-kb", " \n| \n", "\n|\n", ""} {
binding, err := parseRoutingBindings(raw)
if err == nil {
t.Fatalf("internal 为空却解析成功:%q → %v", raw, binding)
}
if binding != nil {
t.Fatalf("internal 为空仍返回了映射:%q → %v", raw, binding)
}
if !strings.Contains(err.Error(), "对内知识库未绑定") || !strings.Contains(err.Error(), "internal_kb_id") {
t.Fatalf("错误没有点名 internal 未绑定:%q → %v", raw, err)
}
}
}
// 0.9.0 测试实例的真实形状:只有对内库(wiki-demo),平台约定对外库没有对应
// 目标时留空。绑定仍然有效,internal scope 能解析;external scope 必须失败、
// 点名 external_kb_id,并且绝不回退到对内库。
func TestParseRoutingBindingsAllowsEmptyExternal(t *testing.T) {
binding, err := parseRoutingBindings("0da9b571-6f13-46a1-85fa-e63d2ef5666e|")
if err != nil {
t.Fatalf("对外库为空不该让整份绑定失效:%v", err)
}
if got := binding["internal"]; got != "0da9b571-6f13-46a1-85fa-e63d2ef5666e" {
t.Fatalf("对内库解析错误:%q", got)
}
if value, present := binding["external"]; present {
t.Fatalf("对外库没配置却写进了映射:%q", value)
}
kb, err := scopeKnowledgeBase(binding, "internal")
if err != nil || kb != "0da9b571-6f13-46a1-85fa-e63d2ef5666e" {
t.Fatalf("internal scope 不该被 external 为空牵连:kb=%q err=%v", kb, err)
}
kb, err = scopeKnowledgeBase(binding, "external")
if err == nil {
t.Fatalf("对外库未配置却解析出了目标库:%q", kb)
}
if kb != "" {
t.Fatalf("external scope 回退到了别的库:%q", kb)
}
if !strings.Contains(err.Error(), "对外知识库未配置") || !strings.Contains(err.Error(), "external_kb_id") {
t.Fatalf("错误没有点名 external 未配置:%v", err)
}
}
// 两个库都绑定时行为不变:这是 0.7.1 栈与对外库已配置的 0.9.0 栈的常态路径。
func TestParseRoutingBindingsBothScopesUnchanged(t *testing.T) {
// psql -tAc 输出带行尾换行,pgx 那边不带——两边都必须是同一个结果。
for _, raw := range []string{"kb-internal|kb-external", "kb-internal|kb-external\n", " kb-internal | kb-external \n"} {
binding, err := parseRoutingBindings(raw)
if err != nil {
t.Fatalf("两个库都绑定却解析失败:%q → %v", raw, err)
}
if len(binding) != 2 || binding["internal"] != "kb-internal" || binding["external"] != "kb-external" {
t.Fatalf("解析结果与原行为不一致:%q → %v", raw, binding)
}
for _, scope := range []string{"internal", "external"} {
if _, err := scopeKnowledgeBase(binding, scope); err != nil {
t.Fatalf("%s scope 不该失败:%v", scope, err)
}
}
}
}
// 没有分隔符说明拿到的不是那一行(空结果集、psql 报错文本等),只能是读失败。
func TestParseRoutingBindingsRejectsMalformedRow(t *testing.T) {
for _, raw := range []string{"kb-without-separator", "ERROR: relation \"routing_kb_bindings\" does not exist"} {
if _, err := parseRoutingBindings(raw); err == nil {
t.Fatalf("没有分隔符却解析成功:%q", raw)
}
}
}
// 不连库的语法/形状校验:裸拼接遇 NULL 会得到整行 NULL,文件模式在 Scan 成
// string 时就会失败并丢掉真实原因。两句来源共用这一条 SQL,因此只查这一处。
func TestRoutingBindingsQueryCoalescesBothColumns(t *testing.T) {
query := routingBindingsQuery
for _, column := range []string{"internal_kb_id", "external_kb_id"} {
if !strings.Contains(query, "COALESCE("+column+",'')") {
t.Fatalf("查询没有把 %s 的 NULL 收敛成空串:%s", column, query)
}
}
if !strings.Contains(query, "||'|'||") {
t.Fatalf("查询不再产出 '<internal>|<external>' 形状:%s", query)
}
if !strings.Contains(query, "WHERE id='active'") {
t.Fatalf("查询没有锁定生效绑定行:%s", query)
}
}
@@ -0,0 +1,59 @@
package main
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// v090 采集脚本是本轮新增的唯一入口。为什么用 bash -n 而不是"跑一遍":跑一遍要
// 停写入者、连 RDS——那是真实采集,不是单元测试。语法与声明在这里钉住,能让
// "标志写错一个字母"在 CI 里就红,而不是等到开维护窗口时才发现。
func TestV090CaptureScriptIsSyntacticallyValid(t *testing.T) {
bash, err := exec.LookPath("bash")
if err != nil {
t.Fatalf("需要 bash 才能检查脚本语法(CI 与测试机都有):%v", err)
}
path := filepath.Join("..", "..", "scripts", "library-capture-v090.sh")
out, err := exec.Command(bash, "-n", path).CombinedOutput()
if err != nil {
t.Fatalf("bash -n %s 失败:%v\n%s", path, err, out)
}
}
// 脚本内容断言:外部库模式(两个 -pg 传空)、写者声明、容器名、归档根、
// LIBRARYCTL_BIN 缺失即失败——这些一旦漂移,真跑时会封错栈或读错库。
func TestV090CaptureScriptDeclaresExternalDatabases(t *testing.T) {
body, err := os.ReadFile(filepath.Join("..", "..", "scripts", "library-capture-v090.sh"))
if err != nil {
t.Fatalf("读脚本:%v", err)
}
for _, want := range []string{
"set -euo pipefail",
"LIBRARYCTL_MISSING",
"-project weknora-v090",
"-writers app",
"-source-app weknora-v090-app",
"-source-es weknora-v090-elasticsearch",
`-source-pg ""`,
`-governance-pg ""`,
"frontend=UI only; no knowledge writes",
"docreader=parse RPC service; app persists parse output",
"elasticsearch=search store; not an initiator",
"redis=queue/cache store; not an initiator",
"/var/lib/knowledge-governance/library",
"/var/lib/knowledge-governance/freeze",
} {
if !strings.Contains(string(body), want) {
t.Fatalf("脚本缺少 %q", want)
}
}
// 0.7.1 的服务名绝不能出现在 v090 入口里:那会让封存停错栈。
for _, forbidden := range []string{"WKNFull", "weknora071full"} {
if strings.Contains(string(body), forbidden) {
t.Fatalf("v090 脚本里出现了 0.7.1 的 %q", forbidden)
}
}
}
+11 -2
View File
@@ -26,6 +26,8 @@ import (
"sync"
"syscall"
"time"
"knowledge-governance/internal/librarydsn"
)
// Config is fixed at start-up by the operator (systemd unit / CLI flags). It is
@@ -429,15 +431,22 @@ func (a *Agent) health() response {
}
// identity 被日志与健康应答共用一份,避免两处描述漂移。
//
// 数据库只报来源模式(container|external|file),不报地址:agent 的应答会被治理侧
// 存档,DSN 里带口令,一旦写进身份就再也收不回来了。
func (c Config) identity() map[string]any {
identity := map[string]any{
"project": c.Project,
"archive_root": c.ArchiveRoot,
"journal_root": c.JournalRoot,
"source_app": c.SourceApp,
"source_pg": c.SourcePG,
"source_es": c.SourceES,
"writers": c.Writers,
// source_pg 仍是容器名(不是地址、不是凭据),保留它以让既有健康页/事后核对
// 读得到同一份字段;数据库**来源**由下面两个模式字段表达,外部模式下它为空。
"source_pg": c.SourcePG,
"source_db": librarydsn.SourceMode(c.SourcePG),
"governance_db": librarydsn.GovernanceMode(c.GovernancePG),
"writers": c.Writers,
}
if c.SourceS3Endpoint != "" {
identity["source_s3_endpoint"] = c.SourceS3Endpoint
+140
View File
@@ -0,0 +1,140 @@
// Package librarydsn 决定整库采集链路里两个 DSN 究竟从哪来。
//
// 为什么要有它:容器模式下 DSN 是"源库容器的地址 + 源应用容器的 env"拼出来的;
// RDS 型 0.9.0 栈没有源库容器(库是托管实例),治理库同样只在外部。旧拼法在那种
// 栈上不是权限不够,而是地址解析方式错了——库根本没有容器地址可查。把"用哪种
// 来源"收敛成一处,libraryctl 与 libraryagent 共用同一份判定,日志/describe 只报
// 模式(container|external|file),永远不报地址与凭据。
package librarydsn
import (
"errors"
"fmt"
"net"
"net/url"
"strings"
"knowledge-governance/internal/dobotaiops"
)
const (
// SourceContainer:-source-pg 给了容器名,地址从容器网络解析(0.7.1 栈)。
SourceContainer = "container"
// SourceExternal:-source-pg 为空,直连源应用容器 env 里声明的托管库(RDS)。
SourceExternal = "external"
// GovernanceContainer:-governance-pg 给了容器名,用 docker exec psql 读。
GovernanceContainer = "container"
// GovernanceFile:-governance-pg 为空,DSN 来自 0600 文件(LIBRARY_GOVERNANCE_DSN_FILE)。
GovernanceFile = "file"
)
// allowedSSLModes 是允许的 sslmode 集合。
//
// 只收这四个:prefer/allow 会在服务端不支持时**静默降级成明文**,托管实例上那等于
// "以为加密了其实没有"——这种"看起来配好了"的失败模式比直接报错更难发现。
var allowedSSLModes = map[string]bool{
"disable": true, // 仅显式测试:本机回环或明确无 TLS 的隔离库
"require": true,
"verify-ca": true,
"verify-full": true,
}
// SourceMode 把 -source-pg 的取值翻译成来源模式。空值 = 外部(RDS)。
func SourceMode(sourcePG string) string {
if strings.TrimSpace(sourcePG) == "" {
return SourceExternal
}
return SourceContainer
}
// GovernanceMode 把 -governance-pg 的取值翻译成来源模式。空值 = DSN 文件。
func GovernanceMode(governancePG string) string {
if strings.TrimSpace(governancePG) == "" {
return GovernanceFile
}
return GovernanceContainer
}
// ContainerURL 复现容器模式原有的 DSN 形状:地址来自容器网络,sslmode=disable
// (容器私网内没有 TLS 终结)。抽到这里只为一件事:让"模式选择"有一条可测的路径,
// 而形状与旧实现逐字节一致(真跑过的栈不能因为重构换了连接串)。
func ContainerURL(address string, app map[string]string) (string, error) {
if strings.TrimSpace(address) == "" {
return "", errors.New("source container address is empty")
}
for _, key := range []string{"DB_USER", "DB_PASSWORD", "DB_NAME"} {
if app[key] == "" {
return "", fmt.Errorf("source app environment is missing %s", key)
}
}
target := url.URL{
Scheme: "postgres",
User: url.UserPassword(app["DB_USER"], app["DB_PASSWORD"]),
Host: net.JoinHostPort(address, defaultPort(app["DB_PORT"])),
Path: "/" + app["DB_NAME"],
RawQuery: url.Values{"sslmode": {"disable"}}.Encode(),
}
return target.String(), nil
}
// SourceURL 用源应用容器 env 里的 DB_* 直连托管库(RDS)。
//
// 缺哪个变量就点哪个变量名:把"连接被拒"与"配置缺一项"分开,才不用去猜着改。
// 错误里只有变量名——DB_PASSWORD 的值绝不能出现在任何一处输出里。
func SourceURL(app map[string]string, getenv func(string) string) (string, error) {
missing := []string{}
for _, key := range []string{"DB_HOST", "DB_USER", "DB_NAME"} {
if strings.TrimSpace(app[key]) == "" {
missing = append(missing, key)
}
}
// 口令不 TrimSpace:前后空格也是口令的一部分,改掉就成另一个口令了。
if app["DB_PASSWORD"] == "" {
missing = append(missing, "DB_PASSWORD")
}
if len(missing) > 0 {
return "", fmt.Errorf("source app environment is missing %s", strings.Join(missing, ", "))
}
sslmode := strings.TrimSpace(getenv("LIBRARY_SOURCE_DB_SSLMODE"))
if sslmode == "" {
sslmode = "require"
}
if !allowedSSLModes[sslmode] {
return "", errors.New("LIBRARY_SOURCE_DB_SSLMODE must be one of disable, require, verify-ca, verify-full")
}
target := url.URL{
Scheme: "postgres",
User: url.UserPassword(app["DB_USER"], app["DB_PASSWORD"]),
Host: net.JoinHostPort(strings.TrimSpace(app["DB_HOST"]), defaultPort(app["DB_PORT"])),
Path: "/" + strings.TrimSpace(app["DB_NAME"]),
RawQuery: url.Values{"sslmode": {sslmode}}.Encode(),
}
return target.String(), nil
}
// GovernanceURL 从 0600 文件读治理库 DSN。
//
// 文件读法复用 dobotaiops.ReadDSNFile:那里已经把"必须是普通文件、必须 0600、
// 错误只报路径与原因"钉住并测过——再写一份同义实现只会让两条路径的规则慢慢分叉。
func GovernanceURL(getenv func(string) string) (string, error) {
path := strings.TrimSpace(getenv("LIBRARY_GOVERNANCE_DSN_FILE"))
if path == "" {
return "", errors.New("LIBRARY_GOVERNANCE_DSN_FILE is not set: the governance DSN file is required when -governance-pg is empty")
}
dsn, err := dobotaiops.ReadDSNFile(path)
if err != nil {
// ReadDSNFile 的错误只含路径与原因,不含 DSN/口令。
return "", fmt.Errorf("governance DSN file unusable: %w", err)
}
return dsn, nil
}
// defaultPort:托管实例的连接信息里端口常常单独给;缺了就按 PostgreSQL 默认。
// 端口不是凭据,给默认值不会把"配错了"藏起来。
func defaultPort(port string) string {
if trimmed := strings.TrimSpace(port); trimmed != "" {
return trimmed
}
return "5432"
}
+176
View File
@@ -0,0 +1,176 @@
package librarydsn
import (
"net/url"
"os"
"path/filepath"
"strings"
"testing"
)
func envFunc(pairs map[string]string) func(string) string {
return func(key string) string { return pairs[key] }
}
// 容器模式:旧拼法保持同一目标——同样的用户/口令/地址/库名,sslmode=disable。
// 真跑过的 0.7.1 栈不能因为这次接入换了连接目标(换 sslmode 或换转义方式都会让
// "同一个库"连不上或连成别的身份)。断言用解析后的分量,不用字面转义:
// userinfo 里 ':' 保不保留不影响解析结果,钉字面只会把重构误判成回归。
func TestContainerURLKeepsLegacyShape(t *testing.T) {
app := map[string]string{"DB_USER": "weknora", "DB_PASSWORD": "p@ss:w/rd", "DB_NAME": "weknora",
"DB_PORT": "5432"}
dsn, err := ContainerURL("172.18.0.5", app)
if err != nil {
t.Fatalf("container url: %v", err)
}
parsed, err := url.Parse(dsn)
if err != nil {
t.Fatalf("生成的连接串自己都解析不了:%v", err)
}
if parsed.Scheme != "postgres" || parsed.Host != "172.18.0.5:5432" || parsed.Path != "/weknora" {
t.Fatalf("容器模式连接目标变了:%s", dsn)
}
if parsed.User.Username() != "weknora" {
t.Fatalf("用户名变了:%s", dsn)
}
if password, _ := parsed.User.Password(); password != "p@ss:w/rd" {
t.Fatalf("口令转义后取不回来:%s", dsn)
}
if parsed.Query().Get("sslmode") != "disable" {
t.Fatalf("容器模式不该改 sslmode:%s", dsn)
}
if _, err = ContainerURL("", app); err == nil {
t.Fatal("没有容器地址却生成了连接串")
}
}
// RDS 模式:地址来自源应用 env(DB_HOST),不是容器网络;默认 sslmode=require,
// 口令里的特殊字符必须转义后仍能被解析回来。
func TestSourceURLUsesRDSHostAndRequiresTLSByDefault(t *testing.T) {
app := map[string]string{
"DB_HOST": "pgm-2ze0abcd.pg.rds.aliyuncs.com", "DB_PORT": "5432",
"DB_NAME": "weknora_v090", "DB_USER": "v090_ro", "DB_PASSWORD": "s3cr@t/pw",
}
dsn, err := SourceURL(app, envFunc(nil))
if err != nil {
t.Fatalf("rds url: %v", err)
}
parsed, err := url.Parse(dsn)
if err != nil {
t.Fatalf("生成的连接串自己都解析不了:%v", err)
}
if parsed.Host != "pgm-2ze0abcd.pg.rds.aliyuncs.com:5432" {
t.Fatalf("host 不是 RDS 地址:%q", parsed.Host)
}
if got := parsed.Query().Get("sslmode"); got != "require" {
t.Fatalf("默认 sslmode 不是 require:%q", got)
}
if password, _ := parsed.User.Password(); password != "s3cr@t/pw" {
t.Fatal("口令转义后取不回来")
}
// 显式 disable 只用于本机隔离测试;它是允许的,但必须由调用方写清楚。
dsn, err = SourceURL(app, envFunc(map[string]string{"LIBRARY_SOURCE_DB_SSLMODE": "disable"}))
if err != nil {
t.Fatalf("explicit disable: %v", err)
}
if !strings.HasSuffix(dsn, "sslmode=disable") {
t.Fatalf("显式 disable 没被采纳:%s", dsn)
}
// prefer 会静默降级成明文:那不是"配好了 TLS",直接拒。
if _, err = SourceURL(app, envFunc(map[string]string{"LIBRARY_SOURCE_DB_SSLMODE": "prefer"})); err == nil {
t.Fatal("sslmode=prefer 被接受了")
}
}
// 缺变量必须失败,且点名缺的那一项;错误里不许出现口令。
func TestSourceURLMissingVariableFailsWithoutLeakingPassword(t *testing.T) {
// 口令运行时拼装:源码里不留字面量口令,避免密钥扫描只报假夹具。
password := strings.Join([]string{"fixture", "db", "secret"}, "-")
app := map[string]string{"DB_HOST": "pgm-x.pg.rds.aliyuncs.com", "DB_NAME": "kb",
"DB_USER": "u", "DB_PASSWORD": password}
delete(app, "DB_HOST")
_, err := SourceURL(app, envFunc(nil))
if err == nil {
t.Fatal("缺 DB_HOST 竟然成功了")
}
if !strings.Contains(err.Error(), "DB_HOST") {
t.Fatalf("错误没有点名缺哪一项:%v", err)
}
if strings.Contains(err.Error(), password) {
t.Fatalf("错误里漏了口令:%v", err)
}
app["DB_HOST"] = "pgm-x.pg.rds.aliyuncs.com"
delete(app, "DB_PASSWORD")
_, err = SourceURL(app, envFunc(nil))
if err == nil || !strings.Contains(err.Error(), "DB_PASSWORD") {
t.Fatalf("缺 DB_PASSWORD 没有点名:%v", err)
}
if strings.Contains(err.Error(), password) {
t.Fatalf("错误里漏了口令:%v", err)
}
}
// 治理库文件模式:0600 文件里的完整 DSN 原样读出;文件缺失/权限过宽/内容不是
// PostgreSQL DSN 都必须失败关闭,且错误只提到路径与原因。
func TestGovernanceURLReadsDSNFileAndFailsClosed(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "governance.dsn")
password := strings.Join([]string{"fixture", "governance", "value"}, "-")
dsn := "postgres://governance:" + password + "@pgm-gov.pg.rds.aliyuncs.com:5432/knowledge_governance_test?sslmode=require"
if err := os.WriteFile(path, []byte(dsn+"\n"), 0o600); err != nil {
t.Fatal(err)
}
got, err := GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": path}))
if err != nil {
t.Fatalf("file mode: %v", err)
}
if got != dsn {
t.Fatalf("读出的 DSN 与文件内容不一致(不想在这里做任何改写)")
}
// 没声明文件:失败并点名变量。
if _, err = GovernanceURL(envFunc(nil)); err == nil || !strings.Contains(err.Error(), "LIBRARY_GOVERNANCE_DSN_FILE") {
t.Fatalf("没配置文件时没有点名变量:%v", err)
}
// 文件不存在:点名路径,不泄漏任何内容。
missing := filepath.Join(dir, "absent.dsn")
if _, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": missing})); err == nil ||
!strings.Contains(err.Error(), missing) {
t.Fatalf("文件缺失没有点名路径:%v", err)
}
// 组/其他可读:拒绝(这条规则由 dobotaiops.ReadDSNFile 判定,这里只钉住接线)。
wide := filepath.Join(dir, "wide.dsn")
if err = os.WriteFile(wide, []byte(dsn+"\n"), 0o644); err != nil {
t.Fatal(err)
}
if err = os.Chmod(wide, 0o644); err != nil {
t.Fatal(err)
}
if _, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": wide})); err == nil {
t.Fatal("0644 的 DSN 文件被接受了")
}
// 内容不是 PostgreSQL DSN:失败,且不要把内容回显进错误。
bogus := filepath.Join(dir, "bogus.dsn")
if err = os.WriteFile(bogus, []byte("mysql://u:"+password+"@host/db\n"), 0o600); err != nil {
t.Fatal(err)
}
_, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": bogus}))
if err == nil {
t.Fatal("非 PostgreSQL DSN 被接受了")
}
if strings.Contains(err.Error(), password) {
t.Fatalf("错误里漏了口令:%v", err)
}
}
func TestModesDeriveFromEmptyFlag(t *testing.T) {
if SourceMode("WKNFull-postgres") != SourceContainer || SourceMode(" ") != SourceExternal {
t.Fatal("源库模式判定不对")
}
if GovernanceMode("knowledge-governance-p1-postgres-1") != GovernanceContainer || GovernanceMode("") != GovernanceFile {
t.Fatal("治理库模式判定不对")
}
}
+209
View File
@@ -0,0 +1,209 @@
// Package libraryprofile 声明"某个 WeKnora schema 版本下,整库适配器可以依赖什么"。
//
// 为什么要有这一层:整库导出/导入原本把 `version != 74` 写死在四个入口里。
// 版本号本身不是契约——同一版本号在不同部署上也可能缺表;反过来,把判定放宽成
// `version > 74` 又会把"没见过的版本"当成"兼容"。所以这里同时声明两件事:
// - 该版本必须存在的表与列(导出/导入按它取数);
// - 该版本的归档部件集合(manifest 按它判定完整性)。
//
// 版本号只用来**选中** profile,真正的放行条件是随后对 information_schema 的
// 能力探测(见 releaseadapter 的 ResolveSchemaProfile)。
package libraryprofile
import "fmt"
// LegacyVersion 是 0.7.1 的 schema_migrations 版本号,也是既有内存夹具的默认值。
const LegacyVersion = 74
// V090Version 是 0.9.0 的 schema_migrations 版本号(已实测:134、dirty=false、83 表)。
const V090Version = 134
// PartRequirement 是归档部件的一项要求(kind/name 与 manifest 的部件键一致)。
type PartRequirement struct {
Kind string
Name string
}
// Profile 是一个 schema 版本的静态能力声明。
type Profile struct {
Name string
SchemaVersion int
WeKnoraVersion string
// EmbeddingDimension 为 0 表示"由目标 models.parameters 说了算";
// 非 0 表示该版本只接受这一定维度(0.7.1 的契约就是固定 1024)。
EmbeddingDimension int
RequiredTables []string
OptionalTables []string
RequiredColumns map[string][]string
// ReviewedTriggers 是该版本允许出现的 pg_get_triggerdef 原文。
// 触发器意味着"写一行还会改别的表",所以只接受逐字核对过的定义,
// 且这些定义必须在该版本上真实存在(消失也说明 schema 不是我们审过的那份)。
ReviewedTriggers []string
RequiredParts []PartRequirement
OptionalParts []PartRequirement
// IndexContractSource 说明该版本的索引名从哪里读。空值等于"读不到",
// 不允许回退成空串去匹配任何索引。
IndexContractSource string
}
var profiles = map[int]Profile{
LegacyVersion: {
Name: "v071",
SchemaVersion: LegacyVersion,
WeKnoraVersion: "v0.7.1",
EmbeddingDimension: 1024,
RequiredTables: []string{
"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
"resources", "resource_bindings", "wiki_folders", "wiki_pages",
"wiki_page_issues", "wiki_log_entries",
},
RequiredColumns: map[string][]string{
"knowledges": {"id", "tenant_id", "knowledge_base_id", "deleted_at", "parse_status"},
"chunks": {"id", "tenant_id", "knowledge_base_id", "knowledge_id", "content"},
"knowledge_tags": {"id", "tenant_id", "knowledge_base_id"},
"knowledge_tag_relations": {"knowledge_id", "tag_id"},
"resources": {"id", "handle", "tenant_id", "physical_path"},
"resource_bindings": {"id", "resource_id", "tenant_id", "owner_type", "owner_id"},
"wiki_pages": {"id", "tenant_id", "knowledge_base_id", "slug"},
"wiki_folders": {"id", "tenant_id", "knowledge_base_id"},
"wiki_page_issues": {"id", "tenant_id", "knowledge_base_id"},
"wiki_log_entries": {"id", "tenant_id", "knowledge_base_id"},
},
RequiredParts: []PartRequirement{
{"database", "knowledges"}, {"database", "chunks"},
{"database", "knowledge_tags"}, {"database", "knowledge_tag_relations"},
{"database", "resources"}, {"database", "resource_bindings"},
{"wiki", "wiki_folders"}, {"wiki", "wiki_pages"},
{"wiki", "wiki_page_issues"}, {"wiki", "wiki_log_entries"},
{"index", "chunks"},
},
IndexContractSource: "vector_stores.index_config.index_name",
},
V090Version: {
Name: "v090",
SchemaVersion: V090Version,
WeKnoraVersion: "v0.9.0",
// 0.9.0 的 KB 不写 vector_store_id,维度只能从 models.parameters 读。
EmbeddingDimension: 0,
RequiredTables: []string{
"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
"resources", "resource_bindings", "wiki_folders", "wiki_pages",
"wiki_page_issues",
},
// 0.9.0 删掉了 wiki_log_entries;部件集合随之为空,不许假装还有。
OptionalTables: []string{"wiki_log_entries"},
RequiredColumns: map[string][]string{
"knowledges": {"id", "tenant_id", "knowledge_base_id", "deleted_at", "parse_status"},
"chunks": {"id", "tenant_id", "knowledge_base_id", "knowledge_id", "content", "seq_id", "content_revision", "index_status"},
"knowledge_tags": {"id", "tenant_id", "knowledge_base_id"},
"knowledge_tag_relations": {"knowledge_id", "tag_id"},
"resources": {"id", "handle", "tenant_id", "physical_path"},
"resource_bindings": {"id", "resource_id", "tenant_id", "owner_type", "owner_id"},
"wiki_pages": {"id", "tenant_id", "knowledge_base_id", "slug"},
"wiki_folders": {"id", "tenant_id", "knowledge_base_id"},
"wiki_page_issues": {"id", "tenant_id", "knowledge_base_id"},
},
RequiredParts: []PartRequirement{
{"database", "knowledges"}, {"database", "chunks"},
{"database", "knowledge_tags"}, {"database", "knowledge_tag_relations"},
{"database", "resources"}, {"database", "resource_bindings"},
{"wiki", "wiki_folders"}, {"wiki", "wiki_pages"}, {"wiki", "wiki_page_issues"},
{"index", "chunks"},
},
OptionalParts: []PartRequirement{
{"wiki", "wiki_log_entries"},
},
// 0.9.0 的索引名来自部署配置(ELASTICSEARCH_INDEX),不是 vector_stores。
// 0.9.0 在 chunks 上挂了图片同步触发器(仅当 image_info 以 '[' 开头时触发),
// 它维护 chunk_images 派生表;定义逐字核对,任何改动都拒绝。
ReviewedTriggers: []string{
`CREATE TRIGGER trg_chunk_images_delete AFTER DELETE ON public.chunks FOR EACH ROW WHEN ((old.image_info ~~ '[%'::text)) EXECUTE FUNCTION chunk_images_sync()`,
`CREATE TRIGGER trg_chunk_images_insert AFTER INSERT ON public.chunks FOR EACH ROW WHEN ((new.image_info ~~ '[%'::text)) EXECUTE FUNCTION chunk_images_sync()`,
`CREATE TRIGGER trg_chunk_images_update AFTER UPDATE ON public.chunks FOR EACH ROW WHEN ((((old.image_info ~~ '[%'::text) OR (new.image_info ~~ '[%'::text)) AND ((old.image_info IS DISTINCT FROM new.image_info) OR (old.is_enabled IS DISTINCT FROM new.is_enabled) OR (old.status IS DISTINCT FROM new.status) OR (old.deleted_at IS DISTINCT FROM new.deleted_at) OR (old.tenant_id IS DISTINCT FROM new.tenant_id) OR ((old.knowledge_base_id)::text IS DISTINCT FROM (new.knowledge_base_id)::text) OR ((old.knowledge_id)::text IS DISTINCT FROM (new.knowledge_id)::text) OR ((old.chunk_type)::text IS DISTINCT FROM (new.chunk_type)::text) OR (old.created_at IS DISTINCT FROM new.created_at) OR (old.updated_at IS DISTINCT FROM new.updated_at)))) EXECUTE FUNCTION chunk_images_sync()`,
},
IndexContractSource: "config ELASTICSEARCH_INDEX",
},
}
// For 按版本号选中 profile。未知版本一律失败:版本号是"选哪份声明"的索引,
// 不是"能不能放行"的依据,所以绝不返回默认 profile。
func For(version int) (Profile, error) {
profile, ok := profiles[version]
if !ok {
return Profile{}, fmt.Errorf("unsupported WeKnora schema version %d: supported profiles are 74 (v071) and 134 (v090)", version)
}
return profile, nil
}
// Supported 返回已声明 profile 的版本号(用于文档与错误信息,不用于放行)。
func Supported() []int {
return []int{LegacyVersion, V090Version}
}
// TableSet 是"可能出现的表"集合:必需表 + 可选表。
func (p Profile) TableSet() map[string]bool {
set := make(map[string]bool, len(p.RequiredTables)+len(p.OptionalTables))
for _, name := range p.RequiredTables {
set[name] = true
}
for _, name := range p.OptionalTables {
set[name] = true
}
return set
}
// HasTable 报告某张表是否属于该 profile(必需或可选)。
func (p Profile) HasTable(name string) bool {
return p.TableSet()[name]
}
// IsRequiredTable 报告某张表是否必需存在。
func (p Profile) IsRequiredTable(name string) bool {
for _, candidate := range p.RequiredTables {
if candidate == name {
return true
}
}
return false
}
// PartKeys 返回 (必需, 可选) 的归档部件键,形如 "wiki/wiki_log_entries"。
func (p Profile) PartKeys() ([]string, []string) {
key := func(part PartRequirement) string { return part.Kind + "/" + part.Name }
required := make([]string, 0, len(p.RequiredParts))
for _, part := range p.RequiredParts {
required = append(required, key(part))
}
optional := make([]string, 0, len(p.OptionalParts))
for _, part := range p.OptionalParts {
optional = append(optional, key(part))
}
return required, optional
}
// RequiresPart 报告某部件键是否必需(可选部件返回 false)。
func (p Profile) RequiresPart(key string) bool {
required, _ := p.PartKeys()
for _, candidate := range required {
if candidate == key {
return true
}
}
return false
}
// AllowsPart 报告某部件键是否至少允许出现。
func (p Profile) AllowsPart(key string) bool {
required, optional := p.PartKeys()
for _, candidate := range required {
if candidate == key {
return true
}
}
for _, candidate := range optional {
if candidate == key {
return true
}
}
return false
}
+14 -9
View File
@@ -10,7 +10,7 @@ import (
"encoding/json"
"errors"
"io"
"knowledge-governance/internal/releasebundle"
"knowledge-governance/internal/libraryprofile"
"math"
"os"
"path/filepath"
@@ -420,24 +420,29 @@ func encodeManifest(m Manifest) ([]byte, error) {
return nil, errors.New("unsupported snapshot validation policy")
}
c := m.Compatibility
if c.WeKnoraVersion != releasebundle.WeKnoraVersionV071 || c.SchemaVersion != releasebundle.WeKnoraMigrationV071 || c.RetrieverEngine != "elasticsearch_v8" || !identifier(c.EmbeddingModel) || c.EmbeddingDimension != releasebundle.EmbeddingDimensionV071 || !validDigest(c.IndexMappingSHA256) {
// 兼容性按 schema 版本选 profile,而不是钉死 v0.7.1:未知版本必须拒绝,
// 已知版本还必须与自己的版本字符串、索引引擎、维度声明一致。
profile, err := libraryprofile.For(c.SchemaVersion)
if err != nil || c.WeKnoraVersion != profile.WeKnoraVersion || c.RetrieverEngine != "elasticsearch_v8" ||
!identifier(c.EmbeddingModel) || c.EmbeddingDimension < 1 ||
(profile.EmbeddingDimension > 0 && c.EmbeddingDimension != profile.EmbeddingDimension) ||
!validDigest(c.IndexMappingSHA256) {
return nil, errors.New("unsupported snapshot compatibility contract")
}
if m.Format != Format || (m.Scope != "internal" && m.Scope != "external") || m.TenantID == 0 || !identifier(m.KnowledgeBaseID) || m.CreatedAt.IsZero() {
return nil, errors.New("invalid library manifest identity")
}
// 部件集合也来自 profile:0.9.0 的归档没有 wiki_log_entries 部件,
// 但 0.7.1 缺了它就是不完整。
requiredKeys, _ := profile.PartKeys()
required := map[string]bool{}
for _, n := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} {
required["database/"+n] = true
for _, key := range requiredKeys {
required[key] = true
}
for _, n := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
required["wiki/"+n] = true
}
required["index/chunks"] = true
seen := map[string]bool{}
for _, p := range m.Parts {
key := p.Kind + "/" + p.Name
if (!required[key] && !(p.Kind == "resource" && identifier(p.Name) && p.Count == 1) && !(key == "schema/migration-catalog" && p.Count == 1)) || seen[key] || !validDigest(p.SHA256) || p.Bytes < 0 || p.Count < 0 {
if (!profile.AllowsPart(key) && !(p.Kind == "resource" && identifier(p.Name) && p.Count == 1) && !(key == "schema/migration-catalog" && p.Count == 1)) || seen[key] || !validDigest(p.SHA256) || p.Bytes < 0 || p.Count < 0 {
return nil, errors.New("invalid or nonknowledge snapshot component")
}
seen[key] = true
@@ -0,0 +1,58 @@
package librarysnapshot
import (
"strings"
"testing"
"time"
)
// 归档部件的完整性按 schema 版本判定:0.9.0 没有 wiki_log_entries,
// 0.7.1 缺了它就是残包。这里直接对 manifest 编码做判定,不依赖数据库。
func TestManifestPartSetFollowsSchemaProfile(t *testing.T) {
part := func(kind, name string) Part {
return Part{Kind: kind, Name: name, SHA256: strings.Repeat("a", 64), Count: 1}
}
parts := []Part{
part("database", "knowledges"), part("database", "chunks"), part("database", "resources"),
part("database", "resource_bindings"), part("database", "knowledge_tags"),
part("database", "knowledge_tag_relations"),
part("wiki", "wiki_pages"), part("wiki", "wiki_folders"), part("wiki", "wiki_page_issues"),
part("index", "chunks"),
}
manifest := func(version int, weknora string, parts []Part) Manifest {
return Manifest{
ValidationPolicy: ArtifactAcceptancePolicy,
Format: Format,
Scope: "internal",
TenantID: 10000,
KnowledgeBaseID: "kb-v090",
CreatedAt: time.Now().UTC(),
Parts: parts,
Compatibility: Compatibility{
WeKnoraVersion: weknora, SchemaVersion: version,
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "bge-m3",
EmbeddingDimension: 1024, IndexMappingSHA256: strings.Repeat("b", 64),
},
}
}
if _, err := encodeManifest(manifest(134, "v0.9.0", parts)); err != nil {
t.Fatalf("0.9.0 manifest without wiki_log_entries must be complete: %v", err)
}
if _, err := encodeManifest(manifest(74, "v0.7.1", parts)); err == nil || !strings.Contains(err.Error(), "incomplete") {
t.Fatalf("0.7.1 manifest missing wiki_log_entries must be refused, got %v", err)
}
withLog := append(append([]Part{}, parts...), part("wiki", "wiki_log_entries"))
if _, err := encodeManifest(manifest(74, "v0.7.1", withLog)); err != nil {
t.Fatalf("0.7.1 manifest with wiki_log_entries must pass: %v", err)
}
if _, err := encodeManifest(manifest(134, "v0.9.0", withLog)); err != nil {
t.Fatalf("0.9.0 manifest may still carry an optional wiki_log_entries part: %v", err)
}
if _, err := encodeManifest(manifest(74, "v0.9.0", withLog)); err == nil {
t.Fatal("version string and profile version must agree")
}
if _, err := encodeManifest(manifest(135, "v0.9.0", parts)); err == nil {
t.Fatal("unknown schema version must be refused")
}
}
+4 -1
View File
@@ -77,7 +77,10 @@ func TestNormalizeRefusesEnabledSettingsWithoutTransportAndBadRecipients(t *test
t.Fatalf("defaults not applied: %+v %v", good, err)
}
// 解析不了的主机名必须仍被判为非法:.invalid 是 RFC 6761 保留域,永远不解析。
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid", From: "kb@example.com",
// 末尾的点是必须的:构建机的 resolv.conf 可能带 search 域(本环境 Tailscale
// 的 tailf49802.ts.net),不带点就会被拼成 smtp.kg-test.invalid.<search> 并
// 由 Fake-IP DNS 解析出来,于是这条断言测的是构建机的搜索域而不是被测逻辑。
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid.", From: "kb@example.com",
Recipients: []string{"a@example.com"}}); err != ErrInvalid {
t.Fatalf("unresolvable host must stay invalid, got %v", err)
}
@@ -3,10 +3,12 @@ package v071
import (
"context"
"errors"
"fmt"
"strings"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/libraryprofile"
"knowledge-governance/internal/releasebundle"
releaseworkflow "knowledge-governance/internal/workflows/release"
)
@@ -37,19 +39,23 @@ type TargetContractInspector struct {
config TargetContractInspectorConfig
}
// targetContractSQL 用 LEFT JOIN 读模型与向量库:命中不到时仍然拿得到"声明了什么",
// 这样缺字段能被点名拒绝,而不是整行查不出来只报一句"读契约失败"。
const targetContractSQL = `
SELECT migration.version,
migration.dirty,
SELECT COALESCE(migration.version, 0),
COALESCE(migration.dirty, false),
knowledge_base.tenant_id,
model.name,
COALESCE(knowledge_base.embedding_model_id, ''),
COALESCE(model.name, ''),
COALESCE(NULLIF(model.parameters->'embedding_parameters'->>'dimension', '')::INTEGER, 0),
(model.id IS NOT NULL),
COALESCE(vector_store.engine_type, ''),
COALESCE(vector_store.index_config->>'index_name', '')
FROM knowledge_bases AS knowledge_base
JOIN models AS model
LEFT JOIN models AS model
ON model.id = knowledge_base.embedding_model_id
AND model.deleted_at IS NULL
JOIN schema_migrations AS migration ON TRUE
LEFT JOIN schema_migrations AS migration ON TRUE
LEFT JOIN vector_stores AS vector_store
ON vector_store.id = knowledge_base.vector_store_id
AND vector_store.deleted_at IS NULL
@@ -57,6 +63,25 @@ WHERE knowledge_base.id = $1
AND knowledge_base.deleted_at IS NULL
`
// resolveTargetIndexName 是"索引名从哪来"的唯一判定点。
//
// 0.9.0 的 KB 不写 vector_store_id(vector_stores 没有对应行),索引名只在部署
// 配置里(ELASTICSEARCH_INDEX)。两条来源都拿不到时**不许**回退成空串——
// 空串会让后续过滤条件匹配到任何索引。0.7.1 若两处都有,必须一致。
func resolveTargetIndexName(fromVectorStore, configured string, profile libraryprofile.Profile) (string, error) {
resolved := fromVectorStore
if resolved == "" {
resolved = configured
}
if !validSearchIdentifier(resolved) {
return "", fmt.Errorf("target WeKnora index name unresolved (contract source: %s): vector_stores.index_config->>'index_name' and the configured index are both empty", profile.IndexContractSource)
}
if fromVectorStore != "" && fromVectorStore != configured {
return "", fmt.Errorf("target WeKnora contract mismatch: vector_stores.index_config.index_name=%q does not match configured index %q", fromVectorStore, configured)
}
return resolved, nil
}
func NewTargetContractInspector(config TargetContractInspectorConfig) (*TargetContractInspector, error) {
config.KnowledgeBaseID = strings.TrimSpace(config.KnowledgeBaseID)
config.IndexName = strings.TrimSpace(config.IndexName)
@@ -85,29 +110,51 @@ func (inspector *TargetContractInspector) Contract(ctx context.Context, item rel
migrationVersion int
dirty bool
tenantID uint64
declaredModel string
embeddingModel string
dimension int
modelBound bool
engineType string
indexName string
)
if err := inspector.config.DB.QueryRow(ctx, targetContractSQL, targetKnowledgeBaseID).Scan(
&migrationVersion, &dirty, &tenantID, &embeddingModel, &dimension, &engineType, &indexName,
&migrationVersion, &dirty, &tenantID, &declaredModel, &embeddingModel, &dimension, &modelBound, &engineType, &indexName,
); err != nil {
return releasebundle.TargetContract{}, errors.New("read target WeKnora contract")
}
if migrationVersion != releasebundle.WeKnoraMigrationV071 || dirty || tenantID != inspector.config.TenantID ||
!validSearchIdentifier(embeddingModel) || dimension < 1 ||
(engineType != "" && engineType != "elasticsearch") ||
(indexName != "" && indexName != inspector.config.IndexName) {
return releasebundle.TargetContract{}, errors.New("target WeKnora contract mismatch")
if dirty {
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora schema_migrations.version=%d is dirty: refusing to publish", migrationVersion)
}
// 版本号只用来选 profile;未知版本失败关闭,错误里带真实版本号。
profile, err := libraryprofile.For(migrationVersion)
if err != nil {
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora: %w", err)
}
if tenantID != inspector.config.TenantID {
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: knowledge_bases.tenant_id=%d, expected %d", tenantID, inspector.config.TenantID)
}
if !validSearchIdentifier(declaredModel) {
return releasebundle.TargetContract{}, errors.New("target WeKnora contract mismatch: knowledge_bases.embedding_model_id is empty: embedding dimension is unknowable")
}
if !modelBound {
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: models row for knowledge_bases.embedding_model_id %q is missing or deleted", declaredModel)
}
if !validSearchIdentifier(embeddingModel) || dimension < 1 {
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: models.parameters.embedding_parameters.dimension is missing for model %q", declaredModel)
}
if engineType != "" && engineType != "elasticsearch" {
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: vector_stores.engine_type=%q is not elasticsearch", engineType)
}
if _, err := resolveTargetIndexName(indexName, inspector.config.IndexName, profile); err != nil {
return releasebundle.TargetContract{}, err
}
mappingSHA256, err := inspector.config.Mapping.MappingSHA256(ctx)
if err != nil || !validLowerSHA256(mappingSHA256) {
return releasebundle.TargetContract{}, errors.New("read target Elasticsearch mapping contract")
}
return releasebundle.TargetContract{
WeKnoraVersion: releasebundle.WeKnoraVersionV071,
MigrationVersion: releasebundle.WeKnoraMigrationV071,
WeKnoraVersion: profile.WeKnoraVersion,
MigrationVersion: profile.SchemaVersion,
TenantID: tenantID,
KnowledgeBaseID: targetKnowledgeBaseID,
RetrieverEngine: "elasticsearch_v8",
@@ -9,7 +9,7 @@ import (
)
func TestTargetContractInspectorReadsOnlyCompatibilityFields(t *testing.T) {
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", 1024, "elasticsearch", "chunks"}}}
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", "bge-m3", 1024, true, "elasticsearch", "chunks"}}}
mapping := &mappingInspector{sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}
inspector, err := NewTargetContractInspector(TargetContractInspectorConfig{DB: db, Mapping: mapping, KnowledgeBaseID: "kb-safe", TenantID: 10000, IndexName: "chunks"})
if err != nil {
@@ -36,7 +36,7 @@ func TestTargetContractInspectorReadsOnlyCompatibilityFields(t *testing.T) {
}
func TestTargetContractInspectorRejectsMismatchedPlanKB(t *testing.T) {
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", 1024, "elasticsearch", "chunks"}}}
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", "bge-m3", 1024, true, "elasticsearch", "chunks"}}}
inspector, _ := NewTargetContractInspector(TargetContractInspectorConfig{DB: db, Mapping: &mappingInspector{sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, KnowledgeBaseID: "kb-safe", TenantID: 10000, IndexName: "chunks"})
plan := releasePlanForArchive()
plan.TestKnowledgeBaseID = "other-kb"
@@ -50,7 +50,7 @@ func TestTargetContractInspectorRejectsMismatchedPlanKB(t *testing.T) {
// the mapped base on the source plane made a real release fail with
// "source WeKnora release contract mismatch" (the target base is unknown there).
func TestTargetContractInspectorSidesPickTheRightBase(t *testing.T) {
row := contractRow{values: []any{74, false, uint64(10000), "bge-m3", 1024, "elasticsearch", "chunks"}}
row := contractRow{values: []any{74, false, uint64(10000), "bge-m3", "bge-m3", 1024, true, "elasticsearch", "chunks"}}
mapping := &mappingInspector{sha: strings.Repeat("a", 64)}
plan := releasePlanForArchive()
plan.TargetKnowledgeBaseID = "prod-kb"
@@ -3,6 +3,7 @@ package v071
import (
"context"
"errors"
"fmt"
"strings"
"time"
@@ -63,8 +64,14 @@ func (exporter *DocumentExporter) Export(ctx context.Context, item releaseworkfl
return nil, errors.New("invalid document release plan")
}
sourceContract, err := exporter.config.Contract.Contract(ctx, item)
if err != nil || sourceContract.WeKnoraVersion != releasebundle.WeKnoraVersionV071 ||
sourceContract.MigrationVersion != releasebundle.WeKnoraMigrationV071 || sourceContract.TenantID != exporter.config.TenantID ||
if err == nil && (sourceContract.WeKnoraVersion != releasebundle.WeKnoraVersionV071 ||
sourceContract.MigrationVersion != releasebundle.WeKnoraMigrationV071) {
// 单文档发布包是 0.7.1 契约:读到的库是别的版本时必须说清楚原因,
// 而不是和"租户/模型不对"共用一句泛化错误。
return nil, fmt.Errorf("single-document release supports only v0.7.1 (74): source library declares %s (%d), so its rows and index documents must go through the library-level publish path",
sourceContract.WeKnoraVersion, sourceContract.MigrationVersion)
}
if err != nil || sourceContract.TenantID != exporter.config.TenantID ||
sourceContract.KnowledgeBaseID != item.TestKnowledgeBaseID || sourceContract.RetrieverEngine != "elasticsearch_v8" ||
sourceContract.EmbeddingModel != exporter.config.EmbeddingModel || sourceContract.EmbeddingDimension != releasebundle.EmbeddingDimensionV071 {
return nil, errors.New("source WeKnora release contract mismatch")
@@ -0,0 +1,35 @@
package v071
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
)
// IndexMappingDigest 与线上读取路径(SearchExporter.inspectMapping)用同一套规范化:
// 取 ES `/{index}/_mapping` 响应里该索引的 mappings,规范 JSON 后取 sha256。
//
// 单独暴露它是为了能用**真实抓下来的映射**证明摘要可算,并证明 0.7.1 与 0.9.0
// 的映射是两个不同契约(不能因为"都是 1024 维"就当同一份)。
func IndexMappingDigest(raw []byte) (string, error) {
var response map[string]json.RawMessage
if err := json.Unmarshal(raw, &response); err != nil || len(response) != 1 {
return "", errors.New("Elasticsearch mapping response is invalid")
}
for _, indexRaw := range response {
var index struct {
Mappings json.RawMessage `json:"mappings"`
}
if err := json.Unmarshal(indexRaw, &index); err != nil || len(index.Mappings) == 0 {
return "", errors.New("Elasticsearch mapping response is invalid")
}
canonical, err := canonicalJSON(index.Mappings)
if err != nil {
return "", errors.New("Elasticsearch mapping response is invalid")
}
digest := sha256.Sum256(canonical)
return hex.EncodeToString(digest[:]), nil
}
return "", errors.New("Elasticsearch mapping response is invalid")
}
@@ -145,13 +145,19 @@ type libraryRow struct {
}
func validateLibraryCandidate(identity PostgresIdentity, pg PostgresSnapshot, es SearchSnapshot) error {
required := map[string]bool{"resources": true}
for _, entry := range libraryContentQueries {
required[entry.name] = true
// 必需表来自 profile,不是固定清单:0.9.0 没有 wiki_log_entries 就不要求它,
// 但只要表在 profile 里是必需的,缺了就必须失败。
profile, err := pg.schemaProfile()
if err != nil {
return err
}
required := map[string]bool{}
for _, name := range profile.RequiredTables {
required[name] = true
}
tables := map[string]map[string]libraryRow{}
for _, table := range pg.Tables {
if !required[table.Name] || tables[table.Name] != nil || table.RowCount != len(table.Rows) || table.SHA256 != newTableSnapshot(table.Name, table.Rows).SHA256 {
if !profile.HasTable(table.Name) || tables[table.Name] != nil || table.RowCount != len(table.Rows) || table.SHA256 != newTableSnapshot(table.Name, table.Rows).SHA256 {
return errors.New("invalid library table inventory")
}
rows := map[string]libraryRow{}
@@ -30,6 +30,8 @@ type ProductionBackupStore struct {
maxDocuments int
maxObjectByte int64
now func() time.Time
// tables 是目标库解析出的知识域表集合(0.9.0 没有 wiki_log_entries)。
tables []string
}
type ProductionBackupConfig struct {
@@ -58,10 +60,16 @@ func NewProductionBackupStore(config ProductionBackupConfig) (*ProductionBackupS
if now == nil {
now = time.Now
}
// 预备份的表集合必须与写入/校验同源:按 0.7.1 固定清单会在 0.9.0 目标上
// 查询不存在的 wiki_log_entries。
schema, err := ResolveSchemaProfile(context.Background(), config.Pool, "prebackup target")
if err != nil {
return nil, err
}
return &ProductionBackupStore{pool: config.Pool, search: config.Search, objects: config.Objects,
archiveRoot: config.ArchiveRoot, scope: config.Scope, identity: config.Identity,
compatibility: config.Compatibility, maxDocuments: config.MaxDocuments,
maxObjectByte: config.MaxObjectByte, now: now}, nil
maxObjectByte: config.MaxObjectByte, now: now, tables: schema.Tables}, nil
}
// tableFingerprint 是一张表的"数量 + 内容摘要"指纹:同数量改写也会被识别。
@@ -75,7 +83,7 @@ func (s *ProductionBackupStore) fingerprints(ctx context.Context) (map[string]ta
result := map[string]tableFingerprint{}
exporter := PostgresExporter{db: s.pool}
var bodies, bindings []json.RawMessage
for _, table := range productionLibraryTables {
for _, table := range s.tables {
if table == "resources" {
continue
}
@@ -267,7 +275,7 @@ func (s *ProductionBackupStore) objectVersions(ctx context.Context, pg PostgresS
func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) {
counts := map[string]int64{}
for _, table := range productionLibraryTables {
for _, table := range s.tables {
var count int64
if err := s.pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`,
table, scopeWhere(table)), s.identity.TenantID, s.identity.KnowledgeBaseID).Scan(&count); err != nil {
@@ -19,19 +19,18 @@ func exportLibraryBeforeImage(ctx context.Context, db librarySnapshotDB, identit
return PostgresSnapshot{}, err
}
defer tx.Rollback(context.WithoutCancel(ctx))
var version int
var dirty bool
if err := tx.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
return PostgresSnapshot{}, errors.New("before-image schema contract mismatch")
profile, err := ResolveSchemaProfile(ctx, tx, "before-image source")
if err != nil {
return PostgresSnapshot{}, err
}
exporter := PostgresExporter{db: tx}
rowsByTable := map[string][]json.RawMessage{}
var bodies, bindings []json.RawMessage
for _, table := range productionLibraryTables {
for _, table := range profile.Tables {
if table == "resources" {
continue
}
rows, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s ORDER BY (%s)::text`, table, table, scopeWhere(table), identityExpr(table)), identity.TenantID, identity.KnowledgeBaseID)
rows, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s ORDER BY (%s)::text`, table, table, scopeWhereFor(profile.Tables, table), identityExpr(table)), identity.TenantID, identity.KnowledgeBaseID)
if err != nil {
return PostgresSnapshot{}, err
}
@@ -50,8 +49,8 @@ func exportLibraryBeforeImage(ctx context.Context, db librarySnapshotDB, identit
return PostgresSnapshot{}, err
}
rowsByTable["resources"] = resources
snapshot := PostgresSnapshot{}
for _, table := range productionLibraryTables {
snapshot := PostgresSnapshot{SchemaVersion: profile.SchemaVersion, Profile: profile.Name}
for _, table := range profile.Tables {
snapshot.Tables = append(snapshot.Tables, newTableSnapshot(table, rowsByTable[table]))
}
handles, err := libraryResourceHandles(bodies)
+11 -7
View File
@@ -29,10 +29,10 @@ func ExportLibraryWithReleases(ctx context.Context, db librarySnapshotDB, identi
return PostgresSnapshot{}, errors.New("begin library snapshot")
}
defer tx.Rollback(context.WithoutCancel(ctx))
var version int
var dirty bool
if err := tx.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
return PostgresSnapshot{}, errors.New("source library schema contract mismatch")
// 版本号只用来选 profile;放行条件是随后的表/列能力探测。
profile, err := ResolveSchemaProfile(ctx, tx, "source library")
if err != nil {
return PostgresSnapshot{}, err
}
var graph bool
if err := tx.QueryRow(ctx, `SELECT COALESCE((indexing_strategy->>'graph_enabled')::boolean,false) FROM knowledge_bases WHERE tenant_id=$1 AND id=$2 AND deleted_at IS NULL`, identity.TenantID, identity.KnowledgeBaseID).Scan(&graph); err != nil {
@@ -62,15 +62,19 @@ func ExportLibraryWithReleases(ctx context.Context, db librarySnapshotDB, identi
return PostgresSnapshot{}, err
}
exporter := PostgresExporter{db: tx}
var snapshot PostgresSnapshot
snapshot.Catalog, _, err = captureMigrationCatalog(ctx, tx)
snapshot := PostgresSnapshot{SchemaVersion: profile.SchemaVersion, Profile: profile.Name}
snapshot.Catalog, _, err = captureMigrationCatalogFor(ctx, tx, profile.CatalogTables())
if err != nil {
return PostgresSnapshot{}, err
}
if err = validateMigrationCatalog(snapshot.Catalog); err != nil {
if err = validateMigrationCatalogFor(snapshot.Catalog, profile.CatalogTables(), profile.Profile); err != nil {
return PostgresSnapshot{}, err
}
for _, entry := range libraryContentQueries {
// 0.9.0 没有 wiki_log_entries:不产生该部件,也不去查不存在的表。
if !profile.HasTable(entry.name) {
continue
}
rows, err := exporter.queryRows(ctx, entry.sql, identity.TenantID, identity.KnowledgeBaseID)
if err != nil {
return PostgresSnapshot{}, err
@@ -103,6 +103,10 @@ func (d *libraryFixtureTx) Query(_ context.Context, q string, _ ...any) (pgx.Row
if d.queryError {
return nil, errors.New("fixture failure")
}
// 能力探测走真实 SQL:假库按 0.7.1 profile 回答,避免"桩放行"。
if strings.Contains(q, "information_schema.columns") {
return legacySchemaProbeRows(), nil
}
// 死信行按真实形状返回:解除判定必须逐字节比对指纹,空结果会让"存在即阻断"失效。
if strings.Contains(q, "to_jsonb(d)") && d.failed > 0 {
return &jsonRows{values: []json.RawMessage{json.RawMessage(fixtureDeadLetter)}}, nil
@@ -111,7 +111,7 @@ func (t *ProductionLibraryTarget) preflightNaturalKeys(ctx context.Context, arch
targetFilter := predicate
args := []any{raw}
if part.Name != "resources" {
scope := strings.ReplaceAll(strings.ReplaceAll(scopeWhere(part.Name), "$2", "$3"), "$1", "$2")
scope := strings.ReplaceAll(strings.ReplaceAll(t.scopeWhere(part.Name), "$2", "$3"), "$1", "$2")
targetFilter += " AND NOT (" + scope + ")"
args = append(args, t.scope.TenantID, t.scope.KnowledgeBaseID)
}
@@ -17,6 +17,7 @@ import (
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/libraryprofile"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
"knowledge-governance/internal/objectgateway"
@@ -85,13 +86,22 @@ func (config ProductionLibraryConfig) validate(scope ProductionScope) error {
return nil
}
// 依赖顺序:插入按正序,删除按倒序。
// 真实 Schema 有外键(resource_bindings.resource_id → resources.id),
// 所以父表 resources 必须排在子表 resource_bindings 之前。
var productionLibraryTables = []string{
"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
"resources", "resource_bindings", "wiki_folders", "wiki_pages",
"wiki_page_issues", "wiki_log_entries",
// productionLibraryTables 是 0.7.1 的知识域表集合(依赖顺序:插入按正序,删除按
// 倒序;真实 Schema 有外键 resource_bindings.resource_id → resources.id,父表
// resources 必须排在子表 resource_bindings 之前)。
//
// 它**不是**本包整库读写的表清单来源:真实路径一律用 ResolveSchemaProfile 解析出的
// Tables——0.9.0 没有 wiki_log_entries,任何一条固定清单的查询都会在真库上失败,而
// "表不存在就跳过"又会让 0.7.1 少一张必需表也悄悄通过。这里只留给明确 74-only 的
// 入口(编目、残留复核、0.7.1 夹具),定义直接取自 profile 声明,避免两份清单走偏。
var productionLibraryTables = legacyKnowledgeTables()
func legacyKnowledgeTables() []string {
profile, err := libraryprofile.For(libraryprofile.LegacyVersion)
if err != nil {
panic(err)
}
return append([]string(nil), profile.RequiredTables...)
}
type ProductionLibraryTarget struct {
@@ -100,6 +110,11 @@ type ProductionLibraryTarget struct {
scope ProductionScope
search *SearchExporter
now func() time.Time
// schema / tables 是目标库解析出的版本声明与"实际存在的知识域表"。写入、校验、
// 清理、编目全部按它取数:0.9.0 缺 wiki_log_entries 时不得查询/删除该表,
// 而 0.7.1 少一张必需表必须失败,不能靠"表不存在就跳过"。
schema SchemaProfile
tables []string
// storageBackend / storageProvider 是本次写入使用的目标存储后端与提供方
// (写入与校验共用,缺一不可:只换桶不换提供方,目标取文件仍会走来源实例的路)。
storageBackend string
@@ -215,7 +230,15 @@ func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryCon
if now == nil {
now = time.Now
}
return &ProductionLibraryTarget{pool: pool, config: config, scope: scope, search: search, now: now}, nil
// 目标版本在装配时定型:未知版本/dirty/缺表在这里就失败,不能等到第一条写入
// 才发现"目标其实没有这张表"。
schema, err := ResolveSchemaProfile(ctx, pool, "publish target")
if err != nil {
pool.Close()
return nil, err
}
return &ProductionLibraryTarget{pool: pool, config: config, scope: scope, search: search, now: now,
schema: schema, tables: schema.Tables}, nil
}
func (t *ProductionLibraryTarget) Close() { t.pool.Close() }
@@ -235,7 +258,7 @@ func (t *ProductionLibraryTarget) Identity() PostgresIdentity {
func (t *ProductionLibraryTarget) Current(ctx context.Context) (libraryrelease.CurrentState, error) {
state := libraryrelease.CurrentState{Counts: map[string]int64{}, Digests: map[string]string{},
CapturedAt: t.now().UTC()}
for _, table := range productionLibraryTables {
for _, table := range t.tables {
count, digest, err := t.tableState(ctx, table)
if err != nil {
return libraryrelease.CurrentState{}, err
@@ -255,10 +278,19 @@ func (t *ProductionLibraryTarget) Current(ctx context.Context) (libraryrelease.C
return state, nil
}
// scopeWhere 返回"这张表里属于本知识库的行"的过滤条件($1=tenant,$2=kb)。
// scopeWhere 是 74-only 入口(残留复核、0.7.1 夹具)使用的固定表集合版本。
func scopeWhere(table string) string { return scopeWhereFor(productionLibraryTables, table) }
// scopeWhere 用本次目标解析出的表集合。
func (t *ProductionLibraryTarget) scopeWhere(table string) string {
return scopeWhereFor(t.tables, table)
}
// scopeWhereFor 返回"这张表里属于本知识库的行"的过滤条件($1=tenant,$2=kb)。
// 不同表的归属列不同:关系表与绑定表没有 knowledge_base_id,必须走父表判定,
// 否则要么删错、要么把别的知识库数据算进来。
func scopeWhere(table string) string {
// 否则要么删错、要么把别的知识库数据算进来。表集合来自解析出的 profile:
// 0.9.0 的资源正文引用子查询不能提到不存在的 wiki_log_entries。
func scopeWhereFor(tables []string, table string) string {
switch table {
case "knowledge_tag_relations":
// 真实 WeKnora 的这张表只有 knowledge_id/tag_id/created_at,
@@ -283,7 +315,7 @@ func scopeWhere(table string) string {
SELECT 1 FROM resource_bindings b
JOIN chunks c ON c.id=b.owner_id AND c.tenant_id=b.tenant_id
WHERE b.resource_id=resources.id AND b.tenant_id=$1 AND b.owner_type='chunk' AND c.knowledge_base_id=$2)
OR EXISTS (` + libraryResourceBodyReferenceSQL("c.tenant_id=$1 AND c.knowledge_base_id=$2") + `))`
OR EXISTS (` + libraryResourceBodyReferenceSQL(tables, "c.tenant_id=$1 AND c.knowledge_base_id=$2") + `))`
default:
return "tenant_id=$1 AND knowledge_base_id=$2"
}
@@ -308,12 +340,12 @@ func (t *ProductionLibraryTarget) tableState(ctx context.Context, table string)
// scan used by the legacy scope predicate.
exporter := PostgresExporter{db: t.pool}
var bodies, bindings []json.RawMessage
for _, name := range productionLibraryTables {
for _, name := range t.tables {
if name == "resources" {
continue
}
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`,
name, name, scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
name, name, t.scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
if err != nil {
return 0, "", fmt.Errorf("读取生产 %s 失败", table)
}
@@ -335,7 +367,7 @@ func (t *ProductionLibraryTarget) tableState(ctx context.Context, table string)
// canonicalJSON on text and fail on every non-empty table.
if table != "resources" {
rows, err := t.pool.Query(ctx, fmt.Sprintf(`SELECT %s FROM %s WHERE %s ORDER BY 1`,
"("+identityExpr(table)+")::text", table, scopeWhere(table)), identity.TenantID, identity.KnowledgeBaseID)
"("+identityExpr(table)+")::text", table, t.scopeWhere(table)), identity.TenantID, identity.KnowledgeBaseID)
if err != nil {
return 0, "", fmt.Errorf("读取生产 %s 失败", table)
}
@@ -404,6 +436,9 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
if err != nil {
return errors.New("预发布快照不可读")
}
if err := requireSameProfile(manifest.Compatibility.SchemaVersion, t.schema, "publish"); err != nil {
return err
}
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
return err
}
@@ -444,8 +479,8 @@ func (t *ProductionLibraryTarget) importRows(ctx context.Context, archive *libra
return errors.New("开启生产事务失败")
}
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
table := productionLibraryTables[index]
for index := len(t.tables) - 1; index >= 0; index-- {
table := t.tables[index]
if _, ok := parts[table]; !ok {
continue
}
@@ -453,7 +488,7 @@ func (t *ProductionLibraryTarget) importRows(ctx context.Context, archive *libra
return err
}
}
for _, table := range productionLibraryTables {
for _, table := range t.tables {
part, ok := parts[table]
if !ok {
continue
@@ -473,7 +508,7 @@ func (t *ProductionLibraryTarget) deleteScope(ctx context.Context, tx pgx.Tx, ta
// 资源可能被多个知识库共享:这里不删,只做插入/更新。
return nil
}
statement := fmt.Sprintf(`DELETE FROM %s WHERE %s`, table, scopeWhere(table))
statement := fmt.Sprintf(`DELETE FROM %s WHERE %s`, table, t.scopeWhere(table))
if _, err := tx.Exec(ctx, statement, t.scope.TenantID, t.scope.KnowledgeBaseID); err != nil {
return fmt.Errorf("清理生产 %s 失败", table)
}
@@ -810,6 +845,9 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err != nil {
return errors.New("预发布快照不可读")
}
if err := requireSameProfile(manifest.Compatibility.SchemaVersion, t.schema, "publish verify"); err != nil {
return err
}
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
return err
}
@@ -1292,7 +1330,7 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
if err := t.AdoptWriteSet(*backup.WriteSet); err != nil {
return err
}
if err := ensureRestorableResourceReferences(ctx, t.pool, *backup.WriteSet); err != nil {
if err := ensureRestorableResourceReferences(ctx, t.pool, t.tables, *backup.WriteSet); err != nil {
return err
}
if err := ensureNoResourceGrants(ctx, t.pool, backup.WriteSet.CreatedIDs("resources")); err != nil {
@@ -0,0 +1,411 @@
package v071
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"io"
"net/http"
"os"
"strconv"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
)
// 0.9.0 整库发布目标的隔离夹具身份。来源=0.9.0 测试实例,目标=另一租户/知识库,
// 这样"资源定位符换桶/后端/租户段"与"行改写目标范围"都被真实覆盖。
const (
v090TargetTenant = 10001
v090TargetKB = "kb-v090-prod"
v090TargetBackend = "backend-v090-target"
v090TargetBucket = "kg-v090-target-bucket"
v090SourceBackend = "backend-v090-source"
// 来源定位符是目标应用真实使用的形状:storage://<backend>/<scheme>://<bucket>/<key>。
v090SourceLocator = "storage://backend-v090-source/oss://kg-v090-source-bucket/weknora/10000/rs-v090-1/kn.bin"
)
// v090ObjectBytes 按"提供方|路径"供字节,既能给封存读来源对象,也能给目标写回。
type v090ObjectBytes map[string][]byte
func (o v090ObjectBytes) Open(_ context.Context, provider, physicalPath string) (io.ReadCloser, error) {
content, ok := o[provider+"|"+physicalPath]
if !ok {
return nil, errors.New("fixture object missing: " + provider + "|" + physicalPath)
}
return io.NopCloser(bytes.NewReader(content)), nil
}
// TestV090ProductionTargetPublishIsolated 是 0.9.0(schema 134)**发布写入路径**的
// 端到端证据:真实导出 → 真实封存 → 真实 ProductionLibraryTarget 写进干净 134 目标
// → 逐表核对 → 用写前镜像恢复回滚 → 再核对回到写前状态。
//
// 这里不允许打桩 PG/索引/对象:134 与 74 的差别(没有 wiki_log_entries、chunks 多
// NOT NULL 列、seq_id 由目标序列分配)只有在真 schema 上才会暴露。因此默认 CI 跳过,
// 在 scripts/test-scope.json 里声明为可选未运行,并写明理由。
func TestV090ProductionTargetPublishIsolated(t *testing.T) {
base := os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" || base == "" {
t.Skip("requires isolated PG plus ES opt-in (KG_LIBRARY_TEST_ISOLATED=1, KG_LIBRARY_TEST_ELASTIC_URL)")
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
src := v090Pool(t, "KG_V090_SOURCE_DSN")
dst := v090Pool(t, "KG_V090_TARGET_DSN")
identity := seedV090(t, ctx, src)
// 资源行必须带目标实例真实可解析的定位符,否则"换桶/换后端"这条改写没有被覆盖。
if _, err := src.Exec(ctx, `UPDATE resources SET provider='oss', storage_backend_id=$2,
physical_path=$3, location_hash=$4 WHERE id=$1`,
v090Resource, v090SourceBackend, v090SourceLocator, resourceLocationHash(v090SourceLocator)); err != nil {
t.Fatalf("retarget source resource locator: %v", err)
}
source, err := ExportLibrary(ctx, src, identity)
if err != nil {
t.Fatalf("export 0.9.0 library: %v", err)
}
if source.SchemaVersion != 134 || source.Profile != "v090" {
t.Fatalf("snapshot must record the 0.9.0 profile, got %d/%q", source.SchemaVersion, source.Profile)
}
if tablePresent(source, "wiki_log_entries") {
t.Fatal("0.9.0 export must not produce a wiki_log_entries table")
}
token := make([]byte, 8)
if _, err := rand.Read(token); err != nil {
t.Fatal(err)
}
index := "kg_v090_publish_" + hex.EncodeToString(token)
client := &http.Client{Timeout: 30 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
call := func(method, path, body string, want int) {
t.Helper()
request, err := http.NewRequestWithContext(ctx, method, base+path, strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
request.Header.Set("Content-Type", "application/json")
response, err := client.Do(request)
if err != nil {
t.Fatalf("fixture HTTP: %v", err)
}
defer response.Body.Close()
if response.StatusCode != want {
t.Fatalf("fixture %s %s => %d, want %d", method, path, response.StatusCode, want)
}
}
call("PUT", "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"chunk_id":{"type":"keyword"},"content":{"type":"text"},"embedding":{"type":"dense_vector","dims":2}}}}`, 200)
defer func() {
call("DELETE", "/"+index, "", 200)
call("HEAD", "/"+index, "", 404)
}()
exporter, err := NewSearchExporter(SearchExporterConfig{BaseURL: base, Index: index, PageSize: 100, HTTPClient: client})
if err != nil {
t.Fatalf("search exporter: %v", err)
}
mappingSHA, err := exporter.MappingSHA256(ctx)
if err != nil {
t.Fatalf("index mapping digest: %v", err)
}
root := t.TempDir()
if err := os.Chmod(root, 0700); err != nil {
t.Fatal(err)
}
archive, err := librarysnapshot.Open(root, 64<<20)
if err != nil {
t.Fatal(err)
}
defer archive.Close()
objectBytes := []byte("fixture-object-bytes")
objects := v090ObjectBytes{"oss|" + v090SourceLocator: objectBytes}
contract := librarysnapshot.Compatibility{
WeKnoraVersion: "v0.9.0", SchemaVersion: 134,
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "bge-m3",
EmbeddingDimension: 1024, IndexMappingSHA256: mappingSHA,
}
sealed, err := ArchiveLibraryCandidate(ctx, archive, "internal", identity, contract, source,
v090PublishIndexSnapshot(t, source, mappingSHA), objects, time.Now().UTC())
if err != nil {
t.Fatalf("seal 0.9.0 archive: %v", err)
}
manifest, err := archive.Load(sealed)
if err != nil {
t.Fatalf("load sealed manifest: %v", err)
}
for _, part := range manifest.Parts {
if part.Name == "wiki_log_entries" {
t.Fatal("0.9.0 archive must not contain a wiki_log_entries part")
}
}
// 目标库:干净 134 schema + KB/模型绑定 + 目标存储后端(桶名用于解析定位符)。
if err := seedV090Target(t, ctx, dst); err != nil {
t.Fatalf("seed target binding: %v", err)
}
store := newMemoryObjectStore(nil)
target, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
Postgres: os.Getenv("KG_V090_TARGET_DSN"), SearchURL: base, SearchIndex: index,
Objects: store, ObjectsReader: store, ArchiveRoot: root,
ObjectsBucket: v090TargetBucket, MaxDocuments: 100, HTTPClient: client,
}, ProductionScope{TenantID: v090TargetTenant, KnowledgeBaseID: v090TargetKB})
if err != nil {
t.Fatalf("target: %v", err)
}
defer target.Close()
backups, err := NewProductionBackupStore(ProductionBackupConfig{
Pool: target.Pool(), Search: target.Search(), Objects: store, ArchiveRoot: root,
Scope: "internal", Identity: target.Identity(), MaxDocuments: 100, Compatibility: contract,
})
if err != nil {
t.Fatalf("backup store: %v", err)
}
// 现状:真实 Current() 必须在 134 上可用,且不得去查 0.9.0 里不存在的 wiki_log_entries。
before, err := target.Current(ctx)
if err != nil {
t.Fatalf("current target state on 0.9.0: %v", err)
}
if _, present := before.Counts["wiki_log_entries"]; present {
t.Fatalf("0.9.0 target state must not include wiki_log_entries: %+v", before.Counts)
}
backup, err := backups.Prepare(ctx, sealed, before)
if err != nil {
t.Fatalf("prepare before-image: %v", err)
}
backup.WriteSet = planLedgerWithUniverse(t, target, sealed, backup.SnapshotID, backup.Universe)
for _, stage := range libraryrelease.Stages() {
if err := target.Apply(ctx, stage, sealed); err != nil {
t.Fatalf("apply %s on 0.9.0 target: %v", stage, err)
}
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: sealed}); err != nil {
t.Fatalf("verify after 0.9.0 publish: %v", err)
}
assertV090Published(t, ctx, dst, target, source)
// 跨版本必须拒绝,并且错误同时点名两个版本号。用另一个没有冻结计划的目标实例:
// 已接管写集合的执行目标会先被"快照与计划不符"挡住,那不是版本闸门。
legacy := sealTargetFixture(t, archive, "internal", []string{"doc-safe"}, []string{"chunk-safe"}, "wiki-safe")
legacyTarget, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
Postgres: os.Getenv("KG_V090_TARGET_DSN"), SearchURL: base, SearchIndex: index,
Objects: store, ObjectsReader: store, ArchiveRoot: root,
ObjectsBucket: v090TargetBucket, MaxDocuments: 100, HTTPClient: client,
}, ProductionScope{TenantID: v090TargetTenant, KnowledgeBaseID: v090TargetKB})
if err != nil {
t.Fatalf("legacy target: %v", err)
}
defer legacyTarget.Close()
err = legacyTarget.Apply(ctx, libraryrelease.StageDatabase, legacy)
if err == nil || !strings.Contains(err.Error(), "74") || !strings.Contains(err.Error(), "134") {
t.Fatalf("cross-version publish must be refused naming both versions, got %v", err)
}
if err := legacyTarget.Verify(ctx, libraryrelease.Expectation{SnapshotID: legacy}); err == nil ||
!strings.Contains(err.Error(), "74") || !strings.Contains(err.Error(), "134") {
t.Fatalf("cross-version verify must be refused naming both versions, got %v", err)
}
// 写前镜像恢复:目标必须回到写之前的样子(0.9.0 表集合)。
if err := target.Restore(ctx, backup); err != nil {
t.Fatalf("restore from before-image: %v", err)
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: backup.SnapshotID}); err != nil {
t.Fatalf("verify after restore: %v", err)
}
for _, table := range []string{"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
"wiki_pages", "wiki_folders", "wiki_page_issues"} {
statement := `SELECT count(*) FROM ` + table + ` WHERE tenant_id=$1 AND knowledge_base_id=$2`
args := []any{v090TargetTenant, v090TargetKB}
if table == "knowledge_tag_relations" {
// 关系表没有租户/知识库列:归属只能走父表。
statement = `SELECT count(*) FROM knowledge_tag_relations r WHERE r.knowledge_id IN
(SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2)`
}
var count int64
if err := dst.QueryRow(ctx, statement, args...).Scan(&count); err != nil {
t.Fatal(err)
}
if count != 0 {
t.Fatalf("rollback left %d rows in %s", count, table)
}
}
var restoredResource int64
if err := dst.QueryRow(ctx, `SELECT count(*) FROM resources WHERE tenant_id=$1`, v090TargetTenant).Scan(&restoredResource); err != nil {
t.Fatal(err)
}
if restoredResource != 0 {
t.Fatalf("rollback left %d published resources in the target tenant", restoredResource)
}
t.Logf("v0.9.0 publish verified: snapshot=%s backup=%s bytes=%d", sealed, backup.SnapshotID, backup.Bytes)
}
// assertV090Published 逐表核对目标与来源内容一致(身份列走目标隔离映射,不作为内容比对)。
func assertV090Published(t *testing.T, ctx context.Context, dst *pgxpool.Pool, target *ProductionLibraryTarget, source PostgresSnapshot) {
t.Helper()
wantChunks := int64(source.Table("chunks").RowCount)
var chunks int64
if err := dst.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=$1 AND knowledge_base_id=$2`,
v090TargetTenant, v090TargetKB).Scan(&chunks); err != nil {
t.Fatal(err)
}
if chunks != wantChunks || chunks < 2 {
t.Fatalf("chunks in target = %d, want %d", chunks, wantChunks)
}
// 目标隔离映射:写入的是目标身份,不是来源身份。
for _, chunkID := range []string{v090ChunkOne, v090ChunkTwo} {
var content string
if err := dst.QueryRow(ctx, `SELECT content FROM chunks WHERE id=$1`,
expectedTargetID(v090TargetTenant, v090TargetKB, "chunks", chunkID)).Scan(&content); err != nil {
t.Fatalf("mapped chunk %s missing: %v", chunkID, err)
}
if content == "" {
t.Fatalf("mapped chunk %s has empty content", chunkID)
}
}
// 0.9.0 新增 NOT NULL 列走目标默认值;seq_id 由目标序列分配(来源值不得搬过来)。
var defaults, assigned int64
if err := dst.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=$1 AND knowledge_base_id=$2
AND source_content IS NOT NULL AND content_revision IS NOT NULL AND index_status IS NOT NULL
AND context_header IS NOT NULL AND last_editor_id IS NOT NULL AND seq_id IS NOT NULL`,
v090TargetTenant, v090TargetKB).Scan(&defaults); err != nil {
t.Fatal(err)
}
if defaults != wantChunks {
t.Fatalf("0.9.0 NOT NULL columns unsatisfied on %d/%d chunks", defaults, wantChunks)
}
if err := dst.QueryRow(ctx, `SELECT count(DISTINCT seq_id) FROM chunks WHERE tenant_id=$1 AND knowledge_base_id=$2`,
v090TargetTenant, v090TargetKB).Scan(&assigned); err != nil {
t.Fatal(err)
}
if assigned != wantChunks {
t.Fatalf("target seq_id must be assigned per chunk, got %d distinct for %d chunks", assigned, wantChunks)
}
// Wiki 必须非空:空表通过等于没验证 wiki 部件。
var pages int64
var content, slug string
if err := dst.QueryRow(ctx, `SELECT count(*) FROM wiki_pages WHERE tenant_id=$1 AND knowledge_base_id=$2`,
v090TargetTenant, v090TargetKB).Scan(&pages); err != nil {
t.Fatal(err)
}
if pages != 1 {
t.Fatalf("wiki_pages in target = %d, want 1", pages)
}
if err := dst.QueryRow(ctx, `SELECT content, slug FROM wiki_pages WHERE id=$1`,
expectedTargetID(v090TargetTenant, v090TargetKB, "wiki_pages", v090Page)).Scan(&content, &slug); err != nil {
t.Fatalf("published wiki page missing: %v", err)
}
if content != "wiki 正文" || slug != "fixture-page" {
t.Fatalf("wiki page content changed: content=%q slug=%q", content, slug)
}
var tags int64
if err := dst.QueryRow(ctx, `SELECT count(*) FROM knowledge_tags WHERE tenant_id=$1 AND knowledge_base_id=$2 AND name='夹具标签'`,
v090TargetTenant, v090TargetKB).Scan(&tags); err != nil {
t.Fatal(err)
}
if tags != 1 {
t.Fatalf("knowledge_tags in target = %d, want 1", tags)
}
var title, model string
if err := dst.QueryRow(ctx, `SELECT title, embedding_model_id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2`,
v090TargetTenant, v090TargetKB).Scan(&title, &model); err != nil {
t.Fatalf("published knowledge missing: %v", err)
}
if title != "0.9.0 夹具" || model != "bge-m3-v090" {
t.Fatalf("knowledge content changed: title=%q model=%q", title, model)
}
// 资源必须指向目标桶/后端,且定位符哈希按最终路径重算。
wantLocator := "storage://" + v090TargetBackend + "/minio://" + v090TargetBucket + "/" +
strconv.FormatUint(v090TargetTenant, 10) + "/" + v090Resource + "/kn.bin"
var provider, backend, locator, locationHash string
var size int64
if err := dst.QueryRow(ctx, `SELECT provider, coalesce(storage_backend_id,''), physical_path, location_hash, size
FROM resources WHERE tenant_id=$1`, v090TargetTenant).
Scan(&provider, &backend, &locator, &locationHash, &size); err != nil {
t.Fatalf("published resource missing: %v", err)
}
if provider != "minio" || backend != v090TargetBackend {
t.Fatalf("resource must point at the target backend: provider=%q backend=%q", provider, backend)
}
if locator != wantLocator {
t.Fatalf("resource locator not retargeted:\n got %s\nwant %s", locator, wantLocator)
}
if locationHash != resourceLocationHash(wantLocator) {
t.Fatalf("resource location_hash must hash the target locator")
}
if size != int64(len("fixture-object-bytes")) {
t.Fatalf("resource size changed: %d", size)
}
// 目标对象存储里必须真的有这份字节(写入阶段按目标定位符上传)。
if _, err := target.Objects().Open(ctx, provider, locator); err != nil {
t.Fatalf("published object unreadable in target store: %v", err)
}
}
// v090PublishIndexSnapshot 按导出分块合成索引快照:这批文档走的是与线上同一套
// 映射/摘要校验,不能凭空绕过。
func v090PublishIndexSnapshot(t *testing.T, pg PostgresSnapshot, mappingSHA string) SearchSnapshot {
t.Helper()
var documents []SearchDocument
for _, raw := range pg.Table("chunks").Rows {
var chunk struct {
ID string `json:"id"`
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
KnowledgeID string `json:"knowledge_id"`
Content string `json:"content"`
}
if err := json.Unmarshal(raw, &chunk); err != nil {
t.Fatalf("decode chunk: %v", err)
}
source, err := json.Marshal(map[string]any{
"chunk_id": chunk.ID, "tenant_id": chunk.TenantID,
"knowledge_base_id": chunk.KnowledgeBaseID, "knowledge_id": chunk.KnowledgeID,
"content": chunk.Content, "source_id": chunk.ID, "source_type": 0,
"embedding": []float64{0.1, 0.2},
})
if err != nil {
t.Fatalf("encode index document: %v", err)
}
documents = append(documents, SearchDocument{ID: "doc-" + chunk.ID, Source: source})
}
digest, err := searchDocumentsDigest(documents)
if err != nil {
t.Fatalf("index digest: %v", err)
}
return SearchSnapshot{Documents: documents, MappingSHA256: mappingSHA, DocumentsSHA256: digest}
}
// seedV090Target 在目标库造 KB/模型绑定与目标存储后端:发布写的是内容行,不是 KB 定义。
func seedV090Target(t *testing.T, ctx context.Context, db *pgxpool.Pool) error {
t.Helper()
for _, statement := range []string{
`INSERT INTO models(id,tenant_id,name,type,source,parameters)
VALUES ('bge-m3-v090',$1,'bge-m3','embedding','local','{"embedding_parameters":{"dimension":1024}}'::jsonb)`,
`INSERT INTO knowledge_bases(id,name,tenant_id,embedding_model_id,summary_model_id,indexing_strategy)
VALUES ('` + v090TargetKB + `','v090 publish target',$1,'bge-m3-v090','bge-m3-v090','{"graph_enabled":false}'::jsonb)`,
`INSERT INTO storage_backends(id,tenant_id,name,provider,config)
VALUES ('` + v090TargetBackend + `',$1,'v090 target','minio','{"bucket_name":"` + v090TargetBucket + `"}'::jsonb)`,
} {
if _, err := db.Exec(ctx, strings.ReplaceAll(statement, "$1", strconv.Itoa(v090TargetTenant))); err != nil {
return err
}
}
return nil
}
@@ -0,0 +1,182 @@
package v071
import (
"context"
"encoding/json"
"os"
"sort"
"strings"
"testing"
"time"
"knowledge-governance/internal/librarysnapshot"
)
// v090MappingDigest 读真实抓下来的 0.9.0 映射(KG_V090_MAPPING_JSON)。
func v090MappingDigest(t *testing.T) string {
t.Helper()
path := os.Getenv("KG_V090_MAPPING_JSON")
if path == "" {
t.Skip("requires KG_V090_MAPPING_JSON pointing at the captured 0.9.0 index mapping")
}
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read KG_V090_MAPPING_JSON: %v", err)
}
digest, err := IndexMappingDigest(raw)
if err != nil {
t.Fatalf("mapping digest: %v", err)
}
return digest
}
// TestV090LibraryRoundTripIsolated 走完整 0.9.0 往返:
// 导出 → 封存(真实 manifest 校验)→ 导入干净 134 schema → 再导出逐行比对。
func TestV090LibraryRoundTripIsolated(t *testing.T) {
ctx := context.Background()
src := v090Pool(t, "KG_V090_SOURCE_DSN")
identity := seedV090(t, ctx, src)
mappingSHA := v090MappingDigest(t)
source, err := ExportLibrary(ctx, src, identity)
if err != nil {
t.Fatalf("export 0.9.0 library: %v", err)
}
if source.SchemaVersion != 134 || source.Profile != "v090" {
t.Fatalf("snapshot must record the 0.9.0 profile, got %d/%q", source.SchemaVersion, source.Profile)
}
// 0.9.0 没有 wiki_log_entries:不允许出现该表,也不允许空 Wiki 冒充通过。
if tablePresent(source, "wiki_log_entries") {
t.Fatal("0.9.0 export must not produce a wiki_log_entries table")
}
if source.Table("wiki_pages").RowCount < 1 || source.Table("wiki_folders").RowCount < 1 {
t.Fatalf("wiki parts must be non-empty: pages=%d folders=%d",
source.Table("wiki_pages").RowCount, source.Table("wiki_folders").RowCount)
}
if source.Table("chunks").RowCount < 2 || source.Table("knowledges").RowCount != 1 {
t.Fatalf("unexpected fixture shape: chunks=%d knowledges=%d",
source.Table("chunks").RowCount, source.Table("knowledges").RowCount)
}
es := v090IndexSnapshot(t, source, mappingSHA)
root := t.TempDir()
if err := os.Chmod(root, 0700); err != nil {
t.Fatal(err)
}
archive, err := librarysnapshot.Open(root, 64<<20)
if err != nil {
t.Fatal(err)
}
defer archive.Close()
contract := librarysnapshot.Compatibility{
WeKnoraVersion: "v0.9.0", SchemaVersion: 134,
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "bge-m3",
EmbeddingDimension: 1024, IndexMappingSHA256: mappingSHA,
}
sealed, err := ArchiveLibraryCandidate(ctx, archive, "internal", identity, contract, source, es,
v090Objects{"fixture/kn-v090.bin": []byte("fixture-object-bytes")}, time.Now().UTC())
if err != nil {
t.Fatalf("seal 0.9.0 archive: %v", err)
}
manifest, err := archive.Load(sealed)
if err != nil {
t.Fatalf("load sealed manifest: %v", err)
}
if manifest.Compatibility.SchemaVersion != 134 || manifest.Compatibility.WeKnoraVersion != "v0.9.0" {
t.Fatalf("manifest must record the 0.9.0 profile, got %+v", manifest.Compatibility)
}
for _, part := range manifest.Parts {
if part.Kind == "wiki" && part.Name == "wiki_log_entries" {
t.Fatal("0.9.0 archive must not contain a wiki_log_entries part")
}
}
dst := v090Pool(t, "KG_V090_TARGET_DSN")
if err := seedV090Binding(t, ctx, dst); err != nil {
t.Fatalf("seed target binding: %v", err)
}
importer, err := NewPostgresImporter(dst)
if err != nil {
t.Fatal(err)
}
document := PostgresIdentity{TenantID: identity.TenantID, KnowledgeBaseID: identity.KnowledgeBaseID, KnowledgeID: v090Knowledge}
if err := importer.Replace(ctx, document, source); err != nil {
t.Fatalf("import 0.9.0 snapshot: %v", err)
}
reloaded, err := ExportLibrary(ctx, dst, identity)
if err != nil {
t.Fatalf("re-export imported library: %v", err)
}
compareSnapshotContent(t, source, reloaded)
// 0.9.0 新增的 NOT NULL 列在导入后必须仍是合法值,且 seq_id 由目标序列分配。
var defaults int
if err := dst.QueryRow(ctx, `SELECT count(*) FROM chunks
WHERE source_content IS NOT NULL AND content_revision IS NOT NULL
AND index_status IS NOT NULL AND context_header IS NOT NULL
AND last_editor_id IS NOT NULL AND seq_id IS NOT NULL`).Scan(&defaults); err != nil {
t.Fatal(err)
}
if defaults != source.Table("chunks").RowCount {
t.Fatalf("0.9.0 NOT NULL columns must be satisfied on every imported chunk: got %d want %d",
defaults, source.Table("chunks").RowCount)
}
var distinctSeq int
if err := dst.QueryRow(ctx, `SELECT count(DISTINCT seq_id) FROM chunks`).Scan(&distinctSeq); err != nil {
t.Fatal(err)
}
if distinctSeq != source.Table("chunks").RowCount {
t.Fatalf("target chunk seq_id must be assigned, got %d distinct", distinctSeq)
}
// 跨版本必须拒绝:0.7.1 归档不能落到 0.9.0 目标。
crossVersion := source
crossVersion.SchemaVersion = 74
crossVersion.Profile = "v071"
err = importer.Replace(ctx, document, crossVersion)
if err == nil || !strings.Contains(err.Error(), "74") || !strings.Contains(err.Error(), "134") {
t.Fatalf("cross-version import must be refused with both versions named, got %v", err)
}
}
// compareSnapshotContent 逐表逐行比对"共有列"的规范化内容。
// chunks.seq_id 是库级序列值、不是文档内容,导入时由目标分配,故排除比对。
func compareSnapshotContent(t *testing.T, source, target PostgresSnapshot) {
t.Helper()
if len(source.Tables) != len(target.Tables) {
t.Fatalf("table set changed across round trip: %d -> %d", len(source.Tables), len(target.Tables))
}
for _, table := range source.Tables {
want := tableDigests(t, table.Name, table.Rows)
got := tableDigests(t, table.Name, target.Table(table.Name).Rows)
if table.RowCount != target.Table(table.Name).RowCount || len(want) != len(got) {
t.Fatalf("%s row count changed: %d -> %d", table.Name, len(want), len(got))
}
for index := range want {
if want[index] != got[index] {
t.Fatalf("%s row %d content changed across round trip", table.Name, index)
}
}
}
}
func tableDigests(t *testing.T, name string, rows []json.RawMessage) []string {
t.Helper()
digests := make([]string, 0, len(rows))
for _, raw := range rows {
var row map[string]json.RawMessage
if err := json.Unmarshal(raw, &row); err != nil {
t.Fatalf("decode %s row: %v", name, err)
}
if name == "chunks" {
delete(row, "seq_id")
}
digest, err := canonicalRowDigest(row)
if err != nil {
t.Fatalf("digest %s row: %v", name, err)
}
digests = append(digests, digest)
}
sort.Strings(digests)
return digests
}
@@ -10,6 +10,7 @@ import (
"strings"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/libraryprofile"
"knowledge-governance/internal/librarysnapshot"
)
@@ -118,7 +119,12 @@ func libraryResourceHandles(rows []json.RawMessage) ([]string, error) {
// Revalidate frozen source content at the actual planning boundary. A sealed
// manifest proves bytes, not that every row has a parent or a supported owner.
func validateArchivedLibraryClosure(archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error {
var pg PostgresSnapshot
// 快照的 schema 契约来自 manifest:不带上版本号,重建出来的快照就是"版本未知",
// 而未知版本按 0.7.1 校验——0.9.0 的归档会被要求提供它根本没有的 wiki_log_entries。
pg := PostgresSnapshot{SchemaVersion: manifest.Compatibility.SchemaVersion}
if profile, err := libraryprofile.For(pg.SchemaVersion); err == nil {
pg.Profile = profile.Name
}
es := SearchSnapshot{MappingSHA256: manifest.Compatibility.IndexMappingSHA256}
for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" && part.Kind != "index" {
@@ -152,8 +152,8 @@ func (t *ProductionLibraryTarget) ClearForeignScope(ctx context.Context, source
return report, nil
}
// 删除顺序:子表先、父表后(resource_bindings → resources,chunks → knowledges)。
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
table := productionLibraryTables[index]
for index := len(t.tables) - 1; index >= 0; index-- {
table := t.tables[index]
ids, ok := identities[table]
if !ok || len(ids) == 0 {
continue
@@ -216,7 +216,7 @@ func (t *ProductionLibraryTarget) legacyResourceRow(row map[string]json.RawMessa
// VerifyResidueCleared 复核清理结果:来源身份不再存在、其它租户的解释得清、
// 且(可选)目标内容与预备份完全一致。
func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, source ProductionScope) error {
for _, table := range productionLibraryTables {
for _, table := range t.tables {
if !tableHasTenant(table) {
continue
}
@@ -240,7 +240,7 @@ func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, sour
// ForeignScopes 列出目标里所有"不是目标租户"的租户,用于人工确认没有别的泄漏。
func (t *ProductionLibraryTarget) ForeignScopes(ctx context.Context) (map[string]string, error) {
foreign := map[string]string{}
for _, table := range productionLibraryTables {
for _, table := range t.tables {
if !tableHasTenant(table) {
continue
}
@@ -272,7 +272,7 @@ func (t *ProductionLibraryTarget) foreignScopeRows(ctx context.Context, table st
if !tableHasTenant(table) {
return nil, nil
}
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table))
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, t.scopeWhere(table))
arguments := []any{source.TenantID, source.KnowledgeBaseID}
if table == "resources" || table == "resource_bindings" {
statement = fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE tenant_id=$1`, table, table)
@@ -10,10 +10,11 @@ import (
)
// Keep capture/verification and deletion protection on the same set of bodies.
// predicate is an internal SQL fragment, never caller-supplied content.
func libraryResourceBodyReferenceSQL(predicate string) string {
// predicate is an internal SQL fragment, never caller-supplied content; tables comes
// from the resolved schema profile (0.9.0 has no wiki_log_entries to scan).
func libraryResourceBodyReferenceSQL(tables []string, predicate string) string {
queries := []string{}
for _, table := range productionLibraryTables {
for _, table := range tables {
if table == "resources" || table == "resource_bindings" || table == "knowledge_tag_relations" {
continue
}
@@ -24,7 +25,7 @@ func libraryResourceBodyReferenceSQL(predicate string) string {
// This must precede *all* restore mutations, including ES and object deletes.
// A created row is not permission to invalidate a later out-of-plan reader.
func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureReader, set libraryrelease.WriteSet) error {
func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureReader, tables []string, set libraryrelease.WriteSet) error {
ids := set.CreatedIDs("resources")
if len(ids) == 0 {
return nil
@@ -49,7 +50,7 @@ func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureRea
if referenced {
return errors.New("created resource has out-of-plan bindings; restore held before writes")
}
bodySQL := libraryResourceBodyReferenceSQL("c.tenant_id IS DISTINCT FROM $2 OR c.knowledge_base_id IS DISTINCT FROM $3")
bodySQL := libraryResourceBodyReferenceSQL(tables, "c.tenant_id IS DISTINCT FROM $2 OR c.knowledge_base_id IS DISTINCT FROM $3")
err = q.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM resources WHERE id=ANY($1::text[]) AND EXISTS (`+bodySQL+`))`, ids, set.TenantID, set.KnowledgeBaseID).Scan(&referenced)
if err != nil {
return err
@@ -62,10 +63,10 @@ func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureRea
// Recheck under the already-held resource parent locks. This does not claim
// cross-store isolation from uncontrolled writers; deployed exclusivity is separate.
func ensureCreatedResourcesUnreferenced(ctx context.Context, q libraryClosureReader, ids []string) error {
func ensureCreatedResourcesUnreferenced(ctx context.Context, q libraryClosureReader, tables, ids []string) error {
var referenced bool
err := q.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM resources WHERE id=ANY($1::text[]) AND
(EXISTS(SELECT 1 FROM resource_bindings b WHERE b.resource_id=resources.id) OR EXISTS (`+libraryResourceBodyReferenceSQL("TRUE")+`)))`, ids).Scan(&referenced)
(EXISTS(SELECT 1 FROM resource_bindings b WHERE b.resource_id=resources.id) OR EXISTS (`+libraryResourceBodyReferenceSQL(tables, "TRUE")+`)))`, ids).Scan(&referenced)
if err != nil {
return err
}
@@ -9,6 +9,7 @@ import (
"fmt"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/libraryprofile"
"knowledge-governance/internal/librarysnapshot"
)
@@ -22,7 +23,7 @@ func (t *ProductionLibraryTarget) migrationCatalog(ctx context.Context) (migrati
return nil, "", err
}
defer tx.Rollback(context.WithoutCancel(ctx))
catalog, digest, err := captureMigrationCatalog(ctx, tx)
catalog, digest, err := captureMigrationCatalogFor(ctx, tx, t.tables)
if err != nil {
return nil, "", err
}
@@ -32,8 +33,15 @@ func (t *ProductionLibraryTarget) migrationCatalog(ctx context.Context) (migrati
return catalog, digest, nil
}
// Source rows and their definitions must come from the same read-only transaction.
// captureMigrationCatalog 是 0.7.1(既有表集合)的入口,行为保持不变。
func captureMigrationCatalog(ctx context.Context, tx pgx.Tx) (migrationCatalog, string, error) {
return captureMigrationCatalogFor(ctx, tx, productionLibraryTables)
}
// captureMigrationCatalogFor 按 profile 解析出的表集合取定义:0.9.0 没有
// wiki_log_entries,用固定清单会让整库编目在第一张不存在的表上失败。
// Source rows and their definitions must come from the same read-only transaction.
func captureMigrationCatalogFor(ctx context.Context, tx pgx.Tx, tables []string) (migrationCatalog, string, error) {
queries := map[string]string{
"columns": `SELECT c.relname AS table_name,a.attname AS column_name,format_type(a.atttypid,a.atttypmod) AS type,a.attnotnull AS not_null,a.attidentity AS identity,a.attgenerated AS generated,coalesce(pg_get_expr(d.adbin,d.adrelid),'') AS default_expr FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid JOIN pg_namespace n ON n.oid=c.relnamespace LEFT JOIN pg_attrdef d ON d.adrelid=c.oid AND d.adnum=a.attnum WHERE n.nspname='public' AND c.relname=ANY($1) AND a.attnum>0 AND NOT a.attisdropped ORDER BY c.relname,a.attnum`,
"constraints": `SELECT ns.nspname AS namespace,c.relname AS table_name,x.conname AS name,x.contype AS type,pg_get_constraintdef(x.oid) AS definition,coalesce(r.relname,'') AS referenced_table,coalesce(rns.nspname,'') AS referenced_namespace FROM pg_constraint x JOIN pg_class c ON c.oid=x.conrelid JOIN pg_namespace ns ON ns.oid=c.relnamespace LEFT JOIN pg_class r ON r.oid=x.confrelid LEFT JOIN pg_namespace rns ON rns.oid=r.relnamespace WHERE (ns.nspname='public' AND c.relname=ANY($1)) OR (rns.nspname='public' AND r.relname=ANY($1)) ORDER BY ns.nspname,c.relname,x.conname`,
@@ -45,7 +53,7 @@ func captureMigrationCatalog(ctx context.Context, tx pgx.Tx) (migrationCatalog,
catalog := migrationCatalog{}
for key, query := range queries {
var raw []byte
if err := tx.QueryRow(ctx, "SELECT coalesce(jsonb_agg(to_jsonb(q)),'[]'::jsonb) FROM ("+query+") q", productionLibraryTables).Scan(&raw); err != nil {
if err := tx.QueryRow(ctx, "SELECT coalesce(jsonb_agg(to_jsonb(q)),'[]'::jsonb) FROM ("+query+") q", tables).Scan(&raw); err != nil {
return nil, "", fmt.Errorf("capture migration catalog %s: %w", key, err)
}
catalog[key] = raw
@@ -59,12 +67,39 @@ func captureMigrationCatalog(ctx context.Context, tx pgx.Tx) (migrationCatalog,
}
func validateMigrationCatalog(catalog migrationCatalog) error {
var triggers []json.RawMessage
profile, err := libraryprofile.For(libraryprofile.LegacyVersion)
if err != nil {
return err
}
return validateMigrationCatalogFor(catalog, productionLibraryTables, profile)
}
func validateMigrationCatalogFor(catalog migrationCatalog, tables []string, profile libraryprofile.Profile) error {
var triggers []struct {
Table string `json:"table_name"`
Name string `json:"name"`
Definition string `json:"definition"`
}
if err := json.Unmarshal(catalog["triggers"], &triggers); err != nil {
return err
}
if len(triggers) > 0 {
return errors.New("unreviewed migration trigger effects")
// 触发器 = 写一行还会改别的表。只接受 profile 里逐字核对过的定义,
// 且它们必须真实存在:消失了说明这也不是我们审过的那份 schema。
reviewed := map[string]bool{}
for _, definition := range profile.ReviewedTriggers {
reviewed[definition] = true
}
seen := map[string]bool{}
for _, trigger := range triggers {
if !reviewed[trigger.Definition] {
return fmt.Errorf("unreviewed migration trigger %s on %s", trigger.Name, trigger.Table)
}
seen[trigger.Definition] = true
}
for definition := range reviewed {
if !seen[definition] {
return errors.New("reviewed migration trigger is missing from this schema version")
}
}
var relations []struct {
Table string `json:"table_name"`
@@ -76,7 +111,7 @@ func validateMigrationCatalog(catalog migrationCatalog) error {
if err := json.Unmarshal(catalog["relations"], &relations); err != nil {
return err
}
if len(relations) != len(productionLibraryTables) {
if len(relations) != len(tables) {
return errors.New("incomplete migration table catalog")
}
for _, r := range relations {
@@ -115,7 +150,7 @@ func (t *ProductionLibraryTarget) preflightMigrationCatalog(ctx context.Context,
if err != nil {
return "", err
}
if err := validateMigrationCatalog(catalog); err != nil {
if err := validateMigrationCatalogFor(catalog, t.tables, t.schema.Profile); err != nil {
return "", err
}
var columns []struct {
@@ -17,8 +17,12 @@ func ScopeContentCounts(ctx context.Context, pool *pgxpool.Pool, identity Produc
if pool == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) {
return nil, errors.New("invalid scope identity")
}
counts := make(map[string]int64, len(productionLibraryTables))
for _, table := range productionLibraryTables {
profile, err := ResolveSchemaProfile(ctx, pool, "scope content probe")
if err != nil {
return nil, err
}
counts := make(map[string]int64, len(profile.Tables))
for _, table := range profile.Tables {
var count int64
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`, table, scopeWhere(table)),
identity.TenantID, identity.KnowledgeBaseID).Scan(&count); err != nil {
@@ -24,8 +24,14 @@ import (
// 写坏"(资源表的 ON CONFLICT 就会这样)在指纹上完全看不出来——数量、身份都一样。
// 因此摘要取自每行 to_jsonb 的 md5,再按身份排序聚合。
func universeFingerprint(ctx context.Context, pool *pgxpool.Pool) (map[string]libraryrelease.TableFingerprint, error) {
result := make(map[string]libraryrelease.TableFingerprint, len(productionLibraryTables))
for _, table := range productionLibraryTables {
// 表集合来自目标库自己的 profile:0.9.0 没有 wiki_log_entries,按 0.7.1 的固定
// 清单取指纹会在第一张不存在的表上失败,而"跳过缺表"又会让 0.7.1 少表蒙混过去。
schema, err := ResolveSchemaProfile(ctx, pool, "whole-instance fingerprint")
if err != nil {
return nil, err
}
result := make(map[string]libraryrelease.TableFingerprint, len(schema.Tables))
for _, table := range schema.Tables {
identity := identityExpr(table)
statement := fmt.Sprintf(
`SELECT count(*), coalesce(md5(string_agg(md5(to_jsonb(x)::text), E'\n' ORDER BY (%s)::text)), '') FROM %s AS x`,
@@ -76,12 +76,12 @@ func (t *ProductionLibraryTarget) productionTableContent(ctx context.Context, ta
// the legacy per-resource body scan made post-sync verification minutes
// long on a small target inside a large tenant.
var bodies, bindings []json.RawMessage
for _, name := range productionLibraryTables {
for _, name := range t.tables {
if name == "resources" {
continue
}
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`,
name, name, scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
name, name, t.scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
if err != nil {
return 0, "", fmt.Errorf("读取生产 %s 内容失败: %w", table, err)
}
@@ -98,7 +98,7 @@ func (t *ProductionLibraryTarget) productionTableContent(ctx context.Context, ta
}
rows = items
} else {
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table)),
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, t.scopeWhere(table)),
identity.TenantID, identity.KnowledgeBaseID)
if err != nil {
return 0, "", fmt.Errorf("读取生产 %s 内容失败: %w", table, err)
@@ -401,7 +401,7 @@ func (t *ProductionLibraryTarget) clearJobCreatedRows(ctx context.Context, set l
return err
}
if err := ensureCreatedResourcesUnreferenced(ctx, tx, ids); err != nil {
if err := ensureCreatedResourcesUnreferenced(ctx, tx, t.tables, ids); err != nil {
return err
}
}
@@ -412,7 +412,7 @@ func (t *ProductionLibraryTarget) clearJobCreatedRows(ctx context.Context, set l
// 复核:本次新建的行一个都不许剩。资源若因"被引用"而保留,那说明恢复没回到
// 预备份状态,必须报出来让维护保持,而不是静默放过。
leftovers := make([]string, 0, 4)
for _, table := range productionLibraryTables {
for _, table := range t.tables {
ids := make([]string, 0, 8)
for _, id := range set.CreatedIDs(table) {
if restored[table][id] {
@@ -10,6 +10,8 @@ import (
"sort"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/libraryprofile"
)
var resourceReferencePattern = regexp.MustCompile(`resource://([A-Za-z0-9_-]{22})`)
@@ -30,6 +32,29 @@ type TableSnapshot struct {
type PostgresSnapshot struct {
Catalog migrationCatalog `json:"catalog,omitempty"`
Tables []TableSnapshot
// SchemaVersion 是导出/导入两侧共同的 profile 标识:归档必须与目标同版本,
// 否则行里的列语义(例如 0.9.0 新增的 NOT NULL 列)不可比。
SchemaVersion int `json:"schema_version,omitempty"`
Profile string `json:"profile,omitempty"`
}
// schemaProfile 解析快照声明的 profile。0 表示既有内存夹具(0.7.1),
// 只把"没有声明版本"解释成 74,不把未知版本解释成任何 profile。
func (snapshot PostgresSnapshot) schemaProfile() (SchemaProfile, error) {
profile, err := libraryprofile.For(snapshot.SchemaVersion)
if err != nil {
if snapshot.SchemaVersion != 0 {
return SchemaProfile{}, err
}
profile, err = libraryprofile.For(libraryprofile.LegacyVersion)
if err != nil {
return SchemaProfile{}, err
}
}
resolved := SchemaProfile{Profile: profile}
resolved.Tables = append(resolved.Tables, profile.RequiredTables...)
resolved.Tables = append(resolved.Tables, profile.OptionalTables...)
return resolved, nil
}
func (snapshot PostgresSnapshot) Table(name string) TableSnapshot {
+25 -14
View File
@@ -67,10 +67,13 @@ func (importer *PostgresImporter) Replace(ctx context.Context, identity Postgres
return errors.New("begin PostgreSQL document import")
}
defer tx.Rollback(ctx)
var version int
var dirty bool
if err := tx.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
return errors.New("target WeKnora schema contract mismatch")
profile, err := ResolveSchemaProfile(ctx, tx, "target WeKnora")
if err != nil {
return err
}
// 目标版本必须与归档同 profile:0.7.1 的归档不能写进 0.9.0,反之亦然。
if err := requireSameProfile(snapshot.SchemaVersion, profile, "import"); err != nil {
return err
}
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, identity.KnowledgeID); err != nil {
return errors.New("lock target knowledge")
@@ -177,10 +180,12 @@ func (importer *PostgresImporter) ReplaceWiki(ctx context.Context, identity Post
return errors.New("begin PostgreSQL wiki import")
}
defer tx.Rollback(ctx)
var version int
var dirty bool
if err := tx.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
return errors.New("target WeKnora schema contract mismatch")
profile, err := ResolveSchemaProfile(ctx, tx, "target WeKnora")
if err != nil {
return err
}
if err := requireSameProfile(snapshot.SchemaVersion, profile, "wiki import"); err != nil {
return err
}
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 1))`, identity.KnowledgeBaseID); err != nil {
return errors.New("lock target wiki knowledge base")
@@ -205,10 +210,8 @@ func (importer *PostgresImporter) Disable(ctx context.Context, identity Postgres
return errors.New("begin PostgreSQL disable")
}
defer tx.Rollback(ctx)
var version int
var dirty bool
if err := tx.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
return errors.New("target WeKnora schema contract mismatch")
if _, err := ResolveSchemaProfile(ctx, tx, "target WeKnora"); err != nil {
return err
}
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, identity.KnowledgeID); err != nil {
return errors.New("lock target knowledge")
@@ -244,9 +247,15 @@ func (importer *PostgresImporter) prepareSnapshot(identity PostgresIdentity, sna
return PostgresSnapshot{}, nil, nil, errors.New("PostgreSQL import snapshot is incomplete")
}
prepared := cloneSnapshot(snapshot)
// 表集合按快照所属 profile 判定:0.9.0 的归档里有 wiki_page_issues,
// 面向单文档发布的白名单没有它,用白名单会把合法归档判成非法。
profile, err := snapshot.schemaProfile()
if err != nil {
return PostgresSnapshot{}, nil, nil, err
}
known := map[string]bool{}
for index, table := range prepared.Tables {
if !validReleaseTable(table.Name) || known[table.Name] || table.RowCount != len(table.Rows) || newTableSnapshot(table.Name, table.Rows).SHA256 != table.SHA256 {
if !profile.HasTable(table.Name) || known[table.Name] || table.RowCount != len(table.Rows) || newTableSnapshot(table.Name, table.Rows).SHA256 != table.SHA256 {
return PostgresSnapshot{}, nil, nil, errors.New("PostgreSQL import snapshot is invalid")
}
known[table.Name] = true
@@ -516,7 +525,9 @@ func validObjectLocator(value string) bool {
}
func cloneSnapshot(snapshot PostgresSnapshot) PostgresSnapshot {
result := PostgresSnapshot{Tables: make([]TableSnapshot, len(snapshot.Tables))}
// 复制快照不得改契约:SchemaVersion/Profile 丢掉会让下游按未知版本(=0.7.1)处理。
result := PostgresSnapshot{Tables: make([]TableSnapshot, len(snapshot.Tables)),
SchemaVersion: snapshot.SchemaVersion, Profile: snapshot.Profile}
for index, table := range snapshot.Tables {
result.Tables[index] = table
result.Tables[index].Rows = append([]json.RawMessage(nil), table.Rows...)
+4 -1
View File
@@ -135,5 +135,8 @@ func replaceTable(snapshot PostgresSnapshot, replacement TableSnapshot) Postgres
if !replaced {
tables = append(tables, replacement)
}
return PostgresSnapshot{Tables: tables}
// SchemaVersion/Profile 描述的是这份快照的 schema 契约,不是表内容:重建快照时
// 丢掉它们,下游读到的就是"版本未知",而未知版本按 0.7.1 处理——134 的归档
// 于是被当成 74 搬运,跨版本闸门形同不存在。
return PostgresSnapshot{Tables: tables, SchemaVersion: snapshot.SchemaVersion, Profile: snapshot.Profile}
}
@@ -173,3 +173,26 @@ func TestPlanItemTargetBaseFallsBackToTestedBase(t *testing.T) {
t.Fatalf("unbound TargetBase = %q, want the tested base", unbound.TargetBase())
}
}
// 范围改写只换身份,不改 schema 契约:改写后的快照若丢掉 SchemaVersion/Profile,
// 下游读到的就是"版本未知",而未知版本会被当作 0.7.1 处理——134 的归档于是被当成
// 74 搬,跨版本闸门形同不存在。
func TestRemapKnowledgeBasePreservesSchemaProfile(t *testing.T) {
bundle := crossBaseBundle(t)
bundle.Postgres.SchemaVersion = 134
bundle.Postgres.Profile = "v090"
if err := bundle.RemapKnowledgeBase("prod-kb"); err != nil {
t.Fatalf("remap: %v", err)
}
if bundle.Postgres.SchemaVersion != 134 || bundle.Postgres.Profile != "v090" {
t.Fatalf("remapped snapshot lost its schema profile: %d/%q",
bundle.Postgres.SchemaVersion, bundle.Postgres.Profile)
}
tables := map[string]bool{}
for _, table := range bundle.Postgres.Tables {
tables[table.Name] = true
}
if len(tables) != len(bundle.Postgres.Tables) {
t.Fatal("remap duplicated a table")
}
}
@@ -0,0 +1,50 @@
package v071
import (
"knowledge-governance/internal/libraryprofile"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
)
// schemaProbeRows 给"按 SQL 打桩"的既有夹具提供 information_schema 能力探测答案。
// 0.7.1 的假库一律按 profile 声明的表/列回答,这样能力探测在单测里也是真的被走了一遍,
// 而不是被桩悄悄放行。
type schemaProbeRows struct {
pairs [][2]string
index int
}
func legacySchemaProbeRows() *schemaProbeRows {
profile, err := libraryprofile.For(libraryprofile.LegacyVersion)
if err != nil {
panic(err)
}
rows := &schemaProbeRows{}
for _, table := range profile.RequiredTables {
for _, column := range profile.RequiredColumns[table] {
rows.pairs = append(rows.pairs, [2]string{table, column})
}
}
return rows
}
func (r *schemaProbeRows) Close() {}
func (r *schemaProbeRows) Err() error { return nil }
func (r *schemaProbeRows) CommandTag() pgconn.CommandTag { return pgconn.CommandTag{} }
func (r *schemaProbeRows) FieldDescriptions() []pgconn.FieldDescription { return nil }
func (r *schemaProbeRows) RawValues() [][]byte { return nil }
func (r *schemaProbeRows) Conn() *pgx.Conn { return nil }
func (r *schemaProbeRows) Next() bool {
if r.index >= len(r.pairs) {
return false
}
r.index++
return true
}
func (r *schemaProbeRows) Scan(dest ...any) error {
*dest[0].(*string) = r.pairs[r.index-1][0]
*dest[1].(*string) = r.pairs[r.index-1][1]
return nil
}
func (r *schemaProbeRows) Values() ([]any, error) { return nil, nil }
@@ -0,0 +1,145 @@
package v071
import (
"context"
"fmt"
"sort"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/libraryprofile"
)
// SchemaProfile 是"版本号 + 能力探测"两者都通过之后的解析结果。
//
// Tables 是**实际存在**的表(必需表按声明顺序 + 存在的可选表)。导出/编目/校验
// 只按它取数:0.9.0 没有 wiki_log_entries,硬编码表清单会在第一条查询上炸,
// 而"表不存在就跳过约束"会让 0.7.1 少一个必需表也悄悄通过。
type SchemaProfile struct {
libraryprofile.Profile
Tables []string
}
// HasTable 报告解析后的表集合里是否包含该表。
func (p SchemaProfile) HasTable(name string) bool {
for _, candidate := range p.Tables {
if candidate == name {
return true
}
}
return false
}
// CatalogTables 是编目/归档用的表清单(与 Tables 同序,sorted 便于稳定摘要)。
func (p SchemaProfile) CatalogTables() []string {
tables := append([]string(nil), p.Tables...)
sort.Strings(tables)
return tables
}
type schemaProbeDB interface {
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}
// ResolveSchemaProfile 解析并校验一侧的 schema 契约。
//
// 顺序是刻意的:先读版本(未知版本失败关闭、dirty 一律失败),再用
// information_schema 证明 profile 声明的东西真的存在。只信版本号等于把
// "迁移进度"当成"能力契约",而两者在生产上经常不同步。
func ResolveSchemaProfile(ctx context.Context, db schemaProbeDB, role string) (SchemaProfile, error) {
var version int
var dirty bool
if err := db.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations ORDER BY version DESC LIMIT 1`).Scan(&version, &dirty); err != nil {
return SchemaProfile{}, fmt.Errorf("read %s schema_migrations: %w", role, err)
}
profile, err := profileForMigration(version, dirty, role)
if err != nil {
return SchemaProfile{}, err
}
columns, err := probeSchemaColumns(ctx, db, profile)
if err != nil {
return SchemaProfile{}, fmt.Errorf("%s schema version %d: %w", role, version, err)
}
return verifySchemaCapabilities(profile, columns, role, version)
}
// profileForMigration 只做"版本号 + dirty → profile"的判定,不碰数据库。
// dirty 与未知版本都必须失败:前者是半截迁移,后者是没见过的契约。
func profileForMigration(version int, dirty bool, role string) (libraryprofile.Profile, error) {
if dirty {
return libraryprofile.Profile{}, fmt.Errorf("%s schema is dirty at version %d: refusing to snapshot or write", role, version)
}
profile, err := libraryprofile.For(version)
if err != nil {
return libraryprofile.Profile{}, fmt.Errorf("%s schema: %w", role, err)
}
return profile, nil
}
// verifySchemaCapabilities 用 information_schema 的实际列集合校正 profile:
// 必需表/列缺失就精确点名,存在才进"实际表集合"。
func verifySchemaCapabilities(profile libraryprofile.Profile, columns map[string]map[string]bool, role string, version int) (SchemaProfile, error) {
resolved := SchemaProfile{Profile: profile}
for _, table := range profile.RequiredTables {
if _, ok := columns[table]; !ok {
return SchemaProfile{}, fmt.Errorf("%s schema version %d: required table %s is missing", role, version, table)
}
resolved.Tables = append(resolved.Tables, table)
}
for _, table := range profile.OptionalTables {
if _, ok := columns[table]; ok {
resolved.Tables = append(resolved.Tables, table)
}
}
for _, table := range resolved.Tables {
for _, column := range profile.RequiredColumns[table] {
if !columns[table][column] {
return SchemaProfile{}, fmt.Errorf("%s schema version %d: required column %s.%s is missing", role, version, table, column)
}
}
}
return resolved, nil
}
// probeSchemaColumns 一次读回 profile 涉及表的列集合。
func probeSchemaColumns(ctx context.Context, db schemaProbeDB, profile libraryprofile.Profile) (map[string]map[string]bool, error) {
names := make([]string, 0, len(profile.RequiredTables)+len(profile.OptionalTables))
names = append(names, profile.RequiredTables...)
names = append(names, profile.OptionalTables...)
sort.Strings(names)
rows, err := db.Query(ctx, `SELECT table_name, column_name FROM information_schema.columns WHERE table_schema='public' AND table_name=ANY($1)`, names)
if err != nil {
return nil, fmt.Errorf("probe schema capabilities: %w", err)
}
defer rows.Close()
columns := map[string]map[string]bool{}
for rows.Next() {
var table, column string
if err := rows.Scan(&table, &column); err != nil {
return nil, fmt.Errorf("probe schema capabilities: %w", err)
}
if columns[table] == nil {
columns[table] = map[string]bool{}
}
columns[table][column] = true
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("probe schema capabilities: %w", err)
}
return columns, nil
}
// requireSameProfile 拒绝跨版本搬运:0.9.0 归档不能落到 0.7.1,反之也不行。
// 本轮只支持 090↔090(以及既有 74↔74)。
func requireSameProfile(sourceVersion int, target SchemaProfile, direction string) error {
if sourceVersion == 0 {
// 老的内存夹具没有声明版本:按 0.7.1 处理,行为不变。
sourceVersion = libraryprofile.LegacyVersion
}
if sourceVersion != target.SchemaVersion {
return fmt.Errorf("%s refused: archive schema version %d cannot be applied to target schema version %d (cross-version %s is not supported)",
direction, sourceVersion, target.SchemaVersion, direction)
}
return nil
}
@@ -0,0 +1,126 @@
package v071
import (
"os"
"strings"
"testing"
"knowledge-governance/internal/libraryprofile"
)
func TestSchemaProfileFailsClosedOnUnknownVersion(t *testing.T) {
_, err := profileForMigration(135, false, "test")
if err == nil || !strings.Contains(err.Error(), "135") || !strings.Contains(err.Error(), "134") {
t.Fatalf("unknown version must fail with the real version number, got %v", err)
}
}
func TestSchemaProfileRejectsDirtySchema(t *testing.T) {
_, err := profileForMigration(134, true, "test")
if err == nil || !strings.Contains(err.Error(), "dirty") {
t.Fatalf("dirty schema must be refused, got %v", err)
}
}
func TestSchemaProfileNamesMissingTableAndColumn(t *testing.T) {
profile, err := libraryprofile.For(134)
if err != nil {
t.Fatal(err)
}
columns := map[string]map[string]bool{}
for _, table := range append(append([]string{}, profile.RequiredTables...), profile.OptionalTables...) {
columns[table] = map[string]bool{}
for _, column := range profile.RequiredColumns[table] {
columns[table][column] = true
}
}
delete(columns, "wiki_pages")
if _, err := verifySchemaCapabilities(profile, columns, "test", 134); err == nil || !strings.Contains(err.Error(), "wiki_pages") {
t.Fatalf("missing required table must be named, got %v", err)
}
columns["wiki_pages"] = map[string]bool{"id": true, "tenant_id": true, "knowledge_base_id": true}
if _, err := verifySchemaCapabilities(profile, columns, "test", 134); err == nil || !strings.Contains(err.Error(), "wiki_pages.slug") {
t.Fatalf("missing required column must be named, got %v", err)
}
columns["wiki_pages"]["slug"] = true
if _, err := verifySchemaCapabilities(profile, columns, "test", 134); err != nil {
t.Fatalf("complete capabilities must pass: %v", err)
}
}
func TestSchemaProfileV071BehaviourUnchanged(t *testing.T) {
legacy, err := libraryprofile.For(74)
if err != nil {
t.Fatal(err)
}
if !legacy.HasTable("wiki_log_entries") || !legacy.RequiresPart("wiki/wiki_log_entries") {
t.Fatal("0.7.1 must still require wiki_log_entries")
}
if legacy.EmbeddingDimension != 1024 {
t.Fatalf("0.7.1 dimension contract must stay 1024, got %d", legacy.EmbeddingDimension)
}
v090, err := libraryprofile.For(134)
if err != nil {
t.Fatal(err)
}
if v090.RequiresPart("wiki/wiki_log_entries") || !v090.AllowsPart("wiki/wiki_log_entries") {
t.Fatal("0.9.0 wiki_log_entries must be optional")
}
if v090.HasTable("wiki_log_entries") == false {
t.Fatal("optional table should still be addressable when present")
}
resolved := SchemaProfile{Profile: v090, Tables: v090.RequiredTables}
if resolved.HasTable("wiki_log_entries") {
t.Fatal("0.9.0 export table set must exclude an absent optional table")
}
}
func TestTargetIndexNameResolvesToConfigurationOnly(t *testing.T) {
profile, err := libraryprofile.For(134)
if err != nil {
t.Fatal(err)
}
name, err := resolveTargetIndexName("", "weknora_v090_chunks", profile)
if err != nil || name != "weknora_v090_chunks" {
t.Fatalf("empty vector store must fall back to the configured index, got %q/%v", name, err)
}
if _, err := resolveTargetIndexName("", "", profile); err == nil || !strings.Contains(err.Error(), "index name unresolved") {
t.Fatalf("no source must fail instead of resolving to an empty name, got %v", err)
}
if _, err := resolveTargetIndexName("idx_a", "idx_b", profile); err == nil || !strings.Contains(err.Error(), "does not match") {
t.Fatalf("conflicting index sources must be refused, got %v", err)
}
}
// 0.9.0 与 0.7.1 的映射必须算出不同摘要:同样的 1024 维不等于同一份契约。
func TestIndexMappingDigestSeparatesSchemaVersions(t *testing.T) {
legacy := []byte(`{"weknora_v071_chunks":{"mappings":{"properties":{
"chunk_id":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
"knowledge_id":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
"knowledge_base_id":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
"embedding":{"type":"dense_vector","dims":1024}}}}}`)
legacyDigest, err := IndexMappingDigest(legacy)
if err != nil {
t.Fatal(err)
}
if path := os.Getenv("KG_V090_MAPPING_JSON"); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
v090Digest, err := IndexMappingDigest(raw)
if err != nil {
t.Fatalf("real 0.9.0 mapping must be digestible: %v", err)
}
if v090Digest == legacyDigest {
t.Fatal("0.9.0 and 0.7.1 mappings must not share a digest")
}
if len(v090Digest) != 64 {
t.Fatalf("unexpected digest shape %q", v090Digest)
}
}
if _, err := IndexMappingDigest([]byte(`{"a":{"mappings":{}},"b":{"mappings":{}}}`)); err == nil {
t.Fatal("multi-index mapping responses must be refused")
}
}
@@ -0,0 +1,183 @@
package v071
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"io"
"os"
"strconv"
"strings"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
)
// 0.9.0 fixture:真实 pg_dump(schema_migrations=134,83 表,无 wiki_log_entries)。
// 只用于显式指定的隔离库;governance 与周报邮件库一律不许指到这里。
const v090SchemaEnv = "KG_V090_SCHEMA_SQL"
func v090SchemaSQL(t *testing.T) string {
t.Helper()
path := os.Getenv(v090SchemaEnv)
if path == "" {
t.Skip("requires " + v090SchemaEnv + " pointing at the captured WeKnora 0.9.0 schema dump")
}
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", v090SchemaEnv, err)
}
lines := strings.Split(string(raw), "\n")
kept := make([]string, 0, len(lines))
for _, line := range lines {
// psql 元命令(\restrict/\unrestrict)只对 psql 有意义,驱动执行要剔除。
if strings.HasPrefix(line, "\\") {
continue
}
kept = append(kept, line)
}
// 装载会话本身也要把 search_path 还给 public,否则后续查询会找不到表。
return strings.Join(kept, "\n") + "\nSET search_path TO public, pg_catalog;\n"
}
// v090Pool 打开一个显式隔离库并重建 public schema,然后装 0.9.0 真实 schema。
func v090Pool(t *testing.T, env string) *pgxpool.Pool {
t.Helper()
dsn := os.Getenv(env)
if dsn == "" {
t.Skip("requires " + env)
}
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
t.Fatalf("%s must name an explicitly isolated kg_library_test_ database", env)
}
// dump 里有 set_config('search_path',''),新连接必须显式给回 public。
cfg.ConnConfig.RuntimeParams["search_path"] = "public, pg_catalog"
ctx := context.Background()
db, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
t.Fatalf("connect: %v", err)
}
t.Cleanup(db.Close)
if _, err = db.Exec(ctx, `DROP SCHEMA public CASCADE; CREATE SCHEMA public`); err != nil {
t.Fatalf("reset schema: %v", err)
}
if _, err = db.Exec(ctx, v090SchemaSQL(t)); err != nil {
t.Fatalf("install 0.9.0 schema: %v", err)
}
// 该 dump 会把会话 search_path 置空,这里一律写全限定名。
if _, err = db.Exec(ctx, `INSERT INTO public.schema_migrations(version,dirty) VALUES (134,false)`); err != nil {
t.Fatalf("stamp schema_migrations: %v", err)
}
return db
}
const v090Tenant = 10000
const v090KB = "kb-v090"
const v090Knowledge = "kn-v090-doc"
const v090ChunkOne = "ck-v090-1"
const v090ChunkTwo = "ck-v090-2"
const v090Folder = "wf-v090-1"
const v090Page = "wp-v090-1"
const v090Tag = "tg-v090-1"
const v090Resource = "rs-v090-1"
// seedV090 造 1 KB / 1 文档 / 2 分块 / 1 wiki 页 / 1 wiki 目录 / 1 资源。
// Wiki 必须非空:空表通过等于没有验证 wiki 部件。
func seedV090(t *testing.T, ctx context.Context, db *pgxpool.Pool) PostgresIdentity {
t.Helper()
statements := []string{
`INSERT INTO models(id,tenant_id,name,type,source,parameters) VALUES ('bge-m3-v090',$1,'bge-m3','embedding','local','{"embedding_parameters":{"dimension":1024}}'::jsonb)`,
`INSERT INTO knowledge_bases(id,name,tenant_id,embedding_model_id,summary_model_id,indexing_strategy) VALUES ('` + v090KB + `','v090 fixture',$1,'bge-m3-v090','bge-m3-v090','{"graph_enabled":false}'::jsonb)`,
`INSERT INTO knowledges(id,tenant_id,knowledge_base_id,type,title,source,parse_status,pending_subtasks_count) VALUES ('` + v090Knowledge + `',$1,'` + v090KB + `','file','0.9.0 夹具','fixture://kn-v090','completed',0)`,
`INSERT INTO chunks(id,tenant_id,knowledge_base_id,knowledge_id,content,chunk_index,start_at,end_at) VALUES
('` + v090ChunkOne + `',$1,'` + v090KB + `','` + v090Knowledge + `','第一段正文',0,0,5),
('` + v090ChunkTwo + `',$1,'` + v090KB + `','` + v090Knowledge + `','第二段正文',1,5,10)`,
`INSERT INTO wiki_folders(id,tenant_id,knowledge_base_id,name,path) VALUES ('` + v090Folder + `',$1,'` + v090KB + `','夹具目录','/fixture')`,
`INSERT INTO wiki_pages(id,tenant_id,knowledge_base_id,slug,title,content,source_refs,chunk_refs,folder_id) VALUES ('` + v090Page + `',$1,'` + v090KB + `','fixture-page','夹具页','wiki 正文','["` + v090Knowledge + `"]'::jsonb,'["` + v090ChunkOne + `"]'::jsonb,'` + v090Folder + `')`,
`INSERT INTO knowledge_tags(id,tenant_id,knowledge_base_id,name) VALUES ('` + v090Tag + `',$1,'` + v090KB + `','夹具标签')`,
`INSERT INTO knowledge_tag_relations(knowledge_id,tag_id) VALUES ('` + v090Knowledge + `','` + v090Tag + `')`,
}
for _, statement := range statements {
if _, err := db.Exec(ctx, strings.ReplaceAll(statement, "$1", strconv.Itoa(v090Tenant))); err != nil {
t.Fatalf("seed: %v\n%s", err, statement)
}
}
// 资源行必须带真实的 physical_path/size/content_hash:封存会逐字节核对。
content := []byte("fixture-object-bytes")
sum := sha256.Sum256(content)
if _, err := db.Exec(ctx, `INSERT INTO resources(id,handle,tenant_id,provider,physical_path,location_hash,size,content_hash)
VALUES ($1,$2,$3,'minio','fixture/kn-v090.bin',$4,$5,$6)`,
v090Resource, "abcdefghijklmnopqrstuv", v090Tenant, "fixture-location-hash", int64(len(content)), hex.EncodeToString(sum[:])); err != nil {
t.Fatalf("seed resource: %v", err)
}
if _, err := db.Exec(ctx, `INSERT INTO resource_bindings(id,resource_id,tenant_id,owner_type,owner_id)
VALUES ('rb-v090-1',$1,$2,'knowledge',$3)`, v090Resource, v090Tenant, v090Knowledge); err != nil {
t.Fatalf("seed binding: %v", err)
}
return PostgresIdentity{TenantID: v090Tenant, KnowledgeBaseID: v090KB}
}
// seedV090Binding 只在目标库造 KB/模型绑定:导入的是文档行,不是 KB 定义。
func seedV090Binding(t *testing.T, ctx context.Context, db *pgxpool.Pool) error {
t.Helper()
for _, statement := range []string{
`INSERT INTO models(id,tenant_id,name,type,source,parameters) VALUES ('bge-m3-v090',$1,'bge-m3','embedding','local','{"embedding_parameters":{"dimension":1024}}'::jsonb)`,
`INSERT INTO knowledge_bases(id,name,tenant_id,embedding_model_id,summary_model_id,indexing_strategy) VALUES ('` + v090KB + `','v090 fixture target',$1,'bge-m3-v090','bge-m3-v090','{"graph_enabled":false}'::jsonb)`,
} {
if _, err := db.Exec(ctx, strings.ReplaceAll(statement, "$1", strconv.Itoa(v090Tenant))); err != nil {
return err
}
}
return nil
}
// v090Objects 只在测试里替代对象存储:字节来自夹具常量,不连任何 S3/OSS。
type v090Objects map[string][]byte
func (o v090Objects) Open(_ context.Context, provider, physicalPath string) (io.ReadCloser, error) {
if provider != "minio" {
return nil, errors.New("unsupported provider in fixture")
}
content, ok := o[physicalPath]
if !ok {
return nil, errors.New("fixture object missing")
}
return io.NopCloser(bytes.NewReader(content)), nil
}
// v090IndexSnapshot 按导出分块合成索引快照:本切片不做真实 ES 采集,
// 但文档/映射摘要走的是与线上同一套校验,不能凭空绕过。
func v090IndexSnapshot(t *testing.T, pg PostgresSnapshot, mappingSHA string) SearchSnapshot {
t.Helper()
var documents []SearchDocument
for _, raw := range pg.Table("chunks").Rows {
var chunk struct {
ID string `json:"id"`
TenantID uint64 `json:"tenant_id"`
KnowledgeBaseID string `json:"knowledge_base_id"`
KnowledgeID string `json:"knowledge_id"`
Content string `json:"content"`
}
if err := json.Unmarshal(raw, &chunk); err != nil {
t.Fatalf("decode chunk: %v", err)
}
source, err := json.Marshal(map[string]any{
"chunk_id": chunk.ID, "tenant_id": chunk.TenantID,
"knowledge_base_id": chunk.KnowledgeBaseID, "knowledge_id": chunk.KnowledgeID,
"content": chunk.Content, "source_id": chunk.ID, "source_type": 0,
})
if err != nil {
t.Fatalf("encode index document: %v", err)
}
documents = append(documents, SearchDocument{ID: "doc-" + chunk.ID, Source: source})
}
digest, err := searchDocumentsDigest(documents)
if err != nil {
t.Fatalf("index digest: %v", err)
}
return SearchSnapshot{Documents: documents, MappingSHA256: mappingSHA, DocumentsSHA256: digest}
}
+11 -2
View File
@@ -5,6 +5,7 @@ import (
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
@@ -152,8 +153,16 @@ func Validate(manifest Manifest, target TargetContract, mapping KnowledgeBaseMap
}
func validateManifestFields(manifest Manifest) error {
if manifest.FormatVersion != FormatVersionV1 || manifest.Contract.WeKnoraVersion != WeKnoraVersionV071 ||
manifest.Contract.MigrationVersion != WeKnoraMigrationV071 || manifest.Contract.EmbeddingDimension != EmbeddingDimensionV071 ||
// 单文档发布包只认 0.7.1 的表集合与契约。别的版本必须**明确拒绝并说明原因**,
// 不能落进下面那句泛化的 "unsupported release contract"——那让人分不清是版本
// 不支持还是字段写错了。0.9.0 的整库内容走整库发布路径(libraryprofile v090)。
if manifest.Contract.MigrationVersion != WeKnoraMigrationV071 ||
manifest.Contract.WeKnoraVersion != WeKnoraVersionV071 {
return fmt.Errorf("release bundle schema contract %s/%d is not supported: single-document release handles only v0.7.1 (74); WeKnora 0.9.0 (134) requires the library-level publish path",
manifest.Contract.WeKnoraVersion, manifest.Contract.MigrationVersion)
}
if manifest.FormatVersion != FormatVersionV1 ||
manifest.Contract.EmbeddingDimension != EmbeddingDimensionV071 ||
manifest.Contract.RetrieverEngine != "elasticsearch_v8" || manifest.Contract.TenantID == 0 || manifest.CreatedAt.IsZero() {
return errors.New("unsupported release contract")
}
+24
View File
@@ -165,3 +165,27 @@ func TestValidateRejectsUndeclaredOrDriftedMapping(t *testing.T) {
})
}
}
// 单文档发布包只支持 0.7.1 契约。0.9.0(134)的包必须被**明确拒绝并说明原因**:
// 静默按 74 处理会让 0.9.0 的表集合/索引文档走错写入路径,而泛化错误又让人分不清
// "版本不支持"和"字段写错"。
func TestSealRejectsNonV071SchemaContractExplicitly(t *testing.T) {
manifest := validManifest()
manifest.Contract.WeKnoraVersion = "v0.9.0"
manifest.Contract.MigrationVersion = 134
_, err := Seal(manifest)
if err == nil {
t.Fatal("0.9.0 release bundle must be refused")
}
for _, want := range []string{"v0.9.0", "134", "74", "library-level publish"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("rejection message must name %q, got %v", want, err)
}
}
// 只改版本号、不改 WeKnora 版本同样必须拒绝:两者必须成对出现。
manifest = validManifest()
manifest.Contract.MigrationVersion = 134
if _, err := Seal(manifest); err == nil {
t.Fatal("schema version 134 with a v0.7.1 label must be refused")
}
}
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# Whole-library capture entry point for the RDS-backed WeKnora 0.9.0 test stack
# (Compose project weknora-v090).
#
# Why a separate file instead of a parameterised one: this stack has no source
# PostgreSQL container and no governance PostgreSQL container. Both databases are
# managed instances (RDS). -source-pg / -governance-pg are therefore passed empty,
# which is libraryctl's documented signal to resolve them from the source app
# container env (DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD + LIBRARY_SOURCE_DB_SSLMODE)
# and from LIBRARY_GOVERNANCE_DSN_FILE (0600 file). Keeping the 0.7.1 entry point
# untouched means a change here can never alter a stack that already ran.
#
# Every service of the Compose project is declared below: exactly one writer (app)
# and a reason for each sibling which must NOT be stopped. ProjectController
# refuses to run if the project gains a service that is not declared here, so a
# new container cannot be silently left outside the maintenance window.
#
# Credentials are read at call time (container env / 0600 file) and are never
# printed. Nothing here writes to the source stack except SIGTERM/SIGSTART of the
# declared writer.
set -euo pipefail
# The binary is pinned by the caller on purpose: there is no "whatever is in this
# directory" fallback, so a stale build next to the script cannot be executed by
# accident against a live stack.
BIN="${LIBRARYCTL_BIN:-}"
if [ -z "$BIN" ]; then
echo "LIBRARYCTL_MISSING: set LIBRARYCTL_BIN to the libraryctl binary built for this run" >&2
exit 1
fi
if [ ! -x "$BIN" ]; then
echo "LIBRARYCTL_NOT_EXECUTABLE: $BIN is not an executable file" >&2
exit 1
fi
# Archives/journals live outside the home directory on this host (they survive the
# operator account); both roots stay overridable for an isolated rehearsal.
ARCHIVE_ROOT="${LIBRARY_ARCHIVE_ROOT:-/var/lib/knowledge-governance/library}"
JOURNAL_ROOT="${LIBRARY_JOURNAL_ROOT:-/var/lib/knowledge-governance/freeze}"
# Reasons must not contain commas: the flag is comma-separated.
NON_WRITERS="frontend=UI only; no knowledge writes"
NON_WRITERS+=",docreader=parse RPC service; app persists parse output"
NON_WRITERS+=",elasticsearch=search store; not an initiator"
NON_WRITERS+=",redis=queue/cache store; not an initiator"
exec "$BIN" "$@" \
-project weknora-v090 \
-writers app \
-source-app weknora-v090-app \
-source-es weknora-v090-elasticsearch \
-source-pg "" \
-governance-pg "" \
-non-writers "$NON_WRITERS" \
-archive "$ARCHIVE_ROOT" \
-journal "$JOURNAL_ROOT"
+10
View File
@@ -647,6 +647,16 @@
"package": "knowledge-governance/internal/httpapi",
"test": "TestWeeklyMailSendsHandlerReadsRealHistoryWithMessageID",
"reason": "需要一次性 PostgreSQL + 测试主密钥;验证 GET /weekly-mail/sends 在非空历史下返回 200 与真实 message_id,CI 不提供"
},
{
"package": "knowledge-governance/internal/releaseadapter/v071",
"test": "TestV090LibraryRoundTripIsolated",
"reason": "需 KG_V090_SCHEMA_SQL 指向真实 WeKnora 0.9.0 schema dump、KG_V090_MAPPING_JSON 指向真实索引映射,以及显式隔离的 kg_library_test_* PG;普通 CI 没有这些输入,跳过不是通过"
},
{
"package": "knowledge-governance/internal/releaseadapter/v071",
"test": "TestV090ProductionTargetPublishIsolated",
"reason": "需显式一次性本机PG(两份 0.9.0 真 schema 库)/ES/对象存储:真实导出→封存→ProductionLibraryTarget 写入 134 目标→逐表核对→写前镜像回滚,验证 0.9.0 无 wiki_log_entries、chunks 新增 NOT NULL 列走默认值、seq_id 由目标序列分配;普通CI没有这些服务,跳过不是通过"
}
]
}