Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5f1a8f565c | |||
| 4fce49478f | |||
| 4d2cb13b0e | |||
| 0aa9f273e6 | |||
| fba2713b4e | |||
| 10568943af | |||
| 3f2626f621 |
@@ -1,5 +1,101 @@
|
||||
# Changelog
|
||||
|
||||
## 开发快照 0.1.380(采集绑定允许对外库未配置)
|
||||
|
||||
### Fixed
|
||||
|
||||
- `libraryctl` 读 `routing_kb_bindings` 不再要求两列都非空:`internal_kb_id` 必填,
|
||||
为空时报 `对内知识库未绑定(routing_kb_bindings.internal_kb_id 为空)`;`external_kb_id`
|
||||
允许为空(平台约定「对外库没有对应目标时留空」),绑定仍然有效,`status` 与
|
||||
`-scope internal` 照常可用。`-scope external` 在未配置时明确失败并报
|
||||
`对外知识库未配置(routing_kb_bindings.external_kb_id 为空)`,不回退到对内库。
|
||||
`preflight` / `capture` / `dead-letters` 三条命令都在连源库、开维护窗口之前
|
||||
定住本次 scope 的目标库,缺绑定不会再退化成 `knowledge base not found in source`。
|
||||
- 读取 SQL 对两列 `COALESCE`:0.9.0 迁移把旧绑定清成 NULL 后,裸拼接会让文件模式
|
||||
(P1c 的 governance DSN)在 `Scan` 到 string 时失败,报出的还是
|
||||
`read routing bindings failed`;NULL 与空串现在同等处理,容器模式与文件模式共用
|
||||
同一条查询与同一份解析语义。没有生效绑定行(空输出)归入「对内库未绑定」。
|
||||
|
||||
### Tests
|
||||
|
||||
- 新增 `cmd/libraryctl` 绑定解析用例:internal 为空拒绝并点名(含 NULL 形状的
|
||||
`|`、空输出、空串);external 为空解析成功且 external scope 被拒、不回退;
|
||||
两列都有时行为不变(含 psql 行尾换行与首尾空格)。另有一条不连库的查询形状
|
||||
断言,锁住两列都必须 `COALESCE`、且仍产出 `<internal>|<external>`。
|
||||
|
||||
## 开发快照 0.1.379(采集接入 RDS 型 0.9.0 栈)
|
||||
|
||||
### Added
|
||||
|
||||
- `libraryctl` / `libraryagent` 的数据库来源可选:`-source-pg` 为空时用 `-source-app`
|
||||
容器环境里的 `DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD` 直连托管实例(RDS),
|
||||
TLS 模式取 `LIBRARY_SOURCE_DB_SSLMODE`(默认 `require`,`disable` 仅在显式测试时允许);
|
||||
`-governance-pg` 为空时从 `LIBRARY_GOVERNANCE_DSN_FILE` 指向的 0600 文件读治理库 DSN。
|
||||
两个都配置时行为与原来完全一致(容器模式),0.7.1 栈不受影响。
|
||||
- 新增 `scripts/library-capture-v090.sh`:Compose 项目 `weknora-v090` 的整库采集入口,
|
||||
唯一写入者 `app`,其余服务逐个声明理由;两个数据库走外部模式(RDS),归档/日志根
|
||||
默认 `/var/lib/knowledge-governance/{library,freeze}` 并可用环境变量覆盖;
|
||||
`LIBRARYCTL_BIN` 未设置或不可执行时明确失败,不做"同目录二进制"回退。
|
||||
- `libraryctl status` 增印 `databases.source_db/governance_db` 模式(`container|external|file`),
|
||||
`libraryagent -describe` 输出同样两项;地址与凭据一律不出现。
|
||||
|
||||
### Tests
|
||||
|
||||
- DSN 来源三条路径(容器 / RDS / 治理库文件)各有用例;缺变量、缺文件、权限过宽、
|
||||
非 PostgreSQL 内容都要求失败关闭,且断言错误里不含口令;`sslmode=prefer` 这类会静默
|
||||
降级的取值直接拒绝。
|
||||
- v090 采集脚本进 CI:`bash -n` 语法检查 + 标志/服务声明断言,并拒绝 0.7.1 的
|
||||
容器名与项目名出现在新入口里。
|
||||
|
||||
## 开发快照 0.1.378(发布写入支持 WeKnora 0.9.0)
|
||||
|
||||
### Added
|
||||
|
||||
- 整库发布写入路径支持 WeKnora 0.9.0(schema 134):生产整库目标在装配时解析目标
|
||||
profile,表集合、清理顺序、编目与整实例指纹都按解析结果取数——0.9.0 没有
|
||||
`wiki_log_entries`,不再对目标查询或删除该表;0.7.1 少一张必需表仍然失败,不靠
|
||||
「表不存在就跳过」放行。
|
||||
- `chunks.seq_id` 语义保持:发布时由目标序列分配,回滚时按写前镜像原值重放;
|
||||
0.9.0 新增的 NOT NULL 列走目标默认值,不用来源值硬填。
|
||||
- 跨版本发布(74 快照 → 134 目标,或反之)明确拒绝,错误同时点名两个版本号。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 范围改写重建快照时会丢掉 `SchemaVersion`/`Profile`(归档重建、快照克隆两处):
|
||||
丢掉后下游按「未知版本 = 0.7.1」处理,0.9.0 的归档会被要求提供它根本没有的
|
||||
`wiki_log_entries`,跨版本闸门也形同不存在。现在快照复制一律带上 schema 契约。
|
||||
- 单文档发布包(`releasebundle`)仍只支持 0.7.1,但改为**明确拒绝并说明原因**:
|
||||
报错点名读到的版本与「0.9.0 应走整库发布路径」,不再与字段写错共用一句泛化错误。
|
||||
|
||||
### Tests
|
||||
|
||||
- 真实 0.9.0 schema + 索引 + 对象存储上的发布路径:导出 → 封存 → 真实
|
||||
`ProductionLibraryTarget` 写入干净 134 目标 → 逐表内容核对(Wiki 非空、资源指向
|
||||
目标桶/后端、`seq_id` 由目标序列分配)→ 用写前镜像恢复回滚并核对回到写前状态;
|
||||
另加 74→134 拒绝(含两个版本号)与「134 目标无 `wiki_log_entries` 不报错」的负例。
|
||||
|
||||
## 开发快照 0.1.376(WeKnora 0.9.0 整库发布 profile)
|
||||
|
||||
### Added
|
||||
|
||||
- 整库发布支持 WeKnora 0.9.0(schema 134):按版本号选中 profile,再用 information_schema
|
||||
探测必需表与必需列,缺项精确报到「表」或「表.列」;未知版本、脏库一律拒绝,不做
|
||||
「版本大于某值就放行」的推断。
|
||||
- 0.9.0 的索引契约改从部署配置读取(该版本知识库不写 `vector_store_id`);两处来源都
|
||||
拿不到时报错,而不是用空串去匹配索引;0.7.1 若两处都有但互相矛盾,同样拒绝。
|
||||
- 归档部件集合随 profile 变化:0.9.0 已删除 `wiki_log_entries`,因此该部件不再要求;
|
||||
其余 Wiki 部件仍是必需,空 Wiki 不能冒充完整。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 采集、导入、Wiki 替换与「去使能」四条路径原先写死 schema 74,遇到 0.9.0 直接失败;
|
||||
现在同版本(74↔74、134↔134)往返可用,跨版本搬运会明确拒绝而不是误放行。
|
||||
|
||||
### Tests
|
||||
|
||||
- 真实 0.9.0 schema 与真实索引映射上跑完整往返:导出 → 封存(真实清单校验)→ 导入
|
||||
干净库 → 再导出逐行内容一致;另加未知版本、脏库、缺表、缺列、索引名解析不到等负例。
|
||||
|
||||
## 开发快照 0.1.375(工作台界面:越疆配色、页头统一与筛选批量条)
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/libraryagent"
|
||||
"knowledge-governance/internal/librarydsn"
|
||||
"knowledge-governance/internal/librarymaintenance"
|
||||
)
|
||||
|
||||
@@ -25,10 +26,10 @@ func main() {
|
||||
flag.StringVar(&config.ArchiveRoot, "archive", filepath.Join(home, ".local/share/knowledge-governance/library"), "sealed artifact root")
|
||||
flag.StringVar(&config.JournalRoot, "journal", filepath.Join(home, ".local/share/knowledge-governance/freeze"), "freeze journal root")
|
||||
flag.StringVar(&config.SourceApp, "source-app", "WKNFull-app", "source container describing the stack")
|
||||
flag.StringVar(&config.SourcePG, "source-pg", "WKNFull-postgres", "source PostgreSQL container")
|
||||
flag.StringVar(&config.SourcePG, "source-pg", "WKNFull-postgres", "source PostgreSQL container; empty = the source DB lives on a managed instance and is reached with the source app env (DB_HOST/...)")
|
||||
flag.StringVar(&config.SourceES, "source-es", "WKNFull-elasticsearch", "source Elasticsearch container")
|
||||
flag.StringVar(&config.SourceS3Endpoint, "source-s3-endpoint", "", "host-reachable source object-store endpoint (overrides the container-internal one)")
|
||||
flag.StringVar(&config.GovernancePG, "governance-pg", "knowledge-governance-p1-postgres-1", "governance PostgreSQL container")
|
||||
flag.StringVar(&config.GovernancePG, "governance-pg", "knowledge-governance-p1-postgres-1", "governance PostgreSQL container; empty = read the DSN from LIBRARY_GOVERNANCE_DSN_FILE")
|
||||
flag.StringVar(&config.Project, "project", "weknora071full", "Compose project owning the writers")
|
||||
flag.StringVar(&config.Writers, "writers", "app", "declared writer services")
|
||||
nonWriters := flag.String("non-writers", defaultNonWriters, "service=reason for every other project service")
|
||||
@@ -41,6 +42,13 @@ func main() {
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if *describe {
|
||||
// describe 只回答"这两个库从哪来"。地址与凭据一律不出现:这段输出会被贴进
|
||||
// 工单与日志,而 RDS 的连接串里带口令。
|
||||
log.Printf("source_db=%s governance_db=%s",
|
||||
librarydsn.SourceMode(config.SourcePG), librarydsn.GovernanceMode(config.GovernancePG))
|
||||
return
|
||||
}
|
||||
// 实例级维护接线(入口闸门 + 后台队列)不在 agent 的 Config 里:它由同一份
|
||||
// 环境声明提供给子进程。启动时先说清楚有没有——缺了封存会在开窗口前被拒,
|
||||
// 不允许"以为接好了"跑到一半才发现没有排空证据。
|
||||
@@ -49,10 +57,6 @@ func main() {
|
||||
} else {
|
||||
log.Printf("library agent: 实例级维护目标 %s 入口 %s 闸门 %s", target.Instance, target.IngressURL, target.IngressService)
|
||||
}
|
||||
if *describe {
|
||||
log.Printf("library agent config: %+v", agent.Describe())
|
||||
return
|
||||
}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
log.Printf("library agent listening on %s (project %s, writers %s)", config.Socket, config.Project, config.Writers)
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 文件模式没配置:必须失败关闭并点名缺的那个变量——"静默退回容器模式"会让
|
||||
// 操作员以为在读 RDS,实际在读本机容器(或者干脆读到不存在的容器)。
|
||||
func TestGovernanceFileModeFailsClosedWithoutDSNFile(t *testing.T) {
|
||||
t.Setenv("LIBRARY_GOVERNANCE_DSN_FILE", "")
|
||||
_, err := readRoutingBindings(context.Background(), options{governancePG: ""})
|
||||
if err == nil {
|
||||
t.Fatal("没有 DSN 文件却读成功了")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "LIBRARY_GOVERNANCE_DSN_FILE") {
|
||||
t.Fatalf("错误没有点名缺哪一项:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 外部 DSN 连不上/写错时,错误必须是固定说法:pgx 的错误里带的是连接串(主机、
|
||||
// 用户名,口令它自己打码),直接透传就等于把地址写进日志与工单。
|
||||
func TestGovernanceFileModeNeverLeaksPasswordInErrors(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "governance.dsn")
|
||||
// 夹具口令运行时拼装:源码里不留“带口令的连接串”字面量,否则密钥扫描只会
|
||||
// 报出一堆必然是假值的夹具,把真泄漏的信号淹掉。
|
||||
password := strings.Join([]string{"fixture", "leak", "probe"}, "-")
|
||||
// 故意用非法 sslmode + 不可达端口:无论失败在解析还是连接阶段,都不许泄露。
|
||||
if err := os.WriteFile(path, []byte("postgres://gov:"+password+"@127.0.0.1:1/gov?sslmode=bogus&connect_timeout=1\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("LIBRARY_GOVERNANCE_DSN_FILE", path)
|
||||
_, err := readRoutingBindings(context.Background(), options{governancePG: ""})
|
||||
if err == nil {
|
||||
t.Fatal("坏 DSN 竟然读成功了")
|
||||
}
|
||||
if strings.Contains(err.Error(), password) {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
}
|
||||
+165
-34
@@ -30,9 +30,11 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/librarydeadletter"
|
||||
"knowledge-governance/internal/librarydsn"
|
||||
"knowledge-governance/internal/libraryfreeze"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/librarymaintenance"
|
||||
@@ -146,9 +148,13 @@ func usage() {
|
||||
|
||||
Flags (defaults target the local WKNFull test stack):
|
||||
-source-app WKNFull-app container whose env describes the source
|
||||
-source-pg WKNFull-postgres source PostgreSQL container (address resolution)
|
||||
-source-pg WKNFull-postgres source PostgreSQL container; empty = the source DB is a
|
||||
managed instance: connect with the source app env
|
||||
(DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD), TLS mode from
|
||||
LIBRARY_SOURCE_DB_SSLMODE (default require)
|
||||
-source-es WKNFull-elasticsearch
|
||||
-governance-pg knowledge-governance-p1-postgres-1 routing bindings read here
|
||||
-governance-pg knowledge-governance-p1-postgres-1 routing bindings read here; empty = read
|
||||
the DSN from LIBRARY_GOVERNANCE_DSN_FILE (0600 file)
|
||||
-project weknora071full Compose project owning the writer group
|
||||
-writers WKNFull-app declared writer services (comma separated)
|
||||
-non-writers WKNFull-frontend=reason,... every other project service
|
||||
@@ -182,9 +188,9 @@ func parseFlags(command string, args []string) (options, []string, error) {
|
||||
set := flag.NewFlagSet(command, flag.ContinueOnError)
|
||||
set.SetOutput(os.Stderr)
|
||||
set.StringVar(&opts.sourceApp, "source-app", opts.sourceApp, "source container describing the stack")
|
||||
set.StringVar(&opts.sourcePG, "source-pg", opts.sourcePG, "source PostgreSQL container")
|
||||
set.StringVar(&opts.sourcePG, "source-pg", opts.sourcePG, "source PostgreSQL container; empty = connect to the source app's external (managed) database")
|
||||
set.StringVar(&opts.sourceES, "source-es", opts.sourceES, "source Elasticsearch container")
|
||||
set.StringVar(&opts.governancePG, "governance-pg", opts.governancePG, "governance PostgreSQL container")
|
||||
set.StringVar(&opts.governancePG, "governance-pg", opts.governancePG, "governance PostgreSQL container; empty = read the DSN from LIBRARY_GOVERNANCE_DSN_FILE")
|
||||
set.StringVar(&opts.project, "project", opts.project, "Compose project owning the writers")
|
||||
set.StringVar(&opts.writers, "writers", opts.writers, "declared writer services")
|
||||
set.StringVar(&opts.nonWriters, "non-writers", "", "service=reason for every other project service")
|
||||
@@ -285,15 +291,25 @@ func discoverSource(ctx context.Context, opts options, binding map[string]string
|
||||
if app["DB_DRIVER"] != "postgres" {
|
||||
return config, errors.New("source database driver is not postgres")
|
||||
}
|
||||
address, err := containerAddress(ctx, opts.sourcePG)
|
||||
if err != nil {
|
||||
return config, err
|
||||
// 两种来源:给了 -source-pg 就按容器地址拼(0.7.1 栈);没给说明库是托管
|
||||
// 实例(RDS 型 0.9.0 栈没有源库容器),用源应用容器 env 里的 DB_* 直连。
|
||||
if librarydsn.SourceMode(opts.sourcePG) == librarydsn.SourceExternal {
|
||||
dsn, dsnErr := librarydsn.SourceURL(app, os.Getenv)
|
||||
if dsnErr != nil {
|
||||
return config, dsnErr
|
||||
}
|
||||
config.DatabaseURL = dsn
|
||||
} else {
|
||||
address, addressErr := containerAddress(ctx, opts.sourcePG)
|
||||
if addressErr != nil {
|
||||
return config, addressErr
|
||||
}
|
||||
dsn, dsnErr := librarydsn.ContainerURL(address, app)
|
||||
if dsnErr != nil {
|
||||
return config, dsnErr
|
||||
}
|
||||
config.DatabaseURL = dsn
|
||||
}
|
||||
dsn := strings.Join([]string{
|
||||
"postgres://" + urlEscape(app["DB_USER"]) + ":" + urlEscape(app["DB_PASSWORD"]) +
|
||||
"@" + address + ":" + app["DB_PORT"] + "/" + urlEscape(app["DB_NAME"]) + "?sslmode=disable",
|
||||
}, "")
|
||||
config.DatabaseURL = dsn
|
||||
config.AppImageID, err = containerImageID(ctx, opts.sourceApp)
|
||||
if err != nil {
|
||||
return config, err
|
||||
@@ -388,17 +404,12 @@ func parseBool(value string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func urlEscape(value string) string {
|
||||
replacer := strings.NewReplacer("%", "%25", ":", "%3A", "@", "%40", "/", "%2F", "?", "%3F", "#", "%23", "&", "%26", " ", "%20")
|
||||
return replacer.Replace(value)
|
||||
}
|
||||
|
||||
// scopeBindings 返回 scope→知识库 映射。治理 API 用的是 LIBRARY_PRODUCTION_KB_*,
|
||||
// CLI 读的是治理库里的 routing bindings;同一台机器上两份不一致时,
|
||||
// CLI 抓到的会是"配置里那个库"而不是 API 认为的目标——所以环境变量优先,
|
||||
// 读不到才退回 bindings。两边讲同一份身份,验收才可比。
|
||||
func scopeBindings(ctx context.Context, container string) (map[string]string, error) {
|
||||
binding, err := readRoutingBindings(ctx, container)
|
||||
func scopeBindings(ctx context.Context, opts options) (map[string]string, error) {
|
||||
binding, err := readRoutingBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -410,17 +421,105 @@ func scopeBindings(ctx context.Context, container string) (map[string]string, er
|
||||
return binding, nil
|
||||
}
|
||||
|
||||
func readRoutingBindings(ctx context.Context, container string) (map[string]string, error) {
|
||||
raw, err := dockerOutput(ctx, "exec", container, "psql", "-U", "governance", "-d", "governance", "-tAc",
|
||||
"SELECT internal_kb_id||'|'||external_kb_id FROM routing_kb_bindings WHERE id='active'")
|
||||
// readRoutingBindings 读治理库里生效的 scope 绑定。两种来源:
|
||||
// - 给了 -governance-pg:docker exec 进治理库容器跑 psql(0.7.1 本机栈);
|
||||
// - 没给:治理库是托管实例(RDS 型 0.9.0 栈没有 governance 容器),DSN 来自
|
||||
// LIBRARY_GOVERNANCE_DSN_FILE 指向的 0600 文件。
|
||||
//
|
||||
// 两条路径只共用行解析,不共用连接方式:容器里没有网络到 RDS 的凭据,宿主机也不该
|
||||
// 为了读两列去 docker exec 一个不存在的容器。
|
||||
func readRoutingBindings(ctx context.Context, opts options) (map[string]string, error) {
|
||||
if librarydsn.GovernanceMode(opts.governancePG) == librarydsn.GovernanceFile {
|
||||
dsn, err := librarydsn.GovernanceURL(os.Getenv)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return readRoutingBindingsFromDSN(ctx, dsn)
|
||||
}
|
||||
raw, err := dockerOutput(ctx, "exec", opts.governancePG, "psql", "-U", "governance", "-d", "governance", "-tAc",
|
||||
routingBindingsQuery)
|
||||
if err != nil {
|
||||
return nil, errors.New("read routing bindings failed")
|
||||
}
|
||||
internal, external, ok := strings.Cut(strings.TrimSpace(raw), "|")
|
||||
if !ok || internal == "" || external == "" {
|
||||
return nil, errors.New("routing bindings are incomplete")
|
||||
return parseRoutingBindings(raw)
|
||||
}
|
||||
|
||||
// COALESCE 不能省:两列都可能是 NULL(0.9.0 迁移把\"旧知识库绑定已解除\"落成 NULL,
|
||||
// 平台约定\"对外库没有对应目标时留空\")。裸拼接遇 NULL 得到整行 NULL,文件模式
|
||||
// 会在 Scan 成 string 时直接失败,报出的还是一句\"read routing bindings failed\"——
|
||||
// 真正的原因(哪一列没绑)被吞掉。这里先拼成固定形状,由 parseRoutingBindings 判。
|
||||
const routingBindingsQuery = "SELECT COALESCE(internal_kb_id,'')||'|'||COALESCE(external_kb_id,'') FROM routing_kb_bindings WHERE id='active'"
|
||||
|
||||
// readRoutingBindingsFromDSN 用外部 DSN 读同一份绑定。
|
||||
//
|
||||
// 只读会话是刻意的:这条路径只是读两列,出了问题必须是"读不到",而不是在治理库上
|
||||
// 留下一次写入。错误一律收敛成固定说法:pgx 的错误里会带连接串(口令它自己打了码,
|
||||
// 但主机与用户名照旧出去),而这里连那份地址都不该出现在日志或工单里。
|
||||
func readRoutingBindingsFromDSN(ctx context.Context, dsn string) (map[string]string, error) {
|
||||
parsed, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
return nil, errors.New("governance database url is invalid")
|
||||
}
|
||||
return map[string]string{"internal": internal, "external": external}, nil
|
||||
parsed.ConnConfig.RuntimeParams["default_transaction_read_only"] = "on"
|
||||
parsed.ConnConfig.RuntimeParams["statement_timeout"] = "30000"
|
||||
parsed.MaxConns = 2
|
||||
pool, err := pgxpool.NewWithConfig(ctx, parsed)
|
||||
if err != nil {
|
||||
return nil, errors.New("open governance database failed")
|
||||
}
|
||||
defer pool.Close()
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, routingBindingsQuery).Scan(&raw); err != nil {
|
||||
return nil, errors.New("read routing bindings failed")
|
||||
}
|
||||
return parseRoutingBindings(raw)
|
||||
}
|
||||
|
||||
// parseRoutingBindings 解一行 `<internal>|<external>`。两条来源(psql -tAc 与 pgx
|
||||
// 只读会话)拼的是同一句 SQL,所以只有这一份解析语义。
|
||||
//
|
||||
// 对内库必填:没有对内库就没有\"整库\"可言,此时必须点名是哪一列没绑,而不是退化成
|
||||
// 泛化的读失败。对外库可空:平台侧约定\"对外库没有对应目标时留空\",绑定本身仍然
|
||||
// 有效(`status`、internal scope 照常可跑);只有真去用 external scope 时才拒绝,
|
||||
// 且绝不回退到对内库——那会把对内库的内容当成对外库封存出去。
|
||||
func parseRoutingBindings(raw string) (map[string]string, error) {
|
||||
trimmed := strings.TrimSpace(raw)
|
||||
// 空输出 = 没有生效绑定行,等于对内库没绑:这跟"读失败"是两件事,
|
||||
// 必须说成前者,否则现场又会退回一句泛化错误。
|
||||
if trimmed == "" {
|
||||
return nil, errInternalKBMissing
|
||||
}
|
||||
internal, external, ok := strings.Cut(trimmed, "|")
|
||||
if !ok {
|
||||
return nil, errors.New("read routing bindings failed")
|
||||
}
|
||||
internal = strings.TrimSpace(internal)
|
||||
external = strings.TrimSpace(external)
|
||||
if internal == "" {
|
||||
return nil, errInternalKBMissing
|
||||
}
|
||||
binding := map[string]string{"internal": internal}
|
||||
if external != "" {
|
||||
binding["external"] = external
|
||||
}
|
||||
return binding, nil
|
||||
}
|
||||
|
||||
var (
|
||||
errInternalKBMissing = errors.New("对内知识库未绑定(routing_kb_bindings.internal_kb_id 为空)")
|
||||
errExternalKBMissing = errors.New("对外知识库未配置(routing_kb_bindings.external_kb_id 为空)")
|
||||
)
|
||||
|
||||
// scopeKnowledgeBase 取本次 scope 的目标库。空值必须在这里就拒绝:往下走会变成
|
||||
// \"knowledge base not found in source\"——那是拿空 UUID 查源库得到的假原因。
|
||||
func scopeKnowledgeBase(binding map[string]string, scope string) (string, error) {
|
||||
if kb := strings.TrimSpace(binding[scope]); kb != "" {
|
||||
return kb, nil
|
||||
}
|
||||
if scope == "external" {
|
||||
return "", errExternalKBMissing
|
||||
}
|
||||
return "", errInternalKBMissing
|
||||
}
|
||||
|
||||
func openSourceDatabase(ctx context.Context, config sourceConfig) (*pgxpool.Pool, error) {
|
||||
@@ -432,11 +531,25 @@ func openSourceDatabase(ctx context.Context, config sourceConfig) (*pgxpool.Pool
|
||||
// that a database-enforced property rather than a convention.
|
||||
parsed.ConnConfig.RuntimeParams["default_transaction_read_only"] = "on"
|
||||
parsed.ConnConfig.RuntimeParams["statement_timeout"] = "120000"
|
||||
// 启动参数不足以保证只读:角色级/库级设置会在启动参数之后生效,托管实例
|
||||
// (RDS)上实测就是这样——SHOW 回来是 off。而池里每条连接是独立的会话,
|
||||
// 对单条连接 SET 管不到其他连接,所以放在 AfterConnect:每条新连接都显式
|
||||
// 置只读,池里就没有能写的会话。下面的 SHOW 只是对任意一条连接再核一次。
|
||||
parsed.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error {
|
||||
_, err := conn.Exec(ctx, "SET SESSION default_transaction_read_only = on")
|
||||
return err
|
||||
}
|
||||
pool, err := pgxpool.NewWithConfig(ctx, parsed)
|
||||
if err != nil {
|
||||
return nil, errors.New("open source database failed")
|
||||
}
|
||||
var readOnly string
|
||||
// 连接失败与"能连上但会话可写"必须分开报:把两者压成同一句,现场只能看到
|
||||
// 一句无信息量的失败,反而会去怀疑权限或只读设置。
|
||||
if err = pool.Ping(ctx); err != nil {
|
||||
pool.Close()
|
||||
return nil, errors.New("source database connection failed")
|
||||
}
|
||||
if err = pool.QueryRow(ctx, "SHOW default_transaction_read_only").Scan(&readOnly); err != nil || readOnly != "on" {
|
||||
pool.Close()
|
||||
return nil, errors.New("source session is not read-only")
|
||||
@@ -587,7 +700,12 @@ func openArchive(opts options) (*librarysnapshot.Archive, error) {
|
||||
|
||||
func runStatus(ctx context.Context, opts options) error {
|
||||
report := map[string]any{}
|
||||
binding, bindingErr := scopeBindings(ctx, opts.governancePG)
|
||||
// 只报来源模式:地址与凭据连 describe/status 都不出现,日志自然也不会带出去。
|
||||
report["databases"] = map[string]string{
|
||||
"source_db": librarydsn.SourceMode(opts.sourcePG),
|
||||
"governance_db": librarydsn.GovernanceMode(opts.governancePG),
|
||||
}
|
||||
binding, bindingErr := scopeBindings(ctx, opts)
|
||||
var config sourceConfig
|
||||
var err error
|
||||
if bindingErr != nil {
|
||||
@@ -680,7 +798,12 @@ func runPreflight(ctx context.Context, opts options) error {
|
||||
if opts.scope != "internal" && opts.scope != "external" {
|
||||
return errors.New("-scope must be internal or external")
|
||||
}
|
||||
binding, err := scopeBindings(ctx, opts.governancePG)
|
||||
binding, err := scopeBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// 绑定缺哪一半在连源库之前就要说清楚:external 未配置不是"源库里没这个库"。
|
||||
kb, err := scopeKnowledgeBase(binding, opts.scope)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -693,7 +816,6 @@ func runPreflight(ctx context.Context, opts options) error {
|
||||
return err
|
||||
}
|
||||
defer pool.Close()
|
||||
kb := config.Scopes[opts.scope]
|
||||
tenant, err := tenantAndKnowledgeBase(ctx, pool, kb)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -773,7 +895,13 @@ func runCapture(ctx context.Context, opts options) error {
|
||||
return err
|
||||
}
|
||||
defer archive.Close()
|
||||
binding, err := scopeBindings(ctx, opts.governancePG)
|
||||
binding, err := scopeBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// 先定住本次 scope 的库:external 没配置时,封存必须在开维护窗口之前就停,
|
||||
// 不能先停写者再发现目标为空。
|
||||
kb, err := scopeKnowledgeBase(binding, opts.scope)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -801,7 +929,7 @@ func runCapture(ctx context.Context, opts options) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
scopedKB := binding[opts.scope]
|
||||
scopedKB := kb
|
||||
if found {
|
||||
manifest, err := replayCapture(job, opts.scope, scopedKB, existing, archive)
|
||||
if err != nil {
|
||||
@@ -814,7 +942,6 @@ func runCapture(ctx context.Context, opts options) error {
|
||||
return err
|
||||
}
|
||||
defer pool.Close()
|
||||
kb := config.Scopes[opts.scope]
|
||||
tenant, err := tenantAndKnowledgeBase(ctx, pool, kb)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -1150,7 +1277,12 @@ func runDeadLetters(ctx context.Context, opts options) error {
|
||||
if opts.scope != "internal" && opts.scope != "external" {
|
||||
return errors.New("-scope must be internal or external")
|
||||
}
|
||||
binding, err := scopeBindings(ctx, opts.governancePG)
|
||||
binding, err := scopeBindings(ctx, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// external 未配置时,死信观测同样要明说"没配置",而不是拿空库去查源库。
|
||||
kb, err := scopeKnowledgeBase(binding, opts.scope)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -1163,7 +1295,6 @@ func runDeadLetters(ctx context.Context, opts options) error {
|
||||
return err
|
||||
}
|
||||
defer pool.Close()
|
||||
kb := config.Scopes[opts.scope]
|
||||
tenant, err := tenantAndKnowledgeBase(ctx, pool, kb)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// internal 为空必须拒绝并点名。实测到的现场就是这一行:0.9.0 迁移后
|
||||
// internal_kb_id 与 external_kb_id 都是 NULL,旧解析退化成
|
||||
// "read routing bindings failed",操作员无法判断是连不上还是没绑定。
|
||||
// NULL 与空串在 SQL 侧由 routingBindingsQuery 的 COALESCE 收敛成同一形状,
|
||||
// 解析侧两者都必须落到同一条错误上。
|
||||
func TestParseRoutingBindingsRejectsEmptyInternal(t *testing.T) {
|
||||
for _, raw := range []string{"|", "|ext-kb", " \n| \n", "\n|\n", ""} {
|
||||
binding, err := parseRoutingBindings(raw)
|
||||
if err == nil {
|
||||
t.Fatalf("internal 为空却解析成功:%q → %v", raw, binding)
|
||||
}
|
||||
if binding != nil {
|
||||
t.Fatalf("internal 为空仍返回了映射:%q → %v", raw, binding)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "对内知识库未绑定") || !strings.Contains(err.Error(), "internal_kb_id") {
|
||||
t.Fatalf("错误没有点名 internal 未绑定:%q → %v", raw, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 0.9.0 测试实例的真实形状:只有对内库(wiki-demo),平台约定对外库没有对应
|
||||
// 目标时留空。绑定仍然有效,internal scope 能解析;external scope 必须失败、
|
||||
// 点名 external_kb_id,并且绝不回退到对内库。
|
||||
func TestParseRoutingBindingsAllowsEmptyExternal(t *testing.T) {
|
||||
binding, err := parseRoutingBindings("0da9b571-6f13-46a1-85fa-e63d2ef5666e|")
|
||||
if err != nil {
|
||||
t.Fatalf("对外库为空不该让整份绑定失效:%v", err)
|
||||
}
|
||||
if got := binding["internal"]; got != "0da9b571-6f13-46a1-85fa-e63d2ef5666e" {
|
||||
t.Fatalf("对内库解析错误:%q", got)
|
||||
}
|
||||
if value, present := binding["external"]; present {
|
||||
t.Fatalf("对外库没配置却写进了映射:%q", value)
|
||||
}
|
||||
|
||||
kb, err := scopeKnowledgeBase(binding, "internal")
|
||||
if err != nil || kb != "0da9b571-6f13-46a1-85fa-e63d2ef5666e" {
|
||||
t.Fatalf("internal scope 不该被 external 为空牵连:kb=%q err=%v", kb, err)
|
||||
}
|
||||
|
||||
kb, err = scopeKnowledgeBase(binding, "external")
|
||||
if err == nil {
|
||||
t.Fatalf("对外库未配置却解析出了目标库:%q", kb)
|
||||
}
|
||||
if kb != "" {
|
||||
t.Fatalf("external scope 回退到了别的库:%q", kb)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "对外知识库未配置") || !strings.Contains(err.Error(), "external_kb_id") {
|
||||
t.Fatalf("错误没有点名 external 未配置:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 两个库都绑定时行为不变:这是 0.7.1 栈与对外库已配置的 0.9.0 栈的常态路径。
|
||||
func TestParseRoutingBindingsBothScopesUnchanged(t *testing.T) {
|
||||
// psql -tAc 输出带行尾换行,pgx 那边不带——两边都必须是同一个结果。
|
||||
for _, raw := range []string{"kb-internal|kb-external", "kb-internal|kb-external\n", " kb-internal | kb-external \n"} {
|
||||
binding, err := parseRoutingBindings(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("两个库都绑定却解析失败:%q → %v", raw, err)
|
||||
}
|
||||
if len(binding) != 2 || binding["internal"] != "kb-internal" || binding["external"] != "kb-external" {
|
||||
t.Fatalf("解析结果与原行为不一致:%q → %v", raw, binding)
|
||||
}
|
||||
for _, scope := range []string{"internal", "external"} {
|
||||
if _, err := scopeKnowledgeBase(binding, scope); err != nil {
|
||||
t.Fatalf("%s scope 不该失败:%v", scope, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 没有分隔符说明拿到的不是那一行(空结果集、psql 报错文本等),只能是读失败。
|
||||
func TestParseRoutingBindingsRejectsMalformedRow(t *testing.T) {
|
||||
for _, raw := range []string{"kb-without-separator", "ERROR: relation \"routing_kb_bindings\" does not exist"} {
|
||||
if _, err := parseRoutingBindings(raw); err == nil {
|
||||
t.Fatalf("没有分隔符却解析成功:%q", raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 不连库的语法/形状校验:裸拼接遇 NULL 会得到整行 NULL,文件模式在 Scan 成
|
||||
// string 时就会失败并丢掉真实原因。两句来源共用这一条 SQL,因此只查这一处。
|
||||
func TestRoutingBindingsQueryCoalescesBothColumns(t *testing.T) {
|
||||
query := routingBindingsQuery
|
||||
for _, column := range []string{"internal_kb_id", "external_kb_id"} {
|
||||
if !strings.Contains(query, "COALESCE("+column+",'')") {
|
||||
t.Fatalf("查询没有把 %s 的 NULL 收敛成空串:%s", column, query)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(query, "||'|'||") {
|
||||
t.Fatalf("查询不再产出 '<internal>|<external>' 形状:%s", query)
|
||||
}
|
||||
if !strings.Contains(query, "WHERE id='active'") {
|
||||
t.Fatalf("查询没有锁定生效绑定行:%s", query)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// v090 采集脚本是本轮新增的唯一入口。为什么用 bash -n 而不是"跑一遍":跑一遍要
|
||||
// 停写入者、连 RDS——那是真实采集,不是单元测试。语法与声明在这里钉住,能让
|
||||
// "标志写错一个字母"在 CI 里就红,而不是等到开维护窗口时才发现。
|
||||
func TestV090CaptureScriptIsSyntacticallyValid(t *testing.T) {
|
||||
bash, err := exec.LookPath("bash")
|
||||
if err != nil {
|
||||
t.Fatalf("需要 bash 才能检查脚本语法(CI 与测试机都有):%v", err)
|
||||
}
|
||||
path := filepath.Join("..", "..", "scripts", "library-capture-v090.sh")
|
||||
out, err := exec.Command(bash, "-n", path).CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("bash -n %s 失败:%v\n%s", path, err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// 脚本内容断言:外部库模式(两个 -pg 传空)、写者声明、容器名、归档根、
|
||||
// LIBRARYCTL_BIN 缺失即失败——这些一旦漂移,真跑时会封错栈或读错库。
|
||||
func TestV090CaptureScriptDeclaresExternalDatabases(t *testing.T) {
|
||||
body, err := os.ReadFile(filepath.Join("..", "..", "scripts", "library-capture-v090.sh"))
|
||||
if err != nil {
|
||||
t.Fatalf("读脚本:%v", err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"set -euo pipefail",
|
||||
"LIBRARYCTL_MISSING",
|
||||
"-project weknora-v090",
|
||||
"-writers app",
|
||||
"-source-app weknora-v090-app",
|
||||
"-source-es weknora-v090-elasticsearch",
|
||||
`-source-pg ""`,
|
||||
`-governance-pg ""`,
|
||||
"frontend=UI only; no knowledge writes",
|
||||
"docreader=parse RPC service; app persists parse output",
|
||||
"elasticsearch=search store; not an initiator",
|
||||
"redis=queue/cache store; not an initiator",
|
||||
"/var/lib/knowledge-governance/library",
|
||||
"/var/lib/knowledge-governance/freeze",
|
||||
} {
|
||||
if !strings.Contains(string(body), want) {
|
||||
t.Fatalf("脚本缺少 %q", want)
|
||||
}
|
||||
}
|
||||
// 0.7.1 的服务名绝不能出现在 v090 入口里:那会让封存停错栈。
|
||||
for _, forbidden := range []string{"WKNFull", "weknora071full"} {
|
||||
if strings.Contains(string(body), forbidden) {
|
||||
t.Fatalf("v090 脚本里出现了 0.7.1 的 %q", forbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -26,6 +26,8 @@ import (
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarydsn"
|
||||
)
|
||||
|
||||
// Config is fixed at start-up by the operator (systemd unit / CLI flags). It is
|
||||
@@ -429,15 +431,22 @@ func (a *Agent) health() response {
|
||||
}
|
||||
|
||||
// identity 被日志与健康应答共用一份,避免两处描述漂移。
|
||||
//
|
||||
// 数据库只报来源模式(container|external|file),不报地址:agent 的应答会被治理侧
|
||||
// 存档,DSN 里带口令,一旦写进身份就再也收不回来了。
|
||||
func (c Config) identity() map[string]any {
|
||||
identity := map[string]any{
|
||||
"project": c.Project,
|
||||
"archive_root": c.ArchiveRoot,
|
||||
"journal_root": c.JournalRoot,
|
||||
"source_app": c.SourceApp,
|
||||
"source_pg": c.SourcePG,
|
||||
"source_es": c.SourceES,
|
||||
"writers": c.Writers,
|
||||
// source_pg 仍是容器名(不是地址、不是凭据),保留它以让既有健康页/事后核对
|
||||
// 读得到同一份字段;数据库**来源**由下面两个模式字段表达,外部模式下它为空。
|
||||
"source_pg": c.SourcePG,
|
||||
"source_db": librarydsn.SourceMode(c.SourcePG),
|
||||
"governance_db": librarydsn.GovernanceMode(c.GovernancePG),
|
||||
"writers": c.Writers,
|
||||
}
|
||||
if c.SourceS3Endpoint != "" {
|
||||
identity["source_s3_endpoint"] = c.SourceS3Endpoint
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
// Package librarydsn 决定整库采集链路里两个 DSN 究竟从哪来。
|
||||
//
|
||||
// 为什么要有它:容器模式下 DSN 是"源库容器的地址 + 源应用容器的 env"拼出来的;
|
||||
// RDS 型 0.9.0 栈没有源库容器(库是托管实例),治理库同样只在外部。旧拼法在那种
|
||||
// 栈上不是权限不够,而是地址解析方式错了——库根本没有容器地址可查。把"用哪种
|
||||
// 来源"收敛成一处,libraryctl 与 libraryagent 共用同一份判定,日志/describe 只报
|
||||
// 模式(container|external|file),永远不报地址与凭据。
|
||||
package librarydsn
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
)
|
||||
|
||||
const (
|
||||
// SourceContainer:-source-pg 给了容器名,地址从容器网络解析(0.7.1 栈)。
|
||||
SourceContainer = "container"
|
||||
// SourceExternal:-source-pg 为空,直连源应用容器 env 里声明的托管库(RDS)。
|
||||
SourceExternal = "external"
|
||||
|
||||
// GovernanceContainer:-governance-pg 给了容器名,用 docker exec psql 读。
|
||||
GovernanceContainer = "container"
|
||||
// GovernanceFile:-governance-pg 为空,DSN 来自 0600 文件(LIBRARY_GOVERNANCE_DSN_FILE)。
|
||||
GovernanceFile = "file"
|
||||
)
|
||||
|
||||
// allowedSSLModes 是允许的 sslmode 集合。
|
||||
//
|
||||
// 只收这四个:prefer/allow 会在服务端不支持时**静默降级成明文**,托管实例上那等于
|
||||
// "以为加密了其实没有"——这种"看起来配好了"的失败模式比直接报错更难发现。
|
||||
var allowedSSLModes = map[string]bool{
|
||||
"disable": true, // 仅显式测试:本机回环或明确无 TLS 的隔离库
|
||||
"require": true,
|
||||
"verify-ca": true,
|
||||
"verify-full": true,
|
||||
}
|
||||
|
||||
// SourceMode 把 -source-pg 的取值翻译成来源模式。空值 = 外部(RDS)。
|
||||
func SourceMode(sourcePG string) string {
|
||||
if strings.TrimSpace(sourcePG) == "" {
|
||||
return SourceExternal
|
||||
}
|
||||
return SourceContainer
|
||||
}
|
||||
|
||||
// GovernanceMode 把 -governance-pg 的取值翻译成来源模式。空值 = DSN 文件。
|
||||
func GovernanceMode(governancePG string) string {
|
||||
if strings.TrimSpace(governancePG) == "" {
|
||||
return GovernanceFile
|
||||
}
|
||||
return GovernanceContainer
|
||||
}
|
||||
|
||||
// ContainerURL 复现容器模式原有的 DSN 形状:地址来自容器网络,sslmode=disable
|
||||
// (容器私网内没有 TLS 终结)。抽到这里只为一件事:让"模式选择"有一条可测的路径,
|
||||
// 而形状与旧实现逐字节一致(真跑过的栈不能因为重构换了连接串)。
|
||||
func ContainerURL(address string, app map[string]string) (string, error) {
|
||||
if strings.TrimSpace(address) == "" {
|
||||
return "", errors.New("source container address is empty")
|
||||
}
|
||||
for _, key := range []string{"DB_USER", "DB_PASSWORD", "DB_NAME"} {
|
||||
if app[key] == "" {
|
||||
return "", fmt.Errorf("source app environment is missing %s", key)
|
||||
}
|
||||
}
|
||||
target := url.URL{
|
||||
Scheme: "postgres",
|
||||
User: url.UserPassword(app["DB_USER"], app["DB_PASSWORD"]),
|
||||
Host: net.JoinHostPort(address, defaultPort(app["DB_PORT"])),
|
||||
Path: "/" + app["DB_NAME"],
|
||||
RawQuery: url.Values{"sslmode": {"disable"}}.Encode(),
|
||||
}
|
||||
return target.String(), nil
|
||||
}
|
||||
|
||||
// SourceURL 用源应用容器 env 里的 DB_* 直连托管库(RDS)。
|
||||
//
|
||||
// 缺哪个变量就点哪个变量名:把"连接被拒"与"配置缺一项"分开,才不用去猜着改。
|
||||
// 错误里只有变量名——DB_PASSWORD 的值绝不能出现在任何一处输出里。
|
||||
func SourceURL(app map[string]string, getenv func(string) string) (string, error) {
|
||||
missing := []string{}
|
||||
for _, key := range []string{"DB_HOST", "DB_USER", "DB_NAME"} {
|
||||
if strings.TrimSpace(app[key]) == "" {
|
||||
missing = append(missing, key)
|
||||
}
|
||||
}
|
||||
// 口令不 TrimSpace:前后空格也是口令的一部分,改掉就成另一个口令了。
|
||||
if app["DB_PASSWORD"] == "" {
|
||||
missing = append(missing, "DB_PASSWORD")
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return "", fmt.Errorf("source app environment is missing %s", strings.Join(missing, ", "))
|
||||
}
|
||||
sslmode := strings.TrimSpace(getenv("LIBRARY_SOURCE_DB_SSLMODE"))
|
||||
if sslmode == "" {
|
||||
sslmode = "require"
|
||||
}
|
||||
if !allowedSSLModes[sslmode] {
|
||||
return "", errors.New("LIBRARY_SOURCE_DB_SSLMODE must be one of disable, require, verify-ca, verify-full")
|
||||
}
|
||||
target := url.URL{
|
||||
Scheme: "postgres",
|
||||
User: url.UserPassword(app["DB_USER"], app["DB_PASSWORD"]),
|
||||
Host: net.JoinHostPort(strings.TrimSpace(app["DB_HOST"]), defaultPort(app["DB_PORT"])),
|
||||
Path: "/" + strings.TrimSpace(app["DB_NAME"]),
|
||||
RawQuery: url.Values{"sslmode": {sslmode}}.Encode(),
|
||||
}
|
||||
return target.String(), nil
|
||||
}
|
||||
|
||||
// GovernanceURL 从 0600 文件读治理库 DSN。
|
||||
//
|
||||
// 文件读法复用 dobotaiops.ReadDSNFile:那里已经把"必须是普通文件、必须 0600、
|
||||
// 错误只报路径与原因"钉住并测过——再写一份同义实现只会让两条路径的规则慢慢分叉。
|
||||
func GovernanceURL(getenv func(string) string) (string, error) {
|
||||
path := strings.TrimSpace(getenv("LIBRARY_GOVERNANCE_DSN_FILE"))
|
||||
if path == "" {
|
||||
return "", errors.New("LIBRARY_GOVERNANCE_DSN_FILE is not set: the governance DSN file is required when -governance-pg is empty")
|
||||
}
|
||||
dsn, err := dobotaiops.ReadDSNFile(path)
|
||||
if err != nil {
|
||||
// ReadDSNFile 的错误只含路径与原因,不含 DSN/口令。
|
||||
return "", fmt.Errorf("governance DSN file unusable: %w", err)
|
||||
}
|
||||
return dsn, nil
|
||||
}
|
||||
|
||||
// defaultPort:托管实例的连接信息里端口常常单独给;缺了就按 PostgreSQL 默认。
|
||||
// 端口不是凭据,给默认值不会把"配错了"藏起来。
|
||||
func defaultPort(port string) string {
|
||||
if trimmed := strings.TrimSpace(port); trimmed != "" {
|
||||
return trimmed
|
||||
}
|
||||
return "5432"
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package librarydsn
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func envFunc(pairs map[string]string) func(string) string {
|
||||
return func(key string) string { return pairs[key] }
|
||||
}
|
||||
|
||||
// 容器模式:旧拼法保持同一目标——同样的用户/口令/地址/库名,sslmode=disable。
|
||||
// 真跑过的 0.7.1 栈不能因为这次接入换了连接目标(换 sslmode 或换转义方式都会让
|
||||
// "同一个库"连不上或连成别的身份)。断言用解析后的分量,不用字面转义:
|
||||
// userinfo 里 ':' 保不保留不影响解析结果,钉字面只会把重构误判成回归。
|
||||
func TestContainerURLKeepsLegacyShape(t *testing.T) {
|
||||
app := map[string]string{"DB_USER": "weknora", "DB_PASSWORD": "p@ss:w/rd", "DB_NAME": "weknora",
|
||||
"DB_PORT": "5432"}
|
||||
dsn, err := ContainerURL("172.18.0.5", app)
|
||||
if err != nil {
|
||||
t.Fatalf("container url: %v", err)
|
||||
}
|
||||
parsed, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("生成的连接串自己都解析不了:%v", err)
|
||||
}
|
||||
if parsed.Scheme != "postgres" || parsed.Host != "172.18.0.5:5432" || parsed.Path != "/weknora" {
|
||||
t.Fatalf("容器模式连接目标变了:%s", dsn)
|
||||
}
|
||||
if parsed.User.Username() != "weknora" {
|
||||
t.Fatalf("用户名变了:%s", dsn)
|
||||
}
|
||||
if password, _ := parsed.User.Password(); password != "p@ss:w/rd" {
|
||||
t.Fatalf("口令转义后取不回来:%s", dsn)
|
||||
}
|
||||
if parsed.Query().Get("sslmode") != "disable" {
|
||||
t.Fatalf("容器模式不该改 sslmode:%s", dsn)
|
||||
}
|
||||
if _, err = ContainerURL("", app); err == nil {
|
||||
t.Fatal("没有容器地址却生成了连接串")
|
||||
}
|
||||
}
|
||||
|
||||
// RDS 模式:地址来自源应用 env(DB_HOST),不是容器网络;默认 sslmode=require,
|
||||
// 口令里的特殊字符必须转义后仍能被解析回来。
|
||||
func TestSourceURLUsesRDSHostAndRequiresTLSByDefault(t *testing.T) {
|
||||
app := map[string]string{
|
||||
"DB_HOST": "pgm-2ze0abcd.pg.rds.aliyuncs.com", "DB_PORT": "5432",
|
||||
"DB_NAME": "weknora_v090", "DB_USER": "v090_ro", "DB_PASSWORD": "s3cr@t/pw",
|
||||
}
|
||||
dsn, err := SourceURL(app, envFunc(nil))
|
||||
if err != nil {
|
||||
t.Fatalf("rds url: %v", err)
|
||||
}
|
||||
parsed, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("生成的连接串自己都解析不了:%v", err)
|
||||
}
|
||||
if parsed.Host != "pgm-2ze0abcd.pg.rds.aliyuncs.com:5432" {
|
||||
t.Fatalf("host 不是 RDS 地址:%q", parsed.Host)
|
||||
}
|
||||
if got := parsed.Query().Get("sslmode"); got != "require" {
|
||||
t.Fatalf("默认 sslmode 不是 require:%q", got)
|
||||
}
|
||||
if password, _ := parsed.User.Password(); password != "s3cr@t/pw" {
|
||||
t.Fatal("口令转义后取不回来")
|
||||
}
|
||||
// 显式 disable 只用于本机隔离测试;它是允许的,但必须由调用方写清楚。
|
||||
dsn, err = SourceURL(app, envFunc(map[string]string{"LIBRARY_SOURCE_DB_SSLMODE": "disable"}))
|
||||
if err != nil {
|
||||
t.Fatalf("explicit disable: %v", err)
|
||||
}
|
||||
if !strings.HasSuffix(dsn, "sslmode=disable") {
|
||||
t.Fatalf("显式 disable 没被采纳:%s", dsn)
|
||||
}
|
||||
// prefer 会静默降级成明文:那不是"配好了 TLS",直接拒。
|
||||
if _, err = SourceURL(app, envFunc(map[string]string{"LIBRARY_SOURCE_DB_SSLMODE": "prefer"})); err == nil {
|
||||
t.Fatal("sslmode=prefer 被接受了")
|
||||
}
|
||||
}
|
||||
|
||||
// 缺变量必须失败,且点名缺的那一项;错误里不许出现口令。
|
||||
func TestSourceURLMissingVariableFailsWithoutLeakingPassword(t *testing.T) {
|
||||
// 口令运行时拼装:源码里不留字面量口令,避免密钥扫描只报假夹具。
|
||||
password := strings.Join([]string{"fixture", "db", "secret"}, "-")
|
||||
app := map[string]string{"DB_HOST": "pgm-x.pg.rds.aliyuncs.com", "DB_NAME": "kb",
|
||||
"DB_USER": "u", "DB_PASSWORD": password}
|
||||
delete(app, "DB_HOST")
|
||||
_, err := SourceURL(app, envFunc(nil))
|
||||
if err == nil {
|
||||
t.Fatal("缺 DB_HOST 竟然成功了")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "DB_HOST") {
|
||||
t.Fatalf("错误没有点名缺哪一项:%v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), password) {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
app["DB_HOST"] = "pgm-x.pg.rds.aliyuncs.com"
|
||||
delete(app, "DB_PASSWORD")
|
||||
_, err = SourceURL(app, envFunc(nil))
|
||||
if err == nil || !strings.Contains(err.Error(), "DB_PASSWORD") {
|
||||
t.Fatalf("缺 DB_PASSWORD 没有点名:%v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), password) {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 治理库文件模式:0600 文件里的完整 DSN 原样读出;文件缺失/权限过宽/内容不是
|
||||
// PostgreSQL DSN 都必须失败关闭,且错误只提到路径与原因。
|
||||
func TestGovernanceURLReadsDSNFileAndFailsClosed(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "governance.dsn")
|
||||
password := strings.Join([]string{"fixture", "governance", "value"}, "-")
|
||||
dsn := "postgres://governance:" + password + "@pgm-gov.pg.rds.aliyuncs.com:5432/knowledge_governance_test?sslmode=require"
|
||||
if err := os.WriteFile(path, []byte(dsn+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": path}))
|
||||
if err != nil {
|
||||
t.Fatalf("file mode: %v", err)
|
||||
}
|
||||
if got != dsn {
|
||||
t.Fatalf("读出的 DSN 与文件内容不一致(不想在这里做任何改写)")
|
||||
}
|
||||
|
||||
// 没声明文件:失败并点名变量。
|
||||
if _, err = GovernanceURL(envFunc(nil)); err == nil || !strings.Contains(err.Error(), "LIBRARY_GOVERNANCE_DSN_FILE") {
|
||||
t.Fatalf("没配置文件时没有点名变量:%v", err)
|
||||
}
|
||||
|
||||
// 文件不存在:点名路径,不泄漏任何内容。
|
||||
missing := filepath.Join(dir, "absent.dsn")
|
||||
if _, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": missing})); err == nil ||
|
||||
!strings.Contains(err.Error(), missing) {
|
||||
t.Fatalf("文件缺失没有点名路径:%v", err)
|
||||
}
|
||||
|
||||
// 组/其他可读:拒绝(这条规则由 dobotaiops.ReadDSNFile 判定,这里只钉住接线)。
|
||||
wide := filepath.Join(dir, "wide.dsn")
|
||||
if err = os.WriteFile(wide, []byte(dsn+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = os.Chmod(wide, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": wide})); err == nil {
|
||||
t.Fatal("0644 的 DSN 文件被接受了")
|
||||
}
|
||||
|
||||
// 内容不是 PostgreSQL DSN:失败,且不要把内容回显进错误。
|
||||
bogus := filepath.Join(dir, "bogus.dsn")
|
||||
if err = os.WriteFile(bogus, []byte("mysql://u:"+password+"@host/db\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = GovernanceURL(envFunc(map[string]string{"LIBRARY_GOVERNANCE_DSN_FILE": bogus}))
|
||||
if err == nil {
|
||||
t.Fatal("非 PostgreSQL DSN 被接受了")
|
||||
}
|
||||
if strings.Contains(err.Error(), password) {
|
||||
t.Fatalf("错误里漏了口令:%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestModesDeriveFromEmptyFlag(t *testing.T) {
|
||||
if SourceMode("WKNFull-postgres") != SourceContainer || SourceMode(" ") != SourceExternal {
|
||||
t.Fatal("源库模式判定不对")
|
||||
}
|
||||
if GovernanceMode("knowledge-governance-p1-postgres-1") != GovernanceContainer || GovernanceMode("") != GovernanceFile {
|
||||
t.Fatal("治理库模式判定不对")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
// Package libraryprofile 声明"某个 WeKnora schema 版本下,整库适配器可以依赖什么"。
|
||||
//
|
||||
// 为什么要有这一层:整库导出/导入原本把 `version != 74` 写死在四个入口里。
|
||||
// 版本号本身不是契约——同一版本号在不同部署上也可能缺表;反过来,把判定放宽成
|
||||
// `version > 74` 又会把"没见过的版本"当成"兼容"。所以这里同时声明两件事:
|
||||
// - 该版本必须存在的表与列(导出/导入按它取数);
|
||||
// - 该版本的归档部件集合(manifest 按它判定完整性)。
|
||||
//
|
||||
// 版本号只用来**选中** profile,真正的放行条件是随后对 information_schema 的
|
||||
// 能力探测(见 releaseadapter 的 ResolveSchemaProfile)。
|
||||
package libraryprofile
|
||||
|
||||
import "fmt"
|
||||
|
||||
// LegacyVersion 是 0.7.1 的 schema_migrations 版本号,也是既有内存夹具的默认值。
|
||||
const LegacyVersion = 74
|
||||
|
||||
// V090Version 是 0.9.0 的 schema_migrations 版本号(已实测:134、dirty=false、83 表)。
|
||||
const V090Version = 134
|
||||
|
||||
// PartRequirement 是归档部件的一项要求(kind/name 与 manifest 的部件键一致)。
|
||||
type PartRequirement struct {
|
||||
Kind string
|
||||
Name string
|
||||
}
|
||||
|
||||
// Profile 是一个 schema 版本的静态能力声明。
|
||||
type Profile struct {
|
||||
Name string
|
||||
SchemaVersion int
|
||||
WeKnoraVersion string
|
||||
// EmbeddingDimension 为 0 表示"由目标 models.parameters 说了算";
|
||||
// 非 0 表示该版本只接受这一定维度(0.7.1 的契约就是固定 1024)。
|
||||
EmbeddingDimension int
|
||||
RequiredTables []string
|
||||
OptionalTables []string
|
||||
RequiredColumns map[string][]string
|
||||
// ReviewedTriggers 是该版本允许出现的 pg_get_triggerdef 原文。
|
||||
// 触发器意味着"写一行还会改别的表",所以只接受逐字核对过的定义,
|
||||
// 且这些定义必须在该版本上真实存在(消失也说明 schema 不是我们审过的那份)。
|
||||
ReviewedTriggers []string
|
||||
RequiredParts []PartRequirement
|
||||
OptionalParts []PartRequirement
|
||||
// IndexContractSource 说明该版本的索引名从哪里读。空值等于"读不到",
|
||||
// 不允许回退成空串去匹配任何索引。
|
||||
IndexContractSource string
|
||||
}
|
||||
|
||||
var profiles = map[int]Profile{
|
||||
LegacyVersion: {
|
||||
Name: "v071",
|
||||
SchemaVersion: LegacyVersion,
|
||||
WeKnoraVersion: "v0.7.1",
|
||||
EmbeddingDimension: 1024,
|
||||
RequiredTables: []string{
|
||||
"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
|
||||
"resources", "resource_bindings", "wiki_folders", "wiki_pages",
|
||||
"wiki_page_issues", "wiki_log_entries",
|
||||
},
|
||||
RequiredColumns: map[string][]string{
|
||||
"knowledges": {"id", "tenant_id", "knowledge_base_id", "deleted_at", "parse_status"},
|
||||
"chunks": {"id", "tenant_id", "knowledge_base_id", "knowledge_id", "content"},
|
||||
"knowledge_tags": {"id", "tenant_id", "knowledge_base_id"},
|
||||
"knowledge_tag_relations": {"knowledge_id", "tag_id"},
|
||||
"resources": {"id", "handle", "tenant_id", "physical_path"},
|
||||
"resource_bindings": {"id", "resource_id", "tenant_id", "owner_type", "owner_id"},
|
||||
"wiki_pages": {"id", "tenant_id", "knowledge_base_id", "slug"},
|
||||
"wiki_folders": {"id", "tenant_id", "knowledge_base_id"},
|
||||
"wiki_page_issues": {"id", "tenant_id", "knowledge_base_id"},
|
||||
"wiki_log_entries": {"id", "tenant_id", "knowledge_base_id"},
|
||||
},
|
||||
RequiredParts: []PartRequirement{
|
||||
{"database", "knowledges"}, {"database", "chunks"},
|
||||
{"database", "knowledge_tags"}, {"database", "knowledge_tag_relations"},
|
||||
{"database", "resources"}, {"database", "resource_bindings"},
|
||||
{"wiki", "wiki_folders"}, {"wiki", "wiki_pages"},
|
||||
{"wiki", "wiki_page_issues"}, {"wiki", "wiki_log_entries"},
|
||||
{"index", "chunks"},
|
||||
},
|
||||
IndexContractSource: "vector_stores.index_config.index_name",
|
||||
},
|
||||
V090Version: {
|
||||
Name: "v090",
|
||||
SchemaVersion: V090Version,
|
||||
WeKnoraVersion: "v0.9.0",
|
||||
// 0.9.0 的 KB 不写 vector_store_id,维度只能从 models.parameters 读。
|
||||
EmbeddingDimension: 0,
|
||||
RequiredTables: []string{
|
||||
"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
|
||||
"resources", "resource_bindings", "wiki_folders", "wiki_pages",
|
||||
"wiki_page_issues",
|
||||
},
|
||||
// 0.9.0 删掉了 wiki_log_entries;部件集合随之为空,不许假装还有。
|
||||
OptionalTables: []string{"wiki_log_entries"},
|
||||
RequiredColumns: map[string][]string{
|
||||
"knowledges": {"id", "tenant_id", "knowledge_base_id", "deleted_at", "parse_status"},
|
||||
"chunks": {"id", "tenant_id", "knowledge_base_id", "knowledge_id", "content", "seq_id", "content_revision", "index_status"},
|
||||
"knowledge_tags": {"id", "tenant_id", "knowledge_base_id"},
|
||||
"knowledge_tag_relations": {"knowledge_id", "tag_id"},
|
||||
"resources": {"id", "handle", "tenant_id", "physical_path"},
|
||||
"resource_bindings": {"id", "resource_id", "tenant_id", "owner_type", "owner_id"},
|
||||
"wiki_pages": {"id", "tenant_id", "knowledge_base_id", "slug"},
|
||||
"wiki_folders": {"id", "tenant_id", "knowledge_base_id"},
|
||||
"wiki_page_issues": {"id", "tenant_id", "knowledge_base_id"},
|
||||
},
|
||||
RequiredParts: []PartRequirement{
|
||||
{"database", "knowledges"}, {"database", "chunks"},
|
||||
{"database", "knowledge_tags"}, {"database", "knowledge_tag_relations"},
|
||||
{"database", "resources"}, {"database", "resource_bindings"},
|
||||
{"wiki", "wiki_folders"}, {"wiki", "wiki_pages"}, {"wiki", "wiki_page_issues"},
|
||||
{"index", "chunks"},
|
||||
},
|
||||
OptionalParts: []PartRequirement{
|
||||
{"wiki", "wiki_log_entries"},
|
||||
},
|
||||
// 0.9.0 的索引名来自部署配置(ELASTICSEARCH_INDEX),不是 vector_stores。
|
||||
// 0.9.0 在 chunks 上挂了图片同步触发器(仅当 image_info 以 '[' 开头时触发),
|
||||
// 它维护 chunk_images 派生表;定义逐字核对,任何改动都拒绝。
|
||||
ReviewedTriggers: []string{
|
||||
`CREATE TRIGGER trg_chunk_images_delete AFTER DELETE ON public.chunks FOR EACH ROW WHEN ((old.image_info ~~ '[%'::text)) EXECUTE FUNCTION chunk_images_sync()`,
|
||||
`CREATE TRIGGER trg_chunk_images_insert AFTER INSERT ON public.chunks FOR EACH ROW WHEN ((new.image_info ~~ '[%'::text)) EXECUTE FUNCTION chunk_images_sync()`,
|
||||
`CREATE TRIGGER trg_chunk_images_update AFTER UPDATE ON public.chunks FOR EACH ROW WHEN ((((old.image_info ~~ '[%'::text) OR (new.image_info ~~ '[%'::text)) AND ((old.image_info IS DISTINCT FROM new.image_info) OR (old.is_enabled IS DISTINCT FROM new.is_enabled) OR (old.status IS DISTINCT FROM new.status) OR (old.deleted_at IS DISTINCT FROM new.deleted_at) OR (old.tenant_id IS DISTINCT FROM new.tenant_id) OR ((old.knowledge_base_id)::text IS DISTINCT FROM (new.knowledge_base_id)::text) OR ((old.knowledge_id)::text IS DISTINCT FROM (new.knowledge_id)::text) OR ((old.chunk_type)::text IS DISTINCT FROM (new.chunk_type)::text) OR (old.created_at IS DISTINCT FROM new.created_at) OR (old.updated_at IS DISTINCT FROM new.updated_at)))) EXECUTE FUNCTION chunk_images_sync()`,
|
||||
},
|
||||
IndexContractSource: "config ELASTICSEARCH_INDEX",
|
||||
},
|
||||
}
|
||||
|
||||
// For 按版本号选中 profile。未知版本一律失败:版本号是"选哪份声明"的索引,
|
||||
// 不是"能不能放行"的依据,所以绝不返回默认 profile。
|
||||
func For(version int) (Profile, error) {
|
||||
profile, ok := profiles[version]
|
||||
if !ok {
|
||||
return Profile{}, fmt.Errorf("unsupported WeKnora schema version %d: supported profiles are 74 (v071) and 134 (v090)", version)
|
||||
}
|
||||
return profile, nil
|
||||
}
|
||||
|
||||
// Supported 返回已声明 profile 的版本号(用于文档与错误信息,不用于放行)。
|
||||
func Supported() []int {
|
||||
return []int{LegacyVersion, V090Version}
|
||||
}
|
||||
|
||||
// TableSet 是"可能出现的表"集合:必需表 + 可选表。
|
||||
func (p Profile) TableSet() map[string]bool {
|
||||
set := make(map[string]bool, len(p.RequiredTables)+len(p.OptionalTables))
|
||||
for _, name := range p.RequiredTables {
|
||||
set[name] = true
|
||||
}
|
||||
for _, name := range p.OptionalTables {
|
||||
set[name] = true
|
||||
}
|
||||
return set
|
||||
}
|
||||
|
||||
// HasTable 报告某张表是否属于该 profile(必需或可选)。
|
||||
func (p Profile) HasTable(name string) bool {
|
||||
return p.TableSet()[name]
|
||||
}
|
||||
|
||||
// IsRequiredTable 报告某张表是否必需存在。
|
||||
func (p Profile) IsRequiredTable(name string) bool {
|
||||
for _, candidate := range p.RequiredTables {
|
||||
if candidate == name {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// PartKeys 返回 (必需, 可选) 的归档部件键,形如 "wiki/wiki_log_entries"。
|
||||
func (p Profile) PartKeys() ([]string, []string) {
|
||||
key := func(part PartRequirement) string { return part.Kind + "/" + part.Name }
|
||||
required := make([]string, 0, len(p.RequiredParts))
|
||||
for _, part := range p.RequiredParts {
|
||||
required = append(required, key(part))
|
||||
}
|
||||
optional := make([]string, 0, len(p.OptionalParts))
|
||||
for _, part := range p.OptionalParts {
|
||||
optional = append(optional, key(part))
|
||||
}
|
||||
return required, optional
|
||||
}
|
||||
|
||||
// RequiresPart 报告某部件键是否必需(可选部件返回 false)。
|
||||
func (p Profile) RequiresPart(key string) bool {
|
||||
required, _ := p.PartKeys()
|
||||
for _, candidate := range required {
|
||||
if candidate == key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// AllowsPart 报告某部件键是否至少允许出现。
|
||||
func (p Profile) AllowsPart(key string) bool {
|
||||
required, optional := p.PartKeys()
|
||||
for _, candidate := range required {
|
||||
if candidate == key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, candidate := range optional {
|
||||
if candidate == key {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"knowledge-governance/internal/releasebundle"
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -420,24 +420,29 @@ func encodeManifest(m Manifest) ([]byte, error) {
|
||||
return nil, errors.New("unsupported snapshot validation policy")
|
||||
}
|
||||
c := m.Compatibility
|
||||
if c.WeKnoraVersion != releasebundle.WeKnoraVersionV071 || c.SchemaVersion != releasebundle.WeKnoraMigrationV071 || c.RetrieverEngine != "elasticsearch_v8" || !identifier(c.EmbeddingModel) || c.EmbeddingDimension != releasebundle.EmbeddingDimensionV071 || !validDigest(c.IndexMappingSHA256) {
|
||||
// 兼容性按 schema 版本选 profile,而不是钉死 v0.7.1:未知版本必须拒绝,
|
||||
// 已知版本还必须与自己的版本字符串、索引引擎、维度声明一致。
|
||||
profile, err := libraryprofile.For(c.SchemaVersion)
|
||||
if err != nil || c.WeKnoraVersion != profile.WeKnoraVersion || c.RetrieverEngine != "elasticsearch_v8" ||
|
||||
!identifier(c.EmbeddingModel) || c.EmbeddingDimension < 1 ||
|
||||
(profile.EmbeddingDimension > 0 && c.EmbeddingDimension != profile.EmbeddingDimension) ||
|
||||
!validDigest(c.IndexMappingSHA256) {
|
||||
return nil, errors.New("unsupported snapshot compatibility contract")
|
||||
}
|
||||
if m.Format != Format || (m.Scope != "internal" && m.Scope != "external") || m.TenantID == 0 || !identifier(m.KnowledgeBaseID) || m.CreatedAt.IsZero() {
|
||||
return nil, errors.New("invalid library manifest identity")
|
||||
}
|
||||
// 部件集合也来自 profile:0.9.0 的归档没有 wiki_log_entries 部件,
|
||||
// 但 0.7.1 缺了它就是不完整。
|
||||
requiredKeys, _ := profile.PartKeys()
|
||||
required := map[string]bool{}
|
||||
for _, n := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} {
|
||||
required["database/"+n] = true
|
||||
for _, key := range requiredKeys {
|
||||
required[key] = true
|
||||
}
|
||||
for _, n := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
|
||||
required["wiki/"+n] = true
|
||||
}
|
||||
required["index/chunks"] = true
|
||||
seen := map[string]bool{}
|
||||
for _, p := range m.Parts {
|
||||
key := p.Kind + "/" + p.Name
|
||||
if (!required[key] && !(p.Kind == "resource" && identifier(p.Name) && p.Count == 1) && !(key == "schema/migration-catalog" && p.Count == 1)) || seen[key] || !validDigest(p.SHA256) || p.Bytes < 0 || p.Count < 0 {
|
||||
if (!profile.AllowsPart(key) && !(p.Kind == "resource" && identifier(p.Name) && p.Count == 1) && !(key == "schema/migration-catalog" && p.Count == 1)) || seen[key] || !validDigest(p.SHA256) || p.Bytes < 0 || p.Count < 0 {
|
||||
return nil, errors.New("invalid or nonknowledge snapshot component")
|
||||
}
|
||||
seen[key] = true
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
package librarysnapshot
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 归档部件的完整性按 schema 版本判定:0.9.0 没有 wiki_log_entries,
|
||||
// 0.7.1 缺了它就是残包。这里直接对 manifest 编码做判定,不依赖数据库。
|
||||
func TestManifestPartSetFollowsSchemaProfile(t *testing.T) {
|
||||
part := func(kind, name string) Part {
|
||||
return Part{Kind: kind, Name: name, SHA256: strings.Repeat("a", 64), Count: 1}
|
||||
}
|
||||
parts := []Part{
|
||||
part("database", "knowledges"), part("database", "chunks"), part("database", "resources"),
|
||||
part("database", "resource_bindings"), part("database", "knowledge_tags"),
|
||||
part("database", "knowledge_tag_relations"),
|
||||
part("wiki", "wiki_pages"), part("wiki", "wiki_folders"), part("wiki", "wiki_page_issues"),
|
||||
part("index", "chunks"),
|
||||
}
|
||||
manifest := func(version int, weknora string, parts []Part) Manifest {
|
||||
return Manifest{
|
||||
ValidationPolicy: ArtifactAcceptancePolicy,
|
||||
Format: Format,
|
||||
Scope: "internal",
|
||||
TenantID: 10000,
|
||||
KnowledgeBaseID: "kb-v090",
|
||||
CreatedAt: time.Now().UTC(),
|
||||
Parts: parts,
|
||||
Compatibility: Compatibility{
|
||||
WeKnoraVersion: weknora, SchemaVersion: version,
|
||||
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "bge-m3",
|
||||
EmbeddingDimension: 1024, IndexMappingSHA256: strings.Repeat("b", 64),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := encodeManifest(manifest(134, "v0.9.0", parts)); err != nil {
|
||||
t.Fatalf("0.9.0 manifest without wiki_log_entries must be complete: %v", err)
|
||||
}
|
||||
if _, err := encodeManifest(manifest(74, "v0.7.1", parts)); err == nil || !strings.Contains(err.Error(), "incomplete") {
|
||||
t.Fatalf("0.7.1 manifest missing wiki_log_entries must be refused, got %v", err)
|
||||
}
|
||||
withLog := append(append([]Part{}, parts...), part("wiki", "wiki_log_entries"))
|
||||
if _, err := encodeManifest(manifest(74, "v0.7.1", withLog)); err != nil {
|
||||
t.Fatalf("0.7.1 manifest with wiki_log_entries must pass: %v", err)
|
||||
}
|
||||
if _, err := encodeManifest(manifest(134, "v0.9.0", withLog)); err != nil {
|
||||
t.Fatalf("0.9.0 manifest may still carry an optional wiki_log_entries part: %v", err)
|
||||
}
|
||||
if _, err := encodeManifest(manifest(74, "v0.9.0", withLog)); err == nil {
|
||||
t.Fatal("version string and profile version must agree")
|
||||
}
|
||||
if _, err := encodeManifest(manifest(135, "v0.9.0", parts)); err == nil {
|
||||
t.Fatal("unknown schema version must be refused")
|
||||
}
|
||||
}
|
||||
@@ -77,7 +77,10 @@ func TestNormalizeRefusesEnabledSettingsWithoutTransportAndBadRecipients(t *test
|
||||
t.Fatalf("defaults not applied: %+v %v", good, err)
|
||||
}
|
||||
// 解析不了的主机名必须仍被判为非法:.invalid 是 RFC 6761 保留域,永远不解析。
|
||||
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid", From: "kb@example.com",
|
||||
// 末尾的点是必须的:构建机的 resolv.conf 可能带 search 域(本环境 Tailscale
|
||||
// 的 tailf49802.ts.net),不带点就会被拼成 smtp.kg-test.invalid.<search> 并
|
||||
// 由 Fake-IP DNS 解析出来,于是这条断言测的是构建机的搜索域而不是被测逻辑。
|
||||
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid.", From: "kb@example.com",
|
||||
Recipients: []string{"a@example.com"}}); err != ErrInvalid {
|
||||
t.Fatalf("unresolvable host must stay invalid, got %v", err)
|
||||
}
|
||||
|
||||
@@ -3,10 +3,12 @@ package v071
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
"knowledge-governance/internal/releasebundle"
|
||||
releaseworkflow "knowledge-governance/internal/workflows/release"
|
||||
)
|
||||
@@ -37,19 +39,23 @@ type TargetContractInspector struct {
|
||||
config TargetContractInspectorConfig
|
||||
}
|
||||
|
||||
// targetContractSQL 用 LEFT JOIN 读模型与向量库:命中不到时仍然拿得到"声明了什么",
|
||||
// 这样缺字段能被点名拒绝,而不是整行查不出来只报一句"读契约失败"。
|
||||
const targetContractSQL = `
|
||||
SELECT migration.version,
|
||||
migration.dirty,
|
||||
SELECT COALESCE(migration.version, 0),
|
||||
COALESCE(migration.dirty, false),
|
||||
knowledge_base.tenant_id,
|
||||
model.name,
|
||||
COALESCE(knowledge_base.embedding_model_id, ''),
|
||||
COALESCE(model.name, ''),
|
||||
COALESCE(NULLIF(model.parameters->'embedding_parameters'->>'dimension', '')::INTEGER, 0),
|
||||
(model.id IS NOT NULL),
|
||||
COALESCE(vector_store.engine_type, ''),
|
||||
COALESCE(vector_store.index_config->>'index_name', '')
|
||||
FROM knowledge_bases AS knowledge_base
|
||||
JOIN models AS model
|
||||
LEFT JOIN models AS model
|
||||
ON model.id = knowledge_base.embedding_model_id
|
||||
AND model.deleted_at IS NULL
|
||||
JOIN schema_migrations AS migration ON TRUE
|
||||
LEFT JOIN schema_migrations AS migration ON TRUE
|
||||
LEFT JOIN vector_stores AS vector_store
|
||||
ON vector_store.id = knowledge_base.vector_store_id
|
||||
AND vector_store.deleted_at IS NULL
|
||||
@@ -57,6 +63,25 @@ WHERE knowledge_base.id = $1
|
||||
AND knowledge_base.deleted_at IS NULL
|
||||
`
|
||||
|
||||
// resolveTargetIndexName 是"索引名从哪来"的唯一判定点。
|
||||
//
|
||||
// 0.9.0 的 KB 不写 vector_store_id(vector_stores 没有对应行),索引名只在部署
|
||||
// 配置里(ELASTICSEARCH_INDEX)。两条来源都拿不到时**不许**回退成空串——
|
||||
// 空串会让后续过滤条件匹配到任何索引。0.7.1 若两处都有,必须一致。
|
||||
func resolveTargetIndexName(fromVectorStore, configured string, profile libraryprofile.Profile) (string, error) {
|
||||
resolved := fromVectorStore
|
||||
if resolved == "" {
|
||||
resolved = configured
|
||||
}
|
||||
if !validSearchIdentifier(resolved) {
|
||||
return "", fmt.Errorf("target WeKnora index name unresolved (contract source: %s): vector_stores.index_config->>'index_name' and the configured index are both empty", profile.IndexContractSource)
|
||||
}
|
||||
if fromVectorStore != "" && fromVectorStore != configured {
|
||||
return "", fmt.Errorf("target WeKnora contract mismatch: vector_stores.index_config.index_name=%q does not match configured index %q", fromVectorStore, configured)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func NewTargetContractInspector(config TargetContractInspectorConfig) (*TargetContractInspector, error) {
|
||||
config.KnowledgeBaseID = strings.TrimSpace(config.KnowledgeBaseID)
|
||||
config.IndexName = strings.TrimSpace(config.IndexName)
|
||||
@@ -85,29 +110,51 @@ func (inspector *TargetContractInspector) Contract(ctx context.Context, item rel
|
||||
migrationVersion int
|
||||
dirty bool
|
||||
tenantID uint64
|
||||
declaredModel string
|
||||
embeddingModel string
|
||||
dimension int
|
||||
modelBound bool
|
||||
engineType string
|
||||
indexName string
|
||||
)
|
||||
if err := inspector.config.DB.QueryRow(ctx, targetContractSQL, targetKnowledgeBaseID).Scan(
|
||||
&migrationVersion, &dirty, &tenantID, &embeddingModel, &dimension, &engineType, &indexName,
|
||||
&migrationVersion, &dirty, &tenantID, &declaredModel, &embeddingModel, &dimension, &modelBound, &engineType, &indexName,
|
||||
); err != nil {
|
||||
return releasebundle.TargetContract{}, errors.New("read target WeKnora contract")
|
||||
}
|
||||
if migrationVersion != releasebundle.WeKnoraMigrationV071 || dirty || tenantID != inspector.config.TenantID ||
|
||||
!validSearchIdentifier(embeddingModel) || dimension < 1 ||
|
||||
(engineType != "" && engineType != "elasticsearch") ||
|
||||
(indexName != "" && indexName != inspector.config.IndexName) {
|
||||
return releasebundle.TargetContract{}, errors.New("target WeKnora contract mismatch")
|
||||
if dirty {
|
||||
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora schema_migrations.version=%d is dirty: refusing to publish", migrationVersion)
|
||||
}
|
||||
// 版本号只用来选 profile;未知版本失败关闭,错误里带真实版本号。
|
||||
profile, err := libraryprofile.For(migrationVersion)
|
||||
if err != nil {
|
||||
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora: %w", err)
|
||||
}
|
||||
if tenantID != inspector.config.TenantID {
|
||||
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: knowledge_bases.tenant_id=%d, expected %d", tenantID, inspector.config.TenantID)
|
||||
}
|
||||
if !validSearchIdentifier(declaredModel) {
|
||||
return releasebundle.TargetContract{}, errors.New("target WeKnora contract mismatch: knowledge_bases.embedding_model_id is empty: embedding dimension is unknowable")
|
||||
}
|
||||
if !modelBound {
|
||||
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: models row for knowledge_bases.embedding_model_id %q is missing or deleted", declaredModel)
|
||||
}
|
||||
if !validSearchIdentifier(embeddingModel) || dimension < 1 {
|
||||
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: models.parameters.embedding_parameters.dimension is missing for model %q", declaredModel)
|
||||
}
|
||||
if engineType != "" && engineType != "elasticsearch" {
|
||||
return releasebundle.TargetContract{}, fmt.Errorf("target WeKnora contract mismatch: vector_stores.engine_type=%q is not elasticsearch", engineType)
|
||||
}
|
||||
if _, err := resolveTargetIndexName(indexName, inspector.config.IndexName, profile); err != nil {
|
||||
return releasebundle.TargetContract{}, err
|
||||
}
|
||||
mappingSHA256, err := inspector.config.Mapping.MappingSHA256(ctx)
|
||||
if err != nil || !validLowerSHA256(mappingSHA256) {
|
||||
return releasebundle.TargetContract{}, errors.New("read target Elasticsearch mapping contract")
|
||||
}
|
||||
return releasebundle.TargetContract{
|
||||
WeKnoraVersion: releasebundle.WeKnoraVersionV071,
|
||||
MigrationVersion: releasebundle.WeKnoraMigrationV071,
|
||||
WeKnoraVersion: profile.WeKnoraVersion,
|
||||
MigrationVersion: profile.SchemaVersion,
|
||||
TenantID: tenantID,
|
||||
KnowledgeBaseID: targetKnowledgeBaseID,
|
||||
RetrieverEngine: "elasticsearch_v8",
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
)
|
||||
|
||||
func TestTargetContractInspectorReadsOnlyCompatibilityFields(t *testing.T) {
|
||||
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", 1024, "elasticsearch", "chunks"}}}
|
||||
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", "bge-m3", 1024, true, "elasticsearch", "chunks"}}}
|
||||
mapping := &mappingInspector{sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}
|
||||
inspector, err := NewTargetContractInspector(TargetContractInspectorConfig{DB: db, Mapping: mapping, KnowledgeBaseID: "kb-safe", TenantID: 10000, IndexName: "chunks"})
|
||||
if err != nil {
|
||||
@@ -36,7 +36,7 @@ func TestTargetContractInspectorReadsOnlyCompatibilityFields(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestTargetContractInspectorRejectsMismatchedPlanKB(t *testing.T) {
|
||||
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", 1024, "elasticsearch", "chunks"}}}
|
||||
db := &contractDB{row: contractRow{values: []any{74, false, uint64(10000), "bge-m3", "bge-m3", 1024, true, "elasticsearch", "chunks"}}}
|
||||
inspector, _ := NewTargetContractInspector(TargetContractInspectorConfig{DB: db, Mapping: &mappingInspector{sha: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}, KnowledgeBaseID: "kb-safe", TenantID: 10000, IndexName: "chunks"})
|
||||
plan := releasePlanForArchive()
|
||||
plan.TestKnowledgeBaseID = "other-kb"
|
||||
@@ -50,7 +50,7 @@ func TestTargetContractInspectorRejectsMismatchedPlanKB(t *testing.T) {
|
||||
// the mapped base on the source plane made a real release fail with
|
||||
// "source WeKnora release contract mismatch" (the target base is unknown there).
|
||||
func TestTargetContractInspectorSidesPickTheRightBase(t *testing.T) {
|
||||
row := contractRow{values: []any{74, false, uint64(10000), "bge-m3", 1024, "elasticsearch", "chunks"}}
|
||||
row := contractRow{values: []any{74, false, uint64(10000), "bge-m3", "bge-m3", 1024, true, "elasticsearch", "chunks"}}
|
||||
mapping := &mappingInspector{sha: strings.Repeat("a", 64)}
|
||||
plan := releasePlanForArchive()
|
||||
plan.TargetKnowledgeBaseID = "prod-kb"
|
||||
|
||||
@@ -3,6 +3,7 @@ package v071
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -63,8 +64,14 @@ func (exporter *DocumentExporter) Export(ctx context.Context, item releaseworkfl
|
||||
return nil, errors.New("invalid document release plan")
|
||||
}
|
||||
sourceContract, err := exporter.config.Contract.Contract(ctx, item)
|
||||
if err != nil || sourceContract.WeKnoraVersion != releasebundle.WeKnoraVersionV071 ||
|
||||
sourceContract.MigrationVersion != releasebundle.WeKnoraMigrationV071 || sourceContract.TenantID != exporter.config.TenantID ||
|
||||
if err == nil && (sourceContract.WeKnoraVersion != releasebundle.WeKnoraVersionV071 ||
|
||||
sourceContract.MigrationVersion != releasebundle.WeKnoraMigrationV071) {
|
||||
// 单文档发布包是 0.7.1 契约:读到的库是别的版本时必须说清楚原因,
|
||||
// 而不是和"租户/模型不对"共用一句泛化错误。
|
||||
return nil, fmt.Errorf("single-document release supports only v0.7.1 (74): source library declares %s (%d), so its rows and index documents must go through the library-level publish path",
|
||||
sourceContract.WeKnoraVersion, sourceContract.MigrationVersion)
|
||||
}
|
||||
if err != nil || sourceContract.TenantID != exporter.config.TenantID ||
|
||||
sourceContract.KnowledgeBaseID != item.TestKnowledgeBaseID || sourceContract.RetrieverEngine != "elasticsearch_v8" ||
|
||||
sourceContract.EmbeddingModel != exporter.config.EmbeddingModel || sourceContract.EmbeddingDimension != releasebundle.EmbeddingDimensionV071 {
|
||||
return nil, errors.New("source WeKnora release contract mismatch")
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
)
|
||||
|
||||
// IndexMappingDigest 与线上读取路径(SearchExporter.inspectMapping)用同一套规范化:
|
||||
// 取 ES `/{index}/_mapping` 响应里该索引的 mappings,规范 JSON 后取 sha256。
|
||||
//
|
||||
// 单独暴露它是为了能用**真实抓下来的映射**证明摘要可算,并证明 0.7.1 与 0.9.0
|
||||
// 的映射是两个不同契约(不能因为"都是 1024 维"就当同一份)。
|
||||
func IndexMappingDigest(raw []byte) (string, error) {
|
||||
var response map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &response); err != nil || len(response) != 1 {
|
||||
return "", errors.New("Elasticsearch mapping response is invalid")
|
||||
}
|
||||
for _, indexRaw := range response {
|
||||
var index struct {
|
||||
Mappings json.RawMessage `json:"mappings"`
|
||||
}
|
||||
if err := json.Unmarshal(indexRaw, &index); err != nil || len(index.Mappings) == 0 {
|
||||
return "", errors.New("Elasticsearch mapping response is invalid")
|
||||
}
|
||||
canonical, err := canonicalJSON(index.Mappings)
|
||||
if err != nil {
|
||||
return "", errors.New("Elasticsearch mapping response is invalid")
|
||||
}
|
||||
digest := sha256.Sum256(canonical)
|
||||
return hex.EncodeToString(digest[:]), nil
|
||||
}
|
||||
return "", errors.New("Elasticsearch mapping response is invalid")
|
||||
}
|
||||
@@ -145,13 +145,19 @@ type libraryRow struct {
|
||||
}
|
||||
|
||||
func validateLibraryCandidate(identity PostgresIdentity, pg PostgresSnapshot, es SearchSnapshot) error {
|
||||
required := map[string]bool{"resources": true}
|
||||
for _, entry := range libraryContentQueries {
|
||||
required[entry.name] = true
|
||||
// 必需表来自 profile,不是固定清单:0.9.0 没有 wiki_log_entries 就不要求它,
|
||||
// 但只要表在 profile 里是必需的,缺了就必须失败。
|
||||
profile, err := pg.schemaProfile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
required := map[string]bool{}
|
||||
for _, name := range profile.RequiredTables {
|
||||
required[name] = true
|
||||
}
|
||||
tables := map[string]map[string]libraryRow{}
|
||||
for _, table := range pg.Tables {
|
||||
if !required[table.Name] || tables[table.Name] != nil || table.RowCount != len(table.Rows) || table.SHA256 != newTableSnapshot(table.Name, table.Rows).SHA256 {
|
||||
if !profile.HasTable(table.Name) || tables[table.Name] != nil || table.RowCount != len(table.Rows) || table.SHA256 != newTableSnapshot(table.Name, table.Rows).SHA256 {
|
||||
return errors.New("invalid library table inventory")
|
||||
}
|
||||
rows := map[string]libraryRow{}
|
||||
|
||||
@@ -30,6 +30,8 @@ type ProductionBackupStore struct {
|
||||
maxDocuments int
|
||||
maxObjectByte int64
|
||||
now func() time.Time
|
||||
// tables 是目标库解析出的知识域表集合(0.9.0 没有 wiki_log_entries)。
|
||||
tables []string
|
||||
}
|
||||
|
||||
type ProductionBackupConfig struct {
|
||||
@@ -58,10 +60,16 @@ func NewProductionBackupStore(config ProductionBackupConfig) (*ProductionBackupS
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
// 预备份的表集合必须与写入/校验同源:按 0.7.1 固定清单会在 0.9.0 目标上
|
||||
// 查询不存在的 wiki_log_entries。
|
||||
schema, err := ResolveSchemaProfile(context.Background(), config.Pool, "prebackup target")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ProductionBackupStore{pool: config.Pool, search: config.Search, objects: config.Objects,
|
||||
archiveRoot: config.ArchiveRoot, scope: config.Scope, identity: config.Identity,
|
||||
compatibility: config.Compatibility, maxDocuments: config.MaxDocuments,
|
||||
maxObjectByte: config.MaxObjectByte, now: now}, nil
|
||||
maxObjectByte: config.MaxObjectByte, now: now, tables: schema.Tables}, nil
|
||||
}
|
||||
|
||||
// tableFingerprint 是一张表的"数量 + 内容摘要"指纹:同数量改写也会被识别。
|
||||
@@ -75,7 +83,7 @@ func (s *ProductionBackupStore) fingerprints(ctx context.Context) (map[string]ta
|
||||
result := map[string]tableFingerprint{}
|
||||
exporter := PostgresExporter{db: s.pool}
|
||||
var bodies, bindings []json.RawMessage
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range s.tables {
|
||||
if table == "resources" {
|
||||
continue
|
||||
}
|
||||
@@ -267,7 +275,7 @@ func (s *ProductionBackupStore) objectVersions(ctx context.Context, pg PostgresS
|
||||
|
||||
func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) {
|
||||
counts := map[string]int64{}
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range s.tables {
|
||||
var count int64
|
||||
if err := s.pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`,
|
||||
table, scopeWhere(table)), s.identity.TenantID, s.identity.KnowledgeBaseID).Scan(&count); err != nil {
|
||||
|
||||
@@ -19,19 +19,18 @@ func exportLibraryBeforeImage(ctx context.Context, db librarySnapshotDB, identit
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
defer tx.Rollback(context.WithoutCancel(ctx))
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := tx.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
|
||||
return PostgresSnapshot{}, errors.New("before-image schema contract mismatch")
|
||||
profile, err := ResolveSchemaProfile(ctx, tx, "before-image source")
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
exporter := PostgresExporter{db: tx}
|
||||
rowsByTable := map[string][]json.RawMessage{}
|
||||
var bodies, bindings []json.RawMessage
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range profile.Tables {
|
||||
if table == "resources" {
|
||||
continue
|
||||
}
|
||||
rows, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s ORDER BY (%s)::text`, table, table, scopeWhere(table), identityExpr(table)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
rows, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s ORDER BY (%s)::text`, table, table, scopeWhereFor(profile.Tables, table), identityExpr(table)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
@@ -50,8 +49,8 @@ func exportLibraryBeforeImage(ctx context.Context, db librarySnapshotDB, identit
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
rowsByTable["resources"] = resources
|
||||
snapshot := PostgresSnapshot{}
|
||||
for _, table := range productionLibraryTables {
|
||||
snapshot := PostgresSnapshot{SchemaVersion: profile.SchemaVersion, Profile: profile.Name}
|
||||
for _, table := range profile.Tables {
|
||||
snapshot.Tables = append(snapshot.Tables, newTableSnapshot(table, rowsByTable[table]))
|
||||
}
|
||||
handles, err := libraryResourceHandles(bodies)
|
||||
|
||||
@@ -29,10 +29,10 @@ func ExportLibraryWithReleases(ctx context.Context, db librarySnapshotDB, identi
|
||||
return PostgresSnapshot{}, errors.New("begin library snapshot")
|
||||
}
|
||||
defer tx.Rollback(context.WithoutCancel(ctx))
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := tx.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
|
||||
return PostgresSnapshot{}, errors.New("source library schema contract mismatch")
|
||||
// 版本号只用来选 profile;放行条件是随后的表/列能力探测。
|
||||
profile, err := ResolveSchemaProfile(ctx, tx, "source library")
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
var graph bool
|
||||
if err := tx.QueryRow(ctx, `SELECT COALESCE((indexing_strategy->>'graph_enabled')::boolean,false) FROM knowledge_bases WHERE tenant_id=$1 AND id=$2 AND deleted_at IS NULL`, identity.TenantID, identity.KnowledgeBaseID).Scan(&graph); err != nil {
|
||||
@@ -62,15 +62,19 @@ func ExportLibraryWithReleases(ctx context.Context, db librarySnapshotDB, identi
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
exporter := PostgresExporter{db: tx}
|
||||
var snapshot PostgresSnapshot
|
||||
snapshot.Catalog, _, err = captureMigrationCatalog(ctx, tx)
|
||||
snapshot := PostgresSnapshot{SchemaVersion: profile.SchemaVersion, Profile: profile.Name}
|
||||
snapshot.Catalog, _, err = captureMigrationCatalogFor(ctx, tx, profile.CatalogTables())
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
if err = validateMigrationCatalog(snapshot.Catalog); err != nil {
|
||||
if err = validateMigrationCatalogFor(snapshot.Catalog, profile.CatalogTables(), profile.Profile); err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
for _, entry := range libraryContentQueries {
|
||||
// 0.9.0 没有 wiki_log_entries:不产生该部件,也不去查不存在的表。
|
||||
if !profile.HasTable(entry.name) {
|
||||
continue
|
||||
}
|
||||
rows, err := exporter.queryRows(ctx, entry.sql, identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
|
||||
@@ -103,6 +103,10 @@ func (d *libraryFixtureTx) Query(_ context.Context, q string, _ ...any) (pgx.Row
|
||||
if d.queryError {
|
||||
return nil, errors.New("fixture failure")
|
||||
}
|
||||
// 能力探测走真实 SQL:假库按 0.7.1 profile 回答,避免"桩放行"。
|
||||
if strings.Contains(q, "information_schema.columns") {
|
||||
return legacySchemaProbeRows(), nil
|
||||
}
|
||||
// 死信行按真实形状返回:解除判定必须逐字节比对指纹,空结果会让"存在即阻断"失效。
|
||||
if strings.Contains(q, "to_jsonb(d)") && d.failed > 0 {
|
||||
return &jsonRows{values: []json.RawMessage{json.RawMessage(fixtureDeadLetter)}}, nil
|
||||
|
||||
@@ -111,7 +111,7 @@ func (t *ProductionLibraryTarget) preflightNaturalKeys(ctx context.Context, arch
|
||||
targetFilter := predicate
|
||||
args := []any{raw}
|
||||
if part.Name != "resources" {
|
||||
scope := strings.ReplaceAll(strings.ReplaceAll(scopeWhere(part.Name), "$2", "$3"), "$1", "$2")
|
||||
scope := strings.ReplaceAll(strings.ReplaceAll(t.scopeWhere(part.Name), "$2", "$3"), "$1", "$2")
|
||||
targetFilter += " AND NOT (" + scope + ")"
|
||||
args = append(args, t.scope.TenantID, t.scope.KnowledgeBaseID)
|
||||
}
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
"knowledge-governance/internal/objectgateway"
|
||||
@@ -85,13 +86,22 @@ func (config ProductionLibraryConfig) validate(scope ProductionScope) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// 依赖顺序:插入按正序,删除按倒序。
|
||||
// 真实 Schema 有外键(resource_bindings.resource_id → resources.id),
|
||||
// 所以父表 resources 必须排在子表 resource_bindings 之前。
|
||||
var productionLibraryTables = []string{
|
||||
"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
|
||||
"resources", "resource_bindings", "wiki_folders", "wiki_pages",
|
||||
"wiki_page_issues", "wiki_log_entries",
|
||||
// productionLibraryTables 是 0.7.1 的知识域表集合(依赖顺序:插入按正序,删除按
|
||||
// 倒序;真实 Schema 有外键 resource_bindings.resource_id → resources.id,父表
|
||||
// resources 必须排在子表 resource_bindings 之前)。
|
||||
//
|
||||
// 它**不是**本包整库读写的表清单来源:真实路径一律用 ResolveSchemaProfile 解析出的
|
||||
// Tables——0.9.0 没有 wiki_log_entries,任何一条固定清单的查询都会在真库上失败,而
|
||||
// "表不存在就跳过"又会让 0.7.1 少一张必需表也悄悄通过。这里只留给明确 74-only 的
|
||||
// 入口(编目、残留复核、0.7.1 夹具),定义直接取自 profile 声明,避免两份清单走偏。
|
||||
var productionLibraryTables = legacyKnowledgeTables()
|
||||
|
||||
func legacyKnowledgeTables() []string {
|
||||
profile, err := libraryprofile.For(libraryprofile.LegacyVersion)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return append([]string(nil), profile.RequiredTables...)
|
||||
}
|
||||
|
||||
type ProductionLibraryTarget struct {
|
||||
@@ -100,6 +110,11 @@ type ProductionLibraryTarget struct {
|
||||
scope ProductionScope
|
||||
search *SearchExporter
|
||||
now func() time.Time
|
||||
// schema / tables 是目标库解析出的版本声明与"实际存在的知识域表"。写入、校验、
|
||||
// 清理、编目全部按它取数:0.9.0 缺 wiki_log_entries 时不得查询/删除该表,
|
||||
// 而 0.7.1 少一张必需表必须失败,不能靠"表不存在就跳过"。
|
||||
schema SchemaProfile
|
||||
tables []string
|
||||
// storageBackend / storageProvider 是本次写入使用的目标存储后端与提供方
|
||||
// (写入与校验共用,缺一不可:只换桶不换提供方,目标取文件仍会走来源实例的路)。
|
||||
storageBackend string
|
||||
@@ -215,7 +230,15 @@ func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryCon
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
return &ProductionLibraryTarget{pool: pool, config: config, scope: scope, search: search, now: now}, nil
|
||||
// 目标版本在装配时定型:未知版本/dirty/缺表在这里就失败,不能等到第一条写入
|
||||
// 才发现"目标其实没有这张表"。
|
||||
schema, err := ResolveSchemaProfile(ctx, pool, "publish target")
|
||||
if err != nil {
|
||||
pool.Close()
|
||||
return nil, err
|
||||
}
|
||||
return &ProductionLibraryTarget{pool: pool, config: config, scope: scope, search: search, now: now,
|
||||
schema: schema, tables: schema.Tables}, nil
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) Close() { t.pool.Close() }
|
||||
@@ -235,7 +258,7 @@ func (t *ProductionLibraryTarget) Identity() PostgresIdentity {
|
||||
func (t *ProductionLibraryTarget) Current(ctx context.Context) (libraryrelease.CurrentState, error) {
|
||||
state := libraryrelease.CurrentState{Counts: map[string]int64{}, Digests: map[string]string{},
|
||||
CapturedAt: t.now().UTC()}
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range t.tables {
|
||||
count, digest, err := t.tableState(ctx, table)
|
||||
if err != nil {
|
||||
return libraryrelease.CurrentState{}, err
|
||||
@@ -255,10 +278,19 @@ func (t *ProductionLibraryTarget) Current(ctx context.Context) (libraryrelease.C
|
||||
return state, nil
|
||||
}
|
||||
|
||||
// scopeWhere 返回"这张表里属于本知识库的行"的过滤条件($1=tenant,$2=kb)。
|
||||
// scopeWhere 是 74-only 入口(残留复核、0.7.1 夹具)使用的固定表集合版本。
|
||||
func scopeWhere(table string) string { return scopeWhereFor(productionLibraryTables, table) }
|
||||
|
||||
// scopeWhere 用本次目标解析出的表集合。
|
||||
func (t *ProductionLibraryTarget) scopeWhere(table string) string {
|
||||
return scopeWhereFor(t.tables, table)
|
||||
}
|
||||
|
||||
// scopeWhereFor 返回"这张表里属于本知识库的行"的过滤条件($1=tenant,$2=kb)。
|
||||
// 不同表的归属列不同:关系表与绑定表没有 knowledge_base_id,必须走父表判定,
|
||||
// 否则要么删错、要么把别的知识库数据算进来。
|
||||
func scopeWhere(table string) string {
|
||||
// 否则要么删错、要么把别的知识库数据算进来。表集合来自解析出的 profile:
|
||||
// 0.9.0 的资源正文引用子查询不能提到不存在的 wiki_log_entries。
|
||||
func scopeWhereFor(tables []string, table string) string {
|
||||
switch table {
|
||||
case "knowledge_tag_relations":
|
||||
// 真实 WeKnora 的这张表只有 knowledge_id/tag_id/created_at,
|
||||
@@ -283,7 +315,7 @@ func scopeWhere(table string) string {
|
||||
SELECT 1 FROM resource_bindings b
|
||||
JOIN chunks c ON c.id=b.owner_id AND c.tenant_id=b.tenant_id
|
||||
WHERE b.resource_id=resources.id AND b.tenant_id=$1 AND b.owner_type='chunk' AND c.knowledge_base_id=$2)
|
||||
OR EXISTS (` + libraryResourceBodyReferenceSQL("c.tenant_id=$1 AND c.knowledge_base_id=$2") + `))`
|
||||
OR EXISTS (` + libraryResourceBodyReferenceSQL(tables, "c.tenant_id=$1 AND c.knowledge_base_id=$2") + `))`
|
||||
default:
|
||||
return "tenant_id=$1 AND knowledge_base_id=$2"
|
||||
}
|
||||
@@ -308,12 +340,12 @@ func (t *ProductionLibraryTarget) tableState(ctx context.Context, table string)
|
||||
// scan used by the legacy scope predicate.
|
||||
exporter := PostgresExporter{db: t.pool}
|
||||
var bodies, bindings []json.RawMessage
|
||||
for _, name := range productionLibraryTables {
|
||||
for _, name := range t.tables {
|
||||
if name == "resources" {
|
||||
continue
|
||||
}
|
||||
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`,
|
||||
name, name, scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
name, name, t.scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return 0, "", fmt.Errorf("读取生产 %s 失败", table)
|
||||
}
|
||||
@@ -335,7 +367,7 @@ func (t *ProductionLibraryTarget) tableState(ctx context.Context, table string)
|
||||
// canonicalJSON on text and fail on every non-empty table.
|
||||
if table != "resources" {
|
||||
rows, err := t.pool.Query(ctx, fmt.Sprintf(`SELECT %s FROM %s WHERE %s ORDER BY 1`,
|
||||
"("+identityExpr(table)+")::text", table, scopeWhere(table)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
"("+identityExpr(table)+")::text", table, t.scopeWhere(table)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return 0, "", fmt.Errorf("读取生产 %s 失败", table)
|
||||
}
|
||||
@@ -404,6 +436,9 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
|
||||
if err != nil {
|
||||
return errors.New("预发布快照不可读")
|
||||
}
|
||||
if err := requireSameProfile(manifest.Compatibility.SchemaVersion, t.schema, "publish"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -444,8 +479,8 @@ func (t *ProductionLibraryTarget) importRows(ctx context.Context, archive *libra
|
||||
return errors.New("开启生产事务失败")
|
||||
}
|
||||
defer func() { _ = tx.Rollback(context.WithoutCancel(ctx)) }()
|
||||
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
|
||||
table := productionLibraryTables[index]
|
||||
for index := len(t.tables) - 1; index >= 0; index-- {
|
||||
table := t.tables[index]
|
||||
if _, ok := parts[table]; !ok {
|
||||
continue
|
||||
}
|
||||
@@ -453,7 +488,7 @@ func (t *ProductionLibraryTarget) importRows(ctx context.Context, archive *libra
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range t.tables {
|
||||
part, ok := parts[table]
|
||||
if !ok {
|
||||
continue
|
||||
@@ -473,7 +508,7 @@ func (t *ProductionLibraryTarget) deleteScope(ctx context.Context, tx pgx.Tx, ta
|
||||
// 资源可能被多个知识库共享:这里不删,只做插入/更新。
|
||||
return nil
|
||||
}
|
||||
statement := fmt.Sprintf(`DELETE FROM %s WHERE %s`, table, scopeWhere(table))
|
||||
statement := fmt.Sprintf(`DELETE FROM %s WHERE %s`, table, t.scopeWhere(table))
|
||||
if _, err := tx.Exec(ctx, statement, t.scope.TenantID, t.scope.KnowledgeBaseID); err != nil {
|
||||
return fmt.Errorf("清理生产 %s 失败", table)
|
||||
}
|
||||
@@ -810,6 +845,9 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err != nil {
|
||||
return errors.New("预发布快照不可读")
|
||||
}
|
||||
if err := requireSameProfile(manifest.Compatibility.SchemaVersion, t.schema, "publish verify"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -1292,7 +1330,7 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
|
||||
if err := t.AdoptWriteSet(*backup.WriteSet); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureRestorableResourceReferences(ctx, t.pool, *backup.WriteSet); err != nil {
|
||||
if err := ensureRestorableResourceReferences(ctx, t.pool, t.tables, *backup.WriteSet); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ensureNoResourceGrants(ctx, t.pool, backup.WriteSet.CreatedIDs("resources")); err != nil {
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// 0.9.0 整库发布目标的隔离夹具身份。来源=0.9.0 测试实例,目标=另一租户/知识库,
|
||||
// 这样"资源定位符换桶/后端/租户段"与"行改写目标范围"都被真实覆盖。
|
||||
const (
|
||||
v090TargetTenant = 10001
|
||||
v090TargetKB = "kb-v090-prod"
|
||||
v090TargetBackend = "backend-v090-target"
|
||||
v090TargetBucket = "kg-v090-target-bucket"
|
||||
v090SourceBackend = "backend-v090-source"
|
||||
// 来源定位符是目标应用真实使用的形状:storage://<backend>/<scheme>://<bucket>/<key>。
|
||||
v090SourceLocator = "storage://backend-v090-source/oss://kg-v090-source-bucket/weknora/10000/rs-v090-1/kn.bin"
|
||||
)
|
||||
|
||||
// v090ObjectBytes 按"提供方|路径"供字节,既能给封存读来源对象,也能给目标写回。
|
||||
type v090ObjectBytes map[string][]byte
|
||||
|
||||
func (o v090ObjectBytes) Open(_ context.Context, provider, physicalPath string) (io.ReadCloser, error) {
|
||||
content, ok := o[provider+"|"+physicalPath]
|
||||
if !ok {
|
||||
return nil, errors.New("fixture object missing: " + provider + "|" + physicalPath)
|
||||
}
|
||||
return io.NopCloser(bytes.NewReader(content)), nil
|
||||
}
|
||||
|
||||
// TestV090ProductionTargetPublishIsolated 是 0.9.0(schema 134)**发布写入路径**的
|
||||
// 端到端证据:真实导出 → 真实封存 → 真实 ProductionLibraryTarget 写进干净 134 目标
|
||||
// → 逐表核对 → 用写前镜像恢复回滚 → 再核对回到写前状态。
|
||||
//
|
||||
// 这里不允许打桩 PG/索引/对象:134 与 74 的差别(没有 wiki_log_entries、chunks 多
|
||||
// NOT NULL 列、seq_id 由目标序列分配)只有在真 schema 上才会暴露。因此默认 CI 跳过,
|
||||
// 在 scripts/test-scope.json 里声明为可选未运行,并写明理由。
|
||||
func TestV090ProductionTargetPublishIsolated(t *testing.T) {
|
||||
base := os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
|
||||
if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" || base == "" {
|
||||
t.Skip("requires isolated PG plus ES opt-in (KG_LIBRARY_TEST_ISOLATED=1, KG_LIBRARY_TEST_ELASTIC_URL)")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
src := v090Pool(t, "KG_V090_SOURCE_DSN")
|
||||
dst := v090Pool(t, "KG_V090_TARGET_DSN")
|
||||
identity := seedV090(t, ctx, src)
|
||||
// 资源行必须带目标实例真实可解析的定位符,否则"换桶/换后端"这条改写没有被覆盖。
|
||||
if _, err := src.Exec(ctx, `UPDATE resources SET provider='oss', storage_backend_id=$2,
|
||||
physical_path=$3, location_hash=$4 WHERE id=$1`,
|
||||
v090Resource, v090SourceBackend, v090SourceLocator, resourceLocationHash(v090SourceLocator)); err != nil {
|
||||
t.Fatalf("retarget source resource locator: %v", err)
|
||||
}
|
||||
|
||||
source, err := ExportLibrary(ctx, src, identity)
|
||||
if err != nil {
|
||||
t.Fatalf("export 0.9.0 library: %v", err)
|
||||
}
|
||||
if source.SchemaVersion != 134 || source.Profile != "v090" {
|
||||
t.Fatalf("snapshot must record the 0.9.0 profile, got %d/%q", source.SchemaVersion, source.Profile)
|
||||
}
|
||||
if tablePresent(source, "wiki_log_entries") {
|
||||
t.Fatal("0.9.0 export must not produce a wiki_log_entries table")
|
||||
}
|
||||
|
||||
token := make([]byte, 8)
|
||||
if _, err := rand.Read(token); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
index := "kg_v090_publish_" + hex.EncodeToString(token)
|
||||
client := &http.Client{Timeout: 30 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||
return http.ErrUseLastResponse
|
||||
}}
|
||||
call := func(method, path, body string, want int) {
|
||||
t.Helper()
|
||||
request, err := http.NewRequestWithContext(ctx, method, base+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
response, err := client.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("fixture HTTP: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != want {
|
||||
t.Fatalf("fixture %s %s => %d, want %d", method, path, response.StatusCode, want)
|
||||
}
|
||||
}
|
||||
call("PUT", "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"chunk_id":{"type":"keyword"},"content":{"type":"text"},"embedding":{"type":"dense_vector","dims":2}}}}`, 200)
|
||||
defer func() {
|
||||
call("DELETE", "/"+index, "", 200)
|
||||
call("HEAD", "/"+index, "", 404)
|
||||
}()
|
||||
exporter, err := NewSearchExporter(SearchExporterConfig{BaseURL: base, Index: index, PageSize: 100, HTTPClient: client})
|
||||
if err != nil {
|
||||
t.Fatalf("search exporter: %v", err)
|
||||
}
|
||||
mappingSHA, err := exporter.MappingSHA256(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("index mapping digest: %v", err)
|
||||
}
|
||||
|
||||
root := t.TempDir()
|
||||
if err := os.Chmod(root, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archive, err := librarysnapshot.Open(root, 64<<20)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
objectBytes := []byte("fixture-object-bytes")
|
||||
objects := v090ObjectBytes{"oss|" + v090SourceLocator: objectBytes}
|
||||
contract := librarysnapshot.Compatibility{
|
||||
WeKnoraVersion: "v0.9.0", SchemaVersion: 134,
|
||||
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "bge-m3",
|
||||
EmbeddingDimension: 1024, IndexMappingSHA256: mappingSHA,
|
||||
}
|
||||
sealed, err := ArchiveLibraryCandidate(ctx, archive, "internal", identity, contract, source,
|
||||
v090PublishIndexSnapshot(t, source, mappingSHA), objects, time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("seal 0.9.0 archive: %v", err)
|
||||
}
|
||||
manifest, err := archive.Load(sealed)
|
||||
if err != nil {
|
||||
t.Fatalf("load sealed manifest: %v", err)
|
||||
}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Name == "wiki_log_entries" {
|
||||
t.Fatal("0.9.0 archive must not contain a wiki_log_entries part")
|
||||
}
|
||||
}
|
||||
|
||||
// 目标库:干净 134 schema + KB/模型绑定 + 目标存储后端(桶名用于解析定位符)。
|
||||
if err := seedV090Target(t, ctx, dst); err != nil {
|
||||
t.Fatalf("seed target binding: %v", err)
|
||||
}
|
||||
store := newMemoryObjectStore(nil)
|
||||
target, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
|
||||
Postgres: os.Getenv("KG_V090_TARGET_DSN"), SearchURL: base, SearchIndex: index,
|
||||
Objects: store, ObjectsReader: store, ArchiveRoot: root,
|
||||
ObjectsBucket: v090TargetBucket, MaxDocuments: 100, HTTPClient: client,
|
||||
}, ProductionScope{TenantID: v090TargetTenant, KnowledgeBaseID: v090TargetKB})
|
||||
if err != nil {
|
||||
t.Fatalf("target: %v", err)
|
||||
}
|
||||
defer target.Close()
|
||||
|
||||
backups, err := NewProductionBackupStore(ProductionBackupConfig{
|
||||
Pool: target.Pool(), Search: target.Search(), Objects: store, ArchiveRoot: root,
|
||||
Scope: "internal", Identity: target.Identity(), MaxDocuments: 100, Compatibility: contract,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("backup store: %v", err)
|
||||
}
|
||||
|
||||
// 现状:真实 Current() 必须在 134 上可用,且不得去查 0.9.0 里不存在的 wiki_log_entries。
|
||||
before, err := target.Current(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("current target state on 0.9.0: %v", err)
|
||||
}
|
||||
if _, present := before.Counts["wiki_log_entries"]; present {
|
||||
t.Fatalf("0.9.0 target state must not include wiki_log_entries: %+v", before.Counts)
|
||||
}
|
||||
backup, err := backups.Prepare(ctx, sealed, before)
|
||||
if err != nil {
|
||||
t.Fatalf("prepare before-image: %v", err)
|
||||
}
|
||||
backup.WriteSet = planLedgerWithUniverse(t, target, sealed, backup.SnapshotID, backup.Universe)
|
||||
|
||||
for _, stage := range libraryrelease.Stages() {
|
||||
if err := target.Apply(ctx, stage, sealed); err != nil {
|
||||
t.Fatalf("apply %s on 0.9.0 target: %v", stage, err)
|
||||
}
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: sealed}); err != nil {
|
||||
t.Fatalf("verify after 0.9.0 publish: %v", err)
|
||||
}
|
||||
|
||||
assertV090Published(t, ctx, dst, target, source)
|
||||
|
||||
// 跨版本必须拒绝,并且错误同时点名两个版本号。用另一个没有冻结计划的目标实例:
|
||||
// 已接管写集合的执行目标会先被"快照与计划不符"挡住,那不是版本闸门。
|
||||
legacy := sealTargetFixture(t, archive, "internal", []string{"doc-safe"}, []string{"chunk-safe"}, "wiki-safe")
|
||||
legacyTarget, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
|
||||
Postgres: os.Getenv("KG_V090_TARGET_DSN"), SearchURL: base, SearchIndex: index,
|
||||
Objects: store, ObjectsReader: store, ArchiveRoot: root,
|
||||
ObjectsBucket: v090TargetBucket, MaxDocuments: 100, HTTPClient: client,
|
||||
}, ProductionScope{TenantID: v090TargetTenant, KnowledgeBaseID: v090TargetKB})
|
||||
if err != nil {
|
||||
t.Fatalf("legacy target: %v", err)
|
||||
}
|
||||
defer legacyTarget.Close()
|
||||
err = legacyTarget.Apply(ctx, libraryrelease.StageDatabase, legacy)
|
||||
if err == nil || !strings.Contains(err.Error(), "74") || !strings.Contains(err.Error(), "134") {
|
||||
t.Fatalf("cross-version publish must be refused naming both versions, got %v", err)
|
||||
}
|
||||
if err := legacyTarget.Verify(ctx, libraryrelease.Expectation{SnapshotID: legacy}); err == nil ||
|
||||
!strings.Contains(err.Error(), "74") || !strings.Contains(err.Error(), "134") {
|
||||
t.Fatalf("cross-version verify must be refused naming both versions, got %v", err)
|
||||
}
|
||||
|
||||
// 写前镜像恢复:目标必须回到写之前的样子(0.9.0 表集合)。
|
||||
if err := target.Restore(ctx, backup); err != nil {
|
||||
t.Fatalf("restore from before-image: %v", err)
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: backup.SnapshotID}); err != nil {
|
||||
t.Fatalf("verify after restore: %v", err)
|
||||
}
|
||||
for _, table := range []string{"knowledges", "chunks", "knowledge_tags", "knowledge_tag_relations",
|
||||
"wiki_pages", "wiki_folders", "wiki_page_issues"} {
|
||||
statement := `SELECT count(*) FROM ` + table + ` WHERE tenant_id=$1 AND knowledge_base_id=$2`
|
||||
args := []any{v090TargetTenant, v090TargetKB}
|
||||
if table == "knowledge_tag_relations" {
|
||||
// 关系表没有租户/知识库列:归属只能走父表。
|
||||
statement = `SELECT count(*) FROM knowledge_tag_relations r WHERE r.knowledge_id IN
|
||||
(SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2)`
|
||||
}
|
||||
var count int64
|
||||
if err := dst.QueryRow(ctx, statement, args...).Scan(&count); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if count != 0 {
|
||||
t.Fatalf("rollback left %d rows in %s", count, table)
|
||||
}
|
||||
}
|
||||
var restoredResource int64
|
||||
if err := dst.QueryRow(ctx, `SELECT count(*) FROM resources WHERE tenant_id=$1`, v090TargetTenant).Scan(&restoredResource); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if restoredResource != 0 {
|
||||
t.Fatalf("rollback left %d published resources in the target tenant", restoredResource)
|
||||
}
|
||||
t.Logf("v0.9.0 publish verified: snapshot=%s backup=%s bytes=%d", sealed, backup.SnapshotID, backup.Bytes)
|
||||
}
|
||||
|
||||
// assertV090Published 逐表核对目标与来源内容一致(身份列走目标隔离映射,不作为内容比对)。
|
||||
func assertV090Published(t *testing.T, ctx context.Context, dst *pgxpool.Pool, target *ProductionLibraryTarget, source PostgresSnapshot) {
|
||||
t.Helper()
|
||||
wantChunks := int64(source.Table("chunks").RowCount)
|
||||
var chunks int64
|
||||
if err := dst.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=$1 AND knowledge_base_id=$2`,
|
||||
v090TargetTenant, v090TargetKB).Scan(&chunks); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if chunks != wantChunks || chunks < 2 {
|
||||
t.Fatalf("chunks in target = %d, want %d", chunks, wantChunks)
|
||||
}
|
||||
// 目标隔离映射:写入的是目标身份,不是来源身份。
|
||||
for _, chunkID := range []string{v090ChunkOne, v090ChunkTwo} {
|
||||
var content string
|
||||
if err := dst.QueryRow(ctx, `SELECT content FROM chunks WHERE id=$1`,
|
||||
expectedTargetID(v090TargetTenant, v090TargetKB, "chunks", chunkID)).Scan(&content); err != nil {
|
||||
t.Fatalf("mapped chunk %s missing: %v", chunkID, err)
|
||||
}
|
||||
if content == "" {
|
||||
t.Fatalf("mapped chunk %s has empty content", chunkID)
|
||||
}
|
||||
}
|
||||
// 0.9.0 新增 NOT NULL 列走目标默认值;seq_id 由目标序列分配(来源值不得搬过来)。
|
||||
var defaults, assigned int64
|
||||
if err := dst.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=$1 AND knowledge_base_id=$2
|
||||
AND source_content IS NOT NULL AND content_revision IS NOT NULL AND index_status IS NOT NULL
|
||||
AND context_header IS NOT NULL AND last_editor_id IS NOT NULL AND seq_id IS NOT NULL`,
|
||||
v090TargetTenant, v090TargetKB).Scan(&defaults); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if defaults != wantChunks {
|
||||
t.Fatalf("0.9.0 NOT NULL columns unsatisfied on %d/%d chunks", defaults, wantChunks)
|
||||
}
|
||||
if err := dst.QueryRow(ctx, `SELECT count(DISTINCT seq_id) FROM chunks WHERE tenant_id=$1 AND knowledge_base_id=$2`,
|
||||
v090TargetTenant, v090TargetKB).Scan(&assigned); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if assigned != wantChunks {
|
||||
t.Fatalf("target seq_id must be assigned per chunk, got %d distinct for %d chunks", assigned, wantChunks)
|
||||
}
|
||||
|
||||
// Wiki 必须非空:空表通过等于没验证 wiki 部件。
|
||||
var pages int64
|
||||
var content, slug string
|
||||
if err := dst.QueryRow(ctx, `SELECT count(*) FROM wiki_pages WHERE tenant_id=$1 AND knowledge_base_id=$2`,
|
||||
v090TargetTenant, v090TargetKB).Scan(&pages); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pages != 1 {
|
||||
t.Fatalf("wiki_pages in target = %d, want 1", pages)
|
||||
}
|
||||
if err := dst.QueryRow(ctx, `SELECT content, slug FROM wiki_pages WHERE id=$1`,
|
||||
expectedTargetID(v090TargetTenant, v090TargetKB, "wiki_pages", v090Page)).Scan(&content, &slug); err != nil {
|
||||
t.Fatalf("published wiki page missing: %v", err)
|
||||
}
|
||||
if content != "wiki 正文" || slug != "fixture-page" {
|
||||
t.Fatalf("wiki page content changed: content=%q slug=%q", content, slug)
|
||||
}
|
||||
|
||||
var tags int64
|
||||
if err := dst.QueryRow(ctx, `SELECT count(*) FROM knowledge_tags WHERE tenant_id=$1 AND knowledge_base_id=$2 AND name='夹具标签'`,
|
||||
v090TargetTenant, v090TargetKB).Scan(&tags); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tags != 1 {
|
||||
t.Fatalf("knowledge_tags in target = %d, want 1", tags)
|
||||
}
|
||||
|
||||
var title, model string
|
||||
if err := dst.QueryRow(ctx, `SELECT title, embedding_model_id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2`,
|
||||
v090TargetTenant, v090TargetKB).Scan(&title, &model); err != nil {
|
||||
t.Fatalf("published knowledge missing: %v", err)
|
||||
}
|
||||
if title != "0.9.0 夹具" || model != "bge-m3-v090" {
|
||||
t.Fatalf("knowledge content changed: title=%q model=%q", title, model)
|
||||
}
|
||||
|
||||
// 资源必须指向目标桶/后端,且定位符哈希按最终路径重算。
|
||||
wantLocator := "storage://" + v090TargetBackend + "/minio://" + v090TargetBucket + "/" +
|
||||
strconv.FormatUint(v090TargetTenant, 10) + "/" + v090Resource + "/kn.bin"
|
||||
var provider, backend, locator, locationHash string
|
||||
var size int64
|
||||
if err := dst.QueryRow(ctx, `SELECT provider, coalesce(storage_backend_id,''), physical_path, location_hash, size
|
||||
FROM resources WHERE tenant_id=$1`, v090TargetTenant).
|
||||
Scan(&provider, &backend, &locator, &locationHash, &size); err != nil {
|
||||
t.Fatalf("published resource missing: %v", err)
|
||||
}
|
||||
if provider != "minio" || backend != v090TargetBackend {
|
||||
t.Fatalf("resource must point at the target backend: provider=%q backend=%q", provider, backend)
|
||||
}
|
||||
if locator != wantLocator {
|
||||
t.Fatalf("resource locator not retargeted:\n got %s\nwant %s", locator, wantLocator)
|
||||
}
|
||||
if locationHash != resourceLocationHash(wantLocator) {
|
||||
t.Fatalf("resource location_hash must hash the target locator")
|
||||
}
|
||||
if size != int64(len("fixture-object-bytes")) {
|
||||
t.Fatalf("resource size changed: %d", size)
|
||||
}
|
||||
// 目标对象存储里必须真的有这份字节(写入阶段按目标定位符上传)。
|
||||
if _, err := target.Objects().Open(ctx, provider, locator); err != nil {
|
||||
t.Fatalf("published object unreadable in target store: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// v090PublishIndexSnapshot 按导出分块合成索引快照:这批文档走的是与线上同一套
|
||||
// 映射/摘要校验,不能凭空绕过。
|
||||
func v090PublishIndexSnapshot(t *testing.T, pg PostgresSnapshot, mappingSHA string) SearchSnapshot {
|
||||
t.Helper()
|
||||
var documents []SearchDocument
|
||||
for _, raw := range pg.Table("chunks").Rows {
|
||||
var chunk struct {
|
||||
ID string `json:"id"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id"`
|
||||
KnowledgeID string `json:"knowledge_id"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &chunk); err != nil {
|
||||
t.Fatalf("decode chunk: %v", err)
|
||||
}
|
||||
source, err := json.Marshal(map[string]any{
|
||||
"chunk_id": chunk.ID, "tenant_id": chunk.TenantID,
|
||||
"knowledge_base_id": chunk.KnowledgeBaseID, "knowledge_id": chunk.KnowledgeID,
|
||||
"content": chunk.Content, "source_id": chunk.ID, "source_type": 0,
|
||||
"embedding": []float64{0.1, 0.2},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encode index document: %v", err)
|
||||
}
|
||||
documents = append(documents, SearchDocument{ID: "doc-" + chunk.ID, Source: source})
|
||||
}
|
||||
digest, err := searchDocumentsDigest(documents)
|
||||
if err != nil {
|
||||
t.Fatalf("index digest: %v", err)
|
||||
}
|
||||
return SearchSnapshot{Documents: documents, MappingSHA256: mappingSHA, DocumentsSHA256: digest}
|
||||
}
|
||||
|
||||
// seedV090Target 在目标库造 KB/模型绑定与目标存储后端:发布写的是内容行,不是 KB 定义。
|
||||
func seedV090Target(t *testing.T, ctx context.Context, db *pgxpool.Pool) error {
|
||||
t.Helper()
|
||||
for _, statement := range []string{
|
||||
`INSERT INTO models(id,tenant_id,name,type,source,parameters)
|
||||
VALUES ('bge-m3-v090',$1,'bge-m3','embedding','local','{"embedding_parameters":{"dimension":1024}}'::jsonb)`,
|
||||
`INSERT INTO knowledge_bases(id,name,tenant_id,embedding_model_id,summary_model_id,indexing_strategy)
|
||||
VALUES ('` + v090TargetKB + `','v090 publish target',$1,'bge-m3-v090','bge-m3-v090','{"graph_enabled":false}'::jsonb)`,
|
||||
`INSERT INTO storage_backends(id,tenant_id,name,provider,config)
|
||||
VALUES ('` + v090TargetBackend + `',$1,'v090 target','minio','{"bucket_name":"` + v090TargetBucket + `"}'::jsonb)`,
|
||||
} {
|
||||
if _, err := db.Exec(ctx, strings.ReplaceAll(statement, "$1", strconv.Itoa(v090TargetTenant))); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// v090MappingDigest 读真实抓下来的 0.9.0 映射(KG_V090_MAPPING_JSON)。
|
||||
func v090MappingDigest(t *testing.T) string {
|
||||
t.Helper()
|
||||
path := os.Getenv("KG_V090_MAPPING_JSON")
|
||||
if path == "" {
|
||||
t.Skip("requires KG_V090_MAPPING_JSON pointing at the captured 0.9.0 index mapping")
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read KG_V090_MAPPING_JSON: %v", err)
|
||||
}
|
||||
digest, err := IndexMappingDigest(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("mapping digest: %v", err)
|
||||
}
|
||||
return digest
|
||||
}
|
||||
|
||||
// TestV090LibraryRoundTripIsolated 走完整 0.9.0 往返:
|
||||
// 导出 → 封存(真实 manifest 校验)→ 导入干净 134 schema → 再导出逐行比对。
|
||||
func TestV090LibraryRoundTripIsolated(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
src := v090Pool(t, "KG_V090_SOURCE_DSN")
|
||||
identity := seedV090(t, ctx, src)
|
||||
mappingSHA := v090MappingDigest(t)
|
||||
|
||||
source, err := ExportLibrary(ctx, src, identity)
|
||||
if err != nil {
|
||||
t.Fatalf("export 0.9.0 library: %v", err)
|
||||
}
|
||||
if source.SchemaVersion != 134 || source.Profile != "v090" {
|
||||
t.Fatalf("snapshot must record the 0.9.0 profile, got %d/%q", source.SchemaVersion, source.Profile)
|
||||
}
|
||||
// 0.9.0 没有 wiki_log_entries:不允许出现该表,也不允许空 Wiki 冒充通过。
|
||||
if tablePresent(source, "wiki_log_entries") {
|
||||
t.Fatal("0.9.0 export must not produce a wiki_log_entries table")
|
||||
}
|
||||
if source.Table("wiki_pages").RowCount < 1 || source.Table("wiki_folders").RowCount < 1 {
|
||||
t.Fatalf("wiki parts must be non-empty: pages=%d folders=%d",
|
||||
source.Table("wiki_pages").RowCount, source.Table("wiki_folders").RowCount)
|
||||
}
|
||||
if source.Table("chunks").RowCount < 2 || source.Table("knowledges").RowCount != 1 {
|
||||
t.Fatalf("unexpected fixture shape: chunks=%d knowledges=%d",
|
||||
source.Table("chunks").RowCount, source.Table("knowledges").RowCount)
|
||||
}
|
||||
|
||||
es := v090IndexSnapshot(t, source, mappingSHA)
|
||||
root := t.TempDir()
|
||||
if err := os.Chmod(root, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archive, err := librarysnapshot.Open(root, 64<<20)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
contract := librarysnapshot.Compatibility{
|
||||
WeKnoraVersion: "v0.9.0", SchemaVersion: 134,
|
||||
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "bge-m3",
|
||||
EmbeddingDimension: 1024, IndexMappingSHA256: mappingSHA,
|
||||
}
|
||||
sealed, err := ArchiveLibraryCandidate(ctx, archive, "internal", identity, contract, source, es,
|
||||
v090Objects{"fixture/kn-v090.bin": []byte("fixture-object-bytes")}, time.Now().UTC())
|
||||
if err != nil {
|
||||
t.Fatalf("seal 0.9.0 archive: %v", err)
|
||||
}
|
||||
manifest, err := archive.Load(sealed)
|
||||
if err != nil {
|
||||
t.Fatalf("load sealed manifest: %v", err)
|
||||
}
|
||||
if manifest.Compatibility.SchemaVersion != 134 || manifest.Compatibility.WeKnoraVersion != "v0.9.0" {
|
||||
t.Fatalf("manifest must record the 0.9.0 profile, got %+v", manifest.Compatibility)
|
||||
}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind == "wiki" && part.Name == "wiki_log_entries" {
|
||||
t.Fatal("0.9.0 archive must not contain a wiki_log_entries part")
|
||||
}
|
||||
}
|
||||
|
||||
dst := v090Pool(t, "KG_V090_TARGET_DSN")
|
||||
if err := seedV090Binding(t, ctx, dst); err != nil {
|
||||
t.Fatalf("seed target binding: %v", err)
|
||||
}
|
||||
importer, err := NewPostgresImporter(dst)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
document := PostgresIdentity{TenantID: identity.TenantID, KnowledgeBaseID: identity.KnowledgeBaseID, KnowledgeID: v090Knowledge}
|
||||
if err := importer.Replace(ctx, document, source); err != nil {
|
||||
t.Fatalf("import 0.9.0 snapshot: %v", err)
|
||||
}
|
||||
reloaded, err := ExportLibrary(ctx, dst, identity)
|
||||
if err != nil {
|
||||
t.Fatalf("re-export imported library: %v", err)
|
||||
}
|
||||
compareSnapshotContent(t, source, reloaded)
|
||||
|
||||
// 0.9.0 新增的 NOT NULL 列在导入后必须仍是合法值,且 seq_id 由目标序列分配。
|
||||
var defaults int
|
||||
if err := dst.QueryRow(ctx, `SELECT count(*) FROM chunks
|
||||
WHERE source_content IS NOT NULL AND content_revision IS NOT NULL
|
||||
AND index_status IS NOT NULL AND context_header IS NOT NULL
|
||||
AND last_editor_id IS NOT NULL AND seq_id IS NOT NULL`).Scan(&defaults); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if defaults != source.Table("chunks").RowCount {
|
||||
t.Fatalf("0.9.0 NOT NULL columns must be satisfied on every imported chunk: got %d want %d",
|
||||
defaults, source.Table("chunks").RowCount)
|
||||
}
|
||||
var distinctSeq int
|
||||
if err := dst.QueryRow(ctx, `SELECT count(DISTINCT seq_id) FROM chunks`).Scan(&distinctSeq); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if distinctSeq != source.Table("chunks").RowCount {
|
||||
t.Fatalf("target chunk seq_id must be assigned, got %d distinct", distinctSeq)
|
||||
}
|
||||
|
||||
// 跨版本必须拒绝:0.7.1 归档不能落到 0.9.0 目标。
|
||||
crossVersion := source
|
||||
crossVersion.SchemaVersion = 74
|
||||
crossVersion.Profile = "v071"
|
||||
err = importer.Replace(ctx, document, crossVersion)
|
||||
if err == nil || !strings.Contains(err.Error(), "74") || !strings.Contains(err.Error(), "134") {
|
||||
t.Fatalf("cross-version import must be refused with both versions named, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// compareSnapshotContent 逐表逐行比对"共有列"的规范化内容。
|
||||
// chunks.seq_id 是库级序列值、不是文档内容,导入时由目标分配,故排除比对。
|
||||
func compareSnapshotContent(t *testing.T, source, target PostgresSnapshot) {
|
||||
t.Helper()
|
||||
if len(source.Tables) != len(target.Tables) {
|
||||
t.Fatalf("table set changed across round trip: %d -> %d", len(source.Tables), len(target.Tables))
|
||||
}
|
||||
for _, table := range source.Tables {
|
||||
want := tableDigests(t, table.Name, table.Rows)
|
||||
got := tableDigests(t, table.Name, target.Table(table.Name).Rows)
|
||||
if table.RowCount != target.Table(table.Name).RowCount || len(want) != len(got) {
|
||||
t.Fatalf("%s row count changed: %d -> %d", table.Name, len(want), len(got))
|
||||
}
|
||||
for index := range want {
|
||||
if want[index] != got[index] {
|
||||
t.Fatalf("%s row %d content changed across round trip", table.Name, index)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func tableDigests(t *testing.T, name string, rows []json.RawMessage) []string {
|
||||
t.Helper()
|
||||
digests := make([]string, 0, len(rows))
|
||||
for _, raw := range rows {
|
||||
var row map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &row); err != nil {
|
||||
t.Fatalf("decode %s row: %v", name, err)
|
||||
}
|
||||
if name == "chunks" {
|
||||
delete(row, "seq_id")
|
||||
}
|
||||
digest, err := canonicalRowDigest(row)
|
||||
if err != nil {
|
||||
t.Fatalf("digest %s row: %v", name, err)
|
||||
}
|
||||
digests = append(digests, digest)
|
||||
}
|
||||
sort.Strings(digests)
|
||||
return digests
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
@@ -118,7 +119,12 @@ func libraryResourceHandles(rows []json.RawMessage) ([]string, error) {
|
||||
// Revalidate frozen source content at the actual planning boundary. A sealed
|
||||
// manifest proves bytes, not that every row has a parent or a supported owner.
|
||||
func validateArchivedLibraryClosure(archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error {
|
||||
var pg PostgresSnapshot
|
||||
// 快照的 schema 契约来自 manifest:不带上版本号,重建出来的快照就是"版本未知",
|
||||
// 而未知版本按 0.7.1 校验——0.9.0 的归档会被要求提供它根本没有的 wiki_log_entries。
|
||||
pg := PostgresSnapshot{SchemaVersion: manifest.Compatibility.SchemaVersion}
|
||||
if profile, err := libraryprofile.For(pg.SchemaVersion); err == nil {
|
||||
pg.Profile = profile.Name
|
||||
}
|
||||
es := SearchSnapshot{MappingSHA256: manifest.Compatibility.IndexMappingSHA256}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" && part.Kind != "index" {
|
||||
|
||||
@@ -152,8 +152,8 @@ func (t *ProductionLibraryTarget) ClearForeignScope(ctx context.Context, source
|
||||
return report, nil
|
||||
}
|
||||
// 删除顺序:子表先、父表后(resource_bindings → resources,chunks → knowledges)。
|
||||
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
|
||||
table := productionLibraryTables[index]
|
||||
for index := len(t.tables) - 1; index >= 0; index-- {
|
||||
table := t.tables[index]
|
||||
ids, ok := identities[table]
|
||||
if !ok || len(ids) == 0 {
|
||||
continue
|
||||
@@ -216,7 +216,7 @@ func (t *ProductionLibraryTarget) legacyResourceRow(row map[string]json.RawMessa
|
||||
// VerifyResidueCleared 复核清理结果:来源身份不再存在、其它租户的解释得清、
|
||||
// 且(可选)目标内容与预备份完全一致。
|
||||
func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, source ProductionScope) error {
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range t.tables {
|
||||
if !tableHasTenant(table) {
|
||||
continue
|
||||
}
|
||||
@@ -240,7 +240,7 @@ func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, sour
|
||||
// ForeignScopes 列出目标里所有"不是目标租户"的租户,用于人工确认没有别的泄漏。
|
||||
func (t *ProductionLibraryTarget) ForeignScopes(ctx context.Context) (map[string]string, error) {
|
||||
foreign := map[string]string{}
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range t.tables {
|
||||
if !tableHasTenant(table) {
|
||||
continue
|
||||
}
|
||||
@@ -272,7 +272,7 @@ func (t *ProductionLibraryTarget) foreignScopeRows(ctx context.Context, table st
|
||||
if !tableHasTenant(table) {
|
||||
return nil, nil
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table))
|
||||
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, t.scopeWhere(table))
|
||||
arguments := []any{source.TenantID, source.KnowledgeBaseID}
|
||||
if table == "resources" || table == "resource_bindings" {
|
||||
statement = fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE tenant_id=$1`, table, table)
|
||||
|
||||
@@ -10,10 +10,11 @@ import (
|
||||
)
|
||||
|
||||
// Keep capture/verification and deletion protection on the same set of bodies.
|
||||
// predicate is an internal SQL fragment, never caller-supplied content.
|
||||
func libraryResourceBodyReferenceSQL(predicate string) string {
|
||||
// predicate is an internal SQL fragment, never caller-supplied content; tables comes
|
||||
// from the resolved schema profile (0.9.0 has no wiki_log_entries to scan).
|
||||
func libraryResourceBodyReferenceSQL(tables []string, predicate string) string {
|
||||
queries := []string{}
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range tables {
|
||||
if table == "resources" || table == "resource_bindings" || table == "knowledge_tag_relations" {
|
||||
continue
|
||||
}
|
||||
@@ -24,7 +25,7 @@ func libraryResourceBodyReferenceSQL(predicate string) string {
|
||||
|
||||
// This must precede *all* restore mutations, including ES and object deletes.
|
||||
// A created row is not permission to invalidate a later out-of-plan reader.
|
||||
func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureReader, set libraryrelease.WriteSet) error {
|
||||
func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureReader, tables []string, set libraryrelease.WriteSet) error {
|
||||
ids := set.CreatedIDs("resources")
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
@@ -49,7 +50,7 @@ func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureRea
|
||||
if referenced {
|
||||
return errors.New("created resource has out-of-plan bindings; restore held before writes")
|
||||
}
|
||||
bodySQL := libraryResourceBodyReferenceSQL("c.tenant_id IS DISTINCT FROM $2 OR c.knowledge_base_id IS DISTINCT FROM $3")
|
||||
bodySQL := libraryResourceBodyReferenceSQL(tables, "c.tenant_id IS DISTINCT FROM $2 OR c.knowledge_base_id IS DISTINCT FROM $3")
|
||||
err = q.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM resources WHERE id=ANY($1::text[]) AND EXISTS (`+bodySQL+`))`, ids, set.TenantID, set.KnowledgeBaseID).Scan(&referenced)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -62,10 +63,10 @@ func ensureRestorableResourceReferences(ctx context.Context, q libraryClosureRea
|
||||
|
||||
// Recheck under the already-held resource parent locks. This does not claim
|
||||
// cross-store isolation from uncontrolled writers; deployed exclusivity is separate.
|
||||
func ensureCreatedResourcesUnreferenced(ctx context.Context, q libraryClosureReader, ids []string) error {
|
||||
func ensureCreatedResourcesUnreferenced(ctx context.Context, q libraryClosureReader, tables, ids []string) error {
|
||||
var referenced bool
|
||||
err := q.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM resources WHERE id=ANY($1::text[]) AND
|
||||
(EXISTS(SELECT 1 FROM resource_bindings b WHERE b.resource_id=resources.id) OR EXISTS (`+libraryResourceBodyReferenceSQL("TRUE")+`)))`, ids).Scan(&referenced)
|
||||
(EXISTS(SELECT 1 FROM resource_bindings b WHERE b.resource_id=resources.id) OR EXISTS (`+libraryResourceBodyReferenceSQL(tables, "TRUE")+`)))`, ids).Scan(&referenced)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"fmt"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
@@ -22,7 +23,7 @@ func (t *ProductionLibraryTarget) migrationCatalog(ctx context.Context) (migrati
|
||||
return nil, "", err
|
||||
}
|
||||
defer tx.Rollback(context.WithoutCancel(ctx))
|
||||
catalog, digest, err := captureMigrationCatalog(ctx, tx)
|
||||
catalog, digest, err := captureMigrationCatalogFor(ctx, tx, t.tables)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
@@ -32,8 +33,15 @@ func (t *ProductionLibraryTarget) migrationCatalog(ctx context.Context) (migrati
|
||||
return catalog, digest, nil
|
||||
}
|
||||
|
||||
// Source rows and their definitions must come from the same read-only transaction.
|
||||
// captureMigrationCatalog 是 0.7.1(既有表集合)的入口,行为保持不变。
|
||||
func captureMigrationCatalog(ctx context.Context, tx pgx.Tx) (migrationCatalog, string, error) {
|
||||
return captureMigrationCatalogFor(ctx, tx, productionLibraryTables)
|
||||
}
|
||||
|
||||
// captureMigrationCatalogFor 按 profile 解析出的表集合取定义:0.9.0 没有
|
||||
// wiki_log_entries,用固定清单会让整库编目在第一张不存在的表上失败。
|
||||
// Source rows and their definitions must come from the same read-only transaction.
|
||||
func captureMigrationCatalogFor(ctx context.Context, tx pgx.Tx, tables []string) (migrationCatalog, string, error) {
|
||||
queries := map[string]string{
|
||||
"columns": `SELECT c.relname AS table_name,a.attname AS column_name,format_type(a.atttypid,a.atttypmod) AS type,a.attnotnull AS not_null,a.attidentity AS identity,a.attgenerated AS generated,coalesce(pg_get_expr(d.adbin,d.adrelid),'') AS default_expr FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid JOIN pg_namespace n ON n.oid=c.relnamespace LEFT JOIN pg_attrdef d ON d.adrelid=c.oid AND d.adnum=a.attnum WHERE n.nspname='public' AND c.relname=ANY($1) AND a.attnum>0 AND NOT a.attisdropped ORDER BY c.relname,a.attnum`,
|
||||
"constraints": `SELECT ns.nspname AS namespace,c.relname AS table_name,x.conname AS name,x.contype AS type,pg_get_constraintdef(x.oid) AS definition,coalesce(r.relname,'') AS referenced_table,coalesce(rns.nspname,'') AS referenced_namespace FROM pg_constraint x JOIN pg_class c ON c.oid=x.conrelid JOIN pg_namespace ns ON ns.oid=c.relnamespace LEFT JOIN pg_class r ON r.oid=x.confrelid LEFT JOIN pg_namespace rns ON rns.oid=r.relnamespace WHERE (ns.nspname='public' AND c.relname=ANY($1)) OR (rns.nspname='public' AND r.relname=ANY($1)) ORDER BY ns.nspname,c.relname,x.conname`,
|
||||
@@ -45,7 +53,7 @@ func captureMigrationCatalog(ctx context.Context, tx pgx.Tx) (migrationCatalog,
|
||||
catalog := migrationCatalog{}
|
||||
for key, query := range queries {
|
||||
var raw []byte
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(jsonb_agg(to_jsonb(q)),'[]'::jsonb) FROM ("+query+") q", productionLibraryTables).Scan(&raw); err != nil {
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(jsonb_agg(to_jsonb(q)),'[]'::jsonb) FROM ("+query+") q", tables).Scan(&raw); err != nil {
|
||||
return nil, "", fmt.Errorf("capture migration catalog %s: %w", key, err)
|
||||
}
|
||||
catalog[key] = raw
|
||||
@@ -59,12 +67,39 @@ func captureMigrationCatalog(ctx context.Context, tx pgx.Tx) (migrationCatalog,
|
||||
}
|
||||
|
||||
func validateMigrationCatalog(catalog migrationCatalog) error {
|
||||
var triggers []json.RawMessage
|
||||
profile, err := libraryprofile.For(libraryprofile.LegacyVersion)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return validateMigrationCatalogFor(catalog, productionLibraryTables, profile)
|
||||
}
|
||||
|
||||
func validateMigrationCatalogFor(catalog migrationCatalog, tables []string, profile libraryprofile.Profile) error {
|
||||
var triggers []struct {
|
||||
Table string `json:"table_name"`
|
||||
Name string `json:"name"`
|
||||
Definition string `json:"definition"`
|
||||
}
|
||||
if err := json.Unmarshal(catalog["triggers"], &triggers); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(triggers) > 0 {
|
||||
return errors.New("unreviewed migration trigger effects")
|
||||
// 触发器 = 写一行还会改别的表。只接受 profile 里逐字核对过的定义,
|
||||
// 且它们必须真实存在:消失了说明这也不是我们审过的那份 schema。
|
||||
reviewed := map[string]bool{}
|
||||
for _, definition := range profile.ReviewedTriggers {
|
||||
reviewed[definition] = true
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, trigger := range triggers {
|
||||
if !reviewed[trigger.Definition] {
|
||||
return fmt.Errorf("unreviewed migration trigger %s on %s", trigger.Name, trigger.Table)
|
||||
}
|
||||
seen[trigger.Definition] = true
|
||||
}
|
||||
for definition := range reviewed {
|
||||
if !seen[definition] {
|
||||
return errors.New("reviewed migration trigger is missing from this schema version")
|
||||
}
|
||||
}
|
||||
var relations []struct {
|
||||
Table string `json:"table_name"`
|
||||
@@ -76,7 +111,7 @@ func validateMigrationCatalog(catalog migrationCatalog) error {
|
||||
if err := json.Unmarshal(catalog["relations"], &relations); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(relations) != len(productionLibraryTables) {
|
||||
if len(relations) != len(tables) {
|
||||
return errors.New("incomplete migration table catalog")
|
||||
}
|
||||
for _, r := range relations {
|
||||
@@ -115,7 +150,7 @@ func (t *ProductionLibraryTarget) preflightMigrationCatalog(ctx context.Context,
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := validateMigrationCatalog(catalog); err != nil {
|
||||
if err := validateMigrationCatalogFor(catalog, t.tables, t.schema.Profile); err != nil {
|
||||
return "", err
|
||||
}
|
||||
var columns []struct {
|
||||
|
||||
@@ -17,8 +17,12 @@ func ScopeContentCounts(ctx context.Context, pool *pgxpool.Pool, identity Produc
|
||||
if pool == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) {
|
||||
return nil, errors.New("invalid scope identity")
|
||||
}
|
||||
counts := make(map[string]int64, len(productionLibraryTables))
|
||||
for _, table := range productionLibraryTables {
|
||||
profile, err := ResolveSchemaProfile(ctx, pool, "scope content probe")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
counts := make(map[string]int64, len(profile.Tables))
|
||||
for _, table := range profile.Tables {
|
||||
var count int64
|
||||
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`, table, scopeWhere(table)),
|
||||
identity.TenantID, identity.KnowledgeBaseID).Scan(&count); err != nil {
|
||||
|
||||
@@ -24,8 +24,14 @@ import (
|
||||
// 写坏"(资源表的 ON CONFLICT 就会这样)在指纹上完全看不出来——数量、身份都一样。
|
||||
// 因此摘要取自每行 to_jsonb 的 md5,再按身份排序聚合。
|
||||
func universeFingerprint(ctx context.Context, pool *pgxpool.Pool) (map[string]libraryrelease.TableFingerprint, error) {
|
||||
result := make(map[string]libraryrelease.TableFingerprint, len(productionLibraryTables))
|
||||
for _, table := range productionLibraryTables {
|
||||
// 表集合来自目标库自己的 profile:0.9.0 没有 wiki_log_entries,按 0.7.1 的固定
|
||||
// 清单取指纹会在第一张不存在的表上失败,而"跳过缺表"又会让 0.7.1 少表蒙混过去。
|
||||
schema, err := ResolveSchemaProfile(ctx, pool, "whole-instance fingerprint")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result := make(map[string]libraryrelease.TableFingerprint, len(schema.Tables))
|
||||
for _, table := range schema.Tables {
|
||||
identity := identityExpr(table)
|
||||
statement := fmt.Sprintf(
|
||||
`SELECT count(*), coalesce(md5(string_agg(md5(to_jsonb(x)::text), E'\n' ORDER BY (%s)::text)), '') FROM %s AS x`,
|
||||
|
||||
@@ -76,12 +76,12 @@ func (t *ProductionLibraryTarget) productionTableContent(ctx context.Context, ta
|
||||
// the legacy per-resource body scan made post-sync verification minutes
|
||||
// long on a small target inside a large tenant.
|
||||
var bodies, bindings []json.RawMessage
|
||||
for _, name := range productionLibraryTables {
|
||||
for _, name := range t.tables {
|
||||
if name == "resources" {
|
||||
continue
|
||||
}
|
||||
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`,
|
||||
name, name, scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
name, name, t.scopeWhere(name)), identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return 0, "", fmt.Errorf("读取生产 %s 内容失败: %w", table, err)
|
||||
}
|
||||
@@ -98,7 +98,7 @@ func (t *ProductionLibraryTarget) productionTableContent(ctx context.Context, ta
|
||||
}
|
||||
rows = items
|
||||
} else {
|
||||
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table)),
|
||||
items, err := exporter.queryRows(ctx, fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, t.scopeWhere(table)),
|
||||
identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return 0, "", fmt.Errorf("读取生产 %s 内容失败: %w", table, err)
|
||||
|
||||
@@ -401,7 +401,7 @@ func (t *ProductionLibraryTarget) clearJobCreatedRows(ctx context.Context, set l
|
||||
return err
|
||||
}
|
||||
|
||||
if err := ensureCreatedResourcesUnreferenced(ctx, tx, ids); err != nil {
|
||||
if err := ensureCreatedResourcesUnreferenced(ctx, tx, t.tables, ids); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -412,7 +412,7 @@ func (t *ProductionLibraryTarget) clearJobCreatedRows(ctx context.Context, set l
|
||||
// 复核:本次新建的行一个都不许剩。资源若因"被引用"而保留,那说明恢复没回到
|
||||
// 预备份状态,必须报出来让维护保持,而不是静默放过。
|
||||
leftovers := make([]string, 0, 4)
|
||||
for _, table := range productionLibraryTables {
|
||||
for _, table := range t.tables {
|
||||
ids := make([]string, 0, 8)
|
||||
for _, id := range set.CreatedIDs(table) {
|
||||
if restored[table][id] {
|
||||
|
||||
@@ -10,6 +10,8 @@ import (
|
||||
"sort"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
)
|
||||
|
||||
var resourceReferencePattern = regexp.MustCompile(`resource://([A-Za-z0-9_-]{22})`)
|
||||
@@ -30,6 +32,29 @@ type TableSnapshot struct {
|
||||
type PostgresSnapshot struct {
|
||||
Catalog migrationCatalog `json:"catalog,omitempty"`
|
||||
Tables []TableSnapshot
|
||||
// SchemaVersion 是导出/导入两侧共同的 profile 标识:归档必须与目标同版本,
|
||||
// 否则行里的列语义(例如 0.9.0 新增的 NOT NULL 列)不可比。
|
||||
SchemaVersion int `json:"schema_version,omitempty"`
|
||||
Profile string `json:"profile,omitempty"`
|
||||
}
|
||||
|
||||
// schemaProfile 解析快照声明的 profile。0 表示既有内存夹具(0.7.1),
|
||||
// 只把"没有声明版本"解释成 74,不把未知版本解释成任何 profile。
|
||||
func (snapshot PostgresSnapshot) schemaProfile() (SchemaProfile, error) {
|
||||
profile, err := libraryprofile.For(snapshot.SchemaVersion)
|
||||
if err != nil {
|
||||
if snapshot.SchemaVersion != 0 {
|
||||
return SchemaProfile{}, err
|
||||
}
|
||||
profile, err = libraryprofile.For(libraryprofile.LegacyVersion)
|
||||
if err != nil {
|
||||
return SchemaProfile{}, err
|
||||
}
|
||||
}
|
||||
resolved := SchemaProfile{Profile: profile}
|
||||
resolved.Tables = append(resolved.Tables, profile.RequiredTables...)
|
||||
resolved.Tables = append(resolved.Tables, profile.OptionalTables...)
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func (snapshot PostgresSnapshot) Table(name string) TableSnapshot {
|
||||
|
||||
@@ -67,10 +67,13 @@ func (importer *PostgresImporter) Replace(ctx context.Context, identity Postgres
|
||||
return errors.New("begin PostgreSQL document import")
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := tx.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
|
||||
return errors.New("target WeKnora schema contract mismatch")
|
||||
profile, err := ResolveSchemaProfile(ctx, tx, "target WeKnora")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// 目标版本必须与归档同 profile:0.7.1 的归档不能写进 0.9.0,反之亦然。
|
||||
if err := requireSameProfile(snapshot.SchemaVersion, profile, "import"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, identity.KnowledgeID); err != nil {
|
||||
return errors.New("lock target knowledge")
|
||||
@@ -177,10 +180,12 @@ func (importer *PostgresImporter) ReplaceWiki(ctx context.Context, identity Post
|
||||
return errors.New("begin PostgreSQL wiki import")
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := tx.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
|
||||
return errors.New("target WeKnora schema contract mismatch")
|
||||
profile, err := ResolveSchemaProfile(ctx, tx, "target WeKnora")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := requireSameProfile(snapshot.SchemaVersion, profile, "wiki import"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 1))`, identity.KnowledgeBaseID); err != nil {
|
||||
return errors.New("lock target wiki knowledge base")
|
||||
@@ -205,10 +210,8 @@ func (importer *PostgresImporter) Disable(ctx context.Context, identity Postgres
|
||||
return errors.New("begin PostgreSQL disable")
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := tx.QueryRow(ctx, `SELECT version, dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
|
||||
return errors.New("target WeKnora schema contract mismatch")
|
||||
if _, err := ResolveSchemaProfile(ctx, tx, "target WeKnora"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1, 0))`, identity.KnowledgeID); err != nil {
|
||||
return errors.New("lock target knowledge")
|
||||
@@ -244,9 +247,15 @@ func (importer *PostgresImporter) prepareSnapshot(identity PostgresIdentity, sna
|
||||
return PostgresSnapshot{}, nil, nil, errors.New("PostgreSQL import snapshot is incomplete")
|
||||
}
|
||||
prepared := cloneSnapshot(snapshot)
|
||||
// 表集合按快照所属 profile 判定:0.9.0 的归档里有 wiki_page_issues,
|
||||
// 面向单文档发布的白名单没有它,用白名单会把合法归档判成非法。
|
||||
profile, err := snapshot.schemaProfile()
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, nil, nil, err
|
||||
}
|
||||
known := map[string]bool{}
|
||||
for index, table := range prepared.Tables {
|
||||
if !validReleaseTable(table.Name) || known[table.Name] || table.RowCount != len(table.Rows) || newTableSnapshot(table.Name, table.Rows).SHA256 != table.SHA256 {
|
||||
if !profile.HasTable(table.Name) || known[table.Name] || table.RowCount != len(table.Rows) || newTableSnapshot(table.Name, table.Rows).SHA256 != table.SHA256 {
|
||||
return PostgresSnapshot{}, nil, nil, errors.New("PostgreSQL import snapshot is invalid")
|
||||
}
|
||||
known[table.Name] = true
|
||||
@@ -516,7 +525,9 @@ func validObjectLocator(value string) bool {
|
||||
}
|
||||
|
||||
func cloneSnapshot(snapshot PostgresSnapshot) PostgresSnapshot {
|
||||
result := PostgresSnapshot{Tables: make([]TableSnapshot, len(snapshot.Tables))}
|
||||
// 复制快照不得改契约:SchemaVersion/Profile 丢掉会让下游按未知版本(=0.7.1)处理。
|
||||
result := PostgresSnapshot{Tables: make([]TableSnapshot, len(snapshot.Tables)),
|
||||
SchemaVersion: snapshot.SchemaVersion, Profile: snapshot.Profile}
|
||||
for index, table := range snapshot.Tables {
|
||||
result.Tables[index] = table
|
||||
result.Tables[index].Rows = append([]json.RawMessage(nil), table.Rows...)
|
||||
|
||||
@@ -135,5 +135,8 @@ func replaceTable(snapshot PostgresSnapshot, replacement TableSnapshot) Postgres
|
||||
if !replaced {
|
||||
tables = append(tables, replacement)
|
||||
}
|
||||
return PostgresSnapshot{Tables: tables}
|
||||
// SchemaVersion/Profile 描述的是这份快照的 schema 契约,不是表内容:重建快照时
|
||||
// 丢掉它们,下游读到的就是"版本未知",而未知版本按 0.7.1 处理——134 的归档
|
||||
// 于是被当成 74 搬运,跨版本闸门形同不存在。
|
||||
return PostgresSnapshot{Tables: tables, SchemaVersion: snapshot.SchemaVersion, Profile: snapshot.Profile}
|
||||
}
|
||||
|
||||
@@ -173,3 +173,26 @@ func TestPlanItemTargetBaseFallsBackToTestedBase(t *testing.T) {
|
||||
t.Fatalf("unbound TargetBase = %q, want the tested base", unbound.TargetBase())
|
||||
}
|
||||
}
|
||||
|
||||
// 范围改写只换身份,不改 schema 契约:改写后的快照若丢掉 SchemaVersion/Profile,
|
||||
// 下游读到的就是"版本未知",而未知版本会被当作 0.7.1 处理——134 的归档于是被当成
|
||||
// 74 搬,跨版本闸门形同不存在。
|
||||
func TestRemapKnowledgeBasePreservesSchemaProfile(t *testing.T) {
|
||||
bundle := crossBaseBundle(t)
|
||||
bundle.Postgres.SchemaVersion = 134
|
||||
bundle.Postgres.Profile = "v090"
|
||||
if err := bundle.RemapKnowledgeBase("prod-kb"); err != nil {
|
||||
t.Fatalf("remap: %v", err)
|
||||
}
|
||||
if bundle.Postgres.SchemaVersion != 134 || bundle.Postgres.Profile != "v090" {
|
||||
t.Fatalf("remapped snapshot lost its schema profile: %d/%q",
|
||||
bundle.Postgres.SchemaVersion, bundle.Postgres.Profile)
|
||||
}
|
||||
tables := map[string]bool{}
|
||||
for _, table := range bundle.Postgres.Tables {
|
||||
tables[table.Name] = true
|
||||
}
|
||||
if len(tables) != len(bundle.Postgres.Tables) {
|
||||
t.Fatal("remap duplicated a table")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
// schemaProbeRows 给"按 SQL 打桩"的既有夹具提供 information_schema 能力探测答案。
|
||||
// 0.7.1 的假库一律按 profile 声明的表/列回答,这样能力探测在单测里也是真的被走了一遍,
|
||||
// 而不是被桩悄悄放行。
|
||||
type schemaProbeRows struct {
|
||||
pairs [][2]string
|
||||
index int
|
||||
}
|
||||
|
||||
func legacySchemaProbeRows() *schemaProbeRows {
|
||||
profile, err := libraryprofile.For(libraryprofile.LegacyVersion)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
rows := &schemaProbeRows{}
|
||||
for _, table := range profile.RequiredTables {
|
||||
for _, column := range profile.RequiredColumns[table] {
|
||||
rows.pairs = append(rows.pairs, [2]string{table, column})
|
||||
}
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func (r *schemaProbeRows) Close() {}
|
||||
func (r *schemaProbeRows) Err() error { return nil }
|
||||
func (r *schemaProbeRows) CommandTag() pgconn.CommandTag { return pgconn.CommandTag{} }
|
||||
func (r *schemaProbeRows) FieldDescriptions() []pgconn.FieldDescription { return nil }
|
||||
func (r *schemaProbeRows) RawValues() [][]byte { return nil }
|
||||
func (r *schemaProbeRows) Conn() *pgx.Conn { return nil }
|
||||
func (r *schemaProbeRows) Next() bool {
|
||||
if r.index >= len(r.pairs) {
|
||||
return false
|
||||
}
|
||||
r.index++
|
||||
return true
|
||||
}
|
||||
func (r *schemaProbeRows) Scan(dest ...any) error {
|
||||
*dest[0].(*string) = r.pairs[r.index-1][0]
|
||||
*dest[1].(*string) = r.pairs[r.index-1][1]
|
||||
return nil
|
||||
}
|
||||
func (r *schemaProbeRows) Values() ([]any, error) { return nil, nil }
|
||||
@@ -0,0 +1,145 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
)
|
||||
|
||||
// SchemaProfile 是"版本号 + 能力探测"两者都通过之后的解析结果。
|
||||
//
|
||||
// Tables 是**实际存在**的表(必需表按声明顺序 + 存在的可选表)。导出/编目/校验
|
||||
// 只按它取数:0.9.0 没有 wiki_log_entries,硬编码表清单会在第一条查询上炸,
|
||||
// 而"表不存在就跳过约束"会让 0.7.1 少一个必需表也悄悄通过。
|
||||
type SchemaProfile struct {
|
||||
libraryprofile.Profile
|
||||
Tables []string
|
||||
}
|
||||
|
||||
// HasTable 报告解析后的表集合里是否包含该表。
|
||||
func (p SchemaProfile) HasTable(name string) bool {
|
||||
for _, candidate := range p.Tables {
|
||||
if candidate == name {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// CatalogTables 是编目/归档用的表清单(与 Tables 同序,sorted 便于稳定摘要)。
|
||||
func (p SchemaProfile) CatalogTables() []string {
|
||||
tables := append([]string(nil), p.Tables...)
|
||||
sort.Strings(tables)
|
||||
return tables
|
||||
}
|
||||
|
||||
type schemaProbeDB interface {
|
||||
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
|
||||
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
|
||||
}
|
||||
|
||||
// ResolveSchemaProfile 解析并校验一侧的 schema 契约。
|
||||
//
|
||||
// 顺序是刻意的:先读版本(未知版本失败关闭、dirty 一律失败),再用
|
||||
// information_schema 证明 profile 声明的东西真的存在。只信版本号等于把
|
||||
// "迁移进度"当成"能力契约",而两者在生产上经常不同步。
|
||||
func ResolveSchemaProfile(ctx context.Context, db schemaProbeDB, role string) (SchemaProfile, error) {
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := db.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations ORDER BY version DESC LIMIT 1`).Scan(&version, &dirty); err != nil {
|
||||
return SchemaProfile{}, fmt.Errorf("read %s schema_migrations: %w", role, err)
|
||||
}
|
||||
profile, err := profileForMigration(version, dirty, role)
|
||||
if err != nil {
|
||||
return SchemaProfile{}, err
|
||||
}
|
||||
columns, err := probeSchemaColumns(ctx, db, profile)
|
||||
if err != nil {
|
||||
return SchemaProfile{}, fmt.Errorf("%s schema version %d: %w", role, version, err)
|
||||
}
|
||||
return verifySchemaCapabilities(profile, columns, role, version)
|
||||
}
|
||||
|
||||
// profileForMigration 只做"版本号 + dirty → profile"的判定,不碰数据库。
|
||||
// dirty 与未知版本都必须失败:前者是半截迁移,后者是没见过的契约。
|
||||
func profileForMigration(version int, dirty bool, role string) (libraryprofile.Profile, error) {
|
||||
if dirty {
|
||||
return libraryprofile.Profile{}, fmt.Errorf("%s schema is dirty at version %d: refusing to snapshot or write", role, version)
|
||||
}
|
||||
profile, err := libraryprofile.For(version)
|
||||
if err != nil {
|
||||
return libraryprofile.Profile{}, fmt.Errorf("%s schema: %w", role, err)
|
||||
}
|
||||
return profile, nil
|
||||
}
|
||||
|
||||
// verifySchemaCapabilities 用 information_schema 的实际列集合校正 profile:
|
||||
// 必需表/列缺失就精确点名,存在才进"实际表集合"。
|
||||
func verifySchemaCapabilities(profile libraryprofile.Profile, columns map[string]map[string]bool, role string, version int) (SchemaProfile, error) {
|
||||
resolved := SchemaProfile{Profile: profile}
|
||||
for _, table := range profile.RequiredTables {
|
||||
if _, ok := columns[table]; !ok {
|
||||
return SchemaProfile{}, fmt.Errorf("%s schema version %d: required table %s is missing", role, version, table)
|
||||
}
|
||||
resolved.Tables = append(resolved.Tables, table)
|
||||
}
|
||||
for _, table := range profile.OptionalTables {
|
||||
if _, ok := columns[table]; ok {
|
||||
resolved.Tables = append(resolved.Tables, table)
|
||||
}
|
||||
}
|
||||
for _, table := range resolved.Tables {
|
||||
for _, column := range profile.RequiredColumns[table] {
|
||||
if !columns[table][column] {
|
||||
return SchemaProfile{}, fmt.Errorf("%s schema version %d: required column %s.%s is missing", role, version, table, column)
|
||||
}
|
||||
}
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
// probeSchemaColumns 一次读回 profile 涉及表的列集合。
|
||||
func probeSchemaColumns(ctx context.Context, db schemaProbeDB, profile libraryprofile.Profile) (map[string]map[string]bool, error) {
|
||||
names := make([]string, 0, len(profile.RequiredTables)+len(profile.OptionalTables))
|
||||
names = append(names, profile.RequiredTables...)
|
||||
names = append(names, profile.OptionalTables...)
|
||||
sort.Strings(names)
|
||||
rows, err := db.Query(ctx, `SELECT table_name, column_name FROM information_schema.columns WHERE table_schema='public' AND table_name=ANY($1)`, names)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("probe schema capabilities: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
columns := map[string]map[string]bool{}
|
||||
for rows.Next() {
|
||||
var table, column string
|
||||
if err := rows.Scan(&table, &column); err != nil {
|
||||
return nil, fmt.Errorf("probe schema capabilities: %w", err)
|
||||
}
|
||||
if columns[table] == nil {
|
||||
columns[table] = map[string]bool{}
|
||||
}
|
||||
columns[table][column] = true
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("probe schema capabilities: %w", err)
|
||||
}
|
||||
return columns, nil
|
||||
}
|
||||
|
||||
// requireSameProfile 拒绝跨版本搬运:0.9.0 归档不能落到 0.7.1,反之也不行。
|
||||
// 本轮只支持 090↔090(以及既有 74↔74)。
|
||||
func requireSameProfile(sourceVersion int, target SchemaProfile, direction string) error {
|
||||
if sourceVersion == 0 {
|
||||
// 老的内存夹具没有声明版本:按 0.7.1 处理,行为不变。
|
||||
sourceVersion = libraryprofile.LegacyVersion
|
||||
}
|
||||
if sourceVersion != target.SchemaVersion {
|
||||
return fmt.Errorf("%s refused: archive schema version %d cannot be applied to target schema version %d (cross-version %s is not supported)",
|
||||
direction, sourceVersion, target.SchemaVersion, direction)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"knowledge-governance/internal/libraryprofile"
|
||||
)
|
||||
|
||||
func TestSchemaProfileFailsClosedOnUnknownVersion(t *testing.T) {
|
||||
_, err := profileForMigration(135, false, "test")
|
||||
if err == nil || !strings.Contains(err.Error(), "135") || !strings.Contains(err.Error(), "134") {
|
||||
t.Fatalf("unknown version must fail with the real version number, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSchemaProfileRejectsDirtySchema(t *testing.T) {
|
||||
_, err := profileForMigration(134, true, "test")
|
||||
if err == nil || !strings.Contains(err.Error(), "dirty") {
|
||||
t.Fatalf("dirty schema must be refused, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSchemaProfileNamesMissingTableAndColumn(t *testing.T) {
|
||||
profile, err := libraryprofile.For(134)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
columns := map[string]map[string]bool{}
|
||||
for _, table := range append(append([]string{}, profile.RequiredTables...), profile.OptionalTables...) {
|
||||
columns[table] = map[string]bool{}
|
||||
for _, column := range profile.RequiredColumns[table] {
|
||||
columns[table][column] = true
|
||||
}
|
||||
}
|
||||
delete(columns, "wiki_pages")
|
||||
if _, err := verifySchemaCapabilities(profile, columns, "test", 134); err == nil || !strings.Contains(err.Error(), "wiki_pages") {
|
||||
t.Fatalf("missing required table must be named, got %v", err)
|
||||
}
|
||||
|
||||
columns["wiki_pages"] = map[string]bool{"id": true, "tenant_id": true, "knowledge_base_id": true}
|
||||
if _, err := verifySchemaCapabilities(profile, columns, "test", 134); err == nil || !strings.Contains(err.Error(), "wiki_pages.slug") {
|
||||
t.Fatalf("missing required column must be named, got %v", err)
|
||||
}
|
||||
columns["wiki_pages"]["slug"] = true
|
||||
if _, err := verifySchemaCapabilities(profile, columns, "test", 134); err != nil {
|
||||
t.Fatalf("complete capabilities must pass: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSchemaProfileV071BehaviourUnchanged(t *testing.T) {
|
||||
legacy, err := libraryprofile.For(74)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !legacy.HasTable("wiki_log_entries") || !legacy.RequiresPart("wiki/wiki_log_entries") {
|
||||
t.Fatal("0.7.1 must still require wiki_log_entries")
|
||||
}
|
||||
if legacy.EmbeddingDimension != 1024 {
|
||||
t.Fatalf("0.7.1 dimension contract must stay 1024, got %d", legacy.EmbeddingDimension)
|
||||
}
|
||||
v090, err := libraryprofile.For(134)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v090.RequiresPart("wiki/wiki_log_entries") || !v090.AllowsPart("wiki/wiki_log_entries") {
|
||||
t.Fatal("0.9.0 wiki_log_entries must be optional")
|
||||
}
|
||||
if v090.HasTable("wiki_log_entries") == false {
|
||||
t.Fatal("optional table should still be addressable when present")
|
||||
}
|
||||
resolved := SchemaProfile{Profile: v090, Tables: v090.RequiredTables}
|
||||
if resolved.HasTable("wiki_log_entries") {
|
||||
t.Fatal("0.9.0 export table set must exclude an absent optional table")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTargetIndexNameResolvesToConfigurationOnly(t *testing.T) {
|
||||
profile, err := libraryprofile.For(134)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
name, err := resolveTargetIndexName("", "weknora_v090_chunks", profile)
|
||||
if err != nil || name != "weknora_v090_chunks" {
|
||||
t.Fatalf("empty vector store must fall back to the configured index, got %q/%v", name, err)
|
||||
}
|
||||
if _, err := resolveTargetIndexName("", "", profile); err == nil || !strings.Contains(err.Error(), "index name unresolved") {
|
||||
t.Fatalf("no source must fail instead of resolving to an empty name, got %v", err)
|
||||
}
|
||||
if _, err := resolveTargetIndexName("idx_a", "idx_b", profile); err == nil || !strings.Contains(err.Error(), "does not match") {
|
||||
t.Fatalf("conflicting index sources must be refused, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 0.9.0 与 0.7.1 的映射必须算出不同摘要:同样的 1024 维不等于同一份契约。
|
||||
func TestIndexMappingDigestSeparatesSchemaVersions(t *testing.T) {
|
||||
legacy := []byte(`{"weknora_v071_chunks":{"mappings":{"properties":{
|
||||
"chunk_id":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
|
||||
"knowledge_id":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
|
||||
"knowledge_base_id":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
|
||||
"embedding":{"type":"dense_vector","dims":1024}}}}}`)
|
||||
legacyDigest, err := IndexMappingDigest(legacy)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if path := os.Getenv("KG_V090_MAPPING_JSON"); path != "" {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v090Digest, err := IndexMappingDigest(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("real 0.9.0 mapping must be digestible: %v", err)
|
||||
}
|
||||
if v090Digest == legacyDigest {
|
||||
t.Fatal("0.9.0 and 0.7.1 mappings must not share a digest")
|
||||
}
|
||||
if len(v090Digest) != 64 {
|
||||
t.Fatalf("unexpected digest shape %q", v090Digest)
|
||||
}
|
||||
}
|
||||
if _, err := IndexMappingDigest([]byte(`{"a":{"mappings":{}},"b":{"mappings":{}}}`)); err == nil {
|
||||
t.Fatal("multi-index mapping responses must be refused")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// 0.9.0 fixture:真实 pg_dump(schema_migrations=134,83 表,无 wiki_log_entries)。
|
||||
// 只用于显式指定的隔离库;governance 与周报邮件库一律不许指到这里。
|
||||
const v090SchemaEnv = "KG_V090_SCHEMA_SQL"
|
||||
|
||||
func v090SchemaSQL(t *testing.T) string {
|
||||
t.Helper()
|
||||
path := os.Getenv(v090SchemaEnv)
|
||||
if path == "" {
|
||||
t.Skip("requires " + v090SchemaEnv + " pointing at the captured WeKnora 0.9.0 schema dump")
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", v090SchemaEnv, err)
|
||||
}
|
||||
lines := strings.Split(string(raw), "\n")
|
||||
kept := make([]string, 0, len(lines))
|
||||
for _, line := range lines {
|
||||
// psql 元命令(\restrict/\unrestrict)只对 psql 有意义,驱动执行要剔除。
|
||||
if strings.HasPrefix(line, "\\") {
|
||||
continue
|
||||
}
|
||||
kept = append(kept, line)
|
||||
}
|
||||
// 装载会话本身也要把 search_path 还给 public,否则后续查询会找不到表。
|
||||
return strings.Join(kept, "\n") + "\nSET search_path TO public, pg_catalog;\n"
|
||||
}
|
||||
|
||||
// v090Pool 打开一个显式隔离库并重建 public schema,然后装 0.9.0 真实 schema。
|
||||
func v090Pool(t *testing.T, env string) *pgxpool.Pool {
|
||||
t.Helper()
|
||||
dsn := os.Getenv(env)
|
||||
if dsn == "" {
|
||||
t.Skip("requires " + env)
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
|
||||
t.Fatalf("%s must name an explicitly isolated kg_library_test_ database", env)
|
||||
}
|
||||
// dump 里有 set_config('search_path',''),新连接必须显式给回 public。
|
||||
cfg.ConnConfig.RuntimeParams["search_path"] = "public, pg_catalog"
|
||||
ctx := context.Background()
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("connect: %v", err)
|
||||
}
|
||||
t.Cleanup(db.Close)
|
||||
if _, err = db.Exec(ctx, `DROP SCHEMA public CASCADE; CREATE SCHEMA public`); err != nil {
|
||||
t.Fatalf("reset schema: %v", err)
|
||||
}
|
||||
if _, err = db.Exec(ctx, v090SchemaSQL(t)); err != nil {
|
||||
t.Fatalf("install 0.9.0 schema: %v", err)
|
||||
}
|
||||
// 该 dump 会把会话 search_path 置空,这里一律写全限定名。
|
||||
if _, err = db.Exec(ctx, `INSERT INTO public.schema_migrations(version,dirty) VALUES (134,false)`); err != nil {
|
||||
t.Fatalf("stamp schema_migrations: %v", err)
|
||||
}
|
||||
return db
|
||||
}
|
||||
|
||||
const v090Tenant = 10000
|
||||
const v090KB = "kb-v090"
|
||||
const v090Knowledge = "kn-v090-doc"
|
||||
const v090ChunkOne = "ck-v090-1"
|
||||
const v090ChunkTwo = "ck-v090-2"
|
||||
const v090Folder = "wf-v090-1"
|
||||
const v090Page = "wp-v090-1"
|
||||
const v090Tag = "tg-v090-1"
|
||||
const v090Resource = "rs-v090-1"
|
||||
|
||||
// seedV090 造 1 KB / 1 文档 / 2 分块 / 1 wiki 页 / 1 wiki 目录 / 1 资源。
|
||||
// Wiki 必须非空:空表通过等于没有验证 wiki 部件。
|
||||
func seedV090(t *testing.T, ctx context.Context, db *pgxpool.Pool) PostgresIdentity {
|
||||
t.Helper()
|
||||
statements := []string{
|
||||
`INSERT INTO models(id,tenant_id,name,type,source,parameters) VALUES ('bge-m3-v090',$1,'bge-m3','embedding','local','{"embedding_parameters":{"dimension":1024}}'::jsonb)`,
|
||||
`INSERT INTO knowledge_bases(id,name,tenant_id,embedding_model_id,summary_model_id,indexing_strategy) VALUES ('` + v090KB + `','v090 fixture',$1,'bge-m3-v090','bge-m3-v090','{"graph_enabled":false}'::jsonb)`,
|
||||
`INSERT INTO knowledges(id,tenant_id,knowledge_base_id,type,title,source,parse_status,pending_subtasks_count) VALUES ('` + v090Knowledge + `',$1,'` + v090KB + `','file','0.9.0 夹具','fixture://kn-v090','completed',0)`,
|
||||
`INSERT INTO chunks(id,tenant_id,knowledge_base_id,knowledge_id,content,chunk_index,start_at,end_at) VALUES
|
||||
('` + v090ChunkOne + `',$1,'` + v090KB + `','` + v090Knowledge + `','第一段正文',0,0,5),
|
||||
('` + v090ChunkTwo + `',$1,'` + v090KB + `','` + v090Knowledge + `','第二段正文',1,5,10)`,
|
||||
`INSERT INTO wiki_folders(id,tenant_id,knowledge_base_id,name,path) VALUES ('` + v090Folder + `',$1,'` + v090KB + `','夹具目录','/fixture')`,
|
||||
`INSERT INTO wiki_pages(id,tenant_id,knowledge_base_id,slug,title,content,source_refs,chunk_refs,folder_id) VALUES ('` + v090Page + `',$1,'` + v090KB + `','fixture-page','夹具页','wiki 正文','["` + v090Knowledge + `"]'::jsonb,'["` + v090ChunkOne + `"]'::jsonb,'` + v090Folder + `')`,
|
||||
`INSERT INTO knowledge_tags(id,tenant_id,knowledge_base_id,name) VALUES ('` + v090Tag + `',$1,'` + v090KB + `','夹具标签')`,
|
||||
`INSERT INTO knowledge_tag_relations(knowledge_id,tag_id) VALUES ('` + v090Knowledge + `','` + v090Tag + `')`,
|
||||
}
|
||||
for _, statement := range statements {
|
||||
if _, err := db.Exec(ctx, strings.ReplaceAll(statement, "$1", strconv.Itoa(v090Tenant))); err != nil {
|
||||
t.Fatalf("seed: %v\n%s", err, statement)
|
||||
}
|
||||
}
|
||||
// 资源行必须带真实的 physical_path/size/content_hash:封存会逐字节核对。
|
||||
content := []byte("fixture-object-bytes")
|
||||
sum := sha256.Sum256(content)
|
||||
if _, err := db.Exec(ctx, `INSERT INTO resources(id,handle,tenant_id,provider,physical_path,location_hash,size,content_hash)
|
||||
VALUES ($1,$2,$3,'minio','fixture/kn-v090.bin',$4,$5,$6)`,
|
||||
v090Resource, "abcdefghijklmnopqrstuv", v090Tenant, "fixture-location-hash", int64(len(content)), hex.EncodeToString(sum[:])); err != nil {
|
||||
t.Fatalf("seed resource: %v", err)
|
||||
}
|
||||
if _, err := db.Exec(ctx, `INSERT INTO resource_bindings(id,resource_id,tenant_id,owner_type,owner_id)
|
||||
VALUES ('rb-v090-1',$1,$2,'knowledge',$3)`, v090Resource, v090Tenant, v090Knowledge); err != nil {
|
||||
t.Fatalf("seed binding: %v", err)
|
||||
}
|
||||
return PostgresIdentity{TenantID: v090Tenant, KnowledgeBaseID: v090KB}
|
||||
}
|
||||
|
||||
// seedV090Binding 只在目标库造 KB/模型绑定:导入的是文档行,不是 KB 定义。
|
||||
func seedV090Binding(t *testing.T, ctx context.Context, db *pgxpool.Pool) error {
|
||||
t.Helper()
|
||||
for _, statement := range []string{
|
||||
`INSERT INTO models(id,tenant_id,name,type,source,parameters) VALUES ('bge-m3-v090',$1,'bge-m3','embedding','local','{"embedding_parameters":{"dimension":1024}}'::jsonb)`,
|
||||
`INSERT INTO knowledge_bases(id,name,tenant_id,embedding_model_id,summary_model_id,indexing_strategy) VALUES ('` + v090KB + `','v090 fixture target',$1,'bge-m3-v090','bge-m3-v090','{"graph_enabled":false}'::jsonb)`,
|
||||
} {
|
||||
if _, err := db.Exec(ctx, strings.ReplaceAll(statement, "$1", strconv.Itoa(v090Tenant))); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// v090Objects 只在测试里替代对象存储:字节来自夹具常量,不连任何 S3/OSS。
|
||||
type v090Objects map[string][]byte
|
||||
|
||||
func (o v090Objects) Open(_ context.Context, provider, physicalPath string) (io.ReadCloser, error) {
|
||||
if provider != "minio" {
|
||||
return nil, errors.New("unsupported provider in fixture")
|
||||
}
|
||||
content, ok := o[physicalPath]
|
||||
if !ok {
|
||||
return nil, errors.New("fixture object missing")
|
||||
}
|
||||
return io.NopCloser(bytes.NewReader(content)), nil
|
||||
}
|
||||
|
||||
// v090IndexSnapshot 按导出分块合成索引快照:本切片不做真实 ES 采集,
|
||||
// 但文档/映射摘要走的是与线上同一套校验,不能凭空绕过。
|
||||
func v090IndexSnapshot(t *testing.T, pg PostgresSnapshot, mappingSHA string) SearchSnapshot {
|
||||
t.Helper()
|
||||
var documents []SearchDocument
|
||||
for _, raw := range pg.Table("chunks").Rows {
|
||||
var chunk struct {
|
||||
ID string `json:"id"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id"`
|
||||
KnowledgeID string `json:"knowledge_id"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &chunk); err != nil {
|
||||
t.Fatalf("decode chunk: %v", err)
|
||||
}
|
||||
source, err := json.Marshal(map[string]any{
|
||||
"chunk_id": chunk.ID, "tenant_id": chunk.TenantID,
|
||||
"knowledge_base_id": chunk.KnowledgeBaseID, "knowledge_id": chunk.KnowledgeID,
|
||||
"content": chunk.Content, "source_id": chunk.ID, "source_type": 0,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encode index document: %v", err)
|
||||
}
|
||||
documents = append(documents, SearchDocument{ID: "doc-" + chunk.ID, Source: source})
|
||||
}
|
||||
digest, err := searchDocumentsDigest(documents)
|
||||
if err != nil {
|
||||
t.Fatalf("index digest: %v", err)
|
||||
}
|
||||
return SearchSnapshot{Documents: documents, MappingSHA256: mappingSHA, DocumentsSHA256: digest}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -152,8 +153,16 @@ func Validate(manifest Manifest, target TargetContract, mapping KnowledgeBaseMap
|
||||
}
|
||||
|
||||
func validateManifestFields(manifest Manifest) error {
|
||||
if manifest.FormatVersion != FormatVersionV1 || manifest.Contract.WeKnoraVersion != WeKnoraVersionV071 ||
|
||||
manifest.Contract.MigrationVersion != WeKnoraMigrationV071 || manifest.Contract.EmbeddingDimension != EmbeddingDimensionV071 ||
|
||||
// 单文档发布包只认 0.7.1 的表集合与契约。别的版本必须**明确拒绝并说明原因**,
|
||||
// 不能落进下面那句泛化的 "unsupported release contract"——那让人分不清是版本
|
||||
// 不支持还是字段写错了。0.9.0 的整库内容走整库发布路径(libraryprofile v090)。
|
||||
if manifest.Contract.MigrationVersion != WeKnoraMigrationV071 ||
|
||||
manifest.Contract.WeKnoraVersion != WeKnoraVersionV071 {
|
||||
return fmt.Errorf("release bundle schema contract %s/%d is not supported: single-document release handles only v0.7.1 (74); WeKnora 0.9.0 (134) requires the library-level publish path",
|
||||
manifest.Contract.WeKnoraVersion, manifest.Contract.MigrationVersion)
|
||||
}
|
||||
if manifest.FormatVersion != FormatVersionV1 ||
|
||||
manifest.Contract.EmbeddingDimension != EmbeddingDimensionV071 ||
|
||||
manifest.Contract.RetrieverEngine != "elasticsearch_v8" || manifest.Contract.TenantID == 0 || manifest.CreatedAt.IsZero() {
|
||||
return errors.New("unsupported release contract")
|
||||
}
|
||||
|
||||
@@ -165,3 +165,27 @@ func TestValidateRejectsUndeclaredOrDriftedMapping(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// 单文档发布包只支持 0.7.1 契约。0.9.0(134)的包必须被**明确拒绝并说明原因**:
|
||||
// 静默按 74 处理会让 0.9.0 的表集合/索引文档走错写入路径,而泛化错误又让人分不清
|
||||
// "版本不支持"和"字段写错"。
|
||||
func TestSealRejectsNonV071SchemaContractExplicitly(t *testing.T) {
|
||||
manifest := validManifest()
|
||||
manifest.Contract.WeKnoraVersion = "v0.9.0"
|
||||
manifest.Contract.MigrationVersion = 134
|
||||
_, err := Seal(manifest)
|
||||
if err == nil {
|
||||
t.Fatal("0.9.0 release bundle must be refused")
|
||||
}
|
||||
for _, want := range []string{"v0.9.0", "134", "74", "library-level publish"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("rejection message must name %q, got %v", want, err)
|
||||
}
|
||||
}
|
||||
// 只改版本号、不改 WeKnora 版本同样必须拒绝:两者必须成对出现。
|
||||
manifest = validManifest()
|
||||
manifest.Contract.MigrationVersion = 134
|
||||
if _, err := Seal(manifest); err == nil {
|
||||
t.Fatal("schema version 134 with a v0.7.1 label must be refused")
|
||||
}
|
||||
}
|
||||
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Whole-library capture entry point for the RDS-backed WeKnora 0.9.0 test stack
|
||||
# (Compose project weknora-v090).
|
||||
#
|
||||
# Why a separate file instead of a parameterised one: this stack has no source
|
||||
# PostgreSQL container and no governance PostgreSQL container. Both databases are
|
||||
# managed instances (RDS). -source-pg / -governance-pg are therefore passed empty,
|
||||
# which is libraryctl's documented signal to resolve them from the source app
|
||||
# container env (DB_HOST/DB_PORT/DB_NAME/DB_USER/DB_PASSWORD + LIBRARY_SOURCE_DB_SSLMODE)
|
||||
# and from LIBRARY_GOVERNANCE_DSN_FILE (0600 file). Keeping the 0.7.1 entry point
|
||||
# untouched means a change here can never alter a stack that already ran.
|
||||
#
|
||||
# Every service of the Compose project is declared below: exactly one writer (app)
|
||||
# and a reason for each sibling which must NOT be stopped. ProjectController
|
||||
# refuses to run if the project gains a service that is not declared here, so a
|
||||
# new container cannot be silently left outside the maintenance window.
|
||||
#
|
||||
# Credentials are read at call time (container env / 0600 file) and are never
|
||||
# printed. Nothing here writes to the source stack except SIGTERM/SIGSTART of the
|
||||
# declared writer.
|
||||
set -euo pipefail
|
||||
|
||||
# The binary is pinned by the caller on purpose: there is no "whatever is in this
|
||||
# directory" fallback, so a stale build next to the script cannot be executed by
|
||||
# accident against a live stack.
|
||||
BIN="${LIBRARYCTL_BIN:-}"
|
||||
if [ -z "$BIN" ]; then
|
||||
echo "LIBRARYCTL_MISSING: set LIBRARYCTL_BIN to the libraryctl binary built for this run" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -x "$BIN" ]; then
|
||||
echo "LIBRARYCTL_NOT_EXECUTABLE: $BIN is not an executable file" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Archives/journals live outside the home directory on this host (they survive the
|
||||
# operator account); both roots stay overridable for an isolated rehearsal.
|
||||
ARCHIVE_ROOT="${LIBRARY_ARCHIVE_ROOT:-/var/lib/knowledge-governance/library}"
|
||||
JOURNAL_ROOT="${LIBRARY_JOURNAL_ROOT:-/var/lib/knowledge-governance/freeze}"
|
||||
|
||||
# Reasons must not contain commas: the flag is comma-separated.
|
||||
NON_WRITERS="frontend=UI only; no knowledge writes"
|
||||
NON_WRITERS+=",docreader=parse RPC service; app persists parse output"
|
||||
NON_WRITERS+=",elasticsearch=search store; not an initiator"
|
||||
NON_WRITERS+=",redis=queue/cache store; not an initiator"
|
||||
|
||||
exec "$BIN" "$@" \
|
||||
-project weknora-v090 \
|
||||
-writers app \
|
||||
-source-app weknora-v090-app \
|
||||
-source-es weknora-v090-elasticsearch \
|
||||
-source-pg "" \
|
||||
-governance-pg "" \
|
||||
-non-writers "$NON_WRITERS" \
|
||||
-archive "$ARCHIVE_ROOT" \
|
||||
-journal "$JOURNAL_ROOT"
|
||||
@@ -647,6 +647,16 @@
|
||||
"package": "knowledge-governance/internal/httpapi",
|
||||
"test": "TestWeeklyMailSendsHandlerReadsRealHistoryWithMessageID",
|
||||
"reason": "需要一次性 PostgreSQL + 测试主密钥;验证 GET /weekly-mail/sends 在非空历史下返回 200 与真实 message_id,CI 不提供"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/releaseadapter/v071",
|
||||
"test": "TestV090LibraryRoundTripIsolated",
|
||||
"reason": "需 KG_V090_SCHEMA_SQL 指向真实 WeKnora 0.9.0 schema dump、KG_V090_MAPPING_JSON 指向真实索引映射,以及显式隔离的 kg_library_test_* PG;普通 CI 没有这些输入,跳过不是通过"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/releaseadapter/v071",
|
||||
"test": "TestV090ProductionTargetPublishIsolated",
|
||||
"reason": "需显式一次性本机PG(两份 0.9.0 真 schema 库)/ES/对象存储:真实导出→封存→ProductionLibraryTarget 写入 134 目标→逐表核对→写前镜像回滚,验证 0.9.0 无 wiki_log_entries、chunks 新增 NOT NULL 列走默认值、seq_id 由目标序列分配;普通CI没有这些服务,跳过不是通过"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user