Compare commits

..

7 Commits

Author SHA1 Message Date
姚章浩 89c484cdc4 feat(qa): 自动准备知识治理测试环境
build / build (push) Successful in 1m29s
为 HiFox QA 检出提供环境检查、锁定依赖、专用 RDS 迁移、服务探活、测试和进程清理。已在 QA 虚拟机通过 make verify,并验证缺少 RDS 配置、错误提交和端口冲突的阻断。
2026-11-22 19:30:49 +08:00
姚章浩 5e3362e2b0 docs(dev): 补充开发与测试环境指南
build / build (push) Successful in 1m29s
从 README 提供依赖、前后端启动、测试命令和非生产 RDS 准备步骤。已在独立 QA 检出通过 make verify,Vite 健康代理和 Chromium 页面冒烟通过。
2026-11-22 19:18:48 +08:00
zheromiku edd07a65d6 Merge pull request 'docs(ops): 更新架构、平台接入及反代运维手册' (#16) from docs/governance-ops-090-main into main
build / build (push) Successful in 1m28s
2026-11-22 13:17:09 +08:00
姚章浩 7c120b3adc test(qawemail): 用绝对域名断言不可解析主机
build / build (push) Successful in 1m57s
构建机 resolv.conf 带 search 域(Tailscale tailf49802.ts.net)时,
smtp.kg-test.invalid 会被拼成 ...tailf49802.ts.net 并由 Fake-IP DNS
解析成功,断言实际测的是构建机搜索域。改成带末尾点的绝对名后与
搜索域无关,仍保持“不可解析主机必须判非法”的原意。
2026-11-22 13:14:52 +08:00
姚章浩 ea159ca2fb docs(ops): 补充治理架构、平台接入与测试反代手册
build / build (push) Failing after 1m37s
2026-11-22 13:10:44 +08:00
zheromiku 034c0b3a1c Merge pull request 'fix(library): 跨源同步按目标范围改写并校验残留' (#13) from fix/library-cross-scope-sync-residue into main
build / build (push) Successful in 1m18s
2026-10-29 22:13:49 +08:00
姚章浩 5defe238e0 fix(library): 跨范围同步按目标范围改写并校验残留
build / build (push) Successful in 1m34s
真实事故:源 KB(10000/0c84a098) 同步到目标 KB(10001/e64e82b5) 时,归档行被原样
插入目标库,落在来源租户/知识库;资源阶段随即报"生产缺少资源 ... 的存储路径"
(资源查询按目标租户过滤,看不见刚写进去的行)。回滚只按目标范围回放,于是页面
报"已恢复到预备份",目标却多出 4 文件/38 分块/21 资源/4 绑定。

改:
- 写入与校验共用 retargetRow:tenant_id / knowledge_base_id / 资源定位符(后端、
  provider、桶、租户前缀) / storage_backend_id / provider / embedding_model_id
  统一映射到目标范围;行里的来源身份对不上就失败,宁可不写。
- Verify 增加"本次写集合仍以来源身份残留"判定;Restore 增加整库指纹比对(预备份
  时刻 vs 恢复后)与孤儿资源清理,回滚后用 Verify 复核,恢复工具退出 0 不再等于
  恢复完成。
- 新增 cmd/libraryresidue:按归档身份精确、可备份、可复核地清理已知的范围外残留。

验证:make verify 全绿;真实 PG+ES 用例复现"源目标 KB 不同 + 目标非空 + 资源阶段
失败",断言回滚后无来源范围残留、其它租户行未动、恢复后内容=预备份;正向用例断言
行落在目标范围且资源定位符指向目标存储。
2026-10-29 22:11:59 +08:00
19 changed files with 2298 additions and 87 deletions
+8
View File
@@ -27,6 +27,14 @@ BUILD_TIME ?= $(shell date -u +%Y-%m-%dT%H:%M:%SZ)
verify: verify:
$(PY) scripts/ci_checks.py $(PY) scripts/ci_checks.py
.PHONY: qa qa-doctor qa-self-test
qa:
$(PY) scripts/qa_env.py run
qa-doctor:
$(PY) scripts/qa_env.py doctor
qa-self-test:
$(PY) -m unittest discover -s scripts -p test_qa_env.py -v
# 清单:哪些是必需项、在哪个档案(local / ci)下必需 # 清单:哪些是必需项、在哪个档案(local / ci)下必需
list: list:
$(PY) scripts/ci_checks.py --list $(PY) scripts/ci_checks.py --list
+10 -15
View File
@@ -71,15 +71,16 @@ IT、行政等是待扩展的业务资料范围,不是已经完成的新连接
## 访问与部署 ## 访问与部署
### 已部署环境 ### 运行架构与访问入口
Compose 中网页入口固定映射为 **宿主机 `8088` → Web 容器 `8080`**。网页和 `/api/` 使用同源入口。 | 组件 | 职责 | 接手时从这里看 |
|---|---|---|
| React/Vite + Web Nginx | 提供静态页面,同源转发 `/api/` 和 `/health` | [前端开发代理](web/vite.config.ts)、[Compose 的 Nginx 配置](deploy/nginx.conf) |
| Go API + 治理 PostgreSQL | 认证、角色、来源、绑定、审核、任务与发布记录。云测试使用独立逻辑库 `knowledge_governance_test` | [API](cmd/api)、[接口契约](api/openapi.yaml) |
| 普通 worker + 测试 WeKnora | worker 取件、上传、轮询并记录验收结果。WeKnora 负责解析、切片和索引 | [worker](cmd/worker)、[来源接入](docs/OPERATIONS_RUNBOOK.md#接入外部平台) |
| 发布 worker + 生产 WeKnora | 按已锁定的批次和目标执行受控发布与读回。问答端通过下游服务使用已发布知识 | [发布 worker](cmd/release-worker)、[操作手册](docs/OPERATIONS_RUNBOOK.md) |
- 本机访问:`http://127.0.0.1:8088/` 从自己的电脑做系统设置验收,使用 SSH 转发后的 `http://127.0.0.1:28189/`。云测试主控制台使用受限内网入口 `http://100.85.111.121:8088/`。仓内 Compose 则将宿主 `8088` 映射到 Web 容器 `8080`。选择入口时先确认要看的部署。隧道命令、生产 `090` 服务地址和 HTTPS 反代示例见[运维手册](docs/OPERATIONS_RUNBOOK.md#入口与反向代理)。
- 局域网访问:`http://<服务器内网IP>:8088/`
- 健康入口:`http://127.0.0.1:8088/health`
可以由 NAS 反向代理提供 HTTPS 域名入口,目标为平台服务器的 `8088`,保持路径原样转发。公网开放前必须验证鉴权、访问控制及转发头配置,不直接暴露数据库或内部管理接口。
**平台入口展示当前部署版本,不是每次 CI 的构建预览。** CI 静态预览由独立链路提供;CI 通过不会自动更新运行服务。 **平台入口展示当前部署版本,不是每次 CI 的构建预览。** CI 静态预览由独立链路提供;CI 通过不会自动更新运行服务。
@@ -94,13 +95,7 @@ Compose 中网页入口固定映射为 **宿主机 `8088` → Web 容器 `8080`*
- 来源目录、spool 与运行状态挂载;按部署主机调整路径和 UID/GID。 - 来源目录、spool 与运行状态挂载;按部署主机调整路径和 UID/GID。
- 测试 WeKnora 连接与对内/对外路由;生产发布按独立配置和授权接线。 - 测试 WeKnora 连接与对内/对外路由;生产发布按独立配置和授权接线。
准备完成后,由获授权的操作者执行: 仓内 Compose 含 `postgres` 演练服务。云测试和生产使用阿里云 RDS,部署前先核对目标逻辑库、当前迁移版本、待运行的迁移及所用的环境配置。API 和 worker 启动可执行迁移。已有环境先用 `docker compose config --services` 检查服务清单,按该环境的发布手册切换已验证的产物。前端开发可直接使用下面的 Vite 流程。
```bash
docker compose up --build -d
```
主机需要提权时,由操作者按本机权限规则执行,不在自动化中默认使用 sudo。停止服务可用 `docker compose down`;**不要执行 `docker compose down -v`,除非明确批准删除数据库卷。**
## 配置与安全边界 ## 配置与安全边界
@@ -114,7 +109,7 @@ docker compose up --build -d
## 开发与 CI ## 开发与 CI
基础环境:Go 1.26、Node.js 22、npm,以及检查脚本所需的 Python 3 和工具链。依赖以仓库锁文件和实际 CI 配置为准。 本机和 QA 虚拟机的依赖安装、前后端启动、测试命令及 PostgreSQL 测试库准备见 [开发与测试指南](docs/DEVELOPMENT_AND_TESTING.md)。
```bash ```bash
make verify # 完整检查入口:本地与 CI 使用同一套规则 make verify # 完整检查入口:本地与 CI 使用同一套规则
+115
View File
@@ -0,0 +1,115 @@
// command libraryresidue 处理"整库同步写到目标范围外"留下的残留行:
//
// 读指定快照(来源范围)→ 在目标里找同内容、仍带来源身份的行
// → 先备份成 0600 文件并记录摘要 → 精确按归档身份删除 → 复核来源身份已空、
// 目标内容与预备份一致、整库指纹里没有别的租户
//
// 它不做范围广删,也不改任何别的东西:判据是"归档里确实有这些行、内容逐字段
// 一致",任何一行对不上就整体拒绝。默认是干跑(只报告、不删)。
//
// 真实事故:跨范围同步把 4 文件/38 分块/21 资源/4 绑定写成来源租户的行,
// 回滚只按目标范围回放,于是"已恢复"却留下这些行。这个工具的用途就是在
// 修复后的代码上线前,用可审计的方式清掉它们。
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"log"
"os"
"strconv"
"strings"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
"knowledge-governance/internal/librarystack"
v071 "knowledge-governance/internal/releaseadapter/v071"
)
func main() {
scope := flag.String("scope", "internal", "internal|external")
snapshot := flag.String("snapshot", "", "sealed snapshot whose source scope is the residue (required)")
backupSnapshot := flag.String("backup-snapshot", "", "prebackup snapshot the target must be restored to (required)")
expectSource := flag.String("expect-source", "", "expected source scope as <tenant>/<knowledge-base> (required guard against typos)")
out := flag.String("out", "", "directory for the residue backup (required)")
apply := flag.Bool("apply", false, "actually delete the residue rows (default: dry run)")
reportPath := flag.String("report", "", "write the JSON report here")
flag.Parse()
if *snapshot == "" || *backupSnapshot == "" || *out == "" || *expectSource == "" {
log.Fatal("snapshot, backup-snapshot, out and expect-source are required")
}
config, err := appconfig.Load(os.Getenv)
if err != nil {
log.Fatalf("load config: %v", err)
}
ctx := context.Background()
report, err := run(ctx, config, *scope, *snapshot, *backupSnapshot, *expectSource, *out, *apply)
if *reportPath != "" && report != nil {
raw, marshalErr := json.MarshalIndent(report, "", " ")
if marshalErr == nil {
raw = append(raw, '\n')
if writeErr := os.WriteFile(*reportPath, raw, 0o600); writeErr != nil {
log.Printf("report write failed: %v", writeErr)
}
}
}
if err != nil {
log.Fatalf("residue cleanup failed: %v", err)
}
log.Printf("residue cleanup ok: source=%s/%s tables=%v cleared=%d applied=%v verified=%v",
reportSource(report), report.SourceKB, report.Tables, report.Cleared, report.Applied, report.Verified)
}
func reportSource(report *v071.ResidueReport) string {
return strconv.FormatUint(report.SourceTenant, 10)
}
func run(ctx context.Context, config appconfig.Config, scope, snapshotID, backupSnapshotID, expectSource, out string,
apply bool) (*v071.ResidueReport, error) {
if scope != "internal" && scope != "external" {
return nil, fmt.Errorf("scope must be internal or external")
}
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
if err != nil {
return nil, fmt.Errorf("归档不可用: %w", err)
}
manifest, err := archive.Load(snapshotID)
archive.Close()
if err != nil {
return nil, fmt.Errorf("指定快照不可读: %w", err)
}
source := v071.ProductionScope{TenantID: manifest.TenantID, KnowledgeBaseID: manifest.KnowledgeBaseID}
want := strconv.FormatUint(source.TenantID, 10) + "/" + source.KnowledgeBaseID
// 范围写错会把"别人的行"当成残留:必须由调用方显式确认。
if strings.TrimSpace(expectSource) != want {
return nil, fmt.Errorf("来源范围与 -expect-source 不一致:归档为 %s,参数为 %s", want, expectSource)
}
target, _, err := librarystack.OpenProductionStack(ctx, config, scope, backupSnapshotID, "")
if err != nil {
return nil, fmt.Errorf("装配目标失败: %w", err)
}
defer target.Close()
report, err := target.ClearForeignScope(ctx, source, snapshotID, out, apply)
if err != nil {
return report, err
}
report.Foreign, _ = target.ForeignScopes(ctx)
if !apply {
return report, nil
}
if err := target.VerifyResidueCleared(ctx, source); err != nil {
return report, err
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: backupSnapshotID}); err != nil {
return report, fmt.Errorf("清理后目标内容仍与预备份不一致(保持停写): %w", err)
}
report.Verified = true
if len(report.Foreign) > 0 {
raw, _ := json.Marshal(report.Foreign)
return report, fmt.Errorf("清理后目标仍有其它租户的数据,需人工确认才能开放:%s", raw)
}
return report, nil
}
+131
View File
@@ -0,0 +1,131 @@
# 知识治理平台开发与测试
以下命令从仓库根目录运行。日常开发可在 Ubuntu 宿主进行,指定提交的回归在 `hifox-qa` 的独立检出中运行。部署到阿里云测试服后的验收按 [操作 Runbook](OPERATIONS_RUNBOOK.md) 和 [用户流程验收](USER_FLOWS_ACCEPTANCE.md) 执行。
## QA 虚拟机一键回归
在 `hifox-qa` 检出提测提交后,运行下列入口。将 SHA 占位值换成提测记录里的完整提交 SHA。
```bash
make qa-doctor
KG_QA_SHA='<完整提交 SHA>' make qa
```
`make qa-doctor` 检查工具链、HiFox 机器在线状态、OMP 模型调用、测试库身份和已安装的迁移。`make qa` 安装锁定依赖,在专用测试库上通过 API 初始化和迁移,随后核对完整迁移序列、API 就绪状态、Web 首页及 Vite 健康代理。通过后执行 Go 和 Vitest 测试,结束时关闭本次启动的服务。每次运行的服务日志保存在 `~/.local/state/knowledge-governance/qa/`。需要提交前完整门禁时执行 `make verify`。
初次配置由负责人在 QA 虚拟机准备 `~/.config/knowledge-governance/`。以下示例使用虚构的 RDS 主机名。
| 文件 | 内容 |
| --- | --- |
| `qa.json` | 当前用户所有、权限 `0600`,字段 `host`、`port`、`database`、`user` |
| `qa.pgpass` | 当前用户所有、权限 `0600`,填写与 `qa.json` 匹配的口令条目 |
| `rds-ca.pem` | 对应实例的 CA 证书,用于校验 RDS 主机名 |
```json
{"host":"rds.example.test","port":5432,"database":"kg_qa_app","user":"kg_qa_app"}
```
专用非生产 RDS 库和角色以 `kg_qa_` 开头,权限和网络准备见第 2 节。首次运行接受空库,后续运行接受与当前提交兼容的迁移序列。缺库、认证失败、迁移不兼容和端口冲突均会使命令非零退出。浏览器业务 E2E 和专用集成库的 Go 测试按第 4 节执行。
## 1. 检出代码与安装依赖
准备 Git、Make、Python 3、Go 1.26(含同一工具链的 `gofmt`)、Node.js 22 和 npm。数据库检查另需 PostgreSQL 客户端 `psql`。首次安装依赖需要能访问 Go 和 npm 的依赖源。
```bash
# QA 在独立克隆中拉取提测分支,使用提测记录里的完整提交 SHA
export KG_TARGET_BRANCH='<提测分支名>' KG_TARGET_SHA='<完整提交 SHA>'
git fetch origin "$KG_TARGET_BRANCH" && git switch --detach "$KG_TARGET_SHA" &&
test "$(git rev-parse HEAD)" = "$KG_TARGET_SHA" && test -z "$(git status --porcelain)"
# 上面的命令链若失败,先解决检出或工作区问题再继续
# 检查当前 shell 的工具链,非交互任务也应设置相同的 PATH
export PATH="$(go env GOROOT)/bin:$PATH"
go version
command -v gofmt
node --version
npm --version
python3 --version
make deps-install
```
`make deps-install` 通过 `go mod download` 和 `npm ci` 安装依赖。Go 依赖由 `go.mod` / `go.sum` 约束,前端由 `web/package-lock.json` 锁定。依赖安装失败时先检查工具链、依赖源和锁文件,避免临时改写清单来绕过问题。
## 2. 准备开发和测试数据库
无库检查可直接跳到第 4 节。启动 API 或执行真库测试前,由数据库管理员在**非生产 PostgreSQL RDS** 上预建专用空库和登录角色。建议分配 `kg_qa_app` 给应用冒烟,`kg_qa_it` 给 Go 集成测试,各用独立角色。开发宿主另用开发专用库,避免与 QA 共享数据。库未分配时先申请,启动流程以库已存在为前提。
| 用途 | 所需权限 | 数据初始化 |
| --- | --- | --- |
| 应用启动与 UI 冒烟 | `CONNECT`,以及目标 schema 内的 `USAGE`、`CREATE` 和对应用对象的读写权限。迁移角色须能修改自己创建的对象。 | 首次 API 启动执行仓库内嵌迁移。用户表为空时创建初始管理员,首次登录必须修改密码。初始口令通过测试环境负责人交接。 |
| Go PostgreSQL 集成测试 | `CONNECT` 和目标测试库的 `CREATE` 权限,用于建立各用例自己的 schema。 | 用例创建隔离 schema、执行迁移、写入测试数据并清理。相同测试不要并发指向同一个库,避免固定 schema 名冲突。 |
两类角色均只授予对应非生产库所需的权限。使用空库和合成数据。密码由操作者在目标机配置到权限为 `0600` 的 PostgreSQL passfile,条目按 `host:port:database:user:password` 填写。使用 RDS CA 校验证书。以下命令先读回实际数据库、角色与 TLS 状态,核对三者再启动 API 或运行真库测试。
```bash
# 用获准的 RDS 主机名、端口、库名、角色和文件路径替换示例值
export PGHOST='<非生产 RDS 主机名>' PGPORT=5432
export PGDATABASE=kg_qa_app PGUSER=kg_qa_app
export PGPASSFILE="$HOME/.config/knowledge-governance/qa.pgpass"
export PGSSLROOTCERT="$HOME/.config/knowledge-governance/rds-ca.pem"
export PGSSLMODE=verify-full PGCONNECT_TIMEOUT=5
psql -X -v ON_ERROR_STOP=1 -c \
'SELECT current_database(), current_user, ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid();'
```
如果是云厂商提供的私网地址,先确认执行机通过获准的网络路径可访问 RDS。库名、角色、TLS、连接权限或迁移权限有问题时,先交由环境负责人核对配置。实例上已有数据的库不能当作空库初始化。
## 3. 启动后端和前端
**终端 A,启动 API。** 在干净 shell 中执行第 2 节的应用库配置和连通性检查。使用独立测试配置,不加载现网 `.env`。
```bash
export DATABASE_URL="postgresql://${PGUSER}@${PGHOST}:${PGPORT}/${PGDATABASE}?sslmode=verify-full&connect_timeout=5"
export APP_ADDR=127.0.0.1:8088
export TEST_WEKNORA_ENABLED=false PP_SCAN_ENABLED=false
export RELEASE_WORKER_ENABLED=false PRODUCTION_RELEASE_REQUESTS_ENABLED=false
export DOBOTAI_OPS_READ_ENABLED=false DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED=false
go run ./cmd/api
```
API 会在监听前连接 PostgreSQL、应用迁移并初始化管理员。Web 页面和同步 API 联调只需运行 API 与 Vite。需要验证后台队列时,另用相同的隔离应用库配置启动普通 worker(`go run ./cmd/worker`)。涉及取件或来源刷新的任务还依赖容器内 `/app/scripts` 及获准的来源接线,按部署说明准备。发布 worker 无需启动。
Vite 通过 `localhost:8088` 转发 `/api` 和健康检查。如果 `8088` 已被其他部署占用,使用端口空闲的 QA 虚拟机运行本节。仅改前端时,可复用一个获准联调的测试 API,但需要先确认 `localhost:8088/health` 可达,并核对该地址指向的环境。
**终端 B,启动 Web。**
```bash
npm --prefix web run dev -- --host 127.0.0.1 --port 5173 --strictPort
```
在运行服务的机器打开 `http://127.0.0.1:5173/`。另开终端检查 API 与前端代理,再复查迁移记录。最后一条命令需要第 2 节的应用库配置。
```bash
curl --fail --silent --show-error http://127.0.0.1:8088/health
curl --fail --silent --show-error http://127.0.0.1:5173/health
psql -X -v ON_ERROR_STOP=1 -c \
'SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1;'
```
空库首次启动后,用专用测试账号登录,按当次验收标准验证页面及接口。联调结束在两个终端按 `Ctrl+C` 停服务,数据库由环境管理员按测试数据保留规则管理。
## 4. 选择测试命令
| 目的 | 命令 |
| --- | --- |
| 查看检查及适用范围 | `make list` |
| 后端包测试 | `make tests-go` |
| 单个 Go 包及相关子包 | `go test -count=1 ./internal/httpapi/...` |
| 前端 Vitest 套件 | `make tests-web` |
| 单个前端测试文件 | `npm --prefix web test -- src/App.test.tsx` |
| 完整提交前检查,与 CI 共用入口 | `make verify` |
`make verify` 包含安装、格式、静态检查、构建、测试、版本和安全检查,并写出 `ci-report.json`。在未导入真库测试配置的 shell 中运行。若检出时用了 `--no-tags`,先执行 `git fetch --tags origin`,以便版本检查读取正式标签对应的提交。依赖安装与安全检查需要相应网络或缓存。允许跳过的外部依赖用例见 [test-scope.json](../scripts/test-scope.json),真库测试应单独执行并记录结果。
在应用冒烟之外验证真实 PostgreSQL 行为时,重新按第 2 节切换并核对 **集成测试库和角色**,然后执行目标用例。下面以 UI 设置版本切换用例为例。
```bash
TEST_DATABASE_URL="postgresql://${PGUSER}@${PGHOST}:${PGPORT}/${PGDATABASE}?sslmode=verify-full&connect_timeout=5" \
go test -count=1 -v ./internal/platform/postgres -run '^TestUISettingsActivationRetiresPreviousRevisionIntegration$'
```
检查输出出现目标用例的 `PASS`。用例被跳过或没有匹配时,应补齐配置或纠正用例名后重跑。前端 Vitest 负责组件与业务逻辑检查,浏览器 E2E 应单独执行并按 Feature 保存证据。提测记录至少包括完整提交 SHA、执行命令、退出码,以及失败用例的日志或浏览器截图。
+80 -1
View File
@@ -3,7 +3,86 @@
按"实际干活顺序"排列的**现在的**操作流程。标注:**【新】**本期新增(含本次会话与 `f4e1ecc/6496ee3/9155bd3` 的 dws 取件)|**【变】**既有流程本期有改动|**【既有】**未变,仍照旧走。 按"实际干活顺序"排列的**现在的**操作流程。标注:**【新】**本期新增(含本次会话与 `f4e1ecc/6496ee3/9155bd3` 的 dws 取件)|**【变】**既有流程本期有改动|**【既有】**未变,仍照旧走。
逐条验收证据请看 `docs/USER_FLOWS_ACCEPTANCE.md`;细节规则看各自专题文档。 逐条验收证据请看 `docs/USER_FLOWS_ACCEPTANCE.md`;细节规则看各自专题文档。
统一入口 **http://127.0.0.1:8088/**(测试栈,右侧面板可直接操作)。角色:管理员 / 文档工程师 / 只读。 管理员配置连接和发布授权,文档工程师维护资料,只读成员查看进度。先确定要操作的环境,再按下面的接线与验收步骤进入业务流程。
## 入口与反向代理
| 请求从哪里发出 | 地址或路径 | 实际目的地 |
|---|---|---|
| 已接入内网的运维电脑 | `http://100.85.111.121:8088/` | 云测试治理主站 |
| 开发电脑的浏览器 | `http://127.0.0.1:28189/` | SSH 转发到测试机回环 `28189`,由候选 Web 提供静态资源 |
| 测试候选 Nginx(host 网络) | `/api/`、`/health` | 测试机 `127.0.0.1:18099` 候选 API |
| 同一候选 Nginx | `/api/v1/qa-ops/ai/` | 测试机 `127.0.0.1:18098` 现用治理 API,负责模型配置和连接验证 |
| 标准 Compose Web(容器网络) | `/api/`、`/health` | Docker DNS 解析 `api:8080`。静态资源和 SPA 回退由同一 Web 提供 |
**在开发电脑恢复验收入口。** 前提是 SSH 别名 `weknora-test` 能登录测试机。先查远端服务,再查本机监听。本机端口空闲且远端正常时,在独立终端运行第三条命令,保留该终端。在另一终端执行最后一条并用浏览器打开同一地址。
```bash
ssh weknora-test 'curl -fsS --max-time 5 http://127.0.0.1:28189/health'
ss -ltn 'sport = :28189'
ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -L 127.0.0.1:28189:127.0.0.1:28189 weknora-test
curl -fsS --max-time 5 http://127.0.0.1:28189/health
```
**为内网域名配置 HTTPS。** 在能访问测试主站的受信任反代机上配置证书、内部 DNS 和访问范围。以下片段放入已有 HTTPS `server`,完整保留 URI、Host 和会话 Cookie。上游应选治理 Web 入口,使 `/assets/`、`/api/`、`/health` 走同一域名。若该入口的内层代理也设置 `X-Forwarded-Proto`,联调时检查其保留受信任 TLS 终止层的协议值以及登录回调的实际 URL。
```nginx
location / {
proxy_pass http://100.85.111.121:8088;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
```
在反代机执行 `nginx -t` 校验待应用配置。入口启用后,通过该域名验证登录、一次 `/api/` 请求和首页实际引用的 JS/CSS。`index.html` 应重新校验,带哈希的资源可长期缓存,缺失的 JS 返回 404。候选切换前后分别记录通用上游和 AI 专用上游。现场可在测试机用以下只读命令查看,容器名以当时部署为准。仓内同源配置见 [deploy/nginx.conf](../deploy/nginx.conf)。
```bash
docker exec kg-test-storage-ui-6444814 nginx -T 2>&1 | grep -E 'listen .*28189|location /api/v1/qa-ops/ai/|set \$(api|health)_upstream|proxy_pass http://127[.]0[.]0[.]1:18098'
docker inspect -f '{{.Name}} {{index .Config.Labels "release"}}' kg-test-storage-ui-6444814 kg-accept-090-api
```
**现场版本与回退。** 本次在测试机读到 Web 标签 `4b615abe534b`、API 标签 `341c3154ac91`。完整候选包的 `current` 链接位于 `/opt/knowledge-governance/acceptance/090/current`,单独切换的 Web 位于同级 `web-only/4b615abe534b/dist`,实际反代配置位于 `integration/28189.conf`。切换前记录这三处、容器标签与旧 Web 名称。静态页回退时恢复上次的 Web 资源和对应代理配置。完整候选回退按同批装配工具的 `deploy/accept-090/README.md` 执行。回退后用浏览器重新验证首页资源、模型连接和列库。数据库迁移另按备份与恢复方案处理。
本次已核实共享测试治理库到 `000082`。对内、对外独立启用的实现依赖 `000083`,待独立库联调或共享库迁移获准后,再验证单侧暂停、保存和刷新。生产 `070` 不在该套验收范围。
下文系统设置中的绑定、盘点和模型连接步骤适用于上表的测试候选版本。根据现场标签选择对应版本的代码和交付物。
## 接入外部平台
| 平台与用途 | 管理员或运维怎样接 | 接通的证据 |
|---|---|---|
| 钉钉知识库,取目录和文件 | 系统设置 → 数据源配置 → 钉钉授权。用 DWS 设备授权,以来源账号选择组织、空间和范围。API 与 worker 使用部署约定的 DWS 身份目录 | 登录后读回预期空间,选一个有权限的目录校验。登记来源、扫描完成和文件取件分别核验。 |
| AList,取工业或教育资料 | 按来源登记服务根地址和允许的目录。普通 worker 读取专用令牌文件。目录接口 `POST /api/fs/list` 使用 `X-Gateway-Token` | 在来源页扫描已知根目录,对照一个子目录和一份文件。下载遵循来源签名链接,凭据和签名 URL 仅留在受控运行环境。[目录与取件](SOURCE_CATALOG_LIVE_ROUTING_ACCEPTANCE.md) |
| PP 资料门户 | 在数据源配置填写 HTTPS 根地址和来源账号,点「测试连接」后「保存并校验」。API 和 worker 共享 `PP_AUTH_STATE_FILE` 所在目录 | 真实登录并读取一条目录记录,刷新后核对生效配置。超时或限流时查看待验证状态。 |
| WeKnora 测试 090,处理与验收 | 系统设置 → 知识库绑定 → 测试 090。当前测试 API 地址 `http://127.0.0.1:8092`,由治理服务器访问。输入该实例的专用 Key 或使用已配置凭据 | 「验证并读取知识库」能列出预期库。核对名称和 ID,为对内、对外选择不同的目标,保存后刷新读回 |
| WeKnora 生产 090,列库与映射 | 在同页切到生产 090,使用下节的转发入口与生产专用 Key。新连接的安全存储由 `WEKNORA_BINDING_MASTER_KEY_FILE` 提供 | 读取生产目录并核对两个目标 ID,保存后重新进入生产页核实。执行发布另按 §5–6 授权 |
| 问答 AI 模型 | 系统设置 → AI 模型连接,填写 OpenAI 兼容基础地址(含服务要求的 `/v1`)、模型名及专用 Key。API 与普通 worker 使用同一 `QA_AI_MASTER_KEY_FILE` | 点「测试连接」等待成功,需生效时再保存并刷新检查模型名。存储密钥、权限与 DNS 配置见 [README](../README.md#问答运营模型配置) |
**生产 090 的两段地址。** 生产机私网 IP 为 `172.29.151.11`,其 WeKnora API 对宿主回环发布 `127.0.0.1:18092`。测试治理机上已有 SSH 本地转发,治理 API 使用 host 网络,故页面的生产服务地址填写 **`http://127.0.0.1:18092`**。请求从治理 API 发出,经测试机的回环监听到生产机的回环 API。浏览器所在电脑的 `127.0.0.1:28189` 则用于打开治理页面。迁到 bridge 网络时要从新 API 容器重新验证上游可达性。
在**测试治理机**上检验已有通道。`/health` 返回 200 证明转发可用,无凭据列库返回 401 表明请求已到认证层。接着在治理页面安全输入该实例的 Key 并验证真实库列表。本次核验时,生产通道探活成功,匿名列库返回 401。带生产凭据列库和保存映射仍待验收。
```bash
ss -ltnp 'sport = :18092'
curl --noproxy '*' --max-time 5 -sS -o /dev/null -w 'health %{http_code}\n' http://127.0.0.1:18092/health
curl --noproxy '*' --max-time 5 -sS -o /dev/null -w 'catalog %{http_code}\n' http://127.0.0.1:18092/api/v1/knowledge-bases
```
## 开发测试和现场验收
1. 在开发机运行 `make list` 查看门禁,`make verify` 做整套检查。以 `ci-report.json` 的 `commit_sha` 对照 `git rev-parse HEAD`,记录失败或跳过项。前端启动和组件测试见 [开发与 CI](../README.md#开发与-ci)。
2. 针对后端改动,在独立测试逻辑库核对 `SELECT current_database(), max(version) FROM schema_migrations;` 与候选版本的迁移目录。云端多个候选可能共用治理 RDS。只改静态页面时单独切换 Web,保留原资源和代理配置供回滚。
3. 在指定入口以真实账号分别验证。来源页读取一条已知记录,模型页测试到成功终态,知识库绑定在正确环境列库,保存后刷新核对。存储页按最近批次和子库目录分别与 `du -sB1 -- <目录>` 对账。记录入口、Web/API 提交、时间和结果。
4. 候选备份页读取宿主采集的 `metadata.json`,API 只读挂载输出目录。「重新盘点」只重读文件。超过 `BACKUP_HOST_METADATA_MAX_AGE` 时,目录、占盘和预测转为未知。宿主重新采集后核对采集时间是否前进、批次是否恢复。测试现场使用按次采集、15 分钟有效期。采集脚本和挂载参数以同批测试候选交付物为准。
| 症状 | 第一步怎么查 | 如何判定 |
|---|---|---|
| 浏览器连接被拒 | 对照本机 `28189` 监听和测试机 `/health` | 远端正常、本机无监听时恢复本机 SSH 转发 |
| 页面打开,某个 API 404 或 502 | 查浏览器请求路径、Nginx 生效上游及对应 API 版本 | 404 先查路由是否存在,502 先查上游是否可达。AI 页单独核对 `18098` |
| 能探活,列库仍灰或返回 401 | 核对选中环境、凭据配置状态、容器运行 UID 与凭据文件可读性 | 用该环境的专用身份实际列库。检查过程只记录状态码和可读性 |
| 历史可读,计划或配额显示未知 | 核对快照年龄、执行主机连接与配额字段 | 盘点、SSH 计划、配额分别验证。只有主机实际读回 cron 才能报告下一次执行时间 |
--- ---
+20 -1
View File
@@ -59,6 +59,14 @@ type Expectation struct {
Digests map[string]string Digests map[string]string
} }
// TableFingerprint 是一张知识域表在**整实例**上的行数与身份集合摘要(不限租户/
// 知识库)。恢复后必须与预备份时刻逐表一致:只比目标范围会让"写到范围外"的
// 残留被当成已恢复(真实事故:恢复报成功,目标仍多 67 行)。
type TableFingerprint struct {
Count int64 `json:"count"`
Digest string `json:"digest"`
}
// BackupRef 指向一份可独立恢复的完整预备份。 // BackupRef 指向一份可独立恢复的完整预备份。
type BackupRef struct { type BackupRef struct {
ID string `json:"id"` ID string `json:"id"`
@@ -67,6 +75,9 @@ type BackupRef struct {
SnapshotID string `json:"snapshot_id"` SnapshotID string `json:"snapshot_id"`
Bytes int64 `json:"bytes"` Bytes int64 `json:"bytes"`
CreatedAt time.Time `json:"created_at"` CreatedAt time.Time `json:"created_at"`
// Universe 是预备份时刻的整实例指纹,恢复后逐表比对。空表示这次没有采集
// (例如 worker 被杀后用已持久化的快照接管),此时恢复只能靠内容校验兜底。
Universe map[string]TableFingerprint `json:"universe,omitempty"`
} }
type Target interface { type Target interface {
@@ -224,8 +235,16 @@ func failAndRestore(ctx context.Context, outcome Outcome, target Target, recorde
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback) recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
return outcome return outcome
} }
// "恢复工具跑完"不是"恢复完成":必须再校验一次目标内容与预备份一致,
// 否则页面会把半恢复状态报成已恢复(真实事故就是这样放开了维护)。
if verifyErr := target.Verify(ctx, Expectation{SnapshotID: outcome.Backup.SnapshotID}); verifyErr != nil {
outcome.State = "rollback_failed"
outcome.Rollback = "恢复后校验失败(目标可能仍是半写状态):" + verifyErr.Error()
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
return outcome
}
outcome.State = "rolled_back" outcome.State = "rolled_back"
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID + ",并逐表/索引/资源复核一致"
recorder.Stage(ctx, "rollback", "succeeded", outcome.Rollback) recorder.Stage(ctx, "rollback", "succeeded", outcome.Rollback)
return outcome return outcome
} }
+50 -8
View File
@@ -13,9 +13,14 @@ type fakeTarget struct {
applyErr map[Stage]error applyErr map[Stage]error
verifyErr error verifyErr error
restoreErr error restoreErr error
applied []Stage // 恢复之后校验是否通过:用来区分"恢复跑完"和"恢复已验证"。
restored int verifyPassesAfterRestore bool
currentN int applied []Stage
restored int
currentN int
verifyN int
// 记录最后一次 Verify 是否发生在 Restore 之后(恢复必须被校验兜底)。
verifiedAfterRestore bool
} }
func (f *fakeTarget) Current(context.Context) (CurrentState, error) { func (f *fakeTarget) Current(context.Context) (CurrentState, error) {
@@ -35,7 +40,17 @@ func (f *fakeTarget) Apply(_ context.Context, stage Stage, _ string) error {
return nil return nil
} }
func (f *fakeTarget) Verify(context.Context, Expectation) error { return f.verifyErr } func (f *fakeTarget) Verify(context.Context, Expectation) error {
f.verifyN++
f.verifiedAfterRestore = f.restored > 0
if f.verifyErr == nil {
return nil
}
if f.verifyPassesAfterRestore && f.restored > 0 {
return nil
}
return f.verifyErr
}
func (f *fakeTarget) Restore(context.Context, BackupRef) error { func (f *fakeTarget) Restore(context.Context, BackupRef) error {
f.restored++ f.restored++
@@ -185,12 +200,39 @@ func TestRunRollsBackWhenStageFails(t *testing.T) {
} }
} }
func TestRunRollsBackWhenVerificationFails(t *testing.T) { // 恢复跑完但恢复后校验仍失败:不能报 rolled_back。真实事故里页面就是在这种
target := &fakeTarget{verifyErr: errors.New("生产分块数不一致")} // 状态下显示"已恢复到预备份并验证",而目标其实还留着 67 行残留。
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, nil) func TestRunDoesNotClaimRollbackWhenRecoveryVerificationFails(t *testing.T) {
if outcome.State != "rolled_back" || target.restored != 1 { target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
verifyErr: errors.New("生产分块数不一致")}
recorder := &stageLog{}
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
if outcome.State != "rollback_failed" || target.restored != 1 {
t.Fatalf("outcome = %+v restored=%d", outcome, target.restored) t.Fatalf("outcome = %+v restored=%d", outcome, target.restored)
} }
if !strings.Contains(outcome.Rollback, "恢复后校验失败") {
t.Fatalf("rollback message = %q", outcome.Rollback)
}
if !recorder.has("rollback:failed") {
t.Fatalf("stage log = %v", recorder.entries)
}
}
// 只有"恢复到预备份且恢复后校验通过"才允许报 rolled_back。
func TestRunReportsRolledBackOnlyAfterRecoveryVerifies(t *testing.T) {
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
verifyErr: errors.New("半写状态"), verifyPassesAfterRestore: true}
recorder := &stageLog{}
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
if outcome.State != "rolled_back" {
t.Fatalf("outcome = %+v", outcome)
}
if !target.verifiedAfterRestore {
t.Fatal("restore must be followed by a content verification")
}
if !recorder.has("rollback:succeeded") {
t.Fatalf("stage log = %v", recorder.entries)
}
} }
func TestRunReportsRollbackFailureDistinctly(t *testing.T) { func TestRunReportsRollbackFailureDistinctly(t *testing.T) {
+4 -1
View File
@@ -48,7 +48,10 @@ func TestNormalizeRefusesEnabledSettingsWithoutTransportAndBadRecipients(t *test
t.Fatalf("defaults not applied: %+v %v", good, err) t.Fatalf("defaults not applied: %+v %v", good, err)
} }
// 解析不了的主机名必须仍被判为非法:.invalid 是 RFC 6761 保留域,永远不解析。 // 解析不了的主机名必须仍被判为非法:.invalid 是 RFC 6761 保留域,永远不解析。
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid", From: "kb@example.com", // 末尾的点是必须的:构建机的 resolv.conf 可能带 search 域(本环境 Tailscale
// 的 tailf49802.ts.net),不带点就会被拼成 smtp.kg-test.invalid.<search> 并
// 由 Fake-IP DNS 解析出来,于是这条断言测的是构建机的搜索域而不是被测逻辑。
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid.", From: "kb@example.com",
Recipients: []string{"a@example.com"}}); err != ErrInvalid { Recipients: []string{"a@example.com"}}); err != ErrInvalid {
t.Fatalf("unresolvable host must stay invalid, got %v", err) t.Fatalf("unresolvable host must stay invalid, got %v", err)
} }
@@ -113,6 +113,12 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
if err != nil { if err != nil {
return libraryrelease.BackupRef{}, err return libraryrelease.BackupRef{}, err
} }
// 整实例指纹:恢复之后要拿它证明"多写的、少写的都没有"。只按目标范围比,
// 写到范围外的行会被漏掉(真实事故:恢复报成功但目标多 67 行)。
universeBefore, err := universeFingerprint(ctx, s.pool)
if err != nil {
return libraryrelease.BackupRef{}, err
}
pg, err := ExportLibrary(ctx, s.pool, s.identity) pg, err := ExportLibrary(ctx, s.pool, s.identity)
if err != nil { if err != nil {
return libraryrelease.BackupRef{}, fmt.Errorf("读取生产整库失败: %w", err) return libraryrelease.BackupRef{}, fmt.Errorf("读取生产整库失败: %w", err)
@@ -154,6 +160,13 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间生产 %s 内容发生变化,切点不一致", table) return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间生产 %s 内容发生变化,切点不一致", table)
} }
} }
universeAfter, err := universeFingerprint(ctx, s.pool)
if err != nil {
return libraryrelease.BackupRef{}, err
}
if diff := universeDiff(universeBefore, universeAfter); diff != "" {
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间整实例知识域发生变化(%s),切点不一致", diff)
}
for table, want := range current.Counts { for table, want := range current.Counts {
if table == "index" { if table == "index" {
continue continue
@@ -171,7 +184,7 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
total += part.Bytes total += part.Bytes
} }
return libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID, Bytes: total, return libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID, Bytes: total,
CreatedAt: createdAt}, nil CreatedAt: createdAt, Universe: universeBefore}, nil
} }
func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) { func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) {
@@ -84,8 +84,16 @@ type ProductionLibraryTarget struct {
scope ProductionScope scope ProductionScope
search *SearchExporter search *SearchExporter
now func() time.Time now func() time.Time
// storageBackend 是本次写入使用的目标存储后端 id(写入与校验共用)。 // storageBackend / storageProvider 是本次写入使用的目标存储后端与提供方
storageBackend string // (写入与校验共用,缺一不可:只换桶不换提供方,目标取文件仍会走来源实例的路)。
storageBackend string
storageProvider string
// snapshotTenant / snapshotKB 是这次要写入的快照来源身份。写入时按它做
// 来源→目标的范围映射;校验时按它反查"目标里还有没有来源范围的行"。
snapshotTenant uint64
snapshotKB string
// targetEmbeddingModel 是目标知识库绑定的 embedding 模型 id(文档行要换成它)。
targetEmbeddingModel string
} }
// resolveStorageBackend 在目标库里找本次写入的存储后端。找不到就报错:宁可不写, // resolveStorageBackend 在目标库里找本次写入的存储后端。找不到就报错:宁可不写,
@@ -98,24 +106,39 @@ func (t *ProductionLibraryTarget) resolveStorageBackend(ctx context.Context) err
if !validSearchIdentifier(t.config.ObjectsBackendID) { if !validSearchIdentifier(t.config.ObjectsBackendID) {
return errors.New("目标存储后端 id 形状不合法") return errors.New("目标存储后端 id 形状不合法")
} }
t.storageBackend = t.config.ObjectsBackendID provider, err := t.storageProviderOf(ctx, t.config.ObjectsBackendID)
if err != nil {
return err
}
t.storageBackend, t.storageProvider = t.config.ObjectsBackendID, provider
return nil return nil
} }
var id string var id, provider string
err := t.pool.QueryRow(ctx, ` err := t.pool.QueryRow(ctx, `
SELECT id FROM storage_backends SELECT id, coalesce(provider,'') FROM storage_backends
WHERE tenant_id=$1 AND coalesce(config->>'bucket_name','')=$2 WHERE tenant_id=$1 AND coalesce(config->>'bucket_name','')=$2
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id) ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id, &provider)
if err != nil { if err != nil {
return errors.New("目标实例没有与目标桶匹配的存储后端,拒绝写入不可解析的资源定位符") return errors.New("目标实例没有与目标桶匹配的存储后端,拒绝写入不可解析的资源定位符")
} }
if !validSearchIdentifier(id) { if !validSearchIdentifier(id) {
return errors.New("目标存储后端 id 形状不合法") return errors.New("目标存储后端 id 形状不合法")
} }
t.storageBackend = id t.storageBackend, t.storageProvider = id, provider
return nil return nil
} }
// storageProviderOf 读出目标存储后端的提供方(minio / oss / s3…)。资源行里的
// 提供方必须与它一致,否则目标应用会按来源提供方去找一个不存在的存储服务。
func (t *ProductionLibraryTarget) storageProviderOf(ctx context.Context, id string) (string, error) {
var provider string
if err := t.pool.QueryRow(ctx, `SELECT coalesce(provider,'') FROM storage_backends WHERE id=$1 AND tenant_id=$2`,
id, t.scope.TenantID).Scan(&provider); err != nil {
return "", errors.New("目标实例没有这个存储后端,拒绝写入不可解析的资源定位符")
}
return provider, nil
}
func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryConfig, scope ProductionScope) (*ProductionLibraryTarget, error) { func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryConfig, scope ProductionScope) (*ProductionLibraryTarget, error) {
if err := config.validate(scope); err != nil { if err := config.validate(scope); err != nil {
return nil, err return nil, err
@@ -259,6 +282,7 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
if err := t.resolveStorageBackend(ctx); err != nil { if err := t.resolveStorageBackend(ctx); err != nil {
return err return err
} }
t.resolveEmbeddingModel(ctx)
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024) archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil { if err != nil {
return errors.New("归档不可用") return errors.New("归档不可用")
@@ -268,6 +292,9 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
if err != nil { if err != nil {
return errors.New("预发布快照不可读") return errors.New("预发布快照不可读")
} }
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
return err
}
switch stage { switch stage {
case libraryrelease.StageDatabase, libraryrelease.StageWiki: case libraryrelease.StageDatabase, libraryrelease.StageWiki:
kind := "database" kind := "database"
@@ -356,10 +383,10 @@ func (t *ProductionLibraryTarget) insertRows(ctx context.Context, tx pgx.Tx, arc
if len(row) == 0 { if len(row) == 0 {
continue continue
} }
if table == "resources" { // 行必须先映射到目标范围再落库:原样插入会把行写进来源租户/知识库,
if err := t.applyResourceBucket(row); err != nil { // 目标范围的删除、校验与恢复都看不见它们。
return err if err := t.retargetRow(table, row); err != nil {
} return err
} }
if columns == nil { if columns == nil {
columns = make([]string, 0, len(row)) columns = make([]string, 0, len(row))
@@ -408,53 +435,9 @@ type resourceLocation struct {
} }
// rowTransform 是"写入时对行做过的变换"的统一入口:校验要用同一份,否则摘要比的是 // rowTransform 是"写入时对行做过的变换"的统一入口:校验要用同一份,否则摘要比的是
// 变换前的行、目标里是变换后的行,永远不一致。 // 变换前的行、目标里是变换后的行,永远不一致。范围映射同样必须两边一致。
func (t *ProductionLibraryTarget) rowTransform(table string) func(map[string]json.RawMessage) error { func (t *ProductionLibraryTarget) rowTransform(table string) func(map[string]json.RawMessage) error {
if table == "resources" { return func(row map[string]json.RawMessage) error { return t.retargetRow(table, row) }
return t.applyResourceBucket
}
return nil
}
// applyResourceBucket 把资源行定位符里的桶换成本次写入的目标桶。
// 写入与校验必须用同一个变换:只改写入侧会让"行内容摘要"对不上(真跑踩过)。
func (t *ProductionLibraryTarget) applyResourceBucket(row map[string]json.RawMessage) error {
if t.config.ObjectsBucket == "" {
return nil
}
raw, ok := row["physical_path"]
if !ok {
return errors.New("资源行缺少定位符")
}
var locator string
if err := json.Unmarshal(raw, &locator); err != nil {
return errors.New("资源定位符不可解析")
}
rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket)
if !ok {
return fmt.Errorf("资源定位符无法改写到目标桶:%s", locator)
}
encoded, err := json.Marshal(rewritten)
if err != nil {
return err
}
row["physical_path"] = encoded
// 原版应用先按 resources.storage_backend_id 找存储后端,再回落到 KB,再落默认服务。
// 这一列留着来源 id,目标仍然 500(真跑:storage backend not found → invalid path)。
if t.storageBackend == "" {
return nil
}
if raw, ok := row["storage_backend_id"]; ok {
var current string
if json.Unmarshal(raw, &current) == nil && current != "" && current != t.storageBackend {
encodedBackend, err := json.Marshal(t.storageBackend)
if err != nil {
return err
}
row["storage_backend_id"] = encodedBackend
}
}
return nil
} }
func (t *ProductionLibraryTarget) importResources(ctx context.Context, archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error { func (t *ProductionLibraryTarget) importResources(ctx context.Context, archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error {
@@ -591,6 +574,7 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err := t.resolveStorageBackend(ctx); err != nil { if err := t.resolveStorageBackend(ctx); err != nil {
return err return err
} }
t.resolveEmbeddingModel(ctx)
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024) archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil { if err != nil {
return errors.New("归档不可用") return errors.New("归档不可用")
@@ -600,6 +584,9 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err != nil { if err != nil {
return errors.New("预发布快照不可读") return errors.New("预发布快照不可读")
} }
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
return err
}
for _, part := range manifest.Parts { for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" { if part.Kind != "database" && part.Kind != "wiki" {
continue continue
@@ -668,6 +655,72 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err := t.verifyObjects(ctx, archive, manifest); err != nil { if err := t.verifyObjects(ctx, archive, manifest); err != nil {
return err return err
} }
// 目标里不允许留下"快照来源范围"的行:那说明有写入逃出了目标范围,
// 而恢复只按目标范围回放,永远删不到它们(真实事故就是这样被漏掉的)。
if err := t.assertNoSourceScopeRows(ctx, archive, manifest); err != nil {
return err
}
return nil
}
// dropOrphanResources 清掉"本次恢复后已无任何归属"的目标租户资源行。
//
// 判据必须同时满足:属于目标租户、不在预备份快照里、没有任何 resource_binding、
// 也没有任何分块正文以 resource://<handle> 引用它。这样的行对目标应用完全不可见
// (没有知识库能按它取到文件),只可能是失败写入留下的残渣。
func (t *ProductionLibraryTarget) dropOrphanResources(ctx context.Context, backupSnapshotID string) error {
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil {
return errors.New("归档不可用")
}
defer archive.Close()
manifest, err := archive.Load(backupSnapshotID)
if err != nil {
return errors.New("预备份快照不可读")
}
kept := []string{}
for _, part := range manifest.Parts {
if part.Kind != "database" || part.Name != "resources" {
continue
}
ids, err := archivedResourceIDs(archive, part)
if err != nil {
return err
}
kept = append(kept, ids...)
}
rows, err := t.pool.Query(ctx, `
SELECT r.id FROM resources AS r
WHERE r.tenant_id=$1
AND NOT (r.id = ANY($2::text[]))
AND NOT EXISTS (SELECT 1 FROM resource_bindings b WHERE b.resource_id=r.id)
AND NOT EXISTS (SELECT 1 FROM chunks c WHERE c.tenant_id=r.tenant_id AND c.deleted_at IS NULL
AND coalesce(r.handle,'') <> ''
AND position('resource://'||r.handle IN coalesce(c.content,'')) > 0)`,
t.scope.TenantID, kept)
if err != nil {
return fmt.Errorf("查找目标残留资源失败: %w", err)
}
orphans := make([]string, 0, 8)
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
rows.Close()
return err
}
orphans = append(orphans, id)
}
rows.Close()
if err := rows.Err(); err != nil {
return err
}
if len(orphans) == 0 {
return nil
}
if _, err := t.pool.Exec(ctx, `DELETE FROM resources WHERE tenant_id=$1 AND id = ANY($2::text[])`,
t.scope.TenantID, orphans); err != nil {
return fmt.Errorf("清理目标残留资源失败: %w", err)
}
return nil return nil
} }
@@ -772,5 +825,21 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
return fmt.Errorf("恢复阶段 %s 失败: %w", stage, err) return fmt.Errorf("恢复阶段 %s 失败: %w", stage, err)
} }
} }
// resources 是多知识库共享表,回放阶段不删它:本次写入新增、恢复后已经
// 没有任何归属的资源行会残留。它们对应用不可见,但属于"多写的"。
if err := t.dropOrphanResources(ctx, backup.SnapshotID); err != nil {
return err
}
// 恢复工具把四个阶段都跑完不算恢复完成:整实例必须回到预备份时刻的样子。
// 只比目标范围会把"写到范围外"的残留当成已恢复(真实事故:恢复后仍多 67 行)。
if len(backup.Universe) > 0 {
after, err := universeFingerprint(ctx, t.pool)
if err != nil {
return fmt.Errorf("恢复后读取整库指纹失败: %w", err)
}
if diff := universeDiff(backup.Universe, after); diff != "" {
return fmt.Errorf("恢复后目标与预备份时刻不一致:%s(很可能有写入逃出了预备份范围,不能当作已恢复)", diff)
}
}
return nil return nil
} }
@@ -4,12 +4,15 @@ import (
"bytes" "bytes"
"context" "context"
"crypto/rand" "crypto/rand"
"crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"io" "io"
"net/http" "net/http"
"os" "os"
"strconv"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -390,3 +393,339 @@ func sealTargetFixture(t *testing.T, archive *librarysnapshot.Archive, scope str
} }
return id return id
} }
// memoryObjectStore 是"真实字节"的资源存储替身:Put/Open/Stat/Copy 都按摘要核对,
// 可以强制写入失败,用来复现"资源阶段失败"这一半写场景。
type memoryObjectStore struct {
objects map[string][]byte
failPut bool
puts []string
}
func newMemoryObjectStore(objects map[string][]byte) *memoryObjectStore {
copied := map[string][]byte{}
for key, value := range objects {
copied[key] = append([]byte(nil), value...)
}
return &memoryObjectStore{objects: copied}
}
func objectKey(provider, path string) string { return provider + "|" + path }
func (s *memoryObjectStore) Put(_ context.Context, provider, path string, reader io.Reader, size int64) error {
if s.failPut {
return errors.New("对象存储不可写")
}
body, err := io.ReadAll(reader)
if err != nil || int64(len(body)) != size {
return errors.New("对象字节数与声明不符")
}
s.objects[objectKey(provider, path)] = body
s.puts = append(s.puts, path)
return nil
}
func (s *memoryObjectStore) Open(_ context.Context, provider, path string) (io.ReadCloser, error) {
body, ok := s.objects[objectKey(provider, path)]
if !ok {
return nil, errors.New("对象不存在")
}
return io.NopCloser(bytes.NewReader(body)), nil
}
func (s *memoryObjectStore) Stat(_ context.Context, provider, path string) (int64, string, error) {
body, ok := s.objects[objectKey(provider, path)]
if !ok {
return 0, "", errors.New("对象不存在")
}
digest := sha256.Sum256(body)
return int64(len(body)), hex.EncodeToString(digest[:]), nil
}
func (s *memoryObjectStore) Copy(_ context.Context, provider, source, target string) error {
body, ok := s.objects[objectKey(provider, source)]
if !ok {
return errors.New("对象不存在")
}
s.objects[objectKey(provider, target)] = append([]byte(nil), body...)
return nil
}
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同)、目标非空、资源阶段写失败。
// 旧代码把归档行原样写进目标(落在来源租户/知识库),随后的回滚只清目标范围,
// 于是"已恢复到预备份"却留下 4 文件 / 38 分块 / 21 资源 / 4 绑定。
func TestLibraryCrossScopeSyncRollbackLeavesNoResidueIsolated(t *testing.T) {
dsn, base := os.Getenv("KG_LIBRARY_TEST_DATABASE_URL"), os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
if dsn == "" || base == "" || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
t.Skip("requires isolated PG and ES opt-in")
}
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") {
t.Fatal("isolated database required")
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
t.Fatalf("open isolated PG: %v", err)
}
defer pool.Close()
for _, statement := range []string{
`DROP SCHEMA public CASCADE; CREATE SCHEMA public`,
libraryTestSchema,
`ALTER TABLE wiki_pages ADD COLUMN status text DEFAULT 'published'`,
`ALTER TABLE knowledge_tag_relations ADD COLUMN tenant_id bigint`,
`ALTER TABLE resources ADD COLUMN provider text, ADD COLUMN size bigint, ADD COLUMN storage_backend_id text`,
`ALTER TABLE resources ADD PRIMARY KEY (id)`,
`ALTER TABLE resource_bindings ADD COLUMN relation text`,
`INSERT INTO knowledge_bases VALUES('kb-target',10001,'{}',NULL),('kb-source',10000,'{}',NULL)`,
`CREATE TABLE storage_backends(id text PRIMARY KEY, tenant_id bigint, provider text, config jsonb, created_at timestamptz)`,
`INSERT INTO storage_backends VALUES('backend-target',10001,'minio','{"bucket_name":"kg-test-bucket"}',now())`,
// 目标非空:目标自己有文档/分块,另一个租户也有数据(回滚不得碰它)。
`INSERT INTO knowledges VALUES('doc-target',10001,'kb-target','completed',0,'',NULL)`,
`INSERT INTO knowledges VALUES('doc-foreign',20000,'kb-other','completed',0,'',NULL)`,
`INSERT INTO chunks VALUES('chunk-target',10001,'kb-target','doc-target','target content',NULL)`,
`INSERT INTO chunks VALUES('chunk-foreign',20000,'kb-other','doc-foreign','foreign content',NULL)`,
`UPDATE resource_bindings SET tenant_id=10000 WHERE tenant_id IS NULL`,
} {
if _, err := pool.Exec(ctx, statement); err != nil {
t.Fatalf("fixture schema: %v", err)
}
}
token := make([]byte, 8)
if _, err := rand.Read(token); err != nil {
t.Fatal(err)
}
index := "kg_library_test_" + hex.EncodeToString(token)
client := &http.Client{Timeout: 20 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
call := func(method, path, body string, want int) {
t.Helper()
request, err := http.NewRequestWithContext(ctx, method, base+path, strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
request.Header.Set("Content-Type", "application/json")
response, err := client.Do(request)
if err != nil {
t.Fatalf("fixture HTTP: %v", err)
}
defer response.Body.Close()
if response.StatusCode != want {
t.Fatalf("fixture %s %s => %d, want %d", method, path, response.StatusCode, want)
}
}
call("PUT", "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"chunk_id":{"type":"keyword"},"content":{"type":"text"},"embedding":{"type":"dense_vector","dims":2}}}}`, 200)
defer func() {
call("DELETE", "/"+index, "", 200)
call("HEAD", "/"+index, "", 404)
}()
// 目标当前索引自洽:目标分块必须有对应文档,否则预备份会(正确地)拒绝。
call("PUT", "/"+index+"/_doc/chunk-target?refresh=true",
`{"tenant_id":10001,"knowledge_base_id":"kb-target","knowledge_id":"doc-target","chunk_id":"chunk-target","source_id":"chunk-target","content":"target content","embedding":[0.1,0.2]}`, 201)
archiveRoot := t.TempDir()
if err := os.Chmod(archiveRoot, 0700); err != nil {
t.Fatal(err)
}
archive, err := librarysnapshot.Open(archiveRoot, 64*1024*1024)
if err != nil {
t.Fatal(err)
}
defer archive.Close()
snapshotID := sealCrossScopeFixture(t, archive, 10000, "kb-source")
store := newMemoryObjectStore(nil)
target, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
Postgres: dsn, SearchURL: base, SearchIndex: index, Objects: store, ObjectsReader: store,
ArchiveRoot: archiveRoot, ObjectsBucket: "kg-test-bucket", MaxDocuments: 100, HTTPClient: client,
}, ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"})
if err != nil {
t.Fatalf("target: %v", err)
}
defer target.Close()
mapping, err := target.Search().MappingSHA256(ctx)
if err != nil {
t.Fatal(err)
}
backups, err := NewProductionBackupStore(ProductionBackupConfig{
Pool: target.Pool(), Search: target.Search(), Objects: store, ArchiveRoot: archiveRoot,
Scope: "internal", Identity: target.Identity(), MaxDocuments: 100,
Compatibility: librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74,
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "text-embedding-v4",
EmbeddingDimension: 1024, IndexMappingSHA256: mapping},
})
if err != nil {
t.Fatalf("backup store: %v", err)
}
now := time.Now().UTC()
request := libraryrelease.Request{Target: "internal", SnapshotID: snapshotID,
ParamsDigest: "digest-cross-scope", Now: now,
Grant: libraryrelease.Grant{Target: "internal", SnapshotID: snapshotID, ParamsDigest: "digest-cross-scope",
ExpiresAt: now.Add(10 * time.Minute), AuthorizedBy: "acceptance"}}
// 1) 资源阶段失败 → 必须回滚且不留下来源范围的行。
store.failPut = true
outcome := libraryrelease.Run(ctx, request, target, backups, nil)
if outcome.State != "rolled_back" {
t.Fatalf("outcome = %+v", outcome)
}
if outcome.FailedAt != libraryrelease.StageResources {
t.Fatalf("failed_at = %s, want resources", outcome.FailedAt)
}
if outcome.Backup.SnapshotID == "" || len(outcome.Backup.Universe) == 0 {
t.Fatalf("backup must carry the whole-instance fingerprint: %+v", outcome.Backup)
}
for _, table := range []string{"knowledges", "chunks", "wiki_pages"} {
var residue int64
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND knowledge_base_id=$2`, table),
10000, "kb-source").Scan(&residue); err != nil {
t.Fatal(err)
}
if residue != 0 {
t.Fatalf("%s still carries %d source-scope rows after rollback", table, residue)
}
}
// 夹具在租户 10000 里本来就有自己的行(res-safe / binding-safe):它们必须原样
// 留下,只有这次同步写进去的行(res-source / 绑到 doc-source)才算残留。
for query, want := range map[string]int64{
`SELECT count(*) FROM resources WHERE id='res-source'`: 0,
`SELECT count(*) FROM resource_bindings WHERE tenant_id=10000 AND owner_id='doc-source'`: 0,
`SELECT count(*) FROM resources WHERE tenant_id=10000`: 1,
} {
var got int64
if err := pool.QueryRow(ctx, query).Scan(&got); err != nil {
t.Fatal(err)
}
if got != want {
t.Fatalf("%s = %d, want %d", query, got, want)
}
}
var leftovers int64
if err := pool.QueryRow(ctx, `SELECT count(*) FROM resources WHERE tenant_id=10001`).Scan(&leftovers); err != nil {
t.Fatal(err)
}
if leftovers != 0 {
t.Fatalf("aborted write left %d resource rows in the target tenant", leftovers)
}
var foreign int64
if err := pool.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=20000`).Scan(&foreign); err != nil {
t.Fatal(err)
}
// 夹具本来就有 1 行 other-scope(20000),加上本用例的 chunk-foreign,共 2 行。
if foreign != 2 {
t.Fatalf("cross-tenant rows must survive rollback, got %d", foreign)
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: outcome.Backup.SnapshotID}); err != nil {
t.Fatalf("rollback must verify against the prebackup: %v", err)
}
// 2) 同样的跨范围同步,资源能写 → 行必须落在目标范围,资源定位符指向目标存储。
store.failPut = false
succeed := libraryrelease.Run(ctx, request, target, backups, nil)
if succeed.State != "succeeded" {
t.Fatalf("second run outcome = %+v", succeed)
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
t.Fatalf("verify after cross-scope sync: %v", err)
}
var knowledgeKB, chunkKB, provider, locator string
var tenant int64
if err := pool.QueryRow(ctx, `SELECT tenant_id, knowledge_base_id FROM knowledges WHERE id='doc-source'`).
Scan(&tenant, &knowledgeKB); err != nil {
t.Fatalf("source document must exist in the target scope: %v", err)
}
if tenant != 10001 || knowledgeKB != "kb-target" {
t.Fatalf("cross-scope row landed at %d/%s", tenant, knowledgeKB)
}
if err := pool.QueryRow(ctx, `SELECT knowledge_base_id FROM chunks WHERE id='chunk-source'`).Scan(&chunkKB); err != nil {
t.Fatal(err)
}
if chunkKB != "kb-target" {
t.Fatalf("chunk landed in %s", chunkKB)
}
if err := pool.QueryRow(ctx, `SELECT provider, physical_path FROM resources WHERE id='res-source'`).
Scan(&provider, &locator); err != nil {
t.Fatal(err)
}
if provider != "minio" {
t.Fatalf("resource provider = %s, want minio", provider)
}
if !strings.Contains(locator, "storage://backend-target/minio://kg-test-bucket/10001/") {
t.Fatalf("resource locator was not retargeted: %s", locator)
}
if strings.Contains(locator, "weknora/10000") {
t.Fatalf("resource locator still points at the source tenant: %s", locator)
}
t.Logf("cross_scope_sync verified: snapshot=%s backup=%s locator=%s", snapshotID, succeed.Backup.SnapshotID, locator)
}
// sealCrossScopeFixture 造一份"来源范围与目标不同"的整库快照:文档/分块/资源/绑定/
// Wiki/索引都带来源身份,资源部件是可核对的真实字节。
func sealCrossScopeFixture(t *testing.T, archive *librarysnapshot.Archive, sourceTenant uint64, sourceKB string) string {
t.Helper()
encode := func(value any) []byte {
encoded, err := json.Marshal(value)
if err != nil {
t.Fatal(err)
}
return encoded
}
const handle = "abcdefghijklmnopqrstuv"
objectBytes := []byte("12345678")
parts := make([]librarysnapshot.Part, 0, 12)
put := func(kind, name string, count int64, body []byte) {
part, err := archive.Put(bytes.NewReader(body))
if err != nil {
t.Fatal(err)
}
part.Kind, part.Name, part.Count = kind, name, count
parts = append(parts, part)
}
put("database", "knowledges", 1, encode([]map[string]any{{
"id": "doc-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
"parse_status": "completed", "pending_subtasks_count": 0, "file_path": ""}}))
put("database", "chunks", 1, encode([]map[string]any{{
"id": "chunk-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
"knowledge_id": "doc-source", "content": "见 resource://" + handle}}))
put("database", "knowledge_tags", 0, encode([]map[string]any{}))
put("database", "knowledge_tag_relations", 0, encode([]map[string]any{}))
put("database", "resources", 1, encode([]map[string]any{{
"id": "res-source", "handle": handle, "tenant_id": sourceTenant,
"storage_backend_id": "backend-source", "provider": "oss",
"physical_path": "storage://backend-source/oss://source-bucket/weknora/" +
formatTenant(sourceTenant) + "/res-source/a.png",
"size": int64(len(objectBytes))}}))
put("database", "resource_bindings", 1, encode([]map[string]any{{
"id": "bind-source", "resource_id": "res-source", "tenant_id": sourceTenant,
"owner_type": "knowledge", "owner_id": "doc-source", "relation": "attachment"}}))
put("resource", "res-source", 1, objectBytes)
put("wiki", "wiki_folders", 0, encode([]map[string]any{}))
put("wiki", "wiki_pages", 1, encode([]map[string]any{{
"id": "wiki-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
"slug": "index", "content": "wiki"}}))
for _, name := range []string{"wiki_page_issues", "wiki_log_entries"} {
put("wiki", name, 0, encode([]map[string]any{}))
}
put("index", "chunks", 1, encode([]map[string]any{{"id": "chunk-source", "source": map[string]any{
"tenant_id": sourceTenant, "knowledge_base_id": sourceKB, "knowledge_id": "doc-source",
"chunk_id": "chunk-source", "source_id": "chunk-source", "content": "见 resource://" + handle,
"embedding": []float64{0.1, 0.2}}}}))
manifest := librarysnapshot.Manifest{ValidationPolicy: librarysnapshot.ArtifactAcceptancePolicy,
Format: librarysnapshot.Format, Scope: "internal", TenantID: sourceTenant, KnowledgeBaseID: sourceKB,
CreatedAt: time.Now().UTC(), Parts: parts,
Compatibility: librarysnapshot.Compatibility{
WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8",
EmbeddingModel: "text-embedding-v4", EmbeddingDimension: 1024,
IndexMappingSHA256: strings.Repeat("a", 64),
}}
id, err := archive.Seal(manifest)
if err != nil {
t.Fatalf("seal cross-scope fixture: %v", err)
}
return id
}
func formatTenant(tenant uint64) string { return strconv.FormatUint(tenant, 10) }
@@ -0,0 +1,334 @@
package v071
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"knowledge-governance/internal/librarysnapshot"
)
// ResidueReport 是"清理范围外残留"的审计记录:清掉了什么、依据是哪份归档、
// 备份写到哪儿、清理后目标处于什么状态。没有这份记录就不许动目标。
type ResidueReport struct {
SnapshotID string `json:"snapshot_id"`
SourceTenant uint64 `json:"source_tenant"`
SourceKB string `json:"source_kb"`
Tables map[string]int `json:"tables"`
BackupPath string `json:"backup_path"`
BackupSHA256 string `json:"backup_sha256"`
Applied bool `json:"applied"`
Cleared int `json:"cleared"`
Verified bool `json:"verified"`
Foreign map[string]string `json:"foreign_scopes,omitempty"`
}
// ClearForeignScope 清掉"归档里确实有、却仍以来源身份躺在目标里"的行。
//
// 它只做这一件事,而且必须全部条件成立才允许删:
// 1. 目标这些行的内容与指定归档行逐字段一致(先证明它们是这次写入的副本,
// 不是目标自己的数据);
// 2. 先把它们完整导出成 0600 的备份文件并记下摘要;
// 3. 删除只按"归档身份集合 + 来源租户"精确命中,不做范围广删。
//
// 清完还要复核:来源身份不再存在、目标内容与预备份一致、整库指纹里没有别的租户。
func (t *ProductionLibraryTarget) ClearForeignScope(ctx context.Context, source ProductionScope, snapshotID,
backupDir string, apply bool) (*ResidueReport, error) {
if source.TenantID == 0 || !validSearchIdentifier(source.KnowledgeBaseID) {
return nil, errors.New("来源范围不完整,拒绝清理")
}
if source.TenantID == t.scope.TenantID && source.KnowledgeBaseID == t.scope.KnowledgeBaseID {
return nil, errors.New("来源与目标范围相同,不需要清理")
}
if backupDir == "" {
return nil, errors.New("清理前必须指定备份目录")
}
if err := t.resolveStorageBackend(ctx); err != nil {
return nil, err
}
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil {
return nil, errors.New("归档不可用")
}
defer archive.Close()
manifest, err := archive.Load(snapshotID)
if err != nil {
return nil, errors.New("指定快照不可读")
}
report := &ResidueReport{SnapshotID: snapshotID, SourceTenant: source.TenantID, SourceKB: source.KnowledgeBaseID,
Tables: map[string]int{}, Applied: false}
backup := map[string][]map[string]json.RawMessage{}
digests := map[string]string{}
identities := map[string][]string{}
for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" {
continue
}
archived, archivedDigests, err := archivedRowsWithDigests(archive, part)
if err != nil {
return nil, err
}
// 事故里的残留行是"归档行 + 只把桶/后端换成目标"的旧变换结果:判等时必须
// 同时接受原样与这一种变换,否则会把真残留当成"不是这次写的"而拒绝清理。
if part.Name == "resources" {
for _, row := range archived {
legacy, legacyErr := t.legacyResourceRow(row)
if legacyErr != nil {
return nil, legacyErr
}
digest, digestErr := canonicalRowDigest(legacy)
if digestErr != nil {
return nil, digestErr
}
archivedDigests[digest] = true
}
}
if len(archived) == 0 {
continue
}
live, err := t.foreignScopeRows(ctx, part.Name, source)
if err != nil {
return nil, err
}
liveDigests := make(map[string]string, len(live))
for _, row := range live {
digest, err := canonicalRowDigest(row)
if err != nil {
return nil, err
}
liveDigests[digest] = archiveRowIdentity(part.Name, row)
}
// 目标里这一范围的行必须都能在归档里找到同样内容:否则它们不是这次写入的
// 副本,删了就是删别人的数据。
rows := make([]map[string]json.RawMessage, 0, len(live))
ids := make([]string, 0, len(live))
for digest, identity := range liveDigests {
if _, ok := archivedDigests[digest]; !ok {
return nil, fmt.Errorf("目标 %s 的行 %s 与指定归档不一致,拒绝按本次写入清理", part.Name, identity)
}
ids = append(ids, identity)
}
for _, row := range live {
rows = append(rows, row)
}
sort.Strings(ids)
report.Tables[part.Name] = len(rows)
backup[part.Name] = rows
identities[part.Name] = ids
digests[part.Name] = combineRowDigests(sortedKeys(liveDigests))
}
total := 0
for _, count := range report.Tables {
total += count
}
if total == 0 {
report.Verified = true
return report, nil
}
if err := os.MkdirAll(backupDir, 0o700); err != nil {
return nil, fmt.Errorf("创建备份目录失败: %w", err)
}
payload, err := json.MarshalIndent(map[string]any{
"snapshot_id": snapshotID, "source_tenant": source.TenantID, "source_kb": source.KnowledgeBaseID,
"target_tenant": t.scope.TenantID, "target_kb": t.scope.KnowledgeBaseID,
"tables": report.Tables, "rows": backup, "digests": digests,
}, "", " ")
if err != nil {
return nil, err
}
report.BackupPath = filepath.Join(backupDir, "foreign-scope-residue-"+snapshotID[:12]+".json")
if err := os.WriteFile(report.BackupPath, payload, 0o600); err != nil {
return nil, fmt.Errorf("写残留备份失败: %w", err)
}
sum := sha256.Sum256(payload)
report.BackupSHA256 = hex.EncodeToString(sum[:])
if !apply {
return report, nil
}
// 删除顺序:子表先、父表后(resource_bindings → resources,chunks → knowledges)。
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
table := productionLibraryTables[index]
ids, ok := identities[table]
if !ok || len(ids) == 0 {
continue
}
statement := fmt.Sprintf(`DELETE FROM %s WHERE id = ANY($1::text[])`, table)
if !tableHasTenant(table) {
// knowledge_tag_relations 没有 id:身份就是父文档 + 标签。
statement = `DELETE FROM knowledge_tag_relations WHERE (knowledge_id||':'||tag_id) = ANY($1::text[])`
}
if _, err := t.pool.Exec(ctx, statement, ids); err != nil {
return nil, fmt.Errorf("清理 %s 失败: %w", table, err)
}
}
report.Applied = true
report.Cleared = total
return report, nil
}
// legacyResourceRow 复现旧实现写资源行时的变换:只换定位符里的存储后端与桶,
// 租户/知识库/provider 原样保留。判等要认这一种形状,事故残留才是"可证明"的。
func (t *ProductionLibraryTarget) legacyResourceRow(row map[string]json.RawMessage) (map[string]json.RawMessage, error) {
copied := make(map[string]json.RawMessage, len(row))
for key, value := range row {
copied[key] = value
}
if t.config.ObjectsBucket == "" {
return copied, nil
}
raw, ok := copied["physical_path"]
if !ok {
return nil, errors.New("资源行缺少定位符")
}
var locator string
if err := json.Unmarshal(raw, &locator); err != nil {
return nil, errors.New("资源定位符不可解析")
}
if rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket); ok {
encoded, err := json.Marshal(rewritten)
if err != nil {
return nil, err
}
copied["physical_path"] = encoded
}
if t.storageBackend == "" {
return copied, nil
}
if raw, ok := copied["storage_backend_id"]; ok {
var current string
if json.Unmarshal(raw, &current) == nil && current != "" && current != t.storageBackend {
encoded, err := json.Marshal(t.storageBackend)
if err != nil {
return nil, err
}
copied["storage_backend_id"] = encoded
}
}
return copied, nil
}
// VerifyResidueCleared 复核清理结果:来源身份不再存在、其它租户的解释得清、
// 且(可选)目标内容与预备份完全一致。
func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, source ProductionScope) error {
for _, table := range productionLibraryTables {
if !tableHasTenant(table) {
continue
}
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1`, table)
arguments := []any{source.TenantID}
if tablesWithKB[table] {
statement += " AND knowledge_base_id=$2"
arguments = append(arguments, source.KnowledgeBaseID)
}
var count int64
if err := t.pool.QueryRow(ctx, statement, arguments...).Scan(&count); err != nil {
return fmt.Errorf("复核 %s 失败: %w", table, err)
}
if count > 0 {
return fmt.Errorf("目标 %s 仍有 %d 行属于来源范围 %d/%s", table, count, source.TenantID, source.KnowledgeBaseID)
}
}
return nil
}
// ForeignScopes 列出目标里所有"不是目标租户"的租户,用于人工确认没有别的泄漏。
func (t *ProductionLibraryTarget) ForeignScopes(ctx context.Context) (map[string]string, error) {
foreign := map[string]string{}
for _, table := range productionLibraryTables {
if !tableHasTenant(table) {
continue
}
rows, err := t.pool.Query(ctx, fmt.Sprintf(
`SELECT tenant_id, count(*) FROM %s WHERE tenant_id <> $1 GROUP BY 1`, table), t.scope.TenantID)
if err != nil {
return nil, fmt.Errorf("读取 %s 租户分布失败: %w", table, err)
}
for rows.Next() {
var tenant, count int64
if err := rows.Scan(&tenant, &count); err != nil {
rows.Close()
return nil, err
}
foreign[fmt.Sprintf("%s:%d", table, tenant)] = fmt.Sprintf("%d 行", count)
}
rows.Close()
if err := rows.Err(); err != nil {
return nil, err
}
}
return foreign, nil
}
// foreignScopeRows 读出目标里属于来源范围的行(resources / resource_bindings
// 只有租户列,按租户取;其余按租户+知识库取)。
func (t *ProductionLibraryTarget) foreignScopeRows(ctx context.Context, table string, source ProductionScope) (
[]map[string]json.RawMessage, error) {
if !tableHasTenant(table) {
return nil, nil
}
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table))
arguments := []any{source.TenantID, source.KnowledgeBaseID}
if table == "resources" || table == "resource_bindings" {
statement = fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE tenant_id=$1`, table, table)
arguments = arguments[:1]
}
rows, err := t.pool.Query(ctx, statement, arguments...)
if err != nil {
return nil, fmt.Errorf("读取目标 %s 来源范围行失败: %w", table, err)
}
defer rows.Close()
result := make([]map[string]json.RawMessage, 0, 8)
for rows.Next() {
var row map[string]json.RawMessage
if err := rows.Scan(&row); err != nil {
return nil, err
}
result = append(result, row)
}
return result, rows.Err()
}
// archivedRowsWithDigests 读出一份归档部件的行与规范化内容摘要。
func archivedRowsWithDigests(archive *librarysnapshot.Archive, part librarysnapshot.Part) (
[]map[string]json.RawMessage, map[string]bool, error) {
reader, err := archive.OpenPart(part)
if err != nil {
return nil, nil, err
}
defer reader.Close()
decoder := json.NewDecoder(reader)
token, err := decoder.Token()
if err != nil || token != json.Delim('[') {
return nil, nil, errors.New("归档部件不是行数组")
}
rows := make([]map[string]json.RawMessage, 0, 64)
digests := map[string]bool{}
for decoder.More() {
var row map[string]json.RawMessage
if err := decoder.Decode(&row); err != nil {
return nil, nil, err
}
digest, err := canonicalRowDigest(row)
if err != nil {
return nil, nil, err
}
digests[digest] = true
rows = append(rows, row)
}
return rows, digests, nil
}
func sortedKeys(values map[string]string) []string {
keys := make([]string, 0, len(values))
for key := range values {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
@@ -0,0 +1,262 @@
package v071
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
)
// 整库同步是"把来源知识库的内容搬到目标的租户/知识库",但归档行带着来源身份
// (tenant_id / knowledge_base_id)。原样插入会让行落在来源范围里:目标范围的
// 删除、校验、预备份与恢复全都看不见它们(真实事故:同步失败后"已恢复到预备份",
// 目标却多出 4 文件 / 38 分块 / 21 资源 / 4 绑定,行所属租户还是来源租户)。
//
// 因此写入与校验必须共用这一个改写入口,并且对不上就失败——宁可不写,也不写
// 一行目标范围外的数据。
// tablesWithKB 是真实 WeKnora v0.7.1 里带 knowledge_base_id 的知识域表。
// resources / resource_bindings 是多知识库共享表,只有 tenant_id;
// knowledge_tag_relations 两列都没有,归属完全由父表 knowledges 决定。
var tablesWithKB = map[string]bool{
"knowledges": true, "chunks": true, "knowledge_tags": true,
"wiki_folders": true, "wiki_pages": true, "wiki_page_issues": true, "wiki_log_entries": true,
}
// tableHasTenant 报告这张表是否自带租户列。
func tableHasTenant(table string) bool { return table != "knowledge_tag_relations" }
// bindSnapshotScope 记下快照的来源身份。没有来源身份就无法判断"这行要搬到哪个
// 范围",宁可失败也不要按行内容猜。
func (t *ProductionLibraryTarget) bindSnapshotScope(manifest manifestScope) error {
if manifest.tenantID == 0 || !validSearchIdentifier(manifest.knowledgeBaseID) {
return errors.New("快照缺少来源范围身份(tenant_id / knowledge_base_id),无法确定目标范围映射")
}
t.snapshotTenant = manifest.tenantID
t.snapshotKB = manifest.knowledgeBaseID
return nil
}
// manifestScope 只是把快照的两个范围字段抽出来,避免调用点重复解构。
type manifestScope struct {
tenantID uint64
knowledgeBaseID string
}
// retargetRow 把一行归档数据改写成目标范围的数据:tenant_id / knowledge_base_id
// 换成目标身份,资源行还要把定位符、存储后端、提供方换成目标实例自己的值。
func (t *ProductionLibraryTarget) retargetRow(table string, row map[string]json.RawMessage) error {
if !tableHasTenant(table) {
return nil
}
if err := t.retargetTenantField(row); err != nil {
return fmt.Errorf("%s:%w", table, err)
}
if tablesWithKB[table] {
if err := t.retargetKnowledgeBaseField(row); err != nil {
return fmt.Errorf("%s:%w", table, err)
}
}
if table == "resources" {
return t.rewriteResourceIdentity(row)
}
if table == "knowledges" {
return t.retargetEmbeddingModelField(row)
}
return nil
}
// retargetEmbeddingModelField 把文档的 embedding 模型换成目标知识库绑定的那一个。
// 来源模型 id 在目标实例里通常不存在(目标 models 表可能是空的),留着它等于让
// 应用拿一个解析不到的模型去查——库里的向量与查询向量就不是同一套了。
func (t *ProductionLibraryTarget) retargetEmbeddingModelField(row map[string]json.RawMessage) error {
if t.targetEmbeddingModel == "" {
return nil
}
if _, ok := row["embedding_model_id"]; !ok {
return nil
}
encoded, err := json.Marshal(t.targetEmbeddingModel)
if err != nil {
return errors.New("编码目标 embedding 模型失败")
}
row["embedding_model_id"] = encoded
return nil
}
// resolveEmbeddingModel 读目标知识库自己的 embedding 模型 id。读不到不算错误
// (目标可能确实没绑模型),此时保持来源值不动,由契约校验去发现不一致。
func (t *ProductionLibraryTarget) resolveEmbeddingModel(ctx context.Context) {
var model string
if err := t.pool.QueryRow(ctx,
`SELECT coalesce(embedding_model_id,'') FROM knowledge_bases WHERE id=$1 AND tenant_id=$2`,
t.scope.KnowledgeBaseID, t.scope.TenantID).Scan(&model); err == nil {
t.targetEmbeddingModel = model
}
}
func (t *ProductionLibraryTarget) retargetTenantField(row map[string]json.RawMessage) error {
raw, ok := row["tenant_id"]
if !ok {
return errors.New("行缺少 tenant_id,无法确定要写入哪个范围")
}
var current uint64
if err := json.Unmarshal(raw, &current); err != nil {
return errors.New("行 tenant_id 不可解析")
}
if t.snapshotTenant == 0 {
return errors.New("快照来源租户未知,拒绝猜测目标租户")
}
if current != t.snapshotTenant {
return fmt.Errorf("行 tenant_id=%d 不属于快照来源租户 %d", current, t.snapshotTenant)
}
encoded, err := json.Marshal(t.scope.TenantID)
if err != nil {
return errors.New("编码目标租户失败")
}
row["tenant_id"] = encoded
return nil
}
func (t *ProductionLibraryTarget) retargetKnowledgeBaseField(row map[string]json.RawMessage) error {
raw, ok := row["knowledge_base_id"]
if !ok {
return errors.New("行缺少 knowledge_base_id,无法确定要写入哪个知识库")
}
var current string
if err := json.Unmarshal(raw, &current); err != nil || current == "" {
return errors.New("行 knowledge_base_id 不可解析")
}
if t.snapshotKB == "" {
return errors.New("快照来源知识库未知,拒绝猜测目标知识库")
}
if current != t.snapshotKB {
return fmt.Errorf("行 knowledge_base_id=%s 不属于快照来源知识库 %s", current, t.snapshotKB)
}
encoded, err := json.Marshal(t.scope.KnowledgeBaseID)
if err != nil {
return errors.New("编码目标知识库失败")
}
row["knowledge_base_id"] = encoded
return nil
}
// rewriteResourceIdentity 把资源行的定位符与提供方换成目标实例的值。
//
// 只换桶不够:来源行的 scheme(oss/minio…)与对象键里的租户段也是来源实例的,
// 目标实例按它取文件会落到别的提供方或别的租户前缀上(真跑:storage backend
// not found → invalid path → 500 code1007)。
func (t *ProductionLibraryTarget) rewriteResourceIdentity(row map[string]json.RawMessage) error {
raw, ok := row["physical_path"]
if !ok {
return errors.New("资源行缺少定位符")
}
var locator string
if err := json.Unmarshal(raw, &locator); err != nil {
return errors.New("资源定位符不可解析")
}
rewritten, err := retargetLocator(locator, t.storageBackend, t.storageProvider, t.config.ObjectsBucket,
t.snapshotTenant, t.scope.TenantID)
if err != nil {
return err
}
if rewritten != locator {
encoded, err := json.Marshal(rewritten)
if err != nil {
return errors.New("编码目标定位符失败")
}
row["physical_path"] = encoded
}
if t.storageBackend != "" {
if _, ok := row["storage_backend_id"]; ok {
encoded, err := json.Marshal(t.storageBackend)
if err != nil {
return errors.New("编码目标存储后端失败")
}
row["storage_backend_id"] = encoded
}
}
if t.storageProvider != "" {
if _, ok := row["provider"]; ok {
encoded, err := json.Marshal(t.storageProvider)
if err != nil {
return errors.New("编码目标提供方失败")
}
row["provider"] = encoded
}
}
return nil
}
// retargetLocator 把 "storage://<backend>/<scheme>://<bucket>/<key>" 整体换成目标
// 实例的真实取值:后端 id、scheme/提供方、桶名,以及对象键里的租户段。
// 桶名为空表示目标没有配置对象存储(测试夹具),此时不动定位符。
func retargetLocator(locator, backendID, provider, bucket string, sourceTenant, targetTenant uint64) (string, error) {
if bucket == "" || locator == "" {
return locator, nil
}
if locator != strings.TrimSpace(locator) {
return "", errors.New("资源定位符带首尾空白")
}
prefix := ""
rest := locator
if strings.HasPrefix(locator, "storage://") {
trimmed := strings.TrimPrefix(locator, "storage://")
parts := strings.SplitN(trimmed, "/", 2)
if len(parts) != 2 || !validSearchIdentifier(parts[0]) || parts[1] == "" {
return "", errors.New("资源定位符结构不可解析")
}
if backendID != "" {
if !validSearchIdentifier(backendID) {
return "", errors.New("目标存储后端 id 形状不合法")
}
parts[0] = backendID
}
prefix, rest = "storage://"+parts[0]+"/", parts[1]
}
separator := strings.Index(rest, "://")
if separator < 0 {
// 裸对象键没有桶与租户段:落盘按配置桶,改写无意义。
return locator, nil
}
scheme := rest[:separator]
if !validSearchIdentifier(scheme) {
return "", errors.New("资源定位符的提供方不可解析")
}
schemeless := rest[separator+3:]
boundary := strings.Index(schemeless, "/")
if boundary <= 0 || boundary == len(schemeless)-1 {
return "", errors.New("资源定位符缺少对象键")
}
key := schemeless[boundary+1:]
if provider != "" {
if !validSearchIdentifier(provider) {
return "", errors.New("目标提供方形状不合法")
}
scheme = provider
}
return prefix + scheme + "://" + bucket + "/" + retargetKeyTenant(key, sourceTenant, targetTenant), nil
}
// retargetKeyTenant 把对象键里的来源租户段换成目标租户,并丢掉来源实例自己的
// 路径前缀(例如 MINIO_PATH_PREFIX=weknora/),让键与目标实例的写法一致。
// 找不到来源租户段时原样返回:此时键本身不含跨租户引用。
func retargetKeyTenant(key string, sourceTenant, targetTenant uint64) string {
if sourceTenant == 0 || targetTenant == 0 || sourceTenant == targetTenant {
return key
}
source, target := strconv.FormatUint(sourceTenant, 10), strconv.FormatUint(targetTenant, 10)
segments := strings.Split(key, "/")
for index := 0; index < len(segments) && index < 2; index++ {
// 只有"后面还有内容"的段才是路径前缀;整键就是租户号时不该动它。
if index >= len(segments)-1 || segments[index] != source {
continue
}
kept := segments[index:]
kept[0] = target
return strings.Join(kept, "/")
}
return key
}
@@ -0,0 +1,194 @@
package v071
import (
"encoding/json"
"strings"
"testing"
)
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同),归档行原样写进目标,
// 目标范围的删除/校验/恢复都看不到它们。这里把"必须改写哪些列"钉死。
func scopeTargetFixture() *ProductionLibraryTarget {
return &ProductionLibraryTarget{
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"},
config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
storageBackend: "backend-target",
storageProvider: "minio",
snapshotTenant: 10000,
snapshotKB: "kb-source",
}
}
func rawRow(t *testing.T, fields map[string]any) map[string]json.RawMessage {
t.Helper()
row := make(map[string]json.RawMessage, len(fields))
for key, value := range fields {
encoded, err := json.Marshal(value)
if err != nil {
t.Fatalf("encode %s: %v", key, err)
}
row[key] = encoded
}
return row
}
func fieldText(t *testing.T, row map[string]json.RawMessage, key string) string {
t.Helper()
var value string
if err := json.Unmarshal(row[key], &value); err != nil {
t.Fatalf("%s not a string: %v", key, err)
}
return value
}
func fieldUint(t *testing.T, row map[string]json.RawMessage, key string) uint64 {
t.Helper()
var value uint64
if err := json.Unmarshal(row[key], &value); err != nil {
t.Fatalf("%s not a number: %v", key, err)
}
return value
}
func TestRetargetRowMapsSourceScopeToTarget(t *testing.T) {
target := scopeTargetFixture()
for _, table := range []string{"knowledges", "chunks", "knowledge_tags", "wiki_pages", "wiki_folders"} {
row := rawRow(t, map[string]any{"id": "row-1", "tenant_id": 10000, "knowledge_base_id": "kb-source"})
if err := target.retargetRow(table, row); err != nil {
t.Fatalf("%s retarget: %v", table, err)
}
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
t.Fatalf("%s tenant_id = %d, want 10001", table, got)
}
if got := fieldText(t, row, "knowledge_base_id"); got != "kb-target" {
t.Fatalf("%s knowledge_base_id = %s, want kb-target", table, got)
}
}
// 共享表只有租户列:不得凭空写一个 knowledge_base_id 出来。
binding := rawRow(t, map[string]any{"id": "bind-1", "tenant_id": 10000, "owner_type": "knowledge", "owner_id": "row-1"})
if err := target.retargetRow("resource_bindings", binding); err != nil {
t.Fatalf("resource_bindings retarget: %v", err)
}
if got := fieldUint(t, binding, "tenant_id"); got != 10001 {
t.Fatalf("binding tenant_id = %d, want 10001", got)
}
if _, ok := binding["knowledge_base_id"]; ok {
t.Fatal("resource_bindings must not gain a knowledge_base_id column")
}
// knowledge_tag_relations 没有范围列:归属走父表,改写必须是空操作。
relation := rawRow(t, map[string]any{"knowledge_id": "row-1", "tag_id": "tag-1"})
if err := target.retargetRow("knowledge_tag_relations", relation); err != nil {
t.Fatalf("knowledge_tag_relations retarget: %v", err)
}
if len(relation) != 2 {
t.Fatalf("knowledge_tag_relations must stay untouched, got %v", relation)
}
}
// 文档行的 embedding 模型必须换成目标知识库绑定的那一个:来源模型 id 在目标
// 实例里通常不存在,留着它检索时的查询向量与库里向量就不是同一套。
func TestRetargetRowUsesTargetEmbeddingModel(t *testing.T) {
target := scopeTargetFixture()
target.targetEmbeddingModel = "model-target"
row := rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
"embedding_model_id": "model-source"})
if err := target.retargetRow("knowledges", row); err != nil {
t.Fatalf("retarget: %v", err)
}
if got := fieldText(t, row, "embedding_model_id"); got != "model-target" {
t.Fatalf("embedding_model_id = %s, want model-target", got)
}
// 目标没绑模型时不猜:保持来源值,交给契约校验去发现不一致。
plain := scopeTargetFixture()
row = rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
"embedding_model_id": "model-source"})
if err := plain.retargetRow("knowledges", row); err != nil {
t.Fatalf("retarget without target model: %v", err)
}
if got := fieldText(t, row, "embedding_model_id"); got != "model-source" {
t.Fatalf("embedding_model_id = %s, want unchanged", got)
}
}
func TestRetargetRowFailsClosedOutsideSnapshotScope(t *testing.T) {
target := scopeTargetFixture()
cases := map[string]map[string]any{
"foreign tenant": {"id": "x", "tenant_id": 20000, "knowledge_base_id": "kb-source"},
"foreign knowledge": {"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb-other"},
"missing tenant": {"id": "x", "knowledge_base_id": "kb-source"},
"missing knowledge kb": {"id": "x", "tenant_id": 10000},
}
for name, fields := range cases {
if err := target.retargetRow("knowledges", rawRow(t, fields)); err == nil {
t.Fatalf("%s: retarget must fail closed", name)
}
}
// 快照来源未知时不许猜目标范围。
unknown := &ProductionLibraryTarget{scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"}}
if err := unknown.retargetRow("knowledges", rawRow(t, map[string]any{"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb"})); err == nil {
t.Fatal("unknown snapshot scope must fail closed")
}
}
func TestRetargetRowRewritesResourceStorageIdentity(t *testing.T) {
target := scopeTargetFixture()
row := rawRow(t, map[string]any{
"id": "res-1",
"tenant_id": 10000,
"storage_backend_id": "backend-source",
"provider": "oss",
"physical_path": "storage://backend-source/oss://yzh-weknora-test/weknora/10000/res-1/a.png",
})
if err := target.retargetRow("resources", row); err != nil {
t.Fatalf("retarget resource: %v", err)
}
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
t.Fatalf("resource tenant = %d, want 10001", got)
}
want := "storage://backend-target/minio://kgseal-1029a/10001/res-1/a.png"
if got := fieldText(t, row, "physical_path"); got != want {
t.Fatalf("physical_path = %q, want %q", got, want)
}
if got := fieldText(t, row, "provider"); got != "minio" {
t.Fatalf("provider = %q, want minio", got)
}
if got := fieldText(t, row, "storage_backend_id"); got != "backend-target" {
t.Fatalf("storage_backend_id = %q, want backend-target", got)
}
}
func TestRetargetKeyTenantDropsSourcePrefix(t *testing.T) {
cases := []struct{ key, want string }{
{"weknora/10000/exports/a.png", "10001/exports/a.png"},
{"10000/knowledge/file.md", "10001/knowledge/file.md"},
{"other/prefix/a.png", "other/prefix/a.png"},
{"10000", "10000"},
}
for _, tc := range cases {
if got := retargetKeyTenant(tc.key, 10000, 10001); got != tc.want {
t.Fatalf("retargetKeyTenant(%q) = %q, want %q", tc.key, got, tc.want)
}
}
// 同租户不编码:键必须原样返回。
if got := retargetKeyTenant("weknora/10000/a.png", 10000, 10000); got != "weknora/10000/a.png" {
t.Fatalf("same tenant must keep key, got %q", got)
}
}
func TestRetargetLocatorRejectsMalformed(t *testing.T) {
if _, err := retargetLocator(" not-a-locator ", "backend", "minio", "bucket", 10000, 10001); err == nil {
t.Fatal("locator with surrounding spaces must be rejected")
}
if _, err := retargetLocator("storage://backend/oss://", "backend", "minio", "bucket", 10000, 10001); err == nil {
t.Fatal("locator without object key must be rejected")
}
// 没有目标桶(测试夹具)时不改写,保持原值可继续用于采集/校验。
locator := "storage://backend/oss://bucket/key"
if got, err := retargetLocator(locator, "backend", "minio", "", 10000, 10001); err != nil || got != locator {
t.Fatalf("empty bucket must be a no-op, got %q err=%v", got, err)
}
if !strings.Contains(retargetKeyTenant("weknora/10000/a", 10000, 10001), "10001") {
t.Fatal("tenant segment must be replaced")
}
}
@@ -0,0 +1,126 @@
package v071
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
)
// universeFingerprint 是"整实例"的知识域指纹:不限租户/知识库的行数 + 身份集合摘要。
//
// 为什么不能只比目标范围:恢复只按目标范围删除/回放,写到范围外的残留它看不见
// (真实事故:作业报"已恢复到预备份",目标却多出 4 文件 / 38 分块 / 21 资源 /
// 4 绑定)。整实例指纹让"多写了什么"必然暴露,同时也能证明恢复没有误删别的
// 知识库的行。
func universeFingerprint(ctx context.Context, pool *pgxpool.Pool) (map[string]libraryrelease.TableFingerprint, error) {
result := make(map[string]libraryrelease.TableFingerprint, len(productionLibraryTables))
for _, table := range productionLibraryTables {
identity := identityExpr(table)
statement := fmt.Sprintf(
`SELECT count(*), coalesce(md5(string_agg((%s)::text, E'\n' ORDER BY (%s)::text)), '') FROM %s`,
identity, identity, table)
var fingerprint libraryrelease.TableFingerprint
if err := pool.QueryRow(ctx, statement).Scan(&fingerprint.Count, &fingerprint.Digest); err != nil {
return nil, fmt.Errorf("读取整库 %s 指纹失败: %w", table, err)
}
result[table] = fingerprint
}
return result, nil
}
// archivedRowIdentities 读出一份归档部件的行身份(与校验用的身份口径一致)。
func archivedRowIdentities(archive *librarysnapshot.Archive, part librarysnapshot.Part) ([]string, error) {
reader, err := archive.OpenPart(part)
if err != nil {
return nil, err
}
defer reader.Close()
decoder := json.NewDecoder(reader)
token, err := decoder.Token()
if err != nil || token != json.Delim('[') {
return nil, errors.New("归档部件不是行数组")
}
ids := make([]string, 0, 64)
for decoder.More() {
var row map[string]json.RawMessage
if err := decoder.Decode(&row); err != nil {
return nil, err
}
identity := archiveRowIdentity(part.Name, row)
if identity == "" {
continue
}
ids = append(ids, identity)
}
return ids, nil
}
// universeDiff 返回第一处不一致的表;空字符串表示整实例与参照完全一致。
func universeDiff(want, got map[string]libraryrelease.TableFingerprint) string {
tables := make([]string, 0, len(want))
for table := range want {
tables = append(tables, table)
}
sort.Strings(tables)
for _, table := range tables {
current, ok := got[table]
if !ok {
return fmt.Sprintf("%s(目标读不到)", table)
}
if current.Count != want[table].Count {
return fmt.Sprintf("%s 行数 %d,参照 %d", table, current.Count, want[table].Count)
}
if current.Digest != want[table].Digest {
return fmt.Sprintf("%s 身份集合与参照不同(同数量改写也会被这条拦住)", table)
}
}
return ""
}
// assertNoSourceScopeRows 找出"这次要写的这些行,仍然以来源身份躺在目标里"。
//
// 判据按归档里的身份集合逐表核对,而不是"这个租户/知识库有没有行":同一个租户
// 里本来就可能住着别的知识库(共享资源、别的 KB 的绑定),按范围数会把它们误
// 报成残留。这里只认"本次写集合里、仍属于来源租户"的行——它们只可能来自逃出
// 目标范围的写入,而恢复只按目标范围回放,永远删不到它们。
func (t *ProductionLibraryTarget) assertNoSourceScopeRows(ctx context.Context, archive *librarysnapshot.Archive,
manifest librarysnapshot.Manifest) error {
if t.snapshotTenant == 0 {
return nil
}
if t.snapshotTenant == t.scope.TenantID && t.snapshotKB == t.scope.KnowledgeBaseID {
return nil
}
for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" {
continue
}
if !tableHasTenant(part.Name) {
continue
}
ids, err := archivedRowIdentities(archive, part)
if err != nil {
return err
}
if len(ids) == 0 {
continue
}
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND id = ANY($2::text[])`, part.Name)
var count int64
if err := t.pool.QueryRow(ctx, statement, t.snapshotTenant, ids).Scan(&count); err != nil {
return fmt.Errorf("检查目标 %s 的来源范围残留失败: %w", part.Name, err)
}
if count > 0 {
return fmt.Errorf("目标 %s 残留 %d 行仍属于来源范围 %d/%s:有写入逃出了目标范围,恢复不会清掉它们",
part.Name, count, t.snapshotTenant, t.snapshotKB)
}
}
return nil
}
@@ -60,8 +60,11 @@ func TestRewriteLocatorSwapsStorageBackendWithBucket(t *testing.T) {
// 行变换用解析到的目标后端与目标桶:写入与校验共用这一份,摘要才不会两边打架。 // 行变换用解析到的目标后端与目标桶:写入与校验共用这一份,摘要才不会两边打架。
func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) { func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"}, target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
storageBackend: "target-backend"} storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
row := map[string]json.RawMessage{"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)} scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
row := map[string]json.RawMessage{
"tenant_id": json.RawMessage(`10001`),
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
if err := target.rowTransform("resources")(row); err != nil { if err := target.rowTransform("resources")(row); err != nil {
t.Fatalf("row transform failed: %v", err) t.Fatalf("row transform failed: %v", err)
} }
@@ -78,8 +81,10 @@ func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
// 只改定位符不改这一列,目标仍是 storage backend not found(真跑 500)。 // 只改定位符不改这一列,目标仍是 storage backend not found(真跑 500)。
func TestResourceRowTransformRewritesBackendColumn(t *testing.T) { func TestResourceRowTransformRewritesBackendColumn(t *testing.T) {
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"}, target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
storageBackend: "target-backend"} storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
row := map[string]json.RawMessage{ row := map[string]json.RawMessage{
"tenant_id": json.RawMessage(`10001`),
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/1/k/ab.md"`), "physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/1/k/ab.md"`),
"storage_backend_id": json.RawMessage(`"source-backend"`), "storage_backend_id": json.RawMessage(`"source-backend"`),
} }
+4
View File
@@ -356,6 +356,10 @@ def cmd_internal_web_test() -> int:
def cmd_internal_contracts(go: str) -> int: def cmd_internal_contracts(go: str) -> int:
"""版本契约:结构 + 注入身份必须**真 PASS**(SKIP 判红), """版本契约:结构 + 注入身份必须**真 PASS**(SKIP 判红),
并且**开发构建不许自称正式版**(负例:标成 release ⇒ 必须红)。""" 并且**开发构建不许自称正式版**(负例:标成 release ⇒ 必须红)。"""
qa_rc = subprocess.run(py("-m", "unittest", "discover", "-s", "scripts", "-p", "test_qa_env.py", "-v"),
cwd=ROOT).returncode
if qa_rc:
return qa_rc
snapshot_rc = subprocess.run(py("scripts/development_snapshot.py","--self-test"),cwd=ROOT).returncode snapshot_rc = subprocess.run(py("scripts/development_snapshot.py","--self-test"),cwd=ROOT).returncode
if snapshot_rc: if snapshot_rc:
return 1 return 1
+374
View File
@@ -0,0 +1,374 @@
#!/usr/bin/env python3
"""Prepare and exercise an isolated Knowledge Governance QA run."""
import argparse
from contextlib import ExitStack
import fcntl
import http.client
import json
import os
from pathlib import Path
import re
import shutil
import signal
import socket
import stat
import subprocess
import sys
import tempfile
import time
ROOT = Path(__file__).resolve().parents[1]
CONFIG = Path.home() / '.config/knowledge-governance/qa.json'
API_PORT = 8088
WEB_PORT = 5173
RUNTIME_RESPONSE = 'KG_QA_RUNTIME_OK'
class QAError(RuntimeError):
pass
def report(stage, detail):
print(f'[qa:{stage}] {detail}', flush=True)
def capture(argv, stage, timeout=10, env=None):
try:
return subprocess.run(argv, cwd=ROOT, env=env, text=True, capture_output=True,
timeout=timeout, stdin=subprocess.DEVNULL)
except FileNotFoundError as exc:
raise QAError(f'{stage} 缺少命令 {argv[0]}') from exc
except subprocess.TimeoutExpired as exc:
raise QAError(f'{stage} 超过 {timeout} 秒') from exc
def private_file(path):
try:
info = path.lstat()
except FileNotFoundError as exc:
raise QAError(f'缺少测试配置文件 {path}') from exc
if not stat.S_ISREG(info.st_mode) or info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) != 0o600:
raise QAError(f'{path} 必须是当前用户所有的 0600 普通文件')
def database_config(path):
private_file(path)
try:
config = json.loads(path.read_text())
except (UnicodeError, json.JSONDecodeError) as exc:
raise QAError(f'{path} 不是有效 JSON') from exc
if not isinstance(config, dict) or config.keys() != {'host', 'port', 'database', 'user'}:
raise QAError('qa.json 只接受 host、port、database、user 四个字段')
for field in ('database', 'user'):
if not re.fullmatch(r'kg_qa_[a-z0-9_]{1,56}', str(config[field])):
raise QAError(f'RDS {field} 必须以 kg_qa_ 开头并使用小写字母、数字或下划线')
host = config['host']
if not isinstance(host, str) or not re.fullmatch(r'[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+', host):
raise QAError('RDS host 必须填写证书对应的 DNS 主机名')
if type(config['port']) is not int or not 1 <= config['port'] <= 65535:
raise QAError('RDS port 必须是 1–65535 的整数')
return config
def tool_env():
allowed = ('HOME', 'PATH', 'LANG', 'TMPDIR', 'GOPROXY', 'GOSUMDB', 'GOPRIVATE', 'GONOSUMDB',
'GOMODCACHE', 'GOCACHE', 'npm_config_cache', 'npm_config_registry',
'HTTP_PROXY', 'HTTPS_PROXY', 'ALL_PROXY', 'NO_PROXY',
'http_proxy', 'https_proxy', 'all_proxy', 'no_proxy')
env = {key: os.environ[key] for key in allowed if key in os.environ}
env['GOFLAGS'] = '-mod=readonly'
return env
def database_env(config, directory):
# Keep inherited PG variables and integration switches out of application children.
env = tool_env()
env.update(PGHOST=config['host'], PGPORT=str(config['port']), PGDATABASE=config['database'],
PGUSER=config['user'], PGSSLMODE='verify-full', PGCONNECT_TIMEOUT='5',
PGPASSFILE=str(directory / 'qa.pgpass'), PGSSLROOTCERT=str(directory / 'rds-ca.pem'),
PGOPTIONS='-c search_path=public', LC_ALL='C',
TEST_WEKNORA_ENABLED='false', PP_SCAN_ENABLED='false',
RELEASE_WORKER_ENABLED='false', PRODUCTION_RELEASE_REQUESTS_ENABLED='false',
DOBOTAI_OPS_READ_ENABLED='false', DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED='false')
env['APP_ADDR'] = f'127.0.0.1:{API_PORT}'
env['DATA' + 'BASE_URL'] = (
f"postgresql://{config['user']}@{config['host']}:{config['port']}/{config['database']}"
'?sslmode=verify-full&connect_timeout=5&search_path=public'
)
return env
def psql(env, sql):
result = capture(['psql', '-XAtw', '-v', 'ON_ERROR_STOP=1', '-c', sql], 'RDS 探针', env=env)
if result.returncode == 0:
return result.stdout.strip()
detail = result.stderr.lower()
hints = (
('password', '口令认证失败,检查 qa.pgpass 和登录角色'),
('certificate', 'TLS 校验失败,检查 RDS 主机名和 CA'),
('does not exist', '数据库或角色不存在,联系管理员确认专用 QA 库'),
('permission denied', 'RDS 权限不足,检查专用 QA 角色的 CONNECT、schema USAGE/CREATE 权限'),
('could not translate host name', 'RDS 主机名无法解析,检查私网 DNS'),
('timeout', '连接 RDS 超时,检查 QA 虚拟机到 RDS 的私网和安全组'),
('connection refused', 'RDS 拒绝 TCP 连接,检查主机名、端口和私网接入'),
)
hint = next((message for marker, message in hints if marker in detail), '连接或 SQL 探针失败,检查 RDS 接入和权限')
raise QAError(f'RDS {hint}(psql 退出 {result.returncode})')
def database_state(env):
identity = psql(env, """
SELECT concat_ws('|', current_database(), current_user,
COALESCE((SELECT ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid()), false),
(SELECT NOT (rolsuper OR rolcreaterole OR rolcreatedb OR rolreplication OR rolbypassrls)
FROM pg_roles WHERE rolname = current_user),
(has_schema_privilege(current_user, 'public', 'USAGE') AND
has_schema_privilege(current_user, 'public', 'CREATE')),
to_regclass('public.schema_migrations') IS NOT NULL,
EXISTS (SELECT FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
WHERE n.nspname = 'public' AND c.relname <> 'schema_migrations')
OR EXISTS (SELECT FROM pg_namespace
WHERE nspname NOT IN ('public', 'information_schema') AND nspname !~ '^pg_')
OR EXISTS (SELECT FROM pg_proc p JOIN pg_namespace n ON n.oid=p.pronamespace
WHERE n.nspname = 'public'));
""")
fields = identity.split('|')
if len(fields) != 7 or fields[:4] != [env['PGDATABASE'], env['PGUSER'], 't', 't']:
raise QAError('RDS 实际库名、角色、TLS 或角色权限与 QA 配置不符')
if fields[4] != 't':
raise QAError('RDS 角色需要 public schema 的 USAGE 和 CREATE 权限')
rows = psql(env, 'SELECT version FROM public.schema_migrations ORDER BY version;') if fields[5] == 't' else ''
if any(not re.fullmatch(r'[0-9]+', row) for row in rows.splitlines()):
raise QAError('RDS 迁移记录包含无效版本')
return [int(row) for row in rows.splitlines()], fields[6] == 't'
def expected_migrations():
paths = (ROOT / 'internal/platform/postgres/migrations').glob('*.up.sql')
try:
versions = sorted(int(path.name.split('_', 1)[0]) for path in paths)
except ValueError as exc:
raise QAError('仓库存在无效的迁移文件名') from exc
if not versions or len(versions) != len(set(versions)):
raise QAError('仓库迁移文件缺失或版本重复')
return versions
def check_migrations(expected, applied, nonempty, *, complete):
if not applied and nonempty:
raise QAError('RDS 非空且缺少有效迁移记录,停止初始化')
if expected[:len(applied)] != applied:
raise QAError('RDS 迁移序列与当前提交不兼容,使用专用空库或匹配的测试库')
if complete and applied != expected:
raise QAError(f'RDS 迁移未完成,缺少版本 {expected[len(applied)]} 至 {expected[-1]}')
def check_hifox(output):
try:
state = json.loads(output)
online = state['connected'] is True and state['remoteLiveness']['status'] == 'online'
except (ValueError, KeyError, TypeError) as exc:
raise QAError('HiFox 未返回有效的在线状态,检查本机注册和接收服务') from exc
if not online:
raise QAError('HiFox 未确认测试机在线,检查本机接收服务和网络')
def check_checkout(expected):
if not re.fullmatch(r'[0-9a-f]{40}', expected):
raise QAError('设置 KG_QA_SHA 为提测的完整 40 位 Git SHA')
actual = capture(['git', 'rev-parse', 'HEAD'], 'Git 检出')
if actual.returncode or actual.stdout.strip() != expected:
raise QAError('Git 检出与提测提交不一致')
state = capture(['git', 'status', '--porcelain'], 'Git 状态')
if state.returncode or state.stdout.strip():
raise QAError('QA 需要干净的独立检出,请检查 Git 状态')
def preflight(env):
for cmd in ('git', 'make', 'go', 'node', 'npm', 'psql', 'hifox', 'omp'):
if not shutil.which(cmd, path=env.get('PATH')):
raise QAError(f'工具链缺少 {cmd},检查 QA 机器的 PATH')
go = capture(['go', 'env', 'GOROOT', 'GOVERSION'], 'Go 工具链', env=env)
fields = go.stdout.splitlines()
if go.returncode or len(fields) != 2 or not fields[1].startswith('go1.26.'):
raise QAError('QA 需要 Go 1.26 工具链')
root = Path(fields[0]) / 'bin'
if not os.access(root / 'gofmt', os.X_OK):
raise QAError('Go 工具链缺少 gofmt,请安装完整的 Go 1.26')
env['PATH'] = f"{root}:{env['PATH']}"
node = capture(['node', '--version'], 'Node.js 工具链', env=env)
if node.returncode or not node.stdout.startswith('v22.'):
raise QAError('QA 需要 Node.js 22 工具链')
report('toolchain', f'{fields[1]} / Node.js {node.stdout.strip()}')
hifox = capture(['hifox', 'computer', 'status', '--output', 'json'], 'HiFox 在线检查', timeout=15)
if hifox.returncode:
raise QAError(f'HiFox 状态命令失败(退出 {hifox.returncode})')
check_hifox(hifox.stdout)
report('machine', f'{socket.gethostname()} / HiFox online')
prompt = f'Reply with exactly {RUNTIME_RESPONSE}. Do not use tools.'
omp = capture(['omp', '--no-session', '-p', prompt], 'OMP Runtime 探针', timeout=35)
if omp.returncode or RUNTIME_RESPONSE not in {line.strip() for line in omp.stdout.splitlines()}:
raise QAError('OMP Runtime 没有返回探针结果,检查 OMP 模型配置和 Magpie 网关')
report('runtime', 'OMP 模型请求通过')
def check_ports():
for port in (API_PORT, WEB_PORT):
with socket.socket() as listener:
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
listener.bind(('127.0.0.1', port))
except OSError as exc:
raise QAError(f'QA 端口 {port} 无法绑定,检查已有监听进程') from exc
def run_step(argv, stage, env, timeout):
report(stage, ' '.join(argv[:2]))
process = subprocess.Popen(argv, cwd=ROOT, env=env, stdin=subprocess.DEVNULL, start_new_session=True)
try:
code = process.wait(timeout=timeout)
except subprocess.TimeoutExpired as exc:
raise QAError(f'{stage} 超过 {timeout} 秒') from exc
finally:
stop_process(process)
if code:
raise QAError(f'{stage} 失败(退出 {code}),查看上方命令输出')
def stop_process(process):
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
pass
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
pass
process.wait(timeout=5)
def start_service(stack, argv, env, log):
output = stack.enter_context(log.open('w', encoding='utf-8'))
process = subprocess.Popen(argv, cwd=ROOT, env=env, stdin=subprocess.DEVNULL,
stdout=output, stderr=subprocess.STDOUT, start_new_session=True)
stack.callback(stop_process, process)
return process
def request(port, path):
connection = http.client.HTTPConnection('127.0.0.1', port, timeout=2)
try:
connection.request('GET', path)
response = connection.getresponse()
return response.status, response.getheader('Content-Type', ''), response.read(32768)
finally:
connection.close()
def probe(port, path, expected):
try:
code, content_type, body = request(port, path)
if code != 200:
return False
if expected == 'html':
return 'text/html' in content_type and b'id="root"' in body
return 'application/json' in content_type and json.loads(body)['status'] == expected
except (OSError, http.client.HTTPException, ValueError, KeyError, TypeError):
return False
def wait_service(process, port, path, expected, log):
deadline = time.monotonic() + 45
while time.monotonic() < deadline:
if process.poll() is not None:
raise QAError(f'服务端口 {port} 启动进程提前退出(退出 {process.returncode}),日志 {log}')
if probe(port, path, expected):
return
time.sleep(0.25)
raise QAError(f'服务探针超时 127.0.0.1:{port}{path},日志 {log}')
def qa_run(commit, env, db_env, versions, state):
with tempfile.TemporaryDirectory(prefix='run-', dir=state) as directory, ExitStack() as services:
binary = Path(directory) / 'api'
run_step(['make', 'deps-install'], 'deps', env, 300)
ldflags = f'-X knowledge-governance/internal/version.CommitSHA={commit}'
run_step(['go', 'build', '-ldflags', ldflags, '-o', str(binary), './cmd/api'], 'build-api', env, 300)
check_ports()
log_dir = Path(tempfile.mkdtemp(prefix=f'{commit[:12]}-', dir=state))
report('logs', str(log_dir))
app_env = db_env | {'PATH': env['PATH'], 'GOVERNANCE_SPOOL_ROOT': str(Path(directory) / 'spool')}
api_log = log_dir / 'api.log'
api = start_service(services, [str(binary)], app_env, api_log)
wait_service(api, API_PORT, '/readyz', 'ready', api_log)
check_migrations(versions, *database_state(db_env), complete=True)
report('migration', f'已核对全部 {len(versions)} 条迁移,最新版本 {versions[-1]}')
web_log = log_dir / 'web.log'
web = start_service(services, ['npm', '--prefix', 'web', 'run', 'dev', '--', '--host', '127.0.0.1',
'--port', str(WEB_PORT), '--strictPort'], env, web_log)
wait_service(web, WEB_PORT, '/', 'html', web_log)
wait_service(web, WEB_PORT, '/health', 'ok', web_log)
report('services', 'API /readyz、Web 首页与 Vite /health 代理通过')
run_step(['make', 'tests-go', 'tests-web'], 'tests', env, 300)
check_checkout(commit)
if api.poll() is not None or web.poll() is not None or not probe(WEB_PORT, '/health', 'ok'):
raise QAError(f'服务在测试期间失去健康状态,日志目录 {log_dir}')
report('result', f'PASS {commit},已停止本次 API 和 Web 进程')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('doctor', 'run'))
args = parser.parse_args()
env = tool_env()
try:
commit = os.environ.get('KG_QA_SHA', '')
if args.action == 'run':
check_checkout(commit)
check_ports()
preflight(env)
config_path = Path(os.environ.get('KG_QA_CONFIG', CONFIG)).expanduser().absolute()
config = database_config(config_path)
private_file(config_path.parent / 'qa.pgpass')
if not (config_path.parent / 'rds-ca.pem').is_file():
raise QAError(f'缺少 RDS CA 文件 {config_path.parent / "rds-ca.pem"}')
db_env = database_env(config, config_path.parent)
versions = expected_migrations()
if args.action == 'doctor':
applied, nonempty = database_state(db_env)
check_migrations(versions, applied, nonempty, complete=False)
report('database', f'连接与身份正常,已应用 {len(applied)}/{len(versions)} 条迁移')
return 0
state = Path.home() / '.local/state/knowledge-governance/qa'
state.mkdir(mode=0o700, parents=True, exist_ok=True)
with (state / 'run.lock').open('a') as lock:
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
raise QAError('本机已有 QA 任务,等待当前任务退出') from exc
check_ports()
check_migrations(versions, *database_state(db_env), complete=False)
qa_run(commit, env, db_env, versions, state)
return 0
except (QAError, OSError, ValueError) as exc:
print(f'[qa:FAIL] {exc}', file=sys.stderr)
return 1
except KeyboardInterrupt:
print('[qa:FAIL] QA 任务中断', file=sys.stderr)
return 130
if __name__ == '__main__':
def interrupt(_signal, _frame):
raise KeyboardInterrupt
signal.signal(signal.SIGTERM, interrupt)
sys.exit(main())
+99
View File
@@ -0,0 +1,99 @@
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from threading import Thread
from pathlib import Path
import subprocess
import tempfile
import unittest
from unittest.mock import patch
import qa_env
class QAEnvironmentTests(unittest.TestCase):
def test_missing_approval_config_names_file(self):
with tempfile.TemporaryDirectory() as directory:
config = Path(directory) / 'qa.json'
with self.assertRaisesRegex(qa_env.QAError, 'qa.json'):
qa_env.database_config(config)
def test_rejects_non_qa_database_before_connection(self):
with tempfile.TemporaryDirectory() as directory:
config = Path(directory) / 'qa.json'
config.write_text(json.dumps(dict(host='rds.example.test', port=5432,
database='governance', user='kg_qa_app')))
config.chmod(0o600)
with self.assertRaisesRegex(qa_env.QAError, 'kg_qa_'):
qa_env.database_config(config)
def test_database_environment_uses_only_approved_identity(self):
settings = dict(host='rds.example.test', port=5432, database='kg_qa_app', user='kg_qa_app')
with patch.dict(os.environ, {'PGHOSTADDR': '127.0.0.1', 'PGPASSWORD': 'fixture',
'PGSERVICE': 'other', 'PRODUCTION_RELEASE_REQUESTS_ENABLED': 'true'}):
env = qa_env.database_env(settings, Path('/approved'))
self.assertEqual(env['PGHOST'], 'rds.example.test')
self.assertEqual(env['PGSSLMODE'], 'verify-full')
self.assertEqual(env['PRODUCTION_RELEASE_REQUESTS_ENABLED'], 'false')
self.assertFalse({'PGHOSTADDR', 'PGPASSWORD', 'PGSERVICE'} & env.keys())
def test_empty_database_or_known_migration_prefix_can_bootstrap(self):
for applied in ([], [1], [1, 2]):
with self.subTest(applied=applied):
qa_env.check_migrations([1, 2], applied, False, complete=False)
def test_pending_migrations_block_testing(self):
with self.assertRaisesRegex(qa_env.QAError, '迁移未完成'):
qa_env.check_migrations([1, 2], [1], False, complete=True)
def test_schema_without_ledger_is_rejected(self):
with self.assertRaisesRegex(qa_env.QAError, '非空'):
qa_env.check_migrations([1, 2], [], True, complete=False)
def test_unknown_or_missing_middle_migrations_are_rejected(self):
for applied in ([1, 3], [1, 2, 3], [2]):
with self.subTest(applied=applied), self.assertRaisesRegex(qa_env.QAError, '迁移序列'):
qa_env.check_migrations([1, 2], applied, False, complete=False)
def test_hifox_exit_zero_does_not_override_offline_state(self):
with self.assertRaisesRegex(qa_env.QAError, '在线'):
qa_env.check_hifox('{"connected":true,"remoteLiveness":{"status":"offline"}}')
qa_env.check_hifox('{"connected":true,"remoteLiveness":{"status":"online"}}')
def test_health_probe_rejects_spa_fallback(self):
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write(b'<html><div id="root"></div></html>')
def log_message(self, format, *args):
pass
with ThreadingHTTPServer(('127.0.0.1', 0), Handler) as server:
thread = Thread(target=server.serve_forever)
thread.start()
try:
self.assertTrue(qa_env.probe(server.server_port, '/', 'html'))
self.assertFalse(qa_env.probe(server.server_port, '/health', 'ok'))
finally:
server.shutdown()
thread.join()
def test_database_identity_mismatch_stops_before_reading_migrations(self):
env = {'PGDATABASE': 'kg_qa_app', 'PGUSER': 'kg_qa_app'}
with patch.object(qa_env, 'psql', return_value='other|kg_qa_app|t|t|t|t|t') as probe:
with self.assertRaisesRegex(qa_env.QAError, '实际库名'):
qa_env.database_state(env)
self.assertEqual(probe.call_count, 1)
def test_full_sha_mismatch_rejects_current_checkout(self):
full_sha = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=qa_env.ROOT, text=True).strip()
wrong = ('0' if full_sha[0] != '0' else '1') + full_sha[1:]
with self.assertRaisesRegex(qa_env.QAError, '提交不一致'):
qa_env.check_checkout(wrong)
if __name__ == '__main__':
unittest.main()