Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 07a0ee2644 | |||
| 0b5de6ef7e | |||
| d7d636792d | |||
| 89c484cdc4 | |||
| 5e3362e2b0 | |||
| edd07a65d6 | |||
| 7c120b3adc | |||
| ea159ca2fb | |||
| 034c0b3a1c | |||
| 5defe238e0 |
@@ -0,0 +1,76 @@
|
||||
name: deploy-production
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
action:
|
||||
description: deploy 或 writeback
|
||||
required: true
|
||||
default: deploy
|
||||
sha:
|
||||
description: 验收过的完整提交 SHA
|
||||
required: true
|
||||
bundle:
|
||||
description: release.json 的完整 SHA256
|
||||
required: true
|
||||
task:
|
||||
description: 对应的 HiFox 任务 UUID
|
||||
required: true
|
||||
confirm:
|
||||
description: deploy-production 或 retry-writeback
|
||||
required: true
|
||||
|
||||
concurrency:
|
||||
group: cloud-test-governance-main-production
|
||||
cancel-in-progress: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
guard:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: 核对手动入口及发布输入
|
||||
run: |
|
||||
set -eu
|
||||
test "$GITHUB_EVENT_NAME" = workflow_dispatch
|
||||
test "$GITHUB_REPOSITORY" = Weknora/knowledge-governance
|
||||
test "$GITHUB_REF" = refs/heads/main
|
||||
case "$GITHUB_SERVER_URL" in
|
||||
https://gitea.zheromiku.top|http://192.168.50.223:3000) ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
git init -q .
|
||||
git fetch -q --depth 200 "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git" refs/heads/main
|
||||
git checkout -q --detach "$GITHUB_SHA"
|
||||
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
|
||||
python3 scripts/production_cd.py guard --event "$GITHUB_EVENT_PATH"
|
||||
|
||||
publish:
|
||||
needs: [guard]
|
||||
runs-on: kg-production-cd
|
||||
timeout-minutes: 40
|
||||
steps:
|
||||
- name: 取同一提交的控制端代码
|
||||
run: |
|
||||
set -eu
|
||||
test "$GITHUB_EVENT_NAME" = workflow_dispatch
|
||||
test "$GITHUB_REPOSITORY" = Weknora/knowledge-governance
|
||||
test "$GITHUB_REF" = refs/heads/main
|
||||
case "$GITHUB_SERVER_URL" in
|
||||
https://gitea.zheromiku.top|http://192.168.50.223:3000) ;;
|
||||
*) exit 1 ;;
|
||||
esac
|
||||
git init -q .
|
||||
git fetch -q --depth 200 "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git" refs/heads/main
|
||||
git checkout -q --detach "$GITHUB_SHA"
|
||||
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
|
||||
- name: 调用云主站受限发布入口并读回回执
|
||||
env:
|
||||
KG_CD_SSH_KEY: ${{ secrets.KG_CD_SSH_KEY }}
|
||||
KG_CD_KNOWN_HOSTS: ${{ secrets.KG_CD_KNOWN_HOSTS }}
|
||||
KG_CD_SSH_HOST: ${{ vars.KG_CD_SSH_HOST }}
|
||||
KG_CD_SSH_PORT: ${{ vars.KG_CD_SSH_PORT }}
|
||||
run: python3 scripts/production_cd.py send --event "$GITHUB_EVENT_PATH"
|
||||
@@ -27,6 +27,14 @@ BUILD_TIME ?= $(shell date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
verify:
|
||||
$(PY) scripts/ci_checks.py
|
||||
|
||||
.PHONY: qa qa-doctor qa-self-test
|
||||
qa:
|
||||
$(PY) scripts/qa_env.py run
|
||||
qa-doctor:
|
||||
$(PY) scripts/qa_env.py doctor
|
||||
qa-self-test:
|
||||
$(PY) -m unittest discover -s scripts -p test_qa_env.py -v
|
||||
|
||||
# 清单:哪些是必需项、在哪个档案(local / ci)下必需
|
||||
list:
|
||||
$(PY) scripts/ci_checks.py --list
|
||||
|
||||
@@ -71,15 +71,16 @@ IT、行政等是待扩展的业务资料范围,不是已经完成的新连接
|
||||
|
||||
## 访问与部署
|
||||
|
||||
### 已部署环境
|
||||
### 运行架构与访问入口
|
||||
|
||||
Compose 中网页入口固定映射为 **宿主机 `8088` → Web 容器 `8080`**。网页和 `/api/` 使用同源入口。
|
||||
| 组件 | 职责 | 接手时从这里看 |
|
||||
|---|---|---|
|
||||
| React/Vite + Web Nginx | 提供静态页面,同源转发 `/api/` 和 `/health` | [前端开发代理](web/vite.config.ts)、[Compose 的 Nginx 配置](deploy/nginx.conf) |
|
||||
| Go API + 治理 PostgreSQL | 认证、角色、来源、绑定、审核、任务与发布记录。云测试使用独立逻辑库 `knowledge_governance_test` | [API](cmd/api)、[接口契约](api/openapi.yaml) |
|
||||
| 普通 worker + 测试 WeKnora | worker 取件、上传、轮询并记录验收结果。WeKnora 负责解析、切片和索引 | [worker](cmd/worker)、[来源接入](docs/OPERATIONS_RUNBOOK.md#接入外部平台) |
|
||||
| 发布 worker + 生产 WeKnora | 按已锁定的批次和目标执行受控发布与读回。问答端通过下游服务使用已发布知识 | [发布 worker](cmd/release-worker)、[操作手册](docs/OPERATIONS_RUNBOOK.md) |
|
||||
|
||||
- 本机访问:`http://127.0.0.1:8088/`
|
||||
- 局域网访问:`http://<服务器内网IP>:8088/`
|
||||
- 健康入口:`http://127.0.0.1:8088/health`
|
||||
|
||||
可以由 NAS 反向代理提供 HTTPS 域名入口,目标为平台服务器的 `8088`,保持路径原样转发。公网开放前必须验证鉴权、访问控制及转发头配置,不直接暴露数据库或内部管理接口。
|
||||
从自己的电脑做系统设置验收,使用 SSH 转发后的 `http://127.0.0.1:28189/`。云测试主控制台使用受限内网入口 `http://100.85.111.121:8088/`。仓内 Compose 则将宿主 `8088` 映射到 Web 容器 `8080`。选择入口时先确认要看的部署。隧道命令、生产 `090` 服务地址和 HTTPS 反代示例见[运维手册](docs/OPERATIONS_RUNBOOK.md#入口与反向代理)。
|
||||
|
||||
**平台入口展示当前部署版本,不是每次 CI 的构建预览。** CI 静态预览由独立链路提供;CI 通过不会自动更新运行服务。
|
||||
|
||||
@@ -94,13 +95,7 @@ Compose 中网页入口固定映射为 **宿主机 `8088` → Web 容器 `8080`*
|
||||
- 来源目录、spool 与运行状态挂载;按部署主机调整路径和 UID/GID。
|
||||
- 测试 WeKnora 连接与对内/对外路由;生产发布按独立配置和授权接线。
|
||||
|
||||
准备完成后,由获授权的操作者执行:
|
||||
|
||||
```bash
|
||||
docker compose up --build -d
|
||||
```
|
||||
|
||||
主机需要提权时,由操作者按本机权限规则执行,不在自动化中默认使用 sudo。停止服务可用 `docker compose down`;**不要执行 `docker compose down -v`,除非明确批准删除数据库卷。**
|
||||
仓内 Compose 含 `postgres` 演练服务。云测试和生产使用阿里云 RDS,部署前先核对目标逻辑库、当前迁移版本、待运行的迁移及所用的环境配置。API 和 worker 启动可执行迁移。已有环境先用 `docker compose config --services` 检查服务清单,按该环境的发布手册切换已验证的产物。前端开发可直接使用下面的 Vite 流程。
|
||||
|
||||
## 配置与安全边界
|
||||
|
||||
@@ -114,7 +109,7 @@ docker compose up --build -d
|
||||
|
||||
## 开发与 CI
|
||||
|
||||
基础环境:Go 1.26、Node.js 22、npm,以及检查脚本所需的 Python 3 和工具链。依赖以仓库锁文件和实际 CI 配置为准。
|
||||
本机和 QA 虚拟机的依赖安装、前后端启动、测试命令及 PostgreSQL 测试库准备见 [开发与测试指南](docs/DEVELOPMENT_AND_TESTING.md)。
|
||||
|
||||
```bash
|
||||
make verify # 完整检查入口:本地与 CI 使用同一套规则
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
// command libraryresidue 处理"整库同步写到目标范围外"留下的残留行:
|
||||
//
|
||||
// 读指定快照(来源范围)→ 在目标里找同内容、仍带来源身份的行
|
||||
// → 先备份成 0600 文件并记录摘要 → 精确按归档身份删除 → 复核来源身份已空、
|
||||
// 目标内容与预备份一致、整库指纹里没有别的租户
|
||||
//
|
||||
// 它不做范围广删,也不改任何别的东西:判据是"归档里确实有这些行、内容逐字段
|
||||
// 一致",任何一行对不上就整体拒绝。默认是干跑(只报告、不删)。
|
||||
//
|
||||
// 真实事故:跨范围同步把 4 文件/38 分块/21 资源/4 绑定写成来源租户的行,
|
||||
// 回滚只按目标范围回放,于是"已恢复"却留下这些行。这个工具的用途就是在
|
||||
// 修复后的代码上线前,用可审计的方式清掉它们。
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"knowledge-governance/internal/appconfig"
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
"knowledge-governance/internal/librarystack"
|
||||
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||
)
|
||||
|
||||
func main() {
|
||||
scope := flag.String("scope", "internal", "internal|external")
|
||||
snapshot := flag.String("snapshot", "", "sealed snapshot whose source scope is the residue (required)")
|
||||
backupSnapshot := flag.String("backup-snapshot", "", "prebackup snapshot the target must be restored to (required)")
|
||||
expectSource := flag.String("expect-source", "", "expected source scope as <tenant>/<knowledge-base> (required guard against typos)")
|
||||
out := flag.String("out", "", "directory for the residue backup (required)")
|
||||
apply := flag.Bool("apply", false, "actually delete the residue rows (default: dry run)")
|
||||
reportPath := flag.String("report", "", "write the JSON report here")
|
||||
flag.Parse()
|
||||
if *snapshot == "" || *backupSnapshot == "" || *out == "" || *expectSource == "" {
|
||||
log.Fatal("snapshot, backup-snapshot, out and expect-source are required")
|
||||
}
|
||||
config, err := appconfig.Load(os.Getenv)
|
||||
if err != nil {
|
||||
log.Fatalf("load config: %v", err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
report, err := run(ctx, config, *scope, *snapshot, *backupSnapshot, *expectSource, *out, *apply)
|
||||
if *reportPath != "" && report != nil {
|
||||
raw, marshalErr := json.MarshalIndent(report, "", " ")
|
||||
if marshalErr == nil {
|
||||
raw = append(raw, '\n')
|
||||
if writeErr := os.WriteFile(*reportPath, raw, 0o600); writeErr != nil {
|
||||
log.Printf("report write failed: %v", writeErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
log.Fatalf("residue cleanup failed: %v", err)
|
||||
}
|
||||
log.Printf("residue cleanup ok: source=%s/%s tables=%v cleared=%d applied=%v verified=%v",
|
||||
reportSource(report), report.SourceKB, report.Tables, report.Cleared, report.Applied, report.Verified)
|
||||
}
|
||||
|
||||
func reportSource(report *v071.ResidueReport) string {
|
||||
return strconv.FormatUint(report.SourceTenant, 10)
|
||||
}
|
||||
|
||||
func run(ctx context.Context, config appconfig.Config, scope, snapshotID, backupSnapshotID, expectSource, out string,
|
||||
apply bool) (*v071.ResidueReport, error) {
|
||||
if scope != "internal" && scope != "external" {
|
||||
return nil, fmt.Errorf("scope must be internal or external")
|
||||
}
|
||||
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("归档不可用: %w", err)
|
||||
}
|
||||
manifest, err := archive.Load(snapshotID)
|
||||
archive.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("指定快照不可读: %w", err)
|
||||
}
|
||||
source := v071.ProductionScope{TenantID: manifest.TenantID, KnowledgeBaseID: manifest.KnowledgeBaseID}
|
||||
want := strconv.FormatUint(source.TenantID, 10) + "/" + source.KnowledgeBaseID
|
||||
// 范围写错会把"别人的行"当成残留:必须由调用方显式确认。
|
||||
if strings.TrimSpace(expectSource) != want {
|
||||
return nil, fmt.Errorf("来源范围与 -expect-source 不一致:归档为 %s,参数为 %s", want, expectSource)
|
||||
}
|
||||
target, _, err := librarystack.OpenProductionStack(ctx, config, scope, backupSnapshotID, "")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("装配目标失败: %w", err)
|
||||
}
|
||||
defer target.Close()
|
||||
report, err := target.ClearForeignScope(ctx, source, snapshotID, out, apply)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
report.Foreign, _ = target.ForeignScopes(ctx)
|
||||
if !apply {
|
||||
return report, nil
|
||||
}
|
||||
if err := target.VerifyResidueCleared(ctx, source); err != nil {
|
||||
return report, err
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: backupSnapshotID}); err != nil {
|
||||
return report, fmt.Errorf("清理后目标内容仍与预备份不一致(保持停写): %w", err)
|
||||
}
|
||||
report.Verified = true
|
||||
if len(report.Foreign) > 0 {
|
||||
raw, _ := json.Marshal(report.Foreign)
|
||||
return report, fmt.Errorf("清理后目标仍有其它租户的数据,需人工确认才能开放:%s", raw)
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
# 知识治理平台开发与测试
|
||||
|
||||
以下命令从仓库根目录运行。日常开发可在 Ubuntu 宿主进行,指定提交的回归在 `hifox-qa` 的独立检出中运行。部署到阿里云测试服后的验收按 [操作 Runbook](OPERATIONS_RUNBOOK.md) 和 [用户流程验收](USER_FLOWS_ACCEPTANCE.md) 执行。
|
||||
|
||||
## QA 虚拟机一键回归
|
||||
|
||||
在 `hifox-qa` 检出提测提交后,运行下列入口。将 SHA 占位值换成提测记录里的完整提交 SHA。
|
||||
|
||||
```bash
|
||||
make qa-doctor
|
||||
KG_QA_SHA='<完整提交 SHA>' make qa
|
||||
```
|
||||
|
||||
`make qa-doctor` 检查工具链、HiFox 机器在线状态、OMP 模型调用、测试库身份和已安装的迁移。`make qa` 安装锁定依赖,在专用测试库上通过 API 初始化和迁移,随后核对完整迁移序列、API 就绪状态、Web 首页及 Vite 健康代理。通过后执行 Go 和 Vitest 测试,结束时关闭本次启动的服务。每次运行的服务日志保存在 `~/.local/state/knowledge-governance/qa/`。需要提交前完整门禁时执行 `make verify`。
|
||||
|
||||
初次配置由负责人在 QA 虚拟机准备 `~/.config/knowledge-governance/`。以下示例使用虚构的 RDS 主机名。
|
||||
|
||||
| 文件 | 内容 |
|
||||
| --- | --- |
|
||||
| `qa.json` | 当前用户所有、权限 `0600`,字段 `host`、`port`、`database`、`user` |
|
||||
| `qa.pgpass` | 当前用户所有、权限 `0600`,填写与 `qa.json` 匹配的口令条目 |
|
||||
| `rds-ca.pem` | 对应实例的 CA 证书,用于校验 RDS 主机名 |
|
||||
|
||||
```json
|
||||
{"host":"rds.example.test","port":5432,"database":"kg_qa_app","user":"kg_qa_app"}
|
||||
```
|
||||
|
||||
专用非生产 RDS 库和角色以 `kg_qa_` 开头,权限和网络准备见第 2 节。首次运行接受空库,后续运行接受与当前提交兼容的迁移序列。缺库、认证失败、迁移不兼容和端口冲突均会使命令非零退出。浏览器业务 E2E 和专用集成库的 Go 测试按第 4 节执行。
|
||||
|
||||
## 1. 检出代码与安装依赖
|
||||
|
||||
准备 Git、Make、Python 3、Go 1.26(含同一工具链的 `gofmt`)、Node.js 22 和 npm。数据库检查另需 PostgreSQL 客户端 `psql`。首次安装依赖需要能访问 Go 和 npm 的依赖源。
|
||||
|
||||
```bash
|
||||
# QA 在独立克隆中拉取提测分支,使用提测记录里的完整提交 SHA
|
||||
export KG_TARGET_BRANCH='<提测分支名>' KG_TARGET_SHA='<完整提交 SHA>'
|
||||
git fetch origin "$KG_TARGET_BRANCH" && git switch --detach "$KG_TARGET_SHA" &&
|
||||
test "$(git rev-parse HEAD)" = "$KG_TARGET_SHA" && test -z "$(git status --porcelain)"
|
||||
# 上面的命令链若失败,先解决检出或工作区问题再继续
|
||||
|
||||
# 检查当前 shell 的工具链,非交互任务也应设置相同的 PATH
|
||||
export PATH="$(go env GOROOT)/bin:$PATH"
|
||||
go version
|
||||
command -v gofmt
|
||||
node --version
|
||||
npm --version
|
||||
python3 --version
|
||||
make deps-install
|
||||
```
|
||||
|
||||
`make deps-install` 通过 `go mod download` 和 `npm ci` 安装依赖。Go 依赖由 `go.mod` / `go.sum` 约束,前端由 `web/package-lock.json` 锁定。依赖安装失败时先检查工具链、依赖源和锁文件,避免临时改写清单来绕过问题。
|
||||
|
||||
## 2. 准备开发和测试数据库
|
||||
|
||||
无库检查可直接跳到第 4 节。启动 API 或执行真库测试前,由数据库管理员在**非生产 PostgreSQL RDS** 上预建专用空库和登录角色。建议分配 `kg_qa_app` 给应用冒烟,`kg_qa_it` 给 Go 集成测试,各用独立角色。开发宿主另用开发专用库,避免与 QA 共享数据。库未分配时先申请,启动流程以库已存在为前提。
|
||||
|
||||
| 用途 | 所需权限 | 数据初始化 |
|
||||
| --- | --- | --- |
|
||||
| 应用启动与 UI 冒烟 | `CONNECT`,以及目标 schema 内的 `USAGE`、`CREATE` 和对应用对象的读写权限。迁移角色须能修改自己创建的对象。 | 首次 API 启动执行仓库内嵌迁移。用户表为空时创建初始管理员,首次登录必须修改密码。初始口令通过测试环境负责人交接。 |
|
||||
| Go PostgreSQL 集成测试 | `CONNECT` 和目标测试库的 `CREATE` 权限,用于建立各用例自己的 schema。 | 用例创建隔离 schema、执行迁移、写入测试数据并清理。相同测试不要并发指向同一个库,避免固定 schema 名冲突。 |
|
||||
|
||||
两类角色均只授予对应非生产库所需的权限。使用空库和合成数据。密码由操作者在目标机配置到权限为 `0600` 的 PostgreSQL passfile,条目按 `host:port:database:user:password` 填写。使用 RDS CA 校验证书。以下命令先读回实际数据库、角色与 TLS 状态,核对三者再启动 API 或运行真库测试。
|
||||
|
||||
```bash
|
||||
# 用获准的 RDS 主机名、端口、库名、角色和文件路径替换示例值
|
||||
export PGHOST='<非生产 RDS 主机名>' PGPORT=5432
|
||||
export PGDATABASE=kg_qa_app PGUSER=kg_qa_app
|
||||
export PGPASSFILE="$HOME/.config/knowledge-governance/qa.pgpass"
|
||||
export PGSSLROOTCERT="$HOME/.config/knowledge-governance/rds-ca.pem"
|
||||
export PGSSLMODE=verify-full PGCONNECT_TIMEOUT=5
|
||||
psql -X -v ON_ERROR_STOP=1 -c \
|
||||
'SELECT current_database(), current_user, ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid();'
|
||||
```
|
||||
|
||||
如果是云厂商提供的私网地址,先确认执行机通过获准的网络路径可访问 RDS。库名、角色、TLS、连接权限或迁移权限有问题时,先交由环境负责人核对配置。实例上已有数据的库不能当作空库初始化。
|
||||
|
||||
## 3. 启动后端和前端
|
||||
|
||||
**终端 A,启动 API。** 在干净 shell 中执行第 2 节的应用库配置和连通性检查。使用独立测试配置,不加载现网 `.env`。
|
||||
|
||||
```bash
|
||||
export DATABASE_URL="postgresql://${PGUSER}@${PGHOST}:${PGPORT}/${PGDATABASE}?sslmode=verify-full&connect_timeout=5"
|
||||
export APP_ADDR=127.0.0.1:8088
|
||||
export TEST_WEKNORA_ENABLED=false PP_SCAN_ENABLED=false
|
||||
export RELEASE_WORKER_ENABLED=false PRODUCTION_RELEASE_REQUESTS_ENABLED=false
|
||||
export DOBOTAI_OPS_READ_ENABLED=false DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED=false
|
||||
go run ./cmd/api
|
||||
```
|
||||
|
||||
API 会在监听前连接 PostgreSQL、应用迁移并初始化管理员。Web 页面和同步 API 联调只需运行 API 与 Vite。需要验证后台队列时,另用相同的隔离应用库配置启动普通 worker(`go run ./cmd/worker`)。涉及取件或来源刷新的任务还依赖容器内 `/app/scripts` 及获准的来源接线,按部署说明准备。发布 worker 无需启动。
|
||||
|
||||
Vite 通过 `localhost:8088` 转发 `/api` 和健康检查。如果 `8088` 已被其他部署占用,使用端口空闲的 QA 虚拟机运行本节。仅改前端时,可复用一个获准联调的测试 API,但需要先确认 `localhost:8088/health` 可达,并核对该地址指向的环境。
|
||||
|
||||
**终端 B,启动 Web。**
|
||||
|
||||
```bash
|
||||
npm --prefix web run dev -- --host 127.0.0.1 --port 5173 --strictPort
|
||||
```
|
||||
|
||||
在运行服务的机器打开 `http://127.0.0.1:5173/`。另开终端检查 API 与前端代理,再复查迁移记录。最后一条命令需要第 2 节的应用库配置。
|
||||
|
||||
```bash
|
||||
curl --fail --silent --show-error http://127.0.0.1:8088/health
|
||||
curl --fail --silent --show-error http://127.0.0.1:5173/health
|
||||
psql -X -v ON_ERROR_STOP=1 -c \
|
||||
'SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1;'
|
||||
```
|
||||
|
||||
空库首次启动后,用专用测试账号登录,按当次验收标准验证页面及接口。联调结束在两个终端按 `Ctrl+C` 停服务,数据库由环境管理员按测试数据保留规则管理。
|
||||
|
||||
## 4. 选择测试命令
|
||||
|
||||
| 目的 | 命令 |
|
||||
| --- | --- |
|
||||
| 查看检查及适用范围 | `make list` |
|
||||
| 后端包测试 | `make tests-go` |
|
||||
| 单个 Go 包及相关子包 | `go test -count=1 ./internal/httpapi/...` |
|
||||
| 前端 Vitest 套件 | `make tests-web` |
|
||||
| 单个前端测试文件 | `npm --prefix web test -- src/App.test.tsx` |
|
||||
| 完整提交前检查,与 CI 共用入口 | `make verify` |
|
||||
|
||||
`make verify` 包含安装、格式、静态检查、构建、测试、版本和安全检查,并写出 `ci-report.json`。在未导入真库测试配置的 shell 中运行。若检出时用了 `--no-tags`,先执行 `git fetch --tags origin`,以便版本检查读取正式标签对应的提交。依赖安装与安全检查需要相应网络或缓存。允许跳过的外部依赖用例见 [test-scope.json](../scripts/test-scope.json),真库测试应单独执行并记录结果。
|
||||
|
||||
在应用冒烟之外验证真实 PostgreSQL 行为时,重新按第 2 节切换并核对 **集成测试库和角色**,然后执行目标用例。下面以 UI 设置版本切换用例为例。
|
||||
|
||||
```bash
|
||||
TEST_DATABASE_URL="postgresql://${PGUSER}@${PGHOST}:${PGPORT}/${PGDATABASE}?sslmode=verify-full&connect_timeout=5" \
|
||||
go test -count=1 -v ./internal/platform/postgres -run '^TestUISettingsActivationRetiresPreviousRevisionIntegration$'
|
||||
```
|
||||
|
||||
检查输出出现目标用例的 `PASS`。用例被跳过或没有匹配时,应补齐配置或纠正用例名后重跑。前端 Vitest 负责组件与业务逻辑检查,浏览器 E2E 应单独执行并按 Feature 保存证据。提测记录至少包括完整提交 SHA、执行命令、退出码,以及失败用例的日志或浏览器截图。
|
||||
@@ -3,7 +3,88 @@
|
||||
按"实际干活顺序"排列的**现在的**操作流程。标注:**【新】**本期新增(含本次会话与 `f4e1ecc/6496ee3/9155bd3` 的 dws 取件)|**【变】**既有流程本期有改动|**【既有】**未变,仍照旧走。
|
||||
逐条验收证据请看 `docs/USER_FLOWS_ACCEPTANCE.md`;细节规则看各自专题文档。
|
||||
|
||||
统一入口 **http://127.0.0.1:8088/**(测试栈,右侧面板可直接操作)。角色:管理员 / 文档工程师 / 只读。
|
||||
管理员配置连接和发布授权,文档工程师维护资料,只读成员查看进度。先确定要操作的环境,再按下面的接线与验收步骤进入业务流程。
|
||||
|
||||
云测试治理主站自身的应用升级见[主站应用 CD](PRODUCTION_CD_RUNBOOK.md)。
|
||||
|
||||
## 入口与反向代理
|
||||
|
||||
| 请求从哪里发出 | 地址或路径 | 实际目的地 |
|
||||
|---|---|---|
|
||||
| 已接入内网的运维电脑 | `http://100.85.111.121:8088/` | 云测试治理主站 |
|
||||
| 开发电脑的浏览器 | `http://127.0.0.1:28189/` | SSH 转发到测试机回环 `28189`,由候选 Web 提供静态资源 |
|
||||
| 测试候选 Nginx(host 网络) | `/api/`、`/health` | 测试机 `127.0.0.1:18099` 候选 API |
|
||||
| 同一候选 Nginx | `/api/v1/qa-ops/ai/` | 测试机 `127.0.0.1:18098` 现用治理 API,负责模型配置和连接验证 |
|
||||
| 标准 Compose Web(容器网络) | `/api/`、`/health` | Docker DNS 解析 `api:8080`。静态资源和 SPA 回退由同一 Web 提供 |
|
||||
|
||||
**在开发电脑恢复验收入口。** 前提是 SSH 别名 `weknora-test` 能登录测试机。先查远端服务,再查本机监听。本机端口空闲且远端正常时,在独立终端运行第三条命令,保留该终端。在另一终端执行最后一条并用浏览器打开同一地址。
|
||||
|
||||
```bash
|
||||
ssh weknora-test 'curl -fsS --max-time 5 http://127.0.0.1:28189/health'
|
||||
ss -ltn 'sport = :28189'
|
||||
ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -L 127.0.0.1:28189:127.0.0.1:28189 weknora-test
|
||||
curl -fsS --max-time 5 http://127.0.0.1:28189/health
|
||||
```
|
||||
|
||||
**为内网域名配置 HTTPS。** 在能访问测试主站的受信任反代机上配置证书、内部 DNS 和访问范围。以下片段放入已有 HTTPS `server`,完整保留 URI、Host 和会话 Cookie。上游应选治理 Web 入口,使 `/assets/`、`/api/`、`/health` 走同一域名。若该入口的内层代理也设置 `X-Forwarded-Proto`,联调时检查其保留受信任 TLS 终止层的协议值以及登录回调的实际 URL。
|
||||
|
||||
```nginx
|
||||
location / {
|
||||
proxy_pass http://100.85.111.121:8088;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
```
|
||||
|
||||
在反代机执行 `nginx -t` 校验待应用配置。入口启用后,通过该域名验证登录、一次 `/api/` 请求和首页实际引用的 JS/CSS。`index.html` 应重新校验,带哈希的资源可长期缓存,缺失的 JS 返回 404。候选切换前后分别记录通用上游和 AI 专用上游。现场可在测试机用以下只读命令查看,容器名以当时部署为准。仓内同源配置见 [deploy/nginx.conf](../deploy/nginx.conf)。
|
||||
|
||||
```bash
|
||||
docker exec kg-test-storage-ui-6444814 nginx -T 2>&1 | grep -E 'listen .*28189|location /api/v1/qa-ops/ai/|set \$(api|health)_upstream|proxy_pass http://127[.]0[.]0[.]1:18098'
|
||||
docker inspect -f '{{.Name}} {{index .Config.Labels "release"}}' kg-test-storage-ui-6444814 kg-accept-090-api
|
||||
```
|
||||
|
||||
**现场版本与回退。** 本次在测试机读到 Web 标签 `4b615abe534b`、API 标签 `341c3154ac91`。完整候选包的 `current` 链接位于 `/opt/knowledge-governance/acceptance/090/current`,单独切换的 Web 位于同级 `web-only/4b615abe534b/dist`,实际反代配置位于 `integration/28189.conf`。切换前记录这三处、容器标签与旧 Web 名称。静态页回退时恢复上次的 Web 资源和对应代理配置。完整候选回退按同批装配工具的 `deploy/accept-090/README.md` 执行。回退后用浏览器重新验证首页资源、模型连接和列库。数据库迁移另按备份与恢复方案处理。
|
||||
|
||||
本次已核实共享测试治理库到 `000082`。对内、对外独立启用的实现依赖 `000083`,待独立库联调或共享库迁移获准后,再验证单侧暂停、保存和刷新。生产 `070` 不在该套验收范围。
|
||||
|
||||
下文系统设置中的绑定、盘点和模型连接步骤适用于上表的测试候选版本。根据现场标签选择对应版本的代码和交付物。
|
||||
|
||||
## 接入外部平台
|
||||
|
||||
| 平台与用途 | 管理员或运维怎样接 | 接通的证据 |
|
||||
|---|---|---|
|
||||
| 钉钉知识库,取目录和文件 | 系统设置 → 数据源配置 → 钉钉授权。用 DWS 设备授权,以来源账号选择组织、空间和范围。API 与 worker 使用部署约定的 DWS 身份目录 | 登录后读回预期空间,选一个有权限的目录校验。登记来源、扫描完成和文件取件分别核验。 |
|
||||
| AList,取工业或教育资料 | 按来源登记服务根地址和允许的目录。普通 worker 读取专用令牌文件。目录接口 `POST /api/fs/list` 使用 `X-Gateway-Token` | 在来源页扫描已知根目录,对照一个子目录和一份文件。下载遵循来源签名链接,凭据和签名 URL 仅留在受控运行环境。[目录与取件](SOURCE_CATALOG_LIVE_ROUTING_ACCEPTANCE.md) |
|
||||
| PP 资料门户 | 在数据源配置填写 HTTPS 根地址和来源账号,点「测试连接」后「保存并校验」。API 和 worker 共享 `PP_AUTH_STATE_FILE` 所在目录 | 真实登录并读取一条目录记录,刷新后核对生效配置。超时或限流时查看待验证状态。 |
|
||||
| WeKnora 测试 090,处理与验收 | 系统设置 → 知识库绑定 → 测试 090。当前测试 API 地址 `http://127.0.0.1:8092`,由治理服务器访问。输入该实例的专用 Key 或使用已配置凭据 | 「验证并读取知识库」能列出预期库。核对名称和 ID,为对内、对外选择不同的目标,保存后刷新读回 |
|
||||
| WeKnora 生产 090,列库与映射 | 在同页切到生产 090,使用下节的转发入口与生产专用 Key。新连接的安全存储由 `WEKNORA_BINDING_MASTER_KEY_FILE` 提供 | 读取生产目录并核对两个目标 ID,保存后重新进入生产页核实。执行发布另按 §5–6 授权 |
|
||||
| 问答 AI 模型 | 系统设置 → AI 模型连接,填写 OpenAI 兼容基础地址(含服务要求的 `/v1`)、模型名及专用 Key。API 与普通 worker 使用同一 `QA_AI_MASTER_KEY_FILE` | 点「测试连接」等待成功,需生效时再保存并刷新检查模型名。存储密钥、权限与 DNS 配置见 [README](../README.md#问答运营模型配置) |
|
||||
|
||||
**生产 090 的两段地址。** 生产机私网 IP 为 `172.29.151.11`,其 WeKnora API 对宿主回环发布 `127.0.0.1:18092`。测试治理机上已有 SSH 本地转发,治理 API 使用 host 网络,故页面的生产服务地址填写 **`http://127.0.0.1:18092`**。请求从治理 API 发出,经测试机的回环监听到生产机的回环 API。浏览器所在电脑的 `127.0.0.1:28189` 则用于打开治理页面。迁到 bridge 网络时要从新 API 容器重新验证上游可达性。
|
||||
|
||||
在**测试治理机**上检验已有通道。`/health` 返回 200 证明转发可用,无凭据列库返回 401 表明请求已到认证层。接着在治理页面安全输入该实例的 Key 并验证真实库列表。本次核验时,生产通道探活成功,匿名列库返回 401。带生产凭据列库和保存映射仍待验收。
|
||||
|
||||
```bash
|
||||
ss -ltnp 'sport = :18092'
|
||||
curl --noproxy '*' --max-time 5 -sS -o /dev/null -w 'health %{http_code}\n' http://127.0.0.1:18092/health
|
||||
curl --noproxy '*' --max-time 5 -sS -o /dev/null -w 'catalog %{http_code}\n' http://127.0.0.1:18092/api/v1/knowledge-bases
|
||||
```
|
||||
|
||||
## 开发测试和现场验收
|
||||
|
||||
1. 在开发机运行 `make list` 查看门禁,`make verify` 做整套检查。以 `ci-report.json` 的 `commit_sha` 对照 `git rev-parse HEAD`,记录失败或跳过项。前端启动和组件测试见 [开发与 CI](../README.md#开发与-ci)。
|
||||
2. 针对后端改动,在独立测试逻辑库核对 `SELECT current_database(), max(version) FROM schema_migrations;` 与候选版本的迁移目录。云端多个候选可能共用治理 RDS。只改静态页面时单独切换 Web,保留原资源和代理配置供回滚。
|
||||
3. 在指定入口以真实账号分别验证。来源页读取一条已知记录,模型页测试到成功终态,知识库绑定在正确环境列库,保存后刷新核对。存储页按最近批次和子库目录分别与 `du -sB1 -- <目录>` 对账。记录入口、Web/API 提交、时间和结果。
|
||||
4. 候选备份页读取宿主采集的 `metadata.json`,API 只读挂载输出目录。「重新盘点」只重读文件。超过 `BACKUP_HOST_METADATA_MAX_AGE` 时,目录、占盘和预测转为未知。宿主重新采集后核对采集时间是否前进、批次是否恢复。测试现场使用按次采集、15 分钟有效期。采集脚本和挂载参数以同批测试候选交付物为准。
|
||||
|
||||
| 症状 | 第一步怎么查 | 如何判定 |
|
||||
|---|---|---|
|
||||
| 浏览器连接被拒 | 对照本机 `28189` 监听和测试机 `/health` | 远端正常、本机无监听时恢复本机 SSH 转发 |
|
||||
| 页面打开,某个 API 404 或 502 | 查浏览器请求路径、Nginx 生效上游及对应 API 版本 | 404 先查路由是否存在,502 先查上游是否可达。AI 页单独核对 `18098` |
|
||||
| 能探活,列库仍灰或返回 401 | 核对选中环境、凭据配置状态、容器运行 UID 与凭据文件可读性 | 用该环境的专用身份实际列库。检查过程只记录状态码和可读性 |
|
||||
| 历史可读,计划或配额显示未知 | 核对快照年龄、执行主机连接与配额字段 | 盘点、SSH 计划、配额分别验证。只有主机实际读回 cron 才能报告下一次执行时间 |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# 云测试治理主站应用 CD
|
||||
|
||||
正式入口为 `http://100.85.111.121:8088/`。云端 Compose 项目为 `knowledge-governance-cloud-test`,工作目录 `/opt/knowledge-governance`,当前配置链为 `compose.json`,服务为 `api`、`web`、`worker`。Web 容器提供 `8088`,通过挂载的 nginx 配置代理 API 的 `127.0.0.1:18098`。应用 CD 与知识库内容发布各自保留原有授权流程。
|
||||
|
||||
## 建立验收产物
|
||||
|
||||
1. 从干净检出的完整提交构建一次 API、worker、web 镜像。构建调用方对三个镜像都注入 `org.opencontainers.image.revision=<完整 SHA>`。API 构建身份沿用 `scripts/version.py --identity` 输出,传给 Dockerfile 的版本、提交和构建时间参数。API 与 worker 各自记录镜像 ID,即使它们来自同一个 Dockerfile。
|
||||
2. 封存后用同一份归档部署到隔离验收环境。输出目录必须是新路径,父目录应已存在。三个 `--api/--web/--worker` 参数是本地镜像的完整 ID。
|
||||
|
||||
```bash
|
||||
python3 scripts/production_cd.py pack --sha "$SHA" --task "$TASK_UUID" \
|
||||
--api "$API_ID" --web "$WEB_ID" --worker "$WORKER_ID" --output "$BUNDLE_DIR"
|
||||
docker load -i "$BUNDLE_DIR/api.tar"
|
||||
docker load -i "$BUNDLE_DIR/web.tar"
|
||||
docker load -i "$BUNDLE_DIR/worker.tar"
|
||||
```
|
||||
|
||||
3. 用验收环境自身的 Compose 和数据库启动这些镜像。用下面的命令读回验收环境运行身份。此处三个服务参数是实际的**测试容器名**。在业务验收入口执行实际用例并将输出、结果与报告链接放在同一份验收记录里。
|
||||
|
||||
```bash
|
||||
python3 scripts/production_cd.py attest --sha "$SHA" --task "$TASK_UUID" \
|
||||
--api "$QA_API" --web "$QA_WEB" --worker "$QA_WORKER" --test-url "$QA_URL" --directory "$BUNDLE_DIR"
|
||||
BUNDLE_SHA=$(sha256sum "$BUNDLE_DIR/release.json" | cut -d' ' -f1)
|
||||
```
|
||||
|
||||
4. 由验收人使用获授权的 HiFox CLI 登记业务结论,读回任务、验收字段和状态。`cd_acceptance` 是 JSON 字符串类型的 metadata,其解析结果为 `{"verdict":"passed","sha":"…","bundle":"…","evidence":"https://…"}`。录入人应先核对真实报告与运行身份。随后把对应任务转至已有「待发布」。用 `task view --output json` 核对实际任务 UUID、空间、状态和验收字段。验收登记重试复用原归档摘要与原记录。以当前版本只接受一个任务,多个任务分别验收。
|
||||
5. 在首次发布前,由运维把四个文件放入云主站的 `/opt/knowledge-governance/cd/bundles/<release.json SHA256>/`。通过安全文件通道传到临时目录,核对摘要与内容后用同一文件系统上的目录重命名发布;已有目录先核对内容,匹配时复用。此目录仅供目标控制器读取,文件由该账号持有且组与其他用户无写权限。
|
||||
|
||||
## 人工确认并自动部署
|
||||
|
||||
在 Gitea Actions 打开 `deploy-production` 并运行工作流。工作流分支选择 `main`,填写 `action=deploy`、已验收的完整 `sha`、归档清单摘要 `bundle`、HiFox 任务 UUID `task` 和 `confirm=deploy-production`。发布人用自己的 Gitea 账号提交一次确认。
|
||||
|
||||
主站侧从 Gitea 读取运行、操作者和通过的校验作业,并验证输入摘要及工作流代码。候选代码可来自仓库内的任务分支,以实际验收 SHA 为准。切换前还核验必需 CI 状态、HiFox 验收记录、三份归档、线上源代码血缘及数据库迁移账本。切换复用 `scripts/deploy-api-web.sh`,仅更换三项 Compose `image`;保留原配置字节以备回滚。切换后自动读回三镜像、API 二进制 SHA、主站健康、API readiness 及实际提供的前端资源。HiFox 回写包含原始运行 URL、完整提交、摘要、镜像 ID 及核验证据。
|
||||
|
||||
目标机已有的 `schema_migrations` 必须与候选完全相同,API 与 worker 的库连接均须核对。应用会在启动时检查迁移。新增或改变的迁移先按独立流程审查、备份、执行并复验;CD 自身只读迁移账本。发布时运营方应保留 prior tag,保障用原 Compose 字节回滚。
|
||||
|
||||
## 失败恢复
|
||||
|
||||
- `rolled_back` 表示自动恢复了原 Compose、旧的镜像 ID、API SHA 与站点健康。`rollback_failed` 或 `cutover_started` 要求巡检现场、按回执恢复并复验。目标机互斥锁覆盖切换及回写。Gitea 的取消记录只能说明流水线结束,回滚以目标机回执和运行时读回为准。
|
||||
- `DEPLOYED_HIFOX_PENDING` 表示镜像切换和核验已经完成。以同一 `sha`、`bundle`、`task` 人工发起 `action=writeback`、`confirm=retry-writeback`。脚本先读原目标回执并复验仍在运行的版本,随后补写 HiFox,保留原部署链接。状态与 metadata 的当前值以 HiFox 读回为准。重复调用核对记录后结束。
|
||||
- 目标回执 `/opt/knowledge-governance/cd/receipts/<bundle>.json`,镜像切换前的 Compose 保存为同名 `.compose.backup`。SSH 超时或强制中断后,先读回执和当前运行态。已存在未决或失败回执时,重复部署被拒。保留原回执作为事故证据。
|
||||
|
||||
## 一次性配置与权限
|
||||
|
||||
- 目标机安装仓库 `scripts/{production_cd.py,cd_migration_probe.py,deploy-api-web.sh,production-cd-ssh.sh}` 到 `/opt/knowledge-governance/cd/bin/`。创建私有 `receipts/`、`bundles/`、`hifox-home/`,初始化 `source.git` 为 bare Git 仓,并将 `.gitea/required-contexts.txt` 安装到该目录。仅发布运维可更新这些文件。
|
||||
- 填写目标机 `/opt/knowledge-governance/cd/deploy-production.yml`,内容与 Gitea 主分支的工作流逐字节相同。控制器用 Gitea 发布人账号 ID 和只读日志验证确认。目标端 `authorized_keys` 为 CI 单独公钥设置 `restrict,command="/opt/knowledge-governance/cd/bin/production-cd-ssh.sh"`。校验云主机 SSH host key 后将其按 `kg-governance-cd` 别名写入 `KG_CD_KNOWN_HOSTS`。对应独立 SSH 私钥保存在 `KG_CD_SSH_KEY`。Runner 标签为 `kg-production-cd`,使用 `KG_CD_SSH_HOST` 和 `KG_CD_SSH_PORT` 接入批准的 Tailscale 或本机 IPv6 隧道。
|
||||
- 将受控 HiFox CLI 安装至 `bin/hifox`,由账号持有人在目标主机使用 `hifox-home` 对应的私有配置目录完成登录。身份须能读取指定空间的任务、写 `cd_release` metadata 并完成对应状态迁移。状态绑定前分别读回组织状态和实际任务类型,复用原有待发布、已发布 ID,保留其余工作流配置。
|
||||
- 常规 push CI 负责验证和静态预览。这个 workflow 的唯一事件是 `workflow_dispatch`。`make verify` 的 `contracts` 子项运行 CD 用例。真实切换的验收要求对应候选、现场前置条件和发布人确认。
|
||||
@@ -3,7 +3,7 @@
|
||||
> 这份文档是**破坏性变化与数据迁移步骤的落脚点**:`CHANGELOG.md` 里以 `**破坏性**` 开头的条目
|
||||
> **必须**链接到本文(或本文的某个小节)。发布记录本身仍在 `CHANGELOG.md` —— 本文不维护第二份变更列表。
|
||||
>
|
||||
> 本仓 CI **只做 CI**:不打 tag、不发 Release、不部署。发布与部署都是**独立的人工审批动作**。
|
||||
> 常规 CI 负责检查、构建与静态预览。版本 tag 与 Release 由人批准后执行。主站应用部署使用独立的[人工确认 CD](PRODUCTION_CD_RUNBOOK.md)。
|
||||
|
||||
## 一、正式发布(由人批准后执行)
|
||||
|
||||
|
||||
@@ -59,6 +59,14 @@ type Expectation struct {
|
||||
Digests map[string]string
|
||||
}
|
||||
|
||||
// TableFingerprint 是一张知识域表在**整实例**上的行数与身份集合摘要(不限租户/
|
||||
// 知识库)。恢复后必须与预备份时刻逐表一致:只比目标范围会让"写到范围外"的
|
||||
// 残留被当成已恢复(真实事故:恢复报成功,目标仍多 67 行)。
|
||||
type TableFingerprint struct {
|
||||
Count int64 `json:"count"`
|
||||
Digest string `json:"digest"`
|
||||
}
|
||||
|
||||
// BackupRef 指向一份可独立恢复的完整预备份。
|
||||
type BackupRef struct {
|
||||
ID string `json:"id"`
|
||||
@@ -67,6 +75,9 @@ type BackupRef struct {
|
||||
SnapshotID string `json:"snapshot_id"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
// Universe 是预备份时刻的整实例指纹,恢复后逐表比对。空表示这次没有采集
|
||||
// (例如 worker 被杀后用已持久化的快照接管),此时恢复只能靠内容校验兜底。
|
||||
Universe map[string]TableFingerprint `json:"universe,omitempty"`
|
||||
}
|
||||
|
||||
type Target interface {
|
||||
@@ -224,8 +235,16 @@ func failAndRestore(ctx context.Context, outcome Outcome, target Target, recorde
|
||||
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
|
||||
return outcome
|
||||
}
|
||||
// "恢复工具跑完"不是"恢复完成":必须再校验一次目标内容与预备份一致,
|
||||
// 否则页面会把半恢复状态报成已恢复(真实事故就是这样放开了维护)。
|
||||
if verifyErr := target.Verify(ctx, Expectation{SnapshotID: outcome.Backup.SnapshotID}); verifyErr != nil {
|
||||
outcome.State = "rollback_failed"
|
||||
outcome.Rollback = "恢复后校验失败(目标可能仍是半写状态):" + verifyErr.Error()
|
||||
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
|
||||
return outcome
|
||||
}
|
||||
outcome.State = "rolled_back"
|
||||
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID
|
||||
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID + ",并逐表/索引/资源复核一致"
|
||||
recorder.Stage(ctx, "rollback", "succeeded", outcome.Rollback)
|
||||
return outcome
|
||||
}
|
||||
|
||||
@@ -13,9 +13,14 @@ type fakeTarget struct {
|
||||
applyErr map[Stage]error
|
||||
verifyErr error
|
||||
restoreErr error
|
||||
applied []Stage
|
||||
restored int
|
||||
currentN int
|
||||
// 恢复之后校验是否通过:用来区分"恢复跑完"和"恢复已验证"。
|
||||
verifyPassesAfterRestore bool
|
||||
applied []Stage
|
||||
restored int
|
||||
currentN int
|
||||
verifyN int
|
||||
// 记录最后一次 Verify 是否发生在 Restore 之后(恢复必须被校验兜底)。
|
||||
verifiedAfterRestore bool
|
||||
}
|
||||
|
||||
func (f *fakeTarget) Current(context.Context) (CurrentState, error) {
|
||||
@@ -35,7 +40,17 @@ func (f *fakeTarget) Apply(_ context.Context, stage Stage, _ string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeTarget) Verify(context.Context, Expectation) error { return f.verifyErr }
|
||||
func (f *fakeTarget) Verify(context.Context, Expectation) error {
|
||||
f.verifyN++
|
||||
f.verifiedAfterRestore = f.restored > 0
|
||||
if f.verifyErr == nil {
|
||||
return nil
|
||||
}
|
||||
if f.verifyPassesAfterRestore && f.restored > 0 {
|
||||
return nil
|
||||
}
|
||||
return f.verifyErr
|
||||
}
|
||||
|
||||
func (f *fakeTarget) Restore(context.Context, BackupRef) error {
|
||||
f.restored++
|
||||
@@ -185,12 +200,39 @@ func TestRunRollsBackWhenStageFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRollsBackWhenVerificationFails(t *testing.T) {
|
||||
target := &fakeTarget{verifyErr: errors.New("生产分块数不一致")}
|
||||
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, nil)
|
||||
if outcome.State != "rolled_back" || target.restored != 1 {
|
||||
// 恢复跑完但恢复后校验仍失败:不能报 rolled_back。真实事故里页面就是在这种
|
||||
// 状态下显示"已恢复到预备份并验证",而目标其实还留着 67 行残留。
|
||||
func TestRunDoesNotClaimRollbackWhenRecoveryVerificationFails(t *testing.T) {
|
||||
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
|
||||
verifyErr: errors.New("生产分块数不一致")}
|
||||
recorder := &stageLog{}
|
||||
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
|
||||
if outcome.State != "rollback_failed" || target.restored != 1 {
|
||||
t.Fatalf("outcome = %+v restored=%d", outcome, target.restored)
|
||||
}
|
||||
if !strings.Contains(outcome.Rollback, "恢复后校验失败") {
|
||||
t.Fatalf("rollback message = %q", outcome.Rollback)
|
||||
}
|
||||
if !recorder.has("rollback:failed") {
|
||||
t.Fatalf("stage log = %v", recorder.entries)
|
||||
}
|
||||
}
|
||||
|
||||
// 只有"恢复到预备份且恢复后校验通过"才允许报 rolled_back。
|
||||
func TestRunReportsRolledBackOnlyAfterRecoveryVerifies(t *testing.T) {
|
||||
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
|
||||
verifyErr: errors.New("半写状态"), verifyPassesAfterRestore: true}
|
||||
recorder := &stageLog{}
|
||||
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
|
||||
if outcome.State != "rolled_back" {
|
||||
t.Fatalf("outcome = %+v", outcome)
|
||||
}
|
||||
if !target.verifiedAfterRestore {
|
||||
t.Fatal("restore must be followed by a content verification")
|
||||
}
|
||||
if !recorder.has("rollback:succeeded") {
|
||||
t.Fatalf("stage log = %v", recorder.entries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportsRollbackFailureDistinctly(t *testing.T) {
|
||||
|
||||
@@ -48,7 +48,10 @@ func TestNormalizeRefusesEnabledSettingsWithoutTransportAndBadRecipients(t *test
|
||||
t.Fatalf("defaults not applied: %+v %v", good, err)
|
||||
}
|
||||
// 解析不了的主机名必须仍被判为非法:.invalid 是 RFC 6761 保留域,永远不解析。
|
||||
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid", From: "kb@example.com",
|
||||
// 末尾的点是必须的:构建机的 resolv.conf 可能带 search 域(本环境 Tailscale
|
||||
// 的 tailf49802.ts.net),不带点就会被拼成 smtp.kg-test.invalid.<search> 并
|
||||
// 由 Fake-IP DNS 解析出来,于是这条断言测的是构建机的搜索域而不是被测逻辑。
|
||||
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid.", From: "kb@example.com",
|
||||
Recipients: []string{"a@example.com"}}); err != ErrInvalid {
|
||||
t.Fatalf("unresolvable host must stay invalid, got %v", err)
|
||||
}
|
||||
|
||||
@@ -113,6 +113,12 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, err
|
||||
}
|
||||
// 整实例指纹:恢复之后要拿它证明"多写的、少写的都没有"。只按目标范围比,
|
||||
// 写到范围外的行会被漏掉(真实事故:恢复报成功但目标多 67 行)。
|
||||
universeBefore, err := universeFingerprint(ctx, s.pool)
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, err
|
||||
}
|
||||
pg, err := ExportLibrary(ctx, s.pool, s.identity)
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, fmt.Errorf("读取生产整库失败: %w", err)
|
||||
@@ -154,6 +160,13 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
|
||||
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间生产 %s 内容发生变化,切点不一致", table)
|
||||
}
|
||||
}
|
||||
universeAfter, err := universeFingerprint(ctx, s.pool)
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, err
|
||||
}
|
||||
if diff := universeDiff(universeBefore, universeAfter); diff != "" {
|
||||
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间整实例知识域发生变化(%s),切点不一致", diff)
|
||||
}
|
||||
for table, want := range current.Counts {
|
||||
if table == "index" {
|
||||
continue
|
||||
@@ -171,7 +184,7 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
|
||||
total += part.Bytes
|
||||
}
|
||||
return libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID, Bytes: total,
|
||||
CreatedAt: createdAt}, nil
|
||||
CreatedAt: createdAt, Universe: universeBefore}, nil
|
||||
}
|
||||
|
||||
func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) {
|
||||
|
||||
@@ -84,8 +84,16 @@ type ProductionLibraryTarget struct {
|
||||
scope ProductionScope
|
||||
search *SearchExporter
|
||||
now func() time.Time
|
||||
// storageBackend 是本次写入使用的目标存储后端 id(写入与校验共用)。
|
||||
storageBackend string
|
||||
// storageBackend / storageProvider 是本次写入使用的目标存储后端与提供方
|
||||
// (写入与校验共用,缺一不可:只换桶不换提供方,目标取文件仍会走来源实例的路)。
|
||||
storageBackend string
|
||||
storageProvider string
|
||||
// snapshotTenant / snapshotKB 是这次要写入的快照来源身份。写入时按它做
|
||||
// 来源→目标的范围映射;校验时按它反查"目标里还有没有来源范围的行"。
|
||||
snapshotTenant uint64
|
||||
snapshotKB string
|
||||
// targetEmbeddingModel 是目标知识库绑定的 embedding 模型 id(文档行要换成它)。
|
||||
targetEmbeddingModel string
|
||||
}
|
||||
|
||||
// resolveStorageBackend 在目标库里找本次写入的存储后端。找不到就报错:宁可不写,
|
||||
@@ -98,24 +106,39 @@ func (t *ProductionLibraryTarget) resolveStorageBackend(ctx context.Context) err
|
||||
if !validSearchIdentifier(t.config.ObjectsBackendID) {
|
||||
return errors.New("目标存储后端 id 形状不合法")
|
||||
}
|
||||
t.storageBackend = t.config.ObjectsBackendID
|
||||
provider, err := t.storageProviderOf(ctx, t.config.ObjectsBackendID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
t.storageBackend, t.storageProvider = t.config.ObjectsBackendID, provider
|
||||
return nil
|
||||
}
|
||||
var id string
|
||||
var id, provider string
|
||||
err := t.pool.QueryRow(ctx, `
|
||||
SELECT id FROM storage_backends
|
||||
SELECT id, coalesce(provider,'') FROM storage_backends
|
||||
WHERE tenant_id=$1 AND coalesce(config->>'bucket_name','')=$2
|
||||
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id)
|
||||
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id, &provider)
|
||||
if err != nil {
|
||||
return errors.New("目标实例没有与目标桶匹配的存储后端,拒绝写入不可解析的资源定位符")
|
||||
}
|
||||
if !validSearchIdentifier(id) {
|
||||
return errors.New("目标存储后端 id 形状不合法")
|
||||
}
|
||||
t.storageBackend = id
|
||||
t.storageBackend, t.storageProvider = id, provider
|
||||
return nil
|
||||
}
|
||||
|
||||
// storageProviderOf 读出目标存储后端的提供方(minio / oss / s3…)。资源行里的
|
||||
// 提供方必须与它一致,否则目标应用会按来源提供方去找一个不存在的存储服务。
|
||||
func (t *ProductionLibraryTarget) storageProviderOf(ctx context.Context, id string) (string, error) {
|
||||
var provider string
|
||||
if err := t.pool.QueryRow(ctx, `SELECT coalesce(provider,'') FROM storage_backends WHERE id=$1 AND tenant_id=$2`,
|
||||
id, t.scope.TenantID).Scan(&provider); err != nil {
|
||||
return "", errors.New("目标实例没有这个存储后端,拒绝写入不可解析的资源定位符")
|
||||
}
|
||||
return provider, nil
|
||||
}
|
||||
|
||||
func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryConfig, scope ProductionScope) (*ProductionLibraryTarget, error) {
|
||||
if err := config.validate(scope); err != nil {
|
||||
return nil, err
|
||||
@@ -259,6 +282,7 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
|
||||
if err := t.resolveStorageBackend(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
t.resolveEmbeddingModel(ctx)
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return errors.New("归档不可用")
|
||||
@@ -268,6 +292,9 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
|
||||
if err != nil {
|
||||
return errors.New("预发布快照不可读")
|
||||
}
|
||||
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
|
||||
return err
|
||||
}
|
||||
switch stage {
|
||||
case libraryrelease.StageDatabase, libraryrelease.StageWiki:
|
||||
kind := "database"
|
||||
@@ -356,10 +383,10 @@ func (t *ProductionLibraryTarget) insertRows(ctx context.Context, tx pgx.Tx, arc
|
||||
if len(row) == 0 {
|
||||
continue
|
||||
}
|
||||
if table == "resources" {
|
||||
if err := t.applyResourceBucket(row); err != nil {
|
||||
return err
|
||||
}
|
||||
// 行必须先映射到目标范围再落库:原样插入会把行写进来源租户/知识库,
|
||||
// 目标范围的删除、校验与恢复都看不见它们。
|
||||
if err := t.retargetRow(table, row); err != nil {
|
||||
return err
|
||||
}
|
||||
if columns == nil {
|
||||
columns = make([]string, 0, len(row))
|
||||
@@ -408,53 +435,9 @@ type resourceLocation struct {
|
||||
}
|
||||
|
||||
// rowTransform 是"写入时对行做过的变换"的统一入口:校验要用同一份,否则摘要比的是
|
||||
// 变换前的行、目标里是变换后的行,永远不一致。
|
||||
// 变换前的行、目标里是变换后的行,永远不一致。范围映射同样必须两边一致。
|
||||
func (t *ProductionLibraryTarget) rowTransform(table string) func(map[string]json.RawMessage) error {
|
||||
if table == "resources" {
|
||||
return t.applyResourceBucket
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyResourceBucket 把资源行定位符里的桶换成本次写入的目标桶。
|
||||
// 写入与校验必须用同一个变换:只改写入侧会让"行内容摘要"对不上(真跑踩过)。
|
||||
func (t *ProductionLibraryTarget) applyResourceBucket(row map[string]json.RawMessage) error {
|
||||
if t.config.ObjectsBucket == "" {
|
||||
return nil
|
||||
}
|
||||
raw, ok := row["physical_path"]
|
||||
if !ok {
|
||||
return errors.New("资源行缺少定位符")
|
||||
}
|
||||
var locator string
|
||||
if err := json.Unmarshal(raw, &locator); err != nil {
|
||||
return errors.New("资源定位符不可解析")
|
||||
}
|
||||
rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket)
|
||||
if !ok {
|
||||
return fmt.Errorf("资源定位符无法改写到目标桶:%s", locator)
|
||||
}
|
||||
encoded, err := json.Marshal(rewritten)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
row["physical_path"] = encoded
|
||||
// 原版应用先按 resources.storage_backend_id 找存储后端,再回落到 KB,再落默认服务。
|
||||
// 这一列留着来源 id,目标仍然 500(真跑:storage backend not found → invalid path)。
|
||||
if t.storageBackend == "" {
|
||||
return nil
|
||||
}
|
||||
if raw, ok := row["storage_backend_id"]; ok {
|
||||
var current string
|
||||
if json.Unmarshal(raw, ¤t) == nil && current != "" && current != t.storageBackend {
|
||||
encodedBackend, err := json.Marshal(t.storageBackend)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
row["storage_backend_id"] = encodedBackend
|
||||
}
|
||||
}
|
||||
return nil
|
||||
return func(row map[string]json.RawMessage) error { return t.retargetRow(table, row) }
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) importResources(ctx context.Context, archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error {
|
||||
@@ -591,6 +574,7 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err := t.resolveStorageBackend(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
t.resolveEmbeddingModel(ctx)
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return errors.New("归档不可用")
|
||||
@@ -600,6 +584,9 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err != nil {
|
||||
return errors.New("预发布快照不可读")
|
||||
}
|
||||
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" {
|
||||
continue
|
||||
@@ -668,6 +655,72 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err := t.verifyObjects(ctx, archive, manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
// 目标里不允许留下"快照来源范围"的行:那说明有写入逃出了目标范围,
|
||||
// 而恢复只按目标范围回放,永远删不到它们(真实事故就是这样被漏掉的)。
|
||||
if err := t.assertNoSourceScopeRows(ctx, archive, manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// dropOrphanResources 清掉"本次恢复后已无任何归属"的目标租户资源行。
|
||||
//
|
||||
// 判据必须同时满足:属于目标租户、不在预备份快照里、没有任何 resource_binding、
|
||||
// 也没有任何分块正文以 resource://<handle> 引用它。这样的行对目标应用完全不可见
|
||||
// (没有知识库能按它取到文件),只可能是失败写入留下的残渣。
|
||||
func (t *ProductionLibraryTarget) dropOrphanResources(ctx context.Context, backupSnapshotID string) error {
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return errors.New("归档不可用")
|
||||
}
|
||||
defer archive.Close()
|
||||
manifest, err := archive.Load(backupSnapshotID)
|
||||
if err != nil {
|
||||
return errors.New("预备份快照不可读")
|
||||
}
|
||||
kept := []string{}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" || part.Name != "resources" {
|
||||
continue
|
||||
}
|
||||
ids, err := archivedResourceIDs(archive, part)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
kept = append(kept, ids...)
|
||||
}
|
||||
rows, err := t.pool.Query(ctx, `
|
||||
SELECT r.id FROM resources AS r
|
||||
WHERE r.tenant_id=$1
|
||||
AND NOT (r.id = ANY($2::text[]))
|
||||
AND NOT EXISTS (SELECT 1 FROM resource_bindings b WHERE b.resource_id=r.id)
|
||||
AND NOT EXISTS (SELECT 1 FROM chunks c WHERE c.tenant_id=r.tenant_id AND c.deleted_at IS NULL
|
||||
AND coalesce(r.handle,'') <> ''
|
||||
AND position('resource://'||r.handle IN coalesce(c.content,'')) > 0)`,
|
||||
t.scope.TenantID, kept)
|
||||
if err != nil {
|
||||
return fmt.Errorf("查找目标残留资源失败: %w", err)
|
||||
}
|
||||
orphans := make([]string, 0, 8)
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
orphans = append(orphans, id)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(orphans) == 0 {
|
||||
return nil
|
||||
}
|
||||
if _, err := t.pool.Exec(ctx, `DELETE FROM resources WHERE tenant_id=$1 AND id = ANY($2::text[])`,
|
||||
t.scope.TenantID, orphans); err != nil {
|
||||
return fmt.Errorf("清理目标残留资源失败: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -772,5 +825,21 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
|
||||
return fmt.Errorf("恢复阶段 %s 失败: %w", stage, err)
|
||||
}
|
||||
}
|
||||
// resources 是多知识库共享表,回放阶段不删它:本次写入新增、恢复后已经
|
||||
// 没有任何归属的资源行会残留。它们对应用不可见,但属于"多写的"。
|
||||
if err := t.dropOrphanResources(ctx, backup.SnapshotID); err != nil {
|
||||
return err
|
||||
}
|
||||
// 恢复工具把四个阶段都跑完不算恢复完成:整实例必须回到预备份时刻的样子。
|
||||
// 只比目标范围会把"写到范围外"的残留当成已恢复(真实事故:恢复后仍多 67 行)。
|
||||
if len(backup.Universe) > 0 {
|
||||
after, err := universeFingerprint(ctx, t.pool)
|
||||
if err != nil {
|
||||
return fmt.Errorf("恢复后读取整库指纹失败: %w", err)
|
||||
}
|
||||
if diff := universeDiff(backup.Universe, after); diff != "" {
|
||||
return fmt.Errorf("恢复后目标与预备份时刻不一致:%s(很可能有写入逃出了预备份范围,不能当作已恢复)", diff)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -4,12 +4,15 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -390,3 +393,339 @@ func sealTargetFixture(t *testing.T, archive *librarysnapshot.Archive, scope str
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// memoryObjectStore 是"真实字节"的资源存储替身:Put/Open/Stat/Copy 都按摘要核对,
|
||||
// 可以强制写入失败,用来复现"资源阶段失败"这一半写场景。
|
||||
type memoryObjectStore struct {
|
||||
objects map[string][]byte
|
||||
failPut bool
|
||||
puts []string
|
||||
}
|
||||
|
||||
func newMemoryObjectStore(objects map[string][]byte) *memoryObjectStore {
|
||||
copied := map[string][]byte{}
|
||||
for key, value := range objects {
|
||||
copied[key] = append([]byte(nil), value...)
|
||||
}
|
||||
return &memoryObjectStore{objects: copied}
|
||||
}
|
||||
|
||||
func objectKey(provider, path string) string { return provider + "|" + path }
|
||||
|
||||
func (s *memoryObjectStore) Put(_ context.Context, provider, path string, reader io.Reader, size int64) error {
|
||||
if s.failPut {
|
||||
return errors.New("对象存储不可写")
|
||||
}
|
||||
body, err := io.ReadAll(reader)
|
||||
if err != nil || int64(len(body)) != size {
|
||||
return errors.New("对象字节数与声明不符")
|
||||
}
|
||||
s.objects[objectKey(provider, path)] = body
|
||||
s.puts = append(s.puts, path)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *memoryObjectStore) Open(_ context.Context, provider, path string) (io.ReadCloser, error) {
|
||||
body, ok := s.objects[objectKey(provider, path)]
|
||||
if !ok {
|
||||
return nil, errors.New("对象不存在")
|
||||
}
|
||||
return io.NopCloser(bytes.NewReader(body)), nil
|
||||
}
|
||||
|
||||
func (s *memoryObjectStore) Stat(_ context.Context, provider, path string) (int64, string, error) {
|
||||
body, ok := s.objects[objectKey(provider, path)]
|
||||
if !ok {
|
||||
return 0, "", errors.New("对象不存在")
|
||||
}
|
||||
digest := sha256.Sum256(body)
|
||||
return int64(len(body)), hex.EncodeToString(digest[:]), nil
|
||||
}
|
||||
|
||||
func (s *memoryObjectStore) Copy(_ context.Context, provider, source, target string) error {
|
||||
body, ok := s.objects[objectKey(provider, source)]
|
||||
if !ok {
|
||||
return errors.New("对象不存在")
|
||||
}
|
||||
s.objects[objectKey(provider, target)] = append([]byte(nil), body...)
|
||||
return nil
|
||||
}
|
||||
|
||||
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同)、目标非空、资源阶段写失败。
|
||||
// 旧代码把归档行原样写进目标(落在来源租户/知识库),随后的回滚只清目标范围,
|
||||
// 于是"已恢复到预备份"却留下 4 文件 / 38 分块 / 21 资源 / 4 绑定。
|
||||
func TestLibraryCrossScopeSyncRollbackLeavesNoResidueIsolated(t *testing.T) {
|
||||
dsn, base := os.Getenv("KG_LIBRARY_TEST_DATABASE_URL"), os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
|
||||
if dsn == "" || base == "" || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
|
||||
t.Skip("requires isolated PG and ES opt-in")
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") {
|
||||
t.Fatal("isolated database required")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||
defer cancel()
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("open isolated PG: %v", err)
|
||||
}
|
||||
defer pool.Close()
|
||||
for _, statement := range []string{
|
||||
`DROP SCHEMA public CASCADE; CREATE SCHEMA public`,
|
||||
libraryTestSchema,
|
||||
`ALTER TABLE wiki_pages ADD COLUMN status text DEFAULT 'published'`,
|
||||
`ALTER TABLE knowledge_tag_relations ADD COLUMN tenant_id bigint`,
|
||||
`ALTER TABLE resources ADD COLUMN provider text, ADD COLUMN size bigint, ADD COLUMN storage_backend_id text`,
|
||||
`ALTER TABLE resources ADD PRIMARY KEY (id)`,
|
||||
`ALTER TABLE resource_bindings ADD COLUMN relation text`,
|
||||
`INSERT INTO knowledge_bases VALUES('kb-target',10001,'{}',NULL),('kb-source',10000,'{}',NULL)`,
|
||||
`CREATE TABLE storage_backends(id text PRIMARY KEY, tenant_id bigint, provider text, config jsonb, created_at timestamptz)`,
|
||||
`INSERT INTO storage_backends VALUES('backend-target',10001,'minio','{"bucket_name":"kg-test-bucket"}',now())`,
|
||||
// 目标非空:目标自己有文档/分块,另一个租户也有数据(回滚不得碰它)。
|
||||
`INSERT INTO knowledges VALUES('doc-target',10001,'kb-target','completed',0,'',NULL)`,
|
||||
`INSERT INTO knowledges VALUES('doc-foreign',20000,'kb-other','completed',0,'',NULL)`,
|
||||
`INSERT INTO chunks VALUES('chunk-target',10001,'kb-target','doc-target','target content',NULL)`,
|
||||
`INSERT INTO chunks VALUES('chunk-foreign',20000,'kb-other','doc-foreign','foreign content',NULL)`,
|
||||
`UPDATE resource_bindings SET tenant_id=10000 WHERE tenant_id IS NULL`,
|
||||
} {
|
||||
if _, err := pool.Exec(ctx, statement); err != nil {
|
||||
t.Fatalf("fixture schema: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
token := make([]byte, 8)
|
||||
if _, err := rand.Read(token); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
index := "kg_library_test_" + hex.EncodeToString(token)
|
||||
client := &http.Client{Timeout: 20 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||
return http.ErrUseLastResponse
|
||||
}}
|
||||
call := func(method, path, body string, want int) {
|
||||
t.Helper()
|
||||
request, err := http.NewRequestWithContext(ctx, method, base+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
response, err := client.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("fixture HTTP: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != want {
|
||||
t.Fatalf("fixture %s %s => %d, want %d", method, path, response.StatusCode, want)
|
||||
}
|
||||
}
|
||||
call("PUT", "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"chunk_id":{"type":"keyword"},"content":{"type":"text"},"embedding":{"type":"dense_vector","dims":2}}}}`, 200)
|
||||
defer func() {
|
||||
call("DELETE", "/"+index, "", 200)
|
||||
call("HEAD", "/"+index, "", 404)
|
||||
}()
|
||||
// 目标当前索引自洽:目标分块必须有对应文档,否则预备份会(正确地)拒绝。
|
||||
call("PUT", "/"+index+"/_doc/chunk-target?refresh=true",
|
||||
`{"tenant_id":10001,"knowledge_base_id":"kb-target","knowledge_id":"doc-target","chunk_id":"chunk-target","source_id":"chunk-target","content":"target content","embedding":[0.1,0.2]}`, 201)
|
||||
|
||||
archiveRoot := t.TempDir()
|
||||
if err := os.Chmod(archiveRoot, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archive, err := librarysnapshot.Open(archiveRoot, 64*1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
snapshotID := sealCrossScopeFixture(t, archive, 10000, "kb-source")
|
||||
|
||||
store := newMemoryObjectStore(nil)
|
||||
target, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
|
||||
Postgres: dsn, SearchURL: base, SearchIndex: index, Objects: store, ObjectsReader: store,
|
||||
ArchiveRoot: archiveRoot, ObjectsBucket: "kg-test-bucket", MaxDocuments: 100, HTTPClient: client,
|
||||
}, ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"})
|
||||
if err != nil {
|
||||
t.Fatalf("target: %v", err)
|
||||
}
|
||||
defer target.Close()
|
||||
mapping, err := target.Search().MappingSHA256(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
backups, err := NewProductionBackupStore(ProductionBackupConfig{
|
||||
Pool: target.Pool(), Search: target.Search(), Objects: store, ArchiveRoot: archiveRoot,
|
||||
Scope: "internal", Identity: target.Identity(), MaxDocuments: 100,
|
||||
Compatibility: librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74,
|
||||
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "text-embedding-v4",
|
||||
EmbeddingDimension: 1024, IndexMappingSHA256: mapping},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("backup store: %v", err)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
request := libraryrelease.Request{Target: "internal", SnapshotID: snapshotID,
|
||||
ParamsDigest: "digest-cross-scope", Now: now,
|
||||
Grant: libraryrelease.Grant{Target: "internal", SnapshotID: snapshotID, ParamsDigest: "digest-cross-scope",
|
||||
ExpiresAt: now.Add(10 * time.Minute), AuthorizedBy: "acceptance"}}
|
||||
|
||||
// 1) 资源阶段失败 → 必须回滚且不留下来源范围的行。
|
||||
store.failPut = true
|
||||
outcome := libraryrelease.Run(ctx, request, target, backups, nil)
|
||||
if outcome.State != "rolled_back" {
|
||||
t.Fatalf("outcome = %+v", outcome)
|
||||
}
|
||||
if outcome.FailedAt != libraryrelease.StageResources {
|
||||
t.Fatalf("failed_at = %s, want resources", outcome.FailedAt)
|
||||
}
|
||||
if outcome.Backup.SnapshotID == "" || len(outcome.Backup.Universe) == 0 {
|
||||
t.Fatalf("backup must carry the whole-instance fingerprint: %+v", outcome.Backup)
|
||||
}
|
||||
for _, table := range []string{"knowledges", "chunks", "wiki_pages"} {
|
||||
var residue int64
|
||||
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND knowledge_base_id=$2`, table),
|
||||
10000, "kb-source").Scan(&residue); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if residue != 0 {
|
||||
t.Fatalf("%s still carries %d source-scope rows after rollback", table, residue)
|
||||
}
|
||||
}
|
||||
// 夹具在租户 10000 里本来就有自己的行(res-safe / binding-safe):它们必须原样
|
||||
// 留下,只有这次同步写进去的行(res-source / 绑到 doc-source)才算残留。
|
||||
for query, want := range map[string]int64{
|
||||
`SELECT count(*) FROM resources WHERE id='res-source'`: 0,
|
||||
`SELECT count(*) FROM resource_bindings WHERE tenant_id=10000 AND owner_id='doc-source'`: 0,
|
||||
`SELECT count(*) FROM resources WHERE tenant_id=10000`: 1,
|
||||
} {
|
||||
var got int64
|
||||
if err := pool.QueryRow(ctx, query).Scan(&got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("%s = %d, want %d", query, got, want)
|
||||
}
|
||||
}
|
||||
var leftovers int64
|
||||
if err := pool.QueryRow(ctx, `SELECT count(*) FROM resources WHERE tenant_id=10001`).Scan(&leftovers); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if leftovers != 0 {
|
||||
t.Fatalf("aborted write left %d resource rows in the target tenant", leftovers)
|
||||
}
|
||||
var foreign int64
|
||||
if err := pool.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=20000`).Scan(&foreign); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// 夹具本来就有 1 行 other-scope(20000),加上本用例的 chunk-foreign,共 2 行。
|
||||
if foreign != 2 {
|
||||
t.Fatalf("cross-tenant rows must survive rollback, got %d", foreign)
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: outcome.Backup.SnapshotID}); err != nil {
|
||||
t.Fatalf("rollback must verify against the prebackup: %v", err)
|
||||
}
|
||||
|
||||
// 2) 同样的跨范围同步,资源能写 → 行必须落在目标范围,资源定位符指向目标存储。
|
||||
store.failPut = false
|
||||
succeed := libraryrelease.Run(ctx, request, target, backups, nil)
|
||||
if succeed.State != "succeeded" {
|
||||
t.Fatalf("second run outcome = %+v", succeed)
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
|
||||
t.Fatalf("verify after cross-scope sync: %v", err)
|
||||
}
|
||||
var knowledgeKB, chunkKB, provider, locator string
|
||||
var tenant int64
|
||||
if err := pool.QueryRow(ctx, `SELECT tenant_id, knowledge_base_id FROM knowledges WHERE id='doc-source'`).
|
||||
Scan(&tenant, &knowledgeKB); err != nil {
|
||||
t.Fatalf("source document must exist in the target scope: %v", err)
|
||||
}
|
||||
if tenant != 10001 || knowledgeKB != "kb-target" {
|
||||
t.Fatalf("cross-scope row landed at %d/%s", tenant, knowledgeKB)
|
||||
}
|
||||
if err := pool.QueryRow(ctx, `SELECT knowledge_base_id FROM chunks WHERE id='chunk-source'`).Scan(&chunkKB); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if chunkKB != "kb-target" {
|
||||
t.Fatalf("chunk landed in %s", chunkKB)
|
||||
}
|
||||
if err := pool.QueryRow(ctx, `SELECT provider, physical_path FROM resources WHERE id='res-source'`).
|
||||
Scan(&provider, &locator); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if provider != "minio" {
|
||||
t.Fatalf("resource provider = %s, want minio", provider)
|
||||
}
|
||||
if !strings.Contains(locator, "storage://backend-target/minio://kg-test-bucket/10001/") {
|
||||
t.Fatalf("resource locator was not retargeted: %s", locator)
|
||||
}
|
||||
if strings.Contains(locator, "weknora/10000") {
|
||||
t.Fatalf("resource locator still points at the source tenant: %s", locator)
|
||||
}
|
||||
t.Logf("cross_scope_sync verified: snapshot=%s backup=%s locator=%s", snapshotID, succeed.Backup.SnapshotID, locator)
|
||||
}
|
||||
|
||||
// sealCrossScopeFixture 造一份"来源范围与目标不同"的整库快照:文档/分块/资源/绑定/
|
||||
// Wiki/索引都带来源身份,资源部件是可核对的真实字节。
|
||||
func sealCrossScopeFixture(t *testing.T, archive *librarysnapshot.Archive, sourceTenant uint64, sourceKB string) string {
|
||||
t.Helper()
|
||||
encode := func(value any) []byte {
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return encoded
|
||||
}
|
||||
const handle = "abcdefghijklmnopqrstuv"
|
||||
objectBytes := []byte("12345678")
|
||||
parts := make([]librarysnapshot.Part, 0, 12)
|
||||
put := func(kind, name string, count int64, body []byte) {
|
||||
part, err := archive.Put(bytes.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
part.Kind, part.Name, part.Count = kind, name, count
|
||||
parts = append(parts, part)
|
||||
}
|
||||
put("database", "knowledges", 1, encode([]map[string]any{{
|
||||
"id": "doc-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
|
||||
"parse_status": "completed", "pending_subtasks_count": 0, "file_path": ""}}))
|
||||
put("database", "chunks", 1, encode([]map[string]any{{
|
||||
"id": "chunk-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
|
||||
"knowledge_id": "doc-source", "content": "见 resource://" + handle}}))
|
||||
put("database", "knowledge_tags", 0, encode([]map[string]any{}))
|
||||
put("database", "knowledge_tag_relations", 0, encode([]map[string]any{}))
|
||||
put("database", "resources", 1, encode([]map[string]any{{
|
||||
"id": "res-source", "handle": handle, "tenant_id": sourceTenant,
|
||||
"storage_backend_id": "backend-source", "provider": "oss",
|
||||
"physical_path": "storage://backend-source/oss://source-bucket/weknora/" +
|
||||
formatTenant(sourceTenant) + "/res-source/a.png",
|
||||
"size": int64(len(objectBytes))}}))
|
||||
put("database", "resource_bindings", 1, encode([]map[string]any{{
|
||||
"id": "bind-source", "resource_id": "res-source", "tenant_id": sourceTenant,
|
||||
"owner_type": "knowledge", "owner_id": "doc-source", "relation": "attachment"}}))
|
||||
put("resource", "res-source", 1, objectBytes)
|
||||
put("wiki", "wiki_folders", 0, encode([]map[string]any{}))
|
||||
put("wiki", "wiki_pages", 1, encode([]map[string]any{{
|
||||
"id": "wiki-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
|
||||
"slug": "index", "content": "wiki"}}))
|
||||
for _, name := range []string{"wiki_page_issues", "wiki_log_entries"} {
|
||||
put("wiki", name, 0, encode([]map[string]any{}))
|
||||
}
|
||||
put("index", "chunks", 1, encode([]map[string]any{{"id": "chunk-source", "source": map[string]any{
|
||||
"tenant_id": sourceTenant, "knowledge_base_id": sourceKB, "knowledge_id": "doc-source",
|
||||
"chunk_id": "chunk-source", "source_id": "chunk-source", "content": "见 resource://" + handle,
|
||||
"embedding": []float64{0.1, 0.2}}}}))
|
||||
manifest := librarysnapshot.Manifest{ValidationPolicy: librarysnapshot.ArtifactAcceptancePolicy,
|
||||
Format: librarysnapshot.Format, Scope: "internal", TenantID: sourceTenant, KnowledgeBaseID: sourceKB,
|
||||
CreatedAt: time.Now().UTC(), Parts: parts,
|
||||
Compatibility: librarysnapshot.Compatibility{
|
||||
WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8",
|
||||
EmbeddingModel: "text-embedding-v4", EmbeddingDimension: 1024,
|
||||
IndexMappingSHA256: strings.Repeat("a", 64),
|
||||
}}
|
||||
id, err := archive.Seal(manifest)
|
||||
if err != nil {
|
||||
t.Fatalf("seal cross-scope fixture: %v", err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func formatTenant(tenant uint64) string { return strconv.FormatUint(tenant, 10) }
|
||||
|
||||
@@ -0,0 +1,334 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// ResidueReport 是"清理范围外残留"的审计记录:清掉了什么、依据是哪份归档、
|
||||
// 备份写到哪儿、清理后目标处于什么状态。没有这份记录就不许动目标。
|
||||
type ResidueReport struct {
|
||||
SnapshotID string `json:"snapshot_id"`
|
||||
SourceTenant uint64 `json:"source_tenant"`
|
||||
SourceKB string `json:"source_kb"`
|
||||
Tables map[string]int `json:"tables"`
|
||||
BackupPath string `json:"backup_path"`
|
||||
BackupSHA256 string `json:"backup_sha256"`
|
||||
Applied bool `json:"applied"`
|
||||
Cleared int `json:"cleared"`
|
||||
Verified bool `json:"verified"`
|
||||
Foreign map[string]string `json:"foreign_scopes,omitempty"`
|
||||
}
|
||||
|
||||
// ClearForeignScope 清掉"归档里确实有、却仍以来源身份躺在目标里"的行。
|
||||
//
|
||||
// 它只做这一件事,而且必须全部条件成立才允许删:
|
||||
// 1. 目标这些行的内容与指定归档行逐字段一致(先证明它们是这次写入的副本,
|
||||
// 不是目标自己的数据);
|
||||
// 2. 先把它们完整导出成 0600 的备份文件并记下摘要;
|
||||
// 3. 删除只按"归档身份集合 + 来源租户"精确命中,不做范围广删。
|
||||
//
|
||||
// 清完还要复核:来源身份不再存在、目标内容与预备份一致、整库指纹里没有别的租户。
|
||||
func (t *ProductionLibraryTarget) ClearForeignScope(ctx context.Context, source ProductionScope, snapshotID,
|
||||
backupDir string, apply bool) (*ResidueReport, error) {
|
||||
if source.TenantID == 0 || !validSearchIdentifier(source.KnowledgeBaseID) {
|
||||
return nil, errors.New("来源范围不完整,拒绝清理")
|
||||
}
|
||||
if source.TenantID == t.scope.TenantID && source.KnowledgeBaseID == t.scope.KnowledgeBaseID {
|
||||
return nil, errors.New("来源与目标范围相同,不需要清理")
|
||||
}
|
||||
if backupDir == "" {
|
||||
return nil, errors.New("清理前必须指定备份目录")
|
||||
}
|
||||
if err := t.resolveStorageBackend(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return nil, errors.New("归档不可用")
|
||||
}
|
||||
defer archive.Close()
|
||||
manifest, err := archive.Load(snapshotID)
|
||||
if err != nil {
|
||||
return nil, errors.New("指定快照不可读")
|
||||
}
|
||||
report := &ResidueReport{SnapshotID: snapshotID, SourceTenant: source.TenantID, SourceKB: source.KnowledgeBaseID,
|
||||
Tables: map[string]int{}, Applied: false}
|
||||
backup := map[string][]map[string]json.RawMessage{}
|
||||
digests := map[string]string{}
|
||||
identities := map[string][]string{}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" {
|
||||
continue
|
||||
}
|
||||
archived, archivedDigests, err := archivedRowsWithDigests(archive, part)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 事故里的残留行是"归档行 + 只把桶/后端换成目标"的旧变换结果:判等时必须
|
||||
// 同时接受原样与这一种变换,否则会把真残留当成"不是这次写的"而拒绝清理。
|
||||
if part.Name == "resources" {
|
||||
for _, row := range archived {
|
||||
legacy, legacyErr := t.legacyResourceRow(row)
|
||||
if legacyErr != nil {
|
||||
return nil, legacyErr
|
||||
}
|
||||
digest, digestErr := canonicalRowDigest(legacy)
|
||||
if digestErr != nil {
|
||||
return nil, digestErr
|
||||
}
|
||||
archivedDigests[digest] = true
|
||||
}
|
||||
}
|
||||
if len(archived) == 0 {
|
||||
continue
|
||||
}
|
||||
live, err := t.foreignScopeRows(ctx, part.Name, source)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
liveDigests := make(map[string]string, len(live))
|
||||
for _, row := range live {
|
||||
digest, err := canonicalRowDigest(row)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
liveDigests[digest] = archiveRowIdentity(part.Name, row)
|
||||
}
|
||||
// 目标里这一范围的行必须都能在归档里找到同样内容:否则它们不是这次写入的
|
||||
// 副本,删了就是删别人的数据。
|
||||
rows := make([]map[string]json.RawMessage, 0, len(live))
|
||||
ids := make([]string, 0, len(live))
|
||||
for digest, identity := range liveDigests {
|
||||
if _, ok := archivedDigests[digest]; !ok {
|
||||
return nil, fmt.Errorf("目标 %s 的行 %s 与指定归档不一致,拒绝按本次写入清理", part.Name, identity)
|
||||
}
|
||||
ids = append(ids, identity)
|
||||
}
|
||||
for _, row := range live {
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
report.Tables[part.Name] = len(rows)
|
||||
backup[part.Name] = rows
|
||||
identities[part.Name] = ids
|
||||
digests[part.Name] = combineRowDigests(sortedKeys(liveDigests))
|
||||
}
|
||||
total := 0
|
||||
for _, count := range report.Tables {
|
||||
total += count
|
||||
}
|
||||
if total == 0 {
|
||||
report.Verified = true
|
||||
return report, nil
|
||||
}
|
||||
if err := os.MkdirAll(backupDir, 0o700); err != nil {
|
||||
return nil, fmt.Errorf("创建备份目录失败: %w", err)
|
||||
}
|
||||
payload, err := json.MarshalIndent(map[string]any{
|
||||
"snapshot_id": snapshotID, "source_tenant": source.TenantID, "source_kb": source.KnowledgeBaseID,
|
||||
"target_tenant": t.scope.TenantID, "target_kb": t.scope.KnowledgeBaseID,
|
||||
"tables": report.Tables, "rows": backup, "digests": digests,
|
||||
}, "", " ")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
report.BackupPath = filepath.Join(backupDir, "foreign-scope-residue-"+snapshotID[:12]+".json")
|
||||
if err := os.WriteFile(report.BackupPath, payload, 0o600); err != nil {
|
||||
return nil, fmt.Errorf("写残留备份失败: %w", err)
|
||||
}
|
||||
sum := sha256.Sum256(payload)
|
||||
report.BackupSHA256 = hex.EncodeToString(sum[:])
|
||||
if !apply {
|
||||
return report, nil
|
||||
}
|
||||
// 删除顺序:子表先、父表后(resource_bindings → resources,chunks → knowledges)。
|
||||
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
|
||||
table := productionLibraryTables[index]
|
||||
ids, ok := identities[table]
|
||||
if !ok || len(ids) == 0 {
|
||||
continue
|
||||
}
|
||||
statement := fmt.Sprintf(`DELETE FROM %s WHERE id = ANY($1::text[])`, table)
|
||||
if !tableHasTenant(table) {
|
||||
// knowledge_tag_relations 没有 id:身份就是父文档 + 标签。
|
||||
statement = `DELETE FROM knowledge_tag_relations WHERE (knowledge_id||':'||tag_id) = ANY($1::text[])`
|
||||
}
|
||||
if _, err := t.pool.Exec(ctx, statement, ids); err != nil {
|
||||
return nil, fmt.Errorf("清理 %s 失败: %w", table, err)
|
||||
}
|
||||
}
|
||||
report.Applied = true
|
||||
report.Cleared = total
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// legacyResourceRow 复现旧实现写资源行时的变换:只换定位符里的存储后端与桶,
|
||||
// 租户/知识库/provider 原样保留。判等要认这一种形状,事故残留才是"可证明"的。
|
||||
func (t *ProductionLibraryTarget) legacyResourceRow(row map[string]json.RawMessage) (map[string]json.RawMessage, error) {
|
||||
copied := make(map[string]json.RawMessage, len(row))
|
||||
for key, value := range row {
|
||||
copied[key] = value
|
||||
}
|
||||
if t.config.ObjectsBucket == "" {
|
||||
return copied, nil
|
||||
}
|
||||
raw, ok := copied["physical_path"]
|
||||
if !ok {
|
||||
return nil, errors.New("资源行缺少定位符")
|
||||
}
|
||||
var locator string
|
||||
if err := json.Unmarshal(raw, &locator); err != nil {
|
||||
return nil, errors.New("资源定位符不可解析")
|
||||
}
|
||||
if rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket); ok {
|
||||
encoded, err := json.Marshal(rewritten)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
copied["physical_path"] = encoded
|
||||
}
|
||||
if t.storageBackend == "" {
|
||||
return copied, nil
|
||||
}
|
||||
if raw, ok := copied["storage_backend_id"]; ok {
|
||||
var current string
|
||||
if json.Unmarshal(raw, ¤t) == nil && current != "" && current != t.storageBackend {
|
||||
encoded, err := json.Marshal(t.storageBackend)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
copied["storage_backend_id"] = encoded
|
||||
}
|
||||
}
|
||||
return copied, nil
|
||||
}
|
||||
|
||||
// VerifyResidueCleared 复核清理结果:来源身份不再存在、其它租户的解释得清、
|
||||
// 且(可选)目标内容与预备份完全一致。
|
||||
func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, source ProductionScope) error {
|
||||
for _, table := range productionLibraryTables {
|
||||
if !tableHasTenant(table) {
|
||||
continue
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1`, table)
|
||||
arguments := []any{source.TenantID}
|
||||
if tablesWithKB[table] {
|
||||
statement += " AND knowledge_base_id=$2"
|
||||
arguments = append(arguments, source.KnowledgeBaseID)
|
||||
}
|
||||
var count int64
|
||||
if err := t.pool.QueryRow(ctx, statement, arguments...).Scan(&count); err != nil {
|
||||
return fmt.Errorf("复核 %s 失败: %w", table, err)
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("目标 %s 仍有 %d 行属于来源范围 %d/%s", table, count, source.TenantID, source.KnowledgeBaseID)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ForeignScopes 列出目标里所有"不是目标租户"的租户,用于人工确认没有别的泄漏。
|
||||
func (t *ProductionLibraryTarget) ForeignScopes(ctx context.Context) (map[string]string, error) {
|
||||
foreign := map[string]string{}
|
||||
for _, table := range productionLibraryTables {
|
||||
if !tableHasTenant(table) {
|
||||
continue
|
||||
}
|
||||
rows, err := t.pool.Query(ctx, fmt.Sprintf(
|
||||
`SELECT tenant_id, count(*) FROM %s WHERE tenant_id <> $1 GROUP BY 1`, table), t.scope.TenantID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("读取 %s 租户分布失败: %w", table, err)
|
||||
}
|
||||
for rows.Next() {
|
||||
var tenant, count int64
|
||||
if err := rows.Scan(&tenant, &count); err != nil {
|
||||
rows.Close()
|
||||
return nil, err
|
||||
}
|
||||
foreign[fmt.Sprintf("%s:%d", table, tenant)] = fmt.Sprintf("%d 行", count)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return foreign, nil
|
||||
}
|
||||
|
||||
// foreignScopeRows 读出目标里属于来源范围的行(resources / resource_bindings
|
||||
// 只有租户列,按租户取;其余按租户+知识库取)。
|
||||
func (t *ProductionLibraryTarget) foreignScopeRows(ctx context.Context, table string, source ProductionScope) (
|
||||
[]map[string]json.RawMessage, error) {
|
||||
if !tableHasTenant(table) {
|
||||
return nil, nil
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table))
|
||||
arguments := []any{source.TenantID, source.KnowledgeBaseID}
|
||||
if table == "resources" || table == "resource_bindings" {
|
||||
statement = fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE tenant_id=$1`, table, table)
|
||||
arguments = arguments[:1]
|
||||
}
|
||||
rows, err := t.pool.Query(ctx, statement, arguments...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("读取目标 %s 来源范围行失败: %w", table, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
result := make([]map[string]json.RawMessage, 0, 8)
|
||||
for rows.Next() {
|
||||
var row map[string]json.RawMessage
|
||||
if err := rows.Scan(&row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result = append(result, row)
|
||||
}
|
||||
return result, rows.Err()
|
||||
}
|
||||
|
||||
// archivedRowsWithDigests 读出一份归档部件的行与规范化内容摘要。
|
||||
func archivedRowsWithDigests(archive *librarysnapshot.Archive, part librarysnapshot.Part) (
|
||||
[]map[string]json.RawMessage, map[string]bool, error) {
|
||||
reader, err := archive.OpenPart(part)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
defer reader.Close()
|
||||
decoder := json.NewDecoder(reader)
|
||||
token, err := decoder.Token()
|
||||
if err != nil || token != json.Delim('[') {
|
||||
return nil, nil, errors.New("归档部件不是行数组")
|
||||
}
|
||||
rows := make([]map[string]json.RawMessage, 0, 64)
|
||||
digests := map[string]bool{}
|
||||
for decoder.More() {
|
||||
var row map[string]json.RawMessage
|
||||
if err := decoder.Decode(&row); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
digest, err := canonicalRowDigest(row)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
digests[digest] = true
|
||||
rows = append(rows, row)
|
||||
}
|
||||
return rows, digests, nil
|
||||
}
|
||||
|
||||
func sortedKeys(values map[string]string) []string {
|
||||
keys := make([]string, 0, len(values))
|
||||
for key := range values {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
@@ -0,0 +1,262 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// 整库同步是"把来源知识库的内容搬到目标的租户/知识库",但归档行带着来源身份
|
||||
// (tenant_id / knowledge_base_id)。原样插入会让行落在来源范围里:目标范围的
|
||||
// 删除、校验、预备份与恢复全都看不见它们(真实事故:同步失败后"已恢复到预备份",
|
||||
// 目标却多出 4 文件 / 38 分块 / 21 资源 / 4 绑定,行所属租户还是来源租户)。
|
||||
//
|
||||
// 因此写入与校验必须共用这一个改写入口,并且对不上就失败——宁可不写,也不写
|
||||
// 一行目标范围外的数据。
|
||||
|
||||
// tablesWithKB 是真实 WeKnora v0.7.1 里带 knowledge_base_id 的知识域表。
|
||||
// resources / resource_bindings 是多知识库共享表,只有 tenant_id;
|
||||
// knowledge_tag_relations 两列都没有,归属完全由父表 knowledges 决定。
|
||||
var tablesWithKB = map[string]bool{
|
||||
"knowledges": true, "chunks": true, "knowledge_tags": true,
|
||||
"wiki_folders": true, "wiki_pages": true, "wiki_page_issues": true, "wiki_log_entries": true,
|
||||
}
|
||||
|
||||
// tableHasTenant 报告这张表是否自带租户列。
|
||||
func tableHasTenant(table string) bool { return table != "knowledge_tag_relations" }
|
||||
|
||||
// bindSnapshotScope 记下快照的来源身份。没有来源身份就无法判断"这行要搬到哪个
|
||||
// 范围",宁可失败也不要按行内容猜。
|
||||
func (t *ProductionLibraryTarget) bindSnapshotScope(manifest manifestScope) error {
|
||||
if manifest.tenantID == 0 || !validSearchIdentifier(manifest.knowledgeBaseID) {
|
||||
return errors.New("快照缺少来源范围身份(tenant_id / knowledge_base_id),无法确定目标范围映射")
|
||||
}
|
||||
t.snapshotTenant = manifest.tenantID
|
||||
t.snapshotKB = manifest.knowledgeBaseID
|
||||
return nil
|
||||
}
|
||||
|
||||
// manifestScope 只是把快照的两个范围字段抽出来,避免调用点重复解构。
|
||||
type manifestScope struct {
|
||||
tenantID uint64
|
||||
knowledgeBaseID string
|
||||
}
|
||||
|
||||
// retargetRow 把一行归档数据改写成目标范围的数据:tenant_id / knowledge_base_id
|
||||
// 换成目标身份,资源行还要把定位符、存储后端、提供方换成目标实例自己的值。
|
||||
func (t *ProductionLibraryTarget) retargetRow(table string, row map[string]json.RawMessage) error {
|
||||
if !tableHasTenant(table) {
|
||||
return nil
|
||||
}
|
||||
if err := t.retargetTenantField(row); err != nil {
|
||||
return fmt.Errorf("%s:%w", table, err)
|
||||
}
|
||||
if tablesWithKB[table] {
|
||||
if err := t.retargetKnowledgeBaseField(row); err != nil {
|
||||
return fmt.Errorf("%s:%w", table, err)
|
||||
}
|
||||
}
|
||||
if table == "resources" {
|
||||
return t.rewriteResourceIdentity(row)
|
||||
}
|
||||
if table == "knowledges" {
|
||||
return t.retargetEmbeddingModelField(row)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// retargetEmbeddingModelField 把文档的 embedding 模型换成目标知识库绑定的那一个。
|
||||
// 来源模型 id 在目标实例里通常不存在(目标 models 表可能是空的),留着它等于让
|
||||
// 应用拿一个解析不到的模型去查——库里的向量与查询向量就不是同一套了。
|
||||
func (t *ProductionLibraryTarget) retargetEmbeddingModelField(row map[string]json.RawMessage) error {
|
||||
if t.targetEmbeddingModel == "" {
|
||||
return nil
|
||||
}
|
||||
if _, ok := row["embedding_model_id"]; !ok {
|
||||
return nil
|
||||
}
|
||||
encoded, err := json.Marshal(t.targetEmbeddingModel)
|
||||
if err != nil {
|
||||
return errors.New("编码目标 embedding 模型失败")
|
||||
}
|
||||
row["embedding_model_id"] = encoded
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveEmbeddingModel 读目标知识库自己的 embedding 模型 id。读不到不算错误
|
||||
// (目标可能确实没绑模型),此时保持来源值不动,由契约校验去发现不一致。
|
||||
func (t *ProductionLibraryTarget) resolveEmbeddingModel(ctx context.Context) {
|
||||
var model string
|
||||
if err := t.pool.QueryRow(ctx,
|
||||
`SELECT coalesce(embedding_model_id,'') FROM knowledge_bases WHERE id=$1 AND tenant_id=$2`,
|
||||
t.scope.KnowledgeBaseID, t.scope.TenantID).Scan(&model); err == nil {
|
||||
t.targetEmbeddingModel = model
|
||||
}
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) retargetTenantField(row map[string]json.RawMessage) error {
|
||||
raw, ok := row["tenant_id"]
|
||||
if !ok {
|
||||
return errors.New("行缺少 tenant_id,无法确定要写入哪个范围")
|
||||
}
|
||||
var current uint64
|
||||
if err := json.Unmarshal(raw, ¤t); err != nil {
|
||||
return errors.New("行 tenant_id 不可解析")
|
||||
}
|
||||
if t.snapshotTenant == 0 {
|
||||
return errors.New("快照来源租户未知,拒绝猜测目标租户")
|
||||
}
|
||||
if current != t.snapshotTenant {
|
||||
return fmt.Errorf("行 tenant_id=%d 不属于快照来源租户 %d", current, t.snapshotTenant)
|
||||
}
|
||||
encoded, err := json.Marshal(t.scope.TenantID)
|
||||
if err != nil {
|
||||
return errors.New("编码目标租户失败")
|
||||
}
|
||||
row["tenant_id"] = encoded
|
||||
return nil
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) retargetKnowledgeBaseField(row map[string]json.RawMessage) error {
|
||||
raw, ok := row["knowledge_base_id"]
|
||||
if !ok {
|
||||
return errors.New("行缺少 knowledge_base_id,无法确定要写入哪个知识库")
|
||||
}
|
||||
var current string
|
||||
if err := json.Unmarshal(raw, ¤t); err != nil || current == "" {
|
||||
return errors.New("行 knowledge_base_id 不可解析")
|
||||
}
|
||||
if t.snapshotKB == "" {
|
||||
return errors.New("快照来源知识库未知,拒绝猜测目标知识库")
|
||||
}
|
||||
if current != t.snapshotKB {
|
||||
return fmt.Errorf("行 knowledge_base_id=%s 不属于快照来源知识库 %s", current, t.snapshotKB)
|
||||
}
|
||||
encoded, err := json.Marshal(t.scope.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return errors.New("编码目标知识库失败")
|
||||
}
|
||||
row["knowledge_base_id"] = encoded
|
||||
return nil
|
||||
}
|
||||
|
||||
// rewriteResourceIdentity 把资源行的定位符与提供方换成目标实例的值。
|
||||
//
|
||||
// 只换桶不够:来源行的 scheme(oss/minio…)与对象键里的租户段也是来源实例的,
|
||||
// 目标实例按它取文件会落到别的提供方或别的租户前缀上(真跑:storage backend
|
||||
// not found → invalid path → 500 code1007)。
|
||||
func (t *ProductionLibraryTarget) rewriteResourceIdentity(row map[string]json.RawMessage) error {
|
||||
raw, ok := row["physical_path"]
|
||||
if !ok {
|
||||
return errors.New("资源行缺少定位符")
|
||||
}
|
||||
var locator string
|
||||
if err := json.Unmarshal(raw, &locator); err != nil {
|
||||
return errors.New("资源定位符不可解析")
|
||||
}
|
||||
rewritten, err := retargetLocator(locator, t.storageBackend, t.storageProvider, t.config.ObjectsBucket,
|
||||
t.snapshotTenant, t.scope.TenantID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if rewritten != locator {
|
||||
encoded, err := json.Marshal(rewritten)
|
||||
if err != nil {
|
||||
return errors.New("编码目标定位符失败")
|
||||
}
|
||||
row["physical_path"] = encoded
|
||||
}
|
||||
if t.storageBackend != "" {
|
||||
if _, ok := row["storage_backend_id"]; ok {
|
||||
encoded, err := json.Marshal(t.storageBackend)
|
||||
if err != nil {
|
||||
return errors.New("编码目标存储后端失败")
|
||||
}
|
||||
row["storage_backend_id"] = encoded
|
||||
}
|
||||
}
|
||||
if t.storageProvider != "" {
|
||||
if _, ok := row["provider"]; ok {
|
||||
encoded, err := json.Marshal(t.storageProvider)
|
||||
if err != nil {
|
||||
return errors.New("编码目标提供方失败")
|
||||
}
|
||||
row["provider"] = encoded
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// retargetLocator 把 "storage://<backend>/<scheme>://<bucket>/<key>" 整体换成目标
|
||||
// 实例的真实取值:后端 id、scheme/提供方、桶名,以及对象键里的租户段。
|
||||
// 桶名为空表示目标没有配置对象存储(测试夹具),此时不动定位符。
|
||||
func retargetLocator(locator, backendID, provider, bucket string, sourceTenant, targetTenant uint64) (string, error) {
|
||||
if bucket == "" || locator == "" {
|
||||
return locator, nil
|
||||
}
|
||||
if locator != strings.TrimSpace(locator) {
|
||||
return "", errors.New("资源定位符带首尾空白")
|
||||
}
|
||||
prefix := ""
|
||||
rest := locator
|
||||
if strings.HasPrefix(locator, "storage://") {
|
||||
trimmed := strings.TrimPrefix(locator, "storage://")
|
||||
parts := strings.SplitN(trimmed, "/", 2)
|
||||
if len(parts) != 2 || !validSearchIdentifier(parts[0]) || parts[1] == "" {
|
||||
return "", errors.New("资源定位符结构不可解析")
|
||||
}
|
||||
if backendID != "" {
|
||||
if !validSearchIdentifier(backendID) {
|
||||
return "", errors.New("目标存储后端 id 形状不合法")
|
||||
}
|
||||
parts[0] = backendID
|
||||
}
|
||||
prefix, rest = "storage://"+parts[0]+"/", parts[1]
|
||||
}
|
||||
separator := strings.Index(rest, "://")
|
||||
if separator < 0 {
|
||||
// 裸对象键没有桶与租户段:落盘按配置桶,改写无意义。
|
||||
return locator, nil
|
||||
}
|
||||
scheme := rest[:separator]
|
||||
if !validSearchIdentifier(scheme) {
|
||||
return "", errors.New("资源定位符的提供方不可解析")
|
||||
}
|
||||
schemeless := rest[separator+3:]
|
||||
boundary := strings.Index(schemeless, "/")
|
||||
if boundary <= 0 || boundary == len(schemeless)-1 {
|
||||
return "", errors.New("资源定位符缺少对象键")
|
||||
}
|
||||
key := schemeless[boundary+1:]
|
||||
if provider != "" {
|
||||
if !validSearchIdentifier(provider) {
|
||||
return "", errors.New("目标提供方形状不合法")
|
||||
}
|
||||
scheme = provider
|
||||
}
|
||||
return prefix + scheme + "://" + bucket + "/" + retargetKeyTenant(key, sourceTenant, targetTenant), nil
|
||||
}
|
||||
|
||||
// retargetKeyTenant 把对象键里的来源租户段换成目标租户,并丢掉来源实例自己的
|
||||
// 路径前缀(例如 MINIO_PATH_PREFIX=weknora/),让键与目标实例的写法一致。
|
||||
// 找不到来源租户段时原样返回:此时键本身不含跨租户引用。
|
||||
func retargetKeyTenant(key string, sourceTenant, targetTenant uint64) string {
|
||||
if sourceTenant == 0 || targetTenant == 0 || sourceTenant == targetTenant {
|
||||
return key
|
||||
}
|
||||
source, target := strconv.FormatUint(sourceTenant, 10), strconv.FormatUint(targetTenant, 10)
|
||||
segments := strings.Split(key, "/")
|
||||
for index := 0; index < len(segments) && index < 2; index++ {
|
||||
// 只有"后面还有内容"的段才是路径前缀;整键就是租户号时不该动它。
|
||||
if index >= len(segments)-1 || segments[index] != source {
|
||||
continue
|
||||
}
|
||||
kept := segments[index:]
|
||||
kept[0] = target
|
||||
return strings.Join(kept, "/")
|
||||
}
|
||||
return key
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同),归档行原样写进目标,
|
||||
// 目标范围的删除/校验/恢复都看不到它们。这里把"必须改写哪些列"钉死。
|
||||
|
||||
func scopeTargetFixture() *ProductionLibraryTarget {
|
||||
return &ProductionLibraryTarget{
|
||||
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"},
|
||||
config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
|
||||
storageBackend: "backend-target",
|
||||
storageProvider: "minio",
|
||||
snapshotTenant: 10000,
|
||||
snapshotKB: "kb-source",
|
||||
}
|
||||
}
|
||||
|
||||
func rawRow(t *testing.T, fields map[string]any) map[string]json.RawMessage {
|
||||
t.Helper()
|
||||
row := make(map[string]json.RawMessage, len(fields))
|
||||
for key, value := range fields {
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatalf("encode %s: %v", key, err)
|
||||
}
|
||||
row[key] = encoded
|
||||
}
|
||||
return row
|
||||
}
|
||||
|
||||
func fieldText(t *testing.T, row map[string]json.RawMessage, key string) string {
|
||||
t.Helper()
|
||||
var value string
|
||||
if err := json.Unmarshal(row[key], &value); err != nil {
|
||||
t.Fatalf("%s not a string: %v", key, err)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func fieldUint(t *testing.T, row map[string]json.RawMessage, key string) uint64 {
|
||||
t.Helper()
|
||||
var value uint64
|
||||
if err := json.Unmarshal(row[key], &value); err != nil {
|
||||
t.Fatalf("%s not a number: %v", key, err)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func TestRetargetRowMapsSourceScopeToTarget(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
for _, table := range []string{"knowledges", "chunks", "knowledge_tags", "wiki_pages", "wiki_folders"} {
|
||||
row := rawRow(t, map[string]any{"id": "row-1", "tenant_id": 10000, "knowledge_base_id": "kb-source"})
|
||||
if err := target.retargetRow(table, row); err != nil {
|
||||
t.Fatalf("%s retarget: %v", table, err)
|
||||
}
|
||||
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
|
||||
t.Fatalf("%s tenant_id = %d, want 10001", table, got)
|
||||
}
|
||||
if got := fieldText(t, row, "knowledge_base_id"); got != "kb-target" {
|
||||
t.Fatalf("%s knowledge_base_id = %s, want kb-target", table, got)
|
||||
}
|
||||
}
|
||||
// 共享表只有租户列:不得凭空写一个 knowledge_base_id 出来。
|
||||
binding := rawRow(t, map[string]any{"id": "bind-1", "tenant_id": 10000, "owner_type": "knowledge", "owner_id": "row-1"})
|
||||
if err := target.retargetRow("resource_bindings", binding); err != nil {
|
||||
t.Fatalf("resource_bindings retarget: %v", err)
|
||||
}
|
||||
if got := fieldUint(t, binding, "tenant_id"); got != 10001 {
|
||||
t.Fatalf("binding tenant_id = %d, want 10001", got)
|
||||
}
|
||||
if _, ok := binding["knowledge_base_id"]; ok {
|
||||
t.Fatal("resource_bindings must not gain a knowledge_base_id column")
|
||||
}
|
||||
// knowledge_tag_relations 没有范围列:归属走父表,改写必须是空操作。
|
||||
relation := rawRow(t, map[string]any{"knowledge_id": "row-1", "tag_id": "tag-1"})
|
||||
if err := target.retargetRow("knowledge_tag_relations", relation); err != nil {
|
||||
t.Fatalf("knowledge_tag_relations retarget: %v", err)
|
||||
}
|
||||
if len(relation) != 2 {
|
||||
t.Fatalf("knowledge_tag_relations must stay untouched, got %v", relation)
|
||||
}
|
||||
}
|
||||
|
||||
// 文档行的 embedding 模型必须换成目标知识库绑定的那一个:来源模型 id 在目标
|
||||
// 实例里通常不存在,留着它检索时的查询向量与库里向量就不是同一套。
|
||||
func TestRetargetRowUsesTargetEmbeddingModel(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
target.targetEmbeddingModel = "model-target"
|
||||
row := rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
|
||||
"embedding_model_id": "model-source"})
|
||||
if err := target.retargetRow("knowledges", row); err != nil {
|
||||
t.Fatalf("retarget: %v", err)
|
||||
}
|
||||
if got := fieldText(t, row, "embedding_model_id"); got != "model-target" {
|
||||
t.Fatalf("embedding_model_id = %s, want model-target", got)
|
||||
}
|
||||
// 目标没绑模型时不猜:保持来源值,交给契约校验去发现不一致。
|
||||
plain := scopeTargetFixture()
|
||||
row = rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
|
||||
"embedding_model_id": "model-source"})
|
||||
if err := plain.retargetRow("knowledges", row); err != nil {
|
||||
t.Fatalf("retarget without target model: %v", err)
|
||||
}
|
||||
if got := fieldText(t, row, "embedding_model_id"); got != "model-source" {
|
||||
t.Fatalf("embedding_model_id = %s, want unchanged", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetRowFailsClosedOutsideSnapshotScope(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
cases := map[string]map[string]any{
|
||||
"foreign tenant": {"id": "x", "tenant_id": 20000, "knowledge_base_id": "kb-source"},
|
||||
"foreign knowledge": {"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb-other"},
|
||||
"missing tenant": {"id": "x", "knowledge_base_id": "kb-source"},
|
||||
"missing knowledge kb": {"id": "x", "tenant_id": 10000},
|
||||
}
|
||||
for name, fields := range cases {
|
||||
if err := target.retargetRow("knowledges", rawRow(t, fields)); err == nil {
|
||||
t.Fatalf("%s: retarget must fail closed", name)
|
||||
}
|
||||
}
|
||||
// 快照来源未知时不许猜目标范围。
|
||||
unknown := &ProductionLibraryTarget{scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"}}
|
||||
if err := unknown.retargetRow("knowledges", rawRow(t, map[string]any{"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb"})); err == nil {
|
||||
t.Fatal("unknown snapshot scope must fail closed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetRowRewritesResourceStorageIdentity(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
row := rawRow(t, map[string]any{
|
||||
"id": "res-1",
|
||||
"tenant_id": 10000,
|
||||
"storage_backend_id": "backend-source",
|
||||
"provider": "oss",
|
||||
"physical_path": "storage://backend-source/oss://yzh-weknora-test/weknora/10000/res-1/a.png",
|
||||
})
|
||||
if err := target.retargetRow("resources", row); err != nil {
|
||||
t.Fatalf("retarget resource: %v", err)
|
||||
}
|
||||
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
|
||||
t.Fatalf("resource tenant = %d, want 10001", got)
|
||||
}
|
||||
want := "storage://backend-target/minio://kgseal-1029a/10001/res-1/a.png"
|
||||
if got := fieldText(t, row, "physical_path"); got != want {
|
||||
t.Fatalf("physical_path = %q, want %q", got, want)
|
||||
}
|
||||
if got := fieldText(t, row, "provider"); got != "minio" {
|
||||
t.Fatalf("provider = %q, want minio", got)
|
||||
}
|
||||
if got := fieldText(t, row, "storage_backend_id"); got != "backend-target" {
|
||||
t.Fatalf("storage_backend_id = %q, want backend-target", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetKeyTenantDropsSourcePrefix(t *testing.T) {
|
||||
cases := []struct{ key, want string }{
|
||||
{"weknora/10000/exports/a.png", "10001/exports/a.png"},
|
||||
{"10000/knowledge/file.md", "10001/knowledge/file.md"},
|
||||
{"other/prefix/a.png", "other/prefix/a.png"},
|
||||
{"10000", "10000"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := retargetKeyTenant(tc.key, 10000, 10001); got != tc.want {
|
||||
t.Fatalf("retargetKeyTenant(%q) = %q, want %q", tc.key, got, tc.want)
|
||||
}
|
||||
}
|
||||
// 同租户不编码:键必须原样返回。
|
||||
if got := retargetKeyTenant("weknora/10000/a.png", 10000, 10000); got != "weknora/10000/a.png" {
|
||||
t.Fatalf("same tenant must keep key, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetLocatorRejectsMalformed(t *testing.T) {
|
||||
if _, err := retargetLocator(" not-a-locator ", "backend", "minio", "bucket", 10000, 10001); err == nil {
|
||||
t.Fatal("locator with surrounding spaces must be rejected")
|
||||
}
|
||||
if _, err := retargetLocator("storage://backend/oss://", "backend", "minio", "bucket", 10000, 10001); err == nil {
|
||||
t.Fatal("locator without object key must be rejected")
|
||||
}
|
||||
// 没有目标桶(测试夹具)时不改写,保持原值可继续用于采集/校验。
|
||||
locator := "storage://backend/oss://bucket/key"
|
||||
if got, err := retargetLocator(locator, "backend", "minio", "", 10000, 10001); err != nil || got != locator {
|
||||
t.Fatalf("empty bucket must be a no-op, got %q err=%v", got, err)
|
||||
}
|
||||
if !strings.Contains(retargetKeyTenant("weknora/10000/a", 10000, 10001), "10001") {
|
||||
t.Fatal("tenant segment must be replaced")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// universeFingerprint 是"整实例"的知识域指纹:不限租户/知识库的行数 + 身份集合摘要。
|
||||
//
|
||||
// 为什么不能只比目标范围:恢复只按目标范围删除/回放,写到范围外的残留它看不见
|
||||
// (真实事故:作业报"已恢复到预备份",目标却多出 4 文件 / 38 分块 / 21 资源 /
|
||||
// 4 绑定)。整实例指纹让"多写了什么"必然暴露,同时也能证明恢复没有误删别的
|
||||
// 知识库的行。
|
||||
func universeFingerprint(ctx context.Context, pool *pgxpool.Pool) (map[string]libraryrelease.TableFingerprint, error) {
|
||||
result := make(map[string]libraryrelease.TableFingerprint, len(productionLibraryTables))
|
||||
for _, table := range productionLibraryTables {
|
||||
identity := identityExpr(table)
|
||||
statement := fmt.Sprintf(
|
||||
`SELECT count(*), coalesce(md5(string_agg((%s)::text, E'\n' ORDER BY (%s)::text)), '') FROM %s`,
|
||||
identity, identity, table)
|
||||
var fingerprint libraryrelease.TableFingerprint
|
||||
if err := pool.QueryRow(ctx, statement).Scan(&fingerprint.Count, &fingerprint.Digest); err != nil {
|
||||
return nil, fmt.Errorf("读取整库 %s 指纹失败: %w", table, err)
|
||||
}
|
||||
result[table] = fingerprint
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// archivedRowIdentities 读出一份归档部件的行身份(与校验用的身份口径一致)。
|
||||
func archivedRowIdentities(archive *librarysnapshot.Archive, part librarysnapshot.Part) ([]string, error) {
|
||||
reader, err := archive.OpenPart(part)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer reader.Close()
|
||||
decoder := json.NewDecoder(reader)
|
||||
token, err := decoder.Token()
|
||||
if err != nil || token != json.Delim('[') {
|
||||
return nil, errors.New("归档部件不是行数组")
|
||||
}
|
||||
ids := make([]string, 0, 64)
|
||||
for decoder.More() {
|
||||
var row map[string]json.RawMessage
|
||||
if err := decoder.Decode(&row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identity := archiveRowIdentity(part.Name, row)
|
||||
if identity == "" {
|
||||
continue
|
||||
}
|
||||
ids = append(ids, identity)
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// universeDiff 返回第一处不一致的表;空字符串表示整实例与参照完全一致。
|
||||
func universeDiff(want, got map[string]libraryrelease.TableFingerprint) string {
|
||||
tables := make([]string, 0, len(want))
|
||||
for table := range want {
|
||||
tables = append(tables, table)
|
||||
}
|
||||
sort.Strings(tables)
|
||||
for _, table := range tables {
|
||||
current, ok := got[table]
|
||||
if !ok {
|
||||
return fmt.Sprintf("%s(目标读不到)", table)
|
||||
}
|
||||
if current.Count != want[table].Count {
|
||||
return fmt.Sprintf("%s 行数 %d,参照 %d", table, current.Count, want[table].Count)
|
||||
}
|
||||
if current.Digest != want[table].Digest {
|
||||
return fmt.Sprintf("%s 身份集合与参照不同(同数量改写也会被这条拦住)", table)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// assertNoSourceScopeRows 找出"这次要写的这些行,仍然以来源身份躺在目标里"。
|
||||
//
|
||||
// 判据按归档里的身份集合逐表核对,而不是"这个租户/知识库有没有行":同一个租户
|
||||
// 里本来就可能住着别的知识库(共享资源、别的 KB 的绑定),按范围数会把它们误
|
||||
// 报成残留。这里只认"本次写集合里、仍属于来源租户"的行——它们只可能来自逃出
|
||||
// 目标范围的写入,而恢复只按目标范围回放,永远删不到它们。
|
||||
func (t *ProductionLibraryTarget) assertNoSourceScopeRows(ctx context.Context, archive *librarysnapshot.Archive,
|
||||
manifest librarysnapshot.Manifest) error {
|
||||
if t.snapshotTenant == 0 {
|
||||
return nil
|
||||
}
|
||||
if t.snapshotTenant == t.scope.TenantID && t.snapshotKB == t.scope.KnowledgeBaseID {
|
||||
return nil
|
||||
}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" {
|
||||
continue
|
||||
}
|
||||
if !tableHasTenant(part.Name) {
|
||||
continue
|
||||
}
|
||||
ids, err := archivedRowIdentities(archive, part)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
continue
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND id = ANY($2::text[])`, part.Name)
|
||||
var count int64
|
||||
if err := t.pool.QueryRow(ctx, statement, t.snapshotTenant, ids).Scan(&count); err != nil {
|
||||
return fmt.Errorf("检查目标 %s 的来源范围残留失败: %w", part.Name, err)
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("目标 %s 残留 %d 行仍属于来源范围 %d/%s:有写入逃出了目标范围,恢复不会清掉它们",
|
||||
part.Name, count, t.snapshotTenant, t.snapshotKB)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -60,8 +60,11 @@ func TestRewriteLocatorSwapsStorageBackendWithBucket(t *testing.T) {
|
||||
// 行变换用解析到的目标后端与目标桶:写入与校验共用这一份,摘要才不会两边打架。
|
||||
func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
|
||||
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
|
||||
storageBackend: "target-backend"}
|
||||
row := map[string]json.RawMessage{"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
|
||||
storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
|
||||
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
|
||||
row := map[string]json.RawMessage{
|
||||
"tenant_id": json.RawMessage(`10001`),
|
||||
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
|
||||
if err := target.rowTransform("resources")(row); err != nil {
|
||||
t.Fatalf("row transform failed: %v", err)
|
||||
}
|
||||
@@ -78,8 +81,10 @@ func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
|
||||
// 只改定位符不改这一列,目标仍是 storage backend not found(真跑 500)。
|
||||
func TestResourceRowTransformRewritesBackendColumn(t *testing.T) {
|
||||
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
|
||||
storageBackend: "target-backend"}
|
||||
storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
|
||||
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
|
||||
row := map[string]json.RawMessage{
|
||||
"tenant_id": json.RawMessage(`10001`),
|
||||
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/1/k/ab.md"`),
|
||||
"storage_backend_id": json.RawMessage(`"source-backend"`),
|
||||
}
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Run inside a governance container. Read the real application DB ledger only."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from urllib.parse import parse_qsl, unquote, urlsplit
|
||||
|
||||
|
||||
def connection_env(env):
|
||||
# Keep only the connection properties actually shared by libpq and pgx.
|
||||
# Unknown DSN options can change the database identity or schema search path.
|
||||
u = urlsplit(env['DATABASE' + '_URL'])
|
||||
if u.scheme not in ('postgres', 'postgresql') or not u.hostname or not u.username or not u.path[1:] or u.fragment:
|
||||
raise ValueError('invalid database URL')
|
||||
query = parse_qsl(u.query, strict_parsing=True, keep_blank_values=True)
|
||||
supported = {'sslmode': 'PGSSLMODE', 'sslrootcert': 'PGSSLROOTCERT', 'sslcert': 'PGSSLCERT', 'sslkey': 'PGSSLKEY'}
|
||||
if len(dict(query)) != len(query) or any(key not in supported or not val for key, val in query):
|
||||
raise ValueError('unknown database connection options')
|
||||
safe = {'PATH': '/usr/bin:/bin', 'HOME': env.get('HOME', '/nonexistent'), 'LC_ALL': 'C',
|
||||
'PGHOST': u.hostname, 'PGPORT': str(u.port or 5432), 'PGDATABASE': unquote(u.path[1:]),
|
||||
'PGUSER': unquote(u.username), 'PGCONNECT_TIMEOUT': '5',
|
||||
'PGOPTIONS': '-c default_transaction_read_only=on -c statement_timeout=5000'}
|
||||
for name in ('PGPASSFILE', 'PGSSLMODE', 'PGSSLROOTCERT', 'PGSSLCERT', 'PGSSLKEY'):
|
||||
if env.get(name):
|
||||
safe[name] = env[name]
|
||||
for name in env:
|
||||
if name.startswith('PG') and name not in safe:
|
||||
raise ValueError('unknown inherited database connection options')
|
||||
safe.update({supported[key]: value for key, value in query})
|
||||
if u.password:
|
||||
safe['PGPASSWORD'] = unquote(u.password)
|
||||
for name in ('PGDATABASE', 'PGUSER'):
|
||||
if any(char in safe[name] for char in '\r\n\x00='):
|
||||
raise ValueError('unsafe database identifier')
|
||||
return safe
|
||||
|
||||
|
||||
def main():
|
||||
env = connection_env(os.environ)
|
||||
sql = """SELECT json_build_object('database', current_database(), 'user', current_user,
|
||||
'read_only', current_setting('transaction_read_only'),
|
||||
'versions', (SELECT json_agg(version ORDER BY version) FROM schema_migrations));"""
|
||||
p = subprocess.run(['psql', '-XAtw', '-v', 'ON_ERROR_STOP=1', '-c', sql],
|
||||
env=env, capture_output=True, timeout=12)
|
||||
if p.returncode:
|
||||
raise ValueError('query failed')
|
||||
data = json.loads(p.stdout)
|
||||
if data['read_only'] != 'on' or data['database'] != env['PGDATABASE'] or data['user'] != env['PGUSER']:
|
||||
raise ValueError('database identity mismatch')
|
||||
print(json.dumps(data['versions']))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, KeyError, subprocess.TimeoutExpired):
|
||||
raise SystemExit('read-only migration probe failed') from None
|
||||
@@ -356,6 +356,14 @@ def cmd_internal_web_test() -> int:
|
||||
def cmd_internal_contracts(go: str) -> int:
|
||||
"""版本契约:结构 + 注入身份必须**真 PASS**(SKIP 判红),
|
||||
并且**开发构建不许自称正式版**(负例:标成 release ⇒ 必须红)。"""
|
||||
qa_rc = subprocess.run(py("-m", "unittest", "discover", "-s", "scripts", "-p", "test_qa_env.py", "-v"),
|
||||
cwd=ROOT).returncode
|
||||
if qa_rc:
|
||||
return qa_rc
|
||||
cd_rc = subprocess.run(py("-m", "unittest", "discover", "-s", "scripts", "-p", "test_production_cd.py", "-v"),
|
||||
cwd=ROOT).returncode
|
||||
if cd_rc:
|
||||
return cd_rc
|
||||
snapshot_rc = subprocess.run(py("scripts/development_snapshot.py","--self-test"),cwd=ROOT).returncode
|
||||
if snapshot_rc:
|
||||
return 1
|
||||
|
||||
@@ -1,6 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# 重建并上线 governance api + web(本机 docker build 需要宿主代理 + --network host)
|
||||
set -u
|
||||
|
||||
# Share the image-only Compose switch with CD without entering the legacy build path.
|
||||
compose_up() {
|
||||
"${compose[@]}" up -d --no-build --no-deps "$@"
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == --verified-compose ]]; then
|
||||
[[ "$#" == 7 && "$5" == api && "$6" == web && "$7" == worker && "$4" == /* ]] || exit 2
|
||||
cd -- "$2" || exit 1
|
||||
compose=(docker compose -p "$3" -f "$4")
|
||||
compose_up --pull never "$5" "$6" "$7"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 1
|
||||
[ -z "$(git status --porcelain --untracked-files=normal)" ] || { printf "%s\n" "REFUSED_DIRTY_SOURCE"; exit 1; }
|
||||
TS=$(date +%Y%m%d-%H%M%S)
|
||||
@@ -103,7 +117,9 @@ if [ "${GATES_PREV}" = "true" ] && [[ "${COMPOSE_FILES_EFFECTIVE}" != *compose.r
|
||||
echo " 确实要关闸:ALLOW_GATE_OFF=1 bash scripts/deploy-api-web.sh"
|
||||
exit 1
|
||||
fi
|
||||
docker compose ${COMPOSE_FILES_EFFECTIVE} up -d --no-build --no-deps api web || { echo "UP_FAILED"; exit 1; }
|
||||
read -r -a compose_file_args <<< "$COMPOSE_FILES_EFFECTIVE"
|
||||
compose=(docker compose "${compose_file_args[@]}")
|
||||
compose_up api web || { echo "UP_FAILED"; exit 1; }
|
||||
|
||||
echo "=== 5. 状态"
|
||||
docker ps --filter name=knowledge-governance-p1 --format '{{.Names}} {{.Status}} {{.Image}}'
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
umask 077
|
||||
[ "${SSH_ORIGINAL_COMMAND:-}" = kg-cd-v1 ] || exit 1
|
||||
exec /usr/bin/env -i HOME=/opt/knowledge-governance/cd PATH=/usr/bin:/bin LANG=C.UTF-8 \
|
||||
GIT_TERMINAL_PROMPT=0 /usr/bin/python3 -I /opt/knowledge-governance/cd/bin/production_cd.py serve
|
||||
@@ -0,0 +1,744 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Governance application CD. The installed host copy owns the release boundary."""
|
||||
|
||||
import argparse
|
||||
from contextlib import contextmanager
|
||||
import copy
|
||||
from datetime import datetime, timezone
|
||||
import fcntl
|
||||
import hashlib
|
||||
from html.parser import HTMLParser
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
from urllib.error import URLError
|
||||
from urllib.parse import urlsplit
|
||||
from urllib.request import HTTPRedirectHandler, ProxyHandler, build_opener
|
||||
|
||||
REPO = 'Weknora/knowledge-governance'
|
||||
FORGE = 'https://gitea.zheromiku.top'
|
||||
TARGET = 'cloud-test-governance-main'
|
||||
ORG = '169e6f7f-e508-4a1f-9a9b-63d240439a3d'
|
||||
SPACE = '106e05d1-4091-40b9-8c57-f76dc94f7825'
|
||||
READY = '3e15556f-ca19-417c-8bfd-666634ea5d29'
|
||||
PUBLISHED = '16185166-b8e9-4761-9926-845a1f41c96d'
|
||||
ROLES = ('api', 'web', 'worker')
|
||||
REVISION = 'org.opencontainers.image.revision'
|
||||
DATA = Path('/opt/knowledge-governance/cd')
|
||||
APP = DATA.parent
|
||||
PROJECT = 'knowledge-governance-cloud-test'
|
||||
SHA = re.compile('[0-9a-f]{40}')
|
||||
HASH = re.compile('[0-9a-f]{64}')
|
||||
IMAGE = re.compile('sha256:[0-9a-f]{64}')
|
||||
UUID = re.compile('[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}')
|
||||
GUARD_PREFIX = 'KG_CD_CONFIRM_V1 '
|
||||
|
||||
|
||||
class CDError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def require(ok, message):
|
||||
if not ok:
|
||||
raise CDError(message)
|
||||
|
||||
|
||||
def stamp():
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def read_json(path):
|
||||
return json.loads(path.read_bytes())
|
||||
|
||||
|
||||
def sync_dir(directory):
|
||||
fd = os.open(directory, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
os.fsync(fd)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def atomic_write(path, data, mode=0o600):
|
||||
fd, temp = tempfile.mkstemp(dir=path.parent)
|
||||
try:
|
||||
with os.fdopen(fd, 'wb') as out:
|
||||
os.fchmod(out.fileno(), mode)
|
||||
out.write(data)
|
||||
out.flush()
|
||||
os.fsync(out.fileno())
|
||||
os.replace(temp, path)
|
||||
sync_dir(path.parent)
|
||||
finally:
|
||||
Path(temp).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def record(path, value):
|
||||
atomic_write(path, (json.dumps(value, ensure_ascii=False, sort_keys=True) + '\n').encode())
|
||||
|
||||
|
||||
@contextmanager
|
||||
def deploy_lock(path):
|
||||
fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
|
||||
with os.fdopen(fd, 'r+') as handle:
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
raise CDError('deployment lock busy') from exc
|
||||
yield
|
||||
|
||||
|
||||
def matches(pattern, text):
|
||||
return isinstance(text, str) and pattern.fullmatch(text) is not None
|
||||
|
||||
|
||||
def validate_request(request):
|
||||
actions = {'deploy': 'deploy-production', 'writeback': 'retry-writeback'}
|
||||
action = request.get('action')
|
||||
require(action in actions and request.get('confirm') == actions[action], 'explicit confirmation required')
|
||||
require(matches(SHA, request.get('sha')) and matches(UUID, request.get('task'))
|
||||
and matches(HASH, request.get('bundle')), 'require full SHA, bundle SHA256 and task UUID')
|
||||
require(matches(re.compile('[1-9][0-9]{0,14}'), request.get('run_id')), 'invalid Gitea run ID')
|
||||
|
||||
|
||||
def guard_proof(request, control_sha):
|
||||
require(matches(SHA, control_sha), 'invalid workflow source SHA')
|
||||
binding = {key: request[key] for key in ('run_id', 'action', 'confirm', 'sha', 'bundle', 'task')}
|
||||
binding.update(repository=REPO, target=TARGET, workflow_sha=control_sha)
|
||||
serialized = json.dumps(binding, sort_keys=True, separators=(',', ':')).encode()
|
||||
return GUARD_PREFIX + hashlib.sha256(serialized).hexdigest()
|
||||
|
||||
|
||||
def verify_guard_log(log, request, control_sha):
|
||||
expected = guard_proof(request, control_sha)
|
||||
pattern = re.compile(r'\d{4}-\d{2}-\d{2}T[0-9:.]+Z (' + GUARD_PREFIX + r'[0-9a-f]{64})')
|
||||
proofs = [match.group(1) for line in log.decode().splitlines() if (match := pattern.fullmatch(line))]
|
||||
require(proofs == [expected], 'Gitea guard dispatch proof missing or different')
|
||||
|
||||
|
||||
def hash_file(path):
|
||||
with path.open('rb') as src:
|
||||
return hashlib.file_digest(src, 'sha256').hexdigest()
|
||||
|
||||
|
||||
def immutable_file(path):
|
||||
info = path.lstat()
|
||||
require(stat.S_ISREG(info.st_mode) and info.st_uid == os.geteuid() and not info.st_mode & 0o022,
|
||||
'bundle requires regular files owned by the deployment account')
|
||||
return path
|
||||
|
||||
|
||||
def verify_bundle(directory, request, archives=True):
|
||||
raw = immutable_file(directory / 'release.json').read_bytes()
|
||||
require(hashlib.sha256(raw).hexdigest() == request['bundle'], 'release manifest digest mismatch')
|
||||
manifest = json.loads(raw)
|
||||
require(manifest.get('schema') == 1 and manifest.get('target') == TARGET
|
||||
and manifest.get('sha') == request['sha'] and manifest.get('task') == request['task'],
|
||||
'bundle commit, task or target mismatch')
|
||||
require(set(manifest.get('images', {})) == set(ROLES), 'bundle must include API, web and worker')
|
||||
for role, image in manifest['images'].items():
|
||||
require(matches(IMAGE, image.get('id')) and matches(HASH, image.get('sha256')), 'invalid image record')
|
||||
if archives:
|
||||
archive = immutable_file(directory / (role + '.tar'))
|
||||
require(hash_file(archive) == image['sha256'], f'{role} archive checksum mismatch')
|
||||
versions = manifest.get('migrations')
|
||||
require(isinstance(versions, list) and versions and all(type(n) is int and n > 0 for n in versions)
|
||||
and versions == sorted(set(versions)), 'invalid migration ledger in manifest')
|
||||
return manifest
|
||||
|
||||
|
||||
def verify_contexts(result, contexts):
|
||||
latest = {}
|
||||
for item in result.get('statuses') or []:
|
||||
latest.setdefault(item.get('context'), item.get('status'))
|
||||
require(bool(contexts) and all(latest.get(context) == 'success' for context in contexts),
|
||||
'required CI status is missing or unsuccessful')
|
||||
|
||||
|
||||
def metadata(task, key):
|
||||
value = (task.get('metadata') or {}).get(key)
|
||||
return json.loads(value) if isinstance(value, str) else None
|
||||
|
||||
|
||||
def check_task(task, request, receipt=None):
|
||||
require(task.get('id') == request['task'] and task.get('spaceId') == SPACE, 'HiFox task or space mismatch')
|
||||
status = (task.get('status') or {}).get('id')
|
||||
same_publication = receipt and metadata(task, 'cd_release') == receipt.get('hifox')
|
||||
require(status == READY or (status == PUBLISHED and same_publication), 'HiFox task is not ready for release')
|
||||
evidence = metadata(task, 'cd_acceptance') or {}
|
||||
url = urlsplit(evidence.get('evidence', ''))
|
||||
require(evidence.get('verdict') == 'passed' and evidence.get('sha') == request['sha']
|
||||
and evidence.get('bundle') == request['bundle'] and url.scheme == 'https' and url.hostname
|
||||
and not (url.username or url.password or url.query or url.fragment),
|
||||
'missing or changed business acceptance for this commit and bundle')
|
||||
return evidence
|
||||
|
||||
|
||||
def release(host, request):
|
||||
validate_request(request)
|
||||
path = host.receipt_path
|
||||
existing = read_json(path) if path.exists() else None
|
||||
if request['action'] == 'writeback':
|
||||
require(existing and existing['state'] in ('deployed_hifox_pending', 'published'),
|
||||
'writeback requires a verified deployment receipt')
|
||||
require(all(existing[k] == request[k] for k in ('sha', 'task', 'bundle')), 'receipt binding mismatch')
|
||||
manifest = verify_bundle(host.bundle_path(request), request, archives=False)
|
||||
host.approve(request)
|
||||
host.verify(manifest)
|
||||
return writeback(host, request, existing)
|
||||
|
||||
require(existing is None, 'existing release receipt: use writeback or inspect interrupted/failed deployment')
|
||||
host.approve(request)
|
||||
manifest = verify_bundle(host.bundle_path(request), request)
|
||||
host.check_ci(request)
|
||||
accepted = check_task(host.task(request['task']), request)
|
||||
before = host.baseline()
|
||||
host.pre_switch(before, manifest)
|
||||
receipt = {k: request[k] for k in ('sha', 'task', 'bundle', 'run_id')}
|
||||
receipt.update(state='preparing', target=TARGET, started_at=stamp(), previous=before)
|
||||
record(path, receipt)
|
||||
try:
|
||||
host.load(host.bundle_path(request), manifest)
|
||||
host.pre_switch(before, manifest)
|
||||
host.approve(request)
|
||||
require(check_task(host.task(request['task']), request) == accepted, 'business acceptance changed before switch')
|
||||
except (CDError, OSError, ValueError, KeyError) as exc:
|
||||
receipt.update(state='failed_before_cutover')
|
||||
record(path, receipt)
|
||||
raise CDError(f'failed_before_cutover: {exc}' if isinstance(exc, CDError) else 'failed_before_cutover') from exc
|
||||
|
||||
receipt.update(state='cutover_started')
|
||||
record(path, receipt)
|
||||
try:
|
||||
host.switch(before, manifest)
|
||||
evidence = host.verify(manifest)
|
||||
except (CDError, OSError, ValueError, KeyError) as exc:
|
||||
status = 'rollback_failed'
|
||||
try:
|
||||
host.rollback(before)
|
||||
status = 'rolled_back'
|
||||
except (CDError, OSError, ValueError, KeyError):
|
||||
pass
|
||||
receipt.update(state=status, finished_at=stamp(), failure=str(exc) if isinstance(exc, CDError) else 'operation failed')
|
||||
record(path, receipt)
|
||||
raise CDError(f'{status}: deployment verification failed') from exc
|
||||
|
||||
receipt.update(state='deployed_hifox_pending', deployed_at=stamp(), evidence=evidence)
|
||||
receipt['hifox'] = {**evidence, 'target': TARGET, 'bundle': request['bundle'], 'deployed_at': receipt['deployed_at'],
|
||||
'run_url': f'{FORGE}/{REPO}/actions/runs/{request["run_id"]}'}
|
||||
record(path, receipt)
|
||||
return writeback(host, request, receipt)
|
||||
|
||||
|
||||
def writeback(host, request, receipt):
|
||||
try:
|
||||
task = host.task(request['task'])
|
||||
check_task(task, request, receipt)
|
||||
host.callback(request['task'], task, receipt.get('hifox'))
|
||||
receipt.update(state='published')
|
||||
record(host.receipt_path, receipt)
|
||||
except (CDError, OSError, ValueError, KeyError) as exc:
|
||||
raise CDError('DEPLOYED_HIFOX_PENDING: retry writeback for this receipt') from exc
|
||||
return receipt
|
||||
|
||||
|
||||
def with_images(config, images):
|
||||
result = copy.deepcopy(config)
|
||||
require(set(result['services']) == set(ROLES), 'unexpected Compose service set')
|
||||
for role in ROLES:
|
||||
result['services'][role]['image'] = images[role]
|
||||
return result
|
||||
|
||||
|
||||
def run(argv, purpose, *, cwd=None, input_data=None, timeout=30, allowed=(0,), env=None):
|
||||
try:
|
||||
result = subprocess.run(argv, cwd=cwd, input=input_data, capture_output=True, timeout=timeout, env=env)
|
||||
except (OSError, subprocess.TimeoutExpired) as exc:
|
||||
raise CDError(f'{purpose} unavailable or timed out') from exc
|
||||
require(result.returncode in allowed, f'{purpose} failed (exit {result.returncode})')
|
||||
return result.stdout
|
||||
|
||||
|
||||
class NoRedirect(HTTPRedirectHandler):
|
||||
def redirect_request(self, *_args, **_kwargs):
|
||||
raise CDError('unexpected HTTP redirect')
|
||||
|
||||
|
||||
def http(url, *, direct=False, limit=2**20):
|
||||
handlers = [NoRedirect()]
|
||||
if direct:
|
||||
handlers.append(ProxyHandler({}))
|
||||
try:
|
||||
with build_opener(*handlers).open(url, timeout=10) as response:
|
||||
data = response.read(limit + 1)
|
||||
require(len(data) <= limit, 'HTTP response exceeds limit')
|
||||
return data
|
||||
except (URLError, TimeoutError) as exc:
|
||||
raise CDError('HTTP read failed') from exc
|
||||
|
||||
|
||||
class Assets(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.paths = []
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = dict(attrs)
|
||||
path = attrs.get('src' if tag == 'script' else 'href', '')
|
||||
if path and re.fullmatch(r'/assets/[A-Za-z0-9_.-]+\.(?:js|css)', path):
|
||||
self.paths.append(path)
|
||||
|
||||
|
||||
def disk_assets(html):
|
||||
parser = Assets()
|
||||
parser.feed(html.decode())
|
||||
require(any(p.endswith('.js') for p in parser.paths), 'web index has no compiled entry point')
|
||||
return sorted(set(parser.paths))
|
||||
|
||||
|
||||
def image_inspect(image):
|
||||
return json.loads(run(['docker', 'image', 'inspect', image], 'read image identity'))[0]
|
||||
|
||||
|
||||
def inspect(name):
|
||||
obj = json.loads(run(['docker', 'container', 'inspect', name], 'read running container'))[0]
|
||||
require(obj['State']['Status'] == 'running' and matches(IMAGE, obj.get('Image')), 'expected container is not running')
|
||||
return obj
|
||||
|
||||
|
||||
def image_label(image):
|
||||
return image_inspect(image)['Config']['Labels'].get(REVISION, '')
|
||||
|
||||
|
||||
def api_identity(container):
|
||||
raw = run(['docker', 'exec', container, '/app/api', '--version-record-status'], 'read API build identity', allowed=(0, 1))
|
||||
obj = json.loads(raw)
|
||||
require(matches(SHA, obj.get('commit_sha')), 'API binary has no full build identity')
|
||||
return obj['commit_sha']
|
||||
|
||||
|
||||
def migration_files(git, sha):
|
||||
paths = git('ls-tree', '-r', '--name-only', sha, 'internal/platform/postgres/migrations').decode().splitlines()
|
||||
return sorted(int(Path(line).name.split('_', 1)[0]) for line in paths if line.endswith('.up.sql'))
|
||||
|
||||
|
||||
def ledger(container):
|
||||
probe = Path(__file__).with_name('cd_migration_probe.py').read_text()
|
||||
data = run(['docker', 'exec', container, 'python3', '-c', probe], 'read migration ledger')
|
||||
result = json.loads(data)
|
||||
require(isinstance(result, list) and result, 'unexpected migration ledger result')
|
||||
return result
|
||||
|
||||
|
||||
def verify_archive(archive, expected):
|
||||
# docker save by ID omits mutable RepoTags. Refuse archives that could retag another app.
|
||||
with tarfile.open(archive, 'r:') as t:
|
||||
entry = t.getmember('manifest.json')
|
||||
require(entry.isfile() and entry.size < 65536, 'unexpected Docker archive manifest')
|
||||
containers = json.load(t.extractfile(entry))
|
||||
require(len(containers) == 1 and not containers[0].get('RepoTags'), 'archive must contain one untagged image')
|
||||
config = t.getmember(containers[0]['Config'])
|
||||
require(config.isfile() and config.size < 2**20, 'unexpected Docker image config')
|
||||
require('sha256:' + hashlib.sha256(t.extractfile(config).read()).hexdigest() == expected,
|
||||
'archive config does not match approved image ID')
|
||||
for member in t:
|
||||
require((member.isdir() or member.isfile()) and not member.name.startswith('/')
|
||||
and '..' not in Path(member.name).parts, 'unexpected Docker archive path')
|
||||
if 'index.json' in t.getnames():
|
||||
index = t.getmember('index.json')
|
||||
require(index.isfile() and index.size < 65536, 'unexpected OCI archive index')
|
||||
oci = json.load(t.extractfile(index))
|
||||
require(len(oci.get('manifests', [])) == 1 and all(
|
||||
key not in descriptor.get('annotations', {}) for descriptor in oci['manifests']
|
||||
for key in ('org.opencontainers.image.ref.name', 'io.containerd.image.name')),
|
||||
'archive OCI index contains a mutable name')
|
||||
|
||||
|
||||
class LiveHost:
|
||||
def __init__(self, request):
|
||||
self.receipt_path = DATA / 'receipts' / (request['bundle'] + '.json')
|
||||
self.compose = APP / 'compose.json'
|
||||
self.git_path = DATA / 'source.git'
|
||||
self.containers = {role: 'kg-cloud-' + role for role in ROLES}
|
||||
self.health_url = 'http://127.0.0.1:8088'
|
||||
self.request = request
|
||||
|
||||
def bundle_path(self, request):
|
||||
return DATA / 'bundles' / request['bundle']
|
||||
|
||||
def git(self, *args):
|
||||
return run(['git', '-C', str(self.git_path), *args], 'release source check', timeout=90)
|
||||
|
||||
def approve(self, request):
|
||||
repo_url = f'{FORGE}/{REPO}'
|
||||
run_id = request['run_id']
|
||||
run_api = f'{FORGE}/api/v1/repos/{REPO}/actions/runs/{run_id}'
|
||||
dispatch = json.loads(http(run_api))
|
||||
control_sha = dispatch.get('head_sha')
|
||||
trigger_actor = dispatch.get('trigger_actor') or dispatch.get('actor') or {}
|
||||
require(str(dispatch.get('id')) == run_id and dispatch.get('event') == 'workflow_dispatch'
|
||||
and dispatch.get('path') == 'deploy-production.yml@refs/heads/main'
|
||||
and dispatch.get('head_branch') == 'main' and dispatch.get('status') in ('in_progress', 'running')
|
||||
and (dispatch.get('actor') or {}).get('id') == 1 and trigger_actor.get('id') == 1
|
||||
and matches(SHA, control_sha), 'Gitea run is not an active deployment dispatched by the release owner')
|
||||
# Repo secrets can be read by other workflows. Only a completed guard from
|
||||
# the installed workflow and controller may authorize the exact inputs.
|
||||
try:
|
||||
self.git('merge-base', '--is-ancestor', control_sha, 'refs/remotes/origin/main')
|
||||
except CDError:
|
||||
self.fetch_source()
|
||||
self.git('merge-base', '--is-ancestor', control_sha, 'refs/remotes/origin/main')
|
||||
trusted = {'scripts/production_cd.py': Path(__file__).resolve(),
|
||||
'.gitea/workflows/deploy-production.yml': DATA / 'deploy-production.yml'}
|
||||
for name, local in trusted.items():
|
||||
require(self.git('show', f'{control_sha}:{name}') == immutable_file(local).read_bytes(),
|
||||
'Gitea approval workflow differs from installed control files')
|
||||
jobs = json.loads(http(run_api + '/jobs')).get('jobs') or []
|
||||
guards = [job for job in jobs if job.get('name') == 'guard' and job.get('conclusion') == 'success'
|
||||
and str(job.get('run_id')) == run_id and job.get('head_sha') == control_sha
|
||||
and matches(re.compile('[1-9][0-9]{0,14}'), str(job.get('id')))]
|
||||
require(len(guards) == 1, 'Gitea approval guard has not succeeded')
|
||||
guard_id = guards[0]['id']
|
||||
verify_guard_log(http(f'{repo_url}/actions/runs/{run_id}/jobs/{guard_id}/logs'), request, control_sha)
|
||||
|
||||
def fetch_source(self):
|
||||
self.git('fetch', '--no-tags', '--prune', f'{FORGE}/{REPO}.git', '+refs/heads/*:refs/remotes/origin/*')
|
||||
|
||||
def check_ci(self, request):
|
||||
self.fetch_source()
|
||||
refs = self.git('for-each-ref', '--format=%(refname)', '--contains', request['sha'], 'refs/remotes/origin/')
|
||||
require(refs.strip(), 'candidate commit is not reachable from a repository branch')
|
||||
contexts = immutable_file(DATA / 'required-contexts.txt').read_text().splitlines()
|
||||
verify_contexts(json.loads(http(f'{FORGE}/api/v1/repos/{REPO}/commits/{request["sha"]}/status')),
|
||||
[line.strip() for line in contexts if line.strip() and not line.lstrip().startswith('#')])
|
||||
|
||||
def hifox(self, *args):
|
||||
cli = DATA / 'bin/hifox'
|
||||
env = {'PATH': '/usr/local/bin:/usr/bin:/bin', 'HOME': str(DATA / 'hifox-home'), 'CI': '1'}
|
||||
raw = run([str(cli), '--organization-id', ORG, *args, '--output', 'json'], 'HiFox operation', env=env)
|
||||
result = json.loads(raw)
|
||||
require(isinstance(result, dict) and result.get('success') is not False and not result.get('error'),
|
||||
'HiFox did not return a successful object')
|
||||
return result
|
||||
|
||||
def task(self, task):
|
||||
return self.hifox('task', 'view', task)
|
||||
|
||||
def callback(self, task_id, task, expected):
|
||||
if metadata(task, 'cd_release') != expected:
|
||||
self.hifox('task', 'metadata', 'set', task_id, 'cd_release', json.dumps(json.dumps(expected)))
|
||||
fresh = self.task(task_id)
|
||||
check_task(fresh, self.request, {'hifox': expected})
|
||||
require(metadata(fresh, 'cd_release') == expected, 'HiFox release receipt readback mismatch')
|
||||
if fresh['status']['id'] != PUBLISHED:
|
||||
self.hifox('task', 'edit', task_id, '--status', PUBLISHED)
|
||||
final = self.task(task_id)
|
||||
final_status = (final.get('status') or {}).get('id')
|
||||
require(final_status == PUBLISHED and metadata(final, 'cd_release') == expected, 'HiFox published readback mismatch')
|
||||
|
||||
def compose_cmd(self, *args):
|
||||
return ['docker', 'compose', '-p', PROJECT, '-f', str(self.compose), *args]
|
||||
|
||||
def rendered_config(self):
|
||||
raw = run(self.compose_cmd('config', '--format', 'json'), 'render Compose configuration', cwd=APP)
|
||||
return json.loads(raw)['services']
|
||||
|
||||
def runtime(self):
|
||||
rendered = self.rendered_config()
|
||||
require(set(rendered) == set(ROLES), 'unexpected Compose service set')
|
||||
result = {}
|
||||
for role, name in self.containers.items():
|
||||
c = inspect(name)
|
||||
labels = c['Config']['Labels']
|
||||
require(labels.get('com.docker.compose.project') == PROJECT
|
||||
and labels.get('com.docker.compose.service') == role
|
||||
and labels.get('com.docker.compose.project.working_dir') == str(APP)
|
||||
and labels.get('com.docker.compose.project.config_files') == str(self.compose),
|
||||
f'{role} does not belong to the approved Compose chain')
|
||||
config = rendered[role]
|
||||
image = image_inspect(c['Image'])
|
||||
require(image_inspect(config['image'])['Id'] == c['Image'], f'{role} Compose image has drifted')
|
||||
env = lambda entries: dict(item.split('=', 1) for item in entries or [])
|
||||
expected_env = env(image['Config'].get('Env'))
|
||||
expected_env.update({key: str(value) for key, value in (config.get('environment') or {}).items()
|
||||
if value is not None})
|
||||
require(env(c['Config'].get('Env')) == expected_env, f'{role} running environment has drifted')
|
||||
expected_mounts = config.get('volumes') or []
|
||||
require(all(mount['type'] == 'bind' for mount in expected_mounts), 'unexpected non-bind Compose volume')
|
||||
mounts = {(v['Source'], v['Destination'], not v['RW']) for v in c.get('Mounts', []) if v['Type'] == 'bind'}
|
||||
require(mounts == {(v['source'], v['target'], v.get('read_only', False)) for v in expected_mounts}
|
||||
and len(mounts) == len(c.get('Mounts', [])), f'{role} running mounts have drifted')
|
||||
require(not config.get('network_mode') or config['network_mode'] == c['HostConfig']['NetworkMode'],
|
||||
f'{role} running network has drifted')
|
||||
result[role] = c['Image']
|
||||
return result
|
||||
|
||||
def baseline(self):
|
||||
images = self.runtime()
|
||||
self.probe_web()
|
||||
return {'compose_sha256': hash_file(self.compose), 'images': images,
|
||||
'revisions': {role: image_label(image) for role, image in images.items()}}
|
||||
|
||||
def pre_switch(self, previous, manifest):
|
||||
require(hash_file(self.compose) == previous['compose_sha256'] and self.runtime() == previous['images'],
|
||||
'target baseline changed during deployment')
|
||||
for revision in set(previous['revisions'].values()):
|
||||
require(matches(SHA, revision), 'running image has no source revision')
|
||||
self.git('merge-base', '--is-ancestor', revision, manifest['sha'])
|
||||
require(migration_files(self.git, manifest['sha']) == manifest['migrations'], 'source migration set differs from bundle')
|
||||
for role in ('api', 'worker'):
|
||||
require(ledger(self.containers[role]) == manifest['migrations'],
|
||||
f'{role} migration ledger mismatch; verify schema changes outside CD')
|
||||
|
||||
def load(self, directory, manifest):
|
||||
archives = {role: directory / (role + '.tar') for role in ROLES}
|
||||
for role, archive in archives.items():
|
||||
verify_archive(archive, manifest['images'][role]['id'])
|
||||
for role, archive in archives.items():
|
||||
image = manifest['images'][role]['id']
|
||||
run(['docker', 'image', 'load', '-i', str(archive)], f'load {role} image', timeout=600)
|
||||
require(image_inspect(image)['Id'] == image and image_label(image) == manifest['sha'],
|
||||
f'{role} loaded image differs from approved build')
|
||||
|
||||
@property
|
||||
def backup(self):
|
||||
return self.receipt_path.with_suffix('.compose.backup')
|
||||
|
||||
def switch(self, previous, manifest):
|
||||
current = self.compose.read_bytes()
|
||||
require(hashlib.sha256(current).hexdigest() == previous['compose_sha256'], 'Compose changed before switch')
|
||||
images = {role: manifest['images'][role]['id'] for role in ROLES}
|
||||
changed = with_images(json.loads(current), images)
|
||||
for role, image in previous['images'].items():
|
||||
run(['docker', 'image', 'tag', image, f'knowledge-governance-cd-rollback:{self.request["bundle"]}-{role}'],
|
||||
'retain rollback image')
|
||||
atomic_write(self.backup, current)
|
||||
atomic_write(self.compose, (json.dumps(changed, indent=2) + '\n').encode(), self.compose.stat().st_mode & 0o777)
|
||||
self.recreate()
|
||||
|
||||
def recreate(self):
|
||||
script = Path(__file__).with_name('deploy-api-web.sh')
|
||||
run(['bash', str(script), '--verified-compose', str(APP), PROJECT, str(self.compose), 'api', 'web', 'worker'],
|
||||
'switch governance Compose services', timeout=120)
|
||||
|
||||
def probe_web(self):
|
||||
origin = self.health_url
|
||||
require(json.loads(http(origin + '/health', direct=True)).get('status') == 'ok', 'web proxy health probe failed')
|
||||
require(json.loads(http('http://127.0.0.1:18098/readyz', direct=True)).get('status') in ('ok', 'ready'),
|
||||
'API readiness probe failed')
|
||||
index = http(origin + '/', direct=True)
|
||||
return index, {path: hashlib.sha256(http(origin + path, direct=True, limit=32 * 2**20)).hexdigest()
|
||||
for path in disk_assets(index)}
|
||||
|
||||
def verify_web_image(self):
|
||||
origin_index, assets = self.probe_web()
|
||||
name = self.containers['web']
|
||||
def read_asset(path):
|
||||
return run(['docker', 'exec', name, 'cat', '/usr/share/nginx/html' + path], 'read running web asset')
|
||||
|
||||
require(origin_index == read_asset('/index.html'), 'served index differs from running web image')
|
||||
for path, served_digest in assets.items():
|
||||
require(hashlib.sha256(read_asset(path)).hexdigest() == served_digest, 'served asset differs from running web image')
|
||||
return assets
|
||||
|
||||
def verify(self, manifest):
|
||||
images = {role: manifest['images'][role]['id'] for role in ROLES}
|
||||
def check():
|
||||
require(self.runtime() == images, 'running image IDs differ from approved bundle')
|
||||
require(api_identity(self.containers['api']) == manifest['sha'], 'running API SHA mismatches approval')
|
||||
assets = self.verify_web_image()
|
||||
return {'sha': manifest['sha'], 'images': images, 'health': 'ok', 'asset_sha256': assets}
|
||||
|
||||
return wait_ready(check)
|
||||
|
||||
def rollback(self, previous):
|
||||
if not self.backup.exists():
|
||||
require(hash_file(self.compose) == previous['compose_sha256'] and self.runtime() == previous['images'],
|
||||
'rollback baseline cannot be established')
|
||||
self.verify_web_image()
|
||||
return
|
||||
original = self.backup.read_bytes()
|
||||
require(hashlib.sha256(original).hexdigest() == previous['compose_sha256'], 'rollback backup checksum mismatch')
|
||||
source = json.loads(original)
|
||||
for role, image in previous['images'].items():
|
||||
reference = source['services'][role]['image']
|
||||
require(image_inspect(reference)['Id'] == image, 'rollback source tag has drifted')
|
||||
atomic_write(self.compose, original, self.compose.stat().st_mode & 0o777)
|
||||
self.recreate()
|
||||
|
||||
def check():
|
||||
require(self.runtime() == previous['images'], 'rollback running images mismatch')
|
||||
require(api_identity(self.containers['api']) == previous['revisions']['api'], 'rollback API SHA mismatch')
|
||||
self.verify_web_image()
|
||||
|
||||
wait_ready(check)
|
||||
|
||||
|
||||
def wait_ready(check, seconds=90):
|
||||
deadline = time.monotonic() + seconds
|
||||
while True:
|
||||
try:
|
||||
return check()
|
||||
except (CDError, OSError, KeyError, ValueError):
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(3)
|
||||
|
||||
|
||||
def event_request(event, env):
|
||||
require(env.get('GITHUB_EVENT_NAME') == 'workflow_dispatch' and env.get('GITHUB_REF') == 'refs/heads/main'
|
||||
and env.get('GITHUB_REPOSITORY') == REPO, 'manual dispatch from main required')
|
||||
inputs = event.get('inputs') or {}
|
||||
request = {name: inputs.get(name, '') for name in ('action', 'sha', 'task', 'bundle', 'confirm')}
|
||||
request['run_id'] = env.get('GITHUB_RUN_ID', '')
|
||||
validate_request(request)
|
||||
return request
|
||||
|
||||
|
||||
def pack(args):
|
||||
require(matches(SHA, args.sha) and matches(UUID, args.task), 'require full SHA and task UUID')
|
||||
require(not args.output.exists(), 'bundle output already exists')
|
||||
source = lambda *argv: run(['git', *argv], 'read candidate source')
|
||||
require(source('rev-parse', args.sha + '^{commit}').decode().strip() == args.sha, 'source commit not found')
|
||||
images = {}
|
||||
versions = migration_files(source, args.sha)
|
||||
manifest = dict(schema=1, target=TARGET, sha=args.sha, task=args.task, images=images, migrations=versions)
|
||||
with tempfile.TemporaryDirectory(dir=args.output.parent) as directory:
|
||||
build = Path(directory) / "sealed"
|
||||
build.mkdir(mode=0o700)
|
||||
for role in ROLES:
|
||||
image = image_inspect(getattr(args, role).removeprefix('sha256:'))
|
||||
image_id = image['Id']
|
||||
require(matches(IMAGE, image_id) and image_label(image_id) == args.sha, f'{role} source label mismatch')
|
||||
archive = build / (role + '.tar')
|
||||
run(['docker', 'image', 'save', '-o', str(archive), image_id], f'archive {role} image', timeout=900)
|
||||
verify_archive(archive, image_id)
|
||||
archive.chmod(0o600)
|
||||
images[role] = {'id': image_id, 'sha256': hash_file(archive)}
|
||||
record(build / 'release.json', manifest)
|
||||
for item in build.iterdir():
|
||||
with item.open("rb") as artifact:
|
||||
os.fsync(artifact.fileno())
|
||||
sync_dir(build)
|
||||
build.rename(args.output)
|
||||
sync_dir(args.output.parent)
|
||||
print(hash_file(args.output / 'release.json'))
|
||||
|
||||
|
||||
def attest(args):
|
||||
request = {'sha': args.sha, 'task': args.task, 'bundle': hash_file(args.directory / 'release.json')}
|
||||
require(matches(SHA, args.sha) and matches(UUID, args.task), 'require full SHA and task UUID')
|
||||
manifest = verify_bundle(args.directory, request)
|
||||
before = {role: inspect(getattr(args, role)) for role in ROLES}
|
||||
for role, obj in before.items():
|
||||
require(obj['Image'] == manifest['images'][role]['id'] and obj['Name'] != '/kg-cloud-' + role,
|
||||
'isolated test runtime image differs from bundle')
|
||||
require(api_identity(args.api) == args.sha and ledger(args.api) == manifest['migrations']
|
||||
and ledger(args.worker) == manifest['migrations'], 'test runtime SHA or schema differs')
|
||||
base = urlsplit(args.test_url)
|
||||
require(base.scheme in ('http', 'https') and base.hostname and not (base.username or base.password or base.query or base.fragment)
|
||||
and base.path in ('', '/') and base.port != 8088, 'provide isolated test endpoint')
|
||||
origin = args.test_url.rstrip('/')
|
||||
require(json.loads(http(origin + '/health', direct=True)).get('status') == 'ok', 'isolated test health probe failed')
|
||||
index = http(origin + '/', direct=True)
|
||||
require(index == run(['docker', 'exec', args.web, 'cat', '/usr/share/nginx/html/index.html'], 'read test web index'),
|
||||
'test endpoint is serving another web image')
|
||||
for role, obj in before.items():
|
||||
require(inspect(getattr(args, role))['Id'] == obj['Id'], 'test instance changed during readback')
|
||||
print(json.dumps({'sha': args.sha, 'bundle': request['bundle'], 'test_url': args.test_url,
|
||||
'images': {role: obj['Image'] for role, obj in before.items()}}, sort_keys=True))
|
||||
|
||||
|
||||
def cli():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
commands = parser.add_subparsers(dest='command', required=True)
|
||||
for command in ('guard', 'send', 'serve', 'pack', 'attest'):
|
||||
sub = commands.add_parser(command)
|
||||
if command in ('guard', 'send'):
|
||||
sub.add_argument('--event', type=Path, required=True)
|
||||
if command in ('pack', 'attest'):
|
||||
sub.add_argument('--sha', required=True)
|
||||
sub.add_argument('--task', required=True)
|
||||
if command in ('pack', 'attest'):
|
||||
for role in ROLES:
|
||||
sub.add_argument('--' + role, required=True)
|
||||
if command == 'pack':
|
||||
sub.add_argument('--output', type=Path, required=True)
|
||||
if command == 'attest':
|
||||
sub.add_argument('--directory', type=Path, required=True)
|
||||
sub.add_argument('--test-url', required=True)
|
||||
args = parser.parse_args()
|
||||
if args.command in ('guard', 'send'):
|
||||
request = event_request(read_json(args.event), os.environ)
|
||||
if args.command == 'guard':
|
||||
print(guard_proof(request, os.environ.get('GITHUB_SHA')))
|
||||
else:
|
||||
send(request)
|
||||
elif args.command == 'pack':
|
||||
pack(args)
|
||||
elif args.command == 'attest':
|
||||
attest(args)
|
||||
else:
|
||||
raw = sys.stdin.buffer.read(8193)
|
||||
require(len(raw) <= 8192, 'request exceeds size limit')
|
||||
request = json.loads(raw)
|
||||
validate_request(request)
|
||||
with deploy_lock(DATA / 'release.lock'):
|
||||
host = LiveHost(request)
|
||||
for item in (DATA / 'receipts').glob('*.json'):
|
||||
if read_json(item)['state'] in ('cutover_started', 'rollback_failed'):
|
||||
raise CDError('previous deployment outcome requires operator reconciliation')
|
||||
result = release(host, request)
|
||||
print(json.dumps({k: result[k] for k in ('state', 'sha', 'task', 'bundle', 'hifox')}, ensure_ascii=False))
|
||||
|
||||
|
||||
def send(request):
|
||||
key = os.environ.get('KG_CD_SSH_KEY', '')
|
||||
known_hosts = os.environ.get('KG_CD_KNOWN_HOSTS', '')
|
||||
host = os.environ.get('KG_CD_SSH_HOST', '')
|
||||
port = os.environ.get('KG_CD_SSH_PORT', '22') or '22'
|
||||
require(key.strip() and known_hosts.strip(), 'missing KG_CD_SSH_KEY or KG_CD_KNOWN_HOSTS')
|
||||
require(host in ('100.85.111.121', '::1') and port.isdecimal() and 1 <= int(port) <= 65535,
|
||||
'configure the direct or approved tunnel address for the target release endpoint')
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
temp = Path(directory)
|
||||
for filename, value in (('key', key), ('known_hosts', known_hosts)):
|
||||
atomic_write(temp / filename, (value.strip() + '\n').encode())
|
||||
argv = ['ssh', '-F', '/dev/null', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', '-o', 'IdentitiesOnly=yes',
|
||||
'-o', 'ClearAllForwardings=yes', '-o', 'StrictHostKeyChecking=yes', '-o', 'HostKeyAlias=kg-governance-cd',
|
||||
'-o', 'GlobalKnownHostsFile=/dev/null', '-o', f'UserKnownHostsFile={temp / "known_hosts"}',
|
||||
'-i', str(temp / 'key'), '-p', port, '-T', 'root@' + host, 'kg-cd-v1']
|
||||
try:
|
||||
result = subprocess.run(argv, input=json.dumps(request).encode(), capture_output=True, timeout=2100)
|
||||
except (OSError, subprocess.TimeoutExpired) as exc:
|
||||
raise CDError('target outcome unknown; inspect durable receipt before retry') from exc
|
||||
if result.returncode:
|
||||
error = result.stderr.decode(errors='replace')
|
||||
for marker in ('DEPLOYED_HIFOX_PENDING', 'rollback_failed:', 'rolled_back:', 'failed_before_cutover:',
|
||||
'target preflight rejected', 'deployment lock busy', 'writeback requires a verified deployment receipt',
|
||||
'Gitea approval guard has not succeeded', 'Gitea guard dispatch proof missing or different'):
|
||||
if marker in error:
|
||||
raise CDError(error.strip().splitlines()[-1])
|
||||
raise CDError('target rejected request or outcome unknown; inspect durable receipt')
|
||||
response = json.loads(result.stdout)
|
||||
require(response.get('state') == 'published' and all(response.get(k) == request[k] for k in ('sha', 'task', 'bundle')),
|
||||
'invalid or unverified deployment receipt')
|
||||
print(json.dumps(response, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
cli()
|
||||
except CDError as error:
|
||||
print(f'CD: {error}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except (OSError, KeyError, ValueError, TypeError, tarfile.TarError) as error:
|
||||
print(f'CD: operation could not be verified ({type(error).__name__})', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,374 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare and exercise an isolated Knowledge Governance QA run."""
|
||||
|
||||
import argparse
|
||||
from contextlib import ExitStack
|
||||
import fcntl
|
||||
import http.client
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CONFIG = Path.home() / '.config/knowledge-governance/qa.json'
|
||||
API_PORT = 8088
|
||||
WEB_PORT = 5173
|
||||
RUNTIME_RESPONSE = 'KG_QA_RUNTIME_OK'
|
||||
|
||||
|
||||
class QAError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def report(stage, detail):
|
||||
print(f'[qa:{stage}] {detail}', flush=True)
|
||||
|
||||
|
||||
def capture(argv, stage, timeout=10, env=None):
|
||||
try:
|
||||
return subprocess.run(argv, cwd=ROOT, env=env, text=True, capture_output=True,
|
||||
timeout=timeout, stdin=subprocess.DEVNULL)
|
||||
except FileNotFoundError as exc:
|
||||
raise QAError(f'{stage} 缺少命令 {argv[0]}') from exc
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise QAError(f'{stage} 超过 {timeout} 秒') from exc
|
||||
|
||||
|
||||
def private_file(path):
|
||||
try:
|
||||
info = path.lstat()
|
||||
except FileNotFoundError as exc:
|
||||
raise QAError(f'缺少测试配置文件 {path}') from exc
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) != 0o600:
|
||||
raise QAError(f'{path} 必须是当前用户所有的 0600 普通文件')
|
||||
|
||||
|
||||
def database_config(path):
|
||||
private_file(path)
|
||||
try:
|
||||
config = json.loads(path.read_text())
|
||||
except (UnicodeError, json.JSONDecodeError) as exc:
|
||||
raise QAError(f'{path} 不是有效 JSON') from exc
|
||||
if not isinstance(config, dict) or config.keys() != {'host', 'port', 'database', 'user'}:
|
||||
raise QAError('qa.json 只接受 host、port、database、user 四个字段')
|
||||
for field in ('database', 'user'):
|
||||
if not re.fullmatch(r'kg_qa_[a-z0-9_]{1,56}', str(config[field])):
|
||||
raise QAError(f'RDS {field} 必须以 kg_qa_ 开头并使用小写字母、数字或下划线')
|
||||
host = config['host']
|
||||
if not isinstance(host, str) or not re.fullmatch(r'[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)+', host):
|
||||
raise QAError('RDS host 必须填写证书对应的 DNS 主机名')
|
||||
if type(config['port']) is not int or not 1 <= config['port'] <= 65535:
|
||||
raise QAError('RDS port 必须是 1–65535 的整数')
|
||||
return config
|
||||
|
||||
|
||||
def tool_env():
|
||||
allowed = ('HOME', 'PATH', 'LANG', 'TMPDIR', 'GOPROXY', 'GOSUMDB', 'GOPRIVATE', 'GONOSUMDB',
|
||||
'GOMODCACHE', 'GOCACHE', 'npm_config_cache', 'npm_config_registry',
|
||||
'HTTP_PROXY', 'HTTPS_PROXY', 'ALL_PROXY', 'NO_PROXY',
|
||||
'http_proxy', 'https_proxy', 'all_proxy', 'no_proxy')
|
||||
env = {key: os.environ[key] for key in allowed if key in os.environ}
|
||||
env['GOFLAGS'] = '-mod=readonly'
|
||||
return env
|
||||
|
||||
|
||||
def database_env(config, directory):
|
||||
# Keep inherited PG variables and integration switches out of application children.
|
||||
env = tool_env()
|
||||
env.update(PGHOST=config['host'], PGPORT=str(config['port']), PGDATABASE=config['database'],
|
||||
PGUSER=config['user'], PGSSLMODE='verify-full', PGCONNECT_TIMEOUT='5',
|
||||
PGPASSFILE=str(directory / 'qa.pgpass'), PGSSLROOTCERT=str(directory / 'rds-ca.pem'),
|
||||
PGOPTIONS='-c search_path=public', LC_ALL='C',
|
||||
TEST_WEKNORA_ENABLED='false', PP_SCAN_ENABLED='false',
|
||||
RELEASE_WORKER_ENABLED='false', PRODUCTION_RELEASE_REQUESTS_ENABLED='false',
|
||||
DOBOTAI_OPS_READ_ENABLED='false', DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED='false')
|
||||
env['APP_ADDR'] = f'127.0.0.1:{API_PORT}'
|
||||
env['DATA' + 'BASE_URL'] = (
|
||||
f"postgresql://{config['user']}@{config['host']}:{config['port']}/{config['database']}"
|
||||
'?sslmode=verify-full&connect_timeout=5&search_path=public'
|
||||
)
|
||||
return env
|
||||
|
||||
|
||||
def psql(env, sql):
|
||||
result = capture(['psql', '-XAtw', '-v', 'ON_ERROR_STOP=1', '-c', sql], 'RDS 探针', env=env)
|
||||
if result.returncode == 0:
|
||||
return result.stdout.strip()
|
||||
detail = result.stderr.lower()
|
||||
hints = (
|
||||
('password', '口令认证失败,检查 qa.pgpass 和登录角色'),
|
||||
('certificate', 'TLS 校验失败,检查 RDS 主机名和 CA'),
|
||||
('does not exist', '数据库或角色不存在,联系管理员确认专用 QA 库'),
|
||||
('permission denied', 'RDS 权限不足,检查专用 QA 角色的 CONNECT、schema USAGE/CREATE 权限'),
|
||||
('could not translate host name', 'RDS 主机名无法解析,检查私网 DNS'),
|
||||
('timeout', '连接 RDS 超时,检查 QA 虚拟机到 RDS 的私网和安全组'),
|
||||
('connection refused', 'RDS 拒绝 TCP 连接,检查主机名、端口和私网接入'),
|
||||
)
|
||||
hint = next((message for marker, message in hints if marker in detail), '连接或 SQL 探针失败,检查 RDS 接入和权限')
|
||||
raise QAError(f'RDS {hint}(psql 退出 {result.returncode})')
|
||||
|
||||
|
||||
def database_state(env):
|
||||
identity = psql(env, """
|
||||
SELECT concat_ws('|', current_database(), current_user,
|
||||
COALESCE((SELECT ssl FROM pg_stat_ssl WHERE pid = pg_backend_pid()), false),
|
||||
(SELECT NOT (rolsuper OR rolcreaterole OR rolcreatedb OR rolreplication OR rolbypassrls)
|
||||
FROM pg_roles WHERE rolname = current_user),
|
||||
(has_schema_privilege(current_user, 'public', 'USAGE') AND
|
||||
has_schema_privilege(current_user, 'public', 'CREATE')),
|
||||
to_regclass('public.schema_migrations') IS NOT NULL,
|
||||
EXISTS (SELECT FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
|
||||
WHERE n.nspname = 'public' AND c.relname <> 'schema_migrations')
|
||||
OR EXISTS (SELECT FROM pg_namespace
|
||||
WHERE nspname NOT IN ('public', 'information_schema') AND nspname !~ '^pg_')
|
||||
OR EXISTS (SELECT FROM pg_proc p JOIN pg_namespace n ON n.oid=p.pronamespace
|
||||
WHERE n.nspname = 'public'));
|
||||
""")
|
||||
fields = identity.split('|')
|
||||
if len(fields) != 7 or fields[:4] != [env['PGDATABASE'], env['PGUSER'], 't', 't']:
|
||||
raise QAError('RDS 实际库名、角色、TLS 或角色权限与 QA 配置不符')
|
||||
if fields[4] != 't':
|
||||
raise QAError('RDS 角色需要 public schema 的 USAGE 和 CREATE 权限')
|
||||
rows = psql(env, 'SELECT version FROM public.schema_migrations ORDER BY version;') if fields[5] == 't' else ''
|
||||
if any(not re.fullmatch(r'[0-9]+', row) for row in rows.splitlines()):
|
||||
raise QAError('RDS 迁移记录包含无效版本')
|
||||
return [int(row) for row in rows.splitlines()], fields[6] == 't'
|
||||
|
||||
|
||||
def expected_migrations():
|
||||
paths = (ROOT / 'internal/platform/postgres/migrations').glob('*.up.sql')
|
||||
try:
|
||||
versions = sorted(int(path.name.split('_', 1)[0]) for path in paths)
|
||||
except ValueError as exc:
|
||||
raise QAError('仓库存在无效的迁移文件名') from exc
|
||||
if not versions or len(versions) != len(set(versions)):
|
||||
raise QAError('仓库迁移文件缺失或版本重复')
|
||||
return versions
|
||||
|
||||
|
||||
def check_migrations(expected, applied, nonempty, *, complete):
|
||||
if not applied and nonempty:
|
||||
raise QAError('RDS 非空且缺少有效迁移记录,停止初始化')
|
||||
if expected[:len(applied)] != applied:
|
||||
raise QAError('RDS 迁移序列与当前提交不兼容,使用专用空库或匹配的测试库')
|
||||
if complete and applied != expected:
|
||||
raise QAError(f'RDS 迁移未完成,缺少版本 {expected[len(applied)]} 至 {expected[-1]}')
|
||||
|
||||
|
||||
def check_hifox(output):
|
||||
try:
|
||||
state = json.loads(output)
|
||||
online = state['connected'] is True and state['remoteLiveness']['status'] == 'online'
|
||||
except (ValueError, KeyError, TypeError) as exc:
|
||||
raise QAError('HiFox 未返回有效的在线状态,检查本机注册和接收服务') from exc
|
||||
if not online:
|
||||
raise QAError('HiFox 未确认测试机在线,检查本机接收服务和网络')
|
||||
|
||||
|
||||
def check_checkout(expected):
|
||||
if not re.fullmatch(r'[0-9a-f]{40}', expected):
|
||||
raise QAError('设置 KG_QA_SHA 为提测的完整 40 位 Git SHA')
|
||||
actual = capture(['git', 'rev-parse', 'HEAD'], 'Git 检出')
|
||||
if actual.returncode or actual.stdout.strip() != expected:
|
||||
raise QAError('Git 检出与提测提交不一致')
|
||||
state = capture(['git', 'status', '--porcelain'], 'Git 状态')
|
||||
if state.returncode or state.stdout.strip():
|
||||
raise QAError('QA 需要干净的独立检出,请检查 Git 状态')
|
||||
|
||||
|
||||
def preflight(env):
|
||||
for cmd in ('git', 'make', 'go', 'node', 'npm', 'psql', 'hifox', 'omp'):
|
||||
if not shutil.which(cmd, path=env.get('PATH')):
|
||||
raise QAError(f'工具链缺少 {cmd},检查 QA 机器的 PATH')
|
||||
go = capture(['go', 'env', 'GOROOT', 'GOVERSION'], 'Go 工具链', env=env)
|
||||
fields = go.stdout.splitlines()
|
||||
if go.returncode or len(fields) != 2 or not fields[1].startswith('go1.26.'):
|
||||
raise QAError('QA 需要 Go 1.26 工具链')
|
||||
root = Path(fields[0]) / 'bin'
|
||||
if not os.access(root / 'gofmt', os.X_OK):
|
||||
raise QAError('Go 工具链缺少 gofmt,请安装完整的 Go 1.26')
|
||||
env['PATH'] = f"{root}:{env['PATH']}"
|
||||
node = capture(['node', '--version'], 'Node.js 工具链', env=env)
|
||||
if node.returncode or not node.stdout.startswith('v22.'):
|
||||
raise QAError('QA 需要 Node.js 22 工具链')
|
||||
report('toolchain', f'{fields[1]} / Node.js {node.stdout.strip()}')
|
||||
|
||||
hifox = capture(['hifox', 'computer', 'status', '--output', 'json'], 'HiFox 在线检查', timeout=15)
|
||||
if hifox.returncode:
|
||||
raise QAError(f'HiFox 状态命令失败(退出 {hifox.returncode})')
|
||||
check_hifox(hifox.stdout)
|
||||
report('machine', f'{socket.gethostname()} / HiFox online')
|
||||
prompt = f'Reply with exactly {RUNTIME_RESPONSE}. Do not use tools.'
|
||||
omp = capture(['omp', '--no-session', '-p', prompt], 'OMP Runtime 探针', timeout=35)
|
||||
if omp.returncode or RUNTIME_RESPONSE not in {line.strip() for line in omp.stdout.splitlines()}:
|
||||
raise QAError('OMP Runtime 没有返回探针结果,检查 OMP 模型配置和 Magpie 网关')
|
||||
report('runtime', 'OMP 模型请求通过')
|
||||
|
||||
|
||||
def check_ports():
|
||||
for port in (API_PORT, WEB_PORT):
|
||||
with socket.socket() as listener:
|
||||
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
try:
|
||||
listener.bind(('127.0.0.1', port))
|
||||
except OSError as exc:
|
||||
raise QAError(f'QA 端口 {port} 无法绑定,检查已有监听进程') from exc
|
||||
|
||||
|
||||
def run_step(argv, stage, env, timeout):
|
||||
report(stage, ' '.join(argv[:2]))
|
||||
process = subprocess.Popen(argv, cwd=ROOT, env=env, stdin=subprocess.DEVNULL, start_new_session=True)
|
||||
try:
|
||||
code = process.wait(timeout=timeout)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise QAError(f'{stage} 超过 {timeout} 秒') from exc
|
||||
finally:
|
||||
stop_process(process)
|
||||
if code:
|
||||
raise QAError(f'{stage} 失败(退出 {code}),查看上方命令输出')
|
||||
|
||||
|
||||
def stop_process(process):
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
try:
|
||||
process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
process.wait(timeout=5)
|
||||
|
||||
|
||||
def start_service(stack, argv, env, log):
|
||||
output = stack.enter_context(log.open('w', encoding='utf-8'))
|
||||
process = subprocess.Popen(argv, cwd=ROOT, env=env, stdin=subprocess.DEVNULL,
|
||||
stdout=output, stderr=subprocess.STDOUT, start_new_session=True)
|
||||
stack.callback(stop_process, process)
|
||||
return process
|
||||
|
||||
|
||||
def request(port, path):
|
||||
connection = http.client.HTTPConnection('127.0.0.1', port, timeout=2)
|
||||
try:
|
||||
connection.request('GET', path)
|
||||
response = connection.getresponse()
|
||||
return response.status, response.getheader('Content-Type', ''), response.read(32768)
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
|
||||
def probe(port, path, expected):
|
||||
try:
|
||||
code, content_type, body = request(port, path)
|
||||
if code != 200:
|
||||
return False
|
||||
if expected == 'html':
|
||||
return 'text/html' in content_type and b'id="root"' in body
|
||||
return 'application/json' in content_type and json.loads(body)['status'] == expected
|
||||
except (OSError, http.client.HTTPException, ValueError, KeyError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
def wait_service(process, port, path, expected, log):
|
||||
deadline = time.monotonic() + 45
|
||||
while time.monotonic() < deadline:
|
||||
if process.poll() is not None:
|
||||
raise QAError(f'服务端口 {port} 启动进程提前退出(退出 {process.returncode}),日志 {log}')
|
||||
if probe(port, path, expected):
|
||||
return
|
||||
time.sleep(0.25)
|
||||
raise QAError(f'服务探针超时 127.0.0.1:{port}{path},日志 {log}')
|
||||
|
||||
|
||||
def qa_run(commit, env, db_env, versions, state):
|
||||
with tempfile.TemporaryDirectory(prefix='run-', dir=state) as directory, ExitStack() as services:
|
||||
binary = Path(directory) / 'api'
|
||||
run_step(['make', 'deps-install'], 'deps', env, 300)
|
||||
ldflags = f'-X knowledge-governance/internal/version.CommitSHA={commit}'
|
||||
run_step(['go', 'build', '-ldflags', ldflags, '-o', str(binary), './cmd/api'], 'build-api', env, 300)
|
||||
check_ports()
|
||||
log_dir = Path(tempfile.mkdtemp(prefix=f'{commit[:12]}-', dir=state))
|
||||
report('logs', str(log_dir))
|
||||
app_env = db_env | {'PATH': env['PATH'], 'GOVERNANCE_SPOOL_ROOT': str(Path(directory) / 'spool')}
|
||||
api_log = log_dir / 'api.log'
|
||||
api = start_service(services, [str(binary)], app_env, api_log)
|
||||
wait_service(api, API_PORT, '/readyz', 'ready', api_log)
|
||||
check_migrations(versions, *database_state(db_env), complete=True)
|
||||
report('migration', f'已核对全部 {len(versions)} 条迁移,最新版本 {versions[-1]}')
|
||||
|
||||
web_log = log_dir / 'web.log'
|
||||
web = start_service(services, ['npm', '--prefix', 'web', 'run', 'dev', '--', '--host', '127.0.0.1',
|
||||
'--port', str(WEB_PORT), '--strictPort'], env, web_log)
|
||||
wait_service(web, WEB_PORT, '/', 'html', web_log)
|
||||
wait_service(web, WEB_PORT, '/health', 'ok', web_log)
|
||||
report('services', 'API /readyz、Web 首页与 Vite /health 代理通过')
|
||||
run_step(['make', 'tests-go', 'tests-web'], 'tests', env, 300)
|
||||
check_checkout(commit)
|
||||
if api.poll() is not None or web.poll() is not None or not probe(WEB_PORT, '/health', 'ok'):
|
||||
raise QAError(f'服务在测试期间失去健康状态,日志目录 {log_dir}')
|
||||
report('result', f'PASS {commit},已停止本次 API 和 Web 进程')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=('doctor', 'run'))
|
||||
args = parser.parse_args()
|
||||
env = tool_env()
|
||||
try:
|
||||
commit = os.environ.get('KG_QA_SHA', '')
|
||||
if args.action == 'run':
|
||||
check_checkout(commit)
|
||||
check_ports()
|
||||
preflight(env)
|
||||
config_path = Path(os.environ.get('KG_QA_CONFIG', CONFIG)).expanduser().absolute()
|
||||
config = database_config(config_path)
|
||||
private_file(config_path.parent / 'qa.pgpass')
|
||||
if not (config_path.parent / 'rds-ca.pem').is_file():
|
||||
raise QAError(f'缺少 RDS CA 文件 {config_path.parent / "rds-ca.pem"}')
|
||||
db_env = database_env(config, config_path.parent)
|
||||
versions = expected_migrations()
|
||||
if args.action == 'doctor':
|
||||
applied, nonempty = database_state(db_env)
|
||||
check_migrations(versions, applied, nonempty, complete=False)
|
||||
report('database', f'连接与身份正常,已应用 {len(applied)}/{len(versions)} 条迁移')
|
||||
return 0
|
||||
|
||||
state = Path.home() / '.local/state/knowledge-governance/qa'
|
||||
state.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
with (state / 'run.lock').open('a') as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError as exc:
|
||||
raise QAError('本机已有 QA 任务,等待当前任务退出') from exc
|
||||
check_ports()
|
||||
check_migrations(versions, *database_state(db_env), complete=False)
|
||||
qa_run(commit, env, db_env, versions, state)
|
||||
return 0
|
||||
except (QAError, OSError, ValueError) as exc:
|
||||
print(f'[qa:FAIL] {exc}', file=sys.stderr)
|
||||
return 1
|
||||
except KeyboardInterrupt:
|
||||
print('[qa:FAIL] QA 任务中断', file=sys.stderr)
|
||||
return 130
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
def interrupt(_signal, _frame):
|
||||
raise KeyboardInterrupt
|
||||
|
||||
signal.signal(signal.SIGTERM, interrupt)
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,157 @@
|
||||
import hashlib
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import Mock
|
||||
|
||||
import production_cd as cd
|
||||
|
||||
|
||||
SHA = 'a' * 40
|
||||
TASK = 'd5abdf29-d0b5-4992-8073-0cb8fffd84ed'
|
||||
IMAGES = {role: 'sha256:' + str(i) * 64 for i, role in enumerate(('api', 'web', 'worker'), 1)}
|
||||
CTX = 'build / build (push)'
|
||||
|
||||
|
||||
class ProductionCD(unittest.TestCase):
|
||||
def setUp(self):
|
||||
temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(temp.cleanup)
|
||||
self.root = Path(temp.name)
|
||||
self.request = dict(sha=SHA, task=TASK, bundle='b' * 64, confirm='deploy-production',
|
||||
action='deploy', run_id='42')
|
||||
self.manifest = dict(schema=1, target=cd.TARGET, sha=SHA, task=TASK, migrations=[1, 2], images={})
|
||||
for role, image in IMAGES.items():
|
||||
archive = self.root / (role + '.tar')
|
||||
archive.write_text(role)
|
||||
self.manifest['images'][role] = dict(id=image, sha256=hashlib.sha256(role.encode()).hexdigest())
|
||||
data = json.dumps(self.manifest).encode()
|
||||
(self.root / 'release.json').write_bytes(data)
|
||||
for path in self.root.iterdir():
|
||||
path.chmod(0o600)
|
||||
self.request['bundle'] = hashlib.sha256(data).hexdigest()
|
||||
self.acceptance = {'verdict': 'passed', 'sha': SHA, 'bundle': self.request['bundle'],
|
||||
'evidence': 'https://example.test/result/42'}
|
||||
self.host = Mock()
|
||||
self.host.task.return_value = {'id': TASK, 'spaceId': cd.SPACE, 'status': {'id': cd.READY},
|
||||
'metadata': {'cd_acceptance': json.dumps(self.acceptance)}}
|
||||
self.host.baseline.return_value = {'before': 'baseline'}
|
||||
self.host.pre_switch.return_value = None
|
||||
self.host.verify.return_value = {'sha': SHA, 'images': IMAGES, 'health': 'ok'}
|
||||
self.host.receipt_path = self.root / 'receipt.json'
|
||||
self.host.bundle_path.return_value = self.root
|
||||
|
||||
def execute(self):
|
||||
return cd.release(self.host, self.request)
|
||||
|
||||
def assert_no_deploy(self):
|
||||
self.host.load.assert_not_called()
|
||||
self.host.switch.assert_not_called()
|
||||
self.host.callback.assert_not_called()
|
||||
|
||||
def test_no_confirmation_has_zero_side_effects(self):
|
||||
self.request['confirm'] = ''
|
||||
with self.assertRaisesRegex(cd.CDError, 'confirmation'):
|
||||
self.execute()
|
||||
self.host.assert_not_called()
|
||||
self.assert_no_deploy()
|
||||
|
||||
def test_preconditions_prevent_loading_and_switching(self):
|
||||
for mode in ('approval', 'ci', 'test', 'changed_bundle', 'tampered_archive', 'migration'):
|
||||
with self.subTest(mode=mode):
|
||||
self.setUp()
|
||||
if mode in ('approval', 'ci', 'migration'):
|
||||
step = {'approval': self.host.approve, 'ci': self.host.check_ci,
|
||||
'migration': self.host.pre_switch}[mode]
|
||||
step.side_effect = cd.CDError(mode)
|
||||
elif mode == 'test':
|
||||
self.host.task.return_value['metadata'].clear()
|
||||
elif mode == 'changed_bundle':
|
||||
self.request['bundle'] = 'f' * 64
|
||||
else:
|
||||
(self.root / 'web.tar').write_bytes(b'other image')
|
||||
with self.assertRaises(cd.CDError):
|
||||
self.execute()
|
||||
self.assert_no_deploy()
|
||||
|
||||
def test_bare_ci_success_cannot_replace_required_context(self):
|
||||
for statuses in ([], [{'context': 'different', 'status': 'success'}],
|
||||
[{'context': CTX, 'status': 'failure'}, {'context': CTX, 'status': 'success'}]):
|
||||
with self.assertRaises(cd.CDError):
|
||||
cd.verify_contexts({'state': 'success', 'statuses': statuses}, [CTX])
|
||||
cd.verify_contexts({'state': 'success', 'statuses': [{'context': CTX, 'status': 'success'}]}, [CTX])
|
||||
|
||||
def test_approval_and_acceptance_rechecked_before_switch(self):
|
||||
self.host.approve.side_effect = [None, cd.CDError('revoked')]
|
||||
with self.assertRaisesRegex(cd.CDError, 'revoked'):
|
||||
self.execute()
|
||||
self.host.switch.assert_not_called()
|
||||
self.host.callback.assert_not_called()
|
||||
|
||||
def test_success_and_writeback_retry_do_not_switch_twice(self):
|
||||
self.host.callback.side_effect = [cd.CDError('HiFox timeout'), None]
|
||||
with self.assertRaisesRegex(cd.CDError, 'DEPLOYED_HIFOX_PENDING'):
|
||||
self.execute()
|
||||
self.assertEqual(cd.read_json(self.host.receipt_path)['state'], 'deployed_hifox_pending')
|
||||
self.request.update(action='writeback', confirm='retry-writeback')
|
||||
self.assertEqual(self.execute()['state'], 'published')
|
||||
self.host.switch.assert_called_once()
|
||||
self.host.load.assert_called_once()
|
||||
self.assertEqual(self.host.verify.call_count, 2)
|
||||
|
||||
def test_writeback_without_receipt_refuses(self):
|
||||
self.request.update(action='writeback', confirm='retry-writeback')
|
||||
with self.assertRaisesRegex(cd.CDError, 'receipt'):
|
||||
self.execute()
|
||||
self.assert_no_deploy()
|
||||
|
||||
def test_failed_cutover_rolls_back_and_never_marks_published(self):
|
||||
for result in ('rolled_back', 'rollback_failed'):
|
||||
with self.subTest(result=result):
|
||||
self.setUp()
|
||||
self.host.verify.side_effect = cd.CDError('new unhealthy')
|
||||
if result == 'rollback_failed':
|
||||
self.host.rollback.side_effect = cd.CDError('old unhealthy')
|
||||
with self.assertRaisesRegex(cd.CDError, result):
|
||||
self.execute()
|
||||
self.assertEqual(cd.read_json(self.host.receipt_path)['state'], result)
|
||||
self.host.rollback.assert_called_once()
|
||||
self.host.callback.assert_not_called()
|
||||
with self.assertRaisesRegex(cd.CDError, 'receipt'):
|
||||
self.execute()
|
||||
self.host.switch.assert_called_once()
|
||||
|
||||
def test_gitea_guard_log_binds_the_exact_dispatch(self):
|
||||
trusted = 'f' * 40
|
||||
proof = cd.guard_proof(self.request, trusted)
|
||||
log = f'2026-11-22T17:31:54.013Z {proof}\n'.encode()
|
||||
cd.verify_guard_log(log, self.request, trusted)
|
||||
for name, altered in (('sha', 'b' * 40), ('bundle', 'c' * 64), ('run_id', '43'),
|
||||
('task', 'd5abdf29-d0b5-4992-8073-0cb8fffd84ee2'), ('confirm', 'retry-writeback')):
|
||||
with self.subTest(name=name), self.assertRaisesRegex(cd.CDError, 'dispatch proof'):
|
||||
cd.verify_guard_log(log, {**self.request, name: altered}, trusted)
|
||||
with self.assertRaisesRegex(cd.CDError, 'dispatch proof'):
|
||||
cd.verify_guard_log(log + log, self.request, trusted)
|
||||
|
||||
def test_target_lock_is_exclusive(self):
|
||||
path = self.root / 'lock'
|
||||
with cd.deploy_lock(path):
|
||||
with self.assertRaisesRegex(cd.CDError, 'deployment lock'):
|
||||
with cd.deploy_lock(path):
|
||||
self.fail('lock was acquired twice')
|
||||
|
||||
def test_persisted_compose_changes_only_images(self):
|
||||
original = {'services': {role: {'image': role + ':old', 'env_file': [{'path': '/kept', 'format': 'raw'}],
|
||||
'volumes': ['/kept:/run/key:ro'], 'environment': {'GATE': 'true'}}
|
||||
for role in IMAGES}, 'name': 'governance'}
|
||||
changed = cd.with_images(original, IMAGES)
|
||||
for role, new in changed['services'].items():
|
||||
self.assertEqual(new.pop('image'), IMAGES[role])
|
||||
old = original['services'][role].copy()
|
||||
old.pop('image')
|
||||
self.assertEqual(new, old)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,99 @@
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
import json
|
||||
import os
|
||||
from threading import Thread
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import qa_env
|
||||
|
||||
|
||||
class QAEnvironmentTests(unittest.TestCase):
|
||||
def test_missing_approval_config_names_file(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
config = Path(directory) / 'qa.json'
|
||||
with self.assertRaisesRegex(qa_env.QAError, 'qa.json'):
|
||||
qa_env.database_config(config)
|
||||
|
||||
def test_rejects_non_qa_database_before_connection(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
config = Path(directory) / 'qa.json'
|
||||
config.write_text(json.dumps(dict(host='rds.example.test', port=5432,
|
||||
database='governance', user='kg_qa_app')))
|
||||
config.chmod(0o600)
|
||||
with self.assertRaisesRegex(qa_env.QAError, 'kg_qa_'):
|
||||
qa_env.database_config(config)
|
||||
|
||||
def test_database_environment_uses_only_approved_identity(self):
|
||||
settings = dict(host='rds.example.test', port=5432, database='kg_qa_app', user='kg_qa_app')
|
||||
with patch.dict(os.environ, {'PGHOSTADDR': '127.0.0.1', 'PGPASSWORD': 'fixture',
|
||||
'PGSERVICE': 'other', 'PRODUCTION_RELEASE_REQUESTS_ENABLED': 'true'}):
|
||||
env = qa_env.database_env(settings, Path('/approved'))
|
||||
self.assertEqual(env['PGHOST'], 'rds.example.test')
|
||||
self.assertEqual(env['PGSSLMODE'], 'verify-full')
|
||||
self.assertEqual(env['PRODUCTION_RELEASE_REQUESTS_ENABLED'], 'false')
|
||||
self.assertFalse({'PGHOSTADDR', 'PGPASSWORD', 'PGSERVICE'} & env.keys())
|
||||
|
||||
def test_empty_database_or_known_migration_prefix_can_bootstrap(self):
|
||||
for applied in ([], [1], [1, 2]):
|
||||
with self.subTest(applied=applied):
|
||||
qa_env.check_migrations([1, 2], applied, False, complete=False)
|
||||
|
||||
def test_pending_migrations_block_testing(self):
|
||||
with self.assertRaisesRegex(qa_env.QAError, '迁移未完成'):
|
||||
qa_env.check_migrations([1, 2], [1], False, complete=True)
|
||||
|
||||
def test_schema_without_ledger_is_rejected(self):
|
||||
with self.assertRaisesRegex(qa_env.QAError, '非空'):
|
||||
qa_env.check_migrations([1, 2], [], True, complete=False)
|
||||
|
||||
def test_unknown_or_missing_middle_migrations_are_rejected(self):
|
||||
for applied in ([1, 3], [1, 2, 3], [2]):
|
||||
with self.subTest(applied=applied), self.assertRaisesRegex(qa_env.QAError, '迁移序列'):
|
||||
qa_env.check_migrations([1, 2], applied, False, complete=False)
|
||||
|
||||
def test_hifox_exit_zero_does_not_override_offline_state(self):
|
||||
with self.assertRaisesRegex(qa_env.QAError, '在线'):
|
||||
qa_env.check_hifox('{"connected":true,"remoteLiveness":{"status":"offline"}}')
|
||||
qa_env.check_hifox('{"connected":true,"remoteLiveness":{"status":"online"}}')
|
||||
|
||||
def test_health_probe_rejects_spa_fallback(self):
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'text/html')
|
||||
self.end_headers()
|
||||
self.wfile.write(b'<html><div id="root"></div></html>')
|
||||
|
||||
def log_message(self, format, *args):
|
||||
pass
|
||||
|
||||
with ThreadingHTTPServer(('127.0.0.1', 0), Handler) as server:
|
||||
thread = Thread(target=server.serve_forever)
|
||||
thread.start()
|
||||
try:
|
||||
self.assertTrue(qa_env.probe(server.server_port, '/', 'html'))
|
||||
self.assertFalse(qa_env.probe(server.server_port, '/health', 'ok'))
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join()
|
||||
|
||||
def test_database_identity_mismatch_stops_before_reading_migrations(self):
|
||||
env = {'PGDATABASE': 'kg_qa_app', 'PGUSER': 'kg_qa_app'}
|
||||
with patch.object(qa_env, 'psql', return_value='other|kg_qa_app|t|t|t|t|t') as probe:
|
||||
with self.assertRaisesRegex(qa_env.QAError, '实际库名'):
|
||||
qa_env.database_state(env)
|
||||
self.assertEqual(probe.call_count, 1)
|
||||
|
||||
def test_full_sha_mismatch_rejects_current_checkout(self):
|
||||
full_sha = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=qa_env.ROOT, text=True).strip()
|
||||
wrong = ('0' if full_sha[0] != '0' else '1') + full_sha[1:]
|
||||
with self.assertRaisesRegex(qa_env.QAError, '提交不一致'):
|
||||
qa_env.check_checkout(wrong)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user