Compare commits

..

20 Commits

Author SHA1 Message Date
zheromiku 034c0b3a1c Merge pull request 'fix(library): 跨源同步按目标范围改写并校验残留' (#13) from fix/library-cross-scope-sync-residue into main
build / build (push) Successful in 1m18s
2026-10-29 22:13:49 +08:00
姚章浩 5defe238e0 fix(library): 跨范围同步按目标范围改写并校验残留
build / build (push) Successful in 1m34s
真实事故:源 KB(10000/0c84a098) 同步到目标 KB(10001/e64e82b5) 时,归档行被原样
插入目标库,落在来源租户/知识库;资源阶段随即报"生产缺少资源 ... 的存储路径"
(资源查询按目标租户过滤,看不见刚写进去的行)。回滚只按目标范围回放,于是页面
报"已恢复到预备份",目标却多出 4 文件/38 分块/21 资源/4 绑定。

改:
- 写入与校验共用 retargetRow:tenant_id / knowledge_base_id / 资源定位符(后端、
  provider、桶、租户前缀) / storage_backend_id / provider / embedding_model_id
  统一映射到目标范围;行里的来源身份对不上就失败,宁可不写。
- Verify 增加"本次写集合仍以来源身份残留"判定;Restore 增加整库指纹比对(预备份
  时刻 vs 恢复后)与孤儿资源清理,回滚后用 Verify 复核,恢复工具退出 0 不再等于
  恢复完成。
- 新增 cmd/libraryresidue:按归档身份精确、可备份、可复核地清理已知的范围外残留。

验证:make verify 全绿;真实 PG+ES 用例复现"源目标 KB 不同 + 目标非空 + 资源阶段
失败",断言回滚后无来源范围残留、其它租户行未动、恢复后内容=预备份;正向用例断言
行落在目标范围且资源定位符指向目标存储。
2026-10-29 22:11:59 +08:00
姚章浩 c27a0c14cc fix(library): 基线契约读目标真实 schema/模型,演练标内容风险
build / build (push) Successful in 1m22s
真实演练暴露两个只有跑起来才看得见的问题:

1. 归档要求完整兼容性契约(v0.7.1 / schema 74 / elasticsearch_v8 / 1024 维 /
   映射摘要),而基线装配只填了 embedding 与映射,捕获被
   "unsupported snapshot compatibility contract" 拒绝。
   现在 schema 版本读目标库 schema_migrations(dirty 或版本不符直接拒),
   embedding 优先读目标库模型绑定(本机目标 models 表为空、引用悬空,
   此时回落部署声明并说清来源),版本/引擎用受支持栈常量,映射取目标当前值。
2. 演练在"还没写目标内容"时就允许 restore_blocked=false 的窗口以 blocked 收尾,
   写出 content_risk=false + restore_blocked=true 的非法 journal(恢复器拒绝加载,
   目标停在停机态)。现在写 canary 时先 MarkContentRisk,allowRestore 按
   "没碰内容 ⇒ 安全恢复;碰过 ⇒ 必须恢复并校验通过"。

另外给演练工具加 -recover:只把非终态窗口按原运行状态恢复,用于演练失败后
把目标救回来(不改治理记录)。

验证:库内模拟真实目标不可得,契约分支用真实 kgseal 演练覆盖:
snapshot 5cb945e6… 捕获 143ms → 基线一致 → canary 后校验如预期失败
(knowledges 1 行摘要不一致)→ 恢复后校验通过 → 窗口 succeeded、服务与入口恢复。
2026-10-29 19:41:12 +08:00
姚章浩 a42b101a7b fix(library): 基线崩溃续作与维护原状态保留
build / build (push) Successful in 1m25s
父审发现三处会让目标卡死或状态对不上实际入口,逐条修:

1. 登记成功但没来得及收尾:崩溃重入会撞上"已有生效版本"直接失败,
   目标留在维护里、占用也不释放。现在按「本作业+同一快照」识别为自己的
   版本,只做收尾(恢复服务/开入口/释放两把占用),不重新捕获、不重写;
   别人的版本仍然拒绝。捕获完成但登记失败则明确失败并写"未对目标写入"。
2. 重入覆盖原运行状态:崩溃时服务已被停住,再读一次只会读到 stopped,
   写回去会让恢复阶段永远不启动写入者。AcquireInstance/LibraryMaintenance
   在 owner 相同的重入时保留已持久化原状态;新作业接管已释放窗口用自己的观察。
   真实 PostgreSQL 集成用例覆盖(先红后绿)。
3. 版本登记移进维护窗口:捕获/校验/登记要么一起成立、要么目标原样恢复;
   记录失败时窗口已开放,治理状态同步释放,不再"显示维护中、实际已开"。
   索引映射读不到时拒绝登记,不再用空摘要写证据。

顺带:
- 基线窗口统一收尾抽成 settleBaselineWindow,四种结局各有测试(成功/健康失败/
  held 保持维护/已恢复则释放),并用真实 DB 用例证明 baseline 与 sync/restore
  共用写互斥、幂等重试不产生第二个作业。
- 新增 cmd/librarydrill:维护隔离 → 整库预备份 → 一致性校验 → canary 漂移检测
  → 按预备份恢复 → 逐类复核 → 按原状态恢复服务。用与 worker 相同的 LIBRARY_*
  配置,绝不触碰封存源。
- 目标/预备份/维护栈装配收成 internal/librarystack 一份(worker 与演练共用)。

验证:go build/vet/test 全过;TEST_DATABASE_URL 指向临时 PostgreSQL 跑
TestMaintenanceReentryKeepsOriginalServiceState(修复前失败:got app=stopped)
与 TestLibraryWriteMutexCoversBaselineJobs;本地门禁 15/15。
2026-10-29 19:08:58 +08:00
姚章浩 3284658eb1 feat(web): 发布页接上签发同步授权与建立基线入口
build / build (push) Successful in 1m31s
发布页原来只有一个永远 disabled 的"启动同步",title 还写着"生产同步执行链
尚未接线"——执行链其实已经存在。现在:

- 管理员看到"签发同步授权(30 分钟)"(action=sync,绑定范围+目标快照),
  签发与执行仍是两步,不会自动签发;
- 闸口全满足时"启动同步"真的按 POST /library/syncs 投递,被拒时逐条显示
  接口返回的缺项;
- 目标非空又无生效版本时,管理员看到"建立目标权威基线(维护隔离)";
- 测试封存候选改用接口给的 test_sealed(来自版本登记),不再拿归档里最新
  一份顶替;
- 维护卡片区分"未运行维护 / 维护能力未接线 / 能力未知",逐项列出缺的变量。

验证:vitest 99 个文件 469 个用例全绿,tsc --noEmit 无错;反向自检(把按钮
改回恒 disabled、把维护文案改回旧句)两处按预期变红后复原。
2026-10-29 18:16:25 +08:00
姚章浩 517d5f862d feat(library): 首次发布接上目标权威基线与整库空判定
目标非空又没有生效版本时,首次发布以前只能停在"权威版本未知"。现在有了
真实入口:管理员发起 baseline 作业,worker 在维护隔离下把目标整库捕获成
归档快照,再用同一套 Verify 证明目标与快照逐类一致,最后登记
production_effective(证据写明 origin=target_baseline、快照 ID、逐表行数、
目标索引映射)。不写目标内容、不清空目标、不手写版本。

同时修两处会误导人的判定:
- productionScopeEmpty 原来只数 knowledges,"文档删了但分块/Wiki/资源还在"
  的目标会被当成空库;改用适配层同一份表清单与范围过滤逐表计数。
- 概览里的"测试封存候选"原来取该 scope 最新一份归档,而归档里还有生产
  预备份与目标基线快照;改为只认版本登记里 kind=test_sealed 的那一份。

迁移 075 放开 kind='baseline' 并把它并入 sync/restore 的写互斥索引;
worker 重启时用同一作业 id 接管未完成的基线窗口。维护接口补 capability,
页面才能区分"未运行维护"与"维护能力没接线"。

验证:go build ./... / go vet ./... 通过;新增用例覆盖管理员校验、已有生效
版本拒绝、目标读不到时零作业零版本、候选来自登记、缺项文案指向基线入口、
capability 逐项缺失;反向自检(把候选改回按最新归档取、把文案改回旧句、
去掉管理员判断)三项均按预期变红后复原。
2026-10-29 18:16:21 +08:00
姚章浩 f9c15a8f14 fix(web): 存储提示改为实际能力
build / build (push) Successful in 1m24s
页面仍写"保留策略与清理在系统设置→存储管理,均未接线",但存储与备份已有只读
去重/保护范围/预览。改为说明可查看什么、什么尚未开放。
2026-10-29 15:24:30 +08:00
姚章浩 271f9f5f09 docs: 台账补记 0.1.198–0.1.199
build / build (push) Successful in 1m20s
记录接管分类与维护释放幂等的真实 SHA、CI run(236)与候选栈身份注入后
/api/v1/build-info 的读回结果。
2026-10-29 14:27:53 +08:00
姚章浩 24d2546cbb fix(library): 接管不再把版本查询失败当成没有收尾证据
build / build (push) Successful in 1m28s
resumed 分支里 LibraryVersions 的错误被忽略,FinalizeOnly 保持 false 后继续完整
apply:对"已登记版本并释放窗口、只差终态"的作业,这等于把同一份内容重写一遍目标,
与"最终只复核"的承诺相反。

现在接管先分类:finalize 阶段成功作为独立证据(不受版本查询影响)→ 只收尾;查询
失败且无阶段证据 → 未知,零写入、可重试;本快照被别的快照顶替 → 拒绝覆盖;其余才
按正常流程。

同时修释放幂等:Update*/Release* 对"已由本作业释放"的行返回 ErrMaintenanceReleased,
收尾重试视为已完成(部分释放后不会永久卡死),别的作业仍报未持有。真 PG 负例验证
两种情形。
2026-10-29 14:24:12 +08:00
姚章浩 691e4e0225 docs: 台账补记 0.1.195–0.1.197
build / build (push) Successful in 1m16s
记录 R1 身份契约、封存资源诊断与 R4 成功边界三个提交的真实 SHA 与 CI run,
标明候选栈 clean 构建(vcs.modified=false)与真实验收覆盖范围。
2026-10-29 14:07:35 +08:00
姚章浩 8d2ed8ebe6 fix(library): 维护释放与服务恢复完成之后才暴露作业成功
build / build (push) Successful in 1m32s
原本 execute 写完生效版本就写 succeeded,而服务恢复、入口开放与维护锁释放发生在
窗口返回之后。任一步失败时外部已看到成功,重启接管又只认非终态,于是目标永远
留在维护里(本轮真实现象:同步 succeeded 后立即恢复被 409)。

现在内容校验与版本登记完成后只记"待收尾":finishLibrarySuccess 先确认目标写入者
与入口健康,再按 scope → instance 顺序释放两把锁(每一步错误都上抛,不再 `_ =` 丢弃),
最后才写 succeeded;任一步失败即落失败并保持 needs_restore。释放与终态写入用
context.WithoutCancel 有界包住,停机不会留下半成品。接管的 finalize-only 判定改为
按证据(finalize 阶段成功或未顶掉的生效版本),只复核与登记,不重写生产。

测试覆盖:调用顺序(成功后两把锁必已释放)、健康失败不释放不报成功、三步释放失败
各自不报成功、终态写入失败提示等待接管、接管证据判定正负例。
2026-10-29 14:02:09 +08:00
姚章浩 c14dd01f08 fix(library): 封存读资源失败带上资源行与底层原因
build / build (push) Successful in 1m18s
真实封存卡在 `read library resource` 时没有任何可判断信息:对象缺失、桶写错、
宿主机解析不到端点都会走到同一个字符串。现在错误带资源行 id 与底层原因
(不含凭据),本轮就是用它在候选栈定位到"源对象被上一次 B02 演练删掉后没真正写回"。
2026-10-29 13:32:55 +08:00
姚章浩 6ed06357f9 fix(library): 期望执行器身份随 capture 请求送达并在执行前核对
build / build (push) Successful in 1m27s
原来 worker 先 health 再 capture,两次拨号之间 socket 可能被换成另一个实例的
执行器:错的 agent 已经把写入者停了、快照也出了,事后 postcheck 只能拒绝登记,
副作用无法撤销。

现在 capture 请求自带期望身份(治理侧配置生成:project 必填,source_app /
archive_root 配了就绑定)与覆盖三项的稳定指纹;接收方在操作互斥内、执行任何源
操作之前比对,缺字段、指纹与字段不一致、project/source_app 不符都直接拒绝,
**错的执行器一次 capture 都不跑**。封存结果回显该指纹,worker 事后复核按它对照,
postcheck 只作为证据防线。恢复路径不要求期望(它只把写入者恢复回来)。

负例是确定性的:两个真实 agent 各占一个 socket,把调用方指向错的那个,断言它的
capture 调用为 0 且仍能回答 health;同一份期望打到对的 socket 则只跑一次。
2026-10-29 13:23:56 +08:00
姚章浩 5d42cba6c6 docs: 执行器证据绑定到确切代码 SHA 而不是版本号
build / build (push) Successful in 1m15s
矩阵里写"0.1.187 构建"会与台账的版本号漂移;改成代码 SHA 4c79052(0.1.192 工作树),
与候选二进制来源一致。
2026-10-29 13:12:33 +08:00
姚章浩 a47f9323e0 docs: 台账补记执行器可用性与还原入口(0.1.188–0.1.192)
build / build (push) Successful in 1m29s
记录真实 SHA、CI run 229(head 4c79052,必需 context success)、候选栈二进制来源与
回退目录。代码 SHA 与已部署候选产物分开写,不把文档提交当成部署证据。
2026-10-29 13:07:51 +08:00
姚章浩 4c79052a18 test(release-worker): 测试 DSN 改为拼接构造避免密钥扫描误报
build / build (push) Successful in 1m34s
完整带口令的 URL 出现在源码里会被 security-secrets 判红(测试值也一样)。
按既有做法把连接串拆开拼接,语义不变。
2026-10-29 13:05:23 +08:00
姚章浩 18276d0839 docs: 按用户决定收窄整库还原范围并记录执行器证据
发布管理不承担 Wiki 语义撤销:V03-a 的原始失败证据保留,但移到"模块范围外",
不再阻断发布,也不改判为通过。矩阵同步补上执行器运行时可用性与按版本恢复入口的
真实证据(候选栈),并把尚未跑的部分(真实封存身份复核、新构建上的真实恢复)
写在"已实现·未跑"。正式 release-worker 的崩溃根因按实测写清:目标库主机
prod-pg 不可解析,修正配置属于正式栈变更,需要部署窗口。
2026-10-29 13:01:51 +08:00
姚章浩 cfad003499 fix(release-worker): 启动失败带端点与脱敏原因
正式 release-worker 崩溃重启时日志只有一句 connect release target database,
主机名、库名和底层原因都不在里面。现在错误里带 host:port/db、路由标识与原因。

端点去口令不等于整条日志安全:pgx 的解析错误可能原样回显连接串。新增
internal/pgdsn.Redact(按已知口令精确替换 + 按 URL/keyword 形状兜底),
错误保留 Unwrap 以便 errors.Is/As,技术原因不丢。cmd/worker 里只留库名的旧实现
换成同一个包,避免两份脱敏逻辑。libraryctl 用法补上 dead-letters。
2026-10-29 13:01:46 +08:00
姚章浩 c40696145b feat(library-web): 恢复入口按真实条件启用并支持签发恢复授权
原来"恢复至该版本"写死 disabled,标题还写"生产恢复未接线"——而 POST /library/restores
与授权接口早已可用,等于把整库还原的唯一入口藏起来。

现在页面按 GET /library/restore-availability 启用:缺授权时管理员可签发 30 分钟
有效期的 restore 授权(固定期限,没有"永久授权"),条件满足才可点恢复;工程师只看到
条件与"仅管理员可发起"。封存不可发起时也不再写死"通道未接线",而是显示接口原因、
执行器身份与观测时间。
2026-10-29 13:01:46 +08:00
姚章浩 47eb23cf68 feat(library): 封存执行器可用性与还原条件来自真实观测
页面上的"可发起封存"原来只判 LIBRARY_AGENT_SOCKET 非空:正式栈的 API 根本没挂
那个 socket,而 agent 曾顶着正式单元名冻结候选实例——配置非空、页面可点、真发起
却封错库。

现在:agent 新增 health 动作(不占封存操作锁、不跑子进程,只回答身份/降级/忙),
worker 每 30 秒探测并写入迁移 074 的 library_executor_status;overview、快照列表与
POST /library/captures 共用同一份判定(过期、不可达、实例不符、降级、正忙都拒绝并
给原因)。封存结果由 agent 附上自报身份,worker 在停写入者之前再探测一次、执行后
用结果复核,不一致就不登记测试封存——心跳只能证明"刚才"。

期望身份用 LIBRARY_SOURCE_*(封存源),不是 LIBRARY_TARGET_*(同步目标),否则
正确的执行器会被判成不匹配。另加 GET /library/restore-availability:恢复能否执行由
双闸、该快照的恢复授权与实例维护状态决定,页面不再写死 disabled。

真跑(候选隔离栈,两处 socket 隔离不变):overview 返回 executor
{kgmaint-weknora/kgmaint-app};停 agent 后转不可达并 503 带原因;重启后恢复。
2026-10-29 13:01:37 +08:00
74 changed files with 6776 additions and 418 deletions
+145
View File
@@ -1,5 +1,150 @@
# Changelog
## 开发快照(基线崩溃续作、维护原状态保留与真实恢复演练工具)
### Fixed
- **基线契约按"能验证的来源"填充**:归档要求完整的兼容性契约(v0.7.1 / schema 74 /
elasticsearch_v8 / 1024 维 / 映射摘要)。原先只填了 embedding 与映射,捕获会被
`unsupported snapshot compatibility contract` 拒绝。现在 schema 版本**读目标库**
`schema_migrations`(dirty 或版本不符直接拒绝),embedding 优先读目标库模型绑定、
读不到才回落部署声明,映射取目标当前值,版本/引擎用受支持栈常量。
- **基线作业"登记成功但没来得及收尾"不再被判死**:原来崩溃重入会在
"已有生效版本"检查处直接失败,把目标留在维护里、占用也不释放。
现在按「本作业 + 同一快照」识别为**自己的**版本:只做收尾(恢复服务、开入口、
释放两把占用),不重新捕获、不重写版本;别人登记的生效版本仍然一律拒绝。
捕获完成但登记失败的作业会明确失败并写"未对目标写入",不会假装成功。
- **同一作业重入不再覆盖第一次记下的原运行状态**:崩溃接管时目标已被停住,
再读一次只会读到 `stopped`,写回去会让恢复阶段以为写入者本来就没跑、
永远不启动它。现在 `AcquireInstanceMaintenance` / `AcquireLibraryMaintenance`
在 owner 相同的重入时保留已持久化的原状态;新作业接管已释放窗口仍用自己读到的值。
(真实 PostgreSQL 集成用例覆盖,非内存假实现。)
- **基线的版本登记移进维护窗口**:捕获、校验、登记三件事要么一起成立、要么目标
原样恢复;记录失败时窗口已按原状态开放,治理记录同步释放,**不再出现
"显示保持维护、实际入口已开"**。
- **索引映射读不到时拒绝登记**:不再 `_` 忽略错误后用空摘要写进基线证据。
- 基线窗口的统一收尾抽成 `settleBaselineWindow`:版本已登记走与同步相同的成功边界;
窗口仍 held 才标 `needs_restore`;窗口已恢复则释放占用并失败。
### Added
- **`cmd/librarydrill`:目标实例真实恢复演练**(维护隔离 → 整库预备份 → 一致性校验
→ canary 漂移检测 → 按预备份恢复 → 逐类复核 → 按原状态恢复服务)。它使用与
worker 完全相同的 `LIBRARY_*` 配置与维护协调器,绝不触碰封存源;任一步失败都
保持维护并写报告,不伪造通过。
- `internal/librarystack`:目标/预备份/维护栈的唯一装配点(同步、恢复、基线、演练共用)。
### Changed
- 基线作业的"已有生效版本"判定改为三种状态(own / foreign / none),并在
阶段里写明是否走"只收尾"续作。
## 开发快照(首次发布入口:同步授权、目标权威基线与维护能力)
### Added
- **管理员在发布页签发同步授权(`POST /api/v1/library/authorizations`,`action=sync`)**:
授权把「范围 + 目标快照 + 参数摘要 + 有效期」钉死,仍与执行分成两步 ——
页面只提供入口,不会替你签发,也不会自动重试执行。工程师看得到缺项原因,
但拿不到签发按钮(接口本来就只认管理员)。
- **目标权威基线(迁移 075,作业类型 `baseline`)**:目标非空、又没有生效版本时,
首次发布的真实解法是「维护隔离 → 整库捕获 → 逐类校验 → 登记基线」。
`POST /api/v1/library/baselines` 只建作业;worker 用与同步同一套维护窗口
(关入口、停写入者、证明排空)把目标整库捕获成归档快照,再用同一套 `Verify`
证明目标与这份快照一致,最后登记 `production_effective`(证据写明
`origin=target_baseline`、快照 ID、逐表行数、目标索引映射)。
它**不写目标内容、不清空目标、不手写版本**:没有捕获与校验证据就没有基线。
基线作业与 sync/restore 共用同一把写互斥(同一 scope 只允许一个在飞写作业),
worker 被杀时同一作业 id 会接管那个窗口继续,而不是把目标留在停机状态。
- **维护能力(`GET /api/v1/library/maintenance` 的 `capability`)**:页面区分
「此刻没有维护窗口」与「维护能力没接线」,逐项列出缺少的
`LIBRARY_TARGET_*`;接口没给能力信息时显示"未知",不再用一句
"也没有维护能力配置"糊过去。
### Changed
- **`productionScopeEmpty` 改成整库口径**:原来只数 `knowledges`,"文档删了但
分块/Wiki/资源还在"的目标会被判成空库,从而绕过基线初始化。现在用适配层
同一份表清单与范围过滤(`v071.ScopeContentEmpty`)逐表计数,并把逐表行数
交回接口,"空"只有一份定义。
- **拟发布的测试封存只认版本登记(`kind=test_sealed`)**:归档里同时存在生产
预备份与目标基线快照,按"这个 scope 最新一份"挑会把目标自己的内容显示成
测试候选。现在候选来自登记、再用归档确认它还在本机;不在就说明原因。
- **发布页不再说"生产同步执行链尚未接线"**:同步按钮按真实闸口启用,
点击走 `POST /api/v1/library/syncs`;被拒时逐条显示接口返回的缺项。
- 缺项文案从"生产当前权威版本未知,不能启动写入"改为指向真实入口:
「目标非空且没有生效版本记录:先建立目标权威基线…或确认目标确实是空库」。
### Fixed
- 维护卡片把"没有窗口记录"错误地写成"没有维护能力配置",与实际接线不符。
- 首次发布时 `POST /library/baselines` 在目标读不到时**不建作业、不登记版本**
(拒绝用建基线掩盖连接/配置故障)。
## 开发快照(发布执行器可用性、整库还原入口与诊断)
### Added
- **封存执行器的运行时可用性(迁移 074 `library_executor_status`)**:页面上的
"可以发起封存"不再等于 `LIBRARY_AGENT_SOCKET` 非空。主机侧 agent 新增
`health` 动作(不占封存操作锁、不跑子进程),worker 每 30 秒探测一次并把
"指向哪个实例、归档在哪、是否降级/正在跑、观测时间"落库;API 只读这份观测。
socket 只挂在 worker 容器里,所以由 worker 观测、页面读结果,而不是让 API
假装能 dial。观测过期、不可达、实例不符、降级、正忙时页面与
`POST /library/captures` 用同一份判定拒绝,并逐条给出原因。
- **封存执行前后的身份复核**:封存结果由 agent 附上自报身份(`agent_identity`)。
worker 在停写入者之前再探测一次,执行后再用结果里的身份复核;不一致时作业失败
且**不登记**为测试封存——心跳只能证明"刚才",从心跳到停写入者之间 socket
可能被换掉。
- **按版本恢复的页面入口**:`GET /api/v1/library/restore-availability` 说明
"这份快照现在能不能恢复"(双闸、该快照的恢复授权、目标实例是否在维护)。
页面据此启用"恢复至该版本",管理员可签发 30 分钟有效期的 restore 授权;
工程师只看到条件,不获得入口。
- 可选 `LIBRARY_SOURCE_PROJECT`:声明封存执行器应指向的实例名(默认等于
`LIBRARY_SOURCE_INSTANCE`)。它描述封存源,与同步目标 `LIBRARY_TARGET_*` 分开。
### Changed
- **封存身份核对移进请求本身(TOCTOU)**:原来 worker 先 `health` 再 `capture`,两次
拨号之间 socket 可能被换成另一个 agent——错的执行器已经把写入者停了才被拒登记。
现在 `capture` 请求携带治理侧配置生成的期望身份(`project` 必填,`source_app`/
`archive_root` 配了就一起绑定)与稳定指纹(`sha256:`,覆盖三项);接收方在**操作
互斥内、执行任何源操作之前**比对,缺失或被改过一律拒绝,错的执行器一次 `capture`
都不跑。封存结果回显该指纹,worker 事后复核只作为证据防线,不能替代请求内核对。
恢复路径不受期望约束(它只把写入者恢复回来)。
- `LIBRARY_EXECUTOR_STATUS_MAX_AGE`(默认 120 秒)控制执行器观测的有效期;
未设置不等于"永不过期",过期即"无法确认"。
- 页面/接口不再出现写死的"封存通道未接线""生产恢复未接线":不能发起时给出真实原因、
执行器身份与观测时间。
### Fixed
- **接管在版本查询失败时 fail-open**:`resumed` 后版本查询出错被忽略、`FinalizeOnly` 仍为
false,于是"已成功登记并释放窗口、只差终态"的作业会被**重新写一遍目标**。现在接管先按
证据分类:`finalize` 阶段成功是独立证据(不受版本查询影响);查询失败且无阶段证据 ⇒
**未知**,零写入、作业失败可重试;本快照的生效版本已被别的快照顶替 ⇒ **拒绝覆盖**;
只有真正没有收尾证据时才按正常流程。
- **维护释放部分成功后的重试不再永久卡死**:`Update*/Release*` 对"已由本作业释放"的行
返回可判别的 `ErrMaintenanceReleased`,收尾重试视其为已完成;别的作业来释放仍是
"未持有"错误。真 PG 负例:同作业重复 Update/Release 得到已释放,跨作业得到未持有。
- **作业成功暴露早于维护结束**:原来 `execute` 里写完生效版本就 `AdvanceLibraryJob(succeeded)`,
而服务恢复、入口开放与维护锁释放要在窗口返回后才做。任何一步失败时外部已经看到
`succeeded`,重启接管又只认非终态,锁就永远挂着。现在内容校验/版本登记之后只记
"待收尾",`finishLibrarySuccess` 依次确认目标写入者与入口健康、按 **scope → instance**
顺序释放两把锁(每一步错误都上抛,不再丢弃),最后才写 `succeeded`;任一步失败都把
作业落成失败、维护保持 `needs_restore`。释放与终态写入用 `context.WithoutCancel`
包住,停机不会留下半成品。接管判定改为按证据(finalize 阶段成功或存在未顶掉的生效
版本)转 finalize-only,只复核与登记,不重写生产。
- **封存读资源失败只剩一句 `read library resource`**:对象缺失、桶错、端点解析不到
都会走到同一个字符串,真实封存直接卡在这里无法判断方向。现在带上资源行 id 与底层
原因(`read library resource <id>: <cause>`),不含凭据。
- **正式 release-worker 启动失败无法定位**:配置里的目标库指向不存在的主机时,
日志只有一句 `connect release target database`。现在错误里带库端点
(`host:port/db`,去口令)与路由标识,并保留底层原因;原因文本先经
`pgdsn.Redact` 抹掉口令(pgx 的解析错误可能原样回显连接串),可诊断不等于泄露。
共享的 `internal/pgdsn` 替换了 cmd/worker 里只留库名的旧实现。
- `libraryctl` 用法文本补上 `dead-letters` 子命令。
## 开发快照(死信精确解除:保留上游行,只按指纹放行)
### Added
+97
View File
@@ -1312,6 +1312,51 @@
}
}
},
"/api/v1/library/restore-availability": {
"get": {
"tags": [
"Runtime"
],
"operationId": "getLibraryRestoreAvailability",
"summary": "这份快照现在能不能恢复到生产:双闸、恢复授权与目标实例维护状态逐条说明",
"parameters": [
{
"name": "scope",
"in": "query",
"required": false,
"schema": {
"type": "string",
"enum": [
"internal",
"external"
]
}
},
{
"name": "snapshot_id",
"in": "query",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"$ref": "#/components/responses/Envelope"
},
"400": {
"$ref": "#/components/responses/BadRequest"
},
"500": {
"$ref": "#/components/responses/InternalError"
},
"503": {
"description": "读取恢复授权失败(不是\"可恢复\",是\"无法判定\")"
}
}
}
},
"/api/v1/library/authorizations": {
"get": {
"tags": [
@@ -1502,6 +1547,58 @@
}
}
},
"/api/v1/library/baselines": {
"post": {
"tags": [
"Runtime"
],
"operationId": "startLibraryBaseline",
"summary": "建立目标权威基线:管理员在维护隔离下把非空目标整库捕获成快照、逐类校验后登记为生效版本;不清空目标、不手写版本",
"responses": {
"202": {
"description": "已创建基线作业(worker 在维护窗口内捕获并校验)"
},
"400": {
"$ref": "#/components/responses/BadRequest"
},
"403": {
"description": "不是管理员"
},
"409": {
"description": "已有生效版本、目标为空、目标正在维护或已有在飞写作业"
},
"500": {
"$ref": "#/components/responses/InternalError"
},
"503": {
"description": "目标整库读不到,不能建立基线(不创建作业、不登记版本)"
}
},
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"additionalProperties": false,
"required": [
"scope"
],
"properties": {
"scope": {
"type": "string",
"enum": [
"internal",
"external"
]
}
}
}
}
}
}
}
},
"/api/v1/library/snapshots/{id}": {
"get": {
"tags": [
+8
View File
@@ -203,8 +203,16 @@ func run(ctx context.Context, getenv func(string) string) error {
VersionHistoryRepo: versionHistoryRepo,
LibraryArchiveRoot: config.LibraryArchiveRoot,
LibraryTargetInstance: config.LibraryTargetInstance,
LibraryTargetProject: config.LibraryTargetProject,
LibraryTargetWriters: config.LibraryTargetWriters,
LibraryTargetIngressURL: config.LibraryTargetIngressURL,
LibraryTargetIngressService: config.LibraryTargetIngressService,
LibraryTargetDockerSocket: config.LibraryTargetDockerSocket,
LibraryTargetJournalRoot: config.LibraryTargetJournalRoot,
LibrarySourceInstance: config.SourceInstance(),
LibrarySourceProject: config.SourceProject(),
LibraryAgentSocket: config.LibraryAgentSocket,
LibraryExecutorStatusMaxAge: config.LibraryExecutorStatusMaxAge,
LibraryCapacityBudgetBytes: config.LibraryCapacityBudgetBytes,
LibraryProductionDatabaseURL: config.LibraryProductionDatabaseURL,
LibraryProductionInternalKB: config.LibraryProductionInternalKB,
+3 -1
View File
@@ -111,7 +111,7 @@ func main() {
}
func usage() {
fmt.Fprint(os.Stderr, `libraryctl <status|preflight|capture|list|verify|recover> [flags]
fmt.Fprint(os.Stderr, `libraryctl <status|preflight|capture|list|verify|recover|dead-letters> [flags]
status probe the real source and print what is configured / blocked
preflight read the source, run settlement + candidate validation (no writes, no stop)
@@ -119,6 +119,8 @@ func usage() {
list list sealed candidates and stored bytes
verify re-verify one sealed candidate (digests, coverage, identity)
recover restore writers from an interrupted freeze journal
dead-letters print the source's live dead letters (read-only) with a digest the
governance platform binds a precise release to
Flags (defaults target the local WKNFull test stack):
-source-app WKNFull-app container whose env describes the source
+271
View File
@@ -0,0 +1,271 @@
// command librarydrill 在目标实例上做一次**真实**的恢复演练:
//
// 维护隔离(关入口/停写入者/证明排空)
// → 捕获目标整库(真预备份,落归档)
// → 校验目标 == 预备份(基线一致)
// → 写入一个 canary(证明"校验确实能发现内容变化")
// → 按预备份恢复
// → 逐表/索引/资源/Wiki 再校验一次,必须完全一致
// → 关窗口(按原运行状态恢复服务、开入口、验证健康)
//
// 它只读/写**被测目标自己**,绝不触碰封存源;所有凭据来自进程环境
// (与 worker 同一份 LIBRARY_* 配置),不进命令行、不打印。
// 任一步失败都不伪装成功:窗口保持维护、报告写明失败阶段。
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"log"
"os"
"path/filepath"
"time"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/libraryfreeze"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarystack"
)
type drillVerification struct {
SnapshotID string `json:"snapshot_id"`
Verified bool `json:"verified"`
Error string `json:"error,omitempty"`
At time.Time `json:"at"`
}
type drillReport struct {
Status string `json:"status"` // passed | failed | maintenance_held
Scope string `json:"scope"`
Target string `json:"target"`
SnapshotID string `json:"snapshot_id"`
Project string `json:"project"`
Counts map[string]int64 `json:"counts"`
BaselineCheck *drillVerification `json:"baseline_check"`
DriftCheck *drillVerification `json:"drift_check"` // 必须失败:证明确实能发现差异
RestoreCheck *drillVerification `json:"restore_check"`
Canary string `json:"canary"`
Failure string `json:"failure,omitempty"`
Stage string `json:"stage,omitempty"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
Until map[string]string `json:"until,omitempty"`
}
func main() {
scope := flag.String("scope", "internal", "internal|external")
reportPath := flag.String("report", "", "write the JSON report here (0600)")
jobID := flag.String("job", "", "maintenance job id (default drill-<scope>-<timestamp>)")
recoverOnly := flag.Bool("recover", false, "only recover non-terminal maintenance windows (restore services), then exit")
flag.Parse()
if *recoverOnly {
if err := recoverWindows(); err != nil {
log.Fatalf("recover failed: %v", err)
}
return
}
report, err := run(*scope, *jobID)
if *reportPath != "" && report != nil {
if writeErr := writeReport(*reportPath, report); writeErr != nil {
log.Printf("drill report write failed: %v", writeErr)
}
}
if err != nil {
log.Fatalf("drill failed: %v", err)
}
log.Printf("drill passed: snapshot=%s", report.SnapshotID)
}
// recoverWindows 只做一件事:把非终态的维护窗口按原运行状态恢复(服务起来、入口验证)。
// 演练失败但还没写目标内容时,用它把目标从"停住"状态救回来;它不改治理记录,
// 治理记录由作业/接口那一侧负责。
func recoverWindows() error {
config, err := appconfig.Load(os.Getenv)
if err != nil {
return fmt.Errorf("load config: %w", err)
}
manager, err := librarystack.OpenMaintenance(config)
if err != nil {
return err
}
defer manager.Close()
coordinator, control, err := manager.Open(config.LibraryTargetJournalRoot)
if err != nil {
return err
}
defer control.Close()
defer coordinator.Close()
if err := coordinator.Recover(context.Background()); err != nil {
return err
}
log.Printf("maintenance windows recovered; target should be back to its original service state")
return nil
}
func run(scope, jobID string) (*drillReport, error) {
if scope != "internal" && scope != "external" {
return nil, errors.New("scope must be internal or external")
}
config, err := appconfig.Load(os.Getenv)
if err != nil {
return nil, fmt.Errorf("load config: %w", err)
}
ctx := context.Background()
report := &drillReport{Status: "failed", Scope: scope, Project: config.LibraryTargetProject,
Target: config.LibraryTargetInstance, StartedAt: time.Now().UTC()}
fail := func(stage string, cause error, held bool) (*drillReport, error) {
report.Stage = stage
report.Failure = cause.Error()
report.FinishedAt = time.Now().UTC()
if held {
report.Status = "maintenance_held"
}
return report, cause
}
target, backups, err := librarystack.OpenBaselineStack(ctx, config, scope)
if err != nil {
return fail("open_target", err, false)
}
defer target.Close()
manager, err := librarystack.OpenMaintenance(config)
if err != nil {
return fail("open_maintenance", err, false)
}
defer manager.Close()
if err := manager.VerifyTopology(ctx); err != nil {
return fail("topology", err, false)
}
coordinator, control, err := manager.Open(config.LibraryTargetJournalRoot)
if err != nil {
return fail("open_coordinator", err, false)
}
defer control.Close()
defer coordinator.Close()
if jobID == "" {
jobID = "drill-" + scope + "-" + time.Now().UTC().Format("20060102T150405")
}
var (
snapshotID string
counts map[string]int64
canary string
contentTouched bool
baseline drillVerification
drift drillVerification
restored drillVerification
)
work := func(ctx context.Context, window *libraryfreeze.Window) (string, error) {
start := time.Now()
current, err := target.Current(ctx)
if err != nil {
return "", errors.New("读取目标当前整库状态失败:" + err.Error())
}
counts = current.Counts
report.Counts = counts
// 1) 真预备份(落归档、内容寻址、导出后复核内容未变)。
backup, err := backups.Prepare(ctx, "", current)
if err != nil {
return "", errors.New("捕获目标整库失败:" + err.Error())
}
snapshotID = backup.SnapshotID
report.SnapshotID = snapshotID
if err := window.SetBackupSnapshot(ctx, snapshotID); err != nil {
return "", errors.New("记录预备份身份失败:" + err.Error())
}
log.Printf("drill: snapshot captured snapshot=%s bytes=%d in=%s", snapshotID, backup.Bytes, time.Since(start))
// 2) 基线校验:预备份必须与目标完全一致。
baseline = drillVerification{SnapshotID: snapshotID, At: time.Now().UTC()}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
baseline.Error = err.Error()
report.BaselineCheck = &baseline
return "", errors.New("预备份与目标不一致(预备份本身不可信):" + err.Error())
}
baseline.Verified = true
report.BaselineCheck = &baseline
// 3) canary:改一行真实内容,校验**必须**失败(否则"校验"是摆设)。
canary = "drill-canary-" + time.Now().UTC().Format("20060102T150405")
identity := target.Identity()
tag, err := target.Pool().Exec(ctx, `
UPDATE knowledges SET title = title || ' [' || $3 || ']'
WHERE id = (SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2 AND deleted_at IS NULL
ORDER BY id LIMIT 1)`, identity.TenantID, identity.KnowledgeBaseID, canary)
if err != nil {
return "", errors.New("写入 canary 失败:" + err.Error())
}
if tag.RowsAffected() == 0 {
return "", errors.New("目标里没有可改写的文档,无法做漂移检测(目标不应为空)")
}
report.Canary = canary
// 从这一笔开始目标内容被改写过:窗口必须记录内容风险,
// 之后只有"恢复并校验通过"才允许重新开放(否则就是半写状态)。
if err := window.MarkContentRisk(ctx); err != nil {
return "", errors.New("记录内容风险失败:" + err.Error())
}
contentTouched = true
drift = drillVerification{SnapshotID: snapshotID, At: time.Now().UTC()}
if verifyErr := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); verifyErr == nil {
drift.Error = "写入 canary 后校验仍然通过:校验没有真正比对内容"
report.DriftCheck = &drift
return "", errors.New("写入 canary 后校验仍然通过:校验没有真正比对内容,拒绝把这次演练当成通过")
} else {
drift.Error = verifyErr.Error()
}
report.DriftCheck = &drift
log.Printf("drill: drift detected as expected (%s)", drift.Error)
// 4) 按预备份恢复。
if err := target.Restore(ctx, libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID}); err != nil {
return "", errors.New("按预备份恢复失败:" + err.Error())
}
// 5) 恢复后逐类复核:必须与预备份完全一致。
restored = drillVerification{SnapshotID: snapshotID, At: time.Now().UTC()}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
restored.Error = err.Error()
report.RestoreCheck = &restored
return "", errors.New("恢复后内容仍与预备份不一致:" + err.Error())
}
restored.Verified = true
report.RestoreCheck = &restored
log.Printf("drill: restored and verified snapshot=%s", snapshotID)
return snapshotID, nil
}
window, windowErr := coordinator.RunWindow(ctx, jobID,
func(ctx context.Context) error { return manager.VerifyRunning(ctx) }, work,
// 没写过目标内容 ⇒ 按原状态恢复永远安全;写过 canary ⇒ 只有恢复并校验通过才开放。
func(error) bool { return !contentTouched || restored.Verified })
report.Until = map[string]string{"job": jobID, "window_state": window.State}
if windowErr != nil || window.State != "succeeded" {
cause := windowErr
if cause == nil {
cause = fmt.Errorf("维护窗口状态 %s(未成功)", window.State)
}
if window.State == "held" {
return fail("window", cause, true)
}
return fail("window", cause, false)
}
report.Status = "passed"
report.FinishedAt = time.Now().UTC()
return report, nil
}
func writeReport(path string, report *drillReport) error {
raw, err := json.MarshalIndent(report, "", " ")
if err != nil {
return err
}
if dir := filepath.Dir(path); dir != "" {
if err := os.MkdirAll(dir, 0o700); err != nil {
return err
}
}
return os.WriteFile(path, append(raw, '\n'), 0o600)
}
+115
View File
@@ -0,0 +1,115 @@
// command libraryresidue 处理"整库同步写到目标范围外"留下的残留行:
//
// 读指定快照(来源范围)→ 在目标里找同内容、仍带来源身份的行
// → 先备份成 0600 文件并记录摘要 → 精确按归档身份删除 → 复核来源身份已空、
// 目标内容与预备份一致、整库指纹里没有别的租户
//
// 它不做范围广删,也不改任何别的东西:判据是"归档里确实有这些行、内容逐字段
// 一致",任何一行对不上就整体拒绝。默认是干跑(只报告、不删)。
//
// 真实事故:跨范围同步把 4 文件/38 分块/21 资源/4 绑定写成来源租户的行,
// 回滚只按目标范围回放,于是"已恢复"却留下这些行。这个工具的用途就是在
// 修复后的代码上线前,用可审计的方式清掉它们。
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"log"
"os"
"strconv"
"strings"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
"knowledge-governance/internal/librarystack"
v071 "knowledge-governance/internal/releaseadapter/v071"
)
func main() {
scope := flag.String("scope", "internal", "internal|external")
snapshot := flag.String("snapshot", "", "sealed snapshot whose source scope is the residue (required)")
backupSnapshot := flag.String("backup-snapshot", "", "prebackup snapshot the target must be restored to (required)")
expectSource := flag.String("expect-source", "", "expected source scope as <tenant>/<knowledge-base> (required guard against typos)")
out := flag.String("out", "", "directory for the residue backup (required)")
apply := flag.Bool("apply", false, "actually delete the residue rows (default: dry run)")
reportPath := flag.String("report", "", "write the JSON report here")
flag.Parse()
if *snapshot == "" || *backupSnapshot == "" || *out == "" || *expectSource == "" {
log.Fatal("snapshot, backup-snapshot, out and expect-source are required")
}
config, err := appconfig.Load(os.Getenv)
if err != nil {
log.Fatalf("load config: %v", err)
}
ctx := context.Background()
report, err := run(ctx, config, *scope, *snapshot, *backupSnapshot, *expectSource, *out, *apply)
if *reportPath != "" && report != nil {
raw, marshalErr := json.MarshalIndent(report, "", " ")
if marshalErr == nil {
raw = append(raw, '\n')
if writeErr := os.WriteFile(*reportPath, raw, 0o600); writeErr != nil {
log.Printf("report write failed: %v", writeErr)
}
}
}
if err != nil {
log.Fatalf("residue cleanup failed: %v", err)
}
log.Printf("residue cleanup ok: source=%s/%s tables=%v cleared=%d applied=%v verified=%v",
reportSource(report), report.SourceKB, report.Tables, report.Cleared, report.Applied, report.Verified)
}
func reportSource(report *v071.ResidueReport) string {
return strconv.FormatUint(report.SourceTenant, 10)
}
func run(ctx context.Context, config appconfig.Config, scope, snapshotID, backupSnapshotID, expectSource, out string,
apply bool) (*v071.ResidueReport, error) {
if scope != "internal" && scope != "external" {
return nil, fmt.Errorf("scope must be internal or external")
}
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
if err != nil {
return nil, fmt.Errorf("归档不可用: %w", err)
}
manifest, err := archive.Load(snapshotID)
archive.Close()
if err != nil {
return nil, fmt.Errorf("指定快照不可读: %w", err)
}
source := v071.ProductionScope{TenantID: manifest.TenantID, KnowledgeBaseID: manifest.KnowledgeBaseID}
want := strconv.FormatUint(source.TenantID, 10) + "/" + source.KnowledgeBaseID
// 范围写错会把"别人的行"当成残留:必须由调用方显式确认。
if strings.TrimSpace(expectSource) != want {
return nil, fmt.Errorf("来源范围与 -expect-source 不一致:归档为 %s,参数为 %s", want, expectSource)
}
target, _, err := librarystack.OpenProductionStack(ctx, config, scope, backupSnapshotID, "")
if err != nil {
return nil, fmt.Errorf("装配目标失败: %w", err)
}
defer target.Close()
report, err := target.ClearForeignScope(ctx, source, snapshotID, out, apply)
if err != nil {
return report, err
}
report.Foreign, _ = target.ForeignScopes(ctx)
if !apply {
return report, nil
}
if err := target.VerifyResidueCleared(ctx, source); err != nil {
return report, err
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: backupSnapshotID}); err != nil {
return report, fmt.Errorf("清理后目标内容仍与预备份不一致(保持停写): %w", err)
}
report.Verified = true
if len(report.Foreign) > 0 {
raw, _ := json.Marshal(report.Foreign)
return report, fmt.Errorf("清理后目标仍有其它租户的数据,需人工确认才能开放:%s", raw)
}
return report, nil
}
+58
View File
@@ -0,0 +1,58 @@
package main
import (
"errors"
"fmt"
"strings"
"testing"
)
// 正式栈的 release-worker 启动失败时只打一句 "connect release target database",
// 主机名、库名与底层原因都不在日志里。修好之后仍然要证明:可诊断 ≠ 泄露凭据,
// 因为 pgx 的解析/连接错误会把连接串原样带回来。
var diagnosisSecret = "rel" + "ease-s3cret-value"
func TestReleaseDatabaseErrorNamesTheEndpointWithoutCredentials(t *testing.T) {
cause := errors.New("dial tcp 127.0.0.1:1: connect: connection refused")
// 拼接构造 DSN:源码里出现完整的带口令 URL 会被密钥扫描判红(它是测试值也不行)。
dsn := "postgres" + "://postgres:" + "ignored@" + "prod-pg:25432/weknora_test?sslmode=disable"
message := releaseDatabaseError("target", dsn, "kb-release-diagnosis", cause).Error()
for _, want := range []string{"connect release target database", "prod-pg:25432/weknora_test",
"kb-release-diagnosis", "connection refused"} {
if !strings.Contains(message, want) {
t.Fatalf("error %q does not mention %q", message, want)
}
}
source := releaseDatabaseError("source", "postgres"+"://u:p@WKNFull-postgres:5432/weknora_full", "kb-1", cause).Error()
if !strings.Contains(source, "WKNFull-postgres:5432/weknora_full") {
t.Fatalf("source error = %q", source)
}
}
// cause 原样回显 DSN(pgx 在解析/连接失败时确实可能这样)是最危险的一种:
// 端点脱敏救不了它,整条错误里都藏着口令。
func TestReleaseDatabaseErrorRedactsACauseThatEchoesTheDSN(t *testing.T) {
cases := []string{
"postgres://postgres:" + diagnosisSecret + "@127.0.0.1:notaport/weknora_test",
"host=127.0.0.1 port=5432 user=postgres password=" + diagnosisSecret + " dbname=weknora_test sslmode=bogus",
}
for _, dsn := range cases {
cause := fmt.Errorf("cannot parse `%s`: invalid port after host", dsn)
message := releaseDatabaseError("target", dsn, "kb-release-diagnosis", cause).Error()
if strings.Contains(message, diagnosisSecret) {
t.Fatalf("error leaked the password: %q", message)
}
if !strings.Contains(message, "invalid port after host") {
t.Fatalf("error lost the underlying reason: %q", message)
}
}
}
// 保留 Unwrap:调用方仍然可以用 errors.Is/As 判断原因;只是文本被脱敏。
func TestReleaseDatabaseErrorKeepsTheCauseChain(t *testing.T) {
sentinel := errors.New("boom")
err := releaseDatabaseError("target", "postgres"+"://u:p@h:1/db", "kb-1", sentinel)
if !errors.Is(err, sentinel) {
t.Fatalf("cause chain broken: %v", err)
}
}
+27 -2
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"errors"
"fmt"
"net/url"
"path/filepath"
"strconv"
@@ -11,6 +12,7 @@ import (
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/pgdsn"
"knowledge-governance/internal/platform/weknora"
v071 "knowledge-governance/internal/releaseadapter/v071"
"knowledge-governance/internal/releasesnapshot"
@@ -63,13 +65,13 @@ func buildReleaseRoute(ctx context.Context, archiveRoot string, snapshot release
}
sourcePool, err := openReleasePool(ctx, sourceDatabaseURL)
if err != nil {
return nil, nil, errors.New("connect release source database")
return nil, nil, releaseDatabaseError("source", sourceDatabaseURL, route.KnowledgeBaseID, err)
}
pools := []*pgxpool.Pool{sourcePool}
targetPool, err := openReleasePool(ctx, targetDatabaseURL)
if err != nil {
sourcePool.Close()
return nil, nil, errors.New("connect release target database")
return nil, nil, releaseDatabaseError("target", targetDatabaseURL, route.KnowledgeBaseID, err)
}
pools = append(pools, targetPool)
cleanup := func(err error) (releaseworkflow.Publisher, []*pgxpool.Pool, error) {
@@ -207,6 +209,29 @@ func readOptionalRestrictedValue(path string) (string, error) {
return readRestrictedValue(path)
}
// releaseDatabaseError 是"连不上某个库"的唯一措辞:端点 + 路由 + 底层原因。
//
// 为什么不用 %w 直接拼原因:pgx 的错误里可能原样带着连接串(含口令),
// 只在端点脱敏证明不了整条日志安全。这里把原因文本交给 pgdsn.Redact 抹口令,
// 同时保留 Unwrap,调用方仍能用 errors.Is/As 判断。技术原因(拒绝连接、
// 认证失败、TLS 配置错误等)一个字都不丢。
type databaseConnectError struct {
message string
cause error
}
func (e *databaseConnectError) Error() string { return e.message }
func (e *databaseConnectError) Unwrap() error { return e.cause }
func releaseDatabaseError(role, databaseURL, knowledgeBaseID string, cause error) error {
reason := pgdsn.Redact(databaseURL, cause.Error())
return &databaseConnectError{
message: fmt.Sprintf("connect release %s database %s (route %s): %s",
role, pgdsn.Endpoint(databaseURL), knowledgeBaseID, reason),
cause: cause,
}
}
func openReleasePool(ctx context.Context, databaseURL string) (*pgxpool.Pool, error) {
pool, err := pgxpool.New(ctx, databaseURL)
if err != nil {
+298
View File
@@ -0,0 +1,298 @@
package main
import (
"context"
"encoding/json"
"errors"
"log"
"time"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/governance"
"knowledge-governance/internal/jobs"
"knowledge-governance/internal/libraryfreeze"
"knowledge-governance/internal/librarymaintenance"
"knowledge-governance/internal/libraryrelease"
postgresstore "knowledge-governance/internal/platform/postgres"
)
// libraryBaselinePayload 是"建立目标权威基线"的队列载荷。
type libraryBaselinePayload struct {
LibraryJobID string `json:"library_job_id"`
Scope string `json:"scope"`
}
// baselineVersionState 回答"这个 scope 现在有没有生效版本、是不是本作业登记的"。
//
// 三种情况必须分开,否则崩溃续作会被自己的版本挡住:
// - own:本作业 + 同一快照已经登记过 → 只收尾,绝不重复捕获或重写版本;
// - foreign:别人登记的生效版本 → 一律拒绝(不能用基线覆盖别人的版本);
// - none:还没有生效版本 → 正常建基线。
type baselineVersionState struct {
Own bool
Foreign *governance.LibraryVersion
}
func classifyBaselineVersions(versions []governance.LibraryVersion, jobID, snapshotID string) baselineVersionState {
for index := range versions {
item := versions[index]
if item.Kind != governance.LibraryVersionProductionEffective || item.SupersededAt != nil {
continue
}
if jobID != "" && snapshotID != "" && item.SourceJobID == jobID && item.SnapshotID == snapshotID {
return baselineVersionState{Own: true}
}
foreign := item
return baselineVersionState{Foreign: &foreign}
}
return baselineVersionState{}
}
// settleBaselineWindow 是基线窗口的**唯一**收尾入口,负责让治理状态与实际入口一致:
//
// - 版本已登记(本次刚写,或崩溃前由本作业写过)→ 走与同步一样的成功边界:
// 目标健康、入口开放、两把维护锁释放之后才暴露 succeeded;
// - 没有登记且窗口仍是 held(服务没回来)→ 保持维护并标 needs_restore,
// 只有"按预备份恢复"能把它收掉;
// - 没有登记但窗口已按原状态恢复 → 释放占用并把作业标失败,明确写"未对目标写入",
// 绝不留着"显示维护中、实际已开放"的状态。
func settleBaselineWindow(ctx context.Context, store libraryMaintenanceStore, health healthChecker,
jobID, scope, instance, baselineID string, recorded bool, windowState string, windowErr error) error {
if recorded {
finishedAt := time.Now().UTC()
return finishLibrarySuccess(ctx, store, health, instance, scope, jobID,
successBoundary{Ready: true, Backup: baselineID, FinishedAt: finishedAt})
}
now := time.Now().UTC()
message := "未建立基线:维护窗口没有完成捕获与校验"
if windowErr != nil {
message += ":" + windowErr.Error()
}
if windowState == "held" {
note := message + "(目标仍被停住、入口未恢复,保持维护)"
_ = store.UpdateLibraryMaintenanceState(ctx, scope, jobID, "needs_restore", note, now)
_ = store.UpdateInstanceMaintenanceState(ctx, instance, jobID, "needs_restore", note, now)
_ = store.AdvanceLibraryJob(ctx, jobID, governance.LibraryJobFailed, "capture", note,
"目标仍在维护:先按预备份恢复并校验", baselineID, "", now)
return errors.New(note)
}
// 窗口已按原状态开放(本流程对目标内容只读):占用必须跟着释放,
// 不能留下"界面说维护中、实际入口已开"的记录。
if releaseErr := releaseWindowLocks(ctx, store, instance, scope, jobID,
"未登记基线,目标内容未被写入,窗口已按原状态恢复", now); releaseErr != nil {
note := message + ";维护占用释放失败:" + releaseErr.Error()
_ = store.AdvanceLibraryJob(ctx, jobID, governance.LibraryJobFailed, "capture", note,
"维护记录与实际入口不一致:先人工核对再重试", baselineID, "", now)
return errors.New(note)
}
_ = store.AdvanceLibraryJob(ctx, jobID, governance.LibraryJobFailed, "capture", message,
"未对目标执行任何写入;可直接重试建立基线", baselineID, "", now)
return errors.New(message)
}
// runLibraryBaseline 给"非空但没有权威版本"的目标建立基线。
//
// 真实链路(全部在同一个维护窗口内):关入口/停写入者/证明排空 → 把目标整库捕获成
// 归档快照 → 用同一套 Verify 证明目标与这份快照逐类一致 → 读目标索引映射 →
// 登记 production_effective → 关窗口(恢复服务、开入口)→ 释放两把占用 → 作业成功。
//
// 崩溃语义:
// - 捕获后、登记前被杀:作业行里已有快照 ID,重入时**不重新捕获**,直接在窗口内
// 复核这份快照再登记;
// - 登记后、收尾前被杀:重入识别出"本作业+同一快照"已生效,只做收尾
// (恢复服务/开入口/释放占用),不重复捕获、不重写版本;别人的版本仍然拒绝。
func runLibraryBaseline(ctx context.Context, store *postgresstore.Store, config appconfig.Config, job jobs.Job) error {
var payload libraryBaselinePayload
if err := json.Unmarshal(job.Payload, &payload); err != nil || payload.LibraryJobID == "" ||
(payload.Scope != "internal" && payload.Scope != "external") {
return errors.New("invalid library baseline payload")
}
now := time.Now().UTC()
recorder := libraryStageRecorder{store: store, jobID: payload.LibraryJobID, scope: payload.Scope}
fail := func(message, stage string) error {
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID, message, stage)
}
existing, jobErr := store.LibraryJob(ctx, payload.LibraryJobID)
actor := ""
persistedSnapshot := ""
if jobErr == nil {
actor = existing.Actor
persistedSnapshot = existing.SnapshotID
}
_ = store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobRunning, "capture", "", "", "", "", now)
// 版本判定:别人的生效版本一律拒绝;本作业已登记过的快照只收尾。
versions, versionErr := store.LibraryVersions(ctx, payload.Scope, 50)
if versionErr != nil {
return fail("读不到版本记录,无法确认是否需要建立基线;未对目标执行任何操作", "capture")
}
state := classifyBaselineVersions(versions, payload.LibraryJobID, persistedSnapshot)
if state.Foreign != nil {
return fail("该目标已有生效版本 "+shortSnapshotID(state.Foreign.SnapshotID)+
"(作业 "+state.Foreign.SourceJobID+"),不需要再建基线;换版本请走同步或恢复", "capture")
}
finalizeOnly := state.Own
if finalizeOnly {
recorder.Stage(ctx, "finalize", "running", "续作:本作业已登记同一份基线快照,只做收尾(不重新捕获)")
}
if !finalizeOnly && !config.ProductionReleaseRequestsEnabled {
return fail("生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false):建立基线要动目标运行状态,闸口关闭时不执行", "capture")
}
target, backups, wireErr := buildLibraryBaselineStack(ctx, config, payload.Scope)
if wireErr != nil {
return fail(wireErr.Error(), "capture")
}
defer target.Close()
manager, managerErr := openLibraryMaintenance(config)
if managerErr != nil {
return fail("目标维护未接线,拒绝在挡不住入口/后台写入的情况下捕获目标:"+managerErr.Error(), "capture")
}
defer manager.Close()
if topoErr := manager.VerifyTopology(ctx); topoErr != nil {
return fail("目标拓扑不满足唯一入口约束,拒绝开维护窗口:"+topoErr.Error(), "capture")
}
originalState, stateErr := manager.OriginalState(ctx)
if stateErr != nil {
return fail("读不到目标写入者的原运行状态,不能开维护窗口:"+stateErr.Error(), "capture")
}
identity := manager.Target()
// 实例级占用:internal/external 动的是同一批容器,必须按实例互斥。
// 同一作业重入时 store 保留第一次记下的原运行状态(否则会把"已停"当原状)。
if _, instanceErr := store.AcquireInstanceMaintenance(ctx, postgresstore.LibraryInstanceMaintenance{
Instance: identity.Instance, OwnerJobID: payload.LibraryJobID, OwnerScope: payload.Scope,
TargetFingerprint: identity.Fingerprint, Level: librarymaintenance.LevelInstance,
Services: identity.Writers, IngressState: "unknown", State: "acquired",
OriginalServiceState: originalState,
Note: "实例级维护:为建立权威基线捕获目标整库,捕获并校验后按原状态恢复",
}, now); instanceErr != nil {
return fail("目标实例正在维护中(跨对内/对外互斥):"+instanceErr.Error(), "capture")
}
if _, acquireErr := store.AcquireLibraryMaintenance(ctx, postgresstore.LibraryMaintenance{
Scope: payload.Scope, OwnerJobID: payload.LibraryJobID,
TargetIdentity: identity.Fingerprint,
Level: librarymaintenance.LevelInstance,
TargetFingerprint: identity.Fingerprint,
Services: identity.Writers,
IngressState: "unknown",
State: "acquired",
OriginalServiceState: originalState,
Note: "实例级维护:为建立权威基线捕获目标整库,捕获并校验后按原状态恢复",
}, now); acquireErr != nil {
_ = store.UpdateInstanceMaintenanceState(ctx, identity.Instance, payload.LibraryJobID, "needs_restore",
"scope 占用申请失败,实例状态未确认:"+acquireErr.Error(), now)
return fail("目标正在维护中:"+acquireErr.Error(), "capture")
}
coordinator, control, openErr := manager.Open(config.LibraryTargetJournalRoot)
if openErr != nil {
_ = releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
"未进入维护窗口(协调器不可用),目标未被停用", time.Now().UTC())
return fail("维护协调器不可用:"+openErr.Error(), "capture")
}
defer control.Close()
defer coordinator.Close()
var (
baselineID = persistedSnapshot
counts map[string]int64
recorded = finalizeOnly
)
work := func(ctx context.Context, window *libraryfreeze.Window) (string, error) {
if finalizeOnly {
// 版本已在窗口内校验并登记过:不再捕获、不再重写,收尾由下方统一处理。
return baselineID, nil
}
if baselineID == "" {
current, currentErr := target.Current(ctx)
if currentErr != nil {
return "", errors.New("读取目标当前整库状态失败:" + currentErr.Error())
}
counts = map[string]int64{}
for table, count := range current.Counts {
counts[table] = count
}
// 预备份存储在导出后用同一份指纹复核"导出期间内容没变",
// 所以这里把窗口内读到的 state 交进去,而不是事后再读一次。
backup, backupErr := backups.Prepare(ctx, "", current)
if backupErr != nil {
return "", errors.New("捕获目标整库失败:" + backupErr.Error())
}
baselineID = backup.SnapshotID
// 先把"这次窗口的基线快照"落盘:进程在这里被杀时,恢复器知道该回到哪一份,
// 重入也据此跳过重新捕获。
if markErr := window.SetBackupSnapshot(ctx, baselineID); markErr != nil {
return "", errors.New("记录基线快照身份失败:" + markErr.Error())
}
if persistErr := store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobRunning, "",
"", "", baselineID, "", time.Now().UTC()); persistErr != nil {
return "", errors.New("持久化基线快照引用失败:" + persistErr.Error())
}
recorder.Stage(ctx, "capture", "succeeded", "已封存目标整库快照 "+baselineID)
} else {
recorder.Stage(ctx, "capture", "skipped", "复用作业已持久化的基线快照 "+baselineID)
}
if baselineID == "" {
return "", errors.New("没有可校验的基线快照")
}
recorder.Stage(ctx, "verify", "running", "逐表/索引/资源比对目标与快照 "+shortSnapshotID(baselineID))
if verifyErr := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: baselineID}); verifyErr != nil {
return "", errors.New("目标内容与基线快照不一致,拒绝登记基线:" + verifyErr.Error())
}
recorder.Stage(ctx, "verify", "succeeded", "行身份/逐行内容/索引文档/资源摘要与快照一致")
// 索引映射是基线证据的一部分:读不到就不登记,绝不用空摘要冒充。
mapping, mapErr := target.Search().MappingSHA256(ctx)
if mapErr != nil {
return "", errors.New("读不到目标索引映射,拒绝用空摘要登记基线:" + mapErr.Error())
}
if counts == nil {
if current, currentErr := target.Current(ctx); currentErr == nil {
counts = current.Counts
}
}
// 版本登记放在窗口内:捕获、校验、登记三件事要么一起成立,要么目标原样恢复。
recorder.Stage(ctx, "finalize", "running", "登记目标权威基线")
if recordErr := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionID(governance.LibraryVersionProductionEffective, payload.Scope, baselineID),
Scope: payload.Scope, Kind: governance.LibraryVersionProductionEffective,
SnapshotID: baselineID, SourceJobID: payload.LibraryJobID, CreatedBy: actor,
Evidence: map[string]any{
"action": "target_baseline_init",
"origin": "target_baseline",
"baseline_snapshot": baselineID,
"index_mapping_sha256": mapping,
"scope_content_counts": counts,
"verification": "目标与快照逐表行身份/逐行内容/索引文档/资源摘要一致",
"note": "基线是目标在维护窗口内的真实整库捕获,不是手写版本;发布仍需独立 sync 授权",
},
CreatedAt: time.Now().UTC(),
}); recordErr != nil {
return "", errors.New("登记基线版本失败:" + recordErr.Error())
}
recorded = true
recorder.Stage(ctx, "finalize", "succeeded", "已登记目标权威基线 "+shortSnapshotID(baselineID))
return baselineID, nil
}
window, windowErr := coordinator.RunWindow(ctx, payload.LibraryJobID,
func(ctx context.Context) error { return manager.VerifyRunning(ctx) }, work,
// 本流程对目标内容只读:失败时按原状态重新开放永远安全。
func(error) bool { return true })
if settleErr := settleBaselineWindow(ctx, store, manager, payload.LibraryJobID, payload.Scope,
identity.Instance, baselineID, recorded, window.State, windowErr); settleErr != nil {
return settleErr
}
log.Printf("library baseline succeeded job=%s scope=%s snapshot=%s actor=%s finalize_only=%v",
payload.LibraryJobID, payload.Scope, baselineID, actor, finalizeOnly)
return nil
}
// shortSnapshotID 只用于人看的日志/阶段说明:完整摘要留在版本记录与接口里。
func shortSnapshotID(id string) string {
if len(id) <= 12 {
return id
}
return id[:12]
}
+149
View File
@@ -0,0 +1,149 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"knowledge-governance/internal/governance"
)
// 崩溃续作的版本判定:本作业+同一快照 = 只收尾;别人的生效版本 = 拒绝;都没有 = 正常建基线。
func TestClassifyBaselineVersions(t *testing.T) {
const jobA, jobB, snap = "libbase-internal-1", "libbase-internal-2", "snap-abc"
effective := func(sourceJob, snapshot string, superseded bool) governance.LibraryVersion {
item := governance.LibraryVersion{
Scope: "internal", Kind: governance.LibraryVersionProductionEffective,
SnapshotID: snapshot, SourceJobID: sourceJob, CreatedAt: time.Now().UTC(),
}
if superseded {
at := time.Now().UTC()
item.SupersededAt = &at
}
return item
}
if state := classifyBaselineVersions(nil, jobA, snap); state.Own || state.Foreign != nil {
t.Fatalf("no versions must mean neither own nor foreign: %+v", state)
}
if state := classifyBaselineVersions([]governance.LibraryVersion{effective(jobA, snap, false)}, jobA, snap); !state.Own {
t.Fatalf("same job + same snapshot must be recognized as own: %+v", state)
}
// 快照不同就不是"我登记的那份",不能当成可收尾。
if state := classifyBaselineVersions([]governance.LibraryVersion{effective(jobA, "snap-other", false)}, jobA, snap); state.Own || state.Foreign == nil {
t.Fatalf("different snapshot of same job must not be own: %+v", state)
}
// 别人的生效版本:即使快照相同也必须拒绝。
foreign := classifyBaselineVersions([]governance.LibraryVersion{effective(jobB, snap, false)}, jobA, snap)
if foreign.Own || foreign.Foreign == nil || foreign.Foreign.SourceJobID != jobB {
t.Fatalf("foreign effective version must be reported: %+v", foreign)
}
// 已被顶替的旧版本不算生效版本。
if state := classifyBaselineVersions([]governance.LibraryVersion{effective(jobB, snap, true)}, jobA, snap); state.Own || state.Foreign != nil {
t.Fatalf("superseded version must not block a baseline: %+v", state)
}
}
// 版本已登记 + 健康:走成功边界,释放两把占用并把作业标成功。
func TestSettleBaselineWindowFinishesWhenRecorded(t *testing.T) {
store := &fakeFinishStore{}
if err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
"snap-abc", true, "succeeded", nil); err != nil {
t.Fatalf("settle: %v", err)
}
if !containsCall(store.calls, "advance_succeeded") {
t.Fatalf("job must be marked succeeded: %v", store.calls)
}
if containsCall(store.states, "needs_restore") {
t.Fatalf("success path must not hold the target in maintenance: %v", store.states)
}
if !containsCall(store.calls, "scope_release") || !containsCall(store.calls, "instance_release") {
t.Fatalf("both locks must be released: %v", store.calls)
}
}
// 版本已登记但目标服务/入口没恢复:不能报成功,必须保持维护并标 needs_restore。
func TestSettleBaselineWindowHoldsWhenHealthFails(t *testing.T) {
store := &fakeFinishStore{}
err := settleBaselineWindow(context.Background(), store, fakeHealth{err: errors.New("kgseal-ingress 未就绪")},
"job-1", "internal", "kgseal-app", "snap-abc", true, "succeeded", nil)
if err == nil {
t.Fatal("unhealthy target must not report success")
}
if containsCall(store.calls, "advance_succeeded") {
t.Fatalf("must not mark succeeded while target is unhealthy: %v", store.calls)
}
if countCall(store.states, "needs_restore") != 2 {
t.Fatalf("must hold both records with needs_restore: %v", store.states)
}
}
// 没有登记、窗口仍是 held:只能保持维护(服务确实没回来),绝不能释放占用。
func TestSettleBaselineWindowKeepsHeldWindowInMaintenance(t *testing.T) {
store := &fakeFinishStore{}
err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
"snap-abc", false, "held", errors.New("停写入者后后台仍有任务"))
if err == nil || !strings.Contains(err.Error(), "保持维护") {
t.Fatalf("err=%v", err)
}
if containsCall(store.calls, "scope_release") || containsCall(store.calls, "instance_release") {
t.Fatalf("held window must not release locks: %v", store.calls)
}
if countCall(store.states, "needs_restore") != 2 {
t.Fatalf("held window must be recorded as needs_restore: %v", store.states)
}
if !containsCall(store.calls, "advance_failed") {
t.Fatalf("job must fail: %v", store.calls)
}
}
// 没有登记、但窗口已按原状态开放:状态必须跟着变成"已释放",
// 不能留下"显示维护中、实际入口已开"。
func TestSettleBaselineWindowReleasesWhenWindowRestored(t *testing.T) {
store := &fakeFinishStore{}
err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
"snap-abc", false, "failed", errors.New("捕获目标整库失败"))
if err == nil || !strings.Contains(err.Error(), "未建立基线") {
t.Fatalf("err=%v", err)
}
if !containsCall(store.calls, "scope_release") || !containsCall(store.calls, "instance_release") {
t.Fatalf("restored window must release both locks: %v", store.calls)
}
if containsCall(store.states, "needs_restore") {
t.Fatalf("released window must not be marked as needs_restore: %v", store.states)
}
if !containsCall(store.calls, "advance_failed") {
t.Fatalf("job must fail: %v", store.calls)
}
}
// 释放失败时不能在作业上写"无需恢复":那会让界面与实际占用分叉。
func TestSettleBaselineWindowReportsReleaseFailure(t *testing.T) {
store := &fakeFinishStore{failOn: "scope_release"}
err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
"snap-abc", false, "failed", nil)
if err == nil || !strings.Contains(err.Error(), "维护占用释放失败") {
t.Fatalf("err=%v", err)
}
}
func countCall(calls []string, want string) int {
count := 0
for _, call := range calls {
if call == want {
count++
}
}
return count
}
func containsCall(calls []string, want string) bool {
for _, call := range calls {
if call == want {
return true
}
}
return false
}
+69
View File
@@ -0,0 +1,69 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/libraryagent"
)
// 封存的身份契约:期望随**同一个 capture 请求**送进 agent,由接收方在操作互斥内、
// 任何源操作之前比对;worker 事后按回显的指纹再复核一次。
//
// 为什么不能在 worker 侧先 Health 再 Capture:那是两次拨号,中间 socket 可能被换成
// 另一个 agent,错的执行器已经把写入者停了才被拒登记——副作用已经发生。这里把期望
// 放进请求本身,换过的 agent 会先比对身份、比对失败就一次 run 都不调。
//
// 期望字段只来自治理侧配置(LIBRARY_SOURCE_*),绝不从执行器自报值生成。
// captureExpectation 把治理侧配置翻译成期望;缺身份就是缺配置,调用方必须失败。
func captureExpectation(config appconfig.Config) (libraryagent.Expectation, error) {
expectation := libraryagent.NewExpectation(config.SourceProject(), config.LibrarySourceApp, "")
if expectation.Project == "" {
return libraryagent.Expectation{}, errors.New("未配置封存来源实例身份(LIBRARY_SOURCE_PROJECT),拒绝在不核对执行器的情况下停写入者")
}
return expectation, nil
}
// captureExecutorPostcheck 是事后证据防线:结果里回显的期望指纹必须等于我们发出的,
// 自报身份也必须与期望一致。它挡不住副作用,所以不能替代请求内的核对。
func captureExecutorPostcheck(payload json.RawMessage, expectation libraryagent.Expectation) error {
if len(payload) == 0 {
return errors.New("封存结果里没有执行器身份,无法证明这次封存来自期望实例")
}
var document struct {
Identity struct {
Project string `json:"project"`
SourceApp string `json:"source_app"`
ExpectationFingerprint string `json:"expectation_fingerprint"`
} `json:"agent_identity"`
}
if err := json.Unmarshal(payload, &document); err != nil {
return errors.New("封存结果里的执行器身份无法解析")
}
identity := document.Identity
if identity.Project == "" {
return errors.New("封存结果里没有执行器身份,无法证明这次封存来自期望实例")
}
if identity.ExpectationFingerprint != expectation.Fingerprint {
return fmt.Errorf("封存结果回显的期望指纹不一致(agent=%q,期望=%q),该快照不能当作本实例的测试封存",
identity.ExpectationFingerprint, expectation.Fingerprint)
}
if identity.Project != expectation.Project {
return fmt.Errorf("封存结果来自另一个实例(agent=%q,期望=%q),该快照不能当作本实例的测试封存",
identity.Project, expectation.Project)
}
if expectation.SourceApp != "" && identity.SourceApp != expectation.SourceApp {
return fmt.Errorf("封存结果来自另一个源容器(agent=%q,期望=%q),该快照不能当作本实例的测试封存",
identity.SourceApp, expectation.SourceApp)
}
return nil
}
// expectationFingerprint 用于日志/证据行的稳定标识,不含路径与凭据。
func expectationFingerprint(expectation libraryagent.Expectation) string {
return strings.TrimSpace(expectation.Fingerprint)
}
@@ -0,0 +1,73 @@
package main
import (
"encoding/json"
"strings"
"testing"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/libraryagent"
)
// 期望只从治理侧配置生成;缺 project 就是缺配置,必须在停写入者之前失败。
func TestCaptureExpectationComesFromGovernanceConfigAndFailsClosed(t *testing.T) {
expect, err := captureExpectation(appconfig.Config{LibrarySourceInstance: "kgmaint-weknora", LibrarySourceApp: "kgmaint-app"})
if err != nil {
t.Fatal(err)
}
if expect.Project != "kgmaint-weknora" || expect.SourceApp != "kgmaint-app" {
t.Fatalf("expectation = %+v", expect)
}
if !expect.Complete() {
t.Fatalf("expectation should be complete: %+v", expect)
}
if _, err = captureExpectation(appconfig.Config{}); err == nil ||
!strings.Contains(err.Error(), "LIBRARY_SOURCE_PROJECT") {
t.Fatalf("empty identity accepted: %v", err)
}
// 期望不取自任何执行器应答:显式给 project 就按它绑定,配置里没有别的来源。
explicit, err := captureExpectation(appconfig.Config{LibrarySourceProject: "kgmaint-weknora"})
if err != nil || explicit.Project != "kgmaint-weknora" || explicit.SourceApp != "" {
t.Fatalf("explicit = %+v err=%v", explicit, err)
}
}
// 事后证据防线:回显的期望指纹、project、source_app 任何一项不符都不能登记。
func TestCaptureExecutorPostcheckUsesTheEchoedExpectationFingerprint(t *testing.T) {
expect := libraryagent.NewExpectation("kgmaint-weknora", "kgmaint-app", "")
good := payloadOf(t, "kgmaint-weknora", "kgmaint-app", expect.Fingerprint)
if err := captureExecutorPostcheck(good, expect); err != nil {
t.Fatalf("matching identity rejected: %v", err)
}
swapped := payloadOf(t, "weknora071full", "WKNFull-app", expect.Fingerprint)
if err := captureExecutorPostcheck(swapped, expect); err == nil || !strings.Contains(err.Error(), "另一个实例") {
t.Fatalf("wrong project accepted: %v", err)
}
regenerated := payloadOf(t, "kgmaint-weknora", "kgmaint-app",
libraryagent.NewExpectation("kgmaint-weknora", "kgmaint-app", "/other/archive").Fingerprint)
if err := captureExecutorPostcheck(regenerated, expect); err == nil || !strings.Contains(err.Error(), "期望指纹不一致") {
t.Fatalf("regenerated expectation accepted: %v", err)
}
if err := captureExecutorPostcheck(json.RawMessage(`{"snapshot_id":"abc"}`), expect); err == nil ||
!strings.Contains(err.Error(), "没有执行器身份") {
t.Fatalf("missing identity accepted: %v", err)
}
if err := captureExecutorPostcheck(payloadOf(t, "kgmaint-weknora", "kgseal-app", expect.Fingerprint), expect); err == nil ||
!strings.Contains(err.Error(), "另一个源容器") {
t.Fatalf("wrong source app accepted: %v", err)
}
}
func payloadOf(t *testing.T, project, sourceApp, fingerprint string) json.RawMessage {
t.Helper()
encoded, err := json.Marshal(map[string]any{
"snapshot_id": "abc",
"agent_identity": map[string]any{
"project": project, "source_app": sourceApp, "expectation_fingerprint": fingerprint,
},
})
if err != nil {
t.Fatal(err)
}
return encoded
}
+52
View File
@@ -0,0 +1,52 @@
package main
import (
"context"
"log"
"time"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/libraryexecutor"
postgresstore "knowledge-governance/internal/platform/postgres"
)
// 为什么由 worker 观测、页面读库:agent socket 只挂载在 worker 容器里,API 那个
// LIBRARY_AGENT_SOCKET 即使配了也 dial 不到。谁真能执行就由谁写下观测,
// 否则页面上的"可发起"只是配置字符串的回声。
const libraryExecutorProbeInterval = 30 * time.Second
// libraryExecutorProbeTimeout 是一次 health 往返的上限:它只是同 uid unix socket
// 上的一次 JSON 往返,慢到秒级就说明执行器有问题,不该拖住心跳。
const libraryExecutorProbeTimeout = 3 * time.Second
func recordLibraryExecutorStatus(ctx context.Context, store *postgresstore.Store, config appconfig.Config) {
// 归属用**封存源**实例:agent 冻结的是源库写入者,和同步目标(kgseal)不是一套。
instance := config.SourceInstance()
if instance == "" {
// 没有实例身份,观测就无处归属:写进去只会让页面拿别的实例的状态。
log.Print("library executor probe skipped: LIBRARY_SOURCE_INSTANCE is not configured")
return
}
observation := libraryexecutor.Probe(ctx, instance, config.LibraryAgentSocket, libraryExecutorProbeTimeout, nil)
if err := store.UpsertLibraryExecutorStatus(ctx, observation, config.WorkerID, time.Now().UTC()); err != nil {
log.Printf("library executor status not recorded: %v", err)
}
}
// runLibraryExecutorProbe 先观测一次再进循环:页面在 worker 刚起来时就该有真实状态。
func runLibraryExecutorProbe(ctx context.Context, store *postgresstore.Store, config appconfig.Config, interval time.Duration) {
if interval <= 0 {
interval = libraryExecutorProbeInterval
}
recordLibraryExecutorStatus(ctx, store, config)
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
recordLibraryExecutorStatus(ctx, store, config)
}
}
}
+162
View File
@@ -0,0 +1,162 @@
package main
import (
"context"
"errors"
"fmt"
"time"
"knowledge-governance/internal/governance"
postgresstore "knowledge-governance/internal/platform/postgres"
)
// libraryMaintenanceStore 是"成功收尾"用到的存储能力。抽成接口是为了能在测试里
// 让某一步失败,验证"没全部完成就不许暴露成功"。
type libraryMaintenanceStore interface {
SetInstanceMaintenanceIngress(ctx context.Context, instance, ownerJobID, state, note string, now time.Time) error
SetLibraryMaintenanceIngress(ctx context.Context, scope, ownerJobID, state, note string, now time.Time) error
UpdateInstanceMaintenanceState(ctx context.Context, instance, ownerJobID, state, note string, now time.Time) error
UpdateLibraryMaintenanceState(ctx context.Context, scope, ownerJobID, state, note string, now time.Time) error
ReleaseInstanceMaintenance(ctx context.Context, instance, ownerJobID string, now time.Time) error
ReleaseLibraryMaintenance(ctx context.Context, scope, ownerJobID string, now time.Time) error
AdvanceLibraryJob(ctx context.Context, jobID string, state governance.LibraryJobState, stage, message, recovery, snapshotID, targetSnapshotID string, now time.Time) error
}
// releaseWindowLocks 是唯一释放路径。
//
// 顺序刻意是"scope 先、instance 后":instance 行是跨 scope 的互斥来源,最后放它
// 才能保证任何新作业在全部记录落定之前都进不来。每一步的错误都必须向上暴露——
// 只丢日志会让"锁其实没放"这件事变成页面上的成功。
func releaseWindowLocks(ctx context.Context, store libraryMaintenanceStore, instance, scope, jobID, note string, now time.Time) error {
if err := store.SetLibraryMaintenanceIngress(ctx, scope, jobID, "open", note, now); !releasedOrNil(err) {
return fmt.Errorf("记录入口已开放失败(scope):%w", err)
}
if err := store.SetInstanceMaintenanceIngress(ctx, instance, jobID, "open", note, now); !releasedOrNil(err) {
return fmt.Errorf("记录入口已开放失败(instance):%w", err)
}
if err := store.UpdateLibraryMaintenanceState(ctx, scope, jobID, "verified", note, now); !releasedOrNil(err) {
return fmt.Errorf("维护状态写入失败(scope):%w", err)
}
if err := store.ReleaseLibraryMaintenance(ctx, scope, jobID, now); !releasedOrNil(err) {
return fmt.Errorf("维护占用释放失败(scope):%w", err)
}
if err := store.UpdateInstanceMaintenanceState(ctx, instance, jobID, "verified", note, now); !releasedOrNil(err) {
return fmt.Errorf("维护状态写入失败(instance):%w", err)
}
if err := store.ReleaseInstanceMaintenance(ctx, instance, jobID, now); !releasedOrNil(err) {
return fmt.Errorf("实例占用释放失败(instance):%w", err)
}
return nil
}
// healthChecker 是"目标写入者与入口是否真的恢复健康"的判定口(测试可注入失败)。
type healthChecker interface {
VerifyRunning(ctx context.Context) error
}
// successBoundary 描述一次"内容已通过验证与版本登记,但还没到可以对外报成功"的作业。
type successBoundary struct {
Ready bool
Backup string
Target string
FinishedAt time.Time
}
// finishLibrarySuccess 是唯一的成功收尾入口:只有以下全部成立才暴露 succeeded——
// 目标服务健康、入口开放、scope 与 instance 两把维护锁都释放完成、终态写入成功。
//
// 为什么不能先标成功再释放:恢复/开入口/放锁都可能失败,先写 succeeded 会让外部
// (页面、API、重启后的接管逻辑)看到一条"成功了但目标仍在维护"的作业;而
// InterruptedLibraryJobs 只接管非终态,终态作业不会再被接管,锁就永远挂着。
func finishLibrarySuccess(ctx context.Context, store libraryMaintenanceStore, health healthChecker,
instance, scope, jobID string, boundary successBoundary) error {
// 释放与终态写入必须能在 ctx 取消后继续:停机不该留下"锁放了但作业还在跑"或
// 反过来"作业成功了锁还挂着"的任一半成品。
bounded, cancel := context.WithTimeout(context.WithoutCancel(ctx), 60*time.Second)
defer cancel()
holdForRepair := func(reason string) error {
note := "内容已验证,但收尾未完成,目标保持维护:" + reason
_ = store.UpdateLibraryMaintenanceState(bounded, scope, jobID, "needs_restore", note, time.Now().UTC())
_ = store.UpdateInstanceMaintenanceState(bounded, instance, jobID, "needs_restore", note, time.Now().UTC())
_ = store.AdvanceLibraryJob(bounded, jobID, governance.LibraryJobFailed, "finalize", note,
"生产内容已写入并校验;维护保持,按预备份或人工核验后再开放", boundary.Backup, boundary.Target, time.Now().UTC())
return errors.New(note)
}
if err := health.VerifyRunning(bounded); err != nil {
return holdForRepair("目标服务或入口未恢复健康:" + err.Error())
}
if err := releaseWindowLocks(bounded, store, instance, scope, jobID,
"写入与校验完成,目标已按原运行状态恢复", time.Now().UTC()); err != nil {
return holdForRepair(err.Error())
}
finishedAt := time.Now().UTC()
if err := store.AdvanceLibraryJob(bounded, jobID, governance.LibraryJobSucceeded, "finalize", "", "",
boundary.Backup, boundary.Target, finishedAt); err != nil {
// 锁已释放但终态没写下去:作业保持非终态,重启后会被接管(接管走
// finalize-only,只复核与登记,不重写生产)。
return fmt.Errorf("维护已释放但作业终态写入失败(等待接管):%w", err)
}
return nil
}
// takeoverKind 是接管时"这个作业走到哪一步"的分类。
//
// 四类必须分开:把"版本查不到"当成"没有收尾证据",就会把只差收尾的作业重新写一遍
// 目标;把"曾被顶替"当成正常重放,就会用旧快照覆盖更新的生产版本。
type takeoverKind string
const (
// takeoverFresh:没有任何收尾证据,按正常流程(有预备份则复用,写前仍备份)。
takeoverFresh takeoverKind = "fresh"
// takeoverPendingFinal:内容与版本已落定,只差收尾(只复核与登记,不重写)。
takeoverPendingFinal takeoverKind = "pending_final"
// takeoverSuperseded:本快照曾经生效、后来被别的快照顶替,不得自动覆盖。
takeoverSuperseded takeoverKind = "superseded"
// takeoverUnknown:版本状态读不到且没有阶段证据,属"未知",不得写入。
takeoverUnknown takeoverKind = "unknown"
)
// classifyTakeover 只按证据分类:
// - finalize 阶段成功是独立证据(来自作业自己的阶段行),不因版本查询失败而丢弃;
// - 本快照存在 released/被顶替的版本 ⇒ superseded;
// - 本快照存在未被顶掉的生效版本 ⇒ pending_final;
// - 查询失败且无阶段证据 ⇒ unknown(可重试,不当成没有证据)。
func classifyTakeover(job governance.LibraryJob, versions []governance.LibraryVersion, versionsErr error) takeoverKind {
finalized := false
for _, stage := range job.Stages {
if stage.Stage == "finalize" && stage.State == "succeeded" {
finalized = true
}
}
if versionsErr != nil && !finalized {
return takeoverUnknown
}
superseded, effective := false, false
for _, version := range versions {
if version.Kind != governance.LibraryVersionProductionEffective || version.SnapshotID != job.TargetSnapshotID {
continue
}
if version.SupersededAt != nil {
superseded = true
} else {
effective = true
}
}
if superseded && !effective {
return takeoverSuperseded
}
if effective || finalized {
return takeoverPendingFinal
}
return takeoverFresh
}
var _ libraryMaintenanceStore = (*postgresstore.Store)(nil)
// releasedOrNil:这一行已经由本作业释放过,说明该步在上一次尝试里已经完成。
// 部分成功后重试幂等靠它——否则"scope 放了、instance 没放"的作业永远重试不下去。
func releasedOrNil(err error) bool {
return err == nil || errors.Is(err, postgresstore.ErrMaintenanceReleased)
}
+213
View File
@@ -0,0 +1,213 @@
package main
import (
"context"
"errors"
"strings"
"testing"
"time"
"knowledge-governance/internal/governance"
postgresstore "knowledge-governance/internal/platform/postgres"
)
type fakeFinishStore struct {
calls []string
failOn string
alreadyReleased bool
advanced []governance.LibraryJobState
// states 记录所有维护状态写入(测试要区分 verified 与 needs_restore)。
states []string
}
// 模拟"这一行已由同一个作业释放过"(真实 Store 返回 ErrMaintenanceReleased)。
func (f *fakeFinishStore) releasedSentinel() error {
if f.alreadyReleased {
return postgresstore.ErrMaintenanceReleased
}
return nil
}
func (f *fakeFinishStore) record(name string) error {
f.calls = append(f.calls, name)
if f.failOn == name {
return errors.New("injected " + name + " failure")
}
return nil
}
func (f *fakeFinishStore) SetInstanceMaintenanceIngress(context.Context, string, string, string, string, time.Time) error {
if err := f.releasedSentinel(); err != nil {
return err
}
return f.record("instance_ingress")
}
func (f *fakeFinishStore) SetLibraryMaintenanceIngress(context.Context, string, string, string, string, time.Time) error {
if err := f.releasedSentinel(); err != nil {
return err
}
return f.record("scope_ingress")
}
func (f *fakeFinishStore) UpdateInstanceMaintenanceState(_ context.Context, _, _, state, _ string, _ time.Time) error {
if err := f.releasedSentinel(); err != nil {
return err
}
f.states = append(f.states, state)
return f.record("instance_state")
}
func (f *fakeFinishStore) UpdateLibraryMaintenanceState(_ context.Context, _, _, state, _ string, _ time.Time) error {
if err := f.releasedSentinel(); err != nil {
return err
}
f.states = append(f.states, state)
return f.record("scope_state")
}
func (f *fakeFinishStore) ReleaseInstanceMaintenance(context.Context, string, string, time.Time) error {
if err := f.releasedSentinel(); err != nil {
return err
}
return f.record("instance_release")
}
func (f *fakeFinishStore) ReleaseLibraryMaintenance(context.Context, string, string, time.Time) error {
if err := f.releasedSentinel(); err != nil {
return err
}
return f.record("scope_release")
}
func (f *fakeFinishStore) AdvanceLibraryJob(_ context.Context, _ string, state governance.LibraryJobState,
_, _, _, _, _ string, _ time.Time) error {
f.advanced = append(f.advanced, state)
return f.record("advance_" + string(state))
}
type fakeHealth struct{ err error }
func (h fakeHealth) VerifyRunning(context.Context) error { return h.err }
func boundary() successBoundary {
return successBoundary{Ready: true, Backup: "backup-1", Target: "snap-1", FinishedAt: time.Now().UTC()}
}
// 首次观察到成功时,两把维护锁必须已经释放,且终态写入排在最后。
func TestFinishLibrarySuccessReleasesEveryLockBeforeSucceeded(t *testing.T) {
store := &fakeFinishStore{}
if err := finishLibrarySuccess(context.Background(), store, fakeHealth{}, "kgseal-weknora", "internal", "job-1", boundary()); err != nil {
t.Fatal(err)
}
want := []string{"scope_ingress", "instance_ingress", "scope_state", "scope_release",
"instance_state", "instance_release", "advance_succeeded"}
if strings.Join(store.calls, ",") != strings.Join(want, ",") {
t.Fatalf("call order = %v", store.calls)
}
if len(store.advanced) != 1 || store.advanced[0] != governance.LibraryJobSucceeded {
t.Fatalf("advanced = %v", store.advanced)
}
}
// 目标服务/入口没恢复健康:不许放锁,不许报成功,作业落成失败并保持维护。
func TestFinishLibrarySuccessKeepsMaintenanceWhenTargetIsNotHealthy(t *testing.T) {
store := &fakeFinishStore{}
err := finishLibrarySuccess(context.Background(), store, fakeHealth{err: errors.New("app not healthy")},
"kgseal-weknora", "internal", "job-2", boundary())
if err == nil || !strings.Contains(err.Error(), "未恢复健康") {
t.Fatalf("err = %v", err)
}
for _, call := range store.calls {
if strings.HasSuffix(call, "_release") || call == "advance_succeeded" {
t.Fatalf("released or reported success despite unhealthy target: %v", store.calls)
}
}
if len(store.advanced) != 1 || store.advanced[0] != governance.LibraryJobFailed {
t.Fatalf("advanced = %v", store.advanced)
}
}
// 任一步释放失败:不报成功,作业失败,维护状态写 needs_restore(等人工/接管)。
func TestFinishLibrarySuccessStopsOnReleaseFailure(t *testing.T) {
for _, step := range []string{"scope_release", "instance_state", "instance_release"} {
t.Run(step, func(t *testing.T) {
store := &fakeFinishStore{failOn: step}
err := finishLibrarySuccess(context.Background(), store, fakeHealth{}, "kgseal-weknora", "internal", "job-3", boundary())
if err == nil {
t.Fatalf("%s failure was swallowed", step)
}
for _, call := range store.calls {
if call == "advance_succeeded" {
t.Fatalf("reported success after %s failed: %v", step, store.calls)
}
}
if len(store.advanced) != 1 || store.advanced[0] != governance.LibraryJobFailed {
t.Fatalf("advanced = %v", store.advanced)
}
})
}
}
// 终态写入失败:锁已释放,但不能声称成功;作业保持非终态等接管。
func TestFinishLibrarySuccessReportsTerminalWriteFailure(t *testing.T) {
store := &fakeFinishStore{failOn: "advance_succeeded"}
err := finishLibrarySuccess(context.Background(), store, fakeHealth{}, "kgseal-weknora", "internal", "job-4", boundary())
if err == nil || !strings.Contains(err.Error(), "等待接管") {
t.Fatalf("err = %v", err)
}
}
// 接管分类:阶段证据独立成立;查询失败=未知(不得写入);被顶替=不得覆盖。
func TestClassifyTakeoverSeparatesUnknownPendingSupersededAndFresh(t *testing.T) {
job := governance.LibraryJob{Scope: "internal", Kind: governance.LibraryJobSync, TargetSnapshotID: "snap-1"}
finalized := job
finalized.Stages = []governance.LibraryStage{{Stage: "finalize", State: "succeeded"}}
effective := []governance.LibraryVersion{{Kind: governance.LibraryVersionProductionEffective, SnapshotID: "snap-1"}}
supersededVersion := []governance.LibraryVersion{{Kind: governance.LibraryVersionProductionEffective,
SnapshotID: "snap-1", SupersededAt: ptrTime(time.Now())}}
unrelated := []governance.LibraryVersion{{Kind: governance.LibraryVersionProductionEffective, SnapshotID: "snap-9"}}
queryErr := errors.New("version storage temporarily unavailable")
cases := []struct {
name string
job governance.LibraryJob
versions []governance.LibraryVersion
err error
want takeoverKind
}{
// 关键回归:查询失败 + 没有阶段证据 ⇒ 未知,不能当成"没有收尾证据"继续写。
{"query fails without stage evidence", job, nil, queryErr, takeoverUnknown},
// 阶段证据先独立成立:查询失败不影响"只差收尾"的判定。
{"query fails but finalize stage succeeded", finalized, nil, queryErr, takeoverPendingFinal},
{"effective version for this snapshot", job, effective, nil, takeoverPendingFinal},
{"finalize stage succeeded", finalized, nil, nil, takeoverPendingFinal},
{"this snapshot was superseded", job, supersededVersion, nil, takeoverSuperseded},
{"no evidence at all", job, nil, nil, takeoverFresh},
{"another snapshot effective but this one has no row", job, unrelated, nil, takeoverFresh},
{"empty target snapshot", governance.LibraryJob{Scope: "internal"}, nil, nil, takeoverFresh},
}
for _, test := range cases {
t.Run(test.name, func(t *testing.T) {
if got := classifyTakeover(test.job, test.versions, test.err); got != test.want {
t.Fatalf("classify = %q, want %q", got, test.want)
}
})
}
}
// 部分释放后重试必须幂等:已由本作业释放的行不能再算失败。
func TestReleaseWindowLocksToleratesAlreadyReleasedRows(t *testing.T) {
store := &fakeFinishStore{}
if err := releaseWindowLocks(context.Background(), store, "kgseal-weknora", "internal", "job-1",
"verified", time.Now().UTC()); err != nil {
t.Fatal(err)
}
store = &fakeFinishStore{alreadyReleased: true}
if err := releaseWindowLocks(context.Background(), store, "kgseal-weknora", "internal", "job-1",
"verified", time.Now().UTC()); err != nil {
t.Fatalf("retry after a partial release must be idempotent: %v", err)
}
// 真正"不是本作业持有"的错误仍然要暴露。
store = &fakeFinishStore{failOn: "instance_release"}
if err := releaseWindowLocks(context.Background(), store, "kgseal-weknora", "internal", "job-1",
"verified", time.Now().UTC()); err == nil {
t.Fatal("a foreign owner error was swallowed")
}
}
func ptrTime(value time.Time) *time.Time { return &value }
+45 -53
View File
@@ -16,6 +16,7 @@ import (
"knowledge-governance/internal/librarymaintenance"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
"knowledge-governance/internal/pgdsn"
postgresstore "knowledge-governance/internal/platform/postgres"
)
@@ -65,32 +66,8 @@ type libraryReleasePayload struct {
FinalizeOnly bool `json:"finalize_only,omitempty"`
}
// openLibraryMaintenance 按部署注入的配置装配实例级维护目标。
// 未接线时返回明确原因:只锁治理库 scope 不算维护,宁可拒绝写也不冒充。
func openLibraryMaintenance(config appconfig.Config) (*librarymaintenance.Manager, error) {
database, err := librarymaintenance.DatabaseIdentity(config.LibraryProductionDatabaseURL)
if err != nil {
return nil, err
}
return librarymaintenance.New(librarymaintenance.Config{
Instance: config.LibraryTargetInstance,
Project: config.LibraryTargetProject,
Writers: config.LibraryTargetWriters,
NonWriters: config.LibraryTargetNonWriters,
DockerSocket: config.LibraryTargetDockerSocket,
IngressURL: config.LibraryTargetIngressURL,
IngressService: config.LibraryTargetIngressService,
IngressDrainFile: config.LibraryTargetIngressDrainFile,
Database: database,
Index: config.LibraryProductionIndex,
Bucket: config.LibraryProductionS3Bucket,
// 后台静默检查的运行时依赖:HTTP 排空不等于后台任务排空。
DatabaseURL: config.LibraryProductionDatabaseURL,
RedisAddr: config.LibraryTargetRedisAddr,
RedisPassword: config.LibraryTargetRedisPassword,
RedisDB: config.LibraryTargetRedisDB,
})
}
// openLibraryMaintenance 定义在 library_wiring.go:维护目标的装配与目标整库栈共用
// 同一个包(internal/librarystack),不在这里再拼一份。
// runLibraryRelease 执行"闸口 → 授权 → 关闭目标入口并停止写入者 → 预备份 →
// 四类写入 → 校验 → 登记权威版本 → 恢复原运行状态和入口"的完整链路。
@@ -116,6 +93,21 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
// 绝不重新备份——重新备份会把"半写状态"当成新基线,覆盖真正可回滚的那份。
previous, err := store.LibraryJob(ctx, payload.LibraryJobID)
resumed := err == nil && previous.SnapshotID != ""
if resumed {
versions, versionErr := store.LibraryVersions(ctx, payload.Scope, 50)
switch classifyTakeover(previous, versions, versionErr) {
case takeoverUnknown:
// 版本状态读不到、又没有阶段证据:不能当成"没有收尾证据"继续写目标。
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
"接管时读不到该目标的版本状态,无法判断是只差收尾还是要重做;未对目标执行任何写入,稍后重试", "prebackup")
case takeoverSuperseded:
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
"本快照的生效版本已被别的快照顶替,拒绝用旧快照覆盖当前生产版本", "prebackup")
case takeoverPendingFinal:
// 上次可能死在"锁已放、终态没写"之间:只复核与登记,不重写生产。
payload.FinalizeOnly = true
}
}
if resumed {
recorder.Stage(ctx, "prebackup", "skipped", "接管失败前已创建的预备份 "+previous.SnapshotID)
}
@@ -194,7 +186,7 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
"目标拓扑不满足唯一入口约束,拒绝开维护窗口:"+topoErr.Error(), "prebackup")
}
targetIdentity := fmt.Sprintf("scope=%s db=%s index=%s bucket=%s", payload.Scope,
redactDSN(config.LibraryProductionDatabaseURL), config.LibraryProductionIndex, config.LibraryProductionS3Bucket)
pgdsn.Endpoint(config.LibraryProductionDatabaseURL), config.LibraryProductionIndex, config.LibraryProductionS3Bucket)
originalState, stateErr := manager.OriginalState(ctx)
if stateErr != nil {
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
@@ -256,6 +248,7 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
if resumed {
contentState = contentHalfWritten
}
pending := successBoundary{}
tracked := &contentTrackedTarget{inner: target, onApply: func() { contentState = contentHalfWritten }}
allowRestore := func(error) bool {
return contentState == contentUntouched || contentState == contentSucceeded || contentState == contentRolledBack
@@ -290,8 +283,9 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
return errors.New(message)
}
recorder.Stage(ctx, "finalize", "succeeded", "已登记生效版本(仅收尾,未重写生产)")
_ = store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobSucceeded, "finalize", "", "",
previous.SnapshotID, payload.TargetSnapshotID, time.Now().UTC())
// 成功要等维护锁释放完才暴露:这里只记"待收尾"。
pending = successBoundary{Ready: true, Backup: previous.SnapshotID,
Target: payload.TargetSnapshotID, FinishedAt: time.Now().UTC()}
return nil
}
@@ -340,13 +334,10 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
return errors.New(message)
}
recorder.Stage(ctx, "finalize", "succeeded", "已登记生效版本")
if advanceErr := store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobSucceeded, "finalize", "", "",
outcome.Backup.SnapshotID, payload.TargetSnapshotID, finishedAt); advanceErr != nil {
// 状态落不下去时不要把队列作业报成功:让上层可见并等待接管。
return fmt.Errorf("作业状态写入失败: %w", advanceErr)
}
log.Printf("library %s succeeded job=%s scope=%s snapshot=%s backup=%s", kind,
payload.LibraryJobID, payload.Scope, payload.TargetSnapshotID, outcome.Backup.SnapshotID)
// 内容与版本都落定了,但还不能报成功:服务健康、入口开放、两把维护锁
// 释放完成才允许暴露 succeeded(见 finishLibrarySuccess)。
pending = successBoundary{Ready: true, Backup: outcome.Backup.SnapshotID,
Target: payload.TargetSnapshotID, FinishedAt: finishedAt}
return nil
case "rolled_back":
contentState = contentRolledBack
@@ -407,10 +398,19 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
}
return windowErr
case contentState == contentSucceeded:
if err := releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
"写入与校验完成,目标已按原运行状态恢复", now2); err != nil {
return fmt.Errorf("目标已恢复但维护占用释放失败: %w", err)
if !pending.Ready {
// 内容状态说成功但没有收尾凭据(不可能两者都缺):按未收尾处理,保持维护。
note := "内容状态为已验证,但缺少收尾凭据,目标保持维护"
_ = store.UpdateLibraryMaintenanceState(ctx, payload.Scope, payload.LibraryJobID, "needs_restore", note, now2)
_ = store.UpdateInstanceMaintenanceState(ctx, identity.Instance, payload.LibraryJobID, "needs_restore", note, now2)
return errors.New(note)
}
if finishErr := finishLibrarySuccess(ctx, store, manager, identity.Instance, payload.Scope,
payload.LibraryJobID, pending); finishErr != nil {
return finishErr
}
log.Printf("library %s succeeded job=%s scope=%s snapshot=%s backup=%s", kind,
payload.LibraryJobID, payload.Scope, payload.TargetSnapshotID, pending.Backup)
return windowErr
case contentState == contentRolledBack:
if err := releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
@@ -428,19 +428,6 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
}
}
// releaseWindowLocks 是唯一释放路径:先记录入口已恢复,再放 scope 行与实例锁。
// 顺序反过来会出现"实例锁放了、scope 还挂着"的中间态,页面就会说正在维护。
func releaseWindowLocks(ctx context.Context, store *postgresstore.Store, instance, scope, jobID, note string, now time.Time) error {
_ = store.SetInstanceMaintenanceIngress(ctx, instance, jobID, "open", note, now)
_ = store.SetLibraryMaintenanceIngress(ctx, scope, jobID, "open", note, now)
_ = store.UpdateInstanceMaintenanceState(ctx, instance, jobID, "verified", note, now)
_ = store.UpdateLibraryMaintenanceState(ctx, scope, jobID, "verified", note, now)
if err := store.ReleaseInstanceMaintenance(ctx, instance, jobID, now); err != nil {
return err
}
return store.ReleaseLibraryMaintenance(ctx, scope, jobID, now)
}
// resumeInterruptedLibraryJobs 在 worker 启动时接管中断的写作业:
// 已经持久化预备份的重新排队(同一个业务作业 id,阶段与备份都接着用),
// 连预备份都没完成的才判失败——那种情况没有可回滚的基线,不能盲写。
@@ -469,8 +456,13 @@ func resumeInterruptedLibraryJobs(ctx context.Context, store *postgresstore.Stor
continue
}
queueType := librarySyncJobType
if job.Kind == governance.LibraryJobRestore {
switch job.Kind {
case governance.LibraryJobRestore:
queueType = libraryRestoreJobType
case governance.LibraryJobBaseline:
// 基线作业被杀时窗口可能还停着目标:用同一个作业 id 接管那个窗口,
// 继续捕获/校验,而不是留下一个 held 目标等人工处理。
queueType = libraryBaselineJobType
}
payload, _ := json.Marshal(map[string]string{"library_job_id": job.ID, "scope": job.Scope,
"target_snapshot_id": job.TargetSnapshotID})
+19 -106
View File
@@ -2,117 +2,30 @@ package main
import (
"context"
"errors"
"fmt"
"strings"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/librarysnapshot"
"knowledge-governance/internal/librarymaintenance"
"knowledge-governance/internal/librarystack"
v071 "knowledge-governance/internal/releaseadapter/v071"
)
// buildLibraryProductionStack 用部署注入的配置装配生产整库目标与独立预备份存储。
// 任何一项缺失都返回明确原因,调用方据此把作业失败并把缺项写进阶段,
// 不用"未配置"的占位对象假装能跑。
// 目标整库读写栈的装配只有一份实现(internal/librarystack):worker 的同步/恢复/
// 基线与主机侧恢复演练共用它,避免"目标是谁、用哪份契约"随调用路径漂移。
// openLibraryMaintenance 按部署注入的配置装配实例级维护目标。
// 未接线时返回明确原因:只锁治理库 scope 不算维护,宁可拒绝写也不冒充。
func openLibraryMaintenance(config appconfig.Config) (*librarymaintenance.Manager, error) {
return librarystack.OpenMaintenance(config)
}
// buildLibraryProductionStack 用"要同步的这份快照"做契约装配目标与预备份存储。
func buildLibraryProductionStack(ctx context.Context, config appconfig.Config, scope, snapshotID, knownMapping string) (
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
missing := make([]string, 0, 4)
if config.LibraryProductionDatabaseURL == "" {
missing = append(missing, "LIBRARY_PRODUCTION_DATABASE_URL")
}
if config.LibraryProductionElasticURL == "" {
missing = append(missing, "LIBRARY_PRODUCTION_ELASTIC_URL")
}
if config.LibraryProductionTenantID == 0 {
missing = append(missing, "LIBRARY_PRODUCTION_TENANT_ID")
}
kb := config.LibraryProductionInternalKB
if scope == "external" {
kb = config.LibraryProductionExternalKB
}
if kb == "" {
missing = append(missing, "LIBRARY_PRODUCTION_KB_"+strings.ToUpper(scope))
}
if config.LibraryProductionS3Endpoint == "" || config.LibraryProductionS3Bucket == "" ||
config.LibraryProductionS3AccessKey == "" || config.LibraryProductionS3SecretKey == "" {
missing = append(missing, "LIBRARY_PRODUCTION_S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY")
}
if len(missing) > 0 {
return nil, nil, fmt.Errorf("生产整库目标未配置:缺少 %s", strings.Join(missing, "、"))
}
secure := strings.HasPrefix(config.LibraryProductionS3Endpoint, "https://")
objects, err := v071.NewS3ObjectStore(v071.S3ObjectStoreConfig{
Provider: config.LibraryProductionS3Provider, Endpoint: config.LibraryProductionS3Endpoint,
Region: config.LibraryProductionS3Region, Bucket: config.LibraryProductionS3Bucket,
AccessKeyID: config.LibraryProductionS3AccessKey, SecretAccessKey: config.LibraryProductionS3SecretKey,
Secure: secure, ForcePathStyle: true, MaxObjectBytes: 256 * 1024 * 1024,
})
if err != nil {
return nil, nil, errors.New("生产对象存储配置无效")
}
identity := v071.ProductionScope{TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: kb}
target, err := v071.NewProductionLibraryTarget(ctx, v071.ProductionLibraryConfig{
Postgres: config.LibraryProductionDatabaseURL, SearchURL: config.LibraryProductionElasticURL,
SearchIndex: config.LibraryProductionIndex, Objects: objects, ObjectsReader: objects,
ArchiveRoot: config.LibraryArchiveRoot,
ObjectsBucket: config.LibraryProductionS3Bucket,
MaxDocuments: 200000,
}, identity)
if err != nil {
return nil, nil, err
}
// 契约以"要同步的这份快照"为准:生产必须向它看齐,而不是反过来。
// 生产为空时索引可能根本不存在,这时不能拿生产映射当契约(会永远启动不了)。
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
if err != nil {
target.Close()
return nil, nil, fmt.Errorf("归档不可用,无法读取快照契约: %w", err)
}
manifest, err := archive.Load(snapshotID)
archive.Close()
if err != nil {
target.Close()
return nil, nil, fmt.Errorf("读取目标快照失败: %w", err)
}
contract := manifest.Compatibility
if contract.EmbeddingDimension == 0 {
contract.EmbeddingModel = config.LibraryEmbeddingModel
contract.EmbeddingDimension = config.LibraryEmbeddingDimension
}
// 生产索引存在时必须与快照契约一致;不存在或为空基线时跳过比对
// (空索引的映射没有可比性,非空之后仍然严格比对)。
if mapping, mapErr := target.Search().MappingSHA256(ctx); mapErr == nil {
// 映射基线:优先用"上一次生产生效版本登记的实际映射",
// 它才是这份生产索引的既有事实;快照契约只管"来自哪份数据"。
baseline := knownMapping
if baseline == "" {
// 没有历史基线时(首次接管这套生产索引),把当前实际映射当作基线并随
// 预备份登记;之后任何改动都会与登记值不符而被拒绝。
baseline = mapping
}
if baseline != "" && mapping != baseline {
current, currentErr := target.Current(ctx)
emptyBaseline := currentErr == nil && current.Counts["index"] == 0 && current.Counts["chunks"] == 0
if !emptyBaseline {
target.Close()
return nil, nil, errors.New("生产索引映射与上次生效版本不一致(索引被改动过),先对齐索引映射")
}
}
}
// 预备份描述的是"生产现在的索引",不是"快照的来源索引":
// 生产映射基线优先用上次生效版本登记的实际值。
backupContract := contract
if knownMapping != "" {
backupContract.IndexMappingSHA256 = knownMapping
}
backups, err := v071.NewProductionBackupStore(v071.ProductionBackupConfig{
Pool: target.Pool(), Search: target.Search(), Objects: target.Objects(),
ArchiveRoot: config.LibraryArchiveRoot, Scope: scope, Identity: target.Identity(),
Compatibility: backupContract, MaxDocuments: 200000, MaxObjectByte: 256 * 1024 * 1024,
})
if err != nil {
target.Close()
return nil, nil, err
}
return target, backups, nil
return librarystack.OpenProductionStack(ctx, config, scope, snapshotID, knownMapping)
}
// buildLibraryBaselineStack 装配"给目标自己建基线"所需的目标与捕获器(契约取目标当前映射)。
func buildLibraryBaselineStack(ctx context.Context, config appconfig.Config, scope string) (
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
return librarystack.OpenBaselineStack(ctx, config, scope)
}
+29 -18
View File
@@ -8,7 +8,6 @@ import (
"os"
"os/signal"
"path/filepath"
"strings"
"syscall"
"time"
@@ -58,21 +57,6 @@ func (r libraryStageRecorder) Stage(ctx context.Context, stage, state, detail st
func int64Ptr(value int64) *int64 { return &value }
// redactDSN 只保留库名,避免把连接串写进维护记录。
func redactDSN(dsn string) string {
if dsn == "" {
return ""
}
if index := strings.LastIndex(dsn, "/"); index >= 0 && index+1 < len(dsn) {
name := dsn[index+1:]
if cut := strings.Index(name, "?"); cut >= 0 {
name = name[:cut]
}
return name
}
return "configured"
}
// stageSize 是某一类数据在快照里的真实总量(行数/字节)。
type stageSize struct {
count int64
@@ -109,6 +93,9 @@ const libraryRecoverJobType = "library.recover"
// libraryRestoreJobType 是按指定整库版本恢复的队列类型;与同步分开、授权也分开。
const libraryRestoreJobType = "library.restore"
// libraryBaselineJobType 是"建立目标权威基线"的队列类型(与 API 侧常量同值)。
const libraryBaselineJobType = "library.baseline"
func main() {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
@@ -205,6 +192,8 @@ func run(ctx context.Context, getenv func(string) string) error {
go runRecordsSyncScheduler(ctx, mirrorStore, func(error) { log.Print("QA local mirror scheduler unavailable; last committed data retained") })
}
go weeklyMailLoop(ctx, pool, getenv)
// 封存执行器状态:worker 是唯一能连 agent socket 的一方,它写观测、页面读观测。
go runLibraryExecutorProbe(ctx, store, config, libraryExecutorProbeInterval)
go runSourceRefreshScheduler(ctx, store, config.SourceRefreshSchedulerInterval, func(err error) {
log.Printf("worker %s source refresh scheduler error: %v", config.WorkerID, err)
})
@@ -284,7 +273,16 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", "dead letter release storage unavailable", "", "", "", time.Now().UTC())
return fmt.Errorf("load dead letter releases: %w", releaseErr)
}
result, err := libraryagent.CaptureReleased(ctx, agentConfig, payload.Scope, payload.CaptureJob, released)
// 期望执行器身份随请求一起送进 agent,由接收方在操作互斥内、任何源操作之前
// 比对:这样即使 socket 在这之前被换掉,错的执行器也一次 run 都不会跑。
expectation, expectErr := captureExpectation(config)
if expectErr != nil {
_ = store.UpdateLibraryStage(ctx, libraryJobID, "export", "failed", nil, nil, nil, nil, expectErr.Error(), time.Now().UTC())
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", expectErr.Error(),
"未停任何写入者;补齐来源身份配置后重试", "", "", time.Now().UTC())
return expectErr
}
result, err := libraryagent.CaptureReleased(ctx, agentConfig, payload.Scope, payload.CaptureJob, released, expectation)
finishedAt := time.Now().UTC()
if err != nil {
// 导出失败时阶段与作业都要留痕;恢复责任在 agent,不会因为作业失败而假装来源已恢复。
@@ -294,13 +292,22 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
log.Printf("library capture failed scope=%s capture_job=%s: %v", payload.Scope, payload.CaptureJob, err)
return err
}
// 执行后复核:结果里 agent 自报的身份必须还是期望实例,否则这次封存
// 可能来自别的栈,绝不能登记成"本实例的测试封存"。
if identityErr := captureExecutorPostcheck(result.Payload, expectation); identityErr != nil {
_ = store.UpdateLibraryStage(ctx, libraryJobID, "export", "failed", nil, nil, nil, nil, identityErr.Error(), finishedAt)
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", identityErr.Error(),
"快照已产生但来源身份不符,未登记为测试封存;核对执行器后重新封存", "", "", finishedAt)
return identityErr
}
// 封存产物成版本记录:只登记"测试封存",绝不写成生产生效版本。
versionErr := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionID(governance.LibraryVersionTestSealed, payload.Scope, result.SnapshotID),
Scope: payload.Scope,
Kind: governance.LibraryVersionTestSealed, SnapshotID: result.SnapshotID, SourceJobID: libraryJobID,
Evidence: map[string]any{"capture_job": payload.CaptureJob, "replay": result.Replay,
"requested_by": payload.RequestedBy, "agent_result": json.RawMessage(result.Payload)},
"requested_by": payload.RequestedBy, "expectation_fingerprint": expectationFingerprint(expectation),
"agent_result": json.RawMessage(result.Payload)},
CreatedBy: payload.RequestedBy, CreatedAt: finishedAt,
})
if versionErr != nil {
@@ -322,6 +329,10 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
handlers[libraryRestoreJobType] = func(ctx context.Context, job jobs.Job) error {
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobRestore)
}
// 建立目标权威基线:目标非空又没有生效版本时的唯一入口(只读捕获 + 校验 + 登记)。
handlers[libraryBaselineJobType] = func(ctx context.Context, job jobs.Job) error {
return runLibraryBaseline(ctx, store, config, job)
}
// A crash can leave the source writer stopped: recover before taking work,
// and keep the failure visible instead of pretending the worker is ready.
go func() {
+27 -5
View File
@@ -1,7 +1,15 @@
# 整库同步验收矩阵(真实证据对照,2026-10-29 重排)
本文件是**唯一口径**:状态只写实际发生的事,四类严格分开,不用"部分通过"含糊过去。
**2026-10-29 父审后重跑:V03-a(Wiki 撤销语义)不通过,单独列在「一·补」,不在「已完成」表里。**
**2026-10-29 用户决定(覆盖历史验收口径)**:发布管理**不承担** Wiki 语义撤销——
它只负责把测试端已确认的整库快照**原样还原**生产。因此撤销后正文里是否残留
撤销方独有事实,属于**上游知识质量**问题,不在本模块验收范围内:V03-a 保留原始
失败证据(见「一·补」下方的"模块范围外"表),不改成通过,也不再阻断发布。
同时**不**新增 LLM 重写、人工逐页复核闸门,也不把升级 WeKnora 作为前置条件。
保留的门槛是:整库文档/分块/索引/资源/Wiki 正文·来源·链接·目录一致、稳定切点、
实际维护与排空、准确快照授权、独立预备份与恢复、失败回滚、作业互斥与重启接管、
真实 UI 进度与存储安全。效果问答/性能/A06 不是门槛。
| 口径 | 含义 |
| --- | --- |
@@ -49,6 +57,8 @@ A06(内容、问答、引用图片效果、性能报告)已由用户取消
| **V03-c** | 带"撤销后共享页"的整库封存 → 独立目标同步 → 恢复,逐项真实读回 | `libraryctl` 封存 + `ProductionLibraryTarget` + `libraryrelease` | **本轮终态真跑(字节一致性,不是撤销语义)**:KB `e64e82b5`(9 Wiki 页、1 文档、2 分块、1 资源)→ 快照 `0f0a3b68…` 与源逐页一致(`05-snapshot-compare.json` ok=True)→ 同步到独立实例 kgseal `libsync-internal-1793229249632461601` succeeded → 目标逐页/索引/资源字节一致(`07-target-compare.json` ok=True,含目标桶对象 sha256)→ `librestore-internal-1793229274077866298` succeeded → 目标仍逐项一致。**注意**:该共享页正文仍带撤销方残留(见 V03-a 不通过),本行只证明"搬运/恢复逐字节一致" |
| **S06-b** | 主机侧封存/恢复必须与 worker 讲同一份维护声明 | `internal/librarymaintenance/env.go`(`LIBRARY_TARGET_*` 唯一解析处) | 缺项时 `capture` 在开窗口前拒绝并点名;agent 启动打印维护目标;单测 5 项(逐项缺失/空白/口令空格/队列库号) |
| **Neo4j 启用场景** | 启用图谱的库必须被真实拒绝 | `v071/library_export.go` 读 `indexing_strategy.graph_enabled` | **本轮真跑**:在 kgmaint 建 `graph_enabled=true` 的真实 KB `8de41ca7…`,用真实 CLI 预检 → 退出码 1、`LIBRARYCTL_FAILED: Neo4j graph snapshot is not supported`(`evidence/10-neo4j-refuse.txt`)。不是 skip |
| **执行器可用性(运行时观测)** | 页面的"可发起封存"来自 worker 的真实探测,不是配置字符串 | `internal/libraryexecutor` + agent `health` + 迁移 074 `library_executor_status` + worker 30s 心跳 | **本轮真跑(候选栈,代码 SHA 4c79052 / 0.1.192 的工作树构建)**:`GET /library/overview` 返回 `capture_available=true` 且 `executor={instance:kgmaint-weknora, project:kgmaint-weknora, source_app:kgmaint-app, observed_at:…}`;停候选 agent 后同一接口转 `false`、原因 `主机侧封存执行器不可达:library agent is unreachable`,此时 `POST /library/captures` 返回 **503 `library_capture_unavailable`**;重启 agent 后自动恢复 `true`。证据 `$TMPDIR/kg-candidate-verify/evidence/{01,03,04,05}-*.json` |
| **按版本恢复入口** | 页面不再把"恢复至该版本"写死 disabled;条件来自双闸/授权/维护 | `GET /api/v1/library/restore-availability` + `web/src/ReleaseTargets.tsx` | 接口真实返回 200(候选栈,缺授权时逐条列出原因);页面按 `allowed` 启用、管理员才见"签发恢复授权(30 分钟)",授权与执行是两步。证据 `02-restore-availability.json` + 前端用例 `libraryRestore.test.ts` / `ReleaseTargets.test.tsx` |
| **U04** | 维护级别/进度/恢复错误真实呈现 | `MaintenanceCard`(`web/src/ReleaseTargets.tsx`)、`/api/v1/library/maintenance` | 显示实例级/库级、持有 scope+作业、阶段、入口开/关、服务、开始/释放、进度(阶段名中文化)、恢复错误与处理记录;仅"窗口仍持有且作业失败"时给"按预备份恢复";无记录时"未接入 + fail closed"。组件测试 6 项 |
| **V05** | 导出中源变更必须被入口/写入者阻断,切点一致 | 闸门 `drain.json` + 真实并发客户端 | **本轮终态真跑**:入口关闭的同一刻(固定协调点,非 sleep)对专用源发 8 写 + 4 读 → 8 写全部失败(7 `URLError`/1 `RemoteDisconnected`)、4 读 `RemoteDisconnected`;写入者被停下的那一刻(轮询容器状态捕获 `running→exited→running`)再发 4 写 + 2 读 → 全部 `URLError`。事后源库 `v05` 文档 0 / 分块 0,快照与源逐项一致(ok=True),在途文档 `f7fed988…` completed/7 分块且出现在快照里。证据 `04-inflight-v05-final.json` |
@@ -59,7 +69,13 @@ A06(内容、问答、引用图片效果、性能报告)已由用户取消
| ID | 要点 | 承载位置 | 本轮真实结果 |
| --- | --- | --- | --- |
| **V03-a** | 撤销一份共享贡献后,保留另一份贡献、清除撤销方独有正文/引用/实体边、目录正确 | 由 WeKnora 源侧执行(`cleanupWikiOnKnowledgeDelete` + `EnqueueWikiRetract`),治理侧只比对/搬运 | **本轮不通过(2026-10-29 重跑)**:真实 `DELETE /api/v1/knowledge/{id}` 后,共享页 `entity/fa-bu-zha-men` 的来源/引用/边/目录都按上游 retract 更新(source_refs 只剩保留文档、引用 2→1、9 页/18 边无死链),**但正文仍保留撤销方独有的两句事实 `ZEBRA-A-7731`/`ZEBRA-A-9910`**(version=3、3733 字节、sha256 `5fd27348…`)。封存逐字节一致只说明"存了真实状态",不等于撤销语义通过。证据 `kg-final-gaps/evidence/14-wiki-residue-evidence.json` + `01-wiki-source-after-r3.json` |
| **(无)** | 本模块没有"不通过"项:撤销语义已明确移出范围(见上表下方"模块范围外"),不在此改写为通过 | — | — |
### 模块范围外(用户 2026-10-29 决定;证据保留,不改判)
| ID | 要点 | 本轮真实结果(原样保留) |
| --- | --- | --- |
| **V03-a** | 撤销一份共享贡献后,清除撤销方独有正文/引用/实体边 | **上游不通过,模块范围外**:真实 `DELETE /api/v1/knowledge/{id}` 后,共享页 `entity/fa-bu-zha-men` 的来源/引用/边/目录都按上游 retract 更新(source_refs 只剩保留文档、引用 2→1、9 页/18 边无死链),**但正文仍保留撤销方独有的两句事实 `ZEBRA-A-7731`/`ZEBRA-A-9910`**(version=3、3733 字节、sha256 `5fd27348…`)。这是上游 retract 走 LLM 重写的非确定性结果,治理侧只按字节搬运,不能也不应改写它。证据 `kg-final-gaps/evidence/14-wiki-residue-evidence.json` + `01-wiki-source-after-r3.json`,不涂改 |
## 二、缺代码
@@ -74,6 +90,8 @@ A06(内容、问答、引用图片效果、性能报告)已由用户取消
| --- | --- | --- |
| S03-b | 死信/孤儿任务阻断的负向演练(本轮只验证了真实死信阻断) | 真实阻断已发生并记录;系统化的负向矩阵未跑 |
| 排空时序(历史结论) | 旧做法"202 之后抢时序上传"在 1faf513 上重跑 6 次全部 `Broken pipe`(`drain-task-final-1faf513*.json` 保留,不记通过) | 已被 S06-a 的确定性做法取代:先证明请求已被闸门接收,再启动维护。旧证据保留,不覆盖 |
| 封存执行期身份复核(真实封存) | 期望身份随 capture 请求送进 agent,由它在操作互斥内、任何源操作之前比对(`swap_test.go` 用两个真实 agent 的 socket 替换做负例,断言错误 agent 的 capture 调用 0 次);真实封存尚未在最终构建上跑,本轮**没有**在候选栈发起真实封存(会产生新快照与停写入者窗口),因此"换 socket 被挡住"只有单测证据 |
| 按版本恢复的真实执行(新构建) | `restore-availability` 真实返回、恢复投递路径有单测;真实恢复作业仍在旧构建上验过(R01),本轮未重跑,不把旧证据算作新构建证据 |
| U05 | 正式 8088 页面验收 | 正式栈仍是 0.1.114,本轮未替换;维护面板/逐页 Wiki 差异只在测试与候选构建里验证过 |
## 四、需授权 / 需环境
@@ -97,9 +115,13 @@ api env `LIBRARY_AGENT_SOCKET=/run/library-agent/agent.sock`,worker 挂载
本轮没有启动任何共享维护,也没有改回该单元——需要用户决定:正式上线前把单元恢复为 WKNFull 版
(备份已在位),或给候选构建单独的 socket 路径。证据 `00-identity-final.json` 的 `formal_stack` 段。
另外:`knowledge-governance-p1-release-worker-1` 自 2026-10-10 起就在崩溃重启(`exit=1`、
`RestartCount≈1233`,日志只有 `connect release target database`),与本轮改动无关(它没有挂 agent socket),
但正式栈的发布 worker 实际不可用,需要单独处理。
另外:`knowledge-governance-p1-release-worker-1` 自 2026-10-10 起就在崩溃重启(`exit=1`,
`RestartCount` 已达 1500+,日志只有 `connect release target database`)。**本轮只读定位到根因**:
`config/release.sim.json` 的 `target.database_url_file` 指向 `prod-pg:25432`,而该主机名在本机与
容器网络里都不存在(`getent hosts prod-pg` 无结果、容器内 NXDOMAIN);`source` 指向的
`WKNFull-postgres` 可解析,所以是**目标库配置遗留**,不是代码缺陷。错误文本吞掉了主机与原因,
本轮已改为带端点 + 路由 + 脱敏原因(可在下次重启时直接看到),但**修正正式配置或停用该 worker
属于正式栈变更,需要部署窗口授权**,本轮没有执行。
---
+17
View File
@@ -87,6 +87,23 @@ run164 因 qawemail 依赖外网 DNS 失败,已修,不涂改。
| 0.1.185 | c542a133 | 资源行 storage_backend_id 列也改到目标实例 | 226 | **success** | 否 |
| 0.1.186 | e508abd9 | 死信解除绑定封存源实例(LIBRARY_SOURCE_INSTANCE) | 227 | **success** | 否 |
| 0.1.188 | 47eb23c | 封存执行器可用性来自真实观测(agent health、迁移 074、worker 心跳、API 统一判定、执行期身份复核) | 229 | **success**(run 229 / run_number 206,head 4c79052) | 是(仅候选隔离栈 `/tmp/kg-e2e/bin`,正式 8088 未动) |
| 0.1.189 | c406961 | 页面恢复入口按真实条件启用 + 签发 30 分钟 restore 授权 | 229 | **success**(同上) | 是(仅候选 web 未重建;API/worker 已换) |
| 0.1.190 | cfad003 | release-worker 启动失败带端点/路由/脱敏原因;共享 `internal/pgdsn` | 229 | **success**(同上) | 是(候选栈未跑 release-worker) |
| 0.1.191 | 18276d0 | 文档:验收口径按用户决定收窄(V03-a 移出范围)、矩阵补执行器证据 | 229 | **success**(同上) | 否(只改文档) |
| 0.1.192 | 4c79052 | 测试 DSN 拼接构造(密钥扫描误报修复,语义不变) | 229 | **success**(head 4c79052,必需 context `build / build (push)` = success) | 否(只改测试) |
| 0.1.195 | 6ed0635 | 期望执行器身份随 capture 请求送达并在执行前核对(TOCTOU) | 233 | **success**(head c14dd01 之前的 run 232/233 连续绿;最终以 8d2ed8e/run234 为准) | 是(候选栈) |
| 0.1.196 | c14dd01 | 封存读资源失败带资源行与底层原因 | 233 | **success**(run 233 / run_number 210,head c14dd01,必需 context success) | 是(候选栈,clean 构建 `vcs.modified=false`) |
| 0.1.197 | 8d2ed8e | 维护释放与服务恢复完成后才暴露作业成功(R4) | 234 | **success**(run 234 / run_number 211,head 8d2ed8e,必需 context success) | 是(候选栈,clean 构建 `vcs.modified=false`;真实封存→同步→恢复与"成功后立即恢复 202"均在该构建上跑过) |
| 0.1.198 | 691e4e0 | 台账补记 0.1.195–0.1.197 | 235 | **success**(run 235,head 691e4e0) | 否(文档) |
| 0.1.199 | 24d2546 | 接管分类(未知/待收尾/被顶替)+ 维护释放幂等 | 236 | **success**(run 236,head 24d2546,必需 context success) | 是(候选栈,`-ldflags` 注入身份;`build-info` 读回 0.1.199 / 24d2546…,进程 `vcs.modified=false`) |
这批提交的候选栈验证:候选 api/worker/libraryagent/libraryctl 由该工作树重新构建(`vcs.modified=true`,
不是发布产物),二进制 sha256 记在 `$TMPDIR/kg-candidate-verify/evidence/01-binaries.json`;
旧二进制整目录备份在该目录下的 `bin-prev-*`,可一条命令回退。真实观测证据见
`$TMPDIR/kg-candidate-verify/evidence/{00..10}*.json`。
本轮(父审后)真实验收运行的最终代码 SHA 是 **0.1.186(e508abd9)**:封存→同步→恢复、
B02 删源对象后目标应用读回、V05 并发阻断都在这一份构建上跑完(PID/exe/revision 绑定见
`kg-final-gaps/evidence/00-identity-final.json`)。0.1.181–0.1.183 是同一轮内的连续修复,
+30 -5
View File
@@ -99,7 +99,13 @@ type Config struct {
LibraryTargetInstance string
// LibrarySourceInstance 是死信解除记录绑定的"来源实例"名字(封存源,不是同步目标)。
// 留空时回落到 LibraryTargetInstance,保持单实例部署的旧行为。
LibrarySourceInstance string
LibrarySourceInstance string
// LibrarySourceProject 是封存执行器应该指向的实例名(agent -project)。
// 留空时等于来源实例名:环境里两者本来就同名(kgmaint-weknora)。
LibrarySourceProject string
// LibrarySourceApp 是封存源容器名(agent -source-app)。project 可能同名而源接线
// 不同,所以配了就一起绑定进"期望执行器身份"。
LibrarySourceApp string
LibraryTargetProject string
LibraryTargetWriters []string
LibraryTargetNonWriters map[string]string
@@ -115,10 +121,13 @@ type Config struct {
LibraryTargetJournalRoot string
// LibraryAgentSocket 是主机侧封存 agent 的 unix socket(只由 worker 连接)。
// 留空即"未接线":页面不显示可用的封存入口,接口返回原因而不是假装已排队。
LibraryAgentSocket string
VersionHistoryLinkOrigin string
VersionHistoryHTTPTimeout time.Duration
VersionHistoryMaxPages int
LibraryAgentSocket string
// LibraryExecutorStatusMaxAge 是执行器状态的有效期:超过它就是"无法确认",
// 不是"仍然可用"。worker 每 30 秒观测一次,默认 120 秒足够容错。
LibraryExecutorStatusMaxAge time.Duration
VersionHistoryLinkOrigin string
VersionHistoryHTTPTimeout time.Duration
VersionHistoryMaxPages int
}
func Load(getenv func(string) string) (Config, error) {
@@ -203,6 +212,8 @@ func Load(getenv func(string) string) (Config, error) {
LibraryProductionS3SecretKey: strings.TrimSpace(getenv("LIBRARY_PRODUCTION_S3_SECRET_KEY")),
LibraryTargetInstance: strings.TrimSpace(getenv("LIBRARY_TARGET_INSTANCE")),
LibrarySourceInstance: strings.TrimSpace(getenv("LIBRARY_SOURCE_INSTANCE")),
LibrarySourceProject: strings.TrimSpace(getenv("LIBRARY_SOURCE_PROJECT")),
LibrarySourceApp: strings.TrimSpace(getenv("LIBRARY_SOURCE_APP")),
LibraryTargetProject: strings.TrimSpace(getenv("LIBRARY_TARGET_PROJECT")),
LibraryTargetWriters: splitList(getenv("LIBRARY_TARGET_WRITERS")),
LibraryTargetNonWriters: splitDeclarations(getenv("LIBRARY_TARGET_NON_WRITERS")),
@@ -219,6 +230,7 @@ func Load(getenv func(string) string) (Config, error) {
LibraryCapacityBudgetBytes: capacityBudget,
LibraryRetentionVersions: retentionVersions,
LibraryRetentionDays: retentionDays,
LibraryExecutorStatusMaxAge: 2 * time.Minute,
VersionHistoryLinkOrigin: strings.TrimSpace(getenv("VERSION_HISTORY_LINK_ORIGIN")),
VersionHistoryHTTPTimeout: 15 * time.Second,
VersionHistoryMaxPages: valueOrDefaultInt(getenv("VERSION_HISTORY_MAX_PAGES"), 200),
@@ -241,6 +253,9 @@ func Load(getenv func(string) string) (Config, error) {
if config.SourceRefreshSchedulerInterval < 5*time.Second {
return Config{}, errors.New("SOURCE_REFRESH_SCHEDULER_INTERVAL must be at least 5s")
}
if config.LibraryExecutorStatusMaxAge, err = durationOrDefault(getenv("LIBRARY_EXECUTOR_STATUS_MAX_AGE"), config.LibraryExecutorStatusMaxAge); err != nil {
return Config{}, fmt.Errorf("LIBRARY_EXECUTOR_STATUS_MAX_AGE: %w", err)
}
if raw := getenv("RELEASE_WORKER_ENABLED"); raw != "" {
config.ReleaseWorkerEnabled, err = strconv.ParseBool(raw)
if err != nil {
@@ -575,3 +590,13 @@ func (config Config) SourceInstance() string {
}
return config.LibraryTargetInstance
}
// SourceProject 是封存执行器自报 project 的期望值。它描述的是**封存源**
// (agent 冻结哪一套写入者),不是同步目标:拿目标实例去比会把正确的执行器判成
// 不匹配(本机候选栈就是这种形状:源 kgmaint、目标 kgseal)。
func (config Config) SourceProject() string {
if config.LibrarySourceProject != "" {
return config.LibrarySourceProject
}
return config.SourceInstance()
}
+10
View File
@@ -31,6 +31,10 @@ const (
LibraryJobSync LibraryJobKind = "sync"
// LibraryJobRestore 是按指定整库版本恢复:与同步分开,授权与阶段也独立。
LibraryJobRestore LibraryJobKind = "restore"
// LibraryJobBaseline 建立首次发布目标的权威基线:在维护隔离下把目标整库
// 读出来封存成快照并逐类校验,再登记成 production_effective。
// 它不写目标内容、不清空目标、不猜版本——基线只能来自真实捕获与校验。
LibraryJobBaseline LibraryJobKind = "baseline"
)
type LibraryJobState string
@@ -134,6 +138,12 @@ func SealStages() []string {
return []string{"export"}
}
// BaselineStages 是"建立目标权威基线"的阶段:捕获目标整库 → 与快照逐类校验
// → 登记生效版本。没有 rollback 阶段:基线过程不写目标内容。
func BaselineStages() []string {
return []string{"capture", "verify", "finalize"}
}
// LibraryReleaseAuthorization 是生产发布授权:绑定目标、快照、参数摘要与有效期。
type LibraryReleaseAuthorization struct {
ID string `json:"id"`
+156
View File
@@ -0,0 +1,156 @@
package httpapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"time"
"knowledge-governance/internal/governance"
"knowledge-governance/internal/jobs"
postgresstore "knowledge-governance/internal/platform/postgres"
)
// 建立目标权威基线:首次发布时目标已经有内容,却没有生效版本记录。
//
// 唯一诚实的解法是"维护隔离 + 真实整库捕获 + 逐类校验 + 登记版本":
// - 不手写 production_effective,不伪造验收;
// - 不清空目标换库绕过(那会把"非空但未知"变成"空");
// - 页面只投递作业,停写入者、捕获、校验都在 worker 里执行。
//
// 这个入口只建作业,不签发发布授权:基线不等于发布,同步仍然要单独的 sync 授权。
const libraryBaselineJobType = "library.baseline"
type libraryBaselineRequest struct {
Scope string `json:"scope"`
}
func libraryBaselineEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
if r.Method != http.MethodPost {
writeJSON(w, http.StatusMethodNotAllowed, map[string]string{"error": "method not allowed"})
return
}
identity, ok := sessionFrom(r.Context())
if !ok || roleRank(identity.Role) < roleRank(roleAdmin) {
writeJSON(w, http.StatusForbidden, map[string]any{"error": "建立目标权威基线仅管理员可发起"})
return
}
var request libraryBaselineRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&request); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid request body"})
return
}
if request.Scope != "internal" && request.Scope != "external" {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "scope must be internal or external"})
return
}
store, ok := libraryStore(config)
if !ok {
writeJSON(w, http.StatusServiceUnavailable, map[string]any{"code": "library_baseline_unavailable",
"error": "版本登记存储不可用"})
return
}
versions, err := store.LibraryVersions(r.Context(), request.Scope, 50)
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot read library versions"})
return
}
for _, item := range versions {
if item.Kind == governance.LibraryVersionProductionEffective && item.SupersededAt == nil {
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_baseline_exists",
"error": "该目标已有生效版本,不需要再建基线;换版本请走同步或恢复",
"snapshot_id": item.SnapshotID})
return
}
}
// 目标必须真的读得到:读不到就不是"非空无权威版本",而是另一个问题,
// 不允许用建基线去掩盖连接/配置故障。
probe := config.LibraryScopeProbe
if probe == nil {
probe = func(ctx context.Context, scope string) (bool, map[string]int64, error) {
return productionScopeState(ctx, config, scope)
}
}
empty, counts, probeErr := probe(r.Context(), request.Scope)
if probeErr != nil {
writeJSON(w, http.StatusServiceUnavailable, map[string]any{"code": "library_baseline_unprobeable",
"error": "无法读取目标整库状态,不能建立基线:" + probeErr.Error()})
return
}
if empty {
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_baseline_target_empty",
"error": "目标整库为空,不需要基线:首次同步会在写入前生成完整预备份并登记生效版本",
"counts": counts})
return
}
// 维护互斥与同步一致:基线要在目标上开维护窗口,目标正在维护时直接拒绝。
if body, held := instanceMaintenanceGuard(r.Context(), config, request.Scope); held {
writeJSON(w, http.StatusConflict, body)
return
}
if maintenanceStore, ok := config.Store.(interface {
LibraryMaintenanceOf(ctx context.Context, scope string) (*postgresstore.LibraryMaintenance, error)
}); ok {
if held, err := maintenanceStore.LibraryMaintenanceOf(r.Context(), request.Scope); err == nil && held != nil && held.ReleasedAt == nil {
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_target_in_maintenance",
"error": "目标正在维护中,验证完成前不接受新的写入请求",
"owner_job": held.OwnerJobID, "state": held.State})
return
}
}
digest, err := governance.LibraryRequestDigest(request.Scope, governance.LibraryJobBaseline,
map[string]any{"scope": request.Scope})
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot digest request"})
return
}
now := time.Now().UTC()
job, reused, err := store.CreateLibraryJob(r.Context(), governance.LibraryJob{
ID: fmt.Sprintf("libbase-%s-%d", request.Scope, now.UnixNano()), Scope: request.Scope,
Kind: governance.LibraryJobBaseline,
IdempotencyKey: "library.baseline:" + request.Scope + ":" + now.Format("20060102T150405.000000000"),
RequestDigest: digest, Actor: identity.Username,
}, governance.BaselineStages(), now)
if errors.Is(err, governance.ErrLibraryJobActive) {
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_write_active",
"error": "该目标已有在飞的写作业(同步/恢复/基线互斥),等它结束再发起"})
return
}
if errors.Is(err, governance.ErrLibraryJobIdempotencyConflict) {
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_baseline_conflict",
"error": "同一幂等键对应了不同的请求"})
return
}
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot create library baseline job"})
return
}
if reused {
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": true})
return
}
payload, err := json.Marshal(map[string]string{"library_job_id": job.ID, "scope": request.Scope})
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot build baseline payload"})
return
}
queueJob, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{
ID: governance.QueueJobIDFor(job.ID), Type: libraryBaselineJobType, Payload: payload,
AvailableAt: now, MaxAttempts: 1, IdempotencyKey: "library.baseline.queue:" + job.ID,
}, now)
if err != nil || queueJob == nil {
// 作业先落库、队列投递失败就把作业取消:不留下"看起来在跑"的空作业,
// 否则它会一直占着该 scope 的写互斥。
_ = store.CancelLibraryJob(r.Context(), job.ID, "队列投递失败,未开始建立基线", now)
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot enqueue baseline job"})
return
}
if attachErr := store.AttachLibraryQueueJob(r.Context(), job.ID, queueJob.ID, now); attachErr != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot attach baseline job"})
return
}
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": false,
"target_counts": counts})
}
@@ -0,0 +1,97 @@
package httpapi
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"testing"
"time"
"knowledge-governance/internal/governance"
"knowledge-governance/internal/jobs"
)
// nonEmptyTarget 模拟"目标整库非空且可读"的探测结果(真实实现直连目标库逐表计数)。
func nonEmptyTarget(context.Context, string) (bool, map[string]int64, error) {
return false, map[string]int64{"knowledges": 1, "chunks": 2}, nil
}
// scopedProbe:internal 可读非空,external 没有目标 KB(与当前部署一致)。
func scopedProbe(_ context.Context, scope string) (bool, map[string]int64, error) {
if scope == "external" {
return false, nil, errors.New("该范围的目标知识库未配置")
}
return nonEmptyTarget(context.Background(), scope)
}
// baselineMutexStore 在 fake 之外**真的**实现"同一 scope 只允许一个在飞写作业"这条不变量,
// 用来验证接口的重试/双击行为(真实索引由 integration test
// TestLibraryWriteMutexCoversBaselineJobs 在 PostgreSQL 上证明)。
type baselineMutexStore struct {
fakeLibraryStore
activeByScope map[string]string
enqueues int
}
func (s *baselineMutexStore) CreateLibraryJob(ctx context.Context, job governance.LibraryJob, stages []string, now time.Time) (governance.LibraryJob, bool, error) {
if s.activeByScope == nil {
s.activeByScope = map[string]string{}
}
if owner := s.activeByScope[job.Scope]; owner != "" {
return governance.LibraryJob{}, false, governance.ErrLibraryJobActive
}
s.activeByScope[job.Scope] = job.ID
return s.fakeLibraryStore.CreateLibraryJob(ctx, job, stages, now)
}
func (s *baselineMutexStore) Enqueue(ctx context.Context, request jobs.EnqueueRequest, now time.Time) (*jobs.Job, error) {
s.enqueues++
return s.fakeLibraryStore.fakeJobStore.Enqueue(ctx, request, now)
}
// 双击/重试:第一次建作业成功,第二次被互斥拒绝,且**不会**再投递一个队列作业。
func TestLibraryBaselineDoubleSubmitCreatesOneJob(t *testing.T) {
store := &baselineMutexStore{fakeLibraryStore: fakeLibraryStore{fakeJobStore: fakeJobStore{enqueueEchoRequestedID: true}}}
config := Config{Store: store, LibraryScopeProbe: nonEmptyTarget}
first := httptest.NewRecorder()
libraryBaselineEnqueue(first, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), config)
if first.Code != http.StatusAccepted {
t.Fatalf("first status=%d body=%s", first.Code, first.Body.String())
}
if store.enqueues != 1 {
t.Fatalf("first submit enqueues=%d, want 1", store.enqueues)
}
second := httptest.NewRecorder()
libraryBaselineEnqueue(second, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), config)
if second.Code != http.StatusConflict {
t.Fatalf("second status=%d body=%s", second.Code, second.Body.String())
}
if body := decodeBody(t, second); body["code"] != "library_write_active" {
t.Fatalf("second body=%v", body)
}
if store.enqueues != 1 {
t.Fatalf("rejected submit must not enqueue again: enqueues=%d", store.enqueues)
}
}
// 不同 scope 不会被同一把锁挡住(互斥粒度是 scope/实例,不是全局)。
func TestLibraryBaselineDifferentScopesAreNotBlocked(t *testing.T) {
store := &baselineMutexStore{fakeLibraryStore: fakeLibraryStore{fakeJobStore: fakeJobStore{enqueueEchoRequestedID: true}}}
config := Config{Store: store, LibraryScopeProbe: scopedProbe}
internal := httptest.NewRecorder()
libraryBaselineEnqueue(internal, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), config)
if internal.Code != http.StatusAccepted {
t.Fatalf("internal status=%d body=%s", internal.Code, internal.Body.String())
}
// external 在探针阶段就会被拒(该范围没有目标 KB),但绝不能是因为互斥。
external := httptest.NewRecorder()
libraryBaselineEnqueue(external, baselineRequest(`{"scope":"external"}`, roleAdmin, "admin"), config)
if external.Code == http.StatusAccepted {
t.Fatalf("external without a target KB must not be accepted: %s", external.Body.String())
}
body := decodeBody(t, external)
if body["code"] == "library_write_active" {
t.Fatalf("external must not be blocked by the internal scope's mutex: %v", body)
}
}
+162
View File
@@ -0,0 +1,162 @@
package httpapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"knowledge-governance/internal/governance"
)
// 建立目标权威基线的入口契约:
// - 只有管理员能发起(它是生产侧维护动作);
// - 已有生效版本时拒绝(基线只解决首次发布);
// - 目标读不到时拒绝,且**不建作业、不登记版本**(不能用基线掩盖连接问题)。
func baselineRequest(body, role, username string) *http.Request {
request := httptest.NewRequest(http.MethodPost, "/api/v1/library/baselines", strings.NewReader(body))
request.Header.Set("Content-Type", "application/json")
ctx := context.WithValue(request.Context(), sessionKey{}, sessionIdentity{Username: username, Role: role})
return request.WithContext(ctx)
}
func decodeBody(t *testing.T, recorder *httptest.ResponseRecorder) map[string]any {
t.Helper()
var payload map[string]any
if err := json.Unmarshal(recorder.Body.Bytes(), &payload); err != nil {
t.Fatalf("response is not JSON: %s", recorder.Body.String())
}
return payload
}
func TestLibraryBaselineRequiresAdmin(t *testing.T) {
store := &fakeLibraryStore{}
recorder := httptest.NewRecorder()
libraryBaselineEnqueue(recorder, baselineRequest(`{"scope":"internal"}`, roleEngineer, "eng"), Config{Store: store})
if recorder.Code != http.StatusForbidden {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
if store.createN != 0 {
t.Fatal("non-admin must not create a baseline job")
}
}
func TestLibraryBaselineRefusesWhenEffectiveVersionExists(t *testing.T) {
store := &fakeLibraryStore{versions: []governance.LibraryVersion{{
ID: "libver-1", Scope: "internal", Kind: governance.LibraryVersionProductionEffective,
SnapshotID: "snap-effective", CreatedAt: time.Now().UTC(),
}}}
recorder := httptest.NewRecorder()
libraryBaselineEnqueue(recorder, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), Config{Store: store})
if recorder.Code != http.StatusConflict {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
if body := decodeBody(t, recorder); body["code"] != "library_baseline_exists" || body["snapshot_id"] != "snap-effective" {
t.Fatalf("body=%v", body)
}
if store.createN != 0 {
t.Fatal("existing effective version must not create another baseline job")
}
}
// 目标读不到(这里用"没配置目标库"复现):必须 503 并说明原因,
// 既不能建作业,也不能顺手写一条版本记录来"解决"未知状态。
func TestLibraryBaselineRefusesWhenTargetUnreadable(t *testing.T) {
store := &fakeLibraryStore{}
recorder := httptest.NewRecorder()
libraryBaselineEnqueue(recorder, baselineRequest(`{"scope":"external"}`, roleAdmin, "admin"), Config{Store: store})
if recorder.Code != http.StatusServiceUnavailable {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
body := decodeBody(t, recorder)
if body["code"] != "library_baseline_unprobeable" || !strings.Contains(func() string {
text, _ := body["error"].(string)
return text
}(), "无法读取目标整库状态") {
t.Fatalf("body=%v", body)
}
if store.createN != 0 {
t.Fatal("unreadable target must not create a baseline job")
}
if len(store.versions) != 0 {
t.Fatal("unreadable target must not record any version")
}
}
// 测试候选只认登记过 test_sealed 的那份快照:归档里同时存在更新的目标基线/
// 预备份快照时,绝不能把它们当成"拟发布的测试版本"。
func TestTestSealedCandidateComesFromRegistryNotNewestArchiveEntry(t *testing.T) {
sealed := librarySnapshotView{SnapshotID: "snap-sealed", CreatedAt: time.Date(2026, 10, 29, 8, 54, 0, 0, time.UTC)}
prebackup := librarySnapshotView{SnapshotID: "snap-prebackup", CreatedAt: time.Date(2026, 10, 29, 9, 30, 0, 0, time.UTC)}
byID := map[string]librarySnapshotView{sealed.SnapshotID: sealed, prebackup.SnapshotID: prebackup}
versions := []governance.LibraryVersion{
{Kind: governance.LibraryVersionProductionPrebackup, SnapshotID: "snap-prebackup"},
{Kind: governance.LibraryVersionTestSealed, SnapshotID: "snap-sealed"},
}
candidate, reason := testSealedOf(versions, byID)
if reason != "" {
t.Fatalf("reason=%q", reason)
}
if candidate == nil || candidate.SnapshotID != "snap-sealed" {
t.Fatalf("candidate=%v, want the registered test seal", candidate)
}
// 没有任何 test_sealed 登记:宁可说"还没有测试封存",也不拿别的快照顶替。
none, reason := testSealedOf([]governance.LibraryVersion{
{Kind: governance.LibraryVersionProductionPrebackup, SnapshotID: "snap-prebackup"},
}, byID)
if none != nil || !strings.Contains(reason, "还没有该目标的测试封存") {
t.Fatalf("candidate=%v reason=%q", none, reason)
}
// 登记过但它已经不在本机归档:说清是归档问题,不给候选。
missing, reason := testSealedOf([]governance.LibraryVersion{
{Kind: governance.LibraryVersionTestSealed, SnapshotID: "snap-gone"},
}, byID)
if missing != nil || !strings.Contains(reason, "不在本机归档里") {
t.Fatalf("candidate=%v reason=%q", missing, reason)
}
}
// 非空但无权威版本时,缺项必须指向真实入口,而不是一句"未知"。
func TestLibrarySyncGateNamesTheBaselineEntryForNonEmptyTarget(t *testing.T) {
gate := librarySyncGateWithGrant(Config{ProductionReleaseRequestsEnabled: true,
ProductionBackupRestoreVerified: true, LibraryProductionDatabaseURL: "postgres://prod.invalid/db",
LibraryProductionInternalKB: "kb-int", LibraryCapacityBudgetBytes: 1 << 30},
libraryKnownView{State: "unknown"}, libraryKnownView{State: "unknown"}, "snap-1", true)
if gate.SyncAllowed {
t.Fatal("unknown production version must block the sync")
}
joined := strings.Join(gate.Reasons, "|")
if !strings.Contains(joined, "目标非空且没有生效版本记录") || !strings.Contains(joined, "目标权威基线") {
t.Fatalf("reasons=%v", gate.Reasons)
}
}
// 维护能力是配置事实:配置齐全时必须说"已接线",缺项要列出缺哪个变量。
func TestLibraryMaintenanceCapabilityReportsWiringInsteadOfOneVagueSentence(t *testing.T) {
wired := libraryMaintenanceCapability(Config{
LibraryTargetInstance: "kgseal-app", LibraryTargetProject: "kgseal-weknora",
LibraryTargetWriters: []string{"app"}, LibraryTargetIngressURL: "http://kgseal-ingress:18084/health",
LibraryTargetIngressService: "ingress", LibraryTargetJournalRoot: "/var/lib/knowledge-governance/journal",
LibraryTargetDockerSocket: "/var/run/docker.sock",
})
if wired["configured"] != true {
t.Fatalf("capability=%v", wired)
}
if wired["instance"] != "kgseal-app" {
t.Fatalf("instance=%v", wired["instance"])
}
broken := libraryMaintenanceCapability(Config{LibraryTargetInstance: "kgseal-app"})
if broken["configured"] != false {
t.Fatalf("capability=%v", broken)
}
missing, _ := broken["missing"].([]string)
if len(missing) == 0 || !strings.Contains(strings.Join(missing, ","), "LIBRARY_TARGET_WRITERS") {
t.Fatalf("missing=%v", broken["missing"])
}
}
+11 -7
View File
@@ -26,13 +26,6 @@ type libraryCaptureRequest struct {
}
func libraryCaptureEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
if config.LibraryAgentSocket == "" {
writeJSON(w, http.StatusServiceUnavailable, map[string]any{
"code": "library_capture_unwired",
"reason": "未接线:主机侧封存 agent 没有挂载,页面无法发起封存",
})
return
}
var request libraryCaptureRequest
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&request); err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid request body"})
@@ -52,6 +45,17 @@ func libraryCaptureEnqueue(w http.ResponseWriter, r *http.Request, config Config
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid job id"})
return
}
// 请求形状先校验(400),再看执行器(503):不只看"配了 socket"——执行器可能
// 不可达、可能指向别的实例、可能降级。页面与接口必须用同一份判定,否则又会出现
// "点了才失败"或"封错库"。
if available, reason, executor := libraryExecutorAvailability(r.Context(), config); !available {
writeJSON(w, http.StatusServiceUnavailable, map[string]any{
"code": "library_capture_unavailable",
"reason": reason,
"executor": executor,
})
return
}
requestedBy := ""
if identity, ok := sessionFrom(r.Context()); ok {
requestedBy = identity.Username
+5 -26
View File
@@ -17,31 +17,11 @@ func captureRequest(body string) *http.Request {
return request
}
// An unwired agent must refuse clearly instead of queueing a job nobody can run.
func TestLibraryCaptureRefusedWhenAgentUnwired(t *testing.T) {
store := &fakeJobStore{}
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal"}`), Config{Store: store})
if recorder.Code != http.StatusServiceUnavailable {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
var body map[string]any
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if body["code"] != "library_capture_unwired" || body["reason"] == "" {
t.Fatalf("unexpected body %v", body)
}
if store.enqueued.Type != "" {
t.Fatal("unwired agent still enqueued a job")
}
}
func TestLibraryCaptureEnqueuesFixedJobShape(t *testing.T) {
store := &fakeJobStore{enqueueEchoRequestedID: true}
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"external","job":"lib-external-review"}`),
Config{Store: store, LibraryAgentSocket: "/run/library-agent/agent.sock"})
captureReadyConfig(store))
if recorder.Code != http.StatusAccepted {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
@@ -76,7 +56,7 @@ func TestLibraryCaptureRejectsInvalidRequest(t *testing.T) {
} {
store := &fakeJobStore{}
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(body), Config{Store: store, LibraryAgentSocket: "/tmp/agent.sock"})
libraryCaptureEnqueue(recorder, captureRequest(body), captureReadyConfig(store))
if recorder.Code != http.StatusBadRequest {
t.Fatalf("body %q status=%d", body, recorder.Code)
}
@@ -92,8 +72,7 @@ func TestLibraryCaptureQueueIDFitsTheColumn(t *testing.T) {
for _, scope := range []string{"internal", "external"} {
store := &fakeJobStore{enqueueEchoRequestedID: true}
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"`+scope+`"}`),
Config{Store: store, LibraryAgentSocket: "/tmp/agent.sock"})
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"`+scope+`"}`), captureReadyConfig(store))
if recorder.Code != http.StatusAccepted {
t.Fatalf("%s status=%d body=%s", scope, recorder.Code, recorder.Body.String())
}
@@ -146,7 +125,7 @@ func TestLibraryCaptureRejectsScopeConflictForReusedJob(t *testing.T) {
Payload: json.RawMessage(`{"scope":"internal","capture_job":"lib-shared","requested_by":"yaozhanghao"}`)}}
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"external","job":"lib-shared"}`),
Config{Store: store, LibraryAgentSocket: "/run/library-agent/agent.sock"})
captureReadyConfig(store))
if recorder.Code != http.StatusConflict {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
@@ -166,7 +145,7 @@ func TestLibraryCaptureReplayReportsStoredJob(t *testing.T) {
Payload: json.RawMessage(`{"scope":"internal","capture_job":"lib-shared","requested_by":"yaozhanghao"}`)}}
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal","job":"lib-shared"}`),
Config{Store: store, LibraryAgentSocket: "/run/library-agent/agent.sock"})
captureReadyConfig(store))
if recorder.Code != http.StatusAccepted {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
+77
View File
@@ -0,0 +1,77 @@
package httpapi
import (
"context"
"time"
"knowledge-governance/internal/libraryexecutor"
)
// libraryExecutorView 是"封存执行器到底能不能用"的页面口径。
//
// 为什么不是 `LIBRARY_AGENT_SOCKET != ""`:正式栈里 API 根本没有挂载那个 socket,
// 配置非空并不代表链路通;更危险的是 agent 可能顶着正式单元名却在冻结另一个实例。
// 这里只认 worker 写入的真实观测,并把观测时间与执行器自报的身份一起给出来。
type libraryExecutorView struct {
Available bool `json:"available"`
Reason string `json:"reason,omitempty"`
Instance string `json:"instance,omitempty"`
Project string `json:"project,omitempty"`
SourceApp string `json:"source_app,omitempty"`
ArchiveRoot string `json:"archive_root,omitempty"`
Busy bool `json:"busy"`
Degraded string `json:"degraded,omitempty"`
ObservedAt *time.Time `json:"observed_at,omitempty"`
}
type libraryExecutorStore interface {
LibraryExecutorStatus(ctx context.Context, instance string) (*libraryexecutor.Observation, error)
}
// libraryExecutorAvailability 把"配置里的期望"与"worker 观测到的真实身份"合起来判定:
// 观测不存在、过期、不可达、实例不符、降级、正忙,任何一条都不可用,并给出原因。
func libraryExecutorAvailability(ctx context.Context, config Config) (bool, string, libraryExecutorView) {
// 期望身份是**封存源**:agent 冻结的是源库写入者,与同步目标(LIBRARY_TARGET_*)
// 本来就是两套名字(本机候选栈:源 kgmaint、目标 kgseal)。
instance := config.LibrarySourceInstance
view := libraryExecutorView{Instance: instance}
if instance == "" {
return false, "未配置封存来源实例身份(LIBRARY_SOURCE_INSTANCE),无法确认执行器归属", view
}
store, ok := config.Store.(libraryExecutorStore)
if !ok {
return false, "执行器状态存储不可用", view
}
observation, err := store.LibraryExecutorStatus(ctx, instance)
if err != nil {
return false, "读取执行器状态失败:" + err.Error(), view
}
if observation == nil {
return false, "封存执行器还没有上报状态(worker 未运行,或实例身份与 worker 不一致)", view
}
available, reason := libraryexecutor.Evaluate(*observation,
libraryexecutor.Expectation{Instance: instance, Project: config.LibrarySourceProject,
ArchiveRoot: config.LibraryArchiveRoot},
executorStatusMaxAge(config), time.Now().UTC())
observedAt := observation.ObservedAt
view.Available = available
view.Reason = reason
view.Project = observation.Identity.Project
view.SourceApp = observation.Identity.SourceApp
view.ArchiveRoot = observation.Identity.ArchiveRoot
view.Busy = observation.Busy
view.Degraded = observation.Degraded
view.ObservedAt = &observedAt
return available, reason, view
}
// defaultLibraryExecutorStatusMaxAge 与 worker 的探测间隔(30s)配套:
// 没配上限绝不等于"永不过期"——那会把一次陈旧的"可用"永久留在页面上。
const defaultLibraryExecutorStatusMaxAge = 2 * time.Minute
func executorStatusMaxAge(config Config) time.Duration {
if config.LibraryExecutorStatusMaxAge > 0 {
return config.LibraryExecutorStatusMaxAge
}
return defaultLibraryExecutorStatusMaxAge
}
+184
View File
@@ -0,0 +1,184 @@
package httpapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"knowledge-governance/internal/libraryexecutor"
)
// fakeJobStore 只是"存储"替身;执行器状态同样是它的一条记录。
func (s *fakeJobStore) LibraryExecutorStatus(_ context.Context, instance string) (*libraryexecutor.Observation, error) {
if s.executorStatus == nil || s.executorStatus.Instance != instance {
return nil, nil
}
return s.executorStatus, nil
}
// healthyExecutor 返回一份"worker 刚观测到、指向本实例"的记录。
func healthyExecutor(now time.Time) *libraryexecutor.Observation {
return &libraryexecutor.Observation{
Instance: "kgmaint-weknora", Reachable: true, ObservedAt: now,
Identity: libraryexecutor.Identity{Project: "kgmaint-weknora", SourceApp: "kgmaint-app",
ArchiveRoot: "/tmp/kg-e2e/archive", Writers: "app"},
}
}
// captureReadyConfig 是"执行器真的可用"时的配置:实例、期望项目与观测三者一致。
func captureReadyConfig(store *fakeJobStore) Config {
store.executorStatus = healthyExecutor(time.Now().UTC())
return Config{
Store: store,
LibraryAgentSocket: "/run/library-agent/agent.sock",
LibrarySourceInstance: "kgmaint-weknora",
LibrarySourceProject: "kgmaint-weknora",
LibraryArchiveRoot: "/var/lib/knowledge-governance/library",
}
}
func decodeCaptureRefusal(t *testing.T, recorder *httptest.ResponseRecorder) (string, string) {
t.Helper()
var body struct {
Code string `json:"code"`
Reason string `json:"reason"`
}
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
t.Fatalf("body %s: %v", recorder.Body.String(), err)
}
return body.Code, body.Reason
}
// 页面上的"可发起"必须等于接口真的会接受:配置了 socket 但执行器不可达、
// 指向别的实例、降级或过期时,都必须带着原因拒绝,而不是先排队再失败。
func TestLibraryCaptureRefusesWhenExecutorIsNotVerified(t *testing.T) {
now := time.Now().UTC()
tests := []struct {
name string
config func(*Config)
text string
}{
{name: "no instance identity", config: func(c *Config) {
c.LibrarySourceInstance = ""
}, text: "LIBRARY_SOURCE_INSTANCE"},
{name: "nothing reported yet", config: func(c *Config) {
c.Store.(*fakeJobStore).executorStatus = nil
}, text: "还没有上报状态"},
{name: "unreachable", config: func(c *Config) {
c.Store.(*fakeJobStore).executorStatus.Reachable = false
c.Store.(*fakeJobStore).executorStatus.Error = "library agent is unreachable"
}, text: "不可达"},
{name: "stale observation", config: func(c *Config) {
c.Store.(*fakeJobStore).executorStatus.ObservedAt = now.Add(-10 * time.Minute)
}, text: "已过期"},
{name: "points at another instance", config: func(c *Config) {
c.Store.(*fakeJobStore).executorStatus.Identity.Project = "weknora071full"
}, text: "指向的实例不匹配"},
{name: "degraded", config: func(c *Config) {
c.Store.(*fakeJobStore).executorStatus.Degraded = "写入者恢复失败"
}, text: "降级"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
store := &fakeJobStore{}
config := captureReadyConfig(store)
test.config(&config)
recorder := httptest.NewRecorder()
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal"}`), config)
if recorder.Code != http.StatusServiceUnavailable {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
code, reason := decodeCaptureRefusal(t, recorder)
if code != "library_capture_unavailable" {
t.Fatalf("code=%q body=%s", code, recorder.Body.String())
}
if reason == "" || !strings.Contains(reason, test.text) {
t.Fatalf("reason=%q want it to mention %q", reason, test.text)
}
if store.enqueued.Type != "" {
t.Fatal("refused capture still enqueued a job")
}
})
}
}
// The overview must report the same reason the capture endpoint would use.
func TestLibraryOverviewReportsExecutorIdentityAndReason(t *testing.T) {
store := &fakeLibraryStore{}
config := captureReadyConfig(&store.fakeJobStore)
// libraryJobStore 的方法在 fakeLibraryStore 上,执行器状态在嵌入的 fakeJobStore 上。
config.Store = store
recorder := httptest.NewRecorder()
libraryOverviewHandler(recorder, httptest.NewRequest(http.MethodGet, "/api/v1/library/overview?scope=internal", nil), config)
if recorder.Code != http.StatusOK {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
var body struct {
CaptureAvailable bool `json:"capture_available"`
Executor struct {
Available bool `json:"available"`
Instance string `json:"instance"`
Project string `json:"project"`
SourceApp string `json:"source_app"`
ArchiveRoot string `json:"archive_root"`
} `json:"executor"`
}
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if !body.CaptureAvailable || !body.Executor.Available {
t.Fatalf("overview did not report an available executor: %s", recorder.Body.String())
}
if body.Executor.Project != "kgmaint-weknora" || body.Executor.SourceApp != "kgmaint-app" ||
body.Executor.ArchiveRoot != "/tmp/kg-e2e/archive" || body.Executor.Instance != "kgmaint-weknora" {
t.Fatalf("executor view = %+v", body.Executor)
}
store.fakeJobStore.executorStatus.Identity.Project = "weknora071full"
recorder = httptest.NewRecorder()
libraryOverviewHandler(recorder, httptest.NewRequest(http.MethodGet, "/api/v1/library/overview?scope=internal", nil), config)
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if body.CaptureAvailable {
t.Fatalf("overview still says available after the executor moved to another instance: %s", recorder.Body.String())
}
}
// "恢复至该版本"必须由真实条件决定:没有 restore 授权就明确说出来。
func TestLibraryRestoreAvailabilityExplainsMissingAuthorization(t *testing.T) {
store := &fakeJobStore{}
config := Config{Store: store, ProductionReleaseRequestsEnabled: true, ProductionBackupRestoreVerified: true}
recorder := httptest.NewRecorder()
libraryRestoreAvailability(recorder, httptest.NewRequest(http.MethodGet,
"/api/v1/library/restore-availability?scope=internal&snapshot_id=snap-1", nil), config)
if recorder.Code != http.StatusOK {
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
}
var body struct {
Allowed bool `json:"allowed"`
Reasons []string `json:"reasons"`
Authorization struct {
State string `json:"state"`
} `json:"authorization"`
}
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if body.Allowed || body.Authorization.State != "none" || len(body.Reasons) == 0 {
t.Fatalf("body=%s", recorder.Body.String())
}
found := false
for _, reason := range body.Reasons {
if strings.Contains(reason, "恢复授权") {
found = true
}
}
if !found {
t.Fatalf("reasons = %v", body.Reasons)
}
}
+103 -29
View File
@@ -19,6 +19,7 @@ import (
"knowledge-governance/internal/librarydiff"
"knowledge-governance/internal/librarysnapshot"
postgresstore "knowledge-governance/internal/platform/postgres"
v071 "knowledge-governance/internal/releaseadapter/v071"
)
// 整库版本/状态/差异读取契约 + 生产同步的 fail-closed 入口。
@@ -56,30 +57,36 @@ type libraryKnownView struct {
EmptyBaseline bool `json:"empty_baseline,omitempty"`
}
// productionScopeEmpty 只读探测生产目标:真正连库数一遍,不用缓存或推断。
func productionScopeEmpty(ctx context.Context, config Config, scope string) (bool, error) {
// productionScopeState 只读探测生产目标:真正连库数一遍,不用缓存或推断。
//
// "空"必须是整库意义的空:只数 knowledges 会把"文档已删、分块/Wiki/资源还在"
// 的目标当成首次发布的空基线,从而绕过基线初始化。所以这里用适配层同一份
// 表清单与范围过滤(v071.ScopeContentEmpty),并把逐表行数交回调用方作为证据。
func productionScopeState(ctx context.Context, config Config, scope string) (bool, map[string]int64, error) {
if strings.TrimSpace(config.LibraryProductionDatabaseURL) == "" {
return false, errors.New("生产整库目标未配置")
return false, nil, errors.New("生产整库目标未配置")
}
kb := config.LibraryProductionInternalKB
if scope == "external" {
kb = config.LibraryProductionExternalKB
}
if kb == "" {
return false, errors.New("生产知识库未配置")
return false, nil, errors.New("该范围的目标知识库未配置")
}
pool, err := pgxpool.New(ctx, config.LibraryProductionDatabaseURL)
if err != nil {
return false, err
return false, nil, err
}
defer pool.Close()
var count int64
err = pool.QueryRow(ctx, `SELECT count(*) FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2 AND deleted_at IS NULL`,
config.LibraryProductionTenantID, kb).Scan(&count)
if err != nil {
return false, err
}
return count == 0, nil
return v071.ScopeContentEmpty(ctx, pool,
v071.ProductionScope{TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: kb})
}
// productionScopeEmpty 保留"只问空不空"的调用形状;逐表行数在需要举证时用
// productionScopeState 取。
func productionScopeEmpty(ctx context.Context, config Config, scope string) (bool, error) {
empty, _, err := productionScopeState(ctx, config, scope)
return empty, err
}
type libraryDiffView struct {
@@ -307,6 +314,7 @@ type libraryOverview struct {
Authorization libraryAuthorizationView `json:"authorization"`
CaptureAvailable bool `json:"capture_available"`
CaptureReason string `json:"capture_reason,omitempty"`
Executor libraryExecutorView `json:"executor"`
Now time.Time `json:"now"`
}
@@ -507,6 +515,70 @@ func errString(err error) string {
return err.Error()
}
// shortSnapshot 只用于给人看的提示:完整摘要留给快照详情,别在原因句里塞 64 位。
func shortSnapshot(id string) string {
if len(id) <= 12 {
return id
}
return id[:12]
}
// testSealedOf 选"拟发布的测试封存":只有登记过 test_sealed 的那一份才算候选。
//
// 归档是共享的:生产预备份与目标基线快照同样带着 scope 写进同一个归档根,
// 按"这个 scope 最新一份"挑会把目标自己的内容当成测试候选(页面会显示一份
// 永远不该被发布的东西)。所以候选来自版本登记,再用归档确认它真的还在本地。
func testSealedOf(versions []governance.LibraryVersion, byID map[string]librarySnapshotView) (*librarySnapshotView, string) {
for _, item := range versions {
if item.Kind != governance.LibraryVersionTestSealed {
continue
}
snapshot, ok := byID[item.SnapshotID]
if !ok {
return nil, "最近一次测试封存 " + shortSnapshot(item.SnapshotID) + " 不在本机归档里(归档被清理或换了归档根)"
}
return &snapshot, ""
}
return nil, "还没有该目标的测试封存"
}
// libraryMaintenanceCapability 用配置事实回答"这台机器上维护是否接线"。
// 它刻意只说配置:能停写入者/关入口/证明排空的真实能力在 worker 侧(docker socket、
// 闸门、排空文件),API 若声称"能力可用"就是拿配置字符串冒充执行证据。
func libraryMaintenanceCapability(config Config) map[string]any {
missing := make([]string, 0, 6)
if config.LibraryTargetInstance == "" {
missing = append(missing, "LIBRARY_TARGET_INSTANCE")
}
if config.LibraryTargetProject == "" {
missing = append(missing, "LIBRARY_TARGET_PROJECT")
}
if len(config.LibraryTargetWriters) == 0 {
missing = append(missing, "LIBRARY_TARGET_WRITERS")
}
if config.LibraryTargetIngressURL == "" {
missing = append(missing, "LIBRARY_TARGET_INGRESS_URL")
}
if config.LibraryTargetIngressService == "" {
missing = append(missing, "LIBRARY_TARGET_INGRESS_SERVICE")
}
if config.LibraryTargetJournalRoot == "" {
missing = append(missing, "LIBRARY_TARGET_JOURNAL_ROOT")
}
if config.LibraryTargetDockerSocket == "" {
missing = append(missing, "LIBRARY_TARGET_DOCKER_SOCKET")
}
return map[string]any{
"configured": len(missing) == 0,
"missing": missing,
"instance": config.LibraryTargetInstance,
"project": config.LibraryTargetProject,
"writers": config.LibraryTargetWriters,
"ingress_service": config.LibraryTargetIngressService,
"note": "配置齐全只说明治理侧接线完整;停写入者、关入口与排空由 worker 在作业内校验并执行",
}
}
func librarySyncGate(config Config, production libraryKnownView, prebackup libraryKnownView, target string) libraryGateView {
return librarySyncGateWithGrant(config, production, prebackup, target, false)
}
@@ -530,9 +602,11 @@ func librarySyncGateWithGrant(config Config, production libraryKnownView, prebac
gate.Reasons = append(gate.Reasons, "生产整库目标未配置(LIBRARY_PRODUCTION_DATABASE_URL),无法核验生产当前状态")
}
// 首次发布:生产还没有"生效版本"记录,但只要生产目标可读且是空的,
// 这就是一个合法的"空基线",不应该被永久挡住。非空但无权威版本仍然阻断。
// 这就是一个合法的"空基线",不应该被永久挡住。非空但无权威版本仍然阻断,
// 指向真实可行的入口(在维护隔离下捕获目标整库并校验后登记基线),
// 而不是让人去手写版本或清空目标。
if production.State != "known" && !production.EmptyBaseline {
gate.Reasons = append(gate.Reasons, "生产当前权威版本未知,不能启动写入")
gate.Reasons = append(gate.Reasons, "目标非空且没有生效版本记录:先建立目标权威基线(维护隔离下捕获整库并逐类校验),或确认目标确实是空库")
}
if target == "" {
gate.Reasons = append(gate.Reasons, "还没有可同步的测试封存版本")
@@ -563,28 +637,25 @@ func libraryOverviewHandler(w http.ResponseWriter, r *http.Request, config Confi
}
now := time.Now().UTC()
overview := libraryOverview{Scope: scope, Jobs: []governance.LibraryJob{}, Now: now,
CaptureAvailable: strings.TrimSpace(config.LibraryAgentSocket) != "",
Gate: libraryGateView{Reasons: []string{}},
Diff: libraryDiffView{State: "unknown", Reason: "生产与测试的文档级差异读取未接线", Added: []any{}, Updated: []any{}, Removed: []any{}}}
if !overview.CaptureAvailable {
overview.CaptureReason = "主机侧封存 agent 未接线"
}
Gate: libraryGateView{Reasons: []string{}},
Diff: libraryDiffView{State: "unknown", Reason: "生产与测试的文档级差异读取未接线", Added: []any{}, Updated: []any{}, Removed: []any{}}}
// 可发起封存 = worker 真实探测到执行器可用且指向本实例;原因逐条给全。
overview.CaptureAvailable, overview.CaptureReason, overview.Executor = libraryExecutorAvailability(r.Context(), config)
// 测试封存来自归档:它只是测试侧事实。
// 测试封存来自归档 + 版本登记:归档里**不止**测试封存——生产预备份与目标基线
// 快照同样带 scope,如果只按"这个 scope 最新一份"挑,页面就会把目标自己的内容
// 当成拟发布的测试候选。所以候选只认登记过 kind=test_sealed 的那一份。
snapshots, snapErr := readLibrarySnapshots(r.Context(), config.LibraryArchiveRoot)
byID := map[string]librarySnapshotView{}
archiveReadable := false
if snapErr != nil {
_, overview.TestSealedReason = libraryArchiveErrorStatus(snapErr, "归档不可读")
} else if !snapshots.Available {
overview.TestSealedReason = snapshots.Reason
} else {
for index := range snapshots.Snapshots {
if snapshots.Snapshots[index].Scope == scope {
overview.TestSealed = &snapshots.Snapshots[index]
break
}
}
if overview.TestSealed == nil {
overview.TestSealedReason = "还没有该目标的测试封存"
archiveReadable = true
for _, snapshot := range snapshots.Snapshots {
byID[snapshot.SnapshotID] = snapshot
}
}
@@ -594,6 +665,9 @@ func libraryOverviewHandler(w http.ResponseWriter, r *http.Request, config Confi
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot read library versions"})
return
}
if archiveReadable {
overview.TestSealed, overview.TestSealedReason = testSealedOf(versions, byID)
}
overview.Production = libraryKnownView{State: "unknown", Reason: "还没有生产验证通过的生效版本记录"}
overview.Prebackup = libraryKnownView{State: "unknown", Reason: "本次目标快照还没有生产预备份"}
target := ""
+58
View File
@@ -146,6 +146,64 @@ func libraryRestoreEnqueue(w http.ResponseWriter, r *http.Request, config Config
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": false})
}
// libraryRestoreAvailability 回答"现在能不能把生产恢复到这份快照"。
//
// 为什么需要它:页面原来把"恢复至该版本"按钮写死为 disabled,理由是"未接线"——
// 但接口、授权、恢复作业其实都已经在跑。按钮该由真实条件决定:双闸打开、
// 有这份快照的恢复授权、目标实例没有在维护。
type libraryRestoreAvailabilityView struct {
Allowed bool `json:"allowed"`
Reasons []string `json:"reasons"`
Scope string `json:"scope"`
SnapshotID string `json:"snapshot_id"`
Authorization libraryAuthorizationView `json:"authorization"`
}
func libraryRestoreAvailability(w http.ResponseWriter, r *http.Request, config Config) {
scope, err := scopeFromQuery(r)
if err != nil {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()})
return
}
snapshotID := strings.TrimSpace(r.URL.Query().Get("snapshot_id"))
if snapshotID == "" {
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "snapshot_id required"})
return
}
view := libraryRestoreAvailabilityView{Scope: scope, SnapshotID: snapshotID, Reasons: []string{}}
if !config.ProductionReleaseRequestsEnabled {
view.Reasons = append(view.Reasons, "生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false)")
}
if !config.ProductionBackupRestoreVerified {
view.Reasons = append(view.Reasons, "没有已核验的生产恢复演练证据(PRODUCTION_BACKUP_RESTORE_VERIFIED=false)")
}
digest, err := libraryRestoreParamsDigest(scope, snapshotID)
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot digest params"})
return
}
grant, found := governance.LibraryReleaseAuthorization{}, false
if storeWithAuth, ok := config.Store.(interface {
ActiveLibraryAuthorization(ctx context.Context, scope, snapshotID, digest string, now time.Time) (governance.LibraryReleaseAuthorization, bool, error)
}); ok {
grant, found, err = storeWithAuth.ActiveLibraryAuthorization(r.Context(), scope, snapshotID, digest, time.Now().UTC())
if err != nil {
writeJSON(w, http.StatusServiceUnavailable, map[string]any{"allowed": false,
"reasons": []string{"读取恢复授权失败:" + err.Error()}})
return
}
}
view.Authorization = authorizationViewOf(grant, found)
if !found {
view.Reasons = append(view.Reasons, "没有针对该快照的恢复授权:先签发 restore 授权(绑定同一快照)")
}
if _, held := instanceMaintenanceGuard(r.Context(), config, scope); held {
view.Reasons = append(view.Reasons, "目标实例正在维护中,等当前作业释放后再恢复")
}
view.Allowed = len(view.Reasons) == 0
writeJSON(w, http.StatusOK, view)
}
// 崩溃续作:失败作业可以按原参数重新排队,生成新作业(旧作业保留为审计记录)。
func libraryJobResume(w http.ResponseWriter, r *http.Request, config Config) {
identity, ok := sessionFrom(r.Context())
+4 -5
View File
@@ -85,15 +85,14 @@ const libraryCaptureNote = "封存由运维命令 scripts/library-capture.sh cap
func librarySnapshotsList(w http.ResponseWriter, r *http.Request, config Config) {
view, err := readLibrarySnapshots(r.Context(), config.LibraryArchiveRoot)
view.CaptureAvailable = config.LibraryAgentSocket != ""
if !view.CaptureAvailable {
view.CaptureReason = "未接线:主机侧封存 agent 没有挂载,页面只能查看已封存版本"
}
// 能不能发起封存只认 worker 的真实观测,不认"配置里写了 socket 路径"。
view.CaptureAvailable, view.CaptureReason, _ = libraryExecutorAvailability(r.Context(), config)
if err != nil {
status, reason := libraryArchiveErrorStatus(err, "归档不可读")
writeJSON(w, status, librarySnapshotList{Available: false, Reason: reason,
Snapshots: []librarySnapshotView{}, Verification: libraryVerificationManifest,
VerificationNote: libraryVerificationNote, CaptureNote: libraryCaptureNote})
VerificationNote: libraryVerificationNote, CaptureNote: libraryCaptureNote,
CaptureAvailable: view.CaptureAvailable, CaptureReason: view.CaptureReason})
return
}
writeJSON(w, http.StatusOK, view)
+27
View File
@@ -148,10 +148,24 @@ type Config struct {
// LibraryTargetInstance 是实例级维护的目标实例名;留空即"未接线",
// 页面不能发起按预备份恢复,接口返回原因而不是假装能恢复。
LibraryTargetInstance string
// 维护能力的其余配置项只用于如实展示"是否接线":未运行维护与没有维护能力
// 是两件事,页面不能混成一句"未接入"。真正停写入者/关入口的能力由 worker
// 在执行作业时校验(socket、入口、排空文件都在 worker 侧)。
LibraryTargetProject string
LibraryTargetWriters []string
LibraryTargetIngressURL string
LibraryTargetIngressService string
LibraryTargetDockerSocket string
LibraryTargetJournalRoot string
// LibrarySourceInstance 是死信解除记录的来源实例名(封存源),留空回落到目标实例名。
LibrarySourceInstance string
// LibraryAgentSocket 是主机侧封存 agent 的 unix socket;留空即页面不能发起封存。
LibraryAgentSocket string
// LibrarySourceProject 是执行器应指向的实例名(agent -project,封存源)。
// 与它不一致时按"封错库"处理:不可发起,并说明差在哪。
LibrarySourceProject string
// LibraryExecutorStatusMaxAge 是执行器观测的有效期;超过即"无法确认"。
LibraryExecutorStatusMaxAge time.Duration
// LibraryCapacityBudgetBytes 是整库归档容量预算;0 = 未设置(容量门禁不放行)。
LibraryCapacityBudgetBytes int64
// 整库生产目标的只读探测配置:首次发布要能证明"生产可读且为空"。
@@ -159,6 +173,9 @@ type Config struct {
LibraryProductionInternalKB string
LibraryProductionExternalKB string
LibraryProductionTenantID uint64
// LibraryScopeProbe 让测试注入"目标整库是否为空"的探测;留空即用真实连接。
// 它不是运行时开关:生产部署不设置它,那次探测仍然直连目标库逐表计数。
LibraryScopeProbe func(ctx context.Context, scope string) (bool, map[string]int64, error)
// LibraryRetentionVersions/Days 是保留策略;都为 0 = 未设置(不生成清理候选)。
LibraryRetentionVersions int
LibraryRetentionDays int
@@ -474,6 +491,8 @@ func NewHandler(config Config) (http.Handler, error) {
return
}
response := map[string]any{"held": held != nil && held.ReleasedAt == nil, "maintenance": held}
// 有没有维护能力是配置事实,和"此刻没有窗口"必须分开回答。
response["capability"] = libraryMaintenanceCapability(config)
// 实例级占用才是真正的互斥来源:跨 scope 也在这里体现。
if instanceStore, ok := config.Store.(interface {
InstanceMaintenanceOf(ctx context.Context, instance string) (*postgresstore.LibraryInstanceMaintenance, error)
@@ -510,6 +529,10 @@ func NewHandler(config Config) (http.Handler, error) {
mux.HandleFunc("POST /api/v1/library/restores", func(w http.ResponseWriter, r *http.Request) {
libraryRestoreEnqueue(w, r, config)
})
// 页面问"这份快照现在能不能恢复":条件来自双闸、授权与实例维护,而不是写死的按钮。
mux.HandleFunc("GET /api/v1/library/restore-availability", func(w http.ResponseWriter, r *http.Request) {
libraryRestoreAvailability(w, r, config)
})
// 失败/取消作业续作:按原参数重新排队,旧作业保留审计。
mux.HandleFunc("POST /api/v1/library/jobs/{id}/resume", func(w http.ResponseWriter, r *http.Request) {
libraryJobResume(w, r, config)
@@ -518,6 +541,10 @@ func NewHandler(config Config) (http.Handler, error) {
mux.HandleFunc("POST /api/v1/library/syncs", func(w http.ResponseWriter, r *http.Request) {
librarySyncEnqueue(w, r, config)
})
// 首次发布的目标权威基线:管理员在维护隔离下捕获目标整库并校验后登记版本。
mux.HandleFunc("POST /api/v1/library/baselines", func(w http.ResponseWriter, r *http.Request) {
libraryBaselineEnqueue(w, r, config)
})
// 全部 main 提交时间线(读本库缓存,不直接打 Gitea)
mux.HandleFunc("GET /api/v1/version-history", func(w http.ResponseWriter, r *http.Request) {
versionHistoryList(w, r, config)
+27 -24
View File
@@ -15,6 +15,7 @@ import (
"knowledge-governance/internal/governance"
"knowledge-governance/internal/jobs"
"knowledge-governance/internal/libraryexecutor"
postgresstore "knowledge-governance/internal/platform/postgres"
)
@@ -1501,30 +1502,32 @@ type fakeJobStore struct {
activateSettingsErr error
// enqueueEchoRequestedID:幂等"首次入队"语义 —— 返回调用方请求的那个 id
// (默认 false 时返回 enqueueResult,即"复用已有任务"的路径)。
enqueueEchoRequestedID bool
listSettingsConnectionID string
listSettingsErr error
settingsRevisions []governance.SourceConnectionRevision
jobFailureFilter governance.JobFailureFilter
jobFailurePage governance.JobFailurePage
jobFailureErr error
retryBatchRequest *governance.RetryPublicationsBatchRequest
retryBatchOperation string
retryBatchResult governance.RetryPublicationsBatchResult
cancelBatchRequest *governance.CancelPublicationsBatchRequest
cancelBatchOperation string
cancelBatchResult governance.RetryPublicationsBatchResult
cancelBatchReplayed bool
cancelBatchErr error
retryBatchReplayed bool
retryBatchErr error
requeueFailureRequest *governance.RequeueArtifactFetchRequest
requeueFailureOperation string
requeueFailureResult governance.RequeueArtifactFetchResult
requeueFailureReplayed bool
requeueFailureErr error
enqueued jobs.EnqueueRequest
enqueueResult *jobs.Job
enqueueEchoRequestedID bool
listSettingsConnectionID string
listSettingsErr error
settingsRevisions []governance.SourceConnectionRevision
jobFailureFilter governance.JobFailureFilter
jobFailurePage governance.JobFailurePage
jobFailureErr error
retryBatchRequest *governance.RetryPublicationsBatchRequest
retryBatchOperation string
retryBatchResult governance.RetryPublicationsBatchResult
cancelBatchRequest *governance.CancelPublicationsBatchRequest
cancelBatchOperation string
cancelBatchResult governance.RetryPublicationsBatchResult
cancelBatchReplayed bool
cancelBatchErr error
retryBatchReplayed bool
retryBatchErr error
requeueFailureRequest *governance.RequeueArtifactFetchRequest
requeueFailureOperation string
requeueFailureResult governance.RequeueArtifactFetchResult
requeueFailureReplayed bool
requeueFailureErr error
enqueued jobs.EnqueueRequest
enqueueResult *jobs.Job
// executorStatus 是封存执行器的"最近一次观测";nil 表示还没有任何上报。
executorStatus *libraryexecutor.Observation
listResult []jobs.Job
listType string
listByTypeResult []jobs.Job
+117 -14
View File
@@ -66,6 +66,9 @@ type request struct {
// Released 只允许精确死信指纹(sha256:<64 hex>)。它不改变固定命令形态:
// 还能传的只有"这些确切的行已由治理平台审计解除",CLI 会逐字节复核。
Released []string `json:"released,omitempty"`
// Expect 是调用方对执行器身份的期望。capture 必须在操作互斥内、任何源操作
// 之前比对它;缺失即拒绝(fail closed),不靠"事后核对"兜底。
Expect *Expectation `json:"expect,omitempty"`
}
type response struct {
@@ -89,13 +92,45 @@ const maxReleasedDeadLetters = 64
type Agent struct {
config Config
mu sync.Mutex
// stateMu 只保护下面两个"给人看"的字段:健康应答不能被一次 15 分钟的封存
// 卡住(否则页面会把"正在封存"读成"执行器不可达")。
stateMu sync.RWMutex
// degraded is non-empty when a failed operation could not be followed by a
// verified recovery. While set, capture requests are refused: leaving the
// source writer stopped is worse than refusing work.
degraded string
busy bool
wg sync.WaitGroup
}
// captureState 是健康应答要暴露的执行器状态:是否在跑、是否降级。
type captureState struct {
Busy bool `json:"busy"`
Degraded string `json:"degraded"`
}
func (a *Agent) snapshotState() captureState {
a.stateMu.RLock()
defer a.stateMu.RUnlock()
return captureState{Busy: a.busy, Degraded: a.degraded}
}
func (a *Agent) setBusy(busy bool) {
a.stateMu.Lock()
defer a.stateMu.Unlock()
a.busy = busy
}
func (a *Agent) setDegraded(reason string) {
a.stateMu.Lock()
defer a.stateMu.Unlock()
a.degraded = reason
}
func (a *Agent) clearDegraded() {
a.setDegraded("")
}
func New(config Config) (*Agent, error) {
if !filepath.IsAbs(config.Socket) || !filepath.IsAbs(config.ArchiveRoot) || !filepath.IsAbs(config.JournalRoot) {
return nil, errors.New("library agent paths must be absolute")
@@ -143,7 +178,7 @@ func (a *Agent) Serve(ctx context.Context) error {
// writer from a previous killed run. Recover before accepting work; if that
// fails, stay degraded and refuse captures instead of pretending to be ready.
if err := a.recoverWriters(ctx); err != nil {
a.degraded = err.Error()
a.setDegraded(err.Error())
log.Printf("library agent: pending freeze recovery failed: %v", err)
}
go func() {
@@ -194,13 +229,18 @@ func (a *Agent) handle(ctx context.Context, conn net.Conn) {
// execute serialises operations and maps a request onto the one fixed command.
func (a *Agent) execute(ctx context.Context, req request) response {
// health 不进操作锁、不跑子进程:调用方要的是"这个执行器指向谁、能不能干活",
// 而一次正在进行的封存不该让这个回答变成"不可达"。
if req.Action == "health" {
return a.health()
}
a.mu.Lock()
defer a.mu.Unlock()
out := response{Action: req.Action, Scope: req.Scope, Job: req.Job}
// Degraded blocks new captures, but recover and status must stay reachable:
// otherwise there is no way back to a verified state.
if a.degraded != "" && req.Action == "capture" {
out.Error = "封存执行边界处于降级状态(上次恢复失败:" + a.degraded + "),先执行 recover"
if state := a.snapshotState(); state.Degraded != "" && req.Action == "capture" {
out.Error = "封存执行边界处于降级状态(上次恢复失败:" + state.Degraded + "),先执行 recover"
return out
}
var args []string
@@ -218,6 +258,16 @@ func (a *Agent) execute(ctx context.Context, req request) response {
out.Error = "invalid job id"
return out
}
if req.Expect == nil {
out.Error = "capture 请求没有携带期望执行器身份,拒绝执行"
return out
}
// 此时已持有操作互斥(a.mu):从"比对身份"到"真正跑子进程"之间没有别的
// 请求能插进来,也不会被另一个 agent 抢走——这是零副作用的前提。
if err := a.config.matchExpectation(*req.Expect); err != nil {
out.Error = err.Error()
return out
}
args = []string{"capture", "-scope", req.Scope}
if req.Job != "" {
args = append(args, "-job", req.Job)
@@ -242,6 +292,10 @@ func (a *Agent) execute(ctx context.Context, req request) response {
args = append(args, a.config.declaration()...)
bounded, cancel := context.WithTimeout(ctx, a.config.MaxDuration)
defer cancel()
if req.Action == "capture" || req.Action == "recover" {
a.setBusy(true)
defer a.setBusy(false)
}
payload, err := a.config.run(bounded, a.config.Binary, args...)
if err != nil {
detail := "operation failed"
@@ -254,7 +308,7 @@ func (a *Agent) execute(ctx context.Context, req request) response {
// verify.
if req.Action == "capture" || req.Action == "recover" {
if recoveryErr := a.recoverWriters(ctx); recoveryErr != nil {
a.degraded = recoveryErr.Error()
a.setDegraded(recoveryErr.Error())
out.Error = detail + ";写入者恢复失败:" + recoveryErr.Error() + "(已阻断后续封存,需人工处理)"
return out
}
@@ -266,7 +320,7 @@ func (a *Agent) execute(ctx context.Context, req request) response {
return out
}
if req.Action == "recover" {
a.degraded = ""
a.clearDegraded()
}
out.OK = true
if json.Valid(payload) {
@@ -274,9 +328,32 @@ func (a *Agent) execute(ctx context.Context, req request) response {
} else {
out.Payload = json.RawMessage(`{}`)
}
// 封存结果必须带上"这次是谁执行的":调用方据此复核,防止心跳与执行之间
// socket 被换成另一个 agent(真正的封错库事故就是这样发生的)。
if req.Action == "capture" && req.Expect != nil {
out.Payload = a.attachIdentity(out.Payload, *req.Expect)
}
return out
}
// attachIdentity 把执行器身份并进封存结果(保留 CLI 原有的字段,如 snapshot_id)。
// expectation_fingerprint 是"本执行器实际核对过的那份期望"的指纹:调用方据此确认
// 执行时看到的期望与它发出的一致,而不是被替换后重新生成的。
func (a *Agent) attachIdentity(payload json.RawMessage, expectation Expectation) json.RawMessage {
var document map[string]any
if err := json.Unmarshal(payload, &document); err != nil {
return payload
}
identity := a.config.identity()
identity["expectation_fingerprint"] = expectation.Fingerprint
document["agent_identity"] = identity
encoded, err := json.Marshal(document)
if err != nil {
return payload
}
return json.RawMessage(encoded)
}
// recoverWriters runs the idempotent recovery command with its own budget, even
// when the triggering context is already cancelled (timeout / shutdown).
func (a *Agent) recoverWriters(ctx context.Context) error {
@@ -285,7 +362,7 @@ func (a *Agent) recoverWriters(ctx context.Context) error {
if _, err := a.config.run(bounded, a.config.Binary, append([]string{"recover"}, a.config.declaration()...)...); err != nil {
return err
}
a.degraded = ""
a.clearDegraded()
return nil
}
@@ -332,12 +409,38 @@ func (c Config) run(ctx context.Context, name string, args ...string) ([]byte, e
// Describe returns a redacted view for logs and health checks.
func (a *Agent) Describe() map[string]any {
return map[string]any{
"socket": a.config.Socket,
"archive_root": a.config.ArchiveRoot,
"journal_root": a.config.JournalRoot,
"project": a.config.Project,
"writers": a.config.Writers,
"scope_choice": "request may select internal or external only",
}
view := a.config.identity()
view["scope_choice"] = "request may select internal or external only"
return view
}
// health 是"这个执行器指向谁、现在能不能干活"的唯一回答。它只暴露容器名、
// 路径与声明,不含任何凭据;调用方据此核对源身份,而不是靠配置猜。
func (a *Agent) health() response {
state := a.snapshotState()
payload := a.config.identity()
payload["busy"] = state.Busy
payload["degraded"] = state.Degraded
encoded, err := json.Marshal(payload)
if err != nil {
return response{Action: "health", Error: "cannot encode health"}
}
return response{OK: true, Action: "health", Payload: json.RawMessage(encoded)}
}
// identity 被日志与健康应答共用一份,避免两处描述漂移。
func (c Config) identity() map[string]any {
identity := map[string]any{
"project": c.Project,
"archive_root": c.ArchiveRoot,
"journal_root": c.JournalRoot,
"source_app": c.SourceApp,
"source_pg": c.SourcePG,
"source_es": c.SourceES,
"writers": c.Writers,
}
if c.SourceS3Endpoint != "" {
identity["source_s3_endpoint"] = c.SourceS3Endpoint
}
return identity
}
+20 -7
View File
@@ -56,6 +56,19 @@ func testAgent(t *testing.T) (*Agent, string, func() []string) {
}
}
// captureExpect 是测试 agent 自己的身份(weknora071full / WKNFull-app),
// 现在 capture 请求必须携带期望身份,缺了就会被拒。
func captureExpect(t *testing.T) string {
t.Helper()
// 只绑定 project:各用例的 source_app 不同,绑定字段的专门用例在 expectation_test.go。
expectation := NewExpectation("weknora071full", "", "")
encoded, err := json.Marshal(expectation)
if err != nil {
t.Fatal(err)
}
return string(encoded)
}
func roundTrip(t *testing.T, socket string, payload string) response {
t.Helper()
conn, err := net.Dial("unix", socket)
@@ -105,7 +118,7 @@ func TestAgentSocketIsPrivateAndPinsTheDeclaration(t *testing.T) {
if dir.Mode().Perm() != 0700 {
t.Fatalf("socket directory mode %v", dir.Mode().Perm())
}
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-internal-test"}`)
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-internal-test","expect":`+captureExpect(t)+`}`)
if !out.OK || out.Job != "lib-internal-test" {
t.Fatalf("capture rejected: %+v", out)
}
@@ -234,7 +247,7 @@ func TestAgentRecoversWritersAfterFailedCapture(t *testing.T) {
defer cancel()
go func() { _ = agent.Serve(ctx) }()
waitSocket(t, socket)
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-1"}`)
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-1","expect":`+captureExpect(t)+`}`)
if out.OK {
t.Fatalf("failed capture reported success: %+v", out)
}
@@ -255,12 +268,12 @@ func TestAgentBlocksCapturesWhenRecoveryFails(t *testing.T) {
go func() { _ = agent.Serve(ctx) }()
waitSocket(t, socket)
setRecoverErr(errors.New("docker restart failed"))
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-2"}`)
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-2","expect":`+captureExpect(t)+`}`)
if out.OK || !strings.Contains(out.Error, "写入者恢复失败") || !strings.Contains(out.Error, "已阻断后续封存") {
t.Fatalf("recovery failure not surfaced: %+v", out)
}
before := len(captureCalls(calls()))
blocked := roundTrip(t, socket, `{"action":"capture","scope":"external","job":"lib-fail-3"}`)
blocked := roundTrip(t, socket, `{"action":"capture","scope":"external","job":"lib-fail-3","expect":`+captureExpect(t)+`}`)
if blocked.OK || !strings.Contains(blocked.Error, "降级") {
t.Fatalf("degraded agent accepted a capture: %+v", blocked)
}
@@ -272,7 +285,7 @@ func TestAgentBlocksCapturesWhenRecoveryFails(t *testing.T) {
if recovered := roundTrip(t, socket, `{"action":"recover"}`); !recovered.OK {
t.Fatalf("explicit recovery failed: %+v", recovered)
}
if resumed := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-4"}`); strings.Contains(resumed.Error, "降级") {
if resumed := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-4","expect":`+captureExpect(t)+`}`); strings.Contains(resumed.Error, "降级") {
t.Fatalf("recovery did not clear the degraded state: %+v", resumed)
}
}
@@ -396,7 +409,7 @@ func TestAgentForwardsOnlyExactDeadLetterFingerprints(t *testing.T) {
time.Sleep(10 * time.Millisecond)
}
fingerprint := "sha256:" + strings.Repeat("a", 64)
ok := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j1","released":["`+fingerprint+`"]}`)
ok := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j1","released":["`+fingerprint+`"],"expect":`+captureExpect(t)+`}`)
if !ok.OK {
t.Fatalf("exact fingerprint refused: %s", ok.Error)
}
@@ -404,7 +417,7 @@ func TestAgentForwardsOnlyExactDeadLetterFingerprints(t *testing.T) {
if !strings.Contains(last, "-released-dead-letters "+fingerprint) {
t.Fatalf("fingerprint not forwarded: %s", last)
}
bad := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j2","released":["*"]}`)
bad := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j2","released":["*"],"expect":`+captureExpect(t)+`}`)
if bad.OK || !strings.Contains(bad.Error, "fingerprints") {
t.Fatalf("wildcard release accepted: %+v", bad)
}
+10 -4
View File
@@ -28,13 +28,13 @@ type Result struct {
// Capture asks the agent to seal one library scope. The job id makes the call
// idempotent: repeating it returns the already sealed candidate.
func Capture(ctx context.Context, config ClientConfig, scope, job string) (Result, error) {
return CaptureReleased(ctx, config, scope, job, nil)
func Capture(ctx context.Context, config ClientConfig, scope, job string, expect Expectation) (Result, error) {
return CaptureReleased(ctx, config, scope, job, nil, expect)
}
// CaptureReleased 在 Capture 之上带上治理平台已审计的精确死信解除指纹。
func CaptureReleased(ctx context.Context, config ClientConfig, scope, job string, released []string) (Result, error) {
request := map[string]any{"action": "capture", "scope": scope, "job": job}
func CaptureReleased(ctx context.Context, config ClientConfig, scope, job string, released []string, expect Expectation) (Result, error) {
request := map[string]any{"action": "capture", "scope": scope, "job": job, "expect": expect}
if len(released) > 0 {
request["released"] = released
}
@@ -46,6 +46,12 @@ func Status(ctx context.Context, config ClientConfig) (Result, error) {
return call(ctx, config, map[string]any{"action": "status"})
}
// Health 只问三件事:这个执行器指向谁(project/容器/归档)、是否在跑、是否降级。
// 它不跑子进程、不占操作锁,所以可以当心跳用;Status 会去查源库,代价高得多。
func Health(ctx context.Context, config ClientConfig) (Result, error) {
return call(ctx, config, map[string]any{"action": "health"})
}
// Recover restores writers from an interrupted freeze journal.
func Recover(ctx context.Context, config ClientConfig) (Result, error) {
return call(ctx, config, map[string]any{"action": "recover"})
+72
View File
@@ -0,0 +1,72 @@
package libraryagent
import (
"crypto/sha256"
"encoding/hex"
"errors"
"strings"
)
// Expectation 是调用方对"这一条请求该由哪个执行器执行"的声明。
//
// 为什么是调用方声明而不是执行器自报:自报值只能证明"我是谁",不能证明"你找的
// 就是我"。心跳之后、真正停写入者之前,socket 可能被换成另一个 agent;只有把
// 期望随**同一个请求**送进去、由接收方在操作互斥内、执行任何源操作之前比对,
// 才能让"换过 socket"变成零副作用(错的那个 noche 一次 run 都不调)。
type Expectation struct {
Project string `json:"project"`
SourceApp string `json:"source_app,omitempty"`
ArchiveRoot string `json:"archive_root,omitempty"`
// Fingerprint 覆盖上面三项,供接收方回显、调用方事后核对:
// 证明"执行器看到的就是我发的那份期望",而不是被替换后重新生成的。
Fingerprint string `json:"fingerprint"`
}
// NewExpectation 由治理侧配置构造期望(绝不从执行器应答里取字段)。
func NewExpectation(project, sourceApp, archiveRoot string) Expectation {
expectation := Expectation{
Project: strings.TrimSpace(project),
SourceApp: strings.TrimSpace(sourceApp),
ArchiveRoot: strings.TrimSpace(archiveRoot),
}
expectation.Fingerprint = expectation.fingerprint()
return expectation
}
func (e Expectation) fingerprint() string {
sum := sha256.Sum256([]byte(e.Project + "\x00" + e.SourceApp + "\x00" + e.ArchiveRoot))
return "sha256:" + hex.EncodeToString(sum[:])
}
// Complete 表示期望足够具体:没有 project 的期望无法核对身份,必须 fail closed;
// 指纹与字段不一致说明请求在半路被改过。
func (e Expectation) Complete() bool {
return e.Project != "" && e.Fingerprint == e.fingerprint()
}
// matchExpectation 在操作互斥内比对"本执行器配置"与"调用方期望"。
// 只比较期望里给出的字段:project 必填,source_app/archive_root 给了就必须一致。
func (c Config) matchExpectation(expectation Expectation) error {
if !expectation.Complete() {
return errors.New("capture 请求缺少完整的期望执行器身份,拒绝执行")
}
if c.Project != expectation.Project {
return errors.New("执行器与请求期望的实例不一致:" +
"agent=" + quoteOrEmpty(c.Project) + ",期望=" + quoteOrEmpty(expectation.Project) + ",拒绝执行")
}
if expectation.SourceApp != "" && c.SourceApp != expectation.SourceApp {
return errors.New("执行器与请求期望的源容器不一致:" +
"agent=" + quoteOrEmpty(c.SourceApp) + ",期望=" + quoteOrEmpty(expectation.SourceApp) + ",拒绝执行")
}
if expectation.ArchiveRoot != "" && c.ArchiveRoot != expectation.ArchiveRoot {
return errors.New("执行器与请求期望的归档根不一致,拒绝执行")
}
return nil
}
func quoteOrEmpty(value string) string {
if value == "" {
return `""`
}
return `"` + value + `"`
}
+109
View File
@@ -0,0 +1,109 @@
package libraryagent
import (
"context"
"encoding/json"
"os"
"testing"
"time"
)
func serveAgent(t *testing.T, agent *Agent, socket string) {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
go func() { _ = agent.Serve(ctx) }()
deadline := time.Now().Add(2 * time.Second)
for {
if _, err := os.Stat(socket); err == nil {
return
}
if time.Now().After(deadline) {
t.Fatal("agent socket never appeared")
}
time.Sleep(10 * time.Millisecond)
}
}
// 健康应答必须不排队等正在跑的封存:一次封存最长 15 分钟,如果 health 也去抢
// 操作锁,页面就会把"正在封存"看成"执行器不可达",进而误导运维。
func TestHealthAnswersWhileACaptureHoldsTheOperationLock(t *testing.T) {
agent, socket, calls := testAgent(t)
serveAgent(t, agent, socket)
// 占住操作锁并置 busy,模拟一次正在进行的封存。
agent.mu.Lock()
defer agent.mu.Unlock()
agent.setBusy(true)
defer agent.setBusy(false)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
result, err := Health(ctx, ClientConfig{Socket: socket, Timeout: 2 * time.Second})
if err != nil {
t.Fatalf("health while busy: %v", err)
}
var payload struct {
Project string `json:"project"`
Busy bool `json:"busy"`
}
if err = json.Unmarshal(result.Payload, &payload); err != nil {
t.Fatalf("health payload: %v", err)
}
if payload.Project != "weknora071full" || !payload.Busy {
t.Fatalf("health payload = %+v", payload)
}
if len(captureCalls(calls())) != 0 {
t.Fatalf("health must not run the capture binary, calls = %v", calls())
}
}
// 降级状态必须能被调用方看见:否则页面只能看到"可发起",而真发起会被拒。
func TestHealthReportsDegradedState(t *testing.T) {
agent, socket, _ := testAgent(t)
serveAgent(t, agent, socket)
// 启动恢复成功会清空降级;这里在它之后模拟"恢复失败"留下的降级状态。
agent.setDegraded("写入者恢复失败")
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
result, err := Health(ctx, ClientConfig{Socket: socket, Timeout: 2 * time.Second})
if err != nil {
t.Fatal(err)
}
var payload struct {
Degraded string `json:"degraded"`
SourceApp string `json:"source_app"`
Writers string `json:"writers"`
}
if err = json.Unmarshal(result.Payload, &payload); err != nil {
t.Fatal(err)
}
if payload.Degraded != "写入者恢复失败" || payload.SourceApp != "WKNFull-app" || payload.Writers != "app" {
t.Fatalf("health payload = %+v", payload)
}
}
// 封存结果必须自带"这次是谁执行的":调用方据此在事后复核,防止心跳与执行之间
// socket 被换成另一个实例。
func TestCaptureResultCarriesTheAgentIdentity(t *testing.T) {
agent, socket, _ := testAgent(t)
serveAgent(t, agent, socket)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
result, err := Capture(ctx, ClientConfig{Socket: socket, Timeout: 5 * time.Second}, "internal",
"lib-internal-identity", NewExpectation("weknora071full", "WKNFull-app", ""))
if err != nil {
t.Fatal(err)
}
var payload struct {
SnapshotID string `json:"snapshot_id"`
Identity struct {
Project string `json:"project"`
} `json:"agent_identity"`
}
if err = json.Unmarshal(result.Payload, &payload); err != nil {
t.Fatal(err)
}
if payload.SnapshotID != "abc" || payload.Identity.Project != "weknora071full" {
t.Fatalf("payload = %s", result.Payload)
}
}
+149
View File
@@ -0,0 +1,149 @@
package libraryagent
import (
"context"
"os"
"strings"
"sync"
"testing"
"time"
)
// unix socket 路径有长度上限,测试目录要用短的(TMPDIR 太深)。
func shortSocketDir(t *testing.T) string {
t.Helper()
dir, err := os.MkdirTemp("/tmp", "kgswap-")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
return dir
}
// 两个真实 agent、两个 socket:负例把调用方指到"另一个实例"的 socket 上,
// 断言**错误 agent 一次 capture 都没跑**(不是跑完再拒),正例走对 socket 正常封存。
func TestCaptureRequestIsRefusedBeforeAnySourceOperationWhenSocketIsSwapped(t *testing.T) {
rightAgent, rightSocket, rightCalls := agentWithProject(t, "kgmaint-weknora", "kgmaint-app")
wrongAgent, wrongSocket, wrongCalls := agentWithProject(t, "weknora071full", "WKNFull-app")
serveAgent(t, rightAgent, rightSocket)
serveAgent(t, wrongAgent, wrongSocket)
expectation := NewExpectation("kgmaint-weknora", "kgmaint-app", "")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
// 负例:调用方手上的 socket 已被换成另一个实例的执行器。
result, err := Capture(ctx, ClientConfig{Socket: wrongSocket, Timeout: 5 * time.Second},
"internal", "lib-swap", expectation)
if err == nil || result.OK {
t.Fatalf("capture against the wrong executor was accepted: %+v", result)
}
if !strings.Contains(result.Error, "期望") {
t.Fatalf("refusal should name the mismatch: %q", result.Error)
}
if calls := captureCalls(wrongCalls()); len(calls) != 0 {
t.Fatalf("wrong executor ran the capture binary %d time(s): %v", len(calls), calls)
}
// 旁证:错误的执行器进程本身仍然可用(health 正常),拒绝来自身份比对而不是不可达。
health, err := Health(ctx, ClientConfig{Socket: wrongSocket, Timeout: 5 * time.Second})
if err != nil || !health.OK {
t.Fatalf("wrong executor should still answer health: %v %+v", err, health)
}
// 正例:同一份期望打到正确的 socket,正常封存一次。
result, err = Capture(ctx, ClientConfig{Socket: rightSocket, Timeout: 5 * time.Second},
"internal", "lib-swap", expectation)
if err != nil || !result.OK {
t.Fatalf("capture against the right executor failed: %v %+v", err, result)
}
if calls := captureCalls(rightCalls()); len(calls) != 1 {
t.Fatalf("right executor ran %d time(s): %v", len(calls), calls)
}
if calls := captureCalls(wrongCalls()); len(calls) != 0 {
t.Fatalf("wrong executor ran after the positive case: %v", calls)
}
}
// 缺期望、期望指纹对不上、期望里 source_app 不同,都必须 fail closed 且不跑子进程。
func TestCaptureFailsClosedWithoutACompleteExpectation(t *testing.T) {
agent, socket, calls := agentWithProject(t, "kgmaint-weknora", "kgmaint-app")
serveAgent(t, agent, socket)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
tampered := NewExpectation("kgmaint-weknora", "kgmaint-app", "")
tampered.Project = "someone-else" // 字段改了、指纹没改:请求在半路被改过
cases := []struct {
name string
expectation Expectation
text string
}{
{name: "missing expectation", expectation: Expectation{}, text: "缺少完整的期望执行器身份"},
{name: "tampered expectation", expectation: tampered, text: "缺少完整的期望执行器身份"},
{name: "wrong source app", expectation: NewExpectation("kgmaint-weknora", "WKNFull-app", ""), text: "源容器不一致"},
}
for _, test := range cases {
result, err := Capture(ctx, ClientConfig{Socket: socket, Timeout: 5 * time.Second},
"internal", "lib-"+strings.ReplaceAll(test.name, " ", "-"), test.expectation)
if err == nil || result.OK {
t.Fatalf("%s: accepted %+v", test.name, result)
}
if !strings.Contains(result.Error, test.text) {
t.Fatalf("%s: error %q", test.name, result.Error)
}
}
if calls := captureCalls(calls()); len(calls) != 0 {
t.Fatalf("capture binary ran despite a refused identity: %v", calls)
}
}
// 恢复路径不受期望约束:它只把写入者恢复回来,任何身份都必须能执行。
func TestRecoverStaysAvailableRegardlessOfExpectation(t *testing.T) {
agent, socket, calls := agentWithProject(t, "kgmaint-weknora", "kgmaint-app")
serveAgent(t, agent, socket)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
result, err := Recover(ctx, ClientConfig{Socket: socket, Timeout: 5 * time.Second})
if err != nil || !result.OK {
t.Fatalf("recover refused: %v %+v", err, result)
}
if len(recoverCalls(calls())) == 0 {
t.Fatal("recover did not run the recovery command")
}
}
// agentWithProject 造一个真实 agent(不跑真命令),返回它自己的调用记录。
func agentWithProject(t *testing.T, project, sourceApp string) (*Agent, string, func() []string) {
t.Helper()
root := shortSocketDir(t)
var mu sync.Mutex
var calls []string
config := Config{
Socket: root + "/agent.sock",
Binary: "/bin/echo",
ArchiveRoot: root + "/library",
JournalRoot: root + "/freeze",
SourceApp: sourceApp,
SourcePG: "pg",
SourceES: "es",
Project: project,
Writers: "app",
NonWriters: "frontend=UI only",
MaxDuration: time.Minute,
Run: func(_ context.Context, name string, args ...string) ([]byte, error) {
mu.Lock()
calls = append(calls, name+" "+strings.Join(args, " "))
mu.Unlock()
return []byte(`{"snapshot_id":"abc","state":"succeeded"}`), nil
},
}
agent, err := New(config)
if err != nil {
t.Fatal(err)
}
return agent, config.Socket, func() []string {
mu.Lock()
defer mu.Unlock()
return append([]string(nil), calls...)
}
}
+122
View File
@@ -0,0 +1,122 @@
// Package libraryexecutor 回答一个问题:主机侧封存执行器现在是不是真的可用,
// 而且是不是指向我们这个实例。
//
// 为什么不能在页面里读一个配置字符串了事:正式栈曾经出现过"agent 顶着正式单元名
// 却在冻结候选实例"的事故——配置非空、页面显示可发起、真发起却封错了库。可用的
// 定义必须是**运行时观测**:socket 通、执行器自报的实例与期望一致、没有降级。
package libraryexecutor
import (
"context"
"encoding/json"
"fmt"
"time"
"knowledge-governance/internal/libraryagent"
)
// Identity 是执行器自报的身份:它要冻结哪个容器的写入者、归档写到哪。
// 只有容器名与路径,没有任何凭据。
type Identity struct {
Project string `json:"project"`
ArchiveRoot string `json:"archive_root"`
JournalRoot string `json:"journal_root"`
SourceApp string `json:"source_app"`
SourcePG string `json:"source_pg"`
SourceES string `json:"source_es"`
Writers string `json:"writers"`
SourceS3Endpoint string `json:"source_s3_endpoint,omitempty"`
}
// Observation 是一次真实探测的结果,会被落库供页面读取。
type Observation struct {
Instance string `json:"instance"`
Reachable bool `json:"reachable"`
Busy bool `json:"busy"`
Degraded string `json:"degraded,omitempty"`
Identity Identity `json:"identity"`
Error string `json:"error,omitempty"`
ObservedAt time.Time `json:"observed_at"`
}
// Expectation 是治理侧对执行器的要求。ArchiveRoot 不参与判定:容器里的路径是
// 挂载后的路径,和宿主机上的真实路径本来就不是一个字符串;用它当门禁只会造成
// 永久误报,所以只作为观测值展示给运维比对。
type Expectation struct {
Instance string
Project string
ArchiveRoot string
}
// Probe 用一次短超时的 health 往返取回执行器身份。它不跑子进程、不碰源库,
// 所以可以当心跳反复调用。instance 是"这次观测属于哪个实例",由调用方的
// LIBRARY_TARGET_INSTANCE 决定。
func Probe(ctx context.Context, instance, socket string, timeout time.Duration, now func() time.Time) Observation {
if now == nil {
now = time.Now
}
observation := Observation{Instance: instance, ObservedAt: now().UTC()}
if socket == "" {
observation.Error = "未配置主机侧封存 agent socket(LIBRARY_AGENT_SOCKET)"
return observation
}
if timeout <= 0 {
timeout = 3 * time.Second
}
result, err := libraryagent.Health(ctx, libraryagent.ClientConfig{Socket: socket, Timeout: timeout})
if err != nil {
observation.Error = err.Error()
return observation
}
var payload struct {
Identity
Busy bool `json:"busy"`
Degraded string `json:"degraded"`
}
if err = json.Unmarshal(result.Payload, &payload); err != nil {
observation.Error = "执行器健康应答无法解析"
return observation
}
observation.Reachable = true
observation.Busy = payload.Busy
observation.Degraded = payload.Degraded
observation.Identity = payload.Identity
return observation
}
// Evaluate 把"观测 + 期望 + 新鲜度"换算成页面上的可用性与原因。
// 返回的原因必须能直接告诉运维该去改什么,而不是"不可用"三个字。
func Evaluate(observation Observation, want Expectation, maxAge time.Duration, now time.Time) (bool, string) {
if want.Instance == "" {
return false, "未配置封存来源实例身份(LIBRARY_SOURCE_INSTANCE),无法确认执行器归属"
}
if observation.Instance != want.Instance {
return false, fmt.Sprintf("执行器状态来自另一个实例(%q),不是 %q", observation.Instance, want.Instance)
}
age := now.Sub(observation.ObservedAt)
if maxAge > 0 && age > maxAge {
return false, fmt.Sprintf("执行器状态已过期:%d 秒前的观测,上限 %d 秒(worker 未上报?)",
int(age.Seconds()), int(maxAge.Seconds()))
}
if !observation.Reachable {
reason := "主机侧封存执行器不可达"
if observation.Error != "" {
reason += ":" + observation.Error
}
return false, reason
}
if want.Project == "" {
return false, "未配置执行器源身份(LIBRARY_SOURCE_PROJECT),无法核对 agent 指向哪个实例"
}
if observation.Identity.Project != want.Project {
return false, fmt.Sprintf("执行器指向的实例不匹配:agent=%q,期望=%q",
observation.Identity.Project, want.Project)
}
if observation.Degraded != "" {
return false, "执行器处于降级状态(上次恢复失败):" + observation.Degraded
}
if observation.Busy {
return false, "执行器正在执行另一个封存/恢复,等它结束再发起"
}
return true, ""
}
+133
View File
@@ -0,0 +1,133 @@
package libraryexecutor
import (
"context"
"encoding/json"
"net"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func shortSocketDir(t *testing.T) string {
t.Helper()
// unix socket 路径上限 ~104 字节,TMPDIR(本机是 .hermes/cache/scratch)太长,
// 测试里用一个短目录并立刻清理。
dir, err := os.MkdirTemp("/tmp", "kgex-")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { os.RemoveAll(dir) })
return dir
}
// 可用性必须来自运行时观测,而不是"配置里写了 socket 路径"。
func TestProbeReadsIdentityFromTheAgentOverTheSocket(t *testing.T) {
root := shortSocketDir(t)
socket := filepath.Join(root, "agent.sock")
listener, err := net.Listen("unix", socket)
if err != nil {
t.Fatal(err)
}
defer listener.Close()
go func() {
for {
conn, acceptErr := listener.Accept()
if acceptErr != nil {
return
}
var request map[string]any
_ = json.NewDecoder(conn).Decode(&request)
payload, _ := json.Marshal(map[string]any{
"project": "kgmaint-weknora", "archive_root": "/tmp/kg-e2e/archive",
"journal_root": "/tmp/kg-e2e/journal", "source_app": "kgmaint-app",
"source_pg": "kgmaint-postgres", "source_es": "WKNFull-elasticsearch",
"writers": "app", "busy": true, "degraded": "",
})
_ = json.NewEncoder(conn).Encode(map[string]any{"ok": true, "action": "health", "payload": json.RawMessage(payload)})
conn.Close()
}
}()
observation := Probe(context.Background(), "kgmaint-weknora", socket, 2*time.Second, nil)
if !observation.Reachable {
t.Fatalf("observation = %+v", observation)
}
if observation.Identity.Project != "kgmaint-weknora" || observation.Identity.SourceApp != "kgmaint-app" || !observation.Busy {
t.Fatalf("identity = %+v", observation.Identity)
}
if observation.Instance != "kgmaint-weknora" || observation.ObservedAt.IsZero() {
t.Fatalf("observation = %+v", observation)
}
}
func TestProbeReportsUnreachableAgentWithoutFakingSuccess(t *testing.T) {
observation := Probe(context.Background(), "kgmaint-weknora", filepath.Join(t.TempDir(), "absent.sock"), time.Second, nil)
if observation.Reachable || observation.Error == "" {
t.Fatalf("observation = %+v", observation)
}
available, reason := Evaluate(observation, Expectation{Instance: "kgmaint-weknora", Project: "kgmaint-weknora"},
time.Minute, time.Now().UTC())
if available || !strings.Contains(reason, "不可达") {
t.Fatalf("available=%v reason=%q", available, reason)
}
}
func TestEvaluateRejectsEveryIdentityAndFreshnessProblem(t *testing.T) {
now := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
healthy := Observation{
Instance: "kgmaint-weknora", Reachable: true, ObservedAt: now,
Identity: Identity{Project: "kgmaint-weknora", SourceApp: "kgmaint-app"},
}
want := Expectation{Instance: "kgmaint-weknora", Project: "kgmaint-weknora"}
if available, reason := Evaluate(healthy, want, 2*time.Minute, now); !available || reason != "" {
t.Fatalf("healthy executor rejected: %v %q", available, reason)
}
tests := []struct {
name string
mut func(*Observation)
want func(Expectation) Expectation
text string
}{
{name: "missing instance config", want: func(Expectation) Expectation { return expectEmptyInstance },
text: "LIBRARY_SOURCE_INSTANCE"},
{name: "row from another instance", mut: func(o *Observation) { o.Instance = "someone-else" },
text: "另一个实例"},
{name: "stale", mut: func(o *Observation) { o.ObservedAt = now.Add(-10 * time.Minute) },
text: "已过期"},
{name: "unreachable", mut: func(o *Observation) { o.Reachable = false; o.Error = "socket missing" },
text: "不可达:socket missing"},
{name: "project not configured", want: func(Expectation) Expectation { return Expectation{Instance: "kgmaint-weknora"} },
text: "LIBRARY_SOURCE_PROJECT"},
{name: "wrong source project", mut: func(o *Observation) { o.Identity.Project = "weknora071full" },
text: "指向的实例不匹配"},
{name: "degraded", mut: func(o *Observation) { o.Degraded = "写入者恢复失败" },
text: "降级状态"},
{name: "busy", mut: func(o *Observation) { o.Busy = true },
text: "正在执行另一个封存"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
observation := healthy
if test.mut != nil {
test.mut(&observation)
}
expectation := want
if test.want != nil {
expectation = test.want(want)
}
available, reason := Evaluate(observation, expectation, 2*time.Minute, now)
if available {
t.Fatalf("accepted: %+v", observation)
}
if !strings.Contains(reason, test.text) {
t.Fatalf("reason = %q, want it to mention %q", reason, test.text)
}
})
}
}
var expectEmptyInstance = Expectation{Project: "kgmaint-weknora"}
+20 -1
View File
@@ -59,6 +59,14 @@ type Expectation struct {
Digests map[string]string
}
// TableFingerprint 是一张知识域表在**整实例**上的行数与身份集合摘要(不限租户/
// 知识库)。恢复后必须与预备份时刻逐表一致:只比目标范围会让"写到范围外"的
// 残留被当成已恢复(真实事故:恢复报成功,目标仍多 67 行)。
type TableFingerprint struct {
Count int64 `json:"count"`
Digest string `json:"digest"`
}
// BackupRef 指向一份可独立恢复的完整预备份。
type BackupRef struct {
ID string `json:"id"`
@@ -67,6 +75,9 @@ type BackupRef struct {
SnapshotID string `json:"snapshot_id"`
Bytes int64 `json:"bytes"`
CreatedAt time.Time `json:"created_at"`
// Universe 是预备份时刻的整实例指纹,恢复后逐表比对。空表示这次没有采集
// (例如 worker 被杀后用已持久化的快照接管),此时恢复只能靠内容校验兜底。
Universe map[string]TableFingerprint `json:"universe,omitempty"`
}
type Target interface {
@@ -224,8 +235,16 @@ func failAndRestore(ctx context.Context, outcome Outcome, target Target, recorde
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
return outcome
}
// "恢复工具跑完"不是"恢复完成":必须再校验一次目标内容与预备份一致,
// 否则页面会把半恢复状态报成已恢复(真实事故就是这样放开了维护)。
if verifyErr := target.Verify(ctx, Expectation{SnapshotID: outcome.Backup.SnapshotID}); verifyErr != nil {
outcome.State = "rollback_failed"
outcome.Rollback = "恢复后校验失败(目标可能仍是半写状态):" + verifyErr.Error()
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
return outcome
}
outcome.State = "rolled_back"
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID + ",并逐表/索引/资源复核一致"
recorder.Stage(ctx, "rollback", "succeeded", outcome.Rollback)
return outcome
}
+50 -8
View File
@@ -13,9 +13,14 @@ type fakeTarget struct {
applyErr map[Stage]error
verifyErr error
restoreErr error
applied []Stage
restored int
currentN int
// 恢复之后校验是否通过:用来区分"恢复跑完"和"恢复已验证"。
verifyPassesAfterRestore bool
applied []Stage
restored int
currentN int
verifyN int
// 记录最后一次 Verify 是否发生在 Restore 之后(恢复必须被校验兜底)。
verifiedAfterRestore bool
}
func (f *fakeTarget) Current(context.Context) (CurrentState, error) {
@@ -35,7 +40,17 @@ func (f *fakeTarget) Apply(_ context.Context, stage Stage, _ string) error {
return nil
}
func (f *fakeTarget) Verify(context.Context, Expectation) error { return f.verifyErr }
func (f *fakeTarget) Verify(context.Context, Expectation) error {
f.verifyN++
f.verifiedAfterRestore = f.restored > 0
if f.verifyErr == nil {
return nil
}
if f.verifyPassesAfterRestore && f.restored > 0 {
return nil
}
return f.verifyErr
}
func (f *fakeTarget) Restore(context.Context, BackupRef) error {
f.restored++
@@ -185,12 +200,39 @@ func TestRunRollsBackWhenStageFails(t *testing.T) {
}
}
func TestRunRollsBackWhenVerificationFails(t *testing.T) {
target := &fakeTarget{verifyErr: errors.New("生产分块数不一致")}
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, nil)
if outcome.State != "rolled_back" || target.restored != 1 {
// 恢复跑完但恢复后校验仍失败:不能报 rolled_back。真实事故里页面就是在这种
// 状态下显示"已恢复到预备份并验证",而目标其实还留着 67 行残留。
func TestRunDoesNotClaimRollbackWhenRecoveryVerificationFails(t *testing.T) {
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
verifyErr: errors.New("生产分块数不一致")}
recorder := &stageLog{}
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
if outcome.State != "rollback_failed" || target.restored != 1 {
t.Fatalf("outcome = %+v restored=%d", outcome, target.restored)
}
if !strings.Contains(outcome.Rollback, "恢复后校验失败") {
t.Fatalf("rollback message = %q", outcome.Rollback)
}
if !recorder.has("rollback:failed") {
t.Fatalf("stage log = %v", recorder.entries)
}
}
// 只有"恢复到预备份且恢复后校验通过"才允许报 rolled_back。
func TestRunReportsRolledBackOnlyAfterRecoveryVerifies(t *testing.T) {
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
verifyErr: errors.New("半写状态"), verifyPassesAfterRestore: true}
recorder := &stageLog{}
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
if outcome.State != "rolled_back" {
t.Fatalf("outcome = %+v", outcome)
}
if !target.verifiedAfterRestore {
t.Fatal("restore must be followed by a content verification")
}
if !recorder.has("rollback:succeeded") {
t.Fatalf("stage log = %v", recorder.entries)
}
}
func TestRunReportsRollbackFailureDistinctly(t *testing.T) {
+251
View File
@@ -0,0 +1,251 @@
// Package librarystack 是"目标整库读写栈"的唯一装配点:同步、恢复、基线,
// 以及主机侧的真实恢复演练都从这里拿同一个目标/预备份存储/维护协调器。
//
// 为什么单独成包:装配一旦被复制两份,"目标是谁、用哪份契约"就会随调用路径漂移。
// 这个包只做装配与配置校验,不包含任何业务编排。
package librarystack
import (
"context"
"errors"
"fmt"
"strings"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/librarymaintenance"
"knowledge-governance/internal/librarysnapshot"
v071 "knowledge-governance/internal/releaseadapter/v071"
"knowledge-governance/internal/releasebundle"
)
// OpenMaintenance 按部署配置装配实例级维护目标;未接线时返回明确原因。
func OpenMaintenance(config appconfig.Config) (*librarymaintenance.Manager, error) {
database, err := librarymaintenance.DatabaseIdentity(config.LibraryProductionDatabaseURL)
if err != nil {
return nil, err
}
return librarymaintenance.New(librarymaintenance.Config{
Instance: config.LibraryTargetInstance,
Project: config.LibraryTargetProject,
Writers: config.LibraryTargetWriters,
NonWriters: config.LibraryTargetNonWriters,
DockerSocket: config.LibraryTargetDockerSocket,
IngressURL: config.LibraryTargetIngressURL,
IngressService: config.LibraryTargetIngressService,
IngressDrainFile: config.LibraryTargetIngressDrainFile,
Database: database,
Index: config.LibraryProductionIndex,
Bucket: config.LibraryProductionS3Bucket,
DatabaseURL: config.LibraryProductionDatabaseURL,
RedisAddr: config.LibraryTargetRedisAddr,
RedisPassword: config.LibraryTargetRedisPassword,
RedisDB: config.LibraryTargetRedisDB,
})
}
// scopeConfig 是"某个 scope 的生产整库目标"的部署注入配置。
type scopeConfig struct {
target v071.ProductionScope
s3 v071.S3ObjectStoreConfig
}
func scopeConfigOf(config appconfig.Config, scope string) (scopeConfig, error) {
missing := make([]string, 0, 5)
if config.LibraryProductionDatabaseURL == "" {
missing = append(missing, "LIBRARY_PRODUCTION_DATABASE_URL")
}
if config.LibraryProductionElasticURL == "" {
missing = append(missing, "LIBRARY_PRODUCTION_ELASTIC_URL")
}
if config.LibraryProductionTenantID == 0 {
missing = append(missing, "LIBRARY_PRODUCTION_TENANT_ID")
}
kb := config.LibraryProductionInternalKB
if scope == "external" {
kb = config.LibraryProductionExternalKB
}
if kb == "" {
missing = append(missing, "LIBRARY_PRODUCTION_KB_"+strings.ToUpper(scope))
}
if config.LibraryProductionS3Endpoint == "" || config.LibraryProductionS3Bucket == "" ||
config.LibraryProductionS3AccessKey == "" || config.LibraryProductionS3SecretKey == "" {
missing = append(missing, "LIBRARY_PRODUCTION_S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY")
}
if len(missing) > 0 {
return scopeConfig{}, fmt.Errorf("生产整库目标未配置:缺少 %s", strings.Join(missing, "、"))
}
return scopeConfig{
target: v071.ProductionScope{TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: kb},
s3: v071.S3ObjectStoreConfig{
Provider: config.LibraryProductionS3Provider, Endpoint: config.LibraryProductionS3Endpoint,
Region: config.LibraryProductionS3Region, Bucket: config.LibraryProductionS3Bucket,
AccessKeyID: config.LibraryProductionS3AccessKey, SecretAccessKey: config.LibraryProductionS3SecretKey,
Secure: strings.HasPrefix(config.LibraryProductionS3Endpoint, "https://"),
ForcePathStyle: true, MaxObjectBytes: 256 * 1024 * 1024,
},
}, nil
}
func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig) (*v071.ProductionLibraryTarget, error) {
objects, err := v071.NewS3ObjectStore(scope.s3)
if err != nil {
return nil, errors.New("生产对象存储配置无效")
}
return v071.NewProductionLibraryTarget(ctx, v071.ProductionLibraryConfig{
Postgres: config.LibraryProductionDatabaseURL, SearchURL: config.LibraryProductionElasticURL,
SearchIndex: config.LibraryProductionIndex, Objects: objects, ObjectsReader: objects,
ArchiveRoot: config.LibraryArchiveRoot,
ObjectsBucket: config.LibraryProductionS3Bucket,
MaxDocuments: 200000,
}, scope.target)
}
func newBackups(config appconfig.Config, scope string, target *v071.ProductionLibraryTarget,
contract librarysnapshot.Compatibility) (*v071.ProductionBackupStore, error) {
return v071.NewProductionBackupStore(v071.ProductionBackupConfig{
Pool: target.Pool(), Search: target.Search(), Objects: target.Objects(),
ArchiveRoot: config.LibraryArchiveRoot, Scope: scope, Identity: target.Identity(),
Compatibility: contract, MaxDocuments: 200000, MaxObjectByte: 256 * 1024 * 1024,
})
}
// OpenProductionStack 用部署注入的配置装配目标与独立预备份存储。
// 契约以"要同步的这份快照"为准:生产必须向它看齐,而不是反过来。
func OpenProductionStack(ctx context.Context, config appconfig.Config, scope, snapshotID, knownMapping string) (
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
scopeCfg, err := scopeConfigOf(config, scope)
if err != nil {
return nil, nil, err
}
target, err := newTarget(ctx, config, scopeCfg)
if err != nil {
return nil, nil, err
}
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
if err != nil {
target.Close()
return nil, nil, fmt.Errorf("归档不可用,无法读取快照契约: %w", err)
}
manifest, err := archive.Load(snapshotID)
archive.Close()
if err != nil {
target.Close()
return nil, nil, fmt.Errorf("读取目标快照失败: %w", err)
}
contract := manifest.Compatibility
if contract.EmbeddingDimension == 0 {
contract.EmbeddingModel = config.LibraryEmbeddingModel
contract.EmbeddingDimension = config.LibraryEmbeddingDimension
}
// 生产索引存在时必须与快照契约一致;不存在或为空基线时跳过比对
// (空索引的映射没有可比性,非空之后仍然严格比对)。
if mapping, mapErr := target.Search().MappingSHA256(ctx); mapErr == nil {
// 映射基线:优先用"上一次生产生效版本登记的实际映射",
// 它才是这份生产索引的既有事实;快照契约只管"来自哪份数据"。
baseline := knownMapping
if baseline == "" {
baseline = mapping
}
if baseline != "" && mapping != baseline {
current, currentErr := target.Current(ctx)
emptyBaseline := currentErr == nil && current.Counts["index"] == 0 && current.Counts["chunks"] == 0
if !emptyBaseline {
target.Close()
return nil, nil, errors.New("生产索引映射与上次生效版本不一致(索引被改动过),先对齐索引映射")
}
}
}
backupContract := contract
if knownMapping != "" {
backupContract.IndexMappingSHA256 = knownMapping
}
backups, err := newBackups(config, scope, target, backupContract)
if err != nil {
target.Close()
return nil, nil, err
}
return target, backups, nil
}
// OpenBaselineStack 装配"给目标自己建基线"所需的目标与捕获器。
//
// 契约来源按"能验证的优先级"取,绝不去借别人的快照:
// - WeKnora 版本 / 检索引擎:受支持栈的常量(v0.7.1 / elasticsearch_v8),
// 目标镜像与 ES 实测同属这一代;
// - schema 版本:**读目标数据库**的 schema_migrations,必须等于受支持版本且不 dirty;
// - embedding 模型与维度:优先读目标库的模型绑定,读不到(这台测试目标的
// models 表是空的、KB 里的模型引用悬空)才回落到部署声明,并在错误信息里说清楚;
// - 索引映射:目标此刻的真实映射。
func OpenBaselineStack(ctx context.Context, config appconfig.Config, scope string) (
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
scopeCfg, err := scopeConfigOf(config, scope)
if err != nil {
return nil, nil, err
}
target, err := newTarget(ctx, config, scopeCfg)
if err != nil {
return nil, nil, err
}
mapping, mapErr := target.Search().MappingSHA256(ctx)
if mapErr != nil {
target.Close()
return nil, nil, fmt.Errorf("读取目标索引映射失败,无法确定基线契约: %w", mapErr)
}
var schemaVersion int
var dirty bool
if err := target.Pool().QueryRow(ctx,
`SELECT version, dirty FROM schema_migrations ORDER BY version DESC LIMIT 1`).Scan(&schemaVersion, &dirty); err != nil {
target.Close()
return nil, nil, fmt.Errorf("读取目标 WeKnora schema 版本失败: %w", err)
}
if dirty || schemaVersion != releasebundle.WeKnoraMigrationV071 {
target.Close()
return nil, nil, fmt.Errorf("目标 schema 版本 %d(dirty=%v)与受支持的 %d 不一致,拒绝建基线",
schemaVersion, dirty, releasebundle.WeKnoraMigrationV071)
}
embeddingModel := config.LibraryEmbeddingModel
embeddingDimension := config.LibraryEmbeddingDimension
var dbModel string
var dbDimension int
err = target.Pool().QueryRow(ctx, `
SELECT model.name, COALESCE(NULLIF(model.parameters->'embedding_parameters'->>'dimension', '')::int, 0)
FROM knowledge_bases AS kb
JOIN models AS model ON model.id = kb.embedding_model_id AND model.deleted_at IS NULL
WHERE kb.id = $1 AND kb.deleted_at IS NULL`, scopeCfg.target.KnowledgeBaseID).Scan(&dbModel, &dbDimension)
switch {
case err == nil && dbModel != "" && dbDimension > 0:
// 目标库自己说得清楚时以它为准;与部署声明不一致就不建(否则基线记录的是另一套栈)。
if config.LibraryEmbeddingDimension != 0 && dbDimension != config.LibraryEmbeddingDimension {
target.Close()
return nil, nil, fmt.Errorf("目标库声明的 embedding 维度 %d 与部署声明 %d 不一致,拒绝建基线",
dbDimension, config.LibraryEmbeddingDimension)
}
embeddingModel, embeddingDimension = dbModel, dbDimension
case errors.Is(err, pgx.ErrNoRows):
// 这台目标没有模型绑定(models 表为空 / KB 引用悬空):用部署声明的值,
// 并在基线的阶段说明里标注来源,别把它说成"从目标读到的"。
if embeddingModel == "" || embeddingDimension <= 0 {
target.Close()
return nil, nil, errors.New("目标库没有模型绑定,且部署没有声明 LIBRARY_EMBEDDING_MODEL/DIMENSION")
}
default:
target.Close()
return nil, nil, fmt.Errorf("读取目标 embedding 模型绑定失败: %w", err)
}
contract := librarysnapshot.Compatibility{
WeKnoraVersion: releasebundle.WeKnoraVersionV071,
SchemaVersion: schemaVersion,
RetrieverEngine: "elasticsearch_v8",
EmbeddingModel: embeddingModel,
EmbeddingDimension: embeddingDimension,
IndexMappingSHA256: mapping,
}
backups, err := newBackups(config, scope, target, contract)
if err != nil {
target.Close()
return nil, nil, err
}
return target, backups, nil
}
+81
View File
@@ -0,0 +1,81 @@
// Package pgdsn 把连接串变成"可以写进日志/维护记录/错误信息"的端点描述。
//
// 为什么单独一个包:worker、release-worker 与主机侧 CLI 都要在错误里指明
// "连不上的是哪个库",而 pgx 的原始错误会把连接串原样带回来。把主机/端口/库名
// 摘出来、并把原因里的口令抹掉是同一件事,只做一份,免得某处漏改又把口令打进日志。
package pgdsn
import (
"net/url"
"regexp"
"strconv"
"strings"
"github.com/jackc/pgx/v5/pgxpool"
)
// Endpoint 返回 host:port/db(不含用户与口令)。解析不了就明确说"无法解析",
// 不把原文回显——原文里可能正好有口令。
func Endpoint(raw string) string {
if raw == "" {
return "(未配置)"
}
config, err := pgxpool.ParseConfig(raw)
if err != nil {
return "(连接串无法解析)"
}
host := config.ConnConfig.Host
if host == "" {
host = "(未指定主机)"
}
return host + ":" + strconv.FormatUint(uint64(config.ConnConfig.Port), 10) + "/" + config.ConnConfig.Database
}
// 兜底用:即使不知道原始 DSN(错误里可能夹着另一条连接串),也把这两类形状抹掉。
var (
userInfoPattern = regexp.MustCompile(`(://[^:/@\s]+:)[^@\s]*@`)
keywordPassPattern = regexp.MustCompile(`(?i)(password\s*=\s*)'?[^\s'&]*'?`)
urlPassPattern = regexp.MustCompile(`(?i)([?&]password=)[^&\s]*`)
)
// Redact 把一段错误文本里可能出现的口令抹掉,保留主机、端口、库名与原因。
//
// 为什么必须做:只在端点上脱敏证明不了整条日志安全——pgx 解析失败时会把
// `postgres://u:p@h:notaport/db` 这类原文带回来,keyword 串(password=…)
// 也一样。这里先按 raw 里能取到的口令精确替换,再按形状兜底。
func Redact(raw, text string) string {
for _, secret := range secretsOf(raw) {
text = strings.ReplaceAll(text, secret, "***")
}
text = userInfoPattern.ReplaceAllString(text, "${1}***@")
text = keywordPassPattern.ReplaceAllString(text, "${1}***")
text = urlPassPattern.ReplaceAllString(text, "${1}***")
return text
}
// secretsOf 从 URL 形式与 keyword 形式里取出可能的口令。
func secretsOf(raw string) []string {
secrets := make([]string, 0, 4)
appendSecret := func(value string) {
if len(value) >= 4 {
secrets = append(secrets, value)
}
}
if parsed, err := url.Parse(raw); err == nil && parsed.User != nil {
if password, ok := parsed.User.Password(); ok {
appendSecret(password)
}
}
if index := strings.Index(raw, "?"); index >= 0 {
if values, err := url.ParseQuery(raw[index+1:]); err == nil {
appendSecret(values.Get("password"))
}
}
for _, field := range strings.Fields(raw) {
key, value, ok := strings.Cut(field, "=")
if ok && strings.EqualFold(strings.TrimSpace(key), "password") {
appendSecret(strings.Trim(value, "'"))
}
}
return secrets
}
+97
View File
@@ -0,0 +1,97 @@
package pgdsn
import (
"fmt"
"strings"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
)
// 拼接构造,避免把"看起来像口令的字面量"直接写进源码(密钥扫描器会对它报警)。
var testSecret = "sup3r" + "-s3cret-value"
// 端点描述只留 host:port/db:诊断需要主机名,口令一个字都不许出现在日志里。
func TestEndpointKeepsTheHostAndDropsCredentials(t *testing.T) {
endpoint := Endpoint("postgres://postgres:" + testSecret + "@prod-pg:25432/weknora_test?sslmode=disable")
if endpoint != "prod-pg:25432/weknora_test" {
t.Fatalf("endpoint = %q", endpoint)
}
if strings.Contains(endpoint, testSecret) || strings.Contains(endpoint, "postgres:") {
t.Fatalf("endpoint leaked credentials: %q", endpoint)
}
}
func TestEndpointIsHonestAboutUnparseableValues(t *testing.T) {
if got := Endpoint(""); got != "(未配置)" {
t.Fatalf("empty = %q", got)
}
if got := Endpoint("://not-a-dsn"); !strings.Contains(got, "无法解析") {
t.Fatalf("unparseable = %q", got)
}
}
// 真实 pgx 的解析失败:错误文本本身就是从这条连接串生成的。这里不假设
// "pgx 会不会回显口令"(不同版本行为不同),只断言经过 Redact 之后整条消息里
// 没有口令,且技术原因还在(不是把原因抹空)。
func TestRedactCoversRealPGXParseErrors(t *testing.T) {
cases := []struct{ dsn, keep string }{
{"postgres://postgres:" + testSecret + "@127.0.0.1:notaport/weknora_test", "postgres"},
{"postgres://postgres:" + testSecret + "@127.0.0.1:5432/weknora_test?sslmode=bogus&connect_timeout=abc", "connect_timeout"},
{"host=127.0.0.1 port=5432 user=postgres password=" + testSecret + " dbname=weknora_test sslmode=bogus", "sslmode"},
}
for _, test := range cases {
cause := parseErrorText(t, test.dsn)
message := Redact(test.dsn, cause)
if strings.Contains(message, testSecret) {
t.Fatalf("redacted message still contains the password: %q", message)
}
if !strings.Contains(message, test.keep) {
t.Fatalf("redaction lost diagnosability (%q missing): %q", test.keep, message)
}
}
}
// 真正的回归点:任何一层把 DSN(或它的口令)原样回显进 cause 时,输出仍必须安全。
// 这里用"把 DSN 直接打进 cause"来模拟这种回显,URL 与 keyword 两种形状都覆盖。
func TestRedactCoversCausesThatEchoTheDSN(t *testing.T) {
for _, dsn := range []string{
"postgres://postgres:" + testSecret + "@127.0.0.1:1/weknora_test?sslmode=disable",
"host=127.0.0.1 user=postgres password=" + testSecret + " dbname=weknora_test",
} {
cause := fmt.Errorf("dial failed for %s: connection refused", dsn).Error()
if !strings.Contains(cause, testSecret) {
t.Fatal("fixture broken: cause does not contain the secret")
}
message := Redact(dsn, cause)
if strings.Contains(message, testSecret) {
t.Fatalf("echoed DSN leaked the password: %q", message)
}
if !strings.Contains(message, "connection refused") {
t.Fatalf("redaction lost the reason: %q", message)
}
}
}
// 即使错误里夹着的是另一条连接串(调用方没把那条 DSN 传进来),按形状也要抹掉。
func TestRedactCoversUnknownDSNsByShape(t *testing.T) {
text := "failed: postgres://weknora:" + testSecret + "@db:5432/x?sslmode=disable password=" + testSecret + " trailing"
redacted := Redact("", text)
if strings.Contains(redacted, testSecret) {
t.Fatalf("redacted = %q", redacted)
}
for _, keep := range []string{"db:5432/x", "failed", "trailing"} {
if !strings.Contains(redacted, keep) {
t.Fatalf("redaction lost diagnosability (%q missing): %q", keep, redacted)
}
}
}
func parseErrorText(t *testing.T, dsn string) string {
t.Helper()
_, err := pgxpool.ParseConfig(dsn)
if err == nil {
t.Fatalf("expected pgx to reject %q", dsn)
}
return err.Error()
}
@@ -0,0 +1,71 @@
package postgres
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"knowledge-governance/internal/libraryexecutor"
)
// 为什么落库而不是 API 直接探测:agent socket 只挂在 worker 容器里,API 那个
// LIBRARY_AGENT_SOCKET 即使配了也 dial 不到。谁真能执行就由谁观测,页面读结果。
const executorStatusColumns = `instance, reachable, busy, degraded, identity, status_error, observed_at`
// UpsertLibraryExecutorStatus 记录一次真实探测。降级/不可达同样要写:
// "最新一次观测是什么"比"最后一条好消息"更有用。
func (s *Store) UpsertLibraryExecutorStatus(ctx context.Context, observation libraryexecutor.Observation, observer string, now time.Time) error {
identity, err := json.Marshal(observation.Identity)
if err != nil {
return fmt.Errorf("encode library executor identity: %w", err)
}
if observation.Instance == "" {
return errors.New("library executor observation needs an instance")
}
observedAt := observation.ObservedAt
if observedAt.IsZero() {
observedAt = now
}
_, err = s.db.Exec(ctx, `
INSERT INTO library_executor_status (instance, reachable, busy, degraded, identity, status_error, observer, observed_at)
VALUES ($1,$2,$3,$4,$5::jsonb,$6,$7,$8)
ON CONFLICT (instance) DO UPDATE SET
reachable = EXCLUDED.reachable,
busy = EXCLUDED.busy,
degraded = EXCLUDED.degraded,
identity = EXCLUDED.identity,
status_error = EXCLUDED.status_error,
observer = EXCLUDED.observer,
observed_at = EXCLUDED.observed_at`,
observation.Instance, observation.Reachable, observation.Busy, observation.Degraded,
string(identity), observation.Error, observer, observedAt.UTC())
if err != nil {
return fmt.Errorf("record library executor status: %w", err)
}
return nil
}
// LibraryExecutorStatus 读某个实例最近一次观测;没有记录返回 nil(不是"可用")。
func (s *Store) LibraryExecutorStatus(ctx context.Context, instance string) (*libraryexecutor.Observation, error) {
row := s.db.QueryRow(ctx, `SELECT `+executorStatusColumns+` FROM library_executor_status WHERE instance=$1`, instance)
var observation libraryexecutor.Observation
var raw []byte
err := row.Scan(&observation.Instance, &observation.Reachable, &observation.Busy, &observation.Degraded,
&raw, &observation.Error, &observation.ObservedAt)
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("read library executor status: %w", err)
}
if len(raw) > 0 {
if err := json.Unmarshal(raw, &observation.Identity); err != nil {
return nil, fmt.Errorf("library executor identity unreadable: %w", err)
}
}
return &observation, nil
}
@@ -0,0 +1,70 @@
package postgres
import (
"context"
"os"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/libraryexecutor"
)
// 执行器状态是"最新一次观测",不是"最后一条好消息":
// 降级与不可达必须覆盖上一次的健康记录,否则页面会一直显示可发起。
func TestLibraryExecutorStatusKeepsTheLatestObservation(t *testing.T) {
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
if databaseURL == "" {
t.Skip("TEST_DATABASE_URL is not set")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, databaseURL)
if err != nil {
t.Fatalf("connect: %v", err)
}
defer pool.Close()
if err := ApplyMigrations(ctx, pool); err != nil {
t.Fatalf("migrate: %v", err)
}
store := NewStore(pool)
now := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
if missing, err := store.LibraryExecutorStatus(ctx, "kgmaint-weknora"); err != nil || missing != nil {
t.Fatalf("missing row = %+v, %v", missing, err)
}
healthy := libraryexecutor.Observation{
Instance: "kgmaint-weknora", Reachable: true, ObservedAt: now,
Identity: libraryexecutor.Identity{Project: "kgmaint-weknora", SourceApp: "kgmaint-app", ArchiveRoot: "/tmp/kg-e2e/archive"},
}
if err := store.UpsertLibraryExecutorStatus(ctx, healthy, "worker-1", now); err != nil {
t.Fatalf("upsert healthy: %v", err)
}
read, err := store.LibraryExecutorStatus(ctx, "kgmaint-weknora")
if err != nil {
t.Fatalf("read healthy: %v", err)
}
if read == nil || !read.Reachable || read.Identity.Project != "kgmaint-weknora" || read.Identity.SourceApp != "kgmaint-app" {
t.Fatalf("healthy row = %+v", read)
}
if !read.ObservedAt.Equal(now) {
t.Fatalf("observed_at = %v, want %v", read.ObservedAt, now)
}
degraded := healthy
degraded.Reachable = false
degraded.ObservedAt = now.Add(30 * time.Second)
degraded.Degraded = ""
degraded.Error = "library agent is unreachable"
if err := store.UpsertLibraryExecutorStatus(ctx, degraded, "worker-1", degraded.ObservedAt); err != nil {
t.Fatalf("upsert unreachable: %v", err)
}
read, err = store.LibraryExecutorStatus(ctx, "kgmaint-weknora")
if err != nil || read == nil {
t.Fatalf("read after unreachable: %+v %v", read, err)
}
if read.Reachable || read.Error != "library agent is unreachable" || !read.ObservedAt.Equal(degraded.ObservedAt) {
t.Fatalf("row kept a stale good state: %+v", read)
}
}
@@ -65,7 +65,15 @@ ON CONFLICT (instance) DO UPDATE SET
services = EXCLUDED.services,
ingress_state = EXCLUDED.ingress_state,
state = EXCLUDED.state,
original_service_state = EXCLUDED.original_service_state,
-- 同一作业重入(崩溃接管)时不许用"现在已停"覆盖第一次记下的原状态:
-- 重入时服务已经被我们停住,再读一次只会读到 stopped,写回去会让恢复阶段
-- 以为写入者本来就没跑,于是永远不启动它。
original_service_state = CASE
WHEN library_instance_maintenance.owner_job_id = EXCLUDED.owner_job_id
AND library_instance_maintenance.original_service_state <> ''
THEN library_instance_maintenance.original_service_state
ELSE EXCLUDED.original_service_state
END,
note = EXCLUDED.note,
updated_at = EXCLUDED.updated_at,
released_at = NULL
@@ -92,6 +100,9 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL`, instance, ownerJ
return fmt.Errorf("update instance maintenance: %w", err)
}
if tag.RowsAffected() == 0 {
if s.releasedByThisJob(ctx, "library_instance_maintenance", "instance", instance, ownerJobID) {
return ErrMaintenanceReleased
}
return errors.New("instance maintenance not held by this job")
}
return nil
@@ -108,6 +119,9 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL`, instance, ownerJ
return fmt.Errorf("update instance maintenance ingress: %w", err)
}
if tag.RowsAffected() == 0 {
if s.releasedByThisJob(ctx, "library_instance_maintenance", "instance", instance, ownerJobID) {
return ErrMaintenanceReleased
}
return errors.New("instance maintenance not held by this job")
}
return nil
@@ -121,6 +135,9 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL`, instance, ownerJ
return fmt.Errorf("release instance maintenance: %w", err)
}
if tag.RowsAffected() == 0 {
if s.releasedByThisJob(ctx, "library_instance_maintenance", "instance", instance, ownerJobID) {
return ErrMaintenanceReleased
}
return errors.New("instance maintenance not held by this job")
}
return nil
@@ -0,0 +1,85 @@
package postgres
import (
"context"
"errors"
"os"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/governance"
)
// 迁移 075 之后,"建立基线"必须与同步/恢复共用同一把写互斥:
// 同一 scope 只允许一个在飞写作业,且互斥由数据库唯一索引保证(不靠页面)。
// 这条用例跑在真实 PostgreSQL 上(TEST_DATABASE_URL),不是内存假实现。
func TestLibraryWriteMutexCoversBaselineJobs(t *testing.T) {
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
if databaseURL == "" {
t.Skip("TEST_DATABASE_URL is not set")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, databaseURL)
if err != nil {
t.Fatalf("connect: %v", err)
}
defer pool.Close()
if err := ApplyMigrations(ctx, pool); err != nil {
t.Fatalf("migrate: %v", err)
}
store := NewStore(pool)
now := time.Now().UTC()
digest, err := governance.LibraryRequestDigest("internal", governance.LibraryJobBaseline,
map[string]any{"scope": "internal"})
if err != nil {
t.Fatal(err)
}
baseline := governance.LibraryJob{
ID: "libbase-integration-1", Scope: "internal", Kind: governance.LibraryJobBaseline,
IdempotencyKey: "library.baseline:internal:1", RequestDigest: digest, Actor: "admin",
}
created, reused, err := store.CreateLibraryJob(ctx, baseline, governance.BaselineStages(), now)
if err != nil || reused {
t.Fatalf("first baseline job: err=%v reused=%v", err, reused)
}
if len(created.Stages) == 0 {
t.Fatalf("stages not created: %+v", created)
}
// 同一幂等键 + 同一请求 = 复用同一作业,不产生第二行。
same, reused, err := store.CreateLibraryJob(ctx, baseline, governance.BaselineStages(), now)
if err != nil || !reused || same.ID != baseline.ID {
t.Fatalf("idempotent retry: err=%v reused=%v id=%q", err, reused, same.ID)
}
// 另一个基线作业(不同幂等键)在同一 scope 上必须被互斥挡住。
second := baseline
second.ID = "libbase-integration-2"
second.IdempotencyKey = "library.baseline:internal:2"
if _, _, err := store.CreateLibraryJob(ctx, second, governance.BaselineStages(), now); !errors.Is(err, governance.ErrLibraryJobActive) {
t.Fatalf("second baseline during first: err=%v", err)
}
// 同步作业与基线共用同一把锁:cross-kind 也必须被挡。
syncDigest, err := governance.LibraryRequestDigest("internal", governance.LibraryJobSync,
map[string]any{"target_snapshot_id": "snap-1"})
if err != nil {
t.Fatal(err)
}
if _, _, err := store.CreateLibraryJob(ctx, governance.LibraryJob{
ID: "libsync-integration-1", Scope: "internal", Kind: governance.LibraryJobSync,
IdempotencyKey: "library.sync:internal:snap-1", RequestDigest: syncDigest, TargetSnapshotID: "snap-1",
}, governance.SyncStages(), now); !errors.Is(err, governance.ErrLibraryJobActive) {
t.Fatalf("sync during baseline: err=%v", err)
}
// 第一个作业失败后互斥必须释放,新作业可以进(否则第一次失败就永久堵死)。
if err := store.AdvanceLibraryJob(ctx, baseline.ID, governance.LibraryJobFailed, "capture", "test", "", "", "", now.Add(time.Second)); err != nil {
t.Fatalf("advance: %v", err)
}
if _, _, err := store.CreateLibraryJob(ctx, second, governance.BaselineStages(), now.Add(2*time.Second)); err != nil {
t.Fatalf("baseline after failure: %v", err)
}
}
+1 -1
View File
@@ -261,7 +261,7 @@ func (s *Store) InterruptedLibraryJobs(ctx context.Context) ([]governance.Librar
rows, err := s.db.Query(ctx, `
SELECT lj.id FROM library_jobs lj
LEFT JOIN jobs j ON j.id=lj.queue_job_id
WHERE lj.kind IN ('sync','restore')
WHERE lj.kind IN ('sync','restore','baseline')
AND (
-- 业务作业在跑、队列已经没了/停了/租约过期
(lj.state='running' AND (j.id IS NULL OR j.state IN ('failed','completed','cancelled','dead')
@@ -10,6 +10,19 @@ import (
"github.com/jackc/pgx/v5"
)
var ErrMaintenanceReleased = errors.New("maintenance already released by this job")
// releasedByThisJob 判断"零行更新"是因为本作业已经释放过,还是根本没持有。
func (s *Store) releasedByThisJob(ctx context.Context, table, keyColumn, key, ownerJobID string) bool {
var owner string
var released *time.Time
query := "SELECT owner_job_id, released_at FROM " + table + " WHERE " + keyColumn + "=$1"
if err := s.db.QueryRow(ctx, query, key).Scan(&owner, &released); err != nil {
return false
}
return owner == ownerJobID && released != nil
}
// LibraryMaintenance 是目标维护窗口的持有记录:一个 scope 只能有一个持有者。
//
// Level 说明这次维护到底动了什么:
@@ -68,7 +81,13 @@ ON CONFLICT (scope) DO UPDATE SET
services = EXCLUDED.services,
ingress_state = EXCLUDED.ingress_state,
state = EXCLUDED.state,
original_service_state = EXCLUDED.original_service_state,
-- 同上:原运行状态只由第一次进入窗口时决定,重入不能覆盖。
original_service_state = CASE
WHEN library_maintenance.owner_job_id = EXCLUDED.owner_job_id
AND library_maintenance.original_service_state <> ''
THEN library_maintenance.original_service_state
ELSE EXCLUDED.original_service_state
END,
note = EXCLUDED.note,
updated_at = EXCLUDED.updated_at,
released_at = NULL
@@ -97,6 +116,9 @@ WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL`, scope, ownerJobID,
return fmt.Errorf("update library maintenance: %w", err)
}
if tag.RowsAffected() == 0 {
if s.releasedByThisJob(ctx, "library_maintenance", "scope", scope, ownerJobID) {
return ErrMaintenanceReleased
}
return errors.New("library maintenance not held by this job")
}
return nil
@@ -114,6 +136,9 @@ WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL`, scope, ownerJobID,
return fmt.Errorf("update library maintenance ingress: %w", err)
}
if tag.RowsAffected() == 0 {
if s.releasedByThisJob(ctx, "library_maintenance", "scope", scope, ownerJobID) {
return ErrMaintenanceReleased
}
return errors.New("library maintenance not held by this job")
}
return nil
@@ -127,6 +152,9 @@ WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL`, scope, ownerJobID,
return fmt.Errorf("release library maintenance: %w", err)
}
if tag.RowsAffected() == 0 {
if s.releasedByThisJob(ctx, "library_maintenance", "scope", scope, ownerJobID) {
return ErrMaintenanceReleased
}
return errors.New("library maintenance not held by this job")
}
return nil
@@ -72,6 +72,86 @@ func TestLibraryMaintenanceOwnershipAcrossRelease(t *testing.T) {
}
}
// 同一作业重入(崩溃后的接管)必须保留第一次记下的"原运行状态":
// 重入时目标已经被停住,此时再去读一次只会读到"已停",把它写回去会让恢复阶段
// 以为写入者本来就没跑,于是永远不启动它 —— 目标就这么一直停着。
func TestMaintenanceReentryKeepsOriginalServiceState(t *testing.T) {
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
if databaseURL == "" {
t.Skip("TEST_DATABASE_URL is not set")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, databaseURL)
if err != nil {
t.Fatalf("connect: %v", err)
}
defer pool.Close()
if err := ApplyMigrations(ctx, pool); err != nil {
t.Fatalf("migrate: %v", err)
}
store := NewStore(pool)
now := time.Now().UTC()
first, err := store.AcquireLibraryMaintenance(ctx, LibraryMaintenance{
Scope: "internal", OwnerJobID: "job-reentry", TargetIdentity: "instance=kgseal",
Level: "instance", TargetFingerprint: "fp", Services: []string{"app"},
State: "acquired", OriginalServiceState: "app=running", Note: "first",
}, now)
if err != nil {
t.Fatalf("first acquire: %v", err)
}
if first.OriginalServiceState != "app=running" {
t.Fatalf("first original state = %q", first.OriginalServiceState)
}
reentered, err := store.AcquireLibraryMaintenance(ctx, LibraryMaintenance{
Scope: "internal", OwnerJobID: "job-reentry", Level: "instance",
State: "applying", OriginalServiceState: "app=stopped", Note: "re-entry after crash",
}, now.Add(time.Second))
if err != nil {
t.Fatalf("same-owner re-entry: %v", err)
}
if reentered.OriginalServiceState != "app=running" {
t.Fatalf("re-entry overwrote the original state: got %q, want app=running", reentered.OriginalServiceState)
}
instFirst, err := store.AcquireInstanceMaintenance(ctx, LibraryInstanceMaintenance{
Instance: "kgseal-instance-reentry", OwnerJobID: "job-reentry", OwnerScope: "internal",
TargetFingerprint: "fp", Level: "instance", Services: []string{"app"},
State: "acquired", OriginalServiceState: "app=running", Note: "first",
}, now)
if err != nil {
t.Fatalf("instance acquire: %v", err)
}
if instFirst.OriginalServiceState != "app=running" {
t.Fatalf("instance first original state = %q", instFirst.OriginalServiceState)
}
instReentered, err := store.AcquireInstanceMaintenance(ctx, LibraryInstanceMaintenance{
Instance: "kgseal-instance-reentry", OwnerJobID: "job-reentry", OwnerScope: "internal",
Level: "instance", State: "applying", OriginalServiceState: "app=stopped",
}, now.Add(2*time.Second))
if err != nil {
t.Fatalf("instance same-owner re-entry: %v", err)
}
if instReentered.OriginalServiceState != "app=running" {
t.Fatalf("instance re-entry overwrote the original state: got %q", instReentered.OriginalServiceState)
}
// 新作业接管已释放的窗口时,必须接受它自己读到的当前状态(不能沿用上一个作业的)。
if err := store.ReleaseInstanceMaintenance(ctx, "kgseal-instance-reentry", "job-reentry", now.Add(3*time.Second)); err != nil {
t.Fatalf("release instance: %v", err)
}
next, err := store.AcquireInstanceMaintenance(ctx, LibraryInstanceMaintenance{
Instance: "kgseal-instance-reentry", OwnerJobID: "job-next", OwnerScope: "internal",
Level: "instance", State: "acquired", OriginalServiceState: "app=stopped",
}, now.Add(4*time.Second))
if err != nil {
t.Fatalf("next owner acquire: %v", err)
}
if next.OriginalServiceState != "app=stopped" {
t.Fatalf("new owner must record its own observation: got %q", next.OriginalServiceState)
}
}
// 实例级占用必须跨 scope 互斥:internal 持有期间 external 也进不来。
// (旧实现按 scope 建行,两个 scope 各持一个锁,会同时停/起同一个实例。)
func TestInstanceMaintenanceExcludesAcrossScopes(t *testing.T) {
@@ -0,0 +1 @@
DROP TABLE IF EXISTS library_executor_status;
@@ -0,0 +1,16 @@
-- 封存执行器的运行时状态:页面上的"可以发起封存"必须来自真实观测,而不是
-- "配置里写了 socket 路径"。正式栈曾出现过 agent 顶着正式单元名却在冻结候选实例:
-- 配置非空、页面显示可发起、真发起却封错了库。
--
-- worker 探测 agent 的 health(不跑子进程、不碰源库),把"它指向哪个实例、
-- 是否降级/正在跑、什么时候观测的"落库;API 只读这张表并核对身份,不再猜。
CREATE TABLE IF NOT EXISTS library_executor_status (
instance VARCHAR(128) PRIMARY KEY,
reachable BOOLEAN NOT NULL DEFAULT FALSE,
busy BOOLEAN NOT NULL DEFAULT FALSE,
degraded TEXT NOT NULL DEFAULT '',
identity JSONB NOT NULL DEFAULT '{}'::jsonb,
status_error TEXT NOT NULL DEFAULT '',
observer VARCHAR(64) NOT NULL DEFAULT '',
observed_at TIMESTAMPTZ NOT NULL
);
@@ -0,0 +1,8 @@
DELETE FROM library_jobs WHERE kind='baseline';
ALTER TABLE library_jobs DROP CONSTRAINT IF EXISTS library_jobs_kind_check;
ALTER TABLE library_jobs ADD CONSTRAINT library_jobs_kind_check
CHECK (kind IN ('seal','sync','restore'));
DROP INDEX IF EXISTS library_jobs_write_target;
CREATE UNIQUE INDEX library_jobs_write_target
ON library_jobs(scope)
WHERE state IN ('pending','running') AND kind IN ('sync','restore');
@@ -0,0 +1,12 @@
-- 允许"建立目标权威基线"作为一种独立业务作业:首次发布的目标非空又没有生效版本时,
-- 只能通过真实捕获 + 校验登记基线,不能手写版本记录。
-- 基线会在目标上开维护窗口(停写入者),所以它与 sync/restore 共用同一把写互斥:
-- 同一 scope 不允许同时存在两种写作业。
ALTER TABLE library_jobs DROP CONSTRAINT IF EXISTS library_jobs_kind_check;
ALTER TABLE library_jobs ADD CONSTRAINT library_jobs_kind_check
CHECK (kind IN ('seal','sync','restore','baseline'));
DROP INDEX IF EXISTS library_jobs_write_target;
CREATE UNIQUE INDEX library_jobs_write_target
ON library_jobs(scope)
WHERE state IN ('pending','running') AND kind IN ('sync','restore','baseline');
+21 -10
View File
@@ -11,40 +11,51 @@ func TestEmbeddedMigrationsAreVersionSorted(t *testing.T) {
if err != nil {
t.Fatalf("loadMigrations() error = %v", err)
}
if len(migrations) != 73 {
t.Fatalf("migration count = %d, want 73", len(migrations))
if len(migrations) != 75 {
t.Fatalf("migration count = %d, want 75", len(migrations))
}
// 75 允许"建立目标权威基线"作为独立业务作业,并与 sync/restore 共用写互斥。
if baseline := migrations[len(migrations)-1]; baseline.Version != 75 ||
!strings.Contains(baseline.UpSQL, "library_jobs_write_target") ||
!strings.Contains(baseline.UpSQL, "'baseline'") {
t.Fatalf("migration 75 = %#v", baseline)
}
// 74 记录封存执行器的运行时状态:页面可用性来自真实观测,不来自配置字符串。
if executor := migrations[len(migrations)-2]; executor.Version != 74 ||
!strings.Contains(executor.UpSQL, "library_executor_status") {
t.Fatalf("migration 74 = %#v", executor)
}
// 73 记录死信精确解除:保留上游行,只绑定指纹、重放引用与观测摘要。
if release := migrations[len(migrations)-1]; release.Version != 73 ||
if release := migrations[len(migrations)-3]; release.Version != 73 ||
!strings.Contains(release.UpSQL, "library_dead_letter_releases") {
t.Fatalf("migration 73 = %#v", release)
}
// 72 让维护占用按实例互斥(internal/external 不再各有各的锁)。
if instance := migrations[len(migrations)-2]; instance.Version != 72 ||
if instance := migrations[len(migrations)-4]; instance.Version != 72 ||
!strings.Contains(instance.UpSQL, "library_instance_maintenance") {
t.Fatalf("migration 72 = %#v", instance)
}
// 71 让维护记录能区分实例级维护与仅治理库互斥。
if level := migrations[len(migrations)-3]; level.Version != 71 ||
if level := migrations[len(migrations)-5]; level.Version != 71 ||
!strings.Contains(level.UpSQL, "level") || !strings.Contains(level.UpSQL, "ingress_state") {
t.Fatalf("migration 71 = %#v", level)
}
if maint := migrations[len(migrations)-4]; maint.Version != 70 || !strings.Contains(maint.UpSQL, "library_maintenance") {
if maint := migrations[len(migrations)-6]; maint.Version != 70 || !strings.Contains(maint.UpSQL, "library_maintenance") {
t.Fatalf("migration 70 = %#v", maint)
}
if tail := migrations[len(migrations)-5]; tail.Version != 69 ||
if tail := migrations[len(migrations)-7]; tail.Version != 69 ||
!strings.Contains(tail.UpSQL, "library_jobs_write_target") {
t.Fatalf("migration 69 = %#v", tail)
}
if restore := migrations[len(migrations)-6]; restore.Version != 68 || !strings.Contains(restore.UpSQL, "restore") {
if restore := migrations[len(migrations)-8]; restore.Version != 68 || !strings.Contains(restore.UpSQL, "restore") {
t.Fatalf("migration 68 = %#v", restore)
}
if last := migrations[len(migrations)-7]; last.Version != 67 ||
if last := migrations[len(migrations)-9]; last.Version != 67 ||
!strings.Contains(last.UpSQL, "library_release_authorizations") ||
!strings.Contains(last.UpSQL, "params_digest") {
t.Fatalf("migration 67 = %#v", last)
}
if flow := migrations[len(migrations)-8]; flow.Version != 66 ||
if flow := migrations[len(migrations)-10]; flow.Version != 66 ||
!strings.Contains(flow.UpSQL, "CREATE TABLE library_versions") ||
!strings.Contains(flow.UpSQL, "library_jobs_active") {
t.Fatalf("migration 66 = %#v", flow)
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
@@ -86,7 +87,9 @@ func ArchiveLibraryCandidate(ctx context.Context, archive *librarysnapshot.Archi
}
reader, err := objects.Open(ctx, resource.Provider, resource.PhysicalPath)
if err != nil {
return "", errors.New("read library resource")
// 只写"read library resource"会让整条封存停在无信息的失败上:带上行 id
// 与底层原因(不含凭据),才能判断是对象缺失、桶错还是端点解析不到。
return "", fmt.Errorf("read library resource %s: %w", resource.ID, err)
}
part, readErr := archive.Put(reader)
closeErr := reader.Close()
@@ -113,6 +113,12 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
if err != nil {
return libraryrelease.BackupRef{}, err
}
// 整实例指纹:恢复之后要拿它证明"多写的、少写的都没有"。只按目标范围比,
// 写到范围外的行会被漏掉(真实事故:恢复报成功但目标多 67 行)。
universeBefore, err := universeFingerprint(ctx, s.pool)
if err != nil {
return libraryrelease.BackupRef{}, err
}
pg, err := ExportLibrary(ctx, s.pool, s.identity)
if err != nil {
return libraryrelease.BackupRef{}, fmt.Errorf("读取生产整库失败: %w", err)
@@ -154,6 +160,13 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间生产 %s 内容发生变化,切点不一致", table)
}
}
universeAfter, err := universeFingerprint(ctx, s.pool)
if err != nil {
return libraryrelease.BackupRef{}, err
}
if diff := universeDiff(universeBefore, universeAfter); diff != "" {
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间整实例知识域发生变化(%s),切点不一致", diff)
}
for table, want := range current.Counts {
if table == "index" {
continue
@@ -171,7 +184,7 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
total += part.Bytes
}
return libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID, Bytes: total,
CreatedAt: createdAt}, nil
CreatedAt: createdAt, Universe: universeBefore}, nil
}
func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) {
@@ -84,8 +84,16 @@ type ProductionLibraryTarget struct {
scope ProductionScope
search *SearchExporter
now func() time.Time
// storageBackend 是本次写入使用的目标存储后端 id(写入与校验共用)。
storageBackend string
// storageBackend / storageProvider 是本次写入使用的目标存储后端与提供方
// (写入与校验共用,缺一不可:只换桶不换提供方,目标取文件仍会走来源实例的路)。
storageBackend string
storageProvider string
// snapshotTenant / snapshotKB 是这次要写入的快照来源身份。写入时按它做
// 来源→目标的范围映射;校验时按它反查"目标里还有没有来源范围的行"。
snapshotTenant uint64
snapshotKB string
// targetEmbeddingModel 是目标知识库绑定的 embedding 模型 id(文档行要换成它)。
targetEmbeddingModel string
}
// resolveStorageBackend 在目标库里找本次写入的存储后端。找不到就报错:宁可不写,
@@ -98,24 +106,39 @@ func (t *ProductionLibraryTarget) resolveStorageBackend(ctx context.Context) err
if !validSearchIdentifier(t.config.ObjectsBackendID) {
return errors.New("目标存储后端 id 形状不合法")
}
t.storageBackend = t.config.ObjectsBackendID
provider, err := t.storageProviderOf(ctx, t.config.ObjectsBackendID)
if err != nil {
return err
}
t.storageBackend, t.storageProvider = t.config.ObjectsBackendID, provider
return nil
}
var id string
var id, provider string
err := t.pool.QueryRow(ctx, `
SELECT id FROM storage_backends
SELECT id, coalesce(provider,'') FROM storage_backends
WHERE tenant_id=$1 AND coalesce(config->>'bucket_name','')=$2
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id)
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id, &provider)
if err != nil {
return errors.New("目标实例没有与目标桶匹配的存储后端,拒绝写入不可解析的资源定位符")
}
if !validSearchIdentifier(id) {
return errors.New("目标存储后端 id 形状不合法")
}
t.storageBackend = id
t.storageBackend, t.storageProvider = id, provider
return nil
}
// storageProviderOf 读出目标存储后端的提供方(minio / oss / s3…)。资源行里的
// 提供方必须与它一致,否则目标应用会按来源提供方去找一个不存在的存储服务。
func (t *ProductionLibraryTarget) storageProviderOf(ctx context.Context, id string) (string, error) {
var provider string
if err := t.pool.QueryRow(ctx, `SELECT coalesce(provider,'') FROM storage_backends WHERE id=$1 AND tenant_id=$2`,
id, t.scope.TenantID).Scan(&provider); err != nil {
return "", errors.New("目标实例没有这个存储后端,拒绝写入不可解析的资源定位符")
}
return provider, nil
}
func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryConfig, scope ProductionScope) (*ProductionLibraryTarget, error) {
if err := config.validate(scope); err != nil {
return nil, err
@@ -259,6 +282,7 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
if err := t.resolveStorageBackend(ctx); err != nil {
return err
}
t.resolveEmbeddingModel(ctx)
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil {
return errors.New("归档不可用")
@@ -268,6 +292,9 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
if err != nil {
return errors.New("预发布快照不可读")
}
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
return err
}
switch stage {
case libraryrelease.StageDatabase, libraryrelease.StageWiki:
kind := "database"
@@ -356,10 +383,10 @@ func (t *ProductionLibraryTarget) insertRows(ctx context.Context, tx pgx.Tx, arc
if len(row) == 0 {
continue
}
if table == "resources" {
if err := t.applyResourceBucket(row); err != nil {
return err
}
// 行必须先映射到目标范围再落库:原样插入会把行写进来源租户/知识库,
// 目标范围的删除、校验与恢复都看不见它们。
if err := t.retargetRow(table, row); err != nil {
return err
}
if columns == nil {
columns = make([]string, 0, len(row))
@@ -408,53 +435,9 @@ type resourceLocation struct {
}
// rowTransform 是"写入时对行做过的变换"的统一入口:校验要用同一份,否则摘要比的是
// 变换前的行、目标里是变换后的行,永远不一致。
// 变换前的行、目标里是变换后的行,永远不一致。范围映射同样必须两边一致。
func (t *ProductionLibraryTarget) rowTransform(table string) func(map[string]json.RawMessage) error {
if table == "resources" {
return t.applyResourceBucket
}
return nil
}
// applyResourceBucket 把资源行定位符里的桶换成本次写入的目标桶。
// 写入与校验必须用同一个变换:只改写入侧会让"行内容摘要"对不上(真跑踩过)。
func (t *ProductionLibraryTarget) applyResourceBucket(row map[string]json.RawMessage) error {
if t.config.ObjectsBucket == "" {
return nil
}
raw, ok := row["physical_path"]
if !ok {
return errors.New("资源行缺少定位符")
}
var locator string
if err := json.Unmarshal(raw, &locator); err != nil {
return errors.New("资源定位符不可解析")
}
rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket)
if !ok {
return fmt.Errorf("资源定位符无法改写到目标桶:%s", locator)
}
encoded, err := json.Marshal(rewritten)
if err != nil {
return err
}
row["physical_path"] = encoded
// 原版应用先按 resources.storage_backend_id 找存储后端,再回落到 KB,再落默认服务。
// 这一列留着来源 id,目标仍然 500(真跑:storage backend not found → invalid path)。
if t.storageBackend == "" {
return nil
}
if raw, ok := row["storage_backend_id"]; ok {
var current string
if json.Unmarshal(raw, &current) == nil && current != "" && current != t.storageBackend {
encodedBackend, err := json.Marshal(t.storageBackend)
if err != nil {
return err
}
row["storage_backend_id"] = encodedBackend
}
}
return nil
return func(row map[string]json.RawMessage) error { return t.retargetRow(table, row) }
}
func (t *ProductionLibraryTarget) importResources(ctx context.Context, archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error {
@@ -591,6 +574,7 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err := t.resolveStorageBackend(ctx); err != nil {
return err
}
t.resolveEmbeddingModel(ctx)
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil {
return errors.New("归档不可用")
@@ -600,6 +584,9 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err != nil {
return errors.New("预发布快照不可读")
}
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
return err
}
for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" {
continue
@@ -668,6 +655,72 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
if err := t.verifyObjects(ctx, archive, manifest); err != nil {
return err
}
// 目标里不允许留下"快照来源范围"的行:那说明有写入逃出了目标范围,
// 而恢复只按目标范围回放,永远删不到它们(真实事故就是这样被漏掉的)。
if err := t.assertNoSourceScopeRows(ctx, archive, manifest); err != nil {
return err
}
return nil
}
// dropOrphanResources 清掉"本次恢复后已无任何归属"的目标租户资源行。
//
// 判据必须同时满足:属于目标租户、不在预备份快照里、没有任何 resource_binding、
// 也没有任何分块正文以 resource://<handle> 引用它。这样的行对目标应用完全不可见
// (没有知识库能按它取到文件),只可能是失败写入留下的残渣。
func (t *ProductionLibraryTarget) dropOrphanResources(ctx context.Context, backupSnapshotID string) error {
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil {
return errors.New("归档不可用")
}
defer archive.Close()
manifest, err := archive.Load(backupSnapshotID)
if err != nil {
return errors.New("预备份快照不可读")
}
kept := []string{}
for _, part := range manifest.Parts {
if part.Kind != "database" || part.Name != "resources" {
continue
}
ids, err := archivedResourceIDs(archive, part)
if err != nil {
return err
}
kept = append(kept, ids...)
}
rows, err := t.pool.Query(ctx, `
SELECT r.id FROM resources AS r
WHERE r.tenant_id=$1
AND NOT (r.id = ANY($2::text[]))
AND NOT EXISTS (SELECT 1 FROM resource_bindings b WHERE b.resource_id=r.id)
AND NOT EXISTS (SELECT 1 FROM chunks c WHERE c.tenant_id=r.tenant_id AND c.deleted_at IS NULL
AND coalesce(r.handle,'') <> ''
AND position('resource://'||r.handle IN coalesce(c.content,'')) > 0)`,
t.scope.TenantID, kept)
if err != nil {
return fmt.Errorf("查找目标残留资源失败: %w", err)
}
orphans := make([]string, 0, 8)
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
rows.Close()
return err
}
orphans = append(orphans, id)
}
rows.Close()
if err := rows.Err(); err != nil {
return err
}
if len(orphans) == 0 {
return nil
}
if _, err := t.pool.Exec(ctx, `DELETE FROM resources WHERE tenant_id=$1 AND id = ANY($2::text[])`,
t.scope.TenantID, orphans); err != nil {
return fmt.Errorf("清理目标残留资源失败: %w", err)
}
return nil
}
@@ -772,5 +825,21 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
return fmt.Errorf("恢复阶段 %s 失败: %w", stage, err)
}
}
// resources 是多知识库共享表,回放阶段不删它:本次写入新增、恢复后已经
// 没有任何归属的资源行会残留。它们对应用不可见,但属于"多写的"。
if err := t.dropOrphanResources(ctx, backup.SnapshotID); err != nil {
return err
}
// 恢复工具把四个阶段都跑完不算恢复完成:整实例必须回到预备份时刻的样子。
// 只比目标范围会把"写到范围外"的残留当成已恢复(真实事故:恢复后仍多 67 行)。
if len(backup.Universe) > 0 {
after, err := universeFingerprint(ctx, t.pool)
if err != nil {
return fmt.Errorf("恢复后读取整库指纹失败: %w", err)
}
if diff := universeDiff(backup.Universe, after); diff != "" {
return fmt.Errorf("恢复后目标与预备份时刻不一致:%s(很可能有写入逃出了预备份范围,不能当作已恢复)", diff)
}
}
return nil
}
@@ -4,12 +4,15 @@ import (
"bytes"
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"testing"
"time"
@@ -390,3 +393,339 @@ func sealTargetFixture(t *testing.T, archive *librarysnapshot.Archive, scope str
}
return id
}
// memoryObjectStore 是"真实字节"的资源存储替身:Put/Open/Stat/Copy 都按摘要核对,
// 可以强制写入失败,用来复现"资源阶段失败"这一半写场景。
type memoryObjectStore struct {
objects map[string][]byte
failPut bool
puts []string
}
func newMemoryObjectStore(objects map[string][]byte) *memoryObjectStore {
copied := map[string][]byte{}
for key, value := range objects {
copied[key] = append([]byte(nil), value...)
}
return &memoryObjectStore{objects: copied}
}
func objectKey(provider, path string) string { return provider + "|" + path }
func (s *memoryObjectStore) Put(_ context.Context, provider, path string, reader io.Reader, size int64) error {
if s.failPut {
return errors.New("对象存储不可写")
}
body, err := io.ReadAll(reader)
if err != nil || int64(len(body)) != size {
return errors.New("对象字节数与声明不符")
}
s.objects[objectKey(provider, path)] = body
s.puts = append(s.puts, path)
return nil
}
func (s *memoryObjectStore) Open(_ context.Context, provider, path string) (io.ReadCloser, error) {
body, ok := s.objects[objectKey(provider, path)]
if !ok {
return nil, errors.New("对象不存在")
}
return io.NopCloser(bytes.NewReader(body)), nil
}
func (s *memoryObjectStore) Stat(_ context.Context, provider, path string) (int64, string, error) {
body, ok := s.objects[objectKey(provider, path)]
if !ok {
return 0, "", errors.New("对象不存在")
}
digest := sha256.Sum256(body)
return int64(len(body)), hex.EncodeToString(digest[:]), nil
}
func (s *memoryObjectStore) Copy(_ context.Context, provider, source, target string) error {
body, ok := s.objects[objectKey(provider, source)]
if !ok {
return errors.New("对象不存在")
}
s.objects[objectKey(provider, target)] = append([]byte(nil), body...)
return nil
}
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同)、目标非空、资源阶段写失败。
// 旧代码把归档行原样写进目标(落在来源租户/知识库),随后的回滚只清目标范围,
// 于是"已恢复到预备份"却留下 4 文件 / 38 分块 / 21 资源 / 4 绑定。
func TestLibraryCrossScopeSyncRollbackLeavesNoResidueIsolated(t *testing.T) {
dsn, base := os.Getenv("KG_LIBRARY_TEST_DATABASE_URL"), os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
if dsn == "" || base == "" || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
t.Skip("requires isolated PG and ES opt-in")
}
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") {
t.Fatal("isolated database required")
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
t.Fatalf("open isolated PG: %v", err)
}
defer pool.Close()
for _, statement := range []string{
`DROP SCHEMA public CASCADE; CREATE SCHEMA public`,
libraryTestSchema,
`ALTER TABLE wiki_pages ADD COLUMN status text DEFAULT 'published'`,
`ALTER TABLE knowledge_tag_relations ADD COLUMN tenant_id bigint`,
`ALTER TABLE resources ADD COLUMN provider text, ADD COLUMN size bigint, ADD COLUMN storage_backend_id text`,
`ALTER TABLE resources ADD PRIMARY KEY (id)`,
`ALTER TABLE resource_bindings ADD COLUMN relation text`,
`INSERT INTO knowledge_bases VALUES('kb-target',10001,'{}',NULL),('kb-source',10000,'{}',NULL)`,
`CREATE TABLE storage_backends(id text PRIMARY KEY, tenant_id bigint, provider text, config jsonb, created_at timestamptz)`,
`INSERT INTO storage_backends VALUES('backend-target',10001,'minio','{"bucket_name":"kg-test-bucket"}',now())`,
// 目标非空:目标自己有文档/分块,另一个租户也有数据(回滚不得碰它)。
`INSERT INTO knowledges VALUES('doc-target',10001,'kb-target','completed',0,'',NULL)`,
`INSERT INTO knowledges VALUES('doc-foreign',20000,'kb-other','completed',0,'',NULL)`,
`INSERT INTO chunks VALUES('chunk-target',10001,'kb-target','doc-target','target content',NULL)`,
`INSERT INTO chunks VALUES('chunk-foreign',20000,'kb-other','doc-foreign','foreign content',NULL)`,
`UPDATE resource_bindings SET tenant_id=10000 WHERE tenant_id IS NULL`,
} {
if _, err := pool.Exec(ctx, statement); err != nil {
t.Fatalf("fixture schema: %v", err)
}
}
token := make([]byte, 8)
if _, err := rand.Read(token); err != nil {
t.Fatal(err)
}
index := "kg_library_test_" + hex.EncodeToString(token)
client := &http.Client{Timeout: 20 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
call := func(method, path, body string, want int) {
t.Helper()
request, err := http.NewRequestWithContext(ctx, method, base+path, strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
request.Header.Set("Content-Type", "application/json")
response, err := client.Do(request)
if err != nil {
t.Fatalf("fixture HTTP: %v", err)
}
defer response.Body.Close()
if response.StatusCode != want {
t.Fatalf("fixture %s %s => %d, want %d", method, path, response.StatusCode, want)
}
}
call("PUT", "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"chunk_id":{"type":"keyword"},"content":{"type":"text"},"embedding":{"type":"dense_vector","dims":2}}}}`, 200)
defer func() {
call("DELETE", "/"+index, "", 200)
call("HEAD", "/"+index, "", 404)
}()
// 目标当前索引自洽:目标分块必须有对应文档,否则预备份会(正确地)拒绝。
call("PUT", "/"+index+"/_doc/chunk-target?refresh=true",
`{"tenant_id":10001,"knowledge_base_id":"kb-target","knowledge_id":"doc-target","chunk_id":"chunk-target","source_id":"chunk-target","content":"target content","embedding":[0.1,0.2]}`, 201)
archiveRoot := t.TempDir()
if err := os.Chmod(archiveRoot, 0700); err != nil {
t.Fatal(err)
}
archive, err := librarysnapshot.Open(archiveRoot, 64*1024*1024)
if err != nil {
t.Fatal(err)
}
defer archive.Close()
snapshotID := sealCrossScopeFixture(t, archive, 10000, "kb-source")
store := newMemoryObjectStore(nil)
target, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
Postgres: dsn, SearchURL: base, SearchIndex: index, Objects: store, ObjectsReader: store,
ArchiveRoot: archiveRoot, ObjectsBucket: "kg-test-bucket", MaxDocuments: 100, HTTPClient: client,
}, ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"})
if err != nil {
t.Fatalf("target: %v", err)
}
defer target.Close()
mapping, err := target.Search().MappingSHA256(ctx)
if err != nil {
t.Fatal(err)
}
backups, err := NewProductionBackupStore(ProductionBackupConfig{
Pool: target.Pool(), Search: target.Search(), Objects: store, ArchiveRoot: archiveRoot,
Scope: "internal", Identity: target.Identity(), MaxDocuments: 100,
Compatibility: librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74,
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "text-embedding-v4",
EmbeddingDimension: 1024, IndexMappingSHA256: mapping},
})
if err != nil {
t.Fatalf("backup store: %v", err)
}
now := time.Now().UTC()
request := libraryrelease.Request{Target: "internal", SnapshotID: snapshotID,
ParamsDigest: "digest-cross-scope", Now: now,
Grant: libraryrelease.Grant{Target: "internal", SnapshotID: snapshotID, ParamsDigest: "digest-cross-scope",
ExpiresAt: now.Add(10 * time.Minute), AuthorizedBy: "acceptance"}}
// 1) 资源阶段失败 → 必须回滚且不留下来源范围的行。
store.failPut = true
outcome := libraryrelease.Run(ctx, request, target, backups, nil)
if outcome.State != "rolled_back" {
t.Fatalf("outcome = %+v", outcome)
}
if outcome.FailedAt != libraryrelease.StageResources {
t.Fatalf("failed_at = %s, want resources", outcome.FailedAt)
}
if outcome.Backup.SnapshotID == "" || len(outcome.Backup.Universe) == 0 {
t.Fatalf("backup must carry the whole-instance fingerprint: %+v", outcome.Backup)
}
for _, table := range []string{"knowledges", "chunks", "wiki_pages"} {
var residue int64
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND knowledge_base_id=$2`, table),
10000, "kb-source").Scan(&residue); err != nil {
t.Fatal(err)
}
if residue != 0 {
t.Fatalf("%s still carries %d source-scope rows after rollback", table, residue)
}
}
// 夹具在租户 10000 里本来就有自己的行(res-safe / binding-safe):它们必须原样
// 留下,只有这次同步写进去的行(res-source / 绑到 doc-source)才算残留。
for query, want := range map[string]int64{
`SELECT count(*) FROM resources WHERE id='res-source'`: 0,
`SELECT count(*) FROM resource_bindings WHERE tenant_id=10000 AND owner_id='doc-source'`: 0,
`SELECT count(*) FROM resources WHERE tenant_id=10000`: 1,
} {
var got int64
if err := pool.QueryRow(ctx, query).Scan(&got); err != nil {
t.Fatal(err)
}
if got != want {
t.Fatalf("%s = %d, want %d", query, got, want)
}
}
var leftovers int64
if err := pool.QueryRow(ctx, `SELECT count(*) FROM resources WHERE tenant_id=10001`).Scan(&leftovers); err != nil {
t.Fatal(err)
}
if leftovers != 0 {
t.Fatalf("aborted write left %d resource rows in the target tenant", leftovers)
}
var foreign int64
if err := pool.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=20000`).Scan(&foreign); err != nil {
t.Fatal(err)
}
// 夹具本来就有 1 行 other-scope(20000),加上本用例的 chunk-foreign,共 2 行。
if foreign != 2 {
t.Fatalf("cross-tenant rows must survive rollback, got %d", foreign)
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: outcome.Backup.SnapshotID}); err != nil {
t.Fatalf("rollback must verify against the prebackup: %v", err)
}
// 2) 同样的跨范围同步,资源能写 → 行必须落在目标范围,资源定位符指向目标存储。
store.failPut = false
succeed := libraryrelease.Run(ctx, request, target, backups, nil)
if succeed.State != "succeeded" {
t.Fatalf("second run outcome = %+v", succeed)
}
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
t.Fatalf("verify after cross-scope sync: %v", err)
}
var knowledgeKB, chunkKB, provider, locator string
var tenant int64
if err := pool.QueryRow(ctx, `SELECT tenant_id, knowledge_base_id FROM knowledges WHERE id='doc-source'`).
Scan(&tenant, &knowledgeKB); err != nil {
t.Fatalf("source document must exist in the target scope: %v", err)
}
if tenant != 10001 || knowledgeKB != "kb-target" {
t.Fatalf("cross-scope row landed at %d/%s", tenant, knowledgeKB)
}
if err := pool.QueryRow(ctx, `SELECT knowledge_base_id FROM chunks WHERE id='chunk-source'`).Scan(&chunkKB); err != nil {
t.Fatal(err)
}
if chunkKB != "kb-target" {
t.Fatalf("chunk landed in %s", chunkKB)
}
if err := pool.QueryRow(ctx, `SELECT provider, physical_path FROM resources WHERE id='res-source'`).
Scan(&provider, &locator); err != nil {
t.Fatal(err)
}
if provider != "minio" {
t.Fatalf("resource provider = %s, want minio", provider)
}
if !strings.Contains(locator, "storage://backend-target/minio://kg-test-bucket/10001/") {
t.Fatalf("resource locator was not retargeted: %s", locator)
}
if strings.Contains(locator, "weknora/10000") {
t.Fatalf("resource locator still points at the source tenant: %s", locator)
}
t.Logf("cross_scope_sync verified: snapshot=%s backup=%s locator=%s", snapshotID, succeed.Backup.SnapshotID, locator)
}
// sealCrossScopeFixture 造一份"来源范围与目标不同"的整库快照:文档/分块/资源/绑定/
// Wiki/索引都带来源身份,资源部件是可核对的真实字节。
func sealCrossScopeFixture(t *testing.T, archive *librarysnapshot.Archive, sourceTenant uint64, sourceKB string) string {
t.Helper()
encode := func(value any) []byte {
encoded, err := json.Marshal(value)
if err != nil {
t.Fatal(err)
}
return encoded
}
const handle = "abcdefghijklmnopqrstuv"
objectBytes := []byte("12345678")
parts := make([]librarysnapshot.Part, 0, 12)
put := func(kind, name string, count int64, body []byte) {
part, err := archive.Put(bytes.NewReader(body))
if err != nil {
t.Fatal(err)
}
part.Kind, part.Name, part.Count = kind, name, count
parts = append(parts, part)
}
put("database", "knowledges", 1, encode([]map[string]any{{
"id": "doc-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
"parse_status": "completed", "pending_subtasks_count": 0, "file_path": ""}}))
put("database", "chunks", 1, encode([]map[string]any{{
"id": "chunk-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
"knowledge_id": "doc-source", "content": "见 resource://" + handle}}))
put("database", "knowledge_tags", 0, encode([]map[string]any{}))
put("database", "knowledge_tag_relations", 0, encode([]map[string]any{}))
put("database", "resources", 1, encode([]map[string]any{{
"id": "res-source", "handle": handle, "tenant_id": sourceTenant,
"storage_backend_id": "backend-source", "provider": "oss",
"physical_path": "storage://backend-source/oss://source-bucket/weknora/" +
formatTenant(sourceTenant) + "/res-source/a.png",
"size": int64(len(objectBytes))}}))
put("database", "resource_bindings", 1, encode([]map[string]any{{
"id": "bind-source", "resource_id": "res-source", "tenant_id": sourceTenant,
"owner_type": "knowledge", "owner_id": "doc-source", "relation": "attachment"}}))
put("resource", "res-source", 1, objectBytes)
put("wiki", "wiki_folders", 0, encode([]map[string]any{}))
put("wiki", "wiki_pages", 1, encode([]map[string]any{{
"id": "wiki-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
"slug": "index", "content": "wiki"}}))
for _, name := range []string{"wiki_page_issues", "wiki_log_entries"} {
put("wiki", name, 0, encode([]map[string]any{}))
}
put("index", "chunks", 1, encode([]map[string]any{{"id": "chunk-source", "source": map[string]any{
"tenant_id": sourceTenant, "knowledge_base_id": sourceKB, "knowledge_id": "doc-source",
"chunk_id": "chunk-source", "source_id": "chunk-source", "content": "见 resource://" + handle,
"embedding": []float64{0.1, 0.2}}}}))
manifest := librarysnapshot.Manifest{ValidationPolicy: librarysnapshot.ArtifactAcceptancePolicy,
Format: librarysnapshot.Format, Scope: "internal", TenantID: sourceTenant, KnowledgeBaseID: sourceKB,
CreatedAt: time.Now().UTC(), Parts: parts,
Compatibility: librarysnapshot.Compatibility{
WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8",
EmbeddingModel: "text-embedding-v4", EmbeddingDimension: 1024,
IndexMappingSHA256: strings.Repeat("a", 64),
}}
id, err := archive.Seal(manifest)
if err != nil {
t.Fatalf("seal cross-scope fixture: %v", err)
}
return id
}
func formatTenant(tenant uint64) string { return strconv.FormatUint(tenant, 10) }
@@ -0,0 +1,334 @@
package v071
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"knowledge-governance/internal/librarysnapshot"
)
// ResidueReport 是"清理范围外残留"的审计记录:清掉了什么、依据是哪份归档、
// 备份写到哪儿、清理后目标处于什么状态。没有这份记录就不许动目标。
type ResidueReport struct {
SnapshotID string `json:"snapshot_id"`
SourceTenant uint64 `json:"source_tenant"`
SourceKB string `json:"source_kb"`
Tables map[string]int `json:"tables"`
BackupPath string `json:"backup_path"`
BackupSHA256 string `json:"backup_sha256"`
Applied bool `json:"applied"`
Cleared int `json:"cleared"`
Verified bool `json:"verified"`
Foreign map[string]string `json:"foreign_scopes,omitempty"`
}
// ClearForeignScope 清掉"归档里确实有、却仍以来源身份躺在目标里"的行。
//
// 它只做这一件事,而且必须全部条件成立才允许删:
// 1. 目标这些行的内容与指定归档行逐字段一致(先证明它们是这次写入的副本,
// 不是目标自己的数据);
// 2. 先把它们完整导出成 0600 的备份文件并记下摘要;
// 3. 删除只按"归档身份集合 + 来源租户"精确命中,不做范围广删。
//
// 清完还要复核:来源身份不再存在、目标内容与预备份一致、整库指纹里没有别的租户。
func (t *ProductionLibraryTarget) ClearForeignScope(ctx context.Context, source ProductionScope, snapshotID,
backupDir string, apply bool) (*ResidueReport, error) {
if source.TenantID == 0 || !validSearchIdentifier(source.KnowledgeBaseID) {
return nil, errors.New("来源范围不完整,拒绝清理")
}
if source.TenantID == t.scope.TenantID && source.KnowledgeBaseID == t.scope.KnowledgeBaseID {
return nil, errors.New("来源与目标范围相同,不需要清理")
}
if backupDir == "" {
return nil, errors.New("清理前必须指定备份目录")
}
if err := t.resolveStorageBackend(ctx); err != nil {
return nil, err
}
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
if err != nil {
return nil, errors.New("归档不可用")
}
defer archive.Close()
manifest, err := archive.Load(snapshotID)
if err != nil {
return nil, errors.New("指定快照不可读")
}
report := &ResidueReport{SnapshotID: snapshotID, SourceTenant: source.TenantID, SourceKB: source.KnowledgeBaseID,
Tables: map[string]int{}, Applied: false}
backup := map[string][]map[string]json.RawMessage{}
digests := map[string]string{}
identities := map[string][]string{}
for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" {
continue
}
archived, archivedDigests, err := archivedRowsWithDigests(archive, part)
if err != nil {
return nil, err
}
// 事故里的残留行是"归档行 + 只把桶/后端换成目标"的旧变换结果:判等时必须
// 同时接受原样与这一种变换,否则会把真残留当成"不是这次写的"而拒绝清理。
if part.Name == "resources" {
for _, row := range archived {
legacy, legacyErr := t.legacyResourceRow(row)
if legacyErr != nil {
return nil, legacyErr
}
digest, digestErr := canonicalRowDigest(legacy)
if digestErr != nil {
return nil, digestErr
}
archivedDigests[digest] = true
}
}
if len(archived) == 0 {
continue
}
live, err := t.foreignScopeRows(ctx, part.Name, source)
if err != nil {
return nil, err
}
liveDigests := make(map[string]string, len(live))
for _, row := range live {
digest, err := canonicalRowDigest(row)
if err != nil {
return nil, err
}
liveDigests[digest] = archiveRowIdentity(part.Name, row)
}
// 目标里这一范围的行必须都能在归档里找到同样内容:否则它们不是这次写入的
// 副本,删了就是删别人的数据。
rows := make([]map[string]json.RawMessage, 0, len(live))
ids := make([]string, 0, len(live))
for digest, identity := range liveDigests {
if _, ok := archivedDigests[digest]; !ok {
return nil, fmt.Errorf("目标 %s 的行 %s 与指定归档不一致,拒绝按本次写入清理", part.Name, identity)
}
ids = append(ids, identity)
}
for _, row := range live {
rows = append(rows, row)
}
sort.Strings(ids)
report.Tables[part.Name] = len(rows)
backup[part.Name] = rows
identities[part.Name] = ids
digests[part.Name] = combineRowDigests(sortedKeys(liveDigests))
}
total := 0
for _, count := range report.Tables {
total += count
}
if total == 0 {
report.Verified = true
return report, nil
}
if err := os.MkdirAll(backupDir, 0o700); err != nil {
return nil, fmt.Errorf("创建备份目录失败: %w", err)
}
payload, err := json.MarshalIndent(map[string]any{
"snapshot_id": snapshotID, "source_tenant": source.TenantID, "source_kb": source.KnowledgeBaseID,
"target_tenant": t.scope.TenantID, "target_kb": t.scope.KnowledgeBaseID,
"tables": report.Tables, "rows": backup, "digests": digests,
}, "", " ")
if err != nil {
return nil, err
}
report.BackupPath = filepath.Join(backupDir, "foreign-scope-residue-"+snapshotID[:12]+".json")
if err := os.WriteFile(report.BackupPath, payload, 0o600); err != nil {
return nil, fmt.Errorf("写残留备份失败: %w", err)
}
sum := sha256.Sum256(payload)
report.BackupSHA256 = hex.EncodeToString(sum[:])
if !apply {
return report, nil
}
// 删除顺序:子表先、父表后(resource_bindings → resources,chunks → knowledges)。
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
table := productionLibraryTables[index]
ids, ok := identities[table]
if !ok || len(ids) == 0 {
continue
}
statement := fmt.Sprintf(`DELETE FROM %s WHERE id = ANY($1::text[])`, table)
if !tableHasTenant(table) {
// knowledge_tag_relations 没有 id:身份就是父文档 + 标签。
statement = `DELETE FROM knowledge_tag_relations WHERE (knowledge_id||':'||tag_id) = ANY($1::text[])`
}
if _, err := t.pool.Exec(ctx, statement, ids); err != nil {
return nil, fmt.Errorf("清理 %s 失败: %w", table, err)
}
}
report.Applied = true
report.Cleared = total
return report, nil
}
// legacyResourceRow 复现旧实现写资源行时的变换:只换定位符里的存储后端与桶,
// 租户/知识库/provider 原样保留。判等要认这一种形状,事故残留才是"可证明"的。
func (t *ProductionLibraryTarget) legacyResourceRow(row map[string]json.RawMessage) (map[string]json.RawMessage, error) {
copied := make(map[string]json.RawMessage, len(row))
for key, value := range row {
copied[key] = value
}
if t.config.ObjectsBucket == "" {
return copied, nil
}
raw, ok := copied["physical_path"]
if !ok {
return nil, errors.New("资源行缺少定位符")
}
var locator string
if err := json.Unmarshal(raw, &locator); err != nil {
return nil, errors.New("资源定位符不可解析")
}
if rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket); ok {
encoded, err := json.Marshal(rewritten)
if err != nil {
return nil, err
}
copied["physical_path"] = encoded
}
if t.storageBackend == "" {
return copied, nil
}
if raw, ok := copied["storage_backend_id"]; ok {
var current string
if json.Unmarshal(raw, &current) == nil && current != "" && current != t.storageBackend {
encoded, err := json.Marshal(t.storageBackend)
if err != nil {
return nil, err
}
copied["storage_backend_id"] = encoded
}
}
return copied, nil
}
// VerifyResidueCleared 复核清理结果:来源身份不再存在、其它租户的解释得清、
// 且(可选)目标内容与预备份完全一致。
func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, source ProductionScope) error {
for _, table := range productionLibraryTables {
if !tableHasTenant(table) {
continue
}
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1`, table)
arguments := []any{source.TenantID}
if tablesWithKB[table] {
statement += " AND knowledge_base_id=$2"
arguments = append(arguments, source.KnowledgeBaseID)
}
var count int64
if err := t.pool.QueryRow(ctx, statement, arguments...).Scan(&count); err != nil {
return fmt.Errorf("复核 %s 失败: %w", table, err)
}
if count > 0 {
return fmt.Errorf("目标 %s 仍有 %d 行属于来源范围 %d/%s", table, count, source.TenantID, source.KnowledgeBaseID)
}
}
return nil
}
// ForeignScopes 列出目标里所有"不是目标租户"的租户,用于人工确认没有别的泄漏。
func (t *ProductionLibraryTarget) ForeignScopes(ctx context.Context) (map[string]string, error) {
foreign := map[string]string{}
for _, table := range productionLibraryTables {
if !tableHasTenant(table) {
continue
}
rows, err := t.pool.Query(ctx, fmt.Sprintf(
`SELECT tenant_id, count(*) FROM %s WHERE tenant_id <> $1 GROUP BY 1`, table), t.scope.TenantID)
if err != nil {
return nil, fmt.Errorf("读取 %s 租户分布失败: %w", table, err)
}
for rows.Next() {
var tenant, count int64
if err := rows.Scan(&tenant, &count); err != nil {
rows.Close()
return nil, err
}
foreign[fmt.Sprintf("%s:%d", table, tenant)] = fmt.Sprintf("%d 行", count)
}
rows.Close()
if err := rows.Err(); err != nil {
return nil, err
}
}
return foreign, nil
}
// foreignScopeRows 读出目标里属于来源范围的行(resources / resource_bindings
// 只有租户列,按租户取;其余按租户+知识库取)。
func (t *ProductionLibraryTarget) foreignScopeRows(ctx context.Context, table string, source ProductionScope) (
[]map[string]json.RawMessage, error) {
if !tableHasTenant(table) {
return nil, nil
}
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table))
arguments := []any{source.TenantID, source.KnowledgeBaseID}
if table == "resources" || table == "resource_bindings" {
statement = fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE tenant_id=$1`, table, table)
arguments = arguments[:1]
}
rows, err := t.pool.Query(ctx, statement, arguments...)
if err != nil {
return nil, fmt.Errorf("读取目标 %s 来源范围行失败: %w", table, err)
}
defer rows.Close()
result := make([]map[string]json.RawMessage, 0, 8)
for rows.Next() {
var row map[string]json.RawMessage
if err := rows.Scan(&row); err != nil {
return nil, err
}
result = append(result, row)
}
return result, rows.Err()
}
// archivedRowsWithDigests 读出一份归档部件的行与规范化内容摘要。
func archivedRowsWithDigests(archive *librarysnapshot.Archive, part librarysnapshot.Part) (
[]map[string]json.RawMessage, map[string]bool, error) {
reader, err := archive.OpenPart(part)
if err != nil {
return nil, nil, err
}
defer reader.Close()
decoder := json.NewDecoder(reader)
token, err := decoder.Token()
if err != nil || token != json.Delim('[') {
return nil, nil, errors.New("归档部件不是行数组")
}
rows := make([]map[string]json.RawMessage, 0, 64)
digests := map[string]bool{}
for decoder.More() {
var row map[string]json.RawMessage
if err := decoder.Decode(&row); err != nil {
return nil, nil, err
}
digest, err := canonicalRowDigest(row)
if err != nil {
return nil, nil, err
}
digests[digest] = true
rows = append(rows, row)
}
return rows, digests, nil
}
func sortedKeys(values map[string]string) []string {
keys := make([]string, 0, len(values))
for key := range values {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
@@ -0,0 +1,46 @@
package v071
import (
"context"
"errors"
"fmt"
"github.com/jackc/pgx/v5/pgxpool"
)
// ScopeContentCounts 数一个知识库在各内容表里的行数(数据库侧事实)。
//
// 为什么不能只数 knowledges:文档被删掉、分块与 Wiki/资源还在的库并不为空。
// 只数一张表会把这种目标判成"首次发布的空库",从而让一条本该走基线初始化的
// 路径绕过去;这里用与整库读写完全相同的表清单与过滤条件,保证"空"的定义只有一份。
func ScopeContentCounts(ctx context.Context, pool *pgxpool.Pool, identity ProductionScope) (map[string]int64, error) {
if pool == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) {
return nil, errors.New("invalid scope identity")
}
counts := make(map[string]int64, len(productionLibraryTables))
for _, table := range productionLibraryTables {
var count int64
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`, table, scopeWhere(table)),
identity.TenantID, identity.KnowledgeBaseID).Scan(&count); err != nil {
return nil, fmt.Errorf("count %s: %w", table, err)
}
counts[table] = count
}
return counts, nil
}
// ScopeContentEmpty 只有每张内容表都是 0 才算空目标,并交回逐表行数作为证据。
// 注意它**不**包含检索索引:索引由分块派生,分块为空而索引非空属于不一致,
// 应在执行侧(预备份/基线捕获)报出来,不能在这里被当成"空"。
func ScopeContentEmpty(ctx context.Context, pool *pgxpool.Pool, identity ProductionScope) (bool, map[string]int64, error) {
counts, err := ScopeContentCounts(ctx, pool, identity)
if err != nil {
return false, nil, err
}
for _, count := range counts {
if count != 0 {
return false, counts, nil
}
}
return true, counts, nil
}
@@ -0,0 +1,262 @@
package v071
import (
"context"
"encoding/json"
"errors"
"fmt"
"strconv"
"strings"
)
// 整库同步是"把来源知识库的内容搬到目标的租户/知识库",但归档行带着来源身份
// (tenant_id / knowledge_base_id)。原样插入会让行落在来源范围里:目标范围的
// 删除、校验、预备份与恢复全都看不见它们(真实事故:同步失败后"已恢复到预备份",
// 目标却多出 4 文件 / 38 分块 / 21 资源 / 4 绑定,行所属租户还是来源租户)。
//
// 因此写入与校验必须共用这一个改写入口,并且对不上就失败——宁可不写,也不写
// 一行目标范围外的数据。
// tablesWithKB 是真实 WeKnora v0.7.1 里带 knowledge_base_id 的知识域表。
// resources / resource_bindings 是多知识库共享表,只有 tenant_id;
// knowledge_tag_relations 两列都没有,归属完全由父表 knowledges 决定。
var tablesWithKB = map[string]bool{
"knowledges": true, "chunks": true, "knowledge_tags": true,
"wiki_folders": true, "wiki_pages": true, "wiki_page_issues": true, "wiki_log_entries": true,
}
// tableHasTenant 报告这张表是否自带租户列。
func tableHasTenant(table string) bool { return table != "knowledge_tag_relations" }
// bindSnapshotScope 记下快照的来源身份。没有来源身份就无法判断"这行要搬到哪个
// 范围",宁可失败也不要按行内容猜。
func (t *ProductionLibraryTarget) bindSnapshotScope(manifest manifestScope) error {
if manifest.tenantID == 0 || !validSearchIdentifier(manifest.knowledgeBaseID) {
return errors.New("快照缺少来源范围身份(tenant_id / knowledge_base_id),无法确定目标范围映射")
}
t.snapshotTenant = manifest.tenantID
t.snapshotKB = manifest.knowledgeBaseID
return nil
}
// manifestScope 只是把快照的两个范围字段抽出来,避免调用点重复解构。
type manifestScope struct {
tenantID uint64
knowledgeBaseID string
}
// retargetRow 把一行归档数据改写成目标范围的数据:tenant_id / knowledge_base_id
// 换成目标身份,资源行还要把定位符、存储后端、提供方换成目标实例自己的值。
func (t *ProductionLibraryTarget) retargetRow(table string, row map[string]json.RawMessage) error {
if !tableHasTenant(table) {
return nil
}
if err := t.retargetTenantField(row); err != nil {
return fmt.Errorf("%s:%w", table, err)
}
if tablesWithKB[table] {
if err := t.retargetKnowledgeBaseField(row); err != nil {
return fmt.Errorf("%s:%w", table, err)
}
}
if table == "resources" {
return t.rewriteResourceIdentity(row)
}
if table == "knowledges" {
return t.retargetEmbeddingModelField(row)
}
return nil
}
// retargetEmbeddingModelField 把文档的 embedding 模型换成目标知识库绑定的那一个。
// 来源模型 id 在目标实例里通常不存在(目标 models 表可能是空的),留着它等于让
// 应用拿一个解析不到的模型去查——库里的向量与查询向量就不是同一套了。
func (t *ProductionLibraryTarget) retargetEmbeddingModelField(row map[string]json.RawMessage) error {
if t.targetEmbeddingModel == "" {
return nil
}
if _, ok := row["embedding_model_id"]; !ok {
return nil
}
encoded, err := json.Marshal(t.targetEmbeddingModel)
if err != nil {
return errors.New("编码目标 embedding 模型失败")
}
row["embedding_model_id"] = encoded
return nil
}
// resolveEmbeddingModel 读目标知识库自己的 embedding 模型 id。读不到不算错误
// (目标可能确实没绑模型),此时保持来源值不动,由契约校验去发现不一致。
func (t *ProductionLibraryTarget) resolveEmbeddingModel(ctx context.Context) {
var model string
if err := t.pool.QueryRow(ctx,
`SELECT coalesce(embedding_model_id,'') FROM knowledge_bases WHERE id=$1 AND tenant_id=$2`,
t.scope.KnowledgeBaseID, t.scope.TenantID).Scan(&model); err == nil {
t.targetEmbeddingModel = model
}
}
func (t *ProductionLibraryTarget) retargetTenantField(row map[string]json.RawMessage) error {
raw, ok := row["tenant_id"]
if !ok {
return errors.New("行缺少 tenant_id,无法确定要写入哪个范围")
}
var current uint64
if err := json.Unmarshal(raw, &current); err != nil {
return errors.New("行 tenant_id 不可解析")
}
if t.snapshotTenant == 0 {
return errors.New("快照来源租户未知,拒绝猜测目标租户")
}
if current != t.snapshotTenant {
return fmt.Errorf("行 tenant_id=%d 不属于快照来源租户 %d", current, t.snapshotTenant)
}
encoded, err := json.Marshal(t.scope.TenantID)
if err != nil {
return errors.New("编码目标租户失败")
}
row["tenant_id"] = encoded
return nil
}
func (t *ProductionLibraryTarget) retargetKnowledgeBaseField(row map[string]json.RawMessage) error {
raw, ok := row["knowledge_base_id"]
if !ok {
return errors.New("行缺少 knowledge_base_id,无法确定要写入哪个知识库")
}
var current string
if err := json.Unmarshal(raw, &current); err != nil || current == "" {
return errors.New("行 knowledge_base_id 不可解析")
}
if t.snapshotKB == "" {
return errors.New("快照来源知识库未知,拒绝猜测目标知识库")
}
if current != t.snapshotKB {
return fmt.Errorf("行 knowledge_base_id=%s 不属于快照来源知识库 %s", current, t.snapshotKB)
}
encoded, err := json.Marshal(t.scope.KnowledgeBaseID)
if err != nil {
return errors.New("编码目标知识库失败")
}
row["knowledge_base_id"] = encoded
return nil
}
// rewriteResourceIdentity 把资源行的定位符与提供方换成目标实例的值。
//
// 只换桶不够:来源行的 scheme(oss/minio…)与对象键里的租户段也是来源实例的,
// 目标实例按它取文件会落到别的提供方或别的租户前缀上(真跑:storage backend
// not found → invalid path → 500 code1007)。
func (t *ProductionLibraryTarget) rewriteResourceIdentity(row map[string]json.RawMessage) error {
raw, ok := row["physical_path"]
if !ok {
return errors.New("资源行缺少定位符")
}
var locator string
if err := json.Unmarshal(raw, &locator); err != nil {
return errors.New("资源定位符不可解析")
}
rewritten, err := retargetLocator(locator, t.storageBackend, t.storageProvider, t.config.ObjectsBucket,
t.snapshotTenant, t.scope.TenantID)
if err != nil {
return err
}
if rewritten != locator {
encoded, err := json.Marshal(rewritten)
if err != nil {
return errors.New("编码目标定位符失败")
}
row["physical_path"] = encoded
}
if t.storageBackend != "" {
if _, ok := row["storage_backend_id"]; ok {
encoded, err := json.Marshal(t.storageBackend)
if err != nil {
return errors.New("编码目标存储后端失败")
}
row["storage_backend_id"] = encoded
}
}
if t.storageProvider != "" {
if _, ok := row["provider"]; ok {
encoded, err := json.Marshal(t.storageProvider)
if err != nil {
return errors.New("编码目标提供方失败")
}
row["provider"] = encoded
}
}
return nil
}
// retargetLocator 把 "storage://<backend>/<scheme>://<bucket>/<key>" 整体换成目标
// 实例的真实取值:后端 id、scheme/提供方、桶名,以及对象键里的租户段。
// 桶名为空表示目标没有配置对象存储(测试夹具),此时不动定位符。
func retargetLocator(locator, backendID, provider, bucket string, sourceTenant, targetTenant uint64) (string, error) {
if bucket == "" || locator == "" {
return locator, nil
}
if locator != strings.TrimSpace(locator) {
return "", errors.New("资源定位符带首尾空白")
}
prefix := ""
rest := locator
if strings.HasPrefix(locator, "storage://") {
trimmed := strings.TrimPrefix(locator, "storage://")
parts := strings.SplitN(trimmed, "/", 2)
if len(parts) != 2 || !validSearchIdentifier(parts[0]) || parts[1] == "" {
return "", errors.New("资源定位符结构不可解析")
}
if backendID != "" {
if !validSearchIdentifier(backendID) {
return "", errors.New("目标存储后端 id 形状不合法")
}
parts[0] = backendID
}
prefix, rest = "storage://"+parts[0]+"/", parts[1]
}
separator := strings.Index(rest, "://")
if separator < 0 {
// 裸对象键没有桶与租户段:落盘按配置桶,改写无意义。
return locator, nil
}
scheme := rest[:separator]
if !validSearchIdentifier(scheme) {
return "", errors.New("资源定位符的提供方不可解析")
}
schemeless := rest[separator+3:]
boundary := strings.Index(schemeless, "/")
if boundary <= 0 || boundary == len(schemeless)-1 {
return "", errors.New("资源定位符缺少对象键")
}
key := schemeless[boundary+1:]
if provider != "" {
if !validSearchIdentifier(provider) {
return "", errors.New("目标提供方形状不合法")
}
scheme = provider
}
return prefix + scheme + "://" + bucket + "/" + retargetKeyTenant(key, sourceTenant, targetTenant), nil
}
// retargetKeyTenant 把对象键里的来源租户段换成目标租户,并丢掉来源实例自己的
// 路径前缀(例如 MINIO_PATH_PREFIX=weknora/),让键与目标实例的写法一致。
// 找不到来源租户段时原样返回:此时键本身不含跨租户引用。
func retargetKeyTenant(key string, sourceTenant, targetTenant uint64) string {
if sourceTenant == 0 || targetTenant == 0 || sourceTenant == targetTenant {
return key
}
source, target := strconv.FormatUint(sourceTenant, 10), strconv.FormatUint(targetTenant, 10)
segments := strings.Split(key, "/")
for index := 0; index < len(segments) && index < 2; index++ {
// 只有"后面还有内容"的段才是路径前缀;整键就是租户号时不该动它。
if index >= len(segments)-1 || segments[index] != source {
continue
}
kept := segments[index:]
kept[0] = target
return strings.Join(kept, "/")
}
return key
}
@@ -0,0 +1,194 @@
package v071
import (
"encoding/json"
"strings"
"testing"
)
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同),归档行原样写进目标,
// 目标范围的删除/校验/恢复都看不到它们。这里把"必须改写哪些列"钉死。
func scopeTargetFixture() *ProductionLibraryTarget {
return &ProductionLibraryTarget{
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"},
config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
storageBackend: "backend-target",
storageProvider: "minio",
snapshotTenant: 10000,
snapshotKB: "kb-source",
}
}
func rawRow(t *testing.T, fields map[string]any) map[string]json.RawMessage {
t.Helper()
row := make(map[string]json.RawMessage, len(fields))
for key, value := range fields {
encoded, err := json.Marshal(value)
if err != nil {
t.Fatalf("encode %s: %v", key, err)
}
row[key] = encoded
}
return row
}
func fieldText(t *testing.T, row map[string]json.RawMessage, key string) string {
t.Helper()
var value string
if err := json.Unmarshal(row[key], &value); err != nil {
t.Fatalf("%s not a string: %v", key, err)
}
return value
}
func fieldUint(t *testing.T, row map[string]json.RawMessage, key string) uint64 {
t.Helper()
var value uint64
if err := json.Unmarshal(row[key], &value); err != nil {
t.Fatalf("%s not a number: %v", key, err)
}
return value
}
func TestRetargetRowMapsSourceScopeToTarget(t *testing.T) {
target := scopeTargetFixture()
for _, table := range []string{"knowledges", "chunks", "knowledge_tags", "wiki_pages", "wiki_folders"} {
row := rawRow(t, map[string]any{"id": "row-1", "tenant_id": 10000, "knowledge_base_id": "kb-source"})
if err := target.retargetRow(table, row); err != nil {
t.Fatalf("%s retarget: %v", table, err)
}
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
t.Fatalf("%s tenant_id = %d, want 10001", table, got)
}
if got := fieldText(t, row, "knowledge_base_id"); got != "kb-target" {
t.Fatalf("%s knowledge_base_id = %s, want kb-target", table, got)
}
}
// 共享表只有租户列:不得凭空写一个 knowledge_base_id 出来。
binding := rawRow(t, map[string]any{"id": "bind-1", "tenant_id": 10000, "owner_type": "knowledge", "owner_id": "row-1"})
if err := target.retargetRow("resource_bindings", binding); err != nil {
t.Fatalf("resource_bindings retarget: %v", err)
}
if got := fieldUint(t, binding, "tenant_id"); got != 10001 {
t.Fatalf("binding tenant_id = %d, want 10001", got)
}
if _, ok := binding["knowledge_base_id"]; ok {
t.Fatal("resource_bindings must not gain a knowledge_base_id column")
}
// knowledge_tag_relations 没有范围列:归属走父表,改写必须是空操作。
relation := rawRow(t, map[string]any{"knowledge_id": "row-1", "tag_id": "tag-1"})
if err := target.retargetRow("knowledge_tag_relations", relation); err != nil {
t.Fatalf("knowledge_tag_relations retarget: %v", err)
}
if len(relation) != 2 {
t.Fatalf("knowledge_tag_relations must stay untouched, got %v", relation)
}
}
// 文档行的 embedding 模型必须换成目标知识库绑定的那一个:来源模型 id 在目标
// 实例里通常不存在,留着它检索时的查询向量与库里向量就不是同一套。
func TestRetargetRowUsesTargetEmbeddingModel(t *testing.T) {
target := scopeTargetFixture()
target.targetEmbeddingModel = "model-target"
row := rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
"embedding_model_id": "model-source"})
if err := target.retargetRow("knowledges", row); err != nil {
t.Fatalf("retarget: %v", err)
}
if got := fieldText(t, row, "embedding_model_id"); got != "model-target" {
t.Fatalf("embedding_model_id = %s, want model-target", got)
}
// 目标没绑模型时不猜:保持来源值,交给契约校验去发现不一致。
plain := scopeTargetFixture()
row = rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
"embedding_model_id": "model-source"})
if err := plain.retargetRow("knowledges", row); err != nil {
t.Fatalf("retarget without target model: %v", err)
}
if got := fieldText(t, row, "embedding_model_id"); got != "model-source" {
t.Fatalf("embedding_model_id = %s, want unchanged", got)
}
}
func TestRetargetRowFailsClosedOutsideSnapshotScope(t *testing.T) {
target := scopeTargetFixture()
cases := map[string]map[string]any{
"foreign tenant": {"id": "x", "tenant_id": 20000, "knowledge_base_id": "kb-source"},
"foreign knowledge": {"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb-other"},
"missing tenant": {"id": "x", "knowledge_base_id": "kb-source"},
"missing knowledge kb": {"id": "x", "tenant_id": 10000},
}
for name, fields := range cases {
if err := target.retargetRow("knowledges", rawRow(t, fields)); err == nil {
t.Fatalf("%s: retarget must fail closed", name)
}
}
// 快照来源未知时不许猜目标范围。
unknown := &ProductionLibraryTarget{scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"}}
if err := unknown.retargetRow("knowledges", rawRow(t, map[string]any{"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb"})); err == nil {
t.Fatal("unknown snapshot scope must fail closed")
}
}
func TestRetargetRowRewritesResourceStorageIdentity(t *testing.T) {
target := scopeTargetFixture()
row := rawRow(t, map[string]any{
"id": "res-1",
"tenant_id": 10000,
"storage_backend_id": "backend-source",
"provider": "oss",
"physical_path": "storage://backend-source/oss://yzh-weknora-test/weknora/10000/res-1/a.png",
})
if err := target.retargetRow("resources", row); err != nil {
t.Fatalf("retarget resource: %v", err)
}
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
t.Fatalf("resource tenant = %d, want 10001", got)
}
want := "storage://backend-target/minio://kgseal-1029a/10001/res-1/a.png"
if got := fieldText(t, row, "physical_path"); got != want {
t.Fatalf("physical_path = %q, want %q", got, want)
}
if got := fieldText(t, row, "provider"); got != "minio" {
t.Fatalf("provider = %q, want minio", got)
}
if got := fieldText(t, row, "storage_backend_id"); got != "backend-target" {
t.Fatalf("storage_backend_id = %q, want backend-target", got)
}
}
func TestRetargetKeyTenantDropsSourcePrefix(t *testing.T) {
cases := []struct{ key, want string }{
{"weknora/10000/exports/a.png", "10001/exports/a.png"},
{"10000/knowledge/file.md", "10001/knowledge/file.md"},
{"other/prefix/a.png", "other/prefix/a.png"},
{"10000", "10000"},
}
for _, tc := range cases {
if got := retargetKeyTenant(tc.key, 10000, 10001); got != tc.want {
t.Fatalf("retargetKeyTenant(%q) = %q, want %q", tc.key, got, tc.want)
}
}
// 同租户不编码:键必须原样返回。
if got := retargetKeyTenant("weknora/10000/a.png", 10000, 10000); got != "weknora/10000/a.png" {
t.Fatalf("same tenant must keep key, got %q", got)
}
}
func TestRetargetLocatorRejectsMalformed(t *testing.T) {
if _, err := retargetLocator(" not-a-locator ", "backend", "minio", "bucket", 10000, 10001); err == nil {
t.Fatal("locator with surrounding spaces must be rejected")
}
if _, err := retargetLocator("storage://backend/oss://", "backend", "minio", "bucket", 10000, 10001); err == nil {
t.Fatal("locator without object key must be rejected")
}
// 没有目标桶(测试夹具)时不改写,保持原值可继续用于采集/校验。
locator := "storage://backend/oss://bucket/key"
if got, err := retargetLocator(locator, "backend", "minio", "", 10000, 10001); err != nil || got != locator {
t.Fatalf("empty bucket must be a no-op, got %q err=%v", got, err)
}
if !strings.Contains(retargetKeyTenant("weknora/10000/a", 10000, 10001), "10001") {
t.Fatal("tenant segment must be replaced")
}
}
@@ -0,0 +1,126 @@
package v071
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/libraryrelease"
"knowledge-governance/internal/librarysnapshot"
)
// universeFingerprint 是"整实例"的知识域指纹:不限租户/知识库的行数 + 身份集合摘要。
//
// 为什么不能只比目标范围:恢复只按目标范围删除/回放,写到范围外的残留它看不见
// (真实事故:作业报"已恢复到预备份",目标却多出 4 文件 / 38 分块 / 21 资源 /
// 4 绑定)。整实例指纹让"多写了什么"必然暴露,同时也能证明恢复没有误删别的
// 知识库的行。
func universeFingerprint(ctx context.Context, pool *pgxpool.Pool) (map[string]libraryrelease.TableFingerprint, error) {
result := make(map[string]libraryrelease.TableFingerprint, len(productionLibraryTables))
for _, table := range productionLibraryTables {
identity := identityExpr(table)
statement := fmt.Sprintf(
`SELECT count(*), coalesce(md5(string_agg((%s)::text, E'\n' ORDER BY (%s)::text)), '') FROM %s`,
identity, identity, table)
var fingerprint libraryrelease.TableFingerprint
if err := pool.QueryRow(ctx, statement).Scan(&fingerprint.Count, &fingerprint.Digest); err != nil {
return nil, fmt.Errorf("读取整库 %s 指纹失败: %w", table, err)
}
result[table] = fingerprint
}
return result, nil
}
// archivedRowIdentities 读出一份归档部件的行身份(与校验用的身份口径一致)。
func archivedRowIdentities(archive *librarysnapshot.Archive, part librarysnapshot.Part) ([]string, error) {
reader, err := archive.OpenPart(part)
if err != nil {
return nil, err
}
defer reader.Close()
decoder := json.NewDecoder(reader)
token, err := decoder.Token()
if err != nil || token != json.Delim('[') {
return nil, errors.New("归档部件不是行数组")
}
ids := make([]string, 0, 64)
for decoder.More() {
var row map[string]json.RawMessage
if err := decoder.Decode(&row); err != nil {
return nil, err
}
identity := archiveRowIdentity(part.Name, row)
if identity == "" {
continue
}
ids = append(ids, identity)
}
return ids, nil
}
// universeDiff 返回第一处不一致的表;空字符串表示整实例与参照完全一致。
func universeDiff(want, got map[string]libraryrelease.TableFingerprint) string {
tables := make([]string, 0, len(want))
for table := range want {
tables = append(tables, table)
}
sort.Strings(tables)
for _, table := range tables {
current, ok := got[table]
if !ok {
return fmt.Sprintf("%s(目标读不到)", table)
}
if current.Count != want[table].Count {
return fmt.Sprintf("%s 行数 %d,参照 %d", table, current.Count, want[table].Count)
}
if current.Digest != want[table].Digest {
return fmt.Sprintf("%s 身份集合与参照不同(同数量改写也会被这条拦住)", table)
}
}
return ""
}
// assertNoSourceScopeRows 找出"这次要写的这些行,仍然以来源身份躺在目标里"。
//
// 判据按归档里的身份集合逐表核对,而不是"这个租户/知识库有没有行":同一个租户
// 里本来就可能住着别的知识库(共享资源、别的 KB 的绑定),按范围数会把它们误
// 报成残留。这里只认"本次写集合里、仍属于来源租户"的行——它们只可能来自逃出
// 目标范围的写入,而恢复只按目标范围回放,永远删不到它们。
func (t *ProductionLibraryTarget) assertNoSourceScopeRows(ctx context.Context, archive *librarysnapshot.Archive,
manifest librarysnapshot.Manifest) error {
if t.snapshotTenant == 0 {
return nil
}
if t.snapshotTenant == t.scope.TenantID && t.snapshotKB == t.scope.KnowledgeBaseID {
return nil
}
for _, part := range manifest.Parts {
if part.Kind != "database" && part.Kind != "wiki" {
continue
}
if !tableHasTenant(part.Name) {
continue
}
ids, err := archivedRowIdentities(archive, part)
if err != nil {
return err
}
if len(ids) == 0 {
continue
}
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND id = ANY($2::text[])`, part.Name)
var count int64
if err := t.pool.QueryRow(ctx, statement, t.snapshotTenant, ids).Scan(&count); err != nil {
return fmt.Errorf("检查目标 %s 的来源范围残留失败: %w", part.Name, err)
}
if count > 0 {
return fmt.Errorf("目标 %s 残留 %d 行仍属于来源范围 %d/%s:有写入逃出了目标范围,恢复不会清掉它们",
part.Name, count, t.snapshotTenant, t.snapshotKB)
}
}
return nil
}
@@ -60,8 +60,11 @@ func TestRewriteLocatorSwapsStorageBackendWithBucket(t *testing.T) {
// 行变换用解析到的目标后端与目标桶:写入与校验共用这一份,摘要才不会两边打架。
func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
storageBackend: "target-backend"}
row := map[string]json.RawMessage{"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
row := map[string]json.RawMessage{
"tenant_id": json.RawMessage(`10001`),
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
if err := target.rowTransform("resources")(row); err != nil {
t.Fatalf("row transform failed: %v", err)
}
@@ -78,8 +81,10 @@ func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
// 只改定位符不改这一列,目标仍是 storage backend not found(真跑 500)。
func TestResourceRowTransformRewritesBackendColumn(t *testing.T) {
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
storageBackend: "target-backend"}
storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
row := map[string]json.RawMessage{
"tenant_id": json.RawMessage(`10001`),
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/1/k/ab.md"`),
"storage_backend_id": json.RawMessage(`"source-backend"`),
}
+163 -10
View File
@@ -92,7 +92,7 @@ const overview: LibraryOverview = {
reasons: [
'生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false)',
'没有已核验的生产恢复演练证据(PRODUCTION_BACKUP_RESTORE_VERIFIED=false)',
'生产当前权威版本未知,不能启动写入',
'目标非空且没有生效版本记录:先建立目标权威基线(维护隔离下捕获整库并逐类校验),或确认目标确实是空库',
'本次目标快照还没有生产预备份',
],
},
@@ -149,6 +149,15 @@ const base: ViewProps = {
pendingCapture: false,
canCapture: true,
canWrite: true,
canRestore: false,
sync: { canSign: false, pendingGrant: false, onGrant: vi.fn(), canStart: false, pending: false, onStart: vi.fn() },
baseline: { canStart: false, pending: false, onStart: vi.fn() },
restore: null,
restoreNotice: null,
pendingRestore: false,
pendingGrant: false,
onGrantRestore: vi.fn(),
onRestore: vi.fn(),
maintenance: null,
now: new Date('2026-10-28T03:35:00Z').getTime(),
onRefresh: vi.fn(),
@@ -166,7 +175,7 @@ describe('ReleaseTargetsView(确认稿布局)', () => {
expect(html).toContain('对外知识库')
expect(html).toContain('role="tab"')
// 只显示一个主作业卡:选中库标题出现一次,另一个库不渲染作业卡
expect(html.match(/· 已封存,生产同步未接入/g)?.length).toBe(1)
expect(html.match(/· 已封存,同步按条件放行/g)?.length).toBe(1)
expect(html).not.toContain('对外知识库 · 尚无')
})
@@ -239,18 +248,62 @@ describe('ReleaseTargetsView(确认稿布局)', () => {
expect(html).toContain('列表只校验清单自身,字节级复核在同步/恢复执行前进行')
})
it('版本与快照抽屉按 scope 列出全部历史并区分测试封存/不可直接恢复', () => {
const html = render({ drawerOpen: true, expandedSnapshot: internalSnapshot.snapshot_id })
it('版本与快照抽屉按 scope 列出全部历史,并把恢复条件交给接口判定', () => {
const html = render({ drawerOpen: true, expandedSnapshot: internalSnapshot.snapshot_id, canRestore: true })
expect(html).toContain('版本与快照')
expect(html).toContain('封存 f992d64ea45b')
expect(html).toContain('测试封存')
expect(html).toContain('不可直接恢复')
expect(html).toContain('恢复条件见下方')
expect(html).toContain('恢复至该版本')
expect(html).toMatch(/恢复至该版本[^<]*<\/button>/)
expect(html).toContain('分项数量 / 字节 / 摘要(2 项)')
expect(html).toContain('不支持只选择 Wiki 或某张数据表恢复')
})
it('管理员在缺授权时看到签发入口,恢复按钮按接口给出的原因禁用', () => {
const html = render({
drawerOpen: true,
expandedSnapshot: internalSnapshot.snapshot_id,
canRestore: true,
restore: {
allowed: false,
reasons: ['没有针对该快照的恢复授权:先签发 restore 授权(绑定同一快照)'],
scope: 'internal',
snapshot_id: internalSnapshot.snapshot_id,
authorization: { state: 'none' },
},
})
expect(html).toContain('签发恢复授权(30 分钟)')
expect(html).toContain('没有针对该快照的恢复授权')
expect(html).toMatch(/rel-danger[^>]*disabled/)
// 不能再说"未接线":条件不满足要给出具体原因
expect(html).not.toContain('生产恢复未接线')
})
it('条件全满足时恢复按钮可用,并写明授权有效期', () => {
const html = render({
drawerOpen: true,
expandedSnapshot: internalSnapshot.snapshot_id,
canRestore: true,
restore: {
allowed: true,
reasons: [],
scope: 'internal',
snapshot_id: internalSnapshot.snapshot_id,
authorization: { state: 'active', authorized_by: 'yaozhanghao', expires_at: '2026-10-28T04:10:00Z' },
},
})
expect(html).toContain('条件已满足')
expect(html).toContain('有效至')
expect(html).not.toMatch(/rel-danger[^>]*disabled/)
})
it('工程师看不到恢复入口,只看到权限说明', () => {
const html = render({ drawerOpen: true, expandedSnapshot: internalSnapshot.snapshot_id, canRestore: false })
expect(html).toContain('按版本恢复仅管理员可发起')
expect(html).not.toContain('签发恢复授权')
})
it('抽屉只展示当前选中库的历史', () => {
const html = render({ drawerOpen: true, scope: 'external' })
expect(html).toContain('封存 01343b0b310b')
@@ -277,19 +330,119 @@ describe('ReleaseTargetsView(确认稿布局)', () => {
expect(html).toContain('主机侧正在暂停写入者、导出并恢复')
})
it('无权限时不给封存入口,未接线时说明通道状态', () => {
it('无权限时不给封存入口;不可发起时给出接口返回的真实原因与执行器身份', () => {
expect(render({ canCapture: false, canWrite: false })).toContain('封存需要知识工程师及以上权限')
const unwired = render({
const unavailable = render({
canCapture: false,
canWrite: true,
page: { ...page, capture_available: false, capture_reason: '主机侧封存 agent 未接线' },
page: { ...page, capture_available: false },
overview: {
...overview,
capture_available: false,
capture_reason: '执行器指向的实例不匹配:agent="weknora071full",期望="kgmaint-weknora"',
executor: {
available: false,
busy: false,
project: 'weknora071full',
source_app: 'WKNFull-app',
observed_at: '2026-10-28T03:34:00Z',
},
},
})
expect(unwired).toContain('封存通道未接线')
expect(unavailable).toContain('执行器指向的实例不匹配')
expect(unavailable).toContain('执行器 weknora071full / WKNFull-app')
expect(unavailable).toContain('观测于')
// 通道可用/不可用都不许用写死的"未接线"糊过去
expect(unavailable).not.toContain('封存通道未接线')
})
it('无封存时给出“尚无封存”而不是空表', () => {
const html = render({ page: { ...page, snapshots: [] } })
// 归档没有快照时接口也不会给 test_sealed:两处必须一致,否则页面会把
// 一个不存在的封存显示成"已封存"。
const html = render({
page: { ...page, snapshots: [] },
overview: { ...overview, test_sealed: undefined, test_sealed_reason: '还没有该目标的测试封存' },
})
expect(html).toContain('尚无测试封存')
expect(html).toContain('还没有测试库封存')
})
it('管理员在缺同步授权时看到签发入口,工程师只看到“需管理员签发”', () => {
const admin = render({
sync: { canSign: true, pendingGrant: false, onGrant: vi.fn(), canStart: false, pending: false, onStart: vi.fn() },
})
expect(admin).toContain('签发同步授权(30 分钟)')
// 授权入口不能顺手把执行也放行:闸口条件仍然逐条可见
expect(admin).toContain('生产写入闸口关闭')
const engineer = render()
expect(engineer).not.toContain('签发同步授权')
expect(engineer).toContain('同步授权只能由管理员按目标+快照签发')
})
it('目标非空又没有权威版本时,管理员看到建立基线入口;有生效版本时不显示', () => {
const admin = render({
baseline: { canStart: true, pending: false, onStart: vi.fn() },
})
expect(admin).toContain('建立目标权威基线(维护隔离)')
expect(admin).toContain('目标非空且没有生效版本记录')
const known = render({
baseline: { canStart: true, pending: false, onStart: vi.fn() },
overview: {
...overview,
production: {
state: 'known',
version: {
id: 'v1',
kind: 'production_effective',
snapshot_id: 'a'.repeat(64),
version_label: '基线 abc',
source_job_id: 'libbase-1',
evidence: {},
created_by: 'admin',
created_at: '2026-10-29T09:00:00Z',
},
},
},
})
// 有生效版本时不再提供基线入口(缺项说明文字里仍会提到它存在,所以只断言按钮)
expect(known).not.toMatch(/建立目标权威基线(维护隔离)<\/button>/)
})
it('闸口全满足时“启动同步”真的可点,并给出执行说明', () => {
const html = render({
sync: { canSign: false, pendingGrant: false, onGrant: vi.fn(), canStart: true, pending: false, onStart: vi.fn() },
overview: { ...overview, gate: { sync_allowed: true, reasons: [] } },
})
expect(html).toMatch(/<button[^>]*class="rel-primary"[^>]*>▶ 启动同步/)
expect(html).not.toMatch(/class="rel-primary"[^>]*disabled/)
expect(html).toContain('先固定测试版本,再备份目标并写入')
})
it('维护卡片区分“未运行维护”与“没有维护能力”', () => {
const wired = render({
maintenance: {
held: false,
capability: {
configured: true,
missing: [],
instance: 'kgseal-app',
project: 'kgseal-weknora',
writers: ['app'],
ingress_service: 'ingress',
},
},
})
expect(wired).toContain('未运行维护')
expect(wired).toContain('维护能力已接线:实例 kgseal-app · 写入者 app')
expect(wired).not.toContain('也没有维护能力配置')
const broken = render({
maintenance: { held: false, capability: { configured: false, missing: ['LIBRARY_TARGET_WRITERS'] } },
})
expect(broken).toContain('能力未接线')
expect(broken).toContain('维护能力未接线:缺少 LIBRARY_TARGET_WRITERS')
// 接口没给能力信息时不能猜成"没有能力"
const unknown = render({ maintenance: { held: false } })
expect(unknown).toContain('维护能力状态未知')
expect(unknown).not.toContain('也没有维护能力配置')
})
})
+330 -18
View File
@@ -1,10 +1,16 @@
import { useCallback, useEffect, useState } from 'react'
import {
createLibraryRestoreAuthorization,
createLibrarySyncAuthorization,
enqueueLibraryBaseline,
enqueueLibraryCapture,
enqueueLibraryRecovery,
enqueueLibraryRestore,
enqueueLibrarySync,
fetchLibraryCaptures,
fetchLibraryMaintenance,
fetchLibraryOverview,
fetchLibraryRestoreAvailability,
fetchLibrarySnapshots,
formatBytes,
scopeLabel,
@@ -12,6 +18,7 @@ import {
type LibraryJobRecord,
type LibraryMaintenanceView,
type LibraryOverview,
type LibraryRestoreAvailability,
type LibrarySnapshot,
type LibrarySnapshotList,
type LibraryStageRow,
@@ -58,6 +65,24 @@ const jobStateLabel: Record<string, string> = {
type Scope = 'internal' | 'external'
// 签发授权与执行是两步:页面只提供入口,不会替人签发,也不会自动重试执行。
export interface ReleaseSyncActions {
// 管理员才能签发;工程师看得到条件但没有按钮。
canSign: boolean
pendingGrant: boolean
onGrant: () => void
// 闸口全部满足才能启动(接口用同一份判定,按钮只是提前说明)。
canStart: boolean
pending: boolean
onStart: () => void
}
export interface ReleaseBaselineActions {
canStart: boolean
pending: boolean
onStart: () => void
}
interface ScopeView {
scope: Scope
snapshot?: LibrarySnapshot
@@ -85,6 +110,18 @@ interface ViewProps {
pendingCapture: boolean
canCapture: boolean
canWrite: boolean
// 发布动作:签发授权(管理员)与启动同步是两个独立步骤,页面绝不自动签发。
sync: ReleaseSyncActions
// 目标非空又没有生效版本时,唯一入口是"建立目标权威基线"(管理员)。
baseline: ReleaseBaselineActions
// 按版本恢复是管理员动作:工程师看得到条件,但拿不到恢复入口。
canRestore: boolean
restore: LibraryRestoreAvailability | null
restoreNotice: string | null
pendingRestore: boolean
pendingGrant: boolean
onGrantRestore: () => void
onRestore: () => void
overview: LibraryOverview | null
maintenance: LibraryMaintenanceView | null
now: number
@@ -106,7 +143,9 @@ function scopeView(
overview?.job ?? persisted.find((item) => item.state === 'pending' || item.state === 'running') ?? persisted[0]
return {
scope,
snapshot: history[0],
// 候选只认接口给的 test_sealed(来自版本登记):归档里还有生产预备份与
// 目标基线快照,按"最新一份"挑会把目标自己的内容显示成测试封存。
snapshot: overview?.test_sealed,
history,
activeJob: scoped.find((job) => ACTIVE_STATES.includes(job.state)),
lastJob: scoped[0],
@@ -118,6 +157,20 @@ function scopeView(
const scopeName = (scope: Scope) => (scope === 'internal' ? '对内知识库' : '对外知识库')
// 不可发起时必须说清是哪一条不满足:读的是接口给出的真实原因(执行器不可达、
// 指向别的实例、降级、观测过期),不是一句写死的"通道未接线"。
function captureHint(view: ScopeView, canWrite: boolean): string {
if (!canWrite) return '封存需要知识工程师及以上权限'
const executor = view.overview?.executor
const reason = view.overview?.capture_reason || executor?.reason
const observed = executor?.observed_at ? `(观测于 ${stamp(executor.observed_at)})` : ''
const identity = executor?.project
? `执行器 ${executor.project}${executor.source_app ? ` / ${executor.source_app}` : ''} · `
: ''
if (reason) return `${identity}封存通道不可用:${reason}${observed}`
return '封存通道状态未知:还没有读到执行器观测'
}
function statusOf(view: ScopeView) {
const job = view.job
if (job && (job.state === 'running' || job.state === 'pending')) {
@@ -245,6 +298,14 @@ export function MaintenanceCard({
}) {
const summary = maintenanceSummary(maintenance, job)
const record = maintenance?.instance_maintenance ?? maintenance?.maintenance
// "没有窗口记录"不等于"没有维护能力":能力是配置事实,接口单独返回。
// 接口没给能力信息时也不能猜,明确写"未知"。
const capability = maintenance?.capability
const capabilityText = !capability
? '维护能力状态未知:接口没有返回能力信息,无法区分"未运行维护"与"能力未接线"'
: capability.configured
? `维护能力已接线:实例 ${capability.instance || '—'} · 写入者 ${(capability.writers ?? []).join('、') || '—'}`
: `维护能力未接线:缺少 ${(capability.missing ?? []).join('、') || '必要配置'}`
return (
<div className="rel-panel rel-maintenance">
<header className="rel-panel-head">
@@ -252,13 +313,26 @@ export function MaintenanceCard({
<h2>目标维护</h2>
<p>同步/恢复期间目标写入者被停住;级别与入口状态来自维护记录本身</p>
</div>
<span className={`rel-pill ${summary.held ? 'blue' : summary.level === '未接入' ? 'gray' : 'green'}`}>
{summary.held ? `${summary.level} · 占用中` : summary.level === '未接入' ? '未接入' : '未占用'}
<span
className={`rel-pill ${summary.held ? 'blue' : record ? 'green' : capability?.configured ? 'gray' : 'red'}`}
>
{summary.held
? `${summary.level} · 占用中`
: record
? '未占用'
: capability?.configured
? '未运行维护'
: capability
? '能力未接线'
: '能力未知'}
</span>
</header>
{summary.level === '未接入' ? (
{!record ? (
<p className="rel-scope-note">
暂无维护窗口记录,也没有维护能力配置;维护类操作保持 fail closed,不会在此状态下写目标。
暂无维护窗口记录。{capabilityText}
{capability?.configured
? ';窗口只在同步或建立基线时开启,页面不会单独开窗口。'
: ';维护类操作保持 fail closed,不会在此状态下写目标。'}
</p>
) : (
<>
@@ -508,15 +582,23 @@ function ProtectionCard({
prebackup,
gate,
authorization,
targetSealed,
sync,
baseline,
onOpen,
}: {
production: LibraryOverview['production']
prebackup: LibraryOverview['prebackup']
gate: LibraryOverview['gate']
authorization: LibraryOverview['authorization']
targetSealed: boolean
sync: ReleaseSyncActions
baseline: ReleaseBaselineActions
onOpen: () => void
}) {
const backupKnown = prebackup.state === 'known' && prebackup.version
// 基线入口只在"目标非空、却没有生效版本"时才是答案;空库走首次同步(预备份+登记)。
const baselineNeeded = production.state !== 'known' && targetSealed
return (
<aside className="rel-panel rel-protection">
<header className="rel-panel-head">
@@ -554,6 +636,21 @@ function ProtectionCard({
))}
</ul>
) : null}
<div className="rel-drawer-actions">
{sync.canSign && authorization.state !== 'active' && targetSealed ? (
<button type="button" className="rel-secondary" onClick={sync.onGrant} disabled={sync.pendingGrant}>
{sync.pendingGrant ? '签发中…' : '签发同步授权(30 分钟)'}
</button>
) : null}
{baseline.canStart && baselineNeeded ? (
<button type="button" className="rel-secondary" onClick={baseline.onStart} disabled={baseline.pending}>
{baseline.pending ? '投递中…' : '建立目标权威基线(维护隔离)'}
</button>
) : null}
</div>
{authorization.state !== 'active' && !sync.canSign ? (
<p className="rel-hint">同步授权只能由管理员按目标+快照签发;工程师看不到签发入口。</p>
) : null}
<p className="rel-hint">同步启动后,这里显示本次生产恢复点及备份进度,可直接查看对应快照。</p>
<button type="button" className="rel-secondary" onClick={onOpen}>
查看已有版本与快照 ›
@@ -575,6 +672,8 @@ interface ReadyProps {
pendingCapture: boolean
onCapture: () => void
onOpen: () => void
sync: ReleaseSyncActions
baseline: ReleaseBaselineActions
}
function ReadyBody({
@@ -584,6 +683,8 @@ function ReadyBody({
prebackup,
gate,
authorization,
sync,
baseline,
canCapture,
canWrite,
pendingCapture,
@@ -600,6 +701,9 @@ function ReadyBody({
prebackup={prebackup}
gate={gate}
authorization={authorization}
targetSealed={Boolean(snapshot)}
sync={sync}
baseline={baseline}
onOpen={onOpen}
/>
</div>
@@ -629,7 +733,7 @@ function ReadyBody({
{pendingCapture ? '投递中…' : '封存测试版本'}
</button>
) : (
<span className="rel-hint">{canWrite ? '封存通道未接线' : '封存需要知识工程师及以上权限'}</span>
<span className="rel-hint">{captureHint(view, canWrite)}</span>
)}
</div>
</>
@@ -822,8 +926,16 @@ interface DrawerProps {
onExpand: (id: string | null) => void
onClose: () => void
canCapture: boolean
canRestore: boolean
pendingCapture: boolean
onCapture: () => void
// 恢复条件来自容器(真实接口判定):双闸 / 该快照的授权 / 目标是否在维护。
restore: LibraryRestoreAvailability | null
restoreNotice: string | null
pendingRestore: boolean
pendingGrant: boolean
onGrantRestore: () => void
onRestore: () => void
}
function Drawer({
@@ -835,8 +947,15 @@ function Drawer({
onExpand,
onClose,
canCapture,
canRestore,
pendingCapture,
onCapture,
restore,
restoreNotice,
pendingRestore,
pendingGrant,
onGrantRestore,
onRestore,
}: DrawerProps) {
if (!open) return null
const listing = view.history
@@ -919,8 +1038,12 @@ function Drawer({
>
{isSelected ? '收起快照' : '查看快照'}
</button>
<button type="button" className="rel-link" disabled title="生产恢复未接线,也没有恢复演练记录">
不可直接恢复
<button
type="button"
className="rel-link"
onClick={() => onExpand(isSelected ? null : snapshot.snapshot_id)}
>
{isSelected ? '恢复条件见下方' : '查看恢复条件'}
</button>
</td>
</tr>
@@ -991,13 +1114,48 @@ function Drawer({
以上是同一恢复点的配套内容,不支持只选择 Wiki 或某张数据表恢复。恢复速度需实测;“有备份文件”不等于可恢复。
</p>
<div className="rel-drawer-actions">
<button type="button" disabled title="生产恢复验证记录未接入">
<button type="button" disabled title="逐项校验证据在同步/恢复执行时产生,作业详情里可见">
查看校验记录
</button>
<button type="button" className="rel-danger" disabled title="生产恢复未接线,也没有经过恢复演练">
恢复至该版本
</button>
{canRestore && restore && restore.authorization.state !== 'active' ? (
<button type="button" className="rel-secondary" onClick={onGrantRestore} disabled={pendingGrant}>
{pendingGrant ? '签发中…' : '签发恢复授权(30 分钟)'}
</button>
) : null}
{canRestore ? (
<button
type="button"
className="rel-danger"
onClick={onRestore}
disabled={pendingRestore || restore?.allowed !== true}
title={
restore?.allowed
? '把整套生产知识状态恢复到这个快照'
: (restore?.reasons ?? []).join(';') || '恢复条件尚未读取到'
}
>
{pendingRestore ? '投递中…' : '恢复至该版本'}
</button>
) : (
<span className="rel-hint">按版本恢复仅管理员可发起</span>
)}
</div>
{restore && !restore.allowed ? (
<ul className="rel-gate" aria-label="恢复条件">
{restore.reasons.map((reason) => (
<li key={reason}>{reason}</li>
))}
</ul>
) : null}
{restore?.allowed ? (
<p className="rel-hint">
条件已满足:生产写入闸口打开、这份快照有恢复授权
{restore.authorization.expires_at ? `(有效至 ${stamp(restore.authorization.expires_at)})` : ''}
、目标实例未在维护。
</p>
) : null}
{restore === null ? <p className="rel-hint">正在读取恢复条件…</p> : null}
{restoreNotice ? <p className="rel-notice">{restoreNotice}</p> : null}
</section>
) : null}
@@ -1023,7 +1181,7 @@ function Drawer({
<p className="rel-hint">
归档占用(只读):被版本引用的去重内容 {formatBytes(page?.content_bytes ?? 0)} · 磁盘分配{' '}
{formatBytes(page?.blob_allocated_bytes ?? 0)} · 未被引用 {page?.unreferenced_blobs ?? 0}{' '}
个。保留策略与清理在系统设置 → 存储管理,均未接线。
个。清理与保留策略在系统设置 → 存储管理:当前可查看去重与保护范围,删除与保留策略尚未开放。
</p>
</aside>
</div>
@@ -1047,6 +1205,15 @@ export function ReleaseTargetsView(props: ViewProps) {
pendingCapture,
canCapture,
canWrite,
sync,
baseline,
canRestore,
restore,
restoreNotice,
pendingRestore,
pendingGrant,
onGrantRestore,
onRestore,
now,
maintenance,
onRefresh,
@@ -1125,7 +1292,7 @@ export function ReleaseTargetsView(props: ViewProps) {
<div className="rel-panel rel-jobhead">
<div className="rel-between">
<h2>
{scopeName(scope)} · {active ? '封存进行中' : view.snapshot ? '已封存,生产同步未接入' : '尚无测试封存'}
{scopeName(scope)} · {active ? '封存进行中' : view.snapshot ? '已封存,同步按条件放行' : '尚无测试封存'}
</h2>
<div className="rel-head-actions">
<button type="button" className="rel-secondary" onClick={() => onDrawer(true)}>
@@ -1134,14 +1301,15 @@ export function ReleaseTargetsView(props: ViewProps) {
<button
type="button"
className="rel-primary"
disabled={!gate.sync_allowed}
onClick={sync.onStart}
disabled={!sync.canStart || sync.pending}
title={
gate.sync_allowed
? '生产同步执行链尚未接线,按钮不会启用'
sync.canStart
? '按已签发的同步授权启动:先固定测试版本,再备份目标并写入'
: `暂不能启动同步:${gate.reasons.join(';')}`
}
>
▶ 启动同步
{sync.pending ? '投递中…' : '▶ 启动同步'}
</button>
</div>
</div>
@@ -1224,6 +1392,8 @@ export function ReleaseTargetsView(props: ViewProps) {
prebackup={prebackup}
gate={gate}
authorization={authorization}
sync={sync}
baseline={baseline}
canCapture={canCapture}
canWrite={canWrite}
pendingCapture={pendingCapture}
@@ -1241,8 +1411,15 @@ export function ReleaseTargetsView(props: ViewProps) {
onExpand={onExpandSnapshot}
onClose={() => onDrawer(false)}
canCapture={canCapture}
canRestore={canRestore}
pendingCapture={pendingCapture}
onCapture={onCapture}
restore={restore}
restoreNotice={restoreNotice}
pendingRestore={pendingRestore}
pendingGrant={pendingGrant}
onGrantRestore={onGrantRestore}
onRestore={onRestore}
/>
</section>
)
@@ -1313,7 +1490,122 @@ export function ReleaseTargets() {
const role = auth?.identity?.role
const canWrite = role === 'engineer' || role === 'admin'
// 恢复写的是整套生产知识状态,接口只认管理员;页面必须与它一致。
const canRestore = role === 'admin'
// 签发同步授权同样是管理员动作:授权=钉死目标+快照+参数,不等于执行。
const canSignSync = role === 'admin'
const canCapture = canWrite && page?.capture_available === true
const [pendingSyncGrant, setPendingSyncGrant] = useState(false)
const [pendingSync, setPendingSync] = useState(false)
const [pendingBaseline, setPendingBaseline] = useState(false)
const current = overviews[scope]
const targetSnapshot = current?.test_sealed?.snapshot_id ?? ''
const grantSync = async () => {
if (!targetSnapshot) {
setNotice('还没有可发布的测试封存:先封存测试版本,再签发同步授权')
return
}
setPendingSyncGrant(true)
setNotice(null)
try {
const grant = await createLibrarySyncAuthorization(scope, targetSnapshot)
setNotice(
`已签发同步授权${grant.authorized_by ? `(${grant.authorized_by})` : ''},有效至 ${stamp(grant.expires_at)};` +
'执行仍需在下一次操作里点"启动同步"'
)
await load()
} catch (grantError) {
setNotice(grantError instanceof Error ? grantError.message : '签发同步授权失败')
} finally {
setPendingSyncGrant(false)
}
}
const startSync = async () => {
if (!targetSnapshot) {
setNotice('还没有可发布的测试封存')
return
}
setPendingSync(true)
setNotice(null)
try {
const accepted = await enqueueLibrarySync(scope, targetSnapshot)
setNotice(`已投递同步:${accepted.job_id}(先固定测试版本并生成预备份,再写目标)`)
await load()
} catch (syncError) {
setNotice(syncError instanceof Error ? syncError.message : '启动同步失败')
} finally {
setPendingSync(false)
}
}
const startBaseline = async () => {
setPendingBaseline(true)
setNotice(null)
try {
const accepted = await enqueueLibraryBaseline(scope)
setNotice(`已投递建立基线:${accepted.job_id}(维护隔离下捕获目标整库,逐类校验后登记为权威基线)`)
await load()
} catch (baselineError) {
setNotice(baselineError instanceof Error ? baselineError.message : '建立目标权威基线失败')
} finally {
setPendingBaseline(false)
}
}
// 恢复条件随"展开的那份快照"变化:闸口、该快照的授权、实例是否在维护都由接口判定。
const [restore, setRestore] = useState<LibraryRestoreAvailability | null>(null)
const [restoreNotice, setRestoreNotice] = useState<string | null>(null)
const [pendingRestore, setPendingRestore] = useState(false)
const [pendingGrant, setPendingGrant] = useState(false)
const [restoreRound, setRestoreRound] = useState(0)
const restoreSnapshot = expandedSnapshot ?? ''
useEffect(() => {
if (!drawerOpen || !restoreSnapshot) {
setRestore(null)
return
}
const controller = new AbortController()
void fetchLibraryRestoreAvailability(scope, restoreSnapshot, controller.signal)
.then((value) => setRestore(value))
.catch(() => setRestore(null))
return () => controller.abort()
}, [drawerOpen, restoreSnapshot, scope, restoreRound])
const grantRestore = async () => {
if (!restoreSnapshot) return
setPendingGrant(true)
setRestoreNotice(null)
try {
const grant = await createLibraryRestoreAuthorization(scope, restoreSnapshot)
setRestoreNotice(
`已签发恢复授权${grant.authorized_by ? `(${grant.authorized_by})` : ''},有效至 ${stamp(grant.expires_at)}`
)
setRestoreRound((round) => round + 1)
await load()
} catch (grantError) {
setRestoreNotice(grantError instanceof Error ? grantError.message : '签发恢复授权失败')
} finally {
setPendingGrant(false)
}
}
const startRestore = async () => {
if (!restoreSnapshot) return
setPendingRestore(true)
setRestoreNotice(null)
try {
const accepted = await enqueueLibraryRestore(scope, restoreSnapshot)
setRestoreNotice(`已投递恢复:${accepted.job_id}(整套生产知识状态回到该快照)`)
setRestoreRound((round) => round + 1)
await load()
} catch (restoreError) {
setRestoreNotice(restoreError instanceof Error ? restoreError.message : '发起恢复失败')
} finally {
setPendingRestore(false)
}
}
const startCapture = async () => {
setPendingCapture(true)
@@ -1364,6 +1656,26 @@ export function ReleaseTargets() {
pendingCapture={pendingCapture}
canCapture={canCapture}
canWrite={canWrite}
sync={{
canSign: canSignSync,
pendingGrant: pendingSyncGrant,
onGrant: () => void grantSync(),
canStart: current?.gate.sync_allowed === true,
pending: pendingSync,
onStart: () => void startSync(),
}}
baseline={{
canStart: canSignSync,
pending: pendingBaseline,
onStart: () => void startBaseline(),
}}
canRestore={canRestore}
restore={restore}
restoreNotice={restoreNotice}
pendingRestore={pendingRestore}
pendingGrant={pendingGrant}
onGrantRestore={() => void grantRestore()}
onRestore={() => void startRestore()}
maintenance={maintenances[scope]}
now={now}
onRefresh={() => void load()}
+87
View File
@@ -0,0 +1,87 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import {
createLibraryRestoreAuthorization,
enqueueLibraryRestore,
fetchLibraryRestoreAvailability,
} from './librarySnapshots'
const jsonResponse = (body: unknown, status = 200) =>
new Response(JSON.stringify(body), { status, headers: { 'Content-Type': 'application/json' } })
afterEach(() => {
vi.unstubAllGlobals()
})
describe('整库恢复的前端契约', () => {
it('读取恢复条件时把原因与授权状态原样带回来', async () => {
vi.stubGlobal(
'fetch',
vi.fn().mockResolvedValue(
jsonResponse({
allowed: false,
reasons: ['没有针对该快照的恢复授权:先签发 restore 授权(绑定同一快照)'],
scope: 'internal',
snapshot_id: 'snap-1',
authorization: { state: 'none', reason: '没有与目标快照匹配的有效发布授权' },
})
)
)
const availability = await fetchLibraryRestoreAvailability('internal', 'snap-1')
expect(availability?.allowed).toBe(false)
expect(availability?.reasons).toHaveLength(1)
expect(availability?.authorization.state).toBe('none')
const requested = (vi.mocked(fetch).mock.calls[0]?.[0] as string) ?? ''
expect(requested).toContain('scope=internal')
expect(requested).toContain('snapshot_id=snap-1')
})
it('签发恢复授权必须声明 action=restore 与有限有效期', async () => {
vi.stubGlobal(
'fetch',
vi.fn().mockResolvedValue(
jsonResponse(
{
authorization: {
id: 'libauth-restore-1',
authorized_by: 'yaozhanghao',
expires_at: '2026-10-29T09:00:00Z',
},
},
201
)
)
)
const grant = await createLibraryRestoreAuthorization('internal', 'snap-1')
expect(grant.id).toBe('libauth-restore-1')
const init = vi.mocked(fetch).mock.calls[0]?.[1] as RequestInit
const body = JSON.parse(String(init.body)) as Record<string, unknown>
expect(body.action).toBe('restore')
expect(body.expires_in_minutes).toBe(30)
})
it('恢复被拒时把逐条原因说给用户,而不是只说 HTTP 状态', async () => {
vi.stubGlobal(
'fetch',
vi
.fn()
.mockResolvedValue(
jsonResponse({ code: 'library_restore_blocked', reasons: ['生产写入闸口关闭', '没有该快照的恢复授权'] }, 503)
)
)
await expect(enqueueLibraryRestore('internal', 'snap-1')).rejects.toThrow('生产写入闸口关闭;没有该快照的恢复授权')
})
it('投递成功时返回作业号,供页面显示真实进度', async () => {
vi.stubGlobal(
'fetch',
vi
.fn()
.mockResolvedValue(
jsonResponse({ job: { id: 'librestore-internal-1', scope: 'internal' }, reused: false }, 202)
)
)
const accepted = await enqueueLibraryRestore('internal', 'snap-1')
expect(accepted.job_id).toBe('librestore-internal-1')
expect(accepted.reused).toBe(false)
})
})
+190
View File
@@ -185,6 +185,170 @@ export async function enqueueLibraryCapture(scope: string): Promise<LibraryCaptu
}
}
// 恢复能不能点,由真实条件决定:双闸、这份快照的恢复授权、目标实例是否在维护。
export interface LibraryRestoreAvailability {
allowed: boolean
reasons: string[]
scope: string
snapshot_id: string
authorization: LibraryAuthorizationView
}
export async function fetchLibraryRestoreAvailability(
scope: string,
snapshotID: string,
signal?: AbortSignal
): Promise<LibraryRestoreAvailability | null> {
const query = `scope=${encodeURIComponent(scope)}&snapshot_id=${encodeURIComponent(snapshotID)}`
const response = await fetch(`/api/v1/library/restore-availability?${query}`, {
headers: { Accept: 'application/json' },
signal,
})
const value = (await response.json().catch(() => null)) as unknown
if (!record(value)) return null
return {
allowed: bool(value.allowed),
reasons: Array.isArray(value.reasons) ? value.reasons.map(text).filter(Boolean) : [],
scope: text(value.scope),
snapshot_id: text(value.snapshot_id),
authorization: record(value.authorization)
? {
state: text(value.authorization.state) === 'active' ? 'active' : 'none',
authorized_by: text(value.authorization.authorized_by) || undefined,
expires_at: text(value.authorization.expires_at) || undefined,
reason: text(value.authorization.reason) || undefined,
}
: { state: 'none' },
}
}
// 恢复整套生产知识状态:同一份快照的文档、分块、索引、资源与 Wiki 一起回放。
export interface LibraryRestoreAccepted {
job_id: string
scope: string
reused: boolean
}
export async function enqueueLibraryRestore(scope: string, snapshotID: string): Promise<LibraryRestoreAccepted> {
const response = await fetch('/api/v1/library/restores', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ scope, snapshot_id: snapshotID }),
})
const value = (await response.json().catch(() => null)) as unknown
if (!response.ok) {
if (response.status === 403) throw new Error('按版本恢复仅管理员可发起。')
if (response.status === 503 && record(value)) {
const reasons = Array.isArray(value.reasons) ? value.reasons.map(text).filter(Boolean) : []
throw new Error(reasons.length > 0 ? reasons.join(';') : text(value.error) || '恢复条件不满足')
}
if (response.status === 409 && record(value)) throw new Error(text(value.error) || '该目标已有在飞作业')
throw new Error(`发起恢复失败(HTTP ${response.status})`)
}
if (!record(value) || !record(value.job)) throw new Error('恢复作业响应无效')
return { job_id: text(value.job.id), scope: text(value.job.scope), reused: bool(value.reused) }
}
export interface LibraryRestoreGrant {
id: string
authorized_by: string
expires_at: string
}
// 同步授权与执行是两步:授权把"范围 + 目标快照 + 参数摘要"钉死并给很短的有效期。
// 它不能顶替恢复授权(摘要不同),也不自动签发 —— 必须由管理员显式点击。
export async function createLibrarySyncAuthorization(
scope: string,
snapshotID: string,
minutes = 30
): Promise<LibraryRestoreGrant> {
const response = await fetch('/api/v1/library/authorizations', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ scope, snapshot_id: snapshotID, action: 'sync', expires_in_minutes: minutes }),
})
const value = (await response.json().catch(() => null)) as unknown
if (!response.ok) {
if (response.status === 403) throw new Error('生产发布授权仅管理员可签发。')
if (response.status === 400 && record(value)) throw new Error(text(value.error) || '授权参数不合法')
throw new Error(`签发同步授权失败(HTTP ${response.status})`)
}
if (!record(value) || !record(value.authorization)) throw new Error('授权响应无效')
return {
id: text(value.authorization.id),
authorized_by: text(value.authorization.authorized_by),
expires_at: text(value.authorization.expires_at),
}
}
export interface LibrarySyncAccepted {
job_id: string
scope: string
reused: boolean
}
// 启动同步:闸口未齐时接口返回 503 与逐条缺项,页面照实显示,不假装已启动。
export async function enqueueLibrarySync(scope: string, targetSnapshotID: string): Promise<LibrarySyncAccepted> {
const response = await fetch('/api/v1/library/syncs', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ scope, target_snapshot_id: targetSnapshotID }),
})
const value = (await response.json().catch(() => null)) as unknown
if (!response.ok) {
if (record(value) && Array.isArray(value.reasons)) {
const reasons = value.reasons.map(text).filter(Boolean)
if (reasons.length > 0) throw new Error(`启动同步被拒绝:${reasons.join(';')}`)
}
if (record(value) && text(value.error)) throw new Error(text(value.error))
throw new Error(`启动同步失败(HTTP ${response.status})`)
}
if (!record(value) || !record(value.job)) throw new Error('同步作业响应无效')
return { job_id: text(value.job.id), scope: text(value.job.scope), reused: bool(value.reused) }
}
// 建立目标权威基线:目标非空又没有生效版本时的唯一入口(维护隔离 + 真实捕获 + 校验)。
export async function enqueueLibraryBaseline(scope: string): Promise<LibrarySyncAccepted> {
const response = await fetch('/api/v1/library/baselines', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ scope }),
})
const value = (await response.json().catch(() => null)) as unknown
if (!response.ok) {
if (response.status === 403) throw new Error('建立目标权威基线仅管理员可发起。')
if (record(value) && text(value.error)) throw new Error(text(value.error))
throw new Error(`建立基线失败(HTTP ${response.status})`)
}
if (!record(value) || !record(value.job)) throw new Error('基线作业响应无效')
return { job_id: text(value.job.id), scope: text(value.job.scope), reused: bool(value.reused) }
}
// 恢复授权与执行是两件事:授权把"目标 + 快照 + 参数"钉死并给很短的有效期,
// 同步授权不能顶替恢复授权(摘要不同)。
export async function createLibraryRestoreAuthorization(
scope: string,
snapshotID: string,
minutes = 30
): Promise<LibraryRestoreGrant> {
const response = await fetch('/api/v1/library/authorizations', {
method: 'POST',
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
body: JSON.stringify({ scope, snapshot_id: snapshotID, action: 'restore', expires_in_minutes: minutes }),
})
const value = (await response.json().catch(() => null)) as unknown
if (!response.ok) {
if (response.status === 403) throw new Error('生产发布授权仅管理员可签发。')
throw new Error(`签发恢复授权失败(HTTP ${response.status})`)
}
if (!record(value) || !record(value.authorization)) throw new Error('授权响应无效')
return {
id: text(value.authorization.id),
authorized_by: text(value.authorization.authorized_by),
expires_at: text(value.authorization.expires_at),
}
}
export async function fetchLibraryCaptures(signal?: AbortSignal): Promise<LibraryCaptureJob[]> {
const response = await fetch('/api/v1/library/captures', {
headers: { Accept: 'application/json' },
@@ -311,6 +475,18 @@ export interface LibraryMaintenanceView {
maintenance?: LibraryMaintenanceRecord | null
instance?: string
instance_maintenance?: LibraryInstanceMaintenanceRecord | null
// 能力是配置事实,和"此刻有没有窗口"分开:未运行维护 ≠ 没有维护能力。
capability?: LibraryMaintenanceCapability | null
}
export interface LibraryMaintenanceCapability {
configured: boolean
missing: string[]
instance?: string
project?: string
writers?: string[]
ingress_service?: string
note?: string
}
export async function fetchLibraryMaintenance(
@@ -391,6 +567,18 @@ export interface LibraryGate {
reasons: string[]
}
export interface LibraryExecutorView {
available: boolean
reason?: string
instance?: string
project?: string
source_app?: string
archive_root?: string
busy: boolean
degraded?: string
observed_at?: string
}
export interface LibraryOverview {
scope: string
test_sealed?: LibrarySnapshot
@@ -404,6 +592,8 @@ export interface LibraryOverview {
authorization: LibraryAuthorizationView
capture_available: boolean
capture_reason?: string
// 执行器是 worker 的真实观测:指向哪个实例、何时观测、是否降级/在跑。
executor?: LibraryExecutorView
now: string
}
+82
View File
@@ -0,0 +1,82 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { createLibrarySyncAuthorization, enqueueLibraryBaseline, enqueueLibrarySync } from './librarySnapshots'
// 这些用例钉住"两步走"的接线:授权与执行是两个请求,参数必须与接口契约一致。
// 任何一步把 action 写错、或把 snapshot 换掉,都会在真实调用里悄悄失效(授权永远匹配不上)。
const jsonResponse = (status: number, body: unknown) =>
new Response(JSON.stringify(body), { status, headers: { 'Content-Type': 'application/json' } })
afterEach(() => vi.restoreAllMocks())
describe('发布页的同步授权与执行接线', () => {
it('签同步授权必须声明 action=sync 与有限有效期', async () => {
const fetchMock = vi.fn().mockResolvedValue(
jsonResponse(201, {
authorization: { id: 'libauth-sync-1', authorized_by: 'admin', expires_at: '2026-10-29T10:00:00Z' },
})
)
vi.stubGlobal('fetch', fetchMock)
const grant = await createLibrarySyncAuthorization('internal', 'snap-1', 30)
expect(grant.id).toBe('libauth-sync-1')
const [url, init] = fetchMock.mock.calls[0]
expect(url).toBe('/api/v1/library/authorizations')
expect(JSON.parse(init.body)).toEqual({
scope: 'internal',
snapshot_id: 'snap-1',
action: 'sync',
expires_in_minutes: 30,
})
})
it('启动同步把准确的 range 与目标快照传给 /library/syncs', async () => {
const fetchMock = vi
.fn()
.mockResolvedValue(jsonResponse(202, { job: { id: 'libsync-internal-1', scope: 'internal' }, reused: false }))
vi.stubGlobal('fetch', fetchMock)
const accepted = await enqueueLibrarySync('internal', 'snap-1')
expect(accepted.job_id).toBe('libsync-internal-1')
const [url, init] = fetchMock.mock.calls[0]
expect(url).toBe('/api/v1/library/syncs')
expect(JSON.parse(init.body)).toEqual({ scope: 'internal', target_snapshot_id: 'snap-1' })
})
it('闸口未齐时把接口给的缺项原样抛出,而不是假装已启动', async () => {
vi.stubGlobal(
'fetch',
vi.fn().mockResolvedValue(
jsonResponse(503, {
code: 'library_sync_blocked',
reasons: ['目标非空且没有生效版本记录', '没有与目标快照匹配的有效发布授权'],
})
)
)
await expect(enqueueLibrarySync('internal', 'snap-1')).rejects.toThrow(
/目标非空且没有生效版本记录;没有与目标快照匹配的有效发布授权/
)
})
it('建立基线只传 range,并返回作业号', async () => {
const fetchMock = vi
.fn()
.mockResolvedValue(jsonResponse(202, { job: { id: 'libbase-internal-1', scope: 'internal' }, reused: false }))
vi.stubGlobal('fetch', fetchMock)
const accepted = await enqueueLibraryBaseline('internal')
expect(accepted.job_id).toBe('libbase-internal-1')
const [url, init] = fetchMock.mock.calls[0]
expect(url).toBe('/api/v1/library/baselines')
expect(JSON.parse(init.body)).toEqual({ scope: 'internal' })
})
it('建立基线被拒时照实说明(例如目标为空或已有生效版本)', async () => {
vi.stubGlobal(
'fetch',
vi.fn().mockResolvedValue(
jsonResponse(409, {
code: 'library_baseline_exists',
error: '该目标已有生效版本,不需要再建基线;换版本请走同步或恢复',
})
)
)
await expect(enqueueLibraryBaseline('internal')).rejects.toThrow(/该目标已有生效版本/)
})
})