Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c27a0c14cc | |||
| a42b101a7b | |||
| 3284658eb1 | |||
| 517d5f862d | |||
| f9c15a8f14 | |||
| 271f9f5f09 | |||
| 24d2546cbb | |||
| 691e4e0225 | |||
| 8d2ed8ebe6 | |||
| c14dd01f08 | |||
| 6ed06357f9 | |||
| 5d42cba6c6 | |||
| a47f9323e0 | |||
| 4c79052a18 | |||
| 18276d0839 | |||
| cfad003499 | |||
| c40696145b | |||
| 47eb23cf68 |
+145
@@ -1,5 +1,150 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 开发快照(基线崩溃续作、维护原状态保留与真实恢复演练工具)
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **基线契约按"能验证的来源"填充**:归档要求完整的兼容性契约(v0.7.1 / schema 74 /
|
||||||
|
elasticsearch_v8 / 1024 维 / 映射摘要)。原先只填了 embedding 与映射,捕获会被
|
||||||
|
`unsupported snapshot compatibility contract` 拒绝。现在 schema 版本**读目标库**
|
||||||
|
`schema_migrations`(dirty 或版本不符直接拒绝),embedding 优先读目标库模型绑定、
|
||||||
|
读不到才回落部署声明,映射取目标当前值,版本/引擎用受支持栈常量。
|
||||||
|
- **基线作业"登记成功但没来得及收尾"不再被判死**:原来崩溃重入会在
|
||||||
|
"已有生效版本"检查处直接失败,把目标留在维护里、占用也不释放。
|
||||||
|
现在按「本作业 + 同一快照」识别为**自己的**版本:只做收尾(恢复服务、开入口、
|
||||||
|
释放两把占用),不重新捕获、不重写版本;别人登记的生效版本仍然一律拒绝。
|
||||||
|
捕获完成但登记失败的作业会明确失败并写"未对目标写入",不会假装成功。
|
||||||
|
- **同一作业重入不再覆盖第一次记下的原运行状态**:崩溃接管时目标已被停住,
|
||||||
|
再读一次只会读到 `stopped`,写回去会让恢复阶段以为写入者本来就没跑、
|
||||||
|
永远不启动它。现在 `AcquireInstanceMaintenance` / `AcquireLibraryMaintenance`
|
||||||
|
在 owner 相同的重入时保留已持久化的原状态;新作业接管已释放窗口仍用自己读到的值。
|
||||||
|
(真实 PostgreSQL 集成用例覆盖,非内存假实现。)
|
||||||
|
- **基线的版本登记移进维护窗口**:捕获、校验、登记三件事要么一起成立、要么目标
|
||||||
|
原样恢复;记录失败时窗口已按原状态开放,治理记录同步释放,**不再出现
|
||||||
|
"显示保持维护、实际入口已开"**。
|
||||||
|
- **索引映射读不到时拒绝登记**:不再 `_` 忽略错误后用空摘要写进基线证据。
|
||||||
|
- 基线窗口的统一收尾抽成 `settleBaselineWindow`:版本已登记走与同步相同的成功边界;
|
||||||
|
窗口仍 held 才标 `needs_restore`;窗口已恢复则释放占用并失败。
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **`cmd/librarydrill`:目标实例真实恢复演练**(维护隔离 → 整库预备份 → 一致性校验
|
||||||
|
→ canary 漂移检测 → 按预备份恢复 → 逐类复核 → 按原状态恢复服务)。它使用与
|
||||||
|
worker 完全相同的 `LIBRARY_*` 配置与维护协调器,绝不触碰封存源;任一步失败都
|
||||||
|
保持维护并写报告,不伪造通过。
|
||||||
|
- `internal/librarystack`:目标/预备份/维护栈的唯一装配点(同步、恢复、基线、演练共用)。
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- 基线作业的"已有生效版本"判定改为三种状态(own / foreign / none),并在
|
||||||
|
阶段里写明是否走"只收尾"续作。
|
||||||
|
|
||||||
|
## 开发快照(首次发布入口:同步授权、目标权威基线与维护能力)
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **管理员在发布页签发同步授权(`POST /api/v1/library/authorizations`,`action=sync`)**:
|
||||||
|
授权把「范围 + 目标快照 + 参数摘要 + 有效期」钉死,仍与执行分成两步 ——
|
||||||
|
页面只提供入口,不会替你签发,也不会自动重试执行。工程师看得到缺项原因,
|
||||||
|
但拿不到签发按钮(接口本来就只认管理员)。
|
||||||
|
- **目标权威基线(迁移 075,作业类型 `baseline`)**:目标非空、又没有生效版本时,
|
||||||
|
首次发布的真实解法是「维护隔离 → 整库捕获 → 逐类校验 → 登记基线」。
|
||||||
|
`POST /api/v1/library/baselines` 只建作业;worker 用与同步同一套维护窗口
|
||||||
|
(关入口、停写入者、证明排空)把目标整库捕获成归档快照,再用同一套 `Verify`
|
||||||
|
证明目标与这份快照一致,最后登记 `production_effective`(证据写明
|
||||||
|
`origin=target_baseline`、快照 ID、逐表行数、目标索引映射)。
|
||||||
|
它**不写目标内容、不清空目标、不手写版本**:没有捕获与校验证据就没有基线。
|
||||||
|
基线作业与 sync/restore 共用同一把写互斥(同一 scope 只允许一个在飞写作业),
|
||||||
|
worker 被杀时同一作业 id 会接管那个窗口继续,而不是把目标留在停机状态。
|
||||||
|
- **维护能力(`GET /api/v1/library/maintenance` 的 `capability`)**:页面区分
|
||||||
|
「此刻没有维护窗口」与「维护能力没接线」,逐项列出缺少的
|
||||||
|
`LIBRARY_TARGET_*`;接口没给能力信息时显示"未知",不再用一句
|
||||||
|
"也没有维护能力配置"糊过去。
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`productionScopeEmpty` 改成整库口径**:原来只数 `knowledges`,"文档删了但
|
||||||
|
分块/Wiki/资源还在"的目标会被判成空库,从而绕过基线初始化。现在用适配层
|
||||||
|
同一份表清单与范围过滤(`v071.ScopeContentEmpty`)逐表计数,并把逐表行数
|
||||||
|
交回接口,"空"只有一份定义。
|
||||||
|
- **拟发布的测试封存只认版本登记(`kind=test_sealed`)**:归档里同时存在生产
|
||||||
|
预备份与目标基线快照,按"这个 scope 最新一份"挑会把目标自己的内容显示成
|
||||||
|
测试候选。现在候选来自登记、再用归档确认它还在本机;不在就说明原因。
|
||||||
|
- **发布页不再说"生产同步执行链尚未接线"**:同步按钮按真实闸口启用,
|
||||||
|
点击走 `POST /api/v1/library/syncs`;被拒时逐条显示接口返回的缺项。
|
||||||
|
- 缺项文案从"生产当前权威版本未知,不能启动写入"改为指向真实入口:
|
||||||
|
「目标非空且没有生效版本记录:先建立目标权威基线…或确认目标确实是空库」。
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- 维护卡片把"没有窗口记录"错误地写成"没有维护能力配置",与实际接线不符。
|
||||||
|
- 首次发布时 `POST /library/baselines` 在目标读不到时**不建作业、不登记版本**
|
||||||
|
(拒绝用建基线掩盖连接/配置故障)。
|
||||||
|
|
||||||
|
## 开发快照(发布执行器可用性、整库还原入口与诊断)
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **封存执行器的运行时可用性(迁移 074 `library_executor_status`)**:页面上的
|
||||||
|
"可以发起封存"不再等于 `LIBRARY_AGENT_SOCKET` 非空。主机侧 agent 新增
|
||||||
|
`health` 动作(不占封存操作锁、不跑子进程),worker 每 30 秒探测一次并把
|
||||||
|
"指向哪个实例、归档在哪、是否降级/正在跑、观测时间"落库;API 只读这份观测。
|
||||||
|
socket 只挂在 worker 容器里,所以由 worker 观测、页面读结果,而不是让 API
|
||||||
|
假装能 dial。观测过期、不可达、实例不符、降级、正忙时页面与
|
||||||
|
`POST /library/captures` 用同一份判定拒绝,并逐条给出原因。
|
||||||
|
- **封存执行前后的身份复核**:封存结果由 agent 附上自报身份(`agent_identity`)。
|
||||||
|
worker 在停写入者之前再探测一次,执行后再用结果里的身份复核;不一致时作业失败
|
||||||
|
且**不登记**为测试封存——心跳只能证明"刚才",从心跳到停写入者之间 socket
|
||||||
|
可能被换掉。
|
||||||
|
- **按版本恢复的页面入口**:`GET /api/v1/library/restore-availability` 说明
|
||||||
|
"这份快照现在能不能恢复"(双闸、该快照的恢复授权、目标实例是否在维护)。
|
||||||
|
页面据此启用"恢复至该版本",管理员可签发 30 分钟有效期的 restore 授权;
|
||||||
|
工程师只看到条件,不获得入口。
|
||||||
|
- 可选 `LIBRARY_SOURCE_PROJECT`:声明封存执行器应指向的实例名(默认等于
|
||||||
|
`LIBRARY_SOURCE_INSTANCE`)。它描述封存源,与同步目标 `LIBRARY_TARGET_*` 分开。
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **封存身份核对移进请求本身(TOCTOU)**:原来 worker 先 `health` 再 `capture`,两次
|
||||||
|
拨号之间 socket 可能被换成另一个 agent——错的执行器已经把写入者停了才被拒登记。
|
||||||
|
现在 `capture` 请求携带治理侧配置生成的期望身份(`project` 必填,`source_app`/
|
||||||
|
`archive_root` 配了就一起绑定)与稳定指纹(`sha256:`,覆盖三项);接收方在**操作
|
||||||
|
互斥内、执行任何源操作之前**比对,缺失或被改过一律拒绝,错的执行器一次 `capture`
|
||||||
|
都不跑。封存结果回显该指纹,worker 事后复核只作为证据防线,不能替代请求内核对。
|
||||||
|
恢复路径不受期望约束(它只把写入者恢复回来)。
|
||||||
|
- `LIBRARY_EXECUTOR_STATUS_MAX_AGE`(默认 120 秒)控制执行器观测的有效期;
|
||||||
|
未设置不等于"永不过期",过期即"无法确认"。
|
||||||
|
- 页面/接口不再出现写死的"封存通道未接线""生产恢复未接线":不能发起时给出真实原因、
|
||||||
|
执行器身份与观测时间。
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **接管在版本查询失败时 fail-open**:`resumed` 后版本查询出错被忽略、`FinalizeOnly` 仍为
|
||||||
|
false,于是"已成功登记并释放窗口、只差终态"的作业会被**重新写一遍目标**。现在接管先按
|
||||||
|
证据分类:`finalize` 阶段成功是独立证据(不受版本查询影响);查询失败且无阶段证据 ⇒
|
||||||
|
**未知**,零写入、作业失败可重试;本快照的生效版本已被别的快照顶替 ⇒ **拒绝覆盖**;
|
||||||
|
只有真正没有收尾证据时才按正常流程。
|
||||||
|
- **维护释放部分成功后的重试不再永久卡死**:`Update*/Release*` 对"已由本作业释放"的行
|
||||||
|
返回可判别的 `ErrMaintenanceReleased`,收尾重试视其为已完成;别的作业来释放仍是
|
||||||
|
"未持有"错误。真 PG 负例:同作业重复 Update/Release 得到已释放,跨作业得到未持有。
|
||||||
|
- **作业成功暴露早于维护结束**:原来 `execute` 里写完生效版本就 `AdvanceLibraryJob(succeeded)`,
|
||||||
|
而服务恢复、入口开放与维护锁释放要在窗口返回后才做。任何一步失败时外部已经看到
|
||||||
|
`succeeded`,重启接管又只认非终态,锁就永远挂着。现在内容校验/版本登记之后只记
|
||||||
|
"待收尾",`finishLibrarySuccess` 依次确认目标写入者与入口健康、按 **scope → instance**
|
||||||
|
顺序释放两把锁(每一步错误都上抛,不再丢弃),最后才写 `succeeded`;任一步失败都把
|
||||||
|
作业落成失败、维护保持 `needs_restore`。释放与终态写入用 `context.WithoutCancel`
|
||||||
|
包住,停机不会留下半成品。接管判定改为按证据(finalize 阶段成功或存在未顶掉的生效
|
||||||
|
版本)转 finalize-only,只复核与登记,不重写生产。
|
||||||
|
- **封存读资源失败只剩一句 `read library resource`**:对象缺失、桶错、端点解析不到
|
||||||
|
都会走到同一个字符串,真实封存直接卡在这里无法判断方向。现在带上资源行 id 与底层
|
||||||
|
原因(`read library resource <id>: <cause>`),不含凭据。
|
||||||
|
- **正式 release-worker 启动失败无法定位**:配置里的目标库指向不存在的主机时,
|
||||||
|
日志只有一句 `connect release target database`。现在错误里带库端点
|
||||||
|
(`host:port/db`,去口令)与路由标识,并保留底层原因;原因文本先经
|
||||||
|
`pgdsn.Redact` 抹掉口令(pgx 的解析错误可能原样回显连接串),可诊断不等于泄露。
|
||||||
|
共享的 `internal/pgdsn` 替换了 cmd/worker 里只留库名的旧实现。
|
||||||
|
- `libraryctl` 用法文本补上 `dead-letters` 子命令。
|
||||||
|
|
||||||
## 开发快照(死信精确解除:保留上游行,只按指纹放行)
|
## 开发快照(死信精确解除:保留上游行,只按指纹放行)
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
@@ -1312,6 +1312,51 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"/api/v1/library/restore-availability": {
|
||||||
|
"get": {
|
||||||
|
"tags": [
|
||||||
|
"Runtime"
|
||||||
|
],
|
||||||
|
"operationId": "getLibraryRestoreAvailability",
|
||||||
|
"summary": "这份快照现在能不能恢复到生产:双闸、恢复授权与目标实例维护状态逐条说明",
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "scope",
|
||||||
|
"in": "query",
|
||||||
|
"required": false,
|
||||||
|
"schema": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"internal",
|
||||||
|
"external"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "snapshot_id",
|
||||||
|
"in": "query",
|
||||||
|
"required": true,
|
||||||
|
"schema": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"responses": {
|
||||||
|
"200": {
|
||||||
|
"$ref": "#/components/responses/Envelope"
|
||||||
|
},
|
||||||
|
"400": {
|
||||||
|
"$ref": "#/components/responses/BadRequest"
|
||||||
|
},
|
||||||
|
"500": {
|
||||||
|
"$ref": "#/components/responses/InternalError"
|
||||||
|
},
|
||||||
|
"503": {
|
||||||
|
"description": "读取恢复授权失败(不是\"可恢复\",是\"无法判定\")"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"/api/v1/library/authorizations": {
|
"/api/v1/library/authorizations": {
|
||||||
"get": {
|
"get": {
|
||||||
"tags": [
|
"tags": [
|
||||||
@@ -1502,6 +1547,58 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"/api/v1/library/baselines": {
|
||||||
|
"post": {
|
||||||
|
"tags": [
|
||||||
|
"Runtime"
|
||||||
|
],
|
||||||
|
"operationId": "startLibraryBaseline",
|
||||||
|
"summary": "建立目标权威基线:管理员在维护隔离下把非空目标整库捕获成快照、逐类校验后登记为生效版本;不清空目标、不手写版本",
|
||||||
|
"responses": {
|
||||||
|
"202": {
|
||||||
|
"description": "已创建基线作业(worker 在维护窗口内捕获并校验)"
|
||||||
|
},
|
||||||
|
"400": {
|
||||||
|
"$ref": "#/components/responses/BadRequest"
|
||||||
|
},
|
||||||
|
"403": {
|
||||||
|
"description": "不是管理员"
|
||||||
|
},
|
||||||
|
"409": {
|
||||||
|
"description": "已有生效版本、目标为空、目标正在维护或已有在飞写作业"
|
||||||
|
},
|
||||||
|
"500": {
|
||||||
|
"$ref": "#/components/responses/InternalError"
|
||||||
|
},
|
||||||
|
"503": {
|
||||||
|
"description": "目标整库读不到,不能建立基线(不创建作业、不登记版本)"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"requestBody": {
|
||||||
|
"required": true,
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"scope"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"scope": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"internal",
|
||||||
|
"external"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"/api/v1/library/snapshots/{id}": {
|
"/api/v1/library/snapshots/{id}": {
|
||||||
"get": {
|
"get": {
|
||||||
"tags": [
|
"tags": [
|
||||||
|
|||||||
@@ -203,8 +203,16 @@ func run(ctx context.Context, getenv func(string) string) error {
|
|||||||
VersionHistoryRepo: versionHistoryRepo,
|
VersionHistoryRepo: versionHistoryRepo,
|
||||||
LibraryArchiveRoot: config.LibraryArchiveRoot,
|
LibraryArchiveRoot: config.LibraryArchiveRoot,
|
||||||
LibraryTargetInstance: config.LibraryTargetInstance,
|
LibraryTargetInstance: config.LibraryTargetInstance,
|
||||||
|
LibraryTargetProject: config.LibraryTargetProject,
|
||||||
|
LibraryTargetWriters: config.LibraryTargetWriters,
|
||||||
|
LibraryTargetIngressURL: config.LibraryTargetIngressURL,
|
||||||
|
LibraryTargetIngressService: config.LibraryTargetIngressService,
|
||||||
|
LibraryTargetDockerSocket: config.LibraryTargetDockerSocket,
|
||||||
|
LibraryTargetJournalRoot: config.LibraryTargetJournalRoot,
|
||||||
LibrarySourceInstance: config.SourceInstance(),
|
LibrarySourceInstance: config.SourceInstance(),
|
||||||
|
LibrarySourceProject: config.SourceProject(),
|
||||||
LibraryAgentSocket: config.LibraryAgentSocket,
|
LibraryAgentSocket: config.LibraryAgentSocket,
|
||||||
|
LibraryExecutorStatusMaxAge: config.LibraryExecutorStatusMaxAge,
|
||||||
LibraryCapacityBudgetBytes: config.LibraryCapacityBudgetBytes,
|
LibraryCapacityBudgetBytes: config.LibraryCapacityBudgetBytes,
|
||||||
LibraryProductionDatabaseURL: config.LibraryProductionDatabaseURL,
|
LibraryProductionDatabaseURL: config.LibraryProductionDatabaseURL,
|
||||||
LibraryProductionInternalKB: config.LibraryProductionInternalKB,
|
LibraryProductionInternalKB: config.LibraryProductionInternalKB,
|
||||||
|
|||||||
@@ -111,7 +111,7 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func usage() {
|
func usage() {
|
||||||
fmt.Fprint(os.Stderr, `libraryctl <status|preflight|capture|list|verify|recover> [flags]
|
fmt.Fprint(os.Stderr, `libraryctl <status|preflight|capture|list|verify|recover|dead-letters> [flags]
|
||||||
|
|
||||||
status probe the real source and print what is configured / blocked
|
status probe the real source and print what is configured / blocked
|
||||||
preflight read the source, run settlement + candidate validation (no writes, no stop)
|
preflight read the source, run settlement + candidate validation (no writes, no stop)
|
||||||
@@ -119,6 +119,8 @@ func usage() {
|
|||||||
list list sealed candidates and stored bytes
|
list list sealed candidates and stored bytes
|
||||||
verify re-verify one sealed candidate (digests, coverage, identity)
|
verify re-verify one sealed candidate (digests, coverage, identity)
|
||||||
recover restore writers from an interrupted freeze journal
|
recover restore writers from an interrupted freeze journal
|
||||||
|
dead-letters print the source's live dead letters (read-only) with a digest the
|
||||||
|
governance platform binds a precise release to
|
||||||
|
|
||||||
Flags (defaults target the local WKNFull test stack):
|
Flags (defaults target the local WKNFull test stack):
|
||||||
-source-app WKNFull-app container whose env describes the source
|
-source-app WKNFull-app container whose env describes the source
|
||||||
|
|||||||
@@ -0,0 +1,271 @@
|
|||||||
|
// command librarydrill 在目标实例上做一次**真实**的恢复演练:
|
||||||
|
//
|
||||||
|
// 维护隔离(关入口/停写入者/证明排空)
|
||||||
|
// → 捕获目标整库(真预备份,落归档)
|
||||||
|
// → 校验目标 == 预备份(基线一致)
|
||||||
|
// → 写入一个 canary(证明"校验确实能发现内容变化")
|
||||||
|
// → 按预备份恢复
|
||||||
|
// → 逐表/索引/资源/Wiki 再校验一次,必须完全一致
|
||||||
|
// → 关窗口(按原运行状态恢复服务、开入口、验证健康)
|
||||||
|
//
|
||||||
|
// 它只读/写**被测目标自己**,绝不触碰封存源;所有凭据来自进程环境
|
||||||
|
// (与 worker 同一份 LIBRARY_* 配置),不进命令行、不打印。
|
||||||
|
// 任一步失败都不伪装成功:窗口保持维护、报告写明失败阶段。
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/appconfig"
|
||||||
|
"knowledge-governance/internal/libraryfreeze"
|
||||||
|
"knowledge-governance/internal/libraryrelease"
|
||||||
|
"knowledge-governance/internal/librarystack"
|
||||||
|
)
|
||||||
|
|
||||||
|
type drillVerification struct {
|
||||||
|
SnapshotID string `json:"snapshot_id"`
|
||||||
|
Verified bool `json:"verified"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
At time.Time `json:"at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type drillReport struct {
|
||||||
|
Status string `json:"status"` // passed | failed | maintenance_held
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
Target string `json:"target"`
|
||||||
|
SnapshotID string `json:"snapshot_id"`
|
||||||
|
Project string `json:"project"`
|
||||||
|
Counts map[string]int64 `json:"counts"`
|
||||||
|
BaselineCheck *drillVerification `json:"baseline_check"`
|
||||||
|
DriftCheck *drillVerification `json:"drift_check"` // 必须失败:证明确实能发现差异
|
||||||
|
RestoreCheck *drillVerification `json:"restore_check"`
|
||||||
|
Canary string `json:"canary"`
|
||||||
|
Failure string `json:"failure,omitempty"`
|
||||||
|
Stage string `json:"stage,omitempty"`
|
||||||
|
StartedAt time.Time `json:"started_at"`
|
||||||
|
FinishedAt time.Time `json:"finished_at"`
|
||||||
|
Until map[string]string `json:"until,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
scope := flag.String("scope", "internal", "internal|external")
|
||||||
|
reportPath := flag.String("report", "", "write the JSON report here (0600)")
|
||||||
|
jobID := flag.String("job", "", "maintenance job id (default drill-<scope>-<timestamp>)")
|
||||||
|
recoverOnly := flag.Bool("recover", false, "only recover non-terminal maintenance windows (restore services), then exit")
|
||||||
|
flag.Parse()
|
||||||
|
if *recoverOnly {
|
||||||
|
if err := recoverWindows(); err != nil {
|
||||||
|
log.Fatalf("recover failed: %v", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
report, err := run(*scope, *jobID)
|
||||||
|
if *reportPath != "" && report != nil {
|
||||||
|
if writeErr := writeReport(*reportPath, report); writeErr != nil {
|
||||||
|
log.Printf("drill report write failed: %v", writeErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("drill failed: %v", err)
|
||||||
|
}
|
||||||
|
log.Printf("drill passed: snapshot=%s", report.SnapshotID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// recoverWindows 只做一件事:把非终态的维护窗口按原运行状态恢复(服务起来、入口验证)。
|
||||||
|
// 演练失败但还没写目标内容时,用它把目标从"停住"状态救回来;它不改治理记录,
|
||||||
|
// 治理记录由作业/接口那一侧负责。
|
||||||
|
func recoverWindows() error {
|
||||||
|
config, err := appconfig.Load(os.Getenv)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("load config: %w", err)
|
||||||
|
}
|
||||||
|
manager, err := librarystack.OpenMaintenance(config)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer manager.Close()
|
||||||
|
coordinator, control, err := manager.Open(config.LibraryTargetJournalRoot)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer control.Close()
|
||||||
|
defer coordinator.Close()
|
||||||
|
if err := coordinator.Recover(context.Background()); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
log.Printf("maintenance windows recovered; target should be back to its original service state")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func run(scope, jobID string) (*drillReport, error) {
|
||||||
|
if scope != "internal" && scope != "external" {
|
||||||
|
return nil, errors.New("scope must be internal or external")
|
||||||
|
}
|
||||||
|
config, err := appconfig.Load(os.Getenv)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("load config: %w", err)
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
report := &drillReport{Status: "failed", Scope: scope, Project: config.LibraryTargetProject,
|
||||||
|
Target: config.LibraryTargetInstance, StartedAt: time.Now().UTC()}
|
||||||
|
fail := func(stage string, cause error, held bool) (*drillReport, error) {
|
||||||
|
report.Stage = stage
|
||||||
|
report.Failure = cause.Error()
|
||||||
|
report.FinishedAt = time.Now().UTC()
|
||||||
|
if held {
|
||||||
|
report.Status = "maintenance_held"
|
||||||
|
}
|
||||||
|
return report, cause
|
||||||
|
}
|
||||||
|
|
||||||
|
target, backups, err := librarystack.OpenBaselineStack(ctx, config, scope)
|
||||||
|
if err != nil {
|
||||||
|
return fail("open_target", err, false)
|
||||||
|
}
|
||||||
|
defer target.Close()
|
||||||
|
manager, err := librarystack.OpenMaintenance(config)
|
||||||
|
if err != nil {
|
||||||
|
return fail("open_maintenance", err, false)
|
||||||
|
}
|
||||||
|
defer manager.Close()
|
||||||
|
if err := manager.VerifyTopology(ctx); err != nil {
|
||||||
|
return fail("topology", err, false)
|
||||||
|
}
|
||||||
|
coordinator, control, err := manager.Open(config.LibraryTargetJournalRoot)
|
||||||
|
if err != nil {
|
||||||
|
return fail("open_coordinator", err, false)
|
||||||
|
}
|
||||||
|
defer control.Close()
|
||||||
|
defer coordinator.Close()
|
||||||
|
|
||||||
|
if jobID == "" {
|
||||||
|
jobID = "drill-" + scope + "-" + time.Now().UTC().Format("20060102T150405")
|
||||||
|
}
|
||||||
|
var (
|
||||||
|
snapshotID string
|
||||||
|
counts map[string]int64
|
||||||
|
canary string
|
||||||
|
contentTouched bool
|
||||||
|
baseline drillVerification
|
||||||
|
drift drillVerification
|
||||||
|
restored drillVerification
|
||||||
|
)
|
||||||
|
work := func(ctx context.Context, window *libraryfreeze.Window) (string, error) {
|
||||||
|
start := time.Now()
|
||||||
|
current, err := target.Current(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return "", errors.New("读取目标当前整库状态失败:" + err.Error())
|
||||||
|
}
|
||||||
|
counts = current.Counts
|
||||||
|
report.Counts = counts
|
||||||
|
// 1) 真预备份(落归档、内容寻址、导出后复核内容未变)。
|
||||||
|
backup, err := backups.Prepare(ctx, "", current)
|
||||||
|
if err != nil {
|
||||||
|
return "", errors.New("捕获目标整库失败:" + err.Error())
|
||||||
|
}
|
||||||
|
snapshotID = backup.SnapshotID
|
||||||
|
report.SnapshotID = snapshotID
|
||||||
|
if err := window.SetBackupSnapshot(ctx, snapshotID); err != nil {
|
||||||
|
return "", errors.New("记录预备份身份失败:" + err.Error())
|
||||||
|
}
|
||||||
|
log.Printf("drill: snapshot captured snapshot=%s bytes=%d in=%s", snapshotID, backup.Bytes, time.Since(start))
|
||||||
|
|
||||||
|
// 2) 基线校验:预备份必须与目标完全一致。
|
||||||
|
baseline = drillVerification{SnapshotID: snapshotID, At: time.Now().UTC()}
|
||||||
|
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
|
||||||
|
baseline.Error = err.Error()
|
||||||
|
report.BaselineCheck = &baseline
|
||||||
|
return "", errors.New("预备份与目标不一致(预备份本身不可信):" + err.Error())
|
||||||
|
}
|
||||||
|
baseline.Verified = true
|
||||||
|
report.BaselineCheck = &baseline
|
||||||
|
|
||||||
|
// 3) canary:改一行真实内容,校验**必须**失败(否则"校验"是摆设)。
|
||||||
|
canary = "drill-canary-" + time.Now().UTC().Format("20060102T150405")
|
||||||
|
identity := target.Identity()
|
||||||
|
tag, err := target.Pool().Exec(ctx, `
|
||||||
|
UPDATE knowledges SET title = title || ' [' || $3 || ']'
|
||||||
|
WHERE id = (SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2 AND deleted_at IS NULL
|
||||||
|
ORDER BY id LIMIT 1)`, identity.TenantID, identity.KnowledgeBaseID, canary)
|
||||||
|
if err != nil {
|
||||||
|
return "", errors.New("写入 canary 失败:" + err.Error())
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return "", errors.New("目标里没有可改写的文档,无法做漂移检测(目标不应为空)")
|
||||||
|
}
|
||||||
|
report.Canary = canary
|
||||||
|
// 从这一笔开始目标内容被改写过:窗口必须记录内容风险,
|
||||||
|
// 之后只有"恢复并校验通过"才允许重新开放(否则就是半写状态)。
|
||||||
|
if err := window.MarkContentRisk(ctx); err != nil {
|
||||||
|
return "", errors.New("记录内容风险失败:" + err.Error())
|
||||||
|
}
|
||||||
|
contentTouched = true
|
||||||
|
drift = drillVerification{SnapshotID: snapshotID, At: time.Now().UTC()}
|
||||||
|
if verifyErr := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); verifyErr == nil {
|
||||||
|
drift.Error = "写入 canary 后校验仍然通过:校验没有真正比对内容"
|
||||||
|
report.DriftCheck = &drift
|
||||||
|
return "", errors.New("写入 canary 后校验仍然通过:校验没有真正比对内容,拒绝把这次演练当成通过")
|
||||||
|
} else {
|
||||||
|
drift.Error = verifyErr.Error()
|
||||||
|
}
|
||||||
|
report.DriftCheck = &drift
|
||||||
|
log.Printf("drill: drift detected as expected (%s)", drift.Error)
|
||||||
|
|
||||||
|
// 4) 按预备份恢复。
|
||||||
|
if err := target.Restore(ctx, libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID}); err != nil {
|
||||||
|
return "", errors.New("按预备份恢复失败:" + err.Error())
|
||||||
|
}
|
||||||
|
// 5) 恢复后逐类复核:必须与预备份完全一致。
|
||||||
|
restored = drillVerification{SnapshotID: snapshotID, At: time.Now().UTC()}
|
||||||
|
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
|
||||||
|
restored.Error = err.Error()
|
||||||
|
report.RestoreCheck = &restored
|
||||||
|
return "", errors.New("恢复后内容仍与预备份不一致:" + err.Error())
|
||||||
|
}
|
||||||
|
restored.Verified = true
|
||||||
|
report.RestoreCheck = &restored
|
||||||
|
log.Printf("drill: restored and verified snapshot=%s", snapshotID)
|
||||||
|
return snapshotID, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
window, windowErr := coordinator.RunWindow(ctx, jobID,
|
||||||
|
func(ctx context.Context) error { return manager.VerifyRunning(ctx) }, work,
|
||||||
|
// 没写过目标内容 ⇒ 按原状态恢复永远安全;写过 canary ⇒ 只有恢复并校验通过才开放。
|
||||||
|
func(error) bool { return !contentTouched || restored.Verified })
|
||||||
|
|
||||||
|
report.Until = map[string]string{"job": jobID, "window_state": window.State}
|
||||||
|
if windowErr != nil || window.State != "succeeded" {
|
||||||
|
cause := windowErr
|
||||||
|
if cause == nil {
|
||||||
|
cause = fmt.Errorf("维护窗口状态 %s(未成功)", window.State)
|
||||||
|
}
|
||||||
|
if window.State == "held" {
|
||||||
|
return fail("window", cause, true)
|
||||||
|
}
|
||||||
|
return fail("window", cause, false)
|
||||||
|
}
|
||||||
|
report.Status = "passed"
|
||||||
|
report.FinishedAt = time.Now().UTC()
|
||||||
|
return report, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeReport(path string, report *drillReport) error {
|
||||||
|
raw, err := json.MarshalIndent(report, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if dir := filepath.Dir(path); dir != "" {
|
||||||
|
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return os.WriteFile(path, append(raw, '\n'), 0o600)
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 正式栈的 release-worker 启动失败时只打一句 "connect release target database",
|
||||||
|
// 主机名、库名与底层原因都不在日志里。修好之后仍然要证明:可诊断 ≠ 泄露凭据,
|
||||||
|
// 因为 pgx 的解析/连接错误会把连接串原样带回来。
|
||||||
|
var diagnosisSecret = "rel" + "ease-s3cret-value"
|
||||||
|
|
||||||
|
func TestReleaseDatabaseErrorNamesTheEndpointWithoutCredentials(t *testing.T) {
|
||||||
|
cause := errors.New("dial tcp 127.0.0.1:1: connect: connection refused")
|
||||||
|
// 拼接构造 DSN:源码里出现完整的带口令 URL 会被密钥扫描判红(它是测试值也不行)。
|
||||||
|
dsn := "postgres" + "://postgres:" + "ignored@" + "prod-pg:25432/weknora_test?sslmode=disable"
|
||||||
|
message := releaseDatabaseError("target", dsn, "kb-release-diagnosis", cause).Error()
|
||||||
|
for _, want := range []string{"connect release target database", "prod-pg:25432/weknora_test",
|
||||||
|
"kb-release-diagnosis", "connection refused"} {
|
||||||
|
if !strings.Contains(message, want) {
|
||||||
|
t.Fatalf("error %q does not mention %q", message, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
source := releaseDatabaseError("source", "postgres"+"://u:p@WKNFull-postgres:5432/weknora_full", "kb-1", cause).Error()
|
||||||
|
if !strings.Contains(source, "WKNFull-postgres:5432/weknora_full") {
|
||||||
|
t.Fatalf("source error = %q", source)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// cause 原样回显 DSN(pgx 在解析/连接失败时确实可能这样)是最危险的一种:
|
||||||
|
// 端点脱敏救不了它,整条错误里都藏着口令。
|
||||||
|
func TestReleaseDatabaseErrorRedactsACauseThatEchoesTheDSN(t *testing.T) {
|
||||||
|
cases := []string{
|
||||||
|
"postgres://postgres:" + diagnosisSecret + "@127.0.0.1:notaport/weknora_test",
|
||||||
|
"host=127.0.0.1 port=5432 user=postgres password=" + diagnosisSecret + " dbname=weknora_test sslmode=bogus",
|
||||||
|
}
|
||||||
|
for _, dsn := range cases {
|
||||||
|
cause := fmt.Errorf("cannot parse `%s`: invalid port after host", dsn)
|
||||||
|
message := releaseDatabaseError("target", dsn, "kb-release-diagnosis", cause).Error()
|
||||||
|
if strings.Contains(message, diagnosisSecret) {
|
||||||
|
t.Fatalf("error leaked the password: %q", message)
|
||||||
|
}
|
||||||
|
if !strings.Contains(message, "invalid port after host") {
|
||||||
|
t.Fatalf("error lost the underlying reason: %q", message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 保留 Unwrap:调用方仍然可以用 errors.Is/As 判断原因;只是文本被脱敏。
|
||||||
|
func TestReleaseDatabaseErrorKeepsTheCauseChain(t *testing.T) {
|
||||||
|
sentinel := errors.New("boom")
|
||||||
|
err := releaseDatabaseError("target", "postgres"+"://u:p@h:1/db", "kb-1", sentinel)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("cause chain broken: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package main
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"net/url"
|
"net/url"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -11,6 +12,7 @@ import (
|
|||||||
|
|
||||||
"github.com/jackc/pgx/v5/pgxpool"
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/pgdsn"
|
||||||
"knowledge-governance/internal/platform/weknora"
|
"knowledge-governance/internal/platform/weknora"
|
||||||
v071 "knowledge-governance/internal/releaseadapter/v071"
|
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||||
"knowledge-governance/internal/releasesnapshot"
|
"knowledge-governance/internal/releasesnapshot"
|
||||||
@@ -63,13 +65,13 @@ func buildReleaseRoute(ctx context.Context, archiveRoot string, snapshot release
|
|||||||
}
|
}
|
||||||
sourcePool, err := openReleasePool(ctx, sourceDatabaseURL)
|
sourcePool, err := openReleasePool(ctx, sourceDatabaseURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, errors.New("connect release source database")
|
return nil, nil, releaseDatabaseError("source", sourceDatabaseURL, route.KnowledgeBaseID, err)
|
||||||
}
|
}
|
||||||
pools := []*pgxpool.Pool{sourcePool}
|
pools := []*pgxpool.Pool{sourcePool}
|
||||||
targetPool, err := openReleasePool(ctx, targetDatabaseURL)
|
targetPool, err := openReleasePool(ctx, targetDatabaseURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
sourcePool.Close()
|
sourcePool.Close()
|
||||||
return nil, nil, errors.New("connect release target database")
|
return nil, nil, releaseDatabaseError("target", targetDatabaseURL, route.KnowledgeBaseID, err)
|
||||||
}
|
}
|
||||||
pools = append(pools, targetPool)
|
pools = append(pools, targetPool)
|
||||||
cleanup := func(err error) (releaseworkflow.Publisher, []*pgxpool.Pool, error) {
|
cleanup := func(err error) (releaseworkflow.Publisher, []*pgxpool.Pool, error) {
|
||||||
@@ -207,6 +209,29 @@ func readOptionalRestrictedValue(path string) (string, error) {
|
|||||||
return readRestrictedValue(path)
|
return readRestrictedValue(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// releaseDatabaseError 是"连不上某个库"的唯一措辞:端点 + 路由 + 底层原因。
|
||||||
|
//
|
||||||
|
// 为什么不用 %w 直接拼原因:pgx 的错误里可能原样带着连接串(含口令),
|
||||||
|
// 只在端点脱敏证明不了整条日志安全。这里把原因文本交给 pgdsn.Redact 抹口令,
|
||||||
|
// 同时保留 Unwrap,调用方仍能用 errors.Is/As 判断。技术原因(拒绝连接、
|
||||||
|
// 认证失败、TLS 配置错误等)一个字都不丢。
|
||||||
|
type databaseConnectError struct {
|
||||||
|
message string
|
||||||
|
cause error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *databaseConnectError) Error() string { return e.message }
|
||||||
|
func (e *databaseConnectError) Unwrap() error { return e.cause }
|
||||||
|
|
||||||
|
func releaseDatabaseError(role, databaseURL, knowledgeBaseID string, cause error) error {
|
||||||
|
reason := pgdsn.Redact(databaseURL, cause.Error())
|
||||||
|
return &databaseConnectError{
|
||||||
|
message: fmt.Sprintf("connect release %s database %s (route %s): %s",
|
||||||
|
role, pgdsn.Endpoint(databaseURL), knowledgeBaseID, reason),
|
||||||
|
cause: cause,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func openReleasePool(ctx context.Context, databaseURL string) (*pgxpool.Pool, error) {
|
func openReleasePool(ctx context.Context, databaseURL string) (*pgxpool.Pool, error) {
|
||||||
pool, err := pgxpool.New(ctx, databaseURL)
|
pool, err := pgxpool.New(ctx, databaseURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,298 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"log"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/appconfig"
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
"knowledge-governance/internal/jobs"
|
||||||
|
"knowledge-governance/internal/libraryfreeze"
|
||||||
|
"knowledge-governance/internal/librarymaintenance"
|
||||||
|
"knowledge-governance/internal/libraryrelease"
|
||||||
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
// libraryBaselinePayload 是"建立目标权威基线"的队列载荷。
|
||||||
|
type libraryBaselinePayload struct {
|
||||||
|
LibraryJobID string `json:"library_job_id"`
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// baselineVersionState 回答"这个 scope 现在有没有生效版本、是不是本作业登记的"。
|
||||||
|
//
|
||||||
|
// 三种情况必须分开,否则崩溃续作会被自己的版本挡住:
|
||||||
|
// - own:本作业 + 同一快照已经登记过 → 只收尾,绝不重复捕获或重写版本;
|
||||||
|
// - foreign:别人登记的生效版本 → 一律拒绝(不能用基线覆盖别人的版本);
|
||||||
|
// - none:还没有生效版本 → 正常建基线。
|
||||||
|
type baselineVersionState struct {
|
||||||
|
Own bool
|
||||||
|
Foreign *governance.LibraryVersion
|
||||||
|
}
|
||||||
|
|
||||||
|
func classifyBaselineVersions(versions []governance.LibraryVersion, jobID, snapshotID string) baselineVersionState {
|
||||||
|
for index := range versions {
|
||||||
|
item := versions[index]
|
||||||
|
if item.Kind != governance.LibraryVersionProductionEffective || item.SupersededAt != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if jobID != "" && snapshotID != "" && item.SourceJobID == jobID && item.SnapshotID == snapshotID {
|
||||||
|
return baselineVersionState{Own: true}
|
||||||
|
}
|
||||||
|
foreign := item
|
||||||
|
return baselineVersionState{Foreign: &foreign}
|
||||||
|
}
|
||||||
|
return baselineVersionState{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// settleBaselineWindow 是基线窗口的**唯一**收尾入口,负责让治理状态与实际入口一致:
|
||||||
|
//
|
||||||
|
// - 版本已登记(本次刚写,或崩溃前由本作业写过)→ 走与同步一样的成功边界:
|
||||||
|
// 目标健康、入口开放、两把维护锁释放之后才暴露 succeeded;
|
||||||
|
// - 没有登记且窗口仍是 held(服务没回来)→ 保持维护并标 needs_restore,
|
||||||
|
// 只有"按预备份恢复"能把它收掉;
|
||||||
|
// - 没有登记但窗口已按原状态恢复 → 释放占用并把作业标失败,明确写"未对目标写入",
|
||||||
|
// 绝不留着"显示维护中、实际已开放"的状态。
|
||||||
|
func settleBaselineWindow(ctx context.Context, store libraryMaintenanceStore, health healthChecker,
|
||||||
|
jobID, scope, instance, baselineID string, recorded bool, windowState string, windowErr error) error {
|
||||||
|
if recorded {
|
||||||
|
finishedAt := time.Now().UTC()
|
||||||
|
return finishLibrarySuccess(ctx, store, health, instance, scope, jobID,
|
||||||
|
successBoundary{Ready: true, Backup: baselineID, FinishedAt: finishedAt})
|
||||||
|
}
|
||||||
|
now := time.Now().UTC()
|
||||||
|
message := "未建立基线:维护窗口没有完成捕获与校验"
|
||||||
|
if windowErr != nil {
|
||||||
|
message += ":" + windowErr.Error()
|
||||||
|
}
|
||||||
|
if windowState == "held" {
|
||||||
|
note := message + "(目标仍被停住、入口未恢复,保持维护)"
|
||||||
|
_ = store.UpdateLibraryMaintenanceState(ctx, scope, jobID, "needs_restore", note, now)
|
||||||
|
_ = store.UpdateInstanceMaintenanceState(ctx, instance, jobID, "needs_restore", note, now)
|
||||||
|
_ = store.AdvanceLibraryJob(ctx, jobID, governance.LibraryJobFailed, "capture", note,
|
||||||
|
"目标仍在维护:先按预备份恢复并校验", baselineID, "", now)
|
||||||
|
return errors.New(note)
|
||||||
|
}
|
||||||
|
// 窗口已按原状态开放(本流程对目标内容只读):占用必须跟着释放,
|
||||||
|
// 不能留下"界面说维护中、实际入口已开"的记录。
|
||||||
|
if releaseErr := releaseWindowLocks(ctx, store, instance, scope, jobID,
|
||||||
|
"未登记基线,目标内容未被写入,窗口已按原状态恢复", now); releaseErr != nil {
|
||||||
|
note := message + ";维护占用释放失败:" + releaseErr.Error()
|
||||||
|
_ = store.AdvanceLibraryJob(ctx, jobID, governance.LibraryJobFailed, "capture", note,
|
||||||
|
"维护记录与实际入口不一致:先人工核对再重试", baselineID, "", now)
|
||||||
|
return errors.New(note)
|
||||||
|
}
|
||||||
|
_ = store.AdvanceLibraryJob(ctx, jobID, governance.LibraryJobFailed, "capture", message,
|
||||||
|
"未对目标执行任何写入;可直接重试建立基线", baselineID, "", now)
|
||||||
|
return errors.New(message)
|
||||||
|
}
|
||||||
|
|
||||||
|
// runLibraryBaseline 给"非空但没有权威版本"的目标建立基线。
|
||||||
|
//
|
||||||
|
// 真实链路(全部在同一个维护窗口内):关入口/停写入者/证明排空 → 把目标整库捕获成
|
||||||
|
// 归档快照 → 用同一套 Verify 证明目标与这份快照逐类一致 → 读目标索引映射 →
|
||||||
|
// 登记 production_effective → 关窗口(恢复服务、开入口)→ 释放两把占用 → 作业成功。
|
||||||
|
//
|
||||||
|
// 崩溃语义:
|
||||||
|
// - 捕获后、登记前被杀:作业行里已有快照 ID,重入时**不重新捕获**,直接在窗口内
|
||||||
|
// 复核这份快照再登记;
|
||||||
|
// - 登记后、收尾前被杀:重入识别出"本作业+同一快照"已生效,只做收尾
|
||||||
|
// (恢复服务/开入口/释放占用),不重复捕获、不重写版本;别人的版本仍然拒绝。
|
||||||
|
func runLibraryBaseline(ctx context.Context, store *postgresstore.Store, config appconfig.Config, job jobs.Job) error {
|
||||||
|
var payload libraryBaselinePayload
|
||||||
|
if err := json.Unmarshal(job.Payload, &payload); err != nil || payload.LibraryJobID == "" ||
|
||||||
|
(payload.Scope != "internal" && payload.Scope != "external") {
|
||||||
|
return errors.New("invalid library baseline payload")
|
||||||
|
}
|
||||||
|
now := time.Now().UTC()
|
||||||
|
recorder := libraryStageRecorder{store: store, jobID: payload.LibraryJobID, scope: payload.Scope}
|
||||||
|
fail := func(message, stage string) error {
|
||||||
|
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID, message, stage)
|
||||||
|
}
|
||||||
|
existing, jobErr := store.LibraryJob(ctx, payload.LibraryJobID)
|
||||||
|
actor := ""
|
||||||
|
persistedSnapshot := ""
|
||||||
|
if jobErr == nil {
|
||||||
|
actor = existing.Actor
|
||||||
|
persistedSnapshot = existing.SnapshotID
|
||||||
|
}
|
||||||
|
_ = store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobRunning, "capture", "", "", "", "", now)
|
||||||
|
|
||||||
|
// 版本判定:别人的生效版本一律拒绝;本作业已登记过的快照只收尾。
|
||||||
|
versions, versionErr := store.LibraryVersions(ctx, payload.Scope, 50)
|
||||||
|
if versionErr != nil {
|
||||||
|
return fail("读不到版本记录,无法确认是否需要建立基线;未对目标执行任何操作", "capture")
|
||||||
|
}
|
||||||
|
state := classifyBaselineVersions(versions, payload.LibraryJobID, persistedSnapshot)
|
||||||
|
if state.Foreign != nil {
|
||||||
|
return fail("该目标已有生效版本 "+shortSnapshotID(state.Foreign.SnapshotID)+
|
||||||
|
"(作业 "+state.Foreign.SourceJobID+"),不需要再建基线;换版本请走同步或恢复", "capture")
|
||||||
|
}
|
||||||
|
finalizeOnly := state.Own
|
||||||
|
if finalizeOnly {
|
||||||
|
recorder.Stage(ctx, "finalize", "running", "续作:本作业已登记同一份基线快照,只做收尾(不重新捕获)")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !finalizeOnly && !config.ProductionReleaseRequestsEnabled {
|
||||||
|
return fail("生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false):建立基线要动目标运行状态,闸口关闭时不执行", "capture")
|
||||||
|
}
|
||||||
|
target, backups, wireErr := buildLibraryBaselineStack(ctx, config, payload.Scope)
|
||||||
|
if wireErr != nil {
|
||||||
|
return fail(wireErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
defer target.Close()
|
||||||
|
|
||||||
|
manager, managerErr := openLibraryMaintenance(config)
|
||||||
|
if managerErr != nil {
|
||||||
|
return fail("目标维护未接线,拒绝在挡不住入口/后台写入的情况下捕获目标:"+managerErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
defer manager.Close()
|
||||||
|
if topoErr := manager.VerifyTopology(ctx); topoErr != nil {
|
||||||
|
return fail("目标拓扑不满足唯一入口约束,拒绝开维护窗口:"+topoErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
originalState, stateErr := manager.OriginalState(ctx)
|
||||||
|
if stateErr != nil {
|
||||||
|
return fail("读不到目标写入者的原运行状态,不能开维护窗口:"+stateErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
identity := manager.Target()
|
||||||
|
// 实例级占用:internal/external 动的是同一批容器,必须按实例互斥。
|
||||||
|
// 同一作业重入时 store 保留第一次记下的原运行状态(否则会把"已停"当原状)。
|
||||||
|
if _, instanceErr := store.AcquireInstanceMaintenance(ctx, postgresstore.LibraryInstanceMaintenance{
|
||||||
|
Instance: identity.Instance, OwnerJobID: payload.LibraryJobID, OwnerScope: payload.Scope,
|
||||||
|
TargetFingerprint: identity.Fingerprint, Level: librarymaintenance.LevelInstance,
|
||||||
|
Services: identity.Writers, IngressState: "unknown", State: "acquired",
|
||||||
|
OriginalServiceState: originalState,
|
||||||
|
Note: "实例级维护:为建立权威基线捕获目标整库,捕获并校验后按原状态恢复",
|
||||||
|
}, now); instanceErr != nil {
|
||||||
|
return fail("目标实例正在维护中(跨对内/对外互斥):"+instanceErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
if _, acquireErr := store.AcquireLibraryMaintenance(ctx, postgresstore.LibraryMaintenance{
|
||||||
|
Scope: payload.Scope, OwnerJobID: payload.LibraryJobID,
|
||||||
|
TargetIdentity: identity.Fingerprint,
|
||||||
|
Level: librarymaintenance.LevelInstance,
|
||||||
|
TargetFingerprint: identity.Fingerprint,
|
||||||
|
Services: identity.Writers,
|
||||||
|
IngressState: "unknown",
|
||||||
|
State: "acquired",
|
||||||
|
OriginalServiceState: originalState,
|
||||||
|
Note: "实例级维护:为建立权威基线捕获目标整库,捕获并校验后按原状态恢复",
|
||||||
|
}, now); acquireErr != nil {
|
||||||
|
_ = store.UpdateInstanceMaintenanceState(ctx, identity.Instance, payload.LibraryJobID, "needs_restore",
|
||||||
|
"scope 占用申请失败,实例状态未确认:"+acquireErr.Error(), now)
|
||||||
|
return fail("目标正在维护中:"+acquireErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
|
||||||
|
coordinator, control, openErr := manager.Open(config.LibraryTargetJournalRoot)
|
||||||
|
if openErr != nil {
|
||||||
|
_ = releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
|
||||||
|
"未进入维护窗口(协调器不可用),目标未被停用", time.Now().UTC())
|
||||||
|
return fail("维护协调器不可用:"+openErr.Error(), "capture")
|
||||||
|
}
|
||||||
|
defer control.Close()
|
||||||
|
defer coordinator.Close()
|
||||||
|
|
||||||
|
var (
|
||||||
|
baselineID = persistedSnapshot
|
||||||
|
counts map[string]int64
|
||||||
|
recorded = finalizeOnly
|
||||||
|
)
|
||||||
|
work := func(ctx context.Context, window *libraryfreeze.Window) (string, error) {
|
||||||
|
if finalizeOnly {
|
||||||
|
// 版本已在窗口内校验并登记过:不再捕获、不再重写,收尾由下方统一处理。
|
||||||
|
return baselineID, nil
|
||||||
|
}
|
||||||
|
if baselineID == "" {
|
||||||
|
current, currentErr := target.Current(ctx)
|
||||||
|
if currentErr != nil {
|
||||||
|
return "", errors.New("读取目标当前整库状态失败:" + currentErr.Error())
|
||||||
|
}
|
||||||
|
counts = map[string]int64{}
|
||||||
|
for table, count := range current.Counts {
|
||||||
|
counts[table] = count
|
||||||
|
}
|
||||||
|
// 预备份存储在导出后用同一份指纹复核"导出期间内容没变",
|
||||||
|
// 所以这里把窗口内读到的 state 交进去,而不是事后再读一次。
|
||||||
|
backup, backupErr := backups.Prepare(ctx, "", current)
|
||||||
|
if backupErr != nil {
|
||||||
|
return "", errors.New("捕获目标整库失败:" + backupErr.Error())
|
||||||
|
}
|
||||||
|
baselineID = backup.SnapshotID
|
||||||
|
// 先把"这次窗口的基线快照"落盘:进程在这里被杀时,恢复器知道该回到哪一份,
|
||||||
|
// 重入也据此跳过重新捕获。
|
||||||
|
if markErr := window.SetBackupSnapshot(ctx, baselineID); markErr != nil {
|
||||||
|
return "", errors.New("记录基线快照身份失败:" + markErr.Error())
|
||||||
|
}
|
||||||
|
if persistErr := store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobRunning, "",
|
||||||
|
"", "", baselineID, "", time.Now().UTC()); persistErr != nil {
|
||||||
|
return "", errors.New("持久化基线快照引用失败:" + persistErr.Error())
|
||||||
|
}
|
||||||
|
recorder.Stage(ctx, "capture", "succeeded", "已封存目标整库快照 "+baselineID)
|
||||||
|
} else {
|
||||||
|
recorder.Stage(ctx, "capture", "skipped", "复用作业已持久化的基线快照 "+baselineID)
|
||||||
|
}
|
||||||
|
if baselineID == "" {
|
||||||
|
return "", errors.New("没有可校验的基线快照")
|
||||||
|
}
|
||||||
|
recorder.Stage(ctx, "verify", "running", "逐表/索引/资源比对目标与快照 "+shortSnapshotID(baselineID))
|
||||||
|
if verifyErr := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: baselineID}); verifyErr != nil {
|
||||||
|
return "", errors.New("目标内容与基线快照不一致,拒绝登记基线:" + verifyErr.Error())
|
||||||
|
}
|
||||||
|
recorder.Stage(ctx, "verify", "succeeded", "行身份/逐行内容/索引文档/资源摘要与快照一致")
|
||||||
|
// 索引映射是基线证据的一部分:读不到就不登记,绝不用空摘要冒充。
|
||||||
|
mapping, mapErr := target.Search().MappingSHA256(ctx)
|
||||||
|
if mapErr != nil {
|
||||||
|
return "", errors.New("读不到目标索引映射,拒绝用空摘要登记基线:" + mapErr.Error())
|
||||||
|
}
|
||||||
|
if counts == nil {
|
||||||
|
if current, currentErr := target.Current(ctx); currentErr == nil {
|
||||||
|
counts = current.Counts
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 版本登记放在窗口内:捕获、校验、登记三件事要么一起成立,要么目标原样恢复。
|
||||||
|
recorder.Stage(ctx, "finalize", "running", "登记目标权威基线")
|
||||||
|
if recordErr := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
|
||||||
|
ID: governance.LibraryVersionID(governance.LibraryVersionProductionEffective, payload.Scope, baselineID),
|
||||||
|
Scope: payload.Scope, Kind: governance.LibraryVersionProductionEffective,
|
||||||
|
SnapshotID: baselineID, SourceJobID: payload.LibraryJobID, CreatedBy: actor,
|
||||||
|
Evidence: map[string]any{
|
||||||
|
"action": "target_baseline_init",
|
||||||
|
"origin": "target_baseline",
|
||||||
|
"baseline_snapshot": baselineID,
|
||||||
|
"index_mapping_sha256": mapping,
|
||||||
|
"scope_content_counts": counts,
|
||||||
|
"verification": "目标与快照逐表行身份/逐行内容/索引文档/资源摘要一致",
|
||||||
|
"note": "基线是目标在维护窗口内的真实整库捕获,不是手写版本;发布仍需独立 sync 授权",
|
||||||
|
},
|
||||||
|
CreatedAt: time.Now().UTC(),
|
||||||
|
}); recordErr != nil {
|
||||||
|
return "", errors.New("登记基线版本失败:" + recordErr.Error())
|
||||||
|
}
|
||||||
|
recorded = true
|
||||||
|
recorder.Stage(ctx, "finalize", "succeeded", "已登记目标权威基线 "+shortSnapshotID(baselineID))
|
||||||
|
return baselineID, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
window, windowErr := coordinator.RunWindow(ctx, payload.LibraryJobID,
|
||||||
|
func(ctx context.Context) error { return manager.VerifyRunning(ctx) }, work,
|
||||||
|
// 本流程对目标内容只读:失败时按原状态重新开放永远安全。
|
||||||
|
func(error) bool { return true })
|
||||||
|
|
||||||
|
if settleErr := settleBaselineWindow(ctx, store, manager, payload.LibraryJobID, payload.Scope,
|
||||||
|
identity.Instance, baselineID, recorded, window.State, windowErr); settleErr != nil {
|
||||||
|
return settleErr
|
||||||
|
}
|
||||||
|
log.Printf("library baseline succeeded job=%s scope=%s snapshot=%s actor=%s finalize_only=%v",
|
||||||
|
payload.LibraryJobID, payload.Scope, baselineID, actor, finalizeOnly)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// shortSnapshotID 只用于人看的日志/阶段说明:完整摘要留在版本记录与接口里。
|
||||||
|
func shortSnapshotID(id string) string {
|
||||||
|
if len(id) <= 12 {
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
return id[:12]
|
||||||
|
}
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 崩溃续作的版本判定:本作业+同一快照 = 只收尾;别人的生效版本 = 拒绝;都没有 = 正常建基线。
|
||||||
|
func TestClassifyBaselineVersions(t *testing.T) {
|
||||||
|
const jobA, jobB, snap = "libbase-internal-1", "libbase-internal-2", "snap-abc"
|
||||||
|
effective := func(sourceJob, snapshot string, superseded bool) governance.LibraryVersion {
|
||||||
|
item := governance.LibraryVersion{
|
||||||
|
Scope: "internal", Kind: governance.LibraryVersionProductionEffective,
|
||||||
|
SnapshotID: snapshot, SourceJobID: sourceJob, CreatedAt: time.Now().UTC(),
|
||||||
|
}
|
||||||
|
if superseded {
|
||||||
|
at := time.Now().UTC()
|
||||||
|
item.SupersededAt = &at
|
||||||
|
}
|
||||||
|
return item
|
||||||
|
}
|
||||||
|
|
||||||
|
if state := classifyBaselineVersions(nil, jobA, snap); state.Own || state.Foreign != nil {
|
||||||
|
t.Fatalf("no versions must mean neither own nor foreign: %+v", state)
|
||||||
|
}
|
||||||
|
if state := classifyBaselineVersions([]governance.LibraryVersion{effective(jobA, snap, false)}, jobA, snap); !state.Own {
|
||||||
|
t.Fatalf("same job + same snapshot must be recognized as own: %+v", state)
|
||||||
|
}
|
||||||
|
// 快照不同就不是"我登记的那份",不能当成可收尾。
|
||||||
|
if state := classifyBaselineVersions([]governance.LibraryVersion{effective(jobA, "snap-other", false)}, jobA, snap); state.Own || state.Foreign == nil {
|
||||||
|
t.Fatalf("different snapshot of same job must not be own: %+v", state)
|
||||||
|
}
|
||||||
|
// 别人的生效版本:即使快照相同也必须拒绝。
|
||||||
|
foreign := classifyBaselineVersions([]governance.LibraryVersion{effective(jobB, snap, false)}, jobA, snap)
|
||||||
|
if foreign.Own || foreign.Foreign == nil || foreign.Foreign.SourceJobID != jobB {
|
||||||
|
t.Fatalf("foreign effective version must be reported: %+v", foreign)
|
||||||
|
}
|
||||||
|
// 已被顶替的旧版本不算生效版本。
|
||||||
|
if state := classifyBaselineVersions([]governance.LibraryVersion{effective(jobB, snap, true)}, jobA, snap); state.Own || state.Foreign != nil {
|
||||||
|
t.Fatalf("superseded version must not block a baseline: %+v", state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 版本已登记 + 健康:走成功边界,释放两把占用并把作业标成功。
|
||||||
|
func TestSettleBaselineWindowFinishesWhenRecorded(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
if err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
|
||||||
|
"snap-abc", true, "succeeded", nil); err != nil {
|
||||||
|
t.Fatalf("settle: %v", err)
|
||||||
|
}
|
||||||
|
if !containsCall(store.calls, "advance_succeeded") {
|
||||||
|
t.Fatalf("job must be marked succeeded: %v", store.calls)
|
||||||
|
}
|
||||||
|
if containsCall(store.states, "needs_restore") {
|
||||||
|
t.Fatalf("success path must not hold the target in maintenance: %v", store.states)
|
||||||
|
}
|
||||||
|
if !containsCall(store.calls, "scope_release") || !containsCall(store.calls, "instance_release") {
|
||||||
|
t.Fatalf("both locks must be released: %v", store.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 版本已登记但目标服务/入口没恢复:不能报成功,必须保持维护并标 needs_restore。
|
||||||
|
func TestSettleBaselineWindowHoldsWhenHealthFails(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
err := settleBaselineWindow(context.Background(), store, fakeHealth{err: errors.New("kgseal-ingress 未就绪")},
|
||||||
|
"job-1", "internal", "kgseal-app", "snap-abc", true, "succeeded", nil)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("unhealthy target must not report success")
|
||||||
|
}
|
||||||
|
if containsCall(store.calls, "advance_succeeded") {
|
||||||
|
t.Fatalf("must not mark succeeded while target is unhealthy: %v", store.calls)
|
||||||
|
}
|
||||||
|
if countCall(store.states, "needs_restore") != 2 {
|
||||||
|
t.Fatalf("must hold both records with needs_restore: %v", store.states)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 没有登记、窗口仍是 held:只能保持维护(服务确实没回来),绝不能释放占用。
|
||||||
|
func TestSettleBaselineWindowKeepsHeldWindowInMaintenance(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
|
||||||
|
"snap-abc", false, "held", errors.New("停写入者后后台仍有任务"))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "保持维护") {
|
||||||
|
t.Fatalf("err=%v", err)
|
||||||
|
}
|
||||||
|
if containsCall(store.calls, "scope_release") || containsCall(store.calls, "instance_release") {
|
||||||
|
t.Fatalf("held window must not release locks: %v", store.calls)
|
||||||
|
}
|
||||||
|
if countCall(store.states, "needs_restore") != 2 {
|
||||||
|
t.Fatalf("held window must be recorded as needs_restore: %v", store.states)
|
||||||
|
}
|
||||||
|
if !containsCall(store.calls, "advance_failed") {
|
||||||
|
t.Fatalf("job must fail: %v", store.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 没有登记、但窗口已按原状态开放:状态必须跟着变成"已释放",
|
||||||
|
// 不能留下"显示维护中、实际入口已开"。
|
||||||
|
func TestSettleBaselineWindowReleasesWhenWindowRestored(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
|
||||||
|
"snap-abc", false, "failed", errors.New("捕获目标整库失败"))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "未建立基线") {
|
||||||
|
t.Fatalf("err=%v", err)
|
||||||
|
}
|
||||||
|
if !containsCall(store.calls, "scope_release") || !containsCall(store.calls, "instance_release") {
|
||||||
|
t.Fatalf("restored window must release both locks: %v", store.calls)
|
||||||
|
}
|
||||||
|
if containsCall(store.states, "needs_restore") {
|
||||||
|
t.Fatalf("released window must not be marked as needs_restore: %v", store.states)
|
||||||
|
}
|
||||||
|
if !containsCall(store.calls, "advance_failed") {
|
||||||
|
t.Fatalf("job must fail: %v", store.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 释放失败时不能在作业上写"无需恢复":那会让界面与实际占用分叉。
|
||||||
|
func TestSettleBaselineWindowReportsReleaseFailure(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{failOn: "scope_release"}
|
||||||
|
err := settleBaselineWindow(context.Background(), store, fakeHealth{}, "job-1", "internal", "kgseal-app",
|
||||||
|
"snap-abc", false, "failed", nil)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "维护占用释放失败") {
|
||||||
|
t.Fatalf("err=%v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func countCall(calls []string, want string) int {
|
||||||
|
count := 0
|
||||||
|
for _, call := range calls {
|
||||||
|
if call == want {
|
||||||
|
count++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsCall(calls []string, want string) bool {
|
||||||
|
for _, call := range calls {
|
||||||
|
if call == want {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/appconfig"
|
||||||
|
"knowledge-governance/internal/libraryagent"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 封存的身份契约:期望随**同一个 capture 请求**送进 agent,由接收方在操作互斥内、
|
||||||
|
// 任何源操作之前比对;worker 事后按回显的指纹再复核一次。
|
||||||
|
//
|
||||||
|
// 为什么不能在 worker 侧先 Health 再 Capture:那是两次拨号,中间 socket 可能被换成
|
||||||
|
// 另一个 agent,错的执行器已经把写入者停了才被拒登记——副作用已经发生。这里把期望
|
||||||
|
// 放进请求本身,换过的 agent 会先比对身份、比对失败就一次 run 都不调。
|
||||||
|
//
|
||||||
|
// 期望字段只来自治理侧配置(LIBRARY_SOURCE_*),绝不从执行器自报值生成。
|
||||||
|
|
||||||
|
// captureExpectation 把治理侧配置翻译成期望;缺身份就是缺配置,调用方必须失败。
|
||||||
|
func captureExpectation(config appconfig.Config) (libraryagent.Expectation, error) {
|
||||||
|
expectation := libraryagent.NewExpectation(config.SourceProject(), config.LibrarySourceApp, "")
|
||||||
|
if expectation.Project == "" {
|
||||||
|
return libraryagent.Expectation{}, errors.New("未配置封存来源实例身份(LIBRARY_SOURCE_PROJECT),拒绝在不核对执行器的情况下停写入者")
|
||||||
|
}
|
||||||
|
return expectation, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// captureExecutorPostcheck 是事后证据防线:结果里回显的期望指纹必须等于我们发出的,
|
||||||
|
// 自报身份也必须与期望一致。它挡不住副作用,所以不能替代请求内的核对。
|
||||||
|
func captureExecutorPostcheck(payload json.RawMessage, expectation libraryagent.Expectation) error {
|
||||||
|
if len(payload) == 0 {
|
||||||
|
return errors.New("封存结果里没有执行器身份,无法证明这次封存来自期望实例")
|
||||||
|
}
|
||||||
|
var document struct {
|
||||||
|
Identity struct {
|
||||||
|
Project string `json:"project"`
|
||||||
|
SourceApp string `json:"source_app"`
|
||||||
|
ExpectationFingerprint string `json:"expectation_fingerprint"`
|
||||||
|
} `json:"agent_identity"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(payload, &document); err != nil {
|
||||||
|
return errors.New("封存结果里的执行器身份无法解析")
|
||||||
|
}
|
||||||
|
identity := document.Identity
|
||||||
|
if identity.Project == "" {
|
||||||
|
return errors.New("封存结果里没有执行器身份,无法证明这次封存来自期望实例")
|
||||||
|
}
|
||||||
|
if identity.ExpectationFingerprint != expectation.Fingerprint {
|
||||||
|
return fmt.Errorf("封存结果回显的期望指纹不一致(agent=%q,期望=%q),该快照不能当作本实例的测试封存",
|
||||||
|
identity.ExpectationFingerprint, expectation.Fingerprint)
|
||||||
|
}
|
||||||
|
if identity.Project != expectation.Project {
|
||||||
|
return fmt.Errorf("封存结果来自另一个实例(agent=%q,期望=%q),该快照不能当作本实例的测试封存",
|
||||||
|
identity.Project, expectation.Project)
|
||||||
|
}
|
||||||
|
if expectation.SourceApp != "" && identity.SourceApp != expectation.SourceApp {
|
||||||
|
return fmt.Errorf("封存结果来自另一个源容器(agent=%q,期望=%q),该快照不能当作本实例的测试封存",
|
||||||
|
identity.SourceApp, expectation.SourceApp)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// expectationFingerprint 用于日志/证据行的稳定标识,不含路径与凭据。
|
||||||
|
func expectationFingerprint(expectation libraryagent.Expectation) string {
|
||||||
|
return strings.TrimSpace(expectation.Fingerprint)
|
||||||
|
}
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/appconfig"
|
||||||
|
"knowledge-governance/internal/libraryagent"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 期望只从治理侧配置生成;缺 project 就是缺配置,必须在停写入者之前失败。
|
||||||
|
func TestCaptureExpectationComesFromGovernanceConfigAndFailsClosed(t *testing.T) {
|
||||||
|
expect, err := captureExpectation(appconfig.Config{LibrarySourceInstance: "kgmaint-weknora", LibrarySourceApp: "kgmaint-app"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if expect.Project != "kgmaint-weknora" || expect.SourceApp != "kgmaint-app" {
|
||||||
|
t.Fatalf("expectation = %+v", expect)
|
||||||
|
}
|
||||||
|
if !expect.Complete() {
|
||||||
|
t.Fatalf("expectation should be complete: %+v", expect)
|
||||||
|
}
|
||||||
|
if _, err = captureExpectation(appconfig.Config{}); err == nil ||
|
||||||
|
!strings.Contains(err.Error(), "LIBRARY_SOURCE_PROJECT") {
|
||||||
|
t.Fatalf("empty identity accepted: %v", err)
|
||||||
|
}
|
||||||
|
// 期望不取自任何执行器应答:显式给 project 就按它绑定,配置里没有别的来源。
|
||||||
|
explicit, err := captureExpectation(appconfig.Config{LibrarySourceProject: "kgmaint-weknora"})
|
||||||
|
if err != nil || explicit.Project != "kgmaint-weknora" || explicit.SourceApp != "" {
|
||||||
|
t.Fatalf("explicit = %+v err=%v", explicit, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 事后证据防线:回显的期望指纹、project、source_app 任何一项不符都不能登记。
|
||||||
|
func TestCaptureExecutorPostcheckUsesTheEchoedExpectationFingerprint(t *testing.T) {
|
||||||
|
expect := libraryagent.NewExpectation("kgmaint-weknora", "kgmaint-app", "")
|
||||||
|
good := payloadOf(t, "kgmaint-weknora", "kgmaint-app", expect.Fingerprint)
|
||||||
|
if err := captureExecutorPostcheck(good, expect); err != nil {
|
||||||
|
t.Fatalf("matching identity rejected: %v", err)
|
||||||
|
}
|
||||||
|
swapped := payloadOf(t, "weknora071full", "WKNFull-app", expect.Fingerprint)
|
||||||
|
if err := captureExecutorPostcheck(swapped, expect); err == nil || !strings.Contains(err.Error(), "另一个实例") {
|
||||||
|
t.Fatalf("wrong project accepted: %v", err)
|
||||||
|
}
|
||||||
|
regenerated := payloadOf(t, "kgmaint-weknora", "kgmaint-app",
|
||||||
|
libraryagent.NewExpectation("kgmaint-weknora", "kgmaint-app", "/other/archive").Fingerprint)
|
||||||
|
if err := captureExecutorPostcheck(regenerated, expect); err == nil || !strings.Contains(err.Error(), "期望指纹不一致") {
|
||||||
|
t.Fatalf("regenerated expectation accepted: %v", err)
|
||||||
|
}
|
||||||
|
if err := captureExecutorPostcheck(json.RawMessage(`{"snapshot_id":"abc"}`), expect); err == nil ||
|
||||||
|
!strings.Contains(err.Error(), "没有执行器身份") {
|
||||||
|
t.Fatalf("missing identity accepted: %v", err)
|
||||||
|
}
|
||||||
|
if err := captureExecutorPostcheck(payloadOf(t, "kgmaint-weknora", "kgseal-app", expect.Fingerprint), expect); err == nil ||
|
||||||
|
!strings.Contains(err.Error(), "另一个源容器") {
|
||||||
|
t.Fatalf("wrong source app accepted: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func payloadOf(t *testing.T, project, sourceApp, fingerprint string) json.RawMessage {
|
||||||
|
t.Helper()
|
||||||
|
encoded, err := json.Marshal(map[string]any{
|
||||||
|
"snapshot_id": "abc",
|
||||||
|
"agent_identity": map[string]any{
|
||||||
|
"project": project, "source_app": sourceApp, "expectation_fingerprint": fingerprint,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return encoded
|
||||||
|
}
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/appconfig"
|
||||||
|
"knowledge-governance/internal/libraryexecutor"
|
||||||
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 为什么由 worker 观测、页面读库:agent socket 只挂载在 worker 容器里,API 那个
|
||||||
|
// LIBRARY_AGENT_SOCKET 即使配了也 dial 不到。谁真能执行就由谁写下观测,
|
||||||
|
// 否则页面上的"可发起"只是配置字符串的回声。
|
||||||
|
const libraryExecutorProbeInterval = 30 * time.Second
|
||||||
|
|
||||||
|
// libraryExecutorProbeTimeout 是一次 health 往返的上限:它只是同 uid unix socket
|
||||||
|
// 上的一次 JSON 往返,慢到秒级就说明执行器有问题,不该拖住心跳。
|
||||||
|
const libraryExecutorProbeTimeout = 3 * time.Second
|
||||||
|
|
||||||
|
func recordLibraryExecutorStatus(ctx context.Context, store *postgresstore.Store, config appconfig.Config) {
|
||||||
|
// 归属用**封存源**实例:agent 冻结的是源库写入者,和同步目标(kgseal)不是一套。
|
||||||
|
instance := config.SourceInstance()
|
||||||
|
if instance == "" {
|
||||||
|
// 没有实例身份,观测就无处归属:写进去只会让页面拿别的实例的状态。
|
||||||
|
log.Print("library executor probe skipped: LIBRARY_SOURCE_INSTANCE is not configured")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
observation := libraryexecutor.Probe(ctx, instance, config.LibraryAgentSocket, libraryExecutorProbeTimeout, nil)
|
||||||
|
if err := store.UpsertLibraryExecutorStatus(ctx, observation, config.WorkerID, time.Now().UTC()); err != nil {
|
||||||
|
log.Printf("library executor status not recorded: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// runLibraryExecutorProbe 先观测一次再进循环:页面在 worker 刚起来时就该有真实状态。
|
||||||
|
func runLibraryExecutorProbe(ctx context.Context, store *postgresstore.Store, config appconfig.Config, interval time.Duration) {
|
||||||
|
if interval <= 0 {
|
||||||
|
interval = libraryExecutorProbeInterval
|
||||||
|
}
|
||||||
|
recordLibraryExecutorStatus(ctx, store, config)
|
||||||
|
ticker := time.NewTicker(interval)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
recordLibraryExecutorStatus(ctx, store, config)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
// libraryMaintenanceStore 是"成功收尾"用到的存储能力。抽成接口是为了能在测试里
|
||||||
|
// 让某一步失败,验证"没全部完成就不许暴露成功"。
|
||||||
|
type libraryMaintenanceStore interface {
|
||||||
|
SetInstanceMaintenanceIngress(ctx context.Context, instance, ownerJobID, state, note string, now time.Time) error
|
||||||
|
SetLibraryMaintenanceIngress(ctx context.Context, scope, ownerJobID, state, note string, now time.Time) error
|
||||||
|
UpdateInstanceMaintenanceState(ctx context.Context, instance, ownerJobID, state, note string, now time.Time) error
|
||||||
|
UpdateLibraryMaintenanceState(ctx context.Context, scope, ownerJobID, state, note string, now time.Time) error
|
||||||
|
ReleaseInstanceMaintenance(ctx context.Context, instance, ownerJobID string, now time.Time) error
|
||||||
|
ReleaseLibraryMaintenance(ctx context.Context, scope, ownerJobID string, now time.Time) error
|
||||||
|
AdvanceLibraryJob(ctx context.Context, jobID string, state governance.LibraryJobState, stage, message, recovery, snapshotID, targetSnapshotID string, now time.Time) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// releaseWindowLocks 是唯一释放路径。
|
||||||
|
//
|
||||||
|
// 顺序刻意是"scope 先、instance 后":instance 行是跨 scope 的互斥来源,最后放它
|
||||||
|
// 才能保证任何新作业在全部记录落定之前都进不来。每一步的错误都必须向上暴露——
|
||||||
|
// 只丢日志会让"锁其实没放"这件事变成页面上的成功。
|
||||||
|
func releaseWindowLocks(ctx context.Context, store libraryMaintenanceStore, instance, scope, jobID, note string, now time.Time) error {
|
||||||
|
if err := store.SetLibraryMaintenanceIngress(ctx, scope, jobID, "open", note, now); !releasedOrNil(err) {
|
||||||
|
return fmt.Errorf("记录入口已开放失败(scope):%w", err)
|
||||||
|
}
|
||||||
|
if err := store.SetInstanceMaintenanceIngress(ctx, instance, jobID, "open", note, now); !releasedOrNil(err) {
|
||||||
|
return fmt.Errorf("记录入口已开放失败(instance):%w", err)
|
||||||
|
}
|
||||||
|
if err := store.UpdateLibraryMaintenanceState(ctx, scope, jobID, "verified", note, now); !releasedOrNil(err) {
|
||||||
|
return fmt.Errorf("维护状态写入失败(scope):%w", err)
|
||||||
|
}
|
||||||
|
if err := store.ReleaseLibraryMaintenance(ctx, scope, jobID, now); !releasedOrNil(err) {
|
||||||
|
return fmt.Errorf("维护占用释放失败(scope):%w", err)
|
||||||
|
}
|
||||||
|
if err := store.UpdateInstanceMaintenanceState(ctx, instance, jobID, "verified", note, now); !releasedOrNil(err) {
|
||||||
|
return fmt.Errorf("维护状态写入失败(instance):%w", err)
|
||||||
|
}
|
||||||
|
if err := store.ReleaseInstanceMaintenance(ctx, instance, jobID, now); !releasedOrNil(err) {
|
||||||
|
return fmt.Errorf("实例占用释放失败(instance):%w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// healthChecker 是"目标写入者与入口是否真的恢复健康"的判定口(测试可注入失败)。
|
||||||
|
type healthChecker interface {
|
||||||
|
VerifyRunning(ctx context.Context) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// successBoundary 描述一次"内容已通过验证与版本登记,但还没到可以对外报成功"的作业。
|
||||||
|
type successBoundary struct {
|
||||||
|
Ready bool
|
||||||
|
Backup string
|
||||||
|
Target string
|
||||||
|
FinishedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// finishLibrarySuccess 是唯一的成功收尾入口:只有以下全部成立才暴露 succeeded——
|
||||||
|
// 目标服务健康、入口开放、scope 与 instance 两把维护锁都释放完成、终态写入成功。
|
||||||
|
//
|
||||||
|
// 为什么不能先标成功再释放:恢复/开入口/放锁都可能失败,先写 succeeded 会让外部
|
||||||
|
// (页面、API、重启后的接管逻辑)看到一条"成功了但目标仍在维护"的作业;而
|
||||||
|
// InterruptedLibraryJobs 只接管非终态,终态作业不会再被接管,锁就永远挂着。
|
||||||
|
func finishLibrarySuccess(ctx context.Context, store libraryMaintenanceStore, health healthChecker,
|
||||||
|
instance, scope, jobID string, boundary successBoundary) error {
|
||||||
|
// 释放与终态写入必须能在 ctx 取消后继续:停机不该留下"锁放了但作业还在跑"或
|
||||||
|
// 反过来"作业成功了锁还挂着"的任一半成品。
|
||||||
|
bounded, cancel := context.WithTimeout(context.WithoutCancel(ctx), 60*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
holdForRepair := func(reason string) error {
|
||||||
|
note := "内容已验证,但收尾未完成,目标保持维护:" + reason
|
||||||
|
_ = store.UpdateLibraryMaintenanceState(bounded, scope, jobID, "needs_restore", note, time.Now().UTC())
|
||||||
|
_ = store.UpdateInstanceMaintenanceState(bounded, instance, jobID, "needs_restore", note, time.Now().UTC())
|
||||||
|
_ = store.AdvanceLibraryJob(bounded, jobID, governance.LibraryJobFailed, "finalize", note,
|
||||||
|
"生产内容已写入并校验;维护保持,按预备份或人工核验后再开放", boundary.Backup, boundary.Target, time.Now().UTC())
|
||||||
|
return errors.New(note)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := health.VerifyRunning(bounded); err != nil {
|
||||||
|
return holdForRepair("目标服务或入口未恢复健康:" + err.Error())
|
||||||
|
}
|
||||||
|
if err := releaseWindowLocks(bounded, store, instance, scope, jobID,
|
||||||
|
"写入与校验完成,目标已按原运行状态恢复", time.Now().UTC()); err != nil {
|
||||||
|
return holdForRepair(err.Error())
|
||||||
|
}
|
||||||
|
finishedAt := time.Now().UTC()
|
||||||
|
if err := store.AdvanceLibraryJob(bounded, jobID, governance.LibraryJobSucceeded, "finalize", "", "",
|
||||||
|
boundary.Backup, boundary.Target, finishedAt); err != nil {
|
||||||
|
// 锁已释放但终态没写下去:作业保持非终态,重启后会被接管(接管走
|
||||||
|
// finalize-only,只复核与登记,不重写生产)。
|
||||||
|
return fmt.Errorf("维护已释放但作业终态写入失败(等待接管):%w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// takeoverKind 是接管时"这个作业走到哪一步"的分类。
|
||||||
|
//
|
||||||
|
// 四类必须分开:把"版本查不到"当成"没有收尾证据",就会把只差收尾的作业重新写一遍
|
||||||
|
// 目标;把"曾被顶替"当成正常重放,就会用旧快照覆盖更新的生产版本。
|
||||||
|
type takeoverKind string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// takeoverFresh:没有任何收尾证据,按正常流程(有预备份则复用,写前仍备份)。
|
||||||
|
takeoverFresh takeoverKind = "fresh"
|
||||||
|
// takeoverPendingFinal:内容与版本已落定,只差收尾(只复核与登记,不重写)。
|
||||||
|
takeoverPendingFinal takeoverKind = "pending_final"
|
||||||
|
// takeoverSuperseded:本快照曾经生效、后来被别的快照顶替,不得自动覆盖。
|
||||||
|
takeoverSuperseded takeoverKind = "superseded"
|
||||||
|
// takeoverUnknown:版本状态读不到且没有阶段证据,属"未知",不得写入。
|
||||||
|
takeoverUnknown takeoverKind = "unknown"
|
||||||
|
)
|
||||||
|
|
||||||
|
// classifyTakeover 只按证据分类:
|
||||||
|
// - finalize 阶段成功是独立证据(来自作业自己的阶段行),不因版本查询失败而丢弃;
|
||||||
|
// - 本快照存在 released/被顶替的版本 ⇒ superseded;
|
||||||
|
// - 本快照存在未被顶掉的生效版本 ⇒ pending_final;
|
||||||
|
// - 查询失败且无阶段证据 ⇒ unknown(可重试,不当成没有证据)。
|
||||||
|
func classifyTakeover(job governance.LibraryJob, versions []governance.LibraryVersion, versionsErr error) takeoverKind {
|
||||||
|
finalized := false
|
||||||
|
for _, stage := range job.Stages {
|
||||||
|
if stage.Stage == "finalize" && stage.State == "succeeded" {
|
||||||
|
finalized = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if versionsErr != nil && !finalized {
|
||||||
|
return takeoverUnknown
|
||||||
|
}
|
||||||
|
superseded, effective := false, false
|
||||||
|
for _, version := range versions {
|
||||||
|
if version.Kind != governance.LibraryVersionProductionEffective || version.SnapshotID != job.TargetSnapshotID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if version.SupersededAt != nil {
|
||||||
|
superseded = true
|
||||||
|
} else {
|
||||||
|
effective = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if superseded && !effective {
|
||||||
|
return takeoverSuperseded
|
||||||
|
}
|
||||||
|
if effective || finalized {
|
||||||
|
return takeoverPendingFinal
|
||||||
|
}
|
||||||
|
return takeoverFresh
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ libraryMaintenanceStore = (*postgresstore.Store)(nil)
|
||||||
|
|
||||||
|
// releasedOrNil:这一行已经由本作业释放过,说明该步在上一次尝试里已经完成。
|
||||||
|
// 部分成功后重试幂等靠它——否则"scope 放了、instance 没放"的作业永远重试不下去。
|
||||||
|
func releasedOrNil(err error) bool {
|
||||||
|
return err == nil || errors.Is(err, postgresstore.ErrMaintenanceReleased)
|
||||||
|
}
|
||||||
@@ -0,0 +1,213 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeFinishStore struct {
|
||||||
|
calls []string
|
||||||
|
failOn string
|
||||||
|
alreadyReleased bool
|
||||||
|
advanced []governance.LibraryJobState
|
||||||
|
// states 记录所有维护状态写入(测试要区分 verified 与 needs_restore)。
|
||||||
|
states []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// 模拟"这一行已由同一个作业释放过"(真实 Store 返回 ErrMaintenanceReleased)。
|
||||||
|
func (f *fakeFinishStore) releasedSentinel() error {
|
||||||
|
if f.alreadyReleased {
|
||||||
|
return postgresstore.ErrMaintenanceReleased
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeFinishStore) record(name string) error {
|
||||||
|
f.calls = append(f.calls, name)
|
||||||
|
if f.failOn == name {
|
||||||
|
return errors.New("injected " + name + " failure")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeFinishStore) SetInstanceMaintenanceIngress(context.Context, string, string, string, string, time.Time) error {
|
||||||
|
if err := f.releasedSentinel(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return f.record("instance_ingress")
|
||||||
|
}
|
||||||
|
func (f *fakeFinishStore) SetLibraryMaintenanceIngress(context.Context, string, string, string, string, time.Time) error {
|
||||||
|
if err := f.releasedSentinel(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return f.record("scope_ingress")
|
||||||
|
}
|
||||||
|
func (f *fakeFinishStore) UpdateInstanceMaintenanceState(_ context.Context, _, _, state, _ string, _ time.Time) error {
|
||||||
|
if err := f.releasedSentinel(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
f.states = append(f.states, state)
|
||||||
|
return f.record("instance_state")
|
||||||
|
}
|
||||||
|
func (f *fakeFinishStore) UpdateLibraryMaintenanceState(_ context.Context, _, _, state, _ string, _ time.Time) error {
|
||||||
|
if err := f.releasedSentinel(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
f.states = append(f.states, state)
|
||||||
|
return f.record("scope_state")
|
||||||
|
}
|
||||||
|
func (f *fakeFinishStore) ReleaseInstanceMaintenance(context.Context, string, string, time.Time) error {
|
||||||
|
if err := f.releasedSentinel(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return f.record("instance_release")
|
||||||
|
}
|
||||||
|
func (f *fakeFinishStore) ReleaseLibraryMaintenance(context.Context, string, string, time.Time) error {
|
||||||
|
if err := f.releasedSentinel(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return f.record("scope_release")
|
||||||
|
}
|
||||||
|
func (f *fakeFinishStore) AdvanceLibraryJob(_ context.Context, _ string, state governance.LibraryJobState,
|
||||||
|
_, _, _, _, _ string, _ time.Time) error {
|
||||||
|
f.advanced = append(f.advanced, state)
|
||||||
|
return f.record("advance_" + string(state))
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeHealth struct{ err error }
|
||||||
|
|
||||||
|
func (h fakeHealth) VerifyRunning(context.Context) error { return h.err }
|
||||||
|
|
||||||
|
func boundary() successBoundary {
|
||||||
|
return successBoundary{Ready: true, Backup: "backup-1", Target: "snap-1", FinishedAt: time.Now().UTC()}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 首次观察到成功时,两把维护锁必须已经释放,且终态写入排在最后。
|
||||||
|
func TestFinishLibrarySuccessReleasesEveryLockBeforeSucceeded(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
if err := finishLibrarySuccess(context.Background(), store, fakeHealth{}, "kgseal-weknora", "internal", "job-1", boundary()); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
want := []string{"scope_ingress", "instance_ingress", "scope_state", "scope_release",
|
||||||
|
"instance_state", "instance_release", "advance_succeeded"}
|
||||||
|
if strings.Join(store.calls, ",") != strings.Join(want, ",") {
|
||||||
|
t.Fatalf("call order = %v", store.calls)
|
||||||
|
}
|
||||||
|
if len(store.advanced) != 1 || store.advanced[0] != governance.LibraryJobSucceeded {
|
||||||
|
t.Fatalf("advanced = %v", store.advanced)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 目标服务/入口没恢复健康:不许放锁,不许报成功,作业落成失败并保持维护。
|
||||||
|
func TestFinishLibrarySuccessKeepsMaintenanceWhenTargetIsNotHealthy(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
err := finishLibrarySuccess(context.Background(), store, fakeHealth{err: errors.New("app not healthy")},
|
||||||
|
"kgseal-weknora", "internal", "job-2", boundary())
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "未恢复健康") {
|
||||||
|
t.Fatalf("err = %v", err)
|
||||||
|
}
|
||||||
|
for _, call := range store.calls {
|
||||||
|
if strings.HasSuffix(call, "_release") || call == "advance_succeeded" {
|
||||||
|
t.Fatalf("released or reported success despite unhealthy target: %v", store.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(store.advanced) != 1 || store.advanced[0] != governance.LibraryJobFailed {
|
||||||
|
t.Fatalf("advanced = %v", store.advanced)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 任一步释放失败:不报成功,作业失败,维护状态写 needs_restore(等人工/接管)。
|
||||||
|
func TestFinishLibrarySuccessStopsOnReleaseFailure(t *testing.T) {
|
||||||
|
for _, step := range []string{"scope_release", "instance_state", "instance_release"} {
|
||||||
|
t.Run(step, func(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{failOn: step}
|
||||||
|
err := finishLibrarySuccess(context.Background(), store, fakeHealth{}, "kgseal-weknora", "internal", "job-3", boundary())
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("%s failure was swallowed", step)
|
||||||
|
}
|
||||||
|
for _, call := range store.calls {
|
||||||
|
if call == "advance_succeeded" {
|
||||||
|
t.Fatalf("reported success after %s failed: %v", step, store.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(store.advanced) != 1 || store.advanced[0] != governance.LibraryJobFailed {
|
||||||
|
t.Fatalf("advanced = %v", store.advanced)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 终态写入失败:锁已释放,但不能声称成功;作业保持非终态等接管。
|
||||||
|
func TestFinishLibrarySuccessReportsTerminalWriteFailure(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{failOn: "advance_succeeded"}
|
||||||
|
err := finishLibrarySuccess(context.Background(), store, fakeHealth{}, "kgseal-weknora", "internal", "job-4", boundary())
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "等待接管") {
|
||||||
|
t.Fatalf("err = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 接管分类:阶段证据独立成立;查询失败=未知(不得写入);被顶替=不得覆盖。
|
||||||
|
func TestClassifyTakeoverSeparatesUnknownPendingSupersededAndFresh(t *testing.T) {
|
||||||
|
job := governance.LibraryJob{Scope: "internal", Kind: governance.LibraryJobSync, TargetSnapshotID: "snap-1"}
|
||||||
|
finalized := job
|
||||||
|
finalized.Stages = []governance.LibraryStage{{Stage: "finalize", State: "succeeded"}}
|
||||||
|
effective := []governance.LibraryVersion{{Kind: governance.LibraryVersionProductionEffective, SnapshotID: "snap-1"}}
|
||||||
|
supersededVersion := []governance.LibraryVersion{{Kind: governance.LibraryVersionProductionEffective,
|
||||||
|
SnapshotID: "snap-1", SupersededAt: ptrTime(time.Now())}}
|
||||||
|
unrelated := []governance.LibraryVersion{{Kind: governance.LibraryVersionProductionEffective, SnapshotID: "snap-9"}}
|
||||||
|
queryErr := errors.New("version storage temporarily unavailable")
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
job governance.LibraryJob
|
||||||
|
versions []governance.LibraryVersion
|
||||||
|
err error
|
||||||
|
want takeoverKind
|
||||||
|
}{
|
||||||
|
// 关键回归:查询失败 + 没有阶段证据 ⇒ 未知,不能当成"没有收尾证据"继续写。
|
||||||
|
{"query fails without stage evidence", job, nil, queryErr, takeoverUnknown},
|
||||||
|
// 阶段证据先独立成立:查询失败不影响"只差收尾"的判定。
|
||||||
|
{"query fails but finalize stage succeeded", finalized, nil, queryErr, takeoverPendingFinal},
|
||||||
|
{"effective version for this snapshot", job, effective, nil, takeoverPendingFinal},
|
||||||
|
{"finalize stage succeeded", finalized, nil, nil, takeoverPendingFinal},
|
||||||
|
{"this snapshot was superseded", job, supersededVersion, nil, takeoverSuperseded},
|
||||||
|
{"no evidence at all", job, nil, nil, takeoverFresh},
|
||||||
|
{"another snapshot effective but this one has no row", job, unrelated, nil, takeoverFresh},
|
||||||
|
{"empty target snapshot", governance.LibraryJob{Scope: "internal"}, nil, nil, takeoverFresh},
|
||||||
|
}
|
||||||
|
for _, test := range cases {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
if got := classifyTakeover(test.job, test.versions, test.err); got != test.want {
|
||||||
|
t.Fatalf("classify = %q, want %q", got, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 部分释放后重试必须幂等:已由本作业释放的行不能再算失败。
|
||||||
|
func TestReleaseWindowLocksToleratesAlreadyReleasedRows(t *testing.T) {
|
||||||
|
store := &fakeFinishStore{}
|
||||||
|
if err := releaseWindowLocks(context.Background(), store, "kgseal-weknora", "internal", "job-1",
|
||||||
|
"verified", time.Now().UTC()); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
store = &fakeFinishStore{alreadyReleased: true}
|
||||||
|
if err := releaseWindowLocks(context.Background(), store, "kgseal-weknora", "internal", "job-1",
|
||||||
|
"verified", time.Now().UTC()); err != nil {
|
||||||
|
t.Fatalf("retry after a partial release must be idempotent: %v", err)
|
||||||
|
}
|
||||||
|
// 真正"不是本作业持有"的错误仍然要暴露。
|
||||||
|
store = &fakeFinishStore{failOn: "instance_release"}
|
||||||
|
if err := releaseWindowLocks(context.Background(), store, "kgseal-weknora", "internal", "job-1",
|
||||||
|
"verified", time.Now().UTC()); err == nil {
|
||||||
|
t.Fatal("a foreign owner error was swallowed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func ptrTime(value time.Time) *time.Time { return &value }
|
||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"knowledge-governance/internal/librarymaintenance"
|
"knowledge-governance/internal/librarymaintenance"
|
||||||
"knowledge-governance/internal/libraryrelease"
|
"knowledge-governance/internal/libraryrelease"
|
||||||
"knowledge-governance/internal/librarysnapshot"
|
"knowledge-governance/internal/librarysnapshot"
|
||||||
|
"knowledge-governance/internal/pgdsn"
|
||||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -65,32 +66,8 @@ type libraryReleasePayload struct {
|
|||||||
FinalizeOnly bool `json:"finalize_only,omitempty"`
|
FinalizeOnly bool `json:"finalize_only,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// openLibraryMaintenance 按部署注入的配置装配实例级维护目标。
|
// openLibraryMaintenance 定义在 library_wiring.go:维护目标的装配与目标整库栈共用
|
||||||
// 未接线时返回明确原因:只锁治理库 scope 不算维护,宁可拒绝写也不冒充。
|
// 同一个包(internal/librarystack),不在这里再拼一份。
|
||||||
func openLibraryMaintenance(config appconfig.Config) (*librarymaintenance.Manager, error) {
|
|
||||||
database, err := librarymaintenance.DatabaseIdentity(config.LibraryProductionDatabaseURL)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
return librarymaintenance.New(librarymaintenance.Config{
|
|
||||||
Instance: config.LibraryTargetInstance,
|
|
||||||
Project: config.LibraryTargetProject,
|
|
||||||
Writers: config.LibraryTargetWriters,
|
|
||||||
NonWriters: config.LibraryTargetNonWriters,
|
|
||||||
DockerSocket: config.LibraryTargetDockerSocket,
|
|
||||||
IngressURL: config.LibraryTargetIngressURL,
|
|
||||||
IngressService: config.LibraryTargetIngressService,
|
|
||||||
IngressDrainFile: config.LibraryTargetIngressDrainFile,
|
|
||||||
Database: database,
|
|
||||||
Index: config.LibraryProductionIndex,
|
|
||||||
Bucket: config.LibraryProductionS3Bucket,
|
|
||||||
// 后台静默检查的运行时依赖:HTTP 排空不等于后台任务排空。
|
|
||||||
DatabaseURL: config.LibraryProductionDatabaseURL,
|
|
||||||
RedisAddr: config.LibraryTargetRedisAddr,
|
|
||||||
RedisPassword: config.LibraryTargetRedisPassword,
|
|
||||||
RedisDB: config.LibraryTargetRedisDB,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// runLibraryRelease 执行"闸口 → 授权 → 关闭目标入口并停止写入者 → 预备份 →
|
// runLibraryRelease 执行"闸口 → 授权 → 关闭目标入口并停止写入者 → 预备份 →
|
||||||
// 四类写入 → 校验 → 登记权威版本 → 恢复原运行状态和入口"的完整链路。
|
// 四类写入 → 校验 → 登记权威版本 → 恢复原运行状态和入口"的完整链路。
|
||||||
@@ -116,6 +93,21 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
// 绝不重新备份——重新备份会把"半写状态"当成新基线,覆盖真正可回滚的那份。
|
// 绝不重新备份——重新备份会把"半写状态"当成新基线,覆盖真正可回滚的那份。
|
||||||
previous, err := store.LibraryJob(ctx, payload.LibraryJobID)
|
previous, err := store.LibraryJob(ctx, payload.LibraryJobID)
|
||||||
resumed := err == nil && previous.SnapshotID != ""
|
resumed := err == nil && previous.SnapshotID != ""
|
||||||
|
if resumed {
|
||||||
|
versions, versionErr := store.LibraryVersions(ctx, payload.Scope, 50)
|
||||||
|
switch classifyTakeover(previous, versions, versionErr) {
|
||||||
|
case takeoverUnknown:
|
||||||
|
// 版本状态读不到、又没有阶段证据:不能当成"没有收尾证据"继续写目标。
|
||||||
|
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
|
||||||
|
"接管时读不到该目标的版本状态,无法判断是只差收尾还是要重做;未对目标执行任何写入,稍后重试", "prebackup")
|
||||||
|
case takeoverSuperseded:
|
||||||
|
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
|
||||||
|
"本快照的生效版本已被别的快照顶替,拒绝用旧快照覆盖当前生产版本", "prebackup")
|
||||||
|
case takeoverPendingFinal:
|
||||||
|
// 上次可能死在"锁已放、终态没写"之间:只复核与登记,不重写生产。
|
||||||
|
payload.FinalizeOnly = true
|
||||||
|
}
|
||||||
|
}
|
||||||
if resumed {
|
if resumed {
|
||||||
recorder.Stage(ctx, "prebackup", "skipped", "接管失败前已创建的预备份 "+previous.SnapshotID)
|
recorder.Stage(ctx, "prebackup", "skipped", "接管失败前已创建的预备份 "+previous.SnapshotID)
|
||||||
}
|
}
|
||||||
@@ -194,7 +186,7 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
"目标拓扑不满足唯一入口约束,拒绝开维护窗口:"+topoErr.Error(), "prebackup")
|
"目标拓扑不满足唯一入口约束,拒绝开维护窗口:"+topoErr.Error(), "prebackup")
|
||||||
}
|
}
|
||||||
targetIdentity := fmt.Sprintf("scope=%s db=%s index=%s bucket=%s", payload.Scope,
|
targetIdentity := fmt.Sprintf("scope=%s db=%s index=%s bucket=%s", payload.Scope,
|
||||||
redactDSN(config.LibraryProductionDatabaseURL), config.LibraryProductionIndex, config.LibraryProductionS3Bucket)
|
pgdsn.Endpoint(config.LibraryProductionDatabaseURL), config.LibraryProductionIndex, config.LibraryProductionS3Bucket)
|
||||||
originalState, stateErr := manager.OriginalState(ctx)
|
originalState, stateErr := manager.OriginalState(ctx)
|
||||||
if stateErr != nil {
|
if stateErr != nil {
|
||||||
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
|
return failLibraryJob(ctx, store, recorder, payload.LibraryJobID,
|
||||||
@@ -256,6 +248,7 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
if resumed {
|
if resumed {
|
||||||
contentState = contentHalfWritten
|
contentState = contentHalfWritten
|
||||||
}
|
}
|
||||||
|
pending := successBoundary{}
|
||||||
tracked := &contentTrackedTarget{inner: target, onApply: func() { contentState = contentHalfWritten }}
|
tracked := &contentTrackedTarget{inner: target, onApply: func() { contentState = contentHalfWritten }}
|
||||||
allowRestore := func(error) bool {
|
allowRestore := func(error) bool {
|
||||||
return contentState == contentUntouched || contentState == contentSucceeded || contentState == contentRolledBack
|
return contentState == contentUntouched || contentState == contentSucceeded || contentState == contentRolledBack
|
||||||
@@ -290,8 +283,9 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
return errors.New(message)
|
return errors.New(message)
|
||||||
}
|
}
|
||||||
recorder.Stage(ctx, "finalize", "succeeded", "已登记生效版本(仅收尾,未重写生产)")
|
recorder.Stage(ctx, "finalize", "succeeded", "已登记生效版本(仅收尾,未重写生产)")
|
||||||
_ = store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobSucceeded, "finalize", "", "",
|
// 成功要等维护锁释放完才暴露:这里只记"待收尾"。
|
||||||
previous.SnapshotID, payload.TargetSnapshotID, time.Now().UTC())
|
pending = successBoundary{Ready: true, Backup: previous.SnapshotID,
|
||||||
|
Target: payload.TargetSnapshotID, FinishedAt: time.Now().UTC()}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -340,13 +334,10 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
return errors.New(message)
|
return errors.New(message)
|
||||||
}
|
}
|
||||||
recorder.Stage(ctx, "finalize", "succeeded", "已登记生效版本")
|
recorder.Stage(ctx, "finalize", "succeeded", "已登记生效版本")
|
||||||
if advanceErr := store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobSucceeded, "finalize", "", "",
|
// 内容与版本都落定了,但还不能报成功:服务健康、入口开放、两把维护锁
|
||||||
outcome.Backup.SnapshotID, payload.TargetSnapshotID, finishedAt); advanceErr != nil {
|
// 释放完成才允许暴露 succeeded(见 finishLibrarySuccess)。
|
||||||
// 状态落不下去时不要把队列作业报成功:让上层可见并等待接管。
|
pending = successBoundary{Ready: true, Backup: outcome.Backup.SnapshotID,
|
||||||
return fmt.Errorf("作业状态写入失败: %w", advanceErr)
|
Target: payload.TargetSnapshotID, FinishedAt: finishedAt}
|
||||||
}
|
|
||||||
log.Printf("library %s succeeded job=%s scope=%s snapshot=%s backup=%s", kind,
|
|
||||||
payload.LibraryJobID, payload.Scope, payload.TargetSnapshotID, outcome.Backup.SnapshotID)
|
|
||||||
return nil
|
return nil
|
||||||
case "rolled_back":
|
case "rolled_back":
|
||||||
contentState = contentRolledBack
|
contentState = contentRolledBack
|
||||||
@@ -407,10 +398,19 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
}
|
}
|
||||||
return windowErr
|
return windowErr
|
||||||
case contentState == contentSucceeded:
|
case contentState == contentSucceeded:
|
||||||
if err := releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
|
if !pending.Ready {
|
||||||
"写入与校验完成,目标已按原运行状态恢复", now2); err != nil {
|
// 内容状态说成功但没有收尾凭据(不可能两者都缺):按未收尾处理,保持维护。
|
||||||
return fmt.Errorf("目标已恢复但维护占用释放失败: %w", err)
|
note := "内容状态为已验证,但缺少收尾凭据,目标保持维护"
|
||||||
|
_ = store.UpdateLibraryMaintenanceState(ctx, payload.Scope, payload.LibraryJobID, "needs_restore", note, now2)
|
||||||
|
_ = store.UpdateInstanceMaintenanceState(ctx, identity.Instance, payload.LibraryJobID, "needs_restore", note, now2)
|
||||||
|
return errors.New(note)
|
||||||
}
|
}
|
||||||
|
if finishErr := finishLibrarySuccess(ctx, store, manager, identity.Instance, payload.Scope,
|
||||||
|
payload.LibraryJobID, pending); finishErr != nil {
|
||||||
|
return finishErr
|
||||||
|
}
|
||||||
|
log.Printf("library %s succeeded job=%s scope=%s snapshot=%s backup=%s", kind,
|
||||||
|
payload.LibraryJobID, payload.Scope, payload.TargetSnapshotID, pending.Backup)
|
||||||
return windowErr
|
return windowErr
|
||||||
case contentState == contentRolledBack:
|
case contentState == contentRolledBack:
|
||||||
if err := releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
|
if err := releaseWindowLocks(ctx, store, identity.Instance, payload.Scope, payload.LibraryJobID,
|
||||||
@@ -428,19 +428,6 @@ func runLibraryRelease(ctx context.Context, store *postgresstore.Store, config a
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// releaseWindowLocks 是唯一释放路径:先记录入口已恢复,再放 scope 行与实例锁。
|
|
||||||
// 顺序反过来会出现"实例锁放了、scope 还挂着"的中间态,页面就会说正在维护。
|
|
||||||
func releaseWindowLocks(ctx context.Context, store *postgresstore.Store, instance, scope, jobID, note string, now time.Time) error {
|
|
||||||
_ = store.SetInstanceMaintenanceIngress(ctx, instance, jobID, "open", note, now)
|
|
||||||
_ = store.SetLibraryMaintenanceIngress(ctx, scope, jobID, "open", note, now)
|
|
||||||
_ = store.UpdateInstanceMaintenanceState(ctx, instance, jobID, "verified", note, now)
|
|
||||||
_ = store.UpdateLibraryMaintenanceState(ctx, scope, jobID, "verified", note, now)
|
|
||||||
if err := store.ReleaseInstanceMaintenance(ctx, instance, jobID, now); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
return store.ReleaseLibraryMaintenance(ctx, scope, jobID, now)
|
|
||||||
}
|
|
||||||
|
|
||||||
// resumeInterruptedLibraryJobs 在 worker 启动时接管中断的写作业:
|
// resumeInterruptedLibraryJobs 在 worker 启动时接管中断的写作业:
|
||||||
// 已经持久化预备份的重新排队(同一个业务作业 id,阶段与备份都接着用),
|
// 已经持久化预备份的重新排队(同一个业务作业 id,阶段与备份都接着用),
|
||||||
// 连预备份都没完成的才判失败——那种情况没有可回滚的基线,不能盲写。
|
// 连预备份都没完成的才判失败——那种情况没有可回滚的基线,不能盲写。
|
||||||
@@ -469,8 +456,13 @@ func resumeInterruptedLibraryJobs(ctx context.Context, store *postgresstore.Stor
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
queueType := librarySyncJobType
|
queueType := librarySyncJobType
|
||||||
if job.Kind == governance.LibraryJobRestore {
|
switch job.Kind {
|
||||||
|
case governance.LibraryJobRestore:
|
||||||
queueType = libraryRestoreJobType
|
queueType = libraryRestoreJobType
|
||||||
|
case governance.LibraryJobBaseline:
|
||||||
|
// 基线作业被杀时窗口可能还停着目标:用同一个作业 id 接管那个窗口,
|
||||||
|
// 继续捕获/校验,而不是留下一个 held 目标等人工处理。
|
||||||
|
queueType = libraryBaselineJobType
|
||||||
}
|
}
|
||||||
payload, _ := json.Marshal(map[string]string{"library_job_id": job.ID, "scope": job.Scope,
|
payload, _ := json.Marshal(map[string]string{"library_job_id": job.ID, "scope": job.Scope,
|
||||||
"target_snapshot_id": job.TargetSnapshotID})
|
"target_snapshot_id": job.TargetSnapshotID})
|
||||||
|
|||||||
+19
-106
@@ -2,117 +2,30 @@ package main
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"knowledge-governance/internal/appconfig"
|
"knowledge-governance/internal/appconfig"
|
||||||
"knowledge-governance/internal/librarysnapshot"
|
"knowledge-governance/internal/librarymaintenance"
|
||||||
|
"knowledge-governance/internal/librarystack"
|
||||||
v071 "knowledge-governance/internal/releaseadapter/v071"
|
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||||
)
|
)
|
||||||
|
|
||||||
// buildLibraryProductionStack 用部署注入的配置装配生产整库目标与独立预备份存储。
|
// 目标整库读写栈的装配只有一份实现(internal/librarystack):worker 的同步/恢复/
|
||||||
// 任何一项缺失都返回明确原因,调用方据此把作业失败并把缺项写进阶段,
|
// 基线与主机侧恢复演练共用它,避免"目标是谁、用哪份契约"随调用路径漂移。
|
||||||
// 不用"未配置"的占位对象假装能跑。
|
|
||||||
|
// openLibraryMaintenance 按部署注入的配置装配实例级维护目标。
|
||||||
|
// 未接线时返回明确原因:只锁治理库 scope 不算维护,宁可拒绝写也不冒充。
|
||||||
|
func openLibraryMaintenance(config appconfig.Config) (*librarymaintenance.Manager, error) {
|
||||||
|
return librarystack.OpenMaintenance(config)
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildLibraryProductionStack 用"要同步的这份快照"做契约装配目标与预备份存储。
|
||||||
func buildLibraryProductionStack(ctx context.Context, config appconfig.Config, scope, snapshotID, knownMapping string) (
|
func buildLibraryProductionStack(ctx context.Context, config appconfig.Config, scope, snapshotID, knownMapping string) (
|
||||||
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
|
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
|
||||||
missing := make([]string, 0, 4)
|
return librarystack.OpenProductionStack(ctx, config, scope, snapshotID, knownMapping)
|
||||||
if config.LibraryProductionDatabaseURL == "" {
|
}
|
||||||
missing = append(missing, "LIBRARY_PRODUCTION_DATABASE_URL")
|
|
||||||
}
|
// buildLibraryBaselineStack 装配"给目标自己建基线"所需的目标与捕获器(契约取目标当前映射)。
|
||||||
if config.LibraryProductionElasticURL == "" {
|
func buildLibraryBaselineStack(ctx context.Context, config appconfig.Config, scope string) (
|
||||||
missing = append(missing, "LIBRARY_PRODUCTION_ELASTIC_URL")
|
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
|
||||||
}
|
return librarystack.OpenBaselineStack(ctx, config, scope)
|
||||||
if config.LibraryProductionTenantID == 0 {
|
|
||||||
missing = append(missing, "LIBRARY_PRODUCTION_TENANT_ID")
|
|
||||||
}
|
|
||||||
kb := config.LibraryProductionInternalKB
|
|
||||||
if scope == "external" {
|
|
||||||
kb = config.LibraryProductionExternalKB
|
|
||||||
}
|
|
||||||
if kb == "" {
|
|
||||||
missing = append(missing, "LIBRARY_PRODUCTION_KB_"+strings.ToUpper(scope))
|
|
||||||
}
|
|
||||||
if config.LibraryProductionS3Endpoint == "" || config.LibraryProductionS3Bucket == "" ||
|
|
||||||
config.LibraryProductionS3AccessKey == "" || config.LibraryProductionS3SecretKey == "" {
|
|
||||||
missing = append(missing, "LIBRARY_PRODUCTION_S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY")
|
|
||||||
}
|
|
||||||
if len(missing) > 0 {
|
|
||||||
return nil, nil, fmt.Errorf("生产整库目标未配置:缺少 %s", strings.Join(missing, "、"))
|
|
||||||
}
|
|
||||||
secure := strings.HasPrefix(config.LibraryProductionS3Endpoint, "https://")
|
|
||||||
objects, err := v071.NewS3ObjectStore(v071.S3ObjectStoreConfig{
|
|
||||||
Provider: config.LibraryProductionS3Provider, Endpoint: config.LibraryProductionS3Endpoint,
|
|
||||||
Region: config.LibraryProductionS3Region, Bucket: config.LibraryProductionS3Bucket,
|
|
||||||
AccessKeyID: config.LibraryProductionS3AccessKey, SecretAccessKey: config.LibraryProductionS3SecretKey,
|
|
||||||
Secure: secure, ForcePathStyle: true, MaxObjectBytes: 256 * 1024 * 1024,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, errors.New("生产对象存储配置无效")
|
|
||||||
}
|
|
||||||
identity := v071.ProductionScope{TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: kb}
|
|
||||||
target, err := v071.NewProductionLibraryTarget(ctx, v071.ProductionLibraryConfig{
|
|
||||||
Postgres: config.LibraryProductionDatabaseURL, SearchURL: config.LibraryProductionElasticURL,
|
|
||||||
SearchIndex: config.LibraryProductionIndex, Objects: objects, ObjectsReader: objects,
|
|
||||||
ArchiveRoot: config.LibraryArchiveRoot,
|
|
||||||
ObjectsBucket: config.LibraryProductionS3Bucket,
|
|
||||||
MaxDocuments: 200000,
|
|
||||||
}, identity)
|
|
||||||
if err != nil {
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
// 契约以"要同步的这份快照"为准:生产必须向它看齐,而不是反过来。
|
|
||||||
// 生产为空时索引可能根本不存在,这时不能拿生产映射当契约(会永远启动不了)。
|
|
||||||
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
|
|
||||||
if err != nil {
|
|
||||||
target.Close()
|
|
||||||
return nil, nil, fmt.Errorf("归档不可用,无法读取快照契约: %w", err)
|
|
||||||
}
|
|
||||||
manifest, err := archive.Load(snapshotID)
|
|
||||||
archive.Close()
|
|
||||||
if err != nil {
|
|
||||||
target.Close()
|
|
||||||
return nil, nil, fmt.Errorf("读取目标快照失败: %w", err)
|
|
||||||
}
|
|
||||||
contract := manifest.Compatibility
|
|
||||||
if contract.EmbeddingDimension == 0 {
|
|
||||||
contract.EmbeddingModel = config.LibraryEmbeddingModel
|
|
||||||
contract.EmbeddingDimension = config.LibraryEmbeddingDimension
|
|
||||||
}
|
|
||||||
// 生产索引存在时必须与快照契约一致;不存在或为空基线时跳过比对
|
|
||||||
// (空索引的映射没有可比性,非空之后仍然严格比对)。
|
|
||||||
if mapping, mapErr := target.Search().MappingSHA256(ctx); mapErr == nil {
|
|
||||||
// 映射基线:优先用"上一次生产生效版本登记的实际映射",
|
|
||||||
// 它才是这份生产索引的既有事实;快照契约只管"来自哪份数据"。
|
|
||||||
baseline := knownMapping
|
|
||||||
if baseline == "" {
|
|
||||||
// 没有历史基线时(首次接管这套生产索引),把当前实际映射当作基线并随
|
|
||||||
// 预备份登记;之后任何改动都会与登记值不符而被拒绝。
|
|
||||||
baseline = mapping
|
|
||||||
}
|
|
||||||
if baseline != "" && mapping != baseline {
|
|
||||||
current, currentErr := target.Current(ctx)
|
|
||||||
emptyBaseline := currentErr == nil && current.Counts["index"] == 0 && current.Counts["chunks"] == 0
|
|
||||||
if !emptyBaseline {
|
|
||||||
target.Close()
|
|
||||||
return nil, nil, errors.New("生产索引映射与上次生效版本不一致(索引被改动过),先对齐索引映射")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// 预备份描述的是"生产现在的索引",不是"快照的来源索引":
|
|
||||||
// 生产映射基线优先用上次生效版本登记的实际值。
|
|
||||||
backupContract := contract
|
|
||||||
if knownMapping != "" {
|
|
||||||
backupContract.IndexMappingSHA256 = knownMapping
|
|
||||||
}
|
|
||||||
backups, err := v071.NewProductionBackupStore(v071.ProductionBackupConfig{
|
|
||||||
Pool: target.Pool(), Search: target.Search(), Objects: target.Objects(),
|
|
||||||
ArchiveRoot: config.LibraryArchiveRoot, Scope: scope, Identity: target.Identity(),
|
|
||||||
Compatibility: backupContract, MaxDocuments: 200000, MaxObjectByte: 256 * 1024 * 1024,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
target.Close()
|
|
||||||
return nil, nil, err
|
|
||||||
}
|
|
||||||
return target, backups, nil
|
|
||||||
}
|
}
|
||||||
|
|||||||
+29
-18
@@ -8,7 +8,6 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -58,21 +57,6 @@ func (r libraryStageRecorder) Stage(ctx context.Context, stage, state, detail st
|
|||||||
|
|
||||||
func int64Ptr(value int64) *int64 { return &value }
|
func int64Ptr(value int64) *int64 { return &value }
|
||||||
|
|
||||||
// redactDSN 只保留库名,避免把连接串写进维护记录。
|
|
||||||
func redactDSN(dsn string) string {
|
|
||||||
if dsn == "" {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
if index := strings.LastIndex(dsn, "/"); index >= 0 && index+1 < len(dsn) {
|
|
||||||
name := dsn[index+1:]
|
|
||||||
if cut := strings.Index(name, "?"); cut >= 0 {
|
|
||||||
name = name[:cut]
|
|
||||||
}
|
|
||||||
return name
|
|
||||||
}
|
|
||||||
return "configured"
|
|
||||||
}
|
|
||||||
|
|
||||||
// stageSize 是某一类数据在快照里的真实总量(行数/字节)。
|
// stageSize 是某一类数据在快照里的真实总量(行数/字节)。
|
||||||
type stageSize struct {
|
type stageSize struct {
|
||||||
count int64
|
count int64
|
||||||
@@ -109,6 +93,9 @@ const libraryRecoverJobType = "library.recover"
|
|||||||
// libraryRestoreJobType 是按指定整库版本恢复的队列类型;与同步分开、授权也分开。
|
// libraryRestoreJobType 是按指定整库版本恢复的队列类型;与同步分开、授权也分开。
|
||||||
const libraryRestoreJobType = "library.restore"
|
const libraryRestoreJobType = "library.restore"
|
||||||
|
|
||||||
|
// libraryBaselineJobType 是"建立目标权威基线"的队列类型(与 API 侧常量同值)。
|
||||||
|
const libraryBaselineJobType = "library.baseline"
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||||
defer stop()
|
defer stop()
|
||||||
@@ -205,6 +192,8 @@ func run(ctx context.Context, getenv func(string) string) error {
|
|||||||
go runRecordsSyncScheduler(ctx, mirrorStore, func(error) { log.Print("QA local mirror scheduler unavailable; last committed data retained") })
|
go runRecordsSyncScheduler(ctx, mirrorStore, func(error) { log.Print("QA local mirror scheduler unavailable; last committed data retained") })
|
||||||
}
|
}
|
||||||
go weeklyMailLoop(ctx, pool, getenv)
|
go weeklyMailLoop(ctx, pool, getenv)
|
||||||
|
// 封存执行器状态:worker 是唯一能连 agent socket 的一方,它写观测、页面读观测。
|
||||||
|
go runLibraryExecutorProbe(ctx, store, config, libraryExecutorProbeInterval)
|
||||||
go runSourceRefreshScheduler(ctx, store, config.SourceRefreshSchedulerInterval, func(err error) {
|
go runSourceRefreshScheduler(ctx, store, config.SourceRefreshSchedulerInterval, func(err error) {
|
||||||
log.Printf("worker %s source refresh scheduler error: %v", config.WorkerID, err)
|
log.Printf("worker %s source refresh scheduler error: %v", config.WorkerID, err)
|
||||||
})
|
})
|
||||||
@@ -284,7 +273,16 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
|
|||||||
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", "dead letter release storage unavailable", "", "", "", time.Now().UTC())
|
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", "dead letter release storage unavailable", "", "", "", time.Now().UTC())
|
||||||
return fmt.Errorf("load dead letter releases: %w", releaseErr)
|
return fmt.Errorf("load dead letter releases: %w", releaseErr)
|
||||||
}
|
}
|
||||||
result, err := libraryagent.CaptureReleased(ctx, agentConfig, payload.Scope, payload.CaptureJob, released)
|
// 期望执行器身份随请求一起送进 agent,由接收方在操作互斥内、任何源操作之前
|
||||||
|
// 比对:这样即使 socket 在这之前被换掉,错的执行器也一次 run 都不会跑。
|
||||||
|
expectation, expectErr := captureExpectation(config)
|
||||||
|
if expectErr != nil {
|
||||||
|
_ = store.UpdateLibraryStage(ctx, libraryJobID, "export", "failed", nil, nil, nil, nil, expectErr.Error(), time.Now().UTC())
|
||||||
|
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", expectErr.Error(),
|
||||||
|
"未停任何写入者;补齐来源身份配置后重试", "", "", time.Now().UTC())
|
||||||
|
return expectErr
|
||||||
|
}
|
||||||
|
result, err := libraryagent.CaptureReleased(ctx, agentConfig, payload.Scope, payload.CaptureJob, released, expectation)
|
||||||
finishedAt := time.Now().UTC()
|
finishedAt := time.Now().UTC()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// 导出失败时阶段与作业都要留痕;恢复责任在 agent,不会因为作业失败而假装来源已恢复。
|
// 导出失败时阶段与作业都要留痕;恢复责任在 agent,不会因为作业失败而假装来源已恢复。
|
||||||
@@ -294,13 +292,22 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
|
|||||||
log.Printf("library capture failed scope=%s capture_job=%s: %v", payload.Scope, payload.CaptureJob, err)
|
log.Printf("library capture failed scope=%s capture_job=%s: %v", payload.Scope, payload.CaptureJob, err)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
// 执行后复核:结果里 agent 自报的身份必须还是期望实例,否则这次封存
|
||||||
|
// 可能来自别的栈,绝不能登记成"本实例的测试封存"。
|
||||||
|
if identityErr := captureExecutorPostcheck(result.Payload, expectation); identityErr != nil {
|
||||||
|
_ = store.UpdateLibraryStage(ctx, libraryJobID, "export", "failed", nil, nil, nil, nil, identityErr.Error(), finishedAt)
|
||||||
|
_ = store.AdvanceLibraryJob(ctx, libraryJobID, governance.LibraryJobFailed, "export", identityErr.Error(),
|
||||||
|
"快照已产生但来源身份不符,未登记为测试封存;核对执行器后重新封存", "", "", finishedAt)
|
||||||
|
return identityErr
|
||||||
|
}
|
||||||
// 封存产物成版本记录:只登记"测试封存",绝不写成生产生效版本。
|
// 封存产物成版本记录:只登记"测试封存",绝不写成生产生效版本。
|
||||||
versionErr := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
|
versionErr := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
|
||||||
ID: governance.LibraryVersionID(governance.LibraryVersionTestSealed, payload.Scope, result.SnapshotID),
|
ID: governance.LibraryVersionID(governance.LibraryVersionTestSealed, payload.Scope, result.SnapshotID),
|
||||||
Scope: payload.Scope,
|
Scope: payload.Scope,
|
||||||
Kind: governance.LibraryVersionTestSealed, SnapshotID: result.SnapshotID, SourceJobID: libraryJobID,
|
Kind: governance.LibraryVersionTestSealed, SnapshotID: result.SnapshotID, SourceJobID: libraryJobID,
|
||||||
Evidence: map[string]any{"capture_job": payload.CaptureJob, "replay": result.Replay,
|
Evidence: map[string]any{"capture_job": payload.CaptureJob, "replay": result.Replay,
|
||||||
"requested_by": payload.RequestedBy, "agent_result": json.RawMessage(result.Payload)},
|
"requested_by": payload.RequestedBy, "expectation_fingerprint": expectationFingerprint(expectation),
|
||||||
|
"agent_result": json.RawMessage(result.Payload)},
|
||||||
CreatedBy: payload.RequestedBy, CreatedAt: finishedAt,
|
CreatedBy: payload.RequestedBy, CreatedAt: finishedAt,
|
||||||
})
|
})
|
||||||
if versionErr != nil {
|
if versionErr != nil {
|
||||||
@@ -322,6 +329,10 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
|
|||||||
handlers[libraryRestoreJobType] = func(ctx context.Context, job jobs.Job) error {
|
handlers[libraryRestoreJobType] = func(ctx context.Context, job jobs.Job) error {
|
||||||
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobRestore)
|
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobRestore)
|
||||||
}
|
}
|
||||||
|
// 建立目标权威基线:目标非空又没有生效版本时的唯一入口(只读捕获 + 校验 + 登记)。
|
||||||
|
handlers[libraryBaselineJobType] = func(ctx context.Context, job jobs.Job) error {
|
||||||
|
return runLibraryBaseline(ctx, store, config, job)
|
||||||
|
}
|
||||||
// A crash can leave the source writer stopped: recover before taking work,
|
// A crash can leave the source writer stopped: recover before taking work,
|
||||||
// and keep the failure visible instead of pretending the worker is ready.
|
// and keep the failure visible instead of pretending the worker is ready.
|
||||||
go func() {
|
go func() {
|
||||||
|
|||||||
@@ -1,7 +1,15 @@
|
|||||||
# 整库同步验收矩阵(真实证据对照,2026-10-29 重排)
|
# 整库同步验收矩阵(真实证据对照,2026-10-29 重排)
|
||||||
|
|
||||||
本文件是**唯一口径**:状态只写实际发生的事,四类严格分开,不用"部分通过"含糊过去。
|
本文件是**唯一口径**:状态只写实际发生的事,四类严格分开,不用"部分通过"含糊过去。
|
||||||
**2026-10-29 父审后重跑:V03-a(Wiki 撤销语义)不通过,单独列在「一·补」,不在「已完成」表里。**
|
|
||||||
|
**2026-10-29 用户决定(覆盖历史验收口径)**:发布管理**不承担** Wiki 语义撤销——
|
||||||
|
它只负责把测试端已确认的整库快照**原样还原**生产。因此撤销后正文里是否残留
|
||||||
|
撤销方独有事实,属于**上游知识质量**问题,不在本模块验收范围内:V03-a 保留原始
|
||||||
|
失败证据(见「一·补」下方的"模块范围外"表),不改成通过,也不再阻断发布。
|
||||||
|
同时**不**新增 LLM 重写、人工逐页复核闸门,也不把升级 WeKnora 作为前置条件。
|
||||||
|
保留的门槛是:整库文档/分块/索引/资源/Wiki 正文·来源·链接·目录一致、稳定切点、
|
||||||
|
实际维护与排空、准确快照授权、独立预备份与恢复、失败回滚、作业互斥与重启接管、
|
||||||
|
真实 UI 进度与存储安全。效果问答/性能/A06 不是门槛。
|
||||||
|
|
||||||
| 口径 | 含义 |
|
| 口径 | 含义 |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -49,6 +57,8 @@ A06(内容、问答、引用图片效果、性能报告)已由用户取消
|
|||||||
| **V03-c** | 带"撤销后共享页"的整库封存 → 独立目标同步 → 恢复,逐项真实读回 | `libraryctl` 封存 + `ProductionLibraryTarget` + `libraryrelease` | **本轮终态真跑(字节一致性,不是撤销语义)**:KB `e64e82b5`(9 Wiki 页、1 文档、2 分块、1 资源)→ 快照 `0f0a3b68…` 与源逐页一致(`05-snapshot-compare.json` ok=True)→ 同步到独立实例 kgseal `libsync-internal-1793229249632461601` succeeded → 目标逐页/索引/资源字节一致(`07-target-compare.json` ok=True,含目标桶对象 sha256)→ `librestore-internal-1793229274077866298` succeeded → 目标仍逐项一致。**注意**:该共享页正文仍带撤销方残留(见 V03-a 不通过),本行只证明"搬运/恢复逐字节一致" |
|
| **V03-c** | 带"撤销后共享页"的整库封存 → 独立目标同步 → 恢复,逐项真实读回 | `libraryctl` 封存 + `ProductionLibraryTarget` + `libraryrelease` | **本轮终态真跑(字节一致性,不是撤销语义)**:KB `e64e82b5`(9 Wiki 页、1 文档、2 分块、1 资源)→ 快照 `0f0a3b68…` 与源逐页一致(`05-snapshot-compare.json` ok=True)→ 同步到独立实例 kgseal `libsync-internal-1793229249632461601` succeeded → 目标逐页/索引/资源字节一致(`07-target-compare.json` ok=True,含目标桶对象 sha256)→ `librestore-internal-1793229274077866298` succeeded → 目标仍逐项一致。**注意**:该共享页正文仍带撤销方残留(见 V03-a 不通过),本行只证明"搬运/恢复逐字节一致" |
|
||||||
| **S06-b** | 主机侧封存/恢复必须与 worker 讲同一份维护声明 | `internal/librarymaintenance/env.go`(`LIBRARY_TARGET_*` 唯一解析处) | 缺项时 `capture` 在开窗口前拒绝并点名;agent 启动打印维护目标;单测 5 项(逐项缺失/空白/口令空格/队列库号) |
|
| **S06-b** | 主机侧封存/恢复必须与 worker 讲同一份维护声明 | `internal/librarymaintenance/env.go`(`LIBRARY_TARGET_*` 唯一解析处) | 缺项时 `capture` 在开窗口前拒绝并点名;agent 启动打印维护目标;单测 5 项(逐项缺失/空白/口令空格/队列库号) |
|
||||||
| **Neo4j 启用场景** | 启用图谱的库必须被真实拒绝 | `v071/library_export.go` 读 `indexing_strategy.graph_enabled` | **本轮真跑**:在 kgmaint 建 `graph_enabled=true` 的真实 KB `8de41ca7…`,用真实 CLI 预检 → 退出码 1、`LIBRARYCTL_FAILED: Neo4j graph snapshot is not supported`(`evidence/10-neo4j-refuse.txt`)。不是 skip |
|
| **Neo4j 启用场景** | 启用图谱的库必须被真实拒绝 | `v071/library_export.go` 读 `indexing_strategy.graph_enabled` | **本轮真跑**:在 kgmaint 建 `graph_enabled=true` 的真实 KB `8de41ca7…`,用真实 CLI 预检 → 退出码 1、`LIBRARYCTL_FAILED: Neo4j graph snapshot is not supported`(`evidence/10-neo4j-refuse.txt`)。不是 skip |
|
||||||
|
| **执行器可用性(运行时观测)** | 页面的"可发起封存"来自 worker 的真实探测,不是配置字符串 | `internal/libraryexecutor` + agent `health` + 迁移 074 `library_executor_status` + worker 30s 心跳 | **本轮真跑(候选栈,代码 SHA 4c79052 / 0.1.192 的工作树构建)**:`GET /library/overview` 返回 `capture_available=true` 且 `executor={instance:kgmaint-weknora, project:kgmaint-weknora, source_app:kgmaint-app, observed_at:…}`;停候选 agent 后同一接口转 `false`、原因 `主机侧封存执行器不可达:library agent is unreachable`,此时 `POST /library/captures` 返回 **503 `library_capture_unavailable`**;重启 agent 后自动恢复 `true`。证据 `$TMPDIR/kg-candidate-verify/evidence/{01,03,04,05}-*.json` |
|
||||||
|
| **按版本恢复入口** | 页面不再把"恢复至该版本"写死 disabled;条件来自双闸/授权/维护 | `GET /api/v1/library/restore-availability` + `web/src/ReleaseTargets.tsx` | 接口真实返回 200(候选栈,缺授权时逐条列出原因);页面按 `allowed` 启用、管理员才见"签发恢复授权(30 分钟)",授权与执行是两步。证据 `02-restore-availability.json` + 前端用例 `libraryRestore.test.ts` / `ReleaseTargets.test.tsx` |
|
||||||
| **U04** | 维护级别/进度/恢复错误真实呈现 | `MaintenanceCard`(`web/src/ReleaseTargets.tsx`)、`/api/v1/library/maintenance` | 显示实例级/库级、持有 scope+作业、阶段、入口开/关、服务、开始/释放、进度(阶段名中文化)、恢复错误与处理记录;仅"窗口仍持有且作业失败"时给"按预备份恢复";无记录时"未接入 + fail closed"。组件测试 6 项 |
|
| **U04** | 维护级别/进度/恢复错误真实呈现 | `MaintenanceCard`(`web/src/ReleaseTargets.tsx`)、`/api/v1/library/maintenance` | 显示实例级/库级、持有 scope+作业、阶段、入口开/关、服务、开始/释放、进度(阶段名中文化)、恢复错误与处理记录;仅"窗口仍持有且作业失败"时给"按预备份恢复";无记录时"未接入 + fail closed"。组件测试 6 项 |
|
||||||
|
|
||||||
| **V05** | 导出中源变更必须被入口/写入者阻断,切点一致 | 闸门 `drain.json` + 真实并发客户端 | **本轮终态真跑**:入口关闭的同一刻(固定协调点,非 sleep)对专用源发 8 写 + 4 读 → 8 写全部失败(7 `URLError`/1 `RemoteDisconnected`)、4 读 `RemoteDisconnected`;写入者被停下的那一刻(轮询容器状态捕获 `running→exited→running`)再发 4 写 + 2 读 → 全部 `URLError`。事后源库 `v05` 文档 0 / 分块 0,快照与源逐项一致(ok=True),在途文档 `f7fed988…` completed/7 分块且出现在快照里。证据 `04-inflight-v05-final.json` |
|
| **V05** | 导出中源变更必须被入口/写入者阻断,切点一致 | 闸门 `drain.json` + 真实并发客户端 | **本轮终态真跑**:入口关闭的同一刻(固定协调点,非 sleep)对专用源发 8 写 + 4 读 → 8 写全部失败(7 `URLError`/1 `RemoteDisconnected`)、4 读 `RemoteDisconnected`;写入者被停下的那一刻(轮询容器状态捕获 `running→exited→running`)再发 4 写 + 2 读 → 全部 `URLError`。事后源库 `v05` 文档 0 / 分块 0,快照与源逐项一致(ok=True),在途文档 `f7fed988…` completed/7 分块且出现在快照里。证据 `04-inflight-v05-final.json` |
|
||||||
@@ -59,7 +69,13 @@ A06(内容、问答、引用图片效果、性能报告)已由用户取消
|
|||||||
|
|
||||||
| ID | 要点 | 承载位置 | 本轮真实结果 |
|
| ID | 要点 | 承载位置 | 本轮真实结果 |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| **V03-a** | 撤销一份共享贡献后,保留另一份贡献、清除撤销方独有正文/引用/实体边、目录正确 | 由 WeKnora 源侧执行(`cleanupWikiOnKnowledgeDelete` + `EnqueueWikiRetract`),治理侧只比对/搬运 | **本轮不通过(2026-10-29 重跑)**:真实 `DELETE /api/v1/knowledge/{id}` 后,共享页 `entity/fa-bu-zha-men` 的来源/引用/边/目录都按上游 retract 更新(source_refs 只剩保留文档、引用 2→1、9 页/18 边无死链),**但正文仍保留撤销方独有的两句事实 `ZEBRA-A-7731`/`ZEBRA-A-9910`**(version=3、3733 字节、sha256 `5fd27348…`)。封存逐字节一致只说明"存了真实状态",不等于撤销语义通过。证据 `kg-final-gaps/evidence/14-wiki-residue-evidence.json` + `01-wiki-source-after-r3.json` |
|
| **(无)** | 本模块没有"不通过"项:撤销语义已明确移出范围(见上表下方"模块范围外"),不在此改写为通过 | — | — |
|
||||||
|
|
||||||
|
### 模块范围外(用户 2026-10-29 决定;证据保留,不改判)
|
||||||
|
|
||||||
|
| ID | 要点 | 本轮真实结果(原样保留) |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| **V03-a** | 撤销一份共享贡献后,清除撤销方独有正文/引用/实体边 | **上游不通过,模块范围外**:真实 `DELETE /api/v1/knowledge/{id}` 后,共享页 `entity/fa-bu-zha-men` 的来源/引用/边/目录都按上游 retract 更新(source_refs 只剩保留文档、引用 2→1、9 页/18 边无死链),**但正文仍保留撤销方独有的两句事实 `ZEBRA-A-7731`/`ZEBRA-A-9910`**(version=3、3733 字节、sha256 `5fd27348…`)。这是上游 retract 走 LLM 重写的非确定性结果,治理侧只按字节搬运,不能也不应改写它。证据 `kg-final-gaps/evidence/14-wiki-residue-evidence.json` + `01-wiki-source-after-r3.json`,不涂改 |
|
||||||
|
|
||||||
## 二、缺代码
|
## 二、缺代码
|
||||||
|
|
||||||
@@ -74,6 +90,8 @@ A06(内容、问答、引用图片效果、性能报告)已由用户取消
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| S03-b | 死信/孤儿任务阻断的负向演练(本轮只验证了真实死信阻断) | 真实阻断已发生并记录;系统化的负向矩阵未跑 |
|
| S03-b | 死信/孤儿任务阻断的负向演练(本轮只验证了真实死信阻断) | 真实阻断已发生并记录;系统化的负向矩阵未跑 |
|
||||||
| 排空时序(历史结论) | 旧做法"202 之后抢时序上传"在 1faf513 上重跑 6 次全部 `Broken pipe`(`drain-task-final-1faf513*.json` 保留,不记通过) | 已被 S06-a 的确定性做法取代:先证明请求已被闸门接收,再启动维护。旧证据保留,不覆盖 |
|
| 排空时序(历史结论) | 旧做法"202 之后抢时序上传"在 1faf513 上重跑 6 次全部 `Broken pipe`(`drain-task-final-1faf513*.json` 保留,不记通过) | 已被 S06-a 的确定性做法取代:先证明请求已被闸门接收,再启动维护。旧证据保留,不覆盖 |
|
||||||
|
| 封存执行期身份复核(真实封存) | 期望身份随 capture 请求送进 agent,由它在操作互斥内、任何源操作之前比对(`swap_test.go` 用两个真实 agent 的 socket 替换做负例,断言错误 agent 的 capture 调用 0 次);真实封存尚未在最终构建上跑,本轮**没有**在候选栈发起真实封存(会产生新快照与停写入者窗口),因此"换 socket 被挡住"只有单测证据 |
|
||||||
|
| 按版本恢复的真实执行(新构建) | `restore-availability` 真实返回、恢复投递路径有单测;真实恢复作业仍在旧构建上验过(R01),本轮未重跑,不把旧证据算作新构建证据 |
|
||||||
| U05 | 正式 8088 页面验收 | 正式栈仍是 0.1.114,本轮未替换;维护面板/逐页 Wiki 差异只在测试与候选构建里验证过 |
|
| U05 | 正式 8088 页面验收 | 正式栈仍是 0.1.114,本轮未替换;维护面板/逐页 Wiki 差异只在测试与候选构建里验证过 |
|
||||||
|
|
||||||
## 四、需授权 / 需环境
|
## 四、需授权 / 需环境
|
||||||
@@ -97,9 +115,13 @@ api env `LIBRARY_AGENT_SOCKET=/run/library-agent/agent.sock`,worker 挂载
|
|||||||
本轮没有启动任何共享维护,也没有改回该单元——需要用户决定:正式上线前把单元恢复为 WKNFull 版
|
本轮没有启动任何共享维护,也没有改回该单元——需要用户决定:正式上线前把单元恢复为 WKNFull 版
|
||||||
(备份已在位),或给候选构建单独的 socket 路径。证据 `00-identity-final.json` 的 `formal_stack` 段。
|
(备份已在位),或给候选构建单独的 socket 路径。证据 `00-identity-final.json` 的 `formal_stack` 段。
|
||||||
|
|
||||||
另外:`knowledge-governance-p1-release-worker-1` 自 2026-10-10 起就在崩溃重启(`exit=1`、
|
另外:`knowledge-governance-p1-release-worker-1` 自 2026-10-10 起就在崩溃重启(`exit=1`,
|
||||||
`RestartCount≈1233`,日志只有 `connect release target database`),与本轮改动无关(它没有挂 agent socket),
|
`RestartCount` 已达 1500+,日志只有 `connect release target database`)。**本轮只读定位到根因**:
|
||||||
但正式栈的发布 worker 实际不可用,需要单独处理。
|
`config/release.sim.json` 的 `target.database_url_file` 指向 `prod-pg:25432`,而该主机名在本机与
|
||||||
|
容器网络里都不存在(`getent hosts prod-pg` 无结果、容器内 NXDOMAIN);`source` 指向的
|
||||||
|
`WKNFull-postgres` 可解析,所以是**目标库配置遗留**,不是代码缺陷。错误文本吞掉了主机与原因,
|
||||||
|
本轮已改为带端点 + 路由 + 脱敏原因(可在下次重启时直接看到),但**修正正式配置或停用该 worker
|
||||||
|
属于正式栈变更,需要部署窗口授权**,本轮没有执行。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -87,6 +87,23 @@ run164 因 qawemail 依赖外网 DNS 失败,已修,不涂改。
|
|||||||
| 0.1.185 | c542a133 | 资源行 storage_backend_id 列也改到目标实例 | 226 | **success** | 否 |
|
| 0.1.185 | c542a133 | 资源行 storage_backend_id 列也改到目标实例 | 226 | **success** | 否 |
|
||||||
| 0.1.186 | e508abd9 | 死信解除绑定封存源实例(LIBRARY_SOURCE_INSTANCE) | 227 | **success** | 否 |
|
| 0.1.186 | e508abd9 | 死信解除绑定封存源实例(LIBRARY_SOURCE_INSTANCE) | 227 | **success** | 否 |
|
||||||
|
|
||||||
|
| 0.1.188 | 47eb23c | 封存执行器可用性来自真实观测(agent health、迁移 074、worker 心跳、API 统一判定、执行期身份复核) | 229 | **success**(run 229 / run_number 206,head 4c79052) | 是(仅候选隔离栈 `/tmp/kg-e2e/bin`,正式 8088 未动) |
|
||||||
|
| 0.1.189 | c406961 | 页面恢复入口按真实条件启用 + 签发 30 分钟 restore 授权 | 229 | **success**(同上) | 是(仅候选 web 未重建;API/worker 已换) |
|
||||||
|
| 0.1.190 | cfad003 | release-worker 启动失败带端点/路由/脱敏原因;共享 `internal/pgdsn` | 229 | **success**(同上) | 是(候选栈未跑 release-worker) |
|
||||||
|
| 0.1.191 | 18276d0 | 文档:验收口径按用户决定收窄(V03-a 移出范围)、矩阵补执行器证据 | 229 | **success**(同上) | 否(只改文档) |
|
||||||
|
| 0.1.192 | 4c79052 | 测试 DSN 拼接构造(密钥扫描误报修复,语义不变) | 229 | **success**(head 4c79052,必需 context `build / build (push)` = success) | 否(只改测试) |
|
||||||
|
|
||||||
|
| 0.1.195 | 6ed0635 | 期望执行器身份随 capture 请求送达并在执行前核对(TOCTOU) | 233 | **success**(head c14dd01 之前的 run 232/233 连续绿;最终以 8d2ed8e/run234 为准) | 是(候选栈) |
|
||||||
|
| 0.1.196 | c14dd01 | 封存读资源失败带资源行与底层原因 | 233 | **success**(run 233 / run_number 210,head c14dd01,必需 context success) | 是(候选栈,clean 构建 `vcs.modified=false`) |
|
||||||
|
| 0.1.197 | 8d2ed8e | 维护释放与服务恢复完成后才暴露作业成功(R4) | 234 | **success**(run 234 / run_number 211,head 8d2ed8e,必需 context success) | 是(候选栈,clean 构建 `vcs.modified=false`;真实封存→同步→恢复与"成功后立即恢复 202"均在该构建上跑过) |
|
||||||
|
|
||||||
|
| 0.1.198 | 691e4e0 | 台账补记 0.1.195–0.1.197 | 235 | **success**(run 235,head 691e4e0) | 否(文档) |
|
||||||
|
| 0.1.199 | 24d2546 | 接管分类(未知/待收尾/被顶替)+ 维护释放幂等 | 236 | **success**(run 236,head 24d2546,必需 context success) | 是(候选栈,`-ldflags` 注入身份;`build-info` 读回 0.1.199 / 24d2546…,进程 `vcs.modified=false`) |
|
||||||
|
这批提交的候选栈验证:候选 api/worker/libraryagent/libraryctl 由该工作树重新构建(`vcs.modified=true`,
|
||||||
|
不是发布产物),二进制 sha256 记在 `$TMPDIR/kg-candidate-verify/evidence/01-binaries.json`;
|
||||||
|
旧二进制整目录备份在该目录下的 `bin-prev-*`,可一条命令回退。真实观测证据见
|
||||||
|
`$TMPDIR/kg-candidate-verify/evidence/{00..10}*.json`。
|
||||||
|
|
||||||
本轮(父审后)真实验收运行的最终代码 SHA 是 **0.1.186(e508abd9)**:封存→同步→恢复、
|
本轮(父审后)真实验收运行的最终代码 SHA 是 **0.1.186(e508abd9)**:封存→同步→恢复、
|
||||||
B02 删源对象后目标应用读回、V05 并发阻断都在这一份构建上跑完(PID/exe/revision 绑定见
|
B02 删源对象后目标应用读回、V05 并发阻断都在这一份构建上跑完(PID/exe/revision 绑定见
|
||||||
`kg-final-gaps/evidence/00-identity-final.json`)。0.1.181–0.1.183 是同一轮内的连续修复,
|
`kg-final-gaps/evidence/00-identity-final.json`)。0.1.181–0.1.183 是同一轮内的连续修复,
|
||||||
|
|||||||
@@ -99,7 +99,13 @@ type Config struct {
|
|||||||
LibraryTargetInstance string
|
LibraryTargetInstance string
|
||||||
// LibrarySourceInstance 是死信解除记录绑定的"来源实例"名字(封存源,不是同步目标)。
|
// LibrarySourceInstance 是死信解除记录绑定的"来源实例"名字(封存源,不是同步目标)。
|
||||||
// 留空时回落到 LibraryTargetInstance,保持单实例部署的旧行为。
|
// 留空时回落到 LibraryTargetInstance,保持单实例部署的旧行为。
|
||||||
LibrarySourceInstance string
|
LibrarySourceInstance string
|
||||||
|
// LibrarySourceProject 是封存执行器应该指向的实例名(agent -project)。
|
||||||
|
// 留空时等于来源实例名:环境里两者本来就同名(kgmaint-weknora)。
|
||||||
|
LibrarySourceProject string
|
||||||
|
// LibrarySourceApp 是封存源容器名(agent -source-app)。project 可能同名而源接线
|
||||||
|
// 不同,所以配了就一起绑定进"期望执行器身份"。
|
||||||
|
LibrarySourceApp string
|
||||||
LibraryTargetProject string
|
LibraryTargetProject string
|
||||||
LibraryTargetWriters []string
|
LibraryTargetWriters []string
|
||||||
LibraryTargetNonWriters map[string]string
|
LibraryTargetNonWriters map[string]string
|
||||||
@@ -115,10 +121,13 @@ type Config struct {
|
|||||||
LibraryTargetJournalRoot string
|
LibraryTargetJournalRoot string
|
||||||
// LibraryAgentSocket 是主机侧封存 agent 的 unix socket(只由 worker 连接)。
|
// LibraryAgentSocket 是主机侧封存 agent 的 unix socket(只由 worker 连接)。
|
||||||
// 留空即"未接线":页面不显示可用的封存入口,接口返回原因而不是假装已排队。
|
// 留空即"未接线":页面不显示可用的封存入口,接口返回原因而不是假装已排队。
|
||||||
LibraryAgentSocket string
|
LibraryAgentSocket string
|
||||||
VersionHistoryLinkOrigin string
|
// LibraryExecutorStatusMaxAge 是执行器状态的有效期:超过它就是"无法确认",
|
||||||
VersionHistoryHTTPTimeout time.Duration
|
// 不是"仍然可用"。worker 每 30 秒观测一次,默认 120 秒足够容错。
|
||||||
VersionHistoryMaxPages int
|
LibraryExecutorStatusMaxAge time.Duration
|
||||||
|
VersionHistoryLinkOrigin string
|
||||||
|
VersionHistoryHTTPTimeout time.Duration
|
||||||
|
VersionHistoryMaxPages int
|
||||||
}
|
}
|
||||||
|
|
||||||
func Load(getenv func(string) string) (Config, error) {
|
func Load(getenv func(string) string) (Config, error) {
|
||||||
@@ -203,6 +212,8 @@ func Load(getenv func(string) string) (Config, error) {
|
|||||||
LibraryProductionS3SecretKey: strings.TrimSpace(getenv("LIBRARY_PRODUCTION_S3_SECRET_KEY")),
|
LibraryProductionS3SecretKey: strings.TrimSpace(getenv("LIBRARY_PRODUCTION_S3_SECRET_KEY")),
|
||||||
LibraryTargetInstance: strings.TrimSpace(getenv("LIBRARY_TARGET_INSTANCE")),
|
LibraryTargetInstance: strings.TrimSpace(getenv("LIBRARY_TARGET_INSTANCE")),
|
||||||
LibrarySourceInstance: strings.TrimSpace(getenv("LIBRARY_SOURCE_INSTANCE")),
|
LibrarySourceInstance: strings.TrimSpace(getenv("LIBRARY_SOURCE_INSTANCE")),
|
||||||
|
LibrarySourceProject: strings.TrimSpace(getenv("LIBRARY_SOURCE_PROJECT")),
|
||||||
|
LibrarySourceApp: strings.TrimSpace(getenv("LIBRARY_SOURCE_APP")),
|
||||||
LibraryTargetProject: strings.TrimSpace(getenv("LIBRARY_TARGET_PROJECT")),
|
LibraryTargetProject: strings.TrimSpace(getenv("LIBRARY_TARGET_PROJECT")),
|
||||||
LibraryTargetWriters: splitList(getenv("LIBRARY_TARGET_WRITERS")),
|
LibraryTargetWriters: splitList(getenv("LIBRARY_TARGET_WRITERS")),
|
||||||
LibraryTargetNonWriters: splitDeclarations(getenv("LIBRARY_TARGET_NON_WRITERS")),
|
LibraryTargetNonWriters: splitDeclarations(getenv("LIBRARY_TARGET_NON_WRITERS")),
|
||||||
@@ -219,6 +230,7 @@ func Load(getenv func(string) string) (Config, error) {
|
|||||||
LibraryCapacityBudgetBytes: capacityBudget,
|
LibraryCapacityBudgetBytes: capacityBudget,
|
||||||
LibraryRetentionVersions: retentionVersions,
|
LibraryRetentionVersions: retentionVersions,
|
||||||
LibraryRetentionDays: retentionDays,
|
LibraryRetentionDays: retentionDays,
|
||||||
|
LibraryExecutorStatusMaxAge: 2 * time.Minute,
|
||||||
VersionHistoryLinkOrigin: strings.TrimSpace(getenv("VERSION_HISTORY_LINK_ORIGIN")),
|
VersionHistoryLinkOrigin: strings.TrimSpace(getenv("VERSION_HISTORY_LINK_ORIGIN")),
|
||||||
VersionHistoryHTTPTimeout: 15 * time.Second,
|
VersionHistoryHTTPTimeout: 15 * time.Second,
|
||||||
VersionHistoryMaxPages: valueOrDefaultInt(getenv("VERSION_HISTORY_MAX_PAGES"), 200),
|
VersionHistoryMaxPages: valueOrDefaultInt(getenv("VERSION_HISTORY_MAX_PAGES"), 200),
|
||||||
@@ -241,6 +253,9 @@ func Load(getenv func(string) string) (Config, error) {
|
|||||||
if config.SourceRefreshSchedulerInterval < 5*time.Second {
|
if config.SourceRefreshSchedulerInterval < 5*time.Second {
|
||||||
return Config{}, errors.New("SOURCE_REFRESH_SCHEDULER_INTERVAL must be at least 5s")
|
return Config{}, errors.New("SOURCE_REFRESH_SCHEDULER_INTERVAL must be at least 5s")
|
||||||
}
|
}
|
||||||
|
if config.LibraryExecutorStatusMaxAge, err = durationOrDefault(getenv("LIBRARY_EXECUTOR_STATUS_MAX_AGE"), config.LibraryExecutorStatusMaxAge); err != nil {
|
||||||
|
return Config{}, fmt.Errorf("LIBRARY_EXECUTOR_STATUS_MAX_AGE: %w", err)
|
||||||
|
}
|
||||||
if raw := getenv("RELEASE_WORKER_ENABLED"); raw != "" {
|
if raw := getenv("RELEASE_WORKER_ENABLED"); raw != "" {
|
||||||
config.ReleaseWorkerEnabled, err = strconv.ParseBool(raw)
|
config.ReleaseWorkerEnabled, err = strconv.ParseBool(raw)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -575,3 +590,13 @@ func (config Config) SourceInstance() string {
|
|||||||
}
|
}
|
||||||
return config.LibraryTargetInstance
|
return config.LibraryTargetInstance
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SourceProject 是封存执行器自报 project 的期望值。它描述的是**封存源**
|
||||||
|
// (agent 冻结哪一套写入者),不是同步目标:拿目标实例去比会把正确的执行器判成
|
||||||
|
// 不匹配(本机候选栈就是这种形状:源 kgmaint、目标 kgseal)。
|
||||||
|
func (config Config) SourceProject() string {
|
||||||
|
if config.LibrarySourceProject != "" {
|
||||||
|
return config.LibrarySourceProject
|
||||||
|
}
|
||||||
|
return config.SourceInstance()
|
||||||
|
}
|
||||||
|
|||||||
@@ -31,6 +31,10 @@ const (
|
|||||||
LibraryJobSync LibraryJobKind = "sync"
|
LibraryJobSync LibraryJobKind = "sync"
|
||||||
// LibraryJobRestore 是按指定整库版本恢复:与同步分开,授权与阶段也独立。
|
// LibraryJobRestore 是按指定整库版本恢复:与同步分开,授权与阶段也独立。
|
||||||
LibraryJobRestore LibraryJobKind = "restore"
|
LibraryJobRestore LibraryJobKind = "restore"
|
||||||
|
// LibraryJobBaseline 建立首次发布目标的权威基线:在维护隔离下把目标整库
|
||||||
|
// 读出来封存成快照并逐类校验,再登记成 production_effective。
|
||||||
|
// 它不写目标内容、不清空目标、不猜版本——基线只能来自真实捕获与校验。
|
||||||
|
LibraryJobBaseline LibraryJobKind = "baseline"
|
||||||
)
|
)
|
||||||
|
|
||||||
type LibraryJobState string
|
type LibraryJobState string
|
||||||
@@ -134,6 +138,12 @@ func SealStages() []string {
|
|||||||
return []string{"export"}
|
return []string{"export"}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// BaselineStages 是"建立目标权威基线"的阶段:捕获目标整库 → 与快照逐类校验
|
||||||
|
// → 登记生效版本。没有 rollback 阶段:基线过程不写目标内容。
|
||||||
|
func BaselineStages() []string {
|
||||||
|
return []string{"capture", "verify", "finalize"}
|
||||||
|
}
|
||||||
|
|
||||||
// LibraryReleaseAuthorization 是生产发布授权:绑定目标、快照、参数摘要与有效期。
|
// LibraryReleaseAuthorization 是生产发布授权:绑定目标、快照、参数摘要与有效期。
|
||||||
type LibraryReleaseAuthorization struct {
|
type LibraryReleaseAuthorization struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
|
|||||||
@@ -0,0 +1,156 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
"knowledge-governance/internal/jobs"
|
||||||
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 建立目标权威基线:首次发布时目标已经有内容,却没有生效版本记录。
|
||||||
|
//
|
||||||
|
// 唯一诚实的解法是"维护隔离 + 真实整库捕获 + 逐类校验 + 登记版本":
|
||||||
|
// - 不手写 production_effective,不伪造验收;
|
||||||
|
// - 不清空目标换库绕过(那会把"非空但未知"变成"空");
|
||||||
|
// - 页面只投递作业,停写入者、捕获、校验都在 worker 里执行。
|
||||||
|
//
|
||||||
|
// 这个入口只建作业,不签发发布授权:基线不等于发布,同步仍然要单独的 sync 授权。
|
||||||
|
const libraryBaselineJobType = "library.baseline"
|
||||||
|
|
||||||
|
type libraryBaselineRequest struct {
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func libraryBaselineEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
|
||||||
|
if r.Method != http.MethodPost {
|
||||||
|
writeJSON(w, http.StatusMethodNotAllowed, map[string]string{"error": "method not allowed"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
identity, ok := sessionFrom(r.Context())
|
||||||
|
if !ok || roleRank(identity.Role) < roleRank(roleAdmin) {
|
||||||
|
writeJSON(w, http.StatusForbidden, map[string]any{"error": "建立目标权威基线仅管理员可发起"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var request libraryBaselineRequest
|
||||||
|
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&request); err != nil {
|
||||||
|
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid request body"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if request.Scope != "internal" && request.Scope != "external" {
|
||||||
|
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "scope must be internal or external"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
store, ok := libraryStore(config)
|
||||||
|
if !ok {
|
||||||
|
writeJSON(w, http.StatusServiceUnavailable, map[string]any{"code": "library_baseline_unavailable",
|
||||||
|
"error": "版本登记存储不可用"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
versions, err := store.LibraryVersions(r.Context(), request.Scope, 50)
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot read library versions"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, item := range versions {
|
||||||
|
if item.Kind == governance.LibraryVersionProductionEffective && item.SupersededAt == nil {
|
||||||
|
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_baseline_exists",
|
||||||
|
"error": "该目标已有生效版本,不需要再建基线;换版本请走同步或恢复",
|
||||||
|
"snapshot_id": item.SnapshotID})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 目标必须真的读得到:读不到就不是"非空无权威版本",而是另一个问题,
|
||||||
|
// 不允许用建基线去掩盖连接/配置故障。
|
||||||
|
probe := config.LibraryScopeProbe
|
||||||
|
if probe == nil {
|
||||||
|
probe = func(ctx context.Context, scope string) (bool, map[string]int64, error) {
|
||||||
|
return productionScopeState(ctx, config, scope)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
empty, counts, probeErr := probe(r.Context(), request.Scope)
|
||||||
|
if probeErr != nil {
|
||||||
|
writeJSON(w, http.StatusServiceUnavailable, map[string]any{"code": "library_baseline_unprobeable",
|
||||||
|
"error": "无法读取目标整库状态,不能建立基线:" + probeErr.Error()})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if empty {
|
||||||
|
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_baseline_target_empty",
|
||||||
|
"error": "目标整库为空,不需要基线:首次同步会在写入前生成完整预备份并登记生效版本",
|
||||||
|
"counts": counts})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// 维护互斥与同步一致:基线要在目标上开维护窗口,目标正在维护时直接拒绝。
|
||||||
|
if body, held := instanceMaintenanceGuard(r.Context(), config, request.Scope); held {
|
||||||
|
writeJSON(w, http.StatusConflict, body)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if maintenanceStore, ok := config.Store.(interface {
|
||||||
|
LibraryMaintenanceOf(ctx context.Context, scope string) (*postgresstore.LibraryMaintenance, error)
|
||||||
|
}); ok {
|
||||||
|
if held, err := maintenanceStore.LibraryMaintenanceOf(r.Context(), request.Scope); err == nil && held != nil && held.ReleasedAt == nil {
|
||||||
|
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_target_in_maintenance",
|
||||||
|
"error": "目标正在维护中,验证完成前不接受新的写入请求",
|
||||||
|
"owner_job": held.OwnerJobID, "state": held.State})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
digest, err := governance.LibraryRequestDigest(request.Scope, governance.LibraryJobBaseline,
|
||||||
|
map[string]any{"scope": request.Scope})
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot digest request"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
now := time.Now().UTC()
|
||||||
|
job, reused, err := store.CreateLibraryJob(r.Context(), governance.LibraryJob{
|
||||||
|
ID: fmt.Sprintf("libbase-%s-%d", request.Scope, now.UnixNano()), Scope: request.Scope,
|
||||||
|
Kind: governance.LibraryJobBaseline,
|
||||||
|
IdempotencyKey: "library.baseline:" + request.Scope + ":" + now.Format("20060102T150405.000000000"),
|
||||||
|
RequestDigest: digest, Actor: identity.Username,
|
||||||
|
}, governance.BaselineStages(), now)
|
||||||
|
if errors.Is(err, governance.ErrLibraryJobActive) {
|
||||||
|
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_write_active",
|
||||||
|
"error": "该目标已有在飞的写作业(同步/恢复/基线互斥),等它结束再发起"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if errors.Is(err, governance.ErrLibraryJobIdempotencyConflict) {
|
||||||
|
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_baseline_conflict",
|
||||||
|
"error": "同一幂等键对应了不同的请求"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot create library baseline job"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if reused {
|
||||||
|
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": true})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
payload, err := json.Marshal(map[string]string{"library_job_id": job.ID, "scope": request.Scope})
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot build baseline payload"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
queueJob, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{
|
||||||
|
ID: governance.QueueJobIDFor(job.ID), Type: libraryBaselineJobType, Payload: payload,
|
||||||
|
AvailableAt: now, MaxAttempts: 1, IdempotencyKey: "library.baseline.queue:" + job.ID,
|
||||||
|
}, now)
|
||||||
|
if err != nil || queueJob == nil {
|
||||||
|
// 作业先落库、队列投递失败就把作业取消:不留下"看起来在跑"的空作业,
|
||||||
|
// 否则它会一直占着该 scope 的写互斥。
|
||||||
|
_ = store.CancelLibraryJob(r.Context(), job.ID, "队列投递失败,未开始建立基线", now)
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot enqueue baseline job"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if attachErr := store.AttachLibraryQueueJob(r.Context(), job.ID, queueJob.ID, now); attachErr != nil {
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot attach baseline job"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": false,
|
||||||
|
"target_counts": counts})
|
||||||
|
}
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
"knowledge-governance/internal/jobs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// nonEmptyTarget 模拟"目标整库非空且可读"的探测结果(真实实现直连目标库逐表计数)。
|
||||||
|
func nonEmptyTarget(context.Context, string) (bool, map[string]int64, error) {
|
||||||
|
return false, map[string]int64{"knowledges": 1, "chunks": 2}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// scopedProbe:internal 可读非空,external 没有目标 KB(与当前部署一致)。
|
||||||
|
func scopedProbe(_ context.Context, scope string) (bool, map[string]int64, error) {
|
||||||
|
if scope == "external" {
|
||||||
|
return false, nil, errors.New("该范围的目标知识库未配置")
|
||||||
|
}
|
||||||
|
return nonEmptyTarget(context.Background(), scope)
|
||||||
|
}
|
||||||
|
|
||||||
|
// baselineMutexStore 在 fake 之外**真的**实现"同一 scope 只允许一个在飞写作业"这条不变量,
|
||||||
|
// 用来验证接口的重试/双击行为(真实索引由 integration test
|
||||||
|
// TestLibraryWriteMutexCoversBaselineJobs 在 PostgreSQL 上证明)。
|
||||||
|
type baselineMutexStore struct {
|
||||||
|
fakeLibraryStore
|
||||||
|
activeByScope map[string]string
|
||||||
|
enqueues int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *baselineMutexStore) CreateLibraryJob(ctx context.Context, job governance.LibraryJob, stages []string, now time.Time) (governance.LibraryJob, bool, error) {
|
||||||
|
if s.activeByScope == nil {
|
||||||
|
s.activeByScope = map[string]string{}
|
||||||
|
}
|
||||||
|
if owner := s.activeByScope[job.Scope]; owner != "" {
|
||||||
|
return governance.LibraryJob{}, false, governance.ErrLibraryJobActive
|
||||||
|
}
|
||||||
|
s.activeByScope[job.Scope] = job.ID
|
||||||
|
return s.fakeLibraryStore.CreateLibraryJob(ctx, job, stages, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *baselineMutexStore) Enqueue(ctx context.Context, request jobs.EnqueueRequest, now time.Time) (*jobs.Job, error) {
|
||||||
|
s.enqueues++
|
||||||
|
return s.fakeLibraryStore.fakeJobStore.Enqueue(ctx, request, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 双击/重试:第一次建作业成功,第二次被互斥拒绝,且**不会**再投递一个队列作业。
|
||||||
|
func TestLibraryBaselineDoubleSubmitCreatesOneJob(t *testing.T) {
|
||||||
|
store := &baselineMutexStore{fakeLibraryStore: fakeLibraryStore{fakeJobStore: fakeJobStore{enqueueEchoRequestedID: true}}}
|
||||||
|
config := Config{Store: store, LibraryScopeProbe: nonEmptyTarget}
|
||||||
|
first := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(first, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), config)
|
||||||
|
if first.Code != http.StatusAccepted {
|
||||||
|
t.Fatalf("first status=%d body=%s", first.Code, first.Body.String())
|
||||||
|
}
|
||||||
|
if store.enqueues != 1 {
|
||||||
|
t.Fatalf("first submit enqueues=%d, want 1", store.enqueues)
|
||||||
|
}
|
||||||
|
second := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(second, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), config)
|
||||||
|
if second.Code != http.StatusConflict {
|
||||||
|
t.Fatalf("second status=%d body=%s", second.Code, second.Body.String())
|
||||||
|
}
|
||||||
|
if body := decodeBody(t, second); body["code"] != "library_write_active" {
|
||||||
|
t.Fatalf("second body=%v", body)
|
||||||
|
}
|
||||||
|
if store.enqueues != 1 {
|
||||||
|
t.Fatalf("rejected submit must not enqueue again: enqueues=%d", store.enqueues)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 不同 scope 不会被同一把锁挡住(互斥粒度是 scope/实例,不是全局)。
|
||||||
|
func TestLibraryBaselineDifferentScopesAreNotBlocked(t *testing.T) {
|
||||||
|
store := &baselineMutexStore{fakeLibraryStore: fakeLibraryStore{fakeJobStore: fakeJobStore{enqueueEchoRequestedID: true}}}
|
||||||
|
config := Config{Store: store, LibraryScopeProbe: scopedProbe}
|
||||||
|
internal := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(internal, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), config)
|
||||||
|
if internal.Code != http.StatusAccepted {
|
||||||
|
t.Fatalf("internal status=%d body=%s", internal.Code, internal.Body.String())
|
||||||
|
}
|
||||||
|
// external 在探针阶段就会被拒(该范围没有目标 KB),但绝不能是因为互斥。
|
||||||
|
external := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(external, baselineRequest(`{"scope":"external"}`, roleAdmin, "admin"), config)
|
||||||
|
if external.Code == http.StatusAccepted {
|
||||||
|
t.Fatalf("external without a target KB must not be accepted: %s", external.Body.String())
|
||||||
|
}
|
||||||
|
body := decodeBody(t, external)
|
||||||
|
if body["code"] == "library_write_active" {
|
||||||
|
t.Fatalf("external must not be blocked by the internal scope's mutex: %v", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 建立目标权威基线的入口契约:
|
||||||
|
// - 只有管理员能发起(它是生产侧维护动作);
|
||||||
|
// - 已有生效版本时拒绝(基线只解决首次发布);
|
||||||
|
// - 目标读不到时拒绝,且**不建作业、不登记版本**(不能用基线掩盖连接问题)。
|
||||||
|
|
||||||
|
func baselineRequest(body, role, username string) *http.Request {
|
||||||
|
request := httptest.NewRequest(http.MethodPost, "/api/v1/library/baselines", strings.NewReader(body))
|
||||||
|
request.Header.Set("Content-Type", "application/json")
|
||||||
|
ctx := context.WithValue(request.Context(), sessionKey{}, sessionIdentity{Username: username, Role: role})
|
||||||
|
return request.WithContext(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeBody(t *testing.T, recorder *httptest.ResponseRecorder) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
var payload map[string]any
|
||||||
|
if err := json.Unmarshal(recorder.Body.Bytes(), &payload); err != nil {
|
||||||
|
t.Fatalf("response is not JSON: %s", recorder.Body.String())
|
||||||
|
}
|
||||||
|
return payload
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLibraryBaselineRequiresAdmin(t *testing.T) {
|
||||||
|
store := &fakeLibraryStore{}
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(recorder, baselineRequest(`{"scope":"internal"}`, roleEngineer, "eng"), Config{Store: store})
|
||||||
|
if recorder.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
if store.createN != 0 {
|
||||||
|
t.Fatal("non-admin must not create a baseline job")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLibraryBaselineRefusesWhenEffectiveVersionExists(t *testing.T) {
|
||||||
|
store := &fakeLibraryStore{versions: []governance.LibraryVersion{{
|
||||||
|
ID: "libver-1", Scope: "internal", Kind: governance.LibraryVersionProductionEffective,
|
||||||
|
SnapshotID: "snap-effective", CreatedAt: time.Now().UTC(),
|
||||||
|
}}}
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(recorder, baselineRequest(`{"scope":"internal"}`, roleAdmin, "admin"), Config{Store: store})
|
||||||
|
if recorder.Code != http.StatusConflict {
|
||||||
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
if body := decodeBody(t, recorder); body["code"] != "library_baseline_exists" || body["snapshot_id"] != "snap-effective" {
|
||||||
|
t.Fatalf("body=%v", body)
|
||||||
|
}
|
||||||
|
if store.createN != 0 {
|
||||||
|
t.Fatal("existing effective version must not create another baseline job")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 目标读不到(这里用"没配置目标库"复现):必须 503 并说明原因,
|
||||||
|
// 既不能建作业,也不能顺手写一条版本记录来"解决"未知状态。
|
||||||
|
func TestLibraryBaselineRefusesWhenTargetUnreadable(t *testing.T) {
|
||||||
|
store := &fakeLibraryStore{}
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
libraryBaselineEnqueue(recorder, baselineRequest(`{"scope":"external"}`, roleAdmin, "admin"), Config{Store: store})
|
||||||
|
if recorder.Code != http.StatusServiceUnavailable {
|
||||||
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
body := decodeBody(t, recorder)
|
||||||
|
if body["code"] != "library_baseline_unprobeable" || !strings.Contains(func() string {
|
||||||
|
text, _ := body["error"].(string)
|
||||||
|
return text
|
||||||
|
}(), "无法读取目标整库状态") {
|
||||||
|
t.Fatalf("body=%v", body)
|
||||||
|
}
|
||||||
|
if store.createN != 0 {
|
||||||
|
t.Fatal("unreadable target must not create a baseline job")
|
||||||
|
}
|
||||||
|
if len(store.versions) != 0 {
|
||||||
|
t.Fatal("unreadable target must not record any version")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 测试候选只认登记过 test_sealed 的那份快照:归档里同时存在更新的目标基线/
|
||||||
|
// 预备份快照时,绝不能把它们当成"拟发布的测试版本"。
|
||||||
|
func TestTestSealedCandidateComesFromRegistryNotNewestArchiveEntry(t *testing.T) {
|
||||||
|
sealed := librarySnapshotView{SnapshotID: "snap-sealed", CreatedAt: time.Date(2026, 10, 29, 8, 54, 0, 0, time.UTC)}
|
||||||
|
prebackup := librarySnapshotView{SnapshotID: "snap-prebackup", CreatedAt: time.Date(2026, 10, 29, 9, 30, 0, 0, time.UTC)}
|
||||||
|
byID := map[string]librarySnapshotView{sealed.SnapshotID: sealed, prebackup.SnapshotID: prebackup}
|
||||||
|
versions := []governance.LibraryVersion{
|
||||||
|
{Kind: governance.LibraryVersionProductionPrebackup, SnapshotID: "snap-prebackup"},
|
||||||
|
{Kind: governance.LibraryVersionTestSealed, SnapshotID: "snap-sealed"},
|
||||||
|
}
|
||||||
|
candidate, reason := testSealedOf(versions, byID)
|
||||||
|
if reason != "" {
|
||||||
|
t.Fatalf("reason=%q", reason)
|
||||||
|
}
|
||||||
|
if candidate == nil || candidate.SnapshotID != "snap-sealed" {
|
||||||
|
t.Fatalf("candidate=%v, want the registered test seal", candidate)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 没有任何 test_sealed 登记:宁可说"还没有测试封存",也不拿别的快照顶替。
|
||||||
|
none, reason := testSealedOf([]governance.LibraryVersion{
|
||||||
|
{Kind: governance.LibraryVersionProductionPrebackup, SnapshotID: "snap-prebackup"},
|
||||||
|
}, byID)
|
||||||
|
if none != nil || !strings.Contains(reason, "还没有该目标的测试封存") {
|
||||||
|
t.Fatalf("candidate=%v reason=%q", none, reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 登记过但它已经不在本机归档:说清是归档问题,不给候选。
|
||||||
|
missing, reason := testSealedOf([]governance.LibraryVersion{
|
||||||
|
{Kind: governance.LibraryVersionTestSealed, SnapshotID: "snap-gone"},
|
||||||
|
}, byID)
|
||||||
|
if missing != nil || !strings.Contains(reason, "不在本机归档里") {
|
||||||
|
t.Fatalf("candidate=%v reason=%q", missing, reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 非空但无权威版本时,缺项必须指向真实入口,而不是一句"未知"。
|
||||||
|
func TestLibrarySyncGateNamesTheBaselineEntryForNonEmptyTarget(t *testing.T) {
|
||||||
|
gate := librarySyncGateWithGrant(Config{ProductionReleaseRequestsEnabled: true,
|
||||||
|
ProductionBackupRestoreVerified: true, LibraryProductionDatabaseURL: "postgres://prod.invalid/db",
|
||||||
|
LibraryProductionInternalKB: "kb-int", LibraryCapacityBudgetBytes: 1 << 30},
|
||||||
|
libraryKnownView{State: "unknown"}, libraryKnownView{State: "unknown"}, "snap-1", true)
|
||||||
|
if gate.SyncAllowed {
|
||||||
|
t.Fatal("unknown production version must block the sync")
|
||||||
|
}
|
||||||
|
joined := strings.Join(gate.Reasons, "|")
|
||||||
|
if !strings.Contains(joined, "目标非空且没有生效版本记录") || !strings.Contains(joined, "目标权威基线") {
|
||||||
|
t.Fatalf("reasons=%v", gate.Reasons)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 维护能力是配置事实:配置齐全时必须说"已接线",缺项要列出缺哪个变量。
|
||||||
|
func TestLibraryMaintenanceCapabilityReportsWiringInsteadOfOneVagueSentence(t *testing.T) {
|
||||||
|
wired := libraryMaintenanceCapability(Config{
|
||||||
|
LibraryTargetInstance: "kgseal-app", LibraryTargetProject: "kgseal-weknora",
|
||||||
|
LibraryTargetWriters: []string{"app"}, LibraryTargetIngressURL: "http://kgseal-ingress:18084/health",
|
||||||
|
LibraryTargetIngressService: "ingress", LibraryTargetJournalRoot: "/var/lib/knowledge-governance/journal",
|
||||||
|
LibraryTargetDockerSocket: "/var/run/docker.sock",
|
||||||
|
})
|
||||||
|
if wired["configured"] != true {
|
||||||
|
t.Fatalf("capability=%v", wired)
|
||||||
|
}
|
||||||
|
if wired["instance"] != "kgseal-app" {
|
||||||
|
t.Fatalf("instance=%v", wired["instance"])
|
||||||
|
}
|
||||||
|
broken := libraryMaintenanceCapability(Config{LibraryTargetInstance: "kgseal-app"})
|
||||||
|
if broken["configured"] != false {
|
||||||
|
t.Fatalf("capability=%v", broken)
|
||||||
|
}
|
||||||
|
missing, _ := broken["missing"].([]string)
|
||||||
|
if len(missing) == 0 || !strings.Contains(strings.Join(missing, ","), "LIBRARY_TARGET_WRITERS") {
|
||||||
|
t.Fatalf("missing=%v", broken["missing"])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -26,13 +26,6 @@ type libraryCaptureRequest struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func libraryCaptureEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
|
func libraryCaptureEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
|
||||||
if config.LibraryAgentSocket == "" {
|
|
||||||
writeJSON(w, http.StatusServiceUnavailable, map[string]any{
|
|
||||||
"code": "library_capture_unwired",
|
|
||||||
"reason": "未接线:主机侧封存 agent 没有挂载,页面无法发起封存",
|
|
||||||
})
|
|
||||||
return
|
|
||||||
}
|
|
||||||
var request libraryCaptureRequest
|
var request libraryCaptureRequest
|
||||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&request); err != nil {
|
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&request); err != nil {
|
||||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid request body"})
|
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid request body"})
|
||||||
@@ -52,6 +45,17 @@ func libraryCaptureEnqueue(w http.ResponseWriter, r *http.Request, config Config
|
|||||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid job id"})
|
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid job id"})
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
// 请求形状先校验(400),再看执行器(503):不只看"配了 socket"——执行器可能
|
||||||
|
// 不可达、可能指向别的实例、可能降级。页面与接口必须用同一份判定,否则又会出现
|
||||||
|
// "点了才失败"或"封错库"。
|
||||||
|
if available, reason, executor := libraryExecutorAvailability(r.Context(), config); !available {
|
||||||
|
writeJSON(w, http.StatusServiceUnavailable, map[string]any{
|
||||||
|
"code": "library_capture_unavailable",
|
||||||
|
"reason": reason,
|
||||||
|
"executor": executor,
|
||||||
|
})
|
||||||
|
return
|
||||||
|
}
|
||||||
requestedBy := ""
|
requestedBy := ""
|
||||||
if identity, ok := sessionFrom(r.Context()); ok {
|
if identity, ok := sessionFrom(r.Context()); ok {
|
||||||
requestedBy = identity.Username
|
requestedBy = identity.Username
|
||||||
|
|||||||
@@ -17,31 +17,11 @@ func captureRequest(body string) *http.Request {
|
|||||||
return request
|
return request
|
||||||
}
|
}
|
||||||
|
|
||||||
// An unwired agent must refuse clearly instead of queueing a job nobody can run.
|
|
||||||
func TestLibraryCaptureRefusedWhenAgentUnwired(t *testing.T) {
|
|
||||||
store := &fakeJobStore{}
|
|
||||||
recorder := httptest.NewRecorder()
|
|
||||||
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal"}`), Config{Store: store})
|
|
||||||
if recorder.Code != http.StatusServiceUnavailable {
|
|
||||||
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
|
||||||
}
|
|
||||||
var body map[string]any
|
|
||||||
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if body["code"] != "library_capture_unwired" || body["reason"] == "" {
|
|
||||||
t.Fatalf("unexpected body %v", body)
|
|
||||||
}
|
|
||||||
if store.enqueued.Type != "" {
|
|
||||||
t.Fatal("unwired agent still enqueued a job")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLibraryCaptureEnqueuesFixedJobShape(t *testing.T) {
|
func TestLibraryCaptureEnqueuesFixedJobShape(t *testing.T) {
|
||||||
store := &fakeJobStore{enqueueEchoRequestedID: true}
|
store := &fakeJobStore{enqueueEchoRequestedID: true}
|
||||||
recorder := httptest.NewRecorder()
|
recorder := httptest.NewRecorder()
|
||||||
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"external","job":"lib-external-review"}`),
|
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"external","job":"lib-external-review"}`),
|
||||||
Config{Store: store, LibraryAgentSocket: "/run/library-agent/agent.sock"})
|
captureReadyConfig(store))
|
||||||
if recorder.Code != http.StatusAccepted {
|
if recorder.Code != http.StatusAccepted {
|
||||||
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
}
|
}
|
||||||
@@ -76,7 +56,7 @@ func TestLibraryCaptureRejectsInvalidRequest(t *testing.T) {
|
|||||||
} {
|
} {
|
||||||
store := &fakeJobStore{}
|
store := &fakeJobStore{}
|
||||||
recorder := httptest.NewRecorder()
|
recorder := httptest.NewRecorder()
|
||||||
libraryCaptureEnqueue(recorder, captureRequest(body), Config{Store: store, LibraryAgentSocket: "/tmp/agent.sock"})
|
libraryCaptureEnqueue(recorder, captureRequest(body), captureReadyConfig(store))
|
||||||
if recorder.Code != http.StatusBadRequest {
|
if recorder.Code != http.StatusBadRequest {
|
||||||
t.Fatalf("body %q status=%d", body, recorder.Code)
|
t.Fatalf("body %q status=%d", body, recorder.Code)
|
||||||
}
|
}
|
||||||
@@ -92,8 +72,7 @@ func TestLibraryCaptureQueueIDFitsTheColumn(t *testing.T) {
|
|||||||
for _, scope := range []string{"internal", "external"} {
|
for _, scope := range []string{"internal", "external"} {
|
||||||
store := &fakeJobStore{enqueueEchoRequestedID: true}
|
store := &fakeJobStore{enqueueEchoRequestedID: true}
|
||||||
recorder := httptest.NewRecorder()
|
recorder := httptest.NewRecorder()
|
||||||
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"`+scope+`"}`),
|
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"`+scope+`"}`), captureReadyConfig(store))
|
||||||
Config{Store: store, LibraryAgentSocket: "/tmp/agent.sock"})
|
|
||||||
if recorder.Code != http.StatusAccepted {
|
if recorder.Code != http.StatusAccepted {
|
||||||
t.Fatalf("%s status=%d body=%s", scope, recorder.Code, recorder.Body.String())
|
t.Fatalf("%s status=%d body=%s", scope, recorder.Code, recorder.Body.String())
|
||||||
}
|
}
|
||||||
@@ -146,7 +125,7 @@ func TestLibraryCaptureRejectsScopeConflictForReusedJob(t *testing.T) {
|
|||||||
Payload: json.RawMessage(`{"scope":"internal","capture_job":"lib-shared","requested_by":"yaozhanghao"}`)}}
|
Payload: json.RawMessage(`{"scope":"internal","capture_job":"lib-shared","requested_by":"yaozhanghao"}`)}}
|
||||||
recorder := httptest.NewRecorder()
|
recorder := httptest.NewRecorder()
|
||||||
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"external","job":"lib-shared"}`),
|
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"external","job":"lib-shared"}`),
|
||||||
Config{Store: store, LibraryAgentSocket: "/run/library-agent/agent.sock"})
|
captureReadyConfig(store))
|
||||||
if recorder.Code != http.StatusConflict {
|
if recorder.Code != http.StatusConflict {
|
||||||
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
}
|
}
|
||||||
@@ -166,7 +145,7 @@ func TestLibraryCaptureReplayReportsStoredJob(t *testing.T) {
|
|||||||
Payload: json.RawMessage(`{"scope":"internal","capture_job":"lib-shared","requested_by":"yaozhanghao"}`)}}
|
Payload: json.RawMessage(`{"scope":"internal","capture_job":"lib-shared","requested_by":"yaozhanghao"}`)}}
|
||||||
recorder := httptest.NewRecorder()
|
recorder := httptest.NewRecorder()
|
||||||
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal","job":"lib-shared"}`),
|
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal","job":"lib-shared"}`),
|
||||||
Config{Store: store, LibraryAgentSocket: "/run/library-agent/agent.sock"})
|
captureReadyConfig(store))
|
||||||
if recorder.Code != http.StatusAccepted {
|
if recorder.Code != http.StatusAccepted {
|
||||||
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,77 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/libraryexecutor"
|
||||||
|
)
|
||||||
|
|
||||||
|
// libraryExecutorView 是"封存执行器到底能不能用"的页面口径。
|
||||||
|
//
|
||||||
|
// 为什么不是 `LIBRARY_AGENT_SOCKET != ""`:正式栈里 API 根本没有挂载那个 socket,
|
||||||
|
// 配置非空并不代表链路通;更危险的是 agent 可能顶着正式单元名却在冻结另一个实例。
|
||||||
|
// 这里只认 worker 写入的真实观测,并把观测时间与执行器自报的身份一起给出来。
|
||||||
|
type libraryExecutorView struct {
|
||||||
|
Available bool `json:"available"`
|
||||||
|
Reason string `json:"reason,omitempty"`
|
||||||
|
Instance string `json:"instance,omitempty"`
|
||||||
|
Project string `json:"project,omitempty"`
|
||||||
|
SourceApp string `json:"source_app,omitempty"`
|
||||||
|
ArchiveRoot string `json:"archive_root,omitempty"`
|
||||||
|
Busy bool `json:"busy"`
|
||||||
|
Degraded string `json:"degraded,omitempty"`
|
||||||
|
ObservedAt *time.Time `json:"observed_at,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type libraryExecutorStore interface {
|
||||||
|
LibraryExecutorStatus(ctx context.Context, instance string) (*libraryexecutor.Observation, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// libraryExecutorAvailability 把"配置里的期望"与"worker 观测到的真实身份"合起来判定:
|
||||||
|
// 观测不存在、过期、不可达、实例不符、降级、正忙,任何一条都不可用,并给出原因。
|
||||||
|
func libraryExecutorAvailability(ctx context.Context, config Config) (bool, string, libraryExecutorView) {
|
||||||
|
// 期望身份是**封存源**:agent 冻结的是源库写入者,与同步目标(LIBRARY_TARGET_*)
|
||||||
|
// 本来就是两套名字(本机候选栈:源 kgmaint、目标 kgseal)。
|
||||||
|
instance := config.LibrarySourceInstance
|
||||||
|
view := libraryExecutorView{Instance: instance}
|
||||||
|
if instance == "" {
|
||||||
|
return false, "未配置封存来源实例身份(LIBRARY_SOURCE_INSTANCE),无法确认执行器归属", view
|
||||||
|
}
|
||||||
|
store, ok := config.Store.(libraryExecutorStore)
|
||||||
|
if !ok {
|
||||||
|
return false, "执行器状态存储不可用", view
|
||||||
|
}
|
||||||
|
observation, err := store.LibraryExecutorStatus(ctx, instance)
|
||||||
|
if err != nil {
|
||||||
|
return false, "读取执行器状态失败:" + err.Error(), view
|
||||||
|
}
|
||||||
|
if observation == nil {
|
||||||
|
return false, "封存执行器还没有上报状态(worker 未运行,或实例身份与 worker 不一致)", view
|
||||||
|
}
|
||||||
|
available, reason := libraryexecutor.Evaluate(*observation,
|
||||||
|
libraryexecutor.Expectation{Instance: instance, Project: config.LibrarySourceProject,
|
||||||
|
ArchiveRoot: config.LibraryArchiveRoot},
|
||||||
|
executorStatusMaxAge(config), time.Now().UTC())
|
||||||
|
observedAt := observation.ObservedAt
|
||||||
|
view.Available = available
|
||||||
|
view.Reason = reason
|
||||||
|
view.Project = observation.Identity.Project
|
||||||
|
view.SourceApp = observation.Identity.SourceApp
|
||||||
|
view.ArchiveRoot = observation.Identity.ArchiveRoot
|
||||||
|
view.Busy = observation.Busy
|
||||||
|
view.Degraded = observation.Degraded
|
||||||
|
view.ObservedAt = &observedAt
|
||||||
|
return available, reason, view
|
||||||
|
}
|
||||||
|
|
||||||
|
// defaultLibraryExecutorStatusMaxAge 与 worker 的探测间隔(30s)配套:
|
||||||
|
// 没配上限绝不等于"永不过期"——那会把一次陈旧的"可用"永久留在页面上。
|
||||||
|
const defaultLibraryExecutorStatusMaxAge = 2 * time.Minute
|
||||||
|
|
||||||
|
func executorStatusMaxAge(config Config) time.Duration {
|
||||||
|
if config.LibraryExecutorStatusMaxAge > 0 {
|
||||||
|
return config.LibraryExecutorStatusMaxAge
|
||||||
|
}
|
||||||
|
return defaultLibraryExecutorStatusMaxAge
|
||||||
|
}
|
||||||
@@ -0,0 +1,184 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/libraryexecutor"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeJobStore 只是"存储"替身;执行器状态同样是它的一条记录。
|
||||||
|
func (s *fakeJobStore) LibraryExecutorStatus(_ context.Context, instance string) (*libraryexecutor.Observation, error) {
|
||||||
|
if s.executorStatus == nil || s.executorStatus.Instance != instance {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return s.executorStatus, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// healthyExecutor 返回一份"worker 刚观测到、指向本实例"的记录。
|
||||||
|
func healthyExecutor(now time.Time) *libraryexecutor.Observation {
|
||||||
|
return &libraryexecutor.Observation{
|
||||||
|
Instance: "kgmaint-weknora", Reachable: true, ObservedAt: now,
|
||||||
|
Identity: libraryexecutor.Identity{Project: "kgmaint-weknora", SourceApp: "kgmaint-app",
|
||||||
|
ArchiveRoot: "/tmp/kg-e2e/archive", Writers: "app"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// captureReadyConfig 是"执行器真的可用"时的配置:实例、期望项目与观测三者一致。
|
||||||
|
func captureReadyConfig(store *fakeJobStore) Config {
|
||||||
|
store.executorStatus = healthyExecutor(time.Now().UTC())
|
||||||
|
return Config{
|
||||||
|
Store: store,
|
||||||
|
LibraryAgentSocket: "/run/library-agent/agent.sock",
|
||||||
|
LibrarySourceInstance: "kgmaint-weknora",
|
||||||
|
LibrarySourceProject: "kgmaint-weknora",
|
||||||
|
LibraryArchiveRoot: "/var/lib/knowledge-governance/library",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeCaptureRefusal(t *testing.T, recorder *httptest.ResponseRecorder) (string, string) {
|
||||||
|
t.Helper()
|
||||||
|
var body struct {
|
||||||
|
Code string `json:"code"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
|
||||||
|
t.Fatalf("body %s: %v", recorder.Body.String(), err)
|
||||||
|
}
|
||||||
|
return body.Code, body.Reason
|
||||||
|
}
|
||||||
|
|
||||||
|
// 页面上的"可发起"必须等于接口真的会接受:配置了 socket 但执行器不可达、
|
||||||
|
// 指向别的实例、降级或过期时,都必须带着原因拒绝,而不是先排队再失败。
|
||||||
|
func TestLibraryCaptureRefusesWhenExecutorIsNotVerified(t *testing.T) {
|
||||||
|
now := time.Now().UTC()
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
config func(*Config)
|
||||||
|
text string
|
||||||
|
}{
|
||||||
|
{name: "no instance identity", config: func(c *Config) {
|
||||||
|
c.LibrarySourceInstance = ""
|
||||||
|
}, text: "LIBRARY_SOURCE_INSTANCE"},
|
||||||
|
{name: "nothing reported yet", config: func(c *Config) {
|
||||||
|
c.Store.(*fakeJobStore).executorStatus = nil
|
||||||
|
}, text: "还没有上报状态"},
|
||||||
|
{name: "unreachable", config: func(c *Config) {
|
||||||
|
c.Store.(*fakeJobStore).executorStatus.Reachable = false
|
||||||
|
c.Store.(*fakeJobStore).executorStatus.Error = "library agent is unreachable"
|
||||||
|
}, text: "不可达"},
|
||||||
|
{name: "stale observation", config: func(c *Config) {
|
||||||
|
c.Store.(*fakeJobStore).executorStatus.ObservedAt = now.Add(-10 * time.Minute)
|
||||||
|
}, text: "已过期"},
|
||||||
|
{name: "points at another instance", config: func(c *Config) {
|
||||||
|
c.Store.(*fakeJobStore).executorStatus.Identity.Project = "weknora071full"
|
||||||
|
}, text: "指向的实例不匹配"},
|
||||||
|
{name: "degraded", config: func(c *Config) {
|
||||||
|
c.Store.(*fakeJobStore).executorStatus.Degraded = "写入者恢复失败"
|
||||||
|
}, text: "降级"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
store := &fakeJobStore{}
|
||||||
|
config := captureReadyConfig(store)
|
||||||
|
test.config(&config)
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
libraryCaptureEnqueue(recorder, captureRequest(`{"scope":"internal"}`), config)
|
||||||
|
if recorder.Code != http.StatusServiceUnavailable {
|
||||||
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
code, reason := decodeCaptureRefusal(t, recorder)
|
||||||
|
if code != "library_capture_unavailable" {
|
||||||
|
t.Fatalf("code=%q body=%s", code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
if reason == "" || !strings.Contains(reason, test.text) {
|
||||||
|
t.Fatalf("reason=%q want it to mention %q", reason, test.text)
|
||||||
|
}
|
||||||
|
if store.enqueued.Type != "" {
|
||||||
|
t.Fatal("refused capture still enqueued a job")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The overview must report the same reason the capture endpoint would use.
|
||||||
|
func TestLibraryOverviewReportsExecutorIdentityAndReason(t *testing.T) {
|
||||||
|
store := &fakeLibraryStore{}
|
||||||
|
config := captureReadyConfig(&store.fakeJobStore)
|
||||||
|
// libraryJobStore 的方法在 fakeLibraryStore 上,执行器状态在嵌入的 fakeJobStore 上。
|
||||||
|
config.Store = store
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
libraryOverviewHandler(recorder, httptest.NewRequest(http.MethodGet, "/api/v1/library/overview?scope=internal", nil), config)
|
||||||
|
if recorder.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
CaptureAvailable bool `json:"capture_available"`
|
||||||
|
Executor struct {
|
||||||
|
Available bool `json:"available"`
|
||||||
|
Instance string `json:"instance"`
|
||||||
|
Project string `json:"project"`
|
||||||
|
SourceApp string `json:"source_app"`
|
||||||
|
ArchiveRoot string `json:"archive_root"`
|
||||||
|
} `json:"executor"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !body.CaptureAvailable || !body.Executor.Available {
|
||||||
|
t.Fatalf("overview did not report an available executor: %s", recorder.Body.String())
|
||||||
|
}
|
||||||
|
if body.Executor.Project != "kgmaint-weknora" || body.Executor.SourceApp != "kgmaint-app" ||
|
||||||
|
body.Executor.ArchiveRoot != "/tmp/kg-e2e/archive" || body.Executor.Instance != "kgmaint-weknora" {
|
||||||
|
t.Fatalf("executor view = %+v", body.Executor)
|
||||||
|
}
|
||||||
|
|
||||||
|
store.fakeJobStore.executorStatus.Identity.Project = "weknora071full"
|
||||||
|
recorder = httptest.NewRecorder()
|
||||||
|
libraryOverviewHandler(recorder, httptest.NewRequest(http.MethodGet, "/api/v1/library/overview?scope=internal", nil), config)
|
||||||
|
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if body.CaptureAvailable {
|
||||||
|
t.Fatalf("overview still says available after the executor moved to another instance: %s", recorder.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// "恢复至该版本"必须由真实条件决定:没有 restore 授权就明确说出来。
|
||||||
|
func TestLibraryRestoreAvailabilityExplainsMissingAuthorization(t *testing.T) {
|
||||||
|
store := &fakeJobStore{}
|
||||||
|
config := Config{Store: store, ProductionReleaseRequestsEnabled: true, ProductionBackupRestoreVerified: true}
|
||||||
|
recorder := httptest.NewRecorder()
|
||||||
|
libraryRestoreAvailability(recorder, httptest.NewRequest(http.MethodGet,
|
||||||
|
"/api/v1/library/restore-availability?scope=internal&snapshot_id=snap-1", nil), config)
|
||||||
|
if recorder.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status=%d body=%s", recorder.Code, recorder.Body.String())
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
Allowed bool `json:"allowed"`
|
||||||
|
Reasons []string `json:"reasons"`
|
||||||
|
Authorization struct {
|
||||||
|
State string `json:"state"`
|
||||||
|
} `json:"authorization"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(recorder.Body.Bytes(), &body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if body.Allowed || body.Authorization.State != "none" || len(body.Reasons) == 0 {
|
||||||
|
t.Fatalf("body=%s", recorder.Body.String())
|
||||||
|
}
|
||||||
|
found := false
|
||||||
|
for _, reason := range body.Reasons {
|
||||||
|
if strings.Contains(reason, "恢复授权") {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("reasons = %v", body.Reasons)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -19,6 +19,7 @@ import (
|
|||||||
"knowledge-governance/internal/librarydiff"
|
"knowledge-governance/internal/librarydiff"
|
||||||
"knowledge-governance/internal/librarysnapshot"
|
"knowledge-governance/internal/librarysnapshot"
|
||||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
|
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||||
)
|
)
|
||||||
|
|
||||||
// 整库版本/状态/差异读取契约 + 生产同步的 fail-closed 入口。
|
// 整库版本/状态/差异读取契约 + 生产同步的 fail-closed 入口。
|
||||||
@@ -56,30 +57,36 @@ type libraryKnownView struct {
|
|||||||
EmptyBaseline bool `json:"empty_baseline,omitempty"`
|
EmptyBaseline bool `json:"empty_baseline,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// productionScopeEmpty 只读探测生产目标:真正连库数一遍,不用缓存或推断。
|
// productionScopeState 只读探测生产目标:真正连库数一遍,不用缓存或推断。
|
||||||
func productionScopeEmpty(ctx context.Context, config Config, scope string) (bool, error) {
|
//
|
||||||
|
// "空"必须是整库意义的空:只数 knowledges 会把"文档已删、分块/Wiki/资源还在"
|
||||||
|
// 的目标当成首次发布的空基线,从而绕过基线初始化。所以这里用适配层同一份
|
||||||
|
// 表清单与范围过滤(v071.ScopeContentEmpty),并把逐表行数交回调用方作为证据。
|
||||||
|
func productionScopeState(ctx context.Context, config Config, scope string) (bool, map[string]int64, error) {
|
||||||
if strings.TrimSpace(config.LibraryProductionDatabaseURL) == "" {
|
if strings.TrimSpace(config.LibraryProductionDatabaseURL) == "" {
|
||||||
return false, errors.New("生产整库目标未配置")
|
return false, nil, errors.New("生产整库目标未配置")
|
||||||
}
|
}
|
||||||
kb := config.LibraryProductionInternalKB
|
kb := config.LibraryProductionInternalKB
|
||||||
if scope == "external" {
|
if scope == "external" {
|
||||||
kb = config.LibraryProductionExternalKB
|
kb = config.LibraryProductionExternalKB
|
||||||
}
|
}
|
||||||
if kb == "" {
|
if kb == "" {
|
||||||
return false, errors.New("生产知识库未配置")
|
return false, nil, errors.New("该范围的目标知识库未配置")
|
||||||
}
|
}
|
||||||
pool, err := pgxpool.New(ctx, config.LibraryProductionDatabaseURL)
|
pool, err := pgxpool.New(ctx, config.LibraryProductionDatabaseURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, err
|
return false, nil, err
|
||||||
}
|
}
|
||||||
defer pool.Close()
|
defer pool.Close()
|
||||||
var count int64
|
return v071.ScopeContentEmpty(ctx, pool,
|
||||||
err = pool.QueryRow(ctx, `SELECT count(*) FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2 AND deleted_at IS NULL`,
|
v071.ProductionScope{TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: kb})
|
||||||
config.LibraryProductionTenantID, kb).Scan(&count)
|
}
|
||||||
if err != nil {
|
|
||||||
return false, err
|
// productionScopeEmpty 保留"只问空不空"的调用形状;逐表行数在需要举证时用
|
||||||
}
|
// productionScopeState 取。
|
||||||
return count == 0, nil
|
func productionScopeEmpty(ctx context.Context, config Config, scope string) (bool, error) {
|
||||||
|
empty, _, err := productionScopeState(ctx, config, scope)
|
||||||
|
return empty, err
|
||||||
}
|
}
|
||||||
|
|
||||||
type libraryDiffView struct {
|
type libraryDiffView struct {
|
||||||
@@ -307,6 +314,7 @@ type libraryOverview struct {
|
|||||||
Authorization libraryAuthorizationView `json:"authorization"`
|
Authorization libraryAuthorizationView `json:"authorization"`
|
||||||
CaptureAvailable bool `json:"capture_available"`
|
CaptureAvailable bool `json:"capture_available"`
|
||||||
CaptureReason string `json:"capture_reason,omitempty"`
|
CaptureReason string `json:"capture_reason,omitempty"`
|
||||||
|
Executor libraryExecutorView `json:"executor"`
|
||||||
Now time.Time `json:"now"`
|
Now time.Time `json:"now"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -507,6 +515,70 @@ func errString(err error) string {
|
|||||||
return err.Error()
|
return err.Error()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// shortSnapshot 只用于给人看的提示:完整摘要留给快照详情,别在原因句里塞 64 位。
|
||||||
|
func shortSnapshot(id string) string {
|
||||||
|
if len(id) <= 12 {
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
return id[:12]
|
||||||
|
}
|
||||||
|
|
||||||
|
// testSealedOf 选"拟发布的测试封存":只有登记过 test_sealed 的那一份才算候选。
|
||||||
|
//
|
||||||
|
// 归档是共享的:生产预备份与目标基线快照同样带着 scope 写进同一个归档根,
|
||||||
|
// 按"这个 scope 最新一份"挑会把目标自己的内容当成测试候选(页面会显示一份
|
||||||
|
// 永远不该被发布的东西)。所以候选来自版本登记,再用归档确认它真的还在本地。
|
||||||
|
func testSealedOf(versions []governance.LibraryVersion, byID map[string]librarySnapshotView) (*librarySnapshotView, string) {
|
||||||
|
for _, item := range versions {
|
||||||
|
if item.Kind != governance.LibraryVersionTestSealed {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
snapshot, ok := byID[item.SnapshotID]
|
||||||
|
if !ok {
|
||||||
|
return nil, "最近一次测试封存 " + shortSnapshot(item.SnapshotID) + " 不在本机归档里(归档被清理或换了归档根)"
|
||||||
|
}
|
||||||
|
return &snapshot, ""
|
||||||
|
}
|
||||||
|
return nil, "还没有该目标的测试封存"
|
||||||
|
}
|
||||||
|
|
||||||
|
// libraryMaintenanceCapability 用配置事实回答"这台机器上维护是否接线"。
|
||||||
|
// 它刻意只说配置:能停写入者/关入口/证明排空的真实能力在 worker 侧(docker socket、
|
||||||
|
// 闸门、排空文件),API 若声称"能力可用"就是拿配置字符串冒充执行证据。
|
||||||
|
func libraryMaintenanceCapability(config Config) map[string]any {
|
||||||
|
missing := make([]string, 0, 6)
|
||||||
|
if config.LibraryTargetInstance == "" {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_INSTANCE")
|
||||||
|
}
|
||||||
|
if config.LibraryTargetProject == "" {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_PROJECT")
|
||||||
|
}
|
||||||
|
if len(config.LibraryTargetWriters) == 0 {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_WRITERS")
|
||||||
|
}
|
||||||
|
if config.LibraryTargetIngressURL == "" {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_INGRESS_URL")
|
||||||
|
}
|
||||||
|
if config.LibraryTargetIngressService == "" {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_INGRESS_SERVICE")
|
||||||
|
}
|
||||||
|
if config.LibraryTargetJournalRoot == "" {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_JOURNAL_ROOT")
|
||||||
|
}
|
||||||
|
if config.LibraryTargetDockerSocket == "" {
|
||||||
|
missing = append(missing, "LIBRARY_TARGET_DOCKER_SOCKET")
|
||||||
|
}
|
||||||
|
return map[string]any{
|
||||||
|
"configured": len(missing) == 0,
|
||||||
|
"missing": missing,
|
||||||
|
"instance": config.LibraryTargetInstance,
|
||||||
|
"project": config.LibraryTargetProject,
|
||||||
|
"writers": config.LibraryTargetWriters,
|
||||||
|
"ingress_service": config.LibraryTargetIngressService,
|
||||||
|
"note": "配置齐全只说明治理侧接线完整;停写入者、关入口与排空由 worker 在作业内校验并执行",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func librarySyncGate(config Config, production libraryKnownView, prebackup libraryKnownView, target string) libraryGateView {
|
func librarySyncGate(config Config, production libraryKnownView, prebackup libraryKnownView, target string) libraryGateView {
|
||||||
return librarySyncGateWithGrant(config, production, prebackup, target, false)
|
return librarySyncGateWithGrant(config, production, prebackup, target, false)
|
||||||
}
|
}
|
||||||
@@ -530,9 +602,11 @@ func librarySyncGateWithGrant(config Config, production libraryKnownView, prebac
|
|||||||
gate.Reasons = append(gate.Reasons, "生产整库目标未配置(LIBRARY_PRODUCTION_DATABASE_URL),无法核验生产当前状态")
|
gate.Reasons = append(gate.Reasons, "生产整库目标未配置(LIBRARY_PRODUCTION_DATABASE_URL),无法核验生产当前状态")
|
||||||
}
|
}
|
||||||
// 首次发布:生产还没有"生效版本"记录,但只要生产目标可读且是空的,
|
// 首次发布:生产还没有"生效版本"记录,但只要生产目标可读且是空的,
|
||||||
// 这就是一个合法的"空基线",不应该被永久挡住。非空但无权威版本仍然阻断。
|
// 这就是一个合法的"空基线",不应该被永久挡住。非空但无权威版本仍然阻断,
|
||||||
|
// 指向真实可行的入口(在维护隔离下捕获目标整库并校验后登记基线),
|
||||||
|
// 而不是让人去手写版本或清空目标。
|
||||||
if production.State != "known" && !production.EmptyBaseline {
|
if production.State != "known" && !production.EmptyBaseline {
|
||||||
gate.Reasons = append(gate.Reasons, "生产当前权威版本未知,不能启动写入")
|
gate.Reasons = append(gate.Reasons, "目标非空且没有生效版本记录:先建立目标权威基线(维护隔离下捕获整库并逐类校验),或确认目标确实是空库")
|
||||||
}
|
}
|
||||||
if target == "" {
|
if target == "" {
|
||||||
gate.Reasons = append(gate.Reasons, "还没有可同步的测试封存版本")
|
gate.Reasons = append(gate.Reasons, "还没有可同步的测试封存版本")
|
||||||
@@ -563,28 +637,25 @@ func libraryOverviewHandler(w http.ResponseWriter, r *http.Request, config Confi
|
|||||||
}
|
}
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
overview := libraryOverview{Scope: scope, Jobs: []governance.LibraryJob{}, Now: now,
|
overview := libraryOverview{Scope: scope, Jobs: []governance.LibraryJob{}, Now: now,
|
||||||
CaptureAvailable: strings.TrimSpace(config.LibraryAgentSocket) != "",
|
Gate: libraryGateView{Reasons: []string{}},
|
||||||
Gate: libraryGateView{Reasons: []string{}},
|
Diff: libraryDiffView{State: "unknown", Reason: "生产与测试的文档级差异读取未接线", Added: []any{}, Updated: []any{}, Removed: []any{}}}
|
||||||
Diff: libraryDiffView{State: "unknown", Reason: "生产与测试的文档级差异读取未接线", Added: []any{}, Updated: []any{}, Removed: []any{}}}
|
// 可发起封存 = worker 真实探测到执行器可用且指向本实例;原因逐条给全。
|
||||||
if !overview.CaptureAvailable {
|
overview.CaptureAvailable, overview.CaptureReason, overview.Executor = libraryExecutorAvailability(r.Context(), config)
|
||||||
overview.CaptureReason = "主机侧封存 agent 未接线"
|
|
||||||
}
|
|
||||||
|
|
||||||
// 测试封存来自归档:它只是测试侧事实。
|
// 测试封存来自归档 + 版本登记:归档里**不止**测试封存——生产预备份与目标基线
|
||||||
|
// 快照同样带 scope,如果只按"这个 scope 最新一份"挑,页面就会把目标自己的内容
|
||||||
|
// 当成拟发布的测试候选。所以候选只认登记过 kind=test_sealed 的那一份。
|
||||||
snapshots, snapErr := readLibrarySnapshots(r.Context(), config.LibraryArchiveRoot)
|
snapshots, snapErr := readLibrarySnapshots(r.Context(), config.LibraryArchiveRoot)
|
||||||
|
byID := map[string]librarySnapshotView{}
|
||||||
|
archiveReadable := false
|
||||||
if snapErr != nil {
|
if snapErr != nil {
|
||||||
_, overview.TestSealedReason = libraryArchiveErrorStatus(snapErr, "归档不可读")
|
_, overview.TestSealedReason = libraryArchiveErrorStatus(snapErr, "归档不可读")
|
||||||
} else if !snapshots.Available {
|
} else if !snapshots.Available {
|
||||||
overview.TestSealedReason = snapshots.Reason
|
overview.TestSealedReason = snapshots.Reason
|
||||||
} else {
|
} else {
|
||||||
for index := range snapshots.Snapshots {
|
archiveReadable = true
|
||||||
if snapshots.Snapshots[index].Scope == scope {
|
for _, snapshot := range snapshots.Snapshots {
|
||||||
overview.TestSealed = &snapshots.Snapshots[index]
|
byID[snapshot.SnapshotID] = snapshot
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if overview.TestSealed == nil {
|
|
||||||
overview.TestSealedReason = "还没有该目标的测试封存"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -594,6 +665,9 @@ func libraryOverviewHandler(w http.ResponseWriter, r *http.Request, config Confi
|
|||||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot read library versions"})
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot read library versions"})
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if archiveReadable {
|
||||||
|
overview.TestSealed, overview.TestSealedReason = testSealedOf(versions, byID)
|
||||||
|
}
|
||||||
overview.Production = libraryKnownView{State: "unknown", Reason: "还没有生产验证通过的生效版本记录"}
|
overview.Production = libraryKnownView{State: "unknown", Reason: "还没有生产验证通过的生效版本记录"}
|
||||||
overview.Prebackup = libraryKnownView{State: "unknown", Reason: "本次目标快照还没有生产预备份"}
|
overview.Prebackup = libraryKnownView{State: "unknown", Reason: "本次目标快照还没有生产预备份"}
|
||||||
target := ""
|
target := ""
|
||||||
|
|||||||
@@ -146,6 +146,64 @@ func libraryRestoreEnqueue(w http.ResponseWriter, r *http.Request, config Config
|
|||||||
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": false})
|
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": false})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// libraryRestoreAvailability 回答"现在能不能把生产恢复到这份快照"。
|
||||||
|
//
|
||||||
|
// 为什么需要它:页面原来把"恢复至该版本"按钮写死为 disabled,理由是"未接线"——
|
||||||
|
// 但接口、授权、恢复作业其实都已经在跑。按钮该由真实条件决定:双闸打开、
|
||||||
|
// 有这份快照的恢复授权、目标实例没有在维护。
|
||||||
|
type libraryRestoreAvailabilityView struct {
|
||||||
|
Allowed bool `json:"allowed"`
|
||||||
|
Reasons []string `json:"reasons"`
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
SnapshotID string `json:"snapshot_id"`
|
||||||
|
Authorization libraryAuthorizationView `json:"authorization"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func libraryRestoreAvailability(w http.ResponseWriter, r *http.Request, config Config) {
|
||||||
|
scope, err := scopeFromQuery(r)
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
snapshotID := strings.TrimSpace(r.URL.Query().Get("snapshot_id"))
|
||||||
|
if snapshotID == "" {
|
||||||
|
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "snapshot_id required"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
view := libraryRestoreAvailabilityView{Scope: scope, SnapshotID: snapshotID, Reasons: []string{}}
|
||||||
|
if !config.ProductionReleaseRequestsEnabled {
|
||||||
|
view.Reasons = append(view.Reasons, "生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false)")
|
||||||
|
}
|
||||||
|
if !config.ProductionBackupRestoreVerified {
|
||||||
|
view.Reasons = append(view.Reasons, "没有已核验的生产恢复演练证据(PRODUCTION_BACKUP_RESTORE_VERIFIED=false)")
|
||||||
|
}
|
||||||
|
digest, err := libraryRestoreParamsDigest(scope, snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot digest params"})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
grant, found := governance.LibraryReleaseAuthorization{}, false
|
||||||
|
if storeWithAuth, ok := config.Store.(interface {
|
||||||
|
ActiveLibraryAuthorization(ctx context.Context, scope, snapshotID, digest string, now time.Time) (governance.LibraryReleaseAuthorization, bool, error)
|
||||||
|
}); ok {
|
||||||
|
grant, found, err = storeWithAuth.ActiveLibraryAuthorization(r.Context(), scope, snapshotID, digest, time.Now().UTC())
|
||||||
|
if err != nil {
|
||||||
|
writeJSON(w, http.StatusServiceUnavailable, map[string]any{"allowed": false,
|
||||||
|
"reasons": []string{"读取恢复授权失败:" + err.Error()}})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
view.Authorization = authorizationViewOf(grant, found)
|
||||||
|
if !found {
|
||||||
|
view.Reasons = append(view.Reasons, "没有针对该快照的恢复授权:先签发 restore 授权(绑定同一快照)")
|
||||||
|
}
|
||||||
|
if _, held := instanceMaintenanceGuard(r.Context(), config, scope); held {
|
||||||
|
view.Reasons = append(view.Reasons, "目标实例正在维护中,等当前作业释放后再恢复")
|
||||||
|
}
|
||||||
|
view.Allowed = len(view.Reasons) == 0
|
||||||
|
writeJSON(w, http.StatusOK, view)
|
||||||
|
}
|
||||||
|
|
||||||
// 崩溃续作:失败作业可以按原参数重新排队,生成新作业(旧作业保留为审计记录)。
|
// 崩溃续作:失败作业可以按原参数重新排队,生成新作业(旧作业保留为审计记录)。
|
||||||
func libraryJobResume(w http.ResponseWriter, r *http.Request, config Config) {
|
func libraryJobResume(w http.ResponseWriter, r *http.Request, config Config) {
|
||||||
identity, ok := sessionFrom(r.Context())
|
identity, ok := sessionFrom(r.Context())
|
||||||
|
|||||||
@@ -85,15 +85,14 @@ const libraryCaptureNote = "封存由运维命令 scripts/library-capture.sh cap
|
|||||||
|
|
||||||
func librarySnapshotsList(w http.ResponseWriter, r *http.Request, config Config) {
|
func librarySnapshotsList(w http.ResponseWriter, r *http.Request, config Config) {
|
||||||
view, err := readLibrarySnapshots(r.Context(), config.LibraryArchiveRoot)
|
view, err := readLibrarySnapshots(r.Context(), config.LibraryArchiveRoot)
|
||||||
view.CaptureAvailable = config.LibraryAgentSocket != ""
|
// 能不能发起封存只认 worker 的真实观测,不认"配置里写了 socket 路径"。
|
||||||
if !view.CaptureAvailable {
|
view.CaptureAvailable, view.CaptureReason, _ = libraryExecutorAvailability(r.Context(), config)
|
||||||
view.CaptureReason = "未接线:主机侧封存 agent 没有挂载,页面只能查看已封存版本"
|
|
||||||
}
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
status, reason := libraryArchiveErrorStatus(err, "归档不可读")
|
status, reason := libraryArchiveErrorStatus(err, "归档不可读")
|
||||||
writeJSON(w, status, librarySnapshotList{Available: false, Reason: reason,
|
writeJSON(w, status, librarySnapshotList{Available: false, Reason: reason,
|
||||||
Snapshots: []librarySnapshotView{}, Verification: libraryVerificationManifest,
|
Snapshots: []librarySnapshotView{}, Verification: libraryVerificationManifest,
|
||||||
VerificationNote: libraryVerificationNote, CaptureNote: libraryCaptureNote})
|
VerificationNote: libraryVerificationNote, CaptureNote: libraryCaptureNote,
|
||||||
|
CaptureAvailable: view.CaptureAvailable, CaptureReason: view.CaptureReason})
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
writeJSON(w, http.StatusOK, view)
|
writeJSON(w, http.StatusOK, view)
|
||||||
|
|||||||
@@ -148,10 +148,24 @@ type Config struct {
|
|||||||
// LibraryTargetInstance 是实例级维护的目标实例名;留空即"未接线",
|
// LibraryTargetInstance 是实例级维护的目标实例名;留空即"未接线",
|
||||||
// 页面不能发起按预备份恢复,接口返回原因而不是假装能恢复。
|
// 页面不能发起按预备份恢复,接口返回原因而不是假装能恢复。
|
||||||
LibraryTargetInstance string
|
LibraryTargetInstance string
|
||||||
|
// 维护能力的其余配置项只用于如实展示"是否接线":未运行维护与没有维护能力
|
||||||
|
// 是两件事,页面不能混成一句"未接入"。真正停写入者/关入口的能力由 worker
|
||||||
|
// 在执行作业时校验(socket、入口、排空文件都在 worker 侧)。
|
||||||
|
LibraryTargetProject string
|
||||||
|
LibraryTargetWriters []string
|
||||||
|
LibraryTargetIngressURL string
|
||||||
|
LibraryTargetIngressService string
|
||||||
|
LibraryTargetDockerSocket string
|
||||||
|
LibraryTargetJournalRoot string
|
||||||
// LibrarySourceInstance 是死信解除记录的来源实例名(封存源),留空回落到目标实例名。
|
// LibrarySourceInstance 是死信解除记录的来源实例名(封存源),留空回落到目标实例名。
|
||||||
LibrarySourceInstance string
|
LibrarySourceInstance string
|
||||||
// LibraryAgentSocket 是主机侧封存 agent 的 unix socket;留空即页面不能发起封存。
|
// LibraryAgentSocket 是主机侧封存 agent 的 unix socket;留空即页面不能发起封存。
|
||||||
LibraryAgentSocket string
|
LibraryAgentSocket string
|
||||||
|
// LibrarySourceProject 是执行器应指向的实例名(agent -project,封存源)。
|
||||||
|
// 与它不一致时按"封错库"处理:不可发起,并说明差在哪。
|
||||||
|
LibrarySourceProject string
|
||||||
|
// LibraryExecutorStatusMaxAge 是执行器观测的有效期;超过即"无法确认"。
|
||||||
|
LibraryExecutorStatusMaxAge time.Duration
|
||||||
// LibraryCapacityBudgetBytes 是整库归档容量预算;0 = 未设置(容量门禁不放行)。
|
// LibraryCapacityBudgetBytes 是整库归档容量预算;0 = 未设置(容量门禁不放行)。
|
||||||
LibraryCapacityBudgetBytes int64
|
LibraryCapacityBudgetBytes int64
|
||||||
// 整库生产目标的只读探测配置:首次发布要能证明"生产可读且为空"。
|
// 整库生产目标的只读探测配置:首次发布要能证明"生产可读且为空"。
|
||||||
@@ -159,6 +173,9 @@ type Config struct {
|
|||||||
LibraryProductionInternalKB string
|
LibraryProductionInternalKB string
|
||||||
LibraryProductionExternalKB string
|
LibraryProductionExternalKB string
|
||||||
LibraryProductionTenantID uint64
|
LibraryProductionTenantID uint64
|
||||||
|
// LibraryScopeProbe 让测试注入"目标整库是否为空"的探测;留空即用真实连接。
|
||||||
|
// 它不是运行时开关:生产部署不设置它,那次探测仍然直连目标库逐表计数。
|
||||||
|
LibraryScopeProbe func(ctx context.Context, scope string) (bool, map[string]int64, error)
|
||||||
// LibraryRetentionVersions/Days 是保留策略;都为 0 = 未设置(不生成清理候选)。
|
// LibraryRetentionVersions/Days 是保留策略;都为 0 = 未设置(不生成清理候选)。
|
||||||
LibraryRetentionVersions int
|
LibraryRetentionVersions int
|
||||||
LibraryRetentionDays int
|
LibraryRetentionDays int
|
||||||
@@ -474,6 +491,8 @@ func NewHandler(config Config) (http.Handler, error) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
response := map[string]any{"held": held != nil && held.ReleasedAt == nil, "maintenance": held}
|
response := map[string]any{"held": held != nil && held.ReleasedAt == nil, "maintenance": held}
|
||||||
|
// 有没有维护能力是配置事实,和"此刻没有窗口"必须分开回答。
|
||||||
|
response["capability"] = libraryMaintenanceCapability(config)
|
||||||
// 实例级占用才是真正的互斥来源:跨 scope 也在这里体现。
|
// 实例级占用才是真正的互斥来源:跨 scope 也在这里体现。
|
||||||
if instanceStore, ok := config.Store.(interface {
|
if instanceStore, ok := config.Store.(interface {
|
||||||
InstanceMaintenanceOf(ctx context.Context, instance string) (*postgresstore.LibraryInstanceMaintenance, error)
|
InstanceMaintenanceOf(ctx context.Context, instance string) (*postgresstore.LibraryInstanceMaintenance, error)
|
||||||
@@ -510,6 +529,10 @@ func NewHandler(config Config) (http.Handler, error) {
|
|||||||
mux.HandleFunc("POST /api/v1/library/restores", func(w http.ResponseWriter, r *http.Request) {
|
mux.HandleFunc("POST /api/v1/library/restores", func(w http.ResponseWriter, r *http.Request) {
|
||||||
libraryRestoreEnqueue(w, r, config)
|
libraryRestoreEnqueue(w, r, config)
|
||||||
})
|
})
|
||||||
|
// 页面问"这份快照现在能不能恢复":条件来自双闸、授权与实例维护,而不是写死的按钮。
|
||||||
|
mux.HandleFunc("GET /api/v1/library/restore-availability", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
libraryRestoreAvailability(w, r, config)
|
||||||
|
})
|
||||||
// 失败/取消作业续作:按原参数重新排队,旧作业保留审计。
|
// 失败/取消作业续作:按原参数重新排队,旧作业保留审计。
|
||||||
mux.HandleFunc("POST /api/v1/library/jobs/{id}/resume", func(w http.ResponseWriter, r *http.Request) {
|
mux.HandleFunc("POST /api/v1/library/jobs/{id}/resume", func(w http.ResponseWriter, r *http.Request) {
|
||||||
libraryJobResume(w, r, config)
|
libraryJobResume(w, r, config)
|
||||||
@@ -518,6 +541,10 @@ func NewHandler(config Config) (http.Handler, error) {
|
|||||||
mux.HandleFunc("POST /api/v1/library/syncs", func(w http.ResponseWriter, r *http.Request) {
|
mux.HandleFunc("POST /api/v1/library/syncs", func(w http.ResponseWriter, r *http.Request) {
|
||||||
librarySyncEnqueue(w, r, config)
|
librarySyncEnqueue(w, r, config)
|
||||||
})
|
})
|
||||||
|
// 首次发布的目标权威基线:管理员在维护隔离下捕获目标整库并校验后登记版本。
|
||||||
|
mux.HandleFunc("POST /api/v1/library/baselines", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
libraryBaselineEnqueue(w, r, config)
|
||||||
|
})
|
||||||
// 全部 main 提交时间线(读本库缓存,不直接打 Gitea)
|
// 全部 main 提交时间线(读本库缓存,不直接打 Gitea)
|
||||||
mux.HandleFunc("GET /api/v1/version-history", func(w http.ResponseWriter, r *http.Request) {
|
mux.HandleFunc("GET /api/v1/version-history", func(w http.ResponseWriter, r *http.Request) {
|
||||||
versionHistoryList(w, r, config)
|
versionHistoryList(w, r, config)
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
|
|
||||||
"knowledge-governance/internal/governance"
|
"knowledge-governance/internal/governance"
|
||||||
"knowledge-governance/internal/jobs"
|
"knowledge-governance/internal/jobs"
|
||||||
|
"knowledge-governance/internal/libraryexecutor"
|
||||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -1501,30 +1502,32 @@ type fakeJobStore struct {
|
|||||||
activateSettingsErr error
|
activateSettingsErr error
|
||||||
// enqueueEchoRequestedID:幂等"首次入队"语义 —— 返回调用方请求的那个 id
|
// enqueueEchoRequestedID:幂等"首次入队"语义 —— 返回调用方请求的那个 id
|
||||||
// (默认 false 时返回 enqueueResult,即"复用已有任务"的路径)。
|
// (默认 false 时返回 enqueueResult,即"复用已有任务"的路径)。
|
||||||
enqueueEchoRequestedID bool
|
enqueueEchoRequestedID bool
|
||||||
listSettingsConnectionID string
|
listSettingsConnectionID string
|
||||||
listSettingsErr error
|
listSettingsErr error
|
||||||
settingsRevisions []governance.SourceConnectionRevision
|
settingsRevisions []governance.SourceConnectionRevision
|
||||||
jobFailureFilter governance.JobFailureFilter
|
jobFailureFilter governance.JobFailureFilter
|
||||||
jobFailurePage governance.JobFailurePage
|
jobFailurePage governance.JobFailurePage
|
||||||
jobFailureErr error
|
jobFailureErr error
|
||||||
retryBatchRequest *governance.RetryPublicationsBatchRequest
|
retryBatchRequest *governance.RetryPublicationsBatchRequest
|
||||||
retryBatchOperation string
|
retryBatchOperation string
|
||||||
retryBatchResult governance.RetryPublicationsBatchResult
|
retryBatchResult governance.RetryPublicationsBatchResult
|
||||||
cancelBatchRequest *governance.CancelPublicationsBatchRequest
|
cancelBatchRequest *governance.CancelPublicationsBatchRequest
|
||||||
cancelBatchOperation string
|
cancelBatchOperation string
|
||||||
cancelBatchResult governance.RetryPublicationsBatchResult
|
cancelBatchResult governance.RetryPublicationsBatchResult
|
||||||
cancelBatchReplayed bool
|
cancelBatchReplayed bool
|
||||||
cancelBatchErr error
|
cancelBatchErr error
|
||||||
retryBatchReplayed bool
|
retryBatchReplayed bool
|
||||||
retryBatchErr error
|
retryBatchErr error
|
||||||
requeueFailureRequest *governance.RequeueArtifactFetchRequest
|
requeueFailureRequest *governance.RequeueArtifactFetchRequest
|
||||||
requeueFailureOperation string
|
requeueFailureOperation string
|
||||||
requeueFailureResult governance.RequeueArtifactFetchResult
|
requeueFailureResult governance.RequeueArtifactFetchResult
|
||||||
requeueFailureReplayed bool
|
requeueFailureReplayed bool
|
||||||
requeueFailureErr error
|
requeueFailureErr error
|
||||||
enqueued jobs.EnqueueRequest
|
enqueued jobs.EnqueueRequest
|
||||||
enqueueResult *jobs.Job
|
enqueueResult *jobs.Job
|
||||||
|
// executorStatus 是封存执行器的"最近一次观测";nil 表示还没有任何上报。
|
||||||
|
executorStatus *libraryexecutor.Observation
|
||||||
listResult []jobs.Job
|
listResult []jobs.Job
|
||||||
listType string
|
listType string
|
||||||
listByTypeResult []jobs.Job
|
listByTypeResult []jobs.Job
|
||||||
|
|||||||
+117
-14
@@ -66,6 +66,9 @@ type request struct {
|
|||||||
// Released 只允许精确死信指纹(sha256:<64 hex>)。它不改变固定命令形态:
|
// Released 只允许精确死信指纹(sha256:<64 hex>)。它不改变固定命令形态:
|
||||||
// 还能传的只有"这些确切的行已由治理平台审计解除",CLI 会逐字节复核。
|
// 还能传的只有"这些确切的行已由治理平台审计解除",CLI 会逐字节复核。
|
||||||
Released []string `json:"released,omitempty"`
|
Released []string `json:"released,omitempty"`
|
||||||
|
// Expect 是调用方对执行器身份的期望。capture 必须在操作互斥内、任何源操作
|
||||||
|
// 之前比对它;缺失即拒绝(fail closed),不靠"事后核对"兜底。
|
||||||
|
Expect *Expectation `json:"expect,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type response struct {
|
type response struct {
|
||||||
@@ -89,13 +92,45 @@ const maxReleasedDeadLetters = 64
|
|||||||
type Agent struct {
|
type Agent struct {
|
||||||
config Config
|
config Config
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
|
// stateMu 只保护下面两个"给人看"的字段:健康应答不能被一次 15 分钟的封存
|
||||||
|
// 卡住(否则页面会把"正在封存"读成"执行器不可达")。
|
||||||
|
stateMu sync.RWMutex
|
||||||
// degraded is non-empty when a failed operation could not be followed by a
|
// degraded is non-empty when a failed operation could not be followed by a
|
||||||
// verified recovery. While set, capture requests are refused: leaving the
|
// verified recovery. While set, capture requests are refused: leaving the
|
||||||
// source writer stopped is worse than refusing work.
|
// source writer stopped is worse than refusing work.
|
||||||
degraded string
|
degraded string
|
||||||
|
busy bool
|
||||||
wg sync.WaitGroup
|
wg sync.WaitGroup
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// captureState 是健康应答要暴露的执行器状态:是否在跑、是否降级。
|
||||||
|
type captureState struct {
|
||||||
|
Busy bool `json:"busy"`
|
||||||
|
Degraded string `json:"degraded"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Agent) snapshotState() captureState {
|
||||||
|
a.stateMu.RLock()
|
||||||
|
defer a.stateMu.RUnlock()
|
||||||
|
return captureState{Busy: a.busy, Degraded: a.degraded}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Agent) setBusy(busy bool) {
|
||||||
|
a.stateMu.Lock()
|
||||||
|
defer a.stateMu.Unlock()
|
||||||
|
a.busy = busy
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Agent) setDegraded(reason string) {
|
||||||
|
a.stateMu.Lock()
|
||||||
|
defer a.stateMu.Unlock()
|
||||||
|
a.degraded = reason
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Agent) clearDegraded() {
|
||||||
|
a.setDegraded("")
|
||||||
|
}
|
||||||
|
|
||||||
func New(config Config) (*Agent, error) {
|
func New(config Config) (*Agent, error) {
|
||||||
if !filepath.IsAbs(config.Socket) || !filepath.IsAbs(config.ArchiveRoot) || !filepath.IsAbs(config.JournalRoot) {
|
if !filepath.IsAbs(config.Socket) || !filepath.IsAbs(config.ArchiveRoot) || !filepath.IsAbs(config.JournalRoot) {
|
||||||
return nil, errors.New("library agent paths must be absolute")
|
return nil, errors.New("library agent paths must be absolute")
|
||||||
@@ -143,7 +178,7 @@ func (a *Agent) Serve(ctx context.Context) error {
|
|||||||
// writer from a previous killed run. Recover before accepting work; if that
|
// writer from a previous killed run. Recover before accepting work; if that
|
||||||
// fails, stay degraded and refuse captures instead of pretending to be ready.
|
// fails, stay degraded and refuse captures instead of pretending to be ready.
|
||||||
if err := a.recoverWriters(ctx); err != nil {
|
if err := a.recoverWriters(ctx); err != nil {
|
||||||
a.degraded = err.Error()
|
a.setDegraded(err.Error())
|
||||||
log.Printf("library agent: pending freeze recovery failed: %v", err)
|
log.Printf("library agent: pending freeze recovery failed: %v", err)
|
||||||
}
|
}
|
||||||
go func() {
|
go func() {
|
||||||
@@ -194,13 +229,18 @@ func (a *Agent) handle(ctx context.Context, conn net.Conn) {
|
|||||||
|
|
||||||
// execute serialises operations and maps a request onto the one fixed command.
|
// execute serialises operations and maps a request onto the one fixed command.
|
||||||
func (a *Agent) execute(ctx context.Context, req request) response {
|
func (a *Agent) execute(ctx context.Context, req request) response {
|
||||||
|
// health 不进操作锁、不跑子进程:调用方要的是"这个执行器指向谁、能不能干活",
|
||||||
|
// 而一次正在进行的封存不该让这个回答变成"不可达"。
|
||||||
|
if req.Action == "health" {
|
||||||
|
return a.health()
|
||||||
|
}
|
||||||
a.mu.Lock()
|
a.mu.Lock()
|
||||||
defer a.mu.Unlock()
|
defer a.mu.Unlock()
|
||||||
out := response{Action: req.Action, Scope: req.Scope, Job: req.Job}
|
out := response{Action: req.Action, Scope: req.Scope, Job: req.Job}
|
||||||
// Degraded blocks new captures, but recover and status must stay reachable:
|
// Degraded blocks new captures, but recover and status must stay reachable:
|
||||||
// otherwise there is no way back to a verified state.
|
// otherwise there is no way back to a verified state.
|
||||||
if a.degraded != "" && req.Action == "capture" {
|
if state := a.snapshotState(); state.Degraded != "" && req.Action == "capture" {
|
||||||
out.Error = "封存执行边界处于降级状态(上次恢复失败:" + a.degraded + "),先执行 recover"
|
out.Error = "封存执行边界处于降级状态(上次恢复失败:" + state.Degraded + "),先执行 recover"
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
var args []string
|
var args []string
|
||||||
@@ -218,6 +258,16 @@ func (a *Agent) execute(ctx context.Context, req request) response {
|
|||||||
out.Error = "invalid job id"
|
out.Error = "invalid job id"
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
if req.Expect == nil {
|
||||||
|
out.Error = "capture 请求没有携带期望执行器身份,拒绝执行"
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
// 此时已持有操作互斥(a.mu):从"比对身份"到"真正跑子进程"之间没有别的
|
||||||
|
// 请求能插进来,也不会被另一个 agent 抢走——这是零副作用的前提。
|
||||||
|
if err := a.config.matchExpectation(*req.Expect); err != nil {
|
||||||
|
out.Error = err.Error()
|
||||||
|
return out
|
||||||
|
}
|
||||||
args = []string{"capture", "-scope", req.Scope}
|
args = []string{"capture", "-scope", req.Scope}
|
||||||
if req.Job != "" {
|
if req.Job != "" {
|
||||||
args = append(args, "-job", req.Job)
|
args = append(args, "-job", req.Job)
|
||||||
@@ -242,6 +292,10 @@ func (a *Agent) execute(ctx context.Context, req request) response {
|
|||||||
args = append(args, a.config.declaration()...)
|
args = append(args, a.config.declaration()...)
|
||||||
bounded, cancel := context.WithTimeout(ctx, a.config.MaxDuration)
|
bounded, cancel := context.WithTimeout(ctx, a.config.MaxDuration)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
if req.Action == "capture" || req.Action == "recover" {
|
||||||
|
a.setBusy(true)
|
||||||
|
defer a.setBusy(false)
|
||||||
|
}
|
||||||
payload, err := a.config.run(bounded, a.config.Binary, args...)
|
payload, err := a.config.run(bounded, a.config.Binary, args...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
detail := "operation failed"
|
detail := "operation failed"
|
||||||
@@ -254,7 +308,7 @@ func (a *Agent) execute(ctx context.Context, req request) response {
|
|||||||
// verify.
|
// verify.
|
||||||
if req.Action == "capture" || req.Action == "recover" {
|
if req.Action == "capture" || req.Action == "recover" {
|
||||||
if recoveryErr := a.recoverWriters(ctx); recoveryErr != nil {
|
if recoveryErr := a.recoverWriters(ctx); recoveryErr != nil {
|
||||||
a.degraded = recoveryErr.Error()
|
a.setDegraded(recoveryErr.Error())
|
||||||
out.Error = detail + ";写入者恢复失败:" + recoveryErr.Error() + "(已阻断后续封存,需人工处理)"
|
out.Error = detail + ";写入者恢复失败:" + recoveryErr.Error() + "(已阻断后续封存,需人工处理)"
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
@@ -266,7 +320,7 @@ func (a *Agent) execute(ctx context.Context, req request) response {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
if req.Action == "recover" {
|
if req.Action == "recover" {
|
||||||
a.degraded = ""
|
a.clearDegraded()
|
||||||
}
|
}
|
||||||
out.OK = true
|
out.OK = true
|
||||||
if json.Valid(payload) {
|
if json.Valid(payload) {
|
||||||
@@ -274,9 +328,32 @@ func (a *Agent) execute(ctx context.Context, req request) response {
|
|||||||
} else {
|
} else {
|
||||||
out.Payload = json.RawMessage(`{}`)
|
out.Payload = json.RawMessage(`{}`)
|
||||||
}
|
}
|
||||||
|
// 封存结果必须带上"这次是谁执行的":调用方据此复核,防止心跳与执行之间
|
||||||
|
// socket 被换成另一个 agent(真正的封错库事故就是这样发生的)。
|
||||||
|
if req.Action == "capture" && req.Expect != nil {
|
||||||
|
out.Payload = a.attachIdentity(out.Payload, *req.Expect)
|
||||||
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// attachIdentity 把执行器身份并进封存结果(保留 CLI 原有的字段,如 snapshot_id)。
|
||||||
|
// expectation_fingerprint 是"本执行器实际核对过的那份期望"的指纹:调用方据此确认
|
||||||
|
// 执行时看到的期望与它发出的一致,而不是被替换后重新生成的。
|
||||||
|
func (a *Agent) attachIdentity(payload json.RawMessage, expectation Expectation) json.RawMessage {
|
||||||
|
var document map[string]any
|
||||||
|
if err := json.Unmarshal(payload, &document); err != nil {
|
||||||
|
return payload
|
||||||
|
}
|
||||||
|
identity := a.config.identity()
|
||||||
|
identity["expectation_fingerprint"] = expectation.Fingerprint
|
||||||
|
document["agent_identity"] = identity
|
||||||
|
encoded, err := json.Marshal(document)
|
||||||
|
if err != nil {
|
||||||
|
return payload
|
||||||
|
}
|
||||||
|
return json.RawMessage(encoded)
|
||||||
|
}
|
||||||
|
|
||||||
// recoverWriters runs the idempotent recovery command with its own budget, even
|
// recoverWriters runs the idempotent recovery command with its own budget, even
|
||||||
// when the triggering context is already cancelled (timeout / shutdown).
|
// when the triggering context is already cancelled (timeout / shutdown).
|
||||||
func (a *Agent) recoverWriters(ctx context.Context) error {
|
func (a *Agent) recoverWriters(ctx context.Context) error {
|
||||||
@@ -285,7 +362,7 @@ func (a *Agent) recoverWriters(ctx context.Context) error {
|
|||||||
if _, err := a.config.run(bounded, a.config.Binary, append([]string{"recover"}, a.config.declaration()...)...); err != nil {
|
if _, err := a.config.run(bounded, a.config.Binary, append([]string{"recover"}, a.config.declaration()...)...); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
a.degraded = ""
|
a.clearDegraded()
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -332,12 +409,38 @@ func (c Config) run(ctx context.Context, name string, args ...string) ([]byte, e
|
|||||||
|
|
||||||
// Describe returns a redacted view for logs and health checks.
|
// Describe returns a redacted view for logs and health checks.
|
||||||
func (a *Agent) Describe() map[string]any {
|
func (a *Agent) Describe() map[string]any {
|
||||||
return map[string]any{
|
view := a.config.identity()
|
||||||
"socket": a.config.Socket,
|
view["scope_choice"] = "request may select internal or external only"
|
||||||
"archive_root": a.config.ArchiveRoot,
|
return view
|
||||||
"journal_root": a.config.JournalRoot,
|
}
|
||||||
"project": a.config.Project,
|
|
||||||
"writers": a.config.Writers,
|
// health 是"这个执行器指向谁、现在能不能干活"的唯一回答。它只暴露容器名、
|
||||||
"scope_choice": "request may select internal or external only",
|
// 路径与声明,不含任何凭据;调用方据此核对源身份,而不是靠配置猜。
|
||||||
}
|
func (a *Agent) health() response {
|
||||||
|
state := a.snapshotState()
|
||||||
|
payload := a.config.identity()
|
||||||
|
payload["busy"] = state.Busy
|
||||||
|
payload["degraded"] = state.Degraded
|
||||||
|
encoded, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
return response{Action: "health", Error: "cannot encode health"}
|
||||||
|
}
|
||||||
|
return response{OK: true, Action: "health", Payload: json.RawMessage(encoded)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// identity 被日志与健康应答共用一份,避免两处描述漂移。
|
||||||
|
func (c Config) identity() map[string]any {
|
||||||
|
identity := map[string]any{
|
||||||
|
"project": c.Project,
|
||||||
|
"archive_root": c.ArchiveRoot,
|
||||||
|
"journal_root": c.JournalRoot,
|
||||||
|
"source_app": c.SourceApp,
|
||||||
|
"source_pg": c.SourcePG,
|
||||||
|
"source_es": c.SourceES,
|
||||||
|
"writers": c.Writers,
|
||||||
|
}
|
||||||
|
if c.SourceS3Endpoint != "" {
|
||||||
|
identity["source_s3_endpoint"] = c.SourceS3Endpoint
|
||||||
|
}
|
||||||
|
return identity
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -56,6 +56,19 @@ func testAgent(t *testing.T) (*Agent, string, func() []string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// captureExpect 是测试 agent 自己的身份(weknora071full / WKNFull-app),
|
||||||
|
// 现在 capture 请求必须携带期望身份,缺了就会被拒。
|
||||||
|
func captureExpect(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
// 只绑定 project:各用例的 source_app 不同,绑定字段的专门用例在 expectation_test.go。
|
||||||
|
expectation := NewExpectation("weknora071full", "", "")
|
||||||
|
encoded, err := json.Marshal(expectation)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return string(encoded)
|
||||||
|
}
|
||||||
|
|
||||||
func roundTrip(t *testing.T, socket string, payload string) response {
|
func roundTrip(t *testing.T, socket string, payload string) response {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
conn, err := net.Dial("unix", socket)
|
conn, err := net.Dial("unix", socket)
|
||||||
@@ -105,7 +118,7 @@ func TestAgentSocketIsPrivateAndPinsTheDeclaration(t *testing.T) {
|
|||||||
if dir.Mode().Perm() != 0700 {
|
if dir.Mode().Perm() != 0700 {
|
||||||
t.Fatalf("socket directory mode %v", dir.Mode().Perm())
|
t.Fatalf("socket directory mode %v", dir.Mode().Perm())
|
||||||
}
|
}
|
||||||
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-internal-test"}`)
|
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-internal-test","expect":`+captureExpect(t)+`}`)
|
||||||
if !out.OK || out.Job != "lib-internal-test" {
|
if !out.OK || out.Job != "lib-internal-test" {
|
||||||
t.Fatalf("capture rejected: %+v", out)
|
t.Fatalf("capture rejected: %+v", out)
|
||||||
}
|
}
|
||||||
@@ -234,7 +247,7 @@ func TestAgentRecoversWritersAfterFailedCapture(t *testing.T) {
|
|||||||
defer cancel()
|
defer cancel()
|
||||||
go func() { _ = agent.Serve(ctx) }()
|
go func() { _ = agent.Serve(ctx) }()
|
||||||
waitSocket(t, socket)
|
waitSocket(t, socket)
|
||||||
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-1"}`)
|
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-1","expect":`+captureExpect(t)+`}`)
|
||||||
if out.OK {
|
if out.OK {
|
||||||
t.Fatalf("failed capture reported success: %+v", out)
|
t.Fatalf("failed capture reported success: %+v", out)
|
||||||
}
|
}
|
||||||
@@ -255,12 +268,12 @@ func TestAgentBlocksCapturesWhenRecoveryFails(t *testing.T) {
|
|||||||
go func() { _ = agent.Serve(ctx) }()
|
go func() { _ = agent.Serve(ctx) }()
|
||||||
waitSocket(t, socket)
|
waitSocket(t, socket)
|
||||||
setRecoverErr(errors.New("docker restart failed"))
|
setRecoverErr(errors.New("docker restart failed"))
|
||||||
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-2"}`)
|
out := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-2","expect":`+captureExpect(t)+`}`)
|
||||||
if out.OK || !strings.Contains(out.Error, "写入者恢复失败") || !strings.Contains(out.Error, "已阻断后续封存") {
|
if out.OK || !strings.Contains(out.Error, "写入者恢复失败") || !strings.Contains(out.Error, "已阻断后续封存") {
|
||||||
t.Fatalf("recovery failure not surfaced: %+v", out)
|
t.Fatalf("recovery failure not surfaced: %+v", out)
|
||||||
}
|
}
|
||||||
before := len(captureCalls(calls()))
|
before := len(captureCalls(calls()))
|
||||||
blocked := roundTrip(t, socket, `{"action":"capture","scope":"external","job":"lib-fail-3"}`)
|
blocked := roundTrip(t, socket, `{"action":"capture","scope":"external","job":"lib-fail-3","expect":`+captureExpect(t)+`}`)
|
||||||
if blocked.OK || !strings.Contains(blocked.Error, "降级") {
|
if blocked.OK || !strings.Contains(blocked.Error, "降级") {
|
||||||
t.Fatalf("degraded agent accepted a capture: %+v", blocked)
|
t.Fatalf("degraded agent accepted a capture: %+v", blocked)
|
||||||
}
|
}
|
||||||
@@ -272,7 +285,7 @@ func TestAgentBlocksCapturesWhenRecoveryFails(t *testing.T) {
|
|||||||
if recovered := roundTrip(t, socket, `{"action":"recover"}`); !recovered.OK {
|
if recovered := roundTrip(t, socket, `{"action":"recover"}`); !recovered.OK {
|
||||||
t.Fatalf("explicit recovery failed: %+v", recovered)
|
t.Fatalf("explicit recovery failed: %+v", recovered)
|
||||||
}
|
}
|
||||||
if resumed := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-4"}`); strings.Contains(resumed.Error, "降级") {
|
if resumed := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"lib-fail-4","expect":`+captureExpect(t)+`}`); strings.Contains(resumed.Error, "降级") {
|
||||||
t.Fatalf("recovery did not clear the degraded state: %+v", resumed)
|
t.Fatalf("recovery did not clear the degraded state: %+v", resumed)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -396,7 +409,7 @@ func TestAgentForwardsOnlyExactDeadLetterFingerprints(t *testing.T) {
|
|||||||
time.Sleep(10 * time.Millisecond)
|
time.Sleep(10 * time.Millisecond)
|
||||||
}
|
}
|
||||||
fingerprint := "sha256:" + strings.Repeat("a", 64)
|
fingerprint := "sha256:" + strings.Repeat("a", 64)
|
||||||
ok := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j1","released":["`+fingerprint+`"]}`)
|
ok := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j1","released":["`+fingerprint+`"],"expect":`+captureExpect(t)+`}`)
|
||||||
if !ok.OK {
|
if !ok.OK {
|
||||||
t.Fatalf("exact fingerprint refused: %s", ok.Error)
|
t.Fatalf("exact fingerprint refused: %s", ok.Error)
|
||||||
}
|
}
|
||||||
@@ -404,7 +417,7 @@ func TestAgentForwardsOnlyExactDeadLetterFingerprints(t *testing.T) {
|
|||||||
if !strings.Contains(last, "-released-dead-letters "+fingerprint) {
|
if !strings.Contains(last, "-released-dead-letters "+fingerprint) {
|
||||||
t.Fatalf("fingerprint not forwarded: %s", last)
|
t.Fatalf("fingerprint not forwarded: %s", last)
|
||||||
}
|
}
|
||||||
bad := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j2","released":["*"]}`)
|
bad := roundTrip(t, socket, `{"action":"capture","scope":"internal","job":"j2","released":["*"],"expect":`+captureExpect(t)+`}`)
|
||||||
if bad.OK || !strings.Contains(bad.Error, "fingerprints") {
|
if bad.OK || !strings.Contains(bad.Error, "fingerprints") {
|
||||||
t.Fatalf("wildcard release accepted: %+v", bad)
|
t.Fatalf("wildcard release accepted: %+v", bad)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -28,13 +28,13 @@ type Result struct {
|
|||||||
|
|
||||||
// Capture asks the agent to seal one library scope. The job id makes the call
|
// Capture asks the agent to seal one library scope. The job id makes the call
|
||||||
// idempotent: repeating it returns the already sealed candidate.
|
// idempotent: repeating it returns the already sealed candidate.
|
||||||
func Capture(ctx context.Context, config ClientConfig, scope, job string) (Result, error) {
|
func Capture(ctx context.Context, config ClientConfig, scope, job string, expect Expectation) (Result, error) {
|
||||||
return CaptureReleased(ctx, config, scope, job, nil)
|
return CaptureReleased(ctx, config, scope, job, nil, expect)
|
||||||
}
|
}
|
||||||
|
|
||||||
// CaptureReleased 在 Capture 之上带上治理平台已审计的精确死信解除指纹。
|
// CaptureReleased 在 Capture 之上带上治理平台已审计的精确死信解除指纹。
|
||||||
func CaptureReleased(ctx context.Context, config ClientConfig, scope, job string, released []string) (Result, error) {
|
func CaptureReleased(ctx context.Context, config ClientConfig, scope, job string, released []string, expect Expectation) (Result, error) {
|
||||||
request := map[string]any{"action": "capture", "scope": scope, "job": job}
|
request := map[string]any{"action": "capture", "scope": scope, "job": job, "expect": expect}
|
||||||
if len(released) > 0 {
|
if len(released) > 0 {
|
||||||
request["released"] = released
|
request["released"] = released
|
||||||
}
|
}
|
||||||
@@ -46,6 +46,12 @@ func Status(ctx context.Context, config ClientConfig) (Result, error) {
|
|||||||
return call(ctx, config, map[string]any{"action": "status"})
|
return call(ctx, config, map[string]any{"action": "status"})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Health 只问三件事:这个执行器指向谁(project/容器/归档)、是否在跑、是否降级。
|
||||||
|
// 它不跑子进程、不占操作锁,所以可以当心跳用;Status 会去查源库,代价高得多。
|
||||||
|
func Health(ctx context.Context, config ClientConfig) (Result, error) {
|
||||||
|
return call(ctx, config, map[string]any{"action": "health"})
|
||||||
|
}
|
||||||
|
|
||||||
// Recover restores writers from an interrupted freeze journal.
|
// Recover restores writers from an interrupted freeze journal.
|
||||||
func Recover(ctx context.Context, config ClientConfig) (Result, error) {
|
func Recover(ctx context.Context, config ClientConfig) (Result, error) {
|
||||||
return call(ctx, config, map[string]any{"action": "recover"})
|
return call(ctx, config, map[string]any{"action": "recover"})
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
package libraryagent
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Expectation 是调用方对"这一条请求该由哪个执行器执行"的声明。
|
||||||
|
//
|
||||||
|
// 为什么是调用方声明而不是执行器自报:自报值只能证明"我是谁",不能证明"你找的
|
||||||
|
// 就是我"。心跳之后、真正停写入者之前,socket 可能被换成另一个 agent;只有把
|
||||||
|
// 期望随**同一个请求**送进去、由接收方在操作互斥内、执行任何源操作之前比对,
|
||||||
|
// 才能让"换过 socket"变成零副作用(错的那个 noche 一次 run 都不调)。
|
||||||
|
type Expectation struct {
|
||||||
|
Project string `json:"project"`
|
||||||
|
SourceApp string `json:"source_app,omitempty"`
|
||||||
|
ArchiveRoot string `json:"archive_root,omitempty"`
|
||||||
|
// Fingerprint 覆盖上面三项,供接收方回显、调用方事后核对:
|
||||||
|
// 证明"执行器看到的就是我发的那份期望",而不是被替换后重新生成的。
|
||||||
|
Fingerprint string `json:"fingerprint"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewExpectation 由治理侧配置构造期望(绝不从执行器应答里取字段)。
|
||||||
|
func NewExpectation(project, sourceApp, archiveRoot string) Expectation {
|
||||||
|
expectation := Expectation{
|
||||||
|
Project: strings.TrimSpace(project),
|
||||||
|
SourceApp: strings.TrimSpace(sourceApp),
|
||||||
|
ArchiveRoot: strings.TrimSpace(archiveRoot),
|
||||||
|
}
|
||||||
|
expectation.Fingerprint = expectation.fingerprint()
|
||||||
|
return expectation
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e Expectation) fingerprint() string {
|
||||||
|
sum := sha256.Sum256([]byte(e.Project + "\x00" + e.SourceApp + "\x00" + e.ArchiveRoot))
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Complete 表示期望足够具体:没有 project 的期望无法核对身份,必须 fail closed;
|
||||||
|
// 指纹与字段不一致说明请求在半路被改过。
|
||||||
|
func (e Expectation) Complete() bool {
|
||||||
|
return e.Project != "" && e.Fingerprint == e.fingerprint()
|
||||||
|
}
|
||||||
|
|
||||||
|
// matchExpectation 在操作互斥内比对"本执行器配置"与"调用方期望"。
|
||||||
|
// 只比较期望里给出的字段:project 必填,source_app/archive_root 给了就必须一致。
|
||||||
|
func (c Config) matchExpectation(expectation Expectation) error {
|
||||||
|
if !expectation.Complete() {
|
||||||
|
return errors.New("capture 请求缺少完整的期望执行器身份,拒绝执行")
|
||||||
|
}
|
||||||
|
if c.Project != expectation.Project {
|
||||||
|
return errors.New("执行器与请求期望的实例不一致:" +
|
||||||
|
"agent=" + quoteOrEmpty(c.Project) + ",期望=" + quoteOrEmpty(expectation.Project) + ",拒绝执行")
|
||||||
|
}
|
||||||
|
if expectation.SourceApp != "" && c.SourceApp != expectation.SourceApp {
|
||||||
|
return errors.New("执行器与请求期望的源容器不一致:" +
|
||||||
|
"agent=" + quoteOrEmpty(c.SourceApp) + ",期望=" + quoteOrEmpty(expectation.SourceApp) + ",拒绝执行")
|
||||||
|
}
|
||||||
|
if expectation.ArchiveRoot != "" && c.ArchiveRoot != expectation.ArchiveRoot {
|
||||||
|
return errors.New("执行器与请求期望的归档根不一致,拒绝执行")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func quoteOrEmpty(value string) string {
|
||||||
|
if value == "" {
|
||||||
|
return `""`
|
||||||
|
}
|
||||||
|
return `"` + value + `"`
|
||||||
|
}
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
package libraryagent
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func serveAgent(t *testing.T, agent *Agent, socket string) {
|
||||||
|
t.Helper()
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
t.Cleanup(cancel)
|
||||||
|
go func() { _ = agent.Serve(ctx) }()
|
||||||
|
deadline := time.Now().Add(2 * time.Second)
|
||||||
|
for {
|
||||||
|
if _, err := os.Stat(socket); err == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatal("agent socket never appeared")
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 健康应答必须不排队等正在跑的封存:一次封存最长 15 分钟,如果 health 也去抢
|
||||||
|
// 操作锁,页面就会把"正在封存"看成"执行器不可达",进而误导运维。
|
||||||
|
func TestHealthAnswersWhileACaptureHoldsTheOperationLock(t *testing.T) {
|
||||||
|
agent, socket, calls := testAgent(t)
|
||||||
|
serveAgent(t, agent, socket)
|
||||||
|
// 占住操作锁并置 busy,模拟一次正在进行的封存。
|
||||||
|
agent.mu.Lock()
|
||||||
|
defer agent.mu.Unlock()
|
||||||
|
agent.setBusy(true)
|
||||||
|
defer agent.setBusy(false)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
result, err := Health(ctx, ClientConfig{Socket: socket, Timeout: 2 * time.Second})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("health while busy: %v", err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
Project string `json:"project"`
|
||||||
|
Busy bool `json:"busy"`
|
||||||
|
}
|
||||||
|
if err = json.Unmarshal(result.Payload, &payload); err != nil {
|
||||||
|
t.Fatalf("health payload: %v", err)
|
||||||
|
}
|
||||||
|
if payload.Project != "weknora071full" || !payload.Busy {
|
||||||
|
t.Fatalf("health payload = %+v", payload)
|
||||||
|
}
|
||||||
|
if len(captureCalls(calls())) != 0 {
|
||||||
|
t.Fatalf("health must not run the capture binary, calls = %v", calls())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 降级状态必须能被调用方看见:否则页面只能看到"可发起",而真发起会被拒。
|
||||||
|
func TestHealthReportsDegradedState(t *testing.T) {
|
||||||
|
agent, socket, _ := testAgent(t)
|
||||||
|
serveAgent(t, agent, socket)
|
||||||
|
// 启动恢复成功会清空降级;这里在它之后模拟"恢复失败"留下的降级状态。
|
||||||
|
agent.setDegraded("写入者恢复失败")
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
result, err := Health(ctx, ClientConfig{Socket: socket, Timeout: 2 * time.Second})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
Degraded string `json:"degraded"`
|
||||||
|
SourceApp string `json:"source_app"`
|
||||||
|
Writers string `json:"writers"`
|
||||||
|
}
|
||||||
|
if err = json.Unmarshal(result.Payload, &payload); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if payload.Degraded != "写入者恢复失败" || payload.SourceApp != "WKNFull-app" || payload.Writers != "app" {
|
||||||
|
t.Fatalf("health payload = %+v", payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 封存结果必须自带"这次是谁执行的":调用方据此在事后复核,防止心跳与执行之间
|
||||||
|
// socket 被换成另一个实例。
|
||||||
|
func TestCaptureResultCarriesTheAgentIdentity(t *testing.T) {
|
||||||
|
agent, socket, _ := testAgent(t)
|
||||||
|
serveAgent(t, agent, socket)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
result, err := Capture(ctx, ClientConfig{Socket: socket, Timeout: 5 * time.Second}, "internal",
|
||||||
|
"lib-internal-identity", NewExpectation("weknora071full", "WKNFull-app", ""))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
SnapshotID string `json:"snapshot_id"`
|
||||||
|
Identity struct {
|
||||||
|
Project string `json:"project"`
|
||||||
|
} `json:"agent_identity"`
|
||||||
|
}
|
||||||
|
if err = json.Unmarshal(result.Payload, &payload); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if payload.SnapshotID != "abc" || payload.Identity.Project != "weknora071full" {
|
||||||
|
t.Fatalf("payload = %s", result.Payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
package libraryagent
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// unix socket 路径有长度上限,测试目录要用短的(TMPDIR 太深)。
|
||||||
|
func shortSocketDir(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
dir, err := os.MkdirTemp("/tmp", "kgswap-")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { os.RemoveAll(dir) })
|
||||||
|
return dir
|
||||||
|
}
|
||||||
|
|
||||||
|
// 两个真实 agent、两个 socket:负例把调用方指到"另一个实例"的 socket 上,
|
||||||
|
// 断言**错误 agent 一次 capture 都没跑**(不是跑完再拒),正例走对 socket 正常封存。
|
||||||
|
func TestCaptureRequestIsRefusedBeforeAnySourceOperationWhenSocketIsSwapped(t *testing.T) {
|
||||||
|
rightAgent, rightSocket, rightCalls := agentWithProject(t, "kgmaint-weknora", "kgmaint-app")
|
||||||
|
wrongAgent, wrongSocket, wrongCalls := agentWithProject(t, "weknora071full", "WKNFull-app")
|
||||||
|
serveAgent(t, rightAgent, rightSocket)
|
||||||
|
serveAgent(t, wrongAgent, wrongSocket)
|
||||||
|
|
||||||
|
expectation := NewExpectation("kgmaint-weknora", "kgmaint-app", "")
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// 负例:调用方手上的 socket 已被换成另一个实例的执行器。
|
||||||
|
result, err := Capture(ctx, ClientConfig{Socket: wrongSocket, Timeout: 5 * time.Second},
|
||||||
|
"internal", "lib-swap", expectation)
|
||||||
|
if err == nil || result.OK {
|
||||||
|
t.Fatalf("capture against the wrong executor was accepted: %+v", result)
|
||||||
|
}
|
||||||
|
if !strings.Contains(result.Error, "期望") {
|
||||||
|
t.Fatalf("refusal should name the mismatch: %q", result.Error)
|
||||||
|
}
|
||||||
|
if calls := captureCalls(wrongCalls()); len(calls) != 0 {
|
||||||
|
t.Fatalf("wrong executor ran the capture binary %d time(s): %v", len(calls), calls)
|
||||||
|
}
|
||||||
|
// 旁证:错误的执行器进程本身仍然可用(health 正常),拒绝来自身份比对而不是不可达。
|
||||||
|
health, err := Health(ctx, ClientConfig{Socket: wrongSocket, Timeout: 5 * time.Second})
|
||||||
|
if err != nil || !health.OK {
|
||||||
|
t.Fatalf("wrong executor should still answer health: %v %+v", err, health)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 正例:同一份期望打到正确的 socket,正常封存一次。
|
||||||
|
result, err = Capture(ctx, ClientConfig{Socket: rightSocket, Timeout: 5 * time.Second},
|
||||||
|
"internal", "lib-swap", expectation)
|
||||||
|
if err != nil || !result.OK {
|
||||||
|
t.Fatalf("capture against the right executor failed: %v %+v", err, result)
|
||||||
|
}
|
||||||
|
if calls := captureCalls(rightCalls()); len(calls) != 1 {
|
||||||
|
t.Fatalf("right executor ran %d time(s): %v", len(calls), calls)
|
||||||
|
}
|
||||||
|
if calls := captureCalls(wrongCalls()); len(calls) != 0 {
|
||||||
|
t.Fatalf("wrong executor ran after the positive case: %v", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 缺期望、期望指纹对不上、期望里 source_app 不同,都必须 fail closed 且不跑子进程。
|
||||||
|
func TestCaptureFailsClosedWithoutACompleteExpectation(t *testing.T) {
|
||||||
|
agent, socket, calls := agentWithProject(t, "kgmaint-weknora", "kgmaint-app")
|
||||||
|
serveAgent(t, agent, socket)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
tampered := NewExpectation("kgmaint-weknora", "kgmaint-app", "")
|
||||||
|
tampered.Project = "someone-else" // 字段改了、指纹没改:请求在半路被改过
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
expectation Expectation
|
||||||
|
text string
|
||||||
|
}{
|
||||||
|
{name: "missing expectation", expectation: Expectation{}, text: "缺少完整的期望执行器身份"},
|
||||||
|
{name: "tampered expectation", expectation: tampered, text: "缺少完整的期望执行器身份"},
|
||||||
|
{name: "wrong source app", expectation: NewExpectation("kgmaint-weknora", "WKNFull-app", ""), text: "源容器不一致"},
|
||||||
|
}
|
||||||
|
for _, test := range cases {
|
||||||
|
result, err := Capture(ctx, ClientConfig{Socket: socket, Timeout: 5 * time.Second},
|
||||||
|
"internal", "lib-"+strings.ReplaceAll(test.name, " ", "-"), test.expectation)
|
||||||
|
if err == nil || result.OK {
|
||||||
|
t.Fatalf("%s: accepted %+v", test.name, result)
|
||||||
|
}
|
||||||
|
if !strings.Contains(result.Error, test.text) {
|
||||||
|
t.Fatalf("%s: error %q", test.name, result.Error)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if calls := captureCalls(calls()); len(calls) != 0 {
|
||||||
|
t.Fatalf("capture binary ran despite a refused identity: %v", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 恢复路径不受期望约束:它只把写入者恢复回来,任何身份都必须能执行。
|
||||||
|
func TestRecoverStaysAvailableRegardlessOfExpectation(t *testing.T) {
|
||||||
|
agent, socket, calls := agentWithProject(t, "kgmaint-weknora", "kgmaint-app")
|
||||||
|
serveAgent(t, agent, socket)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
result, err := Recover(ctx, ClientConfig{Socket: socket, Timeout: 5 * time.Second})
|
||||||
|
if err != nil || !result.OK {
|
||||||
|
t.Fatalf("recover refused: %v %+v", err, result)
|
||||||
|
}
|
||||||
|
if len(recoverCalls(calls())) == 0 {
|
||||||
|
t.Fatal("recover did not run the recovery command")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// agentWithProject 造一个真实 agent(不跑真命令),返回它自己的调用记录。
|
||||||
|
func agentWithProject(t *testing.T, project, sourceApp string) (*Agent, string, func() []string) {
|
||||||
|
t.Helper()
|
||||||
|
root := shortSocketDir(t)
|
||||||
|
var mu sync.Mutex
|
||||||
|
var calls []string
|
||||||
|
config := Config{
|
||||||
|
Socket: root + "/agent.sock",
|
||||||
|
Binary: "/bin/echo",
|
||||||
|
ArchiveRoot: root + "/library",
|
||||||
|
JournalRoot: root + "/freeze",
|
||||||
|
SourceApp: sourceApp,
|
||||||
|
SourcePG: "pg",
|
||||||
|
SourceES: "es",
|
||||||
|
Project: project,
|
||||||
|
Writers: "app",
|
||||||
|
NonWriters: "frontend=UI only",
|
||||||
|
MaxDuration: time.Minute,
|
||||||
|
Run: func(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||||
|
mu.Lock()
|
||||||
|
calls = append(calls, name+" "+strings.Join(args, " "))
|
||||||
|
mu.Unlock()
|
||||||
|
return []byte(`{"snapshot_id":"abc","state":"succeeded"}`), nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
agent, err := New(config)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return agent, config.Socket, func() []string {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
return append([]string(nil), calls...)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
// Package libraryexecutor 回答一个问题:主机侧封存执行器现在是不是真的可用,
|
||||||
|
// 而且是不是指向我们这个实例。
|
||||||
|
//
|
||||||
|
// 为什么不能在页面里读一个配置字符串了事:正式栈曾经出现过"agent 顶着正式单元名
|
||||||
|
// 却在冻结候选实例"的事故——配置非空、页面显示可发起、真发起却封错了库。可用的
|
||||||
|
// 定义必须是**运行时观测**:socket 通、执行器自报的实例与期望一致、没有降级。
|
||||||
|
package libraryexecutor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/libraryagent"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Identity 是执行器自报的身份:它要冻结哪个容器的写入者、归档写到哪。
|
||||||
|
// 只有容器名与路径,没有任何凭据。
|
||||||
|
type Identity struct {
|
||||||
|
Project string `json:"project"`
|
||||||
|
ArchiveRoot string `json:"archive_root"`
|
||||||
|
JournalRoot string `json:"journal_root"`
|
||||||
|
SourceApp string `json:"source_app"`
|
||||||
|
SourcePG string `json:"source_pg"`
|
||||||
|
SourceES string `json:"source_es"`
|
||||||
|
Writers string `json:"writers"`
|
||||||
|
SourceS3Endpoint string `json:"source_s3_endpoint,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Observation 是一次真实探测的结果,会被落库供页面读取。
|
||||||
|
type Observation struct {
|
||||||
|
Instance string `json:"instance"`
|
||||||
|
Reachable bool `json:"reachable"`
|
||||||
|
Busy bool `json:"busy"`
|
||||||
|
Degraded string `json:"degraded,omitempty"`
|
||||||
|
Identity Identity `json:"identity"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
ObservedAt time.Time `json:"observed_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Expectation 是治理侧对执行器的要求。ArchiveRoot 不参与判定:容器里的路径是
|
||||||
|
// 挂载后的路径,和宿主机上的真实路径本来就不是一个字符串;用它当门禁只会造成
|
||||||
|
// 永久误报,所以只作为观测值展示给运维比对。
|
||||||
|
type Expectation struct {
|
||||||
|
Instance string
|
||||||
|
Project string
|
||||||
|
ArchiveRoot string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Probe 用一次短超时的 health 往返取回执行器身份。它不跑子进程、不碰源库,
|
||||||
|
// 所以可以当心跳反复调用。instance 是"这次观测属于哪个实例",由调用方的
|
||||||
|
// LIBRARY_TARGET_INSTANCE 决定。
|
||||||
|
func Probe(ctx context.Context, instance, socket string, timeout time.Duration, now func() time.Time) Observation {
|
||||||
|
if now == nil {
|
||||||
|
now = time.Now
|
||||||
|
}
|
||||||
|
observation := Observation{Instance: instance, ObservedAt: now().UTC()}
|
||||||
|
if socket == "" {
|
||||||
|
observation.Error = "未配置主机侧封存 agent socket(LIBRARY_AGENT_SOCKET)"
|
||||||
|
return observation
|
||||||
|
}
|
||||||
|
if timeout <= 0 {
|
||||||
|
timeout = 3 * time.Second
|
||||||
|
}
|
||||||
|
result, err := libraryagent.Health(ctx, libraryagent.ClientConfig{Socket: socket, Timeout: timeout})
|
||||||
|
if err != nil {
|
||||||
|
observation.Error = err.Error()
|
||||||
|
return observation
|
||||||
|
}
|
||||||
|
var payload struct {
|
||||||
|
Identity
|
||||||
|
Busy bool `json:"busy"`
|
||||||
|
Degraded string `json:"degraded"`
|
||||||
|
}
|
||||||
|
if err = json.Unmarshal(result.Payload, &payload); err != nil {
|
||||||
|
observation.Error = "执行器健康应答无法解析"
|
||||||
|
return observation
|
||||||
|
}
|
||||||
|
observation.Reachable = true
|
||||||
|
observation.Busy = payload.Busy
|
||||||
|
observation.Degraded = payload.Degraded
|
||||||
|
observation.Identity = payload.Identity
|
||||||
|
return observation
|
||||||
|
}
|
||||||
|
|
||||||
|
// Evaluate 把"观测 + 期望 + 新鲜度"换算成页面上的可用性与原因。
|
||||||
|
// 返回的原因必须能直接告诉运维该去改什么,而不是"不可用"三个字。
|
||||||
|
func Evaluate(observation Observation, want Expectation, maxAge time.Duration, now time.Time) (bool, string) {
|
||||||
|
if want.Instance == "" {
|
||||||
|
return false, "未配置封存来源实例身份(LIBRARY_SOURCE_INSTANCE),无法确认执行器归属"
|
||||||
|
}
|
||||||
|
if observation.Instance != want.Instance {
|
||||||
|
return false, fmt.Sprintf("执行器状态来自另一个实例(%q),不是 %q", observation.Instance, want.Instance)
|
||||||
|
}
|
||||||
|
age := now.Sub(observation.ObservedAt)
|
||||||
|
if maxAge > 0 && age > maxAge {
|
||||||
|
return false, fmt.Sprintf("执行器状态已过期:%d 秒前的观测,上限 %d 秒(worker 未上报?)",
|
||||||
|
int(age.Seconds()), int(maxAge.Seconds()))
|
||||||
|
}
|
||||||
|
if !observation.Reachable {
|
||||||
|
reason := "主机侧封存执行器不可达"
|
||||||
|
if observation.Error != "" {
|
||||||
|
reason += ":" + observation.Error
|
||||||
|
}
|
||||||
|
return false, reason
|
||||||
|
}
|
||||||
|
if want.Project == "" {
|
||||||
|
return false, "未配置执行器源身份(LIBRARY_SOURCE_PROJECT),无法核对 agent 指向哪个实例"
|
||||||
|
}
|
||||||
|
if observation.Identity.Project != want.Project {
|
||||||
|
return false, fmt.Sprintf("执行器指向的实例不匹配:agent=%q,期望=%q",
|
||||||
|
observation.Identity.Project, want.Project)
|
||||||
|
}
|
||||||
|
if observation.Degraded != "" {
|
||||||
|
return false, "执行器处于降级状态(上次恢复失败):" + observation.Degraded
|
||||||
|
}
|
||||||
|
if observation.Busy {
|
||||||
|
return false, "执行器正在执行另一个封存/恢复,等它结束再发起"
|
||||||
|
}
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
package libraryexecutor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func shortSocketDir(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
// unix socket 路径上限 ~104 字节,TMPDIR(本机是 .hermes/cache/scratch)太长,
|
||||||
|
// 测试里用一个短目录并立刻清理。
|
||||||
|
dir, err := os.MkdirTemp("/tmp", "kgex-")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { os.RemoveAll(dir) })
|
||||||
|
return dir
|
||||||
|
}
|
||||||
|
|
||||||
|
// 可用性必须来自运行时观测,而不是"配置里写了 socket 路径"。
|
||||||
|
func TestProbeReadsIdentityFromTheAgentOverTheSocket(t *testing.T) {
|
||||||
|
root := shortSocketDir(t)
|
||||||
|
socket := filepath.Join(root, "agent.sock")
|
||||||
|
listener, err := net.Listen("unix", socket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer listener.Close()
|
||||||
|
go func() {
|
||||||
|
for {
|
||||||
|
conn, acceptErr := listener.Accept()
|
||||||
|
if acceptErr != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var request map[string]any
|
||||||
|
_ = json.NewDecoder(conn).Decode(&request)
|
||||||
|
payload, _ := json.Marshal(map[string]any{
|
||||||
|
"project": "kgmaint-weknora", "archive_root": "/tmp/kg-e2e/archive",
|
||||||
|
"journal_root": "/tmp/kg-e2e/journal", "source_app": "kgmaint-app",
|
||||||
|
"source_pg": "kgmaint-postgres", "source_es": "WKNFull-elasticsearch",
|
||||||
|
"writers": "app", "busy": true, "degraded": "",
|
||||||
|
})
|
||||||
|
_ = json.NewEncoder(conn).Encode(map[string]any{"ok": true, "action": "health", "payload": json.RawMessage(payload)})
|
||||||
|
conn.Close()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
observation := Probe(context.Background(), "kgmaint-weknora", socket, 2*time.Second, nil)
|
||||||
|
if !observation.Reachable {
|
||||||
|
t.Fatalf("observation = %+v", observation)
|
||||||
|
}
|
||||||
|
if observation.Identity.Project != "kgmaint-weknora" || observation.Identity.SourceApp != "kgmaint-app" || !observation.Busy {
|
||||||
|
t.Fatalf("identity = %+v", observation.Identity)
|
||||||
|
}
|
||||||
|
if observation.Instance != "kgmaint-weknora" || observation.ObservedAt.IsZero() {
|
||||||
|
t.Fatalf("observation = %+v", observation)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProbeReportsUnreachableAgentWithoutFakingSuccess(t *testing.T) {
|
||||||
|
observation := Probe(context.Background(), "kgmaint-weknora", filepath.Join(t.TempDir(), "absent.sock"), time.Second, nil)
|
||||||
|
if observation.Reachable || observation.Error == "" {
|
||||||
|
t.Fatalf("observation = %+v", observation)
|
||||||
|
}
|
||||||
|
available, reason := Evaluate(observation, Expectation{Instance: "kgmaint-weknora", Project: "kgmaint-weknora"},
|
||||||
|
time.Minute, time.Now().UTC())
|
||||||
|
if available || !strings.Contains(reason, "不可达") {
|
||||||
|
t.Fatalf("available=%v reason=%q", available, reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEvaluateRejectsEveryIdentityAndFreshnessProblem(t *testing.T) {
|
||||||
|
now := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
|
||||||
|
healthy := Observation{
|
||||||
|
Instance: "kgmaint-weknora", Reachable: true, ObservedAt: now,
|
||||||
|
Identity: Identity{Project: "kgmaint-weknora", SourceApp: "kgmaint-app"},
|
||||||
|
}
|
||||||
|
want := Expectation{Instance: "kgmaint-weknora", Project: "kgmaint-weknora"}
|
||||||
|
if available, reason := Evaluate(healthy, want, 2*time.Minute, now); !available || reason != "" {
|
||||||
|
t.Fatalf("healthy executor rejected: %v %q", available, reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mut func(*Observation)
|
||||||
|
want func(Expectation) Expectation
|
||||||
|
text string
|
||||||
|
}{
|
||||||
|
{name: "missing instance config", want: func(Expectation) Expectation { return expectEmptyInstance },
|
||||||
|
text: "LIBRARY_SOURCE_INSTANCE"},
|
||||||
|
{name: "row from another instance", mut: func(o *Observation) { o.Instance = "someone-else" },
|
||||||
|
text: "另一个实例"},
|
||||||
|
{name: "stale", mut: func(o *Observation) { o.ObservedAt = now.Add(-10 * time.Minute) },
|
||||||
|
text: "已过期"},
|
||||||
|
{name: "unreachable", mut: func(o *Observation) { o.Reachable = false; o.Error = "socket missing" },
|
||||||
|
text: "不可达:socket missing"},
|
||||||
|
{name: "project not configured", want: func(Expectation) Expectation { return Expectation{Instance: "kgmaint-weknora"} },
|
||||||
|
text: "LIBRARY_SOURCE_PROJECT"},
|
||||||
|
{name: "wrong source project", mut: func(o *Observation) { o.Identity.Project = "weknora071full" },
|
||||||
|
text: "指向的实例不匹配"},
|
||||||
|
{name: "degraded", mut: func(o *Observation) { o.Degraded = "写入者恢复失败" },
|
||||||
|
text: "降级状态"},
|
||||||
|
{name: "busy", mut: func(o *Observation) { o.Busy = true },
|
||||||
|
text: "正在执行另一个封存"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
observation := healthy
|
||||||
|
if test.mut != nil {
|
||||||
|
test.mut(&observation)
|
||||||
|
}
|
||||||
|
expectation := want
|
||||||
|
if test.want != nil {
|
||||||
|
expectation = test.want(want)
|
||||||
|
}
|
||||||
|
available, reason := Evaluate(observation, expectation, 2*time.Minute, now)
|
||||||
|
if available {
|
||||||
|
t.Fatalf("accepted: %+v", observation)
|
||||||
|
}
|
||||||
|
if !strings.Contains(reason, test.text) {
|
||||||
|
t.Fatalf("reason = %q, want it to mention %q", reason, test.text)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var expectEmptyInstance = Expectation{Project: "kgmaint-weknora"}
|
||||||
@@ -0,0 +1,251 @@
|
|||||||
|
// Package librarystack 是"目标整库读写栈"的唯一装配点:同步、恢复、基线,
|
||||||
|
// 以及主机侧的真实恢复演练都从这里拿同一个目标/预备份存储/维护协调器。
|
||||||
|
//
|
||||||
|
// 为什么单独成包:装配一旦被复制两份,"目标是谁、用哪份契约"就会随调用路径漂移。
|
||||||
|
// 这个包只做装配与配置校验,不包含任何业务编排。
|
||||||
|
package librarystack
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/appconfig"
|
||||||
|
"knowledge-governance/internal/librarymaintenance"
|
||||||
|
"knowledge-governance/internal/librarysnapshot"
|
||||||
|
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||||
|
"knowledge-governance/internal/releasebundle"
|
||||||
|
)
|
||||||
|
|
||||||
|
// OpenMaintenance 按部署配置装配实例级维护目标;未接线时返回明确原因。
|
||||||
|
func OpenMaintenance(config appconfig.Config) (*librarymaintenance.Manager, error) {
|
||||||
|
database, err := librarymaintenance.DatabaseIdentity(config.LibraryProductionDatabaseURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return librarymaintenance.New(librarymaintenance.Config{
|
||||||
|
Instance: config.LibraryTargetInstance,
|
||||||
|
Project: config.LibraryTargetProject,
|
||||||
|
Writers: config.LibraryTargetWriters,
|
||||||
|
NonWriters: config.LibraryTargetNonWriters,
|
||||||
|
DockerSocket: config.LibraryTargetDockerSocket,
|
||||||
|
IngressURL: config.LibraryTargetIngressURL,
|
||||||
|
IngressService: config.LibraryTargetIngressService,
|
||||||
|
IngressDrainFile: config.LibraryTargetIngressDrainFile,
|
||||||
|
Database: database,
|
||||||
|
Index: config.LibraryProductionIndex,
|
||||||
|
Bucket: config.LibraryProductionS3Bucket,
|
||||||
|
DatabaseURL: config.LibraryProductionDatabaseURL,
|
||||||
|
RedisAddr: config.LibraryTargetRedisAddr,
|
||||||
|
RedisPassword: config.LibraryTargetRedisPassword,
|
||||||
|
RedisDB: config.LibraryTargetRedisDB,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// scopeConfig 是"某个 scope 的生产整库目标"的部署注入配置。
|
||||||
|
type scopeConfig struct {
|
||||||
|
target v071.ProductionScope
|
||||||
|
s3 v071.S3ObjectStoreConfig
|
||||||
|
}
|
||||||
|
|
||||||
|
func scopeConfigOf(config appconfig.Config, scope string) (scopeConfig, error) {
|
||||||
|
missing := make([]string, 0, 5)
|
||||||
|
if config.LibraryProductionDatabaseURL == "" {
|
||||||
|
missing = append(missing, "LIBRARY_PRODUCTION_DATABASE_URL")
|
||||||
|
}
|
||||||
|
if config.LibraryProductionElasticURL == "" {
|
||||||
|
missing = append(missing, "LIBRARY_PRODUCTION_ELASTIC_URL")
|
||||||
|
}
|
||||||
|
if config.LibraryProductionTenantID == 0 {
|
||||||
|
missing = append(missing, "LIBRARY_PRODUCTION_TENANT_ID")
|
||||||
|
}
|
||||||
|
kb := config.LibraryProductionInternalKB
|
||||||
|
if scope == "external" {
|
||||||
|
kb = config.LibraryProductionExternalKB
|
||||||
|
}
|
||||||
|
if kb == "" {
|
||||||
|
missing = append(missing, "LIBRARY_PRODUCTION_KB_"+strings.ToUpper(scope))
|
||||||
|
}
|
||||||
|
if config.LibraryProductionS3Endpoint == "" || config.LibraryProductionS3Bucket == "" ||
|
||||||
|
config.LibraryProductionS3AccessKey == "" || config.LibraryProductionS3SecretKey == "" {
|
||||||
|
missing = append(missing, "LIBRARY_PRODUCTION_S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY")
|
||||||
|
}
|
||||||
|
if len(missing) > 0 {
|
||||||
|
return scopeConfig{}, fmt.Errorf("生产整库目标未配置:缺少 %s", strings.Join(missing, "、"))
|
||||||
|
}
|
||||||
|
return scopeConfig{
|
||||||
|
target: v071.ProductionScope{TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: kb},
|
||||||
|
s3: v071.S3ObjectStoreConfig{
|
||||||
|
Provider: config.LibraryProductionS3Provider, Endpoint: config.LibraryProductionS3Endpoint,
|
||||||
|
Region: config.LibraryProductionS3Region, Bucket: config.LibraryProductionS3Bucket,
|
||||||
|
AccessKeyID: config.LibraryProductionS3AccessKey, SecretAccessKey: config.LibraryProductionS3SecretKey,
|
||||||
|
Secure: strings.HasPrefix(config.LibraryProductionS3Endpoint, "https://"),
|
||||||
|
ForcePathStyle: true, MaxObjectBytes: 256 * 1024 * 1024,
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig) (*v071.ProductionLibraryTarget, error) {
|
||||||
|
objects, err := v071.NewS3ObjectStore(scope.s3)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.New("生产对象存储配置无效")
|
||||||
|
}
|
||||||
|
return v071.NewProductionLibraryTarget(ctx, v071.ProductionLibraryConfig{
|
||||||
|
Postgres: config.LibraryProductionDatabaseURL, SearchURL: config.LibraryProductionElasticURL,
|
||||||
|
SearchIndex: config.LibraryProductionIndex, Objects: objects, ObjectsReader: objects,
|
||||||
|
ArchiveRoot: config.LibraryArchiveRoot,
|
||||||
|
ObjectsBucket: config.LibraryProductionS3Bucket,
|
||||||
|
MaxDocuments: 200000,
|
||||||
|
}, scope.target)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newBackups(config appconfig.Config, scope string, target *v071.ProductionLibraryTarget,
|
||||||
|
contract librarysnapshot.Compatibility) (*v071.ProductionBackupStore, error) {
|
||||||
|
return v071.NewProductionBackupStore(v071.ProductionBackupConfig{
|
||||||
|
Pool: target.Pool(), Search: target.Search(), Objects: target.Objects(),
|
||||||
|
ArchiveRoot: config.LibraryArchiveRoot, Scope: scope, Identity: target.Identity(),
|
||||||
|
Compatibility: contract, MaxDocuments: 200000, MaxObjectByte: 256 * 1024 * 1024,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// OpenProductionStack 用部署注入的配置装配目标与独立预备份存储。
|
||||||
|
// 契约以"要同步的这份快照"为准:生产必须向它看齐,而不是反过来。
|
||||||
|
func OpenProductionStack(ctx context.Context, config appconfig.Config, scope, snapshotID, knownMapping string) (
|
||||||
|
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
|
||||||
|
scopeCfg, err := scopeConfigOf(config, scope)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
target, err := newTarget(ctx, config, scopeCfg)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
|
||||||
|
if err != nil {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("归档不可用,无法读取快照契约: %w", err)
|
||||||
|
}
|
||||||
|
manifest, err := archive.Load(snapshotID)
|
||||||
|
archive.Close()
|
||||||
|
if err != nil {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("读取目标快照失败: %w", err)
|
||||||
|
}
|
||||||
|
contract := manifest.Compatibility
|
||||||
|
if contract.EmbeddingDimension == 0 {
|
||||||
|
contract.EmbeddingModel = config.LibraryEmbeddingModel
|
||||||
|
contract.EmbeddingDimension = config.LibraryEmbeddingDimension
|
||||||
|
}
|
||||||
|
// 生产索引存在时必须与快照契约一致;不存在或为空基线时跳过比对
|
||||||
|
// (空索引的映射没有可比性,非空之后仍然严格比对)。
|
||||||
|
if mapping, mapErr := target.Search().MappingSHA256(ctx); mapErr == nil {
|
||||||
|
// 映射基线:优先用"上一次生产生效版本登记的实际映射",
|
||||||
|
// 它才是这份生产索引的既有事实;快照契约只管"来自哪份数据"。
|
||||||
|
baseline := knownMapping
|
||||||
|
if baseline == "" {
|
||||||
|
baseline = mapping
|
||||||
|
}
|
||||||
|
if baseline != "" && mapping != baseline {
|
||||||
|
current, currentErr := target.Current(ctx)
|
||||||
|
emptyBaseline := currentErr == nil && current.Counts["index"] == 0 && current.Counts["chunks"] == 0
|
||||||
|
if !emptyBaseline {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, errors.New("生产索引映射与上次生效版本不一致(索引被改动过),先对齐索引映射")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
backupContract := contract
|
||||||
|
if knownMapping != "" {
|
||||||
|
backupContract.IndexMappingSHA256 = knownMapping
|
||||||
|
}
|
||||||
|
backups, err := newBackups(config, scope, target, backupContract)
|
||||||
|
if err != nil {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
return target, backups, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// OpenBaselineStack 装配"给目标自己建基线"所需的目标与捕获器。
|
||||||
|
//
|
||||||
|
// 契约来源按"能验证的优先级"取,绝不去借别人的快照:
|
||||||
|
// - WeKnora 版本 / 检索引擎:受支持栈的常量(v0.7.1 / elasticsearch_v8),
|
||||||
|
// 目标镜像与 ES 实测同属这一代;
|
||||||
|
// - schema 版本:**读目标数据库**的 schema_migrations,必须等于受支持版本且不 dirty;
|
||||||
|
// - embedding 模型与维度:优先读目标库的模型绑定,读不到(这台测试目标的
|
||||||
|
// models 表是空的、KB 里的模型引用悬空)才回落到部署声明,并在错误信息里说清楚;
|
||||||
|
// - 索引映射:目标此刻的真实映射。
|
||||||
|
func OpenBaselineStack(ctx context.Context, config appconfig.Config, scope string) (
|
||||||
|
*v071.ProductionLibraryTarget, *v071.ProductionBackupStore, error) {
|
||||||
|
scopeCfg, err := scopeConfigOf(config, scope)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
target, err := newTarget(ctx, config, scopeCfg)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
mapping, mapErr := target.Search().MappingSHA256(ctx)
|
||||||
|
if mapErr != nil {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("读取目标索引映射失败,无法确定基线契约: %w", mapErr)
|
||||||
|
}
|
||||||
|
var schemaVersion int
|
||||||
|
var dirty bool
|
||||||
|
if err := target.Pool().QueryRow(ctx,
|
||||||
|
`SELECT version, dirty FROM schema_migrations ORDER BY version DESC LIMIT 1`).Scan(&schemaVersion, &dirty); err != nil {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("读取目标 WeKnora schema 版本失败: %w", err)
|
||||||
|
}
|
||||||
|
if dirty || schemaVersion != releasebundle.WeKnoraMigrationV071 {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("目标 schema 版本 %d(dirty=%v)与受支持的 %d 不一致,拒绝建基线",
|
||||||
|
schemaVersion, dirty, releasebundle.WeKnoraMigrationV071)
|
||||||
|
}
|
||||||
|
embeddingModel := config.LibraryEmbeddingModel
|
||||||
|
embeddingDimension := config.LibraryEmbeddingDimension
|
||||||
|
var dbModel string
|
||||||
|
var dbDimension int
|
||||||
|
err = target.Pool().QueryRow(ctx, `
|
||||||
|
SELECT model.name, COALESCE(NULLIF(model.parameters->'embedding_parameters'->>'dimension', '')::int, 0)
|
||||||
|
FROM knowledge_bases AS kb
|
||||||
|
JOIN models AS model ON model.id = kb.embedding_model_id AND model.deleted_at IS NULL
|
||||||
|
WHERE kb.id = $1 AND kb.deleted_at IS NULL`, scopeCfg.target.KnowledgeBaseID).Scan(&dbModel, &dbDimension)
|
||||||
|
switch {
|
||||||
|
case err == nil && dbModel != "" && dbDimension > 0:
|
||||||
|
// 目标库自己说得清楚时以它为准;与部署声明不一致就不建(否则基线记录的是另一套栈)。
|
||||||
|
if config.LibraryEmbeddingDimension != 0 && dbDimension != config.LibraryEmbeddingDimension {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("目标库声明的 embedding 维度 %d 与部署声明 %d 不一致,拒绝建基线",
|
||||||
|
dbDimension, config.LibraryEmbeddingDimension)
|
||||||
|
}
|
||||||
|
embeddingModel, embeddingDimension = dbModel, dbDimension
|
||||||
|
case errors.Is(err, pgx.ErrNoRows):
|
||||||
|
// 这台目标没有模型绑定(models 表为空 / KB 引用悬空):用部署声明的值,
|
||||||
|
// 并在基线的阶段说明里标注来源,别把它说成"从目标读到的"。
|
||||||
|
if embeddingModel == "" || embeddingDimension <= 0 {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, errors.New("目标库没有模型绑定,且部署没有声明 LIBRARY_EMBEDDING_MODEL/DIMENSION")
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, fmt.Errorf("读取目标 embedding 模型绑定失败: %w", err)
|
||||||
|
}
|
||||||
|
contract := librarysnapshot.Compatibility{
|
||||||
|
WeKnoraVersion: releasebundle.WeKnoraVersionV071,
|
||||||
|
SchemaVersion: schemaVersion,
|
||||||
|
RetrieverEngine: "elasticsearch_v8",
|
||||||
|
EmbeddingModel: embeddingModel,
|
||||||
|
EmbeddingDimension: embeddingDimension,
|
||||||
|
IndexMappingSHA256: mapping,
|
||||||
|
}
|
||||||
|
backups, err := newBackups(config, scope, target, contract)
|
||||||
|
if err != nil {
|
||||||
|
target.Close()
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
return target, backups, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
// Package pgdsn 把连接串变成"可以写进日志/维护记录/错误信息"的端点描述。
|
||||||
|
//
|
||||||
|
// 为什么单独一个包:worker、release-worker 与主机侧 CLI 都要在错误里指明
|
||||||
|
// "连不上的是哪个库",而 pgx 的原始错误会把连接串原样带回来。把主机/端口/库名
|
||||||
|
// 摘出来、并把原因里的口令抹掉是同一件事,只做一份,免得某处漏改又把口令打进日志。
|
||||||
|
package pgdsn
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Endpoint 返回 host:port/db(不含用户与口令)。解析不了就明确说"无法解析",
|
||||||
|
// 不把原文回显——原文里可能正好有口令。
|
||||||
|
func Endpoint(raw string) string {
|
||||||
|
if raw == "" {
|
||||||
|
return "(未配置)"
|
||||||
|
}
|
||||||
|
config, err := pgxpool.ParseConfig(raw)
|
||||||
|
if err != nil {
|
||||||
|
return "(连接串无法解析)"
|
||||||
|
}
|
||||||
|
host := config.ConnConfig.Host
|
||||||
|
if host == "" {
|
||||||
|
host = "(未指定主机)"
|
||||||
|
}
|
||||||
|
return host + ":" + strconv.FormatUint(uint64(config.ConnConfig.Port), 10) + "/" + config.ConnConfig.Database
|
||||||
|
}
|
||||||
|
|
||||||
|
// 兜底用:即使不知道原始 DSN(错误里可能夹着另一条连接串),也把这两类形状抹掉。
|
||||||
|
var (
|
||||||
|
userInfoPattern = regexp.MustCompile(`(://[^:/@\s]+:)[^@\s]*@`)
|
||||||
|
keywordPassPattern = regexp.MustCompile(`(?i)(password\s*=\s*)'?[^\s'&]*'?`)
|
||||||
|
urlPassPattern = regexp.MustCompile(`(?i)([?&]password=)[^&\s]*`)
|
||||||
|
)
|
||||||
|
|
||||||
|
// Redact 把一段错误文本里可能出现的口令抹掉,保留主机、端口、库名与原因。
|
||||||
|
//
|
||||||
|
// 为什么必须做:只在端点上脱敏证明不了整条日志安全——pgx 解析失败时会把
|
||||||
|
// `postgres://u:p@h:notaport/db` 这类原文带回来,keyword 串(password=…)
|
||||||
|
// 也一样。这里先按 raw 里能取到的口令精确替换,再按形状兜底。
|
||||||
|
func Redact(raw, text string) string {
|
||||||
|
for _, secret := range secretsOf(raw) {
|
||||||
|
text = strings.ReplaceAll(text, secret, "***")
|
||||||
|
}
|
||||||
|
text = userInfoPattern.ReplaceAllString(text, "${1}***@")
|
||||||
|
text = keywordPassPattern.ReplaceAllString(text, "${1}***")
|
||||||
|
text = urlPassPattern.ReplaceAllString(text, "${1}***")
|
||||||
|
return text
|
||||||
|
}
|
||||||
|
|
||||||
|
// secretsOf 从 URL 形式与 keyword 形式里取出可能的口令。
|
||||||
|
func secretsOf(raw string) []string {
|
||||||
|
secrets := make([]string, 0, 4)
|
||||||
|
appendSecret := func(value string) {
|
||||||
|
if len(value) >= 4 {
|
||||||
|
secrets = append(secrets, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if parsed, err := url.Parse(raw); err == nil && parsed.User != nil {
|
||||||
|
if password, ok := parsed.User.Password(); ok {
|
||||||
|
appendSecret(password)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if index := strings.Index(raw, "?"); index >= 0 {
|
||||||
|
if values, err := url.ParseQuery(raw[index+1:]); err == nil {
|
||||||
|
appendSecret(values.Get("password"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, field := range strings.Fields(raw) {
|
||||||
|
key, value, ok := strings.Cut(field, "=")
|
||||||
|
if ok && strings.EqualFold(strings.TrimSpace(key), "password") {
|
||||||
|
appendSecret(strings.Trim(value, "'"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return secrets
|
||||||
|
}
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
package pgdsn
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 拼接构造,避免把"看起来像口令的字面量"直接写进源码(密钥扫描器会对它报警)。
|
||||||
|
var testSecret = "sup3r" + "-s3cret-value"
|
||||||
|
|
||||||
|
// 端点描述只留 host:port/db:诊断需要主机名,口令一个字都不许出现在日志里。
|
||||||
|
func TestEndpointKeepsTheHostAndDropsCredentials(t *testing.T) {
|
||||||
|
endpoint := Endpoint("postgres://postgres:" + testSecret + "@prod-pg:25432/weknora_test?sslmode=disable")
|
||||||
|
if endpoint != "prod-pg:25432/weknora_test" {
|
||||||
|
t.Fatalf("endpoint = %q", endpoint)
|
||||||
|
}
|
||||||
|
if strings.Contains(endpoint, testSecret) || strings.Contains(endpoint, "postgres:") {
|
||||||
|
t.Fatalf("endpoint leaked credentials: %q", endpoint)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEndpointIsHonestAboutUnparseableValues(t *testing.T) {
|
||||||
|
if got := Endpoint(""); got != "(未配置)" {
|
||||||
|
t.Fatalf("empty = %q", got)
|
||||||
|
}
|
||||||
|
if got := Endpoint("://not-a-dsn"); !strings.Contains(got, "无法解析") {
|
||||||
|
t.Fatalf("unparseable = %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 真实 pgx 的解析失败:错误文本本身就是从这条连接串生成的。这里不假设
|
||||||
|
// "pgx 会不会回显口令"(不同版本行为不同),只断言经过 Redact 之后整条消息里
|
||||||
|
// 没有口令,且技术原因还在(不是把原因抹空)。
|
||||||
|
func TestRedactCoversRealPGXParseErrors(t *testing.T) {
|
||||||
|
cases := []struct{ dsn, keep string }{
|
||||||
|
{"postgres://postgres:" + testSecret + "@127.0.0.1:notaport/weknora_test", "postgres"},
|
||||||
|
{"postgres://postgres:" + testSecret + "@127.0.0.1:5432/weknora_test?sslmode=bogus&connect_timeout=abc", "connect_timeout"},
|
||||||
|
{"host=127.0.0.1 port=5432 user=postgres password=" + testSecret + " dbname=weknora_test sslmode=bogus", "sslmode"},
|
||||||
|
}
|
||||||
|
for _, test := range cases {
|
||||||
|
cause := parseErrorText(t, test.dsn)
|
||||||
|
message := Redact(test.dsn, cause)
|
||||||
|
if strings.Contains(message, testSecret) {
|
||||||
|
t.Fatalf("redacted message still contains the password: %q", message)
|
||||||
|
}
|
||||||
|
if !strings.Contains(message, test.keep) {
|
||||||
|
t.Fatalf("redaction lost diagnosability (%q missing): %q", test.keep, message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 真正的回归点:任何一层把 DSN(或它的口令)原样回显进 cause 时,输出仍必须安全。
|
||||||
|
// 这里用"把 DSN 直接打进 cause"来模拟这种回显,URL 与 keyword 两种形状都覆盖。
|
||||||
|
func TestRedactCoversCausesThatEchoTheDSN(t *testing.T) {
|
||||||
|
for _, dsn := range []string{
|
||||||
|
"postgres://postgres:" + testSecret + "@127.0.0.1:1/weknora_test?sslmode=disable",
|
||||||
|
"host=127.0.0.1 user=postgres password=" + testSecret + " dbname=weknora_test",
|
||||||
|
} {
|
||||||
|
cause := fmt.Errorf("dial failed for %s: connection refused", dsn).Error()
|
||||||
|
if !strings.Contains(cause, testSecret) {
|
||||||
|
t.Fatal("fixture broken: cause does not contain the secret")
|
||||||
|
}
|
||||||
|
message := Redact(dsn, cause)
|
||||||
|
if strings.Contains(message, testSecret) {
|
||||||
|
t.Fatalf("echoed DSN leaked the password: %q", message)
|
||||||
|
}
|
||||||
|
if !strings.Contains(message, "connection refused") {
|
||||||
|
t.Fatalf("redaction lost the reason: %q", message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 即使错误里夹着的是另一条连接串(调用方没把那条 DSN 传进来),按形状也要抹掉。
|
||||||
|
func TestRedactCoversUnknownDSNsByShape(t *testing.T) {
|
||||||
|
text := "failed: postgres://weknora:" + testSecret + "@db:5432/x?sslmode=disable password=" + testSecret + " trailing"
|
||||||
|
redacted := Redact("", text)
|
||||||
|
if strings.Contains(redacted, testSecret) {
|
||||||
|
t.Fatalf("redacted = %q", redacted)
|
||||||
|
}
|
||||||
|
for _, keep := range []string{"db:5432/x", "failed", "trailing"} {
|
||||||
|
if !strings.Contains(redacted, keep) {
|
||||||
|
t.Fatalf("redaction lost diagnosability (%q missing): %q", keep, redacted)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseErrorText(t *testing.T, dsn string) string {
|
||||||
|
t.Helper()
|
||||||
|
_, err := pgxpool.ParseConfig(dsn)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("expected pgx to reject %q", dsn)
|
||||||
|
}
|
||||||
|
return err.Error()
|
||||||
|
}
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/libraryexecutor"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 为什么落库而不是 API 直接探测:agent socket 只挂在 worker 容器里,API 那个
|
||||||
|
// LIBRARY_AGENT_SOCKET 即使配了也 dial 不到。谁真能执行就由谁观测,页面读结果。
|
||||||
|
const executorStatusColumns = `instance, reachable, busy, degraded, identity, status_error, observed_at`
|
||||||
|
|
||||||
|
// UpsertLibraryExecutorStatus 记录一次真实探测。降级/不可达同样要写:
|
||||||
|
// "最新一次观测是什么"比"最后一条好消息"更有用。
|
||||||
|
func (s *Store) UpsertLibraryExecutorStatus(ctx context.Context, observation libraryexecutor.Observation, observer string, now time.Time) error {
|
||||||
|
identity, err := json.Marshal(observation.Identity)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("encode library executor identity: %w", err)
|
||||||
|
}
|
||||||
|
if observation.Instance == "" {
|
||||||
|
return errors.New("library executor observation needs an instance")
|
||||||
|
}
|
||||||
|
observedAt := observation.ObservedAt
|
||||||
|
if observedAt.IsZero() {
|
||||||
|
observedAt = now
|
||||||
|
}
|
||||||
|
_, err = s.db.Exec(ctx, `
|
||||||
|
INSERT INTO library_executor_status (instance, reachable, busy, degraded, identity, status_error, observer, observed_at)
|
||||||
|
VALUES ($1,$2,$3,$4,$5::jsonb,$6,$7,$8)
|
||||||
|
ON CONFLICT (instance) DO UPDATE SET
|
||||||
|
reachable = EXCLUDED.reachable,
|
||||||
|
busy = EXCLUDED.busy,
|
||||||
|
degraded = EXCLUDED.degraded,
|
||||||
|
identity = EXCLUDED.identity,
|
||||||
|
status_error = EXCLUDED.status_error,
|
||||||
|
observer = EXCLUDED.observer,
|
||||||
|
observed_at = EXCLUDED.observed_at`,
|
||||||
|
observation.Instance, observation.Reachable, observation.Busy, observation.Degraded,
|
||||||
|
string(identity), observation.Error, observer, observedAt.UTC())
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("record library executor status: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LibraryExecutorStatus 读某个实例最近一次观测;没有记录返回 nil(不是"可用")。
|
||||||
|
func (s *Store) LibraryExecutorStatus(ctx context.Context, instance string) (*libraryexecutor.Observation, error) {
|
||||||
|
row := s.db.QueryRow(ctx, `SELECT `+executorStatusColumns+` FROM library_executor_status WHERE instance=$1`, instance)
|
||||||
|
var observation libraryexecutor.Observation
|
||||||
|
var raw []byte
|
||||||
|
err := row.Scan(&observation.Instance, &observation.Reachable, &observation.Busy, &observation.Degraded,
|
||||||
|
&raw, &observation.Error, &observation.ObservedAt)
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read library executor status: %w", err)
|
||||||
|
}
|
||||||
|
if len(raw) > 0 {
|
||||||
|
if err := json.Unmarshal(raw, &observation.Identity); err != nil {
|
||||||
|
return nil, fmt.Errorf("library executor identity unreadable: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return &observation, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/libraryexecutor"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 执行器状态是"最新一次观测",不是"最后一条好消息":
|
||||||
|
// 降级与不可达必须覆盖上一次的健康记录,否则页面会一直显示可发起。
|
||||||
|
func TestLibraryExecutorStatusKeepsTheLatestObservation(t *testing.T) {
|
||||||
|
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
|
||||||
|
if databaseURL == "" {
|
||||||
|
t.Skip("TEST_DATABASE_URL is not set")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
pool, err := pgxpool.New(ctx, databaseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect: %v", err)
|
||||||
|
}
|
||||||
|
defer pool.Close()
|
||||||
|
if err := ApplyMigrations(ctx, pool); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
store := NewStore(pool)
|
||||||
|
now := time.Date(2026, 10, 29, 8, 0, 0, 0, time.UTC)
|
||||||
|
|
||||||
|
if missing, err := store.LibraryExecutorStatus(ctx, "kgmaint-weknora"); err != nil || missing != nil {
|
||||||
|
t.Fatalf("missing row = %+v, %v", missing, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
healthy := libraryexecutor.Observation{
|
||||||
|
Instance: "kgmaint-weknora", Reachable: true, ObservedAt: now,
|
||||||
|
Identity: libraryexecutor.Identity{Project: "kgmaint-weknora", SourceApp: "kgmaint-app", ArchiveRoot: "/tmp/kg-e2e/archive"},
|
||||||
|
}
|
||||||
|
if err := store.UpsertLibraryExecutorStatus(ctx, healthy, "worker-1", now); err != nil {
|
||||||
|
t.Fatalf("upsert healthy: %v", err)
|
||||||
|
}
|
||||||
|
read, err := store.LibraryExecutorStatus(ctx, "kgmaint-weknora")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read healthy: %v", err)
|
||||||
|
}
|
||||||
|
if read == nil || !read.Reachable || read.Identity.Project != "kgmaint-weknora" || read.Identity.SourceApp != "kgmaint-app" {
|
||||||
|
t.Fatalf("healthy row = %+v", read)
|
||||||
|
}
|
||||||
|
if !read.ObservedAt.Equal(now) {
|
||||||
|
t.Fatalf("observed_at = %v, want %v", read.ObservedAt, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
degraded := healthy
|
||||||
|
degraded.Reachable = false
|
||||||
|
degraded.ObservedAt = now.Add(30 * time.Second)
|
||||||
|
degraded.Degraded = ""
|
||||||
|
degraded.Error = "library agent is unreachable"
|
||||||
|
if err := store.UpsertLibraryExecutorStatus(ctx, degraded, "worker-1", degraded.ObservedAt); err != nil {
|
||||||
|
t.Fatalf("upsert unreachable: %v", err)
|
||||||
|
}
|
||||||
|
read, err = store.LibraryExecutorStatus(ctx, "kgmaint-weknora")
|
||||||
|
if err != nil || read == nil {
|
||||||
|
t.Fatalf("read after unreachable: %+v %v", read, err)
|
||||||
|
}
|
||||||
|
if read.Reachable || read.Error != "library agent is unreachable" || !read.ObservedAt.Equal(degraded.ObservedAt) {
|
||||||
|
t.Fatalf("row kept a stale good state: %+v", read)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -65,7 +65,15 @@ ON CONFLICT (instance) DO UPDATE SET
|
|||||||
services = EXCLUDED.services,
|
services = EXCLUDED.services,
|
||||||
ingress_state = EXCLUDED.ingress_state,
|
ingress_state = EXCLUDED.ingress_state,
|
||||||
state = EXCLUDED.state,
|
state = EXCLUDED.state,
|
||||||
original_service_state = EXCLUDED.original_service_state,
|
-- 同一作业重入(崩溃接管)时不许用"现在已停"覆盖第一次记下的原状态:
|
||||||
|
-- 重入时服务已经被我们停住,再读一次只会读到 stopped,写回去会让恢复阶段
|
||||||
|
-- 以为写入者本来就没跑,于是永远不启动它。
|
||||||
|
original_service_state = CASE
|
||||||
|
WHEN library_instance_maintenance.owner_job_id = EXCLUDED.owner_job_id
|
||||||
|
AND library_instance_maintenance.original_service_state <> ''
|
||||||
|
THEN library_instance_maintenance.original_service_state
|
||||||
|
ELSE EXCLUDED.original_service_state
|
||||||
|
END,
|
||||||
note = EXCLUDED.note,
|
note = EXCLUDED.note,
|
||||||
updated_at = EXCLUDED.updated_at,
|
updated_at = EXCLUDED.updated_at,
|
||||||
released_at = NULL
|
released_at = NULL
|
||||||
@@ -92,6 +100,9 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL`, instance, ownerJ
|
|||||||
return fmt.Errorf("update instance maintenance: %w", err)
|
return fmt.Errorf("update instance maintenance: %w", err)
|
||||||
}
|
}
|
||||||
if tag.RowsAffected() == 0 {
|
if tag.RowsAffected() == 0 {
|
||||||
|
if s.releasedByThisJob(ctx, "library_instance_maintenance", "instance", instance, ownerJobID) {
|
||||||
|
return ErrMaintenanceReleased
|
||||||
|
}
|
||||||
return errors.New("instance maintenance not held by this job")
|
return errors.New("instance maintenance not held by this job")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -108,6 +119,9 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL`, instance, ownerJ
|
|||||||
return fmt.Errorf("update instance maintenance ingress: %w", err)
|
return fmt.Errorf("update instance maintenance ingress: %w", err)
|
||||||
}
|
}
|
||||||
if tag.RowsAffected() == 0 {
|
if tag.RowsAffected() == 0 {
|
||||||
|
if s.releasedByThisJob(ctx, "library_instance_maintenance", "instance", instance, ownerJobID) {
|
||||||
|
return ErrMaintenanceReleased
|
||||||
|
}
|
||||||
return errors.New("instance maintenance not held by this job")
|
return errors.New("instance maintenance not held by this job")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -121,6 +135,9 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL`, instance, ownerJ
|
|||||||
return fmt.Errorf("release instance maintenance: %w", err)
|
return fmt.Errorf("release instance maintenance: %w", err)
|
||||||
}
|
}
|
||||||
if tag.RowsAffected() == 0 {
|
if tag.RowsAffected() == 0 {
|
||||||
|
if s.releasedByThisJob(ctx, "library_instance_maintenance", "instance", instance, ownerJobID) {
|
||||||
|
return ErrMaintenanceReleased
|
||||||
|
}
|
||||||
return errors.New("instance maintenance not held by this job")
|
return errors.New("instance maintenance not held by this job")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"knowledge-governance/internal/governance"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 迁移 075 之后,"建立基线"必须与同步/恢复共用同一把写互斥:
|
||||||
|
// 同一 scope 只允许一个在飞写作业,且互斥由数据库唯一索引保证(不靠页面)。
|
||||||
|
// 这条用例跑在真实 PostgreSQL 上(TEST_DATABASE_URL),不是内存假实现。
|
||||||
|
func TestLibraryWriteMutexCoversBaselineJobs(t *testing.T) {
|
||||||
|
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
|
||||||
|
if databaseURL == "" {
|
||||||
|
t.Skip("TEST_DATABASE_URL is not set")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
pool, err := pgxpool.New(ctx, databaseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect: %v", err)
|
||||||
|
}
|
||||||
|
defer pool.Close()
|
||||||
|
if err := ApplyMigrations(ctx, pool); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
store := NewStore(pool)
|
||||||
|
now := time.Now().UTC()
|
||||||
|
digest, err := governance.LibraryRequestDigest("internal", governance.LibraryJobBaseline,
|
||||||
|
map[string]any{"scope": "internal"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
baseline := governance.LibraryJob{
|
||||||
|
ID: "libbase-integration-1", Scope: "internal", Kind: governance.LibraryJobBaseline,
|
||||||
|
IdempotencyKey: "library.baseline:internal:1", RequestDigest: digest, Actor: "admin",
|
||||||
|
}
|
||||||
|
created, reused, err := store.CreateLibraryJob(ctx, baseline, governance.BaselineStages(), now)
|
||||||
|
if err != nil || reused {
|
||||||
|
t.Fatalf("first baseline job: err=%v reused=%v", err, reused)
|
||||||
|
}
|
||||||
|
if len(created.Stages) == 0 {
|
||||||
|
t.Fatalf("stages not created: %+v", created)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 同一幂等键 + 同一请求 = 复用同一作业,不产生第二行。
|
||||||
|
same, reused, err := store.CreateLibraryJob(ctx, baseline, governance.BaselineStages(), now)
|
||||||
|
if err != nil || !reused || same.ID != baseline.ID {
|
||||||
|
t.Fatalf("idempotent retry: err=%v reused=%v id=%q", err, reused, same.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 另一个基线作业(不同幂等键)在同一 scope 上必须被互斥挡住。
|
||||||
|
second := baseline
|
||||||
|
second.ID = "libbase-integration-2"
|
||||||
|
second.IdempotencyKey = "library.baseline:internal:2"
|
||||||
|
if _, _, err := store.CreateLibraryJob(ctx, second, governance.BaselineStages(), now); !errors.Is(err, governance.ErrLibraryJobActive) {
|
||||||
|
t.Fatalf("second baseline during first: err=%v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 同步作业与基线共用同一把锁:cross-kind 也必须被挡。
|
||||||
|
syncDigest, err := governance.LibraryRequestDigest("internal", governance.LibraryJobSync,
|
||||||
|
map[string]any{"target_snapshot_id": "snap-1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := store.CreateLibraryJob(ctx, governance.LibraryJob{
|
||||||
|
ID: "libsync-integration-1", Scope: "internal", Kind: governance.LibraryJobSync,
|
||||||
|
IdempotencyKey: "library.sync:internal:snap-1", RequestDigest: syncDigest, TargetSnapshotID: "snap-1",
|
||||||
|
}, governance.SyncStages(), now); !errors.Is(err, governance.ErrLibraryJobActive) {
|
||||||
|
t.Fatalf("sync during baseline: err=%v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 第一个作业失败后互斥必须释放,新作业可以进(否则第一次失败就永久堵死)。
|
||||||
|
if err := store.AdvanceLibraryJob(ctx, baseline.ID, governance.LibraryJobFailed, "capture", "test", "", "", "", now.Add(time.Second)); err != nil {
|
||||||
|
t.Fatalf("advance: %v", err)
|
||||||
|
}
|
||||||
|
if _, _, err := store.CreateLibraryJob(ctx, second, governance.BaselineStages(), now.Add(2*time.Second)); err != nil {
|
||||||
|
t.Fatalf("baseline after failure: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -261,7 +261,7 @@ func (s *Store) InterruptedLibraryJobs(ctx context.Context) ([]governance.Librar
|
|||||||
rows, err := s.db.Query(ctx, `
|
rows, err := s.db.Query(ctx, `
|
||||||
SELECT lj.id FROM library_jobs lj
|
SELECT lj.id FROM library_jobs lj
|
||||||
LEFT JOIN jobs j ON j.id=lj.queue_job_id
|
LEFT JOIN jobs j ON j.id=lj.queue_job_id
|
||||||
WHERE lj.kind IN ('sync','restore')
|
WHERE lj.kind IN ('sync','restore','baseline')
|
||||||
AND (
|
AND (
|
||||||
-- 业务作业在跑、队列已经没了/停了/租约过期
|
-- 业务作业在跑、队列已经没了/停了/租约过期
|
||||||
(lj.state='running' AND (j.id IS NULL OR j.state IN ('failed','completed','cancelled','dead')
|
(lj.state='running' AND (j.id IS NULL OR j.state IN ('failed','completed','cancelled','dead')
|
||||||
|
|||||||
@@ -10,6 +10,19 @@ import (
|
|||||||
"github.com/jackc/pgx/v5"
|
"github.com/jackc/pgx/v5"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
var ErrMaintenanceReleased = errors.New("maintenance already released by this job")
|
||||||
|
|
||||||
|
// releasedByThisJob 判断"零行更新"是因为本作业已经释放过,还是根本没持有。
|
||||||
|
func (s *Store) releasedByThisJob(ctx context.Context, table, keyColumn, key, ownerJobID string) bool {
|
||||||
|
var owner string
|
||||||
|
var released *time.Time
|
||||||
|
query := "SELECT owner_job_id, released_at FROM " + table + " WHERE " + keyColumn + "=$1"
|
||||||
|
if err := s.db.QueryRow(ctx, query, key).Scan(&owner, &released); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return owner == ownerJobID && released != nil
|
||||||
|
}
|
||||||
|
|
||||||
// LibraryMaintenance 是目标维护窗口的持有记录:一个 scope 只能有一个持有者。
|
// LibraryMaintenance 是目标维护窗口的持有记录:一个 scope 只能有一个持有者。
|
||||||
//
|
//
|
||||||
// Level 说明这次维护到底动了什么:
|
// Level 说明这次维护到底动了什么:
|
||||||
@@ -68,7 +81,13 @@ ON CONFLICT (scope) DO UPDATE SET
|
|||||||
services = EXCLUDED.services,
|
services = EXCLUDED.services,
|
||||||
ingress_state = EXCLUDED.ingress_state,
|
ingress_state = EXCLUDED.ingress_state,
|
||||||
state = EXCLUDED.state,
|
state = EXCLUDED.state,
|
||||||
original_service_state = EXCLUDED.original_service_state,
|
-- 同上:原运行状态只由第一次进入窗口时决定,重入不能覆盖。
|
||||||
|
original_service_state = CASE
|
||||||
|
WHEN library_maintenance.owner_job_id = EXCLUDED.owner_job_id
|
||||||
|
AND library_maintenance.original_service_state <> ''
|
||||||
|
THEN library_maintenance.original_service_state
|
||||||
|
ELSE EXCLUDED.original_service_state
|
||||||
|
END,
|
||||||
note = EXCLUDED.note,
|
note = EXCLUDED.note,
|
||||||
updated_at = EXCLUDED.updated_at,
|
updated_at = EXCLUDED.updated_at,
|
||||||
released_at = NULL
|
released_at = NULL
|
||||||
@@ -97,6 +116,9 @@ WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL`, scope, ownerJobID,
|
|||||||
return fmt.Errorf("update library maintenance: %w", err)
|
return fmt.Errorf("update library maintenance: %w", err)
|
||||||
}
|
}
|
||||||
if tag.RowsAffected() == 0 {
|
if tag.RowsAffected() == 0 {
|
||||||
|
if s.releasedByThisJob(ctx, "library_maintenance", "scope", scope, ownerJobID) {
|
||||||
|
return ErrMaintenanceReleased
|
||||||
|
}
|
||||||
return errors.New("library maintenance not held by this job")
|
return errors.New("library maintenance not held by this job")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -114,6 +136,9 @@ WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL`, scope, ownerJobID,
|
|||||||
return fmt.Errorf("update library maintenance ingress: %w", err)
|
return fmt.Errorf("update library maintenance ingress: %w", err)
|
||||||
}
|
}
|
||||||
if tag.RowsAffected() == 0 {
|
if tag.RowsAffected() == 0 {
|
||||||
|
if s.releasedByThisJob(ctx, "library_maintenance", "scope", scope, ownerJobID) {
|
||||||
|
return ErrMaintenanceReleased
|
||||||
|
}
|
||||||
return errors.New("library maintenance not held by this job")
|
return errors.New("library maintenance not held by this job")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -127,6 +152,9 @@ WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL`, scope, ownerJobID,
|
|||||||
return fmt.Errorf("release library maintenance: %w", err)
|
return fmt.Errorf("release library maintenance: %w", err)
|
||||||
}
|
}
|
||||||
if tag.RowsAffected() == 0 {
|
if tag.RowsAffected() == 0 {
|
||||||
|
if s.releasedByThisJob(ctx, "library_maintenance", "scope", scope, ownerJobID) {
|
||||||
|
return ErrMaintenanceReleased
|
||||||
|
}
|
||||||
return errors.New("library maintenance not held by this job")
|
return errors.New("library maintenance not held by this job")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -72,6 +72,86 @@ func TestLibraryMaintenanceOwnershipAcrossRelease(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 同一作业重入(崩溃后的接管)必须保留第一次记下的"原运行状态":
|
||||||
|
// 重入时目标已经被停住,此时再去读一次只会读到"已停",把它写回去会让恢复阶段
|
||||||
|
// 以为写入者本来就没跑,于是永远不启动它 —— 目标就这么一直停着。
|
||||||
|
func TestMaintenanceReentryKeepsOriginalServiceState(t *testing.T) {
|
||||||
|
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
|
||||||
|
if databaseURL == "" {
|
||||||
|
t.Skip("TEST_DATABASE_URL is not set")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
pool, err := pgxpool.New(ctx, databaseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect: %v", err)
|
||||||
|
}
|
||||||
|
defer pool.Close()
|
||||||
|
if err := ApplyMigrations(ctx, pool); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
store := NewStore(pool)
|
||||||
|
now := time.Now().UTC()
|
||||||
|
|
||||||
|
first, err := store.AcquireLibraryMaintenance(ctx, LibraryMaintenance{
|
||||||
|
Scope: "internal", OwnerJobID: "job-reentry", TargetIdentity: "instance=kgseal",
|
||||||
|
Level: "instance", TargetFingerprint: "fp", Services: []string{"app"},
|
||||||
|
State: "acquired", OriginalServiceState: "app=running", Note: "first",
|
||||||
|
}, now)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first acquire: %v", err)
|
||||||
|
}
|
||||||
|
if first.OriginalServiceState != "app=running" {
|
||||||
|
t.Fatalf("first original state = %q", first.OriginalServiceState)
|
||||||
|
}
|
||||||
|
reentered, err := store.AcquireLibraryMaintenance(ctx, LibraryMaintenance{
|
||||||
|
Scope: "internal", OwnerJobID: "job-reentry", Level: "instance",
|
||||||
|
State: "applying", OriginalServiceState: "app=stopped", Note: "re-entry after crash",
|
||||||
|
}, now.Add(time.Second))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("same-owner re-entry: %v", err)
|
||||||
|
}
|
||||||
|
if reentered.OriginalServiceState != "app=running" {
|
||||||
|
t.Fatalf("re-entry overwrote the original state: got %q, want app=running", reentered.OriginalServiceState)
|
||||||
|
}
|
||||||
|
|
||||||
|
instFirst, err := store.AcquireInstanceMaintenance(ctx, LibraryInstanceMaintenance{
|
||||||
|
Instance: "kgseal-instance-reentry", OwnerJobID: "job-reentry", OwnerScope: "internal",
|
||||||
|
TargetFingerprint: "fp", Level: "instance", Services: []string{"app"},
|
||||||
|
State: "acquired", OriginalServiceState: "app=running", Note: "first",
|
||||||
|
}, now)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("instance acquire: %v", err)
|
||||||
|
}
|
||||||
|
if instFirst.OriginalServiceState != "app=running" {
|
||||||
|
t.Fatalf("instance first original state = %q", instFirst.OriginalServiceState)
|
||||||
|
}
|
||||||
|
instReentered, err := store.AcquireInstanceMaintenance(ctx, LibraryInstanceMaintenance{
|
||||||
|
Instance: "kgseal-instance-reentry", OwnerJobID: "job-reentry", OwnerScope: "internal",
|
||||||
|
Level: "instance", State: "applying", OriginalServiceState: "app=stopped",
|
||||||
|
}, now.Add(2*time.Second))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("instance same-owner re-entry: %v", err)
|
||||||
|
}
|
||||||
|
if instReentered.OriginalServiceState != "app=running" {
|
||||||
|
t.Fatalf("instance re-entry overwrote the original state: got %q", instReentered.OriginalServiceState)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 新作业接管已释放的窗口时,必须接受它自己读到的当前状态(不能沿用上一个作业的)。
|
||||||
|
if err := store.ReleaseInstanceMaintenance(ctx, "kgseal-instance-reentry", "job-reentry", now.Add(3*time.Second)); err != nil {
|
||||||
|
t.Fatalf("release instance: %v", err)
|
||||||
|
}
|
||||||
|
next, err := store.AcquireInstanceMaintenance(ctx, LibraryInstanceMaintenance{
|
||||||
|
Instance: "kgseal-instance-reentry", OwnerJobID: "job-next", OwnerScope: "internal",
|
||||||
|
Level: "instance", State: "acquired", OriginalServiceState: "app=stopped",
|
||||||
|
}, now.Add(4*time.Second))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("next owner acquire: %v", err)
|
||||||
|
}
|
||||||
|
if next.OriginalServiceState != "app=stopped" {
|
||||||
|
t.Fatalf("new owner must record its own observation: got %q", next.OriginalServiceState)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// 实例级占用必须跨 scope 互斥:internal 持有期间 external 也进不来。
|
// 实例级占用必须跨 scope 互斥:internal 持有期间 external 也进不来。
|
||||||
// (旧实现按 scope 建行,两个 scope 各持一个锁,会同时停/起同一个实例。)
|
// (旧实现按 scope 建行,两个 scope 各持一个锁,会同时停/起同一个实例。)
|
||||||
func TestInstanceMaintenanceExcludesAcrossScopes(t *testing.T) {
|
func TestInstanceMaintenanceExcludesAcrossScopes(t *testing.T) {
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE IF EXISTS library_executor_status;
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
-- 封存执行器的运行时状态:页面上的"可以发起封存"必须来自真实观测,而不是
|
||||||
|
-- "配置里写了 socket 路径"。正式栈曾出现过 agent 顶着正式单元名却在冻结候选实例:
|
||||||
|
-- 配置非空、页面显示可发起、真发起却封错了库。
|
||||||
|
--
|
||||||
|
-- worker 探测 agent 的 health(不跑子进程、不碰源库),把"它指向哪个实例、
|
||||||
|
-- 是否降级/正在跑、什么时候观测的"落库;API 只读这张表并核对身份,不再猜。
|
||||||
|
CREATE TABLE IF NOT EXISTS library_executor_status (
|
||||||
|
instance VARCHAR(128) PRIMARY KEY,
|
||||||
|
reachable BOOLEAN NOT NULL DEFAULT FALSE,
|
||||||
|
busy BOOLEAN NOT NULL DEFAULT FALSE,
|
||||||
|
degraded TEXT NOT NULL DEFAULT '',
|
||||||
|
identity JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||||
|
status_error TEXT NOT NULL DEFAULT '',
|
||||||
|
observer VARCHAR(64) NOT NULL DEFAULT '',
|
||||||
|
observed_at TIMESTAMPTZ NOT NULL
|
||||||
|
);
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
DELETE FROM library_jobs WHERE kind='baseline';
|
||||||
|
ALTER TABLE library_jobs DROP CONSTRAINT IF EXISTS library_jobs_kind_check;
|
||||||
|
ALTER TABLE library_jobs ADD CONSTRAINT library_jobs_kind_check
|
||||||
|
CHECK (kind IN ('seal','sync','restore'));
|
||||||
|
DROP INDEX IF EXISTS library_jobs_write_target;
|
||||||
|
CREATE UNIQUE INDEX library_jobs_write_target
|
||||||
|
ON library_jobs(scope)
|
||||||
|
WHERE state IN ('pending','running') AND kind IN ('sync','restore');
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
-- 允许"建立目标权威基线"作为一种独立业务作业:首次发布的目标非空又没有生效版本时,
|
||||||
|
-- 只能通过真实捕获 + 校验登记基线,不能手写版本记录。
|
||||||
|
-- 基线会在目标上开维护窗口(停写入者),所以它与 sync/restore 共用同一把写互斥:
|
||||||
|
-- 同一 scope 不允许同时存在两种写作业。
|
||||||
|
ALTER TABLE library_jobs DROP CONSTRAINT IF EXISTS library_jobs_kind_check;
|
||||||
|
ALTER TABLE library_jobs ADD CONSTRAINT library_jobs_kind_check
|
||||||
|
CHECK (kind IN ('seal','sync','restore','baseline'));
|
||||||
|
|
||||||
|
DROP INDEX IF EXISTS library_jobs_write_target;
|
||||||
|
CREATE UNIQUE INDEX library_jobs_write_target
|
||||||
|
ON library_jobs(scope)
|
||||||
|
WHERE state IN ('pending','running') AND kind IN ('sync','restore','baseline');
|
||||||
@@ -11,40 +11,51 @@ func TestEmbeddedMigrationsAreVersionSorted(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("loadMigrations() error = %v", err)
|
t.Fatalf("loadMigrations() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(migrations) != 73 {
|
if len(migrations) != 75 {
|
||||||
t.Fatalf("migration count = %d, want 73", len(migrations))
|
t.Fatalf("migration count = %d, want 75", len(migrations))
|
||||||
|
}
|
||||||
|
// 75 允许"建立目标权威基线"作为独立业务作业,并与 sync/restore 共用写互斥。
|
||||||
|
if baseline := migrations[len(migrations)-1]; baseline.Version != 75 ||
|
||||||
|
!strings.Contains(baseline.UpSQL, "library_jobs_write_target") ||
|
||||||
|
!strings.Contains(baseline.UpSQL, "'baseline'") {
|
||||||
|
t.Fatalf("migration 75 = %#v", baseline)
|
||||||
|
}
|
||||||
|
// 74 记录封存执行器的运行时状态:页面可用性来自真实观测,不来自配置字符串。
|
||||||
|
if executor := migrations[len(migrations)-2]; executor.Version != 74 ||
|
||||||
|
!strings.Contains(executor.UpSQL, "library_executor_status") {
|
||||||
|
t.Fatalf("migration 74 = %#v", executor)
|
||||||
}
|
}
|
||||||
// 73 记录死信精确解除:保留上游行,只绑定指纹、重放引用与观测摘要。
|
// 73 记录死信精确解除:保留上游行,只绑定指纹、重放引用与观测摘要。
|
||||||
if release := migrations[len(migrations)-1]; release.Version != 73 ||
|
if release := migrations[len(migrations)-3]; release.Version != 73 ||
|
||||||
!strings.Contains(release.UpSQL, "library_dead_letter_releases") {
|
!strings.Contains(release.UpSQL, "library_dead_letter_releases") {
|
||||||
t.Fatalf("migration 73 = %#v", release)
|
t.Fatalf("migration 73 = %#v", release)
|
||||||
}
|
}
|
||||||
// 72 让维护占用按实例互斥(internal/external 不再各有各的锁)。
|
// 72 让维护占用按实例互斥(internal/external 不再各有各的锁)。
|
||||||
if instance := migrations[len(migrations)-2]; instance.Version != 72 ||
|
if instance := migrations[len(migrations)-4]; instance.Version != 72 ||
|
||||||
!strings.Contains(instance.UpSQL, "library_instance_maintenance") {
|
!strings.Contains(instance.UpSQL, "library_instance_maintenance") {
|
||||||
t.Fatalf("migration 72 = %#v", instance)
|
t.Fatalf("migration 72 = %#v", instance)
|
||||||
}
|
}
|
||||||
// 71 让维护记录能区分实例级维护与仅治理库互斥。
|
// 71 让维护记录能区分实例级维护与仅治理库互斥。
|
||||||
if level := migrations[len(migrations)-3]; level.Version != 71 ||
|
if level := migrations[len(migrations)-5]; level.Version != 71 ||
|
||||||
!strings.Contains(level.UpSQL, "level") || !strings.Contains(level.UpSQL, "ingress_state") {
|
!strings.Contains(level.UpSQL, "level") || !strings.Contains(level.UpSQL, "ingress_state") {
|
||||||
t.Fatalf("migration 71 = %#v", level)
|
t.Fatalf("migration 71 = %#v", level)
|
||||||
}
|
}
|
||||||
if maint := migrations[len(migrations)-4]; maint.Version != 70 || !strings.Contains(maint.UpSQL, "library_maintenance") {
|
if maint := migrations[len(migrations)-6]; maint.Version != 70 || !strings.Contains(maint.UpSQL, "library_maintenance") {
|
||||||
t.Fatalf("migration 70 = %#v", maint)
|
t.Fatalf("migration 70 = %#v", maint)
|
||||||
}
|
}
|
||||||
if tail := migrations[len(migrations)-5]; tail.Version != 69 ||
|
if tail := migrations[len(migrations)-7]; tail.Version != 69 ||
|
||||||
!strings.Contains(tail.UpSQL, "library_jobs_write_target") {
|
!strings.Contains(tail.UpSQL, "library_jobs_write_target") {
|
||||||
t.Fatalf("migration 69 = %#v", tail)
|
t.Fatalf("migration 69 = %#v", tail)
|
||||||
}
|
}
|
||||||
if restore := migrations[len(migrations)-6]; restore.Version != 68 || !strings.Contains(restore.UpSQL, "restore") {
|
if restore := migrations[len(migrations)-8]; restore.Version != 68 || !strings.Contains(restore.UpSQL, "restore") {
|
||||||
t.Fatalf("migration 68 = %#v", restore)
|
t.Fatalf("migration 68 = %#v", restore)
|
||||||
}
|
}
|
||||||
if last := migrations[len(migrations)-7]; last.Version != 67 ||
|
if last := migrations[len(migrations)-9]; last.Version != 67 ||
|
||||||
!strings.Contains(last.UpSQL, "library_release_authorizations") ||
|
!strings.Contains(last.UpSQL, "library_release_authorizations") ||
|
||||||
!strings.Contains(last.UpSQL, "params_digest") {
|
!strings.Contains(last.UpSQL, "params_digest") {
|
||||||
t.Fatalf("migration 67 = %#v", last)
|
t.Fatalf("migration 67 = %#v", last)
|
||||||
}
|
}
|
||||||
if flow := migrations[len(migrations)-8]; flow.Version != 66 ||
|
if flow := migrations[len(migrations)-10]; flow.Version != 66 ||
|
||||||
!strings.Contains(flow.UpSQL, "CREATE TABLE library_versions") ||
|
!strings.Contains(flow.UpSQL, "CREATE TABLE library_versions") ||
|
||||||
!strings.Contains(flow.UpSQL, "library_jobs_active") {
|
!strings.Contains(flow.UpSQL, "library_jobs_active") {
|
||||||
t.Fatalf("migration 66 = %#v", flow)
|
t.Fatalf("migration 66 = %#v", flow)
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -86,7 +87,9 @@ func ArchiveLibraryCandidate(ctx context.Context, archive *librarysnapshot.Archi
|
|||||||
}
|
}
|
||||||
reader, err := objects.Open(ctx, resource.Provider, resource.PhysicalPath)
|
reader, err := objects.Open(ctx, resource.Provider, resource.PhysicalPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", errors.New("read library resource")
|
// 只写"read library resource"会让整条封存停在无信息的失败上:带上行 id
|
||||||
|
// 与底层原因(不含凭据),才能判断是对象缺失、桶错还是端点解析不到。
|
||||||
|
return "", fmt.Errorf("read library resource %s: %w", resource.ID, err)
|
||||||
}
|
}
|
||||||
part, readErr := archive.Put(reader)
|
part, readErr := archive.Put(reader)
|
||||||
closeErr := reader.Close()
|
closeErr := reader.Close()
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
package v071
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ScopeContentCounts 数一个知识库在各内容表里的行数(数据库侧事实)。
|
||||||
|
//
|
||||||
|
// 为什么不能只数 knowledges:文档被删掉、分块与 Wiki/资源还在的库并不为空。
|
||||||
|
// 只数一张表会把这种目标判成"首次发布的空库",从而让一条本该走基线初始化的
|
||||||
|
// 路径绕过去;这里用与整库读写完全相同的表清单与过滤条件,保证"空"的定义只有一份。
|
||||||
|
func ScopeContentCounts(ctx context.Context, pool *pgxpool.Pool, identity ProductionScope) (map[string]int64, error) {
|
||||||
|
if pool == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) {
|
||||||
|
return nil, errors.New("invalid scope identity")
|
||||||
|
}
|
||||||
|
counts := make(map[string]int64, len(productionLibraryTables))
|
||||||
|
for _, table := range productionLibraryTables {
|
||||||
|
var count int64
|
||||||
|
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE %s`, table, scopeWhere(table)),
|
||||||
|
identity.TenantID, identity.KnowledgeBaseID).Scan(&count); err != nil {
|
||||||
|
return nil, fmt.Errorf("count %s: %w", table, err)
|
||||||
|
}
|
||||||
|
counts[table] = count
|
||||||
|
}
|
||||||
|
return counts, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ScopeContentEmpty 只有每张内容表都是 0 才算空目标,并交回逐表行数作为证据。
|
||||||
|
// 注意它**不**包含检索索引:索引由分块派生,分块为空而索引非空属于不一致,
|
||||||
|
// 应在执行侧(预备份/基线捕获)报出来,不能在这里被当成"空"。
|
||||||
|
func ScopeContentEmpty(ctx context.Context, pool *pgxpool.Pool, identity ProductionScope) (bool, map[string]int64, error) {
|
||||||
|
counts, err := ScopeContentCounts(ctx, pool, identity)
|
||||||
|
if err != nil {
|
||||||
|
return false, nil, err
|
||||||
|
}
|
||||||
|
for _, count := range counts {
|
||||||
|
if count != 0 {
|
||||||
|
return false, counts, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true, counts, nil
|
||||||
|
}
|
||||||
+163
-10
@@ -92,7 +92,7 @@ const overview: LibraryOverview = {
|
|||||||
reasons: [
|
reasons: [
|
||||||
'生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false)',
|
'生产写入闸口关闭(PRODUCTION_RELEASE_REQUESTS_ENABLED=false)',
|
||||||
'没有已核验的生产恢复演练证据(PRODUCTION_BACKUP_RESTORE_VERIFIED=false)',
|
'没有已核验的生产恢复演练证据(PRODUCTION_BACKUP_RESTORE_VERIFIED=false)',
|
||||||
'生产当前权威版本未知,不能启动写入',
|
'目标非空且没有生效版本记录:先建立目标权威基线(维护隔离下捕获整库并逐类校验),或确认目标确实是空库',
|
||||||
'本次目标快照还没有生产预备份',
|
'本次目标快照还没有生产预备份',
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
@@ -149,6 +149,15 @@ const base: ViewProps = {
|
|||||||
pendingCapture: false,
|
pendingCapture: false,
|
||||||
canCapture: true,
|
canCapture: true,
|
||||||
canWrite: true,
|
canWrite: true,
|
||||||
|
canRestore: false,
|
||||||
|
sync: { canSign: false, pendingGrant: false, onGrant: vi.fn(), canStart: false, pending: false, onStart: vi.fn() },
|
||||||
|
baseline: { canStart: false, pending: false, onStart: vi.fn() },
|
||||||
|
restore: null,
|
||||||
|
restoreNotice: null,
|
||||||
|
pendingRestore: false,
|
||||||
|
pendingGrant: false,
|
||||||
|
onGrantRestore: vi.fn(),
|
||||||
|
onRestore: vi.fn(),
|
||||||
maintenance: null,
|
maintenance: null,
|
||||||
now: new Date('2026-10-28T03:35:00Z').getTime(),
|
now: new Date('2026-10-28T03:35:00Z').getTime(),
|
||||||
onRefresh: vi.fn(),
|
onRefresh: vi.fn(),
|
||||||
@@ -166,7 +175,7 @@ describe('ReleaseTargetsView(确认稿布局)', () => {
|
|||||||
expect(html).toContain('对外知识库')
|
expect(html).toContain('对外知识库')
|
||||||
expect(html).toContain('role="tab"')
|
expect(html).toContain('role="tab"')
|
||||||
// 只显示一个主作业卡:选中库标题出现一次,另一个库不渲染作业卡
|
// 只显示一个主作业卡:选中库标题出现一次,另一个库不渲染作业卡
|
||||||
expect(html.match(/· 已封存,生产同步未接入/g)?.length).toBe(1)
|
expect(html.match(/· 已封存,同步按条件放行/g)?.length).toBe(1)
|
||||||
expect(html).not.toContain('对外知识库 · 尚无')
|
expect(html).not.toContain('对外知识库 · 尚无')
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -239,18 +248,62 @@ describe('ReleaseTargetsView(确认稿布局)', () => {
|
|||||||
expect(html).toContain('列表只校验清单自身,字节级复核在同步/恢复执行前进行')
|
expect(html).toContain('列表只校验清单自身,字节级复核在同步/恢复执行前进行')
|
||||||
})
|
})
|
||||||
|
|
||||||
it('版本与快照抽屉按 scope 列出全部历史并区分测试封存/不可直接恢复', () => {
|
it('版本与快照抽屉按 scope 列出全部历史,并把恢复条件交给接口判定', () => {
|
||||||
const html = render({ drawerOpen: true, expandedSnapshot: internalSnapshot.snapshot_id })
|
const html = render({ drawerOpen: true, expandedSnapshot: internalSnapshot.snapshot_id, canRestore: true })
|
||||||
expect(html).toContain('版本与快照')
|
expect(html).toContain('版本与快照')
|
||||||
expect(html).toContain('封存 f992d64ea45b')
|
expect(html).toContain('封存 f992d64ea45b')
|
||||||
expect(html).toContain('测试封存')
|
expect(html).toContain('测试封存')
|
||||||
expect(html).toContain('不可直接恢复')
|
expect(html).toContain('恢复条件见下方')
|
||||||
expect(html).toContain('恢复至该版本')
|
expect(html).toContain('恢复至该版本')
|
||||||
expect(html).toMatch(/恢复至该版本[^<]*<\/button>/)
|
expect(html).toMatch(/恢复至该版本[^<]*<\/button>/)
|
||||||
expect(html).toContain('分项数量 / 字节 / 摘要(2 项)')
|
expect(html).toContain('分项数量 / 字节 / 摘要(2 项)')
|
||||||
expect(html).toContain('不支持只选择 Wiki 或某张数据表恢复')
|
expect(html).toContain('不支持只选择 Wiki 或某张数据表恢复')
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('管理员在缺授权时看到签发入口,恢复按钮按接口给出的原因禁用', () => {
|
||||||
|
const html = render({
|
||||||
|
drawerOpen: true,
|
||||||
|
expandedSnapshot: internalSnapshot.snapshot_id,
|
||||||
|
canRestore: true,
|
||||||
|
restore: {
|
||||||
|
allowed: false,
|
||||||
|
reasons: ['没有针对该快照的恢复授权:先签发 restore 授权(绑定同一快照)'],
|
||||||
|
scope: 'internal',
|
||||||
|
snapshot_id: internalSnapshot.snapshot_id,
|
||||||
|
authorization: { state: 'none' },
|
||||||
|
},
|
||||||
|
})
|
||||||
|
expect(html).toContain('签发恢复授权(30 分钟)')
|
||||||
|
expect(html).toContain('没有针对该快照的恢复授权')
|
||||||
|
expect(html).toMatch(/rel-danger[^>]*disabled/)
|
||||||
|
// 不能再说"未接线":条件不满足要给出具体原因
|
||||||
|
expect(html).not.toContain('生产恢复未接线')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('条件全满足时恢复按钮可用,并写明授权有效期', () => {
|
||||||
|
const html = render({
|
||||||
|
drawerOpen: true,
|
||||||
|
expandedSnapshot: internalSnapshot.snapshot_id,
|
||||||
|
canRestore: true,
|
||||||
|
restore: {
|
||||||
|
allowed: true,
|
||||||
|
reasons: [],
|
||||||
|
scope: 'internal',
|
||||||
|
snapshot_id: internalSnapshot.snapshot_id,
|
||||||
|
authorization: { state: 'active', authorized_by: 'yaozhanghao', expires_at: '2026-10-28T04:10:00Z' },
|
||||||
|
},
|
||||||
|
})
|
||||||
|
expect(html).toContain('条件已满足')
|
||||||
|
expect(html).toContain('有效至')
|
||||||
|
expect(html).not.toMatch(/rel-danger[^>]*disabled/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('工程师看不到恢复入口,只看到权限说明', () => {
|
||||||
|
const html = render({ drawerOpen: true, expandedSnapshot: internalSnapshot.snapshot_id, canRestore: false })
|
||||||
|
expect(html).toContain('按版本恢复仅管理员可发起')
|
||||||
|
expect(html).not.toContain('签发恢复授权')
|
||||||
|
})
|
||||||
|
|
||||||
it('抽屉只展示当前选中库的历史', () => {
|
it('抽屉只展示当前选中库的历史', () => {
|
||||||
const html = render({ drawerOpen: true, scope: 'external' })
|
const html = render({ drawerOpen: true, scope: 'external' })
|
||||||
expect(html).toContain('封存 01343b0b310b')
|
expect(html).toContain('封存 01343b0b310b')
|
||||||
@@ -277,19 +330,119 @@ describe('ReleaseTargetsView(确认稿布局)', () => {
|
|||||||
expect(html).toContain('主机侧正在暂停写入者、导出并恢复')
|
expect(html).toContain('主机侧正在暂停写入者、导出并恢复')
|
||||||
})
|
})
|
||||||
|
|
||||||
it('无权限时不给封存入口,未接线时说明通道状态', () => {
|
it('无权限时不给封存入口;不可发起时给出接口返回的真实原因与执行器身份', () => {
|
||||||
expect(render({ canCapture: false, canWrite: false })).toContain('封存需要知识工程师及以上权限')
|
expect(render({ canCapture: false, canWrite: false })).toContain('封存需要知识工程师及以上权限')
|
||||||
const unwired = render({
|
const unavailable = render({
|
||||||
canCapture: false,
|
canCapture: false,
|
||||||
canWrite: true,
|
canWrite: true,
|
||||||
page: { ...page, capture_available: false, capture_reason: '主机侧封存 agent 未接线' },
|
page: { ...page, capture_available: false },
|
||||||
|
overview: {
|
||||||
|
...overview,
|
||||||
|
capture_available: false,
|
||||||
|
capture_reason: '执行器指向的实例不匹配:agent="weknora071full",期望="kgmaint-weknora"',
|
||||||
|
executor: {
|
||||||
|
available: false,
|
||||||
|
busy: false,
|
||||||
|
project: 'weknora071full',
|
||||||
|
source_app: 'WKNFull-app',
|
||||||
|
observed_at: '2026-10-28T03:34:00Z',
|
||||||
|
},
|
||||||
|
},
|
||||||
})
|
})
|
||||||
expect(unwired).toContain('封存通道未接线')
|
expect(unavailable).toContain('执行器指向的实例不匹配')
|
||||||
|
expect(unavailable).toContain('执行器 weknora071full / WKNFull-app')
|
||||||
|
expect(unavailable).toContain('观测于')
|
||||||
|
// 通道可用/不可用都不许用写死的"未接线"糊过去
|
||||||
|
expect(unavailable).not.toContain('封存通道未接线')
|
||||||
})
|
})
|
||||||
|
|
||||||
it('无封存时给出“尚无封存”而不是空表', () => {
|
it('无封存时给出“尚无封存”而不是空表', () => {
|
||||||
const html = render({ page: { ...page, snapshots: [] } })
|
// 归档没有快照时接口也不会给 test_sealed:两处必须一致,否则页面会把
|
||||||
|
// 一个不存在的封存显示成"已封存"。
|
||||||
|
const html = render({
|
||||||
|
page: { ...page, snapshots: [] },
|
||||||
|
overview: { ...overview, test_sealed: undefined, test_sealed_reason: '还没有该目标的测试封存' },
|
||||||
|
})
|
||||||
expect(html).toContain('尚无测试封存')
|
expect(html).toContain('尚无测试封存')
|
||||||
expect(html).toContain('还没有测试库封存')
|
expect(html).toContain('还没有测试库封存')
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('管理员在缺同步授权时看到签发入口,工程师只看到“需管理员签发”', () => {
|
||||||
|
const admin = render({
|
||||||
|
sync: { canSign: true, pendingGrant: false, onGrant: vi.fn(), canStart: false, pending: false, onStart: vi.fn() },
|
||||||
|
})
|
||||||
|
expect(admin).toContain('签发同步授权(30 分钟)')
|
||||||
|
// 授权入口不能顺手把执行也放行:闸口条件仍然逐条可见
|
||||||
|
expect(admin).toContain('生产写入闸口关闭')
|
||||||
|
const engineer = render()
|
||||||
|
expect(engineer).not.toContain('签发同步授权')
|
||||||
|
expect(engineer).toContain('同步授权只能由管理员按目标+快照签发')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('目标非空又没有权威版本时,管理员看到建立基线入口;有生效版本时不显示', () => {
|
||||||
|
const admin = render({
|
||||||
|
baseline: { canStart: true, pending: false, onStart: vi.fn() },
|
||||||
|
})
|
||||||
|
expect(admin).toContain('建立目标权威基线(维护隔离)')
|
||||||
|
expect(admin).toContain('目标非空且没有生效版本记录')
|
||||||
|
const known = render({
|
||||||
|
baseline: { canStart: true, pending: false, onStart: vi.fn() },
|
||||||
|
overview: {
|
||||||
|
...overview,
|
||||||
|
production: {
|
||||||
|
state: 'known',
|
||||||
|
version: {
|
||||||
|
id: 'v1',
|
||||||
|
kind: 'production_effective',
|
||||||
|
snapshot_id: 'a'.repeat(64),
|
||||||
|
version_label: '基线 abc',
|
||||||
|
source_job_id: 'libbase-1',
|
||||||
|
evidence: {},
|
||||||
|
created_by: 'admin',
|
||||||
|
created_at: '2026-10-29T09:00:00Z',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
// 有生效版本时不再提供基线入口(缺项说明文字里仍会提到它存在,所以只断言按钮)
|
||||||
|
expect(known).not.toMatch(/建立目标权威基线(维护隔离)<\/button>/)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('闸口全满足时“启动同步”真的可点,并给出执行说明', () => {
|
||||||
|
const html = render({
|
||||||
|
sync: { canSign: false, pendingGrant: false, onGrant: vi.fn(), canStart: true, pending: false, onStart: vi.fn() },
|
||||||
|
overview: { ...overview, gate: { sync_allowed: true, reasons: [] } },
|
||||||
|
})
|
||||||
|
expect(html).toMatch(/<button[^>]*class="rel-primary"[^>]*>▶ 启动同步/)
|
||||||
|
expect(html).not.toMatch(/class="rel-primary"[^>]*disabled/)
|
||||||
|
expect(html).toContain('先固定测试版本,再备份目标并写入')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('维护卡片区分“未运行维护”与“没有维护能力”', () => {
|
||||||
|
const wired = render({
|
||||||
|
maintenance: {
|
||||||
|
held: false,
|
||||||
|
capability: {
|
||||||
|
configured: true,
|
||||||
|
missing: [],
|
||||||
|
instance: 'kgseal-app',
|
||||||
|
project: 'kgseal-weknora',
|
||||||
|
writers: ['app'],
|
||||||
|
ingress_service: 'ingress',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
expect(wired).toContain('未运行维护')
|
||||||
|
expect(wired).toContain('维护能力已接线:实例 kgseal-app · 写入者 app')
|
||||||
|
expect(wired).not.toContain('也没有维护能力配置')
|
||||||
|
const broken = render({
|
||||||
|
maintenance: { held: false, capability: { configured: false, missing: ['LIBRARY_TARGET_WRITERS'] } },
|
||||||
|
})
|
||||||
|
expect(broken).toContain('能力未接线')
|
||||||
|
expect(broken).toContain('维护能力未接线:缺少 LIBRARY_TARGET_WRITERS')
|
||||||
|
// 接口没给能力信息时不能猜成"没有能力"
|
||||||
|
const unknown = render({ maintenance: { held: false } })
|
||||||
|
expect(unknown).toContain('维护能力状态未知')
|
||||||
|
expect(unknown).not.toContain('也没有维护能力配置')
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|||||||
+330
-18
@@ -1,10 +1,16 @@
|
|||||||
import { useCallback, useEffect, useState } from 'react'
|
import { useCallback, useEffect, useState } from 'react'
|
||||||
import {
|
import {
|
||||||
|
createLibraryRestoreAuthorization,
|
||||||
|
createLibrarySyncAuthorization,
|
||||||
|
enqueueLibraryBaseline,
|
||||||
enqueueLibraryCapture,
|
enqueueLibraryCapture,
|
||||||
enqueueLibraryRecovery,
|
enqueueLibraryRecovery,
|
||||||
|
enqueueLibraryRestore,
|
||||||
|
enqueueLibrarySync,
|
||||||
fetchLibraryCaptures,
|
fetchLibraryCaptures,
|
||||||
fetchLibraryMaintenance,
|
fetchLibraryMaintenance,
|
||||||
fetchLibraryOverview,
|
fetchLibraryOverview,
|
||||||
|
fetchLibraryRestoreAvailability,
|
||||||
fetchLibrarySnapshots,
|
fetchLibrarySnapshots,
|
||||||
formatBytes,
|
formatBytes,
|
||||||
scopeLabel,
|
scopeLabel,
|
||||||
@@ -12,6 +18,7 @@ import {
|
|||||||
type LibraryJobRecord,
|
type LibraryJobRecord,
|
||||||
type LibraryMaintenanceView,
|
type LibraryMaintenanceView,
|
||||||
type LibraryOverview,
|
type LibraryOverview,
|
||||||
|
type LibraryRestoreAvailability,
|
||||||
type LibrarySnapshot,
|
type LibrarySnapshot,
|
||||||
type LibrarySnapshotList,
|
type LibrarySnapshotList,
|
||||||
type LibraryStageRow,
|
type LibraryStageRow,
|
||||||
@@ -58,6 +65,24 @@ const jobStateLabel: Record<string, string> = {
|
|||||||
|
|
||||||
type Scope = 'internal' | 'external'
|
type Scope = 'internal' | 'external'
|
||||||
|
|
||||||
|
// 签发授权与执行是两步:页面只提供入口,不会替人签发,也不会自动重试执行。
|
||||||
|
export interface ReleaseSyncActions {
|
||||||
|
// 管理员才能签发;工程师看得到条件但没有按钮。
|
||||||
|
canSign: boolean
|
||||||
|
pendingGrant: boolean
|
||||||
|
onGrant: () => void
|
||||||
|
// 闸口全部满足才能启动(接口用同一份判定,按钮只是提前说明)。
|
||||||
|
canStart: boolean
|
||||||
|
pending: boolean
|
||||||
|
onStart: () => void
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ReleaseBaselineActions {
|
||||||
|
canStart: boolean
|
||||||
|
pending: boolean
|
||||||
|
onStart: () => void
|
||||||
|
}
|
||||||
|
|
||||||
interface ScopeView {
|
interface ScopeView {
|
||||||
scope: Scope
|
scope: Scope
|
||||||
snapshot?: LibrarySnapshot
|
snapshot?: LibrarySnapshot
|
||||||
@@ -85,6 +110,18 @@ interface ViewProps {
|
|||||||
pendingCapture: boolean
|
pendingCapture: boolean
|
||||||
canCapture: boolean
|
canCapture: boolean
|
||||||
canWrite: boolean
|
canWrite: boolean
|
||||||
|
// 发布动作:签发授权(管理员)与启动同步是两个独立步骤,页面绝不自动签发。
|
||||||
|
sync: ReleaseSyncActions
|
||||||
|
// 目标非空又没有生效版本时,唯一入口是"建立目标权威基线"(管理员)。
|
||||||
|
baseline: ReleaseBaselineActions
|
||||||
|
// 按版本恢复是管理员动作:工程师看得到条件,但拿不到恢复入口。
|
||||||
|
canRestore: boolean
|
||||||
|
restore: LibraryRestoreAvailability | null
|
||||||
|
restoreNotice: string | null
|
||||||
|
pendingRestore: boolean
|
||||||
|
pendingGrant: boolean
|
||||||
|
onGrantRestore: () => void
|
||||||
|
onRestore: () => void
|
||||||
overview: LibraryOverview | null
|
overview: LibraryOverview | null
|
||||||
maintenance: LibraryMaintenanceView | null
|
maintenance: LibraryMaintenanceView | null
|
||||||
now: number
|
now: number
|
||||||
@@ -106,7 +143,9 @@ function scopeView(
|
|||||||
overview?.job ?? persisted.find((item) => item.state === 'pending' || item.state === 'running') ?? persisted[0]
|
overview?.job ?? persisted.find((item) => item.state === 'pending' || item.state === 'running') ?? persisted[0]
|
||||||
return {
|
return {
|
||||||
scope,
|
scope,
|
||||||
snapshot: history[0],
|
// 候选只认接口给的 test_sealed(来自版本登记):归档里还有生产预备份与
|
||||||
|
// 目标基线快照,按"最新一份"挑会把目标自己的内容显示成测试封存。
|
||||||
|
snapshot: overview?.test_sealed,
|
||||||
history,
|
history,
|
||||||
activeJob: scoped.find((job) => ACTIVE_STATES.includes(job.state)),
|
activeJob: scoped.find((job) => ACTIVE_STATES.includes(job.state)),
|
||||||
lastJob: scoped[0],
|
lastJob: scoped[0],
|
||||||
@@ -118,6 +157,20 @@ function scopeView(
|
|||||||
|
|
||||||
const scopeName = (scope: Scope) => (scope === 'internal' ? '对内知识库' : '对外知识库')
|
const scopeName = (scope: Scope) => (scope === 'internal' ? '对内知识库' : '对外知识库')
|
||||||
|
|
||||||
|
// 不可发起时必须说清是哪一条不满足:读的是接口给出的真实原因(执行器不可达、
|
||||||
|
// 指向别的实例、降级、观测过期),不是一句写死的"通道未接线"。
|
||||||
|
function captureHint(view: ScopeView, canWrite: boolean): string {
|
||||||
|
if (!canWrite) return '封存需要知识工程师及以上权限'
|
||||||
|
const executor = view.overview?.executor
|
||||||
|
const reason = view.overview?.capture_reason || executor?.reason
|
||||||
|
const observed = executor?.observed_at ? `(观测于 ${stamp(executor.observed_at)})` : ''
|
||||||
|
const identity = executor?.project
|
||||||
|
? `执行器 ${executor.project}${executor.source_app ? ` / ${executor.source_app}` : ''} · `
|
||||||
|
: ''
|
||||||
|
if (reason) return `${identity}封存通道不可用:${reason}${observed}`
|
||||||
|
return '封存通道状态未知:还没有读到执行器观测'
|
||||||
|
}
|
||||||
|
|
||||||
function statusOf(view: ScopeView) {
|
function statusOf(view: ScopeView) {
|
||||||
const job = view.job
|
const job = view.job
|
||||||
if (job && (job.state === 'running' || job.state === 'pending')) {
|
if (job && (job.state === 'running' || job.state === 'pending')) {
|
||||||
@@ -245,6 +298,14 @@ export function MaintenanceCard({
|
|||||||
}) {
|
}) {
|
||||||
const summary = maintenanceSummary(maintenance, job)
|
const summary = maintenanceSummary(maintenance, job)
|
||||||
const record = maintenance?.instance_maintenance ?? maintenance?.maintenance
|
const record = maintenance?.instance_maintenance ?? maintenance?.maintenance
|
||||||
|
// "没有窗口记录"不等于"没有维护能力":能力是配置事实,接口单独返回。
|
||||||
|
// 接口没给能力信息时也不能猜,明确写"未知"。
|
||||||
|
const capability = maintenance?.capability
|
||||||
|
const capabilityText = !capability
|
||||||
|
? '维护能力状态未知:接口没有返回能力信息,无法区分"未运行维护"与"能力未接线"'
|
||||||
|
: capability.configured
|
||||||
|
? `维护能力已接线:实例 ${capability.instance || '—'} · 写入者 ${(capability.writers ?? []).join('、') || '—'}`
|
||||||
|
: `维护能力未接线:缺少 ${(capability.missing ?? []).join('、') || '必要配置'}`
|
||||||
return (
|
return (
|
||||||
<div className="rel-panel rel-maintenance">
|
<div className="rel-panel rel-maintenance">
|
||||||
<header className="rel-panel-head">
|
<header className="rel-panel-head">
|
||||||
@@ -252,13 +313,26 @@ export function MaintenanceCard({
|
|||||||
<h2>目标维护</h2>
|
<h2>目标维护</h2>
|
||||||
<p>同步/恢复期间目标写入者被停住;级别与入口状态来自维护记录本身</p>
|
<p>同步/恢复期间目标写入者被停住;级别与入口状态来自维护记录本身</p>
|
||||||
</div>
|
</div>
|
||||||
<span className={`rel-pill ${summary.held ? 'blue' : summary.level === '未接入' ? 'gray' : 'green'}`}>
|
<span
|
||||||
{summary.held ? `${summary.level} · 占用中` : summary.level === '未接入' ? '未接入' : '未占用'}
|
className={`rel-pill ${summary.held ? 'blue' : record ? 'green' : capability?.configured ? 'gray' : 'red'}`}
|
||||||
|
>
|
||||||
|
{summary.held
|
||||||
|
? `${summary.level} · 占用中`
|
||||||
|
: record
|
||||||
|
? '未占用'
|
||||||
|
: capability?.configured
|
||||||
|
? '未运行维护'
|
||||||
|
: capability
|
||||||
|
? '能力未接线'
|
||||||
|
: '能力未知'}
|
||||||
</span>
|
</span>
|
||||||
</header>
|
</header>
|
||||||
{summary.level === '未接入' ? (
|
{!record ? (
|
||||||
<p className="rel-scope-note">
|
<p className="rel-scope-note">
|
||||||
暂无维护窗口记录,也没有维护能力配置;维护类操作保持 fail closed,不会在此状态下写目标。
|
暂无维护窗口记录。{capabilityText}
|
||||||
|
{capability?.configured
|
||||||
|
? ';窗口只在同步或建立基线时开启,页面不会单独开窗口。'
|
||||||
|
: ';维护类操作保持 fail closed,不会在此状态下写目标。'}
|
||||||
</p>
|
</p>
|
||||||
) : (
|
) : (
|
||||||
<>
|
<>
|
||||||
@@ -508,15 +582,23 @@ function ProtectionCard({
|
|||||||
prebackup,
|
prebackup,
|
||||||
gate,
|
gate,
|
||||||
authorization,
|
authorization,
|
||||||
|
targetSealed,
|
||||||
|
sync,
|
||||||
|
baseline,
|
||||||
onOpen,
|
onOpen,
|
||||||
}: {
|
}: {
|
||||||
production: LibraryOverview['production']
|
production: LibraryOverview['production']
|
||||||
prebackup: LibraryOverview['prebackup']
|
prebackup: LibraryOverview['prebackup']
|
||||||
gate: LibraryOverview['gate']
|
gate: LibraryOverview['gate']
|
||||||
authorization: LibraryOverview['authorization']
|
authorization: LibraryOverview['authorization']
|
||||||
|
targetSealed: boolean
|
||||||
|
sync: ReleaseSyncActions
|
||||||
|
baseline: ReleaseBaselineActions
|
||||||
onOpen: () => void
|
onOpen: () => void
|
||||||
}) {
|
}) {
|
||||||
const backupKnown = prebackup.state === 'known' && prebackup.version
|
const backupKnown = prebackup.state === 'known' && prebackup.version
|
||||||
|
// 基线入口只在"目标非空、却没有生效版本"时才是答案;空库走首次同步(预备份+登记)。
|
||||||
|
const baselineNeeded = production.state !== 'known' && targetSealed
|
||||||
return (
|
return (
|
||||||
<aside className="rel-panel rel-protection">
|
<aside className="rel-panel rel-protection">
|
||||||
<header className="rel-panel-head">
|
<header className="rel-panel-head">
|
||||||
@@ -554,6 +636,21 @@ function ProtectionCard({
|
|||||||
))}
|
))}
|
||||||
</ul>
|
</ul>
|
||||||
) : null}
|
) : null}
|
||||||
|
<div className="rel-drawer-actions">
|
||||||
|
{sync.canSign && authorization.state !== 'active' && targetSealed ? (
|
||||||
|
<button type="button" className="rel-secondary" onClick={sync.onGrant} disabled={sync.pendingGrant}>
|
||||||
|
{sync.pendingGrant ? '签发中…' : '签发同步授权(30 分钟)'}
|
||||||
|
</button>
|
||||||
|
) : null}
|
||||||
|
{baseline.canStart && baselineNeeded ? (
|
||||||
|
<button type="button" className="rel-secondary" onClick={baseline.onStart} disabled={baseline.pending}>
|
||||||
|
{baseline.pending ? '投递中…' : '建立目标权威基线(维护隔离)'}
|
||||||
|
</button>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
{authorization.state !== 'active' && !sync.canSign ? (
|
||||||
|
<p className="rel-hint">同步授权只能由管理员按目标+快照签发;工程师看不到签发入口。</p>
|
||||||
|
) : null}
|
||||||
<p className="rel-hint">同步启动后,这里显示本次生产恢复点及备份进度,可直接查看对应快照。</p>
|
<p className="rel-hint">同步启动后,这里显示本次生产恢复点及备份进度,可直接查看对应快照。</p>
|
||||||
<button type="button" className="rel-secondary" onClick={onOpen}>
|
<button type="button" className="rel-secondary" onClick={onOpen}>
|
||||||
查看已有版本与快照 ›
|
查看已有版本与快照 ›
|
||||||
@@ -575,6 +672,8 @@ interface ReadyProps {
|
|||||||
pendingCapture: boolean
|
pendingCapture: boolean
|
||||||
onCapture: () => void
|
onCapture: () => void
|
||||||
onOpen: () => void
|
onOpen: () => void
|
||||||
|
sync: ReleaseSyncActions
|
||||||
|
baseline: ReleaseBaselineActions
|
||||||
}
|
}
|
||||||
|
|
||||||
function ReadyBody({
|
function ReadyBody({
|
||||||
@@ -584,6 +683,8 @@ function ReadyBody({
|
|||||||
prebackup,
|
prebackup,
|
||||||
gate,
|
gate,
|
||||||
authorization,
|
authorization,
|
||||||
|
sync,
|
||||||
|
baseline,
|
||||||
canCapture,
|
canCapture,
|
||||||
canWrite,
|
canWrite,
|
||||||
pendingCapture,
|
pendingCapture,
|
||||||
@@ -600,6 +701,9 @@ function ReadyBody({
|
|||||||
prebackup={prebackup}
|
prebackup={prebackup}
|
||||||
gate={gate}
|
gate={gate}
|
||||||
authorization={authorization}
|
authorization={authorization}
|
||||||
|
targetSealed={Boolean(snapshot)}
|
||||||
|
sync={sync}
|
||||||
|
baseline={baseline}
|
||||||
onOpen={onOpen}
|
onOpen={onOpen}
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
@@ -629,7 +733,7 @@ function ReadyBody({
|
|||||||
{pendingCapture ? '投递中…' : '封存测试版本'}
|
{pendingCapture ? '投递中…' : '封存测试版本'}
|
||||||
</button>
|
</button>
|
||||||
) : (
|
) : (
|
||||||
<span className="rel-hint">{canWrite ? '封存通道未接线' : '封存需要知识工程师及以上权限'}</span>
|
<span className="rel-hint">{captureHint(view, canWrite)}</span>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
</>
|
</>
|
||||||
@@ -822,8 +926,16 @@ interface DrawerProps {
|
|||||||
onExpand: (id: string | null) => void
|
onExpand: (id: string | null) => void
|
||||||
onClose: () => void
|
onClose: () => void
|
||||||
canCapture: boolean
|
canCapture: boolean
|
||||||
|
canRestore: boolean
|
||||||
pendingCapture: boolean
|
pendingCapture: boolean
|
||||||
onCapture: () => void
|
onCapture: () => void
|
||||||
|
// 恢复条件来自容器(真实接口判定):双闸 / 该快照的授权 / 目标是否在维护。
|
||||||
|
restore: LibraryRestoreAvailability | null
|
||||||
|
restoreNotice: string | null
|
||||||
|
pendingRestore: boolean
|
||||||
|
pendingGrant: boolean
|
||||||
|
onGrantRestore: () => void
|
||||||
|
onRestore: () => void
|
||||||
}
|
}
|
||||||
|
|
||||||
function Drawer({
|
function Drawer({
|
||||||
@@ -835,8 +947,15 @@ function Drawer({
|
|||||||
onExpand,
|
onExpand,
|
||||||
onClose,
|
onClose,
|
||||||
canCapture,
|
canCapture,
|
||||||
|
canRestore,
|
||||||
pendingCapture,
|
pendingCapture,
|
||||||
onCapture,
|
onCapture,
|
||||||
|
restore,
|
||||||
|
restoreNotice,
|
||||||
|
pendingRestore,
|
||||||
|
pendingGrant,
|
||||||
|
onGrantRestore,
|
||||||
|
onRestore,
|
||||||
}: DrawerProps) {
|
}: DrawerProps) {
|
||||||
if (!open) return null
|
if (!open) return null
|
||||||
const listing = view.history
|
const listing = view.history
|
||||||
@@ -919,8 +1038,12 @@ function Drawer({
|
|||||||
>
|
>
|
||||||
{isSelected ? '收起快照' : '查看快照'}
|
{isSelected ? '收起快照' : '查看快照'}
|
||||||
</button>
|
</button>
|
||||||
<button type="button" className="rel-link" disabled title="生产恢复未接线,也没有恢复演练记录">
|
<button
|
||||||
不可直接恢复
|
type="button"
|
||||||
|
className="rel-link"
|
||||||
|
onClick={() => onExpand(isSelected ? null : snapshot.snapshot_id)}
|
||||||
|
>
|
||||||
|
{isSelected ? '恢复条件见下方' : '查看恢复条件'}
|
||||||
</button>
|
</button>
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
@@ -991,13 +1114,48 @@ function Drawer({
|
|||||||
以上是同一恢复点的配套内容,不支持只选择 Wiki 或某张数据表恢复。恢复速度需实测;“有备份文件”不等于可恢复。
|
以上是同一恢复点的配套内容,不支持只选择 Wiki 或某张数据表恢复。恢复速度需实测;“有备份文件”不等于可恢复。
|
||||||
</p>
|
</p>
|
||||||
<div className="rel-drawer-actions">
|
<div className="rel-drawer-actions">
|
||||||
<button type="button" disabled title="生产恢复验证记录未接入">
|
<button type="button" disabled title="逐项校验证据在同步/恢复执行时产生,作业详情里可见">
|
||||||
查看校验记录
|
查看校验记录
|
||||||
</button>
|
</button>
|
||||||
<button type="button" className="rel-danger" disabled title="生产恢复未接线,也没有经过恢复演练">
|
{canRestore && restore && restore.authorization.state !== 'active' ? (
|
||||||
恢复至该版本
|
<button type="button" className="rel-secondary" onClick={onGrantRestore} disabled={pendingGrant}>
|
||||||
</button>
|
{pendingGrant ? '签发中…' : '签发恢复授权(30 分钟)'}
|
||||||
|
</button>
|
||||||
|
) : null}
|
||||||
|
{canRestore ? (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="rel-danger"
|
||||||
|
onClick={onRestore}
|
||||||
|
disabled={pendingRestore || restore?.allowed !== true}
|
||||||
|
title={
|
||||||
|
restore?.allowed
|
||||||
|
? '把整套生产知识状态恢复到这个快照'
|
||||||
|
: (restore?.reasons ?? []).join(';') || '恢复条件尚未读取到'
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{pendingRestore ? '投递中…' : '恢复至该版本'}
|
||||||
|
</button>
|
||||||
|
) : (
|
||||||
|
<span className="rel-hint">按版本恢复仅管理员可发起</span>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
{restore && !restore.allowed ? (
|
||||||
|
<ul className="rel-gate" aria-label="恢复条件">
|
||||||
|
{restore.reasons.map((reason) => (
|
||||||
|
<li key={reason}>{reason}</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
) : null}
|
||||||
|
{restore?.allowed ? (
|
||||||
|
<p className="rel-hint">
|
||||||
|
条件已满足:生产写入闸口打开、这份快照有恢复授权
|
||||||
|
{restore.authorization.expires_at ? `(有效至 ${stamp(restore.authorization.expires_at)})` : ''}
|
||||||
|
、目标实例未在维护。
|
||||||
|
</p>
|
||||||
|
) : null}
|
||||||
|
{restore === null ? <p className="rel-hint">正在读取恢复条件…</p> : null}
|
||||||
|
{restoreNotice ? <p className="rel-notice">{restoreNotice}</p> : null}
|
||||||
</section>
|
</section>
|
||||||
) : null}
|
) : null}
|
||||||
|
|
||||||
@@ -1023,7 +1181,7 @@ function Drawer({
|
|||||||
<p className="rel-hint">
|
<p className="rel-hint">
|
||||||
归档占用(只读):被版本引用的去重内容 {formatBytes(page?.content_bytes ?? 0)} · 磁盘分配{' '}
|
归档占用(只读):被版本引用的去重内容 {formatBytes(page?.content_bytes ?? 0)} · 磁盘分配{' '}
|
||||||
{formatBytes(page?.blob_allocated_bytes ?? 0)} · 未被引用 {page?.unreferenced_blobs ?? 0}{' '}
|
{formatBytes(page?.blob_allocated_bytes ?? 0)} · 未被引用 {page?.unreferenced_blobs ?? 0}{' '}
|
||||||
个。保留策略与清理在系统设置 → 存储管理,均未接线。
|
个。清理与保留策略在系统设置 → 存储管理:当前可查看去重与保护范围,删除与保留策略尚未开放。
|
||||||
</p>
|
</p>
|
||||||
</aside>
|
</aside>
|
||||||
</div>
|
</div>
|
||||||
@@ -1047,6 +1205,15 @@ export function ReleaseTargetsView(props: ViewProps) {
|
|||||||
pendingCapture,
|
pendingCapture,
|
||||||
canCapture,
|
canCapture,
|
||||||
canWrite,
|
canWrite,
|
||||||
|
sync,
|
||||||
|
baseline,
|
||||||
|
canRestore,
|
||||||
|
restore,
|
||||||
|
restoreNotice,
|
||||||
|
pendingRestore,
|
||||||
|
pendingGrant,
|
||||||
|
onGrantRestore,
|
||||||
|
onRestore,
|
||||||
now,
|
now,
|
||||||
maintenance,
|
maintenance,
|
||||||
onRefresh,
|
onRefresh,
|
||||||
@@ -1125,7 +1292,7 @@ export function ReleaseTargetsView(props: ViewProps) {
|
|||||||
<div className="rel-panel rel-jobhead">
|
<div className="rel-panel rel-jobhead">
|
||||||
<div className="rel-between">
|
<div className="rel-between">
|
||||||
<h2>
|
<h2>
|
||||||
{scopeName(scope)} · {active ? '封存进行中' : view.snapshot ? '已封存,生产同步未接入' : '尚无测试封存'}
|
{scopeName(scope)} · {active ? '封存进行中' : view.snapshot ? '已封存,同步按条件放行' : '尚无测试封存'}
|
||||||
</h2>
|
</h2>
|
||||||
<div className="rel-head-actions">
|
<div className="rel-head-actions">
|
||||||
<button type="button" className="rel-secondary" onClick={() => onDrawer(true)}>
|
<button type="button" className="rel-secondary" onClick={() => onDrawer(true)}>
|
||||||
@@ -1134,14 +1301,15 @@ export function ReleaseTargetsView(props: ViewProps) {
|
|||||||
<button
|
<button
|
||||||
type="button"
|
type="button"
|
||||||
className="rel-primary"
|
className="rel-primary"
|
||||||
disabled={!gate.sync_allowed}
|
onClick={sync.onStart}
|
||||||
|
disabled={!sync.canStart || sync.pending}
|
||||||
title={
|
title={
|
||||||
gate.sync_allowed
|
sync.canStart
|
||||||
? '生产同步执行链尚未接线,按钮不会启用'
|
? '按已签发的同步授权启动:先固定测试版本,再备份目标并写入'
|
||||||
: `暂不能启动同步:${gate.reasons.join(';')}`
|
: `暂不能启动同步:${gate.reasons.join(';')}`
|
||||||
}
|
}
|
||||||
>
|
>
|
||||||
▶ 启动同步
|
{sync.pending ? '投递中…' : '▶ 启动同步'}
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -1224,6 +1392,8 @@ export function ReleaseTargetsView(props: ViewProps) {
|
|||||||
prebackup={prebackup}
|
prebackup={prebackup}
|
||||||
gate={gate}
|
gate={gate}
|
||||||
authorization={authorization}
|
authorization={authorization}
|
||||||
|
sync={sync}
|
||||||
|
baseline={baseline}
|
||||||
canCapture={canCapture}
|
canCapture={canCapture}
|
||||||
canWrite={canWrite}
|
canWrite={canWrite}
|
||||||
pendingCapture={pendingCapture}
|
pendingCapture={pendingCapture}
|
||||||
@@ -1241,8 +1411,15 @@ export function ReleaseTargetsView(props: ViewProps) {
|
|||||||
onExpand={onExpandSnapshot}
|
onExpand={onExpandSnapshot}
|
||||||
onClose={() => onDrawer(false)}
|
onClose={() => onDrawer(false)}
|
||||||
canCapture={canCapture}
|
canCapture={canCapture}
|
||||||
|
canRestore={canRestore}
|
||||||
pendingCapture={pendingCapture}
|
pendingCapture={pendingCapture}
|
||||||
onCapture={onCapture}
|
onCapture={onCapture}
|
||||||
|
restore={restore}
|
||||||
|
restoreNotice={restoreNotice}
|
||||||
|
pendingRestore={pendingRestore}
|
||||||
|
pendingGrant={pendingGrant}
|
||||||
|
onGrantRestore={onGrantRestore}
|
||||||
|
onRestore={onRestore}
|
||||||
/>
|
/>
|
||||||
</section>
|
</section>
|
||||||
)
|
)
|
||||||
@@ -1313,7 +1490,122 @@ export function ReleaseTargets() {
|
|||||||
|
|
||||||
const role = auth?.identity?.role
|
const role = auth?.identity?.role
|
||||||
const canWrite = role === 'engineer' || role === 'admin'
|
const canWrite = role === 'engineer' || role === 'admin'
|
||||||
|
// 恢复写的是整套生产知识状态,接口只认管理员;页面必须与它一致。
|
||||||
|
const canRestore = role === 'admin'
|
||||||
|
// 签发同步授权同样是管理员动作:授权=钉死目标+快照+参数,不等于执行。
|
||||||
|
const canSignSync = role === 'admin'
|
||||||
const canCapture = canWrite && page?.capture_available === true
|
const canCapture = canWrite && page?.capture_available === true
|
||||||
|
const [pendingSyncGrant, setPendingSyncGrant] = useState(false)
|
||||||
|
const [pendingSync, setPendingSync] = useState(false)
|
||||||
|
const [pendingBaseline, setPendingBaseline] = useState(false)
|
||||||
|
const current = overviews[scope]
|
||||||
|
const targetSnapshot = current?.test_sealed?.snapshot_id ?? ''
|
||||||
|
|
||||||
|
const grantSync = async () => {
|
||||||
|
if (!targetSnapshot) {
|
||||||
|
setNotice('还没有可发布的测试封存:先封存测试版本,再签发同步授权')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
setPendingSyncGrant(true)
|
||||||
|
setNotice(null)
|
||||||
|
try {
|
||||||
|
const grant = await createLibrarySyncAuthorization(scope, targetSnapshot)
|
||||||
|
setNotice(
|
||||||
|
`已签发同步授权${grant.authorized_by ? `(${grant.authorized_by})` : ''},有效至 ${stamp(grant.expires_at)};` +
|
||||||
|
'执行仍需在下一次操作里点"启动同步"'
|
||||||
|
)
|
||||||
|
await load()
|
||||||
|
} catch (grantError) {
|
||||||
|
setNotice(grantError instanceof Error ? grantError.message : '签发同步授权失败')
|
||||||
|
} finally {
|
||||||
|
setPendingSyncGrant(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const startSync = async () => {
|
||||||
|
if (!targetSnapshot) {
|
||||||
|
setNotice('还没有可发布的测试封存')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
setPendingSync(true)
|
||||||
|
setNotice(null)
|
||||||
|
try {
|
||||||
|
const accepted = await enqueueLibrarySync(scope, targetSnapshot)
|
||||||
|
setNotice(`已投递同步:${accepted.job_id}(先固定测试版本并生成预备份,再写目标)`)
|
||||||
|
await load()
|
||||||
|
} catch (syncError) {
|
||||||
|
setNotice(syncError instanceof Error ? syncError.message : '启动同步失败')
|
||||||
|
} finally {
|
||||||
|
setPendingSync(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const startBaseline = async () => {
|
||||||
|
setPendingBaseline(true)
|
||||||
|
setNotice(null)
|
||||||
|
try {
|
||||||
|
const accepted = await enqueueLibraryBaseline(scope)
|
||||||
|
setNotice(`已投递建立基线:${accepted.job_id}(维护隔离下捕获目标整库,逐类校验后登记为权威基线)`)
|
||||||
|
await load()
|
||||||
|
} catch (baselineError) {
|
||||||
|
setNotice(baselineError instanceof Error ? baselineError.message : '建立目标权威基线失败')
|
||||||
|
} finally {
|
||||||
|
setPendingBaseline(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 恢复条件随"展开的那份快照"变化:闸口、该快照的授权、实例是否在维护都由接口判定。
|
||||||
|
const [restore, setRestore] = useState<LibraryRestoreAvailability | null>(null)
|
||||||
|
const [restoreNotice, setRestoreNotice] = useState<string | null>(null)
|
||||||
|
const [pendingRestore, setPendingRestore] = useState(false)
|
||||||
|
const [pendingGrant, setPendingGrant] = useState(false)
|
||||||
|
const [restoreRound, setRestoreRound] = useState(0)
|
||||||
|
const restoreSnapshot = expandedSnapshot ?? ''
|
||||||
|
useEffect(() => {
|
||||||
|
if (!drawerOpen || !restoreSnapshot) {
|
||||||
|
setRestore(null)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
const controller = new AbortController()
|
||||||
|
void fetchLibraryRestoreAvailability(scope, restoreSnapshot, controller.signal)
|
||||||
|
.then((value) => setRestore(value))
|
||||||
|
.catch(() => setRestore(null))
|
||||||
|
return () => controller.abort()
|
||||||
|
}, [drawerOpen, restoreSnapshot, scope, restoreRound])
|
||||||
|
|
||||||
|
const grantRestore = async () => {
|
||||||
|
if (!restoreSnapshot) return
|
||||||
|
setPendingGrant(true)
|
||||||
|
setRestoreNotice(null)
|
||||||
|
try {
|
||||||
|
const grant = await createLibraryRestoreAuthorization(scope, restoreSnapshot)
|
||||||
|
setRestoreNotice(
|
||||||
|
`已签发恢复授权${grant.authorized_by ? `(${grant.authorized_by})` : ''},有效至 ${stamp(grant.expires_at)}`
|
||||||
|
)
|
||||||
|
setRestoreRound((round) => round + 1)
|
||||||
|
await load()
|
||||||
|
} catch (grantError) {
|
||||||
|
setRestoreNotice(grantError instanceof Error ? grantError.message : '签发恢复授权失败')
|
||||||
|
} finally {
|
||||||
|
setPendingGrant(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const startRestore = async () => {
|
||||||
|
if (!restoreSnapshot) return
|
||||||
|
setPendingRestore(true)
|
||||||
|
setRestoreNotice(null)
|
||||||
|
try {
|
||||||
|
const accepted = await enqueueLibraryRestore(scope, restoreSnapshot)
|
||||||
|
setRestoreNotice(`已投递恢复:${accepted.job_id}(整套生产知识状态回到该快照)`)
|
||||||
|
setRestoreRound((round) => round + 1)
|
||||||
|
await load()
|
||||||
|
} catch (restoreError) {
|
||||||
|
setRestoreNotice(restoreError instanceof Error ? restoreError.message : '发起恢复失败')
|
||||||
|
} finally {
|
||||||
|
setPendingRestore(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
const startCapture = async () => {
|
const startCapture = async () => {
|
||||||
setPendingCapture(true)
|
setPendingCapture(true)
|
||||||
@@ -1364,6 +1656,26 @@ export function ReleaseTargets() {
|
|||||||
pendingCapture={pendingCapture}
|
pendingCapture={pendingCapture}
|
||||||
canCapture={canCapture}
|
canCapture={canCapture}
|
||||||
canWrite={canWrite}
|
canWrite={canWrite}
|
||||||
|
sync={{
|
||||||
|
canSign: canSignSync,
|
||||||
|
pendingGrant: pendingSyncGrant,
|
||||||
|
onGrant: () => void grantSync(),
|
||||||
|
canStart: current?.gate.sync_allowed === true,
|
||||||
|
pending: pendingSync,
|
||||||
|
onStart: () => void startSync(),
|
||||||
|
}}
|
||||||
|
baseline={{
|
||||||
|
canStart: canSignSync,
|
||||||
|
pending: pendingBaseline,
|
||||||
|
onStart: () => void startBaseline(),
|
||||||
|
}}
|
||||||
|
canRestore={canRestore}
|
||||||
|
restore={restore}
|
||||||
|
restoreNotice={restoreNotice}
|
||||||
|
pendingRestore={pendingRestore}
|
||||||
|
pendingGrant={pendingGrant}
|
||||||
|
onGrantRestore={() => void grantRestore()}
|
||||||
|
onRestore={() => void startRestore()}
|
||||||
maintenance={maintenances[scope]}
|
maintenance={maintenances[scope]}
|
||||||
now={now}
|
now={now}
|
||||||
onRefresh={() => void load()}
|
onRefresh={() => void load()}
|
||||||
|
|||||||
@@ -0,0 +1,87 @@
|
|||||||
|
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||||
|
import {
|
||||||
|
createLibraryRestoreAuthorization,
|
||||||
|
enqueueLibraryRestore,
|
||||||
|
fetchLibraryRestoreAvailability,
|
||||||
|
} from './librarySnapshots'
|
||||||
|
|
||||||
|
const jsonResponse = (body: unknown, status = 200) =>
|
||||||
|
new Response(JSON.stringify(body), { status, headers: { 'Content-Type': 'application/json' } })
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals()
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('整库恢复的前端契约', () => {
|
||||||
|
it('读取恢复条件时把原因与授权状态原样带回来', async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
'fetch',
|
||||||
|
vi.fn().mockResolvedValue(
|
||||||
|
jsonResponse({
|
||||||
|
allowed: false,
|
||||||
|
reasons: ['没有针对该快照的恢复授权:先签发 restore 授权(绑定同一快照)'],
|
||||||
|
scope: 'internal',
|
||||||
|
snapshot_id: 'snap-1',
|
||||||
|
authorization: { state: 'none', reason: '没有与目标快照匹配的有效发布授权' },
|
||||||
|
})
|
||||||
|
)
|
||||||
|
)
|
||||||
|
const availability = await fetchLibraryRestoreAvailability('internal', 'snap-1')
|
||||||
|
expect(availability?.allowed).toBe(false)
|
||||||
|
expect(availability?.reasons).toHaveLength(1)
|
||||||
|
expect(availability?.authorization.state).toBe('none')
|
||||||
|
const requested = (vi.mocked(fetch).mock.calls[0]?.[0] as string) ?? ''
|
||||||
|
expect(requested).toContain('scope=internal')
|
||||||
|
expect(requested).toContain('snapshot_id=snap-1')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('签发恢复授权必须声明 action=restore 与有限有效期', async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
'fetch',
|
||||||
|
vi.fn().mockResolvedValue(
|
||||||
|
jsonResponse(
|
||||||
|
{
|
||||||
|
authorization: {
|
||||||
|
id: 'libauth-restore-1',
|
||||||
|
authorized_by: 'yaozhanghao',
|
||||||
|
expires_at: '2026-10-29T09:00:00Z',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
201
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
const grant = await createLibraryRestoreAuthorization('internal', 'snap-1')
|
||||||
|
expect(grant.id).toBe('libauth-restore-1')
|
||||||
|
const init = vi.mocked(fetch).mock.calls[0]?.[1] as RequestInit
|
||||||
|
const body = JSON.parse(String(init.body)) as Record<string, unknown>
|
||||||
|
expect(body.action).toBe('restore')
|
||||||
|
expect(body.expires_in_minutes).toBe(30)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('恢复被拒时把逐条原因说给用户,而不是只说 HTTP 状态', async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
'fetch',
|
||||||
|
vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValue(
|
||||||
|
jsonResponse({ code: 'library_restore_blocked', reasons: ['生产写入闸口关闭', '没有该快照的恢复授权'] }, 503)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
await expect(enqueueLibraryRestore('internal', 'snap-1')).rejects.toThrow('生产写入闸口关闭;没有该快照的恢复授权')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('投递成功时返回作业号,供页面显示真实进度', async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
'fetch',
|
||||||
|
vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValue(
|
||||||
|
jsonResponse({ job: { id: 'librestore-internal-1', scope: 'internal' }, reused: false }, 202)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
const accepted = await enqueueLibraryRestore('internal', 'snap-1')
|
||||||
|
expect(accepted.job_id).toBe('librestore-internal-1')
|
||||||
|
expect(accepted.reused).toBe(false)
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -185,6 +185,170 @@ export async function enqueueLibraryCapture(scope: string): Promise<LibraryCaptu
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 恢复能不能点,由真实条件决定:双闸、这份快照的恢复授权、目标实例是否在维护。
|
||||||
|
export interface LibraryRestoreAvailability {
|
||||||
|
allowed: boolean
|
||||||
|
reasons: string[]
|
||||||
|
scope: string
|
||||||
|
snapshot_id: string
|
||||||
|
authorization: LibraryAuthorizationView
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fetchLibraryRestoreAvailability(
|
||||||
|
scope: string,
|
||||||
|
snapshotID: string,
|
||||||
|
signal?: AbortSignal
|
||||||
|
): Promise<LibraryRestoreAvailability | null> {
|
||||||
|
const query = `scope=${encodeURIComponent(scope)}&snapshot_id=${encodeURIComponent(snapshotID)}`
|
||||||
|
const response = await fetch(`/api/v1/library/restore-availability?${query}`, {
|
||||||
|
headers: { Accept: 'application/json' },
|
||||||
|
signal,
|
||||||
|
})
|
||||||
|
const value = (await response.json().catch(() => null)) as unknown
|
||||||
|
if (!record(value)) return null
|
||||||
|
return {
|
||||||
|
allowed: bool(value.allowed),
|
||||||
|
reasons: Array.isArray(value.reasons) ? value.reasons.map(text).filter(Boolean) : [],
|
||||||
|
scope: text(value.scope),
|
||||||
|
snapshot_id: text(value.snapshot_id),
|
||||||
|
authorization: record(value.authorization)
|
||||||
|
? {
|
||||||
|
state: text(value.authorization.state) === 'active' ? 'active' : 'none',
|
||||||
|
authorized_by: text(value.authorization.authorized_by) || undefined,
|
||||||
|
expires_at: text(value.authorization.expires_at) || undefined,
|
||||||
|
reason: text(value.authorization.reason) || undefined,
|
||||||
|
}
|
||||||
|
: { state: 'none' },
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 恢复整套生产知识状态:同一份快照的文档、分块、索引、资源与 Wiki 一起回放。
|
||||||
|
export interface LibraryRestoreAccepted {
|
||||||
|
job_id: string
|
||||||
|
scope: string
|
||||||
|
reused: boolean
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function enqueueLibraryRestore(scope: string, snapshotID: string): Promise<LibraryRestoreAccepted> {
|
||||||
|
const response = await fetch('/api/v1/library/restores', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
|
||||||
|
body: JSON.stringify({ scope, snapshot_id: snapshotID }),
|
||||||
|
})
|
||||||
|
const value = (await response.json().catch(() => null)) as unknown
|
||||||
|
if (!response.ok) {
|
||||||
|
if (response.status === 403) throw new Error('按版本恢复仅管理员可发起。')
|
||||||
|
if (response.status === 503 && record(value)) {
|
||||||
|
const reasons = Array.isArray(value.reasons) ? value.reasons.map(text).filter(Boolean) : []
|
||||||
|
throw new Error(reasons.length > 0 ? reasons.join(';') : text(value.error) || '恢复条件不满足')
|
||||||
|
}
|
||||||
|
if (response.status === 409 && record(value)) throw new Error(text(value.error) || '该目标已有在飞作业')
|
||||||
|
throw new Error(`发起恢复失败(HTTP ${response.status})`)
|
||||||
|
}
|
||||||
|
if (!record(value) || !record(value.job)) throw new Error('恢复作业响应无效')
|
||||||
|
return { job_id: text(value.job.id), scope: text(value.job.scope), reused: bool(value.reused) }
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface LibraryRestoreGrant {
|
||||||
|
id: string
|
||||||
|
authorized_by: string
|
||||||
|
expires_at: string
|
||||||
|
}
|
||||||
|
|
||||||
|
// 同步授权与执行是两步:授权把"范围 + 目标快照 + 参数摘要"钉死并给很短的有效期。
|
||||||
|
// 它不能顶替恢复授权(摘要不同),也不自动签发 —— 必须由管理员显式点击。
|
||||||
|
export async function createLibrarySyncAuthorization(
|
||||||
|
scope: string,
|
||||||
|
snapshotID: string,
|
||||||
|
minutes = 30
|
||||||
|
): Promise<LibraryRestoreGrant> {
|
||||||
|
const response = await fetch('/api/v1/library/authorizations', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
|
||||||
|
body: JSON.stringify({ scope, snapshot_id: snapshotID, action: 'sync', expires_in_minutes: minutes }),
|
||||||
|
})
|
||||||
|
const value = (await response.json().catch(() => null)) as unknown
|
||||||
|
if (!response.ok) {
|
||||||
|
if (response.status === 403) throw new Error('生产发布授权仅管理员可签发。')
|
||||||
|
if (response.status === 400 && record(value)) throw new Error(text(value.error) || '授权参数不合法')
|
||||||
|
throw new Error(`签发同步授权失败(HTTP ${response.status})`)
|
||||||
|
}
|
||||||
|
if (!record(value) || !record(value.authorization)) throw new Error('授权响应无效')
|
||||||
|
return {
|
||||||
|
id: text(value.authorization.id),
|
||||||
|
authorized_by: text(value.authorization.authorized_by),
|
||||||
|
expires_at: text(value.authorization.expires_at),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface LibrarySyncAccepted {
|
||||||
|
job_id: string
|
||||||
|
scope: string
|
||||||
|
reused: boolean
|
||||||
|
}
|
||||||
|
|
||||||
|
// 启动同步:闸口未齐时接口返回 503 与逐条缺项,页面照实显示,不假装已启动。
|
||||||
|
export async function enqueueLibrarySync(scope: string, targetSnapshotID: string): Promise<LibrarySyncAccepted> {
|
||||||
|
const response = await fetch('/api/v1/library/syncs', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
|
||||||
|
body: JSON.stringify({ scope, target_snapshot_id: targetSnapshotID }),
|
||||||
|
})
|
||||||
|
const value = (await response.json().catch(() => null)) as unknown
|
||||||
|
if (!response.ok) {
|
||||||
|
if (record(value) && Array.isArray(value.reasons)) {
|
||||||
|
const reasons = value.reasons.map(text).filter(Boolean)
|
||||||
|
if (reasons.length > 0) throw new Error(`启动同步被拒绝:${reasons.join(';')}`)
|
||||||
|
}
|
||||||
|
if (record(value) && text(value.error)) throw new Error(text(value.error))
|
||||||
|
throw new Error(`启动同步失败(HTTP ${response.status})`)
|
||||||
|
}
|
||||||
|
if (!record(value) || !record(value.job)) throw new Error('同步作业响应无效')
|
||||||
|
return { job_id: text(value.job.id), scope: text(value.job.scope), reused: bool(value.reused) }
|
||||||
|
}
|
||||||
|
|
||||||
|
// 建立目标权威基线:目标非空又没有生效版本时的唯一入口(维护隔离 + 真实捕获 + 校验)。
|
||||||
|
export async function enqueueLibraryBaseline(scope: string): Promise<LibrarySyncAccepted> {
|
||||||
|
const response = await fetch('/api/v1/library/baselines', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
|
||||||
|
body: JSON.stringify({ scope }),
|
||||||
|
})
|
||||||
|
const value = (await response.json().catch(() => null)) as unknown
|
||||||
|
if (!response.ok) {
|
||||||
|
if (response.status === 403) throw new Error('建立目标权威基线仅管理员可发起。')
|
||||||
|
if (record(value) && text(value.error)) throw new Error(text(value.error))
|
||||||
|
throw new Error(`建立基线失败(HTTP ${response.status})`)
|
||||||
|
}
|
||||||
|
if (!record(value) || !record(value.job)) throw new Error('基线作业响应无效')
|
||||||
|
return { job_id: text(value.job.id), scope: text(value.job.scope), reused: bool(value.reused) }
|
||||||
|
}
|
||||||
|
|
||||||
|
// 恢复授权与执行是两件事:授权把"目标 + 快照 + 参数"钉死并给很短的有效期,
|
||||||
|
// 同步授权不能顶替恢复授权(摘要不同)。
|
||||||
|
export async function createLibraryRestoreAuthorization(
|
||||||
|
scope: string,
|
||||||
|
snapshotID: string,
|
||||||
|
minutes = 30
|
||||||
|
): Promise<LibraryRestoreGrant> {
|
||||||
|
const response = await fetch('/api/v1/library/authorizations', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
|
||||||
|
body: JSON.stringify({ scope, snapshot_id: snapshotID, action: 'restore', expires_in_minutes: minutes }),
|
||||||
|
})
|
||||||
|
const value = (await response.json().catch(() => null)) as unknown
|
||||||
|
if (!response.ok) {
|
||||||
|
if (response.status === 403) throw new Error('生产发布授权仅管理员可签发。')
|
||||||
|
throw new Error(`签发恢复授权失败(HTTP ${response.status})`)
|
||||||
|
}
|
||||||
|
if (!record(value) || !record(value.authorization)) throw new Error('授权响应无效')
|
||||||
|
return {
|
||||||
|
id: text(value.authorization.id),
|
||||||
|
authorized_by: text(value.authorization.authorized_by),
|
||||||
|
expires_at: text(value.authorization.expires_at),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export async function fetchLibraryCaptures(signal?: AbortSignal): Promise<LibraryCaptureJob[]> {
|
export async function fetchLibraryCaptures(signal?: AbortSignal): Promise<LibraryCaptureJob[]> {
|
||||||
const response = await fetch('/api/v1/library/captures', {
|
const response = await fetch('/api/v1/library/captures', {
|
||||||
headers: { Accept: 'application/json' },
|
headers: { Accept: 'application/json' },
|
||||||
@@ -311,6 +475,18 @@ export interface LibraryMaintenanceView {
|
|||||||
maintenance?: LibraryMaintenanceRecord | null
|
maintenance?: LibraryMaintenanceRecord | null
|
||||||
instance?: string
|
instance?: string
|
||||||
instance_maintenance?: LibraryInstanceMaintenanceRecord | null
|
instance_maintenance?: LibraryInstanceMaintenanceRecord | null
|
||||||
|
// 能力是配置事实,和"此刻有没有窗口"分开:未运行维护 ≠ 没有维护能力。
|
||||||
|
capability?: LibraryMaintenanceCapability | null
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface LibraryMaintenanceCapability {
|
||||||
|
configured: boolean
|
||||||
|
missing: string[]
|
||||||
|
instance?: string
|
||||||
|
project?: string
|
||||||
|
writers?: string[]
|
||||||
|
ingress_service?: string
|
||||||
|
note?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function fetchLibraryMaintenance(
|
export async function fetchLibraryMaintenance(
|
||||||
@@ -391,6 +567,18 @@ export interface LibraryGate {
|
|||||||
reasons: string[]
|
reasons: string[]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface LibraryExecutorView {
|
||||||
|
available: boolean
|
||||||
|
reason?: string
|
||||||
|
instance?: string
|
||||||
|
project?: string
|
||||||
|
source_app?: string
|
||||||
|
archive_root?: string
|
||||||
|
busy: boolean
|
||||||
|
degraded?: string
|
||||||
|
observed_at?: string
|
||||||
|
}
|
||||||
|
|
||||||
export interface LibraryOverview {
|
export interface LibraryOverview {
|
||||||
scope: string
|
scope: string
|
||||||
test_sealed?: LibrarySnapshot
|
test_sealed?: LibrarySnapshot
|
||||||
@@ -404,6 +592,8 @@ export interface LibraryOverview {
|
|||||||
authorization: LibraryAuthorizationView
|
authorization: LibraryAuthorizationView
|
||||||
capture_available: boolean
|
capture_available: boolean
|
||||||
capture_reason?: string
|
capture_reason?: string
|
||||||
|
// 执行器是 worker 的真实观测:指向哪个实例、何时观测、是否降级/在跑。
|
||||||
|
executor?: LibraryExecutorView
|
||||||
now: string
|
now: string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||||
|
import { createLibrarySyncAuthorization, enqueueLibraryBaseline, enqueueLibrarySync } from './librarySnapshots'
|
||||||
|
|
||||||
|
// 这些用例钉住"两步走"的接线:授权与执行是两个请求,参数必须与接口契约一致。
|
||||||
|
// 任何一步把 action 写错、或把 snapshot 换掉,都会在真实调用里悄悄失效(授权永远匹配不上)。
|
||||||
|
const jsonResponse = (status: number, body: unknown) =>
|
||||||
|
new Response(JSON.stringify(body), { status, headers: { 'Content-Type': 'application/json' } })
|
||||||
|
|
||||||
|
afterEach(() => vi.restoreAllMocks())
|
||||||
|
|
||||||
|
describe('发布页的同步授权与执行接线', () => {
|
||||||
|
it('签同步授权必须声明 action=sync 与有限有效期', async () => {
|
||||||
|
const fetchMock = vi.fn().mockResolvedValue(
|
||||||
|
jsonResponse(201, {
|
||||||
|
authorization: { id: 'libauth-sync-1', authorized_by: 'admin', expires_at: '2026-10-29T10:00:00Z' },
|
||||||
|
})
|
||||||
|
)
|
||||||
|
vi.stubGlobal('fetch', fetchMock)
|
||||||
|
const grant = await createLibrarySyncAuthorization('internal', 'snap-1', 30)
|
||||||
|
expect(grant.id).toBe('libauth-sync-1')
|
||||||
|
const [url, init] = fetchMock.mock.calls[0]
|
||||||
|
expect(url).toBe('/api/v1/library/authorizations')
|
||||||
|
expect(JSON.parse(init.body)).toEqual({
|
||||||
|
scope: 'internal',
|
||||||
|
snapshot_id: 'snap-1',
|
||||||
|
action: 'sync',
|
||||||
|
expires_in_minutes: 30,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
it('启动同步把准确的 range 与目标快照传给 /library/syncs', async () => {
|
||||||
|
const fetchMock = vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValue(jsonResponse(202, { job: { id: 'libsync-internal-1', scope: 'internal' }, reused: false }))
|
||||||
|
vi.stubGlobal('fetch', fetchMock)
|
||||||
|
const accepted = await enqueueLibrarySync('internal', 'snap-1')
|
||||||
|
expect(accepted.job_id).toBe('libsync-internal-1')
|
||||||
|
const [url, init] = fetchMock.mock.calls[0]
|
||||||
|
expect(url).toBe('/api/v1/library/syncs')
|
||||||
|
expect(JSON.parse(init.body)).toEqual({ scope: 'internal', target_snapshot_id: 'snap-1' })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('闸口未齐时把接口给的缺项原样抛出,而不是假装已启动', async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
'fetch',
|
||||||
|
vi.fn().mockResolvedValue(
|
||||||
|
jsonResponse(503, {
|
||||||
|
code: 'library_sync_blocked',
|
||||||
|
reasons: ['目标非空且没有生效版本记录', '没有与目标快照匹配的有效发布授权'],
|
||||||
|
})
|
||||||
|
)
|
||||||
|
)
|
||||||
|
await expect(enqueueLibrarySync('internal', 'snap-1')).rejects.toThrow(
|
||||||
|
/目标非空且没有生效版本记录;没有与目标快照匹配的有效发布授权/
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('建立基线只传 range,并返回作业号', async () => {
|
||||||
|
const fetchMock = vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValue(jsonResponse(202, { job: { id: 'libbase-internal-1', scope: 'internal' }, reused: false }))
|
||||||
|
vi.stubGlobal('fetch', fetchMock)
|
||||||
|
const accepted = await enqueueLibraryBaseline('internal')
|
||||||
|
expect(accepted.job_id).toBe('libbase-internal-1')
|
||||||
|
const [url, init] = fetchMock.mock.calls[0]
|
||||||
|
expect(url).toBe('/api/v1/library/baselines')
|
||||||
|
expect(JSON.parse(init.body)).toEqual({ scope: 'internal' })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('建立基线被拒时照实说明(例如目标为空或已有生效版本)', async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
'fetch',
|
||||||
|
vi.fn().mockResolvedValue(
|
||||||
|
jsonResponse(409, {
|
||||||
|
code: 'library_baseline_exists',
|
||||||
|
error: '该目标已有生效版本,不需要再建基线;换版本请走同步或恢复',
|
||||||
|
})
|
||||||
|
)
|
||||||
|
)
|
||||||
|
await expect(enqueueLibraryBaseline('internal')).rejects.toThrow(/该目标已有生效版本/)
|
||||||
|
})
|
||||||
|
})
|
||||||
Reference in New Issue
Block a user