fix(library): 收紧恢复派发真实绑定与同事务授权
build / build (push) Has been cancelled

为什么:Astra 复核证实五组实现缺陷——正式装配缺 scope/快照且来源与 before-image
混用;激活撤销不在 CAS 事务内、lease 用事务起始时间;激活只接受任意字符串且撤销可
被 SQL 重置/删除;准入忽略绑定冲突与 prepared 矛盾、收尾只看当前 plan;prepared 已
提交但 mark-intent 未落盘时被误判 legacy。

改了什么:
- 绑定来自原锚点(scope/来源快照/backup/实例/目标指纹/索引 UUID/恢复投递),每次
  重建都实时重读目标身份并与冻结值比较;
- activation 行锁与 CAS 同事务,lease 用 clock_timestamp(),同事务核对 job/operation/
  attempt/维护窗口/恢复 queue 与 operation,同 effect 换 plan 摘要拒绝;
- 激活消费带摘要的证据文件(writer 退出+排空、目标/治理库身份、有效期),撤销终局,
  down 在 activation 有行时拒绝;
- 前置/收尾共用权威窗口全集,绑定冲突与 prepared 矛盾零写 held,合法 dispatching+
  精确 marker 仍可确认;
- prepared 无 mark-intent 仅在绑定一致且无矛盾时补同内容意图,旧无台账意图仍 held。

验证:真 PG 锁等待覆盖撤销先胜/CAS 先胜/等锁跨租约/错锚点/plan 别名/恢复投递;
COMMIT 已到服务端但回包丢失与未决事务分别读回;真 ES+PG+对象存储的 DS01 用例全部
通过;go test ./... 全绿。未迁移正式库、未部署、未激活。
This commit is contained in:
姚章浩
2026-11-02 18:09:26 +08:00
parent 3bce6c7a3f
commit 513aa9e6f5
22 changed files with 2200 additions and 228 deletions
+24
View File
@@ -1,5 +1,29 @@
# Changelog
## 开发快照(DS01 复核修复:真实绑定、同事务授权、可核激活与完整未决分类)
### Fixed
- **正式装配绑定**:`OpenProductionStack` 现在把 scope/实例与本次真实 attempt 身份传入目标;
来源快照与写前镜像分别取自原 `WriteSet.TargetSnapshotID` 与本次 backup,不再把同一个
`RestoreSnapshotID` 同时当来源和 before-image。缺锚点或二者相同在写之前失败关闭。
- **派发线性化**:激活撤销与 `prepared→dispatching` CAS 在同一事务里用 activation 行锁串行化;
lease 在取锁之后用 `clock_timestamp()` 判定;同事务核对原 job/operation/attempt、来源/写集/
维护窗口与恢复投递(queue + recovery operation)绑定;同 effect 换 plan 摘要被拒绝。
- **可核激活**:激活改为消费带 sha256 的证据文件(目标身份/指纹/实例、治理库名、协议构建、
逐 writer 的退出与排空观察、有效期);撤销不可回退为 `NULL`、activation 行不可删除,
down 在仅有 activation/撤销证据时也拒绝。
- **准入/收尾完整分类**:绑定冲突、`prepared` 与 marker/claimed/finalize 矛盾在零目标写之前
held;合法 `dispatching`+精确 marker 仍走确认分支。收尾改用原 window 权威台账全集(跨 plan),
拒绝计划外记录、换 plan 别名与缺 `restored` 证据的 effect。
- **可恢复区段**:`prepared` 已提交但 mark-intent 未落盘时,核对绑定一致且无矛盾后补写同一
意图并继续;无台账的旧意图仍 held,r33 不回填。
### Security
- 绑定冻结实际目标实例/指纹/索引 UUID,同名重建的目标或索引会让旧证明失效;它们用于识别
对象,不用于证明历史未发送。协议仍不声称 ES 端 fence 或旧 `dispatching` owner 零迟到写。
## 开发快照(DS01:删除型恢复的派发台账与唯一发送许可)
### Added
+37 -24
View File
@@ -15,57 +15,70 @@ PG 快照回滚/手工删行之后的协议有效性(这类历史倒退必须
## 状态与唯一发送路径
```
prepared --(CAS, 同一短事务内校验 job/operation/attempt/lease/binding)--> dispatching
prepared --(CAS, 同一短事务内校验 activation/job/operation/attempt/lease/binding)--> dispatching
dispatching --(真实 receipt + 当前 owner/lease)--> marker_confirmed
```
- 发送许可只由 `AcquireRestoreMarker` 在 **COMMIT 明确返回后**给出,`RestoreSendPermit` 消费一次即作废。
- CAS 事务按 `job → operation → activation → active attempt → effect` 取锁:
- 激活撤销(`revoked_at`)与 CAS 在同一事务里线性化:撤销先完成则拒绝,CAS 先取锁则撤销排队;
- lease 用 `clock_timestamp()` 在取锁之后判定,等锁跨过租约即拒绝;
- attempt 必须属于原 window job;job 的 owner/epoch/lease、operation 的 scope/来源快照/
已登记写集摘要、实例维护窗口(未释放且属于该作业)逐项核对;
- 恢复路径另核 recovery queue job(运行中且有 lease)与 recovery operation(`kind=recover`、
绑定同一 backup);
- 同一 effect 换 plan 摘要会被台账拒绝(不能靠新 key 重开证明)。
- 不可变绑定来自原持久锚点:`scope`、来源快照(发布基线)、写前镜像(backup)、目标实例/
实际指纹/索引 UUID、变更/标记/请求摘要与 attempt 身份。来源与写前镜像不允许是同一份快照。
目标指纹与索引 UUID 每次重建绑定都实时重读并与本次恢复冻结值比较:同名重建的目标/索引
无法复用旧证明。
- 恢复期 `_create` 只从 `sendRestoreMarkerOnce` 发出;preflight/故障钩子都在 CAS 之前。
- 请求设置 `GetBody=nil`,Go 传输层不会在 keep-alive 重连时重放该 mutation。
## Restore 准入与收尾
`Restore` 在 `AdoptWriteSet` 之后、任何阶段写入之前调用纯读准入:
`Restore` 在 `AdoptWriteSet` 之后、任何阶段写入之前调用纯读准入,并先对照原 window 的
**权威台账全集**(跨 plan 摘要):出现另一份 plan、计划外 effect 或不可识别状态立即 held。
| 台账/证据 | 结果 |
|---|---|
| 无意图、无台账 | 首次恢复,可建立 `prepared` |
| 无意图、无证据、无台账 | 首次恢复,可建立 `prepared` |
| 旧意图、无台账 | held(旧协议未知历史,不补 prepared) |
| `prepared` | 可竞争一次许可 |
| `prepared` + 绑定冲突(`Held`) | held |
| `prepared` + 已存在 claimed/finalize/目标精确 marker | held(协议证据矛盾) |
| `prepared` + 无矛盾 | 可竞争一次许可 |
| `prepared` + mark-intent 未落盘(文件失败/崩溃区段) | 核对绑定一致后补写**同内容**意图,再竞争 |
| `dispatching` + 目标无精确 marker | held(零新发送) |
| `dispatching` + 精确 marker/内容/世代 | 只走确认分支,不发 create |
| `marker_confirmed` + 无 `restored` 证据 | held |
| `restored` 证据与目标内容/世代一致 | 终态,允许重放 |
收尾前(整库指纹比对之后)再检查一次:仍有 `restored` 缺失或停在 `dispatching` 的 effect ⇒ 拒绝
返回成功,维护窗口不会被释放。
收尾前(整库指纹比对之后)以同一份权威全集再检查一次:任何 `prepared`/`dispatching` 记录、
计划外记录、换 plan 摘要、或缺失 `restored` 文件证据的 effect 都拒绝返回成功,维护窗口不会
被释放。
## 激活(部署动作,不属于本开发包)
新能力默认关闭。启用需要两步,缺一不可:
1. 部署配置 `LIBRARY_RESTORE_DISPATCH_V2=true`(只表示允许装配);
2. 部署流程在治理库登记"旧 writer 已退出并排空"的证据:
2. 部署流程生成一份**证据文件**,并由 `Store.ActivateRestoreDispatch` 登记。代码会:
- 重算文件 sha256 并与调用方给出的摘要比较(不接受文件自报值);
- 严格解析并校验:协议版本、目标身份/实例/指纹、治理库名、协议构建、签署人与时间窗口;
- 要求每个受控 writer 都登记 `stopped + queue_drained + exited_at + drain_evidence`;
- 读取现场 `current_database()` 与证据里的治理库名比较;
- 目标身份/指纹与调用方声明的现场身份一致。
任何一项不满足都不建立 active 行。
```sql
INSERT INTO library_index_restore_dispatch_activation
(target_identity, protocol_version, activated_by, drain_evidence_digest)
VALUES ($1, 1, $2, $3)
ON CONFLICT (target_identity) DO NOTHING;
```
撤销是终局:`revoked_at` 一旦写入,SQL 不能改回 `NULL`、不能改时间,activation 行也不能删除;
重新开放同一物理目标只能换新的目标身份。
`target_identity` 必须等于 `appconfig.Config.LibraryTargetIdentity()` 的值;`drain_evidence_digest`
是部署证据的摘要。撤销:
```sql
UPDATE library_index_restore_dispatch_activation SET revoked_at=now()
WHERE target_identity=$1 AND revoked_at IS NULL;
```
登记的是运维声明,代码只保证"没有登记就不发放许可",不验证声明本身为真。
边界:代码核验的是证据文件完整性、结构完整性与目标/治理库身份;"这些 writer 确实已退出"
仍是部署流程的受控 attestation(附退出时间与排空摘要),不是本进程能独立观察到的外部事实。
## 迁移与回滚
- `000078_library_restore_dispatch.up.sql` 只建治理库表与触发器,不碰 WeKnora Schema。
- `000078_..._down.sql` 在 `library_index_restore_dispatch` 有任意行时**拒绝执行**:这些行是
"是否发送过"的唯一证明,抹掉等于把 unknown 伪装成 prepared。
- `000078_..._down.sql` 在 `library_index_restore_dispatch` 或 activation 表有**任意行**时拒绝执行:
这些行是"是否发送过 / 是否曾被授权"的唯一证明,抹掉等于把 unknown 伪装成 prepared,或把
已撤销的激活悄悄删掉。
@@ -60,16 +60,26 @@ func (k RestoreMarkerKey) Validate() error {
// RestoreMarkerBinding 是派发许可的不可变绑定。目标、快照、计划、变更体与标记体任一
// 变化都必须 held,不能另起一行继续写。
//
// SourceSnapshot 是发布基线(原作业要同步到的那份来源),BackupSnapshot 是本次失败
// 写集的写前镜像;两者必须来自原持久锚点且不能互相代替。TargetInstance /
// TargetFingerprint / IndexUUID 冻结"实际目标",用于识别目标被同名字替换;它们不证明
// 历史上没有发送过任何请求。
type RestoreMarkerBinding struct {
Key RestoreMarkerKey `json:"key"`
Scope string `json:"scope"`
SourceSnapshot string `json:"source_snapshot"`
BackupSnapshot string `json:"backup_snapshot"`
ChangeDigest string `json:"change_digest"`
MarkerDigest string `json:"marker_digest"`
RequestDigest string `json:"request_digest"`
Owner string `json:"owner"`
Epoch int64 `json:"epoch"`
Key RestoreMarkerKey `json:"key"`
Scope string `json:"scope"`
SourceSnapshot string `json:"source_snapshot"`
BackupSnapshot string `json:"backup_snapshot"`
TargetInstance string `json:"target_instance"`
TargetFingerprint string `json:"target_fingerprint"`
IndexUUID string `json:"index_uuid"`
RecoveryJobID string `json:"recovery_job_id,omitempty"`
RecoveryOperationID string `json:"recovery_operation_id,omitempty"`
ChangeDigest string `json:"change_digest"`
MarkerDigest string `json:"marker_digest"`
RequestDigest string `json:"request_digest"`
Owner string `json:"owner"`
Epoch int64 `json:"epoch"`
}
func (b RestoreMarkerBinding) Validate() error {
@@ -78,13 +88,20 @@ func (b RestoreMarkerBinding) Validate() error {
}
for name, value := range map[string]string{
"scope": b.Scope, "source snapshot": b.SourceSnapshot, "backup snapshot": b.BackupSnapshot,
"change digest": b.ChangeDigest, "marker digest": b.MarkerDigest,
"target instance": b.TargetInstance, "target fingerprint": b.TargetFingerprint,
"index uuid": b.IndexUUID, "change digest": b.ChangeDigest, "marker digest": b.MarkerDigest,
"request digest": b.RequestDigest, "owner": b.Owner,
} {
if strings.TrimSpace(value) == "" {
return fmt.Errorf("restore marker binding lacks %s", name)
}
}
if b.SourceSnapshot == b.BackupSnapshot {
return errors.New("restore marker binding uses the source snapshot as the backup image")
}
if b.RecoveryJobID == "" != (b.RecoveryOperationID == "") {
return errors.New("restore marker binding has a partial recovery delivery identity")
}
if b.Epoch <= 0 {
return errors.New("restore marker binding lacks a positive attempt epoch")
}
@@ -97,8 +114,9 @@ func (b RestoreMarkerBinding) Digest() (string, error) {
return "", err
}
joined := strings.Join([]string{
"v1", b.Key.TargetIdentity, b.Key.OperationID, b.Key.WindowJobID, b.Key.PlanDigest,
"v2", b.Key.TargetIdentity, b.Key.OperationID, b.Key.WindowJobID, b.Key.PlanDigest,
b.Key.Index, b.Key.DocumentID, b.Key.Phase, b.Scope, b.SourceSnapshot, b.BackupSnapshot,
b.TargetInstance, b.TargetFingerprint, b.IndexUUID, b.RecoveryJobID, b.RecoveryOperationID,
b.ChangeDigest, b.MarkerDigest, b.RequestDigest,
}, "\x00")
sum := sha256.Sum256([]byte(joined))
@@ -151,13 +169,24 @@ func (p *RestoreSendPermit) Consume(key RestoreMarkerKey, bindingDigest string)
return nil
}
// RestoreDispatchRecord 是权威台账里"这个原 window 还有哪些未决 effect"的一条。
// 收尾与前置都查同一份记录集,不能只遍历当前 plan 的文档列表。
type RestoreDispatchRecord struct {
PlanDigest string `json:"plan_digest"`
Index string `json:"index"`
DocumentID string `json:"document_id"`
State string `json:"state"`
}
// RestoreMarkerGate 是唯一能授予恢复标记 create 发送权的接口。
//
// Inspect 纯读、不返回许可;Prepare 只建立 prepared;Acquire 在独立短事务里做
// prepared→dispatching 的 CAS,只有明确提交后才返回许可;Confirm 只记录真实回执。
// prepared→dispatching 的 CAS,只有明确提交后才返回许可;Confirm 只记录真实回执;
// UnresolvedRestoreDispatches 返回原 window 的权威未决全集(跨 plan,只读)。
type RestoreMarkerGate interface {
InspectRestoreMarker(ctx context.Context, b RestoreMarkerBinding) (RestoreDispatchStatus, bool, error)
PrepareRestoreMarker(ctx context.Context, b RestoreMarkerBinding) (RestoreDispatchStatus, error)
AcquireRestoreMarker(ctx context.Context, b RestoreMarkerBinding) (*RestoreSendPermit, RestoreDispatchStatus, error)
ConfirmRestoreMarker(ctx context.Context, b RestoreMarkerBinding, receipt []byte) error
UnresolvedRestoreDispatches(ctx context.Context, targetIdentity, operationID, windowJobID string) ([]RestoreDispatchRecord, error)
}
@@ -10,7 +10,8 @@ func restoreBinding() RestoreMarkerBinding {
return RestoreMarkerBinding{
Key: RestoreMarkerKey{TargetIdentity: "target-a", OperationID: "op-1", WindowJobID: "job-1",
PlanDigest: "plan-1", Index: "kg-index", DocumentID: "doc-1", Phase: RestoreMarkerPhaseCreate},
Scope: "internal", SourceSnapshot: "snap-1", BackupSnapshot: "snap-1",
Scope: "internal", SourceSnapshot: "snap-src", BackupSnapshot: "snap-backup",
TargetInstance: "inst-1", TargetFingerprint: "fingerprint-1", IndexUUID: "uuid-1",
ChangeDigest: "change-1", MarkerDigest: "marker-1", RequestDigest: "request-1",
Owner: "worker-a#queue#1", Epoch: 3,
}
@@ -18,18 +19,22 @@ func restoreBinding() RestoreMarkerBinding {
func TestRestoreMarkerBindingRejectsIncompleteIdentity(t *testing.T) {
for name, mutate := range map[string]func(*RestoreMarkerBinding){
"target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "" },
"operation": func(b *RestoreMarkerBinding) { b.Key.OperationID = "" },
"window": func(b *RestoreMarkerBinding) { b.Key.WindowJobID = "" },
"plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "" },
"index": func(b *RestoreMarkerBinding) { b.Key.Index = "" },
"document": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "" },
"change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "" },
"marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "" },
"request": func(b *RestoreMarkerBinding) { b.RequestDigest = "" },
"owner": func(b *RestoreMarkerBinding) { b.Owner = "" },
"epoch": func(b *RestoreMarkerBinding) { b.Epoch = 0 },
"phase": func(b *RestoreMarkerBinding) { b.Key.Phase = "apply-create" },
"target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "" },
"operation": func(b *RestoreMarkerBinding) { b.Key.OperationID = "" },
"window": func(b *RestoreMarkerBinding) { b.Key.WindowJobID = "" },
"plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "" },
"index": func(b *RestoreMarkerBinding) { b.Key.Index = "" },
"document": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "" },
"change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "" },
"marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "" },
"request": func(b *RestoreMarkerBinding) { b.RequestDigest = "" },
"owner": func(b *RestoreMarkerBinding) { b.Owner = "" },
"epoch": func(b *RestoreMarkerBinding) { b.Epoch = 0 },
"phase": func(b *RestoreMarkerBinding) { b.Key.Phase = "apply-create" },
"instance": func(b *RestoreMarkerBinding) { b.TargetInstance = "" },
"fingerprint": func(b *RestoreMarkerBinding) { b.TargetFingerprint = "" },
"index uuid": func(b *RestoreMarkerBinding) { b.IndexUUID = "" },
"source as backup": func(b *RestoreMarkerBinding) { b.BackupSnapshot = b.SourceSnapshot },
} {
binding := restoreBinding()
mutate(&binding)
@@ -51,16 +56,20 @@ func TestRestoreMarkerBindingDigestCoversEveryBoundValue(t *testing.T) {
t.Fatal("binding digest is not deterministic")
}
for name, mutate := range map[string]func(*RestoreMarkerBinding){
"target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "target-b" },
"index": func(b *RestoreMarkerBinding) { b.Key.Index = "other-index" },
"doc": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "doc-2" },
"plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "plan-2" },
"backup": func(b *RestoreMarkerBinding) { b.BackupSnapshot = "snap-2" },
"change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "change-2" },
"marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "marker-2" },
"request": func(b *RestoreMarkerBinding) { b.RequestDigest = "request-2" },
"scope": func(b *RestoreMarkerBinding) { b.Scope = "external" },
"source": func(b *RestoreMarkerBinding) { b.SourceSnapshot = "snap-9" },
"target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "target-b" },
"index": func(b *RestoreMarkerBinding) { b.Key.Index = "other-index" },
"doc": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "doc-2" },
"plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "plan-2" },
"backup": func(b *RestoreMarkerBinding) { b.BackupSnapshot = "snap-2" },
"change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "change-2" },
"marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "marker-2" },
"request": func(b *RestoreMarkerBinding) { b.RequestDigest = "request-2" },
"scope": func(b *RestoreMarkerBinding) { b.Scope = "external" },
"source": func(b *RestoreMarkerBinding) { b.SourceSnapshot = "snap-9" },
"instance": func(b *RestoreMarkerBinding) { b.TargetInstance = "inst-2" },
"fingerprint": func(b *RestoreMarkerBinding) { b.TargetFingerprint = "fingerprint-2" },
"index uuid": func(b *RestoreMarkerBinding) { b.IndexUUID = "uuid-2" },
"recovery queue": func(b *RestoreMarkerBinding) { b.RecoveryJobID = "q-1"; b.RecoveryOperationID = "op-r" },
} {
binding := restoreBinding()
mutate(&binding)
@@ -0,0 +1,129 @@
package libraryrelease
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"time"
)
// 恢复派发激活证据(DS01-F3)。
//
// 配置开关与"三个非空字符串"都不是独占证明。激活必须消费一份**可核验的证据文件**:
// - 文件字节的 sha256 由调用方给出并在读取后重算(不是文件里自报的布尔值);
// - 证据只对某一目标身份/目标指纹/治理库生效;
// - 受控 writer 必须逐个登记"已退出 + 队列已排空"以及各自的退出观察;
// - 证据有明确的签署人与时间窗口,过期不接受。
//
// 边界(必须如实写进报告):代码能核验文件完整性、结构完整性、目标/治理库身份与
// 新鲜度;"这些 writer 真的已经停止"仍是部署流程的受控 attestation(附退出时间和
// 排空证据摘要),不是本进程运行时能独立观察到的外部事实。没有这份证据就不建立
// active 行——这一点由代码强制。
const RestoreDispatchActivationMaxAge = 2 * time.Hour
type RestoreDispatchWriterDrain struct {
Name string `json:"name"`
Kind string `json:"kind"`
Stopped bool `json:"stopped"`
ExitedAt *time.Time `json:"exited_at"`
QueueDrained bool `json:"queue_drained"`
DrainEvidence string `json:"drain_evidence"`
}
type RestoreDispatchActivationEvidence struct {
ProtocolVersion int `json:"protocol_version"`
TargetIdentity string `json:"target_identity"`
TargetFingerprint string `json:"target_fingerprint"`
TargetInstance string `json:"target_instance"`
GovernanceDatabase string `json:"governance_database"`
ProtocolBuild string `json:"protocol_build"`
AttestedBy string `json:"attested_by"`
AttestedAt time.Time `json:"attested_at"`
Writers []RestoreDispatchWriterDrain `json:"writers"`
}
func (e RestoreDispatchActivationEvidence) Validate(now time.Time, maxAge time.Duration) error {
if e.ProtocolVersion != 1 {
return fmt.Errorf("restore dispatch activation protocol version %d is not supported", e.ProtocolVersion)
}
for name, value := range map[string]string{
"target identity": e.TargetIdentity, "target fingerprint": e.TargetFingerprint,
"target instance": e.TargetInstance, "governance database": e.GovernanceDatabase,
"protocol build": e.ProtocolBuild, "attested by": e.AttestedBy,
} {
if strings.TrimSpace(value) == "" {
return fmt.Errorf("restore dispatch activation evidence lacks %s", name)
}
}
if len(e.Writers) == 0 {
return errors.New("restore dispatch activation evidence lists no controlled writers")
}
seen := map[string]bool{}
for _, writer := range e.Writers {
if strings.TrimSpace(writer.Name) == "" || strings.TrimSpace(writer.Kind) == "" {
return errors.New("restore dispatch activation writer lacks name or kind")
}
if seen[writer.Name] {
return fmt.Errorf("restore dispatch activation writer %s is listed twice", writer.Name)
}
seen[writer.Name] = true
if !writer.Stopped || !writer.QueueDrained {
return fmt.Errorf("restore dispatch activation writer %s has not exited and drained", writer.Name)
}
if writer.ExitedAt == nil || writer.ExitedAt.IsZero() {
return fmt.Errorf("restore dispatch activation writer %s has no exit observation", writer.Name)
}
if strings.TrimSpace(writer.DrainEvidence) == "" {
return fmt.Errorf("restore dispatch activation writer %s has no drain evidence", writer.Name)
}
if writer.ExitedAt.After(e.AttestedAt) {
return fmt.Errorf("restore dispatch activation writer %s exited after the attestation", writer.Name)
}
}
if e.AttestedAt.IsZero() || e.AttestedAt.After(now.Add(5*time.Minute)) {
return errors.New("restore dispatch activation attestation time is not credible")
}
if maxAge <= 0 {
maxAge = RestoreDispatchActivationMaxAge
}
if now.Sub(e.AttestedAt) > maxAge {
return fmt.Errorf("restore dispatch activation evidence is older than %s", maxAge)
}
return nil
}
// LoadRestoreDispatchActivationEvidence 读取并核验证据文件字节摘要。expectedDigest 必须
// 与文件实际 sha256 相同;不接受"文件里写的摘要"。
func LoadRestoreDispatchActivationEvidence(path, expectedDigest string, now time.Time, maxAge time.Duration) (RestoreDispatchActivationEvidence, error) {
var evidence RestoreDispatchActivationEvidence
if strings.TrimSpace(path) == "" || strings.TrimSpace(expectedDigest) == "" {
return evidence, errors.New("restore dispatch activation requires an evidence file and its digest")
}
raw, err := os.ReadFile(path)
if err != nil {
return evidence, fmt.Errorf("read restore dispatch activation evidence: %w", err)
}
sum := sha256.Sum256(raw)
if got := hex.EncodeToString(sum[:]); got != strings.ToLower(strings.TrimSpace(expectedDigest)) {
return evidence, errors.New("restore dispatch activation evidence digest does not match the file")
}
decoder := json.NewDecoder(strings.NewReader(string(raw)))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&evidence); err != nil {
return RestoreDispatchActivationEvidence{}, fmt.Errorf("parse restore dispatch activation evidence: %w", err)
}
if err := evidence.Validate(now, maxAge); err != nil {
return RestoreDispatchActivationEvidence{}, err
}
return evidence, nil
}
// CanonicalBytes 是写入台账的规范字节(同一份证据只有一种序列化)。
func (e RestoreDispatchActivationEvidence) CanonicalBytes() ([]byte, error) {
return json.Marshal(e)
}
+64 -8
View File
@@ -90,7 +90,49 @@ func scopeConfigOf(config appconfig.Config, scope string) (scopeConfig, error) {
}, nil
}
func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig,
// RestoreDispatchWiring 是恢复派发身份的唯一装配来源:真实 attempt(objectgateway
// claim)、原 window 作业、部署目标实例与 scope。adapter 不接受调用方另填身份。
type RestoreDispatchWiring struct {
TargetIdentity string
Scope string
Instance string
OperationID string
WindowJobID string
AttemptOwner string
AttemptEpoch int64
RecoveryJobID string
RecoveryOperationID string
}
// RestoreDispatchWiringFor 把本次真实执行的 claim 映射成派发身份,并在字段缺失时
// 失败关闭。它不读取快照:来源/写前镜像只能从原 WriteSet 与 backup 引用取得。
func RestoreDispatchWiringFor(config appconfig.Config, scope string, claim objectgateway.Claim) (RestoreDispatchWiring, error) {
if scope != "internal" && scope != "external" {
return RestoreDispatchWiring{}, fmt.Errorf("未知 scope %q", scope)
}
identity := config.LibraryTargetIdentity()
if identity == "" || config.LibraryTargetInstance == "" || config.LibraryProductionDatabaseURL == "" {
return RestoreDispatchWiring{}, errors.New("恢复派发缺少目标身份/实例")
}
if (scope == "internal" && config.LibraryProductionInternalKB == "") ||
(scope == "external" && config.LibraryProductionExternalKB == "") {
return RestoreDispatchWiring{}, errors.New("恢复派发缺少 scope 对应的知识库")
}
if claim.JobID == "" || claim.OperationID == "" || claim.Owner == "" || claim.AttemptEpoch <= 0 {
return RestoreDispatchWiring{}, errors.New("恢复派发缺少本次真实 attempt 身份")
}
if (claim.RecoveryJobID == "") != (claim.RecoveryOperationID == "") {
return RestoreDispatchWiring{}, errors.New("恢复投递身份不完整")
}
return RestoreDispatchWiring{
TargetIdentity: identity, Scope: scope, Instance: config.LibraryTargetInstance,
OperationID: claim.OperationID, WindowJobID: claim.JobID,
AttemptOwner: claim.Owner, AttemptEpoch: claim.AttemptEpoch,
RecoveryJobID: claim.RecoveryJobID, RecoveryOperationID: claim.RecoveryOperationID,
}, nil
}
func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig, scopeName string,
gate libraryrelease.RestoreMarkerGate) (*v071.ProductionLibraryTarget, error) {
objects, err := v071.NewS3ObjectStore(scope.s3)
if err != nil {
@@ -115,16 +157,30 @@ func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig,
// DS01:派发身份只能来自本次真实 attempt(objectgateway claim)与原窗口作业。
// gate 为 nil 时该能力整体关闭,adapter 对新续作 fail closed;baseline 永远传 nil。
claim := objectgateway.ClaimFromContext(ctx)
return v071.NewProductionLibraryTarget(ctx, v071.ProductionLibraryConfig{
targetConfig := v071.ProductionLibraryConfig{
Postgres: config.LibraryProductionDatabaseURL, SearchURL: config.LibraryProductionElasticURL,
SearchIndex: config.LibraryProductionIndex, Objects: writer, ObjectsReader: objects,
ArchiveRoot: config.LibraryArchiveRoot,
ObjectsBucket: config.LibraryProductionS3Bucket,
MaxDocuments: 200000,
RestoreDispatch: gate, RestoreTargetIdentity: config.LibraryTargetIdentity(),
RestoreOperationID: claim.OperationID, RestoreWindowJobID: claim.JobID,
RestoreAttemptOwner: claim.Owner, RestoreAttemptEpoch: claim.AttemptEpoch,
}, scope.target)
RestoreDispatch: gate,
}
if gate != nil {
wiring, err := RestoreDispatchWiringFor(config, scopeName, claim)
if err != nil {
return nil, err
}
targetConfig.RestoreTargetIdentity = wiring.TargetIdentity
targetConfig.RestoreScope = wiring.Scope
targetConfig.RestoreInstance = wiring.Instance
targetConfig.RestoreOperationID = wiring.OperationID
targetConfig.RestoreWindowJobID = wiring.WindowJobID
targetConfig.RestoreAttemptOwner = wiring.AttemptOwner
targetConfig.RestoreAttemptEpoch = wiring.AttemptEpoch
targetConfig.RestoreRecoveryJobID = wiring.RecoveryJobID
targetConfig.RestoreRecoveryOperationID = wiring.RecoveryOperationID
}
return v071.NewProductionLibraryTarget(ctx, targetConfig, scope.target)
}
func newBackups(config appconfig.Config, scope string, target *v071.ProductionLibraryTarget,
@@ -144,7 +200,7 @@ func OpenProductionStack(ctx context.Context, config appconfig.Config, scope, sn
if err != nil {
return nil, nil, err
}
target, err := newTarget(ctx, config, scopeCfg, gate)
target, err := newTarget(ctx, config, scopeCfg, scope, gate)
if err != nil {
return nil, nil, err
}
@@ -226,7 +282,7 @@ func OpenBaselineStack(ctx context.Context, config appconfig.Config, scope strin
if err != nil {
return nil, nil, err
}
target, err := newTarget(ctx, config, scopeCfg, nil)
target, err := newTarget(ctx, config, scopeCfg, scope, nil)
if err != nil {
return nil, nil, err
}
+71
View File
@@ -2,9 +2,80 @@ package librarystack
import (
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/objectgateway"
"testing"
)
// DS01-F1:正式装配必须把 scope/instance/真实 attempt 身份传到 adapter;快照绑定
// 不再来自这里(来源/写前镜像只从原 WriteSet 与 backup 引用取得)。
func TestRestoreDispatchWiringUsesRealClaim(t *testing.T) {
config := appconfig.Config{
LibraryTargetInstance: "inst-prod",
LibraryProductionDatabaseURL: "postgres://user:pass@127.0.0.1:5432/kg_prod?sslmode=disable",
LibraryProductionInternalKB: "kb-internal",
LibraryProductionExternalKB: "kb-external",
}
claim := objectgateway.Claim{
JobID: "job-window", OperationID: "op-original", Owner: "worker-1#queue#job-window",
AttemptEpoch: 3, RecoveryJobID: "queue-recovery", RecoveryOperationID: "op-recovery",
}
wiring, err := RestoreDispatchWiringFor(config, "internal", claim)
if err != nil {
t.Fatalf("real wiring: %v", err)
}
if wiring.Scope != "internal" || wiring.Instance != "inst-prod" {
t.Fatalf("scope/instance = %q/%q", wiring.Scope, wiring.Instance)
}
if wiring.TargetIdentity == "" || wiring.TargetIdentity == "unconfigured" {
t.Fatalf("target identity = %q", wiring.TargetIdentity)
}
if wiring.OperationID != claim.OperationID || wiring.WindowJobID != claim.JobID ||
wiring.AttemptOwner != claim.Owner || wiring.AttemptEpoch != claim.AttemptEpoch {
t.Fatalf("attempt identity not propagated: %+v", wiring)
}
if wiring.RecoveryJobID != claim.RecoveryJobID || wiring.RecoveryOperationID != claim.RecoveryOperationID {
t.Fatalf("recovery delivery identity not propagated: %+v", wiring)
}
// 自动回滚(非恢复 worker)没有 recovery 投递身份,必须保持为空。
rollback, err := RestoreDispatchWiringFor(config, "internal", objectgateway.Claim{
JobID: "job-window", OperationID: "op-original", Owner: "worker-1#queue#job-window", AttemptEpoch: 3})
if err != nil {
t.Fatal(err)
}
if rollback.RecoveryJobID != "" || rollback.RecoveryOperationID != "" {
t.Fatalf("rollback wiring invented a recovery identity: %+v", rollback)
}
}
func TestRestoreDispatchWiringFailsClosed(t *testing.T) {
config := appconfig.Config{
LibraryTargetInstance: "inst-prod",
LibraryProductionDatabaseURL: "postgres://user:pass@127.0.0.1:5432/kg_prod?sslmode=disable",
LibraryProductionInternalKB: "kb-internal",
}
if _, err := RestoreDispatchWiringFor(config, "internal", objectgateway.Claim{}); err == nil {
t.Fatal("missing attempt identity was accepted")
}
if _, err := RestoreDispatchWiringFor(config, "tenant-wide", objectgateway.Claim{
JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1}); err == nil {
t.Fatal("unknown scope was accepted")
}
if _, err := RestoreDispatchWiringFor(config, "external", objectgateway.Claim{
JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1}); err == nil {
t.Fatal("external scope without its knowledge base was accepted")
}
if _, err := RestoreDispatchWiringFor(config, "internal", objectgateway.Claim{
JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1, RecoveryJobID: "q"}); err == nil {
t.Fatal("partial recovery delivery identity was accepted")
}
unconfigured := config
unconfigured.LibraryProductionDatabaseURL = ""
if _, err := RestoreDispatchWiringFor(unconfigured, "internal", objectgateway.Claim{
JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1}); err == nil {
t.Fatal("unconfigured target database was accepted as a dispatch target")
}
}
func TestObjectAddressingMatchesProvider(t *testing.T) {
for _, provider := range []string{"oss", "s3", "minio"} {
t.Run(provider, func(t *testing.T) {
@@ -19,31 +19,62 @@ import (
//
// 关键点不是"多了一张表",而是发送许可的产生方式:
// - Prepare 只建立 prepared(不返回许可);
// - Acquire 在**独立短事务**里按 job→operation→active attempt→effect 的顺序取锁,
// 校验当前 epoch/owner/lease 后做 prepared→dispatching 的 CAS,只有 COMMIT 明确
// 成功才返回一次性许可;
// - Acquire 在**独立短事务**里按 job→operation→activation→active attempt→effect
// 的顺序取锁,校验当前 epoch/owner/lease(数据库时钟)、原作业/operation 锚点、
// 维护窗口、恢复投递身份与不可变绑定后做 prepared→dispatching 的 CAS,只有
// COMMIT 明确成功才返回一次性许可;
// - Inspect 纯读,永不返回许可;
// - 表本身拒绝状态回退、绑定变化与删行(迁移里的触发器)。
// - 表本身拒绝状态回退、绑定变化、删行与激活撤销回退(迁移里的触发器)。
//
// 外层 WithLibraryExecution 的事务只 rollback、从不 commit,所以这里绝不能用它:
// 在那种事务里写 dispatching,发出请求后账会被回滚成"未派发"。
var ErrRestoreDispatchNotActivated = libraryrelease.ErrRestoreDispatchUnavailable
// ActivateRestoreDispatch 登记"该目标已完成旧 writer 退出/排空"的部署证据。
// ActivateRestoreDispatch 消费一份**可核验的**激活证据文件:文件摘要由调用方给出并
// 重算、结构完整、每个受控 writer 都登记了退出与排空、证据在有效期内、目标身份/
// 指纹与治理库身份都与现场一致。任何一项不满足都不建立 active 行。
//
// 它记录的是运维声明,不验证声明为真;代码只能保证没有这条登记就不发放许可。
// 撤销是终局:重新启用必须换一个新的目标身份(或由 DBA 归档旧行后重新登记),
// 不允许用同一条旧证据把已经撤销的目标悄悄打开。
func (s *Store) ActivateRestoreDispatch(ctx context.Context, targetIdentity, activatedBy, drainEvidenceDigest string, now time.Time) error {
if targetIdentity == "" || activatedBy == "" || drainEvidenceDigest == "" {
return errors.New("restore dispatch activation requires target, actor and drain evidence digest")
// 代码核验的是证据文件完整性与身份一致性;"writer 确实已退出"仍是部署流程的受控
// attestation(含退出时间与排空摘要),不是本进程能独立观察到的外部事实。
func (s *Store) ActivateRestoreDispatch(ctx context.Context, evidencePath, evidenceDigest,
targetIdentity, targetFingerprint string, now time.Time) error {
evidence, err := libraryrelease.LoadRestoreDispatchActivationEvidence(evidencePath, evidenceDigest,
now.UTC(), libraryrelease.RestoreDispatchActivationMaxAge)
if err != nil {
return err
}
if evidence.TargetIdentity != targetIdentity {
return fmt.Errorf("restore dispatch activation evidence is for target %q, not %q", evidence.TargetIdentity, targetIdentity)
}
if evidence.TargetFingerprint != targetFingerprint {
return errors.New("restore dispatch activation evidence does not match the live target fingerprint")
}
var database string
var address string
var port int
if err := s.db.QueryRow(ctx,
`SELECT current_database(), COALESCE(host(inet_server_addr()), ''), COALESCE(inet_server_port(), 0)`).
Scan(&database, &address, &port); err != nil {
return fmt.Errorf("read governance database identity: %w", err)
}
if database == "" || database != evidence.GovernanceDatabase {
return fmt.Errorf("restore dispatch activation evidence names governance database %q, live is %q",
evidence.GovernanceDatabase, database)
}
raw, err := evidence.CanonicalBytes()
if err != nil {
return err
}
if _, err := s.db.Exec(ctx, `
INSERT INTO library_index_restore_dispatch_activation (target_identity, protocol_version, activated_by, drain_evidence_digest, activated_at)
VALUES ($1, 1, $2, $3, $4)
INSERT INTO library_index_restore_dispatch_activation
(target_identity, protocol_version, target_fingerprint, target_instance, governance_database,
protocol_build, activated_by, drain_evidence_digest, evidence, attested_at, activated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, $10, $11)
ON CONFLICT (target_identity) DO NOTHING`,
targetIdentity, activatedBy, drainEvidenceDigest, now.UTC()); err != nil {
evidence.TargetIdentity, evidence.ProtocolVersion, evidence.TargetFingerprint, evidence.TargetInstance,
evidence.GovernanceDatabase, evidence.ProtocolBuild, evidence.AttestedBy,
evidenceDigest, string(raw), evidence.AttestedAt.UTC(), now.UTC()); err != nil {
return fmt.Errorf("activate restore dispatch: %w", err)
}
return s.requireRestoreDispatchActivation(ctx, targetIdentity)
@@ -76,6 +107,26 @@ WHERE target_identity=$1 AND protocol_version=1 AND revoked_at IS NULL`, targetI
return nil
}
// lockRestoreDispatchActivation 在**同一个 CAS 事务**里锁激活行。撤销是同一行的
// UPDATE:撤销先完成 ⇒ 这里看到 revoked_at;CAS 先取锁 ⇒ 撤销必须等 CAS 提交,
// 之后按"已派发"的窄边界处理。两者只能有一个线性化顺序。
func lockRestoreDispatchActivation(ctx context.Context, tx pgx.Tx, targetIdentity string) error {
var revoked *time.Time
err := tx.QueryRow(ctx, `
SELECT revoked_at FROM library_index_restore_dispatch_activation
WHERE target_identity=$1 AND protocol_version=1 FOR UPDATE`, targetIdentity).Scan(&revoked)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("%w(目标 %s 没有激活登记)", ErrRestoreDispatchNotActivated, targetIdentity)
}
if err != nil {
return fmt.Errorf("lock restore dispatch activation: %w", err)
}
if revoked != nil {
return fmt.Errorf("%w(目标 %s 的激活已于 %s 撤销)", ErrRestoreDispatchNotActivated, targetIdentity, revoked.UTC().Format(time.RFC3339))
}
return nil
}
func restoreDispatchBindingDigest(b libraryrelease.RestoreMarkerBinding) (string, error) {
return b.Digest()
}
@@ -117,6 +168,49 @@ WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_diges
}
}
// UnresolvedRestoreDispatches 返回原 window 的权威台账全集(跨 plan 摘要)。
// 前置与收尾都查同一份记录:只遍历当前 plan 的文档列表会漏掉计划外/别名 effect。
func (s *Store) UnresolvedRestoreDispatches(ctx context.Context, targetIdentity, operationID, windowJobID string) ([]libraryrelease.RestoreDispatchRecord, error) {
rows, err := s.db.Query(ctx, `
SELECT plan_digest, index_name, document_id, state FROM library_index_restore_dispatch
WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3
ORDER BY plan_digest, index_name, document_id`, targetIdentity, operationID, windowJobID)
if err != nil {
return nil, fmt.Errorf("read restore dispatch window: %w", err)
}
defer rows.Close()
var records []libraryrelease.RestoreDispatchRecord
for rows.Next() {
var record libraryrelease.RestoreDispatchRecord
if err := rows.Scan(&record.PlanDigest, &record.Index, &record.DocumentID, &record.State); err != nil {
return nil, err
}
records = append(records, record)
}
return records, rows.Err()
}
// rejectRestoreDispatchPlanAlias 拒绝同一逻辑 effect 用另一个 plan 摘要另起一行。
// key 里虽然有 plan_digest,但唯一约束允许"同 target/operation/window/index/doc/phase、
// 不同 plan"共存;那正是"换计划别名重开证明"的旁路。
func rejectRestoreDispatchPlanAlias(ctx context.Context, tx pgx.Tx, b libraryrelease.RestoreMarkerBinding) error {
var other string
err := tx.QueryRow(ctx, `
SELECT plan_digest FROM library_index_restore_dispatch
WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3
AND index_name=$4 AND document_id=$5 AND phase=$6 AND plan_digest<>$7
LIMIT 1 FOR UPDATE`,
b.Key.TargetIdentity, b.Key.OperationID, b.Key.WindowJobID,
b.Key.Index, b.Key.DocumentID, b.Key.Phase, b.Key.PlanDigest).Scan(&other)
if errors.Is(err, pgx.ErrNoRows) {
return nil
}
if err != nil {
return fmt.Errorf("read restore dispatch plan alias: %w", err)
}
return fmt.Errorf("%w: effect already belongs to another plan digest", libraryrelease.ErrRestoreDispatchHeld)
}
// PrepareRestoreMarker 只建立 prepared:没有发送权,也不把已有 dispatching 抬回 prepared。
func (s *Store) PrepareRestoreMarker(ctx context.Context, b libraryrelease.RestoreMarkerBinding) (libraryrelease.RestoreDispatchStatus, error) {
digest, err := restoreDispatchBindingDigest(b)
@@ -136,6 +230,12 @@ func (s *Store) PrepareRestoreMarker(ctx context.Context, b libraryrelease.Resto
if err := lockRestoreDispatchIdentity(ctx, tx, b); err != nil {
return libraryrelease.RestoreDispatchStatus{}, err
}
if err := lockRestoreDispatchActivation(ctx, tx, b.Key.TargetIdentity); err != nil {
return libraryrelease.RestoreDispatchStatus{}, err
}
if err := rejectRestoreDispatchPlanAlias(ctx, tx, b); err != nil {
return libraryrelease.RestoreDispatchStatus{}, err
}
if _, err := tx.Exec(ctx, `
INSERT INTO library_index_restore_dispatch
(target_identity, operation_id, window_job_id, plan_digest, index_name, document_id, phase, binding_digest, state)
@@ -168,9 +268,6 @@ func (s *Store) acquireRestoreMarker(ctx context.Context, b libraryrelease.Resto
if err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, err
}
if err := s.requireRestoreDispatchActivation(ctx, b.Key.TargetIdentity); err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, err
}
tx, err := s.db.Begin(ctx)
if err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("dispatch restore marker: %w", err)
@@ -179,14 +276,19 @@ func (s *Store) acquireRestoreMarker(ctx context.Context, b libraryrelease.Resto
if err := lockRestoreDispatchIdentity(ctx, tx, b); err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, err
}
// 当前代次必须在租约内,且就是许可申请者:旧 owner 晚到时租约/epoch 已变,零发送。
// 撤销与 CAS 在同一事务线性化:这里拿激活行锁,撤销的 UPDATE 必须排队。
if err := lockRestoreDispatchActivation(ctx, tx, b.Key.TargetIdentity); err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, err
}
// 当前代次必须属于这个原 window 作业,且租户/epoch/租约都仍有效。lease 用
// clock_timestamp():等锁可能跨越租约到期,事务起始的 now() 不是判据。
var epoch int64
var owner, attemptState string
var owner, attemptState, attemptJob string
var leaseUntil *time.Time
err = tx.QueryRow(ctx, `
SELECT epoch, owner, state, lease_until FROM library_attempts
SELECT epoch, owner, state, job_id, lease_until FROM library_attempts
WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.OperationID).Scan(
&epoch, &owner, &attemptState, &leaseUntil)
&epoch, &owner, &attemptState, &attemptJob, &leaseUntil)
if errors.Is(err, pgx.ErrNoRows) {
return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("%w: no active attempt for operation %s",
governance.ErrLibraryEpochStale, b.Key.OperationID)
@@ -194,8 +296,13 @@ WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.Oper
if err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("read active attempt: %w", err)
}
if attemptJob != b.Key.WindowJobID {
return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("%w: active attempt belongs to job %s, not the original window %s",
governance.ErrLibraryEpochStale, attemptJob, b.Key.WindowJobID)
}
var leaseLive bool
if err := tx.QueryRow(ctx, `SELECT COALESCE($1::timestamptz > now(), false)`, leaseUntil).Scan(&leaseLive); err != nil {
if err := tx.QueryRow(ctx,
`SELECT COALESCE($1::timestamptz > clock_timestamp(), false)`, leaseUntil).Scan(&leaseLive); err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("read attempt lease: %w", err)
}
if owner != b.Owner || epoch != b.Epoch || !leaseLive {
@@ -203,6 +310,9 @@ WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.Oper
Reason: fmt.Sprintf("attempt %s#%d is not the live authorized owner", b.Owner, b.Epoch)},
fmt.Errorf("%w: attempt %s#%d is not current", governance.ErrLibraryEpochStale, b.Owner, b.Epoch)
}
if err := rejectRestoreDispatchPlanAlias(ctx, tx, b); err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, err
}
var id int64
var state, stored string
@@ -235,7 +345,7 @@ WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_diges
tag, err := tx.Exec(ctx, `
UPDATE library_index_restore_dispatch
SET state='dispatching', dispatch_owner=$2, dispatch_epoch=$3, dispatch_nonce=$4,
request_digest=$5, dispatch_committed_at=now(), updated_at=now()
request_digest=$5, dispatch_committed_at=clock_timestamp(), updated_at=clock_timestamp()
WHERE id=$1 AND state='prepared'`, id, b.Owner, b.Epoch, nonce, b.RequestDigest)
if err != nil {
return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("claim restore dispatch: %w", err)
@@ -273,9 +383,6 @@ func (s *Store) ConfirmRestoreMarker(ctx context.Context, b libraryrelease.Resto
if err != nil {
return err
}
if err := s.requireRestoreDispatchActivation(ctx, b.Key.TargetIdentity); err != nil {
return err
}
tx, err := s.db.Begin(ctx)
if err != nil {
return fmt.Errorf("confirm restore marker: %w", err)
@@ -284,25 +391,29 @@ func (s *Store) ConfirmRestoreMarker(ctx context.Context, b libraryrelease.Resto
if err := lockRestoreDispatchIdentity(ctx, tx, b); err != nil {
return err
}
if err := lockRestoreDispatchActivation(ctx, tx, b.Key.TargetIdentity); err != nil {
return err
}
var epoch int64
var owner string
var owner, attemptJob string
var leaseUntil *time.Time
err = tx.QueryRow(ctx, `
SELECT epoch, owner, lease_until FROM library_attempts
WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.OperationID).Scan(&epoch, &owner, &leaseUntil)
SELECT epoch, owner, job_id, lease_until FROM library_attempts
WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.OperationID).Scan(&epoch, &owner, &attemptJob, &leaseUntil)
if err != nil {
return fmt.Errorf("confirm restore marker attempt: %w", err)
}
var leaseLive bool
if err := tx.QueryRow(ctx, `SELECT COALESCE($1::timestamptz > now(), false)`, leaseUntil).Scan(&leaseLive); err != nil {
if err := tx.QueryRow(ctx,
`SELECT COALESCE($1::timestamptz > clock_timestamp(), false)`, leaseUntil).Scan(&leaseLive); err != nil {
return fmt.Errorf("confirm restore marker lease: %w", err)
}
if owner != b.Owner || epoch != b.Epoch || !leaseLive {
if attemptJob != b.Key.WindowJobID || owner != b.Owner || epoch != b.Epoch || !leaseLive {
return fmt.Errorf("%w: confirmation by a stale owner", governance.ErrLibraryEpochStale)
}
tag, err := tx.Exec(ctx, `
UPDATE library_index_restore_dispatch
SET state='marker_confirmed', marker_receipt=$1::jsonb, confirmed_at=now(), updated_at=now()
SET state='marker_confirmed', marker_receipt=$1::jsonb, confirmed_at=clock_timestamp(), updated_at=clock_timestamp()
WHERE target_identity=$2 AND operation_id=$3 AND window_job_id=$4 AND plan_digest=$5
AND index_name=$6 AND document_id=$7 AND phase=$8
AND state='dispatching' AND binding_digest=$9`,
@@ -320,9 +431,18 @@ WHERE target_identity=$2 AND operation_id=$3 AND window_job_id=$4 AND plan_diges
return nil
}
// lockRestoreDispatchIdentity 以 job→operation 锁序把"这份许可是不是仍属于原作业"
// 钉死在原持久锚点上:目标/scope/来源快照、写集摘要(若已登记)、维护窗口与实例
// 指纹、恢复投递的 queue/operation 绑定都必须与不可变绑定一致。
func lockRestoreDispatchIdentity(ctx context.Context, tx pgx.Tx, b libraryrelease.RestoreMarkerBinding) error {
var operationID string
err := tx.QueryRow(ctx, `SELECT operation_id FROM library_jobs WHERE id=$1 FOR UPDATE`, b.Key.WindowJobID).Scan(&operationID)
var operationID, scope, jobSource string
var leaseOwner string
var jobEpoch int64
var jobLease *time.Time
err := tx.QueryRow(ctx, `
SELECT operation_id, scope, target_snapshot_id, lease_owner, attempt_epoch, lease_until
FROM library_jobs WHERE id=$1 FOR UPDATE`, b.Key.WindowJobID).Scan(
&operationID, &scope, &jobSource, &leaseOwner, &jobEpoch, &jobLease)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("window job %s not found", b.Key.WindowJobID)
}
@@ -332,9 +452,93 @@ func lockRestoreDispatchIdentity(ctx context.Context, tx pgx.Tx, b libraryreleas
if operationID != b.Key.OperationID {
return errors.New("window job belongs to another operation")
}
if _, err := tx.Exec(ctx, `SELECT id FROM library_operations WHERE id=$1 FOR UPDATE`, b.Key.OperationID); err != nil {
if scope != b.Scope {
return fmt.Errorf("window job scope %s does not match binding %s", scope, b.Scope)
}
if jobSource == "" || jobSource != b.SourceSnapshot {
return fmt.Errorf("window job source snapshot %q does not match binding %q", jobSource, b.SourceSnapshot)
}
if leaseOwner != b.Owner || jobEpoch != b.Epoch || jobLease == nil {
return fmt.Errorf("%w: window job lease is not held by attempt %s#%d", governance.ErrLibraryEpochStale, b.Owner, b.Epoch)
}
var jobLeaseLive bool
if err := tx.QueryRow(ctx, `SELECT COALESCE($1::timestamptz > clock_timestamp(), false)`, jobLease).Scan(&jobLeaseLive); err != nil {
return fmt.Errorf("read window job lease: %w", err)
}
if !jobLeaseLive {
return fmt.Errorf("%w: window job lease expired while waiting for the dispatch CAS", governance.ErrLibraryEpochStale)
}
var opScope, opKind, opSource, requestDigest, writesetDigest, firstBackup string
err = tx.QueryRow(ctx, `
SELECT scope, kind, target_snapshot_id, request_digest, COALESCE(writeset_digest,''), COALESCE(first_backup_snapshot_id,'')
FROM library_operations WHERE id=$1 FOR UPDATE`, b.Key.OperationID).Scan(
&opScope, &opKind, &opSource, &requestDigest, &writesetDigest, &firstBackup)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("operation %s not found", b.Key.OperationID)
}
if err != nil {
return fmt.Errorf("lock operation: %w", err)
}
if opScope != b.Scope {
return fmt.Errorf("operation scope %s does not match binding %s", opScope, b.Scope)
}
if opSource == "" || opSource != b.SourceSnapshot {
return fmt.Errorf("operation source snapshot %q does not match binding %q", opSource, b.SourceSnapshot)
}
if requestDigest == "" {
return errors.New("operation has no request digest anchor")
}
if writesetDigest != "" && writesetDigest != b.Key.PlanDigest {
return fmt.Errorf("operation plan digest %s does not match binding %s", writesetDigest, b.Key.PlanDigest)
}
if firstBackup != "" && firstBackup != b.BackupSnapshot {
return fmt.Errorf("operation backup snapshot %s does not match binding %s", firstBackup, b.BackupSnapshot)
}
_ = opKind
if b.TargetInstance != "" {
var ownerJob, ownerScope string
var released *time.Time
var state string
err = tx.QueryRow(ctx, `
SELECT owner_job_id, owner_scope, state, released_at
FROM library_instance_maintenance WHERE instance=$1 FOR UPDATE`, b.TargetInstance).Scan(
&ownerJob, &ownerScope, &state, &released)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("target instance %s has no maintenance window", b.TargetInstance)
}
if err != nil {
return fmt.Errorf("lock instance maintenance: %w", err)
}
if released != nil || state == "released" {
return errors.New("target instance maintenance window is already released")
}
if ownerJob != b.Key.WindowJobID || ownerScope != b.Scope {
return fmt.Errorf("maintenance window belongs to %s (scope=%s), not window %s", ownerJob, ownerScope, b.Key.WindowJobID)
}
}
if b.RecoveryJobID != "" {
var queueState string
var lease string
if err := tx.QueryRow(ctx, `SELECT state, COALESCE(lease_owner,'') FROM jobs WHERE id=$1`, b.RecoveryJobID).
Scan(&queueState, &lease); err != nil {
return fmt.Errorf("read recovery queue job: %w", err)
}
if queueState != "running" || lease == "" {
return fmt.Errorf("recovery queue job %s is not a live delivery", b.RecoveryJobID)
}
var recoveryKind, recoveryScope, recoveryBackup string
if err := tx.QueryRow(ctx, `
SELECT kind, scope, target_snapshot_id FROM library_operations WHERE id=$1`, b.RecoveryOperationID).
Scan(&recoveryKind, &recoveryScope, &recoveryBackup); err != nil {
return fmt.Errorf("read recovery operation: %w", err)
}
if recoveryKind != string(governance.LibraryJobRecover) || recoveryScope != b.Scope || recoveryBackup != b.BackupSnapshot {
return fmt.Errorf("recovery operation %s is not bound to this window and backup", b.RecoveryOperationID)
}
}
return nil
}
@@ -0,0 +1,255 @@
package postgres
import (
"context"
"encoding/binary"
"errors"
"io"
"net"
"net/url"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/libraryrelease"
)
// G1:真实 PG 的"COMMIT 已到服务端、客户端收不到成功回包"。beforeCommit 钩子做不到
// 这一点(它根本没调用 Commit)。这里用专用 TCP 转发代理:看到客户端 commit 帧后
// 原样转发,然后立刻切断连接——服务端已提交,客户端拿到的是错误。
func startCommitAckDropProxy(t *testing.T, upstreamHostPort string) (string, *sync.WaitGroup) {
t.Helper()
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for {
client, err := listener.Accept()
if err != nil {
return
}
go func() {
defer client.Close()
server, err := net.Dial("tcp", upstreamHostPort)
if err != nil {
return
}
defer server.Close()
var once sync.Once
go func() {
// 服务端 -> 客户端:正常转发;代理被关闭后自然结束。
_, _ = io.Copy(client, server)
}()
buffer := make([]byte, 32*1024)
for {
n, err := client.Read(buffer)
if n > 0 {
chunk := buffer[:n]
drop := onceFrames(chunk, &once)
if _, werr := server.Write(chunk); werr != nil {
return
}
if drop {
// commit 已经交给服务端;不给客户端任何成功回包。
_ = client.Close()
return
}
}
if err != nil {
return
}
}
}()
}
}()
t.Cleanup(func() {
_ = listener.Close()
wg.Wait()
})
return "127.0.0.1:" + strings.Split(listener.Addr().String(), ":")[1], nil
}
// onceFrames 在客户端字节流里找简单查询帧 'Q' 且语句以 commit 开头。找到一次就返回
// true(后续帧不再触发)。
func onceFrames(chunk []byte, once *sync.Once) bool {
for i := 0; i+5 <= len(chunk); i++ {
if chunk[i] != 'Q' {
continue
}
length := int(binary.BigEndian.Uint32(chunk[i+1 : i+5]))
if length < 5 || i+1+length > len(chunk) {
continue
}
sql := strings.TrimSpace(strings.ToLower(string(chunk[i+5 : i+1+length])))
if strings.HasPrefix(sql, "commit") {
triggered := false
once.Do(func() { triggered = true })
return triggered
}
}
return false
}
func proxyStore(t *testing.T, direct *pgxpool.Pool) (*Store, *pgxpool.Pool) {
t.Helper()
conn := direct.Config().ConnConfig
address, _ := startCommitAckDropProxy(t, net.JoinHostPort(conn.Host, strconv.Itoa(int(conn.Port))))
target := url.URL{Scheme: "postgres", User: url.UserPassword(conn.User, conn.Password),
Host: address, Path: "/" + conn.Database}
query := target.Query()
for key, value := range conn.RuntimeParams {
query.Set(key, value)
}
query.Set("sslmode", "disable")
target.RawQuery = query.Encode()
pool, err := pgxpool.New(context.Background(), target.String())
if err != nil {
t.Fatal(err)
}
t.Cleanup(pool.Close)
return NewStore(pool), pool
}
func TestRestoreDispatchCommitAckLossKeepsDispatchingAndNoPermit(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-ds01-ackloss", "sync", now)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-ackloss", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
binding.Key.WindowJobID = "libsync-ds01-ackloss"
binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil {
t.Fatal(err)
}
proxied, _ := proxyStore(t, pool)
permit, status, err := proxied.AcquireRestoreMarker(ctx, binding)
if err == nil {
t.Fatal("commit acknowledgement loss reported success")
}
if permit != nil {
t.Fatal("a send permit was issued although the client never saw COMMIT succeed")
}
if status.State != libraryrelease.RestoreDispatchDispatching || !status.Held {
t.Fatalf("status after ack loss = %+v", status)
}
// 服务端确实已提交:换一条直连读回 durable dispatching。
time.Sleep(300 * time.Millisecond)
var state string
if err := pool.QueryRow(ctx, `SELECT state FROM library_index_restore_dispatch`).Scan(&state); err != nil {
t.Fatal(err)
}
if state != libraryrelease.RestoreDispatchDispatching {
t.Fatalf("server-side state after committed-but-unacknowledged commit = %s", state)
}
// 下一次调用(直连)不能把 dispatching 当 prepared 再发一次。
permit, _, err = store.AcquireRestoreMarker(ctx, binding)
if permit != nil || !errors.Is(err, libraryrelease.ErrRestoreDispatchHeld) {
t.Fatalf("next call after ack loss = permit=%v err=%v", permit, err)
}
}
func TestRestoreDispatchUndecidedTransactionBlocksThenFollowsRealOutcome(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-ds01-undecided", "sync", now)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-undecided", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
binding.Key.WindowJobID = "libsync-ds01-undecided"
binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil {
t.Fatal(err)
}
// 原事务仍未决:另一条连接已经把行推进到 dispatching,但没有 commit。
open, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
if _, err := open.Exec(ctx, `
UPDATE library_index_restore_dispatch
SET state='dispatching', dispatch_owner='old', dispatch_epoch=1, dispatch_nonce='old-nonce',
request_digest='old-request', dispatch_committed_at=clock_timestamp()
WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_digest=$4
AND index_name=$5 AND document_id=$6 AND phase=$7`,
binding.Key.TargetIdentity, binding.Key.OperationID, binding.Key.WindowJobID, binding.Key.PlanDigest,
binding.Key.Index, binding.Key.DocumentID, binding.Key.Phase); err != nil {
t.Fatal(err)
}
type result struct {
permit *libraryrelease.RestoreSendPermit
err error
}
done := make(chan result, 1)
go func() {
permit, _, err := store.AcquireRestoreMarker(ctx, binding)
done <- result{permit, err}
}()
select {
case got := <-done:
t.Fatalf("acquire returned while the original transaction was undecided: permit=%v err=%v", got.permit, got.err)
case <-time.After(700 * time.Millisecond):
}
if err := open.Rollback(ctx); err != nil {
t.Fatal(err)
}
got := <-done
if got.err != nil || got.permit == nil {
t.Fatalf("after a real rollback the call should win once: permit=%v err=%v", got.permit, got.err)
}
// 第二个 effect:原事务选择 COMMIT,后续调用必须看到 dispatching 而不是 prepared。
second := binding
second.Key.DocumentID = "doc-2"
if _, err := store.PrepareRestoreMarker(ctx, second); err != nil {
t.Fatal(err)
}
open2, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
if _, err := open2.Exec(ctx, `
UPDATE library_index_restore_dispatch
SET state='dispatching', dispatch_owner='old', dispatch_epoch=1, dispatch_nonce='old-nonce-2',
request_digest='old-request-2', dispatch_committed_at=clock_timestamp()
WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_digest=$4
AND index_name=$5 AND document_id=$6 AND phase=$7`,
second.Key.TargetIdentity, second.Key.OperationID, second.Key.WindowJobID, second.Key.PlanDigest,
second.Key.Index, second.Key.DocumentID, second.Key.Phase); err != nil {
t.Fatal(err)
}
done2 := make(chan result, 1)
go func() {
permit, _, err := store.AcquireRestoreMarker(ctx, second)
done2 <- result{permit, err}
}()
time.Sleep(500 * time.Millisecond)
if err := open2.Commit(ctx); err != nil {
t.Fatal(err)
}
got2 := <-done2
if got2.permit != nil || !errors.Is(got2.err, libraryrelease.ErrRestoreDispatchHeld) {
t.Fatalf("after a real commit the call must not treat dispatching as prepared: permit=%v err=%v", got2.permit, got2.err)
}
}
@@ -0,0 +1,401 @@
package postgres
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"os"
"strings"
"testing"
"time"
"knowledge-governance/internal/governance"
"knowledge-governance/internal/libraryrelease"
)
// DS01-F2:授权必须在**同一个 CAS 事务**里与撤销线性化,lease 必须用数据库实时
// 时钟。以下都用真 PG 的锁等待,不用 scripted fake。
// 撤销先完成:Acquire 已经在事务里排队等 job 锁,撤销此时提交。CAS 不得再发许可。
func TestRestoreDispatchRevocationWinsBeforeCASIsRejected(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-ds01-revoke-race", "sync", now)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-revoke-race", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
binding.Key.WindowJobID = "libsync-ds01-revoke-race"
binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil {
t.Fatal(err)
}
lock, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
if _, err := lock.Exec(ctx, `SELECT id FROM library_jobs WHERE id=$1 FOR UPDATE`, binding.Key.WindowJobID); err != nil {
t.Fatal(err)
}
type result struct {
permit *libraryrelease.RestoreSendPermit
err error
}
done := make(chan result, 1)
go func() {
permit, _, err := store.AcquireRestoreMarker(ctx, binding)
done <- result{permit, err}
}()
time.Sleep(400 * time.Millisecond)
if err := store.RevokeRestoreDispatch(ctx, "target-ds01", now); err != nil {
t.Fatalf("revoke: %v", err)
}
if err := lock.Rollback(ctx); err != nil {
t.Fatal(err)
}
got := <-done
if got.permit != nil {
t.Fatal("revocation committed before the CAS but a send permit was still issued")
}
if !errors.Is(got.err, libraryrelease.ErrRestoreDispatchUnavailable) {
t.Fatalf("revocation race error = %v", got.err)
}
var state string
if err := pool.QueryRow(ctx, `SELECT state FROM library_index_restore_dispatch`).Scan(&state); err != nil {
t.Fatal(err)
}
if state != libraryrelease.RestoreDispatchPrepared {
t.Fatalf("row advanced to %s after a winning revocation", state)
}
}
// CAS 先线性化:Acquire 已经锁住激活行、正在等 effect 行;撤销必须排队。CAS 提交后
// 许可属于一次已派发,第二次调用必须 held。
func TestRestoreDispatchCASWinsBeforeRevocationIssuesOnePermit(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-ds01-cas-race", "sync", now)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-cas-race", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
binding.Key.WindowJobID = "libsync-ds01-cas-race"
binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil {
t.Fatal(err)
}
lock, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
if _, err := lock.Exec(ctx, `SELECT id FROM library_index_restore_dispatch WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_digest=$4 AND index_name=$5 AND document_id=$6 AND phase=$7 FOR UPDATE`,
binding.Key.TargetIdentity, binding.Key.OperationID, binding.Key.WindowJobID, binding.Key.PlanDigest,
binding.Key.Index, binding.Key.DocumentID, binding.Key.Phase); err != nil {
t.Fatal(err)
}
type result struct {
permit *libraryrelease.RestoreSendPermit
err error
}
acquired := make(chan result, 1)
go func() {
permit, _, err := store.AcquireRestoreMarker(ctx, binding)
acquired <- result{permit, err}
}()
time.Sleep(400 * time.Millisecond)
revoked := make(chan error, 1)
go func() { revoked <- store.RevokeRestoreDispatch(ctx, "target-ds01", now) }()
time.Sleep(400 * time.Millisecond)
if err := lock.Commit(ctx); err != nil {
t.Fatal(err)
}
got := <-acquired
if got.err != nil || got.permit == nil {
t.Fatalf("CAS should have won: permit=%v err=%v", got.permit, got.err)
}
if err := <-revoked; err != nil {
t.Fatalf("revocation after CAS: %v", err)
}
permit, _, err := store.AcquireRestoreMarker(ctx, binding)
if permit != nil || err == nil {
t.Fatalf("second acquire after dispatching = permit=%v err=%v", permit, err)
}
var state string
if err := pool.QueryRow(ctx, `SELECT state FROM library_index_restore_dispatch`).Scan(&state); err != nil {
t.Fatal(err)
}
if state != libraryrelease.RestoreDispatchDispatching {
t.Fatalf("state = %s", state)
}
}
// 等锁跨过租约:事务开始时刻仍有效、真正做 CAS 时已过期。必须用数据库实时时钟拒绝。
func TestRestoreDispatchLeaseExpiryWhileWaitingForLockIsRejected(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-ds01-lease", "sync", now)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-lease", "worker-a#q1#0", 500*time.Millisecond, now); err != nil {
t.Fatal(err)
}
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
binding.Key.WindowJobID = "libsync-ds01-lease"
binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil {
t.Fatal(err)
}
lock, err := pool.Begin(ctx)
if err != nil {
t.Fatal(err)
}
if _, err := lock.Exec(ctx, `SELECT id FROM library_jobs WHERE id=$1 FOR UPDATE`, binding.Key.WindowJobID); err != nil {
t.Fatal(err)
}
type result struct {
permit *libraryrelease.RestoreSendPermit
err error
}
done := make(chan result, 1)
go func() {
permit, _, err := store.AcquireRestoreMarker(ctx, binding)
done <- result{permit, err}
}()
time.Sleep(900 * time.Millisecond)
if err := lock.Rollback(ctx); err != nil {
t.Fatal(err)
}
got := <-done
if got.permit != nil {
t.Fatal("permit issued although the attempt lease expired while waiting for the job lock")
}
if got.err == nil || (!strings.Contains(got.err.Error(), "not current") &&
!strings.Contains(got.err.Error(), "lease expired")) {
t.Fatalf("expired lease error = %v", got.err)
}
}
// 错目标/错来源/换 plan 别名/错 UUID 都必须在 CAS 前失败,且台账不动。
func TestRestoreDispatchRejectsWrongAnchorsAndPlanAlias(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-ds01-anchors", "sync", now)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-anchors", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
binding.Key.WindowJobID = "libsync-ds01-anchors"
binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil {
t.Fatal(err)
}
wrongSource := binding
wrongSource.SourceSnapshot = "snap-other"
if permit, _, err := store.AcquireRestoreMarker(ctx, wrongSource); permit != nil || err == nil {
t.Fatalf("wrong source acquired: permit=%v err=%v", permit, err)
}
wrongUUID := binding
wrongUUID.IndexUUID = "uuid-replaced"
if permit, _, err := store.AcquireRestoreMarker(ctx, wrongUUID); permit != nil ||
!errors.Is(err, libraryrelease.ErrRestoreDispatchHeld) {
t.Fatalf("replaced index uuid acquired: permit=%v err=%v", permit, err)
}
wrongFingerprint := binding
wrongFingerprint.TargetFingerprint = "fingerprint-other"
if permit, _, err := store.AcquireRestoreMarker(ctx, wrongFingerprint); permit != nil || err == nil {
t.Fatalf("replaced target fingerprint acquired: permit=%v err=%v", permit, err)
}
alias := binding
alias.Key.PlanDigest = "plan-alias"
if _, err := store.PrepareRestoreMarker(ctx, alias); err == nil ||
!strings.Contains(err.Error(), "another plan") {
t.Fatalf("plan alias was accepted: %v", err)
}
var count int
if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_index_restore_dispatch`).Scan(&count); err != nil {
t.Fatal(err)
}
if count != 1 {
t.Fatalf("dispatch rows = %d, want exactly the original plan row", count)
}
// 恢复投递身份缺失/错误:Prepare 就必须失败,不能等到发送前。
recovery := binding
recovery.RecoveryJobID = "queue-missing"
recovery.RecoveryOperationID = "op-missing"
if _, err := store.PrepareRestoreMarker(ctx, recovery); err == nil {
t.Fatal("unknown recovery delivery identity was accepted")
}
// 真实的 recovery queue/operation 绑定可以被验证通过;换成另一份 backup 的
// recovery operation 则必须拒绝。
if _, err := pool.Exec(ctx, `INSERT INTO jobs (id, type, payload, state, lease_owner, idempotency_key)
VALUES ('queue-recovery-1','library.recover','{}'::jsonb,'running','worker-a','queue-recovery-1')`); err != nil {
t.Fatal(err)
}
if _, _, err := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{
ID: "op-recovery-1", Scope: "internal", Kind: governance.LibraryJobRecover,
TargetSnapshotID: "snap-backup", RequestDigest: "digest-recovery", OperationKey: "recovery-1",
Actor: "tester", CreatedAt: now.UTC(),
}); err != nil {
t.Fatal(err)
}
validRecovery := binding
validRecovery.Key.DocumentID = "doc-recovery"
validRecovery.RecoveryJobID = "queue-recovery-1"
validRecovery.RecoveryOperationID = "op-recovery-1"
if _, err := store.PrepareRestoreMarker(ctx, validRecovery); err != nil {
t.Fatalf("valid recovery delivery binding: %v", err)
}
wrongRecovery := validRecovery
wrongRecovery.Key.DocumentID = "doc-recovery-2"
wrongRecovery.RecoveryOperationID = seededOperationID(t, pool, binding.Key.WindowJobID)
if _, err := store.PrepareRestoreMarker(ctx, wrongRecovery); err == nil {
t.Fatal("recovery operation for another intent was accepted")
}
}
// DS01-F3:撤销是终局——SQL 不能把 revoked_at 改回 NULL,也不能删掉整行;down 在
// 只有 activation 证据时同样拒绝。
func TestRestoreDispatchActivationCannotBeResetOrDeleted(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
if err := store.RevokeRestoreDispatch(ctx, "target-ds01", now); err != nil {
t.Fatal(err)
}
if _, err := pool.Exec(ctx, `UPDATE library_index_restore_dispatch_activation SET revoked_at=NULL WHERE target_identity=$1`, "target-ds01"); err == nil {
t.Fatal("revoked_at was reset to NULL")
}
if _, err := pool.Exec(ctx, `DELETE FROM library_index_restore_dispatch_activation WHERE target_identity=$1`, "target-ds01"); err == nil {
t.Fatal("activation row was deleted")
}
down, err := os.ReadFile("migrations/000078_library_restore_dispatch.down.sql")
if err != nil {
t.Fatal(err)
}
if _, err := pool.Exec(ctx, string(down)); err == nil ||
!strings.Contains(err.Error(), "activation evidence") {
t.Fatalf("down with activation-only evidence = %v", err)
}
var revoked *time.Time
if err := pool.QueryRow(ctx, `SELECT revoked_at FROM library_index_restore_dispatch_activation WHERE target_identity=$1`, "target-ds01").Scan(&revoked); err != nil {
t.Fatal(err)
}
if revoked == nil {
t.Fatal("revocation evidence did not survive")
}
}
// 激活只接受结构与身份都可核验的证据文件。
func TestRestoreDispatchActivationRequiresVerifiableEvidence(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
var database string
if err := pool.QueryRow(ctx, `SELECT current_database()`).Scan(&database); err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
exited := now.Add(-time.Minute)
valid := libraryrelease.RestoreDispatchActivationEvidence{
ProtocolVersion: 1, TargetIdentity: "target-ds01", TargetFingerprint: "fingerprint-1",
TargetInstance: "inst-1", GovernanceDatabase: database, ProtocolBuild: "build-1",
AttestedBy: "deploy-bot", AttestedAt: now,
Writers: []libraryrelease.RestoreDispatchWriterDrain{{
Name: "worker-1", Kind: "worker", Stopped: true, ExitedAt: &exited,
QueueDrained: true, DrainEvidence: "sha256:drain",
}},
}
write := func(t *testing.T, mutate func(*libraryrelease.RestoreDispatchActivationEvidence), digest string) (string, string) {
t.Helper()
raw, err := libraryrelease.RestoreDispatchActivationEvidence(valid).CanonicalBytes()
if err != nil {
t.Fatal(err)
}
if mutate != nil {
var copy libraryrelease.RestoreDispatchActivationEvidence
if err := json.Unmarshal(raw, &copy); err != nil {
t.Fatal(err)
}
mutate(&copy)
raw, err = copy.CanonicalBytes()
if err != nil {
t.Fatal(err)
}
}
path := t.TempDir() + "/evidence.json"
if err := os.WriteFile(path, raw, 0o600); err != nil {
t.Fatal(err)
}
if digest == "" {
sum := sha256.Sum256(raw)
digest = hex.EncodeToString(sum[:])
}
return path, digest
}
// 错摘要
path, _ := write(t, nil, "deadbeef")
if err := store.ActivateRestoreDispatch(ctx, path, "deadbeef", "target-ds01", "fingerprint-1", now); err == nil {
t.Fatal("wrong evidence digest was accepted")
}
// 错目标指纹
path, digest := write(t, nil, "")
if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-other", now); err == nil {
t.Fatal("evidence for another target fingerprint was accepted")
}
// 未排空
path, digest = write(t, func(e *libraryrelease.RestoreDispatchActivationEvidence) {
e.Writers[0].QueueDrained = false
}, "")
if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err == nil {
t.Fatal("evidence with an undrained writer was accepted")
}
// 治理库身份不符
path, digest = write(t, func(e *libraryrelease.RestoreDispatchActivationEvidence) {
e.GovernanceDatabase = "some-other-database"
}, "")
if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err == nil {
t.Fatal("evidence for another governance database was accepted")
}
// 过期
path, digest = write(t, func(e *libraryrelease.RestoreDispatchActivationEvidence) {
old := now.Add(-24 * time.Hour)
e.AttestedAt = old
e.Writers[0].ExitedAt = &old
}, "")
if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err == nil {
t.Fatal("stale evidence was accepted")
}
var count int
if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_index_restore_dispatch_activation`).Scan(&count); err != nil {
t.Fatal(err)
}
if count != 0 {
t.Fatalf("invalid evidence created %d activation rows", count)
}
// 有效证据
path, digest = write(t, nil, "")
if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err != nil {
t.Fatalf("valid evidence rejected: %v", err)
}
}
@@ -2,7 +2,12 @@ package postgres
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"sync"
"testing"
@@ -33,7 +38,8 @@ func ds01Binding() libraryrelease.RestoreMarkerBinding {
PlanDigest: "plan-1", Index: "kg-index", DocumentID: "doc-1",
Phase: libraryrelease.RestoreMarkerPhaseCreate,
},
Scope: "internal", SourceSnapshot: "snap-src", BackupSnapshot: "snap-backup",
Scope: "internal", SourceSnapshot: "snap-target", BackupSnapshot: "snap-backup",
TargetInstance: "inst-1", TargetFingerprint: "fingerprint-1", IndexUUID: "uuid-1",
ChangeDigest: "change-1", MarkerDigest: "marker-1", RequestDigest: "request-1",
Owner: "worker-a#q1#0", Epoch: 1,
}
@@ -63,7 +69,7 @@ func TestRestoreDispatchPrepareIsUniqueAndBindingBound(t *testing.T) {
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-prep", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
@@ -101,7 +107,7 @@ func TestRestoreDispatchConcurrentAcquireHasSingleWinner(t *testing.T) {
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-race", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
@@ -156,7 +162,7 @@ func TestRestoreDispatchIsMonotoneAndAppendOnly(t *testing.T) {
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-mono", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
@@ -211,7 +217,7 @@ func TestRestoreDispatchRejectsStaleEpochBeforeSend(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
@@ -258,7 +264,7 @@ func TestRestoreDispatchCommitWithoutExplicitSuccessYieldsNoPermit(t *testing.T)
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-commit", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
binding := ds01Binding()
@@ -310,7 +316,7 @@ func TestRestoreDispatchRevocationIsFinal(t *testing.T) {
if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-revoke", "worker-a#q1#0", 10*time.Minute, now); err != nil {
t.Fatal(err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil {
t.Fatal(err)
}
if err := store.RevokeRestoreDispatch(ctx, "target-ds01", now); err != nil {
@@ -322,7 +328,7 @@ func TestRestoreDispatchRevocationIsFinal(t *testing.T) {
if _, err := store.PrepareRestoreMarker(ctx, binding); !errors.Is(err, libraryrelease.ErrRestoreDispatchUnavailable) {
t.Fatalf("revoked target still prepared: %v", err)
}
if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err == nil {
if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err == nil {
t.Fatal("revoked activation was reopened with the same evidence")
}
var revoked *time.Time
@@ -333,3 +339,38 @@ func TestRestoreDispatchRevocationIsFinal(t *testing.T) {
t.Fatal("revocation was silently cleared")
}
}
// activateRestoreDispatchForTest 写一份结构完整的激活证据并登记到隔离治理库。
// 治理库名取自当前连接:激活 API 会与现场库身份比对。
func activateRestoreDispatchForTest(t *testing.T, store *Store, targetIdentity, fingerprint string) error {
t.Helper()
ctx := context.Background()
var database string
if err := store.db.QueryRow(ctx, `SELECT current_database()`).Scan(&database); err != nil {
t.Fatalf("read governance database: %v", err)
return err
}
now := time.Now().UTC()
exited := now.Add(-time.Minute)
evidence := libraryrelease.RestoreDispatchActivationEvidence{
ProtocolVersion: 1, TargetIdentity: targetIdentity, TargetFingerprint: fingerprint,
TargetInstance: "inst-1", GovernanceDatabase: database, ProtocolBuild: "test-build",
AttestedBy: "ds01-test", AttestedAt: now,
Writers: []libraryrelease.RestoreDispatchWriterDrain{{
Name: "worker", Kind: "worker", Stopped: true, ExitedAt: &exited,
QueueDrained: true, DrainEvidence: "sha256:drain",
}},
}
raw, err := json.Marshal(evidence)
if err != nil {
t.Fatal(err)
return err
}
path := filepath.Join(t.TempDir(), "activation.json")
if err := os.WriteFile(path, raw, 0o600); err != nil {
t.Fatal(err)
return err
}
sum := sha256.Sum256(raw)
return store.ActivateRestoreDispatch(ctx, path, hex.EncodeToString(sum[:]), targetIdentity, fingerprint, now)
}
@@ -1,10 +1,17 @@
-- 已有派发记录时拒绝自动回滚:这些行是"是否发送过"的唯一证明,
-- 用 down 抹掉等于把 unknown 重新伪装成 prepared。人工处置必须先导出并存档。
-- 已有派发记录或激活/撤销证据时拒绝自动回滚:这些行是"是否发送过 / 是否曾被授权"
-- 的唯一证明,用 down 抹掉等于把 unknown 重新伪装成 prepared,或把已撤销的激活
-- 悄悄删掉。人工处置必须先导出并存档。仅有一条已撤销 activation、尚无任何 effect
-- 行时同样拒绝——撤销事实本身也是证据。
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM library_index_restore_dispatch LIMIT 1) THEN
IF to_regclass('library_index_restore_dispatch') IS NOT NULL
AND EXISTS (SELECT 1 FROM library_index_restore_dispatch LIMIT 1) THEN
RAISE EXCEPTION 'refusing to drop restore dispatch evidence with recorded rows';
END IF;
IF to_regclass('library_index_restore_dispatch_activation') IS NOT NULL
AND EXISTS (SELECT 1 FROM library_index_restore_dispatch_activation LIMIT 1) THEN
RAISE EXCEPTION 'refusing to drop restore dispatch activation evidence';
END IF;
END $$;
DROP TRIGGER IF EXISTS library_index_restore_dispatch_monotone ON library_index_restore_dispatch;
@@ -67,26 +67,46 @@ CREATE TRIGGER library_index_restore_dispatch_monotone
BEFORE UPDATE OR DELETE ON library_index_restore_dispatch
FOR EACH ROW EXECUTE FUNCTION library_index_restore_dispatch_guard();
-- 激活登记:配置开关不是独占证明。只有部署流程把"旧 writer 已退出并排空"的证据
-- 登记到这个目标上,协议才对该目标生效;撤销后立即 fail closed。
-- 激活登记:配置开关不是独占证明。只有部署流程把一份可核验的"旧 writer 已退出并
-- 排空"证据文件登记到这个目标上,协议才对该目标生效。
--
-- 除 revoked_at 外全部不可变;撤销是终局(NULL -> 时间戳之后不能再改回 NULL、不能
-- 改时间、不能删除整行)。要重新开放同一物理目标,只能换新的目标身份并重新登记,
-- 不能"把旧行归档/删掉再插一行"。
CREATE TABLE IF NOT EXISTS library_index_restore_dispatch_activation (
target_identity text PRIMARY KEY,
protocol_version integer NOT NULL CHECK (protocol_version = 1),
target_fingerprint text NOT NULL CHECK (target_fingerprint <> ''),
target_instance text NOT NULL CHECK (target_instance <> ''),
governance_database text NOT NULL CHECK (governance_database <> ''),
protocol_build text NOT NULL CHECK (protocol_build <> ''),
activated_by text NOT NULL CHECK (activated_by <> ''),
drain_evidence_digest text NOT NULL CHECK (drain_evidence_digest <> ''),
evidence jsonb NOT NULL,
attested_at timestamptz NOT NULL,
activated_at timestamptz NOT NULL DEFAULT now(),
revoked_at timestamptz
);
CREATE OR REPLACE FUNCTION library_index_restore_dispatch_activation_guard() RETURNS trigger AS $$
BEGIN
IF TG_OP = 'DELETE' THEN
RAISE EXCEPTION 'restore dispatch activation evidence is append-only';
END IF;
IF NEW.target_identity <> OLD.target_identity OR NEW.protocol_version <> OLD.protocol_version OR
NEW.activated_by <> OLD.activated_by OR NEW.drain_evidence_digest <> OLD.drain_evidence_digest OR
NEW.activated_at <> OLD.activated_at THEN
NEW.target_fingerprint <> OLD.target_fingerprint OR NEW.target_instance <> OLD.target_instance OR
NEW.governance_database <> OLD.governance_database OR NEW.protocol_build <> OLD.protocol_build OR
NEW.activated_by <> OLD.activated_by OR
NEW.drain_evidence_digest <> OLD.drain_evidence_digest OR NEW.evidence <> OLD.evidence OR
NEW.attested_at <> OLD.attested_at OR NEW.activated_at <> OLD.activated_at THEN
RAISE EXCEPTION 'restore dispatch activation is immutable except revocation';
END IF;
IF OLD.revoked_at IS NOT NULL AND NEW.revoked_at IS NOT NULL AND NEW.revoked_at <> OLD.revoked_at THEN
RAISE EXCEPTION 'restore dispatch activation revocation is final';
-- 终局:已撤销行既不能改回 NULL,也不能改写成另一个时间。
IF OLD.revoked_at IS NOT NULL THEN
IF NEW.revoked_at IS NULL OR NEW.revoked_at <> OLD.revoked_at THEN
RAISE EXCEPTION 'restore dispatch activation revocation is final';
END IF;
RETURN NEW;
END IF;
RETURN NEW;
END;
@@ -94,5 +114,5 @@ $$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS library_index_restore_dispatch_activation_guard_trg ON library_index_restore_dispatch_activation;
CREATE TRIGGER library_index_restore_dispatch_activation_guard_trg
BEFORE UPDATE ON library_index_restore_dispatch_activation
BEFORE UPDATE OR DELETE ON library_index_restore_dispatch_activation
FOR EACH ROW EXECUTE FUNCTION library_index_restore_dispatch_activation_guard();
@@ -162,6 +162,22 @@ func (p *r38RefusalProxy) setRefuseClean(refuse bool) {
p.flush()
}
// markerCreates 只数**真实转发**到 ES 的恢复标记 create(路径 _create + claim 前缀)。
func (p *r38RefusalProxy) markerCreates() int {
p.mu.Lock()
defer p.mu.Unlock()
count := 0
for _, event := range p.events {
path, _ := event["path"].(string)
request, _ := event["request"].(string)
dispatched, _ := event["dispatched"].(bool)
if dispatched && strings.Contains(path, "/_create/") && strings.Contains(request, indexRestoreClaimPrefix) {
count++
}
}
return count
}
func (p *r38RefusalProxy) armVerifyCorruption() {
p.mu.Lock()
defer p.mu.Unlock()
@@ -404,6 +420,7 @@ func TestFormalMaintenanceRecoveryHeldThenRecovered(t *testing.T) {
objectBranch := scenario == "object-restore"
pgBranch := scenario == "pg-restore"
earlyBranch := scenario == "pg-early"
ds01Branch := scenario == "ds01"
// 三条分支共用同一套 fullshape/真实入库装配;对象分支与两种 PG 分支都让 ES 恢复不成为
// 第一个失败点:ES 分支拒绝 clean 派发,PG 分支在恢复自己的事务里拿到真实 SQLSTATE。
fullShape := objectBranch || pgBranch || earlyBranch
@@ -554,9 +571,10 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
reader := gatewayReadOnlyGovernance(t, ctx, gov, govDSN)
project := os.Getenv("KG_R38_PROJECT")
proxy := newR38RefusalProxy(t, root, fx.searchBase)
if fullShape {
if fullShape || ds01Branch {
// 对象分支要证明的是"对象恢复传输失败",ES 的回滚写入必须放行,
// 否则失败会被误报成旧的 ES clean-dispatch 分支。
// 否则失败会被误报成旧的 ES clean-dispatch 分支。DS01 分支要让恢复在
// 派发 CAS 之前被产品接缝拒绝,ES 传输层不能先拒绝别的写。
proxy.setRefuseClean(false)
}
objectProxy := newR38ObjectRefusalProxy(t, root, "http://"+os.Getenv("KG_R38_GATEWAY_ADDR"))
@@ -599,11 +617,49 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
"KG_TEST_BARRIER_DIR": filepath.Join(root, "barriers"), "KG_TEST_BARRIER_POINTS": "after:wiki",
"KG_TEST_BARRIER_TIMEOUT": "300s",
}
if ds01Branch {
if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
t.Fatal("ds01 branch requires KG_LIBRARY_TEST_ISOLATED=1")
}
env["LIBRARY_RESTORE_DISPATCH_V2"] = "1"
env["KG_DS01_TEST_REFUSE_MARKER"] = fx.index + "/chunk-target"
}
env["PRODUCTION_RELEASE_AUTHORIZER_REF"] = project
env["PRODUCTION_RELEASE_AUTH_TOKEN_FILE"] = filepath.Join(root, "authorization-token")
must(os.WriteFile(env["PRODUCTION_RELEASE_AUTH_TOKEN_FILE"], []byte(objectgateway.Digest([]byte(root+"approval"))), 0600))
cfg, e := appconfig.Load(func(k string) string { return env[k] })
must(e)
var ds01Fingerprint string
if ds01Branch {
// 激活证据必须由**部署流程**登记,且与真实目标身份一致:这里用与 worker 相同的
// 业务库/索引/桶观察目标指纹,并核对治理库名。
probe := fx.target(t)
ds01Fingerprint, _, e = probe.observeRestoreTarget(ctx)
must(e)
var governanceDatabase string
must(gov.QueryRow(ctx, `SELECT current_database()`).Scan(&governanceDatabase))
exited := time.Now().UTC().Add(-time.Minute)
evidence := libraryrelease.RestoreDispatchActivationEvidence{
ProtocolVersion: 1, TargetIdentity: cfg.LibraryTargetIdentity(), TargetFingerprint: ds01Fingerprint,
TargetInstance: project, GovernanceDatabase: governanceDatabase, ProtocolBuild: os.Getenv("KG_R38_VERSION"),
AttestedBy: "r38-runner", AttestedAt: time.Now().UTC(),
Writers: []libraryrelease.RestoreDispatchWriterDrain{{
Name: "writer", Kind: "worker", Stopped: true, ExitedAt: &exited,
QueueDrained: true, DrainEvidence: objectgateway.Digest([]byte(root + "drain")),
}},
}
rawEvidence, merr := json.Marshal(evidence)
must(merr)
evidencePath := filepath.Join(root, "ds01-activation.json")
must(os.WriteFile(evidencePath, rawEvidence, 0600))
sum := sha256.Sum256(rawEvidence)
activationStore := postgresstore.NewStore(gov)
must(activationStore.ActivateRestoreDispatch(ctx, evidencePath, hex.EncodeToString(sum[:]),
cfg.LibraryTargetIdentity(), ds01Fingerprint, time.Now().UTC()))
save("05b-ds01-activation.json", map[string]any{"target": cfg.LibraryTargetIdentity(),
"fingerprint": ds01Fingerprint, "governance_database": governanceDatabase,
"evidence_digest": hex.EncodeToString(sum[:]), "protocol_build": os.Getenv("KG_R38_VERSION")})
}
dataToken := objectgateway.Digest([]byte(root + "data"))
controlToken := objectgateway.Digest([]byte(root + "control"))
env["LIBRARY_OBJECT_GATEWAY_URL"] = objectProxy.url
@@ -883,6 +939,14 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
}
failureText := libraryAfterPublish.Error + "\n" + libraryAfterPublish.Recovery
switch {
case ds01Branch:
if !strings.Contains(failureText, "恢复失败") || !strings.Contains(failureText, "injected DS01 marker refusal before dispatch CAS") {
t.Fatalf("publish failure does not name the DS01 pre-CAS refusal: %q", failureText)
}
if strings.Contains(failureText, "index effect outcome unknown") || strings.Contains(failureText, "SQLSTATE") ||
strings.Contains(failureText, "conditional object cleanup not confirmed") {
t.Fatalf("ds01 branch was mislabelled as another failure: %q", failureText)
}
case objectBranch:
// 失败必须来自对象恢复边界:回滚已通过 ES clean 写入,随后在对象删除派发前被拒。
if !strings.Contains(failureText, "恢复失败") || !strings.Contains(failureText, "conditional object cleanup not confirmed") {
@@ -978,6 +1042,18 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
effects := r38IndexEffects(t, archiveRoot, "chunk-target")
save("12-index-effects-held.json", effects)
switch {
case ds01Branch:
// 派发 CAS 之前被拒:意图已落盘、台账仍是 prepared、没有 claimed/finalize。
for _, phase := range []string{"apply-intent", "applied", "restore-intent", "restore-mark-intent"} {
if _, ok := effects[phase]; !ok {
t.Fatalf("ds01 held boundary missing %s evidence (have %v)", phase, effectPhases(effects))
}
}
for _, phase := range []string{"restore-claimed", "restore-finalize-intent", "restored"} {
if _, ok := effects[phase]; ok {
t.Fatalf("ds01 held boundary must not carry %s evidence", phase)
}
}
case objectBranch, pgBranch:
// 对象分支的 ES clean 写入已成功;PG wiki 分支的 index 阶段先于 wiki 提交。
// 两者都必须有恢复回执,这不是被拒的边界。
@@ -1166,6 +1242,19 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
t.Fatalf("recover with another window => %d %v", code, mismatch)
}
save("17-window-binding-409.json", map[string]any{"http": code, "body": mismatch})
if ds01Branch {
var state, owner, receipt, binding string
must(gov.QueryRow(ctx, `SELECT state, dispatch_owner, COALESCE(marker_receipt::text,''), binding_digest
FROM library_index_restore_dispatch WHERE index_name=$1 AND document_id='chunk-target'`, fx.index).
Scan(&state, &owner, &receipt, &binding))
creates := proxy.markerCreates()
save("17c-ds01-held-ledger.json", map[string]any{"state": state, "owner": owner,
"receipt": receipt, "binding": binding, "marker_creates_forwarded": creates,
"target_fingerprint": ds01Fingerprint})
if state != libraryrelease.RestoreDispatchPrepared || owner != "" || receipt != "" || creates != 0 {
t.Fatalf("DS01 held boundary is not a clean prepared record: state=%s owner=%q creates=%d", state, owner, creates)
}
}
// ---------- 4) 撤故障 → 正常 POST /library/maintenance/recover ----------
proxy.setRefuseClean(false)
@@ -1180,6 +1269,16 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
// 精确解除:只删那一行外部范围的 knowledges 占位行;不换成删除 held 记录或重拍备份。
save("17b-pg-early-fault-disarmed.json", r38DisarmPGEarlyFault(t, fx, conflictKnowledge))
}
if ds01Branch {
// 正式 worker 必须真的换进程:旧执行者退出后由同一冻结二进制的新 worker 消费恢复。
oldPID := worker.Process.Pid
_ = worker.Process.Kill()
must(worker.Wait())
delete(workerEnv, "KG_DS01_TEST_REFUSE_MARKER")
worker = start("worker", "worker-recover", workerEnv)
save("17c2-ds01-worker-restart.json", map[string]any{"old_pid": oldPID, "new_pid": worker.Process.Pid,
"refusal_env_removed": true, "binary": "worker", "sha256": expectedBinaries["worker"]})
}
save("17a-recover-request.json", map[string]any{"window_job_id": job.ID, "scope": "internal",
"backup_snapshot_id": backupID, "original_job": job.ID, "object_fault_removed": objectBranch,
"pg_fault_removed": pgBranch, "pg_early_fault_removed": earlyBranch,
@@ -1201,6 +1300,18 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v
if got := waitJob(recoveryQueueID, func(s string) bool { return s == "completed" }, 300, "recovery must complete"); got != "completed" {
t.Fatalf("recovery state = %s", got)
}
if ds01Branch {
var state, owner, receipt string
must(gov.QueryRow(ctx, `SELECT state, dispatch_owner, COALESCE(marker_receipt::text,'')
FROM library_index_restore_dispatch WHERE index_name=$1 AND document_id='chunk-target'`, fx.index).
Scan(&state, &owner, &receipt))
creates := proxy.markerCreates()
save("18b-ds01-recovered-ledger.json", map[string]any{"state": state, "owner": owner,
"receipt": receipt, "marker_creates_forwarded": creates, "recovery_queue": recoveryQueueID})
if state != libraryrelease.RestoreDispatchMarkerConfirmed || owner == "" || receipt == "" || creates != 1 {
t.Fatalf("DS01 recovery ledger = state=%s owner=%q creates=%d", state, owner, creates)
}
}
recoveredJournal := readR38JSON(t, filepath.Join(root, "maintenance", job.ID+".json"))
save("19-journal-recovered.json", recoveredJournal)
@@ -0,0 +1,283 @@
package v071
import (
"context"
"errors"
"os"
"reflect"
"strings"
"testing"
"knowledge-governance/internal/libraryrelease"
)
// ds01StubGate 只用于分类测试:不产生许可,只返回给定的台账视图。
type ds01StubGate struct {
inspectStatus libraryrelease.RestoreDispatchStatus
inspectFound bool
records []libraryrelease.RestoreDispatchRecord
}
func (g *ds01StubGate) InspectRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding) (libraryrelease.RestoreDispatchStatus, bool, error) {
return g.inspectStatus, g.inspectFound, nil
}
func (g *ds01StubGate) PrepareRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding) (libraryrelease.RestoreDispatchStatus, error) {
return libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchPrepared}, nil
}
func (g *ds01StubGate) AcquireRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding) (*libraryrelease.RestoreSendPermit, libraryrelease.RestoreDispatchStatus, error) {
return nil, g.inspectStatus, libraryrelease.ErrRestoreDispatchHeld
}
func (g *ds01StubGate) ConfirmRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding, []byte) error {
return nil
}
func (g *ds01StubGate) UnresolvedRestoreDispatches(context.Context, string, string, string) ([]libraryrelease.RestoreDispatchRecord, error) {
return g.records, nil
}
func ds01FreezeBindingInputs(t *testing.T, target *ProductionLibraryTarget, backup libraryrelease.BackupRef) {
t.Helper()
target.restoreScope = "internal"
target.restoreSourceSnapshot = backup.WriteSet.TargetSnapshotID
target.restoreBackupSnapshot = backup.SnapshotID
}
// DS01-F1:绑定必须来自原锚点——来源快照 != 写前镜像,且带真实目标实例/指纹/UUID。
func TestDS01BindingUsesFrozenSourceAndRealBackup(t *testing.T) {
fx, snapshot, target, backups, wire := ds01Fixture(t)
authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot)
authority.attach(target)
outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire)
ledger := outcome.Backup.WriteSet
if ledger.TargetSnapshotID == "" || outcome.Backup.SnapshotID == "" {
t.Fatal("fixture lacks a source snapshot or backup image")
}
if ledger.TargetSnapshotID == outcome.Backup.SnapshotID {
t.Fatalf("fixture plan uses one snapshot for both source and backup: %s", ledger.TargetSnapshotID)
}
if err := target.AdoptWriteSet(*ledger); err != nil {
t.Fatal(err)
}
ds01FreezeBindingInputs(t, target, outcome.Backup)
change := indexChangeOf(t, ledger, "chunk-target")
body, err := target.restoreIndexClaimBody(change)
if err != nil {
t.Fatal(err)
}
binding, err := target.restoreMarkerBinding(fx.ctx, change, body)
if err != nil {
t.Fatalf("real binding: %v", err)
}
if binding.SourceSnapshot != ledger.TargetSnapshotID {
t.Fatalf("binding source = %q, plan source = %q", binding.SourceSnapshot, ledger.TargetSnapshotID)
}
if binding.BackupSnapshot != outcome.Backup.SnapshotID {
t.Fatalf("binding backup = %q, real backup = %q", binding.BackupSnapshot, outcome.Backup.SnapshotID)
}
if binding.SourceSnapshot == binding.BackupSnapshot {
t.Fatal("binding still uses the same snapshot for source and backup")
}
if binding.Scope == "" || binding.TargetInstance != ds01TargetInstance ||
binding.TargetFingerprint != authority.targetFingerprint || binding.IndexUUID != authority.indexUUID {
t.Fatalf("binding lacks the real target identity: %+v", binding)
}
status, found, err := authority.store.InspectRestoreMarker(fx.ctx, binding)
if err != nil || !found || status.State != libraryrelease.RestoreDispatchPrepared {
t.Fatalf("real ledger rejected the frozen binding: %+v found=%v err=%v", status, found, err)
}
// 缺任一锚点都必须失败关闭。
target.restoreSourceSnapshot = ""
if _, err := target.restoreMarkerBinding(fx.ctx, change, body); err == nil {
t.Fatal("binding accepted without a source snapshot anchor")
}
}
// DS01-F4:绑定冲突(prepared + Held)必须在任何目标写之前拒绝,不能继续 replay。
func TestDS01BindingConflictHeldBeforeAnyWrite(t *testing.T) {
fx, snapshot, target, backups, wire := ds01Fixture(t)
authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot)
authority.attach(target)
outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire)
if err := target.AdoptWriteSet(*outcome.Backup.WriteSet); err != nil {
t.Fatal(err)
}
ds01FreezeBindingInputs(t, target, outcome.Backup)
target.config.RestoreDispatch = &ds01StubGate{inspectFound: true,
inspectStatus: libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchPrepared, Held: true,
Reason: "binding differs"}}
beforePG := universeDump(t, fx.pool)
beforeIndex := recoveryIndexReadback(t, fx)
beforeObjects := objectVersionSet(t, fx)
beforeDispatch := wire.dispatch
err := target.Restore(fx.ctx, outcome.Backup)
if err == nil || !strings.Contains(err.Error(), "held") {
t.Fatalf("binding conflict admission = %v", err)
}
if wire.dispatch != beforeDispatch {
t.Fatalf("binding conflict forwarded %d marker writes", wire.dispatch-beforeDispatch)
}
if diff := diffUniverse(beforePG, universeDump(t, fx.pool)); diff != "" {
t.Fatalf("binding conflict wrote PG before admission: %s", diff)
}
if after := recoveryIndexReadback(t, fx); !reflect.DeepEqual(beforeIndex, after) {
t.Fatal("binding conflict changed the index")
}
if after := objectVersionSet(t, fx); !reflect.DeepEqual(beforeObjects, after) {
t.Fatal("binding conflict changed objects")
}
}
// DS01-F4:prepared 台账却已在目标上出现精确 marker —— 证据矛盾,必须 held;
// 旧实现会把它当成"已存在的标记"认领并 clean。
func TestDS01PreparedWithMarkerIsContradictionHeld(t *testing.T) {
fx, snapshot, target, backups, wire := ds01Fixture(t)
authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot)
authority.attach(target)
outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire)
if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchPrepared {
t.Fatalf("boundary state = %s", state)
}
ledger := outcome.Backup.WriteSet
if err := target.AdoptWriteSet(*ledger); err != nil {
t.Fatal(err)
}
ds01FreezeBindingInputs(t, target, outcome.Backup)
change := indexChangeOf(t, ledger, "chunk-target")
body, err := target.restoreIndexClaimBody(change)
if err != nil {
t.Fatal(err)
}
// 模拟一个绕过协议的旧 writer:目标上直接出现精确 marker,但台账仍是 prepared。
applied, err := target.loadIndexEffect(change, "applied")
if err != nil || applied == nil || applied.Receipt == nil {
t.Fatalf("applied receipt missing: %v", err)
}
method, path, payload, err := target.indexMutation(change, body, applied.Receipt.SeqNo, applied.Receipt.PrimaryTerm, true)
if err != nil {
t.Fatal(err)
}
var receipt indexReceipt
if err := target.search.requestJSON(fx.ctx, method, path, payload, &receipt); err != nil {
t.Fatalf("foreign marker write: %v", err)
}
beforePG := universeDump(t, fx.pool)
beforeDispatch := wire.dispatch
err = target.Restore(fx.ctx, outcome.Backup)
if err == nil || !strings.Contains(err.Error(), "held") {
t.Fatalf("prepared+marker admission = %v", err)
}
if wire.dispatch != beforeDispatch {
t.Fatalf("prepared+marker forwarded %d marker writes", wire.dispatch-beforeDispatch)
}
if diff := diffUniverse(beforePG, universeDump(t, fx.pool)); diff != "" {
t.Fatalf("prepared+marker wrote PG before admission: %s", diff)
}
requireIndexEffect(t, fx, ledger, change, "restored", false)
if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchPrepared {
t.Fatalf("dispatch state moved to %s", state)
}
}
// DS01-F4:收尾必须用原 window 的权威未决全集;计划外记录与 prepared 记录都拒绝。
func TestDS01CloseoutUsesAuthoritativeWindowSet(t *testing.T) {
fx, snapshot, target, backups, wire := ds01Fixture(t)
authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot)
authority.attach(target)
outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire)
ledger := outcome.Backup.WriteSet
if err := target.AdoptWriteSet(*ledger); err != nil {
t.Fatal(err)
}
ds01FreezeBindingInputs(t, target, outcome.Backup)
change := indexChangeOf(t, ledger, "chunk-target")
// 台账里的 prepared 记录本身就必须拒绝收尾(旧实现只拒绝 dispatching)。
if err := target.verifyRestoreEffectsClosed(fx.ctx); err == nil || !strings.Contains(err.Error(), "未决") {
t.Fatalf("closeout accepted a prepared dispatch record: %v", err)
}
// 计划外原窗口记录:即使当前 plan 的 effect 看起来都已完成,也必须拒绝。
target.config.RestoreDispatch = &ds01StubGate{inspectFound: true,
inspectStatus: libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchMarkerConfirmed},
records: []libraryrelease.RestoreDispatchRecord{{
PlanDigest: ledger.Digest, Index: change.Index, DocumentID: "ghost-document",
State: libraryrelease.RestoreDispatchMarkerConfirmed}}}
if err := target.verifyRestoreEffectsClosed(fx.ctx); err == nil || !strings.Contains(err.Error(), "计划外") {
t.Fatalf("closeout accepted a plan-outside record: %v", err)
}
// 换 plan 别名也必须拒绝。
target.config.RestoreDispatch = &ds01StubGate{inspectFound: true,
inspectStatus: libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchMarkerConfirmed},
records: []libraryrelease.RestoreDispatchRecord{{
PlanDigest: "another-plan", Index: change.Index, DocumentID: change.ID,
State: libraryrelease.RestoreDispatchMarkerConfirmed}}}
if err := target.verifyRestoreEffectsClosed(fx.ctx); err == nil || !strings.Contains(err.Error(), "另一份计划") {
t.Fatalf("closeout accepted a plan alias: %v", err)
}
}
// DS01-F5:prepared 已提交、mark-intent 文件写失败 ⇒ 重启后可补同内容并只派发一次;
// 无台账的旧意图仍按 legacy held(本文件同时守住两个方向)。
func TestDS01PreparedWithoutIntentResumesOnce(t *testing.T) {
fx, snapshot, target, backups, wire := ds01Fixture(t)
authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot)
authority.attach(target)
changeID := "chunk-target"
target.restoreIntentFault = func(c libraryrelease.IndexWrite) error {
if c.ID == changeID {
return errors.New("injected mark-intent file write failure")
}
return nil
}
outcome := libraryrelease.Run(fx.ctx, requestFor(snapshot), contractFailWikiTarget{target}, backups, nil)
if outcome.State != "rollback_failed" || !strings.Contains(outcome.Rollback, "mark-intent") {
t.Fatalf("intent write failure not reached: state=%q rollback=%q", outcome.State, outcome.Rollback)
}
ledger := outcome.Backup.WriteSet
change := indexChangeOf(t, ledger, changeID)
requireIndexEffect(t, fx, ledger, change, "restore-intent", true)
requireIndexEffect(t, fx, ledger, change, "restore-mark-intent", false)
if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchPrepared {
t.Fatalf("after intent failure dispatch row = %s", state)
}
if wire.dispatch != 0 {
t.Fatalf("intent failure forwarded %d marker writes", wire.dispatch)
}
// 新调用(同一 attempt)看到 prepared + 无 mark-intent:补同内容意图后继续。
if err := target.Restore(fx.ctx, outcome.Backup); err != nil {
t.Fatalf("prepared-without-intent resume: %v", err)
}
requireIndexEffect(t, fx, ledger, change, "restore-mark-intent", true)
requireIndexEffect(t, fx, ledger, change, "restored", true)
if wire.dispatch != 1 {
t.Fatalf("resume forwarded %d marker creates, want 1", wire.dispatch)
}
if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchMarkerConfirmed {
t.Fatalf("terminal state = %s", state)
}
}
// 无台账的旧意图仍然 held:补意图的能力只对新协议 prepared 区段开放。
func TestDS01LegacyIntentWithoutLedgerStillHeld(t *testing.T) {
fx, snapshot, target, backups, wire := ds01Fixture(t)
legacy := ds01RunToBoundary(t, fx, snapshot, target, backups, wire)
authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot)
authority.attach(target)
ledger := legacy.Backup.WriteSet
change := indexChangeOf(t, ledger, "chunk-target")
// 删掉 mark-intent,模拟"只剩旧意图、没有新台账"的历史现场。
if err := os.Remove(target.indexEffectPath(change, "restore-mark-intent")); err != nil {
t.Fatal(err)
}
err := target.Restore(fx.ctx, legacy.Backup)
if err == nil || !strings.Contains(err.Error(), "legacy") {
t.Fatalf("legacy intent without ledger = %v", err)
}
if wire.dispatch != 0 {
t.Fatalf("legacy intent forwarded %d marker writes", wire.dispatch)
}
if rows := authority.dispatchRows(t); rows != 0 {
t.Fatalf("legacy intent created %d dispatch rows", rows)
}
}
@@ -2,6 +2,8 @@ package v071
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
@@ -29,17 +31,20 @@ import (
// 这里不制造"prepared 且从未派发"的假正例:prepared 一律由真实 Prepare 产生。
const ds01TargetIdentity = "kg-ds01-test-target"
const ds01TargetInstance = "inst-ds01"
type ds01Authority struct {
store *postgresstore.Store
pool *pgxpool.Pool
dsn string
operationID string
jobID string
owner string
epoch int64
targetIdentity string
snapshot string
store *postgresstore.Store
pool *pgxpool.Pool
dsn string
operationID string
jobID string
owner string
epoch int64
targetIdentity string
snapshot string
targetFingerprint string
indexUUID string
}
func newDS01Authority(t *testing.T, jobID, snapshot string) *ds01Authority {
@@ -78,9 +83,6 @@ func newDS01Authority(t *testing.T, jobID, snapshot string) *ds01Authority {
if err != nil {
t.Fatalf("begin attempt: %v", err)
}
if err := store.ActivateRestoreDispatch(ctx, ds01TargetIdentity, "ds01-test", "sha256:drain-evidence", now); err != nil {
t.Fatalf("activate dispatch: %v", err)
}
return &ds01Authority{store: store, pool: pool, dsn: dsn, operationID: attempt.OperationID,
jobID: attempt.JobID, owner: attempt.Owner, epoch: int64(attempt.Epoch),
targetIdentity: ds01TargetIdentity, snapshot: snapshot}
@@ -95,7 +97,24 @@ func (a *ds01Authority) attach(target *ProductionLibraryTarget) {
target.config.RestoreWindowJobID = a.jobID
target.config.RestoreAttemptOwner = a.owner
target.config.RestoreAttemptEpoch = a.epoch
target.config.RestoreSnapshotID = a.snapshot
target.config.RestoreInstance = ds01TargetInstance
fingerprint, indexUUID, err := target.observeRestoreTarget(context.Background())
if err != nil {
panic(err)
}
if _, err := a.pool.Exec(context.Background(), `
INSERT INTO library_instance_maintenance (instance, owner_job_id, owner_scope, target_fingerprint, state,
original_service_state, note, acquired_at, updated_at)
VALUES ($1,$2,'internal',$3,'acquired','app:running','ds01-test',$4,$4)
ON CONFLICT (instance) DO UPDATE SET owner_job_id=EXCLUDED.owner_job_id, owner_scope=EXCLUDED.owner_scope,
target_fingerprint=EXCLUDED.target_fingerprint, released_at=NULL, state='acquired', updated_at=EXCLUDED.updated_at`,
ds01TargetInstance, a.jobID, fingerprint, time.Now().UTC()); err != nil {
panic(err)
}
a.targetFingerprint, a.indexUUID = fingerprint, indexUUID
if err := activateDS01(context.Background(), a.store, a.pool, ds01TargetIdentity, fingerprint); err != nil {
panic(err)
}
}
func (a *ds01Authority) dispatchRow(t *testing.T) (state, owner string) {
@@ -273,7 +292,7 @@ func TestDS01DispatchingMissingStaysHeldWithZeroWrites(t *testing.T) {
if err != nil {
t.Fatal(err)
}
binding, err := target.restoreMarkerBinding(change, body)
binding, err := target.restoreMarkerBinding(fx.ctx, change, body)
if err != nil {
t.Fatal(err)
}
@@ -329,7 +348,7 @@ func TestDS01CloseoutRejectsUnresolvedDispatch(t *testing.T) {
if err != nil {
t.Fatal(err)
}
binding, err := target.restoreMarkerBinding(change, body)
binding, err := target.restoreMarkerBinding(fx.ctx, change, body)
if err != nil {
t.Fatal(err)
}
@@ -486,7 +505,7 @@ func TestDS01RestoreHelper(t *testing.T) {
target.config.RestoreWindowJobID = os.Getenv("KG_DS01_JOB")
target.config.RestoreAttemptOwner = os.Getenv("KG_DS01_OWNER")
target.config.RestoreAttemptEpoch = epoch
target.config.RestoreSnapshotID = os.Getenv("KG_DS01_SNAPSHOT")
target.config.RestoreInstance = ds01TargetInstance
client := *target.search.client
inner := client.Transport
@@ -521,3 +540,33 @@ func TestDS01RestoreHelper(t *testing.T) {
t.Fatal(err)
}
}
// activateDS01 写一份结构完整的激活证据(治理库名取自隔离库本身)。
func activateDS01(ctx context.Context, store *postgresstore.Store, pool *pgxpool.Pool,
targetIdentity, fingerprint string) error {
var database string
if err := pool.QueryRow(ctx, `SELECT current_database()`).Scan(&database); err != nil {
return err
}
now := time.Now().UTC()
exited := now.Add(-time.Minute)
evidence := libraryrelease.RestoreDispatchActivationEvidence{
ProtocolVersion: 1, TargetIdentity: targetIdentity, TargetFingerprint: fingerprint,
TargetInstance: ds01TargetInstance, GovernanceDatabase: database, ProtocolBuild: "test-build",
AttestedBy: "ds01-test", AttestedAt: now,
Writers: []libraryrelease.RestoreDispatchWriterDrain{{
Name: "worker", Kind: "worker", Stopped: true, ExitedAt: &exited,
QueueDrained: true, DrainEvidence: "sha256:drain",
}},
}
raw, err := json.Marshal(evidence)
if err != nil {
return err
}
path := filepath.Join(os.TempDir(), "ds01-activation-"+targetIdentity+".json")
if err := os.WriteFile(path, raw, 0o600); err != nil {
return err
}
sum := sha256.Sum256(raw)
return store.ActivateRestoreDispatch(ctx, path, hex.EncodeToString(sum[:]), targetIdentity, fingerprint, now)
}
@@ -36,6 +36,13 @@ type indexEffect struct {
// ES 的写计数器是分片生命周期内的统计量(分片迁移/节点替换会清零),
// 不是不可变事务日志,不能作为放行写入的授权依据。
func (t *ProductionLibraryTarget) beginRestoreMarker(ctx context.Context, c libraryrelease.IndexWrite, applied indexReceipt) error {
if t.restoreIntentFault != nil {
fault := t.restoreIntentFault
t.restoreIntentFault = nil
if err := fault(c); err != nil {
return err
}
}
raw, err := json.Marshal(indexEffect{Plan: t.activePlan.Digest, Change: c, Receipt: &applied})
if err != nil {
return err
@@ -84,34 +84,49 @@ func (t *ProductionLibraryTarget) restoreClaimedIndexBefore(ctx context.Context,
}
if markIntent == nil {
if !start {
return errors.New("legacy unknown index restoration lacks attributable marker intent")
}
// 新协议:先在治理库建立 prepared(文件意图之前),再落文件意图。文件失败只留
// prepared(零派发,下一进程可补写同内容意图);反过来"先文件后台账"则会造出
// 无法证明未派发的旧证据。
if t.config.RestoreDispatch != nil && len(c.After) == 0 {
binding, err := t.restoreMarkerBinding(c, body)
// DS01-F5:prepared 已提交、mark-intent 未落盘的窄区段。旧实现一律按
// legacy 拒绝,把设计允许补同内容意图的区段永久堵死。先精确核对:
// 原台账存在、绑定一致、仍是 prepared、且没有任何 claimed/finalize/restored
// 矛盾——全部成立才允许补写同一份意图;否则仍是旧协议未知历史。
resumed, err := t.resumePreparedWithoutIntent(ctx, c, body, applied)
if err != nil {
return err
}
status, err := t.config.RestoreDispatch.PrepareRestoreMarker(ctx, binding)
if err != nil {
if !resumed {
return errors.New("legacy unknown index restoration lacks attributable marker intent")
}
if markIntent, err = t.loadIndexEffect(c, "restore-mark-intent"); err != nil {
return err
}
if status.Held {
return heldRestoreMarker(status, "首次恢复不能建立 prepared")
freshIntent = false
} else {
// 新协议:先在治理库建立 prepared(文件意图之前),再落文件意图。文件失败只留
// prepared(零派发,下一进程可补写同内容意图);反过来"先文件后台账"则会造出
// 无法证明未派发的旧证据。
if t.config.RestoreDispatch != nil && len(c.After) == 0 {
binding, err := t.restoreMarkerBinding(ctx, c, body)
if err != nil {
return err
}
status, err := t.config.RestoreDispatch.PrepareRestoreMarker(ctx, binding)
if err != nil {
return err
}
if status.Held {
return heldRestoreMarker(status, "首次恢复不能建立 prepared")
}
}
if err = t.beginRestoreMarker(ctx, c, applied); err != nil {
return err
}
if markIntent, err = t.loadIndexEffect(c, "restore-mark-intent"); err != nil {
return err
}
}
if err = t.beginRestoreMarker(ctx, c, applied); err != nil {
return err
}
if markIntent, err = t.loadIndexEffect(c, "restore-mark-intent"); err != nil {
return err
}
} else if t.config.RestoreDispatch != nil && len(c.After) == 0 {
// 既有意图必须能在新台账里找到同一 effect:找不到就是旧协议的未知历史,
// 不能借"文件看起来完整"补一份 prepared。
binding, err := t.restoreMarkerBinding(c, body)
binding, err := t.restoreMarkerBinding(ctx, c, body)
if err != nil {
return err
}
@@ -215,6 +230,42 @@ func (t *ProductionLibraryTarget) restoreClaimedIndexBefore(ctx context.Context,
return t.finishRestoreCleanWrite(ctx, c, *claimed.Receipt)
}
// resumePreparedWithoutIntent 处理"prepared 已提交、mark-intent 还没落盘"的窄区段。
//
// 只有同时满足以下各项才允许补写同一份意图并继续:台账存在、绑定一致、状态仍是
// prepared,且没有 claimed/finalize/restored 任何矛盾证据。没有台账的旧意图仍按
// legacy 处理(返回 false),不回填、不推测。
func (t *ProductionLibraryTarget) resumePreparedWithoutIntent(ctx context.Context, c libraryrelease.IndexWrite,
body json.RawMessage, applied indexReceipt) (bool, error) {
if t.config.RestoreDispatch == nil || len(c.After) != 0 {
return false, nil
}
binding, err := t.restoreMarkerBinding(ctx, c, body)
if err != nil {
return false, err
}
status, found, err := t.config.RestoreDispatch.InspectRestoreMarker(ctx, binding)
if err != nil {
return false, err
}
if !found || status.State != libraryrelease.RestoreDispatchPrepared || status.Held {
return false, nil
}
for _, phase := range []string{"restore-claimed", "restore-finalize-intent", "restored"} {
effect, err := t.loadIndexEffect(c, phase)
if err != nil {
return false, err
}
if effect != nil {
return false, fmt.Errorf("%w: prepared 台账却已有 %s 证据", libraryrelease.ErrRestoreDispatchHeld, phase)
}
}
if err := t.beginRestoreMarker(ctx, c, applied); err != nil {
return false, err
}
return true, nil
}
// finishRestoreCleanWrite 派发"标记体 -> 冻结 before-image"的 CAS 收尾,并把回执与
// 实时观察逐位核对。finalize-intent 存在时,只有标记体仍停在它自己的世代上才会
// 走到这里:落地过干净写入必然换掉内容并推进世代。
@@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"fmt"
"os"
"knowledge-governance/internal/libraryrelease"
)
@@ -19,18 +20,32 @@ import (
// 明确不保证:ES 端 fence;已 dispatching 的旧 owner 绝不迟到一次。协议只保证
// 遵守协议的派发者不会得到第二次许可,且 dispatching 永不回退。
func (t *ProductionLibraryTarget) restoreMarkerBinding(c libraryrelease.IndexWrite, body json.RawMessage) (libraryrelease.RestoreMarkerBinding, error) {
// restoreMarkerBinding 用**原持久锚点**构造不可变绑定:
// - scope 来自真实装配;
// - source 是发布基线(WriteSet.TargetSnapshotID),backup 是本次写前镜像;
// - target instance/fingerprint/index UUID 实时观察并与本次恢复的冻结值比较。
func (t *ProductionLibraryTarget) restoreMarkerBinding(ctx context.Context, c libraryrelease.IndexWrite, body json.RawMessage) (libraryrelease.RestoreMarkerBinding, error) {
if t.config.RestoreDispatch == nil {
return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch protocol is not wired")
}
if t.activePlan == nil {
return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch requires the frozen plan")
}
if t.config.RestoreTargetIdentity == "" || t.config.RestoreOperationID == "" ||
t.config.RestoreWindowJobID == "" || t.config.RestoreAttemptOwner == "" ||
t.config.RestoreAttemptEpoch <= 0 {
if t.config.RestoreTargetIdentity == "" || t.config.RestoreInstance == "" ||
t.config.RestoreOperationID == "" || t.config.RestoreWindowJobID == "" ||
t.config.RestoreAttemptOwner == "" || t.config.RestoreAttemptEpoch <= 0 {
return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch lacks the original job/attempt identity")
}
if t.restoreScope == "" || t.restoreSourceSnapshot == "" || t.restoreBackupSnapshot == "" {
return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch lacks the original scope/snapshot anchors")
}
if t.restoreSourceSnapshot == t.restoreBackupSnapshot {
return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch would use one snapshot as both source and backup")
}
fingerprint, indexUUID, err := t.frozenRestoreTarget(ctx)
if err != nil {
return libraryrelease.RestoreMarkerBinding{}, err
}
changeRaw, err := json.Marshal(c)
if err != nil {
return libraryrelease.RestoreMarkerBinding{}, err
@@ -52,14 +67,19 @@ func (t *ProductionLibraryTarget) restoreMarkerBinding(c libraryrelease.IndexWri
DocumentID: c.ID,
Phase: libraryrelease.RestoreMarkerPhaseCreate,
},
Scope: t.config.RestoreScope,
SourceSnapshot: t.config.RestoreSnapshotID,
BackupSnapshot: t.config.RestoreSnapshotID,
ChangeDigest: hex.EncodeToString(changeDigest[:]),
MarkerDigest: hex.EncodeToString(markerDigest[:]),
RequestDigest: hex.EncodeToString(requestDigest[:]),
Owner: t.config.RestoreAttemptOwner,
Epoch: t.config.RestoreAttemptEpoch,
Scope: t.restoreScope,
SourceSnapshot: t.restoreSourceSnapshot,
BackupSnapshot: t.restoreBackupSnapshot,
TargetInstance: t.config.RestoreInstance,
TargetFingerprint: fingerprint,
IndexUUID: indexUUID,
RecoveryJobID: t.config.RestoreRecoveryJobID,
RecoveryOperationID: t.config.RestoreRecoveryOperationID,
ChangeDigest: hex.EncodeToString(changeDigest[:]),
MarkerDigest: hex.EncodeToString(markerDigest[:]),
RequestDigest: hex.EncodeToString(requestDigest[:]),
Owner: t.config.RestoreAttemptOwner,
Epoch: t.config.RestoreAttemptEpoch,
}, nil
}
@@ -67,15 +87,44 @@ func heldRestoreMarker(status libraryrelease.RestoreDispatchStatus, context stri
return fmt.Errorf("%w: %s(state=%s)", libraryrelease.ErrRestoreDispatchHeld, context, status.State)
}
// planEffect 按 index/document 找到当前冻结计划里的 effect(用于权威全集与文件证据对照)。
func (t *ProductionLibraryTarget) planEffect(index, document string) (libraryrelease.IndexWrite, bool) {
for _, c := range t.activePlan.Index {
if c.Index == index && c.ID == document {
return c, true
}
}
return libraryrelease.IndexWrite{}, false
}
// admitRestorePlan 是 Restore 在**任何** PG/对象/索引写之前的纯读准入。
//
// 删除型恢复一旦开始写 recovery 的行/对象,后面才发现某个 effect 的派发结果未知,
// 目标已经从"未动"变成"半恢复"。所以先按整份冻结计划逐 effect 判定:
// 目标已经从"未动"变成"半恢复"。所以先按整份冻结计划逐 effect 判定,并先对照原
// window 的权威台账全集(跨 plan):
// - 台账不存在(旧世界):只有"没有任何恢复意图"才算首次恢复;有意图就 held;
// - prepared:唯一可续作区段;
// - prepared:唯一可续作区段,但出现 marker/claimed/finalize/restored 任一矛盾即 held;
// - dispatching:只有精确 marker+世代才可走确认分支,否则 held;
// - marker_confirmed:只允许确认后的 clean 路径。
func (t *ProductionLibraryTarget) admitRestorePlan(ctx context.Context) error {
gate := t.config.RestoreDispatch
if gate != nil {
records, err := gate.UnresolvedRestoreDispatches(ctx, t.config.RestoreTargetIdentity,
t.config.RestoreOperationID, t.config.RestoreWindowJobID)
if err != nil {
return fmt.Errorf("恢复准入读不到权威派发台账:%w", err)
}
for _, record := range records {
if record.PlanDigest != t.activePlan.Digest {
return fmt.Errorf("%w: 原 window 存在另一份计划的派发记录(plan=%s index=%s doc=%s)",
libraryrelease.ErrRestoreDispatchHeld, record.PlanDigest, record.Index, record.DocumentID)
}
if _, ok := t.planEffect(record.Index, record.DocumentID); !ok {
return fmt.Errorf("%w: 原 window 存在计划外派发记录(index=%s doc=%s state=%s)",
libraryrelease.ErrRestoreDispatchHeld, record.Index, record.DocumentID, record.State)
}
}
}
for _, c := range t.activePlan.Index {
if len(c.After) != 0 {
continue
@@ -88,11 +137,66 @@ func (t *ProductionLibraryTarget) admitRestorePlan(ctx context.Context) error {
if err != nil {
return err
}
claimed, err := t.loadIndexEffect(c, "restore-claimed")
if err != nil {
return err
}
finalize, err := t.loadIndexEffect(c, "restore-finalize-intent")
if err != nil {
return err
}
// 已完成 clean 的 effect 是终态:恢复内容与证据都在时,重放不是未决。
done, err := t.loadIndexEffect(c, "restored")
if err != nil {
return err
}
if gate != nil {
binding, bindErr := t.restoreMarkerBinding(ctx, c, body)
if bindErr != nil {
return bindErr
}
status, found, inspErr := gate.InspectRestoreMarker(ctx, binding)
if inspErr != nil {
return fmt.Errorf("恢复准入读不到派发台账:%w", inspErr)
}
if found {
if done != nil {
if status.State != libraryrelease.RestoreDispatchMarkerConfirmed {
return heldRestoreMarker(status, "已完成 clean 的 effect 台账却不是 marker_confirmed")
}
// 终态:内容与世代由下面的 done 分支核对。
} else {
switch status.State {
case libraryrelease.RestoreDispatchPrepared:
if status.Held {
return heldRestoreMarker(status, "派发绑定与不可变计划冲突")
}
if claimed != nil || finalize != nil {
return heldRestoreMarker(status, "prepared 台账却已有 claimed/finalize 证据")
}
observed, obsErr := t.observePlannedIndex(ctx, c)
if obsErr != nil {
return obsErr
}
if indexBytesEqual(observed, body) {
return heldRestoreMarker(status, "prepared 台账却已存在目标标记:协议证据矛盾")
}
continue
case libraryrelease.RestoreDispatchDispatching, libraryrelease.RestoreDispatchMarkerConfirmed:
observed, obsErr := t.observePlannedIndex(ctx, c)
if obsErr != nil {
return obsErr
}
if indexBytesEqual(observed, body) && observed.SeqNo != nil && observed.PrimaryTerm != nil {
continue // 只允许原确认分支继续,不重发 create
}
return heldRestoreMarker(status, "目标上缺少可归属的精确恢复标记")
default:
return heldRestoreMarker(status, "派发台账状态不可识别")
}
}
}
}
if done != nil {
observed, err := t.observePlannedIndex(ctx, c)
if err != nil {
@@ -104,44 +208,17 @@ func (t *ProductionLibraryTarget) admitRestorePlan(ctx context.Context) error {
}
continue
}
if t.config.RestoreDispatch != nil {
binding, bindErr := t.restoreMarkerBinding(c, body)
if bindErr != nil {
return bindErr
}
status, found, inspErr := t.config.RestoreDispatch.InspectRestoreMarker(ctx, binding)
if inspErr != nil {
return fmt.Errorf("恢复准入读不到派发台账:%w", inspErr)
}
if found {
switch status.State {
case libraryrelease.RestoreDispatchPrepared:
continue
case libraryrelease.RestoreDispatchDispatching, libraryrelease.RestoreDispatchMarkerConfirmed:
observed, obsErr := t.observePlannedIndex(ctx, c)
if obsErr != nil {
return obsErr
}
if len(c.After) == 0 && indexBytesEqual(observed, body) && observed.SeqNo != nil && observed.PrimaryTerm != nil {
continue // 只允许原确认分支继续,不重发 create
}
return heldRestoreMarker(status, "目标上缺少可归属的精确恢复标记")
default:
return heldRestoreMarker(status, "派发台账状态不可识别")
}
}
}
if intent == nil {
if intent == nil && claimed == nil && finalize == nil {
continue
}
observed, err := t.observePlannedIndex(ctx, c)
if err != nil {
return err
}
if indexBytesEqual(observed, body) {
if intent != nil && indexBytesEqual(observed, body) {
continue // 旧标记已在目标上:按既有确认/CAS 规则处理,不派发新 create
}
return fmt.Errorf("%w: 既存恢复意图没有新协议派发台账,无法证明未派发", libraryrelease.ErrRestoreDispatchHeld)
return fmt.Errorf("%w: 既存恢复意图/证据没有新协议派发台账,无法证明未派发", libraryrelease.ErrRestoreDispatchHeld)
}
return nil
}
@@ -152,7 +229,7 @@ func (t *ProductionLibraryTarget) confirmDispatchedMarker(ctx context.Context, c
if t.config.RestoreDispatch == nil || len(c.After) != 0 {
return nil
}
binding, err := t.restoreMarkerBinding(c, body)
binding, err := t.restoreMarkerBinding(ctx, c, body)
if err != nil {
return err
}
@@ -163,6 +240,9 @@ func (t *ProductionLibraryTarget) confirmDispatchedMarker(ctx context.Context, c
if !found {
return fmt.Errorf("%w: 删除型恢复缺少派发台账,拒绝按外观继续", libraryrelease.ErrRestoreDispatchHeld)
}
if status.Held && status.State == libraryrelease.RestoreDispatchPrepared {
return heldRestoreMarker(status, "派发绑定与不可变计划冲突")
}
switch status.State {
case libraryrelease.RestoreDispatchPrepared:
// 还没派发:由调用方按 prepared 竞争许可。
@@ -190,7 +270,7 @@ func (t *ProductionLibraryTarget) dispatchRestoreMarkerCreate(ctx context.Contex
if t.config.RestoreDispatch == nil {
return indexReceipt{}, errors.New("恢复标记派发协议未接线:拒绝发送")
}
binding, err := t.restoreMarkerBinding(c, body)
binding, err := t.restoreMarkerBinding(ctx, c, body)
if err != nil {
return indexReceipt{}, err
}
@@ -198,13 +278,13 @@ func (t *ProductionLibraryTarget) dispatchRestoreMarkerCreate(ctx context.Contex
if err != nil {
return indexReceipt{}, err
}
if !found || status.State != libraryrelease.RestoreDispatchPrepared {
if err != nil {
return indexReceipt{}, err
}
if !found || status.State != libraryrelease.RestoreDispatchPrepared || status.Held {
return indexReceipt{}, heldRestoreMarker(status, "没有可竞争的 prepared 许可")
}
// 控制流故障注入必须在 CAS 之前:拒绝时状态仍是 prepared,可安全重试,零转发。
if err := t.ds01TestRefusal(c); err != nil {
return indexReceipt{}, err
}
if err := t.refuseRestoreWrite(c); err != nil {
return indexReceipt{}, err
}
@@ -255,9 +335,39 @@ func (t *ProductionLibraryTarget) sendRestoreMarkerOnce(ctx context.Context, c l
return receipt, nil
}
// verifyRestoreEffectsClosed 是 Restore 收尾前的最后一道检查:任何一个删除型 effect
// 只要留下恢复意图,就必须已经完成 clean(restored 证据),且台账不得停在 dispatching。
// verifyRestoreEffectsClosed 是 Restore 收尾前的最后一道检查:以原 window 的权威台账
// 全集为准(跨 plan),每个记录都必须是 marker_confirmed,且对应的 restored 文件证据
// 存在。prepared/dispatching、计划外 effect、换 plan 别名、缺 restored 一律拒绝。
func (t *ProductionLibraryTarget) verifyRestoreEffectsClosed(ctx context.Context) error {
if t.config.RestoreDispatch != nil {
records, err := t.config.RestoreDispatch.UnresolvedRestoreDispatches(ctx,
t.config.RestoreTargetIdentity, t.config.RestoreOperationID, t.config.RestoreWindowJobID)
if err != nil {
return fmt.Errorf("恢复收尾读不到权威派发台账:%w", err)
}
for _, record := range records {
if record.PlanDigest != t.activePlan.Digest {
return fmt.Errorf("%w: 收尾时原 window 仍有另一份计划的派发记录(plan=%s)",
libraryrelease.ErrRestoreDispatchHeld, record.PlanDigest)
}
if record.State != libraryrelease.RestoreDispatchMarkerConfirmed {
return fmt.Errorf("%w: 收尾前派发台账仍有未决 effect(index=%s doc=%s state=%s)",
libraryrelease.ErrRestoreDispatchHeld, record.Index, record.DocumentID, record.State)
}
c, ok := t.planEffect(record.Index, record.DocumentID)
if !ok {
return fmt.Errorf("%w: 收尾时存在计划外派发记录(index=%s doc=%s)",
libraryrelease.ErrRestoreDispatchHeld, record.Index, record.DocumentID)
}
done, err := t.loadIndexEffect(c, "restored")
if err != nil {
return err
}
if done == nil {
return fmt.Errorf("%w: 收尾前仍有未 clean 的 effect %s", libraryrelease.ErrRestoreDispatchHeld, c.ID)
}
}
}
for _, c := range t.activePlan.Index {
if len(c.After) != 0 {
continue
@@ -276,24 +386,21 @@ func (t *ProductionLibraryTarget) verifyRestoreEffectsClosed(ctx context.Context
if done == nil {
return fmt.Errorf("%w: 恢复收尾前仍有未决 effect %s", libraryrelease.ErrRestoreDispatchHeld, c.ID)
}
if t.config.RestoreDispatch == nil {
continue
}
body, err := t.restoreIndexClaimBody(c)
if err != nil {
return err
}
binding, err := t.restoreMarkerBinding(c, body)
if err != nil {
return err
}
status, found, err := t.config.RestoreDispatch.InspectRestoreMarker(ctx, binding)
if err != nil {
return err
}
if found && status.State == libraryrelease.RestoreDispatchDispatching {
return heldRestoreMarker(status, "收尾前台账仍停在 dispatching")
}
}
return nil
}
// ds01TestRefusal 是正式 worker 全链验收的最小测试接缝:只有显式隔离运行
// (KG_LIBRARY_TEST_ISOLATED=1)且指定了 effect 时才在 CAS 之前拒绝一次;生产默认
// 全关,且每个进程只触发一次,重启后的 worker 不会重复拒绝。
func (t *ProductionLibraryTarget) ds01TestRefusal(c libraryrelease.IndexWrite) error {
if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" || t.restoreRefusalUsed {
return nil
}
spec := os.Getenv("KG_DS01_TEST_REFUSE_MARKER")
if spec == "" || spec != c.Index+"/"+c.ID {
return nil
}
t.restoreRefusalUsed = true
return errors.New("injected DS01 marker refusal before dispatch CAS")
}
@@ -63,12 +63,14 @@ type ProductionLibraryConfig struct {
RestoreDispatch libraryrelease.RestoreMarkerGate
RestoreTargetIdentity string
RestoreScope string
RestoreInstance string
RestoreOperationID string
RestoreWindowJobID string
RestoreAttemptOwner string
RestoreAttemptEpoch int64
// RestoreSnapshotID 是本次恢复要用的那份预备份(与原 window/plan 绑定)。
RestoreSnapshotID string
// 恢复分发身份:只在恢复 worker 里非空;普通自动回滚为空。
RestoreRecoveryJobID string
RestoreRecoveryOperationID string
}
func (config ProductionLibraryConfig) validate(scope ProductionScope) error {
@@ -114,6 +116,22 @@ type ProductionLibraryTarget struct {
preserveChunkSequence bool
activePlan *libraryrelease.WriteSet
activeSnapshot string
// restoreScope / restoreSourceSnapshot / restoreBackupSnapshot 在 Restore 进入时
// 从**原持久锚点**冻结:scope 来自真实装配,来源快照与写前镜像来自本次 WriteSet
// 与原 backup 引用。它们不能由调用方在恢复中途改写,也不允许同一个快照既当
// 来源又当 before-image。
restoreScope string
restoreSourceSnapshot string
restoreBackupSnapshot string
// restoreFingerprint / restoreIndexUUID 是本次恢复冻结的实际目标身份。派发绑定
// 每次重建时都实时重读并比较:同名重建的目标/索引不可能复用旧证明。
restoreFingerprint string
restoreIndexUUID string
// restoreIntentFault 是一次性文件写失败注入点(生产恒为 nil):在 prepared 已提交、
// mark-intent 尚未落盘处返回错误,用来复现"可补同内容"的窄故障区段。
restoreIntentFault func(libraryrelease.IndexWrite) error
// restoreRefusalUsed 是测试专用接缝的一次性标记(生产恒为 false)。
restoreRefusalUsed bool
// restoreMarkerFault 是一次性控制流故障注入点(生产恒为 nil):它在恢复标记的
// 持久意图落盘之后、网络派发之前返回错误,用来区分"边界上确定没写进去"与
// "写入结果未知"——前者重试时只凭世代证据重做,后者仍按 B19 永久 held。
@@ -971,6 +989,18 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
return fmt.Errorf("写集合记录绑定的是 %d/%s,与目标 %d/%s 不一致",
backup.WriteSet.TenantID, backup.WriteSet.KnowledgeBaseID, t.scope.TenantID, t.scope.KnowledgeBaseID)
}
// 派发绑定只能来自原锚点:来源快照是发布基线,backup 是这份写集的写前镜像。
// 二者缺失或相同都说明装配没有拿到真实身份,必须在任何写之前失败关闭。
if backup.WriteSet.TargetSnapshotID == "" || backup.SnapshotID == "" {
return errors.New("恢复缺少来源快照或写前镜像,拒绝建立派发绑定")
}
if backup.WriteSet.TargetSnapshotID == backup.SnapshotID {
return errors.New("来源快照与写前镜像不能是同一份快照,拒绝建立派发绑定")
}
t.restoreScope = t.config.RestoreScope
t.restoreSourceSnapshot = backup.WriteSet.TargetSnapshotID
t.restoreBackupSnapshot = backup.SnapshotID
t.restoreFingerprint, t.restoreIndexUUID = "", ""
if err := t.AdoptWriteSet(*backup.WriteSet); err != nil {
return err
}
@@ -0,0 +1,56 @@
package v071
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"strings"
)
// 恢复派发只接受"实际目标"的身份:租户/KB、目标库(服务器+库名)、索引名与索引
// UUID、对象桶。它们用于识别"同名但被重建/替换"的目标,让旧绑定无法复用;它们不是
// 历史未发送的证明,也不构成 ES 端 fence。
func (t *ProductionLibraryTarget) observeRestoreTarget(ctx context.Context) (string, string, error) {
if t.pool == nil || t.search == nil {
return "", "", errors.New("restore dispatch requires a live target")
}
var database, address string
var port int
if err := t.pool.QueryRow(ctx,
`SELECT current_database(), COALESCE(host(inet_server_addr()), ''), COALESCE(inet_server_port(), 0)`).
Scan(&database, &address, &port); err != nil {
return "", "", fmt.Errorf("read target database identity: %w", err)
}
if database == "" {
return "", "", errors.New("target database identity is empty")
}
indexUUID, err := t.search.IndexUUID(ctx)
if err != nil {
return "", "", fmt.Errorf("read target index uuid: %w", err)
}
joined := strings.Join([]string{
"v1", fmt.Sprintf("%d", t.scope.TenantID), t.scope.KnowledgeBaseID,
database, address, fmt.Sprintf("%d", port), t.config.SearchIndex, indexUUID, t.config.ObjectsBucket,
}, "\x00")
sum := sha256.Sum256([]byte(joined))
return hex.EncodeToString(sum[:]), indexUUID, nil
}
// frozenRestoreTarget 在一个 Restore 调用内冻结实际目标身份;后续每次重建绑定都
// 重新观察并与冻结值比较。目标/索引在恢复中途被替换时立即失败,而不是继续用旧绑定。
func (t *ProductionLibraryTarget) frozenRestoreTarget(ctx context.Context) (string, string, error) {
fingerprint, indexUUID, err := t.observeRestoreTarget(ctx)
if err != nil {
return "", "", err
}
if t.restoreFingerprint == "" {
t.restoreFingerprint, t.restoreIndexUUID = fingerprint, indexUUID
return fingerprint, indexUUID, nil
}
if fingerprint != t.restoreFingerprint || indexUUID != t.restoreIndexUUID {
return "", "", fmt.Errorf("%w: 目标/索引身份在恢复过程中发生变化", ErrProductionTargetUnconfigured)
}
return t.restoreFingerprint, t.restoreIndexUUID, nil
}
+19
View File
@@ -278,6 +278,25 @@ func (exporter *SearchExporter) requestJSON(ctx context.Context, method, path st
return exporter.requestJSONWithReplay(ctx, method, path, payload, result, true)
}
// IndexUUID 读目标索引的实际 UUID。同名重建会得到新 UUID —— 恢复派发用它识别
// "目标被替换",不是用它证明历史未发送。索引不存在按 errIndexMissing 返回。
func (exporter *SearchExporter) IndexUUID(ctx context.Context) (string, error) {
var rows []struct {
Index string `json:"index"`
UUID string `json:"uuid"`
}
path := "/_cat/indices/" + url.PathEscape(exporter.index) + "?format=json&h=index,uuid"
if err := exporter.requestJSON(ctx, http.MethodGet, path, nil, &rows); err != nil {
return "", err
}
for _, row := range rows {
if row.Index == exporter.index && row.UUID != "" {
return row.UUID, nil
}
}
return "", errors.New("target index uuid is missing")
}
// requestJSONOnce 关闭传输层对该请求的重放能力(GetBody=nil):恢复标记 create
// 每个许可只能落地至多一次真实请求,不能让 keep-alive 重连把它悄悄发第二遍。
// Go 的 Transport 只会自动重试"可重放"的请求;Body 非空且 GetBody 为 nil 时,