diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a171ca..8d4b988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,29 @@ # Changelog +## 开发快照(DS01 复核修复:真实绑定、同事务授权、可核激活与完整未决分类) + +### Fixed + +- **正式装配绑定**:`OpenProductionStack` 现在把 scope/实例与本次真实 attempt 身份传入目标; + 来源快照与写前镜像分别取自原 `WriteSet.TargetSnapshotID` 与本次 backup,不再把同一个 + `RestoreSnapshotID` 同时当来源和 before-image。缺锚点或二者相同在写之前失败关闭。 +- **派发线性化**:激活撤销与 `prepared→dispatching` CAS 在同一事务里用 activation 行锁串行化; + lease 在取锁之后用 `clock_timestamp()` 判定;同事务核对原 job/operation/attempt、来源/写集/ + 维护窗口与恢复投递(queue + recovery operation)绑定;同 effect 换 plan 摘要被拒绝。 +- **可核激活**:激活改为消费带 sha256 的证据文件(目标身份/指纹/实例、治理库名、协议构建、 + 逐 writer 的退出与排空观察、有效期);撤销不可回退为 `NULL`、activation 行不可删除, + down 在仅有 activation/撤销证据时也拒绝。 +- **准入/收尾完整分类**:绑定冲突、`prepared` 与 marker/claimed/finalize 矛盾在零目标写之前 + held;合法 `dispatching`+精确 marker 仍走确认分支。收尾改用原 window 权威台账全集(跨 plan), + 拒绝计划外记录、换 plan 别名与缺 `restored` 证据的 effect。 +- **可恢复区段**:`prepared` 已提交但 mark-intent 未落盘时,核对绑定一致且无矛盾后补写同一 + 意图并继续;无台账的旧意图仍 held,r33 不回填。 + +### Security + +- 绑定冻结实际目标实例/指纹/索引 UUID,同名重建的目标或索引会让旧证明失效;它们用于识别 + 对象,不用于证明历史未发送。协议仍不声称 ES 端 fence 或旧 `dispatching` owner 零迟到写。 + ## 开发快照(DS01:删除型恢复的派发台账与唯一发送许可) ### Added diff --git a/docs/architecture/ds01-restore-dispatch.md b/docs/architecture/ds01-restore-dispatch.md index e469d13..1edd8a1 100644 --- a/docs/architecture/ds01-restore-dispatch.md +++ b/docs/architecture/ds01-restore-dispatch.md @@ -15,57 +15,70 @@ PG 快照回滚/手工删行之后的协议有效性(这类历史倒退必须 ## 状态与唯一发送路径 ``` -prepared --(CAS, 同一短事务内校验 job/operation/attempt/lease/binding)--> dispatching +prepared --(CAS, 同一短事务内校验 activation/job/operation/attempt/lease/binding)--> dispatching dispatching --(真实 receipt + 当前 owner/lease)--> marker_confirmed ``` - 发送许可只由 `AcquireRestoreMarker` 在 **COMMIT 明确返回后**给出,`RestoreSendPermit` 消费一次即作废。 +- CAS 事务按 `job → operation → activation → active attempt → effect` 取锁: + - 激活撤销(`revoked_at`)与 CAS 在同一事务里线性化:撤销先完成则拒绝,CAS 先取锁则撤销排队; + - lease 用 `clock_timestamp()` 在取锁之后判定,等锁跨过租约即拒绝; + - attempt 必须属于原 window job;job 的 owner/epoch/lease、operation 的 scope/来源快照/ + 已登记写集摘要、实例维护窗口(未释放且属于该作业)逐项核对; + - 恢复路径另核 recovery queue job(运行中且有 lease)与 recovery operation(`kind=recover`、 + 绑定同一 backup); + - 同一 effect 换 plan 摘要会被台账拒绝(不能靠新 key 重开证明)。 +- 不可变绑定来自原持久锚点:`scope`、来源快照(发布基线)、写前镜像(backup)、目标实例/ + 实际指纹/索引 UUID、变更/标记/请求摘要与 attempt 身份。来源与写前镜像不允许是同一份快照。 + 目标指纹与索引 UUID 每次重建绑定都实时重读并与本次恢复冻结值比较:同名重建的目标/索引 + 无法复用旧证明。 - 恢复期 `_create` 只从 `sendRestoreMarkerOnce` 发出;preflight/故障钩子都在 CAS 之前。 - 请求设置 `GetBody=nil`,Go 传输层不会在 keep-alive 重连时重放该 mutation。 ## Restore 准入与收尾 -`Restore` 在 `AdoptWriteSet` 之后、任何阶段写入之前调用纯读准入: +`Restore` 在 `AdoptWriteSet` 之后、任何阶段写入之前调用纯读准入,并先对照原 window 的 +**权威台账全集**(跨 plan 摘要):出现另一份 plan、计划外 effect 或不可识别状态立即 held。 | 台账/证据 | 结果 | |---|---| -| 无意图、无台账 | 首次恢复,可建立 `prepared` | +| 无意图、无证据、无台账 | 首次恢复,可建立 `prepared` | | 旧意图、无台账 | held(旧协议未知历史,不补 prepared) | -| `prepared` | 可竞争一次许可 | +| `prepared` + 绑定冲突(`Held`) | held | +| `prepared` + 已存在 claimed/finalize/目标精确 marker | held(协议证据矛盾) | +| `prepared` + 无矛盾 | 可竞争一次许可 | +| `prepared` + mark-intent 未落盘(文件失败/崩溃区段) | 核对绑定一致后补写**同内容**意图,再竞争 | | `dispatching` + 目标无精确 marker | held(零新发送) | | `dispatching` + 精确 marker/内容/世代 | 只走确认分支,不发 create | | `marker_confirmed` + 无 `restored` 证据 | held | | `restored` 证据与目标内容/世代一致 | 终态,允许重放 | -收尾前(整库指纹比对之后)再检查一次:仍有 `restored` 缺失或停在 `dispatching` 的 effect ⇒ 拒绝 -返回成功,维护窗口不会被释放。 +收尾前(整库指纹比对之后)以同一份权威全集再检查一次:任何 `prepared`/`dispatching` 记录、 +计划外记录、换 plan 摘要、或缺失 `restored` 文件证据的 effect 都拒绝返回成功,维护窗口不会 +被释放。 ## 激活(部署动作,不属于本开发包) 新能力默认关闭。启用需要两步,缺一不可: 1. 部署配置 `LIBRARY_RESTORE_DISPATCH_V2=true`(只表示允许装配); -2. 部署流程在治理库登记"旧 writer 已退出并排空"的证据: +2. 部署流程生成一份**证据文件**,并由 `Store.ActivateRestoreDispatch` 登记。代码会: + - 重算文件 sha256 并与调用方给出的摘要比较(不接受文件自报值); + - 严格解析并校验:协议版本、目标身份/实例/指纹、治理库名、协议构建、签署人与时间窗口; + - 要求每个受控 writer 都登记 `stopped + queue_drained + exited_at + drain_evidence`; + - 读取现场 `current_database()` 与证据里的治理库名比较; + - 目标身份/指纹与调用方声明的现场身份一致。 + 任何一项不满足都不建立 active 行。 -```sql -INSERT INTO library_index_restore_dispatch_activation - (target_identity, protocol_version, activated_by, drain_evidence_digest) -VALUES ($1, 1, $2, $3) -ON CONFLICT (target_identity) DO NOTHING; -``` +撤销是终局:`revoked_at` 一旦写入,SQL 不能改回 `NULL`、不能改时间,activation 行也不能删除; +重新开放同一物理目标只能换新的目标身份。 -`target_identity` 必须等于 `appconfig.Config.LibraryTargetIdentity()` 的值;`drain_evidence_digest` -是部署证据的摘要。撤销: - -```sql -UPDATE library_index_restore_dispatch_activation SET revoked_at=now() -WHERE target_identity=$1 AND revoked_at IS NULL; -``` - -登记的是运维声明,代码只保证"没有登记就不发放许可",不验证声明本身为真。 +边界:代码核验的是证据文件完整性、结构完整性与目标/治理库身份;"这些 writer 确实已退出" +仍是部署流程的受控 attestation(附退出时间与排空摘要),不是本进程能独立观察到的外部事实。 ## 迁移与回滚 - `000078_library_restore_dispatch.up.sql` 只建治理库表与触发器,不碰 WeKnora Schema。 -- `000078_..._down.sql` 在 `library_index_restore_dispatch` 有任意行时**拒绝执行**:这些行是 - "是否发送过"的唯一证明,抹掉等于把 unknown 伪装成 prepared。 \ No newline at end of file +- `000078_..._down.sql` 在 `library_index_restore_dispatch` 或 activation 表有**任意行**时拒绝执行: + 这些行是"是否发送过 / 是否曾被授权"的唯一证明,抹掉等于把 unknown 伪装成 prepared,或把 + 已撤销的激活悄悄删掉。 \ No newline at end of file diff --git a/internal/libraryrelease/index_restore_dispatch.go b/internal/libraryrelease/index_restore_dispatch.go index 4a14a95..9dd9ebf 100644 --- a/internal/libraryrelease/index_restore_dispatch.go +++ b/internal/libraryrelease/index_restore_dispatch.go @@ -60,16 +60,26 @@ func (k RestoreMarkerKey) Validate() error { // RestoreMarkerBinding 是派发许可的不可变绑定。目标、快照、计划、变更体与标记体任一 // 变化都必须 held,不能另起一行继续写。 +// +// SourceSnapshot 是发布基线(原作业要同步到的那份来源),BackupSnapshot 是本次失败 +// 写集的写前镜像;两者必须来自原持久锚点且不能互相代替。TargetInstance / +// TargetFingerprint / IndexUUID 冻结"实际目标",用于识别目标被同名字替换;它们不证明 +// 历史上没有发送过任何请求。 type RestoreMarkerBinding struct { - Key RestoreMarkerKey `json:"key"` - Scope string `json:"scope"` - SourceSnapshot string `json:"source_snapshot"` - BackupSnapshot string `json:"backup_snapshot"` - ChangeDigest string `json:"change_digest"` - MarkerDigest string `json:"marker_digest"` - RequestDigest string `json:"request_digest"` - Owner string `json:"owner"` - Epoch int64 `json:"epoch"` + Key RestoreMarkerKey `json:"key"` + Scope string `json:"scope"` + SourceSnapshot string `json:"source_snapshot"` + BackupSnapshot string `json:"backup_snapshot"` + TargetInstance string `json:"target_instance"` + TargetFingerprint string `json:"target_fingerprint"` + IndexUUID string `json:"index_uuid"` + RecoveryJobID string `json:"recovery_job_id,omitempty"` + RecoveryOperationID string `json:"recovery_operation_id,omitempty"` + ChangeDigest string `json:"change_digest"` + MarkerDigest string `json:"marker_digest"` + RequestDigest string `json:"request_digest"` + Owner string `json:"owner"` + Epoch int64 `json:"epoch"` } func (b RestoreMarkerBinding) Validate() error { @@ -78,13 +88,20 @@ func (b RestoreMarkerBinding) Validate() error { } for name, value := range map[string]string{ "scope": b.Scope, "source snapshot": b.SourceSnapshot, "backup snapshot": b.BackupSnapshot, - "change digest": b.ChangeDigest, "marker digest": b.MarkerDigest, + "target instance": b.TargetInstance, "target fingerprint": b.TargetFingerprint, + "index uuid": b.IndexUUID, "change digest": b.ChangeDigest, "marker digest": b.MarkerDigest, "request digest": b.RequestDigest, "owner": b.Owner, } { if strings.TrimSpace(value) == "" { return fmt.Errorf("restore marker binding lacks %s", name) } } + if b.SourceSnapshot == b.BackupSnapshot { + return errors.New("restore marker binding uses the source snapshot as the backup image") + } + if b.RecoveryJobID == "" != (b.RecoveryOperationID == "") { + return errors.New("restore marker binding has a partial recovery delivery identity") + } if b.Epoch <= 0 { return errors.New("restore marker binding lacks a positive attempt epoch") } @@ -97,8 +114,9 @@ func (b RestoreMarkerBinding) Digest() (string, error) { return "", err } joined := strings.Join([]string{ - "v1", b.Key.TargetIdentity, b.Key.OperationID, b.Key.WindowJobID, b.Key.PlanDigest, + "v2", b.Key.TargetIdentity, b.Key.OperationID, b.Key.WindowJobID, b.Key.PlanDigest, b.Key.Index, b.Key.DocumentID, b.Key.Phase, b.Scope, b.SourceSnapshot, b.BackupSnapshot, + b.TargetInstance, b.TargetFingerprint, b.IndexUUID, b.RecoveryJobID, b.RecoveryOperationID, b.ChangeDigest, b.MarkerDigest, b.RequestDigest, }, "\x00") sum := sha256.Sum256([]byte(joined)) @@ -151,13 +169,24 @@ func (p *RestoreSendPermit) Consume(key RestoreMarkerKey, bindingDigest string) return nil } +// RestoreDispatchRecord 是权威台账里"这个原 window 还有哪些未决 effect"的一条。 +// 收尾与前置都查同一份记录集,不能只遍历当前 plan 的文档列表。 +type RestoreDispatchRecord struct { + PlanDigest string `json:"plan_digest"` + Index string `json:"index"` + DocumentID string `json:"document_id"` + State string `json:"state"` +} + // RestoreMarkerGate 是唯一能授予恢复标记 create 发送权的接口。 // // Inspect 纯读、不返回许可;Prepare 只建立 prepared;Acquire 在独立短事务里做 -// prepared→dispatching 的 CAS,只有明确提交后才返回许可;Confirm 只记录真实回执。 +// prepared→dispatching 的 CAS,只有明确提交后才返回许可;Confirm 只记录真实回执; +// UnresolvedRestoreDispatches 返回原 window 的权威未决全集(跨 plan,只读)。 type RestoreMarkerGate interface { InspectRestoreMarker(ctx context.Context, b RestoreMarkerBinding) (RestoreDispatchStatus, bool, error) PrepareRestoreMarker(ctx context.Context, b RestoreMarkerBinding) (RestoreDispatchStatus, error) AcquireRestoreMarker(ctx context.Context, b RestoreMarkerBinding) (*RestoreSendPermit, RestoreDispatchStatus, error) ConfirmRestoreMarker(ctx context.Context, b RestoreMarkerBinding, receipt []byte) error + UnresolvedRestoreDispatches(ctx context.Context, targetIdentity, operationID, windowJobID string) ([]RestoreDispatchRecord, error) } diff --git a/internal/libraryrelease/index_restore_dispatch_test.go b/internal/libraryrelease/index_restore_dispatch_test.go index bc2b194..181b609 100644 --- a/internal/libraryrelease/index_restore_dispatch_test.go +++ b/internal/libraryrelease/index_restore_dispatch_test.go @@ -10,7 +10,8 @@ func restoreBinding() RestoreMarkerBinding { return RestoreMarkerBinding{ Key: RestoreMarkerKey{TargetIdentity: "target-a", OperationID: "op-1", WindowJobID: "job-1", PlanDigest: "plan-1", Index: "kg-index", DocumentID: "doc-1", Phase: RestoreMarkerPhaseCreate}, - Scope: "internal", SourceSnapshot: "snap-1", BackupSnapshot: "snap-1", + Scope: "internal", SourceSnapshot: "snap-src", BackupSnapshot: "snap-backup", + TargetInstance: "inst-1", TargetFingerprint: "fingerprint-1", IndexUUID: "uuid-1", ChangeDigest: "change-1", MarkerDigest: "marker-1", RequestDigest: "request-1", Owner: "worker-a#queue#1", Epoch: 3, } @@ -18,18 +19,22 @@ func restoreBinding() RestoreMarkerBinding { func TestRestoreMarkerBindingRejectsIncompleteIdentity(t *testing.T) { for name, mutate := range map[string]func(*RestoreMarkerBinding){ - "target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "" }, - "operation": func(b *RestoreMarkerBinding) { b.Key.OperationID = "" }, - "window": func(b *RestoreMarkerBinding) { b.Key.WindowJobID = "" }, - "plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "" }, - "index": func(b *RestoreMarkerBinding) { b.Key.Index = "" }, - "document": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "" }, - "change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "" }, - "marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "" }, - "request": func(b *RestoreMarkerBinding) { b.RequestDigest = "" }, - "owner": func(b *RestoreMarkerBinding) { b.Owner = "" }, - "epoch": func(b *RestoreMarkerBinding) { b.Epoch = 0 }, - "phase": func(b *RestoreMarkerBinding) { b.Key.Phase = "apply-create" }, + "target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "" }, + "operation": func(b *RestoreMarkerBinding) { b.Key.OperationID = "" }, + "window": func(b *RestoreMarkerBinding) { b.Key.WindowJobID = "" }, + "plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "" }, + "index": func(b *RestoreMarkerBinding) { b.Key.Index = "" }, + "document": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "" }, + "change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "" }, + "marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "" }, + "request": func(b *RestoreMarkerBinding) { b.RequestDigest = "" }, + "owner": func(b *RestoreMarkerBinding) { b.Owner = "" }, + "epoch": func(b *RestoreMarkerBinding) { b.Epoch = 0 }, + "phase": func(b *RestoreMarkerBinding) { b.Key.Phase = "apply-create" }, + "instance": func(b *RestoreMarkerBinding) { b.TargetInstance = "" }, + "fingerprint": func(b *RestoreMarkerBinding) { b.TargetFingerprint = "" }, + "index uuid": func(b *RestoreMarkerBinding) { b.IndexUUID = "" }, + "source as backup": func(b *RestoreMarkerBinding) { b.BackupSnapshot = b.SourceSnapshot }, } { binding := restoreBinding() mutate(&binding) @@ -51,16 +56,20 @@ func TestRestoreMarkerBindingDigestCoversEveryBoundValue(t *testing.T) { t.Fatal("binding digest is not deterministic") } for name, mutate := range map[string]func(*RestoreMarkerBinding){ - "target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "target-b" }, - "index": func(b *RestoreMarkerBinding) { b.Key.Index = "other-index" }, - "doc": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "doc-2" }, - "plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "plan-2" }, - "backup": func(b *RestoreMarkerBinding) { b.BackupSnapshot = "snap-2" }, - "change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "change-2" }, - "marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "marker-2" }, - "request": func(b *RestoreMarkerBinding) { b.RequestDigest = "request-2" }, - "scope": func(b *RestoreMarkerBinding) { b.Scope = "external" }, - "source": func(b *RestoreMarkerBinding) { b.SourceSnapshot = "snap-9" }, + "target": func(b *RestoreMarkerBinding) { b.Key.TargetIdentity = "target-b" }, + "index": func(b *RestoreMarkerBinding) { b.Key.Index = "other-index" }, + "doc": func(b *RestoreMarkerBinding) { b.Key.DocumentID = "doc-2" }, + "plan": func(b *RestoreMarkerBinding) { b.Key.PlanDigest = "plan-2" }, + "backup": func(b *RestoreMarkerBinding) { b.BackupSnapshot = "snap-2" }, + "change": func(b *RestoreMarkerBinding) { b.ChangeDigest = "change-2" }, + "marker": func(b *RestoreMarkerBinding) { b.MarkerDigest = "marker-2" }, + "request": func(b *RestoreMarkerBinding) { b.RequestDigest = "request-2" }, + "scope": func(b *RestoreMarkerBinding) { b.Scope = "external" }, + "source": func(b *RestoreMarkerBinding) { b.SourceSnapshot = "snap-9" }, + "instance": func(b *RestoreMarkerBinding) { b.TargetInstance = "inst-2" }, + "fingerprint": func(b *RestoreMarkerBinding) { b.TargetFingerprint = "fingerprint-2" }, + "index uuid": func(b *RestoreMarkerBinding) { b.IndexUUID = "uuid-2" }, + "recovery queue": func(b *RestoreMarkerBinding) { b.RecoveryJobID = "q-1"; b.RecoveryOperationID = "op-r" }, } { binding := restoreBinding() mutate(&binding) diff --git a/internal/libraryrelease/restore_dispatch_activation.go b/internal/libraryrelease/restore_dispatch_activation.go new file mode 100644 index 0000000..0705ccf --- /dev/null +++ b/internal/libraryrelease/restore_dispatch_activation.go @@ -0,0 +1,129 @@ +package libraryrelease + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "strings" + "time" +) + +// 恢复派发激活证据(DS01-F3)。 +// +// 配置开关与"三个非空字符串"都不是独占证明。激活必须消费一份**可核验的证据文件**: +// - 文件字节的 sha256 由调用方给出并在读取后重算(不是文件里自报的布尔值); +// - 证据只对某一目标身份/目标指纹/治理库生效; +// - 受控 writer 必须逐个登记"已退出 + 队列已排空"以及各自的退出观察; +// - 证据有明确的签署人与时间窗口,过期不接受。 +// +// 边界(必须如实写进报告):代码能核验文件完整性、结构完整性、目标/治理库身份与 +// 新鲜度;"这些 writer 真的已经停止"仍是部署流程的受控 attestation(附退出时间和 +// 排空证据摘要),不是本进程运行时能独立观察到的外部事实。没有这份证据就不建立 +// active 行——这一点由代码强制。 + +const RestoreDispatchActivationMaxAge = 2 * time.Hour + +type RestoreDispatchWriterDrain struct { + Name string `json:"name"` + Kind string `json:"kind"` + Stopped bool `json:"stopped"` + ExitedAt *time.Time `json:"exited_at"` + QueueDrained bool `json:"queue_drained"` + DrainEvidence string `json:"drain_evidence"` +} + +type RestoreDispatchActivationEvidence struct { + ProtocolVersion int `json:"protocol_version"` + TargetIdentity string `json:"target_identity"` + TargetFingerprint string `json:"target_fingerprint"` + TargetInstance string `json:"target_instance"` + GovernanceDatabase string `json:"governance_database"` + ProtocolBuild string `json:"protocol_build"` + AttestedBy string `json:"attested_by"` + AttestedAt time.Time `json:"attested_at"` + Writers []RestoreDispatchWriterDrain `json:"writers"` +} + +func (e RestoreDispatchActivationEvidence) Validate(now time.Time, maxAge time.Duration) error { + if e.ProtocolVersion != 1 { + return fmt.Errorf("restore dispatch activation protocol version %d is not supported", e.ProtocolVersion) + } + for name, value := range map[string]string{ + "target identity": e.TargetIdentity, "target fingerprint": e.TargetFingerprint, + "target instance": e.TargetInstance, "governance database": e.GovernanceDatabase, + "protocol build": e.ProtocolBuild, "attested by": e.AttestedBy, + } { + if strings.TrimSpace(value) == "" { + return fmt.Errorf("restore dispatch activation evidence lacks %s", name) + } + } + if len(e.Writers) == 0 { + return errors.New("restore dispatch activation evidence lists no controlled writers") + } + seen := map[string]bool{} + for _, writer := range e.Writers { + if strings.TrimSpace(writer.Name) == "" || strings.TrimSpace(writer.Kind) == "" { + return errors.New("restore dispatch activation writer lacks name or kind") + } + if seen[writer.Name] { + return fmt.Errorf("restore dispatch activation writer %s is listed twice", writer.Name) + } + seen[writer.Name] = true + if !writer.Stopped || !writer.QueueDrained { + return fmt.Errorf("restore dispatch activation writer %s has not exited and drained", writer.Name) + } + if writer.ExitedAt == nil || writer.ExitedAt.IsZero() { + return fmt.Errorf("restore dispatch activation writer %s has no exit observation", writer.Name) + } + if strings.TrimSpace(writer.DrainEvidence) == "" { + return fmt.Errorf("restore dispatch activation writer %s has no drain evidence", writer.Name) + } + if writer.ExitedAt.After(e.AttestedAt) { + return fmt.Errorf("restore dispatch activation writer %s exited after the attestation", writer.Name) + } + } + if e.AttestedAt.IsZero() || e.AttestedAt.After(now.Add(5*time.Minute)) { + return errors.New("restore dispatch activation attestation time is not credible") + } + if maxAge <= 0 { + maxAge = RestoreDispatchActivationMaxAge + } + if now.Sub(e.AttestedAt) > maxAge { + return fmt.Errorf("restore dispatch activation evidence is older than %s", maxAge) + } + return nil +} + +// LoadRestoreDispatchActivationEvidence 读取并核验证据文件字节摘要。expectedDigest 必须 +// 与文件实际 sha256 相同;不接受"文件里写的摘要"。 +func LoadRestoreDispatchActivationEvidence(path, expectedDigest string, now time.Time, maxAge time.Duration) (RestoreDispatchActivationEvidence, error) { + var evidence RestoreDispatchActivationEvidence + if strings.TrimSpace(path) == "" || strings.TrimSpace(expectedDigest) == "" { + return evidence, errors.New("restore dispatch activation requires an evidence file and its digest") + } + raw, err := os.ReadFile(path) + if err != nil { + return evidence, fmt.Errorf("read restore dispatch activation evidence: %w", err) + } + sum := sha256.Sum256(raw) + if got := hex.EncodeToString(sum[:]); got != strings.ToLower(strings.TrimSpace(expectedDigest)) { + return evidence, errors.New("restore dispatch activation evidence digest does not match the file") + } + decoder := json.NewDecoder(strings.NewReader(string(raw))) + decoder.DisallowUnknownFields() + if err := decoder.Decode(&evidence); err != nil { + return RestoreDispatchActivationEvidence{}, fmt.Errorf("parse restore dispatch activation evidence: %w", err) + } + if err := evidence.Validate(now, maxAge); err != nil { + return RestoreDispatchActivationEvidence{}, err + } + return evidence, nil +} + +// CanonicalBytes 是写入台账的规范字节(同一份证据只有一种序列化)。 +func (e RestoreDispatchActivationEvidence) CanonicalBytes() ([]byte, error) { + return json.Marshal(e) +} diff --git a/internal/librarystack/stack.go b/internal/librarystack/stack.go index 07f987a..c5a0343 100644 --- a/internal/librarystack/stack.go +++ b/internal/librarystack/stack.go @@ -90,7 +90,49 @@ func scopeConfigOf(config appconfig.Config, scope string) (scopeConfig, error) { }, nil } -func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig, +// RestoreDispatchWiring 是恢复派发身份的唯一装配来源:真实 attempt(objectgateway +// claim)、原 window 作业、部署目标实例与 scope。adapter 不接受调用方另填身份。 +type RestoreDispatchWiring struct { + TargetIdentity string + Scope string + Instance string + OperationID string + WindowJobID string + AttemptOwner string + AttemptEpoch int64 + RecoveryJobID string + RecoveryOperationID string +} + +// RestoreDispatchWiringFor 把本次真实执行的 claim 映射成派发身份,并在字段缺失时 +// 失败关闭。它不读取快照:来源/写前镜像只能从原 WriteSet 与 backup 引用取得。 +func RestoreDispatchWiringFor(config appconfig.Config, scope string, claim objectgateway.Claim) (RestoreDispatchWiring, error) { + if scope != "internal" && scope != "external" { + return RestoreDispatchWiring{}, fmt.Errorf("未知 scope %q", scope) + } + identity := config.LibraryTargetIdentity() + if identity == "" || config.LibraryTargetInstance == "" || config.LibraryProductionDatabaseURL == "" { + return RestoreDispatchWiring{}, errors.New("恢复派发缺少目标身份/实例") + } + if (scope == "internal" && config.LibraryProductionInternalKB == "") || + (scope == "external" && config.LibraryProductionExternalKB == "") { + return RestoreDispatchWiring{}, errors.New("恢复派发缺少 scope 对应的知识库") + } + if claim.JobID == "" || claim.OperationID == "" || claim.Owner == "" || claim.AttemptEpoch <= 0 { + return RestoreDispatchWiring{}, errors.New("恢复派发缺少本次真实 attempt 身份") + } + if (claim.RecoveryJobID == "") != (claim.RecoveryOperationID == "") { + return RestoreDispatchWiring{}, errors.New("恢复投递身份不完整") + } + return RestoreDispatchWiring{ + TargetIdentity: identity, Scope: scope, Instance: config.LibraryTargetInstance, + OperationID: claim.OperationID, WindowJobID: claim.JobID, + AttemptOwner: claim.Owner, AttemptEpoch: claim.AttemptEpoch, + RecoveryJobID: claim.RecoveryJobID, RecoveryOperationID: claim.RecoveryOperationID, + }, nil +} + +func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig, scopeName string, gate libraryrelease.RestoreMarkerGate) (*v071.ProductionLibraryTarget, error) { objects, err := v071.NewS3ObjectStore(scope.s3) if err != nil { @@ -115,16 +157,30 @@ func newTarget(ctx context.Context, config appconfig.Config, scope scopeConfig, // DS01:派发身份只能来自本次真实 attempt(objectgateway claim)与原窗口作业。 // gate 为 nil 时该能力整体关闭,adapter 对新续作 fail closed;baseline 永远传 nil。 claim := objectgateway.ClaimFromContext(ctx) - return v071.NewProductionLibraryTarget(ctx, v071.ProductionLibraryConfig{ + targetConfig := v071.ProductionLibraryConfig{ Postgres: config.LibraryProductionDatabaseURL, SearchURL: config.LibraryProductionElasticURL, SearchIndex: config.LibraryProductionIndex, Objects: writer, ObjectsReader: objects, ArchiveRoot: config.LibraryArchiveRoot, ObjectsBucket: config.LibraryProductionS3Bucket, MaxDocuments: 200000, - RestoreDispatch: gate, RestoreTargetIdentity: config.LibraryTargetIdentity(), - RestoreOperationID: claim.OperationID, RestoreWindowJobID: claim.JobID, - RestoreAttemptOwner: claim.Owner, RestoreAttemptEpoch: claim.AttemptEpoch, - }, scope.target) + RestoreDispatch: gate, + } + if gate != nil { + wiring, err := RestoreDispatchWiringFor(config, scopeName, claim) + if err != nil { + return nil, err + } + targetConfig.RestoreTargetIdentity = wiring.TargetIdentity + targetConfig.RestoreScope = wiring.Scope + targetConfig.RestoreInstance = wiring.Instance + targetConfig.RestoreOperationID = wiring.OperationID + targetConfig.RestoreWindowJobID = wiring.WindowJobID + targetConfig.RestoreAttemptOwner = wiring.AttemptOwner + targetConfig.RestoreAttemptEpoch = wiring.AttemptEpoch + targetConfig.RestoreRecoveryJobID = wiring.RecoveryJobID + targetConfig.RestoreRecoveryOperationID = wiring.RecoveryOperationID + } + return v071.NewProductionLibraryTarget(ctx, targetConfig, scope.target) } func newBackups(config appconfig.Config, scope string, target *v071.ProductionLibraryTarget, @@ -144,7 +200,7 @@ func OpenProductionStack(ctx context.Context, config appconfig.Config, scope, sn if err != nil { return nil, nil, err } - target, err := newTarget(ctx, config, scopeCfg, gate) + target, err := newTarget(ctx, config, scopeCfg, scope, gate) if err != nil { return nil, nil, err } @@ -226,7 +282,7 @@ func OpenBaselineStack(ctx context.Context, config appconfig.Config, scope strin if err != nil { return nil, nil, err } - target, err := newTarget(ctx, config, scopeCfg, nil) + target, err := newTarget(ctx, config, scopeCfg, scope, nil) if err != nil { return nil, nil, err } diff --git a/internal/librarystack/stack_test.go b/internal/librarystack/stack_test.go index a19c982..c2050d4 100644 --- a/internal/librarystack/stack_test.go +++ b/internal/librarystack/stack_test.go @@ -2,9 +2,80 @@ package librarystack import ( "knowledge-governance/internal/appconfig" + "knowledge-governance/internal/objectgateway" "testing" ) +// DS01-F1:正式装配必须把 scope/instance/真实 attempt 身份传到 adapter;快照绑定 +// 不再来自这里(来源/写前镜像只从原 WriteSet 与 backup 引用取得)。 +func TestRestoreDispatchWiringUsesRealClaim(t *testing.T) { + config := appconfig.Config{ + LibraryTargetInstance: "inst-prod", + LibraryProductionDatabaseURL: "postgres://user:pass@127.0.0.1:5432/kg_prod?sslmode=disable", + LibraryProductionInternalKB: "kb-internal", + LibraryProductionExternalKB: "kb-external", + } + claim := objectgateway.Claim{ + JobID: "job-window", OperationID: "op-original", Owner: "worker-1#queue#job-window", + AttemptEpoch: 3, RecoveryJobID: "queue-recovery", RecoveryOperationID: "op-recovery", + } + wiring, err := RestoreDispatchWiringFor(config, "internal", claim) + if err != nil { + t.Fatalf("real wiring: %v", err) + } + if wiring.Scope != "internal" || wiring.Instance != "inst-prod" { + t.Fatalf("scope/instance = %q/%q", wiring.Scope, wiring.Instance) + } + if wiring.TargetIdentity == "" || wiring.TargetIdentity == "unconfigured" { + t.Fatalf("target identity = %q", wiring.TargetIdentity) + } + if wiring.OperationID != claim.OperationID || wiring.WindowJobID != claim.JobID || + wiring.AttemptOwner != claim.Owner || wiring.AttemptEpoch != claim.AttemptEpoch { + t.Fatalf("attempt identity not propagated: %+v", wiring) + } + if wiring.RecoveryJobID != claim.RecoveryJobID || wiring.RecoveryOperationID != claim.RecoveryOperationID { + t.Fatalf("recovery delivery identity not propagated: %+v", wiring) + } + // 自动回滚(非恢复 worker)没有 recovery 投递身份,必须保持为空。 + rollback, err := RestoreDispatchWiringFor(config, "internal", objectgateway.Claim{ + JobID: "job-window", OperationID: "op-original", Owner: "worker-1#queue#job-window", AttemptEpoch: 3}) + if err != nil { + t.Fatal(err) + } + if rollback.RecoveryJobID != "" || rollback.RecoveryOperationID != "" { + t.Fatalf("rollback wiring invented a recovery identity: %+v", rollback) + } +} + +func TestRestoreDispatchWiringFailsClosed(t *testing.T) { + config := appconfig.Config{ + LibraryTargetInstance: "inst-prod", + LibraryProductionDatabaseURL: "postgres://user:pass@127.0.0.1:5432/kg_prod?sslmode=disable", + LibraryProductionInternalKB: "kb-internal", + } + if _, err := RestoreDispatchWiringFor(config, "internal", objectgateway.Claim{}); err == nil { + t.Fatal("missing attempt identity was accepted") + } + if _, err := RestoreDispatchWiringFor(config, "tenant-wide", objectgateway.Claim{ + JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1}); err == nil { + t.Fatal("unknown scope was accepted") + } + if _, err := RestoreDispatchWiringFor(config, "external", objectgateway.Claim{ + JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1}); err == nil { + t.Fatal("external scope without its knowledge base was accepted") + } + if _, err := RestoreDispatchWiringFor(config, "internal", objectgateway.Claim{ + JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1, RecoveryJobID: "q"}); err == nil { + t.Fatal("partial recovery delivery identity was accepted") + } + unconfigured := config + unconfigured.LibraryProductionDatabaseURL = "" + if _, err := RestoreDispatchWiringFor(unconfigured, "internal", objectgateway.Claim{ + JobID: "j", OperationID: "o", Owner: "w", AttemptEpoch: 1}); err == nil { + t.Fatal("unconfigured target database was accepted as a dispatch target") + } +} + func TestObjectAddressingMatchesProvider(t *testing.T) { for _, provider := range []string{"oss", "s3", "minio"} { t.Run(provider, func(t *testing.T) { diff --git a/internal/platform/postgres/library_restore_dispatch.go b/internal/platform/postgres/library_restore_dispatch.go index f044353..4307de3 100644 --- a/internal/platform/postgres/library_restore_dispatch.go +++ b/internal/platform/postgres/library_restore_dispatch.go @@ -19,31 +19,62 @@ import ( // // 关键点不是"多了一张表",而是发送许可的产生方式: // - Prepare 只建立 prepared(不返回许可); -// - Acquire 在**独立短事务**里按 job→operation→active attempt→effect 的顺序取锁, -// 校验当前 epoch/owner/lease 后做 prepared→dispatching 的 CAS,只有 COMMIT 明确 -// 成功才返回一次性许可; +// - Acquire 在**独立短事务**里按 job→operation→activation→active attempt→effect +// 的顺序取锁,校验当前 epoch/owner/lease(数据库时钟)、原作业/operation 锚点、 +// 维护窗口、恢复投递身份与不可变绑定后做 prepared→dispatching 的 CAS,只有 +// COMMIT 明确成功才返回一次性许可; // - Inspect 纯读,永不返回许可; -// - 表本身拒绝状态回退、绑定变化与删行(迁移里的触发器)。 +// - 表本身拒绝状态回退、绑定变化、删行与激活撤销回退(迁移里的触发器)。 // // 外层 WithLibraryExecution 的事务只 rollback、从不 commit,所以这里绝不能用它: // 在那种事务里写 dispatching,发出请求后账会被回滚成"未派发"。 var ErrRestoreDispatchNotActivated = libraryrelease.ErrRestoreDispatchUnavailable -// ActivateRestoreDispatch 登记"该目标已完成旧 writer 退出/排空"的部署证据。 +// ActivateRestoreDispatch 消费一份**可核验的**激活证据文件:文件摘要由调用方给出并 +// 重算、结构完整、每个受控 writer 都登记了退出与排空、证据在有效期内、目标身份/ +// 指纹与治理库身份都与现场一致。任何一项不满足都不建立 active 行。 // -// 它记录的是运维声明,不验证声明为真;代码只能保证没有这条登记就不发放许可。 -// 撤销是终局:重新启用必须换一个新的目标身份(或由 DBA 归档旧行后重新登记), -// 不允许用同一条旧证据把已经撤销的目标悄悄打开。 -func (s *Store) ActivateRestoreDispatch(ctx context.Context, targetIdentity, activatedBy, drainEvidenceDigest string, now time.Time) error { - if targetIdentity == "" || activatedBy == "" || drainEvidenceDigest == "" { - return errors.New("restore dispatch activation requires target, actor and drain evidence digest") +// 代码核验的是证据文件完整性与身份一致性;"writer 确实已退出"仍是部署流程的受控 +// attestation(含退出时间与排空摘要),不是本进程能独立观察到的外部事实。 +func (s *Store) ActivateRestoreDispatch(ctx context.Context, evidencePath, evidenceDigest, + targetIdentity, targetFingerprint string, now time.Time) error { + evidence, err := libraryrelease.LoadRestoreDispatchActivationEvidence(evidencePath, evidenceDigest, + now.UTC(), libraryrelease.RestoreDispatchActivationMaxAge) + if err != nil { + return err + } + if evidence.TargetIdentity != targetIdentity { + return fmt.Errorf("restore dispatch activation evidence is for target %q, not %q", evidence.TargetIdentity, targetIdentity) + } + if evidence.TargetFingerprint != targetFingerprint { + return errors.New("restore dispatch activation evidence does not match the live target fingerprint") + } + var database string + var address string + var port int + if err := s.db.QueryRow(ctx, + `SELECT current_database(), COALESCE(host(inet_server_addr()), ''), COALESCE(inet_server_port(), 0)`). + Scan(&database, &address, &port); err != nil { + return fmt.Errorf("read governance database identity: %w", err) + } + if database == "" || database != evidence.GovernanceDatabase { + return fmt.Errorf("restore dispatch activation evidence names governance database %q, live is %q", + evidence.GovernanceDatabase, database) + } + raw, err := evidence.CanonicalBytes() + if err != nil { + return err } if _, err := s.db.Exec(ctx, ` -INSERT INTO library_index_restore_dispatch_activation (target_identity, protocol_version, activated_by, drain_evidence_digest, activated_at) -VALUES ($1, 1, $2, $3, $4) +INSERT INTO library_index_restore_dispatch_activation + (target_identity, protocol_version, target_fingerprint, target_instance, governance_database, + protocol_build, activated_by, drain_evidence_digest, evidence, attested_at, activated_at) +VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, $10, $11) ON CONFLICT (target_identity) DO NOTHING`, - targetIdentity, activatedBy, drainEvidenceDigest, now.UTC()); err != nil { + evidence.TargetIdentity, evidence.ProtocolVersion, evidence.TargetFingerprint, evidence.TargetInstance, + evidence.GovernanceDatabase, evidence.ProtocolBuild, evidence.AttestedBy, + evidenceDigest, string(raw), evidence.AttestedAt.UTC(), now.UTC()); err != nil { return fmt.Errorf("activate restore dispatch: %w", err) } return s.requireRestoreDispatchActivation(ctx, targetIdentity) @@ -76,6 +107,26 @@ WHERE target_identity=$1 AND protocol_version=1 AND revoked_at IS NULL`, targetI return nil } +// lockRestoreDispatchActivation 在**同一个 CAS 事务**里锁激活行。撤销是同一行的 +// UPDATE:撤销先完成 ⇒ 这里看到 revoked_at;CAS 先取锁 ⇒ 撤销必须等 CAS 提交, +// 之后按"已派发"的窄边界处理。两者只能有一个线性化顺序。 +func lockRestoreDispatchActivation(ctx context.Context, tx pgx.Tx, targetIdentity string) error { + var revoked *time.Time + err := tx.QueryRow(ctx, ` +SELECT revoked_at FROM library_index_restore_dispatch_activation +WHERE target_identity=$1 AND protocol_version=1 FOR UPDATE`, targetIdentity).Scan(&revoked) + if errors.Is(err, pgx.ErrNoRows) { + return fmt.Errorf("%w(目标 %s 没有激活登记)", ErrRestoreDispatchNotActivated, targetIdentity) + } + if err != nil { + return fmt.Errorf("lock restore dispatch activation: %w", err) + } + if revoked != nil { + return fmt.Errorf("%w(目标 %s 的激活已于 %s 撤销)", ErrRestoreDispatchNotActivated, targetIdentity, revoked.UTC().Format(time.RFC3339)) + } + return nil +} + func restoreDispatchBindingDigest(b libraryrelease.RestoreMarkerBinding) (string, error) { return b.Digest() } @@ -117,6 +168,49 @@ WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_diges } } +// UnresolvedRestoreDispatches 返回原 window 的权威台账全集(跨 plan 摘要)。 +// 前置与收尾都查同一份记录:只遍历当前 plan 的文档列表会漏掉计划外/别名 effect。 +func (s *Store) UnresolvedRestoreDispatches(ctx context.Context, targetIdentity, operationID, windowJobID string) ([]libraryrelease.RestoreDispatchRecord, error) { + rows, err := s.db.Query(ctx, ` +SELECT plan_digest, index_name, document_id, state FROM library_index_restore_dispatch +WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 +ORDER BY plan_digest, index_name, document_id`, targetIdentity, operationID, windowJobID) + if err != nil { + return nil, fmt.Errorf("read restore dispatch window: %w", err) + } + defer rows.Close() + var records []libraryrelease.RestoreDispatchRecord + for rows.Next() { + var record libraryrelease.RestoreDispatchRecord + if err := rows.Scan(&record.PlanDigest, &record.Index, &record.DocumentID, &record.State); err != nil { + return nil, err + } + records = append(records, record) + } + return records, rows.Err() +} + +// rejectRestoreDispatchPlanAlias 拒绝同一逻辑 effect 用另一个 plan 摘要另起一行。 +// key 里虽然有 plan_digest,但唯一约束允许"同 target/operation/window/index/doc/phase、 +// 不同 plan"共存;那正是"换计划别名重开证明"的旁路。 +func rejectRestoreDispatchPlanAlias(ctx context.Context, tx pgx.Tx, b libraryrelease.RestoreMarkerBinding) error { + var other string + err := tx.QueryRow(ctx, ` +SELECT plan_digest FROM library_index_restore_dispatch +WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 + AND index_name=$4 AND document_id=$5 AND phase=$6 AND plan_digest<>$7 +LIMIT 1 FOR UPDATE`, + b.Key.TargetIdentity, b.Key.OperationID, b.Key.WindowJobID, + b.Key.Index, b.Key.DocumentID, b.Key.Phase, b.Key.PlanDigest).Scan(&other) + if errors.Is(err, pgx.ErrNoRows) { + return nil + } + if err != nil { + return fmt.Errorf("read restore dispatch plan alias: %w", err) + } + return fmt.Errorf("%w: effect already belongs to another plan digest", libraryrelease.ErrRestoreDispatchHeld) +} + // PrepareRestoreMarker 只建立 prepared:没有发送权,也不把已有 dispatching 抬回 prepared。 func (s *Store) PrepareRestoreMarker(ctx context.Context, b libraryrelease.RestoreMarkerBinding) (libraryrelease.RestoreDispatchStatus, error) { digest, err := restoreDispatchBindingDigest(b) @@ -136,6 +230,12 @@ func (s *Store) PrepareRestoreMarker(ctx context.Context, b libraryrelease.Resto if err := lockRestoreDispatchIdentity(ctx, tx, b); err != nil { return libraryrelease.RestoreDispatchStatus{}, err } + if err := lockRestoreDispatchActivation(ctx, tx, b.Key.TargetIdentity); err != nil { + return libraryrelease.RestoreDispatchStatus{}, err + } + if err := rejectRestoreDispatchPlanAlias(ctx, tx, b); err != nil { + return libraryrelease.RestoreDispatchStatus{}, err + } if _, err := tx.Exec(ctx, ` INSERT INTO library_index_restore_dispatch (target_identity, operation_id, window_job_id, plan_digest, index_name, document_id, phase, binding_digest, state) @@ -168,9 +268,6 @@ func (s *Store) acquireRestoreMarker(ctx context.Context, b libraryrelease.Resto if err != nil { return nil, libraryrelease.RestoreDispatchStatus{}, err } - if err := s.requireRestoreDispatchActivation(ctx, b.Key.TargetIdentity); err != nil { - return nil, libraryrelease.RestoreDispatchStatus{}, err - } tx, err := s.db.Begin(ctx) if err != nil { return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("dispatch restore marker: %w", err) @@ -179,14 +276,19 @@ func (s *Store) acquireRestoreMarker(ctx context.Context, b libraryrelease.Resto if err := lockRestoreDispatchIdentity(ctx, tx, b); err != nil { return nil, libraryrelease.RestoreDispatchStatus{}, err } - // 当前代次必须在租约内,且就是许可申请者:旧 owner 晚到时租约/epoch 已变,零发送。 + // 撤销与 CAS 在同一事务线性化:这里拿激活行锁,撤销的 UPDATE 必须排队。 + if err := lockRestoreDispatchActivation(ctx, tx, b.Key.TargetIdentity); err != nil { + return nil, libraryrelease.RestoreDispatchStatus{}, err + } + // 当前代次必须属于这个原 window 作业,且租户/epoch/租约都仍有效。lease 用 + // clock_timestamp():等锁可能跨越租约到期,事务起始的 now() 不是判据。 var epoch int64 - var owner, attemptState string + var owner, attemptState, attemptJob string var leaseUntil *time.Time err = tx.QueryRow(ctx, ` -SELECT epoch, owner, state, lease_until FROM library_attempts +SELECT epoch, owner, state, job_id, lease_until FROM library_attempts WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.OperationID).Scan( - &epoch, &owner, &attemptState, &leaseUntil) + &epoch, &owner, &attemptState, &attemptJob, &leaseUntil) if errors.Is(err, pgx.ErrNoRows) { return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("%w: no active attempt for operation %s", governance.ErrLibraryEpochStale, b.Key.OperationID) @@ -194,8 +296,13 @@ WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.Oper if err != nil { return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("read active attempt: %w", err) } + if attemptJob != b.Key.WindowJobID { + return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("%w: active attempt belongs to job %s, not the original window %s", + governance.ErrLibraryEpochStale, attemptJob, b.Key.WindowJobID) + } var leaseLive bool - if err := tx.QueryRow(ctx, `SELECT COALESCE($1::timestamptz > now(), false)`, leaseUntil).Scan(&leaseLive); err != nil { + if err := tx.QueryRow(ctx, + `SELECT COALESCE($1::timestamptz > clock_timestamp(), false)`, leaseUntil).Scan(&leaseLive); err != nil { return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("read attempt lease: %w", err) } if owner != b.Owner || epoch != b.Epoch || !leaseLive { @@ -203,6 +310,9 @@ WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.Oper Reason: fmt.Sprintf("attempt %s#%d is not the live authorized owner", b.Owner, b.Epoch)}, fmt.Errorf("%w: attempt %s#%d is not current", governance.ErrLibraryEpochStale, b.Owner, b.Epoch) } + if err := rejectRestoreDispatchPlanAlias(ctx, tx, b); err != nil { + return nil, libraryrelease.RestoreDispatchStatus{}, err + } var id int64 var state, stored string @@ -235,7 +345,7 @@ WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_diges tag, err := tx.Exec(ctx, ` UPDATE library_index_restore_dispatch SET state='dispatching', dispatch_owner=$2, dispatch_epoch=$3, dispatch_nonce=$4, - request_digest=$5, dispatch_committed_at=now(), updated_at=now() + request_digest=$5, dispatch_committed_at=clock_timestamp(), updated_at=clock_timestamp() WHERE id=$1 AND state='prepared'`, id, b.Owner, b.Epoch, nonce, b.RequestDigest) if err != nil { return nil, libraryrelease.RestoreDispatchStatus{}, fmt.Errorf("claim restore dispatch: %w", err) @@ -273,9 +383,6 @@ func (s *Store) ConfirmRestoreMarker(ctx context.Context, b libraryrelease.Resto if err != nil { return err } - if err := s.requireRestoreDispatchActivation(ctx, b.Key.TargetIdentity); err != nil { - return err - } tx, err := s.db.Begin(ctx) if err != nil { return fmt.Errorf("confirm restore marker: %w", err) @@ -284,25 +391,29 @@ func (s *Store) ConfirmRestoreMarker(ctx context.Context, b libraryrelease.Resto if err := lockRestoreDispatchIdentity(ctx, tx, b); err != nil { return err } + if err := lockRestoreDispatchActivation(ctx, tx, b.Key.TargetIdentity); err != nil { + return err + } var epoch int64 - var owner string + var owner, attemptJob string var leaseUntil *time.Time err = tx.QueryRow(ctx, ` -SELECT epoch, owner, lease_until FROM library_attempts -WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.OperationID).Scan(&epoch, &owner, &leaseUntil) +SELECT epoch, owner, job_id, lease_until FROM library_attempts +WHERE operation_id=$1 AND state IN ('claimed','running') FOR UPDATE`, b.Key.OperationID).Scan(&epoch, &owner, &attemptJob, &leaseUntil) if err != nil { return fmt.Errorf("confirm restore marker attempt: %w", err) } var leaseLive bool - if err := tx.QueryRow(ctx, `SELECT COALESCE($1::timestamptz > now(), false)`, leaseUntil).Scan(&leaseLive); err != nil { + if err := tx.QueryRow(ctx, + `SELECT COALESCE($1::timestamptz > clock_timestamp(), false)`, leaseUntil).Scan(&leaseLive); err != nil { return fmt.Errorf("confirm restore marker lease: %w", err) } - if owner != b.Owner || epoch != b.Epoch || !leaseLive { + if attemptJob != b.Key.WindowJobID || owner != b.Owner || epoch != b.Epoch || !leaseLive { return fmt.Errorf("%w: confirmation by a stale owner", governance.ErrLibraryEpochStale) } tag, err := tx.Exec(ctx, ` UPDATE library_index_restore_dispatch -SET state='marker_confirmed', marker_receipt=$1::jsonb, confirmed_at=now(), updated_at=now() +SET state='marker_confirmed', marker_receipt=$1::jsonb, confirmed_at=clock_timestamp(), updated_at=clock_timestamp() WHERE target_identity=$2 AND operation_id=$3 AND window_job_id=$4 AND plan_digest=$5 AND index_name=$6 AND document_id=$7 AND phase=$8 AND state='dispatching' AND binding_digest=$9`, @@ -320,9 +431,18 @@ WHERE target_identity=$2 AND operation_id=$3 AND window_job_id=$4 AND plan_diges return nil } +// lockRestoreDispatchIdentity 以 job→operation 锁序把"这份许可是不是仍属于原作业" +// 钉死在原持久锚点上:目标/scope/来源快照、写集摘要(若已登记)、维护窗口与实例 +// 指纹、恢复投递的 queue/operation 绑定都必须与不可变绑定一致。 func lockRestoreDispatchIdentity(ctx context.Context, tx pgx.Tx, b libraryrelease.RestoreMarkerBinding) error { - var operationID string - err := tx.QueryRow(ctx, `SELECT operation_id FROM library_jobs WHERE id=$1 FOR UPDATE`, b.Key.WindowJobID).Scan(&operationID) + var operationID, scope, jobSource string + var leaseOwner string + var jobEpoch int64 + var jobLease *time.Time + err := tx.QueryRow(ctx, ` +SELECT operation_id, scope, target_snapshot_id, lease_owner, attempt_epoch, lease_until +FROM library_jobs WHERE id=$1 FOR UPDATE`, b.Key.WindowJobID).Scan( + &operationID, &scope, &jobSource, &leaseOwner, &jobEpoch, &jobLease) if errors.Is(err, pgx.ErrNoRows) { return fmt.Errorf("window job %s not found", b.Key.WindowJobID) } @@ -332,9 +452,93 @@ func lockRestoreDispatchIdentity(ctx context.Context, tx pgx.Tx, b libraryreleas if operationID != b.Key.OperationID { return errors.New("window job belongs to another operation") } - if _, err := tx.Exec(ctx, `SELECT id FROM library_operations WHERE id=$1 FOR UPDATE`, b.Key.OperationID); err != nil { + if scope != b.Scope { + return fmt.Errorf("window job scope %s does not match binding %s", scope, b.Scope) + } + if jobSource == "" || jobSource != b.SourceSnapshot { + return fmt.Errorf("window job source snapshot %q does not match binding %q", jobSource, b.SourceSnapshot) + } + if leaseOwner != b.Owner || jobEpoch != b.Epoch || jobLease == nil { + return fmt.Errorf("%w: window job lease is not held by attempt %s#%d", governance.ErrLibraryEpochStale, b.Owner, b.Epoch) + } + var jobLeaseLive bool + if err := tx.QueryRow(ctx, `SELECT COALESCE($1::timestamptz > clock_timestamp(), false)`, jobLease).Scan(&jobLeaseLive); err != nil { + return fmt.Errorf("read window job lease: %w", err) + } + if !jobLeaseLive { + return fmt.Errorf("%w: window job lease expired while waiting for the dispatch CAS", governance.ErrLibraryEpochStale) + } + + var opScope, opKind, opSource, requestDigest, writesetDigest, firstBackup string + err = tx.QueryRow(ctx, ` +SELECT scope, kind, target_snapshot_id, request_digest, COALESCE(writeset_digest,''), COALESCE(first_backup_snapshot_id,'') +FROM library_operations WHERE id=$1 FOR UPDATE`, b.Key.OperationID).Scan( + &opScope, &opKind, &opSource, &requestDigest, &writesetDigest, &firstBackup) + if errors.Is(err, pgx.ErrNoRows) { + return fmt.Errorf("operation %s not found", b.Key.OperationID) + } + if err != nil { return fmt.Errorf("lock operation: %w", err) } + if opScope != b.Scope { + return fmt.Errorf("operation scope %s does not match binding %s", opScope, b.Scope) + } + if opSource == "" || opSource != b.SourceSnapshot { + return fmt.Errorf("operation source snapshot %q does not match binding %q", opSource, b.SourceSnapshot) + } + if requestDigest == "" { + return errors.New("operation has no request digest anchor") + } + if writesetDigest != "" && writesetDigest != b.Key.PlanDigest { + return fmt.Errorf("operation plan digest %s does not match binding %s", writesetDigest, b.Key.PlanDigest) + } + if firstBackup != "" && firstBackup != b.BackupSnapshot { + return fmt.Errorf("operation backup snapshot %s does not match binding %s", firstBackup, b.BackupSnapshot) + } + _ = opKind + + if b.TargetInstance != "" { + var ownerJob, ownerScope string + var released *time.Time + var state string + err = tx.QueryRow(ctx, ` +SELECT owner_job_id, owner_scope, state, released_at +FROM library_instance_maintenance WHERE instance=$1 FOR UPDATE`, b.TargetInstance).Scan( + &ownerJob, &ownerScope, &state, &released) + if errors.Is(err, pgx.ErrNoRows) { + return fmt.Errorf("target instance %s has no maintenance window", b.TargetInstance) + } + if err != nil { + return fmt.Errorf("lock instance maintenance: %w", err) + } + if released != nil || state == "released" { + return errors.New("target instance maintenance window is already released") + } + if ownerJob != b.Key.WindowJobID || ownerScope != b.Scope { + return fmt.Errorf("maintenance window belongs to %s (scope=%s), not window %s", ownerJob, ownerScope, b.Key.WindowJobID) + } + } + + if b.RecoveryJobID != "" { + var queueState string + var lease string + if err := tx.QueryRow(ctx, `SELECT state, COALESCE(lease_owner,'') FROM jobs WHERE id=$1`, b.RecoveryJobID). + Scan(&queueState, &lease); err != nil { + return fmt.Errorf("read recovery queue job: %w", err) + } + if queueState != "running" || lease == "" { + return fmt.Errorf("recovery queue job %s is not a live delivery", b.RecoveryJobID) + } + var recoveryKind, recoveryScope, recoveryBackup string + if err := tx.QueryRow(ctx, ` +SELECT kind, scope, target_snapshot_id FROM library_operations WHERE id=$1`, b.RecoveryOperationID). + Scan(&recoveryKind, &recoveryScope, &recoveryBackup); err != nil { + return fmt.Errorf("read recovery operation: %w", err) + } + if recoveryKind != string(governance.LibraryJobRecover) || recoveryScope != b.Scope || recoveryBackup != b.BackupSnapshot { + return fmt.Errorf("recovery operation %s is not bound to this window and backup", b.RecoveryOperationID) + } + } return nil } diff --git a/internal/platform/postgres/library_restore_dispatch_g1_test.go b/internal/platform/postgres/library_restore_dispatch_g1_test.go new file mode 100644 index 0000000..8d3c193 --- /dev/null +++ b/internal/platform/postgres/library_restore_dispatch_g1_test.go @@ -0,0 +1,255 @@ +package postgres + +import ( + "context" + "encoding/binary" + "errors" + "io" + "net" + "net/url" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/jackc/pgx/v5/pgxpool" + + "knowledge-governance/internal/libraryrelease" +) + +// G1:真实 PG 的"COMMIT 已到服务端、客户端收不到成功回包"。beforeCommit 钩子做不到 +// 这一点(它根本没调用 Commit)。这里用专用 TCP 转发代理:看到客户端 commit 帧后 +// 原样转发,然后立刻切断连接——服务端已提交,客户端拿到的是错误。 + +func startCommitAckDropProxy(t *testing.T, upstreamHostPort string) (string, *sync.WaitGroup) { + t.Helper() + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + for { + client, err := listener.Accept() + if err != nil { + return + } + go func() { + defer client.Close() + server, err := net.Dial("tcp", upstreamHostPort) + if err != nil { + return + } + defer server.Close() + var once sync.Once + go func() { + // 服务端 -> 客户端:正常转发;代理被关闭后自然结束。 + _, _ = io.Copy(client, server) + }() + buffer := make([]byte, 32*1024) + for { + n, err := client.Read(buffer) + if n > 0 { + chunk := buffer[:n] + drop := onceFrames(chunk, &once) + if _, werr := server.Write(chunk); werr != nil { + return + } + if drop { + // commit 已经交给服务端;不给客户端任何成功回包。 + _ = client.Close() + return + } + } + if err != nil { + return + } + } + }() + } + }() + t.Cleanup(func() { + _ = listener.Close() + wg.Wait() + }) + return "127.0.0.1:" + strings.Split(listener.Addr().String(), ":")[1], nil +} + +// onceFrames 在客户端字节流里找简单查询帧 'Q' 且语句以 commit 开头。找到一次就返回 +// true(后续帧不再触发)。 +func onceFrames(chunk []byte, once *sync.Once) bool { + for i := 0; i+5 <= len(chunk); i++ { + if chunk[i] != 'Q' { + continue + } + length := int(binary.BigEndian.Uint32(chunk[i+1 : i+5])) + if length < 5 || i+1+length > len(chunk) { + continue + } + sql := strings.TrimSpace(strings.ToLower(string(chunk[i+5 : i+1+length]))) + if strings.HasPrefix(sql, "commit") { + triggered := false + once.Do(func() { triggered = true }) + return triggered + } + } + return false +} + +func proxyStore(t *testing.T, direct *pgxpool.Pool) (*Store, *pgxpool.Pool) { + t.Helper() + conn := direct.Config().ConnConfig + address, _ := startCommitAckDropProxy(t, net.JoinHostPort(conn.Host, strconv.Itoa(int(conn.Port)))) + target := url.URL{Scheme: "postgres", User: url.UserPassword(conn.User, conn.Password), + Host: address, Path: "/" + conn.Database} + query := target.Query() + for key, value := range conn.RuntimeParams { + query.Set(key, value) + } + query.Set("sslmode", "disable") + target.RawQuery = query.Encode() + pool, err := pgxpool.New(context.Background(), target.String()) + if err != nil { + t.Fatal(err) + } + t.Cleanup(pool.Close) + return NewStore(pool), pool +} + +func TestRestoreDispatchCommitAckLossKeepsDispatchingAndNoPermit(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + seedOperationJob(t, store, pool, "internal", "libsync-ds01-ackloss", "sync", now) + if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-ackloss", "worker-a#q1#0", 10*time.Minute, now); err != nil { + t.Fatal(err) + } + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + binding := ds01Binding() + binding.Key.WindowJobID = "libsync-ds01-ackloss" + binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil { + t.Fatal(err) + } + + proxied, _ := proxyStore(t, pool) + permit, status, err := proxied.AcquireRestoreMarker(ctx, binding) + if err == nil { + t.Fatal("commit acknowledgement loss reported success") + } + if permit != nil { + t.Fatal("a send permit was issued although the client never saw COMMIT succeed") + } + if status.State != libraryrelease.RestoreDispatchDispatching || !status.Held { + t.Fatalf("status after ack loss = %+v", status) + } + // 服务端确实已提交:换一条直连读回 durable dispatching。 + time.Sleep(300 * time.Millisecond) + var state string + if err := pool.QueryRow(ctx, `SELECT state FROM library_index_restore_dispatch`).Scan(&state); err != nil { + t.Fatal(err) + } + if state != libraryrelease.RestoreDispatchDispatching { + t.Fatalf("server-side state after committed-but-unacknowledged commit = %s", state) + } + // 下一次调用(直连)不能把 dispatching 当 prepared 再发一次。 + permit, _, err = store.AcquireRestoreMarker(ctx, binding) + if permit != nil || !errors.Is(err, libraryrelease.ErrRestoreDispatchHeld) { + t.Fatalf("next call after ack loss = permit=%v err=%v", permit, err) + } +} + +func TestRestoreDispatchUndecidedTransactionBlocksThenFollowsRealOutcome(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + seedOperationJob(t, store, pool, "internal", "libsync-ds01-undecided", "sync", now) + if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-undecided", "worker-a#q1#0", 10*time.Minute, now); err != nil { + t.Fatal(err) + } + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + binding := ds01Binding() + binding.Key.WindowJobID = "libsync-ds01-undecided" + binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil { + t.Fatal(err) + } + + // 原事务仍未决:另一条连接已经把行推进到 dispatching,但没有 commit。 + open, err := pool.Begin(ctx) + if err != nil { + t.Fatal(err) + } + if _, err := open.Exec(ctx, ` +UPDATE library_index_restore_dispatch +SET state='dispatching', dispatch_owner='old', dispatch_epoch=1, dispatch_nonce='old-nonce', + request_digest='old-request', dispatch_committed_at=clock_timestamp() +WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_digest=$4 + AND index_name=$5 AND document_id=$6 AND phase=$7`, + binding.Key.TargetIdentity, binding.Key.OperationID, binding.Key.WindowJobID, binding.Key.PlanDigest, + binding.Key.Index, binding.Key.DocumentID, binding.Key.Phase); err != nil { + t.Fatal(err) + } + type result struct { + permit *libraryrelease.RestoreSendPermit + err error + } + done := make(chan result, 1) + go func() { + permit, _, err := store.AcquireRestoreMarker(ctx, binding) + done <- result{permit, err} + }() + select { + case got := <-done: + t.Fatalf("acquire returned while the original transaction was undecided: permit=%v err=%v", got.permit, got.err) + case <-time.After(700 * time.Millisecond): + } + if err := open.Rollback(ctx); err != nil { + t.Fatal(err) + } + got := <-done + if got.err != nil || got.permit == nil { + t.Fatalf("after a real rollback the call should win once: permit=%v err=%v", got.permit, got.err) + } + + // 第二个 effect:原事务选择 COMMIT,后续调用必须看到 dispatching 而不是 prepared。 + second := binding + second.Key.DocumentID = "doc-2" + if _, err := store.PrepareRestoreMarker(ctx, second); err != nil { + t.Fatal(err) + } + open2, err := pool.Begin(ctx) + if err != nil { + t.Fatal(err) + } + if _, err := open2.Exec(ctx, ` +UPDATE library_index_restore_dispatch +SET state='dispatching', dispatch_owner='old', dispatch_epoch=1, dispatch_nonce='old-nonce-2', + request_digest='old-request-2', dispatch_committed_at=clock_timestamp() +WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_digest=$4 + AND index_name=$5 AND document_id=$6 AND phase=$7`, + second.Key.TargetIdentity, second.Key.OperationID, second.Key.WindowJobID, second.Key.PlanDigest, + second.Key.Index, second.Key.DocumentID, second.Key.Phase); err != nil { + t.Fatal(err) + } + done2 := make(chan result, 1) + go func() { + permit, _, err := store.AcquireRestoreMarker(ctx, second) + done2 <- result{permit, err} + }() + time.Sleep(500 * time.Millisecond) + if err := open2.Commit(ctx); err != nil { + t.Fatal(err) + } + got2 := <-done2 + if got2.permit != nil || !errors.Is(got2.err, libraryrelease.ErrRestoreDispatchHeld) { + t.Fatalf("after a real commit the call must not treat dispatching as prepared: permit=%v err=%v", got2.permit, got2.err) + } +} diff --git a/internal/platform/postgres/library_restore_dispatch_linearization_test.go b/internal/platform/postgres/library_restore_dispatch_linearization_test.go new file mode 100644 index 0000000..626dc3d --- /dev/null +++ b/internal/platform/postgres/library_restore_dispatch_linearization_test.go @@ -0,0 +1,401 @@ +package postgres + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "os" + "strings" + "testing" + "time" + + "knowledge-governance/internal/governance" + "knowledge-governance/internal/libraryrelease" +) + +// DS01-F2:授权必须在**同一个 CAS 事务**里与撤销线性化,lease 必须用数据库实时 +// 时钟。以下都用真 PG 的锁等待,不用 scripted fake。 + +// 撤销先完成:Acquire 已经在事务里排队等 job 锁,撤销此时提交。CAS 不得再发许可。 +func TestRestoreDispatchRevocationWinsBeforeCASIsRejected(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + seedOperationJob(t, store, pool, "internal", "libsync-ds01-revoke-race", "sync", now) + if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-revoke-race", "worker-a#q1#0", 10*time.Minute, now); err != nil { + t.Fatal(err) + } + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + binding := ds01Binding() + binding.Key.WindowJobID = "libsync-ds01-revoke-race" + binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil { + t.Fatal(err) + } + + lock, err := pool.Begin(ctx) + if err != nil { + t.Fatal(err) + } + if _, err := lock.Exec(ctx, `SELECT id FROM library_jobs WHERE id=$1 FOR UPDATE`, binding.Key.WindowJobID); err != nil { + t.Fatal(err) + } + type result struct { + permit *libraryrelease.RestoreSendPermit + err error + } + done := make(chan result, 1) + go func() { + permit, _, err := store.AcquireRestoreMarker(ctx, binding) + done <- result{permit, err} + }() + time.Sleep(400 * time.Millisecond) + if err := store.RevokeRestoreDispatch(ctx, "target-ds01", now); err != nil { + t.Fatalf("revoke: %v", err) + } + if err := lock.Rollback(ctx); err != nil { + t.Fatal(err) + } + got := <-done + if got.permit != nil { + t.Fatal("revocation committed before the CAS but a send permit was still issued") + } + if !errors.Is(got.err, libraryrelease.ErrRestoreDispatchUnavailable) { + t.Fatalf("revocation race error = %v", got.err) + } + var state string + if err := pool.QueryRow(ctx, `SELECT state FROM library_index_restore_dispatch`).Scan(&state); err != nil { + t.Fatal(err) + } + if state != libraryrelease.RestoreDispatchPrepared { + t.Fatalf("row advanced to %s after a winning revocation", state) + } +} + +// CAS 先线性化:Acquire 已经锁住激活行、正在等 effect 行;撤销必须排队。CAS 提交后 +// 许可属于一次已派发,第二次调用必须 held。 +func TestRestoreDispatchCASWinsBeforeRevocationIssuesOnePermit(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + seedOperationJob(t, store, pool, "internal", "libsync-ds01-cas-race", "sync", now) + if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-cas-race", "worker-a#q1#0", 10*time.Minute, now); err != nil { + t.Fatal(err) + } + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + binding := ds01Binding() + binding.Key.WindowJobID = "libsync-ds01-cas-race" + binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil { + t.Fatal(err) + } + + lock, err := pool.Begin(ctx) + if err != nil { + t.Fatal(err) + } + if _, err := lock.Exec(ctx, `SELECT id FROM library_index_restore_dispatch WHERE target_identity=$1 AND operation_id=$2 AND window_job_id=$3 AND plan_digest=$4 AND index_name=$5 AND document_id=$6 AND phase=$7 FOR UPDATE`, + binding.Key.TargetIdentity, binding.Key.OperationID, binding.Key.WindowJobID, binding.Key.PlanDigest, + binding.Key.Index, binding.Key.DocumentID, binding.Key.Phase); err != nil { + t.Fatal(err) + } + type result struct { + permit *libraryrelease.RestoreSendPermit + err error + } + acquired := make(chan result, 1) + go func() { + permit, _, err := store.AcquireRestoreMarker(ctx, binding) + acquired <- result{permit, err} + }() + time.Sleep(400 * time.Millisecond) + revoked := make(chan error, 1) + go func() { revoked <- store.RevokeRestoreDispatch(ctx, "target-ds01", now) }() + time.Sleep(400 * time.Millisecond) + if err := lock.Commit(ctx); err != nil { + t.Fatal(err) + } + got := <-acquired + if got.err != nil || got.permit == nil { + t.Fatalf("CAS should have won: permit=%v err=%v", got.permit, got.err) + } + if err := <-revoked; err != nil { + t.Fatalf("revocation after CAS: %v", err) + } + permit, _, err := store.AcquireRestoreMarker(ctx, binding) + if permit != nil || err == nil { + t.Fatalf("second acquire after dispatching = permit=%v err=%v", permit, err) + } + var state string + if err := pool.QueryRow(ctx, `SELECT state FROM library_index_restore_dispatch`).Scan(&state); err != nil { + t.Fatal(err) + } + if state != libraryrelease.RestoreDispatchDispatching { + t.Fatalf("state = %s", state) + } +} + +// 等锁跨过租约:事务开始时刻仍有效、真正做 CAS 时已过期。必须用数据库实时时钟拒绝。 +func TestRestoreDispatchLeaseExpiryWhileWaitingForLockIsRejected(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + seedOperationJob(t, store, pool, "internal", "libsync-ds01-lease", "sync", now) + if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-lease", "worker-a#q1#0", 500*time.Millisecond, now); err != nil { + t.Fatal(err) + } + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + binding := ds01Binding() + binding.Key.WindowJobID = "libsync-ds01-lease" + binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil { + t.Fatal(err) + } + lock, err := pool.Begin(ctx) + if err != nil { + t.Fatal(err) + } + if _, err := lock.Exec(ctx, `SELECT id FROM library_jobs WHERE id=$1 FOR UPDATE`, binding.Key.WindowJobID); err != nil { + t.Fatal(err) + } + type result struct { + permit *libraryrelease.RestoreSendPermit + err error + } + done := make(chan result, 1) + go func() { + permit, _, err := store.AcquireRestoreMarker(ctx, binding) + done <- result{permit, err} + }() + time.Sleep(900 * time.Millisecond) + if err := lock.Rollback(ctx); err != nil { + t.Fatal(err) + } + got := <-done + if got.permit != nil { + t.Fatal("permit issued although the attempt lease expired while waiting for the job lock") + } + if got.err == nil || (!strings.Contains(got.err.Error(), "not current") && + !strings.Contains(got.err.Error(), "lease expired")) { + t.Fatalf("expired lease error = %v", got.err) + } +} + +// 错目标/错来源/换 plan 别名/错 UUID 都必须在 CAS 前失败,且台账不动。 +func TestRestoreDispatchRejectsWrongAnchorsAndPlanAlias(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + seedOperationJob(t, store, pool, "internal", "libsync-ds01-anchors", "sync", now) + if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-anchors", "worker-a#q1#0", 10*time.Minute, now); err != nil { + t.Fatal(err) + } + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + binding := ds01Binding() + binding.Key.WindowJobID = "libsync-ds01-anchors" + binding.Key.OperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, binding); err != nil { + t.Fatal(err) + } + + wrongSource := binding + wrongSource.SourceSnapshot = "snap-other" + if permit, _, err := store.AcquireRestoreMarker(ctx, wrongSource); permit != nil || err == nil { + t.Fatalf("wrong source acquired: permit=%v err=%v", permit, err) + } + wrongUUID := binding + wrongUUID.IndexUUID = "uuid-replaced" + if permit, _, err := store.AcquireRestoreMarker(ctx, wrongUUID); permit != nil || + !errors.Is(err, libraryrelease.ErrRestoreDispatchHeld) { + t.Fatalf("replaced index uuid acquired: permit=%v err=%v", permit, err) + } + wrongFingerprint := binding + wrongFingerprint.TargetFingerprint = "fingerprint-other" + if permit, _, err := store.AcquireRestoreMarker(ctx, wrongFingerprint); permit != nil || err == nil { + t.Fatalf("replaced target fingerprint acquired: permit=%v err=%v", permit, err) + } + alias := binding + alias.Key.PlanDigest = "plan-alias" + if _, err := store.PrepareRestoreMarker(ctx, alias); err == nil || + !strings.Contains(err.Error(), "another plan") { + t.Fatalf("plan alias was accepted: %v", err) + } + var count int + if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_index_restore_dispatch`).Scan(&count); err != nil { + t.Fatal(err) + } + if count != 1 { + t.Fatalf("dispatch rows = %d, want exactly the original plan row", count) + } + // 恢复投递身份缺失/错误:Prepare 就必须失败,不能等到发送前。 + recovery := binding + recovery.RecoveryJobID = "queue-missing" + recovery.RecoveryOperationID = "op-missing" + if _, err := store.PrepareRestoreMarker(ctx, recovery); err == nil { + t.Fatal("unknown recovery delivery identity was accepted") + } + // 真实的 recovery queue/operation 绑定可以被验证通过;换成另一份 backup 的 + // recovery operation 则必须拒绝。 + if _, err := pool.Exec(ctx, `INSERT INTO jobs (id, type, payload, state, lease_owner, idempotency_key) +VALUES ('queue-recovery-1','library.recover','{}'::jsonb,'running','worker-a','queue-recovery-1')`); err != nil { + t.Fatal(err) + } + if _, _, err := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{ + ID: "op-recovery-1", Scope: "internal", Kind: governance.LibraryJobRecover, + TargetSnapshotID: "snap-backup", RequestDigest: "digest-recovery", OperationKey: "recovery-1", + Actor: "tester", CreatedAt: now.UTC(), + }); err != nil { + t.Fatal(err) + } + validRecovery := binding + validRecovery.Key.DocumentID = "doc-recovery" + validRecovery.RecoveryJobID = "queue-recovery-1" + validRecovery.RecoveryOperationID = "op-recovery-1" + if _, err := store.PrepareRestoreMarker(ctx, validRecovery); err != nil { + t.Fatalf("valid recovery delivery binding: %v", err) + } + wrongRecovery := validRecovery + wrongRecovery.Key.DocumentID = "doc-recovery-2" + wrongRecovery.RecoveryOperationID = seededOperationID(t, pool, binding.Key.WindowJobID) + if _, err := store.PrepareRestoreMarker(ctx, wrongRecovery); err == nil { + t.Fatal("recovery operation for another intent was accepted") + } +} + +// DS01-F3:撤销是终局——SQL 不能把 revoked_at 改回 NULL,也不能删掉整行;down 在 +// 只有 activation 证据时同样拒绝。 +func TestRestoreDispatchActivationCannotBeResetOrDeleted(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + now := time.Now().UTC() + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { + t.Fatal(err) + } + if err := store.RevokeRestoreDispatch(ctx, "target-ds01", now); err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, `UPDATE library_index_restore_dispatch_activation SET revoked_at=NULL WHERE target_identity=$1`, "target-ds01"); err == nil { + t.Fatal("revoked_at was reset to NULL") + } + if _, err := pool.Exec(ctx, `DELETE FROM library_index_restore_dispatch_activation WHERE target_identity=$1`, "target-ds01"); err == nil { + t.Fatal("activation row was deleted") + } + down, err := os.ReadFile("migrations/000078_library_restore_dispatch.down.sql") + if err != nil { + t.Fatal(err) + } + if _, err := pool.Exec(ctx, string(down)); err == nil || + !strings.Contains(err.Error(), "activation evidence") { + t.Fatalf("down with activation-only evidence = %v", err) + } + var revoked *time.Time + if err := pool.QueryRow(ctx, `SELECT revoked_at FROM library_index_restore_dispatch_activation WHERE target_identity=$1`, "target-ds01").Scan(&revoked); err != nil { + t.Fatal(err) + } + if revoked == nil { + t.Fatal("revocation evidence did not survive") + } +} + +// 激活只接受结构与身份都可核验的证据文件。 +func TestRestoreDispatchActivationRequiresVerifiableEvidence(t *testing.T) { + store, pool := newIdentityStore(t) + ctx := context.Background() + var database string + if err := pool.QueryRow(ctx, `SELECT current_database()`).Scan(&database); err != nil { + t.Fatal(err) + } + now := time.Now().UTC() + exited := now.Add(-time.Minute) + valid := libraryrelease.RestoreDispatchActivationEvidence{ + ProtocolVersion: 1, TargetIdentity: "target-ds01", TargetFingerprint: "fingerprint-1", + TargetInstance: "inst-1", GovernanceDatabase: database, ProtocolBuild: "build-1", + AttestedBy: "deploy-bot", AttestedAt: now, + Writers: []libraryrelease.RestoreDispatchWriterDrain{{ + Name: "worker-1", Kind: "worker", Stopped: true, ExitedAt: &exited, + QueueDrained: true, DrainEvidence: "sha256:drain", + }}, + } + write := func(t *testing.T, mutate func(*libraryrelease.RestoreDispatchActivationEvidence), digest string) (string, string) { + t.Helper() + raw, err := libraryrelease.RestoreDispatchActivationEvidence(valid).CanonicalBytes() + if err != nil { + t.Fatal(err) + } + if mutate != nil { + var copy libraryrelease.RestoreDispatchActivationEvidence + if err := json.Unmarshal(raw, ©); err != nil { + t.Fatal(err) + } + mutate(©) + raw, err = copy.CanonicalBytes() + if err != nil { + t.Fatal(err) + } + } + path := t.TempDir() + "/evidence.json" + if err := os.WriteFile(path, raw, 0o600); err != nil { + t.Fatal(err) + } + if digest == "" { + sum := sha256.Sum256(raw) + digest = hex.EncodeToString(sum[:]) + } + return path, digest + } + // 错摘要 + path, _ := write(t, nil, "deadbeef") + if err := store.ActivateRestoreDispatch(ctx, path, "deadbeef", "target-ds01", "fingerprint-1", now); err == nil { + t.Fatal("wrong evidence digest was accepted") + } + // 错目标指纹 + path, digest := write(t, nil, "") + if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-other", now); err == nil { + t.Fatal("evidence for another target fingerprint was accepted") + } + // 未排空 + path, digest = write(t, func(e *libraryrelease.RestoreDispatchActivationEvidence) { + e.Writers[0].QueueDrained = false + }, "") + if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err == nil { + t.Fatal("evidence with an undrained writer was accepted") + } + // 治理库身份不符 + path, digest = write(t, func(e *libraryrelease.RestoreDispatchActivationEvidence) { + e.GovernanceDatabase = "some-other-database" + }, "") + if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err == nil { + t.Fatal("evidence for another governance database was accepted") + } + // 过期 + path, digest = write(t, func(e *libraryrelease.RestoreDispatchActivationEvidence) { + old := now.Add(-24 * time.Hour) + e.AttestedAt = old + e.Writers[0].ExitedAt = &old + }, "") + if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err == nil { + t.Fatal("stale evidence was accepted") + } + var count int + if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_index_restore_dispatch_activation`).Scan(&count); err != nil { + t.Fatal(err) + } + if count != 0 { + t.Fatalf("invalid evidence created %d activation rows", count) + } + // 有效证据 + path, digest = write(t, nil, "") + if err := store.ActivateRestoreDispatch(ctx, path, digest, "target-ds01", "fingerprint-1", now); err != nil { + t.Fatalf("valid evidence rejected: %v", err) + } +} diff --git a/internal/platform/postgres/library_restore_dispatch_test.go b/internal/platform/postgres/library_restore_dispatch_test.go index dbe4843..8f7a914 100644 --- a/internal/platform/postgres/library_restore_dispatch_test.go +++ b/internal/platform/postgres/library_restore_dispatch_test.go @@ -2,7 +2,12 @@ package postgres import ( "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" "errors" + "os" + "path/filepath" "strings" "sync" "testing" @@ -33,7 +38,8 @@ func ds01Binding() libraryrelease.RestoreMarkerBinding { PlanDigest: "plan-1", Index: "kg-index", DocumentID: "doc-1", Phase: libraryrelease.RestoreMarkerPhaseCreate, }, - Scope: "internal", SourceSnapshot: "snap-src", BackupSnapshot: "snap-backup", + Scope: "internal", SourceSnapshot: "snap-target", BackupSnapshot: "snap-backup", + TargetInstance: "inst-1", TargetFingerprint: "fingerprint-1", IndexUUID: "uuid-1", ChangeDigest: "change-1", MarkerDigest: "marker-1", RequestDigest: "request-1", Owner: "worker-a#q1#0", Epoch: 1, } @@ -63,7 +69,7 @@ func TestRestoreDispatchPrepareIsUniqueAndBindingBound(t *testing.T) { if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-prep", "worker-a#q1#0", 10*time.Minute, now); err != nil { t.Fatal(err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { t.Fatal(err) } binding := ds01Binding() @@ -101,7 +107,7 @@ func TestRestoreDispatchConcurrentAcquireHasSingleWinner(t *testing.T) { if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-race", "worker-a#q1#0", 10*time.Minute, now); err != nil { t.Fatal(err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { t.Fatal(err) } binding := ds01Binding() @@ -156,7 +162,7 @@ func TestRestoreDispatchIsMonotoneAndAppendOnly(t *testing.T) { if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-mono", "worker-a#q1#0", 10*time.Minute, now); err != nil { t.Fatal(err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { t.Fatal(err) } binding := ds01Binding() @@ -211,7 +217,7 @@ func TestRestoreDispatchRejectsStaleEpochBeforeSend(t *testing.T) { if err != nil { t.Fatal(err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { t.Fatal(err) } binding := ds01Binding() @@ -258,7 +264,7 @@ func TestRestoreDispatchCommitWithoutExplicitSuccessYieldsNoPermit(t *testing.T) if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-commit", "worker-a#q1#0", 10*time.Minute, now); err != nil { t.Fatal(err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { t.Fatal(err) } binding := ds01Binding() @@ -310,7 +316,7 @@ func TestRestoreDispatchRevocationIsFinal(t *testing.T) { if _, err := store.BeginLibraryAttempt(ctx, "libsync-ds01-revoke", "worker-a#q1#0", 10*time.Minute, now); err != nil { t.Fatal(err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err != nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err != nil { t.Fatal(err) } if err := store.RevokeRestoreDispatch(ctx, "target-ds01", now); err != nil { @@ -322,7 +328,7 @@ func TestRestoreDispatchRevocationIsFinal(t *testing.T) { if _, err := store.PrepareRestoreMarker(ctx, binding); !errors.Is(err, libraryrelease.ErrRestoreDispatchUnavailable) { t.Fatalf("revoked target still prepared: %v", err) } - if err := store.ActivateRestoreDispatch(ctx, "target-ds01", "deploy-bot", "sha256:drain", now); err == nil { + if err := activateRestoreDispatchForTest(t, store, "target-ds01", "fingerprint-1"); err == nil { t.Fatal("revoked activation was reopened with the same evidence") } var revoked *time.Time @@ -333,3 +339,38 @@ func TestRestoreDispatchRevocationIsFinal(t *testing.T) { t.Fatal("revocation was silently cleared") } } + +// activateRestoreDispatchForTest 写一份结构完整的激活证据并登记到隔离治理库。 +// 治理库名取自当前连接:激活 API 会与现场库身份比对。 +func activateRestoreDispatchForTest(t *testing.T, store *Store, targetIdentity, fingerprint string) error { + t.Helper() + ctx := context.Background() + var database string + if err := store.db.QueryRow(ctx, `SELECT current_database()`).Scan(&database); err != nil { + t.Fatalf("read governance database: %v", err) + return err + } + now := time.Now().UTC() + exited := now.Add(-time.Minute) + evidence := libraryrelease.RestoreDispatchActivationEvidence{ + ProtocolVersion: 1, TargetIdentity: targetIdentity, TargetFingerprint: fingerprint, + TargetInstance: "inst-1", GovernanceDatabase: database, ProtocolBuild: "test-build", + AttestedBy: "ds01-test", AttestedAt: now, + Writers: []libraryrelease.RestoreDispatchWriterDrain{{ + Name: "worker", Kind: "worker", Stopped: true, ExitedAt: &exited, + QueueDrained: true, DrainEvidence: "sha256:drain", + }}, + } + raw, err := json.Marshal(evidence) + if err != nil { + t.Fatal(err) + return err + } + path := filepath.Join(t.TempDir(), "activation.json") + if err := os.WriteFile(path, raw, 0o600); err != nil { + t.Fatal(err) + return err + } + sum := sha256.Sum256(raw) + return store.ActivateRestoreDispatch(ctx, path, hex.EncodeToString(sum[:]), targetIdentity, fingerprint, now) +} diff --git a/internal/platform/postgres/migrations/000078_library_restore_dispatch.down.sql b/internal/platform/postgres/migrations/000078_library_restore_dispatch.down.sql index 83f0c5a..fdb5094 100644 --- a/internal/platform/postgres/migrations/000078_library_restore_dispatch.down.sql +++ b/internal/platform/postgres/migrations/000078_library_restore_dispatch.down.sql @@ -1,10 +1,17 @@ --- 已有派发记录时拒绝自动回滚:这些行是"是否发送过"的唯一证明, --- 用 down 抹掉等于把 unknown 重新伪装成 prepared。人工处置必须先导出并存档。 +-- 已有派发记录或激活/撤销证据时拒绝自动回滚:这些行是"是否发送过 / 是否曾被授权" +-- 的唯一证明,用 down 抹掉等于把 unknown 重新伪装成 prepared,或把已撤销的激活 +-- 悄悄删掉。人工处置必须先导出并存档。仅有一条已撤销 activation、尚无任何 effect +-- 行时同样拒绝——撤销事实本身也是证据。 DO $$ BEGIN - IF EXISTS (SELECT 1 FROM library_index_restore_dispatch LIMIT 1) THEN + IF to_regclass('library_index_restore_dispatch') IS NOT NULL + AND EXISTS (SELECT 1 FROM library_index_restore_dispatch LIMIT 1) THEN RAISE EXCEPTION 'refusing to drop restore dispatch evidence with recorded rows'; END IF; + IF to_regclass('library_index_restore_dispatch_activation') IS NOT NULL + AND EXISTS (SELECT 1 FROM library_index_restore_dispatch_activation LIMIT 1) THEN + RAISE EXCEPTION 'refusing to drop restore dispatch activation evidence'; + END IF; END $$; DROP TRIGGER IF EXISTS library_index_restore_dispatch_monotone ON library_index_restore_dispatch; diff --git a/internal/platform/postgres/migrations/000078_library_restore_dispatch.up.sql b/internal/platform/postgres/migrations/000078_library_restore_dispatch.up.sql index a8af629..0f74e21 100644 --- a/internal/platform/postgres/migrations/000078_library_restore_dispatch.up.sql +++ b/internal/platform/postgres/migrations/000078_library_restore_dispatch.up.sql @@ -67,26 +67,46 @@ CREATE TRIGGER library_index_restore_dispatch_monotone BEFORE UPDATE OR DELETE ON library_index_restore_dispatch FOR EACH ROW EXECUTE FUNCTION library_index_restore_dispatch_guard(); --- 激活登记:配置开关不是独占证明。只有部署流程把"旧 writer 已退出并排空"的证据 --- 登记到这个目标上,协议才对该目标生效;撤销后立即 fail closed。 +-- 激活登记:配置开关不是独占证明。只有部署流程把一份可核验的"旧 writer 已退出并 +-- 排空"证据文件登记到这个目标上,协议才对该目标生效。 +-- +-- 除 revoked_at 外全部不可变;撤销是终局(NULL -> 时间戳之后不能再改回 NULL、不能 +-- 改时间、不能删除整行)。要重新开放同一物理目标,只能换新的目标身份并重新登记, +-- 不能"把旧行归档/删掉再插一行"。 CREATE TABLE IF NOT EXISTS library_index_restore_dispatch_activation ( target_identity text PRIMARY KEY, protocol_version integer NOT NULL CHECK (protocol_version = 1), + target_fingerprint text NOT NULL CHECK (target_fingerprint <> ''), + target_instance text NOT NULL CHECK (target_instance <> ''), + governance_database text NOT NULL CHECK (governance_database <> ''), + protocol_build text NOT NULL CHECK (protocol_build <> ''), activated_by text NOT NULL CHECK (activated_by <> ''), drain_evidence_digest text NOT NULL CHECK (drain_evidence_digest <> ''), + evidence jsonb NOT NULL, + attested_at timestamptz NOT NULL, activated_at timestamptz NOT NULL DEFAULT now(), revoked_at timestamptz ); CREATE OR REPLACE FUNCTION library_index_restore_dispatch_activation_guard() RETURNS trigger AS $$ BEGIN + IF TG_OP = 'DELETE' THEN + RAISE EXCEPTION 'restore dispatch activation evidence is append-only'; + END IF; IF NEW.target_identity <> OLD.target_identity OR NEW.protocol_version <> OLD.protocol_version OR - NEW.activated_by <> OLD.activated_by OR NEW.drain_evidence_digest <> OLD.drain_evidence_digest OR - NEW.activated_at <> OLD.activated_at THEN + NEW.target_fingerprint <> OLD.target_fingerprint OR NEW.target_instance <> OLD.target_instance OR + NEW.governance_database <> OLD.governance_database OR NEW.protocol_build <> OLD.protocol_build OR + NEW.activated_by <> OLD.activated_by OR + NEW.drain_evidence_digest <> OLD.drain_evidence_digest OR NEW.evidence <> OLD.evidence OR + NEW.attested_at <> OLD.attested_at OR NEW.activated_at <> OLD.activated_at THEN RAISE EXCEPTION 'restore dispatch activation is immutable except revocation'; END IF; - IF OLD.revoked_at IS NOT NULL AND NEW.revoked_at IS NOT NULL AND NEW.revoked_at <> OLD.revoked_at THEN - RAISE EXCEPTION 'restore dispatch activation revocation is final'; + -- 终局:已撤销行既不能改回 NULL,也不能改写成另一个时间。 + IF OLD.revoked_at IS NOT NULL THEN + IF NEW.revoked_at IS NULL OR NEW.revoked_at <> OLD.revoked_at THEN + RAISE EXCEPTION 'restore dispatch activation revocation is final'; + END IF; + RETURN NEW; END IF; RETURN NEW; END; @@ -94,5 +114,5 @@ $$ LANGUAGE plpgsql; DROP TRIGGER IF EXISTS library_index_restore_dispatch_activation_guard_trg ON library_index_restore_dispatch_activation; CREATE TRIGGER library_index_restore_dispatch_activation_guard_trg - BEFORE UPDATE ON library_index_restore_dispatch_activation + BEFORE UPDATE OR DELETE ON library_index_restore_dispatch_activation FOR EACH ROW EXECUTE FUNCTION library_index_restore_dispatch_activation_guard(); \ No newline at end of file diff --git a/internal/releaseadapter/v071/formal_maintenance_recovery_r38_test.go b/internal/releaseadapter/v071/formal_maintenance_recovery_r38_test.go index 8ff1d6a..0dcc78a 100644 --- a/internal/releaseadapter/v071/formal_maintenance_recovery_r38_test.go +++ b/internal/releaseadapter/v071/formal_maintenance_recovery_r38_test.go @@ -162,6 +162,22 @@ func (p *r38RefusalProxy) setRefuseClean(refuse bool) { p.flush() } +// markerCreates 只数**真实转发**到 ES 的恢复标记 create(路径 _create + claim 前缀)。 +func (p *r38RefusalProxy) markerCreates() int { + p.mu.Lock() + defer p.mu.Unlock() + count := 0 + for _, event := range p.events { + path, _ := event["path"].(string) + request, _ := event["request"].(string) + dispatched, _ := event["dispatched"].(bool) + if dispatched && strings.Contains(path, "/_create/") && strings.Contains(request, indexRestoreClaimPrefix) { + count++ + } + } + return count +} + func (p *r38RefusalProxy) armVerifyCorruption() { p.mu.Lock() defer p.mu.Unlock() @@ -404,6 +420,7 @@ func TestFormalMaintenanceRecoveryHeldThenRecovered(t *testing.T) { objectBranch := scenario == "object-restore" pgBranch := scenario == "pg-restore" earlyBranch := scenario == "pg-early" + ds01Branch := scenario == "ds01" // 三条分支共用同一套 fullshape/真实入库装配;对象分支与两种 PG 分支都让 ES 恢复不成为 // 第一个失败点:ES 分支拒绝 clean 派发,PG 分支在恢复自己的事务里拿到真实 SQLSTATE。 fullShape := objectBranch || pgBranch || earlyBranch @@ -554,9 +571,10 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v reader := gatewayReadOnlyGovernance(t, ctx, gov, govDSN) project := os.Getenv("KG_R38_PROJECT") proxy := newR38RefusalProxy(t, root, fx.searchBase) - if fullShape { + if fullShape || ds01Branch { // 对象分支要证明的是"对象恢复传输失败",ES 的回滚写入必须放行, - // 否则失败会被误报成旧的 ES clean-dispatch 分支。 + // 否则失败会被误报成旧的 ES clean-dispatch 分支。DS01 分支要让恢复在 + // 派发 CAS 之前被产品接缝拒绝,ES 传输层不能先拒绝别的写。 proxy.setRefuseClean(false) } objectProxy := newR38ObjectRefusalProxy(t, root, "http://"+os.Getenv("KG_R38_GATEWAY_ADDR")) @@ -599,11 +617,49 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v "KG_TEST_BARRIER_DIR": filepath.Join(root, "barriers"), "KG_TEST_BARRIER_POINTS": "after:wiki", "KG_TEST_BARRIER_TIMEOUT": "300s", } + if ds01Branch { + if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" { + t.Fatal("ds01 branch requires KG_LIBRARY_TEST_ISOLATED=1") + } + env["LIBRARY_RESTORE_DISPATCH_V2"] = "1" + env["KG_DS01_TEST_REFUSE_MARKER"] = fx.index + "/chunk-target" + } env["PRODUCTION_RELEASE_AUTHORIZER_REF"] = project env["PRODUCTION_RELEASE_AUTH_TOKEN_FILE"] = filepath.Join(root, "authorization-token") must(os.WriteFile(env["PRODUCTION_RELEASE_AUTH_TOKEN_FILE"], []byte(objectgateway.Digest([]byte(root+"approval"))), 0600)) cfg, e := appconfig.Load(func(k string) string { return env[k] }) must(e) + var ds01Fingerprint string + if ds01Branch { + // 激活证据必须由**部署流程**登记,且与真实目标身份一致:这里用与 worker 相同的 + // 业务库/索引/桶观察目标指纹,并核对治理库名。 + probe := fx.target(t) + ds01Fingerprint, _, e = probe.observeRestoreTarget(ctx) + must(e) + var governanceDatabase string + must(gov.QueryRow(ctx, `SELECT current_database()`).Scan(&governanceDatabase)) + exited := time.Now().UTC().Add(-time.Minute) + evidence := libraryrelease.RestoreDispatchActivationEvidence{ + ProtocolVersion: 1, TargetIdentity: cfg.LibraryTargetIdentity(), TargetFingerprint: ds01Fingerprint, + TargetInstance: project, GovernanceDatabase: governanceDatabase, ProtocolBuild: os.Getenv("KG_R38_VERSION"), + AttestedBy: "r38-runner", AttestedAt: time.Now().UTC(), + Writers: []libraryrelease.RestoreDispatchWriterDrain{{ + Name: "writer", Kind: "worker", Stopped: true, ExitedAt: &exited, + QueueDrained: true, DrainEvidence: objectgateway.Digest([]byte(root + "drain")), + }}, + } + rawEvidence, merr := json.Marshal(evidence) + must(merr) + evidencePath := filepath.Join(root, "ds01-activation.json") + must(os.WriteFile(evidencePath, rawEvidence, 0600)) + sum := sha256.Sum256(rawEvidence) + activationStore := postgresstore.NewStore(gov) + must(activationStore.ActivateRestoreDispatch(ctx, evidencePath, hex.EncodeToString(sum[:]), + cfg.LibraryTargetIdentity(), ds01Fingerprint, time.Now().UTC())) + save("05b-ds01-activation.json", map[string]any{"target": cfg.LibraryTargetIdentity(), + "fingerprint": ds01Fingerprint, "governance_database": governanceDatabase, + "evidence_digest": hex.EncodeToString(sum[:]), "protocol_build": os.Getenv("KG_R38_VERSION")}) + } dataToken := objectgateway.Digest([]byte(root + "data")) controlToken := objectgateway.Digest([]byte(root + "control")) env["LIBRARY_OBJECT_GATEWAY_URL"] = objectProxy.url @@ -883,6 +939,14 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v } failureText := libraryAfterPublish.Error + "\n" + libraryAfterPublish.Recovery switch { + case ds01Branch: + if !strings.Contains(failureText, "恢复失败") || !strings.Contains(failureText, "injected DS01 marker refusal before dispatch CAS") { + t.Fatalf("publish failure does not name the DS01 pre-CAS refusal: %q", failureText) + } + if strings.Contains(failureText, "index effect outcome unknown") || strings.Contains(failureText, "SQLSTATE") || + strings.Contains(failureText, "conditional object cleanup not confirmed") { + t.Fatalf("ds01 branch was mislabelled as another failure: %q", failureText) + } case objectBranch: // 失败必须来自对象恢复边界:回滚已通过 ES clean 写入,随后在对象删除派发前被拒。 if !strings.Contains(failureText, "恢复失败") || !strings.Contains(failureText, "conditional object cleanup not confirmed") { @@ -978,6 +1042,18 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v effects := r38IndexEffects(t, archiveRoot, "chunk-target") save("12-index-effects-held.json", effects) switch { + case ds01Branch: + // 派发 CAS 之前被拒:意图已落盘、台账仍是 prepared、没有 claimed/finalize。 + for _, phase := range []string{"apply-intent", "applied", "restore-intent", "restore-mark-intent"} { + if _, ok := effects[phase]; !ok { + t.Fatalf("ds01 held boundary missing %s evidence (have %v)", phase, effectPhases(effects)) + } + } + for _, phase := range []string{"restore-claimed", "restore-finalize-intent", "restored"} { + if _, ok := effects[phase]; ok { + t.Fatalf("ds01 held boundary must not carry %s evidence", phase) + } + } case objectBranch, pgBranch: // 对象分支的 ES clean 写入已成功;PG wiki 分支的 index 阶段先于 wiki 提交。 // 两者都必须有恢复回执,这不是被拒的边界。 @@ -1166,6 +1242,19 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v t.Fatalf("recover with another window => %d %v", code, mismatch) } save("17-window-binding-409.json", map[string]any{"http": code, "body": mismatch}) + if ds01Branch { + var state, owner, receipt, binding string + must(gov.QueryRow(ctx, `SELECT state, dispatch_owner, COALESCE(marker_receipt::text,''), binding_digest +FROM library_index_restore_dispatch WHERE index_name=$1 AND document_id='chunk-target'`, fx.index). + Scan(&state, &owner, &receipt, &binding)) + creates := proxy.markerCreates() + save("17c-ds01-held-ledger.json", map[string]any{"state": state, "owner": owner, + "receipt": receipt, "binding": binding, "marker_creates_forwarded": creates, + "target_fingerprint": ds01Fingerprint}) + if state != libraryrelease.RestoreDispatchPrepared || owner != "" || receipt != "" || creates != 0 { + t.Fatalf("DS01 held boundary is not a clean prepared record: state=%s owner=%q creates=%d", state, owner, creates) + } + } // ---------- 4) 撤故障 → 正常 POST /library/maintenance/recover ---------- proxy.setRefuseClean(false) @@ -1180,6 +1269,16 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v // 精确解除:只删那一行外部范围的 knowledges 占位行;不换成删除 held 记录或重拍备份。 save("17b-pg-early-fault-disarmed.json", r38DisarmPGEarlyFault(t, fx, conflictKnowledge)) } + if ds01Branch { + // 正式 worker 必须真的换进程:旧执行者退出后由同一冻结二进制的新 worker 消费恢复。 + oldPID := worker.Process.Pid + _ = worker.Process.Kill() + must(worker.Wait()) + delete(workerEnv, "KG_DS01_TEST_REFUSE_MARKER") + worker = start("worker", "worker-recover", workerEnv) + save("17c2-ds01-worker-restart.json", map[string]any{"old_pid": oldPID, "new_pid": worker.Process.Pid, + "refusal_env_removed": true, "binary": "worker", "sha256": expectedBinaries["worker"]}) + } save("17a-recover-request.json", map[string]any{"window_job_id": job.ID, "scope": "internal", "backup_snapshot_id": backupID, "original_job": job.ID, "object_fault_removed": objectBranch, "pg_fault_removed": pgBranch, "pg_early_fault_removed": earlyBranch, @@ -1201,6 +1300,18 @@ UPDATE knowledge_bases SET embedding_model_id='r38-model',vector_store_id='r38-v if got := waitJob(recoveryQueueID, func(s string) bool { return s == "completed" }, 300, "recovery must complete"); got != "completed" { t.Fatalf("recovery state = %s", got) } + if ds01Branch { + var state, owner, receipt string + must(gov.QueryRow(ctx, `SELECT state, dispatch_owner, COALESCE(marker_receipt::text,'') +FROM library_index_restore_dispatch WHERE index_name=$1 AND document_id='chunk-target'`, fx.index). + Scan(&state, &owner, &receipt)) + creates := proxy.markerCreates() + save("18b-ds01-recovered-ledger.json", map[string]any{"state": state, "owner": owner, + "receipt": receipt, "marker_creates_forwarded": creates, "recovery_queue": recoveryQueueID}) + if state != libraryrelease.RestoreDispatchMarkerConfirmed || owner == "" || receipt == "" || creates != 1 { + t.Fatalf("DS01 recovery ledger = state=%s owner=%q creates=%d", state, owner, creates) + } + } recoveredJournal := readR38JSON(t, filepath.Join(root, "maintenance", job.ID+".json")) save("19-journal-recovered.json", recoveredJournal) diff --git a/internal/releaseadapter/v071/library_index_ds01_fixes_test.go b/internal/releaseadapter/v071/library_index_ds01_fixes_test.go new file mode 100644 index 0000000..6e77ebb --- /dev/null +++ b/internal/releaseadapter/v071/library_index_ds01_fixes_test.go @@ -0,0 +1,283 @@ +package v071 + +import ( + "context" + "errors" + "os" + "reflect" + "strings" + "testing" + + "knowledge-governance/internal/libraryrelease" +) + +// ds01StubGate 只用于分类测试:不产生许可,只返回给定的台账视图。 +type ds01StubGate struct { + inspectStatus libraryrelease.RestoreDispatchStatus + inspectFound bool + records []libraryrelease.RestoreDispatchRecord +} + +func (g *ds01StubGate) InspectRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding) (libraryrelease.RestoreDispatchStatus, bool, error) { + return g.inspectStatus, g.inspectFound, nil +} +func (g *ds01StubGate) PrepareRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding) (libraryrelease.RestoreDispatchStatus, error) { + return libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchPrepared}, nil +} +func (g *ds01StubGate) AcquireRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding) (*libraryrelease.RestoreSendPermit, libraryrelease.RestoreDispatchStatus, error) { + return nil, g.inspectStatus, libraryrelease.ErrRestoreDispatchHeld +} +func (g *ds01StubGate) ConfirmRestoreMarker(context.Context, libraryrelease.RestoreMarkerBinding, []byte) error { + return nil +} +func (g *ds01StubGate) UnresolvedRestoreDispatches(context.Context, string, string, string) ([]libraryrelease.RestoreDispatchRecord, error) { + return g.records, nil +} + +func ds01FreezeBindingInputs(t *testing.T, target *ProductionLibraryTarget, backup libraryrelease.BackupRef) { + t.Helper() + target.restoreScope = "internal" + target.restoreSourceSnapshot = backup.WriteSet.TargetSnapshotID + target.restoreBackupSnapshot = backup.SnapshotID +} + +// DS01-F1:绑定必须来自原锚点——来源快照 != 写前镜像,且带真实目标实例/指纹/UUID。 +func TestDS01BindingUsesFrozenSourceAndRealBackup(t *testing.T) { + fx, snapshot, target, backups, wire := ds01Fixture(t) + authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot) + authority.attach(target) + outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire) + ledger := outcome.Backup.WriteSet + if ledger.TargetSnapshotID == "" || outcome.Backup.SnapshotID == "" { + t.Fatal("fixture lacks a source snapshot or backup image") + } + if ledger.TargetSnapshotID == outcome.Backup.SnapshotID { + t.Fatalf("fixture plan uses one snapshot for both source and backup: %s", ledger.TargetSnapshotID) + } + if err := target.AdoptWriteSet(*ledger); err != nil { + t.Fatal(err) + } + ds01FreezeBindingInputs(t, target, outcome.Backup) + change := indexChangeOf(t, ledger, "chunk-target") + body, err := target.restoreIndexClaimBody(change) + if err != nil { + t.Fatal(err) + } + binding, err := target.restoreMarkerBinding(fx.ctx, change, body) + if err != nil { + t.Fatalf("real binding: %v", err) + } + if binding.SourceSnapshot != ledger.TargetSnapshotID { + t.Fatalf("binding source = %q, plan source = %q", binding.SourceSnapshot, ledger.TargetSnapshotID) + } + if binding.BackupSnapshot != outcome.Backup.SnapshotID { + t.Fatalf("binding backup = %q, real backup = %q", binding.BackupSnapshot, outcome.Backup.SnapshotID) + } + if binding.SourceSnapshot == binding.BackupSnapshot { + t.Fatal("binding still uses the same snapshot for source and backup") + } + if binding.Scope == "" || binding.TargetInstance != ds01TargetInstance || + binding.TargetFingerprint != authority.targetFingerprint || binding.IndexUUID != authority.indexUUID { + t.Fatalf("binding lacks the real target identity: %+v", binding) + } + status, found, err := authority.store.InspectRestoreMarker(fx.ctx, binding) + if err != nil || !found || status.State != libraryrelease.RestoreDispatchPrepared { + t.Fatalf("real ledger rejected the frozen binding: %+v found=%v err=%v", status, found, err) + } + // 缺任一锚点都必须失败关闭。 + target.restoreSourceSnapshot = "" + if _, err := target.restoreMarkerBinding(fx.ctx, change, body); err == nil { + t.Fatal("binding accepted without a source snapshot anchor") + } +} + +// DS01-F4:绑定冲突(prepared + Held)必须在任何目标写之前拒绝,不能继续 replay。 +func TestDS01BindingConflictHeldBeforeAnyWrite(t *testing.T) { + fx, snapshot, target, backups, wire := ds01Fixture(t) + authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot) + authority.attach(target) + outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire) + if err := target.AdoptWriteSet(*outcome.Backup.WriteSet); err != nil { + t.Fatal(err) + } + ds01FreezeBindingInputs(t, target, outcome.Backup) + target.config.RestoreDispatch = &ds01StubGate{inspectFound: true, + inspectStatus: libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchPrepared, Held: true, + Reason: "binding differs"}} + beforePG := universeDump(t, fx.pool) + beforeIndex := recoveryIndexReadback(t, fx) + beforeObjects := objectVersionSet(t, fx) + beforeDispatch := wire.dispatch + + err := target.Restore(fx.ctx, outcome.Backup) + if err == nil || !strings.Contains(err.Error(), "held") { + t.Fatalf("binding conflict admission = %v", err) + } + if wire.dispatch != beforeDispatch { + t.Fatalf("binding conflict forwarded %d marker writes", wire.dispatch-beforeDispatch) + } + if diff := diffUniverse(beforePG, universeDump(t, fx.pool)); diff != "" { + t.Fatalf("binding conflict wrote PG before admission: %s", diff) + } + if after := recoveryIndexReadback(t, fx); !reflect.DeepEqual(beforeIndex, after) { + t.Fatal("binding conflict changed the index") + } + if after := objectVersionSet(t, fx); !reflect.DeepEqual(beforeObjects, after) { + t.Fatal("binding conflict changed objects") + } +} + +// DS01-F4:prepared 台账却已在目标上出现精确 marker —— 证据矛盾,必须 held; +// 旧实现会把它当成"已存在的标记"认领并 clean。 +func TestDS01PreparedWithMarkerIsContradictionHeld(t *testing.T) { + fx, snapshot, target, backups, wire := ds01Fixture(t) + authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot) + authority.attach(target) + outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire) + if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchPrepared { + t.Fatalf("boundary state = %s", state) + } + ledger := outcome.Backup.WriteSet + if err := target.AdoptWriteSet(*ledger); err != nil { + t.Fatal(err) + } + ds01FreezeBindingInputs(t, target, outcome.Backup) + change := indexChangeOf(t, ledger, "chunk-target") + body, err := target.restoreIndexClaimBody(change) + if err != nil { + t.Fatal(err) + } + // 模拟一个绕过协议的旧 writer:目标上直接出现精确 marker,但台账仍是 prepared。 + applied, err := target.loadIndexEffect(change, "applied") + if err != nil || applied == nil || applied.Receipt == nil { + t.Fatalf("applied receipt missing: %v", err) + } + method, path, payload, err := target.indexMutation(change, body, applied.Receipt.SeqNo, applied.Receipt.PrimaryTerm, true) + if err != nil { + t.Fatal(err) + } + var receipt indexReceipt + if err := target.search.requestJSON(fx.ctx, method, path, payload, &receipt); err != nil { + t.Fatalf("foreign marker write: %v", err) + } + beforePG := universeDump(t, fx.pool) + beforeDispatch := wire.dispatch + + err = target.Restore(fx.ctx, outcome.Backup) + if err == nil || !strings.Contains(err.Error(), "held") { + t.Fatalf("prepared+marker admission = %v", err) + } + if wire.dispatch != beforeDispatch { + t.Fatalf("prepared+marker forwarded %d marker writes", wire.dispatch-beforeDispatch) + } + if diff := diffUniverse(beforePG, universeDump(t, fx.pool)); diff != "" { + t.Fatalf("prepared+marker wrote PG before admission: %s", diff) + } + requireIndexEffect(t, fx, ledger, change, "restored", false) + if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchPrepared { + t.Fatalf("dispatch state moved to %s", state) + } +} + +// DS01-F4:收尾必须用原 window 的权威未决全集;计划外记录与 prepared 记录都拒绝。 +func TestDS01CloseoutUsesAuthoritativeWindowSet(t *testing.T) { + fx, snapshot, target, backups, wire := ds01Fixture(t) + authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot) + authority.attach(target) + outcome := ds01RunToBoundary(t, fx, snapshot, target, backups, wire) + ledger := outcome.Backup.WriteSet + if err := target.AdoptWriteSet(*ledger); err != nil { + t.Fatal(err) + } + ds01FreezeBindingInputs(t, target, outcome.Backup) + change := indexChangeOf(t, ledger, "chunk-target") + + // 台账里的 prepared 记录本身就必须拒绝收尾(旧实现只拒绝 dispatching)。 + if err := target.verifyRestoreEffectsClosed(fx.ctx); err == nil || !strings.Contains(err.Error(), "未决") { + t.Fatalf("closeout accepted a prepared dispatch record: %v", err) + } + // 计划外原窗口记录:即使当前 plan 的 effect 看起来都已完成,也必须拒绝。 + target.config.RestoreDispatch = &ds01StubGate{inspectFound: true, + inspectStatus: libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchMarkerConfirmed}, + records: []libraryrelease.RestoreDispatchRecord{{ + PlanDigest: ledger.Digest, Index: change.Index, DocumentID: "ghost-document", + State: libraryrelease.RestoreDispatchMarkerConfirmed}}} + if err := target.verifyRestoreEffectsClosed(fx.ctx); err == nil || !strings.Contains(err.Error(), "计划外") { + t.Fatalf("closeout accepted a plan-outside record: %v", err) + } + // 换 plan 别名也必须拒绝。 + target.config.RestoreDispatch = &ds01StubGate{inspectFound: true, + inspectStatus: libraryrelease.RestoreDispatchStatus{State: libraryrelease.RestoreDispatchMarkerConfirmed}, + records: []libraryrelease.RestoreDispatchRecord{{ + PlanDigest: "another-plan", Index: change.Index, DocumentID: change.ID, + State: libraryrelease.RestoreDispatchMarkerConfirmed}}} + if err := target.verifyRestoreEffectsClosed(fx.ctx); err == nil || !strings.Contains(err.Error(), "另一份计划") { + t.Fatalf("closeout accepted a plan alias: %v", err) + } +} + +// DS01-F5:prepared 已提交、mark-intent 文件写失败 ⇒ 重启后可补同内容并只派发一次; +// 无台账的旧意图仍按 legacy held(本文件同时守住两个方向)。 +func TestDS01PreparedWithoutIntentResumesOnce(t *testing.T) { + fx, snapshot, target, backups, wire := ds01Fixture(t) + authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot) + authority.attach(target) + changeID := "chunk-target" + target.restoreIntentFault = func(c libraryrelease.IndexWrite) error { + if c.ID == changeID { + return errors.New("injected mark-intent file write failure") + } + return nil + } + outcome := libraryrelease.Run(fx.ctx, requestFor(snapshot), contractFailWikiTarget{target}, backups, nil) + if outcome.State != "rollback_failed" || !strings.Contains(outcome.Rollback, "mark-intent") { + t.Fatalf("intent write failure not reached: state=%q rollback=%q", outcome.State, outcome.Rollback) + } + ledger := outcome.Backup.WriteSet + change := indexChangeOf(t, ledger, changeID) + requireIndexEffect(t, fx, ledger, change, "restore-intent", true) + requireIndexEffect(t, fx, ledger, change, "restore-mark-intent", false) + if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchPrepared { + t.Fatalf("after intent failure dispatch row = %s", state) + } + if wire.dispatch != 0 { + t.Fatalf("intent failure forwarded %d marker writes", wire.dispatch) + } + + // 新调用(同一 attempt)看到 prepared + 无 mark-intent:补同内容意图后继续。 + if err := target.Restore(fx.ctx, outcome.Backup); err != nil { + t.Fatalf("prepared-without-intent resume: %v", err) + } + requireIndexEffect(t, fx, ledger, change, "restore-mark-intent", true) + requireIndexEffect(t, fx, ledger, change, "restored", true) + if wire.dispatch != 1 { + t.Fatalf("resume forwarded %d marker creates, want 1", wire.dispatch) + } + if state, _ := authority.dispatchRow(t); state != libraryrelease.RestoreDispatchMarkerConfirmed { + t.Fatalf("terminal state = %s", state) + } +} + +// 无台账的旧意图仍然 held:补意图的能力只对新协议 prepared 区段开放。 +func TestDS01LegacyIntentWithoutLedgerStillHeld(t *testing.T) { + fx, snapshot, target, backups, wire := ds01Fixture(t) + legacy := ds01RunToBoundary(t, fx, snapshot, target, backups, wire) + authority := newDS01Authority(t, requestFor(snapshot).JobID, snapshot) + authority.attach(target) + ledger := legacy.Backup.WriteSet + change := indexChangeOf(t, ledger, "chunk-target") + // 删掉 mark-intent,模拟"只剩旧意图、没有新台账"的历史现场。 + if err := os.Remove(target.indexEffectPath(change, "restore-mark-intent")); err != nil { + t.Fatal(err) + } + err := target.Restore(fx.ctx, legacy.Backup) + if err == nil || !strings.Contains(err.Error(), "legacy") { + t.Fatalf("legacy intent without ledger = %v", err) + } + if wire.dispatch != 0 { + t.Fatalf("legacy intent forwarded %d marker writes", wire.dispatch) + } + if rows := authority.dispatchRows(t); rows != 0 { + t.Fatalf("legacy intent created %d dispatch rows", rows) + } +} diff --git a/internal/releaseadapter/v071/library_index_ds01_test.go b/internal/releaseadapter/v071/library_index_ds01_test.go index 5257d98..5695f75 100644 --- a/internal/releaseadapter/v071/library_index_ds01_test.go +++ b/internal/releaseadapter/v071/library_index_ds01_test.go @@ -2,6 +2,8 @@ package v071 import ( "context" + "crypto/sha256" + "encoding/hex" "encoding/json" "fmt" "io" @@ -29,17 +31,20 @@ import ( // 这里不制造"prepared 且从未派发"的假正例:prepared 一律由真实 Prepare 产生。 const ds01TargetIdentity = "kg-ds01-test-target" +const ds01TargetInstance = "inst-ds01" type ds01Authority struct { - store *postgresstore.Store - pool *pgxpool.Pool - dsn string - operationID string - jobID string - owner string - epoch int64 - targetIdentity string - snapshot string + store *postgresstore.Store + pool *pgxpool.Pool + dsn string + operationID string + jobID string + owner string + epoch int64 + targetIdentity string + snapshot string + targetFingerprint string + indexUUID string } func newDS01Authority(t *testing.T, jobID, snapshot string) *ds01Authority { @@ -78,9 +83,6 @@ func newDS01Authority(t *testing.T, jobID, snapshot string) *ds01Authority { if err != nil { t.Fatalf("begin attempt: %v", err) } - if err := store.ActivateRestoreDispatch(ctx, ds01TargetIdentity, "ds01-test", "sha256:drain-evidence", now); err != nil { - t.Fatalf("activate dispatch: %v", err) - } return &ds01Authority{store: store, pool: pool, dsn: dsn, operationID: attempt.OperationID, jobID: attempt.JobID, owner: attempt.Owner, epoch: int64(attempt.Epoch), targetIdentity: ds01TargetIdentity, snapshot: snapshot} @@ -95,7 +97,24 @@ func (a *ds01Authority) attach(target *ProductionLibraryTarget) { target.config.RestoreWindowJobID = a.jobID target.config.RestoreAttemptOwner = a.owner target.config.RestoreAttemptEpoch = a.epoch - target.config.RestoreSnapshotID = a.snapshot + target.config.RestoreInstance = ds01TargetInstance + fingerprint, indexUUID, err := target.observeRestoreTarget(context.Background()) + if err != nil { + panic(err) + } + if _, err := a.pool.Exec(context.Background(), ` +INSERT INTO library_instance_maintenance (instance, owner_job_id, owner_scope, target_fingerprint, state, + original_service_state, note, acquired_at, updated_at) +VALUES ($1,$2,'internal',$3,'acquired','app:running','ds01-test',$4,$4) +ON CONFLICT (instance) DO UPDATE SET owner_job_id=EXCLUDED.owner_job_id, owner_scope=EXCLUDED.owner_scope, + target_fingerprint=EXCLUDED.target_fingerprint, released_at=NULL, state='acquired', updated_at=EXCLUDED.updated_at`, + ds01TargetInstance, a.jobID, fingerprint, time.Now().UTC()); err != nil { + panic(err) + } + a.targetFingerprint, a.indexUUID = fingerprint, indexUUID + if err := activateDS01(context.Background(), a.store, a.pool, ds01TargetIdentity, fingerprint); err != nil { + panic(err) + } } func (a *ds01Authority) dispatchRow(t *testing.T) (state, owner string) { @@ -273,7 +292,7 @@ func TestDS01DispatchingMissingStaysHeldWithZeroWrites(t *testing.T) { if err != nil { t.Fatal(err) } - binding, err := target.restoreMarkerBinding(change, body) + binding, err := target.restoreMarkerBinding(fx.ctx, change, body) if err != nil { t.Fatal(err) } @@ -329,7 +348,7 @@ func TestDS01CloseoutRejectsUnresolvedDispatch(t *testing.T) { if err != nil { t.Fatal(err) } - binding, err := target.restoreMarkerBinding(change, body) + binding, err := target.restoreMarkerBinding(fx.ctx, change, body) if err != nil { t.Fatal(err) } @@ -486,7 +505,7 @@ func TestDS01RestoreHelper(t *testing.T) { target.config.RestoreWindowJobID = os.Getenv("KG_DS01_JOB") target.config.RestoreAttemptOwner = os.Getenv("KG_DS01_OWNER") target.config.RestoreAttemptEpoch = epoch - target.config.RestoreSnapshotID = os.Getenv("KG_DS01_SNAPSHOT") + target.config.RestoreInstance = ds01TargetInstance client := *target.search.client inner := client.Transport @@ -521,3 +540,33 @@ func TestDS01RestoreHelper(t *testing.T) { t.Fatal(err) } } + +// activateDS01 写一份结构完整的激活证据(治理库名取自隔离库本身)。 +func activateDS01(ctx context.Context, store *postgresstore.Store, pool *pgxpool.Pool, + targetIdentity, fingerprint string) error { + var database string + if err := pool.QueryRow(ctx, `SELECT current_database()`).Scan(&database); err != nil { + return err + } + now := time.Now().UTC() + exited := now.Add(-time.Minute) + evidence := libraryrelease.RestoreDispatchActivationEvidence{ + ProtocolVersion: 1, TargetIdentity: targetIdentity, TargetFingerprint: fingerprint, + TargetInstance: ds01TargetInstance, GovernanceDatabase: database, ProtocolBuild: "test-build", + AttestedBy: "ds01-test", AttestedAt: now, + Writers: []libraryrelease.RestoreDispatchWriterDrain{{ + Name: "worker", Kind: "worker", Stopped: true, ExitedAt: &exited, + QueueDrained: true, DrainEvidence: "sha256:drain", + }}, + } + raw, err := json.Marshal(evidence) + if err != nil { + return err + } + path := filepath.Join(os.TempDir(), "ds01-activation-"+targetIdentity+".json") + if err := os.WriteFile(path, raw, 0o600); err != nil { + return err + } + sum := sha256.Sum256(raw) + return store.ActivateRestoreDispatch(ctx, path, hex.EncodeToString(sum[:]), targetIdentity, fingerprint, now) +} diff --git a/internal/releaseadapter/v071/library_index_effects.go b/internal/releaseadapter/v071/library_index_effects.go index e37b15a..0f09628 100644 --- a/internal/releaseadapter/v071/library_index_effects.go +++ b/internal/releaseadapter/v071/library_index_effects.go @@ -36,6 +36,13 @@ type indexEffect struct { // ES 的写计数器是分片生命周期内的统计量(分片迁移/节点替换会清零), // 不是不可变事务日志,不能作为放行写入的授权依据。 func (t *ProductionLibraryTarget) beginRestoreMarker(ctx context.Context, c libraryrelease.IndexWrite, applied indexReceipt) error { + if t.restoreIntentFault != nil { + fault := t.restoreIntentFault + t.restoreIntentFault = nil + if err := fault(c); err != nil { + return err + } + } raw, err := json.Marshal(indexEffect{Plan: t.activePlan.Digest, Change: c, Receipt: &applied}) if err != nil { return err diff --git a/internal/releaseadapter/v071/library_index_restore_claim.go b/internal/releaseadapter/v071/library_index_restore_claim.go index 6247a2e..071e4ce 100644 --- a/internal/releaseadapter/v071/library_index_restore_claim.go +++ b/internal/releaseadapter/v071/library_index_restore_claim.go @@ -84,34 +84,49 @@ func (t *ProductionLibraryTarget) restoreClaimedIndexBefore(ctx context.Context, } if markIntent == nil { if !start { - return errors.New("legacy unknown index restoration lacks attributable marker intent") - } - // 新协议:先在治理库建立 prepared(文件意图之前),再落文件意图。文件失败只留 - // prepared(零派发,下一进程可补写同内容意图);反过来"先文件后台账"则会造出 - // 无法证明未派发的旧证据。 - if t.config.RestoreDispatch != nil && len(c.After) == 0 { - binding, err := t.restoreMarkerBinding(c, body) + // DS01-F5:prepared 已提交、mark-intent 未落盘的窄区段。旧实现一律按 + // legacy 拒绝,把设计允许补同内容意图的区段永久堵死。先精确核对: + // 原台账存在、绑定一致、仍是 prepared、且没有任何 claimed/finalize/restored + // 矛盾——全部成立才允许补写同一份意图;否则仍是旧协议未知历史。 + resumed, err := t.resumePreparedWithoutIntent(ctx, c, body, applied) if err != nil { return err } - status, err := t.config.RestoreDispatch.PrepareRestoreMarker(ctx, binding) - if err != nil { + if !resumed { + return errors.New("legacy unknown index restoration lacks attributable marker intent") + } + if markIntent, err = t.loadIndexEffect(c, "restore-mark-intent"); err != nil { return err } - if status.Held { - return heldRestoreMarker(status, "首次恢复不能建立 prepared") + freshIntent = false + } else { + // 新协议:先在治理库建立 prepared(文件意图之前),再落文件意图。文件失败只留 + // prepared(零派发,下一进程可补写同内容意图);反过来"先文件后台账"则会造出 + // 无法证明未派发的旧证据。 + if t.config.RestoreDispatch != nil && len(c.After) == 0 { + binding, err := t.restoreMarkerBinding(ctx, c, body) + if err != nil { + return err + } + status, err := t.config.RestoreDispatch.PrepareRestoreMarker(ctx, binding) + if err != nil { + return err + } + if status.Held { + return heldRestoreMarker(status, "首次恢复不能建立 prepared") + } + } + if err = t.beginRestoreMarker(ctx, c, applied); err != nil { + return err + } + if markIntent, err = t.loadIndexEffect(c, "restore-mark-intent"); err != nil { + return err } - } - if err = t.beginRestoreMarker(ctx, c, applied); err != nil { - return err - } - if markIntent, err = t.loadIndexEffect(c, "restore-mark-intent"); err != nil { - return err } } else if t.config.RestoreDispatch != nil && len(c.After) == 0 { // 既有意图必须能在新台账里找到同一 effect:找不到就是旧协议的未知历史, // 不能借"文件看起来完整"补一份 prepared。 - binding, err := t.restoreMarkerBinding(c, body) + binding, err := t.restoreMarkerBinding(ctx, c, body) if err != nil { return err } @@ -215,6 +230,42 @@ func (t *ProductionLibraryTarget) restoreClaimedIndexBefore(ctx context.Context, return t.finishRestoreCleanWrite(ctx, c, *claimed.Receipt) } +// resumePreparedWithoutIntent 处理"prepared 已提交、mark-intent 还没落盘"的窄区段。 +// +// 只有同时满足以下各项才允许补写同一份意图并继续:台账存在、绑定一致、状态仍是 +// prepared,且没有 claimed/finalize/restored 任何矛盾证据。没有台账的旧意图仍按 +// legacy 处理(返回 false),不回填、不推测。 +func (t *ProductionLibraryTarget) resumePreparedWithoutIntent(ctx context.Context, c libraryrelease.IndexWrite, + body json.RawMessage, applied indexReceipt) (bool, error) { + if t.config.RestoreDispatch == nil || len(c.After) != 0 { + return false, nil + } + binding, err := t.restoreMarkerBinding(ctx, c, body) + if err != nil { + return false, err + } + status, found, err := t.config.RestoreDispatch.InspectRestoreMarker(ctx, binding) + if err != nil { + return false, err + } + if !found || status.State != libraryrelease.RestoreDispatchPrepared || status.Held { + return false, nil + } + for _, phase := range []string{"restore-claimed", "restore-finalize-intent", "restored"} { + effect, err := t.loadIndexEffect(c, phase) + if err != nil { + return false, err + } + if effect != nil { + return false, fmt.Errorf("%w: prepared 台账却已有 %s 证据", libraryrelease.ErrRestoreDispatchHeld, phase) + } + } + if err := t.beginRestoreMarker(ctx, c, applied); err != nil { + return false, err + } + return true, nil +} + // finishRestoreCleanWrite 派发"标记体 -> 冻结 before-image"的 CAS 收尾,并把回执与 // 实时观察逐位核对。finalize-intent 存在时,只有标记体仍停在它自己的世代上才会 // 走到这里:落地过干净写入必然换掉内容并推进世代。 diff --git a/internal/releaseadapter/v071/library_index_restore_dispatch.go b/internal/releaseadapter/v071/library_index_restore_dispatch.go index a5868bd..78badaa 100644 --- a/internal/releaseadapter/v071/library_index_restore_dispatch.go +++ b/internal/releaseadapter/v071/library_index_restore_dispatch.go @@ -7,6 +7,7 @@ import ( "encoding/json" "errors" "fmt" + "os" "knowledge-governance/internal/libraryrelease" ) @@ -19,18 +20,32 @@ import ( // 明确不保证:ES 端 fence;已 dispatching 的旧 owner 绝不迟到一次。协议只保证 // 遵守协议的派发者不会得到第二次许可,且 dispatching 永不回退。 -func (t *ProductionLibraryTarget) restoreMarkerBinding(c libraryrelease.IndexWrite, body json.RawMessage) (libraryrelease.RestoreMarkerBinding, error) { +// restoreMarkerBinding 用**原持久锚点**构造不可变绑定: +// - scope 来自真实装配; +// - source 是发布基线(WriteSet.TargetSnapshotID),backup 是本次写前镜像; +// - target instance/fingerprint/index UUID 实时观察并与本次恢复的冻结值比较。 +func (t *ProductionLibraryTarget) restoreMarkerBinding(ctx context.Context, c libraryrelease.IndexWrite, body json.RawMessage) (libraryrelease.RestoreMarkerBinding, error) { if t.config.RestoreDispatch == nil { return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch protocol is not wired") } if t.activePlan == nil { return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch requires the frozen plan") } - if t.config.RestoreTargetIdentity == "" || t.config.RestoreOperationID == "" || - t.config.RestoreWindowJobID == "" || t.config.RestoreAttemptOwner == "" || - t.config.RestoreAttemptEpoch <= 0 { + if t.config.RestoreTargetIdentity == "" || t.config.RestoreInstance == "" || + t.config.RestoreOperationID == "" || t.config.RestoreWindowJobID == "" || + t.config.RestoreAttemptOwner == "" || t.config.RestoreAttemptEpoch <= 0 { return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch lacks the original job/attempt identity") } + if t.restoreScope == "" || t.restoreSourceSnapshot == "" || t.restoreBackupSnapshot == "" { + return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch lacks the original scope/snapshot anchors") + } + if t.restoreSourceSnapshot == t.restoreBackupSnapshot { + return libraryrelease.RestoreMarkerBinding{}, errors.New("restore dispatch would use one snapshot as both source and backup") + } + fingerprint, indexUUID, err := t.frozenRestoreTarget(ctx) + if err != nil { + return libraryrelease.RestoreMarkerBinding{}, err + } changeRaw, err := json.Marshal(c) if err != nil { return libraryrelease.RestoreMarkerBinding{}, err @@ -52,14 +67,19 @@ func (t *ProductionLibraryTarget) restoreMarkerBinding(c libraryrelease.IndexWri DocumentID: c.ID, Phase: libraryrelease.RestoreMarkerPhaseCreate, }, - Scope: t.config.RestoreScope, - SourceSnapshot: t.config.RestoreSnapshotID, - BackupSnapshot: t.config.RestoreSnapshotID, - ChangeDigest: hex.EncodeToString(changeDigest[:]), - MarkerDigest: hex.EncodeToString(markerDigest[:]), - RequestDigest: hex.EncodeToString(requestDigest[:]), - Owner: t.config.RestoreAttemptOwner, - Epoch: t.config.RestoreAttemptEpoch, + Scope: t.restoreScope, + SourceSnapshot: t.restoreSourceSnapshot, + BackupSnapshot: t.restoreBackupSnapshot, + TargetInstance: t.config.RestoreInstance, + TargetFingerprint: fingerprint, + IndexUUID: indexUUID, + RecoveryJobID: t.config.RestoreRecoveryJobID, + RecoveryOperationID: t.config.RestoreRecoveryOperationID, + ChangeDigest: hex.EncodeToString(changeDigest[:]), + MarkerDigest: hex.EncodeToString(markerDigest[:]), + RequestDigest: hex.EncodeToString(requestDigest[:]), + Owner: t.config.RestoreAttemptOwner, + Epoch: t.config.RestoreAttemptEpoch, }, nil } @@ -67,15 +87,44 @@ func heldRestoreMarker(status libraryrelease.RestoreDispatchStatus, context stri return fmt.Errorf("%w: %s(state=%s)", libraryrelease.ErrRestoreDispatchHeld, context, status.State) } +// planEffect 按 index/document 找到当前冻结计划里的 effect(用于权威全集与文件证据对照)。 +func (t *ProductionLibraryTarget) planEffect(index, document string) (libraryrelease.IndexWrite, bool) { + for _, c := range t.activePlan.Index { + if c.Index == index && c.ID == document { + return c, true + } + } + return libraryrelease.IndexWrite{}, false +} + // admitRestorePlan 是 Restore 在**任何** PG/对象/索引写之前的纯读准入。 // // 删除型恢复一旦开始写 recovery 的行/对象,后面才发现某个 effect 的派发结果未知, -// 目标已经从"未动"变成"半恢复"。所以先按整份冻结计划逐 effect 判定: +// 目标已经从"未动"变成"半恢复"。所以先按整份冻结计划逐 effect 判定,并先对照原 +// window 的权威台账全集(跨 plan): // - 台账不存在(旧世界):只有"没有任何恢复意图"才算首次恢复;有意图就 held; -// - prepared:唯一可续作区段; +// - prepared:唯一可续作区段,但出现 marker/claimed/finalize/restored 任一矛盾即 held; // - dispatching:只有精确 marker+世代才可走确认分支,否则 held; // - marker_confirmed:只允许确认后的 clean 路径。 func (t *ProductionLibraryTarget) admitRestorePlan(ctx context.Context) error { + gate := t.config.RestoreDispatch + if gate != nil { + records, err := gate.UnresolvedRestoreDispatches(ctx, t.config.RestoreTargetIdentity, + t.config.RestoreOperationID, t.config.RestoreWindowJobID) + if err != nil { + return fmt.Errorf("恢复准入读不到权威派发台账:%w", err) + } + for _, record := range records { + if record.PlanDigest != t.activePlan.Digest { + return fmt.Errorf("%w: 原 window 存在另一份计划的派发记录(plan=%s index=%s doc=%s)", + libraryrelease.ErrRestoreDispatchHeld, record.PlanDigest, record.Index, record.DocumentID) + } + if _, ok := t.planEffect(record.Index, record.DocumentID); !ok { + return fmt.Errorf("%w: 原 window 存在计划外派发记录(index=%s doc=%s state=%s)", + libraryrelease.ErrRestoreDispatchHeld, record.Index, record.DocumentID, record.State) + } + } + } for _, c := range t.activePlan.Index { if len(c.After) != 0 { continue @@ -88,11 +137,66 @@ func (t *ProductionLibraryTarget) admitRestorePlan(ctx context.Context) error { if err != nil { return err } + claimed, err := t.loadIndexEffect(c, "restore-claimed") + if err != nil { + return err + } + finalize, err := t.loadIndexEffect(c, "restore-finalize-intent") + if err != nil { + return err + } // 已完成 clean 的 effect 是终态:恢复内容与证据都在时,重放不是未决。 done, err := t.loadIndexEffect(c, "restored") if err != nil { return err } + if gate != nil { + binding, bindErr := t.restoreMarkerBinding(ctx, c, body) + if bindErr != nil { + return bindErr + } + status, found, inspErr := gate.InspectRestoreMarker(ctx, binding) + if inspErr != nil { + return fmt.Errorf("恢复准入读不到派发台账:%w", inspErr) + } + if found { + if done != nil { + if status.State != libraryrelease.RestoreDispatchMarkerConfirmed { + return heldRestoreMarker(status, "已完成 clean 的 effect 台账却不是 marker_confirmed") + } + // 终态:内容与世代由下面的 done 分支核对。 + } else { + switch status.State { + case libraryrelease.RestoreDispatchPrepared: + if status.Held { + return heldRestoreMarker(status, "派发绑定与不可变计划冲突") + } + if claimed != nil || finalize != nil { + return heldRestoreMarker(status, "prepared 台账却已有 claimed/finalize 证据") + } + observed, obsErr := t.observePlannedIndex(ctx, c) + if obsErr != nil { + return obsErr + } + if indexBytesEqual(observed, body) { + return heldRestoreMarker(status, "prepared 台账却已存在目标标记:协议证据矛盾") + } + continue + case libraryrelease.RestoreDispatchDispatching, libraryrelease.RestoreDispatchMarkerConfirmed: + observed, obsErr := t.observePlannedIndex(ctx, c) + if obsErr != nil { + return obsErr + } + if indexBytesEqual(observed, body) && observed.SeqNo != nil && observed.PrimaryTerm != nil { + continue // 只允许原确认分支继续,不重发 create + } + return heldRestoreMarker(status, "目标上缺少可归属的精确恢复标记") + default: + return heldRestoreMarker(status, "派发台账状态不可识别") + } + } + } + } if done != nil { observed, err := t.observePlannedIndex(ctx, c) if err != nil { @@ -104,44 +208,17 @@ func (t *ProductionLibraryTarget) admitRestorePlan(ctx context.Context) error { } continue } - if t.config.RestoreDispatch != nil { - binding, bindErr := t.restoreMarkerBinding(c, body) - if bindErr != nil { - return bindErr - } - status, found, inspErr := t.config.RestoreDispatch.InspectRestoreMarker(ctx, binding) - if inspErr != nil { - return fmt.Errorf("恢复准入读不到派发台账:%w", inspErr) - } - if found { - switch status.State { - case libraryrelease.RestoreDispatchPrepared: - continue - case libraryrelease.RestoreDispatchDispatching, libraryrelease.RestoreDispatchMarkerConfirmed: - observed, obsErr := t.observePlannedIndex(ctx, c) - if obsErr != nil { - return obsErr - } - if len(c.After) == 0 && indexBytesEqual(observed, body) && observed.SeqNo != nil && observed.PrimaryTerm != nil { - continue // 只允许原确认分支继续,不重发 create - } - return heldRestoreMarker(status, "目标上缺少可归属的精确恢复标记") - default: - return heldRestoreMarker(status, "派发台账状态不可识别") - } - } - } - if intent == nil { + if intent == nil && claimed == nil && finalize == nil { continue } observed, err := t.observePlannedIndex(ctx, c) if err != nil { return err } - if indexBytesEqual(observed, body) { + if intent != nil && indexBytesEqual(observed, body) { continue // 旧标记已在目标上:按既有确认/CAS 规则处理,不派发新 create } - return fmt.Errorf("%w: 既存恢复意图没有新协议派发台账,无法证明未派发", libraryrelease.ErrRestoreDispatchHeld) + return fmt.Errorf("%w: 既存恢复意图/证据没有新协议派发台账,无法证明未派发", libraryrelease.ErrRestoreDispatchHeld) } return nil } @@ -152,7 +229,7 @@ func (t *ProductionLibraryTarget) confirmDispatchedMarker(ctx context.Context, c if t.config.RestoreDispatch == nil || len(c.After) != 0 { return nil } - binding, err := t.restoreMarkerBinding(c, body) + binding, err := t.restoreMarkerBinding(ctx, c, body) if err != nil { return err } @@ -163,6 +240,9 @@ func (t *ProductionLibraryTarget) confirmDispatchedMarker(ctx context.Context, c if !found { return fmt.Errorf("%w: 删除型恢复缺少派发台账,拒绝按外观继续", libraryrelease.ErrRestoreDispatchHeld) } + if status.Held && status.State == libraryrelease.RestoreDispatchPrepared { + return heldRestoreMarker(status, "派发绑定与不可变计划冲突") + } switch status.State { case libraryrelease.RestoreDispatchPrepared: // 还没派发:由调用方按 prepared 竞争许可。 @@ -190,7 +270,7 @@ func (t *ProductionLibraryTarget) dispatchRestoreMarkerCreate(ctx context.Contex if t.config.RestoreDispatch == nil { return indexReceipt{}, errors.New("恢复标记派发协议未接线:拒绝发送") } - binding, err := t.restoreMarkerBinding(c, body) + binding, err := t.restoreMarkerBinding(ctx, c, body) if err != nil { return indexReceipt{}, err } @@ -198,13 +278,13 @@ func (t *ProductionLibraryTarget) dispatchRestoreMarkerCreate(ctx context.Contex if err != nil { return indexReceipt{}, err } - if !found || status.State != libraryrelease.RestoreDispatchPrepared { - if err != nil { - return indexReceipt{}, err - } + if !found || status.State != libraryrelease.RestoreDispatchPrepared || status.Held { return indexReceipt{}, heldRestoreMarker(status, "没有可竞争的 prepared 许可") } // 控制流故障注入必须在 CAS 之前:拒绝时状态仍是 prepared,可安全重试,零转发。 + if err := t.ds01TestRefusal(c); err != nil { + return indexReceipt{}, err + } if err := t.refuseRestoreWrite(c); err != nil { return indexReceipt{}, err } @@ -255,9 +335,39 @@ func (t *ProductionLibraryTarget) sendRestoreMarkerOnce(ctx context.Context, c l return receipt, nil } -// verifyRestoreEffectsClosed 是 Restore 收尾前的最后一道检查:任何一个删除型 effect -// 只要留下恢复意图,就必须已经完成 clean(restored 证据),且台账不得停在 dispatching。 +// verifyRestoreEffectsClosed 是 Restore 收尾前的最后一道检查:以原 window 的权威台账 +// 全集为准(跨 plan),每个记录都必须是 marker_confirmed,且对应的 restored 文件证据 +// 存在。prepared/dispatching、计划外 effect、换 plan 别名、缺 restored 一律拒绝。 func (t *ProductionLibraryTarget) verifyRestoreEffectsClosed(ctx context.Context) error { + if t.config.RestoreDispatch != nil { + records, err := t.config.RestoreDispatch.UnresolvedRestoreDispatches(ctx, + t.config.RestoreTargetIdentity, t.config.RestoreOperationID, t.config.RestoreWindowJobID) + if err != nil { + return fmt.Errorf("恢复收尾读不到权威派发台账:%w", err) + } + for _, record := range records { + if record.PlanDigest != t.activePlan.Digest { + return fmt.Errorf("%w: 收尾时原 window 仍有另一份计划的派发记录(plan=%s)", + libraryrelease.ErrRestoreDispatchHeld, record.PlanDigest) + } + if record.State != libraryrelease.RestoreDispatchMarkerConfirmed { + return fmt.Errorf("%w: 收尾前派发台账仍有未决 effect(index=%s doc=%s state=%s)", + libraryrelease.ErrRestoreDispatchHeld, record.Index, record.DocumentID, record.State) + } + c, ok := t.planEffect(record.Index, record.DocumentID) + if !ok { + return fmt.Errorf("%w: 收尾时存在计划外派发记录(index=%s doc=%s)", + libraryrelease.ErrRestoreDispatchHeld, record.Index, record.DocumentID) + } + done, err := t.loadIndexEffect(c, "restored") + if err != nil { + return err + } + if done == nil { + return fmt.Errorf("%w: 收尾前仍有未 clean 的 effect %s", libraryrelease.ErrRestoreDispatchHeld, c.ID) + } + } + } for _, c := range t.activePlan.Index { if len(c.After) != 0 { continue @@ -276,24 +386,21 @@ func (t *ProductionLibraryTarget) verifyRestoreEffectsClosed(ctx context.Context if done == nil { return fmt.Errorf("%w: 恢复收尾前仍有未决 effect %s", libraryrelease.ErrRestoreDispatchHeld, c.ID) } - if t.config.RestoreDispatch == nil { - continue - } - body, err := t.restoreIndexClaimBody(c) - if err != nil { - return err - } - binding, err := t.restoreMarkerBinding(c, body) - if err != nil { - return err - } - status, found, err := t.config.RestoreDispatch.InspectRestoreMarker(ctx, binding) - if err != nil { - return err - } - if found && status.State == libraryrelease.RestoreDispatchDispatching { - return heldRestoreMarker(status, "收尾前台账仍停在 dispatching") - } } return nil } + +// ds01TestRefusal 是正式 worker 全链验收的最小测试接缝:只有显式隔离运行 +// (KG_LIBRARY_TEST_ISOLATED=1)且指定了 effect 时才在 CAS 之前拒绝一次;生产默认 +// 全关,且每个进程只触发一次,重启后的 worker 不会重复拒绝。 +func (t *ProductionLibraryTarget) ds01TestRefusal(c libraryrelease.IndexWrite) error { + if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" || t.restoreRefusalUsed { + return nil + } + spec := os.Getenv("KG_DS01_TEST_REFUSE_MARKER") + if spec == "" || spec != c.Index+"/"+c.ID { + return nil + } + t.restoreRefusalUsed = true + return errors.New("injected DS01 marker refusal before dispatch CAS") +} diff --git a/internal/releaseadapter/v071/library_production_target.go b/internal/releaseadapter/v071/library_production_target.go index 8274888..9beddea 100644 --- a/internal/releaseadapter/v071/library_production_target.go +++ b/internal/releaseadapter/v071/library_production_target.go @@ -63,12 +63,14 @@ type ProductionLibraryConfig struct { RestoreDispatch libraryrelease.RestoreMarkerGate RestoreTargetIdentity string RestoreScope string + RestoreInstance string RestoreOperationID string RestoreWindowJobID string RestoreAttemptOwner string RestoreAttemptEpoch int64 - // RestoreSnapshotID 是本次恢复要用的那份预备份(与原 window/plan 绑定)。 - RestoreSnapshotID string + // 恢复分发身份:只在恢复 worker 里非空;普通自动回滚为空。 + RestoreRecoveryJobID string + RestoreRecoveryOperationID string } func (config ProductionLibraryConfig) validate(scope ProductionScope) error { @@ -114,6 +116,22 @@ type ProductionLibraryTarget struct { preserveChunkSequence bool activePlan *libraryrelease.WriteSet activeSnapshot string + // restoreScope / restoreSourceSnapshot / restoreBackupSnapshot 在 Restore 进入时 + // 从**原持久锚点**冻结:scope 来自真实装配,来源快照与写前镜像来自本次 WriteSet + // 与原 backup 引用。它们不能由调用方在恢复中途改写,也不允许同一个快照既当 + // 来源又当 before-image。 + restoreScope string + restoreSourceSnapshot string + restoreBackupSnapshot string + // restoreFingerprint / restoreIndexUUID 是本次恢复冻结的实际目标身份。派发绑定 + // 每次重建时都实时重读并比较:同名重建的目标/索引不可能复用旧证明。 + restoreFingerprint string + restoreIndexUUID string + // restoreIntentFault 是一次性文件写失败注入点(生产恒为 nil):在 prepared 已提交、 + // mark-intent 尚未落盘处返回错误,用来复现"可补同内容"的窄故障区段。 + restoreIntentFault func(libraryrelease.IndexWrite) error + // restoreRefusalUsed 是测试专用接缝的一次性标记(生产恒为 false)。 + restoreRefusalUsed bool // restoreMarkerFault 是一次性控制流故障注入点(生产恒为 nil):它在恢复标记的 // 持久意图落盘之后、网络派发之前返回错误,用来区分"边界上确定没写进去"与 // "写入结果未知"——前者重试时只凭世代证据重做,后者仍按 B19 永久 held。 @@ -971,6 +989,18 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel return fmt.Errorf("写集合记录绑定的是 %d/%s,与目标 %d/%s 不一致", backup.WriteSet.TenantID, backup.WriteSet.KnowledgeBaseID, t.scope.TenantID, t.scope.KnowledgeBaseID) } + // 派发绑定只能来自原锚点:来源快照是发布基线,backup 是这份写集的写前镜像。 + // 二者缺失或相同都说明装配没有拿到真实身份,必须在任何写之前失败关闭。 + if backup.WriteSet.TargetSnapshotID == "" || backup.SnapshotID == "" { + return errors.New("恢复缺少来源快照或写前镜像,拒绝建立派发绑定") + } + if backup.WriteSet.TargetSnapshotID == backup.SnapshotID { + return errors.New("来源快照与写前镜像不能是同一份快照,拒绝建立派发绑定") + } + t.restoreScope = t.config.RestoreScope + t.restoreSourceSnapshot = backup.WriteSet.TargetSnapshotID + t.restoreBackupSnapshot = backup.SnapshotID + t.restoreFingerprint, t.restoreIndexUUID = "", "" if err := t.AdoptWriteSet(*backup.WriteSet); err != nil { return err } diff --git a/internal/releaseadapter/v071/library_restore_target_identity.go b/internal/releaseadapter/v071/library_restore_target_identity.go new file mode 100644 index 0000000..dcb2789 --- /dev/null +++ b/internal/releaseadapter/v071/library_restore_target_identity.go @@ -0,0 +1,56 @@ +package v071 + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "strings" +) + +// 恢复派发只接受"实际目标"的身份:租户/KB、目标库(服务器+库名)、索引名与索引 +// UUID、对象桶。它们用于识别"同名但被重建/替换"的目标,让旧绑定无法复用;它们不是 +// 历史未发送的证明,也不构成 ES 端 fence。 +func (t *ProductionLibraryTarget) observeRestoreTarget(ctx context.Context) (string, string, error) { + if t.pool == nil || t.search == nil { + return "", "", errors.New("restore dispatch requires a live target") + } + var database, address string + var port int + if err := t.pool.QueryRow(ctx, + `SELECT current_database(), COALESCE(host(inet_server_addr()), ''), COALESCE(inet_server_port(), 0)`). + Scan(&database, &address, &port); err != nil { + return "", "", fmt.Errorf("read target database identity: %w", err) + } + if database == "" { + return "", "", errors.New("target database identity is empty") + } + indexUUID, err := t.search.IndexUUID(ctx) + if err != nil { + return "", "", fmt.Errorf("read target index uuid: %w", err) + } + joined := strings.Join([]string{ + "v1", fmt.Sprintf("%d", t.scope.TenantID), t.scope.KnowledgeBaseID, + database, address, fmt.Sprintf("%d", port), t.config.SearchIndex, indexUUID, t.config.ObjectsBucket, + }, "\x00") + sum := sha256.Sum256([]byte(joined)) + return hex.EncodeToString(sum[:]), indexUUID, nil +} + +// frozenRestoreTarget 在一个 Restore 调用内冻结实际目标身份;后续每次重建绑定都 +// 重新观察并与冻结值比较。目标/索引在恢复中途被替换时立即失败,而不是继续用旧绑定。 +func (t *ProductionLibraryTarget) frozenRestoreTarget(ctx context.Context) (string, string, error) { + fingerprint, indexUUID, err := t.observeRestoreTarget(ctx) + if err != nil { + return "", "", err + } + if t.restoreFingerprint == "" { + t.restoreFingerprint, t.restoreIndexUUID = fingerprint, indexUUID + return fingerprint, indexUUID, nil + } + if fingerprint != t.restoreFingerprint || indexUUID != t.restoreIndexUUID { + return "", "", fmt.Errorf("%w: 目标/索引身份在恢复过程中发生变化", ErrProductionTargetUnconfigured) + } + return t.restoreFingerprint, t.restoreIndexUUID, nil +} diff --git a/internal/releaseadapter/v071/search.go b/internal/releaseadapter/v071/search.go index b127ca9..a407fe2 100644 --- a/internal/releaseadapter/v071/search.go +++ b/internal/releaseadapter/v071/search.go @@ -278,6 +278,25 @@ func (exporter *SearchExporter) requestJSON(ctx context.Context, method, path st return exporter.requestJSONWithReplay(ctx, method, path, payload, result, true) } +// IndexUUID 读目标索引的实际 UUID。同名重建会得到新 UUID —— 恢复派发用它识别 +// "目标被替换",不是用它证明历史未发送。索引不存在按 errIndexMissing 返回。 +func (exporter *SearchExporter) IndexUUID(ctx context.Context) (string, error) { + var rows []struct { + Index string `json:"index"` + UUID string `json:"uuid"` + } + path := "/_cat/indices/" + url.PathEscape(exporter.index) + "?format=json&h=index,uuid" + if err := exporter.requestJSON(ctx, http.MethodGet, path, nil, &rows); err != nil { + return "", err + } + for _, row := range rows { + if row.Index == exporter.index && row.UUID != "" { + return row.UUID, nil + } + } + return "", errors.New("target index uuid is missing") +} + // requestJSONOnce 关闭传输层对该请求的重放能力(GetBody=nil):恢复标记 create // 每个许可只能落地至多一次真实请求,不能让 keep-alive 重连把它悄悄发第二遍。 // Go 的 Transport 只会自动重试"可重放"的请求;Body 非空且 GetBody 为 nil 时,