fix(library): 守卫按行取作业身份、同作业多代次、原作业续作与执行域准入
build / build (push) Successful in 1m27s

R1 维护/发布守卫硬编码 j.id=$1:维护类入口的 $1 是 scope/instance,作业号在 $2,
导致 update/release/ingress 六个入口正常路径全部匹配不到作业。改为按被更新行自己的
owner_job_id 生成条件,并同时要求该代次在 library_attempts 里仍活跃。

R2 认领只改 jobs.attempt_epoch 不改 attempts.epoch,第二次接管即错位;唯一索引
(job_id) 与"每次接管新 attempt 且历史不可变"矛盾。改为同作业多代次:锁作业行与
operation 行、同 owner 幂等续租、活跃未过期者让路、过期/本作业旧代次标 abandoned、
新代次 epoch=max+1,并清同 operation 其它作业行租约。迟到 write/renew/finish 一律
ErrLibraryEpochStale(finish 补了"只能在活跃态进来一次")。

R3 续作不再新建 -resume 作业:PrepareLibraryJobResume 让原业务作业回 pending 并显式
结束上一代,维护窗口/journal/预备份/写集合仍按原作业 id 归属,代次由 worker 起手递增。

R4 执行域准入改为注册 handler 前读事实:目标实况(KB/租户/schema/索引)、旧 release
路由绑定的 KB、未释放维护窗口持有者;未知或重叠即拒绝执行(不做维护动作、不停既有服务)。

同时自纠 5 项设计偏离:current 次序键取意图创建时间(旧意图迟到事实被拒)、首次锚点
改事务 FOR UPDATE + 回读且 NULL≠0、版本事件冲突只做加法且 xmax=0 才算新事实、
恢复作为新 kind=recover 意图推进 current、隔离测试加测试库护栏并清理残留 schema。

测试:真实 PG(kg-pgtest-writeset,一用例一 schema)新增 8 条集成用例覆盖上述正负例;
go test ./... 全绿(50 包)、go vet/build rc=0、web tsc + vitest 485 通过、make verify 16 项通过。
This commit is contained in:
姚章浩
2026-10-30 17:20:40 +08:00
parent afe9affc2a
commit a7d643635a
26 changed files with 1551 additions and 215 deletions
+8
View File
@@ -9,6 +9,8 @@
- 按预备份恢复成功后登记**恢复事实**并推进 current:页面、差异与后续作业看到的是恢复出来的内容身份,而不是继续宣称已失败的新版本生效;原失败作业与失败版本保持不变。
- 接管或恢复时,执行目标消费已持久化写集合里的复用决定;**读不懂计划的实现直接拒绝写入**(不再"账本说没动共享对象、实际又重写一遍")。
- 预备份合同改为目标实时状态(schema/模型/映射),不再把来源快照的声明当目标合同。
- **续作沿用原业务作业,只新增执行代次**:维护窗口 owner、freeze journal、预备份与写集合都按业务作业 id 落盘,因此不再为续作另造 `-resume` 作业行;发起续作会显式结束上一代执行(旧代次标记 superseded,行保留)。
- 执行域准入改为**可执行校验**:worker 注册整库发布 handler 之前,读目标实况(知识库/租户/schema/索引)、旧 release 路由绑定的目标知识库、以及未释放的维护窗口持有者;任一读不到、与配置不一致、或旧路由指向同一个知识库,就拒绝执行(明确失败,不做任何维护动作,也不停止既有服务)。
### Fixed
@@ -23,6 +25,12 @@
- 全库搬运改写定位符时重算 `location_hash = sha256(最终 physical_path)`,与文档级导入共用同一实现。
- 整库发布在副作用前核对目标**实时**模型契约:同维不同模型、模型不可读、dirty schema 一律拒绝。
- 容量门禁接到 worker 执行前(预算不足或读不到即拒绝);存储预览按全量依赖判定保护,保护依据读不到时拒绝生成可删候选,备份类型只认显式登记。
- 维护状态/入口/释放的守卫**按被更新行自己的作业号**取身份(此前把 `$1` 硬编码成作业号,而维护类入口的 `$1` 是 scope/instance,导致正常写入与释放全部匹配不到作业);同一代码路径同时用"作业号不匹配必须失败"的负例锁住。
- 同一作业允许**多个不可变执行代次**:认领递增 epoch 并新增 attempt 行(不再复用唯一索引改写旧代次),重复调用幂等续租,别人持有未过期租约时明确让路;被接管的旧代次标记 abandoned,迟到的 write/renew/finish 一律按 `ErrLibraryEpochStale` 拒绝。
- 首次锚点登记改为事务内 `FOR UPDATE` + 只填空 + 逐字段回读核对;`NULL` 与零值不再混淆(真实 PG 上验证,未知 ≠ 0)。
- current 的次序键取**意图创建时间**:旧意图的迟到事实被 `ErrLibraryCurrentStale` 拒绝,不会把当前生效版本推回历史;恢复通过新的 `kind=recover` 意图(内容更旧、意图更新)推进。
- 版本事件冲突只做**加法**(已有键永远赢、来源作业/意图/登记时刻冻结),且只有真正新插入(`xmax=0`)才推进 current,纯重放不再二次推进。
- 集成测试护栏:`TEST_DATABASE_URL` 必须指向看起来是一次性的测试库(或在确属一次性库时显式声明),并先清理上次中断留下的同名 schema;`kg-pgtest-*` 专用容器内一用例一 schema,结束即删。
- 发布页:状态读取核 HTTP 状态与响应结构;按预备份恢复纳入轮询并在维护关闭后读回终态;Wiki 差异覆盖标题/摘要/父级变化。
- CI 版本契约门同时核进程退出码、必需测试真 PASS 与负例失败归因(编译/网络错误不算预期断言生效)。
+28 -17
View File
@@ -51,30 +51,41 @@ func TestPrebackupVersionKeepsFirstRegistrationTimeOnReuse(t *testing.T) {
t.Fatalf("复用的预览份没有 backup_reused=true:%v", version.Evidence)
}
// 反例对照:生效版本行的"重新激活"必须照旧写当前时刻 ——
// 上面那条 CASE 不许漏到别的 kind 上,否则恢复一个旧快照后
// 页面会把"最近一次生效"显示成很早以前。
// 反例对照:真的"重新生效"是**新的意图**(新的作业/新的 operation),
// 它带自己的登记时刻;同一意图里的重复登记则只补证据、时刻不变。
// 这条区分必须成立,否则页面会把"最近一次生效"显示成很早以前。
const effectiveSnap = "snap-effective-reactivated"
effectiveAt := first.Add(time.Hour)
reactivatedAt := effectiveAt.Add(time.Hour)
for _, at := range []time.Time{effectiveAt, reactivatedAt} {
if err := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionID(governance.LibraryVersionProductionEffective, scope, effectiveSnap),
Scope: scope, Kind: governance.LibraryVersionProductionEffective, SnapshotID: effectiveSnap,
SourceJobID: "librestore-internal-reactivate", CreatedBy: "worker",
Evidence: map[string]any{"action": "restore"}, CreatedAt: at,
}); err != nil {
t.Fatalf("record effective at %s: %v", at, err)
}
}
opFirst := testLibraryOperation(t, store, scope, governance.LibraryJobSync, "activate-1", effectiveAt)
recordVersionEvent(t, store, scope, governance.LibraryVersionProductionEffective, effectiveSnap,
"librestore-internal-reactivate-1", opFirst, effectiveAt, map[string]any{"action": "restore"})
// 同一意图内的重复登记:时刻不变(事件不可变)。
recordVersionEvent(t, store, scope, governance.LibraryVersionProductionEffective, effectiveSnap,
"librestore-internal-reactivate-1", opFirst, reactivatedAt, map[string]any{"action": "restore"})
effective, found, err := store.LibraryVersionByRef(ctx, scope, governance.LibraryVersionProductionEffective,
"librestore-internal-reactivate", effectiveSnap)
"librestore-internal-reactivate-1", effectiveSnap)
if err != nil || !found {
t.Fatalf("read effective version: found=%t err=%v", found, err)
}
if !effective.CreatedAt.UTC().Equal(reactivatedAt) {
t.Fatalf("生效版本重新激活后时刻 = %s, want %s(prebackup 的保留规则漏到别的 kind 上了)",
effective.CreatedAt.UTC(), reactivatedAt)
if !effective.CreatedAt.UTC().Equal(effectiveAt) {
t.Fatalf("同一意图的重复登记改写了时刻:%s, want %s", effective.CreatedAt.UTC(), effectiveAt)
}
// 新的意图(另一个作业)重新激活同一快照:新事件、新时刻、成为当前。
opSecond := testLibraryOperation(t, store, scope, governance.LibraryJobSync, "activate-2", reactivatedAt)
recordVersionEvent(t, store, scope, governance.LibraryVersionProductionEffective, effectiveSnap,
"librestore-internal-reactivate-2", opSecond, reactivatedAt, map[string]any{"action": "restore"})
reactivated, found, err := store.LibraryVersionByRef(ctx, scope,
governance.LibraryVersionProductionEffective, "librestore-internal-reactivate-2", effectiveSnap)
if err != nil || !found {
t.Fatalf("read reactivated version: found=%t err=%v", found, err)
}
if !reactivated.CreatedAt.UTC().Equal(reactivatedAt) {
t.Fatalf("重新激活的登记时刻 = %s, want %s", reactivated.CreatedAt.UTC(), reactivatedAt)
}
current, hasCurrent, err := store.LibraryCurrentVersion(ctx, scope)
if err != nil || !hasCurrent || current.ID != reactivated.ID {
t.Fatalf("当前生效版本应指向重新激活的事件:found=%v err=%v %+v", hasCurrent, err, current)
}
}
+1 -1
View File
@@ -111,7 +111,7 @@ func runLibraryBaseline(ctx context.Context, base *postgresstore.Store, config a
if _, admitErr := admitLibraryJob(ctx, base, job, payload.LibraryJobID, payload.Scope, governance.LibraryJobBaseline, true); admitErr != nil {
return admitErr
}
writer, attemptErr := beginLibraryWriter(ctx, base, payload.LibraryJobID, libraryQueueOwner(job, payload.LibraryJobID))
writer, attemptErr := beginLibraryWriter(ctx, base, payload.LibraryJobID, libraryQueueOwner(config.WorkerID, job, payload.LibraryJobID))
if attemptErr != nil {
return attemptErr
}
+8 -5
View File
@@ -123,12 +123,14 @@ func rebuildSuccessBoundary(ctx context.Context, store *libraryAttemptWriter, re
// overview/diff/provenance 仍然宣称失败的新版本生效,后续操作拿错误基线做判断。
//
// 它是独立事件,不改写原来那条失败作业/失败版本;原失败事实保持不变。
func recordRecoveryEvent(ctx context.Context, store *libraryAttemptWriter, scope, jobID, backupSnapshot,
verifyDigest, note string) error {
func recordRecoveryEvent(ctx context.Context, store *libraryAttemptWriter, operation governance.LibraryOperation,
scope, jobID, backupSnapshot, verifyDigest, note string) error {
if backupSnapshot == "" || verifyDigest == "" {
return errors.New("恢复事件缺少预备份身份或校验摘要:不能登记成恢复事实")
}
operationID := store.attempt.OperationID
// 事件挂在**恢复意图**上(不是失败作业的旧意图):current 的次序按意图创建时间,
// 旧意图的迟到事实推不动当前状态。
operationID := operation.ID
evidence := map[string]any{
"backup_snapshot": backupSnapshot, "action": "recover", "verify_digest": verifyDigest,
"window_job": jobID, "note": note, "recovered_at": time.Now().UTC(),
@@ -143,9 +145,10 @@ func recordRecoveryEvent(ctx context.Context, store *libraryAttemptWriter, scope
}
return store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionEventID(governance.LibraryVersionRecoveryVerified, scope, backupSnapshot,
operationID+"|"+jobID),
operationID),
Scope: scope, Kind: governance.LibraryVersionRecoveryVerified, SnapshotID: backupSnapshot,
SourceJobID: jobID, CreatedBy: "worker", Evidence: evidence, CreatedAt: time.Now().UTC(),
SourceJobID: jobID, OperationID: operationID, AttemptEpoch: store.attempt.Epoch,
CreatedBy: "worker", Evidence: evidence, CreatedAt: operation.CreatedAt,
})
}
+104
View File
@@ -0,0 +1,104 @@
package main
import (
"context"
"errors"
"fmt"
"log"
"strings"
"knowledge-governance/internal/appconfig"
"knowledge-governance/internal/jobs"
"knowledge-governance/internal/libraryadmission"
postgresstore "knowledge-governance/internal/platform/postgres"
v071 "knowledge-governance/internal/releaseadapter/v071"
)
// verifyLibraryExecutionDomain 在注册整库发布 handler 之前核对"这个进程能不能独占目标"。
//
// 为什么不在 API 里做:真正的执行者是 worker,能写目标的是它。准入放在执行者起手,
// 才不会被"另一个进程的配置"绕过去。
//
// 判定全部基于**读出来的事实**(目标实况 / 旧的 release 路由绑定 / 未释放的维护窗口),
// 任何一项读不到都按未知处理并拒绝;这里不停止、不修改任何既有服务(停旧 worker 是
// 运维决定),只决定"本进程是否注册整库发布执行器"。
func verifyLibraryExecutionDomain(ctx context.Context, config appconfig.Config, store *postgresstore.Store) error {
if strings.TrimSpace(config.LibraryProductionDatabaseURL) == "" {
return errors.New("整库目标未配置(LIBRARY_PRODUCTION_DATABASE_URL):不注册整库发布执行器")
}
instance := config.LibraryTargetInstance
if instance == "" {
return errors.New("目标实例身份未配置(LIBRARY_TARGET_INSTANCE):无法证明执行域唯一")
}
// 1) 目标实况:连上去读,而不是回声配置。
live, liveErr := v071.ProbeTargetIdentity(ctx, config.LibraryProductionDatabaseURL,
config.LibraryProductionTenantID, config.LibraryProductionInternalKB)
facts := libraryadmission.Facts{LiveErr: liveErr}
if liveErr == nil {
facts.Live = libraryadmission.Target{
DatabaseEndpoint: config.LibraryProductionDatabaseURL, Index: config.LibraryProductionIndex,
Bucket: config.LibraryProductionS3Bucket, KnowledgeBaseID: config.LibraryProductionInternalKB,
TenantID: config.LibraryProductionTenantID, SchemaVersion: live.SchemaVersion,
Instance: instance,
}
facts.LiveDirty = live.Dirty
if live.IndexName != "" {
facts.Live.Index = live.IndexName
}
}
// 2) 旧 release 路由写哪个知识库:production_kb_bindings 是它自己的持久事实。
if internalKB, externalKB, _, bindErr := store.LoadProductionKBBindings(ctx); bindErr != nil {
facts.LegacyErr = bindErr
} else {
// 旧路由把"测试库"发布到 external;两个方向都算重叠。
targetKB := strings.TrimSpace(config.LibraryProductionInternalKB)
for _, candidate := range []string{internalKB, externalKB} {
if candidate != "" && candidate == targetKB {
facts.LegacyReleaseKB = candidate
break
}
}
if facts.LegacyReleaseKB == "" && externalKB != "" {
facts.LegacyReleaseKB = externalKB
}
}
// 3) 未释放的维护窗口:谁此刻持有这个目标。
held, heldErr := store.HeldInstanceMaintenance(ctx)
if heldErr != nil {
facts.HeldErr = heldErr
} else {
for _, row := range held {
if row.ReleasedAt != nil {
continue
}
facts.HeldInstance, facts.HeldJob = row.Instance, row.OwnerJobID
break
}
}
configured := libraryadmission.Target{
DatabaseEndpoint: config.LibraryProductionDatabaseURL, Index: config.LibraryProductionIndex,
Bucket: config.LibraryProductionS3Bucket, KnowledgeBaseID: config.LibraryProductionInternalKB,
TenantID: config.LibraryProductionTenantID, SchemaVersion: 0, // 由目标实况回答;配置不再另声明一份容易过期的版本
Instance: instance,
}
if err := libraryadmission.Verify(configured, facts); err != nil {
return err
}
log.Printf("library execution domain admitted: instance=%s kb=%s schema=%d index=%s",
instance, configured.KnowledgeBaseID, facts.Live.SchemaVersion, configured.Index)
return nil
}
// admittanceGuarded 把一个 handler 包成"准入未通过就拒绝执行"。
//
// 用错误返回而不是不注册:不注册会让作业以 no-handler 进死信、看不出原因;
// 这里让作业**明确失败**并带上准入原因,页面能直接读到,且任何维护动作都没发生。
func admittanceGuarded(admissionErr error, handler func(context.Context, jobs.Job) error) func(context.Context, jobs.Job) error {
if admissionErr == nil {
return handler
}
reason := admissionErr.Error()
return func(context.Context, jobs.Job) error {
return fmt.Errorf("整库发布执行域准入未通过,未做任何维护动作:%s", reason)
}
}
+1 -1
View File
@@ -119,7 +119,7 @@ func runLibraryRelease(ctx context.Context, base *postgresstore.Store, config ap
}
// 执行代次:这一次投递是一个确定的执行者身份。所有治理写入都走 writer
// (同名方法带 epoch 守卫),失租的旧执行从此写不进业务状态/维护锁(F13)。
writer, attemptErr := beginLibraryWriter(ctx, base, payload.LibraryJobID, libraryQueueOwner(job, payload.LibraryJobID))
writer, attemptErr := beginLibraryWriter(ctx, base, payload.LibraryJobID, libraryQueueOwner(config.WorkerID, job, payload.LibraryJobID))
if attemptErr != nil {
return attemptErr
}
@@ -0,0 +1,45 @@
package main
import (
"context"
"testing"
"time"
"knowledge-governance/internal/governance"
postgresstore "knowledge-governance/internal/platform/postgres"
)
// 版本事件的执行身份(父的复核项 R2/F02):一条版本事件属于**一次意图**(operation),
// 不是快照的终身属性。测试要造"同一个快照被另一个作业重新登记",就必须真的换一个意图,
// 而不是靠旧实现的 upsert 改写 source_job_id —— 那条路径现在已经按设计禁止了。
func testLibraryOperation(t *testing.T, store *postgresstore.Store, scope string,
kind governance.LibraryJobKind, key string, at time.Time) string {
t.Helper()
ctx := context.Background()
op, _, err := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{
ID: governance.LibraryOperationID(scope, kind, key), Scope: scope, Kind: kind,
TargetSnapshotID: "target-" + key, RequestDigest: "digest-" + key, OperationKey: key,
Actor: "tester", CreatedAt: at.UTC(),
})
if err != nil {
t.Fatalf("ensure test operation %s: %v", key, err)
}
return op.ID
}
// recordVersionEvent 以给定意图登记一条版本事件(id 与 operation 身份一致)。
func recordVersionEvent(t *testing.T, store *postgresstore.Store, scope string,
kind governance.LibraryVersionKind, snapshot, jobID, operationID string, at time.Time,
evidence map[string]any) {
t.Helper()
if err := store.RecordLibraryVersion(context.Background(), governance.LibraryVersion{
ID: governance.LibraryVersionEventID(kind, scope, snapshot, operationID),
Scope: scope,
Kind: kind,
SnapshotID: snapshot,
SourceJobID: jobID, OperationID: operationID, CreatedBy: "worker",
Evidence: evidence, CreatedAt: at.UTC(),
}); err != nil {
t.Fatalf("record %s version for %s (job %s): %v", kind, snapshot, jobID, err)
}
}
+23 -3
View File
@@ -63,12 +63,32 @@ func runLibraryRecovery(ctx context.Context, store *postgresstore.Store, config
return admitErr
}
// 恢复是一次**新执行代次**:它接管这个窗口,旧代次从此不能写业务状态/释放锁(F13)。
writer, attemptErr := beginLibraryWriter(ctx, store, payload.WindowJobID, libraryQueueOwner(job, payload.WindowJobID))
writer, attemptErr := beginLibraryWriter(ctx, store, payload.WindowJobID, libraryQueueOwner(config.WorkerID, job, payload.WindowJobID))
if attemptErr != nil {
return fmt.Errorf("无法取得恢复窗口的执行权:%w", attemptErr)
}
guarded := writer
prebackup := windowJob.SnapshotID
if prebackup == "" {
return errors.New("原作业没有持久化预备份,无法证明恢复到哪一份")
}
// 恢复是**新意图**(kind=recover):内容更旧、意图更新。把恢复事实挂在失败作业的旧
// operation 上会被 current 的次序(按意图创建时间)判成迟到事实而拒绝——F06 就治不好。
recoveryDigest, digestErr := governance.LibraryRequestDigest(payload.Scope, governance.LibraryJobRecover,
map[string]any{"window_job": payload.WindowJobID, "backup_snapshot": prebackup})
if digestErr != nil {
return fmt.Errorf("无法计算恢复意图摘要:%w", digestErr)
}
recoveryKey := fmt.Sprintf("%s#%d", payload.WindowJobID, writer.attempt.Epoch)
recoveryOp, _, opErr := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{
ID: governance.LibraryOperationID(payload.Scope, governance.LibraryJobRecover, recoveryKey),
Scope: payload.Scope, Kind: governance.LibraryJobRecover, TargetSnapshotID: prebackup,
RequestDigest: recoveryDigest, OperationKey: recoveryKey, Actor: job.LeaseOwner,
CreatedAt: time.Now().UTC(),
})
if opErr != nil {
return fmt.Errorf("无法登记恢复意图:%w", opErr)
}
// 恢复必须带着"这次作业写了什么"的持久记录:没有它就只能按目标现状推断残渣,
// 而那会误删历史孤儿/其它知识库暂未绑定的资源(快照里没有这些行,删了回不来)。
ledger, ledgerErr := loadWriteSetForJob(ctx, store, config.LibraryArchiveRoot, payload.Scope,
@@ -115,8 +135,8 @@ func runLibraryRecovery(ctx context.Context, store *postgresstore.Store, config
if digest == "" {
return errors.New("恢复完成但没有校验摘要:拒绝登记恢复事实(无法证明恢复到了哪一份内容)")
}
if err := recordRecoveryEvent(ctx, writer, payload.Scope, payload.WindowJobID, result.SnapshotID, digest,
"按预备份恢复并逐类校验通过"); err != nil {
if err := recordRecoveryEvent(ctx, writer, recoveryOp, payload.Scope, payload.WindowJobID, result.SnapshotID,
digest, "按预备份恢复并逐类校验通过"); err != nil {
return fmt.Errorf("恢复已完成但恢复事实登记失败:%w", err)
}
// 收尾顺序:先登记事实,再释放占用。反过来会出现"锁已放、current 还指向失败版本"
@@ -57,7 +57,7 @@ func sealTinyBaseline(t *testing.T, root, scope string, now time.Time) string {
// 真实 PostgreSQL 正负例:历史快照被别的作业发布过且已被更新版本顶替,新恢复
// 作业在写入中断后必须能继续;授权/维护/基线任一不成立就必须继续拒绝。
func TestTakeoverHistoricalRestoreOnRealLedger(t *testing.T) {
store, _ := openLedgerStore(t)
store, pool := openLedgerStore(t)
ctx := context.Background()
now := time.Now().UTC()
const scope = "internal"
@@ -111,6 +111,13 @@ func TestTakeoverHistoricalRestoreOnRealLedger(t *testing.T) {
t.Fatalf("record prebackup %s: %v", snapshot, err)
}
}
// 同一份预备份可以被多个意图登记:每次登记是一条独立事件(F02:不许互相改写)。
recordBackupWithIntent := func(source, intentKey, snapshot string, at time.Time) {
t.Helper()
opID := testLibraryOperation(t, store, scope, governance.LibraryJobRestore, intentKey, at)
recordVersionEvent(t, store, scope, governance.LibraryVersionProductionPrebackup, snapshot,
source, opID, at, map[string]any{"backup_snapshot": snapshot})
}
recordEffective(otherJob, target, now.Add(-2*time.Hour))
recordEffective(newerJob, "snap-newer", now.Add(-time.Hour))
recordBackup(jobID, backup)
@@ -178,13 +185,31 @@ func TestTakeoverHistoricalRestoreOnRealLedger(t *testing.T) {
t.Fatalf("re-acquire instance maintenance: %v", err)
}
// 预备份登记被别的作业改写 ⇒ 基线归属不符,拒绝。
recordBackup("libsync-internal-thief", backup)
// 别的作业登记同一份预备份:它只能新增**自己**的事件,改不动本作业的那一条
// (F02 在存储层的形状:来源作业/意图在事件上不可变,upsert 改写这条路已经封死)。
// 因此基线归属仍然成立、接管仍可继续——"偷"在台账上不成立,正是要的行为。
recordBackupWithIntent("libsync-internal-thief", "thief-intent", backup, now)
facts = loadLibraryVersionFacts(ctx, store, scope, target)
if got := classifyTakeover(job, facts, loadTakeoverResumeFacts(ctx, store, archiveRoot, job, now)); got != takeoverSuperseded {
t.Fatalf("预备份归属被改写后的接管 = %q, want %q", got, takeoverSuperseded)
if got := classifyTakeover(job, facts, loadTakeoverResumeFacts(ctx, store, archiveRoot, job, now)); got != takeoverFresh {
t.Fatalf("别的作业另登记同一份预备份不该影响本作业的基线:%q, want %q", got, takeoverFresh)
}
var owners []string
rows, err := pool.Query(ctx, `SELECT source_job_id FROM library_versions
WHERE scope=$1 AND kind='production_prebackup' AND snapshot_id=$2 ORDER BY created_at`, scope, backup)
if err != nil {
t.Fatalf("read prebackup events: %v", err)
}
for rows.Next() {
var owner string
if err := rows.Scan(&owner); err != nil {
t.Fatalf("scan prebackup owner: %v", err)
}
owners = append(owners, owner)
}
rows.Close()
if len(owners) != 2 || owners[0] != jobID || owners[1] != "libsync-internal-thief" {
t.Fatalf("两份登记必须都留在台账上且各自归属不变:%v", owners)
}
recordBackup(jobID, backup)
// 基线清单字节被改 ⇒ 摘要对不上,拒绝。
manifestPath := filepath.Join(archiveRoot, "versions", backup+".json")
+27 -22
View File
@@ -17,7 +17,7 @@ import (
// upsert 并会改写 source_job_id,所以"生产上有这个快照、但它不是本作业发的"是存储层
// 才能造出的真实形状。内存夹具只能造出调用方相信的那个版本。
func TestTakeoverRefusesSameSnapshotRegisteredByAnotherJob(t *testing.T) {
store, _ := openLedgerStore(t)
store, pool := openLedgerStore(t)
ctx := context.Background()
now := time.Now().UTC()
const scope = "internal"
@@ -27,15 +27,10 @@ func TestTakeoverRefusesSameSnapshotRegisteredByAnotherJob(t *testing.T) {
record := func(source string, created time.Time) {
t.Helper()
if err := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionID(governance.LibraryVersionProductionEffective, scope, shared),
Scope: scope, Kind: governance.LibraryVersionProductionEffective, SnapshotID: shared,
SourceJobID: source, CreatedBy: "worker",
Evidence: map[string]any{"action": "sync", "index_mapping_sha256": "mapping-live"},
CreatedAt: created,
}); err != nil {
t.Fatalf("record effective for %s: %v", source, err)
}
opID := testLibraryOperation(t, store, scope, governance.LibraryJobSync, "shared:"+source, created)
recordVersionEvent(t, store, scope, governance.LibraryVersionProductionEffective, shared,
source, opID, created,
map[string]any{"action": "sync", "index_mapping_sha256": "mapping-live"})
}
record(otherJob, now.Add(-time.Hour))
@@ -56,14 +51,25 @@ func TestTakeoverRefusesSameSnapshotRegisteredByAnotherJob(t *testing.T) {
got, takeoverForeignEffective)
}
// 本作业自己登记过同一快照(upsert 把归属改写回来)⇒ 只差收尾。
// 另一条**独立事实**:本作业用自己的意图登记同一快照(归属不会被改写,
// 而是新增一条属于本作业的事件,current 指向最新的那条)。
record(thisJob, now)
facts = loadLibraryVersionFacts(ctx, store, scope, shared)
if facts.Err != nil {
t.Fatalf("read facts after own record: %v", facts.Err)
}
if facts.Current.SourceJobID != thisJob {
t.Fatalf("upsert 没有把归属改成本作业:%+v", facts.Current)
t.Fatalf("本作业的新意图没有成为当前生效事实:%+v", facts.Current)
}
// 旧事件必须原样保留:新意图不许改写别人的登记(F02 的反面证据)。
var otherOwner string
if err := pool.QueryRow(ctx,
`SELECT source_job_id FROM library_versions WHERE scope=$1 AND kind='production_effective'
AND snapshot_id=$2 ORDER BY created_at ASC LIMIT 1`, scope, shared).Scan(&otherOwner); err != nil {
t.Fatalf("read original event: %v", err)
}
if otherOwner != otherJob {
t.Fatalf("别人的历史事件被改写成 %q(必须保持 %q)", otherOwner, otherJob)
}
if got := classifyTakeover(job, facts, takeoverResumeFacts{}); got != takeoverPendingFinal {
t.Fatalf("本作业登记的同一快照分类 = %q, want %q", got, takeoverPendingFinal)
@@ -147,17 +153,15 @@ func TestTakeoverClassificationUsesExactCurrentVersionBeyondHistoryWindow(t *tes
job := governance.LibraryJob{ID: jobID, Scope: scope, Kind: governance.LibraryJobSync,
TargetSnapshotID: targetSnapshot, SnapshotID: "snap-prebackup"}
// 每次登记都是一个**独立意图**(不同 snapshot/job 就是不同 operation):
// current 的次序键取自意图创建时间,测试要证明的正是"按意图时间单调"。
recordEffective := func(sourceJob, snapshot string, created time.Time) {
t.Helper()
if err := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionID(governance.LibraryVersionProductionEffective, scope, snapshot),
Scope: scope, Kind: governance.LibraryVersionProductionEffective, SnapshotID: snapshot,
SourceJobID: sourceJob, CreatedBy: "worker",
Evidence: map[string]any{"action": "sync", "index_mapping_sha256": "mapping-live"},
CreatedAt: created,
}); err != nil {
t.Fatalf("record effective %s: %v", snapshot, err)
}
opID := testLibraryOperation(t, store, scope, governance.LibraryJobSync,
string(governance.LibraryJobSync)+":"+snapshot+":"+sourceJob, created)
recordVersionEvent(t, store, scope, governance.LibraryVersionProductionEffective, snapshot,
sourceJob, opID, created,
map[string]any{"action": "sync", "index_mapping_sha256": "mapping-live"})
}
// 生效版本登记在 48 小时前,然后 250 条更新的历史记录把它挤出任何窗口。
@@ -218,7 +222,8 @@ func TestTakeoverClassificationUsesExactCurrentVersionBeyondHistoryWindow(t *tes
if state := classifyBaselineVersions(current, hasCurrent, "libbase-internal-x", "snap-newer"); state.Own || state.Foreign == nil {
t.Fatalf("别人的生效版本必须挡住基线:%+v", state)
}
recordEffective("libbase-internal-y", "snap-baseline", now)
// 新的意图必须真的更新:晚一分钟,而不是与上一条同刻(同刻只能靠 id 字典序,不是语义)。
recordEffective("libbase-internal-y", "snap-baseline", now.Add(time.Minute))
current, hasCurrent, err = store.LibraryCurrentVersion(ctx, scope)
if err != nil || !hasCurrent {
t.Fatalf("再读当前生效版本失败:found=%v err=%v", hasCurrent, err)
+17 -6
View File
@@ -122,11 +122,22 @@ var (
_ libraryStageStore = (*libraryAttemptWriter)(nil)
)
// libraryQueueOwner 用队列作业 ID 作为执行者身份;缺失时退回业务作业 ID
// (手工/测试路径没有队列身份,但至少不会与他人共用)。
func libraryQueueOwner(job jobs.Job, libraryJobID string) string {
if job.ID != "" {
return job.ID
// libraryQueueOwner 构造**每一次投递**的执行者身份:worker + 队列作业 + 队列尝试序号。
//
// 不能用稳定的 queueJobID 单独当身份(父的复核项 R2):同一个队列作业被重投时
// ID 不变,而执行者是新的——那样第二次投递会被误判成"同一次执行重复调用",
// 直接复用上一代租约而不开新代次。加上 worker 与 attempt 之后:
//
// · 同一次投递重复调用 ⇒ 身份相同 ⇒ 幂等复用;
// · 换 worker 或换队列尝试 ⇒ 身份不同 ⇒ 必须开新代次(旧代次被结束,守卫立即失效)。
func libraryQueueOwner(workerID string, job jobs.Job, libraryJobID string) string {
queueID := job.ID
if queueID == "" {
queueID = libraryJobID
}
return libraryJobID
worker := workerID
if worker == "" {
worker = "worker"
}
return fmt.Sprintf("%s#%s#%d", worker, queueID, job.Attempt)
}
+27 -12
View File
@@ -176,15 +176,23 @@ func run(ctx context.Context, getenv func(string) string) error {
}
// 生产同步/恢复的执行器不依赖主机侧封存 agent:没有 agent socket 也要能跑。
// 这里在全部注册之后无条件覆盖一次,确保 library.sync/library.restore 一定在队列类型里。
handlers[librarySyncJobType] = func(ctx context.Context, job jobs.Job) error {
//
// 执行域准入(R4)在这里统一判定:目标实况、旧 release 路由绑定、未释放维护窗口
// 任何一项未知或重叠 ⇒ 这些 handler 直接拒绝执行(明确失败,无任何维护动作,
// 也不停止任何既有服务)。
admissionErr := verifyLibraryExecutionDomain(context.Background(), config, store)
if admissionErr != nil {
log.Printf("library execution domain NOT admitted: %v", admissionErr)
}
handlers[librarySyncJobType] = admittanceGuarded(admissionErr, func(ctx context.Context, job jobs.Job) error {
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobSync)
}
handlers[libraryRestoreJobType] = func(ctx context.Context, job jobs.Job) error {
})
handlers[libraryRestoreJobType] = admittanceGuarded(admissionErr, func(ctx context.Context, job jobs.Job) error {
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobRestore)
}
handlers[libraryRecoverJobType] = func(ctx context.Context, job jobs.Job) error {
})
handlers[libraryRecoverJobType] = admittanceGuarded(admissionErr, func(ctx context.Context, job jobs.Job) error {
return runLibraryRecovery(ctx, store, config, job)
}
})
worker, err := jobs.NewWorker(jobs.WorkerConfig{
Queue: store,
@@ -333,18 +341,25 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
payload.Scope, payload.CaptureJob, result.SnapshotID, result.Replay, payload.RequestedBy)
return nil
}
// 执行域准入(R4):注册整库发布执行器**之前**核对目标实况、旧 release 路由与
// 未释放的维护窗口。未通过时下面这些 handler 一律直接拒绝执行——明确失败,
// 不带任何维护动作,也不停止任何既有服务。
admissionErr := verifyLibraryExecutionDomain(context.Background(), config, store)
if admissionErr != nil {
log.Printf("library execution domain NOT admitted: %v", admissionErr)
}
// 生产同步:唯一入口在 API(闸口未齐不建作业)。这里仍实现执行器,
// 但默认 fail-closed:没有闸口证据就明确失败,绝不向生产写入半个字节。
handlers[librarySyncJobType] = func(ctx context.Context, job jobs.Job) error {
handlers[librarySyncJobType] = admittanceGuarded(admissionErr, func(ctx context.Context, job jobs.Job) error {
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobSync)
}
handlers[libraryRestoreJobType] = func(ctx context.Context, job jobs.Job) error {
})
handlers[libraryRestoreJobType] = admittanceGuarded(admissionErr, func(ctx context.Context, job jobs.Job) error {
return runLibraryRelease(ctx, store, config, job, governance.LibraryJobRestore)
}
})
// 建立目标权威基线:目标非空又没有生效版本时的唯一入口(只读捕获 + 校验 + 登记)。
handlers[libraryBaselineJobType] = func(ctx context.Context, job jobs.Job) error {
handlers[libraryBaselineJobType] = admittanceGuarded(admissionErr, func(ctx context.Context, job jobs.Job) error {
return runLibraryBaseline(ctx, store, config, job)
}
})
// A crash can leave the source writer stopped: recover before taking work,
// and keep the failure visible instead of pretending the worker is ready.
go func() {
+5
View File
@@ -35,6 +35,11 @@ const (
LibraryJobSync LibraryJobKind = "sync"
// LibraryJobRestore 是按指定整库版本恢复:与同步分开,授权与阶段也独立。
LibraryJobRestore LibraryJobKind = "restore"
// LibraryJobRecover 是"按预备份恢复这个失败窗口"这一次**新意图**。
// 它必须是一条独立意图:恢复要把 current 指回更早发布的快照,而 current 的次序按
// 意图的创建时间排序(见 CurrentOrderKey),拿旧失败作业的身份去写就会被判成"迟到的
// 旧意图"而拒绝——那正是 F06 不能解决的形状。
LibraryJobRecover LibraryJobKind = "recover"
// LibraryJobBaseline 建立首次发布目标的权威基线:在维护隔离下把目标整库
// 读出来封存成快照并逐类校验,再登记成 production_effective。
// 它不写目标内容、不清空目标、不猜版本——基线只能来自真实捕获与校验。
+1 -1
View File
@@ -147,7 +147,7 @@ func ValidateLibraryOperation(op LibraryOperation) error {
return errors.New("library operation requires internal or external scope")
}
switch op.Kind {
case LibraryJobSeal, LibraryJobSync, LibraryJobRestore, LibraryJobBaseline:
case LibraryJobSeal, LibraryJobSync, LibraryJobRestore, LibraryJobBaseline, LibraryJobRecover:
default:
return fmt.Errorf("unsupported library operation kind %q", op.Kind)
}
+3
View File
@@ -354,6 +354,9 @@ type libraryJobStore interface {
AdvanceLibraryJob(ctx context.Context, jobID string, state governance.LibraryJobState, stage, message, recovery, snapshotID, targetSnapshotID string, now time.Time) error
AttachLibraryQueueJob(ctx context.Context, jobID, queueJobID string, now time.Time) error
CancelLibraryJob(ctx context.Context, jobID, reason string, now time.Time) error
// 续作走**原业务作业**(维护窗口/journal/预备份都按它落盘),只新增执行代次。
PrepareLibraryJobResume(ctx context.Context, jobID string, now time.Time) (governance.LibraryJob, bool, error)
ResetQueueJobForResume(ctx context.Context, queueJobID string, now time.Time) error
}
func libraryStore(config Config) (libraryJobStore, bool) {
+20
View File
@@ -157,6 +157,26 @@ func (f *fakeLibraryStore) CancelLibraryJob(_ context.Context, _, _ string, _ ti
return nil
}
// 续作走原作业:只有 failed/cancelled 才能回到 pending。
func (f *fakeLibraryStore) PrepareLibraryJobResume(_ context.Context, jobID string,
_ time.Time) (governance.LibraryJob, bool, error) {
for index, job := range f.jobs {
if job.ID != jobID {
continue
}
if job.State != governance.LibraryJobFailed && job.State != governance.LibraryJobCancelled {
return job, false, nil
}
f.jobs[index].State = governance.LibraryJobPending
return f.jobs[index], true, nil
}
return governance.LibraryJob{}, false, nil
}
func (f *fakeLibraryStore) ResetQueueJobForResume(_ context.Context, _ string, _ time.Time) error {
return nil
}
func TestLibrarySyncGateRequiresEveryPieceOfEvidence(t *testing.T) {
known := libraryKnownView{State: "known", Version: &libraryVersionView{ID: "v1"}}
cases := []struct {
+29 -48
View File
@@ -275,77 +275,58 @@ func libraryJobResume(w http.ResponseWriter, r *http.Request, config Config) {
}
now := time.Now().UTC()
// 幂等键按源作业固定:重复点击返回同一个续作作业,而不是每次新建。
jobID := sourceID + "-resume"
var kind governance.LibraryJobKind
queueType := ""
switch source.Kind {
case governance.LibraryJobSync:
kind, queueType = governance.LibraryJobSync, librarySyncJobType
queueType = librarySyncJobType
case governance.LibraryJobRestore:
kind, queueType = governance.LibraryJobRestore, libraryRestoreJobType
queueType = libraryRestoreJobType
default:
writeJSON(w, http.StatusConflict, map[string]any{"error": "封存作业由封存入口重发,不走续作"})
return
}
// F15:续作是**同一个意图的新执行代次**,不是新意图。沿用原作业的 operation,
// 维护窗口、预备份、写集合的谱系都还挂在同一条意图上;新作业行只是这次执行的载体,
// worker 起手会为它开一个更大的 epoch(旧代次从此不能写)。
operationID := source.OperationID
if operationID == "" {
// 迁移前的老作业没有 operation:按原作业身份补一条(同键同摘要 ⇒ 幂等,
// 不会因为点了两次续作而造出两条意图)。
intent, intentErr := ensureLibraryIntent(r.Context(), store, source.Scope, kind,
source.TargetSnapshotID, digest, identity.Username, "resume:"+sourceID, now)
if intentErr != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot register resume intent"})
return
}
operationID = intent.Operation.ID
}
job, reused, err := store.CreateLibraryJob(r.Context(), governance.LibraryJob{
ID: jobID, Scope: source.Scope, Kind: kind,
IdempotencyKey: "library.job:" + operationID + ":resume",
RequestDigest: digest, OperationID: operationID,
TargetSnapshotID: source.TargetSnapshotID, Actor: identity.Username,
}, governance.SyncStages(), now)
if errors.Is(err, governance.ErrLibraryJobActive) {
writeJSON(w, http.StatusConflict, map[string]any{"error": "该目标已有在飞作业"})
return
}
if errors.Is(err, governance.ErrLibraryJobIdempotencyConflict) {
writeJSON(w, http.StatusConflict, map[string]any{"error": "同一续作请求参数不一致"})
return
}
// R3:续作沿用**原业务作业**,只新增执行代次。
//
// 维护窗口 owner_job_id、freeze journal 文件名、预备份与写集合的身份全部按业务作业
// id 落盘。另建 `-resume` 作业只能继承 operation,其它所有者仍指向旧作业:
// AcquireInstanceMaintenance 会拒绝它、journal 与预备份也查不到——那样"续作"只是
// 一个漂亮的新行,接不上原窗口。原作业 + 新 attempt 才是批准协议的形状。
prepared, resumed, err := store.PrepareLibraryJobResume(r.Context(), sourceID, now)
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot create resumed job"})
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot prepare resumed job"})
return
}
if reused {
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "resumed_from": sourceID, "reused": true})
if !resumed {
// 状态在检查之后被别的执行者推进(已接管/已完成):不许再塞一次。
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_resume_state_changed",
"error": "作业已不在可续作状态(可能已被接管或完成)"})
return
}
// 写入已完成、只差登记版本的情况:续作只做幂等收尾,不重写生产。
body := map[string]any{"library_job_id": job.ID, "scope": source.Scope,
body := map[string]any{"library_job_id": prepared.ID, "scope": source.Scope,
"target_snapshot_id": source.TargetSnapshotID}
if source.Stage == "finalize" && source.SnapshotID != "" {
body["finalize_only"] = true
}
payload, _ := json.Marshal(body)
queueJob, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{
ID: governance.QueueJobIDFor(job.ID), Type: queueType, Payload: payload, AvailableAt: now,
MaxAttempts: 1, IdempotencyKey: "library.resume.queue:" + job.ID,
}, now)
if err != nil || queueJob == nil {
_ = store.CancelLibraryJob(r.Context(), job.ID, "队列投递失败", now)
queueID := governance.QueueJobIDFor(prepared.ID)
if _, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{
ID: queueID, Type: queueType, Payload: payload, AvailableAt: now,
MaxAttempts: 1, IdempotencyKey: "library.resume.queue:" + prepared.ID,
}, now); err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot enqueue resumed job"})
return
}
// 附不上队列号就等于作业没人跑:必须显式失败并取消,不能静默返回 202。
if attachErr := store.AttachLibraryQueueJob(r.Context(), job.ID, queueJob.ID, now); attachErr != nil {
_ = store.CancelLibraryJob(r.Context(), job.ID, "队列归属写入失败", now)
// 队列行可能还是上一次的终态:重置为 pending。仍有效的租约 ⇒ 有执行在跑,让路。
if resetErr := store.ResetQueueJobForResume(r.Context(), queueID, now); resetErr != nil {
writeJSON(w, http.StatusConflict, map[string]any{"code": "library_resume_busy",
"error": "该作业已有执行在跑(租约仍有效),等它结束再续作", "detail": errString(resetErr)})
return
}
if attachErr := store.AttachLibraryQueueJob(r.Context(), prepared.ID, queueID, now); attachErr != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot attach resumed job"})
return
}
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "resumed_from": sourceID, "reused": false})
writeJSON(w, http.StatusAccepted, map[string]any{"job": prepared, "resumed_from": sourceID,
"resumed": true, "next_attempt_epoch": prepared.AttemptEpoch + 1})
}
+91
View File
@@ -0,0 +1,91 @@
// Package libraryadmission 是**执行域准入**:在 worker 注册整库发布 handler 之前,
// 用目标实况与治理库事实回答"这个进程能不能安全地写这个目标"。
//
// 为什么必须有这一层(R4):维护窗口的互斥只覆盖本治理平台的执行者。旧 release 路由
// (独立 release-worker)直接写同一个物理目标(DB/S3/ES),它不经过 library 维护锁。
// 两个执行域并存时,"未知重叠"必须**拒绝**,不能靠某个人去看部署文档。
//
// 这里只做判定(纯函数),探针由调用方注入:判定可以单测,探针只能读、不能改,
// 也绝不停止任何既有服务(停旧 worker 是运维决定,不是这个包的副作用)。
package libraryadmission
import (
"errors"
"fmt"
)
// Target 是一个执行域的目标身份。Configured 来自部署配置,Live 来自目标实况。
type Target struct {
DatabaseEndpoint string
Index string
Bucket string
KnowledgeBaseID string
TenantID uint64
SchemaVersion int
Instance string
}
// Facts 是判定所需的**可核验事实**。任何一个 Err 非空都表示"未知",必须拒绝。
type Facts struct {
// Live 是目标按配置连上去之后**读回来**的身份(不是配置回声)。
Live Target
LiveErr error
// LiveDirty 表示目标 schema 处于 dirty(迁移未完成)。
LiveDirty bool
// LegacyReleaseKB 是旧 release 路由(production_kb_bindings)写入的目标知识库。
// 空串表示旧路由未绑定;读失败必须通过 LegacyErr 表达,不能当"没有"。
LegacyReleaseKB string
LegacyErr error
// HeldInstance/HeldJob 是治理库里**未释放**的维护窗口持有者(空=没有窗口)。
HeldInstance string
HeldJob string
HeldErr error
}
// Verify 判定"当前进程可以独占这个目标"。
//
// 允许的前提全部是**读出来的**:目标实况与配置一致、schema 已知且未 dirty、
// 旧 release 路由不指向同一个知识库、没有别的实例持有未释放的维护窗口。
// 任何一条未知或不一致 ⇒ 返回错误(调用方必须据此不注册 handler,即 fail closed)。
func Verify(configured Target, facts Facts) error {
if facts.LiveErr != nil {
return fmt.Errorf("读不到目标实况,无法证明执行域唯一:%w", facts.LiveErr)
}
if facts.LegacyErr != nil {
return fmt.Errorf("读不到旧 release 路由的目标绑定,无法判断是否重叠:%w", facts.LegacyErr)
}
if facts.HeldErr != nil {
return fmt.Errorf("读不到未释放的维护窗口,无法判断是否重叠:%w", facts.HeldErr)
}
if facts.LiveDirty {
return errors.New("目标 schema 处于 dirty 状态(迁移未完成):拒绝注册整库发布执行器")
}
if configured.KnowledgeBaseID == "" || configured.TenantID == 0 {
return errors.New("整库目标未配置知识库/租户:拒绝注册整库发布执行器")
}
if facts.Live.KnowledgeBaseID != configured.KnowledgeBaseID || facts.Live.TenantID != configured.TenantID {
return fmt.Errorf("目标实况是 %d/%s,配置指向 %d/%s:拒绝在没核对清楚的目标上执行",
facts.Live.TenantID, facts.Live.KnowledgeBaseID, configured.TenantID, configured.KnowledgeBaseID)
}
if facts.Live.SchemaVersion == 0 {
return errors.New("读不到目标 schema 版本:无法证明与整库发布兼容")
}
if configured.SchemaVersion != 0 && facts.Live.SchemaVersion != configured.SchemaVersion {
return fmt.Errorf("目标 schema 版本 %d 与声明 %d 不一致:拒绝执行",
facts.Live.SchemaVersion, configured.SchemaVersion)
}
if configured.Index != "" && facts.Live.Index != "" && configured.Index != facts.Live.Index {
return fmt.Errorf("目标索引 %q 与配置 %q 不一致:拒绝执行", facts.Live.Index, configured.Index)
}
// 旧 release 路由与整库目标指向同一个知识库 ⇒ 两个执行域会互相绕过维护锁。
if facts.LegacyReleaseKB != "" && facts.LegacyReleaseKB == configured.KnowledgeBaseID {
return fmt.Errorf("旧 release 路由已把发布目标绑定到同一个知识库 %s:两个执行域会绕过维护互斥,拒绝注册整库执行器(先隔离或退役旧路由)",
facts.LegacyReleaseKB)
}
// 别的实例正持有未释放的维护窗口 ⇒ 目标此刻归它。
if facts.HeldInstance != "" && configured.Instance != "" && facts.HeldInstance != configured.Instance {
return fmt.Errorf("目标实例 %s 正被 %s(作业 %s)持有未释放的维护窗口:拒绝注册重复执行器",
configured.Instance, facts.HeldInstance, facts.HeldJob)
}
return nil
}
@@ -0,0 +1,93 @@
package libraryadmission
import (
"errors"
"strings"
"testing"
)
func base() (Target, Facts) {
configured := Target{DatabaseEndpoint: "pg:5432/db", Index: "idx", Bucket: "b",
KnowledgeBaseID: "kb-int", TenantID: 10001, SchemaVersion: 71, Instance: "kgseal"}
facts := Facts{Live: Target{DatabaseEndpoint: "pg:5432/db", Index: "idx", Bucket: "b",
KnowledgeBaseID: "kb-int", TenantID: 10001, SchemaVersion: 71, Instance: "kgseal"}}
return configured, facts
}
func TestVerifyAllowsCleanUniqueTarget(t *testing.T) {
configured, facts := base()
if err := Verify(configured, facts); err != nil {
t.Fatalf("干净且唯一的目标应放行:%v", err)
}
}
// 未知一律拒绝:读不到实况/旧路由/维护窗口。
func TestVerifyFailsClosedOnUnknown(t *testing.T) {
configured, facts := base()
cases := map[string]Facts{
"读不到目标实况": {LiveErr: errors.New("connect refused")},
"读不到旧release": {LegacyErr: errors.New("query failed")},
"读不到维护窗口": {HeldErr: errors.New("query failed")},
}
for name, broken := range cases {
if err := Verify(configured, broken); err == nil {
t.Fatalf("%s 时必须拒绝", name)
}
}
if err := Verify(configured, Facts{Live: Target{KnowledgeBaseID: "kb-int", TenantID: 10001}}); err == nil ||
!strings.Contains(err.Error(), "schema") {
t.Fatalf("schema 读不到时必须拒绝:%v", err)
}
_ = facts
}
// 目标实况与配置不一致 ⇒ 拒绝(不能只信配置)。
func TestVerifyRejectsMismatchedLiveTarget(t *testing.T) {
configured, facts := base()
facts.Live.KnowledgeBaseID = "another-kb"
if err := Verify(configured, facts); err == nil {
t.Fatal("目标实况与配置不一致必须拒绝")
}
configured, facts = base()
facts.LiveDirty = true
if err := Verify(configured, facts); err == nil {
t.Fatal("dirty schema 必须拒绝")
}
configured, facts = base()
facts.Live.SchemaVersion = 72
if err := Verify(configured, facts); err == nil {
t.Fatal("schema 版本不一致必须拒绝")
}
}
// 旧 release 路由指向同一个知识库 ⇒ 两个执行域重叠,拒绝。
func TestVerifyRejectsLegacyReleaseOverlap(t *testing.T) {
configured, facts := base()
facts.LegacyReleaseKB = "kb-int"
err := Verify(configured, facts)
if err == nil {
t.Fatal("旧 release 路由与整库目标同一个 KB 时必须拒绝(R4)")
}
if !strings.Contains(err.Error(), "旧 release") {
t.Fatalf("拒绝原因必须指向旧 release 路由:%v", err)
}
// 指向别的知识库则不重叠。
facts.LegacyReleaseKB = "kb-other"
if err := Verify(configured, facts); err != nil {
t.Fatalf("不同知识库不算重叠:%v", err)
}
}
// 别的实例持有未释放窗口 ⇒ 拒绝;自己持有则允许(重启后的自我接管)。
func TestVerifyRejectsForeignHeldWindow(t *testing.T) {
configured, facts := base()
facts.HeldInstance, facts.HeldJob = "other-instance", "libsync-x"
if err := Verify(configured, facts); err == nil {
t.Fatal("别的实例持有未释放窗口时必须拒绝")
}
configured, facts = base()
facts.HeldInstance, facts.HeldJob = "kgseal", "libsync-x"
if err := Verify(configured, facts); err != nil {
t.Fatalf("自己持有窗口时允许注册(重启接管):%v", err)
}
}
@@ -0,0 +1,525 @@
package postgres
import (
"context"
"errors"
"os"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"knowledge-governance/internal/governance"
"knowledge-governance/internal/librarymaintenance"
)
// 真实 PG 上的执行身份/栅栏用例。
//
// 为什么必须是真库(父的复核项 R1/R2):守卫的失败形态是"SQL 条件永远不匹配"——
// 参数序号、列引用、NULL 与空串的差别在内存 fake 里根本不存在。字符串断言证明不了
// 一条 UPDATE 真的命中了一行。
func newIdentityStore(t *testing.T) (*Store, *pgxpool.Pool) {
t.Helper()
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
if databaseURL == "" {
t.Skip("TEST_DATABASE_URL is not set")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, databaseURL)
if err != nil {
t.Fatalf("connect isolated database: %v", err)
}
t.Cleanup(pool.Close)
if err := ApplyMigrations(ctx, pool); err != nil {
t.Fatalf("apply migrations: %v", err)
}
return NewStore(pool), pool
}
// seedOperationJob 建一条"有 operation 的业务作业",返回 operation 与作业 id。
func seedOperationJob(t *testing.T, store *Store, pool *pgxpool.Pool, scope, jobID, kind string, now time.Time) string {
t.Helper()
ctx := context.Background()
op, _, err := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{
ID: governance.LibraryOperationID(scope, governance.LibraryJobKind(kind), jobID),
Scope: scope, Kind: governance.LibraryJobKind(kind), TargetSnapshotID: "snap-target",
RequestDigest: "digest-" + jobID, OperationKey: jobID, Actor: "tester", CreatedAt: now.UTC(),
})
if err != nil {
t.Fatalf("ensure operation: %v", err)
}
if _, _, err := store.CreateLibraryJob(ctx, governance.LibraryJob{
ID: jobID, Scope: scope, Kind: governance.LibraryJobKind(kind), OperationID: op.ID,
IdempotencyKey: "library.job:" + op.ID, RequestDigest: "digest-" + jobID,
TargetSnapshotID: "snap-target", Actor: "tester",
}, governance.SyncStages(), now); err != nil {
t.Fatalf("create library job: %v", err)
}
// 维护窗口由这个作业持有(实例级 + scope 级)。
if _, err := pool.Exec(ctx, `
INSERT INTO library_maintenance (scope, owner_job_id, target_identity, state, original_service_state,
note, acquired_at, updated_at)
VALUES ($1,$2,'target-identity','acquired','app:running,store:running','test',$3,$3)
ON CONFLICT (scope) DO UPDATE SET owner_job_id=EXCLUDED.owner_job_id, released_at=NULL,
state='acquired', updated_at=EXCLUDED.updated_at`,
scope, jobID, now.UTC()); err != nil {
t.Fatalf("seed scope maintenance: %v", err)
}
if _, err := pool.Exec(ctx, `
INSERT INTO library_instance_maintenance (instance, owner_job_id, owner_scope, target_fingerprint, state,
original_service_state, note, acquired_at, updated_at)
VALUES ('inst-1',$1,$2,'fingerprint-1','acquired','app:running','test',$3,$3)
ON CONFLICT (instance) DO UPDATE SET owner_job_id=EXCLUDED.owner_job_id, owner_scope=EXCLUDED.owner_scope,
released_at=NULL, state='acquired', updated_at=EXCLUDED.updated_at`,
jobID, scope, now.UTC()); err != nil {
t.Fatalf("seed instance maintenance: %v", err)
}
return op.ID
}
// settleJob 把一个作业行推到终态:同一 scope 只允许一条在飞作业(部分唯一索引),
// 所以"再建一条兄弟作业"之前必须先让它落地。
func settleJob(t *testing.T, pool *pgxpool.Pool, jobID string, now time.Time) {
t.Helper()
if _, err := pool.Exec(context.Background(),
`UPDATE library_jobs SET state='failed', finished_at=$2, updated_at=$2 WHERE id=$1`,
jobID, now.UTC()); err != nil {
t.Fatalf("settle job %s: %v", jobID, err)
}
}
// R1:维护类守卫的作业 id 必须取自被更新行自己的 owner_job_id,而不是参数 $1。
//
// 旧实现把 `j.id=$1` 写死,$1 是 scope/instance ⇒ 守卫永远匹配不到作业,
// **正常**的维护状态写入与释放全部失败。这里逐个入口走一遍真库。
func TestLibraryMaintenanceFenceUsesRowOwner(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
jobID := "libsync-fence-1"
seedOperationJob(t, store, pool, "internal", jobID, "sync", now)
attempt, err := store.BeginLibraryAttempt(ctx, jobID, "worker-a#q1#0", 10*time.Minute, now)
if err != nil {
t.Fatalf("begin attempt: %v", err)
}
if attempt.OperationID == "" || attempt.Epoch != 1 {
t.Fatalf("attempt = %+v", attempt)
}
// scope 级:状态、入口、释放都必须命中($1 是 scope,不是作业 id)。
if err := store.UpdateLibraryMaintenanceStateGuarded(ctx, "internal", jobID, "applying", "写目标", attempt, now); err != nil {
t.Fatalf("更新 scope 维护状态失败(守卫没命中作业):%v", err)
}
if err := store.SetLibraryMaintenanceIngressGuarded(ctx, "internal", jobID, "closed", "入口已关", attempt, now); err != nil {
t.Fatalf("记录 scope 入口状态失败:%v", err)
}
// 实例级:$1 是 instance。
if err := store.UpdateInstanceMaintenanceStateGuarded(ctx, "inst-1", jobID, "applying", "写目标", attempt, now); err != nil {
t.Fatalf("更新实例维护状态失败:%v", err)
}
if err := store.SetInstanceMaintenanceIngressGuarded(ctx, "inst-1", jobID, "closed", "入口已关", attempt, now); err != nil {
t.Fatalf("记录实例入口状态失败:%v", err)
}
if err := store.ReleaseLibraryMaintenanceGuarded(ctx, "internal", jobID, attempt, now); err != nil {
t.Fatalf("释放 scope 维护失败(正常路径被守卫挡死):%v", err)
}
if err := store.ReleaseInstanceMaintenanceGuarded(ctx, "inst-1", jobID, attempt, now); err != nil {
t.Fatalf("释放实例维护失败:%v", err)
}
var released int
if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_maintenance WHERE released_at IS NOT NULL`).Scan(&released); err != nil {
t.Fatal(err)
}
if released != 1 {
t.Fatalf("scope 维护应有 1 行已释放,实际 %d", released)
}
// 负例(同一代码路径):作业 id 不匹配时守卫必须**拒绝**,而不是默默返回成功。
// 这条与上面的正例一起,才证明 "$1 是 scope、作业 id 取自行" 的契约真的在起作用。
if err := store.ReleaseInstanceMaintenanceGuarded(ctx, "inst-1", "libsync-other-job", attempt, now); err == nil {
t.Fatal("作业 id 不匹配时释放实例维护必须失败(守卫没看作业身份)")
}
if err := store.UpdateLibraryMaintenanceStateGuarded(ctx, "internal", "libsync-other-job", "applying", "x", attempt, now); err == nil {
t.Fatal("作业 id 不匹配时更新维护状态必须失败")
}
// 幂等:再释放一次是"已释放",不是"未持有"。
if err := store.ReleaseLibraryMaintenanceGuarded(ctx, "internal", jobID, attempt, now); !errors.Is(err, ErrMaintenanceReleased) {
t.Fatalf("重复释放应返回 ErrMaintenanceReleased,实际 %v", err)
}
}
// R2:同一作业连续三次认领 ⇒ 三个不可变代次;旧代次的写必须失败。
func TestBeginLibraryAttemptSameJobThreeGenerations(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
jobID := "libsync-gen-1"
opID := seedOperationJob(t, store, pool, "internal", jobID, "sync", now)
var epochs []int
for round := 1; round <= 3; round++ {
attempt, err := store.BeginLibraryAttempt(ctx, jobID, "worker#q#"+string(rune('0'+round)), 10*time.Minute, now.Add(time.Duration(round)*time.Minute))
if err != nil {
t.Fatalf("第 %d 次认领失败:%v", round, err)
}
epochs = append(epochs, attempt.Epoch)
// 结束上一代(模拟执行完毕/崩溃后接手):直接 Finish 或让它过期。
if err := store.FinishLibraryAttempt(ctx, attempt, governance.LibraryAttemptCompleted, "round done", now); err != nil {
t.Fatalf("第 %d 次结束代次失败:%v", round, err)
}
}
if len(epochs) != 3 || epochs[0] != 1 || epochs[1] != 2 || epochs[2] != 3 {
t.Fatalf("epoch 必须按 operation 单调递增:%v", epochs)
}
// 历史不可变:三次认领留下三行。
var rows int
if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_attempts WHERE operation_id=$1`, opID).Scan(&rows); err != nil {
t.Fatal(err)
}
if rows != 3 {
t.Fatalf("同一作业三次认领必须保留 3 条代次(不可变历史),实际 %d", rows)
}
// 作业行指向最后一代。
job, err := store.LibraryJob(ctx, jobID)
if err != nil {
t.Fatal(err)
}
if job.AttemptEpoch != 3 {
t.Fatalf("作业行 attempt_epoch = %d, want 3", job.AttemptEpoch)
}
// 旧代次的守卫写必须失败(它的 attempt 已 completed)。
stale := governance.LibraryAttempt{OperationID: opID, Epoch: 2, JobID: jobID, Owner: "worker#q#2"}
if err := store.AdvanceLibraryJobGuarded(ctx, stale, governance.LibraryJobRunning, "", "", "", "", "", now); !errors.Is(err, governance.ErrLibraryEpochStale) {
t.Fatalf("旧代次写业务状态必须被拒绝,实际 %v", err)
}
// 迟到的 finish 也必须拒绝:否则旧执行能把别人正在跑的那一代"结束"掉。
if err := store.FinishLibraryAttempt(ctx, stale, governance.LibraryAttemptCompleted, "late finish", now); !errors.Is(err, governance.ErrLibraryEpochStale) {
t.Fatalf("旧代次的 finish 必须被拒绝,实际 %v", err)
}
}
// R2:同一次投递重复调用是幂等的(同一代次,不新增行);别人的有效租约必须让路。
func TestBeginLibraryAttemptIdempotentRepeatAndHeldLease(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
jobID := "libsync-repeat-1"
opID := seedOperationJob(t, store, pool, "internal", jobID, "sync", now)
first, err := store.BeginLibraryAttempt(ctx, jobID, "worker-a#q1#0", 10*time.Minute, now)
if err != nil {
t.Fatal(err)
}
second, err := store.BeginLibraryAttempt(ctx, jobID, "worker-a#q1#0", 10*time.Minute, now.Add(time.Second))
if err != nil {
t.Fatalf("同一次投递重复调用必须幂等:%v", err)
}
if second.Epoch != first.Epoch {
t.Fatalf("重复调用不应新增代次:%d vs %d", second.Epoch, first.Epoch)
}
var rows int
if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_attempts WHERE operation_id=$1`, opID).Scan(&rows); err != nil {
t.Fatal(err)
}
if rows != 1 {
t.Fatalf("幂等重复调用不应新增代次行,实际 %d", rows)
}
// 另一个执行者(不同 owner,租约未过期)必须被拒绝。
if _, err := store.BeginLibraryAttempt(ctx, jobID, "worker-b#q1#0", 10*time.Minute, now.Add(2*time.Second)); !errors.Is(err, ErrLibraryAttemptHeld) {
t.Fatalf("别的执行者持有未过期租约时必须拒绝,实际 %v", err)
}
// 租约过期后允许接管,并结束旧代次。
later := now.Add(20 * time.Minute)
takeover, err := store.BeginLibraryAttempt(ctx, jobID, "worker-b#q1#1", 10*time.Minute, later)
if err != nil {
t.Fatalf("租约过期后必须允许接管:%v", err)
}
if takeover.Epoch != first.Epoch+1 {
t.Fatalf("接管必须开新代次:%d", takeover.Epoch)
}
var oldState string
if err := pool.QueryRow(ctx, `SELECT state FROM library_attempts WHERE operation_id=$1 AND epoch=$2`,
opID, first.Epoch).Scan(&oldState); err != nil {
t.Fatal(err)
}
if oldState != "abandoned" {
t.Fatalf("被接管的旧代次应标为 abandoned,实际 %q", oldState)
}
// 旧代次的续租必须失败(失租即停)。
if err := store.RenewLibraryAttempt(ctx, first, time.Minute, later); !errors.Is(err, governance.ErrLibraryEpochStale) {
t.Fatalf("旧代次续租必须失败,实际 %v", err)
}
}
// R2:同一 operation 不同作业(续作留下的兄弟行)——新代次激活后,旧作业行的守卫必须失效。
func TestSiblingJobGuardInvalidatedByNewAttempt(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
firstJob := "libsync-sib-1"
opID := seedOperationJob(t, store, pool, "internal", firstJob, "sync", now)
firstAttempt, err := store.BeginLibraryAttempt(ctx, firstJob, "worker-a#q1#0", 10*time.Minute, now)
if err != nil {
t.Fatal(err)
}
settleJob(t, pool, firstJob, now)
// 兄弟作业行(同一 operation,模拟历史 `-resume` 行/并行执行者)。
secondJob := firstJob + "-resume"
if _, _, err := store.CreateLibraryJob(ctx, governance.LibraryJob{
ID: secondJob, Scope: "internal", Kind: governance.LibraryJobSync, OperationID: opID,
IdempotencyKey: "library.job:" + opID + ":resume", RequestDigest: "digest-sib",
TargetSnapshotID: "snap-target", Actor: "tester",
}, governance.SyncStages(), now); err != nil {
t.Fatalf("create sibling job: %v", err)
}
// 结束第一代后,由兄弟作业接管同一 operation。
if err := store.FinishLibraryAttempt(ctx, firstAttempt, governance.LibraryAttemptFailed, "crashed", now); err != nil {
t.Fatal(err)
}
settleJob(t, pool, firstJob, now)
secondAttempt, err := store.BeginLibraryAttempt(ctx, secondJob, "worker-b#q2#0", 10*time.Minute, now.Add(time.Minute))
if err != nil {
t.Fatalf("兄弟作业接管失败:%v", err)
}
if secondAttempt.OperationID != opID || secondAttempt.Epoch != firstAttempt.Epoch+1 {
t.Fatalf("兄弟作业必须共用同一 operation 并开新一代:%+v", secondAttempt)
}
// 旧作业行上的 epoch/owner 字段还在,但它的守卫必须失败。
if err := store.AdvanceLibraryJobGuarded(ctx, firstAttempt, governance.LibraryJobRunning, "", "", "", "", "", now); !errors.Is(err, governance.ErrLibraryEpochStale) {
t.Fatalf("旧作业的守卫必须失效,实际 %v", err)
}
var siblingLease string
if err := pool.QueryRow(ctx, `SELECT lease_owner FROM library_jobs WHERE id=$1`, firstJob).Scan(&siblingLease); err != nil {
t.Fatal(err)
}
if siblingLease != "" {
t.Fatalf("兄弟接管后旧作业行租约必须被清空,实际 %q", siblingLease)
}
}
// 首次锚点:事务 + 回读核对,NULL 与空串不清混;覆盖被拒绝。
func TestRecordOperationAnchorsTransactionalAndNullSafe(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
jobID := "libsync-anchor-1"
opID := seedOperationJob(t, store, pool, "internal", jobID, "sync", now)
// 只登记快照 id(字节/写集合未知)⇒ 对应列必须是 NULL,而不是 0/空串。
if err := store.RecordOperationAnchors(ctx, opID, governance.LibraryAnchors{
FirstBackupSnapshotID: "backup-snap-1",
}, now); err != nil {
t.Fatalf("首次登记锚点失败:%v", err)
}
var bytesNull, rowsNull, digestNull bool
if err := pool.QueryRow(ctx, `
SELECT first_backup_bytes IS NULL, writeset_rows IS NULL, writeset_digest IS NULL
FROM library_operations WHERE id=$1`, opID).Scan(&bytesNull, &rowsNull, &digestNull); err != nil {
t.Fatal(err)
}
if !bytesNull || !rowsNull || !digestNull {
t.Fatalf("未知锚点必须留 NULL:bytes_null=%v rows_null=%v digest_null=%v", bytesNull, rowsNull, digestNull)
}
// 补齐字节与写集合:允许填空。
bytes := int64(4096)
rows := int64(119)
if err := store.RecordOperationAnchors(ctx, opID, governance.LibraryAnchors{
FirstBackupBytes: &bytes, WriteSetRows: &rows, WriteSetDigest: "ws-digest-1",
}, now); err != nil {
t.Fatalf("补齐锚点失败:%v", err)
}
var gotBytes int64
var gotDigest string
if err := pool.QueryRow(ctx, `SELECT first_backup_bytes, writeset_digest FROM library_operations WHERE id=$1`,
opID).Scan(&gotBytes, &gotDigest); err != nil {
t.Fatal(err)
}
if gotBytes != bytes || gotDigest != "ws-digest-1" {
t.Fatalf("锚点回读不一致:%d/%s", gotBytes, gotDigest)
}
// 覆盖被拒绝(字节与写集合都不许换)。
other := int64(1)
if err := store.RecordOperationAnchors(ctx, opID, governance.LibraryAnchors{FirstBackupBytes: &other}, now); err == nil {
t.Fatal("用不同字节覆盖首次锚点必须失败")
}
if err := store.RecordOperationAnchors(ctx, opID, governance.LibraryAnchors{WriteSetDigest: "ws-digest-2"}, now); err == nil {
t.Fatal("用不同写集合摘要覆盖首次锚点必须失败")
}
// 幂等:同一份锚点重复登记成功且不改值。
if err := store.RecordOperationAnchors(ctx, opID, governance.LibraryAnchors{
FirstBackupSnapshotID: "backup-snap-1", FirstBackupBytes: &bytes, WriteSetDigest: "ws-digest-1",
}, now); err != nil {
t.Fatalf("同一份锚点重复登记必须幂等成功:%v", err)
}
}
// current 指针:旧意图后来登记的事实**不能**把 current 推回历史;新意图(恢复)可以。
func TestLibraryCurrentOrderingRejectsLateOldIntent(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
base := time.Now().UTC()
jobA, jobB := "libsync-ord-a", "libsync-ord-b"
opA := seedOperationJob(t, store, pool, "internal", jobA, "sync", base)
settleJob(t, pool, jobA, base)
opB := seedOperationJob(t, store, pool, "internal", jobB, "sync", base.Add(time.Minute))
effective := func(opID, jobID, snapshot string, at time.Time) error {
return store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionEventID(governance.LibraryVersionProductionEffective, "internal", snapshot, opID),
Scope: "internal", Kind: governance.LibraryVersionProductionEffective, SnapshotID: snapshot,
SourceJobID: jobID, OperationID: opID, CreatedBy: "tester", CreatedAt: at,
Evidence: map[string]any{"backup_snapshot": "backup-" + snapshot},
})
}
if err := effective(opA, jobA, "snap-a", base); err != nil {
t.Fatalf("登记 A 失败:%v", err)
}
if err := effective(opB, jobB, "snap-b", base.Add(time.Minute)); err != nil {
t.Fatalf("登记 B 失败:%v", err)
}
current, found, err := store.LibraryCurrentVersion(ctx, "internal")
if err != nil || !found || current.SnapshotID != "snap-b" {
t.Fatalf("current 应为 snap-b:%v %+v", err, current)
}
// 旧意图(A,创建时间更早)后来又登记了一条新事实:必须被拒绝,不能把 current 拉回去。
late := governance.LibraryVersion{
ID: governance.LibraryVersionEventID(governance.LibraryVersionProductionEffective, "internal", "snap-a2", opA),
Scope: "internal", Kind: governance.LibraryVersionProductionEffective, SnapshotID: "snap-a2",
SourceJobID: jobA, OperationID: opA, CreatedBy: "tester", CreatedAt: base.Add(2 * time.Hour),
Evidence: map[string]any{"backup_snapshot": "backup-snap-a2"},
}
if err := store.RecordLibraryVersion(ctx, late); !errors.Is(err, governance.ErrLibraryCurrentStale) {
t.Fatalf("旧意图的迟到事实必须被拒绝,实际 %v", err)
}
current, found, err = store.LibraryCurrentVersion(ctx, "internal")
if err != nil || !found || current.SnapshotID != "snap-b" {
t.Fatalf("current 不应被旧意图推回:%v %+v", err, current)
}
// 恢复是**新意图**:内容更旧(backup),意图更新 ⇒ 允许推进。
recoveryOp, _, err := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{
ID: governance.LibraryOperationID("internal", governance.LibraryJobRecover, "window-1"),
Scope: "internal", Kind: governance.LibraryJobRecover, TargetSnapshotID: "backup-snap-a",
RequestDigest: "recover-digest", OperationKey: "window-1", Actor: "tester",
CreatedAt: base.Add(3 * time.Hour),
})
if err != nil {
t.Fatalf("登记恢复意图失败:%v", err)
}
if err := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: governance.LibraryVersionEventID(governance.LibraryVersionRecoveryVerified, "internal", "backup-snap-a", recoveryOp.ID),
Scope: "internal", Kind: governance.LibraryVersionRecoveryVerified, SnapshotID: "backup-snap-a",
SourceJobID: jobA, OperationID: recoveryOp.ID, CreatedBy: "worker", CreatedAt: base.Add(3 * time.Hour),
Evidence: map[string]any{"backup_snapshot": "backup-snap-a", "verify_digest": "digest"},
}); err != nil {
t.Fatalf("恢复事实必须能推进 current:%v", err)
}
current, found, err = store.LibraryCurrentVersion(ctx, "internal")
if err != nil || !found || current.SnapshotID != "backup-snap-a" || current.Kind != governance.LibraryVersionRecoveryVerified {
t.Fatalf("恢复后 current 应指向预备份:%v %+v", err, current)
}
}
// R3:续作沿用原业务作业(维护窗口 owner 不变),只新增执行代次。
func TestPrepareLibraryJobResumeKeepsSameJobAndWindow(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
jobID := "libsync-resume-1"
seedOperationJob(t, store, pool, "internal", jobID, "sync", now)
first, err := store.BeginLibraryAttempt(ctx, jobID, "worker-a#q1#0", 10*time.Minute, now)
if err != nil {
t.Fatal(err)
}
if err := store.AdvanceLibraryJobGuarded(ctx, first, governance.LibraryJobFailed, "database",
"写库失败", "按预备份恢复", "backup-snap-1", "", now); err != nil {
t.Fatalf("标记失败:%v", err)
}
// 维护窗口仍由**原作业**持有(这就是续作必须复用原行的原因)。
held, err := store.LibraryMaintenanceOf(ctx, "internal")
if err != nil || held == nil || held.OwnerJobID != jobID || held.ReleasedAt != nil {
t.Fatalf("维护窗口应仍由原作业持有:%v %+v", err, held)
}
prepared, resumed, err := store.PrepareLibraryJobResume(ctx, jobID, now)
if err != nil || !resumed {
t.Fatalf("续作准备失败:%v resumed=%v", err, resumed)
}
if prepared.ID != jobID || prepared.State != governance.LibraryJobPending {
t.Fatalf("续作必须是同一作业行且回到 pending:%+v", prepared)
}
second, err := store.BeginLibraryAttempt(ctx, jobID, "worker-a#q1#1", 10*time.Minute, now.Add(time.Minute))
if err != nil {
t.Fatalf("续作代次失败:%v", err)
}
if second.Epoch != first.Epoch+1 || second.OperationID != first.OperationID {
t.Fatalf("续作必须是同一 operation 的新代次:%+v vs %+v", second, first)
}
// 新代次可以用原窗口释放(owner 仍是同一个作业)。
if err := store.ReleaseLibraryMaintenanceGuarded(ctx, "internal", jobID, second, now.Add(2*time.Minute)); err != nil {
t.Fatalf("续作代次释放原窗口失败:%v", err)
}
}
// 编译期确认维护层级常量可用(避免测试里手写魔法字符串与实现漂移)。
var _ = librarymaintenance.LevelInstance
// 迁移 76 的 down/up 幂等:治理库里这一层只加身份表与 current 指针,
// 撤下来不能带走任何历史事件(版本行必须留着)。真实 PG 上逐条执行,
// 不靠"迁移文件存在"这种静态证据。
func TestMigration76DownAndUpKeepsVersionHistory(t *testing.T) {
store, pool := newIdentityStore(t)
ctx := context.Background()
now := time.Now().UTC()
seedOperationJob(t, store, pool, "internal", "libsync-mig76", "sync", now)
if _, _, err := store.EnsureLibraryOperation(ctx, governance.LibraryOperation{
ID: governance.LibraryOperationID("internal", governance.LibraryJobSync, "legacy-op"),
Scope: "internal", Kind: governance.LibraryJobSync, TargetSnapshotID: "s",
RequestDigest: "d", OperationKey: "k", Actor: "t", CreatedAt: now,
}); err != nil {
t.Fatal(err)
}
if err := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: "libver-mig76-keep", Scope: "internal", Kind: governance.LibraryVersionProductionPrebackup,
SnapshotID: "snap-mig76", SourceJobID: "libsync-mig76", CreatedBy: "worker",
Evidence: map[string]any{"backup_snapshot": "snap-mig76"}, CreatedAt: now,
}); err != nil {
t.Fatal(err)
}
all, err := loadMigrations()
if err != nil {
t.Fatal(err)
}
var target migration
for _, item := range all {
if item.Version == 76 {
target = item
}
}
if target.Version == 0 {
t.Fatal("migration 76 不存在")
}
if _, err := pool.Exec(ctx, target.DownSQL); err != nil {
t.Fatalf("撤销 migration 76 失败:%v", err)
}
var versions int
if err := pool.QueryRow(ctx, `SELECT count(*) FROM library_versions WHERE id='libver-mig76-keep'`).Scan(&versions); err != nil {
t.Fatalf("撤销后版本历史不可读:%v", err)
}
if versions != 1 {
t.Fatalf("撤销 identity 层不许删版本事件:count=%d", versions)
}
if _, err := pool.Exec(ctx, target.UpSQL); err != nil {
t.Fatalf("重新应用 migration 76 失败:%v", err)
}
if err := store.RecordLibraryVersion(ctx, governance.LibraryVersion{
ID: "libver-mig76-after", Scope: "internal", Kind: governance.LibraryVersionProductionPrebackup,
SnapshotID: "snap-mig76-b", SourceJobID: "libsync-mig76", CreatedBy: "worker",
Evidence: map[string]any{"backup_snapshot": "snap-mig76-b"}, CreatedAt: now.Add(time.Minute),
}); err != nil {
t.Fatalf("重新应用后写入失败:%v", err)
}
}
+101 -11
View File
@@ -288,18 +288,35 @@ func (s *Store) RecordLibraryVersion(ctx context.Context, version governance.Lib
}
}
tag, err := tx.Exec(ctx, `
// 事件身份 = (kind, scope, 快照, operation)。同一身份的重复登记只有两种合法形态:
// · 同一执行者补登记**新增**的字段(预备份先登记、写集合随后补上);
// · 完全没有新字段的纯重放。
// 所以冲突时只做**加法**:已有键永远赢(右操作数是已有 evidence),身份字段
// (source_job_id/operation_id/created_at)一格都不许改;换了执行者或换了意图的
// 登记在这里匹配不到行,什么也不写(F02/F09 不能靠 upsert 改写归属)。
// xmax = 0 才是真的新插入:重放同一份证据(或换了执行者被 WHERE 挡住)都不算新事实,
// 否则一次纯重放就能把 current 再"推进"一次,把后来的操作顶掉。
var inserted bool
err = tx.QueryRow(ctx, `
INSERT INTO library_versions (id, scope, kind, snapshot_id, version_label, source_job_id,
operation_id, attempt_epoch, evidence, created_by, created_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)
ON CONFLICT (id) DO NOTHING`,
ON CONFLICT (id) DO UPDATE
SET evidence = EXCLUDED.evidence || library_versions.evidence
WHERE library_versions.source_job_id = EXCLUDED.source_job_id
AND library_versions.operation_id = EXCLUDED.operation_id
RETURNING (xmax = 0)`,
version.ID, version.Scope, string(version.Kind), version.SnapshotID, version.VersionLabel,
version.SourceJobID, version.OperationID, version.AttemptEpoch, evidence, version.CreatedBy,
version.CreatedAt.UTC())
version.CreatedAt.UTC()).Scan(&inserted)
if errors.Is(err, pgx.ErrNoRows) {
// 同一身份已经存在、但归属是别的执行者/别的意图:不覆盖、不报错,也不推进 current。
inserted = false
err = nil
}
if err != nil {
return fmt.Errorf("record library version event: %w", err)
}
inserted := tag.RowsAffected() == 1
if version.Kind == governance.LibraryVersionProductionEffective || version.Kind == governance.LibraryVersionRecoveryVerified {
if err := advanceLibraryCurrent(ctx, tx, version, inserted); err != nil {
@@ -311,10 +328,31 @@ ON CONFLICT (id) DO NOTHING`,
// advanceLibraryCurrent 用单调次序条件推进 current 指针。
//
// 次序键 = (事件登记时间|operation|epoch)。它只回答"哪一条事实更晚被登记",
// 不回答"哪个意图更重要";旧代次能不能写由 epoch 守卫回答,两件事分开才不会互相冒充。
// 次序键 = (意图创建时间 | operation | epoch),**不是**事件登记时间(父的复核项):
// 拿登记时间排序,一次迟到的旧 operation 重放(例如重跑几天前的同步)就会因为"我刚登记"
// 而把 current 推回历史。意图的创建时间不会因为重放而变新,所以旧意图永远推不动
// 比它更新的意图。
//
// 恢复要不要能把 current 指回更早的**内容**?要,但它是通过**新的意图**(kind=recover,
// 创建于恢复时刻)做到的:内容更旧、意图更新,两者不矛盾。旧代次能不能写由 epoch
// 守卫回答(见 libraryFence),不由时间回答。
func advanceLibraryCurrent(ctx context.Context, tx pgx.Tx, version governance.LibraryVersion, inserted bool) error {
orderKey := governance.CurrentOrderKey(version.CreatedAt, version.OperationID, version.AttemptEpoch)
orderAt := version.CreatedAt
if version.OperationID != "" {
var created *time.Time
err := tx.QueryRow(ctx, `SELECT created_at FROM library_operations WHERE id=$1`,
version.OperationID).Scan(&created)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("事件引用的 operation %s 不存在:拒绝推进 current", version.OperationID)
}
if err != nil {
return fmt.Errorf("load library operation for current ordering: %w", err)
}
if created != nil {
orderAt = *created
}
}
orderKey := governance.CurrentOrderKey(orderAt, version.OperationID, version.AttemptEpoch)
if !inserted {
// 重放:事件已经在台账里。确认指针没有被别的更晚事实覆盖,否则只确认自己的历史。
var currentID string
@@ -404,6 +442,47 @@ ORDER BY lj.created_at`)
return jobs, nil
}
// PrepareLibraryJobResume 把**同一个业务作业**放回 pending,供续作/接管重新执行。
//
// 为什么是原作业而不是新开一行(R3):维护窗口 owner_job_id、freeze journal 文件名、
// 预备份与写集合的身份全部按业务作业 id 落在持久层。新开 `-resume` 作业只能继承
// operation_id,其它所有者都还指向旧作业——AcquireInstanceMaintenance 会拒绝它,
// journal/预备份也查不到,于是"续作"只是一个漂亮的新行。
//
// 条件更新:只有 failed/cancelled 才允许置回 pending;终态时间与错误清零,
// 旧 attempt 行保留(历史不可变)。代次由 BeginLibraryAttempt 在 worker 起手时递增。
func (s *Store) PrepareLibraryJobResume(ctx context.Context, jobID string, now time.Time) (governance.LibraryJob, bool, error) {
tx, err := s.db.Begin(ctx)
if err != nil {
return governance.LibraryJob{}, false, fmt.Errorf("prepare library job resume: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
// 续作是人工发起的接管:作业已经 failed/cancelled,它那一代执行者不可能还在合法地写。
// 必须**显式结束**那一代(行保留、标 abandoned),否则它的租约还没过期,
// BeginLibraryAttempt 会认为"有活跃执行者"而拒绝新代次——续作就永远开不起来。
if _, err := tx.Exec(ctx, `
UPDATE library_attempts SET state='abandoned', finished_at=$2, outcome='superseded by operator resume',
lease_until=NULL
WHERE job_id=$1 AND state IN ('claimed','running')`, jobID, now.UTC()); err != nil {
return governance.LibraryJob{}, false, fmt.Errorf("retire attempts for resume: %w", err)
}
tag, err := tx.Exec(ctx, `
UPDATE library_jobs SET state='pending', error='', recovery='', finished_at=NULL, updated_at=$2,
lease_owner='', lease_until=NULL
WHERE id=$1 AND state IN ('failed','cancelled')`, jobID, now.UTC())
if err != nil {
return governance.LibraryJob{}, false, fmt.Errorf("prepare library job resume: %w", err)
}
if err := tx.Commit(ctx); err != nil {
return governance.LibraryJob{}, false, fmt.Errorf("prepare library job resume: %w", err)
}
job, loadErr := s.LibraryJob(ctx, jobID)
if loadErr != nil {
return governance.LibraryJob{}, false, loadErr
}
return job, tag.RowsAffected() == 1, nil
}
// ResetQueueJobForResume 把中断/失败过的队列作业重新排队:attempt 必须归零,
// 否则 attempt<max_attempts 永远不成立,作业再也不会被任何 worker 领取。
//
@@ -612,10 +691,21 @@ ORDER BY created_at DESC, id DESC LIMIT 2`, scope, string(kind), snapshotID)
if err != nil {
return governance.LibraryVersion{}, false, err
}
// 同一快照被多次意图引用时会有多条事件(每次 operation 一条),这是**正常历史**,
// 不是台账损坏:按登记时间取最新一条回答"这个快照最近一次登记成什么"。
// 谁当前生效不在这里判断,那是 current 指针的事实。
if len(found) >= 1 {
// 同一快照可以被多次意图引用,每次 operation 一条事件,这是**正常历史**;
// 但"两条同身份的行"是台账被写坏(或迁移前没有 operation 身份的未知证据),
// 这时挑选任何一条都是替调用方做决定,必须拒绝(未知证据 fail closed)。
if len(found) == 1 {
return found[0], true, nil
}
if len(found) >= 2 {
if found[0].OperationID == "" || found[1].OperationID == "" {
return governance.LibraryVersion{}, false, fmt.Errorf(
"快照 %s 在 %s 下有 %d 条无执行身份的记录:拒绝挑选", snapshotID, kind, len(found))
}
if found[0].OperationID == found[1].OperationID {
return governance.LibraryVersion{}, false, fmt.Errorf(
"快照 %s 在 %s 下同一意图登记了多行:拒绝挑选", snapshotID, kind)
}
return found[0], true, nil
}
return governance.LibraryVersion{}, false, nil
+245 -59
View File
@@ -2,6 +2,7 @@ package postgres
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
@@ -83,45 +84,92 @@ FROM library_operations WHERE id=$1`, id).Scan(&op.ID, &op.Scope, &kind, &op.Tar
// OperationAnchors 读不可变首次锚点。
func (s *Store) OperationAnchors(ctx context.Context, operationID string) (governance.LibraryAnchors, bool, error) {
var a governance.LibraryAnchors
// 锚点列允许 NULL:必须用可空类型扫描,否则空锚点会让整条读取失败
// (真库验证发现:NULL 扫进 *string 直接报错,而内存 fake 永远不会有这个形状)。
var (
snapshot sql.NullString
backupAt sql.NullTime
digest sql.NullString
fingerprint sql.NullString
recordedAt sql.NullTime
)
var bytes, rows sql.NullInt64
err := s.db.QueryRow(ctx, `
SELECT first_backup_snapshot_id, first_backup_bytes, first_backup_at, writeset_digest, writeset_rows,
target_fingerprint, anchors_at
FROM library_operations WHERE id=$1`, operationID).Scan(&a.FirstBackupSnapshotID, &a.FirstBackupBytes,
&a.FirstBackupAt, &a.WriteSetDigest, &a.WriteSetRows, &a.TargetFingerprint, &a.RecordedAt)
FROM library_operations WHERE id=$1`, operationID).Scan(&snapshot, &bytes, &backupAt, &digest, &rows,
&fingerprint, &recordedAt)
if errors.Is(err, pgx.ErrNoRows) {
return governance.LibraryAnchors{}, false, nil
}
if err != nil {
return governance.LibraryAnchors{}, false, fmt.Errorf("load library operation anchors: %w", err)
}
a := scanAnchors(snapshot, bytes, backupAt, digest, rows, fingerprint, recordedAt)
return a, a.RecordedAt != nil, nil
}
// scanAnchors 是锚点唯一的解码实现:NULL 与零值必须分清(未知 ≠ 0)。
func scanAnchors(snapshot sql.NullString, bytes sql.NullInt64, backupAt sql.NullTime, digest sql.NullString,
rows sql.NullInt64, fingerprint sql.NullString, recordedAt sql.NullTime) governance.LibraryAnchors {
a := governance.LibraryAnchors{FirstBackupSnapshotID: snapshot.String, WriteSetDigest: digest.String,
TargetFingerprint: fingerprint.String}
if bytes.Valid {
value := bytes.Int64
a.FirstBackupBytes = &value
}
if rows.Valid {
value := rows.Int64
a.WriteSetRows = &value
}
if backupAt.Valid {
value := backupAt.Time
a.FirstBackupAt = &value
}
if recordedAt.Valid {
value := recordedAt.Time
a.RecordedAt = &value
}
return a
}
// RecordOperationAnchors 只**填空**,不改已存在的锚点。
// 接管/重放会用同一份锚点重复调用;如果新值想覆盖旧值,那说明"首次"被重拍了,
// 直接报错让调用方停下来(把半写状态当新基线等于毁掉唯一可回滚的那份)。
//
// 读-改-写必须在同一个事务里并锁住 operation 行(R2 附带项):两个执行者同时"首次"
// 登记时,先读后写的实现会各自读到 NULL、各自写入,最后一个静默覆盖前一个——
// 首次锚点就不再是首次。这里 FOR UPDATE 串行化,写入后**逐字段回读核对**:
// 任何一个不可变字段与本次声称的值不一致就报错(而不是让调用方以为登记成功)。
func (s *Store) RecordOperationAnchors(ctx context.Context, operationID string, anchors governance.LibraryAnchors, now time.Time) error {
if operationID == "" {
return errors.New("record operation anchors requires an operation")
}
existing, found, err := s.OperationAnchors(ctx, operationID)
tx, err := s.db.Begin(ctx)
if err != nil {
return fmt.Errorf("begin operation anchors: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
var (
exSnapshot, exDigest, exFingerprint sql.NullString
exBytes, exRows sql.NullInt64
exBackupAt, exRecordedAt sql.NullTime
)
err = tx.QueryRow(ctx, `
SELECT first_backup_snapshot_id, first_backup_bytes, first_backup_at, writeset_digest, writeset_rows,
target_fingerprint, anchors_at
FROM library_operations WHERE id=$1 FOR UPDATE`, operationID).Scan(&exSnapshot, &exBytes, &exBackupAt,
&exDigest, &exRows, &exFingerprint, &exRecordedAt)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("operation %s 不存在:拒绝登记首次锚点", operationID)
}
if err != nil {
return fmt.Errorf("load library operation anchors: %w", err)
}
existing := scanAnchors(exSnapshot, exBytes, exBackupAt, exDigest, exRows, exFingerprint, exRecordedAt)
// 已存在且不同 ⇒ 有人想把"首次"换掉(把半写状态当新基线),必须报错。
if err := compareAnchors(operationID, existing, anchors); err != nil {
return err
}
if found {
if existing.FirstBackupSnapshotID != "" && anchors.FirstBackupSnapshotID != "" &&
existing.FirstBackupSnapshotID != anchors.FirstBackupSnapshotID {
return fmt.Errorf("operation %s 的首次预备份是 %s,不能用 %s 覆盖",
operationID, existing.FirstBackupSnapshotID, anchors.FirstBackupSnapshotID)
}
if existing.WriteSetDigest != "" && anchors.WriteSetDigest != "" &&
existing.WriteSetDigest != anchors.WriteSetDigest {
return fmt.Errorf("operation %s 的首次写集合是 %s,不能用 %s 覆盖",
operationID, existing.WriteSetDigest, anchors.WriteSetDigest)
}
}
_, err = s.db.Exec(ctx, `
if _, err := tx.Exec(ctx, `
UPDATE library_operations SET
first_backup_snapshot_id = COALESCE(first_backup_snapshot_id, $2),
first_backup_bytes = COALESCE(first_backup_bytes, $3),
@@ -132,17 +180,98 @@ UPDATE library_operations SET
anchors_at = COALESCE(anchors_at, $8)
WHERE id=$1`, operationID, nullIfEmpty(anchors.FirstBackupSnapshotID), anchors.FirstBackupBytes,
anchors.FirstBackupAt, nullIfEmpty(anchors.WriteSetDigest), anchors.WriteSetRows,
nullIfEmpty(anchors.TargetFingerprint), now.UTC())
if err != nil {
nullIfEmpty(anchors.TargetFingerprint), now.UTC()); err != nil {
return fmt.Errorf("record library operation anchors: %w", err)
}
// 回读核对:NULL 与空串、nil 与 0 必须分清(真实 PG 才有这个差别)。
var (
afSnapshot, afDigest, afFingerprint sql.NullString
afBytes, afRows sql.NullInt64
afBackupAt, afRecordedAt sql.NullTime
)
if err := tx.QueryRow(ctx, `
SELECT first_backup_snapshot_id, first_backup_bytes, first_backup_at, writeset_digest, writeset_rows,
target_fingerprint, anchors_at
FROM library_operations WHERE id=$1`, operationID).Scan(&afSnapshot, &afBytes, &afBackupAt, &afDigest,
&afRows, &afFingerprint, &afRecordedAt); err != nil {
return fmt.Errorf("read back library operation anchors: %w", err)
}
after := scanAnchors(afSnapshot, afBytes, afBackupAt, afDigest, afRows, afFingerprint, afRecordedAt)
if err := verifyAnchorsPersisted(operationID, anchors, after); err != nil {
return err
}
return tx.Commit(ctx)
}
// compareAnchors 拒绝"用新值覆盖已登记的首次锚点"。
func compareAnchors(operationID string, existing, incoming governance.LibraryAnchors) error {
if existing.FirstBackupSnapshotID != "" && incoming.FirstBackupSnapshotID != "" &&
existing.FirstBackupSnapshotID != incoming.FirstBackupSnapshotID {
return fmt.Errorf("operation %s 的首次预备份是 %s,不能用 %s 覆盖",
operationID, existing.FirstBackupSnapshotID, incoming.FirstBackupSnapshotID)
}
if existing.WriteSetDigest != "" && incoming.WriteSetDigest != "" &&
existing.WriteSetDigest != incoming.WriteSetDigest {
return fmt.Errorf("operation %s 的首次写集合是 %s,不能用 %s 覆盖",
operationID, existing.WriteSetDigest, incoming.WriteSetDigest)
}
if existing.FirstBackupBytes != nil && incoming.FirstBackupBytes != nil &&
*existing.FirstBackupBytes != *incoming.FirstBackupBytes {
return fmt.Errorf("operation %s 的首次预备份字节是 %d,不能用 %d 覆盖",
operationID, *existing.FirstBackupBytes, *incoming.FirstBackupBytes)
}
if existing.TargetFingerprint != "" && incoming.TargetFingerprint != "" &&
existing.TargetFingerprint != incoming.TargetFingerprint {
return fmt.Errorf("operation %s 的目标指纹是 %s,不能用 %s 覆盖",
operationID, existing.TargetFingerprint, incoming.TargetFingerprint)
}
return nil
}
// verifyAnchorsPersisted 核对"本次声称的值确实落库了":空值必须仍是 NULL,
// 有值必须逐字段相同。避免"写成功但被 NULL/0 混淆"的静默丢锚点。
func verifyAnchorsPersisted(operationID string, claimed, persisted governance.LibraryAnchors) error {
check := func(name, want, got string) error {
if want != "" && want != got {
return fmt.Errorf("operation %s 的 %s 回读为 %q,与声称的 %q 不一致", operationID, name, got, want)
}
return nil
}
if err := check("first_backup_snapshot_id", claimed.FirstBackupSnapshotID, persisted.FirstBackupSnapshotID); err != nil {
return err
}
if err := check("writeset_digest", claimed.WriteSetDigest, persisted.WriteSetDigest); err != nil {
return err
}
if err := check("target_fingerprint", claimed.TargetFingerprint, persisted.TargetFingerprint); err != nil {
return err
}
if claimed.FirstBackupBytes != nil &&
(persisted.FirstBackupBytes == nil || *persisted.FirstBackupBytes != *claimed.FirstBackupBytes) {
return fmt.Errorf("operation %s 的 first_backup_bytes 没有按声称值落库", operationID)
}
if claimed.WriteSetRows != nil &&
(persisted.WriteSetRows == nil || *persisted.WriteSetRows != *claimed.WriteSetRows) {
return fmt.Errorf("operation %s 的 writeset_rows 没有按声称值落库", operationID)
}
if persisted.RecordedAt == nil {
return fmt.Errorf("operation %s 的 anchors_at 仍为空:锚点没被真正登记", operationID)
}
return nil
}
// BeginLibraryAttempt 取得这个作业的执行权并激活一个新代次。
//
// 三件事必须在同一个事务里:判租约(谁能执行)→ 递增 epoch(谁是当前代次)→
// 写作业行的租约(守卫读它)。分成三步就会出现"两个 worker 都认为自己是当前代次"。
// 顺序与理由(R2):
// 1. 锁作业行、再锁 **operation 行**:epoch 是 operation 维度的单调量,两个作业同时
// 接管同一个 operation 时,只有锁 operation 才能保证"下一个 epoch"不重号。
// 2. 同一次投递重复调用 ⇒ **幂等**:返回同一个活跃代次并续租,不新增谱系。
// 3. 有别的活跃代次:
// - 它的租约仍未过期 ⇒ 拒绝(ErrLibraryAttemptHeld)—— 不能把还在跑的代次踢掉;
// - 租约已过期(或就是本作业的旧代次)⇒ 结束它(abandoned,保留行),再开新代次。
// 4. 新代次 = max(epoch)+1,**新增一行**(历史不可变);同时把同 operation 其它作业行
// 的租约清空,让它们的守卫立刻失效(旧 job 行上的 epoch/owner 字段还会留着,
// 所以守卫必须同时看 attempts 的活跃状态,见 libraryFence)。
func (s *Store) BeginLibraryAttempt(ctx context.Context, jobID, owner string, lease time.Duration, now time.Time) (governance.LibraryAttempt, error) {
if jobID == "" || owner == "" {
return governance.LibraryAttempt{}, errors.New("begin library attempt requires job and owner")
@@ -157,63 +286,116 @@ func (s *Store) BeginLibraryAttempt(ctx context.Context, jobID, owner string, le
defer func() { _ = tx.Rollback(ctx) }()
var operationID, leaseOwner, queueJobID string
var epoch int
var leaseUntil *time.Time
var jobEpoch int
var jobLeaseUntil *time.Time
err = tx.QueryRow(ctx, `
SELECT operation_id, attempt_epoch, lease_owner, lease_until, queue_job_id
FROM library_jobs WHERE id=$1 FOR UPDATE`, jobID).Scan(&operationID, &epoch, &leaseOwner, &leaseUntil, &queueJobID)
FROM library_jobs WHERE id=$1 FOR UPDATE`, jobID).Scan(&operationID, &jobEpoch, &leaseOwner, &jobLeaseUntil, &queueJobID)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return governance.LibraryAttempt{}, fmt.Errorf("library job %s not found", jobID)
}
return governance.LibraryAttempt{}, fmt.Errorf("load library job for attempt: %w", err)
}
// 同一 owner 重入是幂等(重试/队列重投);别人持有未过期租约就必须让路。
if leaseOwner != "" && leaseOwner != owner && leaseUntil != nil && leaseUntil.After(now.UTC()) {
return governance.LibraryAttempt{}, ErrLibraryAttemptHeld
until := now.Add(lease).UTC()
// 旧作业(迁移前没有 operation):没有代次谱系可建,只做租约互斥。
if operationID == "" {
if leaseOwner != "" && leaseOwner != owner && jobLeaseUntil != nil && jobLeaseUntil.After(now.UTC()) {
return governance.LibraryAttempt{}, ErrLibraryAttemptHeld
}
next := jobEpoch + 1
if _, err := tx.Exec(ctx, `
UPDATE library_jobs SET attempt_epoch=$2, lease_owner=$3, lease_until=$4, updated_at=$5 WHERE id=$1`,
jobID, next, owner, until, now.UTC()); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("activate legacy library attempt: %w", err)
}
if err := tx.Commit(ctx); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("commit legacy library attempt: %w", err)
}
return governance.LibraryAttempt{JobID: jobID, Epoch: next, QueueJobID: queueJobID, Owner: owner,
State: governance.LibraryAttemptRunning, StartedAt: now.UTC(), LeaseUntil: &until}, nil
}
nextEpoch := epoch + 1
if nextEpoch < 1 {
nextEpoch = 1
// epoch 分配必须串行到 operation 粒度。
if _, err := tx.Exec(ctx, `SELECT id FROM library_operations WHERE id=$1 FOR UPDATE`, operationID); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("lock library operation for attempt: %w", err)
}
// 代次必须按 operation 单调递增,而不是按作业行:同一个意图的重试/接管会**新开作业行**,
// 若按作业行各自从 1 数起,旧代次(1)与新代次(1)就分不出先后,fencing 失效。
if operationID != "" {
var maxEpoch *int
if err := tx.QueryRow(ctx, `SELECT MAX(epoch) FROM library_attempts WHERE operation_id=$1`,
operationID).Scan(&maxEpoch); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("load library attempt epoch: %w", err)
// 当前活跃代次(部分唯一索引保证至多一行)。
var activeEpoch int
var activeJob, activeOwner, activeState string
var activeUntil *time.Time
activeErr := tx.QueryRow(ctx, `
SELECT epoch, job_id, owner, state, lease_until FROM library_attempts
WHERE operation_id=$1 AND state IN ('claimed','running')`, operationID).Scan(
&activeEpoch, &activeJob, &activeOwner, &activeState, &activeUntil)
switch {
case activeErr != nil && !errors.Is(activeErr, pgx.ErrNoRows):
return governance.LibraryAttempt{}, fmt.Errorf("load active library attempt: %w", activeErr)
case activeErr == nil && activeOwner == owner && activeJob == jobID:
// 同一次投递重复调用:续租,返回同一代次(不新增谱系)。
if _, err := tx.Exec(ctx, `
UPDATE library_attempts SET heartbeat_at=$4, lease_until=$5 WHERE operation_id=$1 AND epoch=$2 AND owner=$3`,
operationID, activeEpoch, owner, now.UTC(), until); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("renew library attempt: %w", err)
}
next := 1
if maxEpoch != nil && *maxEpoch >= next {
next = *maxEpoch + 1
if _, err := tx.Exec(ctx, `
UPDATE library_jobs SET attempt_epoch=$2, lease_owner=$3, lease_until=$4, updated_at=$5 WHERE id=$1`,
jobID, activeEpoch, owner, until, now.UTC()); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("renew library job lease: %w", err)
}
if nextEpoch < next {
nextEpoch = next
if err := tx.Commit(ctx); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("commit library attempt: %w", err)
}
return governance.LibraryAttempt{OperationID: operationID, Epoch: activeEpoch, JobID: jobID,
QueueJobID: queueJobID, Owner: owner, State: governance.LibraryAttemptRunning,
StartedAt: now.UTC(), LeaseUntil: &until}, nil
case activeErr == nil && activeUntil != nil && activeUntil.After(now.UTC()):
// 别人(另一个作业或另一个执行者)仍持有未过期租约:让路,不许抢占。
return governance.LibraryAttempt{}, fmt.Errorf("%w(operation %s 的 %d 代次由 %s/%s 持有)",
ErrLibraryAttemptHeld, operationID, activeEpoch, activeJob, activeOwner)
case activeErr == nil:
// 租约已过期:结束它(行保留),再开新代次。
if _, err := tx.Exec(ctx, `
UPDATE library_attempts SET state='abandoned', finished_at=$3, outcome=$4, lease_until=NULL
WHERE operation_id=$1 AND epoch=$2`, operationID, activeEpoch, now.UTC(),
"lease expired: superseded by a new attempt"); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("abandon stale library attempt: %w", err)
}
}
var maxEpoch *int
if err := tx.QueryRow(ctx, `SELECT MAX(epoch) FROM library_attempts WHERE operation_id=$1`,
operationID).Scan(&maxEpoch); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("load library attempt epoch: %w", err)
}
next := 1
if maxEpoch != nil && *maxEpoch >= next {
next = *maxEpoch + 1
}
attempt := governance.LibraryAttempt{
OperationID: operationID, Epoch: nextEpoch, JobID: jobID, QueueJobID: queueJobID,
Owner: owner, State: governance.LibraryAttemptRunning, StartedAt: now.UTC(),
OperationID: operationID, Epoch: next, JobID: jobID, QueueJobID: queueJobID,
Owner: owner, State: governance.LibraryAttemptRunning, StartedAt: now.UTC(), LeaseUntil: &until,
}
until := now.Add(lease).UTC()
attempt.LeaseUntil = &until
if operationID != "" {
// 一个代次只对应一个作业行:重复 Begin(同 job)不再新增谱系。
if _, err := tx.Exec(ctx, `
if _, err := tx.Exec(ctx, `
INSERT INTO library_attempts (operation_id, epoch, job_id, queue_job_id, owner, state, started_at, heartbeat_at, lease_until)
VALUES ($1,$2,$3,$4,$5,'running',$6,$6,$7)
ON CONFLICT (job_id) DO UPDATE SET
owner=EXCLUDED.owner, state='running', lease_until=EXCLUDED.lease_until, heartbeat_at=EXCLUDED.heartbeat_at`,
operationID, nextEpoch, jobID, queueJobID, owner, now.UTC(), until); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("record library attempt: %w", err)
}
VALUES ($1,$2,$3,$4,$5,'running',$6,$6,$7)`,
operationID, next, jobID, queueJobID, owner, now.UTC(), until); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("record library attempt: %w", err)
}
if _, err := tx.Exec(ctx, `
UPDATE library_jobs SET attempt_epoch=$2, lease_owner=$3, lease_until=$4, updated_at=$5 WHERE id=$1`,
jobID, nextEpoch, owner, until, now.UTC()); err != nil {
jobID, next, owner, until, now.UTC()); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("activate library attempt: %w", err)
}
// 同一 operation 的其它作业行(续作留下/旧作业)立刻失去租约:
// 它们的守卫写从这一刻起失败,即使行上的 epoch/owner 字段还没被改写。
if _, err := tx.Exec(ctx, `
UPDATE library_jobs SET lease_owner='', lease_until=NULL, updated_at=$3
WHERE operation_id=$1 AND id<>$2`, operationID, jobID, now.UTC()); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("revoke sibling library job leases: %w", err)
}
if err := tx.Commit(ctx); err != nil {
return governance.LibraryAttempt{}, fmt.Errorf("commit library attempt: %w", err)
}
@@ -226,9 +408,13 @@ func (s *Store) FinishLibraryAttempt(ctx context.Context, attempt governance.Lib
if attempt.OperationID == "" {
return nil // 旧作业没有 operation:没有代次可结
}
// 只允许结束**还没结束**的那一代:已经 finished/abandoned 的代次是历史事实,
// 迟到的 finish 不能把它的 outcome 改写成"成功"(否则接管后旧执行者一收尾,
// 历史就变成了对自己有利的版本,排障时无从分辨谁真的跑完)。
// 终态集合与服务端的 state CHECK 一致;`completed` 只允许从活跃态进来一次。
tag, err := s.db.Exec(ctx, `
UPDATE library_attempts SET state=$4, finished_at=$5, outcome=$6, lease_until=NULL
WHERE operation_id=$1 AND epoch=$2 AND owner=$3`,
WHERE operation_id=$1 AND epoch=$2 AND owner=$3 AND state IN ('claimed','running')`,
attempt.OperationID, attempt.Epoch, attempt.Owner, string(state), now.UTC(), outcome)
if err != nil {
return fmt.Errorf("finish library attempt: %w", err)
@@ -11,15 +11,40 @@ import (
"knowledge-governance/internal/governance"
)
// 写入栅栏:所有"业务状态 / 维护锁"的写都必须证明调用方仍是当前执行代次。
// 写入栅栏:所有"业务状态 / 维护锁"的写都必须证明调用方仍是**当前活跃代次**。
//
// 为什么单独放一份:F13 的根因不是某一条 SQL 写错了,而是"同一个 job_id 被两个代次
// 同时使用"这件事本身没有被表达出来。只要有一处更新不带代次条件,失租的旧执行就能
// 用同一个 job_id 写业务终态、释放别人刚拿到的维护锁。所以这里给每个可变入口配一个
// 带守卫的版本,worker 只准走这些版本;不带守卫的旧方法留给 API(它代表"当前意图",
// 没有执行代次)。
// 两条条件缺一不可:
// 1. 作业行上的 (attempt_epoch, lease_owner) 就是这次的 (epoch, owner);
// 2. library_attempts 里那一代次仍是 active(claimed/running)。
//
// 为什么第 2 条不能省(R2):接管/续跑会激活新一代次,而**旧作业行上的 epoch/owner
// 字段仍原样保留**。只比较旧作业行,旧执行器的守卫会继续通过——它就能继续写业务状态、
// 释放别人刚拿到的锁。把"该代次仍活跃"作为条件后,旧代次被结束的那一刻起,
// 它的所有守卫写都会失败。
//
// 参数契约(R1):作业 id 取自**被更新行自己的列**(`library_maintenance.owner_job_id`
// 等),不写死 `$1`——维护类更新的 `$1` 是 scope/instance,写死 `$1` 会让守卫永远匹配
// 不到作业,正常写与释放全部失败。列表达式由调用方传入,序号漂移不可能再发生。
const epochFence = `EXISTS (SELECT 1 FROM library_jobs j WHERE j.id=$1 AND j.attempt_epoch=$%d AND j.lease_owner=$%d)`
// libraryFence 生成"调用方仍是当前活跃代次"的谓词。jobIDExpr 是**列表达式**。
func libraryFence(jobIDExpr string, epochParam, ownerParam int) string {
return fmt.Sprintf(`EXISTS (
SELECT 1 FROM library_jobs j
WHERE j.id = %s
AND j.attempt_epoch = $%d
AND j.lease_owner = $%d
AND (
j.operation_id = ''
OR EXISTS (
SELECT 1 FROM library_attempts a
WHERE a.operation_id = j.operation_id
AND a.epoch = j.attempt_epoch
AND a.owner = j.lease_owner
AND a.state IN ('claimed','running')
)
)
)`, jobIDExpr, epochParam, ownerParam)
}
// AdvanceLibraryJobGuarded 只在调用方仍是该作业当前代次时写业务状态。
func (s *Store) AdvanceLibraryJobGuarded(ctx context.Context, attempt governance.LibraryAttempt,
@@ -38,7 +63,7 @@ UPDATE library_jobs SET
target_snapshot_id = CASE WHEN $7='' THEN target_snapshot_id ELSE $7 END,
updated_at=$8,
finished_at=COALESCE($9, finished_at)
WHERE id=$1 AND attempt_epoch=$10 AND lease_owner=$11`,
WHERE id=$1 AND `+libraryFence("$1", 10, 11),
attempt.JobID, string(state), stage, message, recovery, snapshotID, targetSnapshotID, now.UTC(), finished,
attempt.Epoch, attempt.Owner)
if err != nil {
@@ -62,8 +87,7 @@ UPDATE library_job_stages SET
bytes_done = COALESCE($7, bytes_done),
detail = CASE WHEN $8='' THEN detail ELSE $8 END,
updated_at = $9
WHERE job_id=$1 AND stage=$2
AND EXISTS (SELECT 1 FROM library_jobs j WHERE j.id=$1 AND j.attempt_epoch=$10 AND j.lease_owner=$11)`,
WHERE job_id=$1 AND stage=$2 AND `+libraryFence("$1", 10, 11),
attempt.JobID, stage, state, total, completed, bytesTotal, bytesDone, detail, now.UTC(),
attempt.Epoch, attempt.Owner)
if err != nil {
@@ -81,7 +105,8 @@ func (s *Store) ReleaseLibraryMaintenanceGuarded(ctx context.Context, scope, own
tag, err := s.db.Exec(ctx, `
UPDATE library_maintenance SET state='released', released_at=$3, updated_at=$3
WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL
AND `+fmt.Sprintf(epochFence, 4, 5), scope, ownerJobID, now.UTC(), attempt.Epoch, attempt.Owner)
AND `+libraryFence("library_maintenance.owner_job_id", 4, 5), scope, ownerJobID, now.UTC(),
attempt.Epoch, attempt.Owner)
if err != nil {
return fmt.Errorf("release library maintenance (guarded): %w", err)
}
@@ -100,7 +125,8 @@ func (s *Store) ReleaseInstanceMaintenanceGuarded(ctx context.Context, instance,
tag, err := s.db.Exec(ctx, `
UPDATE library_instance_maintenance SET state='released', released_at=$3, updated_at=$3
WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL
AND `+fmt.Sprintf(epochFence, 4, 5), instance, ownerJobID, now.UTC(), attempt.Epoch, attempt.Owner)
AND `+libraryFence("library_instance_maintenance.owner_job_id", 4, 5), instance, ownerJobID, now.UTC(),
attempt.Epoch, attempt.Owner)
if err != nil {
return fmt.Errorf("release instance maintenance (guarded): %w", err)
}
@@ -120,7 +146,8 @@ func (s *Store) UpdateLibraryMaintenanceStateGuarded(ctx context.Context, scope,
tag, err := s.db.Exec(ctx, `
UPDATE library_maintenance SET state=$3, note=$4, updated_at=$5
WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL
AND `+fmt.Sprintf(epochFence, 6, 7), scope, ownerJobID, state, note, now.UTC(), attempt.Epoch, attempt.Owner)
AND `+libraryFence("library_maintenance.owner_job_id", 6, 7), scope, ownerJobID, state, note, now.UTC(),
attempt.Epoch, attempt.Owner)
if err != nil {
return fmt.Errorf("update library maintenance (guarded): %w", err)
}
@@ -138,7 +165,8 @@ func (s *Store) UpdateInstanceMaintenanceStateGuarded(ctx context.Context, insta
tag, err := s.db.Exec(ctx, `
UPDATE library_instance_maintenance SET state=$3, note=$4, updated_at=$5
WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL
AND `+fmt.Sprintf(epochFence, 6, 7), instance, ownerJobID, state, note, now.UTC(), attempt.Epoch, attempt.Owner)
AND `+libraryFence("library_instance_maintenance.owner_job_id", 6, 7), instance, ownerJobID, state, note,
now.UTC(), attempt.Epoch, attempt.Owner)
if err != nil {
return fmt.Errorf("update instance maintenance (guarded): %w", err)
}
@@ -154,17 +182,22 @@ WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL
// fencedWriteError 区分"作业不存在"与"代次已过期":两者都要停下,但文案不能误导运维。
func (s *Store) fencedWriteError(ctx context.Context, attempt governance.LibraryAttempt, action string) error {
var epoch int
var owner string
err := s.db.QueryRow(ctx, `SELECT attempt_epoch, lease_owner FROM library_jobs WHERE id=$1`,
attempt.JobID).Scan(&epoch, &owner)
var owner, operation string
err := s.db.QueryRow(ctx, `SELECT attempt_epoch, lease_owner, operation_id FROM library_jobs WHERE id=$1`,
attempt.JobID).Scan(&epoch, &owner, &operation)
if errors.Is(err, pgx.ErrNoRows) {
return fmt.Errorf("%s:作业 %s 不存在", action, attempt.JobID)
}
if err != nil {
return fmt.Errorf("%s:读不到作业代次:%w", action, err)
}
return fmt.Errorf("%w(%s:本代次 %d/%s,当前 %d/%s)", governance.ErrLibraryEpochStale,
action, attempt.Epoch, attempt.Owner, epoch, owner)
var state string
if operation != "" {
_ = s.db.QueryRow(ctx, `SELECT state FROM library_attempts WHERE operation_id=$1 AND epoch=$2`,
operation, attempt.Epoch).Scan(&state)
}
return fmt.Errorf("%w(%s:本代次 %d/%s,作业行 %d/%s,代次状态 %q)", governance.ErrLibraryEpochStale,
action, attempt.Epoch, attempt.Owner, epoch, owner, state)
}
func (s *Store) SetInstanceMaintenanceIngressGuarded(ctx context.Context, instance, ownerJobID, state, note string,
@@ -175,7 +208,8 @@ func (s *Store) SetInstanceMaintenanceIngressGuarded(ctx context.Context, instan
tag, err := s.db.Exec(ctx, `
UPDATE library_instance_maintenance SET ingress_state=$3, note=$4, updated_at=$5
WHERE instance=$1 AND owner_job_id=$2 AND released_at IS NULL
AND `+fmt.Sprintf(epochFence, 6, 7), instance, ownerJobID, state, note, now.UTC(), attempt.Epoch, attempt.Owner)
AND `+libraryFence("library_instance_maintenance.owner_job_id", 6, 7), instance, ownerJobID, state, note,
now.UTC(), attempt.Epoch, attempt.Owner)
if err != nil {
return fmt.Errorf("update instance maintenance ingress (guarded): %w", err)
}
@@ -198,7 +232,8 @@ func (s *Store) SetLibraryMaintenanceIngressGuarded(ctx context.Context, scope,
tag, err := s.db.Exec(ctx, `
UPDATE library_maintenance SET ingress_state=$3, note=$4, updated_at=$5
WHERE scope=$1 AND owner_job_id=$2 AND released_at IS NULL
AND `+fmt.Sprintf(epochFence, 6, 7), scope, ownerJobID, state, note, now.UTC(), attempt.Epoch, attempt.Owner)
AND `+libraryFence("library_maintenance.owner_job_id", 6, 7), scope, ownerJobID, state, note, now.UTC(),
attempt.Epoch, attempt.Owner)
if err != nil {
return fmt.Errorf("update library maintenance ingress (guarded): %w", err)
}
@@ -49,8 +49,10 @@ CREATE TABLE IF NOT EXISTS library_attempts (
outcome TEXT NOT NULL DEFAULT '',
PRIMARY KEY (operation_id, epoch)
);
-- 一个作业行只对应一个代次:重复 Begin 不新增谱系。
CREATE UNIQUE INDEX IF NOT EXISTS library_attempts_job ON library_attempts(job_id);
-- 一个作业行可以有多条**不可变**代次:接管/续跑本身就是"同一作业的新执行",
-- 若按 (job_id) 唯一,第二次接管只能改写旧行 —— 那就不是不可变历史了(R2)。
-- 这里只保留普通索引,供"这个作业的所有代次"查询。
CREATE INDEX IF NOT EXISTS library_attempts_job ON library_attempts(job_id);
-- 同一 operation 同时只能有一个活跃代次:第二个 resumer 的失败不能释放另一个代次的锁。
CREATE UNIQUE INDEX IF NOT EXISTS library_attempts_active
ON library_attempts(operation_id) WHERE state IN ('claimed','running');
@@ -9,6 +9,8 @@ import (
"crypto/sha256"
"encoding/hex"
"net/url"
"os"
"strings"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
@@ -25,6 +27,14 @@ func IsolatedDatabaseURL(t *testing.T, databaseURL string) string {
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
t.Fatalf("TEST_DATABASE_URL must be a PostgreSQL URL: %v", err)
}
// 护栏:这些用例会在目标库里建/删 schema。指到业务库上就会在业务库里反复
// 建表删表(哪怕只删自己的 schema,也是在业务库上动 DDL)。所以库名必须明确
// 是一次性的(含 test),否则要求显式声明,绝不"顺便"跑在业务库上。
if databaseName := strings.TrimPrefix(parsed.Path, "/"); !strings.Contains(strings.ToLower(databaseName), "test") &&
os.Getenv("KG_ALLOW_NON_TEST_DATABASE_URL") != "1" {
t.Fatalf("TEST_DATABASE_URL 指向的库 %q 看起来不是一次性测试库:拒绝在其中建/删 schema"+
"(确属一次性库请设 KG_ALLOW_NON_TEST_DATABASE_URL=1)", databaseName)
}
digest := sha256.Sum256([]byte(t.Name()))
schema := "kgtest_" + hex.EncodeToString(digest[:8])
ctx := context.Background()
@@ -33,6 +43,11 @@ func IsolatedDatabaseURL(t *testing.T, databaseURL string) string {
t.Fatalf("connect integration database: %v", err)
}
defer admin.Close()
// 上一次被 kill 的运行可能留下同名 schema(cleanup 没跑到):先清掉自己的,
// 否则紧接着的 CREATE 会撞名,整包用例连"跑起来"都做不到。只清本用例自己的名字。
if _, err := admin.Exec(ctx, "DROP SCHEMA IF EXISTS "+schema+" CASCADE"); err != nil {
t.Fatalf("drop stale isolated schema: %v", err)
}
if _, err := admin.Exec(ctx, "CREATE SCHEMA "+schema); err != nil {
t.Fatalf("create isolated integration schema: %v", err)
}
@@ -0,0 +1,43 @@
package v071
import (
"context"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5/pgxpool"
)
// ProbeTargetIdentity 只读地探测"这个 DSN + 知识库"此刻在目标里是什么。
//
// 用途是执行域准入(R4):worker 注册整库发布 handler 之前必须拿到**目标实况**,
// 而不是回声自己的配置。这里复用与发布路径同一份 SQL(liveTargetContractSQL),
// 避免"准入读一份、发布读另一份"的分叉。探测连接短命、只读、失败即返回错误。
func ProbeTargetIdentity(ctx context.Context, dsn string, tenantID uint64, knowledgeBaseID string) (LiveTargetContract, error) {
if dsn == "" || tenantID == 0 || knowledgeBaseID == "" {
return LiveTargetContract{}, errors.New("probe requires dsn, tenant and knowledge base")
}
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return LiveTargetContract{}, fmt.Errorf("connect target for probe: %w", err)
}
defer pool.Close()
probeCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
var (
contract LiveTargetContract
declared string
indexName string
)
if err := pool.QueryRow(probeCtx, liveTargetContractSQL, knowledgeBaseID, tenantID).Scan(
&contract.SchemaVersion, &contract.Dirty, &contract.EmbeddingModel, &contract.EmbeddingDimension,
&contract.ModelBound, &declared, &contract.EngineType, &indexName); err != nil {
return LiveTargetContract{}, fmt.Errorf("read target identity: %w", err)
}
if !contract.ModelBound && declared != "" {
contract.EmbeddingModel = declared
}
contract.IndexName = indexName
return contract, nil
}