feat(library): 快照搬运先于维护、双向校验与云端版本回传
为什么改:云端 release-executor 与目标同机后,本机封存的快照必须"先到、先验证" 才允许开维护窗口。此前只有 pack/unpack 两条人工命令:UI 点发布时执行器读到的是 本机归档路径,云端根本没有这份快照,作业只能在装配阶段失败;而预备份落在云端, 本机页面也读不到。用户已批准补齐这条链路。 搬运(新包 internal/libraryhandoff):本机 Load 逐 blob 重算 → Pack → SSH 流式写入 云端暂存 → 原子改名 → 云端 snapshot-unpack 再逐 blob 校验 → 读回版本清单摘要自证 → scope/tenant/知识库身份与治理库作业一致。只有全部通过才投递 library.sync|restore, 失败时发布作业根本不存在(目标不进维护、不写知识)。远端命令是固定 argv,容器名与 路径只来自部署配置;同 ID 异内容拒绝;同一快照本机 flock 串行化;暂存 bundle 用后即删, 归档本体永不自动清理。 回传:capture 角色按已成功作业的 snapshot_id 扫描,把本机缺失的云端快照 Fetch 回来 (云端 pack → SSH 流回 → 本机 Unpack 逐 blob 校验)。回传成功后 API/页面既有的列表、 详情、差异、存储与历史恢复路径读到的就是真实字节;失败则版本不出现在本机,不会被误报成可恢复。 角色:搬运与回传只注册在 capture;若接线到 full(同时消费发布作业)启动即失败, 避免与云端执行器抢同一批作业。release-executor 仍不拥有治理库 DDL,也不消费 QA/邮件/来源扫描。 验证:internal/libraryhandoff 全绿(跨进程 push/fetch、身份不符拒绝、幂等复用、 并发串行、篡改与未知快照不落版本、路径穿越拒绝、远端错误不当作"没有"); cmd/worker 在隔离 fixture 库(独立测试 PG)上验证"校验通过才入队发布、失败不入队"; httpapi 队列决策单测;相关包 go test 全绿;go vet 全仓通过。 未部署、未推送、未构建镜像:云端接线与验收仍由父执行。真实云链路(含 SSH 通道) 本轮没有云侧写操作,只有本机跨进程与隔离 fixture 证据。
This commit is contained in:
@@ -1,5 +1,32 @@
|
||||
# Changelog
|
||||
|
||||
## 开发快照(跨主机自动归档交接:搬运先于维护 · 双向校验 · 云端版本回传)
|
||||
|
||||
### Added
|
||||
|
||||
- **发布/恢复作业先搬运、后执行**:快照搬运接线(`LIBRARY_HANDOFF_*`)时,API 把作业投递成
|
||||
`library.snapshot.handoff`(capture 角色消费),由它把不可变快照送到云端 release-executor 并
|
||||
**双向校验**(本机 `Load` 逐 blob 重算 → 云端 `snapshot-unpack` 再逐 blob 重算 → 版本清单摘要自证 →
|
||||
scope/tenant/知识库身份与治理库作业一致)之后,才投递真正把目标置入维护窗口的 `library.sync|restore`。
|
||||
搬运失败/超时/校验不过时发布作业**根本不存在**,目标不会被停用、不写任何知识。
|
||||
- 新包 `internal/libraryhandoff`:`Push`(本机→云端)与 `Fetch`(云端→本机)共用一套校验与暂存规则;
|
||||
远端命令是固定 argv(容器名/路径来自部署配置,调用方只能给快照 ID 与期望身份);同一快照用本机
|
||||
flock 串行化,云端内容寻址 + Link 幂等;同 ID 异内容直接拒绝;暂存 bundle 用后即删,归档本体不自动清理。
|
||||
- capture 角色新增**云端版本回传**扫描:按治理库已成功作业的 `snapshot_id`,把本机缺失的云端快照
|
||||
`Fetch` 回本机归档,因此页面/API 既有的列表、详情、差异、存储与历史恢复路径读到的都是真实字节。
|
||||
- `libraryctl snapshot-push` / `snapshot-fetch`:运维可直接搬一份快照并打印逐 blob 身份用于对拍。
|
||||
|
||||
### Changed
|
||||
|
||||
- `LIBRARY_WORKER_ROLE` 的副作用边界继续收敛:搬运与回传只注册在 capture 角色;若把搬运接线到
|
||||
`full` 角色(同时消费发布作业)启动即失败,避免与云端执行器抢同一批作业。
|
||||
- `library_jobs` 的孤儿收拾按角色负责的 kind 过滤;治理库 DDL 仍只属于控制面进程。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 入口闸门不再在 10 秒静默后掐断上游流:建连超时与读超时分离(连上即 `settimeout(None)`),
|
||||
半关闭按 TCP 语义单向传播,避免响应被截断;排空超时仍以 exit 2 表达失败。
|
||||
|
||||
## 开发快照(发布管理统一修复:执行身份 · 终态收尾 · 准入与数据身份)
|
||||
|
||||
### Changed
|
||||
|
||||
@@ -3,6 +3,7 @@ package main
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
@@ -17,6 +18,7 @@ import (
|
||||
"knowledge-governance/internal/backuphost"
|
||||
"knowledge-governance/internal/dingtalkdiscovery"
|
||||
"knowledge-governance/internal/httpapi"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/monitoring"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/platform/weknora"
|
||||
@@ -168,7 +170,14 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
if getenv("DOBOTAI_OPS_TENANT_KEY") != "" {
|
||||
qaSyncStore = qaOrganizationStore
|
||||
}
|
||||
// 快照搬运是否接线:只用于决定"发布/恢复作业先入哪个队列"。
|
||||
// 配置不完整即启动失败(半接线会让作业以"快照不在执行器"失败)。
|
||||
_, handoffConfigured, handoffErr := libraryhandoff.FromEnv(getenv)
|
||||
if handoffErr != nil {
|
||||
return fmt.Errorf("快照搬运接线无效: %w", handoffErr)
|
||||
}
|
||||
handler, err := httpapi.NewHandler(httpapi.Config{
|
||||
LibraryHandoffEnabled: handoffConfigured,
|
||||
QAAI: qaopsai.NewFromEnvironment(pool, getenv, config.DatabaseURL),
|
||||
QAWeeklyMail: qawemail.NewStore(pool, qaSettings.TenantKey, getenv("QA_AI_MASTER_KEY_FILE"), nil),
|
||||
QAAITrustTLSProxy: getenv("QA_AI_TRUST_TLS_PROXY") == "true",
|
||||
|
||||
@@ -34,6 +34,7 @@ import (
|
||||
|
||||
"knowledge-governance/internal/librarydeadletter"
|
||||
"knowledge-governance/internal/libraryfreeze"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/librarymaintenance"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||
@@ -60,6 +61,9 @@ type options struct {
|
||||
job string
|
||||
dryRun bool
|
||||
snapshot string
|
||||
// kb/tenant 是 snapshot-push 要核对的期望身份(来自治理库作业,不是 bundle 自报)。
|
||||
kb string
|
||||
tenant uint64
|
||||
// bundle 是快照跨主机包路径(snapshot-pack 输出 / snapshot-unpack 输入)。
|
||||
bundle string
|
||||
// releasedDeadLetters 是治理平台审计过的精确解除指纹(逗号分隔)。它只让这些
|
||||
@@ -106,6 +110,10 @@ func main() {
|
||||
runErr = runSnapshotPack(opts)
|
||||
case "snapshot-unpack":
|
||||
runErr = runSnapshotUnpack(opts)
|
||||
case "snapshot-push":
|
||||
runErr = runSnapshotPush(opts)
|
||||
case "snapshot-fetch":
|
||||
runErr = runSnapshotFetch(opts)
|
||||
default:
|
||||
usage()
|
||||
os.Exit(2)
|
||||
@@ -130,6 +138,8 @@ func usage() {
|
||||
snapshot-pack export one sealed snapshot into a single verifiable bundle file
|
||||
snapshot-unpack verify a bundle and install it into this archive (content-addressed,
|
||||
version manifest is the commit point; repeated import is idempotent)
|
||||
snapshot-push send one sealed snapshot to the cloud release executor (verify on both sides)
|
||||
snapshot-fetch bring a cloud-held snapshot back into this archive (verified)
|
||||
|
||||
Flags (defaults target the local WKNFull test stack):
|
||||
-source-app WKNFull-app container whose env describes the source
|
||||
@@ -187,6 +197,8 @@ func parseFlags(command string, args []string) (options, []string, error) {
|
||||
set.StringVar(&opts.snapshot, "snapshot", "", "snapshot id for verify")
|
||||
set.StringVar(&opts.releasedDeadLetters, "released-dead-letters", "", "audited dead-letter fingerprints (comma separated)")
|
||||
set.StringVar(&opts.bundle, "bundle", "", "snapshot bundle path (snapshot-pack / snapshot-unpack)")
|
||||
set.StringVar(&opts.kb, "kb", "", "expected knowledge base id (snapshot-push)")
|
||||
set.Uint64Var(&opts.tenant, "tenant", 0, "expected tenant id (snapshot-push)")
|
||||
if err := set.Parse(args); err != nil {
|
||||
return opts, nil, err
|
||||
}
|
||||
@@ -1206,3 +1218,54 @@ func runSnapshotUnpack(opts options) error {
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
// runSnapshotPush 把一份本机封存快照送到云端发布执行器(先校验字节,再校验身份)。
|
||||
// 只读本机归档;远端只写暂存目录与目标归档,不做任何目标库动作。
|
||||
func runSnapshotPush(opts options) error {
|
||||
if opts.snapshot == "" || opts.scope == "" || opts.kb == "" || opts.tenant == 0 {
|
||||
return errors.New("snapshot-push 需要 -snapshot <id> -scope internal|external -tenant <n> -kb <uuid>")
|
||||
}
|
||||
handoff, err := openHandoff(opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
evidence, err := handoff.Push(context.Background(), libraryhandoff.Identity{
|
||||
SnapshotID: opts.snapshot, Scope: opts.scope, TenantID: opts.tenant, KnowledgeBaseID: opts.kb})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
printJSON(evidence)
|
||||
return nil
|
||||
}
|
||||
|
||||
// runSnapshotFetch 把云端的快照取回本机归档(预备份/云端版本的读回)。
|
||||
func runSnapshotFetch(opts options) error {
|
||||
if opts.snapshot == "" {
|
||||
return errors.New("snapshot-fetch 需要 -snapshot <id>")
|
||||
}
|
||||
handoff, err := openHandoff(opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
evidence, err := handoff.Fetch(context.Background(), opts.snapshot)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
printJSON(evidence)
|
||||
return nil
|
||||
}
|
||||
|
||||
// openHandoff 只认部署注入的配置:CLI 不接受调用方指定远端路径/容器/上游地址。
|
||||
func openHandoff(opts options) (*libraryhandoff.Handoff, error) {
|
||||
config, configured, err := libraryhandoff.FromEnv(os.Getenv)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !configured {
|
||||
return nil, errors.New("快照搬运未接线(缺少 LIBRARY_HANDOFF_*)")
|
||||
}
|
||||
if opts.archiveRoot != "" && config.LocalArchive == "" {
|
||||
config.LocalArchive = opts.archiveRoot
|
||||
}
|
||||
return libraryhandoff.New(config, nil)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/appconfig"
|
||||
"knowledge-governance/internal/governance"
|
||||
"knowledge-governance/internal/jobs"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
)
|
||||
|
||||
// libraryHandoffJobType 与 API 侧常量同值:API 用它把"发布/恢复"拆成
|
||||
// "先搬运、后发布"。搬运作业由 capture 角色执行(只有本机有源归档与 SSH 凭据),
|
||||
// release-executor 永不注册它——云端没有源归档,也没有回本机的通道。
|
||||
const libraryHandoffJobType = "library.snapshot.handoff"
|
||||
|
||||
// libraryHandoffStage 是搬运在作业阶段表里的名字(页面能看出卡在搬运还是发布)。
|
||||
const libraryHandoffStage = "handoff"
|
||||
|
||||
type libraryHandoffPayload struct {
|
||||
LibraryJobID string `json:"library_job_id"`
|
||||
Scope string `json:"scope"`
|
||||
TargetSnapshotID string `json:"target_snapshot_id"`
|
||||
NextType string `json:"next_type"`
|
||||
}
|
||||
|
||||
// registerLibraryHandoffHandlers 注册"先把快照送到执行器"的作业。
|
||||
//
|
||||
// 顺序是这条链的安全边界:push 成功(云端逐 blob 校验通过、清单摘要自证、身份一致)
|
||||
// 之后才投递真正的发布作业。push 失败时发布作业根本不存在,executor 不会开维护窗口、
|
||||
// 不会写任何知识数据。
|
||||
func registerLibraryHandoffHandlers(handlers map[string]jobs.Handler, store *postgresstore.Store,
|
||||
config appconfig.Config, handoff *libraryhandoff.Handoff) {
|
||||
handlers[libraryHandoffJobType] = func(ctx context.Context, job jobs.Job) error {
|
||||
var payload libraryHandoffPayload
|
||||
if err := json.Unmarshal(job.Payload, &payload); err != nil ||
|
||||
payload.LibraryJobID == "" || payload.TargetSnapshotID == "" ||
|
||||
(payload.Scope != "internal" && payload.Scope != "external") {
|
||||
return fmt.Errorf("invalid library handoff payload")
|
||||
}
|
||||
return runLibraryHandoff(ctx, store, config, handoff, payload)
|
||||
}
|
||||
}
|
||||
|
||||
func runLibraryHandoff(ctx context.Context, store *postgresstore.Store, config appconfig.Config,
|
||||
handoff *libraryhandoff.Handoff, payload libraryHandoffPayload) error {
|
||||
now := time.Now().UTC()
|
||||
// 作业身份必须与载荷逐项一致:载荷来自队列,业务作业才是治理库的事实。
|
||||
record, err := store.LibraryJob(ctx, payload.LibraryJobID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("读不到业务作业 %s:%w", payload.LibraryJobID, err)
|
||||
}
|
||||
if record.Scope != payload.Scope || record.TargetSnapshotID != payload.TargetSnapshotID {
|
||||
return failLibraryHandoff(ctx, store, payload, "队列载荷与业务作业不一致(scope/快照),拒绝搬运")
|
||||
}
|
||||
switch payload.NextType {
|
||||
case librarySyncJobType:
|
||||
if record.Kind != governance.LibraryJobSync {
|
||||
return failLibraryHandoff(ctx, store, payload, "作业类型是恢复,但搬运要求投递同步,拒绝")
|
||||
}
|
||||
case libraryRestoreJobType:
|
||||
if record.Kind != governance.LibraryJobRestore {
|
||||
return failLibraryHandoff(ctx, store, payload, "作业类型是同步,但搬运要求投递恢复,拒绝")
|
||||
}
|
||||
default:
|
||||
return failLibraryHandoff(ctx, store, payload, "未知的后续作业类型,拒绝搬运")
|
||||
}
|
||||
if record.State == governance.LibraryJobFailed || record.State == governance.LibraryJobCancelled ||
|
||||
record.State == governance.LibraryJobSucceeded {
|
||||
return failLibraryHandoff(ctx, store, payload, "业务作业已终态,不再搬运")
|
||||
}
|
||||
|
||||
knowledgeBase := config.LibraryProductionInternalKB
|
||||
if payload.Scope == "external" {
|
||||
knowledgeBase = config.LibraryProductionExternalKB
|
||||
}
|
||||
if config.LibraryProductionTenantID == 0 || knowledgeBase == "" {
|
||||
return failLibraryHandoff(ctx, store, payload, "目标身份未配置(租户/知识库),无法核对快照归属")
|
||||
}
|
||||
want := libraryhandoff.Identity{SnapshotID: payload.TargetSnapshotID, Scope: payload.Scope,
|
||||
TenantID: config.LibraryProductionTenantID, KnowledgeBaseID: knowledgeBase}
|
||||
|
||||
_ = store.UpdateLibraryStage(ctx, payload.LibraryJobID, libraryHandoffStage, "running", nil, nil, nil, nil,
|
||||
"正在把快照送到发布执行器并校验", now)
|
||||
evidence, err := handoff.Push(ctx, want)
|
||||
if err != nil {
|
||||
message := "快照未送到发布执行器:" + err.Error()
|
||||
_ = store.UpdateLibraryStage(ctx, payload.LibraryJobID, libraryHandoffStage, "failed", nil, nil, nil, nil, message, time.Now().UTC())
|
||||
// 目标没有被停用、没有任何知识写入:恢复责任就是"修好搬运再重试"。
|
||||
_ = store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobFailed, libraryHandoffStage,
|
||||
message, "未进入维护窗口、未写任何知识;修正搬运接线或重新封存后重试", "", "", time.Now().UTC())
|
||||
return fmt.Errorf("%s", message)
|
||||
}
|
||||
|
||||
// 只有搬运校验通过后,发布作业才存在。
|
||||
payloadRaw, marshalErr := json.Marshal(map[string]string{
|
||||
"library_job_id": payload.LibraryJobID, "scope": payload.Scope, "target_snapshot_id": payload.TargetSnapshotID,
|
||||
})
|
||||
if marshalErr != nil {
|
||||
return failLibraryHandoff(ctx, store, payload, "无法构造发布作业载荷")
|
||||
}
|
||||
queueJob, enqueueErr := store.Enqueue(ctx, jobs.EnqueueRequest{
|
||||
ID: governance.QueueJobIDFor(payload.LibraryJobID), Type: payload.NextType, Payload: payloadRaw,
|
||||
AvailableAt: time.Now().UTC(), MaxAttempts: 1,
|
||||
IdempotencyKey: payload.NextType + ".queue:" + payload.LibraryJobID,
|
||||
}, time.Now().UTC())
|
||||
if enqueueErr != nil || queueJob == nil {
|
||||
return failLibraryHandoff(ctx, store, payload, "快照已在执行器就绪,但发布作业投递失败")
|
||||
}
|
||||
if attachErr := store.AttachLibraryQueueJob(ctx, payload.LibraryJobID, queueJob.ID, time.Now().UTC()); attachErr != nil {
|
||||
return failLibraryHandoff(ctx, store, payload, "发布作业已投递但无法关联业务作业")
|
||||
}
|
||||
detail := fmt.Sprintf("快照已在校验后送达执行器(parts=%d bytes=%d reused=%t)", evidence.Parts, evidence.Bytes, evidence.Reused)
|
||||
_ = store.UpdateLibraryStage(ctx, payload.LibraryJobID, libraryHandoffStage, "succeeded", nil, nil, nil, nil, detail, time.Now().UTC())
|
||||
log.Printf("library handoff completed job=%s scope=%s snapshot=%s parts=%d reused=%t",
|
||||
payload.LibraryJobID, payload.Scope, payload.TargetSnapshotID, evidence.Parts, evidence.Reused)
|
||||
return nil
|
||||
}
|
||||
|
||||
func failLibraryHandoff(ctx context.Context, store *postgresstore.Store, payload libraryHandoffPayload, message string) error {
|
||||
_ = store.UpdateLibraryStage(ctx, payload.LibraryJobID, libraryHandoffStage, "failed", nil, nil, nil, nil, message, time.Now().UTC())
|
||||
_ = store.AdvanceLibraryJob(ctx, payload.LibraryJobID, governance.LibraryJobFailed, libraryHandoffStage, message,
|
||||
"未进入维护窗口、未写任何知识;修正后重试", "", "", time.Now().UTC())
|
||||
return fmt.Errorf("%s", message)
|
||||
}
|
||||
|
||||
// runLibraryArchivePullLoop 把云端产生的快照(预备份等)取回本机归档。
|
||||
//
|
||||
// 为什么要回传而不是"页面直接读云端":API 的所有归档消费点(列表/详情/差异/存储/
|
||||
// 容量/恢复寻址)都读本机 LIBRARY_ARCHIVE_ROOT,回传后它们读到的就是真实字节,
|
||||
// 不需要给每条读路径加远端分支,也不会把两端路径当成全局身份(身份始终是 snapshot ID)。
|
||||
//
|
||||
// 失败只记日志:本机没有这份版本时页面按"无此版本/不可读"展示,不会被误报成可恢复。
|
||||
func runLibraryArchivePullLoop(ctx context.Context, store *postgresstore.Store, handoff *libraryhandoff.Handoff,
|
||||
archiveRoot string, interval time.Duration, report func(error)) {
|
||||
if interval <= 0 {
|
||||
interval = 5 * time.Minute
|
||||
}
|
||||
pull := func() {
|
||||
archive, err := librarysnapshot.Open(archiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
report(fmt.Errorf("归档不可读,跳过回传:%w", err))
|
||||
return
|
||||
}
|
||||
defer archive.Close()
|
||||
for _, scope := range []string{"internal", "external"} {
|
||||
ids, listErr := store.LibraryJobSnapshots(ctx, scope, 50)
|
||||
if listErr != nil {
|
||||
report(fmt.Errorf("读不到 %s 的已完成作业快照:%w", scope, listErr))
|
||||
continue
|
||||
}
|
||||
for _, id := range ids {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return
|
||||
}
|
||||
if _, loadErr := archive.LoadManifest(id); loadErr == nil {
|
||||
continue
|
||||
}
|
||||
evidence, fetchErr := handoff.Fetch(ctx, id)
|
||||
if fetchErr != nil {
|
||||
report(fmt.Errorf("回传快照 %s 失败:%w", id, fetchErr))
|
||||
continue
|
||||
}
|
||||
log.Printf("library archive pull snapshot=%s scope=%s parts=%d bytes=%d reused=%t",
|
||||
evidence.Snapshot, evidence.Scope, evidence.Parts, evidence.Bytes, evidence.Reused)
|
||||
}
|
||||
}
|
||||
}
|
||||
pull()
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
pull()
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/appconfig"
|
||||
"knowledge-governance/internal/governance"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
)
|
||||
|
||||
// 这条测试回答一个业务问题:搬运没成功时,发布作业到底存不存在。
|
||||
// 用隔离 fixture 库(KG_TEST_DSN 指向专门的测试库),不碰任何业务库。
|
||||
func TestHandoffChainGatesReleaseJobOnVerifiedTransfer(t *testing.T) {
|
||||
dsn := os.Getenv("KG_TEST_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("KG_TEST_DSN 未设置:跳过隔离 fixture 库上的搬运链测试")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer pool.Close()
|
||||
if err := postgresstore.ApplyMigrations(ctx, pool); err != nil {
|
||||
t.Fatalf("migrate fixture db: %v", err)
|
||||
}
|
||||
store := postgresstore.NewStore(pool)
|
||||
// 隔离 fixture 库会被反复使用:先把上一次运行残留的在飞 fixture 作业收掉,
|
||||
// 否则 (scope,kind) 的唯一索引会把本次新建当成冲突。只动 handoff fixture 的行。
|
||||
if _, err := pool.Exec(ctx, `UPDATE library_jobs SET state='failed', finished_at=now(), updated_at=now()
|
||||
WHERE id LIKE 'libsync-handoff-%' AND state IN ('pending','running')`); err != nil {
|
||||
t.Fatalf("reset fixture jobs: %v", err)
|
||||
}
|
||||
fixture := newHandoffFixture(t)
|
||||
config := appconfig.Config{LibraryProductionTenantID: fixture.tenant,
|
||||
LibraryProductionInternalKB: fixture.kb, LibraryProductionExternalKB: fixture.kb}
|
||||
|
||||
snapshot := fixture.seal(t)
|
||||
job := createFixtureJob(t, store, "internal", snapshot)
|
||||
payload := libraryHandoffPayload{LibraryJobID: job.ID, Scope: "internal",
|
||||
TargetSnapshotID: snapshot, NextType: librarySyncJobType}
|
||||
if err := runLibraryHandoff(ctx, store, config, fixture.handoff, payload); err != nil {
|
||||
t.Fatalf("runLibraryHandoff() error = %v", err)
|
||||
}
|
||||
record, err := store.LibraryJob(ctx, job.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if record.State == governance.LibraryJobFailed {
|
||||
t.Fatalf("verified handoff left the job failed: %s", record.Error)
|
||||
}
|
||||
releaseID := governance.QueueJobIDFor(job.ID)
|
||||
var queueType string
|
||||
if err := pool.QueryRow(ctx, `SELECT type FROM jobs WHERE id=$1`, releaseID).Scan(&queueType); err != nil {
|
||||
t.Fatalf("release queue job missing after verified handoff: %v", err)
|
||||
}
|
||||
if queueType != librarySyncJobType {
|
||||
t.Fatalf("release queue type = %q", queueType)
|
||||
}
|
||||
if record.QueueJobID != releaseID {
|
||||
t.Fatalf("library job not attached to release queue job: %q", record.QueueJobID)
|
||||
}
|
||||
// 云端确实拿到了可校验的快照。
|
||||
if _, err := os.Stat(filepath.Join(fixture.remoteArchive, "versions", snapshot+".json")); err != nil {
|
||||
t.Fatalf("executor archive lacks the handoff snapshot: %v", err)
|
||||
}
|
||||
|
||||
// 负例:快照根本不存在 → 发布作业不得入队,作业必须失败。
|
||||
// 用 external scope:同一 (scope, kind) 只允许一个在飞作业,负例不能挤在 internal 上。
|
||||
unknown := fixture.unknownSnapshotID()
|
||||
badJob := createFixtureJob(t, store, "external", unknown)
|
||||
badPayload := libraryHandoffPayload{LibraryJobID: badJob.ID, Scope: "external",
|
||||
TargetSnapshotID: unknown, NextType: librarySyncJobType}
|
||||
if err := runLibraryHandoff(ctx, store, config, fixture.handoff, badPayload); err == nil {
|
||||
t.Fatal("runLibraryHandoff() reported success for a snapshot that does not exist")
|
||||
}
|
||||
badRecord, err := store.LibraryJob(ctx, badJob.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if badRecord.State != governance.LibraryJobFailed {
|
||||
t.Fatalf("failed handoff left job state = %s", badRecord.State)
|
||||
}
|
||||
var queued int
|
||||
if err := pool.QueryRow(ctx, `SELECT count(*) FROM jobs WHERE id=$1`,
|
||||
governance.QueueJobIDFor(badJob.ID)).Scan(&queued); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if queued != 0 {
|
||||
t.Fatal("a failed handoff still enqueued the release job: the target could be put into maintenance")
|
||||
}
|
||||
}
|
||||
|
||||
func createFixtureJob(t *testing.T, store *postgresstore.Store, scope, snapshot string) governance.LibraryJob {
|
||||
t.Helper()
|
||||
now := time.Now().UTC()
|
||||
// 每次运行用唯一后缀:fixture 库会被反复使用,固定 ID 会撞上上一次的行。
|
||||
unique := fmt.Sprintf("%d", now.UnixNano())
|
||||
job, reused, err := store.CreateLibraryJob(context.Background(), governance.LibraryJob{
|
||||
ID: "libsync-handoff-" + unique, Scope: scope, Kind: governance.LibraryJobSync,
|
||||
IdempotencyKey: "handoff-fixture:" + unique, RequestDigest: "fixture-" + unique,
|
||||
OperationID: "op-" + unique, TargetSnapshotID: snapshot, Actor: "fixture",
|
||||
}, governance.SyncStages(), now)
|
||||
if err != nil {
|
||||
t.Fatalf("create fixture job: %v", err)
|
||||
}
|
||||
if reused {
|
||||
t.Fatalf("fixture job unexpectedly reused")
|
||||
}
|
||||
return job
|
||||
}
|
||||
|
||||
type handoffFixture struct {
|
||||
root string
|
||||
remoteArchive string
|
||||
staging string
|
||||
archiveRoot string
|
||||
tenant uint64
|
||||
kb string
|
||||
handoff *libraryhandoff.Handoff
|
||||
}
|
||||
|
||||
func newHandoffFixture(t *testing.T) handoffFixture {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
remoteArchive := filepath.Join(root, "remote-archive")
|
||||
staging := filepath.Join(root, "remote-staging")
|
||||
bundles := filepath.Join(root, "bundles")
|
||||
localArchive := filepath.Join(root, "local-archive")
|
||||
for _, dir := range []string{remoteArchive, staging, bundles, localArchive} {
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
binary := os.Getenv("KG_TEST_LIBRARYCTL")
|
||||
if binary == "" {
|
||||
t.Skip("KG_TEST_LIBRARYCTL 未设置:搬运链测试需要真实 libraryctl 二进制")
|
||||
}
|
||||
t.Setenv("LIBRARY_HANDOFF_LIBRARYCTL", binary)
|
||||
handoff, err := libraryhandoff.New(libraryhandoff.Config{
|
||||
Transport: libraryhandoff.TransportLocal, LocalRoot: root, LocalArchive: localArchive,
|
||||
ExecutorContainer: "fixture-executor", BundleDir: bundles, RemoteArchive: remoteArchive,
|
||||
StagingDir: filepath.Join(root, "staging"), MaxBundleBytes: 8 << 20, OpTimeout: time.Minute,
|
||||
}, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return handoffFixture{root: root, remoteArchive: remoteArchive, staging: staging,
|
||||
archiveRoot: localArchive, tenant: 10000, kb: "kb-handoff-fixture", handoff: handoff}
|
||||
}
|
||||
|
||||
func (f handoffFixture) unknownSnapshotID() string {
|
||||
value := bytes.Repeat([]byte{9}, 32)
|
||||
return hex.EncodeToString(value)
|
||||
}
|
||||
|
||||
// seal 在本机归档里造一份合法的封存快照(身份与 fixture 的 tenant/kb 一致)。
|
||||
func (f handoffFixture) seal(t *testing.T) string {
|
||||
t.Helper()
|
||||
archive, err := librarysnapshot.Open(f.archiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
part, err := archive.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mapping := sha256.Sum256([]byte("fixture-mapping"))
|
||||
manifest := librarysnapshot.Manifest{
|
||||
Format: librarysnapshot.Format, Scope: "internal", TenantID: f.tenant,
|
||||
KnowledgeBaseID: f.kb, CreatedAt: time.Date(2026, 10, 30, 0, 0, 0, 0, time.UTC),
|
||||
Compatibility: librarysnapshot.Compatibility{
|
||||
WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8",
|
||||
EmbeddingModel: "fixture-model", EmbeddingDimension: 1024,
|
||||
IndexMappingSHA256: hex.EncodeToString(mapping[:]),
|
||||
},
|
||||
}
|
||||
for _, name := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} {
|
||||
current := part
|
||||
current.Kind, current.Name = "database", name
|
||||
manifest.Parts = append(manifest.Parts, current)
|
||||
}
|
||||
for _, name := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
|
||||
current := part
|
||||
current.Kind, current.Name = "wiki", name
|
||||
manifest.Parts = append(manifest.Parts, current)
|
||||
}
|
||||
index := part
|
||||
index.Kind, index.Name = "index", "chunks"
|
||||
manifest.Parts = append(manifest.Parts, index)
|
||||
id, err := archive.Seal(manifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// 保持 encoding/json 参与(payload 结构与 API 一致),避免测试与线上 struct 漂移。
|
||||
var _ = json.Marshal
|
||||
@@ -3,6 +3,7 @@ package main
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
@@ -17,6 +18,7 @@ import (
|
||||
"knowledge-governance/internal/governance"
|
||||
"knowledge-governance/internal/jobs"
|
||||
"knowledge-governance/internal/libraryagent"
|
||||
"knowledge-governance/internal/libraryhandoff"
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
@@ -183,6 +185,24 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
return err
|
||||
}
|
||||
if config.LibraryWorkerRole.ExecutesSourceWork() {
|
||||
// 快照搬运(本机 → 云端执行器)与云端版本回传都只有 capture 角色做:
|
||||
// 它持有源归档与 SSH 凭据;release-executor 在云端没有回本机的通道。
|
||||
if handoffConfig, configured, handoffErr := libraryhandoff.FromEnv(getenv); handoffErr != nil {
|
||||
return fmt.Errorf("快照搬运接线无效: %w", handoffErr)
|
||||
} else if configured {
|
||||
if config.LibraryWorkerRole.ExecutesRelease() {
|
||||
return errors.New("快照搬运只能接在 capture 角色上:full 角色同时消费发布作业,会与云端执行器抢同一批作业")
|
||||
}
|
||||
handoff, buildErr := libraryhandoff.New(handoffConfig, nil)
|
||||
if buildErr != nil {
|
||||
return buildErr
|
||||
}
|
||||
registerLibraryHandoffHandlers(handlers, store, config, handoff)
|
||||
go runLibraryArchivePullLoop(ctx, store, handoff, config.LibraryArchiveRoot, handoffConfig.PullInterval,
|
||||
func(err error) { log.Printf("library archive pull: %v", err) })
|
||||
log.Printf("library handoff enabled: executor=%s archive=%s pull_every=%s",
|
||||
handoffConfig.ExecutorContainer, handoffConfig.RemoteArchive, handoffConfig.PullInterval)
|
||||
}
|
||||
if syncErr := configureDirectorySync(pool, getenv, handlers); syncErr != nil {
|
||||
log.Print("QA directory synchronization unavailable; handler was not registered")
|
||||
}
|
||||
|
||||
@@ -173,4 +173,51 @@ docker compose -f compose.yaml up -d app frontend
|
||||
反向转发),所以阻塞不在"连不上",而在:① 该端点等于让云侧凭据读走整库快照,
|
||||
需要单独的鉴权与审计口径;② 版本证据要新增 `archive_host`,否则 UI 无法判断"该去哪台机器取";
|
||||
③ 拉取失败必须与"目标已进维护窗口"的时序绑定,不能先开窗口再发现取不到。
|
||||
这三条都要父拍板,属于新协议而非配置。
|
||||
这三条都要父拍板,属于新协议而非配置。
|
||||
## 8. 自动快照搬运(run03 已实现,需按本节接线)
|
||||
|
||||
没有这一步,UI 点"发布/恢复"只会在执行器上以"快照不在本机归档"失败(fail-closed,不会开维护窗口)。
|
||||
接线后链路变成:**授权 → 搬运并双向校验 → 才投递发布作业**。
|
||||
|
||||
### 8.1 接线(API 与 capture worker 都要有同一组值)
|
||||
|
||||
| 变量 | 含义 |
|
||||
|---|---|
|
||||
| `LIBRARY_HANDOFF_SSH_CONFIG` | 0600 ssh 配置(与 `BACKUP_HOST_SSH_CONFIG` 同一份即可,指向 `weknora-test`) |
|
||||
| `LIBRARY_HANDOFF_TARGET` | ssh 配置里的 Host 别名(`weknora-test`) |
|
||||
| `LIBRARY_HANDOFF_EXECUTOR` | 云端执行器容器名(`kgtest-release-executor`) |
|
||||
| `LIBRARY_HANDOFF_BUNDLE_DIR` | 云端暂存目录(`/srv/knowledge-governance/bundles`,只放传输中的 bundle) |
|
||||
| `LIBRARY_HANDOFF_REMOTE_ARCHIVE` | 云端归档根(必须等于执行器 env 的 `LIBRARY_ARCHIVE_ROOT`) |
|
||||
| `LIBRARY_HANDOFF_STAGING_DIR` | 本机暂存目录(容器内可见的 0700 目录) |
|
||||
| `LIBRARY_HANDOFF_MAX_BYTES` | 单份 bundle 上限(默认 8GiB) |
|
||||
| `LIBRARY_HANDOFF_TIMEOUT` | 单个远端动作超时(默认 10m) |
|
||||
| `LIBRARY_HANDOFF_PULL_INTERVAL` | 云端版本回传本机的扫描间隔(默认 5m) |
|
||||
|
||||
任一项缺失但其余存在 ⇒ **启动失败**(半接线最坏:API 以为会搬运、worker 搬不动)。
|
||||
全都不配 ⇒ 保持旧行为(本机目标直接发布),不受影响。
|
||||
|
||||
### 8.2 行为
|
||||
|
||||
- 发布/恢复作业先入 `library.snapshot.handoff`(capture 角色执行):本机 `Load` 逐 blob 校验 →
|
||||
`Pack` → SSH 流式写云端暂存 → 原子改名 → 云端 `libraryctl snapshot-unpack`(再逐 blob 校验)→
|
||||
读回版本清单摘要自证 → 身份(scope/tenant/KB)与治理库作业一致 → **才**投递 `library.sync|restore`
|
||||
(ID 仍是 `QueueJobIDFor(jobID)`,接管/续作路径不变)。
|
||||
- 失败即作业失败,文案写明"未进入维护窗口、未写任何知识";发布队列行不存在。
|
||||
- 云端已有同 ID 且校验通过 ⇒ 复用(不发数据);同 ID 异内容 ⇒ 拒绝。
|
||||
- 同一快照并发搬运:本机 flock 串行化,云端内容寻址 + Link 幂等。
|
||||
- 暂存 bundle 用后即删(只删自己这次的传输文件);归档本体(versions/blobs)**永不自动清理**。
|
||||
- 本机回传:capture 角色按 `library_jobs` 里已成功作业的 `snapshot_id` 扫描,本机缺哪个就
|
||||
`Fetch` 哪个(云端 pack → SSH 流回 → 本机 `Unpack` 逐 blob 校验)。回传成功后本机既有
|
||||
API/页面路径(列表/详情/差异/存储/历史恢复)读到的就是真实字节;回传失败则版本不出现在本机,
|
||||
页面不会把它显示成可恢复。
|
||||
|
||||
### 8.3 只读核对命令(父可用)
|
||||
|
||||
```bash
|
||||
# 本机:把某个已封存快照推到执行器(会打印 bundle 每个 blob 的 sha256,可对拍)
|
||||
LIBRARY_HANDOFF_...=... libraryctl snapshot-push -snapshot <id> -scope internal -tenant <n> -kb <uuid>
|
||||
# 本机:把云端快照取回本机归档
|
||||
libraryctl snapshot-fetch -snapshot <id>
|
||||
```
|
||||
|
||||
两条命令都只认部署注入的配置:不接受调用方指定远端路径/容器/上游地址。
|
||||
|
||||
@@ -58,4 +58,13 @@ LIBRARY_AGENT_SOCKET=
|
||||
# 这两个开关只是"本进程允许写"的部署声明;真正的授权是治理库里与快照绑定的
|
||||
# release 授权行。任何一道为 false,发布会在维护窗口之前被拒绝。
|
||||
PRODUCTION_RELEASE_REQUESTS_ENABLED=true
|
||||
PRODUCTION_BACKUP_RESTORE_VERIFIED=true
|
||||
PRODUCTION_BACKUP_RESTORE_VERIFIED=true
|
||||
# ---- 快照搬运(本机 API + capture worker 侧配置;执行器不消费这组变量)----
|
||||
# 只写在这里作对照:真正的 LIBRARY_HANDOFF_* 配在**本机** api/worker 容器,
|
||||
# 执行器只需要 LIBRARY_ARCHIVE_ROOT 与 BUNDLE_DIR 对应的宿主机目录存在。
|
||||
# LIBRARY_HANDOFF_SSH_CONFIG=/run/secrets/backup-host/ssh_config
|
||||
# LIBRARY_HANDOFF_TARGET=weknora-test
|
||||
# LIBRARY_HANDOFF_EXECUTOR=kgtest-release-executor
|
||||
# LIBRARY_HANDOFF_BUNDLE_DIR=/srv/knowledge-governance/bundles
|
||||
# LIBRARY_HANDOFF_REMOTE_ARCHIVE=/var/lib/knowledge-governance/library
|
||||
# LIBRARY_HANDOFF_STAGING_DIR=/var/lib/knowledge-governance/handoff-staging
|
||||
|
||||
@@ -45,6 +45,10 @@ type libraryVersionView struct {
|
||||
// librarySyncJobType 与 worker 的处理器类型一致;单独定义避免 API 依赖 worker 包。
|
||||
const librarySyncJobType = "library.sync"
|
||||
|
||||
// libraryHandoffJobType 是"先把快照送到执行器"的队列类型(capture 角色消费)。
|
||||
// 与发布作业分开:搬运失败时发布作业根本不会入队,目标不会进入维护窗口。
|
||||
const libraryHandoffJobType = "library.snapshot.handoff"
|
||||
|
||||
// libraryRecoverJobType 是"按预备份恢复并校验"的队列类型(同一份定义也在 worker 里)。
|
||||
const libraryRecoverJobType = "library.recover"
|
||||
|
||||
@@ -978,16 +982,15 @@ func librarySyncEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
return
|
||||
}
|
||||
// 业务作业先落库,再投递队列:投递失败就把作业标失败,不留下"看起来在跑"的空作业。
|
||||
payload, err := json.Marshal(map[string]string{
|
||||
"library_job_id": job.ID, "scope": request.Scope, "target_snapshot_id": request.Target,
|
||||
})
|
||||
queueID, queueType, payload, err := libraryQueueRequest(config.LibraryHandoffEnabled, job.ID,
|
||||
request.Scope, request.Target, librarySyncJobType)
|
||||
if err != nil {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot build sync payload"})
|
||||
return
|
||||
}
|
||||
queueJob, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{
|
||||
ID: governance.QueueJobIDFor(job.ID), Type: librarySyncJobType, Payload: payload, AvailableAt: now,
|
||||
MaxAttempts: 1, IdempotencyKey: "library.sync.queue:" + job.ID,
|
||||
ID: queueID, Type: queueType, Payload: payload, AvailableAt: now,
|
||||
MaxAttempts: 1, IdempotencyKey: queueType + ".queue:" + job.ID,
|
||||
}, now)
|
||||
if err != nil || queueJob == nil {
|
||||
log.Printf("library sync enqueue failed job=%s: %v", job.ID, err)
|
||||
@@ -1001,3 +1004,24 @@ func librarySyncEnqueue(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": false})
|
||||
}
|
||||
|
||||
// libraryQueueRequest 决定"这份作业先入哪个队列"。
|
||||
//
|
||||
// 搬运接线时先入 library.snapshot.handoff(capture 角色把快照送到执行器并校验),
|
||||
// 由它在成功后再入 releaseType;搬运未接线时保持原样。两种情况下 releaseType
|
||||
// 作业的 ID 都是 QueueJobIDFor(jobID),所以接管/续作的既有路径不受影响。
|
||||
func libraryQueueRequest(handoff bool, jobID, scope, snapshotID, releaseType string) (string, string, []byte, error) {
|
||||
payload := map[string]string{"library_job_id": jobID, "scope": scope, "target_snapshot_id": snapshotID}
|
||||
queueID := governance.QueueJobIDFor(jobID)
|
||||
queueType := releaseType
|
||||
if handoff {
|
||||
payload["next_type"] = releaseType
|
||||
queueID = governance.QueueJobIDFor(jobID) + ":handoff"
|
||||
queueType = libraryHandoffJobType
|
||||
}
|
||||
raw, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
return queueID, queueType, raw, nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"knowledge-governance/internal/governance"
|
||||
)
|
||||
|
||||
// 搬运接线时,发布作业必须"先搬运":队列类型换成 handoff,释放作业的 ID 仍是基础 ID。
|
||||
func TestLibraryQueueRequestGatesReleaseOnHandoff(t *testing.T) {
|
||||
jobID := "libsync-internal-1"
|
||||
snapshot := "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
|
||||
|
||||
id, queueType, payload, err := libraryQueueRequest(true, jobID, "internal", snapshot, librarySyncJobType)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if queueType != libraryHandoffJobType {
|
||||
t.Fatalf("queue type = %q, want %q", queueType, libraryHandoffJobType)
|
||||
}
|
||||
if id != governance.QueueJobIDFor(jobID)+":handoff" {
|
||||
t.Fatalf("handoff queue id = %q", id)
|
||||
}
|
||||
var decoded map[string]string
|
||||
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if decoded["next_type"] != librarySyncJobType || decoded["target_snapshot_id"] != snapshot {
|
||||
t.Fatalf("handoff payload = %#v", decoded)
|
||||
}
|
||||
|
||||
// 未接线:行为与今天完全一致(直接投递发布作业,ID 是基础 ID)。
|
||||
id, queueType, payload, err = libraryQueueRequest(false, jobID, "internal", snapshot, libraryRestoreJobType)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if queueType != libraryRestoreJobType || id != governance.QueueJobIDFor(jobID) {
|
||||
t.Fatalf("unconfigured queue request = %q %q", queueType, id)
|
||||
}
|
||||
decoded = map[string]string{}
|
||||
if err := json.Unmarshal(payload, &decoded); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, present := decoded["next_type"]; present {
|
||||
t.Fatal("release payload must not carry next_type when handoff is off")
|
||||
}
|
||||
}
|
||||
@@ -138,16 +138,15 @@ func libraryRestoreEnqueue(w http.ResponseWriter, r *http.Request, config Config
|
||||
writeJSON(w, http.StatusAccepted, map[string]any{"job": job, "reused": true})
|
||||
return
|
||||
}
|
||||
payload, err := json.Marshal(map[string]string{
|
||||
"library_job_id": job.ID, "scope": request.Scope, "target_snapshot_id": request.SnapshotID,
|
||||
})
|
||||
queueID, queueType, payload, err := libraryQueueRequest(config.LibraryHandoffEnabled, job.ID,
|
||||
request.Scope, request.SnapshotID, libraryRestoreJobType)
|
||||
if err != nil {
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "cannot build restore payload"})
|
||||
return
|
||||
}
|
||||
queueJob, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{
|
||||
ID: governance.QueueJobIDFor(job.ID), Type: libraryRestoreJobType, Payload: payload, AvailableAt: now,
|
||||
MaxAttempts: 1, IdempotencyKey: "library.restore.queue:" + job.ID,
|
||||
ID: queueID, Type: queueType, Payload: payload, AvailableAt: now,
|
||||
MaxAttempts: 1, IdempotencyKey: queueType + ".queue:" + job.ID,
|
||||
}, now)
|
||||
if err != nil || queueJob == nil {
|
||||
_ = store.CancelLibraryJob(r.Context(), job.ID, "队列投递失败,未开始恢复", now)
|
||||
|
||||
@@ -145,6 +145,10 @@ type Config struct {
|
||||
VersionHistoryRepo string
|
||||
// LibraryArchiveRoot 是整库封存产物的只读归档根;留空即未接线(页面降级)。
|
||||
LibraryArchiveRoot string
|
||||
// LibraryHandoffEnabled:快照搬运已接线时,发布/恢复作业先投递搬运作业,
|
||||
// 由 capture 角色把不可变快照送到云端执行器并校验通过后才投递真正的发布作业。
|
||||
// 未接线即保持今天的行为(本机目标直接发布)。
|
||||
LibraryHandoffEnabled bool
|
||||
// LibraryTargetInstance 是实例级维护的目标实例名;留空即"未接线",
|
||||
// 页面不能发起按预备份恢复,接口返回原因而不是假装能恢复。
|
||||
LibraryTargetInstance string
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package libraryhandoff
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 环境变量名集中在这里:API(决定要不要先做搬运)与 capture worker(真正执行搬运)
|
||||
// 必须看同一组名字,写两遍迟早分叉成"一边以为接线了、另一边没接线"。
|
||||
const (
|
||||
EnvSSHConfig = "LIBRARY_HANDOFF_SSH_CONFIG"
|
||||
EnvTarget = "LIBRARY_HANDOFF_TARGET"
|
||||
EnvExecutor = "LIBRARY_HANDOFF_EXECUTOR"
|
||||
EnvBundleDir = "LIBRARY_HANDOFF_BUNDLE_DIR"
|
||||
EnvRemoteArchive = "LIBRARY_HANDOFF_REMOTE_ARCHIVE"
|
||||
EnvStagingDir = "LIBRARY_HANDOFF_STAGING_DIR"
|
||||
EnvLocalArchive = "LIBRARY_HANDOFF_LOCAL_ARCHIVE"
|
||||
EnvMaxBytes = "LIBRARY_HANDOFF_MAX_BYTES"
|
||||
EnvTimeout = "LIBRARY_HANDOFF_TIMEOUT"
|
||||
EnvTransport = "LIBRARY_HANDOFF_TRANSPORT"
|
||||
EnvLocalRoot = "LIBRARY_HANDOFF_LOCAL_ROOT"
|
||||
// EnvPullInterval 控制"云端版本回传本机"的扫描间隔(capture 角色)。
|
||||
EnvPullInterval = "LIBRARY_HANDOFF_PULL_INTERVAL"
|
||||
)
|
||||
|
||||
// FromEnv 解析搬运配置。
|
||||
//
|
||||
// 返回 configured=false 只表示"这项能力没接线",不是错误:未接线时 API 保持旧行为
|
||||
// (直接投递 library.sync/restore),本机目标部署完全不受影响。
|
||||
//
|
||||
// 但一旦任何一项接线字段出现,就必须**全部齐备**,否则报错——半接线是最坏的情况:
|
||||
// API 以为会搬运,worker 却搬不动,发布作业会以"快照不在执行器"失败。
|
||||
func FromEnv(get func(string) string) (Config, bool, error) {
|
||||
if get == nil {
|
||||
return Config{}, false, errors.New("缺少配置读取入口")
|
||||
}
|
||||
value := func(key string) string { return strings.TrimSpace(get(key)) }
|
||||
config := Config{
|
||||
SSHConfig: value(EnvSSHConfig),
|
||||
SSHTarget: value(EnvTarget),
|
||||
ExecutorContainer: value(EnvExecutor),
|
||||
BundleDir: value(EnvBundleDir),
|
||||
RemoteArchive: value(EnvRemoteArchive),
|
||||
StagingDir: value(EnvStagingDir),
|
||||
LocalArchive: value(EnvLocalArchive),
|
||||
Transport: value(EnvTransport),
|
||||
LocalRoot: value(EnvLocalRoot),
|
||||
OpTimeout: 10 * time.Minute,
|
||||
PullInterval: 5 * time.Minute,
|
||||
}
|
||||
if config.Transport == "" {
|
||||
config.Transport = TransportSSH
|
||||
}
|
||||
if raw := value(EnvMaxBytes); raw != "" {
|
||||
parsed, err := strconv.ParseInt(raw, 10, 64)
|
||||
if err != nil || parsed <= 0 {
|
||||
return Config{}, false, errors.New("LIBRARY_HANDOFF_MAX_BYTES 必须是正整数")
|
||||
}
|
||||
config.MaxBundleBytes = parsed
|
||||
}
|
||||
if raw := value(EnvPullInterval); raw != "" {
|
||||
parsed, err := time.ParseDuration(raw)
|
||||
if err != nil || parsed <= 0 {
|
||||
return Config{}, false, errors.New("LIBRARY_HANDOFF_PULL_INTERVAL 必须是正的时间长度(如 5m)")
|
||||
}
|
||||
config.PullInterval = parsed
|
||||
}
|
||||
if raw := value(EnvTimeout); raw != "" {
|
||||
parsed, err := time.ParseDuration(raw)
|
||||
if err != nil || parsed <= 0 {
|
||||
return Config{}, false, errors.New("LIBRARY_HANDOFF_TIMEOUT 必须是正的时间长度(如 10m)")
|
||||
}
|
||||
config.OpTimeout = parsed
|
||||
}
|
||||
configuredFields := 0
|
||||
for _, item := range []string{EnvSSHConfig, EnvTarget, EnvExecutor, EnvBundleDir, EnvRemoteArchive, EnvStagingDir} {
|
||||
if value(item) != "" {
|
||||
configuredFields++
|
||||
}
|
||||
}
|
||||
if configuredFields == 0 && value(EnvTransport) != TransportLocal {
|
||||
return Config{}, false, nil
|
||||
}
|
||||
if config.Transport == TransportLocal {
|
||||
if value(EnvLocalRoot) == "" {
|
||||
return Config{}, false, errors.New("LIBRARY_HANDOFF_TRANSPORT=local 必须同时给出 LIBRARY_HANDOFF_LOCAL_ROOT(仅限隔离 fixture)")
|
||||
}
|
||||
if value(EnvSSHConfig) != "" || value(EnvTarget) != "" {
|
||||
return Config{}, false, errors.New("local 通道不允许同时配置 SSH 目标:fixture 不能冒充云端通道")
|
||||
}
|
||||
if config.LocalArchive == "" {
|
||||
config.LocalArchive = value("LIBRARY_ARCHIVE_ROOT")
|
||||
}
|
||||
if config.ExecutorContainer == "" {
|
||||
// fixture 不跑容器:给一个固定名字只为让配置校验完整,不会被执行。
|
||||
config.ExecutorContainer = "fixture-executor"
|
||||
}
|
||||
}
|
||||
if err := config.validate(); err != nil {
|
||||
return Config{}, false, err
|
||||
}
|
||||
return config, true, nil
|
||||
}
|
||||
@@ -0,0 +1,632 @@
|
||||
// Package libraryhandoff 把一份已封存快照在"本机控制面"与"云测试发布执行器"之间搬运,
|
||||
// 并保证"字节在到达前不可用"。
|
||||
//
|
||||
// 为什么不是"把归档目录 rsync 过去":发布链路在装配阶段会用 Load 逐 blob 重算 sha256,
|
||||
// 半拷贝的归档会让发布在维护窗口里失败——正好落在最不该失败的时刻。所以搬运单位是
|
||||
// 自描述 bundle(清单先写、blob 以 sha256 命名、下载侧边收边算),并且**只有**
|
||||
// 远端 `libraryctl snapshot-unpack` 成功、版本清单摘要自证、身份与请求一致之后,
|
||||
// 调用方才允许把发布作业入队。
|
||||
//
|
||||
// 通道只用已批准的本机→云端 SSH(BatchMode、0600 配置+私钥)。远端命令是固定 argv:
|
||||
// 容器名、归档根、暂存目录全部来自部署配置,调用方只能给快照 ID 与期望身份。
|
||||
package libraryhandoff
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// Identity 是一次搬运期望的快照身份。它必须来自治理库里的作业/版本记录,
|
||||
// 不能从 bundle 自己读到的值"回填"——那就是让数据自己证明自己。
|
||||
type Identity struct {
|
||||
SnapshotID string
|
||||
Scope string
|
||||
TenantID uint64
|
||||
KnowledgeBaseID string
|
||||
}
|
||||
|
||||
// Config 是部署注入的搬运配置。缺项即未接线:未接线时调用方保持旧行为(本机目标)。
|
||||
type Config struct {
|
||||
// SSHTarget 是 ssh 配置里的 Host 别名;SSHConfig 是 `ssh -F` 用的 0600 文件。
|
||||
SSHTarget string
|
||||
SSHConfig string
|
||||
// ExecutorContainer 是云端 release-executor 容器名(libraryctl 在 /app/libraryctl)。
|
||||
ExecutorContainer string
|
||||
// BundleDir 是云端暂存目录(只放传输中的 bundle,不参与保留策略)。
|
||||
BundleDir string
|
||||
// RemoteArchive 是云端执行器的归档根(与执行器 env 的 LIBRARY_ARCHIVE_ROOT 一致)。
|
||||
RemoteArchive string
|
||||
// LocalArchive 是本机归档根(只读读取;回传时写入)。
|
||||
LocalArchive string
|
||||
// StagingDir 是本机暂存(0600,放本次 pack 出来的 bundle)。
|
||||
StagingDir string
|
||||
// MaxBundleBytes 是单份 bundle 的上限;超过即拒绝(防止把整机磁盘拖垮)。
|
||||
MaxBundleBytes int64
|
||||
// OpTimeout 是单个远端动作的超时。
|
||||
OpTimeout time.Duration
|
||||
// PullInterval 是"云端版本回传本机"的扫描间隔(只有 capture 角色用)。
|
||||
PullInterval time.Duration
|
||||
// Transport 是 "ssh"(默认)或 "local"(仅限隔离 fixture 测试)。
|
||||
Transport string
|
||||
// LocalRoot 是 local transport 的"云端根目录"(fixture 用)。
|
||||
LocalRoot string
|
||||
}
|
||||
|
||||
const (
|
||||
TransportSSH = "ssh"
|
||||
TransportLocal = "local"
|
||||
)
|
||||
|
||||
func (c Config) validate() error {
|
||||
missing := make([]string, 0, 6)
|
||||
if c.Transport == TransportLocal {
|
||||
// fixture 通道没有 ssh,但必须有显式隔离根:否则"本地目录"会被当成云端。
|
||||
if c.LocalRoot == "" || !filepath.IsAbs(c.LocalRoot) {
|
||||
missing = append(missing, "LOCAL_ROOT")
|
||||
}
|
||||
} else {
|
||||
if c.SSHTarget == "" {
|
||||
missing = append(missing, "SSH_TARGET")
|
||||
}
|
||||
if c.SSHConfig == "" || !filepath.IsAbs(c.SSHConfig) {
|
||||
missing = append(missing, "SSH_CONFIG")
|
||||
}
|
||||
}
|
||||
if c.ExecutorContainer == "" {
|
||||
missing = append(missing, "EXECUTOR_CONTAINER")
|
||||
}
|
||||
for name, value := range map[string]string{"BUNDLE_DIR": c.BundleDir, "REMOTE_ARCHIVE": c.RemoteArchive, "LOCAL_ARCHIVE": c.LocalArchive, "STAGING_DIR": c.StagingDir} {
|
||||
if value == "" || !filepath.IsAbs(value) || filepath.Clean(value) != value || strings.Contains(value, "..") {
|
||||
missing = append(missing, name)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
sort.Strings(missing)
|
||||
return fmt.Errorf("快照搬运未接线:缺少 %s", strings.Join(missing, "、"))
|
||||
}
|
||||
if c.MaxBundleBytes <= 0 {
|
||||
c.MaxBundleBytes = 8 << 30
|
||||
}
|
||||
if c.OpTimeout <= 0 {
|
||||
c.OpTimeout = 10 * time.Minute
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Evidence 是一次搬运留下的可审计事实(写进作业阶段/日志,不写凭据)。
|
||||
type Evidence struct {
|
||||
Snapshot string `json:"snapshot"`
|
||||
Scope string `json:"scope"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
KnowledgeBase string `json:"knowledge_base_id"`
|
||||
Parts int `json:"parts"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Reused bool `json:"reused"`
|
||||
Direction string `json:"direction"`
|
||||
VerifiedAt time.Time `json:"verified_at"`
|
||||
}
|
||||
|
||||
// Remote 是"云端那一侧"的动作集合。两种实现:ssh(真实通道)与 local(隔离 fixture)。
|
||||
type Remote interface {
|
||||
PutBundle(ctx context.Context, name string, body io.Reader) error
|
||||
OpenBundle(ctx context.Context, name string) (io.ReadCloser, error)
|
||||
RemoveBundle(ctx context.Context, name string) error
|
||||
RunLibraryctl(ctx context.Context, args ...string) ([]byte, error)
|
||||
ReadVersionManifest(ctx context.Context, id string) ([]byte, error)
|
||||
}
|
||||
|
||||
// Handoff 是搬运的执行者(一套实现,两个方向)。
|
||||
type Handoff struct {
|
||||
config Config
|
||||
remote Remote
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
var snapshotIDPattern = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||
|
||||
// bundleName 只接受内容寻址的快照 ID:暂存目录里不允许出现调用方可控的文件名。
|
||||
func bundleName(id string) (string, error) {
|
||||
if !snapshotIDPattern.MatchString(id) {
|
||||
return "", errors.New("快照 ID 必须是 64 位小写十六进制")
|
||||
}
|
||||
return id + ".tar", nil
|
||||
}
|
||||
|
||||
// New 装配搬运执行器。transport 选择失败一律报错,不做"降级到本机"。
|
||||
func New(config Config, now func() time.Time) (*Handoff, error) {
|
||||
if err := config.validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
if config.Transport == "" {
|
||||
config.Transport = TransportSSH
|
||||
}
|
||||
var remote Remote
|
||||
switch config.Transport {
|
||||
case TransportSSH:
|
||||
remote = &sshRemote{config: config}
|
||||
case TransportLocal:
|
||||
// 只看配置不足以证明这是测试:必须显式给出隔离根,且不得同时配 SSH 目标,
|
||||
// 否则等于把"本地目录"冒充成云端。
|
||||
if config.LocalRoot == "" || !filepath.IsAbs(config.LocalRoot) {
|
||||
return nil, errors.New("local transport 必须显式给出隔离根目录")
|
||||
}
|
||||
remote = &localRemote{config: config}
|
||||
default:
|
||||
return nil, fmt.Errorf("未知搬运通道 %q(只允许 ssh/local)", config.Transport)
|
||||
}
|
||||
return &Handoff{config: config, remote: remote, now: now}, nil
|
||||
}
|
||||
|
||||
// Push 把本机快照送到云端执行器,并在云端**再校验一遍**后才算成功。
|
||||
//
|
||||
// 返回成功即代表:云端 `versions/<id>.json` 存在、摘要 == id、逐 blob 校验通过、
|
||||
// 清单身份与请求一致。任何一步不满足都返回错误,调用方不得入队发布作业。
|
||||
func (h *Handoff) Push(ctx context.Context, want Identity) (Evidence, error) {
|
||||
if err := validateIdentity(want); err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
name, err := bundleName(want.SnapshotID)
|
||||
if err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
// 远端已有同 ID:必须字节一致且身份一致才复用;否则拒绝(同 ID 异内容)。
|
||||
if evidence, ok, err := h.remoteEvidence(ctx, want); err != nil {
|
||||
return Evidence{}, err
|
||||
} else if ok {
|
||||
evidence.Direction = "push"
|
||||
evidence.Bytes = 0
|
||||
return evidence, nil
|
||||
}
|
||||
unlock, err := h.lock(ctx, want.SnapshotID)
|
||||
if err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
defer unlock()
|
||||
// 拿锁后再查一次:并发调用里后到的那个直接复用先到的结果。
|
||||
if evidence, ok, err := h.remoteEvidence(ctx, want); err != nil {
|
||||
return Evidence{}, err
|
||||
} else if ok {
|
||||
evidence.Direction = "push"
|
||||
return evidence, nil
|
||||
}
|
||||
|
||||
archive, err := librarysnapshot.Open(h.config.LocalArchive, 256*1024*1024)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("本机归档不可读,无法搬运:%w", err)
|
||||
}
|
||||
defer archive.Close()
|
||||
// Load 逐 blob 重算 sha256:只搬"在这台机器上验证过"的字节。
|
||||
manifest, err := archive.Load(want.SnapshotID)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("本机快照不可用(可能只存在于云端,走 Fetch):%w", err)
|
||||
}
|
||||
if err := matchesIdentity(manifest, want); err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
|
||||
bundle, cleanup, err := h.packLocally(archive, want.SnapshotID, name)
|
||||
if err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
defer cleanup()
|
||||
file, err := os.Open(bundle)
|
||||
if err != nil {
|
||||
return Evidence{}, errors.New("打开本机 bundle 失败")
|
||||
}
|
||||
defer file.Close()
|
||||
if info, statErr := file.Stat(); statErr != nil || info.Size() <= 0 || info.Size() > h.config.MaxBundleBytes {
|
||||
return Evidence{}, errors.New("bundle 大小超出允许范围")
|
||||
}
|
||||
if err := h.remote.PutBundle(ctx, name, file); err != nil {
|
||||
return Evidence{}, fmt.Errorf("把快照送到执行器失败:%w", err)
|
||||
}
|
||||
if _, err := h.remote.RunLibraryctl(ctx, "snapshot-unpack", "-archive", h.config.RemoteArchive,
|
||||
"-bundle", filepath.Join(h.config.BundleDir, name), "-snapshot", want.SnapshotID); err != nil {
|
||||
return Evidence{}, fmt.Errorf("执行器侧校验/装入失败:%w", err)
|
||||
}
|
||||
manifestRaw, err := h.remote.ReadVersionManifest(ctx, want.SnapshotID)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("执行器侧版本清单不可读:%w", err)
|
||||
}
|
||||
sum := sha256.Sum256(manifestRaw)
|
||||
if hex.EncodeToString(sum[:]) != want.SnapshotID {
|
||||
return Evidence{}, errors.New("执行器侧版本清单摘要与快照 ID 不一致")
|
||||
}
|
||||
remoteManifest, err := librarysnapshot.DecodeManifest(manifestRaw, want.SnapshotID)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("执行器侧清单不可解析:%w", err)
|
||||
}
|
||||
if err := matchesIdentity(remoteManifest, want); err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
// 只删本次传输的暂存文件;归档本体(versions/blobs)一律不动。
|
||||
_ = h.remote.RemoveBundle(ctx, name)
|
||||
return Evidence{Snapshot: want.SnapshotID, Scope: want.Scope, TenantID: want.TenantID,
|
||||
KnowledgeBase: want.KnowledgeBaseID, Parts: len(remoteManifest.Parts),
|
||||
Bytes: manifestBytes(remoteManifest), Direction: "push", VerifiedAt: h.now().UTC()}, nil
|
||||
}
|
||||
|
||||
// Fetch 把云端存在的快照取回本机归档(预备份/云端产物的读回路径)。
|
||||
//
|
||||
// 成功即代表:本机 versions/<id>.json 已落位、逐 blob 校验通过。失败则本机不留下版本,
|
||||
// 页面照旧按"没有这份版本"展示,绝不显示成可恢复。
|
||||
func (h *Handoff) Fetch(ctx context.Context, id string) (Evidence, error) {
|
||||
name, err := bundleName(id)
|
||||
if err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
archive, err := librarysnapshot.Open(h.config.LocalArchive, 256*1024*1024)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("本机归档不可用:%w", err)
|
||||
}
|
||||
defer archive.Close()
|
||||
if manifest, loadErr := archive.Load(id); loadErr == nil {
|
||||
return Evidence{Snapshot: id, Scope: manifest.Scope, TenantID: manifest.TenantID,
|
||||
KnowledgeBase: manifest.KnowledgeBaseID, Parts: len(manifest.Parts),
|
||||
Bytes: manifestBytes(manifest), Reused: true, Direction: "fetch", VerifiedAt: h.now().UTC()}, nil
|
||||
}
|
||||
unlock, err := h.lock(ctx, id)
|
||||
if err != nil {
|
||||
return Evidence{}, err
|
||||
}
|
||||
defer unlock()
|
||||
if manifest, loadErr := archive.Load(id); loadErr == nil {
|
||||
return Evidence{Snapshot: id, Scope: manifest.Scope, TenantID: manifest.TenantID,
|
||||
KnowledgeBase: manifest.KnowledgeBaseID, Parts: len(manifest.Parts),
|
||||
Bytes: manifestBytes(manifest), Reused: true, Direction: "fetch", VerifiedAt: h.now().UTC()}, nil
|
||||
}
|
||||
|
||||
remoteBundle := filepath.Join(h.config.BundleDir, name)
|
||||
defer func() { _ = h.remote.RemoveBundle(context.WithoutCancel(ctx), name) }()
|
||||
if _, err := h.remote.RunLibraryctl(ctx, "snapshot-pack", "-archive", h.config.RemoteArchive,
|
||||
"-snapshot", id, "-bundle", remoteBundle); err != nil {
|
||||
return Evidence{}, fmt.Errorf("云端快照不可打包(可能不存在或校验失败):%w", err)
|
||||
}
|
||||
body, err := h.remote.OpenBundle(ctx, name)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("读取云端 bundle 失败:%w", err)
|
||||
}
|
||||
defer body.Close()
|
||||
localBundle := filepath.Join(h.config.StagingDir, name)
|
||||
_ = os.Remove(localBundle)
|
||||
out, err := os.OpenFile(localBundle, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return Evidence{}, errors.New("无法创建本机暂存 bundle")
|
||||
}
|
||||
written, copyErr := io.Copy(out, io.LimitReader(body, h.config.MaxBundleBytes+1))
|
||||
syncErr := out.Sync()
|
||||
closeErr := out.Close()
|
||||
defer os.Remove(localBundle)
|
||||
if copyErr != nil || syncErr != nil || closeErr != nil {
|
||||
return Evidence{}, errors.New("回传快照字节失败(连接中断或被截断)")
|
||||
}
|
||||
if written > h.config.MaxBundleBytes {
|
||||
return Evidence{}, errors.New("回传 bundle 超过上限,拒绝导入")
|
||||
}
|
||||
manifest, err := archive.Unpack(localBundle, id)
|
||||
if err != nil {
|
||||
return Evidence{}, fmt.Errorf("回传快照校验失败(截断/篡改/缺部件):%w", err)
|
||||
}
|
||||
return Evidence{Snapshot: id, Scope: manifest.Scope, TenantID: manifest.TenantID,
|
||||
KnowledgeBase: manifest.KnowledgeBaseID, Parts: len(manifest.Parts),
|
||||
Bytes: manifestBytes(manifest), Direction: "fetch", VerifiedAt: h.now().UTC()}, nil
|
||||
}
|
||||
|
||||
// remoteEvidence 判断"云端已经有这份快照且身份一致"。
|
||||
func (h *Handoff) remoteEvidence(ctx context.Context, want Identity) (Evidence, bool, error) {
|
||||
raw, err := h.remote.ReadVersionManifest(ctx, want.SnapshotID)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) || strings.Contains(strings.ToLower(err.Error()), "no such file") {
|
||||
return Evidence{}, false, nil
|
||||
}
|
||||
return Evidence{}, false, fmt.Errorf("读取执行器侧版本清单失败:%w", err)
|
||||
}
|
||||
sum := sha256.Sum256(raw)
|
||||
if hex.EncodeToString(sum[:]) != want.SnapshotID {
|
||||
return Evidence{}, false, errors.New("执行器侧已有同名版本但清单摘要不符:拒绝复用")
|
||||
}
|
||||
manifest, err := librarysnapshot.DecodeManifest(raw, want.SnapshotID)
|
||||
if err != nil {
|
||||
return Evidence{}, false, err
|
||||
}
|
||||
if err := matchesIdentity(manifest, want); err != nil {
|
||||
return Evidence{}, false, err
|
||||
}
|
||||
return Evidence{Snapshot: want.SnapshotID, Scope: manifest.Scope, TenantID: manifest.TenantID,
|
||||
KnowledgeBase: manifest.KnowledgeBaseID, Parts: len(manifest.Parts),
|
||||
Bytes: manifestBytes(manifest), Reused: true, VerifiedAt: h.now().UTC()}, true, nil
|
||||
}
|
||||
|
||||
func (h *Handoff) packLocally(archive *librarysnapshot.Archive, id, name string) (string, func(), error) {
|
||||
if err := os.MkdirAll(h.config.StagingDir, 0700); err != nil {
|
||||
return "", func() {}, errors.New("本机暂存目录不可用")
|
||||
}
|
||||
path := filepath.Join(h.config.StagingDir, name)
|
||||
_ = os.Remove(path)
|
||||
if _, err := archive.Pack(id, path); err != nil {
|
||||
return "", func() {}, fmt.Errorf("打包快照失败:%w", err)
|
||||
}
|
||||
return path, func() { _ = os.Remove(path) }, nil
|
||||
}
|
||||
|
||||
// lock 用 flock 串行化"同一份快照"的本机搬运(并发点击/重试不会同时写暂存)。
|
||||
func (h *Handoff) lock(ctx context.Context, id string) (func(), error) {
|
||||
dir := filepath.Join(h.config.LocalArchive, ".handoff")
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
return nil, errors.New("无法创建搬运锁目录")
|
||||
}
|
||||
file, err := os.OpenFile(filepath.Join(dir, id+".lock"), os.O_CREATE|os.O_RDWR, 0600)
|
||||
if err != nil {
|
||||
return nil, errors.New("无法创建搬运锁")
|
||||
}
|
||||
deadline := time.Now().Add(h.config.OpTimeout)
|
||||
for {
|
||||
if err := syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err == nil {
|
||||
return func() {
|
||||
_ = syscall.Flock(int(file.Fd()), syscall.LOCK_UN)
|
||||
_ = file.Close()
|
||||
}, nil
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
file.Close()
|
||||
return nil, errors.New("另一份搬运正在处理同一个快照,等待超时")
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
file.Close()
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func validateIdentity(want Identity) error {
|
||||
if !snapshotIDPattern.MatchString(want.SnapshotID) {
|
||||
return errors.New("快照 ID 必须是 64 位小写十六进制")
|
||||
}
|
||||
if want.Scope != "internal" && want.Scope != "external" {
|
||||
return errors.New("快照 scope 必须是 internal 或 external")
|
||||
}
|
||||
if want.TenantID == 0 {
|
||||
return errors.New("缺少租户身份,无法核对快照归属")
|
||||
}
|
||||
if strings.TrimSpace(want.KnowledgeBaseID) == "" {
|
||||
return errors.New("缺少目标知识库身份,无法核对快照归属")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// matchesIdentity 用治理库里的期望值核对 bundle 自报的身份。
|
||||
func matchesIdentity(manifest librarysnapshot.Manifest, want Identity) error {
|
||||
if manifest.Scope != want.Scope {
|
||||
return fmt.Errorf("快照 scope=%q 与请求 %q 不一致", manifest.Scope, want.Scope)
|
||||
}
|
||||
if manifest.TenantID != want.TenantID {
|
||||
return fmt.Errorf("快照 tenant=%d 与请求 %d 不一致", manifest.TenantID, want.TenantID)
|
||||
}
|
||||
if manifest.KnowledgeBaseID != want.KnowledgeBaseID {
|
||||
return fmt.Errorf("快照知识库 %s 与请求 %s 不一致", manifest.KnowledgeBaseID, want.KnowledgeBaseID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func manifestBytes(manifest librarysnapshot.Manifest) int64 {
|
||||
var total int64
|
||||
seen := map[string]bool{}
|
||||
for _, part := range manifest.Parts {
|
||||
if seen[part.SHA256] {
|
||||
continue
|
||||
}
|
||||
seen[part.SHA256] = true
|
||||
total += part.Bytes
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// ---- ssh 传输:固定 argv,路径全部来自配置 ----
|
||||
|
||||
type sshRemote struct {
|
||||
config Config
|
||||
}
|
||||
|
||||
// remoteExitError 保留远端退出码:只有"文件不存在(44)"才能当成"还没有这份版本",
|
||||
// 网络断、权限错、docker 不可用都必须原样上抛——否则会把"读不到"当成"没有",
|
||||
// 于是继续搬一份没人验证过的快照。
|
||||
type remoteExitError struct {
|
||||
code int
|
||||
detail string
|
||||
}
|
||||
|
||||
func (e *remoteExitError) Error() string { return fmt.Sprintf("ssh exit %d: %s", e.code, e.detail) }
|
||||
|
||||
func (r *sshRemote) run(ctx context.Context, stdin io.Reader, stdout io.Writer, remote string) error {
|
||||
bounded, cancel := context.WithTimeout(ctx, r.config.OpTimeout)
|
||||
defer cancel()
|
||||
argv := []string{"-F", r.config.SSHConfig, "-o", "BatchMode=yes", "-o", "ConnectTimeout=10", r.config.SSHTarget, remote}
|
||||
command := exec.CommandContext(bounded, "ssh", argv...)
|
||||
command.Stdin = stdin
|
||||
command.Stdout = stdout
|
||||
var stderr strings.Builder
|
||||
command.Stderr = &stderr
|
||||
if err := command.Run(); err != nil {
|
||||
if bounded.Err() != nil {
|
||||
return fmt.Errorf("ssh %s 超时", r.config.SSHTarget)
|
||||
}
|
||||
detail := strings.TrimSpace(stderr.String())
|
||||
if len(detail) > 300 {
|
||||
detail = detail[:300]
|
||||
}
|
||||
var exit *exec.ExitError
|
||||
if errors.As(err, &exit) {
|
||||
return &remoteExitError{code: exit.ExitCode(), detail: detail}
|
||||
}
|
||||
return fmt.Errorf("ssh %s 失败: %s", r.config.SSHTarget, detail)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *sshRemote) PutBundle(ctx context.Context, name string, body io.Reader) error {
|
||||
path := filepath.Join(r.config.BundleDir, name)
|
||||
if err := r.run(ctx, nil, io.Discard, fmt.Sprintf("install -d -m 0700 %s", shellQuote(r.config.BundleDir))); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := r.run(ctx, body, io.Discard, fmt.Sprintf("cat > %s", shellQuote(path+".part"))); err != nil {
|
||||
return err
|
||||
}
|
||||
return r.run(ctx, nil, io.Discard, fmt.Sprintf("mv -f %s %s", shellQuote(path+".part"), shellQuote(path)))
|
||||
}
|
||||
|
||||
func (r *sshRemote) OpenBundle(ctx context.Context, name string) (io.ReadCloser, error) {
|
||||
path := filepath.Join(r.config.BundleDir, name)
|
||||
reader, writer := io.Pipe()
|
||||
go func() {
|
||||
writer.CloseWithError(r.run(ctx, nil, writer, fmt.Sprintf("cat %s", shellQuote(path))))
|
||||
}()
|
||||
return reader, nil
|
||||
}
|
||||
|
||||
func (r *sshRemote) RemoveBundle(ctx context.Context, name string) error {
|
||||
path := filepath.Join(r.config.BundleDir, name)
|
||||
return r.run(ctx, nil, io.Discard, fmt.Sprintf("rm -f %s %s", shellQuote(path), shellQuote(path+".part")))
|
||||
}
|
||||
|
||||
func (r *sshRemote) RunLibraryctl(ctx context.Context, args ...string) ([]byte, error) {
|
||||
quoted := make([]string, 0, len(args)+2)
|
||||
quoted = append(quoted, "docker", "exec", shellQuote(r.config.ExecutorContainer), "/app/libraryctl")
|
||||
for _, arg := range args {
|
||||
quoted = append(quoted, shellQuote(arg))
|
||||
}
|
||||
var out strings.Builder
|
||||
if err := r.run(ctx, nil, &out, strings.Join(quoted, " ")); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []byte(out.String()), nil
|
||||
}
|
||||
|
||||
func (r *sshRemote) ReadVersionManifest(ctx context.Context, id string) ([]byte, error) {
|
||||
path := filepath.Join(r.config.RemoteArchive, "versions", id+".json")
|
||||
var out strings.Builder
|
||||
// 只有 "文件不存在" 才算"还没有这份版本";其它失败必须让调用方停下来。
|
||||
command := fmt.Sprintf("if [ -f %s ]; then cat %s; else exit 44; fi", shellQuote(path), shellQuote(path))
|
||||
if err := r.run(ctx, nil, &out, command); err != nil {
|
||||
var exit *remoteExitError
|
||||
if errors.As(err, &exit) && exit.code == 44 {
|
||||
return nil, fmt.Errorf("%w", os.ErrNotExist)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return []byte(out.String()), nil
|
||||
}
|
||||
|
||||
func shellQuote(value string) string {
|
||||
return "'" + strings.ReplaceAll(value, "'", `'\''`) + "'"
|
||||
}
|
||||
|
||||
// ---- local 传输:只用于隔离 fixture(跨进程执行真正的 libraryctl) ----
|
||||
|
||||
type localRemote struct {
|
||||
config Config
|
||||
}
|
||||
|
||||
func (r *localRemote) path(name string) (string, error) {
|
||||
if _, err := bundleName(strings.TrimSuffix(name, ".tar")); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return filepath.Join(r.config.BundleDir, name), nil
|
||||
}
|
||||
|
||||
func (r *localRemote) PutBundle(ctx context.Context, name string, body io.Reader) error {
|
||||
path, err := r.path(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(r.config.BundleDir, 0700); err != nil {
|
||||
return err
|
||||
}
|
||||
out, err := os.OpenFile(path+".part", os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(out, io.LimitReader(body, r.config.MaxBundleBytes+1)); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
if err := out.Sync(); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(path+".part", path)
|
||||
}
|
||||
|
||||
func (r *localRemote) OpenBundle(ctx context.Context, name string) (io.ReadCloser, error) {
|
||||
path, err := r.path(name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return os.Open(path)
|
||||
}
|
||||
|
||||
func (r *localRemote) RemoveBundle(ctx context.Context, name string) error {
|
||||
path, err := r.path(name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, candidate := range []string{path, path + ".part"} {
|
||||
if err := os.Remove(candidate); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RunLibraryctl 在 fixture 里跨进程执行真正的 libraryctl 二进制(不是进程内模拟)。
|
||||
func (r *localRemote) RunLibraryctl(ctx context.Context, args ...string) ([]byte, error) {
|
||||
binary := os.Getenv("LIBRARY_HANDOFF_LIBRARYCTL")
|
||||
if binary == "" {
|
||||
return nil, errors.New("fixture 缺少 LIBRARY_HANDOFF_LIBRARYCTL(libraryctl 可执行路径)")
|
||||
}
|
||||
bounded, cancel := context.WithTimeout(ctx, r.config.OpTimeout)
|
||||
defer cancel()
|
||||
command := exec.CommandContext(bounded, binary, args...)
|
||||
command.Env = append(os.Environ(), "LIBRARY_HANDOFF_FIXTURE=1")
|
||||
var stderr strings.Builder
|
||||
command.Stderr = &stderr
|
||||
out, err := command.Output()
|
||||
if err != nil {
|
||||
detail := strings.TrimSpace(stderr.String())
|
||||
if len(detail) > 300 {
|
||||
detail = detail[:300]
|
||||
}
|
||||
return nil, fmt.Errorf("libraryctl 失败: %s", detail)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (r *localRemote) ReadVersionManifest(ctx context.Context, id string) ([]byte, error) {
|
||||
path := filepath.Join(r.config.RemoteArchive, "versions", id+".json")
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return raw, nil
|
||||
}
|
||||
@@ -0,0 +1,356 @@
|
||||
package libraryhandoff
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
var libraryctlBinary string
|
||||
|
||||
// TestMain 构建真正的 libraryctl 二进制:local 通道的"云端"动作必须跨进程执行,
|
||||
// 否则"跨进程证据"只是同一次调用里的函数调用。
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "handoff-bin-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
binary := filepath.Join(dir, "libraryctl")
|
||||
build := exec.Command("go", "build", "-o", binary, "knowledge-governance/cmd/libraryctl")
|
||||
build.Stderr = os.Stderr
|
||||
if err := build.Run(); err != nil {
|
||||
panic("build libraryctl: " + err.Error())
|
||||
}
|
||||
libraryctlBinary = binary
|
||||
code := m.Run()
|
||||
os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
const (
|
||||
fixtureTenant = 10000
|
||||
fixtureKB = "kb-handoff-fixture"
|
||||
)
|
||||
|
||||
func newArchive(t *testing.T) (string, *librarysnapshot.Archive) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
if err := os.Chmod(root, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archive, err := librarysnapshot.Open(root, 1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { archive.Close() })
|
||||
return root, archive
|
||||
}
|
||||
|
||||
func seal(t *testing.T, archive *librarysnapshot.Archive) string {
|
||||
t.Helper()
|
||||
part, err := archive.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mapping := sha256.Sum256([]byte("fixture-mapping"))
|
||||
manifest := librarysnapshot.Manifest{
|
||||
Format: librarysnapshot.Format, Scope: "internal", TenantID: fixtureTenant,
|
||||
KnowledgeBaseID: fixtureKB, CreatedAt: time.Date(2026, 10, 30, 0, 0, 0, 0, time.UTC),
|
||||
Compatibility: librarysnapshot.Compatibility{
|
||||
WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8",
|
||||
EmbeddingModel: "fixture-model", EmbeddingDimension: 1024,
|
||||
IndexMappingSHA256: hex.EncodeToString(mapping[:]),
|
||||
},
|
||||
}
|
||||
for _, name := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} {
|
||||
current := part
|
||||
current.Kind, current.Name = "database", name
|
||||
manifest.Parts = append(manifest.Parts, current)
|
||||
}
|
||||
for _, name := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
|
||||
current := part
|
||||
current.Kind, current.Name = "wiki", name
|
||||
manifest.Parts = append(manifest.Parts, current)
|
||||
}
|
||||
index := part
|
||||
index.Kind, index.Name = "index", "chunks"
|
||||
manifest.Parts = append(manifest.Parts, index)
|
||||
id, err := archive.Seal(manifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// fixture 在同一个临时根下模拟"云端主机":bundles/ 是暂存,archive/ 是执行器归档根。
|
||||
type fixture struct {
|
||||
root string
|
||||
cfg Config
|
||||
remoteA string
|
||||
}
|
||||
|
||||
func newFixture(t *testing.T, localArchive string) fixture {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
remoteArchive := filepath.Join(root, "archive")
|
||||
staging := filepath.Join(root, "staging")
|
||||
for _, dir := range []string{remoteArchive, staging, filepath.Join(root, "bundles")} {
|
||||
if err := os.MkdirAll(dir, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
t.Setenv("LIBRARY_HANDOFF_LIBRARYCTL", libraryctlBinary)
|
||||
return fixture{root: root, remoteA: remoteArchive, cfg: Config{
|
||||
Transport: TransportLocal, LocalRoot: root,
|
||||
ExecutorContainer: "fixture-executor",
|
||||
BundleDir: filepath.Join(root, "bundles"),
|
||||
RemoteArchive: remoteArchive,
|
||||
LocalArchive: localArchive,
|
||||
StagingDir: staging,
|
||||
MaxBundleBytes: 8 << 20,
|
||||
OpTimeout: 60 * time.Second,
|
||||
}}
|
||||
}
|
||||
|
||||
// withLocalArchive 复用**同一台云端 fixture**(同一 root),只换本机归档根:
|
||||
// 读回必须打到"发布执行器所在的那台机器"的归档,而不是另起一份空目录。
|
||||
func withLocalArchive(t *testing.T, up fixture, localArchive string) Config {
|
||||
t.Helper()
|
||||
staging := filepath.Join(t.TempDir(), "staging")
|
||||
if err := os.MkdirAll(staging, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
config := up.cfg
|
||||
config.LocalArchive = localArchive
|
||||
config.StagingDir = staging
|
||||
return config
|
||||
}
|
||||
|
||||
func TestPushThenFetchKeepsIdentityAndBytesAcrossProcesses(t *testing.T) {
|
||||
sourceRoot, source := newArchive(t)
|
||||
id := seal(t, source)
|
||||
up := newFixture(t, sourceRoot)
|
||||
pusher, err := New(up.cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := Identity{SnapshotID: id, Scope: "internal", TenantID: fixtureTenant, KnowledgeBaseID: fixtureKB}
|
||||
pushed, err := pusher.Push(context.Background(), want)
|
||||
if err != nil {
|
||||
t.Fatalf("Push() error = %v", err)
|
||||
}
|
||||
if pushed.Reused || pushed.Parts != 11 || pushed.Snapshot != id {
|
||||
t.Fatalf("push evidence = %#v", pushed)
|
||||
}
|
||||
// 云端确实落了版本清单,且摘要自证。
|
||||
raw, err := os.ReadFile(filepath.Join(up.remoteA, "versions", id+".json"))
|
||||
if err != nil {
|
||||
t.Fatalf("remote manifest missing: %v", err)
|
||||
}
|
||||
sum := sha256.Sum256(raw)
|
||||
if hex.EncodeToString(sum[:]) != id {
|
||||
t.Fatal("remote manifest digest mismatch")
|
||||
}
|
||||
// 暂存目录不残留传输 bundle。
|
||||
if entries, _ := os.ReadDir(up.cfg.BundleDir); len(entries) != 0 {
|
||||
t.Fatalf("bundle staging not cleaned: %v", entries)
|
||||
}
|
||||
|
||||
// 另一台"本机"取回:跨目录 + 跨进程(local 通道执行真正的 libraryctl)。
|
||||
destinationRoot, _ := newArchive(t)
|
||||
fetcher, err := New(withLocalArchive(t, up, destinationRoot), nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fetched, err := fetcher.Fetch(context.Background(), id)
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch() error = %v", err)
|
||||
}
|
||||
if fetched.Snapshot != id || fetched.Parts != 11 || fetched.KnowledgeBase != fixtureKB {
|
||||
t.Fatalf("fetch evidence = %#v", fetched)
|
||||
}
|
||||
verify, err := librarysnapshot.Open(destinationRoot, 1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer verify.Close()
|
||||
if _, err := verify.Load(id); err != nil {
|
||||
t.Fatalf("fetched snapshot does not verify: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushRejectsIdentityThatDoesNotMatchTheSnapshot(t *testing.T) {
|
||||
sourceRoot, source := newArchive(t)
|
||||
id := seal(t, source)
|
||||
up := newFixture(t, sourceRoot)
|
||||
pusher, err := New(up.cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wrong := Identity{SnapshotID: id, Scope: "internal", TenantID: fixtureTenant, KnowledgeBaseID: "kb-other"}
|
||||
if _, err := pusher.Push(context.Background(), wrong); err == nil {
|
||||
t.Fatal("Push() accepted a snapshot whose KB does not match the request")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(up.remoteA, "versions", id+".json")); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatal("a rejected push left a version on the executor")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushIsIdempotentAndRejectsMissingLocalSnapshot(t *testing.T) {
|
||||
sourceRoot, source := newArchive(t)
|
||||
id := seal(t, source)
|
||||
up := newFixture(t, sourceRoot)
|
||||
pusher, err := New(up.cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := Identity{SnapshotID: id, Scope: "internal", TenantID: fixtureTenant, KnowledgeBaseID: fixtureKB}
|
||||
if _, err := pusher.Push(context.Background(), want); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := pusher.Push(context.Background(), want)
|
||||
if err != nil || !again.Reused {
|
||||
t.Fatalf("second push = %#v, %v (want reused)", again, err)
|
||||
}
|
||||
missing := Identity{SnapshotID: hex.EncodeToString(bytes.Repeat([]byte{7}, 32)), Scope: "internal",
|
||||
TenantID: fixtureTenant, KnowledgeBaseID: fixtureKB}
|
||||
if _, err := pusher.Push(context.Background(), missing); err == nil {
|
||||
t.Fatal("Push() accepted a snapshot that does not exist locally")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConcurrentPushOfSameSnapshotSerialises(t *testing.T) {
|
||||
sourceRoot, source := newArchive(t)
|
||||
id := seal(t, source)
|
||||
up := newFixture(t, sourceRoot)
|
||||
pusher, err := New(up.cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := Identity{SnapshotID: id, Scope: "internal", TenantID: fixtureTenant, KnowledgeBaseID: fixtureKB}
|
||||
var wg sync.WaitGroup
|
||||
errs := make([]error, 3)
|
||||
for i := 0; i < 3; i++ {
|
||||
wg.Add(1)
|
||||
go func(index int) {
|
||||
defer wg.Done()
|
||||
_, errs[index] = pusher.Push(context.Background(), want)
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
for index, err := range errs {
|
||||
if err != nil {
|
||||
t.Fatalf("concurrent push %d failed: %v", index, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchRefusesTamperedRemoteArchive(t *testing.T) {
|
||||
sourceRoot, source := newArchive(t)
|
||||
id := seal(t, source)
|
||||
up := newFixture(t, sourceRoot)
|
||||
pusher, err := New(up.cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := pusher.Push(context.Background(), Identity{SnapshotID: id, Scope: "internal",
|
||||
TenantID: fixtureTenant, KnowledgeBaseID: fixtureKB}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// 篡改云端一个 blob:云端 libraryctl snapshot-pack 会在 Load 时逐 blob 重算并失败。
|
||||
blobs, err := os.ReadDir(filepath.Join(up.remoteA, "blobs"))
|
||||
if err != nil || len(blobs) == 0 {
|
||||
t.Fatalf("remote blobs = %v, %v", blobs, err)
|
||||
}
|
||||
target := filepath.Join(up.remoteA, "blobs", blobs[0].Name())
|
||||
if err := os.Chmod(target, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(target, []byte("tampered"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
destinationRoot, _ := newArchive(t)
|
||||
fetcher, err := New(withLocalArchive(t, up, destinationRoot), nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := fetcher.Fetch(context.Background(), id); err == nil {
|
||||
t.Fatal("Fetch() imported a tampered snapshot")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "versions", id+".json")); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatal("a failed fetch left a version locally")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchRefusesUnknownRemoteSnapshotWithoutClaimingRecoverable(t *testing.T) {
|
||||
destinationRoot, _ := newArchive(t)
|
||||
empty := newFixture(t, destinationRoot)
|
||||
fetcher, err := New(empty.cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
unknown := hex.EncodeToString(bytes.Repeat([]byte{9}, 32))
|
||||
if _, err := fetcher.Fetch(context.Background(), unknown); err == nil {
|
||||
t.Fatal("Fetch() reported success for a snapshot that exists nowhere")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "versions", unknown+".json")); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatal("unknown snapshot produced a local version")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandoffRejectsCallerControlledNamesAndPaths(t *testing.T) {
|
||||
if _, err := bundleName("../../etc/passwd"); err == nil {
|
||||
t.Fatal("bundleName accepted a traversal name")
|
||||
}
|
||||
if _, err := bundleName("ABCDEF"); err == nil {
|
||||
t.Fatal("bundleName accepted a non-digest name")
|
||||
}
|
||||
if _, err := New(Config{Transport: TransportSSH, SSHTarget: "x", SSHConfig: "/tmp/x"}, nil); err == nil {
|
||||
t.Fatal("New() accepted an incomplete SSH configuration")
|
||||
}
|
||||
if _, err := New(Config{Transport: TransportLocal}, nil); err == nil {
|
||||
t.Fatal("New() accepted local transport without an isolation root")
|
||||
}
|
||||
}
|
||||
|
||||
// 断线/超时:远端读版本清单一直失败必须报错,不能当成"没有版本"然后继续搬。
|
||||
func TestPushSurfacesRemoteErrorsInsteadOfAssumingAbsent(t *testing.T) {
|
||||
sourceRoot, source := newArchive(t)
|
||||
id := seal(t, source)
|
||||
up := newFixture(t, sourceRoot)
|
||||
handoff := &Handoff{config: Config{LocalArchive: sourceRoot, MaxBundleBytes: 1 << 20, OpTimeout: time.Second},
|
||||
remote: errorRemote{}, now: time.Now}
|
||||
_, err := handoff.Push(context.Background(), Identity{SnapshotID: id, Scope: "internal",
|
||||
TenantID: fixtureTenant, KnowledgeBaseID: fixtureKB})
|
||||
if err == nil {
|
||||
t.Fatal("Push() ignored a remote transport error")
|
||||
}
|
||||
_ = up
|
||||
}
|
||||
|
||||
type errorRemote struct{}
|
||||
|
||||
func (errorRemote) PutBundle(context.Context, string, io.Reader) error {
|
||||
return errors.New("network down")
|
||||
}
|
||||
func (errorRemote) OpenBundle(context.Context, string) (io.ReadCloser, error) {
|
||||
return nil, errors.New("network down")
|
||||
}
|
||||
func (errorRemote) RemoveBundle(context.Context, string) error { return errors.New("network down") }
|
||||
func (errorRemote) RunLibraryctl(context.Context, ...string) ([]byte, error) {
|
||||
return nil, errors.New("network down")
|
||||
}
|
||||
func (errorRemote) ReadVersionManifest(context.Context, string) ([]byte, error) {
|
||||
return nil, errors.New("network down")
|
||||
}
|
||||
@@ -352,3 +352,17 @@ func decodeBundleManifest(raw []byte, id string) (Manifest, map[string]Part, err
|
||||
}
|
||||
return manifest, expected, nil
|
||||
}
|
||||
|
||||
// DecodeManifest 校验一份"外部拿到的"版本清单字节:摘要必须等于给定快照 ID,
|
||||
// 且必须是规范形式(与 Seal 写出的字节一致)。跨主机搬运时接收侧用它核对
|
||||
// "这份清单是不是它自称的那份",而不是相信对方返回的字段。
|
||||
func DecodeManifest(raw []byte, id string) (Manifest, error) {
|
||||
if !validDigest(id) || digest(raw) != id {
|
||||
return Manifest{}, errors.New("snapshot manifest digest mismatch")
|
||||
}
|
||||
manifest, _, err := decodeBundleManifest(raw, id)
|
||||
if err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
return manifest, nil
|
||||
}
|
||||
|
||||
@@ -792,3 +792,31 @@ FROM library_release_authorizations WHERE scope=$1 ORDER BY created_at DESC, id
|
||||
}
|
||||
return result, rows.Err()
|
||||
}
|
||||
|
||||
// LibraryJobSnapshots 返回该 scope 最近完成作业引用过的快照 ID(预备份/目标快照)。
|
||||
//
|
||||
// 只用于"云端产生的版本回传到本机"这一件事:本机缺哪些、要不要取,按这些 ID 逐个核对。
|
||||
// 只读、按作业终态过滤,不推断内容是否可用——是否可用由归档自身的 Load 决定。
|
||||
func (s *Store) LibraryJobSnapshots(ctx context.Context, scope string, limit int) ([]string, error) {
|
||||
if scope == "" || limit <= 0 || limit > 500 {
|
||||
return nil, errors.New("library job snapshots requires scope and a bounded limit")
|
||||
}
|
||||
rows, err := s.db.Query(ctx, `
|
||||
SELECT DISTINCT snapshot_id FROM library_jobs
|
||||
WHERE scope=$1 AND state='succeeded' AND snapshot_id <> ''
|
||||
ORDER BY snapshot_id
|
||||
LIMIT $2`, scope, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list library job snapshots: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
ids := make([]string, 0, limit)
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids, rows.Err()
|
||||
}
|
||||
|
||||
@@ -267,6 +267,11 @@
|
||||
"package": "knowledge-governance/cmd/worker",
|
||||
"test": "TestTakeoverLedgerRefusesForeignOrMissingBackupEvidence",
|
||||
"reason": "必须在真实 PostgreSQL 上跑:复用未登记的预备份、字节数不一致、写集合被篡改/换作业/换预备份这些负例都依赖版本行身份与 upsert 语义(假 store 造不全);默认 CI 无库 ⇒ 跳过,本机一次性库已实跑(修前红:复用未登记预备份竟然成功)"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/cmd/worker",
|
||||
"test": "TestHandoffChainGatesReleaseJobOnVerifiedTransfer",
|
||||
"reason": "需要隔离的 fixture PostgreSQL(KG_TEST_DSN)与真实 libraryctl 二进制(KG_TEST_LIBRARYCTL);CI 不注入测试库。本机已用独立测试库容器实跑通过,未运行不等于通过"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user