diff --git a/docs/library-sync-acceptance.md b/docs/library-sync-acceptance.md new file mode 100644 index 0000000..7e26d9a --- /dev/null +++ b/docs/library-sync-acceptance.md @@ -0,0 +1,104 @@ +# 知识库级版本同步:开发与验收清单 + +## 交付边界 + +本清单记录已确认需求与验收条件,不表示功能已实现。日常编辑按文档,生产版本及恢复按知识库完整状态;发布管理仅有对内、对外两个目标。开发授权不包含生产同步、生产恢复或真实数据清理。生产执行仍须在发布管理经管理员授权;保留数量、天数、容量预算未设置,自动清理默认关闭。 + +现有文档包发布与执行级补偿只能作为复用基础,不能作为本清单完成证据。静态样图不作为产品运行证据。组织绩效与周报外发不在本次范围。 + +## 已确认:封存隔离方案 A + +用户选择封存期间自动暂停测试 WeKnora 服务,封存后恢复,不修改 WeKnora 源码。测试问答在暂停期间暂不可用,停用时长须实测;不暂停生产服务,也不因此取得生产同步、恢复或真实清理授权。 + +实现要求:先等待处理任务完成,枚举并核实全部测试写入者和副本;暂停后重新检查状态,数据库、索引和对象存储保持可读。持久记录暂停前运行状态及作业归属,覆盖成功、失败、取消和进程重启后的恢复;仅恢复本作业暂停的服务,并读回服务健康。完整归档封存后即可恢复测试服务,后续生产传输从归档读取,不延长测试停用时间。 + +方案已确认不等于暂停/恢复功能已实现;恢复保障验证前不停止运行中的测试服务。 + +## 开发与测试顺序 + +1. 核查实际 WeKnora 的文档删除、Wiki 更新与完成信号,以及跨数据库、对象、索引的快照与恢复契约。 +2. 实现完整版本清单、一致性校验、授权与持久化作业;逐项运行失败用例再实现通过。 +3. 接通实际导出、独立生产预备份、传输、验证、恢复及持久化进度;隔离环境演练故障、重启与重试。 +4. 发布管理接真实接口,存储管理接真实测量及依赖保护;验证浏览器交互与权限。 +5. 完整回归、审查、提交交付;生产实测单列等待授权,不以隔离验收代替。 + +## 验收项目 + +| ID | 范围 | 必须证明的行为 | 状态 | +| --- | --- | --- | --- | +| V01 | 两个目标 | 只按对内、对外操作;身份来自已配置映射,不按名称猜测 | 待验证 | +| V02 | 测试就绪 | 全部解析、索引及 Wiki 就绪;问答与性能验收绑定本版,旧验收不能放行新变化 | 待验证 | +| V03 | Wiki 撤销 | 删除文档后验证共享正文、实体、概念、边与引用的实际变化;删除成功或引用剥离不能代替此证据 | 待验证 | +| V04 | 完整固定版本 | 清单覆盖文档集合、处理结果、资源、索引、完整 Wiki;无部分文档配整库 Wiki 的路径 | 待验证 | +| V05 | 并发修改 | 导出中发生源变更时拒绝不一致版本,或由已证明的固定快照协议隔离;导出前后表面计数相同不算充分证明 | 待验证 | +| V06 | 内容范围 | 仅允许知识数据;账号、会话、凭据及无关知识库不被同步或恢复 | 待验证 | +| B01 | 写前预备份 | 每次生产变更前创建旧状态完整备份,校验恢复资料完整性;失败时零生产知识写入 | 待验证 | +| B02 | 独立恢复依赖 | 资源在测试侧删除后,保留版本仍能恢复,不依赖测试临时对象或短期签名地址 | 待验证 | +| B03 | 完整 Wiki 资产 | 每版保存完整 Wiki 及引用关联,校验摘要、数量和版本绑定 | 待验证 | +| S01 | 数据同步 | 数据库、资源、索引、Wiki 均真实落地并校验,增量仅为传输优化 | 待验证 | +| S02 | 无生产再计算 | 生产侧不解析、Embedding 或生成 Wiki,使用测试验收结果 | 待验证 | +| S03 | 一致生效 | 不宣称不存在的跨系统原子性;明确可见性保护、失败补偿与验证边界 | 待验证 | +| S04 | 故障与恢复 | 各阶段失败、超时与进程重启后可追溯、可恢复,不重复破坏性动作 | 待验证 | +| S05 | 幂等与互斥 | 重复请求不启动重复同步;同目标同步、恢复、清理之间互斥;不同请求体复用键被拒绝 | 待验证 | +| R01 | 版本恢复 | 恢复生产整套文档、资源、索引与 Wiki;测试知识库不回退 | 待验证 | +| R02 | 恢复失败 | 不标记成功、不丢旧恢复依据,明确受影响阶段及后续可执行措施 | 待验证 | +| R03 | 速度 | 分开实测备份与恢复耗时、数据量、吞吐及环境;未经测量不承诺快速 | 待验证 | +| P01 | 作业阶段 | 持久化准备、备份、同步、验证、恢复阶段与开始/更新时间、错误;生产验证成功才完成 | 待验证 | +| P02 | 四类进度 | 数据库记录、资源文件/字节、索引分块、Wiki 页面/关联数据来自实际测量;总量未知不造百分比 | 待验证 | +| P03 | 明细 | 可查看实际对象、数量、字节、校验及异常;筛选与总量不依赖前端当前页 | 待验证 | +| U01 | 主要入口 | 对内/对外切换、启动同步、版本与快照、当前作业预备份入口均可见;旧版历史可追溯 | 待验证 | +| U02 | 权限 | 未就绪明确阻断原因;会话注入操作者;只读/工程师不能获得生产执行权限 | 待验证 | +| U03 | 操作确认 | 启动、恢复显示准确目标及影响并保留授权;运行中禁止重复启动 | 待验证 | +| D01 | 存储策略 | 未设阈值如实显示未设置,自动清理默认关闭,不隐含年龄删除 | 待验证 | +| D02 | 占用测量 | 区分逻辑大小、独占、共享、实际占用;无法测量显示未知,不当成零 | 待验证 | +| D03 | 保护引用 | 当前恢复依据、活跃作业、锁定/保留版本及共享资源受到保护 | 待验证 | +| D04 | 清理预览 | 列明对象、依赖、受影响版本及可测释放量;执行前重新核验引用与活跃作业 | 待验证 | +| D05 | 容量门禁 | 空间不足告警或阻断,不为继续作业偷删受保护备份 | 待验证 | +| D06 | 测试残留 | 覆盖取件缓存、临时解析、同步暂存、失败上传、删除/重解析残留,不删除源文件 | 待验证 | +| D07 | 实际回收 | 区分逻辑删除与实际释放,核查对象历史版本/分片及数据库、索引回收;保留版本清理后仍可恢复 | 待验证 | + +## 证据登记规则 + +每项完成需附实现位置、运行命令与退出码、可回看的证据路径、验证环境及尚未覆盖范围。单元测试、隔离数据库集成、模拟外部服务、真实测试知识库、生产验收分别标注,不能相互替代。 + +真实清理和生产恢复尚未授权的项目保持待授权/待验证;不得为了打勾执行。整项范围未覆盖不得以子切片通过称为本阶段完成。 + +## 已运行的基础模块证据(不等于整体验收) + +实现位置:`internal/librarysnapshot/` 与 `internal/releaseadapter/v071/library_*.go`。 + +- PostgreSQL 整库只读 Repeatable Read 导出:显式包含空 Wiki 表;未完成解析、删除文档残留任务、死信、未知 schema、Neo4j 启用场景拒绝自动放行。 +- Elasticsearch 使用 PIT 与 search_after,核查精确总量、返回范围、重复项与部分结果;PIT 只证明索引自身的视图。 +- 候选归档独立持有资源字节,校验表、scope、资源依赖、索引关联及摘要;内容寻址复用、重开加载、同一文件描述符校验读取、共享/独占引用字节已测试。引用字节不是物理占用或实际可回收字节。 +- `go test -race ./internal/librarysnapshot ./internal/releaseadapter/v071 -count=1`:退出码 0;默认跳过需要显式配置的隔离集成,不能据此声称集成通过。 +- 隔离 PostgreSQL + Elasticsearch 集成另行运行,退出码 0。只创建随机 `kg_library_test_` 前缀夹具;结束后 PostgreSQL 同前缀库残留计数为 0。 +- MVCC 反向自检:仅通过 Go overlay 把 Repeatable Read 改为 Read Committed,同一真实隔离集成退出码 1,失败断言为 `mixed PostgreSQL versions during export`。原始实现复跑退出码 0。 +- `make verify`:退出码 0。本地门禁不是远端 CI,也不是新 UI 或真实知识库完整验收。 + +证据保存在本机仓库忽略目录 `backups/library-sync-foundation-20261027/`:`unit-tests.jsonl`、`kg-library-integration.log`、`kg-library-mvcc-negative.log`、`kg-library-verify.log`。隔离执行脚本副本同目录留存,不含静态凭据;运行时从本地测试容器读取。 + +### 当前不能放行的边界 + +1. `ArchiveLibraryCandidate` 的 seal 只代表归档完整性,不代表跨系统同一时点、Wiki 语义验收通过或生产授权。尚未接到生产执行器。 +2. 数据库 MVCC 与 ES PIT 是各自独立的快照,不能把两者拼接视为一致整版。用户已确认方案 A:封存期间自动暂停测试服务;全部写入者隔离、异常恢复及实际停用时长尚待实现与验证,不修改 WeKnora 源码,不暂停生产。 +3. 已检查本地 v0.7.1 源码:`EnqueueWikiRetract` 在持久化 pending op 失败时记录警告后返回;因此 pending/dead-letter 都为空仍不证明撤销语义已处理。必须补充删除后共享 Wiki 内容的实测与验收绑定。 +4. 生产预备份、独立恢复演练、持久化协调作业、授权 API、真实进度、发布页面与完整存储治理仍未交付;以上验收项目保持待验证。 +5. 当前新增代码未提交、未推送、未部署;未执行生产同步、恢复或真实业务清理。 + +## 方案 A:暂停与恢复原语的实测证据 + +新增 `internal/libraryfreeze/`,包含持久化作业日志、源服务组互斥、暂停前原运行状态、不可变容器 ID、恢复读回,以及本地 Unix socket Docker 控制器。Docker 控制器只接受对应 source 且 `environment=test`、`freeze-role=writer` 的容器;不按模糊名称选取,不控制数据库/索引存储,不使用远程默认 Docker context。运行容器必须有健康检查。 + +已经执行的验证: + +- `go test -race ./internal/libraryfreeze ./internal/librarysnapshot ./internal/releaseadapter/v071 -count=1`,退出码 0。 +- `KG_FREEZE_DOCKER_ISOLATED=1 go test -race -v ./internal/libraryfreeze -run '^TestFreezeIsolatedDocker$' -count=1`,退出码 0。使用已缓存 busybox 创建随机标签、无网络、无宿主挂载的一次性容器,真实验证停止→捕获窗口→重启健康、原本停止容器不启动;控制子进程在停止后退出,再由新恢复调用读持久日志启动原容器。清理后按标签读回无残留。 +- 异常覆盖:停止回包丢失、停止未生效、新副本出现、容器被替换、捕获错误/panic/取消、日志写入失败、恢复健康失败、畸形日志、同源并发及重复恢复。捕获 panic 和日志失写故障先实际报红,修复后通过。 +- 不使用有延迟强杀计时器的停止方式;仅发送 SIGTERM 并等待实际停止。若停止结果不明且服务仍运行,不把“现在运行”误判为已恢复,日志保持待恢复,防止随后迟到的停止把服务留在关闭状态。 +- 新增 Docker 集成用例已在 `scripts/test-scope.json` 精确声明 opt-in;普通 CI 明确跳过,不冒称执行。完整 `make verify` 退出码 0,属于本地门禁。 + +证据:本机 `backups/library-sync-foundation-20261027/kg-freeze-docker-integration.log`、`kg-library-freeze-verify.log`。 + +已新增 `internal/releaseadapter/v071/library_capture.go`,在源服务组互斥下串起预检、暂停、停后复检、PG/ES 导出、独立归档和恢复。协调测试使用 PG/服务控制夹具与 HTTP 搜索夹具,覆盖预检忙时零停止、暂停前后新增任务阻断、搜索失败恢复、旧作业重放不重新暂停、跨 scope 复用拒绝、归档丢失不返回成功;不把这组夹具当成实际业务知识库全链路验收。最终竞态回归、隔离 Docker 重跑与 `make verify` 均退出码 0,最终日志为同目录 `kg-freeze-docker-final.log`、`kg-library-freeze-final-verify.log`。 + +**仍缺少的接线**:真实测试部署的全部写入者枚举及标签/健康契约、全源任务就绪检查的实际提供方、worker 调用捕获协调函数、启动恢复入口和持续恢复调度。单独的本地 Docker 控制器不能控制远端测试主机;远端场景需在测试主机运行受限执行器,不能向浏览器或普通 API 暴露 Docker socket。上述测试不能证明这些接线已完成,尚未暂停实际测试 WeKnora。 diff --git a/internal/libraryfreeze/boundaries_test.go b/internal/libraryfreeze/boundaries_test.go new file mode 100644 index 0000000..e3b0f79 --- /dev/null +++ b/internal/libraryfreeze/boundaries_test.go @@ -0,0 +1,138 @@ +package libraryfreeze + +import ( + "context" + "errors" + "strings" + "testing" +) + +func TestFreezeCancelledCaptureCannotSucceed(t *testing.T) { + _, c, e := freezeFixture(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + result, err := e.Run(ctx, "cancelled", func(context.Context) (string, error) { cancel(); return strings.Repeat("a", 64), nil }) + if err == nil || result.State == "succeeded" { + t.Fatalf("cancelled work succeeded: %+v %v", result, err) + } + if !c.services[0].Running || !c.services[0].Healthy { + t.Fatal("cancel prevented recovery") + } +} + +type faultyControl struct { + *fixtureControl + mode string +} + +func (c *faultyControl) Stop(ctx context.Context, id string) error { + if c.mode == "stop-noop" { + return nil + } + err := c.fixtureControl.Stop(ctx, id) + if c.mode == "lost-ack" { + return errors.New("lost stop acknowledgement") + } + return err +} +func (c *faultyControl) Start(ctx context.Context, id string) error { + err := c.fixtureControl.Start(ctx, id) + if c.mode == "unhealthy" { + for i := range c.services { + if c.services[i].ID == id { + c.services[i].Healthy = false + } + } + } + return err +} +func TestFreezeFailureBoundaries(t *testing.T) { + for _, mode := range []string{"lost-ack", "stop-noop", "unhealthy", "new-replica", "replacement", "capture-error"} { + t.Run(mode, func(t *testing.T) { + _, c, e := freezeFixture(t) + e.control = &faultyControl{fixtureControl: c, mode: mode} + called := false + result, err := e.Run(context.Background(), "fault-job", func(context.Context) (string, error) { + called = true + switch mode { + case "new-replica": + c.services = append(c.services, Service{ID: "unexpected-app", Running: true, Healthy: true}) + case "replacement": + c.services[0].ID = "replacement-app" + case "capture-error": + return "", errors.New("fixture error") + } + return strings.Repeat("a", 64), nil + }) + if err == nil || result.State == "succeeded" { + t.Fatalf("fault reported success: %+v %v", result, err) + } + if (mode == "lost-ack" || mode == "stop-noop") && called { + t.Fatal("capture ran without verified stop") + } + if mode == "replacement" { + for _, call := range c.calls { + if strings.HasPrefix(call, "start:") { + t.Fatal("replacement was started") + } + } + } else if !c.services[0].Running { + t.Fatal("original service not restored") + } + if mode == "unhealthy" || mode == "replacement" { + j, err := e.load("fault-job") + if err != nil || terminal(j.State) { + t.Fatal("unverified recovery marked terminal") + } + } + }) + } +} +func TestFreezeSourceLockBlocksSeparateCoordinator(t *testing.T) { + root, c, e := freezeFixture(t) + other, err := Open(root, "test-source", c) + if err != nil { + t.Fatal(err) + } + defer other.Close() + _, err = e.Run(context.Background(), "outer", func(context.Context) (string, error) { + _, blocked := other.Run(context.Background(), "inner", func(context.Context) (string, error) { t.Fatal("overlapping capture"); return "", nil }) + if blocked == nil { + t.Fatal("source lock did not block") + } + if err := other.Recover(context.Background()); err == nil { + t.Fatal("recovery raced active capture") + } + return strings.Repeat("a", 64), nil + }) + if err != nil { + t.Fatal(err) + } +} +func TestFreezeRejectsMalformedRecoveryJournal(t *testing.T) { + for _, mode := range []string{"unknown-state", "invalid-service", "duplicate-service", "false-ownership"} { + t.Run(mode, func(t *testing.T) { + _, c, e := freezeFixture(t) + j := journal{Version: 1, Source: "test-source", Job: "invalid", Result: Result{State: "stopping"}, Services: []serviceRecord{{Service: Service{ID: "app-a", Running: true}, StopRequested: true}}} + switch mode { + case "unknown-state": + j.State = "invented" + case "invalid-service": + j.Services[0].ID = "../../wrong" + case "duplicate-service": + j.Services = append(j.Services, j.Services[0]) + case "false-ownership": + j.Services[0].Running = false + } + if err := e.save(j); err != nil { + t.Fatal(err) + } + if err := e.Recover(context.Background()); err == nil { + t.Fatal("malformed journal accepted") + } + if len(c.calls) != 0 { + t.Fatalf("malformed journal caused service writes: %v", c.calls) + } + }) + } +} diff --git a/internal/libraryfreeze/docker.go b/internal/libraryfreeze/docker.go new file mode 100644 index 0000000..3a293ff --- /dev/null +++ b/internal/libraryfreeze/docker.go @@ -0,0 +1,185 @@ +package libraryfreeze + +import ( + "context" + "encoding/hex" + "encoding/json" + "errors" + "io" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + "time" +) + +const sourceLabel = "io.knowledge-governance.freeze-source" +const environmentLabel = "io.knowledge-governance.environment" +const roleLabel = "io.knowledge-governance.freeze-role" + +type DockerController struct { + source string + timeout time.Duration + client *http.Client + transport *http.Transport +} + +func NewDockerController(socket, source string, timeout time.Duration) (*DockerController, error) { + if !filepath.IsAbs(socket) || !validID(source) || timeout <= 0 || timeout > 5*time.Minute { + return nil, errors.New("invalid test Docker controller configuration") + } + info, err := os.Stat(socket) + if err != nil || info.Mode()&os.ModeSocket == 0 { + return nil, errors.New("local Docker socket unavailable") + } + transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { + return (&net.Dialer{}).DialContext(ctx, "unix", socket) + }} + client := &http.Client{Transport: transport, Timeout: timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("Docker redirects prohibited") }} + return &DockerController{source: source, timeout: timeout, client: client, transport: transport}, nil +} +func (d *DockerController) Close() { d.transport.CloseIdleConnections() } +func (d *DockerController) request(ctx context.Context, method, path string, out any) error { + req, err := http.NewRequestWithContext(ctx, method, "http://docker"+path, nil) + if err != nil { + return errors.New("invalid Docker request") + } + resp, err := d.client.Do(req) + if err != nil { + return errors.New("test Docker request failed") + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return errors.New("test Docker request rejected") + } + if out != nil { + data, err := io.ReadAll(io.LimitReader(resp.Body, 1024*1024+1)) + if err != nil || len(data) > 1024*1024 || json.Unmarshal(data, out) != nil { + return errors.New("invalid Docker response") + } + } + return nil +} +func dockerID(id string) bool { + b, err := hex.DecodeString(id) + return err == nil && len(b) == 32 && strings.ToLower(id) == id +} + +type dockerInspection struct { + ID string `json:"Id"` + Config struct { + Labels map[string]string `json:"Labels"` + } `json:"Config"` + State struct { + Running bool + Paused bool + Restarting bool + Dead bool + Status string + Health *struct{ Status string } + } `json:"State"` +} + +func (d *DockerController) inspect(ctx context.Context, id string) (Service, error) { + if !dockerID(id) { + return Service{}, errors.New("immutable Docker ID required") + } + var data dockerInspection + if err := d.request(ctx, "GET", "/containers/"+id+"/json", &data); err != nil { + return Service{}, err + } + labels := data.Config.Labels + if data.ID != id || labels[sourceLabel] != d.source || labels[environmentLabel] != "test" || labels[roleLabel] != "writer" { + return Service{}, errors.New("container is not an approved test writer") + } + if data.State.Paused || data.State.Restarting || data.State.Dead { + return Service{}, errors.New("test writer has unstable runtime state") + } + if data.State.Status != "running" && data.State.Status != "exited" && data.State.Status != "created" { + return Service{}, errors.New("unknown test writer runtime state") + } + if data.State.Running != (data.State.Status == "running") { + return Service{}, errors.New("contradictory test writer state") + } + healthy := data.State.Running && data.State.Health != nil && data.State.Health.Status == "healthy" + return Service{ID: id, Running: data.State.Running, Healthy: healthy}, nil +} +func (d *DockerController) Inventory(ctx context.Context) ([]Service, error) { + filters, _ := json.Marshal(map[string][]string{"label": {sourceLabel + "=" + d.source}}) + var containers []struct { + ID string `json:"Id"` + } + if err := d.request(ctx, "GET", "/containers/json?all=1&filters="+url.QueryEscape(string(filters)), &containers); err != nil { + return nil, err + } + if len(containers) == 0 || len(containers) > 256 { + return nil, errors.New("empty or oversized test writer group") + } + items := make([]Service, 0, len(containers)) + seen := map[string]bool{} + for _, container := range containers { + if seen[container.ID] { + return nil, errors.New("duplicate Docker instance") + } + seen[container.ID] = true + s, err := d.inspect(ctx, container.ID) + if err != nil { + return nil, err + } + items = append(items, s) + } + return items, nil +} +func (d *DockerController) wait(ctx context.Context, id string, running bool) error { + ticker := time.NewTicker(100 * time.Millisecond) + defer ticker.Stop() + for { + s, err := d.inspect(ctx, id) + if err != nil { + return err + } + if s.Running == running && (!running || s.Healthy) { + return nil + } + select { + case <-ctx.Done(): + return errors.New("test writer state did not settle before deadline") + case <-ticker.C: + } + } +} + +// Only SIGTERM is sent. There is no delayed SIGKILL timer which could fire after +// recovery starts the service again. An unresolved graceful stop remains pending +// in the coordinator journal; running-now is not evidence of recovery. +func (d *DockerController) Stop(ctx context.Context, id string) error { + bounded, cancel := context.WithTimeout(ctx, d.timeout) + defer cancel() + s, err := d.inspect(bounded, id) + if err != nil { + return err + } + if !s.Running { + return nil + } + if err = d.request(bounded, "POST", "/containers/"+id+"/kill?signal=SIGTERM", nil); err != nil { + return err + } + return d.wait(bounded, id, false) +} +func (d *DockerController) Start(ctx context.Context, id string) error { + bounded, cancel := context.WithTimeout(ctx, d.timeout) + defer cancel() + s, err := d.inspect(bounded, id) + if err != nil { + return err + } + if !s.Running { + if err = d.request(bounded, "POST", "/containers/"+id+"/start", nil); err != nil { + return err + } + } + return d.wait(bounded, id, true) +} diff --git a/internal/libraryfreeze/docker_boundaries_test.go b/internal/libraryfreeze/docker_boundaries_test.go new file mode 100644 index 0000000..fd45056 --- /dev/null +++ b/internal/libraryfreeze/docker_boundaries_test.go @@ -0,0 +1,95 @@ +package libraryfreeze + +import ( + "context" + "encoding/json" + "net" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestDockerControllerRefusesWrongScopeAndUnsafeRuntime(t *testing.T) { + for _, mode := range []string{"production", "wrong-source", "storage", "paused", "restarting", "dead", "no-health", "short-id", "wrong-response-id", "duplicate"} { + t.Run(mode, func(t *testing.T) { + root, err := os.MkdirTemp(os.TempDir(), "kgf-") + if err != nil { + t.Fatal(err) + } + defer os.RemoveAll(root) + socket := filepath.Join(root, "s") + listener, err := net.Listen("unix", socket) + if err != nil { + t.Fatal(err) + } + id := strings.Repeat("a", 64) + if mode == "short-id" { + id = "aaa" + } + writes := 0 + server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != "GET" { + writes++ + w.WriteHeader(204) + return + } + if strings.HasSuffix(r.URL.Path, "/containers/json") { + items := []map[string]string{{"Id": id}} + if mode == "duplicate" { + items = append(items, items[0]) + } + _ = json.NewEncoder(w).Encode(items) + return + } + labels := map[string]string{sourceLabel: "test-source", environmentLabel: "test", roleLabel: "writer"} + switch mode { + case "production": + labels[environmentLabel] = "production" + case "wrong-source": + labels[sourceLabel] = "different-source" + case "storage": + labels[roleLabel] = "database" + } + state := map[string]any{"Running": true, "Status": "running", "Health": map[string]string{"Status": "healthy"}} + switch mode { + case "paused": + state["Paused"] = true + case "restarting": + state["Restarting"] = true + case "dead": + state["Dead"] = true + case "no-health": + delete(state, "Health") + } + actualID := id + if mode == "wrong-response-id" { + actualID = strings.Repeat("b", 64) + } + _ = json.NewEncoder(w).Encode(map[string]any{"Id": actualID, "Config": map[string]any{"Labels": labels}, "State": state}) + })} + go server.Serve(listener) + defer server.Close() + d, err := NewDockerController(socket, "test-source", time.Second) + if err != nil { + t.Fatal(err) + } + defer d.Close() + journalRoot := filepath.Join(root, "j") + if err = os.Mkdir(journalRoot, 0700); err != nil { + t.Fatal(err) + } + e, err := Open(journalRoot, "test-source", d) + if err != nil { + t.Fatal(err) + } + defer e.Close() + _, err = e.Run(context.Background(), "blocked", func(context.Context) (string, error) { t.Fatal("unsafe capture"); return "", nil }) + if err == nil || writes != 0 { + t.Fatalf("unsafe source changed: err=%v writes=%d", err, writes) + } + }) + } +} diff --git a/internal/libraryfreeze/docker_integration_test.go b/internal/libraryfreeze/docker_integration_test.go new file mode 100644 index 0000000..f79c1a9 --- /dev/null +++ b/internal/libraryfreeze/docker_integration_test.go @@ -0,0 +1,127 @@ +package libraryfreeze + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "os" + "os/exec" + "strings" + "testing" + "time" +) + +func TestFreezeIsolatedDocker(t *testing.T) { + if os.Getenv("KG_FREEZE_DOCKER_ISOLATED") != "1" { + t.Skip("explicit opt-in required for disposable local Docker fixture") + } + if os.Getenv("KG_FREEZE_DOCKER_CHILD") == "1" { + d, err := NewDockerController("/var/run/docker.sock", os.Getenv("KG_FREEZE_SOURCE"), 10*time.Second) + if err != nil { + os.Exit(31) + } + e, err := Open(os.Getenv("KG_FREEZE_ROOT"), os.Getenv("KG_FREEZE_SOURCE"), d) + if err != nil { + os.Exit(32) + } + _, _ = e.Run(context.Background(), "crashed-real", func(context.Context) (string, error) { os.Exit(37); return "", nil }) + os.Exit(33) + } + run := func(args ...string) string { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "docker", append([]string{"--host", "unix:///var/run/docker.sock"}, args...)...) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("fixture Docker command failed: %v %s", err, out) + } + return strings.TrimSpace(string(out)) + } + image := run("image", "inspect", "busybox:1.36", "--format", "{{.Id}}") + var token [8]byte + if _, err := rand.Read(token[:]); err != nil { + t.Fatal(err) + } + source := "kg-freeze-test-" + hex.EncodeToString(token[:]) + var ids []string + defer func() { + for _, id := range ids { + run("rm", "-f", id) + } + if remaining := run("ps", "-aq", "--filter", "label="+sourceLabel+"="+source); remaining != "" { + t.Error("disposable freeze containers remain") + } + }() + for _, name := range []string{"running", "originally-off"} { + id := run("create", "--name", source+"-"+name, "--network", "none", "--read-only", "--tmpfs", "/tmp:rw,noexec,nosuid,size=1m", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--memory", "32m", "--cpus", "0.1", "--restart", "unless-stopped", "--label", sourceLabel+"="+source, "--label", environmentLabel+"=test", "--label", roleLabel+"=writer", "--health-cmd", "test -f /tmp/ready", "--health-interval", "1s", "--health-timeout", "1s", "--health-retries", "3", image, "sh", "-c", "trap 'exit 0' TERM; touch /tmp/ready; while :; do sleep 1 & wait $!; done") + ids = append(ids, id) + } + d, err := NewDockerController("/var/run/docker.sock", source, 10*time.Second) + if err != nil { + t.Fatal(err) + } + defer d.Close() + if err = d.Start(context.Background(), ids[0]); err != nil { + t.Fatal(err) + } + root := t.TempDir() + if err = os.Chmod(root, 0700); err != nil { + t.Fatal(err) + } + e, err := Open(root, source, d) + if err != nil { + t.Fatal(err) + } + defer e.Close() + assertRestored := func() { + t.Helper() + a, err := d.inspect(context.Background(), ids[0]) + if err != nil || !a.Running || !a.Healthy { + t.Fatalf("original running fixture not healthy: %+v %v", a, err) + } + b, err := d.inspect(context.Background(), ids[1]) + if err != nil || b.Running { + t.Fatal("originally-off fixture was started") + } + } + started := time.Now() + result, err := e.Run(context.Background(), "real-success", func(context.Context) (string, error) { + items, err := d.Inventory(context.Background()) + if err != nil { + return "", err + } + for _, s := range items { + if s.Running { + t.Fatal("real writer active during capture") + } + } + return strings.Repeat("b", 64), nil + }) + if err != nil || result.State != "succeeded" { + t.Fatalf("real freeze=%+v %v", result, err) + } + assertRestored() + t.Logf("disposable_container_freeze_and_health_restore=%s", time.Since(started)) + cmd := exec.Command(os.Args[0], "-test.run=^TestFreezeIsolatedDocker$") + cmd.Env = append(os.Environ(), "KG_FREEZE_DOCKER_CHILD=1", "KG_FREEZE_SOURCE="+source, "KG_FREEZE_ROOT="+root) + err = cmd.Run() + var exit *exec.ExitError + if !errors.As(err, &exit) || exit.ExitCode() != 37 { + t.Fatalf("crash injection did not reach stopped window: %v", err) + } + stopped, err := d.inspect(context.Background(), ids[0]) + if err != nil || stopped.Running { + t.Fatal("crashed fixture was not stopped") + } + if err = e.Recover(context.Background()); err != nil { + t.Fatal(err) + } + assertRestored() + j, err := e.load("crashed-real") + if err != nil || j.State != "failed" || j.Code != "interrupted" { + t.Fatalf("bad recovery journal: %+v %v", j, err) + } + t.Log("real process exit -> journal reload -> original container healthy; originally-off container remains off") +} diff --git a/internal/libraryfreeze/docker_test.go b/internal/libraryfreeze/docker_test.go new file mode 100644 index 0000000..b9e4934 --- /dev/null +++ b/internal/libraryfreeze/docker_test.go @@ -0,0 +1,78 @@ +package libraryfreeze + +import ( + "context" + "encoding/json" + "net" + "net/http" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestDockerControllerUsesScopedIDsAndGracefulSignal(t *testing.T) { + root, err := os.MkdirTemp(os.TempDir(), "kgf-") + if err != nil { + t.Fatal(err) + } + defer os.RemoveAll(root) + socket := filepath.Join(root, "engine.sock") + listener, err := net.Listen("unix", socket) + if err != nil { + t.Fatal(err) + } + id := strings.Repeat("a", 64) + running := true + var writes []string + server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == "GET" && strings.HasSuffix(r.URL.Path, "/containers/json"): + if !strings.Contains(r.URL.Query().Get("filters"), "test-source") { + t.Error("missing scope filter") + } + _ = json.NewEncoder(w).Encode([]map[string]string{{"Id": id}}) + case r.Method == "GET" && strings.HasSuffix(r.URL.Path, "/json"): + _ = json.NewEncoder(w).Encode(map[string]any{"Id": id, "Config": map[string]any{"Labels": map[string]string{sourceLabel: "test-source", environmentLabel: "test", roleLabel: "writer"}}, "State": map[string]any{"Running": running, "Status": map[bool]string{true: "running", false: "exited"}[running], "Health": map[string]string{"Status": "healthy"}}}) + case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/kill"): + if r.URL.Query().Get("signal") != "SIGTERM" { + t.Error("unsafe stop signal") + } + writes = append(writes, "stop") + running = false + w.WriteHeader(204) + case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/start"): + writes = append(writes, "start") + running = true + w.WriteHeader(204) + default: + w.WriteHeader(404) + } + })} + go server.Serve(listener) + defer server.Close() + control, err := NewDockerController(socket, "test-source", time.Second) + if err != nil { + t.Fatal(err) + } + defer control.Close() + journalRoot := filepath.Join(root, "journal") + if err = os.Mkdir(journalRoot, 0700); err != nil { + t.Fatal(err) + } + engine, err := Open(journalRoot, "test-source", control) + if err != nil { + t.Fatal(err) + } + defer engine.Close() + result, err := engine.Run(context.Background(), "fixture-job", func(context.Context) (string, error) { + if running { + t.Fatal("writer was not stopped") + } + return strings.Repeat("b", 64), nil + }) + if err != nil || result.State != "succeeded" || strings.Join(writes, ",") != "stop,start" { + t.Fatalf("result=%+v err=%v writes=%v", result, err, writes) + } +} diff --git a/internal/libraryfreeze/failure_test.go b/internal/libraryfreeze/failure_test.go new file mode 100644 index 0000000..b8ec6ec --- /dev/null +++ b/internal/libraryfreeze/failure_test.go @@ -0,0 +1,43 @@ +package libraryfreeze + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestFreezeRestoresDespiteCapturePanic(t *testing.T) { + _, c, e := freezeFixture(t) + func() { + defer func() { _ = recover() }() + _, _ = e.Run(context.Background(), "panic-job", func(context.Context) (string, error) { panic("fixture panic") }) + }() + if !c.services[0].Running || !c.services[0].Healthy { + t.Fatal("capture panic left test service stopped") + } + j, err := e.load("panic-job") + if err != nil || j.State != "failed" { + t.Fatalf("journal=%+v err=%v", j, err) + } +} +func TestFreezeRestoresEvenWhenJournalBecomesUnwritable(t *testing.T) { + root, c, e := freezeFixture(t) + _, err := e.Run(context.Background(), "disk-failure", func(context.Context) (string, error) { + path := filepath.Join(root, "disk-failure.json") + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + if err := os.Mkdir(path, 0700); err != nil { + t.Fatal(err) + } + return strings.Repeat("a", 64), nil + }) + if err == nil { + t.Fatal("lost journal must not report success") + } + if !c.services[0].Running || !c.services[0].Healthy { + t.Fatal("journal failure prevented service restoration") + } +} diff --git a/internal/libraryfreeze/freeze.go b/internal/libraryfreeze/freeze.go new file mode 100644 index 0000000..4a652cc --- /dev/null +++ b/internal/libraryfreeze/freeze.go @@ -0,0 +1,472 @@ +// Package libraryfreeze coordinates a test-only maintenance window. It never +// grants production authorization or starts services which were already stopped. +package libraryfreeze + +import ( + "bytes" + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "errors" + "io" + "os" + "path/filepath" + "sort" + "strings" + "syscall" + "time" +) + +type Service struct { + ID string `json:"id"` + Running bool `json:"running"` + Healthy bool `json:"healthy"` +} + +// Inventory must enumerate the complete, independently scoped test writer set, +// using immutable instance IDs. Never implement it as only the currently running +// subset. Start must wait for readiness; callers still read back the result. +type Controller interface { + Inventory(context.Context) ([]Service, error) + Stop(context.Context, string) error + Start(context.Context, string) error +} +type Result struct { + State string `json:"state"` + SnapshotID string `json:"snapshot_id,omitempty"` +} +type serviceRecord struct { + Service + StopRequested bool `json:"stop_requested"` + StopVerified bool `json:"stop_verified"` + Restored bool `json:"restored"` +} +type journal struct { + Version int `json:"version"` + Source string `json:"source"` + Job string `json:"job"` + Result + Services []serviceRecord `json:"services"` + StartedAt time.Time `json:"started_at"` + FinishedAt *time.Time `json:"finished_at,omitempty"` + Code string `json:"code,omitempty"` +} +type Coordinator struct { + root *os.Root + source string + control Controller +} + +func Open(root, source string, c Controller) (*Coordinator, error) { + if !filepath.IsAbs(root) || !validID(source) || c == nil { + return nil, errors.New("invalid freeze configuration") + } + info, err := os.Lstat(root) + if err != nil || !info.IsDir() || info.Mode().Perm() != 0700 { + return nil, errors.New("freeze directory must be private and real") + } + resolved, err := filepath.EvalSymlinks(root) + if err != nil || resolved != filepath.Clean(root) { + return nil, errors.New("freeze directory contains symlink") + } + r, err := os.OpenRoot(root) + if err != nil { + return nil, err + } + return &Coordinator{root: r, source: source, control: c}, nil +} +func (c *Coordinator) Close() error { return c.root.Close() } + +func (c *Coordinator) Recover(ctx context.Context) error { + lock, err := c.lock() + if err != nil { + return err + } + defer unlock(lock) + dir, err := c.root.Open(".") + if err != nil { + return err + } + defer dir.Close() + entries, err := dir.ReadDir(-1) + if err != nil { + return err + } + // Validate every journal before issuing recovery writes. Corrupt records or + // another source's directory must not turn into broad service startup. + var pending []journal + for _, entry := range entries { + if !strings.HasSuffix(entry.Name(), ".json") { + continue + } + job := strings.TrimSuffix(entry.Name(), ".json") + if !validID(job) { + return errors.New("invalid freeze journal name") + } + j, err := c.load(job) + if err != nil { + return err + } + if !terminal(j.State) { + pending = append(pending, j) + } + } + if len(pending) > 1 { + return errors.New("multiple interrupted freezes require operator reconciliation") + } + for _, j := range pending { + if err = c.restore(ctx, &j); err != nil { + return err + } + j.State = "failed" + j.Code = "interrupted" + ended := time.Now().UTC() + j.FinishedAt = &ended + if err = c.save(j); err != nil { + return err + } + } + return nil +} +func invokeCapture(ctx context.Context, capture func(context.Context) (string, error)) (id string, err error) { + defer func() { + if recover() != nil { + id = "" + err = errors.New("candidate capture panicked") + } + }() + return capture(ctx) +} + +func validID(s string) bool { + if len(s) == 0 || len(s) > 128 { + return false + } + for _, r := range s { + if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_') { + return false + } + } + return true +} +func terminal(s string) bool { return s == "succeeded" || s == "failed" } + +// A kernel lock serializes the whole test writer group, not individual KBs. +// A crash releases the lock, but leaves the write-ahead journal for recovery. +func (c *Coordinator) lock() (*os.File, error) { + f, err := c.root.OpenFile("operation.lock", os.O_CREATE|os.O_RDWR|syscall.O_NOFOLLOW, 0600) + if err != nil { + return nil, err + } + if err = syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil { + f.Close() + return nil, errors.New("test source freeze is busy") + } + return f, nil +} +func unlock(f *os.File) { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); _ = f.Close() } +func (c *Coordinator) save(j journal) error { + data, err := json.Marshal(j) + if err != nil { + return err + } + var token [16]byte + if _, err = rand.Read(token[:]); err != nil { + return err + } + name := "write-" + hex.EncodeToString(token[:]) + ".tmp" + f, err := c.root.OpenFile(name, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + return err + } + defer c.root.Remove(name) + defer f.Close() + if _, err = f.Write(data); err != nil { + return err + } + if err = f.Sync(); err != nil { + return err + } + if err = f.Close(); err != nil { + return err + } + if err = c.root.Rename(name, j.Job+".json"); err != nil { + return err + } + dir, err := c.root.Open(".") + if err != nil { + return err + } + defer dir.Close() + return dir.Sync() +} +func (c *Coordinator) load(job string) (journal, error) { + f, err := c.root.OpenFile(job+".json", os.O_RDONLY|syscall.O_NOFOLLOW, 0) + if err != nil { + return journal{}, err + } + defer f.Close() + info, err := f.Stat() + if err != nil || !info.Mode().IsRegular() || info.Size() > 1024*1024 { + return journal{}, errors.New("invalid freeze journal file") + } + data, err := io.ReadAll(io.LimitReader(f, 1024*1024+1)) + if err != nil { + return journal{}, err + } + var j journal + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + if err = dec.Decode(&j); err != nil { + return journal{}, errors.New("invalid freeze journal") + } + var extra any + if dec.Decode(&extra) != io.EOF || j.Version != 1 || j.Source != c.source || j.Job != job || !validID(j.Job) { + return journal{}, errors.New("freeze journal identity mismatch") + } + switch j.State { + case "stopping", "capturing", "restoring", "succeeded", "failed": + default: + return journal{}, errors.New("invalid freeze journal state") + } + if len(j.Services) == 0 || len(j.Services) > 256 { + return journal{}, errors.New("invalid freeze journal services") + } + for i, s := range j.Services { + if !validID(s.ID) || (i > 0 && j.Services[i-1].ID >= s.ID) || s.StopRequested && !s.Running || s.Restored && (!s.StopRequested || !s.Running) { + return journal{}, errors.New("invalid freeze service ownership") + } + } + return j, nil +} +func (c *Coordinator) pending() error { + dir, err := c.root.Open(".") + if err != nil { + return err + } + defer dir.Close() + entries, err := dir.ReadDir(-1) + if err != nil { + return err + } + for _, entry := range entries { + if strings.HasSuffix(entry.Name(), ".json") { + job := strings.TrimSuffix(entry.Name(), ".json") + if !validID(job) { + return errors.New("invalid freeze journal name") + } + j, err := c.load(job) + if err != nil { + return err + } + if !terminal(j.State) { + return errors.New("interrupted test freeze requires recovery") + } + } + } + return nil +} +func inventory(ctx context.Context, control Controller) ([]Service, error) { + items, err := control.Inventory(ctx) + if err != nil { + return nil, errors.New("test writer inventory unavailable") + } + if len(items) == 0 || len(items) > 256 { + return nil, errors.New("invalid test writer inventory") + } + seen := map[string]bool{} + for _, s := range items { + if !validID(s.ID) || seen[s.ID] { + return nil, errors.New("invalid test writer identity") + } + seen[s.ID] = true + } + sort.Slice(items, func(i, j int) bool { return items[i].ID < items[j].ID }) + return items, nil +} +func (c *Coordinator) stopped(ctx context.Context, j journal) error { + items, err := inventory(ctx, c.control) + if err != nil { + return err + } + if len(items) != len(j.Services) { + return errors.New("test writer topology changed") + } + for i, s := range items { + if s.ID != j.Services[i].ID || s.Running { + return errors.New("test writers not isolated") + } + } + return nil +} +func (c *Coordinator) restore(ctx context.Context, j *journal) error { + j.State = "restoring" + // Restoring services is still necessary if the journal disk fails after a + // persisted stop intent. Keep the error, but do not let it suppress recovery. + var failures error = c.save(*j) + for i := range j.Services { + s := &j.Services[i] + if !s.Running || !s.StopRequested { + continue + } + items, err := inventory(ctx, c.control) + if err != nil { + failures = errors.Join(failures, err) + continue + } + found := false + for _, actual := range items { + if actual.ID != s.ID { + continue + } + found = true + if actual.Running && !s.StopVerified { + failures = errors.Join(failures, errors.New("stop outcome remains uncertain; service may still stop later")) + break + } + if !actual.Running { + s.StopVerified = true + if saveErr := c.save(*j); saveErr != nil { + failures = errors.Join(failures, saveErr) + } + if err = c.control.Start(ctx, s.ID); err != nil { + failures = errors.Join(failures, errors.New("test service restart failed")) + break + } + } + after, readErr := inventory(ctx, c.control) + healthy := false + if readErr == nil { + for _, a := range after { + if a.ID == s.ID && a.Running && a.Healthy { + healthy = true + } + } + } + if !healthy { + failures = errors.Join(failures, errors.New("test service health unverified")) + break + } + s.Restored = true + if err = c.save(*j); err != nil { + failures = errors.Join(failures, err) + } + break + } + if !found { + failures = errors.Join(failures, errors.New("original test service missing; replacement not started")) + } + } + return failures +} +func (c *Coordinator) Run(ctx context.Context, job string, capture func(context.Context) (string, error)) (Result, error) { + return c.RunPrepared(ctx, job, func(context.Context) error { return nil }, capture) +} + +// RunPrepared runs readiness checks under the source lock, before any stop +// intent. It does not repeat checks or capture for an already completed job. +func (c *Coordinator) RunPrepared(ctx context.Context, job string, prepare func(context.Context) error, capture func(context.Context) (string, error)) (Result, error) { + if !validID(job) || capture == nil || prepare == nil { + return Result{}, errors.New("invalid freeze request") + } + if err := ctx.Err(); err != nil { + return Result{}, err + } + lock, err := c.lock() + if err != nil { + return Result{}, err + } + defer unlock(lock) + previous, err := c.load(job) + if err == nil { + if previous.State == "succeeded" { + return previous.Result, nil + } + return previous.Result, errors.New("freeze request already exists; recovery or a new request is required") + } + if !errors.Is(err, os.ErrNotExist) { + return Result{}, err + } + if err = c.pending(); err != nil { + return Result{}, err + } + if err = prepare(ctx); err != nil { + return Result{}, err + } + if err = ctx.Err(); err != nil { + return Result{}, err + } + items, err := inventory(ctx, c.control) + if err != nil { + return Result{}, err + } + j := journal{Version: 1, Source: c.source, Job: job, Result: Result{State: "stopping"}, StartedAt: time.Now().UTC()} + for _, s := range items { + if s.Running && !s.Healthy { + return Result{}, errors.New("test writer is not ready") + } + j.Services = append(j.Services, serviceRecord{Service: s}) + } + if err = c.save(j); err != nil { + return Result{}, err + } + var workErr error + for i := range j.Services { + if !j.Services[i].Running { + continue + } + j.Services[i].StopRequested = true + if workErr = c.save(j); workErr != nil { + break + } + if workErr = c.control.Stop(ctx, j.Services[i].ID); workErr != nil { + break + } + } + if workErr == nil { + workErr = c.stopped(ctx, j) + } + if workErr == nil { + for i := range j.Services { + if j.Services[i].StopRequested { + j.Services[i].StopVerified = true + } + } + j.State = "capturing" + workErr = c.save(j) + } + if workErr == nil { + j.SnapshotID, workErr = invokeCapture(ctx, capture) + } + if workErr == nil { + workErr = ctx.Err() + } + if workErr == nil { + decoded, decodeErr := hex.DecodeString(j.SnapshotID) + if decodeErr != nil || len(decoded) != 32 || strings.ToLower(j.SnapshotID) != j.SnapshotID { + workErr = errors.New("invalid candidate snapshot digest") + } + } + if workErr == nil { + workErr = c.stopped(ctx, j) + } + recoveryCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute) + defer cancel() + recoveryErr := c.restore(recoveryCtx, &j) + if recoveryErr != nil { + return j.Result, errors.Join(workErr, recoveryErr) + } + if workErr != nil { + j.State = "failed" + j.Code = "capture_or_isolation_failed" + } else { + j.State = "succeeded" + } + ended := time.Now().UTC() + j.FinishedAt = &ended + if err = c.save(j); err != nil { + return j.Result, err + } + return j.Result, workErr +} diff --git a/internal/libraryfreeze/freeze_test.go b/internal/libraryfreeze/freeze_test.go new file mode 100644 index 0000000..bf569f4 --- /dev/null +++ b/internal/libraryfreeze/freeze_test.go @@ -0,0 +1,77 @@ +package libraryfreeze + +import ( + "context" + "os" + "strings" + "testing" +) + +type fixtureControl struct { + services []Service + calls []string +} + +func (c *fixtureControl) Inventory(context.Context) ([]Service, error) { + return append([]Service(nil), c.services...), nil +} +func (c *fixtureControl) Stop(_ context.Context, id string) error { + c.calls = append(c.calls, "stop:"+id) + for i := range c.services { + if c.services[i].ID == id { + c.services[i].Running = false + c.services[i].Healthy = false + } + } + return nil +} +func (c *fixtureControl) Start(_ context.Context, id string) error { + c.calls = append(c.calls, "start:"+id) + for i := range c.services { + if c.services[i].ID == id { + c.services[i].Running = true + c.services[i].Healthy = true + } + } + return nil +} +func freezeFixture(t *testing.T) (string, *fixtureControl, *Coordinator) { + t.Helper() + root := t.TempDir() + if err := os.Chmod(root, 0700); err != nil { + t.Fatal(err) + } + c := &fixtureControl{services: []Service{{ID: "app-a", Running: true, Healthy: true}, {ID: "app-b"}}} + engine, err := Open(root, "test-source", c) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { engine.Close() }) + return root, c, engine +} +func TestFreezeRestoresOnlyServicesOriginallyRunning(t *testing.T) { + _, c, e := freezeFixture(t) + captures := 0 + result, err := e.Run(context.Background(), "job-a", func(ctx context.Context) (string, error) { + captures++ + for _, s := range c.services { + if s.Running { + t.Fatal("capture started while writer running") + } + } + return strings.Repeat("a", 64), nil + }) + if err != nil { + t.Fatal(err) + } + if result.State != "succeeded" || result.SnapshotID != strings.Repeat("a", 64) || captures != 1 { + t.Fatalf("result=%+v captures=%d", result, captures) + } + if strings.Join(c.calls, ",") != "stop:app-a,start:app-a" { + t.Fatalf("calls=%v", c.calls) + } + replay, err := e.Run(context.Background(), "job-a", func(context.Context) (string, error) { t.Fatal("replay captured again"); return "", nil }) + if err != nil || replay.SnapshotID != result.SnapshotID { + t.Fatalf("replay=%+v err=%v", replay, err) + } +} diff --git a/internal/libraryfreeze/preflight_test.go b/internal/libraryfreeze/preflight_test.go new file mode 100644 index 0000000..4db9aed --- /dev/null +++ b/internal/libraryfreeze/preflight_test.go @@ -0,0 +1,24 @@ +package libraryfreeze + +import ( + "context" + "errors" + "strings" + "testing" +) + +func TestFreezePreflightRejectsWithoutStoppingAndReplaySkipsIt(t *testing.T) { + _, c, e := freezeFixture(t) + _, err := e.RunPrepared(context.Background(), "prepared", func(context.Context) error { return errors.New("tasks active") }, func(context.Context) (string, error) { t.Fatal("capture before ready"); return "", nil }) + if err == nil || len(c.calls) != 0 { + t.Fatal("preflight did not protect services") + } + r, err := e.RunPrepared(context.Background(), "prepared", func(context.Context) error { return nil }, func(context.Context) (string, error) { return strings.Repeat("a", 64), nil }) + if err != nil || r.State != "succeeded" { + t.Fatalf("ready capture: %+v %v", r, err) + } + _, err = e.RunPrepared(context.Background(), "prepared", func(context.Context) error { t.Fatal("completed job consulted changing source"); return nil }, func(context.Context) (string, error) { t.Fatal("completed job captured again"); return "", nil }) + if err != nil { + t.Fatal(err) + } +} diff --git a/internal/libraryfreeze/recovery_test.go b/internal/libraryfreeze/recovery_test.go new file mode 100644 index 0000000..ef55460 --- /dev/null +++ b/internal/libraryfreeze/recovery_test.go @@ -0,0 +1,48 @@ +package libraryfreeze + +import ( + "context" + "errors" + "os" + "os/exec" + "strings" + "testing" +) + +func TestFreezeRecoveryAfterProcessExit(t *testing.T) { + if os.Getenv("KG_FREEZE_CHILD") == "exit-after-stop" { + c := &fixtureControl{services: []Service{{ID: "app-a", Running: true, Healthy: true}, {ID: "app-b"}}} + e, err := Open(os.Getenv("KG_FREEZE_ROOT"), "test-source", c) + if err != nil { + os.Exit(25) + } + _, _ = e.Run(context.Background(), "crashed-job", func(context.Context) (string, error) { os.Exit(27); return "", nil }) + os.Exit(26) + } + root, c, e := freezeFixture(t) + cmd := exec.Command(os.Args[0], "-test.run=^TestFreezeRecoveryAfterProcessExit$") + cmd.Env = append(os.Environ(), "KG_FREEZE_CHILD=exit-after-stop", "KG_FREEZE_ROOT="+root) + err := cmd.Run() + var exit *exec.ExitError + if !errors.As(err, &exit) || exit.ExitCode() != 27 { + t.Fatalf("child did not exit during capture: %v", err) + } + c.services[0].Running = false + c.services[0].Healthy = false + if _, err = e.Run(context.Background(), "new-job", func(context.Context) (string, error) { t.Fatal("new capture before recovery"); return "", nil }); err == nil { + t.Fatal("unfinished job did not block") + } + if err = e.Recover(context.Background()); err != nil { + t.Fatal(err) + } + j, err := e.load("crashed-job") + if err != nil || j.State != "failed" || j.Code != "interrupted" { + t.Fatalf("journal=%+v err=%v", j, err) + } + if strings.Join(c.calls, ",") != "start:app-a" { + t.Fatalf("recovery calls=%v", c.calls) + } + if err = e.Recover(context.Background()); err != nil || len(c.calls) != 1 { + t.Fatalf("recovery is not idempotent: %v", err) + } +} diff --git a/internal/libraryfreeze/uncertain_test.go b/internal/libraryfreeze/uncertain_test.go new file mode 100644 index 0000000..b4524af --- /dev/null +++ b/internal/libraryfreeze/uncertain_test.go @@ -0,0 +1,36 @@ +package libraryfreeze + +import ( + "context" + "errors" + "testing" +) + +type pendingStopControl struct{ *fixtureControl } + +func (c *pendingStopControl) Stop(context.Context, string) error { + return errors.New("stop may complete later") +} +func TestFreezeUncertainStopCannotBecomeTerminalWhileStillRunning(t *testing.T) { + _, c, e := freezeFixture(t) + e.control = &pendingStopControl{c} + _, err := e.Run(context.Background(), "uncertain", func(context.Context) (string, error) { t.Fatal("capture after uncertain stop"); return "", nil }) + if err == nil { + t.Fatal("uncertain stop succeeded") + } + j, err := e.load("uncertain") + if err != nil || terminal(j.State) { + t.Fatal("running now was treated as proof that delayed stop cannot happen") + } + if err = e.Recover(context.Background()); err == nil { + t.Fatal("uncertain running stop falsely recovered") + } + c.services[0].Running = false + c.services[0].Healthy = false + if err = e.Recover(context.Background()); err != nil { + t.Fatal(err) + } + if !c.services[0].Running { + t.Fatal("delayed stop not restored") + } +} diff --git a/internal/librarysnapshot/archive.go b/internal/librarysnapshot/archive.go new file mode 100644 index 0000000..7a6c4e2 --- /dev/null +++ b/internal/librarysnapshot/archive.go @@ -0,0 +1,399 @@ +// Package librarysnapshot stores immutable, content-addressed library artifacts. +// It does not authorize releases or imply cross-system snapshot consistency. +package librarysnapshot + +import ( + "bytes" + "crypto/rand" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "io" + "knowledge-governance/internal/releasebundle" + "math" + "os" + "path/filepath" + "sort" + "syscall" + "time" +) + +const Format = "weknora-library-snapshot/v1" +const maxManifestBytes = 16 * 1024 * 1024 + +type Part struct { + Kind string `json:"kind"` + Name string `json:"name"` + SHA256 string `json:"sha256"` + Bytes int64 `json:"bytes"` + Count int64 `json:"count"` +} +type Compatibility struct { + WeKnoraVersion string `json:"weknora_version"` + SchemaVersion int `json:"schema_version"` + RetrieverEngine string `json:"retriever_engine"` + EmbeddingModel string `json:"embedding_model"` + EmbeddingDimension int `json:"embedding_dimension"` + IndexMappingSHA256 string `json:"index_mapping_sha256"` +} + +type Manifest struct { + Compatibility Compatibility `json:"compatibility"` + Format string `json:"format"` + Scope string `json:"scope"` + TenantID uint64 `json:"tenant_id"` + KnowledgeBaseID string `json:"knowledge_base_id"` + CreatedAt time.Time `json:"created_at"` + Parts []Part `json:"parts"` +} +type Archive struct { + root, blobs, versions *os.Root + maxBlobBytes int64 +} + +func Open(root string, maxBlobBytes int64) (*Archive, error) { + if !filepath.IsAbs(root) || maxBlobBytes < 1 || maxBlobBytes == math.MaxInt64 { + return nil, errors.New("invalid archive configuration") + } + info, err := os.Lstat(root) + if err != nil || !info.IsDir() || info.Mode().Perm() != 0700 { + return nil, errors.New("archive root must be a private real directory") + } + resolved, err := filepath.EvalSymlinks(root) + if err != nil || resolved != filepath.Clean(root) { + return nil, errors.New("archive root contains symlinks") + } + r, err := os.OpenRoot(root) + if err != nil { + return nil, err + } + a := &Archive{root: r, maxBlobBytes: maxBlobBytes} + for _, dir := range []string{"blobs", "versions"} { + if err := r.Mkdir(dir, 0700); err != nil && !errors.Is(err, os.ErrExist) { + a.Close() + return nil, err + } + info, err := r.Lstat(dir) + if err != nil || !info.IsDir() || info.Mode().Perm() != 0700 { + a.Close() + return nil, errors.New("archive subdirectory must be private and real") + } + sub, err := r.OpenRoot(dir) + if err != nil { + a.Close() + return nil, err + } + if dir == "blobs" { + a.blobs = sub + } else { + a.versions = sub + } + } + return a, nil +} +func (a *Archive) Close() error { + var err error + for _, r := range []*os.Root{a.versions, a.blobs, a.root} { + if r != nil { + err = errors.Join(err, r.Close()) + } + } + return err +} + +// Put streams bytes into an archive-owned file; it never keeps a source path or +// signed URL as the recovery dependency. Publishing uses no-overwrite links. +func (a *Archive) Put(r io.Reader) (Part, error) { + if r == nil { + return Part{}, errors.New("nil blob reader") + } + name, f, err := temporary(a.blobs) + if err != nil { + return Part{}, err + } + defer a.blobs.Remove(name) + defer f.Close() + h := sha256.New() + n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(r, a.maxBlobBytes+1)) + if err != nil { + return Part{}, errors.New("read snapshot blob") + } + if n > a.maxBlobBytes { + return Part{}, errors.New("snapshot blob exceeds configured limit") + } + if err = f.Sync(); err != nil { + return Part{}, err + } + if err = f.Close(); err != nil { + return Part{}, err + } + p := Part{SHA256: hex.EncodeToString(h.Sum(nil)), Bytes: n} + if err = a.blobs.Link(name, p.SHA256); err != nil && !errors.Is(err, os.ErrExist) { + return Part{}, err + } + if err = a.verify(p); err != nil { + return Part{}, err + } + if err = syncRoot(a.blobs); err != nil { + return Part{}, err + } + return p, nil +} + +// Seal is the commit point: no version is visible until every dependency has +// been read and hashed locally. Empty tables are explicit, not "leave old data". +func (a *Archive) Seal(m Manifest) (string, error) { + data, err := encodeManifest(m) + if err != nil { + return "", err + } + if err = a.verifyParts(m.Parts); err != nil { + return "", err + } + id := digest(data) + name, f, err := temporary(a.versions) + if err != nil { + return "", err + } + defer a.versions.Remove(name) + defer f.Close() + if _, err = f.Write(data); err != nil { + return "", err + } + if err = f.Sync(); err != nil { + return "", err + } + if err = f.Close(); err != nil { + return "", err + } + if err = a.versions.Link(name, id+".json"); err != nil && !errors.Is(err, os.ErrExist) { + return "", err + } + if _, err = a.Load(id); err != nil { + return "", err + } + if err = syncRoot(a.versions); err != nil { + return "", err + } + return id, nil +} + +// Load revalidates bytes, not just the manifest's claim of recoverability. +func (a *Archive) Load(id string) (Manifest, error) { + if !validDigest(id) { + return Manifest{}, errors.New("invalid snapshot ID") + } + f, err := regular(a.versions, id+".json", maxManifestBytes) + if err != nil { + return Manifest{}, err + } + defer f.Close() + data, err := io.ReadAll(io.LimitReader(f, maxManifestBytes+1)) + if err != nil || len(data) > maxManifestBytes || digest(data) != id { + return Manifest{}, errors.New("snapshot manifest digest mismatch") + } + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + var m Manifest + if err = dec.Decode(&m); err != nil { + return Manifest{}, errors.New("invalid snapshot manifest") + } + var extra any + if dec.Decode(&extra) != io.EOF { + return Manifest{}, errors.New("trailing snapshot manifest data") + } + if _, err = encodeManifest(m); err != nil { + return Manifest{}, err + } + if err = a.verifyParts(m.Parts); err != nil { + return Manifest{}, err + } + return m, nil +} + +func (a *Archive) verifyParts(parts []Part) error { + seen := map[string]int64{} + counts := map[string]int64{} + resourceIDs := map[string]bool{} + archivedResources := map[string]bool{} + for _, p := range parts { + if size, ok := seen[p.SHA256]; ok { + if size != p.Bytes { + return errors.New("conflicting blob sizes") + } + } else { + if err := a.verify(p); err != nil { + return err + } + seen[p.SHA256] = p.Bytes + } + if p.Kind == "resource" { + archivedResources[p.Name] = true + continue + } + count, known := counts[p.SHA256] + if !known || p.Kind == "database" && p.Name == "resources" { + var err error + count, err = a.countRows(p, resourceIDs) + if err != nil { + return err + } + counts[p.SHA256] = count + } + if count != p.Count { + return errors.New("snapshot count does not match archived rows") + } + } + if len(resourceIDs) != len(archivedResources) { + return errors.New("resource archive dependencies incomplete") + } + for id := range resourceIDs { + if !archivedResources[id] { + return errors.New("resource bytes not archived") + } + } + return nil +} + +func (a *Archive) countRows(p Part, resources map[string]bool) (int64, error) { + f, err := regular(a.blobs, p.SHA256, a.maxBlobBytes) + if err != nil { + return 0, err + } + defer f.Close() + dec := json.NewDecoder(io.LimitReader(f, a.maxBlobBytes+1)) + token, err := dec.Token() + if err != nil || token != json.Delim('[') { + return 0, errors.New("snapshot component must be a JSON row array") + } + var n int64 + for dec.More() { + var row map[string]json.RawMessage + if err = dec.Decode(&row); err != nil || row == nil { + return 0, errors.New("invalid archived row") + } + n++ + if p.Kind == "database" && p.Name == "resources" { + var id string + if json.Unmarshal(row["id"], &id) != nil || !identifier(id) || resources[id] { + return 0, errors.New("invalid resource archive identity") + } + resources[id] = true + } + } + if _, err = dec.Token(); err != nil { + return 0, errors.New("incomplete archived row array") + } + var extra any + if dec.Decode(&extra) != io.EOF { + return 0, errors.New("trailing archived row data") + } + return n, nil +} +func (a *Archive) verify(p Part) error { + if !validDigest(p.SHA256) || p.Bytes < 0 || p.Bytes > a.maxBlobBytes { + return errors.New("invalid snapshot blob descriptor") + } + f, err := regular(a.blobs, p.SHA256, a.maxBlobBytes) + if err != nil { + return err + } + defer f.Close() + h := sha256.New() + n, err := io.Copy(h, io.LimitReader(f, a.maxBlobBytes+1)) + if err != nil || n != p.Bytes || hex.EncodeToString(h.Sum(nil)) != p.SHA256 { + return errors.New("snapshot blob integrity mismatch") + } + return nil +} +func regular(root *os.Root, name string, limit int64) (*os.File, error) { + f, err := root.OpenFile(name, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0) + if err != nil { + return nil, errors.New("snapshot file unavailable") + } + st, err := f.Stat() + if err != nil || !st.Mode().IsRegular() || st.Mode().Perm() != 0600 || st.Size() > limit { + f.Close() + return nil, errors.New("unsafe snapshot file") + } + return f, nil +} +func temporary(root *os.Root) (string, *os.File, error) { + var random [16]byte + if _, err := rand.Read(random[:]); err != nil { + return "", nil, err + } + name := ".pending-" + hex.EncodeToString(random[:]) + f, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600) + return name, f, err +} +func syncRoot(root *os.Root) error { + f, err := root.Open(".") + if err != nil { + return err + } + defer f.Close() + return f.Sync() +} +func digest(data []byte) string { h := sha256.Sum256(data); return hex.EncodeToString(h[:]) } +func validDigest(s string) bool { + if len(s) != 64 { + return false + } + for _, c := range s { + if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') { + return false + } + } + return true +} +func identifier(s string) bool { + if len(s) == 0 || len(s) > 128 { + return false + } + for _, c := range s { + if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c == '-' || c == '_' || c == '.') { + return false + } + } + return s != "." && s != ".." +} + +func encodeManifest(m Manifest) ([]byte, error) { + c := m.Compatibility + if c.WeKnoraVersion != releasebundle.WeKnoraVersionV071 || c.SchemaVersion != releasebundle.WeKnoraMigrationV071 || c.RetrieverEngine != "elasticsearch_v8" || !identifier(c.EmbeddingModel) || c.EmbeddingDimension != releasebundle.EmbeddingDimensionV071 || !validDigest(c.IndexMappingSHA256) { + return nil, errors.New("unsupported snapshot compatibility contract") + } + if m.Format != Format || (m.Scope != "internal" && m.Scope != "external") || m.TenantID == 0 || !identifier(m.KnowledgeBaseID) || m.CreatedAt.IsZero() { + return nil, errors.New("invalid library manifest identity") + } + required := map[string]bool{} + for _, n := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} { + required["database/"+n] = true + } + for _, n := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} { + required["wiki/"+n] = true + } + required["index/chunks"] = true + seen := map[string]bool{} + for _, p := range m.Parts { + key := p.Kind + "/" + p.Name + if (!required[key] && !(p.Kind == "resource" && identifier(p.Name) && p.Count == 1)) || seen[key] || !validDigest(p.SHA256) || p.Bytes < 0 || p.Count < 0 { + return nil, errors.New("invalid or nonknowledge snapshot component") + } + seen[key] = true + } + for key := range required { + if !seen[key] { + return nil, errors.New("incomplete library snapshot") + } + } + m.Parts = append([]Part(nil), m.Parts...) + sort.Slice(m.Parts, func(i, j int) bool { return m.Parts[i].Kind+"/"+m.Parts[i].Name < m.Parts[j].Kind+"/"+m.Parts[j].Name }) + data, err := json.Marshal(m) + if err != nil || len(data) > maxManifestBytes { + return nil, errors.New("snapshot manifest too large") + } + return data, nil +} diff --git a/internal/librarysnapshot/archive_test.go b/internal/librarysnapshot/archive_test.go new file mode 100644 index 0000000..3c927f9 --- /dev/null +++ b/internal/librarysnapshot/archive_test.go @@ -0,0 +1,151 @@ +package librarysnapshot + +import ( + "bytes" + "os" + "path/filepath" + "testing" + "time" +) + +func TestArchiveSealsCompleteVersionAndLoadsAfterReopen(t *testing.T) { + root := t.TempDir() + if err := os.Chmod(root, 0700); err != nil { + t.Fatal(err) + } + a, err := Open(root, 1024*1024) + if err != nil { + t.Fatal(err) + } + part, err := a.Put(bytes.NewBufferString("[]")) + if err != nil { + t.Fatal(err) + } + m := testManifest(part) + id, err := a.Seal(m) + if err != nil { + t.Fatal(err) + } + if err = a.Close(); err != nil { + t.Fatal(err) + } + a, err = Open(root, 1024*1024) + if err != nil { + t.Fatal(err) + } + defer a.Close() + got, err := a.Load(id) + if err != nil { + t.Fatal(err) + } + if len(got.Parts) != 11 || got.KnowledgeBaseID != m.KnowledgeBaseID { + t.Fatal("incomplete reopened manifest") + } + again, err := a.Seal(m) + if err != nil || again != id { + t.Fatal("retry changed immutable identity") + } + files, err := os.ReadDir(filepath.Join(root, "blobs")) + if err != nil || len(files) != 1 { + t.Fatal("identical bytes were not shared") + } +} + +func TestArchiveRejectsMissingCorruptAndNonknowledgeParts(t *testing.T) { + root := t.TempDir() + _ = os.Chmod(root, 0700) + a, err := Open(root, 1024) + if err != nil { + t.Fatal(err) + } + defer a.Close() + p, err := a.Put(bytes.NewBufferString("[]")) + if err != nil { + t.Fatal(err) + } + for _, test := range []struct { + name string + mutate func(*Manifest) + }{ + {"unsupported schema", func(m *Manifest) { m.Compatibility.SchemaVersion = 75 }}, + {"missing mapping contract", func(m *Manifest) { m.Compatibility.IndexMappingSHA256 = "" }}, + {"missing Wiki", func(m *Manifest) { m.Parts = m.Parts[:6] }}, + {"duplicate table", func(m *Manifest) { m.Parts = append(m.Parts, m.Parts[0]) }}, + {"account table", func(m *Manifest) { m.Parts[0].Name = "users" }}, + {"bad scope", func(m *Manifest) { m.Scope = "both" }}, + {"wrong size", func(m *Manifest) { m.Parts[0].Bytes++ }}, + {"missing blob", func(m *Manifest) { m.Parts[0].SHA256 = string(bytes.Repeat([]byte("a"), 64)) }}, + {"path traversal", func(m *Manifest) { m.Parts[0].SHA256 = "../../escape" }}, + {"negative count", func(m *Manifest) { m.Parts[0].Count = -1 }}, + } { + t.Run(test.name, func(t *testing.T) { + m := testManifest(p) + test.mutate(&m) + if _, err := a.Seal(m); err == nil { + t.Fatal("unsafe manifest sealed") + } + }) + } + m := testManifest(p) + id, err := a.Seal(m) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "blobs", p.SHA256), []byte("xx"), 0600); err != nil { + t.Fatal(err) + } + if _, err := a.Load(id); err == nil { + t.Fatal("corrupt blob accepted as recoverable") + } +} + +func TestArchiveRefusesInventedCountsAndMissingResourceBytes(t *testing.T) { + root := t.TempDir() + _ = os.Chmod(root, 0700) + a, err := Open(root, 1024) + if err != nil { + t.Fatal(err) + } + defer a.Close() + p, err := a.Put(bytes.NewBufferString("[]")) + if err != nil { + t.Fatal(err) + } + m := testManifest(p) + m.Parts[0].Count = 2 + if _, err = a.Seal(m); err == nil { + t.Fatal("manifest accepted invented progress total") + } + p, err = a.Put(bytes.NewBufferString(`[{"id":"res-safe"}]`)) + if err != nil { + t.Fatal(err) + } + m = testManifest(p) + for i := range m.Parts { + m.Parts[i].Count = 1 + } + if _, err = a.Seal(m); err == nil { + t.Fatal("resource rows have no archived resource bytes") + } +} + +func testManifest(part Part) Manifest { + m := Manifest{Compatibility: Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "fixture-model", EmbeddingDimension: 1024, IndexMappingSHA256: digest([]byte("fixture-mapping"))}, Format: "weknora-library-snapshot/v1", Scope: "internal", TenantID: 10000, KnowledgeBaseID: "kb-safe", CreatedAt: time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)} + for _, name := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} { + p := part + p.Kind = "database" + p.Name = name + m.Parts = append(m.Parts, p) + } + for _, name := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} { + p := part + p.Kind = "wiki" + p.Name = name + m.Parts = append(m.Parts, p) + } + p := part + p.Kind = "index" + p.Name = "chunks" + m.Parts = append(m.Parts, p) + return m +} diff --git a/internal/librarysnapshot/usage.go b/internal/librarysnapshot/usage.go new file mode 100644 index 0000000..d8f1660 --- /dev/null +++ b/internal/librarysnapshot/usage.go @@ -0,0 +1,124 @@ +package librarysnapshot + +import ( + "crypto/sha256" + "encoding/hex" + "errors" + "io" + "math" + "slices" + "sort" + "strings" +) + +type Usage struct { + VersionID string `json:"version_id"` + LogicalBytes int64 `json:"logical_bytes"` + UniqueBytes int64 `json:"unique_bytes"` + SharedBytes int64 `json:"shared_bytes"` + ExclusiveBytes int64 `json:"exclusive_bytes"` +} + +// OpenPart provides archive-owned recovery bytes, never a source-side locator. +// Verify and rewind the same descriptor, so path replacement cannot swap the +// verified file for an unchecked dependency between these two operations. +func (a *Archive) OpenPart(p Part) (io.ReadCloser, error) { + if !validDigest(p.SHA256) || p.Bytes < 0 || p.Bytes > a.maxBlobBytes { + return nil, errors.New("invalid recovery blob") + } + f, err := regular(a.blobs, p.SHA256, a.maxBlobBytes) + if err != nil { + return nil, err + } + h := sha256.New() + n, err := io.Copy(h, io.LimitReader(f, a.maxBlobBytes+1)) + if err != nil || n != p.Bytes || hex.EncodeToString(h.Sum(nil)) != p.SHA256 { + f.Close() + return nil, errors.New("corrupt recovery blob") + } + if _, err = f.Seek(0, io.SeekStart); err != nil { + f.Close() + return nil, err + } + return f, nil +} + +// Usage reports content bytes, not filesystem allocation or reclaimed space. +// It is deliberately read-only: protection/retention and live-job references +// must be acquired from the coordinator before any future cleanup operation. +func (a *Archive) Usage() ([]Usage, error) { + ids, err := a.versionIDs() + if err != nil { + return nil, err + } + type inventory struct { + usage Usage + blobs map[string]int64 + } + inventories := make([]inventory, 0, len(ids)) + refs := map[string]int{} + for _, id := range ids { + m, err := a.Load(id) + if err != nil { + return nil, err + } + v := inventory{usage: Usage{VersionID: id}, blobs: map[string]int64{}} + for _, p := range m.Parts { + if v.usage.LogicalBytes > math.MaxInt64-p.Bytes { + return nil, errors.New("snapshot usage overflow") + } + v.usage.LogicalBytes += p.Bytes + v.blobs[p.SHA256] = p.Bytes + } + for sha := range v.blobs { + refs[sha]++ + } + inventories = append(inventories, v) + } + after, err := a.versionIDs() + if err != nil { + return nil, err + } + if !slices.Equal(ids, after) { + return nil, errors.New("snapshot inventory changed; retry usage read") + } + result := make([]Usage, 0, len(inventories)) + for _, v := range inventories { + for sha, size := range v.blobs { + v.usage.UniqueBytes += size + if refs[sha] > 1 { + v.usage.SharedBytes += size + } else { + v.usage.ExclusiveBytes += size + } + } + result = append(result, v.usage) + } + return result, nil +} + +func (a *Archive) versionIDs() ([]string, error) { + f, err := a.versions.Open(".") + if err != nil { + return nil, err + } + defer f.Close() + entries, err := f.ReadDir(-1) + if err != nil { + return nil, err + } + ids := make([]string, 0, len(entries)) + for _, entry := range entries { + if strings.HasPrefix(entry.Name(), ".pending-") { + continue + } + name := entry.Name() + id := strings.TrimSuffix(name, ".json") + if !strings.HasSuffix(name, ".json") || !validDigest(id) || entry.IsDir() { + return nil, errors.New("unexpected archive version entry") + } + ids = append(ids, id) + } + sort.Strings(ids) + return ids, nil +} diff --git a/internal/librarysnapshot/usage_test.go b/internal/librarysnapshot/usage_test.go new file mode 100644 index 0000000..bdccfba --- /dev/null +++ b/internal/librarysnapshot/usage_test.go @@ -0,0 +1,138 @@ +package librarysnapshot + +import ( + "bytes" + "io" + "os" + "path/filepath" + "sync" + "testing" + "time" +) + +func TestArchiveRecoveryIndependentOfSourceAndUsageByReferences(t *testing.T) { + root := t.TempDir() + _ = os.Chmod(root, 0700) + a, err := Open(root, 1024) + if err != nil { + t.Fatal(err) + } + defer a.Close() + source := filepath.Join(t.TempDir(), "source") + if err := os.WriteFile(source, []byte("first"), 0600); err != nil { + t.Fatal(err) + } + f, err := os.Open(source) + if err != nil { + t.Fatal(err) + } + blob, err := a.Put(f) + f.Close() + if err != nil { + t.Fatal(err) + } + if err = os.Remove(source); err != nil { + t.Fatal(err) + } + reader, err := a.OpenPart(blob) + if err != nil { + t.Fatal(err) + } + restored, err := io.ReadAll(reader) + reader.Close() + if err != nil || string(restored) != "first" { + t.Fatal("restore still depends on source") + } + empty, err := a.Put(bytes.NewBufferString("[]")) + if err != nil { + t.Fatal(err) + } + m := testManifest(empty) + first, err := a.Seal(m) + if err != nil { + t.Fatal(err) + } + m.CreatedAt = m.CreatedAt.Add(time.Second) + second, err := a.Seal(m) + if err != nil { + t.Fatal(err) + } + got, err := a.Usage() + if err != nil { + t.Fatal(err) + } + if len(got) != 2 { + t.Fatalf("versions=%d", len(got)) + } + for _, u := range got { + if u.VersionID != first && u.VersionID != second { + t.Fatal("unexpected version") + } + if u.LogicalBytes != 22 || u.UniqueBytes != 2 || u.SharedBytes != 2 || u.ExclusiveBytes != 0 { + t.Fatalf("bad shared accounting: %+v", u) + } + } + // A digest with no manifest reference must not inflate either version. + if len(restored) != 5 { + t.Fatal("fixture unexpected") + } +} + +func TestArchiveConcurrentWritersAndReadSafety(t *testing.T) { + root := t.TempDir() + _ = os.Chmod(root, 0700) + a, err := Open(root, 16) + if err != nil { + t.Fatal(err) + } + defer a.Close() + b, err := Open(root, 16) + if err != nil { + t.Fatal(err) + } + defer b.Close() + var wg sync.WaitGroup + for i := 0; i < 12; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + archive := a + if i%2 == 0 { + archive = b + } + p, err := archive.Put(bytes.NewBufferString("[]")) + if err != nil { + t.Error(err) + return + } + if _, err = archive.Seal(testManifest(p)); err != nil { + t.Error(err) + } + }(i) + } + wg.Wait() + got, err := a.Usage() + if err != nil || len(got) != 1 { + t.Fatalf("concurrent publication: err=%v versions=%d", err, len(got)) + } + if _, err = a.Put(bytes.NewBufferString("this is more than sixteen bytes")); err == nil { + t.Fatal("oversized blob accepted") + } + p, err := a.Put(bytes.NewBufferString("[]")) + if err != nil { + t.Fatal(err) + } + target := filepath.Join(root, "blobs", p.SHA256) + if err = os.Remove(target); err != nil { + t.Fatal(err) + } + if err = os.Symlink(filepath.Join(t.TempDir(), "outside"), target); err != nil { + t.Fatal(err) + } + if _, err = a.OpenPart(p); err == nil { + t.Fatal("symlink followed") + } + if _, err = a.Usage(); err == nil { + t.Fatal("corrupt dependency reported as safe usage") + } +} diff --git a/internal/releaseadapter/v071/library_archive.go b/internal/releaseadapter/v071/library_archive.go new file mode 100644 index 0000000..6f94dd5 --- /dev/null +++ b/internal/releaseadapter/v071/library_archive.go @@ -0,0 +1,238 @@ +package v071 + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "strings" + "time" + + "knowledge-governance/internal/librarysnapshot" +) + +// ArchiveLibraryCandidate persists independent recovery bytes. "Candidate" is +// intentional: matching IDs/counts does not prove shared Wiki semantic retraction +// or a common cross-system cut. No release worker consumes it as authorization. +func ArchiveLibraryCandidate(ctx context.Context, archive *librarysnapshot.Archive, scope string, identity PostgresIdentity, contract librarysnapshot.Compatibility, pg PostgresSnapshot, es SearchSnapshot, objects ObjectReader, createdAt time.Time) (string, error) { + if archive == nil || objects == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) || identity.KnowledgeID != "" { + return "", errors.New("invalid library candidate configuration") + } + if err := validateLibraryCandidate(identity, pg, es); err != nil { + return "", err + } + if es.MappingSHA256 != contract.IndexMappingSHA256 { + return "", errors.New("library mapping contract mismatch") + } + manifest := librarysnapshot.Manifest{Format: librarysnapshot.Format, Scope: scope, TenantID: identity.TenantID, KnowledgeBaseID: identity.KnowledgeBaseID, CreatedAt: createdAt, Compatibility: contract} + put := func(kind, name string, count int64, data []byte) error { + if err := ctx.Err(); err != nil { + return err + } + p, err := archive.Put(bytes.NewReader(data)) + if err != nil { + return err + } + p.Kind = kind + p.Name = name + p.Count = count + manifest.Parts = append(manifest.Parts, p) + return nil + } + for _, table := range pg.Tables { + rows := table.Rows + if rows == nil { + rows = []json.RawMessage{} + } + data, err := json.Marshal(rows) + if err != nil { + return "", errors.New("encode library rows") + } + kind := "database" + if strings.HasPrefix(table.Name, "wiki_") { + kind = "wiki" + } + if err = put(kind, table.Name, int64(len(rows)), data); err != nil { + return "", err + } + } + documents := es.Documents + if documents == nil { + documents = []SearchDocument{} + } + data, err := json.Marshal(documents) + if err != nil { + return "", errors.New("encode library index") + } + if err = put("index", "chunks", int64(len(documents)), data); err != nil { + return "", err + } + for _, raw := range pg.Table("resources").Rows { + if err = ctx.Err(); err != nil { + return "", err + } + var resource struct { + ID string `json:"id"` + Provider string `json:"provider"` + PhysicalPath string `json:"physical_path"` + Size *int64 `json:"size"` + ContentHash string `json:"content_hash"` + } + if json.Unmarshal(raw, &resource) != nil || resource.Size == nil || *resource.Size < 0 || !validObjectProvider(resource.Provider) || !validObjectKey(resource.PhysicalPath) || resource.ContentHash != "" && !validLowerSHA256(resource.ContentHash) { + return "", errors.New("invalid library resource metadata") + } + reader, err := objects.Open(ctx, resource.Provider, resource.PhysicalPath) + if err != nil { + return "", errors.New("read library resource") + } + part, readErr := archive.Put(reader) + closeErr := reader.Close() + if readErr != nil || closeErr != nil { + return "", errors.New("archive library resource") + } + if part.Bytes != *resource.Size || resource.ContentHash != "" && part.SHA256 != resource.ContentHash { + return "", errors.New("library resource integrity mismatch") + } + part.Kind = "resource" + part.Name = resource.ID + part.Count = 1 + manifest.Parts = append(manifest.Parts, part) + } + if err = ctx.Err(); err != nil { + return "", err + } + return archive.Seal(manifest) +} + +type libraryRow struct { + ID string `json:"id"` + TenantID uint64 `json:"tenant_id"` + KnowledgeBaseID string `json:"knowledge_base_id"` + KnowledgeID string `json:"knowledge_id"` + ChunkID string `json:"chunk_id"` + TagID string `json:"tag_id"` + ResourceID string `json:"resource_id"` + Handle string `json:"handle"` + OwnerType string `json:"owner_type"` + OwnerID string `json:"owner_id"` + Content string `json:"content"` +} + +func validateLibraryCandidate(identity PostgresIdentity, pg PostgresSnapshot, es SearchSnapshot) error { + required := map[string]bool{"resources": true} + for _, entry := range libraryContentQueries { + required[entry.name] = true + } + tables := map[string]map[string]libraryRow{} + for _, table := range pg.Tables { + if !required[table.Name] || tables[table.Name] != nil || table.RowCount != len(table.Rows) || table.SHA256 != newTableSnapshot(table.Name, table.Rows).SHA256 { + return errors.New("invalid library table inventory") + } + rows := map[string]libraryRow{} + tables[table.Name] = rows + for _, raw := range table.Rows { + // Log identities are BIGSERIAL. All other owned rows use string IDs. + if table.Name == "wiki_log_entries" { + var row struct { + ID json.Number `json:"id"` + TenantID uint64 `json:"tenant_id"` + KnowledgeBaseID string `json:"knowledge_base_id"` + } + if json.Unmarshal(raw, &row) != nil || row.ID == "" || row.TenantID != identity.TenantID || row.KnowledgeBaseID != identity.KnowledgeBaseID { + return errors.New("invalid Wiki log scope") + } + if _, ok := rows[string(row.ID)]; ok { + return errors.New("duplicate Wiki log identity") + } + rows[string(row.ID)] = libraryRow{} + continue + } + var row libraryRow + if json.Unmarshal(raw, &row) != nil { + return errors.New("invalid library content row") + } + if table.Name == "knowledge_tag_relations" { + if !validSearchIdentifier(row.KnowledgeID) || !validSearchIdentifier(row.TagID) { + return errors.New("invalid library tag relation") + } + row.ID = row.KnowledgeID + "/" + row.TagID + } else { + if !validSearchIdentifier(row.ID) || row.TenantID != identity.TenantID { + return errors.New("invalid library row identity") + } + if table.Name != "resources" && table.Name != "resource_bindings" && row.KnowledgeBaseID != identity.KnowledgeBaseID { + return errors.New("cross-scope library row") + } + } + if _, ok := rows[row.ID]; ok { + return errors.New("duplicate library row") + } + rows[row.ID] = row + } + } + for name := range required { + if tables[name] == nil { + return errors.New("missing library table") + } + } + for _, chunk := range tables["chunks"] { + if _, ok := tables["knowledges"][chunk.KnowledgeID]; !ok { + return errors.New("orphan library chunk") + } + } + for _, relation := range tables["knowledge_tag_relations"] { + _, hasKnowledge := tables["knowledges"][relation.KnowledgeID] + _, hasTag := tables["knowledge_tags"][relation.TagID] + if !hasKnowledge || !hasTag { + return errors.New("orphan library tag relation") + } + } + for _, binding := range tables["resource_bindings"] { + _, hasResource := tables["resources"][binding.ResourceID] + hasOwner := false + if binding.OwnerType == "knowledge" { + _, hasOwner = tables["knowledges"][binding.OwnerID] + } else if binding.OwnerType == "chunk" { + _, hasOwner = tables["chunks"][binding.OwnerID] + } + if !hasResource || !hasOwner { + return errors.New("orphan library resource binding") + } + } + handles := map[string]bool{} + for _, resource := range tables["resources"] { + if len(resource.Handle) != 22 || handles[resource.Handle] { + return errors.New("invalid resource handle") + } + handles[resource.Handle] = true + } + var bodies []json.RawMessage + for _, name := range []string{"knowledges", "chunks", "wiki_pages"} { + bodies = append(bodies, pg.Table(name).Rows...) + } + for _, handle := range referencedResourceHandles(bodies) { + if !handles[handle] { + return errors.New("unresolved library resource") + } + } + sha, err := searchDocumentsDigest(es.Documents) + if err != nil || !validLowerSHA256(es.MappingSHA256) || sha != es.DocumentsSHA256 { + return errors.New("invalid library index digest") + } + seen := map[string]bool{} + for _, document := range es.Documents { + var row libraryRow + if json.Unmarshal(document.Source, &row) != nil || !validSearchIdentifier(document.ID) || seen[document.ID] { + return errors.New("invalid library search document") + } + seen[document.ID] = true + chunk, ok := tables["chunks"][row.ChunkID] + if !ok || row.KnowledgeID != chunk.KnowledgeID || row.Content != chunk.Content { + return errors.New("library index does not match database chunks") + } + if !searchHitMatches(document.Source, SearchIdentity{TenantID: identity.TenantID, KnowledgeBaseID: identity.KnowledgeBaseID, KnowledgeID: row.KnowledgeID}) { + return errors.New("cross-scope library index") + } + } + return nil +} diff --git a/internal/releaseadapter/v071/library_archive_test.go b/internal/releaseadapter/v071/library_archive_test.go new file mode 100644 index 0000000..2400d34 --- /dev/null +++ b/internal/releaseadapter/v071/library_archive_test.go @@ -0,0 +1,96 @@ +package v071 + +import ( + "bytes" + "context" + "encoding/json" + "io" + "os" + "testing" + "time" + + "knowledge-governance/internal/librarysnapshot" +) + +func TestArchiveLibraryCandidateOwnsEveryLaneAndRejectsMismatch(t *testing.T) { + for _, mode := range []string{"complete", "missing-resource", "bad-resource-hash", "cross-scope", "orphan-search", "missing-wiki", "search-digest", "duplicate-table"} { + t.Run(mode, func(t *testing.T) { + root := t.TempDir() + _ = os.Chmod(root, 0700) + archive, err := librarysnapshot.Open(root, 1024*1024) + if err != nil { + t.Fatal(err) + } + defer archive.Close() + pg, es, reader := libraryCandidateFixture() + switch mode { + case "missing-resource": + reader.data = nil + case "bad-resource-hash": + reader.data = []byte("wrong") + case "cross-scope": + pg.Tables[0] = newTableSnapshot("knowledges", []json.RawMessage{json.RawMessage(`{"id":"doc-safe","tenant_id":20000,"knowledge_base_id":"kb-safe","parse_status":"completed"}`)}) + case "orphan-search": + es.Documents[0].Source = json.RawMessage(`{"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe","chunk_id":"unknown"}`) + es.DocumentsSHA256, _ = searchDocumentsDigest(es.Documents) + case "missing-wiki": + pg.Tables = pg.Tables[:6] + case "search-digest": + es.DocumentsSHA256 = shaBytes([]byte("wrong")) + case "duplicate-table": + pg.Tables = append(pg.Tables, pg.Tables[0]) + } + contract := librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "fixture-model", EmbeddingDimension: 1024, IndexMappingSHA256: es.MappingSHA256} + id, err := ArchiveLibraryCandidate(context.Background(), archive, "internal", PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, contract, pg, es, reader, time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)) + if mode != "complete" { + if err == nil { + t.Fatal("inconsistent candidate archived") + } + return + } + if err != nil { + t.Fatal(err) + } + reader.data = nil + saved, err := archive.Load(id) + if err != nil { + t.Fatal(err) + } + if len(saved.Parts) != 12 { + t.Fatalf("parts=%d", len(saved.Parts)) + } + for _, part := range saved.Parts { + if part.Kind == "resource" { + r, err := archive.OpenPart(part) + if err != nil { + t.Fatal(err) + } + data, err := io.ReadAll(r) + r.Close() + if err != nil || string(data) != "file" { + t.Fatal("resource not independently recoverable") + } + } + } + }) + } +} + +type libraryCandidateReader struct{ data []byte } + +func (r *libraryCandidateReader) Open(context.Context, string, string) (io.ReadCloser, error) { + return io.NopCloser(bytes.NewReader(r.data)), nil +} +func libraryCandidateFixture() (PostgresSnapshot, SearchSnapshot, *libraryCandidateReader) { + tables := []TableSnapshot{ + newTableSnapshot("knowledges", []json.RawMessage{json.RawMessage(`{"id":"doc-safe","tenant_id":10000,"knowledge_base_id":"kb-safe","parse_status":"completed","file_path":"resource://ABCDEFGHIJKLMNOPQRSTUV"}`)}), + newTableSnapshot("chunks", []json.RawMessage{json.RawMessage(`{"id":"chunk-safe","tenant_id":10000,"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe","content":"chunk"}`)}), + newTableSnapshot("resources", []json.RawMessage{json.RawMessage(`{"id":"res-safe","tenant_id":10000,"handle":"ABCDEFGHIJKLMNOPQRSTUV","provider":"oss","physical_path":"oss://fixture/source.txt","size":4}`)}), + } + for _, name := range []string{"resource_bindings", "knowledge_tags", "knowledge_tag_relations", "wiki_folders", "wiki_pages", "wiki_page_issues", "wiki_log_entries"} { + tables = append(tables, newTableSnapshot(name, nil)) + } + es := SearchSnapshot{MappingSHA256: shaBytes([]byte("mapping")), Documents: []SearchDocument{{ID: "es-safe", Source: json.RawMessage(`{"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe","chunk_id":"chunk-safe","content":"chunk"}`)}}} + es.DocumentsSHA256, _ = searchDocumentsDigest(es.Documents) + return PostgresSnapshot{Tables: tables}, es, &libraryCandidateReader{data: []byte("file")} +} diff --git a/internal/releaseadapter/v071/library_capture.go b/internal/releaseadapter/v071/library_capture.go new file mode 100644 index 0000000..636a31f --- /dev/null +++ b/internal/releaseadapter/v071/library_capture.go @@ -0,0 +1,70 @@ +package v071 + +import ( + "context" + "errors" + "time" + + "knowledge-governance/internal/libraryfreeze" + "knowledge-governance/internal/librarysnapshot" +) + +type LibraryCaptureRequest struct { + JobID, Scope string + Identity PostgresIdentity + Compatibility librarysnapshot.Compatibility + CreatedAt time.Time + MaxDocuments int +} + +// CaptureLibrary connects maintenance to the actual PG/PIT/object archive path. +// sourceReady is mandatory and must verify the whole writer group (both scopes, +// ingress and background jobs), not only this KB. It runs before stopping and +// again inside the stopped window. Deployment must establish that the group +// contains every source writer. This function does not certify Wiki semantics, +// approve a release, or give the worker any production credentials. +func CaptureLibrary(ctx context.Context, freeze *libraryfreeze.Coordinator, archive *librarysnapshot.Archive, db librarySnapshotDB, search *SearchExporter, objects ObjectReader, sourceReady func(context.Context) error, req LibraryCaptureRequest) (libraryfreeze.Result, error) { + if freeze == nil || archive == nil || db == nil || search == nil || objects == nil || sourceReady == nil || req.CreatedAt.IsZero() || req.MaxDocuments < 1 || req.Identity.TenantID == 0 || !validSearchIdentifier(req.Identity.KnowledgeBaseID) || req.Identity.KnowledgeID != "" || (req.Scope != "internal" && req.Scope != "external") { + return libraryfreeze.Result{}, errors.New("invalid library capture configuration") + } + result, err := freeze.RunPrepared(ctx, req.JobID, func(ctx context.Context) error { + if err := sourceReady(ctx); err != nil { + return err + } + // Check readiness before a maintenance window, then repeat the export inside + // that window: work can arrive between preflight and the graceful stop. + _, err := ExportLibrary(ctx, db, req.Identity) + return err + }, func(ctx context.Context) (string, error) { + if err := sourceReady(ctx); err != nil { + return "", err + } + pg, err := ExportLibrary(ctx, db, req.Identity) + if err != nil { + return "", err + } + es, err := search.ExportLibrary(ctx, SearchIdentity{TenantID: req.Identity.TenantID, KnowledgeBaseID: req.Identity.KnowledgeBaseID}, req.MaxDocuments) + if err != nil { + return "", err + } + return ArchiveLibraryCandidate(ctx, archive, req.Scope, req.Identity, req.Compatibility, pg, es, objects, req.CreatedAt) + }) + if err != nil { + return result, err + } + // A successful old operation is replayable only while its exact independent + // archive still verifies, and never under another KB/scope/model contract. + saved, err := archive.Load(result.SnapshotID) + if err != nil { + return libraryfreeze.Result{}, errors.New("captured library archive no longer verifies") + } + if saved.Scope != req.Scope || saved.TenantID != req.Identity.TenantID || saved.KnowledgeBaseID != req.Identity.KnowledgeBaseID || saved.Compatibility != req.Compatibility || !saved.CreatedAt.Equal(req.CreatedAt) { + return libraryfreeze.Result{}, errors.New("library capture request conflicts with existing job") + } + for _, part := range saved.Parts { + if part.Kind == "index" && part.Count > int64(req.MaxDocuments) { + return libraryfreeze.Result{}, errors.New("replayed library exceeds requested document limit") + } + } + return result, nil +} diff --git a/internal/releaseadapter/v071/library_capture_test.go b/internal/releaseadapter/v071/library_capture_test.go new file mode 100644 index 0000000..fe96fea --- /dev/null +++ b/internal/releaseadapter/v071/library_capture_test.go @@ -0,0 +1,131 @@ +package v071 + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "os" + "sync/atomic" + "testing" + "time" + + "knowledge-governance/internal/libraryfreeze" + "knowledge-governance/internal/librarysnapshot" +) + +type captureWriter struct { + running atomic.Bool + db *libraryFixture + late bool + stops int +} + +func (c *captureWriter) Inventory(context.Context) ([]libraryfreeze.Service, error) { + r := c.running.Load() + return []libraryfreeze.Service{{ID: "test-writer", Running: r, Healthy: r}}, nil +} +func (c *captureWriter) Stop(context.Context, string) error { + c.stops++ + c.running.Store(false) + if c.late { + c.db.tx.pending = 1 + } + return nil +} +func (c *captureWriter) Start(context.Context, string) error { c.running.Store(true); return nil } +func TestLibraryCaptureMaintenanceOwnsArchiveAndRestoresOnFailure(t *testing.T) { + for _, mode := range []string{"success", "initial-busy", "late-pending", "search-failure"} { + t.Run(mode, func(t *testing.T) { + db := newLibraryFixture() + writer := &captureWriter{db: db, late: mode == "late-pending"} + writer.running.Store(true) + if mode == "initial-busy" { + db.tx.pending = 1 + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/chunks/_mapping": + fmt.Fprint(w, `{"chunks":{"mappings":{"properties":{"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"tenant_id":{"type":"long"}}}}}`) + case "/chunks/_pit": + if writer.running.Load() { + t.Error("PIT opened before writers stopped") + } + fmt.Fprint(w, `{"id":"fixed-pit"}`) + case "/_search": + if mode == "search-failure" { + http.Error(w, "fixture failure", 503) + return + } + fmt.Fprint(w, `{"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"hits":[]}}`) + case "/_pit": + fmt.Fprint(w, `{"succeeded":true}`) + default: + http.Error(w, "unexpected", 400) + } + })) + defer server.Close() + search, err := NewSearchExporter(SearchExporterConfig{BaseURL: server.URL, Index: "chunks", PageSize: 10}) + if err != nil { + t.Fatal(err) + } + mapping, err := search.inspectMapping(context.Background()) + if err != nil { + t.Fatal(err) + } + root := t.TempDir() + _ = os.Chmod(root, 0700) + freeze, err := libraryfreeze.Open(root, "test-source", writer) + if err != nil { + t.Fatal(err) + } + defer freeze.Close() + archiveRoot := t.TempDir() + _ = os.Chmod(archiveRoot, 0700) + archive, err := librarysnapshot.Open(archiveRoot, 1024*1024) + if err != nil { + t.Fatal(err) + } + defer archive.Close() + req := LibraryCaptureRequest{JobID: "capture-job", Scope: "internal", Identity: PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, Compatibility: librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "fixture-model", EmbeddingDimension: 1024, IndexMappingSHA256: mapping.digest}, CreatedAt: time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC), MaxDocuments: 100} + checks := 0 + ready := func(context.Context) error { checks++; return nil } + result, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req) + if !writer.running.Load() { + t.Fatal("test writer left stopped") + } + if mode != "success" { + if err == nil { + t.Fatal("invalid candidate succeeded") + } + if mode == "initial-busy" && writer.stops != 0 { + t.Fatal("busy source stopped") + } + return + } + if err != nil || result.State != "succeeded" { + t.Fatalf("capture=%+v err=%v", result, err) + } + saved, err := archive.Load(result.SnapshotID) + if err != nil || saved.Scope != "internal" || checks != 2 { + t.Fatalf("archive=%+v checks=%d err=%v", saved, checks, err) + } + db.tx.pending = 1 + replay, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req) + if err != nil || replay.SnapshotID != result.SnapshotID || writer.stops != 1 { + t.Fatal("sealed replay consulted current source") + } + req.Scope = "external" + if got, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req); err == nil || got.State == "succeeded" { + t.Fatal("conflicting replay returned a successful candidate") + } + req.Scope = "internal" + if err := os.Remove(archiveRoot + "/versions/" + result.SnapshotID + ".json"); err != nil { + t.Fatal(err) + } + if got, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req); err == nil || got.State == "succeeded" { + t.Fatal("missing archive returned a successful candidate") + } + }) + } +} diff --git a/internal/releaseadapter/v071/library_export.go b/internal/releaseadapter/v071/library_export.go new file mode 100644 index 0000000..7993325 --- /dev/null +++ b/internal/releaseadapter/v071/library_export.go @@ -0,0 +1,118 @@ +package v071 + +import ( + "context" + "encoding/json" + "errors" + "github.com/jackc/pgx/v5" +) + +type librarySnapshotDB interface { + BeginTx(context.Context, pgx.TxOptions) (pgx.Tx, error) +} + +// ExportLibrary reads one PostgreSQL MVCC snapshot, including empty Wiki tables. +// It does not certify semantic quality, freeze Elasticsearch/objects, or authorize +// publication. Those require a separately verified, complete library version. +func ExportLibrary(ctx context.Context, db librarySnapshotDB, identity PostgresIdentity) (PostgresSnapshot, error) { + if db == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) || identity.KnowledgeID != "" { + return PostgresSnapshot{}, errors.New("invalid library snapshot scope") + } + tx, err := db.BeginTx(ctx, pgx.TxOptions{IsoLevel: pgx.RepeatableRead, AccessMode: pgx.ReadOnly}) + if err != nil { + return PostgresSnapshot{}, errors.New("begin library snapshot") + } + defer tx.Rollback(context.WithoutCancel(ctx)) + var version int + var dirty bool + if err := tx.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty { + return PostgresSnapshot{}, errors.New("source library schema contract mismatch") + } + var graph bool + if err := tx.QueryRow(ctx, `SELECT COALESCE((indexing_strategy->>'graph_enabled')::boolean,false) FROM knowledge_bases WHERE tenant_id=$1 AND id=$2 AND deleted_at IS NULL`, identity.TenantID, identity.KnowledgeBaseID).Scan(&graph); err != nil { + return PostgresSnapshot{}, errors.New("source library unavailable") + } + if graph { + return PostgresSnapshot{}, errors.New("Neo4j graph snapshot is not supported") + } + var unsettled, pending, failed int64 + if err := tx.QueryRow(ctx, librarySettlementSQL, identity.TenantID, identity.KnowledgeBaseID).Scan(&unsettled, &pending, &failed); err != nil { + return PostgresSnapshot{}, errors.New("read library settlement") + } + if unsettled != 0 || pending != 0 || failed != 0 { + return PostgresSnapshot{}, errors.New("library has unfinished or failed processing; semantic acceptance is still required after settlement") + } + exporter := PostgresExporter{db: tx} + var snapshot PostgresSnapshot + for _, entry := range libraryContentQueries { + rows, err := exporter.queryRows(ctx, entry.sql, identity.TenantID, identity.KnowledgeBaseID) + if err != nil { + return PostgresSnapshot{}, err + } + snapshot.Tables = append(snapshot.Tables, newTableSnapshot(entry.name, rows)) + } + // Resources can be owned by a document/chunk or only referenced in body text. + // The latter includes Wiki manual pages, which document bundles used to miss. + var bodies []json.RawMessage + for _, name := range []string{"knowledges", "chunks", "wiki_pages"} { + bodies = append(bodies, snapshot.Table(name).Rows...) + } + handles := referencedResourceHandles(bodies) + resources, err := exporter.queryRows(ctx, libraryResourcesSQL, identity.TenantID, identity.KnowledgeBaseID, handles) + if err != nil { + return PostgresSnapshot{}, err + } + resourceIDs, resourceHandles := map[string]bool{}, map[string]bool{} + for _, raw := range resources { + var row struct { + ID string `json:"id"` + Handle string `json:"handle"` + TenantID uint64 `json:"tenant_id"` + } + if json.Unmarshal(raw, &row) != nil || row.ID == "" || row.TenantID != identity.TenantID || resourceIDs[row.ID] || resourceHandles[row.Handle] { + return PostgresSnapshot{}, errors.New("invalid library resource identity") + } + resourceIDs[row.ID] = true + resourceHandles[row.Handle] = true + } + for _, handle := range handles { + if !resourceHandles[handle] { + return PostgresSnapshot{}, errors.New("library references an unavailable resource") + } + } + for _, raw := range snapshot.Table("resource_bindings").Rows { + var row struct { + ResourceID string `json:"resource_id"` + } + if json.Unmarshal(raw, &row) != nil || !resourceIDs[row.ResourceID] { + return PostgresSnapshot{}, errors.New("library binding references an unavailable resource") + } + } + snapshot.Tables = append(snapshot.Tables, newTableSnapshot("resources", resources)) + if err = tx.Commit(ctx); err != nil { + return PostgresSnapshot{}, errors.New("commit library snapshot") + } + return snapshot, nil +} + +// Deleted document retract/finalize tasks are scoped to the KB, not a live +// knowledge row. Never join them to only non-deleted documents. Dead letters +// have no resolution marker in this schema, so presence blocks automatically. +const librarySettlementSQL = `SELECT + (SELECT count(*) FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2 AND deleted_at IS NULL AND (parse_status IS DISTINCT FROM 'completed' OR pending_subtasks_count IS NULL OR pending_subtasks_count<>0)), + (SELECT count(*) FROM task_pending_ops WHERE tenant_id=$1 AND ((scope='knowledge_base' AND scope_id=$2) OR (scope='knowledge' AND scope_id IN (SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2)))), + (SELECT count(*) FROM task_dead_letters WHERE tenant_id=$1 AND ((scope='knowledge_base' AND scope_id=$2) OR (scope='knowledge' AND scope_id IN (SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2))))` + +var libraryContentQueries = []struct{ name, sql string }{ + {"knowledges", `SELECT to_jsonb(k) FROM knowledges k WHERE k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL ORDER BY k.id`}, + {"chunks", `SELECT to_jsonb(c) FROM chunks c JOIN knowledges k ON k.id=c.knowledge_id AND k.tenant_id=c.tenant_id AND k.knowledge_base_id=c.knowledge_base_id WHERE k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL AND c.deleted_at IS NULL ORDER BY c.id`}, + {"knowledge_tags", `SELECT to_jsonb(t) FROM knowledge_tags t WHERE t.tenant_id=$1 AND t.knowledge_base_id=$2 AND t.deleted_at IS NULL ORDER BY t.id`}, + {"knowledge_tag_relations", `SELECT to_jsonb(r) FROM knowledge_tag_relations r JOIN knowledges k ON k.id=r.knowledge_id JOIN knowledge_tags t ON t.id=r.tag_id AND t.tenant_id=k.tenant_id AND t.knowledge_base_id=k.knowledge_base_id WHERE k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL AND t.deleted_at IS NULL ORDER BY r.knowledge_id,r.tag_id`}, + {"resource_bindings", `SELECT to_jsonb(b) FROM resource_bindings b WHERE b.tenant_id=$1 AND ((b.owner_type='knowledge' AND EXISTS (SELECT 1 FROM knowledges k WHERE k.id=b.owner_id AND k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL)) OR (b.owner_type='chunk' AND EXISTS (SELECT 1 FROM chunks c JOIN knowledges k ON k.id=c.knowledge_id AND k.tenant_id=c.tenant_id AND k.knowledge_base_id=c.knowledge_base_id WHERE c.id=b.owner_id AND c.tenant_id=$1 AND c.knowledge_base_id=$2 AND c.deleted_at IS NULL AND k.deleted_at IS NULL))) ORDER BY b.id`}, + {"wiki_pages", `SELECT to_jsonb(w) FROM wiki_pages w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 AND w.deleted_at IS NULL ORDER BY w.id`}, + {"wiki_folders", `SELECT to_jsonb(w) FROM wiki_folders w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 AND w.deleted_at IS NULL ORDER BY w.id`}, + {"wiki_page_issues", `SELECT to_jsonb(w) FROM wiki_page_issues w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 AND w.deleted_at IS NULL ORDER BY w.id`}, + {"wiki_log_entries", `SELECT to_jsonb(w) FROM wiki_log_entries w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 ORDER BY w.id`}, +} + +const libraryResourcesSQL = `SELECT to_jsonb(r) FROM resources r WHERE r.tenant_id=$1 AND r.deleted_at IS NULL AND (r.handle=ANY($3::text[]) OR EXISTS (SELECT 1 FROM resource_bindings b WHERE b.resource_id=r.id AND b.tenant_id=$1 AND ((b.owner_type='knowledge' AND EXISTS (SELECT 1 FROM knowledges k WHERE k.id=b.owner_id AND k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL)) OR (b.owner_type='chunk' AND EXISTS (SELECT 1 FROM chunks c JOIN knowledges k ON k.id=c.knowledge_id AND k.tenant_id=c.tenant_id AND k.knowledge_base_id=c.knowledge_base_id WHERE c.id=b.owner_id AND c.tenant_id=$1 AND c.knowledge_base_id=$2 AND c.deleted_at IS NULL AND k.deleted_at IS NULL))))) ORDER BY r.id` diff --git a/internal/releaseadapter/v071/library_export_integration_test.go b/internal/releaseadapter/v071/library_export_integration_test.go new file mode 100644 index 0000000..cd21f0f --- /dev/null +++ b/internal/releaseadapter/v071/library_export_integration_test.go @@ -0,0 +1,178 @@ +package v071 + +import ( + "context" + "encoding/json" + "os" + "strings" + "testing" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +// This fixture creates only synthetic content in an explicitly named disposable +// database. Never point this test at an application database. +func TestLibraryExportIsolatedPostgres(t *testing.T) { + dsn := os.Getenv("KG_LIBRARY_TEST_DATABASE_URL") + if dsn == "" { + t.Skip("requires explicitly isolated kg_library_test_ PostgreSQL database") + } + cfg, err := pgxpool.ParseConfig(dsn) + if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" { + t.Fatal("isolated database required") + } + ctx := context.Background() + db, err := pgxpool.NewWithConfig(ctx, cfg) + if err != nil { + t.Fatal(err) + } + defer db.Close() + if _, err = db.Exec(ctx, libraryTestSchema); err != nil { + t.Fatal(err) + } + scope := PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"} + run := func() PostgresSnapshot { + t.Helper() + s, e := ExportLibrary(ctx, db, scope) + if e != nil { + t.Fatal(e) + } + return s + } + snap := run() + if snap.Table("knowledges").RowCount != 1 || snap.Table("wiki_pages").RowCount != 1 || snap.Table("resources").RowCount != 1 { + t.Fatal("incorrect scoped export") + } + if snap.Table("wiki_log_entries").RowCount != 1 || snap.Table("wiki_page_issues").RowCount != 1 { + t.Fatal("Wiki associated data lost") + } + for _, tab := range snap.Tables { + for _, raw := range tab.Rows { + if strings.Contains(string(raw), "other-scope") { + t.Fatalf("cross scope leak in %s", tab.Name) + } + } + } + t.Run("MVCC concurrent update", func(t *testing.T) { + wrapper := libraryHookDB{pool: db, hook: func() { + if _, err := db.Exec(ctx, `UPDATE wiki_pages SET content='new-content' WHERE id='wiki-safe'`); err != nil { + t.Fatal(err) + } + }} + old, err := ExportLibrary(ctx, wrapper, scope) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(old.Table("wiki_pages").Rows[0]), "old-content") { + t.Fatal("mixed PostgreSQL versions during export") + } + fresh := run() + if !strings.Contains(string(fresh.Table("wiki_pages").Rows[0]), "new-content") { + t.Fatal("concurrent writer did not actually commit") + } + }) + t.Run("deleted knowledge tasks", func(t *testing.T) { + for _, test := range []struct{ table, task, scope, id string }{ + {"task_pending_ops", "wiki:ingest", "knowledge_base", "kb-safe"}, + {"task_pending_ops", "wiki:finalize", "knowledge_base", "kb-safe"}, + {"task_dead_letters", "wiki:ingest", "knowledge", "deleted-doc"}, + } { + _, err := db.Exec(ctx, `INSERT INTO `+test.table+`(tenant_id,task_type,scope,scope_id) VALUES(10000,$1,$2,$3)`, test.task, test.scope, test.id) + if err != nil { + t.Fatal(err) + } + if _, err = ExportLibrary(ctx, db, scope); err == nil { + t.Fatalf("ignored %s %s", test.table, test.task) + } + if _, err = db.Exec(ctx, `DELETE FROM `+test.table); err != nil { + t.Fatal(err) + } + } + }) + t.Run("dangling resource blocks", func(t *testing.T) { + _, err := db.Exec(ctx, `UPDATE wiki_pages SET content='resource://abcdefghijklmnopqrstuv' WHERE id='wiki-safe'`) + if err != nil { + t.Fatal(err) + } + if _, err = ExportLibrary(ctx, db, scope); err == nil { + t.Fatal("accepted missing Wiki resource dependency") + } + if _, err = db.Exec(ctx, `UPDATE wiki_pages SET content='new-content' WHERE id='wiki-safe'`); err != nil { + t.Fatal(err) + } + }) + t.Run("empty library has explicit empty tables", func(t *testing.T) { + s, err := ExportLibrary(ctx, db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-empty"}) + if err != nil { + t.Fatal(err) + } + if len(s.Tables) != 10 { + t.Fatal("empty library snapshot incomplete") + } + for _, tab := range s.Tables { + if tab.RowCount != 0 { + t.Fatal("empty library leaked content") + } + } + }) + // The exporter may not mutate content: only fixture updates above are visible. + var count int + if err := db.QueryRow(ctx, `SELECT count(*) FROM sessions`).Scan(&count); err != nil || count != 1 { + t.Fatal("nonknowledge data changed") + } + encoded, err := json.Marshal(snap) + if err != nil || strings.Contains(string(encoded), "private-session-marker") { + t.Fatal("nonknowledge data exported") + } +} + +type libraryHookDB struct { + pool *pgxpool.Pool + hook func() +} + +func (d libraryHookDB) BeginTx(ctx context.Context, o pgx.TxOptions) (pgx.Tx, error) { + tx, err := d.pool.BeginTx(ctx, o) + return &libraryHookTx{Tx: tx, hook: d.hook}, err +} + +type libraryHookTx struct { + pgx.Tx + hook func() + fired bool +} + +func (d *libraryHookTx) Query(ctx context.Context, q string, args ...any) (pgx.Rows, error) { + if !d.fired { + d.fired = true + d.hook() + } + return d.Tx.Query(ctx, q, args...) +} + +const libraryTestSchema = ` +CREATE TABLE schema_migrations(version int,dirty bool); INSERT INTO schema_migrations VALUES(74,false); +CREATE TABLE knowledge_bases(id text,tenant_id bigint,indexing_strategy jsonb,deleted_at timestamptz); +INSERT INTO knowledge_bases VALUES('kb-safe',10000,'{}',NULL),('kb-empty',10000,'{}',NULL); +CREATE TABLE knowledges(id text PRIMARY KEY,tenant_id bigint,knowledge_base_id text,parse_status text,pending_subtasks_count int,file_path text,deleted_at timestamptz); +INSERT INTO knowledges VALUES('doc-safe',10000,'kb-safe','completed',0,'resource://ABCDEFGHIJKLMNOPQRSTUV',NULL),('other-scope',20000,'kb-safe','completed',0,'',NULL),('deleted-doc',10000,'kb-safe','completed',0,'',now()); +CREATE TABLE chunks(id text,tenant_id bigint,knowledge_base_id text,knowledge_id text,content text,deleted_at timestamptz); +INSERT INTO chunks VALUES('chunk-safe',10000,'kb-safe','doc-safe','chunk',NULL),('other-scope',20000,'kb-safe','other-scope','secret',NULL); +CREATE TABLE knowledge_tags(id text,tenant_id bigint,knowledge_base_id text,deleted_at timestamptz); +CREATE TABLE knowledge_tag_relations(knowledge_id text,tag_id text); +CREATE TABLE resources(id text,handle text,tenant_id bigint,physical_path text,deleted_at timestamptz); +INSERT INTO resources VALUES('res-safe','ABCDEFGHIJKLMNOPQRSTUV',10000,'source.txt',NULL),('other-scope','abcdefghijklmnopqrstuv',20000,'other-scope',NULL); +CREATE TABLE resource_bindings(id text,tenant_id bigint,resource_id text,owner_type text,owner_id text); +INSERT INTO resource_bindings VALUES('binding-safe',10000,'res-safe','knowledge','doc-safe'); +CREATE TABLE wiki_pages(id text,tenant_id bigint,knowledge_base_id text,slug text,content text,deleted_at timestamptz); +INSERT INTO wiki_pages VALUES('wiki-safe',10000,'kb-safe','index','old-content',NULL),('other-scope',20000,'kb-safe','secret','other-scope',NULL); +CREATE TABLE wiki_folders(id text,tenant_id bigint,knowledge_base_id text,deleted_at timestamptz); +CREATE TABLE wiki_page_issues(id text,tenant_id bigint,knowledge_base_id text,deleted_at timestamptz); +INSERT INTO wiki_page_issues VALUES('issue-safe',10000,'kb-safe',NULL); +CREATE TABLE wiki_log_entries(id bigint,tenant_id bigint,knowledge_base_id text); +INSERT INTO wiki_log_entries VALUES(1,10000,'kb-safe'); +CREATE TABLE task_pending_ops(tenant_id bigint,task_type text,scope text,scope_id text); +CREATE TABLE task_dead_letters(tenant_id bigint,task_type text,scope text,scope_id text); +CREATE TABLE sessions(marker text); INSERT INTO sessions VALUES('private-session-marker'); +` diff --git a/internal/releaseadapter/v071/library_export_test.go b/internal/releaseadapter/v071/library_export_test.go new file mode 100644 index 0000000..b7501c7 --- /dev/null +++ b/internal/releaseadapter/v071/library_export_test.go @@ -0,0 +1,125 @@ +package v071 + +import ( + "context" + "encoding/json" + "errors" + "strings" + "testing" + + "github.com/jackc/pgx/v5" +) + +func TestLibraryExportIncludesEmptyAndCompleteWiki(t *testing.T) { + db := newLibraryFixture() + snapshot, err := ExportLibrary(context.Background(), db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}) + if err != nil { + t.Fatal(err) + } + if len(snapshot.Tables) != 10 { + t.Fatalf("table count=%d, want all ten content tables including empty ones", len(snapshot.Tables)) + } + for _, name := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations", "wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} { + if !tablePresent(snapshot, name) { + t.Fatalf("missing table %s", name) + } + } + if db.options.IsoLevel != pgx.RepeatableRead || db.options.AccessMode != pgx.ReadOnly { + t.Fatalf("unsafe snapshot options: %+v", db.options) + } + if !db.tx.committed { + t.Fatal("snapshot transaction not committed") + } + for _, q := range db.tx.queries { + for _, forbidden := range []string{"from users", "from sessions", "from api_keys", "from messages"} { + if strings.Contains(strings.ToLower(q), forbidden) { + t.Fatalf("nonknowledge export: %s", forbidden) + } + } + } +} + +func TestLibraryExportBlocksUnsettledWorkIncludingDeletedDocumentRetracts(t *testing.T) { + for _, test := range []struct { + name string + setup func(*libraryFixtureTx) + }{ + {"document failed or parsing", func(d *libraryFixtureTx) { d.unsettled = 1 }}, + {"deleted document retract or finalizer", func(d *libraryFixtureTx) { d.pending = 1 }}, + {"dead letter not semantic success", func(d *libraryFixtureTx) { d.failed = 1 }}, + {"unsupported graph data", func(d *libraryFixtureTx) { d.graph = true }}, + } { + t.Run(test.name, func(t *testing.T) { + db := newLibraryFixture() + test.setup(db.tx) + _, err := ExportLibrary(context.Background(), db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}) + if err == nil { + t.Fatal("unsettled library was exported") + } + if db.tx.committed { + t.Fatal("blocked snapshot committed") + } + }) + } +} + +func TestLibraryExportDoesNotReturnPartialSnapshot(t *testing.T) { + db := newLibraryFixture() + db.tx.queryError = true + got, err := ExportLibrary(context.Background(), db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}) + if err == nil || len(got.Tables) != 0 || db.tx.committed { + t.Fatal("partial snapshot escaped") + } +} + +type libraryFixture struct { + options pgx.TxOptions + tx *libraryFixtureTx +} +type libraryFixtureTx struct { + pgx.Tx + queries []string + committed bool + pending, failed, unsettled int64 + graph bool + queryError bool +} + +func newLibraryFixture() *libraryFixture { return &libraryFixture{tx: &libraryFixtureTx{}} } +func (d *libraryFixture) BeginTx(_ context.Context, o pgx.TxOptions) (pgx.Tx, error) { + d.options = o + return d.tx, nil +} +func (d *libraryFixtureTx) Commit(context.Context) error { d.committed = true; return nil } +func (d *libraryFixtureTx) Rollback(context.Context) error { return nil } +func (d *libraryFixtureTx) QueryRow(_ context.Context, q string, _ ...any) pgx.Row { + d.queries = append(d.queries, q) + return libraryFixtureRow{d: d, sql: q} +} +func (d *libraryFixtureTx) Query(_ context.Context, q string, _ ...any) (pgx.Rows, error) { + d.queries = append(d.queries, q) + if d.queryError { + return nil, errors.New("fixture failure") + } + return &jsonRows{values: []json.RawMessage{}}, nil +} + +type libraryFixtureRow struct { + d *libraryFixtureTx + sql string +} + +func (r libraryFixtureRow) Scan(dest ...any) error { + switch { + case strings.Contains(r.sql, "schema_migrations"): + *dest[0].(*int) = 74 + *dest[1].(*bool) = false + case strings.Contains(r.sql, "indexing_strategy"): + *dest[0].(*bool) = r.d.graph + default: + *dest[0].(*int64) = r.d.unsettled + *dest[1].(*int64) = r.d.pending + *dest[2].(*int64) = r.d.failed + } + return nil +} diff --git a/internal/releaseadapter/v071/library_search.go b/internal/releaseadapter/v071/library_search.go new file mode 100644 index 0000000..cfc8c68 --- /dev/null +++ b/internal/releaseadapter/v071/library_search.go @@ -0,0 +1,140 @@ +package v071 + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "sort" + "time" +) + +// ExportLibrary uses one Elasticsearch point-in-time rather than offset paging +// against a changing index. A PIT does not freeze PostgreSQL or object storage; +// the coordinator must still prove cross-lane consistency before publication. +func (e *SearchExporter) ExportLibrary(ctx context.Context, identity SearchIdentity, maxDocuments int) (snapshot SearchSnapshot, resultErr error) { + if identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) || identity.KnowledgeID != "" || maxDocuments < 1 { + return SearchSnapshot{}, errors.New("invalid library search scope or limit") + } + inspection, err := e.inspectMapping(ctx) + if err != nil { + return SearchSnapshot{}, err + } + var opened struct { + ID string `json:"id"` + } + if err = e.requestJSON(ctx, http.MethodPost, "/"+e.index+"/_pit?keep_alive=2m&allow_partial_search_results=false", nil, &opened); err != nil { + return SearchSnapshot{}, err + } + if opened.ID == "" { + return SearchSnapshot{}, errors.New("missing Elasticsearch PIT") + } + pit := opened.ID + defer func() { + cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second) + defer cancel() + var result struct { + Succeeded bool `json:"succeeded"` + } + err := e.requestJSON(cleanup, http.MethodDelete, "/_pit", map[string]any{"id": pit}, &result) + if (err != nil || !result.Succeeded) && resultErr == nil { + snapshot = SearchSnapshot{} + resultErr = errors.New("close library snapshot PIT") + } + }() + filter := []any{map[string]any{"term": map[string]any{inspection.fields.base: identity.KnowledgeBaseID}}} + if inspection.fields.tenant != "" { + filter = append(filter, map[string]any{"term": map[string]any{inspection.fields.tenant: identity.TenantID}}) + } + var cursor json.RawMessage + documents := make([]SearchDocument, 0) + seen := map[string]bool{} + var total *int64 + for { + body := map[string]any{"pit": map[string]any{"id": pit, "keep_alive": "2m"}, "size": e.pageSize, "sort": []any{"_shard_doc"}, "track_total_hits": true, "query": map[string]any{"bool": map[string]any{"filter": filter}}} + if cursor != nil { + body["search_after"] = cursor + } + var page struct { + PITID string `json:"pit_id"` + TimedOut *bool `json:"timed_out"` + Shards struct { + Total int `json:"total"` + Successful int `json:"successful"` + Failed int `json:"failed"` + } `json:"_shards"` + Hits struct { + Total struct { + Value *int64 `json:"value"` + Relation string `json:"relation"` + } `json:"total"` + Hits []struct { + ID string `json:"_id"` + Source json.RawMessage `json:"_source"` + Sort json.RawMessage `json:"sort"` + } `json:"hits"` + } `json:"hits"` + } + if err = e.requestJSON(ctx, http.MethodPost, "/_search?allow_partial_search_results=false", body, &page); err != nil { + return SearchSnapshot{}, err + } + if page.PITID != "" { + pit = page.PITID + } + if page.TimedOut == nil || *page.TimedOut || page.Shards.Total < 1 || page.Shards.Successful != page.Shards.Total || page.Shards.Failed != 0 || page.Hits.Total.Value == nil || page.Hits.Total.Relation != "eq" { + return SearchSnapshot{}, errors.New("incomplete Elasticsearch library snapshot") + } + if total == nil { + total = page.Hits.Total.Value + } + if *total < 0 || *total > int64(maxDocuments) || *total != *page.Hits.Total.Value { + return SearchSnapshot{}, errors.New("Elasticsearch library total changed or exceeds limit") + } + for _, hit := range page.Hits.Hits { + var scope struct { + KnowledgeID string `json:"knowledge_id"` + } + if json.Unmarshal(hit.Source, &scope) != nil || !validSearchIdentifier(scope.KnowledgeID) || !validSearchIdentifier(hit.ID) || seen[hit.ID] { + return SearchSnapshot{}, errors.New("invalid or duplicate library search identity") + } + expected := identity + expected.KnowledgeID = scope.KnowledgeID + if !searchHitMatches(hit.Source, expected) { + return SearchSnapshot{}, errors.New("cross-scope library search document") + } + source, err := canonicalJSON(hit.Source) + if err != nil { + return SearchSnapshot{}, errors.New("invalid library search data") + } + seen[hit.ID] = true + documents = append(documents, SearchDocument{ID: hit.ID, Source: source}) + } + if int64(len(documents)) > *total { + return SearchSnapshot{}, errors.New("library search total mismatch") + } + if len(page.Hits.Hits) > 0 { + next := page.Hits.Hits[len(page.Hits.Hits)-1].Sort + var values []json.RawMessage + if json.Unmarshal(next, &values) != nil || len(values) == 0 || string(next) == string(cursor) { + return SearchSnapshot{}, errors.New("library search cursor did not advance") + } + cursor = next + } + if int64(len(documents)) == *total { + break + } + if len(page.Hits.Hits) == 0 { + return SearchSnapshot{}, errors.New("library search snapshot truncated") + } + } + finalMapping, err := e.inspectMapping(ctx) + if err != nil || finalMapping.digest != inspection.digest { + return SearchSnapshot{}, errors.New("library search mapping changed") + } + sort.Slice(documents, func(i, j int) bool { return documents[i].ID < documents[j].ID }) + sha, err := searchDocumentsDigest(documents) + if err != nil { + return SearchSnapshot{}, err + } + return SearchSnapshot{Documents: documents, MappingSHA256: inspection.digest, DocumentsSHA256: sha}, nil +} diff --git a/internal/releaseadapter/v071/library_search_integration_test.go b/internal/releaseadapter/v071/library_search_integration_test.go new file mode 100644 index 0000000..0ccc118 --- /dev/null +++ b/internal/releaseadapter/v071/library_search_integration_test.go @@ -0,0 +1,81 @@ +package v071 + +import ( + "bytes" + "context" + "crypto/rand" + "encoding/hex" + "fmt" + "net/http" + "os" + "strings" + "testing" + "time" +) + +func TestLibrarySearchIsolatedElasticsearch(t *testing.T) { + base := os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL") + if base == "" { + t.Skip("requires explicit isolated Elasticsearch fixture opt-in") + } + if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" { + t.Fatal("isolated fixture authorization required") + } + var id [8]byte + if _, err := rand.Read(id[:]); err != nil { + t.Fatal(err) + } + index := "kg_library_test_" + hex.EncodeToString(id[:]) + client := &http.Client{Timeout: 20 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} + request := func(method, path, body, contentType string) int { + t.Helper() + r, err := http.NewRequest(method, base+path, strings.NewReader(body)) + if err != nil { + t.Fatal("invalid fixture endpoint") + } + r.Header.Set("Content-Type", contentType) + response, err := client.Do(r) + if err != nil { + t.Fatal("fixture request failed") + } + defer response.Body.Close() + return response.StatusCode + } + if status := request(http.MethodPut, "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"}}}}`, "application/json"); status != 200 { + t.Fatalf("create synthetic index HTTP %d", status) + } + defer func() { + if status := request(http.MethodDelete, "/"+index, "", "application/json"); status != 200 { + t.Errorf("remove fixture index HTTP %d", status) + } + if status := request(http.MethodHead, "/"+index, "", "application/json"); status != 404 { + t.Errorf("fixture index remains HTTP %d", status) + } + }() + var bulk bytes.Buffer + for i := 1; i <= 3; i++ { + baseID := "kb-safe" + if i == 3 { + baseID = "kb-other" + } + fmt.Fprintf(&bulk, "{\"index\":{\"_id\":\"chunk-%d\"}}\n{\"tenant_id\":10000,\"knowledge_base_id\":%q,\"knowledge_id\":\"doc-safe\"}\n", i, baseID) + } + if status := request(http.MethodPost, "/"+index+"/_bulk?refresh=true", bulk.String(), "application/x-ndjson"); status != 200 { + t.Fatalf("fixture bulk HTTP %d", status) + } + exporter, err := NewSearchExporter(SearchExporterConfig{BaseURL: base, Index: index, PageSize: 1, HTTPClient: client}) + if err != nil { + t.Fatal(err) + } + s, err := exporter.ExportLibrary(context.Background(), SearchIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, 10) + if err != nil { + t.Fatal(err) + } + if len(s.Documents) != 2 { + t.Fatalf("documents=%d", len(s.Documents)) + } + empty, err := exporter.ExportLibrary(context.Background(), SearchIdentity{TenantID: 10000, KnowledgeBaseID: "kb-empty"}, 10) + if err != nil || len(empty.Documents) != 0 { + t.Fatalf("empty snapshot err=%v documents=%d", err, len(empty.Documents)) + } +} diff --git a/internal/releaseadapter/v071/library_search_test.go b/internal/releaseadapter/v071/library_search_test.go new file mode 100644 index 0000000..2910ad6 --- /dev/null +++ b/internal/releaseadapter/v071/library_search_test.go @@ -0,0 +1,90 @@ +package v071 + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "testing" +) + +func TestLibrarySearchUsesPITAndChecksExactTotal(t *testing.T) { + for _, mode := range []string{"complete", "partial", "truncated", "wrong-scope", "repeated-cursor", "limit"} { + t.Run(mode, func(t *testing.T) { + searches, closed := 0, false + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + switch { + case r.URL.Path == "/chunks/_mapping": + fmt.Fprint(w, `{"chunks":{"mappings":{"properties":{"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"tenant_id":{"type":"long"}}}}}`) + case r.Method == http.MethodPost && r.URL.Path == "/chunks/_pit": + if r.URL.Query().Get("allow_partial_search_results") != "false" { + t.Error("partial PIT allowed") + } + fmt.Fprint(w, `{"id":"pit-0"}`) + case r.Method == http.MethodPost && r.URL.Path == "/_search": + var body map[string]any + if json.NewDecoder(r.Body).Decode(&body) != nil { + t.Error("bad query") + } + searches++ + if _, ok := body["from"]; ok { + t.Error("offset paging is not a fixed snapshot") + } + pit, _ := body["pit"].(map[string]any) + if pit["id"] != fmt.Sprintf("pit-%d", searches-1) { + t.Error("latest PIT identity not used") + } + if searches > 1 { + if _, ok := body["search_after"]; !ok { + t.Error("missing cursor") + } + } + failed := 0 + if mode == "partial" { + failed = 1 + } + base := "kb-safe" + if mode == "wrong-scope" { + base = "kb-wrong" + } + hits := fmt.Sprintf(`[{"_id":"chunk-%d","_source":{"tenant_id":10000,"knowledge_base_id":%q,"knowledge_id":"doc-safe"},"sort":[%d]}]`, searches, base, searches) + if mode == "truncated" && searches == 2 { + hits = "[]" + } + if mode == "repeated-cursor" && searches == 2 { + hits = `[{"_id":"chunk-2","_source":{"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe"},"sort":[1]}]` + } + fmt.Fprintf(w, `{"pit_id":"pit-%d","timed_out":false,"_shards":{"total":1,"successful":1,"failed":%d},"hits":{"total":{"value":2,"relation":"eq"},"hits":%s}}`, searches, failed, hits) + case r.Method == http.MethodDelete && r.URL.Path == "/_pit": + closed = true + fmt.Fprint(w, `{"succeeded":true,"num_freed":1}`) + default: + t.Errorf("unexpected %s %s", r.Method, r.URL.Path) + http.Error(w, "unexpected", 400) + } + })) + defer server.Close() + exporter, err := NewSearchExporter(SearchExporterConfig{BaseURL: server.URL, Index: "chunks", PageSize: 1}) + if err != nil { + t.Fatal(err) + } + limit := 10 + if mode == "limit" { + limit = 1 + } + snapshot, err := exporter.ExportLibrary(context.Background(), SearchIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, limit) + if mode == "complete" { + if err != nil || len(snapshot.Documents) != 2 { + t.Fatalf("export err=%v documents=%d", err, len(snapshot.Documents)) + } + } else if err == nil { + t.Fatal("invalid snapshot accepted") + } + if !closed { + t.Fatal("PIT leaked") + } + }) + } +} diff --git a/scripts/test-scope.json b/scripts/test-scope.json index 8b76c90..9ed9bc5 100644 --- a/scripts/test-scope.json +++ b/scripts/test-scope.json @@ -9,6 +9,11 @@ "⚠️ 文件缺失或解析不了 ⇒ 门禁红:范围声明不出来时,不许默认放行。" ], "optional": [ + { + "package": "knowledge-governance/internal/libraryfreeze", + "test": "TestFreezeIsolatedDocker", + "reason": "只在显式 KG_FREEZE_DOCKER_ISOLATED=1 下使用本地 Docker 与已缓存 busybox 创建随机标签、无网络/无挂载夹具,验证真实暂停恢复和进程中断;默认 CI 不连接 Docker 或运行中的服务,本地另行实跑" + }, { "package": "knowledge-governance/internal/qaopsmirror", "test": "TestAILiveBailianPersistsAssessmentAndTranslation",