This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
# 知识库级版本同步:开发与验收清单
|
||||
|
||||
## 交付边界
|
||||
|
||||
本清单记录已确认需求与验收条件,不表示功能已实现。日常编辑按文档,生产版本及恢复按知识库完整状态;发布管理仅有对内、对外两个目标。开发授权不包含生产同步、生产恢复或真实数据清理。生产执行仍须在发布管理经管理员授权;保留数量、天数、容量预算未设置,自动清理默认关闭。
|
||||
|
||||
现有文档包发布与执行级补偿只能作为复用基础,不能作为本清单完成证据。静态样图不作为产品运行证据。组织绩效与周报外发不在本次范围。
|
||||
|
||||
## 已确认:封存隔离方案 A
|
||||
|
||||
用户选择封存期间自动暂停测试 WeKnora 服务,封存后恢复,不修改 WeKnora 源码。测试问答在暂停期间暂不可用,停用时长须实测;不暂停生产服务,也不因此取得生产同步、恢复或真实清理授权。
|
||||
|
||||
实现要求:先等待处理任务完成,枚举并核实全部测试写入者和副本;暂停后重新检查状态,数据库、索引和对象存储保持可读。持久记录暂停前运行状态及作业归属,覆盖成功、失败、取消和进程重启后的恢复;仅恢复本作业暂停的服务,并读回服务健康。完整归档封存后即可恢复测试服务,后续生产传输从归档读取,不延长测试停用时间。
|
||||
|
||||
方案已确认不等于暂停/恢复功能已实现;恢复保障验证前不停止运行中的测试服务。
|
||||
|
||||
## 开发与测试顺序
|
||||
|
||||
1. 核查实际 WeKnora 的文档删除、Wiki 更新与完成信号,以及跨数据库、对象、索引的快照与恢复契约。
|
||||
2. 实现完整版本清单、一致性校验、授权与持久化作业;逐项运行失败用例再实现通过。
|
||||
3. 接通实际导出、独立生产预备份、传输、验证、恢复及持久化进度;隔离环境演练故障、重启与重试。
|
||||
4. 发布管理接真实接口,存储管理接真实测量及依赖保护;验证浏览器交互与权限。
|
||||
5. 完整回归、审查、提交交付;生产实测单列等待授权,不以隔离验收代替。
|
||||
|
||||
## 验收项目
|
||||
|
||||
| ID | 范围 | 必须证明的行为 | 状态 |
|
||||
| --- | --- | --- | --- |
|
||||
| V01 | 两个目标 | 只按对内、对外操作;身份来自已配置映射,不按名称猜测 | 待验证 |
|
||||
| V02 | 测试就绪 | 全部解析、索引及 Wiki 就绪;问答与性能验收绑定本版,旧验收不能放行新变化 | 待验证 |
|
||||
| V03 | Wiki 撤销 | 删除文档后验证共享正文、实体、概念、边与引用的实际变化;删除成功或引用剥离不能代替此证据 | 待验证 |
|
||||
| V04 | 完整固定版本 | 清单覆盖文档集合、处理结果、资源、索引、完整 Wiki;无部分文档配整库 Wiki 的路径 | 待验证 |
|
||||
| V05 | 并发修改 | 导出中发生源变更时拒绝不一致版本,或由已证明的固定快照协议隔离;导出前后表面计数相同不算充分证明 | 待验证 |
|
||||
| V06 | 内容范围 | 仅允许知识数据;账号、会话、凭据及无关知识库不被同步或恢复 | 待验证 |
|
||||
| B01 | 写前预备份 | 每次生产变更前创建旧状态完整备份,校验恢复资料完整性;失败时零生产知识写入 | 待验证 |
|
||||
| B02 | 独立恢复依赖 | 资源在测试侧删除后,保留版本仍能恢复,不依赖测试临时对象或短期签名地址 | 待验证 |
|
||||
| B03 | 完整 Wiki 资产 | 每版保存完整 Wiki 及引用关联,校验摘要、数量和版本绑定 | 待验证 |
|
||||
| S01 | 数据同步 | 数据库、资源、索引、Wiki 均真实落地并校验,增量仅为传输优化 | 待验证 |
|
||||
| S02 | 无生产再计算 | 生产侧不解析、Embedding 或生成 Wiki,使用测试验收结果 | 待验证 |
|
||||
| S03 | 一致生效 | 不宣称不存在的跨系统原子性;明确可见性保护、失败补偿与验证边界 | 待验证 |
|
||||
| S04 | 故障与恢复 | 各阶段失败、超时与进程重启后可追溯、可恢复,不重复破坏性动作 | 待验证 |
|
||||
| S05 | 幂等与互斥 | 重复请求不启动重复同步;同目标同步、恢复、清理之间互斥;不同请求体复用键被拒绝 | 待验证 |
|
||||
| R01 | 版本恢复 | 恢复生产整套文档、资源、索引与 Wiki;测试知识库不回退 | 待验证 |
|
||||
| R02 | 恢复失败 | 不标记成功、不丢旧恢复依据,明确受影响阶段及后续可执行措施 | 待验证 |
|
||||
| R03 | 速度 | 分开实测备份与恢复耗时、数据量、吞吐及环境;未经测量不承诺快速 | 待验证 |
|
||||
| P01 | 作业阶段 | 持久化准备、备份、同步、验证、恢复阶段与开始/更新时间、错误;生产验证成功才完成 | 待验证 |
|
||||
| P02 | 四类进度 | 数据库记录、资源文件/字节、索引分块、Wiki 页面/关联数据来自实际测量;总量未知不造百分比 | 待验证 |
|
||||
| P03 | 明细 | 可查看实际对象、数量、字节、校验及异常;筛选与总量不依赖前端当前页 | 待验证 |
|
||||
| U01 | 主要入口 | 对内/对外切换、启动同步、版本与快照、当前作业预备份入口均可见;旧版历史可追溯 | 待验证 |
|
||||
| U02 | 权限 | 未就绪明确阻断原因;会话注入操作者;只读/工程师不能获得生产执行权限 | 待验证 |
|
||||
| U03 | 操作确认 | 启动、恢复显示准确目标及影响并保留授权;运行中禁止重复启动 | 待验证 |
|
||||
| D01 | 存储策略 | 未设阈值如实显示未设置,自动清理默认关闭,不隐含年龄删除 | 待验证 |
|
||||
| D02 | 占用测量 | 区分逻辑大小、独占、共享、实际占用;无法测量显示未知,不当成零 | 待验证 |
|
||||
| D03 | 保护引用 | 当前恢复依据、活跃作业、锁定/保留版本及共享资源受到保护 | 待验证 |
|
||||
| D04 | 清理预览 | 列明对象、依赖、受影响版本及可测释放量;执行前重新核验引用与活跃作业 | 待验证 |
|
||||
| D05 | 容量门禁 | 空间不足告警或阻断,不为继续作业偷删受保护备份 | 待验证 |
|
||||
| D06 | 测试残留 | 覆盖取件缓存、临时解析、同步暂存、失败上传、删除/重解析残留,不删除源文件 | 待验证 |
|
||||
| D07 | 实际回收 | 区分逻辑删除与实际释放,核查对象历史版本/分片及数据库、索引回收;保留版本清理后仍可恢复 | 待验证 |
|
||||
|
||||
## 证据登记规则
|
||||
|
||||
每项完成需附实现位置、运行命令与退出码、可回看的证据路径、验证环境及尚未覆盖范围。单元测试、隔离数据库集成、模拟外部服务、真实测试知识库、生产验收分别标注,不能相互替代。
|
||||
|
||||
真实清理和生产恢复尚未授权的项目保持待授权/待验证;不得为了打勾执行。整项范围未覆盖不得以子切片通过称为本阶段完成。
|
||||
|
||||
## 已运行的基础模块证据(不等于整体验收)
|
||||
|
||||
实现位置:`internal/librarysnapshot/` 与 `internal/releaseadapter/v071/library_*.go`。
|
||||
|
||||
- PostgreSQL 整库只读 Repeatable Read 导出:显式包含空 Wiki 表;未完成解析、删除文档残留任务、死信、未知 schema、Neo4j 启用场景拒绝自动放行。
|
||||
- Elasticsearch 使用 PIT 与 search_after,核查精确总量、返回范围、重复项与部分结果;PIT 只证明索引自身的视图。
|
||||
- 候选归档独立持有资源字节,校验表、scope、资源依赖、索引关联及摘要;内容寻址复用、重开加载、同一文件描述符校验读取、共享/独占引用字节已测试。引用字节不是物理占用或实际可回收字节。
|
||||
- `go test -race ./internal/librarysnapshot ./internal/releaseadapter/v071 -count=1`:退出码 0;默认跳过需要显式配置的隔离集成,不能据此声称集成通过。
|
||||
- 隔离 PostgreSQL + Elasticsearch 集成另行运行,退出码 0。只创建随机 `kg_library_test_` 前缀夹具;结束后 PostgreSQL 同前缀库残留计数为 0。
|
||||
- MVCC 反向自检:仅通过 Go overlay 把 Repeatable Read 改为 Read Committed,同一真实隔离集成退出码 1,失败断言为 `mixed PostgreSQL versions during export`。原始实现复跑退出码 0。
|
||||
- `make verify`:退出码 0。本地门禁不是远端 CI,也不是新 UI 或真实知识库完整验收。
|
||||
|
||||
证据保存在本机仓库忽略目录 `backups/library-sync-foundation-20261027/`:`unit-tests.jsonl`、`kg-library-integration.log`、`kg-library-mvcc-negative.log`、`kg-library-verify.log`。隔离执行脚本副本同目录留存,不含静态凭据;运行时从本地测试容器读取。
|
||||
|
||||
### 当前不能放行的边界
|
||||
|
||||
1. `ArchiveLibraryCandidate` 的 seal 只代表归档完整性,不代表跨系统同一时点、Wiki 语义验收通过或生产授权。尚未接到生产执行器。
|
||||
2. 数据库 MVCC 与 ES PIT 是各自独立的快照,不能把两者拼接视为一致整版。用户已确认方案 A:封存期间自动暂停测试服务;全部写入者隔离、异常恢复及实际停用时长尚待实现与验证,不修改 WeKnora 源码,不暂停生产。
|
||||
3. 已检查本地 v0.7.1 源码:`EnqueueWikiRetract` 在持久化 pending op 失败时记录警告后返回;因此 pending/dead-letter 都为空仍不证明撤销语义已处理。必须补充删除后共享 Wiki 内容的实测与验收绑定。
|
||||
4. 生产预备份、独立恢复演练、持久化协调作业、授权 API、真实进度、发布页面与完整存储治理仍未交付;以上验收项目保持待验证。
|
||||
5. 当前新增代码未提交、未推送、未部署;未执行生产同步、恢复或真实业务清理。
|
||||
|
||||
## 方案 A:暂停与恢复原语的实测证据
|
||||
|
||||
新增 `internal/libraryfreeze/`,包含持久化作业日志、源服务组互斥、暂停前原运行状态、不可变容器 ID、恢复读回,以及本地 Unix socket Docker 控制器。Docker 控制器只接受对应 source 且 `environment=test`、`freeze-role=writer` 的容器;不按模糊名称选取,不控制数据库/索引存储,不使用远程默认 Docker context。运行容器必须有健康检查。
|
||||
|
||||
已经执行的验证:
|
||||
|
||||
- `go test -race ./internal/libraryfreeze ./internal/librarysnapshot ./internal/releaseadapter/v071 -count=1`,退出码 0。
|
||||
- `KG_FREEZE_DOCKER_ISOLATED=1 go test -race -v ./internal/libraryfreeze -run '^TestFreezeIsolatedDocker$' -count=1`,退出码 0。使用已缓存 busybox 创建随机标签、无网络、无宿主挂载的一次性容器,真实验证停止→捕获窗口→重启健康、原本停止容器不启动;控制子进程在停止后退出,再由新恢复调用读持久日志启动原容器。清理后按标签读回无残留。
|
||||
- 异常覆盖:停止回包丢失、停止未生效、新副本出现、容器被替换、捕获错误/panic/取消、日志写入失败、恢复健康失败、畸形日志、同源并发及重复恢复。捕获 panic 和日志失写故障先实际报红,修复后通过。
|
||||
- 不使用有延迟强杀计时器的停止方式;仅发送 SIGTERM 并等待实际停止。若停止结果不明且服务仍运行,不把“现在运行”误判为已恢复,日志保持待恢复,防止随后迟到的停止把服务留在关闭状态。
|
||||
- 新增 Docker 集成用例已在 `scripts/test-scope.json` 精确声明 opt-in;普通 CI 明确跳过,不冒称执行。完整 `make verify` 退出码 0,属于本地门禁。
|
||||
|
||||
证据:本机 `backups/library-sync-foundation-20261027/kg-freeze-docker-integration.log`、`kg-library-freeze-verify.log`。
|
||||
|
||||
已新增 `internal/releaseadapter/v071/library_capture.go`,在源服务组互斥下串起预检、暂停、停后复检、PG/ES 导出、独立归档和恢复。协调测试使用 PG/服务控制夹具与 HTTP 搜索夹具,覆盖预检忙时零停止、暂停前后新增任务阻断、搜索失败恢复、旧作业重放不重新暂停、跨 scope 复用拒绝、归档丢失不返回成功;不把这组夹具当成实际业务知识库全链路验收。最终竞态回归、隔离 Docker 重跑与 `make verify` 均退出码 0,最终日志为同目录 `kg-freeze-docker-final.log`、`kg-library-freeze-final-verify.log`。
|
||||
|
||||
**仍缺少的接线**:真实测试部署的全部写入者枚举及标签/健康契约、全源任务就绪检查的实际提供方、worker 调用捕获协调函数、启动恢复入口和持续恢复调度。单独的本地 Docker 控制器不能控制远端测试主机;远端场景需在测试主机运行受限执行器,不能向浏览器或普通 API 暴露 Docker socket。上述测试不能证明这些接线已完成,尚未暂停实际测试 WeKnora。
|
||||
@@ -0,0 +1,138 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFreezeCancelledCaptureCannotSucceed(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
result, err := e.Run(ctx, "cancelled", func(context.Context) (string, error) { cancel(); return strings.Repeat("a", 64), nil })
|
||||
if err == nil || result.State == "succeeded" {
|
||||
t.Fatalf("cancelled work succeeded: %+v %v", result, err)
|
||||
}
|
||||
if !c.services[0].Running || !c.services[0].Healthy {
|
||||
t.Fatal("cancel prevented recovery")
|
||||
}
|
||||
}
|
||||
|
||||
type faultyControl struct {
|
||||
*fixtureControl
|
||||
mode string
|
||||
}
|
||||
|
||||
func (c *faultyControl) Stop(ctx context.Context, id string) error {
|
||||
if c.mode == "stop-noop" {
|
||||
return nil
|
||||
}
|
||||
err := c.fixtureControl.Stop(ctx, id)
|
||||
if c.mode == "lost-ack" {
|
||||
return errors.New("lost stop acknowledgement")
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (c *faultyControl) Start(ctx context.Context, id string) error {
|
||||
err := c.fixtureControl.Start(ctx, id)
|
||||
if c.mode == "unhealthy" {
|
||||
for i := range c.services {
|
||||
if c.services[i].ID == id {
|
||||
c.services[i].Healthy = false
|
||||
}
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
func TestFreezeFailureBoundaries(t *testing.T) {
|
||||
for _, mode := range []string{"lost-ack", "stop-noop", "unhealthy", "new-replica", "replacement", "capture-error"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
e.control = &faultyControl{fixtureControl: c, mode: mode}
|
||||
called := false
|
||||
result, err := e.Run(context.Background(), "fault-job", func(context.Context) (string, error) {
|
||||
called = true
|
||||
switch mode {
|
||||
case "new-replica":
|
||||
c.services = append(c.services, Service{ID: "unexpected-app", Running: true, Healthy: true})
|
||||
case "replacement":
|
||||
c.services[0].ID = "replacement-app"
|
||||
case "capture-error":
|
||||
return "", errors.New("fixture error")
|
||||
}
|
||||
return strings.Repeat("a", 64), nil
|
||||
})
|
||||
if err == nil || result.State == "succeeded" {
|
||||
t.Fatalf("fault reported success: %+v %v", result, err)
|
||||
}
|
||||
if (mode == "lost-ack" || mode == "stop-noop") && called {
|
||||
t.Fatal("capture ran without verified stop")
|
||||
}
|
||||
if mode == "replacement" {
|
||||
for _, call := range c.calls {
|
||||
if strings.HasPrefix(call, "start:") {
|
||||
t.Fatal("replacement was started")
|
||||
}
|
||||
}
|
||||
} else if !c.services[0].Running {
|
||||
t.Fatal("original service not restored")
|
||||
}
|
||||
if mode == "unhealthy" || mode == "replacement" {
|
||||
j, err := e.load("fault-job")
|
||||
if err != nil || terminal(j.State) {
|
||||
t.Fatal("unverified recovery marked terminal")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
func TestFreezeSourceLockBlocksSeparateCoordinator(t *testing.T) {
|
||||
root, c, e := freezeFixture(t)
|
||||
other, err := Open(root, "test-source", c)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer other.Close()
|
||||
_, err = e.Run(context.Background(), "outer", func(context.Context) (string, error) {
|
||||
_, blocked := other.Run(context.Background(), "inner", func(context.Context) (string, error) { t.Fatal("overlapping capture"); return "", nil })
|
||||
if blocked == nil {
|
||||
t.Fatal("source lock did not block")
|
||||
}
|
||||
if err := other.Recover(context.Background()); err == nil {
|
||||
t.Fatal("recovery raced active capture")
|
||||
}
|
||||
return strings.Repeat("a", 64), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
func TestFreezeRejectsMalformedRecoveryJournal(t *testing.T) {
|
||||
for _, mode := range []string{"unknown-state", "invalid-service", "duplicate-service", "false-ownership"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
j := journal{Version: 1, Source: "test-source", Job: "invalid", Result: Result{State: "stopping"}, Services: []serviceRecord{{Service: Service{ID: "app-a", Running: true}, StopRequested: true}}}
|
||||
switch mode {
|
||||
case "unknown-state":
|
||||
j.State = "invented"
|
||||
case "invalid-service":
|
||||
j.Services[0].ID = "../../wrong"
|
||||
case "duplicate-service":
|
||||
j.Services = append(j.Services, j.Services[0])
|
||||
case "false-ownership":
|
||||
j.Services[0].Running = false
|
||||
}
|
||||
if err := e.save(j); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := e.Recover(context.Background()); err == nil {
|
||||
t.Fatal("malformed journal accepted")
|
||||
}
|
||||
if len(c.calls) != 0 {
|
||||
t.Fatalf("malformed journal caused service writes: %v", c.calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const sourceLabel = "io.knowledge-governance.freeze-source"
|
||||
const environmentLabel = "io.knowledge-governance.environment"
|
||||
const roleLabel = "io.knowledge-governance.freeze-role"
|
||||
|
||||
type DockerController struct {
|
||||
source string
|
||||
timeout time.Duration
|
||||
client *http.Client
|
||||
transport *http.Transport
|
||||
}
|
||||
|
||||
func NewDockerController(socket, source string, timeout time.Duration) (*DockerController, error) {
|
||||
if !filepath.IsAbs(socket) || !validID(source) || timeout <= 0 || timeout > 5*time.Minute {
|
||||
return nil, errors.New("invalid test Docker controller configuration")
|
||||
}
|
||||
info, err := os.Stat(socket)
|
||||
if err != nil || info.Mode()&os.ModeSocket == 0 {
|
||||
return nil, errors.New("local Docker socket unavailable")
|
||||
}
|
||||
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
|
||||
return (&net.Dialer{}).DialContext(ctx, "unix", socket)
|
||||
}}
|
||||
client := &http.Client{Transport: transport, Timeout: timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("Docker redirects prohibited") }}
|
||||
return &DockerController{source: source, timeout: timeout, client: client, transport: transport}, nil
|
||||
}
|
||||
func (d *DockerController) Close() { d.transport.CloseIdleConnections() }
|
||||
func (d *DockerController) request(ctx context.Context, method, path string, out any) error {
|
||||
req, err := http.NewRequestWithContext(ctx, method, "http://docker"+path, nil)
|
||||
if err != nil {
|
||||
return errors.New("invalid Docker request")
|
||||
}
|
||||
resp, err := d.client.Do(req)
|
||||
if err != nil {
|
||||
return errors.New("test Docker request failed")
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return errors.New("test Docker request rejected")
|
||||
}
|
||||
if out != nil {
|
||||
data, err := io.ReadAll(io.LimitReader(resp.Body, 1024*1024+1))
|
||||
if err != nil || len(data) > 1024*1024 || json.Unmarshal(data, out) != nil {
|
||||
return errors.New("invalid Docker response")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func dockerID(id string) bool {
|
||||
b, err := hex.DecodeString(id)
|
||||
return err == nil && len(b) == 32 && strings.ToLower(id) == id
|
||||
}
|
||||
|
||||
type dockerInspection struct {
|
||||
ID string `json:"Id"`
|
||||
Config struct {
|
||||
Labels map[string]string `json:"Labels"`
|
||||
} `json:"Config"`
|
||||
State struct {
|
||||
Running bool
|
||||
Paused bool
|
||||
Restarting bool
|
||||
Dead bool
|
||||
Status string
|
||||
Health *struct{ Status string }
|
||||
} `json:"State"`
|
||||
}
|
||||
|
||||
func (d *DockerController) inspect(ctx context.Context, id string) (Service, error) {
|
||||
if !dockerID(id) {
|
||||
return Service{}, errors.New("immutable Docker ID required")
|
||||
}
|
||||
var data dockerInspection
|
||||
if err := d.request(ctx, "GET", "/containers/"+id+"/json", &data); err != nil {
|
||||
return Service{}, err
|
||||
}
|
||||
labels := data.Config.Labels
|
||||
if data.ID != id || labels[sourceLabel] != d.source || labels[environmentLabel] != "test" || labels[roleLabel] != "writer" {
|
||||
return Service{}, errors.New("container is not an approved test writer")
|
||||
}
|
||||
if data.State.Paused || data.State.Restarting || data.State.Dead {
|
||||
return Service{}, errors.New("test writer has unstable runtime state")
|
||||
}
|
||||
if data.State.Status != "running" && data.State.Status != "exited" && data.State.Status != "created" {
|
||||
return Service{}, errors.New("unknown test writer runtime state")
|
||||
}
|
||||
if data.State.Running != (data.State.Status == "running") {
|
||||
return Service{}, errors.New("contradictory test writer state")
|
||||
}
|
||||
healthy := data.State.Running && data.State.Health != nil && data.State.Health.Status == "healthy"
|
||||
return Service{ID: id, Running: data.State.Running, Healthy: healthy}, nil
|
||||
}
|
||||
func (d *DockerController) Inventory(ctx context.Context) ([]Service, error) {
|
||||
filters, _ := json.Marshal(map[string][]string{"label": {sourceLabel + "=" + d.source}})
|
||||
var containers []struct {
|
||||
ID string `json:"Id"`
|
||||
}
|
||||
if err := d.request(ctx, "GET", "/containers/json?all=1&filters="+url.QueryEscape(string(filters)), &containers); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(containers) == 0 || len(containers) > 256 {
|
||||
return nil, errors.New("empty or oversized test writer group")
|
||||
}
|
||||
items := make([]Service, 0, len(containers))
|
||||
seen := map[string]bool{}
|
||||
for _, container := range containers {
|
||||
if seen[container.ID] {
|
||||
return nil, errors.New("duplicate Docker instance")
|
||||
}
|
||||
seen[container.ID] = true
|
||||
s, err := d.inspect(ctx, container.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, s)
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
func (d *DockerController) wait(ctx context.Context, id string, running bool) error {
|
||||
ticker := time.NewTicker(100 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
s, err := d.inspect(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.Running == running && (!running || s.Healthy) {
|
||||
return nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return errors.New("test writer state did not settle before deadline")
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Only SIGTERM is sent. There is no delayed SIGKILL timer which could fire after
|
||||
// recovery starts the service again. An unresolved graceful stop remains pending
|
||||
// in the coordinator journal; running-now is not evidence of recovery.
|
||||
func (d *DockerController) Stop(ctx context.Context, id string) error {
|
||||
bounded, cancel := context.WithTimeout(ctx, d.timeout)
|
||||
defer cancel()
|
||||
s, err := d.inspect(bounded, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !s.Running {
|
||||
return nil
|
||||
}
|
||||
if err = d.request(bounded, "POST", "/containers/"+id+"/kill?signal=SIGTERM", nil); err != nil {
|
||||
return err
|
||||
}
|
||||
return d.wait(bounded, id, false)
|
||||
}
|
||||
func (d *DockerController) Start(ctx context.Context, id string) error {
|
||||
bounded, cancel := context.WithTimeout(ctx, d.timeout)
|
||||
defer cancel()
|
||||
s, err := d.inspect(bounded, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !s.Running {
|
||||
if err = d.request(bounded, "POST", "/containers/"+id+"/start", nil); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return d.wait(bounded, id, true)
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDockerControllerRefusesWrongScopeAndUnsafeRuntime(t *testing.T) {
|
||||
for _, mode := range []string{"production", "wrong-source", "storage", "paused", "restarting", "dead", "no-health", "short-id", "wrong-response-id", "duplicate"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
root, err := os.MkdirTemp(os.TempDir(), "kgf-")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.RemoveAll(root)
|
||||
socket := filepath.Join(root, "s")
|
||||
listener, err := net.Listen("unix", socket)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
id := strings.Repeat("a", 64)
|
||||
if mode == "short-id" {
|
||||
id = "aaa"
|
||||
}
|
||||
writes := 0
|
||||
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != "GET" {
|
||||
writes++
|
||||
w.WriteHeader(204)
|
||||
return
|
||||
}
|
||||
if strings.HasSuffix(r.URL.Path, "/containers/json") {
|
||||
items := []map[string]string{{"Id": id}}
|
||||
if mode == "duplicate" {
|
||||
items = append(items, items[0])
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(items)
|
||||
return
|
||||
}
|
||||
labels := map[string]string{sourceLabel: "test-source", environmentLabel: "test", roleLabel: "writer"}
|
||||
switch mode {
|
||||
case "production":
|
||||
labels[environmentLabel] = "production"
|
||||
case "wrong-source":
|
||||
labels[sourceLabel] = "different-source"
|
||||
case "storage":
|
||||
labels[roleLabel] = "database"
|
||||
}
|
||||
state := map[string]any{"Running": true, "Status": "running", "Health": map[string]string{"Status": "healthy"}}
|
||||
switch mode {
|
||||
case "paused":
|
||||
state["Paused"] = true
|
||||
case "restarting":
|
||||
state["Restarting"] = true
|
||||
case "dead":
|
||||
state["Dead"] = true
|
||||
case "no-health":
|
||||
delete(state, "Health")
|
||||
}
|
||||
actualID := id
|
||||
if mode == "wrong-response-id" {
|
||||
actualID = strings.Repeat("b", 64)
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"Id": actualID, "Config": map[string]any{"Labels": labels}, "State": state})
|
||||
})}
|
||||
go server.Serve(listener)
|
||||
defer server.Close()
|
||||
d, err := NewDockerController(socket, "test-source", time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer d.Close()
|
||||
journalRoot := filepath.Join(root, "j")
|
||||
if err = os.Mkdir(journalRoot, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e, err := Open(journalRoot, "test-source", d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer e.Close()
|
||||
_, err = e.Run(context.Background(), "blocked", func(context.Context) (string, error) { t.Fatal("unsafe capture"); return "", nil })
|
||||
if err == nil || writes != 0 {
|
||||
t.Fatalf("unsafe source changed: err=%v writes=%d", err, writes)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestFreezeIsolatedDocker(t *testing.T) {
|
||||
if os.Getenv("KG_FREEZE_DOCKER_ISOLATED") != "1" {
|
||||
t.Skip("explicit opt-in required for disposable local Docker fixture")
|
||||
}
|
||||
if os.Getenv("KG_FREEZE_DOCKER_CHILD") == "1" {
|
||||
d, err := NewDockerController("/var/run/docker.sock", os.Getenv("KG_FREEZE_SOURCE"), 10*time.Second)
|
||||
if err != nil {
|
||||
os.Exit(31)
|
||||
}
|
||||
e, err := Open(os.Getenv("KG_FREEZE_ROOT"), os.Getenv("KG_FREEZE_SOURCE"), d)
|
||||
if err != nil {
|
||||
os.Exit(32)
|
||||
}
|
||||
_, _ = e.Run(context.Background(), "crashed-real", func(context.Context) (string, error) { os.Exit(37); return "", nil })
|
||||
os.Exit(33)
|
||||
}
|
||||
run := func(args ...string) string {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "docker", append([]string{"--host", "unix:///var/run/docker.sock"}, args...)...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("fixture Docker command failed: %v %s", err, out)
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
image := run("image", "inspect", "busybox:1.36", "--format", "{{.Id}}")
|
||||
var token [8]byte
|
||||
if _, err := rand.Read(token[:]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
source := "kg-freeze-test-" + hex.EncodeToString(token[:])
|
||||
var ids []string
|
||||
defer func() {
|
||||
for _, id := range ids {
|
||||
run("rm", "-f", id)
|
||||
}
|
||||
if remaining := run("ps", "-aq", "--filter", "label="+sourceLabel+"="+source); remaining != "" {
|
||||
t.Error("disposable freeze containers remain")
|
||||
}
|
||||
}()
|
||||
for _, name := range []string{"running", "originally-off"} {
|
||||
id := run("create", "--name", source+"-"+name, "--network", "none", "--read-only", "--tmpfs", "/tmp:rw,noexec,nosuid,size=1m", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--memory", "32m", "--cpus", "0.1", "--restart", "unless-stopped", "--label", sourceLabel+"="+source, "--label", environmentLabel+"=test", "--label", roleLabel+"=writer", "--health-cmd", "test -f /tmp/ready", "--health-interval", "1s", "--health-timeout", "1s", "--health-retries", "3", image, "sh", "-c", "trap 'exit 0' TERM; touch /tmp/ready; while :; do sleep 1 & wait $!; done")
|
||||
ids = append(ids, id)
|
||||
}
|
||||
d, err := NewDockerController("/var/run/docker.sock", source, 10*time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer d.Close()
|
||||
if err = d.Start(context.Background(), ids[0]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
if err = os.Chmod(root, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e, err := Open(root, source, d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer e.Close()
|
||||
assertRestored := func() {
|
||||
t.Helper()
|
||||
a, err := d.inspect(context.Background(), ids[0])
|
||||
if err != nil || !a.Running || !a.Healthy {
|
||||
t.Fatalf("original running fixture not healthy: %+v %v", a, err)
|
||||
}
|
||||
b, err := d.inspect(context.Background(), ids[1])
|
||||
if err != nil || b.Running {
|
||||
t.Fatal("originally-off fixture was started")
|
||||
}
|
||||
}
|
||||
started := time.Now()
|
||||
result, err := e.Run(context.Background(), "real-success", func(context.Context) (string, error) {
|
||||
items, err := d.Inventory(context.Background())
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, s := range items {
|
||||
if s.Running {
|
||||
t.Fatal("real writer active during capture")
|
||||
}
|
||||
}
|
||||
return strings.Repeat("b", 64), nil
|
||||
})
|
||||
if err != nil || result.State != "succeeded" {
|
||||
t.Fatalf("real freeze=%+v %v", result, err)
|
||||
}
|
||||
assertRestored()
|
||||
t.Logf("disposable_container_freeze_and_health_restore=%s", time.Since(started))
|
||||
cmd := exec.Command(os.Args[0], "-test.run=^TestFreezeIsolatedDocker$")
|
||||
cmd.Env = append(os.Environ(), "KG_FREEZE_DOCKER_CHILD=1", "KG_FREEZE_SOURCE="+source, "KG_FREEZE_ROOT="+root)
|
||||
err = cmd.Run()
|
||||
var exit *exec.ExitError
|
||||
if !errors.As(err, &exit) || exit.ExitCode() != 37 {
|
||||
t.Fatalf("crash injection did not reach stopped window: %v", err)
|
||||
}
|
||||
stopped, err := d.inspect(context.Background(), ids[0])
|
||||
if err != nil || stopped.Running {
|
||||
t.Fatal("crashed fixture was not stopped")
|
||||
}
|
||||
if err = e.Recover(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
assertRestored()
|
||||
j, err := e.load("crashed-real")
|
||||
if err != nil || j.State != "failed" || j.Code != "interrupted" {
|
||||
t.Fatalf("bad recovery journal: %+v %v", j, err)
|
||||
}
|
||||
t.Log("real process exit -> journal reload -> original container healthy; originally-off container remains off")
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDockerControllerUsesScopedIDsAndGracefulSignal(t *testing.T) {
|
||||
root, err := os.MkdirTemp(os.TempDir(), "kgf-")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer os.RemoveAll(root)
|
||||
socket := filepath.Join(root, "engine.sock")
|
||||
listener, err := net.Listen("unix", socket)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
id := strings.Repeat("a", 64)
|
||||
running := true
|
||||
var writes []string
|
||||
server := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == "GET" && strings.HasSuffix(r.URL.Path, "/containers/json"):
|
||||
if !strings.Contains(r.URL.Query().Get("filters"), "test-source") {
|
||||
t.Error("missing scope filter")
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode([]map[string]string{{"Id": id}})
|
||||
case r.Method == "GET" && strings.HasSuffix(r.URL.Path, "/json"):
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"Id": id, "Config": map[string]any{"Labels": map[string]string{sourceLabel: "test-source", environmentLabel: "test", roleLabel: "writer"}}, "State": map[string]any{"Running": running, "Status": map[bool]string{true: "running", false: "exited"}[running], "Health": map[string]string{"Status": "healthy"}}})
|
||||
case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/kill"):
|
||||
if r.URL.Query().Get("signal") != "SIGTERM" {
|
||||
t.Error("unsafe stop signal")
|
||||
}
|
||||
writes = append(writes, "stop")
|
||||
running = false
|
||||
w.WriteHeader(204)
|
||||
case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/start"):
|
||||
writes = append(writes, "start")
|
||||
running = true
|
||||
w.WriteHeader(204)
|
||||
default:
|
||||
w.WriteHeader(404)
|
||||
}
|
||||
})}
|
||||
go server.Serve(listener)
|
||||
defer server.Close()
|
||||
control, err := NewDockerController(socket, "test-source", time.Second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer control.Close()
|
||||
journalRoot := filepath.Join(root, "journal")
|
||||
if err = os.Mkdir(journalRoot, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, err := Open(journalRoot, "test-source", control)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer engine.Close()
|
||||
result, err := engine.Run(context.Background(), "fixture-job", func(context.Context) (string, error) {
|
||||
if running {
|
||||
t.Fatal("writer was not stopped")
|
||||
}
|
||||
return strings.Repeat("b", 64), nil
|
||||
})
|
||||
if err != nil || result.State != "succeeded" || strings.Join(writes, ",") != "stop,start" {
|
||||
t.Fatalf("result=%+v err=%v writes=%v", result, err, writes)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFreezeRestoresDespiteCapturePanic(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
func() {
|
||||
defer func() { _ = recover() }()
|
||||
_, _ = e.Run(context.Background(), "panic-job", func(context.Context) (string, error) { panic("fixture panic") })
|
||||
}()
|
||||
if !c.services[0].Running || !c.services[0].Healthy {
|
||||
t.Fatal("capture panic left test service stopped")
|
||||
}
|
||||
j, err := e.load("panic-job")
|
||||
if err != nil || j.State != "failed" {
|
||||
t.Fatalf("journal=%+v err=%v", j, err)
|
||||
}
|
||||
}
|
||||
func TestFreezeRestoresEvenWhenJournalBecomesUnwritable(t *testing.T) {
|
||||
root, c, e := freezeFixture(t)
|
||||
_, err := e.Run(context.Background(), "disk-failure", func(context.Context) (string, error) {
|
||||
path := filepath.Join(root, "disk-failure.json")
|
||||
if err := os.Remove(path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Mkdir(path, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return strings.Repeat("a", 64), nil
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("lost journal must not report success")
|
||||
}
|
||||
if !c.services[0].Running || !c.services[0].Healthy {
|
||||
t.Fatal("journal failure prevented service restoration")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,472 @@
|
||||
// Package libraryfreeze coordinates a test-only maintenance window. It never
|
||||
// grants production authorization or starts services which were already stopped.
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Service struct {
|
||||
ID string `json:"id"`
|
||||
Running bool `json:"running"`
|
||||
Healthy bool `json:"healthy"`
|
||||
}
|
||||
|
||||
// Inventory must enumerate the complete, independently scoped test writer set,
|
||||
// using immutable instance IDs. Never implement it as only the currently running
|
||||
// subset. Start must wait for readiness; callers still read back the result.
|
||||
type Controller interface {
|
||||
Inventory(context.Context) ([]Service, error)
|
||||
Stop(context.Context, string) error
|
||||
Start(context.Context, string) error
|
||||
}
|
||||
type Result struct {
|
||||
State string `json:"state"`
|
||||
SnapshotID string `json:"snapshot_id,omitempty"`
|
||||
}
|
||||
type serviceRecord struct {
|
||||
Service
|
||||
StopRequested bool `json:"stop_requested"`
|
||||
StopVerified bool `json:"stop_verified"`
|
||||
Restored bool `json:"restored"`
|
||||
}
|
||||
type journal struct {
|
||||
Version int `json:"version"`
|
||||
Source string `json:"source"`
|
||||
Job string `json:"job"`
|
||||
Result
|
||||
Services []serviceRecord `json:"services"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
Code string `json:"code,omitempty"`
|
||||
}
|
||||
type Coordinator struct {
|
||||
root *os.Root
|
||||
source string
|
||||
control Controller
|
||||
}
|
||||
|
||||
func Open(root, source string, c Controller) (*Coordinator, error) {
|
||||
if !filepath.IsAbs(root) || !validID(source) || c == nil {
|
||||
return nil, errors.New("invalid freeze configuration")
|
||||
}
|
||||
info, err := os.Lstat(root)
|
||||
if err != nil || !info.IsDir() || info.Mode().Perm() != 0700 {
|
||||
return nil, errors.New("freeze directory must be private and real")
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(root)
|
||||
if err != nil || resolved != filepath.Clean(root) {
|
||||
return nil, errors.New("freeze directory contains symlink")
|
||||
}
|
||||
r, err := os.OpenRoot(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Coordinator{root: r, source: source, control: c}, nil
|
||||
}
|
||||
func (c *Coordinator) Close() error { return c.root.Close() }
|
||||
|
||||
func (c *Coordinator) Recover(ctx context.Context) error {
|
||||
lock, err := c.lock()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer unlock(lock)
|
||||
dir, err := c.root.Open(".")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
entries, err := dir.ReadDir(-1)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Validate every journal before issuing recovery writes. Corrupt records or
|
||||
// another source's directory must not turn into broad service startup.
|
||||
var pending []journal
|
||||
for _, entry := range entries {
|
||||
if !strings.HasSuffix(entry.Name(), ".json") {
|
||||
continue
|
||||
}
|
||||
job := strings.TrimSuffix(entry.Name(), ".json")
|
||||
if !validID(job) {
|
||||
return errors.New("invalid freeze journal name")
|
||||
}
|
||||
j, err := c.load(job)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !terminal(j.State) {
|
||||
pending = append(pending, j)
|
||||
}
|
||||
}
|
||||
if len(pending) > 1 {
|
||||
return errors.New("multiple interrupted freezes require operator reconciliation")
|
||||
}
|
||||
for _, j := range pending {
|
||||
if err = c.restore(ctx, &j); err != nil {
|
||||
return err
|
||||
}
|
||||
j.State = "failed"
|
||||
j.Code = "interrupted"
|
||||
ended := time.Now().UTC()
|
||||
j.FinishedAt = &ended
|
||||
if err = c.save(j); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func invokeCapture(ctx context.Context, capture func(context.Context) (string, error)) (id string, err error) {
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
id = ""
|
||||
err = errors.New("candidate capture panicked")
|
||||
}
|
||||
}()
|
||||
return capture(ctx)
|
||||
}
|
||||
|
||||
func validID(s string) bool {
|
||||
if len(s) == 0 || len(s) > 128 {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == '-' || r == '_') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
func terminal(s string) bool { return s == "succeeded" || s == "failed" }
|
||||
|
||||
// A kernel lock serializes the whole test writer group, not individual KBs.
|
||||
// A crash releases the lock, but leaves the write-ahead journal for recovery.
|
||||
func (c *Coordinator) lock() (*os.File, error) {
|
||||
f, err := c.root.OpenFile("operation.lock", os.O_CREATE|os.O_RDWR|syscall.O_NOFOLLOW, 0600)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
|
||||
f.Close()
|
||||
return nil, errors.New("test source freeze is busy")
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
func unlock(f *os.File) { _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN); _ = f.Close() }
|
||||
func (c *Coordinator) save(j journal) error {
|
||||
data, err := json.Marshal(j)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var token [16]byte
|
||||
if _, err = rand.Read(token[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
name := "write-" + hex.EncodeToString(token[:]) + ".tmp"
|
||||
f, err := c.root.OpenFile(name, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer c.root.Remove(name)
|
||||
defer f.Close()
|
||||
if _, err = f.Write(data); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = f.Sync(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = f.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = c.root.Rename(name, j.Job+".json"); err != nil {
|
||||
return err
|
||||
}
|
||||
dir, err := c.root.Open(".")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
return dir.Sync()
|
||||
}
|
||||
func (c *Coordinator) load(job string) (journal, error) {
|
||||
f, err := c.root.OpenFile(job+".json", os.O_RDONLY|syscall.O_NOFOLLOW, 0)
|
||||
if err != nil {
|
||||
return journal{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
info, err := f.Stat()
|
||||
if err != nil || !info.Mode().IsRegular() || info.Size() > 1024*1024 {
|
||||
return journal{}, errors.New("invalid freeze journal file")
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(f, 1024*1024+1))
|
||||
if err != nil {
|
||||
return journal{}, err
|
||||
}
|
||||
var j journal
|
||||
dec := json.NewDecoder(bytes.NewReader(data))
|
||||
dec.DisallowUnknownFields()
|
||||
if err = dec.Decode(&j); err != nil {
|
||||
return journal{}, errors.New("invalid freeze journal")
|
||||
}
|
||||
var extra any
|
||||
if dec.Decode(&extra) != io.EOF || j.Version != 1 || j.Source != c.source || j.Job != job || !validID(j.Job) {
|
||||
return journal{}, errors.New("freeze journal identity mismatch")
|
||||
}
|
||||
switch j.State {
|
||||
case "stopping", "capturing", "restoring", "succeeded", "failed":
|
||||
default:
|
||||
return journal{}, errors.New("invalid freeze journal state")
|
||||
}
|
||||
if len(j.Services) == 0 || len(j.Services) > 256 {
|
||||
return journal{}, errors.New("invalid freeze journal services")
|
||||
}
|
||||
for i, s := range j.Services {
|
||||
if !validID(s.ID) || (i > 0 && j.Services[i-1].ID >= s.ID) || s.StopRequested && !s.Running || s.Restored && (!s.StopRequested || !s.Running) {
|
||||
return journal{}, errors.New("invalid freeze service ownership")
|
||||
}
|
||||
}
|
||||
return j, nil
|
||||
}
|
||||
func (c *Coordinator) pending() error {
|
||||
dir, err := c.root.Open(".")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dir.Close()
|
||||
entries, err := dir.ReadDir(-1)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if strings.HasSuffix(entry.Name(), ".json") {
|
||||
job := strings.TrimSuffix(entry.Name(), ".json")
|
||||
if !validID(job) {
|
||||
return errors.New("invalid freeze journal name")
|
||||
}
|
||||
j, err := c.load(job)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !terminal(j.State) {
|
||||
return errors.New("interrupted test freeze requires recovery")
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func inventory(ctx context.Context, control Controller) ([]Service, error) {
|
||||
items, err := control.Inventory(ctx)
|
||||
if err != nil {
|
||||
return nil, errors.New("test writer inventory unavailable")
|
||||
}
|
||||
if len(items) == 0 || len(items) > 256 {
|
||||
return nil, errors.New("invalid test writer inventory")
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, s := range items {
|
||||
if !validID(s.ID) || seen[s.ID] {
|
||||
return nil, errors.New("invalid test writer identity")
|
||||
}
|
||||
seen[s.ID] = true
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].ID < items[j].ID })
|
||||
return items, nil
|
||||
}
|
||||
func (c *Coordinator) stopped(ctx context.Context, j journal) error {
|
||||
items, err := inventory(ctx, c.control)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(items) != len(j.Services) {
|
||||
return errors.New("test writer topology changed")
|
||||
}
|
||||
for i, s := range items {
|
||||
if s.ID != j.Services[i].ID || s.Running {
|
||||
return errors.New("test writers not isolated")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c *Coordinator) restore(ctx context.Context, j *journal) error {
|
||||
j.State = "restoring"
|
||||
// Restoring services is still necessary if the journal disk fails after a
|
||||
// persisted stop intent. Keep the error, but do not let it suppress recovery.
|
||||
var failures error = c.save(*j)
|
||||
for i := range j.Services {
|
||||
s := &j.Services[i]
|
||||
if !s.Running || !s.StopRequested {
|
||||
continue
|
||||
}
|
||||
items, err := inventory(ctx, c.control)
|
||||
if err != nil {
|
||||
failures = errors.Join(failures, err)
|
||||
continue
|
||||
}
|
||||
found := false
|
||||
for _, actual := range items {
|
||||
if actual.ID != s.ID {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
if actual.Running && !s.StopVerified {
|
||||
failures = errors.Join(failures, errors.New("stop outcome remains uncertain; service may still stop later"))
|
||||
break
|
||||
}
|
||||
if !actual.Running {
|
||||
s.StopVerified = true
|
||||
if saveErr := c.save(*j); saveErr != nil {
|
||||
failures = errors.Join(failures, saveErr)
|
||||
}
|
||||
if err = c.control.Start(ctx, s.ID); err != nil {
|
||||
failures = errors.Join(failures, errors.New("test service restart failed"))
|
||||
break
|
||||
}
|
||||
}
|
||||
after, readErr := inventory(ctx, c.control)
|
||||
healthy := false
|
||||
if readErr == nil {
|
||||
for _, a := range after {
|
||||
if a.ID == s.ID && a.Running && a.Healthy {
|
||||
healthy = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !healthy {
|
||||
failures = errors.Join(failures, errors.New("test service health unverified"))
|
||||
break
|
||||
}
|
||||
s.Restored = true
|
||||
if err = c.save(*j); err != nil {
|
||||
failures = errors.Join(failures, err)
|
||||
}
|
||||
break
|
||||
}
|
||||
if !found {
|
||||
failures = errors.Join(failures, errors.New("original test service missing; replacement not started"))
|
||||
}
|
||||
}
|
||||
return failures
|
||||
}
|
||||
func (c *Coordinator) Run(ctx context.Context, job string, capture func(context.Context) (string, error)) (Result, error) {
|
||||
return c.RunPrepared(ctx, job, func(context.Context) error { return nil }, capture)
|
||||
}
|
||||
|
||||
// RunPrepared runs readiness checks under the source lock, before any stop
|
||||
// intent. It does not repeat checks or capture for an already completed job.
|
||||
func (c *Coordinator) RunPrepared(ctx context.Context, job string, prepare func(context.Context) error, capture func(context.Context) (string, error)) (Result, error) {
|
||||
if !validID(job) || capture == nil || prepare == nil {
|
||||
return Result{}, errors.New("invalid freeze request")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
lock, err := c.lock()
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
defer unlock(lock)
|
||||
previous, err := c.load(job)
|
||||
if err == nil {
|
||||
if previous.State == "succeeded" {
|
||||
return previous.Result, nil
|
||||
}
|
||||
return previous.Result, errors.New("freeze request already exists; recovery or a new request is required")
|
||||
}
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
return Result{}, err
|
||||
}
|
||||
if err = c.pending(); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if err = prepare(ctx); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if err = ctx.Err(); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
items, err := inventory(ctx, c.control)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
j := journal{Version: 1, Source: c.source, Job: job, Result: Result{State: "stopping"}, StartedAt: time.Now().UTC()}
|
||||
for _, s := range items {
|
||||
if s.Running && !s.Healthy {
|
||||
return Result{}, errors.New("test writer is not ready")
|
||||
}
|
||||
j.Services = append(j.Services, serviceRecord{Service: s})
|
||||
}
|
||||
if err = c.save(j); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
var workErr error
|
||||
for i := range j.Services {
|
||||
if !j.Services[i].Running {
|
||||
continue
|
||||
}
|
||||
j.Services[i].StopRequested = true
|
||||
if workErr = c.save(j); workErr != nil {
|
||||
break
|
||||
}
|
||||
if workErr = c.control.Stop(ctx, j.Services[i].ID); workErr != nil {
|
||||
break
|
||||
}
|
||||
}
|
||||
if workErr == nil {
|
||||
workErr = c.stopped(ctx, j)
|
||||
}
|
||||
if workErr == nil {
|
||||
for i := range j.Services {
|
||||
if j.Services[i].StopRequested {
|
||||
j.Services[i].StopVerified = true
|
||||
}
|
||||
}
|
||||
j.State = "capturing"
|
||||
workErr = c.save(j)
|
||||
}
|
||||
if workErr == nil {
|
||||
j.SnapshotID, workErr = invokeCapture(ctx, capture)
|
||||
}
|
||||
if workErr == nil {
|
||||
workErr = ctx.Err()
|
||||
}
|
||||
if workErr == nil {
|
||||
decoded, decodeErr := hex.DecodeString(j.SnapshotID)
|
||||
if decodeErr != nil || len(decoded) != 32 || strings.ToLower(j.SnapshotID) != j.SnapshotID {
|
||||
workErr = errors.New("invalid candidate snapshot digest")
|
||||
}
|
||||
}
|
||||
if workErr == nil {
|
||||
workErr = c.stopped(ctx, j)
|
||||
}
|
||||
recoveryCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
|
||||
defer cancel()
|
||||
recoveryErr := c.restore(recoveryCtx, &j)
|
||||
if recoveryErr != nil {
|
||||
return j.Result, errors.Join(workErr, recoveryErr)
|
||||
}
|
||||
if workErr != nil {
|
||||
j.State = "failed"
|
||||
j.Code = "capture_or_isolation_failed"
|
||||
} else {
|
||||
j.State = "succeeded"
|
||||
}
|
||||
ended := time.Now().UTC()
|
||||
j.FinishedAt = &ended
|
||||
if err = c.save(j); err != nil {
|
||||
return j.Result, err
|
||||
}
|
||||
return j.Result, workErr
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fixtureControl struct {
|
||||
services []Service
|
||||
calls []string
|
||||
}
|
||||
|
||||
func (c *fixtureControl) Inventory(context.Context) ([]Service, error) {
|
||||
return append([]Service(nil), c.services...), nil
|
||||
}
|
||||
func (c *fixtureControl) Stop(_ context.Context, id string) error {
|
||||
c.calls = append(c.calls, "stop:"+id)
|
||||
for i := range c.services {
|
||||
if c.services[i].ID == id {
|
||||
c.services[i].Running = false
|
||||
c.services[i].Healthy = false
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c *fixtureControl) Start(_ context.Context, id string) error {
|
||||
c.calls = append(c.calls, "start:"+id)
|
||||
for i := range c.services {
|
||||
if c.services[i].ID == id {
|
||||
c.services[i].Running = true
|
||||
c.services[i].Healthy = true
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func freezeFixture(t *testing.T) (string, *fixtureControl, *Coordinator) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
if err := os.Chmod(root, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := &fixtureControl{services: []Service{{ID: "app-a", Running: true, Healthy: true}, {ID: "app-b"}}}
|
||||
engine, err := Open(root, "test-source", c)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { engine.Close() })
|
||||
return root, c, engine
|
||||
}
|
||||
func TestFreezeRestoresOnlyServicesOriginallyRunning(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
captures := 0
|
||||
result, err := e.Run(context.Background(), "job-a", func(ctx context.Context) (string, error) {
|
||||
captures++
|
||||
for _, s := range c.services {
|
||||
if s.Running {
|
||||
t.Fatal("capture started while writer running")
|
||||
}
|
||||
}
|
||||
return strings.Repeat("a", 64), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.State != "succeeded" || result.SnapshotID != strings.Repeat("a", 64) || captures != 1 {
|
||||
t.Fatalf("result=%+v captures=%d", result, captures)
|
||||
}
|
||||
if strings.Join(c.calls, ",") != "stop:app-a,start:app-a" {
|
||||
t.Fatalf("calls=%v", c.calls)
|
||||
}
|
||||
replay, err := e.Run(context.Background(), "job-a", func(context.Context) (string, error) { t.Fatal("replay captured again"); return "", nil })
|
||||
if err != nil || replay.SnapshotID != result.SnapshotID {
|
||||
t.Fatalf("replay=%+v err=%v", replay, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFreezePreflightRejectsWithoutStoppingAndReplaySkipsIt(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
_, err := e.RunPrepared(context.Background(), "prepared", func(context.Context) error { return errors.New("tasks active") }, func(context.Context) (string, error) { t.Fatal("capture before ready"); return "", nil })
|
||||
if err == nil || len(c.calls) != 0 {
|
||||
t.Fatal("preflight did not protect services")
|
||||
}
|
||||
r, err := e.RunPrepared(context.Background(), "prepared", func(context.Context) error { return nil }, func(context.Context) (string, error) { return strings.Repeat("a", 64), nil })
|
||||
if err != nil || r.State != "succeeded" {
|
||||
t.Fatalf("ready capture: %+v %v", r, err)
|
||||
}
|
||||
_, err = e.RunPrepared(context.Background(), "prepared", func(context.Context) error { t.Fatal("completed job consulted changing source"); return nil }, func(context.Context) (string, error) { t.Fatal("completed job captured again"); return "", nil })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFreezeRecoveryAfterProcessExit(t *testing.T) {
|
||||
if os.Getenv("KG_FREEZE_CHILD") == "exit-after-stop" {
|
||||
c := &fixtureControl{services: []Service{{ID: "app-a", Running: true, Healthy: true}, {ID: "app-b"}}}
|
||||
e, err := Open(os.Getenv("KG_FREEZE_ROOT"), "test-source", c)
|
||||
if err != nil {
|
||||
os.Exit(25)
|
||||
}
|
||||
_, _ = e.Run(context.Background(), "crashed-job", func(context.Context) (string, error) { os.Exit(27); return "", nil })
|
||||
os.Exit(26)
|
||||
}
|
||||
root, c, e := freezeFixture(t)
|
||||
cmd := exec.Command(os.Args[0], "-test.run=^TestFreezeRecoveryAfterProcessExit$")
|
||||
cmd.Env = append(os.Environ(), "KG_FREEZE_CHILD=exit-after-stop", "KG_FREEZE_ROOT="+root)
|
||||
err := cmd.Run()
|
||||
var exit *exec.ExitError
|
||||
if !errors.As(err, &exit) || exit.ExitCode() != 27 {
|
||||
t.Fatalf("child did not exit during capture: %v", err)
|
||||
}
|
||||
c.services[0].Running = false
|
||||
c.services[0].Healthy = false
|
||||
if _, err = e.Run(context.Background(), "new-job", func(context.Context) (string, error) { t.Fatal("new capture before recovery"); return "", nil }); err == nil {
|
||||
t.Fatal("unfinished job did not block")
|
||||
}
|
||||
if err = e.Recover(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
j, err := e.load("crashed-job")
|
||||
if err != nil || j.State != "failed" || j.Code != "interrupted" {
|
||||
t.Fatalf("journal=%+v err=%v", j, err)
|
||||
}
|
||||
if strings.Join(c.calls, ",") != "start:app-a" {
|
||||
t.Fatalf("recovery calls=%v", c.calls)
|
||||
}
|
||||
if err = e.Recover(context.Background()); err != nil || len(c.calls) != 1 {
|
||||
t.Fatalf("recovery is not idempotent: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package libraryfreeze
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type pendingStopControl struct{ *fixtureControl }
|
||||
|
||||
func (c *pendingStopControl) Stop(context.Context, string) error {
|
||||
return errors.New("stop may complete later")
|
||||
}
|
||||
func TestFreezeUncertainStopCannotBecomeTerminalWhileStillRunning(t *testing.T) {
|
||||
_, c, e := freezeFixture(t)
|
||||
e.control = &pendingStopControl{c}
|
||||
_, err := e.Run(context.Background(), "uncertain", func(context.Context) (string, error) { t.Fatal("capture after uncertain stop"); return "", nil })
|
||||
if err == nil {
|
||||
t.Fatal("uncertain stop succeeded")
|
||||
}
|
||||
j, err := e.load("uncertain")
|
||||
if err != nil || terminal(j.State) {
|
||||
t.Fatal("running now was treated as proof that delayed stop cannot happen")
|
||||
}
|
||||
if err = e.Recover(context.Background()); err == nil {
|
||||
t.Fatal("uncertain running stop falsely recovered")
|
||||
}
|
||||
c.services[0].Running = false
|
||||
c.services[0].Healthy = false
|
||||
if err = e.Recover(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !c.services[0].Running {
|
||||
t.Fatal("delayed stop not restored")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,399 @@
|
||||
// Package librarysnapshot stores immutable, content-addressed library artifacts.
|
||||
// It does not authorize releases or imply cross-system snapshot consistency.
|
||||
package librarysnapshot
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"knowledge-governance/internal/releasebundle"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
const Format = "weknora-library-snapshot/v1"
|
||||
const maxManifestBytes = 16 * 1024 * 1024
|
||||
|
||||
type Part struct {
|
||||
Kind string `json:"kind"`
|
||||
Name string `json:"name"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Count int64 `json:"count"`
|
||||
}
|
||||
type Compatibility struct {
|
||||
WeKnoraVersion string `json:"weknora_version"`
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
RetrieverEngine string `json:"retriever_engine"`
|
||||
EmbeddingModel string `json:"embedding_model"`
|
||||
EmbeddingDimension int `json:"embedding_dimension"`
|
||||
IndexMappingSHA256 string `json:"index_mapping_sha256"`
|
||||
}
|
||||
|
||||
type Manifest struct {
|
||||
Compatibility Compatibility `json:"compatibility"`
|
||||
Format string `json:"format"`
|
||||
Scope string `json:"scope"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
Parts []Part `json:"parts"`
|
||||
}
|
||||
type Archive struct {
|
||||
root, blobs, versions *os.Root
|
||||
maxBlobBytes int64
|
||||
}
|
||||
|
||||
func Open(root string, maxBlobBytes int64) (*Archive, error) {
|
||||
if !filepath.IsAbs(root) || maxBlobBytes < 1 || maxBlobBytes == math.MaxInt64 {
|
||||
return nil, errors.New("invalid archive configuration")
|
||||
}
|
||||
info, err := os.Lstat(root)
|
||||
if err != nil || !info.IsDir() || info.Mode().Perm() != 0700 {
|
||||
return nil, errors.New("archive root must be a private real directory")
|
||||
}
|
||||
resolved, err := filepath.EvalSymlinks(root)
|
||||
if err != nil || resolved != filepath.Clean(root) {
|
||||
return nil, errors.New("archive root contains symlinks")
|
||||
}
|
||||
r, err := os.OpenRoot(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a := &Archive{root: r, maxBlobBytes: maxBlobBytes}
|
||||
for _, dir := range []string{"blobs", "versions"} {
|
||||
if err := r.Mkdir(dir, 0700); err != nil && !errors.Is(err, os.ErrExist) {
|
||||
a.Close()
|
||||
return nil, err
|
||||
}
|
||||
info, err := r.Lstat(dir)
|
||||
if err != nil || !info.IsDir() || info.Mode().Perm() != 0700 {
|
||||
a.Close()
|
||||
return nil, errors.New("archive subdirectory must be private and real")
|
||||
}
|
||||
sub, err := r.OpenRoot(dir)
|
||||
if err != nil {
|
||||
a.Close()
|
||||
return nil, err
|
||||
}
|
||||
if dir == "blobs" {
|
||||
a.blobs = sub
|
||||
} else {
|
||||
a.versions = sub
|
||||
}
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
func (a *Archive) Close() error {
|
||||
var err error
|
||||
for _, r := range []*os.Root{a.versions, a.blobs, a.root} {
|
||||
if r != nil {
|
||||
err = errors.Join(err, r.Close())
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// Put streams bytes into an archive-owned file; it never keeps a source path or
|
||||
// signed URL as the recovery dependency. Publishing uses no-overwrite links.
|
||||
func (a *Archive) Put(r io.Reader) (Part, error) {
|
||||
if r == nil {
|
||||
return Part{}, errors.New("nil blob reader")
|
||||
}
|
||||
name, f, err := temporary(a.blobs)
|
||||
if err != nil {
|
||||
return Part{}, err
|
||||
}
|
||||
defer a.blobs.Remove(name)
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(io.MultiWriter(f, h), io.LimitReader(r, a.maxBlobBytes+1))
|
||||
if err != nil {
|
||||
return Part{}, errors.New("read snapshot blob")
|
||||
}
|
||||
if n > a.maxBlobBytes {
|
||||
return Part{}, errors.New("snapshot blob exceeds configured limit")
|
||||
}
|
||||
if err = f.Sync(); err != nil {
|
||||
return Part{}, err
|
||||
}
|
||||
if err = f.Close(); err != nil {
|
||||
return Part{}, err
|
||||
}
|
||||
p := Part{SHA256: hex.EncodeToString(h.Sum(nil)), Bytes: n}
|
||||
if err = a.blobs.Link(name, p.SHA256); err != nil && !errors.Is(err, os.ErrExist) {
|
||||
return Part{}, err
|
||||
}
|
||||
if err = a.verify(p); err != nil {
|
||||
return Part{}, err
|
||||
}
|
||||
if err = syncRoot(a.blobs); err != nil {
|
||||
return Part{}, err
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// Seal is the commit point: no version is visible until every dependency has
|
||||
// been read and hashed locally. Empty tables are explicit, not "leave old data".
|
||||
func (a *Archive) Seal(m Manifest) (string, error) {
|
||||
data, err := encodeManifest(m)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = a.verifyParts(m.Parts); err != nil {
|
||||
return "", err
|
||||
}
|
||||
id := digest(data)
|
||||
name, f, err := temporary(a.versions)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer a.versions.Remove(name)
|
||||
defer f.Close()
|
||||
if _, err = f.Write(data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = f.Sync(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = f.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = a.versions.Link(name, id+".json"); err != nil && !errors.Is(err, os.ErrExist) {
|
||||
return "", err
|
||||
}
|
||||
if _, err = a.Load(id); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err = syncRoot(a.versions); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// Load revalidates bytes, not just the manifest's claim of recoverability.
|
||||
func (a *Archive) Load(id string) (Manifest, error) {
|
||||
if !validDigest(id) {
|
||||
return Manifest{}, errors.New("invalid snapshot ID")
|
||||
}
|
||||
f, err := regular(a.versions, id+".json", maxManifestBytes)
|
||||
if err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
defer f.Close()
|
||||
data, err := io.ReadAll(io.LimitReader(f, maxManifestBytes+1))
|
||||
if err != nil || len(data) > maxManifestBytes || digest(data) != id {
|
||||
return Manifest{}, errors.New("snapshot manifest digest mismatch")
|
||||
}
|
||||
dec := json.NewDecoder(bytes.NewReader(data))
|
||||
dec.DisallowUnknownFields()
|
||||
var m Manifest
|
||||
if err = dec.Decode(&m); err != nil {
|
||||
return Manifest{}, errors.New("invalid snapshot manifest")
|
||||
}
|
||||
var extra any
|
||||
if dec.Decode(&extra) != io.EOF {
|
||||
return Manifest{}, errors.New("trailing snapshot manifest data")
|
||||
}
|
||||
if _, err = encodeManifest(m); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
if err = a.verifyParts(m.Parts); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func (a *Archive) verifyParts(parts []Part) error {
|
||||
seen := map[string]int64{}
|
||||
counts := map[string]int64{}
|
||||
resourceIDs := map[string]bool{}
|
||||
archivedResources := map[string]bool{}
|
||||
for _, p := range parts {
|
||||
if size, ok := seen[p.SHA256]; ok {
|
||||
if size != p.Bytes {
|
||||
return errors.New("conflicting blob sizes")
|
||||
}
|
||||
} else {
|
||||
if err := a.verify(p); err != nil {
|
||||
return err
|
||||
}
|
||||
seen[p.SHA256] = p.Bytes
|
||||
}
|
||||
if p.Kind == "resource" {
|
||||
archivedResources[p.Name] = true
|
||||
continue
|
||||
}
|
||||
count, known := counts[p.SHA256]
|
||||
if !known || p.Kind == "database" && p.Name == "resources" {
|
||||
var err error
|
||||
count, err = a.countRows(p, resourceIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
counts[p.SHA256] = count
|
||||
}
|
||||
if count != p.Count {
|
||||
return errors.New("snapshot count does not match archived rows")
|
||||
}
|
||||
}
|
||||
if len(resourceIDs) != len(archivedResources) {
|
||||
return errors.New("resource archive dependencies incomplete")
|
||||
}
|
||||
for id := range resourceIDs {
|
||||
if !archivedResources[id] {
|
||||
return errors.New("resource bytes not archived")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *Archive) countRows(p Part, resources map[string]bool) (int64, error) {
|
||||
f, err := regular(a.blobs, p.SHA256, a.maxBlobBytes)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
dec := json.NewDecoder(io.LimitReader(f, a.maxBlobBytes+1))
|
||||
token, err := dec.Token()
|
||||
if err != nil || token != json.Delim('[') {
|
||||
return 0, errors.New("snapshot component must be a JSON row array")
|
||||
}
|
||||
var n int64
|
||||
for dec.More() {
|
||||
var row map[string]json.RawMessage
|
||||
if err = dec.Decode(&row); err != nil || row == nil {
|
||||
return 0, errors.New("invalid archived row")
|
||||
}
|
||||
n++
|
||||
if p.Kind == "database" && p.Name == "resources" {
|
||||
var id string
|
||||
if json.Unmarshal(row["id"], &id) != nil || !identifier(id) || resources[id] {
|
||||
return 0, errors.New("invalid resource archive identity")
|
||||
}
|
||||
resources[id] = true
|
||||
}
|
||||
}
|
||||
if _, err = dec.Token(); err != nil {
|
||||
return 0, errors.New("incomplete archived row array")
|
||||
}
|
||||
var extra any
|
||||
if dec.Decode(&extra) != io.EOF {
|
||||
return 0, errors.New("trailing archived row data")
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
func (a *Archive) verify(p Part) error {
|
||||
if !validDigest(p.SHA256) || p.Bytes < 0 || p.Bytes > a.maxBlobBytes {
|
||||
return errors.New("invalid snapshot blob descriptor")
|
||||
}
|
||||
f, err := regular(a.blobs, p.SHA256, a.maxBlobBytes)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, io.LimitReader(f, a.maxBlobBytes+1))
|
||||
if err != nil || n != p.Bytes || hex.EncodeToString(h.Sum(nil)) != p.SHA256 {
|
||||
return errors.New("snapshot blob integrity mismatch")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func regular(root *os.Root, name string, limit int64) (*os.File, error) {
|
||||
f, err := root.OpenFile(name, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
|
||||
if err != nil {
|
||||
return nil, errors.New("snapshot file unavailable")
|
||||
}
|
||||
st, err := f.Stat()
|
||||
if err != nil || !st.Mode().IsRegular() || st.Mode().Perm() != 0600 || st.Size() > limit {
|
||||
f.Close()
|
||||
return nil, errors.New("unsafe snapshot file")
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
func temporary(root *os.Root) (string, *os.File, error) {
|
||||
var random [16]byte
|
||||
if _, err := rand.Read(random[:]); err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
name := ".pending-" + hex.EncodeToString(random[:])
|
||||
f, err := root.OpenFile(name, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
|
||||
return name, f, err
|
||||
}
|
||||
func syncRoot(root *os.Root) error {
|
||||
f, err := root.Open(".")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
return f.Sync()
|
||||
}
|
||||
func digest(data []byte) string { h := sha256.Sum256(data); return hex.EncodeToString(h[:]) }
|
||||
func validDigest(s string) bool {
|
||||
if len(s) != 64 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
func identifier(s string) bool {
|
||||
if len(s) == 0 || len(s) > 128 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c == '-' || c == '_' || c == '.') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return s != "." && s != ".."
|
||||
}
|
||||
|
||||
func encodeManifest(m Manifest) ([]byte, error) {
|
||||
c := m.Compatibility
|
||||
if c.WeKnoraVersion != releasebundle.WeKnoraVersionV071 || c.SchemaVersion != releasebundle.WeKnoraMigrationV071 || c.RetrieverEngine != "elasticsearch_v8" || !identifier(c.EmbeddingModel) || c.EmbeddingDimension != releasebundle.EmbeddingDimensionV071 || !validDigest(c.IndexMappingSHA256) {
|
||||
return nil, errors.New("unsupported snapshot compatibility contract")
|
||||
}
|
||||
if m.Format != Format || (m.Scope != "internal" && m.Scope != "external") || m.TenantID == 0 || !identifier(m.KnowledgeBaseID) || m.CreatedAt.IsZero() {
|
||||
return nil, errors.New("invalid library manifest identity")
|
||||
}
|
||||
required := map[string]bool{}
|
||||
for _, n := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} {
|
||||
required["database/"+n] = true
|
||||
}
|
||||
for _, n := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
|
||||
required["wiki/"+n] = true
|
||||
}
|
||||
required["index/chunks"] = true
|
||||
seen := map[string]bool{}
|
||||
for _, p := range m.Parts {
|
||||
key := p.Kind + "/" + p.Name
|
||||
if (!required[key] && !(p.Kind == "resource" && identifier(p.Name) && p.Count == 1)) || seen[key] || !validDigest(p.SHA256) || p.Bytes < 0 || p.Count < 0 {
|
||||
return nil, errors.New("invalid or nonknowledge snapshot component")
|
||||
}
|
||||
seen[key] = true
|
||||
}
|
||||
for key := range required {
|
||||
if !seen[key] {
|
||||
return nil, errors.New("incomplete library snapshot")
|
||||
}
|
||||
}
|
||||
m.Parts = append([]Part(nil), m.Parts...)
|
||||
sort.Slice(m.Parts, func(i, j int) bool { return m.Parts[i].Kind+"/"+m.Parts[i].Name < m.Parts[j].Kind+"/"+m.Parts[j].Name })
|
||||
data, err := json.Marshal(m)
|
||||
if err != nil || len(data) > maxManifestBytes {
|
||||
return nil, errors.New("snapshot manifest too large")
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package librarysnapshot
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestArchiveSealsCompleteVersionAndLoadsAfterReopen(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := os.Chmod(root, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, err := Open(root, 1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
part, err := a.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := testManifest(part)
|
||||
id, err := a.Seal(m)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = a.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, err = Open(root, 1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer a.Close()
|
||||
got, err := a.Load(id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got.Parts) != 11 || got.KnowledgeBaseID != m.KnowledgeBaseID {
|
||||
t.Fatal("incomplete reopened manifest")
|
||||
}
|
||||
again, err := a.Seal(m)
|
||||
if err != nil || again != id {
|
||||
t.Fatal("retry changed immutable identity")
|
||||
}
|
||||
files, err := os.ReadDir(filepath.Join(root, "blobs"))
|
||||
if err != nil || len(files) != 1 {
|
||||
t.Fatal("identical bytes were not shared")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveRejectsMissingCorruptAndNonknowledgeParts(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
_ = os.Chmod(root, 0700)
|
||||
a, err := Open(root, 1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer a.Close()
|
||||
p, err := a.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
mutate func(*Manifest)
|
||||
}{
|
||||
{"unsupported schema", func(m *Manifest) { m.Compatibility.SchemaVersion = 75 }},
|
||||
{"missing mapping contract", func(m *Manifest) { m.Compatibility.IndexMappingSHA256 = "" }},
|
||||
{"missing Wiki", func(m *Manifest) { m.Parts = m.Parts[:6] }},
|
||||
{"duplicate table", func(m *Manifest) { m.Parts = append(m.Parts, m.Parts[0]) }},
|
||||
{"account table", func(m *Manifest) { m.Parts[0].Name = "users" }},
|
||||
{"bad scope", func(m *Manifest) { m.Scope = "both" }},
|
||||
{"wrong size", func(m *Manifest) { m.Parts[0].Bytes++ }},
|
||||
{"missing blob", func(m *Manifest) { m.Parts[0].SHA256 = string(bytes.Repeat([]byte("a"), 64)) }},
|
||||
{"path traversal", func(m *Manifest) { m.Parts[0].SHA256 = "../../escape" }},
|
||||
{"negative count", func(m *Manifest) { m.Parts[0].Count = -1 }},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
m := testManifest(p)
|
||||
test.mutate(&m)
|
||||
if _, err := a.Seal(m); err == nil {
|
||||
t.Fatal("unsafe manifest sealed")
|
||||
}
|
||||
})
|
||||
}
|
||||
m := testManifest(p)
|
||||
id, err := a.Seal(m)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(root, "blobs", p.SHA256), []byte("xx"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := a.Load(id); err == nil {
|
||||
t.Fatal("corrupt blob accepted as recoverable")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveRefusesInventedCountsAndMissingResourceBytes(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
_ = os.Chmod(root, 0700)
|
||||
a, err := Open(root, 1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer a.Close()
|
||||
p, err := a.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := testManifest(p)
|
||||
m.Parts[0].Count = 2
|
||||
if _, err = a.Seal(m); err == nil {
|
||||
t.Fatal("manifest accepted invented progress total")
|
||||
}
|
||||
p, err = a.Put(bytes.NewBufferString(`[{"id":"res-safe"}]`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m = testManifest(p)
|
||||
for i := range m.Parts {
|
||||
m.Parts[i].Count = 1
|
||||
}
|
||||
if _, err = a.Seal(m); err == nil {
|
||||
t.Fatal("resource rows have no archived resource bytes")
|
||||
}
|
||||
}
|
||||
|
||||
func testManifest(part Part) Manifest {
|
||||
m := Manifest{Compatibility: Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "fixture-model", EmbeddingDimension: 1024, IndexMappingSHA256: digest([]byte("fixture-mapping"))}, Format: "weknora-library-snapshot/v1", Scope: "internal", TenantID: 10000, KnowledgeBaseID: "kb-safe", CreatedAt: time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC)}
|
||||
for _, name := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations"} {
|
||||
p := part
|
||||
p.Kind = "database"
|
||||
p.Name = name
|
||||
m.Parts = append(m.Parts, p)
|
||||
}
|
||||
for _, name := range []string{"wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
|
||||
p := part
|
||||
p.Kind = "wiki"
|
||||
p.Name = name
|
||||
m.Parts = append(m.Parts, p)
|
||||
}
|
||||
p := part
|
||||
p.Kind = "index"
|
||||
p.Name = "chunks"
|
||||
m.Parts = append(m.Parts, p)
|
||||
return m
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package librarysnapshot
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"io"
|
||||
"math"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Usage struct {
|
||||
VersionID string `json:"version_id"`
|
||||
LogicalBytes int64 `json:"logical_bytes"`
|
||||
UniqueBytes int64 `json:"unique_bytes"`
|
||||
SharedBytes int64 `json:"shared_bytes"`
|
||||
ExclusiveBytes int64 `json:"exclusive_bytes"`
|
||||
}
|
||||
|
||||
// OpenPart provides archive-owned recovery bytes, never a source-side locator.
|
||||
// Verify and rewind the same descriptor, so path replacement cannot swap the
|
||||
// verified file for an unchecked dependency between these two operations.
|
||||
func (a *Archive) OpenPart(p Part) (io.ReadCloser, error) {
|
||||
if !validDigest(p.SHA256) || p.Bytes < 0 || p.Bytes > a.maxBlobBytes {
|
||||
return nil, errors.New("invalid recovery blob")
|
||||
}
|
||||
f, err := regular(a.blobs, p.SHA256, a.maxBlobBytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h := sha256.New()
|
||||
n, err := io.Copy(h, io.LimitReader(f, a.maxBlobBytes+1))
|
||||
if err != nil || n != p.Bytes || hex.EncodeToString(h.Sum(nil)) != p.SHA256 {
|
||||
f.Close()
|
||||
return nil, errors.New("corrupt recovery blob")
|
||||
}
|
||||
if _, err = f.Seek(0, io.SeekStart); err != nil {
|
||||
f.Close()
|
||||
return nil, err
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// Usage reports content bytes, not filesystem allocation or reclaimed space.
|
||||
// It is deliberately read-only: protection/retention and live-job references
|
||||
// must be acquired from the coordinator before any future cleanup operation.
|
||||
func (a *Archive) Usage() ([]Usage, error) {
|
||||
ids, err := a.versionIDs()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
type inventory struct {
|
||||
usage Usage
|
||||
blobs map[string]int64
|
||||
}
|
||||
inventories := make([]inventory, 0, len(ids))
|
||||
refs := map[string]int{}
|
||||
for _, id := range ids {
|
||||
m, err := a.Load(id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v := inventory{usage: Usage{VersionID: id}, blobs: map[string]int64{}}
|
||||
for _, p := range m.Parts {
|
||||
if v.usage.LogicalBytes > math.MaxInt64-p.Bytes {
|
||||
return nil, errors.New("snapshot usage overflow")
|
||||
}
|
||||
v.usage.LogicalBytes += p.Bytes
|
||||
v.blobs[p.SHA256] = p.Bytes
|
||||
}
|
||||
for sha := range v.blobs {
|
||||
refs[sha]++
|
||||
}
|
||||
inventories = append(inventories, v)
|
||||
}
|
||||
after, err := a.versionIDs()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !slices.Equal(ids, after) {
|
||||
return nil, errors.New("snapshot inventory changed; retry usage read")
|
||||
}
|
||||
result := make([]Usage, 0, len(inventories))
|
||||
for _, v := range inventories {
|
||||
for sha, size := range v.blobs {
|
||||
v.usage.UniqueBytes += size
|
||||
if refs[sha] > 1 {
|
||||
v.usage.SharedBytes += size
|
||||
} else {
|
||||
v.usage.ExclusiveBytes += size
|
||||
}
|
||||
}
|
||||
result = append(result, v.usage)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (a *Archive) versionIDs() ([]string, error) {
|
||||
f, err := a.versions.Open(".")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
entries, err := f.ReadDir(-1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if strings.HasPrefix(entry.Name(), ".pending-") {
|
||||
continue
|
||||
}
|
||||
name := entry.Name()
|
||||
id := strings.TrimSuffix(name, ".json")
|
||||
if !strings.HasSuffix(name, ".json") || !validDigest(id) || entry.IsDir() {
|
||||
return nil, errors.New("unexpected archive version entry")
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
return ids, nil
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
package librarysnapshot
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestArchiveRecoveryIndependentOfSourceAndUsageByReferences(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
_ = os.Chmod(root, 0700)
|
||||
a, err := Open(root, 1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer a.Close()
|
||||
source := filepath.Join(t.TempDir(), "source")
|
||||
if err := os.WriteFile(source, []byte("first"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f, err := os.Open(source)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
blob, err := a.Put(f)
|
||||
f.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = os.Remove(source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reader, err := a.OpenPart(blob)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, err := io.ReadAll(reader)
|
||||
reader.Close()
|
||||
if err != nil || string(restored) != "first" {
|
||||
t.Fatal("restore still depends on source")
|
||||
}
|
||||
empty, err := a.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := testManifest(empty)
|
||||
first, err := a.Seal(m)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.CreatedAt = m.CreatedAt.Add(time.Second)
|
||||
second, err := a.Seal(m)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := a.Usage()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("versions=%d", len(got))
|
||||
}
|
||||
for _, u := range got {
|
||||
if u.VersionID != first && u.VersionID != second {
|
||||
t.Fatal("unexpected version")
|
||||
}
|
||||
if u.LogicalBytes != 22 || u.UniqueBytes != 2 || u.SharedBytes != 2 || u.ExclusiveBytes != 0 {
|
||||
t.Fatalf("bad shared accounting: %+v", u)
|
||||
}
|
||||
}
|
||||
// A digest with no manifest reference must not inflate either version.
|
||||
if len(restored) != 5 {
|
||||
t.Fatal("fixture unexpected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchiveConcurrentWritersAndReadSafety(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
_ = os.Chmod(root, 0700)
|
||||
a, err := Open(root, 16)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer a.Close()
|
||||
b, err := Open(root, 16)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer b.Close()
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 12; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
archive := a
|
||||
if i%2 == 0 {
|
||||
archive = b
|
||||
}
|
||||
p, err := archive.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
if _, err = archive.Seal(testManifest(p)); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
got, err := a.Usage()
|
||||
if err != nil || len(got) != 1 {
|
||||
t.Fatalf("concurrent publication: err=%v versions=%d", err, len(got))
|
||||
}
|
||||
if _, err = a.Put(bytes.NewBufferString("this is more than sixteen bytes")); err == nil {
|
||||
t.Fatal("oversized blob accepted")
|
||||
}
|
||||
p, err := a.Put(bytes.NewBufferString("[]"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
target := filepath.Join(root, "blobs", p.SHA256)
|
||||
if err = os.Remove(target); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = os.Symlink(filepath.Join(t.TempDir(), "outside"), target); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = a.OpenPart(p); err == nil {
|
||||
t.Fatal("symlink followed")
|
||||
}
|
||||
if _, err = a.Usage(); err == nil {
|
||||
t.Fatal("corrupt dependency reported as safe usage")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// ArchiveLibraryCandidate persists independent recovery bytes. "Candidate" is
|
||||
// intentional: matching IDs/counts does not prove shared Wiki semantic retraction
|
||||
// or a common cross-system cut. No release worker consumes it as authorization.
|
||||
func ArchiveLibraryCandidate(ctx context.Context, archive *librarysnapshot.Archive, scope string, identity PostgresIdentity, contract librarysnapshot.Compatibility, pg PostgresSnapshot, es SearchSnapshot, objects ObjectReader, createdAt time.Time) (string, error) {
|
||||
if archive == nil || objects == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) || identity.KnowledgeID != "" {
|
||||
return "", errors.New("invalid library candidate configuration")
|
||||
}
|
||||
if err := validateLibraryCandidate(identity, pg, es); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if es.MappingSHA256 != contract.IndexMappingSHA256 {
|
||||
return "", errors.New("library mapping contract mismatch")
|
||||
}
|
||||
manifest := librarysnapshot.Manifest{Format: librarysnapshot.Format, Scope: scope, TenantID: identity.TenantID, KnowledgeBaseID: identity.KnowledgeBaseID, CreatedAt: createdAt, Compatibility: contract}
|
||||
put := func(kind, name string, count int64, data []byte) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
p, err := archive.Put(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p.Kind = kind
|
||||
p.Name = name
|
||||
p.Count = count
|
||||
manifest.Parts = append(manifest.Parts, p)
|
||||
return nil
|
||||
}
|
||||
for _, table := range pg.Tables {
|
||||
rows := table.Rows
|
||||
if rows == nil {
|
||||
rows = []json.RawMessage{}
|
||||
}
|
||||
data, err := json.Marshal(rows)
|
||||
if err != nil {
|
||||
return "", errors.New("encode library rows")
|
||||
}
|
||||
kind := "database"
|
||||
if strings.HasPrefix(table.Name, "wiki_") {
|
||||
kind = "wiki"
|
||||
}
|
||||
if err = put(kind, table.Name, int64(len(rows)), data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
documents := es.Documents
|
||||
if documents == nil {
|
||||
documents = []SearchDocument{}
|
||||
}
|
||||
data, err := json.Marshal(documents)
|
||||
if err != nil {
|
||||
return "", errors.New("encode library index")
|
||||
}
|
||||
if err = put("index", "chunks", int64(len(documents)), data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, raw := range pg.Table("resources").Rows {
|
||||
if err = ctx.Err(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
var resource struct {
|
||||
ID string `json:"id"`
|
||||
Provider string `json:"provider"`
|
||||
PhysicalPath string `json:"physical_path"`
|
||||
Size *int64 `json:"size"`
|
||||
ContentHash string `json:"content_hash"`
|
||||
}
|
||||
if json.Unmarshal(raw, &resource) != nil || resource.Size == nil || *resource.Size < 0 || !validObjectProvider(resource.Provider) || !validObjectKey(resource.PhysicalPath) || resource.ContentHash != "" && !validLowerSHA256(resource.ContentHash) {
|
||||
return "", errors.New("invalid library resource metadata")
|
||||
}
|
||||
reader, err := objects.Open(ctx, resource.Provider, resource.PhysicalPath)
|
||||
if err != nil {
|
||||
return "", errors.New("read library resource")
|
||||
}
|
||||
part, readErr := archive.Put(reader)
|
||||
closeErr := reader.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
return "", errors.New("archive library resource")
|
||||
}
|
||||
if part.Bytes != *resource.Size || resource.ContentHash != "" && part.SHA256 != resource.ContentHash {
|
||||
return "", errors.New("library resource integrity mismatch")
|
||||
}
|
||||
part.Kind = "resource"
|
||||
part.Name = resource.ID
|
||||
part.Count = 1
|
||||
manifest.Parts = append(manifest.Parts, part)
|
||||
}
|
||||
if err = ctx.Err(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return archive.Seal(manifest)
|
||||
}
|
||||
|
||||
type libraryRow struct {
|
||||
ID string `json:"id"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id"`
|
||||
KnowledgeID string `json:"knowledge_id"`
|
||||
ChunkID string `json:"chunk_id"`
|
||||
TagID string `json:"tag_id"`
|
||||
ResourceID string `json:"resource_id"`
|
||||
Handle string `json:"handle"`
|
||||
OwnerType string `json:"owner_type"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
func validateLibraryCandidate(identity PostgresIdentity, pg PostgresSnapshot, es SearchSnapshot) error {
|
||||
required := map[string]bool{"resources": true}
|
||||
for _, entry := range libraryContentQueries {
|
||||
required[entry.name] = true
|
||||
}
|
||||
tables := map[string]map[string]libraryRow{}
|
||||
for _, table := range pg.Tables {
|
||||
if !required[table.Name] || tables[table.Name] != nil || table.RowCount != len(table.Rows) || table.SHA256 != newTableSnapshot(table.Name, table.Rows).SHA256 {
|
||||
return errors.New("invalid library table inventory")
|
||||
}
|
||||
rows := map[string]libraryRow{}
|
||||
tables[table.Name] = rows
|
||||
for _, raw := range table.Rows {
|
||||
// Log identities are BIGSERIAL. All other owned rows use string IDs.
|
||||
if table.Name == "wiki_log_entries" {
|
||||
var row struct {
|
||||
ID json.Number `json:"id"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id"`
|
||||
}
|
||||
if json.Unmarshal(raw, &row) != nil || row.ID == "" || row.TenantID != identity.TenantID || row.KnowledgeBaseID != identity.KnowledgeBaseID {
|
||||
return errors.New("invalid Wiki log scope")
|
||||
}
|
||||
if _, ok := rows[string(row.ID)]; ok {
|
||||
return errors.New("duplicate Wiki log identity")
|
||||
}
|
||||
rows[string(row.ID)] = libraryRow{}
|
||||
continue
|
||||
}
|
||||
var row libraryRow
|
||||
if json.Unmarshal(raw, &row) != nil {
|
||||
return errors.New("invalid library content row")
|
||||
}
|
||||
if table.Name == "knowledge_tag_relations" {
|
||||
if !validSearchIdentifier(row.KnowledgeID) || !validSearchIdentifier(row.TagID) {
|
||||
return errors.New("invalid library tag relation")
|
||||
}
|
||||
row.ID = row.KnowledgeID + "/" + row.TagID
|
||||
} else {
|
||||
if !validSearchIdentifier(row.ID) || row.TenantID != identity.TenantID {
|
||||
return errors.New("invalid library row identity")
|
||||
}
|
||||
if table.Name != "resources" && table.Name != "resource_bindings" && row.KnowledgeBaseID != identity.KnowledgeBaseID {
|
||||
return errors.New("cross-scope library row")
|
||||
}
|
||||
}
|
||||
if _, ok := rows[row.ID]; ok {
|
||||
return errors.New("duplicate library row")
|
||||
}
|
||||
rows[row.ID] = row
|
||||
}
|
||||
}
|
||||
for name := range required {
|
||||
if tables[name] == nil {
|
||||
return errors.New("missing library table")
|
||||
}
|
||||
}
|
||||
for _, chunk := range tables["chunks"] {
|
||||
if _, ok := tables["knowledges"][chunk.KnowledgeID]; !ok {
|
||||
return errors.New("orphan library chunk")
|
||||
}
|
||||
}
|
||||
for _, relation := range tables["knowledge_tag_relations"] {
|
||||
_, hasKnowledge := tables["knowledges"][relation.KnowledgeID]
|
||||
_, hasTag := tables["knowledge_tags"][relation.TagID]
|
||||
if !hasKnowledge || !hasTag {
|
||||
return errors.New("orphan library tag relation")
|
||||
}
|
||||
}
|
||||
for _, binding := range tables["resource_bindings"] {
|
||||
_, hasResource := tables["resources"][binding.ResourceID]
|
||||
hasOwner := false
|
||||
if binding.OwnerType == "knowledge" {
|
||||
_, hasOwner = tables["knowledges"][binding.OwnerID]
|
||||
} else if binding.OwnerType == "chunk" {
|
||||
_, hasOwner = tables["chunks"][binding.OwnerID]
|
||||
}
|
||||
if !hasResource || !hasOwner {
|
||||
return errors.New("orphan library resource binding")
|
||||
}
|
||||
}
|
||||
handles := map[string]bool{}
|
||||
for _, resource := range tables["resources"] {
|
||||
if len(resource.Handle) != 22 || handles[resource.Handle] {
|
||||
return errors.New("invalid resource handle")
|
||||
}
|
||||
handles[resource.Handle] = true
|
||||
}
|
||||
var bodies []json.RawMessage
|
||||
for _, name := range []string{"knowledges", "chunks", "wiki_pages"} {
|
||||
bodies = append(bodies, pg.Table(name).Rows...)
|
||||
}
|
||||
for _, handle := range referencedResourceHandles(bodies) {
|
||||
if !handles[handle] {
|
||||
return errors.New("unresolved library resource")
|
||||
}
|
||||
}
|
||||
sha, err := searchDocumentsDigest(es.Documents)
|
||||
if err != nil || !validLowerSHA256(es.MappingSHA256) || sha != es.DocumentsSHA256 {
|
||||
return errors.New("invalid library index digest")
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, document := range es.Documents {
|
||||
var row libraryRow
|
||||
if json.Unmarshal(document.Source, &row) != nil || !validSearchIdentifier(document.ID) || seen[document.ID] {
|
||||
return errors.New("invalid library search document")
|
||||
}
|
||||
seen[document.ID] = true
|
||||
chunk, ok := tables["chunks"][row.ChunkID]
|
||||
if !ok || row.KnowledgeID != chunk.KnowledgeID || row.Content != chunk.Content {
|
||||
return errors.New("library index does not match database chunks")
|
||||
}
|
||||
if !searchHitMatches(document.Source, SearchIdentity{TenantID: identity.TenantID, KnowledgeBaseID: identity.KnowledgeBaseID, KnowledgeID: row.KnowledgeID}) {
|
||||
return errors.New("cross-scope library index")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
func TestArchiveLibraryCandidateOwnsEveryLaneAndRejectsMismatch(t *testing.T) {
|
||||
for _, mode := range []string{"complete", "missing-resource", "bad-resource-hash", "cross-scope", "orphan-search", "missing-wiki", "search-digest", "duplicate-table"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
_ = os.Chmod(root, 0700)
|
||||
archive, err := librarysnapshot.Open(root, 1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
pg, es, reader := libraryCandidateFixture()
|
||||
switch mode {
|
||||
case "missing-resource":
|
||||
reader.data = nil
|
||||
case "bad-resource-hash":
|
||||
reader.data = []byte("wrong")
|
||||
case "cross-scope":
|
||||
pg.Tables[0] = newTableSnapshot("knowledges", []json.RawMessage{json.RawMessage(`{"id":"doc-safe","tenant_id":20000,"knowledge_base_id":"kb-safe","parse_status":"completed"}`)})
|
||||
case "orphan-search":
|
||||
es.Documents[0].Source = json.RawMessage(`{"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe","chunk_id":"unknown"}`)
|
||||
es.DocumentsSHA256, _ = searchDocumentsDigest(es.Documents)
|
||||
case "missing-wiki":
|
||||
pg.Tables = pg.Tables[:6]
|
||||
case "search-digest":
|
||||
es.DocumentsSHA256 = shaBytes([]byte("wrong"))
|
||||
case "duplicate-table":
|
||||
pg.Tables = append(pg.Tables, pg.Tables[0])
|
||||
}
|
||||
contract := librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "fixture-model", EmbeddingDimension: 1024, IndexMappingSHA256: es.MappingSHA256}
|
||||
id, err := ArchiveLibraryCandidate(context.Background(), archive, "internal", PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, contract, pg, es, reader, time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC))
|
||||
if mode != "complete" {
|
||||
if err == nil {
|
||||
t.Fatal("inconsistent candidate archived")
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reader.data = nil
|
||||
saved, err := archive.Load(id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(saved.Parts) != 12 {
|
||||
t.Fatalf("parts=%d", len(saved.Parts))
|
||||
}
|
||||
for _, part := range saved.Parts {
|
||||
if part.Kind == "resource" {
|
||||
r, err := archive.OpenPart(part)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := io.ReadAll(r)
|
||||
r.Close()
|
||||
if err != nil || string(data) != "file" {
|
||||
t.Fatal("resource not independently recoverable")
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type libraryCandidateReader struct{ data []byte }
|
||||
|
||||
func (r *libraryCandidateReader) Open(context.Context, string, string) (io.ReadCloser, error) {
|
||||
return io.NopCloser(bytes.NewReader(r.data)), nil
|
||||
}
|
||||
func libraryCandidateFixture() (PostgresSnapshot, SearchSnapshot, *libraryCandidateReader) {
|
||||
tables := []TableSnapshot{
|
||||
newTableSnapshot("knowledges", []json.RawMessage{json.RawMessage(`{"id":"doc-safe","tenant_id":10000,"knowledge_base_id":"kb-safe","parse_status":"completed","file_path":"resource://ABCDEFGHIJKLMNOPQRSTUV"}`)}),
|
||||
newTableSnapshot("chunks", []json.RawMessage{json.RawMessage(`{"id":"chunk-safe","tenant_id":10000,"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe","content":"chunk"}`)}),
|
||||
newTableSnapshot("resources", []json.RawMessage{json.RawMessage(`{"id":"res-safe","tenant_id":10000,"handle":"ABCDEFGHIJKLMNOPQRSTUV","provider":"oss","physical_path":"oss://fixture/source.txt","size":4}`)}),
|
||||
}
|
||||
for _, name := range []string{"resource_bindings", "knowledge_tags", "knowledge_tag_relations", "wiki_folders", "wiki_pages", "wiki_page_issues", "wiki_log_entries"} {
|
||||
tables = append(tables, newTableSnapshot(name, nil))
|
||||
}
|
||||
es := SearchSnapshot{MappingSHA256: shaBytes([]byte("mapping")), Documents: []SearchDocument{{ID: "es-safe", Source: json.RawMessage(`{"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe","chunk_id":"chunk-safe","content":"chunk"}`)}}}
|
||||
es.DocumentsSHA256, _ = searchDocumentsDigest(es.Documents)
|
||||
return PostgresSnapshot{Tables: tables}, es, &libraryCandidateReader{data: []byte("file")}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/libraryfreeze"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
type LibraryCaptureRequest struct {
|
||||
JobID, Scope string
|
||||
Identity PostgresIdentity
|
||||
Compatibility librarysnapshot.Compatibility
|
||||
CreatedAt time.Time
|
||||
MaxDocuments int
|
||||
}
|
||||
|
||||
// CaptureLibrary connects maintenance to the actual PG/PIT/object archive path.
|
||||
// sourceReady is mandatory and must verify the whole writer group (both scopes,
|
||||
// ingress and background jobs), not only this KB. It runs before stopping and
|
||||
// again inside the stopped window. Deployment must establish that the group
|
||||
// contains every source writer. This function does not certify Wiki semantics,
|
||||
// approve a release, or give the worker any production credentials.
|
||||
func CaptureLibrary(ctx context.Context, freeze *libraryfreeze.Coordinator, archive *librarysnapshot.Archive, db librarySnapshotDB, search *SearchExporter, objects ObjectReader, sourceReady func(context.Context) error, req LibraryCaptureRequest) (libraryfreeze.Result, error) {
|
||||
if freeze == nil || archive == nil || db == nil || search == nil || objects == nil || sourceReady == nil || req.CreatedAt.IsZero() || req.MaxDocuments < 1 || req.Identity.TenantID == 0 || !validSearchIdentifier(req.Identity.KnowledgeBaseID) || req.Identity.KnowledgeID != "" || (req.Scope != "internal" && req.Scope != "external") {
|
||||
return libraryfreeze.Result{}, errors.New("invalid library capture configuration")
|
||||
}
|
||||
result, err := freeze.RunPrepared(ctx, req.JobID, func(ctx context.Context) error {
|
||||
if err := sourceReady(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
// Check readiness before a maintenance window, then repeat the export inside
|
||||
// that window: work can arrive between preflight and the graceful stop.
|
||||
_, err := ExportLibrary(ctx, db, req.Identity)
|
||||
return err
|
||||
}, func(ctx context.Context) (string, error) {
|
||||
if err := sourceReady(ctx); err != nil {
|
||||
return "", err
|
||||
}
|
||||
pg, err := ExportLibrary(ctx, db, req.Identity)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
es, err := search.ExportLibrary(ctx, SearchIdentity{TenantID: req.Identity.TenantID, KnowledgeBaseID: req.Identity.KnowledgeBaseID}, req.MaxDocuments)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return ArchiveLibraryCandidate(ctx, archive, req.Scope, req.Identity, req.Compatibility, pg, es, objects, req.CreatedAt)
|
||||
})
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
// A successful old operation is replayable only while its exact independent
|
||||
// archive still verifies, and never under another KB/scope/model contract.
|
||||
saved, err := archive.Load(result.SnapshotID)
|
||||
if err != nil {
|
||||
return libraryfreeze.Result{}, errors.New("captured library archive no longer verifies")
|
||||
}
|
||||
if saved.Scope != req.Scope || saved.TenantID != req.Identity.TenantID || saved.KnowledgeBaseID != req.Identity.KnowledgeBaseID || saved.Compatibility != req.Compatibility || !saved.CreatedAt.Equal(req.CreatedAt) {
|
||||
return libraryfreeze.Result{}, errors.New("library capture request conflicts with existing job")
|
||||
}
|
||||
for _, part := range saved.Parts {
|
||||
if part.Kind == "index" && part.Count > int64(req.MaxDocuments) {
|
||||
return libraryfreeze.Result{}, errors.New("replayed library exceeds requested document limit")
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/libraryfreeze"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
type captureWriter struct {
|
||||
running atomic.Bool
|
||||
db *libraryFixture
|
||||
late bool
|
||||
stops int
|
||||
}
|
||||
|
||||
func (c *captureWriter) Inventory(context.Context) ([]libraryfreeze.Service, error) {
|
||||
r := c.running.Load()
|
||||
return []libraryfreeze.Service{{ID: "test-writer", Running: r, Healthy: r}}, nil
|
||||
}
|
||||
func (c *captureWriter) Stop(context.Context, string) error {
|
||||
c.stops++
|
||||
c.running.Store(false)
|
||||
if c.late {
|
||||
c.db.tx.pending = 1
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (c *captureWriter) Start(context.Context, string) error { c.running.Store(true); return nil }
|
||||
func TestLibraryCaptureMaintenanceOwnsArchiveAndRestoresOnFailure(t *testing.T) {
|
||||
for _, mode := range []string{"success", "initial-busy", "late-pending", "search-failure"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
db := newLibraryFixture()
|
||||
writer := &captureWriter{db: db, late: mode == "late-pending"}
|
||||
writer.running.Store(true)
|
||||
if mode == "initial-busy" {
|
||||
db.tx.pending = 1
|
||||
}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/chunks/_mapping":
|
||||
fmt.Fprint(w, `{"chunks":{"mappings":{"properties":{"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"tenant_id":{"type":"long"}}}}}`)
|
||||
case "/chunks/_pit":
|
||||
if writer.running.Load() {
|
||||
t.Error("PIT opened before writers stopped")
|
||||
}
|
||||
fmt.Fprint(w, `{"id":"fixed-pit"}`)
|
||||
case "/_search":
|
||||
if mode == "search-failure" {
|
||||
http.Error(w, "fixture failure", 503)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"timed_out":false,"_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"hits":[]}}`)
|
||||
case "/_pit":
|
||||
fmt.Fprint(w, `{"succeeded":true}`)
|
||||
default:
|
||||
http.Error(w, "unexpected", 400)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
search, err := NewSearchExporter(SearchExporterConfig{BaseURL: server.URL, Index: "chunks", PageSize: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mapping, err := search.inspectMapping(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
_ = os.Chmod(root, 0700)
|
||||
freeze, err := libraryfreeze.Open(root, "test-source", writer)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer freeze.Close()
|
||||
archiveRoot := t.TempDir()
|
||||
_ = os.Chmod(archiveRoot, 0700)
|
||||
archive, err := librarysnapshot.Open(archiveRoot, 1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
req := LibraryCaptureRequest{JobID: "capture-job", Scope: "internal", Identity: PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, Compatibility: librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "fixture-model", EmbeddingDimension: 1024, IndexMappingSHA256: mapping.digest}, CreatedAt: time.Date(2026, 10, 27, 0, 0, 0, 0, time.UTC), MaxDocuments: 100}
|
||||
checks := 0
|
||||
ready := func(context.Context) error { checks++; return nil }
|
||||
result, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req)
|
||||
if !writer.running.Load() {
|
||||
t.Fatal("test writer left stopped")
|
||||
}
|
||||
if mode != "success" {
|
||||
if err == nil {
|
||||
t.Fatal("invalid candidate succeeded")
|
||||
}
|
||||
if mode == "initial-busy" && writer.stops != 0 {
|
||||
t.Fatal("busy source stopped")
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || result.State != "succeeded" {
|
||||
t.Fatalf("capture=%+v err=%v", result, err)
|
||||
}
|
||||
saved, err := archive.Load(result.SnapshotID)
|
||||
if err != nil || saved.Scope != "internal" || checks != 2 {
|
||||
t.Fatalf("archive=%+v checks=%d err=%v", saved, checks, err)
|
||||
}
|
||||
db.tx.pending = 1
|
||||
replay, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req)
|
||||
if err != nil || replay.SnapshotID != result.SnapshotID || writer.stops != 1 {
|
||||
t.Fatal("sealed replay consulted current source")
|
||||
}
|
||||
req.Scope = "external"
|
||||
if got, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req); err == nil || got.State == "succeeded" {
|
||||
t.Fatal("conflicting replay returned a successful candidate")
|
||||
}
|
||||
req.Scope = "internal"
|
||||
if err := os.Remove(archiveRoot + "/versions/" + result.SnapshotID + ".json"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, err := CaptureLibrary(context.Background(), freeze, archive, db, search, &libraryCandidateReader{}, ready, req); err == nil || got.State == "succeeded" {
|
||||
t.Fatal("missing archive returned a successful candidate")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
type librarySnapshotDB interface {
|
||||
BeginTx(context.Context, pgx.TxOptions) (pgx.Tx, error)
|
||||
}
|
||||
|
||||
// ExportLibrary reads one PostgreSQL MVCC snapshot, including empty Wiki tables.
|
||||
// It does not certify semantic quality, freeze Elasticsearch/objects, or authorize
|
||||
// publication. Those require a separately verified, complete library version.
|
||||
func ExportLibrary(ctx context.Context, db librarySnapshotDB, identity PostgresIdentity) (PostgresSnapshot, error) {
|
||||
if db == nil || identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) || identity.KnowledgeID != "" {
|
||||
return PostgresSnapshot{}, errors.New("invalid library snapshot scope")
|
||||
}
|
||||
tx, err := db.BeginTx(ctx, pgx.TxOptions{IsoLevel: pgx.RepeatableRead, AccessMode: pgx.ReadOnly})
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, errors.New("begin library snapshot")
|
||||
}
|
||||
defer tx.Rollback(context.WithoutCancel(ctx))
|
||||
var version int
|
||||
var dirty bool
|
||||
if err := tx.QueryRow(ctx, `SELECT version,dirty FROM schema_migrations LIMIT 1`).Scan(&version, &dirty); err != nil || version != 74 || dirty {
|
||||
return PostgresSnapshot{}, errors.New("source library schema contract mismatch")
|
||||
}
|
||||
var graph bool
|
||||
if err := tx.QueryRow(ctx, `SELECT COALESCE((indexing_strategy->>'graph_enabled')::boolean,false) FROM knowledge_bases WHERE tenant_id=$1 AND id=$2 AND deleted_at IS NULL`, identity.TenantID, identity.KnowledgeBaseID).Scan(&graph); err != nil {
|
||||
return PostgresSnapshot{}, errors.New("source library unavailable")
|
||||
}
|
||||
if graph {
|
||||
return PostgresSnapshot{}, errors.New("Neo4j graph snapshot is not supported")
|
||||
}
|
||||
var unsettled, pending, failed int64
|
||||
if err := tx.QueryRow(ctx, librarySettlementSQL, identity.TenantID, identity.KnowledgeBaseID).Scan(&unsettled, &pending, &failed); err != nil {
|
||||
return PostgresSnapshot{}, errors.New("read library settlement")
|
||||
}
|
||||
if unsettled != 0 || pending != 0 || failed != 0 {
|
||||
return PostgresSnapshot{}, errors.New("library has unfinished or failed processing; semantic acceptance is still required after settlement")
|
||||
}
|
||||
exporter := PostgresExporter{db: tx}
|
||||
var snapshot PostgresSnapshot
|
||||
for _, entry := range libraryContentQueries {
|
||||
rows, err := exporter.queryRows(ctx, entry.sql, identity.TenantID, identity.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
snapshot.Tables = append(snapshot.Tables, newTableSnapshot(entry.name, rows))
|
||||
}
|
||||
// Resources can be owned by a document/chunk or only referenced in body text.
|
||||
// The latter includes Wiki manual pages, which document bundles used to miss.
|
||||
var bodies []json.RawMessage
|
||||
for _, name := range []string{"knowledges", "chunks", "wiki_pages"} {
|
||||
bodies = append(bodies, snapshot.Table(name).Rows...)
|
||||
}
|
||||
handles := referencedResourceHandles(bodies)
|
||||
resources, err := exporter.queryRows(ctx, libraryResourcesSQL, identity.TenantID, identity.KnowledgeBaseID, handles)
|
||||
if err != nil {
|
||||
return PostgresSnapshot{}, err
|
||||
}
|
||||
resourceIDs, resourceHandles := map[string]bool{}, map[string]bool{}
|
||||
for _, raw := range resources {
|
||||
var row struct {
|
||||
ID string `json:"id"`
|
||||
Handle string `json:"handle"`
|
||||
TenantID uint64 `json:"tenant_id"`
|
||||
}
|
||||
if json.Unmarshal(raw, &row) != nil || row.ID == "" || row.TenantID != identity.TenantID || resourceIDs[row.ID] || resourceHandles[row.Handle] {
|
||||
return PostgresSnapshot{}, errors.New("invalid library resource identity")
|
||||
}
|
||||
resourceIDs[row.ID] = true
|
||||
resourceHandles[row.Handle] = true
|
||||
}
|
||||
for _, handle := range handles {
|
||||
if !resourceHandles[handle] {
|
||||
return PostgresSnapshot{}, errors.New("library references an unavailable resource")
|
||||
}
|
||||
}
|
||||
for _, raw := range snapshot.Table("resource_bindings").Rows {
|
||||
var row struct {
|
||||
ResourceID string `json:"resource_id"`
|
||||
}
|
||||
if json.Unmarshal(raw, &row) != nil || !resourceIDs[row.ResourceID] {
|
||||
return PostgresSnapshot{}, errors.New("library binding references an unavailable resource")
|
||||
}
|
||||
}
|
||||
snapshot.Tables = append(snapshot.Tables, newTableSnapshot("resources", resources))
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return PostgresSnapshot{}, errors.New("commit library snapshot")
|
||||
}
|
||||
return snapshot, nil
|
||||
}
|
||||
|
||||
// Deleted document retract/finalize tasks are scoped to the KB, not a live
|
||||
// knowledge row. Never join them to only non-deleted documents. Dead letters
|
||||
// have no resolution marker in this schema, so presence blocks automatically.
|
||||
const librarySettlementSQL = `SELECT
|
||||
(SELECT count(*) FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2 AND deleted_at IS NULL AND (parse_status IS DISTINCT FROM 'completed' OR pending_subtasks_count IS NULL OR pending_subtasks_count<>0)),
|
||||
(SELECT count(*) FROM task_pending_ops WHERE tenant_id=$1 AND ((scope='knowledge_base' AND scope_id=$2) OR (scope='knowledge' AND scope_id IN (SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2)))),
|
||||
(SELECT count(*) FROM task_dead_letters WHERE tenant_id=$1 AND ((scope='knowledge_base' AND scope_id=$2) OR (scope='knowledge' AND scope_id IN (SELECT id FROM knowledges WHERE tenant_id=$1 AND knowledge_base_id=$2))))`
|
||||
|
||||
var libraryContentQueries = []struct{ name, sql string }{
|
||||
{"knowledges", `SELECT to_jsonb(k) FROM knowledges k WHERE k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL ORDER BY k.id`},
|
||||
{"chunks", `SELECT to_jsonb(c) FROM chunks c JOIN knowledges k ON k.id=c.knowledge_id AND k.tenant_id=c.tenant_id AND k.knowledge_base_id=c.knowledge_base_id WHERE k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL AND c.deleted_at IS NULL ORDER BY c.id`},
|
||||
{"knowledge_tags", `SELECT to_jsonb(t) FROM knowledge_tags t WHERE t.tenant_id=$1 AND t.knowledge_base_id=$2 AND t.deleted_at IS NULL ORDER BY t.id`},
|
||||
{"knowledge_tag_relations", `SELECT to_jsonb(r) FROM knowledge_tag_relations r JOIN knowledges k ON k.id=r.knowledge_id JOIN knowledge_tags t ON t.id=r.tag_id AND t.tenant_id=k.tenant_id AND t.knowledge_base_id=k.knowledge_base_id WHERE k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL AND t.deleted_at IS NULL ORDER BY r.knowledge_id,r.tag_id`},
|
||||
{"resource_bindings", `SELECT to_jsonb(b) FROM resource_bindings b WHERE b.tenant_id=$1 AND ((b.owner_type='knowledge' AND EXISTS (SELECT 1 FROM knowledges k WHERE k.id=b.owner_id AND k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL)) OR (b.owner_type='chunk' AND EXISTS (SELECT 1 FROM chunks c JOIN knowledges k ON k.id=c.knowledge_id AND k.tenant_id=c.tenant_id AND k.knowledge_base_id=c.knowledge_base_id WHERE c.id=b.owner_id AND c.tenant_id=$1 AND c.knowledge_base_id=$2 AND c.deleted_at IS NULL AND k.deleted_at IS NULL))) ORDER BY b.id`},
|
||||
{"wiki_pages", `SELECT to_jsonb(w) FROM wiki_pages w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 AND w.deleted_at IS NULL ORDER BY w.id`},
|
||||
{"wiki_folders", `SELECT to_jsonb(w) FROM wiki_folders w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 AND w.deleted_at IS NULL ORDER BY w.id`},
|
||||
{"wiki_page_issues", `SELECT to_jsonb(w) FROM wiki_page_issues w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 AND w.deleted_at IS NULL ORDER BY w.id`},
|
||||
{"wiki_log_entries", `SELECT to_jsonb(w) FROM wiki_log_entries w WHERE w.tenant_id=$1 AND w.knowledge_base_id=$2 ORDER BY w.id`},
|
||||
}
|
||||
|
||||
const libraryResourcesSQL = `SELECT to_jsonb(r) FROM resources r WHERE r.tenant_id=$1 AND r.deleted_at IS NULL AND (r.handle=ANY($3::text[]) OR EXISTS (SELECT 1 FROM resource_bindings b WHERE b.resource_id=r.id AND b.tenant_id=$1 AND ((b.owner_type='knowledge' AND EXISTS (SELECT 1 FROM knowledges k WHERE k.id=b.owner_id AND k.tenant_id=$1 AND k.knowledge_base_id=$2 AND k.deleted_at IS NULL)) OR (b.owner_type='chunk' AND EXISTS (SELECT 1 FROM chunks c JOIN knowledges k ON k.id=c.knowledge_id AND k.tenant_id=c.tenant_id AND k.knowledge_base_id=c.knowledge_base_id WHERE c.id=b.owner_id AND c.tenant_id=$1 AND c.knowledge_base_id=$2 AND c.deleted_at IS NULL AND k.deleted_at IS NULL))))) ORDER BY r.id`
|
||||
@@ -0,0 +1,178 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// This fixture creates only synthetic content in an explicitly named disposable
|
||||
// database. Never point this test at an application database.
|
||||
func TestLibraryExportIsolatedPostgres(t *testing.T) {
|
||||
dsn := os.Getenv("KG_LIBRARY_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("requires explicitly isolated kg_library_test_ PostgreSQL database")
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
|
||||
t.Fatal("isolated database required")
|
||||
}
|
||||
ctx := context.Background()
|
||||
db, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if _, err = db.Exec(ctx, libraryTestSchema); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
scope := PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}
|
||||
run := func() PostgresSnapshot {
|
||||
t.Helper()
|
||||
s, e := ExportLibrary(ctx, db, scope)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
return s
|
||||
}
|
||||
snap := run()
|
||||
if snap.Table("knowledges").RowCount != 1 || snap.Table("wiki_pages").RowCount != 1 || snap.Table("resources").RowCount != 1 {
|
||||
t.Fatal("incorrect scoped export")
|
||||
}
|
||||
if snap.Table("wiki_log_entries").RowCount != 1 || snap.Table("wiki_page_issues").RowCount != 1 {
|
||||
t.Fatal("Wiki associated data lost")
|
||||
}
|
||||
for _, tab := range snap.Tables {
|
||||
for _, raw := range tab.Rows {
|
||||
if strings.Contains(string(raw), "other-scope") {
|
||||
t.Fatalf("cross scope leak in %s", tab.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Run("MVCC concurrent update", func(t *testing.T) {
|
||||
wrapper := libraryHookDB{pool: db, hook: func() {
|
||||
if _, err := db.Exec(ctx, `UPDATE wiki_pages SET content='new-content' WHERE id='wiki-safe'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}}
|
||||
old, err := ExportLibrary(ctx, wrapper, scope)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(old.Table("wiki_pages").Rows[0]), "old-content") {
|
||||
t.Fatal("mixed PostgreSQL versions during export")
|
||||
}
|
||||
fresh := run()
|
||||
if !strings.Contains(string(fresh.Table("wiki_pages").Rows[0]), "new-content") {
|
||||
t.Fatal("concurrent writer did not actually commit")
|
||||
}
|
||||
})
|
||||
t.Run("deleted knowledge tasks", func(t *testing.T) {
|
||||
for _, test := range []struct{ table, task, scope, id string }{
|
||||
{"task_pending_ops", "wiki:ingest", "knowledge_base", "kb-safe"},
|
||||
{"task_pending_ops", "wiki:finalize", "knowledge_base", "kb-safe"},
|
||||
{"task_dead_letters", "wiki:ingest", "knowledge", "deleted-doc"},
|
||||
} {
|
||||
_, err := db.Exec(ctx, `INSERT INTO `+test.table+`(tenant_id,task_type,scope,scope_id) VALUES(10000,$1,$2,$3)`, test.task, test.scope, test.id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = ExportLibrary(ctx, db, scope); err == nil {
|
||||
t.Fatalf("ignored %s %s", test.table, test.task)
|
||||
}
|
||||
if _, err = db.Exec(ctx, `DELETE FROM `+test.table); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
t.Run("dangling resource blocks", func(t *testing.T) {
|
||||
_, err := db.Exec(ctx, `UPDATE wiki_pages SET content='resource://abcdefghijklmnopqrstuv' WHERE id='wiki-safe'`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = ExportLibrary(ctx, db, scope); err == nil {
|
||||
t.Fatal("accepted missing Wiki resource dependency")
|
||||
}
|
||||
if _, err = db.Exec(ctx, `UPDATE wiki_pages SET content='new-content' WHERE id='wiki-safe'`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
t.Run("empty library has explicit empty tables", func(t *testing.T) {
|
||||
s, err := ExportLibrary(ctx, db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-empty"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(s.Tables) != 10 {
|
||||
t.Fatal("empty library snapshot incomplete")
|
||||
}
|
||||
for _, tab := range s.Tables {
|
||||
if tab.RowCount != 0 {
|
||||
t.Fatal("empty library leaked content")
|
||||
}
|
||||
}
|
||||
})
|
||||
// The exporter may not mutate content: only fixture updates above are visible.
|
||||
var count int
|
||||
if err := db.QueryRow(ctx, `SELECT count(*) FROM sessions`).Scan(&count); err != nil || count != 1 {
|
||||
t.Fatal("nonknowledge data changed")
|
||||
}
|
||||
encoded, err := json.Marshal(snap)
|
||||
if err != nil || strings.Contains(string(encoded), "private-session-marker") {
|
||||
t.Fatal("nonknowledge data exported")
|
||||
}
|
||||
}
|
||||
|
||||
type libraryHookDB struct {
|
||||
pool *pgxpool.Pool
|
||||
hook func()
|
||||
}
|
||||
|
||||
func (d libraryHookDB) BeginTx(ctx context.Context, o pgx.TxOptions) (pgx.Tx, error) {
|
||||
tx, err := d.pool.BeginTx(ctx, o)
|
||||
return &libraryHookTx{Tx: tx, hook: d.hook}, err
|
||||
}
|
||||
|
||||
type libraryHookTx struct {
|
||||
pgx.Tx
|
||||
hook func()
|
||||
fired bool
|
||||
}
|
||||
|
||||
func (d *libraryHookTx) Query(ctx context.Context, q string, args ...any) (pgx.Rows, error) {
|
||||
if !d.fired {
|
||||
d.fired = true
|
||||
d.hook()
|
||||
}
|
||||
return d.Tx.Query(ctx, q, args...)
|
||||
}
|
||||
|
||||
const libraryTestSchema = `
|
||||
CREATE TABLE schema_migrations(version int,dirty bool); INSERT INTO schema_migrations VALUES(74,false);
|
||||
CREATE TABLE knowledge_bases(id text,tenant_id bigint,indexing_strategy jsonb,deleted_at timestamptz);
|
||||
INSERT INTO knowledge_bases VALUES('kb-safe',10000,'{}',NULL),('kb-empty',10000,'{}',NULL);
|
||||
CREATE TABLE knowledges(id text PRIMARY KEY,tenant_id bigint,knowledge_base_id text,parse_status text,pending_subtasks_count int,file_path text,deleted_at timestamptz);
|
||||
INSERT INTO knowledges VALUES('doc-safe',10000,'kb-safe','completed',0,'resource://ABCDEFGHIJKLMNOPQRSTUV',NULL),('other-scope',20000,'kb-safe','completed',0,'',NULL),('deleted-doc',10000,'kb-safe','completed',0,'',now());
|
||||
CREATE TABLE chunks(id text,tenant_id bigint,knowledge_base_id text,knowledge_id text,content text,deleted_at timestamptz);
|
||||
INSERT INTO chunks VALUES('chunk-safe',10000,'kb-safe','doc-safe','chunk',NULL),('other-scope',20000,'kb-safe','other-scope','secret',NULL);
|
||||
CREATE TABLE knowledge_tags(id text,tenant_id bigint,knowledge_base_id text,deleted_at timestamptz);
|
||||
CREATE TABLE knowledge_tag_relations(knowledge_id text,tag_id text);
|
||||
CREATE TABLE resources(id text,handle text,tenant_id bigint,physical_path text,deleted_at timestamptz);
|
||||
INSERT INTO resources VALUES('res-safe','ABCDEFGHIJKLMNOPQRSTUV',10000,'source.txt',NULL),('other-scope','abcdefghijklmnopqrstuv',20000,'other-scope',NULL);
|
||||
CREATE TABLE resource_bindings(id text,tenant_id bigint,resource_id text,owner_type text,owner_id text);
|
||||
INSERT INTO resource_bindings VALUES('binding-safe',10000,'res-safe','knowledge','doc-safe');
|
||||
CREATE TABLE wiki_pages(id text,tenant_id bigint,knowledge_base_id text,slug text,content text,deleted_at timestamptz);
|
||||
INSERT INTO wiki_pages VALUES('wiki-safe',10000,'kb-safe','index','old-content',NULL),('other-scope',20000,'kb-safe','secret','other-scope',NULL);
|
||||
CREATE TABLE wiki_folders(id text,tenant_id bigint,knowledge_base_id text,deleted_at timestamptz);
|
||||
CREATE TABLE wiki_page_issues(id text,tenant_id bigint,knowledge_base_id text,deleted_at timestamptz);
|
||||
INSERT INTO wiki_page_issues VALUES('issue-safe',10000,'kb-safe',NULL);
|
||||
CREATE TABLE wiki_log_entries(id bigint,tenant_id bigint,knowledge_base_id text);
|
||||
INSERT INTO wiki_log_entries VALUES(1,10000,'kb-safe');
|
||||
CREATE TABLE task_pending_ops(tenant_id bigint,task_type text,scope text,scope_id text);
|
||||
CREATE TABLE task_dead_letters(tenant_id bigint,task_type text,scope text,scope_id text);
|
||||
CREATE TABLE sessions(marker text); INSERT INTO sessions VALUES('private-session-marker');
|
||||
`
|
||||
@@ -0,0 +1,125 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func TestLibraryExportIncludesEmptyAndCompleteWiki(t *testing.T) {
|
||||
db := newLibraryFixture()
|
||||
snapshot, err := ExportLibrary(context.Background(), db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(snapshot.Tables) != 10 {
|
||||
t.Fatalf("table count=%d, want all ten content tables including empty ones", len(snapshot.Tables))
|
||||
}
|
||||
for _, name := range []string{"knowledges", "chunks", "resources", "resource_bindings", "knowledge_tags", "knowledge_tag_relations", "wiki_pages", "wiki_folders", "wiki_page_issues", "wiki_log_entries"} {
|
||||
if !tablePresent(snapshot, name) {
|
||||
t.Fatalf("missing table %s", name)
|
||||
}
|
||||
}
|
||||
if db.options.IsoLevel != pgx.RepeatableRead || db.options.AccessMode != pgx.ReadOnly {
|
||||
t.Fatalf("unsafe snapshot options: %+v", db.options)
|
||||
}
|
||||
if !db.tx.committed {
|
||||
t.Fatal("snapshot transaction not committed")
|
||||
}
|
||||
for _, q := range db.tx.queries {
|
||||
for _, forbidden := range []string{"from users", "from sessions", "from api_keys", "from messages"} {
|
||||
if strings.Contains(strings.ToLower(q), forbidden) {
|
||||
t.Fatalf("nonknowledge export: %s", forbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLibraryExportBlocksUnsettledWorkIncludingDeletedDocumentRetracts(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
setup func(*libraryFixtureTx)
|
||||
}{
|
||||
{"document failed or parsing", func(d *libraryFixtureTx) { d.unsettled = 1 }},
|
||||
{"deleted document retract or finalizer", func(d *libraryFixtureTx) { d.pending = 1 }},
|
||||
{"dead letter not semantic success", func(d *libraryFixtureTx) { d.failed = 1 }},
|
||||
{"unsupported graph data", func(d *libraryFixtureTx) { d.graph = true }},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
db := newLibraryFixture()
|
||||
test.setup(db.tx)
|
||||
_, err := ExportLibrary(context.Background(), db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"})
|
||||
if err == nil {
|
||||
t.Fatal("unsettled library was exported")
|
||||
}
|
||||
if db.tx.committed {
|
||||
t.Fatal("blocked snapshot committed")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLibraryExportDoesNotReturnPartialSnapshot(t *testing.T) {
|
||||
db := newLibraryFixture()
|
||||
db.tx.queryError = true
|
||||
got, err := ExportLibrary(context.Background(), db, PostgresIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"})
|
||||
if err == nil || len(got.Tables) != 0 || db.tx.committed {
|
||||
t.Fatal("partial snapshot escaped")
|
||||
}
|
||||
}
|
||||
|
||||
type libraryFixture struct {
|
||||
options pgx.TxOptions
|
||||
tx *libraryFixtureTx
|
||||
}
|
||||
type libraryFixtureTx struct {
|
||||
pgx.Tx
|
||||
queries []string
|
||||
committed bool
|
||||
pending, failed, unsettled int64
|
||||
graph bool
|
||||
queryError bool
|
||||
}
|
||||
|
||||
func newLibraryFixture() *libraryFixture { return &libraryFixture{tx: &libraryFixtureTx{}} }
|
||||
func (d *libraryFixture) BeginTx(_ context.Context, o pgx.TxOptions) (pgx.Tx, error) {
|
||||
d.options = o
|
||||
return d.tx, nil
|
||||
}
|
||||
func (d *libraryFixtureTx) Commit(context.Context) error { d.committed = true; return nil }
|
||||
func (d *libraryFixtureTx) Rollback(context.Context) error { return nil }
|
||||
func (d *libraryFixtureTx) QueryRow(_ context.Context, q string, _ ...any) pgx.Row {
|
||||
d.queries = append(d.queries, q)
|
||||
return libraryFixtureRow{d: d, sql: q}
|
||||
}
|
||||
func (d *libraryFixtureTx) Query(_ context.Context, q string, _ ...any) (pgx.Rows, error) {
|
||||
d.queries = append(d.queries, q)
|
||||
if d.queryError {
|
||||
return nil, errors.New("fixture failure")
|
||||
}
|
||||
return &jsonRows{values: []json.RawMessage{}}, nil
|
||||
}
|
||||
|
||||
type libraryFixtureRow struct {
|
||||
d *libraryFixtureTx
|
||||
sql string
|
||||
}
|
||||
|
||||
func (r libraryFixtureRow) Scan(dest ...any) error {
|
||||
switch {
|
||||
case strings.Contains(r.sql, "schema_migrations"):
|
||||
*dest[0].(*int) = 74
|
||||
*dest[1].(*bool) = false
|
||||
case strings.Contains(r.sql, "indexing_strategy"):
|
||||
*dest[0].(*bool) = r.d.graph
|
||||
default:
|
||||
*dest[0].(*int64) = r.d.unsettled
|
||||
*dest[1].(*int64) = r.d.pending
|
||||
*dest[2].(*int64) = r.d.failed
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ExportLibrary uses one Elasticsearch point-in-time rather than offset paging
|
||||
// against a changing index. A PIT does not freeze PostgreSQL or object storage;
|
||||
// the coordinator must still prove cross-lane consistency before publication.
|
||||
func (e *SearchExporter) ExportLibrary(ctx context.Context, identity SearchIdentity, maxDocuments int) (snapshot SearchSnapshot, resultErr error) {
|
||||
if identity.TenantID == 0 || !validSearchIdentifier(identity.KnowledgeBaseID) || identity.KnowledgeID != "" || maxDocuments < 1 {
|
||||
return SearchSnapshot{}, errors.New("invalid library search scope or limit")
|
||||
}
|
||||
inspection, err := e.inspectMapping(ctx)
|
||||
if err != nil {
|
||||
return SearchSnapshot{}, err
|
||||
}
|
||||
var opened struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if err = e.requestJSON(ctx, http.MethodPost, "/"+e.index+"/_pit?keep_alive=2m&allow_partial_search_results=false", nil, &opened); err != nil {
|
||||
return SearchSnapshot{}, err
|
||||
}
|
||||
if opened.ID == "" {
|
||||
return SearchSnapshot{}, errors.New("missing Elasticsearch PIT")
|
||||
}
|
||||
pit := opened.ID
|
||||
defer func() {
|
||||
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
|
||||
defer cancel()
|
||||
var result struct {
|
||||
Succeeded bool `json:"succeeded"`
|
||||
}
|
||||
err := e.requestJSON(cleanup, http.MethodDelete, "/_pit", map[string]any{"id": pit}, &result)
|
||||
if (err != nil || !result.Succeeded) && resultErr == nil {
|
||||
snapshot = SearchSnapshot{}
|
||||
resultErr = errors.New("close library snapshot PIT")
|
||||
}
|
||||
}()
|
||||
filter := []any{map[string]any{"term": map[string]any{inspection.fields.base: identity.KnowledgeBaseID}}}
|
||||
if inspection.fields.tenant != "" {
|
||||
filter = append(filter, map[string]any{"term": map[string]any{inspection.fields.tenant: identity.TenantID}})
|
||||
}
|
||||
var cursor json.RawMessage
|
||||
documents := make([]SearchDocument, 0)
|
||||
seen := map[string]bool{}
|
||||
var total *int64
|
||||
for {
|
||||
body := map[string]any{"pit": map[string]any{"id": pit, "keep_alive": "2m"}, "size": e.pageSize, "sort": []any{"_shard_doc"}, "track_total_hits": true, "query": map[string]any{"bool": map[string]any{"filter": filter}}}
|
||||
if cursor != nil {
|
||||
body["search_after"] = cursor
|
||||
}
|
||||
var page struct {
|
||||
PITID string `json:"pit_id"`
|
||||
TimedOut *bool `json:"timed_out"`
|
||||
Shards struct {
|
||||
Total int `json:"total"`
|
||||
Successful int `json:"successful"`
|
||||
Failed int `json:"failed"`
|
||||
} `json:"_shards"`
|
||||
Hits struct {
|
||||
Total struct {
|
||||
Value *int64 `json:"value"`
|
||||
Relation string `json:"relation"`
|
||||
} `json:"total"`
|
||||
Hits []struct {
|
||||
ID string `json:"_id"`
|
||||
Source json.RawMessage `json:"_source"`
|
||||
Sort json.RawMessage `json:"sort"`
|
||||
} `json:"hits"`
|
||||
} `json:"hits"`
|
||||
}
|
||||
if err = e.requestJSON(ctx, http.MethodPost, "/_search?allow_partial_search_results=false", body, &page); err != nil {
|
||||
return SearchSnapshot{}, err
|
||||
}
|
||||
if page.PITID != "" {
|
||||
pit = page.PITID
|
||||
}
|
||||
if page.TimedOut == nil || *page.TimedOut || page.Shards.Total < 1 || page.Shards.Successful != page.Shards.Total || page.Shards.Failed != 0 || page.Hits.Total.Value == nil || page.Hits.Total.Relation != "eq" {
|
||||
return SearchSnapshot{}, errors.New("incomplete Elasticsearch library snapshot")
|
||||
}
|
||||
if total == nil {
|
||||
total = page.Hits.Total.Value
|
||||
}
|
||||
if *total < 0 || *total > int64(maxDocuments) || *total != *page.Hits.Total.Value {
|
||||
return SearchSnapshot{}, errors.New("Elasticsearch library total changed or exceeds limit")
|
||||
}
|
||||
for _, hit := range page.Hits.Hits {
|
||||
var scope struct {
|
||||
KnowledgeID string `json:"knowledge_id"`
|
||||
}
|
||||
if json.Unmarshal(hit.Source, &scope) != nil || !validSearchIdentifier(scope.KnowledgeID) || !validSearchIdentifier(hit.ID) || seen[hit.ID] {
|
||||
return SearchSnapshot{}, errors.New("invalid or duplicate library search identity")
|
||||
}
|
||||
expected := identity
|
||||
expected.KnowledgeID = scope.KnowledgeID
|
||||
if !searchHitMatches(hit.Source, expected) {
|
||||
return SearchSnapshot{}, errors.New("cross-scope library search document")
|
||||
}
|
||||
source, err := canonicalJSON(hit.Source)
|
||||
if err != nil {
|
||||
return SearchSnapshot{}, errors.New("invalid library search data")
|
||||
}
|
||||
seen[hit.ID] = true
|
||||
documents = append(documents, SearchDocument{ID: hit.ID, Source: source})
|
||||
}
|
||||
if int64(len(documents)) > *total {
|
||||
return SearchSnapshot{}, errors.New("library search total mismatch")
|
||||
}
|
||||
if len(page.Hits.Hits) > 0 {
|
||||
next := page.Hits.Hits[len(page.Hits.Hits)-1].Sort
|
||||
var values []json.RawMessage
|
||||
if json.Unmarshal(next, &values) != nil || len(values) == 0 || string(next) == string(cursor) {
|
||||
return SearchSnapshot{}, errors.New("library search cursor did not advance")
|
||||
}
|
||||
cursor = next
|
||||
}
|
||||
if int64(len(documents)) == *total {
|
||||
break
|
||||
}
|
||||
if len(page.Hits.Hits) == 0 {
|
||||
return SearchSnapshot{}, errors.New("library search snapshot truncated")
|
||||
}
|
||||
}
|
||||
finalMapping, err := e.inspectMapping(ctx)
|
||||
if err != nil || finalMapping.digest != inspection.digest {
|
||||
return SearchSnapshot{}, errors.New("library search mapping changed")
|
||||
}
|
||||
sort.Slice(documents, func(i, j int) bool { return documents[i].ID < documents[j].ID })
|
||||
sha, err := searchDocumentsDigest(documents)
|
||||
if err != nil {
|
||||
return SearchSnapshot{}, err
|
||||
}
|
||||
return SearchSnapshot{Documents: documents, MappingSHA256: inspection.digest, DocumentsSHA256: sha}, nil
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLibrarySearchIsolatedElasticsearch(t *testing.T) {
|
||||
base := os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
|
||||
if base == "" {
|
||||
t.Skip("requires explicit isolated Elasticsearch fixture opt-in")
|
||||
}
|
||||
if os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
|
||||
t.Fatal("isolated fixture authorization required")
|
||||
}
|
||||
var id [8]byte
|
||||
if _, err := rand.Read(id[:]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
index := "kg_library_test_" + hex.EncodeToString(id[:])
|
||||
client := &http.Client{Timeout: 20 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
|
||||
request := func(method, path, body, contentType string) int {
|
||||
t.Helper()
|
||||
r, err := http.NewRequest(method, base+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal("invalid fixture endpoint")
|
||||
}
|
||||
r.Header.Set("Content-Type", contentType)
|
||||
response, err := client.Do(r)
|
||||
if err != nil {
|
||||
t.Fatal("fixture request failed")
|
||||
}
|
||||
defer response.Body.Close()
|
||||
return response.StatusCode
|
||||
}
|
||||
if status := request(http.MethodPut, "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"}}}}`, "application/json"); status != 200 {
|
||||
t.Fatalf("create synthetic index HTTP %d", status)
|
||||
}
|
||||
defer func() {
|
||||
if status := request(http.MethodDelete, "/"+index, "", "application/json"); status != 200 {
|
||||
t.Errorf("remove fixture index HTTP %d", status)
|
||||
}
|
||||
if status := request(http.MethodHead, "/"+index, "", "application/json"); status != 404 {
|
||||
t.Errorf("fixture index remains HTTP %d", status)
|
||||
}
|
||||
}()
|
||||
var bulk bytes.Buffer
|
||||
for i := 1; i <= 3; i++ {
|
||||
baseID := "kb-safe"
|
||||
if i == 3 {
|
||||
baseID = "kb-other"
|
||||
}
|
||||
fmt.Fprintf(&bulk, "{\"index\":{\"_id\":\"chunk-%d\"}}\n{\"tenant_id\":10000,\"knowledge_base_id\":%q,\"knowledge_id\":\"doc-safe\"}\n", i, baseID)
|
||||
}
|
||||
if status := request(http.MethodPost, "/"+index+"/_bulk?refresh=true", bulk.String(), "application/x-ndjson"); status != 200 {
|
||||
t.Fatalf("fixture bulk HTTP %d", status)
|
||||
}
|
||||
exporter, err := NewSearchExporter(SearchExporterConfig{BaseURL: base, Index: index, PageSize: 1, HTTPClient: client})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s, err := exporter.ExportLibrary(context.Background(), SearchIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, 10)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(s.Documents) != 2 {
|
||||
t.Fatalf("documents=%d", len(s.Documents))
|
||||
}
|
||||
empty, err := exporter.ExportLibrary(context.Background(), SearchIdentity{TenantID: 10000, KnowledgeBaseID: "kb-empty"}, 10)
|
||||
if err != nil || len(empty.Documents) != 0 {
|
||||
t.Fatalf("empty snapshot err=%v documents=%d", err, len(empty.Documents))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLibrarySearchUsesPITAndChecksExactTotal(t *testing.T) {
|
||||
for _, mode := range []string{"complete", "partial", "truncated", "wrong-scope", "repeated-cursor", "limit"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
searches, closed := 0, false
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch {
|
||||
case r.URL.Path == "/chunks/_mapping":
|
||||
fmt.Fprint(w, `{"chunks":{"mappings":{"properties":{"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"tenant_id":{"type":"long"}}}}}`)
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/chunks/_pit":
|
||||
if r.URL.Query().Get("allow_partial_search_results") != "false" {
|
||||
t.Error("partial PIT allowed")
|
||||
}
|
||||
fmt.Fprint(w, `{"id":"pit-0"}`)
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/_search":
|
||||
var body map[string]any
|
||||
if json.NewDecoder(r.Body).Decode(&body) != nil {
|
||||
t.Error("bad query")
|
||||
}
|
||||
searches++
|
||||
if _, ok := body["from"]; ok {
|
||||
t.Error("offset paging is not a fixed snapshot")
|
||||
}
|
||||
pit, _ := body["pit"].(map[string]any)
|
||||
if pit["id"] != fmt.Sprintf("pit-%d", searches-1) {
|
||||
t.Error("latest PIT identity not used")
|
||||
}
|
||||
if searches > 1 {
|
||||
if _, ok := body["search_after"]; !ok {
|
||||
t.Error("missing cursor")
|
||||
}
|
||||
}
|
||||
failed := 0
|
||||
if mode == "partial" {
|
||||
failed = 1
|
||||
}
|
||||
base := "kb-safe"
|
||||
if mode == "wrong-scope" {
|
||||
base = "kb-wrong"
|
||||
}
|
||||
hits := fmt.Sprintf(`[{"_id":"chunk-%d","_source":{"tenant_id":10000,"knowledge_base_id":%q,"knowledge_id":"doc-safe"},"sort":[%d]}]`, searches, base, searches)
|
||||
if mode == "truncated" && searches == 2 {
|
||||
hits = "[]"
|
||||
}
|
||||
if mode == "repeated-cursor" && searches == 2 {
|
||||
hits = `[{"_id":"chunk-2","_source":{"knowledge_base_id":"kb-safe","knowledge_id":"doc-safe"},"sort":[1]}]`
|
||||
}
|
||||
fmt.Fprintf(w, `{"pit_id":"pit-%d","timed_out":false,"_shards":{"total":1,"successful":1,"failed":%d},"hits":{"total":{"value":2,"relation":"eq"},"hits":%s}}`, searches, failed, hits)
|
||||
case r.Method == http.MethodDelete && r.URL.Path == "/_pit":
|
||||
closed = true
|
||||
fmt.Fprint(w, `{"succeeded":true,"num_freed":1}`)
|
||||
default:
|
||||
t.Errorf("unexpected %s %s", r.Method, r.URL.Path)
|
||||
http.Error(w, "unexpected", 400)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
exporter, err := NewSearchExporter(SearchExporterConfig{BaseURL: server.URL, Index: "chunks", PageSize: 1})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
limit := 10
|
||||
if mode == "limit" {
|
||||
limit = 1
|
||||
}
|
||||
snapshot, err := exporter.ExportLibrary(context.Background(), SearchIdentity{TenantID: 10000, KnowledgeBaseID: "kb-safe"}, limit)
|
||||
if mode == "complete" {
|
||||
if err != nil || len(snapshot.Documents) != 2 {
|
||||
t.Fatalf("export err=%v documents=%d", err, len(snapshot.Documents))
|
||||
}
|
||||
} else if err == nil {
|
||||
t.Fatal("invalid snapshot accepted")
|
||||
}
|
||||
if !closed {
|
||||
t.Fatal("PIT leaked")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,11 @@
|
||||
"⚠️ 文件缺失或解析不了 ⇒ 门禁红:范围声明不出来时,不许默认放行。"
|
||||
],
|
||||
"optional": [
|
||||
{
|
||||
"package": "knowledge-governance/internal/libraryfreeze",
|
||||
"test": "TestFreezeIsolatedDocker",
|
||||
"reason": "只在显式 KG_FREEZE_DOCKER_ISOLATED=1 下使用本地 Docker 与已缓存 busybox 创建随机标签、无网络/无挂载夹具,验证真实暂停恢复和进程中断;默认 CI 不连接 Docker 或运行中的服务,本地另行实跑"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/qaopsmirror",
|
||||
"test": "TestAILiveBailianPersistsAssessmentAndTranslation",
|
||||
|
||||
Reference in New Issue
Block a user