feat(deploy): 接通云治理主站的人工确认发布流水线 #20

Merged
zheromiku merged 2 commits from ci-trial-governance-cd into main 2026-11-23 02:26:49 +08:00
10 changed files with 1111 additions and 2 deletions
+76
View File
@@ -0,0 +1,76 @@
name: deploy-production
on:
workflow_dispatch:
inputs:
action:
description: deploy 或 writeback
required: true
default: deploy
sha:
description: 验收过的完整提交 SHA
required: true
bundle:
description: release.json 的完整 SHA256
required: true
task:
description: 对应的 HiFox 任务 UUID
required: true
confirm:
description: deploy-production 或 retry-writeback
required: true
concurrency:
group: cloud-test-governance-main-production
cancel-in-progress: false
permissions:
contents: read
jobs:
guard:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: 核对手动入口及发布输入
run: |
set -eu
test "$GITHUB_EVENT_NAME" = workflow_dispatch
test "$GITHUB_REPOSITORY" = Weknora/knowledge-governance
test "$GITHUB_REF" = refs/heads/main
case "$GITHUB_SERVER_URL" in
https://gitea.zheromiku.top|http://192.168.50.223:3000) ;;
*) exit 1 ;;
esac
git init -q .
git fetch -q --depth 200 "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git" refs/heads/main
git checkout -q --detach "$GITHUB_SHA"
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
python3 scripts/production_cd.py guard --event "$GITHUB_EVENT_PATH"
publish:
needs: [guard]
runs-on: kg-production-cd
timeout-minutes: 40
steps:
- name: 取同一提交的控制端代码
run: |
set -eu
test "$GITHUB_EVENT_NAME" = workflow_dispatch
test "$GITHUB_REPOSITORY" = Weknora/knowledge-governance
test "$GITHUB_REF" = refs/heads/main
case "$GITHUB_SERVER_URL" in
https://gitea.zheromiku.top|http://192.168.50.223:3000) ;;
*) exit 1 ;;
esac
git init -q .
git fetch -q --depth 200 "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY.git" refs/heads/main
git checkout -q --detach "$GITHUB_SHA"
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
- name: 调用云主站受限发布入口并读回回执
env:
KG_CD_SSH_KEY: ${{ secrets.KG_CD_SSH_KEY }}
KG_CD_KNOWN_HOSTS: ${{ secrets.KG_CD_KNOWN_HOSTS }}
KG_CD_SSH_HOST: ${{ vars.KG_CD_SSH_HOST }}
KG_CD_SSH_PORT: ${{ vars.KG_CD_SSH_PORT }}
run: python3 scripts/production_cd.py send --event "$GITHUB_EVENT_PATH"
+2
View File
@@ -5,6 +5,8 @@
管理员配置连接和发布授权,文档工程师维护资料,只读成员查看进度。先确定要操作的环境,再按下面的接线与验收步骤进入业务流程。
云测试治理主站自身的应用升级见[主站应用 CD](PRODUCTION_CD_RUNBOOK.md)。
## 入口与反向代理
| 请求从哪里发出 | 地址或路径 | 实际目的地 |
+48
View File
@@ -0,0 +1,48 @@
# 云测试治理主站应用 CD
正式入口为 `http://100.85.111.121:8088/`。云端 Compose 项目为 `knowledge-governance-cloud-test`,工作目录 `/opt/knowledge-governance`,当前配置链为 `compose.json`,服务为 `api`、`web`、`worker`。Web 容器提供 `8088`,通过挂载的 nginx 配置代理 API 的 `127.0.0.1:18098`。应用 CD 与知识库内容发布各自保留原有授权流程。
## 建立验收产物
1. 从干净检出的完整提交构建一次 API、worker、web 镜像。构建调用方对三个镜像都注入 `org.opencontainers.image.revision=<完整 SHA>`。API 构建身份沿用 `scripts/version.py --identity` 输出,传给 Dockerfile 的版本、提交和构建时间参数。API 与 worker 各自记录镜像 ID,即使它们来自同一个 Dockerfile。
2. 封存后用同一份归档部署到隔离验收环境。输出目录必须是新路径,父目录应已存在。三个 `--api/--web/--worker` 参数是本地镜像的完整 ID。
```bash
python3 scripts/production_cd.py pack --sha "$SHA" --task "$TASK_UUID" \
--api "$API_ID" --web "$WEB_ID" --worker "$WORKER_ID" --output "$BUNDLE_DIR"
docker load -i "$BUNDLE_DIR/api.tar"
docker load -i "$BUNDLE_DIR/web.tar"
docker load -i "$BUNDLE_DIR/worker.tar"
```
3. 用验收环境自身的 Compose 和数据库启动这些镜像。用下面的命令读回验收环境运行身份。此处三个服务参数是实际的**测试容器名**。在业务验收入口执行实际用例并将输出、结果与报告链接放在同一份验收记录里。
```bash
python3 scripts/production_cd.py attest --sha "$SHA" --task "$TASK_UUID" \
--api "$QA_API" --web "$QA_WEB" --worker "$QA_WORKER" --test-url "$QA_URL" --directory "$BUNDLE_DIR"
BUNDLE_SHA=$(sha256sum "$BUNDLE_DIR/release.json" | cut -d' ' -f1)
```
4. 由验收人使用获授权的 HiFox CLI 登记业务结论,读回任务、验收字段和状态。`cd_acceptance` 是 JSON 字符串类型的 metadata,其解析结果为 `{"verdict":"passed","sha":"…","bundle":"…","evidence":"https://…"}`。录入人应先核对真实报告与运行身份。随后把对应任务转至已有「待发布」。用 `task view --output json` 核对实际任务 UUID、空间、状态和验收字段。验收登记重试复用原归档摘要与原记录。以当前版本只接受一个任务,多个任务分别验收。
5. 在首次发布前,由运维把四个文件放入云主站的 `/opt/knowledge-governance/cd/bundles/<release.json SHA256>/`。通过安全文件通道传到临时目录,核对摘要与内容后用同一文件系统上的目录重命名发布;已有目录先核对内容,匹配时复用。此目录仅供目标控制器读取,文件由该账号持有且组与其他用户无写权限。
## 人工确认并自动部署
在 Gitea Actions 打开 `deploy-production` 并运行工作流。工作流分支选择 `main`,填写 `action=deploy`、已验收的完整 `sha`、归档清单摘要 `bundle`、HiFox 任务 UUID `task` 和 `confirm=deploy-production`。发布人用自己的 Gitea 账号提交一次确认。
主站侧从 Gitea 读取运行、操作者和通过的校验作业,并验证输入摘要及工作流代码。候选代码可来自仓库内的任务分支,以实际验收 SHA 为准。切换前还核验必需 CI 状态、HiFox 验收记录、三份归档、线上源代码血缘及数据库迁移账本。切换复用 `scripts/deploy-api-web.sh`,仅更换三项 Compose `image`;保留原配置字节以备回滚。切换后自动读回三镜像、API 二进制 SHA、主站健康、API readiness 及实际提供的前端资源。HiFox 回写包含原始运行 URL、完整提交、摘要、镜像 ID 及核验证据。
目标机已有的 `schema_migrations` 必须与候选完全相同,API 与 worker 的库连接均须核对。应用会在启动时检查迁移。新增或改变的迁移先按独立流程审查、备份、执行并复验;CD 自身只读迁移账本。发布时运营方应保留 prior tag,保障用原 Compose 字节回滚。
## 失败恢复
- `rolled_back` 表示自动恢复了原 Compose、旧的镜像 ID、API SHA 与站点健康。`rollback_failed` 或 `cutover_started` 要求巡检现场、按回执恢复并复验。目标机互斥锁覆盖切换及回写。Gitea 的取消记录只能说明流水线结束,回滚以目标机回执和运行时读回为准。
- `DEPLOYED_HIFOX_PENDING` 表示镜像切换和核验已经完成。以同一 `sha`、`bundle`、`task` 人工发起 `action=writeback`、`confirm=retry-writeback`。脚本先读原目标回执并复验仍在运行的版本,随后补写 HiFox,保留原部署链接。状态与 metadata 的当前值以 HiFox 读回为准。重复调用核对记录后结束。
- 目标回执 `/opt/knowledge-governance/cd/receipts/<bundle>.json`,镜像切换前的 Compose 保存为同名 `.compose.backup`。SSH 超时或强制中断后,先读回执和当前运行态。已存在未决或失败回执时,重复部署被拒。保留原回执作为事故证据。
## 一次性配置与权限
- 目标机安装仓库 `scripts/{production_cd.py,cd_migration_probe.py,deploy-api-web.sh,production-cd-ssh.sh}` 到 `/opt/knowledge-governance/cd/bin/`。创建私有 `receipts/`、`bundles/`、`hifox-home/`,初始化 `source.git` 为 bare Git 仓,并将 `.gitea/required-contexts.txt` 安装到该目录。仅发布运维可更新这些文件。
- 填写目标机 `/opt/knowledge-governance/cd/deploy-production.yml`,内容与 Gitea 主分支的工作流逐字节相同。控制器用 Gitea 发布人账号 ID 和只读日志验证确认。目标端 `authorized_keys` 为 CI 单独公钥设置 `restrict,command="/opt/knowledge-governance/cd/bin/production-cd-ssh.sh"`。校验云主机 SSH host key 后将其按 `kg-governance-cd` 别名写入 `KG_CD_KNOWN_HOSTS`。对应独立 SSH 私钥保存在 `KG_CD_SSH_KEY`。Runner 标签为 `kg-production-cd`,使用 `KG_CD_SSH_HOST` 和 `KG_CD_SSH_PORT` 接入批准的 Tailscale 或本机 IPv6 隧道。
- 将受控 HiFox CLI 安装至 `bin/hifox`,由账号持有人在目标主机使用 `hifox-home` 对应的私有配置目录完成登录。身份须能读取指定空间的任务、写 `cd_release` metadata 并完成对应状态迁移。状态绑定前分别读回组织状态和实际任务类型,复用原有待发布、已发布 ID,保留其余工作流配置。
- 常规 push CI 负责验证和静态预览。这个 workflow 的唯一事件是 `workflow_dispatch`。`make verify` 的 `contracts` 子项运行 CD 用例。真实切换的验收要求对应候选、现场前置条件和发布人确认。
+1 -1
View File
@@ -3,7 +3,7 @@
> 这份文档是**破坏性变化与数据迁移步骤的落脚点**:`CHANGELOG.md` 里以 `**破坏性**` 开头的条目
> **必须**链接到本文(或本文的某个小节)。发布记录本身仍在 `CHANGELOG.md` —— 本文不维护第二份变更列表。
>
> 本仓 CI **只做 CI**:不打 tag、不发 Release、不部署。发布与部署都是**独立的人工审批动作**。
> 常规 CI 负责检查、构建与静态预览。版本 tag 与 Release 由人批准后执行。主站应用部署使用独立的[人工确认 CD](PRODUCTION_CD_RUNBOOK.md)。
## 一、正式发布(由人批准后执行)
+56
View File
@@ -0,0 +1,56 @@
"""Run inside a governance container. Read the real application DB ledger only."""
import json
import os
import subprocess
from urllib.parse import parse_qsl, unquote, urlsplit
def connection_env(env):
# Keep only the connection properties actually shared by libpq and pgx.
# Unknown DSN options can change the database identity or schema search path.
u = urlsplit(env['DATABASE' + '_URL'])
if u.scheme not in ('postgres', 'postgresql') or not u.hostname or not u.username or not u.path[1:] or u.fragment:
raise ValueError('invalid database URL')
query = parse_qsl(u.query, strict_parsing=True, keep_blank_values=True)
supported = {'sslmode': 'PGSSLMODE', 'sslrootcert': 'PGSSLROOTCERT', 'sslcert': 'PGSSLCERT', 'sslkey': 'PGSSLKEY'}
if len(dict(query)) != len(query) or any(key not in supported or not val for key, val in query):
raise ValueError('unknown database connection options')
safe = {'PATH': '/usr/bin:/bin', 'HOME': env.get('HOME', '/nonexistent'), 'LC_ALL': 'C',
'PGHOST': u.hostname, 'PGPORT': str(u.port or 5432), 'PGDATABASE': unquote(u.path[1:]),
'PGUSER': unquote(u.username), 'PGCONNECT_TIMEOUT': '5',
'PGOPTIONS': '-c default_transaction_read_only=on -c statement_timeout=5000'}
for name in ('PGPASSFILE', 'PGSSLMODE', 'PGSSLROOTCERT', 'PGSSLCERT', 'PGSSLKEY'):
if env.get(name):
safe[name] = env[name]
for name in env:
if name.startswith('PG') and name not in safe:
raise ValueError('unknown inherited database connection options')
safe.update({supported[key]: value for key, value in query})
if u.password:
safe['PGPASSWORD'] = unquote(u.password)
for name in ('PGDATABASE', 'PGUSER'):
if any(char in safe[name] for char in '\r\n\x00='):
raise ValueError('unsafe database identifier')
return safe
def main():
env = connection_env(os.environ)
sql = """SELECT json_build_object('database', current_database(), 'user', current_user,
'read_only', current_setting('transaction_read_only'),
'versions', (SELECT json_agg(version ORDER BY version) FROM schema_migrations));"""
p = subprocess.run(['psql', '-XAtw', '-v', 'ON_ERROR_STOP=1', '-c', sql],
env=env, capture_output=True, timeout=12)
if p.returncode:
raise ValueError('query failed')
data = json.loads(p.stdout)
if data['read_only'] != 'on' or data['database'] != env['PGDATABASE'] or data['user'] != env['PGUSER']:
raise ValueError('database identity mismatch')
print(json.dumps(data['versions']))
if __name__ == '__main__':
try:
main()
except (OSError, ValueError, KeyError, subprocess.TimeoutExpired):
raise SystemExit('read-only migration probe failed') from None
+4
View File
@@ -360,6 +360,10 @@ def cmd_internal_contracts(go: str) -> int:
cwd=ROOT).returncode
if qa_rc:
return qa_rc
cd_rc = subprocess.run(py("-m", "unittest", "discover", "-s", "scripts", "-p", "test_production_cd.py", "-v"),
cwd=ROOT).returncode
if cd_rc:
return cd_rc
snapshot_rc = subprocess.run(py("scripts/development_snapshot.py","--self-test"),cwd=ROOT).returncode
if snapshot_rc:
return 1
+17 -1
View File
@@ -1,6 +1,20 @@
#!/usr/bin/env bash
# 重建并上线 governance api + web(本机 docker build 需要宿主代理 + --network host)
set -u
# Share the image-only Compose switch with CD without entering the legacy build path.
compose_up() {
"${compose[@]}" up -d --no-build --no-deps "$@"
}
if [[ "${1:-}" == --verified-compose ]]; then
[[ "$#" == 7 && "$5" == api && "$6" == web && "$7" == worker && "$4" == /* ]] || exit 2
cd -- "$2" || exit 1
compose=(docker compose -p "$3" -f "$4")
compose_up --pull never "$5" "$6" "$7"
exit $?
fi
cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 1
[ -z "$(git status --porcelain --untracked-files=normal)" ] || { printf "%s\n" "REFUSED_DIRTY_SOURCE"; exit 1; }
TS=$(date +%Y%m%d-%H%M%S)
@@ -103,7 +117,9 @@ if [ "${GATES_PREV}" = "true" ] && [[ "${COMPOSE_FILES_EFFECTIVE}" != *compose.r
echo " 确实要关闸:ALLOW_GATE_OFF=1 bash scripts/deploy-api-web.sh"
exit 1
fi
docker compose ${COMPOSE_FILES_EFFECTIVE} up -d --no-build --no-deps api web || { echo "UP_FAILED"; exit 1; }
read -r -a compose_file_args <<< "$COMPOSE_FILES_EFFECTIVE"
compose=(docker compose "${compose_file_args[@]}")
compose_up api web || { echo "UP_FAILED"; exit 1; }
echo "=== 5. 状态"
docker ps --filter name=knowledge-governance-p1 --format '{{.Names}} {{.Status}} {{.Image}}'
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
set -eu
umask 077
[ "${SSH_ORIGINAL_COMMAND:-}" = kg-cd-v1 ] || exit 1
exec /usr/bin/env -i HOME=/opt/knowledge-governance/cd PATH=/usr/bin:/bin LANG=C.UTF-8 \
GIT_TERMINAL_PROMPT=0 /usr/bin/python3 -I /opt/knowledge-governance/cd/bin/production_cd.py serve
+744
View File
@@ -0,0 +1,744 @@
#!/usr/bin/env python3
"""Governance application CD. The installed host copy owns the release boundary."""
import argparse
from contextlib import contextmanager
import copy
from datetime import datetime, timezone
import fcntl
import hashlib
from html.parser import HTMLParser
import json
import os
from pathlib import Path
import re
import stat
import subprocess
import sys
import tarfile
import tempfile
import time
from urllib.error import URLError
from urllib.parse import urlsplit
from urllib.request import HTTPRedirectHandler, ProxyHandler, build_opener
REPO = 'Weknora/knowledge-governance'
FORGE = 'https://gitea.zheromiku.top'
TARGET = 'cloud-test-governance-main'
ORG = '169e6f7f-e508-4a1f-9a9b-63d240439a3d'
SPACE = '106e05d1-4091-40b9-8c57-f76dc94f7825'
READY = '3e15556f-ca19-417c-8bfd-666634ea5d29'
PUBLISHED = '16185166-b8e9-4761-9926-845a1f41c96d'
ROLES = ('api', 'web', 'worker')
REVISION = 'org.opencontainers.image.revision'
DATA = Path('/opt/knowledge-governance/cd')
APP = DATA.parent
PROJECT = 'knowledge-governance-cloud-test'
SHA = re.compile('[0-9a-f]{40}')
HASH = re.compile('[0-9a-f]{64}')
IMAGE = re.compile('sha256:[0-9a-f]{64}')
UUID = re.compile('[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}')
GUARD_PREFIX = 'KG_CD_CONFIRM_V1 '
class CDError(RuntimeError):
pass
def require(ok, message):
if not ok:
raise CDError(message)
def stamp():
return datetime.now(timezone.utc).isoformat()
def read_json(path):
return json.loads(path.read_bytes())
def sync_dir(directory):
fd = os.open(directory, os.O_RDONLY | os.O_DIRECTORY)
try:
os.fsync(fd)
finally:
os.close(fd)
def atomic_write(path, data, mode=0o600):
fd, temp = tempfile.mkstemp(dir=path.parent)
try:
with os.fdopen(fd, 'wb') as out:
os.fchmod(out.fileno(), mode)
out.write(data)
out.flush()
os.fsync(out.fileno())
os.replace(temp, path)
sync_dir(path.parent)
finally:
Path(temp).unlink(missing_ok=True)
def record(path, value):
atomic_write(path, (json.dumps(value, ensure_ascii=False, sort_keys=True) + '\n').encode())
@contextmanager
def deploy_lock(path):
fd = os.open(path, os.O_CREAT | os.O_RDWR | os.O_NOFOLLOW, 0o600)
with os.fdopen(fd, 'r+') as handle:
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
raise CDError('deployment lock busy') from exc
yield
def matches(pattern, text):
return isinstance(text, str) and pattern.fullmatch(text) is not None
def validate_request(request):
actions = {'deploy': 'deploy-production', 'writeback': 'retry-writeback'}
action = request.get('action')
require(action in actions and request.get('confirm') == actions[action], 'explicit confirmation required')
require(matches(SHA, request.get('sha')) and matches(UUID, request.get('task'))
and matches(HASH, request.get('bundle')), 'require full SHA, bundle SHA256 and task UUID')
require(matches(re.compile('[1-9][0-9]{0,14}'), request.get('run_id')), 'invalid Gitea run ID')
def guard_proof(request, control_sha):
require(matches(SHA, control_sha), 'invalid workflow source SHA')
binding = {key: request[key] for key in ('run_id', 'action', 'confirm', 'sha', 'bundle', 'task')}
binding.update(repository=REPO, target=TARGET, workflow_sha=control_sha)
serialized = json.dumps(binding, sort_keys=True, separators=(',', ':')).encode()
return GUARD_PREFIX + hashlib.sha256(serialized).hexdigest()
def verify_guard_log(log, request, control_sha):
expected = guard_proof(request, control_sha)
pattern = re.compile(r'\d{4}-\d{2}-\d{2}T[0-9:.]+Z (' + GUARD_PREFIX + r'[0-9a-f]{64})')
proofs = [match.group(1) for line in log.decode().splitlines() if (match := pattern.fullmatch(line))]
require(proofs == [expected], 'Gitea guard dispatch proof missing or different')
def hash_file(path):
with path.open('rb') as src:
return hashlib.file_digest(src, 'sha256').hexdigest()
def immutable_file(path):
info = path.lstat()
require(stat.S_ISREG(info.st_mode) and info.st_uid == os.geteuid() and not info.st_mode & 0o022,
'bundle requires regular files owned by the deployment account')
return path
def verify_bundle(directory, request, archives=True):
raw = immutable_file(directory / 'release.json').read_bytes()
require(hashlib.sha256(raw).hexdigest() == request['bundle'], 'release manifest digest mismatch')
manifest = json.loads(raw)
require(manifest.get('schema') == 1 and manifest.get('target') == TARGET
and manifest.get('sha') == request['sha'] and manifest.get('task') == request['task'],
'bundle commit, task or target mismatch')
require(set(manifest.get('images', {})) == set(ROLES), 'bundle must include API, web and worker')
for role, image in manifest['images'].items():
require(matches(IMAGE, image.get('id')) and matches(HASH, image.get('sha256')), 'invalid image record')
if archives:
archive = immutable_file(directory / (role + '.tar'))
require(hash_file(archive) == image['sha256'], f'{role} archive checksum mismatch')
versions = manifest.get('migrations')
require(isinstance(versions, list) and versions and all(type(n) is int and n > 0 for n in versions)
and versions == sorted(set(versions)), 'invalid migration ledger in manifest')
return manifest
def verify_contexts(result, contexts):
latest = {}
for item in result.get('statuses') or []:
latest.setdefault(item.get('context'), item.get('status'))
require(bool(contexts) and all(latest.get(context) == 'success' for context in contexts),
'required CI status is missing or unsuccessful')
def metadata(task, key):
value = (task.get('metadata') or {}).get(key)
return json.loads(value) if isinstance(value, str) else None
def check_task(task, request, receipt=None):
require(task.get('id') == request['task'] and task.get('spaceId') == SPACE, 'HiFox task or space mismatch')
status = (task.get('status') or {}).get('id')
same_publication = receipt and metadata(task, 'cd_release') == receipt.get('hifox')
require(status == READY or (status == PUBLISHED and same_publication), 'HiFox task is not ready for release')
evidence = metadata(task, 'cd_acceptance') or {}
url = urlsplit(evidence.get('evidence', ''))
require(evidence.get('verdict') == 'passed' and evidence.get('sha') == request['sha']
and evidence.get('bundle') == request['bundle'] and url.scheme == 'https' and url.hostname
and not (url.username or url.password or url.query or url.fragment),
'missing or changed business acceptance for this commit and bundle')
return evidence
def release(host, request):
validate_request(request)
path = host.receipt_path
existing = read_json(path) if path.exists() else None
if request['action'] == 'writeback':
require(existing and existing['state'] in ('deployed_hifox_pending', 'published'),
'writeback requires a verified deployment receipt')
require(all(existing[k] == request[k] for k in ('sha', 'task', 'bundle')), 'receipt binding mismatch')
manifest = verify_bundle(host.bundle_path(request), request, archives=False)
host.approve(request)
host.verify(manifest)
return writeback(host, request, existing)
require(existing is None, 'existing release receipt: use writeback or inspect interrupted/failed deployment')
host.approve(request)
manifest = verify_bundle(host.bundle_path(request), request)
host.check_ci(request)
accepted = check_task(host.task(request['task']), request)
before = host.baseline()
host.pre_switch(before, manifest)
receipt = {k: request[k] for k in ('sha', 'task', 'bundle', 'run_id')}
receipt.update(state='preparing', target=TARGET, started_at=stamp(), previous=before)
record(path, receipt)
try:
host.load(host.bundle_path(request), manifest)
host.pre_switch(before, manifest)
host.approve(request)
require(check_task(host.task(request['task']), request) == accepted, 'business acceptance changed before switch')
except (CDError, OSError, ValueError, KeyError) as exc:
receipt.update(state='failed_before_cutover')
record(path, receipt)
raise CDError(f'failed_before_cutover: {exc}' if isinstance(exc, CDError) else 'failed_before_cutover') from exc
receipt.update(state='cutover_started')
record(path, receipt)
try:
host.switch(before, manifest)
evidence = host.verify(manifest)
except (CDError, OSError, ValueError, KeyError) as exc:
status = 'rollback_failed'
try:
host.rollback(before)
status = 'rolled_back'
except (CDError, OSError, ValueError, KeyError):
pass
receipt.update(state=status, finished_at=stamp(), failure=str(exc) if isinstance(exc, CDError) else 'operation failed')
record(path, receipt)
raise CDError(f'{status}: deployment verification failed') from exc
receipt.update(state='deployed_hifox_pending', deployed_at=stamp(), evidence=evidence)
receipt['hifox'] = {**evidence, 'target': TARGET, 'bundle': request['bundle'], 'deployed_at': receipt['deployed_at'],
'run_url': f'{FORGE}/{REPO}/actions/runs/{request["run_id"]}'}
record(path, receipt)
return writeback(host, request, receipt)
def writeback(host, request, receipt):
try:
task = host.task(request['task'])
check_task(task, request, receipt)
host.callback(request['task'], task, receipt.get('hifox'))
receipt.update(state='published')
record(host.receipt_path, receipt)
except (CDError, OSError, ValueError, KeyError) as exc:
raise CDError('DEPLOYED_HIFOX_PENDING: retry writeback for this receipt') from exc
return receipt
def with_images(config, images):
result = copy.deepcopy(config)
require(set(result['services']) == set(ROLES), 'unexpected Compose service set')
for role in ROLES:
result['services'][role]['image'] = images[role]
return result
def run(argv, purpose, *, cwd=None, input_data=None, timeout=30, allowed=(0,), env=None):
try:
result = subprocess.run(argv, cwd=cwd, input=input_data, capture_output=True, timeout=timeout, env=env)
except (OSError, subprocess.TimeoutExpired) as exc:
raise CDError(f'{purpose} unavailable or timed out') from exc
require(result.returncode in allowed, f'{purpose} failed (exit {result.returncode})')
return result.stdout
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, *_args, **_kwargs):
raise CDError('unexpected HTTP redirect')
def http(url, *, direct=False, limit=2**20):
handlers = [NoRedirect()]
if direct:
handlers.append(ProxyHandler({}))
try:
with build_opener(*handlers).open(url, timeout=10) as response:
data = response.read(limit + 1)
require(len(data) <= limit, 'HTTP response exceeds limit')
return data
except (URLError, TimeoutError) as exc:
raise CDError('HTTP read failed') from exc
class Assets(HTMLParser):
def __init__(self):
super().__init__()
self.paths = []
def handle_starttag(self, tag, attrs):
attrs = dict(attrs)
path = attrs.get('src' if tag == 'script' else 'href', '')
if path and re.fullmatch(r'/assets/[A-Za-z0-9_.-]+\.(?:js|css)', path):
self.paths.append(path)
def disk_assets(html):
parser = Assets()
parser.feed(html.decode())
require(any(p.endswith('.js') for p in parser.paths), 'web index has no compiled entry point')
return sorted(set(parser.paths))
def image_inspect(image):
return json.loads(run(['docker', 'image', 'inspect', image], 'read image identity'))[0]
def inspect(name):
obj = json.loads(run(['docker', 'container', 'inspect', name], 'read running container'))[0]
require(obj['State']['Status'] == 'running' and matches(IMAGE, obj.get('Image')), 'expected container is not running')
return obj
def image_label(image):
return image_inspect(image)['Config']['Labels'].get(REVISION, '')
def api_identity(container):
raw = run(['docker', 'exec', container, '/app/api', '--version-record-status'], 'read API build identity', allowed=(0, 1))
obj = json.loads(raw)
require(matches(SHA, obj.get('commit_sha')), 'API binary has no full build identity')
return obj['commit_sha']
def migration_files(git, sha):
paths = git('ls-tree', '-r', '--name-only', sha, 'internal/platform/postgres/migrations').decode().splitlines()
return sorted(int(Path(line).name.split('_', 1)[0]) for line in paths if line.endswith('.up.sql'))
def ledger(container):
probe = Path(__file__).with_name('cd_migration_probe.py').read_text()
data = run(['docker', 'exec', container, 'python3', '-c', probe], 'read migration ledger')
result = json.loads(data)
require(isinstance(result, list) and result, 'unexpected migration ledger result')
return result
def verify_archive(archive, expected):
# docker save by ID omits mutable RepoTags. Refuse archives that could retag another app.
with tarfile.open(archive, 'r:') as t:
entry = t.getmember('manifest.json')
require(entry.isfile() and entry.size < 65536, 'unexpected Docker archive manifest')
containers = json.load(t.extractfile(entry))
require(len(containers) == 1 and not containers[0].get('RepoTags'), 'archive must contain one untagged image')
config = t.getmember(containers[0]['Config'])
require(config.isfile() and config.size < 2**20, 'unexpected Docker image config')
require('sha256:' + hashlib.sha256(t.extractfile(config).read()).hexdigest() == expected,
'archive config does not match approved image ID')
for member in t:
require((member.isdir() or member.isfile()) and not member.name.startswith('/')
and '..' not in Path(member.name).parts, 'unexpected Docker archive path')
if 'index.json' in t.getnames():
index = t.getmember('index.json')
require(index.isfile() and index.size < 65536, 'unexpected OCI archive index')
oci = json.load(t.extractfile(index))
require(len(oci.get('manifests', [])) == 1 and all(
key not in descriptor.get('annotations', {}) for descriptor in oci['manifests']
for key in ('org.opencontainers.image.ref.name', 'io.containerd.image.name')),
'archive OCI index contains a mutable name')
class LiveHost:
def __init__(self, request):
self.receipt_path = DATA / 'receipts' / (request['bundle'] + '.json')
self.compose = APP / 'compose.json'
self.git_path = DATA / 'source.git'
self.containers = {role: 'kg-cloud-' + role for role in ROLES}
self.health_url = 'http://127.0.0.1:8088'
self.request = request
def bundle_path(self, request):
return DATA / 'bundles' / request['bundle']
def git(self, *args):
return run(['git', '-C', str(self.git_path), *args], 'release source check', timeout=90)
def approve(self, request):
repo_url = f'{FORGE}/{REPO}'
run_id = request['run_id']
run_api = f'{FORGE}/api/v1/repos/{REPO}/actions/runs/{run_id}'
dispatch = json.loads(http(run_api))
control_sha = dispatch.get('head_sha')
trigger_actor = dispatch.get('trigger_actor') or dispatch.get('actor') or {}
require(str(dispatch.get('id')) == run_id and dispatch.get('event') == 'workflow_dispatch'
and dispatch.get('path') == 'deploy-production.yml@refs/heads/main'
and dispatch.get('head_branch') == 'main' and dispatch.get('status') in ('in_progress', 'running')
and (dispatch.get('actor') or {}).get('id') == 1 and trigger_actor.get('id') == 1
and matches(SHA, control_sha), 'Gitea run is not an active deployment dispatched by the release owner')
# Repo secrets can be read by other workflows. Only a completed guard from
# the installed workflow and controller may authorize the exact inputs.
try:
self.git('merge-base', '--is-ancestor', control_sha, 'refs/remotes/origin/main')
except CDError:
self.fetch_source()
self.git('merge-base', '--is-ancestor', control_sha, 'refs/remotes/origin/main')
trusted = {'scripts/production_cd.py': Path(__file__).resolve(),
'.gitea/workflows/deploy-production.yml': DATA / 'deploy-production.yml'}
for name, local in trusted.items():
require(self.git('show', f'{control_sha}:{name}') == immutable_file(local).read_bytes(),
'Gitea approval workflow differs from installed control files')
jobs = json.loads(http(run_api + '/jobs')).get('jobs') or []
guards = [job for job in jobs if job.get('name') == 'guard' and job.get('conclusion') == 'success'
and str(job.get('run_id')) == run_id and job.get('head_sha') == control_sha
and matches(re.compile('[1-9][0-9]{0,14}'), str(job.get('id')))]
require(len(guards) == 1, 'Gitea approval guard has not succeeded')
guard_id = guards[0]['id']
verify_guard_log(http(f'{repo_url}/actions/runs/{run_id}/jobs/{guard_id}/logs'), request, control_sha)
def fetch_source(self):
self.git('fetch', '--no-tags', '--prune', f'{FORGE}/{REPO}.git', '+refs/heads/*:refs/remotes/origin/*')
def check_ci(self, request):
self.fetch_source()
refs = self.git('for-each-ref', '--format=%(refname)', '--contains', request['sha'], 'refs/remotes/origin/')
require(refs.strip(), 'candidate commit is not reachable from a repository branch')
contexts = immutable_file(DATA / 'required-contexts.txt').read_text().splitlines()
verify_contexts(json.loads(http(f'{FORGE}/api/v1/repos/{REPO}/commits/{request["sha"]}/status')),
[line.strip() for line in contexts if line.strip() and not line.lstrip().startswith('#')])
def hifox(self, *args):
cli = DATA / 'bin/hifox'
env = {'PATH': '/usr/local/bin:/usr/bin:/bin', 'HOME': str(DATA / 'hifox-home'), 'CI': '1'}
raw = run([str(cli), '--organization-id', ORG, *args, '--output', 'json'], 'HiFox operation', env=env)
result = json.loads(raw)
require(isinstance(result, dict) and result.get('success') is not False and not result.get('error'),
'HiFox did not return a successful object')
return result
def task(self, task):
return self.hifox('task', 'view', task)
def callback(self, task_id, task, expected):
if metadata(task, 'cd_release') != expected:
self.hifox('task', 'metadata', 'set', task_id, 'cd_release', json.dumps(json.dumps(expected)))
fresh = self.task(task_id)
check_task(fresh, self.request, {'hifox': expected})
require(metadata(fresh, 'cd_release') == expected, 'HiFox release receipt readback mismatch')
if fresh['status']['id'] != PUBLISHED:
self.hifox('task', 'edit', task_id, '--status', PUBLISHED)
final = self.task(task_id)
final_status = (final.get('status') or {}).get('id')
require(final_status == PUBLISHED and metadata(final, 'cd_release') == expected, 'HiFox published readback mismatch')
def compose_cmd(self, *args):
return ['docker', 'compose', '-p', PROJECT, '-f', str(self.compose), *args]
def rendered_config(self):
raw = run(self.compose_cmd('config', '--format', 'json'), 'render Compose configuration', cwd=APP)
return json.loads(raw)['services']
def runtime(self):
rendered = self.rendered_config()
require(set(rendered) == set(ROLES), 'unexpected Compose service set')
result = {}
for role, name in self.containers.items():
c = inspect(name)
labels = c['Config']['Labels']
require(labels.get('com.docker.compose.project') == PROJECT
and labels.get('com.docker.compose.service') == role
and labels.get('com.docker.compose.project.working_dir') == str(APP)
and labels.get('com.docker.compose.project.config_files') == str(self.compose),
f'{role} does not belong to the approved Compose chain')
config = rendered[role]
image = image_inspect(c['Image'])
require(image_inspect(config['image'])['Id'] == c['Image'], f'{role} Compose image has drifted')
env = lambda entries: dict(item.split('=', 1) for item in entries or [])
expected_env = env(image['Config'].get('Env'))
expected_env.update({key: str(value) for key, value in (config.get('environment') or {}).items()
if value is not None})
require(env(c['Config'].get('Env')) == expected_env, f'{role} running environment has drifted')
expected_mounts = config.get('volumes') or []
require(all(mount['type'] == 'bind' for mount in expected_mounts), 'unexpected non-bind Compose volume')
mounts = {(v['Source'], v['Destination'], not v['RW']) for v in c.get('Mounts', []) if v['Type'] == 'bind'}
require(mounts == {(v['source'], v['target'], v.get('read_only', False)) for v in expected_mounts}
and len(mounts) == len(c.get('Mounts', [])), f'{role} running mounts have drifted')
require(not config.get('network_mode') or config['network_mode'] == c['HostConfig']['NetworkMode'],
f'{role} running network has drifted')
result[role] = c['Image']
return result
def baseline(self):
images = self.runtime()
self.probe_web()
return {'compose_sha256': hash_file(self.compose), 'images': images,
'revisions': {role: image_label(image) for role, image in images.items()}}
def pre_switch(self, previous, manifest):
require(hash_file(self.compose) == previous['compose_sha256'] and self.runtime() == previous['images'],
'target baseline changed during deployment')
for revision in set(previous['revisions'].values()):
require(matches(SHA, revision), 'running image has no source revision')
self.git('merge-base', '--is-ancestor', revision, manifest['sha'])
require(migration_files(self.git, manifest['sha']) == manifest['migrations'], 'source migration set differs from bundle')
for role in ('api', 'worker'):
require(ledger(self.containers[role]) == manifest['migrations'],
f'{role} migration ledger mismatch; verify schema changes outside CD')
def load(self, directory, manifest):
archives = {role: directory / (role + '.tar') for role in ROLES}
for role, archive in archives.items():
verify_archive(archive, manifest['images'][role]['id'])
for role, archive in archives.items():
image = manifest['images'][role]['id']
run(['docker', 'image', 'load', '-i', str(archive)], f'load {role} image', timeout=600)
require(image_inspect(image)['Id'] == image and image_label(image) == manifest['sha'],
f'{role} loaded image differs from approved build')
@property
def backup(self):
return self.receipt_path.with_suffix('.compose.backup')
def switch(self, previous, manifest):
current = self.compose.read_bytes()
require(hashlib.sha256(current).hexdigest() == previous['compose_sha256'], 'Compose changed before switch')
images = {role: manifest['images'][role]['id'] for role in ROLES}
changed = with_images(json.loads(current), images)
for role, image in previous['images'].items():
run(['docker', 'image', 'tag', image, f'knowledge-governance-cd-rollback:{self.request["bundle"]}-{role}'],
'retain rollback image')
atomic_write(self.backup, current)
atomic_write(self.compose, (json.dumps(changed, indent=2) + '\n').encode(), self.compose.stat().st_mode & 0o777)
self.recreate()
def recreate(self):
script = Path(__file__).with_name('deploy-api-web.sh')
run(['bash', str(script), '--verified-compose', str(APP), PROJECT, str(self.compose), 'api', 'web', 'worker'],
'switch governance Compose services', timeout=120)
def probe_web(self):
origin = self.health_url
require(json.loads(http(origin + '/health', direct=True)).get('status') == 'ok', 'web proxy health probe failed')
require(json.loads(http('http://127.0.0.1:18098/readyz', direct=True)).get('status') in ('ok', 'ready'),
'API readiness probe failed')
index = http(origin + '/', direct=True)
return index, {path: hashlib.sha256(http(origin + path, direct=True, limit=32 * 2**20)).hexdigest()
for path in disk_assets(index)}
def verify_web_image(self):
origin_index, assets = self.probe_web()
name = self.containers['web']
def read_asset(path):
return run(['docker', 'exec', name, 'cat', '/usr/share/nginx/html' + path], 'read running web asset')
require(origin_index == read_asset('/index.html'), 'served index differs from running web image')
for path, served_digest in assets.items():
require(hashlib.sha256(read_asset(path)).hexdigest() == served_digest, 'served asset differs from running web image')
return assets
def verify(self, manifest):
images = {role: manifest['images'][role]['id'] for role in ROLES}
def check():
require(self.runtime() == images, 'running image IDs differ from approved bundle')
require(api_identity(self.containers['api']) == manifest['sha'], 'running API SHA mismatches approval')
assets = self.verify_web_image()
return {'sha': manifest['sha'], 'images': images, 'health': 'ok', 'asset_sha256': assets}
return wait_ready(check)
def rollback(self, previous):
if not self.backup.exists():
require(hash_file(self.compose) == previous['compose_sha256'] and self.runtime() == previous['images'],
'rollback baseline cannot be established')
self.verify_web_image()
return
original = self.backup.read_bytes()
require(hashlib.sha256(original).hexdigest() == previous['compose_sha256'], 'rollback backup checksum mismatch')
source = json.loads(original)
for role, image in previous['images'].items():
reference = source['services'][role]['image']
require(image_inspect(reference)['Id'] == image, 'rollback source tag has drifted')
atomic_write(self.compose, original, self.compose.stat().st_mode & 0o777)
self.recreate()
def check():
require(self.runtime() == previous['images'], 'rollback running images mismatch')
require(api_identity(self.containers['api']) == previous['revisions']['api'], 'rollback API SHA mismatch')
self.verify_web_image()
wait_ready(check)
def wait_ready(check, seconds=90):
deadline = time.monotonic() + seconds
while True:
try:
return check()
except (CDError, OSError, KeyError, ValueError):
if time.monotonic() >= deadline:
raise
time.sleep(3)
def event_request(event, env):
require(env.get('GITHUB_EVENT_NAME') == 'workflow_dispatch' and env.get('GITHUB_REF') == 'refs/heads/main'
and env.get('GITHUB_REPOSITORY') == REPO, 'manual dispatch from main required')
inputs = event.get('inputs') or {}
request = {name: inputs.get(name, '') for name in ('action', 'sha', 'task', 'bundle', 'confirm')}
request['run_id'] = env.get('GITHUB_RUN_ID', '')
validate_request(request)
return request
def pack(args):
require(matches(SHA, args.sha) and matches(UUID, args.task), 'require full SHA and task UUID')
require(not args.output.exists(), 'bundle output already exists')
source = lambda *argv: run(['git', *argv], 'read candidate source')
require(source('rev-parse', args.sha + '^{commit}').decode().strip() == args.sha, 'source commit not found')
images = {}
versions = migration_files(source, args.sha)
manifest = dict(schema=1, target=TARGET, sha=args.sha, task=args.task, images=images, migrations=versions)
with tempfile.TemporaryDirectory(dir=args.output.parent) as directory:
build = Path(directory) / "sealed"
build.mkdir(mode=0o700)
for role in ROLES:
image = image_inspect(getattr(args, role).removeprefix('sha256:'))
image_id = image['Id']
require(matches(IMAGE, image_id) and image_label(image_id) == args.sha, f'{role} source label mismatch')
archive = build / (role + '.tar')
run(['docker', 'image', 'save', '-o', str(archive), image_id], f'archive {role} image', timeout=900)
verify_archive(archive, image_id)
archive.chmod(0o600)
images[role] = {'id': image_id, 'sha256': hash_file(archive)}
record(build / 'release.json', manifest)
for item in build.iterdir():
with item.open("rb") as artifact:
os.fsync(artifact.fileno())
sync_dir(build)
build.rename(args.output)
sync_dir(args.output.parent)
print(hash_file(args.output / 'release.json'))
def attest(args):
request = {'sha': args.sha, 'task': args.task, 'bundle': hash_file(args.directory / 'release.json')}
require(matches(SHA, args.sha) and matches(UUID, args.task), 'require full SHA and task UUID')
manifest = verify_bundle(args.directory, request)
before = {role: inspect(getattr(args, role)) for role in ROLES}
for role, obj in before.items():
require(obj['Image'] == manifest['images'][role]['id'] and obj['Name'] != '/kg-cloud-' + role,
'isolated test runtime image differs from bundle')
require(api_identity(args.api) == args.sha and ledger(args.api) == manifest['migrations']
and ledger(args.worker) == manifest['migrations'], 'test runtime SHA or schema differs')
base = urlsplit(args.test_url)
require(base.scheme in ('http', 'https') and base.hostname and not (base.username or base.password or base.query or base.fragment)
and base.path in ('', '/') and base.port != 8088, 'provide isolated test endpoint')
origin = args.test_url.rstrip('/')
require(json.loads(http(origin + '/health', direct=True)).get('status') == 'ok', 'isolated test health probe failed')
index = http(origin + '/', direct=True)
require(index == run(['docker', 'exec', args.web, 'cat', '/usr/share/nginx/html/index.html'], 'read test web index'),
'test endpoint is serving another web image')
for role, obj in before.items():
require(inspect(getattr(args, role))['Id'] == obj['Id'], 'test instance changed during readback')
print(json.dumps({'sha': args.sha, 'bundle': request['bundle'], 'test_url': args.test_url,
'images': {role: obj['Image'] for role, obj in before.items()}}, sort_keys=True))
def cli():
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest='command', required=True)
for command in ('guard', 'send', 'serve', 'pack', 'attest'):
sub = commands.add_parser(command)
if command in ('guard', 'send'):
sub.add_argument('--event', type=Path, required=True)
if command in ('pack', 'attest'):
sub.add_argument('--sha', required=True)
sub.add_argument('--task', required=True)
if command in ('pack', 'attest'):
for role in ROLES:
sub.add_argument('--' + role, required=True)
if command == 'pack':
sub.add_argument('--output', type=Path, required=True)
if command == 'attest':
sub.add_argument('--directory', type=Path, required=True)
sub.add_argument('--test-url', required=True)
args = parser.parse_args()
if args.command in ('guard', 'send'):
request = event_request(read_json(args.event), os.environ)
if args.command == 'guard':
print(guard_proof(request, os.environ.get('GITHUB_SHA')))
else:
send(request)
elif args.command == 'pack':
pack(args)
elif args.command == 'attest':
attest(args)
else:
raw = sys.stdin.buffer.read(8193)
require(len(raw) <= 8192, 'request exceeds size limit')
request = json.loads(raw)
validate_request(request)
with deploy_lock(DATA / 'release.lock'):
host = LiveHost(request)
for item in (DATA / 'receipts').glob('*.json'):
if read_json(item)['state'] in ('cutover_started', 'rollback_failed'):
raise CDError('previous deployment outcome requires operator reconciliation')
result = release(host, request)
print(json.dumps({k: result[k] for k in ('state', 'sha', 'task', 'bundle', 'hifox')}, ensure_ascii=False))
def send(request):
key = os.environ.get('KG_CD_SSH_KEY', '')
known_hosts = os.environ.get('KG_CD_KNOWN_HOSTS', '')
host = os.environ.get('KG_CD_SSH_HOST', '')
port = os.environ.get('KG_CD_SSH_PORT', '22') or '22'
require(key.strip() and known_hosts.strip(), 'missing KG_CD_SSH_KEY or KG_CD_KNOWN_HOSTS')
require(host in ('100.85.111.121', '::1') and port.isdecimal() and 1 <= int(port) <= 65535,
'configure the direct or approved tunnel address for the target release endpoint')
with tempfile.TemporaryDirectory() as directory:
temp = Path(directory)
for filename, value in (('key', key), ('known_hosts', known_hosts)):
atomic_write(temp / filename, (value.strip() + '\n').encode())
argv = ['ssh', '-F', '/dev/null', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', '-o', 'IdentitiesOnly=yes',
'-o', 'ClearAllForwardings=yes', '-o', 'StrictHostKeyChecking=yes', '-o', 'HostKeyAlias=kg-governance-cd',
'-o', 'GlobalKnownHostsFile=/dev/null', '-o', f'UserKnownHostsFile={temp / "known_hosts"}',
'-i', str(temp / 'key'), '-p', port, '-T', 'root@' + host, 'kg-cd-v1']
try:
result = subprocess.run(argv, input=json.dumps(request).encode(), capture_output=True, timeout=2100)
except (OSError, subprocess.TimeoutExpired) as exc:
raise CDError('target outcome unknown; inspect durable receipt before retry') from exc
if result.returncode:
error = result.stderr.decode(errors='replace')
for marker in ('DEPLOYED_HIFOX_PENDING', 'rollback_failed:', 'rolled_back:', 'failed_before_cutover:',
'target preflight rejected', 'deployment lock busy', 'writeback requires a verified deployment receipt',
'Gitea approval guard has not succeeded', 'Gitea guard dispatch proof missing or different'):
if marker in error:
raise CDError(error.strip().splitlines()[-1])
raise CDError('target rejected request or outcome unknown; inspect durable receipt')
response = json.loads(result.stdout)
require(response.get('state') == 'published' and all(response.get(k) == request[k] for k in ('sha', 'task', 'bundle')),
'invalid or unverified deployment receipt')
print(json.dumps(response, ensure_ascii=False))
if __name__ == '__main__':
try:
cli()
except CDError as error:
print(f'CD: {error}', file=sys.stderr)
sys.exit(1)
except (OSError, KeyError, ValueError, TypeError, tarfile.TarError) as error:
print(f'CD: operation could not be verified ({type(error).__name__})', file=sys.stderr)
sys.exit(1)
+157
View File
@@ -0,0 +1,157 @@
import hashlib
import json
import tempfile
import unittest
from pathlib import Path
from unittest.mock import Mock
import production_cd as cd
SHA = 'a' * 40
TASK = 'd5abdf29-d0b5-4992-8073-0cb8fffd84ed'
IMAGES = {role: 'sha256:' + str(i) * 64 for i, role in enumerate(('api', 'web', 'worker'), 1)}
CTX = 'build / build (push)'
class ProductionCD(unittest.TestCase):
def setUp(self):
temp = tempfile.TemporaryDirectory()
self.addCleanup(temp.cleanup)
self.root = Path(temp.name)
self.request = dict(sha=SHA, task=TASK, bundle='b' * 64, confirm='deploy-production',
action='deploy', run_id='42')
self.manifest = dict(schema=1, target=cd.TARGET, sha=SHA, task=TASK, migrations=[1, 2], images={})
for role, image in IMAGES.items():
archive = self.root / (role + '.tar')
archive.write_text(role)
self.manifest['images'][role] = dict(id=image, sha256=hashlib.sha256(role.encode()).hexdigest())
data = json.dumps(self.manifest).encode()
(self.root / 'release.json').write_bytes(data)
for path in self.root.iterdir():
path.chmod(0o600)
self.request['bundle'] = hashlib.sha256(data).hexdigest()
self.acceptance = {'verdict': 'passed', 'sha': SHA, 'bundle': self.request['bundle'],
'evidence': 'https://example.test/result/42'}
self.host = Mock()
self.host.task.return_value = {'id': TASK, 'spaceId': cd.SPACE, 'status': {'id': cd.READY},
'metadata': {'cd_acceptance': json.dumps(self.acceptance)}}
self.host.baseline.return_value = {'before': 'baseline'}
self.host.pre_switch.return_value = None
self.host.verify.return_value = {'sha': SHA, 'images': IMAGES, 'health': 'ok'}
self.host.receipt_path = self.root / 'receipt.json'
self.host.bundle_path.return_value = self.root
def execute(self):
return cd.release(self.host, self.request)
def assert_no_deploy(self):
self.host.load.assert_not_called()
self.host.switch.assert_not_called()
self.host.callback.assert_not_called()
def test_no_confirmation_has_zero_side_effects(self):
self.request['confirm'] = ''
with self.assertRaisesRegex(cd.CDError, 'confirmation'):
self.execute()
self.host.assert_not_called()
self.assert_no_deploy()
def test_preconditions_prevent_loading_and_switching(self):
for mode in ('approval', 'ci', 'test', 'changed_bundle', 'tampered_archive', 'migration'):
with self.subTest(mode=mode):
self.setUp()
if mode in ('approval', 'ci', 'migration'):
step = {'approval': self.host.approve, 'ci': self.host.check_ci,
'migration': self.host.pre_switch}[mode]
step.side_effect = cd.CDError(mode)
elif mode == 'test':
self.host.task.return_value['metadata'].clear()
elif mode == 'changed_bundle':
self.request['bundle'] = 'f' * 64
else:
(self.root / 'web.tar').write_bytes(b'other image')
with self.assertRaises(cd.CDError):
self.execute()
self.assert_no_deploy()
def test_bare_ci_success_cannot_replace_required_context(self):
for statuses in ([], [{'context': 'different', 'status': 'success'}],
[{'context': CTX, 'status': 'failure'}, {'context': CTX, 'status': 'success'}]):
with self.assertRaises(cd.CDError):
cd.verify_contexts({'state': 'success', 'statuses': statuses}, [CTX])
cd.verify_contexts({'state': 'success', 'statuses': [{'context': CTX, 'status': 'success'}]}, [CTX])
def test_approval_and_acceptance_rechecked_before_switch(self):
self.host.approve.side_effect = [None, cd.CDError('revoked')]
with self.assertRaisesRegex(cd.CDError, 'revoked'):
self.execute()
self.host.switch.assert_not_called()
self.host.callback.assert_not_called()
def test_success_and_writeback_retry_do_not_switch_twice(self):
self.host.callback.side_effect = [cd.CDError('HiFox timeout'), None]
with self.assertRaisesRegex(cd.CDError, 'DEPLOYED_HIFOX_PENDING'):
self.execute()
self.assertEqual(cd.read_json(self.host.receipt_path)['state'], 'deployed_hifox_pending')
self.request.update(action='writeback', confirm='retry-writeback')
self.assertEqual(self.execute()['state'], 'published')
self.host.switch.assert_called_once()
self.host.load.assert_called_once()
self.assertEqual(self.host.verify.call_count, 2)
def test_writeback_without_receipt_refuses(self):
self.request.update(action='writeback', confirm='retry-writeback')
with self.assertRaisesRegex(cd.CDError, 'receipt'):
self.execute()
self.assert_no_deploy()
def test_failed_cutover_rolls_back_and_never_marks_published(self):
for result in ('rolled_back', 'rollback_failed'):
with self.subTest(result=result):
self.setUp()
self.host.verify.side_effect = cd.CDError('new unhealthy')
if result == 'rollback_failed':
self.host.rollback.side_effect = cd.CDError('old unhealthy')
with self.assertRaisesRegex(cd.CDError, result):
self.execute()
self.assertEqual(cd.read_json(self.host.receipt_path)['state'], result)
self.host.rollback.assert_called_once()
self.host.callback.assert_not_called()
with self.assertRaisesRegex(cd.CDError, 'receipt'):
self.execute()
self.host.switch.assert_called_once()
def test_gitea_guard_log_binds_the_exact_dispatch(self):
trusted = 'f' * 40
proof = cd.guard_proof(self.request, trusted)
log = f'2026-11-22T17:31:54.013Z {proof}\n'.encode()
cd.verify_guard_log(log, self.request, trusted)
for name, altered in (('sha', 'b' * 40), ('bundle', 'c' * 64), ('run_id', '43'),
('task', 'd5abdf29-d0b5-4992-8073-0cb8fffd84ee2'), ('confirm', 'retry-writeback')):
with self.subTest(name=name), self.assertRaisesRegex(cd.CDError, 'dispatch proof'):
cd.verify_guard_log(log, {**self.request, name: altered}, trusted)
with self.assertRaisesRegex(cd.CDError, 'dispatch proof'):
cd.verify_guard_log(log + log, self.request, trusted)
def test_target_lock_is_exclusive(self):
path = self.root / 'lock'
with cd.deploy_lock(path):
with self.assertRaisesRegex(cd.CDError, 'deployment lock'):
with cd.deploy_lock(path):
self.fail('lock was acquired twice')
def test_persisted_compose_changes_only_images(self):
original = {'services': {role: {'image': role + ':old', 'env_file': [{'path': '/kept', 'format': 'raw'}],
'volumes': ['/kept:/run/key:ro'], 'environment': {'GATE': 'true'}}
for role in IMAGES}, 'name': 'governance'}
changed = cd.with_images(original, IMAGES)
for role, new in changed['services'].items():
self.assertEqual(new.pop('image'), IMAGES[role])
old = original['services'][role].copy()
old.pop('image')
self.assertEqual(new, old)
if __name__ == '__main__':
unittest.main()