chore(deploy): 部署与备份脚本、示例配置与忽略规则
build / build (push) Failing after 21s

- 新增 scripts/deploy-api-web.sh(API/Web 构建发布)与 scripts/wiki_cleanup_backup.sh(wiki 清理前备份)
- 更新 .env.example、compose.yaml 与 README 说明;忽略 /spool/ingestion 运行期产物
This commit is contained in:
姚章浩
2026-09-30 10:53:25 +08:00
parent 61c0811228
commit e1c5f6895a
7 changed files with 155 additions and 1 deletions
+5
View File
@@ -7,6 +7,11 @@ WORKER_LEASE_DURATION=30s
WORKER_RETRY_DELAY=5s
RELEASE_WORKER_ENABLED=false
PRODUCTION_RELEASE_REQUESTS_ENABLED=false
# Read-only probe of the production WeKnora endpoint (console shows its real
# state); the publish target itself stays in RELEASE_CONFIG_FILE.
PRODUCTION_WEKNORA_BASE_URL=
PRODUCTION_WEKNORA_API_KEY_FILE=
PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP=false
PP_SCAN_ENABLED=false
PP_SCAN_HOST_DATA_ROOT=/home/zheromiku/dobot-portal-sync/data
PP_SCAN_CONTAINER_DATA_ROOT=/home/zheromiku/dobot-portal-sync/data
+4
View File
@@ -31,3 +31,7 @@
__pycache__/
/dist/
config/release.wiki-e2e.json
backups/
# 运行期灌库产物(xlsx/pdf/docx 原件,百兆级),只在本机 spool 里流转,绝不入库
/spool/ingestion/
+44
View File
@@ -140,6 +140,50 @@ RELEASE_WORKER_ENABLED=false
release executions/items/jobs=0/0/0
```
## 服务配置在哪配(四项)
「系统设置 → 服务配置」只保留四项,其他都不在这页:
| 项 | 含义 | 校验 | 谁来写 |
| --- | --- | --- | --- |
| 测试地址 | 控制台探活的测试 WeKnora 地址 | 裸 http(s) 源地址 | 仅管理员 |
| 生产地址 | 控制台探活的生产 WeKnora 地址 | 裸 http(s) 源地址 | 仅管理员 |
| 生产 ID | 发布契约里的租户 ID | 十进制整数(空 = 未配置);保存时真查它是否就是该知识库的租户 | 仅管理员 |
| 知识库 ID | 生产知识库标识 | 不含空格、`/`、`?`、`#`、`\`(空 = 未配置);保存时必须在该地址上真实存在 | 仅管理员 |
一次点「保存并启用服务配置」四项一起提交:服务端先格式校验,再写版本化的活动设置(候选 → 激活 →
旧版本保留审计),网页只录裸地址、绝不夹带凭据。四项里任何一项相对活动版本发生变化都要管理员角色,
普通工程师提交会被 403(只改刷新参数不受影响)。留空表示未配置,界面如实显示「未配置」。
- 地址探活:后端按地址真打 `/health`,显示「端点可达 / 不可达 · 未做接单验证」;测试地址留空时回落到
容器 env `TEST_WEKNORA_BASE_URL`,生产地址回落到 `PRODUCTION_WEKNORA_BASE_URL`(可选
`PRODUCTION_WEKNORA_API_KEY_FILE`、`PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP=false`)。探活只打
`/health`,从不读写知识库、从不参与发布,所以生产端点抖动不会阻断测试流水线。
- **两个 ID 的真校验(保存时执行)**:保存/激活服务配置时,后端拿「生产地址 + 生产 API Key 文件」
调 WeKnora 列库接口(`GET /api/v1/knowledge-bases/{id}`,回落到 `GET /api/v1/knowledge-bases`
列租户)真查一次:
- 查到 → 通过,界面显示知识库名与观测到的租户(`verification.knowledge_base_name` /
`observed_tenant_id`,服务端只读,从不写回配置);
- 端点明确回答「没有这个知识库」或租户与该库不符 → **拒绝保存**(HTTP 400 + 原话),一行都不写;
- 端点抖动、超时、`401/403`、未配置生产凭据 → **只警告不拦**(响应里带 `verification.status`
= `unreachable` / `unauthorized` / `cannot_verify`),保存照常落地——运维不能因为端点故障就改不了地址。
真校验需要 api 容器能读到生产凭据:`PRODUCTION_WEKNORA_API_KEY_FILE` 指向 0600 文件(值永不进网页、
永不进响应);留空时该项显示「未核验:没有可用的生产 API Key 文件」,其余行为不变。
激活前会再查一次(候选可以躺几天,库可能已被删);字段与线上一致的回放不重复核验,幂等重试不受端点影响。
覆盖率要点:租户校验只在「该库的 tenant_id」这一条路径上成立;只填生产 ID 时按列库结果的租户去比对。
- 排列组合之外的东西(依赖探活矩阵、对内/对外知识库绑定下拉)已从这页摘掉;依赖状态在总览页,
绑定数据与 API 仍在(`internal/httpapi/routing_bindings.go`,迁移 000034)。
发布目标与凭据仍然是另一件事,刻意不进网页:
1. 发布通道的目标(唯一真源,Release Worker 读它):`RELEASE_CONFIG_FILE` 指向的 0600 JSON 里
`routes[].target.weknora.base_url`(同级 `api_key_file`),模板见 `config/release.v071.example.json`。
硬约束:绝对路径、0600 普通文件、严格 JSON(多字段即报错)、source/target 不能是同一库或同一桶。
凭据与发布目标刻意不进网页、不回读——它只在管理员受限的运行时配置里。
所以:网页上配的地址管「控制台看到的真实状态」,真正发单时的目标仍由发布运行时配置决定。改了一处却
忘了另一处时,控制台会显示旧地址的真实状态——这是提示,不是故障。
## 本地验证
要求Go 1.26、Node.js 22和npm。
+5
View File
@@ -43,6 +43,11 @@ services:
WORKER_LIVENESS_STALE_AFTER: "${WORKER_LIVENESS_STALE_AFTER:-2m}"
PRODUCTION_RELEASE_REQUESTS_ENABLED: ${PRODUCTION_RELEASE_REQUESTS_ENABLED:-false}
PRODUCTION_BACKUP_RESTORE_VERIFIED: ${PRODUCTION_BACKUP_RESTORE_VERIFIED:-false}
# Read-only probe of the production WeKnora endpoint; publishing keeps
# reading its target from RELEASE_CONFIG_FILE.
PRODUCTION_WEKNORA_BASE_URL: "${PRODUCTION_WEKNORA_BASE_URL:-}"
PRODUCTION_WEKNORA_API_KEY_FILE: "${PRODUCTION_WEKNORA_API_KEY_FILE:-}"
PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP: "${PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP:-false}"
secrets:
- pgpass
- test_weknora_api_key
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# 重建并上线 governance api + web(本机 docker build 需要宿主代理 + --network host)
set -u
cd /home/zheromiku/knowledge-governance || exit 1
TS=$(date +%Y%m%d-%H%M%S)
PROXY_ARGS=(--build-arg HTTP_PROXY=http://127.0.0.1:7890 --build-arg HTTPS_PROXY=http://127.0.0.1:7890 --build-arg NO_PROXY=localhost,127.0.0.1 --build-arg GOPROXY=https://goproxy.cn,direct)
echo "=== 1. 打回滚标签 ($TS)"
docker tag knowledge-governance-p1-api:latest "knowledge-governance-p1-api:rollback-$TS" || exit 1
docker tag knowledge-governance-p1-web:latest "knowledge-governance-p1-web:rollback-$TS" || exit 1
echo "=== 2. 构建 api"
docker build --network host "${PROXY_ARGS[@]}" -f Dockerfile -t knowledge-governance-p1-api:latest . || { echo "API_BUILD_FAILED"; exit 1; }
echo "=== 3. 构建 web"
docker build --network host "${PROXY_ARGS[@]}" -f Dockerfile.web -t knowledge-governance-p1-web:latest . || { echo "WEB_BUILD_FAILED"; exit 1; }
echo "=== 4. 上线 api + web(不动依赖)"
docker compose up -d --no-build --no-deps api web || { echo "UP_FAILED"; exit 1; }
echo "=== 5. 状态"
docker ps --filter name=knowledge-governance-p1 --format '{{.Names}} {{.Status}} {{.Image}}'
echo "api image=$(docker inspect -f '{{.Image}}' knowledge-governance-p1-api-1)"
echo "web image=$(docker inspect -f '{{.Image}}' knowledge-governance-p1-web-1)"
echo "=== 6. 健康"
echo "health=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8088/health)"
echo "=== 7. 生产端点环境变量(只报名字与非空)"
docker exec knowledge-governance-p1-api-1 sh -lc 'for v in PRODUCTION_WEKNORA_BASE_URL PRODUCTION_WEKNORA_API_KEY_FILE PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP; do eval "val=\$$v"; if [ -n "$val" ]; then echo "$v=<set>"; else echo "$v=<empty>"; fi; done'
echo "=== 8. 新包"
docker exec knowledge-governance-p1-web-1 sh -lc 'grep -o "index-[A-Za-z0-9_-]*\.js" /usr/share/nginx/html/index.html | head -1'
echo "DONE"
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# 清理前备份:把待删知识库/平台记录导出成可回滚的文件(只读,不删任何东西)
# 用法:bash scripts/wiki_cleanup_backup.sh [目标目录]
set -uo pipefail
cd "$(dirname "$0")/.."
OUT=${1:-backups/wiki-cleanup-$(date +%Y%m%d-%H%M%S)}
mkdir -p "$OUT"; chmod 700 "$OUT"
PDB=knowledge-governance-p1-postgres-1
SDB=WKNFull-postgres
echo "### 1. 平台库(治理)全量数据备份"
docker exec $PDB pg_dump -U governance -d governance --data-only --no-owner \
-t test_publications -t source_items -t source_item_versions -t routing_kb_bindings \
-t processing_confirmations -t acceptance_results -t job_failures -t content_objects -t jobs \
> "$OUT/platform-data.sql" 2>"$OUT/platform-dump.err" && echo " platform-data.sql $(wc -l < "$OUT/platform-data.sql") 行" || cat "$OUT/platform-dump.err"
echo "### 2. 源端本地 WeKnora 全量数据备份(知识库/文档/分片/wiki/解析跨度)"
docker exec $SDB pg_dump -U weknora -d weknora_full --data-only --no-owner \
-t knowledge_bases -t knowledges -t chunks -t wiki_pages -t wiki_folders \
-t knowledge_processing_spans -t task_dead_letters \
> "$OUT/source-weknora-data.sql" 2>"$OUT/source-dump.err" && echo " source-weknora-data.sql $(wc -l < "$OUT/source-weknora-data.sql") 行" || cat "$OUT/source-dump.err"
echo "### 3. 目标端(测试服 RDS)本次测试库的行备份"
docker run --rm --network host --env-file /tmp/target_pg.env postgres:16-alpine \
pg_dump --data-only --no-owner \
-t knowledges -t chunks -t wiki_pages -t wiki_folders \
> "$OUT/target-testrows-data.sql" 2>"$OUT/target-dump.err" && echo " target-testrows-data.sql $(wc -l < "$OUT/target-testrows-data.sql") 行(全表,含真实库行,只作参照)" || cat "$OUT/target-dump.err"
echo "### 4. 归档 bundle 卷备份"
docker run --rm --user 0:0 -v knowledge-governance-release-bundles:/b -v "$PWD/$OUT":/out alpine:3.20 \
sh -c "cd /b && tar czf /out/release-bundles.tgz ." && echo " release-bundles.tgz $(du -h "$OUT/release-bundles.tgz" | cut -f1)"
echo "### 5. 清单(人读)"
{
echo "# 清理前清单 $(date -Iseconds)"
echo; echo "## 源端知识库"
docker exec $SDB psql -U weknora -d weknora_full -At -P pager=off -c \
"with d as (select kb.id, kb.name, kb.indexing_strategy->>'wiki_enabled' w, (select count(*) from knowledges k where k.knowledge_base_id=kb.id and k.deleted_at is null) docs, (select count(*) from wiki_pages p where p.knowledge_base_id=kb.id) wp from knowledge_bases kb where kb.deleted_at is null) select id||' | '||name||' | docs='||docs||' | wiki_pages='||wp||' | wiki_on='||coalesce(w,'null') from d order by docs desc"
echo; echo "## 平台记录"
docker exec $PDB psql -U governance -d governance -At -P pager=off -c \
"select 'test_publications='||(select count(*) from test_publications)||' (1b52c140='||(select count(*) from test_publications where test_kb_id='1b52c140-2e18-4676-afc5-4c7d8d190a8c')||')'||' jobs='||(select count(*) from jobs)||' processing_confirmations='||(select count(*) from processing_confirmations)||' acceptance_results='||(select count(*) from acceptance_results)||' job_failures='||(select count(*) from job_failures)||' content_objects='||(select count(*) from content_objects)||' source_items='||(select count(*) from source_items)||' catalog_nodes='||(select count(*) from source_catalog_nodes)"
echo; echo "## 目标端知识库"
docker run --rm --network host --env-file /tmp/target_pg.env postgres:16-alpine psql -At -P pager=off -c \
"select kb.id||' | '||kb.name||' | docs='||(select count(*) from knowledges k where k.knowledge_base_id=kb.id and k.deleted_at is null)||' | wiki_pages='||(select count(*) from wiki_pages p where p.knowledge_base_id=kb.id) from knowledge_bases kb where kb.deleted_at is null order by 3 desc"
echo; echo "## 归档 bundle 卷"
docker run --rm --user 0:0 -v knowledge-governance-release-bundles:/b alpine:3.20 sh -c 'for d in /b/*/; do echo "$(basename $d) files=$(ls $d | wc -l)"; done'
} > "$OUT/INVENTORY.md" 2>&1
echo " INVENTORY.md $(wc -l < "$OUT/INVENTORY.md") 行"
cat > "$OUT/ROLLBACK.md" <<'EOF'
# 回滚说明(删了之后想恢复)
1. 平台记录:`docker exec -i knowledge-governance-p1-postgres-1 psql -U governance -d governance < platform-data.sql`
(先确认目标表为空,否则会主键冲突;这是 INSERT 回放,不是幂等 upsert)
2. 源端 WeKnora 知识库/文档/wiki:`docker exec -i WKNFull-postgres psql -U weknora -d weknora_full < source-weknora-data.sql`
注意:这只是把行写回 DB,**不会**重建解析产物/检索索引;恢复后需要对该库重跑解析。
更干净的路子是从钉钉来源目录重新扫描导入(source_items 191 项仍在)。
3. 归档 bundle:`docker run --rm -u 0:0 -v knowledge-governance-release-bundles:/b -v <本目录>:/in alpine:3.20 sh -c 'cd /b && tar xzf /in/release-bundles.tgz'`
EOF
echo " ROLLBACK.md 已写"
echo
echo "备份目录:$OUT"
ls -la "$OUT"
+4 -1
View File
@@ -1 +1,4 @@
# Copy to secrets/alist_api_token, replace this line with the administrator-provided AList read-only API token, then chmod 600 the resulting file.
# Copy to secrets/alist_api_token, replace this line with the administrator-provided
# AList gateway read-only credential (newer gateways issue it in the form gw_... and
# expect it in the X-Gateway-Token header; the legacy x-api-token header is rejected
# with 401), then chmod 600 the resulting file.