Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ad02e8bb9c | |||
| edd07a65d6 | |||
| 7c120b3adc | |||
| ea159ca2fb | |||
| 034c0b3a1c | |||
| 5defe238e0 |
@@ -43,6 +43,12 @@ PRODUCTION_RELEASE_REQUESTS_ENABLED=false
|
||||
PRODUCTION_WEKNORA_BASE_URL=
|
||||
PRODUCTION_WEKNORA_API_KEY_FILE=
|
||||
PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP=false
|
||||
# 连接(地址 + 写-only 密钥)密文存治理库;主密钥是独立的 0600 32 字节文件。
|
||||
WEKNORA_BINDING_MASTER_KEY_FILE=
|
||||
# 同网段 http(生产 18093 反代)放行:只接受 RFC1918/回环/链路本地,公网 http 仍拒绝。
|
||||
WEKNORA_ALLOW_PRIVATE_HTTP=false
|
||||
# 可信 TLS 反代之后录入密钥(X-Forwarded-Proto=https + 同源)。
|
||||
WEKNORA_BINDING_TRUST_TLS_PROXY=false
|
||||
PP_SCAN_ENABLED=false
|
||||
PP_SCAN_HOST_DATA_ROOT=/home/zheromiku/dobot-portal-sync/data
|
||||
PP_SCAN_CONTAINER_DATA_ROOT=/home/zheromiku/dobot-portal-sync/data
|
||||
|
||||
@@ -33,11 +33,22 @@
|
||||
worker 完全相同的 `LIBRARY_*` 配置与维护协调器,绝不触碰封存源;任一步失败都
|
||||
保持维护并写报告,不伪造通过。
|
||||
- `internal/librarystack`:目标/预备份/维护栈的唯一装配点(同步、恢复、基线、演练共用)。
|
||||
- **测试/生产 WeKnora 连接可在页面上配置**(`GET /api/v1/weknora-connections`、
|
||||
`POST /api/v1/weknora-connections/{environment}/test`、
|
||||
`PUT /api/v1/weknora-connections/{environment}`):API Key 以 AES-GCM 密文存治理库
|
||||
(主密钥为独立 0600 文件),页面与接口只回显尾 4 位;「测试连接」真实请求该实例的
|
||||
知识库列表并区分地址不可达 / 密钥无效(401) / 无权限(403) / 成功(库数量),换地址或
|
||||
换密钥必须测通才能保存。密钥存储抽成 `internal/sealedsecret`,QA-AI 侧改为复用同一实现。
|
||||
|
||||
### Changed
|
||||
|
||||
- 基线作业的"已有生效版本"判定改为三种状态(own / foreign / none),并在
|
||||
阶段里写明是否走"只收尾"续作。
|
||||
- WeKnora 的可达性判定收敛为一份(`weknora.HTTPAllowed`):客户端与监控探活共用,
|
||||
不再出现「卡片可达、列库失败」。新增 `WEKNORA_ALLOW_PRIVATE_HTTP` 只放行
|
||||
RFC1918/回环/链路本地 http,公网 http 仍拒绝;生产 `18093` 同网段反代因此可用。
|
||||
- 「生产知识库绑定」只保留下拉选择,删除手输 ID 的降级入口;保存后回报
|
||||
「已冻结目标 / 仍可改指向」条数,已冻结的执行目标不被改写。
|
||||
|
||||
## 开发快照(首次发布入口:同步授权、目标权威基线与维护能力)
|
||||
|
||||
|
||||
@@ -71,15 +71,16 @@ IT、行政等是待扩展的业务资料范围,不是已经完成的新连接
|
||||
|
||||
## 访问与部署
|
||||
|
||||
### 已部署环境
|
||||
### 运行架构与访问入口
|
||||
|
||||
Compose 中网页入口固定映射为 **宿主机 `8088` → Web 容器 `8080`**。网页和 `/api/` 使用同源入口。
|
||||
| 组件 | 职责 | 接手时从这里看 |
|
||||
|---|---|---|
|
||||
| React/Vite + Web Nginx | 提供静态页面,同源转发 `/api/` 和 `/health` | [前端开发代理](web/vite.config.ts)、[Compose 的 Nginx 配置](deploy/nginx.conf) |
|
||||
| Go API + 治理 PostgreSQL | 认证、角色、来源、绑定、审核、任务与发布记录。云测试使用独立逻辑库 `knowledge_governance_test` | [API](cmd/api)、[接口契约](api/openapi.yaml) |
|
||||
| 普通 worker + 测试 WeKnora | worker 取件、上传、轮询并记录验收结果。WeKnora 负责解析、切片和索引 | [worker](cmd/worker)、[来源接入](docs/OPERATIONS_RUNBOOK.md#接入外部平台) |
|
||||
| 发布 worker + 生产 WeKnora | 按已锁定的批次和目标执行受控发布与读回。问答端通过下游服务使用已发布知识 | [发布 worker](cmd/release-worker)、[操作手册](docs/OPERATIONS_RUNBOOK.md) |
|
||||
|
||||
- 本机访问:`http://127.0.0.1:8088/`
|
||||
- 局域网访问:`http://<服务器内网IP>:8088/`
|
||||
- 健康入口:`http://127.0.0.1:8088/health`
|
||||
|
||||
可以由 NAS 反向代理提供 HTTPS 域名入口,目标为平台服务器的 `8088`,保持路径原样转发。公网开放前必须验证鉴权、访问控制及转发头配置,不直接暴露数据库或内部管理接口。
|
||||
从自己的电脑做系统设置验收,使用 SSH 转发后的 `http://127.0.0.1:28189/`。云测试主控制台使用受限内网入口 `http://100.85.111.121:8088/`。仓内 Compose 则将宿主 `8088` 映射到 Web 容器 `8080`。选择入口时先确认要看的部署。隧道命令、生产 `090` 服务地址和 HTTPS 反代示例见[运维手册](docs/OPERATIONS_RUNBOOK.md#入口与反向代理)。
|
||||
|
||||
**平台入口展示当前部署版本,不是每次 CI 的构建预览。** CI 静态预览由独立链路提供;CI 通过不会自动更新运行服务。
|
||||
|
||||
@@ -94,18 +95,12 @@ Compose 中网页入口固定映射为 **宿主机 `8088` → Web 容器 `8080`*
|
||||
- 来源目录、spool 与运行状态挂载;按部署主机调整路径和 UID/GID。
|
||||
- 测试 WeKnora 连接与对内/对外路由;生产发布按独立配置和授权接线。
|
||||
|
||||
准备完成后,由获授权的操作者执行:
|
||||
|
||||
```bash
|
||||
docker compose up --build -d
|
||||
```
|
||||
|
||||
主机需要提权时,由操作者按本机权限规则执行,不在自动化中默认使用 sudo。停止服务可用 `docker compose down`;**不要执行 `docker compose down -v`,除非明确批准删除数据库卷。**
|
||||
仓内 Compose 含 `postgres` 演练服务。云测试和生产使用阿里云 RDS,部署前先核对目标逻辑库、当前迁移版本、待运行的迁移及所用的环境配置。API 和 worker 启动可执行迁移。已有环境先用 `docker compose config --services` 检查服务清单,按该环境的发布手册切换已验证的产物。前端开发可直接使用下面的 Vite 流程。
|
||||
|
||||
## 配置与安全边界
|
||||
|
||||
- 网页服务地址用于控制台探活与配置核验;真正的发布目标由受限发布运行时配置决定,两者不能混为一谈。
|
||||
- 密钥通过文件或 secret 引用提供,不写入 Git、网页响应或日志。示例配置不是真实凭据。
|
||||
- 测试/生产 WeKnora 的地址与 API Key 可在「系统设置 → 测试/生产知识库绑定」里录入:密钥以 AES-GCM 密文存治理库(主密钥是独立 0600 文件),页面与接口只回显尾 4 位,日志、审计与响应都不出现明文。部署期仍可用文件引用作为兜底;示例配置不是真实凭据。
|
||||
- 生产请求与 Release Worker 默认关闭;启用状态必须现场确认。测试处理成功、自动审核或批次锁定都不能绕过生产门禁。
|
||||
- 来源扫描与取件不应改写原始业务资料;目标知识库的发布、移除和回滚属于独立受控动作。
|
||||
- 发布与恢复前必须按目标环境准备备份,并核验可恢复性。
|
||||
@@ -114,7 +109,17 @@ docker compose up --build -d
|
||||
|
||||
## 开发与 CI
|
||||
|
||||
基础环境:Go 1.26、Node.js 22、npm,以及检查脚本所需的 Python 3 和工具链。依赖以仓库锁文件和实际 CI 配置为准。
|
||||
开发环境使用 Go 1.26、Node.js 22、npm 和 Python 3。前端可独立开发。先准备一个获准用于开发的测试治理 API,在开发机的 `localhost:8088` 上确认 `/health` 可达。Vite 按仓内配置将 `/api` 和 `/health` 转给该地址,浏览器中的操作会作用于所连的环境。
|
||||
|
||||
```bash
|
||||
# 仓库根目录,检查测试后端及前端依赖,再启动开发页面
|
||||
curl -fsS --max-time 5 http://localhost:8088/health
|
||||
cd web
|
||||
npm ci
|
||||
npm run dev -- --host 127.0.0.1
|
||||
```
|
||||
|
||||
打开 Vite 打印的地址。修改组件后运行 `npm test` 和 `npm run build`。后端改动在仓库根目录用 `go test ./internal/httpapi` 这类按包测试缩小范围,再运行下方完整门禁。涉及持久化的联调使用独立测试 RDS 逻辑库,启动 API 或 worker 前核实库名和迁移版本。真实页面的通过条件见[运维手册](docs/OPERATIONS_RUNBOOK.md#开发测试和现场验收)。
|
||||
|
||||
```bash
|
||||
make verify # 完整检查入口:本地与 CI 使用同一套规则
|
||||
|
||||
@@ -3387,6 +3387,117 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/weknora-connections": {
|
||||
"get": {
|
||||
"tags": [
|
||||
"Settings"
|
||||
],
|
||||
"operationId": "listWeKnoraConnections",
|
||||
"summary": "读取测试/生产 WeKnora 连接状态(只回显密钥尾4位)",
|
||||
"responses": {
|
||||
"200": {
|
||||
"$ref": "#/components/responses/Envelope"
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
},
|
||||
"500": {
|
||||
"$ref": "#/components/responses/InternalError"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/weknora-connections/{environment}/test": {
|
||||
"post": {
|
||||
"tags": [
|
||||
"Settings"
|
||||
],
|
||||
"operationId": "testWeKnoraConnection",
|
||||
"summary": "用真实列库区分地址不可达/密钥无效/无权限/成功(管理员)",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "environment",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"test",
|
||||
"production"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/WeKnoraConnectionTest"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"$ref": "#/components/responses/Envelope"
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
},
|
||||
"500": {
|
||||
"$ref": "#/components/responses/InternalError"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/weknora-connections/{environment}": {
|
||||
"put": {
|
||||
"tags": [
|
||||
"Settings"
|
||||
],
|
||||
"operationId": "saveWeKnoraConnection",
|
||||
"summary": "保存测通的连接;密钥密文入库,响应只回尾4位(管理员)",
|
||||
"parameters": [
|
||||
{
|
||||
"$ref": "#/components/parameters/IdempotencyKey"
|
||||
},
|
||||
{
|
||||
"name": "environment",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"test",
|
||||
"production"
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/WeKnoraConnectionSave"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"$ref": "#/components/responses/Envelope"
|
||||
},
|
||||
"400": {
|
||||
"$ref": "#/components/responses/BadRequest"
|
||||
},
|
||||
"500": {
|
||||
"$ref": "#/components/responses/InternalError"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/ui-settings": {
|
||||
"get": {
|
||||
"tags": [
|
||||
@@ -5676,6 +5787,57 @@
|
||||
}
|
||||
},
|
||||
"schemas": {
|
||||
"WeKnoraConnectionTest": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"endpoint": {
|
||||
"type": "string"
|
||||
},
|
||||
"api_key": {
|
||||
"type": "string",
|
||||
"writeOnly": true
|
||||
},
|
||||
"key_action": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"keep",
|
||||
"replace",
|
||||
"clear"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"WeKnoraConnectionSave": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"endpoint",
|
||||
"key_action",
|
||||
"base_revision"
|
||||
],
|
||||
"properties": {
|
||||
"endpoint": {
|
||||
"type": "string"
|
||||
},
|
||||
"api_key": {
|
||||
"type": "string",
|
||||
"writeOnly": true
|
||||
},
|
||||
"key_action": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"keep",
|
||||
"replace",
|
||||
"clear"
|
||||
]
|
||||
},
|
||||
"base_revision": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"QAReportedHistoricalRate": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
|
||||
+50
-8
@@ -26,6 +26,7 @@ import (
|
||||
"knowledge-governance/internal/qawemail"
|
||||
"knowledge-governance/internal/serviceverify"
|
||||
"knowledge-governance/internal/version"
|
||||
"knowledge-governance/internal/weknoraconn"
|
||||
)
|
||||
|
||||
// dingtalkSourceLister 是「系统设置 → 数据源配置」的数据源发现实现:只读列举钉钉知识库与
|
||||
@@ -129,17 +130,26 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
}
|
||||
return settings.TestWeKnoraBaseURL
|
||||
}
|
||||
monitorProvider, err := monitoring.NewRealProvider(monitoring.RealProviderConfig{DB: pool, WeKnoraBaseURL: config.TestWeKnoraBaseURL, TestWeKnoraResolver: testEndpointResolver, ProductionWeKnoraBaseURL: config.ProductionWeKnoraBaseURL, ProductionWeKnoraResolver: productionEndpointResolver, MinerUBaseURL: config.MinerUBaseURL, WeKnoraReadiness: weknoraReadiness, WorkerStaleAfter: config.WorkerLivenessStaleAfter})
|
||||
// 「连接」是页面可配的一等状态:地址 + 写-only 密钥密文入库,其他功能只从
|
||||
// 这里解析,不再各自读文件凭据,避免出现两个来源。
|
||||
connectionService := weknoraconn.New(postgresstore.NewStore(pool), weknoraconn.Config{
|
||||
MasterKeyFile: config.WeKnoraBindingMasterKeyFile,
|
||||
AllowPrivateHTTP: config.WeKnoraAllowPrivateHTTP,
|
||||
})
|
||||
monitorProvider, err := monitoring.NewRealProvider(monitoring.RealProviderConfig{DB: pool, WeKnoraBaseURL: config.TestWeKnoraBaseURL, TestWeKnoraResolver: testEndpointResolver, ProductionWeKnoraBaseURL: config.ProductionWeKnoraBaseURL, ProductionWeKnoraResolver: productionEndpointResolver, MinerUBaseURL: config.MinerUBaseURL, WeKnoraReadiness: weknoraReadiness, WorkerStaleAfter: config.WorkerLivenessStaleAfter, AllowPrivateHTTP: config.WeKnoraAllowPrivateHTTP})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
serviceTargetVerifier := serviceverify.New(serviceverify.Config{
|
||||
TestAPIKeyFile: config.TestWeKnoraAPIKeyFile,
|
||||
ProductionAPIKeyFile: config.ProductionWeKnoraAPIKeyFile,
|
||||
TestBaseURLFallback: config.TestWeKnoraBaseURL,
|
||||
ProductionBaseURLFallback: config.ProductionWeKnoraBaseURL,
|
||||
TestAllowDockerHTTP: config.TestWeKnoraAllowDockerHTTP,
|
||||
ProductionAllowDockerHTTP: config.ProductionWeKnoraAllowDockerHTTP,
|
||||
TestAPIKeyFile: config.TestWeKnoraAPIKeyFile,
|
||||
ProductionAPIKeyFile: config.ProductionWeKnoraAPIKeyFile,
|
||||
TestBaseURLFallback: config.TestWeKnoraBaseURL,
|
||||
ProductionBaseURLFallback: config.ProductionWeKnoraBaseURL,
|
||||
TestAllowDockerHTTP: config.TestWeKnoraAllowDockerHTTP,
|
||||
ProductionAllowDockerHTTP: config.ProductionWeKnoraAllowDockerHTTP,
|
||||
ProductionAllowPrivateHTTP: config.WeKnoraAllowPrivateHTTP,
|
||||
TestAllowPrivateHTTP: config.WeKnoraAllowPrivateHTTP,
|
||||
ConnectionResolver: connectionService,
|
||||
})
|
||||
// 「版本记录」:只读查询面在 API 侧接线;真正的抓取在 worker 侧(未配置来源即未接线)。
|
||||
versionHistoryStore, versionHistoryRepo := httpapi.VersionHistoryStore(nil), ""
|
||||
@@ -185,10 +195,14 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
InternalRoutingKnowledgeBaseID: config.InternalRoutingKnowledgeBaseID,
|
||||
ExternalRoutingKnowledgeBaseID: config.ExternalRoutingKnowledgeBaseID,
|
||||
RoutingBindings: postgresstore.NewStore(pool),
|
||||
KnowledgeBases: knowledgeBaseLister,
|
||||
KnowledgeBases: connectionAwareBases{service: connectionService, fallback: knowledgeBaseLister, allowPrivate: config.WeKnoraAllowPrivateHTTP},
|
||||
ProductionBindings: postgresstore.NewStore(pool),
|
||||
ProductionKnowledgeBases: serviceTargetVerifier,
|
||||
ProductionWeKnoraBaseURL: config.ProductionWeKnoraBaseURL,
|
||||
WeKnoraConnections: connectionService,
|
||||
WeKnoraAllowPrivateHTTP: config.WeKnoraAllowPrivateHTTP,
|
||||
WeKnoraTrustTLSProxy: config.WeKnoraBindingTrustTLSProxy,
|
||||
WeKnoraConnectionResolver: connectionService,
|
||||
ServiceTargetVerifier: serviceTargetVerifier,
|
||||
ProductionReleaseRequestsEnabled: config.ProductionReleaseRequestsEnabled,
|
||||
TestAutoReviewPolicy: config.TestAutoReviewPolicy,
|
||||
@@ -255,3 +269,31 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
return server.Shutdown(shutdownCtx)
|
||||
}
|
||||
}
|
||||
|
||||
// connectionAwareBases prefers the admin-managed connection over the bootstrap
|
||||
// env credential, so 「测试知识库绑定」 lists the very instance the connection card
|
||||
// probes — one source, not two.
|
||||
type connectionAwareBases struct {
|
||||
service *weknoraconn.Service
|
||||
fallback httpapi.KnowledgeBaseLister
|
||||
allowPrivate bool
|
||||
}
|
||||
|
||||
func (lister connectionAwareBases) ListKnowledgeBases(ctx context.Context) ([]weknora.KnowledgeBase, error) {
|
||||
if endpoint, apiKey, ok := lister.service.Resolve(ctx, weknoraconn.EnvironmentTest); ok {
|
||||
client, err := weknora.NewClient(weknora.ClientConfig{
|
||||
BaseURL: endpoint,
|
||||
APIKey: apiKey,
|
||||
Timeout: 10 * time.Second,
|
||||
AllowPrivateHTTP: lister.allowPrivate,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return client.ListKnowledgeBases(ctx)
|
||||
}
|
||||
if lister.fallback == nil {
|
||||
return nil, errors.New("测试 WeKnora 未配置:请先在系统设置里录入连接")
|
||||
}
|
||||
return lister.fallback.ListKnowledgeBases(ctx)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
// command libraryresidue 处理"整库同步写到目标范围外"留下的残留行:
|
||||
//
|
||||
// 读指定快照(来源范围)→ 在目标里找同内容、仍带来源身份的行
|
||||
// → 先备份成 0600 文件并记录摘要 → 精确按归档身份删除 → 复核来源身份已空、
|
||||
// 目标内容与预备份一致、整库指纹里没有别的租户
|
||||
//
|
||||
// 它不做范围广删,也不改任何别的东西:判据是"归档里确实有这些行、内容逐字段
|
||||
// 一致",任何一行对不上就整体拒绝。默认是干跑(只报告、不删)。
|
||||
//
|
||||
// 真实事故:跨范围同步把 4 文件/38 分块/21 资源/4 绑定写成来源租户的行,
|
||||
// 回滚只按目标范围回放,于是"已恢复"却留下这些行。这个工具的用途就是在
|
||||
// 修复后的代码上线前,用可审计的方式清掉它们。
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"knowledge-governance/internal/appconfig"
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
"knowledge-governance/internal/librarystack"
|
||||
v071 "knowledge-governance/internal/releaseadapter/v071"
|
||||
)
|
||||
|
||||
func main() {
|
||||
scope := flag.String("scope", "internal", "internal|external")
|
||||
snapshot := flag.String("snapshot", "", "sealed snapshot whose source scope is the residue (required)")
|
||||
backupSnapshot := flag.String("backup-snapshot", "", "prebackup snapshot the target must be restored to (required)")
|
||||
expectSource := flag.String("expect-source", "", "expected source scope as <tenant>/<knowledge-base> (required guard against typos)")
|
||||
out := flag.String("out", "", "directory for the residue backup (required)")
|
||||
apply := flag.Bool("apply", false, "actually delete the residue rows (default: dry run)")
|
||||
reportPath := flag.String("report", "", "write the JSON report here")
|
||||
flag.Parse()
|
||||
if *snapshot == "" || *backupSnapshot == "" || *out == "" || *expectSource == "" {
|
||||
log.Fatal("snapshot, backup-snapshot, out and expect-source are required")
|
||||
}
|
||||
config, err := appconfig.Load(os.Getenv)
|
||||
if err != nil {
|
||||
log.Fatalf("load config: %v", err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
report, err := run(ctx, config, *scope, *snapshot, *backupSnapshot, *expectSource, *out, *apply)
|
||||
if *reportPath != "" && report != nil {
|
||||
raw, marshalErr := json.MarshalIndent(report, "", " ")
|
||||
if marshalErr == nil {
|
||||
raw = append(raw, '\n')
|
||||
if writeErr := os.WriteFile(*reportPath, raw, 0o600); writeErr != nil {
|
||||
log.Printf("report write failed: %v", writeErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
log.Fatalf("residue cleanup failed: %v", err)
|
||||
}
|
||||
log.Printf("residue cleanup ok: source=%s/%s tables=%v cleared=%d applied=%v verified=%v",
|
||||
reportSource(report), report.SourceKB, report.Tables, report.Cleared, report.Applied, report.Verified)
|
||||
}
|
||||
|
||||
func reportSource(report *v071.ResidueReport) string {
|
||||
return strconv.FormatUint(report.SourceTenant, 10)
|
||||
}
|
||||
|
||||
func run(ctx context.Context, config appconfig.Config, scope, snapshotID, backupSnapshotID, expectSource, out string,
|
||||
apply bool) (*v071.ResidueReport, error) {
|
||||
if scope != "internal" && scope != "external" {
|
||||
return nil, fmt.Errorf("scope must be internal or external")
|
||||
}
|
||||
archive, err := librarysnapshot.Open(config.LibraryArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("归档不可用: %w", err)
|
||||
}
|
||||
manifest, err := archive.Load(snapshotID)
|
||||
archive.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("指定快照不可读: %w", err)
|
||||
}
|
||||
source := v071.ProductionScope{TenantID: manifest.TenantID, KnowledgeBaseID: manifest.KnowledgeBaseID}
|
||||
want := strconv.FormatUint(source.TenantID, 10) + "/" + source.KnowledgeBaseID
|
||||
// 范围写错会把"别人的行"当成残留:必须由调用方显式确认。
|
||||
if strings.TrimSpace(expectSource) != want {
|
||||
return nil, fmt.Errorf("来源范围与 -expect-source 不一致:归档为 %s,参数为 %s", want, expectSource)
|
||||
}
|
||||
target, _, err := librarystack.OpenProductionStack(ctx, config, scope, backupSnapshotID, "")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("装配目标失败: %w", err)
|
||||
}
|
||||
defer target.Close()
|
||||
report, err := target.ClearForeignScope(ctx, source, snapshotID, out, apply)
|
||||
if err != nil {
|
||||
return report, err
|
||||
}
|
||||
report.Foreign, _ = target.ForeignScopes(ctx)
|
||||
if !apply {
|
||||
return report, nil
|
||||
}
|
||||
if err := target.VerifyResidueCleared(ctx, source); err != nil {
|
||||
return report, err
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: backupSnapshotID}); err != nil {
|
||||
return report, fmt.Errorf("清理后目标内容仍与预备份不一致(保持停写): %w", err)
|
||||
}
|
||||
report.Verified = true
|
||||
if len(report.Foreign) > 0 {
|
||||
raw, _ := json.Marshal(report.Foreign)
|
||||
return report, fmt.Errorf("清理后目标仍有其它租户的数据,需人工确认才能开放:%s", raw)
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
@@ -70,6 +70,12 @@ services:
|
||||
PRODUCTION_WEKNORA_BASE_URL: "${PRODUCTION_WEKNORA_BASE_URL:-}"
|
||||
PRODUCTION_WEKNORA_API_KEY_FILE: "${PRODUCTION_WEKNORA_API_KEY_FILE:-}"
|
||||
PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP: "${PRODUCTION_WEKNORA_ALLOW_DOCKER_HTTP:-false}"
|
||||
# 连接(地址 + 写-only 密钥)密文存治理库:主密钥是独立的 0600 文件。
|
||||
# 生产 18093 是同网段 http 反代,因此需要显式放行私网 http(只放行
|
||||
# RFC1918/回环/链路本地,公网 http 仍被拒绝)。
|
||||
WEKNORA_BINDING_MASTER_KEY_FILE: "${WEKNORA_BINDING_MASTER_KEY_FILE:-}"
|
||||
WEKNORA_ALLOW_PRIVATE_HTTP: "${WEKNORA_ALLOW_PRIVATE_HTTP:-false}"
|
||||
WEKNORA_BINDING_TRUST_TLS_PROXY: "${WEKNORA_BINDING_TRUST_TLS_PROXY:-false}"
|
||||
# 数据源发现(系统设置 → 数据源配置):api 用 dws 只读列举钉钉知识库与钉盘空间。
|
||||
DWS_BIN: /usr/local/bin/dws
|
||||
DWS_CONFIG_DIR: /data/dws
|
||||
|
||||
@@ -3,7 +3,86 @@
|
||||
按"实际干活顺序"排列的**现在的**操作流程。标注:**【新】**本期新增(含本次会话与 `f4e1ecc/6496ee3/9155bd3` 的 dws 取件)|**【变】**既有流程本期有改动|**【既有】**未变,仍照旧走。
|
||||
逐条验收证据请看 `docs/USER_FLOWS_ACCEPTANCE.md`;细节规则看各自专题文档。
|
||||
|
||||
统一入口 **http://127.0.0.1:8088/**(测试栈,右侧面板可直接操作)。角色:管理员 / 文档工程师 / 只读。
|
||||
管理员配置连接和发布授权,文档工程师维护资料,只读成员查看进度。先确定要操作的环境,再按下面的接线与验收步骤进入业务流程。
|
||||
|
||||
## 入口与反向代理
|
||||
|
||||
| 请求从哪里发出 | 地址或路径 | 实际目的地 |
|
||||
|---|---|---|
|
||||
| 已接入内网的运维电脑 | `http://100.85.111.121:8088/` | 云测试治理主站 |
|
||||
| 开发电脑的浏览器 | `http://127.0.0.1:28189/` | SSH 转发到测试机回环 `28189`,由候选 Web 提供静态资源 |
|
||||
| 测试候选 Nginx(host 网络) | `/api/`、`/health` | 测试机 `127.0.0.1:18099` 候选 API |
|
||||
| 同一候选 Nginx | `/api/v1/qa-ops/ai/` | 测试机 `127.0.0.1:18098` 现用治理 API,负责模型配置和连接验证 |
|
||||
| 标准 Compose Web(容器网络) | `/api/`、`/health` | Docker DNS 解析 `api:8080`。静态资源和 SPA 回退由同一 Web 提供 |
|
||||
|
||||
**在开发电脑恢复验收入口。** 前提是 SSH 别名 `weknora-test` 能登录测试机。先查远端服务,再查本机监听。本机端口空闲且远端正常时,在独立终端运行第三条命令,保留该终端。在另一终端执行最后一条并用浏览器打开同一地址。
|
||||
|
||||
```bash
|
||||
ssh weknora-test 'curl -fsS --max-time 5 http://127.0.0.1:28189/health'
|
||||
ss -ltn 'sport = :28189'
|
||||
ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -L 127.0.0.1:28189:127.0.0.1:28189 weknora-test
|
||||
curl -fsS --max-time 5 http://127.0.0.1:28189/health
|
||||
```
|
||||
|
||||
**为内网域名配置 HTTPS。** 在能访问测试主站的受信任反代机上配置证书、内部 DNS 和访问范围。以下片段放入已有 HTTPS `server`,完整保留 URI、Host 和会话 Cookie。上游应选治理 Web 入口,使 `/assets/`、`/api/`、`/health` 走同一域名。若该入口的内层代理也设置 `X-Forwarded-Proto`,联调时检查其保留受信任 TLS 终止层的协议值以及登录回调的实际 URL。
|
||||
|
||||
```nginx
|
||||
location / {
|
||||
proxy_pass http://100.85.111.121:8088;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
```
|
||||
|
||||
在反代机执行 `nginx -t` 校验待应用配置。入口启用后,通过该域名验证登录、一次 `/api/` 请求和首页实际引用的 JS/CSS。`index.html` 应重新校验,带哈希的资源可长期缓存,缺失的 JS 返回 404。候选切换前后分别记录通用上游和 AI 专用上游。现场可在测试机用以下只读命令查看,容器名以当时部署为准。仓内同源配置见 [deploy/nginx.conf](../deploy/nginx.conf)。
|
||||
|
||||
```bash
|
||||
docker exec kg-test-storage-ui-6444814 nginx -T 2>&1 | grep -E 'listen .*28189|location /api/v1/qa-ops/ai/|set \$(api|health)_upstream|proxy_pass http://127[.]0[.]0[.]1:18098'
|
||||
docker inspect -f '{{.Name}} {{index .Config.Labels "release"}}' kg-test-storage-ui-6444814 kg-accept-090-api
|
||||
```
|
||||
|
||||
**现场版本与回退。** 本次在测试机读到 Web 标签 `4b615abe534b`、API 标签 `341c3154ac91`。完整候选包的 `current` 链接位于 `/opt/knowledge-governance/acceptance/090/current`,单独切换的 Web 位于同级 `web-only/4b615abe534b/dist`,实际反代配置位于 `integration/28189.conf`。切换前记录这三处、容器标签与旧 Web 名称。静态页回退时恢复上次的 Web 资源和对应代理配置。完整候选回退按同批装配工具的 `deploy/accept-090/README.md` 执行。回退后用浏览器重新验证首页资源、模型连接和列库。数据库迁移另按备份与恢复方案处理。
|
||||
|
||||
本次已核实共享测试治理库到 `000082`。对内、对外独立启用的实现依赖 `000083`,待独立库联调或共享库迁移获准后,再验证单侧暂停、保存和刷新。生产 `070` 不在该套验收范围。
|
||||
|
||||
下文系统设置中的绑定、盘点和模型连接步骤适用于上表的测试候选版本。根据现场标签选择对应版本的代码和交付物。
|
||||
|
||||
## 接入外部平台
|
||||
|
||||
| 平台与用途 | 管理员或运维怎样接 | 接通的证据 |
|
||||
|---|---|---|
|
||||
| 钉钉知识库,取目录和文件 | 系统设置 → 数据源配置 → 钉钉授权。用 DWS 设备授权,以来源账号选择组织、空间和范围。API 与 worker 使用部署约定的 DWS 身份目录 | 登录后读回预期空间,选一个有权限的目录校验。登记来源、扫描完成和文件取件分别核验。 |
|
||||
| AList,取工业或教育资料 | 按来源登记服务根地址和允许的目录。普通 worker 读取专用令牌文件。目录接口 `POST /api/fs/list` 使用 `X-Gateway-Token` | 在来源页扫描已知根目录,对照一个子目录和一份文件。下载遵循来源签名链接,凭据和签名 URL 仅留在受控运行环境。[目录与取件](SOURCE_CATALOG_LIVE_ROUTING_ACCEPTANCE.md) |
|
||||
| PP 资料门户 | 在数据源配置填写 HTTPS 根地址和来源账号,点「测试连接」后「保存并校验」。API 和 worker 共享 `PP_AUTH_STATE_FILE` 所在目录 | 真实登录并读取一条目录记录,刷新后核对生效配置。超时或限流时查看待验证状态。 |
|
||||
| WeKnora 测试 090,处理与验收 | 系统设置 → 知识库绑定 → 测试 090。当前测试 API 地址 `http://127.0.0.1:8092`,由治理服务器访问。输入该实例的专用 Key 或使用已配置凭据 | 「验证并读取知识库」能列出预期库。核对名称和 ID,为对内、对外选择不同的目标,保存后刷新读回 |
|
||||
| WeKnora 生产 090,列库与映射 | 在同页切到生产 090,使用下节的转发入口与生产专用 Key。新连接的安全存储由 `WEKNORA_BINDING_MASTER_KEY_FILE` 提供 | 读取生产目录并核对两个目标 ID,保存后重新进入生产页核实。执行发布另按 §5–6 授权 |
|
||||
| 问答 AI 模型 | 系统设置 → AI 模型连接,填写 OpenAI 兼容基础地址(含服务要求的 `/v1`)、模型名及专用 Key。API 与普通 worker 使用同一 `QA_AI_MASTER_KEY_FILE` | 点「测试连接」等待成功,需生效时再保存并刷新检查模型名。存储密钥、权限与 DNS 配置见 [README](../README.md#问答运营模型配置) |
|
||||
|
||||
**生产 090 的两段地址。** 生产机私网 IP 为 `172.29.151.11`,其 WeKnora API 对宿主回环发布 `127.0.0.1:18092`。测试治理机上已有 SSH 本地转发,治理 API 使用 host 网络,故页面的生产服务地址填写 **`http://127.0.0.1:18092`**。请求从治理 API 发出,经测试机的回环监听到生产机的回环 API。浏览器所在电脑的 `127.0.0.1:28189` 则用于打开治理页面。迁到 bridge 网络时要从新 API 容器重新验证上游可达性。
|
||||
|
||||
在**测试治理机**上检验已有通道。`/health` 返回 200 证明转发可用,无凭据列库返回 401 表明请求已到认证层。接着在治理页面安全输入该实例的 Key 并验证真实库列表。本次核验时,生产通道探活成功,匿名列库返回 401。带生产凭据列库和保存映射仍待验收。
|
||||
|
||||
```bash
|
||||
ss -ltnp 'sport = :18092'
|
||||
curl --noproxy '*' --max-time 5 -sS -o /dev/null -w 'health %{http_code}\n' http://127.0.0.1:18092/health
|
||||
curl --noproxy '*' --max-time 5 -sS -o /dev/null -w 'catalog %{http_code}\n' http://127.0.0.1:18092/api/v1/knowledge-bases
|
||||
```
|
||||
|
||||
## 开发测试和现场验收
|
||||
|
||||
1. 在开发机运行 `make list` 查看门禁,`make verify` 做整套检查。以 `ci-report.json` 的 `commit_sha` 对照 `git rev-parse HEAD`,记录失败或跳过项。前端启动和组件测试见 [开发与 CI](../README.md#开发与-ci)。
|
||||
2. 针对后端改动,在独立测试逻辑库核对 `SELECT current_database(), max(version) FROM schema_migrations;` 与候选版本的迁移目录。云端多个候选可能共用治理 RDS。只改静态页面时单独切换 Web,保留原资源和代理配置供回滚。
|
||||
3. 在指定入口以真实账号分别验证。来源页读取一条已知记录,模型页测试到成功终态,知识库绑定在正确环境列库,保存后刷新核对。存储页按最近批次和子库目录分别与 `du -sB1 -- <目录>` 对账。记录入口、Web/API 提交、时间和结果。
|
||||
4. 候选备份页读取宿主采集的 `metadata.json`,API 只读挂载输出目录。「重新盘点」只重读文件。超过 `BACKUP_HOST_METADATA_MAX_AGE` 时,目录、占盘和预测转为未知。宿主重新采集后核对采集时间是否前进、批次是否恢复。测试现场使用按次采集、15 分钟有效期。采集脚本和挂载参数以同批测试候选交付物为准。
|
||||
|
||||
| 症状 | 第一步怎么查 | 如何判定 |
|
||||
|---|---|---|
|
||||
| 浏览器连接被拒 | 对照本机 `28189` 监听和测试机 `/health` | 远端正常、本机无监听时恢复本机 SSH 转发 |
|
||||
| 页面打开,某个 API 404 或 502 | 查浏览器请求路径、Nginx 生效上游及对应 API 版本 | 404 先查路由是否存在,502 先查上游是否可达。AI 页单独核对 `18098` |
|
||||
| 能探活,列库仍灰或返回 401 | 核对选中环境、凭据配置状态、容器运行 UID 与凭据文件可读性 | 用该环境的专用身份实际列库。检查过程只记录状态码和可读性 |
|
||||
| 历史可读,计划或配额显示未知 | 核对快照年龄、执行主机连接与配额字段 | 盘点、SSH 计划、配额分别验证。只有主机实际读回 cron 才能报告下一次执行时间 |
|
||||
|
||||
---
|
||||
|
||||
@@ -81,8 +160,9 @@ bash scripts/target_kb_snapshot.sh # 默认云内执行;PG 8 表 +
|
||||
| 分区 | 谁 | 现在要管什么 |
|
||||
|---|---|---|
|
||||
| 服务配置 | 管理员 | 测试地址 / 生产地址 / 生产 ID / 知识库 ID(保存时真探活,无效才拒) |
|
||||
| 测试知识库绑定 | 管理员 | 标注往哪个测试库路由(对内 ≠ 对外) |
|
||||
| 生产知识库绑定 | 管理员 | 源库 → 目标库映射(发布写哪个生产库) |
|
||||
| 测试知识库绑定 | 管理员 | 顶部连接卡:测试地址 + API Key(测试连接 → 保存;只回显尾4位);下方标注往哪个测试库路由(对内 ≠ 对外) |
|
||||
| 生产连接 | 管理员 | 生产地址填同网段反代 `http://172.29.151.11:18093`(需要 `WEKNORA_ALLOW_PRIVATE_HTTP=true`),密钥录入后只回显尾4位;停止 `ssh -L 18092` 隧道后仍可用 |
|
||||
| 生产知识库绑定 | 管理员 | 源库 → 目标库映射(发布写哪个生产库);选项来自生产实例真实库,禁止手填 ID;保存后回报「已冻结 / 仍可改指向」条数 |
|
||||
| **存储与备份** **【新】** | 管理员 | **定时时间 + 储存天数**(+每月归档、启用开关)→ `保存并下发到主机`;另有 `重新下发当前设置`、`立即备份一次`;页面显示主机上**实际生效**的计划、cron 条目、最近一次结果与"已一致/待下发" |
|
||||
| 刷新策略 | 文档工程师 | 文档/资源刷新间隔、停滞判定、finalizing 时长(版本化 + 原子切换) |
|
||||
| 成员与角色 | 管理员 | 成员与 viewer/engineer/admin |
|
||||
|
||||
@@ -260,6 +260,12 @@
|
||||
- 规则:只配四项(测试地址、生产地址、生产 ID、知识库 ID);**保存时真探测**,无效才拒;凭据只写不读
|
||||
- 状态:✅(见 `P41/P42` 与 `ui_settings_*` 测试)
|
||||
|
||||
### D1b 连接(地址 + API Key,测试 / 生产各一张)
|
||||
- 入口:**系统设置 → 测试知识库绑定 / 生产知识库绑定** 分区顶部的连接卡(管理员)
|
||||
- 步骤:填地址 → 粘贴 API Key(密码框)→ **测试连接** → 成功后才可 **保存连接**;换地址或换密钥必须重新测通
|
||||
- 期望:密钥密文入库、页面只显示「已配置 · 尾4位」;四态文案分别是 地址不可达 / 密钥无效 / 密钥无权限读取知识库列表 / 可达(读到 N 个知识库);保存不触发任何同步或发布
|
||||
- 状态:✅(`internal/weknoraconn` 单测 + `internal/httpapi/weknora_connections_test.go`;迁移 000083)
|
||||
|
||||
### D2 测试知识库绑定(对内 / 对外)
|
||||
- 入口:**系统设置 → 测试知识库绑定** → 选库 → 保存绑定
|
||||
- 期望:保存立即生效并重排"还没上传"的待入库项;对内 ≠ 对外
|
||||
|
||||
@@ -31,28 +31,33 @@ type Config struct {
|
||||
ProductionWeKnoraBaseURL string
|
||||
ProductionWeKnoraAPIKeyFile string
|
||||
ProductionWeKnoraAllowDockerHTTP bool
|
||||
PPScanEnabled bool
|
||||
PPScanAllowedRoot string
|
||||
PPScanLedgerPath string
|
||||
PPScanManifestPath string
|
||||
TestWeKnoraEnabled bool
|
||||
TestWeKnoraAllowDockerHTTP bool
|
||||
TestWeKnoraBaseURL string
|
||||
MinerUBaseURL string
|
||||
TestWeKnoraKBID string
|
||||
InternalRoutingKnowledgeBaseID string
|
||||
ExternalRoutingKnowledgeBaseID string
|
||||
TestWeKnoraAPIKeyFile string
|
||||
GovernanceSpoolRoot string
|
||||
IngestionSpoolRoot string
|
||||
ArtifactFetchMaxBytes int64
|
||||
WorkerLivenessStaleAfter time.Duration
|
||||
TestWeKnoraHTTPTimeout time.Duration
|
||||
TestWeKnoraPollInitial time.Duration
|
||||
TestWeKnoraPollMax time.Duration
|
||||
TestWeKnoraStalledAfter time.Duration
|
||||
TestWeKnoraMaxUploadBytes int64
|
||||
TestAcceptancePolicyVersion string
|
||||
// 连接(地址 + 写-only 密钥)密文存治理库;主密钥是独立的 0600 文件。
|
||||
WeKnoraBindingMasterKeyFile string
|
||||
// 允许同网段 http(生产 18093 反代)与可信 TLS 反代后录入密钥。
|
||||
WeKnoraAllowPrivateHTTP bool
|
||||
WeKnoraBindingTrustTLSProxy bool
|
||||
PPScanEnabled bool
|
||||
PPScanAllowedRoot string
|
||||
PPScanLedgerPath string
|
||||
PPScanManifestPath string
|
||||
TestWeKnoraEnabled bool
|
||||
TestWeKnoraAllowDockerHTTP bool
|
||||
TestWeKnoraBaseURL string
|
||||
MinerUBaseURL string
|
||||
TestWeKnoraKBID string
|
||||
InternalRoutingKnowledgeBaseID string
|
||||
ExternalRoutingKnowledgeBaseID string
|
||||
TestWeKnoraAPIKeyFile string
|
||||
GovernanceSpoolRoot string
|
||||
IngestionSpoolRoot string
|
||||
ArtifactFetchMaxBytes int64
|
||||
WorkerLivenessStaleAfter time.Duration
|
||||
TestWeKnoraHTTPTimeout time.Duration
|
||||
TestWeKnoraPollInitial time.Duration
|
||||
TestWeKnoraPollMax time.Duration
|
||||
TestWeKnoraStalledAfter time.Duration
|
||||
TestWeKnoraMaxUploadBytes int64
|
||||
TestAcceptancePolicyVersion string
|
||||
// TestAutoReviewPolicy:验收通过后是否自动批准(off|internal|all;默认 internal)。
|
||||
TestAutoReviewPolicy string
|
||||
// 目标库所在主机(整库逻辑备份真正执行的地方)。凭据是挂载进来的 0600 ssh
|
||||
@@ -168,6 +173,7 @@ func Load(getenv func(string) string) (Config, error) {
|
||||
ProductionReleaseAuthTokenFile: strings.TrimSpace(getenv("PRODUCTION_RELEASE_AUTH_TOKEN_FILE")),
|
||||
ProductionWeKnoraBaseURL: strings.TrimSpace(getenv("PRODUCTION_WEKNORA_BASE_URL")),
|
||||
ProductionWeKnoraAPIKeyFile: strings.TrimSpace(getenv("PRODUCTION_WEKNORA_API_KEY_FILE")),
|
||||
WeKnoraBindingMasterKeyFile: strings.TrimSpace(getenv("WEKNORA_BINDING_MASTER_KEY_FILE")),
|
||||
PPScanAllowedRoot: strings.TrimSpace(getenv("PP_SCAN_ALLOWED_ROOT")),
|
||||
PPScanLedgerPath: strings.TrimSpace(getenv("PP_SCAN_LEDGER_PATH")),
|
||||
PPScanManifestPath: strings.TrimSpace(getenv("PP_SCAN_MANIFEST_PATH")),
|
||||
@@ -288,6 +294,21 @@ func Load(getenv func(string) string) (Config, error) {
|
||||
return Config{}, fmt.Errorf("PRODUCTION_WEKNORA_BASE_URL: %w", err)
|
||||
}
|
||||
}
|
||||
if raw := getenv("WEKNORA_ALLOW_PRIVATE_HTTP"); raw != "" {
|
||||
config.WeKnoraAllowPrivateHTTP, err = strconv.ParseBool(raw)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("WEKNORA_ALLOW_PRIVATE_HTTP: %w", err)
|
||||
}
|
||||
}
|
||||
if raw := getenv("WEKNORA_BINDING_TRUST_TLS_PROXY"); raw != "" {
|
||||
config.WeKnoraBindingTrustTLSProxy, err = strconv.ParseBool(raw)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("WEKNORA_BINDING_TRUST_TLS_PROXY: %w", err)
|
||||
}
|
||||
}
|
||||
if config.WeKnoraBindingMasterKeyFile != "" && !filepath.IsAbs(config.WeKnoraBindingMasterKeyFile) {
|
||||
return Config{}, errors.New("WEKNORA_BINDING_MASTER_KEY_FILE must be an absolute path")
|
||||
}
|
||||
if config.ProductionWeKnoraAPIKeyFile != "" {
|
||||
if config.ProductionWeKnoraBaseURL == "" {
|
||||
return Config{}, errors.New("PRODUCTION_WEKNORA_API_KEY_FILE requires PRODUCTION_WEKNORA_BASE_URL")
|
||||
|
||||
@@ -138,6 +138,12 @@ func classifyRole(method, path string) string {
|
||||
if path == "/api/v1/routing-bindings" && method != http.MethodGet {
|
||||
return roleAdmin
|
||||
}
|
||||
// Recording an endpoint or an API key changes which instance the whole
|
||||
// platform talks to, and the key is a credential: read is viewer, every
|
||||
// write (including 测试连接, which spends the key) is admin.
|
||||
if strings.HasPrefix(path, "/api/v1/weknora-connections") && method != http.MethodGet {
|
||||
return roleAdmin
|
||||
}
|
||||
// Choosing the production base a release writes into is a production
|
||||
// authority change, exactly like the annotation binding above.
|
||||
if path == "/api/v1/production-kb-bindings" && method != http.MethodGet {
|
||||
|
||||
@@ -411,8 +411,16 @@ func TestClassifyRoleMatrix(t *testing.T) {
|
||||
http.MethodPost + " /api/v1/users": roleAdmin,
|
||||
http.MethodPost + " /api/v1/source-connections/c-1/settings/revisions": roleAdmin,
|
||||
// Choosing a release's production base is a production authority change.
|
||||
http.MethodPut + " /api/v1/production-kb-bindings": roleAdmin,
|
||||
http.MethodGet + " /api/v1/production-kb-bindings": roleViewer,
|
||||
http.MethodPut + " /api/v1/production-kb-bindings": roleAdmin,
|
||||
http.MethodGet + " /api/v1/production-kb-bindings": roleViewer,
|
||||
// An endpoint or an API key decides which instance the platform talks
|
||||
// to; 测试连接 also spends the credential, so every write is admin.
|
||||
http.MethodGet + " /api/v1/weknora-connections": roleViewer,
|
||||
http.MethodPost + " /api/v1/weknora-connections/test/test": roleAdmin,
|
||||
http.MethodPut + " /api/v1/weknora-connections/production": roleAdmin,
|
||||
// The test-side annotation binding is the same authority as production.
|
||||
http.MethodGet + " /api/v1/routing-bindings": roleViewer,
|
||||
http.MethodPut + " /api/v1/routing-bindings": roleAdmin,
|
||||
http.MethodPost + " /api/v1/review-candidates/tp-1/decision": roleEngineer,
|
||||
http.MethodPost + " /api/v1/release-batches/rb-1/seal": roleEngineer,
|
||||
http.MethodPost + " /api/v1/test-publications/start": roleEngineer,
|
||||
|
||||
@@ -16,6 +16,10 @@ import (
|
||||
type ProductionBindingStore interface {
|
||||
LoadProductionKBBindings(context.Context) (string, string, string, error)
|
||||
SaveProductionKBBindings(ctx context.Context, internalID, externalID, updatedBy string, now time.Time) error
|
||||
// ProductionExecutionCounts reports frozen vs still re-targetable release
|
||||
// items. A binding change never rewrites a target that was already frozen
|
||||
// when an execution started; the count is how the page says so.
|
||||
ProductionExecutionCounts(ctx context.Context) (frozen int, retargetable int, err error)
|
||||
}
|
||||
|
||||
// ProductionKnowledgeBaseLister enumerates the knowledge bases of the production
|
||||
@@ -33,11 +37,26 @@ type productionBindingView struct {
|
||||
ProductionBaseURL string `json:"production_base_url"`
|
||||
ProductionConnected bool `json:"production_connected"`
|
||||
ProductionMessage string `json:"production_message"`
|
||||
// Frozen / Retargetable describe what a binding change can still affect.
|
||||
Frozen int `json:"frozen"`
|
||||
Retargetable int `json:"retargetable"`
|
||||
}
|
||||
|
||||
// WeKnoraConnectionResolver reads the admin-managed connection for one
|
||||
// environment. When it answers, that endpoint is the only truth.
|
||||
type WeKnoraConnectionResolver interface {
|
||||
Resolve(ctx context.Context, environment string) (endpoint string, apiKey string, ok bool)
|
||||
}
|
||||
|
||||
// productionTargetBaseURL is the production endpoint the picker reads from: the
|
||||
// operator-managed 「服务配置 · 生产地址」 first, the deployment default second.
|
||||
// connection card first, the operator-managed 「服务配置 · 生产地址」 second, the
|
||||
// deployment default last.
|
||||
func (config Config) productionTargetBaseURL(ctx context.Context) string {
|
||||
if config.WeKnoraConnectionResolver != nil {
|
||||
if endpoint, _, ok := config.WeKnoraConnectionResolver.Resolve(ctx, "production"); ok {
|
||||
return endpoint
|
||||
}
|
||||
}
|
||||
if settings, err := config.Store.GetActiveUISettings(ctx); err == nil {
|
||||
if resolved := strings.TrimSpace(settings.ProductionWeKnoraBaseURL); resolved != "" {
|
||||
return resolved
|
||||
@@ -55,7 +74,7 @@ func (config Config) productionKnowledgeBases(ctx context.Context) ([]weknora.Kn
|
||||
}
|
||||
baseURL := config.productionTargetBaseURL(ctx)
|
||||
if baseURL == "" {
|
||||
return nil, "未配置生产地址(系统设置 → 服务配置)"
|
||||
return nil, "未配置生产地址(系统设置 → 生产知识库绑定 → 连接卡)"
|
||||
}
|
||||
bases, err := config.ProductionKnowledgeBases.ListProductionKnowledgeBases(ctx, baseURL)
|
||||
if err != nil {
|
||||
@@ -74,11 +93,16 @@ func (config Config) productionBindingPayload(ctx context.Context) (productionBi
|
||||
for _, base := range bases {
|
||||
names[base.ID] = base.Name
|
||||
}
|
||||
frozen, retargetable := 0, 0
|
||||
if frozenItems, retargetableItems, countsErr := config.ProductionBindings.ProductionExecutionCounts(ctx); countsErr == nil {
|
||||
frozen, retargetable = frozenItems, retargetableItems
|
||||
}
|
||||
return productionBindingView{
|
||||
InternalKBID: storedInternal, ExternalKBID: storedExternal,
|
||||
InternalKBName: names[storedInternal], ExternalKBName: names[storedExternal],
|
||||
UpdatedBy: updatedBy, ProductionBaseURL: config.productionTargetBaseURL(ctx),
|
||||
ProductionConnected: message == "", ProductionMessage: message,
|
||||
Frozen: frozen, Retargetable: retargetable,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@ type productionBindingStoreFake struct {
|
||||
internal, external, saver string
|
||||
savedInternal, savedExternal string
|
||||
saveCallCount int
|
||||
frozen, retargetable int
|
||||
}
|
||||
|
||||
func (fake *productionBindingStoreFake) LoadProductionKBBindings(context.Context) (string, string, string, error) {
|
||||
@@ -32,6 +33,10 @@ func (fake *productionBindingStoreFake) SaveProductionKBBindings(_ context.Conte
|
||||
return nil
|
||||
}
|
||||
|
||||
func (fake *productionBindingStoreFake) ProductionExecutionCounts(context.Context) (int, int, error) {
|
||||
return fake.frozen, fake.retargetable, nil
|
||||
}
|
||||
|
||||
type productionListerFake struct {
|
||||
bases []weknora.KnowledgeBase
|
||||
err error
|
||||
|
||||
@@ -31,12 +31,15 @@ type routingBindingView struct {
|
||||
ExternalKBID string `json:"external_kb_id"`
|
||||
InternalEnvFallback bool `json:"internal_env_fallback"`
|
||||
ExternalEnvFallback bool `json:"external_env_fallback"`
|
||||
InternalBoundByUI bool `json:"internal_bound_by_ui"`
|
||||
ExternalBoundByUI bool `json:"external_bound_by_ui"`
|
||||
InternalKBName string `json:"internal_kb_name"`
|
||||
ExternalKBName string `json:"external_kb_name"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
WeKnoraConnected bool `json:"weknora_connected"`
|
||||
// 取消绑定时必须说清「回退到了哪个库」,否则「留空」等于没说。
|
||||
InternalFallbackKBID string `json:"internal_fallback_kb_id"`
|
||||
ExternalFallbackKBID string `json:"external_fallback_kb_id"`
|
||||
InternalBoundByUI bool `json:"internal_bound_by_ui"`
|
||||
ExternalBoundByUI bool `json:"external_bound_by_ui"`
|
||||
InternalKBName string `json:"internal_kb_name"`
|
||||
ExternalKBName string `json:"external_kb_name"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
WeKnoraConnected bool `json:"weknora_connected"`
|
||||
// Replanned / RetiredDuplicates describe what THIS save did to 待入库 items
|
||||
// that had not been uploaded yet. A plain read reports zeros.
|
||||
Replanned int `json:"replanned"`
|
||||
@@ -98,9 +101,10 @@ func (config Config) routingBindingPayload(ctx context.Context, replan postgress
|
||||
return routingBindingView{
|
||||
InternalKBID: internal, ExternalKBID: external,
|
||||
InternalBoundByUI: storedInternal != "", ExternalBoundByUI: storedExternal != "",
|
||||
InternalEnvFallback: storedInternal == "" && config.InternalRoutingKnowledgeBaseID != "",
|
||||
ExternalEnvFallback: storedExternal == "" && config.ExternalRoutingKnowledgeBaseID != "",
|
||||
InternalKBName: names[storedInternal], ExternalKBName: names[storedExternal],
|
||||
InternalEnvFallback: storedInternal == "" && config.InternalRoutingKnowledgeBaseID != "",
|
||||
ExternalEnvFallback: storedExternal == "" && config.ExternalRoutingKnowledgeBaseID != "",
|
||||
InternalFallbackKBID: config.InternalRoutingKnowledgeBaseID, ExternalFallbackKBID: config.ExternalRoutingKnowledgeBaseID,
|
||||
InternalKBName: names[storedInternal], ExternalKBName: names[storedExternal],
|
||||
UpdatedBy: updatedBy, WeKnoraConnected: connected,
|
||||
Replanned: replan.Replanned, RetiredDuplicates: replan.RetiredDuplicates,
|
||||
}, nil
|
||||
|
||||
@@ -177,14 +177,24 @@ type Config struct {
|
||||
// 它不是运行时开关:生产部署不设置它,那次探测仍然直连目标库逐表计数。
|
||||
LibraryScopeProbe func(ctx context.Context, scope string) (bool, map[string]int64, error)
|
||||
// LibraryRetentionVersions/Days 是保留策略;都为 0 = 未设置(不生成清理候选)。
|
||||
LibraryRetentionVersions int
|
||||
LibraryRetentionDays int
|
||||
BackupSchedule BackupScheduleStore
|
||||
BackupHost BackupHost
|
||||
BackupRuns *BackupRunTracker
|
||||
ProductionBindings ProductionBindingStore
|
||||
ProductionKnowledgeBases ProductionKnowledgeBaseLister
|
||||
ProductionWeKnoraBaseURL string
|
||||
LibraryRetentionVersions int
|
||||
LibraryRetentionDays int
|
||||
BackupSchedule BackupScheduleStore
|
||||
BackupHost BackupHost
|
||||
BackupRuns *BackupRunTracker
|
||||
ProductionBindings ProductionBindingStore
|
||||
ProductionKnowledgeBases ProductionKnowledgeBaseLister
|
||||
ProductionWeKnoraBaseURL string
|
||||
// WeKnoraConnections 是「连接(地址 + 写-only 密钥)」服务;未接线时三条路由
|
||||
// 返回 503,页面显示「未接线」而不是假装已配置。
|
||||
WeKnoraConnections WeKnoraConnections
|
||||
// WeKnoraAllowPrivateHTTP 允许同网段 http(生产 18093 反代);与客户端、监控
|
||||
// 探活共用同一个判定。WeKnoraTrustTLSProxy 允许在可信 TLS 反代后录入密钥。
|
||||
WeKnoraAllowPrivateHTTP bool
|
||||
WeKnoraTrustTLSProxy bool
|
||||
// WeKnoraConnectionResolver 读页面保存的连接(地址 + 密钥);页面上显示的
|
||||
// 生产地址必须来自这里,否则会出现「连接卡写 18093、绑定页还在用旧地址」。
|
||||
WeKnoraConnectionResolver WeKnoraConnectionResolver
|
||||
ServiceTargetVerifier ServiceTargetVerifier
|
||||
Now func() time.Time
|
||||
RequireWeKnoraReady bool
|
||||
@@ -421,6 +431,15 @@ func NewHandler(config Config) (http.Handler, error) {
|
||||
}
|
||||
config.handleSaveProductionBindings(w, r)
|
||||
})
|
||||
mux.HandleFunc("GET /api/v1/weknora-connections", func(w http.ResponseWriter, r *http.Request) {
|
||||
config.handleWeKnoraConnections(w, r)
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/weknora-connections/{environment}/test", func(w http.ResponseWriter, r *http.Request) {
|
||||
config.handleWeKnoraConnectionTest(w, r)
|
||||
})
|
||||
mux.HandleFunc("PUT /api/v1/weknora-connections/{environment}", func(w http.ResponseWriter, r *http.Request) {
|
||||
config.handleSaveWeKnoraConnection(w, r)
|
||||
})
|
||||
mux.HandleFunc("POST /api/v1/source-catalog/route", func(w http.ResponseWriter, r *http.Request) {
|
||||
routeCatalogFiles(w, r, config)
|
||||
})
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"knowledge-governance/internal/weknoraconn"
|
||||
)
|
||||
|
||||
// WeKnoraConnections is the connection service the routes below call. It is
|
||||
// deliberately narrow: the page may read state, run one real test, and save a
|
||||
// tested configuration — nothing here triggers a sync, a release or a reparse.
|
||||
type WeKnoraConnections interface {
|
||||
View(ctx context.Context, environment string) ([]weknoraconn.View, error)
|
||||
Test(ctx context.Context, environment string, input weknoraconn.TestInput) (weknoraconn.TestResult, error)
|
||||
Save(ctx context.Context, environment string, input weknoraconn.SaveInput, actor string) (weknoraconn.View, error)
|
||||
}
|
||||
|
||||
func weKnoraConnectionError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, weknoraconn.ErrNotFound):
|
||||
writeJSON(w, http.StatusNotFound, map[string]string{"error": "未知的 WeKnora 环境(只支持 test / production)"})
|
||||
case errors.Is(err, weknoraconn.ErrStaleRevision):
|
||||
writeJSON(w, http.StatusConflict, map[string]string{"error": "连接配置已被他人修改,请重新读取后再保存"})
|
||||
case errors.Is(err, weknoraconn.ErrNotTested):
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{
|
||||
"code": "weknora_connection_not_tested",
|
||||
"error": "请先点「测试连接」并成功,才能保存这一侧的配置",
|
||||
})
|
||||
case errors.Is(err, weknoraconn.ErrStorageUnavailable):
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{
|
||||
"error": "密钥存储不可用:主密钥文件缺失或权限不是 0600,本次未保存任何密钥",
|
||||
})
|
||||
case errors.Is(err, weknoraconn.ErrInvalid):
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "连接配置不合法:地址必填,且更换地址必须重新录入密钥"})
|
||||
default:
|
||||
writeJSON(w, http.StatusInternalServerError, map[string]string{"error": "WeKnora 连接配置处理失败"})
|
||||
}
|
||||
}
|
||||
|
||||
// weKnoraSecretTransportWithPrivateHTTP decides whether a plaintext key may be
|
||||
// accepted on this request. It reuses the QA-AI rule (TLS, loopback, trusted
|
||||
// TLS proxy + same origin) and adds the one case this deployment needs: the
|
||||
// operator console itself is reached over plain http on a private address, so a
|
||||
// same-origin fetch from that console is the only accepted non-TLS shape.
|
||||
func weKnoraSecretTransportWithPrivateHTTP(config Config, r *http.Request) bool {
|
||||
if qaAISecretTransport(r, config) {
|
||||
return true
|
||||
}
|
||||
if config.WeKnoraTrustTLSProxy && r.Header.Get("X-Forwarded-Proto") == "https" &&
|
||||
r.Header.Get("Sec-Fetch-Site") == "same-origin" {
|
||||
origin, err := url.Parse(r.Header.Get("Origin"))
|
||||
if err == nil && origin.Scheme == "https" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if !config.WeKnoraAllowPrivateHTTP {
|
||||
return false
|
||||
}
|
||||
if r.Header.Get("Sec-Fetch-Site") != "same-origin" {
|
||||
return false
|
||||
}
|
||||
origin := strings.TrimSpace(r.Header.Get("Origin"))
|
||||
if origin == "" {
|
||||
return false
|
||||
}
|
||||
parsed, err := url.Parse(origin)
|
||||
if err != nil || parsed.Host == "" {
|
||||
return false
|
||||
}
|
||||
if !strings.EqualFold(parsed.Host, r.Host) {
|
||||
return false
|
||||
}
|
||||
host := parsed.Hostname()
|
||||
if ip := net.ParseIP(host); ip != nil {
|
||||
return ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast()
|
||||
}
|
||||
return strings.EqualFold(host, "localhost")
|
||||
}
|
||||
|
||||
func (config Config) handleWeKnoraConnections(w http.ResponseWriter, r *http.Request) {
|
||||
if config.WeKnoraConnections == nil {
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "WeKnora 连接配置未接线"})
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
views, err := config.WeKnoraConnections.View(r.Context(), r.PathValue("environment"))
|
||||
if err != nil {
|
||||
weKnoraConnectionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"data": views})
|
||||
}
|
||||
|
||||
func (config Config) handleWeKnoraConnectionTest(w http.ResponseWriter, r *http.Request) {
|
||||
if config.WeKnoraConnections == nil {
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "WeKnora 连接配置未接线"})
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
var input weknoraconn.TestInput
|
||||
if err := decodeStrictJSON(r, &input); err != nil {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "请求体不合法"})
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(input.APIKey) != "" && !weKnoraSecretTransportWithPrivateHTTP(config, r) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{
|
||||
"code": "weknora_secret_secure_transport_required",
|
||||
"error": "密钥仅可在有效 HTTPS、本机回环地址,或已放行的私网治理台同源页面录入;本次未保存、未发送密钥",
|
||||
})
|
||||
return
|
||||
}
|
||||
result, err := config.WeKnoraConnections.Test(r.Context(), r.PathValue("environment"), input)
|
||||
if err != nil {
|
||||
weKnoraConnectionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"data": result})
|
||||
}
|
||||
|
||||
func (config Config) handleSaveWeKnoraConnection(w http.ResponseWriter, r *http.Request) {
|
||||
if config.WeKnoraConnections == nil {
|
||||
writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "WeKnora 连接配置未接线"})
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
if _, ok := requireOperationID(w, r); !ok {
|
||||
return
|
||||
}
|
||||
var input weknoraconn.SaveInput
|
||||
if err := decodeStrictJSON(r, &input); err != nil {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "请求体不合法"})
|
||||
return
|
||||
}
|
||||
if input.KeyAction != weknoraconn.KeyActionClear && strings.TrimSpace(input.APIKey) != "" &&
|
||||
!weKnoraSecretTransportWithPrivateHTTP(config, r) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{
|
||||
"code": "weknora_secret_secure_transport_required",
|
||||
"error": "密钥仅可在有效 HTTPS、本机回环地址,或已放行的私网治理台同源页面录入;本次未保存、未发送密钥",
|
||||
})
|
||||
return
|
||||
}
|
||||
view, err := config.WeKnoraConnections.Save(r.Context(), r.PathValue("environment"), input, actorFor(r))
|
||||
if err != nil {
|
||||
weKnoraConnectionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"data": view})
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/weknoraconn"
|
||||
)
|
||||
|
||||
type weknoraConnectionsFake struct {
|
||||
views []weknoraconn.View
|
||||
testResult weknoraconn.TestResult
|
||||
testErr error
|
||||
savedResult weknoraconn.View
|
||||
saveErr error
|
||||
lastTest weknoraconn.TestInput
|
||||
lastSave weknoraconn.SaveInput
|
||||
lastActor string
|
||||
}
|
||||
|
||||
func (fake *weknoraConnectionsFake) View(_ context.Context, environment string) ([]weknoraconn.View, error) {
|
||||
if environment == "" {
|
||||
return fake.views, nil
|
||||
}
|
||||
if !weknoraconn.ValidEnvironment(environment) {
|
||||
return nil, weknoraconn.ErrNotFound
|
||||
}
|
||||
for _, view := range fake.views {
|
||||
if view.Environment == environment {
|
||||
return []weknoraconn.View{view}, nil
|
||||
}
|
||||
}
|
||||
return []weknoraconn.View{{Environment: environment, Verification: weknoraconn.VerificationNotConfigured}}, nil
|
||||
}
|
||||
|
||||
func (fake *weknoraConnectionsFake) Test(_ context.Context, environment string, input weknoraconn.TestInput) (weknoraconn.TestResult, error) {
|
||||
if !weknoraconn.ValidEnvironment(environment) {
|
||||
return weknoraconn.TestResult{}, weknoraconn.ErrNotFound
|
||||
}
|
||||
fake.lastTest = input
|
||||
return fake.testResult, fake.testErr
|
||||
}
|
||||
|
||||
func (fake *weknoraConnectionsFake) Save(_ context.Context, environment string, input weknoraconn.SaveInput, actor string) (weknoraconn.View, error) {
|
||||
if !weknoraconn.ValidEnvironment(environment) {
|
||||
return weknoraconn.View{}, weknoraconn.ErrNotFound
|
||||
}
|
||||
fake.lastSave, fake.lastActor = input, actor
|
||||
return fake.savedResult, fake.saveErr
|
||||
}
|
||||
|
||||
func weknoraConnectionsHandler(t *testing.T, fake *weknoraConnectionsFake, allowPrivate bool) http.Handler {
|
||||
t.Helper()
|
||||
handler, err := NewHandler(Config{
|
||||
Store: &fakeJobStore{},
|
||||
WeKnoraConnections: fake,
|
||||
WeKnoraAllowPrivateHTTP: allowPrivate,
|
||||
Now: func() time.Time { return time.Date(2026, time.November, 22, 7, 0, 0, 0, time.UTC) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewHandler: %v", err)
|
||||
}
|
||||
return handler
|
||||
}
|
||||
|
||||
func TestWeKnoraConnectionRoutesProjectStateWithoutSecrets(t *testing.T) {
|
||||
fake := &weknoraConnectionsFake{views: []weknoraconn.View{
|
||||
{Environment: "test", Endpoint: "http://127.0.0.1:8092", KeyConfigured: true, KeyTail: "9f2c", Verification: weknoraconn.OutcomeOK, LibraryCount: 3, Revision: 2, StorageReady: true},
|
||||
{Environment: "production", Verification: weknoraconn.VerificationNotConfigured, StorageReady: true},
|
||||
}}
|
||||
handler := weknoraConnectionsHandler(t, fake, true)
|
||||
rec := httptest.NewRecorder()
|
||||
handler.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/weknora-connections", nil))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if rec.Header().Get("Cache-Control") != "no-store" {
|
||||
t.Fatalf("cache-control = %q", rec.Header().Get("Cache-Control"))
|
||||
}
|
||||
body := rec.Body.String()
|
||||
if !strings.Contains(body, `"key_tail":"9f2c"`) || strings.Contains(body, "key_ciphertext") || strings.Contains(body, "api_key") {
|
||||
t.Fatalf("body leaks or loses key projection: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWeKnoraConnectionTestKeepsFourAnswers(t *testing.T) {
|
||||
for _, outcome := range []string{weknoraconn.OutcomeOK, weknoraconn.OutcomeUnreachable, weknoraconn.OutcomeKeyInvalid, weknoraconn.OutcomeKeyForbidden} {
|
||||
fake := &weknoraConnectionsFake{testResult: weknoraconn.TestResult{
|
||||
Environment: "production", Endpoint: "http://172.29.151.11:18093", Outcome: outcome, Message: outcome,
|
||||
}}
|
||||
handler := weknoraConnectionsHandler(t, fake, true)
|
||||
rec := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPost, "/api/v1/weknora-connections/production/test", strings.NewReader(`{"endpoint":"http://172.29.151.11:18093","key_action":"keep"}`))
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
handler.ServeHTTP(rec, request)
|
||||
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), `"outcome":"`+outcome+`"`) {
|
||||
t.Fatalf("%s: status = %d body = %s", outcome, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWeKnoraConnectionTestRejectsPlainKeyFromAPublicOrigin(t *testing.T) {
|
||||
fake := &weknoraConnectionsFake{}
|
||||
handler := weknoraConnectionsHandler(t, fake, true)
|
||||
rec := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPost, "/api/v1/weknora-connections/test/test", strings.NewReader(`{"endpoint":"http://127.0.0.1:8092","api_key":"sk-live","key_action":"replace"}`))
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
handler.ServeHTTP(rec, request)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "weknora_secret_secure_transport_required") {
|
||||
t.Fatalf("status = %d body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if fake.lastTest.APIKey != "" {
|
||||
t.Fatal("a rejected key must never reach the service")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWeKnoraConnectionTestAcceptsSameOriginPrivateConsole(t *testing.T) {
|
||||
fake := &weknoraConnectionsFake{testResult: weknoraconn.TestResult{Environment: "test", Outcome: weknoraconn.OutcomeOK}}
|
||||
handler := weknoraConnectionsHandler(t, fake, true)
|
||||
rec := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPost, "/api/v1/weknora-connections/test/test", strings.NewReader(`{"endpoint":"http://127.0.0.1:8092","api_key":"sk-live","key_action":"replace"}`))
|
||||
request.Host = "172.29.151.10:8088"
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
request.Header.Set("Origin", "http://172.29.151.10:8088")
|
||||
request.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
handler.ServeHTTP(rec, request)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if fake.lastTest.APIKey != "sk-live" {
|
||||
t.Fatalf("service saw key %q", fake.lastTest.APIKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveWeKnoraConnectionRequiresIdempotencyKeyAndMapsNotTested(t *testing.T) {
|
||||
fake := &weknoraConnectionsFake{saveErr: weknoraconn.ErrNotTested}
|
||||
handler := weknoraConnectionsHandler(t, fake, false)
|
||||
rec := httptest.NewRecorder()
|
||||
handler.ServeHTTP(rec, httptest.NewRequest(http.MethodPut, "/api/v1/weknora-connections/production", strings.NewReader(`{"endpoint":"https://prod.example","key_action":"keep","base_revision":1}`)))
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "Idempotency-Key") {
|
||||
t.Fatalf("missing idempotency key: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
rec = httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPut, "/api/v1/weknora-connections/production", strings.NewReader(`{"endpoint":"https://prod.example","key_action":"keep","base_revision":1}`))
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
request.Header.Set("Idempotency-Key", "op-1")
|
||||
handler.ServeHTTP(rec, request)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "weknora_connection_not_tested") {
|
||||
t.Fatalf("not tested mapping: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestWeKnoraConnectionUnknownEnvironmentIs404(t *testing.T) {
|
||||
fake := &weknoraConnectionsFake{}
|
||||
handler := weknoraConnectionsHandler(t, fake, false)
|
||||
rec := httptest.NewRecorder()
|
||||
handler.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/api/v1/weknora-connections?environment=staging", nil))
|
||||
// The list route reads the path value, not the query: unknown environments
|
||||
// only matter on the per-environment routes.
|
||||
rec = httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPost, "/api/v1/weknora-connections/staging/test", strings.NewReader(`{"endpoint":"https://x.example","key_action":"keep"}`))
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
handler.ServeHTTP(rec, request)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
var _ = json.Marshal
|
||||
@@ -59,6 +59,14 @@ type Expectation struct {
|
||||
Digests map[string]string
|
||||
}
|
||||
|
||||
// TableFingerprint 是一张知识域表在**整实例**上的行数与身份集合摘要(不限租户/
|
||||
// 知识库)。恢复后必须与预备份时刻逐表一致:只比目标范围会让"写到范围外"的
|
||||
// 残留被当成已恢复(真实事故:恢复报成功,目标仍多 67 行)。
|
||||
type TableFingerprint struct {
|
||||
Count int64 `json:"count"`
|
||||
Digest string `json:"digest"`
|
||||
}
|
||||
|
||||
// BackupRef 指向一份可独立恢复的完整预备份。
|
||||
type BackupRef struct {
|
||||
ID string `json:"id"`
|
||||
@@ -67,6 +75,9 @@ type BackupRef struct {
|
||||
SnapshotID string `json:"snapshot_id"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
// Universe 是预备份时刻的整实例指纹,恢复后逐表比对。空表示这次没有采集
|
||||
// (例如 worker 被杀后用已持久化的快照接管),此时恢复只能靠内容校验兜底。
|
||||
Universe map[string]TableFingerprint `json:"universe,omitempty"`
|
||||
}
|
||||
|
||||
type Target interface {
|
||||
@@ -224,8 +235,16 @@ func failAndRestore(ctx context.Context, outcome Outcome, target Target, recorde
|
||||
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
|
||||
return outcome
|
||||
}
|
||||
// "恢复工具跑完"不是"恢复完成":必须再校验一次目标内容与预备份一致,
|
||||
// 否则页面会把半恢复状态报成已恢复(真实事故就是这样放开了维护)。
|
||||
if verifyErr := target.Verify(ctx, Expectation{SnapshotID: outcome.Backup.SnapshotID}); verifyErr != nil {
|
||||
outcome.State = "rollback_failed"
|
||||
outcome.Rollback = "恢复后校验失败(目标可能仍是半写状态):" + verifyErr.Error()
|
||||
recorder.Stage(ctx, "rollback", "failed", outcome.Rollback)
|
||||
return outcome
|
||||
}
|
||||
outcome.State = "rolled_back"
|
||||
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID
|
||||
outcome.Rollback = "已恢复到预备份 " + outcome.Backup.ID + ",并逐表/索引/资源复核一致"
|
||||
recorder.Stage(ctx, "rollback", "succeeded", outcome.Rollback)
|
||||
return outcome
|
||||
}
|
||||
|
||||
@@ -13,9 +13,14 @@ type fakeTarget struct {
|
||||
applyErr map[Stage]error
|
||||
verifyErr error
|
||||
restoreErr error
|
||||
applied []Stage
|
||||
restored int
|
||||
currentN int
|
||||
// 恢复之后校验是否通过:用来区分"恢复跑完"和"恢复已验证"。
|
||||
verifyPassesAfterRestore bool
|
||||
applied []Stage
|
||||
restored int
|
||||
currentN int
|
||||
verifyN int
|
||||
// 记录最后一次 Verify 是否发生在 Restore 之后(恢复必须被校验兜底)。
|
||||
verifiedAfterRestore bool
|
||||
}
|
||||
|
||||
func (f *fakeTarget) Current(context.Context) (CurrentState, error) {
|
||||
@@ -35,7 +40,17 @@ func (f *fakeTarget) Apply(_ context.Context, stage Stage, _ string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeTarget) Verify(context.Context, Expectation) error { return f.verifyErr }
|
||||
func (f *fakeTarget) Verify(context.Context, Expectation) error {
|
||||
f.verifyN++
|
||||
f.verifiedAfterRestore = f.restored > 0
|
||||
if f.verifyErr == nil {
|
||||
return nil
|
||||
}
|
||||
if f.verifyPassesAfterRestore && f.restored > 0 {
|
||||
return nil
|
||||
}
|
||||
return f.verifyErr
|
||||
}
|
||||
|
||||
func (f *fakeTarget) Restore(context.Context, BackupRef) error {
|
||||
f.restored++
|
||||
@@ -185,12 +200,39 @@ func TestRunRollsBackWhenStageFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRollsBackWhenVerificationFails(t *testing.T) {
|
||||
target := &fakeTarget{verifyErr: errors.New("生产分块数不一致")}
|
||||
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, nil)
|
||||
if outcome.State != "rolled_back" || target.restored != 1 {
|
||||
// 恢复跑完但恢复后校验仍失败:不能报 rolled_back。真实事故里页面就是在这种
|
||||
// 状态下显示"已恢复到预备份并验证",而目标其实还留着 67 行残留。
|
||||
func TestRunDoesNotClaimRollbackWhenRecoveryVerificationFails(t *testing.T) {
|
||||
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
|
||||
verifyErr: errors.New("生产分块数不一致")}
|
||||
recorder := &stageLog{}
|
||||
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
|
||||
if outcome.State != "rollback_failed" || target.restored != 1 {
|
||||
t.Fatalf("outcome = %+v restored=%d", outcome, target.restored)
|
||||
}
|
||||
if !strings.Contains(outcome.Rollback, "恢复后校验失败") {
|
||||
t.Fatalf("rollback message = %q", outcome.Rollback)
|
||||
}
|
||||
if !recorder.has("rollback:failed") {
|
||||
t.Fatalf("stage log = %v", recorder.entries)
|
||||
}
|
||||
}
|
||||
|
||||
// 只有"恢复到预备份且恢复后校验通过"才允许报 rolled_back。
|
||||
func TestRunReportsRolledBackOnlyAfterRecoveryVerifies(t *testing.T) {
|
||||
target := &fakeTarget{applyErr: map[Stage]error{StageResources: errors.New("资源写失败")},
|
||||
verifyErr: errors.New("半写状态"), verifyPassesAfterRestore: true}
|
||||
recorder := &stageLog{}
|
||||
outcome := Run(context.Background(), request(time.Now()), target, &fakeBackups{}, recorder)
|
||||
if outcome.State != "rolled_back" {
|
||||
t.Fatalf("outcome = %+v", outcome)
|
||||
}
|
||||
if !target.verifiedAfterRestore {
|
||||
t.Fatal("restore must be followed by a content verification")
|
||||
}
|
||||
if !recorder.has("rollback:succeeded") {
|
||||
t.Fatalf("stage log = %v", recorder.entries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportsRollbackFailureDistinctly(t *testing.T) {
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/governance"
|
||||
"knowledge-governance/internal/platform/weknora"
|
||||
)
|
||||
|
||||
// P10: every configured dependency reports Liveness ("process reachable") and
|
||||
@@ -197,10 +198,14 @@ func (provider *RealProvider) productionWeKnoraProbe(ctx context.Context, now ti
|
||||
func (provider *RealProvider) httpProbeOutcome(ctx context.Context, baseURL string, now time.Time) ProbeOutcome {
|
||||
parsed, err := url.Parse(baseURL)
|
||||
outcome := ProbeOutcome{ObservedAt: now}
|
||||
if err != nil || parsed.Scheme != "http" && parsed.Scheme != "https" || parsed.Host == "" {
|
||||
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" {
|
||||
outcome.Reachability = errors.New("探测配置无效")
|
||||
return outcome
|
||||
}
|
||||
if !weknora.HTTPAllowed(baseURL, weknora.HTTPPolicy{AllowPrivateHTTP: provider.allowPrivateHTTP}) {
|
||||
outcome.Reachability = errors.New("探测配置无效:该地址不被允许(https 或已放行的私网/回环 http)")
|
||||
return outcome
|
||||
}
|
||||
parsed.Path = strings.TrimSuffix(parsed.Path, "/") + "/health"
|
||||
parsed.RawQuery, parsed.Fragment = "", ""
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, parsed.String(), nil)
|
||||
|
||||
@@ -50,6 +50,10 @@ type RealProviderConfig struct {
|
||||
// WorkerStaleAfter bounds how long a missing worker heartbeat still counts
|
||||
// as alive (P10).
|
||||
WorkerStaleAfter time.Duration
|
||||
// AllowPrivateHTTP relaxes the plain-http rule for same-LAN addresses. The
|
||||
// console probes use the very same judgement as the WeKnora client, so a
|
||||
// card can never say 「可达」 while the client refuses the same URL.
|
||||
AllowPrivateHTTP bool
|
||||
}
|
||||
|
||||
type RealProvider struct {
|
||||
@@ -63,6 +67,7 @@ type RealProvider struct {
|
||||
productionWeKnoraResolver func(ctx context.Context) string
|
||||
weknoraReadiness func(ctx context.Context) error
|
||||
workerStaleAfter time.Duration
|
||||
allowPrivateHTTP bool
|
||||
health *healthTracker
|
||||
}
|
||||
|
||||
@@ -103,7 +108,7 @@ func NewRealProvider(config RealProviderConfig) (*RealProvider, error) {
|
||||
if stale <= 0 {
|
||||
stale = time.Minute
|
||||
}
|
||||
return &RealProvider{db: config.DB, now: config.Now, httpClient: config.HTTPClient, weknoraBaseURL: strings.TrimSpace(config.WeKnoraBaseURL), minerUBaseURL: strings.TrimSpace(config.MinerUBaseURL), testWeKnoraResolver: config.TestWeKnoraResolver, productionWeKnoraBaseURL: strings.TrimSpace(config.ProductionWeKnoraBaseURL), productionWeKnoraResolver: config.ProductionWeKnoraResolver, weknoraReadiness: config.WeKnoraReadiness, workerStaleAfter: stale, health: newHealthTracker()}, nil
|
||||
return &RealProvider{db: config.DB, now: config.Now, httpClient: config.HTTPClient, weknoraBaseURL: strings.TrimSpace(config.WeKnoraBaseURL), minerUBaseURL: strings.TrimSpace(config.MinerUBaseURL), testWeKnoraResolver: config.TestWeKnoraResolver, productionWeKnoraBaseURL: strings.TrimSpace(config.ProductionWeKnoraBaseURL), productionWeKnoraResolver: config.ProductionWeKnoraResolver, weknoraReadiness: config.WeKnoraReadiness, workerStaleAfter: stale, allowPrivateHTTP: config.AllowPrivateHTTP, health: newHealthTracker()}, nil
|
||||
}
|
||||
|
||||
func (provider *RealProvider) Overview(ctx context.Context) (governance.MonitorOverview, error) {
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
DROP TABLE IF EXISTS weknora_environment_connection_tests;
|
||||
DROP TABLE IF EXISTS weknora_environment_connection_history;
|
||||
DROP TABLE IF EXISTS weknora_environment_connections;
|
||||
@@ -0,0 +1,45 @@
|
||||
-- 连接是「地址 + 写-only 密钥」的一份状态,绑定仍然是「对内/对外目标库」的另一份
|
||||
-- 状态(routing_kb_bindings / production_kb_bindings)。两者刻意不合并:现场那张
|
||||
-- 把两者塞进一张表的孤表正是重复列的来源。
|
||||
--
|
||||
-- 编号从 000083 起:现场 0.9.0 的 schema_migrations 已登记到 82,而迁移器按号跳过、
|
||||
-- 没有顺序守卫 —— 76..82 号会被静默丢弃。仓内最大号到 75 不影响 83 的排序与执行。
|
||||
CREATE TABLE weknora_environment_connections (
|
||||
environment VARCHAR(16) PRIMARY KEY CHECK (environment IN ('test','production')),
|
||||
revision BIGINT NOT NULL DEFAULT 1 CHECK (revision > 0),
|
||||
endpoint TEXT NOT NULL DEFAULT '',
|
||||
key_ciphertext BYTEA NOT NULL DEFAULT '\x',
|
||||
key_tail VARCHAR(8) NOT NULL DEFAULT '',
|
||||
verification TEXT NOT NULL DEFAULT 'not_configured'
|
||||
CHECK (verification IN ('not_configured','ok','unreachable','key_invalid','key_forbidden')),
|
||||
library_count INTEGER NOT NULL DEFAULT 0 CHECK (library_count >= 0),
|
||||
checked_at TIMESTAMPTZ,
|
||||
updated_by TEXT NOT NULL DEFAULT '',
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- 历史只记「有没有配过密钥」,不记尾号也不记密文:一个可审计的时间线不需要能
|
||||
-- 反推出任何一位密钥。
|
||||
CREATE TABLE weknora_environment_connection_history (
|
||||
environment VARCHAR(16) NOT NULL CHECK (environment IN ('test','production')),
|
||||
revision BIGINT NOT NULL,
|
||||
endpoint TEXT NOT NULL,
|
||||
key_configured BOOLEAN NOT NULL,
|
||||
verification TEXT NOT NULL,
|
||||
actor TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (environment, revision)
|
||||
);
|
||||
|
||||
-- 「测通才可保存」需要一条可核对的证据:这次测试用的是哪个地址、哪把密钥(只存
|
||||
-- 指纹)、结论如何。保存时按 (environment, endpoint, key_fingerprint, ok) 匹配。
|
||||
CREATE TABLE weknora_environment_connection_tests (
|
||||
environment VARCHAR(16) PRIMARY KEY CHECK (environment IN ('test','production')),
|
||||
endpoint TEXT NOT NULL,
|
||||
key_fingerprint CHAR(64) NOT NULL CHECK (key_fingerprint ~ '^[0-9a-f]{64}$'),
|
||||
outcome TEXT NOT NULL
|
||||
CHECK (outcome IN ('ok','unreachable','key_invalid','key_forbidden')),
|
||||
library_count INTEGER NOT NULL DEFAULT 0 CHECK (library_count >= 0),
|
||||
message TEXT NOT NULL DEFAULT '',
|
||||
checked_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
@@ -11,51 +11,59 @@ func TestEmbeddedMigrationsAreVersionSorted(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("loadMigrations() error = %v", err)
|
||||
}
|
||||
if len(migrations) != 75 {
|
||||
t.Fatalf("migration count = %d, want 75", len(migrations))
|
||||
if len(migrations) != 76 {
|
||||
t.Fatalf("migration count = %d, want 76", len(migrations))
|
||||
}
|
||||
// 83 把「连接(地址 + 写-only 密钥)」从文件引用搬进治理库;编号必须 ≥83,
|
||||
// 因为现场 0.9.0 的 schema_migrations 已到 82,迁移器按号跳过且无顺序守卫。
|
||||
if connections := migrations[len(migrations)-1]; connections.Version != 83 ||
|
||||
!strings.Contains(connections.UpSQL, "weknora_environment_connections") ||
|
||||
!strings.Contains(connections.UpSQL, "key_ciphertext") ||
|
||||
!strings.Contains(connections.DownSQL, "DROP TABLE IF EXISTS weknora_environment_connections") {
|
||||
t.Fatalf("migration 83 = %#v", connections)
|
||||
}
|
||||
// 75 允许"建立目标权威基线"作为独立业务作业,并与 sync/restore 共用写互斥。
|
||||
if baseline := migrations[len(migrations)-1]; baseline.Version != 75 ||
|
||||
if baseline := migrations[len(migrations)-2]; baseline.Version != 75 ||
|
||||
!strings.Contains(baseline.UpSQL, "library_jobs_write_target") ||
|
||||
!strings.Contains(baseline.UpSQL, "'baseline'") {
|
||||
t.Fatalf("migration 75 = %#v", baseline)
|
||||
}
|
||||
// 74 记录封存执行器的运行时状态:页面可用性来自真实观测,不来自配置字符串。
|
||||
if executor := migrations[len(migrations)-2]; executor.Version != 74 ||
|
||||
if executor := migrations[len(migrations)-3]; executor.Version != 74 ||
|
||||
!strings.Contains(executor.UpSQL, "library_executor_status") {
|
||||
t.Fatalf("migration 74 = %#v", executor)
|
||||
}
|
||||
// 73 记录死信精确解除:保留上游行,只绑定指纹、重放引用与观测摘要。
|
||||
if release := migrations[len(migrations)-3]; release.Version != 73 ||
|
||||
if release := migrations[len(migrations)-4]; release.Version != 73 ||
|
||||
!strings.Contains(release.UpSQL, "library_dead_letter_releases") {
|
||||
t.Fatalf("migration 73 = %#v", release)
|
||||
}
|
||||
// 72 让维护占用按实例互斥(internal/external 不再各有各的锁)。
|
||||
if instance := migrations[len(migrations)-4]; instance.Version != 72 ||
|
||||
if instance := migrations[len(migrations)-5]; instance.Version != 72 ||
|
||||
!strings.Contains(instance.UpSQL, "library_instance_maintenance") {
|
||||
t.Fatalf("migration 72 = %#v", instance)
|
||||
}
|
||||
// 71 让维护记录能区分实例级维护与仅治理库互斥。
|
||||
if level := migrations[len(migrations)-5]; level.Version != 71 ||
|
||||
if level := migrations[len(migrations)-6]; level.Version != 71 ||
|
||||
!strings.Contains(level.UpSQL, "level") || !strings.Contains(level.UpSQL, "ingress_state") {
|
||||
t.Fatalf("migration 71 = %#v", level)
|
||||
}
|
||||
if maint := migrations[len(migrations)-6]; maint.Version != 70 || !strings.Contains(maint.UpSQL, "library_maintenance") {
|
||||
if maint := migrations[len(migrations)-7]; maint.Version != 70 || !strings.Contains(maint.UpSQL, "library_maintenance") {
|
||||
t.Fatalf("migration 70 = %#v", maint)
|
||||
}
|
||||
if tail := migrations[len(migrations)-7]; tail.Version != 69 ||
|
||||
if tail := migrations[len(migrations)-8]; tail.Version != 69 ||
|
||||
!strings.Contains(tail.UpSQL, "library_jobs_write_target") {
|
||||
t.Fatalf("migration 69 = %#v", tail)
|
||||
}
|
||||
if restore := migrations[len(migrations)-8]; restore.Version != 68 || !strings.Contains(restore.UpSQL, "restore") {
|
||||
if restore := migrations[len(migrations)-9]; restore.Version != 68 || !strings.Contains(restore.UpSQL, "restore") {
|
||||
t.Fatalf("migration 68 = %#v", restore)
|
||||
}
|
||||
if last := migrations[len(migrations)-9]; last.Version != 67 ||
|
||||
if last := migrations[len(migrations)-10]; last.Version != 67 ||
|
||||
!strings.Contains(last.UpSQL, "library_release_authorizations") ||
|
||||
!strings.Contains(last.UpSQL, "params_digest") {
|
||||
t.Fatalf("migration 67 = %#v", last)
|
||||
}
|
||||
if flow := migrations[len(migrations)-10]; flow.Version != 66 ||
|
||||
if flow := migrations[len(migrations)-11]; flow.Version != 66 ||
|
||||
!strings.Contains(flow.UpSQL, "CREATE TABLE library_versions") ||
|
||||
!strings.Contains(flow.UpSQL, "library_jobs_active") {
|
||||
t.Fatalf("migration 66 = %#v", flow)
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// WeKnoraConnection is one environment's 地址 + 写-only 密钥. The ciphertext is
|
||||
// opened only in the service layer; everything below this line treats it as an
|
||||
// opaque blob so no query result can accidentally be rendered as a secret.
|
||||
type WeKnoraConnection struct {
|
||||
Environment string
|
||||
Revision int64
|
||||
Endpoint string
|
||||
KeyCiphertext []byte
|
||||
KeyTail string
|
||||
Verification string
|
||||
LibraryCount int
|
||||
CheckedAt *time.Time
|
||||
UpdatedBy string
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
// WeKnoraConnectionTest is the evidence a save needs: which endpoint, which key
|
||||
// (fingerprint only) and what the last real listing said.
|
||||
type WeKnoraConnectionTest struct {
|
||||
Environment string
|
||||
Endpoint string
|
||||
KeyFingerprint string
|
||||
Outcome string
|
||||
LibraryCount int
|
||||
Message string
|
||||
CheckedAt time.Time
|
||||
}
|
||||
|
||||
// LoadWeKnoraConnections reads both environments. A missing row is simply absent
|
||||
// from the result: 「未配置」 is a state, not an error.
|
||||
func (s *Store) LoadWeKnoraConnections(ctx context.Context) ([]WeKnoraConnection, error) {
|
||||
rows, err := s.db.Query(ctx, `
|
||||
SELECT environment,revision,endpoint,key_ciphertext,key_tail,verification,library_count,checked_at,updated_by,updated_at
|
||||
FROM weknora_environment_connections
|
||||
ORDER BY environment`)
|
||||
if err != nil {
|
||||
return nil, errors.New("load weknora connections")
|
||||
}
|
||||
defer rows.Close()
|
||||
connections := make([]WeKnoraConnection, 0, 2)
|
||||
for rows.Next() {
|
||||
var connection WeKnoraConnection
|
||||
if err := rows.Scan(&connection.Environment, &connection.Revision, &connection.Endpoint,
|
||||
&connection.KeyCiphertext, &connection.KeyTail, &connection.Verification,
|
||||
&connection.LibraryCount, &connection.CheckedAt, &connection.UpdatedBy, &connection.UpdatedAt); err != nil {
|
||||
return nil, errors.New("load weknora connections")
|
||||
}
|
||||
connections = append(connections, connection)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
return nil, errors.New("load weknora connections")
|
||||
}
|
||||
return connections, nil
|
||||
}
|
||||
|
||||
// LoadWeKnoraConnection reads one environment. The boolean is false when the
|
||||
// environment was never configured.
|
||||
func (s *Store) LoadWeKnoraConnection(ctx context.Context, environment string) (WeKnoraConnection, bool, error) {
|
||||
var connection WeKnoraConnection
|
||||
err := s.db.QueryRow(ctx, `
|
||||
SELECT environment,revision,endpoint,key_ciphertext,key_tail,verification,library_count,checked_at,updated_by,updated_at
|
||||
FROM weknora_environment_connections WHERE environment=$1`, environment).
|
||||
Scan(&connection.Environment, &connection.Revision, &connection.Endpoint,
|
||||
&connection.KeyCiphertext, &connection.KeyTail, &connection.Verification,
|
||||
&connection.LibraryCount, &connection.CheckedAt, &connection.UpdatedBy, &connection.UpdatedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return WeKnoraConnection{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return WeKnoraConnection{}, false, errors.New("load weknora connection")
|
||||
}
|
||||
return connection, true, nil
|
||||
}
|
||||
|
||||
// SaveWeKnoraConnection writes one revision and its history row in a single
|
||||
// transaction. The advisory lock is per environment so two concurrent saves
|
||||
// cannot both pick the same revision.
|
||||
func (s *Store) SaveWeKnoraConnection(ctx context.Context, connection WeKnoraConnection, now time.Time) error {
|
||||
if now.IsZero() {
|
||||
return errors.New("weknora connection save requires a timestamp")
|
||||
}
|
||||
tx, err := s.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin weknora connection save: %w", err)
|
||||
}
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtext('weknora_connection:'||$1))`, connection.Environment); err != nil {
|
||||
return fmt.Errorf("lock weknora connection: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO weknora_environment_connections(environment,revision,endpoint,key_ciphertext,key_tail,verification,library_count,checked_at,updated_by,updated_at)
|
||||
VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
|
||||
ON CONFLICT(environment) DO UPDATE SET
|
||||
revision=EXCLUDED.revision,
|
||||
endpoint=EXCLUDED.endpoint,
|
||||
key_ciphertext=EXCLUDED.key_ciphertext,
|
||||
key_tail=EXCLUDED.key_tail,
|
||||
verification=EXCLUDED.verification,
|
||||
library_count=EXCLUDED.library_count,
|
||||
checked_at=EXCLUDED.checked_at,
|
||||
updated_by=EXCLUDED.updated_by,
|
||||
updated_at=EXCLUDED.updated_at`,
|
||||
connection.Environment, connection.Revision, connection.Endpoint, connection.KeyCiphertext,
|
||||
connection.KeyTail, connection.Verification, connection.LibraryCount, connection.CheckedAt,
|
||||
connection.UpdatedBy, now); err != nil {
|
||||
return errors.New("save weknora connection")
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO weknora_environment_connection_history(environment,revision,endpoint,key_configured,verification,actor,created_at)
|
||||
VALUES($1,$2,$3,$4,$5,$6,$7)
|
||||
ON CONFLICT(environment,revision) DO NOTHING`,
|
||||
connection.Environment, connection.Revision, connection.Endpoint, len(connection.KeyCiphertext) > 0,
|
||||
connection.Verification, connection.UpdatedBy, now); err != nil {
|
||||
return errors.New("record weknora connection history")
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("commit weknora connection save: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SaveWeKnoraConnectionTest stores the outcome of the real listing a save will
|
||||
// later point at. Only the key fingerprint is kept — never the key.
|
||||
func (s *Store) SaveWeKnoraConnectionTest(ctx context.Context, result WeKnoraConnectionTest, now time.Time) error {
|
||||
if now.IsZero() {
|
||||
return errors.New("weknora connection test requires a timestamp")
|
||||
}
|
||||
if _, err := s.db.Exec(ctx, `
|
||||
INSERT INTO weknora_environment_connection_tests(environment,endpoint,key_fingerprint,outcome,library_count,message,checked_at)
|
||||
VALUES($1,$2,$3,$4,$5,$6,$7)
|
||||
ON CONFLICT(environment) DO UPDATE SET
|
||||
endpoint=EXCLUDED.endpoint,
|
||||
key_fingerprint=EXCLUDED.key_fingerprint,
|
||||
outcome=EXCLUDED.outcome,
|
||||
library_count=EXCLUDED.library_count,
|
||||
message=EXCLUDED.message,
|
||||
checked_at=EXCLUDED.checked_at`,
|
||||
result.Environment, result.Endpoint, result.KeyFingerprint, result.Outcome,
|
||||
result.LibraryCount, result.Message, now); err != nil {
|
||||
return errors.New("save weknora connection test")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// LoadWeKnoraConnectionTest reads the last test for one environment.
|
||||
func (s *Store) LoadWeKnoraConnectionTest(ctx context.Context, environment string) (WeKnoraConnectionTest, bool, error) {
|
||||
var result WeKnoraConnectionTest
|
||||
err := s.db.QueryRow(ctx, `
|
||||
SELECT environment,endpoint,key_fingerprint,outcome,library_count,message,checked_at
|
||||
FROM weknora_environment_connection_tests WHERE environment=$1`, environment).
|
||||
Scan(&result.Environment, &result.Endpoint, &result.KeyFingerprint, &result.Outcome,
|
||||
&result.LibraryCount, &result.Message, &result.CheckedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return WeKnoraConnectionTest{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return WeKnoraConnectionTest{}, false, errors.New("load weknora connection test")
|
||||
}
|
||||
return result, true, nil
|
||||
}
|
||||
|
||||
// UpdateWeKnoraConnectionVerification refreshes the card state without touching
|
||||
// the saved endpoint or ciphertext.
|
||||
func (s *Store) UpdateWeKnoraConnectionVerification(ctx context.Context, environment, verification string, libraryCount int, checkedAt time.Time, now time.Time) error {
|
||||
if _, err := s.db.Exec(ctx, `
|
||||
UPDATE weknora_environment_connections
|
||||
SET verification=$2, library_count=$3, checked_at=$4, updated_at=$5
|
||||
WHERE environment=$1`, environment, verification, libraryCount, checkedAt, now); err != nil {
|
||||
return errors.New("update weknora connection verification")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ProductionExecutionCounts reports, for executions that are still in flight,
|
||||
// how many items already have a frozen production target and how many do not yet.
|
||||
// A production binding change never rewrites the frozen ones: an execution that
|
||||
// already started keeps the knowledge base it was authorized against. Lifetime
|
||||
// counts are deliberately excluded — completed executions are history, not
|
||||
// something a binding change could still affect.
|
||||
func (s *Store) ProductionExecutionCounts(ctx context.Context) (int, int, error) {
|
||||
var frozen, retargetable int
|
||||
if err := s.db.QueryRow(ctx, `
|
||||
SELECT
|
||||
COUNT(*) FILTER (WHERE item.target_knowledge_base_id <> ''),
|
||||
COUNT(*) FILTER (WHERE item.target_knowledge_base_id = '')
|
||||
FROM release_item_executions item
|
||||
JOIN release_executions execution ON execution.id = item.release_execution_id
|
||||
WHERE execution.state IN ('pending','running')`).Scan(&frozen, &retargetable); err != nil {
|
||||
return 0, 0, errors.New("load production execution counts")
|
||||
}
|
||||
return frozen, retargetable, nil
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// 连接(地址 + 写-only 密钥)是新的持久状态:一环境一行、修订递增、历史只记
|
||||
// 「有没有配过密钥」。测试必须证明 000083 真能 up/down,而不是只证明 SQL 解析通过。
|
||||
func TestWeKnoraConnectionsMigrationUpDownAndStore(t *testing.T) {
|
||||
databaseURL := isolatedIntegrationDatabaseURL(t, os.Getenv("TEST_DATABASE_URL"))
|
||||
if databaseURL == "" {
|
||||
t.Skip("TEST_DATABASE_URL is not set")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, databaseURL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer pool.Close()
|
||||
if err = ApplyMigrations(ctx, pool); err != nil {
|
||||
t.Fatalf("apply migrations: %v", err)
|
||||
}
|
||||
var applied bool
|
||||
if err = pool.QueryRow(ctx, "SELECT EXISTS (SELECT 1 FROM schema_migrations WHERE version = 83)").Scan(&applied); err != nil || !applied {
|
||||
t.Fatalf("migration 83 not recorded: %v %v", applied, err)
|
||||
}
|
||||
|
||||
store := NewStore(pool)
|
||||
now := time.Date(2026, time.November, 22, 7, 0, 0, 0, time.UTC)
|
||||
checked := now.Add(-time.Minute)
|
||||
connection := WeKnoraConnection{
|
||||
Environment: "production",
|
||||
Revision: 2,
|
||||
Endpoint: "http://172.29.151.11:18093",
|
||||
KeyCiphertext: []byte{1, 2, 3, 4},
|
||||
KeyTail: "9f2c",
|
||||
Verification: "ok",
|
||||
LibraryCount: 4,
|
||||
CheckedAt: &checked,
|
||||
UpdatedBy: "admin",
|
||||
}
|
||||
if err := store.SaveWeKnoraConnection(ctx, connection, now); err != nil {
|
||||
t.Fatalf("save connection: %v", err)
|
||||
}
|
||||
loaded, exists, err := store.LoadWeKnoraConnection(ctx, "production")
|
||||
if err != nil || !exists {
|
||||
t.Fatalf("load connection: %v %v", exists, err)
|
||||
}
|
||||
if loaded.Endpoint != connection.Endpoint || loaded.Revision != 2 || loaded.KeyTail != "9f2c" || string(loaded.KeyCiphertext) != string(connection.KeyCiphertext) {
|
||||
t.Fatalf("loaded = %+v", loaded)
|
||||
}
|
||||
var configured bool
|
||||
var actor string
|
||||
if err := pool.QueryRow(ctx, `SELECT key_configured,actor FROM weknora_environment_connection_history WHERE environment='production' AND revision=2`).Scan(&configured, &actor); err != nil {
|
||||
t.Fatalf("history row: %v", err)
|
||||
}
|
||||
if !configured || actor != "admin" {
|
||||
t.Fatalf("history = %v %q", configured, actor)
|
||||
}
|
||||
// 历史里不能出现尾号或密文:只有布尔。
|
||||
var historyColumns int
|
||||
if err := pool.QueryRow(ctx, `SELECT COUNT(*) FROM information_schema.columns WHERE table_schema=current_schema() AND table_name='weknora_environment_connection_history' AND column_name IN ('key_tail','key_ciphertext')`).Scan(&historyColumns); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if historyColumns != 0 {
|
||||
t.Fatalf("history table exposes %d secret columns", historyColumns)
|
||||
}
|
||||
|
||||
// 测试证据:同一环境只保留最后一次,且只存密钥指纹。
|
||||
fingerprint := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
if err := store.SaveWeKnoraConnectionTest(ctx, WeKnoraConnectionTest{
|
||||
Environment: "production", Endpoint: connection.Endpoint, KeyFingerprint: fingerprint,
|
||||
Outcome: "ok", LibraryCount: 4, Message: "接口可达",
|
||||
}, now); err != nil {
|
||||
t.Fatalf("save test evidence: %v", err)
|
||||
}
|
||||
proof, found, err := store.LoadWeKnoraConnectionTest(ctx, "production")
|
||||
if err != nil || !found || proof.KeyFingerprint != fingerprint || proof.Outcome != "ok" {
|
||||
t.Fatalf("load test evidence = %+v %v %v", proof, found, err)
|
||||
}
|
||||
|
||||
// 冻结/未冻结计数按 target_knowledge_base_id 判定,不能用 IS NULL。
|
||||
frozen, retargetable, err := store.ProductionExecutionCounts(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("production execution counts: %v", err)
|
||||
}
|
||||
if frozen != 0 || retargetable != 0 {
|
||||
t.Fatalf("empty execution counts = %d/%d", frozen, retargetable)
|
||||
}
|
||||
|
||||
// down 必须只删这三张新表,不碰既有绑定表。
|
||||
if err := RollbackLatestMigration(ctx, pool); err != nil {
|
||||
t.Fatalf("rollback 83: %v", err)
|
||||
}
|
||||
for _, table := range []string{"weknora_environment_connections", "weknora_environment_connection_history", "weknora_environment_connection_tests"} {
|
||||
var present bool
|
||||
if err := pool.QueryRow(ctx, `SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema=current_schema() AND table_name=$1)`, table).Scan(&present); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if present {
|
||||
t.Fatalf("%s survived the down migration", table)
|
||||
}
|
||||
}
|
||||
for _, table := range []string{"routing_kb_bindings", "production_kb_bindings"} {
|
||||
var present bool
|
||||
if err := pool.QueryRow(ctx, `SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema=current_schema() AND table_name=$1)`, table).Scan(&present); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !present {
|
||||
t.Fatalf("%s must not be dropped by migration 83 down", table)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -48,6 +48,43 @@ type ClientConfig struct {
|
||||
Timeout time.Duration
|
||||
MaxResponseBytes int64
|
||||
AllowDockerHTTP bool
|
||||
// AllowPrivateHTTP relaxes the http-only rule for RFC1918, link-local and
|
||||
// ULA literals. It exists because the production instance sits behind a
|
||||
// same-LAN nginx that speaks plain http; https stays the default and any
|
||||
// public http host is still refused.
|
||||
AllowPrivateHTTP bool
|
||||
}
|
||||
|
||||
// HTTPPolicy is the single reachability judgement shared by the WeKnora client
|
||||
// and the console probes: two callers must never disagree about whether a
|
||||
// configured endpoint is acceptable.
|
||||
type HTTPPolicy struct {
|
||||
AllowPrivateHTTP bool
|
||||
AllowDockerHTTP bool
|
||||
}
|
||||
|
||||
// HTTPAllowed reports whether baseURL may be called at all. https is always
|
||||
// allowed; http only for loopback, an opted-in private address, or (legacy)
|
||||
// a single-label docker host.
|
||||
func HTTPAllowed(rawURL string, policy HTTPPolicy) bool {
|
||||
parsed, err := url.Parse(strings.TrimSpace(rawURL))
|
||||
if err != nil || parsed.Host == "" || parsed.User != nil {
|
||||
return false
|
||||
}
|
||||
switch parsed.Scheme {
|
||||
case "https":
|
||||
return true
|
||||
case "http":
|
||||
host := parsed.Hostname()
|
||||
if isLoopbackHost(host) {
|
||||
return true
|
||||
}
|
||||
if policy.AllowPrivateHTTP && isPrivateHost(host) {
|
||||
return true
|
||||
}
|
||||
return policy.AllowDockerHTTP && isSingleLabelDockerHost(host)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type Client struct {
|
||||
@@ -65,7 +102,7 @@ func NewClient(config ClientConfig) (*Client, error) {
|
||||
if baseURL.Scheme == "" || baseURL.Host == "" || baseURL.User != nil || baseURL.RawQuery != "" || baseURL.Fragment != "" {
|
||||
return nil, errors.New("invalid WeKnora base URL")
|
||||
}
|
||||
if baseURL.Scheme != "https" && !(baseURL.Scheme == "http" && (isLoopbackHost(baseURL.Hostname()) || (config.AllowDockerHTTP && isSingleLabelDockerHost(baseURL.Hostname())))) {
|
||||
if !HTTPAllowed(config.BaseURL, HTTPPolicy{AllowPrivateHTTP: config.AllowPrivateHTTP, AllowDockerHTTP: config.AllowDockerHTTP}) {
|
||||
return nil, errors.New("WeKnora base URL must use HTTPS")
|
||||
}
|
||||
baseURL.Path = strings.TrimRight(baseURL.Path, "/")
|
||||
@@ -99,6 +136,17 @@ func isLoopbackHost(host string) bool {
|
||||
return ip != nil && ip.IsLoopback()
|
||||
}
|
||||
|
||||
// isPrivateHost reports the addresses a same-LAN deployment legitimately uses:
|
||||
// RFC1918, link-local and IPv6 ULA. A public name is never private, so opting
|
||||
// in cannot widen the client to the internet.
|
||||
func isPrivateHost(host string) bool {
|
||||
ip := net.ParseIP(host)
|
||||
if ip == nil {
|
||||
return false
|
||||
}
|
||||
return ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast()
|
||||
}
|
||||
|
||||
func isSingleLabelDockerHost(host string) bool {
|
||||
if host == "" || strings.Contains(host, ".") || net.ParseIP(host) != nil || len(host) > 63 {
|
||||
return false
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
package weknora
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestHTTPAllowedSharesOneJudgement(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
url string
|
||||
policy HTTPPolicy
|
||||
want bool
|
||||
}{
|
||||
{"https always", "https://weknora.example", HTTPPolicy{}, true},
|
||||
{"loopback http", "http://127.0.0.1:8092", HTTPPolicy{}, true},
|
||||
{"localhost http", "http://localhost:8092", HTTPPolicy{}, true},
|
||||
{"private http refused by default", "http://172.29.151.11:18093", HTTPPolicy{}, false},
|
||||
{"private http allowed when opted in", "http://172.29.151.11:18093", HTTPPolicy{AllowPrivateHTTP: true}, true},
|
||||
{"rfc1918 /8", "http://10.1.2.3", HTTPPolicy{AllowPrivateHTTP: true}, true},
|
||||
{"rfc1918 /16", "http://192.168.1.10", HTTPPolicy{AllowPrivateHTTP: true}, true},
|
||||
{"link-local", "http://169.254.10.10", HTTPPolicy{AllowPrivateHTTP: true}, true},
|
||||
{"public http still refused", "http://203.0.113.9", HTTPPolicy{AllowPrivateHTTP: true}, false},
|
||||
{"public name still refused", "http://weknora.example", HTTPPolicy{AllowPrivateHTTP: true}, false},
|
||||
{"docker single label opt-in", "http://weknora", HTTPPolicy{AllowDockerHTTP: true}, true},
|
||||
{"docker single label off", "http://weknora", HTTPPolicy{}, false},
|
||||
{"credentials refused", "http://user:pass@127.0.0.1:8092", HTTPPolicy{}, false},
|
||||
{"empty host", "http://", HTTPPolicy{}, false},
|
||||
{"other scheme", "ftp://127.0.0.1", HTTPPolicy{}, false},
|
||||
}
|
||||
for _, testCase := range cases {
|
||||
if got := HTTPAllowed(testCase.url, testCase.policy); got != testCase.want {
|
||||
t.Fatalf("%s: HTTPAllowed(%q, %+v) = %v, want %v", testCase.name, testCase.url, testCase.policy, got, testCase.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewClientAcceptsPrivateHTTPOnlyWhenOptedIn(t *testing.T) {
|
||||
base := ClientConfig{BaseURL: "http://172.29.151.11:18093", APIKey: "k", Timeout: 1}
|
||||
if _, err := NewClient(base); err == nil {
|
||||
t.Fatal("private http must be refused without the opt-in")
|
||||
}
|
||||
base.AllowPrivateHTTP = true
|
||||
if _, err := NewClient(base); err != nil {
|
||||
t.Fatalf("private http must be accepted with the opt-in: %v", err)
|
||||
}
|
||||
}
|
||||
+21
-41
@@ -1,55 +1,35 @@
|
||||
package qaopsai
|
||||
|
||||
import (
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/rand"
|
||||
"fmt"
|
||||
"os"
|
||||
"knowledge-governance/internal/sealedsecret"
|
||||
)
|
||||
|
||||
// The independently provisioned master key never lives in source, settings JSON or database backups.
|
||||
func (s *Store) masterKey() ([]byte, error) {
|
||||
info, err := os.Lstat(s.options.KeyFile)
|
||||
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm() != 0600 {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
raw, err := os.ReadFile(s.options.KeyFile)
|
||||
if err != nil || len(raw) != 32 {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return raw, nil
|
||||
}
|
||||
func (s *Store) aead() (cipher.AEAD, error) {
|
||||
key, err := s.masterKey()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
block, err := aes.NewCipher(key)
|
||||
if err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return cipher.NewGCM(block)
|
||||
}
|
||||
// QA-AI keeps one write-only credential per revision. The AES-GCM convention
|
||||
// itself lives in internal/sealedsecret so a second feature cannot grow a
|
||||
// second, slightly different crypto path; only the AAD domain and the error
|
||||
// vocabulary stay here.
|
||||
const qaAISecretScope = "qa-ai"
|
||||
|
||||
func (s *Store) seal(secret string, revision int64) ([]byte, error) {
|
||||
aead, err := s.aead()
|
||||
key, err := sealedsecret.LoadKeyFile(s.options.KeyFile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
nonce := make([]byte, aead.NonceSize())
|
||||
if _, err = rand.Read(nonce); err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return aead.Seal(nonce, nonce, []byte(secret), []byte(fmt.Sprintf("qa-ai:%s:%d", s.tenant, revision))), nil
|
||||
}
|
||||
func (s *Store) unseal(sealed []byte, revision int64) (string, error) {
|
||||
aead, err := s.aead()
|
||||
if err != nil || len(sealed) < aead.NonceSize() {
|
||||
return "", ErrUnavailable
|
||||
sealed, err := sealedsecret.Seal(key, secret, sealedsecret.AAD(qaAISecretScope, s.tenant, revision))
|
||||
if err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
raw, err := aead.Open(nil, sealed[:aead.NonceSize()], sealed[aead.NonceSize():], []byte(fmt.Sprintf("qa-ai:%s:%d", s.tenant, revision)))
|
||||
return sealed, nil
|
||||
}
|
||||
|
||||
func (s *Store) unseal(sealed []byte, revision int64) (string, error) {
|
||||
key, err := sealedsecret.LoadKeyFile(s.options.KeyFile)
|
||||
if err != nil {
|
||||
return "", ErrUnavailable
|
||||
}
|
||||
return string(raw), nil
|
||||
plain, err := sealedsecret.Open(key, sealed, sealedsecret.AAD(qaAISecretScope, s.tenant, revision))
|
||||
if err != nil {
|
||||
return "", ErrUnavailable
|
||||
}
|
||||
return plain, nil
|
||||
}
|
||||
|
||||
@@ -9,6 +9,8 @@ import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/sealedsecret"
|
||||
)
|
||||
|
||||
type Options struct {
|
||||
@@ -67,7 +69,7 @@ func (s *Store) readSettings(ctx context.Context, q settingsQuery) (privateSetti
|
||||
return out, ErrUnavailable
|
||||
}
|
||||
out.KeyConfigured = len(out.sealed) > 0
|
||||
_, keyErr := s.masterKey()
|
||||
_, keyErr := sealedsecret.LoadKeyFile(s.options.KeyFile)
|
||||
out.StorageReady = keyErr == nil
|
||||
return out, nil
|
||||
}
|
||||
|
||||
@@ -48,7 +48,10 @@ func TestNormalizeRefusesEnabledSettingsWithoutTransportAndBadRecipients(t *test
|
||||
t.Fatalf("defaults not applied: %+v %v", good, err)
|
||||
}
|
||||
// 解析不了的主机名必须仍被判为非法:.invalid 是 RFC 6761 保留域,永远不解析。
|
||||
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid", From: "kb@example.com",
|
||||
// 末尾的点是必须的:构建机的 resolv.conf 可能带 search 域(本环境 Tailscale
|
||||
// 的 tailf49802.ts.net),不带点就会被拼成 smtp.kg-test.invalid.<search> 并
|
||||
// 由 Fake-IP DNS 解析出来,于是这条断言测的是构建机的搜索域而不是被测逻辑。
|
||||
if _, err := normalize(SettingsInput{Host: "smtp.kg-test.invalid.", From: "kb@example.com",
|
||||
Recipients: []string{"a@example.com"}}); err != ErrInvalid {
|
||||
t.Fatalf("unresolvable host must stay invalid, got %v", err)
|
||||
}
|
||||
|
||||
@@ -113,6 +113,12 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, err
|
||||
}
|
||||
// 整实例指纹:恢复之后要拿它证明"多写的、少写的都没有"。只按目标范围比,
|
||||
// 写到范围外的行会被漏掉(真实事故:恢复报成功但目标多 67 行)。
|
||||
universeBefore, err := universeFingerprint(ctx, s.pool)
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, err
|
||||
}
|
||||
pg, err := ExportLibrary(ctx, s.pool, s.identity)
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, fmt.Errorf("读取生产整库失败: %w", err)
|
||||
@@ -154,6 +160,13 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
|
||||
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间生产 %s 内容发生变化,切点不一致", table)
|
||||
}
|
||||
}
|
||||
universeAfter, err := universeFingerprint(ctx, s.pool)
|
||||
if err != nil {
|
||||
return libraryrelease.BackupRef{}, err
|
||||
}
|
||||
if diff := universeDiff(universeBefore, universeAfter); diff != "" {
|
||||
return libraryrelease.BackupRef{}, fmt.Errorf("预备份期间整实例知识域发生变化(%s),切点不一致", diff)
|
||||
}
|
||||
for table, want := range current.Counts {
|
||||
if table == "index" {
|
||||
continue
|
||||
@@ -171,7 +184,7 @@ func (s *ProductionBackupStore) Prepare(ctx context.Context, _ string, current l
|
||||
total += part.Bytes
|
||||
}
|
||||
return libraryrelease.BackupRef{ID: "backup-" + snapshotID, SnapshotID: snapshotID, Bytes: total,
|
||||
CreatedAt: createdAt}, nil
|
||||
CreatedAt: createdAt, Universe: universeBefore}, nil
|
||||
}
|
||||
|
||||
func (s *ProductionBackupStore) rowCounts(ctx context.Context) (map[string]int64, error) {
|
||||
|
||||
@@ -84,8 +84,16 @@ type ProductionLibraryTarget struct {
|
||||
scope ProductionScope
|
||||
search *SearchExporter
|
||||
now func() time.Time
|
||||
// storageBackend 是本次写入使用的目标存储后端 id(写入与校验共用)。
|
||||
storageBackend string
|
||||
// storageBackend / storageProvider 是本次写入使用的目标存储后端与提供方
|
||||
// (写入与校验共用,缺一不可:只换桶不换提供方,目标取文件仍会走来源实例的路)。
|
||||
storageBackend string
|
||||
storageProvider string
|
||||
// snapshotTenant / snapshotKB 是这次要写入的快照来源身份。写入时按它做
|
||||
// 来源→目标的范围映射;校验时按它反查"目标里还有没有来源范围的行"。
|
||||
snapshotTenant uint64
|
||||
snapshotKB string
|
||||
// targetEmbeddingModel 是目标知识库绑定的 embedding 模型 id(文档行要换成它)。
|
||||
targetEmbeddingModel string
|
||||
}
|
||||
|
||||
// resolveStorageBackend 在目标库里找本次写入的存储后端。找不到就报错:宁可不写,
|
||||
@@ -98,24 +106,39 @@ func (t *ProductionLibraryTarget) resolveStorageBackend(ctx context.Context) err
|
||||
if !validSearchIdentifier(t.config.ObjectsBackendID) {
|
||||
return errors.New("目标存储后端 id 形状不合法")
|
||||
}
|
||||
t.storageBackend = t.config.ObjectsBackendID
|
||||
provider, err := t.storageProviderOf(ctx, t.config.ObjectsBackendID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
t.storageBackend, t.storageProvider = t.config.ObjectsBackendID, provider
|
||||
return nil
|
||||
}
|
||||
var id string
|
||||
var id, provider string
|
||||
err := t.pool.QueryRow(ctx, `
|
||||
SELECT id FROM storage_backends
|
||||
SELECT id, coalesce(provider,'') FROM storage_backends
|
||||
WHERE tenant_id=$1 AND coalesce(config->>'bucket_name','')=$2
|
||||
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id)
|
||||
ORDER BY created_at LIMIT 1`, t.scope.TenantID, t.config.ObjectsBucket).Scan(&id, &provider)
|
||||
if err != nil {
|
||||
return errors.New("目标实例没有与目标桶匹配的存储后端,拒绝写入不可解析的资源定位符")
|
||||
}
|
||||
if !validSearchIdentifier(id) {
|
||||
return errors.New("目标存储后端 id 形状不合法")
|
||||
}
|
||||
t.storageBackend = id
|
||||
t.storageBackend, t.storageProvider = id, provider
|
||||
return nil
|
||||
}
|
||||
|
||||
// storageProviderOf 读出目标存储后端的提供方(minio / oss / s3…)。资源行里的
|
||||
// 提供方必须与它一致,否则目标应用会按来源提供方去找一个不存在的存储服务。
|
||||
func (t *ProductionLibraryTarget) storageProviderOf(ctx context.Context, id string) (string, error) {
|
||||
var provider string
|
||||
if err := t.pool.QueryRow(ctx, `SELECT coalesce(provider,'') FROM storage_backends WHERE id=$1 AND tenant_id=$2`,
|
||||
id, t.scope.TenantID).Scan(&provider); err != nil {
|
||||
return "", errors.New("目标实例没有这个存储后端,拒绝写入不可解析的资源定位符")
|
||||
}
|
||||
return provider, nil
|
||||
}
|
||||
|
||||
func NewProductionLibraryTarget(ctx context.Context, config ProductionLibraryConfig, scope ProductionScope) (*ProductionLibraryTarget, error) {
|
||||
if err := config.validate(scope); err != nil {
|
||||
return nil, err
|
||||
@@ -259,6 +282,7 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
|
||||
if err := t.resolveStorageBackend(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
t.resolveEmbeddingModel(ctx)
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return errors.New("归档不可用")
|
||||
@@ -268,6 +292,9 @@ func (t *ProductionLibraryTarget) Apply(ctx context.Context, stage libraryreleas
|
||||
if err != nil {
|
||||
return errors.New("预发布快照不可读")
|
||||
}
|
||||
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
|
||||
return err
|
||||
}
|
||||
switch stage {
|
||||
case libraryrelease.StageDatabase, libraryrelease.StageWiki:
|
||||
kind := "database"
|
||||
@@ -356,10 +383,10 @@ func (t *ProductionLibraryTarget) insertRows(ctx context.Context, tx pgx.Tx, arc
|
||||
if len(row) == 0 {
|
||||
continue
|
||||
}
|
||||
if table == "resources" {
|
||||
if err := t.applyResourceBucket(row); err != nil {
|
||||
return err
|
||||
}
|
||||
// 行必须先映射到目标范围再落库:原样插入会把行写进来源租户/知识库,
|
||||
// 目标范围的删除、校验与恢复都看不见它们。
|
||||
if err := t.retargetRow(table, row); err != nil {
|
||||
return err
|
||||
}
|
||||
if columns == nil {
|
||||
columns = make([]string, 0, len(row))
|
||||
@@ -408,53 +435,9 @@ type resourceLocation struct {
|
||||
}
|
||||
|
||||
// rowTransform 是"写入时对行做过的变换"的统一入口:校验要用同一份,否则摘要比的是
|
||||
// 变换前的行、目标里是变换后的行,永远不一致。
|
||||
// 变换前的行、目标里是变换后的行,永远不一致。范围映射同样必须两边一致。
|
||||
func (t *ProductionLibraryTarget) rowTransform(table string) func(map[string]json.RawMessage) error {
|
||||
if table == "resources" {
|
||||
return t.applyResourceBucket
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyResourceBucket 把资源行定位符里的桶换成本次写入的目标桶。
|
||||
// 写入与校验必须用同一个变换:只改写入侧会让"行内容摘要"对不上(真跑踩过)。
|
||||
func (t *ProductionLibraryTarget) applyResourceBucket(row map[string]json.RawMessage) error {
|
||||
if t.config.ObjectsBucket == "" {
|
||||
return nil
|
||||
}
|
||||
raw, ok := row["physical_path"]
|
||||
if !ok {
|
||||
return errors.New("资源行缺少定位符")
|
||||
}
|
||||
var locator string
|
||||
if err := json.Unmarshal(raw, &locator); err != nil {
|
||||
return errors.New("资源定位符不可解析")
|
||||
}
|
||||
rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket)
|
||||
if !ok {
|
||||
return fmt.Errorf("资源定位符无法改写到目标桶:%s", locator)
|
||||
}
|
||||
encoded, err := json.Marshal(rewritten)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
row["physical_path"] = encoded
|
||||
// 原版应用先按 resources.storage_backend_id 找存储后端,再回落到 KB,再落默认服务。
|
||||
// 这一列留着来源 id,目标仍然 500(真跑:storage backend not found → invalid path)。
|
||||
if t.storageBackend == "" {
|
||||
return nil
|
||||
}
|
||||
if raw, ok := row["storage_backend_id"]; ok {
|
||||
var current string
|
||||
if json.Unmarshal(raw, ¤t) == nil && current != "" && current != t.storageBackend {
|
||||
encodedBackend, err := json.Marshal(t.storageBackend)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
row["storage_backend_id"] = encodedBackend
|
||||
}
|
||||
}
|
||||
return nil
|
||||
return func(row map[string]json.RawMessage) error { return t.retargetRow(table, row) }
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) importResources(ctx context.Context, archive *librarysnapshot.Archive, manifest librarysnapshot.Manifest) error {
|
||||
@@ -591,6 +574,7 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err := t.resolveStorageBackend(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
t.resolveEmbeddingModel(ctx)
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return errors.New("归档不可用")
|
||||
@@ -600,6 +584,9 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err != nil {
|
||||
return errors.New("预发布快照不可读")
|
||||
}
|
||||
if err := t.bindSnapshotScope(manifestScope{tenantID: manifest.TenantID, knowledgeBaseID: manifest.KnowledgeBaseID}); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" {
|
||||
continue
|
||||
@@ -668,6 +655,72 @@ func (t *ProductionLibraryTarget) Verify(ctx context.Context, expect libraryrele
|
||||
if err := t.verifyObjects(ctx, archive, manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
// 目标里不允许留下"快照来源范围"的行:那说明有写入逃出了目标范围,
|
||||
// 而恢复只按目标范围回放,永远删不到它们(真实事故就是这样被漏掉的)。
|
||||
if err := t.assertNoSourceScopeRows(ctx, archive, manifest); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// dropOrphanResources 清掉"本次恢复后已无任何归属"的目标租户资源行。
|
||||
//
|
||||
// 判据必须同时满足:属于目标租户、不在预备份快照里、没有任何 resource_binding、
|
||||
// 也没有任何分块正文以 resource://<handle> 引用它。这样的行对目标应用完全不可见
|
||||
// (没有知识库能按它取到文件),只可能是失败写入留下的残渣。
|
||||
func (t *ProductionLibraryTarget) dropOrphanResources(ctx context.Context, backupSnapshotID string) error {
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return errors.New("归档不可用")
|
||||
}
|
||||
defer archive.Close()
|
||||
manifest, err := archive.Load(backupSnapshotID)
|
||||
if err != nil {
|
||||
return errors.New("预备份快照不可读")
|
||||
}
|
||||
kept := []string{}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" || part.Name != "resources" {
|
||||
continue
|
||||
}
|
||||
ids, err := archivedResourceIDs(archive, part)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
kept = append(kept, ids...)
|
||||
}
|
||||
rows, err := t.pool.Query(ctx, `
|
||||
SELECT r.id FROM resources AS r
|
||||
WHERE r.tenant_id=$1
|
||||
AND NOT (r.id = ANY($2::text[]))
|
||||
AND NOT EXISTS (SELECT 1 FROM resource_bindings b WHERE b.resource_id=r.id)
|
||||
AND NOT EXISTS (SELECT 1 FROM chunks c WHERE c.tenant_id=r.tenant_id AND c.deleted_at IS NULL
|
||||
AND coalesce(r.handle,'') <> ''
|
||||
AND position('resource://'||r.handle IN coalesce(c.content,'')) > 0)`,
|
||||
t.scope.TenantID, kept)
|
||||
if err != nil {
|
||||
return fmt.Errorf("查找目标残留资源失败: %w", err)
|
||||
}
|
||||
orphans := make([]string, 0, 8)
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
orphans = append(orphans, id)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(orphans) == 0 {
|
||||
return nil
|
||||
}
|
||||
if _, err := t.pool.Exec(ctx, `DELETE FROM resources WHERE tenant_id=$1 AND id = ANY($2::text[])`,
|
||||
t.scope.TenantID, orphans); err != nil {
|
||||
return fmt.Errorf("清理目标残留资源失败: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -772,5 +825,21 @@ func (t *ProductionLibraryTarget) Restore(ctx context.Context, backup libraryrel
|
||||
return fmt.Errorf("恢复阶段 %s 失败: %w", stage, err)
|
||||
}
|
||||
}
|
||||
// resources 是多知识库共享表,回放阶段不删它:本次写入新增、恢复后已经
|
||||
// 没有任何归属的资源行会残留。它们对应用不可见,但属于"多写的"。
|
||||
if err := t.dropOrphanResources(ctx, backup.SnapshotID); err != nil {
|
||||
return err
|
||||
}
|
||||
// 恢复工具把四个阶段都跑完不算恢复完成:整实例必须回到预备份时刻的样子。
|
||||
// 只比目标范围会把"写到范围外"的残留当成已恢复(真实事故:恢复后仍多 67 行)。
|
||||
if len(backup.Universe) > 0 {
|
||||
after, err := universeFingerprint(ctx, t.pool)
|
||||
if err != nil {
|
||||
return fmt.Errorf("恢复后读取整库指纹失败: %w", err)
|
||||
}
|
||||
if diff := universeDiff(backup.Universe, after); diff != "" {
|
||||
return fmt.Errorf("恢复后目标与预备份时刻不一致:%s(很可能有写入逃出了预备份范围,不能当作已恢复)", diff)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -4,12 +4,15 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -390,3 +393,339 @@ func sealTargetFixture(t *testing.T, archive *librarysnapshot.Archive, scope str
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// memoryObjectStore 是"真实字节"的资源存储替身:Put/Open/Stat/Copy 都按摘要核对,
|
||||
// 可以强制写入失败,用来复现"资源阶段失败"这一半写场景。
|
||||
type memoryObjectStore struct {
|
||||
objects map[string][]byte
|
||||
failPut bool
|
||||
puts []string
|
||||
}
|
||||
|
||||
func newMemoryObjectStore(objects map[string][]byte) *memoryObjectStore {
|
||||
copied := map[string][]byte{}
|
||||
for key, value := range objects {
|
||||
copied[key] = append([]byte(nil), value...)
|
||||
}
|
||||
return &memoryObjectStore{objects: copied}
|
||||
}
|
||||
|
||||
func objectKey(provider, path string) string { return provider + "|" + path }
|
||||
|
||||
func (s *memoryObjectStore) Put(_ context.Context, provider, path string, reader io.Reader, size int64) error {
|
||||
if s.failPut {
|
||||
return errors.New("对象存储不可写")
|
||||
}
|
||||
body, err := io.ReadAll(reader)
|
||||
if err != nil || int64(len(body)) != size {
|
||||
return errors.New("对象字节数与声明不符")
|
||||
}
|
||||
s.objects[objectKey(provider, path)] = body
|
||||
s.puts = append(s.puts, path)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *memoryObjectStore) Open(_ context.Context, provider, path string) (io.ReadCloser, error) {
|
||||
body, ok := s.objects[objectKey(provider, path)]
|
||||
if !ok {
|
||||
return nil, errors.New("对象不存在")
|
||||
}
|
||||
return io.NopCloser(bytes.NewReader(body)), nil
|
||||
}
|
||||
|
||||
func (s *memoryObjectStore) Stat(_ context.Context, provider, path string) (int64, string, error) {
|
||||
body, ok := s.objects[objectKey(provider, path)]
|
||||
if !ok {
|
||||
return 0, "", errors.New("对象不存在")
|
||||
}
|
||||
digest := sha256.Sum256(body)
|
||||
return int64(len(body)), hex.EncodeToString(digest[:]), nil
|
||||
}
|
||||
|
||||
func (s *memoryObjectStore) Copy(_ context.Context, provider, source, target string) error {
|
||||
body, ok := s.objects[objectKey(provider, source)]
|
||||
if !ok {
|
||||
return errors.New("对象不存在")
|
||||
}
|
||||
s.objects[objectKey(provider, target)] = append([]byte(nil), body...)
|
||||
return nil
|
||||
}
|
||||
|
||||
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同)、目标非空、资源阶段写失败。
|
||||
// 旧代码把归档行原样写进目标(落在来源租户/知识库),随后的回滚只清目标范围,
|
||||
// 于是"已恢复到预备份"却留下 4 文件 / 38 分块 / 21 资源 / 4 绑定。
|
||||
func TestLibraryCrossScopeSyncRollbackLeavesNoResidueIsolated(t *testing.T) {
|
||||
dsn, base := os.Getenv("KG_LIBRARY_TEST_DATABASE_URL"), os.Getenv("KG_LIBRARY_TEST_ELASTIC_URL")
|
||||
if dsn == "" || base == "" || os.Getenv("KG_LIBRARY_TEST_ISOLATED") != "1" {
|
||||
t.Skip("requires isolated PG and ES opt-in")
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil || !strings.HasPrefix(cfg.ConnConfig.Database, "kg_library_test_") {
|
||||
t.Fatal("isolated database required")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||
defer cancel()
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("open isolated PG: %v", err)
|
||||
}
|
||||
defer pool.Close()
|
||||
for _, statement := range []string{
|
||||
`DROP SCHEMA public CASCADE; CREATE SCHEMA public`,
|
||||
libraryTestSchema,
|
||||
`ALTER TABLE wiki_pages ADD COLUMN status text DEFAULT 'published'`,
|
||||
`ALTER TABLE knowledge_tag_relations ADD COLUMN tenant_id bigint`,
|
||||
`ALTER TABLE resources ADD COLUMN provider text, ADD COLUMN size bigint, ADD COLUMN storage_backend_id text`,
|
||||
`ALTER TABLE resources ADD PRIMARY KEY (id)`,
|
||||
`ALTER TABLE resource_bindings ADD COLUMN relation text`,
|
||||
`INSERT INTO knowledge_bases VALUES('kb-target',10001,'{}',NULL),('kb-source',10000,'{}',NULL)`,
|
||||
`CREATE TABLE storage_backends(id text PRIMARY KEY, tenant_id bigint, provider text, config jsonb, created_at timestamptz)`,
|
||||
`INSERT INTO storage_backends VALUES('backend-target',10001,'minio','{"bucket_name":"kg-test-bucket"}',now())`,
|
||||
// 目标非空:目标自己有文档/分块,另一个租户也有数据(回滚不得碰它)。
|
||||
`INSERT INTO knowledges VALUES('doc-target',10001,'kb-target','completed',0,'',NULL)`,
|
||||
`INSERT INTO knowledges VALUES('doc-foreign',20000,'kb-other','completed',0,'',NULL)`,
|
||||
`INSERT INTO chunks VALUES('chunk-target',10001,'kb-target','doc-target','target content',NULL)`,
|
||||
`INSERT INTO chunks VALUES('chunk-foreign',20000,'kb-other','doc-foreign','foreign content',NULL)`,
|
||||
`UPDATE resource_bindings SET tenant_id=10000 WHERE tenant_id IS NULL`,
|
||||
} {
|
||||
if _, err := pool.Exec(ctx, statement); err != nil {
|
||||
t.Fatalf("fixture schema: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
token := make([]byte, 8)
|
||||
if _, err := rand.Read(token); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
index := "kg_library_test_" + hex.EncodeToString(token)
|
||||
client := &http.Client{Timeout: 20 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||
return http.ErrUseLastResponse
|
||||
}}
|
||||
call := func(method, path, body string, want int) {
|
||||
t.Helper()
|
||||
request, err := http.NewRequestWithContext(ctx, method, base+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
response, err := client.Do(request)
|
||||
if err != nil {
|
||||
t.Fatalf("fixture HTTP: %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != want {
|
||||
t.Fatalf("fixture %s %s => %d, want %d", method, path, response.StatusCode, want)
|
||||
}
|
||||
}
|
||||
call("PUT", "/"+index, `{"mappings":{"properties":{"tenant_id":{"type":"long"},"knowledge_base_id":{"type":"keyword"},"knowledge_id":{"type":"keyword"},"chunk_id":{"type":"keyword"},"content":{"type":"text"},"embedding":{"type":"dense_vector","dims":2}}}}`, 200)
|
||||
defer func() {
|
||||
call("DELETE", "/"+index, "", 200)
|
||||
call("HEAD", "/"+index, "", 404)
|
||||
}()
|
||||
// 目标当前索引自洽:目标分块必须有对应文档,否则预备份会(正确地)拒绝。
|
||||
call("PUT", "/"+index+"/_doc/chunk-target?refresh=true",
|
||||
`{"tenant_id":10001,"knowledge_base_id":"kb-target","knowledge_id":"doc-target","chunk_id":"chunk-target","source_id":"chunk-target","content":"target content","embedding":[0.1,0.2]}`, 201)
|
||||
|
||||
archiveRoot := t.TempDir()
|
||||
if err := os.Chmod(archiveRoot, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archive, err := librarysnapshot.Open(archiveRoot, 64*1024*1024)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer archive.Close()
|
||||
snapshotID := sealCrossScopeFixture(t, archive, 10000, "kb-source")
|
||||
|
||||
store := newMemoryObjectStore(nil)
|
||||
target, err := NewProductionLibraryTarget(ctx, ProductionLibraryConfig{
|
||||
Postgres: dsn, SearchURL: base, SearchIndex: index, Objects: store, ObjectsReader: store,
|
||||
ArchiveRoot: archiveRoot, ObjectsBucket: "kg-test-bucket", MaxDocuments: 100, HTTPClient: client,
|
||||
}, ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"})
|
||||
if err != nil {
|
||||
t.Fatalf("target: %v", err)
|
||||
}
|
||||
defer target.Close()
|
||||
mapping, err := target.Search().MappingSHA256(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
backups, err := NewProductionBackupStore(ProductionBackupConfig{
|
||||
Pool: target.Pool(), Search: target.Search(), Objects: store, ArchiveRoot: archiveRoot,
|
||||
Scope: "internal", Identity: target.Identity(), MaxDocuments: 100,
|
||||
Compatibility: librarysnapshot.Compatibility{WeKnoraVersion: "v0.7.1", SchemaVersion: 74,
|
||||
RetrieverEngine: "elasticsearch_v8", EmbeddingModel: "text-embedding-v4",
|
||||
EmbeddingDimension: 1024, IndexMappingSHA256: mapping},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("backup store: %v", err)
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
request := libraryrelease.Request{Target: "internal", SnapshotID: snapshotID,
|
||||
ParamsDigest: "digest-cross-scope", Now: now,
|
||||
Grant: libraryrelease.Grant{Target: "internal", SnapshotID: snapshotID, ParamsDigest: "digest-cross-scope",
|
||||
ExpiresAt: now.Add(10 * time.Minute), AuthorizedBy: "acceptance"}}
|
||||
|
||||
// 1) 资源阶段失败 → 必须回滚且不留下来源范围的行。
|
||||
store.failPut = true
|
||||
outcome := libraryrelease.Run(ctx, request, target, backups, nil)
|
||||
if outcome.State != "rolled_back" {
|
||||
t.Fatalf("outcome = %+v", outcome)
|
||||
}
|
||||
if outcome.FailedAt != libraryrelease.StageResources {
|
||||
t.Fatalf("failed_at = %s, want resources", outcome.FailedAt)
|
||||
}
|
||||
if outcome.Backup.SnapshotID == "" || len(outcome.Backup.Universe) == 0 {
|
||||
t.Fatalf("backup must carry the whole-instance fingerprint: %+v", outcome.Backup)
|
||||
}
|
||||
for _, table := range []string{"knowledges", "chunks", "wiki_pages"} {
|
||||
var residue int64
|
||||
if err := pool.QueryRow(ctx, fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND knowledge_base_id=$2`, table),
|
||||
10000, "kb-source").Scan(&residue); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if residue != 0 {
|
||||
t.Fatalf("%s still carries %d source-scope rows after rollback", table, residue)
|
||||
}
|
||||
}
|
||||
// 夹具在租户 10000 里本来就有自己的行(res-safe / binding-safe):它们必须原样
|
||||
// 留下,只有这次同步写进去的行(res-source / 绑到 doc-source)才算残留。
|
||||
for query, want := range map[string]int64{
|
||||
`SELECT count(*) FROM resources WHERE id='res-source'`: 0,
|
||||
`SELECT count(*) FROM resource_bindings WHERE tenant_id=10000 AND owner_id='doc-source'`: 0,
|
||||
`SELECT count(*) FROM resources WHERE tenant_id=10000`: 1,
|
||||
} {
|
||||
var got int64
|
||||
if err := pool.QueryRow(ctx, query).Scan(&got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("%s = %d, want %d", query, got, want)
|
||||
}
|
||||
}
|
||||
var leftovers int64
|
||||
if err := pool.QueryRow(ctx, `SELECT count(*) FROM resources WHERE tenant_id=10001`).Scan(&leftovers); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if leftovers != 0 {
|
||||
t.Fatalf("aborted write left %d resource rows in the target tenant", leftovers)
|
||||
}
|
||||
var foreign int64
|
||||
if err := pool.QueryRow(ctx, `SELECT count(*) FROM chunks WHERE tenant_id=20000`).Scan(&foreign); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// 夹具本来就有 1 行 other-scope(20000),加上本用例的 chunk-foreign,共 2 行。
|
||||
if foreign != 2 {
|
||||
t.Fatalf("cross-tenant rows must survive rollback, got %d", foreign)
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: outcome.Backup.SnapshotID}); err != nil {
|
||||
t.Fatalf("rollback must verify against the prebackup: %v", err)
|
||||
}
|
||||
|
||||
// 2) 同样的跨范围同步,资源能写 → 行必须落在目标范围,资源定位符指向目标存储。
|
||||
store.failPut = false
|
||||
succeed := libraryrelease.Run(ctx, request, target, backups, nil)
|
||||
if succeed.State != "succeeded" {
|
||||
t.Fatalf("second run outcome = %+v", succeed)
|
||||
}
|
||||
if err := target.Verify(ctx, libraryrelease.Expectation{SnapshotID: snapshotID}); err != nil {
|
||||
t.Fatalf("verify after cross-scope sync: %v", err)
|
||||
}
|
||||
var knowledgeKB, chunkKB, provider, locator string
|
||||
var tenant int64
|
||||
if err := pool.QueryRow(ctx, `SELECT tenant_id, knowledge_base_id FROM knowledges WHERE id='doc-source'`).
|
||||
Scan(&tenant, &knowledgeKB); err != nil {
|
||||
t.Fatalf("source document must exist in the target scope: %v", err)
|
||||
}
|
||||
if tenant != 10001 || knowledgeKB != "kb-target" {
|
||||
t.Fatalf("cross-scope row landed at %d/%s", tenant, knowledgeKB)
|
||||
}
|
||||
if err := pool.QueryRow(ctx, `SELECT knowledge_base_id FROM chunks WHERE id='chunk-source'`).Scan(&chunkKB); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if chunkKB != "kb-target" {
|
||||
t.Fatalf("chunk landed in %s", chunkKB)
|
||||
}
|
||||
if err := pool.QueryRow(ctx, `SELECT provider, physical_path FROM resources WHERE id='res-source'`).
|
||||
Scan(&provider, &locator); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if provider != "minio" {
|
||||
t.Fatalf("resource provider = %s, want minio", provider)
|
||||
}
|
||||
if !strings.Contains(locator, "storage://backend-target/minio://kg-test-bucket/10001/") {
|
||||
t.Fatalf("resource locator was not retargeted: %s", locator)
|
||||
}
|
||||
if strings.Contains(locator, "weknora/10000") {
|
||||
t.Fatalf("resource locator still points at the source tenant: %s", locator)
|
||||
}
|
||||
t.Logf("cross_scope_sync verified: snapshot=%s backup=%s locator=%s", snapshotID, succeed.Backup.SnapshotID, locator)
|
||||
}
|
||||
|
||||
// sealCrossScopeFixture 造一份"来源范围与目标不同"的整库快照:文档/分块/资源/绑定/
|
||||
// Wiki/索引都带来源身份,资源部件是可核对的真实字节。
|
||||
func sealCrossScopeFixture(t *testing.T, archive *librarysnapshot.Archive, sourceTenant uint64, sourceKB string) string {
|
||||
t.Helper()
|
||||
encode := func(value any) []byte {
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return encoded
|
||||
}
|
||||
const handle = "abcdefghijklmnopqrstuv"
|
||||
objectBytes := []byte("12345678")
|
||||
parts := make([]librarysnapshot.Part, 0, 12)
|
||||
put := func(kind, name string, count int64, body []byte) {
|
||||
part, err := archive.Put(bytes.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
part.Kind, part.Name, part.Count = kind, name, count
|
||||
parts = append(parts, part)
|
||||
}
|
||||
put("database", "knowledges", 1, encode([]map[string]any{{
|
||||
"id": "doc-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
|
||||
"parse_status": "completed", "pending_subtasks_count": 0, "file_path": ""}}))
|
||||
put("database", "chunks", 1, encode([]map[string]any{{
|
||||
"id": "chunk-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
|
||||
"knowledge_id": "doc-source", "content": "见 resource://" + handle}}))
|
||||
put("database", "knowledge_tags", 0, encode([]map[string]any{}))
|
||||
put("database", "knowledge_tag_relations", 0, encode([]map[string]any{}))
|
||||
put("database", "resources", 1, encode([]map[string]any{{
|
||||
"id": "res-source", "handle": handle, "tenant_id": sourceTenant,
|
||||
"storage_backend_id": "backend-source", "provider": "oss",
|
||||
"physical_path": "storage://backend-source/oss://source-bucket/weknora/" +
|
||||
formatTenant(sourceTenant) + "/res-source/a.png",
|
||||
"size": int64(len(objectBytes))}}))
|
||||
put("database", "resource_bindings", 1, encode([]map[string]any{{
|
||||
"id": "bind-source", "resource_id": "res-source", "tenant_id": sourceTenant,
|
||||
"owner_type": "knowledge", "owner_id": "doc-source", "relation": "attachment"}}))
|
||||
put("resource", "res-source", 1, objectBytes)
|
||||
put("wiki", "wiki_folders", 0, encode([]map[string]any{}))
|
||||
put("wiki", "wiki_pages", 1, encode([]map[string]any{{
|
||||
"id": "wiki-source", "tenant_id": sourceTenant, "knowledge_base_id": sourceKB,
|
||||
"slug": "index", "content": "wiki"}}))
|
||||
for _, name := range []string{"wiki_page_issues", "wiki_log_entries"} {
|
||||
put("wiki", name, 0, encode([]map[string]any{}))
|
||||
}
|
||||
put("index", "chunks", 1, encode([]map[string]any{{"id": "chunk-source", "source": map[string]any{
|
||||
"tenant_id": sourceTenant, "knowledge_base_id": sourceKB, "knowledge_id": "doc-source",
|
||||
"chunk_id": "chunk-source", "source_id": "chunk-source", "content": "见 resource://" + handle,
|
||||
"embedding": []float64{0.1, 0.2}}}}))
|
||||
manifest := librarysnapshot.Manifest{ValidationPolicy: librarysnapshot.ArtifactAcceptancePolicy,
|
||||
Format: librarysnapshot.Format, Scope: "internal", TenantID: sourceTenant, KnowledgeBaseID: sourceKB,
|
||||
CreatedAt: time.Now().UTC(), Parts: parts,
|
||||
Compatibility: librarysnapshot.Compatibility{
|
||||
WeKnoraVersion: "v0.7.1", SchemaVersion: 74, RetrieverEngine: "elasticsearch_v8",
|
||||
EmbeddingModel: "text-embedding-v4", EmbeddingDimension: 1024,
|
||||
IndexMappingSHA256: strings.Repeat("a", 64),
|
||||
}}
|
||||
id, err := archive.Seal(manifest)
|
||||
if err != nil {
|
||||
t.Fatalf("seal cross-scope fixture: %v", err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func formatTenant(tenant uint64) string { return strconv.FormatUint(tenant, 10) }
|
||||
|
||||
@@ -0,0 +1,334 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// ResidueReport 是"清理范围外残留"的审计记录:清掉了什么、依据是哪份归档、
|
||||
// 备份写到哪儿、清理后目标处于什么状态。没有这份记录就不许动目标。
|
||||
type ResidueReport struct {
|
||||
SnapshotID string `json:"snapshot_id"`
|
||||
SourceTenant uint64 `json:"source_tenant"`
|
||||
SourceKB string `json:"source_kb"`
|
||||
Tables map[string]int `json:"tables"`
|
||||
BackupPath string `json:"backup_path"`
|
||||
BackupSHA256 string `json:"backup_sha256"`
|
||||
Applied bool `json:"applied"`
|
||||
Cleared int `json:"cleared"`
|
||||
Verified bool `json:"verified"`
|
||||
Foreign map[string]string `json:"foreign_scopes,omitempty"`
|
||||
}
|
||||
|
||||
// ClearForeignScope 清掉"归档里确实有、却仍以来源身份躺在目标里"的行。
|
||||
//
|
||||
// 它只做这一件事,而且必须全部条件成立才允许删:
|
||||
// 1. 目标这些行的内容与指定归档行逐字段一致(先证明它们是这次写入的副本,
|
||||
// 不是目标自己的数据);
|
||||
// 2. 先把它们完整导出成 0600 的备份文件并记下摘要;
|
||||
// 3. 删除只按"归档身份集合 + 来源租户"精确命中,不做范围广删。
|
||||
//
|
||||
// 清完还要复核:来源身份不再存在、目标内容与预备份一致、整库指纹里没有别的租户。
|
||||
func (t *ProductionLibraryTarget) ClearForeignScope(ctx context.Context, source ProductionScope, snapshotID,
|
||||
backupDir string, apply bool) (*ResidueReport, error) {
|
||||
if source.TenantID == 0 || !validSearchIdentifier(source.KnowledgeBaseID) {
|
||||
return nil, errors.New("来源范围不完整,拒绝清理")
|
||||
}
|
||||
if source.TenantID == t.scope.TenantID && source.KnowledgeBaseID == t.scope.KnowledgeBaseID {
|
||||
return nil, errors.New("来源与目标范围相同,不需要清理")
|
||||
}
|
||||
if backupDir == "" {
|
||||
return nil, errors.New("清理前必须指定备份目录")
|
||||
}
|
||||
if err := t.resolveStorageBackend(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
archive, err := librarysnapshot.Open(t.config.ArchiveRoot, 256*1024*1024)
|
||||
if err != nil {
|
||||
return nil, errors.New("归档不可用")
|
||||
}
|
||||
defer archive.Close()
|
||||
manifest, err := archive.Load(snapshotID)
|
||||
if err != nil {
|
||||
return nil, errors.New("指定快照不可读")
|
||||
}
|
||||
report := &ResidueReport{SnapshotID: snapshotID, SourceTenant: source.TenantID, SourceKB: source.KnowledgeBaseID,
|
||||
Tables: map[string]int{}, Applied: false}
|
||||
backup := map[string][]map[string]json.RawMessage{}
|
||||
digests := map[string]string{}
|
||||
identities := map[string][]string{}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" {
|
||||
continue
|
||||
}
|
||||
archived, archivedDigests, err := archivedRowsWithDigests(archive, part)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 事故里的残留行是"归档行 + 只把桶/后端换成目标"的旧变换结果:判等时必须
|
||||
// 同时接受原样与这一种变换,否则会把真残留当成"不是这次写的"而拒绝清理。
|
||||
if part.Name == "resources" {
|
||||
for _, row := range archived {
|
||||
legacy, legacyErr := t.legacyResourceRow(row)
|
||||
if legacyErr != nil {
|
||||
return nil, legacyErr
|
||||
}
|
||||
digest, digestErr := canonicalRowDigest(legacy)
|
||||
if digestErr != nil {
|
||||
return nil, digestErr
|
||||
}
|
||||
archivedDigests[digest] = true
|
||||
}
|
||||
}
|
||||
if len(archived) == 0 {
|
||||
continue
|
||||
}
|
||||
live, err := t.foreignScopeRows(ctx, part.Name, source)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
liveDigests := make(map[string]string, len(live))
|
||||
for _, row := range live {
|
||||
digest, err := canonicalRowDigest(row)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
liveDigests[digest] = archiveRowIdentity(part.Name, row)
|
||||
}
|
||||
// 目标里这一范围的行必须都能在归档里找到同样内容:否则它们不是这次写入的
|
||||
// 副本,删了就是删别人的数据。
|
||||
rows := make([]map[string]json.RawMessage, 0, len(live))
|
||||
ids := make([]string, 0, len(live))
|
||||
for digest, identity := range liveDigests {
|
||||
if _, ok := archivedDigests[digest]; !ok {
|
||||
return nil, fmt.Errorf("目标 %s 的行 %s 与指定归档不一致,拒绝按本次写入清理", part.Name, identity)
|
||||
}
|
||||
ids = append(ids, identity)
|
||||
}
|
||||
for _, row := range live {
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
report.Tables[part.Name] = len(rows)
|
||||
backup[part.Name] = rows
|
||||
identities[part.Name] = ids
|
||||
digests[part.Name] = combineRowDigests(sortedKeys(liveDigests))
|
||||
}
|
||||
total := 0
|
||||
for _, count := range report.Tables {
|
||||
total += count
|
||||
}
|
||||
if total == 0 {
|
||||
report.Verified = true
|
||||
return report, nil
|
||||
}
|
||||
if err := os.MkdirAll(backupDir, 0o700); err != nil {
|
||||
return nil, fmt.Errorf("创建备份目录失败: %w", err)
|
||||
}
|
||||
payload, err := json.MarshalIndent(map[string]any{
|
||||
"snapshot_id": snapshotID, "source_tenant": source.TenantID, "source_kb": source.KnowledgeBaseID,
|
||||
"target_tenant": t.scope.TenantID, "target_kb": t.scope.KnowledgeBaseID,
|
||||
"tables": report.Tables, "rows": backup, "digests": digests,
|
||||
}, "", " ")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
report.BackupPath = filepath.Join(backupDir, "foreign-scope-residue-"+snapshotID[:12]+".json")
|
||||
if err := os.WriteFile(report.BackupPath, payload, 0o600); err != nil {
|
||||
return nil, fmt.Errorf("写残留备份失败: %w", err)
|
||||
}
|
||||
sum := sha256.Sum256(payload)
|
||||
report.BackupSHA256 = hex.EncodeToString(sum[:])
|
||||
if !apply {
|
||||
return report, nil
|
||||
}
|
||||
// 删除顺序:子表先、父表后(resource_bindings → resources,chunks → knowledges)。
|
||||
for index := len(productionLibraryTables) - 1; index >= 0; index-- {
|
||||
table := productionLibraryTables[index]
|
||||
ids, ok := identities[table]
|
||||
if !ok || len(ids) == 0 {
|
||||
continue
|
||||
}
|
||||
statement := fmt.Sprintf(`DELETE FROM %s WHERE id = ANY($1::text[])`, table)
|
||||
if !tableHasTenant(table) {
|
||||
// knowledge_tag_relations 没有 id:身份就是父文档 + 标签。
|
||||
statement = `DELETE FROM knowledge_tag_relations WHERE (knowledge_id||':'||tag_id) = ANY($1::text[])`
|
||||
}
|
||||
if _, err := t.pool.Exec(ctx, statement, ids); err != nil {
|
||||
return nil, fmt.Errorf("清理 %s 失败: %w", table, err)
|
||||
}
|
||||
}
|
||||
report.Applied = true
|
||||
report.Cleared = total
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// legacyResourceRow 复现旧实现写资源行时的变换:只换定位符里的存储后端与桶,
|
||||
// 租户/知识库/provider 原样保留。判等要认这一种形状,事故残留才是"可证明"的。
|
||||
func (t *ProductionLibraryTarget) legacyResourceRow(row map[string]json.RawMessage) (map[string]json.RawMessage, error) {
|
||||
copied := make(map[string]json.RawMessage, len(row))
|
||||
for key, value := range row {
|
||||
copied[key] = value
|
||||
}
|
||||
if t.config.ObjectsBucket == "" {
|
||||
return copied, nil
|
||||
}
|
||||
raw, ok := copied["physical_path"]
|
||||
if !ok {
|
||||
return nil, errors.New("资源行缺少定位符")
|
||||
}
|
||||
var locator string
|
||||
if err := json.Unmarshal(raw, &locator); err != nil {
|
||||
return nil, errors.New("资源定位符不可解析")
|
||||
}
|
||||
if rewritten, ok := rewriteLocatorTarget(locator, t.storageBackend, t.config.ObjectsBucket); ok {
|
||||
encoded, err := json.Marshal(rewritten)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
copied["physical_path"] = encoded
|
||||
}
|
||||
if t.storageBackend == "" {
|
||||
return copied, nil
|
||||
}
|
||||
if raw, ok := copied["storage_backend_id"]; ok {
|
||||
var current string
|
||||
if json.Unmarshal(raw, ¤t) == nil && current != "" && current != t.storageBackend {
|
||||
encoded, err := json.Marshal(t.storageBackend)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
copied["storage_backend_id"] = encoded
|
||||
}
|
||||
}
|
||||
return copied, nil
|
||||
}
|
||||
|
||||
// VerifyResidueCleared 复核清理结果:来源身份不再存在、其它租户的解释得清、
|
||||
// 且(可选)目标内容与预备份完全一致。
|
||||
func (t *ProductionLibraryTarget) VerifyResidueCleared(ctx context.Context, source ProductionScope) error {
|
||||
for _, table := range productionLibraryTables {
|
||||
if !tableHasTenant(table) {
|
||||
continue
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1`, table)
|
||||
arguments := []any{source.TenantID}
|
||||
if tablesWithKB[table] {
|
||||
statement += " AND knowledge_base_id=$2"
|
||||
arguments = append(arguments, source.KnowledgeBaseID)
|
||||
}
|
||||
var count int64
|
||||
if err := t.pool.QueryRow(ctx, statement, arguments...).Scan(&count); err != nil {
|
||||
return fmt.Errorf("复核 %s 失败: %w", table, err)
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("目标 %s 仍有 %d 行属于来源范围 %d/%s", table, count, source.TenantID, source.KnowledgeBaseID)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ForeignScopes 列出目标里所有"不是目标租户"的租户,用于人工确认没有别的泄漏。
|
||||
func (t *ProductionLibraryTarget) ForeignScopes(ctx context.Context) (map[string]string, error) {
|
||||
foreign := map[string]string{}
|
||||
for _, table := range productionLibraryTables {
|
||||
if !tableHasTenant(table) {
|
||||
continue
|
||||
}
|
||||
rows, err := t.pool.Query(ctx, fmt.Sprintf(
|
||||
`SELECT tenant_id, count(*) FROM %s WHERE tenant_id <> $1 GROUP BY 1`, table), t.scope.TenantID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("读取 %s 租户分布失败: %w", table, err)
|
||||
}
|
||||
for rows.Next() {
|
||||
var tenant, count int64
|
||||
if err := rows.Scan(&tenant, &count); err != nil {
|
||||
rows.Close()
|
||||
return nil, err
|
||||
}
|
||||
foreign[fmt.Sprintf("%s:%d", table, tenant)] = fmt.Sprintf("%d 行", count)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return foreign, nil
|
||||
}
|
||||
|
||||
// foreignScopeRows 读出目标里属于来源范围的行(resources / resource_bindings
|
||||
// 只有租户列,按租户取;其余按租户+知识库取)。
|
||||
func (t *ProductionLibraryTarget) foreignScopeRows(ctx context.Context, table string, source ProductionScope) (
|
||||
[]map[string]json.RawMessage, error) {
|
||||
if !tableHasTenant(table) {
|
||||
return nil, nil
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE %s`, table, table, scopeWhere(table))
|
||||
arguments := []any{source.TenantID, source.KnowledgeBaseID}
|
||||
if table == "resources" || table == "resource_bindings" {
|
||||
statement = fmt.Sprintf(`SELECT to_jsonb(%s) FROM %s WHERE tenant_id=$1`, table, table)
|
||||
arguments = arguments[:1]
|
||||
}
|
||||
rows, err := t.pool.Query(ctx, statement, arguments...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("读取目标 %s 来源范围行失败: %w", table, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
result := make([]map[string]json.RawMessage, 0, 8)
|
||||
for rows.Next() {
|
||||
var row map[string]json.RawMessage
|
||||
if err := rows.Scan(&row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result = append(result, row)
|
||||
}
|
||||
return result, rows.Err()
|
||||
}
|
||||
|
||||
// archivedRowsWithDigests 读出一份归档部件的行与规范化内容摘要。
|
||||
func archivedRowsWithDigests(archive *librarysnapshot.Archive, part librarysnapshot.Part) (
|
||||
[]map[string]json.RawMessage, map[string]bool, error) {
|
||||
reader, err := archive.OpenPart(part)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
defer reader.Close()
|
||||
decoder := json.NewDecoder(reader)
|
||||
token, err := decoder.Token()
|
||||
if err != nil || token != json.Delim('[') {
|
||||
return nil, nil, errors.New("归档部件不是行数组")
|
||||
}
|
||||
rows := make([]map[string]json.RawMessage, 0, 64)
|
||||
digests := map[string]bool{}
|
||||
for decoder.More() {
|
||||
var row map[string]json.RawMessage
|
||||
if err := decoder.Decode(&row); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
digest, err := canonicalRowDigest(row)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
digests[digest] = true
|
||||
rows = append(rows, row)
|
||||
}
|
||||
return rows, digests, nil
|
||||
}
|
||||
|
||||
func sortedKeys(values map[string]string) []string {
|
||||
keys := make([]string, 0, len(values))
|
||||
for key := range values {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
@@ -0,0 +1,262 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// 整库同步是"把来源知识库的内容搬到目标的租户/知识库",但归档行带着来源身份
|
||||
// (tenant_id / knowledge_base_id)。原样插入会让行落在来源范围里:目标范围的
|
||||
// 删除、校验、预备份与恢复全都看不见它们(真实事故:同步失败后"已恢复到预备份",
|
||||
// 目标却多出 4 文件 / 38 分块 / 21 资源 / 4 绑定,行所属租户还是来源租户)。
|
||||
//
|
||||
// 因此写入与校验必须共用这一个改写入口,并且对不上就失败——宁可不写,也不写
|
||||
// 一行目标范围外的数据。
|
||||
|
||||
// tablesWithKB 是真实 WeKnora v0.7.1 里带 knowledge_base_id 的知识域表。
|
||||
// resources / resource_bindings 是多知识库共享表,只有 tenant_id;
|
||||
// knowledge_tag_relations 两列都没有,归属完全由父表 knowledges 决定。
|
||||
var tablesWithKB = map[string]bool{
|
||||
"knowledges": true, "chunks": true, "knowledge_tags": true,
|
||||
"wiki_folders": true, "wiki_pages": true, "wiki_page_issues": true, "wiki_log_entries": true,
|
||||
}
|
||||
|
||||
// tableHasTenant 报告这张表是否自带租户列。
|
||||
func tableHasTenant(table string) bool { return table != "knowledge_tag_relations" }
|
||||
|
||||
// bindSnapshotScope 记下快照的来源身份。没有来源身份就无法判断"这行要搬到哪个
|
||||
// 范围",宁可失败也不要按行内容猜。
|
||||
func (t *ProductionLibraryTarget) bindSnapshotScope(manifest manifestScope) error {
|
||||
if manifest.tenantID == 0 || !validSearchIdentifier(manifest.knowledgeBaseID) {
|
||||
return errors.New("快照缺少来源范围身份(tenant_id / knowledge_base_id),无法确定目标范围映射")
|
||||
}
|
||||
t.snapshotTenant = manifest.tenantID
|
||||
t.snapshotKB = manifest.knowledgeBaseID
|
||||
return nil
|
||||
}
|
||||
|
||||
// manifestScope 只是把快照的两个范围字段抽出来,避免调用点重复解构。
|
||||
type manifestScope struct {
|
||||
tenantID uint64
|
||||
knowledgeBaseID string
|
||||
}
|
||||
|
||||
// retargetRow 把一行归档数据改写成目标范围的数据:tenant_id / knowledge_base_id
|
||||
// 换成目标身份,资源行还要把定位符、存储后端、提供方换成目标实例自己的值。
|
||||
func (t *ProductionLibraryTarget) retargetRow(table string, row map[string]json.RawMessage) error {
|
||||
if !tableHasTenant(table) {
|
||||
return nil
|
||||
}
|
||||
if err := t.retargetTenantField(row); err != nil {
|
||||
return fmt.Errorf("%s:%w", table, err)
|
||||
}
|
||||
if tablesWithKB[table] {
|
||||
if err := t.retargetKnowledgeBaseField(row); err != nil {
|
||||
return fmt.Errorf("%s:%w", table, err)
|
||||
}
|
||||
}
|
||||
if table == "resources" {
|
||||
return t.rewriteResourceIdentity(row)
|
||||
}
|
||||
if table == "knowledges" {
|
||||
return t.retargetEmbeddingModelField(row)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// retargetEmbeddingModelField 把文档的 embedding 模型换成目标知识库绑定的那一个。
|
||||
// 来源模型 id 在目标实例里通常不存在(目标 models 表可能是空的),留着它等于让
|
||||
// 应用拿一个解析不到的模型去查——库里的向量与查询向量就不是同一套了。
|
||||
func (t *ProductionLibraryTarget) retargetEmbeddingModelField(row map[string]json.RawMessage) error {
|
||||
if t.targetEmbeddingModel == "" {
|
||||
return nil
|
||||
}
|
||||
if _, ok := row["embedding_model_id"]; !ok {
|
||||
return nil
|
||||
}
|
||||
encoded, err := json.Marshal(t.targetEmbeddingModel)
|
||||
if err != nil {
|
||||
return errors.New("编码目标 embedding 模型失败")
|
||||
}
|
||||
row["embedding_model_id"] = encoded
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveEmbeddingModel 读目标知识库自己的 embedding 模型 id。读不到不算错误
|
||||
// (目标可能确实没绑模型),此时保持来源值不动,由契约校验去发现不一致。
|
||||
func (t *ProductionLibraryTarget) resolveEmbeddingModel(ctx context.Context) {
|
||||
var model string
|
||||
if err := t.pool.QueryRow(ctx,
|
||||
`SELECT coalesce(embedding_model_id,'') FROM knowledge_bases WHERE id=$1 AND tenant_id=$2`,
|
||||
t.scope.KnowledgeBaseID, t.scope.TenantID).Scan(&model); err == nil {
|
||||
t.targetEmbeddingModel = model
|
||||
}
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) retargetTenantField(row map[string]json.RawMessage) error {
|
||||
raw, ok := row["tenant_id"]
|
||||
if !ok {
|
||||
return errors.New("行缺少 tenant_id,无法确定要写入哪个范围")
|
||||
}
|
||||
var current uint64
|
||||
if err := json.Unmarshal(raw, ¤t); err != nil {
|
||||
return errors.New("行 tenant_id 不可解析")
|
||||
}
|
||||
if t.snapshotTenant == 0 {
|
||||
return errors.New("快照来源租户未知,拒绝猜测目标租户")
|
||||
}
|
||||
if current != t.snapshotTenant {
|
||||
return fmt.Errorf("行 tenant_id=%d 不属于快照来源租户 %d", current, t.snapshotTenant)
|
||||
}
|
||||
encoded, err := json.Marshal(t.scope.TenantID)
|
||||
if err != nil {
|
||||
return errors.New("编码目标租户失败")
|
||||
}
|
||||
row["tenant_id"] = encoded
|
||||
return nil
|
||||
}
|
||||
|
||||
func (t *ProductionLibraryTarget) retargetKnowledgeBaseField(row map[string]json.RawMessage) error {
|
||||
raw, ok := row["knowledge_base_id"]
|
||||
if !ok {
|
||||
return errors.New("行缺少 knowledge_base_id,无法确定要写入哪个知识库")
|
||||
}
|
||||
var current string
|
||||
if err := json.Unmarshal(raw, ¤t); err != nil || current == "" {
|
||||
return errors.New("行 knowledge_base_id 不可解析")
|
||||
}
|
||||
if t.snapshotKB == "" {
|
||||
return errors.New("快照来源知识库未知,拒绝猜测目标知识库")
|
||||
}
|
||||
if current != t.snapshotKB {
|
||||
return fmt.Errorf("行 knowledge_base_id=%s 不属于快照来源知识库 %s", current, t.snapshotKB)
|
||||
}
|
||||
encoded, err := json.Marshal(t.scope.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return errors.New("编码目标知识库失败")
|
||||
}
|
||||
row["knowledge_base_id"] = encoded
|
||||
return nil
|
||||
}
|
||||
|
||||
// rewriteResourceIdentity 把资源行的定位符与提供方换成目标实例的值。
|
||||
//
|
||||
// 只换桶不够:来源行的 scheme(oss/minio…)与对象键里的租户段也是来源实例的,
|
||||
// 目标实例按它取文件会落到别的提供方或别的租户前缀上(真跑:storage backend
|
||||
// not found → invalid path → 500 code1007)。
|
||||
func (t *ProductionLibraryTarget) rewriteResourceIdentity(row map[string]json.RawMessage) error {
|
||||
raw, ok := row["physical_path"]
|
||||
if !ok {
|
||||
return errors.New("资源行缺少定位符")
|
||||
}
|
||||
var locator string
|
||||
if err := json.Unmarshal(raw, &locator); err != nil {
|
||||
return errors.New("资源定位符不可解析")
|
||||
}
|
||||
rewritten, err := retargetLocator(locator, t.storageBackend, t.storageProvider, t.config.ObjectsBucket,
|
||||
t.snapshotTenant, t.scope.TenantID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if rewritten != locator {
|
||||
encoded, err := json.Marshal(rewritten)
|
||||
if err != nil {
|
||||
return errors.New("编码目标定位符失败")
|
||||
}
|
||||
row["physical_path"] = encoded
|
||||
}
|
||||
if t.storageBackend != "" {
|
||||
if _, ok := row["storage_backend_id"]; ok {
|
||||
encoded, err := json.Marshal(t.storageBackend)
|
||||
if err != nil {
|
||||
return errors.New("编码目标存储后端失败")
|
||||
}
|
||||
row["storage_backend_id"] = encoded
|
||||
}
|
||||
}
|
||||
if t.storageProvider != "" {
|
||||
if _, ok := row["provider"]; ok {
|
||||
encoded, err := json.Marshal(t.storageProvider)
|
||||
if err != nil {
|
||||
return errors.New("编码目标提供方失败")
|
||||
}
|
||||
row["provider"] = encoded
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// retargetLocator 把 "storage://<backend>/<scheme>://<bucket>/<key>" 整体换成目标
|
||||
// 实例的真实取值:后端 id、scheme/提供方、桶名,以及对象键里的租户段。
|
||||
// 桶名为空表示目标没有配置对象存储(测试夹具),此时不动定位符。
|
||||
func retargetLocator(locator, backendID, provider, bucket string, sourceTenant, targetTenant uint64) (string, error) {
|
||||
if bucket == "" || locator == "" {
|
||||
return locator, nil
|
||||
}
|
||||
if locator != strings.TrimSpace(locator) {
|
||||
return "", errors.New("资源定位符带首尾空白")
|
||||
}
|
||||
prefix := ""
|
||||
rest := locator
|
||||
if strings.HasPrefix(locator, "storage://") {
|
||||
trimmed := strings.TrimPrefix(locator, "storage://")
|
||||
parts := strings.SplitN(trimmed, "/", 2)
|
||||
if len(parts) != 2 || !validSearchIdentifier(parts[0]) || parts[1] == "" {
|
||||
return "", errors.New("资源定位符结构不可解析")
|
||||
}
|
||||
if backendID != "" {
|
||||
if !validSearchIdentifier(backendID) {
|
||||
return "", errors.New("目标存储后端 id 形状不合法")
|
||||
}
|
||||
parts[0] = backendID
|
||||
}
|
||||
prefix, rest = "storage://"+parts[0]+"/", parts[1]
|
||||
}
|
||||
separator := strings.Index(rest, "://")
|
||||
if separator < 0 {
|
||||
// 裸对象键没有桶与租户段:落盘按配置桶,改写无意义。
|
||||
return locator, nil
|
||||
}
|
||||
scheme := rest[:separator]
|
||||
if !validSearchIdentifier(scheme) {
|
||||
return "", errors.New("资源定位符的提供方不可解析")
|
||||
}
|
||||
schemeless := rest[separator+3:]
|
||||
boundary := strings.Index(schemeless, "/")
|
||||
if boundary <= 0 || boundary == len(schemeless)-1 {
|
||||
return "", errors.New("资源定位符缺少对象键")
|
||||
}
|
||||
key := schemeless[boundary+1:]
|
||||
if provider != "" {
|
||||
if !validSearchIdentifier(provider) {
|
||||
return "", errors.New("目标提供方形状不合法")
|
||||
}
|
||||
scheme = provider
|
||||
}
|
||||
return prefix + scheme + "://" + bucket + "/" + retargetKeyTenant(key, sourceTenant, targetTenant), nil
|
||||
}
|
||||
|
||||
// retargetKeyTenant 把对象键里的来源租户段换成目标租户,并丢掉来源实例自己的
|
||||
// 路径前缀(例如 MINIO_PATH_PREFIX=weknora/),让键与目标实例的写法一致。
|
||||
// 找不到来源租户段时原样返回:此时键本身不含跨租户引用。
|
||||
func retargetKeyTenant(key string, sourceTenant, targetTenant uint64) string {
|
||||
if sourceTenant == 0 || targetTenant == 0 || sourceTenant == targetTenant {
|
||||
return key
|
||||
}
|
||||
source, target := strconv.FormatUint(sourceTenant, 10), strconv.FormatUint(targetTenant, 10)
|
||||
segments := strings.Split(key, "/")
|
||||
for index := 0; index < len(segments) && index < 2; index++ {
|
||||
// 只有"后面还有内容"的段才是路径前缀;整键就是租户号时不该动它。
|
||||
if index >= len(segments)-1 || segments[index] != source {
|
||||
continue
|
||||
}
|
||||
kept := segments[index:]
|
||||
kept[0] = target
|
||||
return strings.Join(kept, "/")
|
||||
}
|
||||
return key
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 真实事故的形状:来源 KB 与目标 KB 不同(租户也不同),归档行原样写进目标,
|
||||
// 目标范围的删除/校验/恢复都看不到它们。这里把"必须改写哪些列"钉死。
|
||||
|
||||
func scopeTargetFixture() *ProductionLibraryTarget {
|
||||
return &ProductionLibraryTarget{
|
||||
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"},
|
||||
config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
|
||||
storageBackend: "backend-target",
|
||||
storageProvider: "minio",
|
||||
snapshotTenant: 10000,
|
||||
snapshotKB: "kb-source",
|
||||
}
|
||||
}
|
||||
|
||||
func rawRow(t *testing.T, fields map[string]any) map[string]json.RawMessage {
|
||||
t.Helper()
|
||||
row := make(map[string]json.RawMessage, len(fields))
|
||||
for key, value := range fields {
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatalf("encode %s: %v", key, err)
|
||||
}
|
||||
row[key] = encoded
|
||||
}
|
||||
return row
|
||||
}
|
||||
|
||||
func fieldText(t *testing.T, row map[string]json.RawMessage, key string) string {
|
||||
t.Helper()
|
||||
var value string
|
||||
if err := json.Unmarshal(row[key], &value); err != nil {
|
||||
t.Fatalf("%s not a string: %v", key, err)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func fieldUint(t *testing.T, row map[string]json.RawMessage, key string) uint64 {
|
||||
t.Helper()
|
||||
var value uint64
|
||||
if err := json.Unmarshal(row[key], &value); err != nil {
|
||||
t.Fatalf("%s not a number: %v", key, err)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func TestRetargetRowMapsSourceScopeToTarget(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
for _, table := range []string{"knowledges", "chunks", "knowledge_tags", "wiki_pages", "wiki_folders"} {
|
||||
row := rawRow(t, map[string]any{"id": "row-1", "tenant_id": 10000, "knowledge_base_id": "kb-source"})
|
||||
if err := target.retargetRow(table, row); err != nil {
|
||||
t.Fatalf("%s retarget: %v", table, err)
|
||||
}
|
||||
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
|
||||
t.Fatalf("%s tenant_id = %d, want 10001", table, got)
|
||||
}
|
||||
if got := fieldText(t, row, "knowledge_base_id"); got != "kb-target" {
|
||||
t.Fatalf("%s knowledge_base_id = %s, want kb-target", table, got)
|
||||
}
|
||||
}
|
||||
// 共享表只有租户列:不得凭空写一个 knowledge_base_id 出来。
|
||||
binding := rawRow(t, map[string]any{"id": "bind-1", "tenant_id": 10000, "owner_type": "knowledge", "owner_id": "row-1"})
|
||||
if err := target.retargetRow("resource_bindings", binding); err != nil {
|
||||
t.Fatalf("resource_bindings retarget: %v", err)
|
||||
}
|
||||
if got := fieldUint(t, binding, "tenant_id"); got != 10001 {
|
||||
t.Fatalf("binding tenant_id = %d, want 10001", got)
|
||||
}
|
||||
if _, ok := binding["knowledge_base_id"]; ok {
|
||||
t.Fatal("resource_bindings must not gain a knowledge_base_id column")
|
||||
}
|
||||
// knowledge_tag_relations 没有范围列:归属走父表,改写必须是空操作。
|
||||
relation := rawRow(t, map[string]any{"knowledge_id": "row-1", "tag_id": "tag-1"})
|
||||
if err := target.retargetRow("knowledge_tag_relations", relation); err != nil {
|
||||
t.Fatalf("knowledge_tag_relations retarget: %v", err)
|
||||
}
|
||||
if len(relation) != 2 {
|
||||
t.Fatalf("knowledge_tag_relations must stay untouched, got %v", relation)
|
||||
}
|
||||
}
|
||||
|
||||
// 文档行的 embedding 模型必须换成目标知识库绑定的那一个:来源模型 id 在目标
|
||||
// 实例里通常不存在,留着它检索时的查询向量与库里向量就不是同一套。
|
||||
func TestRetargetRowUsesTargetEmbeddingModel(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
target.targetEmbeddingModel = "model-target"
|
||||
row := rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
|
||||
"embedding_model_id": "model-source"})
|
||||
if err := target.retargetRow("knowledges", row); err != nil {
|
||||
t.Fatalf("retarget: %v", err)
|
||||
}
|
||||
if got := fieldText(t, row, "embedding_model_id"); got != "model-target" {
|
||||
t.Fatalf("embedding_model_id = %s, want model-target", got)
|
||||
}
|
||||
// 目标没绑模型时不猜:保持来源值,交给契约校验去发现不一致。
|
||||
plain := scopeTargetFixture()
|
||||
row = rawRow(t, map[string]any{"id": "doc-1", "tenant_id": 10000, "knowledge_base_id": "kb-source",
|
||||
"embedding_model_id": "model-source"})
|
||||
if err := plain.retargetRow("knowledges", row); err != nil {
|
||||
t.Fatalf("retarget without target model: %v", err)
|
||||
}
|
||||
if got := fieldText(t, row, "embedding_model_id"); got != "model-source" {
|
||||
t.Fatalf("embedding_model_id = %s, want unchanged", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetRowFailsClosedOutsideSnapshotScope(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
cases := map[string]map[string]any{
|
||||
"foreign tenant": {"id": "x", "tenant_id": 20000, "knowledge_base_id": "kb-source"},
|
||||
"foreign knowledge": {"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb-other"},
|
||||
"missing tenant": {"id": "x", "knowledge_base_id": "kb-source"},
|
||||
"missing knowledge kb": {"id": "x", "tenant_id": 10000},
|
||||
}
|
||||
for name, fields := range cases {
|
||||
if err := target.retargetRow("knowledges", rawRow(t, fields)); err == nil {
|
||||
t.Fatalf("%s: retarget must fail closed", name)
|
||||
}
|
||||
}
|
||||
// 快照来源未知时不许猜目标范围。
|
||||
unknown := &ProductionLibraryTarget{scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb-target"}}
|
||||
if err := unknown.retargetRow("knowledges", rawRow(t, map[string]any{"id": "x", "tenant_id": 10000, "knowledge_base_id": "kb"})); err == nil {
|
||||
t.Fatal("unknown snapshot scope must fail closed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetRowRewritesResourceStorageIdentity(t *testing.T) {
|
||||
target := scopeTargetFixture()
|
||||
row := rawRow(t, map[string]any{
|
||||
"id": "res-1",
|
||||
"tenant_id": 10000,
|
||||
"storage_backend_id": "backend-source",
|
||||
"provider": "oss",
|
||||
"physical_path": "storage://backend-source/oss://yzh-weknora-test/weknora/10000/res-1/a.png",
|
||||
})
|
||||
if err := target.retargetRow("resources", row); err != nil {
|
||||
t.Fatalf("retarget resource: %v", err)
|
||||
}
|
||||
if got := fieldUint(t, row, "tenant_id"); got != 10001 {
|
||||
t.Fatalf("resource tenant = %d, want 10001", got)
|
||||
}
|
||||
want := "storage://backend-target/minio://kgseal-1029a/10001/res-1/a.png"
|
||||
if got := fieldText(t, row, "physical_path"); got != want {
|
||||
t.Fatalf("physical_path = %q, want %q", got, want)
|
||||
}
|
||||
if got := fieldText(t, row, "provider"); got != "minio" {
|
||||
t.Fatalf("provider = %q, want minio", got)
|
||||
}
|
||||
if got := fieldText(t, row, "storage_backend_id"); got != "backend-target" {
|
||||
t.Fatalf("storage_backend_id = %q, want backend-target", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetKeyTenantDropsSourcePrefix(t *testing.T) {
|
||||
cases := []struct{ key, want string }{
|
||||
{"weknora/10000/exports/a.png", "10001/exports/a.png"},
|
||||
{"10000/knowledge/file.md", "10001/knowledge/file.md"},
|
||||
{"other/prefix/a.png", "other/prefix/a.png"},
|
||||
{"10000", "10000"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := retargetKeyTenant(tc.key, 10000, 10001); got != tc.want {
|
||||
t.Fatalf("retargetKeyTenant(%q) = %q, want %q", tc.key, got, tc.want)
|
||||
}
|
||||
}
|
||||
// 同租户不编码:键必须原样返回。
|
||||
if got := retargetKeyTenant("weknora/10000/a.png", 10000, 10000); got != "weknora/10000/a.png" {
|
||||
t.Fatalf("same tenant must keep key, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetargetLocatorRejectsMalformed(t *testing.T) {
|
||||
if _, err := retargetLocator(" not-a-locator ", "backend", "minio", "bucket", 10000, 10001); err == nil {
|
||||
t.Fatal("locator with surrounding spaces must be rejected")
|
||||
}
|
||||
if _, err := retargetLocator("storage://backend/oss://", "backend", "minio", "bucket", 10000, 10001); err == nil {
|
||||
t.Fatal("locator without object key must be rejected")
|
||||
}
|
||||
// 没有目标桶(测试夹具)时不改写,保持原值可继续用于采集/校验。
|
||||
locator := "storage://backend/oss://bucket/key"
|
||||
if got, err := retargetLocator(locator, "backend", "minio", "", 10000, 10001); err != nil || got != locator {
|
||||
t.Fatalf("empty bucket must be a no-op, got %q err=%v", got, err)
|
||||
}
|
||||
if !strings.Contains(retargetKeyTenant("weknora/10000/a", 10000, 10001), "10001") {
|
||||
t.Fatal("tenant segment must be replaced")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package v071
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"knowledge-governance/internal/libraryrelease"
|
||||
"knowledge-governance/internal/librarysnapshot"
|
||||
)
|
||||
|
||||
// universeFingerprint 是"整实例"的知识域指纹:不限租户/知识库的行数 + 身份集合摘要。
|
||||
//
|
||||
// 为什么不能只比目标范围:恢复只按目标范围删除/回放,写到范围外的残留它看不见
|
||||
// (真实事故:作业报"已恢复到预备份",目标却多出 4 文件 / 38 分块 / 21 资源 /
|
||||
// 4 绑定)。整实例指纹让"多写了什么"必然暴露,同时也能证明恢复没有误删别的
|
||||
// 知识库的行。
|
||||
func universeFingerprint(ctx context.Context, pool *pgxpool.Pool) (map[string]libraryrelease.TableFingerprint, error) {
|
||||
result := make(map[string]libraryrelease.TableFingerprint, len(productionLibraryTables))
|
||||
for _, table := range productionLibraryTables {
|
||||
identity := identityExpr(table)
|
||||
statement := fmt.Sprintf(
|
||||
`SELECT count(*), coalesce(md5(string_agg((%s)::text, E'\n' ORDER BY (%s)::text)), '') FROM %s`,
|
||||
identity, identity, table)
|
||||
var fingerprint libraryrelease.TableFingerprint
|
||||
if err := pool.QueryRow(ctx, statement).Scan(&fingerprint.Count, &fingerprint.Digest); err != nil {
|
||||
return nil, fmt.Errorf("读取整库 %s 指纹失败: %w", table, err)
|
||||
}
|
||||
result[table] = fingerprint
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// archivedRowIdentities 读出一份归档部件的行身份(与校验用的身份口径一致)。
|
||||
func archivedRowIdentities(archive *librarysnapshot.Archive, part librarysnapshot.Part) ([]string, error) {
|
||||
reader, err := archive.OpenPart(part)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer reader.Close()
|
||||
decoder := json.NewDecoder(reader)
|
||||
token, err := decoder.Token()
|
||||
if err != nil || token != json.Delim('[') {
|
||||
return nil, errors.New("归档部件不是行数组")
|
||||
}
|
||||
ids := make([]string, 0, 64)
|
||||
for decoder.More() {
|
||||
var row map[string]json.RawMessage
|
||||
if err := decoder.Decode(&row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identity := archiveRowIdentity(part.Name, row)
|
||||
if identity == "" {
|
||||
continue
|
||||
}
|
||||
ids = append(ids, identity)
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
// universeDiff 返回第一处不一致的表;空字符串表示整实例与参照完全一致。
|
||||
func universeDiff(want, got map[string]libraryrelease.TableFingerprint) string {
|
||||
tables := make([]string, 0, len(want))
|
||||
for table := range want {
|
||||
tables = append(tables, table)
|
||||
}
|
||||
sort.Strings(tables)
|
||||
for _, table := range tables {
|
||||
current, ok := got[table]
|
||||
if !ok {
|
||||
return fmt.Sprintf("%s(目标读不到)", table)
|
||||
}
|
||||
if current.Count != want[table].Count {
|
||||
return fmt.Sprintf("%s 行数 %d,参照 %d", table, current.Count, want[table].Count)
|
||||
}
|
||||
if current.Digest != want[table].Digest {
|
||||
return fmt.Sprintf("%s 身份集合与参照不同(同数量改写也会被这条拦住)", table)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// assertNoSourceScopeRows 找出"这次要写的这些行,仍然以来源身份躺在目标里"。
|
||||
//
|
||||
// 判据按归档里的身份集合逐表核对,而不是"这个租户/知识库有没有行":同一个租户
|
||||
// 里本来就可能住着别的知识库(共享资源、别的 KB 的绑定),按范围数会把它们误
|
||||
// 报成残留。这里只认"本次写集合里、仍属于来源租户"的行——它们只可能来自逃出
|
||||
// 目标范围的写入,而恢复只按目标范围回放,永远删不到它们。
|
||||
func (t *ProductionLibraryTarget) assertNoSourceScopeRows(ctx context.Context, archive *librarysnapshot.Archive,
|
||||
manifest librarysnapshot.Manifest) error {
|
||||
if t.snapshotTenant == 0 {
|
||||
return nil
|
||||
}
|
||||
if t.snapshotTenant == t.scope.TenantID && t.snapshotKB == t.scope.KnowledgeBaseID {
|
||||
return nil
|
||||
}
|
||||
for _, part := range manifest.Parts {
|
||||
if part.Kind != "database" && part.Kind != "wiki" {
|
||||
continue
|
||||
}
|
||||
if !tableHasTenant(part.Name) {
|
||||
continue
|
||||
}
|
||||
ids, err := archivedRowIdentities(archive, part)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
continue
|
||||
}
|
||||
statement := fmt.Sprintf(`SELECT count(*) FROM %s WHERE tenant_id=$1 AND id = ANY($2::text[])`, part.Name)
|
||||
var count int64
|
||||
if err := t.pool.QueryRow(ctx, statement, t.snapshotTenant, ids).Scan(&count); err != nil {
|
||||
return fmt.Errorf("检查目标 %s 的来源范围残留失败: %w", part.Name, err)
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("目标 %s 残留 %d 行仍属于来源范围 %d/%s:有写入逃出了目标范围,恢复不会清掉它们",
|
||||
part.Name, count, t.snapshotTenant, t.snapshotKB)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -60,8 +60,11 @@ func TestRewriteLocatorSwapsStorageBackendWithBucket(t *testing.T) {
|
||||
// 行变换用解析到的目标后端与目标桶:写入与校验共用这一份,摘要才不会两边打架。
|
||||
func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
|
||||
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
|
||||
storageBackend: "target-backend"}
|
||||
row := map[string]json.RawMessage{"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
|
||||
storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
|
||||
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
|
||||
row := map[string]json.RawMessage{
|
||||
"tenant_id": json.RawMessage(`10001`),
|
||||
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/10001/k/ab.md"`)}
|
||||
if err := target.rowTransform("resources")(row); err != nil {
|
||||
t.Fatalf("row transform failed: %v", err)
|
||||
}
|
||||
@@ -78,8 +81,10 @@ func TestResourceRowTransformRewritesBackendAndBucket(t *testing.T) {
|
||||
// 只改定位符不改这一列,目标仍是 storage backend not found(真跑 500)。
|
||||
func TestResourceRowTransformRewritesBackendColumn(t *testing.T) {
|
||||
target := &ProductionLibraryTarget{config: ProductionLibraryConfig{ObjectsBucket: "kgseal-1029a"},
|
||||
storageBackend: "target-backend"}
|
||||
storageBackend: "target-backend", snapshotTenant: 10001, snapshotKB: "kb",
|
||||
scope: ProductionScope{TenantID: 10001, KnowledgeBaseID: "kb"}}
|
||||
row := map[string]json.RawMessage{
|
||||
"tenant_id": json.RawMessage(`10001`),
|
||||
"physical_path": json.RawMessage(`"storage://source-backend/minio://kgmaint-1028a/1/k/ab.md"`),
|
||||
"storage_backend_id": json.RawMessage(`"source-backend"`),
|
||||
}
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
// Package sealedsecret holds the one AES-GCM convention this platform uses to
|
||||
// keep a write-only credential (model key, WeKnora key) inside a database row
|
||||
// without ever storing the plaintext or a reversible copy elsewhere.
|
||||
//
|
||||
// Two rules make it safe to have only one implementation:
|
||||
// - the master key is an independently provisioned 32-byte 0600 file, never
|
||||
// in source, settings JSON or database backups;
|
||||
// - every ciphertext is bound to a caller-supplied AAD (tenant + revision),
|
||||
// so a sealed value cannot be replayed onto a different row or revision.
|
||||
package sealedsecret
|
||||
|
||||
import (
|
||||
"crypto/aes"
|
||||
"crypto/cipher"
|
||||
"crypto/rand"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
)
|
||||
|
||||
// ErrUnavailable means the master key is not usable, so sealing and opening are
|
||||
// both impossible. Callers degrade honestly (they must not pretend it works).
|
||||
var ErrUnavailable = errors.New("sealed secret storage is unavailable")
|
||||
|
||||
// KeySize is the only accepted master-key length (AES-256).
|
||||
const KeySize = 32
|
||||
|
||||
// LoadKeyFile reads the master key. A missing file, a non-regular file, group or
|
||||
// other permissions, or a wrong length are all ErrUnavailable: a half-configured
|
||||
// secret store must never be silently substituted by a weaker one.
|
||||
func LoadKeyFile(path string) ([]byte, error) {
|
||||
if path == "" {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm() != 0o600 {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil || len(raw) != KeySize {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return raw, nil
|
||||
}
|
||||
|
||||
func aead(key []byte) (cipher.AEAD, error) {
|
||||
if len(key) != KeySize {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
block, err := aes.NewCipher(key)
|
||||
if err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return cipher.NewGCM(block)
|
||||
}
|
||||
|
||||
// Seal returns nonce||ciphertext bound to aad.
|
||||
func Seal(key []byte, plaintext, aad string) ([]byte, error) {
|
||||
sealer, err := aead(key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
nonce := make([]byte, sealer.NonceSize())
|
||||
if _, err = rand.Read(nonce); err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return sealer.Seal(nonce, nonce, []byte(plaintext), []byte(aad)), nil
|
||||
}
|
||||
|
||||
// Open reverses Seal. A different aad, a truncated value or a rotated key all
|
||||
// return ErrUnavailable rather than an empty credential.
|
||||
func Open(key, sealed []byte, aad string) (string, error) {
|
||||
opener, err := aead(key)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(sealed) < opener.NonceSize() {
|
||||
return "", ErrUnavailable
|
||||
}
|
||||
raw, err := opener.Open(nil, sealed[:opener.NonceSize()], sealed[opener.NonceSize():], []byte(aad))
|
||||
if err != nil {
|
||||
return "", ErrUnavailable
|
||||
}
|
||||
return string(raw), nil
|
||||
}
|
||||
|
||||
// AAD builds the canonical associated data for one row revision. Every caller
|
||||
// goes through it so two features cannot accidentally share a binding domain.
|
||||
func AAD(scope, subject string, revision int64) string {
|
||||
return fmt.Sprintf("%s:%s:%d", scope, subject, revision)
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package sealedsecret
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func writeKey(t *testing.T, content []byte, mode os.FileMode) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "master.key")
|
||||
if err := os.WriteFile(path, content, mode); err != nil {
|
||||
t.Fatalf("write key file: %v", err)
|
||||
}
|
||||
if err := os.Chmod(path, mode); err != nil {
|
||||
t.Fatalf("chmod key file: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestSealOpenRoundTripBindsAAD(t *testing.T) {
|
||||
key := bytes.Repeat([]byte{0x7a}, KeySize)
|
||||
sealed, err := Seal(key, "sk-live-1234", AAD("weknora-connection", "test", 3))
|
||||
if err != nil {
|
||||
t.Fatalf("Seal() error = %v", err)
|
||||
}
|
||||
if bytes.Contains(sealed, []byte("sk-live-1234")) {
|
||||
t.Fatal("ciphertext must not contain the plaintext")
|
||||
}
|
||||
plain, err := Open(key, sealed, AAD("weknora-connection", "test", 3))
|
||||
if err != nil || plain != "sk-live-1234" {
|
||||
t.Fatalf("Open() = %q, %v", plain, err)
|
||||
}
|
||||
if _, err := Open(key, sealed, AAD("weknora-connection", "test", 4)); err == nil {
|
||||
t.Fatal("a different revision must not open the same ciphertext")
|
||||
}
|
||||
if _, err := Open(key, sealed, AAD("weknora-connection", "production", 3)); err == nil {
|
||||
t.Fatal("a different environment must not open the same ciphertext")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenRejectsTruncatedAndRotatedKey(t *testing.T) {
|
||||
key := bytes.Repeat([]byte{0x11}, KeySize)
|
||||
sealed, err := Seal(key, "secret", AAD("qa-ai", "tenant", 1))
|
||||
if err != nil {
|
||||
t.Fatalf("Seal() error = %v", err)
|
||||
}
|
||||
if _, err := Open(key, sealed[:4], AAD("qa-ai", "tenant", 1)); err == nil {
|
||||
t.Fatal("truncated ciphertext must not open")
|
||||
}
|
||||
rotated := bytes.Repeat([]byte{0x22}, KeySize)
|
||||
if _, err := Open(rotated, sealed, AAD("qa-ai", "tenant", 1)); err == nil {
|
||||
t.Fatal("a rotated master key must not open old ciphertext")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadKeyFileRequirements(t *testing.T) {
|
||||
good := writeKey(t, bytes.Repeat([]byte{0x01}, KeySize), 0o600)
|
||||
if _, err := LoadKeyFile(good); err != nil {
|
||||
t.Fatalf("LoadKeyFile(0600, 32B) error = %v", err)
|
||||
}
|
||||
cases := map[string]string{
|
||||
"missing": "",
|
||||
"short": writeKey(t, bytes.Repeat([]byte{0x01}, 16), 0o600),
|
||||
"group-visible": writeKey(t, bytes.Repeat([]byte{0x01}, KeySize), 0o640),
|
||||
}
|
||||
for name, path := range cases {
|
||||
if _, err := LoadKeyFile(path); err != ErrUnavailable {
|
||||
t.Fatalf("%s: LoadKeyFile() error = %v, want ErrUnavailable", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,17 @@ import (
|
||||
// the operator to paste an id. A missing endpoint or credential is an error the
|
||||
// caller reports honestly rather than a silently empty picker.
|
||||
func (verifier *Verifier) ListProductionKnowledgeBases(ctx context.Context, baseURL string) ([]weknora.KnowledgeBase, error) {
|
||||
// The admin-managed connection wins: the page is where operators record the
|
||||
// endpoint and key, and a second (file) source would silently disagree.
|
||||
if verifier.config.ConnectionResolver != nil {
|
||||
if endpoint, apiKey, ok := verifier.config.ConnectionResolver.Resolve(ctx, "production"); ok {
|
||||
client, err := verifier.clientWithPolicy(endpoint, apiKey, verifier.config.ProductionAllowDockerHTTP, verifier.config.ProductionAllowPrivateHTTP)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return client.ListKnowledgeBases(ctx)
|
||||
}
|
||||
}
|
||||
resolved := firstNonEmpty(baseURL, verifier.config.ProductionBaseURLFallback)
|
||||
if resolved == "" {
|
||||
return nil, errors.New("production WeKnora endpoint is not configured")
|
||||
@@ -21,7 +32,7 @@ func (verifier *Verifier) ListProductionKnowledgeBases(ctx context.Context, base
|
||||
if apiKey == "" {
|
||||
return nil, errors.New("production WeKnora API key file is not available")
|
||||
}
|
||||
client, err := verifier.client(resolved, apiKey, verifier.config.ProductionAllowDockerHTTP)
|
||||
client, err := verifier.clientWithPolicy(resolved, apiKey, verifier.config.ProductionAllowDockerHTTP, verifier.config.ProductionAllowPrivateHTTP)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -40,8 +40,19 @@ type Config struct {
|
||||
ProductionBaseURLFallback string
|
||||
TestAllowDockerHTTP bool
|
||||
ProductionAllowDockerHTTP bool
|
||||
Timeout time.Duration
|
||||
Now func() time.Time
|
||||
// ProductionAllowPrivateHTTP mirrors WEKNORA_ALLOW_PRIVATE_HTTP: the
|
||||
// production instance is reachable only as a same-LAN http endpoint.
|
||||
ProductionAllowPrivateHTTP bool
|
||||
TestAllowPrivateHTTP bool
|
||||
// ConnectionResolver reads the admin-managed connection (endpoint + key)
|
||||
// from the governance database. When it answers, it wins over the env
|
||||
// defaults and over the credential file — the page is the place operators
|
||||
// configure these, and a second source would drift.
|
||||
ConnectionResolver interface {
|
||||
Resolve(ctx context.Context, environment string) (string, string, bool)
|
||||
}
|
||||
Timeout time.Duration
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Verifier asks endpoints what they really hold. It is safe for concurrent use.
|
||||
@@ -75,12 +86,20 @@ func (verifier *Verifier) VerifyServiceTarget(ctx context.Context, target govern
|
||||
testBaseURL := firstNonEmpty(target.TestBaseURL, verifier.config.TestBaseURLFallback)
|
||||
productionKey := verifier.apiKey(verifier.config.ProductionAPIKeyFile)
|
||||
testKey := verifier.apiKey(verifier.config.TestAPIKeyFile)
|
||||
if verifier.config.ConnectionResolver != nil {
|
||||
if endpoint, key, ok := verifier.config.ConnectionResolver.Resolve(ctx, "test"); ok {
|
||||
testBaseURL, testKey = endpoint, key
|
||||
}
|
||||
if endpoint, key, ok := verifier.config.ConnectionResolver.Resolve(ctx, "production"); ok {
|
||||
productionBaseURL, productionKey = endpoint, key
|
||||
}
|
||||
}
|
||||
|
||||
verification := governance.ServiceTargetVerification{
|
||||
Status: governance.ServiceTargetNotConfigured,
|
||||
Message: "未配置:两项 ID 都留空,没有可核验的值",
|
||||
TestEndpoint: verifier.probeEndpoint(ctx, testBaseURL, testKey, "测试地址", verifier.config.TestAllowDockerHTTP),
|
||||
ProductionEndpoint: verifier.probeEndpoint(ctx, productionBaseURL, productionKey, "生产地址", verifier.config.ProductionAllowDockerHTTP),
|
||||
TestEndpoint: verifier.probeEndpointWithPolicy(ctx, testBaseURL, testKey, "测试地址", verifier.config.TestAllowDockerHTTP, verifier.config.TestAllowPrivateHTTP),
|
||||
ProductionEndpoint: verifier.probeEndpointWithPolicy(ctx, productionBaseURL, productionKey, "生产地址", verifier.config.ProductionAllowDockerHTTP, verifier.config.ProductionAllowPrivateHTTP),
|
||||
CheckedAt: verifier.config.Now().UTC(),
|
||||
}
|
||||
if target.ProductionKnowledgeBaseID == "" && target.ProductionTenantID == "" {
|
||||
@@ -96,7 +115,7 @@ func (verifier *Verifier) VerifyServiceTarget(ctx context.Context, target govern
|
||||
verification.Message = "未核验:没有可用的生产 API Key 文件(PRODUCTION_WEKNORA_API_KEY_FILE),保存未被阻断"
|
||||
return verification
|
||||
}
|
||||
client, err := verifier.client(productionBaseURL, productionKey, verifier.config.ProductionAllowDockerHTTP)
|
||||
client, err := verifier.clientWithPolicy(productionBaseURL, productionKey, verifier.config.ProductionAllowDockerHTTP, verifier.config.ProductionAllowPrivateHTTP)
|
||||
if err != nil {
|
||||
verification.Status = governance.ServiceTargetCannotVerify
|
||||
verification.Message = "未核验:生产地址不能作为 WeKnora 端点(" + err.Error() + "),保存未被阻断"
|
||||
@@ -198,11 +217,15 @@ func (verifier *Verifier) verifyTenantOnly(ctx context.Context, client *weknora.
|
||||
// is available, otherwise a plain /health liveness call. Advisory — the caller
|
||||
// only reports it.
|
||||
func (verifier *Verifier) probeEndpoint(ctx context.Context, baseURL, apiKey, label string, allowDockerHTTP bool) governance.ServiceEndpointStatus {
|
||||
return verifier.probeEndpointWithPolicy(ctx, baseURL, apiKey, label, allowDockerHTTP, false)
|
||||
}
|
||||
|
||||
func (verifier *Verifier) probeEndpointWithPolicy(ctx context.Context, baseURL, apiKey, label string, allowDockerHTTP, allowPrivateHTTP bool) governance.ServiceEndpointStatus {
|
||||
if strings.TrimSpace(baseURL) == "" {
|
||||
return governance.ServiceEndpointStatus{Status: governance.ServiceEndpointNotConfigured, Message: label + "未配置"}
|
||||
}
|
||||
if apiKey != "" {
|
||||
if client, err := verifier.client(baseURL, apiKey, allowDockerHTTP); err == nil {
|
||||
if client, err := verifier.clientWithPolicy(baseURL, apiKey, allowDockerHTTP, allowPrivateHTTP); err == nil {
|
||||
bases, listErr := client.ListKnowledgeBases(ctx)
|
||||
switch {
|
||||
case listErr == nil:
|
||||
@@ -271,12 +294,13 @@ func (verifier *Verifier) candidateSuffix(ctx context.Context, client *weknora.C
|
||||
return fmt.Sprintf(";该地址现有 %d 个知识库:%s", len(bases), strings.Join(listed, "、"))
|
||||
}
|
||||
|
||||
func (verifier *Verifier) client(baseURL, apiKey string, allowDockerHTTP bool) (*weknora.Client, error) {
|
||||
func (verifier *Verifier) clientWithPolicy(baseURL, apiKey string, allowDockerHTTP, allowPrivateHTTP bool) (*weknora.Client, error) {
|
||||
return weknora.NewClient(weknora.ClientConfig{
|
||||
BaseURL: baseURL,
|
||||
APIKey: apiKey,
|
||||
Timeout: verifier.config.Timeout,
|
||||
AllowDockerHTTP: allowDockerHTTP,
|
||||
BaseURL: baseURL,
|
||||
APIKey: apiKey,
|
||||
Timeout: verifier.config.Timeout,
|
||||
AllowDockerHTTP: allowDockerHTTP,
|
||||
AllowPrivateHTTP: allowPrivateHTTP,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,454 @@
|
||||
// Package weknoraconn owns the two WeKnora connections the console configures:
|
||||
// endpoint + write-only API key per environment, with a real listing as the only
|
||||
// proof a configuration is usable.
|
||||
//
|
||||
// It deliberately stores no knowledge-base binding. Which base receives 对内 /
|
||||
// 对外 content stays in routing_kb_bindings / production_kb_bindings, because one
|
||||
// meaning may only have one home.
|
||||
package weknoraconn
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/platform/weknora"
|
||||
"knowledge-governance/internal/sealedsecret"
|
||||
)
|
||||
|
||||
const (
|
||||
EnvironmentTest = "test"
|
||||
EnvironmentProduction = "production"
|
||||
|
||||
KeyActionKeep = "keep"
|
||||
KeyActionReplace = "replace"
|
||||
KeyActionClear = "clear"
|
||||
|
||||
// Outcome vocabulary shared by the API, the stored evidence and the page.
|
||||
OutcomeOK = "ok"
|
||||
OutcomeUnreachable = "unreachable"
|
||||
OutcomeKeyInvalid = "key_invalid"
|
||||
OutcomeKeyForbidden = "key_forbidden"
|
||||
|
||||
// VerificationNotConfigured means there is nothing to verify yet.
|
||||
VerificationNotConfigured = "not_configured"
|
||||
|
||||
// proofTTL bounds how long a successful 测试连接 vouches for a candidate.
|
||||
proofTTL = 15 * time.Minute
|
||||
)
|
||||
|
||||
var (
|
||||
ErrInvalid = errors.New("weknora connection request is invalid")
|
||||
ErrNotFound = errors.New("unknown weknora environment")
|
||||
ErrStaleRevision = errors.New("weknora connection revision is stale")
|
||||
ErrNotTested = errors.New("weknora connection must be tested before saving")
|
||||
ErrStorageUnavailable = errors.New("weknora connection secret storage is unavailable")
|
||||
)
|
||||
|
||||
// Store is the persistence the service needs. The postgres store implements it.
|
||||
type Store interface {
|
||||
LoadWeKnoraConnection(ctx context.Context, environment string) (postgresstore.WeKnoraConnection, bool, error)
|
||||
LoadWeKnoraConnections(ctx context.Context) ([]postgresstore.WeKnoraConnection, error)
|
||||
SaveWeKnoraConnection(ctx context.Context, connection postgresstore.WeKnoraConnection, now time.Time) error
|
||||
SaveWeKnoraConnectionTest(ctx context.Context, result postgresstore.WeKnoraConnectionTest, now time.Time) error
|
||||
LoadWeKnoraConnectionTest(ctx context.Context, environment string) (postgresstore.WeKnoraConnectionTest, bool, error)
|
||||
UpdateWeKnoraConnectionVerification(ctx context.Context, environment, verification string, libraryCount int, checkedAt time.Time, now time.Time) error
|
||||
}
|
||||
|
||||
// KnowledgeBaseLister is the narrow client the service needs; tests inject it.
|
||||
type KnowledgeBaseLister interface {
|
||||
ListKnowledgeBases(ctx context.Context) ([]weknora.KnowledgeBase, error)
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
// MasterKeyFile is the 0600 32-byte master key. Empty or unreadable means
|
||||
// the page can read state but cannot store a key — reported, never faked.
|
||||
MasterKeyFile string
|
||||
// AllowPrivateHTTP follows WEKNORA_ALLOW_PRIVATE_HTTP: same-LAN plain http
|
||||
// is usable, public http is not.
|
||||
AllowPrivateHTTP bool
|
||||
Timeout time.Duration
|
||||
Now func() time.Time
|
||||
// NewLister is overridable in tests. Production leaves it nil.
|
||||
NewLister func(endpoint, apiKey string) (KnowledgeBaseLister, error)
|
||||
}
|
||||
|
||||
type Service struct {
|
||||
store Store
|
||||
config Config
|
||||
}
|
||||
|
||||
func New(store Store, config Config) *Service {
|
||||
if config.Timeout <= 0 {
|
||||
config.Timeout = 10 * time.Second
|
||||
}
|
||||
if config.Now == nil {
|
||||
config.Now = time.Now
|
||||
}
|
||||
return &Service{store: store, config: config}
|
||||
}
|
||||
|
||||
// View is the page projection. It never carries the key or the ciphertext.
|
||||
type View struct {
|
||||
Environment string `json:"environment"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
KeyConfigured bool `json:"key_configured"`
|
||||
KeyTail string `json:"key_tail"`
|
||||
Verification string `json:"verification"`
|
||||
LibraryCount int `json:"library_count"`
|
||||
CheckedAt *time.Time `json:"checked_at"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
Revision int64 `json:"revision"`
|
||||
StorageReady bool `json:"storage_ready"`
|
||||
}
|
||||
|
||||
// TestInput is one 测试连接 request. An empty endpoint means "the saved one"; an
|
||||
// empty key_action means keep.
|
||||
type TestInput struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
APIKey string `json:"api_key"`
|
||||
KeyAction string `json:"key_action"`
|
||||
}
|
||||
|
||||
type TestResult struct {
|
||||
Environment string `json:"environment"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Outcome string `json:"outcome"`
|
||||
LibraryCount int `json:"library_count"`
|
||||
Message string `json:"message"`
|
||||
CheckedAt time.Time `json:"checked_at"`
|
||||
}
|
||||
|
||||
type SaveInput struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
APIKey string `json:"api_key"`
|
||||
KeyAction string `json:"key_action"`
|
||||
BaseRevision int64 `json:"base_revision"`
|
||||
}
|
||||
|
||||
func ValidEnvironment(value string) bool {
|
||||
return value == EnvironmentTest || value == EnvironmentProduction
|
||||
}
|
||||
|
||||
// View reads both environments, or one when environment is non-empty. A missing
|
||||
// row is projected as 未配置 so the page has a stable shape.
|
||||
func (service *Service) View(ctx context.Context, environment string) ([]View, error) {
|
||||
stored, err := service.store.LoadWeKnoraConnections(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
byEnvironment := make(map[string]postgresstore.WeKnoraConnection, len(stored))
|
||||
for _, connection := range stored {
|
||||
byEnvironment[connection.Environment] = connection
|
||||
}
|
||||
environments := []string{EnvironmentTest, EnvironmentProduction}
|
||||
if environment != "" {
|
||||
if !ValidEnvironment(environment) {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
environments = []string{environment}
|
||||
}
|
||||
_, masterKeyErr := sealedsecret.LoadKeyFile(service.config.MasterKeyFile)
|
||||
views := make([]View, 0, len(environments))
|
||||
for _, name := range environments {
|
||||
connection := byEnvironment[name]
|
||||
verification := connection.Verification
|
||||
if verification == "" {
|
||||
verification = VerificationNotConfigured
|
||||
}
|
||||
views = append(views, View{
|
||||
Environment: name,
|
||||
Endpoint: connection.Endpoint,
|
||||
KeyConfigured: len(connection.KeyCiphertext) > 0,
|
||||
KeyTail: connection.KeyTail,
|
||||
Verification: verification,
|
||||
LibraryCount: connection.LibraryCount,
|
||||
CheckedAt: connection.CheckedAt,
|
||||
UpdatedBy: connection.UpdatedBy,
|
||||
Revision: connection.Revision,
|
||||
StorageReady: masterKeyErr == nil,
|
||||
})
|
||||
}
|
||||
return views, nil
|
||||
}
|
||||
|
||||
// Test performs the real GET /api/v1/knowledge-bases against the candidate
|
||||
// endpoint and key, and records the outcome as the evidence a later save needs.
|
||||
func (service *Service) Test(ctx context.Context, environment string, input TestInput) (TestResult, error) {
|
||||
if !ValidEnvironment(environment) {
|
||||
return TestResult{}, ErrNotFound
|
||||
}
|
||||
saved, exists, err := service.store.LoadWeKnoraConnection(ctx, environment)
|
||||
if err != nil {
|
||||
return TestResult{}, err
|
||||
}
|
||||
endpoint := strings.TrimSpace(input.Endpoint)
|
||||
if endpoint == "" && exists {
|
||||
endpoint = saved.Endpoint
|
||||
}
|
||||
if endpoint == "" {
|
||||
return TestResult{}, ErrInvalid
|
||||
}
|
||||
action := normalizeKeyAction(input.KeyAction)
|
||||
apiKey, err := service.candidateKey(saved, exists, action, input.APIKey)
|
||||
if err != nil {
|
||||
return TestResult{}, err
|
||||
}
|
||||
now := service.config.Now().UTC()
|
||||
result := TestResult{Environment: environment, Endpoint: endpoint, CheckedAt: now}
|
||||
if !weknora.HTTPAllowed(endpoint, weknora.HTTPPolicy{AllowPrivateHTTP: service.config.AllowPrivateHTTP}) {
|
||||
result.Outcome = OutcomeUnreachable
|
||||
result.Message = "地址不可达:只接受 https,或已放行的私网/回环 http 地址"
|
||||
_ = service.recordTest(ctx, result, apiKey)
|
||||
return result, nil
|
||||
}
|
||||
lister, err := service.newLister(endpoint, apiKey)
|
||||
if err != nil {
|
||||
result.Outcome = OutcomeUnreachable
|
||||
result.Message = "地址不可达:" + err.Error()
|
||||
_ = service.recordTest(ctx, result, apiKey)
|
||||
return result, nil
|
||||
}
|
||||
probeCtx, cancel := context.WithTimeout(ctx, service.config.Timeout)
|
||||
defer cancel()
|
||||
bases, listErr := lister.ListKnowledgeBases(probeCtx)
|
||||
if listErr != nil {
|
||||
result.Outcome, result.Message = classifyListError(listErr)
|
||||
_ = service.recordTest(ctx, result, apiKey)
|
||||
return result, nil
|
||||
}
|
||||
result.Outcome = OutcomeOK
|
||||
result.LibraryCount = len(bases)
|
||||
result.Message = "接口可达 · 已读到 " + strconv.Itoa(len(bases)) + " 个知识库"
|
||||
if recordErr := service.recordTest(ctx, result, apiKey); recordErr != nil {
|
||||
// The listing worked but the proof did not land: say so now, because the
|
||||
// save gate will refuse this configuration a moment later.
|
||||
result.Message += "(测试证据未能落库,保存会被拒绝,请重试)"
|
||||
}
|
||||
// The card may only claim 「可达」 for the configuration it actually probed.
|
||||
if exists && action != KeyActionReplace && endpoint == saved.Endpoint {
|
||||
_ = service.store.UpdateWeKnoraConnectionVerification(ctx, environment, result.Outcome, result.LibraryCount, result.CheckedAt, now)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Save stores a tested configuration. A new key or a new endpoint must have a
|
||||
// successful test behind it; keeping a tested configuration is not blocked by a
|
||||
// jittery endpoint.
|
||||
func (service *Service) Save(ctx context.Context, environment string, input SaveInput, actor string) (View, error) {
|
||||
if !ValidEnvironment(environment) {
|
||||
return View{}, ErrNotFound
|
||||
}
|
||||
saved, exists, err := service.store.LoadWeKnoraConnection(ctx, environment)
|
||||
if err != nil {
|
||||
return View{}, err
|
||||
}
|
||||
endpoint := strings.TrimSpace(input.Endpoint)
|
||||
if endpoint == "" {
|
||||
return View{}, ErrInvalid
|
||||
}
|
||||
action := normalizeKeyAction(input.KeyAction)
|
||||
if exists && input.BaseRevision != saved.Revision {
|
||||
return View{}, ErrStaleRevision
|
||||
}
|
||||
if exists && endpoint != saved.Endpoint && action != KeyActionReplace {
|
||||
return View{}, ErrInvalid
|
||||
}
|
||||
apiKey, err := service.candidateKey(saved, exists, action, input.APIKey)
|
||||
if err != nil {
|
||||
return View{}, err
|
||||
}
|
||||
now := service.config.Now().UTC()
|
||||
revision := int64(1)
|
||||
if exists {
|
||||
revision = saved.Revision + 1
|
||||
}
|
||||
verification := VerificationNotConfigured
|
||||
libraryCount := 0
|
||||
var checkedAt *time.Time
|
||||
if apiKey != "" {
|
||||
if !weknora.HTTPAllowed(endpoint, weknora.HTTPPolicy{AllowPrivateHTTP: service.config.AllowPrivateHTTP}) {
|
||||
return View{}, ErrInvalid
|
||||
}
|
||||
if !exists || action == KeyActionReplace || endpoint != saved.Endpoint {
|
||||
proof, found, proofErr := service.store.LoadWeKnoraConnectionTest(ctx, environment)
|
||||
if proofErr != nil {
|
||||
return View{}, proofErr
|
||||
}
|
||||
if !found || proof.Outcome != OutcomeOK || proof.Endpoint != endpoint ||
|
||||
proof.KeyFingerprint != fingerprint(apiKey) || now.Sub(proof.CheckedAt) > proofTTL {
|
||||
return View{}, ErrNotTested
|
||||
}
|
||||
libraryCount = proof.LibraryCount
|
||||
checkedAt = &proof.CheckedAt
|
||||
} else {
|
||||
libraryCount = saved.LibraryCount
|
||||
checkedAt = saved.CheckedAt
|
||||
}
|
||||
verification = OutcomeOK
|
||||
}
|
||||
sealed := []byte{}
|
||||
if apiKey != "" {
|
||||
masterKey, keyErr := sealedsecret.LoadKeyFile(service.config.MasterKeyFile)
|
||||
if keyErr != nil {
|
||||
return View{}, ErrStorageUnavailable
|
||||
}
|
||||
sealed, err = sealedsecret.Seal(masterKey, apiKey, sealedsecret.AAD("weknora-connection", environment, revision))
|
||||
if err != nil {
|
||||
return View{}, ErrStorageUnavailable
|
||||
}
|
||||
}
|
||||
connection := postgresstore.WeKnoraConnection{
|
||||
Environment: environment,
|
||||
Revision: revision,
|
||||
Endpoint: endpoint,
|
||||
KeyCiphertext: sealed,
|
||||
KeyTail: keyTail(apiKey),
|
||||
Verification: verification,
|
||||
LibraryCount: libraryCount,
|
||||
CheckedAt: checkedAt,
|
||||
UpdatedBy: actor,
|
||||
}
|
||||
if err := service.store.SaveWeKnoraConnection(ctx, connection, now); err != nil {
|
||||
return View{}, err
|
||||
}
|
||||
views, err := service.View(ctx, environment)
|
||||
if err != nil {
|
||||
return View{}, err
|
||||
}
|
||||
return views[0], nil
|
||||
}
|
||||
|
||||
// Resolve returns the endpoint and key the platform should use for one
|
||||
// environment. It is the single source other features read instead of guessing
|
||||
// from environment variables.
|
||||
func (service *Service) Resolve(ctx context.Context, environment string) (string, string, bool) {
|
||||
if !ValidEnvironment(environment) {
|
||||
return "", "", false
|
||||
}
|
||||
saved, exists, err := service.store.LoadWeKnoraConnection(ctx, environment)
|
||||
if err != nil || !exists || saved.Endpoint == "" || len(saved.KeyCiphertext) == 0 {
|
||||
return "", "", false
|
||||
}
|
||||
masterKey, keyErr := sealedsecret.LoadKeyFile(service.config.MasterKeyFile)
|
||||
if keyErr != nil {
|
||||
return "", "", false
|
||||
}
|
||||
apiKey, openErr := sealedsecret.Open(masterKey, saved.KeyCiphertext, sealedsecret.AAD("weknora-connection", environment, saved.Revision))
|
||||
if openErr != nil {
|
||||
return "", "", false
|
||||
}
|
||||
return saved.Endpoint, apiKey, true
|
||||
}
|
||||
|
||||
// HTTPPolicy exposes the endpoint policy the client uses, so a caller that
|
||||
// builds its own client stays in agreement with the service.
|
||||
func (service *Service) HTTPPolicy() weknora.HTTPPolicy {
|
||||
return weknora.HTTPPolicy{AllowPrivateHTTP: service.config.AllowPrivateHTTP}
|
||||
}
|
||||
|
||||
func (service *Service) candidateKey(saved postgresstore.WeKnoraConnection, exists bool, action, inputKey string) (string, error) {
|
||||
switch action {
|
||||
case KeyActionClear:
|
||||
return "", nil
|
||||
case KeyActionReplace:
|
||||
trimmed := strings.TrimSpace(inputKey)
|
||||
if trimmed == "" {
|
||||
return "", ErrInvalid
|
||||
}
|
||||
return trimmed, nil
|
||||
default:
|
||||
if !exists || len(saved.KeyCiphertext) == 0 {
|
||||
return "", ErrInvalid
|
||||
}
|
||||
masterKey, err := sealedsecret.LoadKeyFile(service.config.MasterKeyFile)
|
||||
if err != nil {
|
||||
return "", ErrStorageUnavailable
|
||||
}
|
||||
apiKey, err := sealedsecret.Open(masterKey, saved.KeyCiphertext, sealedsecret.AAD("weknora-connection", saved.Environment, saved.Revision))
|
||||
if err != nil {
|
||||
return "", ErrStorageUnavailable
|
||||
}
|
||||
return apiKey, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (service *Service) newLister(endpoint, apiKey string) (KnowledgeBaseLister, error) {
|
||||
if service.config.NewLister != nil {
|
||||
return service.config.NewLister(endpoint, apiKey)
|
||||
}
|
||||
return weknora.NewClient(weknora.ClientConfig{
|
||||
BaseURL: endpoint,
|
||||
APIKey: apiKey,
|
||||
Timeout: service.config.Timeout,
|
||||
AllowPrivateHTTP: service.config.AllowPrivateHTTP,
|
||||
})
|
||||
}
|
||||
|
||||
func (service *Service) recordTest(ctx context.Context, result TestResult, apiKey string) error {
|
||||
return service.store.SaveWeKnoraConnectionTest(ctx, postgresstore.WeKnoraConnectionTest{
|
||||
Environment: result.Environment,
|
||||
Endpoint: result.Endpoint,
|
||||
KeyFingerprint: fingerprint(apiKey),
|
||||
Outcome: result.Outcome,
|
||||
LibraryCount: result.LibraryCount,
|
||||
Message: result.Message,
|
||||
}, result.CheckedAt)
|
||||
}
|
||||
|
||||
// classifyListError maps one failed listing onto the four answers the page
|
||||
// promises, so 「密钥无效」 and 「密钥无权限」 never collapse into one message.
|
||||
func classifyListError(err error) (string, string) {
|
||||
var httpErr *weknora.HTTPError
|
||||
if errors.As(err, &httpErr) {
|
||||
switch httpErr.StatusCode {
|
||||
case 401:
|
||||
return OutcomeKeyInvalid, "密钥无效:该实例拒绝了这把 API Key(401)"
|
||||
case 403:
|
||||
return OutcomeKeyForbidden, "密钥无权限读取知识库列表(403)"
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
return OutcomeUnreachable, "地址不可达:请求超时"
|
||||
default:
|
||||
var netErr interface{ Timeout() bool }
|
||||
if errors.As(err, &netErr) && netErr.Timeout() {
|
||||
return OutcomeUnreachable, "地址不可达:请求超时"
|
||||
}
|
||||
return OutcomeUnreachable, "地址不可达:连接失败"
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeKeyAction(action string) string {
|
||||
switch strings.TrimSpace(action) {
|
||||
case KeyActionReplace:
|
||||
return KeyActionReplace
|
||||
case KeyActionClear:
|
||||
return KeyActionClear
|
||||
default:
|
||||
return KeyActionKeep
|
||||
}
|
||||
}
|
||||
|
||||
func fingerprint(apiKey string) string {
|
||||
sum := sha256.Sum256([]byte(apiKey))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// keyTail is the only part of a key the page may show: the last four characters.
|
||||
func keyTail(apiKey string) string {
|
||||
if apiKey == "" {
|
||||
return ""
|
||||
}
|
||||
runes := []rune(apiKey)
|
||||
if len(runes) <= 4 {
|
||||
return string(runes)
|
||||
}
|
||||
return string(runes[len(runes)-4:])
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
package weknoraconn
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/platform/weknora"
|
||||
)
|
||||
|
||||
type fakeStore struct {
|
||||
connections map[string]postgres.WeKnoraConnection
|
||||
tests map[string]postgres.WeKnoraConnectionTest
|
||||
}
|
||||
|
||||
func newFakeStore() *fakeStore {
|
||||
return &fakeStore{
|
||||
connections: map[string]postgres.WeKnoraConnection{},
|
||||
tests: map[string]postgres.WeKnoraConnectionTest{},
|
||||
}
|
||||
}
|
||||
|
||||
func (store *fakeStore) LoadWeKnoraConnection(_ context.Context, environment string) (postgres.WeKnoraConnection, bool, error) {
|
||||
connection, ok := store.connections[environment]
|
||||
return connection, ok, nil
|
||||
}
|
||||
|
||||
func (store *fakeStore) LoadWeKnoraConnections(context.Context) ([]postgres.WeKnoraConnection, error) {
|
||||
result := make([]postgres.WeKnoraConnection, 0, len(store.connections))
|
||||
for _, connection := range store.connections {
|
||||
result = append(result, connection)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (store *fakeStore) SaveWeKnoraConnection(_ context.Context, connection postgres.WeKnoraConnection, _ time.Time) error {
|
||||
store.connections[connection.Environment] = connection
|
||||
return nil
|
||||
}
|
||||
|
||||
func (store *fakeStore) SaveWeKnoraConnectionTest(_ context.Context, result postgres.WeKnoraConnectionTest, now time.Time) error {
|
||||
result.CheckedAt = now
|
||||
store.tests[result.Environment] = result
|
||||
return nil
|
||||
}
|
||||
|
||||
func (store *fakeStore) LoadWeKnoraConnectionTest(_ context.Context, environment string) (postgres.WeKnoraConnectionTest, bool, error) {
|
||||
result, ok := store.tests[environment]
|
||||
return result, ok, nil
|
||||
}
|
||||
|
||||
func (store *fakeStore) UpdateWeKnoraConnectionVerification(_ context.Context, environment, verification string, libraryCount int, checkedAt time.Time, _ time.Time) error {
|
||||
connection, ok := store.connections[environment]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
connection.Verification = verification
|
||||
connection.LibraryCount = libraryCount
|
||||
connection.CheckedAt = &checkedAt
|
||||
store.connections[environment] = connection
|
||||
return nil
|
||||
}
|
||||
|
||||
type fakeLister struct {
|
||||
bases []weknora.KnowledgeBase
|
||||
err error
|
||||
}
|
||||
|
||||
func (lister fakeLister) ListKnowledgeBases(context.Context) ([]weknora.KnowledgeBase, error) {
|
||||
return lister.bases, lister.err
|
||||
}
|
||||
|
||||
func newService(t *testing.T, store Store, lister fakeLister) *Service {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "master.key")
|
||||
if err := os.WriteFile(path, []byte("0123456789abcdef0123456789abcdef"), 0o600); err != nil {
|
||||
t.Fatalf("write master key: %v", err)
|
||||
}
|
||||
if err := os.Chmod(path, 0o600); err != nil {
|
||||
t.Fatalf("chmod master key: %v", err)
|
||||
}
|
||||
now := time.Date(2026, 11, 22, 7, 0, 0, 0, time.UTC)
|
||||
return New(store, Config{
|
||||
MasterKeyFile: path,
|
||||
AllowPrivateHTTP: true,
|
||||
Timeout: time.Second,
|
||||
Now: func() time.Time { return now },
|
||||
NewLister: func(endpoint, apiKey string) (KnowledgeBaseLister, error) {
|
||||
if endpoint == "" || apiKey == "" {
|
||||
return nil, errors.New("missing endpoint or key")
|
||||
}
|
||||
return lister, nil
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func TestTestClassifiesTheFourAnswers(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantOutcome string
|
||||
wantMessage string
|
||||
}{
|
||||
{"unauthorized", &weknora.HTTPError{StatusCode: 401}, OutcomeKeyInvalid, "密钥无效"},
|
||||
{"forbidden", &weknora.HTTPError{StatusCode: 403}, OutcomeKeyForbidden, "密钥无权限"},
|
||||
{"timeout", context.DeadlineExceeded, OutcomeUnreachable, "地址不可达"},
|
||||
{"refused", errors.New("dial tcp: connection refused"), OutcomeUnreachable, "地址不可达"},
|
||||
}
|
||||
for _, testCase := range cases {
|
||||
service := newService(t, newFakeStore(), fakeLister{err: testCase.err})
|
||||
result, err := service.Test(context.Background(), EnvironmentTest, TestInput{
|
||||
Endpoint: "http://127.0.0.1:8092", APIKey: "sk-test-1234", KeyAction: KeyActionReplace,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: Test() error = %v", testCase.name, err)
|
||||
}
|
||||
if result.Outcome != testCase.wantOutcome {
|
||||
t.Fatalf("%s: outcome = %q, want %q", testCase.name, result.Outcome, testCase.wantOutcome)
|
||||
}
|
||||
if !contains(result.Message, testCase.wantMessage) {
|
||||
t.Fatalf("%s: message = %q, want it to mention %q", testCase.name, result.Message, testCase.wantMessage)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestSuccessReportsLibraryCount(t *testing.T) {
|
||||
service := newService(t, newFakeStore(), fakeLister{bases: []weknora.KnowledgeBase{{ID: "a"}, {ID: "b"}, {ID: "c"}}})
|
||||
result, err := service.Test(context.Background(), EnvironmentProduction, TestInput{
|
||||
Endpoint: "http://172.29.151.11:18093", APIKey: "prod-key-9999", KeyAction: KeyActionReplace,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Test() error = %v", err)
|
||||
}
|
||||
if result.Outcome != OutcomeOK || result.LibraryCount != 3 {
|
||||
t.Fatalf("result = %+v", result)
|
||||
}
|
||||
if !contains(result.Message, "3 个知识库") {
|
||||
t.Fatalf("message = %q", result.Message)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveRequiresTestProofAndProjectsTailOnly(t *testing.T) {
|
||||
store := newFakeStore()
|
||||
service := newService(t, store, fakeLister{bases: []weknora.KnowledgeBase{{ID: "a"}, {ID: "b"}}})
|
||||
ctx := context.Background()
|
||||
save := SaveInput{Endpoint: "http://127.0.0.1:8092", APIKey: "sk-live-abcd", KeyAction: KeyActionReplace}
|
||||
if _, err := service.Save(ctx, EnvironmentTest, save, "admin"); !errors.Is(err, ErrNotTested) {
|
||||
t.Fatalf("Save() without a test = %v, want ErrNotTested", err)
|
||||
}
|
||||
if _, err := service.Test(ctx, EnvironmentTest, TestInput{Endpoint: save.Endpoint, APIKey: save.APIKey, KeyAction: KeyActionReplace}); err != nil {
|
||||
t.Fatalf("Test() error = %v", err)
|
||||
}
|
||||
view, err := service.Save(ctx, EnvironmentTest, save, "admin")
|
||||
if err != nil {
|
||||
t.Fatalf("Save() error = %v", err)
|
||||
}
|
||||
if !view.KeyConfigured || view.KeyTail != "abcd" || view.Verification != OutcomeOK || view.LibraryCount != 2 {
|
||||
t.Fatalf("view = %+v", view)
|
||||
}
|
||||
if view.Revision != 1 {
|
||||
t.Fatalf("revision = %d", view.Revision)
|
||||
}
|
||||
endpoint, apiKey, ok := service.Resolve(ctx, EnvironmentTest)
|
||||
if !ok || endpoint != save.Endpoint || apiKey != save.APIKey {
|
||||
t.Fatalf("Resolve() = %q %q %v", endpoint, apiKey, ok)
|
||||
}
|
||||
// The proof is bound to the key: another key must not reuse it.
|
||||
other := SaveInput{Endpoint: save.Endpoint, APIKey: "sk-live-zzzz", KeyAction: KeyActionReplace, BaseRevision: 1}
|
||||
if _, err := service.Save(ctx, EnvironmentTest, other, "admin"); !errors.Is(err, ErrNotTested) {
|
||||
t.Fatalf("Save() with a different untested key = %v, want ErrNotTested", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveRejectsEndpointChangeWithoutReenteringKey(t *testing.T) {
|
||||
store := newFakeStore()
|
||||
service := newService(t, store, fakeLister{bases: []weknora.KnowledgeBase{{ID: "a"}}})
|
||||
ctx := context.Background()
|
||||
first := SaveInput{Endpoint: "http://127.0.0.1:8092", APIKey: "sk-live-abcd", KeyAction: KeyActionReplace}
|
||||
if _, err := service.Test(ctx, EnvironmentTest, TestInput{Endpoint: first.Endpoint, APIKey: first.APIKey, KeyAction: KeyActionReplace}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := service.Save(ctx, EnvironmentTest, first, "admin"); err != nil {
|
||||
t.Fatalf("first Save() error = %v", err)
|
||||
}
|
||||
if _, err := service.Save(ctx, EnvironmentTest, SaveInput{Endpoint: "http://127.0.0.1:9092", KeyAction: KeyActionKeep, BaseRevision: 1}, "admin"); !errors.Is(err, ErrInvalid) {
|
||||
t.Fatalf("endpoint change with keep = %v, want ErrInvalid", err)
|
||||
}
|
||||
if _, err := service.Save(ctx, EnvironmentTest, SaveInput{Endpoint: "http://127.0.0.1:8092", KeyAction: KeyActionKeep, BaseRevision: 7}, "admin"); !errors.Is(err, ErrStaleRevision) {
|
||||
t.Fatalf("stale revision = %v, want ErrStaleRevision", err)
|
||||
}
|
||||
// Keeping the tested configuration must not be blocked by proof age.
|
||||
view, err := service.Save(ctx, EnvironmentTest, SaveInput{Endpoint: "http://127.0.0.1:8092", KeyAction: KeyActionKeep, BaseRevision: 1}, "admin")
|
||||
if err != nil || view.Verification != OutcomeOK || view.KeyTail != "abcd" {
|
||||
t.Fatalf("keep Save() = %+v, %v", view, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClearKeyResetsVerification(t *testing.T) {
|
||||
store := newFakeStore()
|
||||
service := newService(t, store, fakeLister{bases: []weknora.KnowledgeBase{{ID: "a"}}})
|
||||
ctx := context.Background()
|
||||
if _, err := service.Test(ctx, EnvironmentProduction, TestInput{Endpoint: "https://prod.example", APIKey: "prod-key", KeyAction: KeyActionReplace}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := service.Save(ctx, EnvironmentProduction, SaveInput{Endpoint: "https://prod.example", APIKey: "prod-key", KeyAction: KeyActionReplace}, "admin"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
view, err := service.Save(ctx, EnvironmentProduction, SaveInput{Endpoint: "https://prod.example", KeyAction: KeyActionClear, BaseRevision: 1}, "admin")
|
||||
if err != nil {
|
||||
t.Fatalf("clear Save() error = %v", err)
|
||||
}
|
||||
if view.KeyConfigured || view.KeyTail != "" || view.Verification != VerificationNotConfigured {
|
||||
t.Fatalf("cleared view = %+v", view)
|
||||
}
|
||||
if _, _, ok := service.Resolve(ctx, EnvironmentProduction); ok {
|
||||
t.Fatal("a cleared connection must not resolve")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPublicHTTPIsRefusedEvenWhenPrivateHTTPIsAllowed(t *testing.T) {
|
||||
service := newService(t, newFakeStore(), fakeLister{})
|
||||
result, err := service.Test(context.Background(), EnvironmentTest, TestInput{Endpoint: "http://weknora.example", APIKey: "k", KeyAction: KeyActionReplace})
|
||||
if err != nil {
|
||||
t.Fatalf("Test() error = %v", err)
|
||||
}
|
||||
if result.Outcome != OutcomeUnreachable {
|
||||
t.Fatalf("outcome = %q", result.Outcome)
|
||||
}
|
||||
if _, err := service.Save(context.Background(), EnvironmentTest, SaveInput{Endpoint: "http://weknora.example", APIKey: "k", KeyAction: KeyActionReplace}, "admin"); err == nil {
|
||||
t.Fatal("saving a public http endpoint must fail")
|
||||
}
|
||||
}
|
||||
|
||||
func contains(value, needle string) bool {
|
||||
return len(needle) == 0 || (len(value) >= len(needle) && indexOf(value, needle) >= 0)
|
||||
}
|
||||
|
||||
func indexOf(value, needle string) int {
|
||||
for i := 0; i+len(needle) <= len(value); i++ {
|
||||
if value[i:i+len(needle)] == needle {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
import { useCallback, useEffect, useState } from 'react'
|
||||
|
||||
import { useOptionalAuth } from './auth'
|
||||
import { WeKnoraConnections } from './WeKnoraConnections'
|
||||
import { fetchProductionBindingOptions, fetchProductionBindings, saveProductionBindings } from './productionBindings'
|
||||
import type { ProductionBindings } from './productionBindings'
|
||||
import type { KnowledgeBaseOption } from './routingBindings'
|
||||
@@ -41,9 +42,12 @@ export function ProductionBindingsSection() {
|
||||
const next = await saveProductionBindings({ internal_kb_id: internalDraft, external_kb_id: externalDraft })
|
||||
setBindings(next)
|
||||
const where = `对内 ${next.internal_kb_name || next.internal_kb_id.slice(0, 8) || '未绑定'} · 对外 ${next.external_kb_name || next.external_kb_id.slice(0, 8) || '未绑定'}`
|
||||
const frozen = next.frozen + next.retargetable > 0
|
||||
? `;已冻结目标 ${next.frozen} 条(不会被改写),仍可改指向 ${next.retargetable} 条`
|
||||
: ''
|
||||
setMessage(next.production_connected
|
||||
? `发布目标已更新:${where}。之后新发布的“对内/对外”内容会写入这两个生产库(已发布过的条目仍回到原目标)。`
|
||||
: `发布目标已保存:${where};但当前核对不了生产端(${next.production_message}),下次发布前请确认。`)
|
||||
? `发布目标已更新:${where}${frozen}。之后新发布的“对内/对外”内容会写入这两个生产库。`
|
||||
: `发布目标已保存:${where}${frozen};但当前核对不了生产端(${next.production_message}),下次发布前请确认。`)
|
||||
} catch (cause) { setError(cause instanceof Error ? cause.message : '保存生产知识库绑定失败') }
|
||||
finally { setSaving(false) }
|
||||
}
|
||||
@@ -62,11 +66,14 @@ export function ProductionBindingsSection() {
|
||||
<option value="">{placeholder}</option>
|
||||
{options.map((base) => <option key={base.id} value={base.id}>{optionLabel(base)}</option>)}
|
||||
</select>
|
||||
: <input value={value} onChange={(event) => onChange(event.target.value.trim())} placeholder={`${placeholder}(输入生产知识库 ID)`} aria-label={placeholder} />
|
||||
: <select value="" disabled aria-label={placeholder}>
|
||||
<option value="">读不到生产库列表:先在上方把生产连接测通</option>
|
||||
</select>
|
||||
)
|
||||
|
||||
return <section className="settings-panel wide" id="settings-production-bindings">
|
||||
<div className="settings-section-heading"><div><h2>生产知识库绑定</h2><p>发布链路的“源库 → 目标库”:新发布的对内 / 对外内容按这里的绑定写入生产 WeKnora 的对应知识库。留空表示未绑定,按同 ID 语义发布。</p></div><button onClick={() => void load()} disabled={loading}>{loading ? '读取中…' : '重新读取'}</button></div>
|
||||
<WeKnoraConnections environment="production" title="生产 WeKnora" placeholder="http://172.29.151.11:18093" onSaved={() => void load()} />
|
||||
{error ? <p className="error-message" role="alert">{error}</p> : null}
|
||||
{bindings ? <div className="kb-binding-grid">
|
||||
{renderSide('对内生产知识库', bindings.internal_kb_id, bindings.internal_kb_name)}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { useCallback, useEffect, useState } from 'react'
|
||||
|
||||
import { useOptionalAuth } from './auth'
|
||||
import { WeKnoraConnections } from './WeKnoraConnections'
|
||||
import { fetchRoutingBindingOptions, fetchRoutingBindings, saveRoutingBindings } from './routingBindings'
|
||||
import type { KnowledgeBaseOption, RoutingBindings } from './routingBindings'
|
||||
|
||||
@@ -17,67 +18,171 @@ export function RoutingBindingsSection() {
|
||||
const [message, setMessage] = useState('')
|
||||
|
||||
const load = useCallback(async () => {
|
||||
setLoading(true); setError(null)
|
||||
setLoading(true)
|
||||
setError(null)
|
||||
try {
|
||||
const next = await fetchRoutingBindings()
|
||||
setBindings(next)
|
||||
setInternalDraft(next.internal_bound_by_ui ? next.internal_kb_id : '')
|
||||
setExternalDraft(next.external_bound_by_ui ? next.external_kb_id : '')
|
||||
try { setOptions(await fetchRoutingBindingOptions()) } catch { setOptions([]) }
|
||||
} catch (cause) { setError(cause instanceof Error ? cause.message : '读取测试知识库绑定失败') }
|
||||
finally { setLoading(false) }
|
||||
try {
|
||||
setOptions(await fetchRoutingBindingOptions())
|
||||
} catch {
|
||||
setOptions([])
|
||||
}
|
||||
} catch (cause) {
|
||||
setError(cause instanceof Error ? cause.message : '读取测试知识库绑定失败')
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [])
|
||||
useEffect(() => { void load() }, [load])
|
||||
useEffect(() => {
|
||||
void load()
|
||||
}, [load])
|
||||
|
||||
const dirty = bindings !== null && (internalDraft !== (bindings.internal_bound_by_ui ? bindings.internal_kb_id : '') || externalDraft !== (bindings.external_bound_by_ui ? bindings.external_kb_id : ''))
|
||||
const dirty =
|
||||
bindings !== null &&
|
||||
(internalDraft !== (bindings.internal_bound_by_ui ? bindings.internal_kb_id : '') ||
|
||||
externalDraft !== (bindings.external_bound_by_ui ? bindings.external_kb_id : ''))
|
||||
const sameSelection = internalDraft !== '' && internalDraft === externalDraft
|
||||
|
||||
const save = async () => {
|
||||
setSaving(true); setError(null); setMessage('')
|
||||
setSaving(true)
|
||||
setError(null)
|
||||
setMessage('')
|
||||
try {
|
||||
const next = await saveRoutingBindings({ internal_kb_id: internalDraft, external_kb_id: externalDraft })
|
||||
setBindings(next)
|
||||
const replan = next.replanned || next.retired_duplicates
|
||||
? `;同时把 ${next.replanned} 项还没上传的待入库重新指向新库${next.retired_duplicates ? `,${next.retired_duplicates} 项因新库里已有同篇而被退休` : ''}`
|
||||
: ''
|
||||
setMessage(`绑定已生效:对内 ${next.internal_kb_name || next.internal_kb_id.slice(0, 8) || '未绑定'} · 对外 ${next.external_kb_name || next.external_kb_id.slice(0, 8) || '未绑定'}。此后“来源与目录”的标记按新绑定路由${replan}。`)
|
||||
} catch (cause) { setError(cause instanceof Error ? cause.message : '保存测试知识库绑定失败') }
|
||||
finally { setSaving(false) }
|
||||
const replan =
|
||||
next.replanned || next.retired_duplicates
|
||||
? `;同时把 ${next.replanned} 项还没上传的待入库重新指向新库${next.retired_duplicates ? `,${next.retired_duplicates} 项因新库里已有同篇而被退休` : ''}`
|
||||
: ''
|
||||
const fallbacks: string[] = []
|
||||
if (!next.internal_bound_by_ui)
|
||||
fallbacks.push(
|
||||
`对内回退到部署默认库 ${next.internal_fallback_kb_id ? `${next.internal_fallback_kb_id.slice(0, 8)}…` : '(未设置)'}`
|
||||
)
|
||||
if (!next.external_bound_by_ui)
|
||||
fallbacks.push(
|
||||
`对外回退到部署默认库 ${next.external_fallback_kb_id ? `${next.external_fallback_kb_id.slice(0, 8)}…` : '(未设置)'}`
|
||||
)
|
||||
setMessage(
|
||||
`绑定已生效:对内 ${next.internal_kb_name || next.internal_kb_id.slice(0, 8) || '未绑定'} · 对外 ${next.external_kb_name || next.external_kb_id.slice(0, 8) || '未绑定'}${fallbacks.length ? `;${fallbacks.join(';')}` : ''}。此后“来源与目录”的标记按新绑定路由${replan}。`
|
||||
)
|
||||
} catch (cause) {
|
||||
setError(cause instanceof Error ? cause.message : '保存测试知识库绑定失败')
|
||||
} finally {
|
||||
setSaving(false)
|
||||
}
|
||||
}
|
||||
|
||||
const optionLabel = (base: KnowledgeBaseOption) => `${base.name}(${base.knowledge_count} 文档 · ${base.id.slice(0, 8)}…)`
|
||||
const renderSide = (label: string, boundID: string, boundName: string, fromEnv: boolean, boundByUI: boolean) => (
|
||||
const optionLabel = (base: KnowledgeBaseOption) =>
|
||||
`${base.name}(${base.knowledge_count} 文档 · ${base.id.slice(0, 8)}…)`
|
||||
const renderSide = (
|
||||
label: string,
|
||||
boundID: string,
|
||||
boundName: string,
|
||||
fromEnv: boolean,
|
||||
boundByUI: boolean,
|
||||
fallbackID = ''
|
||||
) => (
|
||||
<div className={`kb-binding-side ${boundID ? 'bound' : 'warn'}`}>
|
||||
<b>{label}</b>
|
||||
<span>{boundID ? (boundName || `${boundID.slice(0, 8)}…`) : '未绑定 · 该方向的标记会被拒绝'}</span>
|
||||
<small>{boundID ? (boundByUI ? (boundName ? `UI 绑定 · ${boundID.slice(0, 8)}…` : 'UI 绑定') : fromEnv ? '来自部署默认值(env)' : '') : '请绑定一个测试知识库'}</small>
|
||||
<span>{boundID ? boundName || `${boundID.slice(0, 8)}…` : '未绑定 · 该方向的标记会被拒绝'}</span>
|
||||
<small>
|
||||
{boundID
|
||||
? boundByUI
|
||||
? boundName
|
||||
? `UI 绑定 · ${boundID.slice(0, 8)}…`
|
||||
: 'UI 绑定'
|
||||
: fromEnv
|
||||
? `回退到部署默认库 ${fallbackID ? `${fallbackID.slice(0, 8)}…` : '(未设置)'}`
|
||||
: ''
|
||||
: '请绑定一个测试知识库'}
|
||||
</small>
|
||||
</div>
|
||||
)
|
||||
const editorFor = (value: string, onChange: (next: string) => void, placeholder: string) => (
|
||||
options.length > 0
|
||||
? <select value={value} onChange={(event) => onChange(event.target.value)} aria-label={placeholder}>
|
||||
<option value="">{placeholder}</option>
|
||||
{options.map((base) => <option key={base.id} value={base.id}>{optionLabel(base)}</option>)}
|
||||
</select>
|
||||
: <input value={value} onChange={(event) => onChange(event.target.value.trim())} placeholder={`${placeholder}(输入知识库 ID)`} aria-label={placeholder} />
|
||||
)
|
||||
const editorFor = (value: string, onChange: (next: string) => void, placeholder: string) =>
|
||||
options.length > 0 ? (
|
||||
<select value={value} onChange={(event) => onChange(event.target.value)} aria-label={placeholder}>
|
||||
<option value="">{placeholder}</option>
|
||||
{options.map((base) => (
|
||||
<option key={base.id} value={base.id}>
|
||||
{optionLabel(base)}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
) : (
|
||||
<select value="" disabled aria-label={placeholder}>
|
||||
<option value="">读不到测试库列表:先在上方把测试连接测通</option>
|
||||
</select>
|
||||
)
|
||||
|
||||
return <section className="settings-panel wide" id="settings-routing-bindings">
|
||||
<div className="settings-section-heading"><div><h2>测试知识库绑定</h2><p>“来源与目录”里的对内 / 对外标记按这里的绑定分发到对应的测试 WeKnora 知识库;保存后立即生效,无需重启。发布写入的生产知识库见「生产知识库绑定」。</p></div><button onClick={() => void load()} disabled={loading}>{loading ? '读取中…' : '重新读取'}</button></div>
|
||||
{error ? <p className="error-message" role="alert">{error}</p> : null}
|
||||
{bindings ? <div className="kb-binding-grid">
|
||||
{renderSide('对内知识库', bindings.internal_kb_id, bindings.internal_kb_name, bindings.internal_env_fallback, bindings.internal_bound_by_ui)}
|
||||
{renderSide('对外知识库', bindings.external_kb_id, bindings.external_kb_name, bindings.external_env_fallback, bindings.external_bound_by_ui)}
|
||||
</div> : null}
|
||||
{bindings && !bindings.weknora_connected ? <p className="hint">当前读不到 WeKnora 的库列表,只能按 ID 填写;连接恢复后可改为下拉选择。</p> : null}
|
||||
{isAdmin && bindings ? <div className="kb-binding-editor">
|
||||
<label>对内绑定 {editorFor(internalDraft, setInternalDraft, '选择对内知识库')}</label>
|
||||
<label>对外绑定 {editorFor(externalDraft, setExternalDraft, '选择对外知识库')}</label>
|
||||
<p className="hint">留空 = 取消 UI 绑定,回退到部署默认值;对内与对外必须是不同的库。</p>
|
||||
{sameSelection ? <p className="error-message">对内与对外必须绑定不同的知识库</p> : null}
|
||||
<button onClick={() => void save()} disabled={saving || !dirty || sameSelection}>{saving ? '正在校验并保存…' : '保存绑定'}</button>
|
||||
{message ? <p className="success-message" role="status">{message}</p> : null}
|
||||
</div> : null}
|
||||
{!isAdmin ? <p className="hint">仅管理员可修改测试知识库绑定。</p> : null}
|
||||
</section>
|
||||
return (
|
||||
<section className="settings-panel wide" id="settings-routing-bindings">
|
||||
<div className="settings-section-heading">
|
||||
<div>
|
||||
<h2>测试知识库绑定</h2>
|
||||
<p>
|
||||
“来源与目录”里的对内 / 对外标记按这里的绑定分发到对应的测试 WeKnora
|
||||
知识库;保存后立即生效,无需重启。发布写入的生产知识库见「生产知识库绑定」。
|
||||
</p>
|
||||
</div>
|
||||
<button onClick={() => void load()} disabled={loading}>
|
||||
{loading ? '读取中…' : '重新读取'}
|
||||
</button>
|
||||
</div>
|
||||
<WeKnoraConnections
|
||||
environment="test"
|
||||
title="测试 WeKnora"
|
||||
placeholder="http://127.0.0.1:8092"
|
||||
onSaved={() => void load()}
|
||||
/>
|
||||
{error ? (
|
||||
<p className="error-message" role="alert">
|
||||
{error}
|
||||
</p>
|
||||
) : null}
|
||||
{bindings ? (
|
||||
<div className="kb-binding-grid">
|
||||
{renderSide(
|
||||
'对内知识库',
|
||||
bindings.internal_kb_id,
|
||||
bindings.internal_kb_name,
|
||||
bindings.internal_env_fallback,
|
||||
bindings.internal_bound_by_ui,
|
||||
bindings.internal_fallback_kb_id
|
||||
)}
|
||||
{renderSide(
|
||||
'对外知识库',
|
||||
bindings.external_kb_id,
|
||||
bindings.external_kb_name,
|
||||
bindings.external_env_fallback,
|
||||
bindings.external_bound_by_ui,
|
||||
bindings.external_fallback_kb_id
|
||||
)}
|
||||
</div>
|
||||
) : null}
|
||||
{bindings && !bindings.weknora_connected ? (
|
||||
<p className="hint">当前读不到 WeKnora 的库列表:先在上面的连接卡把地址与密钥配好并测通,再回来选库。</p>
|
||||
) : null}
|
||||
{isAdmin && bindings ? (
|
||||
<div className="kb-binding-editor">
|
||||
<label>对内绑定 {editorFor(internalDraft, setInternalDraft, '选择对内知识库')}</label>
|
||||
<label>对外绑定 {editorFor(externalDraft, setExternalDraft, '选择对外知识库')}</label>
|
||||
<p className="hint">留空 = 取消 UI 绑定,回退到部署默认值;对内与对外必须是不同的库。</p>
|
||||
{sameSelection ? <p className="error-message">对内与对外必须绑定不同的知识库</p> : null}
|
||||
<button onClick={() => void save()} disabled={saving || !dirty || sameSelection}>
|
||||
{saving ? '正在校验并保存…' : '保存绑定'}
|
||||
</button>
|
||||
{message ? (
|
||||
<p className="success-message" role="status">
|
||||
{message}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
) : null}
|
||||
{!isAdmin ? <p className="hint">仅管理员可修改测试知识库绑定。</p> : null}
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -424,12 +424,14 @@ describe('SystemSettingsView', () => {
|
||||
expect(on).toContain('测试 WeKnora')
|
||||
expect(on).toContain('可达 · 可接单')
|
||||
expect(on).toContain('延迟 86ms')
|
||||
expect(on).toContain('已启用 · 可提交生产请求')
|
||||
expect(on).toContain('生产写入:开启 · 可提交生产请求')
|
||||
expect(on).toContain('生产写入:开启')
|
||||
expect(on).toContain('发布目标与凭据:管理员受限配置')
|
||||
const off = view(false, false)
|
||||
expect(off).toContain('未启用')
|
||||
// 连接状态与生产写入是两行:闸门关掉不代表「连接不可达」,两个事实分开说。
|
||||
expect(off).toContain('连接:可达')
|
||||
expect(off).toContain('生产写入:关闭')
|
||||
expect(off).not.toContain('被阻断')
|
||||
// 卡片状态由真实探测/门禁驱动,写死的“已连接 / 未连接 / 生产映射:未配置”必须消失。
|
||||
for (const removed of ['本地真实连接', '生产映射:未配置']) expect(off).not.toContain(removed)
|
||||
// 生产端点可达性来自真实探活,且只说真话:探活不等于验证过接单能力。
|
||||
@@ -453,7 +455,7 @@ describe('SystemSettingsView', () => {
|
||||
initialSection="services"
|
||||
/>
|
||||
)
|
||||
expect(blocked).toContain('已启用 · 被阻断')
|
||||
expect(blocked).toContain('生产写入:开启 · 恢复演练未核验')
|
||||
// 未配置生产端点时必须显式说「未配置」,不得靠文案含糊过去。
|
||||
expect(blocked).toContain('端点探测:暂无真实探测结果')
|
||||
const unconfigured = renderToStaticMarkup(
|
||||
|
||||
+23
-26
@@ -119,6 +119,15 @@ export const productionWeKnoraProbe = (monitor: MonitorOverview | null): string
|
||||
return `端点探测:${label}${detail ? `(${detail})` : ''}`
|
||||
}
|
||||
|
||||
// 生产卡片拆两行:连接是否可达,与生产写入是否开启是两件事 —— 以前把
|
||||
// 「被阻断」混进连接状态,页面会自相矛盾(端点明明可达却显示被阻断)。
|
||||
const productionConnectionLine = (monitor: MonitorOverview | null): string => {
|
||||
const health = findHealth(monitor, '生产 WeKnora')
|
||||
if (!health) return '连接:暂无探测结果'
|
||||
if (!health.configured) return '连接:未配置'
|
||||
return health.liveness === 'up' ? '连接:可达' : '连接:不可达'
|
||||
}
|
||||
|
||||
export function SystemSettingsView({
|
||||
sources,
|
||||
connections,
|
||||
@@ -199,6 +208,12 @@ export function SystemSettingsView({
|
||||
setDirty(false)
|
||||
onSave(candidateInput())
|
||||
}
|
||||
const productionConnectionLineText = productionConnectionLine(monitor)
|
||||
const productionWriteLine = !productionEnabled
|
||||
? '生产写入:关闭'
|
||||
: productionBackupRestoreVerified
|
||||
? '生产写入:开启 · 可提交生产请求'
|
||||
: '生产写入:开启 · 恢复演练未核验'
|
||||
return (
|
||||
<main className="page-shell system-settings">
|
||||
<header className="hero compact">
|
||||
@@ -278,14 +293,8 @@ export function SystemSettingsView({
|
||||
</article>
|
||||
<article className="production">
|
||||
<b>生产 WeKnora</b>
|
||||
<span>
|
||||
{!productionEnabled
|
||||
? '未启用'
|
||||
: productionBackupRestoreVerified
|
||||
? '已启用 · 可提交生产请求'
|
||||
: '已启用 · 被阻断'}
|
||||
</span>
|
||||
<small>{`${productionWeKnoraProbe(monitor)};生产写入:${productionEnabled ? '开启' : '关闭'};发布目标与凭据:管理员受限配置,本页不可回读`}</small>
|
||||
<span>{productionConnectionLineText}</span>
|
||||
<small>{`${productionWeKnoraProbe(monitor)};${productionWriteLine};发布目标与凭据:管理员受限配置,本页不可回读`}</small>
|
||||
</article>
|
||||
</div>
|
||||
<div className="safe-setting-grid production-endpoint-form">
|
||||
@@ -293,27 +302,11 @@ export function SystemSettingsView({
|
||||
<>
|
||||
<label>
|
||||
测试地址
|
||||
<input
|
||||
value={testEndpoint}
|
||||
onChange={(event) => {
|
||||
setTestEndpoint(event.target.value)
|
||||
setDirty(true)
|
||||
}}
|
||||
placeholder="https://weknora.test.example"
|
||||
aria-invalid={!testEndpointValid}
|
||||
/>
|
||||
<input value={testEndpoint || '未配置'} readOnly disabled />
|
||||
</label>
|
||||
<label>
|
||||
生产地址
|
||||
<input
|
||||
value={endpoint}
|
||||
onChange={(event) => {
|
||||
setEndpoint(event.target.value)
|
||||
setDirty(true)
|
||||
}}
|
||||
placeholder="https://weknora.prod.example"
|
||||
aria-invalid={!endpointValid}
|
||||
/>
|
||||
<input value={endpoint || '未配置'} readOnly disabled />
|
||||
</label>
|
||||
<label>
|
||||
生产 ID
|
||||
@@ -376,6 +369,10 @@ export function SystemSettingsView({
|
||||
原子切换);后端同时对测试地址/生产地址做真实探活,并用生产地址真查这两个 ID(WeKnora
|
||||
列库接口):端点回答"没有这个知识库"或租户不符会拒绝保存,端点抖动、鉴权失败或未配置生产凭据只警告不拦,核验原话与知识库名在保存后显示。
|
||||
</small>
|
||||
<p className="hint">
|
||||
地址已移到「测试知识库绑定 /
|
||||
生产知识库绑定」分区顶部的连接卡维护(可在那里测试连接);这里是回退值,只读。
|
||||
</p>
|
||||
{serviceSettingsValid ? null : (
|
||||
<p className="error-message" role="alert">
|
||||
格式不合法:地址需 http:// 或 https:// 开头的纯地址,不能带用户名密码、? 查询串或 # 片段;生产
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
import { useCallback, useEffect, useState } from 'react'
|
||||
|
||||
import { PasswordInput } from './PasswordInput'
|
||||
import { useOptionalAuth } from './auth'
|
||||
import {
|
||||
fetchWeKnoraConnections,
|
||||
keyLabel,
|
||||
saveWeKnoraConnection,
|
||||
testWeKnoraConnection,
|
||||
verificationLabel,
|
||||
} from './weknoraConnections'
|
||||
import type { WeKnoraConnection, WeKnoraEnvironment } from './weknoraConnections'
|
||||
|
||||
interface Props {
|
||||
environment: WeKnoraEnvironment
|
||||
title: string
|
||||
placeholder: string
|
||||
onSaved?: () => void
|
||||
}
|
||||
|
||||
// 连接卡:地址 + 写-only 密钥 + 真实列库测试。它只改「连哪台实例、用哪把钥匙」,
|
||||
// 不碰绑定,也不触发任何同步或发布。
|
||||
export function WeKnoraConnections({ environment, title, placeholder, onSaved }: Props) {
|
||||
const auth = useOptionalAuth()
|
||||
const isAdmin = auth?.identity?.role === 'admin'
|
||||
const [connection, setConnection] = useState<WeKnoraConnection | null>(null)
|
||||
const [endpointDraft, setEndpointDraft] = useState('')
|
||||
const [keyDraft, setKeyDraft] = useState('')
|
||||
const [testing, setTesting] = useState(false)
|
||||
const [saving, setSaving] = useState(false)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [message, setMessage] = useState('')
|
||||
const [testedEndpoint, setTestedEndpoint] = useState('')
|
||||
|
||||
const load = useCallback(async () => {
|
||||
try {
|
||||
const all = await fetchWeKnoraConnections()
|
||||
const found = all.find((item) => item.environment === environment) ?? null
|
||||
setConnection(found)
|
||||
setEndpointDraft(found?.endpoint ?? '')
|
||||
} catch (cause) {
|
||||
setError(cause instanceof Error ? cause.message : '读取连接失败')
|
||||
}
|
||||
}, [environment])
|
||||
useEffect(() => {
|
||||
void load()
|
||||
}, [load])
|
||||
|
||||
const endpoint = endpointDraft.trim()
|
||||
const keyEntered = keyDraft.trim() !== ''
|
||||
const endpointChanged = connection !== null && endpoint !== connection.endpoint
|
||||
const canTest = endpoint !== '' && !testing && !saving
|
||||
const tested = testedEndpoint !== '' && testedEndpoint === endpoint
|
||||
const canSave = endpoint !== '' && !saving && !testing && (keyEntered || endpointChanged ? tested : true)
|
||||
|
||||
const test = async () => {
|
||||
setTesting(true)
|
||||
setError(null)
|
||||
setMessage('')
|
||||
try {
|
||||
const result = await testWeKnoraConnection(environment, {
|
||||
endpoint,
|
||||
...(keyEntered
|
||||
? { api_key: keyDraft.trim(), key_action: 'replace' as const }
|
||||
: { key_action: 'keep' as const }),
|
||||
})
|
||||
setMessage(result.message)
|
||||
setTestedEndpoint(result.outcome === 'ok' ? endpoint : '')
|
||||
if (result.outcome !== 'ok') setError(result.message)
|
||||
} catch (cause) {
|
||||
setError(cause instanceof Error ? cause.message : '测试连接失败')
|
||||
} finally {
|
||||
setTesting(false)
|
||||
}
|
||||
}
|
||||
|
||||
const save = async (action: 'keep' | 'replace' | 'clear') => {
|
||||
setSaving(true)
|
||||
setError(null)
|
||||
try {
|
||||
const next = await saveWeKnoraConnection(environment, {
|
||||
endpoint,
|
||||
...(action === 'replace' ? { api_key: keyDraft.trim() } : {}),
|
||||
key_action: action,
|
||||
base_revision: connection?.revision ?? 0,
|
||||
})
|
||||
setConnection(next)
|
||||
setEndpointDraft(next.endpoint)
|
||||
setKeyDraft('')
|
||||
setTestedEndpoint('')
|
||||
setMessage(`已保存:${keyLabel(next)};最近一次测试:${verificationLabel(next)}。保存未触发任何同步或发布。`)
|
||||
onSaved?.()
|
||||
} catch (cause) {
|
||||
setError(cause instanceof Error ? cause.message : '保存连接失败')
|
||||
} finally {
|
||||
setSaving(false)
|
||||
}
|
||||
}
|
||||
|
||||
const clearKey = () => {
|
||||
if (!window.confirm('清除后该侧将没有任何可用密钥,发布与列库都会失败。确定清除?')) return
|
||||
void save('clear')
|
||||
}
|
||||
|
||||
const saveAction = keyEntered ? 'replace' : 'keep'
|
||||
|
||||
return (
|
||||
<div className="weknora-connection-card" id={`settings-weknora-connection-${environment}`}>
|
||||
<div className="kb-binding-grid">
|
||||
<div className={`kb-binding-side ${connection?.verification === 'ok' ? 'bound' : 'warn'}`}>
|
||||
<b>{title}</b>
|
||||
<span>连接:{verificationLabel(connection)}</span>
|
||||
<small>
|
||||
{connection?.endpoint
|
||||
? `地址:${connection.endpoint} · 密钥:${keyLabel(connection)}`
|
||||
: '还没有配置地址与密钥'}
|
||||
</small>
|
||||
</div>
|
||||
</div>
|
||||
{connection && !connection.storage_ready ? (
|
||||
<p className="hint">密钥存储不可用(主密钥文件缺失或权限不是 0600):可以读取当前状态,但保存会被拒绝。</p>
|
||||
) : null}
|
||||
{isAdmin ? (
|
||||
<div className="kb-binding-editor">
|
||||
<label>
|
||||
{title}地址
|
||||
<input
|
||||
value={endpointDraft}
|
||||
onChange={(event) => {
|
||||
setEndpointDraft(event.target.value)
|
||||
setTestedEndpoint('')
|
||||
}}
|
||||
placeholder={placeholder}
|
||||
aria-label={`${title}地址`}
|
||||
/>
|
||||
</label>
|
||||
<label>
|
||||
API Key
|
||||
<PasswordInput
|
||||
autoComplete="new-password"
|
||||
value={keyDraft}
|
||||
onChange={(event) => {
|
||||
setKeyDraft(event.target.value)
|
||||
setTestedEndpoint('')
|
||||
}}
|
||||
placeholder={connection?.key_configured ? '留空 = 保留原密钥' : '粘贴 API Key'}
|
||||
aria-label={`${title} API Key`}
|
||||
/>
|
||||
</label>
|
||||
<p className="hint">
|
||||
「测试连接」会真实请求该实例的知识库列表,并区分地址不可达 / 密钥无效 / 无权限 /
|
||||
成功;换了地址或密钥必须先测通才能保存。撤销绑定请把下方目标库留空。
|
||||
</p>
|
||||
<div className="kb-binding-actions">
|
||||
<button onClick={() => void test()} disabled={!canTest}>
|
||||
{testing ? '正在测试…' : '测试连接'}
|
||||
</button>
|
||||
<button onClick={() => void save(saveAction)} disabled={!canSave}>
|
||||
{saving ? '正在保存…' : '保存连接'}
|
||||
</button>
|
||||
{connection?.key_configured ? (
|
||||
<button className="danger" onClick={clearKey} disabled={saving || testing}>
|
||||
清除密钥
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
{error ? (
|
||||
<p className="error-message" role="alert">
|
||||
{error}
|
||||
</p>
|
||||
) : null}
|
||||
{message ? (
|
||||
<p className="success-message" role="status">
|
||||
{message}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
) : (
|
||||
<p className="hint">仅管理员可修改连接与密钥;当前密钥只显示尾 4 位,任何接口都不会回读明文。</p>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -11,6 +11,8 @@ export interface ProductionBindings {
|
||||
production_base_url: string
|
||||
production_connected: boolean
|
||||
production_message: string
|
||||
frozen: number
|
||||
retargetable: number
|
||||
}
|
||||
|
||||
const record = (value: unknown): value is Record<string, unknown> => Boolean(value) && typeof value === 'object' && !Array.isArray(value)
|
||||
@@ -25,6 +27,8 @@ function projection(item: Record<string, unknown>): ProductionBindings {
|
||||
production_base_url: typeof item.production_base_url === 'string' ? item.production_base_url : '',
|
||||
production_connected: item.production_connected === true,
|
||||
production_message: typeof item.production_message === 'string' ? item.production_message : '',
|
||||
frozen: typeof item.frozen === 'number' ? item.frozen : 0,
|
||||
retargetable: typeof item.retargetable === 'number' ? item.retargetable : 0,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -5,6 +5,8 @@ export interface RoutingBindings {
|
||||
external_kb_id: string
|
||||
internal_env_fallback: boolean
|
||||
external_env_fallback: boolean
|
||||
internal_fallback_kb_id: string
|
||||
external_fallback_kb_id: string
|
||||
internal_bound_by_ui: boolean
|
||||
external_bound_by_ui: boolean
|
||||
internal_kb_name: string
|
||||
@@ -30,6 +32,8 @@ function bindingProjection(item: Record<string, unknown>): RoutingBindings {
|
||||
external_kb_id: typeof item.external_kb_id === 'string' ? item.external_kb_id : '',
|
||||
internal_env_fallback: item.internal_env_fallback === true,
|
||||
external_env_fallback: item.external_env_fallback === true,
|
||||
internal_fallback_kb_id: typeof item.internal_fallback_kb_id === 'string' ? item.internal_fallback_kb_id : '',
|
||||
external_fallback_kb_id: typeof item.external_fallback_kb_id === 'string' ? item.external_fallback_kb_id : '',
|
||||
internal_bound_by_ui: item.internal_bound_by_ui === true,
|
||||
external_bound_by_ui: item.external_bound_by_ui === true,
|
||||
internal_kb_name: typeof item.internal_kb_name === 'string' ? item.internal_kb_name : '',
|
||||
|
||||
@@ -882,3 +882,29 @@ button.link {
|
||||
font-size: 0.8rem;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
/* 连接卡:地址 + 写-only 密钥 + 真实列库测试。它嵌在既有的「测试/生产知识库绑定」
|
||||
分区里,所以只做一个轻量内框,不另起一套分区样式。 */
|
||||
.weknora-connection-card {
|
||||
display: grid;
|
||||
gap: 8px;
|
||||
margin: 10px 0 16px;
|
||||
padding: 12px;
|
||||
border: 1px solid var(--border-base);
|
||||
border-radius: 10px;
|
||||
background: var(--bg-soft);
|
||||
}
|
||||
.weknora-connection-card .kb-binding-grid {
|
||||
margin: 0;
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.kb-binding-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
.kb-binding-actions button.danger {
|
||||
border-color: var(--c-fail);
|
||||
background: #fff;
|
||||
color: var(--c-fail);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import {
|
||||
fetchWeKnoraConnections,
|
||||
keyLabel,
|
||||
saveWeKnoraConnection,
|
||||
testWeKnoraConnection,
|
||||
verificationLabel,
|
||||
} from './weknoraConnections'
|
||||
import type { WeKnoraConnection, WeKnoraVerification } from './weknoraConnections'
|
||||
|
||||
const connection: WeKnoraConnection = {
|
||||
environment: 'production',
|
||||
endpoint: 'http://172.29.151.11:18093',
|
||||
key_configured: true,
|
||||
key_tail: '9f2c',
|
||||
verification: 'ok',
|
||||
library_count: 4,
|
||||
checked_at: '2026-11-22T07:00:00Z',
|
||||
updated_by: 'admin',
|
||||
revision: 3,
|
||||
storage_ready: true,
|
||||
}
|
||||
|
||||
describe('WeKnora connection API', () => {
|
||||
it('projects only the safe fields and never surfaces a key', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn().mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
data: [{ ...connection, api_key: 'placeholder-key-value', key_ciphertext: 'AAAA' }],
|
||||
}),
|
||||
{ status: 200 }
|
||||
)
|
||||
)
|
||||
)
|
||||
const [projected] = await fetchWeKnoraConnections()
|
||||
expect(projected.key_tail).toBe('9f2c')
|
||||
expect(projected).not.toHaveProperty('api_key')
|
||||
expect(projected).not.toHaveProperty('key_ciphertext')
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('renders the four answers distinctly', () => {
|
||||
const withVerification = (verification: WeKnoraVerification): WeKnoraConnection => ({ ...connection, verification })
|
||||
expect(verificationLabel({ ...connection, library_count: 4 })).toBe('可达 · 已读到 4 个知识库')
|
||||
expect(verificationLabel(withVerification('key_invalid'))).toBe('密钥无效')
|
||||
expect(verificationLabel(withVerification('key_forbidden'))).toBe('密钥无权限读取知识库列表')
|
||||
expect(verificationLabel(withVerification('unreachable'))).toBe('地址不可达')
|
||||
expect(verificationLabel(null)).toBe('未配置')
|
||||
expect(keyLabel(connection)).toBe('已配置 · 尾9f2c')
|
||||
expect(keyLabel({ ...connection, key_configured: false, key_tail: '' })).toBe('未配置')
|
||||
})
|
||||
|
||||
it('tests through the environment route and keeps the outcome', async () => {
|
||||
const fetchMock = vi.fn().mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
data: {
|
||||
environment: 'production',
|
||||
endpoint: connection.endpoint,
|
||||
outcome: 'key_forbidden',
|
||||
library_count: 0,
|
||||
message: '密钥无权限读取知识库列表(403)',
|
||||
checked_at: '2026-11-22T07:00:00Z',
|
||||
},
|
||||
}),
|
||||
{ status: 200 }
|
||||
)
|
||||
)
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
const result = await testWeKnoraConnection('production', {
|
||||
endpoint: connection.endpoint,
|
||||
api_key: 'sk-live-x',
|
||||
key_action: 'replace',
|
||||
})
|
||||
expect(result.outcome).toBe('key_forbidden')
|
||||
expect(result.message).toContain('无权限')
|
||||
expect(fetchMock.mock.calls[0][0]).toBe('/api/v1/weknora-connections/production/test')
|
||||
expect(JSON.parse(fetchMock.mock.calls[0][1].body)).toMatchObject({ key_action: 'replace' })
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('saves with the revision and an idempotency key', async () => {
|
||||
const fetchMock = vi.fn().mockResolvedValue(new Response(JSON.stringify({ data: connection }), { status: 200 }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
const saved = await saveWeKnoraConnection('production', {
|
||||
endpoint: connection.endpoint,
|
||||
key_action: 'keep',
|
||||
base_revision: 2,
|
||||
})
|
||||
expect(saved.revision).toBe(3)
|
||||
const [url, init] = fetchMock.mock.calls[0]
|
||||
expect(url).toBe('/api/v1/weknora-connections/production')
|
||||
expect(init.method).toBe('PUT')
|
||||
expect(init.headers['Idempotency-Key']).toBeTruthy()
|
||||
expect(JSON.parse(init.body)).toMatchObject({ key_action: 'keep', base_revision: 2 })
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
it('surfaces the server reason when a save is refused', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi
|
||||
.fn()
|
||||
.mockResolvedValue(
|
||||
new Response(JSON.stringify({ error: '请先点「测试连接」并成功,才能保存这一侧的配置' }), { status: 400 })
|
||||
)
|
||||
)
|
||||
await expect(
|
||||
saveWeKnoraConnection('test', { endpoint: 'http://127.0.0.1:8092', key_action: 'keep', base_revision: 0 })
|
||||
).rejects.toThrow('测试连接')
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,145 @@
|
||||
import { newOperationID } from './operationKey'
|
||||
|
||||
// 连接状态:地址 + 「有没有配过密钥」+ 上一次真实列库的结论。密钥本身永远不回到
|
||||
// 页面,只回显尾 4 位。
|
||||
export type WeKnoraVerification = 'not_configured' | 'ok' | 'unreachable' | 'key_invalid' | 'key_forbidden'
|
||||
export type WeKnoraTestOutcome = 'ok' | 'unreachable' | 'key_invalid' | 'key_forbidden'
|
||||
export type WeKnoraEnvironment = 'test' | 'production'
|
||||
|
||||
export interface WeKnoraConnection {
|
||||
environment: WeKnoraEnvironment
|
||||
endpoint: string
|
||||
key_configured: boolean
|
||||
key_tail: string
|
||||
verification: WeKnoraVerification
|
||||
library_count: number
|
||||
checked_at: string | null
|
||||
updated_by: string
|
||||
revision: number
|
||||
storage_ready: boolean
|
||||
}
|
||||
|
||||
export interface WeKnoraConnectionTestResult {
|
||||
environment: WeKnoraEnvironment
|
||||
endpoint: string
|
||||
outcome: WeKnoraTestOutcome
|
||||
library_count: number
|
||||
message: string
|
||||
checked_at: string
|
||||
}
|
||||
|
||||
type KeyAction = 'keep' | 'replace' | 'clear'
|
||||
|
||||
const record = (value: unknown): value is Record<string, unknown> =>
|
||||
Boolean(value) && typeof value === 'object' && !Array.isArray(value)
|
||||
|
||||
const verificationOf = (value: unknown): WeKnoraVerification =>
|
||||
value === 'ok' ||
|
||||
value === 'unreachable' ||
|
||||
value === 'key_invalid' ||
|
||||
value === 'key_forbidden' ||
|
||||
value === 'not_configured'
|
||||
? value
|
||||
: 'not_configured'
|
||||
|
||||
function projection(item: Record<string, unknown>): WeKnoraConnection {
|
||||
const environment = item.environment === 'production' ? 'production' : 'test'
|
||||
return {
|
||||
environment,
|
||||
endpoint: typeof item.endpoint === 'string' ? item.endpoint : '',
|
||||
key_configured: item.key_configured === true,
|
||||
key_tail: typeof item.key_tail === 'string' ? item.key_tail : '',
|
||||
verification: verificationOf(item.verification),
|
||||
library_count: typeof item.library_count === 'number' ? item.library_count : 0,
|
||||
checked_at: typeof item.checked_at === 'string' ? item.checked_at : null,
|
||||
updated_by: typeof item.updated_by === 'string' ? item.updated_by : '',
|
||||
revision: typeof item.revision === 'number' ? item.revision : 0,
|
||||
storage_ready: item.storage_ready === true,
|
||||
}
|
||||
}
|
||||
|
||||
async function reason(response: Response, fallback: string): Promise<string> {
|
||||
try {
|
||||
const body = (await response.json()) as unknown
|
||||
if (record(body) && typeof body.error === 'string' && body.error) return body.error
|
||||
} catch {
|
||||
/* keep the status text */
|
||||
}
|
||||
return `${fallback}(HTTP ${response.status})`
|
||||
}
|
||||
|
||||
export async function fetchWeKnoraConnections(signal?: AbortSignal): Promise<WeKnoraConnection[]> {
|
||||
const response = await fetch('/api/v1/weknora-connections', { signal, headers: { Accept: 'application/json' } })
|
||||
if (!response.ok) throw new Error(await reason(response, '读取 WeKnora 连接失败'))
|
||||
const value = (await response.json()) as unknown
|
||||
if (!record(value) || !Array.isArray(value.data)) throw new Error('WeKnora 连接响应无效')
|
||||
return value.data.filter(record).map(projection)
|
||||
}
|
||||
|
||||
export async function testWeKnoraConnection(
|
||||
environment: WeKnoraEnvironment,
|
||||
input: { endpoint: string; api_key?: string; key_action?: KeyAction },
|
||||
signal?: AbortSignal
|
||||
): Promise<WeKnoraConnectionTestResult> {
|
||||
const response = await fetch(`/api/v1/weknora-connections/${environment}/test`, {
|
||||
method: 'POST',
|
||||
signal,
|
||||
headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
|
||||
body: JSON.stringify(input),
|
||||
})
|
||||
if (!response.ok) throw new Error(await reason(response, '测试连接失败'))
|
||||
const value = (await response.json()) as unknown
|
||||
if (!record(value) || !record(value.data)) throw new Error('测试连接响应无效')
|
||||
const data = value.data
|
||||
return {
|
||||
environment: data.environment === 'production' ? 'production' : 'test',
|
||||
endpoint: typeof data.endpoint === 'string' ? data.endpoint : input.endpoint,
|
||||
outcome:
|
||||
data.outcome === 'key_invalid' || data.outcome === 'key_forbidden' || data.outcome === 'unreachable'
|
||||
? data.outcome
|
||||
: 'ok',
|
||||
library_count: typeof data.library_count === 'number' ? data.library_count : 0,
|
||||
message: typeof data.message === 'string' ? data.message : '',
|
||||
checked_at: typeof data.checked_at === 'string' ? data.checked_at : '',
|
||||
}
|
||||
}
|
||||
|
||||
export async function saveWeKnoraConnection(
|
||||
environment: WeKnoraEnvironment,
|
||||
input: { endpoint: string; api_key?: string; key_action: KeyAction; base_revision: number },
|
||||
signal?: AbortSignal
|
||||
): Promise<WeKnoraConnection> {
|
||||
const response = await fetch(`/api/v1/weknora-connections/${environment}`, {
|
||||
method: 'PUT',
|
||||
signal,
|
||||
headers: { 'Content-Type': 'application/json', Accept: 'application/json', 'Idempotency-Key': newOperationID() },
|
||||
body: JSON.stringify(input),
|
||||
})
|
||||
if (!response.ok) throw new Error(await reason(response, '保存连接失败'))
|
||||
const value = (await response.json()) as unknown
|
||||
if (!record(value) || !record(value.data)) throw new Error('保存连接响应无效')
|
||||
return projection(value.data)
|
||||
}
|
||||
|
||||
// verificationLabel 是页面唯一的口径:四态说清「哪里不通」,成功时带库数量。
|
||||
export function verificationLabel(connection: WeKnoraConnection | null): string {
|
||||
if (!connection || !connection.endpoint) return '未配置'
|
||||
switch (connection.verification) {
|
||||
case 'ok':
|
||||
return `可达 · 已读到 ${connection.library_count} 个知识库`
|
||||
case 'key_invalid':
|
||||
return '密钥无效'
|
||||
case 'key_forbidden':
|
||||
return '密钥无权限读取知识库列表'
|
||||
case 'unreachable':
|
||||
return '地址不可达'
|
||||
default:
|
||||
// 填了地址但还没测过:说「尚未验证」,不要谎报不可达。
|
||||
return '尚未验证'
|
||||
}
|
||||
}
|
||||
|
||||
export function keyLabel(connection: WeKnoraConnection | null): string {
|
||||
if (!connection || !connection.key_configured) return '未配置'
|
||||
return `已配置 · 尾${connection.key_tail || '—'}`
|
||||
}
|
||||
Reference in New Issue
Block a user