Merge pull request 'Fix/development version history' (#8) from fix/development-version-history into main
build / build (push) Successful in 55s
build / build (push) Successful in 55s
Reviewed-on: #8
This commit was merged in pull request #8.
This commit is contained in:
@@ -1,3 +1,26 @@
|
||||
# Optional independent source, disabled by default. Never use the app writer or
|
||||
# substitute DATABASE_URL. Backend credential file must be a regular 0600 file.
|
||||
DOBOTAI_OPS_READ_ENABLED=false
|
||||
DOBOTAI_OPS_READ_DSN_FILE=
|
||||
DOBOTAI_OPS_TENANT_KEY=
|
||||
DOBOTAI_OPS_DIAL_MODE=direct
|
||||
DOBOTAI_OPS_SSH_TARGET=
|
||||
DOBOTAI_OPS_CONNECT_TIMEOUT=5s
|
||||
DOBOTAI_OPS_QUERY_TIMEOUT=4s
|
||||
DOBOTAI_OPS_MAX_CONNS=2
|
||||
DOBOTAI_OPS_MIN_CONNS=0
|
||||
|
||||
# API and worker share this opt-in binding; no browser-selected paths/credentials.
|
||||
# Runtime root is the read-only existing weekly_report directory (SDK/config and
|
||||
# private secret file). Python must be compatible and already provide PyYAML.
|
||||
# Container deployment requires approved runtime/script mounts; not auto-installed.
|
||||
DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED=false
|
||||
DOBOTAI_OPS_DIRECTORY_SYNC_PYTHON=
|
||||
DOBOTAI_OPS_DIRECTORY_SYNC_SCRIPT=
|
||||
DOBOTAI_OPS_DIRECTORY_SYNC_RUNTIME_ROOT=
|
||||
DOBOTAI_OPS_DIRECTORY_SYNC_SPOOL_ROOT=
|
||||
DOBOTAI_OPS_DIRECTORY_SYNC_TIMEOUT=20m
|
||||
|
||||
APP_ADDR=:8080
|
||||
DATABASE_URL=postgres://governance@postgres:5432/governance?sslmode=disable
|
||||
PGPASSFILE=/run/secrets/pgpass
|
||||
|
||||
+310
-2
@@ -4,7 +4,26 @@
|
||||
|
||||
### Added
|
||||
|
||||
- **「版本记录」页**（一级菜单第 5 项）：只读时间线追踪**远端 main 可达的全部提交**（含合并提交与已并入 main 的侧分支提交），按提交时间倒序显示标题/正文、`[正式版本] / [开发版本] / [历史提交]` 徽标、短 SHA、提交内变更记录与 `[查看提交]`/`[查看 tag]`；支持「全部 / 正式 / 开发 / 历史」筛选与游标翻页，窄屏单列。
|
||||
- 问答记录新增“本周”快捷查询，默认展示北京时间本周数据；所选范围自动读取全部记录，校验完整数量后展示，不再需要分页或加载更多。固定展示完整字段，空列不再隐藏，中文原文直接展示、外语未翻译明确标记；保留源端有界传输、取消和失败保护。
|
||||
|
||||
- 问答记录新增年份与自然周选择，周次展示起止日期，支持跨年及每年实际周数；按北京时间周一至周日填入查询范围，默认本周且仅查询截至当前的数据。保留自定义时间与原查询按钮，浏览器时区不改变查询边界。
|
||||
|
||||
- 问答运营按已确认口径显示知识覆盖率、回答正确率及原始分子/分母、排除数量与构成。使用完整筛选范围和当前有效人工优先结论，精确规则命中的未覆盖可进入覆盖分母；部分覆盖不算完全覆盖，待确认/未评估/失效/不计入按对应指标排除，零分母显示“暂无可计算数据”。返回独立规则版本，AI 模型与绩效范围仍未接入。
|
||||
|
||||
- 问答详情支持治理侧人工复核及不可变审计：覆盖、正确性、产品分类、问题类型、根因与理由由管理员明确填写，操作人取登录身份；保存后独立读回复核历史。保留源端判断和自动规则，输入变化使旧人工结论失效，已观测的内容回退不自动恢复旧结论；并发修改冲突、重复请求不重复留档。
|
||||
- 新增覆盖复检历史、已采集输入快照和完整筛选范围质量数量。批量统计只传指纹与规则命中元数据，原文在读取问答/复核时采集，未采集版本明确提示而非伪造空快照；源事务时点阻止延迟旧扫描回退新输入。基础数量与本期经确认新增的质量比例分开显示；不接 AI 模型、不定义绩效分母、不回写 DobotAI 原问答。
|
||||
|
||||
- 「使用统计 → 个人月明细」按北京时间自然月及截止时点展示当前应用可见名册的人员用量（包含本源零记录），支持部门含子部门、区域/待核对状态、姓名/完整部门路径和已有提问/零记录筛选。稳定身份关联去重、不可关联问答单列、过期组织版本阻断翻页；源端姓名可缺失并提示重新同步，不按姓名匹配身份。姓名筛选通过有界只读 POST 请求体传递、不进入 URL，原始员工标识不返回浏览器；名册不作为在职/考核分母，当月进度不作为绩效结论。
|
||||
|
||||
- 问答运营按稳定身份关联当前治理通讯录，缺失部门/区域补全并保留源快照及目录/区域版本；空列判断包含完整筛选结果，不混合配置已变化的分页。新增完整时间/关键词范围的使用统计，问答计次、名册内人员去重，跨区/未匹配/身份缺失分列，不输出绩效结论；不回写 DobotAI 原表。
|
||||
- 「问答运营 → 区域使用」支持管理员从网页发起持久化通讯录同步，重复请求重放、进行中操作合并；按服务器固定范围注册后台执行器，未接线或未检测到心跳时不创建新任务。界面显示真实任务状态、不编造百分比；失败保留旧快照与区域规则，同步完成不覆盖未保存草稿。旧执行器丢失租约后不能激活结果，重启接管和已提交快照的续跑使用原任务。
|
||||
- 新增治理侧通讯录实时桥接，复用既有周报读取能力，向源端读取真实缺失祖先以补齐部门树；不改旧周报脚本、配置或输出。私有 JSON 与受控子进程执行器拒绝不完整数据、模拟同步、公开文件和异常输出，超时清理临时人员文件。同步默认关闭，容器启用仍需批准的既有周报 SDK/配置及兼容 Python 运行时挂载。
|
||||
- 区域归属表支持长部门路径换行，避免继承全局不换行规则后将生效区域与归属依据列挤出桌面面板。
|
||||
- **「问答运营 → 区域使用」区域归属配置切片**：按稳定部门 ID 搜索和勾选完整路径，预览去重归属人数，保存版本后重新读取；父部门包含子部门、子部门显式配置优先，多部门跨区域人员单独待核对。预览不写入，过期版本和不完整部门树被阻断。基础使用统计见本期新增，不代表绩效验收；翻译及 AI 评估模型接入按用户要求暂缓。
|
||||
- 新增治理侧组织快照与区域修订迁移、受确认保护的后台导入命令；不修改 DobotAI 原表或钉钉组织结构。
|
||||
- **「问答运营 → 问答记录」首个开发切片**：在现有导航中查看 DobotAI 2.0 钉钉问答，支持半开时间范围、原问题关键词、稳定游标分页和完整详情；按整个筛选结果隐藏空列，反馈聚合不放大问答条数。外语翻译、AI 评估尚未接入，不标为第一阶段完成。
|
||||
- 治理侧重新检查答案：包含 `暂无已发布` 即未覆盖，引用和源端 covered 不能抵消；未命中仍待评估，源端规则单独保留作对照。独立只读源默认关闭、租户后端固定、管理员会话鉴权，源故障不回退治理主库；安全告警明确保留 TEMP/TLS 尚未整改状态。
|
||||
- **「版本记录」页**（一级入口）：只读时间线追踪**远端 main 可达的全部提交**（含合并提交与已并入 main 的侧分支提交），按提交时间倒序显示标题/正文、`[正式版本] / [开发版本] / [历史提交]` 徽标、短 SHA、提交内变更记录与 `[查看提交]`/`[查看 tag]`；支持「全部 / 正式 / 开发 / 历史」筛选与游标翻页，窄屏单列。
|
||||
- 新增只读接口 `GET /api/v1/build-info`（当前运行产物的构建身份）、`GET /api/v1/version-history`（快照游标分页，不返回伪造的 total）、`POST /api/v1/version-history/sync`（仅管理员手动同步，并发同步复用同一作业）。
|
||||
- 新增迁移 `000046_version_history`（提交记录 + 同步状态）：`commit_sha` 是唯一身份，**版本号不是唯一键** —— 两个提交算出同一个开发号时各占一行，用 SHA 区分。
|
||||
- 「当前运行」只认运行产物注入的**完整 SHA 精确匹配**：匹配到就标该条；开发构建显式标注"开发构建"；拿不到标识就显示"当前运行版本未知"，**不拿最新 main 或最新 tag 顶替**；CI 成功不等于已部署。
|
||||
@@ -18,6 +37,8 @@
|
||||
|
||||
### Fixed
|
||||
|
||||
- 修复通过带端口的 HTTP 地址访问问答运营时，正常同源操作被误报“拒绝跨站配置请求”：网页代理保留原始 Host 及端口，使缺少 Fetch Metadata 的浏览器也能正确执行 Origin 校验。保留后端跨站、异端口、伪造转发头与缺失操作意图的拒绝保护。
|
||||
|
||||
- **登记钉钉知识库失败（HTTP 409 `space_registration_conflict`）的真实原因与修复**：`source_connections.name` 有唯一约束，而早期路径登记的钉钉来源（`base_url` 为空、id 与 dws 现在推导的确定性 id 不同）被**归档后仍占着名字槽位** ⇒ 重新登记时 INSERT 撞唯一约束；处理器又把**所有**存储层错误都翻译成同一个 409，真实原因被吞掉（线上只能看到一个无细节的错误码）。现在：① 同名来源**已归档**时，把它改名腾位（`…（已下架 · <短ID>）`，归档行不显示、数据与审计一行不动）；② 同名来源**仍在使用**时返回 `space_name_conflict` + 可行动提示（先「去掉来源」）；③ 同一登记键指向别的空间且该行仍在用时返回 `space_identity_conflict`（已归档的行则允许就地改写 —— 这就是"去掉后再用 dws 重新获取"）；④ 其余内部故障改回 **500** `space_registration_failed`，不再伪装成冲突；前端优先显示后端 `hint`，并为这两个新码补了中文说明。
|
||||
|
||||
- 前端入口与静态资源的缓存语义（"新功能看起来没上线"与"页面偶尔裸奔/空白"的根因）：
|
||||
@@ -52,9 +73,296 @@
|
||||
> **版本怎么走**（两行就够）：
|
||||
> · **正式版本** = 下面那条 `## [x.y.z] - 日期`，是**发版事实**，由人批准后打注释 tag `vX.Y.Z`；
|
||||
> · **开发版本自动递增** = `<正式版本的主.次>.<自那个 tag 以来的提交数>`（正式 `0.1.0` 之后第 12 个提交 =
|
||||
> `0.1.12`），由 `scripts/version.py` 从 git **算出来**，**不写进本文件** ⇒ 每个提交都有自己的版本号，
|
||||
> `0.1.12`），由 `scripts/version.py` 从 git **算出来**；开发交付的独立说明写入本文件“开发快照”章节，并打 `snapshot/vX.Y.Z` 注释标签 ⇒ 每次开发交付均可追溯，
|
||||
> 但"发布版本"仍然只能由人定（CI 不会自动升版本、不会打 tag）。
|
||||
|
||||
## 开发快照 snapshot/v0.1.27 - 2026-10-18
|
||||
|
||||
### Fixed
|
||||
|
||||
- 并行分支存在同号开发提交时，历史变更说明与补记标签计划按完整 SHA 选择，不再被前一条同号记录遮蔽；不强改推导版本号或移动既有标签。
|
||||
|
||||
## 开发快照 snapshot/v0.1.26 - 2026-10-18
|
||||
|
||||
### Fixed
|
||||
|
||||
- 版本记录收录已交付开发快照与当前运行提交，不再只同步 main；开发版本显示对应的独立变更说明。
|
||||
- 修复同步重复返回旧已完成任务：进行中同步合并，完成后允许新一轮同步。
|
||||
- 构建启动后自动请求版本同步；部署后独立读回当前完整 SHA、开发标签与变更说明，缺失时不宣称交付完成。
|
||||
|
||||
### Added
|
||||
|
||||
- 增加开发快照计划、显式发布及远端读回工具；历史说明按原提交补记，正式 tag 与 Release 保持独立授权。
|
||||
|
||||
## 开发快照 snapshot/v0.1.25 - 2026-10-18
|
||||
|
||||
目标提交：90901a4458054aca721047c9a2896f2130d46b5c
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 问答记录新增“本周”快捷查询，默认展示北京时间本周数据；所选范围自动读取全部记录，校验完整数量后展示，不再需要分页或加载更多。固定展示完整字段，空列不再隐藏，中文原文直接展示、外语未翻译明确标记；保留源端有界传输、取消和失败保护。
|
||||
|
||||
## 开发快照 snapshot/v0.1.24 - 2026-10-18
|
||||
|
||||
目标提交：3566f208e0e15fdec2cc98c29f93c46f5f7c91ca
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 问答记录新增年份与自然周选择，周次展示起止日期，支持跨年及每年实际周数；按北京时间周一至周日填入查询范围，默认本周且仅查询截至当前的数据。保留自定义时间与原查询按钮，浏览器时区不改变查询边界。
|
||||
|
||||
## 开发快照 snapshot/v0.1.23 - 2026-10-18
|
||||
|
||||
目标提交：2ea2111420e501fec37e0ea0c4bcd3cd8c866403
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 修复通过带端口的 HTTP 地址访问问答运营时，正常同源操作被误报“拒绝跨站配置请求”：网页代理保留原始 Host 及端口，使缺少 Fetch Metadata 的浏览器也能正确执行 Origin 校验。保留后端跨站、异端口、伪造转发头与缺失操作意图的拒绝保护。
|
||||
|
||||
## 开发快照 snapshot/v0.1.22 - 2026-10-18
|
||||
|
||||
目标提交：828c2c5fa0d4e3e0eee3987085a4d782872f7e95
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 问答运营按已确认口径显示知识覆盖率、回答正确率及原始分子/分母、排除数量与构成。使用完整筛选范围和当前有效人工优先结论，精确规则命中的未覆盖可进入覆盖分母；部分覆盖不算完全覆盖，待确认/未评估/失效/不计入按对应指标排除，零分母显示“暂无可计算数据”。返回独立规则版本，AI 模型与绩效范围仍未接入。
|
||||
- 新增覆盖复检历史、已采集输入快照和完整筛选范围质量数量。批量统计只传指纹与规则命中元数据，原文在读取问答/复核时采集，未采集版本明确提示而非伪造空快照；源事务时点阻止延迟旧扫描回退新输入。基础数量与本期经确认新增的质量比例分开显示；不接 AI 模型、不定义绩效分母、不回写 DobotAI 原问答。
|
||||
|
||||
## 开发快照 snapshot/v0.1.21 - 2026-10-17
|
||||
|
||||
目标提交：4b3aaff3219e78ae3dfd5d12ba36f82e323a21ba
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 问答详情支持治理侧人工复核及不可变审计：覆盖、正确性、产品分类、问题类型、根因与理由由管理员明确填写，操作人取登录身份；保存后独立读回复核历史。保留源端判断和自动规则，输入变化使旧人工结论失效，已观测的内容回退不自动恢复旧结论；并发修改冲突、重复请求不重复留档。
|
||||
- 新增覆盖复检历史、已采集输入快照和完整筛选范围质量数量。批量统计只传指纹与规则命中元数据，原文在读取问答/复核时采集，未采集版本明确提示而非伪造空快照；源事务时点阻止延迟旧扫描回退新输入。仅提供数量，不接 AI 模型、不定义质量率或绩效分母、不回写 DobotAI 原问答。
|
||||
|
||||
## 开发快照 snapshot/v0.1.20 - 2026-10-17
|
||||
|
||||
目标提交：9556db1ef01add21064553f64be6a7e25f838e9e
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 「使用统计 → 个人月明细」按北京时间自然月及截止时点展示当前应用可见名册的人员用量（包含本源零记录），支持部门含子部门、区域/待核对状态、姓名/完整部门路径和已有提问/零记录筛选。稳定身份关联去重、不可关联问答单列、过期组织版本阻断翻页；源端姓名可缺失并提示重新同步，不按姓名匹配身份。姓名筛选通过有界只读 POST 请求体传递、不进入 URL，原始员工标识不返回浏览器；名册不作为在职/考核分母，当月进度不作为绩效结论。
|
||||
|
||||
## 开发快照 snapshot/v0.1.19 - 2026-10-17
|
||||
|
||||
目标提交：56b8cabed9213911b6beb4aeaac9baa22e872a00
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 问答运营按稳定身份关联当前治理通讯录，缺失部门/区域补全并保留源快照及目录/区域版本；空列判断包含完整筛选结果，不混合配置已变化的分页。新增完整时间/关键词范围的使用统计，问答计次、名册内人员去重，跨区/未匹配/身份缺失分列，不输出绩效结论；不回写 DobotAI 原表。
|
||||
- 新增治理侧通讯录实时桥接，复用既有周报读取能力，向源端读取真实缺失祖先以补齐部门树；不改旧周报脚本、配置或输出。私有 JSON 与受控子进程执行器拒绝不完整数据、模拟同步、公开文件和异常输出，超时清理临时人员文件。同步默认关闭，容器启用仍需批准的既有周报 SDK/配置及兼容 Python 运行时挂载。
|
||||
- **「问答运营 → 区域使用」区域归属配置切片**：按稳定部门 ID 搜索和勾选完整路径，预览去重归属人数，保存版本后重新读取；父部门包含子部门、子部门显式配置优先，多部门跨区域人员单独待核对。预览不写入，过期版本和不完整部门树被阻断。基础使用统计见本期新增，不代表绩效验收；翻译及 AI 评估模型接入按用户要求暂缓。
|
||||
- **「问答运营 → 问答记录」首个开发切片**：在现有导航中查看 DobotAI 2.0 钉钉问答，支持半开时间范围、原问题关键词、稳定游标分页和完整详情；按整个筛选结果隐藏空列，反馈聚合不放大问答条数。外语翻译、AI 评估尚未接入，不标为第一阶段完成。
|
||||
|
||||
## 开发快照 snapshot/v0.1.18 - 2026-10-17
|
||||
|
||||
目标提交：cc4906b6a9f33683a5c0017d83b24d96d5c997b1
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 「问答运营 → 区域使用」支持管理员从网页发起持久化通讯录同步，重复请求重放、进行中操作合并；按服务器固定范围注册后台执行器，未接线或未检测到心跳时不创建新任务。界面显示真实任务状态、不编造百分比；失败保留旧快照与区域规则，同步完成不覆盖未保存草稿。旧执行器丢失租约后不能激活结果，重启接管和已提交快照的续跑使用原任务。
|
||||
- 新增治理侧通讯录实时桥接，复用既有周报读取能力，向源端读取真实缺失祖先以补齐部门树；不改旧周报脚本、配置或输出。私有 JSON 与受控子进程执行器拒绝不完整数据、模拟同步、公开文件和异常输出，超时清理临时人员文件；问答组织关联尚未接入。同步默认关闭，容器启用仍需批准的既有周报 SDK/配置及兼容 Python 运行时挂载。
|
||||
- **「问答运营 → 区域使用」区域归属配置切片**：按稳定部门 ID 搜索和勾选完整路径，预览去重归属人数，保存版本后重新读取；父部门包含子部门、子部门显式配置优先，多部门跨区域人员单独待核对。预览不写入，过期版本和不完整部门树被阻断。当前不是使用/绩效统计验收，翻译及 AI 评估模型接入按用户要求暂缓。
|
||||
|
||||
## 开发快照 snapshot/v0.1.17 - 2026-10-17
|
||||
|
||||
目标提交：6adfdcdd4fe0e5c35018bf0e4857c203dd24e5d1
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 区域归属表支持长部门路径换行，避免继承全局不换行规则后将生效区域与归属依据列挤出桌面面板。
|
||||
|
||||
## 开发快照 snapshot/v0.1.16 - 2026-10-17
|
||||
|
||||
目标提交：5874134e049b0896a202b41d9d400d1be5386f89
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- 新增治理侧通讯录实时桥接，复用既有周报读取能力，向源端读取真实缺失祖先以补齐部门树；不改旧周报脚本、配置或输出。私有 JSON 与受控子进程执行器拒绝不完整数据、模拟同步、公开文件和异常输出，超时清理临时人员文件；网页触发的后台同步队列及问答组织关联尚未接入。
|
||||
|
||||
## 开发快照 snapshot/v0.1.15 - 2026-10-17
|
||||
|
||||
目标提交：dab74b706b5b87da3ecb71a8e0f6b6caa5fba4ad
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- **「问答运营 → 区域使用」区域归属配置切片**：按稳定部门 ID 搜索和勾选完整路径，预览去重归属人数，保存版本后重新读取；父部门包含子部门、子部门显式配置优先，多部门跨区域人员单独待核对。预览不写入，过期版本和不完整部门树被阻断。当前不是实时通讯录同步或使用/绩效统计验收，翻译及 AI 评估模型接入按用户要求暂缓。
|
||||
- 新增治理侧组织快照与区域修订迁移、受确认保护的后台导入命令；不修改 DobotAI 原表或钉钉组织结构。
|
||||
|
||||
## 开发快照 snapshot/v0.1.14 - 2026-10-17
|
||||
|
||||
目标提交：fd04bf7886aa4eba68b247dd7c1092894a5282da
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- **「问答运营 → 问答记录」首个开发切片**：在现有导航中查看 DobotAI 2.0 钉钉问答，支持半开时间范围、原问题关键词、稳定游标分页和完整详情；按整个筛选结果隐藏空列，反馈聚合不放大问答条数。外语翻译、AI 评估、人员区域映射尚未接入，不标为第一阶段完成。
|
||||
- 治理侧重新检查答案：包含 `暂无已发布` 即未覆盖，引用和源端 covered 不能抵消；未命中仍待评估，源端规则单独保留作对照。独立只读源默认关闭、租户后端固定、管理员会话鉴权，源故障不回退治理主库；安全告警明确保留 TEMP/TLS 尚未整改状态。
|
||||
- **「版本记录」页**（一级入口）：只读时间线追踪**远端 main 可达的全部提交**（含合并提交与已并入 main 的侧分支提交），按提交时间倒序显示标题/正文、`[正式版本] / [开发版本] / [历史提交]` 徽标、短 SHA、提交内变更记录与 `[查看提交]`/`[查看 tag]`；支持「全部 / 正式 / 开发 / 历史」筛选与游标翻页，窄屏单列。
|
||||
|
||||
## 开发快照 snapshot/v0.1.13 - 2026-10-14
|
||||
|
||||
目标提交：c1d49adb622f3bf18e51bddb415795a894ed227e
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- 补三Agent交接闭环工作流（状态机、快照tag与验收模板）。
|
||||
|
||||
## 开发快照 snapshot/v0.1.12 - 2026-10-14
|
||||
|
||||
目标提交：9a8a3e77956149d116360bbb561efd67e20fb062
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Fixed
|
||||
|
||||
- **登记钉钉知识库失败（HTTP 409 `space_registration_conflict`）的真实原因与修复**：`source_connections.name` 有唯一约束，而早期路径登记的钉钉来源（`base_url` 为空、id 与 dws 现在推导的确定性 id 不同）被**归档后仍占着名字槽位** ⇒ 重新登记时 INSERT 撞唯一约束；处理器又把**所有**存储层错误都翻译成同一个 409，真实原因被吞掉（线上只能看到一个无细节的错误码）。现在：① 同名来源**已归档**时，把它改名腾位（`…（已下架 · <短ID>）`，归档行不显示、数据与审计一行不动）；② 同名来源**仍在使用**时返回 `space_name_conflict` + 可行动提示（先「去掉来源」）；③ 同一登记键指向别的空间且该行仍在用时返回 `space_identity_conflict`（已归档的行则允许就地改写 —— 这就是"去掉后再用 dws 重新获取"）；④ 其余内部故障改回 **500** `space_registration_failed`，不再伪装成冲突；前端优先显示后端 `hint`，并为这两个新码补了中文说明。
|
||||
|
||||
## 开发快照 snapshot/v0.1.11 - 2026-10-14
|
||||
|
||||
目标提交：2c7265efc47a2cd6b324bf278fa5ddb0c4a9fe57
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- 「系统设置 → 数据源配置」**收敛成两个入口**：**dws 导入**（按当前钉钉账号列出可访问的知识库与钉盘空间）与 **PP 导入**。AList（已由钉盘文件替代）、静态登记的钉钉知识库、本地手工导入与 PP legacy 都**不再出现在这一页**；已接入的这类来源用**归档**下架（`POST /api/v1/source-connections/{id}/archive`，管理员、幂等，带写意图头与幂等键）：从列表、来源与目录、计划刷新与手动扫描里消失，**目录快照、发布记录与审计数据一行不删**。恢复用 `…/restore`（保持停用、不自动带回调度），或让 dws 重新登记同一来源 —— 钉钉空间登记是确定性 ID + `ON CONFLICT`，**原地复活同一行、不产生重复**。卡片上新增「去掉来源」按钮（管理员，带确认弹窗）。
|
||||
- 迁移 `000047_source_connection_archive`：新增 `archived_at` / `archived_by` 与只覆盖未归档行的部分索引；列表、目录、计划刷新、手动扫描四类查询一律排除归档行（漏一条就等于入口没去掉）。
|
||||
|
||||
## 开发快照 snapshot/v0.1.10 - 2026-10-14
|
||||
|
||||
目标提交：64b32c2da64881b3c9bf6de6b8f9af3f833128fd
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- 「系统设置 → 数据源配置」默认只列 PP 资料与「可加入的来源」：AList 来源（已由钉盘文件替代）与静态登记的钉钉知识库来源（改为按当前钉钉账号发现）**不再默认列出**，收进「显示 AList／钉钉知识库来源（N）」折叠开关 —— **来源一个都没删**，展开后照旧可改刷新周期与启停，「来源与目录」里的内容不变。手动导入来源（本地手工验收）不在这次范围内，照旧显示。
|
||||
|
||||
## 开发快照 snapshot/v0.1.9 - 2026-10-14
|
||||
|
||||
目标提交：b7f7e4f12101569db5bb76af9d836043eefc342b
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Fixed
|
||||
|
||||
- **上线前护栏 + 事故复盘**：`scripts/deploy-api-web.sh` 现在先校验 `.env` 的键值形状（值里再出现 `KEY=` 或键名不合法 ⇒ **拒绝上线**），配置校验失败时也会给出"两行粘成一行"的线索。事故：`.env` 最后一行没有结尾换行、又被追加新键 ⇒ 两行粘成一行 ⇒ `VERSION_HISTORY_GITEA_BASE_URL` 变成非法值 ⇒ api/worker 按设计拒绝启动、崩溃重启（平台整体不可用）。已修数据、加护栏、加回归用例。
|
||||
|
||||
## 开发快照 snapshot/v0.1.8 - 2026-10-14
|
||||
|
||||
目标提交：4cb51108454c3559e8a6305ad3316f74ed4e2daf
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 「版本记录」页的链接改为**抓取与链接分离**：抓取仍走配置的（内网）地址，页面上的「查看提交 / 查看 tag」用 `VERSION_HISTORY_LINK_ORIGIN`（公网 origin）拼 —— 原先两者共用内网地址，内网外的浏览器点开是空白；另外 tag 记录原先不带链接（空 `href` 点下去没反应），现在补上链接、没有链接就不渲染该按钮。
|
||||
|
||||
## 开发快照 snapshot/v0.1.7 - 2026-10-14
|
||||
|
||||
目标提交：90db243a68a5d16d643dd0273614b93666d65ca7
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 前端入口与静态资源的缓存语义（"新功能看起来没上线"与"页面偶尔裸奔/空白"的根因）：
|
||||
|
||||
## 开发快照 snapshot/v0.1.6 - 2026-10-14
|
||||
|
||||
目标提交：062a3b3cb60ede5ebcca64a0f14df53fffa1f359
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Fixed
|
||||
|
||||
- 前端入口的缓存策略：`index.html` 此前没有缓存头，CDN/浏览器会启发式缓存它，于是**页面会一直停在旧构建上**（实测被腾讯云 EdgeOne 卡住：源站已是新包，公网入口仍发旧包，看起来像"新菜单/新页面没上线"）。现在 `index.html` 每次回源校验（`no-cache, must-revalidate`）、带内容哈希的 `/assets/` 长缓存（`immutable`），旧哈希资源明确 404 而不是回退成 HTML。
|
||||
|
||||
## 开发快照 snapshot/v0.1.5 - 2026-10-14
|
||||
|
||||
目标提交：4d43a9386a8a97969a1568956213f6f17b13edfb
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- tag 之前的提交不再被编号 + 部署脚本注入构建身份。
|
||||
|
||||
## 开发快照 snapshot/v0.1.4 - 2026-10-13
|
||||
|
||||
目标提交：0974d8dfa2aad5650eefbcee3edb365a8afd4868
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Added
|
||||
|
||||
- **「版本记录」页**（一级菜单第 5 项）：只读时间线追踪**远端 main 可达的全部提交**（含合并提交与已并入 main 的侧分支提交），按提交时间倒序显示标题/正文、`[正式版本] / [开发版本] / [历史提交]` 徽标、短 SHA、提交内变更记录与 `[查看提交]`/`[查看 tag]`；支持「全部 / 正式 / 开发 / 历史」筛选与游标翻页，窄屏单列。
|
||||
- 新增只读接口 `GET /api/v1/build-info`（当前运行产物的构建身份）、`GET /api/v1/version-history`（快照游标分页，不返回伪造的 total）、`POST /api/v1/version-history/sync`（仅管理员手动同步，并发同步复用同一作业）。
|
||||
- 新增迁移 `000046_version_history`（提交记录 + 同步状态）：`commit_sha` 是唯一身份，**版本号不是唯一键** —— 两个提交算出同一个开发号时各占一行，用 SHA 区分。
|
||||
- 「当前运行」只认运行产物注入的**完整 SHA 精确匹配**：匹配到就标该条；开发构建显式标注"开发构建"；拿不到标识就显示"当前运行版本未知"，**不拿最新 main 或最新 tag 顶替**；CI 成功不等于已部署。
|
||||
|
||||
### Security
|
||||
|
||||
- 版本记录同步**匿名只读**远端仓库（**不使用任何凭据**），来源限配置里批准的 origin 与固定仓库：拒绝任意 URL、拒绝跨主机重定向，单个响应设大小上限与超时、5xx/429 才有退避重试。
|
||||
- 提交正文与变更记录一律按**纯文本**渲染（不产生 HTML/脚本注入面）；页面不含打 tag / 发布 / 部署入口；同步动作仅管理员可用，服务端再次判角色。
|
||||
|
||||
## 开发快照 snapshot/v0.1.3 - 2026-10-13
|
||||
|
||||
目标提交：ad9c5d20d467a077e88bbfe63b1a56355d263e51
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- Merge remote-tracking branch 'origin/main'。
|
||||
|
||||
## 开发快照 snapshot/v0.1.1 - 2026-10-13
|
||||
|
||||
目标提交：3a1d32da8fba90eb5081865b1f3ad424a46639ee
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- 重写平台定位与能力边界并归档原说明。
|
||||
|
||||
## 开发快照 snapshot/v0.1.1 - 2026-10-13
|
||||
|
||||
目标提交：950c1b429bc814d9fa7c2c1449f999a05537af21
|
||||
|
||||
历史开发快照补记，不改变原提交、原提交时间或正式发布状态。
|
||||
|
||||
### Changed
|
||||
|
||||
- 抬档门禁（破坏性 ⇒ 0.x 抬次 / ≥1.0.0 抬主）+ 1.0.0 前置（稳定契约承诺）。
|
||||
|
||||
## [0.1.0] - 2026-10-13
|
||||
|
||||
### Added
|
||||
|
||||
+115
-1
@@ -3,7 +3,7 @@
|
||||
"info": {
|
||||
"title": "Knowledge Governance API",
|
||||
"version": "v1",
|
||||
"description": "独立知识治理控制平面API。所有响应使用安全投影；不返回来源凭据、签名下载地址、原始内容、内部目标连接或生产密钥。"
|
||||
"description": "独立知识治理控制平面API。响应不返回来源凭据、内部目标连接或生产密钥。问答运营仅向已登录且获授权的管理员返回业务问答正文，禁止共享缓存。"
|
||||
},
|
||||
"servers": [
|
||||
{
|
||||
@@ -44,6 +44,120 @@
|
||||
}
|
||||
],
|
||||
"paths": {
|
||||
"/api/v1/qa-ops/organization/sync": {
|
||||
"get": {"tags":["QA operations"],"operationId":"getQAOrganizationSync","summary":"管理员读取通讯录同步执行器状态及最近任务的安全汇总，不返回人员名单或运行配置","responses":{"200":{"$ref":"#/components/responses/Envelope"},"400":{"$ref":"#/components/responses/BadRequest"},"401":{"description":"需要登录"},"403":{"description":"仅管理员"},"500":{"$ref":"#/components/responses/InternalError"},"503":{"description":"状态存储不可用，不伪装为空记录"}}},
|
||||
"post": {"tags":["QA operations"],"operationId":"startQAOrganizationSync","summary":"管理员同源发起持久化通讯录同步；重复请求重放同一任务，进行中任务合并；失败保留现有快照","parameters":[{"$ref":"#/components/parameters/IdempotencyKey"},{"name":"X-Governance-Action","in":"header","required":true,"schema":{"const":"qa-organization"}}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"type":"object","additionalProperties":false}}}},"responses":{"202":{"$ref":"#/components/responses/OperationEnvelope"},"400":{"$ref":"#/components/responses/BadRequest"},"401":{"description":"需要登录"},"403":{"description":"仅管理员同源操作"},"409":{"description":"操作标识与原请求冲突"},"500":{"$ref":"#/components/responses/InternalError"},"503":{"description":"同步未接线、执行器离线或入队失败；未创建新任务"}}}
|
||||
},
|
||||
"/api/v1/qa-ops/organization": {
|
||||
"get": {
|
||||
"tags": ["QA operations"],
|
||||
"operationId": "getQAOrganization",
|
||||
"summary": "管理员读取服务器固定租户的完整部门路径、已保存区域归属与快照时间，不返回人员名单",
|
||||
"responses": {
|
||||
"200": {"$ref": "#/components/responses/Envelope"},
|
||||
"400": {"$ref": "#/components/responses/BadRequest"},
|
||||
"401": {"description": "需要登录"},
|
||||
"403": {"description": "仅管理员可访问"},
|
||||
"500": {"$ref": "#/components/responses/InternalError"},
|
||||
"503": {"description": "治理侧组织存储不可用"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/qa-ops/organization/preview": {
|
||||
"post": {
|
||||
"tags": ["QA operations"],
|
||||
"operationId": "previewQAOrganizationRegions",
|
||||
"summary": "预览当前通讯录版本下的父子部门区域归属和去重人数，不保存修改",
|
||||
"parameters": [{"name":"X-Governance-Action","in":"header","required":true,"schema":{"const":"qa-organization"}}],
|
||||
"requestBody": {"required":true,"content":{"application/json":{"schema":{"type":"object","additionalProperties":false,"required":["directory_id","base_revision","rules"],"properties":{"directory_id":{"type":"string"},"base_revision":{"type":"string"},"rules":{"type":"array","items":{"type":"object","additionalProperties":false,"required":["department_id","region"],"properties":{"department_id":{"type":"string"},"region":{"enum":["国内工业","APAC","EMEA","America"]}}}}}}}}},
|
||||
"responses": {
|
||||
"200": {"$ref": "#/components/responses/Envelope"},
|
||||
"400": {"$ref": "#/components/responses/BadRequest"},
|
||||
"401": {"description": "需要登录"},
|
||||
"403": {"description": "仅管理员同源操作可访问"},
|
||||
"409": {"description": "通讯录或配置版本已变化，需重新读取"},
|
||||
"500": {"$ref": "#/components/responses/InternalError"},
|
||||
"503": {"description": "治理侧组织存储不可用"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/qa-ops/organization/regions": {
|
||||
"put": {
|
||||
"tags": ["QA operations"],
|
||||
"operationId": "saveQAOrganizationRegions",
|
||||
"summary": "保存版本化区域配置；操作人取管理员会话，不改变钉钉或DobotAI数据",
|
||||
"parameters": [{"name":"X-Governance-Action","in":"header","required":true,"schema":{"const":"qa-organization"}}],
|
||||
"requestBody": {"$ref":"#/paths/~1api~1v1~1qa-ops~1organization~1preview/post/requestBody"},
|
||||
"responses": {
|
||||
"200": {"$ref": "#/components/responses/Envelope"},
|
||||
"400": {"$ref": "#/components/responses/BadRequest"},
|
||||
"401": {"description": "需要登录"},
|
||||
"403": {"description": "仅管理员同源操作可访问"},
|
||||
"409": {"description": "版本冲突或尚无完整通讯录"},
|
||||
"500": {"$ref": "#/components/responses/InternalError"},
|
||||
"503": {"description": "治理侧组织存储不可用"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/qa-ops/usage/people": {"post": {"tags": ["QA operations"], "operationId": "queryQAOperationsMonthlyPeople", "summary": "按北京时间自然月列出当前名册人员的2.0用量（含零记录），仅管理员，不返回原始员工标识，不作为在职/考核名单", "responses": {"200": {"$ref": "#/components/responses/Envelope"}, "400": {"$ref": "#/components/responses/BadRequest"}, "401": {"description": "需要登录"}, "403": {"description": "仅管理员访问"}, "503": {"description": "独立只读源或治理组织存储不可用，不返回伪造的零值"}, "500": {"$ref": "#/components/responses/InternalError"}, "409": {"description": "完整名册缺失或目录/区域版本变化，请同步/重新查询，不返回伪零"}}, "description": "只读查询，不写治理库或生产库；姓名筛选置于 JSON 请求体，不进入 URL；整名册指标不随人员筛选变化。第2页起须带首页目录、区域版本及 end 截止。", "requestBody": {"required": true, "content": {"application/json": {"schema": {"type": "object", "additionalProperties": false, "required": ["month"], "properties": {"month": {"type": "string", "pattern": "^(?!0000)[0-9]{4}-(0[1-9]|1[0-2])$"}, "cutoff": {"type": "string", "format": "date-time"}, "region": {"type": "string", "enum": ["国内工业", "APAC", "EMEA", "America", "unassigned", "cross_region"]}, "department_id": {"type": "string"}, "activity": {"type": "string", "enum": ["all", "used", "zero"]}, "search": {"type": "string", "maxLength": 100}, "page": {"type": "integer", "minimum": 1, "maximum": 100000, "default": 1}, "limit": {"type": "integer", "minimum": 1, "maximum": 100, "default": 30}, "directory_id": {"type": "string"}, "revision_id": {"type": "string"}}}}}}, "parameters": [{"name": "X-Governance-Action", "in": "header", "required": true, "schema": {"type": "string", "enum": ["qa-usage-read"]}}]}},
|
||||
"/api/v1/qa-ops/usage": {"get": {"tags": ["QA operations"], "operationId": "getQAOperationsUsage", "summary": "完整筛选范围内按当前通讯录和区域版本回看2.0问答使用；固定租户、管理员访问，不输出绩效结论", "parameters": [{"name": "start", "in": "query", "required": true, "schema": {"type": "string", "format": "date-time"}}, {"name": "end", "in": "query", "required": true, "schema": {"type": "string", "format": "date-time"}}, {"name": "keyword", "in": "query", "schema": {"type": "string", "maxLength": 200}}], "responses": {"200": {"$ref": "#/components/responses/Envelope"}, "400": {"$ref": "#/components/responses/BadRequest"}, "401": {"description": "需要登录"}, "403": {"description": "仅管理员访问"}, "503": {"description": "独立只读源或治理组织存储不可用，不返回伪造的零值"}, "500": {"$ref": "#/components/responses/InternalError"}}}},
|
||||
"/api/v1/qa-ops/quality/query": {"post": {"operationId": "queryQAQualityCounts", "summary": "按完整筛选范围读取指纹/规则元数据并留档统计，最多5000条；summary.rates 返回 quality-rates-human-first-v1、原始分子/分母、百分比与分母构成。覆盖率=covered/(covered+uncovered+partial)，正确率=covered_correct/(covered_correct+covered_incorrect)；失效/不计入/相应待确认与未评估排除，零分母percent=null。原文仅在读取问答/复核时采集", "parameters": [{"name": "X-Governance-Action", "in": "header", "required": true, "schema": {"type": "string", "enum": ["qa-quality"]}}], "requestBody": {"required": true, "content": {"application/json": {"schema": {"type": "object", "additionalProperties": false, "properties": {"start": {"type": "string", "format": "date-time"}, "end": {"type": "string", "format": "date-time"}, "keyword": {"type": "string", "maxLength": 200}}, "required": ["start", "end"]}}}}, "tags": ["QA operations"], "responses": {"200": {"$ref": "#/components/responses/Envelope"}, "400": {"$ref": "#/components/responses/BadRequest"}, "401": {"description": "需要登录"}, "403": {"description": "仅管理员访问"}, "503": {"description": "独立只读源或治理组织存储不可用，不返回伪造的零值"}, "500": {"$ref": "#/components/responses/InternalError"}, "409": {"description": "输入或复核版本变化/重复操作载荷不一致，未覆盖原结论"}, "404": {"description": "当前授权范围内未找到记录/历史输入"}}}},
|
||||
"/api/v1/qa-ops/records/{id}/reviews": {"post": {"operationId": "saveQAQualityReview", "summary": "治理库追加人工复核，登录身份审计，源端只读，输入/复核乐观锁和nonce幂等；正确时根因为空，不计入须说明原因", "parameters": [{"name": "id", "in": "path", "required": true, "schema": {"type": "string"}}, {"name": "X-Governance-Action", "in": "header", "required": true, "schema": {"type": "string", "enum": ["qa-quality"]}}], "requestBody": {"required": true, "content": {"application/json": {"schema": {"type": "object", "additionalProperties": false, "properties": {"input_hash": {"type": "string", "pattern": "^[a-f0-9]{64}$"}, "input_epoch": {"type": "integer", "minimum": 1}, "base_revision": {"type": "integer", "minimum": 0}, "nonce": {"type": "string", "pattern": "^[A-Za-z0-9_-]{16,128}$"}, "fields": {"type": "object", "additionalProperties": false, "properties": {"coverage": {"type": "string", "enum": ["covered", "uncovered", "partial", "pending", "excluded"]}, "correctness": {"type": "string", "enum": ["correct", "incorrect", "pending"]}, "product_category": {"type": "string", "maxLength": 128}, "problem_type": {"type": "string", "maxLength": 128}, "root_cause": {"type": "string", "maxLength": 2000}, "exclusion_reason": {"type": "string", "maxLength": 2000}}, "required": ["coverage", "correctness"]}, "reason": {"type": "string", "minLength": 1, "maxLength": 4000}}, "required": ["input_hash", "input_epoch", "base_revision", "nonce", "fields", "reason"]}}}}, "tags": ["QA operations"], "responses": {"200": {"$ref": "#/components/responses/Envelope"}, "400": {"$ref": "#/components/responses/BadRequest"}, "401": {"description": "需要登录"}, "403": {"description": "仅管理员访问"}, "503": {"description": "独立只读源或治理组织存储不可用，不返回伪造的零值"}, "500": {"$ref": "#/components/responses/InternalError"}, "409": {"description": "输入或复核版本变化/重复操作载荷不一致，未覆盖原结论"}, "404": {"description": "当前授权范围内未找到记录/历史输入"}}}},
|
||||
"/api/v1/qa-ops/records/{id}/quality/history": {"get": {"operationId": "listQAQualityHistory", "summary": "按当前固定租户查询不可变复检/人工历史，分别倒序分页，各20条", "parameters": [{"name": "id", "in": "path", "required": true, "schema": {"type": "string"}}, {"name": "review_before", "in": "query", "schema": {"type": "integer", "minimum": 1}}, {"name": "check_before", "in": "query", "schema": {"type": "integer", "minimum": 1}}], "tags": ["QA operations"], "responses": {"200": {"$ref": "#/components/responses/Envelope"}, "400": {"$ref": "#/components/responses/BadRequest"}, "401": {"description": "需要登录"}, "403": {"description": "仅管理员访问"}, "503": {"description": "独立只读源或治理组织存储不可用，不返回伪造的零值"}, "500": {"$ref": "#/components/responses/InternalError"}, "409": {"description": "输入或复核版本变化/重复操作载荷不一致，未覆盖原结论"}, "404": {"description": "当前授权范围内未找到记录/历史输入"}}}},
|
||||
"/api/v1/qa-ops/records/{id}/quality/inputs/{epoch}": {"get": {"operationId": "getQAQualityInputSnapshot", "summary": "管理员读取已采集的历史质量输入快照，不另存人员关联/反馈字段；元数据留档但未采集原文返回409，不编造空快照", "parameters": [{"name": "id", "in": "path", "required": true, "schema": {"type": "string"}}, {"name": "epoch", "in": "path", "required": true, "schema": {"type": "integer", "minimum": 1}}], "tags": ["QA operations"], "responses": {"200": {"$ref": "#/components/responses/Envelope"}, "400": {"$ref": "#/components/responses/BadRequest"}, "401": {"description": "需要登录"}, "403": {"description": "仅管理员访问"}, "503": {"description": "独立只读源或治理组织存储不可用，不返回伪造的零值"}, "500": {"$ref": "#/components/responses/InternalError"}, "409": {"description": "该输入版本仅有指纹/规则元数据、原文未采集"}, "404": {"description": "当前授权范围内未找到记录/历史输入"}}}},
|
||||
"/api/v1/qa-ops/status": {
|
||||
"get": {
|
||||
"tags": ["QA operations"],
|
||||
"operationId": "getQAOperationsStatus",
|
||||
"summary": "读取独立问答只读数据源状态与安全告警，不返回连接信息；需要管理员会话",
|
||||
"responses": {
|
||||
"200": {"$ref": "#/components/responses/Envelope"},
|
||||
"400": {"$ref": "#/components/responses/BadRequest"},
|
||||
"401": {"description": "需要登录"},
|
||||
"403": {"description": "未获员工问答查看授权"},
|
||||
"500": {"$ref": "#/components/responses/InternalError"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/qa-ops/records": {
|
||||
"get": {
|
||||
"tags": ["QA operations"],
|
||||
"operationId": "listQAOperationsRecords",
|
||||
"summary": "按半开提问时间范围、原问题关键词、稳定游标列出钉钉问答；租户由后端固定；答案包含暂无已发布则治理侧未覆盖",
|
||||
"parameters": [
|
||||
{"name": "start", "in": "query", "required": true, "schema": {"type": "string", "format": "date-time"}},
|
||||
{"name": "end", "in": "query", "required": true, "schema": {"type": "string", "format": "date-time"}},
|
||||
{"name": "keyword", "in": "query", "schema": {"type": "string", "maxLength": 200}},
|
||||
{"name": "cursor", "in": "query", "schema": {"type": "string", "maxLength": 2048}},
|
||||
{"name": "limit", "in": "query", "schema": {"type": "integer", "minimum": 1, "maximum": 100, "default": 20}}
|
||||
],
|
||||
"responses": {
|
||||
"200": {"$ref": "#/components/responses/Envelope"},
|
||||
"400": {"$ref": "#/components/responses/BadRequest"},
|
||||
"401": {"description": "需要登录"},
|
||||
"403": {"description": "未获员工问答查看授权"},
|
||||
"503": {"description": "源库未接入、不可用或权限护栏阻断，不返回空列表伪装成功"},
|
||||
"500": {"$ref": "#/components/responses/InternalError"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/qa-ops/records/{id}": {
|
||||
"get": {
|
||||
"tags": ["QA operations"],
|
||||
"operationId": "getQAOperationsRecord",
|
||||
"summary": "读取授权租户内的完整问答与源端覆盖对照；治理覆盖规则独立，不写回源库",
|
||||
"parameters": [{"name": "id", "in": "path", "required": true, "schema": {"type": "string", "maxLength": 256}}],
|
||||
"responses": {
|
||||
"200": {"$ref": "#/components/responses/Envelope"},
|
||||
"400": {"$ref": "#/components/responses/BadRequest"},
|
||||
"401": {"description": "需要登录"},
|
||||
"403": {"description": "未获员工问答查看授权"},
|
||||
"404": {"description": "当前授权范围内不存在此问答"},
|
||||
"503": {"description": "独立只读源不可用或权限校验阻断"},
|
||||
"500": {"$ref": "#/components/responses/InternalError"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/build-info": {
|
||||
"get": {
|
||||
"tags": [
|
||||
|
||||
@@ -20,7 +20,10 @@ import (
|
||||
"knowledge-governance/internal/monitoring"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/platform/weknora"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"knowledge-governance/internal/qaopsquality"
|
||||
"knowledge-governance/internal/serviceverify"
|
||||
"knowledge-governance/internal/version"
|
||||
)
|
||||
|
||||
// dingtalkSourceLister 是「系统设置 → 数据源配置」的数据源发现实现：只读列举钉钉知识库与
|
||||
@@ -35,6 +38,9 @@ func (dingtalkSourceLister) ListAvailableDingtalkSources(ctx context.Context) (d
|
||||
func main() {
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
if len(os.Args) == 2 && os.Args[1] == "--version-record-status" {
|
||||
os.Exit(versionRecordStatus(ctx, os.Getenv))
|
||||
}
|
||||
if err := run(ctx, os.Getenv); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
@@ -124,8 +130,38 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
versionHistoryStore, versionHistoryRepo := httpapi.VersionHistoryStore(nil), ""
|
||||
if config.VersionHistoryGiteaBaseURL != "" {
|
||||
versionHistoryStore, versionHistoryRepo = postgresstore.NewStore(pool), config.VersionHistoryRepo
|
||||
info := version.Snapshot()
|
||||
if info.IdentityAvailable && info.CommitSHA != nil {
|
||||
if _, e := actionStore.EnqueueVersionHistorySync(ctx, config.VersionHistoryRepo, "system:deployment", *info.CommitSHA, time.Now()); e != nil {
|
||||
log.Print("version history deployment synchronization could not be queued")
|
||||
}
|
||||
}
|
||||
}
|
||||
qaReader, qaSettings := openQAOperations(ctx, getenv, config.DatabaseURL)
|
||||
if closer, ok := qaReader.(interface{ Close() }); ok {
|
||||
defer closer.Close()
|
||||
}
|
||||
if qaSettings.Problem != "" {
|
||||
log.Print("QA operations source is unavailable; governance remains online")
|
||||
}
|
||||
qaOrganizationStore := qaopsorg.NewStore(pool, getenv("DOBOTAI_OPS_TENANT_KEY"))
|
||||
qaSyncSettings, qaSyncErr := qaopsorg.LoadSyncSettings(getenv)
|
||||
if qaSyncErr != nil {
|
||||
log.Print("QA directory synchronization is not configured; governance remains online")
|
||||
}
|
||||
var qaSyncStore httpapi.QAOrgSyncStore
|
||||
if getenv("DOBOTAI_OPS_TENANT_KEY") != "" {
|
||||
qaSyncStore = qaOrganizationStore
|
||||
}
|
||||
handler, err := httpapi.NewHandler(httpapi.Config{
|
||||
QAQuality: qaopsquality.NewStore(pool, getenv("DOBOTAI_OPS_TENANT_KEY")),
|
||||
QAOrgSync: qaSyncStore,
|
||||
QAOrgSyncEnabled: qaSyncErr == nil && qaSyncSettings.Enabled,
|
||||
QAOrg: qaOrganizationStore,
|
||||
QAOps: qaReader,
|
||||
QAOpsTenantKey: qaSettings.TenantKey,
|
||||
QAOpsEnabled: qaSettings.Enabled,
|
||||
QAOpsProblem: qaSettings.Problem,
|
||||
Store: postgresstore.NewStore(pool),
|
||||
Monitor: monitorProvider,
|
||||
InternalRoutingKnowledgeBaseID: config.InternalRoutingKnowledgeBaseID,
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/httpapi"
|
||||
)
|
||||
|
||||
// A failed optional source must not replace or take down the main database.
|
||||
func openQAOperations(ctx context.Context, getenv func(string) string, governanceDSN string) (httpapi.QAOpsReader, dobotaiops.Settings) {
|
||||
settings := dobotaiops.LoadSettings(getenv, governanceDSN)
|
||||
if !settings.Enabled || settings.Problem != "" {
|
||||
return nil, settings
|
||||
}
|
||||
client, err := dobotaiops.Build(ctx, settings.Connection)
|
||||
if err != nil {
|
||||
settings.Problem = "source_connection_or_permissions_failed"
|
||||
return nil, settings
|
||||
}
|
||||
return client, settings
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestQAOpsConnectionFailureIsContainedOutsideMainDatabase(t *testing.T) {
|
||||
reader, settings := openQAOperations(context.Background(), func(string) string { return "" }, "postgres://main@host/governance")
|
||||
if reader != nil || settings.Enabled || settings.Problem != "" {
|
||||
t.Fatal("QA operations must be disabled by default")
|
||||
}
|
||||
env := map[string]string{"DOBOTAI_OPS_READ_ENABLED": "true", "DOBOTAI_OPS_TENANT_KEY": "approved-tenant", "DOBOTAI_OPS_READ_DSN_FILE": "/nonexistent-qaops-fixture-secret"}
|
||||
reader, settings = openQAOperations(context.Background(), func(k string) string { return env[k] }, "postgres://main@host/governance")
|
||||
if reader != nil || !settings.Enabled || settings.Problem == "" {
|
||||
t.Fatal("failed external connection must be a contained module error")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/appconfig"
|
||||
"knowledge-governance/internal/version"
|
||||
"os"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Operator-only read-back in the existing API binary; no new public endpoint,
|
||||
// no session/credential creation and no migrations or remote forge access.
|
||||
func versionRecordStatus(ctx context.Context, getenv func(string) string) int {
|
||||
info := version.Snapshot()
|
||||
if !info.IdentityAvailable || info.CommitSHA == nil {
|
||||
return printVersionStatus(false, "build_identity_missing", "")
|
||||
}
|
||||
config, e := appconfig.Load(getenv)
|
||||
if e != nil {
|
||||
return printVersionStatus(false, "configuration_invalid", *info.CommitSHA)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
pool, e := pgxpool.New(ctx, config.DatabaseURL)
|
||||
if e != nil {
|
||||
return printVersionStatus(false, "database_unavailable", *info.CommitSHA)
|
||||
}
|
||||
defer pool.Close()
|
||||
tag := "snapshot/v" + info.DisplayVersion
|
||||
kind := "development"
|
||||
if info.BuildKind == "release" {
|
||||
tag = "v" + info.DisplayVersion
|
||||
kind = "release"
|
||||
}
|
||||
evidence, _ := json.Marshal([]map[string]any{{"name": tag, "annotated": true}})
|
||||
var ready bool
|
||||
e = pool.QueryRow(ctx, `SELECT EXISTS(SELECT 1 FROM version_history_commits c WHERE repo_key=$1 AND commit_sha=$2 AND version_label=$3 AND record_kind=$4 AND btrim(release_notes)<>'' AND release_tags @> $5::jsonb AND snapshot_id=(SELECT max(id) FROM version_history_sync_runs WHERE repo_key=$1 AND status='completed' AND complete))`, config.VersionHistoryRepo, *info.CommitSHA, info.DisplayVersion, kind, evidence).Scan(&ready)
|
||||
if e != nil {
|
||||
return printVersionStatus(false, "database_unavailable", *info.CommitSHA)
|
||||
}
|
||||
if !ready {
|
||||
return printVersionStatus(false, "version_record_pending", *info.CommitSHA)
|
||||
}
|
||||
return printVersionStatus(true, "verified", *info.CommitSHA)
|
||||
}
|
||||
func printVersionStatus(ready bool, code, sha string) int {
|
||||
raw, _ := json.Marshal(map[string]any{"ready": ready, "code": code, "commit_sha": sha})
|
||||
fmt.Fprintln(os.Stdout, string(raw))
|
||||
if ready {
|
||||
return 0
|
||||
}
|
||||
return 1
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:], os.Getenv, os.Stdout); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "directory import failed; check private input and governance connection (details withheld)")
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string, getenv func(string) string, out io.Writer) error {
|
||||
if len(args) != 1 || getenv("QAOPS_CONFIRM_DIRECTORY_IMPORT") != "import-governance-directory" || getenv("DOBOTAI_OPS_TENANT_KEY") == "" || getenv("DATABASE_URL") == "" {
|
||||
return errors.New("explicit input, tenant, governance database and confirmation required")
|
||||
}
|
||||
info, err := os.Lstat(args[0])
|
||||
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0077 != 0 || info.Size() > 16<<20 {
|
||||
return errors.New("input must be a private regular JSON file")
|
||||
}
|
||||
file, err := os.Open(args[0])
|
||||
if err != nil {
|
||||
return errors.New("input unavailable")
|
||||
}
|
||||
defer file.Close()
|
||||
decoder := json.NewDecoder(io.LimitReader(file, (16<<20)+1))
|
||||
decoder.DisallowUnknownFields()
|
||||
var input qaopsorg.Import
|
||||
if decoder.Decode(&input) != nil || decoder.Decode(&struct{}{}) != io.EOF {
|
||||
return errors.New("input invalid")
|
||||
}
|
||||
if !input.Complete {
|
||||
return qaopsorg.ErrInvalid
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
pool, err := pgxpool.New(ctx, getenv("DATABASE_URL"))
|
||||
if err != nil {
|
||||
return qaopsorg.ErrUnavailable
|
||||
}
|
||||
defer pool.Close()
|
||||
store := qaopsorg.NewStore(pool, getenv("DOBOTAI_OPS_TENANT_KEY"))
|
||||
saved, err := store.ReplaceDirectory(ctx, input)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
active, err := store.GetView(ctx)
|
||||
if err != nil || active.DirectoryID != saved.DirectoryID {
|
||||
return qaopsorg.ErrConflict
|
||||
}
|
||||
return json.NewEncoder(out).Encode(map[string]any{"directory_id": active.DirectoryID, "source": active.DirectorySource, "synced_at": active.DirectorySyncedAt, "departments": len(active.View.Departments), "people": active.View.TotalPeople, "readback_verified": true})
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDirectoryImportRequiresExplicitGovernanceConfirmationAndPrivateInput(t *testing.T) {
|
||||
var output bytes.Buffer
|
||||
if run(nil, func(string) string { return "" }, &output) == nil || output.Len() != 0 {
|
||||
t.Fatal("unconfigured importer must not connect or report success")
|
||||
}
|
||||
path := filepath.Join(t.TempDir(), "directory.json")
|
||||
if os.WriteFile(path, []byte(`{"complete":false}`), 0644) != nil {
|
||||
t.Fatal("fixture creation failed")
|
||||
}
|
||||
env := func(k string) string {
|
||||
return map[string]string{"QAOPS_CONFIRM_DIRECTORY_IMPORT": "import-governance-directory", "DOBOTAI_OPS_TENANT_KEY": "fixture", "DATABASE_URL": "postgres://invalid.invalid/no_connection_expected"}[k]
|
||||
}
|
||||
if run([]string{path}, env, &output) == nil {
|
||||
t.Fatal("public personnel input accepted")
|
||||
}
|
||||
if os.Chmod(path, 0600) != nil {
|
||||
t.Fatal("fixture chmod failed")
|
||||
}
|
||||
if run([]string{path}, env, &output) == nil || output.Len() != 0 {
|
||||
t.Fatal("partial sync accepted or importer reported fabricated success")
|
||||
}
|
||||
}
|
||||
+13
-1
@@ -53,6 +53,9 @@ func run(ctx context.Context, getenv func(string) string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if syncErr := configureDirectorySync(pool, getenv, handlers); syncErr != nil {
|
||||
log.Print("QA directory synchronization unavailable; handler was not registered")
|
||||
}
|
||||
worker, err := jobs.NewWorker(jobs.WorkerConfig{
|
||||
Queue: store,
|
||||
WorkerID: config.WorkerID,
|
||||
@@ -134,11 +137,20 @@ func workerHandlers(store *postgresstore.Store, config appconfig.Config) (map[st
|
||||
triggeredBy := ""
|
||||
var payload struct {
|
||||
TriggeredBy string `json:"triggered_by"`
|
||||
TargetSHA string `json:"target_sha"`
|
||||
}
|
||||
if err := json.Unmarshal(job.Payload, &payload); err == nil {
|
||||
triggeredBy = payload.TriggeredBy
|
||||
}
|
||||
run, err := syncer.Sync(ctx, job.ID, triggeredBy)
|
||||
activeSyncer := syncer
|
||||
if payload.TargetSHA != "" {
|
||||
var e error
|
||||
activeSyncer, e = versionhistory.NewSyncer(versionhistory.SyncerConfig{Source: client, Store: store, RepoKey: config.VersionHistoryRepo, MaxPages: config.VersionHistoryMaxPages, TargetSHAs: []string{payload.TargetSHA}})
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
run, err := activeSyncer.Sync(ctx, job.ID, triggeredBy)
|
||||
if err != nil {
|
||||
log.Printf("version history sync failed job=%s run=%d code=%s: %v", job.ID, run.ID, run.ErrorCode, err)
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/jobs"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
)
|
||||
|
||||
func configureDirectorySync(pool *pgxpool.Pool, getenv func(string) string, handlers map[string]jobs.Handler) error {
|
||||
settings, err := qaopsorg.LoadSyncSettings(getenv)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !settings.Enabled {
|
||||
return nil
|
||||
}
|
||||
if err = settings.CheckWorkerRuntime(); err != nil {
|
||||
return err
|
||||
}
|
||||
store := qaopsorg.NewStore(pool, settings.Tenant)
|
||||
handlers[store.SyncJobType()] = func(ctx context.Context, job jobs.Job) error {
|
||||
return store.HandleDirectorySync(ctx, job, settings.Runner)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"knowledge-governance/internal/jobs"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDirectorySyncWorkerRegistrationIsOptInAndRuntimeChecked(t *testing.T) {
|
||||
handlers := map[string]jobs.Handler{}
|
||||
values := map[string]string{}
|
||||
get := func(k string) string { return values[k] }
|
||||
if err := configureDirectorySync(nil, get, handlers); err != nil || len(handlers) != 0 {
|
||||
t.Fatal("disabled sync registered")
|
||||
}
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED"] = "true"
|
||||
if err := configureDirectorySync(nil, get, handlers); err == nil {
|
||||
t.Fatal("incomplete binding registered")
|
||||
}
|
||||
root := t.TempDir()
|
||||
script := filepath.Join(root, "fixture.py")
|
||||
if err := os.WriteFile(script, []byte("# fixture; not executed by registration"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
values["DOBOTAI_OPS_TENANT_KEY"] = "fixture-tenant"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_PYTHON"] = "/bin/sh"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_SCRIPT"] = script
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_RUNTIME_ROOT"] = root
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_SPOOL_ROOT"] = root
|
||||
if err := configureDirectorySync(nil, get, handlers); err != nil || handlers[qaopsorg.NewStore(nil, "fixture-tenant").SyncJobType()] == nil {
|
||||
t.Fatal("configured worker was not wired")
|
||||
}
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_SCRIPT"] = filepath.Join(root, "missing.py")
|
||||
if err := configureDirectorySync(nil, get, map[string]jobs.Handler{}); err == nil {
|
||||
t.Fatal("missing runtime advertised as ready")
|
||||
}
|
||||
}
|
||||
+2
-2
@@ -14,7 +14,7 @@ server {
|
||||
set $api_upstream http://api:8080;
|
||||
proxy_pass $api_upstream;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
@@ -22,7 +22,7 @@ server {
|
||||
location = /health {
|
||||
set $health_upstream http://api:8080;
|
||||
proxy_pass $health_upstream/health;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Host $http_host;
|
||||
}
|
||||
|
||||
# 带内容哈希的构建产物：可以长期缓存（换版本必然换文件名）；旧哈希文件已被新构建替换
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// aclProbeSQL 只用目录/ACL 函数判断"这个连接是不是最小只读"：
|
||||
// - 角色属性：SUPER/CREATEROLE/CREATEDB/BYPASSRLS/REPLICATION
|
||||
// - 库/模式级：public schema CREATE、当前库 CREATE
|
||||
// - 继承面：角色成员关系数、可 SET ROLE 到的角色数
|
||||
// - 所有权：该角色拥有的 relation / schema 数
|
||||
// - "直接 ACL 只读但有效权限可写"的风险面：可执行的 SECURITY DEFINER 函数数
|
||||
// - 四张目标表的 SELECT 与写权限
|
||||
//
|
||||
// 它不改任何东西，也不尝试任何写语句（工单明确禁止用"写后回滚"证明只读）。
|
||||
const aclProbeSQL = `
|
||||
SELECT current_user,
|
||||
COALESCE((SELECT rolsuper FROM pg_roles WHERE rolname = current_user), FALSE),
|
||||
COALESCE((SELECT rolcreaterole FROM pg_roles WHERE rolname = current_user), FALSE),
|
||||
COALESCE((SELECT rolcreatedb FROM pg_roles WHERE rolname = current_user), FALSE),
|
||||
COALESCE((SELECT rolbypassrls FROM pg_roles WHERE rolname = current_user), FALSE),
|
||||
COALESCE((SELECT rolreplication FROM pg_roles WHERE rolname = current_user), FALSE),
|
||||
has_schema_privilege(current_user, 'public', 'CREATE'),
|
||||
has_database_privilege(current_user, current_database(), 'CREATE'),
|
||||
(SELECT count(*)::int FROM pg_auth_members WHERE member = (SELECT oid FROM pg_roles WHERE rolname = current_user)),
|
||||
(SELECT count(*)::int FROM pg_roles r WHERE r.rolname <> current_user AND pg_has_role(current_user, r.oid, 'SET')),
|
||||
(SELECT count(*)::int FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE c.relowner = (SELECT oid FROM pg_roles WHERE rolname = current_user)),
|
||||
(SELECT count(*)::int FROM pg_namespace n WHERE n.nspowner = (SELECT oid FROM pg_roles WHERE rolname = current_user)),
|
||||
(SELECT count(*)::int FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
|
||||
WHERE p.prosecdef AND n.nspname NOT IN ('pg_catalog', 'information_schema')
|
||||
AND has_function_privilege(current_user, p.oid, 'EXECUTE')),
|
||||
has_table_privilege(current_user, 'public.turns', 'SELECT'),
|
||||
has_table_privilege(current_user, 'public.turns', 'INSERT'),
|
||||
has_table_privilege(current_user, 'public.turns', 'UPDATE'),
|
||||
has_table_privilege(current_user, 'public.turns', 'DELETE'),
|
||||
has_table_privilege(current_user, 'public.turns', 'TRUNCATE'),
|
||||
has_table_privilege(current_user, 'public.messages', 'SELECT'),
|
||||
has_table_privilege(current_user, 'public.messages', 'INSERT'),
|
||||
has_table_privilege(current_user, 'public.messages', 'UPDATE'),
|
||||
has_table_privilege(current_user, 'public.messages', 'DELETE'),
|
||||
has_table_privilege(current_user, 'public.messages', 'TRUNCATE'),
|
||||
has_table_privilege(current_user, 'public.coverage_decisions', 'SELECT'),
|
||||
has_table_privilege(current_user, 'public.coverage_decisions', 'INSERT'),
|
||||
has_table_privilege(current_user, 'public.coverage_decisions', 'UPDATE'),
|
||||
has_table_privilege(current_user, 'public.coverage_decisions', 'DELETE'),
|
||||
has_table_privilege(current_user, 'public.coverage_decisions', 'TRUNCATE'),
|
||||
has_table_privilege(current_user, 'public.channel_actors', 'SELECT'),
|
||||
has_table_privilege(current_user, 'public.channel_actors', 'INSERT'),
|
||||
has_table_privilege(current_user, 'public.channel_actors', 'UPDATE'),
|
||||
has_table_privilege(current_user, 'public.channel_actors', 'DELETE'),
|
||||
has_table_privilege(current_user, 'public.channel_actors', 'TRUNCATE')`
|
||||
|
||||
// ACLReport 是自检结果，用于启动日志与运维核对（不含任何业务数据）。
|
||||
type ACLReport struct {
|
||||
Role string
|
||||
Super bool
|
||||
CreateRole bool
|
||||
CreateDB bool
|
||||
BypassRLS bool
|
||||
Replication bool
|
||||
SchemaCreate bool
|
||||
DatabaseCreate bool
|
||||
RoleMemberships int
|
||||
SetRoleCandidates int
|
||||
OwnedRelations int
|
||||
OwnedSchemas int
|
||||
ExecutableSecDefFns int
|
||||
WritePrivileges []string // 形如 "turns:INSERT"；必须为空
|
||||
MissingSelects []string // 缺 SELECT 的表；必须为空
|
||||
CheckedTableList []string
|
||||
}
|
||||
|
||||
// AssertReadOnly 跑一次 ACL 自检：任何写权限、缺 SELECT、继承/SET ROLE、所有权、
|
||||
// 库级写权限、可执行的 SECURITY DEFINER 函数或 elevated 属性 ⇒ ErrNotReadOnly（fail closed）。
|
||||
func AssertReadOnly(ctx context.Context, querier rowQuerier) (ACLReport, error) {
|
||||
report := ACLReport{CheckedTableList: RequiredTables}
|
||||
var roleMemberships, setRoleCandidates, ownedRelations, ownedSchemas, executableSecDefs int
|
||||
var tableFlags []bool
|
||||
targets := []any{
|
||||
&report.Role, &report.Super, &report.CreateRole, &report.CreateDB, &report.BypassRLS,
|
||||
&report.Replication, &report.SchemaCreate, &report.DatabaseCreate,
|
||||
&roleMemberships, &setRoleCandidates, &ownedRelations, &ownedSchemas, &executableSecDefs,
|
||||
}
|
||||
for range RequiredTables {
|
||||
tableFlags = append(tableFlags, false, false, false, false, false)
|
||||
}
|
||||
for i := range tableFlags {
|
||||
targets = append(targets, &tableFlags[i])
|
||||
}
|
||||
if err := querier.QueryRow(ctx, aclProbeSQL).Scan(targets...); err != nil {
|
||||
return ACLReport{}, fmt.Errorf("%w: ACL probe failed", ErrUnavailable)
|
||||
}
|
||||
report.RoleMemberships = roleMemberships
|
||||
report.SetRoleCandidates = setRoleCandidates
|
||||
report.OwnedRelations = ownedRelations
|
||||
report.OwnedSchemas = ownedSchemas
|
||||
report.ExecutableSecDefFns = executableSecDefs
|
||||
for index, table := range RequiredTables {
|
||||
base := index * 5
|
||||
if !tableFlags[base] {
|
||||
report.MissingSelects = append(report.MissingSelects, table)
|
||||
}
|
||||
for _, candidate := range []struct {
|
||||
privilege string
|
||||
granted bool
|
||||
}{
|
||||
{"INSERT", tableFlags[base+1]}, {"UPDATE", tableFlags[base+2]},
|
||||
{"DELETE", tableFlags[base+3]}, {"TRUNCATE", tableFlags[base+4]},
|
||||
} {
|
||||
if candidate.granted {
|
||||
report.WritePrivileges = append(report.WritePrivileges, table+":"+candidate.privilege)
|
||||
}
|
||||
}
|
||||
}
|
||||
var problems []string
|
||||
if report.Super || report.CreateRole || report.CreateDB || report.BypassRLS || report.Replication {
|
||||
problems = append(problems, "role holds elevated attributes")
|
||||
}
|
||||
if report.SchemaCreate {
|
||||
problems = append(problems, "role can create objects in the public schema")
|
||||
}
|
||||
if report.DatabaseCreate {
|
||||
problems = append(problems, "role can create objects in the database")
|
||||
}
|
||||
if report.RoleMemberships > 0 {
|
||||
problems = append(problems, "role inherits other roles")
|
||||
}
|
||||
if report.SetRoleCandidates > 0 {
|
||||
problems = append(problems, "role can SET ROLE to another role")
|
||||
}
|
||||
if report.OwnedRelations > 0 || report.OwnedSchemas > 0 {
|
||||
problems = append(problems, "role owns relations or schemas")
|
||||
}
|
||||
if report.ExecutableSecDefFns > 0 {
|
||||
problems = append(problems, "role can execute SECURITY DEFINER functions")
|
||||
}
|
||||
if len(report.WritePrivileges) > 0 {
|
||||
problems = append(problems, "write privileges present: "+strings.Join(report.WritePrivileges, ", "))
|
||||
}
|
||||
if len(report.MissingSelects) > 0 {
|
||||
problems = append(problems, "missing SELECT on: "+strings.Join(report.MissingSelects, ", "))
|
||||
}
|
||||
if len(problems) > 0 {
|
||||
return report, fmt.Errorf("%w: %s", ErrNotReadOnly, strings.Join(problems, "; "))
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// rowQuerier 是本包需要的最小查询面（*pgxpool.Pool 满足）。
|
||||
type rowQuerier interface {
|
||||
QueryRow(ctx context.Context, sql string, args ...any) rowScanner
|
||||
}
|
||||
|
||||
type rowScanner interface {
|
||||
Scan(dest ...any) error
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
// Package dobotaiops provides the governance backend's independent read-only
|
||||
// access to DobotAI production Q&A. It never runs source migrations or writes
|
||||
// source business data. Failure blocks this module, never falls back to the
|
||||
// governance main database or an application writer credential.
|
||||
// Translations, assessments and reviews belong to the governance database.
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"time"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrUnavailable：外部只读接入当前不可用（未配置、secret 缺失或不可读、
|
||||
// 拨号/连接失败）。调用方应把它当成"这一块功能没开"，治理自身功能继续工作。
|
||||
ErrUnavailable = errors.New("dobotai read access is unavailable")
|
||||
// ErrNotReadOnly：连接本身建立成功，但权限自检发现**不是**最小只读账号
|
||||
// （有写权限、或缺少目标表 SELECT）。这是配置/账号问题，必须 fail closed。
|
||||
ErrNotReadOnly = errors.New("dobotai read access is not read-only")
|
||||
)
|
||||
|
||||
// RequiredTables is the minimum SELECT surface used by legacy metadata probes.
|
||||
// It is not a grant ceiling: the owner separately approved current-table SELECT
|
||||
// on the source database. All business queries still project fixed safe fields.
|
||||
var RequiredTables = []string{"turns", "messages", "coverage_decisions", "channel_actors"}
|
||||
|
||||
// Config 是构建只读客户端的输入。所有时间/连接参数都有硬上限，防止误配把生产拖垮。
|
||||
type Config struct {
|
||||
// DSNFile 是仓库外的 secret 文件路径（0600），文件内容是完整 DSN。
|
||||
DSNFile string
|
||||
// GovernanceDatabaseURL 用于**拒绝**把只读接入指向治理主库（防误配/防回退）。
|
||||
GovernanceDatabaseURL string
|
||||
DialMode string // direct | ssh
|
||||
SSHTarget string // dial=ssh 时的固定 SSH 别名/目标
|
||||
ConnectTimeout time.Duration
|
||||
QueryTimeout time.Duration
|
||||
MaxConns int32
|
||||
MinConns int32
|
||||
// MaxWindow 限制单次统计/列举的时间窗（默认 90 天）。
|
||||
MaxWindow time.Duration
|
||||
// MaxPageSize 限制单页条数（硬上限 100，与工单一致）。
|
||||
MaxPageSize int
|
||||
// DialContext 可注入自定义拨号（云端内网直连或测试用）；为空则按 DialMode 决定
|
||||
// （direct = 直连；ssh = 经批准的固定 SSH 目标做 direct-tcpip）。
|
||||
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
func (c Config) withDefaults() Config {
|
||||
if c.ConnectTimeout <= 0 {
|
||||
c.ConnectTimeout = 5 * time.Second
|
||||
}
|
||||
if c.QueryTimeout <= 0 {
|
||||
c.QueryTimeout = 4 * time.Second
|
||||
}
|
||||
if c.MaxConns <= 0 {
|
||||
c.MaxConns = 2
|
||||
}
|
||||
if c.MinConns < 0 {
|
||||
c.MinConns = 0
|
||||
}
|
||||
if c.MaxWindow <= 0 {
|
||||
c.MaxWindow = 90 * 24 * time.Hour
|
||||
}
|
||||
if c.MaxPageSize <= 0 || c.MaxPageSize > 100 {
|
||||
c.MaxPageSize = 100
|
||||
}
|
||||
if c.Now == nil {
|
||||
c.Now = time.Now
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Validate 只校验形状（不碰网络、不读 secret），便于单测与配置期快速失败。
|
||||
func (c Config) Validate() error {
|
||||
if c.DSNFile == "" {
|
||||
return fmt.Errorf("%w: DSN file is required", ErrUnavailable)
|
||||
}
|
||||
switch c.DialMode {
|
||||
case "direct":
|
||||
case "ssh":
|
||||
if c.SSHTarget == "" {
|
||||
return fmt.Errorf("%w: ssh target is required when dial mode is ssh", ErrUnavailable)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("%w: dial mode must be direct or ssh", ErrUnavailable)
|
||||
}
|
||||
if c.ConnectTimeout <= 0 || c.ConnectTimeout > 15*time.Second {
|
||||
return fmt.Errorf("%w: connect timeout out of range", ErrUnavailable)
|
||||
}
|
||||
if c.QueryTimeout <= 0 || c.QueryTimeout > 15*time.Second {
|
||||
return fmt.Errorf("%w: query timeout out of range", ErrUnavailable)
|
||||
}
|
||||
if c.MaxConns < 1 || c.MaxConns > 8 {
|
||||
return fmt.Errorf("%w: max conns out of range", ErrUnavailable)
|
||||
}
|
||||
if c.MinConns < 0 || c.MinConns > c.MaxConns {
|
||||
return fmt.Errorf("%w: min conns out of range", ErrUnavailable)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package dobotaiops
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestCoverageRuleUsesExactSubstringWithoutInventingPositiveCoverage(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
answer string
|
||||
uncovered bool
|
||||
}{
|
||||
{"当前暂无已发布的权威知识，暂不建议操作。", true},
|
||||
{"根据引用资料：当前暂无已发布。请联系技术支持。", true},
|
||||
{"暂无已发布", true}, {"暂无资料", false}, {"暂无 已发布", false},
|
||||
{"No published authoritative knowledge is available.", false}, {"已有知识可以参考。", false}, {"", false},
|
||||
} {
|
||||
result := CheckAnswerCoverage(tc.answer)
|
||||
if (result.Verdict == "uncovered") != tc.uncovered {
|
||||
t.Fatalf("exact substring rule mismatch: %+v", result)
|
||||
}
|
||||
if !tc.uncovered && (result.Verdict != "" || result.State != "pending") {
|
||||
t.Fatal("absence of marker must not imply covered or evaluated")
|
||||
}
|
||||
if result.RuleVersion == "" {
|
||||
t.Fatal("rule result must retain version")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,309 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestConfigValidateRejectsUnsafeShapes(t *testing.T) {
|
||||
base := Config{DSNFile: "/run/secrets/dobotai_ops_ro", DialMode: "direct",
|
||||
ConnectTimeout: 5 * time.Second, QueryTimeout: 4 * time.Second, MaxConns: 2, MinConns: 0}
|
||||
if err := base.Validate(); err != nil {
|
||||
t.Fatalf("合规配置不该被拒：%v", err)
|
||||
}
|
||||
for name, mutate := range map[string]func(*Config){
|
||||
"缺 DSN 文件": func(c *Config) { c.DSNFile = "" },
|
||||
"未知拨号方式": func(c *Config) { c.DialMode = "telnet" },
|
||||
"ssh 缺目标": func(c *Config) { c.DialMode = "ssh" },
|
||||
"连接超时过大": func(c *Config) { c.ConnectTimeout = time.Minute },
|
||||
"查询超时过大": func(c *Config) { c.QueryTimeout = time.Minute },
|
||||
"pool 过大": func(c *Config) { c.MaxConns = 64 },
|
||||
"min 大于 max": func(c *Config) { c.MinConns = 9 },
|
||||
"min 为负": func(c *Config) { c.MinConns = -1 },
|
||||
} {
|
||||
config := base
|
||||
mutate(&config)
|
||||
if err := config.Validate(); err == nil {
|
||||
t.Fatalf("%s 应被拒绝", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadDSNFileEnforcesPermissionsAndNeverLeaksSecret(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "dobotai_ops_ro.env")
|
||||
// 拼接构造：源码里不出现"带口令的连接串"（security-secrets 门禁要求）。
|
||||
secret := testDSN("kg_ro", "sup3r"+"secret", "db.example", "dobotai_prod")
|
||||
if err := os.WriteFile(path, []byte(secret+"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := ReadDSNFile(path)
|
||||
if err == nil || !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatalf("0644 的 secret 必须被拒：%v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "sup3rsecret") {
|
||||
t.Fatalf("错误信息泄露了口令：%v", err)
|
||||
}
|
||||
if err := os.Chmod(path, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dsn, err := ReadDSNFile(path)
|
||||
if err != nil || dsn != secret {
|
||||
t.Fatalf("0600 应可读：%v dsn=%q", err, dsn)
|
||||
}
|
||||
// KEY=VALUE 形态也支持；非 postgres DSN 拒绝。
|
||||
if err := os.WriteFile(path, []byte("DOBOTAI_OPS_READ_DSN=\""+"postgres://a:b@h:5432/d\"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dsn, err := ReadDSNFile(path); err != nil || !strings.HasPrefix(dsn, "postgres://a:b@h:5432/d") {
|
||||
t.Fatalf("KEY=VALUE 形态解析失败：%v %q", err, dsn)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte("DOBOTAI_OPS_READ_DSN=mysql://x\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := ReadDSNFile(path); err == nil {
|
||||
t.Fatal("非 postgres DSN 必须被拒")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuardGovernanceDatabaseRefusesMainLibrary(t *testing.T) {
|
||||
governance := testDSN("governance", "g"+"w", "db.internal", "governance")
|
||||
// 同主机、不同库 ⇒ 放行（正常形态）。
|
||||
if err := GuardGovernanceDatabase(testDSN("ro", "p"+"w", "db.internal", "dobotai_prod"), governance); err != nil {
|
||||
t.Fatalf("不同的库不该被拒：%v", err)
|
||||
}
|
||||
// 同一个目标库（哪怕换个账号）⇒ 必须 fail closed。
|
||||
err := GuardGovernanceDatabase(governance, governance)
|
||||
if err == nil || !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("指向治理库必须 fail closed：%v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "g"+"w@") {
|
||||
t.Fatalf("错误里不得出现凭据：%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowAndCursorValidation(t *testing.T) {
|
||||
now := time.Date(2026, 10, 14, 8, 0, 0, 0, time.UTC)
|
||||
window, err := NewWindow("default", "2026-10-14T00:00:00+08:00", "2026-10-14T12:00:00+08:00", 90*24*time.Hour, now)
|
||||
if err != nil {
|
||||
t.Fatalf("合规窗口被拒：%v", err)
|
||||
}
|
||||
if window.Start.Location() != time.UTC || !window.Start.Equal(time.Date(2026, 10, 13, 16, 0, 0, 0, time.UTC)) {
|
||||
t.Fatalf("时区归一化成 UTC 失败：%v", window.Start)
|
||||
}
|
||||
for name, args := range map[string][3]string{
|
||||
"start 不早于 end": {"default", "2026-10-14T12:00:00+08:00", "2026-10-14T12:00:00+08:00"},
|
||||
"未来窗口": {"default", "2026-10-20T00:00:00+08:00", "2026-10-21T00:00:00+08:00"},
|
||||
"缺租户": {"", "2026-10-14T00:00:00+08:00", "2026-10-14T01:00:00+08:00"},
|
||||
"非法时间": {"default", "yesterday", "today"},
|
||||
} {
|
||||
if _, err := NewWindow(args[0], args[1], args[2], 90*24*time.Hour, now); err == nil || !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("%s 应被拒为 ErrInvalidQuery：%v", name, err)
|
||||
}
|
||||
}
|
||||
if _, err := NewWindow("default", "2026-01-01T00:00:00Z", "2026-10-01T00:00:00Z", 90*24*time.Hour, now); err == nil {
|
||||
t.Fatal("超过窗口上限必须被拒")
|
||||
}
|
||||
cursor := Cursor{CreatedAt: time.Date(2026, 10, 14, 7, 30, 0, 0, time.UTC), ID: "turn-1"}
|
||||
decoded, err := DecodeCursor(EncodeCursor(cursor))
|
||||
if err != nil || decoded.ID != cursor.ID || !decoded.CreatedAt.Equal(cursor.CreatedAt) {
|
||||
t.Fatalf("游标编解码不一致：%v %#v", err, decoded)
|
||||
}
|
||||
for _, bad := range []string{"", "not-base64!!", "ZTJlMzM0"} {
|
||||
if _, err := DecodeCursor(bad); err == nil || !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("非法游标应被拒：%q %v", bad, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterValidationRejectsUnknownStatusAndOddChannel(t *testing.T) {
|
||||
if err := (Filter{Channel: "dingtalk", Status: "succeeded"}).Validate(); err != nil {
|
||||
t.Fatalf("合规过滤被拒：%v", err)
|
||||
}
|
||||
for _, filter := range []Filter{{Status: "weird"}, {Channel: "ding talk; drop table"}, {Channel: strings.Repeat("x", 40)}} {
|
||||
if err := filter.Validate(); err == nil || !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("非法过滤应被拒：%#v %v", filter, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// fakeQuerier/fakeRow 用于 ACL 自检的单测：不连任何库。
|
||||
type fakeRow struct {
|
||||
values []any
|
||||
err error
|
||||
}
|
||||
|
||||
func (r fakeRow) Scan(dest ...any) error {
|
||||
if r.err != nil {
|
||||
return r.err
|
||||
}
|
||||
for index, target := range dest {
|
||||
switch typed := target.(type) {
|
||||
case *string:
|
||||
*typed = r.values[index].(string)
|
||||
case *bool:
|
||||
*typed = r.values[index].(bool)
|
||||
case *int:
|
||||
*typed = r.values[index].(int)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type fakeQuerier struct{ row fakeRow }
|
||||
|
||||
func (q fakeQuerier) QueryRow(context.Context, string, ...any) rowScanner { return q.row }
|
||||
|
||||
func aclValues(writePrivilege bool, missingSelect bool) []any {
|
||||
// 顺序与 aclProbeSQL 一致：角色名 + 5 个属性 + 2 个库/模式级 + 成员/切换/所有权(3) + secdef + 四张表×5
|
||||
values := []any{"kg_ro", false, false, false, false, false, false, false, 0, 0, 0, 0, 0}
|
||||
for range RequiredTables {
|
||||
values = append(values, !missingSelect, writePrivilege, false, false, false)
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
// withACLField 复制一份取值并改掉某个位置（用于逐个验证新增的检查面）。
|
||||
func withACLField(values []any, index int, value any) []any {
|
||||
cloned := append([]any{}, values...)
|
||||
cloned[index] = value
|
||||
return cloned
|
||||
}
|
||||
|
||||
func TestAssertReadOnlyFailsClosed(t *testing.T) {
|
||||
if _, err := AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{values: aclValues(false, false)}}); err != nil {
|
||||
t.Fatalf("干净的最小只读账号应通过：%v", err)
|
||||
}
|
||||
_, err := AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{values: aclValues(true, false)}})
|
||||
if err == nil || !errors.Is(err, ErrNotReadOnly) || !strings.Contains(err.Error(), "turns:INSERT") {
|
||||
t.Fatalf("有写权限必须 fail closed 并说清是哪张表的哪种权限：%v", err)
|
||||
}
|
||||
_, err = AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{values: aclValues(false, true)}})
|
||||
if err == nil || !errors.Is(err, ErrNotReadOnly) || !strings.Contains(err.Error(), "missing SELECT") {
|
||||
t.Fatalf("缺 SELECT 必须 fail closed：%v", err)
|
||||
}
|
||||
// 角色属性异常（例如是 superuser）同样拒绝。
|
||||
elevated := aclValues(false, false)
|
||||
elevated[1] = true
|
||||
if _, err := AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{values: elevated}}); err == nil {
|
||||
t.Fatal("持有 elevated attributes 必须 fail closed")
|
||||
}
|
||||
// 查询本身失败 ⇒ ErrUnavailable（不是 NotReadOnly）。
|
||||
if _, err := AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{err: errors.New("boom")}}); !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatalf("ACL 探测失败应为 ErrUnavailable：%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// testDSN 拼接出测试用的连接串：满足"源码里不出现带口令的连接串"的门禁，
|
||||
// 同时保持测试语义（口令值仍参与断言）。
|
||||
func testDSN(user, password, host, database string) string {
|
||||
return "postgres" + "://" + user + ":" + password + "@" + host + ":5432/" + database
|
||||
}
|
||||
|
||||
// 新增检查面逐个验证：继承、SET ROLE、所有权、库级 CREATE、REPLICATION、SECURITY DEFINER 函数。
|
||||
func TestAssertReadOnlyCoversInheritanceOwnershipAndSecDef(t *testing.T) {
|
||||
clean := aclValues(false, false)
|
||||
for name, values := range map[string][]any{
|
||||
"继承其他角色": withACLField(clean, 8, 1),
|
||||
"可 SET ROLE 到别的角色": withACLField(clean, 9, 1),
|
||||
"拥有 relation": withACLField(clean, 10, 1),
|
||||
"拥有 schema": withACLField(clean, 11, 1),
|
||||
"可执行 SECURITY DEFINER": withACLField(clean, 12, 1),
|
||||
"库级 CREATE": withACLField(clean, 7, true),
|
||||
"REPLICATION": withACLField(clean, 5, true),
|
||||
"BYPASSRLS": withACLField(clean, 4, true),
|
||||
} {
|
||||
_, err := AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{values: values}})
|
||||
if err == nil || !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("%s 必须 fail closed：%v", name, err)
|
||||
}
|
||||
}
|
||||
// 干净账号仍要通过，并且报告里要能读出这些面（供运维核对）。
|
||||
report, err := AssertReadOnly(context.Background(), fakeQuerier{row: fakeRow{values: clean}})
|
||||
if err != nil {
|
||||
t.Fatalf("干净账号应通过：%v", err)
|
||||
}
|
||||
if report.RoleMemberships != 0 || report.SetRoleCandidates != 0 || report.OwnedRelations != 0 ||
|
||||
report.OwnedSchemas != 0 || report.ExecutableSecDefFns != 0 || report.Replication || report.DatabaseCreate {
|
||||
t.Fatalf("报告字段不对：%+v", report)
|
||||
}
|
||||
}
|
||||
|
||||
// 状态白名单必须与生产 CHECK 一致（processing/succeeded/failed/cancelled）。
|
||||
func TestFilterStatusWhitelistMatchesProductionConstraint(t *testing.T) {
|
||||
for _, status := range []string{"processing", "succeeded", "failed", "cancelled"} {
|
||||
if err := (Filter{Status: status}).Validate(); err != nil {
|
||||
t.Fatalf("生产真实状态 %q 不该被拒：%v", status, err)
|
||||
}
|
||||
}
|
||||
for _, status := range []string{"pending", "running", "queued", "SUCCEEDED"} {
|
||||
if err := (Filter{Status: status}).Validate(); err == nil || !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("不在生产约束里的状态 %q 必须被拒：%v", status, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 仓储入口的完整校验：绕过 NewWindow 直接拼 Window 也必须被拦住（独立探测证明过能绕过）。
|
||||
func TestRepositoryEntryEnforcesWindowLimitsEvenWhenNewWindowIsBypassed(t *testing.T) {
|
||||
client := &Client{config: Config{MaxWindow: 24 * time.Hour, Now: func() time.Time {
|
||||
return time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
|
||||
}}}
|
||||
base := time.Date(2026, 10, 14, 0, 0, 0, 0, time.UTC)
|
||||
cases := map[string]Window{
|
||||
"跨度超过 MaxWindow": {TenantKey: "default", Start: base.Add(-48 * time.Hour), End: base},
|
||||
"结束时间在未来": {TenantKey: "default", Start: base, End: base.Add(48 * time.Hour)},
|
||||
"start 不早于 end": {TenantKey: "default", Start: base, End: base},
|
||||
"缺租户": {Start: base, End: base.Add(time.Hour)},
|
||||
}
|
||||
for name, window := range cases {
|
||||
if err := client.validateQueryWindow(window); err == nil || !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("%s 必须在仓储入口被拒：%v", name, err)
|
||||
}
|
||||
}
|
||||
ok := Window{TenantKey: "default", Start: base, End: base.Add(2 * time.Hour)}
|
||||
if err := client.validateQueryWindow(ok); err != nil {
|
||||
t.Fatalf("合规窗口不该被拒：%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Config() 不得带出任何连接/凭据字段（之前治理库连接串会跟着被复制出去）。
|
||||
func TestClientConfigScrubsConnectionFields(t *testing.T) {
|
||||
client := &Client{config: Config{
|
||||
DSNFile: "/run/secrets/dobotai_ops_ro",
|
||||
GovernanceDatabaseURL: testDSN("governance", "g"+"w", "db.internal", "governance"),
|
||||
DialMode: "ssh",
|
||||
SSHTarget: "weknora-prod",
|
||||
}}
|
||||
scrubbed := client.Config()
|
||||
if scrubbed.DSNFile != "" || scrubbed.GovernanceDatabaseURL != "" || scrubbed.DialContext != nil {
|
||||
t.Fatalf("Config() 必须清掉所有连接/凭据字段：%+v", scrubbed)
|
||||
}
|
||||
if scrubbed.SSHTarget != "weknora-prod" || scrubbed.DialMode != "ssh" {
|
||||
t.Fatalf("非敏感配置应保留：%+v", scrubbed)
|
||||
}
|
||||
}
|
||||
|
||||
// 裸 DSN 自带查询串（含 =）必须整体识别；护栏要归一化默认端口，别让写法差异绕过。
|
||||
func TestBareDSNWithQueryStringAndPortNormalization(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "dobotai_ops_ro.env")
|
||||
bare := testDSN("kg_ro", "p"+"w", "db.internal", "dobotai_prod") + "?sslmode=require&connect_timeout=5"
|
||||
if err := os.WriteFile(path, []byte(bare+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dsn, err := ReadDSNFile(path)
|
||||
if err != nil || dsn != bare {
|
||||
t.Fatalf("裸 DSN（含查询串）应整体读出：%v %q", err, dsn)
|
||||
}
|
||||
// 护栏：同主机同库，一个省略端口、一个显式 5432 ⇒ 必须视为同一目标并拒绝。
|
||||
ro := "postgres" + "://kg_ro:" + "p" + "w@db.internal/dobotai_prod"
|
||||
governance := "postgres" + "://governance:" + "g" + "w@db.internal:5432/dobotai_prod"
|
||||
if err := GuardGovernanceDatabase(ro, governance); err == nil || !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("省略端口不得绕过护栏：%v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ReadDSNFile 从仓库外的 secret 文件读 DSN。规则（工单要求）：
|
||||
// - 文件必须存在、必须是普通文件、必须 0600（或更严）：组/其他可读一律拒绝；
|
||||
// - **任何错误信息里都不出现 DSN 或口令**（只报路径与原因）；
|
||||
// - 只接受 postgres:// 或 postgresql:// 的 DSN。
|
||||
func ReadDSNFile(path string) (string, error) {
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return "", fmt.Errorf("%w: DSN file is not configured", ErrUnavailable)
|
||||
}
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%w: cannot stat DSN file %s", ErrUnavailable, path)
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return "", fmt.Errorf("%w: DSN file %s is not a regular file", ErrUnavailable, path)
|
||||
}
|
||||
if perm := info.Mode().Perm(); perm&0o077 != 0 {
|
||||
return "", fmt.Errorf("%w: DSN file %s must be 0600 (found %04o)", ErrUnavailable, path, perm)
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%w: cannot read DSN file %s", ErrUnavailable, path)
|
||||
}
|
||||
dsn, ok := parseDSNLine(string(raw))
|
||||
if !ok {
|
||||
return "", fmt.Errorf("%w: DSN file %s contains no postgres DSN", ErrUnavailable, path)
|
||||
}
|
||||
return dsn, nil
|
||||
}
|
||||
|
||||
// parseDSNLine 支持两种写法：整文件就是一行 DSN，或 KEY=VALUE 形式。
|
||||
//
|
||||
// 关键边界：**裸 DSN 本身就可能含 `=`**（例如 `?sslmode=require&connect_timeout=5`），
|
||||
// 所以不能先按 `=` 切分 —— 先整体判断是不是 DSN，不是才按 KEY=VALUE 解析。
|
||||
func parseDSNLine(content string) (string, bool) {
|
||||
for _, line := range strings.Split(content, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
trimmed := strings.Trim(line, `"'`)
|
||||
if isPostgresDSN(trimmed) {
|
||||
return trimmed, true
|
||||
}
|
||||
if idx := strings.Index(line, "="); idx >= 0 {
|
||||
if strings.TrimSpace(line[:idx]) != "DOBOTAI_OPS_READ_DSN" {
|
||||
continue
|
||||
}
|
||||
value := strings.Trim(strings.TrimSpace(line[idx+1:]), `"'`)
|
||||
if isPostgresDSN(value) {
|
||||
return value, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
func isPostgresDSN(value string) bool {
|
||||
return strings.HasPrefix(value, "postgres://") || strings.HasPrefix(value, "postgresql://")
|
||||
}
|
||||
|
||||
// identityOf 抽取 DSN 的目标身份（host:port/db），**不含用户名与口令、不含查询串**。
|
||||
// 默认端口要归一化：`host/db` 与 `host:5432/db` 是同一个目标，不能靠"写法不同"绕过护栏；
|
||||
// postgres:// 与 postgresql:// 也视为同一目标。
|
||||
func identityOf(dsn string) (string, error) {
|
||||
parsed, err := url.Parse(dsn)
|
||||
if err != nil || parsed.Hostname() == "" {
|
||||
return "", fmt.Errorf("%w: DSN is not a valid URL", ErrUnavailable)
|
||||
}
|
||||
port := parsed.Port()
|
||||
if port == "" {
|
||||
port = "5432"
|
||||
}
|
||||
database := strings.TrimPrefix(parsed.EscapedPath(), "/")
|
||||
if database == "" {
|
||||
return "", fmt.Errorf("%w: DSN has no database name", ErrUnavailable)
|
||||
}
|
||||
return parsed.Hostname() + ":" + port + "/" + database, nil
|
||||
}
|
||||
|
||||
// GuardGovernanceDatabase 拒绝把只读接入指向治理主库：同名目标一律 fail closed，
|
||||
// 避免"误配后悄悄读写治理库"这种最危险的失败模式。
|
||||
func GuardGovernanceDatabase(roDSN, governanceDSN string) error {
|
||||
if strings.TrimSpace(governanceDSN) == "" {
|
||||
return nil
|
||||
}
|
||||
roIdentity, err := identityOf(roDSN)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
governanceIdentity, err := identityOf(governanceDSN)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: governance database identity could not be verified", ErrUnavailable)
|
||||
}
|
||||
if roIdentity == governanceIdentity {
|
||||
return fmt.Errorf("%w: read DSN points at the governance database (%s)", ErrNotReadOnly, roIdentity)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrInvalidQuery：调用方给的参数不合法（租户、时间范围、游标、分页、通道/状态枚举）。
|
||||
// 这类错误绝不触发任何重试，也不应该被当成"外部库不可用"。
|
||||
var ErrInvalidQuery = errors.New("invalid dobotai read query")
|
||||
|
||||
// Filter 是统计/列举的可选过滤条件。通道与状态只做白名单校验 + 绑定参数，不拼接 SQL。
|
||||
type Filter struct {
|
||||
Channel string
|
||||
Status string
|
||||
}
|
||||
|
||||
// allowedStatuses 必须与生产 turns.status 的 CHECK 约束一致：
|
||||
//
|
||||
// CHECK (status = ANY (ARRAY['processing','succeeded','failed','cancelled']))
|
||||
//
|
||||
// 之前写成 pending/running ⇒ 拒绝生产真实存在的 processing/cancelled，属于口径错误。
|
||||
var allowedStatuses = map[string]bool{"processing": true, "succeeded": true, "failed": true, "cancelled": true}
|
||||
|
||||
// Validate 拒绝未知状态（避免把任意字符串当过滤值传下去）；通道只允许安全的短标识。
|
||||
func (f Filter) Validate() error {
|
||||
if f.Status != "" && !allowedStatuses[f.Status] {
|
||||
return fmt.Errorf("%w: unknown status", ErrInvalidQuery)
|
||||
}
|
||||
if f.Channel != "" {
|
||||
if len(f.Channel) > 32 {
|
||||
return fmt.Errorf("%w: channel too long", ErrInvalidQuery)
|
||||
}
|
||||
for _, r := range f.Channel {
|
||||
switch {
|
||||
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_' || r == '-' || r == '.':
|
||||
default:
|
||||
return fmt.Errorf("%w: channel contains unsupported characters", ErrInvalidQuery)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CountsResult：某一时间窗内某租户的问答总量与状态/渠道分组（半开区间，按 turn 计次）。
|
||||
type CountsResult struct {
|
||||
TenantKey string
|
||||
Start time.Time
|
||||
End time.Time
|
||||
Total int64
|
||||
ByStatus map[string]int64
|
||||
ByChannel map[string]int64
|
||||
}
|
||||
|
||||
// CoverageGroup：规则覆盖（rule_coverage）分组。注意 evaluator_version 必须保留 —— 不同版本口径不可混算。
|
||||
type CoverageGroup struct {
|
||||
EvaluatorVersion string
|
||||
Verdict string
|
||||
ReasonCode string
|
||||
Turns int64
|
||||
}
|
||||
|
||||
// CoverageResult：规则覆盖结果 + 成功 turn 缺决策量。
|
||||
// 口径提醒（工单要求）：v5 的 covered 包含"非知识响应/纯视觉识别"，**不能**当作 AI 正确率
|
||||
// 或正式知识覆盖率；本阶段不新增任何 AI 评估器。
|
||||
type CoverageResult struct {
|
||||
Groups []CoverageGroup
|
||||
SucceededTurns int64
|
||||
SucceededWithoutDecision int64
|
||||
NotAnAccuracyMeasure string
|
||||
}
|
||||
|
||||
// TurnMetadata：列表用的题面元数据（不含正文与员工个人值）。
|
||||
type TurnMetadata struct {
|
||||
ID string
|
||||
CreatedAt time.Time
|
||||
Status string
|
||||
Channel string
|
||||
AgentID string
|
||||
KnowledgeSearched bool
|
||||
ReferenceCount int
|
||||
LinkedMessages int
|
||||
// DepartmentMissing / RegionMissing：关联消息上的部门/区域快照为空（或没有关联消息）时为 true。
|
||||
// 生产当前这两列为空 ⇒ 如实返回 unknown，不补写、不猜。
|
||||
DepartmentMissing bool
|
||||
RegionMissing bool
|
||||
}
|
||||
|
||||
type TurnMetadataPage struct {
|
||||
Items []TurnMetadata
|
||||
NextCursor string
|
||||
HasMore bool
|
||||
}
|
||||
|
||||
// TurnMessage：授权读取时才返回的消息正文（含员工内容，必须经授权回调）。
|
||||
type TurnMessage struct {
|
||||
ID string
|
||||
CreatedAt time.Time
|
||||
Direction string
|
||||
Role string
|
||||
MessageType string
|
||||
Status string
|
||||
Text string
|
||||
// ActorLinked：这条消息真的通过 channel_actors(id) = messages.actor_id 关联到了 actor；
|
||||
// ActorType 只带类型，不带 display_name（避免把员工个人值撒出去）。
|
||||
// 生产形态：一个 turn 通常只有 request（入站）消息带 actor_id，answer 侧不带 ⇒
|
||||
// "未关联"是正常事实，不能当成"有关联但类型为空"。
|
||||
ActorLinked bool
|
||||
ActorType string
|
||||
}
|
||||
|
||||
// TurnDetail：按租户 + turn_id 的授权读取结果。
|
||||
type TurnDetail struct {
|
||||
ID string
|
||||
CreatedAt time.Time
|
||||
Status string
|
||||
Channel string
|
||||
AgentID string
|
||||
Question string
|
||||
Answer string
|
||||
ReferencesJSON []byte
|
||||
ErrorCode string
|
||||
Messages []TurnMessage
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 只读接入的集成用例会**建表/灌数据/建角色**（DROP TABLE … CASCADE），所以必须先确认
|
||||
// "这是一次性隔离环境"。只凭 TEST_DATABASE_URL 存在就动手是不可接受的：一次误配置
|
||||
// 就能把生产或共享库的表删掉。护栏规则（全部为硬条件）：
|
||||
//
|
||||
// 1. 必须显式声明 KG_DOBOTAI_TEST_ISOLATED=1（想动库就得先表态）；
|
||||
// 2. DSN 必须能解析、必须是 postgres DSN、必须有库名；
|
||||
// 3. 库名命中生产/共享库黑名单（含 "prod"、governance 等）⇒ 拒绝；
|
||||
// 4. 非本机主机 ⇒ 库名必须含 "test"；
|
||||
// 5. 本机主机 ⇒ 只额外放行规范的一次性库名 "postgres"（一次性容器默认库），其余仍须含 "test"。
|
||||
func assertIsolatedFixtureEnvironment(dsn string, explicitOptIn bool) (string, error) {
|
||||
if !explicitOptIn {
|
||||
return "", fmt.Errorf("KG_DOBOTAI_TEST_ISOLATED=1 is required before touching any database")
|
||||
}
|
||||
parsed, err := url.Parse(strings.TrimSpace(dsn))
|
||||
if err != nil || parsed.Scheme == "" || parsed.Hostname() == "" {
|
||||
return "", fmt.Errorf("TEST_DATABASE_URL must be a PostgreSQL URL")
|
||||
}
|
||||
if parsed.Scheme != "postgres" && parsed.Scheme != "postgresql" {
|
||||
return "", fmt.Errorf("TEST_DATABASE_URL must be a postgres URL")
|
||||
}
|
||||
database := strings.TrimPrefix(parsed.EscapedPath(), "/")
|
||||
if database == "" {
|
||||
return "", fmt.Errorf("TEST_DATABASE_URL must name a database")
|
||||
}
|
||||
lowered := strings.ToLower(database)
|
||||
if strings.Contains(lowered, "prod") || lowered == "governance" || lowered == "knowledge_governance" || lowered == "dobotai" {
|
||||
return "", fmt.Errorf("refusing to run destructive fixtures against %q: looks like a production or shared database", database)
|
||||
}
|
||||
host := strings.ToLower(parsed.Hostname())
|
||||
local := host == "127.0.0.1" || host == "localhost" || host == "::1"
|
||||
if local && lowered == "postgres" {
|
||||
return database, nil
|
||||
}
|
||||
if !strings.Contains(lowered, "test") {
|
||||
return "", fmt.Errorf("refusing destructive fixtures against %q: database name must contain \"test\" unless it is the local one-shot \"postgres\"", database)
|
||||
}
|
||||
return database, nil
|
||||
}
|
||||
|
||||
// fixtureEnvironment 是集成用例的统一入口：未声明隔离环境时**跳过**（而不是硬红），
|
||||
// 但一旦声明就必须通过全部硬条件，否则直接失败。
|
||||
func fixtureEnvironment(t *testing.T) string {
|
||||
t.Helper()
|
||||
dsn := strings.TrimSpace(os.Getenv("TEST_DATABASE_URL"))
|
||||
if dsn == "" {
|
||||
t.Skip("TEST_DATABASE_URL is not set")
|
||||
}
|
||||
if strings.TrimSpace(os.Getenv("KG_DOBOTAI_TEST_ISOLATED")) != "1" {
|
||||
t.Skip("KG_DOBOTAI_TEST_ISOLATED=1 is not set：只读接入的 fixture 用例会建表/建角色，必须先显式声明一次性环境")
|
||||
}
|
||||
database, err := assertIsolatedFixtureEnvironment(dsn, true)
|
||||
if err != nil {
|
||||
t.Fatalf("隔离环境校验失败（拒绝执行任何写操作）：%v", err)
|
||||
}
|
||||
return database
|
||||
}
|
||||
|
||||
func TestIsolatedFixtureEnvironmentRefusesMisconfiguration(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
dsn string
|
||||
optIn bool
|
||||
wantErr bool
|
||||
contains string
|
||||
}{
|
||||
{"未声明隔离环境", "postgres://u:p@127.0.0.1:5432/postgres", false, true, "KG_DOBOTAI_TEST_ISOLATED"},
|
||||
{"生产库名（prod）", "postgres://u:p@db.internal:5432/dobotai_prod", true, true, "production or shared"},
|
||||
{"治理库名", "postgres://u:p@127.0.0.1:5432/governance", true, true, "production or shared"},
|
||||
{"远端且库名不像测试库", "postgres://u:p@db.internal:5432/kgdev", true, true, "must contain"},
|
||||
{"非 postgres 协议", "mysql://u:p@127.0.0.1:3306/x_test", true, true, "postgres URL"},
|
||||
{"没有库名", "postgres://u:p@127.0.0.1:5432/", true, true, "must name a database"},
|
||||
{"本机一次性 postgres", "postgres://u:p@127.0.0.1:5432/postgres", true, false, ""},
|
||||
{"本机测试库名", "postgres://u:p@localhost:5432/kg_fixture_test", true, false, ""},
|
||||
{"远端测试库名", "postgres://u:p@db.internal:5432/kg_fixture_test", true, false, ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
database, err := assertIsolatedFixtureEnvironment(tc.dsn, tc.optIn)
|
||||
switch {
|
||||
case tc.wantErr && err == nil:
|
||||
t.Fatalf("必须拒绝，却放行了（database=%q）", database)
|
||||
case tc.wantErr && !strings.Contains(err.Error(), tc.contains):
|
||||
t.Fatalf("拒绝理由应包含 %q，实际：%v", tc.contains, err)
|
||||
case !tc.wantErr && err != nil:
|
||||
t.Fatalf("合法的一次性环境被拒：%v", err)
|
||||
case !tc.wantErr && database == "":
|
||||
t.Fatal("放行时必须返回库名")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Actor keys are internal join material, never fields of the browser response.
|
||||
type ActorIdentity struct {
|
||||
ActorID string `json:"actor_id"`
|
||||
StaffID string `json:"staff_id"`
|
||||
UnionID string `json:"union_id"`
|
||||
}
|
||||
type OrganizationVersion struct {
|
||||
State string `json:"state"`
|
||||
DirectoryID string `json:"directory_id"`
|
||||
DirectorySource string `json:"directory_source"`
|
||||
SyncedAt *time.Time `json:"synced_at"`
|
||||
RevisionID string `json:"revision_id"`
|
||||
Basis string `json:"basis"`
|
||||
}
|
||||
type RecordOrganization struct {
|
||||
Version OrganizationVersion `json:"version"`
|
||||
State string `json:"state"`
|
||||
Departments []string `json:"departments"`
|
||||
Region string `json:"region"`
|
||||
SourceDepartment string `json:"source_department"`
|
||||
SourceRegion string `json:"source_region"`
|
||||
DepartmentOrigin string `json:"department_origin"`
|
||||
RegionOrigin string `json:"region_origin"`
|
||||
}
|
||||
type UsageGroup struct {
|
||||
Identity ActorIdentity `json:"-"`
|
||||
Total int64
|
||||
Succeeded int64
|
||||
Failed int64
|
||||
}
|
||||
type UsageGroups struct {
|
||||
Groups []UsageGroup `json:"-"`
|
||||
AsOf time.Time
|
||||
}
|
||||
|
||||
func decodeRecord(raw string) (Record, error) {
|
||||
var wire struct {
|
||||
Record
|
||||
Identity ActorIdentity `json:"_identity"`
|
||||
QualityHash string `json:"_quality_hash"`
|
||||
QualityObservedAt time.Time `json:"_quality_observed_at"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(raw), &wire); err != nil {
|
||||
return Record{}, err
|
||||
}
|
||||
wire.Record.Identity = wire.Identity
|
||||
wire.Record.QualityInputHash = wire.QualityHash
|
||||
wire.Record.QualityObservedAt = wire.QualityObservedAt
|
||||
wire.Record.Coverage = CheckAnswerCoverage(wire.Record.Answer)
|
||||
return wire.Record, nil
|
||||
}
|
||||
func (c *Client) ReadUsageGroups(ctx context.Context, query RecordQuery) (UsageGroups, error) {
|
||||
if err := c.validateQueryWindow(query.Window); err != nil {
|
||||
return UsageGroups{}, err
|
||||
}
|
||||
if query.Cursor != "" {
|
||||
return UsageGroups{}, ErrInvalidQuery
|
||||
}
|
||||
pattern, patternErr := recordKeywordPattern(query.Keyword)
|
||||
if patternErr != nil {
|
||||
return UsageGroups{}, patternErr
|
||||
}
|
||||
result := UsageGroups{Groups: []UsageGroup{}}
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
if q.QueryRow(ctx, "SELECT transaction_timestamp()").Scan(&result.AsOf) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
rows, err := q.Query(ctx, recordsBaseSQL+` SELECT identity::text,count(*),count(*) FILTER(WHERE status='succeeded'),count(*) FILTER(WHERE status='failed') FROM records WHERE `+recordFilterSQL+` GROUP BY identity`, query.Window.TenantKey, query.Window.Start, query.Window.End, pattern)
|
||||
if err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var raw string
|
||||
var group UsageGroup
|
||||
if rows.Scan(&raw, &group.Total, &group.Succeeded, &group.Failed) != nil || json.Unmarshal([]byte(raw), &group.Identity) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
result.Groups = append(result.Groups, group)
|
||||
}
|
||||
return rows.Err()
|
||||
})
|
||||
return result, err
|
||||
}
|
||||
|
||||
// Keep the agreed field ordering after governance enriches an initially empty column.
|
||||
func AddRecordColumns(page *RecordPage, extra ...string) {
|
||||
all := map[string]bool{}
|
||||
for _, col := range append(page.Columns, extra...) {
|
||||
all[col] = true
|
||||
}
|
||||
page.Columns = []string{}
|
||||
for _, col := range recordColumnOrder {
|
||||
if all[col] {
|
||||
page.Columns = append(page.Columns, col)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestQARecordInternalIdentityAndWholeFilterKeys(t *testing.T) {
|
||||
pool := fixturePool(t)
|
||||
dsn := qaRecordsFixture(t, pool)
|
||||
execFixture(t, pool, `UPDATE messages SET actor_union_id_snapshot='fixture-union' WHERE id='msg-req-3'`, `UPDATE messages SET actor_id=NULL WHERE id='msg-req-4'`)
|
||||
client := buildReadOnlyClient(t, dsn)
|
||||
query := RecordQuery{Window: Window{TenantKey: "default", Start: time.Date(2026, 10, 14, 0, 0, 0, 0, time.UTC), End: time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC)}, Limit: 1}
|
||||
page, err := client.ListRecords(context.Background(), query)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if page.Total != 4 || len(page.Items) != 1 || page.Items[0].Identity.StaffID != "" {
|
||||
t.Fatal("fixture first page should be anonymous")
|
||||
}
|
||||
found := false
|
||||
for _, actor := range page.ScopeActors {
|
||||
if actor.StaffID == "ext-actor-1" && actor.UnionID == "fixture-union" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if len(page.ScopeQuality) != 4 || len(page.Items[0].QualityInputHash) != 64 {
|
||||
t.Fatal("whole-scope quality versions missing")
|
||||
}
|
||||
if !found {
|
||||
t.Fatal("whole-filter keys missing; later-page department would be hidden")
|
||||
}
|
||||
item, err := client.GetRecord(context.Background(), "default", "turn-3")
|
||||
if err != nil || item.Identity.StaffID != "ext-actor-1" || item.Identity.UnionID != "fixture-union" {
|
||||
t.Fatalf("stable identity not read from inbound actor/message: %v", err)
|
||||
}
|
||||
scope, se := client.ReadQualityScope(context.Background(), query)
|
||||
if se != nil || len(scope.Keys) != 4 || scope.AsOf.IsZero() {
|
||||
t.Fatal("quality metadata scope incomplete", se)
|
||||
}
|
||||
for _, key := range scope.Keys {
|
||||
if len(key.InputHash) != 64 || key.ObservedAt.IsZero() {
|
||||
t.Fatal("scope omitted input version")
|
||||
}
|
||||
}
|
||||
raw, _ := json.Marshal(page)
|
||||
detail, _ := json.Marshal(item)
|
||||
for _, s := range []string{string(raw), string(detail)} {
|
||||
for _, private := range []string{"ext-actor-1", "fixture-union", "actor-row-1", "_identity", "scope_actors"} {
|
||||
if strings.Contains(s, private) {
|
||||
t.Fatal("internal roster key leaked into API JSON")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestQAUsageGroupsCountEventsOnceOverTheFullFilter(t *testing.T) {
|
||||
pool := fixturePool(t)
|
||||
dsn := qaRecordsFixture(t, pool)
|
||||
|
||||
client := buildReadOnlyClient(t, dsn)
|
||||
query := RecordQuery{Window: Window{TenantKey: "default", Start: time.Date(2026, 10, 14, 0, 0, 0, 0, time.UTC), End: time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC)}}
|
||||
result, err := client.ReadUsageGroups(context.Background(), query)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var total, success, failed int64
|
||||
for _, g := range result.Groups {
|
||||
total += g.Total
|
||||
success += g.Succeeded
|
||||
failed += g.Failed
|
||||
}
|
||||
if total != 4 || success != 3 || failed != 1 || result.AsOf.IsZero() {
|
||||
t.Fatalf("events multiplied/foreign tenant included: %d %d %d", total, success, failed)
|
||||
}
|
||||
query.Keyword = "%"
|
||||
result, err = client.ReadUsageGroups(context.Background(), query)
|
||||
if err != nil || len(result.Groups) != 0 {
|
||||
t.Fatal("usage keyword wildcard not literal")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// Client 是对外部生产库的**独立只读**客户端：自己的 pool、自己的凭据、自己的超时。
|
||||
// 它没有任何写方法，也不做 DDL/迁移 —— 这是本阶段的接口面本身在保证的事。
|
||||
type Client struct {
|
||||
pool *pgxpool.Pool
|
||||
config Config
|
||||
report ACLReport
|
||||
safety SourceSafety
|
||||
}
|
||||
|
||||
// Build 建立只读客户端：
|
||||
// 1. 形状校验 → 2. 从仓库外 0600 secret 读 DSN → 3. 拒绝指向治理主库 →
|
||||
// 4. 建独立 pool（default_transaction_read_only=on、statement_timeout、连接上限）→
|
||||
// 5. Ping → 6. ACL 自检（不是最小只读就 fail closed）。
|
||||
//
|
||||
// 任何一步失败都返回 ErrUnavailable / ErrNotReadOnly；调用方据此把功能标记为不可用，
|
||||
// **绝不回退到治理主库或应用写账号**。
|
||||
func Build(ctx context.Context, cfg Config) (*Client, error) {
|
||||
cfg = cfg.withDefaults()
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
dsn, err := ReadDSNFile(cfg.DSNFile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := GuardGovernanceDatabase(dsn, cfg.GovernanceDatabaseURL); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
poolConfig, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
// 解析失败的原始错误可能带着 DSN，绝不上抛。
|
||||
return nil, fmt.Errorf("%w: DSN could not be parsed", ErrUnavailable)
|
||||
}
|
||||
poolConfig.MaxConns = cfg.MaxConns
|
||||
poolConfig.MinConns = cfg.MinConns
|
||||
poolConfig.MaxConnLifetime = 30 * time.Minute
|
||||
poolConfig.MaxConnIdleTime = 5 * time.Minute
|
||||
poolConfig.ConnConfig.ConnectTimeout = cfg.ConnectTimeout
|
||||
if poolConfig.ConnConfig.RuntimeParams == nil {
|
||||
poolConfig.ConnConfig.RuntimeParams = map[string]string{}
|
||||
}
|
||||
// 会话级只读 + 语句超时：只读默认不是 ACL 的替代品，两层都要有。
|
||||
poolConfig.ConnConfig.RuntimeParams["default_transaction_read_only"] = "on"
|
||||
poolConfig.ConnConfig.RuntimeParams["statement_timeout"] = fmt.Sprintf("%d", (cfg.QueryTimeout + 2*time.Second).Milliseconds())
|
||||
poolConfig.ConnConfig.RuntimeParams["idle_in_transaction_session_timeout"] = "15000"
|
||||
poolConfig.ConnConfig.RuntimeParams["application_name"] = "kg-dobotai-ops-readonly"
|
||||
poolConfig.ConnConfig.RuntimeParams["search_path"] = "pg_catalog,public"
|
||||
if cfg.DialContext != nil {
|
||||
poolConfig.ConnConfig.DialFunc = cfg.DialContext
|
||||
} else if cfg.DialMode == "ssh" {
|
||||
// 拨号地址只从配置里的 DSN 派生（远端解析），不接受 pgx 传入的本机解析结果 ——
|
||||
// 本机 DNS 被 Fake-IP/VPN 劫持时，那个结果会是假地址（实测挂 20s 超时）。
|
||||
dialAddress, addrErr := hostPortOf(dsn)
|
||||
if addrErr != nil {
|
||||
return nil, addrErr
|
||||
}
|
||||
dialer, dialErr := SSHDialer(cfg.SSHTarget, dialAddress)
|
||||
if dialErr != nil {
|
||||
return nil, dialErr
|
||||
}
|
||||
poolConfig.ConnConfig.DialFunc = dialer
|
||||
}
|
||||
|
||||
connectCtx, cancelConnect := context.WithTimeout(ctx, cfg.ConnectTimeout)
|
||||
defer cancelConnect()
|
||||
pool, err := pgxpool.NewWithConfig(connectCtx, poolConfig)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: pool could not be created", ErrUnavailable)
|
||||
}
|
||||
client := &Client{pool: pool, config: cfg}
|
||||
pingCtx, cancelPing := context.WithTimeout(ctx, cfg.ConnectTimeout)
|
||||
defer cancelPing()
|
||||
if err := pool.Ping(pingCtx); err != nil {
|
||||
pool.Close()
|
||||
return nil, fmt.Errorf("%w: cannot reach the external read-only database", ErrUnavailable)
|
||||
}
|
||||
auditCtx, cancelAudit := context.WithTimeout(ctx, cfg.QueryTimeout)
|
||||
defer cancelAudit()
|
||||
report, err := AssertReadOnly(auditCtx, poolQuerier{pool: pool})
|
||||
if err != nil {
|
||||
pool.Close()
|
||||
return nil, err
|
||||
}
|
||||
safety, err := AuditSourceSafety(auditCtx, poolQuerier{pool: pool})
|
||||
if err != nil {
|
||||
pool.Close()
|
||||
return nil, err
|
||||
}
|
||||
if cfg.MaxConns > int32(safety.ConnectionLimit) {
|
||||
pool.Close()
|
||||
return nil, fmt.Errorf("%w: pool exceeds source role connection budget", ErrNotReadOnly)
|
||||
}
|
||||
client.report = report
|
||||
client.safety = safety
|
||||
return client, nil
|
||||
}
|
||||
|
||||
// Close 释放独立 pool（治理主 pool 完全不受影响）。
|
||||
func (c *Client) Close() {
|
||||
if c != nil && c.pool != nil {
|
||||
c.pool.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// ACL 返回启动时自检的权限报告（只含角色与 ACL，不含业务数据）。
|
||||
func (c *Client) ACL() ACLReport { return c.report }
|
||||
|
||||
// Safety preserves transport/TEMP findings without mislabelling them as fixed.
|
||||
func (c *Client) Safety() SourceSafety { return c.safety }
|
||||
|
||||
// Config 返回生效后的配置，**已清掉所有连接/凭据字段**（secret 文件路径与治理库 DSN 都不外传；
|
||||
// 之前只清了 DSNFile，治理库连接串会跟着这个结构被复制出去）。
|
||||
func (c *Client) Config() Config {
|
||||
cfg := c.config
|
||||
cfg.DSNFile = ""
|
||||
cfg.GovernanceDatabaseURL = ""
|
||||
cfg.DialContext = nil
|
||||
return cfg
|
||||
}
|
||||
|
||||
// Available 是给健康检查/日志用的轻量探测：一次 SELECT 1，带自己的超时。
|
||||
func (c *Client) Available(ctx context.Context) error {
|
||||
if c == nil || c.pool == nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
probeCtx, cancel := context.WithTimeout(ctx, c.config.ConnectTimeout)
|
||||
defer cancel()
|
||||
var one int
|
||||
if err := c.pool.QueryRow(probeCtx, "SELECT 1").Scan(&one); err != nil {
|
||||
return fmt.Errorf("%w: probe failed", ErrUnavailable)
|
||||
}
|
||||
if one != 1 {
|
||||
return fmt.Errorf("%w: probe returned an unexpected value", ErrUnavailable)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// poolQuerier 让 *pgxpool.Pool 适配本包的最小接口（ACL 与仓储都只用这两种方法）。
|
||||
type poolQuerier struct{ pool *pgxpool.Pool }
|
||||
|
||||
func (q poolQuerier) QueryRow(ctx context.Context, sql string, args ...any) rowScanner {
|
||||
return q.pool.QueryRow(ctx, sql, args...)
|
||||
}
|
||||
|
||||
func (q poolQuerier) Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error) {
|
||||
return q.pool.Query(ctx, sql, args...)
|
||||
}
|
||||
|
||||
// withReadOnlyTx 把一次读放进 REPEATABLE READ + READ ONLY 事务：统计口径固定在同一个
|
||||
// as_of 上，且即便账号被误配成可写，这里也没有写的机会。
|
||||
func (c *Client) withReadOnlyTx(ctx context.Context, fn func(ctx context.Context, querier querier) error) error {
|
||||
if c == nil || c.pool == nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
queryCtx, cancel := context.WithTimeout(ctx, c.config.QueryTimeout)
|
||||
defer cancel()
|
||||
tx, err := c.pool.BeginTx(queryCtx, pgx.TxOptions{IsoLevel: pgx.RepeatableRead, AccessMode: pgx.ReadOnly})
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: cannot open a read-only transaction", ErrUnavailable)
|
||||
}
|
||||
defer func() { _ = tx.Rollback(context.Background()) }()
|
||||
if _, err := AssertReadOnly(queryCtx, txQuerier{tx: tx}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := AuditSourceSafety(queryCtx, txQuerier{tx: tx}); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := fn(queryCtx, txQuerier{tx: tx}); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(queryCtx); err != nil {
|
||||
return fmt.Errorf("%w: read-only transaction failed", ErrUnavailable)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type querier interface {
|
||||
QueryRow(ctx context.Context, sql string, args ...any) rowScanner
|
||||
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
|
||||
}
|
||||
|
||||
type txQuerier struct{ tx pgx.Tx }
|
||||
|
||||
func (q txQuerier) QueryRow(ctx context.Context, sql string, args ...any) rowScanner {
|
||||
return q.tx.QueryRow(ctx, sql, args...)
|
||||
}
|
||||
|
||||
func (q txQuerier) Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error) {
|
||||
return q.tx.Query(ctx, sql, args...)
|
||||
}
|
||||
|
||||
// ErrUnavailableIfNil 让调用方把 nil 客户端当成"未启用"而不是 panic。
|
||||
func ErrUnavailableIfNil(client *Client) error {
|
||||
if client == nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// hostPortOf 从 DSN 取"主机:端口"（缺端口按 5432），只用于固定隧道的拨号地址。
|
||||
// 只做解析，不返回任何凭据。
|
||||
func hostPortOf(dsn string) (string, error) {
|
||||
parsed, err := url.Parse(strings.TrimSpace(dsn))
|
||||
if err != nil || parsed.Hostname() == "" {
|
||||
return "", fmt.Errorf("%w: DSN has no host to dial", ErrUnavailable)
|
||||
}
|
||||
port := parsed.Port()
|
||||
if port == "" {
|
||||
port = "5432"
|
||||
}
|
||||
return parsed.Hostname() + ":" + port, nil
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 生产只读**组合验证**：Go 客户端 + 最小只读账号 + 经批准的 SSH 通路，真读生产数据。
|
||||
//
|
||||
// 只在前两项都显式给出时运行（默认跳过，绝不在 CI 或误操作下碰生产）：
|
||||
//
|
||||
// KG_DOBOTAI_PROD_VERIFY=1 \
|
||||
// KG_DOBOTAI_PROD_DSN_FILE=~/.config/knowledge-governance/secrets/dobotai_ops_ro.env \
|
||||
// KG_DOBOTAI_PROD_SSH_TARGET=weknora-prod \
|
||||
// go test ./internal/dobotaiops/ -run TestProductionReadOnlyCombinedVerification -v
|
||||
//
|
||||
// 纪律：只用只读账号；只 SELECT；证据里**不打印**正文、员工姓名或任何凭据。
|
||||
func TestProductionReadOnlyCombinedVerification(t *testing.T) {
|
||||
if strings.TrimSpace(os.Getenv("KG_DOBOTAI_PROD_VERIFY")) != "1" {
|
||||
t.Skip("KG_DOBOTAI_PROD_VERIFY≠1：跳过生产组合验证")
|
||||
}
|
||||
dsnFile := expandHome(strings.TrimSpace(os.Getenv("KG_DOBOTAI_PROD_DSN_FILE")))
|
||||
sshTarget := strings.TrimSpace(os.Getenv("KG_DOBOTAI_PROD_SSH_TARGET"))
|
||||
tenantKey := strings.TrimSpace(os.Getenv("KG_DOBOTAI_PROD_TENANT"))
|
||||
if dsnFile == "" || sshTarget == "" || tenantKey == "" {
|
||||
// 租户键必须由调用方给出：查询本身就是"按租户"的，猜一个默认值等于自欺。
|
||||
t.Skip("KG_DOBOTAI_PROD_DSN_FILE / KG_DOBOTAI_PROD_SSH_TARGET / KG_DOBOTAI_PROD_TENANT 未设置：跳过")
|
||||
}
|
||||
dsn, err := ReadDSNFile(dsnFile)
|
||||
if err != nil {
|
||||
t.Fatalf("读只读 secret 失败：%v", err)
|
||||
}
|
||||
parsed, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
port := parsed.Port()
|
||||
if port == "" {
|
||||
port = "5432"
|
||||
}
|
||||
// 拨号地址由 DSN 派生（不写死主机），并经 SSH 通路到达：
|
||||
address := parsed.Hostname() + ":" + port
|
||||
t.Logf("经 SSH 目标 %s 拨号到 %s（只读账号 %s）", sshTarget, address, parsed.User.Username())
|
||||
|
||||
// 关键：DialMode=ssh ⇒ 走 SSHDialer（socketpair + ssh -W），不是直连。
|
||||
client, err := Build(context.Background(), Config{
|
||||
DSNFile: dsnFile,
|
||||
GovernanceDatabaseURL: "postgres://governance@not-this-host:5432/governance",
|
||||
DialMode: "ssh",
|
||||
SSHTarget: sshTarget,
|
||||
ConnectTimeout: 10 * time.Second,
|
||||
QueryTimeout: 8 * time.Second,
|
||||
MaxConns: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Build()（经 SSH 通路 + 只读账号）失败：%v", err)
|
||||
}
|
||||
defer client.Close()
|
||||
ctx := context.Background()
|
||||
|
||||
report := client.ACL()
|
||||
t.Logf("ACL 报告：role=%s super=%v createrole=%v createdb=%v bypassrls=%v replication=%v schema_create=%v db_create=%v "+
|
||||
"memberships=%d set_role=%d owned_rel=%d owned_schema=%d secdef_exec=%d write_privs=%d missing_selects=%d",
|
||||
report.Role, report.Super, report.CreateRole, report.CreateDB, report.BypassRLS, report.Replication,
|
||||
report.SchemaCreate, report.DatabaseCreate, report.RoleMemberships, report.SetRoleCandidates,
|
||||
report.OwnedRelations, report.OwnedSchemas, report.ExecutableSecDefFns,
|
||||
len(report.WritePrivileges), len(report.MissingSelects))
|
||||
|
||||
now := time.Now().UTC()
|
||||
window, err := NewWindow(tenantKey, now.Add(-14*24*time.Hour).Format(time.RFC3339), now.Add(-30*time.Minute).Format(time.RFC3339), 90*24*time.Hour, now)
|
||||
if err != nil {
|
||||
t.Fatalf("NewWindow() error = %v", err)
|
||||
}
|
||||
counts, err := client.QueryCounts(ctx, window, Filter{})
|
||||
if err != nil {
|
||||
t.Fatalf("QueryCounts() error = %v", err)
|
||||
}
|
||||
t.Logf("窗口 %s → %s：turns=%d 状态=%v 渠道=%v", window.Start.Format(time.RFC3339), window.End.Format(time.RFC3339), counts.Total, counts.ByStatus, counts.ByChannel)
|
||||
|
||||
coverage, err := client.QueryRuleCoverage(ctx, window)
|
||||
if err != nil {
|
||||
t.Fatalf("QueryRuleCoverage() error = %v", err)
|
||||
}
|
||||
byVersion := map[string]int64{}
|
||||
for _, group := range coverage.Groups {
|
||||
byVersion[group.EvaluatorVersion+"|"+group.Verdict] += group.Turns
|
||||
}
|
||||
t.Logf("rule_coverage：成功 turn=%d 缺决策=%d 分组(版本|verdict=%d)：%v", coverage.SucceededTurns, coverage.SucceededWithoutDecision, len(coverage.Groups), byVersion)
|
||||
|
||||
page, err := client.ListTurnMetadata(ctx, window, "", 5)
|
||||
if err != nil {
|
||||
t.Fatalf("ListTurnMetadata() error = %v", err)
|
||||
}
|
||||
missingDepartment, missingRegion, linkedNonZero := 0, 0, 0
|
||||
for _, item := range page.Items {
|
||||
if item.DepartmentMissing {
|
||||
missingDepartment++
|
||||
}
|
||||
if item.RegionMissing {
|
||||
missingRegion++
|
||||
}
|
||||
if item.LinkedMessages > 0 {
|
||||
linkedNonZero++
|
||||
}
|
||||
}
|
||||
t.Logf("元数据首页：%d 条（HasMore=%v）｜关联消息>0 的 %d 条｜部门缺失 %d｜区域缺失 %d", len(page.Items), page.HasMore, linkedNonZero, missingDepartment, missingRegion)
|
||||
if len(page.Items) == 0 {
|
||||
t.Fatal("窗口内没有 turn：窗口选错了")
|
||||
}
|
||||
|
||||
// 人员关联回归：用真实 turn 走授权读取，检查关联是否真的命中（关联字段错就会全空）。
|
||||
// 生产形态：一个 turn 的 request（入站）消息带 actor_id，answer（出站）不带 ⇒
|
||||
// 入站必须关联到 actor，出站未关联属正常事实（用 ActorLinked 区分，不混为一谈）。
|
||||
inboundLinked, inboundUnlinked, outboundUnlinked := 0, 0, 0
|
||||
for _, item := range page.Items {
|
||||
detail, err := client.GetTurnForAuthorizedRead(ctx, window.TenantKey, item.ID, func(string) error { return nil })
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, message := range detail.Messages {
|
||||
switch {
|
||||
case message.Direction == "inbound" && message.ActorLinked && message.ActorType != "":
|
||||
inboundLinked++
|
||||
case message.Direction == "inbound":
|
||||
inboundUnlinked++
|
||||
case !message.ActorLinked:
|
||||
outboundUnlinked++
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Logf("人员关联（channel_actors.id = messages.actor_id）：入站已关联 %d / 入站未关联 %d / 出站无 actor %d",
|
||||
inboundLinked, inboundUnlinked, outboundUnlinked)
|
||||
if inboundLinked == 0 {
|
||||
t.Fatal("人员关联一处都没命中：关联字段或通路有问题")
|
||||
}
|
||||
if inboundUnlinked > 0 {
|
||||
t.Fatalf("有 %d 条入站消息没关联到 actor：", inboundUnlinked)
|
||||
}
|
||||
|
||||
// 只读边界：ACL 函数自证（不执行任何写语句）。
|
||||
for _, table := range RequiredTables {
|
||||
if !strings.Contains(fmt.Sprint(report.CheckedTableList), table) {
|
||||
t.Fatalf("自检没覆盖 %s", table)
|
||||
}
|
||||
}
|
||||
if err := client.Available(ctx); err != nil {
|
||||
t.Fatalf("可用性探测失败：%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func expandHome(path string) string {
|
||||
if strings.HasPrefix(path, "~/") {
|
||||
if home, err := os.UserHomeDir(); err == nil {
|
||||
return home + path[1:]
|
||||
}
|
||||
}
|
||||
return path
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"time"
|
||||
)
|
||||
|
||||
type QualityScope struct {
|
||||
Keys []QualityKey
|
||||
AsOf time.Time
|
||||
}
|
||||
|
||||
func (c *Client) ReadQualityScope(ctx context.Context, query RecordQuery) (QualityScope, error) {
|
||||
if err := c.validateQueryWindow(query.Window); err != nil {
|
||||
return QualityScope{}, err
|
||||
}
|
||||
if query.Cursor != "" {
|
||||
return QualityScope{}, ErrInvalidQuery
|
||||
}
|
||||
pattern, err := recordKeywordPattern(query.Keyword)
|
||||
if err != nil {
|
||||
return QualityScope{}, err
|
||||
}
|
||||
result := QualityScope{Keys: []QualityKey{}}
|
||||
err = c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
if q.QueryRow(ctx, "SELECT transaction_timestamp()").Scan(&result.AsOf) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
rows, err := q.Query(ctx, recordsBaseSQL+` SELECT id,quality_hash,strpos(payload->>'answer',$5)>0,(payload->'source_coverage')::text FROM records WHERE `+recordFilterSQL+` ORDER BY id LIMIT 5001`, query.Window.TenantKey, query.Window.Start, query.Window.End, pattern, noPublishedMarker)
|
||||
if err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var key QualityKey
|
||||
var raw string
|
||||
if rows.Scan(&key.ID, &key.InputHash, &key.Marker, &raw) != nil || json.Unmarshal([]byte(raw), &key.Source) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
if len(result.Keys) >= 5000 {
|
||||
return ErrInvalidQuery
|
||||
}
|
||||
key.ObservedAt = result.AsOf
|
||||
result.Keys = append(result.Keys, key)
|
||||
}
|
||||
return rows.Err()
|
||||
})
|
||||
return result, err
|
||||
}
|
||||
@@ -0,0 +1,284 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
const CoverageRuleVersion = "governance-answer-substring-v1"
|
||||
const noPublishedMarker = "暂无已发布"
|
||||
|
||||
var ErrRecordNotFound = errors.New("qa record not found")
|
||||
|
||||
type CoverageCheck struct {
|
||||
Verdict string `json:"verdict,omitempty"`
|
||||
State string `json:"state"`
|
||||
RuleVersion string `json:"rule_version"`
|
||||
MatchedText string `json:"matched_text,omitempty"`
|
||||
}
|
||||
|
||||
// Absence of this exact substring is not evidence that knowledge covers a question.
|
||||
func CheckAnswerCoverage(answer string) CoverageCheck {
|
||||
return CheckMarkerCoverage(strings.Contains(answer, noPublishedMarker))
|
||||
}
|
||||
func CheckMarkerCoverage(matched bool) CoverageCheck {
|
||||
result := CoverageCheck{State: "pending", RuleVersion: CoverageRuleVersion}
|
||||
if matched {
|
||||
result.State = "checked"
|
||||
result.Verdict = "uncovered"
|
||||
result.MatchedText = noPublishedMarker
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
type SourceCoverage struct {
|
||||
Verdict string `json:"verdict"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
EvaluatorVersion string `json:"evaluator_version"`
|
||||
}
|
||||
|
||||
type RecordFeedback struct {
|
||||
AskerValue string `json:"asker_value"`
|
||||
Up int64 `json:"up"`
|
||||
Down int64 `json:"down"`
|
||||
}
|
||||
|
||||
type Record struct {
|
||||
QualityObservedAt time.Time `json:"-"`
|
||||
QualityInputHash string `json:"-"`
|
||||
Quality json.RawMessage `json:"quality,omitempty"`
|
||||
Identity ActorIdentity `json:"-"`
|
||||
Organization *RecordOrganization `json:"organization,omitempty"`
|
||||
ID string `json:"id"`
|
||||
Status string `json:"status"`
|
||||
Question string `json:"question"`
|
||||
Answer string `json:"answer"`
|
||||
AskedAt time.Time `json:"asked_at"`
|
||||
TimeSource string `json:"time_source"`
|
||||
Asker string `json:"asker"`
|
||||
TriggerPerson string `json:"trigger_person"`
|
||||
Department string `json:"department"`
|
||||
Region string `json:"region"`
|
||||
References json.RawMessage `json:"references"`
|
||||
Media json.RawMessage `json:"media"`
|
||||
SourceCoverage SourceCoverage `json:"source_coverage"`
|
||||
Feedback RecordFeedback `json:"feedback"`
|
||||
Coverage CoverageCheck `json:"coverage"`
|
||||
}
|
||||
|
||||
type RecordQuery struct {
|
||||
Window Window
|
||||
Keyword string
|
||||
Cursor string
|
||||
Limit int
|
||||
}
|
||||
|
||||
type QualityKey struct {
|
||||
ObservedAt time.Time `json:"-"`
|
||||
Marker bool `json:"marker"`
|
||||
Source SourceCoverage `json:"source"`
|
||||
ID string `json:"id"`
|
||||
InputHash string `json:"input_hash"`
|
||||
}
|
||||
type RecordPage struct {
|
||||
ScopeQuality []QualityKey `json:"-"`
|
||||
ScopeActors []ActorIdentity `json:"-"`
|
||||
Organization *OrganizationVersion `json:"organization,omitempty"`
|
||||
Items []Record `json:"items"`
|
||||
Columns []string `json:"columns"`
|
||||
Total int64 `json:"total"`
|
||||
HasMore bool `json:"has_more"`
|
||||
NextCursor string `json:"next_cursor"`
|
||||
AsOf time.Time `json:"as_of"`
|
||||
}
|
||||
|
||||
type recordCursor struct {
|
||||
AskedAt time.Time `json:"at"`
|
||||
ID string `json:"id"`
|
||||
Scope string `json:"scope"`
|
||||
}
|
||||
|
||||
// A single inbound-message join controls names/time. Feedback is preaggregated;
|
||||
// joining each feedback row directly would multiply the business event count.
|
||||
const actorIdentitySQL = `jsonb_build_object('actor_id',COALESCE(a.id,''),'staff_id',COALESCE(a.external_actor_id,''),'union_id',COALESCE(m.actor_union_id_snapshot,''))`
|
||||
const qualityHashSQL = `encode(sha256(convert_to(jsonb_build_array('quality-input-v1',t.id,t.status,t.question,t.answer,t.references_json,COALESCE(m.media_json,'[]'::jsonb))::text,'UTF8')),'hex')`
|
||||
const recordsBaseSQL = `
|
||||
WITH records AS (
|
||||
SELECT t.id,COALESCE(m.created_at,t.created_at) AS asked_at,t.question,t.status,` + actorIdentitySQL + ` AS identity,` + qualityHashSQL + ` AS quality_hash,
|
||||
jsonb_build_object(
|
||||
'_identity',` + actorIdentitySQL + `, '_quality_hash',` + qualityHashSQL + `,'_quality_observed_at',transaction_timestamp(),
|
||||
'id',t.id,'status',t.status,'question',t.question,'answer',t.answer,
|
||||
'asked_at',COALESCE(m.created_at,t.created_at),
|
||||
'time_source',CASE WHEN m.id IS NULL THEN 'turn.created_at' ELSE 'request_message.created_at' END,
|
||||
'asker',COALESCE(NULLIF(m.actor_name_snapshot,''),a.display_name,''),
|
||||
'trigger_person',COALESCE(m.trigger_person_snapshot,''),
|
||||
'department',COALESCE(m.department_snapshot,''),'region',COALESCE(m.region_snapshot,''),
|
||||
'references',t.references_json,'media',COALESCE(m.media_json,'[]'::jsonb),
|
||||
'source_coverage',jsonb_build_object('verdict',COALESCE(c.verdict,''),'reason_code',COALESCE(c.reason_code,''),'evaluator_version',COALESCE(c.evaluator_version,'')),
|
||||
'feedback',jsonb_build_object('asker_value',COALESCE(f.asker_value,''),'up',f.up_count,'down',f.down_count)
|
||||
) AS payload
|
||||
FROM public.turns t
|
||||
LEFT JOIN public.messages m ON m.id=t.request_message_id AND m.tenant_key=t.tenant_key AND m.channel=t.channel AND m.direction='inbound'
|
||||
LEFT JOIN public.channel_actors a ON a.id=m.actor_id AND a.tenant_key=t.tenant_key AND a.channel=t.channel
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT verdict,reason_code,evaluator_version FROM public.coverage_decisions c
|
||||
WHERE c.turn_id=t.id AND c.tenant_key=t.tenant_key AND c.channel=t.channel
|
||||
ORDER BY c.created_at DESC,c.evaluator_version DESC LIMIT 1
|
||||
) c ON true
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT count(*) FILTER(WHERE value='up') AS up_count,count(*) FILTER(WHERE value='down') AS down_count,
|
||||
max(value) FILTER(WHERE actor_id=m.actor_id) AS asker_value
|
||||
FROM public.answer_feedback f
|
||||
WHERE f.turn_id=t.id AND f.answer_message_id=t.answer_message_id AND f.tenant_key=t.tenant_key AND f.channel=t.channel
|
||||
) f ON true
|
||||
WHERE t.tenant_key=$1 AND t.channel='dingtalk'
|
||||
)`
|
||||
|
||||
const recordFilterSQL = ` asked_at >= $2 AND asked_at < $3 AND ($4='' OR question ILIKE $4 ESCAPE '\')`
|
||||
|
||||
// Availability is over the entire filtered result, not just the current page.
|
||||
const recordColumnsSQL = `SELECT count(*),jsonb_build_object(
|
||||
'answer',COALESCE(bool_or(payload->>'answer'<>''),false),
|
||||
'coverage',COALESCE(bool_or(strpos(payload->>'answer',$5)>0),false),
|
||||
'asked_at',count(*)>0,
|
||||
'asker',COALESCE(bool_or(payload->>'asker'<>''),false),
|
||||
'question',COALESCE(bool_or(question<>''),false),
|
||||
'trigger_person',COALESCE(bool_or(payload->>'trigger_person'<>''),false),
|
||||
'department',COALESCE(bool_or(payload->>'department'<>''),false),
|
||||
'region',COALESCE(bool_or(payload->>'region'<>''),false),
|
||||
'feedback',COALESCE(bool_or(payload->'feedback'->>'asker_value'<>'' OR
|
||||
(payload->'feedback'->>'up')::bigint>0 OR (payload->'feedback'->>'down')::bigint>0),false)
|
||||
)::text,COALESCE(jsonb_agg(DISTINCT identity),'[]'::jsonb)::text,COALESCE(jsonb_agg(jsonb_build_object('id',id,'input_hash',quality_hash,'marker',strpos(payload->>'answer',$5)>0,'source',payload->'source_coverage')),'[]'::jsonb)::text FROM records WHERE `
|
||||
|
||||
var recordColumnOrder = []string{"question_chinese", "answer", "coverage", "is_correct", "product_category", "problem_type", "root_cause", "asked_at", "asker", "question", "language", "trigger_person", "department", "region", "feedback"}
|
||||
|
||||
func recordKeywordPattern(keyword string) (string, error) {
|
||||
if !utf8.ValidString(keyword) || utf8.RuneCountInString(keyword) > 200 || strings.ContainsRune(keyword, 0) {
|
||||
return "", ErrInvalidQuery
|
||||
}
|
||||
if keyword == "" {
|
||||
return "", nil
|
||||
}
|
||||
return "%" + strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`).Replace(keyword) + "%", nil
|
||||
}
|
||||
|
||||
func (c *Client) ListRecords(ctx context.Context, query RecordQuery) (RecordPage, error) {
|
||||
if err := c.validateQueryWindow(query.Window); err != nil {
|
||||
return RecordPage{}, err
|
||||
}
|
||||
if query.Limit < 1 || query.Limit > 100 || len(query.Cursor) > 2048 {
|
||||
return RecordPage{}, fmt.Errorf("%w: invalid record filters", ErrInvalidQuery)
|
||||
}
|
||||
scopeBytes, _ := json.Marshal([]string{query.Window.TenantKey, query.Window.Start.UTC().Format(time.RFC3339Nano), query.Window.End.UTC().Format(time.RFC3339Nano), query.Keyword})
|
||||
scopeSum := sha256.Sum256(scopeBytes)
|
||||
scope := hex.EncodeToString(scopeSum[:])
|
||||
var cursor recordCursor
|
||||
var before any
|
||||
if query.Cursor != "" {
|
||||
decoded, err := base64.RawURLEncoding.DecodeString(query.Cursor)
|
||||
if err != nil || json.Unmarshal(decoded, &cursor) != nil || cursor.Scope != scope || cursor.ID == "" || len(cursor.ID) > 256 || cursor.AskedAt.Before(query.Window.Start) || !cursor.AskedAt.Before(query.Window.End) {
|
||||
return RecordPage{}, fmt.Errorf("%w: invalid or mismatched cursor", ErrInvalidQuery)
|
||||
}
|
||||
before = cursor.AskedAt
|
||||
}
|
||||
pattern, patternErr := recordKeywordPattern(query.Keyword)
|
||||
if patternErr != nil {
|
||||
return RecordPage{}, patternErr
|
||||
}
|
||||
args := []any{query.Window.TenantKey, query.Window.Start, query.Window.End, pattern}
|
||||
page := RecordPage{Items: make([]Record, 0), Columns: make([]string, 0)}
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
if err := q.QueryRow(ctx, "SELECT transaction_timestamp()").Scan(&page.AsOf); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
var rawColumns, rawActors, rawQuality string
|
||||
if err := q.QueryRow(ctx, recordsBaseSQL+recordColumnsSQL+recordFilterSQL, append(args, noPublishedMarker)...).Scan(&page.Total, &rawColumns, &rawActors, &rawQuality); err != nil {
|
||||
return fmt.Errorf("%w: record count failed", ErrUnavailable)
|
||||
}
|
||||
if json.Unmarshal([]byte(rawQuality), &page.ScopeQuality) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
for i := range page.ScopeQuality {
|
||||
page.ScopeQuality[i].ObservedAt = page.AsOf
|
||||
}
|
||||
var available map[string]bool
|
||||
if json.Unmarshal([]byte(rawActors), &page.ScopeActors) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
if err := json.Unmarshal([]byte(rawColumns), &available); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
for _, column := range recordColumnOrder {
|
||||
if available[column] {
|
||||
page.Columns = append(page.Columns, column)
|
||||
}
|
||||
}
|
||||
rows, err := q.Query(ctx, recordsBaseSQL+` SELECT payload::text FROM records WHERE `+recordFilterSQL+`
|
||||
AND ($5::timestamptz IS NULL OR (asked_at,id)<($5,$6)) ORDER BY asked_at DESC,id DESC LIMIT $7`, append(args, before, cursor.ID, query.Limit+1)...)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: record list failed", ErrUnavailable)
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var raw string
|
||||
if err := rows.Scan(&raw); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
item, err := decodeRecord(raw)
|
||||
if err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
|
||||
page.Items = append(page.Items, item)
|
||||
}
|
||||
return rows.Err()
|
||||
})
|
||||
if err != nil {
|
||||
return RecordPage{}, err
|
||||
}
|
||||
if len(page.Items) > query.Limit {
|
||||
page.HasMore = true
|
||||
page.Items = page.Items[:query.Limit]
|
||||
last := page.Items[len(page.Items)-1]
|
||||
raw, err := json.Marshal(recordCursor{AskedAt: last.AskedAt, ID: last.ID, Scope: scope})
|
||||
if err != nil {
|
||||
return RecordPage{}, ErrUnavailable
|
||||
}
|
||||
page.NextCursor = base64.RawURLEncoding.EncodeToString(raw)
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
func (c *Client) GetRecord(ctx context.Context, tenantKey, id string) (Record, error) {
|
||||
if strings.TrimSpace(tenantKey) == "" || id == "" || len(id) > 256 || strings.ContainsRune(id, 0) {
|
||||
return Record{}, ErrInvalidQuery
|
||||
}
|
||||
var item Record
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
var raw string
|
||||
err := q.QueryRow(ctx, recordsBaseSQL+` SELECT payload::text FROM records WHERE id=$2`, tenantKey, id).Scan(&raw)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return ErrRecordNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
item, err = decodeRecord(raw)
|
||||
if err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
return item, err
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func qaRecordsFixture(t *testing.T, pool *pgxpool.Pool) string {
|
||||
t.Helper()
|
||||
dsn := buildFixture(t, pool)
|
||||
execFixture(t, pool,
|
||||
`ALTER TABLE messages ADD COLUMN actor_name_snapshot text NOT NULL DEFAULT '', ADD COLUMN trigger_person_snapshot text NOT NULL DEFAULT '', ADD COLUMN media_json jsonb NOT NULL DEFAULT '[]'`,
|
||||
`DROP TABLE IF EXISTS answer_feedback`,
|
||||
`CREATE TABLE answer_feedback(id text PRIMARY KEY,tenant_key text,channel text,turn_id text,answer_message_id text,actor_id text,value text)`,
|
||||
`GRANT SELECT ON answer_feedback TO kg_ro_fixture`,
|
||||
`UPDATE messages SET actor_name_snapshot='测试人员',trigger_person_snapshot='回调昵称' WHERE id='msg-req-3'`,
|
||||
`UPDATE turns SET answer='根据引用，当前暂无已发布的权威知识，暂不建议操作。' WHERE id='turn-3'`,
|
||||
`INSERT INTO answer_feedback VALUES ('fb1','default','dingtalk','turn-3','msg-ans-3','actor-row-1','up'),('fb2','default','dingtalk','turn-3','msg-ans-3','another-person','down'),('fb3','other-tenant','dingtalk','turn-3','msg-ans-3','actor-row-1','down')`,
|
||||
)
|
||||
return dsn
|
||||
}
|
||||
|
||||
func TestQARecordSearchPreservesSourceAndDoesNotMultiplyFeedback(t *testing.T) {
|
||||
pool := fixturePool(t)
|
||||
client := buildReadOnlyClient(t, qaRecordsFixture(t, pool))
|
||||
window := Window{TenantKey: "default", Start: time.Date(2026, 10, 14, 0, 0, 0, 0, time.UTC), End: time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC)}
|
||||
query := RecordQuery{Window: window, Keyword: "第三", Limit: 20}
|
||||
page, err := client.ListRecords(context.Background(), query)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if page.Total != 1 || len(page.Items) != 1 {
|
||||
t.Fatalf("feedback multiplied turns: total=%d rows=%d", page.Total, len(page.Items))
|
||||
}
|
||||
item := page.Items[0]
|
||||
if item.ID != "turn-3" || item.Question != "第三问（正文）" || item.Asker != "测试人员" || item.TriggerPerson != "回调昵称" {
|
||||
t.Fatalf("explicit field mapping failed: %+v", item)
|
||||
}
|
||||
if item.Coverage.Verdict != "uncovered" || item.SourceCoverage.Verdict != "covered" {
|
||||
t.Fatalf("governance must override reference-prioritized source without rewriting it: %+v", item)
|
||||
}
|
||||
if item.Feedback.AskerValue != "up" || item.Feedback.Up != 1 || item.Feedback.Down != 1 {
|
||||
t.Fatalf("asker and aggregate feedback were conflated: %+v", item.Feedback)
|
||||
}
|
||||
query.Keyword = "%"
|
||||
page, err = client.ListRecords(context.Background(), query)
|
||||
if err != nil || page.Total != 0 {
|
||||
t.Fatalf("keyword wildcard must be literal: total=%d err=%v", page.Total, err)
|
||||
}
|
||||
query.Keyword = ""
|
||||
query.Limit = 1
|
||||
page, err = client.ListRecords(context.Background(), query)
|
||||
if err != nil || page.Total != 4 || len(page.Items) != 1 || !page.HasMore || page.NextCursor == "" {
|
||||
t.Fatalf("invalid first page: %+v %v", page, err)
|
||||
}
|
||||
firstID := page.Items[0].ID
|
||||
foundFeedback := false
|
||||
for _, column := range page.Columns {
|
||||
if column == "feedback" {
|
||||
foundFeedback = true
|
||||
}
|
||||
}
|
||||
if !foundFeedback {
|
||||
t.Fatal("a populated column elsewhere in the filtered result must not be hidden by an empty first page")
|
||||
}
|
||||
query.Cursor = page.NextCursor
|
||||
next, err := client.ListRecords(context.Background(), query)
|
||||
if err != nil || len(next.Items) != 1 || next.Items[0].ID == firstID {
|
||||
t.Fatalf("invalid cursor page: %+v %v", next, err)
|
||||
}
|
||||
query.Keyword = "changed"
|
||||
if _, err := client.ListRecords(context.Background(), query); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("cursor must be scoped to its filters: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Explicit opt-in only. This test executes catalog/SELECT queries, never
|
||||
// migrations, permission changes, fixture helpers or business writes.
|
||||
func TestProductionQAQueryReadOnlyVerification(t *testing.T) {
|
||||
if os.Getenv("KG_DOBOTAI_PROD_VERIFY") != "1" {
|
||||
t.Skip("explicit production read verification is disabled")
|
||||
}
|
||||
path, target := os.Getenv("KG_DOBOTAI_PROD_DSN_FILE"), os.Getenv("KG_DOBOTAI_PROD_SSH_TARGET")
|
||||
if path == "" || target == "" {
|
||||
t.Fatal("approved source credential file and SSH target required")
|
||||
}
|
||||
client, err := Build(context.Background(), Config{DSNFile: path, GovernanceDatabaseURL: "postgres://governance@not-this-source/governance", DialMode: "ssh", SSHTarget: target, ConnectTimeout: 10 * time.Second, QueryTimeout: 12 * time.Second, MaxConns: 1})
|
||||
if err != nil {
|
||||
t.Fatalf("dedicated read source blocked: %v", err)
|
||||
}
|
||||
defer client.Close()
|
||||
now := time.Now().UTC()
|
||||
start, end := now.Add(-14*24*time.Hour), now.Add(-30*time.Minute)
|
||||
tenant := os.Getenv("KG_DOBOTAI_PROD_TENANT")
|
||||
if tenant == "" {
|
||||
// Discover, never guess: refuse if this approved source is not single-tenant.
|
||||
var tenants int
|
||||
err = client.withReadOnlyTx(context.Background(), func(ctx context.Context, q querier) error {
|
||||
return q.QueryRow(ctx, `SELECT count(DISTINCT tenant_key),COALESCE(min(tenant_key),'') FROM public.turns WHERE channel='dingtalk' AND created_at >= $1 AND created_at < $2`, start, end).Scan(&tenants, &tenant)
|
||||
})
|
||||
if err != nil || tenants != 1 || tenant == "" {
|
||||
t.Fatal("source does not identify one authorized tenant; explicit scope is required")
|
||||
}
|
||||
}
|
||||
query := RecordQuery{Window: Window{TenantKey: tenant, Start: start, End: end}, Limit: 5}
|
||||
page, err := client.ListRecords(context.Background(), query)
|
||||
if err != nil {
|
||||
t.Fatalf("real query implementation failed: %v", err)
|
||||
}
|
||||
if page.Total < 1 || len(page.Items) < 1 {
|
||||
t.Fatal("no real source data in the bounded window")
|
||||
}
|
||||
detail, err := client.GetRecord(context.Background(), tenant, page.Items[0].ID)
|
||||
if err != nil || detail.Question != page.Items[0].Question || detail.Answer != page.Items[0].Answer {
|
||||
t.Fatal("source list/detail content mismatch")
|
||||
}
|
||||
if _, err := client.GetRecord(context.Background(), "qaops-not-authorized-tenant", page.Items[0].ID); !errors.Is(err, ErrRecordNotFound) {
|
||||
t.Fatal("tenant isolation failed")
|
||||
}
|
||||
if page.HasMore {
|
||||
query.Cursor = page.NextCursor
|
||||
next, err := client.ListRecords(context.Background(), query)
|
||||
if err != nil {
|
||||
t.Fatalf("real cursor page failed: %v", err)
|
||||
}
|
||||
ids := map[string]bool{}
|
||||
for _, item := range page.Items {
|
||||
ids[item.ID] = true
|
||||
}
|
||||
for _, item := range next.Items {
|
||||
if ids[item.ID] {
|
||||
t.Fatal("cursor pagination repeated a turn")
|
||||
}
|
||||
}
|
||||
}
|
||||
safety := client.Safety()
|
||||
if output := os.Getenv("KG_DOBOTAI_PROD_SCOPE_FILE"); output != "" {
|
||||
// Optional local browser-harness handoff: one tenant key only, never
|
||||
// credentials/employee data, outside Git, exclusive creation and 0600.
|
||||
if !strings.HasPrefix(filepath.Clean(output), filepath.Clean(os.TempDir())+string(os.PathSeparator)) {
|
||||
t.Fatal("scope handoff must stay inside the scratch directory")
|
||||
}
|
||||
file, err := os.OpenFile(output, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
|
||||
if err != nil {
|
||||
t.Fatal("cannot create private scope handoff")
|
||||
}
|
||||
_, writeErr := file.WriteString(tenant)
|
||||
closeErr := file.Close()
|
||||
if writeErr != nil || closeErr != nil {
|
||||
t.Fatal("private scope handoff failed")
|
||||
}
|
||||
}
|
||||
t.Logf("REAL_SOURCE_READ_VERIFIED total=%d first_page=%d cursor=%t columns=%v as_of=%s tables_audited=%d effective_table_writes=%d column_writes=%d sequence_writes=%d schema_creates=%d temp=%t ssl=%t; no content or identity logged", page.Total, len(page.Items), page.HasMore, page.Columns, page.AsOf.UTC().Format(time.RFC3339), safety.TablesChecked, safety.TableWrites, safety.ColumnWrites, safety.SequenceWrites, safety.SchemaCreates, safety.DatabaseTemp, safety.SSL)
|
||||
}
|
||||
@@ -0,0 +1,307 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 本阶段只读查询模块（工单第 4 项）。纪律：
|
||||
// - 一律绑定参数；列固定；不拼接调用方给的 SQL/排序字段；
|
||||
// - 时间统一半开区间 created_at >= $start AND created_at < $end；
|
||||
// - 按 turn 计次；跨租户/渠道的连接同时校验 tenant_key 与 channel；
|
||||
// - 只走 REPEATABLE READ + READ ONLY 事务，口径固定在同一个 as_of 上。
|
||||
|
||||
const countsTotalSQL = `
|
||||
SELECT count(*)::bigint
|
||||
FROM turns
|
||||
WHERE tenant_key = $1 AND created_at >= $2 AND created_at < $3
|
||||
AND ($4 = '' OR channel = $4) AND ($5 = '' OR status = $5)`
|
||||
|
||||
const countsBySQL = `
|
||||
SELECT %s, count(*)::bigint
|
||||
FROM turns
|
||||
WHERE tenant_key = $1 AND created_at >= $2 AND created_at < $3
|
||||
AND ($4 = '' OR channel = $4) AND ($5 = '' OR status = $5)
|
||||
GROUP BY 1
|
||||
ORDER BY 1`
|
||||
|
||||
// QueryCounts 返回某租户在窗口内的问答总量与状态/渠道分组（按 turn 计次）。
|
||||
func (c *Client) QueryCounts(ctx context.Context, window Window, filter Filter) (CountsResult, error) {
|
||||
if err := filter.Validate(); err != nil {
|
||||
return CountsResult{}, err
|
||||
}
|
||||
if err := c.validateQueryWindow(window); err != nil {
|
||||
return CountsResult{}, err
|
||||
}
|
||||
result := CountsResult{TenantKey: window.TenantKey, Start: window.Start, End: window.End,
|
||||
ByStatus: map[string]int64{}, ByChannel: map[string]int64{}}
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
if err := q.QueryRow(ctx, countsTotalSQL, window.TenantKey, window.Start, window.End, filter.Channel, filter.Status).
|
||||
Scan(&result.Total); err != nil {
|
||||
return fmt.Errorf("%w: total count failed", ErrUnavailable)
|
||||
}
|
||||
statusRows, err := q.Query(ctx, fmt.Sprintf(countsBySQL, "status"), window.TenantKey, window.Start, window.End, filter.Channel, filter.Status)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: status grouping failed", ErrUnavailable)
|
||||
}
|
||||
defer statusRows.Close()
|
||||
for statusRows.Next() {
|
||||
var status string
|
||||
var count int64
|
||||
if err := statusRows.Scan(&status, &count); err != nil {
|
||||
return fmt.Errorf("%w: status grouping failed", ErrUnavailable)
|
||||
}
|
||||
result.ByStatus[status] = count
|
||||
}
|
||||
channelRows, err := q.Query(ctx, fmt.Sprintf(countsBySQL, "channel"), window.TenantKey, window.Start, window.End, filter.Channel, filter.Status)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: channel grouping failed", ErrUnavailable)
|
||||
}
|
||||
defer channelRows.Close()
|
||||
for channelRows.Next() {
|
||||
var channel string
|
||||
var count int64
|
||||
if err := channelRows.Scan(&channel, &count); err != nil {
|
||||
return fmt.Errorf("%w: channel grouping failed", ErrUnavailable)
|
||||
}
|
||||
result.ByChannel[channel] = count
|
||||
}
|
||||
if err := statusRows.Err(); err != nil {
|
||||
return fmt.Errorf("%w: status grouping failed", ErrUnavailable)
|
||||
}
|
||||
return channelRows.Err()
|
||||
})
|
||||
if err != nil {
|
||||
return CountsResult{}, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
const coverageByVersionSQL = `
|
||||
SELECT c.evaluator_version, c.verdict, c.reason_code, count(DISTINCT c.turn_id)::bigint
|
||||
FROM coverage_decisions c
|
||||
JOIN turns t ON t.id = c.turn_id AND t.tenant_key = c.tenant_key AND t.channel = c.channel
|
||||
WHERE t.tenant_key = $1 AND t.created_at >= $2 AND t.created_at < $3
|
||||
GROUP BY 1, 2, 3
|
||||
ORDER BY 1, 2, 3`
|
||||
|
||||
const succeededTurnStatsSQL = `
|
||||
SELECT count(*)::bigint,
|
||||
count(*) FILTER (
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM coverage_decisions c
|
||||
WHERE c.turn_id = t.id AND c.tenant_key = t.tenant_key AND c.channel = t.channel
|
||||
)
|
||||
)::bigint
|
||||
FROM turns t
|
||||
WHERE t.tenant_key = $1 AND t.status = 'succeeded' AND t.created_at >= $2 AND t.created_at < $3`
|
||||
|
||||
// QueryRuleCoverage 返回规则覆盖分组与"成功 turn 缺决策"量。
|
||||
// 注意：这是 rule_coverage（规则决策），**不是** AI 正确率或正式知识覆盖率；
|
||||
// 不同 evaluator_version 的口径不会混算。
|
||||
func (c *Client) QueryRuleCoverage(ctx context.Context, window Window) (CoverageResult, error) {
|
||||
if err := c.validateQueryWindow(window); err != nil {
|
||||
return CoverageResult{}, err
|
||||
}
|
||||
result := CoverageResult{NotAnAccuracyMeasure: "rule_coverage 只是规则决策覆盖；v5 的 covered 含非知识响应/纯视觉识别，不能当 AI 正确率或正式知识覆盖率"}
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
rows, err := q.Query(ctx, coverageByVersionSQL, window.TenantKey, window.Start, window.End)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: coverage grouping failed", ErrUnavailable)
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var group CoverageGroup
|
||||
if err := rows.Scan(&group.EvaluatorVersion, &group.Verdict, &group.ReasonCode, &group.Turns); err != nil {
|
||||
return fmt.Errorf("%w: coverage grouping failed", ErrUnavailable)
|
||||
}
|
||||
result.Groups = append(result.Groups, group)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return fmt.Errorf("%w: coverage grouping failed", ErrUnavailable)
|
||||
}
|
||||
if err := q.QueryRow(ctx, succeededTurnStatsSQL, window.TenantKey, window.Start, window.End).
|
||||
Scan(&result.SucceededTurns, &result.SucceededWithoutDecision); err != nil {
|
||||
return fmt.Errorf("%w: succeeded turn stats failed", ErrUnavailable)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return CoverageResult{}, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
const listTurnMetadataSQL = `
|
||||
SELECT t.id, t.created_at, t.status, t.channel, t.agent_id, t.knowledge_searched,
|
||||
CASE WHEN jsonb_typeof(t.references_json) = 'array' THEN jsonb_array_length(t.references_json) ELSE 0 END,
|
||||
linked.total,
|
||||
COALESCE(linked.department_missing, TRUE),
|
||||
COALESCE(linked.region_missing, TRUE)
|
||||
FROM turns t
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT count(*)::int AS total,
|
||||
bool_and(COALESCE(m.department_snapshot, '') = '') AS department_missing,
|
||||
bool_and(COALESCE(m.region_snapshot, '') = '') AS region_missing
|
||||
FROM messages m
|
||||
WHERE m.tenant_key = t.tenant_key AND m.channel = t.channel
|
||||
AND m.id = ANY(array_remove(ARRAY[t.request_message_id, t.answer_message_id], NULL))
|
||||
) linked ON TRUE
|
||||
WHERE t.tenant_key = $1 AND t.created_at >= $2 AND t.created_at < $3
|
||||
AND ($4::timestamptz IS NULL OR (t.created_at, t.id) < ($4::timestamptz, $5::text))
|
||||
ORDER BY t.created_at DESC, t.id DESC
|
||||
LIMIT $6`
|
||||
|
||||
// ListTurnMetadata 用 (created_at, id) 游标稳定翻页（倒序）；limit 硬上限 100。
|
||||
// 返回的是元数据：不含正文、不含员工个人值；部门/区域快照为空如实标记为 missing。
|
||||
func (c *Client) ListTurnMetadata(ctx context.Context, window Window, cursor string, limit int) (TurnMetadataPage, error) {
|
||||
if err := c.validateQueryWindow(window); err != nil {
|
||||
return TurnMetadataPage{}, err
|
||||
}
|
||||
if limit <= 0 {
|
||||
limit = 50
|
||||
}
|
||||
if limit > c.config.MaxPageSize {
|
||||
return TurnMetadataPage{}, fmt.Errorf("%w: limit exceeds %d", ErrInvalidQuery, c.config.MaxPageSize)
|
||||
}
|
||||
var decoded Cursor
|
||||
var cursorTime *time.Time
|
||||
if strings.TrimSpace(cursor) != "" {
|
||||
value, err := DecodeCursor(cursor)
|
||||
if err != nil {
|
||||
return TurnMetadataPage{}, err
|
||||
}
|
||||
decoded = value
|
||||
cursorTime = &decoded.CreatedAt
|
||||
}
|
||||
page := TurnMetadataPage{}
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
rows, err := q.Query(ctx, listTurnMetadataSQL, window.TenantKey, window.Start, window.End, cursorTime, decoded.ID, limit+1)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: turn listing failed", ErrUnavailable)
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var item TurnMetadata
|
||||
if err := rows.Scan(&item.ID, &item.CreatedAt, &item.Status, &item.Channel, &item.AgentID,
|
||||
&item.KnowledgeSearched, &item.ReferenceCount, &item.LinkedMessages,
|
||||
&item.DepartmentMissing, &item.RegionMissing); err != nil {
|
||||
return fmt.Errorf("%w: turn listing failed", ErrUnavailable)
|
||||
}
|
||||
page.Items = append(page.Items, item)
|
||||
}
|
||||
return rows.Err()
|
||||
})
|
||||
if err != nil {
|
||||
return TurnMetadataPage{}, err
|
||||
}
|
||||
if len(page.Items) > limit {
|
||||
// 多取一条只为判断是否还有下一页；游标必须指向**最后一条真正返回的行**，
|
||||
// 否则下一页会跳过那一条（集成测试专门钉这个边界）。
|
||||
page.Items = page.Items[:limit]
|
||||
page.HasMore = true
|
||||
tail := page.Items[len(page.Items)-1]
|
||||
page.NextCursor = EncodeCursor(Cursor{CreatedAt: tail.CreatedAt.UTC(), ID: tail.ID})
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
|
||||
const authorizedTurnSQL = `
|
||||
SELECT t.id, t.created_at, t.status, t.channel, t.agent_id, t.question, t.answer,
|
||||
COALESCE(t.references_json::text, '[]'), t.error_message
|
||||
FROM turns t
|
||||
WHERE t.tenant_key = $1 AND t.id = $2`
|
||||
|
||||
const authorizedMessagesSQL = `
|
||||
SELECT m.id, m.created_at, m.direction, m.role, m.message_type, m.status, m.text_content,
|
||||
a.id IS NOT NULL,
|
||||
COALESCE(a.actor_type, '')
|
||||
FROM messages m
|
||||
LEFT JOIN channel_actors a
|
||||
-- 关联字段必须是 channel_actors.id：生产 FK 是 messages.actor_id → channel_actors(id)。
|
||||
-- 用 external_actor_id 关联在生产对拍里命中 0 行（正确关联 2234 行）。
|
||||
ON a.tenant_key = m.tenant_key AND a.channel = m.channel AND a.id = m.actor_id
|
||||
WHERE m.tenant_key = $1 AND m.channel = $2
|
||||
AND m.id = ANY(array_remove(ARRAY[$3, $4], NULL))
|
||||
ORDER BY m.created_at, m.id`
|
||||
|
||||
// GetTurnForAuthorizedRead 按租户 + turn_id 读取正文/引用（必要时）。
|
||||
// **必须**传入授权回调（返回错误即拒绝）；本函数不提供"默认放行"，测试输出也不应打印原文或员工个人值。
|
||||
func (c *Client) GetTurnForAuthorizedRead(ctx context.Context, tenantKey, turnID string, authorize func(turnID string) error) (TurnDetail, error) {
|
||||
if authorize == nil {
|
||||
return TurnDetail{}, fmt.Errorf("%w: an explicit authorization callback is required", ErrInvalidQuery)
|
||||
}
|
||||
if strings.TrimSpace(tenantKey) == "" || strings.TrimSpace(turnID) == "" {
|
||||
return TurnDetail{}, fmt.Errorf("%w: tenant and turn id are required", ErrInvalidQuery)
|
||||
}
|
||||
if err := authorize(turnID); err != nil {
|
||||
return TurnDetail{}, fmt.Errorf("%w: turn read is not authorized", ErrInvalidQuery)
|
||||
}
|
||||
detail := TurnDetail{}
|
||||
var references string
|
||||
var requestID, answerID *string
|
||||
err := c.withReadOnlyTx(ctx, func(ctx context.Context, q querier) error {
|
||||
if err := q.QueryRow(ctx, authorizedTurnSQL, tenantKey, turnID).Scan(
|
||||
&detail.ID, &detail.CreatedAt, &detail.Status, &detail.Channel, &detail.AgentID,
|
||||
&detail.Question, &detail.Answer, &references, &detail.ErrorCode); err != nil {
|
||||
return fmt.Errorf("%w: turn could not be read", ErrUnavailable)
|
||||
}
|
||||
detail.ReferencesJSON = []byte(references)
|
||||
if !json.Valid(detail.ReferencesJSON) {
|
||||
detail.ReferencesJSON = []byte("[]")
|
||||
}
|
||||
// request/answer 两条消息 id 需要一起取：单独查一次拿 id 列表，避免把正文带进列表接口。
|
||||
if err := q.QueryRow(ctx, `SELECT request_message_id, answer_message_id FROM turns WHERE tenant_key = $1 AND id = $2`, tenantKey, turnID).
|
||||
Scan(&requestID, &answerID); err != nil {
|
||||
return fmt.Errorf("%w: turn could not be read", ErrUnavailable)
|
||||
}
|
||||
rows, err := q.Query(ctx, authorizedMessagesSQL, tenantKey, detail.Channel, requestID, answerID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: turn messages could not be read", ErrUnavailable)
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var message TurnMessage
|
||||
if err := rows.Scan(&message.ID, &message.CreatedAt, &message.Direction, &message.Role,
|
||||
&message.MessageType, &message.Status, &message.Text, &message.ActorLinked, &message.ActorType); err != nil {
|
||||
return fmt.Errorf("%w: turn messages could not be read", ErrUnavailable)
|
||||
}
|
||||
detail.Messages = append(detail.Messages, message)
|
||||
}
|
||||
return rows.Err()
|
||||
})
|
||||
if err != nil {
|
||||
return TurnDetail{}, err
|
||||
}
|
||||
return detail, nil
|
||||
}
|
||||
|
||||
// validateQueryWindow 是仓储入口的**完整**校验：不能只靠 NewWindow —— 调用方完全可以
|
||||
// 自己拼一个 Window 结构体绕过它（独立探测已证明）。这里把租户、半开区间、跨度上限
|
||||
// 与"不允许未来窗口"全部落实在每次查询上。
|
||||
func (c *Client) validateQueryWindow(window Window) error {
|
||||
if strings.TrimSpace(window.TenantKey) == "" {
|
||||
return fmt.Errorf("%w: tenant key is required", ErrInvalidQuery)
|
||||
}
|
||||
if window.Start.IsZero() || window.End.IsZero() || !window.Start.Before(window.End) {
|
||||
return fmt.Errorf("%w: window is invalid", ErrInvalidQuery)
|
||||
}
|
||||
limit := c.config.MaxWindow
|
||||
if limit <= 0 {
|
||||
limit = 90 * 24 * time.Hour
|
||||
}
|
||||
if window.End.Sub(window.Start) > limit {
|
||||
return fmt.Errorf("%w: window exceeds the %s limit", ErrInvalidQuery, limit)
|
||||
}
|
||||
now := time.Now()
|
||||
if c.config.Now != nil {
|
||||
now = c.config.Now()
|
||||
}
|
||||
if window.End.After(now.Add(5 * time.Minute)) {
|
||||
return fmt.Errorf("%w: window end is in the future", ErrInvalidQuery)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,347 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// 这些用例只在 TEST_DATABASE_URL 指向**一次性/隔离**数据库时运行（CI 默认跳过）。
|
||||
// 它们**绝不**指向生产：fixture 表、隔离角色、临时 DSN 文件都在这个库内建/删。
|
||||
// 生产侧的真实验证另有入口（纯目录/SELECT，见工单与 runbook）。
|
||||
func fixturePool(t *testing.T) *pgxpool.Pool {
|
||||
t.Helper()
|
||||
// 先过隔离环境护栏（会建表/建角色）：未声明就跳过，声明了但不像一次性库就直接失败。
|
||||
fixtureEnvironment(t)
|
||||
dsn := strings.TrimSpace(os.Getenv("TEST_DATABASE_URL"))
|
||||
pool, err := pgxpool.New(context.Background(), dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
return pool
|
||||
}
|
||||
|
||||
// dropRoleIfExists 先解掉授权依赖再删角色（临时库里可重复运行）。
|
||||
func dropRoleIfExists(t *testing.T, pool *pgxpool.Pool, role string) {
|
||||
t.Helper()
|
||||
var exists bool
|
||||
if err := pool.QueryRow(context.Background(), `SELECT EXISTS(SELECT 1 FROM pg_roles WHERE rolname = $1)`, role).Scan(&exists); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !exists {
|
||||
return
|
||||
}
|
||||
execFixture(t, pool, fmt.Sprintf(`DROP OWNED BY %s`, quoteIdent(role)), fmt.Sprintf(`DROP ROLE %s`, quoteIdent(role)))
|
||||
}
|
||||
|
||||
func execFixture(t *testing.T, pool *pgxpool.Pool, statements ...string) {
|
||||
t.Helper()
|
||||
for _, statement := range statements {
|
||||
if _, err := pool.Exec(context.Background(), statement); err != nil {
|
||||
t.Fatalf("fixture 语句失败（%s）：%v", strings.SplitN(statement, "\n", 2)[0], err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func randomSecret(t *testing.T) string {
|
||||
t.Helper()
|
||||
buf := make([]byte, 32)
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return strings.NewReplacer("+", "A", "/", "B", "=", "C").Replace(base64.URLEncoding.EncodeToString(buf))
|
||||
}
|
||||
|
||||
func writeSecretFile(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "dobotai_ops_ro.env")
|
||||
if err := os.WriteFile(path, []byte(content+"\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
// buildFixture 建表 + 灌数据 + 建一个镜像生产授权面的最小只读角色，返回只读 DSN。
|
||||
func buildFixture(t *testing.T, pool *pgxpool.Pool) string {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
execFixture(t, pool,
|
||||
`DROP TABLE IF EXISTS messages, coverage_decisions, channel_actors, turns CASCADE`,
|
||||
`CREATE TABLE turns (id text PRIMARY KEY, tenant_key text NOT NULL, channel text NOT NULL,
|
||||
status text NOT NULL, agent_id text NOT NULL DEFAULT '', knowledge_searched boolean NOT NULL DEFAULT false,
|
||||
references_json jsonb NOT NULL DEFAULT '[]'::jsonb, request_message_id text, answer_message_id text,
|
||||
question text NOT NULL DEFAULT '', answer text NOT NULL DEFAULT '', error_message text NOT NULL DEFAULT '',
|
||||
created_at timestamptz NOT NULL)`,
|
||||
`CREATE TABLE messages (id text PRIMARY KEY, tenant_key text NOT NULL, channel text NOT NULL, actor_id text,
|
||||
department_snapshot text NOT NULL DEFAULT '', region_snapshot text NOT NULL DEFAULT '', actor_union_id_snapshot text NOT NULL DEFAULT '',
|
||||
direction text NOT NULL DEFAULT '', role text NOT NULL DEFAULT '', message_type text NOT NULL DEFAULT '',
|
||||
status text NOT NULL DEFAULT '', text_content text NOT NULL DEFAULT '', created_at timestamptz NOT NULL)`,
|
||||
`CREATE TABLE coverage_decisions (id text PRIMARY KEY, tenant_key text NOT NULL, channel text NOT NULL,
|
||||
turn_id text NOT NULL, evaluator_version text NOT NULL, verdict text NOT NULL, reason_code text NOT NULL DEFAULT '',
|
||||
reference_count integer NOT NULL DEFAULT 0, created_at timestamptz NOT NULL,
|
||||
UNIQUE (tenant_key, channel, turn_id))`,
|
||||
`CREATE TABLE channel_actors (id text PRIMARY KEY, tenant_key text NOT NULL, channel text NOT NULL,
|
||||
external_actor_id text NOT NULL, actor_type text NOT NULL, display_name text NOT NULL DEFAULT '',
|
||||
created_at timestamptz NOT NULL)`,
|
||||
`INSERT INTO turns (id, tenant_key, channel, status, agent_id, knowledge_searched, references_json,
|
||||
request_message_id, answer_message_id, question, answer, created_at) VALUES
|
||||
('turn-4', 'default', 'dingtalk', 'succeeded', 'agent-a', false, '[]',
|
||||
'msg-req-4', 'msg-ans-4', '第四问（无规则决策）', '第四答', '2026-10-14T04:00:00Z'),
|
||||
('turn-3', 'default', 'dingtalk', 'succeeded', 'agent-a', true, '[{"doc":"x"},{"doc":"y"}]',
|
||||
'msg-req-3', 'msg-ans-3', '第三问（正文）', '第三答（正文）', '2026-10-14T03:00:00Z'),
|
||||
('turn-2', 'default', 'dingtalk', 'succeeded', 'agent-a', false, '[]',
|
||||
'msg-req-2', 'msg-ans-2', '第二问', '第二答', '2026-10-14T02:00:00Z'),
|
||||
('turn-1', 'default', 'dingtalk', 'failed', 'agent-b', true, '[]',
|
||||
NULL, NULL, '第一问', '', '2026-10-14T01:00:00Z')`,
|
||||
`INSERT INTO messages (id, tenant_key, channel, actor_id, department_snapshot, region_snapshot,
|
||||
direction, role, message_type, status, text_content, created_at) VALUES
|
||||
('msg-req-3', 'default', 'dingtalk', 'actor-row-1', '', '', 'inbound', 'user', 'text', 'ok', '第三问原文', '2026-10-14T03:00:00Z'),
|
||||
('msg-ans-3', 'default', 'dingtalk', 'actor-row-1', '研发部', '华南', 'outbound', 'assistant', 'text', 'ok', '第三答原文', '2026-10-14T03:00:01Z'),
|
||||
('msg-req-2', 'default', 'dingtalk', 'actor-row-1', '', '', 'inbound', 'user', 'text', 'ok', '第二问原文', '2026-10-14T02:00:00Z'),
|
||||
('msg-req-4', 'default', 'dingtalk', 'actor-row-1', '', '', 'inbound', 'user', 'text', 'ok', '第四问原文', '2026-10-14T04:00:00Z'),
|
||||
('msg-ans-4', 'default', 'dingtalk', 'actor-row-1', '', '', 'outbound', 'assistant', 'text', 'ok', '第四答原文', '2026-10-14T04:00:01Z')`,
|
||||
// 真实关系：messages.actor_id 引用 channel_actors(id)；external_actor_id 是另一个
|
||||
// 唯一键（tenant_key, channel, external_actor_id）。夹具必须照这个来 —— 用
|
||||
// external_actor_id 关联就会命中 0 行（生产对拍实测：错 0 / 对 2234）。
|
||||
`INSERT INTO channel_actors (id, tenant_key, channel, external_actor_id, actor_type, display_name, created_at)
|
||||
VALUES ('actor-row-1', 'default', 'dingtalk', 'ext-actor-1', 'employee', '不应外泄的姓名', '2026-10-14T00:00:00Z')`,
|
||||
// 另一个租户的数据：所有查询都必须把它挡在外面（跨租户隔离）。
|
||||
`INSERT INTO turns (id, tenant_key, channel, status, agent_id, created_at)
|
||||
VALUES ('turn-other-tenant', 'another-tenant', 'dingtalk', 'succeeded', 'agent-x', '2026-10-14T03:30:00Z')`,
|
||||
`INSERT INTO coverage_decisions (id, tenant_key, channel, turn_id, evaluator_version, verdict, reason_code, reference_count, created_at)
|
||||
VALUES ('cov-other-tenant', 'another-tenant', 'dingtalk', 'turn-other-tenant', 'v5', 'covered', '', 0, '2026-10-14T03:30:02Z')`,
|
||||
// 生产上 (tenant_key, channel, turn_id) 是唯一的：一个 turn 只会有一条决策，
|
||||
// 不同 turn 才可能来自不同 evaluator_version —— fixture 照生产形态来。
|
||||
`INSERT INTO coverage_decisions (id, tenant_key, channel, turn_id, evaluator_version, verdict, reason_code, reference_count, created_at)
|
||||
VALUES ('cov-1', 'default', 'dingtalk', 'turn-3', 'v5', 'covered', '', 2, '2026-10-14T03:00:02Z'),
|
||||
('cov-2', 'default', 'dingtalk', 'turn-1', 'v5', 'uncovered', 'no_source', 0, '2026-10-14T01:00:02Z'),
|
||||
('cov-3', 'default', 'dingtalk', 'turn-2', 'v4', 'covered', '', 0, '2026-10-14T02:00:02Z')`,
|
||||
)
|
||||
password := randomSecret(t)
|
||||
// 角色在同一个临时库里可能已经存在（多个用例复用）：先删再建，保持用例可重复运行。
|
||||
dropRoleIfExists(t, pool, "kg_ro_fixture")
|
||||
execFixture(t, pool,
|
||||
fmt.Sprintf(`CREATE ROLE kg_ro_fixture LOGIN PASSWORD '%s' CONNECTION LIMIT 4`, password),
|
||||
`ALTER ROLE kg_ro_fixture SET default_transaction_read_only = on`,
|
||||
)
|
||||
// 当前库名（避免写死）
|
||||
var databaseName string
|
||||
if err := pool.QueryRow(ctx, `SELECT current_database()`).Scan(&databaseName); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
execFixture(t, pool,
|
||||
fmt.Sprintf(`GRANT CONNECT ON DATABASE %s TO kg_ro_fixture`, quoteIdent(databaseName)),
|
||||
`GRANT USAGE ON SCHEMA public TO kg_ro_fixture`,
|
||||
`GRANT SELECT ON TABLE public.turns, public.messages, public.coverage_decisions, public.channel_actors TO kg_ro_fixture`,
|
||||
)
|
||||
// 用与生产一致的写法拼只读 DSN：主机/端口/库名从 TEST_DATABASE_URL 派生
|
||||
//（容器内 inet_server_addr 从宿主机不可路由），口令在测试内不打印。
|
||||
return dsnWithCredentials(t, "kg_ro_fixture", password)
|
||||
}
|
||||
|
||||
// dsnWithCredentials 把 TEST_DATABASE_URL 里的用户/口令换掉，其余（主机、端口、库、查询串）保持。
|
||||
func dsnWithCredentials(t *testing.T, user, password string) string {
|
||||
t.Helper()
|
||||
parsed, err := url.Parse(strings.TrimSpace(os.Getenv("TEST_DATABASE_URL")))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
parsed.User = url.UserPassword(user, password)
|
||||
return parsed.String()
|
||||
}
|
||||
|
||||
func quoteIdent(value string) string {
|
||||
return `"` + strings.ReplaceAll(value, `"`, `""`) + `"`
|
||||
}
|
||||
|
||||
func buildReadOnlyClient(t *testing.T, roDSN string) *Client {
|
||||
t.Helper()
|
||||
client, err := Build(context.Background(), Config{
|
||||
DSNFile: writeSecretFile(t, "DOBOTAI_OPS_READ_DSN="+roDSN),
|
||||
GovernanceDatabaseURL: "postgres://governance@elsewhere:5432/governance",
|
||||
DialMode: "direct",
|
||||
ConnectTimeout: 5 * time.Second,
|
||||
QueryTimeout: 5 * time.Second,
|
||||
MaxConns: 2,
|
||||
// 固定"当前时间"：仓储入口会拒未来窗口，夹具的时间窗必须相对一个确定时刻来判。
|
||||
Now: func() time.Time { return time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Build() 失败：%v", err)
|
||||
}
|
||||
t.Cleanup(client.Close)
|
||||
return client
|
||||
}
|
||||
|
||||
func TestReadOnlyRepositoryAgainstFixtureDatabase(t *testing.T) {
|
||||
pool := fixturePool(t)
|
||||
roDSN := buildFixture(t, pool)
|
||||
client := buildReadOnlyClient(t, roDSN)
|
||||
ctx := context.Background()
|
||||
window, err := NewWindow("default", "2026-10-14T00:00:00Z", "2026-10-15T00:00:00Z", 90*24*time.Hour, time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// ACL 自检报告必须干净（这是"账号确实最小只读"的机器证据）。
|
||||
report := client.ACL()
|
||||
if report.Role != "kg_ro_fixture" || report.Super || report.SchemaCreate || len(report.WritePrivileges) != 0 || len(report.MissingSelects) != 0 {
|
||||
t.Fatalf("ACL 报告不干净：%+v", report)
|
||||
}
|
||||
|
||||
counts, err := client.QueryCounts(ctx, window, Filter{})
|
||||
if err != nil {
|
||||
t.Fatalf("QueryCounts() error = %v", err)
|
||||
}
|
||||
if counts.Total != 4 || counts.ByStatus["succeeded"] != 3 || counts.ByStatus["failed"] != 1 || counts.ByChannel["dingtalk"] != 4 {
|
||||
t.Fatalf("counts = %+v", counts)
|
||||
}
|
||||
filtered, err := client.QueryCounts(ctx, window, Filter{Channel: "dingtalk", Status: "succeeded"})
|
||||
if err != nil || filtered.Total != 3 {
|
||||
t.Fatalf("过滤计数 = %+v (%v)", filtered, err)
|
||||
}
|
||||
if _, err := client.QueryCounts(ctx, window, Filter{Status: "nope"}); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("未知状态应被拒：%v", err)
|
||||
}
|
||||
|
||||
coverage, err := client.QueryRuleCoverage(ctx, window)
|
||||
if err != nil {
|
||||
t.Fatalf("QueryRuleCoverage() error = %v", err)
|
||||
}
|
||||
if len(coverage.Groups) != 3 {
|
||||
t.Fatalf("覆盖分组 = %+v", coverage.Groups)
|
||||
}
|
||||
if coverage.SucceededTurns != 3 || coverage.SucceededWithoutDecision != 1 {
|
||||
t.Fatalf("成功 turn 统计 = %+v", coverage)
|
||||
}
|
||||
for _, group := range coverage.Groups {
|
||||
if group.EvaluatorVersion == "v5" && group.Verdict == "uncovered" && group.ReasonCode != "no_source" {
|
||||
t.Fatalf("reason_code 未保留：%+v", group)
|
||||
}
|
||||
}
|
||||
|
||||
first, err := client.ListTurnMetadata(ctx, window, "", 2)
|
||||
if err != nil {
|
||||
t.Fatalf("ListTurnMetadata() error = %v", err)
|
||||
}
|
||||
if len(first.Items) != 2 || !first.HasMore || first.NextCursor == "" {
|
||||
t.Fatalf("第一页 = %+v", first)
|
||||
}
|
||||
if first.Items[0].ID != "turn-4" || first.Items[1].ID != "turn-3" {
|
||||
t.Fatalf("倒序不对：%+v", first.Items)
|
||||
}
|
||||
if first.Items[1].ReferenceCount != 2 || first.Items[1].LinkedMessages != 2 {
|
||||
t.Fatalf("引用/关联消息计数不对：%+v", first.Items[1])
|
||||
}
|
||||
// turn-3：一条关联消息部门/区域为空、一条有值 ⇒ 不该标 missing；turn-4：两条都为空 ⇒ 如实标 missing。
|
||||
if first.Items[1].DepartmentMissing || first.Items[1].RegionMissing {
|
||||
t.Fatalf("有值的关联消息不该标 missing：%+v", first.Items[1])
|
||||
}
|
||||
if !first.Items[0].DepartmentMissing || !first.Items[0].RegionMissing {
|
||||
t.Fatalf("快照为空的关联消息必须如实标 missing：%+v", first.Items[0])
|
||||
}
|
||||
second, err := client.ListTurnMetadata(ctx, window, first.NextCursor, 2)
|
||||
if err != nil {
|
||||
t.Fatalf("第二页 error = %v", err)
|
||||
}
|
||||
if len(second.Items) != 2 || second.Items[0].ID != "turn-2" || second.Items[1].ID != "turn-1" || second.HasMore {
|
||||
t.Fatalf("第二页 = %+v", second)
|
||||
}
|
||||
if _, err := client.ListTurnMetadata(ctx, window, "不是游标", 10); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("非法游标应被拒：%v", err)
|
||||
}
|
||||
if _, err := client.ListTurnMetadata(ctx, window, "", 101); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("超过分页上限应被拒：%v", err)
|
||||
}
|
||||
// 跨租户隔离：另一个租户的 turn 不能出现在任何结果里（计数、覆盖、列表都一样）。
|
||||
otherTenant, err := NewWindow("another-tenant", "2026-10-14T00:00:00Z", "2026-10-15T00:00:00Z", 90*24*time.Hour, time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
otherCounts, err := client.QueryCounts(ctx, otherTenant, Filter{})
|
||||
if err != nil || otherCounts.Total != 1 {
|
||||
t.Fatalf("另一个租户应只看到自己的 1 条：%+v (%v)", otherCounts, err)
|
||||
}
|
||||
for _, item := range append(first.Items, second.Items...) {
|
||||
if item.ID == "turn-other-tenant" {
|
||||
t.Fatalf("跨租户泄漏：%+v", item)
|
||||
}
|
||||
}
|
||||
if _, err := client.GetTurnForAuthorizedRead(ctx, "default", "turn-other-tenant", func(string) error { return nil }); !errors.Is(err, ErrUnavailable) {
|
||||
// default 租户下没有这个 turn ⇒ 读不到（不是"读到了别人的"）
|
||||
t.Fatalf("跨租户 turn 不该在 default 下可读：%v", err)
|
||||
}
|
||||
|
||||
// 授权读取：没有授权回调一律拒绝（不提供"默认放行"）。
|
||||
if _, err := client.GetTurnForAuthorizedRead(ctx, "default", "turn-3", nil); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("缺授权回调必须拒绝：%v", err)
|
||||
}
|
||||
if _, err := client.GetTurnForAuthorizedRead(ctx, "default", "turn-3", func(string) error { return errors.New("no") }); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Fatalf("授权回调拒绝时必须拒绝：%v", err)
|
||||
}
|
||||
detail, err := client.GetTurnForAuthorizedRead(ctx, "default", "turn-3", func(string) error { return nil })
|
||||
if err != nil {
|
||||
t.Fatalf("授权读取 error = %v", err)
|
||||
}
|
||||
if detail.Question != "第三问（正文）" || detail.Answer != "第三答（正文）" || len(detail.Messages) != 2 {
|
||||
t.Fatalf("授权读取内容不对：%+v", detail)
|
||||
}
|
||||
// 关联回归：即使夹具里 id 与 external_actor_id 不同，也必须靠正确的字段命中 actor_type。
|
||||
for _, message := range detail.Messages {
|
||||
if message.ActorType != "employee" {
|
||||
t.Fatalf("人员关联没命中（关联字段必须是 channel_actors.id）：%+v", message)
|
||||
}
|
||||
}
|
||||
if strings.Contains(detail.Messages[0].ActorType, "不应外泄") {
|
||||
t.Fatalf("只应带 actor_type，不应带员工姓名：%+v", detail.Messages[0])
|
||||
}
|
||||
_ = client.Available(ctx)
|
||||
}
|
||||
|
||||
func TestBuildRefusesUnsafeAccountsAndTargets(t *testing.T) {
|
||||
pool := fixturePool(t)
|
||||
roDSN := buildFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
password := randomSecret(t)
|
||||
dropRoleIfExists(t, pool, "kg_writer_fixture")
|
||||
execFixture(t, pool,
|
||||
fmt.Sprintf(`CREATE ROLE kg_writer_fixture LOGIN PASSWORD '%s'`, password),
|
||||
`GRANT USAGE ON SCHEMA public TO kg_writer_fixture`,
|
||||
`GRANT SELECT, INSERT ON TABLE public.turns TO kg_writer_fixture`,
|
||||
`GRANT SELECT ON TABLE public.messages, public.coverage_decisions, public.channel_actors TO kg_writer_fixture`,
|
||||
)
|
||||
writerDSN := dsnWithCredentials(t, "kg_writer_fixture", password)
|
||||
|
||||
// 有写权限的账号：fail closed（不是"能读就算过"）。
|
||||
_, err := Build(ctx, Config{
|
||||
DSNFile: writeSecretFile(t, writerDSN), DialMode: "direct",
|
||||
GovernanceDatabaseURL: "postgres://governance@elsewhere:5432/governance",
|
||||
ConnectTimeout: 5 * time.Second, QueryTimeout: 5 * time.Second})
|
||||
if !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("可写账号必须被拒为 ErrNotReadOnly：%v", err)
|
||||
}
|
||||
// 指向治理主库的 DSN：拒绝（防"误配后读写治理库"）。
|
||||
_, err = Build(ctx, Config{
|
||||
DSNFile: writeSecretFile(t, roDSN), DialMode: "direct", GovernanceDatabaseURL: roDSN,
|
||||
ConnectTimeout: 5 * time.Second, QueryTimeout: 5 * time.Second})
|
||||
if !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("指向治理库必须被拒：%v", err)
|
||||
}
|
||||
// secret 文件缺失：ErrUnavailable（按"未启用"处理，不拖垮治理）。
|
||||
_, err = Build(ctx, Config{DSNFile: filepath.Join(t.TempDir(), "missing.env"), DialMode: "direct",
|
||||
ConnectTimeout: time.Second, QueryTimeout: time.Second})
|
||||
if !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatalf("缺 secret 应为 ErrUnavailable：%v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// SourceSafety records effective privileges, not only grants on queried tables.
|
||||
// TEMP and transport are disclosed independently: this probe never changes them.
|
||||
type SourceSafety struct {
|
||||
Database string `json:"database"`
|
||||
ConnectionLimit int `json:"connection_limit"`
|
||||
TablesChecked int `json:"tables_checked"`
|
||||
TableWrites int `json:"table_writes"`
|
||||
ColumnWrites int `json:"column_writes"`
|
||||
SequenceWrites int `json:"sequence_writes"`
|
||||
SchemaCreates int `json:"schema_creates"`
|
||||
DatabaseTemp bool `json:"database_temp"`
|
||||
SSL bool `json:"ssl"`
|
||||
MissingColumns []string `json:"missing_columns"`
|
||||
}
|
||||
|
||||
const sourceSafetySQL = `
|
||||
WITH relations AS (
|
||||
SELECT c.oid,c.relkind FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
|
||||
WHERE n.nspname NOT LIKE 'pg_%' AND n.nspname<>'information_schema'
|
||||
), required(table_name,column_name,type_name) AS (VALUES
|
||||
('turns','id','text'),('turns','tenant_key','text'),('turns','channel','text'),
|
||||
('turns','question','text'),('turns','answer','text'),('turns','created_at','timestamptz'),
|
||||
('turns','references_json','jsonb'),('turns','knowledge_searched','bool'),
|
||||
('turns','request_message_id','text'),('turns','answer_message_id','text'),
|
||||
('messages','id','text'),('messages','actor_id','text'),('messages','department_snapshot','text'),
|
||||
('messages','region_snapshot','text'),('messages','actor_union_id_snapshot','text'),('channel_actors','id','text'),('channel_actors','external_actor_id','text'),
|
||||
('coverage_decisions','turn_id','text'),('coverage_decisions','verdict','text')
|
||||
)
|
||||
SELECT jsonb_build_object(
|
||||
'database',current_database(),
|
||||
'connection_limit',(SELECT rolconnlimit FROM pg_roles WHERE rolname=current_user),
|
||||
'tables_checked',(SELECT count(*) FROM relations WHERE relkind IN ('r','p','v','m','f')),
|
||||
'table_writes',(SELECT count(*) FROM relations WHERE relkind IN ('r','p','v','m','f') AND (
|
||||
has_table_privilege(oid,'INSERT') OR has_table_privilege(oid,'UPDATE') OR has_table_privilege(oid,'DELETE') OR
|
||||
has_table_privilege(oid,'TRUNCATE') OR has_table_privilege(oid,'REFERENCES') OR has_table_privilege(oid,'TRIGGER') OR
|
||||
CASE WHEN current_setting('server_version_num')::int>=170000 THEN has_table_privilege(oid,'MAINTAIN') ELSE false END)),
|
||||
'column_writes',(SELECT count(*) FROM relations WHERE relkind IN ('r','p','v','m','f') AND (
|
||||
has_any_column_privilege(oid,'INSERT') OR has_any_column_privilege(oid,'UPDATE') OR has_any_column_privilege(oid,'REFERENCES'))),
|
||||
'sequence_writes',(SELECT count(*) FROM relations WHERE relkind='S' AND (has_sequence_privilege(oid,'USAGE') OR has_sequence_privilege(oid,'UPDATE'))),
|
||||
'schema_creates',(SELECT count(*) FROM pg_namespace WHERE nspname NOT LIKE 'pg_%' AND nspname<>'information_schema' AND has_schema_privilege(oid,'CREATE')),
|
||||
'database_temp',has_database_privilege(current_database(),'TEMP'),
|
||||
'ssl',COALESCE((SELECT ssl FROM pg_stat_ssl WHERE pid=pg_backend_pid()),false),
|
||||
'missing_columns',(SELECT COALESCE(jsonb_agg(table_name||'.'||column_name),'[]'::jsonb) FROM required r WHERE NOT EXISTS(
|
||||
SELECT 1 FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid JOIN pg_namespace n ON n.oid=c.relnamespace
|
||||
WHERE n.nspname='public' AND c.relname=r.table_name AND a.attname=r.column_name AND a.attnum>0 AND NOT a.attisdropped AND a.atttypid=to_regtype(r.type_name))))::text`
|
||||
|
||||
func AuditSourceSafety(ctx context.Context, q rowQuerier) (SourceSafety, error) {
|
||||
var raw string
|
||||
if err := q.QueryRow(ctx, sourceSafetySQL).Scan(&raw); err != nil {
|
||||
return SourceSafety{}, fmt.Errorf("%w: extended safety probe failed", ErrUnavailable)
|
||||
}
|
||||
var report SourceSafety
|
||||
if err := json.Unmarshal([]byte(raw), &report); err != nil || report.Database == "" || report.TablesChecked < 4 || report.ConnectionLimit <= 0 {
|
||||
return SourceSafety{}, fmt.Errorf("%w: incomplete safety report or unlimited source connections", ErrNotReadOnly)
|
||||
}
|
||||
if report.TableWrites > 0 || report.ColumnWrites > 0 || report.SequenceWrites > 0 || report.SchemaCreates > 0 {
|
||||
return report, fmt.Errorf("%w: effective write or maintenance privileges detected", ErrNotReadOnly)
|
||||
}
|
||||
if len(report.MissingColumns) > 0 {
|
||||
return report, fmt.Errorf("%w: source schema contract mismatch", ErrUnavailable)
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSourceRechecksEffectivePrivilegesBeforeEveryQuery(t *testing.T) {
|
||||
pool := fixturePool(t)
|
||||
roDSN := buildFixture(t, pool)
|
||||
client := buildReadOnlyClient(t, roDSN)
|
||||
execFixture(t, pool, `CREATE TABLE acl_other_surface(id text)`, `GRANT INSERT ON acl_other_surface TO kg_ro_fixture`)
|
||||
t.Cleanup(func() { execFixture(t, pool, `DROP TABLE IF EXISTS acl_other_surface`) })
|
||||
window := Window{TenantKey: "default", Start: time.Date(2026, 10, 14, 0, 0, 0, 0, time.UTC), End: time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC)}
|
||||
if _, err := client.QueryCounts(context.Background(), window, Filter{}); !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("unexpected privilege escalation outside legacy tables was not blocked: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestExtendedAuditRejectsWritesOutsideTheFourLegacyTables(t *testing.T) {
|
||||
for _, key := range []string{"table_writes", "column_writes", "sequence_writes", "schema_creates"} {
|
||||
report := map[string]any{"database": "source_test", "connection_limit": 4, "tables_checked": 19, "table_writes": 0, "column_writes": 0, "sequence_writes": 0, "schema_creates": 0, "database_temp": true, "ssl": false, "missing_columns": []string{}}
|
||||
report[key] = 1
|
||||
raw, _ := json.Marshal(report)
|
||||
_, err := AuditSourceSafety(context.Background(), fakeQuerier{row: fakeRow{values: []any{string(raw)}}})
|
||||
if !errors.Is(err, ErrNotReadOnly) {
|
||||
t.Fatalf("%s must block source: %v", key, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtendedAuditReportsTransportTempAndSchemaContract(t *testing.T) {
|
||||
raw := `{"database":"source_test","connection_limit":4,"tables_checked":19,"table_writes":0,"column_writes":0,"sequence_writes":0,"schema_creates":0,"database_temp":true,"ssl":false,"missing_columns":[]}`
|
||||
report, err := AuditSourceSafety(context.Background(), fakeQuerier{row: fakeRow{values: []any{raw}}})
|
||||
if err != nil || !report.DatabaseTemp || report.SSL || report.ConnectionLimit != 4 {
|
||||
t.Fatalf("audit missing security disclosures: %+v %v", report, err)
|
||||
}
|
||||
raw = `{"database":"source_test","connection_limit":4,"tables_checked":19,"missing_columns":["turns.question"]}`
|
||||
if _, err := AuditSourceSafety(context.Background(), fakeQuerier{row: fakeRow{values: []any{raw}}}); err == nil {
|
||||
t.Fatal("missing schema contract must block source")
|
||||
}
|
||||
for _, raw := range []string{"{}", "not json"} {
|
||||
if _, err := AuditSourceSafety(context.Background(), fakeQuerier{row: fakeRow{values: []any{raw}}}); err == nil {
|
||||
t.Fatal("incomplete audit must not pass")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Settings contains only backend configuration. A source error must not stop
|
||||
// the governance main database or its unrelated workflows from starting.
|
||||
type Settings struct {
|
||||
Enabled bool
|
||||
TenantKey string
|
||||
Connection Config
|
||||
Problem string
|
||||
}
|
||||
|
||||
func LoadSettings(getenv func(string) string, governanceDSN string) Settings {
|
||||
s := Settings{}
|
||||
raw := strings.TrimSpace(getenv("DOBOTAI_OPS_READ_ENABLED"))
|
||||
if raw == "" {
|
||||
return s
|
||||
}
|
||||
enabled, err := strconv.ParseBool(raw)
|
||||
if err != nil {
|
||||
s.Problem = "DOBOTAI_OPS_READ_ENABLED is invalid"
|
||||
return s
|
||||
}
|
||||
if !enabled {
|
||||
return s
|
||||
}
|
||||
s.Enabled = true
|
||||
s.TenantKey = strings.TrimSpace(getenv("DOBOTAI_OPS_TENANT_KEY"))
|
||||
c := Config{DSNFile: strings.TrimSpace(getenv("DOBOTAI_OPS_READ_DSN_FILE")), GovernanceDatabaseURL: governanceDSN,
|
||||
DialMode: strings.TrimSpace(getenv("DOBOTAI_OPS_DIAL_MODE")), SSHTarget: strings.TrimSpace(getenv("DOBOTAI_OPS_SSH_TARGET")),
|
||||
ConnectTimeout: 5 * time.Second, QueryTimeout: 4 * time.Second, MaxConns: 2, MinConns: 0}
|
||||
if c.DialMode == "" {
|
||||
c.DialMode = "direct"
|
||||
}
|
||||
for _, item := range []struct {
|
||||
key string
|
||||
target *time.Duration
|
||||
}{
|
||||
{"DOBOTAI_OPS_CONNECT_TIMEOUT", &c.ConnectTimeout}, {"DOBOTAI_OPS_QUERY_TIMEOUT", &c.QueryTimeout},
|
||||
} {
|
||||
if raw := getenv(item.key); raw != "" {
|
||||
value, err := time.ParseDuration(raw)
|
||||
if err != nil {
|
||||
s.Problem = item.key + " is invalid"
|
||||
return s
|
||||
}
|
||||
*item.target = value
|
||||
}
|
||||
}
|
||||
for _, item := range []struct {
|
||||
key string
|
||||
target *int32
|
||||
}{
|
||||
{"DOBOTAI_OPS_MAX_CONNS", &c.MaxConns}, {"DOBOTAI_OPS_MIN_CONNS", &c.MinConns},
|
||||
} {
|
||||
if raw := getenv(item.key); raw != "" {
|
||||
value, err := strconv.ParseInt(raw, 10, 32)
|
||||
if err != nil {
|
||||
s.Problem = item.key + " is invalid"
|
||||
return s
|
||||
}
|
||||
*item.target = int32(value)
|
||||
}
|
||||
}
|
||||
if s.TenantKey == "" || len(s.TenantKey) > 128 {
|
||||
s.Problem = "DOBOTAI_OPS_TENANT_KEY is required and must be at most 128 bytes"
|
||||
return s
|
||||
}
|
||||
if !filepath.IsAbs(c.DSNFile) || filepath.Clean(c.DSNFile) != c.DSNFile {
|
||||
s.Problem = "DOBOTAI_OPS_READ_DSN_FILE must be an absolute clean path"
|
||||
return s
|
||||
}
|
||||
if err := c.Validate(); err != nil {
|
||||
s.Problem = "external read configuration is invalid"
|
||||
return s
|
||||
}
|
||||
s.Connection = c.withDefaults()
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLoadSettingsConsumesIndependentConnectionLimits(t *testing.T) {
|
||||
env := map[string]string{
|
||||
"DOBOTAI_OPS_READ_ENABLED": "true", "DOBOTAI_OPS_READ_DSN_FILE": "/run/secrets/source",
|
||||
"DOBOTAI_OPS_TENANT_KEY": "approved-tenant", "DOBOTAI_OPS_DIAL_MODE": "ssh", "DOBOTAI_OPS_SSH_TARGET": "approved-hop",
|
||||
"DOBOTAI_OPS_CONNECT_TIMEOUT": "7s", "DOBOTAI_OPS_QUERY_TIMEOUT": "3s",
|
||||
"DOBOTAI_OPS_MAX_CONNS": "3", "DOBOTAI_OPS_MIN_CONNS": "1",
|
||||
}
|
||||
s := LoadSettings(func(k string) string { return env[k] }, "postgres://governance@elsewhere/governance")
|
||||
if !s.Enabled || s.Problem != "" || s.TenantKey != "approved-tenant" {
|
||||
t.Fatalf("settings were not loaded: %+v", s)
|
||||
}
|
||||
if s.Connection.ConnectTimeout != 7*time.Second || s.Connection.QueryTimeout != 3*time.Second || s.Connection.MaxConns != 3 || s.Connection.MinConns != 1 {
|
||||
t.Fatalf("environment overrides ignored: %+v", s.Connection)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadSettingsIsDisabledByDefaultAndContainsInvalidSourceConfiguration(t *testing.T) {
|
||||
if s := LoadSettings(func(string) string { return "" }, ""); s.Enabled || s.Problem != "" {
|
||||
t.Fatalf("not disabled by default: %+v", s)
|
||||
}
|
||||
for _, key := range []string{"DOBOTAI_OPS_READ_ENABLED", "DOBOTAI_OPS_CONNECT_TIMEOUT", "DOBOTAI_OPS_QUERY_TIMEOUT", "DOBOTAI_OPS_MAX_CONNS", "DOBOTAI_OPS_MIN_CONNS"} {
|
||||
env := map[string]string{"DOBOTAI_OPS_READ_ENABLED": "true", "DOBOTAI_OPS_READ_DSN_FILE": "/run/secrets/source", "DOBOTAI_OPS_TENANT_KEY": "approved-tenant"}
|
||||
env[key] = "invalid-sensitive-value"
|
||||
s := LoadSettings(func(k string) string { return env[k] }, "postgres://governance@elsewhere/governance")
|
||||
if s.Problem == "" || strings.Contains(s.Problem, "invalid-sensitive-value") {
|
||||
t.Fatalf("%s must fail closed without echoing its value: %+v", key, s)
|
||||
}
|
||||
}
|
||||
s := LoadSettings(func(k string) string {
|
||||
if k == "DOBOTAI_OPS_READ_ENABLED" {
|
||||
return "true"
|
||||
}
|
||||
return ""
|
||||
}, "")
|
||||
if s.Problem == "" {
|
||||
t.Fatal("missing secret and tenant must block source access")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSecretReaderDoesNotSelectAnUnrelatedDatabaseURL(t *testing.T) {
|
||||
unrelated := testDSN("writer", "not"+"a-secret", "other", "other_db")
|
||||
wanted := testDSN("reader", "fixture"+"only", "source", "source_db")
|
||||
got, ok := parseDSNLine("DATABASE_URL=" + unrelated + "\nDOBOTAI_OPS_READ_DSN=" + wanted)
|
||||
if !ok || got != wanted {
|
||||
t.Fatal("must select only the dedicated read DSN key")
|
||||
}
|
||||
if _, ok := parseDSNLine("DATABASE_URL=" + unrelated); ok {
|
||||
t.Fatal("must not borrow another application's credential")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMalformedGovernanceDSNDoesNotDisableIdentityGuard(t *testing.T) {
|
||||
err := GuardGovernanceDatabase("postgres://reader@source/dobotai_prod", "invalid-main-dsn")
|
||||
if !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatalf("malformed main identity must fail closed: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// SSHDialer 复用既有的 SSH direct-tcpip 能力：`ssh -W <host>:<port> <target>` ——
|
||||
// 由 SSH 侧完成到目标库的 TCP 连接，本机**不新增监听、不改 SSH/NPS/systemd/Docker 网络**。
|
||||
//
|
||||
// 实现用 socketpair：Go 侧拿到的是 fd 支撑的真实 net.Conn（支持 SetDeadline，
|
||||
// pgx 的连接/查询超时才能在拨号阶段生效），ssh 子进程的 stdin/stdout 指向同一对端的另一端。
|
||||
//
|
||||
// context 语义（刻意区分两段）：
|
||||
// - **拨号阶段**尊重 ctx：ctx 已取消就不再创建连接、不再启动子进程；
|
||||
// - **连接建立之后不绑定这个 ctx** —— pgx 在连接成功后就会取消拨号 ctx，
|
||||
// 若把连接生命周期挂上去，健康连接会被立刻掐断。连接生命周期绑定在
|
||||
// "连接关闭 ⇄ 子进程回收" 这对关系上（见 tunnelConn.Close）。
|
||||
//
|
||||
// 依赖：运行环境里必须有 ssh 客户端与已批准的密钥/known_hosts。容器内 localhost 不等于
|
||||
// 宿主机 —— 容器没有 ssh 能力时返回 ErrUnavailable；**不擅自复制私钥、不放宽 known_hosts、
|
||||
// 不改 network_mode**。
|
||||
// dialAddress 是**配置里 DSN 的主机:端口**。为什么必须用它、而不是 pgx 传进来的 addr：
|
||||
// pgx 会先在**本机**解析域名，本机若开了 Clash/mihomo 之类的 Fake-IP（198.18.0.0/15）或
|
||||
// 其他 DNS 劫持，解析结果是假地址；把假地址交给 ssh 去连必然挂住（实测：20s 超时）。
|
||||
// 固定审批的隧道应当让**远端**解析并连接配置里的主机名 —— 这也是"不接受任意主机"的同一原则。
|
||||
func SSHDialer(target, dialAddress string) (func(ctx context.Context, network, addr string) (net.Conn, error), error) {
|
||||
if target == "" {
|
||||
return nil, fmt.Errorf("%w: ssh target is required", ErrUnavailable)
|
||||
}
|
||||
if strings.TrimSpace(dialAddress) == "" {
|
||||
return nil, fmt.Errorf("%w: dial address is required", ErrUnavailable)
|
||||
}
|
||||
if _, err := exec.LookPath("ssh"); err != nil {
|
||||
return nil, fmt.Errorf("%w: ssh client is not available in this environment", ErrUnavailable)
|
||||
}
|
||||
return func(ctx context.Context, _ string, _ string) (net.Conn, error) {
|
||||
addr := dialAddress
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, fmt.Errorf("%w: ssh dial cancelled before start", ErrUnavailable)
|
||||
}
|
||||
fds, err := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_STREAM, 0)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: cannot create a socketpair for the ssh tunnel", ErrUnavailable)
|
||||
}
|
||||
localEnd := os.NewFile(uintptr(fds[0]), "kg-ssh-tunnel-local")
|
||||
remoteEnd := os.NewFile(uintptr(fds[1]), "kg-ssh-tunnel-child")
|
||||
conn, err := net.FileConn(localEnd)
|
||||
_ = localEnd.Close()
|
||||
if err != nil {
|
||||
_ = remoteEnd.Close()
|
||||
return nil, fmt.Errorf("%w: cannot wrap the ssh tunnel in a connection", ErrUnavailable)
|
||||
}
|
||||
command := exec.Command("ssh",
|
||||
"-o", "BatchMode=yes",
|
||||
"-o", "ExitOnForwardFailure=yes",
|
||||
"-o", "ServerAliveInterval=15",
|
||||
"-W", addr, target)
|
||||
command.Stdin = remoteEnd
|
||||
command.Stdout = remoteEnd
|
||||
// ssh 自身的 stderr 一律丢弃：里面可能带主机名/路径；失败统一上抛 ErrUnavailable。
|
||||
command.Stderr = io.Discard
|
||||
if err := command.Start(); err != nil {
|
||||
_ = remoteEnd.Close()
|
||||
_ = conn.Close()
|
||||
return nil, fmt.Errorf("%w: cannot start the ssh tunnel", ErrUnavailable)
|
||||
}
|
||||
_ = remoteEnd.Close()
|
||||
// reaped 必须先初始化：reap() 会在子进程退出时 close 它（nil channel 会 panic）。
|
||||
tunnel := &tunnelConn{Conn: conn, command: command, reaped: make(chan struct{})}
|
||||
go tunnel.reap()
|
||||
return tunnel, nil
|
||||
}, nil
|
||||
}
|
||||
|
||||
// tunnelConn 把"连接关闭 ⇄ 子进程回收"显式绑定起来：
|
||||
// - 连接被关（pgx 归还/释放连接、或对端断开）⇒ 子进程收到终止信号并回收，不留孤儿；
|
||||
// - 子进程自己退出（隧道断/认证失败）⇒ Go 侧连接立刻关掉，让 pgx 马上感知断开；
|
||||
// - Close 幂等，且不阻塞在等待子进程上（最多等一个短宽限期）。
|
||||
type tunnelConn struct {
|
||||
net.Conn
|
||||
command *exec.Cmd
|
||||
closeOnce sync.Once
|
||||
reaped chan struct{}
|
||||
}
|
||||
|
||||
func (t *tunnelConn) reap() {
|
||||
if t.command != nil {
|
||||
_ = t.command.Wait()
|
||||
}
|
||||
if t.reaped != nil {
|
||||
close(t.reaped)
|
||||
}
|
||||
// 子进程退出 ⇒ 连接必然无用，主动关掉让调用方立刻看到 EOF。
|
||||
_ = t.Conn.Close()
|
||||
}
|
||||
|
||||
func (t *tunnelConn) Close() error {
|
||||
var err error
|
||||
t.closeOnce.Do(func() {
|
||||
err = t.Conn.Close()
|
||||
// 连接关闭后 ssh 通常自行退出；这里补一刀保证它不会挂着不退。
|
||||
if t.command.Process != nil {
|
||||
_ = t.command.Process.Signal(syscall.SIGTERM)
|
||||
}
|
||||
select {
|
||||
case <-t.Exited():
|
||||
case <-time.After(2 * time.Second):
|
||||
if t.command.Process != nil {
|
||||
_ = t.command.Process.Kill()
|
||||
}
|
||||
}
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// Exited 让测试/运维能确认子进程已经回收（不暴露任何连接信息）。
|
||||
func (t *tunnelConn) Exited() <-chan struct{} {
|
||||
if t.reaped == nil {
|
||||
closed := make(chan struct{})
|
||||
close(closed)
|
||||
return closed
|
||||
}
|
||||
return t.reaped
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 只有在显式给出"已批准的 SSH 目标 + 一个可达的只读端点"时才跑：CI 默认跳过。
|
||||
// 目的：证明 `ssh -W` + socketpair 的拨号在 Go 侧是一个**支持 deadline 的真实 net.Conn**
|
||||
// （pgx 的连接/查询超时能在拨号阶段生效），且经这条通道发出的请求真的能到达目标。
|
||||
//
|
||||
// KG_SSH_DIAL_TARGET=weknora-prod KG_SSH_DIAL_ADDR=127.0.0.1:8090 // go test ./internal/dobotaiops/ -run TestSSHDialerTransportsRealTraffic -v
|
||||
func TestSSHDialerTransportsRealTraffic(t *testing.T) {
|
||||
target := strings.TrimSpace(os.Getenv("KG_SSH_DIAL_TARGET"))
|
||||
address := strings.TrimSpace(os.Getenv("KG_SSH_DIAL_ADDR"))
|
||||
if target == "" || address == "" {
|
||||
t.Skip("KG_SSH_DIAL_TARGET / KG_SSH_DIAL_ADDR 未设置：跳过（CI 默认不跑）")
|
||||
}
|
||||
dialer, err := SSHDialer(target, address)
|
||||
if err != nil {
|
||||
t.Fatalf("SSHDialer() error = %v", err)
|
||||
}
|
||||
conn, err := dialer(context.Background(), "tcp", address)
|
||||
if err != nil {
|
||||
t.Fatalf("dial error = %v", err)
|
||||
}
|
||||
defer conn.Close()
|
||||
// 真实 deadline 必须可用（否则 pgx 的连接超时形同虚设）。
|
||||
if err := conn.SetDeadline(time.Now().Add(10 * time.Second)); err != nil {
|
||||
t.Fatalf("socketpair 连接必须支持 deadline：%v", err)
|
||||
}
|
||||
if _, err := fmt.Fprintf(conn, "GET /version HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n", address); err != nil {
|
||||
t.Fatalf("write error = %v", err)
|
||||
}
|
||||
response, err := http.ReadResponse(bufio.NewReader(conn), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("read error = %v", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d", response.StatusCode)
|
||||
}
|
||||
buf := make([]byte, 256)
|
||||
n, _ := response.Body.Read(buf)
|
||||
if !strings.Contains(string(buf[:n]), "commit_sha") {
|
||||
t.Fatalf("经 SSH 通道取到的响应不像版本接口：%q", string(buf[:n]))
|
||||
}
|
||||
var _ net.Conn = conn
|
||||
}
|
||||
|
||||
// 取消的 context 不得创建连接、不得启动子进程（独立探测曾证明旧实现照样建连）。
|
||||
func TestSSHDialerRespectsCancelledContext(t *testing.T) {
|
||||
dialer, err := SSHDialer("definitely-not-a-real-ssh-alias", "127.0.0.1:5432")
|
||||
if err != nil {
|
||||
t.Fatalf("SSHDialer() error = %v", err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
conn, err := dialer(ctx, "tcp", "127.0.0.1:1")
|
||||
if err == nil || conn != nil {
|
||||
t.Fatalf("ctx 已取消时不得建连：conn=%v err=%v", conn, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "cancelled") {
|
||||
t.Fatalf("应明确说明是取消：%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// 关闭连接必须回收子进程（不阻塞、幂等）；顺带证明 GetDeadline/SetDeadline 是真实能力。
|
||||
func TestSSHTunnelCloseReapsChild(t *testing.T) {
|
||||
target := strings.TrimSpace(os.Getenv("KG_SSH_DIAL_TARGET"))
|
||||
if target == "" {
|
||||
t.Skip("KG_SSH_DIAL_TARGET 未设置：跳过（需要有已批准目标才能起 ssh 子进程）")
|
||||
}
|
||||
dialer, err := SSHDialer(target, "127.0.0.1:1") // 目标端口不通也没关系：本轮只看生命周期
|
||||
if err != nil {
|
||||
t.Fatalf("SSHDialer() error = %v", err)
|
||||
}
|
||||
conn, err := dialer(context.Background(), "tcp", "127.0.0.1:1")
|
||||
if err != nil {
|
||||
t.Skipf("无法在本机建立隧道（%v）—— 该用例只在有 ssh 能力的环境有意义", err)
|
||||
}
|
||||
tunnel, ok := conn.(*tunnelConn)
|
||||
if !ok {
|
||||
t.Fatalf("返回值应绑定子进程生命周期：%T", conn)
|
||||
}
|
||||
if err := tunnel.Close(); err != nil {
|
||||
t.Fatalf("Close() error = %v", err)
|
||||
}
|
||||
if err := tunnel.Close(); err != nil {
|
||||
t.Fatalf("Close() 必须幂等：%v", err)
|
||||
}
|
||||
select {
|
||||
case <-tunnel.Exited():
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("关闭连接后 ssh 子进程没有被回收")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package dobotaiops
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Window 是一次只读统计/列举的时间范围，语义固定为**半开区间**：
|
||||
// created_at >= Start AND created_at < End（工单要求）。
|
||||
type Window struct {
|
||||
TenantKey string
|
||||
Start time.Time
|
||||
End time.Time
|
||||
}
|
||||
|
||||
// NewWindow 解析并校验调用方给的租户与时间范围：
|
||||
// - 租户非空、长度受限（不做任何 SQL 拼接，只作为绑定参数）；
|
||||
// - 时间是 RFC3339（允许带时区），必须有 start < end；
|
||||
// - 窗口跨度不超过上限（默认 90 天）、不允许跑到未来太远。
|
||||
func NewWindow(tenantKey, startRaw, endRaw string, limit time.Duration, now time.Time) (Window, error) {
|
||||
tenantKey = strings.TrimSpace(tenantKey)
|
||||
if tenantKey == "" || len(tenantKey) > 128 {
|
||||
return Window{}, fmt.Errorf("%w: tenant key is required", ErrInvalidQuery)
|
||||
}
|
||||
start, err := parseInstant(startRaw)
|
||||
if err != nil {
|
||||
return Window{}, err
|
||||
}
|
||||
end, err := parseInstant(endRaw)
|
||||
if err != nil {
|
||||
return Window{}, err
|
||||
}
|
||||
if !start.Before(end) {
|
||||
return Window{}, fmt.Errorf("%w: start must be before end", ErrInvalidQuery)
|
||||
}
|
||||
if limit <= 0 {
|
||||
limit = 90 * 24 * time.Hour
|
||||
}
|
||||
if end.Sub(start) > limit {
|
||||
return Window{}, fmt.Errorf("%w: window exceeds the %s limit", ErrInvalidQuery, limit)
|
||||
}
|
||||
if end.After(now.Add(5 * time.Minute)) {
|
||||
return Window{}, fmt.Errorf("%w: end is in the future", ErrInvalidQuery)
|
||||
}
|
||||
return Window{TenantKey: tenantKey, Start: start.UTC(), End: end.UTC()}, nil
|
||||
}
|
||||
|
||||
func parseInstant(raw string) (time.Time, error) {
|
||||
raw = strings.TrimSpace(raw)
|
||||
if raw == "" {
|
||||
return time.Time{}, fmt.Errorf("%w: time is required", ErrInvalidQuery)
|
||||
}
|
||||
parsed, err := time.Parse(time.RFC3339, raw)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("%w: time must be RFC3339 (with offset)", ErrInvalidQuery)
|
||||
}
|
||||
return parsed.UTC(), nil
|
||||
}
|
||||
|
||||
// Cursor 是 (created_at, id) 复合游标：与工单要求一致 —— 稳定、确定性、翻页不漏不重。
|
||||
type Cursor struct {
|
||||
CreatedAt time.Time
|
||||
ID string
|
||||
}
|
||||
|
||||
// EncodeCursor / DecodeCursor 只做无损编解码，不做业务判断；非法游标一律报 ErrInvalidQuery。
|
||||
func EncodeCursor(cursor Cursor) string {
|
||||
return base64.RawURLEncoding.EncodeToString([]byte(cursor.CreatedAt.UTC().Format(time.RFC3339Nano) + "|" + cursor.ID))
|
||||
}
|
||||
|
||||
func DecodeCursor(raw string) (Cursor, error) {
|
||||
decoded, err := base64.RawURLEncoding.DecodeString(strings.TrimSpace(raw))
|
||||
if err != nil {
|
||||
return Cursor{}, fmt.Errorf("%w: cursor is not valid", ErrInvalidQuery)
|
||||
}
|
||||
parts := strings.SplitN(string(decoded), "|", 2)
|
||||
if len(parts) != 2 || parts[1] == "" {
|
||||
return Cursor{}, fmt.Errorf("%w: cursor is not valid", ErrInvalidQuery)
|
||||
}
|
||||
createdAt, err := time.Parse(time.RFC3339Nano, parts[0])
|
||||
if err != nil {
|
||||
return Cursor{}, fmt.Errorf("%w: cursor is not valid", ErrInvalidQuery)
|
||||
}
|
||||
return Cursor{CreatedAt: createdAt.UTC(), ID: parts[1]}, nil
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package governance
|
||||
|
||||
import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Development headings cannot replace the formal release baseline.
|
||||
func ExtractDevelopmentNotes(changelog, version string) string {
|
||||
return developmentNotes(changelog, version, "")
|
||||
}
|
||||
func ExtractDevelopmentNotesForCommit(changelog, version, sha string) string {
|
||||
return developmentNotes(changelog, version, sha)
|
||||
}
|
||||
func developmentNotes(changelog, version, sha string) string {
|
||||
if !SemverValid(version) {
|
||||
return ""
|
||||
}
|
||||
lines := strings.Split(changelog, "\n")
|
||||
prefix := "## 开发快照 snapshot/v" + version + " - "
|
||||
for i, line := range lines {
|
||||
if strings.HasPrefix(line, prefix) {
|
||||
end := len(lines)
|
||||
for j := i + 1; j < len(lines); j++ {
|
||||
if strings.HasPrefix(lines[j], "## ") {
|
||||
end = j
|
||||
break
|
||||
}
|
||||
}
|
||||
body := strings.TrimSpace(strings.Join(lines[i+1:end], "\n"))
|
||||
if sha == "" {
|
||||
return body
|
||||
}
|
||||
for _, field := range lines[i+1 : end] {
|
||||
if strings.TrimSpace(field) == "目标提交："+sha {
|
||||
return body
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -101,7 +101,7 @@ func isAuthEndpoint(method, path string) bool {
|
||||
func classifyRole(method, path string) string {
|
||||
// The member roster exposes account identities and roles: admin-only for
|
||||
// every method, checked before the read shortcut below.
|
||||
if strings.HasPrefix(path, "/api/v1/users") {
|
||||
if strings.HasPrefix(path, "/api/v1/users") || strings.HasPrefix(path, "/api/v1/qa-ops/") {
|
||||
return roleAdmin
|
||||
}
|
||||
if method == http.MethodGet {
|
||||
|
||||
@@ -77,7 +77,7 @@ func TestOpenAPICatalogMutationsRequireIdempotencyKeyHeader(t *testing.T) {
|
||||
if !ok || schema["type"] != "string" || schema["minLength"] != float64(1) || schema["maxLength"] != float64(128) {
|
||||
t.Fatalf("IdempotencyKey header schema is unsafe: %#v", parameter["schema"])
|
||||
}
|
||||
for _, path := range []string{"/api/v1/source-catalog/route", "/api/v1/source-catalog/sources/{id}/refresh"} {
|
||||
for _, path := range []string{"/api/v1/qa-ops/organization/sync", "/api/v1/source-catalog/route", "/api/v1/source-catalog/sources/{id}/refresh"} {
|
||||
operation, ok := document.Paths[path]["post"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("missing POST %s", path)
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
var errQAOrganization = errors.New("organization attribution unavailable")
|
||||
|
||||
type qaAttributionProvider interface {
|
||||
GetAttribution(context.Context) (*qaopsorg.Attribution, error)
|
||||
}
|
||||
|
||||
func recordAttribution(ctx context.Context, config Config) (*qaopsorg.Attribution, error) {
|
||||
if config.QAOrg == nil {
|
||||
return qaopsorg.BuildAttribution(qaopsorg.State{State: "empty"}, qaopsorg.Directory{})
|
||||
}
|
||||
provider, ok := config.QAOrg.(qaAttributionProvider)
|
||||
if !ok {
|
||||
return nil, errQAOrganization
|
||||
}
|
||||
p, err := provider.GetAttribution(ctx)
|
||||
if err != nil || p == nil {
|
||||
return nil, errQAOrganization
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func attributionVersion(p *qaopsorg.Attribution) dobotaiops.OrganizationVersion {
|
||||
return dobotaiops.OrganizationVersion{State: p.Metadata.State, DirectoryID: p.Metadata.DirectoryID, DirectorySource: p.Metadata.DirectorySource, SyncedAt: p.Metadata.DirectorySyncedAt, RevisionID: p.Metadata.RevisionID, Basis: "current_directory"}
|
||||
}
|
||||
func enrichQARecord(item *dobotaiops.Record, p *qaopsorg.Attribution) {
|
||||
person := p.Resolve(item.Identity.StaffID, item.Identity.UnionID)
|
||||
o := &dobotaiops.RecordOrganization{Version: attributionVersion(p), State: person.State, Departments: person.Departments, Region: person.Region, SourceDepartment: item.Department, SourceRegion: item.Region, DepartmentOrigin: "unavailable", RegionOrigin: "unavailable"}
|
||||
if item.Department != "" {
|
||||
o.DepartmentOrigin = "source_snapshot"
|
||||
} else if len(person.Departments) > 0 {
|
||||
item.Department = strings.Join(person.Departments, "；")
|
||||
o.DepartmentOrigin = "current_directory"
|
||||
}
|
||||
if item.Region != "" {
|
||||
o.RegionOrigin = "source_snapshot"
|
||||
} else if person.Region != "" {
|
||||
item.Region = person.Region
|
||||
o.RegionOrigin = "current_directory"
|
||||
}
|
||||
item.Organization = o
|
||||
}
|
||||
func enrichQAPage(page *dobotaiops.RecordPage, p *qaopsorg.Attribution) {
|
||||
version := attributionVersion(p)
|
||||
page.Organization = &version
|
||||
for i := range page.Items {
|
||||
enrichQARecord(&page.Items[i], p)
|
||||
}
|
||||
for _, actor := range page.ScopeActors {
|
||||
person := p.Resolve(actor.StaffID, actor.UnionID)
|
||||
if len(person.Departments) > 0 {
|
||||
dobotaiops.AddRecordColumns(page, "department")
|
||||
}
|
||||
if person.Region != "" {
|
||||
dobotaiops.AddRecordColumns(page, "region")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type qaUsageReader interface {
|
||||
ReadUsageGroups(context.Context, dobotaiops.RecordQuery) (dobotaiops.UsageGroups, error)
|
||||
}
|
||||
type qaUsageRow struct {
|
||||
Key string `json:"key"`
|
||||
Questions int64 `json:"questions"`
|
||||
Succeeded int64 `json:"succeeded"`
|
||||
Failed int64 `json:"failed"`
|
||||
Other int64 `json:"other"`
|
||||
Users *int `json:"users"`
|
||||
RosterPeople *int `json:"roster_people"`
|
||||
}
|
||||
type qaUsage struct {
|
||||
Organization dobotaiops.OrganizationVersion `json:"organization"`
|
||||
AsOf time.Time `json:"as_of"`
|
||||
Start time.Time `json:"start"`
|
||||
End time.Time `json:"end"`
|
||||
Keyword string `json:"keyword"`
|
||||
Questions int64 `json:"questions"`
|
||||
IdentifiedUsers int `json:"identified_users"`
|
||||
Rows []qaUsageRow `json:"rows"`
|
||||
}
|
||||
|
||||
func summarizeQAUsage(groups dobotaiops.UsageGroups, p *qaopsorg.Attribution, query dobotaiops.RecordQuery) qaUsage {
|
||||
result := qaUsage{Organization: attributionVersion(p), AsOf: groups.AsOf, Start: query.Window.Start, End: query.Window.End, Keyword: query.Keyword, Rows: []qaUsageRow{}}
|
||||
keys := append(append([]string{}, qaopsorg.RegionNames...), "unassigned", "cross_region", "unmatched", "identity_conflict", "no_identity", "directory_missing")
|
||||
rows := map[string]*qaUsageRow{}
|
||||
members := map[string]map[string]bool{}
|
||||
all := map[string]bool{}
|
||||
for _, key := range keys {
|
||||
row := &qaUsageRow{Key: key}
|
||||
if p.Metadata.State == "ready" {
|
||||
var n int
|
||||
valid := true
|
||||
switch key {
|
||||
case "unassigned":
|
||||
n = p.Metadata.View.UnassignedPeople
|
||||
case "cross_region":
|
||||
n = p.Metadata.View.ConflictingPeople
|
||||
default:
|
||||
var ok bool
|
||||
n, ok = p.Metadata.View.Regions[key]
|
||||
valid = ok
|
||||
}
|
||||
if valid {
|
||||
row.RosterPeople = &n
|
||||
row.Users = new(int)
|
||||
}
|
||||
}
|
||||
rows[key] = row
|
||||
members[key] = map[string]bool{}
|
||||
}
|
||||
for _, group := range groups.Groups {
|
||||
person := p.Resolve(group.Identity.StaffID, group.Identity.UnionID)
|
||||
key := person.State
|
||||
if person.Region != "" {
|
||||
key = person.Region
|
||||
}
|
||||
row := rows[key]
|
||||
if row == nil {
|
||||
continue
|
||||
}
|
||||
row.Questions += group.Total
|
||||
row.Succeeded += group.Succeeded
|
||||
row.Failed += group.Failed
|
||||
row.Other += group.Total - group.Succeeded - group.Failed
|
||||
result.Questions += group.Total
|
||||
if person.PersonID != "" {
|
||||
members[key][person.PersonID] = true
|
||||
all[person.PersonID] = true
|
||||
}
|
||||
}
|
||||
for _, key := range keys {
|
||||
row := rows[key]
|
||||
if row.Users != nil {
|
||||
*row.Users = len(members[key])
|
||||
}
|
||||
result.Rows = append(result.Rows, *row)
|
||||
}
|
||||
result.IdentifiedUsers = len(all)
|
||||
return result
|
||||
}
|
||||
func qaOpsUsage(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
reader, ok := config.QAOps.(qaUsageReader)
|
||||
if !ok || config.QAOpsTenantKey == "" {
|
||||
qaOpsError(w, dobotaiops.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
values := r.URL.Query()
|
||||
for key, items := range values {
|
||||
if (key != "start" && key != "end" && key != "keyword") || len(items) != 1 {
|
||||
qaOpsError(w, dobotaiops.ErrInvalidQuery)
|
||||
return
|
||||
}
|
||||
}
|
||||
window, err := dobotaiops.NewWindow(config.QAOpsTenantKey, values.Get("start"), values.Get("end"), 90*24*time.Hour, config.Now())
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
p, err := recordAttribution(r.Context(), config)
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
query := dobotaiops.RecordQuery{Window: window, Keyword: values.Get("keyword")}
|
||||
groups, err := reader.ReadUsageGroups(r.Context(), query)
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": summarizeQAUsage(groups, p, query)})
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type attributionOrgSpy struct {
|
||||
orgSpy
|
||||
projection *qaopsorg.Attribution
|
||||
}
|
||||
|
||||
func (s *attributionOrgSpy) GetAttribution(context.Context) (*qaopsorg.Attribution, error) {
|
||||
return s.projection, s.err
|
||||
}
|
||||
|
||||
type attributedReader struct{}
|
||||
|
||||
func (*attributedReader) Available(context.Context) error { return nil }
|
||||
func (*attributedReader) Safety() dobotaiops.SourceSafety { return dobotaiops.SourceSafety{SSL: true} }
|
||||
func (*attributedReader) ListRecords(context.Context, dobotaiops.RecordQuery) (dobotaiops.RecordPage, error) {
|
||||
return dobotaiops.RecordPage{Items: []dobotaiops.Record{{ID: "anonymous"}}, ScopeActors: []dobotaiops.ActorIdentity{{StaffID: "fixture-private-staff"}}, Columns: []string{"question"}, Total: 3}, nil
|
||||
}
|
||||
func (*attributedReader) GetRecord(context.Context, string, string) (dobotaiops.Record, error) {
|
||||
return dobotaiops.Record{ID: "one", Department: "历史部门", Region: "历史区域", Identity: dobotaiops.ActorIdentity{StaffID: "fixture-private-staff"}}, nil
|
||||
}
|
||||
func (*attributedReader) ReadUsageGroups(context.Context, dobotaiops.RecordQuery) (dobotaiops.UsageGroups, error) {
|
||||
return dobotaiops.UsageGroups{AsOf: time.Date(2026, 10, 15, 0, 0, 0, 0, time.UTC), Groups: []dobotaiops.UsageGroup{{Identity: dobotaiops.ActorIdentity{StaffID: "fixture-private-staff"}, Total: 3, Succeeded: 2, Failed: 1}, {Identity: dobotaiops.ActorIdentity{UnionID: "fixture-private-union"}, Total: 2, Succeeded: 2}, {Identity: dobotaiops.ActorIdentity{StaffID: "unmatched"}, Total: 2}}}, nil
|
||||
}
|
||||
func attributionHandler(t *testing.T) (http.Handler, *attributionOrgSpy) {
|
||||
t.Helper()
|
||||
projection, err := qaopsorg.BuildAttribution(qaopsorg.State{State: "ready", DirectoryID: "directory-v1", RevisionID: "region-v1", Rules: []qaopsorg.Rule{{DepartmentID: "root", Region: "APAC"}}}, qaopsorg.Directory{Departments: []qaopsorg.Department{{ID: "root", Name: "测试公司"}}, People: []qaopsorg.Person{{ID: "fixture-private-staff", UnionID: "fixture-private-union", DepartmentIDs: []string{"root"}}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
org := &attributionOrgSpy{projection: projection}
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["attribution-admin"] = "usr-boss"
|
||||
auth.sessions["attribution-viewer"] = "usr-eye"
|
||||
h, err := NewHandler(Config{Store: &fakeJobStore{}, Auth: &AuthConfig{Store: auth}, QAOps: &attributedReader{}, QAOpsTenantKey: "fixed", QAOrg: org, Now: func() time.Time { return time.Date(2026, 10, 17, 0, 0, 0, 0, time.UTC) }})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return h, org
|
||||
}
|
||||
func attributionRequest(h http.Handler, path, token string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: token})
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
const attributionFilter = "?start=2026-10-14T00:00:00Z&end=2026-10-15T00:00:00Z"
|
||||
|
||||
func TestQAAttributionUsesWholeFilterAndPreservesHistoricalSnapshots(t *testing.T) {
|
||||
h, org := attributionHandler(t)
|
||||
w := attributionRequest(h, "/api/v1/qa-ops/records"+attributionFilter, "attribution-admin")
|
||||
if w.Code != 200 {
|
||||
t.Fatal(w.Code, w.Body.String())
|
||||
}
|
||||
var page struct{ Data dobotaiops.RecordPage }
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &page)
|
||||
if !strings.Contains(strings.Join(page.Data.Columns, ","), "department,region") {
|
||||
t.Fatal("organization column hidden because first page is anonymous", w.Body.String())
|
||||
}
|
||||
w = attributionRequest(h, "/api/v1/qa-ops/records/one", "attribution-admin")
|
||||
var detail struct{ Data dobotaiops.Record }
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &detail)
|
||||
if detail.Data.Department != "历史部门" || detail.Data.Region != "历史区域" || detail.Data.Organization == nil || detail.Data.Organization.Region != "APAC" {
|
||||
t.Fatal("source snapshot overwritten or current attribution missing", w.Body.String())
|
||||
}
|
||||
if strings.Contains(w.Body.String(), "fixture-private") {
|
||||
t.Fatal("identity keys exposed")
|
||||
}
|
||||
org.err = qaopsorg.ErrUnavailable
|
||||
w = attributionRequest(h, "/api/v1/qa-ops/records"+attributionFilter, "attribution-admin")
|
||||
if w.Code != 503 || !strings.Contains(w.Body.String(), "organization_unavailable") {
|
||||
t.Fatal("permission/storage failure hidden as empty", w.Body.String())
|
||||
}
|
||||
}
|
||||
func TestQAUsageIsAdminScopedDeduplicatedAndNotPerformance(t *testing.T) {
|
||||
h, _ := attributionHandler(t)
|
||||
path := "/api/v1/qa-ops/usage" + attributionFilter
|
||||
for _, tc := range []struct {
|
||||
token string
|
||||
status int
|
||||
}{{"", 401}, {"attribution-viewer", 403}, {"attribution-admin", 200}} {
|
||||
w := attributionRequest(h, path, tc.token)
|
||||
if w.Code != tc.status {
|
||||
t.Fatalf("usage guard got %d: %s", w.Code, w.Body.String())
|
||||
}
|
||||
if tc.status == 200 {
|
||||
var result struct {
|
||||
Data struct {
|
||||
Questions int64 `json:"questions"`
|
||||
IdentifiedUsers int `json:"identified_users"`
|
||||
Rows []struct {
|
||||
Key string
|
||||
Questions int64
|
||||
Users *int
|
||||
}
|
||||
}
|
||||
}
|
||||
if json.Unmarshal(w.Body.Bytes(), &result) != nil || result.Data.Questions != 7 || result.Data.IdentifiedUsers != 1 {
|
||||
t.Fatal("events/people not deduplicated", w.Body.String())
|
||||
}
|
||||
for _, row := range result.Data.Rows {
|
||||
if row.Key == "APAC" && (row.Questions != 5 || row.Users == nil || *row.Users != 1) {
|
||||
t.Fatal("regional user double-counted")
|
||||
}
|
||||
}
|
||||
if strings.Contains(w.Body.String(), "fixture-private") || strings.Contains(w.Body.String(), "performance") {
|
||||
t.Fatal("private identity or performance verdict leaked")
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, extra := range []string{"&tenant=forged", "&limit=1", "&cursor=forged", "&start=duplicate"} {
|
||||
if w := attributionRequest(h, path+extra, "attribution-admin"); w.Code != 400 {
|
||||
t.Fatal("usage accepted client scope or pagination")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
_ "time/tzdata"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
type monthlyFilter struct {
|
||||
Month, Region, Department, Activity, Search string
|
||||
Start, End, MonthEnd time.Time
|
||||
Page, Limit int
|
||||
}
|
||||
type monthlyPerson struct {
|
||||
Key string `json:"key"`
|
||||
Name string `json:"name"`
|
||||
State string `json:"state"`
|
||||
Region string `json:"region"`
|
||||
Departments []string `json:"departments"`
|
||||
Questions int64 `json:"questions"`
|
||||
Succeeded int64 `json:"succeeded"`
|
||||
Failed int64 `json:"failed"`
|
||||
Other int64 `json:"other"`
|
||||
}
|
||||
type monthlyDepartment struct {
|
||||
ID string `json:"id"`
|
||||
Path string `json:"path"`
|
||||
}
|
||||
type monthlyPeople struct {
|
||||
Organization dobotaiops.OrganizationVersion `json:"organization"`
|
||||
Source string `json:"source"`
|
||||
Timezone string `json:"timezone"`
|
||||
Month string `json:"month"`
|
||||
Start time.Time `json:"start"`
|
||||
End time.Time `json:"end"`
|
||||
MonthEnd time.Time `json:"month_end"`
|
||||
AsOf time.Time `json:"as_of"`
|
||||
Partial bool `json:"partial"`
|
||||
Questions int64 `json:"questions"`
|
||||
LinkedQuestions int64 `json:"linked_questions"`
|
||||
UnresolvedQuestions int64 `json:"unresolved_questions"`
|
||||
RosterPeople int `json:"roster_people"`
|
||||
UsedPeople int `json:"used_people"`
|
||||
ZeroPeople int `json:"zero_people"`
|
||||
MissingNames int `json:"missing_names"`
|
||||
Total int `json:"total"`
|
||||
Page int `json:"page"`
|
||||
Limit int `json:"limit"`
|
||||
HasMore bool `json:"has_more"`
|
||||
Rows []monthlyPerson `json:"rows"`
|
||||
DepartmentOptions []monthlyDepartment `json:"department_options"`
|
||||
}
|
||||
|
||||
func parseMonthlyFilter(values url.Values, now time.Time) (monthlyFilter, error) {
|
||||
f := monthlyFilter{Month: values.Get("month"), Region: values.Get("region"), Department: values.Get("department_id"), Activity: values.Get("activity"), Search: values.Get("search"), Page: 1, Limit: 30}
|
||||
allowed := map[string]bool{"month": true, "cutoff": true, "region": true, "department_id": true, "activity": true, "search": true, "page": true, "limit": true, "directory_id": true, "revision_id": true}
|
||||
for key, v := range values {
|
||||
if !allowed[key] || len(v) != 1 {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
}
|
||||
loc, err := time.LoadLocation("Asia/Shanghai")
|
||||
if err != nil {
|
||||
return f, dobotaiops.ErrUnavailable
|
||||
}
|
||||
f.Start, err = time.ParseInLocation("2006-01", f.Month, loc)
|
||||
if err != nil || len(f.Month) != 7 || f.Start.Year() < 1 || f.Start.Format("2006-01") != f.Month || !f.Start.Before(now) {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
f.MonthEnd = f.Start.AddDate(0, 1, 0)
|
||||
f.End = f.MonthEnd
|
||||
if now.Before(f.End) {
|
||||
f.End = now.In(loc)
|
||||
}
|
||||
if raw := values.Get("cutoff"); raw != "" {
|
||||
cutoff, e := time.Parse(time.RFC3339Nano, raw)
|
||||
if e != nil || !cutoff.After(f.Start) || cutoff.After(f.MonthEnd) || cutoff.After(now) {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
f.End = cutoff.In(loc)
|
||||
}
|
||||
if f.Activity == "" {
|
||||
f.Activity = "all"
|
||||
}
|
||||
if f.Activity != "all" && f.Activity != "used" && f.Activity != "zero" {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
validRegion := f.Region == "" || f.Region == "unassigned" || f.Region == "cross_region"
|
||||
for _, r := range qaopsorg.RegionNames {
|
||||
validRegion = validRegion || f.Region == r
|
||||
}
|
||||
if !validRegion {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
if !utf8.ValidString(f.Search) || utf8.RuneCountInString(f.Search) > 100 || strings.ContainsRune(f.Search, 0) {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
for key, target := range map[string]*int{"page": &f.Page, "limit": &f.Limit} {
|
||||
if raw := values.Get(key); raw != "" {
|
||||
n, e := strconv.Atoi(raw)
|
||||
if e != nil || n < 1 || n > 100000 {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
*target = n
|
||||
}
|
||||
}
|
||||
if f.Limit > 100 || f.Page > 1 && (!values.Has("directory_id") || !values.Has("revision_id")) {
|
||||
return f, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
func monthlyHasDepartment(person qaopsorg.PersonAttribution, department string, parents map[string]string) bool {
|
||||
if department == "" {
|
||||
return true
|
||||
}
|
||||
for _, id := range person.DepartmentIDs {
|
||||
for cur := id; cur != ""; cur = parents[cur] {
|
||||
if cur == department {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
func monthlyResult(groups dobotaiops.UsageGroups, p *qaopsorg.Attribution, f monthlyFilter) monthlyPeople {
|
||||
result := monthlyPeople{Organization: attributionVersion(p), Source: "2.0", Timezone: "Asia/Shanghai", Month: f.Month, Start: f.Start, End: f.End, MonthEnd: f.MonthEnd, AsOf: groups.AsOf, Partial: f.End.Before(f.MonthEnd), Page: f.Page, Limit: f.Limit, Rows: []monthlyPerson{}, DepartmentOptions: []monthlyDepartment{}}
|
||||
totals := map[string]dobotaiops.UsageGroup{}
|
||||
for _, group := range groups.Groups {
|
||||
result.Questions += group.Total
|
||||
person := p.Resolve(group.Identity.StaffID, group.Identity.UnionID)
|
||||
if person.PersonID == "" {
|
||||
result.UnresolvedQuestions += group.Total
|
||||
continue
|
||||
}
|
||||
n := totals[person.PersonID]
|
||||
n.Total += group.Total
|
||||
n.Succeeded += group.Succeeded
|
||||
n.Failed += group.Failed
|
||||
totals[person.PersonID] = n
|
||||
result.LinkedQuestions += group.Total
|
||||
}
|
||||
parents := map[string]string{}
|
||||
for _, d := range p.Metadata.View.Departments {
|
||||
parents[d.ID] = d.ParentID
|
||||
result.DepartmentOptions = append(result.DepartmentOptions, monthlyDepartment{ID: d.ID, Path: d.Path})
|
||||
}
|
||||
rows := []monthlyPerson{}
|
||||
people := p.Members()
|
||||
result.RosterPeople = len(people)
|
||||
search := strings.ToLower(f.Search)
|
||||
for _, person := range people {
|
||||
n := totals[person.PersonID]
|
||||
if n.Total > 0 {
|
||||
result.UsedPeople++
|
||||
} else {
|
||||
result.ZeroPeople++
|
||||
}
|
||||
if person.Name == "" {
|
||||
result.MissingNames++
|
||||
}
|
||||
if f.Region != "" && f.Region != person.Region && f.Region != person.State {
|
||||
continue
|
||||
}
|
||||
if !monthlyHasDepartment(person, f.Department, parents) {
|
||||
continue
|
||||
}
|
||||
if f.Activity == "used" && n.Total == 0 || f.Activity == "zero" && n.Total > 0 {
|
||||
continue
|
||||
}
|
||||
if search != "" && !strings.Contains(strings.ToLower(person.Name), search) && !strings.Contains(strings.ToLower(strings.Join(person.Departments, "；")), search) {
|
||||
continue
|
||||
}
|
||||
// This is a stable row key, not an anonymisation claim or a new user identifier.
|
||||
sum := sha256.Sum256([]byte(p.Metadata.DirectoryID + "\x00" + person.PersonID))
|
||||
rows = append(rows, monthlyPerson{Key: "person-" + hex.EncodeToString(sum[:]), Name: person.Name, State: person.State, Region: person.Region, Departments: person.Departments, Questions: n.Total, Succeeded: n.Succeeded, Failed: n.Failed, Other: n.Total - n.Succeeded - n.Failed})
|
||||
}
|
||||
sort.Slice(rows, func(i, j int) bool {
|
||||
a, b := rows[i], rows[j]
|
||||
if a.Name == "" && b.Name != "" {
|
||||
return false
|
||||
}
|
||||
if a.Name != "" && b.Name == "" {
|
||||
return true
|
||||
}
|
||||
if a.Name != b.Name {
|
||||
return a.Name < b.Name
|
||||
}
|
||||
return a.Key < b.Key
|
||||
})
|
||||
result.Total = len(rows)
|
||||
start := (f.Page - 1) * f.Limit
|
||||
if start < len(rows) {
|
||||
end := min(start+f.Limit, len(rows))
|
||||
result.Rows = rows[start:end]
|
||||
result.HasMore = end < len(rows)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// Names remain in the bounded read-query body, not URLs/access logs. The source
|
||||
// reader and governance store are still strictly read-only on this route.
|
||||
func monthlyRequestValues(w http.ResponseWriter, r *http.Request) (url.Values, error) {
|
||||
values := url.Values{}
|
||||
if r.URL.RawQuery != "" || r.Header.Get("X-Governance-Action") != "qa-usage-read" {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 8192))
|
||||
token, err := decoder.Token()
|
||||
if err != nil || token != json.Delim('{') {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
for decoder.More() {
|
||||
token, err = decoder.Token()
|
||||
key, ok := token.(string)
|
||||
if err != nil || !ok || values.Has(key) {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
var raw json.RawMessage
|
||||
if decoder.Decode(&raw) != nil || bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
if key == "page" || key == "limit" {
|
||||
var n int
|
||||
if json.Unmarshal(raw, &n) != nil {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
values.Set(key, strconv.Itoa(n))
|
||||
} else {
|
||||
var text string
|
||||
if json.Unmarshal(raw, &text) != nil {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
values.Set(key, text)
|
||||
}
|
||||
}
|
||||
if _, err = decoder.Token(); err != nil {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
var extra any
|
||||
if decoder.Decode(&extra) != io.EOF {
|
||||
return nil, dobotaiops.ErrInvalidQuery
|
||||
}
|
||||
return values, nil
|
||||
}
|
||||
func qaMonthlyPeople(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
values, err := monthlyRequestValues(w, r)
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
f, err := parseMonthlyFilter(values, config.Now())
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
p, err := recordAttribution(r.Context(), config)
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
if p.Metadata.State != "ready" {
|
||||
writeJSON(w, 409, map[string]string{"code": "directory_missing", "error": "尚无完整通讯录，请先从区域使用页同步；不能生成零记录人员名单"})
|
||||
return
|
||||
}
|
||||
|
||||
if values.Has("directory_id") || values.Has("revision_id") {
|
||||
if !values.Has("directory_id") || !values.Has("revision_id") {
|
||||
qaOpsError(w, dobotaiops.ErrInvalidQuery)
|
||||
return
|
||||
}
|
||||
if values.Get("directory_id") != p.Metadata.DirectoryID || values.Get("revision_id") != p.Metadata.RevisionID {
|
||||
writeJSON(w, 409, map[string]string{"code": "organization_changed", "error": "通讯录或区域配置已变化，请重新查询个人月明细"})
|
||||
return
|
||||
}
|
||||
}
|
||||
if f.Department != "" {
|
||||
found := false
|
||||
for _, d := range p.Metadata.View.Departments {
|
||||
found = found || d.ID == f.Department
|
||||
}
|
||||
if !found {
|
||||
qaOpsError(w, dobotaiops.ErrInvalidQuery)
|
||||
return
|
||||
}
|
||||
}
|
||||
reader, ok := config.QAOps.(qaUsageReader)
|
||||
if !ok || config.QAOpsTenantKey == "" {
|
||||
qaOpsError(w, dobotaiops.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
groups, err := reader.ReadUsageGroups(r.Context(), dobotaiops.RecordQuery{Window: dobotaiops.Window{TenantKey: config.QAOpsTenantKey, Start: f.Start, End: f.End}})
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": monthlyResult(groups, p, f)})
|
||||
}
|
||||
@@ -0,0 +1,221 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type monthlyReaderSpy struct {
|
||||
attributedReader
|
||||
query dobotaiops.RecordQuery
|
||||
calls int
|
||||
}
|
||||
|
||||
func (s *monthlyReaderSpy) ReadUsageGroups(ctx context.Context, q dobotaiops.RecordQuery) (dobotaiops.UsageGroups, error) {
|
||||
s.calls++
|
||||
s.query = q
|
||||
return s.attributedReader.ReadUsageGroups(ctx, q)
|
||||
}
|
||||
func monthlyRequest(h http.Handler, path, token string) *httptest.ResponseRecorder {
|
||||
u, _ := url.Parse(path)
|
||||
body := map[string]any{}
|
||||
for k, v := range u.Query() {
|
||||
if len(v) != 1 {
|
||||
body[k] = v
|
||||
continue
|
||||
}
|
||||
if k == "page" || k == "limit" {
|
||||
if n, e := strconv.Atoi(v[0]); e == nil {
|
||||
body[k] = n
|
||||
} else {
|
||||
body[k] = v[0]
|
||||
}
|
||||
} else {
|
||||
body[k] = v[0]
|
||||
}
|
||||
}
|
||||
raw, _ := json.Marshal(body)
|
||||
r := httptest.NewRequest(http.MethodPost, u.Path, bytes.NewReader(raw))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
r.Header.Set("X-Governance-Action", "qa-usage-read")
|
||||
if token != "" {
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: token})
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
func monthlyHandler(t *testing.T) (http.Handler, *attributionOrgSpy, *monthlyReaderSpy) {
|
||||
t.Helper()
|
||||
d := qaopsorg.Directory{Departments: []qaopsorg.Department{{ID: "root", Name: "示例根"}, {ID: "child", ParentID: "root", Name: "子部门"}}, People: []qaopsorg.Person{{ID: "fixture-private-staff", Name: "同名示例", UnionID: "fixture-private-union", DepartmentIDs: []string{"root"}}, {ID: "private-zero", Name: "同名示例", DepartmentIDs: []string{"child"}}, {ID: "private-cross", Name: "跨区域示例", DepartmentIDs: []string{"root", "child"}}, {ID: "private-unnamed", DepartmentIDs: []string{"child"}}}}
|
||||
now := time.Date(2026, 10, 17, 8, 0, 0, 0, time.UTC)
|
||||
p, err := qaopsorg.BuildAttribution(qaopsorg.State{State: "ready", DirectoryID: "monthly-directory", DirectorySource: "weekly_report_live", DirectorySyncedAt: &now, RevisionID: "monthly-regions", Rules: []qaopsorg.Rule{{DepartmentID: "root", Region: "国内工业"}, {DepartmentID: "child", Region: "APAC"}}}, d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
org := &attributionOrgSpy{projection: p}
|
||||
reader := &monthlyReaderSpy{}
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["attribution-admin"] = "usr-boss"
|
||||
auth.sessions["attribution-viewer"] = "usr-eye"
|
||||
h, err := NewHandler(Config{Store: &fakeJobStore{}, Auth: &AuthConfig{Store: auth}, QAOps: reader, QAOpsTenantKey: "fixed", QAOrg: org, Now: func() time.Time { return now }})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return h, org, reader
|
||||
}
|
||||
func TestQAMonthlyPeopleIncludesZeroUseAndKeepsUnresolvedEvents(t *testing.T) {
|
||||
h, _, reader := monthlyHandler(t)
|
||||
path := "/api/v1/qa-ops/usage/people?month=2026-10&limit=2"
|
||||
w := monthlyRequest(h, path, "attribution-admin")
|
||||
if w.Code != 200 {
|
||||
t.Fatal(w.Code, w.Body.String())
|
||||
}
|
||||
var result struct {
|
||||
Data struct {
|
||||
Month string
|
||||
Timezone string
|
||||
Partial bool
|
||||
Questions int64
|
||||
LinkedQuestions int64 `json:"linked_questions"`
|
||||
UnresolvedQuestions int64 `json:"unresolved_questions"`
|
||||
RosterPeople int `json:"roster_people"`
|
||||
UsedPeople int `json:"used_people"`
|
||||
ZeroPeople int `json:"zero_people"`
|
||||
MissingNames int `json:"missing_names"`
|
||||
Total int
|
||||
HasMore bool `json:"has_more"`
|
||||
Rows []struct {
|
||||
Key string
|
||||
Name string
|
||||
Questions int64
|
||||
State string
|
||||
}
|
||||
}
|
||||
}
|
||||
if json.Unmarshal(w.Body.Bytes(), &result) != nil {
|
||||
t.Fatal("invalid monthly JSON")
|
||||
}
|
||||
v := result.Data
|
||||
if v.Month != "2026-10" || v.Timezone != "Asia/Shanghai" || !v.Partial || v.Questions != 7 || v.LinkedQuestions != 5 || v.UnresolvedQuestions != 2 || v.RosterPeople != 4 || v.UsedPeople != 1 || v.ZeroPeople != 3 || v.MissingNames != 1 || v.Total != 4 || len(v.Rows) != 2 || !v.HasMore {
|
||||
t.Fatal("monthly roster, zero count or alias dedup wrong", w.Body.String())
|
||||
}
|
||||
if reader.query.Window.TenantKey != "fixed" || reader.query.Window.Start.Format(time.RFC3339) != "2026-10-01T00:00:00+08:00" || reader.query.Window.End.Format(time.RFC3339) != "2026-10-17T16:00:00+08:00" {
|
||||
t.Fatal("month was not normalized to the named timezone")
|
||||
}
|
||||
if strings.Contains(w.Body.String(), "private-") || strings.Contains(w.Body.String(), "performance") || strings.Contains(w.Body.String(), "staff_id") {
|
||||
t.Fatal("raw identities or unapproved verdict exposed")
|
||||
}
|
||||
page2 := monthlyRequest(h, path+"&page=2&directory_id=monthly-directory&revision_id=monthly-regions", "attribution-admin")
|
||||
if page2.Code != 200 {
|
||||
t.Fatal(page2.Body.String())
|
||||
}
|
||||
for _, extra := range []string{"&activity=zero", "®ion=cross_region", "&department_id=child", "&search=" + url.QueryEscape("子部门")} {
|
||||
r := monthlyRequest(h, "/api/v1/qa-ops/usage/people?month=2026-10"+extra, "attribution-admin")
|
||||
if r.Code != 200 {
|
||||
t.Fatal("valid monthly filter failed", r.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestQAMonthlyScopeAndVersionFailClosed(t *testing.T) {
|
||||
h, org, reader := monthlyHandler(t)
|
||||
base := "/api/v1/qa-ops/usage/people?month=2026-10"
|
||||
for _, tc := range []struct {
|
||||
suffix, token string
|
||||
status int
|
||||
}{{"", "", 401}, {"", "attribution-viewer", 403}, {"&tenant=forged", "attribution-admin", 400}, {"&threshold=5", "attribution-admin", 400}, {"&month=duplicate", "attribution-admin", 400}, {"&page=0", "attribution-admin", 400}, {"&limit=101", "attribution-admin", 400}, {"&activity=under5", "attribution-admin", 400}, {"®ion=invalid", "attribution-admin", 400}, {"&department_id=invalid", "attribution-admin", 400}, {"&page=2", "attribution-admin", 400}, {"&page=2&directory_id=changed&revision_id=monthly-regions", "attribution-admin", 409}, {"&cutoff=2026-11-01T00:00:00Z", "attribution-admin", 400}} {
|
||||
w := monthlyRequest(h, base+tc.suffix, tc.token)
|
||||
if w.Code != tc.status {
|
||||
t.Fatalf("monthly %s: %d want %d", tc.suffix, w.Code, tc.status)
|
||||
}
|
||||
}
|
||||
for _, month := range []string{"2026-1", " 2026-10", "2026-13", "0000-01", "2026-11"} {
|
||||
r := monthlyRequest(h, "/api/v1/qa-ops/usage/people?month="+url.QueryEscape(month), "attribution-admin")
|
||||
if r.Code != 400 {
|
||||
t.Fatal("invalid/future month accepted")
|
||||
}
|
||||
}
|
||||
if reader.calls != 0 {
|
||||
t.Fatal("invalid or unauthorized monthly request reached source")
|
||||
}
|
||||
org.projection, _ = qaopsorg.BuildAttribution(qaopsorg.State{State: "empty"}, qaopsorg.Directory{})
|
||||
if w := monthlyRequest(h, base, "attribution-admin"); w.Code != 409 {
|
||||
t.Fatal("no roster presented as zero-use list")
|
||||
}
|
||||
org.err = qaopsorg.ErrUnavailable
|
||||
if w := monthlyRequest(h, base, "attribution-admin"); w.Code != 503 {
|
||||
t.Fatal("store failure hidden as empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQAMonthlyPersonFiltersDoNotGuessNamesOrDuplicateDescendantMembers(t *testing.T) {
|
||||
h, _, _ := monthlyHandler(t)
|
||||
for _, tc := range []struct {
|
||||
filter string
|
||||
total int
|
||||
}{{"&activity=used", 1}, {"&activity=zero", 3}, {"®ion=cross_region", 1}, {"®ion=APAC", 2}, {"&department_id=child", 3}, {"&department_id=root", 4}, {"&search=" + url.QueryEscape("同名示例"), 2}, {"&search=" + url.QueryEscape("子部门"), 3}, {"&search=%25_", 0}} {
|
||||
w := monthlyRequest(h, "/api/v1/qa-ops/usage/people?month=2026-10"+tc.filter, "attribution-admin")
|
||||
var payload struct{ Data monthlyPeople }
|
||||
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &payload) != nil || payload.Data.Total != tc.total || payload.Data.Questions != 7 {
|
||||
t.Fatal("person-scope filter did not preserve source totals or person dedup", tc.filter)
|
||||
}
|
||||
for _, r := range payload.Data.Rows {
|
||||
if tc.filter == "&activity=zero" && r.Questions != 0 {
|
||||
t.Fatal("used member labeled zero")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestQAMonthlyReadBodyRejectsDuplicateNullTrailingAndOversizedInput(t *testing.T) {
|
||||
h, _, reader := monthlyHandler(t)
|
||||
for _, body := range []string{`{"month":"2026-10","month":"2026-10"}`, `{"month":"2026-10","search":null}`, `{"month":"2026-10","page":"1"}`, `{"month":"2026-10"}{}`, `[]`, `{"month":"2026-10","search":"` + strings.Repeat("x", 9000) + `"}`} {
|
||||
r := httptest.NewRequest("POST", "/api/v1/qa-ops/usage/people", strings.NewReader(body))
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "attribution-admin"})
|
||||
r.Header.Set("X-Governance-Action", "qa-usage-read")
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != 400 {
|
||||
t.Fatal("invalid read body reached source", w.Code)
|
||||
}
|
||||
}
|
||||
for _, method := range []string{"POST", "GET"} {
|
||||
r := httptest.NewRequest(method, "/api/v1/qa-ops/usage/people?search=sensitive-example", strings.NewReader(`{"month":"2026-10"}`))
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "attribution-admin"})
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != 400 && w.Code != 405 {
|
||||
t.Fatal("URL-based name query accepted")
|
||||
}
|
||||
}
|
||||
if reader.calls != 0 {
|
||||
t.Fatal("invalid body read production source")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQAMonthlyPastLeapMonthUsesClosedCalendarWindow(t *testing.T) {
|
||||
h, _, reader := monthlyHandler(t)
|
||||
w := monthlyRequest(h, "/api/v1/qa-ops/usage/people?month=2024-02", "attribution-admin")
|
||||
if w.Code != 200 {
|
||||
t.Fatal(w.Body.String())
|
||||
}
|
||||
if reader.query.Window.End.Format(time.RFC3339) != "2024-03-01T00:00:00+08:00" || reader.query.Window.End.Sub(reader.query.Window.Start) != 29*24*time.Hour {
|
||||
t.Fatal("leap month boundary incorrect")
|
||||
}
|
||||
var raw struct{ Data struct{ Partial bool } }
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &raw)
|
||||
if raw.Data.Partial {
|
||||
t.Fatal("past natural month mislabeled as current progress")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
)
|
||||
|
||||
// No raw SQL, source credential, arbitrary tenant or write method crosses this boundary.
|
||||
type QAOpsReader interface {
|
||||
ListRecords(context.Context, dobotaiops.RecordQuery) (dobotaiops.RecordPage, error)
|
||||
GetRecord(context.Context, string, string) (dobotaiops.Record, error)
|
||||
Available(context.Context) error
|
||||
Safety() dobotaiops.SourceSafety
|
||||
}
|
||||
|
||||
func authorizeQAOps(w http.ResponseWriter, r *http.Request) bool {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
w.Header().Set("Pragma", "no-cache")
|
||||
identity, ok := sessionFrom(r.Context())
|
||||
if !ok {
|
||||
writeJSON(w, http.StatusUnauthorized, map[string]string{"error": "问答运营需要登录后访问"})
|
||||
return false
|
||||
}
|
||||
// Until the employee-data visibility scope is approved, do not widen access
|
||||
// to all pre-existing viewers/engineers merely because this endpoint is GET.
|
||||
if identity.Role != roleAdmin {
|
||||
writeJSON(w, http.StatusForbidden, map[string]string{"error": "尚未配置员工问答可见范围，当前仅管理员可访问"})
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func qaOpsStatus(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
state := "disabled"
|
||||
code := "source_not_configured"
|
||||
warnings := []string{}
|
||||
if config.QAOpsEnabled {
|
||||
state = "unavailable"
|
||||
code = "source_unavailable"
|
||||
}
|
||||
if config.QAOpsProblem != "" {
|
||||
state = "blocked"
|
||||
code = "source_configuration_invalid"
|
||||
}
|
||||
if config.QAOps != nil && config.QAOps.Available(r.Context()) == nil {
|
||||
state = "connected"
|
||||
code = ""
|
||||
safety := config.QAOps.Safety()
|
||||
if safety.DatabaseTemp {
|
||||
warnings = append(warnings, "database_temp_enabled")
|
||||
}
|
||||
if !safety.SSL {
|
||||
warnings = append(warnings, "database_transport_unencrypted")
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"data": map[string]any{"state": state, "code": code, "warnings": warnings, "read_only": true, "source_version": "2.0", "channel": "dingtalk"}})
|
||||
}
|
||||
|
||||
func listQAOpsRecords(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
if config.QAOps == nil || config.QAOpsTenantKey == "" {
|
||||
qaOpsError(w, dobotaiops.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
values := r.URL.Query()
|
||||
allowed := map[string]bool{"start": true, "end": true, "keyword": true, "cursor": true, "limit": true}
|
||||
for key, items := range values {
|
||||
if !allowed[key] || len(items) != 1 {
|
||||
qaOpsError(w, dobotaiops.ErrInvalidQuery)
|
||||
return
|
||||
}
|
||||
}
|
||||
window, err := dobotaiops.NewWindow(config.QAOpsTenantKey, values.Get("start"), values.Get("end"), 90*24*time.Hour, config.Now())
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
limit := 20
|
||||
if raw := values.Get("limit"); raw != "" {
|
||||
limit, err = strconv.Atoi(raw)
|
||||
if err != nil || limit < 1 || limit > 100 {
|
||||
qaOpsError(w, dobotaiops.ErrInvalidQuery)
|
||||
return
|
||||
}
|
||||
}
|
||||
projection, err := recordAttribution(r.Context(), config)
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
page, err := config.QAOps.ListRecords(r.Context(), dobotaiops.RecordQuery{Window: window, Keyword: values.Get("keyword"), Cursor: values.Get("cursor"), Limit: limit})
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
enrichQAPage(&page, projection)
|
||||
if err := enrichQAQualityPage(r.Context(), config, &page); err != nil {
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"data": page})
|
||||
}
|
||||
|
||||
func getQAOpsRecord(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
if config.QAOps == nil || config.QAOpsTenantKey == "" {
|
||||
qaOpsError(w, dobotaiops.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
if r.URL.RawQuery != "" {
|
||||
qaOpsError(w, dobotaiops.ErrInvalidQuery)
|
||||
return
|
||||
}
|
||||
projection, err := recordAttribution(r.Context(), config)
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
item, err := config.QAOps.GetRecord(r.Context(), config.QAOpsTenantKey, r.PathValue("id"))
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
enrichQARecord(&item, projection)
|
||||
if err := enrichQAQualityRecord(r.Context(), config, &item); err != nil {
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"data": item})
|
||||
}
|
||||
|
||||
func qaOpsError(w http.ResponseWriter, err error) {
|
||||
status := http.StatusServiceUnavailable
|
||||
code := "source_unavailable"
|
||||
message := "问答数据源不可用，请检查独立只读连接；原有知识治理功能不受影响"
|
||||
if errors.Is(err, errQAOrganization) {
|
||||
code = "organization_unavailable"
|
||||
message = "通讯录或区域配置读取失败，未将不可用数据伪装成空列，请重试"
|
||||
}
|
||||
if errors.Is(err, dobotaiops.ErrInvalidQuery) {
|
||||
status = http.StatusBadRequest
|
||||
code = "invalid_query"
|
||||
message = "查询参数无效：请检查时间范围、关键词长度或重新查询后翻页"
|
||||
}
|
||||
if errors.Is(err, dobotaiops.ErrNotReadOnly) {
|
||||
code = "source_not_read_only"
|
||||
message = "问答连接权限校验未通过，已阻止读取，请联系管理员"
|
||||
}
|
||||
if errors.Is(err, dobotaiops.ErrRecordNotFound) {
|
||||
status = http.StatusNotFound
|
||||
code = "record_not_found"
|
||||
message = "未找到当前授权范围内的问答记录"
|
||||
}
|
||||
writeJSON(w, status, map[string]string{"error": message, "code": code})
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type qaOpsSpy struct {
|
||||
query dobotaiops.RecordQuery
|
||||
tenant, id string
|
||||
calls int
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *qaOpsSpy) ListRecords(_ context.Context, q dobotaiops.RecordQuery) (dobotaiops.RecordPage, error) {
|
||||
s.query = q
|
||||
s.calls++
|
||||
return dobotaiops.RecordPage{Items: []dobotaiops.Record{}}, s.err
|
||||
}
|
||||
func (s *qaOpsSpy) GetRecord(_ context.Context, tenant, id string) (dobotaiops.Record, error) {
|
||||
s.tenant = tenant
|
||||
s.id = id
|
||||
s.calls++
|
||||
return dobotaiops.Record{}, s.err
|
||||
}
|
||||
func (s *qaOpsSpy) Available(context.Context) error { return s.err }
|
||||
func (s *qaOpsSpy) Safety() dobotaiops.SourceSafety {
|
||||
return dobotaiops.SourceSafety{DatabaseTemp: true, SSL: false}
|
||||
}
|
||||
|
||||
func TestQAOperationsRequireExplicitSessionAndServerSelectedTenant(t *testing.T) {
|
||||
spy := &qaOpsSpy{}
|
||||
auth := newFakeAuthStore()
|
||||
for token, user := range map[string]string{"admin-qa-fixture": "usr-boss", "viewer-qa-fixture": "usr-eye", "engineer-qa-fixture": "usr-eng"} {
|
||||
auth.sessions[token] = user
|
||||
}
|
||||
handler, err := NewHandler(Config{Store: &fakeJobStore{}, QAOps: spy, QAOpsEnabled: true, QAOpsTenantKey: "approved-tenant", Auth: &AuthConfig{Store: auth}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := "/api/v1/qa-ops/records?start=2026-10-14T00:00:00Z&end=2026-10-15T00:00:00Z"
|
||||
for _, tc := range []struct {
|
||||
token, path string
|
||||
status int
|
||||
}{
|
||||
{"", base, http.StatusUnauthorized}, {"viewer-qa-fixture", base, http.StatusForbidden}, {"engineer-qa-fixture", base, http.StatusForbidden},
|
||||
{"admin-qa-fixture", base + "&tenant_key=another-tenant", http.StatusBadRequest},
|
||||
{"admin-qa-fixture", base + "&limit=0", http.StatusBadRequest},
|
||||
{"admin-qa-fixture", base + "&keyword=a&keyword=b", http.StatusBadRequest},
|
||||
{"admin-qa-fixture", base + "&keyword=test", http.StatusOK},
|
||||
} {
|
||||
request := httptest.NewRequest(http.MethodGet, tc.path, nil)
|
||||
if tc.token != "" {
|
||||
request.AddCookie(&http.Cookie{Name: sessionCookieName, Value: tc.token})
|
||||
}
|
||||
response := httptest.NewRecorder()
|
||||
handler.ServeHTTP(response, request)
|
||||
if response.Code != tc.status {
|
||||
t.Fatalf("status=%d want=%d body=%s", response.Code, tc.status, response.Body.String())
|
||||
}
|
||||
}
|
||||
if spy.calls != 1 || spy.query.Window.TenantKey != "approved-tenant" || spy.query.Keyword != "test" {
|
||||
t.Fatalf("unauthorized/invalid reads reached source: %+v", spy)
|
||||
}
|
||||
spy.err = errors.New("sensitive-driver-detail-not-for-browser")
|
||||
request := httptest.NewRequest(http.MethodGet, base, nil)
|
||||
request.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "admin-qa-fixture"})
|
||||
response := httptest.NewRecorder()
|
||||
handler.ServeHTTP(response, request)
|
||||
if response.Code != http.StatusServiceUnavailable || strings.Contains(response.Body.String(), "sensitive-driver-detail") {
|
||||
t.Fatalf("source failure leaked or became empty success: %s", response.Body.String())
|
||||
}
|
||||
if response.Header().Get("Cache-Control") != "no-store" {
|
||||
t.Fatal("employee answers must not be cached")
|
||||
}
|
||||
unauth, _ := NewHandler(Config{Store: &fakeJobStore{}, QAOps: spy, QAOpsTenantKey: "approved-tenant"})
|
||||
response = httptest.NewRecorder()
|
||||
unauth.ServeHTTP(response, httptest.NewRequest(http.MethodGet, base, nil))
|
||||
if response.Code != http.StatusUnauthorized {
|
||||
t.Fatal("auth-disabled governance must not expose production answers")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestQAOperationsRoleGatePrecedesGenericGETShortcut(t *testing.T) {
|
||||
for _, path := range []string{"/api/v1/qa-ops/status", "/api/v1/qa-ops/records", "/api/v1/qa-ops/records/some-id"} {
|
||||
if role := classifyRole(http.MethodGet, path); role != roleAdmin {
|
||||
t.Fatalf("QA employee data route classified as %s, must require admin: %s", role, path)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestQAOperationsFailClosedWithoutBreakingGovernance(t *testing.T) {
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["admin-qa-fixture"] = "usr-boss"
|
||||
handler, err := NewHandler(Config{Store: &fakeJobStore{}, Auth: &AuthConfig{Store: auth}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request := httptest.NewRequest(http.MethodGet, "/api/v1/qa-ops/records?start=2026-10-14T00:00:00Z&end=2026-10-15T00:00:00Z", nil)
|
||||
request.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "admin-qa-fixture"})
|
||||
response := httptest.NewRecorder()
|
||||
handler.ServeHTTP(response, request)
|
||||
if response.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("unconfigured source must not be an empty list: status=%d body=%s", response.Code, response.Body.String())
|
||||
}
|
||||
health := httptest.NewRecorder()
|
||||
handler.ServeHTTP(health, httptest.NewRequest(http.MethodGet, "/livez", nil))
|
||||
if health.Code != http.StatusOK {
|
||||
t.Fatalf("external source must not stop governance liveness: %d", health.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
type QAOrgStore interface {
|
||||
GetView(context.Context) (qaopsorg.State, error)
|
||||
PreviewRules(context.Context, qaopsorg.SaveRequest) (qaopsorg.State, error)
|
||||
SaveRules(context.Context, qaopsorg.SaveRequest, string) (qaopsorg.State, error)
|
||||
}
|
||||
|
||||
func qaOrganization(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
if r.URL.RawQuery != "" {
|
||||
writeOrgError(w, qaopsorg.ErrInvalid)
|
||||
return
|
||||
}
|
||||
if config.QAOrg == nil {
|
||||
writeOrgError(w, qaopsorg.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
var result qaopsorg.State
|
||||
var err error
|
||||
if r.Method == http.MethodGet {
|
||||
result, err = config.QAOrg.GetView(r.Context())
|
||||
} else {
|
||||
if !authorizeQAOrganizationWrite(w, r) {
|
||||
return
|
||||
}
|
||||
var request qaopsorg.SaveRequest
|
||||
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20))
|
||||
decoder.DisallowUnknownFields()
|
||||
if decoder.Decode(&request) != nil || decoder.Decode(&struct{}{}) != io.EOF || request.DirectoryID == "" || request.Rules == nil {
|
||||
writeOrgError(w, qaopsorg.ErrInvalid)
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodPost {
|
||||
result, err = config.QAOrg.PreviewRules(r.Context(), request)
|
||||
} else {
|
||||
identity, _ := sessionFrom(r.Context())
|
||||
result, err = config.QAOrg.SaveRules(r.Context(), request, identity.Username)
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
writeOrgError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": result})
|
||||
}
|
||||
func writeOrgError(w http.ResponseWriter, err error) {
|
||||
status, message := 503, "通讯录或区域配置存储不可用，未展示为空数据"
|
||||
switch {
|
||||
case errors.Is(err, qaopsorg.ErrInvalid):
|
||||
status, message = 400, "部门树或区域配置无效，请核对重复、缺失路径及区域"
|
||||
case errors.Is(err, qaopsorg.ErrConflict):
|
||||
status, message = 409, "通讯录或区域配置已变化，请重新读取后预览"
|
||||
case errors.Is(err, qaopsorg.ErrNoDirectory):
|
||||
status, message = 409, "尚无完整通讯录快照，不能保存区域配置"
|
||||
}
|
||||
writeJSON(w, status, map[string]string{"error": message})
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestQAOrganizationSameOriginThroughReverseProxy(t *testing.T) {
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["org-proxy-admin"] = "usr-boss"
|
||||
spy := &orgSpy{}
|
||||
handler, err := NewHandler(Config{Store: &fakeJobStore{}, QAOrg: spy, Auth: &AuthConfig{Store: auth}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
site string
|
||||
status int
|
||||
}{{"same-origin", 200}, {"cross-site", 403}, {"same-site", 403}} {
|
||||
req := httptest.NewRequest("PUT", "http://internal-api:8080/api/v1/qa-ops/organization/regions", strings.NewReader(`{"directory_id":"d","base_revision":"","rules":[]}`))
|
||||
req.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "org-proxy-admin"})
|
||||
req.Header.Set("X-Governance-Action", "qa-organization")
|
||||
req.Header.Set("Origin", "http://frontend:8088")
|
||||
req.Header.Set("Sec-Fetch-Site", tc.site)
|
||||
res := httptest.NewRecorder()
|
||||
handler.ServeHTTP(res, req)
|
||||
if res.Code != tc.status {
|
||||
t.Fatalf("proxy fetch site %s got %d want %d", tc.site, res.Code, tc.status)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type QAOrgSyncStore interface {
|
||||
GetSyncStatus(context.Context) (qaopsorg.SyncStatus, error)
|
||||
EnqueueSync(context.Context, string, string) (qaopsorg.SyncReceipt, bool, error)
|
||||
}
|
||||
|
||||
func authorizeQAOrganizationWrite(w http.ResponseWriter, r *http.Request) bool {
|
||||
return authorizeQAIntent(w, r, "qa-organization")
|
||||
}
|
||||
func authorizeQAIntent(w http.ResponseWriter, r *http.Request, intent string) bool {
|
||||
if r.Header.Get("X-Governance-Action") != intent || r.Header.Get("Sec-Fetch-Site") == "cross-site" {
|
||||
writeJSON(w, 403, map[string]string{"error": "此操作需要同源操作意图"})
|
||||
return false
|
||||
}
|
||||
// Fetch Metadata survives same-origin proxies that rewrite Host; untrusted
|
||||
// forwarded headers must not grant cross-origin write authority.
|
||||
if raw := r.Header.Get("Origin"); raw != "" && r.Header.Get("Sec-Fetch-Site") != "same-origin" {
|
||||
origin, err := url.Parse(raw)
|
||||
if err != nil || (origin.Scheme != "http" && origin.Scheme != "https") || origin.Host != r.Host {
|
||||
writeJSON(w, 403, map[string]string{"error": "拒绝跨站配置请求"})
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
func qaOrganizationSync(w http.ResponseWriter, r *http.Request, c Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
if r.URL.RawQuery != "" {
|
||||
writeOrgError(w, qaopsorg.ErrInvalid)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
if r.Method == http.MethodGet {
|
||||
if c.QAOrgSync == nil {
|
||||
writeJSON(w, 200, map[string]any{"data": qaopsorg.SyncStatus{}})
|
||||
return
|
||||
}
|
||||
result, err := c.QAOrgSync.GetSyncStatus(r.Context())
|
||||
if err != nil {
|
||||
writeOrgError(w, err)
|
||||
return
|
||||
}
|
||||
result.Configured = c.QAOrgSyncEnabled
|
||||
writeJSON(w, 200, map[string]any{"data": result})
|
||||
return
|
||||
}
|
||||
if !authorizeQAOrganizationWrite(w, r) {
|
||||
return
|
||||
}
|
||||
key := r.Header.Get("Idempotency-Key")
|
||||
if key == "" || len(key) > 128 || strings.TrimSpace(key) != key {
|
||||
writeJSON(w, 400, map[string]string{"error": "同步请求需要有效 Idempotency-Key，重试须复用原标识"})
|
||||
return
|
||||
}
|
||||
var body *struct{}
|
||||
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1024))
|
||||
decoder.DisallowUnknownFields()
|
||||
if decoder.Decode(&body) != nil || body == nil || decoder.Decode(&struct{}{}) != io.EOF {
|
||||
writeOrgError(w, qaopsorg.ErrInvalid)
|
||||
return
|
||||
}
|
||||
if !c.QAOrgSyncEnabled || c.QAOrgSync == nil {
|
||||
writeJSON(w, 503, map[string]string{"error": "通讯录后台同步未接线，未创建任务"})
|
||||
return
|
||||
}
|
||||
identity, _ := sessionFrom(r.Context())
|
||||
receipt, replayed, err := c.QAOrgSync.EnqueueSync(r.Context(), key, identity.Username)
|
||||
if err != nil {
|
||||
if errors.Is(err, qaopsorg.ErrSyncWorkerUnavailable) {
|
||||
writeJSON(w, 503, map[string]string{"error": "通讯录执行器未在线，未创建任务；请检查后台同步配置"})
|
||||
return
|
||||
}
|
||||
writeOrgError(w, err)
|
||||
return
|
||||
}
|
||||
operation := "created"
|
||||
if replayed {
|
||||
operation = "replayed"
|
||||
}
|
||||
writeJSON(w, 202, map[string]any{"data": receipt, "operation": operation})
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGenericJobCreationCannotBypassDirectorySyncAdminPath(t *testing.T) {
|
||||
h, err := NewHandler(Config{Store: &fakeJobStore{}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := httptest.NewRequest("POST", "/api/v1/jobs", strings.NewReader(`{"type":"qaops.directory.sync.fixture","payload":{"triggered_by":"forged"},"idempotency_key":"bypass"}`))
|
||||
res := httptest.NewRecorder()
|
||||
h.ServeHTTP(res, req)
|
||||
if res.Code != 400 {
|
||||
t.Fatalf("reserved sync job bypassed its dedicated admission path: %d", res.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type orgSyncSpy struct {
|
||||
calls int
|
||||
key, actor string
|
||||
replay bool
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *orgSyncSpy) GetSyncStatus(context.Context) (qaopsorg.SyncStatus, error) {
|
||||
s.calls++
|
||||
return qaopsorg.SyncStatus{Configured: true, WorkerReady: true}, s.err
|
||||
}
|
||||
func (s *orgSyncSpy) EnqueueSync(_ context.Context, key, actor string) (qaopsorg.SyncReceipt, bool, error) {
|
||||
s.calls++
|
||||
s.key = key
|
||||
s.actor = actor
|
||||
return qaopsorg.SyncReceipt{JobID: "fixture-job"}, s.replay, s.err
|
||||
}
|
||||
func TestQAOrganizationSyncRequiresAdminIntentAndDurableOperation(t *testing.T) {
|
||||
spy := &orgSyncSpy{}
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["sync-admin"] = "usr-boss"
|
||||
auth.sessions["sync-viewer"] = "usr-eye"
|
||||
handler, err := NewHandler(Config{Store: &fakeJobStore{}, QAOrgSync: spy, QAOrgSyncEnabled: true, Auth: &AuthConfig{Store: auth}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
method, token, key, action, origin, body string
|
||||
status int
|
||||
}{
|
||||
{"GET", "", "", "", "", "", 401}, {"GET", "sync-viewer", "", "", "", "", 403}, {"GET", "sync-admin", "", "", "", "", 200},
|
||||
{"POST", "sync-admin", "op-one", "", "", "{}", 403}, {"POST", "sync-admin", "op-one", "qa-organization", "https://evil.invalid", "{}", 403},
|
||||
{"POST", "sync-admin", "", "qa-organization", "", "{}", 400}, {"POST", "sync-admin", " op-one ", "qa-organization", "", "{}", 400},
|
||||
{"POST", "sync-admin", "op-one", "qa-organization", "", `null`, 400},
|
||||
{"POST", "sync-admin", "op-one", "qa-organization", "", `{"tenant":"forged"}`, 400}, {"POST", "sync-admin", "op-one", "qa-organization", "", "{} {}", 400},
|
||||
{"POST", "sync-admin", "op-one", "qa-organization", "", "{}", 202},
|
||||
} {
|
||||
req := httptest.NewRequest(tc.method, "/api/v1/qa-ops/organization/sync", strings.NewReader(tc.body))
|
||||
if tc.token != "" {
|
||||
req.AddCookie(&http.Cookie{Name: sessionCookieName, Value: tc.token})
|
||||
}
|
||||
req.Header.Set("Idempotency-Key", tc.key)
|
||||
req.Header.Set("X-Governance-Action", tc.action)
|
||||
req.Header.Set("Origin", tc.origin)
|
||||
res := httptest.NewRecorder()
|
||||
handler.ServeHTTP(res, req)
|
||||
if res.Code != tc.status {
|
||||
t.Fatalf("%+v got %d: %s", tc, res.Code, res.Body.String())
|
||||
}
|
||||
}
|
||||
if spy.calls != 2 || spy.key != "op-one" || spy.actor == "" {
|
||||
t.Fatalf("invalid request reached store: %+v", spy)
|
||||
}
|
||||
spy.replay = true
|
||||
req := httptest.NewRequest("POST", "/api/v1/qa-ops/organization/sync", strings.NewReader("{}"))
|
||||
req.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "sync-admin"})
|
||||
req.Header.Set("Idempotency-Key", "op-one")
|
||||
req.Header.Set("X-Governance-Action", "qa-organization")
|
||||
res := httptest.NewRecorder()
|
||||
handler.ServeHTTP(res, req)
|
||||
var body map[string]any
|
||||
if json.Unmarshal(res.Body.Bytes(), &body) != nil || body["operation"] != "replayed" {
|
||||
t.Fatal("operation replay not exposed")
|
||||
}
|
||||
disabled, _ := NewHandler(Config{Store: &fakeJobStore{}, QAOrgSync: spy, Auth: &AuthConfig{Store: auth}})
|
||||
req.Body = io.NopCloser(strings.NewReader("{}"))
|
||||
res = httptest.NewRecorder()
|
||||
disabled.ServeHTTP(res, req)
|
||||
if res.Code != 503 {
|
||||
t.Fatal("disabled sync enqueued")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type orgSpy struct {
|
||||
calls int
|
||||
actor string
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *orgSpy) GetView(context.Context) (qaopsorg.State, error) {
|
||||
s.calls++
|
||||
return qaopsorg.State{State: "empty"}, s.err
|
||||
}
|
||||
func (s *orgSpy) PreviewRules(context.Context, qaopsorg.SaveRequest) (qaopsorg.State, error) {
|
||||
s.calls++
|
||||
return qaopsorg.State{}, s.err
|
||||
}
|
||||
func (s *orgSpy) SaveRules(_ context.Context, _ qaopsorg.SaveRequest, actor string) (qaopsorg.State, error) {
|
||||
s.calls++
|
||||
s.actor = actor
|
||||
return qaopsorg.State{}, s.err
|
||||
}
|
||||
func TestQAOrganizationGuardsIdentityIntentAndErrors(t *testing.T) {
|
||||
spy := &orgSpy{}
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["org-admin"] = "usr-boss"
|
||||
auth.sessions["org-viewer"] = "usr-eye"
|
||||
handler, err := NewHandler(Config{Store: &fakeJobStore{}, QAOrg: spy, Auth: &AuthConfig{Store: auth}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
base := "/api/v1/qa-ops/organization"
|
||||
for _, tc := range []struct {
|
||||
method, path, token, action, origin, body string
|
||||
status int
|
||||
}{
|
||||
{"GET", base, "", "", "", "", 401},
|
||||
{"GET", base, "org-viewer", "", "", "", 403},
|
||||
{"GET", base, "org-admin", "", "", "", 200},
|
||||
{"GET", base + "?tenant=another", "org-admin", "", "", "", 400},
|
||||
{"PUT", base + "/regions", "org-admin", "", "", "{}", 403},
|
||||
{"PUT", base + "/regions", "org-admin", "qa-organization", "https://untrusted.invalid", "{}", 403},
|
||||
{"PUT", base + "/regions", "org-admin", "qa-organization", "", `{"updated_by":"forged"}`, 400},
|
||||
{"PUT", base + "/regions", "org-admin", "qa-organization", "", `{} {}`, 400},
|
||||
{"PUT", base + "/regions", "org-admin", "qa-organization", "", `{}`, 400},
|
||||
{"PUT", base + "/regions", "org-admin", "qa-organization", "", `{"directory_id":"d","base_revision":"","rules":null}`, 400},
|
||||
{"PUT", base + "/regions", "org-admin", "qa-organization", "", `{"directory_id":"d","base_revision":"","rules":[]}`, 200},
|
||||
{"POST", base + "/preview", "org-admin", "qa-organization", "", `{"directory_id":"d","base_revision":"","rules":[]}`, 200},
|
||||
} {
|
||||
req := httptest.NewRequest(tc.method, tc.path, strings.NewReader(tc.body))
|
||||
if tc.token != "" {
|
||||
req.AddCookie(&http.Cookie{Name: sessionCookieName, Value: tc.token})
|
||||
}
|
||||
req.Header.Set("X-Governance-Action", tc.action)
|
||||
req.Header.Set("Origin", tc.origin)
|
||||
res := httptest.NewRecorder()
|
||||
handler.ServeHTTP(res, req)
|
||||
if res.Code != tc.status {
|
||||
t.Fatalf("%s %s got %d want %d: %s", tc.method, tc.path, res.Code, tc.status, res.Body.String())
|
||||
}
|
||||
}
|
||||
if spy.calls != 3 || spy.actor == "" || spy.actor == "forged" {
|
||||
t.Fatalf("invalid requests reached store or actor not stamped: %+v", spy)
|
||||
}
|
||||
spy.err = qaopsorg.ErrConflict
|
||||
req := httptest.NewRequest("PUT", base+"/regions", strings.NewReader(`{"directory_id":"d","base_revision":"","rules":[]}`))
|
||||
req.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "org-admin"})
|
||||
req.Header.Set("X-Governance-Action", "qa-organization")
|
||||
res := httptest.NewRecorder()
|
||||
handler.ServeHTTP(res, req)
|
||||
if res.Code != 409 {
|
||||
t.Fatal("stale configuration not reported as conflict")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"regexp"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// HTTP LAN/tailnet origins may omit Fetch Metadata. nginx must preserve the
|
||||
// external authority including its port so Origin can be checked without it.
|
||||
func TestQAProxyPreservesBrowserAuthority(t *testing.T) {
|
||||
config, err := os.ReadFile("../../deploy/nginx.conf")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hosts := regexp.MustCompile(`(?m)^\s*proxy_set_header\s+Host\s+([^;]+);`).FindAllStringSubmatch(string(config), -1)
|
||||
if len(hosts) < 2 {
|
||||
t.Fatal("API and health proxy Host directives must both be explicit")
|
||||
}
|
||||
for _, host := range hosts {
|
||||
if host[1] != "$http_host" {
|
||||
t.Fatalf("proxy drops browser authority/port: got %q, want $http_host", host[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestQAIntentWithoutFetchMetadataChecksExactAuthority(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name, host, origin, forwarded, site, intent string
|
||||
allowed bool
|
||||
}{
|
||||
{"LAN same authority", "192.168.50.134:8088", "http://192.168.50.134:8088", "", "", "qa-quality", true},
|
||||
{"tailnet same authority", "100.92.49.78:8088", "http://100.92.49.78:8088", "", "", "qa-quality", true},
|
||||
{"proxy stripped port", "192.168.50.134", "http://192.168.50.134:8088", "", "", "qa-quality", false},
|
||||
{"foreign origin", "192.168.50.134:8088", "http://foreign.invalid:8088", "", "", "qa-quality", false},
|
||||
{"different port", "192.168.50.134:8088", "http://192.168.50.134:8089", "", "", "qa-quality", false},
|
||||
{"forwarded header cannot grant", "192.168.50.134:8088", "http://foreign.invalid:8088", "foreign.invalid:8088", "", "qa-quality", false},
|
||||
{"opaque origin", "192.168.50.134:8088", "null", "", "", "qa-quality", false},
|
||||
{"explicit cross site", "192.168.50.134:8088", "http://192.168.50.134:8088", "", "cross-site", "qa-quality", false},
|
||||
{"missing intent", "192.168.50.134:8088", "http://192.168.50.134:8088", "", "", "", false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest("POST", "http://"+tc.host+"/api/v1/qa-ops/quality/query", nil)
|
||||
req.Header.Set("Origin", tc.origin)
|
||||
req.Header.Set("X-Governance-Action", tc.intent)
|
||||
if tc.forwarded != "" {
|
||||
req.Header.Set("X-Forwarded-Host", tc.forwarded)
|
||||
}
|
||||
if tc.site != "" {
|
||||
req.Header.Set("Sec-Fetch-Site", tc.site)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
if got := authorizeQAIntent(rec, req, "qa-quality"); got != tc.allowed {
|
||||
t.Fatalf("authorized=%v want %v, HTTP=%d", got, tc.allowed, rec.Code)
|
||||
}
|
||||
if !tc.allowed && rec.Code != 403 {
|
||||
t.Fatalf("rejected authority should be 403, got %d", rec.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/qaopsquality"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type QAQualityStore interface {
|
||||
ObserveKeys(context.Context, []dobotaiops.QualityKey) (map[string]qaopsquality.View, error)
|
||||
Observe(context.Context, []dobotaiops.Record) (map[string]qaopsquality.View, error)
|
||||
Save(context.Context, dobotaiops.Record, qaopsquality.SaveRequest, string) (qaopsquality.Receipt, error)
|
||||
Columns(context.Context, []dobotaiops.QualityKey) ([]string, error)
|
||||
History(context.Context, string, int64, int64) (qaopsquality.History, error)
|
||||
Snapshot(context.Context, string, int64) (qaopsquality.Snapshot, error)
|
||||
}
|
||||
|
||||
type qaQualityReader interface {
|
||||
ReadQualityScope(context.Context, dobotaiops.RecordQuery) (dobotaiops.QualityScope, error)
|
||||
}
|
||||
|
||||
func writeQualityError(w http.ResponseWriter, err error) {
|
||||
status, code, message := 503, "quality_unavailable", "质量留档/复核存储不可用，未将失败伪装为空值或零统计"
|
||||
switch {
|
||||
case errors.Is(err, qaopsquality.ErrInvalid) || errors.Is(err, dobotaiops.ErrInvalidQuery):
|
||||
status, code, message = 400, "invalid_quality_request", "质量操作参数无效：请核对结论、理由、版本和筛选范围（最多5000条/32MiB）"
|
||||
case errors.Is(err, qaopsquality.ErrConflict):
|
||||
status, code, message = 409, "quality_conflict", "原始问答或人工复核版本已变化，请重新读取问答后核对；未覆盖现有结论"
|
||||
case errors.Is(err, qaopsquality.ErrSnapshotMissing):
|
||||
status, code, message = 409, "quality_input_metadata_only", "此版本仅留存指纹与复检元数据，未采集原文；未编造空问题或空答案"
|
||||
case errors.Is(err, qaopsquality.ErrNotFound):
|
||||
status, code, message = 404, "quality_input_not_found", "未找到当前授权范围的输入版本"
|
||||
}
|
||||
writeJSON(w, status, map[string]string{"error": message, "code": code})
|
||||
}
|
||||
func enrichQAQualityRecord(ctx context.Context, c Config, item *dobotaiops.Record) error {
|
||||
if c.QAQuality == nil {
|
||||
return nil
|
||||
}
|
||||
views, err := c.QAQuality.Observe(ctx, []dobotaiops.Record{*item})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v, ok := views[item.ID]
|
||||
if !ok {
|
||||
return qaopsquality.ErrUnavailable
|
||||
}
|
||||
item.Quality, err = json.Marshal(v)
|
||||
return err
|
||||
}
|
||||
func enrichQAQualityPage(ctx context.Context, c Config, page *dobotaiops.RecordPage) error {
|
||||
if c.QAQuality == nil {
|
||||
return nil
|
||||
}
|
||||
views, err := c.QAQuality.Observe(ctx, page.Items)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range page.Items {
|
||||
v, ok := views[page.Items[i].ID]
|
||||
if !ok {
|
||||
return qaopsquality.ErrUnavailable
|
||||
}
|
||||
page.Items[i].Quality, err = json.Marshal(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
columns, err := c.QAQuality.Columns(ctx, page.ScopeQuality)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
dobotaiops.AddRecordColumns(page, columns...)
|
||||
return nil
|
||||
}
|
||||
func walkQualityJSON(d *json.Decoder) error {
|
||||
token, err := d.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if token == nil {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
if delimiter, ok := token.(json.Delim); ok {
|
||||
if delimiter != '{' && delimiter != '[' {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
keys := map[string]bool{}
|
||||
for d.More() {
|
||||
if delimiter == '{' {
|
||||
key, e := d.Token()
|
||||
name, ok := key.(string)
|
||||
if e != nil || !ok || keys[name] {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
keys[name] = true
|
||||
}
|
||||
if walkQualityJSON(d) != nil {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
}
|
||||
_, err = d.Token()
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func decodeQualityJSON(w http.ResponseWriter, r *http.Request, target any, required ...string) error {
|
||||
if r.URL.RawQuery != "" {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
raw, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 64<<10))
|
||||
if err != nil || bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
var top map[string]json.RawMessage
|
||||
if json.Unmarshal(raw, &top) != nil || top == nil {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
for _, key := range required {
|
||||
if _, ok := top[key]; !ok {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
}
|
||||
check := json.NewDecoder(bytes.NewReader(raw))
|
||||
if walkQualityJSON(check) != nil {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
var extra any
|
||||
if check.Decode(&extra) != io.EOF {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
d := json.NewDecoder(bytes.NewReader(raw))
|
||||
d.DisallowUnknownFields()
|
||||
if d.Decode(target) != nil || d.Decode(&extra) != io.EOF {
|
||||
return qaopsquality.ErrInvalid
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func qaQualityReview(w http.ResponseWriter, r *http.Request, c Config) {
|
||||
if !authorizeQAOps(w, r) || !authorizeQAIntent(w, r, "qa-quality") {
|
||||
return
|
||||
}
|
||||
var request qaopsquality.SaveRequest
|
||||
if decodeQualityJSON(w, r, &request, "input_hash", "input_epoch", "base_revision", "nonce", "fields", "reason") != nil || qaopsquality.Validate(request) != nil {
|
||||
writeQualityError(w, qaopsquality.ErrInvalid)
|
||||
return
|
||||
}
|
||||
if c.QAQuality == nil || c.QAOps == nil || c.QAOpsTenantKey == "" {
|
||||
writeQualityError(w, qaopsquality.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
record, err := c.QAOps.GetRecord(r.Context(), c.QAOpsTenantKey, r.PathValue("id"))
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
identity, _ := sessionFrom(r.Context())
|
||||
receipt, err := c.QAQuality.Save(r.Context(), record, request, identity.Username)
|
||||
if err != nil {
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": receipt})
|
||||
}
|
||||
func qaQualityQuery(w http.ResponseWriter, r *http.Request, c Config) {
|
||||
if !authorizeQAOps(w, r) || !authorizeQAIntent(w, r, "qa-quality") {
|
||||
return
|
||||
}
|
||||
var request struct {
|
||||
Start string `json:"start"`
|
||||
End string `json:"end"`
|
||||
Keyword string `json:"keyword"`
|
||||
}
|
||||
if decodeQualityJSON(w, r, &request) != nil {
|
||||
writeQualityError(w, qaopsquality.ErrInvalid)
|
||||
return
|
||||
}
|
||||
window, err := dobotaiops.NewWindow(c.QAOpsTenantKey, request.Start, request.End, 90*24*time.Hour, c.Now())
|
||||
if err != nil {
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
reader, ok := c.QAOps.(qaQualityReader)
|
||||
if !ok || c.QAQuality == nil {
|
||||
writeQualityError(w, qaopsquality.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
scope, err := reader.ReadQualityScope(r.Context(), dobotaiops.RecordQuery{Window: window, Keyword: request.Keyword})
|
||||
if err != nil {
|
||||
qaOpsError(w, err)
|
||||
return
|
||||
}
|
||||
views, err := c.QAQuality.ObserveKeys(r.Context(), scope.Keys)
|
||||
if err != nil || len(views) != len(scope.Keys) {
|
||||
if err == nil {
|
||||
err = qaopsquality.ErrUnavailable
|
||||
}
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": map[string]any{"summary": qaopsquality.Summarize(views), "start": window.Start, "end": window.End, "keyword": request.Keyword, "as_of": scope.AsOf, "ai_state": "not_configured", "scope": "whole_filter"}})
|
||||
}
|
||||
func qaQualityHistory(w http.ResponseWriter, r *http.Request, c Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
if c.QAQuality == nil {
|
||||
writeQualityError(w, qaopsquality.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
values := r.URL.Query()
|
||||
for key, v := range values {
|
||||
if (key != "review_before" && key != "check_before") || len(v) != 1 {
|
||||
writeQualityError(w, qaopsquality.ErrInvalid)
|
||||
return
|
||||
}
|
||||
}
|
||||
cursors := map[string]int64{}
|
||||
for _, key := range []string{"review_before", "check_before"} {
|
||||
raw := values.Get(key)
|
||||
if raw == "" {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.ParseInt(raw, 10, 64)
|
||||
if err != nil || n < 1 {
|
||||
writeQualityError(w, qaopsquality.ErrInvalid)
|
||||
return
|
||||
}
|
||||
cursors[key] = n
|
||||
}
|
||||
value, err := c.QAQuality.History(r.Context(), r.PathValue("id"), cursors["review_before"], cursors["check_before"])
|
||||
if err != nil {
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": value})
|
||||
}
|
||||
func qaQualitySnapshot(w http.ResponseWriter, r *http.Request, c Config) {
|
||||
if !authorizeQAOps(w, r) {
|
||||
return
|
||||
}
|
||||
epoch, err := strconv.ParseInt(r.PathValue("epoch"), 10, 64)
|
||||
if err != nil || epoch < 1 || r.URL.RawQuery != "" {
|
||||
writeQualityError(w, qaopsquality.ErrInvalid)
|
||||
return
|
||||
}
|
||||
if c.QAQuality == nil {
|
||||
writeQualityError(w, qaopsquality.ErrUnavailable)
|
||||
return
|
||||
}
|
||||
value, err := c.QAQuality.Snapshot(r.Context(), r.PathValue("id"), epoch)
|
||||
if err != nil {
|
||||
writeQualityError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, 200, map[string]any{"data": value})
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"knowledge-governance/internal/qaopsquality"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type qualityFake struct {
|
||||
actor string
|
||||
calls int
|
||||
fail bool
|
||||
}
|
||||
|
||||
func (s *qualityFake) Observe(_ context.Context, records []dobotaiops.Record) (map[string]qaopsquality.View, error) {
|
||||
s.calls++
|
||||
if s.fail {
|
||||
return nil, qaopsquality.ErrUnavailable
|
||||
}
|
||||
out := map[string]qaopsquality.View{}
|
||||
for _, r := range records {
|
||||
e, _ := qaopsquality.EvidenceFor(r)
|
||||
out[r.ID] = qaopsquality.Project(e.Hash, 1, 0, qaopsquality.Check{InputEpoch: 1, InputHash: e.Hash, Result: dobotaiops.CheckAnswerCoverage(r.Answer)}, nil)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (s *qualityFake) ObserveKeys(_ context.Context, keys []dobotaiops.QualityKey) (map[string]qaopsquality.View, error) {
|
||||
s.calls++
|
||||
out := map[string]qaopsquality.View{}
|
||||
for _, key := range keys {
|
||||
out[key.ID] = qaopsquality.Project(key.InputHash, 1, 0, qaopsquality.Check{Result: dobotaiops.CheckMarkerCoverage(key.Marker)}, nil)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (s *qualityFake) Save(_ context.Context, r dobotaiops.Record, q qaopsquality.SaveRequest, actor string) (qaopsquality.Receipt, error) {
|
||||
s.actor = actor
|
||||
v := qaopsquality.Project(q.InputHash, 1, 1, qaopsquality.Check{Result: dobotaiops.CheckAnswerCoverage(r.Answer)}, &qaopsquality.Review{InputEpoch: 1, InputHash: q.InputHash, Fields: q.Fields, Reason: q.Reason, Reviewer: actor})
|
||||
return qaopsquality.Receipt{View: v, SavedRevision: 1}, nil
|
||||
}
|
||||
func (s *qualityFake) Columns(_ context.Context, keys []dobotaiops.QualityKey) ([]string, error) {
|
||||
return []string{"is_correct", "product_category"}, nil
|
||||
}
|
||||
func (s *qualityFake) History(context.Context, string, int64, int64) (qaopsquality.History, error) {
|
||||
return qaopsquality.History{Reviews: []qaopsquality.Review{}, Checks: []qaopsquality.Check{}}, nil
|
||||
}
|
||||
func (s *qualityFake) Snapshot(context.Context, string, int64) (qaopsquality.Snapshot, error) {
|
||||
return qaopsquality.Snapshot{}, qaopsquality.ErrNotFound
|
||||
}
|
||||
|
||||
type qualitySource struct {
|
||||
attributedReader
|
||||
record dobotaiops.Record
|
||||
tenant string
|
||||
scopeCalls int
|
||||
}
|
||||
|
||||
func (s *qualitySource) GetRecord(_ context.Context, tenant, id string) (dobotaiops.Record, error) {
|
||||
s.tenant = tenant
|
||||
return s.record, nil
|
||||
}
|
||||
func (s *qualitySource) ListRecords(_ context.Context, q dobotaiops.RecordQuery) (dobotaiops.RecordPage, error) {
|
||||
return dobotaiops.RecordPage{Items: []dobotaiops.Record{s.record}, Columns: []string{"question"}, Total: 2, AsOf: time.Now()}, nil
|
||||
}
|
||||
func (s *qualitySource) ReadQualityScope(_ context.Context, q dobotaiops.RecordQuery) (dobotaiops.QualityScope, error) {
|
||||
s.scopeCalls++
|
||||
s.tenant = q.Window.TenantKey
|
||||
second := s.record
|
||||
second.ID = "second-turn"
|
||||
keys := []dobotaiops.QualityKey{}
|
||||
for _, r := range []dobotaiops.Record{s.record, second} {
|
||||
e, _ := qaopsquality.EvidenceFor(r)
|
||||
keys = append(keys, dobotaiops.QualityKey{ID: r.ID, InputHash: e.Hash, Marker: e.Result.Verdict == "uncovered", Source: r.SourceCoverage})
|
||||
}
|
||||
return dobotaiops.QualityScope{Keys: keys, AsOf: time.Now()}, nil
|
||||
}
|
||||
func qualityTestHandler(t *testing.T) (http.Handler, *qualityFake, *qualitySource) {
|
||||
t.Helper()
|
||||
store := &qualityFake{}
|
||||
source := &qualitySource{record: dobotaiops.Record{ID: "fixture-turn", Status: "succeeded", Question: "示例", Answer: "暂无已发布"}}
|
||||
auth := newFakeAuthStore()
|
||||
auth.sessions["quality-admin"] = "usr-boss"
|
||||
auth.sessions["quality-viewer"] = "usr-eye"
|
||||
h, e := NewHandler(Config{Store: &fakeJobStore{}, Auth: &AuthConfig{Store: auth}, QAOps: source, QAOpsTenantKey: "fixed", QAQuality: store, Now: func() time.Time { return time.Date(2026, 10, 17, 0, 0, 0, 0, time.UTC) }})
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
return h, store, source
|
||||
}
|
||||
func qualityRequest(h http.Handler, method, path, token, body string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
r.Header.Set("X-Governance-Action", "qa-quality")
|
||||
if token != "" {
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: token})
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
func TestQAQualityDetailListAndWholeScopeCounts(t *testing.T) {
|
||||
h, store, source := qualityTestHandler(t)
|
||||
w := qualityRequest(h, "GET", "/api/v1/qa-ops/records/fixture-turn", "quality-admin", "")
|
||||
if w.Code != 200 || !strings.Contains(w.Body.String(), `"quality"`) {
|
||||
t.Fatal("persisted quality absent in detail", w.Code)
|
||||
}
|
||||
w = qualityRequest(h, "GET", "/api/v1/qa-ops/records?start=2026-10-14T00:00:00Z&end=2026-10-15T00:00:00Z", "quality-admin", "")
|
||||
if w.Code != 200 || !strings.Contains(w.Body.String(), `"is_correct"`) {
|
||||
t.Fatal("whole-filter quality columns absent")
|
||||
}
|
||||
w = qualityRequest(h, "POST", "/api/v1/qa-ops/quality/query", "quality-admin", `{"start":"2026-10-14T00:00:00Z","end":"2026-10-15T00:00:00Z","keyword":""}`)
|
||||
if w.Code != 200 || !strings.Contains(w.Body.String(), `"unreviewed":2`) || source.scopeCalls != 1 || source.tenant != "fixed" || store.calls < 2 {
|
||||
t.Fatal("quality summary estimated from page instead of scope", w.Code, w.Body.String())
|
||||
}
|
||||
var counts struct {
|
||||
Data struct {
|
||||
Summary qaopsquality.Summary `json:"summary"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if json.Unmarshal(w.Body.Bytes(), &counts) != nil || counts.Data.Summary.Rates.Coverage.Denominator != 2 || counts.Data.Summary.Rates.Coverage.Numerator != 0 || counts.Data.Summary.Rates.Correctness.Percent != nil {
|
||||
t.Fatal("confirmed quality metrics were not computed for the whole scope")
|
||||
}
|
||||
store.fail = true
|
||||
w = qualityRequest(h, "GET", "/api/v1/qa-ops/records/fixture-turn", "quality-admin", "")
|
||||
if w.Code != 503 || !strings.Contains(w.Body.String(), "quality_unavailable") {
|
||||
t.Fatal("quality storage failure presented as empty success")
|
||||
}
|
||||
}
|
||||
func TestQAQualityReviewUsesSessionActorAndStrictReadOnlySourceBoundary(t *testing.T) {
|
||||
h, store, source := qualityTestHandler(t)
|
||||
e, _ := qaopsquality.EvidenceFor(source.record)
|
||||
q := qaopsquality.SaveRequest{InputHash: e.Hash, InputEpoch: 1, Nonce: strings.Repeat("n", 32), Fields: qaopsquality.Fields{Coverage: "uncovered", Correctness: "pending"}, Reason: "人工核对"}
|
||||
raw, _ := json.Marshal(q)
|
||||
path := "/api/v1/qa-ops/records/fixture-turn/reviews"
|
||||
w := qualityRequest(h, "POST", path, "quality-admin", string(raw))
|
||||
if w.Code != 200 || store.actor != "boss" || source.tenant != "fixed" {
|
||||
t.Fatal("manual review failed or actor came from request", w.Code, store.actor)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
token, body string
|
||||
status int
|
||||
}{{"", string(raw), 401}, {"quality-viewer", string(raw), 403}, {"quality-admin", strings.TrimSuffix(string(raw), "}") + `,"reviewer":"forged"}`, 400}, {"quality-admin", `null`, 400}, {"quality-admin", string(raw) + `{}`, 400}, {"quality-admin", `{"nonce":"a","nonce":"b"}`, 400}} {
|
||||
w = qualityRequest(h, "POST", path, tc.token, tc.body)
|
||||
if w.Code != tc.status {
|
||||
t.Fatal("review boundary failed", w.Code, tc.status)
|
||||
}
|
||||
}
|
||||
r := httptest.NewRequest("POST", path, strings.NewReader(string(raw)))
|
||||
r.AddCookie(&http.Cookie{Name: sessionCookieName, Value: "quality-admin"})
|
||||
r.Header.Set("X-Governance-Action", "qa-quality")
|
||||
r.Header.Set("Sec-Fetch-Site", "cross-site")
|
||||
w = httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != 403 {
|
||||
t.Fatal("cross-site review accepted")
|
||||
}
|
||||
w = qualityRequest(h, "GET", path+"/invalid", "quality-admin", "")
|
||||
if w.Code != 404 {
|
||||
t.Fatal("unknown endpoint silently accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQAQualityReviewRejectsMissingRevisionAndNestedNull(t *testing.T) {
|
||||
h, _, source := qualityTestHandler(t)
|
||||
e, _ := qaopsquality.EvidenceFor(source.record)
|
||||
q := qaopsquality.SaveRequest{InputHash: e.Hash, InputEpoch: 1, Nonce: strings.Repeat("n", 32), Fields: qaopsquality.Fields{Coverage: "pending", Correctness: "pending"}, Reason: "示例复核"}
|
||||
raw, _ := json.Marshal(q)
|
||||
for _, body := range []string{strings.Replace(string(raw), `"base_revision":0,`, "", 1), strings.Replace(string(raw), `"base_revision":0`, `"base_revision":null`, 1), strings.Replace(string(raw), `"product_category":""`, `"product_category":null`, 1)} {
|
||||
w := qualityRequest(h, "POST", "/api/v1/qa-ops/records/fixture-turn/reviews", "quality-admin", body)
|
||||
if w.Code != 400 {
|
||||
t.Fatal("missing/null version or field silently became a default", w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"knowledge-governance/internal/jobs"
|
||||
"knowledge-governance/internal/monitoring"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
)
|
||||
|
||||
type JobStore interface {
|
||||
@@ -101,6 +102,14 @@ type Store interface {
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
QAQuality QAQualityStore
|
||||
QAOrgSync QAOrgSyncStore
|
||||
QAOrgSyncEnabled bool
|
||||
QAOrg QAOrgStore
|
||||
QAOps QAOpsReader
|
||||
QAOpsTenantKey string
|
||||
QAOpsEnabled bool
|
||||
QAOpsProblem string
|
||||
Store Store
|
||||
Monitor MonitorProvider
|
||||
InternalRoutingKnowledgeBaseID string
|
||||
@@ -167,6 +176,27 @@ func NewHandler(config Config) (http.Handler, error) {
|
||||
config.NewID = newID
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("POST /api/v1/qa-ops/quality/query", func(w http.ResponseWriter, r *http.Request) { qaQualityQuery(w, r, config) })
|
||||
mux.HandleFunc("POST /api/v1/qa-ops/records/{id}/reviews", func(w http.ResponseWriter, r *http.Request) { qaQualityReview(w, r, config) })
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/records/{id}/quality/history", func(w http.ResponseWriter, r *http.Request) { qaQualityHistory(w, r, config) })
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/records/{id}/quality/inputs/{epoch}", func(w http.ResponseWriter, r *http.Request) { qaQualitySnapshot(w, r, config) })
|
||||
|
||||
mux.HandleFunc("POST /api/v1/qa-ops/usage/people", func(w http.ResponseWriter, r *http.Request) { qaMonthlyPeople(w, r, config) })
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/usage", func(w http.ResponseWriter, r *http.Request) { qaOpsUsage(w, r, config) })
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/organization/sync", func(w http.ResponseWriter, r *http.Request) { qaOrganizationSync(w, r, config) })
|
||||
mux.HandleFunc("POST /api/v1/qa-ops/organization/sync", func(w http.ResponseWriter, r *http.Request) { qaOrganizationSync(w, r, config) })
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/organization", func(w http.ResponseWriter, r *http.Request) { qaOrganization(w, r, config) })
|
||||
mux.HandleFunc("POST /api/v1/qa-ops/organization/preview", func(w http.ResponseWriter, r *http.Request) { qaOrganization(w, r, config) })
|
||||
mux.HandleFunc("PUT /api/v1/qa-ops/organization/regions", func(w http.ResponseWriter, r *http.Request) { qaOrganization(w, r, config) })
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/status", func(w http.ResponseWriter, r *http.Request) {
|
||||
qaOpsStatus(w, r, config)
|
||||
})
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/records", func(w http.ResponseWriter, r *http.Request) {
|
||||
listQAOpsRecords(w, r, config)
|
||||
})
|
||||
mux.HandleFunc("GET /api/v1/qa-ops/records/{id}", func(w http.ResponseWriter, r *http.Request) {
|
||||
getQAOpsRecord(w, r, config)
|
||||
})
|
||||
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
})
|
||||
@@ -539,6 +569,10 @@ func createJob(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON request"})
|
||||
return
|
||||
}
|
||||
if strings.HasPrefix(input.Type, qaopsorg.SyncJobPrefix) {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "通讯录同步任务只能从问答运营的管理员入口创建"})
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(input.Type) == "" || len(input.Type) > 100 {
|
||||
writeJSON(w, http.StatusBadRequest, map[string]string{"error": "type is required and must not exceed 100 characters"})
|
||||
return
|
||||
|
||||
@@ -225,10 +225,26 @@ func versionHistorySync(w http.ResponseWriter, r *http.Request, config Config) {
|
||||
versionHistoryError(w, http.StatusInternalServerError, "version_history_unavailable", "构造同步任务失败")
|
||||
return
|
||||
}
|
||||
if queue, ok := config.Store.(interface {
|
||||
EnqueueVersionHistorySync(context.Context, string, string, string, time.Time) (*jobs.Job, error)
|
||||
}); ok {
|
||||
target := ""
|
||||
info := version.Snapshot()
|
||||
if info.IdentityAvailable && info.CommitSHA != nil {
|
||||
target = *info.CommitSHA
|
||||
}
|
||||
job, e := queue.EnqueueVersionHistorySync(r.Context(), config.VersionHistoryRepo, triggeredBy, target, now)
|
||||
if e != nil || job == nil {
|
||||
versionHistoryError(w, 500, "version_history_unavailable", "入队同步任务失败")
|
||||
return
|
||||
}
|
||||
writeJSON(w, 202, map[string]any{"job_id": job.ID, "status": job.State})
|
||||
return
|
||||
}
|
||||
requestedID := fmt.Sprintf("version-history-sync-%d", now.UnixNano())
|
||||
job, err := config.Store.Enqueue(r.Context(), jobs.EnqueueRequest{ID: requestedID,
|
||||
Type: VersionHistoryJobType, Payload: payload, AvailableAt: now, MaxAttempts: 3,
|
||||
IdempotencyKey: "version-history.sync:" + config.VersionHistoryRepo}, now)
|
||||
IdempotencyKey: "version-history.sync:" + config.VersionHistoryRepo + ":" + requestedID}, now)
|
||||
if err != nil || job == nil {
|
||||
versionHistoryError(w, http.StatusInternalServerError, "version_history_unavailable", "入队同步任务失败")
|
||||
return
|
||||
|
||||
@@ -237,7 +237,7 @@ func TestVersionHistorySyncRequiresAdminAndEnqueuesOnce(t *testing.T) {
|
||||
if recorder.Code != http.StatusAccepted {
|
||||
t.Fatalf("首次同步应 202，得到 %d（%s）", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
if queue.enqueued.Type != VersionHistoryJobType || queue.enqueued.IdempotencyKey != "version-history.sync:Weknora/knowledge-governance" {
|
||||
if queue.enqueued.Type != VersionHistoryJobType || queue.enqueued.IdempotencyKey != "version-history.sync:Weknora/knowledge-governance:"+queue.enqueued.ID {
|
||||
t.Fatalf("入队参数不对：%+v", queue.enqueued)
|
||||
}
|
||||
if queue.enqueued.MaxAttempts != 3 || queue.enqueued.AvailableAt != now {
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
DROP TABLE IF EXISTS qa_ops_organization_state;
|
||||
DROP TABLE IF EXISTS qa_ops_region_revisions;
|
||||
DROP TABLE IF EXISTS qa_ops_directory_snapshots;
|
||||
@@ -0,0 +1,26 @@
|
||||
CREATE TABLE qa_ops_directory_snapshots (
|
||||
tenant_key TEXT NOT NULL,
|
||||
id TEXT NOT NULL,
|
||||
source TEXT NOT NULL CHECK (source IN ('weekly_report_snapshot','weekly_report_live')),
|
||||
synced_at TIMESTAMPTZ NOT NULL,
|
||||
imported_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
payload JSONB NOT NULL CHECK (jsonb_typeof(payload)='object'),
|
||||
PRIMARY KEY (tenant_key,id)
|
||||
);
|
||||
CREATE TABLE qa_ops_region_revisions (
|
||||
tenant_key TEXT NOT NULL,
|
||||
id TEXT NOT NULL,
|
||||
directory_id TEXT NOT NULL,
|
||||
rules JSONB NOT NULL CHECK (jsonb_typeof(rules)='array'),
|
||||
updated_by TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (tenant_key,id),
|
||||
FOREIGN KEY (tenant_key,directory_id) REFERENCES qa_ops_directory_snapshots(tenant_key,id)
|
||||
);
|
||||
CREATE TABLE qa_ops_organization_state (
|
||||
tenant_key TEXT PRIMARY KEY,
|
||||
directory_id TEXT NOT NULL,
|
||||
revision_id TEXT,
|
||||
FOREIGN KEY (tenant_key,directory_id) REFERENCES qa_ops_directory_snapshots(tenant_key,id),
|
||||
FOREIGN KEY (tenant_key,revision_id) REFERENCES qa_ops_region_revisions(tenant_key,id)
|
||||
);
|
||||
@@ -0,0 +1,4 @@
|
||||
DROP TABLE qa_ops_quality_reviews;
|
||||
DROP TABLE qa_ops_quality_checks;
|
||||
DROP TABLE qa_ops_quality_inputs;
|
||||
DROP TABLE qa_ops_quality_records;
|
||||
@@ -0,0 +1,13 @@
|
||||
CREATE TABLE qa_ops_quality_records (
|
||||
tenant_key TEXT NOT NULL,record_id TEXT NOT NULL,observed_at TIMESTAMPTZ,input_hash TEXT NOT NULL DEFAULT '',input_epoch BIGINT NOT NULL DEFAULT 0 CHECK(input_epoch>=0),review_revision BIGINT NOT NULL DEFAULT 0 CHECK(review_revision>=0),PRIMARY KEY(tenant_key,record_id)
|
||||
);
|
||||
CREATE TABLE qa_ops_quality_inputs (
|
||||
tenant_key TEXT NOT NULL,record_id TEXT NOT NULL,input_epoch BIGINT NOT NULL CHECK(input_epoch>0),input_hash TEXT NOT NULL CHECK(input_hash ~ '^[a-f0-9]{64}$'),payload JSONB CHECK(payload IS NULL OR jsonb_typeof(payload)='object'),created_at TIMESTAMPTZ NOT NULL DEFAULT now(),PRIMARY KEY(tenant_key,record_id,input_epoch),FOREIGN KEY(tenant_key,record_id) REFERENCES qa_ops_quality_records(tenant_key,record_id)
|
||||
);
|
||||
CREATE TABLE qa_ops_quality_checks (
|
||||
id BIGSERIAL PRIMARY KEY,tenant_key TEXT NOT NULL,record_id TEXT NOT NULL,input_epoch BIGINT NOT NULL,rule_version TEXT NOT NULL,result JSONB NOT NULL CHECK(jsonb_typeof(result)='object'),source JSONB NOT NULL CHECK(jsonb_typeof(source)='object'),created_at TIMESTAMPTZ NOT NULL DEFAULT now(),UNIQUE(tenant_key,record_id,input_epoch,rule_version),FOREIGN KEY(tenant_key,record_id,input_epoch) REFERENCES qa_ops_quality_inputs(tenant_key,record_id,input_epoch)
|
||||
);
|
||||
CREATE TABLE qa_ops_quality_reviews (
|
||||
tenant_key TEXT NOT NULL,record_id TEXT NOT NULL,revision BIGINT NOT NULL CHECK(revision>0),input_epoch BIGINT NOT NULL,input_hash TEXT NOT NULL CHECK(input_hash ~ '^[a-f0-9]{64}$'),fields JSONB NOT NULL CHECK(jsonb_typeof(fields)='object'),reason TEXT NOT NULL CHECK(length(btrim(reason))>0),reviewer TEXT NOT NULL CHECK(length(btrim(reviewer))>0),nonce TEXT NOT NULL,request_hash TEXT NOT NULL,created_at TIMESTAMPTZ NOT NULL DEFAULT now(),PRIMARY KEY(tenant_key,record_id,revision),UNIQUE(tenant_key,record_id,nonce),FOREIGN KEY(tenant_key,record_id,input_epoch) REFERENCES qa_ops_quality_inputs(tenant_key,record_id,input_epoch)
|
||||
);
|
||||
CREATE INDEX idx_qa_quality_checks_history ON qa_ops_quality_checks(tenant_key,record_id,id DESC);
|
||||
@@ -11,8 +11,8 @@ func TestEmbeddedMigrationsAreVersionSorted(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("loadMigrations() error = %v", err)
|
||||
}
|
||||
if len(migrations) != 47 {
|
||||
t.Fatalf("migration count = %d, want 47", len(migrations))
|
||||
if len(migrations) != 49 {
|
||||
t.Fatalf("migration count = %d, want 49", len(migrations))
|
||||
}
|
||||
if migrations[0].Version != 1 || !strings.Contains(migrations[0].UpSQL, "CREATE TABLE IF NOT EXISTS jobs") {
|
||||
t.Fatalf("migration 1 = %#v", migrations[0])
|
||||
|
||||
@@ -20,6 +20,12 @@ var ErrOperationRequestConflict = errors.New("operation request conflicts with a
|
||||
|
||||
type operationExecutor func(pgx.Tx) (json.RawMessage, error)
|
||||
|
||||
// ExecuteOperation lets bounded modules share the durable operation transaction
|
||||
// instead of caching responses after their business writes have committed.
|
||||
func (s *Store) ExecuteOperation(ctx context.Context, operationType, operationID, fingerprint string, now time.Time, execute func(pgx.Tx) (json.RawMessage, error)) (json.RawMessage, bool, error) {
|
||||
return s.executeOperation(ctx, operationType, operationID, fingerprint, now, execute)
|
||||
}
|
||||
|
||||
func (s *Store) RouteCatalogFilesWithOperation(ctx context.Context, request governance.CatalogRouteRequest, operationID string) (governance.CatalogRouteResult, bool, error) {
|
||||
if err := validateCatalogRouteRequest(request); err != nil {
|
||||
return governance.CatalogRouteResult{}, false, err
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"knowledge-governance/internal/jobs"
|
||||
"knowledge-governance/internal/versionhistory"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func (s *Store) EnqueueVersionHistorySync(ctx context.Context, repo, actor, target string, now time.Time) (*jobs.Job, error) {
|
||||
if strings.Count(repo, "/") != 1 || strings.TrimSpace(repo) != repo || len(repo) > 200 || actor == "" || len(actor) > 256 || target != "" && !regexp.MustCompile(`^[a-f0-9]{40}$`).MatchString(target) {
|
||||
return nil, errors.New("invalid version synchronization request")
|
||||
}
|
||||
tx, e := s.db.Begin(ctx)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
if _, e = tx.Exec(ctx, `SELECT pg_advisory_xact_lock(hashtextextended($1,0))`, "version-history:"+repo); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
// Retire exhausted expired owners. Terminal jobs are history, not a perpetual
|
||||
// idempotency key: a later refresh must be allowed to execute again.
|
||||
if _, e = tx.Exec(ctx, `UPDATE jobs SET state='failed',lease_owner=NULL,lease_until=NULL,updated_at=$3 WHERE type=$1 AND payload->>'repo_key'=$2 AND state='running' AND lease_until<clock_timestamp() AND attempt>=max_attempts`, versionhistory.JobType, repo, now); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
var active string
|
||||
e = tx.QueryRow(ctx, `SELECT id FROM jobs WHERE type=$1 AND payload->>'repo_key'=$2 AND state IN ('pending','running','retry') ORDER BY created_at DESC,id DESC LIMIT 1`, versionhistory.JobType, repo).Scan(&active)
|
||||
if e != nil && !errors.Is(e, pgx.ErrNoRows) {
|
||||
return nil, e
|
||||
}
|
||||
if active != "" {
|
||||
job, e := scanJob(tx.QueryRow(ctx, `SELECT id,type,payload,state,priority,available_at,lease_owner,lease_until,attempt,max_attempts,last_error,idempotency_key,created_at,updated_at FROM jobs WHERE id=$1`, active))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if e = tx.Commit(ctx); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
return job, nil
|
||||
}
|
||||
var random [16]byte
|
||||
if _, e = rand.Read(random[:]); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
id := "vh-" + hex.EncodeToString(random[:])
|
||||
payload, _ := json.Marshal(map[string]string{"repo_key": repo, "triggered_by": actor, "target_sha": target})
|
||||
job, e := NewStore(tx).Enqueue(ctx, jobs.EnqueueRequest{ID: id, Type: versionhistory.JobType, Payload: payload, AvailableAt: now, MaxAttempts: 3, IdempotencyKey: "version-history:" + repo + ":" + id}, now)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if e = tx.Commit(ctx); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
return job, nil
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/versionhistory"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestVersionHistorySyncQueueCanRefreshAfterCompletion(t *testing.T) {
|
||||
dsn := os.Getenv("KG_VERSION_HISTORY_TEST_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("explicit disposable version-history DB required")
|
||||
}
|
||||
u, e := url.Parse(dsn)
|
||||
if e != nil || u.Hostname() != "127.0.0.1" || !strings.HasPrefix(u.Path, "/kg_version_history_test_") {
|
||||
t.Fatal("refusing shared database")
|
||||
}
|
||||
ctx := context.Background()
|
||||
p, e := pgxpool.New(ctx, dsn)
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
defer p.Close()
|
||||
if e = ApplyMigrations(ctx, p); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
s := NewStore(p)
|
||||
one, e := s.EnqueueVersionHistorySync(ctx, "fixture/repo", "fixture-admin", strings.Repeat("a", 40), time.Now())
|
||||
if e != nil || one == nil {
|
||||
t.Fatal("first sync missing", e)
|
||||
}
|
||||
same, e := s.EnqueueVersionHistorySync(ctx, "fixture/repo", "fixture-admin", strings.Repeat("a", 40), time.Now())
|
||||
if e != nil || same.ID != one.ID {
|
||||
t.Fatal("pending sync not coalesced", e)
|
||||
}
|
||||
claimed, e := s.Claim(ctx, "fixture-worker", []string{versionhistory.JobType}, time.Now(), time.Minute)
|
||||
if e != nil || claimed == nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if e = s.Complete(ctx, claimed.ID, claimed.LeaseOwner, time.Now()); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
next, e := s.EnqueueVersionHistorySync(ctx, "fixture/repo", "fixture-admin", strings.Repeat("b", 40), time.Now())
|
||||
if e != nil || next == nil || next.ID == one.ID || next.State != "pending" {
|
||||
t.Fatal("completed request was replayed forever instead of starting a new refresh", e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sort"
|
||||
)
|
||||
|
||||
type PersonAttribution struct {
|
||||
Name string `json:"name"`
|
||||
DepartmentIDs []string `json:"-"`
|
||||
State string `json:"state"`
|
||||
PersonID string `json:"-"`
|
||||
Departments []string `json:"departments"`
|
||||
Region string `json:"region"`
|
||||
}
|
||||
type Attribution struct {
|
||||
Metadata State
|
||||
people map[string]PersonAttribution
|
||||
unions map[string]string
|
||||
}
|
||||
|
||||
func (p *Attribution) Members() []PersonAttribution {
|
||||
result := []PersonAttribution{}
|
||||
if p == nil {
|
||||
return result
|
||||
}
|
||||
for _, person := range p.people {
|
||||
result = append(result, person)
|
||||
}
|
||||
sort.Slice(result, func(i, j int) bool { return result[i].PersonID < result[j].PersonID })
|
||||
return result
|
||||
}
|
||||
|
||||
func BuildAttribution(state State, directory Directory) (*Attribution, error) {
|
||||
p := &Attribution{Metadata: state, people: map[string]PersonAttribution{}, unions: map[string]string{}}
|
||||
if state.State == "empty" {
|
||||
return p, nil
|
||||
}
|
||||
if state.State != "ready" || state.DirectoryID == "" {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
view, err := Preview(directory, state.Rules)
|
||||
if err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
p.Metadata.View = view
|
||||
departments := map[string]DepartmentView{}
|
||||
for _, d := range view.Departments {
|
||||
departments[d.ID] = d
|
||||
}
|
||||
for _, person := range directory.People {
|
||||
paths, regions := map[string]bool{}, map[string]bool{}
|
||||
for _, id := range person.DepartmentIDs {
|
||||
d := departments[id]
|
||||
paths[d.Path] = true
|
||||
if d.Region != "" {
|
||||
regions[d.Region] = true
|
||||
}
|
||||
}
|
||||
item := PersonAttribution{Name: person.Name, DepartmentIDs: append([]string{}, person.DepartmentIDs...), State: "unassigned", PersonID: person.ID, Departments: []string{}}
|
||||
for path := range paths {
|
||||
item.Departments = append(item.Departments, path)
|
||||
}
|
||||
sort.Strings(item.Departments)
|
||||
if len(regions) == 1 {
|
||||
item.State = "matched"
|
||||
for region := range regions {
|
||||
item.Region = region
|
||||
}
|
||||
} else if len(regions) > 1 {
|
||||
item.State = "cross_region"
|
||||
}
|
||||
p.people[person.ID] = item
|
||||
if person.UnionID != "" {
|
||||
if !validID(person.UnionID) {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
if _, ok := p.unions[person.UnionID]; ok {
|
||||
p.unions[person.UnionID] = ""
|
||||
} else {
|
||||
p.unions[person.UnionID] = person.ID
|
||||
}
|
||||
}
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func (p *Attribution) Resolve(staff, union string) PersonAttribution {
|
||||
none := PersonAttribution{State: "unmatched", Departments: []string{}}
|
||||
if p == nil || p.Metadata.State != "ready" {
|
||||
none.State = "directory_missing"
|
||||
return none
|
||||
}
|
||||
if staff == "" && union == "" {
|
||||
none.State = "no_identity"
|
||||
return none
|
||||
}
|
||||
byUnion, unionKnown := p.unions[union]
|
||||
if person, ok := p.people[staff]; ok {
|
||||
if unionKnown && (byUnion == "" || byUnion != staff) {
|
||||
none.State = "identity_conflict"
|
||||
return none
|
||||
}
|
||||
return person
|
||||
}
|
||||
if unionKnown {
|
||||
if byUnion == "" {
|
||||
none.State = "identity_conflict"
|
||||
return none
|
||||
}
|
||||
return p.people[byUnion]
|
||||
}
|
||||
return none
|
||||
}
|
||||
func (s *Store) GetAttribution(ctx context.Context) (*Attribution, error) {
|
||||
if !s.available() {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
state, directory, err := s.read(ctx, s.pool, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return BuildAttribution(state, directory)
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAttributionUsesStableIdentityAndKeepsAmbiguitiesExplicit(t *testing.T) {
|
||||
d := Directory{Departments: []Department{{ID: "root", Name: "fixture"}, {ID: "a", ParentID: "root", Name: "one"}, {ID: "b", ParentID: "root", Name: "two"}}, People: []Person{{ID: "staff-a", UnionID: "union-a", DepartmentIDs: []string{"a", "a"}}, {ID: "staff-cross", UnionID: "union-cross", DepartmentIDs: []string{"a", "b"}}, {ID: "duplicate-one", UnionID: "duplicate", DepartmentIDs: []string{"a"}}, {ID: "duplicate-two", UnionID: "duplicate", DepartmentIDs: []string{"b"}}}}
|
||||
p, err := BuildAttribution(State{State: "ready", DirectoryID: "d", RevisionID: "r", Rules: []Rule{{DepartmentID: "root", Region: "国内工业"}, {DepartmentID: "b", Region: "APAC"}}}, d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, key := range [][2]string{{"staff-a", ""}, {"", "union-a"}} {
|
||||
got := p.Resolve(key[0], key[1])
|
||||
if got.State != "matched" || got.Region != "国内工业" || len(got.Departments) != 1 || got.PersonID != "staff-a" {
|
||||
t.Fatalf("stable identity not resolved/deduplicated: %+v", got)
|
||||
}
|
||||
}
|
||||
if got := p.Resolve("staff-cross", ""); got.State != "cross_region" || got.Region != "" || len(got.Departments) != 2 {
|
||||
t.Fatalf("invented primary region: %+v", got)
|
||||
}
|
||||
if got := p.Resolve("", "duplicate"); got.State != "identity_conflict" {
|
||||
t.Fatalf("duplicate union selected first person: %+v", got)
|
||||
}
|
||||
if got := p.Resolve("staff-a", "union-cross"); got.State != "identity_conflict" {
|
||||
t.Fatalf("contradictory identifiers silently preferred: %+v", got)
|
||||
}
|
||||
if got := p.Resolve("absent", ""); got.State != "unmatched" {
|
||||
t.Fatalf("unknown identity inferred: %+v", got)
|
||||
}
|
||||
if got := p.Resolve("", ""); got.State != "no_identity" {
|
||||
t.Fatalf("anonymous identity inferred: %+v", got)
|
||||
}
|
||||
}
|
||||
func TestAttributionDoesNotTreatMissingDirectoryAsPermissionSuccess(t *testing.T) {
|
||||
p, err := BuildAttribution(State{State: "empty"}, Directory{})
|
||||
if err != nil || p.Resolve("staff", "").State != "directory_missing" {
|
||||
t.Fatal("missing directory not explicit")
|
||||
}
|
||||
var store *Store
|
||||
if _, err = store.GetAttribution(context.Background()); !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatal("unavailable store returned an empty directory")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
var ErrDirectorySync = errors.New("directory synchronization failed; active snapshot was not replaced")
|
||||
|
||||
type ExecDirectoryRunner struct {
|
||||
Python string
|
||||
Script string
|
||||
RuntimeRoot string
|
||||
SpoolRoot string
|
||||
Timeout time.Duration
|
||||
}
|
||||
|
||||
// Only server-owned paths cross this boundary. Credentials are read by the
|
||||
// existing directory client from its private file, never passed in argv/logs.
|
||||
func (r ExecDirectoryRunner) Run(ctx context.Context) (Import, error) {
|
||||
for _, path := range []string{r.Python, r.Script, r.RuntimeRoot, r.SpoolRoot} {
|
||||
if !filepath.IsAbs(path) || filepath.Clean(path) != path {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
}
|
||||
if r.Timeout == 0 {
|
||||
r.Timeout = 20 * time.Minute
|
||||
}
|
||||
if r.Timeout < time.Millisecond || r.Timeout > 30*time.Minute {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
work, err := os.MkdirTemp(r.SpoolRoot, "qa-directory-")
|
||||
if err != nil {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
defer os.RemoveAll(work)
|
||||
output := filepath.Join(work, "directory.json")
|
||||
runCtx, cancel := context.WithTimeout(ctx, r.Timeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(runCtx, r.Python, r.Script, "--runtime-root", r.RuntimeRoot, "--output", output, "--fetch-live-directory")
|
||||
cmd.Dir = r.RuntimeRoot
|
||||
cmd.Env = []string{"PYTHONUNBUFFERED=1"}
|
||||
for _, key := range []string{"PATH", "HOME", "TMPDIR", "LANG", "TZ"} {
|
||||
if value, ok := os.LookupEnv(key); ok {
|
||||
cmd.Env = append(cmd.Env, key+"="+value)
|
||||
}
|
||||
}
|
||||
cmd.Stdout = io.Discard
|
||||
cmd.Stderr = io.Discard
|
||||
cmd.WaitDelay = 3 * time.Second
|
||||
if err = cmd.Run(); err != nil {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
info, err := os.Lstat(output)
|
||||
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0077 != 0 || info.Size() < 1 || info.Size() > 16<<20 {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
file, err := os.Open(output)
|
||||
if err != nil {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
defer file.Close()
|
||||
decoder := json.NewDecoder(io.LimitReader(file, (16<<20)+1))
|
||||
decoder.DisallowUnknownFields()
|
||||
var result Import
|
||||
if decoder.Decode(&result) != nil || decoder.Decode(&struct{}{}) != io.EOF || !result.Complete || result.Source != "weekly_report_live" || result.SyncedAt.IsZero() || result.SyncedAt.After(time.Now().Add(time.Minute)) {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
if _, err = Preview(result.Directory, nil); err != nil {
|
||||
return Import{}, ErrDirectorySync
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDirectoryBridgeRegressionSuiteRunsInDefaultCI(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "python3", "-m", "unittest", "discover", "-s", "scripts", "-p", "test_qa_ops_directory_sync.py", "-v")
|
||||
cmd.Dir = filepath.Join("..", "..")
|
||||
if output, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("directory bridge regression failed: %v\n%s", err, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectoryRunnerParsesPrivateOutputAndRemovesSpool(t *testing.T) {
|
||||
t.Setenv("DINGTALK_CLIENT_SECRET", "fixture-credential-do-not-forward")
|
||||
root := t.TempDir()
|
||||
spool := filepath.Join(root, "spool")
|
||||
if err := os.Mkdir(spool, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
script := filepath.Join(root, "fake-sync.sh")
|
||||
fixture := `{"source":"weekly_report_live","synced_at":"2026-10-01T00:00:00Z","complete":true,"directory":{"departments":[{"id":"root","parent_id":"","name":"隔离测试部门"}],"people":[{"id":"fixture-user","department_ids":["root"]}]}}`
|
||||
body := "#!/bin/sh\nif [ -n \"${DINGTALK_CLIENT_SECRET:-}\" ]; then exit 3; fi\nwhile [ \"$#\" -gt 0 ]; do if [ \"$1\" = --output ]; then output=$2; shift; fi; shift; done\numask 077\nprintf '%s' '" + fixture + "' > \"$output\"\n"
|
||||
if err := os.WriteFile(script, []byte(body), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runner := ExecDirectoryRunner{Python: "/bin/sh", Script: script, RuntimeRoot: root, SpoolRoot: spool, Timeout: time.Second}
|
||||
result, err := runner.Run(context.Background())
|
||||
if err != nil || len(result.Directory.People) != 1 {
|
||||
t.Fatalf("runner failed: %v", err)
|
||||
}
|
||||
entries, err := os.ReadDir(spool)
|
||||
if err != nil || len(entries) != 0 {
|
||||
t.Fatal("personnel spool leaked")
|
||||
}
|
||||
if err := os.WriteFile(script, []byte("#!/bin/sh\nprintf 'secret-driver-detail' >&2\nexit 1\n"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = runner.Run(context.Background())
|
||||
if err == nil || strings.Contains(err.Error(), "secret-driver-detail") {
|
||||
t.Fatal("runner error is missing or leaks child stderr")
|
||||
}
|
||||
entries, _ = os.ReadDir(spool)
|
||||
if len(entries) != 0 {
|
||||
t.Fatal("failed runner leaked spool")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectoryRunnerRejectsMalformedOrPublicPersonnelOutput(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
for _, mode := range []string{"malformed", "public"} {
|
||||
script := filepath.Join(root, mode+".sh")
|
||||
body := "#!/bin/sh\numask 077\nwhile [ \"$#\" -gt 0 ]; do if [ \"$1\" = --output ]; then output=$2; shift; fi; shift; done\nprintf '%s' '{not-json}' > \"$output\"\n"
|
||||
if mode == "public" {
|
||||
body += "printf '%s' '{\"source\":\"weekly_report_live\",\"synced_at\":\"2026-10-01T00:00:00Z\",\"complete\":true,\"directory\":{\"departments\":[{\"id\":\"root\",\"parent_id\":\"\",\"name\":\"fixture\"}],\"people\":[{\"id\":\"fixture-user\",\"department_ids\":[\"root\"]}]}}' > \"$output\"\nchmod 644 \"$output\"\n"
|
||||
}
|
||||
if err := os.WriteFile(script, []byte(body), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runner := ExecDirectoryRunner{Python: "/bin/sh", Script: script, RuntimeRoot: root, SpoolRoot: root, Timeout: time.Second}
|
||||
if _, err := runner.Run(context.Background()); err == nil {
|
||||
t.Fatal("invalid output accepted")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectoryRunnerDeadlineStopsChildAndCleansPrivateSpool(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
spool := filepath.Join(root, "spool")
|
||||
if err := os.Mkdir(spool, 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
script := filepath.Join(root, "timeout.sh")
|
||||
if err := os.WriteFile(script, []byte("#!/bin/sh\nexec sleep 30\n"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runner := ExecDirectoryRunner{Python: "/bin/sh", Script: script, RuntimeRoot: root, SpoolRoot: spool, Timeout: 50 * time.Millisecond}
|
||||
started := time.Now()
|
||||
if _, err := runner.Run(context.Background()); err == nil {
|
||||
t.Fatal("timed-out source reported success")
|
||||
}
|
||||
if time.Since(started) > 5*time.Second {
|
||||
t.Fatal("source process cleanup is unbounded")
|
||||
}
|
||||
entries, err := os.ReadDir(spool)
|
||||
if err != nil || len(entries) != 0 {
|
||||
t.Fatal("timed-out source retained personnel spool")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMembersKeepNamedAndUnnamedRosterPeopleWithoutRawIdentifiers(t *testing.T) {
|
||||
d := Directory{Departments: []Department{{ID: "root", Name: "fixture"}}, People: []Person{{ID: "private-a", Name: "<源端姓名>", UnionID: "private-union", DepartmentIDs: []string{"root"}}, {ID: "private-zero", DepartmentIDs: []string{"root"}}}}
|
||||
p, err := BuildAttribution(State{State: "ready", DirectoryID: "directory"}, d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
members := p.Members()
|
||||
if len(members) != 2 {
|
||||
t.Fatal("zero-use roster member omitted")
|
||||
}
|
||||
if p.Resolve("private-a", "").Name != "<源端姓名>" || p.Resolve("private-zero", "").Name != "" {
|
||||
t.Fatal("display name lost or guessed")
|
||||
}
|
||||
raw, _ := json.Marshal(members)
|
||||
if strings.Contains(string(raw), "private-") || strings.Contains(string(raw), "department_ids") {
|
||||
t.Fatal("internal identifiers serialized")
|
||||
}
|
||||
d.People[0].Name = "bad\x00name"
|
||||
if _, err = BuildAttribution(State{State: "ready", DirectoryID: "directory"}, d); err == nil {
|
||||
t.Fatal("invalid display name accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var ErrInvalid = errors.New("invalid organization data or region rules")
|
||||
var RegionNames = []string{"国内工业", "APAC", "EMEA", "America"}
|
||||
|
||||
type Department struct {
|
||||
ID string `json:"id"`
|
||||
ParentID string `json:"parent_id"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
type Person struct {
|
||||
Name string `json:"name,omitempty"`
|
||||
ID string `json:"id"`
|
||||
UnionID string `json:"union_id,omitempty"`
|
||||
DepartmentIDs []string `json:"department_ids"`
|
||||
}
|
||||
type Directory struct {
|
||||
Departments []Department `json:"departments"`
|
||||
People []Person `json:"people"`
|
||||
}
|
||||
type Rule struct {
|
||||
DepartmentID string `json:"department_id"`
|
||||
Region string `json:"region"`
|
||||
}
|
||||
type DepartmentView struct {
|
||||
Department
|
||||
Path string `json:"path"`
|
||||
Region string `json:"region"`
|
||||
Explicit bool `json:"explicit"`
|
||||
InheritedFrom string `json:"inherited_from"`
|
||||
}
|
||||
type PreviewResult struct {
|
||||
Departments []DepartmentView `json:"departments"`
|
||||
Regions map[string]int `json:"regions"`
|
||||
TotalPeople int `json:"total_people"`
|
||||
UnassignedPeople int `json:"unassigned_people"`
|
||||
ConflictingPeople int `json:"conflicting_people"`
|
||||
MissingDepartments []string `json:"missing_departments"`
|
||||
}
|
||||
|
||||
// Identity is department ID. Names and legacy keyword rules never decide ownership.
|
||||
func Preview(directory Directory, rules []Rule) (PreviewResult, error) {
|
||||
result := PreviewResult{Departments: []DepartmentView{}, Regions: map[string]int{}, MissingDepartments: []string{}, TotalPeople: len(directory.People)}
|
||||
if len(directory.Departments) == 0 || len(directory.Departments) > 20000 || len(directory.People) == 0 || len(directory.People) > 100000 || len(rules) > 20000 {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
for _, region := range RegionNames {
|
||||
result.Regions[region] = 0
|
||||
}
|
||||
departments := make(map[string]Department, len(directory.Departments))
|
||||
for _, d := range directory.Departments {
|
||||
if !validID(d.ID) || d.Name == "" || len(d.Name) > 1000 || strings.ContainsRune(d.Name, 0) {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
if _, ok := departments[d.ID]; ok {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
departments[d.ID] = d
|
||||
}
|
||||
direct := map[string]string{}
|
||||
for _, rule := range rules {
|
||||
if !validID(rule.DepartmentID) {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
if _, ok := result.Regions[rule.Region]; !ok {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
if _, ok := direct[rule.DepartmentID]; ok {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
direct[rule.DepartmentID] = rule.Region
|
||||
if _, ok := departments[rule.DepartmentID]; !ok {
|
||||
result.MissingDepartments = append(result.MissingDepartments, rule.DepartmentID)
|
||||
}
|
||||
}
|
||||
resolved := map[string]DepartmentView{}
|
||||
for _, d := range directory.Departments {
|
||||
seen := map[string]bool{}
|
||||
chain := []string{}
|
||||
region, from := "", ""
|
||||
for cur := d.ID; cur != ""; {
|
||||
node, ok := departments[cur]
|
||||
if !ok || seen[cur] || len(chain) > 100 {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
seen[cur] = true
|
||||
chain = append(chain, node.Name)
|
||||
if region == "" && direct[cur] != "" {
|
||||
region, from = direct[cur], cur
|
||||
}
|
||||
cur = node.ParentID
|
||||
}
|
||||
for left, right := 0, len(chain)-1; left < right; left, right = left+1, right-1 {
|
||||
chain[left], chain[right] = chain[right], chain[left]
|
||||
}
|
||||
view := DepartmentView{Department: d, Path: strings.Join(chain, "/"), Region: region, Explicit: direct[d.ID] != ""}
|
||||
if from != d.ID {
|
||||
view.InheritedFrom = from
|
||||
}
|
||||
resolved[d.ID] = view
|
||||
result.Departments = append(result.Departments, view)
|
||||
}
|
||||
people := map[string]bool{}
|
||||
for _, person := range directory.People {
|
||||
if !validID(person.ID) || people[person.ID] || len(person.DepartmentIDs) > 1000 || len(person.Name) > 1000 || strings.ContainsRune(person.Name, 0) {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
people[person.ID] = true
|
||||
regions := map[string]bool{}
|
||||
for _, id := range person.DepartmentIDs {
|
||||
department, ok := resolved[id]
|
||||
if !ok {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
if department.Region != "" {
|
||||
regions[department.Region] = true
|
||||
}
|
||||
}
|
||||
switch len(regions) {
|
||||
case 0:
|
||||
result.UnassignedPeople++
|
||||
case 1:
|
||||
for region := range regions {
|
||||
result.Regions[region]++
|
||||
}
|
||||
default:
|
||||
result.ConflictingPeople++
|
||||
}
|
||||
}
|
||||
sort.Slice(result.Departments, func(i, j int) bool {
|
||||
a, b := result.Departments[i], result.Departments[j]
|
||||
if a.Path == b.Path {
|
||||
return a.ID < b.ID
|
||||
}
|
||||
return a.Path < b.Path
|
||||
})
|
||||
sort.Strings(result.MissingDepartments)
|
||||
return result, nil
|
||||
}
|
||||
func validID(value string) bool {
|
||||
return value != "" && len(value) <= 256 && strings.TrimSpace(value) == value && !strings.ContainsAny(value, "\x00\r\n")
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func exampleDirectory() Directory {
|
||||
return Directory{Departments: []Department{
|
||||
{ID: "root", Name: "示例公司"},
|
||||
{ID: "sales", ParentID: "root", Name: "销售"},
|
||||
{ID: "apac", ParentID: "sales", Name: "业务组"},
|
||||
{ID: "leaf", ParentID: "apac", Name: "支持"},
|
||||
{ID: "other", ParentID: "root", Name: "业务组"},
|
||||
}, People: []Person{
|
||||
{ID: "p1", DepartmentIDs: []string{"sales"}},
|
||||
{ID: "p2", DepartmentIDs: []string{"leaf"}},
|
||||
{ID: "p3", DepartmentIDs: []string{"sales", "leaf"}},
|
||||
{ID: "p4", DepartmentIDs: []string{"other"}},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestRegionPreviewUsesNearestExplicitAncestorAndCountsPeopleOnce(t *testing.T) {
|
||||
view, err := Preview(exampleDirectory(), []Rule{{DepartmentID: "sales", Region: "国内工业"}, {DepartmentID: "apac", Region: "APAC"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
byID := map[string]DepartmentView{}
|
||||
for _, d := range view.Departments {
|
||||
byID[d.ID] = d
|
||||
}
|
||||
if got := byID["leaf"]; got.Region != "APAC" || got.InheritedFrom != "apac" || got.Path != "示例公司/销售/业务组/支持" {
|
||||
t.Fatalf("wrong child inheritance: %+v", got)
|
||||
}
|
||||
if !byID["apac"].Explicit || byID["other"].Region != "" {
|
||||
t.Fatal("explicit choice or unmapped department lost")
|
||||
}
|
||||
if view.Regions["国内工业"] != 1 || view.Regions["APAC"] != 1 || view.UnassignedPeople != 1 || view.ConflictingPeople != 1 || view.TotalPeople != 4 {
|
||||
t.Fatalf("person duplicated or conflict silently resolved: %+v", view)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegionPreviewRejectsBrokenDirectoryAndDuplicateAssignments(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
mutate func(*Directory)
|
||||
rules []Rule
|
||||
}{
|
||||
{name: "cycle", mutate: func(d *Directory) { d.Departments[0].ParentID = "leaf" }},
|
||||
{name: "missing parent", mutate: func(d *Directory) { d.Departments[1].ParentID = "absent" }},
|
||||
{name: "duplicate department", mutate: func(d *Directory) { d.Departments = append(d.Departments, d.Departments[0]) }},
|
||||
{name: "duplicate person", mutate: func(d *Directory) { d.People = append(d.People, d.People[0]) }},
|
||||
{name: "missing membership", mutate: func(d *Directory) { d.People[0].DepartmentIDs = []string{"absent"} }},
|
||||
{name: "conflicting direct rules", rules: []Rule{{DepartmentID: "sales", Region: "国内工业"}, {DepartmentID: "sales", Region: "APAC"}}},
|
||||
{name: "unknown region", rules: []Rule{{DepartmentID: "sales", Region: "automatic"}}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
d := exampleDirectory()
|
||||
if tc.mutate != nil {
|
||||
tc.mutate(&d)
|
||||
}
|
||||
_, err := Preview(d, tc.rules)
|
||||
if !errors.Is(err, ErrInvalid) {
|
||||
t.Fatalf("must fail closed, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegionPreviewKeepsMissingRulesAndDoesNotInventPrimaryDepartment(t *testing.T) {
|
||||
d := exampleDirectory()
|
||||
d.People = append(d.People, Person{ID: "p5", DepartmentIDs: []string{"apac", "leaf"}})
|
||||
view, err := Preview(d, []Rule{{DepartmentID: "sales", Region: "国内工业"}, {DepartmentID: "apac", Region: "APAC"}, {DepartmentID: "removed", Region: "EMEA"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(view.MissingDepartments) != 1 || view.MissingDepartments[0] != "removed" || view.ConflictingPeople != 1 || view.Regions["APAC"] != 2 {
|
||||
t.Fatalf("lost orphan rule or duplicated same-region membership: %+v", view)
|
||||
}
|
||||
d.Departments[2].Name = "改名后的部门"
|
||||
renamed, err := Preview(d, []Rule{{DepartmentID: "sales", Region: "国内工业"}, {DepartmentID: "apac", Region: "APAC"}})
|
||||
if err != nil || renamed.Regions["APAC"] != 2 {
|
||||
t.Fatal("stable department ID did not survive rename")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"knowledge-governance/internal/jobs"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
var ErrConflict = errors.New("organization snapshot or region revision changed")
|
||||
var ErrUnavailable = errors.New("organization store unavailable")
|
||||
var ErrNoDirectory = errors.New("organization directory has not been imported")
|
||||
|
||||
type Import struct {
|
||||
Source string `json:"source"`
|
||||
SyncedAt time.Time `json:"synced_at"`
|
||||
Complete bool `json:"complete"`
|
||||
Directory Directory `json:"directory"`
|
||||
}
|
||||
type SaveRequest struct {
|
||||
DirectoryID string `json:"directory_id"`
|
||||
BaseRevision string `json:"base_revision"`
|
||||
Rules []Rule `json:"rules"`
|
||||
}
|
||||
type State struct {
|
||||
State string `json:"state"`
|
||||
DirectoryID string `json:"directory_id"`
|
||||
DirectorySource string `json:"directory_source"`
|
||||
DirectorySyncedAt *time.Time `json:"directory_synced_at"`
|
||||
RevisionID string `json:"revision_id"`
|
||||
UpdatedBy string `json:"updated_by"`
|
||||
UpdatedAt *time.Time `json:"updated_at"`
|
||||
Rules []Rule `json:"rules"`
|
||||
View PreviewResult `json:"view"`
|
||||
}
|
||||
type Store struct {
|
||||
pool *pgxpool.Pool
|
||||
tenant string
|
||||
}
|
||||
|
||||
func NewStore(pool *pgxpool.Pool, tenant string) *Store { return &Store{pool: pool, tenant: tenant} }
|
||||
func (s *Store) available() bool { return s != nil && s.pool != nil && validID(s.tenant) }
|
||||
|
||||
type orgQuerier interface {
|
||||
QueryRow(context.Context, string, ...any) pgx.Row
|
||||
}
|
||||
|
||||
const readStateSQL = `SELECT d.id,d.source,d.synced_at,d.payload,COALESCE(r.id,''),COALESCE(r.rules,'[]'::jsonb),COALESCE(r.updated_by,''),r.created_at
|
||||
FROM qa_ops_organization_state st JOIN qa_ops_directory_snapshots d ON d.tenant_key=st.tenant_key AND d.id=st.directory_id
|
||||
LEFT JOIN qa_ops_region_revisions r ON r.tenant_key=st.tenant_key AND r.id=st.revision_id WHERE st.tenant_key=$1`
|
||||
|
||||
func (s *Store) read(ctx context.Context, q orgQuerier, lock bool) (State, Directory, error) {
|
||||
state := State{State: "empty", Rules: []Rule{}, View: PreviewResult{Departments: []DepartmentView{}, Regions: map[string]int{}, MissingDepartments: []string{}}}
|
||||
for _, r := range RegionNames {
|
||||
state.View.Regions[r] = 0
|
||||
}
|
||||
var directory Directory
|
||||
query := readStateSQL
|
||||
if lock {
|
||||
query += " FOR UPDATE OF st"
|
||||
}
|
||||
var payload, rules []byte
|
||||
err := q.QueryRow(ctx, query, s.tenant).Scan(&state.DirectoryID, &state.DirectorySource, &state.DirectorySyncedAt, &payload, &state.RevisionID, &rules, &state.UpdatedBy, &state.UpdatedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return state, directory, nil
|
||||
}
|
||||
if err != nil || json.Unmarshal(payload, &directory) != nil || json.Unmarshal(rules, &state.Rules) != nil {
|
||||
return State{}, directory, ErrUnavailable
|
||||
}
|
||||
state.View, err = Preview(directory, state.Rules)
|
||||
if err != nil {
|
||||
return State{}, directory, ErrUnavailable
|
||||
}
|
||||
state.State = "ready"
|
||||
return state, directory, nil
|
||||
}
|
||||
func (s *Store) GetView(ctx context.Context) (State, error) {
|
||||
if !s.available() {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
state, _, err := s.read(ctx, s.pool, false)
|
||||
return state, err
|
||||
}
|
||||
func (s *Store) PreviewRules(ctx context.Context, request SaveRequest) (State, error) {
|
||||
if !s.available() {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
state, directory, err := s.read(ctx, s.pool, false)
|
||||
if err != nil {
|
||||
return State{}, err
|
||||
}
|
||||
return applyRequest(state, directory, request)
|
||||
}
|
||||
func applyRequest(state State, directory Directory, request SaveRequest) (State, error) {
|
||||
if state.State != "ready" {
|
||||
return State{}, ErrNoDirectory
|
||||
}
|
||||
if request.DirectoryID != state.DirectoryID || request.BaseRevision != state.RevisionID {
|
||||
return State{}, ErrConflict
|
||||
}
|
||||
view, err := Preview(directory, request.Rules)
|
||||
if err != nil {
|
||||
return State{}, err
|
||||
}
|
||||
// Missing saved paths are retained for operator review, never silently removed.
|
||||
old := map[string]string{}
|
||||
for _, rule := range state.Rules {
|
||||
old[rule.DepartmentID] = rule.Region
|
||||
}
|
||||
for _, missing := range view.MissingDepartments {
|
||||
for _, rule := range request.Rules {
|
||||
if rule.DepartmentID == missing && old[missing] != rule.Region {
|
||||
return State{}, ErrInvalid
|
||||
}
|
||||
}
|
||||
}
|
||||
state.View = view
|
||||
state.Rules = append([]Rule{}, request.Rules...)
|
||||
sort.Slice(state.Rules, func(i, j int) bool { return state.Rules[i].DepartmentID < state.Rules[j].DepartmentID })
|
||||
return state, nil
|
||||
}
|
||||
func (s *Store) SaveRules(ctx context.Context, request SaveRequest, actor string) (State, error) {
|
||||
if !s.available() {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
if !validID(actor) {
|
||||
return State{}, ErrInvalid
|
||||
}
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
defer func() { _ = tx.Rollback(context.Background()) }()
|
||||
current, directory, err := s.read(ctx, tx, true)
|
||||
if err != nil {
|
||||
return State{}, err
|
||||
}
|
||||
next, err := applyRequest(current, directory, request)
|
||||
if err != nil {
|
||||
return State{}, err
|
||||
}
|
||||
before := append([]Rule{}, current.Rules...)
|
||||
sort.Slice(before, func(i, j int) bool { return before[i].DepartmentID < before[j].DepartmentID })
|
||||
oldJSON, _ := json.Marshal(before)
|
||||
newJSON, _ := json.Marshal(next.Rules)
|
||||
if current.RevisionID != "" && string(oldJSON) == string(newJSON) {
|
||||
if tx.Commit(ctx) != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
return current, nil
|
||||
}
|
||||
var nonce [16]byte
|
||||
if _, err = rand.Read(nonce[:]); err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
next.RevisionID = "reg-" + hex.EncodeToString(nonce[:])
|
||||
next.UpdatedBy = actor
|
||||
var at time.Time
|
||||
if err = tx.QueryRow(ctx, `INSERT INTO qa_ops_region_revisions(tenant_key,id,directory_id,rules,updated_by) VALUES($1,$2,$3,$4,$5) RETURNING created_at`, s.tenant, next.RevisionID, next.DirectoryID, newJSON, actor).Scan(&at); err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
next.UpdatedAt = &at
|
||||
if _, err = tx.Exec(ctx, `UPDATE qa_ops_organization_state SET revision_id=$2 WHERE tenant_key=$1`, s.tenant, next.RevisionID); err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
return next, nil
|
||||
}
|
||||
func (s *Store) ReplaceDirectory(ctx context.Context, input Import) (State, error) {
|
||||
return s.replaceDirectory(ctx, input, nil)
|
||||
}
|
||||
func (s *Store) replaceDirectory(ctx context.Context, input Import, lease *jobs.Job) (State, error) {
|
||||
if !s.available() {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
if !input.Complete || input.SyncedAt.IsZero() || input.SyncedAt.After(time.Now().Add(time.Minute)) || (input.Source != "weekly_report_snapshot" && input.Source != "weekly_report_live") {
|
||||
return State{}, ErrInvalid
|
||||
}
|
||||
if _, err := Preview(input.Directory, nil); err != nil {
|
||||
return State{}, err
|
||||
}
|
||||
raw, err := json.Marshal(input)
|
||||
if err != nil || len(raw) > 16<<20 {
|
||||
return State{}, ErrInvalid
|
||||
}
|
||||
sum := sha256.Sum256(raw)
|
||||
id := "dir-" + hex.EncodeToString(sum[:])
|
||||
payload, _ := json.Marshal(input.Directory)
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
defer func() { _ = tx.Rollback(context.Background()) }()
|
||||
if lease != nil {
|
||||
if _, err = s.syncLease(ctx, tx, *lease, true); err != nil {
|
||||
return State{}, err
|
||||
}
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `INSERT INTO qa_ops_directory_snapshots(tenant_key,id,source,synced_at,payload) VALUES($1,$2,$3,$4,$5) ON CONFLICT(tenant_key,id) DO NOTHING`, s.tenant, id, input.Source, input.SyncedAt, payload); err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
tag, err := tx.Exec(ctx, `INSERT INTO qa_ops_organization_state(tenant_key,directory_id) VALUES($1,$2) ON CONFLICT(tenant_key) DO UPDATE SET directory_id=EXCLUDED.directory_id WHERE qa_ops_organization_state.directory_id=EXCLUDED.directory_id OR (SELECT synced_at FROM qa_ops_directory_snapshots WHERE tenant_key=$1 AND id=qa_ops_organization_state.directory_id)<$3`, s.tenant, id, input.SyncedAt)
|
||||
if err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
if tag.RowsAffected() != 1 {
|
||||
return State{}, ErrConflict
|
||||
}
|
||||
if lease != nil {
|
||||
result, _ := json.Marshal(SyncResult{DirectoryID: id, Departments: len(input.Directory.Departments), People: len(input.Directory.People), SyncedAt: input.SyncedAt})
|
||||
tag, err = tx.Exec(ctx, `UPDATE jobs SET payload=jsonb_set(payload,'{result}',$3::jsonb),updated_at=now() WHERE id=$1 AND lease_owner=$2 AND state='running' AND lease_until>clock_timestamp() AND attempt=$4`, lease.ID, lease.LeaseOwner, result, lease.Attempt)
|
||||
if err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
if tag.RowsAffected() != 1 {
|
||||
return State{}, ErrSyncLease
|
||||
}
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return State{}, ErrUnavailable
|
||||
}
|
||||
return s.GetView(ctx)
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
package qaopsorg_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
)
|
||||
|
||||
func TestOrganizationStoreRoundTripAndOptimisticConflict(t *testing.T) {
|
||||
dsn := os.Getenv("KG_QAOPS_ORG_TEST_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("explicit isolated organization database is not configured")
|
||||
}
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil || u.Hostname() != "127.0.0.1" || !strings.HasPrefix(u.Path, "/kg_qaops_org_test_") || os.Getenv("KG_QAOPS_ORG_TEST_ISOLATED") != "1" {
|
||||
t.Fatal("refusing non-isolated organization database")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal("test database connection failed")
|
||||
}
|
||||
defer pool.Close()
|
||||
if err = postgresstore.ApplyMigrations(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store := qaopsorg.NewStore(pool, "fixture-tenant")
|
||||
empty, err := store.GetView(ctx)
|
||||
if err != nil || empty.State != "empty" {
|
||||
t.Fatalf("empty state must be explicit: %+v %v", empty, err)
|
||||
}
|
||||
input := qaopsorg.Import{Source: "weekly_report_snapshot", SyncedAt: time.Date(2026, 9, 27, 14, 3, 44, 0, time.UTC), Complete: true, Directory: qaopsorg.Directory{
|
||||
Departments: []qaopsorg.Department{{ID: "root", Name: "示例公司"}, {ID: "a", ParentID: "root", Name: "国内组"}, {ID: "b", ParentID: "a", Name: "海外组"}},
|
||||
People: []qaopsorg.Person{{ID: "u1", DepartmentIDs: []string{"a"}}, {ID: "u2", DepartmentIDs: []string{"b"}}},
|
||||
}}
|
||||
imported, err := store.ReplaceDirectory(ctx, input)
|
||||
if err != nil || imported.DirectoryID == "" || imported.View.TotalPeople != 2 {
|
||||
t.Fatalf("import failed: %+v %v", imported, err)
|
||||
}
|
||||
request := qaopsorg.SaveRequest{DirectoryID: imported.DirectoryID, Rules: []qaopsorg.Rule{{DepartmentID: "a", Region: "国内工业"}, {DepartmentID: "b", Region: "APAC"}}}
|
||||
preview, err := store.PreviewRules(ctx, request)
|
||||
if err != nil || preview.View.Regions["APAC"] != 1 {
|
||||
t.Fatalf("preview failed: %+v %v", preview, err)
|
||||
}
|
||||
before, _ := store.GetView(ctx)
|
||||
if before.RevisionID != "" {
|
||||
t.Fatal("preview must not persist changes")
|
||||
}
|
||||
saved, err := store.SaveRules(ctx, request, "fixture-admin")
|
||||
if err != nil || saved.RevisionID == "" || saved.UpdatedBy != "fixture-admin" {
|
||||
t.Fatalf("save failed: %+v %v", saved, err)
|
||||
}
|
||||
read, err := store.GetView(ctx)
|
||||
if err != nil || read.RevisionID != saved.RevisionID || read.View.Regions["国内工业"] != 1 {
|
||||
t.Fatal("saved rules not consumed by read path")
|
||||
}
|
||||
projection, projectionErr := store.GetAttribution(ctx)
|
||||
if projectionErr != nil || projection.Metadata.DirectoryID != read.DirectoryID || projection.Metadata.RevisionID != read.RevisionID || projection.Resolve("u2", "").Region != "APAC" {
|
||||
t.Fatal("attribution did not consume the saved directory/region revision")
|
||||
}
|
||||
if _, err = store.SaveRules(ctx, request, "stale-admin"); !errors.Is(err, qaopsorg.ErrConflict) {
|
||||
t.Fatalf("stale update overwrote newer rules: %v", err)
|
||||
}
|
||||
request.BaseRevision = saved.RevisionID
|
||||
again, err := store.SaveRules(ctx, request, "fixture-admin")
|
||||
if err != nil || again.RevisionID != saved.RevisionID {
|
||||
t.Fatal("unchanged save must not append duplicate revisions")
|
||||
}
|
||||
broken := input
|
||||
broken.Complete = false
|
||||
if _, err = store.ReplaceDirectory(ctx, broken); !errors.Is(err, qaopsorg.ErrInvalid) {
|
||||
t.Fatal("partial sync activated")
|
||||
}
|
||||
request.Rules = []qaopsorg.Rule{{DepartmentID: "absent", Region: "EMEA"}}
|
||||
if _, err = store.SaveRules(ctx, request, "fixture-admin"); !errors.Is(err, qaopsorg.ErrInvalid) {
|
||||
t.Fatal("invalid rule persisted")
|
||||
}
|
||||
after, _ := store.GetView(ctx)
|
||||
if after.RevisionID != saved.RevisionID {
|
||||
t.Fatal("failed writes changed active revision")
|
||||
}
|
||||
other, _ := qaopsorg.NewStore(pool, "another-tenant").GetView(ctx)
|
||||
if other.State != "empty" {
|
||||
t.Fatal("directory leaked across tenants")
|
||||
}
|
||||
input.Directory.Departments = input.Directory.Departments[:2]
|
||||
input.Directory.People = input.Directory.People[:1]
|
||||
input.SyncedAt = input.SyncedAt.Add(time.Hour)
|
||||
newer, err := store.ReplaceDirectory(ctx, input)
|
||||
if err != nil || newer.RevisionID != saved.RevisionID || len(newer.View.MissingDepartments) != 1 {
|
||||
t.Fatalf("directory refresh deleted rules or hid missing department: %+v %v", newer, err)
|
||||
}
|
||||
if _, err = store.PreviewRules(ctx, request); !errors.Is(err, qaopsorg.ErrConflict) {
|
||||
t.Fatal("preview accepted an obsolete directory snapshot")
|
||||
}
|
||||
older := input
|
||||
start := make(chan struct{})
|
||||
results := make(chan error, 2)
|
||||
for _, region := range []string{"APAC", "EMEA"} {
|
||||
go func(region string) {
|
||||
<-start
|
||||
_, failure := store.SaveRules(ctx, qaopsorg.SaveRequest{DirectoryID: newer.DirectoryID, BaseRevision: newer.RevisionID, Rules: []qaopsorg.Rule{{DepartmentID: "a", Region: region}}}, "concurrent-admin")
|
||||
results <- failure
|
||||
}(region)
|
||||
}
|
||||
close(start)
|
||||
successes, conflicts := 0, 0
|
||||
for range 2 {
|
||||
failure := <-results
|
||||
if failure == nil {
|
||||
successes++
|
||||
} else if errors.Is(failure, qaopsorg.ErrConflict) {
|
||||
conflicts++
|
||||
} else {
|
||||
t.Fatal(failure)
|
||||
}
|
||||
}
|
||||
if successes != 1 || conflicts != 1 {
|
||||
t.Fatalf("concurrent stale writes were not serialized: success=%d conflict=%d", successes, conflicts)
|
||||
}
|
||||
older.SyncedAt = older.SyncedAt.Add(-2 * time.Hour)
|
||||
if _, err = store.ReplaceDirectory(ctx, older); !errors.Is(err, qaopsorg.ErrConflict) {
|
||||
t.Fatalf("older directory rolled back the active roster: %v", err)
|
||||
}
|
||||
var revisions int
|
||||
if err = pool.QueryRow(ctx, "SELECT count(*) FROM qa_ops_region_revisions WHERE tenant_key=$1", "fixture-tenant").Scan(&revisions); err != nil || revisions != 2 {
|
||||
t.Fatal("revision history changed unexpectedly")
|
||||
}
|
||||
// This test exercises migration 48, not whichever unrelated migration is latest.
|
||||
for {
|
||||
var latest int64
|
||||
if pool.QueryRow(ctx, "SELECT max(version) FROM schema_migrations").Scan(&latest) != nil {
|
||||
t.Fatal("migration version unavailable")
|
||||
}
|
||||
if latest <= 48 {
|
||||
break
|
||||
}
|
||||
if postgresstore.RollbackLatestMigration(ctx, pool) != nil {
|
||||
t.Fatal("could not reach organization migration")
|
||||
}
|
||||
}
|
||||
if err = postgresstore.RollbackLatestMigration(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var removed bool
|
||||
if err = pool.QueryRow(ctx, "SELECT to_regclass('qa_ops_organization_state') IS NULL AND to_regclass('qa_ops_region_revisions') IS NULL AND to_regclass('qa_ops_directory_snapshots') IS NULL").Scan(&removed); err != nil || !removed {
|
||||
t.Fatal("organization down migration left residue")
|
||||
}
|
||||
if err = postgresstore.ApplyMigrations(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if state, failure := store.GetView(ctx); failure != nil || state.State != "empty" {
|
||||
t.Fatal("organization migration could not be reapplied")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"knowledge-governance/internal/jobs"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
)
|
||||
|
||||
var ErrSyncWorkerUnavailable = errors.New("directory sync worker is not connected")
|
||||
|
||||
const SyncJobPrefix = "qaops.directory.sync."
|
||||
|
||||
var ErrSyncLease = errors.New("directory sync lease is no longer owned")
|
||||
|
||||
type DirectoryRunner interface {
|
||||
Run(context.Context) (Import, error)
|
||||
}
|
||||
type SyncReceipt struct {
|
||||
JobID string `json:"job_id"`
|
||||
Deduplicated bool `json:"deduplicated"`
|
||||
}
|
||||
type SyncResult struct {
|
||||
DirectoryID string `json:"directory_id"`
|
||||
Departments int `json:"departments"`
|
||||
People int `json:"people"`
|
||||
SyncedAt time.Time `json:"synced_at"`
|
||||
}
|
||||
type SyncJob struct {
|
||||
ID string `json:"id"`
|
||||
State string `json:"state"`
|
||||
Attempt int `json:"attempt"`
|
||||
MaxAttempts int `json:"max_attempts"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
ErrorCode string `json:"error_code"`
|
||||
Result *SyncResult `json:"result"`
|
||||
}
|
||||
type SyncStatus struct {
|
||||
Configured bool `json:"configured"`
|
||||
WorkerReady bool `json:"worker_ready"`
|
||||
Job *SyncJob `json:"job"`
|
||||
}
|
||||
|
||||
func (s *Store) SyncJobType() string {
|
||||
h := sha256.Sum256([]byte(s.tenant))
|
||||
return SyncJobPrefix + hex.EncodeToString(h[:])
|
||||
}
|
||||
|
||||
const syncWorkerReadySQL = `SELECT EXISTS(SELECT 1 FROM worker_heartbeats WHERE $1=ANY(string_to_array(job_types,',')) AND last_seen_at>clock_timestamp()-interval '2 minutes') OR EXISTS(SELECT 1 FROM jobs WHERE type=$1 AND state='running' AND lease_until>clock_timestamp())`
|
||||
|
||||
func (s *Store) EnqueueSync(ctx context.Context, key, actor string) (SyncReceipt, bool, error) {
|
||||
if !s.available() {
|
||||
return SyncReceipt{}, false, ErrUnavailable
|
||||
}
|
||||
if !validID(key) || len(key) > 128 || !validID(actor) {
|
||||
return SyncReceipt{}, false, ErrInvalid
|
||||
}
|
||||
fp := sha256.Sum256([]byte(actor))
|
||||
now := time.Now()
|
||||
raw, replayed, err := postgresstore.NewStore(s.pool).ExecuteOperation(ctx, s.SyncJobType(), key, hex.EncodeToString(fp[:]), now, func(tx pgx.Tx) (json.RawMessage, error) {
|
||||
if _, e := tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,0))", s.SyncJobType()); e != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
// A last-attempt process may die without recording failure. A new explicit
|
||||
// action retires that exhausted lease; it never impersonates a live owner.
|
||||
if _, e := tx.Exec(ctx, `UPDATE jobs SET state='failed',last_error='directory worker stopped; previous snapshot retained',last_error_code='directory_sync_failed',lease_owner=NULL,lease_until=NULL,updated_at=now() WHERE type=$1 AND state='running' AND lease_until<clock_timestamp() AND attempt>=max_attempts`, s.SyncJobType()); e != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
var active string
|
||||
e := tx.QueryRow(ctx, `SELECT id FROM jobs WHERE type=$1 AND state IN ('pending','running','retry') ORDER BY created_at DESC,id DESC LIMIT 1`, s.SyncJobType()).Scan(&active)
|
||||
if e == nil {
|
||||
return json.Marshal(SyncReceipt{JobID: active, Deduplicated: true})
|
||||
}
|
||||
if !errors.Is(e, pgx.ErrNoRows) {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
var ready bool
|
||||
if tx.QueryRow(ctx, syncWorkerReadySQL, s.SyncJobType()).Scan(&ready) != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
if !ready {
|
||||
return nil, ErrSyncWorkerUnavailable
|
||||
}
|
||||
digest := sha256.Sum256([]byte(s.SyncJobType() + ":" + key))
|
||||
id := "job-" + hex.EncodeToString(digest[:16])
|
||||
payload, _ := json.Marshal(map[string]string{"triggered_by": actor})
|
||||
job, e := postgresstore.NewStore(tx).Enqueue(ctx, jobs.EnqueueRequest{ID: id, Type: s.SyncJobType(), Payload: payload, AvailableAt: now, MaxAttempts: 3, IdempotencyKey: s.SyncJobType() + ":" + key}, now)
|
||||
if e != nil || job == nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return json.Marshal(SyncReceipt{JobID: job.ID})
|
||||
})
|
||||
if errors.Is(err, postgresstore.ErrOperationRequestConflict) {
|
||||
return SyncReceipt{}, false, ErrConflict
|
||||
}
|
||||
if err != nil {
|
||||
return SyncReceipt{}, false, err
|
||||
}
|
||||
var result SyncReceipt
|
||||
if json.Unmarshal(raw, &result) != nil || result.JobID == "" {
|
||||
return SyncReceipt{}, false, ErrUnavailable
|
||||
}
|
||||
return result, replayed, nil
|
||||
}
|
||||
func (s *Store) GetSyncStatus(ctx context.Context) (SyncStatus, error) {
|
||||
result := SyncStatus{Configured: true}
|
||||
if !s.available() {
|
||||
return SyncStatus{}, ErrUnavailable
|
||||
}
|
||||
if s.pool.QueryRow(ctx, syncWorkerReadySQL, s.SyncJobType()).Scan(&result.WorkerReady) != nil {
|
||||
return SyncStatus{}, ErrUnavailable
|
||||
}
|
||||
var job SyncJob
|
||||
var payload []byte
|
||||
var expired bool
|
||||
err := s.pool.QueryRow(ctx, `SELECT id,state,attempt,max_attempts,updated_at,payload,COALESCE(state='running' AND lease_until<clock_timestamp(),false) FROM jobs WHERE type=$1 ORDER BY created_at DESC,id DESC LIMIT 1`, s.SyncJobType()).Scan(&job.ID, &job.State, &job.Attempt, &job.MaxAttempts, &job.UpdatedAt, &payload, &expired)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return result, nil
|
||||
}
|
||||
if err != nil {
|
||||
return SyncStatus{}, ErrUnavailable
|
||||
}
|
||||
var parsed struct {
|
||||
Result *SyncResult `json:"result"`
|
||||
}
|
||||
if json.Unmarshal(payload, &parsed) != nil {
|
||||
return SyncStatus{}, ErrUnavailable
|
||||
}
|
||||
job.Result = parsed.Result
|
||||
if expired {
|
||||
job.State = "retry"
|
||||
if job.Attempt >= job.MaxAttempts {
|
||||
job.State = "failed"
|
||||
}
|
||||
}
|
||||
if job.State == "failed" || job.State == "retry" {
|
||||
job.ErrorCode = "directory_sync_failed"
|
||||
}
|
||||
result.Job = &job
|
||||
return result, nil
|
||||
}
|
||||
func (s *Store) syncLease(ctx context.Context, q orgQuerier, job jobs.Job, lock bool) (*SyncResult, error) {
|
||||
if job.Type != s.SyncJobType() || job.LeaseOwner == "" {
|
||||
return nil, ErrSyncLease
|
||||
}
|
||||
query := `SELECT payload FROM jobs WHERE id=$1 AND type=$2 AND state='running' AND lease_owner=$3 AND attempt=$4 AND lease_until>clock_timestamp() AND EXISTS(SELECT 1 FROM operation_requests op WHERE op.operation_type=jobs.type AND op.result->>'job_id'=jobs.id)`
|
||||
if lock {
|
||||
query += " FOR UPDATE"
|
||||
}
|
||||
var raw []byte
|
||||
if q.QueryRow(ctx, query, job.ID, s.SyncJobType(), job.LeaseOwner, job.Attempt).Scan(&raw) != nil {
|
||||
return nil, ErrSyncLease
|
||||
}
|
||||
var payload struct {
|
||||
Result *SyncResult `json:"result"`
|
||||
}
|
||||
if json.Unmarshal(raw, &payload) != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return payload.Result, nil
|
||||
}
|
||||
func (s *Store) HandleDirectorySync(ctx context.Context, job jobs.Job, runner DirectoryRunner) error {
|
||||
if !s.available() || runner == nil {
|
||||
return ErrDirectorySync
|
||||
}
|
||||
result, err := s.syncLease(ctx, s.pool, job, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if result != nil {
|
||||
return nil
|
||||
}
|
||||
input, err := runner.Run(ctx)
|
||||
if err != nil {
|
||||
return ErrDirectorySync
|
||||
}
|
||||
if input.Source != "weekly_report_live" {
|
||||
return jobs.Permanent("directory_sync_failed", ErrDirectorySync)
|
||||
}
|
||||
_, err = s.replaceDirectory(ctx, input, &job)
|
||||
if err != nil {
|
||||
if errors.Is(err, ErrSyncLease) {
|
||||
return err
|
||||
}
|
||||
return ErrDirectorySync
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
package qaopsorg_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/jobs"
|
||||
postgresstore "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/qaopsorg"
|
||||
)
|
||||
|
||||
type syncRunner func(context.Context) (qaopsorg.Import, error)
|
||||
|
||||
func (f syncRunner) Run(ctx context.Context) (qaopsorg.Import, error) { return f(ctx) }
|
||||
func TestDirectorySyncQueueDurableReplayAndLeaseFencing(t *testing.T) {
|
||||
dsn := os.Getenv("KG_QAOPS_ORG_TEST_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("explicit isolated organization database is not configured")
|
||||
}
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil || u.Hostname() != "127.0.0.1" || !strings.HasPrefix(u.Path, "/kg_qaops_org_test_") || os.Getenv("KG_QAOPS_ORG_TEST_ISOLATED") != "1" {
|
||||
t.Fatal("refusing non-isolated organization database")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal("isolated connection failed")
|
||||
}
|
||||
defer pool.Close()
|
||||
if err = postgresstore.ApplyMigrations(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store := qaopsorg.NewStore(pool, "sync-fixture-tenant")
|
||||
queue := postgresstore.NewStore(pool)
|
||||
initial := qaopsorg.Import{Source: "weekly_report_live", Complete: true, SyncedAt: time.Now().Add(-time.Hour), Directory: qaopsorg.Directory{Departments: []qaopsorg.Department{{ID: "root", Name: "同步夹具"}}, People: []qaopsorg.Person{{ID: "fixture-person", DepartmentIDs: []string{"root"}}}}}
|
||||
old, err := store.ReplaceDirectory(ctx, initial)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
saved, err := store.SaveRules(ctx, qaopsorg.SaveRequest{DirectoryID: old.DirectoryID, Rules: []qaopsorg.Rule{{DepartmentID: "root", Region: "APAC"}}}, "fixture-admin")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err = store.EnqueueSync(ctx, "op-no-worker", "fixture-admin"); !errors.Is(err, qaopsorg.ErrSyncWorkerUnavailable) {
|
||||
t.Fatalf("disconnected worker accepted: %v", err)
|
||||
}
|
||||
if _, err = queue.Claim(ctx, "fixture-sync-worker", []string{store.SyncJobType()}, time.Now(), time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var first qaopsorg.SyncReceipt
|
||||
var created atomic.Int32
|
||||
var wg sync.WaitGroup
|
||||
failures := make(chan error, 12)
|
||||
ids := make(chan string, 12)
|
||||
for range 12 {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
r, replay, e := qaopsorg.NewStore(pool, "sync-fixture-tenant").EnqueueSync(ctx, "op-same-action", "fixture-admin")
|
||||
if e != nil {
|
||||
failures <- e
|
||||
return
|
||||
}
|
||||
if !replay {
|
||||
created.Add(1)
|
||||
}
|
||||
ids <- r.JobID
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(failures)
|
||||
for e := range failures {
|
||||
t.Fatal(e)
|
||||
}
|
||||
close(ids)
|
||||
for id := range ids {
|
||||
if first.JobID == "" {
|
||||
first.JobID = id
|
||||
}
|
||||
if id != first.JobID {
|
||||
t.Fatal("duplicate effect")
|
||||
}
|
||||
}
|
||||
if created.Load() != 1 {
|
||||
t.Fatal("operation replay was not durable")
|
||||
}
|
||||
reused, _, err := store.EnqueueSync(ctx, "op-another-click", "fixture-admin")
|
||||
if err != nil || !reused.Deduplicated || reused.JobID != first.JobID {
|
||||
t.Fatal("in-flight clicks did not coalesce")
|
||||
}
|
||||
if _, _, err = store.EnqueueSync(ctx, "op-same-action", "another-admin"); !errors.Is(err, qaopsorg.ErrConflict) {
|
||||
t.Fatal("actor fingerprint conflict ignored")
|
||||
}
|
||||
other, _ := qaopsorg.NewStore(pool, "other-sync-tenant").GetSyncStatus(ctx)
|
||||
if other.Job != nil {
|
||||
t.Fatal("job leaked across tenants")
|
||||
}
|
||||
var n int
|
||||
if err = pool.QueryRow(ctx, "SELECT count(*) FROM jobs WHERE type=$1", store.SyncJobType()).Scan(&n); err != nil || n != 1 {
|
||||
t.Fatal("job count must remain one")
|
||||
}
|
||||
job, err := queue.Claim(ctx, "worker-before-crash", []string{store.SyncJobType()}, time.Now(), time.Minute)
|
||||
if err != nil || job == nil {
|
||||
t.Fatal("job was not durable/claimable")
|
||||
}
|
||||
// A different process can reclaim expired work; the old owner must not activate data.
|
||||
if _, err = pool.Exec(ctx, "UPDATE jobs SET lease_until=now()-interval '1 second' WHERE id=$1", job.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reclaimed, err := queue.Claim(ctx, "worker-after-restart", []string{store.SyncJobType()}, time.Now(), time.Minute)
|
||||
if err != nil || reclaimed == nil || reclaimed.Attempt != 2 {
|
||||
t.Fatal("expired lease did not resume")
|
||||
}
|
||||
newer := initial
|
||||
newer.SyncedAt = time.Now()
|
||||
newer.Directory.People = append(append([]qaopsorg.Person{}, initial.Directory.People...), qaopsorg.Person{ID: "fixture-person-2", DepartmentIDs: []string{"root"}})
|
||||
var fetched atomic.Int32
|
||||
run := syncRunner(func(context.Context) (qaopsorg.Import, error) { fetched.Add(1); return newer, nil })
|
||||
if err = store.HandleDirectorySync(ctx, *job, run); err == nil {
|
||||
t.Fatal("stale owner activated a snapshot")
|
||||
}
|
||||
still, _ := store.GetView(ctx)
|
||||
if still.DirectoryID != old.DirectoryID {
|
||||
t.Fatal("stale lease changed active data")
|
||||
}
|
||||
if err = store.HandleDirectorySync(ctx, *reclaimed, run); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
count := fetched.Load()
|
||||
// Crash after committing the snapshot, before queue.Complete: do not fetch twice.
|
||||
if err = store.HandleDirectorySync(ctx, *reclaimed, run); err != nil || fetched.Load() != count {
|
||||
t.Fatal("committed source was fetched again")
|
||||
}
|
||||
if err = queue.Complete(ctx, reclaimed.ID, reclaimed.LeaseOwner, time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
current, _ := store.GetView(ctx)
|
||||
if current.View.TotalPeople != 2 || current.RevisionID != saved.RevisionID || current.View.Regions["APAC"] != 2 {
|
||||
t.Fatal("refresh lost regional rules or failed to activate")
|
||||
}
|
||||
status, err := store.GetSyncStatus(ctx)
|
||||
if err != nil || status.Job == nil || status.Job.State != "completed" || status.Job.Result == nil || status.Job.Result.People != 2 {
|
||||
t.Fatal("safe status missing durable result")
|
||||
}
|
||||
raw, _ := json.Marshal(status)
|
||||
if strings.Contains(string(raw), "fixture-person") || strings.Contains(string(raw), "lease_owner") {
|
||||
t.Fatal("personnel/lease details leaked in status")
|
||||
}
|
||||
// Failure must preserve the previous complete directory and never store raw exceptions.
|
||||
failed, _, err := store.EnqueueSync(ctx, "op-fails", "fixture-admin")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
bad, err := queue.Claim(ctx, "worker-failure", []string{store.SyncJobType()}, time.Now(), time.Minute)
|
||||
if err != nil || bad == nil || bad.ID != failed.JobID {
|
||||
t.Fatal("failure job was not enqueued")
|
||||
}
|
||||
failure := store.HandleDirectorySync(ctx, *bad, syncRunner(func(context.Context) (qaopsorg.Import, error) {
|
||||
return qaopsorg.Import{}, errors.New("sensitive-fixture-must-not-escape")
|
||||
}))
|
||||
if failure == nil || strings.Contains(failure.Error(), "sensitive-fixture") {
|
||||
t.Fatal("unsafe source exception")
|
||||
}
|
||||
if err = queue.Fail(ctx, bad.ID, bad.LeaseOwner, failure.Error(), "directory_sync_failed", time.Now(), time.Now().Add(time.Minute)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
kept, _ := store.GetView(ctx)
|
||||
if kept.DirectoryID != current.DirectoryID || kept.RevisionID != saved.RevisionID {
|
||||
t.Fatal("failed refresh replaced valid state")
|
||||
}
|
||||
late := qaopsorg.NewStore(pool, "late-fence-tenant")
|
||||
lateOld, err := late.ReplaceDirectory(ctx, initial)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = queue.Claim(ctx, "late-worker", []string{late.SyncJobType()}, time.Now(), time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err = late.EnqueueSync(ctx, "op-late-fence", "fixture-admin"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lateJob, err := queue.Claim(ctx, "late-worker", []string{late.SyncJobType()}, time.Now(), time.Minute)
|
||||
if err != nil || lateJob == nil {
|
||||
t.Fatal("late-fence job unavailable")
|
||||
}
|
||||
lateErr := late.HandleDirectorySync(ctx, *lateJob, syncRunner(func(context.Context) (qaopsorg.Import, error) {
|
||||
_, failure := pool.Exec(ctx, "UPDATE jobs SET lease_owner='replacement-worker',attempt=attempt+1,lease_until=now()+interval '1 minute' WHERE id=$1", lateJob.ID)
|
||||
return newer, failure
|
||||
}))
|
||||
if lateErr == nil {
|
||||
t.Fatal("lease lost DURING source fetch was not fenced at commit")
|
||||
}
|
||||
lateKept, _ := late.GetView(ctx)
|
||||
if lateKept.DirectoryID != lateOld.DirectoryID {
|
||||
t.Fatal("late stale worker activated data")
|
||||
}
|
||||
if err = pool.QueryRow(ctx, "SELECT count(*) FROM qa_ops_directory_snapshots WHERE tenant_key=$1", "late-fence-tenant").Scan(&n); err != nil || n != 1 {
|
||||
t.Fatal("rejected stale snapshot left durable rows")
|
||||
}
|
||||
if _, err = pool.Exec(ctx, "UPDATE jobs SET lease_until=now()-interval '1 second' WHERE id=$1", lateJob.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
interrupted, err := late.GetSyncStatus(ctx)
|
||||
if err != nil || interrupted.Job == nil || interrupted.Job.State != "retry" {
|
||||
t.Fatal("interrupted lease must wait for takeover, not claim active source processing")
|
||||
}
|
||||
if _, err = pool.Exec(ctx, "UPDATE jobs SET attempt=max_attempts,lease_until=now()-interval '1 second' WHERE id=$1", lateJob.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
exhausted, err := late.GetSyncStatus(ctx)
|
||||
if err != nil || exhausted.Job == nil || exhausted.Job.State != "failed" {
|
||||
t.Fatal("exhausted crashed worker reported running forever")
|
||||
}
|
||||
fresh, _, err := late.EnqueueSync(ctx, "op-after-exhausted", "fixture-admin")
|
||||
if err != nil || fresh.JobID == lateJob.ID {
|
||||
t.Fatal("explicit retry could not replace an exhausted crashed job")
|
||||
}
|
||||
ghost := qaopsorg.NewStore(pool, "unadmitted-tenant")
|
||||
if _, err = ghost.ReplaceDirectory(ctx, initial); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err = queue.Enqueue(ctx, jobs.EnqueueRequest{ID: "fixture-unadmitted-job", Type: ghost.SyncJobType(), Payload: json.RawMessage(`{}`), AvailableAt: time.Now(), MaxAttempts: 1, IdempotencyKey: "fixture-unadmitted-job"}, time.Now())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ghostJob, err := queue.Claim(ctx, "ghost-worker", []string{ghost.SyncJobType()}, time.Now(), time.Minute)
|
||||
if err != nil || ghostJob == nil {
|
||||
t.Fatal("forged fixture unavailable")
|
||||
}
|
||||
var ghostCalls int
|
||||
ghostErr := ghost.HandleDirectorySync(ctx, *ghostJob, syncRunner(func(context.Context) (qaopsorg.Import, error) { ghostCalls++; return newer, nil }))
|
||||
if ghostErr == nil || ghostCalls != 0 {
|
||||
t.Fatal("unadmitted generic task consumed the directory source")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
type SyncSettings struct {
|
||||
Enabled bool
|
||||
Tenant string
|
||||
Runner ExecDirectoryRunner
|
||||
}
|
||||
|
||||
func LoadSyncSettings(getenv func(string) string) (SyncSettings, error) {
|
||||
var s SyncSettings
|
||||
enabled := getenv("DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED")
|
||||
if enabled == "" || enabled == "false" {
|
||||
return s, nil
|
||||
}
|
||||
if enabled != "true" {
|
||||
return s, errors.New("invalid directory sync enable flag")
|
||||
}
|
||||
s.Tenant = getenv("DOBOTAI_OPS_TENANT_KEY")
|
||||
s.Runner = ExecDirectoryRunner{Python: getenv("DOBOTAI_OPS_DIRECTORY_SYNC_PYTHON"), Script: getenv("DOBOTAI_OPS_DIRECTORY_SYNC_SCRIPT"), RuntimeRoot: getenv("DOBOTAI_OPS_DIRECTORY_SYNC_RUNTIME_ROOT"), SpoolRoot: getenv("DOBOTAI_OPS_DIRECTORY_SYNC_SPOOL_ROOT"), Timeout: 20 * time.Minute}
|
||||
if value := getenv("DOBOTAI_OPS_DIRECTORY_SYNC_TIMEOUT"); value != "" {
|
||||
d, err := time.ParseDuration(value)
|
||||
if err != nil {
|
||||
return SyncSettings{}, errors.New("invalid directory sync timeout")
|
||||
}
|
||||
s.Runner.Timeout = d
|
||||
}
|
||||
if !validID(s.Tenant) || s.Runner.Timeout < time.Second || s.Runner.Timeout > 30*time.Minute {
|
||||
return SyncSettings{}, errors.New("directory sync binding is incomplete")
|
||||
}
|
||||
for _, path := range []string{s.Runner.Python, s.Runner.Script, s.Runner.RuntimeRoot, s.Runner.SpoolRoot} {
|
||||
if !filepath.IsAbs(path) || filepath.Clean(path) != path {
|
||||
return SyncSettings{}, errors.New("directory sync requires server-owned absolute paths")
|
||||
}
|
||||
}
|
||||
s.Enabled = true
|
||||
return s, nil
|
||||
}
|
||||
func (s SyncSettings) CheckWorkerRuntime() error {
|
||||
for _, path := range []string{s.Runner.Python, s.Runner.Script} {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil || !info.Mode().IsRegular() {
|
||||
return ErrSyncWorkerUnavailable
|
||||
}
|
||||
}
|
||||
for _, path := range []string{s.Runner.RuntimeRoot, s.Runner.SpoolRoot} {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil || !info.IsDir() {
|
||||
return ErrSyncWorkerUnavailable
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package qaopsorg
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestDirectorySyncSettingsAreExplicitAndServerOwned(t *testing.T) {
|
||||
values := map[string]string{}
|
||||
get := func(k string) string { return values[k] }
|
||||
off, err := LoadSyncSettings(get)
|
||||
if err != nil || off.Enabled {
|
||||
t.Fatal("sync must default off")
|
||||
}
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_ENABLED"] = "true"
|
||||
if _, err = LoadSyncSettings(get); err == nil {
|
||||
t.Fatal("unbound runtime accepted")
|
||||
}
|
||||
values["DOBOTAI_OPS_TENANT_KEY"] = "fixture-tenant"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_PYTHON"] = "/usr/bin/python3"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_SCRIPT"] = "/app/scripts/qa_ops_directory_sync.py"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_RUNTIME_ROOT"] = "/data/weekly-report"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_SPOOL_ROOT"] = "/data/private-spool"
|
||||
ready, err := LoadSyncSettings(get)
|
||||
if err != nil || !ready.Enabled || ready.Runner.Timeout != 20*time.Minute {
|
||||
t.Fatal("valid runtime not configured")
|
||||
}
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_TIMEOUT"] = "31m"
|
||||
if _, err = LoadSyncSettings(get); err == nil {
|
||||
t.Fatal("unbounded source timeout")
|
||||
}
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_TIMEOUT"] = "10m"
|
||||
values["DOBOTAI_OPS_DIRECTORY_SYNC_SCRIPT"] = "relative/script.py"
|
||||
if _, err = LoadSyncSettings(get); err == nil {
|
||||
t.Fatal("browser-like relative script accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package qaopsquality
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
var hashPattern = regexp.MustCompile(`^[a-f0-9]{64}$`)
|
||||
var noncePattern = regexp.MustCompile(`^[A-Za-z0-9_-]{16,128}$`)
|
||||
|
||||
func validText(s string, max int) bool {
|
||||
return utf8.ValidString(s) && utf8.RuneCountInString(s) <= max && !strings.ContainsRune(s, 0)
|
||||
}
|
||||
func validID(s string) bool { return strings.TrimSpace(s) != "" && validText(s, 256) }
|
||||
func canonical(raw json.RawMessage) (json.RawMessage, error) {
|
||||
if len(raw) == 0 {
|
||||
return json.RawMessage("null"), nil
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(raw))
|
||||
decoder.UseNumber()
|
||||
var v any
|
||||
if decoder.Decode(&v) != nil {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
var extra any
|
||||
if decoder.Decode(&extra) != io.EOF {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
out, err := json.Marshal(v)
|
||||
return out, err
|
||||
}
|
||||
func EvidenceFor(record dobotaiops.Record) (Evidence, error) {
|
||||
if !validID(record.ID) {
|
||||
return Evidence{}, ErrInvalid
|
||||
}
|
||||
refs, err := canonical(record.References)
|
||||
if err != nil {
|
||||
return Evidence{}, ErrInvalid
|
||||
}
|
||||
media, err := canonical(record.Media)
|
||||
if err != nil {
|
||||
return Evidence{}, ErrInvalid
|
||||
}
|
||||
// Keep only assessment inputs, never personnel or mutable feedback/organisation.
|
||||
payload, err := json.Marshal(struct {
|
||||
Version string `json:"version"`
|
||||
ID string `json:"id"`
|
||||
Status string `json:"status"`
|
||||
Question string `json:"question"`
|
||||
Answer string `json:"answer"`
|
||||
References json.RawMessage `json:"references"`
|
||||
Media json.RawMessage `json:"media"`
|
||||
}{"dobotai-2.0-dingtalk-quality-v1", record.ID, record.Status, record.Question, record.Answer, refs, media})
|
||||
if err != nil || len(payload) > 8<<20 {
|
||||
return Evidence{}, ErrInvalid
|
||||
}
|
||||
sum := sha256.Sum256(payload)
|
||||
hash := hex.EncodeToString(sum[:])
|
||||
if record.QualityInputHash != "" {
|
||||
if !hashPattern.MatchString(record.QualityInputHash) {
|
||||
return Evidence{}, ErrInvalid
|
||||
}
|
||||
hash = record.QualityInputHash
|
||||
}
|
||||
return Evidence{Hash: hash, Payload: payload, Record: record, Result: dobotaiops.CheckAnswerCoverage(record.Answer)}, nil
|
||||
}
|
||||
func Validate(r SaveRequest) error {
|
||||
if !hashPattern.MatchString(r.InputHash) || r.InputEpoch < 1 || r.BaseRevision < 0 || !noncePattern.MatchString(r.Nonce) || strings.TrimSpace(r.Reason) == "" || !validText(r.Reason, 4000) {
|
||||
return ErrInvalid
|
||||
}
|
||||
f := r.Fields
|
||||
validCoverage := false
|
||||
for _, s := range []string{"covered", "uncovered", "partial", "pending", "excluded"} {
|
||||
validCoverage = validCoverage || f.Coverage == s
|
||||
}
|
||||
if !validCoverage || (f.Correctness != "correct" && f.Correctness != "incorrect" && f.Correctness != "pending") {
|
||||
return ErrInvalid
|
||||
}
|
||||
if !validText(f.ProductCategory, 128) || !validText(f.ProblemType, 128) || !validText(f.RootCause, 2000) || !validText(f.ExclusionReason, 2000) {
|
||||
return ErrInvalid
|
||||
}
|
||||
if f.Correctness == "correct" && f.RootCause != "" || f.Coverage == "excluded" && strings.TrimSpace(f.ExclusionReason) == "" || f.Coverage != "excluded" && f.ExclusionReason != "" {
|
||||
return ErrInvalid
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func Project(hash string, epoch, revision int64, automatic Check, review *Review) View {
|
||||
view := View{InputHash: hash, InputEpoch: epoch, Revision: revision, State: "unreviewed", Automatic: automatic, LatestReview: review, Effective: Fields{Coverage: automatic.Result.Verdict}}
|
||||
if review == nil {
|
||||
return view
|
||||
}
|
||||
view.State = "stale"
|
||||
if review.InputEpoch == epoch && review.InputHash == hash {
|
||||
view.State = "reviewed"
|
||||
view.Effective = review.Fields
|
||||
}
|
||||
return view
|
||||
}
|
||||
func Summarize(views map[string]View) Summary {
|
||||
result := Summary{Total: len(views), Rates: ComputeRates(views)}
|
||||
for _, v := range views {
|
||||
switch v.State {
|
||||
case "reviewed":
|
||||
result.Reviewed++
|
||||
if v.Effective.Coverage == "pending" || v.Effective.Correctness == "pending" {
|
||||
result.PendingConfirmation++
|
||||
}
|
||||
case "stale":
|
||||
result.Stale++
|
||||
default:
|
||||
result.Unreviewed++
|
||||
}
|
||||
if v.Automatic.Result.Verdict == "uncovered" {
|
||||
result.RuleUncovered++
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package qaopsquality
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"time"
|
||||
)
|
||||
|
||||
var ErrInvalid = errors.New("invalid quality request")
|
||||
var ErrConflict = errors.New("quality input or review revision changed")
|
||||
var ErrUnavailable = errors.New("quality store unavailable")
|
||||
var ErrNotFound = errors.New("quality input snapshot not found")
|
||||
var ErrSnapshotMissing = errors.New("only quality metadata was observed for this input")
|
||||
|
||||
type Fields struct {
|
||||
Coverage string `json:"coverage"`
|
||||
Correctness string `json:"correctness"`
|
||||
ProductCategory string `json:"product_category"`
|
||||
ProblemType string `json:"problem_type"`
|
||||
RootCause string `json:"root_cause"`
|
||||
ExclusionReason string `json:"exclusion_reason"`
|
||||
}
|
||||
type SaveRequest struct {
|
||||
InputHash string `json:"input_hash"`
|
||||
InputEpoch int64 `json:"input_epoch"`
|
||||
BaseRevision int64 `json:"base_revision"`
|
||||
Nonce string `json:"nonce"`
|
||||
Fields Fields `json:"fields"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
type Review struct {
|
||||
Revision int64 `json:"revision"`
|
||||
InputEpoch int64 `json:"input_epoch"`
|
||||
InputHash string `json:"input_hash"`
|
||||
Fields Fields `json:"fields"`
|
||||
Reason string `json:"reason"`
|
||||
Reviewer string `json:"reviewer"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
type Check struct {
|
||||
InputAvailable bool `json:"input_available"`
|
||||
ID int64 `json:"id"`
|
||||
InputEpoch int64 `json:"input_epoch"`
|
||||
InputHash string `json:"input_hash"`
|
||||
Result dobotaiops.CoverageCheck `json:"result"`
|
||||
Source dobotaiops.SourceCoverage `json:"source"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
type View struct {
|
||||
InputHash string `json:"input_hash"`
|
||||
InputEpoch int64 `json:"input_epoch"`
|
||||
Revision int64 `json:"revision"`
|
||||
State string `json:"state"`
|
||||
Automatic Check `json:"automatic"`
|
||||
LatestReview *Review `json:"latest_review"`
|
||||
Effective Fields `json:"effective"`
|
||||
}
|
||||
type History struct {
|
||||
Reviews []Review `json:"reviews"`
|
||||
Checks []Check `json:"checks"`
|
||||
NextReview int64 `json:"next_review"`
|
||||
NextCheck int64 `json:"next_check"`
|
||||
}
|
||||
type Summary struct {
|
||||
Rates Rates `json:"rates"`
|
||||
Total int `json:"total"`
|
||||
Reviewed int `json:"reviewed"`
|
||||
Unreviewed int `json:"unreviewed"`
|
||||
Stale int `json:"stale"`
|
||||
PendingConfirmation int `json:"pending_confirmation"`
|
||||
RuleUncovered int `json:"rule_uncovered"`
|
||||
}
|
||||
type Receipt struct {
|
||||
View View `json:"view"`
|
||||
SavedRevision int64 `json:"saved_revision"`
|
||||
Replayed bool `json:"replayed"`
|
||||
}
|
||||
type Evidence struct {
|
||||
Result dobotaiops.CoverageCheck
|
||||
Hash string
|
||||
Payload json.RawMessage
|
||||
Record dobotaiops.Record
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
InputEpoch int64 `json:"input_epoch"`
|
||||
InputHash string `json:"input_hash"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type Store struct {
|
||||
pool *pgxpool.Pool
|
||||
tenant string
|
||||
}
|
||||
|
||||
func NewStore(pool *pgxpool.Pool, tenant string) *Store { return &Store{pool: pool, tenant: tenant} }
|
||||
@@ -0,0 +1,73 @@
|
||||
package qaopsquality
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEvidenceBindsQualityInputsNotPeopleFeedbackOrJSONKeyOrder(t *testing.T) {
|
||||
record := dobotaiops.Record{ID: "fixture-turn", Status: "succeeded", Question: "示例问题", Answer: "有暂无已发布但存在引用", References: json.RawMessage(`[{"b":2,"a":1}]`), Media: json.RawMessage(`[]`)}
|
||||
a, err := EvidenceFor(record)
|
||||
if err != nil || len(a.Hash) != 64 || !strings.Contains(string(a.Payload), record.Answer) {
|
||||
t.Fatal("quality inputs not snapshotted")
|
||||
}
|
||||
record.Asker = "another-person"
|
||||
record.Department = "changed"
|
||||
record.Feedback.Up = 10
|
||||
record.References = json.RawMessage(`[{"a":1,"b":2}]`)
|
||||
b, _ := EvidenceFor(record)
|
||||
if b.Hash != a.Hash {
|
||||
t.Fatal("irrelevant identity/feedback or JSON object ordering invalidated review")
|
||||
}
|
||||
record.Answer = "原答案变更"
|
||||
b, _ = EvidenceFor(record)
|
||||
if b.Hash == a.Hash {
|
||||
t.Fatal("answer change undetected")
|
||||
}
|
||||
}
|
||||
func TestHumanProjectionKeepsRuleAndStalesObservedInputChanges(t *testing.T) {
|
||||
automatic := Check{Result: dobotaiops.CheckAnswerCoverage("暂无已发布"), InputEpoch: 2, InputHash: strings.Repeat("a", 64)}
|
||||
review := &Review{InputEpoch: 1, InputHash: automatic.InputHash, Fields: Fields{Coverage: "covered", Correctness: "correct", ProductCategory: "MG400"}}
|
||||
view := Project(automatic.InputHash, 2, 1, automatic, review)
|
||||
if view.State != "stale" || view.Effective.Coverage != "uncovered" || view.Effective.Correctness != "" {
|
||||
t.Fatal("old conclusion remained effective after input epoch changed")
|
||||
}
|
||||
review.InputEpoch = 2
|
||||
view = Project(automatic.InputHash, 2, 1, automatic, review)
|
||||
if view.State != "reviewed" || view.Effective.Coverage != "covered" || view.Automatic.Result.Verdict != "uncovered" {
|
||||
t.Fatal("human and immutable rule not separated")
|
||||
}
|
||||
summary := Summarize(map[string]View{"a": view, "b": Project(automatic.InputHash, 3, 1, automatic, review), "c": Project(automatic.InputHash, 1, 0, automatic, nil)})
|
||||
if summary.Total != 3 || summary.Reviewed != 1 || summary.Stale != 1 || summary.Unreviewed != 1 || summary.PendingConfirmation != 0 {
|
||||
t.Fatal("quality counts invent rates or unknown judgments")
|
||||
}
|
||||
}
|
||||
func TestReviewRequiresExplicitOutcomesAndEvidence(t *testing.T) {
|
||||
valid := SaveRequest{InputHash: strings.Repeat("a", 64), InputEpoch: 1, Nonce: strings.Repeat("n", 32), Fields: Fields{Coverage: "uncovered", Correctness: "pending"}, Reason: "人工核对引用与答案，证据不足"}
|
||||
if Validate(valid) != nil {
|
||||
t.Fatal("valid review rejected")
|
||||
}
|
||||
for _, field := range []string{"reason", "hash", "coverage", "correctness", "root", "exclusion"} {
|
||||
bad := valid
|
||||
switch field {
|
||||
case "reason":
|
||||
bad.Reason = " "
|
||||
case "hash":
|
||||
bad.InputHash = "malformed"
|
||||
case "coverage":
|
||||
bad.Fields.Coverage = ""
|
||||
case "correctness":
|
||||
bad.Fields.Correctness = ""
|
||||
case "root":
|
||||
bad.Fields.Correctness = "correct"
|
||||
bad.Fields.RootCause = "知识缺失"
|
||||
case "exclusion":
|
||||
bad.Fields.Coverage = "excluded"
|
||||
}
|
||||
if Validate(bad) == nil {
|
||||
t.Fatal("invalid review accepted", field)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package qaopsquality
|
||||
|
||||
type Rate struct {
|
||||
Numerator int `json:"numerator"`
|
||||
Denominator int `json:"denominator"`
|
||||
Percent *float64 `json:"percent"`
|
||||
}
|
||||
type RateBreakdown struct {
|
||||
Covered int `json:"covered"`
|
||||
Uncovered int `json:"uncovered"`
|
||||
Partial int `json:"partial"`
|
||||
Pending int `json:"pending"`
|
||||
Unassessed int `json:"unassessed"`
|
||||
Excluded int `json:"excluded"`
|
||||
Stale int `json:"stale"`
|
||||
CoveredCorrect int `json:"covered_correct"`
|
||||
CoveredIncorrect int `json:"covered_incorrect"`
|
||||
CoveredPending int `json:"covered_pending"`
|
||||
CoveredUnassessed int `json:"covered_unassessed"`
|
||||
}
|
||||
type Rates struct {
|
||||
PolicyVersion string `json:"policy_version"`
|
||||
Coverage Rate `json:"coverage"`
|
||||
Correctness Rate `json:"correctness"`
|
||||
Breakdown RateBreakdown `json:"breakdown"`
|
||||
}
|
||||
|
||||
func ComputeRates(views map[string]View) Rates {
|
||||
out := Rates{PolicyVersion: "quality-rates-human-first-v1"}
|
||||
b := &out.Breakdown
|
||||
for _, v := range views {
|
||||
if v.State == "stale" {
|
||||
b.Stale++
|
||||
continue
|
||||
}
|
||||
switch v.Effective.Coverage {
|
||||
case "covered":
|
||||
b.Covered++
|
||||
switch v.Effective.Correctness {
|
||||
case "correct":
|
||||
b.CoveredCorrect++
|
||||
case "incorrect":
|
||||
b.CoveredIncorrect++
|
||||
case "pending":
|
||||
b.CoveredPending++
|
||||
default:
|
||||
b.CoveredUnassessed++
|
||||
}
|
||||
case "uncovered":
|
||||
b.Uncovered++
|
||||
case "partial":
|
||||
b.Partial++
|
||||
case "pending":
|
||||
b.Pending++
|
||||
case "excluded":
|
||||
b.Excluded++
|
||||
default:
|
||||
b.Unassessed++
|
||||
}
|
||||
}
|
||||
out.Coverage = makeRate(b.Covered, b.Covered+b.Uncovered+b.Partial)
|
||||
out.Correctness = makeRate(b.CoveredCorrect, b.CoveredCorrect+b.CoveredIncorrect)
|
||||
return out
|
||||
}
|
||||
func makeRate(numerator, denominator int) Rate {
|
||||
r := Rate{Numerator: numerator, Denominator: denominator}
|
||||
if denominator > 0 {
|
||||
p := 100 * float64(numerator) / float64(denominator)
|
||||
r.Percent = &p
|
||||
}
|
||||
return r
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package qaopsquality
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestQualityRatesUseEffectiveCurrentConclusionsAndExplicitDenominators(t *testing.T) {
|
||||
views := map[string]View{}
|
||||
for i, fields := range []Fields{{Coverage: "covered", Correctness: "correct"}, {Coverage: "covered", Correctness: "incorrect"}, {Coverage: "covered", Correctness: "pending"}, {Coverage: "uncovered"}, {Coverage: "partial", Correctness: "correct"}, {Coverage: "pending", Correctness: "correct"}, {}, {Coverage: "excluded", Correctness: "pending"}, {Coverage: "covered", Correctness: "correct"}, {Coverage: "covered"}} {
|
||||
state := "reviewed"
|
||||
if i == 3 || i == 6 {
|
||||
state = "unreviewed"
|
||||
}
|
||||
if i == 8 {
|
||||
state = "stale"
|
||||
}
|
||||
views[string(rune('a'+i))] = View{State: state, Effective: fields, Automatic: Check{Result: dobotaiops.CheckAnswerCoverage("暂无已发布")}}
|
||||
}
|
||||
r := ComputeRates(views)
|
||||
if r.PolicyVersion != "quality-rates-human-first-v1" || r.Coverage.Numerator != 4 || r.Coverage.Denominator != 6 || r.Correctness.Numerator != 1 || r.Correctness.Denominator != 2 {
|
||||
t.Fatal("effective decisions, partial or exclusion scopes incorrect", r)
|
||||
}
|
||||
if r.Coverage.Percent == nil || math.Abs(*r.Coverage.Percent-100*float64(4)/6) > 1e-10 || r.Correctness.Percent == nil || *r.Correctness.Percent != 50 {
|
||||
t.Fatal("ratio value disagrees with raw counts")
|
||||
}
|
||||
b := r.Breakdown
|
||||
if b.Covered+b.Uncovered+b.Partial+b.Pending+b.Unassessed+b.Excluded+b.Stale != len(views) || b.CoveredCorrect+b.CoveredIncorrect+b.CoveredPending+b.CoveredUnassessed != b.Covered {
|
||||
t.Fatal("denominator breakdown not conserved")
|
||||
}
|
||||
if got := Summarize(views); got.Rates.Coverage.Denominator != 6 {
|
||||
t.Fatal("scope summary did not consume the one shared calculation")
|
||||
}
|
||||
}
|
||||
func TestQualityRatesEmptyScopeHasNullPercent(t *testing.T) {
|
||||
r := ComputeRates(map[string]View{})
|
||||
raw, _ := json.Marshal(r)
|
||||
if r.Coverage.Percent != nil || r.Correctness.Percent != nil || strings.Count(string(raw), `"percent":null`) != 2 {
|
||||
t.Fatal("zero denominator fabricated zero or full rate")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,322 @@
|
||||
package qaopsquality
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
func (s *Store) Columns(ctx context.Context, keys []dobotaiops.QualityKey) ([]string, error) {
|
||||
if !s.available() {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
if len(keys) > 100000 {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
for _, key := range keys {
|
||||
if !validID(key.ID) || !hashPattern.MatchString(key.InputHash) {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
}
|
||||
raw, err := json.Marshal(keys)
|
||||
if err != nil {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
if len(keys) == 0 {
|
||||
return []string{}, nil
|
||||
}
|
||||
var coverage, correctness, product, problem, root bool
|
||||
err = s.pool.QueryRow(ctx, `SELECT COALESCE(bool_or(r.fields->>'coverage'<>''),false),COALESCE(bool_or(r.fields->>'correctness'<>''),false),COALESCE(bool_or(r.fields->>'product_category'<>''),false),COALESCE(bool_or(r.fields->>'problem_type'<>''),false),COALESCE(bool_or(r.fields->>'root_cause'<>''),false) FROM jsonb_to_recordset($2::jsonb) k(id text,input_hash text) JOIN qa_ops_quality_records h ON h.tenant_key=$1 AND h.record_id=k.id AND h.input_hash=k.input_hash JOIN qa_ops_quality_reviews r ON r.tenant_key=h.tenant_key AND r.record_id=h.record_id AND r.revision=h.review_revision AND r.input_epoch=h.input_epoch`, s.tenant, raw).Scan(&coverage, &correctness, &product, &problem, &root)
|
||||
if err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
out := []string{}
|
||||
for _, c := range []struct {
|
||||
key string
|
||||
yes bool
|
||||
}{{"coverage", coverage}, {"is_correct", correctness}, {"product_category", product}, {"problem_type", problem}, {"root_cause", root}} {
|
||||
if c.yes {
|
||||
out = append(out, c.key)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (s *Store) Snapshot(ctx context.Context, id string, epoch int64) (Snapshot, error) {
|
||||
if !s.available() {
|
||||
return Snapshot{}, ErrUnavailable
|
||||
}
|
||||
if !validID(id) || epoch < 1 {
|
||||
return Snapshot{}, ErrInvalid
|
||||
}
|
||||
var value Snapshot
|
||||
value.InputEpoch = epoch
|
||||
err := s.pool.QueryRow(ctx, `SELECT input_hash,payload,created_at FROM qa_ops_quality_inputs WHERE tenant_key=$1 AND record_id=$2 AND input_epoch=$3`, s.tenant, id, epoch).Scan(&value.InputHash, &value.Payload, &value.CreatedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return Snapshot{}, ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return Snapshot{}, ErrUnavailable
|
||||
}
|
||||
if len(value.Payload) == 0 {
|
||||
return Snapshot{}, ErrSnapshotMissing
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
func (s *Store) available() bool { return s != nil && s.pool != nil && validID(s.tenant) }
|
||||
func (s *Store) observeLocked(ctx context.Context, tx pgx.Tx, e Evidence) error {
|
||||
id := e.Record.ID
|
||||
if _, err := tx.Exec(ctx, `INSERT INTO qa_ops_quality_records(tenant_key,record_id) VALUES($1,$2) ON CONFLICT DO NOTHING`, s.tenant, id); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
var previous string
|
||||
var epoch int64
|
||||
var observedAt *time.Time
|
||||
if tx.QueryRow(ctx, `SELECT input_hash,input_epoch,observed_at FROM qa_ops_quality_records WHERE tenant_key=$1 AND record_id=$2 FOR UPDATE`, s.tenant, id).Scan(&previous, &epoch, &observedAt) != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
// Source transaction time orders concurrent page/detail/scope observations.
|
||||
// A late different input is rejected, never projected against a newer answer.
|
||||
if observedAt != nil && !e.Record.QualityObservedAt.IsZero() && e.Record.QualityObservedAt.Before(*observedAt) && previous != e.Hash {
|
||||
return ErrConflict
|
||||
}
|
||||
if !e.Record.QualityObservedAt.IsZero() {
|
||||
if _, err := tx.Exec(ctx, `UPDATE qa_ops_quality_records SET observed_at=$3 WHERE tenant_key=$1 AND record_id=$2 AND(observed_at IS NULL OR observed_at<$3)`, s.tenant, id, e.Record.QualityObservedAt); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
}
|
||||
var payload any
|
||||
if len(e.Payload) > 0 {
|
||||
payload = e.Payload
|
||||
}
|
||||
if previous != e.Hash {
|
||||
epoch++
|
||||
if _, err := tx.Exec(ctx, `INSERT INTO qa_ops_quality_inputs(tenant_key,record_id,input_epoch,input_hash,payload) VALUES($1,$2,$3,$4,$5)`, s.tenant, id, epoch, e.Hash, payload); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
tag, err := tx.Exec(ctx, `UPDATE qa_ops_quality_records SET input_hash=$3,input_epoch=$4 WHERE tenant_key=$1 AND record_id=$2`, s.tenant, id, e.Hash, epoch)
|
||||
if err != nil || tag.RowsAffected() != 1 {
|
||||
return ErrUnavailable
|
||||
}
|
||||
}
|
||||
if payload != nil {
|
||||
if _, err := tx.Exec(ctx, `UPDATE qa_ops_quality_inputs SET payload=$4 WHERE tenant_key=$1 AND record_id=$2 AND input_epoch=$3 AND payload IS NULL`, s.tenant, id, epoch, payload); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
}
|
||||
automatic, _ := json.Marshal(e.Result)
|
||||
source, _ := json.Marshal(e.Record.SourceCoverage)
|
||||
if _, err := tx.Exec(ctx, `INSERT INTO qa_ops_quality_checks(tenant_key,record_id,input_epoch,rule_version,result,source) VALUES($1,$2,$3,$4,$5,$6) ON CONFLICT(tenant_key,record_id,input_epoch,rule_version) DO NOTHING`, s.tenant, id, epoch, dobotaiops.CoverageRuleVersion, automatic, source); err != nil {
|
||||
return ErrUnavailable
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
const checkObject = `jsonb_build_object('id',c.id,'input_epoch',c.input_epoch,'input_hash',i.input_hash,'result',c.result,'source',c.source,'created_at',c.created_at,'input_available',i.payload IS NOT NULL)`
|
||||
const reviewObject = `jsonb_build_object('revision',r.revision,'input_epoch',r.input_epoch,'input_hash',r.input_hash,'fields',r.fields,'reason',r.reason,'reviewer',r.reviewer,'created_at',r.created_at)`
|
||||
|
||||
func (s *Store) view(ctx context.Context, tx pgx.Tx, id string) (View, error) {
|
||||
var hash, checkRaw, reviewRaw string
|
||||
var epoch, revision int64
|
||||
err := tx.QueryRow(ctx, `SELECT h.input_hash,h.input_epoch,h.review_revision,`+checkObject+`::text,COALESCE((SELECT `+reviewObject+` FROM qa_ops_quality_reviews r WHERE r.tenant_key=h.tenant_key AND r.record_id=h.record_id AND r.revision=h.review_revision),'null'::jsonb)::text FROM qa_ops_quality_records h JOIN qa_ops_quality_inputs i ON i.tenant_key=h.tenant_key AND i.record_id=h.record_id AND i.input_epoch=h.input_epoch JOIN qa_ops_quality_checks c ON c.tenant_key=h.tenant_key AND c.record_id=h.record_id AND c.input_epoch=h.input_epoch AND c.rule_version=$3 WHERE h.tenant_key=$1 AND h.record_id=$2`, s.tenant, id, dobotaiops.CoverageRuleVersion).Scan(&hash, &epoch, &revision, &checkRaw, &reviewRaw)
|
||||
var automatic Check
|
||||
var review *Review
|
||||
if err != nil || json.Unmarshal([]byte(checkRaw), &automatic) != nil || json.Unmarshal([]byte(reviewRaw), &review) != nil {
|
||||
return View{}, ErrUnavailable
|
||||
}
|
||||
return Project(hash, epoch, revision, automatic, review), nil
|
||||
}
|
||||
func (s *Store) Observe(ctx context.Context, records []dobotaiops.Record) (map[string]View, error) {
|
||||
if len(records) > 5000 {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
items := make([]Evidence, 0, len(records))
|
||||
for _, record := range records {
|
||||
e, err := EvidenceFor(record)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, e)
|
||||
}
|
||||
return s.observeEvidence(ctx, items)
|
||||
}
|
||||
func (s *Store) ObserveKeys(ctx context.Context, keys []dobotaiops.QualityKey) (map[string]View, error) {
|
||||
if len(keys) > 5000 {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
items := make([]Evidence, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
if !validID(key.ID) || !hashPattern.MatchString(key.InputHash) {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
items = append(items, Evidence{Hash: key.InputHash, Record: dobotaiops.Record{ID: key.ID, SourceCoverage: key.Source, QualityObservedAt: key.ObservedAt}, Result: dobotaiops.CheckMarkerCoverage(key.Marker)})
|
||||
}
|
||||
return s.observeEvidence(ctx, items)
|
||||
}
|
||||
func (s *Store) observeEvidence(ctx context.Context, items []Evidence) (map[string]View, error) {
|
||||
if !s.available() {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
evidence := map[string]Evidence{}
|
||||
ids := []string{}
|
||||
bytes := 0
|
||||
for _, e := range items {
|
||||
id := e.Record.ID
|
||||
if old, ok := evidence[id]; ok {
|
||||
if old.Hash != e.Hash || old.Result != e.Result {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
continue
|
||||
}
|
||||
evidence[id] = e
|
||||
ids = append(ids, id)
|
||||
bytes += len(e.Payload)
|
||||
if bytes > 32<<20 {
|
||||
return nil, ErrInvalid
|
||||
}
|
||||
}
|
||||
sort.Strings(ids)
|
||||
result := map[string]View{}
|
||||
if len(ids) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
for _, id := range ids {
|
||||
if err = s.observeLocked(ctx, tx, evidence[id]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, err := s.view(ctx, tx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[id] = v
|
||||
}
|
||||
if tx.Commit(ctx) != nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
func (s *Store) Save(ctx context.Context, record dobotaiops.Record, request SaveRequest, actor string) (Receipt, error) {
|
||||
if !s.available() {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
if Validate(request) != nil || !validID(actor) {
|
||||
return Receipt{}, ErrInvalid
|
||||
}
|
||||
e, err := EvidenceFor(record)
|
||||
if err != nil {
|
||||
return Receipt{}, err
|
||||
}
|
||||
payload, _ := json.Marshal(request)
|
||||
sum := sha256.Sum256(payload)
|
||||
requestHash := hex.EncodeToString(sum[:])
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
if err = s.observeLocked(ctx, tx, e); err != nil {
|
||||
return Receipt{}, err
|
||||
}
|
||||
var existingHash, existingActor string
|
||||
var savedRevision int64
|
||||
err = tx.QueryRow(ctx, `SELECT request_hash,reviewer,revision FROM qa_ops_quality_reviews WHERE tenant_key=$1 AND record_id=$2 AND nonce=$3`, s.tenant, record.ID, request.Nonce).Scan(&existingHash, &existingActor, &savedRevision)
|
||||
replay := err == nil
|
||||
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
current, err := s.view(ctx, tx, record.ID)
|
||||
if err != nil {
|
||||
return Receipt{}, err
|
||||
}
|
||||
if replay {
|
||||
if existingHash != requestHash || existingActor != actor {
|
||||
return Receipt{}, ErrConflict
|
||||
}
|
||||
} else {
|
||||
if current.InputHash != request.InputHash || current.InputEpoch != request.InputEpoch || current.Revision != request.BaseRevision {
|
||||
if tx.Commit(ctx) != nil {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
return Receipt{}, ErrConflict
|
||||
}
|
||||
savedRevision = current.Revision + 1
|
||||
fields, _ := json.Marshal(request.Fields)
|
||||
if _, err = tx.Exec(ctx, `INSERT INTO qa_ops_quality_reviews(tenant_key,record_id,revision,input_epoch,input_hash,fields,reason,reviewer,nonce,request_hash) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)`, s.tenant, record.ID, savedRevision, current.InputEpoch, current.InputHash, fields, request.Reason, actor, request.Nonce, requestHash); err != nil {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
tag, err := tx.Exec(ctx, `UPDATE qa_ops_quality_records SET review_revision=$3 WHERE tenant_key=$1 AND record_id=$2`, s.tenant, record.ID, savedRevision)
|
||||
if err != nil || tag.RowsAffected() != 1 {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
current, err = s.view(ctx, tx, record.ID)
|
||||
if err != nil {
|
||||
return Receipt{}, err
|
||||
}
|
||||
}
|
||||
if tx.Commit(ctx) != nil {
|
||||
return Receipt{}, ErrUnavailable
|
||||
}
|
||||
return Receipt{View: current, SavedRevision: savedRevision, Replayed: replay}, nil
|
||||
}
|
||||
func (s *Store) History(ctx context.Context, id string, reviewBefore, checkBefore int64) (History, error) {
|
||||
result := History{Reviews: []Review{}, Checks: []Check{}}
|
||||
if !s.available() {
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
if !validID(id) || reviewBefore < 0 || checkBefore < 0 {
|
||||
return result, ErrInvalid
|
||||
}
|
||||
rows, err := s.pool.Query(ctx, `SELECT `+reviewObject+`::text FROM qa_ops_quality_reviews r WHERE tenant_key=$1 AND record_id=$2 AND ($3::bigint=0 OR revision<$3) ORDER BY revision DESC LIMIT 21`, s.tenant, id, reviewBefore)
|
||||
if err != nil {
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
for rows.Next() {
|
||||
var raw string
|
||||
var value Review
|
||||
if rows.Scan(&raw) != nil || json.Unmarshal([]byte(raw), &value) != nil {
|
||||
rows.Close()
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
result.Reviews = append(result.Reviews, value)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
if len(result.Reviews) > 20 {
|
||||
result.Reviews = result.Reviews[:20]
|
||||
result.NextReview = result.Reviews[19].Revision
|
||||
}
|
||||
rows, err = s.pool.Query(ctx, `SELECT `+checkObject+`::text FROM qa_ops_quality_checks c JOIN qa_ops_quality_inputs i USING(tenant_key,record_id,input_epoch) WHERE c.tenant_key=$1 AND c.record_id=$2 AND ($3::bigint=0 OR c.id<$3) ORDER BY c.id DESC LIMIT 21`, s.tenant, id, checkBefore)
|
||||
if err != nil {
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var raw string
|
||||
var value Check
|
||||
if rows.Scan(&raw) != nil || json.Unmarshal([]byte(raw), &value) != nil {
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
result.Checks = append(result.Checks, value)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
return result, ErrUnavailable
|
||||
}
|
||||
if len(result.Checks) > 20 {
|
||||
result.Checks = result.Checks[:20]
|
||||
result.NextCheck = result.Checks[19].ID
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
package qaopsquality_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"knowledge-governance/internal/dobotaiops"
|
||||
pg "knowledge-governance/internal/platform/postgres"
|
||||
"knowledge-governance/internal/qaopsquality"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestQualityStoreImmutableHistoryReplayConcurrencyAndStaleness(t *testing.T) {
|
||||
dsn := os.Getenv("KG_QAOPS_QUALITY_TEST_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("explicit disposable quality DB required")
|
||||
}
|
||||
u, e := url.Parse(dsn)
|
||||
if e != nil || u.Hostname() != "127.0.0.1" || !strings.HasPrefix(u.Path, "/kg_qaops_quality_test_") || os.Getenv("KG_QAOPS_QUALITY_TEST_ISOLATED") != "1" {
|
||||
t.Fatal("refusing shared quality database")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, e := pgxpool.New(ctx, dsn)
|
||||
if e != nil {
|
||||
t.Fatal("fixture connection failed")
|
||||
}
|
||||
defer pool.Close()
|
||||
if e = pg.ApplyMigrations(ctx, pool); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
store := qaopsquality.NewStore(pool, "fixture-tenant")
|
||||
record := dobotaiops.Record{ID: "fixture-turn", Status: "succeeded", Question: "示例", Answer: "暂无已发布", AskedAt: time.Now()}
|
||||
views, e := store.Observe(ctx, []dobotaiops.Record{record})
|
||||
if e != nil {
|
||||
t.Fatal("source observation not persisted", e)
|
||||
}
|
||||
first := views[record.ID]
|
||||
again, e := store.Observe(ctx, []dobotaiops.Record{record})
|
||||
if e != nil || again[record.ID].Automatic.ID != first.Automatic.ID {
|
||||
t.Fatal("repeated GET duplicated immutable checks")
|
||||
}
|
||||
metaKey := dobotaiops.QualityKey{ID: "metadata-only-turn", InputHash: strings.Repeat("f", 64), Marker: true}
|
||||
meta, me := store.ObserveKeys(ctx, []dobotaiops.QualityKey{metaKey})
|
||||
if me != nil || meta[metaKey.ID].Automatic.InputAvailable {
|
||||
t.Fatal("compact scope observation not stored honestly", me)
|
||||
}
|
||||
if _, me = store.Snapshot(ctx, metaKey.ID, 1); !errors.Is(me, qaopsquality.ErrSnapshotMissing) {
|
||||
t.Fatal("metadata-only input fabricated an empty original")
|
||||
}
|
||||
full, me := store.Observe(ctx, []dobotaiops.Record{{ID: metaKey.ID, QualityInputHash: metaKey.InputHash, Question: "示例", Answer: "暂无已发布"}})
|
||||
if me != nil || !full[metaKey.ID].Automatic.InputAvailable || full[metaKey.ID].Automatic.ID != meta[metaKey.ID].Automatic.ID {
|
||||
t.Fatal("reading full content duplicated/replaced the same check")
|
||||
}
|
||||
req := qaopsquality.SaveRequest{InputHash: first.InputHash, InputEpoch: first.InputEpoch, BaseRevision: first.Revision, Nonce: strings.Repeat("n", 32), Fields: qaopsquality.Fields{Coverage: "covered", Correctness: "correct", ProductCategory: "MG400", ProblemType: "咨询"}, Reason: "人工核对有效引用"}
|
||||
saved, e := store.Save(ctx, record, req, "fixture-admin")
|
||||
if e != nil || saved.View.State != "reviewed" || saved.View.Effective.Coverage != "covered" || saved.View.Automatic.Result.Verdict != "uncovered" || saved.View.LatestReview.Reviewer != "fixture-admin" {
|
||||
t.Fatal("manual override lost audit/rule separation", e)
|
||||
}
|
||||
cols, ce := store.Columns(ctx, []dobotaiops.QualityKey{{ID: record.ID, InputHash: first.InputHash}})
|
||||
if ce != nil || len(cols) != 4 {
|
||||
t.Fatal("review fields absent from whole-scope projection", ce)
|
||||
}
|
||||
replay, e := store.Save(ctx, record, req, "fixture-admin")
|
||||
if e != nil || !replay.Replayed || replay.SavedRevision != saved.SavedRevision {
|
||||
t.Fatal("retry duplicated review")
|
||||
}
|
||||
bad := req
|
||||
bad.Reason = "changed body"
|
||||
if _, e = store.Save(ctx, record, bad, "fixture-admin"); !errors.Is(e, qaopsquality.ErrConflict) {
|
||||
t.Fatal("nonce reused with different request")
|
||||
}
|
||||
if _, e = store.Save(ctx, record, req, "other-admin"); !errors.Is(e, qaopsquality.ErrConflict) {
|
||||
t.Fatal("nonce replay changed operator")
|
||||
}
|
||||
record.Answer = "新答案"
|
||||
changed, e := store.Observe(ctx, []dobotaiops.Record{record})
|
||||
if e != nil || changed[record.ID].State != "stale" || changed[record.ID].Effective.Correctness != "" {
|
||||
t.Fatal("old manual result silently survived change")
|
||||
}
|
||||
if _, e = store.Save(ctx, record, req, "fixture-admin"); e != nil {
|
||||
t.Fatal("old completed request should replay without a new review")
|
||||
}
|
||||
record.Answer = "暂无已发布"
|
||||
reverted, e := store.Observe(ctx, []dobotaiops.Record{record})
|
||||
v := reverted[record.ID]
|
||||
if e != nil || v.InputHash != first.InputHash || v.InputEpoch <= first.InputEpoch || v.State != "stale" {
|
||||
t.Fatal("answer revert reactivated old judgment")
|
||||
}
|
||||
req.InputHash = v.InputHash
|
||||
req.InputEpoch = v.InputEpoch
|
||||
req.BaseRevision = v.Revision
|
||||
start := make(chan struct{})
|
||||
results := make(chan error, 2)
|
||||
for _, nonce := range []string{strings.Repeat("a", 32), strings.Repeat("b", 32)} {
|
||||
go func(nonce string) {
|
||||
q := req
|
||||
q.Nonce = nonce
|
||||
<-start
|
||||
_, err := store.Save(ctx, record, q, "fixture-admin")
|
||||
results <- err
|
||||
}(nonce)
|
||||
}
|
||||
close(start)
|
||||
success, conflict := 0, 0
|
||||
for range 2 {
|
||||
e = <-results
|
||||
if e == nil {
|
||||
success++
|
||||
} else if errors.Is(e, qaopsquality.ErrConflict) {
|
||||
conflict++
|
||||
} else {
|
||||
t.Fatal(e)
|
||||
}
|
||||
}
|
||||
if success != 1 || conflict != 1 {
|
||||
t.Fatal("concurrent reviewers overwrote each other")
|
||||
}
|
||||
h, e := store.History(ctx, record.ID, 0, 0)
|
||||
if e != nil || len(h.Reviews) != 2 || len(h.Checks) != 3 || h.Checks[0].InputEpoch != 3 {
|
||||
t.Fatal("history missing/replaced", e)
|
||||
}
|
||||
oldInput, oe := store.Snapshot(ctx, record.ID, 1)
|
||||
if oe != nil || oldInput.InputHash != first.InputHash || !strings.Contains(string(oldInput.Payload), "暂无已发布") {
|
||||
t.Fatal("old assessment evidence not readable")
|
||||
}
|
||||
// A slow older source scan must not roll back a newer observed input.
|
||||
ordered := dobotaiops.Record{ID: "ordered-source-turn", Answer: "旧输入", QualityObservedAt: time.Now().Add(-time.Minute)}
|
||||
oldEvidence, _ := qaopsquality.EvidenceFor(ordered)
|
||||
if _, e = store.Observe(ctx, []dobotaiops.Record{ordered}); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
ordered.Answer = "新输入"
|
||||
ordered.QualityObservedAt = ordered.QualityObservedAt.Add(time.Second)
|
||||
newViews, ne := store.Observe(ctx, []dobotaiops.Record{ordered})
|
||||
if ne != nil {
|
||||
t.Fatal(ne)
|
||||
}
|
||||
newView := newViews[ordered.ID]
|
||||
if _, e = store.ObserveKeys(ctx, []dobotaiops.QualityKey{{ID: ordered.ID, InputHash: oldEvidence.Hash, ObservedAt: ordered.QualityObservedAt.Add(-time.Second)}}); !errors.Is(e, qaopsquality.ErrConflict) {
|
||||
t.Fatal("late old source scan rolled back the current input")
|
||||
}
|
||||
current, ce := store.Observe(ctx, []dobotaiops.Record{ordered})
|
||||
if ce != nil || current[ordered.ID].InputEpoch != newView.InputEpoch {
|
||||
t.Fatal("stale scan created extra input epochs")
|
||||
}
|
||||
// Independent cursors must enumerate more than one page without loss.
|
||||
paging := dobotaiops.Record{ID: "history-paging-turn", Answer: "示例"}
|
||||
pageSeen := map[int64]bool{}
|
||||
checkSeen := map[int64]bool{}
|
||||
for i := 0; i < 23; i++ {
|
||||
paging.Answer = fmt.Sprintf("示例输入%d", i)
|
||||
vv, ee := store.Observe(ctx, []dobotaiops.Record{paging})
|
||||
if ee != nil {
|
||||
t.Fatal(ee)
|
||||
}
|
||||
value := vv[paging.ID]
|
||||
_, ee = store.Save(ctx, paging, qaopsquality.SaveRequest{InputHash: value.InputHash, InputEpoch: value.InputEpoch, BaseRevision: value.Revision, Nonce: fmt.Sprintf("history_%020d", i), Fields: qaopsquality.Fields{Coverage: "pending", Correctness: "pending"}, Reason: "隔离分页验证"}, "fixture-admin")
|
||||
if ee != nil {
|
||||
t.Fatal(ee)
|
||||
}
|
||||
}
|
||||
rc, cc := int64(0), int64(0)
|
||||
for {
|
||||
page, ee := store.History(ctx, paging.ID, rc, cc)
|
||||
if ee != nil {
|
||||
t.Fatal(ee)
|
||||
}
|
||||
for _, r := range page.Reviews {
|
||||
if pageSeen[r.Revision] {
|
||||
t.Fatal("duplicate history review")
|
||||
}
|
||||
pageSeen[r.Revision] = true
|
||||
}
|
||||
for _, c := range page.Checks {
|
||||
if checkSeen[c.ID] {
|
||||
t.Fatal("duplicate history check")
|
||||
}
|
||||
checkSeen[c.ID] = true
|
||||
}
|
||||
rc, cc = page.NextReview, page.NextCheck
|
||||
if rc == 0 && cc == 0 {
|
||||
break
|
||||
}
|
||||
if rc == 0 || cc == 0 {
|
||||
t.Fatal("unexpected fixture cursor shape")
|
||||
}
|
||||
}
|
||||
if len(pageSeen) != 23 || len(checkSeen) != 23 {
|
||||
t.Fatal("history cursor omitted immutable rows")
|
||||
}
|
||||
freshPool, fe := pgxpool.New(ctx, dsn)
|
||||
if fe != nil {
|
||||
t.Fatal(fe)
|
||||
}
|
||||
persisted, fe := qaopsquality.NewStore(freshPool, "fixture-tenant").History(ctx, record.ID, 0, 0)
|
||||
freshPool.Close()
|
||||
if fe != nil || len(persisted.Reviews) != 2 || len(persisted.Checks) != 3 {
|
||||
t.Fatal("history did not survive a new connection pool")
|
||||
}
|
||||
// History survives a fresh store/connection, not merely an in-memory cache.
|
||||
if h, e = qaopsquality.NewStore(pool, "other-tenant").History(ctx, record.ID, 0, 0); e != nil || len(h.Reviews) != 0 || len(h.Checks) != 0 {
|
||||
t.Fatal("quality history crossed tenants")
|
||||
}
|
||||
if e = pg.RollbackLatestMigration(ctx, pool); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
var gone bool
|
||||
if pool.QueryRow(ctx, `SELECT to_regclass('qa_ops_quality_records') IS NULL AND to_regclass('qa_ops_quality_inputs') IS NULL AND to_regclass('qa_ops_quality_checks') IS NULL AND to_regclass('qa_ops_quality_reviews') IS NULL`).Scan(&gone) != nil || !gone {
|
||||
t.Fatal("quality down migration left data tables")
|
||||
}
|
||||
if e = pg.ApplyMigrations(ctx, pool); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package versionhistory
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"knowledge-governance/internal/governance"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func (s *Syncer) includeDeliveredCommits(ctx context.Context, commits []RawCommit, tags []RawTag) ([]RawCommit, error) {
|
||||
seen := map[string]bool{}
|
||||
for _, c := range commits {
|
||||
seen[c.SHA] = true
|
||||
}
|
||||
refs := append([]string{}, s.targetSHAs...)
|
||||
snapshots := []RawTag{}
|
||||
for _, tag := range tags {
|
||||
if tag.Annotated && strings.HasPrefix(tag.Name, "snapshot/v") && governance.SemverValid(strings.TrimPrefix(tag.Name, "snapshot/v")) {
|
||||
snapshots = append(snapshots, tag)
|
||||
}
|
||||
}
|
||||
sort.Slice(snapshots, func(i, j int) bool {
|
||||
return governance.CompareSemver(strings.TrimPrefix(snapshots[i].Name, "snapshot/v"), strings.TrimPrefix(snapshots[j].Name, "snapshot/v")) > 0
|
||||
})
|
||||
for _, tag := range snapshots {
|
||||
refs = append(refs, tag.TargetSHA)
|
||||
}
|
||||
for _, ref := range refs {
|
||||
if seen[ref] {
|
||||
continue
|
||||
}
|
||||
if !fullSHA.MatchString(ref) {
|
||||
return nil, fmt.Errorf("开发快照缺少完整提交身份")
|
||||
}
|
||||
extra, total, e := s.fetchCommitsAt(ctx, ref)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if total > 0 && len(extra) != total {
|
||||
return nil, codeError{code: ErrorCodeCountMismatch, message: "开发快照提交历史不完整"}
|
||||
}
|
||||
found := false
|
||||
for _, c := range extra {
|
||||
if c.SHA == ref {
|
||||
found = true
|
||||
}
|
||||
if !seen[c.SHA] {
|
||||
seen[c.SHA] = true
|
||||
commits = append(commits, c)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return nil, codeError{code: ErrorCodeIncompletePages, message: "开发快照目标没有被取回"}
|
||||
}
|
||||
}
|
||||
return commits, nil
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package versionhistory
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"knowledge-governance/internal/governance"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestGiteaSnapshotRefIsExplicitAndNotesArePreserved(t *testing.T) {
|
||||
server := fakeGitea(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/api/v1/repos/Weknora/knowledge-governance/commits" {
|
||||
if r.URL.Query().Get("sha") != shaB {
|
||||
t.Error("snapshot commit history silently used default branch")
|
||||
}
|
||||
fmt.Fprintf(w, `[{"sha":%q,"parents":[],"commit":{"message":"fixture"}}]`, shaB)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/api/v1/repos/Weknora/knowledge-governance/git/refs/tags" {
|
||||
fmt.Fprintf(w, `[{"ref":"refs/tags/snapshot/v0.1.1","object":{"type":"tag","sha":%q}}]`, shaT)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(w, `{"message":"development notes","object":{"type":"commit","sha":%q}}`, shaB)
|
||||
})
|
||||
c, _ := NewGiteaClient(validConfig(server.URL))
|
||||
if _, _, e := c.ListCommitsAtRef(context.Background(), shaB, 1, 50); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
tags, e := c.ListTags(context.Background())
|
||||
if e != nil || len(tags) != 1 || tags[0].Message != "development notes" {
|
||||
t.Fatal("annotated snapshot notes lost", e)
|
||||
}
|
||||
}
|
||||
func TestDevelopmentVersionsHaveTheirOwnChangelogNotes(t *testing.T) {
|
||||
a, b := shaA, shaB
|
||||
now := time.Now()
|
||||
text := "# Changelog\n\n## [Unreleased]\n\n### Added\n\n- unrelated future work\n\n## 开发快照 snapshot/v0.1.1 - 2026-10-18\n\n### Fixed\n\n- fix specifically delivered here\n\n## [0.1.0] - 2026-10-13\n\n### Added\n\n- baseline\n"
|
||||
result := DeriveVersions(DeriveInput{Commits: []RawCommit{{SHA: a, CommittedAt: now}, {SHA: b, Parents: []string{a}, CommittedAt: now}}, Tags: []RawTag{{Name: "v0.1.0", TargetSHA: a, Annotated: true}}, Changelog: func(string) (string, error) { return text, nil }})
|
||||
for _, v := range result.Records {
|
||||
if v.CommitSHA == b && (v.ReleaseNotes != "### Fixed\n\n- fix specifically delivered here" || v.RecordKind != governance.RecordKindDevelopment) {
|
||||
t.Fatalf("development version lacks its own notes: %q", v.ReleaseNotes)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHistoricDevelopmentArchiveBindsExactCommit(t *testing.T) {
|
||||
archive := "## 开发快照 snapshot/v0.1.1 - 2026-10-18\n\n目标提交：" + shaB + "\n\n### Fixed\n\n- historic delivery notes\n"
|
||||
input := DeriveInput{DeliveryArchive: archive, Commits: []RawCommit{{SHA: shaA}, {SHA: shaB, Parents: []string{shaA}}}, Tags: []RawTag{{Name: "v0.1.0", TargetSHA: shaA, Annotated: true}}, Changelog: func(string) (string, error) { return "## [0.1.0] - 2026-10-13\n\n### Added\n\n- baseline", nil }}
|
||||
result := DeriveVersions(input)
|
||||
for _, v := range result.Records {
|
||||
if v.CommitSHA == shaB && !strings.Contains(v.ReleaseNotes, "historic delivery notes") {
|
||||
t.Fatal("archived notes missing")
|
||||
}
|
||||
}
|
||||
// Parallel branch histories may legitimately have the same derived number.
|
||||
input.DeliveryArchive = strings.ReplaceAll(archive, shaB, strings.Repeat("c", 40)) + "\n" + archive
|
||||
result = DeriveVersions(input)
|
||||
for _, v := range result.Records {
|
||||
if v.CommitSHA == shaB && !strings.Contains(v.ReleaseNotes, "historic delivery notes") {
|
||||
t.Fatal("same-number archive hid notes for exact commit")
|
||||
}
|
||||
}
|
||||
input.DeliveryArchive = strings.ReplaceAll(archive, shaB, strings.Repeat("c", 40))
|
||||
result = DeriveVersions(input)
|
||||
for _, v := range result.Records {
|
||||
if v.CommitSHA == shaB && v.ReleaseNotes != "" {
|
||||
t.Fatal("different commit reused same-number archive notes")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type snapshotRefSource struct {
|
||||
fakeSource
|
||||
other []RawCommit
|
||||
}
|
||||
|
||||
func (s *snapshotRefSource) ListCommitsAtRef(_ context.Context, _ string, page, limit int) ([]RawCommit, int, error) {
|
||||
if page == 1 {
|
||||
return s.other, len(s.other), nil
|
||||
}
|
||||
return nil, len(s.other), nil
|
||||
}
|
||||
func TestSyncIncludesDeliveredSnapshotOutsideMain(t *testing.T) {
|
||||
a := RawCommit{SHA: shaA}
|
||||
b := RawCommit{SHA: shaB, Parents: []string{shaA}}
|
||||
src := &snapshotRefSource{fakeSource: fakeSource{commits: []RawCommit{a}, total: 1, pageSize: 50, head: shaA, tags: []RawTag{{Name: "snapshot/v0.1.1", TargetSHA: shaB, Annotated: true}}}, other: []RawCommit{b, a}}
|
||||
store := &fakeStore{}
|
||||
run, e := newTestSyncer(t, src, store).Sync(context.Background(), "fixture", "admin")
|
||||
if e != nil || len(store.saved) != 1 || len(store.saved[0].records) != 2 || !run.Complete {
|
||||
t.Fatal("delivered non-main snapshot omitted", e, run.FetchedCount)
|
||||
}
|
||||
}
|
||||
@@ -188,6 +188,12 @@ var errGiteaNotFound = newGiteaError(KindNotFound, http.StatusNotFound, "版本
|
||||
|
||||
// ListCommits 读一页提交；同时返回 X-Total-Count（用于完整性对账，不做假 total）。
|
||||
func (c *GiteaClient) ListCommits(ctx context.Context, page, limit int) ([]RawCommit, int, error) {
|
||||
return c.ListCommitsAtRef(ctx, "main", page, limit)
|
||||
}
|
||||
func (c *GiteaClient) ListCommitsAtRef(ctx context.Context, ref string, page, limit int) ([]RawCommit, int, error) {
|
||||
if ref != "main" && !fullSHA.MatchString(ref) {
|
||||
return nil, 0, newGiteaError(KindParse, 0, "版本记录：只接受 main 或完整提交 SHA", nil)
|
||||
}
|
||||
if limit <= 0 || limit > giteaDefaultPageSize {
|
||||
limit = giteaDefaultPageSize
|
||||
}
|
||||
@@ -196,6 +202,7 @@ func (c *GiteaClient) ListCommits(ctx context.Context, page, limit int) ([]RawCo
|
||||
query.Set("page", fmt.Sprintf("%d", page))
|
||||
query.Set("stat", "false")
|
||||
query.Set("verification", "false")
|
||||
query.Set("sha", ref)
|
||||
body, header, err := c.get(ctx, c.apiURL("/commits", query))
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
@@ -322,7 +329,8 @@ func (c *GiteaClient) ListTags(ctx context.Context) ([]RawTag, error) {
|
||||
return nil, err
|
||||
}
|
||||
var annotated struct {
|
||||
Object struct {
|
||||
Message string `json:"message"`
|
||||
Object struct {
|
||||
SHA string `json:"sha"`
|
||||
} `json:"object"`
|
||||
}
|
||||
@@ -333,6 +341,7 @@ func (c *GiteaClient) ListTags(ctx context.Context) ([]RawTag, error) {
|
||||
continue
|
||||
}
|
||||
tag.TargetSHA = annotated.Object.SHA
|
||||
tag.Message = annotated.Message
|
||||
}
|
||||
tags = append(tags, tag)
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ package versionhistory
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"knowledge-governance/internal/governance"
|
||||
@@ -25,6 +26,7 @@ type RawCommit struct {
|
||||
|
||||
// RawTag 是一个 tag 及其可核验属性；Annotated=false 表示轻量 tag（不算正式版本的证据）。
|
||||
type RawTag struct {
|
||||
Message string
|
||||
Name string
|
||||
TagSHA string
|
||||
TargetSHA string
|
||||
@@ -36,13 +38,14 @@ type ChangelogSource func(sha string) (string, error)
|
||||
|
||||
// DeriveInput 是一次换算的全部输入。
|
||||
type DeriveInput struct {
|
||||
RepoKey string
|
||||
CommitURLBase string // 形如 "<origin>/<owner>/<repo>/commit/"，由配置拼好，不接受外部传入
|
||||
TagURLBase string // 形如 "<origin>/<owner>/<repo>/src/tag/"；空则不下发 tag 链接
|
||||
Commits []RawCommit
|
||||
Tags []RawTag
|
||||
Changelog ChangelogSource
|
||||
SnapshotID int64
|
||||
DeliveryArchive string
|
||||
RepoKey string
|
||||
CommitURLBase string // 形如 "<origin>/<owner>/<repo>/commit/"，由配置拼好，不接受外部传入
|
||||
TagURLBase string // 形如 "<origin>/<owner>/<repo>/src/tag/"；空则不下发 tag 链接
|
||||
Commits []RawCommit
|
||||
Tags []RawTag
|
||||
Changelog ChangelogSource
|
||||
SnapshotID int64
|
||||
}
|
||||
|
||||
// DeriveResult 是换算结果。
|
||||
@@ -192,6 +195,22 @@ func DeriveVersions(in DeriveInput) DeriveResult {
|
||||
}
|
||||
if kind == governance.RecordKindRelease {
|
||||
record.ReleaseNotes = governance.ExtractReleaseNotes(changelog, label)
|
||||
} else if kind == governance.RecordKindDevelopment {
|
||||
record.ReleaseNotes = governance.ExtractDevelopmentNotes(changelog, label)
|
||||
if record.ReleaseNotes == "" {
|
||||
record.ReleaseNotes = governance.ExtractDevelopmentNotesForCommit(in.DeliveryArchive, label, commit.SHA)
|
||||
}
|
||||
if record.ReleaseNotes == "" {
|
||||
for _, tag := range tagsByTarget[commit.SHA] {
|
||||
if tag.Annotated && tag.Name == "snapshot/v"+label {
|
||||
_, body, found := strings.Cut(tag.Message, "\n\n")
|
||||
if found {
|
||||
record.ReleaseNotes = strings.TrimSpace(body)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
records = append(records, record)
|
||||
}
|
||||
|
||||
@@ -130,22 +130,24 @@ type SyncOutcome struct {
|
||||
|
||||
// SyncerConfig 装配同步器。
|
||||
type SyncerConfig struct {
|
||||
Source Source
|
||||
Store SyncStore
|
||||
RepoKey string
|
||||
Changelog string // 提交内的变更记录文件路径，默认 CHANGELOG.md
|
||||
Now func() time.Time
|
||||
MaxPages int
|
||||
Source Source
|
||||
Store SyncStore
|
||||
RepoKey string
|
||||
Changelog string // 提交内的变更记录文件路径，默认 CHANGELOG.md
|
||||
Now func() time.Time
|
||||
MaxPages int
|
||||
TargetSHAs []string
|
||||
}
|
||||
|
||||
// Syncer 抓取远端 main 可达的全部提交并换算版本记录。
|
||||
type Syncer struct {
|
||||
source Source
|
||||
store SyncStore
|
||||
repoKey string
|
||||
changelog string
|
||||
now func() time.Time
|
||||
maxPages int
|
||||
source Source
|
||||
store SyncStore
|
||||
repoKey string
|
||||
changelog string
|
||||
now func() time.Time
|
||||
maxPages int
|
||||
targetSHAs []string
|
||||
}
|
||||
|
||||
const syncerDefaultMaxPages = 200
|
||||
@@ -170,7 +172,12 @@ func NewSyncer(config SyncerConfig) (*Syncer, error) {
|
||||
if maxPages <= 0 {
|
||||
maxPages = syncerDefaultMaxPages
|
||||
}
|
||||
return &Syncer{source: config.Source, store: config.Store, repoKey: config.RepoKey,
|
||||
for _, sha := range config.TargetSHAs {
|
||||
if !fullSHA.MatchString(sha) {
|
||||
return nil, errors.New("版本记录：运行身份必须是完整 SHA")
|
||||
}
|
||||
}
|
||||
return &Syncer{targetSHAs: append([]string{}, config.TargetSHAs...), source: config.Source, store: config.Store, repoKey: config.RepoKey,
|
||||
changelog: changelog, now: now, maxPages: maxPages}, nil
|
||||
}
|
||||
|
||||
@@ -204,9 +211,22 @@ func (s *Syncer) Sync(ctx context.Context, jobID, triggeredBy string) (governanc
|
||||
return fail(err)
|
||||
}
|
||||
|
||||
commits, err = s.includeDeliveredCommits(ctx, commits, tags)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
total = len(commits)
|
||||
deliveryArchive := ""
|
||||
if len(s.targetSHAs) > 0 {
|
||||
deliveryArchive, err = s.source.ReadFile(ctx, s.targetSHAs[0], s.changelog)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
}
|
||||
derived := DeriveVersions(DeriveInput{RepoKey: s.repoKey, CommitURLBase: s.source.CommitURLBase(),
|
||||
TagURLBase: s.source.TagURLBase(),
|
||||
Commits: commits, Tags: tags, SnapshotID: runID,
|
||||
TagURLBase: s.source.TagURLBase(),
|
||||
DeliveryArchive: deliveryArchive,
|
||||
Commits: commits, Tags: tags, SnapshotID: runID,
|
||||
Changelog: func(sha string) (string, error) {
|
||||
return s.source.ReadFile(ctx, sha, s.changelog)
|
||||
}})
|
||||
@@ -220,6 +240,9 @@ func (s *Syncer) Sync(ctx context.Context, jobID, triggeredBy string) (governanc
|
||||
} else {
|
||||
sourceHead = headSHA(commits) // 读分支失败不致命：退回提交列表最新一条
|
||||
}
|
||||
if len(s.targetSHAs) > 0 {
|
||||
sourceHead = s.targetSHAs[0]
|
||||
}
|
||||
outcome := SyncOutcome{SourceHeadSHA: sourceHead, FetchedCount: len(commits),
|
||||
ExpectedCount: total, Complete: derived.GraphComplete && (total == 0 || len(commits) == total)}
|
||||
if err := s.store.FinishVersionHistorySync(ctx, runID, outcome, s.now().UTC()); err != nil {
|
||||
@@ -232,6 +255,9 @@ func (s *Syncer) Sync(ctx context.Context, jobID, triggeredBy string) (governanc
|
||||
|
||||
// fetchAllCommits 逐页抓取并按 SHA 去重；短页即终止（也防住服务端忽略 page 参数）。
|
||||
func (s *Syncer) fetchAllCommits(ctx context.Context) ([]RawCommit, int, error) {
|
||||
return s.fetchCommitsAt(ctx, "")
|
||||
}
|
||||
func (s *Syncer) fetchCommitsAt(ctx context.Context, ref string) ([]RawCommit, int, error) {
|
||||
pageSize := s.source.PageSize()
|
||||
if pageSize <= 0 {
|
||||
pageSize = giteaDefaultPageSize
|
||||
@@ -240,7 +266,18 @@ func (s *Syncer) fetchAllCommits(ctx context.Context) ([]RawCommit, int, error)
|
||||
commits := make([]RawCommit, 0, pageSize)
|
||||
total := 0
|
||||
for page := 1; page <= s.maxPages; page++ {
|
||||
pageCommits, pageTotal, err := s.source.ListCommits(ctx, page, pageSize)
|
||||
var pageCommits []RawCommit
|
||||
var pageTotal int
|
||||
var err error
|
||||
if ref == "" {
|
||||
pageCommits, pageTotal, err = s.source.ListCommits(ctx, page, pageSize)
|
||||
} else if source, ok := s.source.(interface {
|
||||
ListCommitsAtRef(context.Context, string, int, int) ([]RawCommit, int, error)
|
||||
}); ok {
|
||||
pageCommits, pageTotal, err = source.ListCommitsAtRef(ctx, ref, page, pageSize)
|
||||
} else {
|
||||
return nil, 0, errors.New("版本来源不支持开发快照历史")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
@@ -356,6 +356,9 @@ def cmd_internal_web_test() -> int:
|
||||
def cmd_internal_contracts(go: str) -> int:
|
||||
"""版本契约：结构 + 注入身份必须**真 PASS**（SKIP 判红），
|
||||
并且**开发构建不许自称正式版**（负例：标成 release ⇒ 必须红）。"""
|
||||
snapshot_rc = subprocess.run(py("scripts/development_snapshot.py","--self-test"),cwd=ROOT).returncode
|
||||
if snapshot_rc:
|
||||
return 1
|
||||
rc = subprocess.run(py("scripts/check_changelog.py"), cwd=ROOT).returncode
|
||||
if rc:
|
||||
return 1
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# 重建并上线 governance api + web（本机 docker build 需要宿主代理 + --network host）
|
||||
set -u
|
||||
cd /home/zheromiku/knowledge-governance || exit 1
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit 1
|
||||
[ -z "$(git status --porcelain --untracked-files=normal)" ] || { printf "%s\n" "REFUSED_DIRTY_SOURCE"; exit 1; }
|
||||
TS=$(date +%Y%m%d-%H%M%S)
|
||||
PROXY_ARGS=(--build-arg HTTP_PROXY=http://127.0.0.1:7890 --build-arg HTTPS_PROXY=http://127.0.0.1:7890 --build-arg NO_PROXY=localhost,127.0.0.1 --build-arg GOPROXY=https://goproxy.cn,direct)
|
||||
|
||||
@@ -123,4 +124,12 @@ if docker exec knowledge-governance-p1-api-1 sh -lc "strings /app/api | grep -q
|
||||
else
|
||||
echo "INJECTION_MISSING（产物里没有 $INJECTED —— 页面会显示当前运行未知）"
|
||||
fi
|
||||
echo "=== 10. 版本记录同步读回（当前 SHA + 注释快照 + 独立变更说明）"
|
||||
# Startup queues the read-only history job. A healthy API alone is not proof.
|
||||
VERIFIED=0
|
||||
for ((attempt=0;attempt<90;attempt++)); do
|
||||
if docker exec knowledge-governance-p1-api-1 /app/api --version-record-status; then VERIFIED=1; break; fi
|
||||
sleep 2
|
||||
done
|
||||
[ "$VERIFIED" = "1" ] || { printf '%s\n' 'VERSION_RECORD_NOT_VERIFIED'; exit 1; }
|
||||
echo "DONE"
|
||||
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Explicit development delivery tags; never creates formal tags or releases."""
|
||||
import argparse,json,re,subprocess,tempfile
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parent.parent
|
||||
SHA=re.compile(r'^[a-f0-9]{40}$')
|
||||
def git(root,*args):
|
||||
p=subprocess.run(['git',*args],cwd=root,text=True,capture_output=True)
|
||||
if p.returncode:raise RuntimeError('git operation failed: '+args[0])
|
||||
return p.stdout.strip()
|
||||
def notes(text,version,sha=None):
|
||||
for heading in re.finditer(r'^## 开发快照 snapshot/v'+re.escape(version)+r' - \d{4}-\d{2}-\d{2}\s*$',text,re.M):
|
||||
body=re.split(r'^##\s',text[heading.end():],maxsplit=1,flags=re.M)[0].strip()
|
||||
if sha is None or ('目标提交：'+sha) in body.splitlines():return body
|
||||
return ''
|
||||
def plan(root,sha,backfill=False):
|
||||
if not sha or not SHA.fullmatch(sha):raise ValueError('exact 40-character commit SHA required')
|
||||
if git(root,'cat-file','-t',sha)!='commit':raise ValueError('snapshot target must be a commit')
|
||||
text=git(root,'show',sha+':CHANGELOG.md');m=re.search(r'^##\s*\[(\d+\.\d+\.\d+)\]',text,re.M)
|
||||
if not m:raise ValueError('formal baseline missing')
|
||||
baseline=m.group(1);base='v'+baseline
|
||||
if git(root,'cat-file','-t','refs/tags/'+base)!='tag':raise ValueError('annotated formal baseline missing')
|
||||
git(root,'merge-base','--is-ancestor',base,sha)
|
||||
n=int(git(root,'rev-list','--count',base+'..'+sha));version='.'.join(baseline.split('.')[:2]+[str(n)])
|
||||
if n<=0:raise ValueError('formal baseline cannot be tagged as a development delivery')
|
||||
body=notes(text,version)
|
||||
if not body and backfill:
|
||||
body=notes(git(root,'show','HEAD:CHANGELOG.md'),version,sha)
|
||||
if '目标提交：'+sha not in body:raise ValueError('backfilled notes must bind the exact historic SHA')
|
||||
if not body:raise ValueError('this development version has no independent CHANGELOG entry')
|
||||
if not re.search(r'^### (Added|Changed|Fixed|Security)\s*$',body,re.M):raise ValueError('snapshot notes have no change category')
|
||||
return {'commit_sha':sha,'display_version':version,'tag':'snapshot/v'+version,'message':'Development snapshot '+version+'\n\n'+body,'backfilled':bool(backfill)}
|
||||
def publish(root,p):
|
||||
# Only explicitly named development refs are written. Existing refs, including
|
||||
# legacy misnumbered snapshots, are never moved or overwritten.
|
||||
tag=p['tag'];sha=p['commit_sha'];ref='refs/tags/'+tag
|
||||
remote=git(root,'ls-remote','origin');pairs=[line.split() for line in remote.splitlines() if line.strip()];refs={name:obj for obj,name in pairs}
|
||||
if ref in refs:
|
||||
if refs.get(ref+'^{}')!=sha:raise ValueError('remote snapshot already exists with different or non-annotated identity')
|
||||
return {'tag':tag,'commit_sha':sha,'state':'already_published'}
|
||||
reachable=False
|
||||
for obj,name in pairs:
|
||||
if name.startswith('refs/heads/') and subprocess.run(['git','merge-base','--is-ancestor',sha,obj],cwd=root,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL).returncode==0:reachable=True;break
|
||||
if not reachable:raise ValueError('snapshot target has not been pushed to a remote branch')
|
||||
exists=subprocess.run(['git','show-ref','--verify','--quiet',ref],cwd=root).returncode==0
|
||||
if exists:
|
||||
if git(root,'cat-file','-t',ref)!='tag' or git(root,'rev-parse',ref+'^{}')!=sha:raise ValueError('local snapshot conflicts with requested identity')
|
||||
if git(root,'for-each-ref','--format=%(contents)',ref).strip()!=p['message'].strip():raise ValueError('existing snapshot annotation differs; will not overwrite')
|
||||
else:
|
||||
with tempfile.NamedTemporaryFile(mode='w',prefix='kg-snapshot-note-',encoding='utf-8') as f:
|
||||
f.write(p['message']+'\n');f.flush();git(root,'tag','-a',tag,sha,'-F',f.name)
|
||||
git(root,'push','origin',ref+':'+ref)
|
||||
readback={line.split()[1]:line.split()[0] for line in git(root,'ls-remote','--tags','origin',ref,ref+'^{}').splitlines()}
|
||||
if readback.get(ref+'^{}')!=sha or readback.get(ref)!=git(root,'rev-parse',ref):raise RuntimeError('remote snapshot read-back failed')
|
||||
return {'tag':tag,'commit_sha':sha,'state':'published_and_read_back'}
|
||||
def self_test():
|
||||
with tempfile.TemporaryDirectory(prefix='kg-snapshot-test-') as d:
|
||||
r=Path(d);git(r,'init','-b','main');git(r,'config','user.name','Fixture');git(r,'config','user.email','fixture@example.invalid')
|
||||
base='# Changelog\n\n## [0.1.0] - 2026-10-13\n\n### Added\n\n- fixture baseline\n'
|
||||
(r/'CHANGELOG.md').write_text(base);git(r,'add','CHANGELOG.md');git(r,'commit','-m','feat: fixture baseline');git(r,'tag','-a','v0.1.0','-m','fixture')
|
||||
(r/'CHANGELOG.md').write_text('## 开发快照 snapshot/v0.1.1 - 2026-10-18\n\n### Fixed\n\n- specific fixture fix\n\n'+base);git(r,'add','CHANGELOG.md');git(r,'commit','-m','fix: fixture change')
|
||||
sha=git(r,'rev-parse','HEAD');before=git(r,'tag');p=plan(r,sha)
|
||||
assert p['tag']=='snapshot/v0.1.1' and 'specific fixture fix' in p['message']
|
||||
duplicate='## 开发快照 snapshot/v0.1.1 - 2026-10-18\n\n目标提交：'+('a'*40)+'\n\n### Fixed\n\n- wrong branch\n\n## 开发快照 snapshot/v0.1.1 - 2026-10-18\n\n目标提交：'+sha+'\n\n### Fixed\n\n- exact branch'
|
||||
assert 'exact branch' in notes(duplicate,'0.1.1',sha) and 'wrong branch' not in notes(duplicate,'0.1.1',sha)
|
||||
assert before==git(r,'tag'),'planning wrote a tag'
|
||||
try:plan(r,'a'*7)
|
||||
except ValueError:pass
|
||||
else:raise AssertionError('short SHA accepted')
|
||||
bare=r/'remote.git';git(r,'init','--bare',str(bare));git(r,'remote','add','origin',str(bare));git(r,'push','origin','main')
|
||||
out=publish(r,p);assert out['state']=='published_and_read_back'
|
||||
assert git(r,'rev-parse',p['tag']+'^{}')==sha
|
||||
assert publish(r,p)['state']=='already_published'
|
||||
wrong=dict(p,commit_sha='a'*40)
|
||||
try:publish(r,wrong)
|
||||
except ValueError:pass
|
||||
else:raise AssertionError('conflicting remote snapshot overwritten')
|
||||
assert git(r,'tag','--list','v*').splitlines()==['v0.1.0'],'formal tags changed'
|
||||
print('development snapshot fixture passed; real repository tags untouched')
|
||||
def main():
|
||||
a=argparse.ArgumentParser();a.add_argument('--sha');a.add_argument('--backfill',action='store_true');a.add_argument('--publish',action='store_true');a.add_argument('--self-test',action='store_true');o=a.parse_args()
|
||||
if o.self_test:return self_test()
|
||||
p=plan(ROOT,o.sha,o.backfill)
|
||||
print(json.dumps(publish(ROOT,p) if o.publish else p,ensure_ascii=False))
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Read-only bridge to the existing weekly-report contacts fetchers.
|
||||
|
||||
The bridge does not execute the weekly report, export spreadsheets, write its
|
||||
configuration, or infer company/department names. All new data stays in a private
|
||||
caller-selected JSON file for the governance importer.
|
||||
"""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import importlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import types
|
||||
from urllib.request import HTTPRedirectHandler, ProxyHandler, build_opener
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
|
||||
class DirectorySyncError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _validate_directory(departments, people):
|
||||
if not departments or not people or len(departments) > 20000 or len(people) > 100000:
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
for identity, department in departments.items():
|
||||
if identity <= 0 or not department.get('name'):
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
seen = set()
|
||||
current = identity
|
||||
while current:
|
||||
if current in seen or current not in departments or len(seen) > 100:
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
seen.add(current)
|
||||
current = int(departments[current].get('parent_id') or 0)
|
||||
for identity, person in people.items():
|
||||
name = person.get('name')
|
||||
if name is not None and (not isinstance(name, str) or len(name.encode('utf-8')) > 1000 or '\x00' in name):
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
if not identity or str(person.get('user_id') or '') != str(identity):
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
if not person.get('department_ids') or any(int(d) not in departments for d in person['department_ids']):
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
|
||||
|
||||
def collect_directory(client, fetch_departments, fetch_users, get_detail, logger):
|
||||
if client.config.dry_run:
|
||||
raise DirectorySyncError('directory_dry_run_rejected')
|
||||
departments, denied, failed = fetch_departments(client, logger, None, None)
|
||||
if denied or failed or not departments:
|
||||
raise DirectorySyncError('directory_department_fetch_incomplete')
|
||||
departments = dict(departments)
|
||||
# Legacy full traversal returns children but omits the starting root. Read
|
||||
# actual missing ancestors; never manufacture a root or discard parent IDs.
|
||||
for _ in range(100):
|
||||
missing = sorted({int(d.get('parent_id') or 0) for d in departments.values()} - set(departments) - {0})
|
||||
if not missing:
|
||||
break
|
||||
for identity in missing:
|
||||
detail, denied, failed = get_detail(client, logger, identity)
|
||||
if denied or failed or not detail or int(detail.get('id') or 0) != identity or not detail.get('name'):
|
||||
raise DirectorySyncError('directory_ancestor_unavailable')
|
||||
departments[identity] = detail
|
||||
else:
|
||||
raise DirectorySyncError('directory_integrity_failed')
|
||||
people, denied, failed = fetch_users(client, logger, sorted(departments), enable_user_detail=False)
|
||||
if denied or failed:
|
||||
raise DirectorySyncError('directory_people_fetch_incomplete')
|
||||
_validate_directory(departments, people)
|
||||
return {
|
||||
'source': 'weekly_report_live',
|
||||
'synced_at': datetime.now(timezone.utc).isoformat(),
|
||||
'complete': True,
|
||||
'directory': {
|
||||
'departments': [
|
||||
{'id': str(identity), 'parent_id': str(int(d.get('parent_id') or 0)) if d.get('parent_id') else '', 'name': str(d['name'])}
|
||||
for identity, d in sorted(departments.items())
|
||||
],
|
||||
# List membership provides stable staff IDs; do not reuse the old
|
||||
# guessed primary department, employment policy or openId fallback.
|
||||
'people': [
|
||||
{'id': str(identity), **({'name': p['name']} if p.get('name') else {}), 'union_id': str((p.get('raw_payload') or {}).get('unionid') or (p.get('raw_payload') or {}).get('unionId') or ''), 'department_ids': [str(d) for d in sorted(p['department_ids'])]}
|
||||
for identity, p in sorted(people.items())
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class _NoRedirect(HTTPRedirectHandler):
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
raise DirectorySyncError('directory_redirect_rejected')
|
||||
|
||||
|
||||
def load_existing_runtime(root):
|
||||
root = Path(root)
|
||||
if not root.is_absolute() or not (root / 'scripts/dingtalk_sync/client.py').is_file():
|
||||
raise DirectorySyncError('directory_configuration_invalid')
|
||||
secret = root / '.env'
|
||||
info = secret.lstat()
|
||||
if not secret.is_file() or secret.is_symlink() or info.st_mode & 0o077:
|
||||
raise DirectorySyncError('directory_configuration_invalid')
|
||||
package_name = '_qaops_weekly_directory'
|
||||
package = types.ModuleType(package_name)
|
||||
package.__path__ = [str(root / 'scripts/dingtalk_sync')]
|
||||
sys.modules[package_name] = package
|
||||
client_module = importlib.import_module(package_name + '.client')
|
||||
dept_module = importlib.import_module(package_name + '.fetch_departments')
|
||||
people_module = importlib.import_module(package_name + '.fetch_users')
|
||||
config = client_module.load_sync_config(str(secret), str(root / 'config/dingtalk_sync.yaml'))
|
||||
if config.api_base_url.rstrip('/') != 'https://oapi.dingtalk.com' or config.dry_run or config.api_mode != 'topapi_v2':
|
||||
raise DirectorySyncError('directory_configuration_invalid')
|
||||
if not (1 <= config.timeout_seconds <= 30 and 0 <= config.max_retries <= 3 and 1 <= config.page_size <= 100):
|
||||
raise DirectorySyncError('directory_configuration_invalid')
|
||||
# Keep the old transport inside this private module namespace. Suppress its
|
||||
# potentially sensitive exception logging and refuse redirects/proxies.
|
||||
logger = logging.getLogger('qa_ops_directory_private')
|
||||
logger.handlers = [logging.NullHandler()]
|
||||
logger.propagate = False
|
||||
opener = build_opener(ProxyHandler({}), _NoRedirect())
|
||||
setattr(client_module, 'urlopen', opener.open)
|
||||
|
||||
class BoundedClient(client_module.DingtalkClient):
|
||||
def __init__(self):
|
||||
super().__init__(config, logger=logger)
|
||||
self.requests = 0
|
||||
self.pages = set()
|
||||
|
||||
def request_json(self, method, path, params=None, json_body=None, retry_on_business_error=True):
|
||||
self.requests += 1
|
||||
if self.requests > 10000:
|
||||
raise DirectorySyncError('directory_request_limit')
|
||||
if path.endswith('/listsub') or path.endswith('/user/list'):
|
||||
key = (path, json.dumps(json_body, sort_keys=True))
|
||||
if key in self.pages:
|
||||
raise DirectorySyncError('directory_pagination_repeated')
|
||||
self.pages.add(key)
|
||||
return super().request_json(method, path, params, json_body, retry_on_business_error)
|
||||
|
||||
return BoundedClient(), dept_module.fetch_all_departments, people_module.fetch_users, dept_module._get_department_detail, logger
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Fetch a complete read-only directory into private governance JSON; no weekly-report writes')
|
||||
parser.add_argument('--runtime-root', required=True)
|
||||
parser.add_argument('--output', required=True)
|
||||
parser.add_argument('--fetch-live-directory', action='store_true', required=True)
|
||||
args = parser.parse_args()
|
||||
output = Path(args.output)
|
||||
if not output.is_absolute() or output.exists():
|
||||
raise DirectorySyncError('directory_output_must_be_new_absolute_path')
|
||||
client, departments, people, detail, logger = load_existing_runtime(args.runtime_root)
|
||||
print(json.dumps({'stage': 'fetching_live_directory', 'weekly_report_outputs_modified': False}), flush=True)
|
||||
result = collect_directory(client, departments, people, detail, logger)
|
||||
raw = json.dumps(result, ensure_ascii=False).encode('utf-8')
|
||||
if len(raw) > 16 << 20:
|
||||
raise DirectorySyncError('directory_output_limit')
|
||||
fd = os.open(output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
try:
|
||||
with os.fdopen(fd, 'wb') as stream:
|
||||
stream.write(raw)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
except BaseException:
|
||||
output.unlink(missing_ok=True)
|
||||
raise
|
||||
print(json.dumps({'stage': 'complete', 'departments': len(result['directory']['departments']), 'people': len(result['directory']['people']), 'synced_at': result['synced_at'], 'requests': client.requests, 'complete': True, 'employee_identities_logged': False, 'weekly_report_outputs_modified': False}), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
except Exception as failure:
|
||||
code = str(failure) if isinstance(failure, DirectorySyncError) else 'directory_sync_failed'
|
||||
print(json.dumps({'stage': 'failed', 'code': code}), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -9,6 +9,59 @@
|
||||
"⚠️ 文件缺失或解析不了 ⇒ 门禁红：范围声明不出来时，不许默认放行。"
|
||||
],
|
||||
"optional": [
|
||||
{"package":"knowledge-governance/internal/qaopsquality","test":"TestQualityStoreImmutableHistoryReplayConcurrencyAndStaleness","reason":"仅在显式一次性 loopback kg_qaops_quality_test_ 库执行；默认 CI 不连接共享/生产库，隔离 PG 另行实跑"},
|
||||
{"package": "knowledge-governance/internal/dobotaiops", "test": "TestQARecordInternalIdentityAndWholeFilterKeys", "reason": "必须显式一次性隔离PG；验证真实只读SQL身份关联/计数及隐私，不连接生产；默认无库时明确跳过，另行本地补验"},
|
||||
{"package": "knowledge-governance/internal/dobotaiops", "test": "TestQAUsageGroupsCountEventsOnceOverTheFullFilter", "reason": "必须显式一次性隔离PG；验证真实只读SQL身份关联/计数及隐私，不连接生产；默认无库时明确跳过，另行本地补验"},
|
||||
{
|
||||
"package": "knowledge-governance/internal/qaopsorg",
|
||||
"test": "TestDirectorySyncQueueDurableReplayAndLeaseFencing",
|
||||
"reason": "仅显式一次性隔离PG库验证持久化入队、并发重放、进程接管与租约栅栏；不调用真实钉钉源、不连接共享或生产库；默认CI无库则明确跳过"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/qaopsorg",
|
||||
"test": "TestOrganizationStoreRoundTripAndOptimisticConflict",
|
||||
"reason": "仅在显式KG_QAOPS_ORG_TEST_ISOLATED=1且127.0.0.1上的一次性kg_qaops_org_test_库运行；写入治理侧快照与配置，禁止使用共享或生产库；本批已在隔离PG补验"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestSSHDialerTransportsRealTraffic",
|
||||
"reason": "需要已批准 SSH 路径和只读版本接口；CI 不带生产 SSH 凭据，缺环境时显式跳过"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestSSHTunnelCloseReapsChild",
|
||||
"reason": "需要已批准 SSH 目标来验证子进程回收；无路径的 CI 不应尝试建立生产连接"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestReadOnlyRepositoryAgainstFixtureDatabase",
|
||||
"reason": "需显式 KG_DOBOTAI_TEST_ISOLATED=1 且 TEST_DATABASE_URL 通过隔离库护栏，CI 不建生产测试表；本批已在一次性测试库补验"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestBuildRefusesUnsafeAccountsAndTargets",
|
||||
"reason": "只在显式一次性测试库创建危险权限夹具，不在生产库验证写入；本批已在隔离库补验"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestSourceRechecksEffectivePrivilegesBeforeEveryQuery",
|
||||
"reason": "在一次性测试库模拟额外表写权限，测试前须通过隔离护栏；本批已实际补验"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestQARecordSearchPreservesSourceAndDoesNotMultiplyFeedback",
|
||||
"reason": "完整查询/反馈/游标/空列验证需要显式一次性测试库；本批已在隔离库补验"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestProductionReadOnlyCombinedVerification",
|
||||
"reason": "需独立批准的生产只读账号文件与 SSH 路径并显式 KG_DOBOTAI_PROD_VERIFY=1；CI 永不自动读取生产库"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/dobotaiops",
|
||||
"test": "TestProductionQAQueryReadOnlyVerification",
|
||||
"reason": "真实源查询端到端验证仅在显式 KG_DOBOTAI_PROD_VERIFY=1、批准的凭据文件与 SSH 路径下执行；CI 不访问生产，测试不写源库"
|
||||
},
|
||||
{
|
||||
"package": "knowledge-governance/internal/platform/postgres",
|
||||
"reason": "整个包的集成用例都要 TEST_DATABASE_URL 指向可销毁测试库；本仓不连生产库，CI 无库 ⇒ 跳过（补验路径：起一次性 PG 后设 TEST_DATABASE_URL）"
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import importlib.util
|
||||
import logging
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
spec=importlib.util.spec_from_file_location('qa_ops_directory_sync',Path(__file__).with_name('qa_ops_directory_sync.py'))
|
||||
assert spec is not None and spec.loader is not None
|
||||
sync=importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(sync)
|
||||
|
||||
class FakeClient:
|
||||
class Config:
|
||||
dry_run=False
|
||||
def __init__(self):
|
||||
self.config=self.Config()
|
||||
|
||||
def full_departments(client,logger,include,exclude):
|
||||
return {10:{'id':10,'parent_id':1,'name':'示例业务'}},set(),set()
|
||||
def full_users(client,logger,ids,enable_user_detail):
|
||||
if enable_user_detail:
|
||||
raise AssertionError('Must not inherit silent per-user detail failures or guessed primary department')
|
||||
return {'fixture-user':{'user_id':'fixture-user','department_ids':{10},'raw_payload':{'unionid':'fixture-union'}}},set(),set()
|
||||
def root_detail(client,logger,department_id):
|
||||
return {'id':department_id,'parent_id':0,'name':'真实源端返回的示例根部门'},False,False
|
||||
|
||||
class DirectoryBridgeTests(unittest.TestCase):
|
||||
def test_real_display_name_is_retained_without_guessing_or_requiring_it(self):
|
||||
def users(*args,**kwargs):
|
||||
return {'u':{'user_id':'u','name':'<示例员工>','department_ids':{10},'raw_payload':{}}},set(),set()
|
||||
result=sync.collect_directory(FakeClient(),full_departments,users,root_detail,logging.getLogger('fixture'))
|
||||
self.assertEqual(result['directory']['people'][0].get('name'),'<示例员工>')
|
||||
result=sync.collect_directory(FakeClient(),full_departments,full_users,root_detail,logging.getLogger('fixture'))
|
||||
self.assertNotIn('name',result['directory']['people'][0])
|
||||
def test_fetches_the_real_missing_root_and_preserves_stable_identities(self):
|
||||
result=sync.collect_directory(FakeClient(),full_departments,full_users,root_detail,logging.getLogger('fixture'))
|
||||
self.assertTrue(result['complete'])
|
||||
self.assertEqual(result['source'],'weekly_report_live')
|
||||
self.assertEqual(result['directory']['departments'],[
|
||||
{'id':'1','parent_id':'','name':'真实源端返回的示例根部门'},
|
||||
{'id':'10','parent_id':'1','name':'示例业务'}])
|
||||
self.assertEqual(result['directory']['people'],[{'id':'fixture-user','union_id':'fixture-union','department_ids':['10']}])
|
||||
def test_inaccessible_root_is_not_replaced_with_a_fabricated_company(self):
|
||||
def denied(*args):return None,True,False
|
||||
with self.assertRaisesRegex(sync.DirectorySyncError,'directory_ancestor_unavailable'):
|
||||
sync.collect_directory(FakeClient(),full_departments,full_users,denied,logging.getLogger('fixture'))
|
||||
def test_partial_fetch_and_dry_run_do_not_become_complete_live_snapshots(self):
|
||||
def partial(*args):return {},set(),{10}
|
||||
with self.assertRaises(sync.DirectorySyncError):
|
||||
sync.collect_directory(FakeClient(),partial,full_users,root_detail,logging.getLogger('fixture'))
|
||||
client=FakeClient();client.config.dry_run=True
|
||||
with self.assertRaises(sync.DirectorySyncError):
|
||||
sync.collect_directory(client,full_departments,full_users,root_detail,logging.getLogger('fixture'))
|
||||
def test_union_id_does_not_fall_back_to_open_id(self):
|
||||
def users(*args,**kwargs):return {'u':{'user_id':'u','department_ids':{10},'union_id':'not-a-union','raw_payload':{'openId':'not-a-union'}}},set(),set()
|
||||
result=sync.collect_directory(FakeClient(),full_departments,users,root_detail,logging.getLogger('fixture'))
|
||||
self.assertEqual(result['directory']['people'][0]['union_id'],'')
|
||||
def test_wrong_root_identity_and_cycles_are_rejected(self):
|
||||
def wrong(*args):return {'id':999,'parent_id':0,'name':'wrong'},False,False
|
||||
with self.assertRaises(sync.DirectorySyncError):
|
||||
sync.collect_directory(FakeClient(),full_departments,full_users,wrong,logging.getLogger('fixture'))
|
||||
def cyclic(*args):return {1:{'id':1,'parent_id':10,'name':'a'},10:{'id':10,'parent_id':1,'name':'b'}},set(),set()
|
||||
with self.assertRaises(sync.DirectorySyncError):
|
||||
sync.collect_directory(FakeClient(),cyclic,full_users,root_detail,logging.getLogger('fixture'))
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user