Commit Graph

26 Commits

Author SHA1 Message Date
姚章浩 f8cef26f4a feat(web): 登录/改密/成员表单加密码可见切换 2026-09-30 10:54:07 +08:00
姚章浩 83d0347ca7 fix(web): 请求时解析 api 上游以免重建后 502 2026-09-30 10:54:07 +08:00
姚章浩 c84f848849 perf(catalog): resolve filename filters once per query, not per row
The 000036 evaluator probes the rules table and runs two regexes for every
catalog file row; the first-level page of the 8k-file education source took
2.5-5s per query and the browser hung on 'reading first-level directory'.

- source_catalog_nodes gains a STORED generated ext column (000038)
- catalog_filter_context() resolves the connection's source-level allow/
  exclude sets once; hot queries match node.ext against arrays (000039)
- filename_is_filtered stays the exact fallback, used only when the
  connection has directory-scoped rules
- ListCatalogSources, ListCatalogNodes (current/subtree/total/items) and
  RouteCatalogFiles fan-out adopt the fast path

Education root page: 4770ms -> 12ms; all-source counts: 54ms.
2026-09-30 10:54:07 +08:00
姚章浩 e6ef5a4693 feat(filters): allowlist mode for filename filters
- filter rules gain mode ('exclude'|'allow'); nearest allow rule governs a
  directory subtree, extensions outside it are filtered before fetch
- filename_extension() now requires a well-formed suffix, so dated names like
  'CRA_V3.1手册_2024-07-31_en' no longer fake an extension; extensionless
  files are never filtered
- API accepts {mode, extensions}; UI editor becomes grouped format chips with
  keep/exclude modes (keep-list default: docs/text/scripts/images)
- migration 000036 + integration tests for allow, nesting and restore
2026-09-30 10:54:07 +08:00
姚章浩 f7c65571e4 feat(settings): 文件名过滤编辑器预置默认排除集 2026-09-30 10:54:07 +08:00
姚章浩 257be88ecf fix(fetch): UTF-8 sniff boundary must not reject valid documents
validate_magic read exactly 4096 bytes and strict-decoded them; a multibyte
character straddling the boundary made perfectly valid .md/.txt files fail
with invalid_format. Only tolerate 'unexpected end of data' when the file
actually continues past the sniff window; genuinely corrupt bytes still fail.
2026-09-30 10:54:07 +08:00
姚章浩 b3c9331966 feat(ingest): 源与目录文件名过滤及钉钉文档导出
- filename_filter_rules table (000035) with source-root and per-directory rows
- additive inheritance: source default + every ancestor directory contributes
- filename_is_filtered() SQL predicate applied to catalog browsing, subtree
  counts, change-set diffing, routing fan-out and artifact fetch manifests
- fetch_state 'filtered' (CHECK extended); reconcile cancels/restores fetch
  jobs, retires planned publications and blocks items when rules change
- GET/PUT/DELETE /api/v1/filename-filters (Idempotency-Key, normalized
  extensions) + Settings and Catalog filter editors
- failJobSQL no longer swallows the claimed row when the failure projection
  upserts (fixes spurious 'lease lost' retries)
- DingTalk live fetch: adoc/axls now use the export-job pipeline (HTTP 403
  from file/download fixed), header-signature downloads supported
2026-09-30 10:54:07 +08:00
姚章浩 2005e4e7da fix(settings): resume DingTalk space registration after login 2026-09-30 10:54:07 +08:00
姚章浩 f82283abd4 fix(settings): make DingTalk QR sessions retryable 2026-09-30 10:54:07 +08:00
姚章浩 64d0883e30 fix(settings): complete DingTalk scan login flow 2026-09-30 10:54:07 +08:00
姚章浩 e60b10256a fix(settings): add DingTalk spaces and render the real login QR
Address two gaps in the first DingTalk-session UI:
- add an explicit admin-only '+ 新增钉钉知识库' action on the data-source
  panel; it accepts an alidocs space URL, resolves the live name/root through
  the authenticated sidecar, writes the inventory root atomically, upserts a
  deterministic source connection, and enqueues its first catalog refresh
- read DingTalk's real QR canvas pixels rather than taking a DOM screenshot;
  select by dark-pixel content, composite onto a white quiet zone, and render
  at 320x320 so the dialog contains a large complete code, not a page thumbnail

The live refresh path resolves registered roots from the connection locator,
while existing name-based DingTalk sources remain compatible.
2026-09-30 10:54:07 +08:00
姚章浩 173b44abca feat(settings): refresh DingTalk scan sessions from the data-source page
DingTalk source scanning depends on a shared browser session whose capture
file had to be replaced by hand whenever it expired (every scan 401s).

- services/dingtalk-login: admin-triggered Playwright sidecar that opens the
  space page in a persistent headless Chromium, publishes the QR through
  session polling, verifies the first authenticated XHR against the box API,
  then writes the capture in place (same inode keeps the worker's ro bind
  mount current); tokens never leave the sidecar
- api: POST/GET/DELETE /api/v1/dingtalk-session authenticated relay; QR is a
  login artifact, so the whole prefix is admin-only in classifyRole; OpenAPI
  contract and proxy/503/hostile-ID tests included
- web: 系统设置 → 数据源配置 钉钉卡片新增扫码刷新会话(仅管理员可见),
  QR 弹窗 + 1.5s 轮询 + 成功/过期/失败状态
- compose: sidecar service owns the ONLY rw mount of secrets/; worker keeps ro
2026-09-30 10:54:07 +08:00
姚章浩 021e91eae4 feat(settings): bind internal/external knowledge bases from the UI
Routing targets move from env-only to a UI-managed binding (migration
000034 routing_kb_bindings, single active row):
- GET/PUT /api/v1/routing-bindings + /options: env defaults act as the
  bootstrap fallback; a UI binding wins and takes effect immediately
  (per-request resolution, no restart), and clearing falls back to env
- PUT validates against the connected WeKnora instance (real knowledge
  base list via the scoped API key) and rejects same-KB bindings;
  saves require the admin role
- route/catalog capabilities now resolve through the binding so both
  annotation paths share one source of truth
- system settings > 服务配置 gains a 知识库绑定 panel: dropdown picker
  (or raw ID when WeKnora is unreachable) showing bound name, origin
  (UI vs env default) and last author
- contract tests: OpenAPI, handler matrix (fallback, validation,
  same-KB rejection, capability resolution), frontend fetch/save layer
2026-09-30 10:54:07 +08:00
姚章浩 4c86c5e659 fix(httpapi): accept planned in the test-publications state filter
validProcessingState predated policy-driven ingestion and rejected
processing_state=planned with a 400, so the 待入库 lane silently fetched
an empty plan list while annotation, fetch and planning had all succeeded.
2026-09-30 10:54:07 +08:00
姚章浩 d24e402ef6 fix(orchestration): test poll/validate schedules carry the retry round
jobs.idempotency_key is globally unique, but test-poll:<pub>:<seq> and
test-validate:<pub>:<seq> were reused verbatim by every manual retry. The
first round's completed rows made ON CONFLICT DO NOTHING silently swallow
the next round's schedule SQL: a retried publication re-uploaded, polled
once, and then froze in pending/uploaded with no successor job and no
error. Keys and job ids are now derived from (publication, retry_count,
sequence) inside the schedule SQL so every retry round owns fresh
orchestration rows.
2026-09-30 10:54:07 +08:00
姚章浩 c2d093bc90 fix(fetch): AList download route authenticates with x-api-token
This AList deployment (sales.dobotoa.cn family) only recognizes x-api-token on
the /p/ download route; an Authorization header gets 302'd to login.dingtalk.com.
Probe x-api-token first, fall back to Authorization, then fail-closed.
2026-09-30 10:54:07 +08:00
姚章浩 fad8b10247 feat(web): annotation result copy matches policy-driven ingestion
Saved strategies now report intents/fetch/planned/blocked counts and point at
解析监控 → 待入库; the blocked note explains the 需下架 lane instead of the
deprecated snapshot-diff confirmation step.
2026-09-30 10:54:07 +08:00
姚章浩 63f3b2c503 feat(governance): policy-driven ingestion from catalog annotations
Annotation in 来源与目录 is now the only import decision surface:
- migration 000033: catalog_import_intents (durable, rescan-proof per-file
  intent with locator snapshot + fetch lifecycle), source_items.policy_state,
  planned publication state, ui_setting_revisions.ingestion_concurrency
- marking fans a folder decision out to per-file intents, materializes
  versioned items, reuses already-verified local bytes (compare, don't pull),
  and plans one publication per routed KB; blocked retires planned rows
  while live publications stay untouched (需下架 shows in the KB diff)
- artifact.fetch worker lane (scripts/artifact_fetch.py): DingTalk export
  tree/live box API, AList /api/fs/get direct download with SSO-redirect
  fail-closed, PP ledger via locator hash; content-level dedupe on sha256
- 开始处理 promotes planned->pending inside the claim-time concurrency budget
  from system settings; claims enforce it in SQL
- monitor overview rebuilt as knowledge-base diff: planned/needs_removal per
  KB + fetch pipeline summary; frontend lanes 待入库/需下架 replace the
  deprecated snapshot change-set confirmation
- spool: images (png/jpg/gif/bmp/webp) accepted, ppt/pptx rejected in favour
  of PDF twins; DingTalk PPT resolves its converted PDF pair
2026-09-30 10:54:07 +08:00
zheromiku f6cb90cd70 feat(web): login gate, session identity UI and member management
- main.tsx 挂 AuthProvider 门控:未登录→登录页;must_change_password→强制改密页
- 登录页/改密页/顶栏身份(头像+姓名+角色徽章+退出)按 A·Data-console token 样式
- 全局 fetch 拦截器:任意业务请求 401 自动回登录页(会话过期/被撤销不留死表单)
- 系统设置新增「成员与角色」分区(仅 admin 可见):名单、角色调整、启停、重置临时密码、新增成员
- 消灭全部手填操作人:审核/封批/确认计划集/范围规则/来源调度/UI 设置/开始处理/批量重试/
  目录差异确认共 9 处输入框删除,请求体不再携带 actor 字段(服务端从会话注入)
- 测试断言方向翻转:显式验证请求体不含 operator_ref/created_by 等字段
- 附带修复:登录失败计数 SQL 参数类型无法推断导致 500(pgx 扩展协议),显式 cast 解决

前端门禁:tsc 0 错、125 用例全绿、build 通过
2026-09-30 10:54:07 +08:00
zheromiku fbe80ad784 feat(auth): accounts, sessions, RBAC and server-stamped actors
上线账号权限管理(后端):
- 迁移 000032:users + user_sessions(token 只存 sha256,服务端可即时撤销)
- bcrypt 密码、8 次失败锁 15 分钟、统一 401 错误防用户名枚举
- 三角色 viewer/engineer/admin;HttpOnly+SameSite=Lax 会话 cookie,滑动续期
- 强制改密锁定:must_change_password 会话只能走 auth 端点;首登 dobotai
- bootstrap:users 表为空时创建 admin(初始密码 dobotai,强制改密)
- 生产双闸授权/回滚/失败重试收敛为 admin 角色;AuthorizedBy 用会话身份
- 全部业务写端点删除手填 actor 字段(operator_ref/reviewer_ref/created_by/
  sealed_by/confirmed_by),服务端从会话注入;auth 关闭的开发 socket 回退 system
- 成员管理 API:列表/创建/改角色/启停/重置密码,最后一名 active admin 保护
- RBAC 矩阵纯函数契约测试 + 中间件行为测试(登录/锁定/角色/审计戳)
- 附带修复:GET /api/v1/users 曾被 GET=viewer 规则误放行(测试抓到)
2026-09-30 10:54:07 +08:00
zheromiku 6592df01b6 chore(spool): T2/T3 演练用合成 fixture
- t2-success-3.md:故意碰撞出 tp-0af1… < tp-204f… 的排序,用于
  「成功项先 verified → 后续项永久失败 → 自动回滚撤销成功项」演练
- t3-bulk-200..259.md:51 篇大批次压测样本(跨 batch_size=50 两批)
所有文档均为合成内容,无业务信息
2026-09-30 10:54:07 +08:00
zheromiku 1f0c8be812 fix(release): rewrite nested bucket in target resource locators
模拟发布暴露的第 8 个契约缺陷:PG 通道改写资源行时只替换了外层
storage://<backend-id> 前缀,内层 oss://<bucket>/<key> 照搬源桶。
WeKnora v0.7.1 官方 GetKnowledgeFile 从路径内层桶名直接 GetObject,
导致目标端所有文档预览以生产凭据读源桶失败(403),前端表现为预览空白。

- PostgresImporterConfig 新增 TargetBucket;nested locator 导入时改写
  内层桶名为目标桶,location_hash 随改写后的路径重算
- 未配置目标桶时保持旧行为透传,不做猜测
- 4 个单测锁定:桶改写+hash重算、非嵌套 locator 不变、无配置透传、非法桶名拒绝
- 端到端复验:重封 canary 批次重新发布 4/4 verified,目标端资源行
  全部指向 dobot-weknora-test,location_hash 与路径逐行比对一致
2026-09-30 10:54:07 +08:00
zheromiku dcdc8d1ddd feat: 知识治理平台(含已验证发布/回滚契约)
全栈知识治理平台首次入库(P1-P10 + 发布链路模拟验证闭环):

- 治理核心:源目录/变更集/人工审核/发布批次/发布执行(fail-closed 闸口)
- 测试通道:WeKnora 测试 KB 上传/轮询/验收;poll 完成时写入
  parser_generation/index_generation,修复 manual/seed 路径发布闸口永久拦截
- 发布适配器 v071(本地 source=测试角色 / 云端 target=生产角色 模拟验证通过):
  * ES 导出/导入 mapping 自适应(.keyword 子字段、可选 tenant_id)
  * OSS locator 支持嵌套 scheme;chunks.seq_id 由目标序列分配
  * bundle 归档同身份重试原子替换;verifier 按 bundle chunk_type 计数
  * 回滚 job payload 支持 prior_execution_id(strict 解码)
- 三通道发布+回滚演练实测:publish 4/4 verified,检索命中,
  rollback 4/4 rolled_back,目标端 PG is_enabled=false、ES is_enabled=false
- 前端 A·Data-console 风格全站统一,120 用例通过
- 门禁:go vet / go test 全包通过,gofmt 干净
2026-09-30 10:54:06 +08:00
zheromiku ed48c4ceda chore(web): 增加 CI 接入自检页作为预览发布样例
build / build (push) Successful in 3s
- public/index.html 作为静态产物目录样例(构建成功即发布可访问预览链接)
- 第 1/2 仓的预览链路同源,第三仓接通知 + 网页回报
2026-09-30 10:21:39 +08:00
姚章浩 91430dfcad chore(ci): 接入 Gitea Actions 本机编译与微信通知
build / build (push) Successful in 2s
- 提交信息规范校验(Conventional Commits 权威闸门,非法提交不编译)
- 提交触发本机 Runner 编译(go / python / node 自动识别)
- 构建成功自动上报 ci-bridge:微信通知 + 静态网页预览链接
2026-09-30 10:19:29 +08:00
zheromiku b968abbb58 Initial commit 2026-09-30 10:18:38 +08:00