knowledge-governance

Weknora/knowledge-governance

Fork 0

Commit Graph

Author	SHA1	Message	Date
姚章浩	d29e04e647	fix(artifact): 白名单环境漏传 dws 变量导致钉钉取件整体失败端到端验收实测（2026-10-10）：标注后取件以 artifact_fetch_rejected: credentials 终态失败。根因：artifactfetch runner 给子进程的是白名单环境（防凭据外泄），但没有透传 dws 取件器必需的 ARTIFACT_DINGTALK_FETCHER/DWS_BIN/DWS_CONFIG_DIR/DWS_HOME/DWS_DISABLE_KEYCHAIN → dws 找不到配置目录与文件型 keychain（$HOME/.local/share/dws-cli 落空）→ 报未登录 → 映射成 credentials 终态（不可重试）→ 且不会回落老路径。脚本单跑正常，因为绕过了白名单。 - handler.go: childEnv() 显式透传 dws 变量（仅非空值）；失败时把 stderr 尾部脱敏后带进错误，否则只能看到裸 code 无法定位（本次排查就卡在这） - 新增 3 个回归测试：环境透传 / 空值不透传 / stderr 上报与敏感内容抑制 - 实测：修复后重新入队 → artifact.fetch completed，产物 sn-753f….docx 4799B（adoc→docx 导出）， content_objects 记为 docx/verified	2026-10-10 09:57:41 +08:00
姚章浩	b3c9331966	feat(ingest): 源与目录文件名过滤及钉钉文档导出 - filename_filter_rules table (000035) with source-root and per-directory rows - additive inheritance: source default + every ancestor directory contributes - filename_is_filtered() SQL predicate applied to catalog browsing, subtree counts, change-set diffing, routing fan-out and artifact fetch manifests - fetch_state 'filtered' (CHECK extended); reconcile cancels/restores fetch jobs, retires planned publications and blocks items when rules change - GET/PUT/DELETE /api/v1/filename-filters (Idempotency-Key, normalized extensions) + Settings and Catalog filter editors - failJobSQL no longer swallows the claimed row when the failure projection upserts (fixes spurious 'lease lost' retries) - DingTalk live fetch: adoc/axls now use the export-job pipeline (HTTP 403 from file/download fixed), header-signature downloads supported	2026-09-30 10:54:07 +08:00
姚章浩	63f3b2c503	feat(governance): policy-driven ingestion from catalog annotations Annotation in 来源与目录 is now the only import decision surface: - migration 000033: catalog_import_intents (durable, rescan-proof per-file intent with locator snapshot + fetch lifecycle), source_items.policy_state, planned publication state, ui_setting_revisions.ingestion_concurrency - marking fans a folder decision out to per-file intents, materializes versioned items, reuses already-verified local bytes (compare, don't pull), and plans one publication per routed KB; blocked retires planned rows while live publications stay untouched (需下架 shows in the KB diff) - artifact.fetch worker lane (scripts/artifact_fetch.py): DingTalk export tree/live box API, AList /api/fs/get direct download with SSO-redirect fail-closed, PP ledger via locator hash; content-level dedupe on sha256 - 开始处理 promotes planned->pending inside the claim-time concurrency budget from system settings; claims enforce it in SQL - monitor overview rebuilt as knowledge-base diff: planned/needs_removal per KB + fetch pipeline summary; frontend lanes 待入库/需下架 replace the deprecated snapshot change-set confirmation - spool: images (png/jpg/gif/bmp/webp) accepted, ppt/pptx rejected in favour of PDF twins; DingTalk PPT resolves its converted PDF pair	2026-09-30 10:54:07 +08:00

Author

SHA1

Message

Date

姚章浩

d29e04e647

fix(artifact): 白名单环境漏传 dws 变量导致钉钉取件整体失败

端到端验收实测（2026-10-10）：标注后取件以 artifact_fetch_rejected: credentials 终态失败。
根因：artifactfetch runner 给子进程的是白名单环境（防凭据外泄），但没有透传 dws 取件器
必需的 ARTIFACT_DINGTALK_FETCHER/DWS_BIN/DWS_CONFIG_DIR/DWS_HOME/DWS_DISABLE_KEYCHAIN
→ dws 找不到配置目录与文件型 keychain（$HOME/.local/share/dws-cli 落空）→ 报未登录 →
映射成 credentials 终态（不可重试）→ 且不会回落老路径。脚本单跑正常，因为绕过了白名单。

- handler.go: childEnv() 显式透传 dws 变量（仅非空值）；失败时把 stderr 尾部脱敏后带进错误，
  否则只能看到裸 code 无法定位（本次排查就卡在这）
- 新增 3 个回归测试：环境透传 / 空值不透传 / stderr 上报与敏感内容抑制
- 实测：修复后重新入队 → artifact.fetch completed，产物 sn-753f….docx 4799B（adoc→docx 导出），
  content_objects 记为 docx/verified

2026-10-10 09:57:41 +08:00

姚章浩

b3c9331966

feat(ingest): 源与目录文件名过滤及钉钉文档导出

- filename_filter_rules table (000035) with source-root and per-directory rows
- additive inheritance: source default + every ancestor directory contributes
- filename_is_filtered() SQL predicate applied to catalog browsing, subtree
  counts, change-set diffing, routing fan-out and artifact fetch manifests
- fetch_state 'filtered' (CHECK extended); reconcile cancels/restores fetch
  jobs, retires planned publications and blocks items when rules change
- GET/PUT/DELETE /api/v1/filename-filters (Idempotency-Key, normalized
  extensions) + Settings and Catalog filter editors
- failJobSQL no longer swallows the claimed row when the failure projection
  upserts (fixes spurious 'lease lost' retries)
- DingTalk live fetch: adoc/axls now use the export-job pipeline (HTTP 403
  from file/download fixed), header-signature downloads supported

2026-09-30 10:54:07 +08:00

姚章浩

63f3b2c503

feat(governance): policy-driven ingestion from catalog annotations

Annotation in 来源与目录 is now the only import decision surface:
- migration 000033: catalog_import_intents (durable, rescan-proof per-file
  intent with locator snapshot + fetch lifecycle), source_items.policy_state,
  planned publication state, ui_setting_revisions.ingestion_concurrency
- marking fans a folder decision out to per-file intents, materializes
  versioned items, reuses already-verified local bytes (compare, don't pull),
  and plans one publication per routed KB; blocked retires planned rows
  while live publications stay untouched (需下架 shows in the KB diff)
- artifact.fetch worker lane (scripts/artifact_fetch.py): DingTalk export
  tree/live box API, AList /api/fs/get direct download with SSO-redirect
  fail-closed, PP ledger via locator hash; content-level dedupe on sha256
- 开始处理 promotes planned->pending inside the claim-time concurrency budget
  from system settings; claims enforce it in SQL
- monitor overview rebuilt as knowledge-base diff: planned/needs_removal per
  KB + fetch pipeline summary; frontend lanes 待入库/需下架 replace the
  deprecated snapshot change-set confirmation
- spool: images (png/jpg/gif/bmp/webp) accepted, ppt/pptx rejected in favour
  of PDF twins; DingTalk PPT resolves its converted PDF pair

2026-09-30 10:54:07 +08:00

3 Commits