knowledge-governance

Weknora/knowledge-governance

Fork 0

Commit Graph

Author	SHA1	Message	Date
姚章浩	f4e1ecc5f1	feat(artifact): 钉钉取件优先走 dws（DingTalk Workspace CLI），老路径保留为回退 - artifact_fetch.py: 新增 dingtalk dws 取件器（adoc→doc export、axls→sheet 转换、其余→drive 直下）， ARTIFACT_DINGTALK_FETCHER=dws\|auto\|legacy 三级开关，auto 仅对可重试错误回落老路径；平台侧判定（pick_suffix/SLIDES/magic/大小上限/sha256/spool/空壳检查）全部保留；结果 JSON 字段不变（Go runner 用 DisallowUnknownFields 解析 stdout），取件器来源只写 stderr。 - axls_to_xlsx.py: 纯标准库 axls→xlsx（文本写入 sharedStrings 以通过空壳判定）。 - dws_error_map.py: dws 错误→平台 {ok,code,retryable,missing}；未分类错误默认不重试。 - Dockerfile: alpine 增加 gcompat/libc6-compat；dws v1.0.63 固定版本+sha256 校验；构建期 dws version 自检。 - compose.yaml: worker 增加 dws 环境与凭据挂载（DWS_HOME 方案，XDG_DATA_HOME 不被 dws 识别）。 - 测试: artifact_fetch_test 新增 11 项（路由/回落语义/契约字段锁定）；dws_error_map 17 断言。 - 实测: 真容器(--read-only --tmpfs --user 1000:1000) 端到端 adoc/axls/pdf 3/3 成功，docx sha 与平台基准一致。	2026-10-09 10:57:11 +08:00
姚章浩	257be88ecf	fix(fetch): UTF-8 sniff boundary must not reject valid documents validate_magic read exactly 4096 bytes and strict-decoded them; a multibyte character straddling the boundary made perfectly valid .md/.txt files fail with invalid_format. Only tolerate 'unexpected end of data' when the file actually continues past the sniff window; genuinely corrupt bytes still fail.	2026-09-30 10:54:07 +08:00
姚章浩	b3c9331966	feat(ingest): 源与目录文件名过滤及钉钉文档导出 - filename_filter_rules table (000035) with source-root and per-directory rows - additive inheritance: source default + every ancestor directory contributes - filename_is_filtered() SQL predicate applied to catalog browsing, subtree counts, change-set diffing, routing fan-out and artifact fetch manifests - fetch_state 'filtered' (CHECK extended); reconcile cancels/restores fetch jobs, retires planned publications and blocks items when rules change - GET/PUT/DELETE /api/v1/filename-filters (Idempotency-Key, normalized extensions) + Settings and Catalog filter editors - failJobSQL no longer swallows the claimed row when the failure projection upserts (fixes spurious 'lease lost' retries) - DingTalk live fetch: adoc/axls now use the export-job pipeline (HTTP 403 from file/download fixed), header-signature downloads supported	2026-09-30 10:54:07 +08:00

Author

SHA1

Message

Date

姚章浩

f4e1ecc5f1

feat(artifact): 钉钉取件优先走 dws（DingTalk Workspace CLI），老路径保留为回退

- artifact_fetch.py: 新增 dingtalk dws 取件器（adoc→doc export、axls→sheet 转换、其余→drive 直下），
  ARTIFACT_DINGTALK_FETCHER=dws|auto|legacy 三级开关，auto 仅对可重试错误回落老路径；
  平台侧判定（pick_suffix/SLIDES/magic/大小上限/sha256/spool/空壳检查）全部保留；
  结果 JSON 字段不变（Go runner 用 DisallowUnknownFields 解析 stdout），取件器来源只写 stderr。
- axls_to_xlsx.py: 纯标准库 axls→xlsx（文本写入 sharedStrings 以通过空壳判定）。
- dws_error_map.py: dws 错误→平台 {ok,code,retryable,missing}；未分类错误默认不重试。
- Dockerfile: alpine 增加 gcompat/libc6-compat；dws v1.0.63 固定版本+sha256 校验；构建期 dws version 自检。
- compose.yaml: worker 增加 dws 环境与凭据挂载（DWS_HOME 方案，XDG_DATA_HOME 不被 dws 识别）。
- 测试: artifact_fetch_test 新增 11 项（路由/回落语义/契约字段锁定）；dws_error_map 17 断言。
- 实测: 真容器(--read-only --tmpfs --user 1000:1000) 端到端 adoc/axls/pdf 3/3 成功，docx sha 与平台基准一致。

2026-10-09 10:57:11 +08:00

姚章浩

257be88ecf

fix(fetch): UTF-8 sniff boundary must not reject valid documents

validate_magic read exactly 4096 bytes and strict-decoded them; a multibyte
character straddling the boundary made perfectly valid .md/.txt files fail
with invalid_format. Only tolerate 'unexpected end of data' when the file
actually continues past the sniff window; genuinely corrupt bytes still fail.

2026-09-30 10:54:07 +08:00

姚章浩

b3c9331966

feat(ingest): 源与目录文件名过滤及钉钉文档导出

- filename_filter_rules table (000035) with source-root and per-directory rows
- additive inheritance: source default + every ancestor directory contributes
- filename_is_filtered() SQL predicate applied to catalog browsing, subtree
  counts, change-set diffing, routing fan-out and artifact fetch manifests
- fetch_state 'filtered' (CHECK extended); reconcile cancels/restores fetch
  jobs, retires planned publications and blocks items when rules change
- GET/PUT/DELETE /api/v1/filename-filters (Idempotency-Key, normalized
  extensions) + Settings and Catalog filter editors
- failJobSQL no longer swallows the claimed row when the failure projection
  upserts (fixes spurious 'lease lost' retries)
- DingTalk live fetch: adoc/axls now use the export-job pipeline (HTTP 403
  from file/download fixed), header-signature downloads supported

2026-09-30 10:54:07 +08:00

3 Commits