feat(governance): policy-driven ingestion from catalog annotations

Annotation in 来源与目录 is now the only import decision surface:
- migration 000033: catalog_import_intents (durable, rescan-proof per-file
  intent with locator snapshot + fetch lifecycle), source_items.policy_state,
  planned publication state, ui_setting_revisions.ingestion_concurrency
- marking fans a folder decision out to per-file intents, materializes
  versioned items, reuses already-verified local bytes (compare, don't pull),
  and plans one publication per routed KB; blocked retires planned rows
  while live publications stay untouched (需下架 shows in the KB diff)
- artifact.fetch worker lane (scripts/artifact_fetch.py): DingTalk export
  tree/live box API, AList /api/fs/get direct download with SSO-redirect
  fail-closed, PP ledger via locator hash; content-level dedupe on sha256
- 开始处理 promotes planned->pending inside the claim-time concurrency budget
  from system settings; claims enforce it in SQL
- monitor overview rebuilt as knowledge-base diff: planned/needs_removal per
  KB + fetch pipeline summary; frontend lanes 待入库/需下架 replace the
  deprecated snapshot change-set confirmation
- spool: images (png/jpg/gif/bmp/webp) accepted, ppt/pptx rejected in favour
  of PDF twins; DingTalk PPT resolves its converted PDF pair
This commit is contained in:
姚章浩
2026-09-13 22:17:12 +08:00
parent f6cb90cd70
commit 63f3b2c503
38 changed files with 1597 additions and 307 deletions
+21 -9
View File
@@ -13,17 +13,29 @@ RETURNING id, type, payload, state, priority, available_at,
`
const claimJobSQL = `
WITH candidate AS (
SELECT id
FROM jobs
WITH budget AS (
SELECT COALESCE((
SELECT ingestion_concurrency
FROM ui_setting_revisions
WHERE state = 'active'
LIMIT 1
), 2) AS ingestion_slots
), candidate AS (
SELECT jobs.id
FROM jobs, budget
WHERE (
(state IN ('pending', 'retry') AND available_at <= $2)
OR (state = 'running' AND lease_until < $2)
(jobs.state IN ('pending', 'retry') AND jobs.available_at <= $2)
OR (jobs.state = 'running' AND jobs.lease_until < $2)
)
AND attempt < max_attempts
AND type = ANY($4::text[])
ORDER BY priority DESC, available_at ASC, created_at ASC, id ASC
FOR UPDATE SKIP LOCKED
AND jobs.attempt < jobs.max_attempts
AND jobs.type = ANY($4::text[])
AND (
jobs.type <> 'test.upload'
OR (SELECT COUNT(*) FROM jobs active
WHERE active.type = 'test.upload' AND active.state = 'running') < budget.ingestion_slots
)
ORDER BY jobs.priority DESC, jobs.available_at ASC, jobs.created_at ASC, jobs.id ASC
FOR UPDATE OF jobs SKIP LOCKED
LIMIT 1
)
UPDATE jobs AS j