feat(governance): policy-driven ingestion from catalog annotations
Annotation in 来源与目录 is now the only import decision surface: - migration 000033: catalog_import_intents (durable, rescan-proof per-file intent with locator snapshot + fetch lifecycle), source_items.policy_state, planned publication state, ui_setting_revisions.ingestion_concurrency - marking fans a folder decision out to per-file intents, materializes versioned items, reuses already-verified local bytes (compare, don't pull), and plans one publication per routed KB; blocked retires planned rows while live publications stay untouched (需下架 shows in the KB diff) - artifact.fetch worker lane (scripts/artifact_fetch.py): DingTalk export tree/live box API, AList /api/fs/get direct download with SSO-redirect fail-closed, PP ledger via locator hash; content-level dedupe on sha256 - 开始处理 promotes planned->pending inside the claim-time concurrency budget from system settings; claims enforce it in SQL - monitor overview rebuilt as knowledge-base diff: planned/needs_removal per KB + fetch pipeline summary; frontend lanes 待入库/需下架 replace the deprecated snapshot change-set confirmation - spool: images (png/jpg/gif/bmp/webp) accepted, ppt/pptx rejected in favour of PDF twins; DingTalk PPT resolves its converted PDF pair
This commit is contained in:
@@ -13,17 +13,29 @@ RETURNING id, type, payload, state, priority, available_at,
|
||||
`
|
||||
|
||||
const claimJobSQL = `
|
||||
WITH candidate AS (
|
||||
SELECT id
|
||||
FROM jobs
|
||||
WITH budget AS (
|
||||
SELECT COALESCE((
|
||||
SELECT ingestion_concurrency
|
||||
FROM ui_setting_revisions
|
||||
WHERE state = 'active'
|
||||
LIMIT 1
|
||||
), 2) AS ingestion_slots
|
||||
), candidate AS (
|
||||
SELECT jobs.id
|
||||
FROM jobs, budget
|
||||
WHERE (
|
||||
(state IN ('pending', 'retry') AND available_at <= $2)
|
||||
OR (state = 'running' AND lease_until < $2)
|
||||
(jobs.state IN ('pending', 'retry') AND jobs.available_at <= $2)
|
||||
OR (jobs.state = 'running' AND jobs.lease_until < $2)
|
||||
)
|
||||
AND attempt < max_attempts
|
||||
AND type = ANY($4::text[])
|
||||
ORDER BY priority DESC, available_at ASC, created_at ASC, id ASC
|
||||
FOR UPDATE SKIP LOCKED
|
||||
AND jobs.attempt < jobs.max_attempts
|
||||
AND jobs.type = ANY($4::text[])
|
||||
AND (
|
||||
jobs.type <> 'test.upload'
|
||||
OR (SELECT COUNT(*) FROM jobs active
|
||||
WHERE active.type = 'test.upload' AND active.state = 'running') < budget.ingestion_slots
|
||||
)
|
||||
ORDER BY jobs.priority DESC, jobs.available_at ASC, jobs.created_at ASC, jobs.id ASC
|
||||
FOR UPDATE OF jobs SKIP LOCKED
|
||||
LIMIT 1
|
||||
)
|
||||
UPDATE jobs AS j
|
||||
|
||||
Reference in New Issue
Block a user