feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements

Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
This commit is contained in:
wizardchen
2026-06-03 12:00:09 +08:00
committed by lyingbug
parent 7b1bb1054f
commit ef1047bf67
50 changed files with 4352 additions and 304 deletions
+36 -1
View File
@@ -562,6 +562,39 @@ DOCREADER_TRANSPORT=grpc
# 渲染页图的最大长边像素(防止超大页面 PDF 渲染出 100+MP 图、撑爆 gRPC 消息上限)
# 调小可进一步减小图片体积;过小会影响 OCR 识别(密集中文建议 >=1600)
# DOCREADER_PDF_RENDER_MAX_EDGE=2000
# Layout text: insert spaces when glyph gaps exceed this × median char width (default 0.4).
# DOCREADER_PDF_WORD_GAP_WIDTH_RATIO=0.4
# Native PDF layout: drop narrow margin columns (arXiv sidebar) below this page-width ratio (default 0.12).
# DOCREADER_PDF_MARGIN_COL_WIDTH_RATIO=0.12
# DOCREADER_PDF_MIN_HEADING_LINE_CHARS=8
# Remove U+FFFE/soft-hyphen artifacts; strip vector chart axis text; render chart areas as JPEG.
# DOCREADER_PDF_SANITIZE_TEXT=true
# DOCREADER_PDF_STRIP_CHART_DEBRIS=true
# DOCREADER_PDF_RENDER_VECTOR_FIGURES=true
# OpenDataLoader PDF(知识库 parser_engine_rules 指定 engine: opendataloader)
# 需 Java 11+;docreader 镜像已包含 openjdk-17-jre-headless。
# DOCREADER_ODL_MAX_WORKERS=1
# fast 模式(默认):DOCREADER_ODL_HYBRID=off
# hybrid 需另起服务;镜像/模型较大,默认 pull/up 与 --full 均不含 odl-hybrid。
# 默认 --no-ocr(不做 EasyOCR)。
# 开发:make dev-start DEV_ARGS=--odl-hybrid(本地 build)
# 生产/docker-compose.yml(需 DOCREADER_ODL_HYBRID=docling-fast 等):
# docker compose --profile odl-hybrid up -d --build odl-hybrid
# 该镜像未发布到 Docker Hub(本地 tag: weknora-odl-hybrid:local),make pull-images 不会拉取,只能按需 build。
# 修改 Dockerfile.odl-hybrid 后需重建:docker compose --profile odl-hybrid build --no-cache odl-hybrid
# ODL_HYBRID_EXTRA_ARGS=--no-ocr
# 扫描件不要用 hybrid OCR,请用 builtin 扫描渲染 + Go OCR,或 MinerU;若坚持 hybrid OCR:
# ODL_HYBRID_EXTRA_ARGS=--force-ocr
# DOCREADER_ODL_HYBRID=docling-fast
# DOCREADER_ODL_HYBRID_URL=http://odl-hybrid:5002
# 开发环境 hybrid:make dev-start DEV_ARGS=--odl-hybrid
# 仅用 fast 模式(不需 odl-hybrid 容器)时请保持 DOCREADER_ODL_HYBRID=off。
# ODL_HYBRID_PORT=5002
# ODL_HYBRID_STARTUP_WAIT_SEC=180
# DOCREADER_ODL_HYBRID_MODE=auto
# DOCREADER_ODL_HYBRID_FALLBACK=false
# DOCREADER_ODL_MARKDOWN_WITH_HTML=false
# VLM(视觉模型)单次 HTTP 请求的整体超时时间(秒)。
# 扫描件整页 OCR(全文+版式抽取)在慢端点上很容易超过默认值,
@@ -648,7 +681,9 @@ DOCREADER_TRANSPORT=grpc
# --- Async pipeline tuning (optional) -----------------------------------------
# Worker pool size for the asynq server. Default 16 — raise it on machines
# that handle many concurrent uploads (default Go runtime.NumCPU() under-
# provisions for the I/O-bound document pipeline).
# provisions for the I/O-bound document pipeline). Can also be set in the
# management UI under Settings → System settings (asynq.concurrency);
# UI changes require a process restart.
# WEKNORA_ASYNQ_CONCURRENCY=16
# Read/write timeout (in milliseconds) the asynq client uses against Redis.