feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing. Optional odl-hybrid sidecar stays local-build only and is excluded from default dev-start and full profiles.
This commit is contained in:
+36
-1
@@ -562,6 +562,39 @@ DOCREADER_TRANSPORT=grpc
|
||||
# 渲染页图的最大长边像素(防止超大页面 PDF 渲染出 100+MP 图、撑爆 gRPC 消息上限)
|
||||
# 调小可进一步减小图片体积;过小会影响 OCR 识别(密集中文建议 >=1600)
|
||||
# DOCREADER_PDF_RENDER_MAX_EDGE=2000
|
||||
# Layout text: insert spaces when glyph gaps exceed this × median char width (default 0.4).
|
||||
# DOCREADER_PDF_WORD_GAP_WIDTH_RATIO=0.4
|
||||
# Native PDF layout: drop narrow margin columns (arXiv sidebar) below this page-width ratio (default 0.12).
|
||||
# DOCREADER_PDF_MARGIN_COL_WIDTH_RATIO=0.12
|
||||
# DOCREADER_PDF_MIN_HEADING_LINE_CHARS=8
|
||||
# Remove U+FFFE/soft-hyphen artifacts; strip vector chart axis text; render chart areas as JPEG.
|
||||
# DOCREADER_PDF_SANITIZE_TEXT=true
|
||||
# DOCREADER_PDF_STRIP_CHART_DEBRIS=true
|
||||
# DOCREADER_PDF_RENDER_VECTOR_FIGURES=true
|
||||
|
||||
# OpenDataLoader PDF(知识库 parser_engine_rules 指定 engine: opendataloader)
|
||||
# 需 Java 11+;docreader 镜像已包含 openjdk-17-jre-headless。
|
||||
# DOCREADER_ODL_MAX_WORKERS=1
|
||||
# fast 模式(默认):DOCREADER_ODL_HYBRID=off
|
||||
# hybrid 需另起服务;镜像/模型较大,默认 pull/up 与 --full 均不含 odl-hybrid。
|
||||
# 默认 --no-ocr(不做 EasyOCR)。
|
||||
# 开发:make dev-start DEV_ARGS=--odl-hybrid(本地 build)
|
||||
# 生产/docker-compose.yml(需 DOCREADER_ODL_HYBRID=docling-fast 等):
|
||||
# docker compose --profile odl-hybrid up -d --build odl-hybrid
|
||||
# 该镜像未发布到 Docker Hub(本地 tag: weknora-odl-hybrid:local),make pull-images 不会拉取,只能按需 build。
|
||||
# 修改 Dockerfile.odl-hybrid 后需重建:docker compose --profile odl-hybrid build --no-cache odl-hybrid
|
||||
# ODL_HYBRID_EXTRA_ARGS=--no-ocr
|
||||
# 扫描件不要用 hybrid OCR,请用 builtin 扫描渲染 + Go OCR,或 MinerU;若坚持 hybrid OCR:
|
||||
# ODL_HYBRID_EXTRA_ARGS=--force-ocr
|
||||
# DOCREADER_ODL_HYBRID=docling-fast
|
||||
# DOCREADER_ODL_HYBRID_URL=http://odl-hybrid:5002
|
||||
# 开发环境 hybrid:make dev-start DEV_ARGS=--odl-hybrid
|
||||
# 仅用 fast 模式(不需 odl-hybrid 容器)时请保持 DOCREADER_ODL_HYBRID=off。
|
||||
# ODL_HYBRID_PORT=5002
|
||||
# ODL_HYBRID_STARTUP_WAIT_SEC=180
|
||||
# DOCREADER_ODL_HYBRID_MODE=auto
|
||||
# DOCREADER_ODL_HYBRID_FALLBACK=false
|
||||
# DOCREADER_ODL_MARKDOWN_WITH_HTML=false
|
||||
|
||||
# VLM(视觉模型)单次 HTTP 请求的整体超时时间(秒)。
|
||||
# 扫描件整页 OCR(全文+版式抽取)在慢端点上很容易超过默认值,
|
||||
@@ -648,7 +681,9 @@ DOCREADER_TRANSPORT=grpc
|
||||
# --- Async pipeline tuning (optional) -----------------------------------------
|
||||
# Worker pool size for the asynq server. Default 16 — raise it on machines
|
||||
# that handle many concurrent uploads (default Go runtime.NumCPU() under-
|
||||
# provisions for the I/O-bound document pipeline).
|
||||
# provisions for the I/O-bound document pipeline). Can also be set in the
|
||||
# management UI under Settings → System settings (asynq.concurrency);
|
||||
# UI changes require a process restart.
|
||||
# WEKNORA_ASYNQ_CONCURRENCY=16
|
||||
|
||||
# Read/write timeout (in milliseconds) the asynq client uses against Redis.
|
||||
|
||||
Reference in New Issue
Block a user