Files
Enterprise-Knowledge-Base/docreader/pyproject.toml
T
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00

26 lines
603 B
TOML

[project]
name = "docreader"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.10.18"
dependencies = [
"beautifulsoup4>=4.14.2",
"grpcio>=1.78.0",
"grpcio-health-checking>=1.78.0",
"grpcio-tools>=1.78.0",
"liteparse>=2.0.4",
"lxml>=6.1.0",
"markitdown[docx,pdf,xls,xlsx]>=0.1.3",
"pillow>=12.0.0",
"playwright>=1.55.0",
"protobuf>=6.33.0",
"pydantic>=2.13.4",
"pypdf>=6.1.3",
"pypdfium2>=5.8.0",
"python-docx>=1.2.0",
"requests>=2.32.5",
"textract==1.5.0",
"trafilatura>=2.0.0",
]