7b1bb1054f
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:
docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
thread-safe so we fan out across processes. Configurable via
DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
image, so documents with many page images are no longer capped by the unary
gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
backward compatibility; the Go production reader switches to ReadStream.
VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
the default 90s -> 180s so dense scanned-page OCR does not time out with
"context deadline exceeded".
Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
a single large document cannot saturate the shared worker pool and block
user-facing document parsing:
image:multimodal -> "multimodal"
chunk:extract -> "graph"
question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
scanned-queue set (so cancelling a knowledge still purges its pending tasks).
26 lines
603 B
TOML
26 lines
603 B
TOML
[project]
|
|
name = "docreader"
|
|
version = "0.1.0"
|
|
description = "Add your description here"
|
|
readme = "README.md"
|
|
requires-python = ">=3.10.18"
|
|
dependencies = [
|
|
"beautifulsoup4>=4.14.2",
|
|
"grpcio>=1.78.0",
|
|
"grpcio-health-checking>=1.78.0",
|
|
"grpcio-tools>=1.78.0",
|
|
"liteparse>=2.0.4",
|
|
"lxml>=6.1.0",
|
|
"markitdown[docx,pdf,xls,xlsx]>=0.1.3",
|
|
"pillow>=12.0.0",
|
|
"playwright>=1.55.0",
|
|
"protobuf>=6.33.0",
|
|
"pydantic>=2.13.4",
|
|
"pypdf>=6.1.3",
|
|
"pypdfium2>=5.8.0",
|
|
"python-docx>=1.2.0",
|
|
"requests>=2.32.5",
|
|
"textract==1.5.0",
|
|
"trafilatura>=2.0.0",
|
|
]
|