Files
Enterprise-Knowledge-Base/internal
wizardchen ad69240ac1 fix(chunker): heuristic splitter drops boundaries inside protected spans
Heuristic boundary detection (numbered sections, all-caps headings,
\\n{3,} blank blocks, etc.) ran on the raw text and could land inside
atomic regions handled by protectedPatterns — most notably LaTeX
$$...$$ blocks, Markdown tables, fenced code, and image/link refs.
A boundary inside such a region would cause the bin-packer to slice
through protected content, defeating the protection.

Convert protectedSpans output to rune offsets once and filter the
boundary list before bin-packing. Boundaries on a span edge are kept
(they align with the span) — only strictly-interior ones are dropped.
2026-05-06 21:16:48 +08:00
..
2025-08-05 15:08:07 +08:00