Files
Enterprise-Knowledge-Base/internal
wizardchen 2f86e5b13d feat(chunker): coalesce tiny adjacent chunks in heading splitter
Documents with many short headings (FAQ-style, quick refs) used to fail
the heading-tier validator with "too many tiny chunks" and silently fall
back to legacy splitting, defeating the purpose of heading-aware mode.

Merge physically adjacent sections whose combined size still fits within
ChunkSize, deriving a shared breadcrumb via commonHeadingPrefix so the
merged chunk's ContextHeader stays meaningful.

Tests cover the merge path, position-invariant preservation after merge,
ChunkSize ceiling, and the breadcrumb prefix helper. Existing tests that
relied on tiny fixtures were grown so each section stays distinct.
2026-05-06 20:23:05 +08:00
..
2025-08-05 15:08:07 +08:00