The heuristic splitter relied on two regexes that were too strict for
real-world Chinese technical documents:
- ChineseChapterPattern required 第 / numeral / unit to be adjacent, so
the very common "第 1 章 引言" form never matched.
- NumberedSectionPattern required a trailing dot after the numeral, so
multi-level numbering such as "1.1 文档目的" or "2.2.1 用户与权限"
was missed.
As a result, documents like the CHAPTER_SAMPLE shipped with the chunking
debug drawer collected zero heuristic markers, and ProfileDocument fell
all the way through to the character-level Legacy tier, producing chunks
that ignored the document's explicit chapter structure.
- Loosen ChineseChapterPattern to tolerate spaces around 第 / 数字 / 单位.
- Allow the multi-level branch of NumberedSectionPattern to drop its
trailing dot; keep the single-level / roman branch dot-required to
avoid false positives on version strings.
- Extend patterns_test.go with positive and negative cases covering the
CHAPTER_SAMPLE wording plus deep-nesting and lone-numeral regressions.
Adds the building blocks for tiered chunking strategies — multilingual
patterns, token approximation, document profiling, validation, and a
strategy resolver that today still falls through to the legacy
SplitText. Default Strategy is empty/legacy so existing knowledge bases
continue to chunk identically.
- tokens.go: char/token approximation per language (en/de/zh/mixed)
and a coarse DetectLanguage heuristic
- patterns.go: source-of-truth regex bibliothek for headings, numbered
sections, multilingual chapter markers, visual separators, etc.
- profiler.go: single-pass document scan + SelectStrategy chain
- validator.go: rejects obviously broken chunk-sets to drive fallbacks
- strategy.go: public Split / SplitParentChild entry points; resolves
StrategyAuto via the profiler, anything else maps to a fixed tier
- splitter.go: SplitterConfig gains Strategy / TokenLimit / Languages
fields and DefaultChunkSize / DefaultChunkOverlap constants (overlap
raised to 80 ≈ 15%)
- knowledge.go: switches both call sites to chunker.Split /
SplitParentChild — behavior unchanged for empty Strategy
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k