feat(chunker): recursive priority splitting + token-aware budget

The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).

Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
  per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
  chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
  consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
  derived from TokenLimit so chunks stay safe under embedding model
  token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
  English (~4 chars/tok).

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
This commit is contained in:
Claude
2026-05-02 09:22:02 +00:00
committed by lyingbug
parent ae6fde2310
commit 1585a6f12c
5 changed files with 97 additions and 20 deletions
+8 -2
View File
@@ -1786,17 +1786,23 @@ type ProcessChunksOptions struct {
}
// buildSplitterConfig creates a SplitterConfig with fallbacks from a KnowledgeBase.
// Defaults mirror chunker.DefaultChunkSize / DefaultChunkOverlap so behavior is
// identical whether callers come through this path or invoke the chunker
// directly with a zero-value config.
func buildSplitterConfig(kb *types.KnowledgeBase) chunker.SplitterConfig {
chunkCfg := chunker.SplitterConfig{
ChunkSize: kb.ChunkingConfig.ChunkSize,
ChunkOverlap: kb.ChunkingConfig.ChunkOverlap,
Separators: kb.ChunkingConfig.Separators,
Strategy: kb.ChunkingConfig.Strategy,
TokenLimit: kb.ChunkingConfig.TokenLimit,
Languages: kb.ChunkingConfig.Languages,
}
if chunkCfg.ChunkSize <= 0 {
chunkCfg.ChunkSize = 512
chunkCfg.ChunkSize = chunker.DefaultChunkSize
}
if chunkCfg.ChunkOverlap <= 0 {
chunkCfg.ChunkOverlap = 50
chunkCfg.ChunkOverlap = chunker.DefaultChunkOverlap
}
if len(chunkCfg.Separators) == 0 {
chunkCfg.Separators = []string{"\n\n", "\n", "。"}