feat(chunker): recursive priority splitting + token-aware budget
The Go splitter previously combined all separators into one regex, losing the priority-based recursive behavior of the Python reference. This pass walks the separator list in order: the first separator that yields >1 piece wins, mirroring docreader/splitter/splitter.py. Larger semantic boundaries (\n\n) now win over smaller ones (\n). Additional cross-cutting changes: - ChunkingConfig gains Strategy / TokenLimit / Languages fields so the per-KB config can pick a tier and pin embedding token budgets. - buildSplitterConfig in the knowledge service uses the new chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap consolidated to 80 ≈ 15%) instead of the previous hardcoded 50. - ensureDefaults clamps ChunkSize to a per-language character budget derived from TokenLimit so chunks stay safe under embedding model token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than English (~4 chars/tok). https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
This commit is contained in:
@@ -1786,17 +1786,23 @@ type ProcessChunksOptions struct {
|
||||
}
|
||||
|
||||
// buildSplitterConfig creates a SplitterConfig with fallbacks from a KnowledgeBase.
|
||||
// Defaults mirror chunker.DefaultChunkSize / DefaultChunkOverlap so behavior is
|
||||
// identical whether callers come through this path or invoke the chunker
|
||||
// directly with a zero-value config.
|
||||
func buildSplitterConfig(kb *types.KnowledgeBase) chunker.SplitterConfig {
|
||||
chunkCfg := chunker.SplitterConfig{
|
||||
ChunkSize: kb.ChunkingConfig.ChunkSize,
|
||||
ChunkOverlap: kb.ChunkingConfig.ChunkOverlap,
|
||||
Separators: kb.ChunkingConfig.Separators,
|
||||
Strategy: kb.ChunkingConfig.Strategy,
|
||||
TokenLimit: kb.ChunkingConfig.TokenLimit,
|
||||
Languages: kb.ChunkingConfig.Languages,
|
||||
}
|
||||
if chunkCfg.ChunkSize <= 0 {
|
||||
chunkCfg.ChunkSize = 512
|
||||
chunkCfg.ChunkSize = chunker.DefaultChunkSize
|
||||
}
|
||||
if chunkCfg.ChunkOverlap <= 0 {
|
||||
chunkCfg.ChunkOverlap = 50
|
||||
chunkCfg.ChunkOverlap = chunker.DefaultChunkOverlap
|
||||
}
|
||||
if len(chunkCfg.Separators) == 0 {
|
||||
chunkCfg.Separators = []string{"\n\n", "\n", "。"}
|
||||
|
||||
Reference in New Issue
Block a user