docs(chunking): align overlap default + sharpen UI/source/repo docs
Three documentation passes around the adaptive chunking work: UI - Frontend ChunkOverlap default consolidated to 80 (was 100), matching chunker.DefaultChunkOverlap on the backend. Both DEFAULT_CHUNKING_PRESET and initFormData updated. The KB-load fallback also uses 80 when a loaded KB has no chunk_overlap stored. - All four locales (en-US, zh-CN, ko-KR, ru-RU) get rewritten chunking setting descriptions: each now states the validated range, the default, and the situations where you'd deviate (FAQ vs narrative, embedder token limits, language-specific corpora). Source code - splitter.go: DefaultChunkSize / DefaultChunkOverlap constants get a longer block-comment explaining the per-language token math and the use-case sweet spots, plus the migration note on what the old inconsistent defaults were. - KBChunkingSettings.vue: new comment block above ChunkingConfig documents the slider min/max for each setting, why those bounds exist, and the recommended TokenLimit values per embedding model. Repo docs - New docs/CHUNKING.md: end-to-end guide covering why chunking matters, the adaptive 3-tier architecture, per-setting reference with ranges and sweet spots, parent-child explanation, the token-limit table per embedder (OpenAI / Voyage / Cohere / BGE / MiniLM / Jina), 7 use-case presets, the debug panel workflow, the API surface, and known trade-offs (recursive strategy hidden from UI, no auto-reindex on strategy switch, OCR limitations). - CHANGELOG.md gets a new [Unreleased] section consolidating all the adaptive-chunking work shipped on this branch: 5 features, 8 improvements, 6 fixes, 1 docs entry. The entry references docs/CHUNKING.md for deeper explanation. https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
This commit is contained in:
@@ -409,10 +409,13 @@ const WIKI_ONLY_CHUNKING_PRESET = {
|
||||
enableParentChild: false,
|
||||
} as const
|
||||
|
||||
// 非 Wiki-only 场景下回落到的默认值(与 initFormData 保持一致)。
|
||||
// Non-Wiki-only fallback. Mirrors chunker.DefaultChunkSize and
|
||||
// DefaultChunkOverlap on the backend so a freshly created KB uses
|
||||
// the same numbers whether the editor sets them or the splitter
|
||||
// falls back to its package defaults.
|
||||
const DEFAULT_CHUNKING_PRESET = {
|
||||
chunkSize: 512,
|
||||
chunkOverlap: 100,
|
||||
chunkOverlap: 80,
|
||||
enableParentChild: true,
|
||||
} as const
|
||||
|
||||
@@ -485,7 +488,9 @@ const initFormData = (type: 'document' | 'faq' = 'document') => {
|
||||
},
|
||||
chunkingConfig: {
|
||||
chunkSize: 512,
|
||||
chunkOverlap: 100,
|
||||
// 80 ≈ 15% of chunkSize — community-recommended sweet spot.
|
||||
// Aligned with chunker.DefaultChunkOverlap on the backend.
|
||||
chunkOverlap: 80,
|
||||
separators: ['\n\n', '\n', '。', '!', '?', ';', ';'],
|
||||
parserEngineRules: undefined as any,
|
||||
enableParentChild: true,
|
||||
@@ -586,7 +591,9 @@ const loadKBData = async () => {
|
||||
},
|
||||
chunkingConfig: {
|
||||
chunkSize: kb.chunking_config?.chunk_size || 512,
|
||||
chunkOverlap: kb.chunking_config?.chunk_overlap || 100,
|
||||
// Fallback only used when the loaded KB has no chunk_overlap stored.
|
||||
// Aligned with chunker.DefaultChunkOverlap on the backend.
|
||||
chunkOverlap: kb.chunking_config?.chunk_overlap || 80,
|
||||
separators: kb.chunking_config?.separators || ['\n\n', '\n', '。', '!', '?', ';', ';'],
|
||||
parserEngineRules: kb.chunking_config?.parser_engine_rules || undefined,
|
||||
enableParentChild: kb.chunking_config?.enable_parent_child || false,
|
||||
|
||||
Reference in New Issue
Block a user