docs(chunking): align overlap default + sharpen UI/source/repo docs

Three documentation passes around the adaptive chunking work:

UI
- Frontend ChunkOverlap default consolidated to 80 (was 100), matching
  chunker.DefaultChunkOverlap on the backend. Both DEFAULT_CHUNKING_PRESET
  and initFormData updated. The KB-load fallback also uses 80 when a
  loaded KB has no chunk_overlap stored.
- All four locales (en-US, zh-CN, ko-KR, ru-RU) get rewritten chunking
  setting descriptions: each now states the validated range, the default,
  and the situations where you'd deviate (FAQ vs narrative, embedder
  token limits, language-specific corpora).

Source code
- splitter.go: DefaultChunkSize / DefaultChunkOverlap constants get a
  longer block-comment explaining the per-language token math and the
  use-case sweet spots, plus the migration note on what the old
  inconsistent defaults were.
- KBChunkingSettings.vue: new comment block above ChunkingConfig
  documents the slider min/max for each setting, why those bounds
  exist, and the recommended TokenLimit values per embedding model.

Repo docs
- New docs/CHUNKING.md: end-to-end guide covering why chunking matters,
  the adaptive 3-tier architecture, per-setting reference with ranges
  and sweet spots, parent-child explanation, the token-limit table per
  embedder (OpenAI / Voyage / Cohere / BGE / MiniLM / Jina), 7 use-case
  presets, the debug panel workflow, the API surface, and known
  trade-offs (recursive strategy hidden from UI, no auto-reindex on
  strategy switch, OCR limitations).
- CHANGELOG.md gets a new [Unreleased] section consolidating all the
  adaptive-chunking work shipped on this branch: 5 features, 8
  improvements, 6 fixes, 1 docs entry. The entry references
  docs/CHUNKING.md for deeper explanation.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
This commit is contained in:
Claude
2026-05-03 11:57:03 +00:00
committed by lyingbug
parent 2cc0322004
commit 49adf0d73c
9 changed files with 277 additions and 52 deletions
@@ -409,10 +409,13 @@ const WIKI_ONLY_CHUNKING_PRESET = {
enableParentChild: false,
} as const
// 非 Wiki-only 场景下回落到的默认值(与 initFormData 保持一致)。
// Non-Wiki-only fallback. Mirrors chunker.DefaultChunkSize and
// DefaultChunkOverlap on the backend so a freshly created KB uses
// the same numbers whether the editor sets them or the splitter
// falls back to its package defaults.
const DEFAULT_CHUNKING_PRESET = {
chunkSize: 512,
chunkOverlap: 100,
chunkOverlap: 80,
enableParentChild: true,
} as const
@@ -485,7 +488,9 @@ const initFormData = (type: 'document' | 'faq' = 'document') => {
},
chunkingConfig: {
chunkSize: 512,
chunkOverlap: 100,
// 80 ≈ 15% of chunkSize — community-recommended sweet spot.
// Aligned with chunker.DefaultChunkOverlap on the backend.
chunkOverlap: 80,
separators: ['\n\n', '\n', '。', '!', '?', ';', ';'],
parserEngineRules: undefined as any,
enableParentChild: true,
@@ -586,7 +591,9 @@ const loadKBData = async () => {
},
chunkingConfig: {
chunkSize: kb.chunking_config?.chunk_size || 512,
chunkOverlap: kb.chunking_config?.chunk_overlap || 100,
// Fallback only used when the loaded KB has no chunk_overlap stored.
// Aligned with chunker.DefaultChunkOverlap on the backend.
chunkOverlap: kb.chunking_config?.chunk_overlap || 80,
separators: kb.chunking_config?.separators || ['\n\n', '\n', '。', '!', '?', ';', ';'],
parserEngineRules: kb.chunking_config?.parser_engine_rules || undefined,
enableParentChild: kb.chunking_config?.enable_parent_child || false,