Files
Enterprise-Knowledge-Base/internal/infrastructure/chunker/tokens.go
T
Claude 8388a50c23 fix(chunker): post-review fixes for preview endpoint robustness
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.

Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
  not accept a context.Context, so when previewTimeout fires the
  worker keeps running. Bounding input size keeps worst-case CPU
  per request well under a second on commodity hardware. Sized so
  10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
  follow-up: real cancellation needs the splitter to take a ctx.

Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
  reuse a single rune-count per chunk for stats + size + token
  estimation. Eliminates the previous triple []rune allocation per
  chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
  instead of []chunker.Chunk; sumSq computed in float64 to avoid
  the int*int overflow at l > ~46k.

Correctness / UX
- Preview panel sends strategy / token_limit / languages
  unconditionally, mirroring the buildSubmitData convention so the
  preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
  gin's cryptic 'Field validation failed on the required tag'.

Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
  TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
  made the original name a misnomer; the test still asserts the
  right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
  TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
  profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
  cases (empty / single / varying / no-variance underflow) plus
  PreviewChunking httptest scenarios (auto path, legacy strategy,
  empty-text rejection, oversize rejection, chunk truncation with
  full-set stats).

Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
  obsolete since the heading and heuristic splitters shipped.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00

160 lines
4.2 KiB
Go

// Package chunker - tokens.go provides language-aware token count approximation.
//
// We avoid pulling in a tokenizer dependency (e.g. tiktoken) and instead use
// per-language chars-per-token ratios derived from common embedding model
// vocabularies. The numbers are conservative — they tend to slightly
// over-estimate token counts so that chunks stay safely under model limits.
package chunker
import (
"unicode"
"unicode/utf8"
)
// Language identifiers used by the token estimator and the heuristic splitter.
const (
LangEnglish = "en"
LangGerman = "de"
LangChinese = "zh"
LangMixed = "mixed"
)
// charsPerToken holds approximate chars/token ratios per language.
// Numbers err on the conservative side so estimates over-shoot a little.
var charsPerToken = map[string]float64{
LangEnglish: 4.0,
LangGerman: 4.5,
LangChinese: 1.7,
LangMixed: 3.0,
}
// ApproxTokenCount returns a conservative token estimate for s in the given
// language. An empty or unknown lang falls back to "mixed".
func ApproxTokenCount(s string, lang string) int {
if s == "" {
return 0
}
return ApproxTokenCountFromRuneLen(utf8.RuneCountInString(s), lang)
}
// ApproxTokenCountFromRuneLen is the allocation-free variant of
// ApproxTokenCount when the caller has already computed the rune length.
// Use this in hot loops where the same content's rune count would
// otherwise be recomputed multiple times (e.g. preview endpoint emitting
// per-chunk stats).
func ApproxTokenCountFromRuneLen(runeLen int, lang string) int {
if runeLen <= 0 {
return 0
}
ratio, ok := charsPerToken[lang]
if !ok {
ratio = charsPerToken[LangMixed]
}
approx := float64(runeLen) / ratio
if approx < 1 {
return 1
}
return int(approx + 0.5)
}
// DetectLanguage returns a coarse language label by counting CJK runes vs.
// Latin runes. The result is one of LangChinese, LangGerman, LangEnglish or
// LangMixed. Detection is cheap and meant only for heuristic dispatch — it
// is NOT a replacement for proper language identification.
func DetectLanguage(s string) string {
if s == "" {
return LangMixed
}
var cjk, latin, umlaut int
for _, r := range s {
switch {
case unicode.Is(unicode.Han, r) || unicode.Is(unicode.Hangul, r) || unicode.Is(unicode.Hiragana, r) || unicode.Is(unicode.Katakana, r):
cjk++
case isGermanUmlaut(r):
umlaut++
latin++
case (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z'):
latin++
}
}
total := cjk + latin
if total == 0 {
return LangMixed
}
cjkRatio := float64(cjk) / float64(total)
latinRatio := float64(latin) / float64(total)
// Mixed: meaningful presence of both scripts (>=15% each).
if cjkRatio >= 0.15 && latinRatio >= 0.15 {
return LangMixed
}
if cjkRatio > 0.3 {
return LangChinese
}
if umlaut > 0 || hasGermanWords(s) {
return LangGerman
}
return LangEnglish
}
func isGermanUmlaut(r rune) bool {
switch r {
case 'ä', 'ö', 'ü', 'Ä', 'Ö', 'Ü', 'ß':
return true
}
return false
}
// hasGermanWords does a tiny stop-word check to bias towards "de" when the
// text uses common German function words. Cheap heuristic — false positives
// on borrowed terms are acceptable.
func hasGermanWords(s string) bool {
const sample = 512
if len(s) > sample {
s = s[:sample]
}
for _, w := range []string{" der ", " die ", " das ", " und ", " ist ", " nicht ", " mit ", " auf "} {
if containsLower(s, w) {
return true
}
}
return false
}
func containsLower(haystack, needle string) bool {
if len(haystack) < len(needle) {
return false
}
for i := 0; i+len(needle) <= len(haystack); i++ {
match := true
for j := 0; j < len(needle); j++ {
h := haystack[i+j]
if h >= 'A' && h <= 'Z' {
h += 'a' - 'A'
}
if h != needle[j] {
match = false
break
}
}
if match {
return true
}
}
return false
}
// CharsForTokenLimit converts a token limit into an approximate character
// budget for a given language. Used to size chunks so they fit within an
// embedding model's max-token window with a small safety margin.
func CharsForTokenLimit(tokens int, lang string) int {
if tokens <= 0 {
return 0
}
ratio, ok := charsPerToken[lang]
if !ok {
ratio = charsPerToken[LangMixed]
}
// 0.9 safety factor so we under-shoot the model limit instead of overshooting.
return int(float64(tokens) * ratio * 0.9)
}