fix(chunker): post-review fixes for preview endpoint robustness

Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.

Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
  not accept a context.Context, so when previewTimeout fires the
  worker keeps running. Bounding input size keeps worst-case CPU
  per request well under a second on commodity hardware. Sized so
  10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
  follow-up: real cancellation needs the splitter to take a ctx.

Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
  reuse a single rune-count per chunk for stats + size + token
  estimation. Eliminates the previous triple []rune allocation per
  chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
  instead of []chunker.Chunk; sumSq computed in float64 to avoid
  the int*int overflow at l > ~46k.

Correctness / UX
- Preview panel sends strategy / token_limit / languages
  unconditionally, mirroring the buildSubmitData convention so the
  preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
  gin's cryptic 'Field validation failed on the required tag'.

Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
  TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
  made the original name a misnomer; the test still asserts the
  right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
  TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
  profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
  cases (empty / single / varying / no-variance underflow) plus
  PreviewChunking httptest scenarios (auto path, legacy strategy,
  empty-text rejection, oversize rejection, chunk truncation with
  full-set stats).

Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
  obsolete since the heading and heuristic splitters shipped.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
This commit is contained in:
Claude
2026-05-03 07:16:47 +00:00
committed by lyingbug
parent beb7d005e9
commit 8388a50c23
6 changed files with 324 additions and 40 deletions
@@ -161,7 +161,8 @@ interface Props {
const props = defineProps<Props>()
const { t } = useI18n()
const MAX_CHARS = 256 * 1024
// Mirrors handler.previewMaxChars on the backend. Keep in sync.
const MAX_CHARS = 64 * 1024
const open = ref(false)
const sample = ref('')
@@ -181,15 +182,18 @@ const runPreview = async () => {
result.value = null
expandedChunks.value = new Set()
try {
// Send all fields explicitly (including empty / 0 / []) so the
// preview faithfully reflects what would happen on save. Mirrors
// the buildSubmitData convention in KnowledgeBaseEditorModal.
const resp = await previewChunking({
text: sample.value,
chunking_config: {
chunk_size: props.config.chunkSize,
chunk_overlap: props.config.chunkOverlap,
separators: props.config.separators,
strategy: props.config.strategy || undefined,
token_limit: props.config.tokenLimit || undefined,
languages: props.config.languages?.length ? props.config.languages : undefined
strategy: props.config.strategy ?? '',
token_limit: props.config.tokenLimit ?? 0,
languages: props.config.languages ?? []
}
})
if (!resp.success) {