Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.
Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
not accept a context.Context, so when previewTimeout fires the
worker keeps running. Bounding input size keeps worst-case CPU
per request well under a second on commodity hardware. Sized so
10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
follow-up: real cancellation needs the splitter to take a ctx.
Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
reuse a single rune-count per chunk for stats + size + token
estimation. Eliminates the previous triple []rune allocation per
chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
instead of []chunker.Chunk; sumSq computed in float64 to avoid
the int*int overflow at l > ~46k.
Correctness / UX
- Preview panel sends strategy / token_limit / languages
unconditionally, mirroring the buildSubmitData convention so the
preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
gin's cryptic 'Field validation failed on the required tag'.
Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
made the original name a misnomer; the test still asserts the
right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
cases (empty / single / varying / no-variance underflow) plus
PreviewChunking httptest scenarios (auto path, legacy strategy,
empty-text rejection, oversize rejection, chunk truncation with
full-set stats).
Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
obsolete since the heading and heuristic splitters shipped.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by the post-Phase-3 self-review.
High severity
- Strategy / TokenLimit / Languages can now be reset. The KB-config
handler uses pointer DTOs so the absence of a field in the payload
means "no change" while an explicit empty string / 0 / [] clears
the value. The frontend now sends the fields unconditionally so
the user can revert to defaults via the clear icon — previously
these fields were write-once.
- DocProfile is no longer computed twice in the preview endpoint.
Diagnostics carries the profile that drove tier selection (when
auto-strategy ran); the handler only re-profiles for explicit
strategies that bypass the profiler.
- Chunk-size stats (avg / min / max / stddev) are now computed over
the FULL chunk set before the response is truncated to
previewMaxChunks. Avoids systematically misleading metrics for
large documents.
- SplitWithDiagnostics defaults SelectedTier to TierLegacy so an
empty-text request never leaves the diag with a blank tier string.
Medium severity
- Split() reverts to its own loop instead of delegating to
SplitWithDiagnostics. Diagnostics struct allocation is reserved
for callers that actually want the trace (preview endpoint).
Matters in SplitParentChild where Split runs once per parent.
- Preview endpoint runs the splitter on a goroutine wrapped in a
5s context.WithTimeout. CJK input at the 256k-rune ceiling
could otherwise tie up the worker for several seconds.
Low severity
- KBChunkingSettings emits arrays via spread copies so parent-side
mutation cannot drift the form's reactive state.
- KBChunkingDebug font-family includes a monospace fallback so the
chunk content stays monospaced even when the TDesign CSS variable
is missing.
Documentation
- DocProfile JSON shape is now annotated as part of the public
preview API; field renames would be breaking.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds POST /api/v1/chunker/preview — a read-only endpoint that runs the
adaptive chunker on supplied text and returns the chunks plus
diagnostic info (which tier won, which were rejected and why, the
DocProfile that the selector saw). Used by the upcoming KB editor
debug panel so users can experiment with chunking parameters without
re-uploading documents.
Backend pieces:
- chunker.SplitWithDiagnostics(text, cfg) returns chunks plus a
Diagnostics struct (SelectedTier, TierChain, Rejected). Split() now
delegates here.
- chunker.DocProfile gets JSON tags so it serializes cleanly.
- handler.PreviewChunking caps input at 256k chars and chunks at 500
per response, computes a size-distribution summary, and never
touches DB or embedding APIs.
Auth: lives under the v1 group so the standard JWT middleware applies.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Claude