fix(chunker): post-review fixes for preview endpoint robustness

Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.

Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
  not accept a context.Context, so when previewTimeout fires the
  worker keeps running. Bounding input size keeps worst-case CPU
  per request well under a second on commodity hardware. Sized so
  10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
  follow-up: real cancellation needs the splitter to take a ctx.

Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
  reuse a single rune-count per chunk for stats + size + token
  estimation. Eliminates the previous triple []rune allocation per
  chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
  instead of []chunker.Chunk; sumSq computed in float64 to avoid
  the int*int overflow at l > ~46k.

Correctness / UX
- Preview panel sends strategy / token_limit / languages
  unconditionally, mirroring the buildSubmitData convention so the
  preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
  gin's cryptic 'Field validation failed on the required tag'.

Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
  TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
  made the original name a misnomer; the test still asserts the
  right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
  TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
  profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
  cases (empty / single / varying / no-variance underflow) plus
  PreviewChunking httptest scenarios (auto path, legacy strategy,
  empty-text rejection, oversize rejection, chunk truncation with
  full-set stats).

Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
  obsolete since the heading and heuristic splitters shipped.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
This commit is contained in:
Claude
2026-05-03 07:16:47 +00:00
committed by lyingbug
parent beb7d005e9
commit 8388a50c23
6 changed files with 324 additions and 40 deletions
+53 -28
View File
@@ -8,17 +8,28 @@ import (
"context"
"math"
"net/http"
"strings"
"time"
"unicode/utf8"
"github.com/Tencent/WeKnora/internal/infrastructure/chunker"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/gin-gonic/gin"
)
// previewMaxChars caps the input text size so callers can't tie up the
// server with arbitrarily large payloads. Chosen to stay well below the
// 256 KB byte limit even for ASCII (256 KB / ~1 byte/rune).
const previewMaxChars = 256 * 1024
// previewMaxChars caps the input text size so a single preview request
// cannot tie up the splitter for long. Chosen to bound worst-case CPU
// well under the previewTimeout: at 64k runes even the heaviest tier
// chain finishes in well under a second on commodity hardware.
//
// SECURITY NOTE: the splitter is CPU-bound and does NOT accept a
// context.Context. When previewTimeout fires the handler returns to the
// caller, but the worker goroutine keeps running until the splitter
// finishes naturally. The 64k ceiling is the primary mitigation against
// goroutine pile-up under repeated authenticated requests. If the
// splitter ever gains context-awareness, the goroutine-wrapper in
// PreviewChunking should switch to it for true cancellation.
const previewMaxChars = 64 * 1024
// previewMaxChunks caps the number of chunks returned in a single preview
// response so the UI doesn't choke on pathological splits. Stats are
@@ -26,14 +37,15 @@ const previewMaxChars = 256 * 1024
// avg/min/max/stddev stay representative.
const previewMaxChunks = 500
// previewTimeout caps how long the splitter is allowed to run for a single
// preview call. CJK input at the 256k-rune ceiling can otherwise take
// several seconds across all four tier attempts.
// previewTimeout caps how long the handler waits for the splitter
// goroutine before returning a 504. See note above on previewMaxChars.
const previewTimeout = 5 * time.Second
// PreviewChunkingRequest is the body shape accepted by /chunker/preview.
// Text is checked manually below so we can return a friendlier error than
// gin's default "Field validation for 'Text' failed on the 'required' tag".
type PreviewChunkingRequest struct {
Text string `json:"text" binding:"required"`
Text string `json:"text"`
ChunkingConfig PreviewChunkingPayload `json:"chunking_config"`
}
@@ -100,7 +112,15 @@ func PreviewChunking(c *gin.Context) {
return
}
if len([]rune(req.Text)) > previewMaxChars {
if strings.TrimSpace(req.Text) == "" {
c.JSON(http.StatusBadRequest, gin.H{
"success": false,
"error": "text is empty — paste a sample to preview chunking",
})
return
}
if utf8.RuneCountInString(req.Text) > previewMaxChars {
c.JSON(http.StatusRequestEntityTooLarge, gin.H{
"success": false,
"error": "text exceeds preview limit",
@@ -159,24 +179,32 @@ func PreviewChunking(c *gin.Context) {
lang = profile.DetectedLangs[0]
}
// Compute rune lengths once per chunk; reused for stats and result
// payload below. Avoids the previous triple-pass over each chunk's
// content (stats + result + ApproxTokenCount each rune-counted).
runeLens := make([]int, len(chunks))
for i, ch := range chunks {
runeLens[i] = utf8.RuneCountInString(ch.Content)
}
// Compute stats over the FULL chunk set first so the metrics stay
// representative even when we trim the response to previewMaxChunks.
totalCount := len(chunks)
stats := computeChunkSizeStats(chunks, lang)
stats := computeChunkSizeStats(runeLens)
if totalCount > previewMaxChunks {
stats.TruncatedTo = totalCount
chunks = chunks[:previewMaxChunks]
runeLens = runeLens[:previewMaxChunks]
}
results := make([]PreviewChunkResult, 0, len(chunks))
for _, ch := range chunks {
runeLen := len([]rune(ch.Content))
for i, ch := range chunks {
results = append(results, PreviewChunkResult{
Seq: ch.Seq,
Start: ch.Start,
End: ch.End,
SizeChars: runeLen,
SizeTokensApprox: chunker.ApproxTokenCount(ch.Content, lang),
SizeChars: runeLens[i],
SizeTokensApprox: chunker.ApproxTokenCountFromRuneLen(runeLens[i], lang),
ContextHeader: ch.ContextHeader,
Content: ch.Content,
})
@@ -193,23 +221,20 @@ func PreviewChunking(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"success": true, "data": resp})
}
// computeChunkSizeStats walks the full chunk slice once and returns the
// size distribution stats. Operates directly on chunker.Chunk so we don't
// need to materialize PreviewChunkResult before truncation.
//
// lang is forwarded to ApproxTokenCount only if callers extend the stats
// later — currently the result struct only tracks chars.
func computeChunkSizeStats(chunks []chunker.Chunk, _ string) PreviewChunkingStats {
stats := PreviewChunkingStats{Count: len(chunks)}
if len(chunks) == 0 {
// computeChunkSizeStats summarizes count / avg / min / max / stddev from
// a pre-computed rune-length slice. Decoupling from chunker.Chunk lets
// the caller compute rune lengths once and reuse them for the response
// payload (avoids a second []rune allocation per chunk).
func computeChunkSizeStats(runeLens []int) PreviewChunkingStats {
stats := PreviewChunkingStats{Count: len(runeLens)}
if len(runeLens) == 0 {
return stats
}
var sum, sumSq float64
minLen, maxLen := math.MaxInt32, 0
for _, ch := range chunks {
l := len([]rune(ch.Content))
for _, l := range runeLens {
sum += float64(l)
sumSq += float64(l * l)
sumSq += float64(l) * float64(l)
if l < minLen {
minLen = l
}
@@ -217,8 +242,8 @@ func computeChunkSizeStats(chunks []chunker.Chunk, _ string) PreviewChunkingStat
maxLen = l
}
}
avg := sum / float64(len(chunks))
variance := sumSq/float64(len(chunks)) - avg*avg
avg := sum / float64(len(runeLens))
variance := sumSq/float64(len(runeLens)) - avg*avg
if variance < 0 {
// Float precision can push the variance slightly below zero on
// near-uniform inputs; clamp so sqrt doesn't return NaN.