8388a50c23
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.
Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
not accept a context.Context, so when previewTimeout fires the
worker keeps running. Bounding input size keeps worst-case CPU
per request well under a second on commodity hardware. Sized so
10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
follow-up: real cancellation needs the splitter to take a ctx.
Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
reuse a single rune-count per chunk for stats + size + token
estimation. Eliminates the previous triple []rune allocation per
chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
instead of []chunker.Chunk; sumSq computed in float64 to avoid
the int*int overflow at l > ~46k.
Correctness / UX
- Preview panel sends strategy / token_limit / languages
unconditionally, mirroring the buildSubmitData convention so the
preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
gin's cryptic 'Field validation failed on the required tag'.
Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
made the original name a misnomer; the test still asserts the
right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
cases (empty / single / varying / no-variance underflow) plus
PreviewChunking httptest scenarios (auto path, legacy strategy,
empty-text rejection, oversize rejection, chunk truncation with
full-set stats).
Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
obsolete since the heading and heuristic splitters shipped.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
264 lines
9.2 KiB
Go
264 lines
9.2 KiB
Go
// Package chunker - strategy.go is the public entry point for adaptive
|
|
// chunking. Callers invoke Split / SplitParentChild instead of the legacy
|
|
// SplitText / SplitTextParentChild functions; the strategy resolver picks
|
|
// a tier based on document profile and the SplitterConfig.Strategy hint.
|
|
//
|
|
// The legacy entry points still exist in splitter.go for backwards
|
|
// compatibility — strategy.go simply layers a tier-selector on top.
|
|
package chunker
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/Tencent/WeKnora/internal/logger"
|
|
)
|
|
|
|
// Strategy values for SplitterConfig.Strategy.
|
|
const (
|
|
StrategyAuto = "auto"
|
|
StrategyHeading = "heading"
|
|
StrategyHeuristic = "heuristic"
|
|
StrategyRecursive = "recursive"
|
|
StrategyLegacy = "legacy"
|
|
)
|
|
|
|
// Split chunks text using the strategy configured in cfg. When cfg.Strategy
|
|
// is empty or "auto" the document profiler picks the tier. The function
|
|
// always returns a non-nil result: on tier failure the chain falls through
|
|
// to the legacy splitter, which is the original Tier 3 implementation.
|
|
//
|
|
// Hot path: avoids the Diagnostics struct allocation that
|
|
// SplitWithDiagnostics performs (matters in SplitParentChild where
|
|
// Split is called once per parent).
|
|
func Split(text string, cfg SplitterConfig) []Chunk {
|
|
if text == "" {
|
|
return nil
|
|
}
|
|
cfg = ensureDefaults(cfg)
|
|
chain, _ := resolveChainWithProfile(text, cfg)
|
|
totalChars := len([]rune(text))
|
|
|
|
var lastOut []Chunk
|
|
for i, tier := range chain {
|
|
out := runTier(tier, text, cfg)
|
|
if v := ValidateChunks(out, totalChars, cfg.ChunkSize); v.OK {
|
|
return out
|
|
} else {
|
|
logger.Debugf(context.Background(), "chunker: tier %s rejected: %s", tier, v.Reason)
|
|
}
|
|
if tier == TierLegacy && i == len(chain)-1 {
|
|
lastOut = out
|
|
}
|
|
}
|
|
if lastOut != nil {
|
|
return lastOut
|
|
}
|
|
return SplitText(text, cfg)
|
|
}
|
|
|
|
// TierRejection records why a tier was rejected by the validator and the
|
|
// chain advanced to the next tier. Surfaced by SplitWithDiagnostics for
|
|
// the debug/preview endpoint.
|
|
type TierRejection struct {
|
|
Tier StrategyTier `json:"tier"`
|
|
Reason string `json:"reason"`
|
|
}
|
|
|
|
// Diagnostics captures which tier produced the returned chunks plus the
|
|
// chain that was attempted, any rejected tiers along the way, and the
|
|
// document profile that drove tier selection.
|
|
//
|
|
// Useful for surfacing in a debug UI; not produced by the normal Split
|
|
// path. The JSON shape is part of the public preview-endpoint API — keep
|
|
// field names stable.
|
|
type Diagnostics struct {
|
|
SelectedTier StrategyTier `json:"selected_tier"`
|
|
TierChain []StrategyTier `json:"tier_chain"`
|
|
Rejected []TierRejection `json:"rejected"`
|
|
// Profile is set when the auto strategy resolved the chain via the
|
|
// document profiler. nil when an explicit Strategy bypassed profiling.
|
|
Profile *DocProfile `json:"profile,omitempty"`
|
|
}
|
|
|
|
// SplitWithDiagnostics is the same as Split but also returns the
|
|
// diagnostic trace (selected tier, full chain, rejection reasons,
|
|
// profile when available). Use this for the chunker preview endpoint
|
|
// where the caller wants to know which tier won and why others lost.
|
|
func SplitWithDiagnostics(text string, cfg SplitterConfig) ([]Chunk, *Diagnostics) {
|
|
// Default selected tier to legacy so an empty diag never carries the
|
|
// zero string — that would render as a blank tag in the debug UI.
|
|
diag := &Diagnostics{SelectedTier: TierLegacy}
|
|
if text == "" {
|
|
return nil, diag
|
|
}
|
|
cfg = ensureDefaults(cfg)
|
|
chain, profile := resolveChainWithProfile(text, cfg)
|
|
diag.TierChain = chain
|
|
diag.Profile = profile
|
|
totalChars := len([]rune(text))
|
|
|
|
var lastOut []Chunk
|
|
var lastTier StrategyTier
|
|
for i, tier := range chain {
|
|
out := runTier(tier, text, cfg)
|
|
v := ValidateChunks(out, totalChars, cfg.ChunkSize)
|
|
if v.OK {
|
|
diag.SelectedTier = tier
|
|
return out, diag
|
|
}
|
|
diag.Rejected = append(diag.Rejected, TierRejection{Tier: tier, Reason: v.Reason})
|
|
logger.Debugf(context.Background(), "chunker: tier %s rejected: %s", tier, v.Reason)
|
|
if tier == TierLegacy && i == len(chain)-1 {
|
|
lastOut = out
|
|
lastTier = tier
|
|
}
|
|
}
|
|
if lastOut != nil {
|
|
diag.SelectedTier = lastTier
|
|
return lastOut, diag
|
|
}
|
|
// Defensive last-ditch fallback.
|
|
return SplitText(text, cfg), diag
|
|
}
|
|
|
|
// SplitParentChild is the strategy-aware analog of SplitTextParentChild.
|
|
// It runs the tier selector for parent splitting, then re-splits each
|
|
// parent into children with the small-chunk config.
|
|
//
|
|
// Children are forced to recursive splitting — they are sub-pieces of an
|
|
// already-segmented parent, so re-profiling per parent would cost N extra
|
|
// O(N) document scans without any real chance of picking a better tier.
|
|
func SplitParentChild(text string, parentCfg, childCfg SplitterConfig) ParentChildResult {
|
|
if text == "" {
|
|
return ParentChildResult{}
|
|
}
|
|
parentCfg = ensureDefaults(parentCfg)
|
|
childCfg = ensureDefaults(childCfg)
|
|
|
|
parents := Split(text, parentCfg)
|
|
if len(parents) == 0 {
|
|
return ParentChildResult{}
|
|
}
|
|
|
|
// Pin children to the recursive tier so SplitText runs directly without
|
|
// the profiler/strategy chain re-deciding per parent. Preserves Tier-1
|
|
// breadcrumb context (already attached to each parent) since we only
|
|
// re-split parent content, not re-detect headings.
|
|
childCfg.Strategy = StrategyRecursive
|
|
|
|
var newParents []Chunk
|
|
var children []ChildChunk
|
|
childSeq := 0
|
|
for _, parent := range parents {
|
|
subs := Split(parent.Content, childCfg)
|
|
|
|
parentIndex := -1
|
|
if len(subs) > 1 || (len(subs) == 1 && subs[0].Content != parent.Content) {
|
|
parentIndex = len(newParents)
|
|
newParents = append(newParents, parent)
|
|
}
|
|
for _, sub := range subs {
|
|
sub.Seq = childSeq
|
|
sub.Start += parent.Start
|
|
sub.End += parent.Start
|
|
// Children inherit the parent's breadcrumb so embedding still
|
|
// sees the section context — but only if the child itself does
|
|
// not already carry one (sub-splits don't, but defensive).
|
|
if sub.ContextHeader == "" {
|
|
sub.ContextHeader = parent.ContextHeader
|
|
}
|
|
children = append(children, ChildChunk{Chunk: sub, ParentIndex: parentIndex})
|
|
childSeq++
|
|
}
|
|
}
|
|
return ParentChildResult{Parents: newParents, Children: children}
|
|
}
|
|
|
|
// resolveChainWithProfile returns the strategy chain to attempt and, when
|
|
// the chain was selected by the profiler (auto strategy), the DocProfile
|
|
// that drove the selection. Profile is nil for explicit non-auto strategies
|
|
// so callers don't pay for an unused profiling pass.
|
|
func resolveChainWithProfile(text string, cfg SplitterConfig) ([]StrategyTier, *DocProfile) {
|
|
switch cfg.Strategy {
|
|
case StrategyHeading:
|
|
return []StrategyTier{TierHeading, TierLegacy}, nil
|
|
case StrategyHeuristic:
|
|
return []StrategyTier{TierHeuristic, TierLegacy}, nil
|
|
case StrategyRecursive:
|
|
return []StrategyTier{TierRecursive, TierLegacy}, nil
|
|
case StrategyLegacy, "":
|
|
// Empty == legacy preserves backwards compatibility with stored
|
|
// ChunkingConfig rows that pre-date the Strategy field.
|
|
return []StrategyTier{TierLegacy}, nil
|
|
case StrategyAuto:
|
|
fallthrough
|
|
default:
|
|
profile := ProfileDocument(text)
|
|
return SelectStrategy(profile), profile
|
|
}
|
|
}
|
|
|
|
// runTier dispatches the splitter implementation for the given tier.
|
|
// splitByHeadings / splitByHeuristics are package-level vars overridden
|
|
// from heading_splitter.go / heuristic_splitter.go via init(); recursive
|
|
// and legacy share the same SplitText path. The default branch is kept
|
|
// as defensive belt-and-suspenders for future StrategyTier additions.
|
|
func runTier(tier StrategyTier, text string, cfg SplitterConfig) []Chunk {
|
|
switch tier {
|
|
case TierHeading:
|
|
return splitByHeadings(text, cfg)
|
|
case TierHeuristic:
|
|
return splitByHeuristics(text, cfg)
|
|
case TierRecursive, TierLegacy:
|
|
return SplitText(text, cfg)
|
|
}
|
|
return SplitText(text, cfg)
|
|
}
|
|
|
|
// ensureDefaults fills in zero-value config fields with sane defaults.
|
|
// Mirrors buildSplitterConfig in internal/application/service/knowledge.go
|
|
// so direct callers of this package get the same numbers.
|
|
//
|
|
// When cfg.TokenLimit is set, ChunkSize is clamped to the character budget
|
|
// that fits within that token limit (with a 10% safety factor). This makes
|
|
// chunks safe for embedding APIs that have hard token caps.
|
|
func ensureDefaults(cfg SplitterConfig) SplitterConfig {
|
|
if cfg.ChunkSize <= 0 {
|
|
cfg.ChunkSize = DefaultChunkSize
|
|
}
|
|
if cfg.ChunkOverlap <= 0 {
|
|
cfg.ChunkOverlap = DefaultChunkOverlap
|
|
}
|
|
if len(cfg.Separators) == 0 {
|
|
cfg.Separators = []string{"\n\n", "\n", "。"}
|
|
}
|
|
if cfg.TokenLimit > 0 {
|
|
lang := LangMixed
|
|
if len(cfg.Languages) > 0 {
|
|
lang = cfg.Languages[0]
|
|
}
|
|
charBudget := CharsForTokenLimit(cfg.TokenLimit, lang)
|
|
if charBudget > 0 && (cfg.ChunkSize == 0 || charBudget < cfg.ChunkSize) {
|
|
cfg.ChunkSize = charBudget
|
|
}
|
|
}
|
|
// Guard against pathological overlap configurations: if Overlap exceeds
|
|
// half of ChunkSize, almost every chunk is duplicate content. Cap it at
|
|
// ChunkSize/2 so Overlap stays a useful smoothing band rather than a
|
|
// near-clone of the previous chunk.
|
|
if cfg.ChunkOverlap > cfg.ChunkSize/2 && cfg.ChunkSize > 0 {
|
|
cfg.ChunkOverlap = cfg.ChunkSize / 2
|
|
}
|
|
return cfg
|
|
}
|
|
|
|
// splitByHeadings is overridden by heading_splitter.go.
|
|
var splitByHeadings = func(text string, cfg SplitterConfig) []Chunk {
|
|
return SplitText(text, cfg)
|
|
}
|
|
|
|
// splitByHeuristics is overridden by heuristic_splitter.go.
|
|
var splitByHeuristics = func(text string, cfg SplitterConfig) []Chunk {
|
|
return SplitText(text, cfg)
|
|
}
|