Add per-stream once-only logs at the OpenAI-protocol layer so we can
triage streaming behavior (natural-stop vs tool-call, TTFC, ordering of
reasoning/content/tool_calls) without grepping through every delta.
streamState now tracks:
- streamStartedAt : baseline for elapsed_ms on each fire-once log
- firstContentSeen : delta.Content first appearance
- firstReasoningSeen : reasoning_content first appearance
- firstToolCallSeen / noToolCallStopLogged (existing flags retained)
Logs emitted at most once per stream:
[LLM Stream] First reasoning_content at OpenAI layer (len, preview, elapsed_ms)
[LLM Stream] First delta.Content at OpenAI layer (len, preview, tool_call_seen, thinking_seen, elapsed_ms)
[LLM Stream] First tool_calls delta at OpenAI layer (count, first_id, first_name, first_content_seen, thinking_seen, elapsed_ms)
[LLM Stream] Natural-stop at OpenAI layer (finish=stop, tool_calls field never observed, thinking_seen, first_content_seen, elapsed_ms)
Together with the existing agent-layer "Natural-stop candidate detected"
log this lets a single grep reconstruct the temporal layout of one stream
(reasoning -> content -> tool_call OR natural-stop) and tell apart:
(A) tool_calls field truly absent (real natural-stop)
(B) tool_calls field arrived but high-level marker not yet emitted
Pure logging change; no behavior change. Previews use the existing
truncateForDebug helper (rune-safe, capped at 80 chars).
Add explicit response logging for the branch where the model returns
finish=stop with zero tool calls. This makes it easier to identify the
natural-stop path early in logs and correlate rounds that are likely to
be treated as final-answer candidates by analyzeResponse.
- include `tool_calls=0` in the no-tool summary log
- emit a dedicated "Natural-stop candidate detected" info log when
finish_reason is stop and no tool calls are present
Closes#1113
The graph extraction pipeline relies on a strict markdown fence regex to
pull JSON out of LLM responses. In production the regex misses ~84% of
real-world responses, in three ways:
1. The LLM hits max_tokens mid-output and produces no closing fence.
2. The opening fence is malformed or surrounded by prose, so the
non-greedy regex fails to anchor a match.
3. The model returns raw JSON with stray backticks but no real fence.
In every case extractContent fell through to returning the raw text,
which then failed json.Unmarshal with errors like
"invalid character '` + "`" + "' looking for beginning of value".
This change keeps the existing happy path untouched and adds a
conservative recovery step in the default branch of extractContent:
- If an opening ``` is present, take everything after it, drop a
likely language tag on the first line, cut at any trailing closing
fence, and trim stray backticks/whitespace.
- Otherwise, look for an outermost JSON object/array in the text using
a small bracket-balanced scanner that respects string literals, so
embedded {} or [] inside JSON strings don't confuse it.
- Only fall back to the original raw-text behavior when neither
strategy yields anything plausible.
The recovery helpers (stripFencesAndExtract, extractJSONLike,
isLikelyLanguageTag) are package-private and have table-driven tests in
extract_entity_test.go covering the three failure patterns described in
the issue, plus the previously-working fenced and bare-JSON shapes to
guard against regressions.
The heuristic splitter relied on two regexes that were too strict for
real-world Chinese technical documents:
- ChineseChapterPattern required 第 / numeral / unit to be adjacent, so
the very common "第 1 章 引言" form never matched.
- NumberedSectionPattern required a trailing dot after the numeral, so
multi-level numbering such as "1.1 文档目的" or "2.2.1 用户与权限"
was missed.
As a result, documents like the CHAPTER_SAMPLE shipped with the chunking
debug drawer collected zero heuristic markers, and ProfileDocument fell
all the way through to the character-level Legacy tier, producing chunks
that ignored the document's explicit chapter structure.
- Loosen ChineseChapterPattern to tolerate spaces around 第 / 数字 / 单位.
- Allow the multi-level branch of NumberedSectionPattern to drop its
trailing dot; keep the single-level / roman branch dot-required to
avoid false positives on version strings.
- Extend patterns_test.go with positive and negative cases covering the
CHAPTER_SAMPLE wording plus deep-nesting and lone-numeral regressions.
Children were forced to StrategyRecursive on the assumption that
re-profiling each parent would be too expensive. In practice profiling
each parent is bounded by O(sum(parent_size)) ≈ O(N) total, the same
order as the original parent profiling pass — and the gain is real:
when a parent (e.g. an H1 chapter) contains its own sub-headings, the
heading splitter on child input now picks them up and generates a
finer-grained breadcrumb instead of every child sharing the parent's
top-level breadcrumb.
Add mergeBreadcrumbs to combine the parent ContextHeader with the
child's freshly-derived one, dropping the duplicated seam line that
appears when the child's first heading equals the parent's last.
Tests cover (a) sub-headings now appearing in child breadcrumbs and
(b) the merge dedup against duplicated seam lines.
Heuristic and recursive splitters preserve original byte positions in
Chunk.Content (the End-Start == RuneCountInString invariant is required
by document-reconstruction code paths), which means a chunk sliced at a
boundary often carries leading/trailing newlines from the boundary
itself. Feeding that whitespace into the embedding model dilutes the
vector and wastes tokens for no benefit.
TrimSpace the body inside EmbeddingContent for both chunker.Chunk and
the two types.Chunk / types.ParsedChunk mirrors. Inner whitespace is
preserved; positions/Content are unchanged.
Also clean up appendChunk: the previous code computed a trimmed string
for the empty-check then stored a separate untrimmed copy, which read
as if it might be intentionally divergent. Replace with an explicit
'raw text, skip if pure whitespace' shape and document the invariant.
Heuristic boundary detection (numbered sections, all-caps headings,
\\n{3,} blank blocks, etc.) ran on the raw text and could land inside
atomic regions handled by protectedPatterns — most notably LaTeX
$$...$$ blocks, Markdown tables, fenced code, and image/link refs.
A boundary inside such a region would cause the bin-packer to slice
through protected content, defeating the protection.
Convert protectedSpans output to rune offsets once and filter the
boundary list before bin-packing. Boundaries on a span edge are kept
(they align with the span) — only strictly-interior ones are dropped.
Strategy.Split / SplitWithDiagnostics already run ProfileDocument once
when the auto strategy resolves the chain. splitByHeadingsImpl was
rerunning the same O(N) pass on entry, so every auto-mode call paid
2x scan cost; SplitParentChild paid 2x per parent.
Pass the profile down through runTier and let splitters compute their
own only when called outside the auto path (where profile == nil).
applyOverlapAligned searched for the latest boundary in
[curEnd-2*overlap, curEnd], but curEnd itself is always one of the
boundaries (the bin-packer only flushes at boundary positions). The
loop therefore always returned curEnd, producing chunkStart == curEnd
and zero overlap regardless of cfg.ChunkOverlap.
Exclude curEnd from the search window so an earlier boundary can be
picked, restoring the intended overlap behaviour.
TierRecursive and TierLegacy both invoked SplitText with identical
output, but only TierLegacy got the "always-return-on-validation-failure"
safety-net behavior. Auto chains thus ran SplitText twice on the
fallback path and produced two-line debug rejection traces with the same
reason.
Inline TierRecursive into TierLegacy: SelectStrategy and the explicit
StrategyRecursive entry point now emit single-legacy chains. The
StrategyRecursive public constant stays so existing ChunkingConfig rows
keep parsing — it's just an alias for legacy now.
No user-visible behavior change; one fewer SplitText call per failed
auto fallback.
Documents with many short headings (FAQ-style, quick refs) used to fail
the heading-tier validator with "too many tiny chunks" and silently fall
back to legacy splitting, defeating the purpose of heading-aware mode.
Merge physically adjacent sections whose combined size still fits within
ChunkSize, deriving a shared breadcrumb via commonHeadingPrefix so the
merged chunk's ContextHeader stays meaningful.
Tests cover the merge path, position-invariant preservation after merge,
ChunkSize ceiling, and the breadcrumb prefix helper. Existing tests that
relied on tiny fixtures were grown so each section stays distinct.
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
Three documentation passes around the adaptive chunking work:
UI
- Frontend ChunkOverlap default consolidated to 80 (was 100), matching
chunker.DefaultChunkOverlap on the backend. Both DEFAULT_CHUNKING_PRESET
and initFormData updated. The KB-load fallback also uses 80 when a
loaded KB has no chunk_overlap stored.
- All four locales (en-US, zh-CN, ko-KR, ru-RU) get rewritten chunking
setting descriptions: each now states the validated range, the default,
and the situations where you'd deviate (FAQ vs narrative, embedder
token limits, language-specific corpora).
Source code
- splitter.go: DefaultChunkSize / DefaultChunkOverlap constants get a
longer block-comment explaining the per-language token math and the
use-case sweet spots, plus the migration note on what the old
inconsistent defaults were.
- KBChunkingSettings.vue: new comment block above ChunkingConfig
documents the slider min/max for each setting, why those bounds
exist, and the recommended TokenLimit values per embedding model.
Repo docs
- New docs/CHUNKING.md: end-to-end guide covering why chunking matters,
the adaptive 3-tier architecture, per-setting reference with ranges
and sweet spots, parent-child explanation, the token-limit table per
embedder (OpenAI / Voyage / Cohere / BGE / MiniLM / Jina), 7 use-case
presets, the debug panel workflow, the API surface, and known
trade-offs (recursive strategy hidden from UI, no auto-reindex on
strategy switch, OCR limitations).
- CHANGELOG.md gets a new [Unreleased] section consolidating all the
adaptive-chunking work shipped on this branch: 5 features, 8
improvements, 6 fixes, 1 docs entry. The entry references
docs/CHUNKING.md for deeper explanation.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.
Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
not accept a context.Context, so when previewTimeout fires the
worker keeps running. Bounding input size keeps worst-case CPU
per request well under a second on commodity hardware. Sized so
10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
follow-up: real cancellation needs the splitter to take a ctx.
Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
reuse a single rune-count per chunk for stats + size + token
estimation. Eliminates the previous triple []rune allocation per
chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
instead of []chunker.Chunk; sumSq computed in float64 to avoid
the int*int overflow at l > ~46k.
Correctness / UX
- Preview panel sends strategy / token_limit / languages
unconditionally, mirroring the buildSubmitData convention so the
preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
gin's cryptic 'Field validation failed on the required tag'.
Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
made the original name a misnomer; the test still asserts the
right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
cases (empty / single / varying / no-variance underflow) plus
PreviewChunking httptest scenarios (auto path, legacy strategy,
empty-text rejection, oversize rejection, chunk truncation with
full-set stats).
Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
obsolete since the heading and heuristic splitters shipped.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by the post-Phase-3 self-review.
High severity
- Strategy / TokenLimit / Languages can now be reset. The KB-config
handler uses pointer DTOs so the absence of a field in the payload
means "no change" while an explicit empty string / 0 / [] clears
the value. The frontend now sends the fields unconditionally so
the user can revert to defaults via the clear icon — previously
these fields were write-once.
- DocProfile is no longer computed twice in the preview endpoint.
Diagnostics carries the profile that drove tier selection (when
auto-strategy ran); the handler only re-profiles for explicit
strategies that bypass the profiler.
- Chunk-size stats (avg / min / max / stddev) are now computed over
the FULL chunk set before the response is truncated to
previewMaxChunks. Avoids systematically misleading metrics for
large documents.
- SplitWithDiagnostics defaults SelectedTier to TierLegacy so an
empty-text request never leaves the diag with a blank tier string.
Medium severity
- Split() reverts to its own loop instead of delegating to
SplitWithDiagnostics. Diagnostics struct allocation is reserved
for callers that actually want the trace (preview endpoint).
Matters in SplitParentChild where Split runs once per parent.
- Preview endpoint runs the splitter on a goroutine wrapped in a
5s context.WithTimeout. CJK input at the 256k-rune ceiling
could otherwise tie up the worker for several seconds.
Low severity
- KBChunkingSettings emits arrays via spread copies so parent-side
mutation cannot drift the form's reactive state.
- KBChunkingDebug font-family includes a monospace fallback so the
chunk content stays monospaced even when the TDesign CSS variable
is missing.
Documentation
- DocProfile JSON shape is now annotated as part of the public
preview API; field renames would be breaking.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds POST /api/v1/chunker/preview — a read-only endpoint that runs the
adaptive chunker on supplied text and returns the chunks plus
diagnostic info (which tier won, which were rejected and why, the
DocProfile that the selector saw). Used by the upcoming KB editor
debug panel so users can experiment with chunking parameters without
re-uploading documents.
Backend pieces:
- chunker.SplitWithDiagnostics(text, cfg) returns chunks plus a
Diagnostics struct (SelectedTier, TierChain, Rejected). Split() now
delegates here.
- chunker.DocProfile gets JSON tags so it serializes cleanly.
- handler.PreviewChunking caps input at 256k chars and chunks at 500
per response, computes a size-distribution summary, and never
touches DB or embedding APIs.
Auth: lives under the v1 group so the standard JWT middleware applies.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Locks in two invariants exposed during the integration audit:
- ContextHeader on types.Chunk must never leak into the JSON
representation served to API clients (json:"-" + gorm:"-").
- types.ParsedChunk.EmbeddingContent prepends ContextHeader with a
blank line separator and falls back to plain Content when the
header is unset.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.
Critical
- Heading splitter no longer prepends breadcrumb to Content. The
breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
combines header + content for the embedder. This restores the
End-Start == len(Content) invariant relied on by knowledge.go's
document-reconstruction path (summary generation, UI highlight) and
also eliminates the duplicate-heading regression where the section
heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
leaves a piece > ChunkSize, the next-priority separator is applied
inside that piece. Mirrors the Python reference. Without this, a
document with one paragraph break followed by a long run of
newline-separated lines could emit a single ~1950-rune chunk for
ChunkSize=300.
Medium
- SplitParentChild forces children onto the recursive tier, skipping
the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
whole chain rejects rather than running SplitText a second time.
Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
configs (TokenLimit clamping ChunkSize tiny while Overlap stays
high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
KBs that stored ChunkOverlap=0.
ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
knowledge service can build embedding inputs as
titlePrefix + chunk.EmbeddingContent() without a DB schema change.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Reciprocal Rank Fusion previously hard-coded k=60 and weights 0.7/0.3.
Both are now sourced from RetrievalConfig (RRFK, RRFVectorWeight,
RRFKeywordWeight) with the same numbers as defaults via three new
GetEffective accessors. fuseOrDeduplicate / fuseWithRRF accept the
config; HybridSearch reads it from tenantInfo and forwards.
Lets corpus owners tune fusion behavior — large corpora benefit from
larger k, keyword-heavy domains from a higher keyword weight, etc. —
without changing application code.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
batchEmbedWithBackoff replaces the previous fixed 100ms × 5 retry loop
with proper exponential backoff (200/400/800/1600/3200 ms) and respects
context cancellation between attempts. sanitizeForEmbedding caps any
single embedding input at 20k runes — well above realistic chunk
budgets, so it only triggers on pathologically large inputs and logs
a warning when it does.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Wires the new ChunkingConfig fields (Strategy, TokenLimit, Languages)
through the documentSplitting DTO of the KB-config update endpoint and
reflects them back in the GET-config response. Existing payloads stay
fully backwards-compatible — only set the fields you want to change.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
splitByHeuristicsImpl handles documents that lack proper Markdown
headings but contain other structural cues — form-feeds, numbered
sections, multilingual chapter markers (DE/EN/ZH), all-caps section
titles, visual separators (---/===), repeated page footers, and runs
of blank lines. Boundaries are sorted by position; greedy bin-packing
fills chunks up to cfg.ChunkSize and oversize blocks between two
boundaries fall back to the legacy splitter for inner segmentation.
Boundary detection skips fenced code blocks and de-duplicates
overlapping offsets keeping the highest-priority candidate.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
splitByHeadingsImpl detects the dominant Markdown heading level via the
profiler and splits the document at headings of that level (or shallower).
Each resulting chunk gets a breadcrumb prefix like
'# Chapter 1\n## Section 1.2' so downstream embedding sees the section
context, not just the body. Sections larger than ChunkSize fall back to
the legacy splitter for inner segmentation, with the breadcrumb prepended
to every sub-chunk and ChunkSize reduced by the breadcrumb length so the
final chunk still fits the budget.
- DominantHeadingLevel: prefer the lowest level with ≥3 hits, else the
deepest level with ≥1 hit (so small docs with H1+H2 still split at H2)
- HeadingHierarchy state is updated for sub-headings within a section so
later primary sections see the right active stack
- Code-block contents are skipped during heading detection so a `# foo`
inside ``` doesn't create a fake boundary
- strategy.go uses context.Background() for the fallback log entry
(GetLogger panics on nil context)
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds HeadingHierarchy, a level-stack that tracks Markdown heading
nesting (#, ##, ###, ...). When a level-N heading is observed, all
deeper entries are popped — so Breadcrumb() always reflects the active
context path. Used by the upcoming heading-aware splitter to prepend a
context breadcrumb (e.g. '# Chapter 1\n## Section 1.2') to every chunk
inside a section, lifting embedding quality for non-trivial documents.
The existing header_tracker (table headers) uses a binary start/end hook
model that does not fit headings — a heading does not have an explicit
end, it is replaced by the next heading of equal or shallower depth.
A separate level-stack is the right shape.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).
Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
derived from TokenLimit so chunks stay safe under embedding model
token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
English (~4 chars/tok).
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds the building blocks for tiered chunking strategies — multilingual
patterns, token approximation, document profiling, validation, and a
strategy resolver that today still falls through to the legacy
SplitText. Default Strategy is empty/legacy so existing knowledge bases
continue to chunk identically.
- tokens.go: char/token approximation per language (en/de/zh/mixed)
and a coarse DetectLanguage heuristic
- patterns.go: source-of-truth regex bibliothek for headings, numbered
sections, multilingual chapter markers, visual separators, etc.
- profiler.go: single-pass document scan + SelectStrategy chain
- validator.go: rejects obviously broken chunk-sets to drive fallbacks
- strategy.go: public Split / SplitParentChild entry points; resolves
StrategyAuto via the profiler, anything else maps to a fixed tier
- splitter.go: SplitterConfig gains Strategy / TokenLimit / Languages
fields and DefaultChunkSize / DefaultChunkOverlap constants (overlap
raised to 80 ≈ 15%)
- knowledge.go: switches both call sites to chunker.Split /
SplitParentChild — behavior unchanged for empty Strategy
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.
- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
so fall-through to the tenant-merge path is intentional rather than an
unrecognised provider sliding past the switch. The legacy `cos_config` column
predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
handler/system.go StorageEngineStatusItem.Name comment now list all 7
providers (matched against types.StorageEngineConfig and ParseProviderScheme).
Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
StorageConfig data is only consumed for cos/minio and skipped for the
others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.
Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
fixing it requires API surface additions; can ship separately.
Closes#1117 (the buildStorageConfig switches; remaining items deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
Addresses two pre-merge review notes:
1. Extract the sentinel-returning threshold check from getSummary into
a small package-private helper (checkSufficientSummaryContent) and
add a unit test for it. The new test pins the behavior the caller
in ProcessSummaryGeneration relies on: the sentinel is returned for
empty / image-only / sub-threshold / scanned-PDF-without-OCR inputs,
and nil is returned for legitimate short notes and scanned PDFs
where VLM OCR succeeded.
The downstream "errors.Is(err, errInsufficientSummaryContent) →
SummaryStatusFailed + skip summary chunk" branch is mechanical (no
branching beyond the errors.Is check); covering it end-to-end would
require mocking KnowledgeRepository / ChunkRepository / chat.Chat,
for which there is no precedent in this package.
2. Switch minTextContentRunes from const to var so tests can override
the threshold and so future config plumbing can adjust it at
runtime without a rebuild. The new test exercises the override.
Drive-by comment fix: extractEntitiesAndConceptsNoUpsert still claimed
summary slugs are code-generated from the document title. Since the
previous commit they are derived from the knowledge ID instead.
Pure documentation change, no behavior impact.
Self-review of the prior commit surfaced one critical regression and
several smaller issues. This change addresses them.
Critical: the old stripImageMarkup stripped the WHOLE <image>...</image>
block, but searchutil.EnrichContentWithImageInfo wraps successful VLM OCR
output in exactly that shape — so a scanned PDF where OCR worked was
being rejected by hasSufficientTextContent and routed to the same
"failed" path as the OCR-empty case. The fix splits stripping into:
* <image_original>...</image_original> blocks removed entirely
(they only carry the redundant Markdown ref already stripped above)
* <image>, <images>, <image_caption>, <image_ocr> tags removed at
the tag level, INNER TEXT PRESERVED, so OCR / caption text reaches
the threshold check and the LLM call.
A regression test pins this down with realistic enriched-image XML that
the previous code failed silently on.
Other fixes:
* Threshold lowered from 50 to 10 runes — short legitimate documents
(brief memos, single-line notes) must still pass.
* Summary chunk content no longer prefixes "# Document\n<filename>";
only the LLM-generated summary is indexed, so filenames cannot
leak back into RAG retrieval context.
* Wiki summary slug is now derived from the knowledge ID rather than
the upload filename, eliminating filename-based slugs like
"summary/mx5280-pdf" that downstream wiki-edit prompts can read.
* Citation sourceRef likewise drops the "|<docTitle>" suffix.
* Dead docTitle parameter removed from extractCandidateSlugs,
extractEntitiesAndConceptsNoUpsert, classifyChunkCitations.
* The redundant stripImageMarkup pass in the warning log is gone;
realTextRuneCount caches the count once and uses
utf8.RuneCountInString to avoid the rune-slice allocation.
Wiki-page display titles still flow through DocTitle (filename) into
later wiki-edit / index-rebuild prompts. That is a larger refactor of
the wiki pipeline and is intentionally left for a separate PR.
Scanned PDFs without a successful OCR pass (e.g. when no VLM is configured
on the KB, or when the VLM returned a "no text content" reply that the
sanitizer discarded) reach the summary and wiki-ingest pipelines with
nothing but markdown image references for content. The downstream LLM
prompts used to receive the filename, file type, and title alongside that
empty content, and the generate_summary template explicitly forbade the
model from refusing — so the model would invent a topic from the filename
alone, producing e.g. a "Canon scanner manual" summary for an MX5280.pdf
that is actually a scanned legal letter.
Two changes shut this down:
1. Drop filename / title / file-type from every LLM prompt input. The
wiki summary, knowledge extract, candidate slug, and chunk citation
prompts now see only the document content. The knowledge summary
path no longer prepends the "Document Type: X / File Name: Y" intro.
2. Add a content-sufficiency guard in front of the LLM call. A new
helper strips markdown image references and HTML image tags and
refuses to invoke the model when fewer than 50 real-text runes
remain. getSummary returns a sentinel error in that case and the
caller marks the knowledge as SummaryStatusFailed instead of
surfacing the first chunk's bare image reference. Wiki ingest skips
LLM extraction with a clear log message.
The generate_summary.yaml prompt is also updated: the "NEVER output
refusal phrases" clause is removed and replaced with an explicit
empty-content rule that returns a deterministic placeholder line.
Real OCR (Tesseract / PaddleOCR) and filename-pattern sanitisation are
intentionally left for separate PRs.
resolveFileServiceForKnowledge was reading os.Getenv("LOCAL_STORAGE_BASE_DIR")
on every request. This causes a mismatch when the variable is not in the
process environment at request time (e.g. it was set programmatically before
app startup but not exported, or the DI-constructed fileService uses a
different configured path).
Fix: store the value in a new localBaseDir field at NewDataAnalysisTool
construction time — the same moment the DI container assembles the local
FileService — and use t.localBaseDir in resolveFileServiceForKnowledge
instead. This guarantees the data-analysis tool always resolves files
under the same base directory that the local FileService was configured
with, eliminating the "path not found" errors when LOCAL storage is used
for a knowledge base while the global storage backend is minio/oss/cos.
Fixes#1040
After ~34 failed reconnects, math.Pow(2, attempt-1) * baseDelay overflowed
int64 and produced a negative time.Duration, which bypassed the max-delay
cap. time.After(-1s) fires immediately, so a single bad credential (e.g.
WeCom bot_id/secret error 853000) turned into thousands of reconnects
per minute and massive log spam ("reconnecting in -1s (attempt 4601)").
Replace math.Pow with a bit-shift and cap the exponent at 30, which is
well above the shift needed to reach the 30s max delay. Same fix applied
to the WeChat long-poll client. Added a regression test that exercises
attempt=4601.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Kingsoft Cloud KS3 as a supported storage provider, including
backend service implementation, storage config UI, and provider
scheme recognition throughout the codebase.
AgentFinalAnswer may carry Done:true more than once (e.g. finish chunk
plus EOF). Guard AddEpisode with sync.Once so memory persists once per
turn, consistent with qa.go completion handling.
Co-authored-by: Cursor <cursoragent@cursor.com>
Reproduce:
1. Run WeKnora pointing OPENAI_BASE_URL at any OpenAI-compatible
embedding endpoint.
2. Stop that endpoint.
3. Issue any RAG-mode chat / knowledge_search query — the pipeline
reaches chunk_search_parallel → knowledgeBaseService.GetQueryEmbedding
→ OpenAIEmbedder.Embed → OpenAIEmbedder.BatchEmbed.
4. Observe the entire WeKnora process SIGSEGV instead of returning a
graceful 5xx.
Stack (real, observed in production):
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x40]
internal/models/embedding/openai.go:195 +0x458 BatchEmbed
internal/models/embedding/openai.go:93 +0xb0 Embed
internal/application/service/knowledgebase_search.go:35
GetQueryEmbedding
internal/application/service/chat_pipeline/search.go:371
searchByTargets.func1
Root cause — variable shadowing in doRequestWithRetry:
func (e *OpenAIEmbedder) doRequestWithRetry(...) (*http.Response, error) {
var resp *http.Response
var err error // outer err
...
for i := 0; i <= e.maxRetries; i++ {
...
req, err := http.NewRequestWithContext(...) // ← `:=` shadows
// outer `err`
// because `req`
// is new in
// the loop scope
if err != nil { ... continue }
...
resp, err = e.httpClient.Do(req) // writes to the
// shadowed
// loop-local err
if err == nil { return resp, nil }
logger.Errorf("attempt %d failed: %v", i+1, err) // logs the inner err
}
return nil, err // returns OUTER
// err — never set
}
When httpClient.Do returns connection errors on every attempt, the loop
logs them faithfully but the outer err stays nil. The function returns
(nil, nil). Caller in BatchEmbed reads:
resp, err := e.doRequestWithRetry(ctx, jsonData)
if err != nil { return nil, ... } // skipped (err is nil)
if resp.Body != nil { // ← line 195 · resp itself is nil
defer resp.Body.Close() // nil pointer dereference
} // → SIGSEGV → process dies
Fix — declare `req` separately so the shared `err` variable is not shadowed:
var req *http.Request
req, err = http.NewRequestWithContext(...) // outer err
After the fix, when all retries fail, doRequestWithRetry returns the real
error (e.g. `dial tcp 127.0.0.1:11434: connect: connection refused`),
the caller propagates it as a regular error, the chat pipeline returns a
5xx to the client, and the process keeps running.
Verified by stopping the embedding upstream while WeKnora was serving
queries · process stayed up · returned graceful errors · resumed normal
operation when the upstream came back. Without the patch the same
sequence killed the process within one query.
- Apply the session-list additions to the SQLite init migration (Lite build)
and make QueryPaged dialect-aware: LOWER(..) LIKE on SQLite, ILIKE on
Postgres, drop NULLS LAST on SQLite. Escape LIKE wildcards in the keyword
via the existing escapeLikeKeyword helper.
- SetPinned now returns rowsAffected so the handler can respond 404 on
unknown / unauthorized session IDs instead of a misleading 200.
- GET /sessions always returns the enriched shape (pin state + IM origin
fields) so the frontend never needs a second roundtrip; the dual-path
legacy branch in the handler is gone.
- Register pin routes with the wildcard name that matches each verb's
existing radix tree (POST :session_id, DELETE :id) and accept either
param name in the handler; avoids gin's "wildcard conflicts" panic.
- Drop the redundant [platform] prefix from IM session titles now that
the list renders a platform icon alongside the title; add unit tests
for shortID / buildUserSessionTitle / buildThreadSessionTitle.
- Frontend: remove the submenu search input and its i18n/keyword wiring
(search lives elsewhere in the app); pin icon uses the TDesign `pin`
glyph and inherits color so the active session turns green; optimistic
pin moves the item to the top of the list so it shows up at the top
of the Pinned group; IM text badge replaced by the platform SVG icons
from assets/img/im, desaturated by default and full-color on hover
or when the session is active.
Sessions today are a flat list ordered by updated_at. Two gaps showed up in
practice:
- Users cannot find specific chats as the list grows beyond a screen.
- IM-created sessions (WeCom/Feishu/Slack/...) are indistinguishable:
every title was "IM-<platform>" or "IM-<platform>-<username>" and the
list API hid the underlying im_channel_sessions mapping, so admins had
no way to tell which Feishu group a session came from.
Backend
- Migration 000039 adds sessions.user_id (owner), is_pinned, pinned_at
plus a composite index for the list query. Existing rows keep user_id
NULL and stay visible at the tenant level for backward compatibility.
- CreateSession now writes the caller's user_id from auth context.
- GET /sessions accepts keyword / source / agent_id. When any filter is
set, the response switches to enriched items that LEFT JOIN
im_channel_sessions and expose im_platform / im_chat_id / im_thread_id
/ im_user_id / im_agent_id / im_channel_id. No filters => legacy shape,
existing clients unaffected.
- Ordering: is_pinned DESC, pinned_at DESC NULLS LAST, updated_at DESC.
- POST/DELETE /sessions/:id/pin for user-scoped pin/unpin.
- IM session titles: "[platform] <user|chat|thread>" with short ID
suffixes so group/DM/thread sessions are visually distinct without
needing a round-trip to fetch a display name from the IM adapter.
Frontend
- Search input debounced at 300ms drives the keyword filter.
- Pinned chats render in a dedicated group above the time-based groups,
with a pin icon and a pin/unpin entry in the per-chat dropdown.
- IM chats get a short [platform] badge in the list.
- Pin toggle is optimistic and guards against double-clicks.
- zh/en/ko/ru i18n keys added for the new strings.