Commit Graph

826 Commits

Author SHA1 Message Date
Claude 49adf0d73c docs(chunking): align overlap default + sharpen UI/source/repo docs
Three documentation passes around the adaptive chunking work:

UI
- Frontend ChunkOverlap default consolidated to 80 (was 100), matching
  chunker.DefaultChunkOverlap on the backend. Both DEFAULT_CHUNKING_PRESET
  and initFormData updated. The KB-load fallback also uses 80 when a
  loaded KB has no chunk_overlap stored.
- All four locales (en-US, zh-CN, ko-KR, ru-RU) get rewritten chunking
  setting descriptions: each now states the validated range, the default,
  and the situations where you'd deviate (FAQ vs narrative, embedder
  token limits, language-specific corpora).

Source code
- splitter.go: DefaultChunkSize / DefaultChunkOverlap constants get a
  longer block-comment explaining the per-language token math and the
  use-case sweet spots, plus the migration note on what the old
  inconsistent defaults were.
- KBChunkingSettings.vue: new comment block above ChunkingConfig
  documents the slider min/max for each setting, why those bounds
  exist, and the recommended TokenLimit values per embedding model.

Repo docs
- New docs/CHUNKING.md: end-to-end guide covering why chunking matters,
  the adaptive 3-tier architecture, per-setting reference with ranges
  and sweet spots, parent-child explanation, the token-limit table per
  embedder (OpenAI / Voyage / Cohere / BGE / MiniLM / Jina), 7 use-case
  presets, the debug panel workflow, the API surface, and known
  trade-offs (recursive strategy hidden from UI, no auto-reindex on
  strategy switch, OCR limitations).
- CHANGELOG.md gets a new [Unreleased] section consolidating all the
  adaptive-chunking work shipped on this branch: 5 features, 8
  improvements, 6 fixes, 1 docs entry. The entry references
  docs/CHUNKING.md for deeper explanation.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 8388a50c23 fix(chunker): post-review fixes for preview endpoint robustness
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.

Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
  not accept a context.Context, so when previewTimeout fires the
  worker keeps running. Bounding input size keeps worst-case CPU
  per request well under a second on commodity hardware. Sized so
  10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
  follow-up: real cancellation needs the splitter to take a ctx.

Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
  reuse a single rune-count per chunk for stats + size + token
  estimation. Eliminates the previous triple []rune allocation per
  chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
  instead of []chunker.Chunk; sumSq computed in float64 to avoid
  the int*int overflow at l > ~46k.

Correctness / UX
- Preview panel sends strategy / token_limit / languages
  unconditionally, mirroring the buildSubmitData convention so the
  preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
  gin's cryptic 'Field validation failed on the required tag'.

Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
  TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
  made the original name a misnomer; the test still asserts the
  right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
  TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
  profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
  cases (empty / single / varying / no-variance underflow) plus
  PreviewChunking httptest scenarios (auto path, legacy strategy,
  empty-text rejection, oversize rejection, chunk truncation with
  full-set stats).

Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
  obsolete since the heading and heuristic splitters shipped.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude beb7d005e9 fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness
Resolves the issues surfaced by the post-Phase-3 self-review.

High severity
- Strategy / TokenLimit / Languages can now be reset. The KB-config
  handler uses pointer DTOs so the absence of a field in the payload
  means "no change" while an explicit empty string / 0 / [] clears
  the value. The frontend now sends the fields unconditionally so
  the user can revert to defaults via the clear icon — previously
  these fields were write-once.
- DocProfile is no longer computed twice in the preview endpoint.
  Diagnostics carries the profile that drove tier selection (when
  auto-strategy ran); the handler only re-profiles for explicit
  strategies that bypass the profiler.
- Chunk-size stats (avg / min / max / stddev) are now computed over
  the FULL chunk set before the response is truncated to
  previewMaxChunks. Avoids systematically misleading metrics for
  large documents.
- SplitWithDiagnostics defaults SelectedTier to TierLegacy so an
  empty-text request never leaves the diag with a blank tier string.

Medium severity
- Split() reverts to its own loop instead of delegating to
  SplitWithDiagnostics. Diagnostics struct allocation is reserved
  for callers that actually want the trace (preview endpoint).
  Matters in SplitParentChild where Split runs once per parent.
- Preview endpoint runs the splitter on a goroutine wrapped in a
  5s context.WithTimeout. CJK input at the 256k-rune ceiling
  could otherwise tie up the worker for several seconds.

Low severity
- KBChunkingSettings emits arrays via spread copies so parent-side
  mutation cannot drift the form's reactive state.
- KBChunkingDebug font-family includes a monospace fallback so the
  chunk content stays monospaced even when the TDesign CSS variable
  is missing.

Documentation
- DocProfile JSON shape is now annotated as part of the public
  preview API; field renames would be breaking.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude a867b1b29d feat(handler): add chunker preview endpoint for debugging
Adds POST /api/v1/chunker/preview — a read-only endpoint that runs the
adaptive chunker on supplied text and returns the chunks plus
diagnostic info (which tier won, which were rejected and why, the
DocProfile that the selector saw). Used by the upcoming KB editor
debug panel so users can experiment with chunking parameters without
re-uploading documents.

Backend pieces:
- chunker.SplitWithDiagnostics(text, cfg) returns chunks plus a
  Diagnostics struct (SelectedTier, TierChain, Rejected). Split() now
  delegates here.
- chunker.DocProfile gets JSON tags so it serializes cleanly.
- handler.PreviewChunking caps input at 256k chars and chunks at 500
  per response, computes a size-distribution summary, and never
  touches DB or embedding APIs.

Auth: lives under the v1 group so the standard JWT middleware applies.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 2feab191bb test(types): regression test that ContextHeader is hidden from JSON
Locks in two invariants exposed during the integration audit:
- ContextHeader on types.Chunk must never leak into the JSON
  representation served to API clients (json:"-" + gorm:"-").
- types.ParsedChunk.EmbeddingContent prepends ContextHeader with a
  blank line separator and falls back to plain Content when the
  header is unset.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude bc44c31cec fix(chunker): QA audit — position invariant, recursive split, perf
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.

Critical
- Heading splitter no longer prepends breadcrumb to Content. The
  breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
  combines header + content for the embedder. This restores the
  End-Start == len(Content) invariant relied on by knowledge.go's
  document-reconstruction path (summary generation, UI highlight) and
  also eliminates the duplicate-heading regression where the section
  heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
  leaves a piece > ChunkSize, the next-priority separator is applied
  inside that piece. Mirrors the Python reference. Without this, a
  document with one paragraph break followed by a long run of
  newline-separated lines could emit a single ~1950-rune chunk for
  ChunkSize=300.

Medium
- SplitParentChild forces children onto the recursive tier, skipping
  the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
  boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
  whole chain rejects rather than running SplitText a second time.

Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
  configs (TokenLimit clamping ChunkSize tiny while Overlap stays
  high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
  KBs that stored ChunkOverlap=0.

ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
  EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
  knowledge service can build embedding inputs as
  titlePrefix + chunk.EmbeddingContent() without a DB schema change.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 2850ac78ce feat(retrieval): configurable RRF parameters per tenant
Reciprocal Rank Fusion previously hard-coded k=60 and weights 0.7/0.3.
Both are now sourced from RetrievalConfig (RRFK, RRFVectorWeight,
RRFKeywordWeight) with the same numbers as defaults via three new
GetEffective accessors. fuseOrDeduplicate / fuseWithRRF accept the
config; HybridSearch reads it from tenantInfo and forwards.

Lets corpus owners tune fusion behavior — large corpora benefit from
larger k, keyword-heavy domains from a higher keyword weight, etc. —
without changing application code.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 0dc32cb0c4 feat(retriever): exponential backoff + safety truncation for embeddings
batchEmbedWithBackoff replaces the previous fixed 100ms × 5 retry loop
with proper exponential backoff (200/400/800/1600/3200 ms) and respects
context cancellation between attempts. sanitizeForEmbedding caps any
single embedding input at 20k runes — well above realistic chunk
budgets, so it only triggers on pathologically large inputs and logs
a warning when it does.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 82b0e504b0 feat(handler): expose chunking strategy / token-limit / languages in API
Wires the new ChunkingConfig fields (Strategy, TokenLimit, Languages)
through the documentSplitting DTO of the KB-config update endpoint and
reflects them back in the GET-config response. Existing payloads stay
fully backwards-compatible — only set the fields you want to change.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude d31acf1063 feat(chunker): heuristic Tier 2 splitter
splitByHeuristicsImpl handles documents that lack proper Markdown
headings but contain other structural cues — form-feeds, numbered
sections, multilingual chapter markers (DE/EN/ZH), all-caps section
titles, visual separators (---/===), repeated page footers, and runs
of blank lines. Boundaries are sorted by position; greedy bin-packing
fills chunks up to cfg.ChunkSize and oversize blocks between two
boundaries fall back to the legacy splitter for inner segmentation.

Boundary detection skips fenced code blocks and de-duplicates
overlapping offsets keeping the highest-priority candidate.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude cf8789f84c feat(chunker): heading-aware Tier 1 splitter
splitByHeadingsImpl detects the dominant Markdown heading level via the
profiler and splits the document at headings of that level (or shallower).
Each resulting chunk gets a breadcrumb prefix like
'# Chapter 1\n## Section 1.2' so downstream embedding sees the section
context, not just the body. Sections larger than ChunkSize fall back to
the legacy splitter for inner segmentation, with the breadcrumb prepended
to every sub-chunk and ChunkSize reduced by the breadcrumb length so the
final chunk still fits the budget.

- DominantHeadingLevel: prefer the lowest level with ≥3 hits, else the
  deepest level with ≥1 hit (so small docs with H1+H2 still split at H2)
- HeadingHierarchy state is updated for sub-headings within a section so
  later primary sections see the right active stack
- Code-block contents are skipped during heading detection so a `# foo`
  inside ``` doesn't create a fake boundary
- strategy.go uses context.Background() for the fallback log entry
  (GetLogger panics on nil context)

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 592d3618dc feat(chunker): heading hierarchy tracker
Adds HeadingHierarchy, a level-stack that tracks Markdown heading
nesting (#, ##, ###, ...). When a level-N heading is observed, all
deeper entries are popped — so Breadcrumb() always reflects the active
context path. Used by the upcoming heading-aware splitter to prepend a
context breadcrumb (e.g. '# Chapter 1\n## Section 1.2') to every chunk
inside a section, lifting embedding quality for non-trivial documents.

The existing header_tracker (table headers) uses a binary start/end hook
model that does not fit headings — a heading does not have an explicit
end, it is replaced by the next heading of equal or shallower depth.
A separate level-stack is the right shape.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude 1585a6f12c feat(chunker): recursive priority splitting + token-aware budget
The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).

Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
  per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
  chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
  consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
  derived from TokenLimit so chunks stay safe under embedding model
  token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
  English (~4 chars/tok).

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
Claude ae6fde2310 feat(chunker): foundation for adaptive 3-tier chunking
Adds the building blocks for tiered chunking strategies — multilingual
patterns, token approximation, document profiling, validation, and a
strategy resolver that today still falls through to the legacy
SplitText. Default Strategy is empty/legacy so existing knowledge bases
continue to chunk identically.

- tokens.go: char/token approximation per language (en/de/zh/mixed)
  and a coarse DetectLanguage heuristic
- patterns.go: source-of-truth regex bibliothek for headings, numbered
  sections, multilingual chapter markers, visual separators, etc.
- profiler.go: single-pass document scan + SelectStrategy chain
- validator.go: rejects obviously broken chunk-sets to drive fallbacks
- strategy.go: public Split / SplitParentChild entry points; resolves
  StrategyAuto via the profiler, anything else maps to a fixed tier
- splitter.go: SplitterConfig gains Strategy / TokenLimit / Languages
  fields and DefaultChunkSize / DefaultChunkOverlap constants (overlap
  raised to 80 ≈ 15%)
- knowledge.go: switches both call sites to chunker.Split /
  SplitParentChild — behavior unchanged for empty Strategy

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
voidborne-d 46cab52fb0 fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117)
knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.

- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
  StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
  Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
  SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
  so fall-through to the tenant-merge path is intentional rather than an
  unrecognised provider sliding past the switch. The legacy `cos_config` column
  predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
  handler/system.go StorageEngineStatusItem.Name comment now list all 7
  providers (matched against types.StorageEngineConfig and ParseProviderScheme).

Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
  population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
  regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
  StorageConfig data is only consumed for cos/minio and skipped for the
  others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.

Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
  a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
  the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
  fixing it requires API surface additions; can ship separately.

Closes #1117 (the buildStorageConfig switches; remaining items deferred).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 17:09:42 +08:00
lyingbug 0d8540c93c Revert "refactor(knowledge): split 9.8k-line service file by responsibility"
This reverts commit f67d0b53ce.
2026-05-06 17:09:24 +08:00
wizardchen f67d0b53ce refactor(knowledge): split 9.8k-line service file by responsibility
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.

Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.

  knowledge.go              497  struct + constructor + errs + CRUD reads + Search
  knowledge_create.go      1144  CreateFrom{File,URL,Passage,Manual} + helpers
  knowledge_delete.go       686  Delete + wiki cleanup + ProcessKnowledgeListDelete
  knowledge_process.go     2407  ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
  knowledge_clone_move.go  1049  Clone* + Move* + progress persistence
  knowledge_faq.go         1932  FAQ CRUD + Search + Export
  knowledge_faq_import.go  1939  FAQ import/validate/index + runningFAQImportInfo
  knowledge_util.go         352  file-type/url/hash + VLM+Storage config + resolveFileService

Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
2026-05-06 17:06:05 +08:00
sqkstwj 0e19c978ae fix(files): fall back to global file service when tenant storage config is unavailable 2026-05-06 17:04:38 +08:00
Claude c3e53a3fc4 test(ingest): cover insufficient-content gate; allow threshold override
Addresses two pre-merge review notes:

1. Extract the sentinel-returning threshold check from getSummary into
   a small package-private helper (checkSufficientSummaryContent) and
   add a unit test for it. The new test pins the behavior the caller
   in ProcessSummaryGeneration relies on: the sentinel is returned for
   empty / image-only / sub-threshold / scanned-PDF-without-OCR inputs,
   and nil is returned for legitimate short notes and scanned PDFs
   where VLM OCR succeeded.

   The downstream "errors.Is(err, errInsufficientSummaryContent) →
   SummaryStatusFailed + skip summary chunk" branch is mechanical (no
   branching beyond the errors.Is check); covering it end-to-end would
   require mocking KnowledgeRepository / ChunkRepository / chat.Chat,
   for which there is no precedent in this package.

2. Switch minTextContentRunes from const to var so tests can override
   the threshold and so future config plumbing can adjust it at
   runtime without a rebuild. The new test exercises the override.
2026-05-06 11:23:32 +08:00
Claude 7ca4b2f53c chore(comment): update slug source to knowledge ID
Drive-by comment fix: extractEntitiesAndConceptsNoUpsert still claimed
summary slugs are code-generated from the document title. Since the
previous commit they are derived from the knowledge ID instead.
Pure documentation change, no behavior impact.
2026-05-06 11:23:32 +08:00
Claude 6744e1cdd5 fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks
Self-review of the prior commit surfaced one critical regression and
several smaller issues. This change addresses them.

Critical: the old stripImageMarkup stripped the WHOLE <image>...</image>
block, but searchutil.EnrichContentWithImageInfo wraps successful VLM OCR
output in exactly that shape — so a scanned PDF where OCR worked was
being rejected by hasSufficientTextContent and routed to the same
"failed" path as the OCR-empty case. The fix splits stripping into:
  * <image_original>...</image_original> blocks removed entirely
    (they only carry the redundant Markdown ref already stripped above)
  * <image>, <images>, <image_caption>, <image_ocr> tags removed at
    the tag level, INNER TEXT PRESERVED, so OCR / caption text reaches
    the threshold check and the LLM call.

A regression test pins this down with realistic enriched-image XML that
the previous code failed silently on.

Other fixes:
  * Threshold lowered from 50 to 10 runes — short legitimate documents
    (brief memos, single-line notes) must still pass.
  * Summary chunk content no longer prefixes "# Document\n<filename>";
    only the LLM-generated summary is indexed, so filenames cannot
    leak back into RAG retrieval context.
  * Wiki summary slug is now derived from the knowledge ID rather than
    the upload filename, eliminating filename-based slugs like
    "summary/mx5280-pdf" that downstream wiki-edit prompts can read.
  * Citation sourceRef likewise drops the "|<docTitle>" suffix.
  * Dead docTitle parameter removed from extractCandidateSlugs,
    extractEntitiesAndConceptsNoUpsert, classifyChunkCitations.
  * The redundant stripImageMarkup pass in the warning log is gone;
    realTextRuneCount caches the count once and uses
    utf8.RuneCountInString to avoid the rune-slice allocation.

Wiki-page display titles still flow through DocTitle (filename) into
later wiki-edit / index-rebuild prompts. That is a larger refactor of
the wiki pipeline and is intentionally left for a separate PR.
2026-05-06 11:23:32 +08:00
Claude 15550d14d4 fix(ingest): stop hallucinating summaries from filenames on empty PDFs
Scanned PDFs without a successful OCR pass (e.g. when no VLM is configured
on the KB, or when the VLM returned a "no text content" reply that the
sanitizer discarded) reach the summary and wiki-ingest pipelines with
nothing but markdown image references for content. The downstream LLM
prompts used to receive the filename, file type, and title alongside that
empty content, and the generate_summary template explicitly forbade the
model from refusing — so the model would invent a topic from the filename
alone, producing e.g. a "Canon scanner manual" summary for an MX5280.pdf
that is actually a scanned legal letter.

Two changes shut this down:

1. Drop filename / title / file-type from every LLM prompt input. The
   wiki summary, knowledge extract, candidate slug, and chunk citation
   prompts now see only the document content. The knowledge summary
   path no longer prepends the "Document Type: X / File Name: Y" intro.

2. Add a content-sufficiency guard in front of the LLM call. A new
   helper strips markdown image references and HTML image tags and
   refuses to invoke the model when fewer than 50 real-text runes
   remain. getSummary returns a sentinel error in that case and the
   caller marks the knowledge as SummaryStatusFailed instead of
   surfacing the first chunk's bare image reference. Wiki ingest skips
   LLM extraction with a clear log message.

The generate_summary.yaml prompt is also updated: the "NEVER output
refusal phrases" clause is removed and replaced with an explicit
empty-content rule that returns a deterministic placeholder line.

Real OCR (Tesseract / PaddleOCR) and filename-pattern sanitisation are
intentionally left for separate PRs.
2026-05-06 11:23:32 +08:00
guangyang1206 d3d433574f fix(data-analysis): capture LOCAL_STORAGE_BASE_DIR at init time (#1040)
resolveFileServiceForKnowledge was reading os.Getenv("LOCAL_STORAGE_BASE_DIR")
on every request.  This causes a mismatch when the variable is not in the
process environment at request time (e.g. it was set programmatically before
app startup but not exported, or the DI-constructed fileService uses a
different configured path).

Fix: store the value in a new localBaseDir field at NewDataAnalysisTool
construction time — the same moment the DI container assembles the local
FileService — and use t.localBaseDir in resolveFileServiceForKnowledge
instead.  This guarantees the data-analysis tool always resolves files
under the same base directory that the local FileService was configured
with, eliminating the "path not found" errors when LOCAL storage is used
for a knowledge base while the global storage backend is minio/oss/cos.

Fixes #1040
2026-05-06 11:19:56 +08:00
c e938f1c6ae test(agent): expand graph query regression fixture 2026-05-06 11:04:55 +08:00
c b2ff29cc0f fix(agent): correct graph config reporting 2026-05-06 11:04:55 +08:00
c f08192ba26 fix: include shared knowledge in knowledge search 2026-05-05 01:04:49 +08:00
wizardchen 6c1442aaf9 fix(im): clamp reconnect backoff to avoid int64 overflow → busy loop
After ~34 failed reconnects, math.Pow(2, attempt-1) * baseDelay overflowed
int64 and produced a negative time.Duration, which bypassed the max-delay
cap. time.After(-1s) fires immediately, so a single bad credential (e.g.
WeCom bot_id/secret error 853000) turned into thousands of reconnects
per minute and massive log spam ("reconnecting in -1s (attempt 4601)").

Replace math.Pow with a bit-shift and cap the exponent at 30, which is
well above the shift needed to reach the 30s max delay. Same fix applied
to the WeChat long-poll client. Added a regression test that exercises
attempt=4601.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 00:36:18 +08:00
ChenRussell 42348eb40d feat(ks3): add KS3 object storage support
Add Kingsoft Cloud KS3 as a supported storage provider, including
  backend service implementation, storage config UI, and provider
  scheme recognition throughout the codebase.
2026-05-05 00:22:27 +08:00
IFeynman 19cd17c0a6 fix(wiki): apply reviewer's detailed resolution for contradiction handling 2026-05-04 23:52:38 +08:00
IFeynman cf8e62fa9e fix(wiki): address PR review feedbacks regarding citation format, rule conflicts, and search loops 2026-05-04 23:52:38 +08:00
IFeynman b313da20ae feat(wiki): enforce strict citation tracing and ontology reuse in Wiki prompts 2026-05-04 23:52:38 +08:00
Andy Yang 3d94a476bf fix(memory): dedupe episode write on duplicate streaming Done
AgentFinalAnswer may carry Done:true more than once (e.g. finish chunk
plus EOF). Guard AddEpisode with sync.Once so memory persists once per
turn, consistent with qa.go completion handling.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-04 23:49:29 +08:00
c 48aeafcc3f fix(chat): include history in model fallback 2026-05-04 23:24:21 +08:00
toy0116 8955bf1aa8 fix(embedding): OpenAIEmbedder.doRequestWithRetry shadows err · returns (nil, nil) on connection failure · caller nil-derefs and SIGSEGVs
Reproduce:

  1. Run WeKnora pointing OPENAI_BASE_URL at any OpenAI-compatible
     embedding endpoint.
  2. Stop that endpoint.
  3. Issue any RAG-mode chat / knowledge_search query — the pipeline
     reaches chunk_search_parallel → knowledgeBaseService.GetQueryEmbedding
     → OpenAIEmbedder.Embed → OpenAIEmbedder.BatchEmbed.
  4. Observe the entire WeKnora process SIGSEGV instead of returning a
     graceful 5xx.

Stack (real, observed in production):

  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x2 addr=0x40]

  internal/models/embedding/openai.go:195 +0x458   BatchEmbed
  internal/models/embedding/openai.go:93  +0xb0    Embed
  internal/application/service/knowledgebase_search.go:35
                                                   GetQueryEmbedding
  internal/application/service/chat_pipeline/search.go:371
                                                   searchByTargets.func1

Root cause — variable shadowing in doRequestWithRetry:

  func (e *OpenAIEmbedder) doRequestWithRetry(...) (*http.Response, error) {
      var resp *http.Response
      var err error                                        // outer err
      ...
      for i := 0; i <= e.maxRetries; i++ {
          ...
          req, err := http.NewRequestWithContext(...)      // ← `:=` shadows
                                                           //   outer `err`
                                                           //   because `req`
                                                           //   is new in
                                                           //   the loop scope
          if err != nil { ... continue }
          ...
          resp, err = e.httpClient.Do(req)                 // writes to the
                                                           // shadowed
                                                           // loop-local err
          if err == nil { return resp, nil }
          logger.Errorf("attempt %d failed: %v", i+1, err) // logs the inner err
      }
      return nil, err                                      // returns OUTER
                                                           // err — never set
  }

When httpClient.Do returns connection errors on every attempt, the loop
logs them faithfully but the outer err stays nil. The function returns
(nil, nil). Caller in BatchEmbed reads:

  resp, err := e.doRequestWithRetry(ctx, jsonData)
  if err != nil { return nil, ... }       // skipped (err is nil)
  if resp.Body != nil {                   // ← line 195 · resp itself is nil
      defer resp.Body.Close()             //   nil pointer dereference
  }                                       //   → SIGSEGV → process dies

Fix — declare `req` separately so the shared `err` variable is not shadowed:

  var req *http.Request
  req, err = http.NewRequestWithContext(...)               // outer err

After the fix, when all retries fail, doRequestWithRetry returns the real
error (e.g. `dial tcp 127.0.0.1:11434: connect: connection refused`),
the caller propagates it as a regular error, the chat pipeline returns a
5xx to the client, and the process keeps running.

Verified by stopping the embedding upstream while WeKnora was serving
queries · process stayed up · returned graceful errors · resumed normal
operation when the upstream came back. Without the patch the same
sequence killed the process within one query.
2026-05-04 23:18:42 +08:00
wizardchen fe0d24ae87 fix(sessions): review fixes for keyword search / pinning / IM titles
- Apply the session-list additions to the SQLite init migration (Lite build)
  and make QueryPaged dialect-aware: LOWER(..) LIKE on SQLite, ILIKE on
  Postgres, drop NULLS LAST on SQLite. Escape LIKE wildcards in the keyword
  via the existing escapeLikeKeyword helper.
- SetPinned now returns rowsAffected so the handler can respond 404 on
  unknown / unauthorized session IDs instead of a misleading 200.
- GET /sessions always returns the enriched shape (pin state + IM origin
  fields) so the frontend never needs a second roundtrip; the dual-path
  legacy branch in the handler is gone.
- Register pin routes with the wildcard name that matches each verb's
  existing radix tree (POST :session_id, DELETE :id) and accept either
  param name in the handler; avoids gin's "wildcard conflicts" panic.
- Drop the redundant [platform] prefix from IM session titles now that
  the list renders a platform icon alongside the title; add unit tests
  for shortID / buildUserSessionTitle / buildThreadSessionTitle.
- Frontend: remove the submenu search input and its i18n/keyword wiring
  (search lives elsewhere in the app); pin icon uses the TDesign `pin`
  glyph and inherits color so the active session turns green; optimistic
  pin moves the item to the top of the list so it shows up at the top
  of the Pinned group; IM text badge replaced by the platform SVG icons
  from assets/img/im, desaturated by default and full-color on hover
  or when the session is active.
2026-04-30 16:23:00 +08:00
wizardchen dbd804d6e3 feat(sessions): add keyword search, user-scoped pinning, and IM source visibility
Sessions today are a flat list ordered by updated_at. Two gaps showed up in
practice:
- Users cannot find specific chats as the list grows beyond a screen.
- IM-created sessions (WeCom/Feishu/Slack/...) are indistinguishable:
  every title was "IM-<platform>" or "IM-<platform>-<username>" and the
  list API hid the underlying im_channel_sessions mapping, so admins had
  no way to tell which Feishu group a session came from.

Backend
- Migration 000039 adds sessions.user_id (owner), is_pinned, pinned_at
  plus a composite index for the list query. Existing rows keep user_id
  NULL and stay visible at the tenant level for backward compatibility.
- CreateSession now writes the caller's user_id from auth context.
- GET /sessions accepts keyword / source / agent_id. When any filter is
  set, the response switches to enriched items that LEFT JOIN
  im_channel_sessions and expose im_platform / im_chat_id / im_thread_id
  / im_user_id / im_agent_id / im_channel_id. No filters => legacy shape,
  existing clients unaffected.
- Ordering: is_pinned DESC, pinned_at DESC NULLS LAST, updated_at DESC.
- POST/DELETE /sessions/:id/pin for user-scoped pin/unpin.
- IM session titles: "[platform] <user|chat|thread>" with short ID
  suffixes so group/DM/thread sessions are visually distinct without
  needing a round-trip to fetch a display name from the IM adapter.

Frontend
- Search input debounced at 300ms drives the keyword filter.
- Pinned chats render in a dedicated group above the time-based groups,
  with a pin icon and a pin/unpin entry in the per-chat dropdown.
- IM chats get a short [platform] badge in the list.
- Pin toggle is optimistic and guards against double-clicks.
- zh/en/ko/ru i18n keys added for the new strings.
2026-04-30 16:23:00 +08:00
wizardchen e029d31f86 feat(im): tenant-wide IM channels overview under the user menu
Until now the IM channels a tenant has connected were only visible from
inside each agent's editor (AgentEditorModal → IMChannelPanel). Finding
out which bots are live across all agents meant clicking through every
one of them. Add a tenant-scoped overview so the set of connected IMs is
always one hop away from the avatar.

Backend

- im.Service.ListChannelsByTenant(tenantID) returns all non-deleted
  channels in the tenant, LEFT JOIN'd with custom_agents.name so
  built-in agents (which don't have rows in custom_agents) still show
  up with an empty agent_name; the frontend substitutes a localized
  "built-in agent" label.
- Credentials are intentionally stripped from this response — the list
  view is read-only and the editor route (GET /agents/:id/im-channels)
  remains the only source for secrets.
- New ChannelWithAgent DTO + IMHandler.ListAllIMChannels wired at
  GET /api/v1/im-channels under the existing RegisterIMChannelRoutes
  group, so auth middleware is inherited.

Frontend

- IMChannelsOverviewPanel.vue: floating submenu pane, stacked rows
  (agent on top, channel below) with identical 20px avatars and 12px
  labels so neither identity dominates. Each row is the whole click
  target (jumps to the agent editor); the switch uses @click.stop to
  toggle in place without triggering navigation.
- UserMenu.vue: new hover-driven submenu entry ("已接入的 IM" + link
  icon). The pane is teleported to <body> because the sidebar container
  has overflow:hidden that would otherwise clip right-flying content.
  Position is computed from the menu item's rect and clamps/flips
  against the viewport edges.
- "Live" indicator: a pulsing green dot on the menu item when at least
  one channel is enabled. UserMenu prefetches the list once on mount;
  the panel re-emits channels-changed after every load/toggle so the
  dot stays in sync. Respects prefers-reduced-motion.
- AgentList.vue: supplement the existing onMounted-only
  checkAndOpenEditModal with a watch(route.query.edit) so
  router.push({ path: '/platform/agents', query: { edit, section } })
  from an already-mounted AgentList (the common case when navigating
  from the overview) opens the editor immediately instead of only
  after a hard refresh.

Assets

- Seven platform SVGs under assets/img/im/, pulled from iconify
  (simple-icons / logos / tdesign / icon-park / remix icon) and baked
  with brand colors so monochrome sources (wecom/wechat/dingtalk/
  feishu) don't render black.

i18n

- New imOverview namespace in zh-CN / en-US / ko-KR / ru-RU covering
  the menu label, panel title/subtitle, column headers, builtin-agent
  fallback, and the live-indicator tooltip.

Tested

- go build ./...
- go test ./internal/im/...
- vue-tsc --noEmit
- npm run build
2026-04-30 15:29:14 +08:00
sqkstwj a5f80b747c fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
wizardchen 3b23713c54 fix(crypto): fail loudly when encrypted DB fields cannot be decrypted
GORM Scan / AfterFind hooks for every AES-encrypted column followed
the same lenient pattern:

    if decrypted, err := utils.DecryptAESGCM(c.APIKey, key); err == nil {
        c.APIKey = decrypted
    }

When SYSTEM_AES_KEY was missing, rotated, or the wrong length the
decryption error was swallowed and the in-memory struct kept the raw
"enc:v1:..." ciphertext. The application then happily forwarded the
ciphertext upstream as the actual API key / password, surfacing as
401/403/SignatureDoesNotMatch from third-party vendors. Worse, a
ciphertext snippet of a customer credential was leaking into the
external provider's request logs.

Introduce utils.DecryptStoredSecret that:
  - returns "" / legacy plaintext untouched (no false positives for
    pre-encryption rows);
  - returns ErrEncryptedDataMissingKey when the value carries the
    enc:v1: prefix but no AES key is configured;
  - propagates any GCM auth-tag failure from a rotated key.

Wire it into the five Scan / AfterFind sites that currently swallow
the error:
  - Tenant.AfterFind (tenants.api_key)
  - CredentialsConfig.Scan (tenants.we_knora_cloud.app_secret)
  - ModelParameters.Scan (models.parameters.api_key, .app_secret)
  - ConnectionConfig.Scan (vector_store_connections.password, .api_key)
  - WebSearchProviderParameters.Scan (web_search_providers.api_key)

The error is wrapped with the originating column so the failure is
diagnosable in the logs. Operators must restore the previous
SYSTEM_AES_KEY (or rotate the affected secrets) instead of receiving
silent vendor 401s.

Add table-driven tests covering empty input, legacy plaintext,
round-trip, missing key, wrong-length key, and rotated key. The
strict path explicitly asserts that ciphertext does NOT leak in the
returned plaintext on any error path.
2026-04-30 15:17:56 +08:00
wizardchen 097c9d0ad5 fix(im): revert tenant-from-context for presigned URL
Reverts the tenant-ID-from-context change in da5fcd3. The storage path
encodes the resource owner's tenant, and the presigned-URL verifier
uses that ID to look up the owning tenant's StorageEngineConfig. Using
the caller's tenant ID from context would break cross-tenant shared
resources — e.g. when tenant Y reads an image from a KB shared by
tenant X, signing with Y would cause the verifier to open Y's storage
backend and 404 on X's file.

Keep:
- presignDefaultTTL shortened from 24h to 2h (independent improvement).
- godoc note on ParseTenantIDFromStoragePath flagging the ambiguity
  for cloud paths with numeric bucket/region names.
- Unit tests for path-based tenant extraction and the no-external-URL
  backward-compat path.
2026-04-30 11:40:30 +08:00
wizardchen f27a1e083c fix(im): prefer tenant ID from context, shorten presigned URL TTL
Addresses review feedback on the IM storage URL rewrite:

- localFileService.GetFileURL now reads tenant ID from request context
  first, falling back to ParseTenantIDFromStoragePath only when context
  is absent. Fixes ambiguity for cloud providers whose paths embed
  numeric bucket/region names before the tenant segment, which could
  mint presigned URLs bound to the wrong tenant ID.
- Shorten presigned URL default TTL from 24h to 2h. A leaked HMAC key
  authorizes cross-tenant file reads, so URLs should expire quickly;
  IM clients fetch referenced images within seconds anyway.
- Document ParseTenantIDFromStoragePath as a best-effort fallback.
- Add unit tests covering context-first, path-fallback, and the
  no-external-URL backward-compat path.
2026-04-30 11:40:30 +08:00
wizardchen 7fd566bc15 fix(im): rewrite private storage URLs to HTTP in IM channel replies
IM platforms (Feishu, Slack, Telegram, DingTalk, Mattermost, WeCom) cannot
render provider:// URLs (local://, minio://, s3://, etc.) that appear in
LLM answers containing knowledge base images. The web frontend handles
these via the authenticated /files endpoint, but IM clients need publicly
resolvable HTTP URLs.

Changes:
- Add HMAC-SHA256 presigned URL utility (internal/utils/presign.go) for
  generating time-limited, signature-verified file access URLs
- Add GET /api/v1/files/presigned endpoint that serves files without
  session auth, verified by HMAC signature and expiry
- Update localFileService.GetFileURL() to return presigned HTTP URLs
  when APP_EXTERNAL_URL is configured (cloud backends already return
  presigned URLs via their SDKs)
- Add IM content rewriting pipeline: strip <image> XML tags, strip
  citation tags, rewrite storage URLs to HTTP — applied at all IM
  output points (streaming flush, non-streaming reply, fallback)
- Add holdback buffer in streaming flush to prevent URL/tag splitting
  across chunk boundaries

Closes #1058
2026-04-30 11:40:30 +08:00
Mao Meng b48adde3e6 fix:attachment_document_failed 2026-04-29 22:34:13 +08:00
wizardchen b2975f7d1d refactor(im): move adapter factories into per-platform subpackages
registerIMAdapterFactories in internal/container/container.go had grown
to ~290 lines of platform-specific config parsing and runtime start/stop
logic across 7 IM platforms, plus three private credential helpers. The
container package is meant to wire dependencies, not own each platform's
credentials schema and websocket lifecycle.

Extract each platform's factory into its own subpackage file and expose
the credential helpers on the im package:

- New im.ParseCredentials / im.GetString / im.GetBool (formerly the
  unexported parseCredentials / getString / credentialBool in container).
- New im.ResolveMode helper collapses the six identical "default to
  websocket when Mode is empty" blocks.
- New {wecom,feishu,slack,telegram,dingtalk,mattermost,wechat}.NewFactory
  returns an im.AdapterFactory; container.go now calls each one in a
  single line.

Behavior is preserved: adapter construction, goroutine start/stop,
log messages, default modes (mattermost still defaults to "webhook";
others to "websocket"), and error strings are unchanged. The WeCom
corp_agent_id float64/int switch and the WeChat long-poll-only path
are preserved as-is.

Net: container.go drops ~320 lines and no longer needs to be touched
when adding or tuning an IM platform.
2026-04-29 20:08:22 +08:00
wizardchen 7fcd92994b feat(tenant): expose API Key reset from the API Info page
Add a danger-themed reset button next to the existing copy/eye buttons
on the API Info settings page. Confirmation dialog warns that the old
key is revoked immediately; on success the new plaintext key replaces
the displayed value and is auto-revealed so the user can copy it.

Backend: new POST /api/v1/tenants/:id/api-key handler that wraps the
existing TenantService.UpdateAPIKey; access is gated by the same
authorizeTenantAccess check as other tenant endpoints. The handler
returns the freshly generated plaintext key, while the encrypted form
is persisted to the database.

Frontend: new resetTenantApiKey API client, reset button + confirm
dialog wiring in ApiInfo.vue, plus matching i18n entries for zh-CN,
en-US, ko-KR, and ru-RU.
2026-04-29 20:07:38 +08:00
lyingbug cbdfba0075 Merge pull request #1086 from Windfarer/storage-allow-list
feat: add STORAGE_ALLOW_LIST env var
2026-04-29 19:49:46 +08:00
wizardchen c89fd5da98 fix(tenant): return plaintext API key after create/reset
CreateTenant and UpdateAPIKey both overwrote tenant.APIKey with the
AES-encrypted ciphertext before returning, causing the UI to display
"enc:v1:..." right after generating or resetting an API key. Save the
plaintext in a local variable, encrypt only the value handed to
UpdateTenant, and return the plaintext to callers.

Subsequent reads via GetTenantByID continue to rely on the AfterFind
hook for transparent decryption, so this only affects the initial
write-back path.
2026-04-29 19:46:21 +08:00
Windfarer 20be935fe0 Merge branch 'main' into storage-allow-list 2026-04-29 19:35:13 +08:00
Windfarer 6601a483be feat: add STORAGE_ALLOW_LIST env var 2026-04-29 18:58:09 +08:00
wizardchen da90867dd4 feat(ui): refactor IM channels list layout and form\n\n- Change IM channel form from dialog to drawer for better UX\n- Convert channel list from vertical layout to responsive grid cards\n- Move edit/delete actions into a dropdown menu for cleaner card design\n- Replace platform radio group with a select dropdown in the add channel form 2026-04-29 17:34:20 +08:00