The Agent ran a parallel context cache (Redis/in-memory) on top of the
messages table to feed multi-turn history into the LLM. That dual-write
caused subtle drift (e.g. compression diverging from DB, system-prompt
swaps lost on restart) and required a separate ClearContext path on
session/IM clear.
Make the messages table the single source of truth:
- Add service.LoadAgentHistory: rebuilds chat.Message history per turn
from the persisted messages, expanding AgentSteps into proper OpenAI
assistant_with_tool_calls + tool messages and replaying the canonical
final answer (with <think> blocks stripped). final_answer tool calls
are filtered to avoid duplicating the trailing answer.
- Make AgentEngine stateless across turns: drop ContextManager / sessionID
cache plumbing from the engine, agent service, and CreateAgentEngine
signature. The engine only uses sessionID for logging/event emission.
- Wire AgentQA to load history from DB on demand using HistoryTurns
(default 5) when MultiTurnEnabled, otherwise run with empty history.
- Delete the llmcontext package (ContextManager interface, Redis/memory
storage, factory) and the SessionService.ClearContext API path; IM
/clear and session message clear no longer need to invalidate cache.
Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with
empty AgentSteps just produce the single canonical assistant message),
and Agent-mode turns now consistently see prior tool calls and results.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
items" had two args), and harden validMerge against un-prefixed
slugs whose strings.Index returned -1 and silently passed the type
check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
letter row instead of a hard-coded 0; tighten payloadProbe to the
set of field names with consistent semantics across payloads
(drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
outside an asynq worker ctx so the semantics stay pinned.
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
When a doc's entity/concept page generation hit a transient LLM error in
the reduce phase, the page never got written, but the doc's summary page
(generated in parallel with the slug list baked into its content) was
already persisted with [[entity/foo|name]] links pointing at the missing
page. The wiki log feed also surfaced the failed slugs as clickable
entries that 404'd. cleanDeadLinks only ran for retract batches, so the
debris persisted indefinitely.
reduceSlugUpdates now reports addition-path failures back to the batch
driver, which collects them and:
- rewrites this batch's summary pages, replacing dead [[slug|display]]
refs with plain display text (or a humanized slug tail);
- filters dead slugs out of docResults[].Pages before the wiki log
entries are flushed;
- broadens the cleanDeadLinks trigger to fire whenever any page was
touched or any slug failed, as a long-tail safety net.
Pure text replacement; no extra LLM calls. Failed slugs are picked up
naturally on the next ingest of the same document via the existing slug-
continuity rules, so the system self-heals without manual reingest.
Avoid treating Redis pending-length read failures as empty queues, reset stale fail-count keys on fresh ingest/drop paths, and reduce wiki ingest MaxRetry to a moderate shared constant to limit queue churn.
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: requeueFailedOps unconditionally RPush'd every failed op back to
wiki:pending, causing unbounded growth when a document consistently triggers
LLM timeouts. Observed: 553 entries in wiki:pending:08134644 (one KB), mostly
duplicates for ~5 unique documents that each timed out on every batch cycle.
Fix:
- Add wikiFailCountKeyPrefix ("wiki:failcount:") and wikiMaxFailRetries (5)
constants in wiki_ingest.go
- requeueFailedOps now atomically Incr(wiki:failcount:{kbID}:{knowledgeID})
before each RPush; skips re-queue (drops op permanently) if count > 5
- Expire the fail-count key at wikiPendingTTL (24h) on every update
- wiki_ingest_batch.go: Del the fail-count key at ingestSucceeded++ so
transient errors don't permanently burn through a document's retry budget
Evidence of problem severity: 5 persistently-failing docs × ~110 requeue
cycles each = 553 pending entries; had to use Lua atomic dedup to recover.
Behaviour after this fix:
- First 5 failures → normal retry (re-queued, follow-up batch scheduled)
- 6th failure → logged as WARN "dropping op … after 6 failures (limit 5)",
not re-queued; prevents the feedback loop
- If document is fixed upstream and re-ingested via EnqueueWikiIngest, the
op is a fresh entry with no existing fail-count key → full 5-retry budget
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: when multiple wiki:ingest tasks queue up for the same KB,
they all hit ErrWikiIngestConcurrent and retry every 15 s. With
MaxRetry=10 (150 s window), any batch taking longer than ~2.5 min
caused pending tasks to exhaust retries and land in archived.
Confirmed via Redis inspection: archived wiki:ingest task had
retry=10, retried=10, error="concurrent wiki task active".
Two-part fix:
1. Early-exit at lock conflict when pending list is empty.
If another batch is active and the pending list is already empty,
there is nothing left to process — the concurrent task will handle
everything. Return nil (success) immediately instead of burning
through retry slots on a guaranteed no-op. Only retry when the
pending list still has items, i.e., we have real work to do.
2. Increase MaxRetry 10 → 25 across all four enqueue sites.
25 × 15 s = ~6 min window; accommodates large KB batches and the
60 s orphan-lock expiry with generous headroom. Combined with the
early-exit above, redundant tasks now consume 0 retries instead of
10, so the remaining budget is entirely available for genuine
concurrency conflicts.
No API or schema changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Attachments uploaded from the UI were correctly parsed by the handler and
set on QARequest.Attachments, but the Agent path dropped them before
calling engine.Execute. Only the KnowledgeQA pipeline injected them via
Attachments.BuildPrompt(), so in "smart reasoning" (Agent) mode the model
never saw the uploaded file and answered as if no attachment existed.
Append Attachments.BuildPrompt() to agentQuery alongside QuotedContext,
mirroring chat_pipeline/into_chat_message.go. This keeps the engine
signature unchanged and matches the existing KnowledgeQA behavior.
Wikis with tens of thousands of pages used to crash the browser trying
to render the entire link graph at once (30MB+ JSON, 100k+ SVG elements).
This change moves the graph viewer from "fetch everything, render
everything" to "fetch a slice, expand on demand" with several matching
UX improvements.
Backend
* GET /wiki/graph now accepts `mode` (overview | ego), `center`, `depth`,
`types`, `limit` query params. Default is overview top-500 by
link_count, capped at 2000. Response includes a Meta object so the
frontend can render a truncation hint and drive the UI.
* Pure helper `computeGraphSubset` extracted for testability; six unit
tests cover overview truncation / type filter / ego BFS / missing
center error.
* WikiLintService passes Limit=0 (uncapped) so link integrity checks
still walk every page.
* Repo `Search` adds CASE-based relevance ranking (title 4, slug 3,
summary 2, content 1) so full-text results put the most obvious
matches first instead of whatever was updated most recently.
Frontend
* Graph viewer fetches overview on entry; ego pivot on double-click,
search, wiki-link click, URL ?slug=, global issues jump.
* Shift+click or hover ⊕ button blooms neighbors onto the current
canvas additively. Bloom tracks generations and LRU-evicts oldest
when total exceeds 1500 nodes; ego center / selected node / latest
anchor are always protected.
* "Grow frontier (N)" legend action expands every dashed-ring node in
one click (6-way concurrency) while skipping Index/Log super-nodes
that would otherwise dump the whole wiki onto the canvas.
* Node dashed expansion ring + drawer "X/Y neighbors shown" hint tell
users which nodes still have neighbors to load.
* Search dropdown uses remote full-text search (debounced, sequence-
numbered to drop stale responses) and falls back to the overview
top-500 snapshot when the input is empty.
* Type filters now round-trip to the server so top-N is always
computed from the user's active type set rather than hiding nodes
client-side and shrinking the view.
* Status card replaces the cramped "centered on X · N hops · M nodes"
line with a structured focus/overview summary, resolving slugs to
page titles.
* Help popover in the legend lists every canvas shortcut.
i18n entries added for zh-CN / en-US / ko-KR / ru-RU.
Closes#1113
The graph extraction pipeline relies on a strict markdown fence regex to
pull JSON out of LLM responses. In production the regex misses ~84% of
real-world responses, in three ways:
1. The LLM hits max_tokens mid-output and produces no closing fence.
2. The opening fence is malformed or surrounded by prose, so the
non-greedy regex fails to anchor a match.
3. The model returns raw JSON with stray backticks but no real fence.
In every case extractContent fell through to returning the raw text,
which then failed json.Unmarshal with errors like
"invalid character '` + "`" + "' looking for beginning of value".
This change keeps the existing happy path untouched and adds a
conservative recovery step in the default branch of extractContent:
- If an opening ``` is present, take everything after it, drop a
likely language tag on the first line, cut at any trailing closing
fence, and trim stray backticks/whitespace.
- Otherwise, look for an outermost JSON object/array in the text using
a small bracket-balanced scanner that respects string literals, so
embedded {} or [] inside JSON strings don't confuse it.
- Only fall back to the original raw-text behavior when neither
strategy yields anything plausible.
The recovery helpers (stripFencesAndExtract, extractJSONLike,
isLikelyLanguageTag) are package-private and have table-driven tests in
extract_entity_test.go covering the three failure patterns described in
the issue, plus the previously-working fenced and bare-JSON shapes to
guard against regressions.
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.
Critical
- Heading splitter no longer prepends breadcrumb to Content. The
breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
combines header + content for the embedder. This restores the
End-Start == len(Content) invariant relied on by knowledge.go's
document-reconstruction path (summary generation, UI highlight) and
also eliminates the duplicate-heading regression where the section
heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
leaves a piece > ChunkSize, the next-priority separator is applied
inside that piece. Mirrors the Python reference. Without this, a
document with one paragraph break followed by a long run of
newline-separated lines could emit a single ~1950-rune chunk for
ChunkSize=300.
Medium
- SplitParentChild forces children onto the recursive tier, skipping
the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
whole chain rejects rather than running SplitText a second time.
Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
configs (TokenLimit clamping ChunkSize tiny while Overlap stays
high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
KBs that stored ChunkOverlap=0.
ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
knowledge service can build embedding inputs as
titlePrefix + chunk.EmbeddingContent() without a DB schema change.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Reciprocal Rank Fusion previously hard-coded k=60 and weights 0.7/0.3.
Both are now sourced from RetrievalConfig (RRFK, RRFVectorWeight,
RRFKeywordWeight) with the same numbers as defaults via three new
GetEffective accessors. fuseOrDeduplicate / fuseWithRRF accept the
config; HybridSearch reads it from tenantInfo and forwards.
Lets corpus owners tune fusion behavior — large corpora benefit from
larger k, keyword-heavy domains from a higher keyword weight, etc. —
without changing application code.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
batchEmbedWithBackoff replaces the previous fixed 100ms × 5 retry loop
with proper exponential backoff (200/400/800/1600/3200 ms) and respects
context cancellation between attempts. sanitizeForEmbedding caps any
single embedding input at 20k runes — well above realistic chunk
budgets, so it only triggers on pathologically large inputs and logs
a warning when it does.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).
Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
derived from TokenLimit so chunks stay safe under embedding model
token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
English (~4 chars/tok).
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds the building blocks for tiered chunking strategies — multilingual
patterns, token approximation, document profiling, validation, and a
strategy resolver that today still falls through to the legacy
SplitText. Default Strategy is empty/legacy so existing knowledge bases
continue to chunk identically.
- tokens.go: char/token approximation per language (en/de/zh/mixed)
and a coarse DetectLanguage heuristic
- patterns.go: source-of-truth regex bibliothek for headings, numbered
sections, multilingual chapter markers, visual separators, etc.
- profiler.go: single-pass document scan + SelectStrategy chain
- validator.go: rejects obviously broken chunk-sets to drive fallbacks
- strategy.go: public Split / SplitParentChild entry points; resolves
StrategyAuto via the profiler, anything else maps to a fixed tier
- splitter.go: SplitterConfig gains Strategy / TokenLimit / Languages
fields and DefaultChunkSize / DefaultChunkOverlap constants (overlap
raised to 80 ≈ 15%)
- knowledge.go: switches both call sites to chunker.Split /
SplitParentChild — behavior unchanged for empty Strategy
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.
- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
so fall-through to the tenant-merge path is intentional rather than an
unrecognised provider sliding past the switch. The legacy `cos_config` column
predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
handler/system.go StorageEngineStatusItem.Name comment now list all 7
providers (matched against types.StorageEngineConfig and ParseProviderScheme).
Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
StorageConfig data is only consumed for cos/minio and skipped for the
others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.
Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
fixing it requires API surface additions; can ship separately.
Closes#1117 (the buildStorageConfig switches; remaining items deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
internal/application/service/knowledge.go had grown to 9883 lines /
149 functions spanning CRUD, document processing, summary/question
generation, clone/move, FAQ (CRUD+import+index+export), wiki cleanup,
multimodal image, and file utilities. Navigating it, or diffing
changes in it, was increasingly painful.
Purely mechanical split into 8 files in the same package. No
signatures, behaviors, or types were modified. knowledgeService
struct, NewKnowledgeService, and package-level errors remain in
knowledge.go unchanged. Per-file imports were trimmed by goimports.
knowledge.go 497 struct + constructor + errs + CRUD reads + Search
knowledge_create.go 1144 CreateFrom{File,URL,Passage,Manual} + helpers
knowledge_delete.go 686 Delete + wiki cleanup + ProcessKnowledgeListDelete
knowledge_process.go 2407 ProcessDocument/Summary/Question + Reparse + UpdateImageInfo
knowledge_clone_move.go 1049 Clone* + Move* + progress persistence
knowledge_faq.go 1932 FAQ CRUD + Search + Export
knowledge_faq_import.go 1939 FAQ import/validate/index + runningFAQImportInfo
knowledge_util.go 352 file-type/url/hash + VLM+Storage config + resolveFileService
Verification: go build, go vet, gofmt on the 8 files, and
golangci-lint all clean (22 issues post-split matches 22 pre-split
in knowledge.go — no new issues introduced). The pre-existing
unused warnings on getVLMConfig / buildStorageConfig were left as-is
rather than removing dead code beyond the scope of this task.
Addresses two pre-merge review notes:
1. Extract the sentinel-returning threshold check from getSummary into
a small package-private helper (checkSufficientSummaryContent) and
add a unit test for it. The new test pins the behavior the caller
in ProcessSummaryGeneration relies on: the sentinel is returned for
empty / image-only / sub-threshold / scanned-PDF-without-OCR inputs,
and nil is returned for legitimate short notes and scanned PDFs
where VLM OCR succeeded.
The downstream "errors.Is(err, errInsufficientSummaryContent) →
SummaryStatusFailed + skip summary chunk" branch is mechanical (no
branching beyond the errors.Is check); covering it end-to-end would
require mocking KnowledgeRepository / ChunkRepository / chat.Chat,
for which there is no precedent in this package.
2. Switch minTextContentRunes from const to var so tests can override
the threshold and so future config plumbing can adjust it at
runtime without a rebuild. The new test exercises the override.
Drive-by comment fix: extractEntitiesAndConceptsNoUpsert still claimed
summary slugs are code-generated from the document title. Since the
previous commit they are derived from the knowledge ID instead.
Pure documentation change, no behavior impact.
Self-review of the prior commit surfaced one critical regression and
several smaller issues. This change addresses them.
Critical: the old stripImageMarkup stripped the WHOLE <image>...</image>
block, but searchutil.EnrichContentWithImageInfo wraps successful VLM OCR
output in exactly that shape — so a scanned PDF where OCR worked was
being rejected by hasSufficientTextContent and routed to the same
"failed" path as the OCR-empty case. The fix splits stripping into:
* <image_original>...</image_original> blocks removed entirely
(they only carry the redundant Markdown ref already stripped above)
* <image>, <images>, <image_caption>, <image_ocr> tags removed at
the tag level, INNER TEXT PRESERVED, so OCR / caption text reaches
the threshold check and the LLM call.
A regression test pins this down with realistic enriched-image XML that
the previous code failed silently on.
Other fixes:
* Threshold lowered from 50 to 10 runes — short legitimate documents
(brief memos, single-line notes) must still pass.
* Summary chunk content no longer prefixes "# Document\n<filename>";
only the LLM-generated summary is indexed, so filenames cannot
leak back into RAG retrieval context.
* Wiki summary slug is now derived from the knowledge ID rather than
the upload filename, eliminating filename-based slugs like
"summary/mx5280-pdf" that downstream wiki-edit prompts can read.
* Citation sourceRef likewise drops the "|<docTitle>" suffix.
* Dead docTitle parameter removed from extractCandidateSlugs,
extractEntitiesAndConceptsNoUpsert, classifyChunkCitations.
* The redundant stripImageMarkup pass in the warning log is gone;
realTextRuneCount caches the count once and uses
utf8.RuneCountInString to avoid the rune-slice allocation.
Wiki-page display titles still flow through DocTitle (filename) into
later wiki-edit / index-rebuild prompts. That is a larger refactor of
the wiki pipeline and is intentionally left for a separate PR.
Scanned PDFs without a successful OCR pass (e.g. when no VLM is configured
on the KB, or when the VLM returned a "no text content" reply that the
sanitizer discarded) reach the summary and wiki-ingest pipelines with
nothing but markdown image references for content. The downstream LLM
prompts used to receive the filename, file type, and title alongside that
empty content, and the generate_summary template explicitly forbade the
model from refusing — so the model would invent a topic from the filename
alone, producing e.g. a "Canon scanner manual" summary for an MX5280.pdf
that is actually a scanned legal letter.
Two changes shut this down:
1. Drop filename / title / file-type from every LLM prompt input. The
wiki summary, knowledge extract, candidate slug, and chunk citation
prompts now see only the document content. The knowledge summary
path no longer prepends the "Document Type: X / File Name: Y" intro.
2. Add a content-sufficiency guard in front of the LLM call. A new
helper strips markdown image references and HTML image tags and
refuses to invoke the model when fewer than 50 real-text runes
remain. getSummary returns a sentinel error in that case and the
caller marks the knowledge as SummaryStatusFailed instead of
surfacing the first chunk's bare image reference. Wiki ingest skips
LLM extraction with a clear log message.
The generate_summary.yaml prompt is also updated: the "NEVER output
refusal phrases" clause is removed and replaced with an explicit
empty-content rule that returns a deterministic placeholder line.
Real OCR (Tesseract / PaddleOCR) and filename-pattern sanitisation are
intentionally left for separate PRs.
Add Kingsoft Cloud KS3 as a supported storage provider, including
backend service implementation, storage config UI, and provider
scheme recognition throughout the codebase.
AgentFinalAnswer may carry Done:true more than once (e.g. finish chunk
plus EOF). Guard AddEpisode with sync.Once so memory persists once per
turn, consistent with qa.go completion handling.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Apply the session-list additions to the SQLite init migration (Lite build)
and make QueryPaged dialect-aware: LOWER(..) LIKE on SQLite, ILIKE on
Postgres, drop NULLS LAST on SQLite. Escape LIKE wildcards in the keyword
via the existing escapeLikeKeyword helper.
- SetPinned now returns rowsAffected so the handler can respond 404 on
unknown / unauthorized session IDs instead of a misleading 200.
- GET /sessions always returns the enriched shape (pin state + IM origin
fields) so the frontend never needs a second roundtrip; the dual-path
legacy branch in the handler is gone.
- Register pin routes with the wildcard name that matches each verb's
existing radix tree (POST :session_id, DELETE :id) and accept either
param name in the handler; avoids gin's "wildcard conflicts" panic.
- Drop the redundant [platform] prefix from IM session titles now that
the list renders a platform icon alongside the title; add unit tests
for shortID / buildUserSessionTitle / buildThreadSessionTitle.
- Frontend: remove the submenu search input and its i18n/keyword wiring
(search lives elsewhere in the app); pin icon uses the TDesign `pin`
glyph and inherits color so the active session turns green; optimistic
pin moves the item to the top of the list so it shows up at the top
of the Pinned group; IM text badge replaced by the platform SVG icons
from assets/img/im, desaturated by default and full-color on hover
or when the session is active.
Sessions today are a flat list ordered by updated_at. Two gaps showed up in
practice:
- Users cannot find specific chats as the list grows beyond a screen.
- IM-created sessions (WeCom/Feishu/Slack/...) are indistinguishable:
every title was "IM-<platform>" or "IM-<platform>-<username>" and the
list API hid the underlying im_channel_sessions mapping, so admins had
no way to tell which Feishu group a session came from.
Backend
- Migration 000039 adds sessions.user_id (owner), is_pinned, pinned_at
plus a composite index for the list query. Existing rows keep user_id
NULL and stay visible at the tenant level for backward compatibility.
- CreateSession now writes the caller's user_id from auth context.
- GET /sessions accepts keyword / source / agent_id. When any filter is
set, the response switches to enriched items that LEFT JOIN
im_channel_sessions and expose im_platform / im_chat_id / im_thread_id
/ im_user_id / im_agent_id / im_channel_id. No filters => legacy shape,
existing clients unaffected.
- Ordering: is_pinned DESC, pinned_at DESC NULLS LAST, updated_at DESC.
- POST/DELETE /sessions/:id/pin for user-scoped pin/unpin.
- IM session titles: "[platform] <user|chat|thread>" with short ID
suffixes so group/DM/thread sessions are visually distinct without
needing a round-trip to fetch a display name from the IM adapter.
Frontend
- Search input debounced at 300ms drives the keyword filter.
- Pinned chats render in a dedicated group above the time-based groups,
with a pin icon and a pin/unpin entry in the per-chat dropdown.
- IM chats get a short [platform] badge in the list.
- Pin toggle is optimistic and guards against double-clicks.
- zh/en/ko/ru i18n keys added for the new strings.
Reverts the tenant-ID-from-context change in da5fcd3. The storage path
encodes the resource owner's tenant, and the presigned-URL verifier
uses that ID to look up the owning tenant's StorageEngineConfig. Using
the caller's tenant ID from context would break cross-tenant shared
resources — e.g. when tenant Y reads an image from a KB shared by
tenant X, signing with Y would cause the verifier to open Y's storage
backend and 404 on X's file.
Keep:
- presignDefaultTTL shortened from 24h to 2h (independent improvement).
- godoc note on ParseTenantIDFromStoragePath flagging the ambiguity
for cloud paths with numeric bucket/region names.
- Unit tests for path-based tenant extraction and the no-external-URL
backward-compat path.
Addresses review feedback on the IM storage URL rewrite:
- localFileService.GetFileURL now reads tenant ID from request context
first, falling back to ParseTenantIDFromStoragePath only when context
is absent. Fixes ambiguity for cloud providers whose paths embed
numeric bucket/region names before the tenant segment, which could
mint presigned URLs bound to the wrong tenant ID.
- Shorten presigned URL default TTL from 24h to 2h. A leaked HMAC key
authorizes cross-tenant file reads, so URLs should expire quickly;
IM clients fetch referenced images within seconds anyway.
- Document ParseTenantIDFromStoragePath as a best-effort fallback.
- Add unit tests covering context-first, path-fallback, and the
no-external-URL backward-compat path.
IM platforms (Feishu, Slack, Telegram, DingTalk, Mattermost, WeCom) cannot
render provider:// URLs (local://, minio://, s3://, etc.) that appear in
LLM answers containing knowledge base images. The web frontend handles
these via the authenticated /files endpoint, but IM clients need publicly
resolvable HTTP URLs.
Changes:
- Add HMAC-SHA256 presigned URL utility (internal/utils/presign.go) for
generating time-limited, signature-verified file access URLs
- Add GET /api/v1/files/presigned endpoint that serves files without
session auth, verified by HMAC signature and expiry
- Update localFileService.GetFileURL() to return presigned HTTP URLs
when APP_EXTERNAL_URL is configured (cloud backends already return
presigned URLs via their SDKs)
- Add IM content rewriting pipeline: strip <image> XML tags, strip
citation tags, rewrite storage URLs to HTTP — applied at all IM
output points (streaming flush, non-streaming reply, fallback)
- Add holdback buffer in streaming flush to prevent URL/tag splitting
across chunk boundaries
Closes#1058
CreateTenant and UpdateAPIKey both overwrote tenant.APIKey with the
AES-encrypted ciphertext before returning, causing the UI to display
"enc:v1:..." right after generating or resetting an API key. Save the
plaintext in a local variable, encrypt only the value handed to
UpdateTenant, and return the plaintext to callers.
Subsequent reads via GetTenantByID continue to rely on the AfterFind
hook for transparent decryption, so this only affects the initial
write-back path.
When users ask broad queries like "请整理知识库中的数据" in RAG mode,
vector/keyword search returns nothing because the query has no specific
content to match. The user needs to see what documents exist, not search
results.
- Add buildKBDocumentListing() to fetch document titles/filenames from
the knowledge base and inject them into the fallback prompt via
{{kb_documents}} placeholder
- Update fallback prompt templates (model_fallback, default_fallback_prompt)
to include document listing context so the LLM can guide users
- Improve rewrite prompt: tighten summarize vs kb_search classification,
expand kb_search to cover browse/organize/list operations, add examples
Closes#959
- Add per-KB sync.Map lock in Lite mode to mirror Redis SetNX concurrency.
- Requeue failed wiki ops in Lite by enqueueing asynq tasks with delay/tracing.
- Teach SyncTaskExecutor to honor ProcessIn delay and MaxRetry like Redis asynq.
- Introduced a mechanism to track and requeue failed operations during the wiki ingest process.
- Added a new `requeueFailedOps` function to append failed operations back to the Redis pending list for retry in subsequent batches.
- Updated the `ProcessWikiIngest` method to collect failed operations and ensure they are retried after trimming the pending list.
This change improves the robustness of the ingest process by preventing data loss from transient failures.
Two critical fixes to the peekPendingList function:
1. Add error logging for JSON unmarshal failures (line 303)
- Previously: Malformed JSON items were silently skipped with no indication
- Now: Each failure is logged with the error and raw item content (first 100 chars)
- This makes data loss visible and debuggable
2. Fix trim count mismatch (line 325)
- Previously: Returned len(result) - the count of raw Redis items peeked
- Now: Returns len(ops) - the count of successfully parsed items
- Impact: Malformed items now stay in Redis for retry instead of being discarded
- This prevents silent data loss when JSON encoding issues occur
Root cause of the reported issue:
- When 8 files are uploaded in a batch, if one has encoding issues that cause
JSON.Unmarshal to fail, it was silently dropped from processing
- The file was trimmed from Redis without being processed
- Result: Only 7 of 8 files appeared in the Wiki
Fix approach:
- Make unmarshal failures visible in logs
- Retry malformed items in the next batch instead of discarding them
- Admin can see the errors and take corrective action
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.
This change threads one trace end-to-end:
- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
worker can attach to an upstream trace without emitting a duplicate
trace-create; StartGeneration now auto-picks parentObservationId from
ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
payloads so trace_id / parent_obs_id / user_id / session_id serialise
into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
resumes the upstream trace or opens a standalone asynq.<type> trace
for scheduled jobs, and wraps the handler in a SPAN with task metadata
(id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
ResumeTrace no-trace-create guarantee; asynq_test.go covers
InjectTracing round-trip, middleware resume path, and standalone
trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
and documents the cross-process propagation model.
No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
Closes#620#497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
wizardchen