perf(wiki): move ingest log to event table and index to on-demand API

Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
wizardchen
2026-05-08 21:43:53 +08:00
committed by lyingbug
parent bf12868923
commit d0144f3586
25 changed files with 2353 additions and 272 deletions
+77 -4
View File
@@ -115,12 +115,85 @@ export function deleteWikiPage(kbId: string, slug: string) {
return del(`/api/v1/knowledgebase/${kbId}/wiki/pages/${encodeSlugPath(slug)}`);
}
export function getWikiIndex(kbId: string) {
return get(`/api/v1/knowledgebase/${kbId}/wiki/index`);
export interface WikiIndexEntryDTO {
slug: string;
title: string;
summary: string;
}
export function getWikiLog(kbId: string) {
return get(`/api/v1/knowledgebase/${kbId}/wiki/log`);
export interface WikiIndexGroup {
type: string;
total: number;
items: WikiIndexEntryDTO[];
next_cursor?: string;
}
export interface WikiIndexResponse {
intro: string;
version: number;
groups: WikiIndexGroup[];
}
// getWikiIndex fetches the structured index view for a wiki KB. The
// backend replaced the legacy "markdown blob of intro + directory" with
// { intro, groups } so a 40k-page wiki no longer round-trips multiple
// megabytes on every index open. Pass `types` to restrict which
// page_type buckets come back; `limit` bounds the per-group window;
// `cursor` resumes from a previous response.
export function getWikiIndex(
kbId: string,
params?: { types?: string[]; limit?: number; cursor?: string },
) {
const query = new URLSearchParams();
if (params) {
if (params.types && params.types.length > 0) query.set('types', params.types.join(','));
if (params.limit !== undefined) query.set('limit', String(params.limit));
if (params.cursor) query.set('cursor', params.cursor);
}
const qs = query.toString();
const suffix = qs ? `?${qs}` : '';
return get(`/api/v1/knowledgebase/${kbId}/wiki/index${suffix}`);
}
export interface WikiLogPageRef {
slug: string;
title?: string;
}
export interface WikiLogEntry {
id: number;
tenant_id: number;
knowledge_base_id: string;
action: string;
knowledge_id: string;
doc_title: string;
summary: string;
// Each ref carries both slug (for navigation) and title (captured at
// ingest time for display). Legacy rows written before the title
// column was added surface as refs with an empty title; render falls
// back to the slug in that case.
pages_affected: WikiLogPageRef[];
created_at: string;
}
export interface WikiLogListResponse {
entries: WikiLogEntry[];
next_cursor?: string;
}
// getWikiLog fetches a page of wiki operation events (newest first). Pass the
// `next_cursor` from the previous response back as `cursor` to load more;
// an empty / missing `next_cursor` signals end-of-feed. `limit` is clamped
// server-side to [1, 200] and defaults to 50.
export function getWikiLog(kbId: string, params?: { cursor?: string; limit?: number }) {
const query = new URLSearchParams();
if (params) {
if (params.cursor) query.set('cursor', params.cursor);
if (params.limit !== undefined) query.set('limit', String(params.limit));
}
const qs = query.toString();
const suffix = qs ? `?${qs}` : '';
return get(`/api/v1/knowledgebase/${kbId}/wiki/log${suffix}`);
}
export interface WikiGraphQueryParams {