d0144f3586
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
33 lines
1.6 KiB
SQL
33 lines
1.6 KiB
SQL
-- Migration: 000040_wiki_log_entries
|
|
-- Description: Dedicated append-only event table for wiki operation log entries.
|
|
-- Replaces the "single giant TEXT column on slug='log' wiki_pages
|
|
-- row" model that caused O(n^2) write amplification as KBs grew:
|
|
-- every ingest/retract op previously did GetLog + UpdatePage
|
|
-- rewriting the entire (potentially multi-MB) TEXT column.
|
|
-- Here each event is one INSERT; reads paginate by (kb_id, id DESC).
|
|
|
|
DO $$ BEGIN RAISE NOTICE '[Migration 000040] Applying wiki_log_entries schema'; END $$;
|
|
|
|
CREATE TABLE IF NOT EXISTS wiki_log_entries (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
tenant_id BIGINT NOT NULL,
|
|
knowledge_base_id VARCHAR(36) NOT NULL,
|
|
action VARCHAR(32) NOT NULL,
|
|
knowledge_id VARCHAR(36) NOT NULL DEFAULT '',
|
|
doc_title TEXT NOT NULL DEFAULT '',
|
|
summary TEXT NOT NULL DEFAULT '',
|
|
pages_affected JSONB NOT NULL DEFAULT '[]'::JSONB,
|
|
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
|
|
);
|
|
|
|
-- Primary list query: cursor-paginated feed per KB, newest first.
|
|
-- Using id (BIGSERIAL, monotonic) as the cursor sidesteps duplicate-timestamp
|
|
-- tie-breaking that a created_at cursor would require.
|
|
CREATE INDEX IF NOT EXISTS idx_wiki_log_entries_kb_id_desc
|
|
ON wiki_log_entries (knowledge_base_id, id DESC);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_wiki_log_entries_tenant_id
|
|
ON wiki_log_entries (tenant_id);
|
|
|
|
DO $$ BEGIN RAISE NOTICE '[Migration 000040] wiki_log_entries schema applied successfully'; END $$;
|