wizardchen
d0144f3586
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
52 lines
1.8 KiB
Go
52 lines
1.8 KiB
Go
package service
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
|
)
|
|
|
|
// wikiLogEntryService is a thin wrapper over WikiLogEntryRepository. The
|
|
// event layer carries no business rules today, so the service just
|
|
// forwards to the repo — the indirection exists so handlers / ingest
|
|
// batch code depend on an interface rather than a concrete repo type.
|
|
type wikiLogEntryService struct {
|
|
repo interfaces.WikiLogEntryRepository
|
|
}
|
|
|
|
// NewWikiLogEntryService constructs a WikiLogEntryService backed by the
|
|
// given repository.
|
|
func NewWikiLogEntryService(repo interfaces.WikiLogEntryRepository) interfaces.WikiLogEntryService {
|
|
return &wikiLogEntryService{repo: repo}
|
|
}
|
|
|
|
// AppendBatch records the given events in one database round trip. Empty
|
|
// batches are a no-op (the repo handles that).
|
|
func (s *wikiLogEntryService) AppendBatch(ctx context.Context, entries []*types.WikiLogEntry) error {
|
|
return s.repo.AppendBatch(ctx, entries)
|
|
}
|
|
|
|
// List paginates the per-KB event feed. See repo.List for cursor semantics.
|
|
func (s *wikiLogEntryService) List(ctx context.Context, kbID string, cursor string, limit int) (*types.WikiLogEntryListResponse, error) {
|
|
entries, nextCursor, err := s.repo.List(ctx, kbID, cursor, limit)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if entries == nil {
|
|
// Normalise to an empty slice so clients don't need to
|
|
// distinguish `null` from `[]`.
|
|
entries = []*types.WikiLogEntry{}
|
|
}
|
|
return &types.WikiLogEntryListResponse{
|
|
Entries: entries,
|
|
NextCursor: nextCursor,
|
|
}, nil
|
|
}
|
|
|
|
// DeleteByKB removes the log feed for a KB. Called when the KB itself is
|
|
// being deleted, so no further reads happen.
|
|
func (s *wikiLogEntryService) DeleteByKB(ctx context.Context, kbID string) error {
|
|
return s.repo.DeleteByKB(ctx, kbID)
|
|
}
|