Files
Enterprise-Knowledge-Base/internal/application/service/wiki_page.go
T
wizardchen 04e8105728 perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.

This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.

  - migration 000041: task_pending_ops + task_dead_letters tables, plus
    three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
    source_refs text fallback, lower(title) trgm).
  - TaskPendingOpsRepository + TaskDeadLetterRepository in
    internal/application/repository/task_queue.go: cursor-paginated
    list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
    delete that refuses empty keys so a buggy caller can't wipe a KB's
    queue.
  - internal/middleware/asynqdl: writes a task_dead_letters row when an
    asynq task exhausts its retry budget. Payload-agnostic — a small
    probe struct extracts TenantID + scope hints across every existing
    payload type, so summary:generation / image:multimodal /
    faq_import / kb:clone / etc. all dead-letter without per-handler
    code. Best-effort: an insert failure never masks the underlying
    task error. Installed in router/task.go before the langfuse mw so
    it sees raw errors.

  - Pending queue moved Redis → PG (no TTL, restart-durable). Redis
    keeps just the active-batch lock and the delete tombstone.
  - In-batch retry budget (wikiMaxFailRetries=5) tracked via
    pendingRepo.IncrFailCount; over the cap the row is moved to
    task_dead_letters with task_type=wiki:ingest, related_id=knowledge
    id. Asynq retries (10) are handled by the new middleware.
  - Removed the per-batch ListAllPages. WikiBatchContext now carries
    lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
    mutex-protected caches; reduce reaches for titles / summaries on
    demand instead of pre-loading the whole KB.
  - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
    which the new GIN index on source_refs serves as a Bitmap Index
    Scan instead of a sequential text LIKE.
  - Dedup pre-filter uses pg_trgm via FindSimilarPages
    (idx_wiki_pages_title_trgm). For each new entity/concept (and each
    of its aliases) we ask the DB for the top-K trigram-similar
    existing pages and union the results — bounded prompt size, no Go-
    side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
  - cleanDeadLinks / injectCrossLinks are scoped to the batch's
    affected slugs (a few dozen) instead of every page in the KB. Both
    use the new lite ListBySlugs / ExistsSlugs repo methods so they
    pull only slug + title + outlinks, not full content.
  - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
    [[bad-slug|display]] the cleanup path now tries display-text
    reverse lookup → hyphen/case normalized equality → char-bigram
    Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
    break drift case ("shang-hai-tower" vs "shanghai-tower") in place
    instead of replacing the link with plain text.
  - rebuildIndexPage uses ListByTypeRecent(200) for the first-time
    intro and drops the full DocumentSummaries blob from the
    incremental update prompt, so its context stays bounded regardless
    of KB size.
  - Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
    IngestMapParallel / IngestReduceParallel, with sensible defaults
    via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
    follow-ups don't waste asynq retry slots bouncing on the active
    lock.

  - RunLint walks pages via the new ListPagesCursor in 200-page
    windows and computes the live-slug set with a one-column
    ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
    every node + edge. Memory is now bounded; 40k pages walks in
    constant ~4MB.

  - 14 GORM tests for both repos against an in-memory SQLite mirror of
    the production DDL (task_queue_test.go).
  - 6 tests for the dead-letter middleware covering retry budget
    detection, payload-agnostic scope inference, error truncation, and
    repo-failure isolation.
  - 7 tests for the slug fuzzy resolve helper covering the three
    resolution stages, display-text priority over normalized equality,
    bigram fallback acceptance, and rejection of unrelated slugs.
  - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
    new fetcher / cursor APIs.
2026-05-10 00:01:06 +08:00

1035 lines
34 KiB
Go

package service
import (
"context"
"errors"
"fmt"
"regexp"
"sort"
"strconv"
"strings"
"time"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/google/uuid"
"github.com/redis/go-redis/v9"
)
// wikiLinkRegex matches [[wiki-link]] syntax in markdown content
var wikiLinkRegex = regexp.MustCompile(`\[\[([^\]]+)\]\]`)
// wikiPageService implements the WikiPageService interface
type wikiPageService struct {
repo interfaces.WikiPageRepository
chunkRepo interfaces.ChunkRepository
kbService interfaces.KnowledgeBaseService
taskPendingRepo interfaces.TaskPendingOpsRepository
redisClient *redis.Client
}
// NewWikiPageService creates a new wiki page service
func NewWikiPageService(
repo interfaces.WikiPageRepository,
chunkRepo interfaces.ChunkRepository,
kbService interfaces.KnowledgeBaseService,
taskPendingRepo interfaces.TaskPendingOpsRepository,
redisClient *redis.Client,
) interfaces.WikiPageService {
return &wikiPageService{
repo: repo,
chunkRepo: chunkRepo,
kbService: kbService,
taskPendingRepo: taskPendingRepo,
redisClient: redisClient,
}
}
// CreatePage creates a new wiki page
func (s *wikiPageService) CreatePage(ctx context.Context, page *types.WikiPage) (*types.WikiPage, error) {
if page.ID == "" {
page.ID = uuid.New().String()
}
if page.Slug == "" {
return nil, errors.New("wiki page slug is required")
}
if page.KnowledgeBaseID == "" {
return nil, errors.New("knowledge_base_id is required")
}
if page.Status == "" {
page.Status = types.WikiPageStatusPublished
}
if page.Version == 0 {
page.Version = 1
}
// Parse outbound links from content
page.OutLinks = s.parseOutLinks(page.Content)
now := time.Now()
page.CreatedAt = now
page.UpdatedAt = now
if err := s.repo.Create(ctx, page); err != nil {
return nil, fmt.Errorf("create wiki page: %w", err)
}
// Update inbound links on target pages
s.updateInLinks(ctx, page.KnowledgeBaseID, page.Slug, page.OutLinks)
return page, nil
}
// UpdatePage updates an existing wiki page.
//
// Version bump policy: the `version` column is intended to track the user-
// visible content revision, not every row rewrite. We therefore bump it only
// when at least one of the user-facing fields actually changes — title,
// content, summary, page_type, or status. Bookkeeping-only writes (refreshing
// source_refs after re-ingest when the body is identical, rebuilding the index
// page with the same directory, cross-link injection that ends up replacing
// nothing, etc.) still persist through `UpdateMeta` but leave `version`
// untouched so consumers can treat a bump as a real edit signal.
func (s *wikiPageService) UpdatePage(ctx context.Context, page *types.WikiPage) (*types.WikiPage, error) {
existing, err := s.repo.GetBySlug(ctx, page.KnowledgeBaseID, page.Slug)
if err != nil {
return nil, fmt.Errorf("get existing page: %w", err)
}
oldOutLinks := existing.OutLinks
// Snapshot user-visible fields BEFORE mutation so we can decide whether
// this is a real content change or just bookkeeping.
contentChanged := existing.Title != page.Title ||
existing.Content != page.Content ||
existing.Summary != page.Summary ||
existing.PageType != page.PageType ||
existing.Status != page.Status
existing.Title = page.Title
existing.Content = page.Content
existing.Summary = page.Summary
existing.PageType = page.PageType
existing.SourceRefs = page.SourceRefs
existing.ChunkRefs = page.ChunkRefs
existing.PageMetadata = page.PageMetadata
existing.Status = page.Status
existing.UpdatedAt = time.Now()
// Outbound links are a pure derivative of content, so they only shift
// when content shifts. Re-parse unconditionally to stay consistent with
// the stored body.
existing.OutLinks = s.parseOutLinks(existing.Content)
if contentChanged {
if err := s.repo.Update(ctx, existing); err != nil {
return nil, fmt.Errorf("update wiki page: %w", err)
}
} else {
// No user-visible change — persist bookkeeping fields but preserve
// the version so downstream consumers can rely on it.
if err := s.repo.UpdateMeta(ctx, existing); err != nil {
return nil, fmt.Errorf("update wiki page meta: %w", err)
}
}
// Update inbound links: remove old, add new. If content didn't change,
// oldOutLinks == existing.OutLinks and these calls are effectively no-ops.
s.removeInLinks(ctx, existing.KnowledgeBaseID, existing.Slug, oldOutLinks)
s.updateInLinks(ctx, existing.KnowledgeBaseID, existing.Slug, existing.OutLinks)
return existing, nil
}
// UpdatePageMeta updates only metadata (status, source_refs) without version bump or link re-parse.
func (s *wikiPageService) UpdatePageMeta(ctx context.Context, page *types.WikiPage) error {
page.UpdatedAt = time.Now()
return s.repo.UpdateMeta(ctx, page)
}
// UpdateAutoLinkedContent persists content produced by machine-only link
// decorators (cross-link injection / dead-link cleanup) without bumping
// `version`. Out-links are re-parsed from the new body and bidirectional
// in-link references on target pages are refreshed so link navigation stays
// consistent — only the user-facing revision counter is preserved.
func (s *wikiPageService) UpdateAutoLinkedContent(ctx context.Context, page *types.WikiPage) error {
existing, err := s.repo.GetBySlug(ctx, page.KnowledgeBaseID, page.Slug)
if err != nil {
return fmt.Errorf("get existing page: %w", err)
}
oldOutLinks := existing.OutLinks
existing.Content = page.Content
existing.OutLinks = s.parseOutLinks(existing.Content)
existing.UpdatedAt = time.Now()
if err := s.repo.UpdateAutoLinkedContent(ctx, existing); err != nil {
return fmt.Errorf("update auto-linked content: %w", err)
}
s.removeInLinks(ctx, existing.KnowledgeBaseID, existing.Slug, oldOutLinks)
s.updateInLinks(ctx, existing.KnowledgeBaseID, existing.Slug, existing.OutLinks)
return nil
}
// GetPageBySlug retrieves a wiki page by its slug
func (s *wikiPageService) GetPageBySlug(ctx context.Context, kbID string, slug string) (*types.WikiPage, error) {
return s.repo.GetBySlug(ctx, kbID, slug)
}
// GetPageByID retrieves a wiki page by its ID
func (s *wikiPageService) GetPageByID(ctx context.Context, id string) (*types.WikiPage, error) {
return s.repo.GetByID(ctx, id)
}
// ListPages lists wiki pages with optional filtering and pagination
func (s *wikiPageService) ListPages(ctx context.Context, req *types.WikiPageListRequest) (*types.WikiPageListResponse, error) {
pages, total, err := s.repo.List(ctx, req)
if err != nil {
return nil, err
}
pageSize := req.PageSize
if pageSize < 1 {
pageSize = 20
}
page := req.Page
if page < 1 {
page = 1
}
totalPages := int(total) / pageSize
if int(total)%pageSize > 0 {
totalPages++
}
return &types.WikiPageListResponse{
Pages: pages,
Total: total,
Page: page,
PageSize: pageSize,
TotalPages: totalPages,
}, nil
}
// DeletePage soft-deletes a wiki page
func (s *wikiPageService) DeletePage(ctx context.Context, kbID string, slug string) error {
page, err := s.repo.GetBySlug(ctx, kbID, slug)
if err != nil {
return err
}
// Remove inbound link references from pages this page links to
s.removeInLinks(ctx, kbID, slug, page.OutLinks)
// Delete the page
if err := s.repo.Delete(ctx, kbID, slug); err != nil {
return err
}
// Delete synced chunk
s.deleteChunkForPage(ctx, page)
return nil
}
// GetIndex returns the index page for a knowledge base
func (s *wikiPageService) GetIndex(ctx context.Context, kbID string) (*types.WikiPage, error) {
page, err := s.repo.GetBySlug(ctx, kbID, "index")
if err != nil {
if errors.Is(err, repository.ErrWikiPageNotFound) {
// Create default index page
return s.createDefaultPage(ctx, kbID, "index", "Index", types.WikiPageTypeIndex,
"# Wiki Index\n\nThis is the index page. It will be automatically updated as pages are added.\n")
}
return nil, err
}
return page, nil
}
// wikiIndexContentPageTypes enumerates the page types that make up a wiki's
// user-visible directory. System pages (index/log) are excluded; any
// LLM-created type we do not recognize surfaces under a generic "other"
// bucket.
var wikiIndexContentPageTypes = []string{
types.WikiPageTypeSummary,
types.WikiPageTypeEntity,
types.WikiPageTypeConcept,
types.WikiPageTypeSynthesis,
types.WikiPageTypeComparison,
}
// GetIndexView builds the structured index response without ever
// materializing a multi-MB directory markdown string. Intro is read from
// the index wiki_page row (which now carries only intro text — see
// rebuildIndexPage). Each requested page_type is paginated independently
// with ListByTypeLight so reads stay O(page_size) rather than O(total
// pages in the KB).
//
// `pageTypes` narrows which groups to include; empty = all content types.
// `limit` is the per-group window size (defaults to 50, capped at 200).
// `cursor` is an opaque offset string; currently we use the stringified
// offset so clients can resume where they left off. Because different
// page_types paginate independently, `cursor` applies uniformly to every
// group — if the caller wants per-group cursors it should request one
// type at a time via `pageTypes`. That simplifies the wire format and
// matches the frontend's tabbed UX.
func (s *wikiPageService) GetIndexView(
ctx context.Context,
kbID string,
pageTypes []string,
limit int,
cursor string,
) (*types.WikiIndexResponse, error) {
indexPage, err := s.GetIndex(ctx, kbID)
if err != nil {
return nil, fmt.Errorf("load index page: %w", err)
}
if limit <= 0 {
limit = 50
}
if limit > 200 {
limit = 200
}
offset := 0
if cursor != "" {
v, parseErr := strconv.Atoi(cursor)
if parseErr != nil || v < 0 {
return nil, fmt.Errorf("invalid cursor %q", cursor)
}
offset = v
}
// Default to every known content type when the caller passes no
// filter. Any unknown request-time type is passed through verbatim so
// future page types (declared in types/wiki_page.go) start showing
// up in the index the moment the LLM starts creating them, without a
// handler change.
selected := pageTypes
if len(selected) == 0 {
selected = append([]string{}, wikiIndexContentPageTypes...)
}
groups := make([]types.WikiIndexGroup, 0, len(selected))
for _, pt := range selected {
entries, total, listErr := s.repo.ListByTypeLight(ctx, kbID, pt, limit, offset)
if listErr != nil {
return nil, fmt.Errorf("list %s pages: %w", pt, listErr)
}
if entries == nil {
entries = []types.WikiIndexEntry{}
}
next := ""
// Only emit a cursor when a full page was returned AND more rows
// remain past `offset + limit`. A short page or one that exactly
// consumed the remainder should signal end-of-feed.
if len(entries) == limit && int64(offset+len(entries)) < total {
next = strconv.Itoa(offset + limit)
}
groups = append(groups, types.WikiIndexGroup{
Type: pt,
Total: total,
Items: entries,
NextCursor: next,
})
}
// The intro used to be stored on indexPage.Summary while
// indexPage.Content held intro + directory markdown. After the
// directory was lifted out of wiki_pages the content column holds
// only the intro. Fall back to Summary for KBs that haven't been
// re-ingested since the change so the response is never blank.
intro := indexPage.Content
if strings.TrimSpace(intro) == "" {
intro = indexPage.Summary
}
return &types.WikiIndexResponse{
Intro: intro,
Version: indexPage.Version,
Groups: groups,
}, nil
}
// GetLog returns the wiki_pages row for slug='log' if it exists.
//
// Log events are now stored in the dedicated `wiki_log_entries` table and
// paginated via wikiLogEntryService — the per-KB log is no longer a single
// TEXT column on a wiki_pages row (that model caused O(n^2) write
// amplification as logs grew). This method is retained for callers that
// still probe the legacy row (wiki_lint, knowledge delete, etc.), but it
// no longer auto-creates the placeholder page on miss; a missing row is a
// normal state and the helper returns `nil, nil`.
func (s *wikiPageService) GetLog(ctx context.Context, kbID string) (*types.WikiPage, error) {
page, err := s.repo.GetBySlug(ctx, kbID, "log")
if err != nil {
if errors.Is(err, repository.ErrWikiPageNotFound) {
return nil, nil
}
return nil, err
}
return page, nil
}
// GetGraph returns a slice of the wiki link graph for visualization.
//
// Two modes are supported:
//
// - WikiGraphModeOverview (default): returns the top `Limit` pages sorted
// by link_count (in+out), plus every edge that connects two surviving
// nodes. This is what the frontend fetches on the first graph open —
// 4万-page wikis would otherwise ship ~30MB of JSON and crash the
// browser trying to render 100k SVG elements.
//
// - WikiGraphModeEgo: returns the BFS neighborhood of `Center` up to
// `Depth` undirected hops, capped at `Limit` total nodes. The
// frontend uses this to drill down when the user clicks / searches a
// node in the overview.
//
// `Types` is an optional page_type allow-list applied to both the candidate
// node set and (in ego mode) the frontier expansion. Leaving it empty means
// no type filter.
//
// `Limit <= 0` disables the cap entirely and is reserved for internal
// callers like the lint service that need to walk every page. The HTTP
// handler always clamps Limit into a safe range so external traffic can
// never opt out of truncation.
//
// Implementation note: pages are still fetched via repo.ListAll. At 4万
// pages that's ~10MB of rows + deserialization, which is already on the
// expensive side but still tractable and keeps the repository interface
// unchanged. Pushing the filter/top-N down into SQL is a follow-up step
// (cache layer + DB-side projection) — see CLAUDE.md plan.
func (s *wikiPageService) GetGraph(ctx context.Context, req *types.WikiGraphRequest) (*types.WikiGraphData, error) {
if req == nil {
return nil, errors.New("wiki graph request is required")
}
pages, err := s.repo.ListAll(ctx, req.KnowledgeBaseID)
if err != nil {
return nil, err
}
return computeGraphSubset(pages, req)
}
// computeGraphSubset is the pure I/O-free core of GetGraph. It takes the
// full page list and a request description and returns the subgraph the
// caller asked for. Extracted from GetGraph so tests can exercise the
// mode/limit/type-filter behavior without plumbing a full repository mock.
func computeGraphSubset(pages []*types.WikiPage, req *types.WikiGraphRequest) (*types.WikiGraphData, error) {
mode := req.Mode
if mode == "" {
mode = types.WikiGraphModeOverview
}
// Pre-compute link_count and the type allow-list used for candidate
// filtering. We keep the full page list around so ego mode can still
// traverse through neighbors whose type is in the allow-list.
typeAllow := make(map[string]bool, len(req.Types))
for _, t := range req.Types {
if t != "" {
typeAllow[t] = true
}
}
hasTypeFilter := len(typeAllow) > 0
pageBySlug := make(map[string]*types.WikiPage, len(pages))
linkCount := make(map[string]int, len(pages))
for _, p := range pages {
pageBySlug[p.Slug] = p
linkCount[p.Slug] = len(p.InLinks) + len(p.OutLinks)
}
// Select the node slug set for the requested slice.
var selected map[string]struct{}
switch mode {
case types.WikiGraphModeEgo:
if req.Center == "" {
return nil, errors.New("ego graph requires a center slug")
}
if _, ok := pageBySlug[req.Center]; !ok {
return nil, fmt.Errorf("ego center slug %q not found", req.Center)
}
depth := req.Depth
if depth < 1 {
depth = 1
}
selected = bfsEgoSlugs(pageBySlug, req.Center, depth, typeAllow, req.Limit)
default:
// overview: keep only type-allowed candidates, sort by link_count desc, cap.
candidates := make([]*types.WikiPage, 0, len(pages))
for _, p := range pages {
if hasTypeFilter && !typeAllow[p.PageType] {
continue
}
candidates = append(candidates, p)
}
sort.SliceStable(candidates, func(i, j int) bool {
li := linkCount[candidates[i].Slug]
lj := linkCount[candidates[j].Slug]
if li != lj {
return li > lj
}
// Stable tiebreaker keeps the API deterministic between calls.
return candidates[i].Slug < candidates[j].Slug
})
if req.Limit > 0 && len(candidates) > req.Limit {
candidates = candidates[:req.Limit]
}
selected = make(map[string]struct{}, len(candidates))
for _, p := range candidates {
selected[p.Slug] = struct{}{}
}
}
// Build nodes from the selected set.
nodes := make([]types.WikiGraphNode, 0, len(selected))
for slug := range selected {
p := pageBySlug[slug]
nodes = append(nodes, types.WikiGraphNode{
Slug: p.Slug,
Title: p.Title,
PageType: p.PageType,
LinkCount: linkCount[slug],
})
}
// Deterministic node ordering — the map iteration above is random.
sort.Slice(nodes, func(i, j int) bool {
if nodes[i].LinkCount != nodes[j].LinkCount {
return nodes[i].LinkCount > nodes[j].LinkCount
}
return nodes[i].Slug < nodes[j].Slug
})
// Build edges, keeping only edges whose endpoints both survived selection.
var edges []types.WikiGraphEdge
for _, p := range pages {
if _, ok := selected[p.Slug]; !ok {
continue
}
for _, target := range p.OutLinks {
if _, ok := selected[target]; !ok {
continue
}
edges = append(edges, types.WikiGraphEdge{
Source: p.Slug,
Target: target,
})
}
}
// total is the count of candidate nodes before truncation — i.e. the
// population the frontend would need to fetch if it asked for the
// whole graph. For overview this respects the type filter; for ego
// it is the total KB page count (the user still sees "X of Y" based
// on the full wiki, not a filtered denominator).
total := len(pages)
if mode == types.WikiGraphModeOverview && hasTypeFilter {
total = 0
for _, p := range pages {
if typeAllow[p.PageType] {
total++
}
}
}
meta := types.WikiGraphMeta{
Mode: mode,
Total: total,
Returned: len(nodes),
Truncated: len(nodes) < total,
}
if mode == types.WikiGraphModeEgo {
meta.Center = req.Center
meta.Depth = req.Depth
if meta.Depth < 1 {
meta.Depth = 1
}
}
return &types.WikiGraphData{
Nodes: nodes,
Edges: edges,
Meta: meta,
}, nil
}
// bfsEgoSlugs computes the undirected BFS neighborhood of `center` up to
// `depth` hops using both inbound and outbound links. Type-filtered pages
// are excluded from the result but are also NOT traversed through — so a
// filter that hides "index" pages will not leak the whole wiki via the
// index. The caller guarantees center exists in pageBySlug.
func bfsEgoSlugs(
pageBySlug map[string]*types.WikiPage,
center string,
depth int,
typeAllow map[string]bool,
limit int,
) map[string]struct{} {
hasTypeFilter := len(typeAllow) > 0
centerPage, ok := pageBySlug[center]
if !ok {
return map[string]struct{}{}
}
// If the center itself fails the type filter we honor the filter and
// return an empty set — the handler will surface Returned=0.
if hasTypeFilter && !typeAllow[centerPage.PageType] {
return map[string]struct{}{}
}
visited := map[string]struct{}{center: {}}
frontier := []string{center}
for hop := 0; hop < depth; hop++ {
if limit > 0 && len(visited) >= limit {
break
}
next := make([]string, 0, len(frontier))
for _, slug := range frontier {
p, ok := pageBySlug[slug]
if !ok {
continue
}
neighbors := make([]string, 0, len(p.OutLinks)+len(p.InLinks))
neighbors = append(neighbors, p.OutLinks...)
neighbors = append(neighbors, p.InLinks...)
for _, nb := range neighbors {
if _, seen := visited[nb]; seen {
continue
}
np, exists := pageBySlug[nb]
if !exists {
continue
}
if hasTypeFilter && !typeAllow[np.PageType] {
continue
}
visited[nb] = struct{}{}
next = append(next, nb)
if limit > 0 && len(visited) >= limit {
break
}
}
if limit > 0 && len(visited) >= limit {
break
}
}
frontier = next
if len(frontier) == 0 {
break
}
}
return visited
}
// GetStats returns aggregate statistics about the wiki
func (s *wikiPageService) GetStats(ctx context.Context, kbID string) (*types.WikiStats, error) {
counts, err := s.repo.CountByType(ctx, kbID)
if err != nil {
return nil, err
}
var total int64
for _, c := range counts {
total += c
}
orphans, err := s.repo.CountOrphans(ctx, kbID)
if err != nil {
return nil, err
}
// Count total links
pages, err := s.repo.ListAll(ctx, kbID)
if err != nil {
return nil, err
}
var totalLinks int64
for _, p := range pages {
totalLinks += int64(len(p.OutLinks))
}
// Get recent updates (last 10)
listReq := &types.WikiPageListRequest{
KnowledgeBaseID: kbID,
Page: 1,
PageSize: 10,
SortBy: "updated_at",
SortOrder: "desc",
}
recentPages, _, err := s.repo.List(ctx, listReq)
if err != nil {
return nil, err
}
var pendingTasks int64
var pendingIssues int64
var isActive bool
if s.taskPendingRepo != nil {
// Pending wiki ingest ops live in task_pending_ops keyed by
// (task_type="wiki:ingest", scope="knowledge_base", scope_id=kbID).
pendingTasks, _ = s.taskPendingRepo.PendingCount(ctx, wikiTaskType, wikiTaskScope, kbID)
}
if s.redisClient != nil {
// The "active batch in progress" flag is still a Redis-only
// short-lived signal (per-process lock with TTL renew); not
// worth migrating since it carries no durable state.
activeFlag, _ := s.redisClient.Exists(ctx, "wiki:active:"+kbID).Result()
isActive = activeFlag > 0
}
issues, _ := s.ListIssues(ctx, kbID, "", "pending")
pendingIssues = int64(len(issues))
return &types.WikiStats{
TotalPages: total,
PagesByType: counts,
TotalLinks: totalLinks,
OrphanCount: orphans,
RecentUpdates: recentPages,
PendingTasks: pendingTasks,
PendingIssues: pendingIssues,
IsActive: isActive,
}, nil
}
// RebuildLinks re-parses all pages and rebuilds bidirectional link references
func (s *wikiPageService) RebuildLinks(ctx context.Context, kbID string) error {
pages, err := s.repo.ListAll(ctx, kbID)
if err != nil {
return err
}
// Build slug-to-page map
pageMap := make(map[string]*types.WikiPage)
for _, p := range pages {
pageMap[p.Slug] = p
}
// Clear all inbound links first
for _, p := range pages {
p.InLinks = types.StringArray{}
}
// Re-parse outbound links and rebuild inbound links
for _, p := range pages {
p.OutLinks = s.parseOutLinks(p.Content)
for _, target := range p.OutLinks {
if tp, exists := pageMap[target]; exists {
tp.InLinks = append(tp.InLinks, p.Slug)
}
}
}
// Save all pages (link rebuild is metadata-only, no version bump)
for _, p := range pages {
p.UpdatedAt = time.Now()
if err := s.repo.UpdateMeta(ctx, p); err != nil {
logger.Warnf(ctx, "wiki: failed to update links for page %s: %v", p.Slug, err)
}
}
return nil
}
// ListAllPages retrieves all wiki pages without pagination.
func (s *wikiPageService) ListAllPages(ctx context.Context, kbID string) ([]*types.WikiPage, error) {
return s.repo.ListAll(ctx, kbID)
}
// ListByType returns every wiki page of a given type for a KB. Exposed so
// callers like intro regeneration can load only the page type they need
// (summaries) instead of paying for the full ListAll scan.
func (s *wikiPageService) ListByType(ctx context.Context, kbID string, pageType string) ([]*types.WikiPage, error) {
return s.repo.ListByType(ctx, kbID, pageType)
}
// ListPagesBySourceRef exposes the repository's source-ref lookup so higher
// layers (delete flow, retract reconciliation) can re-query the current wiki
// state without depending on a stale caller-captured slug list.
func (s *wikiPageService) ListPagesBySourceRef(ctx context.Context, kbID string, knowledgeID string) ([]*types.WikiPage, error) {
return s.repo.ListBySourceRef(ctx, kbID, knowledgeID)
}
// ListSlugsBySourceRef returns just the slugs of pages that cite the given
// knowledge id. Backed by the source_refs GIN index added in migration
// 000041 — the wiki ingest pipeline uses it as a cheap "before" snapshot
// when reconciling old vs new extraction sets.
func (s *wikiPageService) ListSlugsBySourceRef(ctx context.Context, kbID string, knowledgeID string) ([]string, error) {
return s.repo.ListSlugsBySourceRef(ctx, kbID, knowledgeID)
}
// ListBySlugs is the lazy fetcher used by wiki ingest's batch context.
// Returns lightweight projections (no content / source_refs / chunk_refs)
// for the requested slugs, in a single IN query. Used in place of the
// pre-batch ListAllPages dump that historically pulled hundreds of MB
// for KBs in the tens of thousands of pages.
func (s *wikiPageService) ListBySlugs(ctx context.Context, kbID string, slugs []string) (map[string]*types.WikiPageLite, error) {
return s.repo.ListBySlugs(ctx, kbID, slugs)
}
// ListSummariesByKnowledgeIDs is the lazy fetcher for the retract /
// reparse branches of reduceSlugUpdates. Returns the content of each
// surviving summary page keyed by its source knowledge id.
func (s *wikiPageService) ListSummariesByKnowledgeIDs(ctx context.Context, kbID string, kids []string) (map[string]string, error) {
return s.repo.ListSummariesByKnowledgeIDs(ctx, kbID, kids)
}
// ExistsSlugs reports which of the given slugs are live (non-archived,
// non-deleted) in the KB. Used by cleanDeadLinks to validate out-link
// targets before stripping them.
func (s *wikiPageService) ExistsSlugs(ctx context.Context, kbID string, slugs []string) (map[string]bool, error) {
return s.repo.ExistsSlugs(ctx, kbID, slugs)
}
// ListAllSlugs returns every non-archived slug in the KB. Used by lint
// to compute the live-slug set without paying for ListAll's full row
// materialization.
func (s *wikiPageService) ListAllSlugs(ctx context.Context, kbID string) ([]string, error) {
return s.repo.ListAllSlugs(ctx, kbID)
}
// ListPagesCursor is the lint-side cursor pagination over wiki_pages.
func (s *wikiPageService) ListPagesCursor(ctx context.Context, kbID string, cursor string, limit int) ([]*types.WikiPage, string, error) {
return s.repo.ListPagesCursor(ctx, kbID, cursor, limit)
}
// ListByTypeRecent caps the page count for first-time index intro
// generation so the LLM prompt stays bounded on large KBs.
func (s *wikiPageService) ListByTypeRecent(ctx context.Context, kbID string, pageType string, limit int) ([]types.WikiIndexEntry, error) {
return s.repo.ListByTypeRecent(ctx, kbID, pageType, limit)
}
// FindSimilarPages performs a pg_trgm similarity search; used by the
// dedup pre-filter to surface candidate merge targets.
func (s *wikiPageService) FindSimilarPages(ctx context.Context, kbID string, query string, pageTypes []string, limit int) ([]*types.WikiPageLite, error) {
return s.repo.FindSimilarPages(ctx, kbID, query, pageTypes, limit)
}
// CountByType is a service-layer pass-through over the repo. Used by
// the index intro path to frame the LLM prompt's "showing N of M" hint.
func (s *wikiPageService) CountByType(ctx context.Context, kbID string) (map[string]int64, error) {
return s.repo.CountByType(ctx, kbID)
}
// SearchPages performs full-text search over wiki pages
func (s *wikiPageService) SearchPages(ctx context.Context, kbID string, query string, limit int) ([]*types.WikiPage, error) {
return s.repo.Search(ctx, kbID, query, limit)
}
// --- Internal helpers ---
// parseOutLinks extracts [[wiki-link]] slugs from markdown content
func (s *wikiPageService) parseOutLinks(content string) types.StringArray {
matches := wikiLinkRegex.FindAllStringSubmatch(content, -1)
seen := make(map[string]bool)
var links types.StringArray
for _, match := range matches {
if len(match) > 1 {
slug := strings.TrimSpace(match[1])
// Handle [[slug|display name]] format — slug is the first part
if parts := strings.SplitN(slug, "|", 2); len(parts) == 2 {
slug = strings.TrimSpace(parts[0])
}
slug = normalizeSlug(slug)
if slug != "" && !seen[slug] {
seen[slug] = true
links = append(links, slug)
}
}
}
return links
}
// normalizeSlug normalizes a wiki link slug
func normalizeSlug(slug string) string {
slug = strings.ToLower(strings.TrimSpace(slug))
slug = strings.ReplaceAll(slug, " ", "-")
return slug
}
// updateInLinks adds the source slug to the in_links of target pages
func (s *wikiPageService) updateInLinks(ctx context.Context, kbID string, sourceSlug string, targets types.StringArray) {
for _, targetSlug := range targets {
targetPage, err := s.repo.GetBySlug(ctx, kbID, targetSlug)
if err != nil {
continue // target page may not exist yet
}
if !containsString(targetPage.InLinks, sourceSlug) {
targetPage.InLinks = append(targetPage.InLinks, sourceSlug)
targetPage.UpdatedAt = time.Now()
if err := s.repo.UpdateMeta(ctx, targetPage); err != nil {
logger.Warnf(ctx, "wiki: failed to update in_links for %s: %v", targetSlug, err)
}
}
}
}
// removeInLinks removes the source slug from the in_links of target pages
func (s *wikiPageService) removeInLinks(ctx context.Context, kbID string, sourceSlug string, targets types.StringArray) {
for _, targetSlug := range targets {
targetPage, err := s.repo.GetBySlug(ctx, kbID, targetSlug)
if err != nil {
continue
}
newInLinks := removeString(targetPage.InLinks, sourceSlug)
if len(newInLinks) != len(targetPage.InLinks) {
targetPage.InLinks = newInLinks
targetPage.UpdatedAt = time.Now()
if err := s.repo.UpdateMeta(ctx, targetPage); err != nil {
logger.Warnf(ctx, "wiki: failed to update in_links for %s: %v", targetSlug, err)
}
}
}
}
// deleteChunkForPage removes the synced chunk for a wiki page
func (s *wikiPageService) deleteChunkForPage(ctx context.Context, page *types.WikiPage) {
chunkID := "wp-" + page.ID
if err := s.chunkRepo.DeleteChunk(ctx, page.TenantID, chunkID); err != nil {
logger.Warnf(ctx, "wiki: failed to delete chunk for page %s: %v", page.Slug, err)
}
}
// createDefaultPage creates a default system page (index, log)
func (s *wikiPageService) createDefaultPage(ctx context.Context, kbID string, slug string, title string, pageType string, content string) (*types.WikiPage, error) {
// Get KB to get tenant ID
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, kbID)
if err != nil {
return nil, fmt.Errorf("get knowledge base: %w", err)
}
page := &types.WikiPage{
ID: uuid.New().String(),
TenantID: kb.TenantID,
KnowledgeBaseID: kbID,
Slug: slug,
Title: title,
PageType: pageType,
Status: types.WikiPageStatusPublished,
Content: content,
Summary: title,
Version: 1,
}
if err := s.repo.Create(ctx, page); err != nil {
return nil, fmt.Errorf("create default %s page: %w", slug, err)
}
return page, nil
}
// containsString checks if a string slice contains a given string
func containsString(slice []string, s string) bool {
for _, v := range slice {
if v == s {
return true
}
}
return false
}
// removeString removes a string from a slice
func removeString(slice []string, s string) types.StringArray {
result := make(types.StringArray, 0, len(slice))
for _, v := range slice {
if v != s {
result = append(result, v)
}
}
return result
}
// CreateIssue logs a new issue for a wiki page
func (s *wikiPageService) CreateIssue(ctx context.Context, issue *types.WikiPageIssue) (*types.WikiPageIssue, error) {
if issue.ID == "" {
issue.ID = uuid.New().String()
}
if err := s.repo.CreateIssue(ctx, issue); err != nil {
return nil, fmt.Errorf("create wiki page issue: %w", err)
}
return issue, nil
}
// ListIssues retrieves issues for a knowledge base
func (s *wikiPageService) ListIssues(ctx context.Context, kbID string, slug string, status string) ([]*types.WikiPageIssue, error) {
return s.repo.ListIssues(ctx, kbID, slug, status)
}
// UpdateIssueStatus updates an issue's status
func (s *wikiPageService) UpdateIssueStatus(ctx context.Context, issueID string, status string) error {
return s.repo.UpdateIssueStatus(ctx, issueID, status)
}
// InjectCrossLinks scans affected pages and injects [[wiki-links]] for mentions
// of other wiki page titles in the content. Pure text replacement, no LLM call.
// Shares the linkifyContent helper with the ingest pipeline so both paths honor
// the same code-block / existing-link / word-boundary rules.
func (s *wikiPageService) InjectCrossLinks(ctx context.Context, kbID string, affectedSlugs []string) {
allPages, err := s.ListAllPages(ctx, kbID)
if err != nil || len(allPages) < 2 {
return
}
refs := collectLinkRefs(allPages)
if len(refs) == 0 {
return
}
affectedSet := make(map[string]bool, len(affectedSlugs))
for _, slug := range affectedSlugs {
affectedSet[slug] = true
}
var updated int
for _, p := range allPages {
if !affectedSet[p.Slug] {
continue
}
if p.PageType == types.WikiPageTypeIndex || p.PageType == types.WikiPageTypeLog {
continue
}
newContent, changed := linkifyContent(p.Content, refs, p.Slug)
if !changed {
continue
}
p.Content = newContent
if err := s.UpdateAutoLinkedContent(ctx, p); err != nil {
logger.Warnf(ctx, "wiki: cross-link injection failed for %s: %v", p.Slug, err)
continue
}
updated++
}
if updated > 0 {
logger.Infof(ctx, "wiki: injected cross-links in %d pages", updated)
}
}
// RebuildIndexPage was historically called by agent write/rename tools to
// refresh the index page's directory listing after a page mutation.
//
// The directory is no longer persisted in wiki_pages.content — it is
// assembled on demand by GetIndexView from the lightweight ListByTypeLight
// projection, so individual page writes don't need to redo O(N) string
// concatenation and rewrite a multi-MB TEXT column anymore. Keeping the
// method name lets existing agent tool call sites (wiki_write_page,
// wiki_rename_page) compile unchanged; the body is now intentionally a
// no-op.
//
// The intro that still lives on the index row is managed separately by
// the ingest pipeline (see wikiIngestService.rebuildIndexPage) on batch
// completion, which is where we actually have the LLM + change description
// context needed to rewrite it.
func (s *wikiPageService) RebuildIndexPage(ctx context.Context, kbID string) error {
_ = ctx
_ = kbID
return nil
}