Files
Enterprise-Knowledge-Base/internal/application/service/slug_fuzzy.go
T
wizardchen 04e8105728 perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.

This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.

  - migration 000041: task_pending_ops + task_dead_letters tables, plus
    three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
    source_refs text fallback, lower(title) trgm).
  - TaskPendingOpsRepository + TaskDeadLetterRepository in
    internal/application/repository/task_queue.go: cursor-paginated
    list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
    delete that refuses empty keys so a buggy caller can't wipe a KB's
    queue.
  - internal/middleware/asynqdl: writes a task_dead_letters row when an
    asynq task exhausts its retry budget. Payload-agnostic — a small
    probe struct extracts TenantID + scope hints across every existing
    payload type, so summary:generation / image:multimodal /
    faq_import / kb:clone / etc. all dead-letter without per-handler
    code. Best-effort: an insert failure never masks the underlying
    task error. Installed in router/task.go before the langfuse mw so
    it sees raw errors.

  - Pending queue moved Redis → PG (no TTL, restart-durable). Redis
    keeps just the active-batch lock and the delete tombstone.
  - In-batch retry budget (wikiMaxFailRetries=5) tracked via
    pendingRepo.IncrFailCount; over the cap the row is moved to
    task_dead_letters with task_type=wiki:ingest, related_id=knowledge
    id. Asynq retries (10) are handled by the new middleware.
  - Removed the per-batch ListAllPages. WikiBatchContext now carries
    lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
    mutex-protected caches; reduce reaches for titles / summaries on
    demand instead of pre-loading the whole KB.
  - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
    which the new GIN index on source_refs serves as a Bitmap Index
    Scan instead of a sequential text LIKE.
  - Dedup pre-filter uses pg_trgm via FindSimilarPages
    (idx_wiki_pages_title_trgm). For each new entity/concept (and each
    of its aliases) we ask the DB for the top-K trigram-similar
    existing pages and union the results — bounded prompt size, no Go-
    side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
  - cleanDeadLinks / injectCrossLinks are scoped to the batch's
    affected slugs (a few dozen) instead of every page in the KB. Both
    use the new lite ListBySlugs / ExistsSlugs repo methods so they
    pull only slug + title + outlinks, not full content.
  - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
    [[bad-slug|display]] the cleanup path now tries display-text
    reverse lookup → hyphen/case normalized equality → char-bigram
    Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
    break drift case ("shang-hai-tower" vs "shanghai-tower") in place
    instead of replacing the link with plain text.
  - rebuildIndexPage uses ListByTypeRecent(200) for the first-time
    intro and drops the full DocumentSummaries blob from the
    incremental update prompt, so its context stays bounded regardless
    of KB size.
  - Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
    IngestMapParallel / IngestReduceParallel, with sensible defaults
    via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
    follow-ups don't waste asynq retry slots bouncing on the active
    lock.

  - RunLint walks pages via the new ListPagesCursor in 200-page
    windows and computes the live-slug set with a one-column
    ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
    every node + edge. Memory is now bounded; 40k pages walks in
    constant ~4MB.

  - 14 GORM tests for both repos against an in-memory SQLite mirror of
    the production DDL (task_queue_test.go).
  - 6 tests for the dead-letter middleware covering retry budget
    detection, payload-agnostic scope inference, error truncation, and
    repo-failure isolation.
  - 7 tests for the slug fuzzy resolve helper covering the three
    resolution stages, display-text priority over normalized equality,
    bigram fallback acceptance, and rejection of unrelated slugs.
  - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
    new fetcher / cursor APIs.
2026-05-10 00:01:06 +08:00

189 lines
6.7 KiB
Go

package service
import (
"strings"
"github.com/Tencent/WeKnora/internal/searchutil"
)
// Fuzzy slug resolution: rescue LLM-generated `[[slug|display]]` links
// where the slug is *almost* right but not exactly the one in the KB.
//
// Background: the wiki ingest LLM is given an `<available_wiki_pages>`
// block listing exact `[[slug]] = title` entries. Empirically — even
// when the prompt explicitly says "copy-paste the slug verbatim" —
// open-source models such as deepseek-v3 routinely munge slugs in
// predictable ways:
//
// - inserting hyphens that happen to look like pinyin word breaks
// ("shanghaitower" → "shang-hai-tower"),
// - dropping or duplicating hyphens,
// - normalizing the case of an ASCII slug,
// - latinizing names that should have stayed CJK.
//
// The display text is almost always still correct (the model just
// copy-pastes the title), so we have two levers to find the *real*
// slug: a tolerant string compare on the slug itself, and a reverse
// lookup from display text. Combined, these recover the vast majority
// of broken links without any LLM cost — the alternative being
// stripDeadWikiLinks, which gives up and emits plain text.
//
// Cleanup callers (stripDeadWikiLinks, cleanDeadLinks) consult
// resolveDeadSlug FIRST and only fall back to "strip to plain text"
// when no live candidate is close enough to be safe.
// slugResolveBigramThreshold is the minimum char-bigram Jaccard
// similarity required for the bigram fallback to accept a candidate.
// 0.8 is intentionally conservative: at this level "shang-hai-tower"
// and "shanghai-tower" still match (most bigrams overlap), but
// "user-profile" and "user-permissions" do NOT (different stems even
// though the prefix matches). Anything weaker risks linking to the
// wrong page, which is worse than emitting plain text.
const slugResolveBigramThreshold = 0.8
// normalizeSlugForCompare collapses cosmetic slug variations so two
// slugs that differ only in hyphenation or case compare equal.
//
// Specifically:
// - lowercases ASCII letters (slugify already does this, so most
// real slugs are already lowercase, but the LLM occasionally
// produces capitalized forms),
// - removes every hyphen and underscore.
//
// Hyphens in wiki slugs are purely visual separators — slugify uses
// them between word fragments. Removing them lets us treat
// "shang-hai-tower" and "shanghai-tower" as the same logical token bag.
//
// CJK runes are preserved verbatim: they ARE the meaningful identity
// of CJK slugs, so collapsing them would over-merge.
func normalizeSlugForCompare(slug string) string {
slug = strings.ToLower(slug)
slug = strings.ReplaceAll(slug, "-", "")
slug = strings.ReplaceAll(slug, "_", "")
return slug
}
// resolveDeadSlug attempts to map a dead `[[slug]]` reference back to
// a live KB slug, using progressively more permissive heuristics:
//
// 1. Display-text reverse lookup. If the LLM emitted "[[bad-slug|上海
// 中心大厦]]" and there's a live page whose Title or alias is
// "上海中心大厦", return its slug. This is the most common case
// by far (the model copies the title correctly even when it
// mangles the slug).
//
// 2. Hyphen/case-normalized slug equality. Two slugs that compare
// equal under normalizeSlugForCompare are treated as the same
// logical page. Catches the LLM's "insert pinyin breaks" pattern.
//
// 3. Bigram Jaccard similarity ≥ slugResolveBigramThreshold (0.8).
// Catches typos and minor character substitutions that the
// normalize step doesn't fix. Computed via searchutil.Jaccard
// against character bigrams of each candidate slug.
//
// Returns the resolved live slug and true on success; "" and false
// when no candidate is close enough to be safe.
//
// The candidate pool is supplied by the caller as `liveSlugs`. The
// caller is also responsible for passing a `titleToSlug` map keyed
// by exact (case-sensitive) page title and alias surface forms. Both
// maps are consulted only — never mutated.
func resolveDeadSlug(
deadSlug string,
displayText string,
liveSlugs map[string]struct{},
titleToSlug map[string]string,
) (string, bool) {
if deadSlug == "" {
return "", false
}
// Already live? No-op success.
if _, ok := liveSlugs[deadSlug]; ok {
return deadSlug, true
}
// (1) Display-text reverse lookup. Trim whitespace because LLM
// occasionally emits `[[slug| display ]]` with stray spaces, and
// the user-side title would have been written without them.
if dt := strings.TrimSpace(displayText); dt != "" {
if slug, ok := titleToSlug[dt]; ok && slug != "" {
if _, live := liveSlugs[slug]; live {
return slug, true
}
}
}
// (2) Normalized-equality check. Compute the dead slug's normal
// form once and compare against every live candidate's normal
// form. O(N) but N is bounded — this only fires on the dead-link
// path, and `liveSlugs` is the per-batch affected set, not the
// full KB.
deadNorm := normalizeSlugForCompare(deadSlug)
if deadNorm == "" {
// Slug was nothing but hyphens / underscores after normalize
// — nothing meaningful to compare against.
return "", false
}
for cand := range liveSlugs {
if normalizeSlugForCompare(cand) == deadNorm {
return cand, true
}
}
// (3) Bigram Jaccard fallback. Use searchutil.Jaccard against
// the character bigrams of each slug. We restrict the bigram
// alphabet to the slug's normalized form (lowercase, no hyphens)
// so a hyphenation difference doesn't dominate the bigram set
// the way it would for raw slug strings.
deadGrams := slugCharBigrams(deadNorm)
if len(deadGrams) == 0 {
return "", false
}
var (
bestSlug string
bestScore float64
)
for cand := range liveSlugs {
candNorm := normalizeSlugForCompare(cand)
if candNorm == "" {
continue
}
candGrams := slugCharBigrams(candNorm)
if len(candGrams) == 0 {
continue
}
score := searchutil.Jaccard(deadGrams, candGrams)
if score > bestScore {
bestScore = score
bestSlug = cand
}
}
if bestScore >= slugResolveBigramThreshold {
return bestSlug, true
}
return "", false
}
// slugCharBigrams returns a character-bigram set for the given (already
// normalized) slug. Single-rune slugs degrade to a 1-gram so they still
// contribute a comparable signal.
//
// Mirrors surfaceGrams in wiki_ingest_dedup.go but operates on a slug
// (which is already lowercased / hyphen-stripped) rather than a raw
// surface form, so we don't redo the lowercasing or punctuation
// stripping work.
func slugCharBigrams(s string) map[string]struct{} {
if s == "" {
return nil
}
runes := []rune(s)
if len(runes) == 1 {
return map[string]struct{}{string(runes): {}}
}
out := make(map[string]struct{}, len(runes))
for i := 0; i < len(runes)-1; i++ {
out[string(runes[i:i+2])] = struct{}{}
}
return out
}