04e8105728
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
189 lines
6.7 KiB
Go
189 lines
6.7 KiB
Go
package service
|
|
|
|
import (
|
|
"strings"
|
|
|
|
"github.com/Tencent/WeKnora/internal/searchutil"
|
|
)
|
|
|
|
// Fuzzy slug resolution: rescue LLM-generated `[[slug|display]]` links
|
|
// where the slug is *almost* right but not exactly the one in the KB.
|
|
//
|
|
// Background: the wiki ingest LLM is given an `<available_wiki_pages>`
|
|
// block listing exact `[[slug]] = title` entries. Empirically — even
|
|
// when the prompt explicitly says "copy-paste the slug verbatim" —
|
|
// open-source models such as deepseek-v3 routinely munge slugs in
|
|
// predictable ways:
|
|
//
|
|
// - inserting hyphens that happen to look like pinyin word breaks
|
|
// ("shanghaitower" → "shang-hai-tower"),
|
|
// - dropping or duplicating hyphens,
|
|
// - normalizing the case of an ASCII slug,
|
|
// - latinizing names that should have stayed CJK.
|
|
//
|
|
// The display text is almost always still correct (the model just
|
|
// copy-pastes the title), so we have two levers to find the *real*
|
|
// slug: a tolerant string compare on the slug itself, and a reverse
|
|
// lookup from display text. Combined, these recover the vast majority
|
|
// of broken links without any LLM cost — the alternative being
|
|
// stripDeadWikiLinks, which gives up and emits plain text.
|
|
//
|
|
// Cleanup callers (stripDeadWikiLinks, cleanDeadLinks) consult
|
|
// resolveDeadSlug FIRST and only fall back to "strip to plain text"
|
|
// when no live candidate is close enough to be safe.
|
|
|
|
// slugResolveBigramThreshold is the minimum char-bigram Jaccard
|
|
// similarity required for the bigram fallback to accept a candidate.
|
|
// 0.8 is intentionally conservative: at this level "shang-hai-tower"
|
|
// and "shanghai-tower" still match (most bigrams overlap), but
|
|
// "user-profile" and "user-permissions" do NOT (different stems even
|
|
// though the prefix matches). Anything weaker risks linking to the
|
|
// wrong page, which is worse than emitting plain text.
|
|
const slugResolveBigramThreshold = 0.8
|
|
|
|
// normalizeSlugForCompare collapses cosmetic slug variations so two
|
|
// slugs that differ only in hyphenation or case compare equal.
|
|
//
|
|
// Specifically:
|
|
// - lowercases ASCII letters (slugify already does this, so most
|
|
// real slugs are already lowercase, but the LLM occasionally
|
|
// produces capitalized forms),
|
|
// - removes every hyphen and underscore.
|
|
//
|
|
// Hyphens in wiki slugs are purely visual separators — slugify uses
|
|
// them between word fragments. Removing them lets us treat
|
|
// "shang-hai-tower" and "shanghai-tower" as the same logical token bag.
|
|
//
|
|
// CJK runes are preserved verbatim: they ARE the meaningful identity
|
|
// of CJK slugs, so collapsing them would over-merge.
|
|
func normalizeSlugForCompare(slug string) string {
|
|
slug = strings.ToLower(slug)
|
|
slug = strings.ReplaceAll(slug, "-", "")
|
|
slug = strings.ReplaceAll(slug, "_", "")
|
|
return slug
|
|
}
|
|
|
|
// resolveDeadSlug attempts to map a dead `[[slug]]` reference back to
|
|
// a live KB slug, using progressively more permissive heuristics:
|
|
//
|
|
// 1. Display-text reverse lookup. If the LLM emitted "[[bad-slug|上海
|
|
// 中心大厦]]" and there's a live page whose Title or alias is
|
|
// "上海中心大厦", return its slug. This is the most common case
|
|
// by far (the model copies the title correctly even when it
|
|
// mangles the slug).
|
|
//
|
|
// 2. Hyphen/case-normalized slug equality. Two slugs that compare
|
|
// equal under normalizeSlugForCompare are treated as the same
|
|
// logical page. Catches the LLM's "insert pinyin breaks" pattern.
|
|
//
|
|
// 3. Bigram Jaccard similarity ≥ slugResolveBigramThreshold (0.8).
|
|
// Catches typos and minor character substitutions that the
|
|
// normalize step doesn't fix. Computed via searchutil.Jaccard
|
|
// against character bigrams of each candidate slug.
|
|
//
|
|
// Returns the resolved live slug and true on success; "" and false
|
|
// when no candidate is close enough to be safe.
|
|
//
|
|
// The candidate pool is supplied by the caller as `liveSlugs`. The
|
|
// caller is also responsible for passing a `titleToSlug` map keyed
|
|
// by exact (case-sensitive) page title and alias surface forms. Both
|
|
// maps are consulted only — never mutated.
|
|
func resolveDeadSlug(
|
|
deadSlug string,
|
|
displayText string,
|
|
liveSlugs map[string]struct{},
|
|
titleToSlug map[string]string,
|
|
) (string, bool) {
|
|
if deadSlug == "" {
|
|
return "", false
|
|
}
|
|
// Already live? No-op success.
|
|
if _, ok := liveSlugs[deadSlug]; ok {
|
|
return deadSlug, true
|
|
}
|
|
|
|
// (1) Display-text reverse lookup. Trim whitespace because LLM
|
|
// occasionally emits `[[slug| display ]]` with stray spaces, and
|
|
// the user-side title would have been written without them.
|
|
if dt := strings.TrimSpace(displayText); dt != "" {
|
|
if slug, ok := titleToSlug[dt]; ok && slug != "" {
|
|
if _, live := liveSlugs[slug]; live {
|
|
return slug, true
|
|
}
|
|
}
|
|
}
|
|
|
|
// (2) Normalized-equality check. Compute the dead slug's normal
|
|
// form once and compare against every live candidate's normal
|
|
// form. O(N) but N is bounded — this only fires on the dead-link
|
|
// path, and `liveSlugs` is the per-batch affected set, not the
|
|
// full KB.
|
|
deadNorm := normalizeSlugForCompare(deadSlug)
|
|
if deadNorm == "" {
|
|
// Slug was nothing but hyphens / underscores after normalize
|
|
// — nothing meaningful to compare against.
|
|
return "", false
|
|
}
|
|
for cand := range liveSlugs {
|
|
if normalizeSlugForCompare(cand) == deadNorm {
|
|
return cand, true
|
|
}
|
|
}
|
|
|
|
// (3) Bigram Jaccard fallback. Use searchutil.Jaccard against
|
|
// the character bigrams of each slug. We restrict the bigram
|
|
// alphabet to the slug's normalized form (lowercase, no hyphens)
|
|
// so a hyphenation difference doesn't dominate the bigram set
|
|
// the way it would for raw slug strings.
|
|
deadGrams := slugCharBigrams(deadNorm)
|
|
if len(deadGrams) == 0 {
|
|
return "", false
|
|
}
|
|
var (
|
|
bestSlug string
|
|
bestScore float64
|
|
)
|
|
for cand := range liveSlugs {
|
|
candNorm := normalizeSlugForCompare(cand)
|
|
if candNorm == "" {
|
|
continue
|
|
}
|
|
candGrams := slugCharBigrams(candNorm)
|
|
if len(candGrams) == 0 {
|
|
continue
|
|
}
|
|
score := searchutil.Jaccard(deadGrams, candGrams)
|
|
if score > bestScore {
|
|
bestScore = score
|
|
bestSlug = cand
|
|
}
|
|
}
|
|
if bestScore >= slugResolveBigramThreshold {
|
|
return bestSlug, true
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// slugCharBigrams returns a character-bigram set for the given (already
|
|
// normalized) slug. Single-rune slugs degrade to a 1-gram so they still
|
|
// contribute a comparable signal.
|
|
//
|
|
// Mirrors surfaceGrams in wiki_ingest_dedup.go but operates on a slug
|
|
// (which is already lowercased / hyphen-stripped) rather than a raw
|
|
// surface form, so we don't redo the lowercasing or punctuation
|
|
// stripping work.
|
|
func slugCharBigrams(s string) map[string]struct{} {
|
|
if s == "" {
|
|
return nil
|
|
}
|
|
runes := []rune(s)
|
|
if len(runes) == 1 {
|
|
return map[string]struct{}{string(runes): {}}
|
|
}
|
|
out := make(map[string]struct{}, len(runes))
|
|
for i := 0; i < len(runes)-1; i++ {
|
|
out[string(runes[i:i+2])] = struct{}{}
|
|
}
|
|
return out
|
|
}
|