04e8105728
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
138 lines
3.9 KiB
Go
138 lines
3.9 KiB
Go
package service
|
|
|
|
import "testing"
|
|
|
|
func TestNormalizeSlugForCompare(t *testing.T) {
|
|
cases := map[string]string{
|
|
"shang-hai-tower": "shanghaitower",
|
|
"shanghai-tower": "shanghaitower",
|
|
"SHANG-HAI-TOWER": "shanghaitower",
|
|
"under_score_slug": "underscoreslug",
|
|
"entity/shanghai-tower": "entity/shanghaitower",
|
|
"": "",
|
|
"---": "",
|
|
"中文-slug": "中文slug",
|
|
}
|
|
for input, want := range cases {
|
|
t.Run(input, func(t *testing.T) {
|
|
if got := normalizeSlugForCompare(input); got != want {
|
|
t.Errorf("normalizeSlugForCompare(%q) = %q, want %q", input, got, want)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_HyphenVariation(t *testing.T) {
|
|
// The exact bug pattern reported in production: the LLM inserted a
|
|
// pinyin word break in a CJK-derived slug ("shang-hai" instead of
|
|
// "shanghai") even though the display text was copied verbatim from
|
|
// the canonical title. Resolver should heal the hyphenation drift
|
|
// using the display-text reverse lookup.
|
|
live := map[string]struct{}{
|
|
"entity/shanghai-tower": {},
|
|
}
|
|
titleToSlug := map[string]string{
|
|
"上海中心大厦": "entity/shanghai-tower",
|
|
}
|
|
got, ok := resolveDeadSlug(
|
|
"entity/shang-hai-tower",
|
|
"上海中心大厦",
|
|
live, titleToSlug,
|
|
)
|
|
if !ok {
|
|
t.Fatal("expected resolution success on pinyin-break drift case")
|
|
}
|
|
if got != "entity/shanghai-tower" {
|
|
t.Errorf("expected canonical slug, got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_ExactMatchPasses(t *testing.T) {
|
|
live := map[string]struct{}{
|
|
"entity/foo": {},
|
|
}
|
|
got, ok := resolveDeadSlug("entity/foo", "", live, nil)
|
|
if !ok || got != "entity/foo" {
|
|
t.Errorf("expected pass-through for live slug, got (%q, %v)", got, ok)
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_DisplayTextLookupPriority(t *testing.T) {
|
|
// Even when normalized-equality would also find a candidate,
|
|
// display-text lookup should win because it's the highest-
|
|
// confidence signal.
|
|
live := map[string]struct{}{
|
|
"entity/exact-match": {},
|
|
"entity/exactmatch": {},
|
|
}
|
|
titleToSlug := map[string]string{
|
|
"Exact Match": "entity/exactmatch",
|
|
}
|
|
got, ok := resolveDeadSlug(
|
|
"entity/exact-match-typo",
|
|
"Exact Match",
|
|
live, titleToSlug,
|
|
)
|
|
if !ok {
|
|
t.Fatal("expected resolution")
|
|
}
|
|
if got != "entity/exactmatch" {
|
|
t.Errorf("display-text lookup should have won; got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_DisplayTextStillRequiresLive(t *testing.T) {
|
|
// titleToSlug points at a slug that's NOT in liveSlugs; we
|
|
// shouldn't return it.
|
|
live := map[string]struct{}{
|
|
"entity/something-else": {},
|
|
}
|
|
titleToSlug := map[string]string{
|
|
"Title": "entity/dead-target",
|
|
}
|
|
if _, ok := resolveDeadSlug("entity/x", "Title", live, titleToSlug); ok {
|
|
t.Fatal("must not return a slug that isn't in liveSlugs")
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_BigramFallback(t *testing.T) {
|
|
// Slug differs from canonical by a single character (typo).
|
|
// Normalized-equality won't catch this; bigram Jaccard should.
|
|
live := map[string]struct{}{
|
|
"entity/zhongguo-yinhang": {},
|
|
}
|
|
got, ok := resolveDeadSlug(
|
|
"entity/zhongguo-yinghang", // h↔gh transposition typo
|
|
"",
|
|
live, nil,
|
|
)
|
|
if !ok {
|
|
t.Fatal("expected bigram fallback to recover the typo")
|
|
}
|
|
if got != "entity/zhongguo-yinhang" {
|
|
t.Errorf("got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_RejectsUnrelated(t *testing.T) {
|
|
// Completely unrelated candidates must not match — better to
|
|
// strip the link than to silently link to the wrong page.
|
|
live := map[string]struct{}{
|
|
"entity/foo": {},
|
|
"entity/bar": {},
|
|
"entity/baz": {},
|
|
}
|
|
if got, ok := resolveDeadSlug("entity/quuxasdf", "", live, nil); ok {
|
|
t.Errorf("expected no match for unrelated slug; got %q", got)
|
|
}
|
|
}
|
|
|
|
func TestResolveDeadSlug_EmptyInputs(t *testing.T) {
|
|
if _, ok := resolveDeadSlug("", "any", nil, nil); ok {
|
|
t.Error("empty deadSlug must return false")
|
|
}
|
|
if _, ok := resolveDeadSlug("entity/foo", "", nil, nil); ok {
|
|
t.Error("empty liveSlugs / titleToSlug must return false")
|
|
}
|
|
}
|