feat: Enhance wiki extraction and configuration options
- Introduced extraction granularity settings in the wiki configuration, allowing users to control the level of detail in entity and concept extraction (focused, standard, exhaustive). - Updated the UI to reflect these new options, including tooltips for better user guidance. - Refactored the knowledge base editor to support the new extraction granularity feature, ensuring a seamless user experience. - Improved the backend logic for wiki ingest tasks to utilize the new granularity settings, enhancing the accuracy of extracted content. - Added tests to validate the new extraction granularity functionality and ensure robust performance across different configurations.
This commit is contained in:
@@ -112,6 +112,152 @@ Output ONLY valid JSON. Example:
|
||||
]
|
||||
}`
|
||||
|
||||
// WikiCandidateSlugPrompt (Pass 0 of the chunk-cited pipeline) asks the LLM to
|
||||
// scan a document and output the SKELETON of all entities/concepts it contains:
|
||||
// name, slug, aliases, a short description, and a short details tiebreaker.
|
||||
// The heavy lifting — linking each slug to concrete supporting chunks — is
|
||||
// done in a second pass (see WikiChunkCitationPrompt). Because this prompt no
|
||||
// longer has to carry full facts per item, it stays cheap even for long docs.
|
||||
const WikiCandidateSlugPrompt = `You are a knowledge extraction system. Analyze the following document and list all significant entities AND key concepts as a lightweight candidate set. Another pass will later attach concrete supporting chunks to each item, so you do NOT need to write exhaustive per-item facts here.
|
||||
|
||||
<document>
|
||||
<title>{{.Title}}</title>
|
||||
<content>
|
||||
{{.Content}}
|
||||
</content>
|
||||
</document>
|
||||
|
||||
<previous_slugs>
|
||||
{{.PreviousSlugs}}
|
||||
</previous_slugs>
|
||||
|
||||
<instructions>
|
||||
Return a JSON object with two arrays: "entities" and "concepts".
|
||||
**IMPORTANT: Write ALL names, descriptions, and details in {{.Language}}**.
|
||||
|
||||
### Extraction Scope (Granularity: {{.Granularity}})
|
||||
{{.GranularityGuidance}}
|
||||
|
||||
### Slug Continuity Rules
|
||||
If previous slugs are provided above, you MUST follow these rules:
|
||||
- If an entity or concept from the previous extraction still exists in the current document, **reuse its exact slug** from the previous list. Do NOT generate a new slug for the same thing.
|
||||
- If an entity or concept no longer appears in the document, **do NOT include it** in the output.
|
||||
- Only generate new slugs for entities/concepts that are genuinely new (not present in the previous list).
|
||||
- This ensures slug stability across document updates.
|
||||
|
||||
### Entities (people, organizations, products, places, technologies, events, etc.)
|
||||
Each entity should have:
|
||||
- "name": The entity name in {{.Language}} (human-readable).
|
||||
- "slug": URL-friendly slug, format "entity/<lowercase-hyphenated-name>" (use romanized/pinyin form for non-Latin names). **Reuse previous slug if the entity was extracted before.**
|
||||
- "aliases": An array of strings representing names that refer to THE EXACT SAME entity. Only include: official abbreviations (e.g. "IBM" for "International Business Machines"), full/short name variants (e.g. "腾讯" for "腾讯控股有限公司"), translations, and well-known alternate names. Do NOT include parent categories, related products, generic terms, or broader concepts. Provide [] if none.
|
||||
- "description": **Index listing summary** — one sentence, 15-40 words, in {{.Language}}. Describes WHAT this entity IS and its role in the document. Must be self-contained. This will be displayed in the wiki index.
|
||||
- "details": A short 1-3 sentence fallback summary in {{.Language}}. This is ONLY used when chunk-level citation fails downstream, so it does NOT need to be exhaustive. Keep it under 300 characters.
|
||||
|
||||
Apply the Extraction Scope rules above. Never promote trivially-mentioned names into entities.
|
||||
|
||||
### Concepts (topics, themes, methodologies, theories, etc.)
|
||||
Each concept should have:
|
||||
- "name": The concept name in {{.Language}} (human-readable).
|
||||
- "slug": URL-friendly slug, format "concept/<lowercase-hyphenated-name>" (use romanized/pinyin form for non-Latin names). **Reuse previous slug if the concept was extracted before.**
|
||||
- "aliases": An array of strings representing names that refer to THE EXACT SAME concept. Only include: official abbreviations (e.g. "RAG" for "Retrieval-Augmented Generation"), full/short name variants, and well-known synonyms used interchangeably in the field. Do NOT include sub-topics, related techniques, broader categories, or implementation details. Provide [] if none.
|
||||
- "description": **Index listing summary** — one sentence, 15-40 words, in {{.Language}}. Defines WHAT this concept IS. Must be self-contained.
|
||||
- "details": A short 1-3 sentence fallback summary in {{.Language}}. Keep it under 300 characters.
|
||||
|
||||
Apply the Extraction Scope rules above. Skip concepts that are merely name-dropped without discussion.
|
||||
|
||||
### Deduplication Rules
|
||||
- If something is a specific named thing (person, company, product, place), put it ONLY in "entities".
|
||||
- If something is an abstract idea, methodology, or theory, put it ONLY in "concepts".
|
||||
- Never duplicate items across the two arrays.
|
||||
|
||||
### JSON Formatting Rules
|
||||
- **CRITICAL**: Do NOT use literal newline characters inside JSON string values. If you need a newline in a string, you MUST use the escaped sequence \n.
|
||||
</instructions>
|
||||
|
||||
Output ONLY valid JSON. Example:
|
||||
{
|
||||
"entities": [
|
||||
{
|
||||
"name": "Acme Corp",
|
||||
"slug": "entity/acme-corp",
|
||||
"aliases": ["Acme", "Acme Corporation"],
|
||||
"description": "A technology company specializing in AI solutions.",
|
||||
"details": "Founded in 2020, focuses on enterprise AI products."
|
||||
}
|
||||
],
|
||||
"concepts": [
|
||||
{
|
||||
"name": "Retrieval-Augmented Generation",
|
||||
"slug": "concept/retrieval-augmented-generation",
|
||||
"aliases": ["RAG"],
|
||||
"description": "A technique that combines information retrieval with language model generation.",
|
||||
"details": "Retrieves documents, then feeds them as context to an LLM."
|
||||
}
|
||||
]
|
||||
}`
|
||||
|
||||
// WikiChunkCitationPrompt (Pass 1..N of the chunk-cited pipeline) asks the LLM
|
||||
// to read a batch of chunks and, for each candidate entity/concept, list the
|
||||
// chunk IDs that substantively discuss it. This keeps per-slug "facts" in
|
||||
// their verbatim form (the chunk text) instead of asking the LLM to paraphrase.
|
||||
const WikiChunkCitationPrompt = `You are a precise citation system. Your job is to scan a batch of document chunks and decide, for each candidate entity/concept below, which chunks substantively discuss it.
|
||||
|
||||
<document_title>{{.DocTitle}}</document_title>
|
||||
|
||||
<candidate_slugs>
|
||||
{{.CandidateSlugs}}
|
||||
</candidate_slugs>
|
||||
|
||||
<chunks>
|
||||
{{.ChunksXML}}
|
||||
</chunks>
|
||||
|
||||
<instructions>
|
||||
**IMPORTANT: Write ALL names, descriptions, and details in {{.Language}}**.
|
||||
|
||||
### Primary task
|
||||
For each candidate slug above, select the chunk IDs (from the <chunks> block) that **substantively discuss** that entity/concept. "Substantively" means the chunk states at least one concrete fact, attribute, step, date, number, relationship, or other useful piece of information about the candidate — not a passing mention.
|
||||
|
||||
- Only cite chunks that appear in the <chunks> block above.
|
||||
- Use the "id" attribute of each <c> element verbatim (e.g. "c003").
|
||||
- If a candidate is not meaningfully discussed in ANY chunk in this batch, omit it from the output (do not include empty arrays).
|
||||
- A chunk CAN be cited by multiple candidates if it genuinely discusses multiple of them.
|
||||
- If a chunk is overly long or mixes unrelated topics, still cite it for every candidate it discusses.
|
||||
|
||||
### Secondary task: new slugs
|
||||
If this batch reveals a significant entity/concept that is **NOT** in <candidate_slugs>, you may add it under "new_slugs" so it gets incorporated. Only add genuinely new, substantively-discussed items. Do NOT rediscover items already listed above — reuse their slug if they are already candidates.
|
||||
|
||||
Each new slug must include:
|
||||
- "type": "entity" or "concept"
|
||||
- "name", "slug", "aliases", "description", "details" (same semantics as the candidate list)
|
||||
- "source_chunks": list of chunk IDs in the current batch that discuss it
|
||||
|
||||
### JSON Formatting Rules
|
||||
- **CRITICAL**: Do NOT use literal newline characters inside JSON string values. If needed, use \n.
|
||||
- Output ONLY valid JSON, no preamble.
|
||||
</instructions>
|
||||
|
||||
Output format:
|
||||
{
|
||||
"citations": {
|
||||
"entity/xxx": ["c001", "c003"],
|
||||
"concept/yyy": ["c002"]
|
||||
},
|
||||
"new_slugs": [
|
||||
{
|
||||
"type": "entity",
|
||||
"name": "Example",
|
||||
"slug": "entity/example",
|
||||
"aliases": [],
|
||||
"description": "...",
|
||||
"details": "...",
|
||||
"source_chunks": ["c005"]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
If nothing in this batch is cite-worthy, return: {"citations": {}, "new_slugs": []}`
|
||||
|
||||
// WikiPageModifyPrompt updates an existing wiki page with new additions and removes stale/deleted information in a single pass.
|
||||
const WikiPageModifyPrompt = `You are a wiki editor tasked with updating an existing wiki page. You must process a set of NEW information to add, AND/OR a set of deleted documents whose exclusive contributions must be REMOVED.
|
||||
|
||||
@@ -132,6 +278,8 @@ This wiki page is specifically about **{{.PageTitle}}** (a {{.PageType}}). Every
|
||||
<new_information>
|
||||
{{.NewContent}}
|
||||
</new_information>
|
||||
|
||||
The <new_information> block above is assembled from VERBATIM source chunks that were already cited as directly supporting this page. An optional <source_context> block inside each document is a document-level summary that tells you BOTH what the document is about AND what KIND of document it is (e.g. a resume, an announcement, a product page, a schedule) — use it to calibrate tone, stay on-topic, and avoid over-promotion. Do NOT quote the source_context text into the page; it is framing only.
|
||||
{{end}}
|
||||
|
||||
{{if .HasRetractions}}
|
||||
@@ -154,13 +302,17 @@ This wiki page is specifically about **{{.PageTitle}}** (a {{.PageType}}). Every
|
||||
2. REMOVE facts/claims that were ONLY sourced from the <deleted_documents> and are NOT present in any <remaining_source_documents> or <new_information>.
|
||||
{{end}}
|
||||
{{if .HasAdditions}}
|
||||
3. ADD and MERGE the facts, details, and context from the <new_information> into the page.
|
||||
3. ADD and MERGE the facts from <new_information> into the page. You are a COMPILER, not a writer:
|
||||
- **CRITICAL CONFLICT CHECK**: First verify that the <new_information> is actually about **{{.PageTitle}}** (as declared in <page_metadata>). If a piece of new info clearly belongs to a DIFFERENT but related thing (e.g., this page is about "Hunyuan Model" but the new info is about "Qwen3"; or this page is about "居民身份证" but the new info is about "工作居住证"), you MUST REJECT that part of the new information and DO NOT add it.
|
||||
- If it is genuinely about {{.PageTitle}} and contradicts old content, prefer the newer information.
|
||||
- **Stay close to source wording.** The chunks are verbatim. Reuse the source's own sentences; you MAY lightly reorder, deduplicate, and join related sentences, but do NOT rephrase for style, do NOT expand short statements into longer ones, and do NOT invent transitional sentences.
|
||||
- **Do NOT over-structure.** Only introduce a section heading (##, ###) if the source itself uses that heading OR the page already has one from existing content. For a new page with flat source text, a single "# {{.PageTitle}}" heading plus 1-2 short paragraphs and a flat bullet list of facts is PREFERRED over inventing a hierarchy of subsections.
|
||||
- **Do NOT add rhetorical filler.** Phrases like "旨在帮助…", "该平台致力于…", "具有重要意义", "designed to…", "aims to provide…" MUST NOT appear unless they are literally present in the source chunks.
|
||||
- **Scope discipline.** The source_context tells you whether the document is self-reported (e.g. a resume) or third-party authoritative. If the source is self-reported, do NOT elevate claims into industry-wide statements — stay descriptive and attribute when useful ("根据简历所述…" / "as described by…" is acceptable when the source is first-person).
|
||||
{{end}}
|
||||
4. Preserve existing information that is still valid and still about {{.PageTitle}}.
|
||||
5. Keep [[slug|name]] wiki-link references ONLY if the slug appears in the <valid_wiki_links> list above. Remove any [[slug|name]] whose slug is NOT in that list. Do NOT invent new wiki-link slugs. The page's own slug ({{.PageSlug}}) MUST NOT appear as a [[...]] link inside its own content.
|
||||
6. Maintain the existing page structure and formatting style. Use "# {{.PageTitle}}" as the top-level heading if the page does not already have one.
|
||||
6. Maintain the existing page structure and formatting style. Use "# {{.PageTitle}}" as the top-level heading if the page does not already have one. Do NOT introduce new heading levels beyond what the source or existing page justifies.
|
||||
7. **Image rule**: Include relevant images using Markdown syntax:  from new information if applicable.
|
||||
{{if .HasRetractions}}
|
||||
8. If after removing deleted content the page becomes nearly empty and there is no new information to add, output just: "SUMMARY: (empty page)\n# {{.PageTitle}}\n\n*This page's primary source document was removed.*"
|
||||
@@ -219,7 +371,6 @@ const WikiLogEntryTemplate = `## [{{.Date}}] {{.Operation}} | {{.Title}}
|
||||
- **Summary**: {{.Summary}}
|
||||
`
|
||||
|
||||
|
||||
// WikiDeduplicationPrompt asks the LLM to identify duplicate entities/concepts
|
||||
// between newly extracted items and existing wiki pages.
|
||||
const WikiDeduplicationPrompt = `You are a strict deduplication system. Given a list of newly extracted items and a list of existing wiki pages, determine which new items refer to the **exact same** real-world entity or concept as an existing page.
|
||||
@@ -268,3 +419,72 @@ If no items match any existing pages, return: {"merges": {}}
|
||||
|
||||
Output ONLY valid JSON. Example:
|
||||
{"merges": {"entity/acme-corporation": "entity/acme-corp", "concept/rag": "concept/retrieval-augmented-generation"}}`
|
||||
|
||||
// Granularity guidance blocks injected into WikiCandidateSlugPrompt. The
|
||||
// pipeline resolves a KnowledgeBase's configured granularity to one of these
|
||||
// strings via WikiGranularityGuidance().
|
||||
//
|
||||
// The three levels form a spectrum from "only the document's main subjects"
|
||||
// to "every named thing you see". Moving down the list monotonically
|
||||
// increases the candidate slug count, the downstream chunk-citation cost,
|
||||
// and the noise-to-signal ratio of the wiki index.
|
||||
const (
|
||||
WikiGranularityGuidanceFocused = `**FOCUSED mode — aggressive pruning.**
|
||||
Extract ONLY the document's primary subjects: the handful of entities/concepts that this document is fundamentally ABOUT.
|
||||
|
||||
INCLUDE:
|
||||
- The document's main subject(s) — e.g. for a resume: the person and their named projects; for an announcement: the announcing organization and the event/product being announced; for a product page: the product itself and its maker.
|
||||
- At most 3-7 items total across entities and concepts combined.
|
||||
|
||||
EXCLUDE (even if named explicitly):
|
||||
- Technology stacks / libraries / frameworks mentioned in passing (e.g. a resume listing "Spring Boot, MySQL, Redis" — do NOT extract these).
|
||||
- Generic concepts and methodologies that are merely referenced (e.g. "microservices", "async processing", "stateless authentication", "streaming response" mentioned as an implementation detail).
|
||||
- Places, schools, or organizations mentioned only as background (e.g. alma mater of a resume owner, unless the document is ABOUT the school itself).
|
||||
- Anything that would normally get a one-sentence description because there is not enough content to say more.
|
||||
|
||||
If you are unsure whether an item belongs, LEAVE IT OUT. A clean, focused index is more valuable than a comprehensive but noisy one.`
|
||||
|
||||
WikiGranularityGuidanceStandard = `**STANDARD mode — balanced (default).**
|
||||
Extract the document's main subjects PLUS entities/concepts that are substantively discussed — meaning they have a dedicated paragraph, multiple bullet points, or at least 2-3 sentences of context.
|
||||
|
||||
INCLUDE:
|
||||
- The document's main subject(s).
|
||||
- Secondary entities/concepts that receive a concrete block of content (a paragraph, a multi-point list, or a dedicated sub-section).
|
||||
- Named methodologies, architectures, or techniques when the document explains HOW the subject uses them — not merely names them.
|
||||
|
||||
EXCLUDE:
|
||||
- Items mentioned only in a comma-separated list of technologies without any further explanation (e.g. "Tech stack: A, B, C, D" — none of A/B/C/D are extracted unless they each also receive their own paragraph elsewhere).
|
||||
- One-off mentions, parenthetical references, and generic infrastructure nouns.
|
||||
- Items whose entire contribution to the document would fit in a single short sentence.
|
||||
|
||||
Aim for a tight, curated index. When in doubt about a marginal item, prefer to EXCLUDE it.`
|
||||
|
||||
WikiGranularityGuidanceExhaustive = `**EXHAUSTIVE mode — maximum recall.**
|
||||
Extract every named entity and every recognizable concept, including technologies, tools, standards, and methodologies mentioned even once by name, provided they are concrete and well-known (not generic terms like "database" or "function").
|
||||
|
||||
INCLUDE:
|
||||
- All main and secondary subjects.
|
||||
- All named technologies, libraries, frameworks, databases, services, protocols, or standards.
|
||||
- All recognizable concepts and methodologies that have widely-used names (e.g. RAG, microservices, async processing, SSE, JWT).
|
||||
|
||||
EXCLUDE ONLY:
|
||||
- Truly generic terms (e.g. "server", "function", "data").
|
||||
- Items that appear only inside URL paths or reference citations.
|
||||
|
||||
Use this mode when the knowledge base functions as a technical glossary rather than a curated narrative wiki.`
|
||||
)
|
||||
|
||||
// WikiGranularityGuidance returns the guidance text to inject into the
|
||||
// WikiCandidateSlugPrompt template for the given granularity. Accepts the
|
||||
// raw string value stored in WikiConfig.ExtractionGranularity; callers do
|
||||
// NOT need to Normalize() first — unknown values fall through to standard.
|
||||
func WikiGranularityGuidance(granularity string) string {
|
||||
switch granularity {
|
||||
case "focused":
|
||||
return WikiGranularityGuidanceFocused
|
||||
case "exhaustive":
|
||||
return WikiGranularityGuidanceExhaustive
|
||||
default:
|
||||
return WikiGranularityGuidanceStandard
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestWikiGranularityGuidance_RoutesByKey(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"focused": WikiGranularityGuidanceFocused,
|
||||
"standard": WikiGranularityGuidanceStandard,
|
||||
"exhaustive": WikiGranularityGuidanceExhaustive,
|
||||
}
|
||||
for key, want := range cases {
|
||||
if got := WikiGranularityGuidance(key); got != want {
|
||||
t.Errorf("WikiGranularityGuidance(%q) returned unexpected block", key)
|
||||
_ = got
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWikiGranularityGuidance_UnknownDefaultsToStandard(t *testing.T) {
|
||||
unknowns := []string{"", "FOCUSED", "detailed", "minimal", "full", "unknown"}
|
||||
for _, k := range unknowns {
|
||||
if WikiGranularityGuidance(k) != WikiGranularityGuidanceStandard {
|
||||
t.Errorf("WikiGranularityGuidance(%q) should fall back to STANDARD block", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sanity check that the three guidance blocks are meaningfully different.
|
||||
// A regression (e.g. two constants accidentally pointing at the same string)
|
||||
// would silently disable the user-facing level control.
|
||||
func TestWikiGranularityGuidance_BlocksAreDistinct(t *testing.T) {
|
||||
blocks := []string{
|
||||
WikiGranularityGuidanceFocused,
|
||||
WikiGranularityGuidanceStandard,
|
||||
WikiGranularityGuidanceExhaustive,
|
||||
}
|
||||
seen := make(map[string]bool, len(blocks))
|
||||
for _, b := range blocks {
|
||||
if b == "" {
|
||||
t.Error("granularity guidance block must not be empty")
|
||||
continue
|
||||
}
|
||||
if seen[b] {
|
||||
t.Error("granularity guidance blocks must be distinct")
|
||||
}
|
||||
seen[b] = true
|
||||
}
|
||||
|
||||
// Each block should name its mode, so the LLM can't silently get the
|
||||
// wrong guidance without us noticing in review.
|
||||
if !strings.Contains(WikiGranularityGuidanceFocused, "FOCUSED") {
|
||||
t.Error("focused block should self-identify")
|
||||
}
|
||||
if !strings.Contains(WikiGranularityGuidanceStandard, "STANDARD") {
|
||||
t.Error("standard block should self-identify")
|
||||
}
|
||||
if !strings.Contains(WikiGranularityGuidanceExhaustive, "EXHAUSTIVE") {
|
||||
t.Error("exhaustive block should self-identify")
|
||||
}
|
||||
}
|
||||
@@ -123,8 +123,8 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
|
||||
}
|
||||
}
|
||||
|
||||
// 6. Spawn Wiki Ingest Task if wiki indexing is enabled in IndexingStrategy and auto ingest is on
|
||||
if kb.IndexingStrategy.WikiEnabled && kb.WikiConfig != nil && kb.WikiConfig.AutoIngest && len(textChunks) > 0 {
|
||||
// 6. Spawn Wiki Ingest Task if wiki indexing is enabled in IndexingStrategy
|
||||
if kb.IndexingStrategy.WikiEnabled && len(textChunks) > 0 {
|
||||
EnqueueWikiIngest(ctx, s.taskEnqueuer, s.redisClient, payload.TenantID, payload.KnowledgeBaseID, payload.KnowledgeID)
|
||||
logger.Infof(ctx, "[KnowledgePostProcess] Enqueued wiki ingest task for %s", payload.KnowledgeID)
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package service
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
@@ -19,6 +20,15 @@ import (
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
// ErrWikiIngestConcurrent is returned by the wiki ingest handler when another
|
||||
// batch is already running for the same KB (i.e. the `wiki:active:<kbID>`
|
||||
// Redis lock is held). The asynq server's RetryDelayFunc uses errors.Is on
|
||||
// this sentinel to apply a short, fixed retry delay instead of asynq's default
|
||||
// exponential backoff — otherwise a freshly orphaned lock (e.g. from a crash
|
||||
// or restart) would force newcomers to wait minutes even after the lock
|
||||
// naturally expires.
|
||||
var ErrWikiIngestConcurrent = errors.New("concurrent wiki task active")
|
||||
|
||||
const (
|
||||
// maxContentForWiki limits the document content sent to LLM for wiki generation
|
||||
maxContentForWiki = 32768
|
||||
@@ -54,6 +64,20 @@ const (
|
||||
// wikiDeletedTTL bounds how long we remember a deletion. Must comfortably
|
||||
// exceed the longest plausible ingest run (LLM extraction + reduce).
|
||||
wikiDeletedTTL = 1 * time.Hour
|
||||
|
||||
// wikiActiveLockTTL is the TTL for the per-KB "batch in progress" flag.
|
||||
// Kept short (relative to total batch runtime) so that if the owning
|
||||
// process crashes without running its `defer Del`, the orphaned lock
|
||||
// expires quickly and newcomers aren't blocked. A periodic renew
|
||||
// (wikiActiveLockRenew) keeps the lock alive while the handler is
|
||||
// genuinely still running.
|
||||
wikiActiveLockTTL = 60 * time.Second
|
||||
|
||||
// wikiActiveLockRenew is how often the in-flight handler bumps the TTL.
|
||||
// Must be comfortably shorter than wikiActiveLockTTL so a single missed
|
||||
// tick (GC pause, Redis blip) doesn't let the lock slip out from under a
|
||||
// live handler.
|
||||
wikiActiveLockRenew = 20 * time.Second
|
||||
)
|
||||
|
||||
// WikiDeletedTombstoneKey returns the Redis key used to mark a knowledge as
|
||||
@@ -318,6 +342,12 @@ type WikiBatchContext struct {
|
||||
AllPages []*types.WikiPage
|
||||
SlugTitleMap map[string]string
|
||||
SummaryContentByKnowledgeID map[string]string
|
||||
// ExtractionGranularity drives Pass 0 (candidate slug extraction)
|
||||
// aggressiveness. Resolved once per batch from the KnowledgeBase's
|
||||
// WikiConfig so every doc in the batch sees the same scope rules.
|
||||
// Already Normalize()'d — consumers can assume it is one of the
|
||||
// three valid values.
|
||||
ExtractionGranularity types.WikiExtractionGranularity
|
||||
}
|
||||
|
||||
// SlugUpdate represents a single update operation for a specific slug
|
||||
@@ -332,6 +362,19 @@ type SlugUpdate struct {
|
||||
SummaryBody string // For summary
|
||||
SummaryLine string // For summary
|
||||
RetractDocContent string // For retract / retractStale
|
||||
// SourceChunks lists the chunk IDs (within KnowledgeID) that substantively
|
||||
// support this update. Mirrors Item.SourceChunks for convenience — the
|
||||
// Reduce phase reads from here to avoid an extra field hop.
|
||||
SourceChunks []string
|
||||
// DocSummary is the document-level summary body produced by
|
||||
// WikiSummaryPrompt (everything after the SUMMARY: ... headline, falling
|
||||
// back to the raw output if no headline could be parsed out). Carried
|
||||
// here so the Reduce phase can frame cited chunks with a rich
|
||||
// <source_context> block that tells the editor model what the document
|
||||
// is about AND what kind of document it is (resume vs announcement vs
|
||||
// product page). The one-line headline alone was too terse to keep the
|
||||
// editor grounded on longer / multi-topic source documents.
|
||||
DocSummary string
|
||||
}
|
||||
|
||||
func previewText(s string, maxRunes int) string {
|
||||
@@ -537,13 +580,19 @@ func (s *wikiIngestService) getExistingPageSlugsForKnowledge(ctx context.Context
|
||||
|
||||
// Multi-source → remove ref, queue retract
|
||||
|
||||
// extractedItem represents a single extracted entity or concept
|
||||
// extractedItem represents a single extracted entity or concept.
|
||||
//
|
||||
// SourceChunks holds the stable chunk IDs (from the source document) that
|
||||
// substantively discuss this item. Populated by the chunk-citation pass; when
|
||||
// non-empty the Reduce phase uses these chunks verbatim as the item's
|
||||
// evidence instead of the shorter Description/Details fields.
|
||||
type extractedItem struct {
|
||||
Name string `json:"name"`
|
||||
Slug string `json:"slug"`
|
||||
Aliases []string `json:"aliases"`
|
||||
Description string `json:"description"`
|
||||
Details string `json:"details"`
|
||||
Name string `json:"name"`
|
||||
Slug string `json:"slug"`
|
||||
Aliases []string `json:"aliases"`
|
||||
Description string `json:"description"`
|
||||
Details string `json:"details"`
|
||||
SourceChunks []string `json:"source_chunks,omitempty"`
|
||||
}
|
||||
|
||||
// combinedExtraction represents the parsed result of the combined entity+concept extraction
|
||||
@@ -779,11 +828,29 @@ func (s *wikiIngestService) deduplicateExtractedBatch(
|
||||
return entities, concepts
|
||||
}
|
||||
|
||||
if len(allPages) == 0 {
|
||||
return entities, concepts
|
||||
}
|
||||
|
||||
// Pre-filter the candidate existing-pages set. Passing the full corpus
|
||||
// to the LLM on large KBs bloats the prompt and empirically lets the
|
||||
// model hallucinate merges between totally unrelated slugs. The filter
|
||||
// is conservative (high recall) and the downstream validMerge check
|
||||
// remains in place for defense in depth.
|
||||
newItems := make([]extractedItem, 0, len(entities)+len(concepts))
|
||||
newItems = append(newItems, entities...)
|
||||
newItems = append(newItems, concepts...)
|
||||
candidatePages := selectDedupCandidatePages(newItems, allPages)
|
||||
if len(candidatePages) == 0 {
|
||||
return entities, concepts
|
||||
}
|
||||
if origCount := countEntityConceptPages(allPages); origCount > len(candidatePages) {
|
||||
logger.Infof(ctx, "wiki ingest: dedup candidate filter kept %d/%d existing pages for %d new items",
|
||||
len(candidatePages), origCount, len(newItems))
|
||||
}
|
||||
|
||||
var existingBuf strings.Builder
|
||||
for _, p := range allPages {
|
||||
if p.PageType != types.WikiPageTypeEntity && p.PageType != types.WikiPageTypeConcept {
|
||||
continue
|
||||
}
|
||||
for _, p := range candidatePages {
|
||||
writeDedupItemXML(&existingBuf, p.Slug, p.Title, string(p.PageType), []string(p.Aliases))
|
||||
}
|
||||
if existingBuf.Len() == 0 {
|
||||
|
||||
@@ -101,16 +101,22 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
|
||||
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
|
||||
}
|
||||
|
||||
// Try to acquire the "active batch" flag (non-blocking)
|
||||
// Try to acquire the "active batch" flag (non-blocking).
|
||||
//
|
||||
// TTL is intentionally short (wikiActiveLockTTL ≈ 60s) so that if the
|
||||
// owning process dies without releasing the lock (crash, kill -9,
|
||||
// container restart), the orphaned key expires within ~1 minute and new
|
||||
// tasks aren't starved. A renew goroutine keeps the lock alive while
|
||||
// the handler is genuinely running.
|
||||
if s.redisClient != nil {
|
||||
activeKey := wikiActiveKeyPrefix + payload.KnowledgeBaseID
|
||||
acquired, err := s.redisClient.SetNX(ctx, activeKey, "1", 5*time.Minute).Result()
|
||||
acquired, err := s.redisClient.SetNX(ctx, activeKey, "1", wikiActiveLockTTL).Result()
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki ingest: redis SetNX failed: %v", err)
|
||||
} else if !acquired {
|
||||
exitStatus = "active_lock_conflict"
|
||||
logger.Infof(ctx, "wiki ingest: another batch active for KB %s, deferring to asynq retry", payload.KnowledgeBaseID)
|
||||
return fmt.Errorf("concurrent wiki task active, please retry")
|
||||
return ErrWikiIngestConcurrent
|
||||
}
|
||||
lockAcquired = acquired
|
||||
|
||||
@@ -121,14 +127,14 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
|
||||
}()
|
||||
|
||||
go func() {
|
||||
ticker := time.NewTicker(2 * time.Minute)
|
||||
ticker := time.NewTicker(wikiActiveLockRenew)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-lockCtx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.redisClient.Expire(context.Background(), activeKey, 5*time.Minute)
|
||||
s.redisClient.Expire(context.Background(), activeKey, wikiActiveLockTTL)
|
||||
}
|
||||
}
|
||||
}()
|
||||
@@ -145,13 +151,11 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
|
||||
exitStatus = "kb_not_wiki_enabled"
|
||||
return fmt.Errorf("wiki ingest: KB %s is not wiki type", kb.ID)
|
||||
}
|
||||
if kb.WikiConfig == nil || !kb.WikiConfig.AutoIngest {
|
||||
exitStatus = "auto_ingest_disabled"
|
||||
logger.Infof(ctx, "wiki ingest: auto_ingest disabled for KB %s, skipping", kb.ID)
|
||||
return nil
|
||||
}
|
||||
|
||||
synthesisModelID := kb.WikiConfig.SynthesisModelID
|
||||
var synthesisModelID string
|
||||
if kb.WikiConfig != nil {
|
||||
synthesisModelID = kb.WikiConfig.SynthesisModelID
|
||||
}
|
||||
if synthesisModelID == "" {
|
||||
synthesisModelID = kb.SummaryModelID
|
||||
}
|
||||
@@ -189,10 +193,21 @@ func (s *wikiIngestService) ProcessWikiIngest(ctx context.Context, t *asynq.Task
|
||||
|
||||
// Fetch all existing pages once, shared across Map and Reduce phases
|
||||
allPages, _ := s.wikiService.ListAllPages(ctx, payload.KnowledgeBaseID)
|
||||
|
||||
// Resolve extraction granularity once per batch. Historical rows with
|
||||
// empty/unknown values fall back to Standard via Normalize(). Failures
|
||||
// to load the KB (unlikely since we're already acting on it) also
|
||||
// degrade gracefully to Standard.
|
||||
granularity := types.WikiExtractionStandard
|
||||
if kb, kbErr := s.kbService.GetKnowledgeBaseByID(ctx, payload.KnowledgeBaseID); kbErr == nil && kb != nil && kb.WikiConfig != nil {
|
||||
granularity = kb.WikiConfig.ExtractionGranularity.Normalize()
|
||||
}
|
||||
|
||||
batchCtx := &WikiBatchContext{
|
||||
AllPages: allPages,
|
||||
SlugTitleMap: make(map[string]string),
|
||||
SummaryContentByKnowledgeID: make(map[string]string),
|
||||
ExtractionGranularity: granularity,
|
||||
}
|
||||
for _, p := range allPages {
|
||||
if p.PageType != types.WikiPageTypeIndex && p.PageType != types.WikiPageTypeLog && p.Status != types.WikiPageStatusArchived {
|
||||
@@ -457,22 +472,35 @@ func (s *wikiIngestService) mapOneDocument(
|
||||
sourceRef := fmt.Sprintf("%s|%s", knowledgeID, docTitle)
|
||||
oldPageSlugs := s.getExistingPageSlugsForKnowledge(ctx, payload.KnowledgeBaseID, knowledgeID)
|
||||
|
||||
logger.Infof(ctx, "wiki ingest: extracting entities and concepts for %s", knowledgeID)
|
||||
extractedEntities, extractedConcepts, slugItems, err := s.extractEntitiesAndConceptsNoUpsert(ctx, chatModel, content, docTitle, lang, oldPageSlugs, batchCtx)
|
||||
// Pass 0: lightweight candidate slug extraction (skeleton only).
|
||||
// On failure we fall back to the legacy single-shot extractor so the doc
|
||||
// still gets ingested, just without chunk-level citations.
|
||||
var (
|
||||
extractedEntities []extractedItem
|
||||
extractedConcepts []extractedItem
|
||||
slugItems map[string]extractedItem
|
||||
pass0Failed bool
|
||||
)
|
||||
logger.Infof(ctx, "wiki ingest: pass 0 — extracting candidate slugs for %s", knowledgeID)
|
||||
extractedEntities, extractedConcepts, slugItems, err = s.extractCandidateSlugs(ctx, chatModel, content, docTitle, lang, oldPageSlugs, batchCtx)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki ingest: knowledge extraction failed for %s: %v", knowledgeID, err)
|
||||
return nil, nil, err
|
||||
logger.Warnf(ctx, "wiki ingest: pass 0 failed for %s (%v) — falling back to legacy extractor", knowledgeID, err)
|
||||
pass0Failed = true
|
||||
extractedEntities, extractedConcepts, slugItems, err = s.extractEntitiesAndConceptsNoUpsert(ctx, chatModel, content, docTitle, lang, oldPageSlugs, batchCtx)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki ingest: legacy fallback also failed for %s: %v", knowledgeID, err)
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
|
||||
var extractedPages []string
|
||||
// Build slug listing for Summary's wiki-link input.
|
||||
var summaryExtractedPages []string
|
||||
for slug := range slugItems {
|
||||
extractedPages = append(extractedPages, slug)
|
||||
summaryExtractedPages = append(summaryExtractedPages, slug)
|
||||
}
|
||||
|
||||
// Summary
|
||||
summarySlug := fmt.Sprintf("summary/%s", slugify(docTitle))
|
||||
var slugListing string
|
||||
for _, slug := range extractedPages {
|
||||
for _, slug := range summaryExtractedPages {
|
||||
if item, ok := slugItems[slug]; ok {
|
||||
aliases := ""
|
||||
if len(item.Aliases) > 0 {
|
||||
@@ -484,20 +512,87 @@ func (s *wikiIngestService) mapOneDocument(
|
||||
}
|
||||
}
|
||||
|
||||
var docSummaryLine string
|
||||
summaryContent, err := s.generateWithTemplate(ctx, chatModel, agent.WikiSummaryPrompt, map[string]string{
|
||||
"Title": docTitle,
|
||||
"FileName": docTitle,
|
||||
"FileType": "document",
|
||||
"Content": content,
|
||||
"Language": lang,
|
||||
"ExtractedSlugs": slugListing,
|
||||
})
|
||||
// Summary and chunk classification are independent given Pass 0 output —
|
||||
// run them in parallel. Summary handles wiki-link injection; classification
|
||||
// attaches concrete chunk IDs to each candidate slug.
|
||||
var (
|
||||
summaryContent string
|
||||
summaryErr error
|
||||
citations map[string][]string
|
||||
newSlugs []newSlugFromCitation
|
||||
batchCount int
|
||||
)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(2)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
summaryContent, summaryErr = s.generateWithTemplate(ctx, chatModel, agent.WikiSummaryPrompt, map[string]string{
|
||||
"Title": docTitle,
|
||||
"FileName": docTitle,
|
||||
"FileType": "document",
|
||||
"Content": content,
|
||||
"Language": lang,
|
||||
"ExtractedSlugs": slugListing,
|
||||
})
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
// Skip citation pass when Pass 0 has fallen back to the legacy path —
|
||||
// the legacy output already contains paraphrased Details, so chunk
|
||||
// citations would be redundant and we'd spend LLM calls for nothing.
|
||||
if pass0Failed {
|
||||
citations = map[string][]string{}
|
||||
return
|
||||
}
|
||||
candidatesXML := renderCandidateSlugsXML(extractedEntities, extractedConcepts)
|
||||
citations, newSlugs, batchCount = s.classifyChunkCitations(ctx, chatModel, candidatesXML, docTitle, chunks, lang)
|
||||
}()
|
||||
wg.Wait()
|
||||
|
||||
// Merge citations back into the item structs (non-failing; items without
|
||||
// citations simply keep their Description+Details fallback).
|
||||
var uncited int
|
||||
extractedEntities, extractedConcepts, uncited = mergeCitationsIntoItems(extractedEntities, extractedConcepts, citations, newSlugs)
|
||||
|
||||
// Rebuild slugItems so stale entries (for slugs that did not survive the
|
||||
// merge) and brand-new slugs discovered by the citation pass are both
|
||||
// reflected in summaryExtractedPages tracking.
|
||||
slugItems = make(map[string]extractedItem, len(extractedEntities)+len(extractedConcepts))
|
||||
for _, item := range extractedEntities {
|
||||
if item.Slug != "" && item.Name != "" {
|
||||
slugItems[item.Slug] = item
|
||||
}
|
||||
}
|
||||
for _, item := range extractedConcepts {
|
||||
if item.Slug != "" && item.Name != "" {
|
||||
slugItems[item.Slug] = item
|
||||
}
|
||||
}
|
||||
|
||||
extractedPages := make([]string, 0, len(slugItems)+1)
|
||||
for slug := range slugItems {
|
||||
extractedPages = append(extractedPages, slug)
|
||||
}
|
||||
|
||||
// Count total distinct chunks cited across all slugs for logging.
|
||||
citedChunkSet := make(map[string]bool)
|
||||
for _, ids := range citations {
|
||||
for _, id := range ids {
|
||||
citedChunkSet[id] = true
|
||||
}
|
||||
}
|
||||
|
||||
var updates []SlugUpdate
|
||||
// docSummaryLine is the one-sentence headline used for terse log/audit
|
||||
// previews and for <document_added> blocks in retract prompts.
|
||||
// docSummary is the full summary body attached to each entity/concept
|
||||
// update so the editor model gets rich framing in <source_context>.
|
||||
var docSummaryLine string
|
||||
var docSummary string
|
||||
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "wiki ingest: generate summary failed for %s: %v", knowledgeID, err)
|
||||
if summaryErr != nil {
|
||||
logger.Errorf(ctx, "wiki ingest: generate summary failed for %s: %v", knowledgeID, summaryErr)
|
||||
} else {
|
||||
sumLine, sumBody := splitSummaryLine(summaryContent)
|
||||
if sumBody == "" {
|
||||
@@ -507,6 +602,10 @@ func (s *wikiIngestService) mapOneDocument(
|
||||
sumLine = docTitle
|
||||
}
|
||||
docSummaryLine = sumLine
|
||||
docSummary = sumBody
|
||||
if strings.TrimSpace(docSummary) == "" {
|
||||
docSummary = sumLine
|
||||
}
|
||||
updates = append(updates, SlugUpdate{
|
||||
Slug: summarySlug,
|
||||
Type: types.WikiPageTypeSummary,
|
||||
@@ -524,13 +623,15 @@ func (s *wikiIngestService) mapOneDocument(
|
||||
for _, item := range extractedEntities {
|
||||
if item.Slug != "" {
|
||||
updates = append(updates, SlugUpdate{
|
||||
Slug: item.Slug,
|
||||
Type: types.WikiPageTypeEntity,
|
||||
Item: item,
|
||||
DocTitle: docTitle,
|
||||
KnowledgeID: knowledgeID,
|
||||
SourceRef: sourceRef,
|
||||
Language: lang,
|
||||
Slug: item.Slug,
|
||||
Type: types.WikiPageTypeEntity,
|
||||
Item: item,
|
||||
DocTitle: docTitle,
|
||||
KnowledgeID: knowledgeID,
|
||||
SourceRef: sourceRef,
|
||||
Language: lang,
|
||||
SourceChunks: item.SourceChunks,
|
||||
DocSummary: docSummary,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -539,13 +640,15 @@ func (s *wikiIngestService) mapOneDocument(
|
||||
for _, item := range extractedConcepts {
|
||||
if item.Slug != "" {
|
||||
updates = append(updates, SlugUpdate{
|
||||
Slug: item.Slug,
|
||||
Type: types.WikiPageTypeConcept,
|
||||
Item: item,
|
||||
DocTitle: docTitle,
|
||||
KnowledgeID: knowledgeID,
|
||||
SourceRef: sourceRef,
|
||||
Language: lang,
|
||||
Slug: item.Slug,
|
||||
Type: types.WikiPageTypeConcept,
|
||||
Item: item,
|
||||
DocTitle: docTitle,
|
||||
KnowledgeID: knowledgeID,
|
||||
SourceRef: sourceRef,
|
||||
Language: lang,
|
||||
SourceChunks: item.SourceChunks,
|
||||
DocSummary: docSummary,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -571,8 +674,13 @@ func (s *wikiIngestService) mapOneDocument(
|
||||
}
|
||||
}
|
||||
|
||||
logger.Infof(ctx, "wiki ingest: mapped knowledge %s title=%q generated_updates=%d elapsed=%s",
|
||||
knowledgeID, previewText(docTitle, 80), len(updates), time.Since(docStartedAt).Round(time.Millisecond))
|
||||
logger.Infof(ctx,
|
||||
"wiki ingest: mapped knowledge %s title=%q candidates=%d chunks=%d batches=%d cited_chunks=%d uncited_slugs=%d new_slugs=%d updates=%d pass0_fallback=%v elapsed=%s",
|
||||
knowledgeID, previewText(docTitle, 80),
|
||||
len(slugItems), len(chunks), batchCount, len(citedChunkSet), uncited, len(newSlugs),
|
||||
len(updates), pass0Failed,
|
||||
time.Since(docStartedAt).Round(time.Millisecond),
|
||||
)
|
||||
|
||||
return &docIngestResult{
|
||||
KnowledgeID: knowledgeID,
|
||||
@@ -716,6 +824,11 @@ func (s *wikiIngestService) reduceSlugUpdates(
|
||||
page.Summary = summaryUpdate.SummaryLine
|
||||
page.PageType = types.WikiPageTypeSummary
|
||||
page.SourceRefs = appendUnique(page.SourceRefs, summaryUpdate.SourceRef)
|
||||
// Summary pages don't carry chunk-level citations (they are document-
|
||||
// level synopses generated from the whole content). Clear any stale
|
||||
// chunk refs that may remain if this slug was once an entity page
|
||||
// and got converted to a summary page.
|
||||
page.ChunkRefs = types.StringArray{}
|
||||
changed = true
|
||||
|
||||
if exists {
|
||||
@@ -786,9 +899,38 @@ func (s *wikiIngestService) reduceSlugUpdates(
|
||||
|
||||
if len(additions) > 0 {
|
||||
language = additions[0].Language
|
||||
|
||||
// Resolve SourceChunks → chunk contents in a single batched query per
|
||||
// knowledge ID, so the <new_information> block can quote the chunks
|
||||
// verbatim instead of relying on the short Details paraphrase.
|
||||
chunkContentByID := s.resolveCitedChunks(ctx, tenantID, additions)
|
||||
|
||||
for _, add := range additions {
|
||||
fmt.Fprintf(&newContentBuilder, "<document>\n<title>%s</title>\n<content>\n**%s**: %s\n\n%s\n</content>\n</document>\n\n",
|
||||
add.DocTitle, add.Item.Name, add.Item.Description, add.Item.Details)
|
||||
cited := collectCitedChunkContent(add.SourceChunks, chunkContentByID)
|
||||
// Frame the chunks with the document-level summary body so the
|
||||
// editor model knows BOTH what the document is about AND what
|
||||
// kind of document it is (resume vs announcement vs product
|
||||
// page vs schedule). The one-sentence headline alone was too
|
||||
// terse to keep the editor grounded on longer or multi-topic
|
||||
// source documents, and calibrating tone (self-reported vs
|
||||
// third-party authoritative) benefits from the richer context.
|
||||
sourceCtx := strings.TrimSpace(add.DocSummary)
|
||||
sourceCtxBlock := ""
|
||||
if sourceCtx != "" {
|
||||
sourceCtxBlock = fmt.Sprintf("<source_context>\n%s\n</source_context>\n", sourceCtx)
|
||||
}
|
||||
if cited != "" {
|
||||
fmt.Fprintf(&newContentBuilder,
|
||||
"<document>\n<title>%s</title>\n%s<content>\n**%s**: %s\n\n%s\n</content>\n</document>\n\n",
|
||||
add.DocTitle, sourceCtxBlock, add.Item.Name, add.Item.Description, cited)
|
||||
} else {
|
||||
// Fallback: no citations available (legacy path, citation pass
|
||||
// failed, or bad chunk IDs were filtered out) — stick with
|
||||
// the short Details summary so the page still gets real text.
|
||||
fmt.Fprintf(&newContentBuilder,
|
||||
"<document>\n<title>%s</title>\n%s<content>\n**%s**: %s\n\n%s\n</content>\n</document>\n\n",
|
||||
add.DocTitle, sourceCtxBlock, add.Item.Name, add.Item.Description, add.Item.Details)
|
||||
}
|
||||
docTitles = appendUnique(docTitles, add.DocTitle)
|
||||
|
||||
for _, alias := range add.Item.Aliases {
|
||||
@@ -872,6 +1014,11 @@ func (s *wikiIngestService) reduceSlugUpdates(
|
||||
}
|
||||
|
||||
if changed {
|
||||
// Refresh chunk refs in-place on the page so they persist alongside
|
||||
// the rest of the row. Retract-only updates (no additions) preserve
|
||||
// the existing refs; addition rounds append the newly-cited chunks
|
||||
// on top of what was already there, deduplicated.
|
||||
page.ChunkRefs = mergeChunkRefs(page.ChunkRefs, additions)
|
||||
if exists {
|
||||
_, err = s.wikiService.UpdatePage(ctx, page)
|
||||
} else {
|
||||
@@ -882,3 +1029,34 @@ func (s *wikiIngestService) reduceSlugUpdates(
|
||||
|
||||
return false, "", nil
|
||||
}
|
||||
|
||||
// mergeChunkRefs unions the chunk IDs currently on the page with the ones
|
||||
// cited by this batch's additions, preserving insertion order and dropping
|
||||
// duplicates. Empty strings are filtered out so a malformed source_chunks
|
||||
// array can't leave junk in the column.
|
||||
//
|
||||
// A retract round with no additions leaves the current refs untouched —
|
||||
// retract-only paths don't carry chunk IDs (only knowledge IDs), and we
|
||||
// can't surgically filter without that info. The next time the slug is
|
||||
// re-materialized via additions the fresh chunks will overlay on top.
|
||||
func mergeChunkRefs(current types.StringArray, additions []SlugUpdate) types.StringArray {
|
||||
seen := make(map[string]bool, len(current))
|
||||
out := make(types.StringArray, 0, len(current))
|
||||
for _, id := range current {
|
||||
if id == "" || seen[id] {
|
||||
continue
|
||||
}
|
||||
seen[id] = true
|
||||
out = append(out, id)
|
||||
}
|
||||
for _, add := range additions {
|
||||
for _, chunkID := range add.SourceChunks {
|
||||
if chunkID == "" || seen[chunkID] {
|
||||
continue
|
||||
}
|
||||
seen[chunkID] = true
|
||||
out = append(out, chunkID)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -0,0 +1,566 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/agent"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/models/chat"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"golang.org/x/sync/errgroup"
|
||||
)
|
||||
|
||||
const (
|
||||
// maxRunesPerCitationBatch bounds the size of a single chunk-citation
|
||||
// batch by rune count (a fast approximation of token count). Small enough
|
||||
// that batches stay comfortably inside the LLM's context and output
|
||||
// budgets, large enough that most short/medium documents classify in a
|
||||
// single batch. Tune via benchmarks if needed.
|
||||
maxRunesPerCitationBatch = 12000
|
||||
|
||||
// maxCitationBatchConcurrency limits parallelism across chunk-citation
|
||||
// batches so a single long document can't saturate the synthesis model.
|
||||
maxCitationBatchConcurrency = 4
|
||||
)
|
||||
|
||||
// citationBatchResult is the JSON shape we expect back from one invocation of
|
||||
// WikiChunkCitationPrompt.
|
||||
type citationBatchResult struct {
|
||||
Citations map[string][]string `json:"citations"`
|
||||
NewSlugs []newSlugFromCitation `json:"new_slugs"`
|
||||
}
|
||||
|
||||
// newSlugFromCitation is the shape of an entry in the "new_slugs" array of
|
||||
// WikiChunkCitationPrompt. Mirrors extractedItem but also carries a "type"
|
||||
// tag because this prompt emits entities and concepts in a single array.
|
||||
type newSlugFromCitation struct {
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Slug string `json:"slug"`
|
||||
Aliases []string `json:"aliases"`
|
||||
Description string `json:"description"`
|
||||
Details string `json:"details"`
|
||||
SourceChunks []string `json:"source_chunks"`
|
||||
}
|
||||
|
||||
// citationPipelineOutcome carries the raw numbers produced by the Pass
|
||||
// 0 + classification flow so callers (mapOneDocument) can log a single
|
||||
// unified stat line.
|
||||
type citationPipelineOutcome struct {
|
||||
CandidateCount int
|
||||
ChunkCount int
|
||||
BatchCount int
|
||||
CitedChunks int
|
||||
UncitedSlugs int
|
||||
NewSlugCount int
|
||||
}
|
||||
|
||||
// extractCandidateSlugs runs Pass 0 of the chunk-cited pipeline: it scans the
|
||||
// full (reconstructed) document text and returns a lightweight skeleton of
|
||||
// every significant entity/concept. Unlike the legacy single-shot extraction,
|
||||
// this pass explicitly does NOT ask the LLM to paraphrase full facts per item;
|
||||
// those will come from the chunk-citation pass instead.
|
||||
//
|
||||
// Returns (entities, concepts, slugItems, error). On LLM or parse failure it
|
||||
// returns an error — the caller can then fall back to the legacy extractor.
|
||||
func (s *wikiIngestService) extractCandidateSlugs(
|
||||
ctx context.Context,
|
||||
chatModel chat.Chat,
|
||||
content, docTitle, lang string,
|
||||
oldPageSlugs map[string]bool,
|
||||
batchCtx *WikiBatchContext,
|
||||
) ([]extractedItem, []extractedItem, map[string]extractedItem, error) {
|
||||
var prevSlugsText string
|
||||
if len(oldPageSlugs) > 0 {
|
||||
var sb strings.Builder
|
||||
for slug := range oldPageSlugs {
|
||||
if !strings.HasPrefix(slug, "entity/") && !strings.HasPrefix(slug, "concept/") {
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&sb, "- %s\n", slug)
|
||||
}
|
||||
prevSlugsText = sb.String()
|
||||
}
|
||||
if prevSlugsText == "" {
|
||||
prevSlugsText = "(none — this is a new document)"
|
||||
}
|
||||
|
||||
granularity := batchCtx.ExtractionGranularity.Normalize()
|
||||
raw, err := s.generateWithTemplate(ctx, chatModel, agent.WikiCandidateSlugPrompt, map[string]string{
|
||||
"Title": docTitle,
|
||||
"Content": content,
|
||||
"Language": lang,
|
||||
"PreviousSlugs": prevSlugsText,
|
||||
"Granularity": string(granularity),
|
||||
"GranularityGuidance": agent.WikiGranularityGuidance(string(granularity)),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, nil, nil, fmt.Errorf("candidate slug extraction failed: %w", err)
|
||||
}
|
||||
|
||||
raw = cleanLLMJSON(raw)
|
||||
|
||||
var result combinedExtraction
|
||||
if err := json.Unmarshal([]byte(raw), &result); err != nil {
|
||||
logger.Warnf(ctx, "wiki ingest: failed to parse candidate slug JSON: %v\nRaw: %s", err, raw)
|
||||
return nil, nil, nil, fmt.Errorf("parse candidate slug JSON: %w", err)
|
||||
}
|
||||
|
||||
result.Entities, result.Concepts = s.deduplicateExtractedBatch(
|
||||
ctx, chatModel, result.Entities, result.Concepts, batchCtx.AllPages,
|
||||
)
|
||||
|
||||
slugItems := make(map[string]extractedItem, len(result.Entities)+len(result.Concepts))
|
||||
for _, item := range result.Entities {
|
||||
if item.Slug != "" && item.Name != "" {
|
||||
slugItems[item.Slug] = item
|
||||
}
|
||||
}
|
||||
for _, item := range result.Concepts {
|
||||
if item.Slug != "" && item.Name != "" {
|
||||
slugItems[item.Slug] = item
|
||||
}
|
||||
}
|
||||
|
||||
return result.Entities, result.Concepts, slugItems, nil
|
||||
}
|
||||
|
||||
// chunkBatch groups chunks that will be sent in a single WikiChunkCitationPrompt call.
|
||||
type chunkBatch struct {
|
||||
chunks []*types.Chunk
|
||||
aliasToID map[string]string
|
||||
totalRuneLen int
|
||||
}
|
||||
|
||||
// splitChunksIntoCitationBatches partitions chunks into batches whose total
|
||||
// rune count stays under maxRunesPerCitationBatch. Chunk order (by ChunkIndex)
|
||||
// is preserved, and a chunk that by itself exceeds the budget occupies its
|
||||
// own batch so we never silently drop content. Each chunk is assigned a short
|
||||
// alias ("c000", "c001", ...) that the prompt uses in place of the raw UUID;
|
||||
// the alias → real ID map is kept on the batch so the caller can translate
|
||||
// LLM responses back to stable chunk IDs.
|
||||
func splitChunksIntoCitationBatches(chunks []*types.Chunk) []chunkBatch {
|
||||
// Only cite text chunks — image/ocr chunks are already merged into the
|
||||
// text content via reconstructEnrichedContent and the LLM doesn't see
|
||||
// them as standalone units.
|
||||
filtered := make([]*types.Chunk, 0, len(chunks))
|
||||
for _, c := range chunks {
|
||||
if c == nil || c.Content == "" {
|
||||
continue
|
||||
}
|
||||
if c.ChunkType != types.ChunkTypeText && c.ChunkType != "" {
|
||||
continue
|
||||
}
|
||||
filtered = append(filtered, c)
|
||||
}
|
||||
if len(filtered) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Preserve document order for human-readable citation
|
||||
sort.Slice(filtered, func(i, j int) bool {
|
||||
if filtered[i].ChunkIndex == filtered[j].ChunkIndex {
|
||||
return filtered[i].StartAt < filtered[j].StartAt
|
||||
}
|
||||
return filtered[i].ChunkIndex < filtered[j].ChunkIndex
|
||||
})
|
||||
|
||||
var batches []chunkBatch
|
||||
current := chunkBatch{aliasToID: make(map[string]string)}
|
||||
aliasCounter := 0
|
||||
|
||||
flush := func() {
|
||||
if len(current.chunks) == 0 {
|
||||
return
|
||||
}
|
||||
batches = append(batches, current)
|
||||
current = chunkBatch{aliasToID: make(map[string]string)}
|
||||
}
|
||||
|
||||
for _, c := range filtered {
|
||||
runeLen := len([]rune(c.Content))
|
||||
// If adding this chunk would exceed the budget and the current batch
|
||||
// isn't empty, flush first so the oversized chunk is not merged with
|
||||
// already-queued ones. Oversized chunks still go through — they just
|
||||
// get their own batch.
|
||||
if len(current.chunks) > 0 && current.totalRuneLen+runeLen > maxRunesPerCitationBatch {
|
||||
flush()
|
||||
}
|
||||
|
||||
alias := fmt.Sprintf("c%03d", aliasCounter)
|
||||
aliasCounter++
|
||||
current.aliasToID[alias] = c.ID
|
||||
current.chunks = append(current.chunks, c)
|
||||
current.totalRuneLen += runeLen
|
||||
}
|
||||
flush()
|
||||
|
||||
// Restart alias numbering per batch so the LLM sees a small, local ID
|
||||
// space (helps it follow exact-match citation rules). We do this by
|
||||
// rebuilding the alias map in a second pass.
|
||||
for bi := range batches {
|
||||
newAliasToID := make(map[string]string, len(batches[bi].chunks))
|
||||
for idx, c := range batches[bi].chunks {
|
||||
alias := fmt.Sprintf("c%03d", idx)
|
||||
newAliasToID[alias] = c.ID
|
||||
}
|
||||
batches[bi].aliasToID = newAliasToID
|
||||
}
|
||||
|
||||
return batches
|
||||
}
|
||||
|
||||
// renderCandidateSlugsXML renders candidate slugs as a compact list suitable
|
||||
// for the prompt's <candidate_slugs> block.
|
||||
func renderCandidateSlugsXML(entities, concepts []extractedItem) string {
|
||||
var sb strings.Builder
|
||||
write := func(item extractedItem, kind string) {
|
||||
aliases := ""
|
||||
if len(item.Aliases) > 0 {
|
||||
aliases = fmt.Sprintf(" aliases=%q", strings.Join(item.Aliases, ", "))
|
||||
}
|
||||
fmt.Fprintf(&sb, "- slug: %s, type: %s, name: %q%s, description: %s\n",
|
||||
item.Slug, kind, item.Name, aliases, item.Description)
|
||||
}
|
||||
for _, item := range entities {
|
||||
if item.Slug == "" || item.Name == "" {
|
||||
continue
|
||||
}
|
||||
write(item, "entity")
|
||||
}
|
||||
for _, item := range concepts {
|
||||
if item.Slug == "" || item.Name == "" {
|
||||
continue
|
||||
}
|
||||
write(item, "concept")
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// renderChunksXML formats one batch's chunks into the <chunks> block, using
|
||||
// per-batch aliases (c000, c001, ...) instead of raw UUIDs.
|
||||
func renderChunksXML(batch chunkBatch) string {
|
||||
aliases := make([]string, 0, len(batch.aliasToID))
|
||||
for a := range batch.aliasToID {
|
||||
aliases = append(aliases, a)
|
||||
}
|
||||
sort.Strings(aliases)
|
||||
|
||||
idToAlias := make(map[string]string, len(batch.aliasToID))
|
||||
for a, id := range batch.aliasToID {
|
||||
idToAlias[id] = a
|
||||
}
|
||||
|
||||
var sb strings.Builder
|
||||
for _, c := range batch.chunks {
|
||||
alias := idToAlias[c.ID]
|
||||
fmt.Fprintf(&sb, "<c id=%q index=\"%d\">\n%s\n</c>\n", alias, c.ChunkIndex, c.Content)
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// classifyChunkCitations runs Pass 1..N of the chunk-cited pipeline: given a
|
||||
// set of candidate slugs (from Pass 0) and the document's chunks, it asks the
|
||||
// LLM which chunks substantively discuss each candidate. Results across
|
||||
// batches are merged into a single slug → union(chunk_id) map, and any
|
||||
// "new_slugs" that Pass 0 missed are collected separately.
|
||||
//
|
||||
// Returns (citations, newSlugs, batchCount). citations is keyed by slug and
|
||||
// contains real chunk UUIDs (already translated from batch aliases). newSlugs
|
||||
// likewise carry real chunk UUIDs in SourceChunks.
|
||||
func (s *wikiIngestService) classifyChunkCitations(
|
||||
ctx context.Context,
|
||||
chatModel chat.Chat,
|
||||
candidatesXML string,
|
||||
docTitle string,
|
||||
chunks []*types.Chunk,
|
||||
lang string,
|
||||
) (map[string][]string, []newSlugFromCitation, int) {
|
||||
batches := splitChunksIntoCitationBatches(chunks)
|
||||
if len(batches) == 0 || strings.TrimSpace(candidatesXML) == "" {
|
||||
return map[string][]string{}, nil, 0
|
||||
}
|
||||
|
||||
// Merge state. Using sets keyed by (slug, chunkID) to dedup across
|
||||
// batches; order is re-imposed from chunk ChunkIndex at the end.
|
||||
var mu sync.Mutex
|
||||
citationSet := make(map[string]map[string]bool) // slug → set of real chunk IDs
|
||||
var newSlugsAll []newSlugFromCitation
|
||||
|
||||
eg, ectx := errgroup.WithContext(ctx)
|
||||
eg.SetLimit(maxCitationBatchConcurrency)
|
||||
|
||||
for bi := range batches {
|
||||
batch := batches[bi]
|
||||
batchIdx := bi
|
||||
eg.Go(func() error {
|
||||
chunksXML := renderChunksXML(batch)
|
||||
raw, err := s.generateWithTemplate(ectx, chatModel, agent.WikiChunkCitationPrompt, map[string]string{
|
||||
"DocTitle": docTitle,
|
||||
"CandidateSlugs": candidatesXML,
|
||||
"ChunksXML": chunksXML,
|
||||
"Language": lang,
|
||||
})
|
||||
if err != nil {
|
||||
logger.Warnf(ectx, "wiki ingest: citation batch %d failed: %v", batchIdx, err)
|
||||
return nil // don't abort peer batches
|
||||
}
|
||||
raw = cleanLLMJSON(raw)
|
||||
|
||||
var parsed citationBatchResult
|
||||
if jerr := json.Unmarshal([]byte(raw), &parsed); jerr != nil {
|
||||
logger.Warnf(ectx, "wiki ingest: citation batch %d parse failed: %v\nRaw: %s", batchIdx, jerr, raw)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Translate aliases → real chunk UUIDs; drop unknown aliases.
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
|
||||
for slug, aliasList := range parsed.Citations {
|
||||
if slug == "" {
|
||||
continue
|
||||
}
|
||||
set, ok := citationSet[slug]
|
||||
if !ok {
|
||||
set = make(map[string]bool)
|
||||
citationSet[slug] = set
|
||||
}
|
||||
for _, alias := range aliasList {
|
||||
realID, known := batch.aliasToID[alias]
|
||||
if !known {
|
||||
logger.Warnf(ectx, "wiki ingest: citation batch %d referenced unknown chunk alias %q for slug %s", batchIdx, alias, slug)
|
||||
continue
|
||||
}
|
||||
set[realID] = true
|
||||
}
|
||||
}
|
||||
|
||||
for _, ns := range parsed.NewSlugs {
|
||||
if ns.Slug == "" || ns.Name == "" {
|
||||
continue
|
||||
}
|
||||
real := make([]string, 0, len(ns.SourceChunks))
|
||||
for _, alias := range ns.SourceChunks {
|
||||
if id, ok := batch.aliasToID[alias]; ok {
|
||||
real = append(real, id)
|
||||
}
|
||||
}
|
||||
ns.SourceChunks = real
|
||||
newSlugsAll = append(newSlugsAll, ns)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
_ = eg.Wait()
|
||||
|
||||
// Build a stable chunk-order so the final citations come out in document order.
|
||||
chunkOrder := make(map[string]int, len(chunks))
|
||||
for _, c := range chunks {
|
||||
chunkOrder[c.ID] = c.ChunkIndex
|
||||
}
|
||||
|
||||
out := make(map[string][]string, len(citationSet))
|
||||
for slug, set := range citationSet {
|
||||
ids := make([]string, 0, len(set))
|
||||
for id := range set {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.SliceStable(ids, func(i, j int) bool {
|
||||
return chunkOrder[ids[i]] < chunkOrder[ids[j]]
|
||||
})
|
||||
out[slug] = ids
|
||||
}
|
||||
|
||||
return out, newSlugsAll, len(batches)
|
||||
}
|
||||
|
||||
// resolveCitedChunks loads the content of every chunk referenced by the
|
||||
// given additions in a single batched query per knowledge ID, returning a
|
||||
// map[chunkID]content. Missing / out-of-tenant chunk IDs are silently skipped
|
||||
// (logged at warn level) so the Reduce phase can gracefully fall back to the
|
||||
// Details paraphrase.
|
||||
func (s *wikiIngestService) resolveCitedChunks(
|
||||
ctx context.Context,
|
||||
tenantID uint64,
|
||||
additions []SlugUpdate,
|
||||
) map[string]string {
|
||||
// Group chunk IDs by knowledge ID — chunk repo queries are scoped by
|
||||
// tenant but not by knowledge, so we could fetch everything in one go.
|
||||
// We batch per knowledge anyway to keep the IN(...) list bounded for
|
||||
// large multi-knowledge reduces.
|
||||
byKnowledge := make(map[string]map[string]bool)
|
||||
for _, add := range additions {
|
||||
if add.KnowledgeID == "" {
|
||||
continue
|
||||
}
|
||||
for _, chunkID := range add.SourceChunks {
|
||||
if chunkID == "" {
|
||||
continue
|
||||
}
|
||||
set, ok := byKnowledge[add.KnowledgeID]
|
||||
if !ok {
|
||||
set = make(map[string]bool)
|
||||
byKnowledge[add.KnowledgeID] = set
|
||||
}
|
||||
set[chunkID] = true
|
||||
}
|
||||
}
|
||||
if len(byKnowledge) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
out := make(map[string]string)
|
||||
for _, idSet := range byKnowledge {
|
||||
ids := make([]string, 0, len(idSet))
|
||||
for id := range idSet {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
continue
|
||||
}
|
||||
chunks, err := s.chunkRepo.ListChunksByID(ctx, tenantID, ids)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki ingest: failed to resolve cited chunks: %v", err)
|
||||
continue
|
||||
}
|
||||
for _, c := range chunks {
|
||||
if c == nil || c.Content == "" {
|
||||
continue
|
||||
}
|
||||
out[c.ID] = c.Content
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// collectCitedChunkContent materializes the verbatim content of every
|
||||
// referenced chunk, concatenated in the order provided. Chunk IDs that can't
|
||||
// be resolved are silently dropped (logged upstream).
|
||||
func collectCitedChunkContent(chunkIDs []string, contentByID map[string]string) string {
|
||||
if len(chunkIDs) == 0 || len(contentByID) == 0 {
|
||||
return ""
|
||||
}
|
||||
var sb strings.Builder
|
||||
for _, id := range chunkIDs {
|
||||
content, ok := contentByID[id]
|
||||
if !ok || strings.TrimSpace(content) == "" {
|
||||
continue
|
||||
}
|
||||
if sb.Len() > 0 {
|
||||
sb.WriteString("\n\n")
|
||||
}
|
||||
sb.WriteString(content)
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// mergeCitationsIntoItems backfills SourceChunks on every extractedItem from
|
||||
// the citation map, and augments the item slices with any genuinely new slugs
|
||||
// the citation pass discovered. Items whose slug is not in citations are left
|
||||
// untouched — Reduce will fall back to Description/Details for them.
|
||||
//
|
||||
// Returns the updated entity and concept slices plus a count of candidates
|
||||
// that ended with zero citations (for observability).
|
||||
func mergeCitationsIntoItems(
|
||||
entities, concepts []extractedItem,
|
||||
citations map[string][]string,
|
||||
newSlugs []newSlugFromCitation,
|
||||
) ([]extractedItem, []extractedItem, int) {
|
||||
uncited := 0
|
||||
|
||||
for i := range entities {
|
||||
ids := citations[entities[i].Slug]
|
||||
entities[i].SourceChunks = ids
|
||||
if len(ids) == 0 {
|
||||
uncited++
|
||||
}
|
||||
}
|
||||
for i := range concepts {
|
||||
ids := citations[concepts[i].Slug]
|
||||
concepts[i].SourceChunks = ids
|
||||
if len(ids) == 0 {
|
||||
uncited++
|
||||
}
|
||||
}
|
||||
|
||||
// Append new_slugs discovered by the citation pass, avoiding duplicates.
|
||||
existingSlugs := make(map[string]bool, len(entities)+len(concepts))
|
||||
for _, e := range entities {
|
||||
existingSlugs[e.Slug] = true
|
||||
}
|
||||
for _, c := range concepts {
|
||||
existingSlugs[c.Slug] = true
|
||||
}
|
||||
|
||||
// Aggregate per-slug across batches so the same "newly discovered" slug
|
||||
// surfacing in multiple batches merges into a single item.
|
||||
type mergedNew struct {
|
||||
item extractedItem
|
||||
typ string
|
||||
}
|
||||
merged := make(map[string]*mergedNew)
|
||||
slugOrder := []string{}
|
||||
|
||||
for _, ns := range newSlugs {
|
||||
if ns.Slug == "" || ns.Name == "" {
|
||||
continue
|
||||
}
|
||||
if existingSlugs[ns.Slug] {
|
||||
// Treat as citation for existing candidate
|
||||
continue
|
||||
}
|
||||
kind := strings.TrimSpace(strings.ToLower(ns.Type))
|
||||
if kind == "" {
|
||||
if strings.HasPrefix(ns.Slug, "concept/") {
|
||||
kind = "concept"
|
||||
} else {
|
||||
kind = "entity"
|
||||
}
|
||||
}
|
||||
existing, ok := merged[ns.Slug]
|
||||
if !ok {
|
||||
existing = &mergedNew{
|
||||
item: extractedItem{
|
||||
Name: ns.Name,
|
||||
Slug: ns.Slug,
|
||||
Aliases: append([]string(nil), ns.Aliases...),
|
||||
Description: ns.Description,
|
||||
Details: ns.Details,
|
||||
SourceChunks: append([]string(nil), ns.SourceChunks...),
|
||||
},
|
||||
typ: kind,
|
||||
}
|
||||
merged[ns.Slug] = existing
|
||||
slugOrder = append(slugOrder, ns.Slug)
|
||||
continue
|
||||
}
|
||||
// Union source chunks
|
||||
seen := make(map[string]bool, len(existing.item.SourceChunks))
|
||||
for _, id := range existing.item.SourceChunks {
|
||||
seen[id] = true
|
||||
}
|
||||
for _, id := range ns.SourceChunks {
|
||||
if !seen[id] {
|
||||
existing.item.SourceChunks = append(existing.item.SourceChunks, id)
|
||||
seen[id] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for _, slug := range slugOrder {
|
||||
m := merged[slug]
|
||||
if m.typ == "concept" {
|
||||
concepts = append(concepts, m.item)
|
||||
} else {
|
||||
entities = append(entities, m.item)
|
||||
}
|
||||
}
|
||||
|
||||
return entities, concepts, uncited
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// TestMergeCitationsIntoItems_PopulatesSourceChunksOnCandidates verifies that
|
||||
// citations returned by the chunk-classification pass are attached back onto
|
||||
// the matching candidate items while non-cited candidates are left untouched.
|
||||
func TestMergeCitationsIntoItems_PopulatesSourceChunksOnCandidates(t *testing.T) {
|
||||
entities := []extractedItem{
|
||||
{Name: "Acme", Slug: "entity/acme"},
|
||||
{Name: "Beta", Slug: "entity/beta"},
|
||||
}
|
||||
concepts := []extractedItem{
|
||||
{Name: "RAG", Slug: "concept/rag"},
|
||||
}
|
||||
citations := map[string][]string{
|
||||
"entity/acme": {"chunk-1", "chunk-3"},
|
||||
"concept/rag": {"chunk-2"},
|
||||
}
|
||||
|
||||
gotE, gotC, uncited := mergeCitationsIntoItems(entities, concepts, citations, nil)
|
||||
|
||||
if len(gotE) != 2 || len(gotC) != 1 {
|
||||
t.Fatalf("expected 2 entities + 1 concept, got %d + %d", len(gotE), len(gotC))
|
||||
}
|
||||
acme := findBySlug(gotE, "entity/acme")
|
||||
if acme == nil {
|
||||
t.Fatalf("entity/acme missing from result")
|
||||
}
|
||||
if !equalStrings(acme.SourceChunks, []string{"chunk-1", "chunk-3"}) {
|
||||
t.Errorf("entity/acme source_chunks = %v, want [chunk-1 chunk-3]", acme.SourceChunks)
|
||||
}
|
||||
beta := findBySlug(gotE, "entity/beta")
|
||||
if beta == nil {
|
||||
t.Fatalf("entity/beta missing from result")
|
||||
}
|
||||
if len(beta.SourceChunks) != 0 {
|
||||
t.Errorf("entity/beta should have no citations, got %v", beta.SourceChunks)
|
||||
}
|
||||
rag := findBySlug(gotC, "concept/rag")
|
||||
if rag == nil {
|
||||
t.Fatalf("concept/rag missing")
|
||||
}
|
||||
if !equalStrings(rag.SourceChunks, []string{"chunk-2"}) {
|
||||
t.Errorf("concept/rag source_chunks = %v, want [chunk-2]", rag.SourceChunks)
|
||||
}
|
||||
if uncited != 1 {
|
||||
t.Errorf("uncited = %d, want 1", uncited)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMergeCitationsIntoItems_AddsNewSlugsAndUnionsChunksAcrossBatches checks
|
||||
// that genuinely new slugs (ones Pass 0 missed) are appended to the right
|
||||
// type slice, and that a slug surfacing in two batches ends up with the union
|
||||
// of its source chunks.
|
||||
func TestMergeCitationsIntoItems_AddsNewSlugsAndUnionsChunksAcrossBatches(t *testing.T) {
|
||||
entities := []extractedItem{
|
||||
{Name: "Known", Slug: "entity/known"},
|
||||
}
|
||||
concepts := []extractedItem{}
|
||||
|
||||
newSlugs := []newSlugFromCitation{
|
||||
{
|
||||
Type: "entity",
|
||||
Name: "Fresh Entity",
|
||||
Slug: "entity/fresh",
|
||||
Description: "desc",
|
||||
Details: "details",
|
||||
SourceChunks: []string{"c001", "c002"},
|
||||
},
|
||||
{
|
||||
// Same slug as above, appears in another batch — must union.
|
||||
Type: "entity",
|
||||
Name: "Fresh Entity",
|
||||
Slug: "entity/fresh",
|
||||
SourceChunks: []string{"c002", "c003"},
|
||||
},
|
||||
{
|
||||
Type: "concept",
|
||||
Name: "New Concept",
|
||||
Slug: "concept/new-concept",
|
||||
SourceChunks: []string{"c010"},
|
||||
},
|
||||
{
|
||||
// Duplicate of an existing candidate — should NOT produce a
|
||||
// duplicate entry (Known already exists in `entities`).
|
||||
Type: "entity",
|
||||
Name: "Known",
|
||||
Slug: "entity/known",
|
||||
SourceChunks: []string{"c020"},
|
||||
},
|
||||
}
|
||||
|
||||
gotE, gotC, _ := mergeCitationsIntoItems(entities, concepts, nil, newSlugs)
|
||||
|
||||
if len(gotE) != 2 {
|
||||
t.Fatalf("expected 2 entities, got %d (%+v)", len(gotE), gotE)
|
||||
}
|
||||
if len(gotC) != 1 {
|
||||
t.Fatalf("expected 1 concept, got %d (%+v)", len(gotC), gotC)
|
||||
}
|
||||
fresh := findBySlug(gotE, "entity/fresh")
|
||||
if fresh == nil {
|
||||
t.Fatalf("entity/fresh missing")
|
||||
}
|
||||
sort.Strings(fresh.SourceChunks)
|
||||
if !equalStrings(fresh.SourceChunks, []string{"c001", "c002", "c003"}) {
|
||||
t.Errorf("entity/fresh source_chunks = %v, want union [c001 c002 c003]", fresh.SourceChunks)
|
||||
}
|
||||
newC := findBySlug(gotC, "concept/new-concept")
|
||||
if newC == nil || !equalStrings(newC.SourceChunks, []string{"c010"}) {
|
||||
t.Errorf("concept/new-concept missing or wrong chunks: %+v", newC)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSplitChunksIntoCitationBatches_RespectsBudgetAndOrder verifies that the
|
||||
// batcher never puts too many runes in one batch, preserves document order,
|
||||
// and that an oversized chunk gets its own batch.
|
||||
func TestSplitChunksIntoCitationBatches_RespectsBudgetAndOrder(t *testing.T) {
|
||||
// Each small chunk is 5k runes → 3 of them should fit in one batch
|
||||
// (15k > 12k limit would spill to a second batch).
|
||||
mk := func(idx int, runes int, id string) *types.Chunk {
|
||||
return &types.Chunk{
|
||||
ID: id,
|
||||
ChunkIndex: idx,
|
||||
Content: repeatRune('a', runes),
|
||||
ChunkType: types.ChunkTypeText,
|
||||
}
|
||||
}
|
||||
chunks := []*types.Chunk{
|
||||
mk(0, 5000, "id-0"),
|
||||
mk(1, 5000, "id-1"),
|
||||
mk(2, 5000, "id-2"), // this should start a new batch (15k > 12k)
|
||||
// An oversized chunk gets a dedicated batch.
|
||||
mk(3, 20000, "id-big"),
|
||||
mk(4, 1000, "id-small"),
|
||||
}
|
||||
batches := splitChunksIntoCitationBatches(chunks)
|
||||
if len(batches) < 3 {
|
||||
t.Fatalf("expected at least 3 batches, got %d", len(batches))
|
||||
}
|
||||
// All input IDs should show up in some batch, exactly once, in order.
|
||||
seen := []string{}
|
||||
for _, b := range batches {
|
||||
for _, c := range b.chunks {
|
||||
seen = append(seen, c.ID)
|
||||
}
|
||||
}
|
||||
wantOrder := []string{"id-0", "id-1", "id-2", "id-big", "id-small"}
|
||||
if !equalStrings(seen, wantOrder) {
|
||||
t.Errorf("batch order = %v, want %v", seen, wantOrder)
|
||||
}
|
||||
|
||||
// Verify alias → id map is populated per batch with unique aliases.
|
||||
for bi, b := range batches {
|
||||
if len(b.aliasToID) != len(b.chunks) {
|
||||
t.Errorf("batch %d alias count %d != chunk count %d", bi, len(b.aliasToID), len(b.chunks))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- helpers ---
|
||||
|
||||
func findBySlug(items []extractedItem, slug string) *extractedItem {
|
||||
for i := range items {
|
||||
if items[i].Slug == slug {
|
||||
return &items[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func equalStrings(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func repeatRune(r rune, n int) string {
|
||||
out := make([]rune, n)
|
||||
for i := range out {
|
||||
out[i] = r
|
||||
}
|
||||
return string(out)
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/searchutil"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// Pre-filtering candidate existing pages before the dedup LLM call.
|
||||
//
|
||||
// Without pre-filtering, the dedup prompt packs the entire entity+concept
|
||||
// page corpus into <existing_pages>. On knowledge bases with 100+ pages
|
||||
// this inflates input tokens and — more importantly — gives weaker LLMs
|
||||
// enough rope to hallucinate merges between totally unrelated slugs just
|
||||
// because the output looked plausible. Observed cases include
|
||||
// "城镇登记失业人员" → "中华优秀传统文化" (zero shared characters).
|
||||
//
|
||||
// The filter below keeps only pages that share at least some cheap
|
||||
// surface-level signal with one of the new items. Fast to compute, no
|
||||
// external calls, and it only ever *removes* candidates from the prompt —
|
||||
// the downstream validMerge check still guards the final write.
|
||||
const (
|
||||
// dedupCandidateTopK is how many existing pages we keep per new item
|
||||
// even if none pass the score floor. Guarantees the LLM sees a small
|
||||
// ordered shortlist rather than an empty list when nothing matches
|
||||
// strongly.
|
||||
dedupCandidateTopK = 20
|
||||
|
||||
// dedupCandidateScoreFloor is the Jaccard floor. Pairs at or above
|
||||
// this similarity are always included regardless of the top-K cap.
|
||||
// Tuned so that "城镇登记失业人员" vs "中华优秀传统文化" (Jaccard 0)
|
||||
// is excluded while "Acme Corp" vs "Acme Corporation" (Jaccard ≈ 0.5)
|
||||
// clearly passes.
|
||||
dedupCandidateScoreFloor = 0.08
|
||||
|
||||
// dedupSmallCorpusBypass skips pre-filtering entirely when the
|
||||
// existing-page corpus is already small enough to fit in the prompt
|
||||
// without degrading the LLM. The filter only earns its keep on large
|
||||
// KBs; on small ones it risks cutting legitimate matches with no
|
||||
// real token savings.
|
||||
dedupSmallCorpusBypass = 25
|
||||
)
|
||||
|
||||
// dedupSurface is the pre-computed similarity feature set for one side of
|
||||
// a (new item, existing page) comparison.
|
||||
type dedupSurface struct {
|
||||
// slugTokens are the kebab-case tokens from the slug base (after "/").
|
||||
// Slugs are an orthogonal signal to the surface names — Chinese pages
|
||||
// carry their pinyin here, which keeps the filter useful on purely
|
||||
// Latin-script new items too.
|
||||
slugTokens map[string]struct{}
|
||||
|
||||
// nameGramSets holds one char-bigram set per surface form (name and
|
||||
// each alias). We keep them separate so the pair score is max-over-
|
||||
// surfaces — a rare alias match shouldn't be diluted by the primary
|
||||
// name disagreeing.
|
||||
nameGramSets []map[string]struct{}
|
||||
}
|
||||
|
||||
// countEntityConceptPages returns how many of the given pages are
|
||||
// entity- or concept-typed. Used only for logging the prefilter's
|
||||
// reduction ratio.
|
||||
func countEntityConceptPages(pages []*types.WikiPage) int {
|
||||
n := 0
|
||||
for _, p := range pages {
|
||||
if p == nil {
|
||||
continue
|
||||
}
|
||||
if p.PageType == types.WikiPageTypeEntity || p.PageType == types.WikiPageTypeConcept {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// selectDedupCandidatePages returns the subset of allPages plausibly
|
||||
// related to at least one of newItems. Non-entity/concept pages are
|
||||
// dropped unconditionally. The returned slice preserves the input order
|
||||
// so the downstream prompt stays stable across runs.
|
||||
//
|
||||
// On small corpora (<= dedupSmallCorpusBypass entries) this is a no-op
|
||||
// aside from the page-type filter.
|
||||
func selectDedupCandidatePages(
|
||||
newItems []extractedItem,
|
||||
allPages []*types.WikiPage,
|
||||
) []*types.WikiPage {
|
||||
pages := make([]*types.WikiPage, 0, len(allPages))
|
||||
for _, p := range allPages {
|
||||
if p == nil {
|
||||
continue
|
||||
}
|
||||
if p.PageType != types.WikiPageTypeEntity && p.PageType != types.WikiPageTypeConcept {
|
||||
continue
|
||||
}
|
||||
pages = append(pages, p)
|
||||
}
|
||||
if len(pages) == 0 {
|
||||
return pages
|
||||
}
|
||||
if len(newItems) == 0 || len(pages) <= dedupSmallCorpusBypass {
|
||||
return pages
|
||||
}
|
||||
|
||||
pageFeats := make([]dedupSurface, len(pages))
|
||||
for i, p := range pages {
|
||||
surfaces := make([]string, 0, 1+len(p.Aliases))
|
||||
surfaces = append(surfaces, p.Title)
|
||||
surfaces = append(surfaces, []string(p.Aliases)...)
|
||||
pageFeats[i] = dedupSurface{
|
||||
slugTokens: slugBaseTokens(p.Slug),
|
||||
nameGramSets: gramsPerSurface(surfaces),
|
||||
}
|
||||
}
|
||||
|
||||
selected := make(map[int]bool, len(pages))
|
||||
for _, it := range newItems {
|
||||
surfaces := make([]string, 0, 1+len(it.Aliases))
|
||||
surfaces = append(surfaces, it.Name)
|
||||
surfaces = append(surfaces, it.Aliases...)
|
||||
itemFeat := dedupSurface{
|
||||
slugTokens: slugBaseTokens(it.Slug),
|
||||
nameGramSets: gramsPerSurface(surfaces),
|
||||
}
|
||||
if len(itemFeat.slugTokens) == 0 && len(itemFeat.nameGramSets) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
scores := make([]struct {
|
||||
idx int
|
||||
score float64
|
||||
}, len(pageFeats))
|
||||
for i := range pageFeats {
|
||||
scores[i].idx = i
|
||||
scores[i].score = dedupPairScore(itemFeat, pageFeats[i])
|
||||
}
|
||||
// Stable sort so ties break deterministically by original index.
|
||||
sort.SliceStable(scores, func(i, j int) bool {
|
||||
return scores[i].score > scores[j].score
|
||||
})
|
||||
|
||||
topKRemaining := dedupCandidateTopK
|
||||
for _, s := range scores {
|
||||
if s.score >= dedupCandidateScoreFloor {
|
||||
selected[s.idx] = true
|
||||
continue
|
||||
}
|
||||
// Below the floor but we still owe the LLM some candidates so
|
||||
// it can decline cleanly — fill the top-K budget with the
|
||||
// highest-scoring remaining pages, as long as the score isn't
|
||||
// flatly zero (a zero score means we have nothing in common
|
||||
// with this page and including it just invites hallucination).
|
||||
if topKRemaining > 0 && s.score > 0 {
|
||||
selected[s.idx] = true
|
||||
topKRemaining--
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
out := make([]*types.WikiPage, 0, len(selected))
|
||||
for i, p := range pages {
|
||||
if selected[i] {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// dedupPairScore is the max similarity between any surface form of a and
|
||||
// b (plus the slug-token similarity). Slug and name signals live in
|
||||
// different symbol spaces (ASCII pinyin vs raw surface form) so we take
|
||||
// the max rather than e.g. their average.
|
||||
func dedupPairScore(a, b dedupSurface) float64 {
|
||||
best := searchutil.Jaccard(a.slugTokens, b.slugTokens)
|
||||
for _, ag := range a.nameGramSets {
|
||||
for _, bg := range b.nameGramSets {
|
||||
if v := searchutil.Jaccard(ag, bg); v > best {
|
||||
best = v
|
||||
}
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
// slugBaseTokens returns the kebab-case tokens of a slug's base component.
|
||||
// "entity/beijing-nongshang-yinxing" → {"beijing", "nongshang", "yinxing"}.
|
||||
func slugBaseTokens(slug string) map[string]struct{} {
|
||||
if slug == "" {
|
||||
return nil
|
||||
}
|
||||
base := slug
|
||||
if i := strings.Index(slug, "/"); i >= 0 {
|
||||
base = slug[i+1:]
|
||||
}
|
||||
base = strings.ToLower(base)
|
||||
fields := strings.FieldsFunc(base, func(r rune) bool {
|
||||
return r == '-' || r == '_' || r == '.' || unicode.IsSpace(r)
|
||||
})
|
||||
if len(fields) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]struct{}, len(fields))
|
||||
for _, tok := range fields {
|
||||
if tok == "" {
|
||||
continue
|
||||
}
|
||||
out[tok] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// gramsPerSurface computes a gram set per non-empty surface form.
|
||||
func gramsPerSurface(surfaces []string) []map[string]struct{} {
|
||||
out := make([]map[string]struct{}, 0, len(surfaces))
|
||||
for _, s := range surfaces {
|
||||
g := surfaceGrams(s)
|
||||
if len(g) > 0 {
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// surfaceGrams returns a character-bigram set for a surface form after
|
||||
// lowercasing and stripping non-letter/digit runes. Bigrams work well
|
||||
// across both CJK (where each bigram approximates a word) and Latin
|
||||
// (where they catch stem overlap like "corporation" ↔ "corp"). Single-
|
||||
// rune strings fall back to a 1-gram so they still contribute a signal.
|
||||
func surfaceGrams(s string) map[string]struct{} {
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
var b strings.Builder
|
||||
b.Grow(len(s))
|
||||
for _, r := range strings.ToLower(s) {
|
||||
if unicode.IsLetter(r) || unicode.IsDigit(r) {
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
runes := []rune(b.String())
|
||||
if len(runes) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]struct{}, len(runes))
|
||||
if len(runes) == 1 {
|
||||
out[string(runes)] = struct{}{}
|
||||
return out
|
||||
}
|
||||
for i := 0; i < len(runes)-1; i++ {
|
||||
out[string(runes[i:i+2])] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// helper: build a minimal entity/concept WikiPage.
|
||||
func makePage(slug, title, typ string, aliases ...string) *types.WikiPage {
|
||||
return &types.WikiPage{
|
||||
Slug: slug,
|
||||
Title: title,
|
||||
PageType: typ,
|
||||
Aliases: types.StringArray(aliases),
|
||||
}
|
||||
}
|
||||
|
||||
func pageSlugs(pages []*types.WikiPage) []string {
|
||||
out := make([]string, 0, len(pages))
|
||||
for _, p := range pages {
|
||||
out = append(out, p.Slug)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func containsSlug(pages []*types.WikiPage, slug string) bool {
|
||||
for _, p := range pages {
|
||||
if p.Slug == slug {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Regression for the observed hallucination in llm_debug/20260422_171316.675.log:
|
||||
// deepseek-v3.2 merged concept/chengzhen-dengji-shiye-renyuan into
|
||||
// concept/zhong-hua-you-xiu-chuan-tong-wen-hua despite zero character overlap.
|
||||
// With the prefilter in place, the traditional-culture page should not even
|
||||
// be offered to the LLM as a candidate for that new item.
|
||||
func TestSelectDedupCandidatePages_FiltersUnrelatedHallucinationTarget(t *testing.T) {
|
||||
newItems := []extractedItem{
|
||||
{Slug: "concept/chengzhen-dengji-shiye-renyuan", Name: "城镇登记失业人员", Aliases: []string{"登记失业人员"}},
|
||||
{Slug: "entity/beijing-nongshang-yinxing", Name: "北京农商银行"},
|
||||
}
|
||||
|
||||
// Build a corpus that mirrors the real log's shape: ~90 unrelated pages
|
||||
// plus a few that share tokens with the new items (so the prefilter has
|
||||
// plausible near-matches to keep).
|
||||
pages := []*types.WikiPage{
|
||||
makePage("concept/zhong-hua-you-xiu-chuan-tong-wen-hua", "中华优秀传统文化", "concept"),
|
||||
makePage("concept/jiuye-jineng-peixun", "就业技能培训", "concept"),
|
||||
makePage("concept/peixun-kaohe-pingjia", "培训考核评价", "concept"),
|
||||
makePage("concept/ren-gong-zhi-neng-an-quan", "人工智能安全", "concept", "AI安全"),
|
||||
makePage("entity/bei-jing-shi-jiao-yu-wei-yuan-hui", "北京市教育委员会", "entity", "北京市教委"),
|
||||
makePage("entity/beijingshi-changping-zhiye-xuexiao", "北京市昌平职业学校", "entity"),
|
||||
}
|
||||
// Pad with filler so we exceed dedupSmallCorpusBypass.
|
||||
for i := 0; i < 40; i++ {
|
||||
pages = append(pages, makePage(
|
||||
"concept/filler-"+strings.Repeat("x", i+1),
|
||||
"填充概念"+strings.Repeat("占位", i+1),
|
||||
"concept",
|
||||
))
|
||||
}
|
||||
|
||||
got := selectDedupCandidatePages(newItems, pages)
|
||||
|
||||
if containsSlug(got, "concept/zhong-hua-you-xiu-chuan-tong-wen-hua") {
|
||||
t.Fatalf("expected unrelated page to be filtered out, but got it in candidates: %v",
|
||||
pageSlugs(got))
|
||||
}
|
||||
if len(got) >= len(pages) {
|
||||
t.Fatalf("expected prefilter to shrink the corpus (%d pages), but kept %d",
|
||||
len(pages), len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// A related page (shares tokens / characters with a new item) must survive
|
||||
// the prefilter so the LLM can still evaluate the merge.
|
||||
func TestSelectDedupCandidatePages_KeepsRelatedPages(t *testing.T) {
|
||||
newItems := []extractedItem{
|
||||
{Slug: "concept/chengzhen-dengji-shiye-renyuan", Name: "城镇登记失业人员", Aliases: []string{"登记失业人员"}},
|
||||
}
|
||||
// Build > dedupSmallCorpusBypass pages so the filter actually runs.
|
||||
pages := []*types.WikiPage{
|
||||
// Directly related: existing page whose title overlaps with the new
|
||||
// item on "登记失业人员". Prefilter MUST keep this.
|
||||
makePage("concept/deng-ji-shi-ye-ren-yuan", "登记失业人员", "concept", "城镇登记失业人员"),
|
||||
// Unrelated.
|
||||
makePage("concept/zhong-hua-you-xiu-chuan-tong-wen-hua", "中华优秀传统文化", "concept"),
|
||||
}
|
||||
for i := 0; i < 30; i++ {
|
||||
pages = append(pages, makePage(
|
||||
"entity/filler-"+strings.Repeat("x", i+1),
|
||||
"填充实体"+strings.Repeat("占位", i+1),
|
||||
"entity",
|
||||
))
|
||||
}
|
||||
|
||||
got := selectDedupCandidatePages(newItems, pages)
|
||||
|
||||
if !containsSlug(got, "concept/deng-ji-shi-ye-ren-yuan") {
|
||||
t.Fatalf("expected strongly-related page to be kept, got: %v", pageSlugs(got))
|
||||
}
|
||||
}
|
||||
|
||||
// On small corpora the filter should be a no-op (minus page-type filtering):
|
||||
// passing every page through is cheap and avoids cutting legitimate matches
|
||||
// when the prompt is already small.
|
||||
func TestSelectDedupCandidatePages_SmallCorpusBypass(t *testing.T) {
|
||||
newItems := []extractedItem{
|
||||
{Slug: "concept/a", Name: "A"},
|
||||
}
|
||||
pages := []*types.WikiPage{
|
||||
makePage("concept/wholly-unrelated-1", "毫不相关一", "concept"),
|
||||
makePage("concept/wholly-unrelated-2", "毫不相关二", "concept"),
|
||||
makePage("concept/wholly-unrelated-3", "毫不相关三", "concept"),
|
||||
}
|
||||
got := selectDedupCandidatePages(newItems, pages)
|
||||
if len(got) != len(pages) {
|
||||
t.Fatalf("expected bypass on small corpus: got %d, want %d", len(got), len(pages))
|
||||
}
|
||||
}
|
||||
|
||||
// Non-entity/concept pages (summaries, logs, …) must be stripped regardless
|
||||
// of corpus size — they are never valid merge targets.
|
||||
func TestSelectDedupCandidatePages_DropsNonEntityConcept(t *testing.T) {
|
||||
newItems := []extractedItem{
|
||||
{Slug: "concept/foo", Name: "Foo"},
|
||||
}
|
||||
pages := []*types.WikiPage{
|
||||
makePage("summary/some-doc", "Some Doc Summary", types.WikiPageTypeSummary),
|
||||
makePage("log/batch-123", "Batch 123", types.WikiPageTypeLog),
|
||||
makePage("concept/foo-related", "Foo Related", types.WikiPageTypeConcept),
|
||||
}
|
||||
got := selectDedupCandidatePages(newItems, pages)
|
||||
for _, p := range got {
|
||||
if p.PageType != types.WikiPageTypeEntity && p.PageType != types.WikiPageTypeConcept {
|
||||
t.Fatalf("non-entity/concept page should have been filtered: %s (%s)",
|
||||
p.Slug, p.PageType)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// surfaceGrams must yield empty intersection for the real hallucinated pair,
|
||||
// confirming the underlying similarity signal is doing its job.
|
||||
func TestSurfaceGrams_UnrelatedCJKPair(t *testing.T) {
|
||||
a := surfaceGrams("城镇登记失业人员")
|
||||
b := surfaceGrams("中华优秀传统文化")
|
||||
for k := range a {
|
||||
if _, ok := b[k]; ok {
|
||||
t.Fatalf("expected zero bigram overlap, but shared %q", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Latin abbreviation ↔ full-name pair must score highly (> floor) so the
|
||||
// filter keeps legitimate merge candidates like "Acme Corp" ↔ "Acme Corporation".
|
||||
func TestDedupPairScore_AcmeCorpVariant(t *testing.T) {
|
||||
a := dedupSurface{
|
||||
slugTokens: slugBaseTokens("entity/acme-corp"),
|
||||
nameGramSets: gramsPerSurface([]string{"Acme Corp"}),
|
||||
}
|
||||
b := dedupSurface{
|
||||
slugTokens: slugBaseTokens("entity/acme-corporation"),
|
||||
nameGramSets: gramsPerSurface([]string{"Acme Corporation"}),
|
||||
}
|
||||
score := dedupPairScore(a, b)
|
||||
if score < dedupCandidateScoreFloor {
|
||||
t.Fatalf("expected Acme Corp ↔ Corporation score above floor %v, got %v",
|
||||
dedupCandidateScoreFloor, score)
|
||||
}
|
||||
}
|
||||
|
||||
// Unrelated CJK pair (the exact case observed in production) must score 0.
|
||||
func TestDedupPairScore_UnrelatedCJKPair(t *testing.T) {
|
||||
a := dedupSurface{
|
||||
slugTokens: slugBaseTokens("concept/chengzhen-dengji-shiye-renyuan"),
|
||||
nameGramSets: gramsPerSurface([]string{"城镇登记失业人员", "登记失业人员"}),
|
||||
}
|
||||
b := dedupSurface{
|
||||
slugTokens: slugBaseTokens("concept/zhong-hua-you-xiu-chuan-tong-wen-hua"),
|
||||
nameGramSets: gramsPerSurface([]string{"中华优秀传统文化"}),
|
||||
}
|
||||
if s := dedupPairScore(a, b); s >= dedupCandidateScoreFloor {
|
||||
t.Fatalf("expected unrelated CJK pair to score below floor %v, got %v",
|
||||
dedupCandidateScoreFloor, s)
|
||||
}
|
||||
}
|
||||
@@ -92,6 +92,7 @@ func (s *wikiPageService) UpdatePage(ctx context.Context, page *types.WikiPage)
|
||||
existing.Summary = page.Summary
|
||||
existing.PageType = page.PageType
|
||||
existing.SourceRefs = page.SourceRefs
|
||||
existing.ChunkRefs = page.ChunkRefs
|
||||
existing.PageMetadata = page.PageMetadata
|
||||
existing.Status = page.Status
|
||||
existing.UpdatedAt = time.Now()
|
||||
|
||||
+84
-30
@@ -65,8 +65,19 @@ type WikiPage struct {
|
||||
Summary string `json:"summary" gorm:"type:text"`
|
||||
// Alternate names, abbreviations, acronyms or translated names
|
||||
Aliases StringArray `json:"aliases" gorm:"type:json"`
|
||||
// References to source knowledge IDs that contributed to this page
|
||||
// References to source knowledge IDs that contributed to this page.
|
||||
// Format matches the legacy "<knowledge_id>|<doc_title>" convention used
|
||||
// across the ingest pipeline, so retract / display code can split on `|`
|
||||
// to recover the title. Document-level granularity.
|
||||
SourceRefs StringArray `json:"source_refs" gorm:"type:json"`
|
||||
// ChunkRefs records the specific source-document chunks this page was
|
||||
// built from — one UUID per cited chunk. Populated during ingest from
|
||||
// the chunk-citation pass; refreshed wholesale whenever the page is
|
||||
// re-materialized. Empty for summary pages (they are document-level
|
||||
// synopses and don't carry chunk-level citations). Use this when you
|
||||
// need to surface the underlying evidence for a wiki page, or to
|
||||
// retract citations when a source document is deleted.
|
||||
ChunkRefs StringArray `json:"chunk_refs" gorm:"type:json"`
|
||||
// Slugs of pages that link TO this page (backlinks)
|
||||
InLinks StringArray `json:"in_links" gorm:"type:json"`
|
||||
// Slugs of pages this page links to (outbound links)
|
||||
@@ -88,18 +99,61 @@ func (WikiPage) TableName() string {
|
||||
return "wiki_pages"
|
||||
}
|
||||
|
||||
// WikiExtractionGranularity controls how aggressive Pass 0 (candidate slug
|
||||
// extraction) is. Higher granularity = more slugs, lower = tighter focus on
|
||||
// the document's main subjects.
|
||||
type WikiExtractionGranularity string
|
||||
|
||||
const (
|
||||
// WikiExtractionFocused keeps only the document's main subjects (e.g.
|
||||
// a resume yields the person + their projects, nothing else). Most
|
||||
// aggressive slug pruning; avoids index bloat from incidental technology
|
||||
// names and generic concepts.
|
||||
WikiExtractionFocused WikiExtractionGranularity = "focused"
|
||||
|
||||
// WikiExtractionStandard is the default: main subjects plus entities /
|
||||
// concepts that are substantively discussed (a dedicated paragraph or
|
||||
// multiple bullet points). Skips one-off mentions and commodity terms.
|
||||
WikiExtractionStandard WikiExtractionGranularity = "standard"
|
||||
|
||||
// WikiExtractionExhaustive extracts every named entity and recognizable
|
||||
// concept, including stacks/libs mentioned in passing. Matches the
|
||||
// pre-granularity behavior. Useful when the KB is being used as a
|
||||
// glossary rather than a curated wiki.
|
||||
WikiExtractionExhaustive WikiExtractionGranularity = "exhaustive"
|
||||
)
|
||||
|
||||
// IsValid reports whether g is one of the three recognized levels.
|
||||
func (g WikiExtractionGranularity) IsValid() bool {
|
||||
switch g {
|
||||
case WikiExtractionFocused, WikiExtractionStandard, WikiExtractionExhaustive:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Normalize returns g if valid, otherwise WikiExtractionStandard. Callers
|
||||
// pipe config through this so historical rows with empty / unknown values
|
||||
// don't surprise the extraction prompt.
|
||||
func (g WikiExtractionGranularity) Normalize() WikiExtractionGranularity {
|
||||
if g.IsValid() {
|
||||
return g
|
||||
}
|
||||
return WikiExtractionStandard
|
||||
}
|
||||
|
||||
// WikiConfig stores wiki-specific configuration for a knowledge base.
|
||||
// Applicable to document-type knowledge bases with wiki feature enabled.
|
||||
// When Enabled is true, document ingestion triggers automatic wiki page generation.
|
||||
// Whether the wiki feature is turned on is controlled by IndexingStrategy.WikiEnabled;
|
||||
// this struct only carries wiki-specific tunables.
|
||||
type WikiConfig struct {
|
||||
// Enabled activates the wiki feature for this knowledge base
|
||||
Enabled bool `yaml:"enabled" json:"enabled"`
|
||||
// AutoIngest triggers wiki page generation/update when new documents are added
|
||||
AutoIngest bool `yaml:"auto_ingest" json:"auto_ingest"`
|
||||
// SynthesisModelID is the LLM model ID used for wiki page generation and updates
|
||||
SynthesisModelID string `yaml:"synthesis_model_id" json:"synthesis_model_id"`
|
||||
// MaxPagesPerIngest limits pages created/updated per ingest operation (0 = no limit)
|
||||
MaxPagesPerIngest int `yaml:"max_pages_per_ingest" json:"max_pages_per_ingest"`
|
||||
// ExtractionGranularity controls how many candidate slugs Pass 0 extracts
|
||||
// per document. Empty / unknown value is treated as WikiExtractionStandard.
|
||||
ExtractionGranularity WikiExtractionGranularity `yaml:"extraction_granularity" json:"extraction_granularity,omitempty"`
|
||||
}
|
||||
|
||||
// Value implements the driver.Valuer interface
|
||||
@@ -122,13 +176,13 @@ func (c *WikiConfig) Scan(value interface{}) error {
|
||||
// WikiPageListRequest represents a request to list wiki pages with filtering
|
||||
type WikiPageListRequest struct {
|
||||
KnowledgeBaseID string `json:"knowledge_base_id"`
|
||||
PageType string `json:"page_type,omitempty"` // filter by type
|
||||
Status string `json:"status,omitempty"` // filter by status
|
||||
Query string `json:"query,omitempty"` // full-text search
|
||||
Page int `json:"page,omitempty"` // pagination page (1-based)
|
||||
PageSize int `json:"page_size,omitempty"` // pagination size
|
||||
SortBy string `json:"sort_by,omitempty"` // "updated_at", "created_at", "title"
|
||||
SortOrder string `json:"sort_order,omitempty"` // "asc" or "desc"
|
||||
PageType string `json:"page_type,omitempty"` // filter by type
|
||||
Status string `json:"status,omitempty"` // filter by status
|
||||
Query string `json:"query,omitempty"` // full-text search
|
||||
Page int `json:"page,omitempty"` // pagination page (1-based)
|
||||
PageSize int `json:"page_size,omitempty"` // pagination size
|
||||
SortBy string `json:"sort_by,omitempty"` // "updated_at", "created_at", "title"
|
||||
SortOrder string `json:"sort_order,omitempty"` // "asc" or "desc"
|
||||
}
|
||||
|
||||
// WikiPageListResponse represents a paginated list of wiki pages
|
||||
@@ -166,28 +220,28 @@ type WikiStats struct {
|
||||
TotalPages int64 `json:"total_pages"`
|
||||
PagesByType map[string]int64 `json:"pages_by_type"`
|
||||
TotalLinks int64 `json:"total_links"`
|
||||
OrphanCount int64 `json:"orphan_count"` // pages with no inbound links
|
||||
RecentUpdates []*WikiPage `json:"recent_updates"` // last N updated pages
|
||||
PendingTasks int64 `json:"pending_tasks"` // number of documents waiting to be ingested
|
||||
PendingIssues int64 `json:"pending_issues"` // number of pending wiki issues
|
||||
IsActive bool `json:"is_active"` // whether wiki ingestion is currently running
|
||||
OrphanCount int64 `json:"orphan_count"` // pages with no inbound links
|
||||
RecentUpdates []*WikiPage `json:"recent_updates"` // last N updated pages
|
||||
PendingTasks int64 `json:"pending_tasks"` // number of documents waiting to be ingested
|
||||
PendingIssues int64 `json:"pending_issues"` // number of pending wiki issues
|
||||
IsActive bool `json:"is_active"` // whether wiki ingestion is currently running
|
||||
}
|
||||
|
||||
// WikiPageIssue represents an issue flagged on a specific wiki page.
|
||||
// These issues are typically identified by agents or linters and stored for review.
|
||||
type WikiPageIssue struct {
|
||||
ID string `json:"id" gorm:"type:varchar(36);primaryKey"`
|
||||
TenantID uint64 `json:"tenant_id" gorm:"index"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id" gorm:"type:varchar(36);index"`
|
||||
Slug string `json:"slug" gorm:"type:varchar(255);index"`
|
||||
IssueType string `json:"issue_type" gorm:"type:varchar(50)"`
|
||||
Description string `json:"description" gorm:"type:text"`
|
||||
SuspectedKnowledgeIDs StringArray `json:"suspected_knowledge_ids" gorm:"type:json"`
|
||||
Status string `json:"status" gorm:"type:varchar(20);default:'pending';index"`
|
||||
ReportedBy string `json:"reported_by" gorm:"type:varchar(100)"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
DeletedAt gorm.DeletedAt `json:"deleted_at" gorm:"index"`
|
||||
ID string `json:"id" gorm:"type:varchar(36);primaryKey"`
|
||||
TenantID uint64 `json:"tenant_id" gorm:"index"`
|
||||
KnowledgeBaseID string `json:"knowledge_base_id" gorm:"type:varchar(36);index"`
|
||||
Slug string `json:"slug" gorm:"type:varchar(255);index"`
|
||||
IssueType string `json:"issue_type" gorm:"type:varchar(50)"`
|
||||
Description string `json:"description" gorm:"type:text"`
|
||||
SuspectedKnowledgeIDs StringArray `json:"suspected_knowledge_ids" gorm:"type:json"`
|
||||
Status string `json:"status" gorm:"type:varchar(20);default:'pending';index"`
|
||||
ReportedBy string `json:"reported_by" gorm:"type:varchar(100)"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
DeletedAt gorm.DeletedAt `json:"deleted_at" gorm:"index"`
|
||||
}
|
||||
|
||||
// TableName specifies the database table name
|
||||
|
||||
@@ -29,11 +29,8 @@ func TestWikiPageTypes(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestWikiConfigValueScan(t *testing.T) {
|
||||
// Test Value (serialize)
|
||||
config := WikiConfig{
|
||||
Enabled: true,
|
||||
AutoIngest: true,
|
||||
SynthesisModelID: "model-123",
|
||||
SynthesisModelID: "model-123",
|
||||
MaxPagesPerIngest: 20,
|
||||
}
|
||||
|
||||
@@ -42,7 +39,6 @@ func TestWikiConfigValueScan(t *testing.T) {
|
||||
t.Fatalf("WikiConfig.Value() error: %v", err)
|
||||
}
|
||||
|
||||
// Test Scan (deserialize)
|
||||
var restored WikiConfig
|
||||
b, ok := val.([]byte)
|
||||
if !ok {
|
||||
@@ -52,9 +48,6 @@ func TestWikiConfigValueScan(t *testing.T) {
|
||||
t.Fatalf("WikiConfig.Scan() error: %v", err)
|
||||
}
|
||||
|
||||
if restored.AutoIngest != true {
|
||||
t.Error("AutoIngest mismatch")
|
||||
}
|
||||
if restored.SynthesisModelID != "model-123" {
|
||||
t.Error("SynthesisModelID mismatch")
|
||||
}
|
||||
@@ -115,25 +108,21 @@ func TestStringArrayEmpty(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestKnowledgeBaseEnsureDefaultsWiki(t *testing.T) {
|
||||
kb := &KnowledgeBase{Type: KnowledgeBaseTypeDocument, WikiConfig: &WikiConfig{Enabled: true, AutoIngest: true}}
|
||||
kb := &KnowledgeBase{
|
||||
Type: KnowledgeBaseTypeDocument,
|
||||
WikiConfig: &WikiConfig{SynthesisModelID: "m-1"},
|
||||
}
|
||||
kb.EnsureDefaults()
|
||||
|
||||
if kb.WikiConfig == nil {
|
||||
t.Fatal("EnsureDefaults should preserve WikiConfig for wiki-enabled KB")
|
||||
}
|
||||
if kb.WikiConfig.AutoIngest != true {
|
||||
t.Error("AutoIngest should be preserved")
|
||||
if kb.WikiConfig.SynthesisModelID != "m-1" {
|
||||
t.Error("SynthesisModelID should be preserved")
|
||||
}
|
||||
if kb.FAQConfig != nil {
|
||||
t.Error("Document KB should not have FAQConfig")
|
||||
}
|
||||
|
||||
// Test that user can disable AutoIngest
|
||||
kb2 := &KnowledgeBase{Type: KnowledgeBaseTypeDocument, WikiConfig: &WikiConfig{Enabled: true, AutoIngest: false}}
|
||||
kb2.EnsureDefaults()
|
||||
if kb2.WikiConfig.AutoIngest != false {
|
||||
t.Error("EnsureDefaults should not override user's AutoIngest=false setting")
|
||||
}
|
||||
}
|
||||
|
||||
func TestKnowledgeBaseEnsureDefaultsDocumentWithoutWiki(t *testing.T) {
|
||||
@@ -150,16 +139,16 @@ func TestKnowledgeBaseEnsureDefaultsDocumentWithoutWiki(t *testing.T) {
|
||||
|
||||
func TestWikiPageJSON(t *testing.T) {
|
||||
page := WikiPage{
|
||||
ID: "test-id",
|
||||
Slug: "entity/test",
|
||||
Title: "Test Entity",
|
||||
PageType: WikiPageTypeEntity,
|
||||
Content: "# Test\n\nSome content with [[concept/related]]",
|
||||
Summary: "A test entity page",
|
||||
SourceRefs: StringArray{"source-1", "source-2"},
|
||||
OutLinks: StringArray{"concept/related"},
|
||||
InLinks: StringArray{"summary/doc1"},
|
||||
Version: 3,
|
||||
ID: "test-id",
|
||||
Slug: "entity/test",
|
||||
Title: "Test Entity",
|
||||
PageType: WikiPageTypeEntity,
|
||||
Content: "# Test\n\nSome content with [[concept/related]]",
|
||||
Summary: "A test entity page",
|
||||
SourceRefs: StringArray{"source-1", "source-2"},
|
||||
OutLinks: StringArray{"concept/related"},
|
||||
InLinks: StringArray{"summary/doc1"},
|
||||
Version: 3,
|
||||
}
|
||||
|
||||
// Serialize
|
||||
@@ -225,3 +214,73 @@ func TestChunkTypeWikiPage(t *testing.T) {
|
||||
t.Errorf("ChunkTypeWikiPage should be 'wiki_page', got '%s'", ChunkTypeWikiPage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWikiExtractionGranularity_IsValid(t *testing.T) {
|
||||
valid := []WikiExtractionGranularity{
|
||||
WikiExtractionFocused,
|
||||
WikiExtractionStandard,
|
||||
WikiExtractionExhaustive,
|
||||
}
|
||||
for _, g := range valid {
|
||||
if !g.IsValid() {
|
||||
t.Errorf("%q should be valid", g)
|
||||
}
|
||||
}
|
||||
|
||||
invalid := []WikiExtractionGranularity{"", "FOCUSED", "strict", "none", "FULL"}
|
||||
for _, g := range invalid {
|
||||
if g.IsValid() {
|
||||
t.Errorf("%q should NOT be valid", g)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWikiExtractionGranularity_Normalize(t *testing.T) {
|
||||
cases := map[WikiExtractionGranularity]WikiExtractionGranularity{
|
||||
"": WikiExtractionStandard, // legacy / unset
|
||||
"unknown": WikiExtractionStandard,
|
||||
"FOCUSED": WikiExtractionStandard, // case-sensitive on purpose
|
||||
"focused": WikiExtractionFocused,
|
||||
"standard": WikiExtractionStandard,
|
||||
"exhaustive": WikiExtractionExhaustive,
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := in.Normalize(); got != want {
|
||||
t.Errorf("Normalize(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWikiConfig_JSONRoundTrip_WithGranularity(t *testing.T) {
|
||||
original := WikiConfig{
|
||||
SynthesisModelID: "m-1",
|
||||
MaxPagesPerIngest: 20,
|
||||
ExtractionGranularity: WikiExtractionFocused,
|
||||
}
|
||||
data, err := json.Marshal(original)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal error: %v", err)
|
||||
}
|
||||
var restored WikiConfig
|
||||
if err := json.Unmarshal(data, &restored); err != nil {
|
||||
t.Fatalf("Unmarshal error: %v", err)
|
||||
}
|
||||
if restored.ExtractionGranularity != WikiExtractionFocused {
|
||||
t.Errorf("granularity round-trip failed: got %q", restored.ExtractionGranularity)
|
||||
}
|
||||
|
||||
// Legacy row without the granularity field. We also include the removed
|
||||
// "enabled" / "auto_ingest" keys to verify the decoder tolerates unknown
|
||||
// fields on rows written before those columns were retired.
|
||||
legacy := []byte(`{"enabled":true,"auto_ingest":true,"synthesis_model_id":"","max_pages_per_ingest":0}`)
|
||||
var oldCfg WikiConfig
|
||||
if err := json.Unmarshal(legacy, &oldCfg); err != nil {
|
||||
t.Fatalf("Unmarshal legacy error: %v", err)
|
||||
}
|
||||
if oldCfg.ExtractionGranularity != "" {
|
||||
t.Errorf("legacy row should decode to empty granularity, got %q", oldCfg.ExtractionGranularity)
|
||||
}
|
||||
if oldCfg.ExtractionGranularity.Normalize() != WikiExtractionStandard {
|
||||
t.Errorf("legacy row should normalize to standard")
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user