fix(wiki): mask image URLs before LLM to prevent UUID/path corruption
Wiki summary/entity/concept/page-modify prompts instructed the LLM to reproduce  verbatim, so high-entropy image URLs passed through generation and got corrupted (dropped UUID segments, duplicated or mangled path prefixes, even CJK chars inserted into the bucket name), producing object keys that don't exist -> 404. Mask all image URLs to opaque placeholders at the generateWithTemplate chokepoint and deterministically rehydrate after generation; drop corrupted/hallucinated placeholders so broken links never reach storage.
This commit is contained in:
@@ -30,7 +30,7 @@ const WikiSummaryPrompt = `You are a wiki editor. Given the following document c
|
||||
3. Include the key facts, arguments, and conclusions.
|
||||
4. Use proper heading hierarchy (## for sections, ### for subsections).
|
||||
5. **Wiki-link rule**: The available_wiki_pages list above maps slugs to display names and their aliases (format: "[[slug]] = display name (Aliases: a, b)"). Whenever you mention a name or alias that matches a listed entry, you MUST write it as [[slug|display name]] (e.g. [[entity/zhong-guo|中国]]), NOT as bold (**name**) or bare [[slug]]. Use the EXACT slugs provided — do NOT invent new slugs.
|
||||
6. **Image rule**: If the document contains <images> tags with <image> elements, you SHOULD include the relevant images in your summary using the Markdown syntax: . Place the images where they are contextually relevant to the text.
|
||||
6. **Image rule**: If the document contains <images> tags with <image> elements, you SHOULD include the relevant images in your summary using the Markdown syntax: . Place the images where they are contextually relevant to the text. The URL inside  is an opaque token; reproduce it EXACTLY and VERBATIM, do not alter, shorten, or normalize it.
|
||||
7. At the end, include a "## Key Takeaways" section with bullet points.
|
||||
8. Write in {{.Language}}.
|
||||
9. Keep the summary concise but thorough (500-1500 words depending on document length).
|
||||
@@ -73,7 +73,7 @@ Each entity should have:
|
||||
- "slug": URL-friendly slug, format "entity/<lowercase-hyphenated-name>" (use romanized/pinyin form for non-Latin names). **Reuse previous slug if the entity was extracted before.**
|
||||
- "aliases": An array of strings representing names that refer to THE EXACT SAME entity. Only include: official abbreviations (e.g. "IBM" for "International Business Machines"), full/short name variants (e.g. "腾讯" for "腾讯控股有限公司"), translations (e.g. "Apple" for "苹果公司"), and well-known alternate names (e.g. "Alphabet" for "Google母公司"). Do NOT include parent categories, related products, generic terms, or broader concepts. Provide [] if none.
|
||||
- "description": **Index listing summary** — one sentence, 15-40 words, in {{.Language}}. Describes WHAT this entity IS and its role in the document. Must be self-contained (understandable without reading the full page). This will be displayed in the wiki index.
|
||||
- "details": A 2-5 sentence summary in {{.Language}} of key facts from the document. **Image rule**: If the document contains relevant <image> elements in an <images> tag, include them in the details using Markdown syntax: .
|
||||
- "details": A 2-5 sentence summary in {{.Language}} of key facts from the document. **Image rule**: If the document contains relevant <image> elements in an <images> tag, include them in the details using Markdown syntax: . The URL inside  is an opaque token; reproduce it EXACTLY and VERBATIM, do not alter, shorten, or normalize it.
|
||||
|
||||
Only include entities that are substantively discussed (mentioned at least twice or described in detail). Do NOT include generic terms.
|
||||
|
||||
@@ -83,7 +83,7 @@ Each concept should have:
|
||||
- "slug": URL-friendly slug, format "concept/<lowercase-hyphenated-name>" (use romanized/pinyin form for non-Latin names). **Reuse previous slug if the concept was extracted before.**
|
||||
- "aliases": An array of strings representing names that refer to THE EXACT SAME concept. Only include: official abbreviations (e.g. "RAG" for "Retrieval-Augmented Generation"), full/short name variants, and well-known synonyms used interchangeably in the field. Do NOT include sub-topics, related techniques, broader categories, or implementation details. Provide [] if none.
|
||||
- "description": **Index listing summary** — one sentence, 15-40 words, in {{.Language}}. Defines WHAT this concept IS. Must be self-contained (understandable without reading the full page). This will be displayed in the wiki index.
|
||||
- "details": A 2-5 sentence explanation in {{.Language}} as discussed in the document. **Image rule**: If the document contains relevant <image> elements in an <images> tag, include them in the details using Markdown syntax: .
|
||||
- "details": A 2-5 sentence explanation in {{.Language}} as discussed in the document. **Image rule**: If the document contains relevant <image> elements in an <images> tag, include them in the details using Markdown syntax: . The URL inside  is an opaque token; reproduce it EXACTLY and VERBATIM, do not alter, shorten, or normalize it.
|
||||
|
||||
Only include concepts that are substantively discussed. Skip trivial or overly generic concepts.
|
||||
|
||||
@@ -323,7 +323,7 @@ The <new_information> block above is assembled from VERBATIM source chunks that
|
||||
4. Preserve existing information that is still valid and still about {{.PageTitle}}.
|
||||
5. Keep [[slug|name]] wiki-link references ONLY if the slug appears in the <valid_wiki_links> list above. Remove any [[slug|name]] whose slug is NOT in that list. Do NOT invent new wiki-link slugs. The page's own slug ({{.PageSlug}}) MUST NOT appear as a [[...]] link inside its own content.
|
||||
6. Maintain the existing page structure and formatting style. Use "# {{.PageTitle}}" as the top-level heading if the page does not already have one. Do NOT introduce new heading levels beyond what the source or existing page justifies.
|
||||
7. **Image rule**: Include relevant images using Markdown syntax:  from new information if applicable.
|
||||
7. **Image rule**: Include relevant images using Markdown syntax:  from new information if applicable. The URL inside  is an opaque token; reproduce it EXACTLY and VERBATIM, do not alter, shorten, or normalize it.
|
||||
{{if .HasRetractions}}
|
||||
8. If after removing deleted content the page becomes nearly empty and there is no new information to add, output just: "SUMMARY: (empty page)\n# {{.PageTitle}}\n\n*This page's primary source document was removed.*"
|
||||
{{end}}
|
||||
|
||||
@@ -1693,8 +1693,10 @@ func (s *wikiIngestService) generateWithTemplate(ctx context.Context, chatModel
|
||||
return "", fmt.Errorf("parse template: %w", err)
|
||||
}
|
||||
|
||||
maskedData, urlMap := maskTemplateDataImageURLs(data)
|
||||
|
||||
var buf strings.Builder
|
||||
if err := tmpl.Execute(&buf, data); err != nil {
|
||||
if err := tmpl.Execute(&buf, maskedData); err != nil {
|
||||
return "", fmt.Errorf("execute template: %w", err)
|
||||
}
|
||||
|
||||
@@ -1710,7 +1712,7 @@ func (s *wikiIngestService) generateWithTemplate(ctx context.Context, chatModel
|
||||
Thinking: &thinking,
|
||||
})
|
||||
if err == nil {
|
||||
return response.Content, nil
|
||||
return unmaskImageURLs(response.Content, urlMap), nil
|
||||
}
|
||||
lastErr = err
|
||||
|
||||
@@ -1999,8 +2001,160 @@ var (
|
||||
// <image_caption>...</image_caption> blocks, and stripping the content
|
||||
// would silently destroy the very text we want to keep.
|
||||
imageWrapperTagRE = regexp.MustCompile(`(?i)</?image[a-z_]*\b[^>]*/?>`)
|
||||
|
||||
// Markdown image references with the URL captured separately so LLM-bound
|
||||
// image URLs can be frozen while captions remain editable.
|
||||
mdImageURLRE = regexp.MustCompile(`!\[[^\]]*\]\(([^)]*)\)`)
|
||||
|
||||
// Enriched image blocks store the original object URL as an attribute,
|
||||
// e.g. <image url="...">. Capture both double- and single-quoted forms.
|
||||
imageURLAttrRE = regexp.MustCompile(`(?i)<image\b[^>]*\surl\s*=\s*(?:"([^"]*)"|'([^']*)')`)
|
||||
|
||||
imagePlaceholderTokenRE = regexp.MustCompile(`wkimg:[A-Za-z0-9_-]+`)
|
||||
)
|
||||
|
||||
func maskTemplateDataImageURLs(data map[string]string) (map[string]string, map[string]string) {
|
||||
if len(data) == 0 {
|
||||
return data, nil
|
||||
}
|
||||
|
||||
masked := make(map[string]string, len(data))
|
||||
urlToToken := make(map[string]string)
|
||||
tokenToURL := make(map[string]string)
|
||||
|
||||
keys := make([]string, 0, len(data))
|
||||
for key := range data {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
for _, key := range keys {
|
||||
masked[key] = maskImageURLsWithState(data[key], urlToToken, tokenToURL)
|
||||
}
|
||||
|
||||
return masked, tokenToURL
|
||||
}
|
||||
|
||||
// maskImageURLs replaces image URLs with low-entropy placeholders. It only
|
||||
// freezes URLs; alt/caption text remains in place for the LLM to edit.
|
||||
func maskImageURLs(s string) (string, map[string]string) {
|
||||
urlToToken := make(map[string]string)
|
||||
tokenToURL := make(map[string]string)
|
||||
return maskImageURLsWithState(s, urlToToken, tokenToURL), tokenToURL
|
||||
}
|
||||
|
||||
func maskImageURLsWithState(s string, urlToToken, tokenToURL map[string]string) string {
|
||||
urls := collectMaskableImageURLs(s)
|
||||
if len(urls) == 0 {
|
||||
return s
|
||||
}
|
||||
|
||||
for _, url := range urls {
|
||||
if _, ok := urlToToken[url]; ok {
|
||||
continue
|
||||
}
|
||||
token := fmt.Sprintf("wkimg:%04d", len(tokenToURL)+1)
|
||||
urlToToken[url] = token
|
||||
tokenToURL[token] = url
|
||||
}
|
||||
|
||||
replaceURLs := append([]string(nil), urls...)
|
||||
sort.SliceStable(replaceURLs, func(i, j int) bool {
|
||||
return len(replaceURLs[i]) > len(replaceURLs[j])
|
||||
})
|
||||
|
||||
masked := s
|
||||
for _, url := range replaceURLs {
|
||||
masked = strings.ReplaceAll(masked, url, urlToToken[url])
|
||||
}
|
||||
return masked
|
||||
}
|
||||
|
||||
func collectMaskableImageURLs(s string) []string {
|
||||
seen := make(map[string]struct{})
|
||||
var urls []string
|
||||
|
||||
addURL := func(url string) {
|
||||
url = strings.TrimSpace(url)
|
||||
if url == "" {
|
||||
return
|
||||
}
|
||||
if _, ok := seen[url]; ok {
|
||||
return
|
||||
}
|
||||
seen[url] = struct{}{}
|
||||
urls = append(urls, url)
|
||||
}
|
||||
|
||||
for _, match := range mdImageURLRE.FindAllStringSubmatch(s, -1) {
|
||||
addURL(match[1])
|
||||
}
|
||||
for _, match := range imageURLAttrRE.FindAllStringSubmatch(s, -1) {
|
||||
if match[1] != "" {
|
||||
addURL(match[1])
|
||||
continue
|
||||
}
|
||||
addURL(match[2])
|
||||
}
|
||||
|
||||
return urls
|
||||
}
|
||||
|
||||
// unmaskImageURLs restores known placeholders and drops any corrupted or
|
||||
// invented image placeholders so broken image links never reach storage.
|
||||
func unmaskImageURLs(out string, urlMap map[string]string) string {
|
||||
out = mdImageURLRE.ReplaceAllStringFunc(out, func(match string) string {
|
||||
parts := mdImageURLRE.FindStringSubmatch(match)
|
||||
if len(parts) != 2 {
|
||||
return match
|
||||
}
|
||||
url := strings.TrimSpace(parts[1])
|
||||
if realURL, ok := urlMap[url]; ok {
|
||||
idx := strings.LastIndex(match, "(")
|
||||
if idx < 0 {
|
||||
return match
|
||||
}
|
||||
return match[:idx+1] + realURL + ")"
|
||||
}
|
||||
if strings.HasPrefix(url, "wkimg:") {
|
||||
return ""
|
||||
}
|
||||
return match
|
||||
})
|
||||
|
||||
return replaceImagePlaceholderTokensOutsideMarkdown(out, urlMap)
|
||||
}
|
||||
|
||||
func replaceImagePlaceholderTokensOutsideMarkdown(s string, urlMap map[string]string) string {
|
||||
matches := mdImageURLRE.FindAllStringIndex(s, -1)
|
||||
if len(matches) == 0 {
|
||||
return replaceImagePlaceholderTokens(s, urlMap)
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
last := 0
|
||||
for _, match := range matches {
|
||||
if match[0] > last {
|
||||
b.WriteString(replaceImagePlaceholderTokens(s[last:match[0]], urlMap))
|
||||
}
|
||||
b.WriteString(s[match[0]:match[1]])
|
||||
last = match[1]
|
||||
}
|
||||
if last < len(s) {
|
||||
b.WriteString(replaceImagePlaceholderTokens(s[last:], urlMap))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func replaceImagePlaceholderTokens(s string, urlMap map[string]string) string {
|
||||
return imagePlaceholderTokenRE.ReplaceAllStringFunc(s, func(token string) string {
|
||||
if realURL, ok := urlMap[token]; ok {
|
||||
return realURL
|
||||
}
|
||||
return ""
|
||||
})
|
||||
}
|
||||
|
||||
// stripImageMarkup removes image-only placeholders (Markdown image refs,
|
||||
// <img> tags, <image_original> redundancy blocks) and unwraps the
|
||||
// <image>/<image_caption>/<image_ocr> XML wrappers produced by the search
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/models/chat"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
@@ -19,8 +22,8 @@ func TestSlugify(t *testing.T) {
|
||||
{"Special!@#Chars", "specialchars"},
|
||||
{"CamelCase", "camelcase"},
|
||||
{"", ""},
|
||||
{"a/b/c", "a/b/c"}, // preserve slashes for hierarchical slugs
|
||||
{"中文标题", "中文标题"}, // preserve CJK
|
||||
{"a/b/c", "a/b/c"}, // preserve slashes for hierarchical slugs
|
||||
{"中文标题", "中文标题"}, // preserve CJK
|
||||
{"Mix 中英文 Test", "mix-中英文-test"}, // mixed
|
||||
}
|
||||
|
||||
@@ -201,3 +204,154 @@ func TestHasSufficientTextContent(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMaskImageURLs(t *testing.T) {
|
||||
const urlA = "minio://kb/10000/exports/4135-aaaa-bbbb-cccc/page_1.jpg"
|
||||
const urlB = "local://kb/10000/exports/9999-dddd-eeee-ffff/page_2.png"
|
||||
|
||||
t.Run("single markdown image round trips exact URL", func(t *testing.T) {
|
||||
input := "Intro  outro"
|
||||
masked, urlMap := maskImageURLs(input)
|
||||
|
||||
if masked != "Intro  outro" {
|
||||
t.Fatalf("masked = %q", masked)
|
||||
}
|
||||
if got := urlMap["wkimg:0001"]; got != urlA {
|
||||
t.Fatalf("urlMap[wkimg:0001] = %q, want %q", got, urlA)
|
||||
}
|
||||
if got := unmaskImageURLs(masked, urlMap); got != input {
|
||||
t.Fatalf("unmaskImageURLs() = %q, want %q", got, input)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("enriched image attribute and original markdown share token", func(t *testing.T) {
|
||||
input := `<image url="` + urlA + `">
|
||||
<image_original></image_original>
|
||||
<image_caption>caption text</image_caption>
|
||||
</image>`
|
||||
masked, urlMap := maskImageURLs(input)
|
||||
|
||||
if len(urlMap) != 1 {
|
||||
t.Fatalf("len(urlMap) = %d, want 1", len(urlMap))
|
||||
}
|
||||
if strings.Count(masked, "wkimg:0001") != 2 {
|
||||
t.Fatalf("masked should contain the same placeholder twice: %q", masked)
|
||||
}
|
||||
if strings.Contains(masked, urlA) {
|
||||
t.Fatalf("masked still contains real URL: %q", masked)
|
||||
}
|
||||
if got := unmaskImageURLs(masked, urlMap); got != input {
|
||||
t.Fatalf("unmaskImageURLs() = %q, want %q", got, input)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("same URL repeated and distinct URLs map correctly", func(t *testing.T) {
|
||||
input := "\n\n"
|
||||
masked, urlMap := maskImageURLs(input)
|
||||
|
||||
if strings.Count(masked, "wkimg:0001") != 2 {
|
||||
t.Fatalf("same URL should reuse wkimg:0001: %q", masked)
|
||||
}
|
||||
if strings.Count(masked, "wkimg:0002") != 1 {
|
||||
t.Fatalf("second URL should use wkimg:0002: %q", masked)
|
||||
}
|
||||
if got := unmaskImageURLs(masked, urlMap); got != input {
|
||||
t.Fatalf("unmaskImageURLs() = %q, want %q", got, input)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("caption is preserved even when it resembles a placeholder", func(t *testing.T) {
|
||||
input := ""
|
||||
masked, urlMap := maskImageURLs(input)
|
||||
got := unmaskImageURLs(masked, urlMap)
|
||||
|
||||
if got != input {
|
||||
t.Fatalf("unmaskImageURLs() = %q, want %q", got, input)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("non image text is unchanged", func(t *testing.T) {
|
||||
input := "slug: entity/example\nlanguage: zh"
|
||||
masked, urlMap := maskImageURLs(input)
|
||||
|
||||
if masked != input {
|
||||
t.Fatalf("maskImageURLs() = %q, want %q", masked, input)
|
||||
}
|
||||
if len(urlMap) != 0 {
|
||||
t.Fatalf("len(urlMap) = %d, want 0", len(urlMap))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestUnmaskImageURLsDropsUnknownPlaceholders(t *testing.T) {
|
||||
urlMap := map[string]string{"wkimg:0001": "minio://kb/exports/real.jpg"}
|
||||
input := `{"details":"keep  drop  and wkimg:9999"}`
|
||||
got := unmaskImageURLs(input, urlMap)
|
||||
|
||||
if !strings.Contains(got, "") {
|
||||
t.Fatalf("known placeholder was not restored: %q", got)
|
||||
}
|
||||
if strings.Contains(got, "wkimg:") || strings.Contains(got, "![bad]") {
|
||||
t.Fatalf("unknown placeholders should be dropped: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateWithTemplateMasksImageURLsBeforeLLM(t *testing.T) {
|
||||
const realURL = "minio://kb/10000/exports/4135-aaaa-bbbb-cccc/page_1.jpg"
|
||||
model := &templateCaptureChatModel{
|
||||
response: `{"details":"Model kept "}`,
|
||||
}
|
||||
service := &wikiIngestService{}
|
||||
|
||||
got, err := service.generateWithTemplate(
|
||||
context.Background(),
|
||||
model,
|
||||
`Content={{.Content}} Existing={{.ExistingContent}}`,
|
||||
map[string]string{
|
||||
"Content": "new ",
|
||||
"ExistingContent": "old ",
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("generateWithTemplate() error = %v", err)
|
||||
}
|
||||
if strings.Contains(model.prompt, realURL) {
|
||||
t.Fatalf("LLM prompt contains real URL: %q", model.prompt)
|
||||
}
|
||||
if strings.Count(model.prompt, "wkimg:0001") != 2 {
|
||||
t.Fatalf("same URL across fields should share wkimg:0001: %q", model.prompt)
|
||||
}
|
||||
if strings.Contains(got, "wkimg:") {
|
||||
t.Fatalf("returned content still contains placeholder: %q", got)
|
||||
}
|
||||
if !strings.Contains(got, realURL) {
|
||||
t.Fatalf("returned content does not contain restored real URL: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
type templateCaptureChatModel struct {
|
||||
prompt string
|
||||
response string
|
||||
}
|
||||
|
||||
func (m *templateCaptureChatModel) Chat(
|
||||
_ context.Context,
|
||||
messages []chat.Message,
|
||||
_ *chat.ChatOptions,
|
||||
) (*types.ChatResponse, error) {
|
||||
if len(messages) > 0 {
|
||||
m.prompt = messages[0].Content
|
||||
}
|
||||
return &types.ChatResponse{Content: m.response}, nil
|
||||
}
|
||||
|
||||
func (m *templateCaptureChatModel) ChatStream(
|
||||
context.Context,
|
||||
[]chat.Message,
|
||||
*chat.ChatOptions,
|
||||
) (<-chan types.StreamResponse, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (m *templateCaptureChatModel) GetModelName() string { return "capture" }
|
||||
func (m *templateCaptureChatModel) GetModelID() string { return "capture" }
|
||||
|
||||
Reference in New Issue
Block a user