fix(chat): enforce retrieved image output in answers

When retrieved context or tool results contain Markdown images, dynamically
append output requirements to system prompts and final-answer instructions.
Also normalize fullwidth image parentheses in the frontend renderer so
localized model output still renders correctly.
This commit is contained in:
wizardchen
2026-07-15 18:09:17 +08:00
committed by lyingbug
parent 951f95903b
commit fbc9db85d1
11 changed files with 264 additions and 5 deletions
@@ -161,7 +161,7 @@ templates:
- WRONG (all grouped at end of answer):
<kb doc="文档A" chunk_id="xxx" /><kb doc="文档B" chunk_id="yyy" />
* **Structured:** Clear hierarchy and logic.
* **Rich Media (Markdown with Images):** When retrieved chunks contain Markdown images or an "images" field with URLs, reuse relevant images in your response using standard Markdown image syntax: ![description](image_url). Preserve each image URL exactly and place images at contextually appropriate positions within the answer. Images help users better understand the content, especially for diagrams, charts, screenshots, or visual explanations.
* **Rich Media (Markdown with Images — REQUIRED):** When any retrieved chunk or tool result contains Markdown images or an "images" field with URLs, treat those images as relevant by default. Unless the user explicitly requests text-only output or every retrieved image is clearly unrelated, the final answer MUST include at least one relevant image using standard Markdown syntax: `![description](image_url)`. Copy the complete Markdown image and URL exactly. Parentheses MUST be ASCII half-width `(` and `)`; NEVER use full-width `(` or `)`. Place each image immediately after the paragraph it supports. When multiple images support different sections, distribute them across those sections instead of stopping after the first image. Silently verify this requirement before finishing.
### System Status
Web Search: {{web_search_status}}
@@ -263,7 +263,7 @@ templates:
- **External MCP Tools (if any are exposed in your tool list):** Use them when the user's question requires real-time data, external system lookups, or actions that do NOT live inside the wiki (e.g. querying a ticketing system, calling an internal API, fetching live metrics). Treat their responses as additional evidence, not as a replacement for wiki content.
- **Skills (if an `### Available Skills` section appears later in this prompt):** Before answering, scan the listed skills. If the user's intent matches a skill's triggers, call `read_skill(skill_name="...")` to load its full instructions, then follow them. Skills are especially useful for specialized output formatting or domain-specific procedures.
- When the user @MCP or @Skill, a short `<must_use>` block also appears before the question — follow it with highest priority; wiki content remains the primary source of truth for domain facts, and MCP/skills are complementary.
6. **Synthesize:** Once you have gathered sufficient information from reading multiple interconnected pages (and optionally source documents, MCP results, or skill guidance), synthesize your answer and deliver it by writing it as your reply, then stop (no further tool calls in that final message). If you encountered relevant Markdown images in the source documents, reuse them in your answer and preserve their URLs exactly.
6. **Synthesize:** Once you have gathered sufficient information from reading multiple interconnected pages (and optionally source documents, MCP results, or skill guidance), synthesize your answer and deliver it by writing it as your reply, then stop (no further tool calls in that final message). If retrieved source documents contain Markdown images, the final answer MUST include at least one unless the user explicitly requests text-only output or every image is clearly unrelated. Copy the complete Markdown image syntax and URL exactly, using ASCII half-width `(` and `)` only—never full-width `(` or `)`. Place images after the paragraphs they support. When multiple images support different sections, distribute them across those sections instead of stopping after the first image. Verify image inclusion before finishing.
7. **Flag Issues (If Necessary):** If you discover that a wiki page contains factual errors, mixed entities (e.g., two different products combined into one page), or outdated information, OR if the user points out such errors, use the `wiki_flag_issue` tool to submit a maintenance report for that page before you write your final answer.
</workflow>
+5 -1
View File
@@ -28,7 +28,11 @@ templates:
- If the user's question cannot be answered, honestly inform the user and provide reasonable suggestions
## Output Format
- Output your final result in Markdown format with images when applicable
- Output your final result in Markdown format
- When retrieved information contains Markdown images, treat them as relevant by default. Unless the user explicitly requests text-only output or every image is clearly unrelated, the final answer MUST include at least one relevant image copied from the retrieved information with its URL preserved exactly
- Image Markdown MUST use ASCII half-width parentheses exactly as `![alt](url)`; never use full-width `(` or `)`
- Place each image immediately after the paragraph it supports; before finishing, silently verify that the answer satisfies this image requirement
- When multiple retrieved images support different sections, distribute them across those sections instead of stopping after the first image
- Ensure the output is concise yet comprehensive, well-organized, clear, and non-repetitive
## CRITICAL: Language Rule
@@ -6,6 +6,7 @@ import {
closeDanglingStreamingEmphasis,
createChatMarkdownRenderer,
markStandaloneStrongParagraphs,
normalizeFullwidthMarkdownImageParentheses,
normalizeLegacyImageContextMarkup,
preprocessMathDelimiters,
renderChatMarkdown,
@@ -46,6 +47,56 @@ test('replaceIncompleteImageWithPlaceholder hides an unfinished streaming image'
)
})
test('normalizeFullwidthMarkdownImageParentheses repairs localized image delimiters', () => {
const ref = 'resource://yB7V7wE1gls7h9WonCDq5Q'
assert.equal(
normalizeFullwidthMarkdownImageParentheses(`![](${ref})`),
`![](${ref})`,
)
assert.equal(
normalizeFullwidthMarkdownImageParentheses(`![流程图](${ref})`),
`![流程图](${ref})`,
)
assert.equal(
normalizeFullwidthMarkdownImageParentheses(`普通中文(说明)和 ![示例](unknown://value)`),
`普通中文(说明)和 ![示例](unknown://value)`,
)
})
test('normalizeFullwidthMarkdownImageParentheses preserves literal examples in code', () => {
const code = '`![](resource://example)`\n```md\n![](resource://example)\n```'
assert.equal(normalizeFullwidthMarkdownImageParentheses(code), code)
})
test('renderChatMarkdown safely renders an image with fullwidth parentheses', () => {
const renderer = createChatMarkdownRenderer({
imageRenderer: ({ href, text }) => `<img src="${href}" alt="${text}">`,
isValidImageUrl: (href) => href.startsWith('resource://'),
})
const html = renderChatMarkdown('![](resource://yB7V7wE1gls7h9WonCDq5Q)', {
renderer,
escapeMarkdown: (text) => text,
sanitizeHtml: (value) => value,
streaming: false,
})
assert.match(html, /<img src="resource:\/\/yB7V7wE1gls7h9WonCDq5Q" alt="">/)
assert.doesNotMatch(html, /(|)/)
})
test('renderChatMarkdown hides an unfinished fullwidth-parenthesis image while streaming', () => {
const renderer = createChatMarkdownRenderer()
const html = renderChatMarkdown('before ![流程图](resource://yB7V7wE1gls7h9WonCDq5Q', {
renderer,
escapeMarkdown: (text) => text,
sanitizeHtml: (value) => value,
streaming: true,
})
assert.match(html, /streaming-image-loading/)
assert.doesNotMatch(html, /resource:\/\/|(/)
})
test('normalizeLegacyImageContextMarkup converts copied image XML to Markdown', () => {
const input = [
'before',
+29 -1
View File
@@ -84,6 +84,33 @@ const LEGACY_IMAGE_ORIGINAL_RE = /<image_original>([\s\S]*?)<\/image_original>/i
const LEGACY_IMAGE_CAPTION_RE = /<image_caption>([\s\S]*?)<\/image_caption>/i
const LEGACY_IMAGE_OCR_RE = /<image_ocr>([\s\S]*?)<\/image_ocr>/i
const COMPLETE_MARKDOWN_CODE_RE = /(```[\s\S]*?```|~~~[\s\S]*?~~~|`[^`\n]*`)/g
const IMAGE_URL_SCHEME = '(?:https?|resource|storage|local|minio|s3|cos|tos|oss|obs|ks3)'
const FULLWIDTH_IMAGE_OPEN_RE = new RegExp(`(!\\[[^\\]\\n]*\\])((?=${IMAGE_URL_SCHEME}://)`, 'gi')
const FULLWIDTH_IMAGE_CLOSE_RE = new RegExp(
`(!\\[[^\\]\\n]*\\]\\(${IMAGE_URL_SCHEME}://[^)\\n]*?))`,
'gi',
)
/**
* Repair image Markdown when a model localizes its destination parentheses:
* `![alt](resource://…)` -> `![alt](resource://…)`.
*
* The rewrite is limited to supported image URL schemes and skips code, so
* ordinary Chinese punctuation and literal Markdown examples stay unchanged.
* Normalizing an opening parenthesis before the closing one streams in also
* lets the existing incomplete-image skeleton suppress the unstable tail.
*/
export function normalizeFullwidthMarkdownImageParentheses(content: string): string {
if (!content || (!content.includes('(') && !content.includes(')'))) return content
const parts = content.split(COMPLETE_MARKDOWN_CODE_RE)
for (let i = 0; i < parts.length; i += 2) {
parts[i] = parts[i]
.replace(FULLWIDTH_IMAGE_OPEN_RE, '$1(')
.replace(FULLWIDTH_IMAGE_CLOSE_RE, '$1)')
}
return parts.join('')
}
function legacyImageField(body: string, pattern: RegExp): string {
const value = body.match(pattern)?.[1] || ''
@@ -437,7 +464,8 @@ export function renderChatMarkdown(rawMarkdown: unknown, options: RenderChatMark
)
const citationSafeText = stripIncompleteCitationTag(imageContextSafeText)
const { text: tagSafe, tags } = preserveCitationTags(citationSafeText)
const imageSafe = replaceIncompleteImageWithPlaceholder(tagSafe)
const normalizedImageMarkdown = normalizeFullwidthMarkdownImageParentheses(tagSafe)
const imageSafe = replaceIncompleteImageWithPlaceholder(normalizedImageMarkdown)
const mathSafe = preprocessMathDelimiters(imageSafe)
const restoredTags = restoreCitationTags(mathSafe, tags)
const inlineTags = joinCitationTagsToPreviousLine(restoredTags)
+18 -1
View File
@@ -10,9 +10,19 @@ import (
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/searchutil"
"github.com/Tencent/WeKnora/internal/types"
)
func finalAnswerImageRequirement(hasRetrievedImage bool) string {
if !hasRetrievedImage {
return ""
}
return `
6. Retrieved tool results contain Markdown images. Unless the user explicitly requested text-only output or every image is clearly unrelated, the final answer MUST include at least one relevant Markdown image copied verbatim from the tool results. Preserve its complete URL exactly. Use ASCII half-width parentheses exactly as ![alt](url) and never use full-width ( or ). Place the image immediately after the paragraph it supports. When multiple images support different sections, distribute them across those sections instead of stopping after the first image.
7. Before finishing, silently verify that the answer contains a Markdown image when requirement 6 applies.`
}
// streamFinalAnswerToEventBus streams the final answer generation through EventBus
func (e *AgentEngine) streamFinalAnswerToEventBus(
ctx context.Context,
@@ -41,9 +51,13 @@ func (e *AgentEngine) streamFinalAnswerToEventBus(
// Add all tool call results as context
toolResultCount := 0
hasRetrievedImage := false
for stepIdx, step := range state.RoundSteps {
for toolIdx, toolCall := range step.ToolCalls {
toolResultCount++
if searchutil.MarkdownImageRegex.MatchString(toolCall.Result.Output) {
hasRetrievedImage = true
}
messages = append(messages, chat.Message{
Role: "user",
Content: fmt.Sprintf("Tool %s returned: %s", toolCall.Name, toolCall.Result.Output),
@@ -56,6 +70,8 @@ func (e *AgentEngine) streamFinalAnswerToEventBus(
logger.Debugf(ctx, "[Agent][FinalAnswer] Built context: %d messages, %d tool results",
len(messages), toolResultCount)
imageRequirement := finalAnswerImageRequirement(hasRetrievedImage)
// Add final answer prompt
finalPrompt := fmt.Sprintf(`Based on the above tool call results, generate a complete answer for the user's question.
@@ -67,8 +83,9 @@ Requirements:
3. Organize the answer in a structured format
4. If information is insufficient, honestly state so
5. IMPORTANT: Respond in the same language as the user's question
%s
Now generate the final answer:`, query)
Now generate the final answer:`, query, imageRequirement)
messages = append(messages, chat.Message{
Role: "user",
@@ -0,0 +1,24 @@
package agent
import (
"strings"
"testing"
)
func TestFinalAnswerImageRequirement(t *testing.T) {
if got := finalAnswerImageRequirement(false); got != "" {
t.Fatalf("text-only tool results should not add an image requirement: %q", got)
}
got := finalAnswerImageRequirement(true)
for _, required := range []string{
"MUST include at least one relevant Markdown image",
"Preserve its complete URL exactly",
"ASCII half-width parentheses",
"silently verify",
} {
if !strings.Contains(got, required) {
t.Fatalf("expected %q in final-answer image requirement:\n%s", required, got)
}
}
}
+46
View File
@@ -0,0 +1,46 @@
package agent
import (
"strings"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/searchutil"
"github.com/Tencent/WeKnora/internal/types"
)
const agentRetrievedImageRequirementMarker = "## Retrieved Image Output Requirement"
const agentRetrievedImageSystemRequirement = `
## Retrieved Image Output Requirement
Retrieved tool results for this turn contain Markdown images. Treat images attached to retrieved passages as relevant by default.
- Unless the user explicitly requests text-only output, or every retrieved image is clearly unrelated, the final answer MUST include at least one relevant Markdown image copied verbatim from the tool results.
- Preserve the complete Markdown image syntax and URL exactly; never invent, shorten, normalize, or replace the URL.
- Use ASCII half-width parentheses exactly as ![alt](url); never use full-width ( or ).
- Place each image immediately after the paragraph it supports.
- When multiple retrieved images support different sections, distribute them across those sections instead of stopping after the first image.
- Before finishing, silently verify that the answer contains a Markdown image whenever this requirement applies.`
func stepContainsMarkdownImage(step types.AgentStep) bool {
for _, toolCall := range step.ToolCalls {
if toolCall.Result != nil &&
toolCall.Result.Success &&
searchutil.MarkdownImageRegex.MatchString(toolCall.Result.Output) {
return true
}
}
return false
}
func appendAgentRetrievedImageRequirement(messages []chat.Message) []chat.Message {
for i := range messages {
if messages[i].Role != "system" {
continue
}
if !strings.Contains(messages[i].Content, agentRetrievedImageRequirementMarker) {
messages[i].Content = strings.TrimRight(messages[i].Content, " \t\r\n") + agentRetrievedImageSystemRequirement
}
break
}
return messages
}
+7
View File
@@ -424,6 +424,13 @@ func (e *AgentEngine) appendToolResults(
messages []chat.Message,
step types.AgentStep,
) []chat.Message {
if stepContainsMarkdownImage(step) {
// Keep the requirement at system priority even when a custom Agent prompt
// replaces the built-in template. Appending it once, when image-bearing
// evidence first appears, also avoids burdening text-only turns.
messages = appendAgentRetrievedImageRequirement(messages)
}
// Add assistant message with tool calls (if any)
if step.Thought != "" || len(step.ToolCalls) > 0 || step.ReasoningContent != "" {
assistantMsg := chat.Message{
+32
View File
@@ -3,6 +3,7 @@ package agent
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
@@ -162,6 +163,37 @@ func TestAppendToolResults_PreservesReasoningContent(t *testing.T) {
})
}
func TestAppendToolResults_AddsDynamicImageRequirementToCustomSystemPrompt(t *testing.T) {
engine := &AgentEngine{}
prior := []chat.Message{
{Role: "system", Content: "Custom agent prompt."},
{Role: "user", Content: "解释流程"},
}
step := types.AgentStep{
ToolCalls: []types.ToolCall{{
ID: "call-image",
Name: "knowledge_search",
Result: &types.ToolResult{
Success: true,
Output: "结果\n![流程图](resource://AbCdEfGhIjKlMnOpQrStUv)",
},
}},
}
out := engine.appendToolResults(prior, step)
require.Len(t, out, 4)
assert.Contains(t, out[0].Content, "Custom agent prompt.")
assert.Contains(t, out[0].Content, agentRetrievedImageRequirementMarker)
assert.Contains(t, out[0].Content, "MUST include at least one relevant Markdown image")
assert.Contains(t, out[0].Content, "ASCII half-width parentheses")
assert.Equal(t, "tool", out[3].Role)
assert.Contains(t, out[3].Content, "![流程图](resource://AbCdEfGhIjKlMnOpQrStUv)")
// A later image-bearing step must not duplicate the system requirement.
out = engine.appendToolResults(out, step)
assert.Equal(t, 1, strings.Count(out[0].Content, agentRetrievedImageRequirementMarker))
}
func TestBuildRuntimeContextBlock_PinnedDocuments(t *testing.T) {
block := buildRuntimeContextBlock(
"sess-1",
@@ -11,12 +11,31 @@ import (
"github.com/Tencent/WeKnora/internal/common"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/searchutil"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
var regThinkTags = regexp.MustCompile(`(?s)<think>.*?</think>`)
const retrievedImageOutputRequirement = `
## Retrieved Image Output Requirement
The retrieved context for this turn contains Markdown images. Images attached to retrieved passages should be treated as relevant by default.
- Unless the user explicitly requests text-only output, or every retrieved image is clearly unrelated to the answer, the final answer MUST include at least one relevant Markdown image copied from the retrieved context.
- Copy the complete Markdown image syntax and its URL verbatim. Never invent, shorten, normalize, or replace the URL.
- Use ASCII half-width parentheses in image Markdown exactly as ![alt](url). Never use full-width ( or ).
- Place each image immediately after the paragraph it supports, rather than collecting images at the end.
- When multiple retrieved images support different sections of a multi-section answer, include them in their corresponding sections instead of stopping after the first image.
- Before finishing, silently verify that the answer contains a Markdown image whenever this requirement applies.`
func appendRetrievedImageOutputRequirement(systemPrompt, renderedContexts string) string {
if !searchutil.MarkdownImageRegex.MatchString(renderedContexts) {
return systemPrompt
}
return strings.TrimRight(systemPrompt, " \t\r\n") + retrievedImageOutputRequirement
}
// pipelineInfo logs pipeline info level entries.
func pipelineInfo(ctx context.Context, stage, action string, fields map[string]interface{}) {
common.PipelineInfo(ctx, stage, action, fields)
@@ -75,6 +94,7 @@ func prepareMessagesWithHistory(chatManage *types.ChatManage) []chat.Message {
"language": chatManage.Language,
"contexts": chatManage.RenderedContexts,
})
systemPrompt = appendRetrievedImageOutputRequirement(systemPrompt, chatManage.RenderedContexts)
chatMessages := []chat.Message{
{Role: "system", Content: systemPrompt},
@@ -0,0 +1,30 @@
package chatpipeline
import (
"strings"
"testing"
)
func TestAppendRetrievedImageOutputRequirement(t *testing.T) {
base := "Answer from retrieved evidence."
withImage := appendRetrievedImageOutputRequirement(
base,
"context\n![流程图](resource://AbCdEfGhIjKlMnOpQrStUv)",
)
for _, required := range []string{
base,
"MUST include at least one relevant Markdown image",
"Copy the complete Markdown image syntax and its URL verbatim",
"ASCII half-width parentheses",
"immediately after the paragraph it supports",
} {
if !strings.Contains(withImage, required) {
t.Fatalf("expected %q in dynamic image requirement:\n%s", required, withImage)
}
}
withoutImage := appendRetrievedImageOutputRequirement(base, "text-only retrieved context")
if withoutImage != base {
t.Fatalf("text-only context should not change the system prompt: %q", withoutImage)
}
}