feat(llm): introduce request-local source reference aliases

Replace raw chunk/document/KB IDs in LLM context with short cN/dN/bN/wN
aliases, decode tool arguments on the way in, and expand compact <ref/>
citations to the public <kb>/<web> format before user-facing output.
This commit is contained in:
wizardchen
2026-07-16 21:23:20 +08:00
committed by lyingbug
parent 79e8f3d894
commit 014164ae44
33 changed files with 1962 additions and 149 deletions
@@ -76,7 +76,7 @@ templates:
### Critical Constraints (ABSOLUTE RULES)
1. **Evidence-Based Facts:** For factual claims about documents or domain knowledge, rely on KB/Web retrieval rather than internal knowledge. However, you MAY answer directly when the user's question is about image content you can see, conversational context, or general interaction.
2. **Mandatory Deep Read:** Whenever grep_chunks or knowledge_search returns matches, you **MUST** read full content before answering. For **FAQ** hits use `list_knowledge_chunks(faq_id=...)` or `get_document_info(faq_ids=[...])` with the **faq_id** from the result — do **not** use the parent knowledge_id of the FAQ container. For **documents**, use `list_knowledge_chunks(knowledge_id=...)` or `get_document_info(knowledge_ids=...)`. Do not rely on search snippets alone.
2. **Mandatory Deep Read:** Whenever grep_chunks or knowledge_search returns matches, you **MUST** read full content before answering. For **FAQ** hits use `list_knowledge_chunks(faq_id=cN)` or `get_document_info(faq_ids=[cN])` with the chunk's short **cN** ID — do **not** use its parent dN document ID. For **documents**, use `list_knowledge_chunks(knowledge_id=dN)` or `get_document_info(knowledge_ids=[dN])`. Do not rely on search snippets alone.
3. **Knowledge Base Priority:** When retrieval IS needed, always exhaust knowledge base strategies (including the Deep Read) before attempting Web Search (if enabled).
4. **Always Re-Retrieve for Each New Question:** You MUST perform fresh knowledge base retrieval for EVERY new user question that requires factual or domain-specific information, even if a similar or identical question was asked earlier in the conversation. NEVER rely on previously retrieved knowledge base content from the conversation history — the knowledge base may have been updated, switched, or had content removed since the last retrieval. Treat each new question as if you have no prior knowledge from previous retrievals.
5. **User-Friendly Communication:** In ALL outputs visible to users (including your thinking/reasoning process), you MUST:
@@ -134,7 +134,7 @@ templates:
1. **Entity Anchoring (grep_chunks):** Regex search over chunk content using PostgreSQL POSIX `~*` (case-insensitive; `REGEXP` on MySQL/SQLite). Behaves like `grep -E -i`. Input field is a single `query` string — pack 2-3 terms into ONE alternation regex (e.g. `stardust|skyvault|psionic`) rather than firing several calls. Plain literal text also works (`engine` matches anywhere in chunk content). Each match returns a `<match_snippet>` you can use to judge relevance before deep-reading.
2. **Semantic Expansion (knowledge_search):** Use vector search for context (filter by IDs from step 1 if applicable).
3. **Deep Contextualization (list_knowledge_chunks): MANDATORY.**
* Rule: After Step 1 or 2 returns knowledge_ids, you MUST call this tool.
* Rule: After Step 1 or 2 returns dN document IDs, you MUST call this tool with `knowledge_id=dN`.
* Frequency: Call it frequently for multiple IDs to ensure you have the full results. **Do not be lazy; fetch the content.**
4. **Graph Exploration (query_knowledge_graph):** Optional for relationships.
5. **Web Fallback (web_search):** Use ONLY if Web Search is Enabled AND the Deep Read in Step 3 confirms the data is missing or irrelevant.
@@ -149,17 +149,17 @@ templates:
### Final Output Standards
* **Definitive:** Based strictly on the "Deep Read" content.
* **Sourced (Inline Citations):** Factual claims must be cited using <kb doc="..." chunk_id="..." /> or <web url="..." title="..." /> (if from web).
**Citation rules (STRICT):**
- The citation tag must be placed **on the same line** as the last sentence of the paragraph it supports, with NO line break before it. It is part of the sentence/paragraph text flow.
- Do NOT repeat the same citation after every sentence. One citation per paragraph per source is enough.
- NEVER group all citations at the bottom of the answer. They must be distributed inline throughout the text.
- CORRECT: 扩容业务影响时间由天级降低到分钟级。<kb doc="文档A" chunk_id="xxx" />
- WRONG (line break before tag):
扩容业务影响时间由天级降低到分钟级。
<kb doc="文档A" chunk_id="xxx" />
- WRONG (all grouped at end of answer):
<kb doc="文档A" chunk_id="xxx" /><kb doc="文档B" chunk_id="yyy" />
* **Sourced (Inline Citations):** Factual claims must cite the short source ID attached to the supporting chunk or web page, using `<ref id="cN"/>` or `<ref id="wN"/>` exactly. The system expands it to the public citation format.
**Citation rules (STRICT):**
- The citation tag must be placed **on the same line** as the last sentence of the paragraph it supports, with NO line break before it. It is part of the sentence/paragraph text flow.
- Do NOT repeat the same citation after every sentence. One citation per paragraph per source is enough.
- NEVER group all citations at the bottom of the answer. They must be distributed inline throughout the text.
- CORRECT: 扩容业务影响时间由天级降低到分钟级。<ref id="c1"/>
- WRONG (line break before tag):
扩容业务影响时间由天级降低到分钟级。
<ref id="c1"/>
- WRONG (all grouped at end of answer):
<ref id="c1"/><ref id="c2"/>
* **Structured:** Clear hierarchy and logic.
* **Rich Media (Markdown with Images — REQUIRED):** When any retrieved chunk or tool result contains Markdown images or an "images" field with URLs, treat those images as relevant by default. Unless the user explicitly requests text-only output or every retrieved image is clearly unrelated, the final answer MUST include at least one relevant image using standard Markdown syntax: `![description](image_url)`. Copy the complete Markdown image and URL exactly. Parentheses MUST be ASCII half-width `(` and `)`; NEVER use full-width `(` or `)`. Place each image immediately after the paragraph it supports. When multiple images support different sections, distribute them across those sections instead of stopping after the first image. Silently verify this requirement before finishing.
@@ -204,7 +204,7 @@ templates:
### Workflow
1. **Understand:** Call data_schema to get table name, columns, types, and row count.
2. **Plan:** For complex questions, break the analysis into sub-queries. Plan internally by default; if `thinking` is in your tool list, you MAY reason out loud there, and if `todo_write` is available you MAY also track tasks with it.
3. **Query:** Call data_analysis with the knowledge_id and SQL query.
3. **Query:** Call data_analysis with the short dN document ID in knowledge_id and the SQL query.
4. **Analyze:** Interpret results and provide insights.
### SQL Best Practices for DuckDB
@@ -258,7 +258,7 @@ templates:
- **Outgoing links (`Links to`):** Use these to dive deeper into specific concepts, people, or items mentioned in the current page's text.
- **Incoming links (`Linked from`):** Use these to find broader context, discover what other projects/entities reference this topic, or see where this concept is applied.
If the current page doesn't fully answer the question, you MUST call `wiki_read_page` again on these related slugs (1-2 hops).
4. **Drill-down Fallback (Optional):** Wiki pages summarize information from source documents. If the user asks for highly specific details (e.g. exact quotes, raw data, or specific code snippets) that are not present in the wiki page, check the `<sources>` section of that wiki page. Use the `wiki_read_source_doc` tool with the provided `knowledge_id` to read the original raw document, optionally using a regex query to find the specific detail.
4. **Drill-down Fallback (Optional):** Wiki pages summarize information from source documents. If the user asks for highly specific details (e.g. exact quotes, raw data, or specific code snippets) that are not present in the wiki page, check the `<sources>` section of that wiki page. Use the `wiki_read_source_doc` tool with the provided short dN `knowledge_id` to read the original raw document, optionally using a regex query to find the specific detail.
5. **Auxiliary Tools (Optional):** Beyond the wiki itself, you may leverage auxiliary capabilities when they clearly help answer the question:
- **External MCP Tools (if any are exposed in your tool list):** Use them when the user's question requires real-time data, external system lookups, or actions that do NOT live inside the wiki (e.g. querying a ticketing system, calling an internal API, fetching live metrics). Treat their responses as additional evidence, not as a replacement for wiki content.
- **Skills (if an `### Available Skills` section appears later in this prompt):** Before answering, scan the listed skills. If the user's intent matches a skill's triggers, call `read_skill(skill_name="...")` to load its full instructions, then follow them. Skills are especially useful for specialized output formatting or domain-specific procedures.
@@ -282,8 +282,8 @@ templates:
<tool_guidelines>
* **wiki_search:** Use this to find entry points. STRONGLY PREFER using PostgreSQL POSIX regular expressions (~* operator) to search for multiple concepts at once efficiently. Examples: alternation ("stardust|skyvault|psionic"), multiple terms ("psionic.*engine"), prefix ("^entity/.*"). Do not use simple plain text queries if you are looking for multiple things. You can provide multiple queries at once to search in parallel.
* **wiki_read_page:** Your primary tool. Use it to read the full content of pages found via search, linked from other pages, or the special `index` and `log` pages. You can provide multiple slugs at once to read multiple pages in parallel.
* **wiki_read_source_doc:** Use this ONLY as a fallback when a wiki page's content is insufficient and you need to dive into the raw source document listed in the page's `<sources>` section. Use the `knowledge_id` and an optional regex `query` to efficiently find specific quotes or details, OR use `start_chunk_index` and `end_chunk_index` to fetch a contiguous range of chunks.
* **wiki_flag_issue:** Use this tool when you or the user identifies that a wiki page is factually incorrect, contains outdated information, or wrongly merges distinct entities (e.g., merging a competitor's product into the current page). This logs an issue for human review or automated maintenance. Provide a clear description and any suspected source `knowledge_id`s that might be causing the problem.
* **wiki_read_source_doc:** Use this ONLY as a fallback when a wiki page's content is insufficient and you need to dive into the raw source document listed in the page's `<sources>` section. Use its short dN `knowledge_id` and an optional regex `query` to efficiently find specific quotes or details, OR use `start_chunk_index` and `end_chunk_index` to fetch a contiguous range of chunks.
* **wiki_flag_issue:** Use this tool when you or the user identifies that a wiki page is factually incorrect, contains outdated information, or wrongly merges distinct entities (e.g., merging a competitor's product into the current page). This logs an issue for human review or automated maintenance. Provide a clear description and any suspected short dN source document IDs that might be causing the problem.
* **MCP Tools (dynamic):** Any tools whose names do NOT start with `wiki_` and are not the built-in `thinking` / `todo_write` are external MCP tools exposed by connected services. Use them when the task needs real-time or external data that the wiki cannot provide. Their results are evidence — cite them inline and never let them override facts already confirmed from the wiki.
* **read_skill / execute_skill_script (only if skills are exposed):** Load and run skills listed under `### Available Skills`. `read_skill` fetches full skill instructions; `execute_skill_script` runs a script shipped with a skill (only available when the sandbox is enabled).
* **Ending the turn:** When your research is complete, write your complete answer (with wiki-links, inline citations, and relevant images) as plain text and stop — do not request any tools in that final message.
@@ -322,7 +322,7 @@ templates:
- Call `wiki_update_issue` to mark the issue as "resolved" with a note.
- Do NOT make any edits to the page.
- Skip to step 8 and deliver a brief "already resolved" message as your reply. Do not continue to steps 4–7.
4. **Investigate Sources:** If the issue is confirmed to still exist and involves conflicting facts or mixed entities, the current wiki page might be poisoned. You MUST call `wiki_read_source_doc` using the `knowledge_id`s listed in the page's `<sources>` or provided in the issue description. Read the raw text to discover the truth.
4. **Investigate Sources:** If the issue is confirmed to still exist and involves conflicting facts or mixed entities, the current wiki page might be poisoned. You MUST call `wiki_read_source_doc` using the short dN `knowledge_id`s listed in the page's `<sources>` or provided in the issue description. Read the raw text to discover the truth.
5. **Determine the Fix Strategy:**
- *Correction:* Fix minor errors efficiently using the `wiki_replace_text` tool, or rewrite the page using `wiki_write_page`.
- *Renaming:* If the page title or slug is fundamentally wrong, use `wiki_rename_page` to change the slug. Incoming links will be updated automatically!
@@ -352,7 +352,7 @@ templates:
- Include a "## Key Takeaways" section with bullet points at the end.
- Preserve any valid image links `![alt](url)`.
- Use wiki-links `[[slug|display name]]` whenever mentioning entities/concepts that exist in the wiki.
9. **Source Refs:** When calling `wiki_write_page` or `wiki_replace_text`, you MUST provide the `source_refs` array containing the `knowledge_id`s of the source documents you used to verify the information.
9. **Source Refs:** When calling `wiki_write_page` or `wiki_replace_text`, you MUST provide the `source_refs` array containing the short dN IDs of the source documents you used to verify the information.
10. **Always End by Answering:** Your LAST action of every turn MUST be writing a concise summary of what was fixed (or why no fix was needed) as your reply, then stopping (no further tool calls in that final message).
11. **Strict Ontology & Anti-Duplication:** BEFORE creating any new page via `wiki_write_page`, you MUST perform a targeted deduplication check: use `wiki_search` (maximum 1-2 regex queries using alternation for synonyms/aliases). If a canonical page is found, you must MERGE the information into it rather than creating a duplicate graph node.
12. **Strict Citation Tracing:** Any new factual information injected via `wiki_replace_text` or `wiki_write_page` MUST be strictly grounded in the raw documents (`wiki_read_source_doc`). You are strictly forbidden from synthesizing or hallucinating external knowledge that is not present in the provided source chunks.
@@ -441,7 +441,7 @@ templates:
3. **Chunk Grounding (only when `CHUNK_KBS` is non-empty):**
MANDATORY for any factual claim the user will verify (exact quotes, numbers, code, names).
- Call `knowledge_search` (semantic) and/or `grep_chunks` (regex) scoped to `CHUNK_KBS`.
- Whenever a search returns matched `knowledge_id`s / `chunk_id`s, you MUST call `list_knowledge_chunks` to read the full content. Do NOT answer from search snippets alone.
- Whenever a search returns matched dN document IDs / cN chunk IDs, you MUST call `list_knowledge_chunks` to read the full content. Do NOT answer from search snippets alone.
- If the first pass returns nothing relevant AND `WIKI_KBS` is also non-empty, fall back to a wiki search before giving up — the wiki may still list the concept under a different phrasing.
- If `CHUNK_KBS` is empty, skip this step and rely on wiki, but make that limitation explicit in the final answer.
@@ -463,21 +463,21 @@ templates:
<citation_format>
Use EXACTLY these two formats (any other syntax renders as plain text):
- Chunk (verifiable facts: quotes, numbers, code, names): `<kb doc="..." chunk_id="..." kb_id="..." />`
- Chunk (verifiable facts: quotes, numbers, code, names): `<ref id="cN"/>`
- Wiki page (concepts / overviews / navigation): `[[slug|display name]]`
All values MUST come from tool results of this turn — never fabricate a `chunk_id`, `kb_id`, or `slug`.
The source ID MUST come from tool results of this turn — never fabricate a `cN`, `wN`, or `slug`.
Placement: inline on the same line as the sentence it supports, one per source per paragraph, never grouped at the end, never wrapped in backticks.
Example: WeKnora 采用混合检索。<kb doc="架构设计.md" chunk_id="abc123" kb_id="kb-01" /> 详见 [[hybrid-retrieval|混合检索]]。
Example: WeKnora 采用混合检索。<ref id="c1"/> 详见 [[hybrid-retrieval|混合检索]]。
</citation_format>
<constraints>
ABSOLUTE RULES:
1. **Evidence-Based Facts:** Ground every factual claim in content you retrieved *this turn*, either from wiki pages or from source chunks.
2. **Wiki Never Replaces RAG for Exact Quotes:** Wiki pages are summaries. If the user asks for a direct quote, numeric value, code, or table row, you MUST read the underlying chunks via `list_knowledge_chunks` or `wiki_read_source_doc` and cite them with `<kb .../>`.
3. **Mandatory Deep Read:** Whenever `grep_chunks` or `knowledge_search` returns matched `knowledge_id`s / `chunk_id`s, you MUST immediately call `list_knowledge_chunks` to read the full content. Do NOT rely on search snippets.
2. **Wiki Never Replaces RAG for Exact Quotes:** Wiki pages are summaries. If the user asks for a direct quote, numeric value, code, or table row, you MUST read the underlying chunks via `list_knowledge_chunks` or `wiki_read_source_doc` and cite the supporting chunk with `<ref id="cN"/>`.
3. **Mandatory Deep Read:** Whenever `grep_chunks` or `knowledge_search` returns matched dN document IDs / cN chunk IDs, you MUST immediately call `list_knowledge_chunks` to read the full content. Do NOT rely on search snippets.
4. **Mandatory Wiki Read:** `wiki_search` returns summaries only. If a wiki hit is relevant, you MUST call `wiki_read_page` before citing it via `[[slug|name]]`.
5. **Always Re-Retrieve:** Every new user question triggers fresh retrieval. Do not reuse retrieved content from earlier turns.
6. **Cite Both Sides When Both Contributed:** If a claim came from a wiki overview AND was verified against a source chunk, cite both.
@@ -500,7 +500,7 @@ templates:
* **list_knowledge_chunks:** MANDATORY after any chunk search — loads the full text of the matched chunks.
* **get_document_info:** Fetch metadata (title, upload time, page count) when you need to cite a document properly.
* **wiki_flag_issue:** Use when wiki and chunks disagree, or when the user points out a wiki error.
* **Ending the turn:** When your answer is ready, write it as plain text (with inline `<kb .../>` citations and `[[slug|display name]]` wiki-links) and stop — do not request any tools in that final message.
* **Ending the turn:** When your answer is ready, write it as plain text (with inline `<ref id="cN"/>` citations and `[[slug|display name]]` wiki-links) and stop — do not request any tools in that final message.
</tool_guidelines>
<system_status>
+5 -1
View File
@@ -14,6 +14,7 @@ import (
"github.com/Tencent/WeKnora/internal/common"
appconfig "github.com/Tencent/WeKnora/internal/config"
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/llmresource"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
@@ -51,6 +52,7 @@ type AgentEngine struct {
lastUsage types.TokenUsage // Token usage from the most recent LLM call
lastSentMsgCount int // Number of messages sent in the most recent LLM call
resourceRefs *llmresource.Registry // request-local aliases for durable resource references
sourceRefs *llmreference.Registry // request-local chunk/document/web aliases and citations
}
// ImageDescriberFunc generates a text description of an image.
@@ -86,6 +88,7 @@ func NewAgentEngine(
systemPromptTemplate: systemPromptTemplate,
tokenEstimator: tokenEst,
resourceRefs: llmresource.NewRegistry(),
sourceRefs: llmreference.NewRegistry(),
}
// Initialize memory consolidator if context window management is configured
@@ -116,12 +119,13 @@ func (e *AgentEngine) systemPromptOptions(ctx context.Context) *BuildSystemPromp
}
func (e *AgentEngine) buildSystemPrompt(ctx context.Context) string {
return BuildSystemPromptWithOptions(
prompt := BuildSystemPromptWithOptions(
e.knowledgeBasesInfo,
e.config.WebSearchEnabled,
e.systemPromptOptions(ctx),
e.systemPromptTemplate,
)
return strings.TrimRight(prompt, " \t\r\n") + llmreference.ProtocolPrompt
}
// NewAgentEngineWithSkills creates a new agent engine with skills support
+17
View File
@@ -7,6 +7,7 @@ import (
"testing"
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/assert"
@@ -69,6 +70,22 @@ func TestStreamLLMResourceAliasesRoundTrip(t *testing.T) {
require.Equal(t, "source=res://0001", model.calls[0][0].Content)
}
func TestStreamLLMChunkReferenceExpandsBeforeEmission(t *testing.T) {
model := &mockChat{responses: []mockResponse{{chunks: []types.StreamResponse{
{ResponseType: types.ResponseTypeAnswer, Content: `answer <ref id="`},
{ResponseType: types.ResponseTypeAnswer, Content: `c1"/>`, Done: true},
}}}}
engine := newTestEngine(t, model)
engine.sourceRefs.RegisterChunk(llmreference.ChunkReference{
ChunkID: "chunk-1",
KnowledgeBaseID: "kb-1",
DocumentTitle: "Doc",
})
result, err := engine.streamLLMToEventBus(context.Background(), nil, nil, nil)
require.NoError(t, err)
require.Equal(t, `answer <kb doc="Doc" chunk_id="chunk-1" kb_id="kb-1" />`, result.Content)
}
func (m *mockChat) Chat(_ context.Context, _ []chat.Message, _ *chat.ChatOptions) (*types.ChatResponse, error) {
return nil, fmt.Errorf("not implemented")
}
+3 -2
View File
@@ -58,9 +58,10 @@ func (e *AgentEngine) streamFinalAnswerToEventBus(
if searchutil.MarkdownImageRegex.MatchString(toolCall.Result.Output) {
hasRetrievedImage = true
}
modelOutput := e.sourceRefs.ModelOutput(toolCall.Result)
messages = append(messages, chat.Message{
Role: "user",
Content: fmt.Sprintf("Tool %s returned: %s", toolCall.Name, toolCall.Result.Output),
Content: fmt.Sprintf("Tool %s returned: %s", toolCall.Name, modelOutput),
})
logger.Debugf(ctx, "[Agent][FinalAnswer] Added tool result [Step-%d][Tool-%d]: %s (output: %d chars)",
stepIdx+1, toolIdx+1, toolCall.Name, len(toolCall.Result.Output))
@@ -79,7 +80,7 @@ User question: %s
Requirements:
1. Answer based on the actually retrieved content
2. Cite sources by document title or name; never expose chunk_id, knowledge_id, tool names, or other internal identifiers
2. Cite supporting knowledge chunks and web pages only with their supplied compact form: <ref id="cN"/> or <ref id="wN"/>
3. Organize the answer in a structured format
4. If information is insufficient, honestly state so
5. IMPORTANT: Respond in the same language as the user's question
+55 -9
View File
@@ -11,6 +11,7 @@ import (
agenttools "github.com/Tencent/WeKnora/internal/agent/tools"
"github.com/Tencent/WeKnora/internal/common"
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
@@ -374,11 +375,59 @@ func commonStringPrefix(a, b string) string {
// (runtime_context + must_use + query). Used by Execute and finalize paths only;
// not written to rendered_content / history.
func (e *AgentEngine) RenderUserTurnContent(sessionID, query string) string {
e.registerRuntimeReferences()
runtimeCtx := buildRuntimeContextBlock(sessionID, e.knowledgeBasesInfo, e.selectedDocs)
runtimeCtx = e.sourceRefs.CompactKnownText(runtimeCtx)
mustUse := buildMustUseBlock(e.pinnedMCPServices, e.pinnedSkills)
return composeUserTurnContent(runtimeCtx, mustUse, query)
}
// registerRuntimeReferences makes bound KBs, pinned documents and recent
// chunks addressable without exposing their durable IDs to the model.
func (e *AgentEngine) registerRuntimeReferences() {
if e == nil || e.sourceRefs == nil {
return
}
for _, kb := range e.knowledgeBasesInfo {
if kb == nil {
continue
}
e.sourceRefs.RegisterKnowledgeBase(kb.ID)
for _, doc := range kb.RecentDocs {
e.sourceRefs.RegisterDocument(doc.KnowledgeID)
if doc.ChunkID != "" {
title := doc.Title
if title == "" {
title = doc.FileName
}
e.sourceRefs.RegisterChunk(llmreference.ChunkReference{
ChunkID: doc.ChunkID,
KnowledgeID: doc.KnowledgeID,
KnowledgeBaseID: firstNonEmptyAgent(doc.KnowledgeBaseID, kb.ID),
DocumentTitle: title,
ChunkType: doc.Type,
})
}
}
}
for _, doc := range e.selectedDocs {
if doc == nil {
continue
}
e.sourceRefs.RegisterDocument(doc.KnowledgeID)
e.sourceRefs.RegisterKnowledgeBase(doc.KnowledgeBaseID)
}
}
func firstNonEmptyAgent(values ...string) string {
for _, value := range values {
if value != "" {
return value
}
}
return ""
}
func composeUserTurnContent(parts ...string) string {
nonEmpty := make([]string, 0, len(parts))
for _, part := range parts {
@@ -463,10 +512,7 @@ func (e *AgentEngine) appendToolResults(
// Add tool result messages (role: "tool", following OpenAI format)
for _, toolCall := range step.ToolCalls {
resultContent := toolCall.Result.Output
if !toolCall.Result.Success {
resultContent = fmt.Sprintf("Error: %s", toolCall.Result.Error)
}
resultContent := e.sourceRefs.ModelOutput(toolCall.Result)
toolMsg := chat.Message{
Role: "tool",
@@ -556,13 +602,13 @@ func (e *AgentEngine) buildMessagesWithLLMContext(
}
}
// Build user message with per-turn scope envelopes (current turn only).
// Historical user messages in llmContext stay as bare Content from the DB.
runtimeCtx := buildRuntimeContextBlock(sessionID, e.knowledgeBasesInfo, e.selectedDocs)
mustUse := buildMustUseBlock(e.pinnedMCPServices, e.pinnedSkills)
// Build the current user message through the same registration path used by
// final synthesis. Calling buildRuntimeContextBlock directly here would put
// durable bound-KB/document IDs into the first model request before the
// request-local source registry had seen them.
userMsg := chat.Message{
Role: "user",
Content: composeUserTurnContent(runtimeCtx, mustUse, currentQuery),
Content: e.RenderUserTurnContent(sessionID, currentQuery),
Images: imageURLs,
}
messages = append(messages, userMsg)
+33
View File
@@ -8,6 +8,7 @@ import (
"time"
agenttools "github.com/Tencent/WeKnora/internal/agent/tools"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/assert"
@@ -271,6 +272,38 @@ func TestRenderUserTurnContent_IncludesScopeBlocks(t *testing.T) {
assert.Contains(t, out, "hello")
}
func TestBuildMessagesWithLLMContextRegistersBoundScopeBeforeFirstModelCall(t *testing.T) {
engine := &AgentEngine{
sourceRefs: llmreference.NewRegistry(),
knowledgeBasesInfo: []*KnowledgeBaseInfo{{
ID: "kb-real-id",
Name: "Docs",
RecentDocs: []RecentDocInfo{{
ChunkID: "chunk-real-id",
KnowledgeID: "doc-real-id",
KnowledgeBaseID: "kb-real-id",
Title: "Guide",
}},
}},
selectedDocs: []*SelectedDocumentInfo{{
KnowledgeID: "selected-doc-real-id",
KnowledgeBaseID: "kb-real-id",
Title: "Selected",
}},
}
messages := engine.buildMessagesWithLLMContext("system", "question", "session", nil, nil)
require.Len(t, messages, 2)
userContent := messages[1].Content
assert.Contains(t, userContent, `knowledge_base id="b1"`)
assert.Contains(t, userContent, `knowledge_id="d1"`)
assert.Contains(t, userContent, `knowledge_id="d2"`)
assert.Equal(t, "c1", engine.sourceRefs.ChunkAlias("chunk-real-id"))
assert.NotContains(t, userContent, "kb-real-id")
assert.NotContains(t, userContent, "chunk-real-id")
assert.NotContains(t, userContent, "doc-real-id")
}
func TestBuildMustUseBlock_MultiWordServicePrefix(t *testing.T) {
// Service "My Service" -> tools mcp_my_service_*; the prefix must be the
// full service slug, not the first underscore segment (mcp_my_).
+12 -4
View File
@@ -9,6 +9,7 @@ import (
agenttools "github.com/Tencent/WeKnora/internal/agent/tools"
"github.com/Tencent/WeKnora/internal/common"
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/llmresource"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
@@ -38,6 +39,7 @@ func (e *AgentEngine) streamLLMToEventBus(
llmCtx, llmCancel := context.WithTimeout(ctx, e.getLLMCallTimeout())
defer llmCancel()
messages = e.sourceRefs.EncodeMessages(messages)
messages = e.resourceRefs.EncodeMessages(messages)
stream, err := e.chatModel.ChatStream(llmCtx, messages, opts)
if err != nil {
@@ -51,6 +53,8 @@ func (e *AgentEngine) streamLLMToEventBus(
firstChunkTime := time.Time{}
answerDecoder := llmresource.NewStreamDecoder(e.resourceRefs)
thinkingDecoder := llmresource.NewStreamDecoder(e.resourceRefs)
answerRefExpander := llmreference.NewStreamExpander(e.sourceRefs)
thinkingRefExpander := llmreference.NewStreamExpander(e.sourceRefs)
for chunk := range stream {
chunkCount++
@@ -67,11 +71,12 @@ func (e *AgentEngine) streamLLMToEventBus(
continue
}
if chunk.ResponseType == types.ResponseTypeThinking {
chunk.Content = thinkingDecoder.Feed(chunk.Content)
chunk.Content = thinkingRefExpander.Feed(thinkingDecoder.Feed(chunk.Content))
} else {
chunk.Content = answerDecoder.Feed(chunk.Content)
chunk.Content = answerRefExpander.Feed(answerDecoder.Feed(chunk.Content))
}
e.resourceRefs.DecodeToolCalls(chunk.ToolCalls)
e.sourceRefs.DecodeToolCalls(chunk.ToolCalls)
if chunk.Content != "" {
isExtracted := chunk.Data != nil && chunk.Data["source"] != nil
@@ -100,12 +105,15 @@ func (e *AgentEngine) streamLLMToEventBus(
emitFunc(&chunk, result.Content)
}
}
result.Content += answerDecoder.Flush()
result.ReasoningContent += thinkingDecoder.Flush()
result.Content += answerRefExpander.Feed(answerDecoder.Flush())
result.Content += answerRefExpander.Flush()
result.ReasoningContent += thinkingRefExpander.Feed(thinkingDecoder.Flush())
result.ReasoningContent += thinkingRefExpander.Flush()
// Some providers stream tool-call argument fragments but expose the
// accumulated call in the final chunk. Decode once more after assembly so
// aliases split across provider chunks cannot leak into tool execution.
e.resourceRefs.DecodeToolCalls(result.ToolCalls)
e.sourceRefs.DecodeToolCalls(result.ToolCalls)
if orphans := e.resourceRefs.OrphanAliases(result.Content); len(orphans) > 0 {
logger.Warnf(ctx, "[Agent][Stream] Model emitted %d unresolvable resource alias(es): %v",
len(orphans), orphans)
+1 -1
View File
@@ -103,7 +103,7 @@ func buildMissingColumnSuggestion(sqlErr error, schema *TableSchema) string {
}
type DataAnalysisInput struct {
KnowledgeID string `json:"knowledge_id" jsonschema:"id of the knowledge to query"`
KnowledgeID string `json:"knowledge_id" jsonschema:"short dN document ID to query"`
Sql string `json:"sql" jsonschema:"SQL to be executed on knowledge"`
}
+1 -1
View File
@@ -17,7 +17,7 @@ var dataSchemaTool = BaseTool{
}
type DataSchemaInput struct {
KnowledgeID string `json:"knowledge_id" jsonschema:"id of the knowledge to query"`
KnowledgeID string `json:"knowledge_id" jsonschema:"short dN document ID to query"`
}
type DataSchemaTool struct {
+4 -4
View File
@@ -43,20 +43,20 @@ Do not use when:
- Can check document processing status (parse_status)
## IDs
- knowledge_ids: regular documents knowledges
- faq_ids: individual FAQ entries. Returns the standard question and answers, not the container title.`,
- knowledge_ids: regular documents, using the short dN IDs from retrieval results
- faq_ids: individual FAQ entries, using the short cN chunk IDs. Returns the standard question and answers, not the container title.`,
schema: json.RawMessage(`{
"type": "object",
"properties": {
"knowledge_ids": {
"type": "array",
"items": { "type": "string" },
"description": "Document/knowledge IDs for regular documents"
"description": "Short dN document IDs for regular documents"
},
"faq_ids": {
"type": "array",
"items": { "type": "string" },
"description": "FAQ entry IDs (= chunk_id from grep_chunks). Use instead of knowledge_ids for a single FAQ Q&A."
"description": "Short cN FAQ chunk IDs from retrieval results. Use instead of knowledge_ids for a single FAQ Q&A."
}
}
}`),
+3 -3
View File
@@ -20,7 +20,7 @@ var grepChunksTool = BaseTool{
name: ToolGrepChunks,
description: `Search knowledge base chunk content with a single POSIX regular expression, applied directly in the database (PostgreSQL ~* / MySQL/SQLite REGEXP, case-insensitive). Behaves like ` + "`grep -E -i`" + `.
Pack multiple concepts into ONE regex using ` + "`|`" + ` alternation — do not call this tool repeatedly for synonyms.
Returns matching chunks with hit counts and a <match_snippet> around the first match (each tagged with its knowledge_id and chunk_id).
Returns matching chunks with a short cN chunk source ID, a parent dN document ID, and a <match> snippet around the first match.
Examples:
- Alternation (RECOMMENDED): "stardust|skyvault|psionic" (matches any of the words)
- Multiple terms in order: "psionic.*engine" (matches both words in order)
@@ -30,8 +30,8 @@ IMPORTANT — JSON escaping: every backslash in a regex MUST be written as \\ in
Use this to locate candidate chunks by exact identifiers, error codes, product names, or recurring terms.
## Deep read after grep:
- **FAQ hit** (chunk type faq): call list_knowledge_chunks with **faq_id** from the grep result (NOT the parent knowledge_id).
- **Document hit**: call list_knowledge_chunks with **knowledge_id**, or get_document_info with **knowledge_ids**.`,
- **FAQ hit** (chunk type faq): call list_knowledge_chunks with **faq_id=cN** from the grep result (NOT the parent dN document ID).
- **Document hit**: call list_knowledge_chunks with **knowledge_id=dN**, or get_document_info with **knowledge_ids=[dN]**.`,
schema: json.RawMessage(`{
"type": "object",
"properties": {
+2 -2
View File
@@ -75,7 +75,7 @@ Avoid:
## Output
Returns chunks ranked by semantic similarity, reranked when applicable.
Results represent conceptual relevance, not literal keyword overlap.`,
Each chunk has a short cN source ID and belongs to a dN document ID. Results represent conceptual relevance, not literal keyword overlap. Use dN for document-level follow-up tool calls and cN for chunk citations.`,
schema: json.RawMessage(`{
"type": "object",
"properties": {
@@ -90,7 +90,7 @@ Results represent conceptual relevance, not literal keyword overlap.`,
},
"knowledge_base_ids": {
"type": "array",
"description": "Optional: KB IDs to search",
"description": "Optional: bound knowledge-base IDs (the short bN values shown in runtime context)",
"items": {
"type": "string"
},
@@ -16,13 +16,13 @@ var listKnowledgeChunksTool = BaseTool{
description: `Retrieve full chunk content for a document or a single FAQ entry.
## Use After grep_chunks or knowledge_search:
- **FAQ hit** (type faq): list_knowledge_chunks(faq_id="<chunk_id from search>") — reads that one FAQ entry with answers from metadata.
- **Document hit**: list_knowledge_chunks(knowledge_id="<document id>") — pages through all chunks.
- **FAQ hit** (type faq): list_knowledge_chunks(faq_id="cN") — reads that one FAQ chunk with answers from metadata.
- **Document hit**: list_knowledge_chunks(knowledge_id="dN") — pages through all chunks.
## Parameters (provide exactly one id target):
- faq_id (optional): FAQ entry ID from grep_chunks / knowledge_search.
- chunk_id (optional): Single non-FAQ chunk ID (do not use for FAQ — use faq_id).
- knowledge_id (optional): Document/knowledge ID to page through all chunks.
- faq_id (optional): Short cN ID for an FAQ chunk from grep_chunks / knowledge_search.
- chunk_id (optional): Short cN ID for a single non-FAQ chunk.
- knowledge_id (optional): Short dN document ID to page through all chunks.
- limit / offset: Only for knowledge_id paging (default limit 20, max 100).
## Output:
@@ -32,15 +32,15 @@ Full chunk content. FAQ entries include <faq> with <answer> from metadata.`,
"properties": {
"faq_id": {
"type": "string",
"description": "FAQ entry ID (same as chunk_id). Use for FAQ hits instead of knowledge_id."
"description": "Short cN FAQ chunk ID. Use for FAQ hits instead of the parent dN document ID."
},
"chunk_id": {
"type": "string",
"description": "Single chunk ID (alias of faq_id)"
"description": "Short cN ID for one non-FAQ chunk"
},
"knowledge_id": {
"type": "string",
"description": "Document/knowledge ID to list all chunks"
"description": "Short dN document ID to list all chunks"
},
"limit": {
"type": "integer",
+13 -10
View File
@@ -37,7 +37,7 @@ Explores relationships between entities in knowledge bases that have graph extra
- Need exact document content → use knowledge_search
## Parameters
- **knowledge_base_ids** (required): Array of knowledge base IDs (1-10). Only KBs with graph extraction configured will be effective.
- **knowledge_base_ids** (required): Array of short bN knowledge base IDs (1-10). Only KBs with graph extraction configured will be effective.
- **query** (required): Query content - can be entity name, relationship query, or concept search.
## Graph Configuration
@@ -61,7 +61,7 @@ If KB is not configured with graph, tool will return regular search results.
// QueryKnowledgeGraphInput defines the input parameters for query knowledge graph tool
type QueryKnowledgeGraphInput struct {
KnowledgeBaseIDs []string `json:"knowledge_base_ids" jsonschema:"Array of knowledge base IDs to query"`
KnowledgeBaseIDs []string `json:"knowledge_base_ids" jsonschema:"Array of short bN knowledge base IDs to query"`
Query string `json:"query" jsonschema:"Query content (entity name or query text)"`
}
@@ -299,14 +299,17 @@ func (t *QueryKnowledgeGraphTool) Execute(ctx context.Context, args json.RawMess
output += fmt.Sprintf(" 🆔 chunk_id: %s\n\n", result.ID)
formattedResults = append(formattedResults, map[string]interface{}{
"result_index": i + 1,
"chunk_id": result.ID,
"content": result.Content,
"score": result.Score,
"relevance_level": relevanceLevel,
"knowledge_id": result.KnowledgeID,
"knowledge_title": result.KnowledgeTitle,
"match_type": FormatMatchType(result.MatchType),
"result_index": i + 1,
"chunk_id": result.ID,
"chunk_index": result.ChunkIndex,
"chunk_type": result.ChunkType,
"content": result.Content,
"score": result.Score,
"relevance_level": relevanceLevel,
"knowledge_id": result.KnowledgeID,
"knowledge_base_id": result.KnowledgeBaseID,
"knowledge_title": result.KnowledgeTitle,
"match_type": FormatMatchType(result.MatchType),
})
}
+4 -4
View File
@@ -29,10 +29,10 @@ const (
var webFetchTool = BaseTool{
name: ToolWebFetch,
description: `Fetch detailed web content from previously discovered URLs and analyze it with an LLM.
description: `Fetch detailed web content from previously discovered short wN web page IDs and analyze it with an LLM.
## Usage
- Receive one or more {url, prompt} combinations
- Receive one or more {url: "wN", prompt} combinations; the field name stays url, but its value is the short page ID
- Fetch web page content and convert to Markdown text
- Use prompt to call small model for analysis and summary (if model is available)
- Return summary result and original content fragment
@@ -45,12 +45,12 @@ var webFetchTool = BaseTool{
// WebFetchInput defines the input parameters for web fetch tool
type WebFetchInput struct {
Items []WebFetchItem `json:"items" jsonschema:"Batch fetch tasks, each containing a url and prompt"`
Items []WebFetchItem `json:"items" jsonschema:"Batch fetch tasks, each containing a short wN web page ID and prompt"`
}
// WebFetchItem represents a single web fetch task
type WebFetchItem struct {
URL string `json:"url" jsonschema:"URL of the web page to fetch, should come from web_search results"`
URL string `json:"url" jsonschema:"Short wN web page ID from web_search results"`
Prompt string `json:"prompt" jsonschema:"Prompt for analyzing the fetched web page content"`
}
+3 -3
View File
@@ -41,7 +41,7 @@ var webSearchTool = BaseTool{
**Parameters**:
- query (required): Search query string
**Returns**: Web search results with title, URL, snippet, and content (up to %d results)
**Returns**: Web search results with title, short wN page ID, snippet, and content (up to %d results)
## Examples
@@ -62,8 +62,8 @@ var webSearchTool = BaseTool{
- Results are automatically compressed using RAG to extract relevant content
- Search results are stored in a temporary knowledge base for the session
- Use this tool when knowledge bases don't have the information you need
- Results include URL, title, snippet, and content snippet (may be truncated)
- **CRITICAL**: If content is truncated or you need full details, use **web_fetch** to fetch complete page content
- Results include a short wN page ID, title, snippet, and content snippet (may be truncated)
- **CRITICAL**: If content is truncated or you need full details, pass that wN value to **web_fetch** to fetch complete page content
- Maximum %d results will be returned per search`,
schema: utils.GenerateSchema[WebSearchInput](),
}
+3 -3
View File
@@ -42,7 +42,7 @@ This will log an issue for human review or automated maintenance.`,
"suspected_knowledge_ids": {
"type": "array",
"items": { "type": "string" },
"description": "Optional list of knowledge_ids (from the <sources> block) that you suspect are causing the pollution or error."
"description": "Optional list of short dN document IDs from the <sources> block that you suspect are causing the pollution or error."
}
},
"required": ["slug", "issue_type", "description"]
@@ -72,7 +72,7 @@ func (t *wikiFlagIssueTool) Execute(ctx context.Context, args json.RawMessage) (
if len(t.kbIDs) == 0 {
return &types.ToolResult{Success: false, Error: "No knowledge bases available for issue tracking"}, nil
}
// Default to first KB ID if multiple (normally there's only one in this context)
kbID := t.kbIDs[0]
@@ -102,4 +102,4 @@ func (t *wikiFlagIssueTool) Execute(ctx context.Context, args json.RawMessage) (
Success: true,
Output: fmt.Sprintf("Successfully flagged issue for %s. A maintenance ticket has been created for review.", slug),
}, nil
}
}
+32 -2
View File
@@ -31,7 +31,7 @@ If neither query nor range is provided, it returns the beginning of the document
"properties": {
"knowledge_id": {
"type": "string",
"description": "The ID of the source document to read"
"description": "The short dN source document ID from the <sources> block"
},
"query": {
"type": "string",
@@ -161,12 +161,27 @@ func (t *wikiReadSourceDocTool) Execute(ctx context.Context, args json.RawMessag
}
var chunksOutput strings.Builder
formattedChunks := make([]map[string]interface{}, 0)
totalChunks := int64(0)
reachedMax := false
var prevChunk *types.Chunk
var forceOutputNext bool
outputtedIndices := make(map[int]bool)
appendFormattedChunk := func(chunk *types.Chunk, content string) {
if chunk == nil {
return
}
formattedChunks = append(formattedChunks, map[string]interface{}{
"chunk_id": chunk.ID,
"chunk_index": chunk.ChunkIndex,
"chunk_type": chunk.ChunkType,
"content": content,
"knowledge_id": knowledgeID,
"knowledge_base": knowledge.KnowledgeBaseID,
"knowledge_title": knowledge.Title,
})
}
for {
pagination := &types.Pagination{
@@ -213,6 +228,7 @@ func (t *wikiReadSourceDocTool) Execute(ctx context.Context, args json.RawMessag
break
}
fmt.Fprintf(&chunksOutput, "<chunk index=\"%d\" type=\"range\">\n%s\n</chunk>\n", chunkNum, chunkContent)
appendFormattedChunk(c, chunkContent)
matchCount++
continue
}
@@ -232,6 +248,7 @@ func (t *wikiReadSourceDocTool) Execute(ctx context.Context, args json.RawMessag
if prevChunk != nil && !outputtedIndices[prevChunk.ChunkIndex] {
prevContent := enrichChunkContent(prevChunk)
fmt.Fprintf(&chunksOutput, "<chunk index=\"%d\" type=\"context_before\">\n%s\n</chunk>\n", prevChunk.ChunkIndex+1, prevContent)
appendFormattedChunk(prevChunk, prevContent)
outputtedIndices[prevChunk.ChunkIndex] = true
}
}
@@ -242,6 +259,7 @@ func (t *wikiReadSourceDocTool) Execute(ctx context.Context, args json.RawMessag
matchAttr = ` type="match"`
}
fmt.Fprintf(&chunksOutput, "<chunk index=\"%d\"%s>\n%s\n</chunk>\n", c.ChunkIndex+1, matchAttr, chunkContent)
appendFormattedChunk(c, chunkContent)
outputtedIndices[c.ChunkIndex] = true
}
@@ -251,6 +269,7 @@ func (t *wikiReadSourceDocTool) Execute(ctx context.Context, args json.RawMessag
} else if forceOutputNext {
if !outputtedIndices[c.ChunkIndex] {
fmt.Fprintf(&chunksOutput, "<chunk index=\"%d\" type=\"context_after\">\n%s\n</chunk>\n", c.ChunkIndex+1, chunkContent)
appendFormattedChunk(c, chunkContent)
outputtedIndices[c.ChunkIndex] = true
}
forceOutputNext = false
@@ -312,5 +331,16 @@ func (t *wikiReadSourceDocTool) Execute(ctx context.Context, args json.RawMessag
sb.WriteString("</source_document>")
return &types.ToolResult{Success: true, Output: sb.String()}, nil
return &types.ToolResult{
Success: true,
Output: sb.String(),
Data: map[string]interface{}{
"display_type": "knowledge_chunks_list",
"knowledge_id": knowledgeID,
"knowledge_title": knowledge.Title,
"total_chunks": totalChunks,
"fetched_chunks": len(formattedChunks),
"chunks": formattedChunks,
},
}, nil
}
+1 -1
View File
@@ -41,7 +41,7 @@ func NewWikiReplaceTextTool(wikiPageService interfaces.WikiPageService, kbIDs []
"source_refs": {
"type": "array",
"items": {"type": "string"},
"description": "An optional list of source knowledge IDs (UUIDs only) that justify this change. If provided, these will COMPLETELY REPLACE the existing source_refs of the page."
"description": "An optional list of short dN source document IDs that justify this change. If provided, these will COMPLETELY REPLACE the existing source_refs of the page."
}
},
"required": ["slug", "old_text", "new_text"]
+4 -4
View File
@@ -284,7 +284,7 @@ func NewWikiReadPageTool(
ToolWikiReadPage,
`Read one or more wiki pages by their slugs. Returns the full markdown content, metadata, and links.
Use this to read specific wiki pages when you know their slug (e.g. "entity/acme-corp", "concept/rag").
When the same slug exists in multiple knowledge bases, all matching pages are returned (each tagged with its knowledge_base_id). Pass "knowledge_base_id" to limit to a specific KB.`,
When the same slug exists in multiple knowledge bases, all matching pages are returned (each tagged with its short bN knowledge_base_id). Pass that bN value in "knowledge_base_id" to limit to a specific KB.`,
json.RawMessage(`{
"type": "object",
"properties": {
@@ -295,7 +295,7 @@ When the same slug exists in multiple knowledge bases, all matching pages are re
},
"knowledge_base_id": {
"type": "string",
"description": "Optional: specific knowledge base ID. If omitted, reads the slug from every wiki KB in scope (all matches returned)."
"description": "Optional: specific short bN knowledge base ID. If omitted, reads the slug from every wiki KB in scope (all matches returned)."
}
},
"required": ["slugs"]
@@ -576,7 +576,7 @@ func NewWikiSearchTool(
ToolWikiSearch,
`Search wiki pages using PostgreSQL POSIX regular expressions (~* operator, case-insensitive).
STRONGLY PREFER using regex to search for multiple concepts at once rather than simple plain text queries.
Returns matching pages with titles, slugs, and summaries (each tagged with its knowledge_base_id).
Returns matching pages with titles, slugs, and summaries (each tagged with its short bN knowledge_base_id).
Examples:
- Alternation (RECOMMENDED): "stardust|skyvault" (matches either word)
- Multiple terms (RECOMMENDED): "psionic.*engine" (matches both words in order)
@@ -598,7 +598,7 @@ Use this to find relevant wiki pages when you don't know the exact slug.`,
},
"knowledge_base_id": {
"type": "string",
"description": "Optional: restrict search to a single knowledge base ID in scope."
"description": "Optional: restrict search to a single short bN knowledge base ID in scope."
}
},
"required": ["queries"]
+1 -1
View File
@@ -56,7 +56,7 @@ func NewWikiWritePageTool(wikiPageService interfaces.WikiPageService, kbIDs []st
"source_refs": {
"type": "array",
"items": {"type": "string"},
"description": "A list of source knowledge IDs (UUIDs only) that contributed to this page. If provided, these will COMPLETELY REPLACE the existing source_refs of the page."
"description": "A list of short dN source document IDs that contributed to this page. If provided, these will COMPLETELY REPLACE the existing source_refs of the page."
}
},
"required": ["slug", "title", "summary", "content", "page_type"]
+10 -19
View File
@@ -117,27 +117,18 @@ func LoadAgentHistory(
}
// buildUserHistoryMessage converts a stored user message into the chat.Message
// form that should appear in LLM history. It prefers RenderedContent (which
// already contains any retrieval-context augmentation from prior turns) and
// falls back to Content. When the user uploaded images, the captions extracted
// during ingestion are appended so subsequent turns retain a textual reference
// to those images even after the binary payload is gone.
// form that should appear in LLM history. It deliberately ignores
// RenderedContent: that field is a snapshot of the old prompt and retrieval
// context format, which must not be mixed into the current request protocol.
// Image captions and attachments are reconstructed from their canonical DB
// columns so useful user-provided context is retained without stale RAG data.
func buildUserHistoryMessage(m *types.Message) chat.Message {
content := m.RenderedContent
if content == "" {
content = m.Content
content := m.Content
if captions := extractImageCaptionsFromMessage(m.Images); captions != "" {
content += "\n\n[用户上传图片内容]\n" + captions
}
// Only append fallbacks when RenderedContent is absent — when present, it
// already carries the augmented version persisted by the original turn.
// Agent-mode turns do not persist RenderedContent (scope envelopes are
// injected only for the current LLM call, not replayed from history).
if m.RenderedContent == "" {
if captions := extractImageCaptionsFromMessage(m.Images); captions != "" {
content += "\n\n[用户上传图片内容]\n" + captions
}
if len(m.Attachments) > 0 {
content += m.Attachments.BuildPrompt()
}
if len(m.Attachments) > 0 {
content += m.Attachments.BuildPrompt()
}
return chat.Message{Role: "user", Content: content}
}
@@ -11,12 +11,9 @@ import (
"github.com/stretchr/testify/require"
)
// TestBuildUserHistoryMessage_PrefersRenderedContent verifies the user side of
// history rebuild: RenderedContent (the RAG-augmented version) wins over the
// raw Content, and image captions are appended only when there's no rendered
// override (the rendered version already carries any retrieval context that
// would have included the caption upstream).
func TestBuildUserHistoryMessage_PrefersRenderedContent(t *testing.T) {
// TestBuildUserHistoryMessage_IgnoresLegacyRenderedContent verifies that old
// prompt/context snapshots never re-enter the current model context.
func TestBuildUserHistoryMessage_IgnoresLegacyRenderedContent(t *testing.T) {
msg := &types.Message{
Role: "user",
Content: "what about the chart?",
@@ -27,7 +24,8 @@ func TestBuildUserHistoryMessage_PrefersRenderedContent(t *testing.T) {
}
got := buildUserHistoryMessage(msg)
assert.Equal(t, "user", got.Role)
assert.Equal(t, "what about the chart? [augmented]", got.Content)
assert.Equal(t, "what about the chart?\n\n[用户上传图片内容]\na bar chart", got.Content)
assert.NotContains(t, got.Content, "[augmented]")
}
func TestBuildUserHistoryMessage_FallsBackToContentWithCaptions(t *testing.T) {
@@ -69,10 +67,10 @@ func TestBuildUserHistoryMessage_AppendsAttachmentsWhenNoRenderedContent(t *test
assert.Contains(t, got.Content, "hello world")
}
// TestBuildUserHistoryMessage_RenderedContentSkipsAttachmentReplay ensures the
// KnowledgeQA path (where RenderedContent already includes the attachment
// prompt persisted by the pipeline) does not double-inject attachments.
func TestBuildUserHistoryMessage_RenderedContentSkipsAttachmentReplay(t *testing.T) {
// TestBuildUserHistoryMessage_RenderedContentRebuildsAttachment verifies that
// canonical attachment data is retained while the legacy rendered snapshot is
// discarded.
func TestBuildUserHistoryMessage_RenderedContentRebuildsAttachment(t *testing.T) {
msg := &types.Message{
Role: "user",
Content: "summarize this",
@@ -82,8 +80,9 @@ func TestBuildUserHistoryMessage_RenderedContentSkipsAttachmentReplay(t *testing
},
}
got := buildUserHistoryMessage(msg)
assert.Equal(t, "summarize this [with retrieval context already included]", got.Content)
assert.NotContains(t, got.Content, "<attachment")
assert.Contains(t, got.Content, "summarize this")
assert.NotContains(t, got.Content, "[with retrieval context already included]")
assert.Contains(t, got.Content, "<attachment")
}
// TestBuildAssistantHistoryMessages_NaturalFinishEmitsSingleAnswer covers the
@@ -51,8 +51,9 @@ func (p *PluginChatCompletion) OnEvent(
pipelineInfo(ctx, "Completion", "messages_ready", map[string]interface{}{
"message_count": len(chatManage.History) + 2,
})
chatMessages := prepareMessagesWithHistory(chatManage)
chatMessages, sourceRefs := prepareMessagesWithReferences(ctx, chatManage)
resourceRefs := llmresource.NewRegistry()
chatMessages = sourceRefs.EncodeMessages(chatMessages)
chatMessages = resourceRefs.EncodeMessages(chatMessages)
// Call the chat model to generate response
@@ -68,6 +69,7 @@ func (p *PluginChatCompletion) OnEvent(
return ErrModelCall.WithError(err)
}
resourceRefs.DecodeResponse(chatResponse)
sourceRefs.ExpandResponse(chatResponse)
if orphans := resourceRefs.OrphanAliases(chatResponse.Content); len(orphans) > 0 {
pipelineWarn(ctx, "Completion", "orphan_resource_aliases", map[string]interface{}{
"session_id": chatManage.SessionID,
@@ -6,6 +6,7 @@ import (
"fmt"
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/llmresource"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
@@ -55,8 +56,9 @@ func (p *PluginChatCompletionStream) OnEvent(ctx context.Context,
// Prepare base messages without history
chatMessages := prepareMessagesWithHistory(chatManage)
chatMessages, sourceRefs := prepareMessagesWithReferences(ctx, chatManage)
resourceRefs := llmresource.NewRegistry()
chatMessages = sourceRefs.EncodeMessages(chatMessages)
chatMessages = resourceRefs.EncodeMessages(chatMessages)
pipelineInfo(ctx, "Stream", "messages_ready", map[string]interface{}{
"message_count": len(chatMessages),
@@ -110,6 +112,8 @@ func (p *PluginChatCompletionStream) OnEvent(ctx context.Context,
go func() {
answerDecoder := llmresource.NewStreamDecoder(resourceRefs)
thinkingDecoder := llmresource.NewStreamDecoder(resourceRefs)
answerRefExpander := llmreference.NewStreamExpander(sourceRefs)
thinkingRefExpander := llmreference.NewStreamExpander(sourceRefs)
thinkingID := fmt.Sprintf("%s-thinking", uuid.New().String()[:8])
answerID := fmt.Sprintf("%s-answer", uuid.New().String()[:8])
thinkingOpen := false
@@ -135,20 +139,22 @@ func (p *PluginChatCompletionStream) OnEvent(ctx context.Context,
// reference in flight at teardown is silently dropped (and never
// persisted, since the assistant message is saved from these events).
flushDecoders := func() {
if tail := thinkingDecoder.Flush(); tail != "" {
thinkingTail := thinkingRefExpander.Feed(thinkingDecoder.Flush()) + thinkingRefExpander.Flush()
if thinkingTail != "" {
_ = eventBus.Emit(ctx, types.Event{
ID: thinkingID,
Type: types.EventType(event.EventAgentThought),
SessionID: chatManage.SessionID,
Data: event.AgentThoughtData{Content: tail},
Data: event.AgentThoughtData{Content: thinkingTail},
})
}
if tail := answerDecoder.Flush(); tail != "" {
answerTail := answerRefExpander.Feed(answerDecoder.Flush()) + answerRefExpander.Flush()
if answerTail != "" {
_ = eventBus.Emit(ctx, types.Event{
ID: answerID,
Type: types.EventType(event.EventAgentFinalAnswer),
SessionID: chatManage.SessionID,
Data: event.AgentFinalAnswerData{Content: tail},
Data: event.AgentFinalAnswerData{Content: answerTail},
})
}
}
@@ -192,7 +198,7 @@ func (p *PluginChatCompletionStream) OnEvent(ctx context.Context,
}
if response.ResponseType == types.ResponseTypeThinking {
response.Content = thinkingDecoder.Feed(response.Content)
response.Content = thinkingRefExpander.Feed(thinkingDecoder.Feed(response.Content))
if response.Content != "" {
thinkingOpen = true
eventBus.Emit(ctx, types.Event{
@@ -212,7 +218,7 @@ func (p *PluginChatCompletionStream) OnEvent(ctx context.Context,
}
if response.ResponseType == types.ResponseTypeAnswer {
response.Content = answerDecoder.Feed(response.Content)
response.Content = answerRefExpander.Feed(answerDecoder.Feed(response.Content))
closeThinking()
eventBus.Emit(ctx, types.Event{
ID: answerID,
@@ -145,25 +145,18 @@ func loadAndProcessHistory(
h = &types.History{}
}
if message.Role == "user" {
if message.RenderedContent != "" {
h.Query = message.RenderedContent
} else {
h.Query = message.Content
}
// RenderedContent is a snapshot of the prompt/context format used by
// the original turn. Replaying it would mix legacy <context id="…">
// envelopes and old citation instructions into the current protocol.
// Historical references are carried separately in KnowledgeReferences
// and can be re-merged into this turn's freshly rendered context.
h.Query = message.Content
h.CreateAt = message.CreatedAt
// When RenderedContent is absent (e.g. the pure-chat, non-RAG path
// and the search-nothing fallback path never persist it), replay the
// previous turn's image captions and attachment content from the
// stored message so the model retains a textual reference to them.
// When RenderedContent is present it already carries this
// augmentation, so we must not double-append.
if message.RenderedContent == "" {
if desc := extractImageCaptions(message.Images); desc != "" {
h.Query += "\n\n[用户上传图片内容]\n" + desc
}
if len(message.Attachments) > 0 {
h.Query += message.Attachments.BuildPrompt()
}
if desc := extractImageCaptions(message.Images); desc != "" {
h.Query += "\n\n[用户上传图片内容]\n" + desc
}
if len(message.Attachments) > 0 {
h.Query += message.Attachments.BuildPrompt()
}
} else {
h.Answer = regThinkTags.ReplaceAllString(message.Content, "")
@@ -0,0 +1,135 @@
package chatpipeline
import (
"context"
"strings"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
)
// prepareMessagesWithReferences replaces the pipeline's positional context IDs
// with request-local chunk references in the copy sent to the model. Persisted
// rendered_content and the public citation contract remain unchanged.
func prepareMessagesWithReferences(
ctx context.Context,
chatManage *types.ChatManage,
) ([]chat.Message, *llmreference.Registry) {
refs := llmreference.NewRegistry()
if chatManage == nil {
return nil, refs
}
messages := prepareMessagesWithHistory(chatManage)
if len(messages) > 0 {
messages[0].Content = strings.TrimRight(messages[0].Content, " \t\r\n") + llmreference.ProtocolPrompt
}
if len(chatManage.MergeResult) == 0 || len(messages) == 0 {
return messages, refs
}
ordered := orderedPipelineReferences(chatManage)
knowledgeResults := make([]*types.SearchResult, 0, len(ordered))
knowledgeRows := make([]map[string]interface{}, 0, len(ordered))
webRows := make([]map[string]interface{}, 0)
for _, result := range ordered {
if result == nil {
continue
}
if isPipelineWebReference(result) {
webRows = append(webRows, map[string]interface{}{
"url": result.ID,
"title": firstPipelineTitle(result),
"snippet": result.Content,
"published_at": result.Metadata["published_at"],
})
continue
}
knowledgeResults = append(knowledgeResults, result)
knowledgeRows = append(knowledgeRows, map[string]interface{}{
"chunk_id": result.ID,
"knowledge_id": result.KnowledgeID,
"knowledge_base_id": result.KnowledgeBaseID,
"knowledge_title": firstPipelineTitle(result),
"chunk_index": result.ChunkIndex,
"chunk_type": result.ChunkType,
"content": getEnrichedPassageForChat(ctx, result),
})
}
refs.RegisterSearchResults(knowledgeResults)
var contextParts []string
if len(knowledgeRows) > 0 {
contextParts = append(contextParts, refs.ModelOutput(&types.ToolResult{
Success: true,
Data: map[string]interface{}{
"display_type": "search_results",
"results": knowledgeRows,
},
}))
}
if len(webRows) > 0 {
contextParts = append(contextParts, refs.ModelOutput(&types.ToolResult{
Success: true,
Data: map[string]interface{}{
"display_type": "web_search_results",
"results": webRows,
},
}))
}
modelContexts := strings.Join(contextParts, "\n")
if strings.TrimSpace(modelContexts) == "" {
return messages, refs
}
last := len(messages) - 1
replaced := false
for _, index := range []int{0, last} {
if chatManage.RenderedContexts != "" && strings.Contains(messages[index].Content, chatManage.RenderedContexts) {
messages[index].Content = strings.ReplaceAll(messages[index].Content, chatManage.RenderedContexts, modelContexts)
replaced = true
}
}
if !replaced {
messages[last].Content = modelContexts + "\n\n" + messages[last].Content
}
return messages, refs
}
func isPipelineWebReference(result *types.SearchResult) bool {
if result == nil {
return false
}
return strings.EqualFold(result.ChunkType, string(types.ChunkTypeWebSearch)) ||
strings.EqualFold(result.KnowledgeSource, "web_search")
}
func orderedPipelineReferences(chatManage *types.ChatManage) []*types.SearchResult {
if chatManage == nil {
return nil
}
if !chatManage.FAQPriorityEnabled {
return chatManage.MergeResult
}
ordered := make([]*types.SearchResult, 0, len(chatManage.MergeResult))
for _, result := range chatManage.MergeResult {
if result != nil && result.ChunkType == string(types.ChunkTypeFAQ) {
ordered = append(ordered, result)
}
}
for _, result := range chatManage.MergeResult {
if result != nil && result.ChunkType != string(types.ChunkTypeFAQ) {
ordered = append(ordered, result)
}
}
return ordered
}
func firstPipelineTitle(result *types.SearchResult) string {
if result == nil {
return ""
}
if result.KnowledgeTitle != "" {
return result.KnowledgeTitle
}
return result.KnowledgeFilename
}
@@ -0,0 +1,116 @@
package chatpipeline
import (
"context"
"strings"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestPrepareMessagesWithReferencesUsesChunkCentricContext(t *testing.T) {
rendered := `<context id="1">first content</context><context id="2">second content</context>`
manage := &types.ChatManage{
PipelineRequest: types.PipelineRequest{
Query: "question",
SummaryConfig: types.SummaryConfig{
Prompt: "system",
},
},
PipelineState: types.PipelineState{
RenderedContexts: rendered,
UserContent: "References:\n" + rendered + "\nQuestion: question",
MergeResult: []*types.SearchResult{
{ID: "chunk-1", KnowledgeID: "doc-1", KnowledgeBaseID: "kb-1", KnowledgeTitle: "Doc", ChunkIndex: 1, Content: "first content"},
{ID: "chunk-2", KnowledgeID: "doc-1", KnowledgeBaseID: "kb-1", KnowledgeTitle: "Doc", ChunkIndex: 2, Content: "second content"},
},
},
}
messages, refs := prepareMessagesWithReferences(context.Background(), manage)
require.Len(t, messages, 2)
require.Contains(t, messages[0].Content, "Source reference protocol")
require.Contains(t, messages[1].Content, `<document id="d1" kb="b1" title="Doc">`)
require.Contains(t, messages[1].Content, `<chunk id="c1" index="1" view="full">`)
require.Contains(t, messages[1].Content, `<chunk id="c2" index="2" view="full">`)
require.False(t, strings.Contains(messages[1].Content, "chunk-1"))
require.Equal(t,
`<kb doc="Doc" chunk_id="chunk-1" kb_id="kb-1" />`,
refs.ExpandText(`<ref id="c1"/>`),
)
}
func TestPrepareMessagesWithReferencesReplacesSystemPromptContextAndHistoryCitations(t *testing.T) {
rendered := `<context id="1">first content</context>`
manage := &types.ChatManage{
PipelineRequest: types.PipelineRequest{
Query: "question",
SummaryConfig: types.SummaryConfig{
Prompt: `System references: {{contexts}}`,
},
},
PipelineState: types.PipelineState{
RenderedContexts: rendered,
UserContent: "Question: question",
History: []*types.History{{
Query: "previous",
Answer: `Previous <kb doc="Old" chunk_id="old-chunk" kb_id="old-kb" />`,
}},
MergeResult: []*types.SearchResult{{
ID: "current-chunk", KnowledgeID: "current-doc", KnowledgeBaseID: "current-kb", KnowledgeTitle: "Current", Content: "first content",
}},
},
}
messages, refs := prepareMessagesWithReferences(context.Background(), manage)
messages = refs.EncodeMessages(messages)
require.NotContains(t, messages[0].Content, rendered)
require.Contains(t, messages[0].Content, `<chunk id="c1"`)
require.Contains(t, messages[2].Content, `<ref id="c2"/>`)
require.NotContains(t, messages[2].Content, "old-chunk")
}
func TestPrepareMessagesWithReferencesKeepsWebSeparateFromChunks(t *testing.T) {
manage := &types.ChatManage{
PipelineRequest: types.PipelineRequest{Query: "question", SummaryConfig: types.SummaryConfig{Prompt: "system"}},
PipelineState: types.PipelineState{
UserContent: "question",
MergeResult: []*types.SearchResult{{
ID: "https://example.com/page",
KnowledgeTitle: "Example",
Content: "web content",
ChunkType: string(types.ChunkTypeWebSearch),
KnowledgeSource: "web_search",
}},
},
}
messages, refs := prepareMessagesWithReferences(context.Background(), manage)
require.Contains(t, messages[1].Content, `<retrieval type="web" mode="search">`)
require.Contains(t, messages[1].Content, `<page id="w1" title="Example">`)
require.NotContains(t, messages[1].Content, `<chunk id="c1"`)
require.Equal(t,
`<web url="https://example.com/page" title="Example" />`,
refs.ExpandText(`<ref id="w1"/>`),
)
}
func TestPrepareMessagesWithReferencesCompactsHistoryWithoutCurrentRetrieval(t *testing.T) {
manage := &types.ChatManage{
PipelineRequest: types.PipelineRequest{Query: "follow-up", SummaryConfig: types.SummaryConfig{Prompt: "system"}},
PipelineState: types.PipelineState{
UserContent: "follow-up",
History: []*types.History{{
Query: "previous",
Answer: `Previous <web url="https://example.com/old" title="Old" />`,
}},
},
}
messages, refs := prepareMessagesWithReferences(context.Background(), manage)
messages = refs.EncodeMessages(messages)
require.Contains(t, messages[0].Content, "Source reference protocol")
require.Contains(t, messages[2].Content, `<ref id="w1"/>`)
require.NotContains(t, messages[2].Content, "https://example.com/old")
}
@@ -79,3 +79,20 @@ func TestBuildFallbackMessages_AttachesImagesToUserTurn(t *testing.T) {
withVision := buildFallbackMessages(cm, "fallback")
assert.Equal(t, cm.Images, withVision[len(withVision)-1].Images)
}
func TestPrepareFallbackMessagesMigratesHistoricalCitations(t *testing.T) {
cm := &types.ChatManage{}
cm.Query = "follow-up"
cm.History = []*types.History{{
Query: "previous",
Answer: `Previous <kb doc="Legacy" chunk_id="legacy-chunk" kb_id="legacy-kb" />`,
}}
messages, refs := prepareFallbackMessages(cm, "legacy fallback prompt")
require.Contains(t, messages[0].Content, "Source reference protocol")
require.Equal(t, `Previous <ref id="c1"/>`, messages[2].Content)
require.Equal(t,
`<kb doc="Legacy" chunk_id="legacy-chunk" kb_id="legacy-kb" />`,
refs.ExpandText(`<ref id="c1"/>`),
)
}
@@ -10,6 +10,7 @@ import (
chatpipeline "github.com/Tencent/WeKnora/internal/application/service/chat_pipeline"
"github.com/Tencent/WeKnora/internal/common"
"github.com/Tencent/WeKnora/internal/event"
"github.com/Tencent/WeKnora/internal/llmreference"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
@@ -971,7 +972,8 @@ func (s *sessionService) handleModelFallback(ctx context.Context, chatManage *ty
}
// Start streaming response
responseChan, err := chatModel.ChatStream(ctx, buildFallbackMessages(chatManage, promptContent), opt)
fallbackMessages, sourceRefs := prepareFallbackMessages(chatManage, promptContent)
responseChan, err := chatModel.ChatStream(ctx, fallbackMessages, opt)
if err != nil {
logger.Errorf(ctx, "Failed to start streaming fallback response: %v, falling back to fixed response", err)
s.handleFixedFallback(ctx, chatManage)
@@ -985,7 +987,21 @@ func (s *sessionService) handleModelFallback(ctx context.Context, chatManage *ty
}
// Start goroutine to consume stream and emit events
go s.consumeFallbackStream(ctx, chatManage, responseChan)
go s.consumeFallbackStream(ctx, chatManage, responseChan, sourceRefs)
}
func prepareFallbackMessages(
chatManage *types.ChatManage,
promptContent string,
) ([]chat.Message, *llmreference.Registry) {
messages := buildFallbackMessages(chatManage, promptContent)
refs := llmreference.NewRegistry()
if len(messages) > 0 && messages[0].Role == "system" {
messages[0].Content = strings.TrimRight(messages[0].Content, " \t\r\n") + llmreference.ProtocolPrompt
} else {
messages = append([]chat.Message{{Role: "system", Content: strings.TrimSpace(llmreference.ProtocolPrompt)}}, messages...)
}
return refs.EncodeMessages(messages), refs
}
func buildFallbackMessages(chatManage *types.ChatManage, promptContent string) []chat.Message {
@@ -1118,15 +1134,21 @@ func (s *sessionService) consumeFallbackStream(
ctx context.Context,
chatManage *types.ChatManage,
responseChan <-chan types.StreamResponse,
sourceRefs *llmreference.Registry,
) {
fallbackID := generateEventID("fallback")
eventBus := chatManage.EventBus
var finalContent string
streamCompleted := false
refExpander := llmreference.NewStreamExpander(sourceRefs)
for response := range responseChan {
// Emit event for each answer chunk
if response.ResponseType == types.ResponseTypeAnswer {
response.Content = refExpander.Feed(response.Content)
if response.Done {
response.Content += refExpander.Flush()
}
finalContent += response.Content
if err := eventBus.Emit(ctx, types.Event{
ID: fallbackID,
+429
View File
@@ -0,0 +1,429 @@
package llmreference
import (
"encoding/json"
"fmt"
"sort"
"strings"
"github.com/Tencent/WeKnora/internal/types"
)
// ModelOutput returns a compact, source-centric representation for the LLM.
// The canonical ToolResult.Output remains untouched for UI, logs and storage.
func (r *Registry) ModelOutput(result *types.ToolResult) string {
if result == nil {
return ""
}
if !result.Success {
if result.Error != "" {
return "Error: " + r.CompactKnownText(result.Error)
}
return "Error: tool call failed"
}
displayType := stringValue(result.Data, "display_type")
switch displayType {
case "grep_results":
return r.modelKnowledgeOutput("keyword", mapsValue(result.Data["chunk_results"]), result.Output)
case "search_results":
return r.modelKnowledgeOutput("semantic", mapsValue(result.Data["results"]), result.Output)
case "knowledge_chunks_list":
return r.modelKnowledgeChunksOutput(result.Data, result.Output)
case "document_info":
return r.modelDocumentInfoOutput(mapsValue(result.Data["documents"]), result.Output)
case "graph_query_results":
return r.modelKnowledgeOutput("graph", mapsValue(result.Data["results"]), result.Output)
case "web_search_results":
return r.modelWebSearchOutput(mapsValue(result.Data["results"]), result.Output)
case "web_fetch_results":
return r.modelWebFetchOutput(mapsValue(result.Data["results"]), result.Output)
default:
r.registerLabeledReferences(result.Output)
return r.CompactKnownText(result.Output)
}
}
func (r *Registry) modelDocumentInfoOutput(rows []map[string]interface{}, fallback string) string {
if len(rows) == 0 {
return r.CompactKnownText(fallback)
}
var b strings.Builder
b.WriteString("<documents>\n")
count := 0
for _, row := range rows {
knowledgeID := stringValue(row, "knowledge_id")
docAlias := r.RegisterDocument(knowledgeID)
if boolValue(row, "is_faq") {
chunkID := stringValue(row, "faq_id")
if chunkID == "" {
continue
}
title := firstNonEmpty(stringValue(row, "faq_question"), stringValue(row, "title"))
chunkAlias := r.RegisterChunk(ChunkReference{
ChunkID: chunkID,
KnowledgeID: knowledgeID,
DocumentTitle: title,
ChunkType: "faq",
})
fmt.Fprintf(&b, " <document id=\"%s\" type=\"faq\">\n", escapeAttr(docAlias))
fmt.Fprintf(&b, " <chunk id=\"%s\" type=\"faq\">\n", escapeAttr(chunkAlias))
if title != "" {
fmt.Fprintf(&b, " <question>%s</question>\n", escapeText(title))
}
for _, answer := range stringSliceValue(row["faq_answers"]) {
fmt.Fprintf(&b, " <answer>%s</answer>\n", escapeText(answer))
}
b.WriteString(" </chunk>\n </document>\n")
count++
continue
}
if docAlias == "" {
continue
}
fmt.Fprintf(&b, " <document id=\"%s\"", escapeAttr(docAlias))
if title := stringValue(row, "title"); title != "" {
fmt.Fprintf(&b, " title=\"%s\"", escapeAttr(title))
}
if docType := stringValue(row, "type"); docType != "" {
fmt.Fprintf(&b, " type=\"%s\"", escapeAttr(docType))
}
if fileType := stringValue(row, "file_type"); fileType != "" {
fmt.Fprintf(&b, " file_type=\"%s\"", escapeAttr(fileType))
}
fmt.Fprintf(&b, " chunk_count=\"%d\">\n", intValue(row, "chunk_count"))
if description := stringValue(row, "description"); description != "" {
fmt.Fprintf(&b, " <description>%s</description>\n", escapeText(description))
}
b.WriteString(" </document>\n")
count++
}
b.WriteString("</documents>")
if count == 0 {
return r.CompactKnownText(fallback)
}
return b.String()
}
type modelChunk struct {
alias string
docAlias string
kbAlias string
title string
chunkType string
index int
view string
match string
content string
question string
answers []string
images []map[string]interface{}
docRealID string
kbRealID string
chunkReal string
inputOrder int
}
func (r *Registry) modelKnowledgeOutput(mode string, rows []map[string]interface{}, fallback string) string {
chunks := make([]modelChunk, 0, len(rows))
for idx, row := range rows {
chunkID := firstNonEmpty(stringValue(row, "chunk_id"), stringValue(row, "faq_id"), stringValue(row, "id"))
knowledgeID := stringValue(row, "knowledge_id")
kbID := firstNonEmpty(stringValue(row, "knowledge_base_id"), stringValue(row, "knowledge_base"))
title := firstNonEmpty(stringValue(row, "knowledge_title"), stringValue(row, "title"))
if chunkID == "" {
continue
}
chunkType := stringValue(row, "chunk_type")
if stringValue(row, "faq_id") != "" && chunkType == "" {
chunkType = "faq"
}
chunkIndex := intValue(row, "chunk_index")
if chunkIndex == 0 {
chunkIndex = intValue(row, "index")
}
chunkAlias := r.RegisterChunk(ChunkReference{
ChunkID: chunkID,
KnowledgeID: knowledgeID,
KnowledgeBaseID: kbID,
DocumentTitle: title,
ChunkIndex: chunkIndex,
ChunkType: chunkType,
})
chunks = append(chunks, modelChunk{
alias: chunkAlias,
docAlias: r.RegisterDocument(knowledgeID),
kbAlias: r.RegisterKnowledgeBase(kbID),
title: title,
chunkType: chunkType,
index: chunkIndex,
view: viewForRow(row, mode),
match: firstNonEmpty(stringValue(row, "match_snippet"), stringValue(row, "matched_content")),
content: stringValue(row, "content"),
question: firstNonEmpty(stringValue(row, "faq_question"), stringValue(row, "faq_standard_question")),
answers: stringSliceValue(row["faq_answers"]),
images: mapsValue(row["images"]),
docRealID: knowledgeID,
kbRealID: kbID,
chunkReal: chunkID,
inputOrder: idx,
})
}
if len(chunks) == 0 {
return r.CompactKnownText(fallback)
}
return renderKnowledgeChunks(mode, chunks)
}
func viewForRow(row map[string]interface{}, mode string) string {
if stringValue(row, "content") != "" {
return "full"
}
if mode == "deep_read" {
return "full"
}
return "match"
}
func (r *Registry) modelKnowledgeChunksOutput(data map[string]interface{}, fallback string) string {
rows := mapsValue(data["chunks"])
title := stringValue(data, "knowledge_title")
knowledgeID := stringValue(data, "knowledge_id")
for _, row := range rows {
if stringValue(row, "knowledge_id") == "" {
row["knowledge_id"] = knowledgeID
}
if stringValue(row, "knowledge_title") == "" {
row["knowledge_title"] = title
}
}
output := r.modelKnowledgeOutput("deep_read", rows, fallback)
if len(rows) == 0 {
return output
}
remaining := intValue(data, "total_chunks") - intValue(data, "fetched_chunks")
if remaining > 0 {
output = strings.TrimSuffix(output, "</retrieval>")
output += fmt.Sprintf(" <pagination remaining=\"%d\" page=\"%d\" page_size=\"%d\" />\n</retrieval>",
remaining, intValue(data, "page"), intValue(data, "page_size"))
}
return output
}
func renderKnowledgeChunks(mode string, chunks []modelChunk) string {
type docGroup struct {
alias string
kbAlias string
title string
chunks []modelChunk
order int
}
groupsByKey := make(map[string]*docGroup)
var groups []*docGroup
for _, chunk := range chunks {
key := chunk.docAlias
if key == "" {
key = "chunk:" + chunk.alias
}
group := groupsByKey[key]
if group == nil {
group = &docGroup{alias: chunk.docAlias, kbAlias: chunk.kbAlias, title: chunk.title, order: chunk.inputOrder}
groupsByKey[key] = group
groups = append(groups, group)
}
group.chunks = append(group.chunks, chunk)
}
sort.SliceStable(groups, func(i, j int) bool { return groups[i].order < groups[j].order })
var b strings.Builder
fmt.Fprintf(&b, "<retrieval type=\"knowledge\" mode=\"%s\">\n", escapeAttr(mode))
for _, group := range groups {
b.WriteString(" <document")
if group.alias != "" {
fmt.Fprintf(&b, " id=\"%s\"", escapeAttr(group.alias))
}
if group.kbAlias != "" {
fmt.Fprintf(&b, " kb=\"%s\"", escapeAttr(group.kbAlias))
}
if group.title != "" {
fmt.Fprintf(&b, " title=\"%s\"", escapeAttr(group.title))
}
b.WriteString(">\n")
for _, chunk := range group.chunks {
fmt.Fprintf(&b, " <chunk id=\"%s\" index=\"%d\" view=\"%s\"", chunk.alias, chunk.index, chunk.view)
if chunk.chunkType != "" {
fmt.Fprintf(&b, " type=\"%s\"", escapeAttr(chunk.chunkType))
}
b.WriteString(">\n")
if chunk.question != "" {
fmt.Fprintf(&b, " <question>%s</question>\n", escapeText(chunk.question))
}
if chunk.match != "" {
fmt.Fprintf(&b, " <match>%s</match>\n", escapeText(chunk.match))
}
if chunk.content != "" {
fmt.Fprintf(&b, " <content>%s</content>\n", escapeText(chunk.content))
}
for _, answer := range chunk.answers {
fmt.Fprintf(&b, " <answer>%s</answer>\n", escapeText(answer))
}
for _, image := range chunk.images {
imageURL := stringValue(image, "url")
if imageURL == "" {
continue
}
caption := stringValue(image, "caption")
fmt.Fprintf(&b, " ![%s](%s)\n", caption, imageURL)
}
b.WriteString(" </chunk>\n")
}
b.WriteString(" </document>\n")
}
b.WriteString("</retrieval>")
return b.String()
}
func (r *Registry) modelWebSearchOutput(rows []map[string]interface{}, fallback string) string {
if len(rows) == 0 {
return r.CompactKnownText(fallback)
}
var b strings.Builder
b.WriteString("<retrieval type=\"web\" mode=\"search\">\n")
count := 0
for _, row := range rows {
rawURL := stringValue(row, "url")
if rawURL == "" {
continue
}
alias := r.RegisterWeb(rawURL, stringValue(row, "title"))
fmt.Fprintf(&b, " <page id=\"%s\" title=\"%s\">\n", alias, escapeAttr(stringValue(row, "title")))
if snippet := stringValue(row, "snippet"); snippet != "" {
fmt.Fprintf(&b, " <match>%s</match>\n", escapeText(snippet))
}
if published := stringValue(row, "published_at"); published != "" {
fmt.Fprintf(&b, " <published>%s</published>\n", escapeText(published))
}
b.WriteString(" </page>\n")
count++
}
b.WriteString("</retrieval>")
if count == 0 {
return r.CompactKnownText(fallback)
}
return b.String()
}
func (r *Registry) modelWebFetchOutput(rows []map[string]interface{}, fallback string) string {
if len(rows) == 0 {
return r.CompactKnownText(fallback)
}
var b strings.Builder
b.WriteString("<retrieval type=\"web\" mode=\"fetch\">\n")
count := 0
for _, row := range rows {
rawURL := stringValue(row, "url")
if rawURL == "" {
continue
}
title := stringValue(row, "title")
alias := r.RegisterWeb(rawURL, title)
fmt.Fprintf(&b, " <page id=\"%s\"", alias)
if title != "" {
fmt.Fprintf(&b, " title=\"%s\"", escapeAttr(title))
}
b.WriteString(" view=\"full\">\n")
if summary := stringValue(row, "summary"); summary != "" {
fmt.Fprintf(&b, " <summary>%s</summary>\n", escapeText(summary))
}
if content := stringValue(row, "raw_content"); content != "" {
fmt.Fprintf(&b, " <content>%s</content>\n", escapeText(content))
}
b.WriteString(" </page>\n")
count++
}
b.WriteString("</retrieval>")
if count == 0 {
return r.CompactKnownText(fallback)
}
return b.String()
}
func mapsValue(value interface{}) []map[string]interface{} {
if value == nil {
return nil
}
encoded, err := json.Marshal(value)
if err != nil {
return nil
}
var rows []map[string]interface{}
if err := json.Unmarshal(encoded, &rows); err != nil {
return nil
}
return rows
}
func stringValue(values map[string]interface{}, key string) string {
if values == nil {
return ""
}
switch value := values[key].(type) {
case string:
return value
case fmt.Stringer:
return value.String()
case json.Number:
return value.String()
default:
return ""
}
}
func intValue(values map[string]interface{}, key string) int {
if values == nil {
return 0
}
switch value := values[key].(type) {
case int:
return value
case int32:
return int(value)
case int64:
return int(value)
case float64:
return int(value)
case json.Number:
result, _ := value.Int64()
return int(result)
default:
return 0
}
}
func boolValue(values map[string]interface{}, key string) bool {
if values == nil {
return false
}
value, _ := values[key].(bool)
return value
}
func stringSliceValue(value interface{}) []string {
if value == nil {
return nil
}
encoded, err := json.Marshal(value)
if err != nil {
return nil
}
var values []string
if err := json.Unmarshal(encoded, &values); err != nil {
return nil
}
return values
}
func escapeText(value string) string {
replacer := strings.NewReplacer("&", "&amp;", "<", "&lt;", ">", "&gt;")
return replacer.Replace(value)
}
+680
View File
@@ -0,0 +1,680 @@
// Package llmreference keeps internal source identifiers out of LLM-facing
// context. It assigns request-local aliases to chunks, documents, knowledge
// bases and web pages, decodes aliases used in tool arguments, and expands the
// model's compact <ref/> citations before application code consumes them.
package llmreference
import (
"encoding/json"
"fmt"
"html"
"net/url"
"regexp"
"sort"
"strings"
"sync"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
)
const ProtocolPrompt = `
## Source reference protocol (system-owned)
Retrieved knowledge chunks and web pages carry short source IDs.
- Cite a knowledge chunk with exactly: <ref id="cN"/>
- Cite a web page with exactly: <ref id="wN"/>
- Copy only IDs that appeared in the supplied context or tool results.
- dN identifies a document and bN identifies a knowledge base for tool calls; they are not citable.
- Never put raw chunk IDs, knowledge IDs, knowledge-base IDs, or source URLs in citations; never output <kb> or <web> tags yourself. This does not change separate instructions to preserve retrieved Markdown image URLs.
- These rules supersede any earlier, saved, or custom prompt instructions that ask for raw IDs, raw source URLs, or model-authored <kb>/<web> citations.
- Keep each <ref/> inline on the same line as the claim it supports.`
type ChunkReference struct {
Alias string
ChunkID string
KnowledgeID string
KnowledgeBaseID string
DocumentTitle string
ChunkIndex int
ChunkType string
}
type WebReference struct {
Alias string
URL string
Title string
}
// Registry is scoped to one assistant response (including every Agent tool
// round). Aliases are never persisted or accepted across requests.
type Registry struct {
mu sync.RWMutex
chunkByID map[string]*ChunkReference
chunkByAlias map[string]*ChunkReference
docToAlias map[string]string
aliasToDoc map[string]string
kbToAlias map[string]string
aliasToKB map[string]string
webByURL map[string]*WebReference
webByAlias map[string]*WebReference
}
func NewRegistry() *Registry {
return &Registry{
chunkByID: make(map[string]*ChunkReference),
chunkByAlias: make(map[string]*ChunkReference),
docToAlias: make(map[string]string),
aliasToDoc: make(map[string]string),
kbToAlias: make(map[string]string),
aliasToKB: make(map[string]string),
webByURL: make(map[string]*WebReference),
webByAlias: make(map[string]*WebReference),
}
}
func (r *Registry) Count() int {
if r == nil {
return 0
}
r.mu.RLock()
defer r.mu.RUnlock()
return len(r.chunkByAlias) + len(r.webByAlias)
}
func (r *Registry) RegisterChunk(ref ChunkReference) string {
if r == nil || strings.TrimSpace(ref.ChunkID) == "" {
return ""
}
r.mu.Lock()
defer r.mu.Unlock()
if existing := r.chunkByID[ref.ChunkID]; existing != nil {
mergeChunkReference(existing, ref)
return existing.Alias
}
ref.Alias = fmt.Sprintf("c%d", len(r.chunkByAlias)+1)
copyRef := ref
r.chunkByID[ref.ChunkID] = &copyRef
r.chunkByAlias[ref.Alias] = &copyRef
return ref.Alias
}
func mergeChunkReference(dst *ChunkReference, src ChunkReference) {
if dst.KnowledgeID == "" {
dst.KnowledgeID = src.KnowledgeID
}
if dst.KnowledgeBaseID == "" {
dst.KnowledgeBaseID = src.KnowledgeBaseID
}
if dst.DocumentTitle == "" {
dst.DocumentTitle = src.DocumentTitle
}
if dst.ChunkIndex == 0 {
dst.ChunkIndex = src.ChunkIndex
}
if dst.ChunkType == "" {
dst.ChunkType = src.ChunkType
}
}
func (r *Registry) RegisterDocument(id string) string {
if r == nil || strings.TrimSpace(id) == "" {
return ""
}
r.mu.Lock()
defer r.mu.Unlock()
if alias := r.docToAlias[id]; alias != "" {
return alias
}
alias := fmt.Sprintf("d%d", len(r.aliasToDoc)+1)
r.docToAlias[id] = alias
r.aliasToDoc[alias] = id
return alias
}
func (r *Registry) RegisterKnowledgeBase(id string) string {
if r == nil || strings.TrimSpace(id) == "" {
return ""
}
r.mu.Lock()
defer r.mu.Unlock()
if alias := r.kbToAlias[id]; alias != "" {
return alias
}
alias := fmt.Sprintf("b%d", len(r.aliasToKB)+1)
r.kbToAlias[id] = alias
r.aliasToKB[alias] = id
return alias
}
func (r *Registry) RegisterWeb(rawURL, title string) string {
if r == nil || strings.TrimSpace(rawURL) == "" {
return ""
}
key := canonicalWebURL(rawURL)
r.mu.Lock()
defer r.mu.Unlock()
if existing := r.webByURL[key]; existing != nil {
if existing.Title == "" && title != "" {
existing.Title = title
}
return existing.Alias
}
ref := &WebReference{
Alias: fmt.Sprintf("w%d", len(r.webByAlias)+1),
URL: rawURL,
Title: title,
}
r.webByURL[key] = ref
r.webByAlias[ref.Alias] = ref
return ref.Alias
}
func canonicalWebURL(raw string) string {
parsed, err := url.Parse(strings.TrimSpace(raw))
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return strings.TrimSpace(raw)
}
parsed.Fragment = ""
return parsed.String()
}
func (r *Registry) RegisterSearchResults(results []*types.SearchResult) {
for _, result := range results {
if result == nil {
continue
}
r.RegisterDocument(result.KnowledgeID)
r.RegisterKnowledgeBase(result.KnowledgeBaseID)
r.RegisterChunk(ChunkReference{
ChunkID: result.ID,
KnowledgeID: result.KnowledgeID,
KnowledgeBaseID: result.KnowledgeBaseID,
DocumentTitle: firstNonEmpty(result.KnowledgeTitle, result.KnowledgeFilename),
ChunkIndex: result.ChunkIndex,
ChunkType: result.ChunkType,
})
}
}
func firstNonEmpty(values ...string) string {
for _, value := range values {
if strings.TrimSpace(value) != "" {
return value
}
}
return ""
}
func (r *Registry) ChunkAlias(id string) string {
r.mu.RLock()
defer r.mu.RUnlock()
if ref := r.chunkByID[id]; ref != nil {
return ref.Alias
}
return ""
}
func (r *Registry) DocumentAlias(id string) string {
r.mu.RLock()
defer r.mu.RUnlock()
return r.docToAlias[id]
}
func (r *Registry) KnowledgeBaseAlias(id string) string {
r.mu.RLock()
defer r.mu.RUnlock()
return r.kbToAlias[id]
}
// DecodeToolCalls restores exact alias-valued JSON strings before tools parse
// or validate their arguments. It never performs substring replacement.
func (r *Registry) DecodeToolCalls(toolCalls []types.LLMToolCall) {
for i := range toolCalls {
toolCalls[i].Function.Arguments = r.decodeJSON(toolCalls[i].Function.Arguments, false)
}
}
// EncodeMessages compacts known real identifiers in assistant tool-call replay.
func (r *Registry) EncodeMessages(messages []chat.Message) []chat.Message {
if r == nil || len(messages) == 0 {
return messages
}
out := make([]chat.Message, len(messages))
copy(out, messages)
// First register every durable identifier present in historical tool calls
// and canonical assistant citations. This two-pass shape lets an early tool
// message reuse metadata that appears only in the turn's final answer.
for i := range out {
if out[i].Role == "assistant" || out[i].Role == "tool" {
out[i].Content = r.CompactPublicCitations(out[i].Content)
out[i].ReasoningContent = r.CompactPublicCitations(out[i].ReasoningContent)
}
if len(out[i].MultiContent) > 0 {
out[i].MultiContent = append([]chat.MessageContentPart(nil), out[i].MultiContent...)
for j := range out[i].MultiContent {
if out[i].MultiContent[j].Type == "text" && (out[i].Role == "assistant" || out[i].Role == "tool") {
out[i].MultiContent[j].Text = r.CompactPublicCitations(out[i].MultiContent[j].Text)
}
}
}
if len(out[i].ToolCalls) > 0 {
out[i].ToolCalls = append([]chat.ToolCall(nil), out[i].ToolCalls...)
for j := range out[i].ToolCalls {
r.registerToolArguments(out[i].ToolCalls[j].Function.Arguments)
}
}
}
for i := range out {
if out[i].Role == "tool" {
r.registerLegacyToolReferences(out[i].Content)
out[i].Content = r.CompactKnownText(out[i].Content)
}
for j := range out[i].ToolCalls {
out[i].ToolCalls[j].Function.Arguments = r.decodeJSON(out[i].ToolCalls[j].Function.Arguments, true)
}
}
return out
}
var shortSourceAliasRE = regexp.MustCompile(`(?i)^[cdbw][1-9][0-9]*$`)
func (r *Registry) registerToolArguments(raw string) {
if r == nil || strings.TrimSpace(raw) == "" {
return
}
var value interface{}
if err := json.Unmarshal([]byte(raw), &value); err != nil {
return
}
r.registerToolArgumentValue("", value)
}
func (r *Registry) registerToolArgumentValue(key string, value interface{}) {
switch typed := value.(type) {
case string:
value := strings.TrimSpace(typed)
if value == "" || shortSourceAliasRE.MatchString(value) {
return
}
switch strings.ToLower(key) {
case "chunk_id", "faq_id", "chunk_ids", "faq_ids":
r.RegisterChunk(ChunkReference{ChunkID: value})
case "knowledge_id", "knowledge_ids", "suspected_knowledge_ids", "source_refs":
r.RegisterDocument(value)
case "knowledge_base", "knowledge_base_id", "knowledge_base_ids", "kb_id", "kb_ids":
r.RegisterKnowledgeBase(value)
case "url", "urls":
if parsed, err := url.Parse(value); err == nil && (parsed.Scheme == "http" || parsed.Scheme == "https") {
r.RegisterWeb(value, "")
}
}
case []interface{}:
for _, item := range typed {
r.registerToolArgumentValue(key, item)
}
case map[string]interface{}:
for childKey, item := range typed {
r.registerToolArgumentValue(childKey, item)
}
}
}
var (
publicKBTagRE = regexp.MustCompile(`(?is)<kb\b[^>]*>`)
publicWebTagRE = regexp.MustCompile(`(?is)<web\b[^>]*>`)
docAttrRE = regexp.MustCompile(`(?i)\bdoc\s*=\s*"([^"]*)"`)
chunkAttrRE = regexp.MustCompile(`(?i)\bchunk_id\s*=\s*"([^"]+)"`)
publicKBAttrRE = regexp.MustCompile(`(?i)\bkb_id\s*=\s*"([^"]*)"`)
urlAttrRE = regexp.MustCompile(`(?i)\burl\s*=\s*"([^"]+)"`)
titleAttrRE = regexp.MustCompile(`(?i)\btitle\s*=\s*"([^"]*)"`)
legacyChunkRE = regexp.MustCompile(`(?is)<(?:chunk|faq)\b[^>]*>`)
faqAttrRE = regexp.MustCompile(`(?i)\bfaq_id\s*=\s*"([^"]+)"`)
knowledgeTitleAttrRE = regexp.MustCompile(`(?i)\bknowledge_title\s*=\s*"([^"]*)"`)
)
func (r *Registry) registerLegacyToolReferences(text string) {
if r == nil || text == "" {
return
}
r.registerLabeledReferences(text)
for _, tag := range legacyChunkRE.FindAllString(text, -1) {
chunkID := firstNonEmpty(publicAttr(chunkAttrRE, tag), publicAttr(faqAttrRE, tag))
if chunkID == "" {
continue
}
r.RegisterChunk(ChunkReference{
ChunkID: chunkID,
KnowledgeID: publicAttr(documentAttrRE, tag),
KnowledgeBaseID: firstNonEmpty(publicAttr(kbAttrRE, tag), publicAttr(publicKBAttrRE, tag)),
DocumentTitle: firstNonEmpty(publicAttr(knowledgeTitleAttrRE, tag), publicAttr(docAttrRE, tag)),
})
}
}
// CompactPublicCitations folds canonical citations from prior assistant turns
// back into this request's private protocol. This prevents durable chunk IDs
// and web URLs in conversation history from becoming model-visible again.
func (r *Registry) CompactPublicCitations(text string) string {
if r == nil || text == "" {
return text
}
text = publicKBTagRE.ReplaceAllStringFunc(text, func(tag string) string {
chunkID := publicAttr(chunkAttrRE, tag)
if chunkID == "" {
return tag
}
alias := r.RegisterChunk(ChunkReference{
ChunkID: chunkID,
KnowledgeBaseID: publicAttr(publicKBAttrRE, tag),
DocumentTitle: publicAttr(docAttrRE, tag),
})
return `<ref id="` + alias + `"/>`
})
return publicWebTagRE.ReplaceAllStringFunc(text, func(tag string) string {
rawURL := publicAttr(urlAttrRE, tag)
if rawURL == "" {
return tag
}
alias := r.RegisterWeb(rawURL, publicAttr(titleAttrRE, tag))
return `<ref id="` + alias + `"/>`
})
}
func publicAttr(expression *regexp.Regexp, tag string) string {
match := expression.FindStringSubmatch(tag)
if len(match) != 2 {
return ""
}
return html.UnescapeString(match[1])
}
func (r *Registry) decodeJSON(raw string, encode bool) string {
if r == nil || strings.TrimSpace(raw) == "" {
return raw
}
var value interface{}
if err := json.Unmarshal([]byte(raw), &value); err != nil {
return raw
}
value = r.walkJSON(value, encode)
encoded, err := json.Marshal(value)
if err != nil {
return raw
}
return string(encoded)
}
func (r *Registry) walkJSON(value interface{}, encode bool) interface{} {
switch typed := value.(type) {
case string:
if encode {
if alias := r.aliasForRealValue(typed); alias != "" {
return alias
}
return typed
}
if real := r.realForAlias(typed); real != "" {
return real
}
return typed
case []interface{}:
for i := range typed {
typed[i] = r.walkJSON(typed[i], encode)
}
case map[string]interface{}:
for key, item := range typed {
typed[key] = r.walkJSON(item, encode)
}
}
return value
}
func (r *Registry) aliasForRealValue(real string) string {
r.mu.RLock()
defer r.mu.RUnlock()
if ref := r.chunkByID[real]; ref != nil {
return ref.Alias
}
if alias := r.docToAlias[real]; alias != "" {
return alias
}
if alias := r.kbToAlias[real]; alias != "" {
return alias
}
if ref := r.webByURL[canonicalWebURL(real)]; ref != nil {
return ref.Alias
}
return ""
}
func (r *Registry) realForAlias(alias string) string {
r.mu.RLock()
defer r.mu.RUnlock()
if ref := r.chunkByAlias[alias]; ref != nil {
return ref.ChunkID
}
if real := r.aliasToDoc[alias]; real != "" {
return real
}
if real := r.aliasToKB[alias]; real != "" {
return real
}
if ref := r.webByAlias[alias]; ref != nil {
return ref.URL
}
return ""
}
// CompactKnownText is intentionally limited to identifiers already registered
// from structured runtime/tool data. It is used for metadata envelopes, not
// arbitrary retrieved prose.
func (r *Registry) CompactKnownText(text string) string {
if r == nil || text == "" {
return text
}
type pair struct{ real, alias string }
r.mu.RLock()
pairs := make([]pair, 0, len(r.chunkByID)+len(r.docToAlias)+len(r.kbToAlias)+len(r.webByURL))
for real, ref := range r.chunkByID {
pairs = append(pairs, pair{real, ref.Alias})
}
for real, alias := range r.docToAlias {
pairs = append(pairs, pair{real, alias})
}
for real, alias := range r.kbToAlias {
pairs = append(pairs, pair{real, alias})
}
for _, ref := range r.webByURL {
pairs = append(pairs, pair{ref.URL, ref.Alias})
}
r.mu.RUnlock()
sort.SliceStable(pairs, func(i, j int) bool { return len(pairs[i].real) > len(pairs[j].real) })
for _, item := range pairs {
if item.real != "" {
text = strings.ReplaceAll(text, item.real, item.alias)
}
}
return text
}
var (
refTagRE = regexp.MustCompile(`(?i)<ref\s+id\s*=\s*"([^"]+)"\s*/?>`)
refCandidateRE = regexp.MustCompile(`(?is)<ref(?:\s|$)[^>]*(?:>|$)`)
modelKBTagRE = regexp.MustCompile(`(?is)<kb(?:\s|$)[^>]*(?:>|$)`)
modelWebTagRE = regexp.MustCompile(`(?is)<web(?:\s|$)[^>]*(?:>|$)`)
)
var (
documentAttrRE = regexp.MustCompile(`(?i)\bknowledge_id\s*=\s*"([^"]+)"`)
documentElementRE = regexp.MustCompile(`(?is)<knowledge_id>\s*([^<]+?)\s*</knowledge_id>`)
kbAttrRE = regexp.MustCompile(`(?i)\b(?:knowledge_base_id|kb_id)\s*=\s*"([^"]+)"`)
kbElementRE = regexp.MustCompile(`(?is)<(?:knowledge_base_id|kb_id)>\s*([^<]+?)\s*</(?:knowledge_base_id|kb_id)>`)
)
// registerLabeledReferences covers metadata-oriented tools that do not have a
// dedicated compact renderer. Only explicit ID labels are recognized; UUID-like
// text in retrieved content is never guessed to be a source identifier.
func (r *Registry) registerLabeledReferences(text string) {
if r == nil || text == "" {
return
}
for _, expression := range []*regexp.Regexp{documentAttrRE, documentElementRE} {
for _, match := range expression.FindAllStringSubmatch(text, -1) {
if len(match) == 2 {
r.RegisterDocument(strings.TrimSpace(match[1]))
}
}
}
for _, expression := range []*regexp.Regexp{kbAttrRE, kbElementRE} {
for _, match := range expression.FindAllStringSubmatch(text, -1) {
if len(match) == 2 {
r.RegisterKnowledgeBase(strings.TrimSpace(match[1]))
}
}
}
}
// ExpandText converts the private model protocol into the existing public
// <kb/> / <web/> contract. Unknown aliases fail closed and disappear.
func (r *Registry) ExpandText(text string) string {
if r == nil || text == "" {
return text
}
// Public citation tags are output-only. Drop any instance written directly
// by the model, then create canonical tags solely from registered aliases.
text = modelKBTagRE.ReplaceAllString(text, "")
text = modelWebTagRE.ReplaceAllString(text, "")
return refCandidateRE.ReplaceAllStringFunc(text, func(tag string) string {
match := refTagRE.FindStringSubmatch(tag)
if len(match) != 2 {
return ""
}
alias := strings.ToLower(match[1])
r.mu.RLock()
chunkRef := r.chunkByAlias[alias]
webRef := r.webByAlias[alias]
r.mu.RUnlock()
if chunkRef != nil {
attrs := fmt.Sprintf(`doc="%s" chunk_id="%s"`, escapeAttr(chunkRef.DocumentTitle), escapeAttr(chunkRef.ChunkID))
if chunkRef.KnowledgeBaseID != "" {
attrs += fmt.Sprintf(` kb_id="%s"`, escapeAttr(chunkRef.KnowledgeBaseID))
}
return "<kb " + attrs + " />"
}
if webRef != nil {
return fmt.Sprintf(`<web url="%s" title="%s" />`, escapeAttr(webRef.URL), escapeAttr(webRef.Title))
}
return ""
})
}
func escapeAttr(value string) string { return html.EscapeString(value) }
func (r *Registry) ExpandResponse(response *types.ChatResponse) {
if response == nil {
return
}
response.Content = r.ExpandText(response.Content)
response.ReasoningContent = r.ExpandText(response.ReasoningContent)
r.DecodeToolCalls(response.ToolCalls)
}
// StreamExpander prevents partial private <ref/> tags from reaching SSE while
// preserving normal streaming for all other content.
type StreamExpander struct {
registry *Registry
pending string
}
func NewStreamExpander(registry *Registry) *StreamExpander {
return &StreamExpander{registry: registry}
}
func (d *StreamExpander) Feed(chunk string) string {
if d == nil || d.registry == nil {
return chunk
}
data := d.pending + chunk
d.pending = ""
var out strings.Builder
for data != "" {
idx := strings.Index(data, "<")
if idx < 0 {
out.WriteString(data)
break
}
out.WriteString(data[:idx])
data = data[idx:]
lower := strings.ToLower(data)
if isSourceTagPending(lower) && !strings.Contains(data, ">") {
d.pending = data
break
}
if isRefTagStart(lower) {
end := strings.IndexByte(data, '>')
if end < 0 {
d.pending = data
break
}
tag := data[:end+1]
if refTagRE.MatchString(tag) {
out.WriteString(d.registry.ExpandText(tag))
}
data = data[end+1:]
continue
}
if isNamedTagStart(lower, "kb") || isNamedTagStart(lower, "web") {
end := strings.IndexByte(data, '>')
if end < 0 {
d.pending = data
break
}
data = data[end+1:]
continue
}
out.WriteByte('<')
data = data[1:]
}
return out.String()
}
func isRefTagStart(value string) bool {
return isNamedTagStart(value, "ref")
}
func isNamedTagStart(value, name string) bool {
prefix := "<" + name
if !strings.HasPrefix(value, prefix) {
return false
}
if len(value) == len(prefix) {
return true
}
next := value[len(prefix)]
return next == ' ' || next == '\t' || next == '\r' || next == '\n' || next == '>'
}
func isSourceTagPending(value string) bool {
for _, name := range []string{"ref", "kb", "web"} {
prefix := "<" + name
if (len(value) <= len(prefix) && strings.HasPrefix(prefix, value)) || isNamedTagStart(value, name) {
return true
}
}
return false
}
func (d *StreamExpander) Flush() string {
if d == nil {
return ""
}
pending := d.pending
d.pending = ""
lower := strings.ToLower(pending)
if isSourceTagPending(lower) {
return ""
}
return d.registry.ExpandText(pending)
}
+281
View File
@@ -0,0 +1,281 @@
package llmreference
import (
"strings"
"testing"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestRegistryChunkAliasIsStableAndExpandsCanonicalCitation(t *testing.T) {
registry := NewRegistry()
first := registry.RegisterChunk(ChunkReference{
ChunkID: "chunk-uuid-1",
KnowledgeID: "knowledge-uuid-1",
KnowledgeBaseID: "kb-uuid-1",
DocumentTitle: "Architecture.md",
ChunkIndex: 7,
})
second := registry.RegisterChunk(ChunkReference{
ChunkID: "chunk-uuid-1",
KnowledgeID: "knowledge-uuid-1",
DocumentTitle: "Architecture.md",
})
require.Equal(t, "c1", first)
require.Equal(t, first, second)
require.Equal(t,
`claim <kb doc="Architecture.md" chunk_id="chunk-uuid-1" kb_id="kb-uuid-1" />`,
registry.ExpandText(`claim <ref id="c1"/>`),
)
}
func TestRegistryDecodesAliasesInNestedToolArguments(t *testing.T) {
registry := NewRegistry()
registry.RegisterDocument("knowledge-uuid-1")
registry.RegisterKnowledgeBase("kb-uuid-1")
registry.RegisterWeb("https://example.com/page", "Example")
registry.RegisterChunk(ChunkReference{ChunkID: "chunk-uuid-1"})
calls := []types.LLMToolCall{{
Function: types.FunctionCall{
Arguments: `{"knowledge_id":"d1","knowledge_base_ids":["b1"],"url":"w1","chunk_id":"c1"}`,
},
}}
registry.DecodeToolCalls(calls)
require.JSONEq(t,
`{"knowledge_id":"knowledge-uuid-1","knowledge_base_ids":["kb-uuid-1"],"url":"https://example.com/page","chunk_id":"chunk-uuid-1"}`,
calls[0].Function.Arguments,
)
}
func TestStreamExpanderHoldsSplitReferenceAndDropsUnknown(t *testing.T) {
registry := NewRegistry()
registry.RegisterChunk(ChunkReference{ChunkID: "chunk-1", DocumentTitle: "Doc"})
expander := NewStreamExpander(registry)
require.Equal(t, "before ", expander.Feed(`before <ref id="`))
require.Equal(t, `<kb doc="Doc" chunk_id="chunk-1" /> after`, expander.Feed(`c1"/> after`))
require.Empty(t, expander.Flush())
require.Equal(t, "x y", registry.ExpandText(`x <ref id="c999"/> y`))
require.Equal(t, "x y", registry.ExpandText(`x <ref id="d1"/> y`))
require.Equal(t, "x y", registry.ExpandText(`x <ref id='c1'/> y`))
require.Equal(t, "x ", registry.ExpandText(`x <ref id="c1"`))
require.Equal(t, "x y", registry.ExpandText(`x <kb doc="forged" chunk_id="forged" /> y`))
require.Equal(t, "x ", expander.Feed(`x <we`))
require.Equal(t, " y", expander.Feed(`b url="https://forged" /> y`))
}
func TestEncodeMessagesCompactsCanonicalCitationsFromHistory(t *testing.T) {
registry := NewRegistry()
messages := []chat.Message{{
Role: "assistant",
Content: `Knowledge <kb doc="A &amp; B.pdf" chunk_id="chunk-real" kb_id="kb-real" />; ` +
`web <web url="https://example.com/a?x=1&amp;y=2" title="Example &amp; More" />`,
}}
encoded := registry.EncodeMessages(messages)
require.Equal(t, `Knowledge <ref id="c1"/>; web <ref id="w1"/>`, encoded[0].Content)
require.NotContains(t, encoded[0].Content, "chunk-real")
require.NotContains(t, encoded[0].Content, "https://example.com")
require.Equal(t,
`<kb doc="A &amp; B.pdf" chunk_id="chunk-real" kb_id="kb-real" /> <web url="https://example.com/a?x=1&amp;y=2" title="Example &amp; More" />`,
registry.ExpandText(`<ref id="c1"/> <ref id="w1"/>`),
)
}
func TestEncodeMessagesMigratesLegacyToolHistoryAtReadTime(t *testing.T) {
registry := NewRegistry()
messages := []chat.Message{
{
Role: "assistant",
ToolCalls: []chat.ToolCall{{
Function: chat.FunctionCall{
Name: "knowledge_search",
Arguments: `{"knowledge_base_ids":["kb-real"],"knowledge_ids":["doc-real"]}`,
},
}},
},
{
Role: "tool",
Name: "knowledge_search",
Content: `<chunk chunk_id="chunk-real" knowledge_id="doc-real" knowledge_base_id="kb-real" ` +
`knowledge_title="Legacy Doc">legacy content</chunk>`,
},
{
Role: "assistant",
Content: `Legacy answer <kb doc="Legacy Doc" chunk_id="chunk-real" kb_id="kb-real" />`,
},
}
encoded := registry.EncodeMessages(messages)
require.JSONEq(t, `{"knowledge_base_ids":["b1"],"knowledge_ids":["d1"]}`,
encoded[0].ToolCalls[0].Function.Arguments)
require.Contains(t, encoded[1].Content, `chunk_id="c1"`)
require.Contains(t, encoded[1].Content, `knowledge_id="d1"`)
require.Contains(t, encoded[1].Content, `knowledge_base_id="b1"`)
require.Equal(t, `Legacy answer <ref id="c1"/>`, encoded[2].Content)
require.Equal(t,
`<kb doc="Legacy Doc" chunk_id="chunk-real" kb_id="kb-real" />`,
registry.ExpandText(`<ref id="c1"/>`),
)
}
func TestEncodeMessagesDoesNotTreatLegacyPromptExampleAsARealSource(t *testing.T) {
registry := NewRegistry()
messages := []chat.Message{{
Role: "system",
Content: `Old rule: cite <kb doc="..." chunk_id="..." />`,
}}
encoded := registry.EncodeMessages(messages)
require.Equal(t, messages[0].Content, encoded[0].Content)
require.Zero(t, registry.Count())
}
func TestModelOutputGroupsChunksAndReusesAliasAcrossTools(t *testing.T) {
registry := NewRegistry()
search := &types.ToolResult{
Success: true,
Output: "raw UUID output",
Data: map[string]interface{}{
"display_type": "search_results",
"results": []map[string]interface{}{
{
"chunk_id": "chunk-uuid-1",
"knowledge_id": "knowledge-uuid-1",
"knowledge_base_id": "kb-uuid-1",
"knowledge_title": "Doc A",
"chunk_index": 3,
"content": "full content",
},
},
},
}
first := registry.ModelOutput(search)
require.Contains(t, first, `<document id="d1" kb="b1" title="Doc A">`)
require.Contains(t, first, `<chunk id="c1" index="3" view="full">`)
require.NotContains(t, first, "chunk-uuid-1")
require.NotContains(t, first, "knowledge-uuid-1")
deepRead := &types.ToolResult{
Success: true,
Data: map[string]interface{}{
"display_type": "knowledge_chunks_list",
"knowledge_id": "knowledge-uuid-1",
"knowledge_title": "Doc A",
"total_chunks": int64(1),
"fetched_chunks": 1,
"chunks": []map[string]interface{}{
{"chunk_id": "chunk-uuid-1", "knowledge_id": "knowledge-uuid-1", "knowledge_base": "kb-uuid-1", "chunk_index": 3, "content": "deep content"},
},
},
}
second := registry.ModelOutput(deepRead)
require.Contains(t, second, `<chunk id="c1"`)
require.False(t, strings.Contains(second, "c2"))
}
func TestModelOutputWebAliasExpandsToWebCitation(t *testing.T) {
registry := NewRegistry()
output := registry.ModelOutput(&types.ToolResult{
Success: true,
Data: map[string]interface{}{
"display_type": "web_search_results",
"results": []map[string]interface{}{
{"title": "Example", "url": "https://example.com/a", "snippet": "snippet"},
},
},
})
require.Contains(t, output, `<page id="w1" title="Example">`)
require.NotContains(t, output, "https://example.com/a")
require.Equal(t,
`<web url="https://example.com/a" title="Example" />`,
registry.ExpandText(`<ref id="w1"/>`),
)
}
func TestModelOutputDocumentInfoUsesDocumentAndFAQAliases(t *testing.T) {
registry := NewRegistry()
result := &types.ToolResult{
Success: true,
Output: "raw IDs must not be used",
Data: map[string]interface{}{
"display_type": "document_info",
"documents": []map[string]interface{}{
{
"knowledge_id": "doc-real-id",
"title": "Architecture",
"description": "System overview",
"type": "file",
"chunk_count": 12,
},
{
"faq_id": "faq-chunk-real-id",
"knowledge_id": "faq-container-real-id",
"faq_question": "How does it work?",
"faq_answers": []string{"With short aliases."},
"is_faq": true,
},
},
},
}
output := registry.ModelOutput(result)
for _, raw := range []string{"doc-real-id", "faq-chunk-real-id", "faq-container-real-id"} {
require.NotContains(t, output, raw)
}
require.Contains(t, output, `<document id="d1" title="Architecture"`)
require.Contains(t, output, `<chunk id="c1" type="faq">`)
require.Contains(t, registry.ExpandText(`Answer <ref id="c1"/>`), `chunk_id="faq-chunk-real-id"`)
}
func TestModelOutputCompactsLabeledWikiReferences(t *testing.T) {
registry := NewRegistry()
result := &types.ToolResult{
Success: true,
Output: `<wiki_page>
<knowledge_base_id>kb-real-id</knowledge_base_id>
<sources><source knowledge_id="doc-real-id">Source</source></sources>
</wiki_page>`,
}
output := registry.ModelOutput(result)
require.NotContains(t, output, "kb-real-id")
require.NotContains(t, output, "doc-real-id")
require.Contains(t, output, `<knowledge_base_id>b1</knowledge_base_id>`)
require.Contains(t, output, `knowledge_id="d1"`)
toolCalls := []types.LLMToolCall{{Function: types.FunctionCall{Arguments: `{"knowledge_id":"d1","knowledge_base_id":"b1"}`}}}
registry.DecodeToolCalls(toolCalls)
require.JSONEq(t, `{"knowledge_id":"doc-real-id","knowledge_base_id":"kb-real-id"}`, toolCalls[0].Function.Arguments)
}
func TestModelOutputGraphResultsUseChunkAliases(t *testing.T) {
registry := NewRegistry()
output := registry.ModelOutput(&types.ToolResult{
Success: true,
Data: map[string]interface{}{
"display_type": "graph_query_results",
"results": []map[string]interface{}{{
"chunk_id": "graph-chunk-real",
"chunk_index": 4,
"knowledge_id": "graph-doc-real",
"knowledge_base_id": "graph-kb-real",
"knowledge_title": "Graph Source",
"content": "A relates to B.",
}},
},
})
require.Contains(t, output, `<retrieval type="knowledge" mode="graph">`)
require.Contains(t, output, `<chunk id="c1" index="4" view="full">`)
require.NotContains(t, output, "graph-chunk-real")
require.Equal(t,
`<kb doc="Graph Source" chunk_id="graph-chunk-real" kb_id="graph-kb-real" />`,
registry.ExpandText(`<ref id="c1"/>`),
)
}