feat: Enhance agent retrieval strategies and regex handling

- Updated the agent system prompt to emphasize the use of regex for chunk searches, improving retrieval accuracy and efficiency.
- Introduced a new `Capabilities` field in the `KnowledgeBaseInfo` struct to define the retrieval surfaces available for each knowledge base, guiding the agent's strategy selection.
- Enhanced the `grep_chunks` tool to support regex queries, allowing for more flexible and powerful text pattern matching in knowledge base chunks.
- Improved JSON handling in the `RepairJSON` function to address invalid escape sequences, particularly for regex patterns, ensuring robust parsing and error handling.
- Updated documentation and comments across various files to clarify usage and expectations for regex and knowledge base capabilities.

These changes significantly enhance the agent's ability to retrieve relevant information while ensuring the integrity of input data handling.
This commit is contained in:
wizardchen
2026-04-21 19:19:57 +08:00
parent 4022f134bd
commit 37a517612d
9 changed files with 688 additions and 328 deletions
@@ -128,7 +128,7 @@ templates:
### Core Retrieval Strategy (Strict Sequence)
For every retrieval attempt (Phase 1 or Phase 3), follow this exact chain:
1. **Entity Anchoring (grep_chunks):** Use short keywords (1-3 words) to find candidate documents.
1. **Entity Anchoring (grep_chunks):** Regex search over chunk content using PostgreSQL POSIX `~*` (case-insensitive; `REGEXP` on MySQL/SQLite). STRONGLY PREFER using regex to search for multiple concepts at once — pack 2-3 terms into one alternation query (e.g. `stardust|skyvault|psionic`) rather than firing several single-keyword calls. Plain literal text also works (`engine` matches anywhere in chunk content). Each match returns a `<match_snippet>` you can use to judge relevance before deep-reading. Input field is `queries` (array, 1-5).
2. **Semantic Expansion (knowledge_search):** Use vector search for context (filter by IDs from step 1 if applicable).
3. **Deep Contextualization (list_knowledge_chunks): MANDATORY.**
* Rule: After Step 1 or 2 returns knowledge_ids, you MUST call this tool.
@@ -137,7 +137,7 @@ templates:
5. **Web Fallback (web_search):** Use ONLY if Web Search is Enabled AND the Deep Read in Step 3 confirms the data is missing or irrelevant.
### Tool Selection Guidelines
* **grep_chunks / knowledge_search:** Your "Index". Use these to find *where* the information might be.
* **grep_chunks / knowledge_search:** Your "Index". Use these to find *where* the information might be. `grep_chunks` uses PostgreSQL POSIX `~*` regex (case-insensitive; `REGEXP` on MySQL/SQLite) — input field is `queries` (1–5 regex strings); STRONGLY PREFER one alternation query (`a|b|c`) over multiple single-keyword calls. Literal text like `engine` also works. `knowledge_search` accepts 1–5 semantic `queries` and returns `<chunk>` entries with full content, scores, and a `<match_snippet>` per result.
* **list_knowledge_chunks:** Your "Eyes". MUST be used after every search. Use to read what the information is.
* **web_search / web_fetch:** Use these ONLY when Web Search is Enabled and KB retrieval is insufficient.
* **todo_write (optional, only if enabled):** Your "Manager" for tracking multi-step research. Only use it when the user has added it to the tool list.
@@ -379,12 +379,16 @@ templates:
mode: "smart-reasoning"
content: |
<role>
You are WeKnora Hybrid Researcher, an intelligent retrieval assistant developed by Tencent. The Knowledge Base(s) you work on expose TWO complementary indexes over the same source documents:
You are WeKnora Hybrid Researcher, an intelligent retrieval assistant developed by Tencent. Your bound Knowledge Base(s) may expose up to TWO complementary indexes over the same source documents:
- **Wiki Index** — LLM-synthesized Markdown pages (summaries, entities, concepts) organized as an interlinked graph. Best for navigation, relations, "what is X", and high-level context.
- **Chunk Index (RAG)** — raw document chunks with vector + BM25 retrieval. Best for precise quotes, numbers, code snippets, recent documents, and anything the wiki hasn't synthesized yet.
Your job is to orchestrate BOTH intelligently, using each where it is strongest.
Not every bound KB exposes both surfaces. Each `<knowledge_base>` entry below declares its available surfaces via a `capabilities="..."` attribute; consult it before choosing a retrieval strategy. Your job is to orchestrate whichever surfaces are available, using each where it is strongest.
</role>
<bound_knowledge_bases>
{{knowledge_bases}}
</bound_knowledge_bases>
<mission>
Deliver accurate, traceable, and well-cited answers. For most questions, start from the wiki to establish the conceptual map, then drill into chunks for ground-truth evidence. For fact-heavy or "give me the exact quote" questions, go straight to chunks. Never rely on internal parametric knowledge.
</mission>
@@ -392,41 +396,51 @@ templates:
<workflow>
IMPORTANT: The KB(s) you are bound to may expose any subset of {wiki, chunks}.
The bound KB set is heterogeneous — some KBs may be wiki-only, some may be
chunk-only (pure RAG), and some may have both. Do NOT assume both are always
available. Your FIRST move on a fresh question is to probe what is reachable,
THEN pick a strategy.
chunk-only (pure RAG), and some may have both. Do NOT probe this via
search — it is declared deterministically on each `<knowledge_base>` entry
in the `<knowledge_bases>` block above, in the `capabilities="..."`
attribute. Trust that attribute as the source of truth.
Default "Wiki-first, RAG-verify" cycle (adapt as needed):
0. **Read Surface Capabilities (no tool call needed):**
For every KB in scope, look up its `capabilities` attribute:
* `capabilities="wiki,chunks"` → full hybrid available
* `capabilities="wiki"` → wiki only (no chunk retrieval)
* `capabilities="chunks"` → pure RAG (no wiki)
* missing / empty → treat as chunks-only and rely on `list_knowledge_chunks`
Let `WIKI_KBS` = the subset whose capabilities contain `wiki`, and
`CHUNK_KBS` = the subset whose capabilities contain `chunks`. The rest
of this workflow is conditional on which of these is non-empty.
0. **Probe Available Surfaces (MANDATORY first action on every new question):**
- Issue ONE parallel round: call `wiki_search` AND `knowledge_search` with the same keywords.
- Inspect the results to decide the effective mode for this turn:
* Both have hits → run the full "Wiki-first, RAG-verify" cycle below.
* Only wiki has hits → skip steps 3–4, answer from wiki only (step 2 + step 5 with wiki citations only).
* Only chunks have hits → skip step 2, go straight to step 3 (pure RAG).
* Neither has hits → widen keywords and retry once; if still empty, tell the user the KB has no relevant material.
1. **Classify the Query (only branches reachable given surface availability):**
- *Conceptual / relational / "what is / how does":* prefer wiki when `WIKI_KBS` is non-empty; otherwise fall back to chunks.
- *Specific fact / exact quote / numeric / code-level:* prefer chunks when `CHUNK_KBS` is non-empty; otherwise fall back to wiki.
- *Mixed / unclear:* if both sides are available, start from wiki (cheaper context). Otherwise use whichever is available.
1. **Classify the Query (if both surfaces are available):**
- *Conceptual / relational / "what is / how does":* start from wiki.
- *Specific fact / exact quote / numeric / code-level:* start from RAG chunks.
- *Mixed / unclear:* start from wiki — it's cheaper context.
2. **Wiki Entry Point (if wiki-leaning AND wiki is available):**
- `wiki_search` with 1–3 keyword regex queries to discover slugs (prefer regex alternation over multiple calls).
- Call `wiki_read_page` on the top 1–2 slugs to load their full Markdown plus linked-page summaries.
2. **Wiki Entry Point (only when `WIKI_KBS` is non-empty AND wiki was chosen):**
- `wiki_search` with 1–3 regex queries (prefer one alternation query over multiple calls), restricted to `WIKI_KBS`.
- Call `wiki_read_page` on the top 1–2 slugs to load full Markdown plus linked-page summaries.
- Follow `Links to` / `Linked from` for 1 extra hop if the answer isn't yet complete.
3. **Chunk Grounding (MANDATORY for any factual claim, WHEN chunks are available):**
- After reading wiki, or when the query demands precise evidence, call `knowledge_search` and/or `grep_chunks` to find chunks in the source documents.
- Whenever a search returns matched `knowledge_id`s / `chunk_id`s, you MUST call `list_knowledge_chunks` to read the full content of those chunks. Do not answer from search snippets alone.
- If chunks are NOT available in any bound KB, skip this step and rely on wiki — but make the limitation explicit in your final answer.
3. **Chunk Grounding (only when `CHUNK_KBS` is non-empty):**
MANDATORY for any factual claim the user will verify (exact quotes, numbers, code, names).
- Call `knowledge_search` (semantic) and/or `grep_chunks` (regex) scoped to `CHUNK_KBS`.
- Whenever a search returns matched `knowledge_id`s / `chunk_id`s, you MUST call `list_knowledge_chunks` to read the full content. Do NOT answer from search snippets alone.
- If the first pass returns nothing relevant AND `WIKI_KBS` is also non-empty, fall back to a wiki search before giving up — the wiki may still list the concept under a different phrasing.
- If `CHUNK_KBS` is empty, skip this step and rely on wiki, but make that limitation explicit in the final answer.
4. **Cross-Check (WHEN both surfaces contributed):** If wiki claims and raw chunks disagree, trust the raw chunks (they are the source of truth) and call `wiki_flag_issue` on the offending wiki page to report the discrepancy for human review.
4. **Cross-Check (only when BOTH surfaces contributed to the answer):**
If wiki claims and raw chunks disagree, trust the raw chunks and call `wiki_flag_issue` on the offending wiki page to report the discrepancy for human review.
5. **Synthesize:**
- Build an answer that is grounded in the actual content you retrieved this turn, structured with whatever navigation scaffolding you gathered.
- Cite every source you used: wiki pages as `[[slug|display name]]`, and chunks as `[knowledge_id#chunk_id]` or the citation style expected by the product.
- If you fell back to a single surface, say so briefly ("The bound KB only exposes X, so this answer is drawn entirely from X").
5. **Fallback on Empty First Pass (single retry, then stop):**
If the primary surface returned nothing useful:
* Widen / re-phrase the query (alternation, synonyms, romanized vs CJK form) and retry ONCE on the same surface.
* If still empty and the *other* surface is available, try it once.
* If both come up empty, tell the user the bound KB has no relevant material — do NOT invent.
6. **Synthesize:**
- Build an answer grounded in content actually retrieved this turn.
- Cite every source: wiki pages as `[[slug|display name]]`, chunks as `[knowledge_id#chunk_id]` (or the citation style expected by the product).
- When only one surface was reachable, note it briefly ("The bound KB only exposes X, so this answer is drawn entirely from X").
</workflow>
<constraints>
@@ -444,7 +458,7 @@ templates:
* **wiki_search:** Wiki entry point. Prefer POSIX regex (`~*`) with alternation to cover multiple concepts in one call.
* **wiki_read_page:** Load the full content (and linked summaries) of 1–3 top slugs. Batch multiple slugs in a single call when possible.
* **knowledge_search:** Semantic retrieval over raw chunks. Use when the user is looking for concepts, paraphrased information, or when wiki didn't cover the answer.
* **grep_chunks:** Lexical / keyword retrieval over raw chunks. Use for exact tokens (error messages, function names, product codes) or when semantic search misses.
* **grep_chunks:** Regex retrieval over raw chunks using PostgreSQL POSIX `~*` (case-insensitive; `REGEXP` on MySQL/SQLite). Input field is `queries` (1–5 regex strings). STRONGLY PREFER using regex to search for multiple concepts at once — one alternation query `error_code_a|error_code_b` beats multiple single-keyword calls. Literal text also works. Use for exact tokens (error messages, function names, product codes) or when semantic search misses.
* **list_knowledge_chunks:** MANDATORY after any chunk search — loads the full text of the matched chunks.
* **get_document_info:** Fetch metadata (title, upload time, page count) when you need to cite a document properly.
* **wiki_flag_issue:** Use when wiki and chunks disagree, or when the user points out a wiki error.
+13 -3
View File
@@ -78,7 +78,13 @@ type KnowledgeBaseInfo struct {
Type string // Knowledge base type: "document" or "faq"
Description string
DocCount int
RecentDocs []RecentDocInfo // Recently added documents (up to 10)
// Capabilities lists the retrieval surfaces this KB exposes. Any subset of
// {"wiki", "chunks"}. "chunks" is present when the KB has vector and/or
// keyword (BM25) indexing enabled. This is the *deterministic* source of
// truth the agent should consult before picking a retrieval strategy —
// significantly more reliable than running probing searches.
Capabilities []string
RecentDocs []RecentDocInfo // Recently added documents (up to 10)
}
// PlaceholderDefinition defines a placeholder exposed to UI/configuration
@@ -118,8 +124,12 @@ func formatKnowledgeBaseList(kbInfos []*KnowledgeBaseInfo) string {
if kbType == "" {
kbType = "document"
}
b.WriteString(fmt.Sprintf("<knowledge_base id=\"%s\" name=\"%s\" type=\"%s\" doc_count=\"%d\">\n",
kb.ID, kb.Name, kbType, kb.DocCount))
capsAttr := ""
if len(kb.Capabilities) > 0 {
capsAttr = fmt.Sprintf(" capabilities=\"%s\"", strings.Join(kb.Capabilities, ","))
}
b.WriteString(fmt.Sprintf("<knowledge_base id=\"%s\" name=\"%s\" type=\"%s\" doc_count=\"%d\"%s>\n",
kb.ID, kb.Name, kbType, kb.DocCount, capsAttr))
if kb.Description != "" {
b.WriteString(fmt.Sprintf("<description>%s</description>\n", kb.Description))
}
+318 -250
View File
@@ -5,8 +5,10 @@ import (
"encoding/json"
"fmt"
"math"
"regexp"
"sort"
"strings"
"sync"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/searchutil"
@@ -16,86 +18,70 @@ import (
var grepChunksTool = BaseTool{
name: ToolGrepChunks,
description: `Unix-style text pattern matching tool for knowledge base chunks.
Searches for text patterns in chunk content using strict literal text matching (fixed-string search). This tool performs exact keyword lookup, not semantic search.
## Core Function
Performs exact, literal text pattern matching. Accepts multiple patterns and returns chunks matching any of them (OR logic).
## CRITICAL – Keyword Extraction Rules
This tool MUST receive **short, high-value keywords** only.
**Do NOT use long phrases, sentences, or multi-word expressions.**
Provide only the **minimal core entities** extracted from user query, such as:
- Proper nouns
- Key concepts
- Domain terms
- Distinct entities that define the query
### Requirements
- Keywords should be **1–3 words maximum**
- Focus exclusively on **core entities**, not descriptions
- Break complex input into individual, essential keywords
- Avoid phrases, explanations, or anything that reduces match probability
- Preserve precision details embedded in the query (e.g., version numbers, build IDs) when they materially define the entity being matched.
Long phrases dramatically reduce recall because chunks rarely contain identical wording.
Only short, atomic keywords ensure accurate matching and avoid unrelated retrieval.
## Usage
grep_chunks scans enabled chunks across the specified knowledge bases and returns those containing any provided keyword. Matching is case-insensitive, with chunk indices and local context included.
## When to Use
- Extracting core entities from user input
- Exact keyword presence checks
- Fast preliminary filtering before semantic search
- Situations requiring deterministic text search
`,
description: `Search knowledge base chunk content using PostgreSQL POSIX regular expressions (~* operator, case-insensitive; REGEXP on MySQL/SQLite).
STRONGLY PREFER using regex to search for multiple concepts at once rather than simple plain text queries.
Returns matching chunks with per-pattern hit counts and a <match_snippet> around the first match (each tagged with its knowledge_id and chunk_id).
Examples:
- Alternation (RECOMMENDED): "stardust|skyvault" (matches either word)
- Multiple terms (RECOMMENDED): "psionic.*engine" (matches both words in order)
- Word boundary / anchor: "\\brag\\b" or "^chapter\\s+\\d+"
- Plain text: "engine" (matches literal substring anywhere in chunk content)
IMPORTANT — JSON escaping: every backslash in a regex MUST be written as \\ inside the JSON tool arguments (e.g. to search for literal "C++" write "C\\+\\+", NOT "C\+\+"; for "\d+" write "\\d+"). Plain "\+" / "\d" etc. are invalid JSON escapes and will fail to parse.
Use this to locate candidate chunks by exact identifiers, error codes, product names, or recurring terms. Pair with list_knowledge_chunks afterwards to read the full context around any promising chunk_id.`,
schema: json.RawMessage(`{
"type": "object",
"properties": {
"patterns": {
"queries": {
"type": "array",
"description": "REQUIRED: Text patterns to search for. Can be a single pattern or multiple patterns. Treated as literal text (fixed string matching). Results match any of the patterns (OR logic).",
"items": {
"type": "string"
},
"minItems": 1
"items": { "type": "string" },
"description": "List of regex queries to run. A chunk matches when ANY query matches its content. Prefer one alternation query (\"a|b|c\") over multiple single-keyword queries.",
"minItems": 1,
"maxItems": 5
},
"knowledge_base_ids": {
"type": "array",
"description": "Filter by knowledge base IDs. If empty, searches all allowed KBs.",
"items": {
"type": "string"
}
"items": { "type": "string" },
"description": "Optional: restrict search to specific KB IDs within the agent scope."
},
"max_results": {
"limit": {
"type": "integer",
"description": "Maximum number of matching chunks to return (default: 50, max: 200)",
"default": 50,
"description": "Max matching chunks to return (default 30, max 100).",
"default": 30,
"minimum": 1,
"maximum": 200
"maximum": 100
}
},
"required": ["patterns"]
"required": ["queries"]
}`),
}
// GrepChunksInput defines the input parameters for grep chunks tool
// GrepChunksInput defines the input parameters for grep chunks tool.
// The canonical parameter names are `queries` and `limit` (mirroring
// wiki_search). The legacy `patterns` and `max_results` keys remain accepted
// so older model outputs or external callers don't break silently.
type GrepChunksInput struct {
Patterns []string `json:"patterns" `
Queries []string `json:"queries,omitempty"`
Patterns []string `json:"patterns,omitempty"` // legacy alias for queries
KnowledgeBaseIDs []string `json:"knowledge_base_ids,omitempty"`
MaxResults int `json:"max_results,omitempty"`
Limit int `json:"limit,omitempty"`
MaxResults int `json:"max_results,omitempty"` // legacy alias for limit
}
// GrepChunksTool performs text pattern matching in knowledge base chunks
// Similar to grep command in Unix-like systems, but operates on knowledge base content
// GrepChunksTool performs regex pattern matching across knowledge base chunks.
// PostgreSQL: uses the case-insensitive POSIX operator ~*.
// MySQL/SQLite: falls back to REGEXP.
//
// The tool tracks previously-returned chunk IDs per-instance (one instance per
// agent session) so that a subsequent search hitting the same chunk can be
// rendered compactly with an `already_seen="true"` marker instead of replaying
// the snippet, mirroring the UX of wiki_search.
type GrepChunksTool struct {
BaseTool
db *gorm.DB
searchTargets types.SearchTargets // Pre-computed unified search targets with KB-tenant mapping
searchTargets types.SearchTargets
mu sync.Mutex
seenChunks map[string]bool
}
// NewGrepChunksTool creates a new grep chunks tool
@@ -104,6 +90,7 @@ func NewGrepChunksTool(db *gorm.DB, searchTargets types.SearchTargets) *GrepChun
BaseTool: grepChunksTool,
db: db,
searchTargets: searchTargets,
seenChunks: make(map[string]bool),
}
}
@@ -111,7 +98,6 @@ func NewGrepChunksTool(db *gorm.DB, searchTargets types.SearchTargets) *GrepChun
func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*types.ToolResult, error) {
logger.Infof(ctx, "[Tool][GrepChunks] Execute started")
// Parse args from json.RawMessage
var input GrepChunksInput
if err := json.Unmarshal(args, &input); err != nil {
logger.Errorf(ctx, "[Tool][GrepChunks] Failed to parse args: %v", err)
@@ -121,36 +107,60 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty
}, err
}
// Parse pattern parameter (required) - support multiple patterns
patterns := input.Patterns
// Accept both canonical (`queries`) and legacy (`patterns`) field names.
// When both are present we concatenate, preserving whichever came first,
// so a caller migrating between the two won't end up with nothing.
rawQueries := append([]string{}, input.Queries...)
rawQueries = append(rawQueries, input.Patterns...)
// Validate patterns
if len(patterns) == 0 {
logger.Errorf(ctx, "[Tool][GrepChunks] Missing or invalid patterns parameter")
return &types.ToolResult{
Success: false,
Error: "pattern parameter is required and must contain at least one non-empty pattern",
}, fmt.Errorf("missing pattern parameter")
}
// Use default values for all options
countOnly := false // default: show results
maxResults := 50
if input.MaxResults > 0 {
maxResults = input.MaxResults
if maxResults < 1 {
maxResults = 1
} else if maxResults > 200 {
maxResults = 200
queries := make([]string, 0, len(rawQueries))
for _, q := range rawQueries {
if strings.TrimSpace(q) != "" {
queries = append(queries, q)
}
}
// Get allowed KBs from searchTargets
if len(queries) == 0 {
logger.Errorf(ctx, "[Tool][GrepChunks] Missing or empty queries parameter")
return &types.ToolResult{
Success: false,
Error: "queries parameter is required and must contain at least one non-empty regex query",
}, fmt.Errorf("missing queries parameter")
}
if len(queries) > 5 {
queries = queries[:5]
}
// Compile queries with (?i) prefix for case-insensitive Go-side matching.
// Compilation also validates the regex syntax before we send it to the DB.
compiled := make([]*regexp.Regexp, 0, len(queries))
for _, q := range queries {
re, err := regexp.Compile("(?i)" + q)
if err != nil {
logger.Errorf(ctx, "[Tool][GrepChunks] Invalid regex %q: %v", q, err)
return &types.ToolResult{
Success: false,
Error: fmt.Sprintf("invalid regex query %q: %v", q, err),
}, err
}
compiled = append(compiled, re)
}
// Canonical `limit`, with `max_results` accepted as legacy alias.
limit := input.Limit
if limit <= 0 && input.MaxResults > 0 {
limit = input.MaxResults
}
if limit <= 0 {
limit = 30
}
if limit > 100 {
limit = 100
}
allowedKBIDs := t.searchTargets.GetAllKnowledgeBaseIDs()
kbTenantMap := t.searchTargets.GetKBTenantMap()
// Collect all specific knowledge IDs from searchTargets
var allowedKnowledgeIDs []string
for _, target := range t.searchTargets {
if target.Type == types.SearchTargetTypeKnowledge && len(target.KnowledgeIDs) > 0 {
@@ -158,12 +168,10 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty
}
}
// Parse knowledge_base_ids filter from input
kbIDs := input.KnowledgeBaseIDs
if len(kbIDs) == 0 {
kbIDs = allowedKBIDs
} else {
// Validate input KBs against allowed KBs
validKBs := make([]string, 0)
for _, kbID := range kbIDs {
if t.searchTargets.ContainsKB(kbID) {
@@ -173,11 +181,10 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty
kbIDs = validKBs
}
logger.Infof(ctx, "[Tool][GrepChunks] Patterns: %v, MaxResults: %d, KBs: %v, KnowledgeIDs: %v, KBTenantMap: %v",
patterns, maxResults, kbIDs, allowedKnowledgeIDs, kbTenantMap)
logger.Infof(ctx, "[Tool][GrepChunks] Queries: %v, Limit: %d, KBs: %v, KnowledgeIDs: %v",
queries, limit, kbIDs, allowedKnowledgeIDs)
// Build and execute query with tenant info
results, totalCount, err := t.searchChunks(ctx, patterns, kbIDs, allowedKnowledgeIDs, kbTenantMap)
results, err := t.searchChunks(ctx, queries, kbIDs, allowedKnowledgeIDs, kbTenantMap)
if err != nil {
logger.Errorf(ctx, "[Tool][GrepChunks] Search failed: %v", err)
return &types.ToolResult{
@@ -188,36 +195,28 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty
logger.Infof(ctx, "[Tool][GrepChunks] Found %d matching chunks", len(results))
// Apply deduplication to remove duplicate or near-duplicate chunks
deduplicatedResults := t.deduplicateChunks(ctx, results)
logger.Infof(ctx, "[Tool][GrepChunks] After deduplication: %d chunks (from %d)",
len(deduplicatedResults), len(results))
// Calculate match scores for sorting (based on match count and position)
scoredResults := t.scoreChunks(ctx, deduplicatedResults, patterns)
// Score chunks using compiled regex (counts + earliest-position boost).
scoredResults := t.scoreChunks(ctx, deduplicatedResults, compiled)
// Apply MMR to reduce redundancy if we have many results
finalResults := scoredResults
if len(scoredResults) > 10 {
// Use MMR when we have more than 10 results
mmrK := len(scoredResults)
if maxResults > 0 && mmrK > maxResults {
mmrK = maxResults
if limit > 0 && mmrK > limit {
mmrK = limit
}
logger.Debugf(
ctx,
"[Tool][GrepChunks] Applying MMR: k=%d, lambda=0.7, input=%d results",
mmrK,
len(scoredResults),
)
mmrResults := t.applyMMR(ctx, scoredResults, patterns, mmrK, 0.7)
logger.Debugf(ctx, "[Tool][GrepChunks] Applying MMR: k=%d, lambda=0.7, input=%d results",
mmrK, len(scoredResults))
mmrResults := t.applyMMR(ctx, scoredResults, mmrK, 0.7)
if len(mmrResults) > 0 {
finalResults = mmrResults
logger.Infof(ctx, "[Tool][GrepChunks] MMR completed: %d results selected", len(finalResults))
}
}
// Sort by match score (descending), then by chunk index
sort.Slice(finalResults, func(i, j int) bool {
if finalResults[i].MatchedPatterns != finalResults[j].MatchedPatterns {
return finalResults[i].MatchedPatterns > finalResults[j].MatchedPatterns
@@ -228,29 +227,30 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty
return finalResults[i].ChunkIndex < finalResults[j].ChunkIndex
})
aggregatedResults := t.aggregateByKnowledge(finalResults, patterns)
totalKnowledge := len(aggregatedResults)
if len(finalResults) > limit {
finalResults = finalResults[:limit]
}
// Aggregation by knowledge is still useful for the frontend summary view.
aggregatedResults := t.aggregateByKnowledge(finalResults, queries, compiled)
if len(aggregatedResults) > 20 {
aggregatedResults = aggregatedResults[:20]
}
logger.Infof(ctx, "[Tool][GrepChunks] Aggregated results: %d", len(aggregatedResults))
// Format output
output := t.formatOutput(ctx, aggregatedResults, totalCount, patterns, countOnly)
output := t.formatOutput(ctx, finalResults, queries, compiled)
return &types.ToolResult{
Success: true,
Output: output,
Data: map[string]interface{}{
"patterns": patterns,
"queries": queries,
"patterns": queries, // legacy alias; frontend currently reads `patterns`
"knowledge_results": aggregatedResults,
"result_count": len(aggregatedResults),
"total_matches": totalKnowledge,
"total_matches": len(finalResults),
"knowledge_base_ids": kbIDs,
"max_results": maxResults,
"limit": limit,
"max_results": limit, // legacy alias
"display_type": "grep_results",
},
}, nil
@@ -259,33 +259,42 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty
type chunkWithTitle struct {
types.Chunk
KnowledgeTitle string `json:"knowledge_title" gorm:"column:knowledge_title"`
MatchScore float64 `json:"match_score" gorm:"column:match_score"` // Score based on match count and position
MatchedPatterns int `json:"matched_patterns"` // Number of unique patterns matched
MatchScore float64 `json:"match_score" gorm:"column:match_score"`
MatchedPatterns int `json:"matched_patterns"`
TotalChunkCount int `json:"total_chunk_count" gorm:"column:total_chunk_count"`
}
// searchChunks performs the database search with pattern matching
// kbTenantMap provides KB-to-tenant mapping for cross-tenant queries
// regexOperatorForDialect returns the SQL operator used to apply a POSIX
// regular expression to a text column for the current dialect.
// PostgreSQL ~* is case-insensitive by default; MySQL/SQLite REGEXP relies on
// collation / driver extensions.
func (t *GrepChunksTool) regexOperatorForDialect() string {
switch t.db.Dialector.Name() {
case "postgres":
return "~*"
default:
// MySQL, SQLite (with the go-sqlite3 REGEXP extension), or anything else
// that understands the REGEXP keyword.
return "REGEXP"
}
}
// searchChunks performs the database search using regex queries.
func (t *GrepChunksTool) searchChunks(
ctx context.Context,
patterns []string,
queries []string,
kbIDs []string,
knowledgeIDs []string,
kbTenantMap map[string]uint64,
) ([]chunkWithTitle, int64, error) {
) ([]chunkWithTitle, error) {
if len(kbIDs) == 0 && len(knowledgeIDs) == 0 {
logger.Warnf(ctx, "[Tool][GrepChunks] No kbIDs or knowledgeIDs specified, returning empty results")
return nil, 0, nil
return nil, nil
}
// PostgreSQL uses ILIKE for case-insensitive matching;
// MySQL and SQLite LIKE is already case-insensitive under default collation.
likeOp := "LIKE"
if t.db.Dialector.Name() == "postgres" {
likeOp = "ILIKE"
}
regexOp := t.regexOperatorForDialect()
query := t.db.Debug().WithContext(ctx).Table("chunks").
query := t.db.WithContext(ctx).Table("chunks").
Select("chunks.id, chunks.content, chunks.chunk_index, chunks.knowledge_id, "+
"chunks.knowledge_base_id, chunks.chunk_type, chunks.created_at, "+
"knowledges.title as knowledge_title").
@@ -311,28 +320,27 @@ func (t *GrepChunksTool) searchChunks(
query = query.Where("("+strings.Join(conditions, " OR ")+")", args...)
} else {
logger.Warnf(ctx, "[Tool][GrepChunks] No valid KB-tenant pairs found")
return nil, 0, nil
return nil, nil
}
}
if len(patterns) == 1 {
query = query.Where("chunks.content "+likeOp+" ?", "%"+patterns[0]+"%")
} else {
var conditions []string
var args []interface{}
for _, pattern := range patterns {
conditions = append(conditions, "chunks.content "+likeOp+" ?")
args = append(args, "%"+pattern+"%")
}
query = query.Where("("+strings.Join(conditions, " OR ")+")", args...)
// For MySQL/SQLite REGEXP case-insensitivity we rely on the column's default
// collation (utf8mb4_general_ci etc.) OR the driver's REGEXP implementation,
// which mirrors what wiki_search already ships in this codebase.
var regexConditions []string
var regexArgs []interface{}
for _, q := range queries {
regexConditions = append(regexConditions, fmt.Sprintf("chunks.content %s ?", regexOp))
regexArgs = append(regexArgs, q)
}
query = query.Where("("+strings.Join(regexConditions, " OR ")+")", regexArgs...)
const maxFetchLimit = 500
var results []chunkWithTitle
if err := query.Order("chunks.created_at DESC").Limit(maxFetchLimit).Find(&results).Error; err != nil {
logger.Errorf(ctx, "[Tool][GrepChunks] Failed to fetch results: %v", err)
return nil, 0, err
return nil, err
}
if len(results) > 0 {
@@ -371,27 +379,25 @@ func (t *GrepChunksTool) searchChunks(
}
}
return results, int64(len(results)), nil
return results, nil
}
// formatOutput formats the search results as XML
// formatOutput emits per-chunk XML with <match_snippet> and <query_hit>
// elements, mirroring the wiki_search output shape. Chunks that were already
// surfaced by a previous call to this tool in the same session are rendered
// compactly with `already_seen="true"` so the LLM doesn't waste context
// re-reading the same snippet.
func (t *GrepChunksTool) formatOutput(
ctx context.Context,
results []knowledgeAggregation,
totalCount int64,
patterns []string,
countOnly bool,
results []chunkWithTitle,
queries []string,
compiled []*regexp.Regexp,
) string {
var b strings.Builder
if countOnly {
b.WriteString(fmt.Sprintf("<grep_results count=\"%d\" />\n", totalCount))
return b.String()
}
b.WriteString(fmt.Sprintf("<grep_results match_count=\"%d\">\n", len(results)))
for _, p := range patterns {
b.WriteString(fmt.Sprintf("<pattern>%s</pattern>\n", p))
b.WriteString(fmt.Sprintf("<grep_results chunk_count=\"%d\">\n", len(results)))
for _, q := range queries {
b.WriteString(fmt.Sprintf("<query>%s</query>\n", xmlEscape(q)))
}
if len(results) == 0 {
@@ -399,19 +405,57 @@ func (t *GrepChunksTool) formatOutput(
return b.String()
}
for _, result := range results {
b.WriteString(fmt.Sprintf("<match knowledge_id=\"%s\" title=\"%s\" chunk_hits=\"%d\" chunk_total=\"%d\">\n",
result.KnowledgeID, result.KnowledgeTitle, result.ChunkHitCount, result.TotalChunkCount))
for _, pattern := range patterns {
count := result.PatternCounts[pattern]
if count > 0 {
b.WriteString(fmt.Sprintf("<pattern_hit pattern=\"%s\" count=\"%d\" />\n", pattern, count))
for _, r := range results {
counts := countRegexHits(r.Content, compiled, queries)
snippet := extractSnippetRegex(r.Content, compiled)
t.mu.Lock()
seen := t.seenChunks[r.ID]
t.seenChunks[r.ID] = true
t.mu.Unlock()
if seen {
b.WriteString(fmt.Sprintf(
"<chunk chunk_id=\"%s\" knowledge_id=\"%s\" knowledge_title=\"%s\" chunk_index=\"%d\" score=\"%.3f\" already_seen=\"true\">\n",
xmlEscape(r.ID),
xmlEscape(r.KnowledgeID),
xmlEscape(r.KnowledgeTitle),
r.ChunkIndex,
r.MatchScore,
))
for _, q := range queries {
if c := counts[q]; c > 0 {
b.WriteString(fmt.Sprintf("<query_hit query=\"%s\" count=\"%d\" />\n",
xmlEscape(q), c))
}
}
b.WriteString("<note>(snippet omitted, already returned in a previous grep_chunks call this session)</note>\n")
b.WriteString("</chunk>\n")
continue
}
b.WriteString(fmt.Sprintf(
"<chunk chunk_id=\"%s\" knowledge_id=\"%s\" knowledge_title=\"%s\" chunk_index=\"%d\" score=\"%.3f\">\n",
xmlEscape(r.ID),
xmlEscape(r.KnowledgeID),
xmlEscape(r.KnowledgeTitle),
r.ChunkIndex,
r.MatchScore,
))
for _, q := range queries {
if c := counts[q]; c > 0 {
b.WriteString(fmt.Sprintf("<query_hit query=\"%s\" count=\"%d\" />\n",
xmlEscape(q), c))
}
}
b.WriteString("</match>\n")
if snippet != "" {
b.WriteString(fmt.Sprintf("<match_snippet>%s</match_snippet>\n", xmlEscape(snippet)))
}
b.WriteString("</chunk>\n")
}
b.WriteString("</grep_results>")
_ = ctx
return b.String()
}
@@ -426,17 +470,21 @@ type knowledgeAggregation struct {
DistinctPatterns int `json:"distinct_patterns"`
}
func (t *GrepChunksTool) aggregateByKnowledge(results []chunkWithTitle, patterns []string) []knowledgeAggregation {
func (t *GrepChunksTool) aggregateByKnowledge(
results []chunkWithTitle,
queries []string,
compiled []*regexp.Regexp,
) []knowledgeAggregation {
if len(results) == 0 {
return nil
}
patternKeys := make([]string, 0, len(patterns))
for _, p := range patterns {
if strings.TrimSpace(p) == "" {
queryKeys := make([]string, 0, len(queries))
for _, q := range queries {
if strings.TrimSpace(q) == "" {
continue
}
patternKeys = append(patternKeys, p)
queryKeys = append(queryKeys, q)
}
aggregated := make(map[string]*knowledgeAggregation)
@@ -456,23 +504,23 @@ func (t *GrepChunksTool) aggregateByKnowledge(results []chunkWithTitle, patterns
KnowledgeBaseID: chunk.KnowledgeBaseID,
KnowledgeTitle: title,
TotalChunkCount: chunk.TotalChunkCount,
PatternCounts: make(map[string]int, len(patternKeys)),
PatternCounts: make(map[string]int, len(queryKeys)),
}
for _, pKey := range patternKeys {
aggregated[knowledgeID].PatternCounts[pKey] = 0
for _, qKey := range queryKeys {
aggregated[knowledgeID].PatternCounts[qKey] = 0
}
}
entry := aggregated[knowledgeID]
entry.ChunkHitCount++
patternOccurrences := t.countPatternOccurrences(chunk.Content, patternKeys)
for _, p := range patternKeys {
count := patternOccurrences[p]
occurrences := countRegexHits(chunk.Content, compiled, queryKeys)
for _, q := range queryKeys {
count := occurrences[q]
if count == 0 {
continue
}
entry.PatternCounts[p] += count
entry.PatternCounts[q] += count
entry.TotalPatternHits += count
}
}
@@ -504,38 +552,89 @@ func (t *GrepChunksTool) aggregateByKnowledge(results []chunkWithTitle, patterns
return resultSlice
}
func (t *GrepChunksTool) countPatternOccurrences(content string, patterns []string) map[string]int {
// countRegexHits returns the total number of matches per (compiled) pattern
// within content, keyed by the original (uncompiled) pattern string.
func countRegexHits(content string, compiled []*regexp.Regexp, patterns []string) map[string]int {
counts := make(map[string]int, len(patterns))
if content == "" || len(patterns) == 0 {
if content == "" || len(compiled) == 0 {
return counts
}
contentLower := strings.ToLower(content)
for _, pattern := range patterns {
p := strings.ToLower(pattern)
if strings.TrimSpace(p) == "" {
for i, re := range compiled {
if re == nil {
continue
}
counts[pattern] = countOccurrences(contentLower, p)
matches := re.FindAllStringIndex(content, -1)
counts[patterns[i]] = len(matches)
}
return counts
}
func countOccurrences(text string, pattern string) int {
if pattern == "" {
return 0
// extractSnippetRegex returns a short context snippet around the earliest
// regex match across any of the provided compiled patterns. Result is
// compressed to a single line and bounded in length on both sides of the
// match to keep the XML output concise.
func extractSnippetRegex(content string, compiled []*regexp.Regexp) string {
if content == "" || len(compiled) == 0 {
return ""
}
count := 0
index := 0
for index < len(text) {
pos := strings.Index(text[index:], pattern)
if pos == -1 {
break
earliest := -1
earliestEnd := -1
for _, re := range compiled {
if re == nil {
continue
}
loc := re.FindStringIndex(content)
if loc == nil {
continue
}
if earliest < 0 || loc[0] < earliest {
earliest = loc[0]
earliestEnd = loc[1]
}
count++
index += pos + len(pattern)
}
return count
if earliest < 0 {
return ""
}
matchStr := content[earliest:earliestEnd]
before := content[:earliest]
after := content[earliestEnd:]
const contextRunes = 60
beforeRunes := []rune(before)
if len(beforeRunes) > contextRunes {
beforeRunes = beforeRunes[len(beforeRunes)-contextRunes:]
}
afterRunes := []rune(after)
if len(afterRunes) > contextRunes {
afterRunes = afterRunes[:contextRunes]
}
matchRunes := []rune(matchStr)
if len(matchRunes) > 120 {
matchRunes = append(matchRunes[:120], []rune("...")...)
}
snippet := string(beforeRunes) + string(matchRunes) + string(afterRunes)
snippet = strings.ReplaceAll(snippet, "\n", " ")
for strings.Contains(snippet, " ") {
snippet = strings.ReplaceAll(snippet, " ", " ")
}
return "... " + strings.TrimSpace(snippet) + " ..."
}
// xmlEscape replaces characters that would break simple XML attribute /
// element values. It is intentionally minimal because the rendered output is
// consumed by the LLM (forgiving parser) rather than a strict XML processor.
func xmlEscape(s string) string {
replacer := strings.NewReplacer(
"&", "&amp;",
"<", "&lt;",
">", "&gt;",
"\"", "&quot;",
"'", "&apos;",
)
return replacer.Replace(s)
}
// deduplicateChunks removes duplicate or near-duplicate chunks using content signature
@@ -545,7 +644,6 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW
uniqueResults := make([]chunkWithTitle, 0)
for _, r := range results {
// Build multiple keys for deduplication
keys := []string{r.ID}
if r.ParentChunkID != "" {
keys = append(keys, "parent:"+r.ParentChunkID)
@@ -554,7 +652,6 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW
keys = append(keys, fmt.Sprintf("kb:%s#%d", r.KnowledgeID, r.ChunkIndex))
}
// Check if any key is already seen
dup := false
for _, k := range keys {
if seen[k] {
@@ -566,7 +663,6 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW
continue
}
// Check content signature for near-duplicate content
sig := t.buildContentSignature(r.Content)
if sig != "" {
if contentSig[sig] {
@@ -575,15 +671,12 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW
contentSig[sig] = true
}
// Mark all keys as seen
for _, k := range keys {
seen[k] = true
}
uniqueResults = append(uniqueResults, r)
}
// If we have duplicates by ID, keep the first one
seenByID := make(map[string]bool)
deduplicated := make([]chunkWithTitle, 0)
for _, r := range uniqueResults {
@@ -592,7 +685,7 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW
deduplicated = append(deduplicated, r)
}
}
_ = ctx
return deduplicated
}
@@ -601,53 +694,55 @@ func (t *GrepChunksTool) buildContentSignature(content string) string {
return searchutil.BuildContentSignature(content)
}
// scoreChunks calculates match scores for chunks based on pattern matches
// scoreChunks calculates match scores for chunks based on regex matches.
func (t *GrepChunksTool) scoreChunks(
ctx context.Context,
results []chunkWithTitle,
patterns []string,
compiled []*regexp.Regexp,
) []chunkWithTitle {
scored := make([]chunkWithTitle, len(results))
for i := range results {
scored[i] = results[i]
score, patternCount := t.calculateMatchScore(results[i].Content, patterns)
score, patternCount := t.calculateMatchScore(results[i].Content, compiled)
scored[i].MatchScore = score
scored[i].MatchedPatterns = patternCount
}
_ = ctx
return scored
}
// calculateMatchScore calculates a score based on how many patterns match and their positions
func (t *GrepChunksTool) calculateMatchScore(content string, patterns []string) (float64, int) {
if content == "" || len(patterns) == 0 {
// calculateMatchScore counts how many regex patterns match the content and
// applies a small boost for earlier match positions.
func (t *GrepChunksTool) calculateMatchScore(content string, compiled []*regexp.Regexp) (float64, int) {
if content == "" || len(compiled) == 0 {
return 0.0, 0
}
contentLower := strings.ToLower(content)
matchCount := 0
earliestPos := len(content)
// Count how many patterns match and find earliest position
for _, pattern := range patterns {
patternLower := strings.ToLower(pattern)
if strings.Contains(contentLower, patternLower) {
matchCount++
// Find position of first match
pos := strings.Index(contentLower, patternLower)
if pos >= 0 && pos < earliestPos {
earliestPos = pos
}
for _, re := range compiled {
if re == nil {
continue
}
loc := re.FindStringIndex(content)
if loc == nil {
continue
}
matchCount++
if loc[0] < earliestPos {
earliestPos = loc[0]
}
}
// Score: higher for more matches, slightly higher for earlier positions
// Base score: match ratio (0.0 to 1.0)
baseScore := float64(matchCount) / float64(len(patterns))
if matchCount == 0 {
return 0.0, 0
}
baseScore := float64(matchCount) / float64(len(compiled))
// Position bonus: earlier matches get slight boost (max 0.1)
positionBonus := 0.0
if earliestPos < len(content) {
// Normalize position to [0, 1] and apply small bonus
positionRatio := 1.0 - float64(earliestPos)/float64(len(content))
positionBonus = positionRatio * 0.1
}
@@ -659,7 +754,6 @@ func (t *GrepChunksTool) calculateMatchScore(content string, patterns []string)
func (t *GrepChunksTool) applyMMR(
ctx context.Context,
results []chunkWithTitle,
patterns []string,
k int,
lambda float64,
) []chunkWithTitle {
@@ -671,18 +765,16 @@ func (t *GrepChunksTool) applyMMR(
lambda, k, len(results))
selected := make([]chunkWithTitle, 0, k)
selectedTokenSets := make([]map[string]struct{}, 0, k) // cache of token sets
selectedTokenSets := make([]map[string]struct{}, 0, k)
candidates := make([]chunkWithTitle, len(results))
copy(candidates, results)
// Pre-compute token sets for all candidates
tokenSets := make([]map[string]struct{}, len(candidates))
for i, r := range candidates {
tokenSets[i] = t.tokenizeSimple(r.Content)
}
// MMR selection loop
for len(selected) < k && len(candidates) > 0 {
bestIdx := 0
bestScore := -1.0
@@ -690,13 +782,9 @@ func (t *GrepChunksTool) applyMMR(
for i, r := range candidates {
relevance := r.MatchScore
redundancy := 0.0
// Calculate maximum redundancy with already selected results
for _, selectedTS := range selectedTokenSets {
redundancy = math.Max(redundancy, t.jaccard(tokenSets[i], selectedTS))
}
// MMR score: balance relevance and diversity
mmr := lambda*relevance - (1.0-lambda)*redundancy
if mmr > bestScore {
bestScore = mmr
@@ -704,11 +792,9 @@ func (t *GrepChunksTool) applyMMR(
}
}
// Add best candidate to selected and remove from candidates
selected = append(selected, candidates[bestIdx])
selectedTokenSets = append(selectedTokenSets, tokenSets[bestIdx])
// Remove corresponding token set. Use swap deletion
last := len(candidates) - 1
candidates[bestIdx] = candidates[last]
tokenSets[bestIdx] = tokenSets[last]
@@ -716,24 +802,6 @@ func (t *GrepChunksTool) applyMMR(
tokenSets = tokenSets[:last]
}
// Compute average redundancy among selected results
avgRed := 0.0
if len(selected) > 1 {
pairs := 0
for i := 0; i < len(selected); i++ {
for j := i + 1; j < len(selected); j++ {
avgRed += t.jaccard(selectedTokenSets[i], selectedTokenSets[j]) // read token from cache
pairs++
}
}
if pairs > 0 {
avgRed /= float64(pairs)
}
}
logger.Debugf(ctx, "[Tool][GrepChunks] MMR completed: selected=%d, avg_redundancy=%.4f",
len(selected), avgRed)
return selected
}
+63
View File
@@ -11,6 +11,8 @@ import (
// - Trailing commas before closing brackets
// - Single quotes instead of double quotes
// - Unquoted keys
// - Invalid backslash escapes inside strings (e.g. regex patterns like
// "C\+\+", "\d+", "\.log$") where the LLM forgot to double-escape.
//
// Returns the repaired JSON string. If repair is not possible,
// returns the original string unchanged (caller should handle parse errors).
@@ -30,6 +32,10 @@ func RepairJSON(s string) string {
}
}
// Fix invalid backslash escapes before anything else, because unbalanced
// strings confuse the comma/bracket trackers below.
s = fixInvalidEscapes(s)
// Fix trailing commas: ,} or ,]
s = fixTrailingCommas(s)
@@ -39,6 +45,63 @@ func RepairJSON(s string) string {
return s
}
// fixInvalidEscapes turns invalid JSON string escapes into literal backslash
// sequences. JSON only allows the escape letters \" \\ \/ \b \f \n \r \t \u;
// any other backslash inside a string (e.g. \+ \d \w \. \| \( \)) is a parse
// error. LLMs frequently emit these when they meant to pass a regex pattern
// without double-escaping — rewriting "\+" to "\\+" recovers the LLM's intent
// (the unmarshalled Go string becomes "\+", which is what the regex engine
// expects) and is idempotent on already-valid JSON.
func fixInvalidEscapes(s string) string {
var out strings.Builder
out.Grow(len(s) + 8)
runes := []rune(s)
inString := false
for i := 0; i < len(runes); i++ {
r := runes[i]
if !inString {
out.WriteRune(r)
if r == '"' {
inString = true
}
continue
}
if r == '"' {
out.WriteRune(r)
inString = false
continue
}
if r != '\\' {
out.WriteRune(r)
continue
}
// Found a backslash inside a string. Peek at the next rune to decide
// whether this is a valid JSON escape.
if i+1 >= len(runes) {
// Dangling backslash at EOF — escape it so we don't leave the
// string in a "pending escape" state for downstream parsers.
out.WriteString(`\\`)
continue
}
next := runes[i+1]
switch next {
case '"', '\\', '/', 'b', 'f', 'n', 'r', 't', 'u':
// Valid JSON escape — pass through unchanged.
out.WriteRune(r)
out.WriteRune(next)
i++
default:
// Invalid escape — most likely a regex metachar that the LLM
// forgot to double-escape. Emit a literal backslash + the next
// rune, so the final Go string preserves the \X sequence.
out.WriteString(`\\`)
out.WriteRune(next)
i++
}
}
return out.String()
}
// fixTrailingCommas removes trailing commas before closing brackets/braces.
func fixTrailingCommas(s string) string {
// Simple state machine to handle strings
+49
View File
@@ -5,6 +5,7 @@ import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestRepairJSON(t *testing.T) {
@@ -74,4 +75,52 @@ func TestRepairJSON(t *testing.T) {
result := RepairJSON(input)
assert.True(t, json.Valid([]byte(result)), "result: %s", result)
})
t.Run("invalid escape: regex plus (C\\+\\+)", func(t *testing.T) {
// The LLM emits a raw regex without double-escaping — \+ is not a
// legal JSON escape. Repair should rewrite it to \\+ so the
// unmarshalled string is the regex pattern `C\+\+`, matching
// literal `C++`.
input := `{"queries": ["C\+\+"]}`
result := RepairJSON(input)
assert.True(t, json.Valid([]byte(result)), "result: %s", result)
var parsed struct {
Queries []string `json:"queries"`
}
require.NoError(t, json.Unmarshal([]byte(result), &parsed))
assert.Equal(t, []string{`C\+\+`}, parsed.Queries)
})
t.Run("invalid escape: common regex shorthand", func(t *testing.T) {
input := `{"queries": ["\d+\.\d+", "\bword\b", "foo\|bar"]}`
result := RepairJSON(input)
assert.True(t, json.Valid([]byte(result)), "result: %s", result)
var parsed struct {
Queries []string `json:"queries"`
}
require.NoError(t, json.Unmarshal([]byte(result), &parsed))
// \b is a VALID JSON escape (backspace) — the LLM gets what it asked
// for (a backspace char), we don't silently rewrite valid escapes.
// \d, \., \| are invalid → repaired to literal backslash sequences.
assert.Equal(t, `\d+\.\d+`, parsed.Queries[0])
assert.Equal(t, "\bword\b", parsed.Queries[1])
assert.Equal(t, `foo\|bar`, parsed.Queries[2])
})
t.Run("invalid escape repair is idempotent", func(t *testing.T) {
input := `{"queries": ["C\+\+"]}`
once := RepairJSON(input)
twice := RepairJSON(once)
assert.Equal(t, once, twice)
})
t.Run("valid escapes are preserved", func(t *testing.T) {
// Valid JSON escapes must pass through untouched.
input := `{"path": "C:\\Users\\x", "line": "a\nb", "quote": "say \"hi\""}`
result := RepairJSON(input)
assert.Equal(t, input, result)
assert.True(t, json.Valid([]byte(result)))
})
}
+167 -35
View File
@@ -117,7 +117,10 @@ type searchResultWithMeta struct {
KnowledgeBaseType string // Type of the knowledge base (document, faq, etc.)
}
// KnowledgeSearchTool searches knowledge bases with flexible query modes
// KnowledgeSearchTool searches knowledge bases with flexible query modes.
// seenChunks lets repeated calls in the same session surface previously-
// returned chunks in a compact form (mirroring wiki_search's de-duping UX)
// so the LLM doesn't burn tokens re-reading identical content.
type KnowledgeSearchTool struct {
BaseTool
knowledgeBaseService interfaces.KnowledgeBaseService
@@ -127,6 +130,9 @@ type KnowledgeSearchTool struct {
rerankModel rerank.Reranker
chatModel chat.Chat // Optional chat model for LLM-based reranking
config *config.Config // Global config for fallback values
seenMu sync.Mutex
seenChunks map[string]bool
}
// NewKnowledgeSearchTool creates a new knowledge search tool
@@ -148,6 +154,7 @@ func NewKnowledgeSearchTool(
rerankModel: rerankModel,
chatModel: chatModel,
config: cfg,
seenChunks: make(map[string]bool),
}
}
@@ -1086,9 +1093,15 @@ func (t *KnowledgeSearchTool) formatOutput(
kbCounts[r.KnowledgeID]++
}
// Format individual results as XML
// Format individual results as XML. Tag names are kept in sync with
// wiki_search (`<search_results>`, per-entry element, `<query>`) so that
// agents and downstream consumers see a single consistent shape across
// all retrieval tools.
var ob strings.Builder
ob.WriteString(fmt.Sprintf("<search_results count=\"%d\">\n", len(results)))
for _, q := range queries {
ob.WriteString(fmt.Sprintf("<query>%s</query>\n", xmlEscape(q)))
}
formattedResults := make([]map[string]interface{}, 0, len(results))
@@ -1136,46 +1149,82 @@ func (t *KnowledgeSearchTool) formatOutput(
}
}
ob.WriteString(fmt.Sprintf("<result id=\"%d\" chunk_id=\"%s\" chunk_index=\"%d\" knowledge_id=\"%s\" source=\"%s\">\n",
i+1, result.ID, result.ChunkIndex, result.KnowledgeID, result.KnowledgeTitle))
ob.WriteString(fmt.Sprintf("<content>%s</content>\n", result.Content))
t.seenMu.Lock()
seen := t.seenChunks[result.ID]
t.seenChunks[result.ID] = true
t.seenMu.Unlock()
if result.ImageInfo != "" {
var imageInfos []types.ImageInfo
if err := json.Unmarshal([]byte(result.ImageInfo), &imageInfos); err == nil && len(imageInfos) > 0 {
for _, img := range imageInfos {
ob.WriteString(fmt.Sprintf("<image url=\"%s\">\n", img.URL))
if img.Caption != "" {
ob.WriteString(fmt.Sprintf("<image_caption>%s</image_caption>\n", img.Caption))
if seen {
// Compact rendering for chunks we already returned in a previous
// knowledge_search call during this session. The model has the
// content in context already, so re-emitting it only burns tokens.
ob.WriteString(fmt.Sprintf(
"<chunk rank=\"%d\" chunk_id=\"%s\" chunk_index=\"%d\" knowledge_id=\"%s\" knowledge_base_id=\"%s\" knowledge_title=\"%s\" score=\"%.3f\" source_query=\"%s\" already_seen=\"true\">\n",
i+1,
xmlEscape(result.ID),
result.ChunkIndex,
xmlEscape(result.KnowledgeID),
xmlEscape(result.KnowledgeBaseID),
xmlEscape(result.KnowledgeTitle),
result.Score,
xmlEscape(result.SourceQuery),
))
ob.WriteString("<note>(content omitted, already returned in a previous knowledge_search call this session)</note>\n")
ob.WriteString("</chunk>\n")
} else {
ob.WriteString(fmt.Sprintf(
"<chunk rank=\"%d\" chunk_id=\"%s\" chunk_index=\"%d\" knowledge_id=\"%s\" knowledge_base_id=\"%s\" knowledge_title=\"%s\" score=\"%.3f\" source_query=\"%s\">\n",
i+1,
xmlEscape(result.ID),
result.ChunkIndex,
xmlEscape(result.KnowledgeID),
xmlEscape(result.KnowledgeBaseID),
xmlEscape(result.KnowledgeTitle),
result.Score,
xmlEscape(result.SourceQuery),
))
if snippet := extractSnippetForQueries(result.Content, queries); snippet != "" {
ob.WriteString(fmt.Sprintf("<match_snippet>%s</match_snippet>\n", xmlEscape(snippet)))
}
ob.WriteString(fmt.Sprintf("<content>%s</content>\n", result.Content))
if result.ImageInfo != "" {
var imageInfos []types.ImageInfo
if err := json.Unmarshal([]byte(result.ImageInfo), &imageInfos); err == nil && len(imageInfos) > 0 {
for _, img := range imageInfos {
ob.WriteString(fmt.Sprintf("<image url=\"%s\">\n", xmlEscape(img.URL)))
if img.Caption != "" {
ob.WriteString(fmt.Sprintf("<image_caption>%s</image_caption>\n", xmlEscape(img.Caption)))
}
if img.OCRText != "" {
ob.WriteString(fmt.Sprintf("<image_ocr>%s</image_ocr>\n", xmlEscape(img.OCRText)))
}
ob.WriteString("</image>\n")
}
if img.OCRText != "" {
ob.WriteString(fmt.Sprintf("<image_ocr>%s</image_ocr>\n", img.OCRText))
}
}
if faqMeta != nil {
ob.WriteString("<faq>\n")
if faqMeta.StandardQuestion != "" {
ob.WriteString(fmt.Sprintf("<question>%s</question>\n", xmlEscape(faqMeta.StandardQuestion)))
}
if len(faqMeta.SimilarQuestions) > 0 {
for _, sq := range faqMeta.SimilarQuestions {
ob.WriteString(fmt.Sprintf("<similar_question>%s</similar_question>\n", xmlEscape(sq)))
}
ob.WriteString("</image>\n")
}
if len(faqMeta.Answers) > 0 {
for _, ans := range faqMeta.Answers {
ob.WriteString(fmt.Sprintf("<answer>%s</answer>\n", xmlEscape(ans)))
}
}
ob.WriteString("</faq>\n")
}
}
if faqMeta != nil {
ob.WriteString("<faq>\n")
if faqMeta.StandardQuestion != "" {
ob.WriteString(fmt.Sprintf("<question>%s</question>\n", faqMeta.StandardQuestion))
}
if len(faqMeta.SimilarQuestions) > 0 {
for _, sq := range faqMeta.SimilarQuestions {
ob.WriteString(fmt.Sprintf("<similar_question>%s</similar_question>\n", sq))
}
}
if len(faqMeta.Answers) > 0 {
for _, ans := range faqMeta.Answers {
ob.WriteString(fmt.Sprintf("<answer>%s</answer>\n", ans))
}
}
ob.WriteString("</faq>\n")
ob.WriteString("</chunk>\n")
}
ob.WriteString("</result>\n")
formattedResults = append(formattedResults, map[string]interface{}{
"result_index": i + 1,
"chunk_id": result.ID,
@@ -1238,7 +1287,7 @@ func (t *KnowledgeSearchTool) formatOutput(
remaining := totalChunks - int64(retrievedCount)
percentage := float64(retrievedCount) / float64(totalChunks) * 100
ob.WriteString(fmt.Sprintf("<document_stat knowledge_id=\"%s\" title=\"%s\" total_chunks=\"%d\" retrieved=\"%d\" remaining=\"%d\" coverage=\"%.1f%%\" />\n",
knowledgeID, title, totalChunks, retrievedCount, remaining, percentage))
xmlEscape(knowledgeID), xmlEscape(title), totalChunks, retrievedCount, remaining, percentage))
}
}
ob.WriteString("</retrieval_statistics>\n")
@@ -1438,6 +1487,89 @@ func (t *KnowledgeSearchTool) tokenizeSimple(text string) map[string]struct{} {
return searchutil.TokenizeSimple(text)
}
// extractSnippetForQueries tries to produce a short contextual snippet around
// the first occurrence of any token extracted from the provided queries.
// When no token matches (common for fully paraphrased semantic queries) it
// falls back to the leading 160 runes of content so callers always get
// something to scan. The snippet is single-lined and bounded in length to
// keep the rendered XML compact.
func extractSnippetForQueries(content string, queries []string) string {
content = strings.TrimSpace(content)
if content == "" {
return ""
}
tokens := make([]string, 0, 8)
seen := make(map[string]struct{})
for _, q := range queries {
for _, tok := range strings.FieldsFunc(q, func(r rune) bool {
// Split on whitespace and common punctuation; keep CJK as whole tokens.
switch r {
case ' ', '\t', '\n', '\r', ',', '.', ';', ':', '?', '!',
'(', ')', '[', ']', '{', '}', '"', '\'':
return true
}
return false
}) {
tok = strings.ToLower(strings.TrimSpace(tok))
// Skip trivially-short stopwords. Three rune floor covers most
// English function words without chopping CJK bigrams.
if len([]rune(tok)) < 2 {
continue
}
if _, ok := seen[tok]; ok {
continue
}
seen[tok] = struct{}{}
tokens = append(tokens, tok)
}
}
lowered := strings.ToLower(content)
earliest := -1
earliestEnd := -1
for _, tok := range tokens {
idx := strings.Index(lowered, tok)
if idx < 0 {
continue
}
end := idx + len(tok)
if earliest < 0 || idx < earliest {
earliest = idx
earliestEnd = end
}
}
const contextRunes = 60
if earliest < 0 {
runes := []rune(content)
if len(runes) > contextRunes*2 {
return strings.TrimSpace(string(runes[:contextRunes*2])) + " ..."
}
return content
}
matchStr := content[earliest:earliestEnd]
before := content[:earliest]
after := content[earliestEnd:]
beforeRunes := []rune(before)
if len(beforeRunes) > contextRunes {
beforeRunes = beforeRunes[len(beforeRunes)-contextRunes:]
}
afterRunes := []rune(after)
if len(afterRunes) > contextRunes {
afterRunes = afterRunes[:contextRunes]
}
snippet := string(beforeRunes) + matchStr + string(afterRunes)
snippet = strings.ReplaceAll(snippet, "\n", " ")
for strings.Contains(snippet, " ") {
snippet = strings.ReplaceAll(snippet, " ", " ")
}
return "... " + strings.TrimSpace(snippet) + " ..."
}
// jaccard calculates Jaccard similarity between two token sets
func (t *KnowledgeSearchTool) jaccard(a, b map[string]struct{}) float64 {
return searchutil.Jaccard(a, b)
+1 -1
View File
@@ -35,7 +35,7 @@ If neither query nor range is provided, it returns the beginning of the document
},
"query": {
"type": "string",
"description": "Optional: A regex query to filter the document chunks. Use this to find specific quotes or details efficiently."
"description": "Optional: A regex query to filter the document chunks. Use this to find specific quotes or details efficiently. Remember to double-escape backslashes for JSON: write \"C\\\\+\\\\+\" (NOT \"C\\+\\+\") and \"\\\\d+\" (NOT \"\\d+\")."
},
"start_chunk_index": {
"type": "integer",
+1
View File
@@ -439,6 +439,7 @@ Examples:
- Multiple terms (RECOMMENDED): "psionic.*engine" (matches both words in order)
- Prefix matching: "^entity/.*" (finds all entities)
- Plain text: "engine" (matches anywhere in title/content/slug/summary)
IMPORTANT — JSON escaping: every backslash in a regex MUST be written as \\ inside the JSON tool arguments (e.g. to search for literal "C++" write "C\\+\\+", NOT "C\+\+"; for "\d+" write "\\d+"). Plain "\+" / "\d" etc. are invalid JSON escapes and will fail to parse.
Use this to find relevant wiki pages when you don't know the exact slug.`,
json.RawMessage(`{
"type": "object",
+29 -6
View File
@@ -709,18 +709,41 @@ func (s *agentService) getKnowledgeBaseInfos(ctx context.Context, kbIDs []string
kbType = "document" // Default type
}
kbInfos = append(kbInfos, &agent.KnowledgeBaseInfo{
ID: kb.ID,
Name: kb.Name,
Type: kbType,
Description: kb.Description,
DocCount: docCount,
RecentDocs: recentDocs,
ID: kb.ID,
Name: kb.Name,
Type: kbType,
Description: kb.Description,
DocCount: docCount,
Capabilities: kbRetrievalCapabilities(kb),
RecentDocs: recentDocs,
})
}
return kbInfos, nil
}
// kbRetrievalCapabilities reports which retrieval surfaces a KB exposes.
// Surfaces are the static facts the hybrid agent prompt consults to pick its
// retrieval strategy — the agent should NOT need to probe this via search.
//
// Returned values are a subset of {"wiki", "chunks"}:
// - "wiki" → the KB has wiki ingestion enabled (wiki_search / wiki_read_page)
// - "chunks" → the KB has vector and/or keyword (BM25) indexing enabled
// (knowledge_search / grep_chunks)
func kbRetrievalCapabilities(kb *types.KnowledgeBase) []string {
if kb == nil {
return nil
}
caps := make([]string, 0, 2)
if kb.IsWikiEnabled() {
caps = append(caps, "wiki")
}
if kb.IsVectorEnabled() || kb.IsKeywordEnabled() {
caps = append(caps, "chunks")
}
return caps
}
// getSelectedDocumentInfos retrieves detailed information for user-selected documents (via @ mention)
// This loads the actual content of the documents to include in the system prompt
func (s *agentService) getSelectedDocumentInfos(ctx context.Context, knowledgeIDs []string) ([]*agent.SelectedDocumentInfo, error) {