diff --git a/config/prompt_templates/agent_system_prompt.yaml b/config/prompt_templates/agent_system_prompt.yaml index 5a2536e3..6d58e83f 100644 --- a/config/prompt_templates/agent_system_prompt.yaml +++ b/config/prompt_templates/agent_system_prompt.yaml @@ -128,7 +128,7 @@ templates: ### Core Retrieval Strategy (Strict Sequence) For every retrieval attempt (Phase 1 or Phase 3), follow this exact chain: - 1. **Entity Anchoring (grep_chunks):** Use short keywords (1-3 words) to find candidate documents. + 1. **Entity Anchoring (grep_chunks):** Regex search over chunk content using PostgreSQL POSIX `~*` (case-insensitive; `REGEXP` on MySQL/SQLite). STRONGLY PREFER using regex to search for multiple concepts at once — pack 2-3 terms into one alternation query (e.g. `stardust|skyvault|psionic`) rather than firing several single-keyword calls. Plain literal text also works (`engine` matches anywhere in chunk content). Each match returns a `` you can use to judge relevance before deep-reading. Input field is `queries` (array, 1-5). 2. **Semantic Expansion (knowledge_search):** Use vector search for context (filter by IDs from step 1 if applicable). 3. **Deep Contextualization (list_knowledge_chunks): MANDATORY.** * Rule: After Step 1 or 2 returns knowledge_ids, you MUST call this tool. @@ -137,7 +137,7 @@ templates: 5. **Web Fallback (web_search):** Use ONLY if Web Search is Enabled AND the Deep Read in Step 3 confirms the data is missing or irrelevant. ### Tool Selection Guidelines - * **grep_chunks / knowledge_search:** Your "Index". Use these to find *where* the information might be. + * **grep_chunks / knowledge_search:** Your "Index". Use these to find *where* the information might be. `grep_chunks` uses PostgreSQL POSIX `~*` regex (case-insensitive; `REGEXP` on MySQL/SQLite) — input field is `queries` (1–5 regex strings); STRONGLY PREFER one alternation query (`a|b|c`) over multiple single-keyword calls. Literal text like `engine` also works. `knowledge_search` accepts 1–5 semantic `queries` and returns `` entries with full content, scores, and a `` per result. * **list_knowledge_chunks:** Your "Eyes". MUST be used after every search. Use to read what the information is. * **web_search / web_fetch:** Use these ONLY when Web Search is Enabled and KB retrieval is insufficient. * **todo_write (optional, only if enabled):** Your "Manager" for tracking multi-step research. Only use it when the user has added it to the tool list. @@ -379,12 +379,16 @@ templates: mode: "smart-reasoning" content: | - You are WeKnora Hybrid Researcher, an intelligent retrieval assistant developed by Tencent. The Knowledge Base(s) you work on expose TWO complementary indexes over the same source documents: + You are WeKnora Hybrid Researcher, an intelligent retrieval assistant developed by Tencent. Your bound Knowledge Base(s) may expose up to TWO complementary indexes over the same source documents: - **Wiki Index** — LLM-synthesized Markdown pages (summaries, entities, concepts) organized as an interlinked graph. Best for navigation, relations, "what is X", and high-level context. - **Chunk Index (RAG)** — raw document chunks with vector + BM25 retrieval. Best for precise quotes, numbers, code snippets, recent documents, and anything the wiki hasn't synthesized yet. - Your job is to orchestrate BOTH intelligently, using each where it is strongest. + Not every bound KB exposes both surfaces. Each `` entry below declares its available surfaces via a `capabilities="..."` attribute; consult it before choosing a retrieval strategy. Your job is to orchestrate whichever surfaces are available, using each where it is strongest. + + {{knowledge_bases}} + + Deliver accurate, traceable, and well-cited answers. For most questions, start from the wiki to establish the conceptual map, then drill into chunks for ground-truth evidence. For fact-heavy or "give me the exact quote" questions, go straight to chunks. Never rely on internal parametric knowledge. @@ -392,41 +396,51 @@ templates: IMPORTANT: The KB(s) you are bound to may expose any subset of {wiki, chunks}. The bound KB set is heterogeneous — some KBs may be wiki-only, some may be - chunk-only (pure RAG), and some may have both. Do NOT assume both are always - available. Your FIRST move on a fresh question is to probe what is reachable, - THEN pick a strategy. + chunk-only (pure RAG), and some may have both. Do NOT probe this via + search — it is declared deterministically on each `` entry + in the `` block above, in the `capabilities="..."` + attribute. Trust that attribute as the source of truth. - Default "Wiki-first, RAG-verify" cycle (adapt as needed): + 0. **Read Surface Capabilities (no tool call needed):** + For every KB in scope, look up its `capabilities` attribute: + * `capabilities="wiki,chunks"` → full hybrid available + * `capabilities="wiki"` → wiki only (no chunk retrieval) + * `capabilities="chunks"` → pure RAG (no wiki) + * missing / empty → treat as chunks-only and rely on `list_knowledge_chunks` + Let `WIKI_KBS` = the subset whose capabilities contain `wiki`, and + `CHUNK_KBS` = the subset whose capabilities contain `chunks`. The rest + of this workflow is conditional on which of these is non-empty. - 0. **Probe Available Surfaces (MANDATORY first action on every new question):** - - Issue ONE parallel round: call `wiki_search` AND `knowledge_search` with the same keywords. - - Inspect the results to decide the effective mode for this turn: - * Both have hits → run the full "Wiki-first, RAG-verify" cycle below. - * Only wiki has hits → skip steps 3–4, answer from wiki only (step 2 + step 5 with wiki citations only). - * Only chunks have hits → skip step 2, go straight to step 3 (pure RAG). - * Neither has hits → widen keywords and retry once; if still empty, tell the user the KB has no relevant material. + 1. **Classify the Query (only branches reachable given surface availability):** + - *Conceptual / relational / "what is / how does":* prefer wiki when `WIKI_KBS` is non-empty; otherwise fall back to chunks. + - *Specific fact / exact quote / numeric / code-level:* prefer chunks when `CHUNK_KBS` is non-empty; otherwise fall back to wiki. + - *Mixed / unclear:* if both sides are available, start from wiki (cheaper context). Otherwise use whichever is available. - 1. **Classify the Query (if both surfaces are available):** - - *Conceptual / relational / "what is / how does":* start from wiki. - - *Specific fact / exact quote / numeric / code-level:* start from RAG chunks. - - *Mixed / unclear:* start from wiki — it's cheaper context. - - 2. **Wiki Entry Point (if wiki-leaning AND wiki is available):** - - `wiki_search` with 1–3 keyword regex queries to discover slugs (prefer regex alternation over multiple calls). - - Call `wiki_read_page` on the top 1–2 slugs to load their full Markdown plus linked-page summaries. + 2. **Wiki Entry Point (only when `WIKI_KBS` is non-empty AND wiki was chosen):** + - `wiki_search` with 1–3 regex queries (prefer one alternation query over multiple calls), restricted to `WIKI_KBS`. + - Call `wiki_read_page` on the top 1–2 slugs to load full Markdown plus linked-page summaries. - Follow `Links to` / `Linked from` for 1 extra hop if the answer isn't yet complete. - 3. **Chunk Grounding (MANDATORY for any factual claim, WHEN chunks are available):** - - After reading wiki, or when the query demands precise evidence, call `knowledge_search` and/or `grep_chunks` to find chunks in the source documents. - - Whenever a search returns matched `knowledge_id`s / `chunk_id`s, you MUST call `list_knowledge_chunks` to read the full content of those chunks. Do not answer from search snippets alone. - - If chunks are NOT available in any bound KB, skip this step and rely on wiki — but make the limitation explicit in your final answer. + 3. **Chunk Grounding (only when `CHUNK_KBS` is non-empty):** + MANDATORY for any factual claim the user will verify (exact quotes, numbers, code, names). + - Call `knowledge_search` (semantic) and/or `grep_chunks` (regex) scoped to `CHUNK_KBS`. + - Whenever a search returns matched `knowledge_id`s / `chunk_id`s, you MUST call `list_knowledge_chunks` to read the full content. Do NOT answer from search snippets alone. + - If the first pass returns nothing relevant AND `WIKI_KBS` is also non-empty, fall back to a wiki search before giving up — the wiki may still list the concept under a different phrasing. + - If `CHUNK_KBS` is empty, skip this step and rely on wiki, but make that limitation explicit in the final answer. - 4. **Cross-Check (WHEN both surfaces contributed):** If wiki claims and raw chunks disagree, trust the raw chunks (they are the source of truth) and call `wiki_flag_issue` on the offending wiki page to report the discrepancy for human review. + 4. **Cross-Check (only when BOTH surfaces contributed to the answer):** + If wiki claims and raw chunks disagree, trust the raw chunks and call `wiki_flag_issue` on the offending wiki page to report the discrepancy for human review. - 5. **Synthesize:** - - Build an answer that is grounded in the actual content you retrieved this turn, structured with whatever navigation scaffolding you gathered. - - Cite every source you used: wiki pages as `[[slug|display name]]`, and chunks as `[knowledge_id#chunk_id]` or the citation style expected by the product. - - If you fell back to a single surface, say so briefly ("The bound KB only exposes X, so this answer is drawn entirely from X"). + 5. **Fallback on Empty First Pass (single retry, then stop):** + If the primary surface returned nothing useful: + * Widen / re-phrase the query (alternation, synonyms, romanized vs CJK form) and retry ONCE on the same surface. + * If still empty and the *other* surface is available, try it once. + * If both come up empty, tell the user the bound KB has no relevant material — do NOT invent. + + 6. **Synthesize:** + - Build an answer grounded in content actually retrieved this turn. + - Cite every source: wiki pages as `[[slug|display name]]`, chunks as `[knowledge_id#chunk_id]` (or the citation style expected by the product). + - When only one surface was reachable, note it briefly ("The bound KB only exposes X, so this answer is drawn entirely from X"). @@ -444,7 +458,7 @@ templates: * **wiki_search:** Wiki entry point. Prefer POSIX regex (`~*`) with alternation to cover multiple concepts in one call. * **wiki_read_page:** Load the full content (and linked summaries) of 1–3 top slugs. Batch multiple slugs in a single call when possible. * **knowledge_search:** Semantic retrieval over raw chunks. Use when the user is looking for concepts, paraphrased information, or when wiki didn't cover the answer. - * **grep_chunks:** Lexical / keyword retrieval over raw chunks. Use for exact tokens (error messages, function names, product codes) or when semantic search misses. + * **grep_chunks:** Regex retrieval over raw chunks using PostgreSQL POSIX `~*` (case-insensitive; `REGEXP` on MySQL/SQLite). Input field is `queries` (1–5 regex strings). STRONGLY PREFER using regex to search for multiple concepts at once — one alternation query `error_code_a|error_code_b` beats multiple single-keyword calls. Literal text also works. Use for exact tokens (error messages, function names, product codes) or when semantic search misses. * **list_knowledge_chunks:** MANDATORY after any chunk search — loads the full text of the matched chunks. * **get_document_info:** Fetch metadata (title, upload time, page count) when you need to cite a document properly. * **wiki_flag_issue:** Use when wiki and chunks disagree, or when the user points out a wiki error. diff --git a/internal/agent/prompts.go b/internal/agent/prompts.go index 06960a69..c60c928c 100644 --- a/internal/agent/prompts.go +++ b/internal/agent/prompts.go @@ -78,7 +78,13 @@ type KnowledgeBaseInfo struct { Type string // Knowledge base type: "document" or "faq" Description string DocCount int - RecentDocs []RecentDocInfo // Recently added documents (up to 10) + // Capabilities lists the retrieval surfaces this KB exposes. Any subset of + // {"wiki", "chunks"}. "chunks" is present when the KB has vector and/or + // keyword (BM25) indexing enabled. This is the *deterministic* source of + // truth the agent should consult before picking a retrieval strategy — + // significantly more reliable than running probing searches. + Capabilities []string + RecentDocs []RecentDocInfo // Recently added documents (up to 10) } // PlaceholderDefinition defines a placeholder exposed to UI/configuration @@ -118,8 +124,12 @@ func formatKnowledgeBaseList(kbInfos []*KnowledgeBaseInfo) string { if kbType == "" { kbType = "document" } - b.WriteString(fmt.Sprintf("\n", - kb.ID, kb.Name, kbType, kb.DocCount)) + capsAttr := "" + if len(kb.Capabilities) > 0 { + capsAttr = fmt.Sprintf(" capabilities=\"%s\"", strings.Join(kb.Capabilities, ",")) + } + b.WriteString(fmt.Sprintf("\n", + kb.ID, kb.Name, kbType, kb.DocCount, capsAttr)) if kb.Description != "" { b.WriteString(fmt.Sprintf("%s\n", kb.Description)) } diff --git a/internal/agent/tools/grep_chunks.go b/internal/agent/tools/grep_chunks.go index 4637d7f5..e11ff126 100644 --- a/internal/agent/tools/grep_chunks.go +++ b/internal/agent/tools/grep_chunks.go @@ -5,8 +5,10 @@ import ( "encoding/json" "fmt" "math" + "regexp" "sort" "strings" + "sync" "github.com/Tencent/WeKnora/internal/logger" "github.com/Tencent/WeKnora/internal/searchutil" @@ -16,86 +18,70 @@ import ( var grepChunksTool = BaseTool{ name: ToolGrepChunks, - description: `Unix-style text pattern matching tool for knowledge base chunks. - -Searches for text patterns in chunk content using strict literal text matching (fixed-string search). This tool performs exact keyword lookup, not semantic search. - -## Core Function -Performs exact, literal text pattern matching. Accepts multiple patterns and returns chunks matching any of them (OR logic). - -## CRITICAL – Keyword Extraction Rules -This tool MUST receive **short, high-value keywords** only. -**Do NOT use long phrases, sentences, or multi-word expressions.** - -Provide only the **minimal core entities** extracted from user query, such as: -- Proper nouns -- Key concepts -- Domain terms -- Distinct entities that define the query - -### Requirements -- Keywords should be **1–3 words maximum** -- Focus exclusively on **core entities**, not descriptions -- Break complex input into individual, essential keywords -- Avoid phrases, explanations, or anything that reduces match probability -- Preserve precision details embedded in the query (e.g., version numbers, build IDs) when they materially define the entity being matched. - -Long phrases dramatically reduce recall because chunks rarely contain identical wording. -Only short, atomic keywords ensure accurate matching and avoid unrelated retrieval. - - -## Usage -grep_chunks scans enabled chunks across the specified knowledge bases and returns those containing any provided keyword. Matching is case-insensitive, with chunk indices and local context included. - -## When to Use -- Extracting core entities from user input -- Exact keyword presence checks -- Fast preliminary filtering before semantic search -- Situations requiring deterministic text search -`, + description: `Search knowledge base chunk content using PostgreSQL POSIX regular expressions (~* operator, case-insensitive; REGEXP on MySQL/SQLite). +STRONGLY PREFER using regex to search for multiple concepts at once rather than simple plain text queries. +Returns matching chunks with per-pattern hit counts and a around the first match (each tagged with its knowledge_id and chunk_id). +Examples: +- Alternation (RECOMMENDED): "stardust|skyvault" (matches either word) +- Multiple terms (RECOMMENDED): "psionic.*engine" (matches both words in order) +- Word boundary / anchor: "\\brag\\b" or "^chapter\\s+\\d+" +- Plain text: "engine" (matches literal substring anywhere in chunk content) +IMPORTANT — JSON escaping: every backslash in a regex MUST be written as \\ inside the JSON tool arguments (e.g. to search for literal "C++" write "C\\+\\+", NOT "C\+\+"; for "\d+" write "\\d+"). Plain "\+" / "\d" etc. are invalid JSON escapes and will fail to parse. +Use this to locate candidate chunks by exact identifiers, error codes, product names, or recurring terms. Pair with list_knowledge_chunks afterwards to read the full context around any promising chunk_id.`, schema: json.RawMessage(`{ "type": "object", "properties": { - "patterns": { + "queries": { "type": "array", - "description": "REQUIRED: Text patterns to search for. Can be a single pattern or multiple patterns. Treated as literal text (fixed string matching). Results match any of the patterns (OR logic).", - "items": { - "type": "string" - }, - "minItems": 1 + "items": { "type": "string" }, + "description": "List of regex queries to run. A chunk matches when ANY query matches its content. Prefer one alternation query (\"a|b|c\") over multiple single-keyword queries.", + "minItems": 1, + "maxItems": 5 }, "knowledge_base_ids": { "type": "array", - "description": "Filter by knowledge base IDs. If empty, searches all allowed KBs.", - "items": { - "type": "string" - } + "items": { "type": "string" }, + "description": "Optional: restrict search to specific KB IDs within the agent scope." }, - "max_results": { + "limit": { "type": "integer", - "description": "Maximum number of matching chunks to return (default: 50, max: 200)", - "default": 50, + "description": "Max matching chunks to return (default 30, max 100).", + "default": 30, "minimum": 1, - "maximum": 200 + "maximum": 100 } }, - "required": ["patterns"] + "required": ["queries"] }`), } -// GrepChunksInput defines the input parameters for grep chunks tool +// GrepChunksInput defines the input parameters for grep chunks tool. +// The canonical parameter names are `queries` and `limit` (mirroring +// wiki_search). The legacy `patterns` and `max_results` keys remain accepted +// so older model outputs or external callers don't break silently. type GrepChunksInput struct { - Patterns []string `json:"patterns" ` + Queries []string `json:"queries,omitempty"` + Patterns []string `json:"patterns,omitempty"` // legacy alias for queries KnowledgeBaseIDs []string `json:"knowledge_base_ids,omitempty"` - MaxResults int `json:"max_results,omitempty"` + Limit int `json:"limit,omitempty"` + MaxResults int `json:"max_results,omitempty"` // legacy alias for limit } -// GrepChunksTool performs text pattern matching in knowledge base chunks -// Similar to grep command in Unix-like systems, but operates on knowledge base content +// GrepChunksTool performs regex pattern matching across knowledge base chunks. +// PostgreSQL: uses the case-insensitive POSIX operator ~*. +// MySQL/SQLite: falls back to REGEXP. +// +// The tool tracks previously-returned chunk IDs per-instance (one instance per +// agent session) so that a subsequent search hitting the same chunk can be +// rendered compactly with an `already_seen="true"` marker instead of replaying +// the snippet, mirroring the UX of wiki_search. type GrepChunksTool struct { BaseTool db *gorm.DB - searchTargets types.SearchTargets // Pre-computed unified search targets with KB-tenant mapping + searchTargets types.SearchTargets + + mu sync.Mutex + seenChunks map[string]bool } // NewGrepChunksTool creates a new grep chunks tool @@ -104,6 +90,7 @@ func NewGrepChunksTool(db *gorm.DB, searchTargets types.SearchTargets) *GrepChun BaseTool: grepChunksTool, db: db, searchTargets: searchTargets, + seenChunks: make(map[string]bool), } } @@ -111,7 +98,6 @@ func NewGrepChunksTool(db *gorm.DB, searchTargets types.SearchTargets) *GrepChun func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*types.ToolResult, error) { logger.Infof(ctx, "[Tool][GrepChunks] Execute started") - // Parse args from json.RawMessage var input GrepChunksInput if err := json.Unmarshal(args, &input); err != nil { logger.Errorf(ctx, "[Tool][GrepChunks] Failed to parse args: %v", err) @@ -121,36 +107,60 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty }, err } - // Parse pattern parameter (required) - support multiple patterns - patterns := input.Patterns + // Accept both canonical (`queries`) and legacy (`patterns`) field names. + // When both are present we concatenate, preserving whichever came first, + // so a caller migrating between the two won't end up with nothing. + rawQueries := append([]string{}, input.Queries...) + rawQueries = append(rawQueries, input.Patterns...) - // Validate patterns - if len(patterns) == 0 { - logger.Errorf(ctx, "[Tool][GrepChunks] Missing or invalid patterns parameter") - return &types.ToolResult{ - Success: false, - Error: "pattern parameter is required and must contain at least one non-empty pattern", - }, fmt.Errorf("missing pattern parameter") - } - - // Use default values for all options - countOnly := false // default: show results - - maxResults := 50 - if input.MaxResults > 0 { - maxResults = input.MaxResults - if maxResults < 1 { - maxResults = 1 - } else if maxResults > 200 { - maxResults = 200 + queries := make([]string, 0, len(rawQueries)) + for _, q := range rawQueries { + if strings.TrimSpace(q) != "" { + queries = append(queries, q) } } - // Get allowed KBs from searchTargets + if len(queries) == 0 { + logger.Errorf(ctx, "[Tool][GrepChunks] Missing or empty queries parameter") + return &types.ToolResult{ + Success: false, + Error: "queries parameter is required and must contain at least one non-empty regex query", + }, fmt.Errorf("missing queries parameter") + } + if len(queries) > 5 { + queries = queries[:5] + } + + // Compile queries with (?i) prefix for case-insensitive Go-side matching. + // Compilation also validates the regex syntax before we send it to the DB. + compiled := make([]*regexp.Regexp, 0, len(queries)) + for _, q := range queries { + re, err := regexp.Compile("(?i)" + q) + if err != nil { + logger.Errorf(ctx, "[Tool][GrepChunks] Invalid regex %q: %v", q, err) + return &types.ToolResult{ + Success: false, + Error: fmt.Sprintf("invalid regex query %q: %v", q, err), + }, err + } + compiled = append(compiled, re) + } + + // Canonical `limit`, with `max_results` accepted as legacy alias. + limit := input.Limit + if limit <= 0 && input.MaxResults > 0 { + limit = input.MaxResults + } + if limit <= 0 { + limit = 30 + } + if limit > 100 { + limit = 100 + } + allowedKBIDs := t.searchTargets.GetAllKnowledgeBaseIDs() kbTenantMap := t.searchTargets.GetKBTenantMap() - // Collect all specific knowledge IDs from searchTargets var allowedKnowledgeIDs []string for _, target := range t.searchTargets { if target.Type == types.SearchTargetTypeKnowledge && len(target.KnowledgeIDs) > 0 { @@ -158,12 +168,10 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty } } - // Parse knowledge_base_ids filter from input kbIDs := input.KnowledgeBaseIDs if len(kbIDs) == 0 { kbIDs = allowedKBIDs } else { - // Validate input KBs against allowed KBs validKBs := make([]string, 0) for _, kbID := range kbIDs { if t.searchTargets.ContainsKB(kbID) { @@ -173,11 +181,10 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty kbIDs = validKBs } - logger.Infof(ctx, "[Tool][GrepChunks] Patterns: %v, MaxResults: %d, KBs: %v, KnowledgeIDs: %v, KBTenantMap: %v", - patterns, maxResults, kbIDs, allowedKnowledgeIDs, kbTenantMap) + logger.Infof(ctx, "[Tool][GrepChunks] Queries: %v, Limit: %d, KBs: %v, KnowledgeIDs: %v", + queries, limit, kbIDs, allowedKnowledgeIDs) - // Build and execute query with tenant info - results, totalCount, err := t.searchChunks(ctx, patterns, kbIDs, allowedKnowledgeIDs, kbTenantMap) + results, err := t.searchChunks(ctx, queries, kbIDs, allowedKnowledgeIDs, kbTenantMap) if err != nil { logger.Errorf(ctx, "[Tool][GrepChunks] Search failed: %v", err) return &types.ToolResult{ @@ -188,36 +195,28 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty logger.Infof(ctx, "[Tool][GrepChunks] Found %d matching chunks", len(results)) - // Apply deduplication to remove duplicate or near-duplicate chunks deduplicatedResults := t.deduplicateChunks(ctx, results) logger.Infof(ctx, "[Tool][GrepChunks] After deduplication: %d chunks (from %d)", len(deduplicatedResults), len(results)) - // Calculate match scores for sorting (based on match count and position) - scoredResults := t.scoreChunks(ctx, deduplicatedResults, patterns) + // Score chunks using compiled regex (counts + earliest-position boost). + scoredResults := t.scoreChunks(ctx, deduplicatedResults, compiled) - // Apply MMR to reduce redundancy if we have many results finalResults := scoredResults if len(scoredResults) > 10 { - // Use MMR when we have more than 10 results mmrK := len(scoredResults) - if maxResults > 0 && mmrK > maxResults { - mmrK = maxResults + if limit > 0 && mmrK > limit { + mmrK = limit } - logger.Debugf( - ctx, - "[Tool][GrepChunks] Applying MMR: k=%d, lambda=0.7, input=%d results", - mmrK, - len(scoredResults), - ) - mmrResults := t.applyMMR(ctx, scoredResults, patterns, mmrK, 0.7) + logger.Debugf(ctx, "[Tool][GrepChunks] Applying MMR: k=%d, lambda=0.7, input=%d results", + mmrK, len(scoredResults)) + mmrResults := t.applyMMR(ctx, scoredResults, mmrK, 0.7) if len(mmrResults) > 0 { finalResults = mmrResults logger.Infof(ctx, "[Tool][GrepChunks] MMR completed: %d results selected", len(finalResults)) } } - // Sort by match score (descending), then by chunk index sort.Slice(finalResults, func(i, j int) bool { if finalResults[i].MatchedPatterns != finalResults[j].MatchedPatterns { return finalResults[i].MatchedPatterns > finalResults[j].MatchedPatterns @@ -228,29 +227,30 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty return finalResults[i].ChunkIndex < finalResults[j].ChunkIndex }) - aggregatedResults := t.aggregateByKnowledge(finalResults, patterns) - - totalKnowledge := len(aggregatedResults) + if len(finalResults) > limit { + finalResults = finalResults[:limit] + } + // Aggregation by knowledge is still useful for the frontend summary view. + aggregatedResults := t.aggregateByKnowledge(finalResults, queries, compiled) if len(aggregatedResults) > 20 { aggregatedResults = aggregatedResults[:20] } - logger.Infof(ctx, "[Tool][GrepChunks] Aggregated results: %d", len(aggregatedResults)) - - // Format output - output := t.formatOutput(ctx, aggregatedResults, totalCount, patterns, countOnly) + output := t.formatOutput(ctx, finalResults, queries, compiled) return &types.ToolResult{ Success: true, Output: output, Data: map[string]interface{}{ - "patterns": patterns, + "queries": queries, + "patterns": queries, // legacy alias; frontend currently reads `patterns` "knowledge_results": aggregatedResults, "result_count": len(aggregatedResults), - "total_matches": totalKnowledge, + "total_matches": len(finalResults), "knowledge_base_ids": kbIDs, - "max_results": maxResults, + "limit": limit, + "max_results": limit, // legacy alias "display_type": "grep_results", }, }, nil @@ -259,33 +259,42 @@ func (t *GrepChunksTool) Execute(ctx context.Context, args json.RawMessage) (*ty type chunkWithTitle struct { types.Chunk KnowledgeTitle string `json:"knowledge_title" gorm:"column:knowledge_title"` - MatchScore float64 `json:"match_score" gorm:"column:match_score"` // Score based on match count and position - MatchedPatterns int `json:"matched_patterns"` // Number of unique patterns matched + MatchScore float64 `json:"match_score" gorm:"column:match_score"` + MatchedPatterns int `json:"matched_patterns"` TotalChunkCount int `json:"total_chunk_count" gorm:"column:total_chunk_count"` } -// searchChunks performs the database search with pattern matching -// kbTenantMap provides KB-to-tenant mapping for cross-tenant queries +// regexOperatorForDialect returns the SQL operator used to apply a POSIX +// regular expression to a text column for the current dialect. +// PostgreSQL ~* is case-insensitive by default; MySQL/SQLite REGEXP relies on +// collation / driver extensions. +func (t *GrepChunksTool) regexOperatorForDialect() string { + switch t.db.Dialector.Name() { + case "postgres": + return "~*" + default: + // MySQL, SQLite (with the go-sqlite3 REGEXP extension), or anything else + // that understands the REGEXP keyword. + return "REGEXP" + } +} + +// searchChunks performs the database search using regex queries. func (t *GrepChunksTool) searchChunks( ctx context.Context, - patterns []string, + queries []string, kbIDs []string, knowledgeIDs []string, kbTenantMap map[string]uint64, -) ([]chunkWithTitle, int64, error) { +) ([]chunkWithTitle, error) { if len(kbIDs) == 0 && len(knowledgeIDs) == 0 { logger.Warnf(ctx, "[Tool][GrepChunks] No kbIDs or knowledgeIDs specified, returning empty results") - return nil, 0, nil + return nil, nil } - // PostgreSQL uses ILIKE for case-insensitive matching; - // MySQL and SQLite LIKE is already case-insensitive under default collation. - likeOp := "LIKE" - if t.db.Dialector.Name() == "postgres" { - likeOp = "ILIKE" - } + regexOp := t.regexOperatorForDialect() - query := t.db.Debug().WithContext(ctx).Table("chunks"). + query := t.db.WithContext(ctx).Table("chunks"). Select("chunks.id, chunks.content, chunks.chunk_index, chunks.knowledge_id, "+ "chunks.knowledge_base_id, chunks.chunk_type, chunks.created_at, "+ "knowledges.title as knowledge_title"). @@ -311,28 +320,27 @@ func (t *GrepChunksTool) searchChunks( query = query.Where("("+strings.Join(conditions, " OR ")+")", args...) } else { logger.Warnf(ctx, "[Tool][GrepChunks] No valid KB-tenant pairs found") - return nil, 0, nil + return nil, nil } } - if len(patterns) == 1 { - query = query.Where("chunks.content "+likeOp+" ?", "%"+patterns[0]+"%") - } else { - var conditions []string - var args []interface{} - for _, pattern := range patterns { - conditions = append(conditions, "chunks.content "+likeOp+" ?") - args = append(args, "%"+pattern+"%") - } - query = query.Where("("+strings.Join(conditions, " OR ")+")", args...) + // For MySQL/SQLite REGEXP case-insensitivity we rely on the column's default + // collation (utf8mb4_general_ci etc.) OR the driver's REGEXP implementation, + // which mirrors what wiki_search already ships in this codebase. + var regexConditions []string + var regexArgs []interface{} + for _, q := range queries { + regexConditions = append(regexConditions, fmt.Sprintf("chunks.content %s ?", regexOp)) + regexArgs = append(regexArgs, q) } + query = query.Where("("+strings.Join(regexConditions, " OR ")+")", regexArgs...) const maxFetchLimit = 500 var results []chunkWithTitle if err := query.Order("chunks.created_at DESC").Limit(maxFetchLimit).Find(&results).Error; err != nil { logger.Errorf(ctx, "[Tool][GrepChunks] Failed to fetch results: %v", err) - return nil, 0, err + return nil, err } if len(results) > 0 { @@ -371,27 +379,25 @@ func (t *GrepChunksTool) searchChunks( } } - return results, int64(len(results)), nil + return results, nil } -// formatOutput formats the search results as XML +// formatOutput emits per-chunk XML with and +// elements, mirroring the wiki_search output shape. Chunks that were already +// surfaced by a previous call to this tool in the same session are rendered +// compactly with `already_seen="true"` so the LLM doesn't waste context +// re-reading the same snippet. func (t *GrepChunksTool) formatOutput( ctx context.Context, - results []knowledgeAggregation, - totalCount int64, - patterns []string, - countOnly bool, + results []chunkWithTitle, + queries []string, + compiled []*regexp.Regexp, ) string { var b strings.Builder - if countOnly { - b.WriteString(fmt.Sprintf("\n", totalCount)) - return b.String() - } - - b.WriteString(fmt.Sprintf("\n", len(results))) - for _, p := range patterns { - b.WriteString(fmt.Sprintf("%s\n", p)) + b.WriteString(fmt.Sprintf("\n", len(results))) + for _, q := range queries { + b.WriteString(fmt.Sprintf("%s\n", xmlEscape(q))) } if len(results) == 0 { @@ -399,19 +405,57 @@ func (t *GrepChunksTool) formatOutput( return b.String() } - for _, result := range results { - b.WriteString(fmt.Sprintf("\n", - result.KnowledgeID, result.KnowledgeTitle, result.ChunkHitCount, result.TotalChunkCount)) - for _, pattern := range patterns { - count := result.PatternCounts[pattern] - if count > 0 { - b.WriteString(fmt.Sprintf("\n", pattern, count)) + for _, r := range results { + counts := countRegexHits(r.Content, compiled, queries) + snippet := extractSnippetRegex(r.Content, compiled) + + t.mu.Lock() + seen := t.seenChunks[r.ID] + t.seenChunks[r.ID] = true + t.mu.Unlock() + + if seen { + b.WriteString(fmt.Sprintf( + "\n", + xmlEscape(r.ID), + xmlEscape(r.KnowledgeID), + xmlEscape(r.KnowledgeTitle), + r.ChunkIndex, + r.MatchScore, + )) + for _, q := range queries { + if c := counts[q]; c > 0 { + b.WriteString(fmt.Sprintf("\n", + xmlEscape(q), c)) + } + } + b.WriteString("(snippet omitted, already returned in a previous grep_chunks call this session)\n") + b.WriteString("\n") + continue + } + + b.WriteString(fmt.Sprintf( + "\n", + xmlEscape(r.ID), + xmlEscape(r.KnowledgeID), + xmlEscape(r.KnowledgeTitle), + r.ChunkIndex, + r.MatchScore, + )) + for _, q := range queries { + if c := counts[q]; c > 0 { + b.WriteString(fmt.Sprintf("\n", + xmlEscape(q), c)) } } - b.WriteString("\n") + if snippet != "" { + b.WriteString(fmt.Sprintf("%s\n", xmlEscape(snippet))) + } + b.WriteString("\n") } b.WriteString("") + _ = ctx return b.String() } @@ -426,17 +470,21 @@ type knowledgeAggregation struct { DistinctPatterns int `json:"distinct_patterns"` } -func (t *GrepChunksTool) aggregateByKnowledge(results []chunkWithTitle, patterns []string) []knowledgeAggregation { +func (t *GrepChunksTool) aggregateByKnowledge( + results []chunkWithTitle, + queries []string, + compiled []*regexp.Regexp, +) []knowledgeAggregation { if len(results) == 0 { return nil } - patternKeys := make([]string, 0, len(patterns)) - for _, p := range patterns { - if strings.TrimSpace(p) == "" { + queryKeys := make([]string, 0, len(queries)) + for _, q := range queries { + if strings.TrimSpace(q) == "" { continue } - patternKeys = append(patternKeys, p) + queryKeys = append(queryKeys, q) } aggregated := make(map[string]*knowledgeAggregation) @@ -456,23 +504,23 @@ func (t *GrepChunksTool) aggregateByKnowledge(results []chunkWithTitle, patterns KnowledgeBaseID: chunk.KnowledgeBaseID, KnowledgeTitle: title, TotalChunkCount: chunk.TotalChunkCount, - PatternCounts: make(map[string]int, len(patternKeys)), + PatternCounts: make(map[string]int, len(queryKeys)), } - for _, pKey := range patternKeys { - aggregated[knowledgeID].PatternCounts[pKey] = 0 + for _, qKey := range queryKeys { + aggregated[knowledgeID].PatternCounts[qKey] = 0 } } entry := aggregated[knowledgeID] entry.ChunkHitCount++ - patternOccurrences := t.countPatternOccurrences(chunk.Content, patternKeys) - for _, p := range patternKeys { - count := patternOccurrences[p] + occurrences := countRegexHits(chunk.Content, compiled, queryKeys) + for _, q := range queryKeys { + count := occurrences[q] if count == 0 { continue } - entry.PatternCounts[p] += count + entry.PatternCounts[q] += count entry.TotalPatternHits += count } } @@ -504,38 +552,89 @@ func (t *GrepChunksTool) aggregateByKnowledge(results []chunkWithTitle, patterns return resultSlice } -func (t *GrepChunksTool) countPatternOccurrences(content string, patterns []string) map[string]int { +// countRegexHits returns the total number of matches per (compiled) pattern +// within content, keyed by the original (uncompiled) pattern string. +func countRegexHits(content string, compiled []*regexp.Regexp, patterns []string) map[string]int { counts := make(map[string]int, len(patterns)) - if content == "" || len(patterns) == 0 { + if content == "" || len(compiled) == 0 { return counts } - - contentLower := strings.ToLower(content) - for _, pattern := range patterns { - p := strings.ToLower(pattern) - if strings.TrimSpace(p) == "" { + for i, re := range compiled { + if re == nil { continue } - counts[pattern] = countOccurrences(contentLower, p) + matches := re.FindAllStringIndex(content, -1) + counts[patterns[i]] = len(matches) } return counts } -func countOccurrences(text string, pattern string) int { - if pattern == "" { - return 0 +// extractSnippetRegex returns a short context snippet around the earliest +// regex match across any of the provided compiled patterns. Result is +// compressed to a single line and bounded in length on both sides of the +// match to keep the XML output concise. +func extractSnippetRegex(content string, compiled []*regexp.Regexp) string { + if content == "" || len(compiled) == 0 { + return "" } - count := 0 - index := 0 - for index < len(text) { - pos := strings.Index(text[index:], pattern) - if pos == -1 { - break + + earliest := -1 + earliestEnd := -1 + for _, re := range compiled { + if re == nil { + continue + } + loc := re.FindStringIndex(content) + if loc == nil { + continue + } + if earliest < 0 || loc[0] < earliest { + earliest = loc[0] + earliestEnd = loc[1] } - count++ - index += pos + len(pattern) } - return count + if earliest < 0 { + return "" + } + + matchStr := content[earliest:earliestEnd] + before := content[:earliest] + after := content[earliestEnd:] + + const contextRunes = 60 + beforeRunes := []rune(before) + if len(beforeRunes) > contextRunes { + beforeRunes = beforeRunes[len(beforeRunes)-contextRunes:] + } + afterRunes := []rune(after) + if len(afterRunes) > contextRunes { + afterRunes = afterRunes[:contextRunes] + } + matchRunes := []rune(matchStr) + if len(matchRunes) > 120 { + matchRunes = append(matchRunes[:120], []rune("...")...) + } + + snippet := string(beforeRunes) + string(matchRunes) + string(afterRunes) + snippet = strings.ReplaceAll(snippet, "\n", " ") + for strings.Contains(snippet, " ") { + snippet = strings.ReplaceAll(snippet, " ", " ") + } + return "... " + strings.TrimSpace(snippet) + " ..." +} + +// xmlEscape replaces characters that would break simple XML attribute / +// element values. It is intentionally minimal because the rendered output is +// consumed by the LLM (forgiving parser) rather than a strict XML processor. +func xmlEscape(s string) string { + replacer := strings.NewReplacer( + "&", "&", + "<", "<", + ">", ">", + "\"", """, + "'", "'", + ) + return replacer.Replace(s) } // deduplicateChunks removes duplicate or near-duplicate chunks using content signature @@ -545,7 +644,6 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW uniqueResults := make([]chunkWithTitle, 0) for _, r := range results { - // Build multiple keys for deduplication keys := []string{r.ID} if r.ParentChunkID != "" { keys = append(keys, "parent:"+r.ParentChunkID) @@ -554,7 +652,6 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW keys = append(keys, fmt.Sprintf("kb:%s#%d", r.KnowledgeID, r.ChunkIndex)) } - // Check if any key is already seen dup := false for _, k := range keys { if seen[k] { @@ -566,7 +663,6 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW continue } - // Check content signature for near-duplicate content sig := t.buildContentSignature(r.Content) if sig != "" { if contentSig[sig] { @@ -575,15 +671,12 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW contentSig[sig] = true } - // Mark all keys as seen for _, k := range keys { seen[k] = true } - uniqueResults = append(uniqueResults, r) } - // If we have duplicates by ID, keep the first one seenByID := make(map[string]bool) deduplicated := make([]chunkWithTitle, 0) for _, r := range uniqueResults { @@ -592,7 +685,7 @@ func (t *GrepChunksTool) deduplicateChunks(ctx context.Context, results []chunkW deduplicated = append(deduplicated, r) } } - + _ = ctx return deduplicated } @@ -601,53 +694,55 @@ func (t *GrepChunksTool) buildContentSignature(content string) string { return searchutil.BuildContentSignature(content) } -// scoreChunks calculates match scores for chunks based on pattern matches +// scoreChunks calculates match scores for chunks based on regex matches. func (t *GrepChunksTool) scoreChunks( ctx context.Context, results []chunkWithTitle, - patterns []string, + compiled []*regexp.Regexp, ) []chunkWithTitle { scored := make([]chunkWithTitle, len(results)) for i := range results { scored[i] = results[i] - score, patternCount := t.calculateMatchScore(results[i].Content, patterns) + score, patternCount := t.calculateMatchScore(results[i].Content, compiled) scored[i].MatchScore = score scored[i].MatchedPatterns = patternCount } + _ = ctx return scored } -// calculateMatchScore calculates a score based on how many patterns match and their positions -func (t *GrepChunksTool) calculateMatchScore(content string, patterns []string) (float64, int) { - if content == "" || len(patterns) == 0 { +// calculateMatchScore counts how many regex patterns match the content and +// applies a small boost for earlier match positions. +func (t *GrepChunksTool) calculateMatchScore(content string, compiled []*regexp.Regexp) (float64, int) { + if content == "" || len(compiled) == 0 { return 0.0, 0 } - contentLower := strings.ToLower(content) matchCount := 0 earliestPos := len(content) - // Count how many patterns match and find earliest position - for _, pattern := range patterns { - patternLower := strings.ToLower(pattern) - if strings.Contains(contentLower, patternLower) { - matchCount++ - // Find position of first match - pos := strings.Index(contentLower, patternLower) - if pos >= 0 && pos < earliestPos { - earliestPos = pos - } + for _, re := range compiled { + if re == nil { + continue + } + loc := re.FindStringIndex(content) + if loc == nil { + continue + } + matchCount++ + if loc[0] < earliestPos { + earliestPos = loc[0] } } - // Score: higher for more matches, slightly higher for earlier positions - // Base score: match ratio (0.0 to 1.0) - baseScore := float64(matchCount) / float64(len(patterns)) + if matchCount == 0 { + return 0.0, 0 + } + + baseScore := float64(matchCount) / float64(len(compiled)) - // Position bonus: earlier matches get slight boost (max 0.1) positionBonus := 0.0 if earliestPos < len(content) { - // Normalize position to [0, 1] and apply small bonus positionRatio := 1.0 - float64(earliestPos)/float64(len(content)) positionBonus = positionRatio * 0.1 } @@ -659,7 +754,6 @@ func (t *GrepChunksTool) calculateMatchScore(content string, patterns []string) func (t *GrepChunksTool) applyMMR( ctx context.Context, results []chunkWithTitle, - patterns []string, k int, lambda float64, ) []chunkWithTitle { @@ -671,18 +765,16 @@ func (t *GrepChunksTool) applyMMR( lambda, k, len(results)) selected := make([]chunkWithTitle, 0, k) - selectedTokenSets := make([]map[string]struct{}, 0, k) // cache of token sets + selectedTokenSets := make([]map[string]struct{}, 0, k) candidates := make([]chunkWithTitle, len(results)) copy(candidates, results) - // Pre-compute token sets for all candidates tokenSets := make([]map[string]struct{}, len(candidates)) for i, r := range candidates { tokenSets[i] = t.tokenizeSimple(r.Content) } - // MMR selection loop for len(selected) < k && len(candidates) > 0 { bestIdx := 0 bestScore := -1.0 @@ -690,13 +782,9 @@ func (t *GrepChunksTool) applyMMR( for i, r := range candidates { relevance := r.MatchScore redundancy := 0.0 - - // Calculate maximum redundancy with already selected results for _, selectedTS := range selectedTokenSets { redundancy = math.Max(redundancy, t.jaccard(tokenSets[i], selectedTS)) } - - // MMR score: balance relevance and diversity mmr := lambda*relevance - (1.0-lambda)*redundancy if mmr > bestScore { bestScore = mmr @@ -704,11 +792,9 @@ func (t *GrepChunksTool) applyMMR( } } - // Add best candidate to selected and remove from candidates selected = append(selected, candidates[bestIdx]) selectedTokenSets = append(selectedTokenSets, tokenSets[bestIdx]) - // Remove corresponding token set. Use swap deletion last := len(candidates) - 1 candidates[bestIdx] = candidates[last] tokenSets[bestIdx] = tokenSets[last] @@ -716,24 +802,6 @@ func (t *GrepChunksTool) applyMMR( tokenSets = tokenSets[:last] } - // Compute average redundancy among selected results - avgRed := 0.0 - if len(selected) > 1 { - pairs := 0 - for i := 0; i < len(selected); i++ { - for j := i + 1; j < len(selected); j++ { - avgRed += t.jaccard(selectedTokenSets[i], selectedTokenSets[j]) // read token from cache - pairs++ - } - } - if pairs > 0 { - avgRed /= float64(pairs) - } - } - - logger.Debugf(ctx, "[Tool][GrepChunks] MMR completed: selected=%d, avg_redundancy=%.4f", - len(selected), avgRed) - return selected } diff --git a/internal/agent/tools/json_repair.go b/internal/agent/tools/json_repair.go index 8bbb8cc4..eb8bf121 100644 --- a/internal/agent/tools/json_repair.go +++ b/internal/agent/tools/json_repair.go @@ -11,6 +11,8 @@ import ( // - Trailing commas before closing brackets // - Single quotes instead of double quotes // - Unquoted keys +// - Invalid backslash escapes inside strings (e.g. regex patterns like +// "C\+\+", "\d+", "\.log$") where the LLM forgot to double-escape. // // Returns the repaired JSON string. If repair is not possible, // returns the original string unchanged (caller should handle parse errors). @@ -30,6 +32,10 @@ func RepairJSON(s string) string { } } + // Fix invalid backslash escapes before anything else, because unbalanced + // strings confuse the comma/bracket trackers below. + s = fixInvalidEscapes(s) + // Fix trailing commas: ,} or ,] s = fixTrailingCommas(s) @@ -39,6 +45,63 @@ func RepairJSON(s string) string { return s } +// fixInvalidEscapes turns invalid JSON string escapes into literal backslash +// sequences. JSON only allows the escape letters \" \\ \/ \b \f \n \r \t \u; +// any other backslash inside a string (e.g. \+ \d \w \. \| \( \)) is a parse +// error. LLMs frequently emit these when they meant to pass a regex pattern +// without double-escaping — rewriting "\+" to "\\+" recovers the LLM's intent +// (the unmarshalled Go string becomes "\+", which is what the regex engine +// expects) and is idempotent on already-valid JSON. +func fixInvalidEscapes(s string) string { + var out strings.Builder + out.Grow(len(s) + 8) + runes := []rune(s) + inString := false + for i := 0; i < len(runes); i++ { + r := runes[i] + if !inString { + out.WriteRune(r) + if r == '"' { + inString = true + } + continue + } + if r == '"' { + out.WriteRune(r) + inString = false + continue + } + if r != '\\' { + out.WriteRune(r) + continue + } + // Found a backslash inside a string. Peek at the next rune to decide + // whether this is a valid JSON escape. + if i+1 >= len(runes) { + // Dangling backslash at EOF — escape it so we don't leave the + // string in a "pending escape" state for downstream parsers. + out.WriteString(`\\`) + continue + } + next := runes[i+1] + switch next { + case '"', '\\', '/', 'b', 'f', 'n', 'r', 't', 'u': + // Valid JSON escape — pass through unchanged. + out.WriteRune(r) + out.WriteRune(next) + i++ + default: + // Invalid escape — most likely a regex metachar that the LLM + // forgot to double-escape. Emit a literal backslash + the next + // rune, so the final Go string preserves the \X sequence. + out.WriteString(`\\`) + out.WriteRune(next) + i++ + } + } + return out.String() +} + // fixTrailingCommas removes trailing commas before closing brackets/braces. func fixTrailingCommas(s string) string { // Simple state machine to handle strings diff --git a/internal/agent/tools/json_repair_test.go b/internal/agent/tools/json_repair_test.go index 50be289f..bcfea7a3 100644 --- a/internal/agent/tools/json_repair_test.go +++ b/internal/agent/tools/json_repair_test.go @@ -5,6 +5,7 @@ import ( "testing" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) func TestRepairJSON(t *testing.T) { @@ -74,4 +75,52 @@ func TestRepairJSON(t *testing.T) { result := RepairJSON(input) assert.True(t, json.Valid([]byte(result)), "result: %s", result) }) + + t.Run("invalid escape: regex plus (C\\+\\+)", func(t *testing.T) { + // The LLM emits a raw regex without double-escaping — \+ is not a + // legal JSON escape. Repair should rewrite it to \\+ so the + // unmarshalled string is the regex pattern `C\+\+`, matching + // literal `C++`. + input := `{"queries": ["C\+\+"]}` + result := RepairJSON(input) + assert.True(t, json.Valid([]byte(result)), "result: %s", result) + + var parsed struct { + Queries []string `json:"queries"` + } + require.NoError(t, json.Unmarshal([]byte(result), &parsed)) + assert.Equal(t, []string{`C\+\+`}, parsed.Queries) + }) + + t.Run("invalid escape: common regex shorthand", func(t *testing.T) { + input := `{"queries": ["\d+\.\d+", "\bword\b", "foo\|bar"]}` + result := RepairJSON(input) + assert.True(t, json.Valid([]byte(result)), "result: %s", result) + + var parsed struct { + Queries []string `json:"queries"` + } + require.NoError(t, json.Unmarshal([]byte(result), &parsed)) + // \b is a VALID JSON escape (backspace) — the LLM gets what it asked + // for (a backspace char), we don't silently rewrite valid escapes. + // \d, \., \| are invalid → repaired to literal backslash sequences. + assert.Equal(t, `\d+\.\d+`, parsed.Queries[0]) + assert.Equal(t, "\bword\b", parsed.Queries[1]) + assert.Equal(t, `foo\|bar`, parsed.Queries[2]) + }) + + t.Run("invalid escape repair is idempotent", func(t *testing.T) { + input := `{"queries": ["C\+\+"]}` + once := RepairJSON(input) + twice := RepairJSON(once) + assert.Equal(t, once, twice) + }) + + t.Run("valid escapes are preserved", func(t *testing.T) { + // Valid JSON escapes must pass through untouched. + input := `{"path": "C:\\Users\\x", "line": "a\nb", "quote": "say \"hi\""}` + result := RepairJSON(input) + assert.Equal(t, input, result) + assert.True(t, json.Valid([]byte(result))) + }) } diff --git a/internal/agent/tools/knowledge_search.go b/internal/agent/tools/knowledge_search.go index 412d9490..8adcd89e 100644 --- a/internal/agent/tools/knowledge_search.go +++ b/internal/agent/tools/knowledge_search.go @@ -117,7 +117,10 @@ type searchResultWithMeta struct { KnowledgeBaseType string // Type of the knowledge base (document, faq, etc.) } -// KnowledgeSearchTool searches knowledge bases with flexible query modes +// KnowledgeSearchTool searches knowledge bases with flexible query modes. +// seenChunks lets repeated calls in the same session surface previously- +// returned chunks in a compact form (mirroring wiki_search's de-duping UX) +// so the LLM doesn't burn tokens re-reading identical content. type KnowledgeSearchTool struct { BaseTool knowledgeBaseService interfaces.KnowledgeBaseService @@ -127,6 +130,9 @@ type KnowledgeSearchTool struct { rerankModel rerank.Reranker chatModel chat.Chat // Optional chat model for LLM-based reranking config *config.Config // Global config for fallback values + + seenMu sync.Mutex + seenChunks map[string]bool } // NewKnowledgeSearchTool creates a new knowledge search tool @@ -148,6 +154,7 @@ func NewKnowledgeSearchTool( rerankModel: rerankModel, chatModel: chatModel, config: cfg, + seenChunks: make(map[string]bool), } } @@ -1086,9 +1093,15 @@ func (t *KnowledgeSearchTool) formatOutput( kbCounts[r.KnowledgeID]++ } - // Format individual results as XML + // Format individual results as XML. Tag names are kept in sync with + // wiki_search (``, per-entry element, ``) so that + // agents and downstream consumers see a single consistent shape across + // all retrieval tools. var ob strings.Builder ob.WriteString(fmt.Sprintf("\n", len(results))) + for _, q := range queries { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(q))) + } formattedResults := make([]map[string]interface{}, 0, len(results)) @@ -1136,46 +1149,82 @@ func (t *KnowledgeSearchTool) formatOutput( } } - ob.WriteString(fmt.Sprintf("\n", - i+1, result.ID, result.ChunkIndex, result.KnowledgeID, result.KnowledgeTitle)) - ob.WriteString(fmt.Sprintf("%s\n", result.Content)) + t.seenMu.Lock() + seen := t.seenChunks[result.ID] + t.seenChunks[result.ID] = true + t.seenMu.Unlock() - if result.ImageInfo != "" { - var imageInfos []types.ImageInfo - if err := json.Unmarshal([]byte(result.ImageInfo), &imageInfos); err == nil && len(imageInfos) > 0 { - for _, img := range imageInfos { - ob.WriteString(fmt.Sprintf("\n", img.URL)) - if img.Caption != "" { - ob.WriteString(fmt.Sprintf("%s\n", img.Caption)) + if seen { + // Compact rendering for chunks we already returned in a previous + // knowledge_search call during this session. The model has the + // content in context already, so re-emitting it only burns tokens. + ob.WriteString(fmt.Sprintf( + "\n", + i+1, + xmlEscape(result.ID), + result.ChunkIndex, + xmlEscape(result.KnowledgeID), + xmlEscape(result.KnowledgeBaseID), + xmlEscape(result.KnowledgeTitle), + result.Score, + xmlEscape(result.SourceQuery), + )) + ob.WriteString("(content omitted, already returned in a previous knowledge_search call this session)\n") + ob.WriteString("\n") + } else { + ob.WriteString(fmt.Sprintf( + "\n", + i+1, + xmlEscape(result.ID), + result.ChunkIndex, + xmlEscape(result.KnowledgeID), + xmlEscape(result.KnowledgeBaseID), + xmlEscape(result.KnowledgeTitle), + result.Score, + xmlEscape(result.SourceQuery), + )) + if snippet := extractSnippetForQueries(result.Content, queries); snippet != "" { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(snippet))) + } + ob.WriteString(fmt.Sprintf("%s\n", result.Content)) + + if result.ImageInfo != "" { + var imageInfos []types.ImageInfo + if err := json.Unmarshal([]byte(result.ImageInfo), &imageInfos); err == nil && len(imageInfos) > 0 { + for _, img := range imageInfos { + ob.WriteString(fmt.Sprintf("\n", xmlEscape(img.URL))) + if img.Caption != "" { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(img.Caption))) + } + if img.OCRText != "" { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(img.OCRText))) + } + ob.WriteString("\n") } - if img.OCRText != "" { - ob.WriteString(fmt.Sprintf("%s\n", img.OCRText)) + } + } + + if faqMeta != nil { + ob.WriteString("\n") + if faqMeta.StandardQuestion != "" { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(faqMeta.StandardQuestion))) + } + if len(faqMeta.SimilarQuestions) > 0 { + for _, sq := range faqMeta.SimilarQuestions { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(sq))) } - ob.WriteString("\n") } + if len(faqMeta.Answers) > 0 { + for _, ans := range faqMeta.Answers { + ob.WriteString(fmt.Sprintf("%s\n", xmlEscape(ans))) + } + } + ob.WriteString("\n") } - } - if faqMeta != nil { - ob.WriteString("\n") - if faqMeta.StandardQuestion != "" { - ob.WriteString(fmt.Sprintf("%s\n", faqMeta.StandardQuestion)) - } - if len(faqMeta.SimilarQuestions) > 0 { - for _, sq := range faqMeta.SimilarQuestions { - ob.WriteString(fmt.Sprintf("%s\n", sq)) - } - } - if len(faqMeta.Answers) > 0 { - for _, ans := range faqMeta.Answers { - ob.WriteString(fmt.Sprintf("%s\n", ans)) - } - } - ob.WriteString("\n") + ob.WriteString("\n") } - ob.WriteString("\n") - formattedResults = append(formattedResults, map[string]interface{}{ "result_index": i + 1, "chunk_id": result.ID, @@ -1238,7 +1287,7 @@ func (t *KnowledgeSearchTool) formatOutput( remaining := totalChunks - int64(retrievedCount) percentage := float64(retrievedCount) / float64(totalChunks) * 100 ob.WriteString(fmt.Sprintf("\n", - knowledgeID, title, totalChunks, retrievedCount, remaining, percentage)) + xmlEscape(knowledgeID), xmlEscape(title), totalChunks, retrievedCount, remaining, percentage)) } } ob.WriteString("\n") @@ -1438,6 +1487,89 @@ func (t *KnowledgeSearchTool) tokenizeSimple(text string) map[string]struct{} { return searchutil.TokenizeSimple(text) } +// extractSnippetForQueries tries to produce a short contextual snippet around +// the first occurrence of any token extracted from the provided queries. +// When no token matches (common for fully paraphrased semantic queries) it +// falls back to the leading 160 runes of content so callers always get +// something to scan. The snippet is single-lined and bounded in length to +// keep the rendered XML compact. +func extractSnippetForQueries(content string, queries []string) string { + content = strings.TrimSpace(content) + if content == "" { + return "" + } + + tokens := make([]string, 0, 8) + seen := make(map[string]struct{}) + for _, q := range queries { + for _, tok := range strings.FieldsFunc(q, func(r rune) bool { + // Split on whitespace and common punctuation; keep CJK as whole tokens. + switch r { + case ' ', '\t', '\n', '\r', ',', '.', ';', ':', '?', '!', + '(', ')', '[', ']', '{', '}', '"', '\'': + return true + } + return false + }) { + tok = strings.ToLower(strings.TrimSpace(tok)) + // Skip trivially-short stopwords. Three rune floor covers most + // English function words without chopping CJK bigrams. + if len([]rune(tok)) < 2 { + continue + } + if _, ok := seen[tok]; ok { + continue + } + seen[tok] = struct{}{} + tokens = append(tokens, tok) + } + } + + lowered := strings.ToLower(content) + earliest := -1 + earliestEnd := -1 + for _, tok := range tokens { + idx := strings.Index(lowered, tok) + if idx < 0 { + continue + } + end := idx + len(tok) + if earliest < 0 || idx < earliest { + earliest = idx + earliestEnd = end + } + } + + const contextRunes = 60 + if earliest < 0 { + runes := []rune(content) + if len(runes) > contextRunes*2 { + return strings.TrimSpace(string(runes[:contextRunes*2])) + " ..." + } + return content + } + + matchStr := content[earliest:earliestEnd] + before := content[:earliest] + after := content[earliestEnd:] + + beforeRunes := []rune(before) + if len(beforeRunes) > contextRunes { + beforeRunes = beforeRunes[len(beforeRunes)-contextRunes:] + } + afterRunes := []rune(after) + if len(afterRunes) > contextRunes { + afterRunes = afterRunes[:contextRunes] + } + + snippet := string(beforeRunes) + matchStr + string(afterRunes) + snippet = strings.ReplaceAll(snippet, "\n", " ") + for strings.Contains(snippet, " ") { + snippet = strings.ReplaceAll(snippet, " ", " ") + } + return "... " + strings.TrimSpace(snippet) + " ..." +} + // jaccard calculates Jaccard similarity between two token sets func (t *KnowledgeSearchTool) jaccard(a, b map[string]struct{}) float64 { return searchutil.Jaccard(a, b) diff --git a/internal/agent/tools/wiki_read_source_doc.go b/internal/agent/tools/wiki_read_source_doc.go index e09de05c..009e54fd 100644 --- a/internal/agent/tools/wiki_read_source_doc.go +++ b/internal/agent/tools/wiki_read_source_doc.go @@ -35,7 +35,7 @@ If neither query nor range is provided, it returns the beginning of the document }, "query": { "type": "string", - "description": "Optional: A regex query to filter the document chunks. Use this to find specific quotes or details efficiently." + "description": "Optional: A regex query to filter the document chunks. Use this to find specific quotes or details efficiently. Remember to double-escape backslashes for JSON: write \"C\\\\+\\\\+\" (NOT \"C\\+\\+\") and \"\\\\d+\" (NOT \"\\d+\")." }, "start_chunk_index": { "type": "integer", diff --git a/internal/agent/tools/wiki_tools.go b/internal/agent/tools/wiki_tools.go index f94f6deb..3212e131 100644 --- a/internal/agent/tools/wiki_tools.go +++ b/internal/agent/tools/wiki_tools.go @@ -439,6 +439,7 @@ Examples: - Multiple terms (RECOMMENDED): "psionic.*engine" (matches both words in order) - Prefix matching: "^entity/.*" (finds all entities) - Plain text: "engine" (matches anywhere in title/content/slug/summary) +IMPORTANT — JSON escaping: every backslash in a regex MUST be written as \\ inside the JSON tool arguments (e.g. to search for literal "C++" write "C\\+\\+", NOT "C\+\+"; for "\d+" write "\\d+"). Plain "\+" / "\d" etc. are invalid JSON escapes and will fail to parse. Use this to find relevant wiki pages when you don't know the exact slug.`, json.RawMessage(`{ "type": "object", diff --git a/internal/application/service/agent_service.go b/internal/application/service/agent_service.go index 7d16c4b4..38e1b6a6 100644 --- a/internal/application/service/agent_service.go +++ b/internal/application/service/agent_service.go @@ -709,18 +709,41 @@ func (s *agentService) getKnowledgeBaseInfos(ctx context.Context, kbIDs []string kbType = "document" // Default type } kbInfos = append(kbInfos, &agent.KnowledgeBaseInfo{ - ID: kb.ID, - Name: kb.Name, - Type: kbType, - Description: kb.Description, - DocCount: docCount, - RecentDocs: recentDocs, + ID: kb.ID, + Name: kb.Name, + Type: kbType, + Description: kb.Description, + DocCount: docCount, + Capabilities: kbRetrievalCapabilities(kb), + RecentDocs: recentDocs, }) } return kbInfos, nil } +// kbRetrievalCapabilities reports which retrieval surfaces a KB exposes. +// Surfaces are the static facts the hybrid agent prompt consults to pick its +// retrieval strategy — the agent should NOT need to probe this via search. +// +// Returned values are a subset of {"wiki", "chunks"}: +// - "wiki" → the KB has wiki ingestion enabled (wiki_search / wiki_read_page) +// - "chunks" → the KB has vector and/or keyword (BM25) indexing enabled +// (knowledge_search / grep_chunks) +func kbRetrievalCapabilities(kb *types.KnowledgeBase) []string { + if kb == nil { + return nil + } + caps := make([]string, 0, 2) + if kb.IsWikiEnabled() { + caps = append(caps, "wiki") + } + if kb.IsVectorEnabled() || kb.IsKeywordEnabled() { + caps = append(caps, "chunks") + } + return caps +} + // getSelectedDocumentInfos retrieves detailed information for user-selected documents (via @ mention) // This loads the actual content of the documents to include in the system prompt func (s *agentService) getSelectedDocumentInfos(ctx context.Context, knowledgeIDs []string) ([]*agent.SelectedDocumentInfo, error) {