- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings. - The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities. - Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system. These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
15 KiB
WeKnora Wiki System - Executive Summary
Document: WIKI_SYSTEM_SUMMARY.md
Date: 2026-04-20
Status: Complete Technical Analysis
What is the Wiki System?
WeKnora's Wiki is a document synthesis and knowledge graph feature for document-type knowledge bases. It:
- Automatically extracts entities and concepts from uploaded documents using LLM
- Generates LLM-synthesized wiki pages that aggregate information across documents
- Maintains bidirectional link graphs showing relationships between concepts
- Provides visualization and statistics for knowledge exploration
- Boosts retrieval of wiki pages in search results
- Tracks health with linting and issue detection
Critical: Wiki is NOT a KB type — it's a feature flag (WikiConfig.Enabled) on document-type KBs.
Quick Architecture Map
Document Upload (KB with wiki enabled)
↓
ChunkService (creates chunks)
↓
EnqueueWikiIngest() → Redis pending list + Asynq task (30s delay)
↓
ProcessWikiIngest (batch handler) ─────┬─ MAP Phase (extract entities/concepts)
├─ 10 parallel goroutines │
├─ LLM calls for each document │
└─ Accumulate updates │
├─ REDUCE Phase (merge & synthesize)
├─ 10 parallel goroutines
├─ LLM calls for page synthesis
└─ Save to DB
↓
Post-Processing
├─ Append log entries
├─ Rebuild index page (LLM-updated intro)
├─ Clean dead links
├─ Inject cross-links (pure text matching)
├─ Publish draft pages
└─ Schedule follow-up if more pending
↓
Wiki pages now indexed as chunks (ChunkType="wiki_page")
↓
Search retrieval → WikiBoost plugin (1.3x score multiplier)
↓
Agents can read/write wiki pages via tools
Core Data Structures
WikiPage (Database Entity)
type WikiPage struct {
ID string // UUID
KnowledgeBaseID string // Parent KB
Slug string // URL-friendly: "entity/acme-corp"
Title string // Human title
PageType string // summary|entity|concept|index|log|synthesis|comparison
Status string // draft|published|archived
Content string // Full markdown
Summary string // One-line for index
Aliases []string // Alternative names
SourceRefs []string // Knowledge IDs: "id|title"
InLinks []string // Pages linking to this
OutLinks []string // Pages this links to
Version int // Optimistic locking
}
WikiConfig (Feature Configuration)
type WikiConfig struct {
Enabled bool // Feature flag
AutoIngest bool // Auto-generate from uploads
SynthesisModelID string // LLM model (defaults to KB.SummaryModelID)
MaxPagesPerIngest int // Batch size limit (0 = unlimited)
}
Key Services
WikiPageService
Responsibilities:
- CRUD operations on wiki pages
- Link parsing and bidirectional reference management
- Index/log page creation and updates
- Graph generation for visualization
- Statistics aggregation
- Cross-link injection (mentions → auto-links)
Key Methods:
CreatePage(),UpdatePage(),DeletePage()GetPageBySlug(),ListPages(),SearchPages()GetIndex(),GetLog()GetGraph()(for visualization)GetStats()(aggregate statistics)RebuildLinks()(full rebuild)InjectCrossLinks()(auto-link mentions)
WikiIngestService
Responsibilities:
- Orchestrate async wiki generation from documents
- Manage Redis pending queue and distributed locks
- Coordinate MAP-REDUCE batch processing
- Call LLM prompts for extraction and synthesis
- Handle retract operations (document deletion)
- Maintain operation log
Key Methods:
EnqueueWikiIngest()(queue a document)EnqueueWikiRetract()(queue a deletion)ProcessWikiIngest()(main batch handler)mapOneDocument()(extract entities/concepts)reduceSlugUpdates()(merge and synthesize)
Async Pipeline Details
Debouncing Mechanism
Time Action
─────────────────────────────────────
0s doc1 uploaded → RPush + Enqueue(delay=30s)
5s doc2 uploaded → RPush + Enqueue(delay=30s)
10s doc3 uploaded → RPush + Enqueue(delay=30s)
30s Task 1 fires → Drains [doc1,doc2,doc3] → Process all
35s Task 2 fires → Drains [] → No-op return
40s Task 3 fires → Drains [] → No-op return
Benefit: Natural batching without explicit deduplication logic.
MAP Phase
Parallel extraction (10 goroutines max):
- For each document: Extract entities & concepts from chunks
- Call LLM:
WikiKnowledgeExtractPrompt - Result: List of SlugUpdate operations
Per Document:
- Read chunks from ChunkRepository
- Reconstruct content (truncated to 32KB)
- Call LLM extraction (single call for both entities + concepts)
- Deduplicate against existing pages (single LLM call for batch)
- Generate summary page
- Identify stale pages (from previous ingest)
REDUCE Phase
Parallel merging (10 goroutines max):
- For each affected slug: Apply all accumulated updates
- Call LLM:
WikiPageModifyPrompt(for entity/concept synthesis) - Save page to DB
Per Slug:
- Separate updates into: additions, retracts, summary
- Build LLM prompt with context (existing content, new info, remaining sources)
- Call LLM to synthesize merged content
- Update page record
Post-Processing
1. Append log entries (one per operation)
2. Rebuild index page (LLM updates intro, code rebuilds directory)
3. Clean dead links (remove [[broken-links]])
4. Inject cross-links (auto-link mentions of other pages)
5. Publish draft pages (transition to published)
6. Trim processed items from Redis pending list
7. Schedule follow-up if more items remain
Cross-Link Injection
Purpose: Auto-detect mentions of existing wiki page titles and wrap with [[wiki-links]].
Implementation: Pure text processing (no LLM cost)
Algorithm:
- Sort refs by matchText length (descending) → longer strings match first
- Compute forbidden spans → regions untouched (code blocks, existing links, etc.)
- For each ref:
- Skip if already linked elsewhere
- Find first safe match outside forbidden spans
- Check word boundaries (for ASCII text)
- Wrap with
[[slug|matchText]]
Forbidden Regions:
- Fenced code blocks (```/~~~)
- Inline code (backticks)
- Existing wiki links
- Markdown links text
- Images
- Reference-style links
- Autolinks
Retrieval Integration
ChunkType Classification
ChunkTypeWikiPage = "wiki_page"
Used to:
1. Differentiate wiki pages from document chunks
2. Apply retrieval boost (1.3x score multiplier)
3. Filter results if needed
WikiBoost Plugin
Phase: CHUNK_RERANK (after initial retrieval and reranking)
Algorithm:
- Check if any chunks have
ChunkType == "wiki_page" - Verify at least one search target is a wiki KB
- Multiply wiki page scores by 1.3x
- Re-sort results
Rationale: Wiki pages are pre-synthesized and cross-referenced, making them more valuable than raw document chunks.
Agent Integration
Wiki Tools Available
| Tool | Purpose |
|---|---|
wiki_read_page |
Retrieve page content for inspection |
wiki_write_page |
Create or overwrite page (synthesis/comparison) |
wiki_delete_page |
Soft-delete page |
wiki_rename_page |
Rename slug and update references |
wiki_read_source_doc |
Verify facts against original document |
wiki_flag_issue |
Report problem on page |
wiki_read_issue |
Review flagged issues |
wiki_update_issue |
Mark issue resolved/ignored |
Common Agent Workflows
Creating a Synthesis Page:
1. Agent analyzes multiple documents
2. Calls wiki_write_page with:
- slug: "synthesis/..."
- page_type: "synthesis"
- content: Full markdown with [[wiki-links]]
- source_refs: Knowledge IDs of documents analyzed
3. Tool parses links from content
4. Tool calls InjectCrossLinks() (auto-link mentions)
5. Tool calls RebuildIndexPage() (add to index)
6. Page now appears in search and graph
API Endpoints
Base: /api/v1/knowledgebase/{kb_id}/wiki
| Method | Path | Purpose |
|---|---|---|
| GET | /pages |
List with pagination/filtering |
| POST | /pages |
Create page |
| GET | /pages/{slug} |
Get single page |
| PUT | /pages/{slug} |
Update page |
| DELETE | /pages/{slug} |
Soft-delete |
| GET | /index |
Get index (auto-create) |
| GET | /log |
Get log (auto-create) |
| GET | /graph |
Link graph for visualization |
| GET | /stats |
Aggregate statistics |
| GET | /lint |
Health check |
| POST | /rebuild-links |
Force rebuild |
| GET | /search |
Full-text search |
Configuration
KB-Level Settings
{
"enabled": true,
"auto_ingest": true,
"synthesis_model_id": "gpt-4-turbo",
"max_pages_per_ingest": 0
}
Updated via: PUT /api/v1/knowledgebase/{id} with wiki_config field.
Agent Access
type AgentConfig struct {
KnowledgeBases []string // KB IDs agent can access
KnowledgeIDs []string // Individual docs
// Tools available:
// - wiki_read_page, wiki_write_page, wiki_delete_page, etc.
}
Performance & Scaling
Optimization Techniques
| Technique | Benefit |
|---|---|
| Debouncing (30s) | Reduces task scheduling overhead |
| Batching (5 docs) | Amortizes LLM costs |
| Parallel processing (10 goroutines) | Utilizes multi-core efficiently |
| Batch dedup (1 LLM call) | Entities + concepts in single call |
| Pre-loading (WikiBatchContext) | All pages fetched once per batch |
| Fast path (wiki boost) | Skip work if no wiki chunks |
| Content truncation (32KB) | Limits LLM input size |
Scaling Limits
- Redis pending list: 24-hour TTL (auto-cleanup)
- Active lock: 5-minute TTL with 2-minute refresh
- Task timeout: 60 minutes
- Max retries: 10 (outlasts lock TTL)
- Sequential per KB: One batch at a time per knowledge base
Health Monitoring
Linting Issues Detected
| Issue | Severity | Fixable |
|---|---|---|
| Orphan page (no inbound links) | Warning | No |
| Broken link (target deleted) | Error | Yes |
| Empty content (<50 chars) | Warning | Yes |
| Missing cross-reference (mention not linked) | Info | No |
| Stale reference (doc deleted) | Warning | No |
Health Score
Base: 100
- Each orphan (if >50%): -25
- Each broken link: -2
- Each stale ref: -1
API: GET /api/v1/knowledgebase/{id}/wiki/lint
Common Questions
Q: Is wiki a separate KB type?
A: No. Wiki is a feature flag on document-type KBs. You enable it via WikiConfig.Enabled.
Q: What triggers wiki generation?
A: Document upload (if WikiConfig.AutoIngest = true). System calls EnqueueWikiIngest().
Q: Can I disable auto-generation?
A: Yes. Set AutoIngest = false. Manual wiki_write_page tool still works.
Q: How is wiki content indexed?
A: Wiki pages are converted to chunks with ChunkType = "wiki_page" and indexed like normal.
Q: How does wiki affect search results?
A: WikiBoost plugin increases wiki page scores by 1.3x during reranking.
Q: Can agents create wiki pages?
A: Yes. Use wiki_write_page tool. Agent decides slug, content, page type.
Q: What happens to wiki when I delete a document?
A: EnqueueWikiRetract() is called. Pages referencing the document are updated/deleted.
Q: How do cross-links work?
A: Pure text matching. System finds mentions of page titles and wraps with [[slug|title]].
Q: What's the performance impact?
A: Minimal. Debouncing/batching amortizes LLM costs. Boosting adds negligible overhead.
File Structure Overview
internal/
├── types/
│ ├── wiki_page.go # Data models
│ ├── chunk.go # ChunkType constants
│ └── interfaces/wiki_page.go # Service interfaces
├── application/
│ ├── service/
│ │ ├── wiki_page.go # WikiPageService
│ │ ├── wiki_ingest.go # EnqueueWikiIngest
│ │ ├── wiki_ingest_batch.go # ProcessWikiIngest (MAP-REDUCE)
│ │ ├── wiki_linkify.go # Cross-link injection
│ │ ├── wiki_lint.go # Health checking
│ │ ├── chat_pipeline/wiki_boost.go # Retrieval boosting
│ │ └── chunk.go # Chunk operations
│ ├── repository/
│ │ └── wiki_page.go # Database layer
│ └── handler/
│ └── wiki_page.go # HTTP endpoints
├── agent/
│ ├── tools/
│ │ ├── wiki_write_page.go # Create/update pages
│ │ ├── wiki_read_page.go # Read pages
│ │ ├── wiki_delete_page.go # Delete pages
│ │ └── ... # Other wiki tools
│ └── prompts_wiki.go # LLM prompts
└── router/router.go # Route registration
Next Steps for Developers
Understanding Specific Areas
- Page Creation Flow: Read
wiki_page.go:CreatePage()+wiki_page.go:parseOutLinks() - Document Ingestion: Read
wiki_ingest_batch.go:ProcessWikiIngest()+mapOneDocument() - Link Synthesis: Read
wiki_ingest_batch.go:reduceSlugUpdates()+ LLM prompts - Cross-linking: Read
wiki_linkify.go:linkifyContent()+computeForbiddenSpans() - Retrieval Boost: Read
chat_pipeline/wiki_boost.go+ chunk indexing flow
Common Modifications
- Change LLM prompts: Edit
internal/agent/prompts_wiki.go - Adjust batch size: Change
wikiMaxDocsPerBatchinwiki_ingest.go - Change boost factor: Modify
wikiBoostFactorinwiki_boost.go - Add issue type: Extend
WikiLintIssueTypeenum inwiki_lint.go - New agent tool: Create
internal/agent/tools/wiki_*.go+ register
References
- Full Technical Deep Dive: See
WIKI_SYSTEM_DEEP_DIVE.md(948 lines) - Architecture Analysis: See
KB_WIKI_ARCHITECTURE_ANALYSIS.md(34KB) - File Index: See
KB_WIKI_FILE_INDEX.md
End of Summary
Generated: 2026-04-20