- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings. - The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities. - Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system. These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
14 KiB
Wiki System Documentation Index
Purpose: Central reference for all wiki system documentation
Last Updated: April 2026
Quick Navigation
For First-Time Readers
-
Start with WIKI_SYSTEM_SUMMARY.md (5 min read)
- Overview of wiki functionality
- Key architectural decisions
- Common workflows
-
Then read WIKI_SYSTEM_DEEP_DIVE.md (15 min read)
- Detailed service descriptions
- Data flow diagrams
- Complete API endpoints
For Developers Implementing Features
-
WIKI_IMPLEMENTATION_GUIDE.md (primary reference)
- Common tasks with code examples
- Testing patterns
- Error handling
- Performance optimization
- Debugging techniques
-
WIKI_TECHNICAL_ANALYSIS.md (deep reference)
- Architectural patterns explained
- Edge cases documented
- Operational considerations
- Known limitations
For Architects and Code Reviewers
-
WIKI_TECHNICAL_ANALYSIS.md (full read recommended)
- Part 2: Core Concepts and Patterns
- Part 3: Detailed Implementation Patterns
- Part 4: Edge Cases and Gotchas
- Part 5: Operational Considerations
- Part 7: Future Enhancements
-
Source Code (in this order)
internal/types/wiki_page.go— data modelsinternal/application/service/wiki_page.go— core serviceinternal/application/service/wiki_ingest.go— async pipelineinternal/application/service/wiki_ingest_batch.go— MAP-REDUCE handlerinternal/application/service/wiki_linkify.go— cross-link injectioninternal/application/service/wiki_lint.go— health checking
Documentation Overview
WIKI_SYSTEM_SUMMARY.md
Type: Executive Summary
Length: ~470 lines
Audience: Everyone
Key Sections:
- What is wiki?
- Quick architecture map
- Core data structures
- Key services overview
- Async pipeline simplified
- Cross-link injection overview
- Retrieval integration
- Agent integration
- API endpoints quick reference
- Configuration overview
- FAQ
When to Read: First introduction to wiki system
WIKI_SYSTEM_DEEP_DIVE.md
Type: Technical Reference
Length: ~950 lines
Audience: Developers, architects
Key Sections:
- Complete system overview
- Core data models with full struct definitions
- Service architecture with dependency graphs
- Exhaustive async pipeline documentation
- Cross-link injection algorithm details
- Wiki chunking and retrieval integration
- Agent integration with workflows
- Complete API endpoints table
- Configuration and error handling
- Performance optimization
- Common workflows and FAQ
- File structure appendix
When to Read: When you need comprehensive understanding of a subsystem
WIKI_TECHNICAL_ANALYSIS.md
Type: Deep Technical Analysis
Length: ~1,170 lines
Audience: Architects, advanced developers
Key Sections:
-
Part 1: Architectural Overview (1.1-1.3)
- System context and purpose
- Wiki as feature flag (key design decision)
- Core data flow diagram
- Key files and responsibilities
-
Part 2: Core Concepts and Patterns (2.1-2.5)
- Bidirectional link references
- Optimistic locking with version numbers
- Async batch processing with debouncing
- MAP-REDUCE pattern for extraction
- Cross-link injection with forbidden spans
-
Part 3: Detailed Implementation Patterns (3.1-3.4)
- Repository layer: optimistic locking
- Service layer: multi-step operations
- Async pipeline: WikiIngestService
- Linting and health monitoring
-
Part 4: Edge Cases and Gotchas (4.1-4.9)
- Concurrent update conflicts
- Link consistency windows
- Self-link prevention
- LLM conflict detection
- Slug reuse across updates
- CJK text handling
- Forbidden span edge cases
- Redis timing issues
- LLM parsing failures
-
Part 5: Operational Considerations (5.1-5.4)
- Performance scaling
- Storage considerations
- Monitoring and debugging
- Common pitfalls and solutions
-
Part 6: Integration Points (6.1-6.3)
- Chat pipeline integration
- Agent tool integration
- Chunk synchronization
-
Part 7: Future Enhancements (7.1-7.2)
- Potential improvements
- Known limitations
When to Read: When designing new features or debugging complex issues
WIKI_IMPLEMENTATION_GUIDE.md
Type: Practical How-To Guide
Length: ~750 lines
Audience: Developers implementing wiki features
Key Sections:
-
Part 1: Common Tasks and Patterns
- Task 1: Create a new wiki page programmatically
- Task 2: Update an existing page
- Task 3: Work with links programmatically
- Task 4: Iterate over all pages efficiently
- Task 5: Work with cross-link injection
-
Part 2: Testing Patterns
- Unit tests
- Integration tests
-
Part 3: Error Handling
- ErrWikiPageNotFound
- ErrWikiPageConflict
- LLM extraction failures
-
Part 4: Performance Optimization
- Batch creation
- Query optimization
- Link rebuild efficiency
-
Part 5: Debugging and Troubleshooting
- Check page consistency
- Monitor wiki health
-
Part 6: Common Mistakes
- 10 common mistakes with fixes
-
Appendix A: Key functions reference
-
Appendix B: Configuration
-
Appendix C: SQL queries for debugging
When to Read: Before implementing any wiki feature
From Previous Sessions: Supporting Documentation
KB_WIKI_ARCHITECTURE_ANALYSIS.md
Type: Initial architecture analysis
Length: ~750 lines
Key Content:
- High-level system design
- Data model analysis
- Service relationships
- Batch processing flow
- Integration points
KB_WIKI_QUICK_REFERENCE.md
Type: Quick reference
Length: ~300 lines
Key Content:
- Quick navigation to common tasks
- API endpoint quick list
- Configuration quick reference
- Common errors quick lookup
KB_WIKI_FILE_INDEX.md
Type: File directory and location reference
Length: ~250 lines
Key Content:
- All wiki-related files with locations
- File purposes and responsibilities
- Cross-references
- Updated file counts and statistics
Source Code Quick Reference
Core Data Models
File: internal/types/wiki_page.go
WikiPagestruct (84 lines)WikiConfigstruct (12 lines)WikiPageTypeconstantsWikiPageStatusconstantsWikiStats,WikiGraphNode,WikiGraphEdgestructsWikiPageIssuestruct
Service Layer
File: internal/application/service/wiki_page.go
WikiPageServiceimplementation- CreatePage, UpdatePage, UpdatePageMeta
- GetPageBySlug, GetPageByID, ListPages
- DeletePage, RebuildLinks, InjectCrossLinks
- Helper functions: parseOutLinks, normalizeSlug, updateInLinks, removeInLinks
File: internal/application/service/wiki_lint.go
WikiLintServiceimplementation- RunLint (comprehensive health check)
- AutoFix (automatic repair)
Async Pipeline
File: internal/application/service/wiki_ingest.go
WikiIngestServiceimplementation- EnqueueWikiIngest, EnqueueWikiRetract
- Debouncing mechanism
- Redis queue management
File: internal/application/service/wiki_ingest_batch.go
- ProcessWikiIngest handler (MAP-REDUCE)
- mapOneDocument (extraction)
- reduceSlugUpdates (synthesis)
- Post-processing steps
Text Processing
File: internal/application/service/wiki_linkify.go
- linkifyContent (main algorithm)
- computeForbiddenSpans (markdown structure detection)
- findFirstSafeMatch (safe occurrence finding)
- hasWordBoundary, isASCIIWordRune (boundary checking)
- Comprehensive forbidden span detection
Repository Layer
File: internal/application/repository/wiki_page.go
WikiPageRepositoryimplementation- Create, Update, UpdateMeta operations
- GetByID, GetBySlug queries
- List with filtering and pagination
- Optimistic locking implementation
HTTP Handlers
File: internal/handler/wiki_page.go
WikiPageHandler(HTTP endpoints)- ListPages, CreatePage, GetPage, UpdatePage, DeletePage
- GetGraph, GetStats, RebuildLinks, RebuildIndexPage
- Health checking and stats endpoints
Integration Plugins
File: internal/application/service/chat_pipeline/wiki_boost.go
PluginWikiBoostretrieval scoring plugin- 1.3x score multiplier for wiki pages
- Ranks wiki content higher in search results
Agent Tools
File: internal/agent/tools/wiki_write_page.go
wikiWritePageTool(create/update pages from agent)- Parameters: slug, title, summary, content, page_type
File: internal/agent/tools/wiki_delete_page.go
wikiDeletePageTool(delete pages from agent)
File: internal/agent/tools/wiki_replace_text.go
wikiReplaceTextTool(find/replace within page)
LLM Prompts
File: internal/agent/prompts_wiki.go
WikiSummaryPrompt(summary generation)WikiKnowledgeExtractPrompt(entity/concept extraction)WikiPageModifyPrompt(page updating)
Cross-Document References
By Topic
Understanding Concurrency:
- WIKI_TECHNICAL_ANALYSIS.md → Part 4.1 (Concurrent update conflicts)
- WIKI_IMPLEMENTATION_GUIDE.md → Part 3 (Error handling)
- wiki_page.go (repo) → Update method (lines 36-60)
Understanding Links:
- WIKI_TECHNICAL_ANALYSIS.md → Part 2.1 (Bidirectional links)
- wiki_page.go (service) → updateInLinks, removeInLinks methods
- WIKI_IMPLEMENTATION_GUIDE.md → Task 3 (Work with links)
Understanding Async Processing:
- WIKI_TECHNICAL_ANALYSIS.md → Part 2.3 & 2.4 (Debouncing and MAP-REDUCE)
- WIKI_SYSTEM_DEEP_DIVE.md → Async pipeline section
- wiki_ingest.go → EnqueueWikiIngest method
- wiki_ingest_batch.go → ProcessWikiIngest handler
Understanding Cross-Link Injection:
- WIKI_TECHNICAL_ANALYSIS.md → Part 2.5 (Forbidden spans)
- WIKI_IMPLEMENTATION_GUIDE.md → Task 5 (Cross-link injection)
- wiki_linkify.go → Full implementation
- wiki_linkify_test.go → Test cases (coverage)
Understanding Health Checking:
- WIKI_TECHNICAL_ANALYSIS.md → Part 3.4 (Linting)
- wiki_lint.go → Full implementation
- WIKI_IMPLEMENTATION_GUIDE.md → Part 5 (Monitoring)
Key Diagrams and Flows
Data Flow (ASCII)
Document Upload
↓
EnqueueWikiIngest → Redis List (30s debounce)
↓
ProcessWikiIngest [Async]
├─ MAP: Extract entities/concepts (parallel)
├─ REDUCE: Synthesize pages (parallel)
├─ Post-processing: Links, index, cross-link injection
└─ Schedule follow-up if pending remain
↓
Wiki Pages in DB
├─ Synchronized to chunks (ChunkType="wiki_page")
├─ Integrated in retrieval pipeline
└─ Boosted 1.3x in chat search
↓
Chat Integration
└─ Higher-ranked results → better answers
Service Dependencies
WikiPageService
├─ WikiPageRepository (data access)
├─ ChunkRepository (sync to retrieval)
├─ KnowledgeBaseService (KB validation)
└─ Redis (pending queue, locking)
WikiIngestService
├─ WikiPageService (create/update pages)
├─ KnowledgeBaseService (KB config)
├─ KnowledgeService (fetch document chunks)
├─ ChunkRepository (sync results)
├─ ModelService (LLM calls)
├─ Task queue (Asynq)
└─ Redis (debouncing, locking)
WikiLintService
├─ WikiPageService (get pages, graph)
├─ KnowledgeBaseService (KB validation)
└─ No database write access (read-only)
State Transitions
WikiPage States:
draft ──→ published ──→ archived
↑
└─── (UpdatePage increments version)
WikiPageIssue States:
pending ──→ acknowledged ──→ resolved
Page Creation Flow:
1. Create page + parse OutLinks
2. Insert to DB
3. Update target InLinks (async, fire-and-forget)
4. Return to caller
Page Update Flow:
1. Fetch existing (get current version)
2. Update fields + re-parse OutLinks
3. Optimistic lock check (WHERE version = X)
4. Update InLinks on old targets (remove)
5. Update InLinks on new targets (add)
6. Return updated page or ErrWikiPageConflict
Decision Matrix: Which Document to Read
| Question | Document |
|---|---|
| What is wiki and why do we have it? | WIKI_SYSTEM_SUMMARY.md |
| How do I create a new wiki page? | WIKI_IMPLEMENTATION_GUIDE.md (Task 1) |
| How does async processing work? | WIKI_TECHNICAL_ANALYSIS.md (Part 2.3-2.4) |
| What are the edge cases I should know about? | WIKI_TECHNICAL_ANALYSIS.md (Part 4) |
| How do I test my wiki feature? | WIKI_IMPLEMENTATION_GUIDE.md (Part 2) |
| What's the complete API? | WIKI_SYSTEM_DEEP_DIVE.md |
| How do I debug a broken link? | WIKI_IMPLEMENTATION_GUIDE.md (Part 5) |
| What's the data model? | internal/types/wiki_page.go |
| How does link injection work? | WIKI_TECHNICAL_ANALYSIS.md (Part 2.5) |
| How do I handle version conflicts? | WIKI_IMPLEMENTATION_GUIDE.md (Part 3) |
| What performance considerations exist? | WIKI_TECHNICAL_ANALYSIS.md (Part 5.1) |
| How is wiki integrated with chat? | WIKI_TECHNICAL_ANALYSIS.md (Part 6.1) |
| What are the LLM prompts? | internal/agent/prompts_wiki.go |
| How do I monitor wiki health? | WIKI_IMPLEMENTATION_GUIDE.md (Part 5) + wiki_lint.go |
| What's the full system architecture? | WIKI_TECHNICAL_ANALYSIS.md (Part 1) |
Maintenance and Updates
This documentation set should be updated when:
- New API endpoints are added or modified
- Database schema changes
- LLM prompt templates change significantly
- Performance characteristics change
- New edge cases discovered
- Configuration options change
- Integration points change
To update documentation:
- Identify which docs are affected
- Update in order: code first, then summaries
- Regenerate any tables or metrics
- Update cross-references
- Review for accuracy and clarity
Feedback and Questions
If documentation is:
- Unclear: Note the specific section and what's confusing
- Incomplete: Note what's missing
- Outdated: Note what's changed in the code
- Inaccurate: Note what's wrong and what's correct
Post improvements as pull requests with clear descriptions.
Statistics
| Metric | Value |
|---|---|
| Total documentation | ~3,900 lines |
| Source code (wiki only) | ~7,300 lines |
| Total wiki-related files | ~20 files |
| Test coverage | ~285 lines (wiki_linkify_test.go + wiki_page_test.go) |
| LLM prompts | 3 major templates |
| Database tables | 2 (wiki_pages, wiki_page_issues) |
| Key services | 3 (WikiPageService, WikiIngestService, WikiLintService) |
| API endpoints | 20+ (see WIKI_SYSTEM_DEEP_DIVE.md) |