Files
Enterprise-Knowledge-Base/WIKI_DOCS_INDEX.md
T
wizardchen 28deb8282f feat: Add comprehensive documentation for WeKnora knowledge base and wiki features
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings.
- The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities.
- Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system.

These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
2026-04-22 21:18:10 +08:00

483 lines
14 KiB
Markdown

# Wiki System Documentation Index
**Purpose:** Central reference for all wiki system documentation
**Last Updated:** April 2026
---
## Quick Navigation
### For First-Time Readers
1. Start with **WIKI_SYSTEM_SUMMARY.md** (5 min read)
- Overview of wiki functionality
- Key architectural decisions
- Common workflows
2. Then read **WIKI_SYSTEM_DEEP_DIVE.md** (15 min read)
- Detailed service descriptions
- Data flow diagrams
- Complete API endpoints
### For Developers Implementing Features
1. **WIKI_IMPLEMENTATION_GUIDE.md** (primary reference)
- Common tasks with code examples
- Testing patterns
- Error handling
- Performance optimization
- Debugging techniques
2. **WIKI_TECHNICAL_ANALYSIS.md** (deep reference)
- Architectural patterns explained
- Edge cases documented
- Operational considerations
- Known limitations
### For Architects and Code Reviewers
1. **WIKI_TECHNICAL_ANALYSIS.md** (full read recommended)
- Part 2: Core Concepts and Patterns
- Part 3: Detailed Implementation Patterns
- Part 4: Edge Cases and Gotchas
- Part 5: Operational Considerations
- Part 7: Future Enhancements
2. **Source Code** (in this order)
- `internal/types/wiki_page.go` — data models
- `internal/application/service/wiki_page.go` — core service
- `internal/application/service/wiki_ingest.go` — async pipeline
- `internal/application/service/wiki_ingest_batch.go` — MAP-REDUCE handler
- `internal/application/service/wiki_linkify.go` — cross-link injection
- `internal/application/service/wiki_lint.go` — health checking
---
## Documentation Overview
### WIKI_SYSTEM_SUMMARY.md
**Type:** Executive Summary
**Length:** ~470 lines
**Audience:** Everyone
**Key Sections:**
- What is wiki?
- Quick architecture map
- Core data structures
- Key services overview
- Async pipeline simplified
- Cross-link injection overview
- Retrieval integration
- Agent integration
- API endpoints quick reference
- Configuration overview
- FAQ
**When to Read:** First introduction to wiki system
---
### WIKI_SYSTEM_DEEP_DIVE.md
**Type:** Technical Reference
**Length:** ~950 lines
**Audience:** Developers, architects
**Key Sections:**
- Complete system overview
- Core data models with full struct definitions
- Service architecture with dependency graphs
- Exhaustive async pipeline documentation
- Cross-link injection algorithm details
- Wiki chunking and retrieval integration
- Agent integration with workflows
- Complete API endpoints table
- Configuration and error handling
- Performance optimization
- Common workflows and FAQ
- File structure appendix
**When to Read:** When you need comprehensive understanding of a subsystem
---
### WIKI_TECHNICAL_ANALYSIS.md
**Type:** Deep Technical Analysis
**Length:** ~1,170 lines
**Audience:** Architects, advanced developers
**Key Sections:**
- **Part 1:** Architectural Overview (1.1-1.3)
- System context and purpose
- Wiki as feature flag (key design decision)
- Core data flow diagram
- Key files and responsibilities
- **Part 2:** Core Concepts and Patterns (2.1-2.5)
- Bidirectional link references
- Optimistic locking with version numbers
- Async batch processing with debouncing
- MAP-REDUCE pattern for extraction
- Cross-link injection with forbidden spans
- **Part 3:** Detailed Implementation Patterns (3.1-3.4)
- Repository layer: optimistic locking
- Service layer: multi-step operations
- Async pipeline: WikiIngestService
- Linting and health monitoring
- **Part 4:** Edge Cases and Gotchas (4.1-4.9)
- Concurrent update conflicts
- Link consistency windows
- Self-link prevention
- LLM conflict detection
- Slug reuse across updates
- CJK text handling
- Forbidden span edge cases
- Redis timing issues
- LLM parsing failures
- **Part 5:** Operational Considerations (5.1-5.4)
- Performance scaling
- Storage considerations
- Monitoring and debugging
- Common pitfalls and solutions
- **Part 6:** Integration Points (6.1-6.3)
- Chat pipeline integration
- Agent tool integration
- Chunk synchronization
- **Part 7:** Future Enhancements (7.1-7.2)
- Potential improvements
- Known limitations
**When to Read:** When designing new features or debugging complex issues
---
### WIKI_IMPLEMENTATION_GUIDE.md
**Type:** Practical How-To Guide
**Length:** ~750 lines
**Audience:** Developers implementing wiki features
**Key Sections:**
- **Part 1:** Common Tasks and Patterns
- Task 1: Create a new wiki page programmatically
- Task 2: Update an existing page
- Task 3: Work with links programmatically
- Task 4: Iterate over all pages efficiently
- Task 5: Work with cross-link injection
- **Part 2:** Testing Patterns
- Unit tests
- Integration tests
- **Part 3:** Error Handling
- ErrWikiPageNotFound
- ErrWikiPageConflict
- LLM extraction failures
- **Part 4:** Performance Optimization
- Batch creation
- Query optimization
- Link rebuild efficiency
- **Part 5:** Debugging and Troubleshooting
- Check page consistency
- Monitor wiki health
- **Part 6:** Common Mistakes
- 10 common mistakes with fixes
- **Appendix A:** Key functions reference
- **Appendix B:** Configuration
- **Appendix C:** SQL queries for debugging
**When to Read:** Before implementing any wiki feature
---
### From Previous Sessions: Supporting Documentation
#### KB_WIKI_ARCHITECTURE_ANALYSIS.md
**Type:** Initial architecture analysis
**Length:** ~750 lines
**Key Content:**
- High-level system design
- Data model analysis
- Service relationships
- Batch processing flow
- Integration points
#### KB_WIKI_QUICK_REFERENCE.md
**Type:** Quick reference
**Length:** ~300 lines
**Key Content:**
- Quick navigation to common tasks
- API endpoint quick list
- Configuration quick reference
- Common errors quick lookup
#### KB_WIKI_FILE_INDEX.md
**Type:** File directory and location reference
**Length:** ~250 lines
**Key Content:**
- All wiki-related files with locations
- File purposes and responsibilities
- Cross-references
- Updated file counts and statistics
---
## Source Code Quick Reference
### Core Data Models
**File:** `internal/types/wiki_page.go`
- `WikiPage` struct (84 lines)
- `WikiConfig` struct (12 lines)
- `WikiPageType` constants
- `WikiPageStatus` constants
- `WikiStats`, `WikiGraphNode`, `WikiGraphEdge` structs
- `WikiPageIssue` struct
### Service Layer
**File:** `internal/application/service/wiki_page.go`
- `WikiPageService` implementation
- CreatePage, UpdatePage, UpdatePageMeta
- GetPageBySlug, GetPageByID, ListPages
- DeletePage, RebuildLinks, InjectCrossLinks
- Helper functions: parseOutLinks, normalizeSlug, updateInLinks, removeInLinks
**File:** `internal/application/service/wiki_lint.go`
- `WikiLintService` implementation
- RunLint (comprehensive health check)
- AutoFix (automatic repair)
### Async Pipeline
**File:** `internal/application/service/wiki_ingest.go`
- `WikiIngestService` implementation
- EnqueueWikiIngest, EnqueueWikiRetract
- Debouncing mechanism
- Redis queue management
**File:** `internal/application/service/wiki_ingest_batch.go`
- ProcessWikiIngest handler (MAP-REDUCE)
- mapOneDocument (extraction)
- reduceSlugUpdates (synthesis)
- Post-processing steps
### Text Processing
**File:** `internal/application/service/wiki_linkify.go`
- linkifyContent (main algorithm)
- computeForbiddenSpans (markdown structure detection)
- findFirstSafeMatch (safe occurrence finding)
- hasWordBoundary, isASCIIWordRune (boundary checking)
- Comprehensive forbidden span detection
### Repository Layer
**File:** `internal/application/repository/wiki_page.go`
- `WikiPageRepository` implementation
- Create, Update, UpdateMeta operations
- GetByID, GetBySlug queries
- List with filtering and pagination
- Optimistic locking implementation
### HTTP Handlers
**File:** `internal/handler/wiki_page.go`
- `WikiPageHandler` (HTTP endpoints)
- ListPages, CreatePage, GetPage, UpdatePage, DeletePage
- GetGraph, GetStats, RebuildLinks, RebuildIndexPage
- Health checking and stats endpoints
### Integration Plugins
**File:** `internal/application/service/chat_pipeline/wiki_boost.go`
- `PluginWikiBoost` retrieval scoring plugin
- 1.3x score multiplier for wiki pages
- Ranks wiki content higher in search results
### Agent Tools
**File:** `internal/agent/tools/wiki_write_page.go`
- `wikiWritePageTool` (create/update pages from agent)
- Parameters: slug, title, summary, content, page_type
**File:** `internal/agent/tools/wiki_delete_page.go`
- `wikiDeletePageTool` (delete pages from agent)
**File:** `internal/agent/tools/wiki_replace_text.go`
- `wikiReplaceTextTool` (find/replace within page)
### LLM Prompts
**File:** `internal/agent/prompts_wiki.go`
- `WikiSummaryPrompt` (summary generation)
- `WikiKnowledgeExtractPrompt` (entity/concept extraction)
- `WikiPageModifyPrompt` (page updating)
---
## Cross-Document References
### By Topic
**Understanding Concurrency:**
1. WIKI_TECHNICAL_ANALYSIS.md → Part 4.1 (Concurrent update conflicts)
2. WIKI_IMPLEMENTATION_GUIDE.md → Part 3 (Error handling)
3. wiki_page.go (repo) → Update method (lines 36-60)
**Understanding Links:**
1. WIKI_TECHNICAL_ANALYSIS.md → Part 2.1 (Bidirectional links)
2. wiki_page.go (service) → updateInLinks, removeInLinks methods
3. WIKI_IMPLEMENTATION_GUIDE.md → Task 3 (Work with links)
**Understanding Async Processing:**
1. WIKI_TECHNICAL_ANALYSIS.md → Part 2.3 & 2.4 (Debouncing and MAP-REDUCE)
2. WIKI_SYSTEM_DEEP_DIVE.md → Async pipeline section
3. wiki_ingest.go → EnqueueWikiIngest method
4. wiki_ingest_batch.go → ProcessWikiIngest handler
**Understanding Cross-Link Injection:**
1. WIKI_TECHNICAL_ANALYSIS.md → Part 2.5 (Forbidden spans)
2. WIKI_IMPLEMENTATION_GUIDE.md → Task 5 (Cross-link injection)
3. wiki_linkify.go → Full implementation
4. wiki_linkify_test.go → Test cases (coverage)
**Understanding Health Checking:**
1. WIKI_TECHNICAL_ANALYSIS.md → Part 3.4 (Linting)
2. wiki_lint.go → Full implementation
3. WIKI_IMPLEMENTATION_GUIDE.md → Part 5 (Monitoring)
---
## Key Diagrams and Flows
### Data Flow (ASCII)
```
Document Upload
↓
EnqueueWikiIngest → Redis List (30s debounce)
↓
ProcessWikiIngest [Async]
├─ MAP: Extract entities/concepts (parallel)
├─ REDUCE: Synthesize pages (parallel)
├─ Post-processing: Links, index, cross-link injection
└─ Schedule follow-up if pending remain
↓
Wiki Pages in DB
├─ Synchronized to chunks (ChunkType="wiki_page")
├─ Integrated in retrieval pipeline
└─ Boosted 1.3x in chat search
↓
Chat Integration
└─ Higher-ranked results → better answers
```
### Service Dependencies
```
WikiPageService
├─ WikiPageRepository (data access)
├─ ChunkRepository (sync to retrieval)
├─ KnowledgeBaseService (KB validation)
└─ Redis (pending queue, locking)
WikiIngestService
├─ WikiPageService (create/update pages)
├─ KnowledgeBaseService (KB config)
├─ KnowledgeService (fetch document chunks)
├─ ChunkRepository (sync results)
├─ ModelService (LLM calls)
├─ Task queue (Asynq)
└─ Redis (debouncing, locking)
WikiLintService
├─ WikiPageService (get pages, graph)
├─ KnowledgeBaseService (KB validation)
└─ No database write access (read-only)
```
### State Transitions
```
WikiPage States:
draft ──→ published ──→ archived
↑
└─── (UpdatePage increments version)
WikiPageIssue States:
pending ──→ acknowledged ──→ resolved
Page Creation Flow:
1. Create page + parse OutLinks
2. Insert to DB
3. Update target InLinks (async, fire-and-forget)
4. Return to caller
Page Update Flow:
1. Fetch existing (get current version)
2. Update fields + re-parse OutLinks
3. Optimistic lock check (WHERE version = X)
4. Update InLinks on old targets (remove)
5. Update InLinks on new targets (add)
6. Return updated page or ErrWikiPageConflict
```
---
## Decision Matrix: Which Document to Read
| Question | Document |
|----------|----------|
| What is wiki and why do we have it? | WIKI_SYSTEM_SUMMARY.md |
| How do I create a new wiki page? | WIKI_IMPLEMENTATION_GUIDE.md (Task 1) |
| How does async processing work? | WIKI_TECHNICAL_ANALYSIS.md (Part 2.3-2.4) |
| What are the edge cases I should know about? | WIKI_TECHNICAL_ANALYSIS.md (Part 4) |
| How do I test my wiki feature? | WIKI_IMPLEMENTATION_GUIDE.md (Part 2) |
| What's the complete API? | WIKI_SYSTEM_DEEP_DIVE.md |
| How do I debug a broken link? | WIKI_IMPLEMENTATION_GUIDE.md (Part 5) |
| What's the data model? | internal/types/wiki_page.go |
| How does link injection work? | WIKI_TECHNICAL_ANALYSIS.md (Part 2.5) |
| How do I handle version conflicts? | WIKI_IMPLEMENTATION_GUIDE.md (Part 3) |
| What performance considerations exist? | WIKI_TECHNICAL_ANALYSIS.md (Part 5.1) |
| How is wiki integrated with chat? | WIKI_TECHNICAL_ANALYSIS.md (Part 6.1) |
| What are the LLM prompts? | internal/agent/prompts_wiki.go |
| How do I monitor wiki health? | WIKI_IMPLEMENTATION_GUIDE.md (Part 5) + wiki_lint.go |
| What's the full system architecture? | WIKI_TECHNICAL_ANALYSIS.md (Part 1) |
---
## Maintenance and Updates
This documentation set should be updated when:
- New API endpoints are added or modified
- Database schema changes
- LLM prompt templates change significantly
- Performance characteristics change
- New edge cases discovered
- Configuration options change
- Integration points change
**To update documentation:**
1. Identify which docs are affected
2. Update in order: code first, then summaries
3. Regenerate any tables or metrics
4. Update cross-references
5. Review for accuracy and clarity
---
## Feedback and Questions
If documentation is:
- **Unclear:** Note the specific section and what's confusing
- **Incomplete:** Note what's missing
- **Outdated:** Note what's changed in the code
- **Inaccurate:** Note what's wrong and what's correct
Post improvements as pull requests with clear descriptions.
---
## Statistics
| Metric | Value |
|--------|-------|
| Total documentation | ~3,900 lines |
| Source code (wiki only) | ~7,300 lines |
| Total wiki-related files | ~20 files |
| Test coverage | ~285 lines (wiki_linkify_test.go + wiki_page_test.go) |
| LLM prompts | 3 major templates |
| Database tables | 2 (wiki_pages, wiki_page_issues) |
| Key services | 3 (WikiPageService, WikiIngestService, WikiLintService) |
| API endpoints | 20+ (see WIKI_SYSTEM_DEEP_DIVE.md) |