Files
Enterprise-Knowledge-Base/WIKI_DOCS_INDEX.md
T
wizardchen 28deb8282f feat: Add comprehensive documentation for WeKnora knowledge base and wiki features
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings.
- The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities.
- Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system.

These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
2026-04-22 21:18:10 +08:00

14 KiB

Wiki System Documentation Index

Purpose: Central reference for all wiki system documentation
Last Updated: April 2026


Quick Navigation

For First-Time Readers

  1. Start with WIKI_SYSTEM_SUMMARY.md (5 min read)

    • Overview of wiki functionality
    • Key architectural decisions
    • Common workflows
  2. Then read WIKI_SYSTEM_DEEP_DIVE.md (15 min read)

    • Detailed service descriptions
    • Data flow diagrams
    • Complete API endpoints

For Developers Implementing Features

  1. WIKI_IMPLEMENTATION_GUIDE.md (primary reference)

    • Common tasks with code examples
    • Testing patterns
    • Error handling
    • Performance optimization
    • Debugging techniques
  2. WIKI_TECHNICAL_ANALYSIS.md (deep reference)

    • Architectural patterns explained
    • Edge cases documented
    • Operational considerations
    • Known limitations

For Architects and Code Reviewers

  1. WIKI_TECHNICAL_ANALYSIS.md (full read recommended)

    • Part 2: Core Concepts and Patterns
    • Part 3: Detailed Implementation Patterns
    • Part 4: Edge Cases and Gotchas
    • Part 5: Operational Considerations
    • Part 7: Future Enhancements
  2. Source Code (in this order)

    • internal/types/wiki_page.go — data models
    • internal/application/service/wiki_page.go — core service
    • internal/application/service/wiki_ingest.go — async pipeline
    • internal/application/service/wiki_ingest_batch.go — MAP-REDUCE handler
    • internal/application/service/wiki_linkify.go — cross-link injection
    • internal/application/service/wiki_lint.go — health checking

Documentation Overview

WIKI_SYSTEM_SUMMARY.md

Type: Executive Summary
Length: ~470 lines
Audience: Everyone
Key Sections:

  • What is wiki?
  • Quick architecture map
  • Core data structures
  • Key services overview
  • Async pipeline simplified
  • Cross-link injection overview
  • Retrieval integration
  • Agent integration
  • API endpoints quick reference
  • Configuration overview
  • FAQ

When to Read: First introduction to wiki system


WIKI_SYSTEM_DEEP_DIVE.md

Type: Technical Reference
Length: ~950 lines
Audience: Developers, architects
Key Sections:

  • Complete system overview
  • Core data models with full struct definitions
  • Service architecture with dependency graphs
  • Exhaustive async pipeline documentation
  • Cross-link injection algorithm details
  • Wiki chunking and retrieval integration
  • Agent integration with workflows
  • Complete API endpoints table
  • Configuration and error handling
  • Performance optimization
  • Common workflows and FAQ
  • File structure appendix

When to Read: When you need comprehensive understanding of a subsystem


WIKI_TECHNICAL_ANALYSIS.md

Type: Deep Technical Analysis
Length: ~1,170 lines
Audience: Architects, advanced developers
Key Sections:

  • Part 1: Architectural Overview (1.1-1.3)

    • System context and purpose
    • Wiki as feature flag (key design decision)
    • Core data flow diagram
    • Key files and responsibilities
  • Part 2: Core Concepts and Patterns (2.1-2.5)

    • Bidirectional link references
    • Optimistic locking with version numbers
    • Async batch processing with debouncing
    • MAP-REDUCE pattern for extraction
    • Cross-link injection with forbidden spans
  • Part 3: Detailed Implementation Patterns (3.1-3.4)

    • Repository layer: optimistic locking
    • Service layer: multi-step operations
    • Async pipeline: WikiIngestService
    • Linting and health monitoring
  • Part 4: Edge Cases and Gotchas (4.1-4.9)

    • Concurrent update conflicts
    • Link consistency windows
    • Self-link prevention
    • LLM conflict detection
    • Slug reuse across updates
    • CJK text handling
    • Forbidden span edge cases
    • Redis timing issues
    • LLM parsing failures
  • Part 5: Operational Considerations (5.1-5.4)

    • Performance scaling
    • Storage considerations
    • Monitoring and debugging
    • Common pitfalls and solutions
  • Part 6: Integration Points (6.1-6.3)

    • Chat pipeline integration
    • Agent tool integration
    • Chunk synchronization
  • Part 7: Future Enhancements (7.1-7.2)

    • Potential improvements
    • Known limitations

When to Read: When designing new features or debugging complex issues


WIKI_IMPLEMENTATION_GUIDE.md

Type: Practical How-To Guide
Length: ~750 lines
Audience: Developers implementing wiki features
Key Sections:

  • Part 1: Common Tasks and Patterns

    • Task 1: Create a new wiki page programmatically
    • Task 2: Update an existing page
    • Task 3: Work with links programmatically
    • Task 4: Iterate over all pages efficiently
    • Task 5: Work with cross-link injection
  • Part 2: Testing Patterns

    • Unit tests
    • Integration tests
  • Part 3: Error Handling

    • ErrWikiPageNotFound
    • ErrWikiPageConflict
    • LLM extraction failures
  • Part 4: Performance Optimization

    • Batch creation
    • Query optimization
    • Link rebuild efficiency
  • Part 5: Debugging and Troubleshooting

    • Check page consistency
    • Monitor wiki health
  • Part 6: Common Mistakes

    • 10 common mistakes with fixes
  • Appendix A: Key functions reference

  • Appendix B: Configuration

  • Appendix C: SQL queries for debugging

When to Read: Before implementing any wiki feature


From Previous Sessions: Supporting Documentation

KB_WIKI_ARCHITECTURE_ANALYSIS.md

Type: Initial architecture analysis
Length: ~750 lines
Key Content:

  • High-level system design
  • Data model analysis
  • Service relationships
  • Batch processing flow
  • Integration points

KB_WIKI_QUICK_REFERENCE.md

Type: Quick reference
Length: ~300 lines
Key Content:

  • Quick navigation to common tasks
  • API endpoint quick list
  • Configuration quick reference
  • Common errors quick lookup

KB_WIKI_FILE_INDEX.md

Type: File directory and location reference
Length: ~250 lines
Key Content:

  • All wiki-related files with locations
  • File purposes and responsibilities
  • Cross-references
  • Updated file counts and statistics

Source Code Quick Reference

Core Data Models

File: internal/types/wiki_page.go

  • WikiPage struct (84 lines)
  • WikiConfig struct (12 lines)
  • WikiPageType constants
  • WikiPageStatus constants
  • WikiStats, WikiGraphNode, WikiGraphEdge structs
  • WikiPageIssue struct

Service Layer

File: internal/application/service/wiki_page.go

  • WikiPageService implementation
  • CreatePage, UpdatePage, UpdatePageMeta
  • GetPageBySlug, GetPageByID, ListPages
  • DeletePage, RebuildLinks, InjectCrossLinks
  • Helper functions: parseOutLinks, normalizeSlug, updateInLinks, removeInLinks

File: internal/application/service/wiki_lint.go

  • WikiLintService implementation
  • RunLint (comprehensive health check)
  • AutoFix (automatic repair)

Async Pipeline

File: internal/application/service/wiki_ingest.go

  • WikiIngestService implementation
  • EnqueueWikiIngest, EnqueueWikiRetract
  • Debouncing mechanism
  • Redis queue management

File: internal/application/service/wiki_ingest_batch.go

  • ProcessWikiIngest handler (MAP-REDUCE)
  • mapOneDocument (extraction)
  • reduceSlugUpdates (synthesis)
  • Post-processing steps

Text Processing

File: internal/application/service/wiki_linkify.go

  • linkifyContent (main algorithm)
  • computeForbiddenSpans (markdown structure detection)
  • findFirstSafeMatch (safe occurrence finding)
  • hasWordBoundary, isASCIIWordRune (boundary checking)
  • Comprehensive forbidden span detection

Repository Layer

File: internal/application/repository/wiki_page.go

  • WikiPageRepository implementation
  • Create, Update, UpdateMeta operations
  • GetByID, GetBySlug queries
  • List with filtering and pagination
  • Optimistic locking implementation

HTTP Handlers

File: internal/handler/wiki_page.go

  • WikiPageHandler (HTTP endpoints)
  • ListPages, CreatePage, GetPage, UpdatePage, DeletePage
  • GetGraph, GetStats, RebuildLinks, RebuildIndexPage
  • Health checking and stats endpoints

Integration Plugins

File: internal/application/service/chat_pipeline/wiki_boost.go

  • PluginWikiBoost retrieval scoring plugin
  • 1.3x score multiplier for wiki pages
  • Ranks wiki content higher in search results

Agent Tools

File: internal/agent/tools/wiki_write_page.go

  • wikiWritePageTool (create/update pages from agent)
  • Parameters: slug, title, summary, content, page_type

File: internal/agent/tools/wiki_delete_page.go

  • wikiDeletePageTool (delete pages from agent)

File: internal/agent/tools/wiki_replace_text.go

  • wikiReplaceTextTool (find/replace within page)

LLM Prompts

File: internal/agent/prompts_wiki.go

  • WikiSummaryPrompt (summary generation)
  • WikiKnowledgeExtractPrompt (entity/concept extraction)
  • WikiPageModifyPrompt (page updating)

Cross-Document References

By Topic

Understanding Concurrency:

  1. WIKI_TECHNICAL_ANALYSIS.md → Part 4.1 (Concurrent update conflicts)
  2. WIKI_IMPLEMENTATION_GUIDE.md → Part 3 (Error handling)
  3. wiki_page.go (repo) → Update method (lines 36-60)

Understanding Links:

  1. WIKI_TECHNICAL_ANALYSIS.md → Part 2.1 (Bidirectional links)
  2. wiki_page.go (service) → updateInLinks, removeInLinks methods
  3. WIKI_IMPLEMENTATION_GUIDE.md → Task 3 (Work with links)

Understanding Async Processing:

  1. WIKI_TECHNICAL_ANALYSIS.md → Part 2.3 & 2.4 (Debouncing and MAP-REDUCE)
  2. WIKI_SYSTEM_DEEP_DIVE.md → Async pipeline section
  3. wiki_ingest.go → EnqueueWikiIngest method
  4. wiki_ingest_batch.go → ProcessWikiIngest handler

Understanding Cross-Link Injection:

  1. WIKI_TECHNICAL_ANALYSIS.md → Part 2.5 (Forbidden spans)
  2. WIKI_IMPLEMENTATION_GUIDE.md → Task 5 (Cross-link injection)
  3. wiki_linkify.go → Full implementation
  4. wiki_linkify_test.go → Test cases (coverage)

Understanding Health Checking:

  1. WIKI_TECHNICAL_ANALYSIS.md → Part 3.4 (Linting)
  2. wiki_lint.go → Full implementation
  3. WIKI_IMPLEMENTATION_GUIDE.md → Part 5 (Monitoring)

Key Diagrams and Flows

Data Flow (ASCII)

Document Upload
    ↓
EnqueueWikiIngest → Redis List (30s debounce)
    ↓
ProcessWikiIngest [Async]
    ├─ MAP: Extract entities/concepts (parallel)
    ├─ REDUCE: Synthesize pages (parallel)
    ├─ Post-processing: Links, index, cross-link injection
    └─ Schedule follow-up if pending remain
    ↓
Wiki Pages in DB
    ├─ Synchronized to chunks (ChunkType="wiki_page")
    ├─ Integrated in retrieval pipeline
    └─ Boosted 1.3x in chat search
    ↓
Chat Integration
    └─ Higher-ranked results → better answers

Service Dependencies

WikiPageService
    ├─ WikiPageRepository (data access)
    ├─ ChunkRepository (sync to retrieval)
    ├─ KnowledgeBaseService (KB validation)
    └─ Redis (pending queue, locking)

WikiIngestService
    ├─ WikiPageService (create/update pages)
    ├─ KnowledgeBaseService (KB config)
    ├─ KnowledgeService (fetch document chunks)
    ├─ ChunkRepository (sync results)
    ├─ ModelService (LLM calls)
    ├─ Task queue (Asynq)
    └─ Redis (debouncing, locking)

WikiLintService
    ├─ WikiPageService (get pages, graph)
    ├─ KnowledgeBaseService (KB validation)
    └─ No database write access (read-only)

State Transitions

WikiPage States:
    draft ──→ published ──→ archived
                  ↑
                  └─── (UpdatePage increments version)

WikiPageIssue States:
    pending ──→ acknowledged ──→ resolved

Page Creation Flow:
    1. Create page + parse OutLinks
    2. Insert to DB
    3. Update target InLinks (async, fire-and-forget)
    4. Return to caller

Page Update Flow:
    1. Fetch existing (get current version)
    2. Update fields + re-parse OutLinks
    3. Optimistic lock check (WHERE version = X)
    4. Update InLinks on old targets (remove)
    5. Update InLinks on new targets (add)
    6. Return updated page or ErrWikiPageConflict

Decision Matrix: Which Document to Read

Question Document
What is wiki and why do we have it? WIKI_SYSTEM_SUMMARY.md
How do I create a new wiki page? WIKI_IMPLEMENTATION_GUIDE.md (Task 1)
How does async processing work? WIKI_TECHNICAL_ANALYSIS.md (Part 2.3-2.4)
What are the edge cases I should know about? WIKI_TECHNICAL_ANALYSIS.md (Part 4)
How do I test my wiki feature? WIKI_IMPLEMENTATION_GUIDE.md (Part 2)
What's the complete API? WIKI_SYSTEM_DEEP_DIVE.md
How do I debug a broken link? WIKI_IMPLEMENTATION_GUIDE.md (Part 5)
What's the data model? internal/types/wiki_page.go
How does link injection work? WIKI_TECHNICAL_ANALYSIS.md (Part 2.5)
How do I handle version conflicts? WIKI_IMPLEMENTATION_GUIDE.md (Part 3)
What performance considerations exist? WIKI_TECHNICAL_ANALYSIS.md (Part 5.1)
How is wiki integrated with chat? WIKI_TECHNICAL_ANALYSIS.md (Part 6.1)
What are the LLM prompts? internal/agent/prompts_wiki.go
How do I monitor wiki health? WIKI_IMPLEMENTATION_GUIDE.md (Part 5) + wiki_lint.go
What's the full system architecture? WIKI_TECHNICAL_ANALYSIS.md (Part 1)

Maintenance and Updates

This documentation set should be updated when:

  • New API endpoints are added or modified
  • Database schema changes
  • LLM prompt templates change significantly
  • Performance characteristics change
  • New edge cases discovered
  • Configuration options change
  • Integration points change

To update documentation:

  1. Identify which docs are affected
  2. Update in order: code first, then summaries
  3. Regenerate any tables or metrics
  4. Update cross-references
  5. Review for accuracy and clarity

Feedback and Questions

If documentation is:

  • Unclear: Note the specific section and what's confusing
  • Incomplete: Note what's missing
  • Outdated: Note what's changed in the code
  • Inaccurate: Note what's wrong and what's correct

Post improvements as pull requests with clear descriptions.


Statistics

Metric Value
Total documentation ~3,900 lines
Source code (wiki only) ~7,300 lines
Total wiki-related files ~20 files
Test coverage ~285 lines (wiki_linkify_test.go + wiki_page_test.go)
LLM prompts 3 major templates
Database tables 2 (wiki_pages, wiki_page_issues)
Key services 3 (WikiPageService, WikiIngestService, WikiLintService)
API endpoints 20+ (see WIKI_SYSTEM_DEEP_DIVE.md)