Files
Enterprise-Knowledge-Base/WIKI_IMPLEMENTATION_GUIDE.md
T
wizardchen 28deb8282f feat: Add comprehensive documentation for WeKnora knowledge base and wiki features
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings.
- The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities.
- Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system.

These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
2026-04-22 21:18:10 +08:00

757 lines
22 KiB
Markdown

# Wiki System Implementation Guide
**Target Audience:** Developers implementing wiki features or extending the system
**Last Updated:** April 2026
---
## Part 1: Common Tasks and Patterns
### Task 1: Create a New Wiki Page Programmatically
**Scenario:** Agent tool or batch process needs to create a page
```go
import (
"context"
"time"
"github.com/google/uuid"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
func createEntityPage(
ctx context.Context,
svc interfaces.WikiPageService,
kbID string,
name string,
details string,
) (*types.WikiPage, error) {
page := &types.WikiPage{
ID: uuid.New().String(),
KnowledgeBaseID: kbID,
Slug: slugify(name), // "Acme Corp" → "entity/acme-corp"
Title: name,
PageType: types.WikiPageTypeEntity,
Status: types.WikiPageStatusPublished,
Content: formatMarkdown(details),
Summary: firstSentence(details),
Aliases: []string{}, // Alternative names
SourceRefs: []string{}, // Knowledge IDs
Version: 1,
}
// Service handles link parsing and inbound link updates
return svc.CreatePage(ctx, page)
}
// Helper: normalize slug like the service does
func slugify(s string) string {
s = strings.ToLower(strings.TrimSpace(s))
s = strings.ReplaceAll(s, " ", "-")
return "entity/" + s
}
```
**Key Points:**
- Service automatically parses OutLinks from content
- InLinks are updated on target pages automatically
- Version starts at 1
- Timestamps set by service
### Task 2: Update an Existing Page
**Scenario:** Need to modify page content (e.g., agent synthesis)
```go
func updatePageContent(
ctx context.Context,
svc interfaces.WikiPageService,
kbID string,
slug string,
newContent string,
newSummary string,
) (*types.WikiPage, error) {
// 1. Fetch existing page
existing, err := svc.GetPageBySlug(ctx, kbID, slug)
if err != nil {
return nil, fmt.Errorf("page not found: %w", err)
}
// 2. Update only the fields you want to change
existing.Content = newContent
existing.Summary = newSummary
// DO NOT modify Version, OutLinks, InLinks, etc.
// 3. Call UpdatePage
// - Service re-parses OutLinks
// - Service updates InLinks on target pages
// - Service increments version (optimistic lock)
updated, err := svc.UpdatePage(ctx, existing)
if err != nil {
// Might be ErrWikiPageConflict if version changed
// Retry with fresh fetch if needed
return nil, fmt.Errorf("update failed: %w", err)
}
return updated, nil
}
// INCORRECT: Do this instead
// Wrong: page.OutLinks = []string{...} // Don't manually set!
// Wrong: page.InLinks = []string{...} // Service maintains these!
```
**Handling Conflicts:**
```go
// If update fails with ErrWikiPageConflict, retry
func updatePageWithRetry(ctx context.Context, svc interfaces.WikiPageService,
kbID, slug, newContent string) (*types.WikiPage, error) {
maxRetries := 3
for attempt := 1; attempt <= maxRetries; attempt++ {
existing, _ := svc.GetPageBySlug(ctx, kbID, slug)
existing.Content = newContent
updated, err := svc.UpdatePage(ctx, existing)
if err == nil {
return updated, nil
}
if !strings.Contains(err.Error(), "conflict") {
return nil, err // Not a conflict, real error
}
// Conflict: retry with fresh page
}
return nil, fmt.Errorf("max retries exceeded")
}
```
### Task 3: Work with Links Programmatically
**Scenario:** Need to query the link graph
```go
func analyzePageConnections(
ctx context.Context,
svc interfaces.WikiPageService,
kbID string,
slug string,
) {
// Get the page
page, _ := svc.GetPageBySlug(ctx, kbID, slug)
// Pages that link TO this page (backlinks)
fmt.Printf("Inbound links: %v\n", page.InLinks) // []{"page-a", "page-b"}
// Pages this page links to (references)
fmt.Printf("Outbound links: %v\n", page.OutLinks) // []{"page-c", "page-d"}
// Get the full graph
graph, _ := svc.GetGraph(ctx, kbID)
// Find all pages that mention this page
incomingReferences := make(map[string]string)
for _, node := range graph.Nodes {
p, _ := svc.GetPageBySlug(ctx, kbID, node.Slug)
for _, outLink := range p.OutLinks {
if outLink == slug {
incomingReferences[node.Slug] = node.Title
}
}
}
}
```
**Link Consistency Guarantees:**
```go
// If you fetch a page and it has OutLinks, those pages should exist
// (unless the link is broken)
page, _ := svc.GetPageBySlug(ctx, kbID, slug)
for _, target := range page.OutLinks {
targetPage, err := svc.GetPageBySlug(ctx, kbID, target)
if err != nil {
// Broken link! Linter would flag this
fmt.Printf("BROKEN: %s links to nonexistent %s\n", slug, target)
}
}
// If you see page.InLinks, that page should have you in its OutLinks
// (Unless there's a race condition or bug)
page, _ := svc.GetPageBySlug(ctx, kbID, slug)
for _, inLink := range page.InLinks {
inLinkPage, _ := svc.GetPageBySlug(ctx, kbID, inLink)
if !contains(inLinkPage.OutLinks, slug) {
fmt.Printf("INCONSISTENT: %s claims to link to us, but we're not in their OutLinks\n", inLink)
}
}
```
### Task 4: Iterate Over All Pages Efficiently
**Scenario:** Batch processing, statistics generation
```go
func iterateAllPages(ctx context.Context, svc interfaces.WikiPageService, kbID string) error {
// Option 1: ListAllPages (simple, loads everything into memory)
pages, err := svc.ListAllPages(ctx, kbID)
for _, page := range pages {
processPage(page)
}
// Option 2: Paginated List (memory-efficient, used by UI)
pageSize := 100
for page := 1; ; page++ {
req := &types.WikiPageListRequest{
KnowledgeBaseID: kbID,
Page: page,
PageSize: pageSize,
SortBy: "updated_at",
SortOrder: "desc",
}
resp, _ := svc.ListPages(ctx, req)
for _, p := range resp.Pages {
processPage(p)
}
if resp.Page * resp.PageSize >= int(resp.Total) {
break // Last page
}
}
return nil
}
// Filter by type
func getConceptPages(ctx context.Context, svc interfaces.WikiPageService, kbID string) ([]*types.WikiPage, error) {
pages, _ := svc.ListAllPages(ctx, kbID)
var concepts []*types.WikiPage
for _, p := range pages {
if p.PageType == types.WikiPageTypeConcept {
concepts = append(concepts, p)
}
}
return concepts, nil
}
```
### Task 5: Work with Cross-Link Injection
**Scenario:** Need to understand or extend linkification
```go
import (
"github.com/Tencent/WeKnora/internal/application/service"
)
// The service-level function
func injectCrossLinksExample(ctx context.Context, svc interfaces.WikiPageService, kbID string) {
// Get all pages to collect link references
allPages, _ := svc.ListAllPages(ctx, kbID)
// Build refs from entity/concept pages
var refs []linkRef
for _, p := range allPages {
if p.PageType == types.WikiPageTypeEntity || p.PageType == types.WikiPageTypeConcept {
// Each page becomes a potential link target
ref := linkRef{
slug: p.Slug,
matchText: p.Title, // or p.Aliases[i]
}
refs = append(refs, ref)
}
}
// Call the service (internal function, not exported)
// Typically called by InjectCrossLinks() which batches this
for _, p := range allPages {
if p.PageType == types.WikiPageTypeIndex || p.PageType == types.WikiPageTypeLog {
continue // Don't inject into system pages
}
// linkifyContent() is in wiki_linkify.go, NOT exported
// This is called internally by the service
}
}
```
**Key Implementation Detail:**
The `linkifyContent` function is package-private (lowercase). You shouldn't call it directly. Instead:
```go
// Correct: Use service-level InjectCrossLinks
svc.InjectCrossLinks(ctx, kbID, []string{"entity/acme"})
// Incorrect: Don't try to call linkifyContent directly
// content, _ := linkifyContent(page.Content, refs, page.Slug)
// (Won't compile, it's in service package not exported)
```
---
## Part 2: Testing Patterns
### Unit Test: Page Creation and Links
```go
package service
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/types"
)
func TestCreatePageWithLinks(t *testing.T) {
ctx := context.Background()
// Setup
repo := NewMockWikiPageRepository()
svc := NewWikiPageService(repo, ...)
// Create first page
page1 := &types.WikiPage{
Slug: "entity/acme",
Title: "Acme Corp",
Content: "[[entity/bob]] founded Acme.",
}
created1, _ := svc.CreatePage(ctx, page1)
// Verify OutLinks parsed
if len(created1.OutLinks) != 1 || created1.OutLinks[0] != "entity/bob" {
t.Fatalf("OutLinks not parsed correctly")
}
// Create target page
page2 := &types.WikiPage{
Slug: "entity/bob",
Title: "Bob Smith",
}
created2, _ := svc.CreatePage(ctx, page2)
// Verify InLinks updated
// (Requires mocking updateInLinks or using integration test)
refreshed2, _ := svc.GetPageBySlug(ctx, "entity/bob")
if !contains(refreshed2.InLinks, "entity/acme") {
t.Fatalf("InLinks not updated")
}
}
```
### Integration Test: Full Pipeline
```go
func TestWikiIngestToRetrieval(t *testing.T) {
// Setup: real DB, real LLM mock
ctx := context.Background()
db := setupTestDB()
modelSvc := NewMockModelService() // Mock LLM responses
// 1. Create KB with wiki enabled
kb := &types.KnowledgeBase{
ID: "test-kb",
WikiConfig: types.WikiConfig{
Enabled: true,
AutoIngest: true,
SynthesisModelID: "test-model",
},
}
// ... save to DB
// 2. Enqueue document
ingestSvc := NewWikiIngestService(...)
ingestSvc.EnqueueWikiIngest(ctx, kb.ID, "doc1", "Document Title")
// 3. Manually trigger batch processing (or wait + poll)
handler := NewProcessWikiIngestHandler(ingestSvc, ...)
task, _ := asynq.NewTask("wiki:ingest", payload)
handler(task)
// 4. Verify pages created
pages, _ := pageSvc.ListAllPages(ctx, kb.ID)
if len(pages) == 0 {
t.Fatalf("No pages created")
}
// 5. Verify chunks synchronized
chunks, _ := chunkService.GetChunks(ctx, "wp-*") // Wiki page chunks
if len(chunks) != len(pages) {
t.Fatalf("Not all pages chunked")
}
// 6. Search and verify boost
results, _ := retriever.Search(ctx, "query")
// Wiki page chunks should rank higher due to 1.3x boost
}
```
---
## Part 3: Error Handling
### Common Errors and Recovery
**ErrWikiPageNotFound**
```go
import (
"errors"
"github.com/Tencent/WeKnora/internal/application/repository"
)
page, err := svc.GetPageBySlug(ctx, kbID, slug)
if errors.Is(err, repository.ErrWikiPageNotFound) {
// Page doesn't exist
// Safe to create new page with this slug
return svc.CreatePage(ctx, &types.WikiPage{Slug: slug, ...})
}
```
**ErrWikiPageConflict**
```go
existing, _ := svc.GetPageBySlug(ctx, kbID, slug)
existing.Content = newContent
updated, err := svc.UpdatePage(ctx, existing)
if errors.Is(err, repository.ErrWikiPageConflict) {
// Another process updated the page
// Retry: fetch fresh version and reapply changes
existing, _ = svc.GetPageBySlug(ctx, kbID, slug)
existing.Content = newContent
updated, _ = svc.UpdatePage(ctx, existing) // Should succeed now
}
```
**LLM Extraction Failures**
```go
// During wiki ingest, if LLM call fails:
// 1. Error logged, extraction returns empty
// 2. Document marked as "processed" anyway
// 3. No automatic retry
// Recovery: Manually re-upload document
// Check for failed extractions
for _, doc := range pendingOps {
if doc.Status == "failed" {
fmt.Printf("Document %s failed to extract\n", doc.KnowledgeID)
// Re-upload or check logs for details
}
}
```
---
## Part 4: Performance Optimization
### Batch Creation
**Don't do this:**
```go
// N separate Create calls → N DB round-trips, N link updates
for _, data := range largeBatch {
page := mapToPage(data)
svc.CreatePage(ctx, &page) // Slow!
}
```
**Do this instead:**
```go
// Batch in the ingest service (30s debounce)
for _, data := range largeBatch {
ingestSvc.EnqueueWikiIngest(ctx, kbID, data.KnowledgeID, data.Title)
}
// Service batches and processes in one async task
```
### Query Optimization
**List with Filters**
```go
// Good: Let repository handle filtering
req := &types.WikiPageListRequest{
KnowledgeBaseID: kbID,
PageType: "entity", // Filter in DB
Status: "published", // Filter in DB
Query: "acme", // Full-text search in DB
PageSize: 50,
}
resp, _ := svc.ListPages(ctx, req)
// Bad: Fetch all and filter in memory
pages, _ := svc.ListAllPages(ctx, kbID) // Loads everything!
var entities []*types.WikiPage
for _, p := range pages {
if p.PageType == "entity" { entities = append(entities, p) }
}
```
### Link Rebuild Efficiency
```go
// RebuildLinks is O(n²) in worst case (all pages link to all others)
// For large wikis (>1000 pages):
// - Run in background
// - Consider scheduling off-peak
// - Monitor resource usage
// Example: scheduled maintenance
func scheduleNightlyRebuild(svc interfaces.WikiPageService) {
go func() {
ticker := time.NewTicker(24 * time.Hour)
for range ticker.C {
if shouldRebuild() { // Check health score
svc.RebuildLinks(context.Background(), kbID)
logger.Infof(context.Background(), "nightly rebuild complete")
}
}
}()
}
```
---
## Part 5: Debugging and Troubleshooting
### Check Page Consistency
```go
func validatePageConsistency(ctx context.Context,
svc interfaces.WikiPageService, kbID, slug string) []string {
var issues []string
page, _ := svc.GetPageBySlug(ctx, kbID, slug)
// 1. Check OutLinks exist
for _, target := range page.OutLinks {
if target == page.Slug {
issues = append(issues, fmt.Sprintf("Self-link detected"))
}
tp, err := svc.GetPageBySlug(ctx, kbID, target)
if err != nil {
issues = append(issues, fmt.Sprintf("Broken OutLink: %s", target))
} else if !contains(tp.InLinks, slug) {
issues = append(issues, fmt.Sprintf("InLink inconsistency: %s should have inbound from %s", target, slug))
}
}
// 2. Check InLinks reciprocal
for _, source := range page.InLinks {
sp, _ := svc.GetPageBySlug(ctx, kbID, source)
if !contains(sp.OutLinks, slug) {
issues = append(issues, fmt.Sprintf("InLink inconsistency: %s claims to link here but doesn't", source))
}
}
// 3. Check content matches OutLinks
foundLinks := parseLinks(page.Content)
for _, link := range foundLinks {
if !contains(page.OutLinks, link) {
issues = append(issues, fmt.Sprintf("Content has link [[%s]] not in OutLinks", link))
}
}
return issues
}
// Usage
issues := validatePageConsistency(ctx, svc, kbID, "entity/acme")
if len(issues) > 0 {
logger.Warnf(ctx, "Page consistency issues: %v", issues)
// Consider running RebuildLinks to fix
}
```
### Monitor Wiki Health
```go
func monitorWikiHealth(ctx context.Context,
lintSvc *service.WikiLintService, kbID string) {
report, _ := lintSvc.RunLint(ctx, kbID)
fmt.Printf("Health Score: %d/100\n", report.HealthScore)
fmt.Printf("Total Issues: %d\n", len(report.Issues))
for _, issue := range report.Issues {
fmt.Printf("[%s] %s: %s\n",
issue.Severity, issue.Type, issue.Description)
}
if report.HealthScore < 70 {
logger.Warnf(ctx, "Wiki health below threshold: %d/100", report.HealthScore)
// Consider triggering AutoFix
fixed, _ := lintSvc.AutoFix(ctx, kbID)
logger.Infof(ctx, "AutoFix resolved %d issues", fixed)
}
}
```
---
## Part 6: Common Mistakes and How to Avoid Them
| Mistake | Problem | Fix |
|---------|---------|-----|
| Manually setting OutLinks | Links get out of sync | Let service parse from Content |
| Manually setting InLinks | Inbound refs become stale | Let service maintain via updateInLinks |
| Calling linkifyContent directly | Package is private | Use service.InjectCrossLinks |
| Not handling ErrWikiPageConflict | Updates silently fail | Add retry logic for conflicts |
| Storing page without timestamp | Audit trail lost | Set CreatedAt/UpdatedAt before Create |
| Mixing wiki and non-wiki KBs | Feature gating fails | Check IsWikiEnabled() always |
| Assuming slug format | Future formats might change | Use parseSlug() helper |
| Long-running UpdatePage loops | Timeout/deadlock risk | Batch with EnqueueWikiIngest |
| Ignoring LLM extraction errors | Partial data persists | Log and monitor failures |
| Direct database updates | Links become inconsistent | Always use service layer |
---
## Appendix A: Key Functions Reference
### WikiPageService
| Method | Purpose | When to Use |
|--------|---------|------------|
| CreatePage | Insert new page | Creating extracted/synthesized pages |
| UpdatePage | Update content (version bump) | Modifying page text, title, summary |
| UpdatePageMeta | Update metadata (no version bump) | Internal link maintenance |
| GetPageBySlug | Fetch by (kbID, slug) | Retrieve for display/editing |
| GetPageByID | Fetch by UUID | Less common, used internally |
| ListPages | Paginated list with filters | UI pagination, search |
| ListAllPages | All pages without pagination | Batch processing, analysis |
| SearchPages | Full-text search | User search queries |
| DeletePage | Soft delete | Remove pages |
| GetIndex | Fetch/create index page | Getting wiki directory |
| GetLog | Fetch/create log page | Getting operation history |
| GetGraph | Build link graph | Visualization, analysis |
| GetStats | Aggregate statistics | Dashboard, monitoring |
| RebuildLinks | Re-establish all references | Recovery after bugs |
| InjectCrossLinks | Auto-link pages | Post-ingest cross-linking |
| RebuildIndexPage | Regenerate index | After page changes |
### WikiLintService
| Method | Purpose | Returns |
|--------|---------|---------|
| RunLint | Health check | WikiLintReport (issues, score, stats) |
| AutoFix | Auto-repair issues | Number of fixed issues |
### WikiIngestService
| Method | Purpose | When to Use |
|--------|---------|------------|
| EnqueueWikiIngest | Queue document for processing | After document upload |
| EnqueueWikiRetract | Queue document deletion | When document deleted |
---
## Appendix B: Configuration
```yaml
# Example wiki config in knowledge base
wiki_config:
enabled: true
auto_ingest: true
synthesis_model_id: "gpt-4" # or "claude-3-opus", etc.
max_pages_per_ingest: 0 # 0 = unlimited
# Environment-level defaults
WIKI_INGEST_DELAY: 30s # Debounce window
WIKI_MAX_DOCS_PER_BATCH: 5 # Docs per batch
WIKI_MAX_CONTENT_SIZE: 33554432 # 32 MB per document
WIKI_PENDING_TTL: 86400 # 24 hours
WIKI_LOCK_TTL: 300 # 5 minutes
# LLM-specific settings
WIKI_SYNTHESIS_MODEL: "gpt-4-turbo"
WIKI_EXTRACTION_MODEL: "gpt-4"
WIKI_TEMPERATURE: 0.3 # Lower = more deterministic
WIKI_MAX_TOKENS: 4000 # Per LLM call
```
---
## Appendix C: SQL Queries for Debugging
```sql
-- Find all orphan pages
SELECT slug, title, page_type, updated_at
FROM wiki_pages
WHERE knowledge_base_id = 'kb123'
AND page_type NOT IN ('index', 'log')
AND JSON_LENGTH(in_links) = 0
ORDER BY updated_at DESC;
-- Find broken links
SELECT DISTINCT p.slug, p.title,
JSON_UNQUOTE(JSON_EXTRACT(p.out_links, '$[*]')) as broken_target
FROM wiki_pages p
WHERE knowledge_base_id = 'kb123'
AND JSON_LENGTH(p.out_links) > 0
AND NOT EXISTS (
SELECT 1 FROM wiki_pages t
WHERE t.knowledge_base_id = p.knowledge_base_id
AND t.slug = JSON_UNQUOTE(JSON_EXTRACT(p.out_links, '$[0]'))
);
-- Find highest-linked pages (hub pages)
SELECT slug, title,
JSON_LENGTH(in_links) as inbound_count,
JSON_LENGTH(out_links) as outbound_count
FROM wiki_pages
WHERE knowledge_base_id = 'kb123'
AND page_type NOT IN ('index', 'log')
ORDER BY JSON_LENGTH(in_links) DESC
LIMIT 20;
-- Find pages by type distribution
SELECT page_type, COUNT(*) as count,
AVG(LENGTH(content)) as avg_size,
AVG(JSON_LENGTH(in_links)) as avg_inlinks
FROM wiki_pages
WHERE knowledge_base_id = 'kb123'
AND deleted_at IS NULL
GROUP BY page_type
ORDER BY count DESC;
-- Monitor recent activity
SELECT slug, title, page_type, updated_at, version
FROM wiki_pages
WHERE knowledge_base_id = 'kb123'
AND deleted_at IS NULL
ORDER BY updated_at DESC
LIMIT 20;
-- Find pages by source knowledge
SELECT slug, title, COUNT(*) as source_count
FROM wiki_pages, JSON_TABLE(
source_refs, '$[*]' COLUMNS (ref VARCHAR(100) PATH '$')
) jt
WHERE knowledge_base_id = 'kb123'
AND knowledge_id = 'knowledgeID123'
GROUP BY slug, title;
```
---
## Summary
This guide covers the essential patterns for working with the wiki system. Key principles:
1. **Always use the service layer** — don't access repository directly
2. **Let the service maintain links** — don't manually set InLinks/OutLinks
3. **Handle concurrency** — retry on ErrWikiPageConflict
4. **Use the ingest pipeline** — batch operations via EnqueueWikiIngest
5. **Monitor health** — run RunLint periodically
6. **Debug with tools** — use GetGraph, GetStats, linter for troubleshooting
For more details, see `WIKI_TECHNICAL_ANALYSIS.md` for deep dives into architecture and edge cases.