28deb8282f
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings. - The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities. - Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system. These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
5.5 KiB
5.5 KiB
Knowledge Base Data Model - Quick Reference
Core Structs at a Glance
1. KnowledgeBase (internal/types/knowledgebase.go:40-98)
Main entity defining a knowledge base with configuration for:
- Chunking:
ChunkingConfig(chunk size, overlap, parent-child strategy) - Indexing: Determined by TENANT config via
Tenant.RetrieverEngines - Multimodal:
VLMConfig(vision/image processing),ASRConfig(speech recognition) - Wiki:
WikiConfig(wiki page generation on document ingest) - Question Generation:
QuestionGenerationConfig(auto-generate Q&A from chunks) - Graph Extraction:
ExtractConfig(knowledge graph entity/relation extraction) - Embedding Model:
EmbeddingModelID(for vector search) - Summary Model:
SummaryModelID(for LLM-based summarization)
2. WikiConfig (internal/types/wiki_page.go:91-103)
Controls wiki feature for a knowledge base:
Enabled: bool - Feature toggleAutoIngest: bool - Auto-generate pages when documents addedSynthesisModelID: string - LLM model for page generationMaxPagesPerIngest: int - Rate limit (0 = unlimited)
3. ChunkingConfig (internal/types/knowledgebase.go:118-141)
Controls document chunking strategy:
ChunkSize: int - Main chunk size in charactersChunkOverlap: int - Overlap between chunksParserEngineRules: []ParserEngineRule - Parser selection per file typeEnableParentChild: bool - Two-level chunking (context + matching)ParentChunkSize: int - Large chunks for context (default 4096)ChildChunkSize: int - Small chunks for embedding (default 384)
Processing Pipeline Steps
Step 1: Document Ingestion
Document Upload → ProcessDocument Task (async)
File: internal/application/service/knowledge.go:7732
Key:
- Validates KB config (multimodal, ASR enabled)
- Downloads file from URL if provided
- Calls
convert()to parse document
Step 2: Chunk Creation & Indexing
processChunks() → Save chunks to DB → BatchIndex()
File: internal/application/service/knowledge.go:1681
Key:
- Creates chunks from parsed content
- Parent chunks stored but NOT indexed (parent-child mode)
- Child/text chunks prepared for indexing
retrieveEngine.BatchIndex()uses TENANT's RetrieverEngines config- Supports both vector and keyword indexing (can be combined)
Step 3: Post-Processing
KnowledgePostProcess → Spawn async tasks
File: internal/application/service/knowledge_post_process.go:42
Key:
- Summary Generation (LLM summarization)
- Question Generation (if enabled)
- Graph RAG Extraction (if enabled)
- Wiki Ingest (if enabled AND AutoIngest=true AND has text chunks)
Vector vs Keyword Indexing Decision
Decision Location: Tenant configuration, NOT knowledge base level
How it works:
- Tenant has
RetrieverEnginesconfig specifying retriever types - Each engine can be:
"vector","keywords", or both - During
BatchIndex(), all configured types are used - Enables hybrid search combining results from multiple index types
Config Path:
- Table:
tenants - Column:
retriever_engines(JSONB) - Type:
[]RetrieverEngineParams - Each param specifies:
{RetrieverEngineType, RetrieverType}
Wiki Ingest Trigger
Exact Condition (knowledge_post_process.go:123-126):
if kb.IsWikiEnabled() && kb.WikiConfig.AutoIngest && len(textChunks) > 0 {
EnqueueWikiIngest(...)
}
Breakdown:
kb.IsWikiEnabled()⟹WikiConfig != nil && Enabled == truekb.WikiConfig.AutoIngest⟹ Must be explicitly enabledlen(textChunks) > 0⟹ Must have text content (not just images)
Result:
- Enqueues
TypeWikiIngesttask - LLM generates/updates wiki pages based on ingested content
- Uses
SynthesisModelIDfrom WikiConfig
Retrieval Configuration
File: internal/types/retrieval_config.go:14-27
Stored on: tenants.retrieval_config (JSONB)
Controls:
EmbeddingTopK(50): Max results from vector searchVectorThreshold(0.15): Min similarity score for vectorsKeywordThreshold(0.3): Min score for keywordsRerankTopK(10): Results after rerankingRerankModelID: LLM for reranking results
API Endpoints
Create Knowledge Base
POST /knowledge-bases
Body: types.KnowledgeBase
Update Knowledge Base
PUT /knowledge-bases/{id}
Body: UpdateKnowledgeBaseRequest {
Name: string
Description: string
Config: *KnowledgeBaseConfig { ChunkingConfig, WikiConfig, ... }
}
Delete Knowledge Base
DELETE /knowledge-bases/{id}
Important Constants
Knowledge Base Types
"document"- Regular documents with chunking/search"faq"- Q&A pairs"wiki"- Wiki pages (not a separate type, but feature on documents)
Retriever Types
"vector"- Vector/embedding-based search"keywords"- Full-text/keyword search"websearch"- Web search
Chunk Types
"text"- Main text content"parent_text"- Parent chunk (context, not indexed)"image_ocr"- OCR text from images"image_caption"- AI-generated image captions"summary"- Document summary chunk
Key Takeaways
- Indexing is tenant-scoped, not per-KB
- Hybrid search uses multiple index types simultaneously
- Wiki ingest requires three conditions: enabled, auto_ingest, content
- Parent-child chunking splits semantic context from search terms
- Post-processing is async and orchestrates multiple LLM tasks
- Multimodal (images/speech) requires explicit config + KB support