28deb8282f
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings. - The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities. - Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system. These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
416 lines
14 KiB
Markdown
416 lines
14 KiB
Markdown
# Knowledge Base & Wiki System - File Index & Cross-Reference
|
|
|
|
## COMPLETE FILE LISTING BY COMPONENT
|
|
|
|
### 1. DATA MODELS (internal/types/)
|
|
|
|
| File | Purpose | Key Types |
|
|
|------|---------|-----------|
|
|
| `knowledgebase.go` | KB data model | `KnowledgeBase`, `WikiConfig`, `ChunkingConfig`, `VLMConfig`, `ASRConfig`, `ExtractConfig`, `FAQConfig` |
|
|
| `knowledge.go` | Document/knowledge model | `Knowledge`, `ManualKnowledgeMetadata`, `Knowledge.Type` (manual, faq) |
|
|
| `wiki_page.go` | Wiki page model | `WikiPage`, `WikiPageIssue`, `WikiConfig`, `WikiStats`, `WikiGraphData` |
|
|
| `agent.go` | Agent configuration | `AgentConfig`, `SessionAgentConfig` (includes KnowledgeBases) |
|
|
| `chunk.go` | Chunk model | Indexed content units (not detailed in search but referenced) |
|
|
| `message.go` | Chat messages | Not detailed - used by sessions |
|
|
| `tag.go` | Tags/categories | For knowledge organization |
|
|
|
|
### 2. INTERFACES (internal/types/interfaces/)
|
|
|
|
| File | Purpose | Contracts |
|
|
|------|---------|-----------|
|
|
| `knowledgebase.go` | KB service interface | `KnowledgeBaseService`, `KnowledgeBaseRepository` |
|
|
| `knowledge.go` | Knowledge service interface | `KnowledgeService`, `KnowledgeRepository` |
|
|
| `wiki_page.go` | Wiki service interface | `WikiPageService`, `WikiPageRepository` |
|
|
| `agent.go` | Agent tool interface | Generic tool registration |
|
|
| `chunk.go` | Chunk access interface | `ChunkRepository`, `ChunkService` |
|
|
|
|
### 3. SERVICES (internal/application/service/)
|
|
|
|
| File | Purpose | Methods |
|
|
|------|---------|---------|
|
|
| `knowledgebase.go` | KB business logic | Create, Update, Get, List, HybridSearch |
|
|
| `knowledge.go` | Doc ingest pipeline | CreateKnowledge, UpdateKnowledge, Delete, Search |
|
|
| `wiki_ingest.go` | Wiki LLM generation | WikiIngestService, ProcessWikiIngest, ScheduleFollowUp |
|
|
| `wiki_ingest_batch.go` | Wiki batch processing | Batch LLM calls, link rebuilding |
|
|
| `wiki_page.go` | Wiki CRUD operations | CreatePage, UpdatePage, GetPageBySlug, ListPages, RebuildLinks |
|
|
| `wiki_linkify.go` | Wiki link parsing | Parse [[wiki-links]] syntax |
|
|
| `wiki_lint.go` | Wiki quality checks | Validate, find broken links, orphaned pages |
|
|
| `knowledge_post_process.go` | Post-processing docs | Summary generation, extraction |
|
|
| `chat_pipeline/search.go` | Retrieval in chat | Knowledge search during chat |
|
|
| `chat_pipeline/wiki_boost.go` | Wiki integration in chat | Boost wiki results in chat context |
|
|
| `chunk.go` | Chunk service | Chunk CRUD, retrieval |
|
|
| `retriever/keywords_vector_hybrid_indexer.go` | Hybrid search engine | Index, BatchIndex, Retrieve |
|
|
|
|
### 4. REPOSITORIES (internal/application/repository/)
|
|
|
|
| File | Purpose | Storage |
|
|
|------|---------|---------|
|
|
| `knowledgebase.go` | KB persistence | GORM: knowledge_bases table |
|
|
| `knowledge.go` | Knowledge persistence | GORM: knowledges table |
|
|
| `wiki_page.go` | Wiki persistence | GORM: wiki_pages, wiki_page_issues |
|
|
| `chunk.go` | Chunk persistence | GORM: chunks table |
|
|
| `chunk_sqlite_test.go` | SQLite chunk backend | For testing |
|
|
| `retriever/weaviate/repository.go` | Weaviate backend | Vector search |
|
|
| `retriever/elasticsearch/v7/repository.go` | Elasticsearch v7 | Vector + keyword |
|
|
| `retriever/elasticsearch/v8/repository.go` | Elasticsearch v8 | Vector + keyword |
|
|
| `retriever/qdrant/repository.go` | Qdrant backend | Vector search |
|
|
| `retriever/milvus/repository.go` | Milvus backend | Vector search |
|
|
| `retriever/postgres/repository.go` | PostgreSQL backend | Vector search with pgvector |
|
|
| `retriever/sqlite/repository.go` | SQLite backend | Local vector search |
|
|
| `tag.go` | Tag persistence | GORM: tags table |
|
|
|
|
### 5. HTTP HANDLERS (internal/handler/)
|
|
|
|
| File | Purpose | Endpoints |
|
|
|------|---------|-----------|
|
|
| `knowledgebase.go` | KB HTTP API | POST/PUT/GET/DELETE /api/v1/knowledge-bases |
|
|
| `wiki_page.go` | Wiki HTTP API | GET/POST/PUT/DELETE /api/v1/knowledgebase/{id}/wiki/* |
|
|
| `knowledge.go` | Knowledge HTTP API | File upload, URL add, manual create |
|
|
| `chunk.go` | Chunk HTTP API | Chunk retrieval, preview |
|
|
| `session/qa.go` | Session chat QA | Knowledge search in chat context |
|
|
|
|
### 6. ROUTING (internal/router/)
|
|
|
|
| File | Purpose | Route Registration |
|
|
|------|---------|-----------------|
|
|
| `router.go` | Main router | RegisterWikiPageRoutes (~line 839) |
|
|
| `task.go` | Task routing | Async task registration |
|
|
|
|
### 7. AGENT TOOLS (internal/agent/tools/)
|
|
|
|
| File | Purpose | Tool Name |
|
|
|------|---------|-----------|
|
|
| `wiki_write_page.go` | Agent creates page | `wiki_write_page` |
|
|
| `wiki_read_source_doc.go` | Agent reads original doc | `wiki_read_source_doc` |
|
|
| `wiki_tools.go` | Tool registration | WikiToolRegistry |
|
|
| `wiki_delete_page.go` | Agent deletes page | `wiki_delete_page` |
|
|
| `wiki_replace_text.go` | Agent edits page | `wiki_replace_text` |
|
|
| `wiki_rename_page.go` | Agent renames page | `wiki_rename_page` |
|
|
| `wiki_update_issue.go` | Agent resolves issue | `wiki_update_issue` |
|
|
| `wiki_read_issue.go` | Agent checks issue | `wiki_read_issue` |
|
|
| `wiki_flag_issue.go` | Agent flags problem | `wiki_flag_issue` |
|
|
| `knowledge_search.go` | Agent searches KB | `knowledge_search` |
|
|
| `grep_chunks.go` | Agent regex search | `grep_chunks` |
|
|
| `list_knowledge_chunks.go` | Agent lists docs | `list_knowledge_chunks` |
|
|
| `get_document_info.go` | Agent gets doc info | `get_document_info` |
|
|
| `web_search.go` | Agent web search | `web_search` |
|
|
| `definitions.go` | Tool definitions | Tool registry |
|
|
|
|
### 8. AGENT CONFIG (internal/agent/)
|
|
|
|
| File | Purpose | Content |
|
|
|------|---------|---------|
|
|
| `prompts_wiki.go` | Wiki-specific prompts | Agent instructions for wiki tools |
|
|
| `prompts.go` | General prompts | System prompt templates |
|
|
| `engine.go` | Agent loop | ReAct execution engine |
|
|
| `observe.go` | Agent observation | Tool result interpretation |
|
|
|
|
### 9. ASYNC TASKS (internal/types/ & services)
|
|
|
|
| Task Type | File | Handler | Payload |
|
|
|-----------|------|---------|---------|
|
|
| `TypeWikiIngest` | wiki_ingest.go | ProcessWikiIngest | WikiIngestPayload |
|
|
| (Knowledge ingest) | knowledge.go | Various | KnowledgePayload |
|
|
|
|
### 10. FRONTEND (frontend/src/)
|
|
|
|
| File | Purpose | Exports |
|
|
|------|---------|---------|
|
|
| `api/knowledge-base/index.ts` | KB API client | createKnowledgeBase, updateKnowledgeBase, getKnowledgeBaseById, etc. |
|
|
| `api/wiki/index.ts` | Wiki API client | listWikiPages, getWikiPage, updateWikiPage, etc. |
|
|
| `stores/knowledge.ts` | State management | Knowledge store (Pinia) |
|
|
| `stores/settings.ts` | Settings store | App settings |
|
|
| `hooks/useKnowledgeBase.ts` | KB hook | KB operations hook |
|
|
| `hooks/useKnowledgeBaseCreationNavigation.ts` | KB creation | Navigation after KB creation |
|
|
|
|
---
|
|
|
|
## KEY DATA FLOWS BY FILE
|
|
|
|
### Document Ingestion Flow
|
|
```
|
|
1. frontend/src/api/knowledge-base/index.ts
|
|
→ uploadKnowledgeFile(kbId, file)
|
|
|
|
2. internal/handler/knowledge.go
|
|
→ KnowledgeHandler.CreateKnowledge()
|
|
|
|
3. internal/application/service/knowledge.go
|
|
→ KnowledgeService.CreateKnowledge()
|
|
→ Creates Knowledge entity (ParseStatus=pending)
|
|
→ Enqueues ingest task
|
|
|
|
4. Async task processed:
|
|
→ Chunks created (chunks table)
|
|
→ Embeddings indexed (vector DB: Weaviate, Elasticsearch, etc.)
|
|
→ IF WikiConfig.AutoIngest:
|
|
→ internal/application/service/wiki_ingest.go
|
|
→ Enqueue WikiIngest task (30s delay)
|
|
|
|
5. Wiki ingest task:
|
|
→ internal/application/service/wiki_ingest_batch.go
|
|
→ ProcessWikiIngest()
|
|
→ LLM generates pages
|
|
→ internal/application/service/wiki_page.go
|
|
→ CreatePage() for each generated wiki page
|
|
→ internal/application/repository/wiki_page.go
|
|
→ Save to wiki_pages table
|
|
|
|
6. Link maintenance:
|
|
→ internal/application/service/wiki_linkify.go
|
|
→ Parse [[wiki-links]]
|
|
→ internal/application/service/wiki_page.go
|
|
→ RebuildLinks() updates InLinks/OutLinks
|
|
```
|
|
|
|
### Settings Update Flow
|
|
```
|
|
1. frontend/src/api/knowledge-base/index.ts
|
|
→ updateKnowledgeBase(id, config)
|
|
|
|
2. internal/handler/knowledgebase.go
|
|
→ KnowledgeBaseHandler.UpdateKnowledgeBase()
|
|
→ Validates config
|
|
|
|
3. internal/application/service/knowledgebase.go
|
|
→ KnowledgeBaseService.UpdateKnowledgeBase()
|
|
→ Merges config (including wiki_config)
|
|
→ Calls EnsureDefaults()
|
|
|
|
4. internal/application/repository/knowledgebase.go
|
|
→ KnowledgeBaseRepository.UpdateKnowledgeBase()
|
|
→ UPDATE knowledge_bases SET wiki_config = JSON_VALUE
|
|
```
|
|
|
|
### Agent Using Wiki
|
|
```
|
|
1. Session starts:
|
|
→ internal/handler/session/handler.go
|
|
→ Load session with agent_config
|
|
|
|
2. Agent receives query:
|
|
→ internal/types/agent.go
|
|
→ AgentConfig.KnowledgeBases → resolved to KB objects
|
|
→ For each wiki-enabled KB:
|
|
→ Add wiki tools to available tools
|
|
|
|
3. Agent calls wiki_read_page:
|
|
→ internal/agent/tools/wiki_tools.go
|
|
→ Call WikiPageService.GetPageBySlug()
|
|
→ internal/application/service/wiki_page.go
|
|
→ internal/application/repository/wiki_page.go
|
|
→ SELECT FROM wiki_pages WHERE knowledge_base_id = ? AND slug = ?
|
|
|
|
4. Agent calls wiki_write_page:
|
|
→ internal/agent/tools/wiki_write_page.go
|
|
→ Call WikiPageService.CreatePage()
|
|
→ Same flow as #5-6 in document ingestion
|
|
```
|
|
|
|
---
|
|
|
|
## DIRECT FILE RELATIONSHIPS
|
|
|
|
### KnowledgeBase ↔ WikiConfig
|
|
```
|
|
internal/types/knowledgebase.go::KnowledgeBase
|
|
└─ WikiConfig *types.WikiConfig
|
|
└─ Stored in knowledge_bases.wiki_config (JSON)
|
|
|
|
Update path:
|
|
PUT handler (knowledgebase.go)
|
|
→ Service (knowledgebase.go)
|
|
→ Repository (knowledgebase.go)
|
|
→ Database: UPDATE knowledge_bases SET wiki_config = ?
|
|
```
|
|
|
|
### Knowledge ↔ Chunks
|
|
```
|
|
internal/types/knowledge.go::Knowledge
|
|
└─ FK: knowledge_base_id
|
|
|
|
internal/types/chunk.go::Chunk
|
|
└─ FK: knowledge_id → knowledge.id
|
|
└─ Indexed in multiple backends:
|
|
├─ elasticsearch/v7/repository.go
|
|
├─ elasticsearch/v8/repository.go
|
|
├─ weaviate/repository.go
|
|
├─ qdrant/repository.go
|
|
├─ milvus/repository.go
|
|
├─ postgres/repository.go
|
|
└─ sqlite/repository.go
|
|
```
|
|
|
|
### WikiPage ↔ KnowledgeBase
|
|
```
|
|
internal/types/wiki_page.go::WikiPage
|
|
└─ FK: knowledge_base_id
|
|
└─ Unique constraint: (knowledge_base_id, slug)
|
|
└─ SourceRefs: knowledge IDs that contributed
|
|
```
|
|
|
|
### Agent ↔ KnowledgeBase
|
|
```
|
|
internal/types/agent.go::AgentConfig
|
|
└─ KnowledgeBases []string (KB IDs)
|
|
└─ Resolved at runtime to KnowledgeBase objects
|
|
└─ Tool selection based on KB.IsWikiEnabled()
|
|
```
|
|
|
|
---
|
|
|
|
## SERVICE DEPENDENCY INJECTION
|
|
|
|
### Container (internal/container/container.go)
|
|
```
|
|
Provides instances of all services:
|
|
- KnowledgeBaseService
|
|
- KnowledgeService
|
|
- WikiPageService
|
|
- WikiLintService
|
|
- Retrieval engines
|
|
- Repositories
|
|
- HTTP handlers
|
|
```
|
|
|
|
### Service Constructor Dependencies
|
|
|
|
**KnowledgeBaseService:**
|
|
```go
|
|
repo KnowledgeBaseRepository
|
|
kgRepo KnowledgeRepository
|
|
chunkRepo ChunkRepository
|
|
shareRepo KBShareRepository
|
|
kbShareService KBShareService
|
|
modelService ModelService
|
|
retrieveEngine RetrieveEngineRegistry
|
|
tenantRepo TenantRepository
|
|
fileSvc FileService
|
|
graphEngine RetrieveGraphRepository
|
|
asynqClient TaskEnqueuer
|
|
```
|
|
|
|
**WikiPageService:**
|
|
```go
|
|
repo WikiPageRepository
|
|
chunkRepo ChunkRepository
|
|
kbService KnowledgeBaseService
|
|
redisClient *redis.Client
|
|
```
|
|
|
|
**WikiIngestService:**
|
|
```go
|
|
wikiService WikiPageService
|
|
kbService KnowledgeBaseService
|
|
knowledgeSvc KnowledgeService
|
|
chunkRepo ChunkRepository
|
|
modelService ModelService
|
|
task TaskEnqueuer
|
|
redisClient *redis.Client
|
|
```
|
|
|
|
---
|
|
|
|
## CONFIGURATION FILES & ENVIRONMENT
|
|
|
|
### Backend Config (internal/config/config.go)
|
|
```
|
|
Reads from:
|
|
- .env file
|
|
- Environment variables
|
|
- YAML config files
|
|
|
|
Key config sections:
|
|
- Database (PostgreSQL, MySQL, SQLite)
|
|
- Redis (for async tasks)
|
|
- Retrieval engines (Elasticsearch, Weaviate, etc.)
|
|
- LLM models (OpenAI, Claude, local, etc.)
|
|
- Storage (local, S3, Minio, etc.)
|
|
```
|
|
|
|
### Environment Variables (.env)
|
|
```
|
|
# Database
|
|
DATABASE_URL=...
|
|
|
|
# Redis
|
|
REDIS_URL=...
|
|
|
|
# LLM
|
|
LLM_API_KEY=...
|
|
LLM_MODEL_ID=...
|
|
|
|
# Vector DB
|
|
WEAVIATE_URL=...
|
|
ELASTICSEARCH_URL=...
|
|
|
|
# Storage
|
|
STORAGE_PROVIDER=...
|
|
```
|
|
|
|
---
|
|
|
|
## BUILD & DEPLOYMENT
|
|
|
|
### Binary
|
|
- `server` - Main Go binary (compiled from cmd/desktop/main.go)
|
|
- `WeKnora` - Desktop app version
|
|
|
|
### Docker
|
|
- `Dockerfile` - Backend container
|
|
- `frontend/Dockerfile` - Frontend container
|
|
- `docker-compose.yml` - Full stack
|
|
|
|
### Kubernetes/Helm
|
|
- `helm/` directory - Helm charts
|
|
|
|
---
|
|
|
|
## SUMMARY TABLE
|
|
|
|
| Component | Count | Key Files |
|
|
|-----------|-------|-----------|
|
|
| Models | 7+ | types/*.go |
|
|
| Services | 10+ | service/*.go |
|
|
| Repositories | 10+ | repository/*.go |
|
|
| Handlers | 5+ | handler/*.go |
|
|
| Agent Tools | 13+ | tools/wiki_*.go, knowledge_*.go |
|
|
| Retrieval Backends | 6 | retriever/*/repository.go |
|
|
| Frontend APIs | 2 | api/knowledge-base/, api/wiki/ |
|
|
| Interfaces | 5+ | types/interfaces/*.go |
|
|
|
|
---
|
|
|
|
## QUICK FILE REFERENCE BY QUESTION
|
|
|
|
**Q: Where is WikiConfig defined?**
|
|
A: `internal/types/knowledgebase.go` (line ~94-103)
|
|
|
|
**Q: Where is wiki auto-ingest logic?**
|
|
A: `internal/application/service/wiki_ingest.go` and `wiki_ingest_batch.go`
|
|
|
|
**Q: How are KB settings persisted?**
|
|
A: `internal/handler/knowledgebase.go` → `service/knowledgebase.go` → `repository/knowledgebase.go`
|
|
|
|
**Q: Where do agents access KBs?**
|
|
A: `internal/types/agent.go` (AgentConfig.KnowledgeBases) → resolved in service layer
|
|
|
|
**Q: Where is wiki page model?**
|
|
A: `internal/types/wiki_page.go` (WikiPage struct, ~line 43-84)
|
|
|
|
**Q: How are wiki links maintained?**
|
|
A: `internal/application/service/wiki_linkify.go` and `wiki_page.go`::RebuildLinks
|
|
|
|
**Q: Where is hybrid retrieval implemented?**
|
|
A: `internal/application/service/retriever/keywords_vector_hybrid_indexer.go`
|
|
|
|
**Q: How does chunk embedding happen?**
|
|
A: Multiple backends in `retriever/*/repository.go` (Elasticsearch, Weaviate, etc.)
|
|
|
|
**Q: Frontend API for KB updates?**
|
|
A: `frontend/src/api/knowledge-base/index.ts`::updateKnowledgeBase()
|
|
|
|
**Q: Where is async task handling?**
|
|
A: Asynq queue, triggered in service layer, workers in `wiki_ingest.go`
|
|
|