First half of the gated OpenSearch k-NN driver introduced in PR 1 (#1445) by way of #1440. PR 2a ships a hollow, interface-compliant shell of the `internal/application/repository/retriever/opensearch/` package — every behavioural method (Save / BatchSave / DeleteBy* / Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* / EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled` or a conservative sentinel value. PR 2b lands the real read/write implementations in dedicated files (`query.go` + `retrieve.go` + `crud.go`) and replaces the stubs accordingly. Strict feature-gate (unchanged from PR 1): no entry is added to validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping / BuildEnvVectorStores / container env path / engine factory switch, so the driver remains unreachable. Attempting to register an `engine_type=opensearch` VectorStore continues to fail with the existing "not a valid engine type" error. What lands in PR 2a ------------------- Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC): - `repository.go` — Repository struct + NewRepository constructor that validates cluster reachability + OS version (2.4+ / 3.x; primary tested 3.3.2) + k-NN plugin presence on every cluster node. sync.Once-guarded ensureReady(ctx, dim) for lazy per- dimension index creation, with transient errors not cached so a momentary cluster blip does not permanently poison a dim. sanitizeIndexName enforces a strict OS-compatible name spec. probeVersion uses robust strings.Split/Atoi parsing for pre-release suffixes and missing-patch versions. EngineType returns the PR 1 constant; Support returns [keywords, vector]. - `transport.go` — newOpenSearchClient ships TLS posture (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy- only cipher list) and transport tuning for the driver. Caller exists only in PR 3 (container.go + engine_factory.go); PR 2a remains gated dead code. - `mapping.go` — buildIndexMapping(cfg, dim) produces the full knn_vector + HNSW + content-analyzer mapping with every *_id field as an explicit keyword and source_type as integer. buildKeywordsMapping ships the dim-less keyword-only index mapping used by the no-embedding save path. createIndexAndAlias creates <alias>_v1 and aliases <alias> to it, with best-effort orphan cleanup and mapping-drift detection. - `config.go` — internalCfg (value type) applying OpenSearch defaults (hnsw_m=16, ef_construction=100, ef_search=100, shards=4, replicas=1, engine=lucene). - `errors.go` — nine sentinels (ErrIndexNotFound, ErrDimensionMismatch, ErrAuth, ErrTransport, ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled, ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports apperrors; PR 3's engine factory wraps these to typed AppError 2200/2201. - `stubs.go` — every behavioural method returns ErrFeatureNotEnabled. EstimateStorageSize returns a conservative HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard fails-closed for non-empty KBs. Tests (~1115 LoC, 50 cases): - `repository_test.go` — interface satisfaction, sentinel mapping, sanitizeIndexName positive/negative matrix, semver parsing (pre-release / missing-patch), buildIndexMapping JSON shape pin (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 / 2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin multi-node coverage, ensureReady concurrency + per-dim isolation + transient retry, NewRepository storeID validation, all 11 stubs (CopyIndices, BatchUpdate*, EstimateStorageSize, SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*), wrapTransport sentinel mapping + leak guard, isNotFound / isAlreadyExistsError, drainAndClose / limitedDecode helpers. - `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify / TLS 1.2 pinning / cipher list / transport tuning. Single dependency addition: github.com/opensearch-project/ opensearch-go/v4 v4.6.0 in go.mod/go.sum. SDK quirks discovered (opensearch-go v4.6.0) -------------------------------------------- PR 2a includes the workarounds for two of three SDK limitations that landed during full implementation (the third, Refresh:*bool, only affects the delete path that ships in PR 2b): - AliasExists method passes dataPointer=nil to its internal do(), which means non-2xx responses come back as a plain *errors.errorString ("status: 404 Not Found") rather than as *opensearch.StructError. aliasExists therefore inspects resp.StatusCode directly (resp is returned even when err is non-nil) and only falls back to wrapTransport for the "no response at all" case. - sync.OnceReset is not in the standard library; the keyword-only index uses a mutex + ready/err flag pattern so transient failures can be retried by the next caller. The per-dimension path uses the `once map[int]*sync.Once` delete-and-recreate trick. Test fixes folded in -------------------- While doing a full `go test ./...` against PR 1-merged main, two deterministic regressions surfaced that block a clean run-everything signal. Both are unrelated to the driver and are folded into PR 2a so the PR's own CI run is green: (1) Follow-up to #1445 — fanout test missed the new normalizer policy (internal/application/service/knowledgebase_search_fanout_test.go, +46 / -6). #1445 changed EngineAwareNormalizer for ES / ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01 passthrough (those engines surface non-negative cosine to the normalizer per Lucene script_score non-negative invariant for ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for OpenSearch, engine-internal or IR-normalized conversions for the rest). Milvus is now the only engine that still surfaces raw signed cosine in [-1, 1]. TestRetrieveFromStores_MixedEngine_Normalizes still asserted the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3) which under passthrough now becomes clamp01(-0.4) = 0. The normalizer's own _test.go was updated at #1445 time, but this fan-out integration test was not. Fix: rewrite the godoc to spell out the two engine groups; restate sub-case 2 as ES passthrough on a production-possible mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3 pinning the cosine-shift branch via Milvus -0.4 → 0.3. (2) Pre-existing — SSRF whitelist singleton race surfaced by this run (internal/utils/security.go + internal/utils/security_test.go + internal/infrastructure/web_search/searxng_test.go, +33 / -9). loadSSRFWhitelist in internal/utils/security.go is cached via sync.Once on first call. The internal reset helper resetSSRFWhitelistForTest was unexported, so tests in other packages could not reset and saw whatever whitelist was cached by the first sync.Once.Do() in the same test binary. In internal/infrastructure/web_search/, TestValidateProxyURL runs before TestValidateSearxngBaseURL alphabetically and exercises ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty whitelist; the later setenv in searxng_test then has no effect and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted". Pre-existing on main; surfaced now because this PR was the first to do a full `go test ./...` run on top of #1445. Fix: capitalize the helper to ResetSSRFWhitelistForTest (the ForTest suffix is the test-only contract); update in-package callers; in web_search/searxng_test.go import internal/utils and call ResetSSRFWhitelistForTest around the env mutation in both TestValidateSearxngBaseURL and TestSearxngProvider_Search. No production code path changes. Roadmap ------- - PR 2b (next, depends on this PR) — read/write implementations: query.go + retrieve.go + crud.go land their real bodies; stubs for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are removed; corresponding CRUD/retrieve/filter test cases (~430 LoC) join repository_test.go. - PR 3 — activation switch + async paths (CopyIndices, BatchUpdate*, large-batch async deletes) + i18n + docker-compose dev profile. After PR 3 merges, the OpenSearch driver becomes reachable via either `engine_type=opensearch` VectorStore or `RETRIEVE_DRIVER=opensearch` env. Backward compatibility ---------------------- - New package — additive only. No existing file modified except go.mod / go.sum, the two test files in (1)/(2), and the test-only export rename in utils/security.go. - Driver is unreachable: no registry path activates it. - No SQL migration. - The PR 1 normalizer case for OpenSearch remains unreachable here (no driver instance produces a result yet). Test plan --------- - [x] go build ./... clean - [x] go vet ./... clean - [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes - [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/ shows only PR 1's normalizer case + this driver's EngineType() and tests — no activation path. - [x] grep -r "case \"opensearch\"" internal/ shows no hits.
| English | 简体中文 | 日本語 | 한국어 |
Overview • Architecture • Key Features • Getting Started • API Reference • Developer Guide
💡 WeKnora — Turn Documents into Living Knowledge with RAG, Agents and Auto-Wiki
📌 Overview
WeKnora is an open-source, LLM-powered knowledge framework built for enterprise-grade document understanding, semantic retrieval, and autonomous reasoning.
It is organized around three core capabilities: RAG-based Quick Q&A for everyday lookups, a ReAct Agent that autonomously orchestrates retrieval, MCP tools and web search to handle complex multi-step tasks, and a brand-new Wiki Mode in which agents distill raw documents into a self-maintaining, interlinked markdown knowledge base with an interactive knowledge graph. Combined with multi-source ingestion (Feishu / Notion / Yuque, and growing), 20+ LLM provider integrations, full Langfuse observability, enterprise-ready multi-tenant RBAC (4-tier role matrix + per-resource ownership + per-tenant audit log), and a fully self-hostable modular architecture, WeKnora turns scattered documents into a queryable, reasoning-capable, continuously evolving knowledge asset.
The framework supports auto-syncing knowledge from Feishu, Notion, and Yuque (more data sources coming soon), handles 10+ document formats including PDF, Word, images, and Excel, and can serve Q&A directly through IM channels like WeCom, Feishu, Slack, and Telegram. It is compatible with major LLM providers including OpenAI, DeepSeek, Qwen (Alibaba Cloud), Zhipu, Hunyuan, Gemini, MiniMax, NVIDIA, and Ollama. Its fully modular design allows swapping LLMs, vector databases, and storage backends, with support for local and private cloud deployment ensuring complete data sovereignty. WeKnora also integrates with Langfuse for comprehensive observability into agent reasoning, token usage, and pipeline tracing.
✨ Latest Updates
- v0.6.0 — Tenant RBAC (4-tier role matrix
Owner/Admin/Contributor/Viewer+ per-KB ownership + per-tenant audit log), tenant member management & multi-workspace UX, self-service workspaces;weknoraCLI v0.4 GA withmcp serve; KB retrieval fan-out across vector stores; AES-256-GCM credential encryption + docreader gRPC TLS + Token; Zhipu embedder + Huawei OBS; server-side user preferences; Go 1.26.0. Seedocs/RBAC说明.mdandCHANGELOG.md. - v0.5.2 — Wiki ingest scales to 40k-document KBs (task queue + DLQ); MCP human-in-the-loop tool approval; Anthropic / Apache Doris / Tencent VectorDB / KS3 / SearXNG backends; adaptive 3-tier chunking with live preview; global ⌘K command palette; Yuque connector + WeChat Mini Program;
weknoraCLI preview. - v0.5.1 — Knowledge-base batch management; tenant-wide IM channels overview; session search + user-scoped pinning; unified Model / Web Search / MCP settings cards; per-agent LLM timeout; desktop tenant switching.
- v0.5.0 — Wiki Mode GA — agents auto-generate structured, interlinked Markdown wiki pages with a knowledge graph; wiki browser + visual graph in the UI.
- v0.4.0 — WeKnora Cloud (hosted LLM + parsing); Chrome Extension; ClawHub Skill; WeChat IM; attachment processing; Azure OpenAI / Alibaba OSS; Notion connector; Baidu + Ollama web search; VectorStore management.
- v0.3.6 — ASR (audio); Feishu data-source auto-sync; OIDC; IM quote-reply context + thread-based sessions; document summarization; Tavily search; parallel tool calling; agent @mention scope restriction.
- v0.3.5 — Telegram / DingTalk / Mattermost IM; IM slash commands + QA queue; suggested questions; VLM auto-describe MCP tool images; Novita AI; channel tracking.
- v0.3.4 — WeCom / Feishu / Slack IM; multimodal image support; NVIDIA model API; Weaviate; AWS S3; AES-256-GCM API-key encryption; built-in MCP service; hybrid-search optimization;
final_answertool. - v0.3.3 — Parent-child chunking; KB pinning; fallback response; passage cleaning for rerank; storage auto-creation; Milvus.
- v0.3.2 — Knowledge Search entry; per-source parser & storage engine config; image rendering in local storage; document preview; Volcengine TOS; Mermaid rendering; batch session management; memory graph preview.
- v0.3.0 — Shared Space; Agent Skills + sandboxed execution; custom agents; Data Analyst agent; thinking mode; Bing / Google web search; API Key auth; Helm chart; Korean i18n; Qdrant.
- v0.2.0 — Agent Mode (ReACT); multi-type knowledge bases (FAQ + document); conversation strategy config; DuckDuckGo web search; MCP tool integration; new UI with agent mode switching; MQ async task management.
📱 Interface Showcase
💬 Intelligent Q&A Conversation![]() |
|
📖 Wiki Browser![]() |
🕸️ Wiki Knowledge Graph![]() |
🤖 Agent Mode · Tool Call Process![]() |
⚙️ Conversation Settings![]() |
🔭 Observability · Langfuse Tracing![]() |
|
🏗️ Architecture
Fully modular pipeline from document parsing, vectorization, and retrieval to LLM inference — every component is swappable and extensible. Supports local / private cloud deployment with full data sovereignty and a zero-barrier Web UI for quick onboarding.
🧩 Feature Overview
Intelligent Conversation
| Capability | Details |
|---|---|
| Intelligent Reasoning | ReACT progressive multi-step reasoning, autonomously orchestrating knowledge retrieval, MCP tools, and web search; custom agent support |
| Quick Q&A | RAG-based Q&A over knowledge bases for fast and accurate answers |
| Wiki Mode | Agent-driven auto-generation of structured, interlinked markdown Wiki pages from raw documents |
| Tool Calling | Built-in tools, MCP tools, web search |
| Conversation Strategy | Online Prompt editing, retrieval threshold tuning, multi-turn context awareness |
| Suggested Questions | Auto-generated question suggestions based on knowledge base content |
Knowledge Management
| Capability | Details |
|---|---|
| Knowledge Base Types | FAQ / Document / Wiki with folder import, URL import, tag management, and online entry |
| Data Source Import | Auto-sync from Feishu / Notion / Yuque (more data sources coming soon); incremental and full sync |
| Document Formats | PDF / Word / Txt / Markdown / HTML / Images / CSV / Excel / PPT / JSON |
| Retrieval Strategies | BM25 sparse / Dense retrieval / GraphRAG / parent-child chunking / multi-dimensional indexing |
| E2E Testing | Full-pipeline visualization with recall hit rate, BLEU / ROUGE metric evaluation |
Integrations & Extensions
| Capability | Details |
|---|---|
| LLMs | OpenAI / Azure OpenAI / Anthropic (Claude) / DeepSeek / Qwen (Alibaba Cloud) / Zhipu / Hunyuan / Doubao (Volcengine) / Gemini / MiniMax / NVIDIA / Novita AI / SiliconFlow / OpenRouter / Ollama |
| Embeddings | Ollama / BGE / GTE / Zhipu / OpenAI-compatible APIs |
| Vector DBs | PostgreSQL (pgvector) / Elasticsearch / Milvus / Weaviate / Qdrant / Apache Doris / Tencent VectorDB |
| Object Storage | Local / MinIO / AWS S3 / Volcengine TOS / Alibaba Cloud OSS / Kingsoft Cloud KS3 / Huawei Cloud OBS |
| IM Channels | WeCom / Feishu / Slack / Telegram / DingTalk / Mattermost / WeChat |
| Web Search | DuckDuckGo / Bing / Google / Tavily / Baidu / Ollama / SearXNG |
Platform
| Capability | Details |
|---|---|
| Deployment | Local / Docker / Kubernetes (Helm) with private and offline support |
| UI | Web UI / RESTful API / CLI (weknora) / Chrome Extension / WeChat Mini Program |
| Access Control | Tenant RBAC with 4-tier role matrix (Owner / Admin / Contributor / Viewer), per-KB resource ownership, per-tenant audit log, invite-only workspaces, self-service tenant creation, cross-tenant superuser |
| Security | AES-256-GCM at-rest encryption for API keys and MCP / data-source credentials with graceful key rotation; gRPC TLS + Token between app and docreader; SSRF-safe HTTP client; sandbox isolation for agent skills |
| Observability | Integrated Langfuse for ReAct loops, token tracking, tool calls, and pipeline tracing |
| Task Management | MQ async tasks, automatic database migration on version upgrade |
| Model Management | Centralized config, per-knowledge-base model selection, multi-tenant built-in model sharing, WeKnora Cloud hosted models and parsing |
🧩 Chrome Extension
WeKnora Chrome Extension lets you capture web content directly into your WeKnora knowledge base. Select text, images, or entire pages in the browser and save them as knowledge entries with one click — no copy-paste or file upload needed.
📱 WeChat Mini Program
The WeKnora Mini Program provides a lightweight mobile client for configuring WeKnora API access, selecting knowledge bases, importing URLs, and asking knowledge chat from WeChat.
🦞 ClawHub Skill
WeKnora ClawHub Skill is a WeKnora skill published on the ClawHub platform. Once installed, it enables document import (file / URL / Markdown), hybrid search (vector + keyword) across knowledge bases, and knowledge entry management — all through the WeKnora REST API.
- Document Import — Upload files, import web pages, or write Markdown knowledge via the agent
- Hybrid Search — Search within or across knowledge bases with vector + keyword retrieval
- Knowledge Management — List, browse, edit, and delete knowledge entries programmatically
⌨️ Command-Line Interface
weknora is the official CLI for driving the API from a terminal or AI agent.
The command surface mirrors gh CLI's <noun> <verb> convention; output is
human-readable by default and switches to a stable JSON envelope with --json.
weknora auth login --host https://kb.example.com
weknora kb list
weknora link --kb my-knowledge-base # bind the current directory
weknora doc upload notes.md
weknora chat "summarise the design doc"
See cli/README.md for install + 5-minute quickstart
and cli/AGENTS.md for the operational contract that
AI agents (Claude Code, Cursor, Aider, …) can rely on.
🚀 Getting Started
🛠 Prerequisites
📦 Installation & Launch
git clone https://github.com/Tencent/WeKnora.git
cd WeKnora
cp .env.example .env # Edit .env as needed, see comments in the file
docker compose up -d # Start core services
Once started, visit http://localhost to get started.
To use a local Ollama model, run
ollama serve > /dev/null 2>&1 &first.
🔧 Optional Services (Docker Compose Profiles)
Add --profile flags to enable additional components. Multiple profiles can be combined:
| Profile | Description | Command |
|---|---|---|
| (default) | Core services | docker compose up -d |
full |
All features | docker compose --profile full up -d |
neo4j |
Knowledge Graph (Neo4j) | docker compose --profile neo4j up -d |
minio |
Object Storage (MinIO) | docker compose --profile minio up -d |
langfuse |
Tracing (Langfuse) | docker compose --profile langfuse up -d |
Combine profiles: docker compose --profile neo4j --profile minio up -d
Stop services: docker compose down
🌐 Service URLs
| Service | URL |
|---|---|
| Web UI | http://localhost |
| Backend API | http://localhost:8080 |
| Langfuse Tracing | http://localhost:3000 |
MCP Server
Please refer to the MCP Configuration Guide for the necessary setup.
🔌 Using WeChat Dialog Open Platform
WeKnora serves as the core technology framework for the WeChat Dialog Open Platform, providing a more convenient usage approach:
- Zero-code Deployment: Simply upload knowledge to quickly deploy intelligent Q&A services within the WeChat ecosystem, achieving an "ask and answer" experience
- Efficient Question Management: Support for categorized management of high-frequency questions, with rich data tools to ensure accurate, reliable, and easily maintainable answers
- WeChat Ecosystem Integration: Through the WeChat Dialog Open Platform, WeKnora's intelligent Q&A capabilities can be seamlessly integrated into WeChat Official Accounts, Mini Programs, and other WeChat scenarios, enhancing user interaction experiences
📘 API Reference
Troubleshooting FAQ: Troubleshooting FAQ
Detailed API documentation is available at: API Docs
Product plans and upcoming features: Roadmap
🧭 Developer Guide
⚡ Fast Development Mode (Recommended)
If you need to frequently modify code, you don't need to rebuild Docker images every time! Use fast development mode:
# Start infrastructure
make dev-start
# Start backend (new terminal)
make dev-app
# Start frontend (new terminal)
make dev-frontend
Development Advantages:
- ✅ Frontend modifications auto hot-reload (no restart needed)
- ✅ Backend modifications quick restart (5-10 seconds, supports Air hot-reload)
- ✅ No need to rebuild Docker images
- ✅ Support IDE breakpoint debugging
Detailed Documentation: Development Environment Quick Start
🤝 Contributing
Welcome to submit Issues or Pull Requests.
Process: Fork → Create branch → Commit changes → Open PR
Standards: Format code with gofmt, follow Conventional Commits (feat: / fix: / docs: / test: / refactor:)
🔒 Security Notice
Important: Starting from v0.1.3, WeKnora includes login authentication functionality to enhance system security. For production deployments, we strongly recommend:
- Deploy WeKnora services in internal/private network environments rather than public internet
- Avoid exposing the service directly to public networks to prevent potential information leakage
- Configure proper firewall rules and access controls for your deployment environment
- Regularly update to the latest version for security patches and improvements
👥 Contributors
Thanks to these excellent contributors:
📄 License
This project is licensed under the MIT License. You are free to use, modify, and distribute the code with proper attribution.







