Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
Stop hardcoding DOCREADER_PDF_RENDER_PARALLELISM=4 in compose so low-core
containers use min(4,cpu). Clarify WEKNORA_LANGUAGE precedence and align
docreader JPEG quality docs with the 85 default.
* feat(compose): align env vars in docker-compose and .env.example with code
docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
(CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod
.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY
helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
so GraphRAG was silently broken: setting neo4j.enabled=true gave the
app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment
TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Let users upload documents in chat without ingesting them into a knowledge base: files are parsed in the background, referenced by ID on send, and expire after a configurable TTL.
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
Drop the standalone OIDC_AUTH_DEFAULT_TENANT_MODE / oidc_auth.default_tenant_mode
knob so OIDC auto-provisioning and local password registration share the single
auth.default_tenant_mode policy (resolved by the callback handler). Repair
tenantless sessions by falling back to the first active membership on the login
happy path and stale-preference paths, allow tenantless PUT /auth/me/preferences,
clear the tenant snapshot on /auth/me responses without a tenant, and default the
frontend canCreateTenant capability to fail-closed until /auth/me confirms it.
Allow deployments to register users without auto-creating a personal tenant, expose creation policy via /auth/me, and guide tenantless accounts through a workspace onboarding flow.
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
Add a tenant.auto_create_api_key setting (env
WEKNORA_TENANT_AUTO_CREATE_API_KEY, default false) that restores the
pre-break-change behaviour of minting a full_access API key when a
tenant is created and returning its plaintext token in the create
response (data.api_key).
- system_setting: register tenant.auto_create_api_key bool spec
- handler: gate auto key creation behind the setting; embed the token
via tenantWithAPIKey without re-adding a persisted api_key column;
key-creation failures log a warning and do not fail tenant creation
- client: add optional Tenant.APIKey field populated only on create
- docs/env: document the switch (.env.example, docker-compose,
tenant.md, regenerated swagger)
Increase the default value of WEKNORA_MODEL_MAX_CONCURRENCY from 8 to 32 in .env.example, docker-compose.yml, and related application code. This change enhances the concurrency limit for background tasks, ensuring better performance and resource management across the system.
Add WEKNORA_MODEL_MAX_CONCURRENCY to the example environment file and docker-compose configuration, setting a default value of 8. This change supports the new model concurrency feature for background tasks, ensuring proper configuration across environments.
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.
- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
concurrency, default 16), split from the parse pool (NewParseAsynqServer).
Both run the same mux over disjoint queue sets. Replaces the old
weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
batches per KB; Map/Reduce fan-out sized within a batch. New config knob
+ system setting + swagger docs.
- docreader: serialize all pdfium access behind a global lock. pypdfium2
is process-global and not thread-safe, so concurrent PDF uploads could
corrupt its shared state and deadlock the whole process (requests stuck
in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
Replace 7 hardcoded service host/port values with environment variable
references that preserve the existing defaults, so users can override them
via .env to connect to external infrastructure (DB, Redis, Qdrant, Milvus,
Neo4j) without editing docker-compose.yml.
- DB_DRIVER, DB_HOST, DB_PORT
- QDRANT_HOST
- MILVUS_ADDRESS
- REDIS_ADDR
- NEO4J_URI
- Added a new script `build_frontend_dist.sh` to handle the building of frontend static assets.
- Updated `docker-compose.yml` to include a note about running the build script before building the frontend service.
- Modified the GitHub Actions workflow to integrate the new build script for frontend assets.
- Adjusted the Dockerfile to copy built assets directly from the `dist` directory.
- Refined the `.dockerignore` to exclude unnecessary files while keeping the `dist` directory.
- Added a new build argument `COMMIT_ID_ARG` to pass the commit ID during the Docker build process.
- Updated the Dockerfile to set an environment variable `VITE_FRONTEND_COMMIT` with the commit ID.
- Enhanced the Vite configuration to resolve the frontend commit ID from the environment or Git.
- Modified the SystemInfo component to display the frontend commit ID in the UI.
- Implemented a version info preparation step in the GitHub Actions workflow to capture the commit ID.
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
- Increased the default worker pool size for the asynq server from 16 to 32 to improve performance during concurrent uploads.
- Updated the .env.example and docker-compose.yml files to reflect the new default concurrency setting.
- Enhanced the handling of FAQ entries in the tool results, ensuring distinct identification through standard questions, improving user experience and clarity in the output.
- Refactored related components to utilize the new FAQ attributes, ensuring consistent display and interaction across the application.
Vector store create and raw connection-test accepted user-supplied
connection addresses without SSRF validation, while every other
admin-entered endpoint (e.g. model BaseURL) already runs
ValidateURLForSSRF. An admin could point an engine address at an
internal host and use the connection probe as an SSRF oracle.
Add validateConnectionAddrSSRF, applied at the two user-input
boundaries only:
- CreateStore: validated before any network dial.
- TestStoreRaw: a new TestRawConnection wrapper runs an engine-type
allowlist, required-field validation, and SSRF validation before
delegating to TestConnection. TestConnection stays validation-free
for trusted callers (env stores, stored configs already validated
at create time).
Validation covers every address-bearing field a driver dials
(Addr, qdrant Host+Port, weaviate Host and GrpcAddress) and
fails closed for unmapped engine types. The Elasticsearch probe no
longer follows redirects, closing a 302-to-internal bypass.
docker-compose whitelists the bundled vector-store service hostnames
(qdrant, milvus, weaviate, doris-fe) via SSRF_WHITELIST_EXTRA so the
out-of-box experience is preserved; postgres is intentionally not
whitelisted. .env.example and the OpenSearch dev guide document how
to whitelist external or local-dev addresses.
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
PR #1453 introduced the map form of env_file:
env_file:
- path: .env
required: false
which is only recognised by Docker Compose v2.24+ (Jan 2024). Older
deploys would refuse to parse docker-compose.yml even when builtin_models
YAML is not enabled — breaking the "no impact unless opted in" promise.
Switch to the array form (`env_file: [.env]`) which has been supported
since the earliest Compose schemas. The trade-off is that Compose now
errors when .env is absent, so:
- scripts/start_all.sh already calls check_env_file (cp .env.example .env
if missing) before docker compose up; that path is unaffected.
- The bare `make docker-run` / `make docker-restart` targets in the
Makefile are taught the same fallback: touch / copy .env.example before
invoking docker-compose, so fresh clones keep working.
The docker-compose.yml comment block is updated to explain the version
trade-off so future maintainers don't re-introduce the map form.
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.
Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.
The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
Replace the English docs/rbac.md with a comprehensive Chinese
docs/RBAC说明.md and a wiki-style summary under docs/wiki/安全认证/.
Explain how tenant RBAC relates to the shared space feature (they
are orthogonal: tenant RBAC is the vertical defense, shared space
is the horizontal collaboration channel) and cross-link the two
docs in both the flat and wiki trees. Update inbound references in
.env.example, docker-compose.yml, and the auth legacy env test to
point at the new file name.
Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.
Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.
Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:
* auto (default/recommended): probe Doris server on first use to detect available vector
functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
(older builds lacking inner_product_approximate)
* legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
builds without inner_product_approximate support)
* inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
(for modern Doris 4.0+ and current SelectDB with normalized embeddings)
Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level
Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
Refactor the tenant RBAC configuration to change the default value from false to true, enabling role enforcement by default. This change allows operators to opt into a logging-only rollout window by explicitly setting the configuration to false.
Updates include:
- Modifications to .env.example and docker-compose.yml to reflect the new default.
- Adjustments in rbac.md documentation to clarify the new default behavior and the opt-in process.
- Code changes across various files to utilize the new pointer-based configuration for EnableRBAC, ensuring nil safety and clearer intent.
No functional changes were introduced; the adjustments primarily enhance clarity and maintainability of the RBAC feature.
Surface two existing config.go env overrides to the canonical
deployment artifacts so operators can flip them without reading the
Go source:
* WEKNORA_TENANT_ENABLE_RBAC — observe / enforce switch for
tenant-level role enforcement (PR 1303). Default false keeps the
current behaviour; flip to true once role assignments have been
audited per docs/rbac.md.
* WEKNORA_TENANT_MAX_OWNED_PER_USER — cap on tenants a single
non-superuser can self-create. Uses the existing <0 / 0 / >0
sentinel semantics documented on TenantConfig.MaxOwnedPerUser.
docker-compose.yml passes both through to the app container, and
.env.example gains a "Tenant / RBAC" section with the default
values and the same sentinel rules inline so the example is the
sole reference operators need.
No functional change — both env vars were already honoured by
config.go.applyAuthAndTenantDefaults.
Follow-up to #1359. Addresses a set of correctness and security gaps in
the initial docreader auth implementation.
- docker-compose: inject GRPC_TLS_*/GRPC_TLS_SERVER_NAME/GRPC_AUTH_TOKEN
into the WeKnora-app service. Without this the Go client never saw the
knobs, so enabling token auth on the server broke every RPC.
- client: bind tokenAuth.RequireTransportSecurity() to TLSEnabled so a
bearer token cannot be sent over an insecure channel once TLS is on.
- server: load_tls_credentials now raises TLSConfigError on misconfig
(cert/key missing, file unreadable, mTLS without CA); main.py exits 1
instead of silently downgrading to insecure.
- server: replace endswith("/Check"|"/Watch") health bypass with exact
match against /grpc.health.v1.Health/{Check,Watch}.
- server: compare tokens with hmac.compare_digest, warn on tokens < 16B.
- server: AuthInterceptor now returns an abort handler matching the
original RPC kind (unary/stream) and uses context.abort, so streaming
RPCs surface UNAUTHENTICATED instead of INTERNAL.
- internal/infrastructure/docparser/grpc_parser.go: drop the duplicated
TLS/tokenAuth block and reuse docreader/client.LoadAuthConfigFromEnv +
BuildDialOptions. Single source of truth for client-side auth.
- Add GRPC_TLS_SERVER_NAME (client SNI override) and
GRPC_MTLS_REQUIRE_CLIENT_CERT (server explicit mTLS toggle); document
the differing CA semantics between client and server in .env*.example.
- Reject half-configured client mTLS (cert XOR key) loudly.
- Fix missing trailing newline in .env.lite.example.
Verified locally: go build ./... and go vet ./... clean; auth.py
fail-fast / token paths smoke-tested.
The docreader gRPC service has no authentication or TLS, but the
default `ports: ["50051:50051"]` mapping in docker-compose.yml binds
to 0.0.0.0, exposing an unauthenticated document parser (with URL
fetch capability) on every host interface.
The app container reaches docreader through the internal Docker
network via `docreader:50051` (the default `DOCREADER_ADDR`), so the
host port mapping is not required for normal operation.
Replace the `ports` entry with `expose: ["50051"]` so the port is
only reachable inside the WeKnora-network. Operators who need to
call docreader from the host (for debugging, etc.) can re-add a
`ports:` entry in a local override, preferably bound to 127.0.0.1.
`${SEARXNG_SECRET:?...}` made the variable mandatory at compose parse time,
which forced *any* compose command (default profile included) to fail when
SEARXNG_SECRET was unset, with a message confusingly claiming the searxng
profile was being started.
Switch to `${SEARXNG_SECRET:-weknora-default-searxng-secret-...}` so the
searxng profile starts zero-config. Default deployments bind searxng to
127.0.0.1 only, so a shared default secret is acceptable; .env.example
now explicitly warns to rotate it before flipping SEARXNG_BIND=0.0.0.0,
since secret_key signs image-proxy URLs.
- Updated .env.example to clarify SEARXNG_SECRET generation and added SSRF_WHITELIST_EXTRA for improved security.
- Modified docker-compose files to bind SearXNG to localhost by default and introduced a one-time initialization service to set up settings.yml correctly.
- Enhanced SearxngProvider with stricter URL validation, ensuring no query or fragment is present in the base URL.
- Added unit tests for SearXNG validation and date parsing to ensure robustness.
- Updated frontend WebSearchSettings to reflect changes in SearXNG instance URL handling.
This commit improves the security and usability of the SearXNG integration, addressing potential misconfigurations and enhancing the developer experience.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
Closes#620#497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
Previously MINIO_ENDPOINT was hardcoded to minio:9000 in docker-compose.yml,
preventing users from connecting to an external MinIO service. Now it supports
override via .env while keeping the same default for backward compatibility.
wizardchen