Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.
- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
(WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.
- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
(default 30m). Without this, a hung docreader pinned a worker for the
full DocumentProcessTimeout window.
- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
failing single image no longer strands the parent — the asynq retry
is allowed to run, but on the final attempt we count the image
regardless of outcome. Redis DECR errors fall back to enqueuing the
post-process task instead of returning silently.
- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
ManualProcess exhausts retries, immediately mark the corresponding
Knowledge as failed with the last error. This surfaces the failure
in the UI without waiting for the housekeeping sweep.
- HousekeepingService: 5-minute cron that flips knowledge rows stuck
in "processing" past DocumentProcessTimeout + 10m to failed, plus
summary rows stuck > 1h. Catches anything the other safety nets
miss (worker SIGKILL mid-handler, etc.). Disable with
WEKNORA_HOUSEKEEPING_ENABLED=false.
- Distributed startup recovery: previously the post-restart sweep was
skipped whenever REDIS_ADDR was set, even though Asynq does not
reschedule the task that was actively running on the dead instance.
Now the sweep runs in distributed mode too, but only against rows
older than 30 minutes to avoid racing peer instances.
* feat(rbac): add multi-use share-link invitations for invite_only mode
When the auth.registration_mode toggle flips to invite_only, the public
/auth/register endpoint returns 403 — but until now there was no
channel for unregistered users to actually join a tenant. The existing
invitation table required a registered invitee, so new emails could
not enter the system at all.
This change introduces a share-link model:
- Owner generates a multi-use registration link with a per-tenant role
- Recipient opens the link, registers with their own email, and is
added to the tenant on the same request
- Token is stored in plaintext (not hashed) so the management UI can
re-display the URL on demand without a "copy now or revoke" trap.
Threat model is bounded by 7-day TTL, revocability, and the fact
that all the link grants is membership in one tenant.
- accepted_count tracks how many users joined through the link so the
Owner can tell whether a link is fresh or has already been spread
Per-user invitations (registered email -> in-app inbox accept) are
unchanged. Share-link rows use empty invitee_user_id as discriminator;
the partial unique index on (tenant_id, invitee_user_id) was relaxed
to skip empty values so multiple share links can coexist per tenant.
Frontend reuses Login.vue for /register?token=xxx instead of a
parallel page, and adds icon-only revoke / remove actions plus a
distinct "active" status tag for share-link rows.
Migration 000054 adds the token column, the accepted_count column,
and the relaxed pending unique index in a single step.
* feat(auth): update invitation token handling and add rate limiting
- Refactor the invitation token retrieval to use POST instead of GET, enhancing security by preventing the plaintext token from appearing in logs and browser history.
- Update the API endpoint from `/auth/invitations/:token` to `/auth/invitations/lookup` to reflect the new method.
- Introduce a rate limiter for unauthenticated share-link endpoints to mitigate brute-force attacks and abuse, ensuring a maximum of 30 requests per minute per IP.
- Adjust the registration flow to maintain user experience while securing the invitation process.
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.
Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.
The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
Replace the English docs/rbac.md with a comprehensive Chinese
docs/RBAC说明.md and a wiki-style summary under docs/wiki/安全认证/.
Explain how tenant RBAC relates to the shared space feature (they
are orthogonal: tenant RBAC is the vertical defense, shared space
is the horizontal collaboration channel) and cross-link the two
docs in both the flat and wiki trees. Update inbound references in
.env.example, docker-compose.yml, and the auth legacy env test to
point at the new file name.
Refactor the tenant RBAC configuration to change the default value from false to true, enabling role enforcement by default. This change allows operators to opt into a logging-only rollout window by explicitly setting the configuration to false.
Updates include:
- Modifications to .env.example and docker-compose.yml to reflect the new default.
- Adjustments in rbac.md documentation to clarify the new default behavior and the opt-in process.
- Code changes across various files to utilize the new pointer-based configuration for EnableRBAC, ensuring nil safety and clearer intent.
No functional changes were introduced; the adjustments primarily enhance clarity and maintainability of the RBAC feature.
DISABLE_REGISTRATION=true used to block /auth/register at the handler
layer only, leaving /auth/config still reporting self_serve. The
frontend therefore kept showing the Register entry even when the env
var was set — clicking it just hit the 403. Two gates, out of sync.
Wire DISABLE_REGISTRATION=true through applyAuthAndTenantDefaults so
it coerces auth.registration_mode to invite_only (env wins over YAML,
matching docs/rbac.md). The handler-side os.Getenv check is now
redundant with IsInviteOnly() and is removed, leaving a single
enforcement path.
Add config tests pinning down the env/YAML matrix, including the
explicit-self_serve override case that would otherwise be the easy
regression to ship.
DISABLE_REGISTRATION=true (handler layer) and
WEKNORA_AUTH_REGISTRATION_MODE=invite_only (config layer) were two
env-level ways of saying the same thing: block /auth/register.
Keeping both invites confusion about which wins and risks operators
setting one while expecting the other.
Remove the WEKNORA_AUTH_REGISTRATION_MODE env override so the env
layer exposes a single knob (DISABLE_REGISTRATION). The
auth.registration_mode config field stays — operators who want the
richer behaviour (frontend hides the registration entry via
/auth/config in addition to the server-side 403) flip it in
config.yaml.
No behaviour change for deployments that did not set the env var.
Update docs/rbac.md to document the deliberate split.
Added a new configuration option to limit the number of tenants a non-superuser can create via self-service. Introduced a new error type for handling cases where users exceed this limit, returning a 429 status code. Updated the tenant creation and update handlers to enforce this limit and provide appropriate feedback to users. Additionally, refactored the tenant update request structure to ensure only mutable fields are allowed.
Refs: #1303
The audit_logs table grows monotonically until something purges it.
PR 6 explicitly listed retention as out of scope for v1; this picks
that up.
Adds an `audit.retention_days` config (default 90, 0 disables) and a
small background goroutine `AuditLogRetentionRunner` that fires once
~10 minutes after boot and then every 24h, calling
`AuditLogService.Purge` which DELETEs rows older than the cutoff in a
single statement. The dependency surface is intentionally minimal —
no robfig/cron, no asynq, no migration — because retention has no
wall-clock alignment requirement.
Wiring is the same shape as the existing data source scheduler:
container `Provide(NewAuditLogRetentionRunner)`, an `Invoke` that
calls `Start`, and a `ResourceCleaner.RegisterWithName` for graceful
shutdown.
Defaults preserve operator intent:
- `audit:` section omitted from YAML -> retention_days=90.
- explicit `audit.retention_days: 0` in YAML -> purge disabled.
- `WEKNORA_AUDIT_RETENTION_DAYS=N` env override (incl. N=0).
- `retention_days < 0` is rejected by ValidateConfig.
Tests cover the service Purge contract (no-op when disabled, cutoff
math, error propagation) and the runner lifecycle (Start no-op when
disabled, idempotent Start/Stop, ticker cadence, runOnce swallows
errors). 10 new tests, all green.
docs/rbac.md updated with the new YAML / env / behaviour.
Refs: #1303
Sweep over the route table found four classes of mutating endpoints
that had no role guard:
- POST /tenants — creating a brand-new tenant. Now g.CrossTenant():
only org-level superusers may create tenants, matching /tenants/all
and /tenants/search.
- POST /initialization/initialize/:kbId, PUT /initialization/config/:kbId
— these change a KB's embedding/parser/storage configuration, which
is at least as sensitive as PUT /knowledge-bases/:id. New
KBCreatorLookupFromKbIDParam (sister of KBCreatorLookup but reading
:kbId) lets us reuse the OwnedKBOrAdmin matrix; routes now use the
new g.OwnedKBOrAdminFromKbIDParam() guard.
- POST /initialization/{ollama,remote,embedding,rerank,asr,multimodal,
extract}/* — system-level model probes / downloads / extraction
pipelines. These trigger LLM calls and outbound network fanout
with tenant credentials; gated to Admin+. The corresponding GETs
(status / list / progress) drop to Viewer+ since they're read-only
observability.
- POST /system/{parser-engines,docreader,storage-engine}/* — same
pattern: GETs that report state are Viewer+, the *-check /
/reconnect actions are Admin+.
POST /evaluation/ runs the eval harness which fans out LLM calls
across the tenant — gated to Admin+.
Operational improvement: log the resolved tenant RBAC state at
LoadConfig completion. air's hot-reload only rebuilds the binary on
Go-source changes; it does NOT re-source .env, so flipping
WEKNORA_TENANT_ENABLE_RBAC=true while the dev loop is already
running has no effect until the parent dev-app process restarts.
This was a real diagnostic loop in PR review — printing the value at
startup makes the "I edited .env but the gates still aren't firing"
trap obvious from the first console line.
The Printf includes both the resolved cfg.Tenant.EnableRBAC value
and the raw env var, so a mismatch between "what the .env file
says" and "what this process actually inherited" is one glance.
Lay the foundation for tenant-level RBAC without enforcing it yet.
Schema, types, middleware lookup, and login-flow shape changes ship now;
RequireRole/route annotations and member-management endpoints follow in
PR 2 and PR 3.
Highlights:
- New tenant_members table (Postgres + SQLite) with (user_id, tenant_id)
unique key and (owner|admin|contributor|viewer) role values. Migration
000043 backfills one row per existing user, marking the earliest-created
active user in each tenant as Owner and the rest as Contributor.
- KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so
PR 2 can wire ownership-based and Viewer-runtime checks.
- Auth middleware loads the active TenantMember row, sets a new
TenantRoleContextKey, and falls back via three escape hatches:
cross-tenant superusers (CanAccessAllTenants) get Admin in the target
tenant; orphan tenants (zero members) auto-promote the first authenticating
human to Owner; and when tenant.enable_rbac is false (the v1 default)
unmatched lookups fail open with Admin to preserve current behaviour.
- Register / OIDC provisioning now insert an Owner membership for the
registrant via TenantMemberService.EnsureOwner.
- LoginResponse renamed tenant -> active_tenant and gained a memberships
array. POST /auth/switch-tenant issues a fresh token pair scoped to a
target membership; GET /auth/config exposes auth.registration_mode for
the frontend register-tab gating.
- New config keys: auth.registration_mode (self_serve | invite_only,
defaults to self_serve) and tenant.enable_rbac (defaults to false).
Both have WEKNORA_* env overrides.
- Frontend store persists memberships and exposes currentTenantRole;
Login.vue calls /auth/config and hides the Register tab in invite_only
mode. Go SDK types updated to match the new LoginResponse shape.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
- Added a new configuration file `agent_type_presets.yaml` to define various agent type presets, including RAG Q&A, Wiki Q&A, Hybrid (Wiki + RAG), and Data Analysis.
- Updated the `builtin_agents.yaml` to associate built-in agents with the new agent type presets, enhancing their configuration options.
- Enhanced the agent system prompt templates to reflect the new agent types and their functionalities.
- Implemented API endpoints to retrieve agent type presets, allowing for dynamic loading in the frontend.
- Updated the frontend to support agent type selection in the agent editor modal, improving user experience by auto-filling relevant configurations based on selected presets.
These changes significantly enhance the flexibility and usability of the agent system, allowing users to easily configure agents based on predefined types.
- Added max_input_chars configuration to limit input size for summary generation.
- Updated generate_summary.yaml to clarify summary generation steps and requirements.
- Introduced a new summary section in the document content view to display generated summaries or loading states.
- Refactored doc-content.vue to utilize computed properties for improved performance and readability.
- Enhanced knowledge base hooks to include description and summary status fields for better data handling.
- Updated internationalization files to include new summary-related labels in multiple languages.
- Updated context template to include runtime metadata such as current time and week, improving contextual awareness in user queries.
- Enhanced rewrite template with critical instructions for intent classification, ensuring that rewritten questions preserve essential entities and keywords.
- Refined intent classification logic to prioritize user intents more effectively, improving the accuracy of responses based on user queries.
- Added examples to clarify expected input and output formats for intent classification, enhancing usability for developers.
- Updated the assistant descriptions in various prompt templates to specify that they are developed by Tencent.
- Added new intent-specific system prompt templates for greeting, chitchat, follow-up, image analysis, summarization, and web search unavailability scenarios.
- Enhanced citation rules in the system prompts to ensure proper inline citation formatting.
- Refactored the rewrite prompt to classify user intent more accurately and output structured JSON responses.
- Introduced a configurable LLM call timeout with a default value, allowing for better control over LLM call durations.
- Added logic to retry transient errors (e.g., timeouts, rate limits) up to a specified maximum number of retries, improving robustness in error handling.
- Implemented parameter casting for tool arguments to ensure correct types are used, addressing common LLM quirks.
- Enhanced tool execution error messages with guidance for retrying with different approaches.
- Added validation for configuration values to prevent runtime errors.
- Added timezone and language settings to the environment configuration.
- Introduced built-in agent configurations with multilingual support for various agents.
- Updated Docker Compose to utilize new environment variables for timezone and language.
- Created new prompt templates for question generation, summary generation, and keywords extraction.
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.
This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
- Added configuration for enabling cross-tenant access in the application.
- Introduced new API endpoints for listing and searching tenants, accessible only to users with the appropriate permissions.
- Updated user and tenant models to include the `can_access_all_tenants` field.
- Enhanced the authentication middleware to handle cross-tenant requests and validate user permissions.
- Created a new TenantSelector component for selecting tenants in the frontend, improving user experience.
- Updated internationalization files to support new tenant-related strings and messages.
- Added support for query expansion in conversation settings, allowing the system to generate additional queries when recall is low.
- Updated localization files in English, Russian, and Chinese to include new strings for the query expansion feature.
- Enhanced UI in AgentSettings.vue to allow users to toggle query expansion settings.
- Refactored related API and service logic to accommodate the new query expansion functionality in session management and chat processing.
- Updated agent configuration to support separate system prompts for web search enabled and disabled states.
- Removed deprecated agent configuration parameters to streamline settings management.
- Enhanced UI components in AgentSettings.vue to allow configuration of custom prompts based on web search status.
- Improved localization in English, Russian, and Chinese for new prompt settings and UI elements.
- Refactored related API and service logic to accommodate changes in agent configuration structure.
- Introduced web search configuration in config.yaml, allowing multiple providers with specific settings.
- Updated InputField component to manage agent mode and web search toggles, improving user interaction.
- Enhanced chat API to include web search status in requests, enabling dynamic query handling.
- Implemented backend support for web search integration, merging results with knowledge base queries.
- Added tenant-level web search configuration management, allowing customization per tenant.