Commit Graph

37 Commits

Author SHA1 Message Date
wizardchen 0e1282c2da test: add document process task options tests
Centralize DefaultDocumentProcessTimeout in the config package and
reuse DocumentProcessTimeout() from documentProcessTaskOptions.
2026-05-28 20:27:53 +08:00
wizardchen 3ae3ea97c5 fix(knowledge): prevent documents from getting stuck in "processing"
Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.

- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
  WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
  CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
  (WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.

- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
  (default 30m). Without this, a hung docreader pinned a worker for the
  full DocumentProcessTimeout window.

- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
  failing single image no longer strands the parent — the asynq retry
  is allowed to run, but on the final attempt we count the image
  regardless of outcome. Redis DECR errors fall back to enqueuing the
  post-process task instead of returning silently.

- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
  ManualProcess exhausts retries, immediately mark the corresponding
  Knowledge as failed with the last error. This surfaces the failure
  in the UI without waiting for the housekeeping sweep.

- HousekeepingService: 5-minute cron that flips knowledge rows stuck
  in "processing" past DocumentProcessTimeout + 10m to failed, plus
  summary rows stuck > 1h. Catches anything the other safety nets
  miss (worker SIGKILL mid-handler, etc.). Disable with
  WEKNORA_HOUSEKEEPING_ENABLED=false.

- Distributed startup recovery: previously the post-restart sweep was
  skipped whenever REDIS_ADDR was set, even though Asynq does not
  reschedule the task that was actively running on the dead instance.
  Now the sweep runs in distributed mode too, but only against rows
  older than 30 minutes to avoid racing peer instances.
2026-05-28 15:14:45 +08:00
lyingbug 3611a8607d feat(rbac): add multi-use share-link invitations for invite_only mode (#1504)
* feat(rbac): add multi-use share-link invitations for invite_only mode

When the auth.registration_mode toggle flips to invite_only, the public
/auth/register endpoint returns 403 — but until now there was no
channel for unregistered users to actually join a tenant. The existing
invitation table required a registered invitee, so new emails could
not enter the system at all.

This change introduces a share-link model:

- Owner generates a multi-use registration link with a per-tenant role
- Recipient opens the link, registers with their own email, and is
  added to the tenant on the same request
- Token is stored in plaintext (not hashed) so the management UI can
  re-display the URL on demand without a "copy now or revoke" trap.
  Threat model is bounded by 7-day TTL, revocability, and the fact
  that all the link grants is membership in one tenant.
- accepted_count tracks how many users joined through the link so the
  Owner can tell whether a link is fresh or has already been spread

Per-user invitations (registered email -> in-app inbox accept) are
unchanged. Share-link rows use empty invitee_user_id as discriminator;
the partial unique index on (tenant_id, invitee_user_id) was relaxed
to skip empty values so multiple share links can coexist per tenant.

Frontend reuses Login.vue for /register?token=xxx instead of a
parallel page, and adds icon-only revoke / remove actions plus a
distinct "active" status tag for share-link rows.

Migration 000054 adds the token column, the accepted_count column,
and the relaxed pending unique index in a single step.

* feat(auth): update invitation token handling and add rate limiting

- Refactor the invitation token retrieval to use POST instead of GET, enhancing security by preventing the plaintext token from appearing in logs and browser history.
- Update the API endpoint from `/auth/invitations/:token` to `/auth/invitations/lookup` to reflect the new method.
- Introduce a rate limiter for unauthenticated share-link endpoints to mitigate brute-force attacks and abuse, ensuring a maximum of 30 requests per minute per IP.
- Adjust the registration flow to maintain user experience while securing the invitation process.
2026-05-27 19:42:01 +08:00
jackson.jia d439b3ae07 feat(builtin-models): add YAML-based declarative config with ${ENV} interpolation
Allow built-in models to be declared in config/builtin_models.yaml
instead of inserting rows via SQL. On every startup the file is read
and each entry is UPSERT-ed into the models table (is_builtin=true)
by stable id.

Any string field may reference an environment variable with ${NAME}.
Unset variables are left as the literal placeholder so
misconfiguration surfaces clearly in provider calls rather than
failing silently with an empty token.

The file is optional: missing file, parse errors, and per-entry
upsert failures all log a warning without aborting startup.
docker-compose.yml adds env_file (.env, required:false) so
deployment-specific variables are passed through automatically.
2026-05-26 11:31:01 +08:00
wizardchen 7ebb29cd3e docs: add Chinese RBAC guide and link with shared space docs
Replace the English docs/rbac.md with a comprehensive Chinese
docs/RBAC说明.md and a wiki-style summary under docs/wiki/安全认证/.
Explain how tenant RBAC relates to the shared space feature (they
are orthogonal: tenant RBAC is the vertical defense, shared space
is the horizontal collaboration channel) and cross-link the two
docs in both the flat and wiki trees. Update inbound references in
.env.example, docker-compose.yml, and the auth legacy env test to
point at the new file name.
2026-05-21 12:28:31 +08:00
wizardchen 7444c2190d chore(rbac): update default behavior for tenant RBAC configuration
Refactor the tenant RBAC configuration to change the default value from false to true, enabling role enforcement by default. This change allows operators to opt into a logging-only rollout window by explicitly setting the configuration to false.

Updates include:
- Modifications to .env.example and docker-compose.yml to reflect the new default.
- Adjustments in rbac.md documentation to clarify the new default behavior and the opt-in process.
- Code changes across various files to utilize the new pointer-based configuration for EnableRBAC, ensuring nil safety and clearer intent.

No functional changes were introduced; the adjustments primarily enhance clarity and maintainability of the RBAC feature.
2026-05-18 21:24:28 +08:00
wizardchen 49f3ad185a fix(auth): make DISABLE_REGISTRATION drive registration_mode too
DISABLE_REGISTRATION=true used to block /auth/register at the handler
layer only, leaving /auth/config still reporting self_serve. The
frontend therefore kept showing the Register entry even when the env
var was set — clicking it just hit the 403. Two gates, out of sync.

Wire DISABLE_REGISTRATION=true through applyAuthAndTenantDefaults so
it coerces auth.registration_mode to invite_only (env wins over YAML,
matching docs/rbac.md). The handler-side os.Getenv check is now
redundant with IsInviteOnly() and is removed, leaving a single
enforcement path.

Add config tests pinning down the env/YAML matrix, including the
explicit-self_serve override case that would otherwise be the easy
regression to ship.
2026-05-18 21:15:01 +08:00
wizardchen 2ab54f1542 chore(auth): drop redundant WEKNORA_AUTH_REGISTRATION_MODE env override
DISABLE_REGISTRATION=true (handler layer) and
WEKNORA_AUTH_REGISTRATION_MODE=invite_only (config layer) were two
env-level ways of saying the same thing: block /auth/register.
Keeping both invites confusion about which wins and risks operators
setting one while expecting the other.

Remove the WEKNORA_AUTH_REGISTRATION_MODE env override so the env
layer exposes a single knob (DISABLE_REGISTRATION). The
auth.registration_mode config field stays — operators who want the
richer behaviour (frontend hides the registration entry via
/auth/config in addition to the server-side 403) flip it in
config.yaml.

No behaviour change for deployments that did not set the env var.
Update docs/rbac.md to document the deliberate split.
2026-05-18 21:15:01 +08:00
wizardchen 7d030a6f6a feat(tenant): implement tenant creation limit and error handling
Added a new configuration option to limit the number of tenants a non-superuser can create via self-service. Introduced a new error type for handling cases where users exceed this limit, returning a 429 status code. Updated the tenant creation and update handlers to enforce this limit and provide appropriate feedback to users. Additionally, refactored the tenant update request structure to ensure only mutable fields are allowed.

Refs: #1303
2026-05-18 17:28:58 +08:00
wizardchen ff1f03ce0d feat(rbac): daily retention sweep for audit_logs (default 90 days)
The audit_logs table grows monotonically until something purges it.
PR 6 explicitly listed retention as out of scope for v1; this picks
that up.

Adds an `audit.retention_days` config (default 90, 0 disables) and a
small background goroutine `AuditLogRetentionRunner` that fires once
~10 minutes after boot and then every 24h, calling
`AuditLogService.Purge` which DELETEs rows older than the cutoff in a
single statement. The dependency surface is intentionally minimal —
no robfig/cron, no asynq, no migration — because retention has no
wall-clock alignment requirement.

Wiring is the same shape as the existing data source scheduler:
container `Provide(NewAuditLogRetentionRunner)`, an `Invoke` that
calls `Start`, and a `ResourceCleaner.RegisterWithName` for graceful
shutdown.

Defaults preserve operator intent:
  - `audit:` section omitted from YAML  -> retention_days=90.
  - explicit `audit.retention_days: 0` in YAML  -> purge disabled.
  - `WEKNORA_AUDIT_RETENTION_DAYS=N` env override (incl. N=0).
  - `retention_days < 0` is rejected by ValidateConfig.

Tests cover the service Purge contract (no-op when disabled, cutoff
math, error propagation) and the runner lifecycle (Start no-op when
disabled, idempotent Start/Stop, ticker cadence, runOnce swallows
errors). 10 new tests, all green.

docs/rbac.md updated with the new YAML / env / behaviour.

Refs: #1303
2026-05-18 17:28:58 +08:00
wizardchen 0a44077498 fix(rbac): plug ungated mutating routes; log RBAC state at startup
Sweep over the route table found four classes of mutating endpoints
that had no role guard:

- POST /tenants — creating a brand-new tenant. Now g.CrossTenant():
  only org-level superusers may create tenants, matching /tenants/all
  and /tenants/search.
- POST /initialization/initialize/:kbId, PUT /initialization/config/:kbId
  — these change a KB's embedding/parser/storage configuration, which
  is at least as sensitive as PUT /knowledge-bases/:id. New
  KBCreatorLookupFromKbIDParam (sister of KBCreatorLookup but reading
  :kbId) lets us reuse the OwnedKBOrAdmin matrix; routes now use the
  new g.OwnedKBOrAdminFromKbIDParam() guard.
- POST /initialization/{ollama,remote,embedding,rerank,asr,multimodal,
  extract}/* — system-level model probes / downloads / extraction
  pipelines. These trigger LLM calls and outbound network fanout
  with tenant credentials; gated to Admin+. The corresponding GETs
  (status / list / progress) drop to Viewer+ since they're read-only
  observability.
- POST /system/{parser-engines,docreader,storage-engine}/* — same
  pattern: GETs that report state are Viewer+, the *-check /
  /reconnect actions are Admin+.

POST /evaluation/ runs the eval harness which fans out LLM calls
across the tenant — gated to Admin+.

Operational improvement: log the resolved tenant RBAC state at
LoadConfig completion. air's hot-reload only rebuilds the binary on
Go-source changes; it does NOT re-source .env, so flipping
WEKNORA_TENANT_ENABLE_RBAC=true while the dev loop is already
running has no effect until the parent dev-app process restarts.
This was a real diagnostic loop in PR review — printing the value at
startup makes the "I edited .env but the gates still aren't firing"
trap obvious from the first console line.

The Printf includes both the resolved cfg.Tenant.EnableRBAC value
and the raw env var, so a mismatch between "what the .env file
says" and "what this process actually inherited" is one glance.
2026-05-18 17:28:58 +08:00
wizardchen 7f7a48880f feat(rbac): add tenant_members schema and auth wiring (PR 1 of 3, #1303)
Lay the foundation for tenant-level RBAC without enforcing it yet.
Schema, types, middleware lookup, and login-flow shape changes ship now;
RequireRole/route annotations and member-management endpoints follow in
PR 2 and PR 3.

Highlights:
- New tenant_members table (Postgres + SQLite) with (user_id, tenant_id)
  unique key and (owner|admin|contributor|viewer) role values. Migration
  000043 backfills one row per existing user, marking the earliest-created
  active user in each tenant as Owner and the rest as Contributor.
- KnowledgeBase.creator_id and CustomAgent.runnable_by_viewer added so
  PR 2 can wire ownership-based and Viewer-runtime checks.
- Auth middleware loads the active TenantMember row, sets a new
  TenantRoleContextKey, and falls back via three escape hatches:
  cross-tenant superusers (CanAccessAllTenants) get Admin in the target
  tenant; orphan tenants (zero members) auto-promote the first authenticating
  human to Owner; and when tenant.enable_rbac is false (the v1 default)
  unmatched lookups fail open with Admin to preserve current behaviour.
- Register / OIDC provisioning now insert an Owner membership for the
  registrant via TenantMemberService.EnsureOwner.
- LoginResponse renamed tenant -> active_tenant and gained a memberships
  array. POST /auth/switch-tenant issues a fresh token pair scoped to a
  target membership; GET /auth/config exposes auth.registration_mode for
  the frontend register-tab gating.
- New config keys: auth.registration_mode (self_serve | invite_only,
  defaults to self_serve) and tenant.enable_rbac (defaults to false).
  Both have WEKNORA_* env overrides.
- Frontend store persists memberships and exposes currentTenantRole;
  Login.vue calls /auth/config and hides the Register tab in invite_only
  mode. Go SDK types updated to match the new LoginResponse shape.
2026-05-18 17:28:58 +08:00
weitong a27f12a6cb 验证完毕 2026-05-15 20:47:01 +08:00
wizardchen 5510ea8f5a feat(agent): human-in-the-loop approval for MCP tool calls (#1173)
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.

Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
  required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
  delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
  replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
  (without the per-tool 60s timeout) for the wait, and re-derives a
  fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
  EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
  WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
  duration).

Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
  args, validation feedback, mm:ss countdown that turns warning/danger
  near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.

Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
  considerations (Redis cross-instance, restart limitations).
2026-05-10 22:57:12 +08:00
wizardchen 4022f134bd feat: Introduce agent type presets for smart-reasoning mode
- Added a new configuration file `agent_type_presets.yaml` to define various agent type presets, including RAG Q&A, Wiki Q&A, Hybrid (Wiki + RAG), and Data Analysis.
- Updated the `builtin_agents.yaml` to associate built-in agents with the new agent type presets, enhancing their configuration options.
- Enhanced the agent system prompt templates to reflect the new agent types and their functionalities.
- Implemented API endpoints to retrieve agent type presets, allowing for dynamic loading in the frontend.
- Updated the frontend to support agent type selection in the agent editor modal, improving user experience by auto-filling relevant configurations based on selected presets.

These changes significantly enhance the flexibility and usability of the agent system, allowing users to easily configure agents based on predefined types.
2026-04-22 21:18:20 +08:00
AndyYang d5ecc150e0 feat(agent): support customizable LLM call timeout and add docker-compose mapping 2026-04-07 11:26:52 +08:00
wizardchen cb36570cde feat(summary): enhance document summarization capabilities and UI
- Added max_input_chars configuration to limit input size for summary generation.
- Updated generate_summary.yaml to clarify summary generation steps and requirements.
- Introduced a new summary section in the document content view to display generated summaries or loading states.
- Refactored doc-content.vue to utilize computed properties for improved performance and readability.
- Enhanced knowledge base hooks to include description and summary status fields for better data handling.
- Updated internationalization files to include new summary-related labels in multiple languages.
2026-04-02 18:34:39 +08:00
Windfarer c1816fe6d6 add oidc 2026-03-30 11:13:44 +08:00
wizardchen 72dfb9ce75 feat(prompt): enhance context and intent classification in templates
- Updated context template to include runtime metadata such as current time and week, improving contextual awareness in user queries.
- Enhanced rewrite template with critical instructions for intent classification, ensuring that rewritten questions preserve essential entities and keywords.
- Refined intent classification logic to prioritize user intents more effectively, improving the accuracy of responses based on user queries.
- Added examples to clarify expected input and output formats for intent classification, enhancing usability for developers.
2026-03-26 18:57:12 +08:00
wizardchen 052be53c42 feat(prompts): update assistant descriptions and add intent-specific templates
- Updated the assistant descriptions in various prompt templates to specify that they are developed by Tencent.
- Added new intent-specific system prompt templates for greeting, chitchat, follow-up, image analysis, summarization, and web search unavailability scenarios.
- Enhanced citation rules in the system prompts to ensure proper inline citation formatting.
- Refactored the rewrite prompt to classify user intent more accurately and output structured JSON responses.
2026-03-25 22:08:29 +08:00
wizardchen e936e0b347 feat(agent): implement LLM call timeout and transient error handling
- Introduced a configurable LLM call timeout with a default value, allowing for better control over LLM call durations.
- Added logic to retry transient errors (e.g., timeouts, rate limits) up to a specified maximum number of retries, improving robustness in error handling.
- Implemented parameter casting for tool arguments to ensure correct types are used, addressing common LLM quirks.
- Enhanced tool execution error messages with guidance for retrying with different approaches.
- Added validation for configuration values to prevent runtime errors.
2026-03-25 22:08:29 +08:00
nullkey f01dcd4129 feat(im): add multi-instance support with Redis-based distributed coordination
- config: add IMConfig for tunable workers, queue size, rate limit params
- qaqueue: add Redis global per-user limits and global concurrency gate
- ratelimit: add distributedLimiter with Redis ZSET sliding window + local fallback
- service: add Redis message dedup, WebSocket leader election, cross-instance /stop
2026-03-20 15:49:59 +08:00
wizardchen 139a9c40ff feat: enhance localization and configuration support
- Added timezone and language settings to the environment configuration.
- Introduced built-in agent configurations with multilingual support for various agents.
- Updated Docker Compose to utilize new environment variables for timezone and language.
- Created new prompt templates for question generation, summary generation, and keywords extraction.
2026-03-17 22:23:24 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
Dounx 6e03f1ea79 feat(redis): add REDIS_USERNAME support for Redis ACL 2026-02-04 19:38:40 +08:00
wizardchen fad72fc272 refactor: web search provider registry pattern 2026-01-20 18:19:44 +08:00
wizardchen 0d020ce30c feat: 添加思考模式配置支持
在SummaryConfig结构体中新增Thinking字段,支持启用思考模式功能。更新相关配置文件、类型定义和客户端代码以支持该功能。
2025-12-31 15:36:20 +08:00
wizardchen abfe87eaed feat: 重构提示词模板系统,新增多场景模板配置支持;优化智能体知识库选择模式,支持全部/指定/禁用三种模式 2025-12-25 18:22:36 +08:00
wizardchen 16562eebbb feat: 添加AI问题生成功能,支持配置生成数量并在前端展示生成的问题 2025-12-03 01:42:31 +08:00
wizardchen cdf576eb18 feat: Implement cross-tenant access functionality
- Added configuration for enabling cross-tenant access in the application.
- Introduced new API endpoints for listing and searching tenants, accessible only to users with the appropriate permissions.
- Updated user and tenant models to include the `can_access_all_tenants` field.
- Enhanced the authentication middleware to handle cross-tenant requests and validate user permissions.
- Created a new TenantSelector component for selecting tenants in the frontend, improving user experience.
- Updated internationalization files to support new tenant-related strings and messages.
2025-12-02 12:17:39 +08:00
wizardchen eb8bfddfd1 refactor: Update function signatures across multiple files to improve readability by adding context parameters and enhancing code structure 2025-11-28 16:17:52 +08:00
wizardchen b9c1191dd2 feat: Introduce query expansion feature and enhance related configurations
- Added support for query expansion in conversation settings, allowing the system to generate additional queries when recall is low.
- Updated localization files in English, Russian, and Chinese to include new strings for the query expansion feature.
- Enhanced UI in AgentSettings.vue to allow users to toggle query expansion settings.
- Refactored related API and service logic to accommodate the new query expansion functionality in session management and chat processing.
2025-11-19 20:23:30 +08:00
wizardchen 3bf2a14499 feat: Enhance agent configuration and localization for web search prompts
- Updated agent configuration to support separate system prompts for web search enabled and disabled states.
- Removed deprecated agent configuration parameters to streamline settings management.
- Enhanced UI components in AgentSettings.vue to allow configuration of custom prompts based on web search status.
- Improved localization in English, Russian, and Chinese for new prompt settings and UI elements.
- Refactored related API and service logic to accommodate changes in agent configuration structure.
2025-11-19 19:27:25 +08:00
wizardchen 07a8c1de4c feat: Add web search functionality and enhance agent mode integration
- Introduced web search configuration in config.yaml, allowing multiple providers with specific settings.
- Updated InputField component to manage agent mode and web search toggles, improving user interaction.
- Enhanced chat API to include web search status in requests, enabling dynamic query handling.
- Implemented backend support for web search integration, merging results with knowledge base queries.
- Added tenant-level web search configuration management, allowing customization per tenant.
2025-11-10 03:14:12 +08:00
wizardchen 4fa3adbf3b feat: Add agent configuration and cleanup scripts for database migrations 2025-11-05 23:18:44 +08:00
begoniezhao b6d8be634a feat: 新增异步任务提取服务 2025-10-16 17:48:21 +08:00
wizardchen 56eb2bce33 init commit 2025-08-05 15:08:07 +08:00