Commit Graph

1225 Commits

Author SHA1 Message Date
wizardchen d60614475e test(api-keys): add unit tests for tenant API key validation
Introduce unit tests for the tenant API key validation logic, ensuring that scoped keys require capabilities, while full-access keys are validated without them. The tests cover various scenarios, reinforcing the expected behavior of the API key validation function.
2026-07-07 12:00:13 +08:00
wizardchen 2c6f082062 Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
wizardchen b5cc7d2a28 feat(api-keys): implement throttled last_used_at updates for API key authentication
Enhance the tenant API key service by introducing asynchronous throttling for the last_used_at updates during API key authentication. This change reduces database write frequency under high request rates, improving performance. Additionally, update tests to validate the throttling behavior, ensuring that last_used_at is only updated once per key within the defined interval. This implementation aligns with the goal of optimizing API key management and maintaining system efficiency.
2026-07-07 12:00:13 +08:00
wizardchen 97c36f634a refactor(api-keys): remove API key reset endpoint and update documentation
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
2026-07-07 12:00:13 +08:00
wizardchen 1d1e1e3d25 refactor(api-keys): update role handling and enhance API key restrictions
Change the default role for tenant API keys from 'admin' to 'owner' to align with the new authorization model. Update related tests to reflect this change, ensuring that API keys with the owner role are correctly denied access to management paths and system-admin routes. Introduce new tests to validate that API keys are properly restricted from accessing sensitive endpoints, reinforcing the security measures in place.
2026-07-07 12:00:13 +08:00
wizardchen a6759d61b5 feat(api-keys): implement DenyAPIKeyPrincipal middleware and related tests
Add DenyAPIKeyPrincipal middleware to explicitly block API-key principals from accessing certain routes, ensuring security for endpoints outside the /api/v1 group. Introduce tests to validate that API keys are correctly denied access while allowing JWT sessions to pass through. This enhances the overall authorization model by reinforcing the default-deny approach for API keys.
2026-07-07 12:00:13 +08:00
wizardchen d2ffc1b973 refactor(api-keys): simplify authorization to a single-role gate
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.

- Add APIKeyGate as the single authority for API-key authorization: routes
  declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
  routes are denied by default. JWT principals short-circuit the gate and the
  JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
  is a pure downstream data filter); owner is a full-access tenant tier that
  unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
  writes on API-key encryption failure, deny KB-restricted keys on /files, and
  backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
  role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
  gin paths before comparison; add regression tests.
2026-07-07 12:00:13 +08:00
wizardchen fedb6c3f7e test(api-keys): add tests for tenant API key management path rejection
Introduce tests to validate that tenant API keys are correctly restricted from accessing management paths. Implemented new test cases for both allowed and forbidden paths, ensuring proper enforcement of API key scopes. Additionally, refactored existing tests to utilize the new rejection logic.
2026-07-07 12:00:13 +08:00
wizardchen 9a76a24ce6 refactor(api-keys): use role field and route-level API key guards
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
2026-07-07 12:00:13 +08:00
wizardchen c65ffa3d9c fix(api-keys): enforce scope from request context after attach
Use c.Request.Context() in authenticateAPIKeyRequest so scoped keys are
actually restricted, update dto tests for removed tenant.api_key, and add
a regression test for the attach-context contract.
2026-07-07 12:00:13 +08:00
wizardchen 94d18ea15a fix(api-keys): close KB scope gaps in handlers and CloneContext
Enforce per-key knowledge base restrictions on batch/search read paths,
centralize scope checks in shared KB access validators, and propagate
TenantAPIKeyScope through logger.CloneContext so async QA cannot bypass it.
2026-07-07 12:00:13 +08:00
wizardchen 3f516d7f31 fix(api-keys): harden scoped key auth after code review
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
2026-07-07 12:00:13 +08:00
wizardchen 505bc7ddec feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
langcaiye 1708d6895a fix(search): apply tag filters to knowledge search 2026-07-06 23:30:33 +08:00
cao_j b16cb0bb56 fix(agent): treat end_turn as natural stop 2026-07-06 23:16:18 +08:00
养乐多 baada32cb6 fix(im/feishu): actually close WebSocket long connection when channel is stopped
Deleting or disabling a Feishu channel left its WebSocket long connection
alive: the Lark SDK's Start blocks on a bare select{} and never observes
ctx, and v3.6.1 exposed no public Close(), so the wsCancel() cleanup was a
no-op. The socket stayed open, the SDK kept auto-reconnecting, and Feishu
kept pushing messages — causing cross-environment message duplication when
the same bot was bound elsewhere.

Fix in three layers:

1. Connection teardown
   - Upgrade oapi-sdk-go v3.6.1 -> v3.9.7 (adds Client.Close()).
   - Add LongConnClient.Close() wrapping wsClient.Close(); factory's cleanup
     now calls Close() first, then wsCancel() as a belt-and-braces fallback.

2. Multi-instance coordination
   - wsLeaderRetryLoop: after acquiring leadership, re-check the DB before
     starting; abort if the channel was deleted/disabled while waiting.
   - wsLeaderRenewLoop: on each renewal, re-check the DB; step down if the
     channel was deleted/disabled on another instance (HTTP request may have
     been served elsewhere), so the leader doesn't keep the connection open
     until process restart.

3. Robustness
   - DB checks distinguish gorm.ErrRecordNotFound (channel truly gone ->
     stop/abort) from transient DB errors (skip this round, retry next tick)
     so a DB hiccup can't cause a spurious stop or permanent leader-abort.
   - startChannelInternal now stops any existing channelState for the same
     ID before overwriting, making it idempotent against concurrent start
     paths and preventing adapter/long-connection leaks.

No interface changes; webhook and long-poll modes unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 23:10:45 +08:00
wizardchen 4f2dca0b63 fix(redis): harden TLS tests and surface TLS config at startup
Isolate Redis TLS unit tests from parent-process env vars and log
REDIS_USE_TLS settings at boot so misconfiguration is easier to spot.
2026-07-06 23:04:52 +08:00
qilifan d5b8d2f886 feat(redis): support TLS for Redis connections (#1930)
[Feature]: Support TLS for Redis connections
2026-07-06 23:02:02 +08:00
nullkey 445a4330fa fix(server): case-insensitive doc keyword search + not-found classification
Make document keyword search case-insensitive (LOWER(...) LIKE LOWER(?)) in the
list filter and SearchKnowledge / SearchKnowledgeInScopes paths so a query and
its differently-cased variant return the same results.

Classify not-found consistently as HTTP 404 (not 500) so clients read a
permanent resource.not_found instead of a retryable 5xx: map the document /
chunk not-found sentinels in the KB-access middleware (typed ErrChunkNotFound
via errors.Is), and return 404 for a missing message on the continue-stream and
message-delete endpoints.
2026-07-06 22:53:01 +08:00
wizardchen 1b0502e562 fix(feishu): upload card markdown images 2026-07-06 14:52:29 +08:00
wizardchen 1661827d6e fix(security): harden model client redirects 2026-07-02 19:15:46 +08:00
wizardchen 663d13eb58 fix(security): harden outbound requests and file previews 2026-07-02 19:15:46 +08:00
wizardchen 7099a7e181 fix(security): redact IM list credentials and harden MinerU SSRF
Stop returning bot credentials from Viewer-accessible IM channel list
endpoints and validate self-hosted MinerU URLs at config save and fetch time.
2026-07-02 17:58:20 +08:00
wizardchen 235e86b876 fix(security): harden refresh token validation and add auth tests
Reject refresh attempts when the DB token type is not refresh_token,
and add service-layer tests for bearer refresh rejection, logout
revocation, and expired-token logout handling.
2026-07-02 17:08:17 +08:00
wizardchen 040504447e fix(security): block refresh tokens as bearer and sandbox MCP uploads
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
2026-07-02 17:08:17 +08:00
wizardchen cd65665894 fix(security): close remaining GHSA gaps from security triage
Block scheme-prefixed loopback storage checks, validate Notion attachment
URLs, sanitize wiki markdown HTML, and require MCP_SERVER_AUTH_TOKEN for
network MCP transports.
2026-07-02 15:03:53 +08:00
wizardchen f23c674bad fix(security): close six open GHSA findings from security triage
Harden share-link listing, password rotation, OIDC callback binding,
org invite search, knowledge batch mutations, and connector redirects.
2026-07-02 15:03:53 +08:00
wizardchen fc64751f4e fix(security): close init config and parser-check redaction gaps
Strip model base URLs from viewer-accessible initialization config
responses and merge redacted parser secrets before engine check overrides
so admin test flows keep working after masked GET responses.
2026-07-02 14:41:42 +08:00
wizardchen 1a264e09c1 fix(security): harden secret redaction follow-ups for PR #1884
Restore PreserveIfRedacted merge on tenant KV PUT so Admin settings saves
no longer overwrite stored secrets with placeholder values. Redact cross-
tenant tenant listings, tighten Viewer MCP/model responses, and align init
config frontend types with credentials metadata.
2026-07-02 14:41:42 +08:00
wizardchen 512d2e5c25 fix(security): redact secrets in login and tenant KV update responses
Apply the same tenant/integration secret policy to auth login, OIDC
callback, switch-tenant, and invite-register payloads, and return masked
config objects from Admin-only tenant KV PUT handlers.
2026-07-02 14:41:42 +08:00
wizardchen 0b73080bb8 fix(security): redact integration secrets from viewer-accessible endpoints
Prevent low-privileged tenant members from reading raw API keys and
infrastructure credentials via tenant profile, KV config, model/MCP/web
search listings, and KB initialization config responses.
2026-07-02 14:41:42 +08:00
wizardchen 8325dac738 fix(security): harden web_fetch and datasource connector SSRF paths
Use SSRF-safe HTTP clients in chat pipeline web_fetch and Feishu/Yuque/Notion
connectors, validating tenant-supplied base_url values before outbound requests.
2026-07-02 12:27:48 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
wizardchen a17a2250ed fix(security): enforce KB access on wiki reads and related IDOR paths
Gate wiki GET routes and initialization config reads with KBAccessRead,
and bind async task progress endpoints to the caller tenant via task ID
parsing. Closes cross-tenant read paths reported in GHSA-ph8f-4qwx-89rr
and related initialization/progress leaks.
2026-07-02 12:06:22 +08:00
wizardchen ab790bcbd9 fix(security): close SQL validator bypass in agent database tools
Harden validateNode with default-deny semantics and PG17 SQL/JSON node
recursion so dangerous functions cannot hide inside JSON_SCALAR/JSON_VALUE.
Also recurse into FROM subqueries for DuckDB data_analysis and block
read_text/read_blob and related file-access functions.
2026-07-02 11:42:11 +08:00
wizardchen 2203c4bb14 feat(im): include per-service MCP OAuth URLs in IM auth notices
IM channels cannot complete in-conversation OAuth prompts, so when an
MCP service needs authorization the reply now includes clickable
authorization URLs (when APP_EXTERNAL_URL and OAuth manager are
available) instead of only directing users to the admin console.
2026-07-01 17:53:54 +08:00
wizardchen c694fcf6cd test: address PR review follow-ups for OAuth and logger cleanup
Restore logger config via t.Cleanup after t.Setenv so Yuque token tests
do not leak global state, add unit tests for MCP OAuth non-interactive
context helpers and IM auth notice formatting, and fix EmbedChatCore
indentation.
2026-07-01 17:31:42 +08:00
wizardchen 8d40959f88 fix lint and test failures 2026-07-01 17:31:42 +08:00
wizardchen 4c962a9403 feat(agent): add configurable MCP OAuth wait timeout per agent
Allow agents to set how long in-conversation OAuth prompts wait before
auto-skipping, with UI in the agent editor and bounded gate behavior.
2026-07-01 14:37:37 +08:00
lawrence fdfc7297de fix(vlm): allow configurable temperature for VLM models
Some VLM models (e.g. certain vision models) only accept temperature=1.
Previously temperature was hardcoded to 0.1. Now reads from model's
ExtraConfig["temperature"] with fallback to default 0.1.

Usage: set ExtraConfig.temperature=1 in the VLM model configuration.
2026-07-01 12:20:09 +08:00
wizardchen ee1014f87c feat(frontend): add API integration playground 2026-07-01 11:14:51 +08:00
wizardchen 5074344c16 feat(embed): scope OAuth and sessions per visitor principal
Add embed visitor principals (X-Embed-Visitor), widen sessions.user_id,
embed MCP OAuth/approval routes, OAuth callback context detachment, and
CloneContext principal preservation for async QA streams.
2026-06-30 20:01:05 +08:00
wizardchen 97bca74a31 fix: harden API principal config after review
Add require_direct_header toggle, external user ID validation, JWT TTL
limits, SaveToken error handling, ru-RU i18n, API docs, and Go SDK methods.
2026-06-30 20:01:05 +08:00
wizardchen cb99736d72 feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
wizardchen b5a1c88f75 fix(mcp): add OAuth skip endpoint and streamline in-chat interaction cards
Add a dedicated cancel route so users can skip mid-conversation OAuth without
token verification, and restyle OAuth/approval cards to match AgentStreamDisplay
with compact inline actions.
2026-06-29 23:22:28 +08:00
wizardchen 505195865a fix(mcp): trigger OAuth prompt during tool registration and dedupe helpers
OAuth authorization was only handled in MCPTool.Execute, so unauthenticated
services failed silently during RegisterMCPTools before any tool reached the
agent. Wire EventBus/session metadata into registration and reuse the same
getOrCreateMCPClientWithOAuthRetry path used at execution time.

Extract MCP OAuth session/wait/connect logic into mcp_oauth.go and remove the
duplicate waitForOAuthAndRetry + RegisterMCPContext layering in mcp_tool.go.
2026-06-29 23:22:28 +08:00
wizardchen 9fd7a7f522 feat(mcp): prompt and resume OAuth authorization mid-conversation
When an agent invokes an OAuth-enabled MCP service that the current user has
not authorized yet, the tool call now pauses and surfaces an "Authorize" card
in the chat instead of failing with an opaque error. After the user completes
the per-user OAuth popup, the backend verifies a token exists and resumes the
paused tool call automatically.

Backend:
- Add mcp_oauth_required / mcp_oauth_resolved events and response types.
- Add OAuthPendingRequest + Gate.RequestOAuthAndWait, reusing the existing
  approval waiter, Resolve, Redis cross-instance fan-out and timeout.
- MCPTool.Execute pauses on an authorization-required error for OAuth services
  and retries once (with a fresh ctx and recycled connection) after the user
  authorizes.
- Forward the new events to the SSE stream.
- Add ResolveMCPOAuth handler + route; it verifies authorization before
  unblocking so a premature resolve cannot resume into another failure.

Frontend:
- Add McpOAuthCard with an Authorize button reusing the popup + status-poll
  flow, then resolve the paused run.
- Handle the new stream events (dedup multiple pending cards per service).
- Add resolveMCPOAuth API client and i18n keys (zh/en/ko/ru).
2026-06-29 23:22:28 +08:00
wizardchen f2686e9e39 feat(mcp): drive auth by AuthType and auto-detect OAuth on test
Select the auth header for the configured strategy only, instead of
emitting both X-API-Key and Authorization whenever the fields happened
to be set (which double-authed after switching strategies). AuthType=""
keeps the legacy infer-from-fields behavior for pre-existing rows.

Add a configurable api_key_header (non-secret, defaults to X-API-Key) so
services expecting the key in a different header work without plaintext
custom headers.

On connection test, upgrade a bare 401 that advertises RFC 9728
protected-resource metadata into an explicit OAuthRequired signal so the
UI can guide the user to switch to OAuth 2.0. Surface the server's
description from the initialize handshake in the test result.
2026-06-29 22:37:52 +08:00
wizardchen c29761331c fix(agent): enforce tag scopes across knowledge tools 2026-06-29 19:17:42 +08:00
wizardchen 17ffd902ac fix(agent): avoid expanding tag scopes for grep chunks 2026-06-29 19:17:42 +08:00