Commit Graph

1219 Commits

Author SHA1 Message Date
wizardchen d2ffc1b973 refactor(api-keys): simplify authorization to a single-role gate
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.

- Add APIKeyGate as the single authority for API-key authorization: routes
  declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
  routes are denied by default. JWT principals short-circuit the gate and the
  JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
  is a pure downstream data filter); owner is a full-access tenant tier that
  unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
  writes on API-key encryption failure, deny KB-restricted keys on /files, and
  backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
  role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
  gin paths before comparison; add regression tests.
2026-07-07 12:00:13 +08:00
wizardchen fedb6c3f7e test(api-keys): add tests for tenant API key management path rejection
Introduce tests to validate that tenant API keys are correctly restricted from accessing management paths. Implemented new test cases for both allowed and forbidden paths, ensuring proper enforcement of API key scopes. Additionally, refactored existing tests to utilize the new rejection logic.
2026-07-07 12:00:13 +08:00
wizardchen 9a76a24ce6 refactor(api-keys): use role field and route-level API key guards
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
2026-07-07 12:00:13 +08:00
wizardchen c65ffa3d9c fix(api-keys): enforce scope from request context after attach
Use c.Request.Context() in authenticateAPIKeyRequest so scoped keys are
actually restricted, update dto tests for removed tenant.api_key, and add
a regression test for the attach-context contract.
2026-07-07 12:00:13 +08:00
wizardchen 94d18ea15a fix(api-keys): close KB scope gaps in handlers and CloneContext
Enforce per-key knowledge base restrictions on batch/search read paths,
centralize scope checks in shared KB access validators, and propagate
TenantAPIKeyScope through logger.CloneContext so async QA cannot bypass it.
2026-07-07 12:00:13 +08:00
wizardchen 3f516d7f31 fix(api-keys): harden scoped key auth after code review
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
2026-07-07 12:00:13 +08:00
wizardchen 505bc7ddec feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
langcaiye 1708d6895a fix(search): apply tag filters to knowledge search 2026-07-06 23:30:33 +08:00
cao_j b16cb0bb56 fix(agent): treat end_turn as natural stop 2026-07-06 23:16:18 +08:00
养乐多 baada32cb6 fix(im/feishu): actually close WebSocket long connection when channel is stopped
Deleting or disabling a Feishu channel left its WebSocket long connection
alive: the Lark SDK's Start blocks on a bare select{} and never observes
ctx, and v3.6.1 exposed no public Close(), so the wsCancel() cleanup was a
no-op. The socket stayed open, the SDK kept auto-reconnecting, and Feishu
kept pushing messages — causing cross-environment message duplication when
the same bot was bound elsewhere.

Fix in three layers:

1. Connection teardown
   - Upgrade oapi-sdk-go v3.6.1 -> v3.9.7 (adds Client.Close()).
   - Add LongConnClient.Close() wrapping wsClient.Close(); factory's cleanup
     now calls Close() first, then wsCancel() as a belt-and-braces fallback.

2. Multi-instance coordination
   - wsLeaderRetryLoop: after acquiring leadership, re-check the DB before
     starting; abort if the channel was deleted/disabled while waiting.
   - wsLeaderRenewLoop: on each renewal, re-check the DB; step down if the
     channel was deleted/disabled on another instance (HTTP request may have
     been served elsewhere), so the leader doesn't keep the connection open
     until process restart.

3. Robustness
   - DB checks distinguish gorm.ErrRecordNotFound (channel truly gone ->
     stop/abort) from transient DB errors (skip this round, retry next tick)
     so a DB hiccup can't cause a spurious stop or permanent leader-abort.
   - startChannelInternal now stops any existing channelState for the same
     ID before overwriting, making it idempotent against concurrent start
     paths and preventing adapter/long-connection leaks.

No interface changes; webhook and long-poll modes unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 23:10:45 +08:00
wizardchen 4f2dca0b63 fix(redis): harden TLS tests and surface TLS config at startup
Isolate Redis TLS unit tests from parent-process env vars and log
REDIS_USE_TLS settings at boot so misconfiguration is easier to spot.
2026-07-06 23:04:52 +08:00
qilifan d5b8d2f886 feat(redis): support TLS for Redis connections (#1930)
[Feature]: Support TLS for Redis connections
2026-07-06 23:02:02 +08:00
nullkey 445a4330fa fix(server): case-insensitive doc keyword search + not-found classification
Make document keyword search case-insensitive (LOWER(...) LIKE LOWER(?)) in the
list filter and SearchKnowledge / SearchKnowledgeInScopes paths so a query and
its differently-cased variant return the same results.

Classify not-found consistently as HTTP 404 (not 500) so clients read a
permanent resource.not_found instead of a retryable 5xx: map the document /
chunk not-found sentinels in the KB-access middleware (typed ErrChunkNotFound
via errors.Is), and return 404 for a missing message on the continue-stream and
message-delete endpoints.
2026-07-06 22:53:01 +08:00
wizardchen 1b0502e562 fix(feishu): upload card markdown images 2026-07-06 14:52:29 +08:00
wizardchen 1661827d6e fix(security): harden model client redirects 2026-07-02 19:15:46 +08:00
wizardchen 663d13eb58 fix(security): harden outbound requests and file previews 2026-07-02 19:15:46 +08:00
wizardchen 7099a7e181 fix(security): redact IM list credentials and harden MinerU SSRF
Stop returning bot credentials from Viewer-accessible IM channel list
endpoints and validate self-hosted MinerU URLs at config save and fetch time.
2026-07-02 17:58:20 +08:00
wizardchen 235e86b876 fix(security): harden refresh token validation and add auth tests
Reject refresh attempts when the DB token type is not refresh_token,
and add service-layer tests for bearer refresh rejection, logout
revocation, and expired-token logout handling.
2026-07-02 17:08:17 +08:00
wizardchen 040504447e fix(security): block refresh tokens as bearer and sandbox MCP uploads
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
2026-07-02 17:08:17 +08:00
wizardchen cd65665894 fix(security): close remaining GHSA gaps from security triage
Block scheme-prefixed loopback storage checks, validate Notion attachment
URLs, sanitize wiki markdown HTML, and require MCP_SERVER_AUTH_TOKEN for
network MCP transports.
2026-07-02 15:03:53 +08:00
wizardchen f23c674bad fix(security): close six open GHSA findings from security triage
Harden share-link listing, password rotation, OIDC callback binding,
org invite search, knowledge batch mutations, and connector redirects.
2026-07-02 15:03:53 +08:00
wizardchen fc64751f4e fix(security): close init config and parser-check redaction gaps
Strip model base URLs from viewer-accessible initialization config
responses and merge redacted parser secrets before engine check overrides
so admin test flows keep working after masked GET responses.
2026-07-02 14:41:42 +08:00
wizardchen 1a264e09c1 fix(security): harden secret redaction follow-ups for PR #1884
Restore PreserveIfRedacted merge on tenant KV PUT so Admin settings saves
no longer overwrite stored secrets with placeholder values. Redact cross-
tenant tenant listings, tighten Viewer MCP/model responses, and align init
config frontend types with credentials metadata.
2026-07-02 14:41:42 +08:00
wizardchen 512d2e5c25 fix(security): redact secrets in login and tenant KV update responses
Apply the same tenant/integration secret policy to auth login, OIDC
callback, switch-tenant, and invite-register payloads, and return masked
config objects from Admin-only tenant KV PUT handlers.
2026-07-02 14:41:42 +08:00
wizardchen 0b73080bb8 fix(security): redact integration secrets from viewer-accessible endpoints
Prevent low-privileged tenant members from reading raw API keys and
infrastructure credentials via tenant profile, KV config, model/MCP/web
search listings, and KB initialization config responses.
2026-07-02 14:41:42 +08:00
wizardchen 8325dac738 fix(security): harden web_fetch and datasource connector SSRF paths
Use SSRF-safe HTTP clients in chat pipeline web_fetch and Feishu/Yuque/Notion
connectors, validating tenant-supplied base_url values before outbound requests.
2026-07-02 12:27:48 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
wizardchen a17a2250ed fix(security): enforce KB access on wiki reads and related IDOR paths
Gate wiki GET routes and initialization config reads with KBAccessRead,
and bind async task progress endpoints to the caller tenant via task ID
parsing. Closes cross-tenant read paths reported in GHSA-ph8f-4qwx-89rr
and related initialization/progress leaks.
2026-07-02 12:06:22 +08:00
wizardchen ab790bcbd9 fix(security): close SQL validator bypass in agent database tools
Harden validateNode with default-deny semantics and PG17 SQL/JSON node
recursion so dangerous functions cannot hide inside JSON_SCALAR/JSON_VALUE.
Also recurse into FROM subqueries for DuckDB data_analysis and block
read_text/read_blob and related file-access functions.
2026-07-02 11:42:11 +08:00
wizardchen 2203c4bb14 feat(im): include per-service MCP OAuth URLs in IM auth notices
IM channels cannot complete in-conversation OAuth prompts, so when an
MCP service needs authorization the reply now includes clickable
authorization URLs (when APP_EXTERNAL_URL and OAuth manager are
available) instead of only directing users to the admin console.
2026-07-01 17:53:54 +08:00
wizardchen c694fcf6cd test: address PR review follow-ups for OAuth and logger cleanup
Restore logger config via t.Cleanup after t.Setenv so Yuque token tests
do not leak global state, add unit tests for MCP OAuth non-interactive
context helpers and IM auth notice formatting, and fix EmbedChatCore
indentation.
2026-07-01 17:31:42 +08:00
wizardchen 8d40959f88 fix lint and test failures 2026-07-01 17:31:42 +08:00
wizardchen 4c962a9403 feat(agent): add configurable MCP OAuth wait timeout per agent
Allow agents to set how long in-conversation OAuth prompts wait before
auto-skipping, with UI in the agent editor and bounded gate behavior.
2026-07-01 14:37:37 +08:00
lawrence fdfc7297de fix(vlm): allow configurable temperature for VLM models
Some VLM models (e.g. certain vision models) only accept temperature=1.
Previously temperature was hardcoded to 0.1. Now reads from model's
ExtraConfig["temperature"] with fallback to default 0.1.

Usage: set ExtraConfig.temperature=1 in the VLM model configuration.
2026-07-01 12:20:09 +08:00
wizardchen ee1014f87c feat(frontend): add API integration playground 2026-07-01 11:14:51 +08:00
wizardchen 5074344c16 feat(embed): scope OAuth and sessions per visitor principal
Add embed visitor principals (X-Embed-Visitor), widen sessions.user_id,
embed MCP OAuth/approval routes, OAuth callback context detachment, and
CloneContext principal preservation for async QA streams.
2026-06-30 20:01:05 +08:00
wizardchen 97bca74a31 fix: harden API principal config after review
Add require_direct_header toggle, external user ID validation, JWT TTL
limits, SaveToken error handling, ru-RU i18n, API docs, and Go SDK methods.
2026-06-30 20:01:05 +08:00
wizardchen cb99736d72 feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
wizardchen b5a1c88f75 fix(mcp): add OAuth skip endpoint and streamline in-chat interaction cards
Add a dedicated cancel route so users can skip mid-conversation OAuth without
token verification, and restyle OAuth/approval cards to match AgentStreamDisplay
with compact inline actions.
2026-06-29 23:22:28 +08:00
wizardchen 505195865a fix(mcp): trigger OAuth prompt during tool registration and dedupe helpers
OAuth authorization was only handled in MCPTool.Execute, so unauthenticated
services failed silently during RegisterMCPTools before any tool reached the
agent. Wire EventBus/session metadata into registration and reuse the same
getOrCreateMCPClientWithOAuthRetry path used at execution time.

Extract MCP OAuth session/wait/connect logic into mcp_oauth.go and remove the
duplicate waitForOAuthAndRetry + RegisterMCPContext layering in mcp_tool.go.
2026-06-29 23:22:28 +08:00
wizardchen 9fd7a7f522 feat(mcp): prompt and resume OAuth authorization mid-conversation
When an agent invokes an OAuth-enabled MCP service that the current user has
not authorized yet, the tool call now pauses and surfaces an "Authorize" card
in the chat instead of failing with an opaque error. After the user completes
the per-user OAuth popup, the backend verifies a token exists and resumes the
paused tool call automatically.

Backend:
- Add mcp_oauth_required / mcp_oauth_resolved events and response types.
- Add OAuthPendingRequest + Gate.RequestOAuthAndWait, reusing the existing
  approval waiter, Resolve, Redis cross-instance fan-out and timeout.
- MCPTool.Execute pauses on an authorization-required error for OAuth services
  and retries once (with a fresh ctx and recycled connection) after the user
  authorizes.
- Forward the new events to the SSE stream.
- Add ResolveMCPOAuth handler + route; it verifies authorization before
  unblocking so a premature resolve cannot resume into another failure.

Frontend:
- Add McpOAuthCard with an Authorize button reusing the popup + status-poll
  flow, then resolve the paused run.
- Handle the new stream events (dedup multiple pending cards per service).
- Add resolveMCPOAuth API client and i18n keys (zh/en/ko/ru).
2026-06-29 23:22:28 +08:00
wizardchen f2686e9e39 feat(mcp): drive auth by AuthType and auto-detect OAuth on test
Select the auth header for the configured strategy only, instead of
emitting both X-API-Key and Authorization whenever the fields happened
to be set (which double-authed after switching strategies). AuthType=""
keeps the legacy infer-from-fields behavior for pre-existing rows.

Add a configurable api_key_header (non-secret, defaults to X-API-Key) so
services expecting the key in a different header work without plaintext
custom headers.

On connection test, upgrade a bare 401 that advertises RFC 9728
protected-resource metadata into an explicit OAuthRequired signal so the
UI can guide the user to switch to OAuth 2.0. Surface the server's
description from the initialize handshake in the test result.
2026-06-29 22:37:52 +08:00
wizardchen c29761331c fix(agent): enforce tag scopes across knowledge tools 2026-06-29 19:17:42 +08:00
wizardchen 17ffd902ac fix(agent): avoid expanding tag scopes for grep chunks 2026-06-29 19:17:42 +08:00
wizardchen d3071fbe03 test(agent): add unit tests for buildMustUseBlock functionality
- Introduced tests for buildMustUseBlock to validate multi-word service prefix handling and name sanitization.
- Ensured that service names are correctly sanitized to prevent newline injection in must_use blocks.
- Verified that the generated block includes the correct tool name prefixes and does not contain unwanted characters.
2026-06-29 18:09:44 +08:00
wizardchen 6ad75d19fd test(knowledge): unbreak service test package after tag-loading change
Fixing the knowledge_create_test compile error in this branch let the
internal/application/service test package run for the first time, which
exposed two latent failures introduced when GetKnowledgeByID started
eagerly loading tags:

- GetOwningKBCreatorID went through the service-level GetKnowledgeByID
  (which now calls repo.GetKnowledgeTags); ownership resolution only
  needs the knowledge -> kb_id link, so resolve via repo.GetKnowledgeByID
  directly (tenant-scoped) and drop the extra tag query.
- createKnowledgeFileRepoStub lacked GetKnowledgeTags, which the create
  path always invokes via setAndAttachKnowledgeTags; add the stub method.
2026-06-29 18:09:44 +08:00
wizardchen d24e830c85 refactor(agent): fix grep scope loss and consolidate per-turn @mention scoping
- Fix grep_chunks dropping the full-KB scope when a turn mixes @KB with
  @tag/@file: extract scopeClause and OR knowledge-ID and (kb,tenant)
  predicates together, matching knowledge_search's fan-out.
- Consolidate the three scattered per-request @Skill/@MCP scope blocks in
  buildAgentConfig into applyPerRequestSkillScope / applyPerRequestMCPScope,
  each narrowing the whitelist and computing the must_use pin in one place.
- Dedupe the two identical selection-mode normalizers in Input-field.vue.
- Fix knowledge_create_test passing "" for the tagIDs []string parameter,
  which made the whole service test package fail to compile.
- Add unit tests for scopeClause and the per-request scope helpers.
2026-06-29 18:09:44 +08:00
wizardchen b86efe3f92 fix(agent): harden @mention scope and simplify must_use hints
Tighten per-turn @MCP/@Skill/@tag handling after review: enforce agent
MCP/Skill whitelist, ignore mentions when selection mode is none, tag-scope
grep_chunks, and simplified English must_use lines without tool dumps.
Skip persisting scope envelopes to history rendered_content.
2026-06-29 18:09:44 +08:00
wizardchen 049f85139d feat(chat): add @Skill/@MCP mentions with scoped agent runtime
Wire per-turn @mentions for skills, MCP services, tags, and files through
QA handlers into agent config, runtime_context/must_use user blocks, and
tool registration. Fix tag-only scopes dropping KB tools, refresh suggested
questions on mention changes, and strengthen MCP must_use so @selected
services are invoked before local KB search.
2026-06-29 18:09:44 +08:00
wizardchen 40f217c228 test: fix stale Doris SQL-shape and flaky OSS bucket assertions
Two pre-existing test failures on main, unrelated to any feature change:

- Doris VectorRetrieve/KeywordsRetrieve inline LIMIT as a literal (Doris
  does not support placeholders for LIMIT/OFFSET), but the SQL-shape tests
  still expected `LIMIT ?` with TopK as a bound arg. Match the literal
  LIMIT and drop the extra argument.
- TestOssEnsureBucket_CreateFails used the bucket name "test-bucket",
  which already exists globally on OSS, so IsBucketExist short-circuited
  and the create path never ran. Use a non-existent bucket name so the
  create-with-invalid-credentials path is actually exercised.
2026-06-29 17:44:56 +08:00