Commit Graph

1234 Commits

Author SHA1 Message Date
wizardchen 6ac1703198 Simplify KB duplicate API and improve naming/docs coverage.
Drop client-supplied target_id, generate IDs server-side, localize duplicate
names from request language, document the endpoint, and expand handler/client tests.
2026-07-08 20:55:54 +08:00
wizardchen 58b79876ae Add knowledge base duplicate flow 2026-07-08 20:55:54 +08:00
wizardchen a1b49e2664 fix(faq): guard against division by zero in import timing logs
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-08 15:27:51 +08:00
wizardchen d9bb7aa8f2 fix(search): harden tag scope resolution after #1907
Default missing KB metadata to document tag relation lookup, propagate tag resolution failures instead of returning empty results, and reject ambiguous bare tag_ids when multiple knowledge bases are in scope.
2026-07-08 10:55:42 +08:00
wizardchen 3209ed77fd feat(quick-qa): add references drawer and polish retrieval source UI
Add a side panel for web and knowledge-base citations with citation-badge
highlighting, refine collapsed RAG pipeline summary layout, and emit
knowledge_base_id from agent knowledge_search results.
2026-07-08 00:26:34 +08:00
wizardchen 45eab4a95f fix(quick-qa): distinguish web and knowledge base retrieval in pipeline UI
Emit search_source metadata from RAG progress events so quick-answer steps,
summaries, and IM labels no longer mislabel web-only results as knowledge base search.
2026-07-08 00:26:34 +08:00
wizardchen 77cf6a5789 fix(web-search): polish Keenable provider integration after #1946
Expose optional API key in settings UI, map longer API snippets into Content
for RAG, align maxResults fallback with other providers, sync Swagger enum,
and add follow-up unit tests.
2026-07-07 17:14:39 +08:00
Ilya Bogin 8cf9f1c882 feat: add Keenable as a configurable web search provider
Add Keenable Search alongside the existing DuckDuckGo / Bing / Google /
Tavily / Ollama / Baidu / SearXNG providers.

- internal/infrastructure/web_search/keenable.go: KeenableProvider
  implementing the WebSearchProvider interface, following the Tavily
  provider pattern. Keyless by default: with no API key it calls the
  public endpoint (/v1/search/public, rate-limited); a key switches to
  /v1/search and lifts the cap. Base URL is hardcoded (SSRF-safe), proxy
  is honored.
- Registered "keenable" in the container registry.
- Added the keenable provider type + type metadata (RequiresAPIKey=false)
  and service validation (no required key, like DuckDuckGo).
- Provider-specific empty-results diagnostic (keyless rate-limit hint).
- Frontend: extend the provider union type with 'keenable'.
- Unit tests: keyless vs keyed endpoint, attribution header, maxResults
  trim, result mapping, error status, name.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:58:26 +08:00
wizardchen 974231190c Add API key member and space capabilities 2026-07-07 12:00:13 +08:00
wizardchen d60614475e test(api-keys): add unit tests for tenant API key validation
Introduce unit tests for the tenant API key validation logic, ensuring that scoped keys require capabilities, while full-access keys are validated without them. The tests cover various scenarios, reinforcing the expected behavior of the API key validation function.
2026-07-07 12:00:13 +08:00
wizardchen 2c6f082062 Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
wizardchen b5cc7d2a28 feat(api-keys): implement throttled last_used_at updates for API key authentication
Enhance the tenant API key service by introducing asynchronous throttling for the last_used_at updates during API key authentication. This change reduces database write frequency under high request rates, improving performance. Additionally, update tests to validate the throttling behavior, ensuring that last_used_at is only updated once per key within the defined interval. This implementation aligns with the goal of optimizing API key management and maintaining system efficiency.
2026-07-07 12:00:13 +08:00
wizardchen 97c36f634a refactor(api-keys): remove API key reset endpoint and update documentation
Eliminate the POST /tenants/{id}/api-key endpoint for resetting tenant API keys, transitioning to a more granular API key management approach. Update related documentation and API specifications to reflect the new structure, including changes to the tenant API key management paths. Adjust frontend code and internationalization files to remove references to the deprecated API key reset functionality, enhancing clarity and consistency across the application.
2026-07-07 12:00:13 +08:00
wizardchen 1d1e1e3d25 refactor(api-keys): update role handling and enhance API key restrictions
Change the default role for tenant API keys from 'admin' to 'owner' to align with the new authorization model. Update related tests to reflect this change, ensuring that API keys with the owner role are correctly denied access to management paths and system-admin routes. Introduce new tests to validate that API keys are properly restricted from accessing sensitive endpoints, reinforcing the security measures in place.
2026-07-07 12:00:13 +08:00
wizardchen a6759d61b5 feat(api-keys): implement DenyAPIKeyPrincipal middleware and related tests
Add DenyAPIKeyPrincipal middleware to explicitly block API-key principals from accessing certain routes, ensuring security for endpoints outside the /api/v1 group. Introduce tests to validate that API keys are correctly denied access while allowing JWT sessions to pass through. This enhances the overall authorization model by reinforcing the default-deny approach for API keys.
2026-07-07 12:00:13 +08:00
wizardchen d2ffc1b973 refactor(api-keys): simplify authorization to a single-role gate
Redesign tenant API key authorization around one dimension (role) plus a
central default-deny gate, replacing the previous role + KB-scoped flag model.

- Add APIKeyGate as the single authority for API-key authorization: routes
  declare one policy (MinRole) via apiKeyGroup/apiKeyRoute helpers; undeclared
  routes are denied by default. JWT principals short-circuit the gate and the
  JWT-centric RBAC guards short-circuit for API-key principals.
- Roles: viewer/contributor/admin are knowledge-base data-plane only (KB scope
  is a pure downstream data filter); owner is a full-access tenant tier that
  unlocks tenant-level infrastructure routes and is required for every method.
- Drop the KBScoped policy flag and the KB-restricted-vs-tenant branching.
- Security hardening: never disclose HMAC secret (return presence only), abort
  writes on API-key encryption failure, deny KB-restricted keys on /files, and
  backfill legacy key hashes at startup instead of O(N) plaintext scans.
- Frontend: two-level create dialog (access type -> permission level), rename
  role labels to reflect actual scope, sync en/zh/ko/ru locales.
- Fix startup self-check false-positive on trailing-slash routes and normalize
  gin paths before comparison; add regression tests.
2026-07-07 12:00:13 +08:00
wizardchen fedb6c3f7e test(api-keys): add tests for tenant API key management path rejection
Introduce tests to validate that tenant API keys are correctly restricted from accessing management paths. Implemented new test cases for both allowed and forbidden paths, ensuring proper enforcement of API key scopes. Additionally, refactored existing tests to utilize the new rejection logic.
2026-07-07 12:00:13 +08:00
wizardchen 9a76a24ce6 refactor(api-keys): use role field and route-level API key guards
Replace scopes JSON with a viewer/contributor/admin role column, move
unsafe-method authorization from path-prefix middleware to declarative
APIKey* guards on routes, and close remaining KB-scope gaps in handlers.
2026-07-07 12:00:13 +08:00
wizardchen c65ffa3d9c fix(api-keys): enforce scope from request context after attach
Use c.Request.Context() in authenticateAPIKeyRequest so scoped keys are
actually restricted, update dto tests for removed tenant.api_key, and add
a regression test for the attach-context contract.
2026-07-07 12:00:13 +08:00
wizardchen 94d18ea15a fix(api-keys): close KB scope gaps in handlers and CloneContext
Enforce per-key knowledge base restrictions on batch/search read paths,
centralize scope checks in shared KB access validators, and propagate
TenantAPIKeyScope through logger.CloneContext so async QA cannot bypass it.
2026-07-07 12:00:13 +08:00
wizardchen 3f516d7f31 fix(api-keys): harden scoped key auth after code review
Revoke existing keys on reset, unify scope enforcement, default empty
scopes to read-only, map API key scopes to tenant roles, and remove dead
key-generation helpers left over from the tenant api_key column removal.
2026-07-07 12:00:13 +08:00
wizardchen 505bc7ddec feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
langcaiye 1708d6895a fix(search): apply tag filters to knowledge search 2026-07-06 23:30:33 +08:00
cao_j b16cb0bb56 fix(agent): treat end_turn as natural stop 2026-07-06 23:16:18 +08:00
养乐多 baada32cb6 fix(im/feishu): actually close WebSocket long connection when channel is stopped
Deleting or disabling a Feishu channel left its WebSocket long connection
alive: the Lark SDK's Start blocks on a bare select{} and never observes
ctx, and v3.6.1 exposed no public Close(), so the wsCancel() cleanup was a
no-op. The socket stayed open, the SDK kept auto-reconnecting, and Feishu
kept pushing messages — causing cross-environment message duplication when
the same bot was bound elsewhere.

Fix in three layers:

1. Connection teardown
   - Upgrade oapi-sdk-go v3.6.1 -> v3.9.7 (adds Client.Close()).
   - Add LongConnClient.Close() wrapping wsClient.Close(); factory's cleanup
     now calls Close() first, then wsCancel() as a belt-and-braces fallback.

2. Multi-instance coordination
   - wsLeaderRetryLoop: after acquiring leadership, re-check the DB before
     starting; abort if the channel was deleted/disabled while waiting.
   - wsLeaderRenewLoop: on each renewal, re-check the DB; step down if the
     channel was deleted/disabled on another instance (HTTP request may have
     been served elsewhere), so the leader doesn't keep the connection open
     until process restart.

3. Robustness
   - DB checks distinguish gorm.ErrRecordNotFound (channel truly gone ->
     stop/abort) from transient DB errors (skip this round, retry next tick)
     so a DB hiccup can't cause a spurious stop or permanent leader-abort.
   - startChannelInternal now stops any existing channelState for the same
     ID before overwriting, making it idempotent against concurrent start
     paths and preventing adapter/long-connection leaks.

No interface changes; webhook and long-poll modes unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 23:10:45 +08:00
wizardchen 4f2dca0b63 fix(redis): harden TLS tests and surface TLS config at startup
Isolate Redis TLS unit tests from parent-process env vars and log
REDIS_USE_TLS settings at boot so misconfiguration is easier to spot.
2026-07-06 23:04:52 +08:00
qilifan d5b8d2f886 feat(redis): support TLS for Redis connections (#1930)
[Feature]: Support TLS for Redis connections
2026-07-06 23:02:02 +08:00
nullkey 445a4330fa fix(server): case-insensitive doc keyword search + not-found classification
Make document keyword search case-insensitive (LOWER(...) LIKE LOWER(?)) in the
list filter and SearchKnowledge / SearchKnowledgeInScopes paths so a query and
its differently-cased variant return the same results.

Classify not-found consistently as HTTP 404 (not 500) so clients read a
permanent resource.not_found instead of a retryable 5xx: map the document /
chunk not-found sentinels in the KB-access middleware (typed ErrChunkNotFound
via errors.Is), and return 404 for a missing message on the continue-stream and
message-delete endpoints.
2026-07-06 22:53:01 +08:00
wizardchen 1b0502e562 fix(feishu): upload card markdown images 2026-07-06 14:52:29 +08:00
wizardchen 1661827d6e fix(security): harden model client redirects 2026-07-02 19:15:46 +08:00
wizardchen 663d13eb58 fix(security): harden outbound requests and file previews 2026-07-02 19:15:46 +08:00
wizardchen 7099a7e181 fix(security): redact IM list credentials and harden MinerU SSRF
Stop returning bot credentials from Viewer-accessible IM channel list
endpoints and validate self-hosted MinerU URLs at config save and fetch time.
2026-07-02 17:58:20 +08:00
wizardchen 235e86b876 fix(security): harden refresh token validation and add auth tests
Reject refresh attempts when the DB token type is not refresh_token,
and add service-layer tests for bearer refresh rejection, logout
revocation, and expired-token logout handling.
2026-07-02 17:08:17 +08:00
wizardchen 040504447e fix(security): block refresh tokens as bearer and sandbox MCP uploads
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
2026-07-02 17:08:17 +08:00
wizardchen cd65665894 fix(security): close remaining GHSA gaps from security triage
Block scheme-prefixed loopback storage checks, validate Notion attachment
URLs, sanitize wiki markdown HTML, and require MCP_SERVER_AUTH_TOKEN for
network MCP transports.
2026-07-02 15:03:53 +08:00
wizardchen f23c674bad fix(security): close six open GHSA findings from security triage
Harden share-link listing, password rotation, OIDC callback binding,
org invite search, knowledge batch mutations, and connector redirects.
2026-07-02 15:03:53 +08:00
wizardchen fc64751f4e fix(security): close init config and parser-check redaction gaps
Strip model base URLs from viewer-accessible initialization config
responses and merge redacted parser secrets before engine check overrides
so admin test flows keep working after masked GET responses.
2026-07-02 14:41:42 +08:00
wizardchen 1a264e09c1 fix(security): harden secret redaction follow-ups for PR #1884
Restore PreserveIfRedacted merge on tenant KV PUT so Admin settings saves
no longer overwrite stored secrets with placeholder values. Redact cross-
tenant tenant listings, tighten Viewer MCP/model responses, and align init
config frontend types with credentials metadata.
2026-07-02 14:41:42 +08:00
wizardchen 512d2e5c25 fix(security): redact secrets in login and tenant KV update responses
Apply the same tenant/integration secret policy to auth login, OIDC
callback, switch-tenant, and invite-register payloads, and return masked
config objects from Admin-only tenant KV PUT handlers.
2026-07-02 14:41:42 +08:00
wizardchen 0b73080bb8 fix(security): redact integration secrets from viewer-accessible endpoints
Prevent low-privileged tenant members from reading raw API keys and
infrastructure credentials via tenant profile, KV config, model/MCP/web
search listings, and KB initialization config responses.
2026-07-02 14:41:42 +08:00
wizardchen 8325dac738 fix(security): harden web_fetch and datasource connector SSRF paths
Use SSRF-safe HTTP clients in chat pipeline web_fetch and Feishu/Yuque/Notion
connectors, validating tenant-supplied base_url values before outbound requests.
2026-07-02 12:27:48 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
wizardchen a17a2250ed fix(security): enforce KB access on wiki reads and related IDOR paths
Gate wiki GET routes and initialization config reads with KBAccessRead,
and bind async task progress endpoints to the caller tenant via task ID
parsing. Closes cross-tenant read paths reported in GHSA-ph8f-4qwx-89rr
and related initialization/progress leaks.
2026-07-02 12:06:22 +08:00
wizardchen ab790bcbd9 fix(security): close SQL validator bypass in agent database tools
Harden validateNode with default-deny semantics and PG17 SQL/JSON node
recursion so dangerous functions cannot hide inside JSON_SCALAR/JSON_VALUE.
Also recurse into FROM subqueries for DuckDB data_analysis and block
read_text/read_blob and related file-access functions.
2026-07-02 11:42:11 +08:00
wizardchen 2203c4bb14 feat(im): include per-service MCP OAuth URLs in IM auth notices
IM channels cannot complete in-conversation OAuth prompts, so when an
MCP service needs authorization the reply now includes clickable
authorization URLs (when APP_EXTERNAL_URL and OAuth manager are
available) instead of only directing users to the admin console.
2026-07-01 17:53:54 +08:00
wizardchen c694fcf6cd test: address PR review follow-ups for OAuth and logger cleanup
Restore logger config via t.Cleanup after t.Setenv so Yuque token tests
do not leak global state, add unit tests for MCP OAuth non-interactive
context helpers and IM auth notice formatting, and fix EmbedChatCore
indentation.
2026-07-01 17:31:42 +08:00
wizardchen 8d40959f88 fix lint and test failures 2026-07-01 17:31:42 +08:00
wizardchen 4c962a9403 feat(agent): add configurable MCP OAuth wait timeout per agent
Allow agents to set how long in-conversation OAuth prompts wait before
auto-skipping, with UI in the agent editor and bounded gate behavior.
2026-07-01 14:37:37 +08:00
lawrence fdfc7297de fix(vlm): allow configurable temperature for VLM models
Some VLM models (e.g. certain vision models) only accept temperature=1.
Previously temperature was hardcoded to 0.1. Now reads from model's
ExtraConfig["temperature"] with fallback to default 0.1.

Usage: set ExtraConfig.temperature=1 in the VLM model configuration.
2026-07-01 12:20:09 +08:00
wizardchen ee1014f87c feat(frontend): add API integration playground 2026-07-01 11:14:51 +08:00