Commit Graph

688 Commits

Author SHA1 Message Date
nullkey 445a4330fa fix(server): case-insensitive doc keyword search + not-found classification
Make document keyword search case-insensitive (LOWER(...) LIKE LOWER(?)) in the
list filter and SearchKnowledge / SearchKnowledgeInScopes paths so a query and
its differently-cased variant return the same results.

Classify not-found consistently as HTTP 404 (not 500) so clients read a
permanent resource.not_found instead of a retryable 5xx: map the document /
chunk not-found sentinels in the KB-access middleware (typed ErrChunkNotFound
via errors.Is), and return 404 for a missing message on the continue-stream and
message-delete endpoints.
2026-07-06 22:53:01 +08:00
wizardchen 663d13eb58 fix(security): harden outbound requests and file previews 2026-07-02 19:15:46 +08:00
wizardchen 235e86b876 fix(security): harden refresh token validation and add auth tests
Reject refresh attempts when the DB token type is not refresh_token,
and add service-layer tests for bearer refresh rejection, logout
revocation, and expired-token logout handling.
2026-07-02 17:08:17 +08:00
wizardchen 040504447e fix(security): block refresh tokens as bearer and sandbox MCP uploads
Reject refresh JWTs in ValidateToken, revoke all user sessions on logout,
and restrict create_knowledge_from_file to allowed directories on network MCP transports.
2026-07-02 17:08:17 +08:00
wizardchen f23c674bad fix(security): close six open GHSA findings from security triage
Harden share-link listing, password rotation, OIDC callback binding,
org invite search, knowledge batch mutations, and connector redirects.
2026-07-02 15:03:53 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
wizardchen 4c962a9403 feat(agent): add configurable MCP OAuth wait timeout per agent
Allow agents to set how long in-conversation OAuth prompts wait before
auto-skipping, with UI in the agent editor and bounded gate behavior.
2026-07-01 14:37:37 +08:00
wizardchen 5074344c16 feat(embed): scope OAuth and sessions per visitor principal
Add embed visitor principals (X-Embed-Visitor), widen sessions.user_id,
embed MCP OAuth/approval routes, OAuth callback context detachment, and
CloneContext principal preservation for async QA streams.
2026-06-30 20:01:05 +08:00
wizardchen 97bca74a31 fix: harden API principal config after review
Add require_direct_header toggle, external user ID validation, JWT TTL
limits, SaveToken error handling, ru-RU i18n, API docs, and Go SDK methods.
2026-06-30 20:01:05 +08:00
wizardchen cb99736d72 feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
wizardchen 505195865a fix(mcp): trigger OAuth prompt during tool registration and dedupe helpers
OAuth authorization was only handled in MCPTool.Execute, so unauthenticated
services failed silently during RegisterMCPTools before any tool reached the
agent. Wire EventBus/session metadata into registration and reuse the same
getOrCreateMCPClientWithOAuthRetry path used at execution time.

Extract MCP OAuth session/wait/connect logic into mcp_oauth.go and remove the
duplicate waitForOAuthAndRetry + RegisterMCPContext layering in mcp_tool.go.
2026-06-29 23:22:28 +08:00
wizardchen f2686e9e39 feat(mcp): drive auth by AuthType and auto-detect OAuth on test
Select the auth header for the configured strategy only, instead of
emitting both X-API-Key and Authorization whenever the fields happened
to be set (which double-authed after switching strategies). AuthType=""
keeps the legacy infer-from-fields behavior for pre-existing rows.

Add a configurable api_key_header (non-secret, defaults to X-API-Key) so
services expecting the key in a different header work without plaintext
custom headers.

On connection test, upgrade a bare 401 that advertises RFC 9728
protected-resource metadata into an explicit OAuthRequired signal so the
UI can guide the user to switch to OAuth 2.0. Surface the server's
description from the initialize handshake in the test result.
2026-06-29 22:37:52 +08:00
wizardchen c29761331c fix(agent): enforce tag scopes across knowledge tools 2026-06-29 19:17:42 +08:00
wizardchen 6ad75d19fd test(knowledge): unbreak service test package after tag-loading change
Fixing the knowledge_create_test compile error in this branch let the
internal/application/service test package run for the first time, which
exposed two latent failures introduced when GetKnowledgeByID started
eagerly loading tags:

- GetOwningKBCreatorID went through the service-level GetKnowledgeByID
  (which now calls repo.GetKnowledgeTags); ownership resolution only
  needs the knowledge -> kb_id link, so resolve via repo.GetKnowledgeByID
  directly (tenant-scoped) and drop the extra tag query.
- createKnowledgeFileRepoStub lacked GetKnowledgeTags, which the create
  path always invokes via setAndAttachKnowledgeTags; add the stub method.
2026-06-29 18:09:44 +08:00
wizardchen d24e830c85 refactor(agent): fix grep scope loss and consolidate per-turn @mention scoping
- Fix grep_chunks dropping the full-KB scope when a turn mixes @KB with
  @tag/@file: extract scopeClause and OR knowledge-ID and (kb,tenant)
  predicates together, matching knowledge_search's fan-out.
- Consolidate the three scattered per-request @Skill/@MCP scope blocks in
  buildAgentConfig into applyPerRequestSkillScope / applyPerRequestMCPScope,
  each narrowing the whitelist and computing the must_use pin in one place.
- Dedupe the two identical selection-mode normalizers in Input-field.vue.
- Fix knowledge_create_test passing "" for the tagIDs []string parameter,
  which made the whole service test package fail to compile.
- Add unit tests for scopeClause and the per-request scope helpers.
2026-06-29 18:09:44 +08:00
wizardchen b86efe3f92 fix(agent): harden @mention scope and simplify must_use hints
Tighten per-turn @MCP/@Skill/@tag handling after review: enforce agent
MCP/Skill whitelist, ignore mentions when selection mode is none, tag-scope
grep_chunks, and simplified English must_use lines without tool dumps.
Skip persisting scope envelopes to history rendered_content.
2026-06-29 18:09:44 +08:00
wizardchen 049f85139d feat(chat): add @Skill/@MCP mentions with scoped agent runtime
Wire per-turn @mentions for skills, MCP services, tags, and files through
QA handlers into agent config, runtime_context/must_use user blocks, and
tool registration. Fix tag-only scopes dropping KB tools, refresh suggested
questions on mention changes, and strengthen MCP must_use so @selected
services are invoked before local KB search.
2026-06-29 18:09:44 +08:00
wizardchen 40f217c228 test: fix stale Doris SQL-shape and flaky OSS bucket assertions
Two pre-existing test failures on main, unrelated to any feature change:

- Doris VectorRetrieve/KeywordsRetrieve inline LIMIT as a literal (Doris
  does not support placeholders for LIMIT/OFFSET), but the SQL-shape tests
  still expected `LIMIT ?` with TopK as a bound arg. Match the literal
  LIMIT and drop the extra argument.
- TestOssEnsureBucket_CreateFails used the bucket name "test-bucket",
  which already exists globally on OSS, so IsBucketExist short-circuited
  and the create path never ran. Use a non-existent bucket name so the
  create-with-invalid-credentials path is actually exercised.
2026-06-29 17:44:56 +08:00
wizardchen 2f29054110 fix(knowledge): harden multi-tag data integrity and UX follow-ups
Clear tag relations on move/delete, validate tags on create, batch-load tags,
restore KB-scoped reference counts, dedupe SetKnowledgeTags, fix filter
double-fetch and ResizeObserver leaks, and regenerate Swagger for tag_ids.
2026-06-26 10:52:52 +08:00
Anky c007a897cf feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge

- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)

* feat(knowledge): tag edit dialog with dynamic overflow and inline creation

- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder

* fix: exclude soft-deleted knowledge from tag reference counts

CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.

* fix: refresh tag counts after async knowledge deletion completes

Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.

---------

Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-26 00:18:48 +08:00
wizardchen 97b7bb9890 fix(knowledge): harden batch reparse UX and async processing
Move batch reparse work to an asynq task to avoid HTTP timeouts, add i18n
for all locales, guard in-flight documents, and prevent concurrent delete/reparse.
2026-06-25 23:57:54 +08:00
wizardchen 47478491c9 fix(agent): address readiness review findings
Tighten shared-agent readiness checks, surface not-ready toasts in the
selector, validate chat-model overrides in resolveChatModelID, and
improve configure deep-links for shared agents.
2026-06-25 23:33:39 +08:00
wizardchen 9ca4e1f23a feat(agent): enforce model readiness and polish selector UX
Block chat when agents lack an explicit chat or rerank model, guide users
through the selector and editor with field highlight, and align session QA
checks with the same readiness rules on backend and frontend.
2026-06-25 23:33:39 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
wizardchen 635d7c9643 feat(integrations): polish embed/IM channel editors and agent rebind
Align embed channel setup with a step wizard, simplify deploy layout,
and allow changing bound agents on update for embed and IM channels.
2026-06-24 20:09:19 +08:00
wizardchen 7855f36799 fix(model): block deletion when model is referenced by KB or agent
Reject model delete with HTTP 400 when the current tenant still has
knowledge bases or custom agents bound to the model ID.
2026-06-23 20:36:16 +08:00
wizardchen bc9ad3aca0 fix(qa): wire agent thinking toggle and show reasoning in RAG timeline
Default unset agent thinking to explicit false so thinking_type models
receive disabled/enabled wire params. Render reasoning inside the quick-answer
pipeline timeline only when thinking events arrive, avoiding a flash of the
thinking step when thinking mode is off.
2026-06-23 20:17:19 +08:00
wizardchen 35f717f7fa fix(datasource): optimize RSS incremental sync and surface partial failures
Skip article-page fetches when feed signals are unchanged, preserve sync
cursor across transient feed outages, and record partial multi-feed syncs.
2026-06-23 17:30:38 +08:00
wizardchen 11d461a4f7 fix(knowledgebase): delete data sources when knowledge base is removed
Stop cron schedules and cancel in-flight sync logs when a KB is deleted,
and skip datasource sync tasks if the target KB no longer exists.
2026-06-23 17:30:38 +08:00
wizardchen 3cb0e906a7 fix(datasource): harden RSS connector and editor UX after review
Scope auth headers to feed fetches, use feed-scoped external IDs and
content-based incremental fingerprints, tolerate partial feed failures,
store feed URLs in settings, and fix credential configured detection.
2026-06-23 17:30:38 +08:00
langcaiye 3eb49da746 feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
wizardchen 7f3f219f58 feat(mcp): add OAuth2 authorization for MCP services
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.

- Persist per-user access/refresh tokens and per-service DCR client
  credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
  in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
  the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
  hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
  authorization status/actions, and render the connection test result
  inline in the editor drawer instead of a stacked centered dialog
2026-06-22 20:19:31 +08:00
wizardchen 5d0d317a36 feat(datasource): reveal pre-existing selections in lazily-loaded picker
When editing a data source whose tree loads one level at a time (Feishu
wiki), a saved deep-node selection was invisible: only top-level spaces
were listed, so the selected node could not be seen and its ancestor
space rendered as unchecked.

- Add Connector.ResolveResourceAncestors: given selected resource IDs,
  return the ancestor IDs that must be expanded to reveal them. Feishu
  walks up via GetWikiNode (O(depth) per selection, no full-tree walk);
  Notion/Yuque return empty.
- Fix wikiNode.ParentNodeID JSON tag: the Feishu API field is
  parent_node_token, not parent_node_id, so GetWikiNode can return the
  parent and the ancestor walk works in production.
- Expose POST /datasource/{id}/resource-ancestors (service + handler +
  route) and a matching frontend API client.
- Edit-mode picker reveals hidden selections by expanding and lazily
  loading each ancestor level, so selections are visible and checked.
- Gate child fetching on treeFullyLoaded instead of a partial-children
  heuristic, avoiding a redundant request for full-tree connectors.
2026-06-22 19:24:08 +08:00
wizardchen d8f13ada4a fix(datasource): lazy-load Feishu wiki resources to avoid listing timeout (#1672)
Listing resources for a Feishu data source eagerly walked the entire wiki
tree (one API call per node across every space), which exceeded the request
timeout for large knowledge bases. ListResources now takes a parentID and
returns a single level at a time: empty lists the wiki spaces, a space/node
ID lists only its direct children. The frontend fetches children on demand
when a node is expanded. Notion (full tree in one call) and Yuque (flat)
keep their existing behavior by ignoring non-empty parentID.

Refs: https://github.com/Tencent/WeKnora/issues/1672
2026-06-22 17:53:30 +08:00
wizardchen c3cd629da7 fix(tenant): purge memberships on delete and filter stale switcher rows
Cascade soft-delete tenant_members when a workspace is removed so the
tenant switcher no longer shows ghost entries as "#<id>". Also filter
orphaned memberships in /auth/me and switch to another workspace after
delete when one remains.
2026-06-17 10:31:53 +08:00
“jiangyijun” 6bc8e9ed49 fix(vectordb): 修复 question 数据复制到向量数据库的 ID 映射问题
主要改进:
1. CopyIndices 新增分页查询逻辑(每页500条),避免大批量数据查询失败
2. 修复 ID 字段优先级:优先使用 IndexInfo.ID,兜底使用 ChunkID
3. 新增 translateSourceID 函数,智能处理 question 类型数据的 SourceID 映射:
   - 当 SourceID == ChunkID 时,直接映射为目标 ChunkID
   - 当 SourceID 为 "ChunkID-QuestionID" 格式时,保留 QuestionID 部分
   - 其他情况使用 SHA256 生成稳定的映射 ID

影响范围:
- 修改文件:internal/application/repository/retriever/tencentvectordb/repository.go
- 新增函数:copySourceQueryParams、remapCopiedEmbeddings、translateSourceID
2026-06-17 05:08:50 +08:00
wizardchen 16363623ab feat(wiki): enhance anchored form popups and internationalization updates
Add new styles for anchored form popups in the wiki browser, improving the user interface for folder actions and confirmations. Update internationalization strings for move confirmation titles in English, Korean, Russian, and Chinese locales to enhance user experience across multiple languages. Refactor existing components to integrate the new popup styles and ensure consistent behavior.
2026-06-17 00:28:30 +08:00
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen 88c10da0b0 refactor(wiki): unify category cleaning and push directory listing to SQL
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.

Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.

Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen efe8b6f41d feat(agent): add tool result persistence trimming for large outputs
Introduce display_type-based output stripping for SSE/DB storage and
add knowledge_chunks_list display type for list_knowledge_chunks.
2026-06-16 01:09:23 +08:00
wizardchen de525a4f02 feat(observability): enhance Langfuse retrieve and rerank tracing
Add shared retrieval observation helpers, expand rerank generation output
with previews and score stats, and enrich retrieve/rerank pipeline spans.
2026-06-16 01:09:23 +08:00
wizardchen 9e916839ef feat(backend): add RAG pipeline progress events
Emit query-understanding and knowledge-search progress over SSE, emit
references before stream completion, and wire progress into session QA.
2026-06-16 01:09:07 +08:00
wizardchen 2906c6656e feat(backend): improve parent-child chunk merge and imageinfo matching
Scope image info to matched text children during parent resolution and
add range/URL filtering utilities with unit tests.
2026-06-16 01:09:07 +08:00
FFFFFFpy a997441392 fix(wiki): mask image URLs before LLM to prevent UUID/path corruption
Wiki summary/entity/concept/page-modify prompts instructed the LLM to
reproduce ![caption](url) verbatim, so high-entropy image URLs passed
through generation and got corrupted (dropped UUID segments, duplicated
or mangled path prefixes, even CJK chars inserted into the bucket name),
producing object keys that don't exist -> 404.

Mask all image URLs to opaque placeholders at the generateWithTemplate
chokepoint and deterministically rehydrate after generation; drop
corrupted/hallucinated placeholders so broken links never reach storage.
2026-06-15 17:27:56 +08:00
wizardchen ed9802ff2b feat(embed): add backend embed channels with auth and rate limiting (#1657)
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
2026-06-12 23:10:39 +08:00