Commit Graph

1164 Commits

Author SHA1 Message Date
Anky c007a897cf feat: support multi-tag for document knowledge (#1790)
* feat: support multi-tag for document knowledge

- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)

* feat(knowledge): tag edit dialog with dynamic overflow and inline creation

- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder

* fix: exclude soft-deleted knowledge from tag reference counts

CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.

* fix: refresh tag counts after async knowledge deletion completes

Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.

---------

Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-06-26 00:18:48 +08:00
wizardchen 97b7bb9890 fix(knowledge): harden batch reparse UX and async processing
Move batch reparse work to an asynq task to avoid HTTP timeouts, add i18n
for all locales, guard in-flight documents, and prevent concurrent delete/reparse.
2026-06-25 23:57:54 +08:00
jw_mac a14c1d87aa feat(knowledge): add batch reparse capability
This commit introduces the Batch Reparse feature for the Knowledge Base.
Users can now select multiple documents and trigger a re-parsing pipeline in batches, streamlining the recovery from parse failures.
2026-06-25 23:43:27 +08:00
wizardchen 47478491c9 fix(agent): address readiness review findings
Tighten shared-agent readiness checks, surface not-ready toasts in the
selector, validate chat-model overrides in resolveChatModelID, and
improve configure deep-links for shared agents.
2026-06-25 23:33:39 +08:00
wizardchen 9ca4e1f23a feat(agent): enforce model readiness and polish selector UX
Block chat when agents lack an explicit chat or rerank model, guide users
through the selector and editor with field highlight, and align session QA
checks with the same readiness rules on backend and frontend.
2026-06-25 23:33:39 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
wizardchen 2c44a41f2a feat(models): add model test debugger for saved model configs
Let admins send real requests against configured chat, embedding, rerank,
VLM, and ASR models from settings, with thinking controls shown only when
the model supports them.
2026-06-25 17:05:33 +08:00
wizardchen 122408b283 feat(chat): unify resource chips and browse recent files in @ mention
Add recent=true to knowledge search so the mention picker can list recent
files without a keyword, and consolidate KB/FAQ/file chip styling across
chat input and message history.
2026-06-24 20:58:16 +08:00
wizardchen babe3f2315 fix(integrations): polish agent filter UX and embed session rebind
Add compact agent filter beside channel list titles with selected name
display, fix embed localStorage agentId read on rebind, add IM agent
rebind tests, and deep-link agent editor counts to the matching tab.
2026-06-24 20:09:19 +08:00
wizardchen 635d7c9643 feat(integrations): polish embed/IM channel editors and agent rebind
Align embed channel setup with a step wizard, simplify deploy layout,
and allow changing bound agents on update for embed and IM channels.
2026-06-24 20:09:19 +08:00
wizardchen fa8e2bb087 fix(web-search): improve SearXNG test connection error messages
Return provider-specific diagnostics when connectivity tests return zero
results, including SearXNG unresponsive_engines details instead of a
misleading API key hint.
2026-06-24 14:28:49 +08:00
wizardchen 7c8b15f022 fix(im): harden stream finalize and drop collapsed think header
Add agent complete wait timeout and answer fallbacks for cancel paths,
restore Telegram Markdown on finalize with plain-text retry, and show
answer-only final IM frames instead of the collapsed think summary.
2026-06-24 10:33:55 +08:00
wizardchen e6b3ef4f73 fix(im): unblock agent stream wait when QA fails before complete event
Close completeDone on pipeline errors and AgentQA early failures so
handleMessageStream and runQA do not hang waiting for EventAgentComplete.
2026-06-24 10:33:55 +08:00
wizardchen da3d65c6f2 fix(im): address stream display review feedback
Wait for agent complete without a short timeout, unify Telegram stream
parse mode, remove obsolete FinalAnswer handling, isolate per-section
newline tracking, add lifecycle tests, and update IM streaming docs.
2026-06-24 10:33:55 +08:00
wizardchen c3b036cb00 feat(im): align agent stream display with web thinking and tool progress
Mirror Web AgentStreamDisplay/RagPipelineProgress in IM text streams:
retract live answers into 思考过程, show Web-aligned tool titles with
query/summary from structured data, and collapse thinking on finalize
instead of dropping it entirely.
2026-06-24 10:33:55 +08:00
wizardchen 7855f36799 fix(model): block deletion when model is referenced by KB or agent
Reject model delete with HTTP 400 when the current tenant still has
knowledge bases or custom agents bound to the model ID.
2026-06-23 20:36:16 +08:00
wizardchen bc9ad3aca0 fix(qa): wire agent thinking toggle and show reasoning in RAG timeline
Default unset agent thinking to explicit false so thinking_type models
receive disabled/enabled wire params. Render reasoning inside the quick-answer
pipeline timeline only when thinking events arrive, avoiding a flash of the
thinking step when thinking mode is off.
2026-06-23 20:17:19 +08:00
wizardchen be26563f86 fix(agent): remove unused RunnableByViewer gate blocking viewer chat
The runnable_by_viewer field was never populated for built-in agents,
so TenantRoleViewer users always hit 403 on agent-chat. Drop the field
and RBAC check so agent-chat matches knowledge-chat access for viewers.
2026-06-23 18:21:23 +08:00
wizardchen 35f717f7fa fix(datasource): optimize RSS incremental sync and surface partial failures
Skip article-page fetches when feed signals are unchanged, preserve sync
cursor across transient feed outages, and record partial multi-feed syncs.
2026-06-23 17:30:38 +08:00
wizardchen 11d461a4f7 fix(knowledgebase): delete data sources when knowledge base is removed
Stop cron schedules and cancel in-flight sync logs when a KB is deleted,
and skip datasource sync tasks if the target KB no longer exists.
2026-06-23 17:30:38 +08:00
wizardchen 3cb0e906a7 fix(datasource): harden RSS connector and editor UX after review
Scope auth headers to feed fetches, use feed-scoped external IDs and
content-based incremental fingerprints, tolerate partial feed failures,
store feed URLs in settings, and fix credential configured detection.
2026-06-23 17:30:38 +08:00
wizardchen e40ad802d7 feat(datasource): add RSS feed connector with full-text ingestion
Enable private RSS feeds as a knowledge base data source with custom
headers, article body fetching, and Markdown output for incremental sync.
2026-06-23 17:30:38 +08:00
langcaiye 3eb49da746 feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
wizardchen 7f3f219f58 feat(mcp): add OAuth2 authorization for MCP services
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.

- Persist per-user access/refresh tokens and per-service DCR client
  credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
  in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
  the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
  hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
  authorization status/actions, and render the connection test result
  inline in the editor drawer instead of a stacked centered dialog
2026-06-22 20:19:31 +08:00
wizardchen 5d0d317a36 feat(datasource): reveal pre-existing selections in lazily-loaded picker
When editing a data source whose tree loads one level at a time (Feishu
wiki), a saved deep-node selection was invisible: only top-level spaces
were listed, so the selected node could not be seen and its ancestor
space rendered as unchecked.

- Add Connector.ResolveResourceAncestors: given selected resource IDs,
  return the ancestor IDs that must be expanded to reveal them. Feishu
  walks up via GetWikiNode (O(depth) per selection, no full-tree walk);
  Notion/Yuque return empty.
- Fix wikiNode.ParentNodeID JSON tag: the Feishu API field is
  parent_node_token, not parent_node_id, so GetWikiNode can return the
  parent and the ancestor walk works in production.
- Expose POST /datasource/{id}/resource-ancestors (service + handler +
  route) and a matching frontend API client.
- Edit-mode picker reveals hidden selections by expanding and lazily
  loading each ancestor level, so selections are visible and checked.
- Gate child fetching on treeFullyLoaded instead of a partial-children
  heuristic, avoiding a redundant request for full-tree connectors.
2026-06-22 19:24:08 +08:00
wizardchen d8f13ada4a fix(datasource): lazy-load Feishu wiki resources to avoid listing timeout (#1672)
Listing resources for a Feishu data source eagerly walked the entire wiki
tree (one API call per node across every space), which exceeded the request
timeout for large knowledge bases. ListResources now takes a parentID and
returns a single level at a time: empty lists the wiki spaces, a space/node
ID lists only its direct children. The frontend fetches children on demand
when a node is expanded. Notion (full tree in one call) and Yuque (flat)
keep their existing behavior by ignoring non-empty parentID.

Refs: https://github.com/Tencent/WeKnora/issues/1672
2026-06-22 17:53:30 +08:00
wizardchen 74c845a1f4 fix(docparser): normalize PaddleOCR-VL HTML tables (#1725)
PaddleOCR-VL renders tables as HTML with per-cell text-align styles.
These waste tokens on layout markup and bypass the chunker's
table-protection logic (which only recognizes Markdown pipe tables),
causing large tables to be split mid-row.

Convert inline HTML <table> blocks to GFM Markdown tables when possible,
and strip presentational attributes (style/class/align/...) for tables
using rowspan/colspan that Markdown cannot express.

Refs: https://github.com/Tencent/WeKnora/issues/1725
2026-06-22 17:31:26 +08:00
wizardchen abf6a99b90 fix(api): add POST for hybrid-search while keeping GET compatibility (#1727)
GET with a JSON body was non-standard and confused integrators. Register
POST as the preferred method and keep GET for existing clients; update SDK,
MCP server, and API docs accordingly.

Refs: https://github.com/Tencent/WeKnora/issues/1727
2026-06-22 17:22:35 +08:00
wizardchen 632cd55018 feat(im): integrate IM service for custom agent deletion
- Add IM service dependency to CustomAgentHandler for managing agent-related IM channels.
- Implement DeleteChannelsByAgent method to stop and soft-delete IM channels associated with a custom agent upon its removal.
- Update ListChannelsByTenant to exclude soft-deleted custom agents, ensuring consistent overview lists after agent removal.
2026-06-22 17:16:07 +08:00
wizardchen ecc27e9b0f fix(docparser): rename close variable to avoid shadowing builtin
The angle-bracket image target parser declared a local variable named
`close`, which shadows Go's built-in `close` function and is flagged by
revive's redefines-builtin-id rule, breaking `make lint`. Rename it to
`closeIdx`.
2026-06-22 15:57:41 +08:00
FFFFFFpy a6a20175e5 fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
wizardchen 01819b5bc2 fix(im): prevent DingTalk stream connection from going stale
DingTalk websocket (Stream) channels stopped delivering @-mentions after
a day or two and only recovered when the IM channel was toggled off/on.
The factory fired the SDK's Start once and relied entirely on the SDK's
internal auto-reconnect, with no liveness supervision on WeKnora's side.
On long-running connections the SDK can enter a zombie state where the
connection object stays alive but no longer receives messages, leaving no
way to recover automatically. The cancel func was also ineffective because
the SDK ignores the context after the initial dial, so stopping a channel
leaked the old connection.

Add a reusable RunSupervised helper that keeps a long connection alive and
proactively recycles it every 6h on top of the SDK's own reconnect, which
bounds the worst-case outage and clears stuck connections. Wire DingTalk to
it, add LongConnClient.Close (disables AutoReconnect before closing so the
SDK does not resurrect the connection), and make the cancel func actually
tear the connection down. Drop the now-unused stream client reference from
the adapter, which only sends replies via sessionWebhook or OpenAPI.
2026-06-22 15:04:23 +08:00
adawang 3b317f7eef perf(utils): compile SQL-validation regexps once instead of per call
checkSQLInjectionRisks and InjectAndConditions compiled their regexps
with regexp.MustCompile on every invocation. Both run on the datasource
SQL path — InjectAndConditions is called once per security filter
(tenant, soft-delete, hidden-KB, search-scope), so a single secured query
recompiled ~20 patterns. InjectAndConditions also compiled the identical
tail-clause pattern twice.

Hoist them to package-level vars (compiled once at init). Behavior is
unchanged — same patterns, covered by the existing TestValidateSQL_*
and TestInjectAndConditions tests.

Benchmarks (apple M, -benchmem):
  InjectAndConditions    4347 ns, 72 allocs -> 1023 ns, 7 allocs
  checkSQLInjectionRisks 36074 ns, 560 allocs -> 12984 ns, 12 allocs

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 14:17:17 +08:00
wizardchen c3cd629da7 fix(tenant): purge memberships on delete and filter stale switcher rows
Cascade soft-delete tenant_members when a workspace is removed so the
tenant switcher no longer shows ghost entries as "#<id>". Also filter
orphaned memberships in /auth/me and switch to another workspace after
delete when one remains.
2026-06-17 10:31:53 +08:00
“jiangyijun” 6bc8e9ed49 fix(vectordb): 修复 question 数据复制到向量数据库的 ID 映射问题
主要改进:
1. CopyIndices 新增分页查询逻辑(每页500条),避免大批量数据查询失败
2. 修复 ID 字段优先级:优先使用 IndexInfo.ID,兜底使用 ChunkID
3. 新增 translateSourceID 函数,智能处理 question 类型数据的 SourceID 映射:
   - 当 SourceID == ChunkID 时,直接映射为目标 ChunkID
   - 当 SourceID 为 "ChunkID-QuestionID" 格式时,保留 QuestionID 部分
   - 其他情况使用 SHA256 生成稳定的映射 ID

影响范围:
- 修改文件:internal/application/repository/retriever/tencentvectordb/repository.go
- 新增函数:copySourceQueryParams、remapCopiedEmbeddings、translateSourceID
2026-06-17 05:08:50 +08:00
wizardchen 16363623ab feat(wiki): enhance anchored form popups and internationalization updates
Add new styles for anchored form popups in the wiki browser, improving the user interface for folder actions and confirmations. Update internationalization strings for move confirmation titles in English, Korean, Russian, and Chinese locales to enhance user experience across multiple languages. Refactor existing components to integrate the new popup styles and ensure consistent behavior.
2026-06-17 00:28:30 +08:00
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen 88c10da0b0 refactor(wiki): unify category cleaning and push directory listing to SQL
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.

Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.

Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
adawang ec66a3dac7 perf(common): compile LLM-JSON code-fence regexp once
ParseLLMJsonResponse recompiled its ```json fence-extraction regexp on
every call that fell through the direct-unmarshal fast path — i.e. every
fenced LLM response on the graph-extraction path. Hoist it to a
package-level var.

Adds a test (none existed) covering direct, fenced, no-lang and
prose-wrapped inputs, plus a benchmark.

Benchmark, fenced payload (-benchmem):
  before: 3205 ns/op, 5712 B/op, 54 allocs/op
  after:  1315 ns/op,  693 B/op, 15 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:52:00 +08:00
adawang 8f4b3333f2 perf(sandbox): compile command-substitution regexps once
hasCommandSubstitution recompiled its three regexps with regexp.MustCompile
on every call, and ValidateArgs calls it once per argument — so validating
an N-arg command compiled 3N patterns. Hoist them to a package-level slice
compiled once at init.

Behavior is unchanged (same patterns); covered by the existing
TestScriptValidator_ValidateArgs / ValidateAll tests.

Benchmark, ValidateArgs with 8 args (-benchmem):
  before: 16374 ns/op, 40142 B/op, 505 allocs/op
  after:   2092 ns/op,    32 B/op,   1 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:51:32 +08:00
adawang 1c735ebb9e feat(wiki): front-load static rules in chunk-citation prompt for prefix cache
WikiChunkCitationPrompt placed the per-batch <chunks> block before the
static rules, output schema and the per-document-stable <candidate_slugs>.
Within one document only ChunksXML changes between batches, so the stable
content trailing the chunks could not form a long common prefix and
providers re-billed the static rules on every citation batch (issue #1687
measured ~6.4% prefix-cache hit rate on wiki synthesis).

Reorder to [rules | output schema | candidate_slugs | chunks] so every
batch after the first shares the long [rules | candidate_slugs] prefix.
No instruction text was dropped; only block order and the inline
above/below directional references changed. Verified the cacheable prefix
grows from 281 to 2366 bytes and is byte-identical across batches.

Addresses #1687

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:58:12 +08:00
adawang 8d3030d339 fix(chunker): carry deep sub-heading context into large-section chunks
When the dominant heading level repeats and becomes the split backbone,
a long section is sub-split via SplitText and every sub-chunk inherited
the section-level breadcrumb. Deep `###`/`####` headings that define a
sub-chunk's meaning were dropped from Chunk.ContextHeader, so queries
about content nested far below a shallow heading failed to retrieve it.

Track the heading hierarchy across the section and assign each sub-chunk
the breadcrumb in effect at its start offset, so the deepest active
heading reaches the breadcrumb. Adds a regression test.

Fixes #1674

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:56:57 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen efe8b6f41d feat(agent): add tool result persistence trimming for large outputs
Introduce display_type-based output stripping for SSE/DB storage and
add knowledge_chunks_list display type for list_knowledge_chunks.
2026-06-16 01:09:23 +08:00
wizardchen 16f2a46465 feat(agent): rerank all knowledge search hits with threshold filtering
Remove FAQ bypass in agent rerank, apply composite model scores, and
filter low-confidence results using configurable rerank_threshold.
2026-06-16 01:09:23 +08:00
wizardchen de525a4f02 feat(observability): enhance Langfuse retrieve and rerank tracing
Add shared retrieval observation helpers, expand rerank generation output
with previews and score stats, and enrich retrieve/rerank pipeline spans.
2026-06-16 01:09:23 +08:00