Commit Graph

2098 Commits

Author SHA1 Message Date
langcaiye fd5eecb4e3 feat: add tenant workspace deletion UI 2026-06-17 05:09:46 +08:00
“jiangyijun” 6bc8e9ed49 fix(vectordb): 修复 question 数据复制到向量数据库的 ID 映射问题
主要改进:
1. CopyIndices 新增分页查询逻辑(每页500条),避免大批量数据查询失败
2. 修复 ID 字段优先级:优先使用 IndexInfo.ID,兜底使用 ChunkID
3. 新增 translateSourceID 函数,智能处理 question 类型数据的 SourceID 映射:
   - 当 SourceID == ChunkID 时,直接映射为目标 ChunkID
   - 当 SourceID 为 "ChunkID-QuestionID" 格式时,保留 QuestionID 部分
   - 其他情况使用 SHA256 生成稳定的映射 ID

影响范围:
- 修改文件:internal/application/repository/retriever/tencentvectordb/repository.go
- 新增函数:copySourceQueryParams、remapCopiedEmbeddings、translateSourceID
2026-06-17 05:08:50 +08:00
FFFFFFpy 408364c6c9 fix(docreader): restore epub and mhtml parser registration 2026-06-17 00:30:51 +08:00
wizardchen 16363623ab feat(wiki): enhance anchored form popups and internationalization updates
Add new styles for anchored form popups in the wiki browser, improving the user interface for folder actions and confirmations. Update internationalization strings for move confirmation titles in English, Korean, Russian, and Chinese locales to enhance user experience across multiple languages. Refactor existing components to integrate the new popup styles and ensure consistent behavior.
2026-06-17 00:28:30 +08:00
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 07e0f6f21a feat(session-sidebar): enhance menu positioning and behavior
Refactor the session sidebar menu to use a Teleport component for improved positioning and visibility. Introduce a trigger reference for dynamic menu placement based on viewport dimensions. Update event listeners for menu interactions to ensure proper opening and closing behavior, enhancing user experience. Adjust CSS for fixed positioning and z-index to prevent overlap issues.
2026-06-16 23:51:29 +08:00
wizardchen c8cb4fab39 feat(wiki): show category hierarchy and navigation in wiki browser
Render the wiki page category hierarchy in the browser, wire up the new
category-path/availability API calls, and add the corresponding i18n strings
across en-US, ko-KR, ru-RU and zh-CN locales.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen 88c10da0b0 refactor(wiki): unify category cleaning and push directory listing to SQL
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.

Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.

Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
adawang ec66a3dac7 perf(common): compile LLM-JSON code-fence regexp once
ParseLLMJsonResponse recompiled its ```json fence-extraction regexp on
every call that fell through the direct-unmarshal fast path — i.e. every
fenced LLM response on the graph-extraction path. Hoist it to a
package-level var.

Adds a test (none existed) covering direct, fenced, no-lang and
prose-wrapped inputs, plus a benchmark.

Benchmark, fenced payload (-benchmem):
  before: 3205 ns/op, 5712 B/op, 54 allocs/op
  after:  1315 ns/op,  693 B/op, 15 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:52:00 +08:00
adawang 8f4b3333f2 perf(sandbox): compile command-substitution regexps once
hasCommandSubstitution recompiled its three regexps with regexp.MustCompile
on every call, and ValidateArgs calls it once per argument — so validating
an N-arg command compiled 3N patterns. Hoist them to a package-level slice
compiled once at init.

Behavior is unchanged (same patterns); covered by the existing
TestScriptValidator_ValidateArgs / ValidateAll tests.

Benchmark, ValidateArgs with 8 args (-benchmem):
  before: 16374 ns/op, 40142 B/op, 505 allocs/op
  after:   2092 ns/op,    32 B/op,   1 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:51:32 +08:00
adawang 1c735ebb9e feat(wiki): front-load static rules in chunk-citation prompt for prefix cache
WikiChunkCitationPrompt placed the per-batch <chunks> block before the
static rules, output schema and the per-document-stable <candidate_slugs>.
Within one document only ChunksXML changes between batches, so the stable
content trailing the chunks could not form a long common prefix and
providers re-billed the static rules on every citation batch (issue #1687
measured ~6.4% prefix-cache hit rate on wiki synthesis).

Reorder to [rules | output schema | candidate_slugs | chunks] so every
batch after the first shares the long [rules | candidate_slugs] prefix.
No instruction text was dropped; only block order and the inline
above/below directional references changed. Verified the cacheable prefix
grows from 281 to 2366 bytes and is byte-identical across batches.

Addresses #1687

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:58:12 +08:00
adawang 8d3030d339 fix(chunker): carry deep sub-heading context into large-section chunks
When the dominant heading level repeats and becomes the split backbone,
a long section is sub-split via SplitText and every sub-chunk inherited
the section-level breadcrumb. Deep `###`/`####` headings that define a
sub-chunk's meaning were dropped from Chunk.ContextHeader, so queries
about content nested far below a shallow heading failed to retrieve it.

Track the heading hierarchy across the section and assign each sub-chunk
the breadcrumb in effect at its start offset, so the deepest active
heading reaches the breadcrumb. Adds a regression test.

Fixes #1674

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:56:57 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
adawang 6efd2093f5 fix(frontend): list writable shared KBs in manual knowledge editor
The "add to knowledge base" action on the chat page opens the manual
editor, whose target picker was populated only from listKnowledgeBases()
— the caller's own KBs. A user who was granted editor access to a shared
KB therefore could not select it, even though the backend
(CreateManualKnowledge) accepts editor/admin permission.

Also fetch shared knowledge bases and merge those the user can write to
(editor/admin) into the picker, de-duplicated against owned KBs and
limited to document-type KBs. Viewer-only shares stay excluded.

Fixes #1689

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:41:38 +08:00
wizardchen b58232e65b feat(docreader): add quick availability probe for opendataloader parser
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
2026-06-16 16:40:54 +08:00
FFFFFFpy 6d07e20204 fix(wiki): use placeholder src for protected provider images to avoid failed initial loads 2026-06-16 12:24:01 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen fe7aded648 feat(i18n): add translations for citations, RAG progress, and tool results
Sync embed and locale strings for citation popover, pipeline progress,
agent tool status, and knowledge chunks list summaries.
2026-06-16 01:21:19 +08:00
wizardchen 283dbbd577 feat(chat): improve chat layout and suggested questions UX
Refine input field, session sidebar, suggested questions refresh, and
embed channel preview for a cleaner chat creation experience.
2026-06-16 01:21:19 +08:00
wizardchen c6f1b54f73 feat(chat): refactor agent stream display and tool result rendering
Add RagPipelineProgress timeline, simplify AgentStreamDisplay structure,
group grep hits per document, and add KnowledgeChunksList tool result UI.
2026-06-16 01:21:19 +08:00
wizardchen 05f39b859f feat(chat): add citation popover and chunk caching
Introduce ChatCitationFloat hover UI, session-scoped citation chunk cache,
and unified citation markdown handling for main chat and embed views.
2026-06-16 01:09:55 +08:00
wizardchen ee1c3c732c feat(chat): add shared markdown rendering pipeline
Extract chat markdown styles, renderer, DOMPurify config, syntax
highlighting, and mermaid support into reusable utilities with tests.
2026-06-16 01:09:55 +08:00
wizardchen efe8b6f41d feat(agent): add tool result persistence trimming for large outputs
Introduce display_type-based output stripping for SSE/DB storage and
add knowledge_chunks_list display type for list_knowledge_chunks.
2026-06-16 01:09:23 +08:00
wizardchen 16f2a46465 feat(agent): rerank all knowledge search hits with threshold filtering
Remove FAQ bypass in agent rerank, apply composite model scores, and
filter low-confidence results using configurable rerank_threshold.
2026-06-16 01:09:23 +08:00
wizardchen de525a4f02 feat(observability): enhance Langfuse retrieve and rerank tracing
Add shared retrieval observation helpers, expand rerank generation output
with previews and score stats, and enrich retrieve/rerank pipeline spans.
2026-06-16 01:09:23 +08:00
wizardchen 9e916839ef feat(backend): add RAG pipeline progress events
Emit query-understanding and knowledge-search progress over SSE, emit
references before stream completion, and wire progress into session QA.
2026-06-16 01:09:07 +08:00
wizardchen 2906c6656e feat(backend): improve parent-child chunk merge and imageinfo matching
Scope image info to matched text children during parent resolution and
add range/URL filtering utilities with unit tests.
2026-06-16 01:09:07 +08:00
wizardchen 8dbc20b9b3 feat(im): enhance IMChannelPanel and SettingDrawer components
Refactor settings drawer for consistency, add role-based empty states,
and improve save loading UX for IM channel configuration.
2026-06-16 00:55:23 +08:00
wizardchen 53e0333a53 Add embedding dimension override support 2026-06-15 19:18:19 +08:00
wizardchen 90f8e8b390 feat(auth): clear session resource caches on logout to prevent data leakage 2026-06-15 17:46:08 +08:00
MaoMeng cf0a64b88e fix:dimensions not sent in embedding API requests (#1654)
fix:dimensions not sent in embedding API requests
2026-06-15 17:32:23 +08:00
FFFFFFpy a997441392 fix(wiki): mask image URLs before LLM to prevent UUID/path corruption
Wiki summary/entity/concept/page-modify prompts instructed the LLM to
reproduce ![caption](url) verbatim, so high-entropy image URLs passed
through generation and got corrupted (dropped UUID segments, duplicated
or mangled path prefixes, even CJK chars inserted into the bucket name),
producing object keys that don't exist -> 404.

Mask all image URLs to opaque placeholders at the generateWithTemplate
chokepoint and deterministically rehydrate after generation; drop
corrupted/hallucinated placeholders so broken links never reach storage.
2026-06-15 17:27:56 +08:00
wizardchen f0dcf85528 feat(i18n): add 'Open documentation' string to multiple language files 2026-06-15 17:20:40 +08:00
wizardchen 20b01a6cd5 feat(embed): session sidebar grouping and embed theme alignment (#1657)
Group embed sessions in the sidebar by channel, replace session source filter
with grouping buckets, and align shared chat components with embed theme colors.
2026-06-12 23:10:39 +08:00
wizardchen 08852ab19d feat(embed): add widget, visitor chat, and management UI (#1657)
Add embed.html entrypoint, floating widget SDK, secure-mode snippets,
management panel with preview, visitor chat composables, citation rendering,
and integration docs for subdomain and server-side token exchange.
2026-06-12 23:10:39 +08:00
wizardchen ed9802ff2b feat(embed): add backend embed channels with auth and rate limiting (#1657)
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
2026-06-12 23:10:39 +08:00
langcaiye a0786b6fe7 fix(wiki): clear stale citation chunks on deletion 2026-06-12 23:08:20 +08:00
langcaiye 903b014992 fix(tcvdb): keep default replica number at one 2026-06-12 22:54:44 +08:00
langcaiye d07a62af06 fix(tcvdb): improve collection compatibility 2026-06-12 22:54:44 +08:00
kshitijshresth 1e98739dbf Validate sync log limit parameter 2026-06-12 22:52:25 +08:00
langcaiye d832dae818 fix(datasource): clarify edit does not sync 2026-06-12 22:49:37 +08:00
nullkey 5851c64ba1 feat(im): IM session improvements — titles, in-flight recovery, classification
Three related improvements to the IM-session experience:

- Auto-generated titles: a new IM session starts untitled and gets a
  content-based title from its first message via the same GenerateTitleAsync
  flow web sessions use, instead of an identity title like "user · dm". Falls
  back to the identity title when the first message has no text. All platforms.

- In-flight reply recovery (web chat): opening a session whose reply is still
  being generated on an IM channel made the web client attach via
  continue-stream and fail (the IM answer never streams through this server),
  flashing a "stream failed" toast and forcing a manual refresh. Treat a failed
  attach to an IM reply (channel === 'im') as recoverable — poll until it
  completes, then reload the thread; on timeout surface the standard error.
  Web/api replies keep the original behaviour, untouched.

- Source-filter classification: the sidebar filter treats "web" as sessions
  with no IM mapping, but the list join excluded soft-deleted mappings, so
  /clear'd IM sessions leaked into the user's own web chats. Include soft-deleted
  mappings so a session ever bound to an IM channel stays under its platform.
  Adds a repository regression test.
2026-06-12 17:01:28 +08:00
nullkey ac84dacc00 feat(frontend): add session source filter to sidebar (#1560)
IM-channel sessions were mixed in with the user's own chats. The sidebar
now defaults to web sessions; tenants with a configured IM channel get a
source switcher in the "对话" header to reach a specific platform.

- getSessionsList: optional `source` query param
- getMessageList: token-guarded loads (racing source switches / route
  changes can't append stale rows) + per-source pagination reset
- SourceSwitcherDropdown: scope dropdown mirroring KBSwitcherDropdown,
  scales to many platforms, reuses platform logos, closes on select
- empty state when a source has no sessions
- pure logic in sessionSourceFilter.ts with unit tests; zh/en/ko/ru i18n
2026-06-12 17:01:28 +08:00
jw_mac 232280d959 fix:lack the completed status set 2026-06-12 10:59:18 +08:00
wizardchen ebe7bfe0ae refactor(settings): unify empty state handling across settings views
- Updated McpSettings.vue, ModelSettings.vue, VectorStoreSettings.vue, and WebSearchSettings.vue to streamline the empty state display for services and models.
- Removed admin-specific add buttons from empty states, enhancing clarity for non-admin users.
- Improved consistency in the presentation of empty states across different settings components.
2026-06-11 17:18:53 +08:00
wizardchen 8112cdbf85 feat(menu): enhance menu component with new chat functionality and localization updates
- Updated the menu component to include a new chat option, improving user navigation.
- Refactored the menu layout to allow for sticky behavior during scrolling, enhancing usability.
- Added localization support for the new chat feature in English, Korean, Russian, and Chinese.
- Adjusted the menu store to reflect the new chat option and maintain consistent menu structure.
2026-06-11 16:28:16 +08:00
wizardchen 00ad8adb71 feat(menu): enhance sidebar functionality with search and toggle buttons
- Updated the menu component to include a search button alongside the sidebar toggle in the expanded view, improving accessibility to search functionality.
- Refactored the layout to maintain a consistent design for the logo actions, ensuring a seamless user experience.
- Adjusted CSS styles for the sidebar toggle and search button for better alignment and responsiveness.
- Improved tooltip handling for the search feature to provide clear user guidance.
2026-06-11 14:38:50 +08:00