Commit Graph

2107 Commits

Author SHA1 Message Date
wizardchen b5acb89b4d feat(frontend): enhance chat stream handling with assistant message management
- Introduce mechanisms to mark assistant messages as completed and handle in-flight messages during user queries.
- Implement functions to prepare for new outgoing messages and manage the stopping of assistant messages.
- Update chat stream handling to ensure proper message state management and prevent reuse of incomplete messages.
- Integrate new functionalities into existing composables for improved chat experience.
2026-06-18 18:42:16 +08:00
wizardchen c6b5d2e77b feat(frontend): enhance chat scroll handling for improved user experience
- Implement logic to accurately detect user scroll direction, preventing false positives during content loading.
- Update scroll handling in both `useEmbedChatSession` and chat view to maintain auto-follow functionality during streaming.
- Introduce a mechanism to differentiate between actual upward scrolls and content growth, enhancing chat usability.
2026-06-18 18:42:16 +08:00
wizardchen 80bcf200b6 feat(frontend): enhance chat image handling and typewriter effect
- Introduce a stable skeleton box for protected images during loading to prevent flickering.
- Implement a new composable `useStickyBottomOnResize` to maintain chat scroll position as content loads.
- Add `useTypewriter` for smoother text rendering in chat, improving user experience during streaming.
- Create a directive `vStableHtml` to efficiently update HTML content without losing image states.
- Enhance chat markdown rendering to handle incomplete citation tags and trailing horizontal rules during streaming.
- Add tests for new typewriter functionality and citation handling.
2026-06-18 18:42:16 +08:00
wizardchen 95b97dc379 style(frontend): update component styles for consistency and usability
- Change background colors in menu and user message components for better visual coherence
- Adjust text colors and font weights in UserMenu for improved readability
- Enhance suggested questions layout with updated spacing and alignment
- Refine chat message styles for better responsiveness and user experience
- Update hover effects and transitions for suggested question cards
2026-06-18 18:42:16 +08:00
wizardchen 4e2d568149 feat(frontend): polish chat suggestions, citations, and drawers
- unify suggested-question styling across chat entry points
- keep citation markers inline with their preceding list content
- prevent manual editor viewport shifts and synchronize drawer motion
- align embedded preview and editor drawers in the body stacking context
2026-06-18 18:42:16 +08:00
wizardchen e0d3c1dafd feat(frontend): improve manual knowledge editor and chat citation rendering
- Replace manual knowledge editor dialog with SettingDrawer and refine toolbar/footer layout
- Add draft status resolution for manual knowledge items in the knowledge base
- Fix citation placement near fenced code blocks and add regression tests
- Update ChatRequestInfoButton icon and chat message/citation styles
- Add i18n strings for manual knowledge editor features
2026-06-17 20:52:37 +08:00
JojaCola312 4c5379df9c test(docreader): align pdf JPEG quality default assertion 2026-06-17 17:18:57 +08:00
adawang 3b317f7eef perf(utils): compile SQL-validation regexps once instead of per call
checkSQLInjectionRisks and InjectAndConditions compiled their regexps
with regexp.MustCompile on every invocation. Both run on the datasource
SQL path — InjectAndConditions is called once per security filter
(tenant, soft-delete, hidden-KB, search-scope), so a single secured query
recompiled ~20 patterns. InjectAndConditions also compiled the identical
tail-clause pattern twice.

Hoist them to package-level vars (compiled once at init). Behavior is
unchanged — same patterns, covered by the existing TestValidateSQL_*
and TestInjectAndConditions tests.

Benchmarks (apple M, -benchmem):
  InjectAndConditions    4347 ns, 72 allocs -> 1023 ns, 7 allocs
  checkSQLInjectionRisks 36074 ns, 560 allocs -> 12984 ns, 12 allocs

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 14:17:17 +08:00
wizardchen c3cd629da7 fix(tenant): purge memberships on delete and filter stale switcher rows
Cascade soft-delete tenant_members when a workspace is removed so the
tenant switcher no longer shows ghost entries as "#<id>". Also filter
orphaned memberships in /auth/me and switch to another workspace after
delete when one remains.
2026-06-17 10:31:53 +08:00
langcaiye fd5eecb4e3 feat: add tenant workspace deletion UI 2026-06-17 05:09:46 +08:00
“jiangyijun” 6bc8e9ed49 fix(vectordb): 修复 question 数据复制到向量数据库的 ID 映射问题
主要改进:
1. CopyIndices 新增分页查询逻辑(每页500条),避免大批量数据查询失败
2. 修复 ID 字段优先级:优先使用 IndexInfo.ID,兜底使用 ChunkID
3. 新增 translateSourceID 函数,智能处理 question 类型数据的 SourceID 映射:
   - 当 SourceID == ChunkID 时,直接映射为目标 ChunkID
   - 当 SourceID 为 "ChunkID-QuestionID" 格式时,保留 QuestionID 部分
   - 其他情况使用 SHA256 生成稳定的映射 ID

影响范围:
- 修改文件:internal/application/repository/retriever/tencentvectordb/repository.go
- 新增函数:copySourceQueryParams、remapCopiedEmbeddings、translateSourceID
2026-06-17 05:08:50 +08:00
FFFFFFpy 408364c6c9 fix(docreader): restore epub and mhtml parser registration 2026-06-17 00:30:51 +08:00
wizardchen 16363623ab feat(wiki): enhance anchored form popups and internationalization updates
Add new styles for anchored form popups in the wiki browser, improving the user interface for folder actions and confirmations. Update internationalization strings for move confirmation titles in English, Korean, Russian, and Chinese locales to enhance user experience across multiple languages. Refactor existing components to integrate the new popup styles and ensure consistent behavior.
2026-06-17 00:28:30 +08:00
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 07e0f6f21a feat(session-sidebar): enhance menu positioning and behavior
Refactor the session sidebar menu to use a Teleport component for improved positioning and visibility. Introduce a trigger reference for dynamic menu placement based on viewport dimensions. Update event listeners for menu interactions to ensure proper opening and closing behavior, enhancing user experience. Adjust CSS for fixed positioning and z-index to prevent overlap issues.
2026-06-16 23:51:29 +08:00
wizardchen c8cb4fab39 feat(wiki): show category hierarchy and navigation in wiki browser
Render the wiki page category hierarchy in the browser, wire up the new
category-path/availability API calls, and add the corresponding i18n strings
across en-US, ko-KR, ru-RU and zh-CN locales.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen 88c10da0b0 refactor(wiki): unify category cleaning and push directory listing to SQL
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.

Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.

Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
adawang ec66a3dac7 perf(common): compile LLM-JSON code-fence regexp once
ParseLLMJsonResponse recompiled its ```json fence-extraction regexp on
every call that fell through the direct-unmarshal fast path — i.e. every
fenced LLM response on the graph-extraction path. Hoist it to a
package-level var.

Adds a test (none existed) covering direct, fenced, no-lang and
prose-wrapped inputs, plus a benchmark.

Benchmark, fenced payload (-benchmem):
  before: 3205 ns/op, 5712 B/op, 54 allocs/op
  after:  1315 ns/op,  693 B/op, 15 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:52:00 +08:00
adawang 8f4b3333f2 perf(sandbox): compile command-substitution regexps once
hasCommandSubstitution recompiled its three regexps with regexp.MustCompile
on every call, and ValidateArgs calls it once per argument — so validating
an N-arg command compiled 3N patterns. Hoist them to a package-level slice
compiled once at init.

Behavior is unchanged (same patterns); covered by the existing
TestScriptValidator_ValidateArgs / ValidateAll tests.

Benchmark, ValidateArgs with 8 args (-benchmem):
  before: 16374 ns/op, 40142 B/op, 505 allocs/op
  after:   2092 ns/op,    32 B/op,   1 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:51:32 +08:00
adawang 1c735ebb9e feat(wiki): front-load static rules in chunk-citation prompt for prefix cache
WikiChunkCitationPrompt placed the per-batch <chunks> block before the
static rules, output schema and the per-document-stable <candidate_slugs>.
Within one document only ChunksXML changes between batches, so the stable
content trailing the chunks could not form a long common prefix and
providers re-billed the static rules on every citation batch (issue #1687
measured ~6.4% prefix-cache hit rate on wiki synthesis).

Reorder to [rules | output schema | candidate_slugs | chunks] so every
batch after the first shares the long [rules | candidate_slugs] prefix.
No instruction text was dropped; only block order and the inline
above/below directional references changed. Verified the cacheable prefix
grows from 281 to 2366 bytes and is byte-identical across batches.

Addresses #1687

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:58:12 +08:00
adawang 8d3030d339 fix(chunker): carry deep sub-heading context into large-section chunks
When the dominant heading level repeats and becomes the split backbone,
a long section is sub-split via SplitText and every sub-chunk inherited
the section-level breadcrumb. Deep `###`/`####` headings that define a
sub-chunk's meaning were dropped from Chunk.ContextHeader, so queries
about content nested far below a shallow heading failed to retrieve it.

Track the heading hierarchy across the section and assign each sub-chunk
the breadcrumb in effect at its start offset, so the deepest active
heading reaches the breadcrumb. Adds a regression test.

Fixes #1674

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:56:57 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
adawang 6efd2093f5 fix(frontend): list writable shared KBs in manual knowledge editor
The "add to knowledge base" action on the chat page opens the manual
editor, whose target picker was populated only from listKnowledgeBases()
— the caller's own KBs. A user who was granted editor access to a shared
KB therefore could not select it, even though the backend
(CreateManualKnowledge) accepts editor/admin permission.

Also fetch shared knowledge bases and merge those the user can write to
(editor/admin) into the picker, de-duplicated against owned KBs and
limited to document-type KBs. Viewer-only shares stay excluded.

Fixes #1689

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:41:38 +08:00
wizardchen b58232e65b feat(docreader): add quick availability probe for opendataloader parser
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
2026-06-16 16:40:54 +08:00
FFFFFFpy 6d07e20204 fix(wiki): use placeholder src for protected provider images to avoid failed initial loads 2026-06-16 12:24:01 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen fe7aded648 feat(i18n): add translations for citations, RAG progress, and tool results
Sync embed and locale strings for citation popover, pipeline progress,
agent tool status, and knowledge chunks list summaries.
2026-06-16 01:21:19 +08:00
wizardchen 283dbbd577 feat(chat): improve chat layout and suggested questions UX
Refine input field, session sidebar, suggested questions refresh, and
embed channel preview for a cleaner chat creation experience.
2026-06-16 01:21:19 +08:00
wizardchen c6f1b54f73 feat(chat): refactor agent stream display and tool result rendering
Add RagPipelineProgress timeline, simplify AgentStreamDisplay structure,
group grep hits per document, and add KnowledgeChunksList tool result UI.
2026-06-16 01:21:19 +08:00
wizardchen 05f39b859f feat(chat): add citation popover and chunk caching
Introduce ChatCitationFloat hover UI, session-scoped citation chunk cache,
and unified citation markdown handling for main chat and embed views.
2026-06-16 01:09:55 +08:00
wizardchen ee1c3c732c feat(chat): add shared markdown rendering pipeline
Extract chat markdown styles, renderer, DOMPurify config, syntax
highlighting, and mermaid support into reusable utilities with tests.
2026-06-16 01:09:55 +08:00
wizardchen efe8b6f41d feat(agent): add tool result persistence trimming for large outputs
Introduce display_type-based output stripping for SSE/DB storage and
add knowledge_chunks_list display type for list_knowledge_chunks.
2026-06-16 01:09:23 +08:00
wizardchen 16f2a46465 feat(agent): rerank all knowledge search hits with threshold filtering
Remove FAQ bypass in agent rerank, apply composite model scores, and
filter low-confidence results using configurable rerank_threshold.
2026-06-16 01:09:23 +08:00
wizardchen de525a4f02 feat(observability): enhance Langfuse retrieve and rerank tracing
Add shared retrieval observation helpers, expand rerank generation output
with previews and score stats, and enrich retrieve/rerank pipeline spans.
2026-06-16 01:09:23 +08:00
wizardchen 9e916839ef feat(backend): add RAG pipeline progress events
Emit query-understanding and knowledge-search progress over SSE, emit
references before stream completion, and wire progress into session QA.
2026-06-16 01:09:07 +08:00
wizardchen 2906c6656e feat(backend): improve parent-child chunk merge and imageinfo matching
Scope image info to matched text children during parent resolution and
add range/URL filtering utilities with unit tests.
2026-06-16 01:09:07 +08:00
wizardchen 8dbc20b9b3 feat(im): enhance IMChannelPanel and SettingDrawer components
Refactor settings drawer for consistency, add role-based empty states,
and improve save loading UX for IM channel configuration.
2026-06-16 00:55:23 +08:00
wizardchen 53e0333a53 Add embedding dimension override support 2026-06-15 19:18:19 +08:00
wizardchen 90f8e8b390 feat(auth): clear session resource caches on logout to prevent data leakage 2026-06-15 17:46:08 +08:00
MaoMeng cf0a64b88e fix:dimensions not sent in embedding API requests (#1654)
fix:dimensions not sent in embedding API requests
2026-06-15 17:32:23 +08:00
FFFFFFpy a997441392 fix(wiki): mask image URLs before LLM to prevent UUID/path corruption
Wiki summary/entity/concept/page-modify prompts instructed the LLM to
reproduce ![caption](url) verbatim, so high-entropy image URLs passed
through generation and got corrupted (dropped UUID segments, duplicated
or mangled path prefixes, even CJK chars inserted into the bucket name),
producing object keys that don't exist -> 404.

Mask all image URLs to opaque placeholders at the generateWithTemplate
chokepoint and deterministically rehydrate after generation; drop
corrupted/hallucinated placeholders so broken links never reach storage.
2026-06-15 17:27:56 +08:00
wizardchen f0dcf85528 feat(i18n): add 'Open documentation' string to multiple language files 2026-06-15 17:20:40 +08:00
wizardchen 20b01a6cd5 feat(embed): session sidebar grouping and embed theme alignment (#1657)
Group embed sessions in the sidebar by channel, replace session source filter
with grouping buckets, and align shared chat components with embed theme colors.
2026-06-12 23:10:39 +08:00
wizardchen 08852ab19d feat(embed): add widget, visitor chat, and management UI (#1657)
Add embed.html entrypoint, floating widget SDK, secure-mode snippets,
management panel with preview, visitor chat composables, citation rendering,
and integration docs for subdomain and server-side token exchange.
2026-06-12 23:10:39 +08:00
wizardchen ed9802ff2b feat(embed): add backend embed channels with auth and rate limiting (#1657)
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
2026-06-12 23:10:39 +08:00
langcaiye a0786b6fe7 fix(wiki): clear stale citation chunks on deletion 2026-06-12 23:08:20 +08:00
langcaiye 903b014992 fix(tcvdb): keep default replica number at one 2026-06-12 22:54:44 +08:00