Commit Graph

2124 Commits

Author SHA1 Message Date
wizardchen 74c845a1f4 fix(docparser): normalize PaddleOCR-VL HTML tables (#1725)
PaddleOCR-VL renders tables as HTML with per-cell text-align styles.
These waste tokens on layout markup and bypass the chunker's
table-protection logic (which only recognizes Markdown pipe tables),
causing large tables to be split mid-row.

Convert inline HTML <table> blocks to GFM Markdown tables when possible,
and strip presentational attributes (style/class/align/...) for tables
using rowspan/colspan that Markdown cannot express.

Refs: https://github.com/Tencent/WeKnora/issues/1725
2026-06-22 17:31:26 +08:00
wizardchen abf6a99b90 fix(api): add POST for hybrid-search while keeping GET compatibility (#1727)
GET with a JSON body was non-standard and confused integrators. Register
POST as the preferred method and keep GET for existing clients; update SDK,
MCP server, and API docs accordingly.

Refs: https://github.com/Tencent/WeKnora/issues/1727
2026-06-22 17:22:35 +08:00
wizardchen 632cd55018 feat(im): integrate IM service for custom agent deletion
- Add IM service dependency to CustomAgentHandler for managing agent-related IM channels.
- Implement DeleteChannelsByAgent method to stop and soft-delete IM channels associated with a custom agent upon its removal.
- Update ListChannelsByTenant to exclude soft-deleted custom agents, ensuring consistent overview lists after agent removal.
2026-06-22 17:16:07 +08:00
wizardchen ddc5460988 fix(frontend): update UserMenu to use localized string for system management
Replace hardcoded "系统管理" text with the localized key for system management in UserMenu.vue, enhancing internationalization support.
2026-06-22 15:57:41 +08:00
wizardchen ecc27e9b0f fix(docparser): rename close variable to avoid shadowing builtin
The angle-bracket image target parser declared a local variable named
`close`, which shadows Go's built-in `close` function and is flagged by
revive's redefines-builtin-id rule, breaking `make lint`. Rename it to
`closeIdx`.
2026-06-22 15:57:41 +08:00
FFFFFFpy a6a20175e5 fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
wizardchen 5f7ed2ffe9 feat(i18n): add key descriptions for various settings in multiple languages
- Introduce detailed descriptions for authentication, SSRF protection, tenant management, and async task worker settings in English, Korean, Russian, and Chinese localization files.
- Enhance user understanding of configuration options by providing context-sensitive help directly within the settings interface.
- Update the Settings and SystemSettings components to utilize these new descriptions for improved user experience.
2026-06-22 15:21:50 +08:00
wizardchen 01819b5bc2 fix(im): prevent DingTalk stream connection from going stale
DingTalk websocket (Stream) channels stopped delivering @-mentions after
a day or two and only recovered when the IM channel was toggled off/on.
The factory fired the SDK's Start once and relied entirely on the SDK's
internal auto-reconnect, with no liveness supervision on WeKnora's side.
On long-running connections the SDK can enter a zombie state where the
connection object stays alive but no longer receives messages, leaving no
way to recover automatically. The cancel func was also ineffective because
the SDK ignores the context after the initial dial, so stopping a channel
leaked the old connection.

Add a reusable RunSupervised helper that keeps a long connection alive and
proactively recycles it every 6h on top of the SDK's own reconnect, which
bounds the worst-case outage and clears stuck connections. Wire DingTalk to
it, add LongConnClient.Close (disables AutoReconnect before closing so the
SDK does not resurrect the connection), and make the cancel func actually
tear the connection down. Drop the now-unused stream client reference from
the adapter, which only sends replies via sessionWebhook or OpenAPI.
2026-06-22 15:04:23 +08:00
wizardchen 10e63b2227 feat(frontend): add MCP config section and unify modal navigation styles
- Add dedicated MCP service configuration section in AgentEditorModal (agent mode only)
- Unify sidebar nav hover/active states across settings and editor modals
- Refine tool card and tag chip colors to use design tokens and color-mix
- Tighten suggested questions button padding and border-radius
2026-06-22 14:33:54 +08:00
wizardchen 8ada130ee1 fix(frontend): enhance repairFlankingEmphasis to handle opening delimiters correctly
- Update regex patterns to account for opening delimiters preceded by letters/numbers and followed by punctuation, ensuring proper rendering of emphasis in markdown.
- Add tests to validate new behavior for various edge cases, including punctuation-adjacent emphasis and exponent markers.
- Improve overall markdown rendering consistency in chat components.
2026-06-22 12:31:34 +08:00
wizardchen 483f507104 feat(frontend): add wiki tool support and enhance agent tool icon mapping
- Implement `getWikiPageText` function to handle persisted slugs and JSON input for wiki page text retrieval.
- Update `getAgentToolIconName` to include mappings for `wiki_search` and `wiki_read_page` tools.
- Enhance tests to cover new functionality for wiki tools and ensure proper icon mapping in the agent stream display.
- Modify `AgentStreamDisplay` to utilize new wiki tool features and improve localization for wiki-related actions.
2026-06-22 12:25:28 +08:00
Binks Riverton 0c3c866f0b fix(frontend): prevent settings modal from overflowing viewport on small screens
Add max-height: calc(100vh - 40px) to .settings-modal so it never exceeds
the viewport height minus overlay padding, keeping the close button always
visible without requiring F11 fullscreen.
2026-06-22 12:03:40 +08:00
Binks Riverton 9739ebc2ee @
fix(frontend): lower UploadConfirmDialog z-index so Settings modal appears above it

Reduce the overlay z-index from 2500 to 1000 so that the global
Settings modal (z-index: 1100) opens on top of the upload confirm
dialog instead of behind it.

Previously, when a user uploaded image files without a multimodal
model configured, clicking "前往全局设置添加模型" would open the
Settings modal behind the upload confirm dialog, forcing the user
to close the dialog first. With the z-index lowered to the
project-wide modal convention (1000), the Settings modal now
displays correctly above this dialog.
@
2026-06-22 12:02:54 +08:00
wizardchen 31f05cf425 feat(frontend): add directory view toggle and improve localization
- Introduce a view mode toggle for directory display in the WikiBrowser component, allowing users to switch between tree and list views.
- Update localization files for English, Korean, Russian, and Chinese to include new strings for the view mode toggle and its associated labels.
- Enhance user experience by providing clear visual options for navigating directory structures.
2026-06-22 11:56:47 +08:00
wizardchen cc9de85c25 feat(frontend): enhance chat streaming visuals with shimmer effect and fade tail
- Introduce a shimmer effect for in-progress step titles in chat components, providing visual feedback during processing.
- Implement a trailing fade effect for the most recently streamed characters in chat markdown, enhancing the typing experience.
- Update CSS styles to support the new shimmer and fade effects, ensuring smooth transitions and visual coherence.
- Add tests to verify the functionality of the shimmer effect and the application of the fade tail in markdown rendering.
2026-06-22 10:48:53 +08:00
wizardchen 58461a5b68 feat(frontend): enhance chat markdown rendering and streaming behavior
- Introduce functions to stabilize rendering of dangling inline markers during streaming, preventing flickering and layout shifts.
- Implement tagging for standalone strong paragraphs to ensure consistent styling as subtitles.
- Update CSS for chat components to improve animation smoothness and visual feedback during typing.
- Add tests for new markdown rendering features, including handling of incomplete emphasis and citation tags.
2026-06-22 10:48:53 +08:00
wizardchen ae9038732a style(frontend): update ListSpaceSidebar styles for improved hover and active states
- Change background colors for hover and active states to enhance visual feedback.
- Adjust icon label colors and remove unnecessary font weight properties for better consistency.
- Refine item count background color to align with updated design standards.
2026-06-18 18:50:19 +08:00
wizardchen b5acb89b4d feat(frontend): enhance chat stream handling with assistant message management
- Introduce mechanisms to mark assistant messages as completed and handle in-flight messages during user queries.
- Implement functions to prepare for new outgoing messages and manage the stopping of assistant messages.
- Update chat stream handling to ensure proper message state management and prevent reuse of incomplete messages.
- Integrate new functionalities into existing composables for improved chat experience.
2026-06-18 18:42:16 +08:00
wizardchen c6b5d2e77b feat(frontend): enhance chat scroll handling for improved user experience
- Implement logic to accurately detect user scroll direction, preventing false positives during content loading.
- Update scroll handling in both `useEmbedChatSession` and chat view to maintain auto-follow functionality during streaming.
- Introduce a mechanism to differentiate between actual upward scrolls and content growth, enhancing chat usability.
2026-06-18 18:42:16 +08:00
wizardchen 80bcf200b6 feat(frontend): enhance chat image handling and typewriter effect
- Introduce a stable skeleton box for protected images during loading to prevent flickering.
- Implement a new composable `useStickyBottomOnResize` to maintain chat scroll position as content loads.
- Add `useTypewriter` for smoother text rendering in chat, improving user experience during streaming.
- Create a directive `vStableHtml` to efficiently update HTML content without losing image states.
- Enhance chat markdown rendering to handle incomplete citation tags and trailing horizontal rules during streaming.
- Add tests for new typewriter functionality and citation handling.
2026-06-18 18:42:16 +08:00
wizardchen 95b97dc379 style(frontend): update component styles for consistency and usability
- Change background colors in menu and user message components for better visual coherence
- Adjust text colors and font weights in UserMenu for improved readability
- Enhance suggested questions layout with updated spacing and alignment
- Refine chat message styles for better responsiveness and user experience
- Update hover effects and transitions for suggested question cards
2026-06-18 18:42:16 +08:00
wizardchen 4e2d568149 feat(frontend): polish chat suggestions, citations, and drawers
- unify suggested-question styling across chat entry points
- keep citation markers inline with their preceding list content
- prevent manual editor viewport shifts and synchronize drawer motion
- align embedded preview and editor drawers in the body stacking context
2026-06-18 18:42:16 +08:00
wizardchen e0d3c1dafd feat(frontend): improve manual knowledge editor and chat citation rendering
- Replace manual knowledge editor dialog with SettingDrawer and refine toolbar/footer layout
- Add draft status resolution for manual knowledge items in the knowledge base
- Fix citation placement near fenced code blocks and add regression tests
- Update ChatRequestInfoButton icon and chat message/citation styles
- Add i18n strings for manual knowledge editor features
2026-06-17 20:52:37 +08:00
JojaCola312 4c5379df9c test(docreader): align pdf JPEG quality default assertion 2026-06-17 17:18:57 +08:00
adawang 3b317f7eef perf(utils): compile SQL-validation regexps once instead of per call
checkSQLInjectionRisks and InjectAndConditions compiled their regexps
with regexp.MustCompile on every invocation. Both run on the datasource
SQL path — InjectAndConditions is called once per security filter
(tenant, soft-delete, hidden-KB, search-scope), so a single secured query
recompiled ~20 patterns. InjectAndConditions also compiled the identical
tail-clause pattern twice.

Hoist them to package-level vars (compiled once at init). Behavior is
unchanged — same patterns, covered by the existing TestValidateSQL_*
and TestInjectAndConditions tests.

Benchmarks (apple M, -benchmem):
  InjectAndConditions    4347 ns, 72 allocs -> 1023 ns, 7 allocs
  checkSQLInjectionRisks 36074 ns, 560 allocs -> 12984 ns, 12 allocs

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 14:17:17 +08:00
wizardchen c3cd629da7 fix(tenant): purge memberships on delete and filter stale switcher rows
Cascade soft-delete tenant_members when a workspace is removed so the
tenant switcher no longer shows ghost entries as "#<id>". Also filter
orphaned memberships in /auth/me and switch to another workspace after
delete when one remains.
2026-06-17 10:31:53 +08:00
langcaiye fd5eecb4e3 feat: add tenant workspace deletion UI 2026-06-17 05:09:46 +08:00
“jiangyijun” 6bc8e9ed49 fix(vectordb): 修复 question 数据复制到向量数据库的 ID 映射问题
主要改进:
1. CopyIndices 新增分页查询逻辑(每页500条),避免大批量数据查询失败
2. 修复 ID 字段优先级:优先使用 IndexInfo.ID,兜底使用 ChunkID
3. 新增 translateSourceID 函数,智能处理 question 类型数据的 SourceID 映射:
   - 当 SourceID == ChunkID 时,直接映射为目标 ChunkID
   - 当 SourceID 为 "ChunkID-QuestionID" 格式时,保留 QuestionID 部分
   - 其他情况使用 SHA256 生成稳定的映射 ID

影响范围:
- 修改文件:internal/application/repository/retriever/tencentvectordb/repository.go
- 新增函数:copySourceQueryParams、remapCopiedEmbeddings、translateSourceID
2026-06-17 05:08:50 +08:00
FFFFFFpy 408364c6c9 fix(docreader): restore epub and mhtml parser registration 2026-06-17 00:30:51 +08:00
wizardchen 16363623ab feat(wiki): enhance anchored form popups and internationalization updates
Add new styles for anchored form popups in the wiki browser, improving the user interface for folder actions and confirmations. Update internationalization strings for move confirmation titles in English, Korean, Russian, and Chinese locales to enhance user experience across multiple languages. Refactor existing components to integrate the new popup styles and ensure consistent behavior.
2026-06-17 00:28:30 +08:00
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 07e0f6f21a feat(session-sidebar): enhance menu positioning and behavior
Refactor the session sidebar menu to use a Teleport component for improved positioning and visibility. Introduce a trigger reference for dynamic menu placement based on viewport dimensions. Update event listeners for menu interactions to ensure proper opening and closing behavior, enhancing user experience. Adjust CSS for fixed positioning and z-index to prevent overlap issues.
2026-06-16 23:51:29 +08:00
wizardchen c8cb4fab39 feat(wiki): show category hierarchy and navigation in wiki browser
Render the wiki page category hierarchy in the browser, wire up the new
category-path/availability API calls, and add the corresponding i18n strings
across en-US, ko-KR, ru-RU and zh-CN locales.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen 88c10da0b0 refactor(wiki): unify category cleaning and push directory listing to SQL
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.

Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.

Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
adawang ec66a3dac7 perf(common): compile LLM-JSON code-fence regexp once
ParseLLMJsonResponse recompiled its ```json fence-extraction regexp on
every call that fell through the direct-unmarshal fast path — i.e. every
fenced LLM response on the graph-extraction path. Hoist it to a
package-level var.

Adds a test (none existed) covering direct, fenced, no-lang and
prose-wrapped inputs, plus a benchmark.

Benchmark, fenced payload (-benchmem):
  before: 3205 ns/op, 5712 B/op, 54 allocs/op
  after:  1315 ns/op,  693 B/op, 15 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:52:00 +08:00
adawang 8f4b3333f2 perf(sandbox): compile command-substitution regexps once
hasCommandSubstitution recompiled its three regexps with regexp.MustCompile
on every call, and ValidateArgs calls it once per argument — so validating
an N-arg command compiled 3N patterns. Hoist them to a package-level slice
compiled once at init.

Behavior is unchanged (same patterns); covered by the existing
TestScriptValidator_ValidateArgs / ValidateAll tests.

Benchmark, ValidateArgs with 8 args (-benchmem):
  before: 16374 ns/op, 40142 B/op, 505 allocs/op
  after:   2092 ns/op,    32 B/op,   1 allocs/op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 19:51:32 +08:00
adawang 1c735ebb9e feat(wiki): front-load static rules in chunk-citation prompt for prefix cache
WikiChunkCitationPrompt placed the per-batch <chunks> block before the
static rules, output schema and the per-document-stable <candidate_slugs>.
Within one document only ChunksXML changes between batches, so the stable
content trailing the chunks could not form a long common prefix and
providers re-billed the static rules on every citation batch (issue #1687
measured ~6.4% prefix-cache hit rate on wiki synthesis).

Reorder to [rules | output schema | candidate_slugs | chunks] so every
batch after the first shares the long [rules | candidate_slugs] prefix.
No instruction text was dropped; only block order and the inline
above/below directional references changed. Verified the cacheable prefix
grows from 281 to 2366 bytes and is byte-identical across batches.

Addresses #1687

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:58:12 +08:00
adawang 8d3030d339 fix(chunker): carry deep sub-heading context into large-section chunks
When the dominant heading level repeats and becomes the split backbone,
a long section is sub-split via SplitText and every sub-chunk inherited
the section-level breadcrumb. Deep `###`/`####` headings that define a
sub-chunk's meaning were dropped from Chunk.ContextHeader, so queries
about content nested far below a shallow heading failed to retrieve it.

Track the heading hierarchy across the section and assign each sub-chunk
the breadcrumb in effect at its start offset, so the deepest active
heading reaches the breadcrumb. Adds a regression test.

Fixes #1674

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:56:57 +08:00
langcaiye 0c3d45d3cd fix: mark knowledge delete failures after retry exhaustion 2026-06-16 16:47:46 +08:00
adawang 6efd2093f5 fix(frontend): list writable shared KBs in manual knowledge editor
The "add to knowledge base" action on the chat page opens the manual
editor, whose target picker was populated only from listKnowledgeBases()
— the caller's own KBs. A user who was granted editor access to a shared
KB therefore could not select it, even though the backend
(CreateManualKnowledge) accepts editor/admin permission.

Also fetch shared knowledge bases and merge those the user can write to
(editor/admin) into the picker, de-duplicated against owned KBs and
limited to document-type KBs. Viewer-only shares stay excluded.

Fixes #1689

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 16:41:38 +08:00
wizardchen b58232e65b feat(docreader): add quick availability probe for opendataloader parser
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
2026-06-16 16:40:54 +08:00
FFFFFFpy 6d07e20204 fix(wiki): use placeholder src for protected provider images to avoid failed initial loads 2026-06-16 12:24:01 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen fe7aded648 feat(i18n): add translations for citations, RAG progress, and tool results
Sync embed and locale strings for citation popover, pipeline progress,
agent tool status, and knowledge chunks list summaries.
2026-06-16 01:21:19 +08:00
wizardchen 283dbbd577 feat(chat): improve chat layout and suggested questions UX
Refine input field, session sidebar, suggested questions refresh, and
embed channel preview for a cleaner chat creation experience.
2026-06-16 01:21:19 +08:00
wizardchen c6f1b54f73 feat(chat): refactor agent stream display and tool result rendering
Add RagPipelineProgress timeline, simplify AgentStreamDisplay structure,
group grep hits per document, and add KnowledgeChunksList tool result UI.
2026-06-16 01:21:19 +08:00
wizardchen 05f39b859f feat(chat): add citation popover and chunk caching
Introduce ChatCitationFloat hover UI, session-scoped citation chunk cache,
and unified citation markdown handling for main chat and embed views.
2026-06-16 01:09:55 +08:00