PaddleOCR-VL renders tables as HTML with per-cell text-align styles.
These waste tokens on layout markup and bypass the chunker's
table-protection logic (which only recognizes Markdown pipe tables),
causing large tables to be split mid-row.
Convert inline HTML <table> blocks to GFM Markdown tables when possible,
and strip presentational attributes (style/class/align/...) for tables
using rowspan/colspan that Markdown cannot express.
Refs: https://github.com/Tencent/WeKnora/issues/1725
GET with a JSON body was non-standard and confused integrators. Register
POST as the preferred method and keep GET for existing clients; update SDK,
MCP server, and API docs accordingly.
Refs: https://github.com/Tencent/WeKnora/issues/1727
- Add IM service dependency to CustomAgentHandler for managing agent-related IM channels.
- Implement DeleteChannelsByAgent method to stop and soft-delete IM channels associated with a custom agent upon its removal.
- Update ListChannelsByTenant to exclude soft-deleted custom agents, ensuring consistent overview lists after agent removal.
The angle-bracket image target parser declared a local variable named
`close`, which shadows Go's built-in `close` function and is flagged by
revive's redefines-builtin-id rule, breaking `make lint`. Rename it to
`closeIdx`.
DingTalk websocket (Stream) channels stopped delivering @-mentions after
a day or two and only recovered when the IM channel was toggled off/on.
The factory fired the SDK's Start once and relied entirely on the SDK's
internal auto-reconnect, with no liveness supervision on WeKnora's side.
On long-running connections the SDK can enter a zombie state where the
connection object stays alive but no longer receives messages, leaving no
way to recover automatically. The cancel func was also ineffective because
the SDK ignores the context after the initial dial, so stopping a channel
leaked the old connection.
Add a reusable RunSupervised helper that keeps a long connection alive and
proactively recycles it every 6h on top of the SDK's own reconnect, which
bounds the worst-case outage and clears stuck connections. Wire DingTalk to
it, add LongConnClient.Close (disables AutoReconnect before closing so the
SDK does not resurrect the connection), and make the cancel func actually
tear the connection down. Drop the now-unused stream client reference from
the adapter, which only sends replies via sessionWebhook or OpenAPI.
checkSQLInjectionRisks and InjectAndConditions compiled their regexps
with regexp.MustCompile on every invocation. Both run on the datasource
SQL path — InjectAndConditions is called once per security filter
(tenant, soft-delete, hidden-KB, search-scope), so a single secured query
recompiled ~20 patterns. InjectAndConditions also compiled the identical
tail-clause pattern twice.
Hoist them to package-level vars (compiled once at init). Behavior is
unchanged — same patterns, covered by the existing TestValidateSQL_*
and TestInjectAndConditions tests.
Benchmarks (apple M, -benchmem):
InjectAndConditions 4347 ns, 72 allocs -> 1023 ns, 7 allocs
checkSQLInjectionRisks 36074 ns, 560 allocs -> 12984 ns, 12 allocs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cascade soft-delete tenant_members when a workspace is removed so the
tenant switcher no longer shows ghost entries as "#<id>". Also filter
orphaned memberships in /auth/me and switch to another workspace after
delete when one remains.
Add new styles for anchored form popups in the wiki browser, improving the user interface for folder actions and confirmations. Update internationalization strings for move confirmation titles in English, Korean, Russian, and Chinese locales to enhance user experience across multiple languages. Refactor existing components to integrate the new popup styles and ensure consistent behavior.
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.
Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
Centralize the wiki category-path normalization (separator/quote cleanup,
type-label stripping, dedup, depth cap) into a single types helper shared by
the service, repository, and taxonomy layers so stored and queried paths are
always cleaned identically and cannot drift.
Flatten category_path into fixed category_l1/l2/l3 columns (kept in sync on
write) and rewrite ListDistinctCategoryPathsByType to group, count, and
paginate directly in cross-dialect SQL, returning the page of child folders
plus the total distinct count. This removes the prior full-column scan and
in-memory pagination (whose maxPaths cap was effectively dead) and makes Total
and later pages correct.
Add the level columns + composite index to the wiki migrations and the repo
test schema, and cover the SQL grouping/counting/pagination with a new test.
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
ParseLLMJsonResponse recompiled its ```json fence-extraction regexp on
every call that fell through the direct-unmarshal fast path — i.e. every
fenced LLM response on the graph-extraction path. Hoist it to a
package-level var.
Adds a test (none existed) covering direct, fenced, no-lang and
prose-wrapped inputs, plus a benchmark.
Benchmark, fenced payload (-benchmem):
before: 3205 ns/op, 5712 B/op, 54 allocs/op
after: 1315 ns/op, 693 B/op, 15 allocs/op
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hasCommandSubstitution recompiled its three regexps with regexp.MustCompile
on every call, and ValidateArgs calls it once per argument — so validating
an N-arg command compiled 3N patterns. Hoist them to a package-level slice
compiled once at init.
Behavior is unchanged (same patterns); covered by the existing
TestScriptValidator_ValidateArgs / ValidateAll tests.
Benchmark, ValidateArgs with 8 args (-benchmem):
before: 16374 ns/op, 40142 B/op, 505 allocs/op
after: 2092 ns/op, 32 B/op, 1 allocs/op
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WikiChunkCitationPrompt placed the per-batch <chunks> block before the
static rules, output schema and the per-document-stable <candidate_slugs>.
Within one document only ChunksXML changes between batches, so the stable
content trailing the chunks could not form a long common prefix and
providers re-billed the static rules on every citation batch (issue #1687
measured ~6.4% prefix-cache hit rate on wiki synthesis).
Reorder to [rules | output schema | candidate_slugs | chunks] so every
batch after the first shares the long [rules | candidate_slugs] prefix.
No instruction text was dropped; only block order and the inline
above/below directional references changed. Verified the cacheable prefix
grows from 281 to 2366 bytes and is byte-identical across batches.
Addresses #1687
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the dominant heading level repeats and becomes the split backbone,
a long section is sub-split via SplitText and every sub-chunk inherited
the section-level breadcrumb. Deep `###`/`####` headings that define a
sub-chunk's meaning were dropped from Chunk.ContextHeader, so queries
about content nested far below a shallow heading failed to retrieve it.
Track the heading hierarchy across the section and assign each sub-chunk
the breadcrumb in effect at its start offset, so the deepest active
heading reaches the breadcrumb. Adds a regression test.
Fixes#1674
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wiki summary/entity/concept/page-modify prompts instructed the LLM to
reproduce  verbatim, so high-entropy image URLs passed
through generation and got corrupted (dropped UUID segments, duplicated
or mangled path prefixes, even CJK chars inserted into the bucket name),
producing object keys that don't exist -> 404.
Mask all image URLs to opaque placeholders at the generateWithTemplate
chokepoint and deterministically rehydrate after generation; drop
corrupted/hallucinated placeholders so broken links never reach storage.
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
Three related improvements to the IM-session experience:
- Auto-generated titles: a new IM session starts untitled and gets a
content-based title from its first message via the same GenerateTitleAsync
flow web sessions use, instead of an identity title like "user · dm". Falls
back to the identity title when the first message has no text. All platforms.
- In-flight reply recovery (web chat): opening a session whose reply is still
being generated on an IM channel made the web client attach via
continue-stream and fail (the IM answer never streams through this server),
flashing a "stream failed" toast and forcing a manual refresh. Treat a failed
attach to an IM reply (channel === 'im') as recoverable — poll until it
completes, then reload the thread; on timeout surface the standard error.
Web/api replies keep the original behaviour, untouched.
- Source-filter classification: the sidebar filter treats "web" as sessions
with no IM mapping, but the list join excluded soft-deleted mappings, so
/clear'd IM sessions leaked into the user's own web chats. Include soft-deleted
mappings so a session ever bound to an IM channel stays under its platform.
Adds a repository regression test.
- Removed dependency on utils for schema generation, replacing it with a hardcoded JSON schema for GetDocumentInfoInput.
- Updated the input struct to use omitempty for JSON fields, clarifying that either knowledge_ids or faq_ids may be provided, but both are optional.
- Enhanced documentation for GetDocumentInfoInput to specify the usage of knowledge_ids and faq_ids.
- Enhanced the `reparseKnowledge` function to accept optional process configuration overrides, allowing users to adjust parsing settings during reparse operations.
- Updated the frontend components to integrate reparse capabilities, including the `UploadConfirmDialog` and `KnowledgeBase` views, providing a user-friendly interface for confirming reparse actions.
- Added new localization strings for reparse confirmation and process configuration details in multiple languages.
- Modified the backend API to handle reparse requests with optional configuration, ensuring proper validation and persistence of overrides.
- Introduced new types and interfaces to manage reparse-specific data structures in the store, improving type safety and clarity in the codebase.
- Removed the `enable_multimodal` field from the `ChunkingConfigOverride` interface and related components to streamline configuration.
- Updated the logic in `UploadConfirmDialog.vue` to default `enableParentChild` to false when initializing from knowledge base info.
- Modified the `mergeChunkingConfig` function to ensure that `EnableParentChild` is authoritative, allowing explicit control over its state.
- Added a new test case in `knowledge_process_config_test.go` to verify the behavior of parent-child toggles in process configuration resolution.
- Updated the logic for `graph_enabled` in `UploadConfirmDialog.vue` to require both `nodeExtractConfig.enabled` and `graphEnabled` to be true.
- Added a new test case in `knowledge_process_config_test.go` to ensure that `GraphEnabled` is false when `ExtractConfig.Enabled` is false.
- Adjusted the `ResolveProcessConfig` function to reflect the updated dependency between graph and extract configurations, ensuring proper behavior in the process configuration resolution.
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides.
- Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations.
- Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads.
- Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process.
- Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic.
- Enhanced user experience by providing detailed configuration options during the upload process.
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
- Updated the sanitization logic in `newStreamPacketDumper` to exclude the '.' character, enhancing security against path traversal vulnerabilities.
- Added checks to ensure the cleaned file path remains within the specified directory, preventing unauthorized access to the filesystem.
- Improved overall robustness of file handling by enforcing stricter path validation rules.
The global Auth middleware allowlist (noAuthAPI) was missing the
share-link routes, so an unauthenticated invitee opening /register?token=
hit a 401 on /auth/invitations/lookup and the frontend bounced them to
/login instead of showing the register page. Add both share-link
endpoints to the allowlist and stop the request interceptor from treating
their 401s as a session-expiry redirect.
Refs: https://github.com/Tencent/WeKnora/issues/1617
Moving knowledge with mode=reuse_vectors copies vector index entries
directly between knowledge bases, which only works when both KBs are
bound to the same vector store (CopyIndices is routed through the source
store). Nothing rejected a cross-store reuse_vectors move, so it would
copy target-KB rows into the source store and then delete the source
indices — corrupting vector data. The move endpoint validated only the
embedding model, and ListMoveTargets surfaces same-model KBs regardless
of their store binding, so a cross-store target is reachable.
Reject reuse_vectors moves whose source and target KBs are bound to
different vector stores:
- MoveKnowledge handler: synchronous 400 before enqueueing, with a
message pointing the caller at reparse mode.
- moveOneKnowledge: re-check before marking the knowledge as processing,
so a rejected move leaves it untouched rather than stranded.
- moveKnowledgeReuseVectors: a top-of-function guard as defense in depth
for any directly enqueued task, satisfying the pre-existing contract
noted in the comment ("callers must reject cross-store moves").
reparse mode is unaffected: it re-indexes into the target store from
scratch and is the supported way to move across stores.
wizardchen