The SearXNG provider forwarded `parameters.api_key` as
`Authorization: Bearer <key>` for reverse-proxy auth, but the frontend
never rendered the API key input for self-hosted providers
(`requires_api_key=false`), so the path was unreachable from the UI.
Rather than expose another UI knob for a niche reverse-proxy setup that
the project does not currently support, remove the Authorization header
branch and the `apiKey` field on `SearxngProvider`. Tenants who put
SearXNG behind an authenticating reverse proxy can still front it with
network-level auth (mTLS, IP allowlist) without changes here.
`WebSearchProviderParameters.APIKey` is generic and used by other
providers, so it stays.
- Updated .env.example to clarify SEARXNG_SECRET generation and added SSRF_WHITELIST_EXTRA for improved security.
- Modified docker-compose files to bind SearXNG to localhost by default and introduced a one-time initialization service to set up settings.yml correctly.
- Enhanced SearxngProvider with stricter URL validation, ensuring no query or fragment is present in the base URL.
- Added unit tests for SearXNG validation and date parsing to ensure robustness.
- Updated frontend WebSearchSettings to reflect changes in SearXNG instance URL handling.
This commit improves the security and usability of the SearXNG integration, addressing potential misconfigurations and enhancing the developer experience.
The heuristic splitter relied on two regexes that were too strict for
real-world Chinese technical documents:
- ChineseChapterPattern required 第 / numeral / unit to be adjacent, so
the very common "第 1 章 引言" form never matched.
- NumberedSectionPattern required a trailing dot after the numeral, so
multi-level numbering such as "1.1 文档目的" or "2.2.1 用户与权限"
was missed.
As a result, documents like the CHAPTER_SAMPLE shipped with the chunking
debug drawer collected zero heuristic markers, and ProfileDocument fell
all the way through to the character-level Legacy tier, producing chunks
that ignored the document's explicit chapter structure.
- Loosen ChineseChapterPattern to tolerate spaces around 第 / 数字 / 单位.
- Allow the multi-level branch of NumberedSectionPattern to drop its
trailing dot; keep the single-level / roman branch dot-required to
avoid false positives on version strings.
- Extend patterns_test.go with positive and negative cases covering the
CHAPTER_SAMPLE wording plus deep-nesting and lone-numeral regressions.
Children were forced to StrategyRecursive on the assumption that
re-profiling each parent would be too expensive. In practice profiling
each parent is bounded by O(sum(parent_size)) ≈ O(N) total, the same
order as the original parent profiling pass — and the gain is real:
when a parent (e.g. an H1 chapter) contains its own sub-headings, the
heading splitter on child input now picks them up and generates a
finer-grained breadcrumb instead of every child sharing the parent's
top-level breadcrumb.
Add mergeBreadcrumbs to combine the parent ContextHeader with the
child's freshly-derived one, dropping the duplicated seam line that
appears when the child's first heading equals the parent's last.
Tests cover (a) sub-headings now appearing in child breadcrumbs and
(b) the merge dedup against duplicated seam lines.
Heuristic and recursive splitters preserve original byte positions in
Chunk.Content (the End-Start == RuneCountInString invariant is required
by document-reconstruction code paths), which means a chunk sliced at a
boundary often carries leading/trailing newlines from the boundary
itself. Feeding that whitespace into the embedding model dilutes the
vector and wastes tokens for no benefit.
TrimSpace the body inside EmbeddingContent for both chunker.Chunk and
the two types.Chunk / types.ParsedChunk mirrors. Inner whitespace is
preserved; positions/Content are unchanged.
Also clean up appendChunk: the previous code computed a trimmed string
for the empty-check then stored a separate untrimmed copy, which read
as if it might be intentionally divergent. Replace with an explicit
'raw text, skip if pure whitespace' shape and document the invariant.
Heuristic boundary detection (numbered sections, all-caps headings,
\\n{3,} blank blocks, etc.) ran on the raw text and could land inside
atomic regions handled by protectedPatterns — most notably LaTeX
$$...$$ blocks, Markdown tables, fenced code, and image/link refs.
A boundary inside such a region would cause the bin-packer to slice
through protected content, defeating the protection.
Convert protectedSpans output to rune offsets once and filter the
boundary list before bin-packing. Boundaries on a span edge are kept
(they align with the span) — only strictly-interior ones are dropped.
Strategy.Split / SplitWithDiagnostics already run ProfileDocument once
when the auto strategy resolves the chain. splitByHeadingsImpl was
rerunning the same O(N) pass on entry, so every auto-mode call paid
2x scan cost; SplitParentChild paid 2x per parent.
Pass the profile down through runTier and let splitters compute their
own only when called outside the auto path (where profile == nil).
applyOverlapAligned searched for the latest boundary in
[curEnd-2*overlap, curEnd], but curEnd itself is always one of the
boundaries (the bin-packer only flushes at boundary positions). The
loop therefore always returned curEnd, producing chunkStart == curEnd
and zero overlap regardless of cfg.ChunkOverlap.
Exclude curEnd from the search window so an earlier boundary can be
picked, restoring the intended overlap behaviour.
TierRecursive and TierLegacy both invoked SplitText with identical
output, but only TierLegacy got the "always-return-on-validation-failure"
safety-net behavior. Auto chains thus ran SplitText twice on the
fallback path and produced two-line debug rejection traces with the same
reason.
Inline TierRecursive into TierLegacy: SelectStrategy and the explicit
StrategyRecursive entry point now emit single-legacy chains. The
StrategyRecursive public constant stays so existing ChunkingConfig rows
keep parsing — it's just an alias for legacy now.
No user-visible behavior change; one fewer SplitText call per failed
auto fallback.
Documents with many short headings (FAQ-style, quick refs) used to fail
the heading-tier validator with "too many tiny chunks" and silently fall
back to legacy splitting, defeating the purpose of heading-aware mode.
Merge physically adjacent sections whose combined size still fits within
ChunkSize, deriving a shared breadcrumb via commonHeadingPrefix so the
merged chunk's ContextHeader stays meaningful.
Tests cover the merge path, position-invariant preservation after merge,
ChunkSize ceiling, and the breadcrumb prefix helper. Existing tests that
relied on tiny fixtures were grown so each section stays distinct.
Three documentation passes around the adaptive chunking work:
UI
- Frontend ChunkOverlap default consolidated to 80 (was 100), matching
chunker.DefaultChunkOverlap on the backend. Both DEFAULT_CHUNKING_PRESET
and initFormData updated. The KB-load fallback also uses 80 when a
loaded KB has no chunk_overlap stored.
- All four locales (en-US, zh-CN, ko-KR, ru-RU) get rewritten chunking
setting descriptions: each now states the validated range, the default,
and the situations where you'd deviate (FAQ vs narrative, embedder
token limits, language-specific corpora).
Source code
- splitter.go: DefaultChunkSize / DefaultChunkOverlap constants get a
longer block-comment explaining the per-language token math and the
use-case sweet spots, plus the migration note on what the old
inconsistent defaults were.
- KBChunkingSettings.vue: new comment block above ChunkingConfig
documents the slider min/max for each setting, why those bounds
exist, and the recommended TokenLimit values per embedding model.
Repo docs
- New docs/CHUNKING.md: end-to-end guide covering why chunking matters,
the adaptive 3-tier architecture, per-setting reference with ranges
and sweet spots, parent-child explanation, the token-limit table per
embedder (OpenAI / Voyage / Cohere / BGE / MiniLM / Jina), 7 use-case
presets, the debug panel workflow, the API surface, and known
trade-offs (recursive strategy hidden from UI, no auto-reindex on
strategy switch, OCR limitations).
- CHANGELOG.md gets a new [Unreleased] section consolidating all the
adaptive-chunking work shipped on this branch: 5 features, 8
improvements, 6 fixes, 1 docs entry. The entry references
docs/CHUNKING.md for deeper explanation.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.
Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
not accept a context.Context, so when previewTimeout fires the
worker keeps running. Bounding input size keeps worst-case CPU
per request well under a second on commodity hardware. Sized so
10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
follow-up: real cancellation needs the splitter to take a ctx.
Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
reuse a single rune-count per chunk for stats + size + token
estimation. Eliminates the previous triple []rune allocation per
chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
instead of []chunker.Chunk; sumSq computed in float64 to avoid
the int*int overflow at l > ~46k.
Correctness / UX
- Preview panel sends strategy / token_limit / languages
unconditionally, mirroring the buildSubmitData convention so the
preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
gin's cryptic 'Field validation failed on the required tag'.
Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
made the original name a misnomer; the test still asserts the
right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
cases (empty / single / varying / no-variance underflow) plus
PreviewChunking httptest scenarios (auto path, legacy strategy,
empty-text rejection, oversize rejection, chunk truncation with
full-set stats).
Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
obsolete since the heading and heuristic splitters shipped.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by the post-Phase-3 self-review.
High severity
- Strategy / TokenLimit / Languages can now be reset. The KB-config
handler uses pointer DTOs so the absence of a field in the payload
means "no change" while an explicit empty string / 0 / [] clears
the value. The frontend now sends the fields unconditionally so
the user can revert to defaults via the clear icon — previously
these fields were write-once.
- DocProfile is no longer computed twice in the preview endpoint.
Diagnostics carries the profile that drove tier selection (when
auto-strategy ran); the handler only re-profiles for explicit
strategies that bypass the profiler.
- Chunk-size stats (avg / min / max / stddev) are now computed over
the FULL chunk set before the response is truncated to
previewMaxChunks. Avoids systematically misleading metrics for
large documents.
- SplitWithDiagnostics defaults SelectedTier to TierLegacy so an
empty-text request never leaves the diag with a blank tier string.
Medium severity
- Split() reverts to its own loop instead of delegating to
SplitWithDiagnostics. Diagnostics struct allocation is reserved
for callers that actually want the trace (preview endpoint).
Matters in SplitParentChild where Split runs once per parent.
- Preview endpoint runs the splitter on a goroutine wrapped in a
5s context.WithTimeout. CJK input at the 256k-rune ceiling
could otherwise tie up the worker for several seconds.
Low severity
- KBChunkingSettings emits arrays via spread copies so parent-side
mutation cannot drift the form's reactive state.
- KBChunkingDebug font-family includes a monospace fallback so the
chunk content stays monospaced even when the TDesign CSS variable
is missing.
Documentation
- DocProfile JSON shape is now annotated as part of the public
preview API; field renames would be breaking.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds POST /api/v1/chunker/preview — a read-only endpoint that runs the
adaptive chunker on supplied text and returns the chunks plus
diagnostic info (which tier won, which were rejected and why, the
DocProfile that the selector saw). Used by the upcoming KB editor
debug panel so users can experiment with chunking parameters without
re-uploading documents.
Backend pieces:
- chunker.SplitWithDiagnostics(text, cfg) returns chunks plus a
Diagnostics struct (SelectedTier, TierChain, Rejected). Split() now
delegates here.
- chunker.DocProfile gets JSON tags so it serializes cleanly.
- handler.PreviewChunking caps input at 256k chars and chunks at 500
per response, computes a size-distribution summary, and never
touches DB or embedding APIs.
Auth: lives under the v1 group so the standard JWT middleware applies.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.
Critical
- Heading splitter no longer prepends breadcrumb to Content. The
breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
combines header + content for the embedder. This restores the
End-Start == len(Content) invariant relied on by knowledge.go's
document-reconstruction path (summary generation, UI highlight) and
also eliminates the duplicate-heading regression where the section
heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
leaves a piece > ChunkSize, the next-priority separator is applied
inside that piece. Mirrors the Python reference. Without this, a
document with one paragraph break followed by a long run of
newline-separated lines could emit a single ~1950-rune chunk for
ChunkSize=300.
Medium
- SplitParentChild forces children onto the recursive tier, skipping
the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
whole chain rejects rather than running SplitText a second time.
Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
configs (TokenLimit clamping ChunkSize tiny while Overlap stays
high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
KBs that stored ChunkOverlap=0.
ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
knowledge service can build embedding inputs as
titlePrefix + chunk.EmbeddingContent() without a DB schema change.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
splitByHeuristicsImpl handles documents that lack proper Markdown
headings but contain other structural cues — form-feeds, numbered
sections, multilingual chapter markers (DE/EN/ZH), all-caps section
titles, visual separators (---/===), repeated page footers, and runs
of blank lines. Boundaries are sorted by position; greedy bin-packing
fills chunks up to cfg.ChunkSize and oversize blocks between two
boundaries fall back to the legacy splitter for inner segmentation.
Boundary detection skips fenced code blocks and de-duplicates
overlapping offsets keeping the highest-priority candidate.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
splitByHeadingsImpl detects the dominant Markdown heading level via the
profiler and splits the document at headings of that level (or shallower).
Each resulting chunk gets a breadcrumb prefix like
'# Chapter 1\n## Section 1.2' so downstream embedding sees the section
context, not just the body. Sections larger than ChunkSize fall back to
the legacy splitter for inner segmentation, with the breadcrumb prepended
to every sub-chunk and ChunkSize reduced by the breadcrumb length so the
final chunk still fits the budget.
- DominantHeadingLevel: prefer the lowest level with ≥3 hits, else the
deepest level with ≥1 hit (so small docs with H1+H2 still split at H2)
- HeadingHierarchy state is updated for sub-headings within a section so
later primary sections see the right active stack
- Code-block contents are skipped during heading detection so a `# foo`
inside ``` doesn't create a fake boundary
- strategy.go uses context.Background() for the fallback log entry
(GetLogger panics on nil context)
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds HeadingHierarchy, a level-stack that tracks Markdown heading
nesting (#, ##, ###, ...). When a level-N heading is observed, all
deeper entries are popped — so Breadcrumb() always reflects the active
context path. Used by the upcoming heading-aware splitter to prepend a
context breadcrumb (e.g. '# Chapter 1\n## Section 1.2') to every chunk
inside a section, lifting embedding quality for non-trivial documents.
The existing header_tracker (table headers) uses a binary start/end hook
model that does not fit headings — a heading does not have an explicit
end, it is replaced by the next heading of equal or shallower depth.
A separate level-stack is the right shape.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).
Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
derived from TokenLimit so chunks stay safe under embedding model
token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
English (~4 chars/tok).
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Adds the building blocks for tiered chunking strategies — multilingual
patterns, token approximation, document profiling, validation, and a
strategy resolver that today still falls through to the legacy
SplitText. Default Strategy is empty/legacy so existing knowledge bases
continue to chunk identically.
- tokens.go: char/token approximation per language (en/de/zh/mixed)
and a coarse DetectLanguage heuristic
- patterns.go: source-of-truth regex bibliothek for headings, numbered
sections, multilingual chapter markers, visual separators, etc.
- profiler.go: single-pass document scan + SelectStrategy chain
- validator.go: rejects obviously broken chunk-sets to drive fallbacks
- strategy.go: public Split / SplitParentChild entry points; resolves
StrategyAuto via the profiler, anything else maps to a fixed tier
- splitter.go: SplitterConfig gains Strategy / TokenLimit / Languages
fields and DefaultChunkSize / DefaultChunkOverlap constants (overlap
raised to 80 ≈ 15%)
- knowledge.go: switches both call sites to chunker.Split /
SplitParentChild — behavior unchanged for empty Strategy
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
When the uploaded file is itself an image, the image reference now carries
an IsOriginal flag so ResolveAndStore skips the small-icon size filter.
Otherwise small standalone images (e.g. avatars below 64x64) were silently
dropped before reaching multimodal OCR/caption processing.
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.
These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
- Enhanced the logging of the tenant access token in the Feishu client to handle variable lengths for prefix and suffix, ensuring accurate display even for shorter tokens.
- Updated the test cases in `connector_test.go` to reflect changes in the endpoint path for file downloads, improving test reliability.
- Adjusted example outputs in `example_test.go` for clarity and consistency.
- Modified image resolution tests to correct expected outcomes based on updated criteria for icon images.
- Added environment variable settings in image resolver tests to ensure localhost accessibility during testing.
These changes enhance the robustness of logging and testing mechanisms across the application.
- Updated the SplitTextParentChild function to maintain a newParents slice, ensuring accurate parent indexing when splitting text into child chunks.
- This change enhances the integrity of parent-child relationships by correctly associating child chunks with their respective parents, improving overall chunk processing.
- Replaced references to Docreader credentials with WeKnoraCloud credentials across multiple services and handlers.
- Introduced a new CredentialsConfig structure to manage third-party provider credentials, specifically for WeKnoraCloud.
- Updated database schema to include a credentials column for storing WeKnoraCloud AppID and AppSecret.
- Enhanced methods for retrieving and utilizing WeKnoraCloud credentials, ensuring proper encryption and decryption during storage and retrieval.
This update improves the management of WeKnoraCloud credentials, streamlining the integration and enhancing security measures.
- Removed support for video file uploads across the application, including updates to the knowledge base and document processing services.
- Updated internationalization files to reflect the removal of video-related terminology, changing labels and descriptions to focus solely on audio processing.
- Adjusted file type handling to exclude video formats, ensuring users receive accurate feedback regarding supported file types.
- Enhanced error handling to provide clear messages when users attempt to upload unsupported video files.
This update streamlines the application by focusing on audio processing capabilities and improving user experience through clearer messaging.
- Changed 'Audio Processing' to 'Audio & Video' in English, Korean, Russian, and Chinese translations to reflect expanded functionality.
- Updated ASR configuration titles and descriptions to include video processing capabilities, enhancing user understanding of the feature.
- Renamed file type labels from 'Audio Files' to 'Audio & Video' across multiple languages for consistency and clarity.
- Enhanced document upload handling in the KnowledgeBase to filter unsupported file types, providing user feedback for filtered images, videos, and audio files.
This update improves the internationalization of the application and clarifies the capabilities of the ASR feature.
- Introduced a new WeKnoraCloudSettings component for managing APPID and APPSECRET.
- Added functionality to save WeKnoraCloud credentials without automatically creating models.
- Implemented credential status checks across various components to provide user feedback on configuration status.
- Updated API endpoints to handle credential saving and status retrieval.
- Enhanced UI elements to guide users in configuring WeKnoraCloud settings.
This update improves the integration of WeKnoraCloud by allowing users to manage their credentials more effectively and receive real-time feedback on their configuration status.
- Remove infrastructure/crypto package (CryptoService, config, PBKDF2-based encryption)
- Replace with lightweight utils.GetAESKey/EncryptAESGCM/DecryptAESGCM helpers
- Remove cryptoSvc dependency from modelService, tenantService, weKnoraCloudService
- Remove crypto state persistence logic from container initialization
- Add TruncatePromptTokens field to WeKnoraCloud embed request
- Update frontend hint text for WeKnoraCloud credential errors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added ASR configuration options to the knowledge base and model settings.
- Implemented ASR model selection and transcription capabilities in the knowledge processing pipeline.
- Enhanced file upload support to include audio formats for ASR processing.
- Updated localization files to include ASR-related labels and descriptions.
- Refactored model handling to accommodate ASR as a new model type.
- Added headerTracker to preserve context of Markdown table headers across split chunks.
- Implemented logic to prepend headers to subsequent chunks after the first.
- Enhanced merging logic to ensure headers are not duplicated in overlapping content.
- Added comprehensive tests for header tracking and chunking behavior with Markdown tables.
- Keep strict mode blocking all direct IPs (IPv4 and IPv6 uniformly)
- Unify all SSRF call sites to use ValidateURLForSSRF (whitelist-aware)
- Add Teredo (2001:0000::/32) and 6to4 (2002::/16) tunnel detection
- Make redirect handler and DNS pinning respect SSRF_WHITELIST
- Unexport isSSRFSafeURL to prevent future callers bypassing whitelist
- Add scheme validation for whitelisted redirect targets
- Document IPv6 whitelist syntax in .env.example
- Add comprehensive IPv6 test coverage
- Updated relevant files to include provider registration, implementation, and metadata.
- Enhanced frontend components to support provider management and configuration.
- Added localization for new provider settings and messages.
- Implemented backend repository methods for CRUD operations on web search providers.
- Introduced a new task enqueuer in the ImageMultimodalService to handle question generation for image knowledge.
- Implemented logic to enqueue question generation tasks when valid captions or OCR content are available, enhancing the knowledge base's interactivity.
- Updated image reference handling to ensure markdown URLs are correctly formatted, accommodating filenames with spaces.
- Enhanced regex pattern in image resolver to support filenames with spaces, improving image processing accuracy.
- Updated regex patterns in MarkdownImageUtil to support alt text containing brackets and handle MIME types with hyphens.
- Implemented new functions in ImageResolver for resolving HTML <img> tags with data URIs and bare base64 content, improving image handling in markdown.
- Added comprehensive tests for various image scenarios, ensuring robust handling of data URIs and base64 images.
- Modified the sortedKeys function to sort keys numerically when all keys are numeric strings, ensuring correct order for keys like "2" and "10".
- Updated the NeedsKBRetrieval method to include IntentSummarize as a valid case for knowledge base retrieval.
- Added a new test data file for chat import to support testing of document parsing and processing functionalities.
- Introduced support for JSON file types in English, Korean, Russian, and Chinese locale files.
- Updated the file type acceptance criteria in the document upload component to include JSON.
- Enhanced the knowledge base parser settings to recognize JSON as a valid file type.
- Implemented JSON conversion functionality in the document parser, allowing for JSON content to be processed and converted to markdown.
- Added comprehensive tests for JSON to markdown conversion to ensure functionality and reliability.
- Added functionality to resolve embedded data:image;base64 images in markdown, storing them and replacing references with stable URLs.
- Updated the knowledge service to process data URIs before handling remote images, ensuring all images are resolved consistently.
- Improved logging for resolved images, providing better insights during manual processing.
- Added unit tests to validate the new data URI resolution functionality and ensure reliability.
Prevents 400 Bad Request errors when uploading documents with large
tables or code blocks by adding hard limits in buildUnitsWithProtection
and mergeUnits functions.
wizardchen