Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.
Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.
Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.
Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.
Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.
- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
renderIndexOverviewForAgent output, isTransientLLMError classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wikis with tens of thousands of pages used to crash the browser trying
to render the entire link graph at once (30MB+ JSON, 100k+ SVG elements).
This change moves the graph viewer from "fetch everything, render
everything" to "fetch a slice, expand on demand" with several matching
UX improvements.
Backend
* GET /wiki/graph now accepts `mode` (overview | ego), `center`, `depth`,
`types`, `limit` query params. Default is overview top-500 by
link_count, capped at 2000. Response includes a Meta object so the
frontend can render a truncation hint and drive the UI.
* Pure helper `computeGraphSubset` extracted for testability; six unit
tests cover overview truncation / type filter / ego BFS / missing
center error.
* WikiLintService passes Limit=0 (uncapped) so link integrity checks
still walk every page.
* Repo `Search` adds CASE-based relevance ranking (title 4, slug 3,
summary 2, content 1) so full-text results put the most obvious
matches first instead of whatever was updated most recently.
Frontend
* Graph viewer fetches overview on entry; ego pivot on double-click,
search, wiki-link click, URL ?slug=, global issues jump.
* Shift+click or hover ⊕ button blooms neighbors onto the current
canvas additively. Bloom tracks generations and LRU-evicts oldest
when total exceeds 1500 nodes; ego center / selected node / latest
anchor are always protected.
* "Grow frontier (N)" legend action expands every dashed-ring node in
one click (6-way concurrency) while skipping Index/Log super-nodes
that would otherwise dump the whole wiki onto the canvas.
* Node dashed expansion ring + drawer "X/Y neighbors shown" hint tell
users which nodes still have neighbors to load.
* Search dropdown uses remote full-text search (debounced, sequence-
numbered to drop stale responses) and falls back to the overview
top-500 snapshot when the input is empty.
* Type filters now round-trip to the server so top-N is always
computed from the user's active type set rather than hiding nodes
client-side and shrinking the view.
* Status card replaces the cramped "centered on X · N hops · M nodes"
line with a structured focus/overview summary, resolving slugs to
page titles.
* Help popover in the legend lists every canvas shortcut.
i18n entries added for zh-CN / en-US / ko-KR / ru-RU.
Heuristic and recursive splitters preserve original byte positions in
Chunk.Content (the End-Start == RuneCountInString invariant is required
by document-reconstruction code paths), which means a chunk sliced at a
boundary often carries leading/trailing newlines from the boundary
itself. Feeding that whitespace into the embedding model dilutes the
vector and wastes tokens for no benefit.
TrimSpace the body inside EmbeddingContent for both chunker.Chunk and
the two types.Chunk / types.ParsedChunk mirrors. Inner whitespace is
preserved; positions/Content are unchanged.
Also clean up appendChunk: the previous code computed a trimmed string
for the empty-check then stored a separate untrimmed copy, which read
as if it might be intentionally divergent. Replace with an explicit
'raw text, skip if pure whitespace' shape and document the invariant.
Locks in two invariants exposed during the integration audit:
- ContextHeader on types.Chunk must never leak into the JSON
representation served to API clients (json:"-" + gorm:"-").
- types.ParsedChunk.EmbeddingContent prepends ContextHeader with a
blank line separator and falls back to plain Content when the
header is unset.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.
Critical
- Heading splitter no longer prepends breadcrumb to Content. The
breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
combines header + content for the embedder. This restores the
End-Start == len(Content) invariant relied on by knowledge.go's
document-reconstruction path (summary generation, UI highlight) and
also eliminates the duplicate-heading regression where the section
heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
leaves a piece > ChunkSize, the next-priority separator is applied
inside that piece. Mirrors the Python reference. Without this, a
document with one paragraph break followed by a long run of
newline-separated lines could emit a single ~1950-rune chunk for
ChunkSize=300.
Medium
- SplitParentChild forces children onto the recursive tier, skipping
the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
whole chain rejects rather than running SplitText a second time.
Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
configs (TokenLimit clamping ChunkSize tiny while Overlap stays
high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
KBs that stored ChunkOverlap=0.
ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
knowledge service can build embedding inputs as
titlePrefix + chunk.EmbeddingContent() without a DB schema change.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
Reciprocal Rank Fusion previously hard-coded k=60 and weights 0.7/0.3.
Both are now sourced from RetrievalConfig (RRFK, RRFVectorWeight,
RRFKeywordWeight) with the same numbers as defaults via three new
GetEffective accessors. fuseOrDeduplicate / fuseWithRRF accept the
config; HybridSearch reads it from tenantInfo and forwards.
Lets corpus owners tune fusion behavior — large corpora benefit from
larger k, keyword-heavy domains from a higher keyword weight, etc. —
without changing application code.
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
The Go splitter previously combined all separators into one regex, losing
the priority-based recursive behavior of the Python reference. This pass
walks the separator list in order: the first separator that yields >1
piece wins, mirroring docreader/splitter/splitter.py. Larger semantic
boundaries (\n\n) now win over smaller ones (\n).
Additional cross-cutting changes:
- ChunkingConfig gains Strategy / TokenLimit / Languages fields so the
per-KB config can pick a tier and pin embedding token budgets.
- buildSplitterConfig in the knowledge service uses the new
chunker.DefaultChunkSize / DefaultChunkOverlap constants (overlap
consolidated to 80 ≈ 15%) instead of the previous hardcoded 50.
- ensureDefaults clamps ChunkSize to a per-language character budget
derived from TokenLimit so chunks stay safe under embedding model
token caps. Chinese (~1.7 chars/tok) gets a tighter clamp than
English (~4 chars/tok).
https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.
- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
so fall-through to the tenant-merge path is intentional rather than an
unrecognised provider sliding past the switch. The legacy `cos_config` column
predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
handler/system.go StorageEngineStatusItem.Name comment now list all 7
providers (matched against types.StorageEngineConfig and ParseProviderScheme).
Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
StorageConfig data is only consumed for cos/minio and skipped for the
others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.
Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
fixing it requires API surface additions; can ship separately.
Closes#1117 (the buildStorageConfig switches; remaining items deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Kingsoft Cloud KS3 as a supported storage provider, including
backend service implementation, storage config UI, and provider
scheme recognition throughout the codebase.
- Apply the session-list additions to the SQLite init migration (Lite build)
and make QueryPaged dialect-aware: LOWER(..) LIKE on SQLite, ILIKE on
Postgres, drop NULLS LAST on SQLite. Escape LIKE wildcards in the keyword
via the existing escapeLikeKeyword helper.
- SetPinned now returns rowsAffected so the handler can respond 404 on
unknown / unauthorized session IDs instead of a misleading 200.
- GET /sessions always returns the enriched shape (pin state + IM origin
fields) so the frontend never needs a second roundtrip; the dual-path
legacy branch in the handler is gone.
- Register pin routes with the wildcard name that matches each verb's
existing radix tree (POST :session_id, DELETE :id) and accept either
param name in the handler; avoids gin's "wildcard conflicts" panic.
- Drop the redundant [platform] prefix from IM session titles now that
the list renders a platform icon alongside the title; add unit tests
for shortID / buildUserSessionTitle / buildThreadSessionTitle.
- Frontend: remove the submenu search input and its i18n/keyword wiring
(search lives elsewhere in the app); pin icon uses the TDesign `pin`
glyph and inherits color so the active session turns green; optimistic
pin moves the item to the top of the list so it shows up at the top
of the Pinned group; IM text badge replaced by the platform SVG icons
from assets/img/im, desaturated by default and full-color on hover
or when the session is active.
Sessions today are a flat list ordered by updated_at. Two gaps showed up in
practice:
- Users cannot find specific chats as the list grows beyond a screen.
- IM-created sessions (WeCom/Feishu/Slack/...) are indistinguishable:
every title was "IM-<platform>" or "IM-<platform>-<username>" and the
list API hid the underlying im_channel_sessions mapping, so admins had
no way to tell which Feishu group a session came from.
Backend
- Migration 000039 adds sessions.user_id (owner), is_pinned, pinned_at
plus a composite index for the list query. Existing rows keep user_id
NULL and stay visible at the tenant level for backward compatibility.
- CreateSession now writes the caller's user_id from auth context.
- GET /sessions accepts keyword / source / agent_id. When any filter is
set, the response switches to enriched items that LEFT JOIN
im_channel_sessions and expose im_platform / im_chat_id / im_thread_id
/ im_user_id / im_agent_id / im_channel_id. No filters => legacy shape,
existing clients unaffected.
- Ordering: is_pinned DESC, pinned_at DESC NULLS LAST, updated_at DESC.
- POST/DELETE /sessions/:id/pin for user-scoped pin/unpin.
- IM session titles: "[platform] <user|chat|thread>" with short ID
suffixes so group/DM/thread sessions are visually distinct without
needing a round-trip to fetch a display name from the IM adapter.
Frontend
- Search input debounced at 300ms drives the keyword filter.
- Pinned chats render in a dedicated group above the time-based groups,
with a pin icon and a pin/unpin entry in the per-chat dropdown.
- IM chats get a short [platform] badge in the list.
- Pin toggle is optimistic and guards against double-clicks.
- zh/en/ko/ru i18n keys added for the new strings.
GORM Scan / AfterFind hooks for every AES-encrypted column followed
the same lenient pattern:
if decrypted, err := utils.DecryptAESGCM(c.APIKey, key); err == nil {
c.APIKey = decrypted
}
When SYSTEM_AES_KEY was missing, rotated, or the wrong length the
decryption error was swallowed and the in-memory struct kept the raw
"enc:v1:..." ciphertext. The application then happily forwarded the
ciphertext upstream as the actual API key / password, surfacing as
401/403/SignatureDoesNotMatch from third-party vendors. Worse, a
ciphertext snippet of a customer credential was leaking into the
external provider's request logs.
Introduce utils.DecryptStoredSecret that:
- returns "" / legacy plaintext untouched (no false positives for
pre-encryption rows);
- returns ErrEncryptedDataMissingKey when the value carries the
enc:v1: prefix but no AES key is configured;
- propagates any GCM auth-tag failure from a rotated key.
Wire it into the five Scan / AfterFind sites that currently swallow
the error:
- Tenant.AfterFind (tenants.api_key)
- CredentialsConfig.Scan (tenants.we_knora_cloud.app_secret)
- ModelParameters.Scan (models.parameters.api_key, .app_secret)
- ConnectionConfig.Scan (vector_store_connections.password, .api_key)
- WebSearchProviderParameters.Scan (web_search_providers.api_key)
The error is wrapped with the originating column so the failure is
diagnosable in the logs. Operators must restore the previous
SYSTEM_AES_KEY (or rotate the affected secrets) instead of receiving
silent vendor 401s.
Add table-driven tests covering empty input, legacy plaintext,
round-trip, missing key, wrong-length key, and rotated key. The
strict path explicitly asserts that ciphertext does NOT leak in the
returned plaintext on any error path.
When the uploaded file is itself an image, the image reference now carries
an IsOriginal flag so ResolveAndStore skips the small-icon size filter.
Otherwise small standalone images (e.g. avatars below 64x64) were silently
dropped before reaching multimodal OCR/caption processing.
- Update Scan method to handle both legacy bare-array format (e.g. [{...}, {...}]) and current object-wrapped format (e.g. {"engines": [{...}, {...}]}).
- Improve error handling for unmarshalling failures, providing clearer feedback on format issues.
This change ensures backward compatibility while allowing for the new data structure.
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.
This change threads one trace end-to-end:
- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
worker can attach to an upstream trace without emitting a duplicate
trace-create; StartGeneration now auto-picks parentObservationId from
ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
payloads so trace_id / parent_obs_id / user_id / session_id serialise
into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
resumes the upstream trace or opens a standalone asynq.<type> trace
for scheduled jobs, and wraps the handler in a SPAN with task metadata
(id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
ResumeTrace no-trace-create guarantee; asynq_test.go covers
InjectTracing round-trip, middleware resume path, and standalone
trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
and documents the cross-process propagation model.
No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
Closes#620#497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).
Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
so logger.CloneContext can preserve it across handler / goroutine
boundaries (otherwise each LLM call auto-created an orphan trace,
fragmenting one request into many)
Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes
Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.
Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
reuses WeKnora's existing PostgreSQL (separate database via an idempotent
init container that fixes ICU collation drift) and Redis (separate DB
number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
to avoid Prisma P1013 when passwords contain @ / # / etc.
Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
Introduce per-model `custom_headers` config (similar to OpenAI Python SDK's
`extra_headers`) so users can inject gateway auth tokens, trace IDs, etc.
into every outbound API request for chat / embedding / rerank / VLLM / ASR
models. Reserved headers like Authorization / Content-Type are always
preserved to avoid breaking auth or signing flows.
Along the way, unify production and "test connection" paths onto a single
`ConfigFromModel(*types.Model, appID, appSecret)` constructor in each
model package. Both `service.modelService.GetXxxModel` and the four
`handler.initialization.Check*Model` / `TestEmbeddingModel` endpoints now
go through the same field mapping, so new parameters only need to be
added in one place.
Backend:
- Add `CustomHeaders map[string]string` to `types.ModelParameters` with
JSON/YAML `omitempty` tags for forward compatibility.
- New `internal/utils/extraheaders.go` providing `ApplyCustomHeaders`
(for hand-rolled HTTP paths) and `WrapHTTPClientWithHeaders` /
`CustomHeadersRoundTripper` (for SDK paths like go-openai). Reserved
headers (Authorization, api-key, Content-Type, Accept, Host,
Content-Length, User-Agent) are filtered out.
- Inject headers in every remote model path: chat (SDK + raw HTTP),
embedding (openai/aliyun/jina/volcengine/nvidia/azure_openai), rerank
(remote_api/aliyun/jina/nvidia/zhipu — via shared `customHeaderSetter`
interface), VLM and ASR (via http.Client wrapping).
- Add `ConfigFromModel` to each of chat/embedding/rerank/vlm/asr,
consolidating field mapping (ExtraConfig, CustomHeaders, InterfaceType
defaulting, WeKnoraCloud credentials, etc.) and cover it with
dedicated unit tests per package.
- Refactor `service/model.go`: replace ~100 lines of hand-written Config
literals with one-line `ConfigFromModel` calls; drop the now-unused
`stringMapToAnyMap` helper.
- Refactor `handler/initialization.go` test-connection endpoints: merge
four ad-hoc request structs into one `ModelTestRequest`, extract
`buildTestModel` and `resolveTenantWeKnoraCloudCreds` helpers, and
route all four endpoints through `ConfigFromModel` + `NewXxx` so the
test path is now behaviorally identical to production.
Frontend:
- Add `custom_headers` to the `ModelConfig` API type.
- `ModelEditorDialog.vue`: new Key-Value header editor (add/remove rows),
auto-converts map <-> array when loading/saving, and passes the
current header set to the `Test Connection` button so the preview
reflects exactly what production will send.
- `ModelSettings.vue`: serialize the header array back into a map for
the backend (drops empty rows).
- i18n: add custom-header labels / descriptions / placeholders to
zh-CN, en-US, ru-RU, ko-KR.
Tests:
- `internal/utils/extraheaders_test.go` covers reserved-header filtering
and round-tripper wrapping (including nil-client fallback).
- New `config_from_model_test.go` in chat / embedding / rerank / vlm /
asr verifies all fields (CustomHeaders, ExtraConfig, InterfaceType
defaulting, AppID/AppSecret) are propagated end-to-end.
- Introduced UpdateAutoLinkedContent method to persist changes from machine-only link decorators without incrementing the version number, ensuring user-facing revisions remain consistent.
- Updated existing methods to utilize the new auto-linked content update logic, enhancing the handling of cross-link injection and dead-link cleanup.
- Enhanced the KnowledgeBase and WikiBrowser components to improve user experience during wiki operations, including status tracking and indexing indicators.
- Added scheduling for wiki status probes to provide timely feedback on indexing processes after user-triggered actions.
- Enhanced the UpdatePage and UpdateMeta methods to differentiate between user-visible content changes and bookkeeping updates, ensuring the version number is only incremented for actual content modifications.
- Updated the WikiPage struct and associated interfaces to reflect the new versioning policy, improving clarity on when the version is modified.
- Improved documentation for methods to clarify their intended use and behavior regarding versioning and metadata updates.
- Introduced extraction granularity settings in the wiki configuration, allowing users to control the level of detail in entity and concept extraction (focused, standard, exhaustive).
- Updated the UI to reflect these new options, including tooltips for better user guidance.
- Refactored the knowledge base editor to support the new extraction granularity feature, ensuring a seamless user experience.
- Improved the backend logic for wiki ingest tasks to utilize the new granularity settings, enhancing the accuracy of extracted content.
- Added tests to validate the new extraction granularity functionality and ensure robust performance across different configurations.
- Updated the knowledge base service to ensure WikiConfig is created when wiki indexing is enabled, enhancing backward compatibility.
- Removed legacy syncing of WikiConfig.Enabled to IndexingStrategy.WikiEnabled, streamlining the logic for wiki enablement.
- Introduced a custom retry delay function for asynq tasks to handle wiki ingest lock conflicts more effectively, improving task management.
- Cleaned up the container file by removing unnecessary whitespace, contributing to better code readability.
- Added comments to clarify the purpose of the `builtinAgentIDsOrdered` list, specifically noting the exclusion of `BuiltinWikiFixerID` to prevent clutter in the user-facing agent list.
- This change enhances the understanding of the built-in agent configuration and maintains a cleaner user interface for agent selection.
- Added a new configuration file `agent_type_presets.yaml` to define various agent type presets, including RAG Q&A, Wiki Q&A, Hybrid (Wiki + RAG), and Data Analysis.
- Updated the `builtin_agents.yaml` to associate built-in agents with the new agent type presets, enhancing their configuration options.
- Enhanced the agent system prompt templates to reflect the new agent types and their functionalities.
- Implemented API endpoints to retrieve agent type presets, allowing for dynamic loading in the frontend.
- Updated the frontend to support agent type selection in the agent editor modal, improving user experience by auto-filling relevant configurations based on selected presets.
These changes significantly enhance the flexibility and usability of the agent system, allowing users to easily configure agents based on predefined types.
- Updated the SuggestedQuestion interface to include 'wiki' as a new source type, allowing for more diverse question origins.
- Implemented ListRecentForSuggestions method in the wikiPageRepository to retrieve recent wiki pages for fallback suggestions, improving the agent's ability to generate relevant questions when other sources are unavailable.
- Enhanced the customAgentService to utilize the new wiki page suggestions, ensuring a broader range of questions can be generated from wiki content.
These changes significantly improve the agent's suggestion capabilities by incorporating wiki content, enhancing user interaction and information retrieval.
- Enhanced the `cleanupWikiOnKnowledgeDelete` function to pass the full knowledge object, allowing for more accurate title and summary retrieval during wiki cleanup.
- Introduced a tombstone mechanism in Redis to mark recently deleted knowledge, enabling in-flight wiki ingest tasks to skip processing for deleted documents.
- Updated the `isKnowledgeGone` function to check for deleted knowledge using the tombstone, improving efficiency in handling concurrent delete and ingest operations.
- Added logic in the `wikiIngestService` to filter out updates for deleted knowledge, ensuring that no ghost references are created during the ingest process.
- Implemented lint checks for stale source references in the `WikiLintService`, allowing for automatic cleanup of pages referencing deleted knowledge.
These changes significantly enhance the robustness of the wiki system, ensuring consistency and accuracy in handling knowledge deletions and related wiki pages.
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings.
- The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities.
- Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system.
These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members.
- Added a new built-in agent, "Wiki Fixer," designed to repair and optimize Wiki pages based on linter issues, with multilingual support for enhanced accessibility.
- Implemented a corresponding system prompt detailing the agent's role, mission, and workflow for effective issue resolution.
- Introduced new API functions for listing and updating wiki issues, allowing for better management of content conflicts and errors.
- Enhanced the frontend with new UI elements to display pending issues and facilitate auto-fixing, improving user interaction with the Wiki content.
These changes significantly enhance the agent's capabilities for maintaining the accuracy and quality of Wiki pages, fostering a more reliable knowledge base.
- Added a new tool, `wiki_flag_issue`, enabling users to report factual errors, mixed entities, or outdated information on wiki pages.
- Updated the agent system prompt and tool definitions to include the new tool, enhancing the agent's capabilities for maintaining wiki accuracy.
- Implemented backend functionality for creating, listing, and updating the status of flagged issues, ensuring effective tracking and resolution.
These changes significantly improve the agent's ability to manage and report issues within the wiki, fostering a more reliable knowledge base.
- Introduced a new test file for JSON handling to ensure that unmarshalling correctly copies input data.
- Validated that the stored JSON remains intact after input mutation, and confirmed successful marshalling of the output.
- This test enhances the reliability of JSON operations within the types package.
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.
These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
- Introduced a new method to list all wiki pages without pagination, facilitating index rebuilding and cross-linking.
- Updated the wiki ingest and retract processes to include document summaries, improving clarity in log entries and index updates.
- Enhanced the cleanup process for deleted knowledge, ensuring accurate summary retention and index regeneration.
- Streamlined the handling of page updates and deletions, optimizing the overall wiki management workflow.
These changes improve the efficiency and reliability of wiki content management and indexing.
Implemented new features in the WikiBrowser component to allow users to toggle the visibility of arrows on graph edges. This includes:
1. **Localization Updates**: Added translations for "Show arrows" and "Hide arrows" in English, Korean, Russian, and Chinese.
2. **Graph Rendering Enhancements**: Introduced logic to dynamically update SVG line attributes based on the toggle state, improving visual clarity in graph representations.
3. **State Management**: Added reactive state management for arrow visibility, ensuring seamless user interaction.
These changes enhance the user experience by providing more control over graph visualizations.
Added functionality to enable updates for the `wiki_config` field in the `KnowledgeBaseConfig` struct. This includes:
1. **Type Definition Update**: Added `WikiConfig` field to `KnowledgeBaseConfig` to allow JSON unmarshaling during update requests.
2. **Service Layer Enhancement**: Updated the `UpdateKnowledgeBase` method to persist changes to `wiki_config` when provided in the request.
3. **Frontend TypeScript Adjustments**: Enhanced type definitions in the frontend to support `wiki_config`, improving IDE autocomplete and type checking.
This fix addresses the critical issue where `wiki_config` was not being updated due to its absence in the `KnowledgeBaseConfig` struct, ensuring proper handling of wiki configurations during knowledge base updates.
Implemented three critical fixes to enable wiki_config persistence during
KB creation and updates:
1. Add WikiConfig field to KnowledgeBaseConfig struct
- File: internal/types/knowledgebase.go
- Enables JSON unmarshaling of wiki_config from update requests
- Previously the field was silently dropped during deserialization
2. Update UpdateKnowledgeBase service to persist wiki_config
- File: internal/application/service/knowledgebase.go
- Added logic to apply wiki_config changes when config parameter is provided
- Mirrors existing logic for FAQConfig persistence
3. Update frontend TypeScript types for IDE completion
- File: frontend/src/api/knowledge-base/index.ts
- Added WikiConfig type to createKnowledgeBase function signature
- Added structured WikiConfig type to updateKnowledgeBase config parameter
- Enables TypeScript IDE autocomplete and type checking
Flow:
- Frontend: updateKnowledgeBase({name, description, config: {wiki_config}})
- Handler: Parses into UpdateKnowledgeBaseRequest via KnowledgeBaseConfig
- Service: Applies wiki_config to KB entity via UpdateKnowledgeBase
- Database: Persists wiki_config JSONB field to postgres
Root cause analysis:
Previously, the KnowledgeBaseConfig struct was missing the WikiConfig field,
causing JSON unmarshaling to silently drop any wiki_config data sent in
PUT requests. KB creation worked because it used the full KnowledgeBase struct.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes:
**Backend Changes:**
- Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log)
- Database schema: wiki_pages table with full migration support
- WikiPageService: CRUD operations and page management
- WikiPageRepository: GORM-based persistence layer
- Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents
* Summary page generation using LLM
* Entity and concept extraction in a single LLM call
* Synthesis opportunity detection
* Index page rebuilding
* Log page maintenance
- Wiki boost feature: Enhance chat retrieval with wiki context
- Wiki linting: Maintenance and validation utilities
- Agent wiki tools: Enable agents to query and interact with wiki pages
- Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks
- Language support: Reuse existing middleware language infrastructure for LLM prompts
**Frontend Changes:**
- Wiki browser UI: View all wiki pages with filtering and search
- Wiki API client: Knowledge base wiki management endpoints
- Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model)
- i18n updates: Support for English, Korean, Russian, and Chinese interfaces
**Configuration:**
- Container DI: Wire up all wiki services
- Router: Register wiki API endpoints
- Task handling: Support async wiki ingest tasks
**Testing:**
- Unit tests for wiki page types
- Service layer tests
- Endpoint tests for wiki operations
- Integration tests with LLM mocking
**Documentation:**
- Language refactoring analysis and guides
- Implementation completion reports
- Quick reference guides for developers
**Key Features:**
✅ LLM-powered wiki page generation from documents
✅ Multi-language support (9+ languages)
✅ Automatic extraction of entities and concepts
✅ Synthesis opportunity detection
✅ Index and log page maintenance
✅ Progressive wiki building across multiple documents
✅ Agent-based wiki interaction
✅ Chat retrieval enhancement with wiki context
✅ Full frontend UI for wiki browsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Syncs personal and team Yuque knowledge bases into WeKnora as Markdown,
with full and incremental sync + deletion detection. Mirrors the Feishu
connector pattern; fetches V2Doc.body directly (no async export step).
Config: api_token (required) + base_url (optional, defaults to public cloud).
401/403 wraps ErrInvalidCredentials; 429 honors Retry-After; filters to
status=1 Doc-type at list stage to conserve API quota. Body is trusted as
Markdown for format=markdown|lake — empirically verified against the v2
API: Lake XML lives separately in body_lake, body itself is the Markdown
view. Other formats (e.g. html) are skipped with a placeholder item so
they surface in sync logs instead of silently feeding HTML to the parser.
v2Doc.status accepts both int and string shapes — Yuque's OpenAPI spec
declares string but the live API returns int. sanitizeFileName truncates
at a UTF-8 rune boundary so multi-byte titles don't emit invalid UTF-8
that downstream filename validation rejects.
Also adds logger.SetOutput test hook, used by the token-redaction test.
- Extend RetrieveEngineRegistry with byStoreID map for DB store engines
- Add StoreRegistry interface and EngineFactory function type
- Create engine factory with per-driver SDK client construction
- Load VectorStore records from DB into registry at startup
- Wire Create/Delete to dynamically register/unregister engines
- Add comprehensive unit tests for registry and service
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
Introduce KnowledgePostProcess task to decouple image multimodal
processing from downstream operations like summary, question
generation, and knowledge graph extraction.
This allows unifying the processing logic across documents and
images and ensures all text content, including from images,
is properly extracted before post-processing.
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):
1. Redis nil pointer panic on FAQ import (knowledge.go)
- Add in-memory fallback (sync.Map) for FAQ import progress tracking
- Add nil guards for running task locks and progress queries
2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
- GORM autoIncrement on non-PK columns doesn't work in SQLite
- Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
- Add BeforeCreate hook for KnowledgeTag
- Use Unscoped() to include soft-deleted records in MAX(seq_id) query
3. NOW() function not available in SQLite (chunk.go)
- Replace raw SQL NOW() with time.Now() or datetime('now')
- Use dialector check for raw SQL that must differ between PG and SQLite
4. FAQ KB should reject file uploads (knowledge.go)
- Add kb.Type == "faq" check in CreateKnowledgeFromFile
5. Linux/macOS build compatibility (Makefile)
- Conditionally apply macOS-only linker flag based on uname
6. Add SQLite integration tests (chunk_sqlite_test.go)
- TestCreateChunks_SQLite_SeqIDAutoAssigned
- TestCreateChunks_SQLite_SeqIDContinuesFromExisting
- TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
- TestCreateChunks_SQLite_SeqIDAfterSoftDelete
- TestKnowledgeTag_SQLite_SeqIDAutoAssigned
- TestUpdateChunk_SQLite_NoNOWError
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issunion