e8dcbb07d09db88894601b564ebabe21dda6830d
19 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wizardchen
|
d496d0fc73 |
feat(task-queue): split upstream workers into isolated core/enrichment/maintenance pools
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology. |
||
|
wizardchen
|
ed9e780c8c |
feat(client): enhance knowledge file upload with process configuration support
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides. - Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations. - Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads. - Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process. - Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic. - Enhanced user experience by providing detailed configuration options during the upload process. |
||
|
wizardchen
|
7b1bb1054f |
feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:
docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
thread-safe so we fan out across processes. Configurable via
DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
image, so documents with many page images are no longer capped by the unary
gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
backward compatibility; the Go production reader switches to ReadStream.
VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
the default 90s -> 180s so dense scanned-page OCR does not time out with
"context deadline exceeded".
Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
a single large document cannot saturate the shared worker pool and block
user-facing document parsing:
image:multimodal -> "multimodal"
chunk:extract -> "graph"
question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
scanned-queue set (so cancelling a knowledge still purges its pending tasks).
|
||
|
wizardchen
|
cea6ef0ce3 |
test(knowledge): add comprehensive tests for finalizing subtask behavior
Introduced a new test file for the knowledge repository, implementing various scenarios to ensure the reliability of the finalizing subtask counter. Key tests include: - Concurrent promotion of subtasks to verify atomic behavior. - Handling of partial decrements to ensure the row remains in "finalizing" state. - Safety checks to prevent underflow of the pending subtask count. - Regression tests to confirm that updates do not overwrite the pending counter. These tests address previous issues with the finalizing process and enhance overall stability. |
||
|
wizardchen
|
84bd95275d |
fix(knowledge): make finalizing subtask counter reliable + async question fan-out
The finalizing subtask counter (pending_subtasks_count) could get stuck at a non-zero value, leaving knowledge permanently in "finalizing" and never promoted to "completed". Root cause and related hardening: - UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the omit list, so any concurrent enrichment subtask that loaded the row, did slow work (LLM call), then saved an unrelated field wrote back the STALE counter, clobbering decrements other subtasks performed in between. Add PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it. - FinalizeSubtask no longer gates the finalizing->completed promote on a separate re-read of the counter. Every caller unconditionally runs the guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single authoritative atomic check, so a racy/stale read can't strand the row. - Decrements run on a context detached from the caller's cancellation (context.WithoutCancel + timeout) so graceful shutdown / preemption can't silently skip a decrement. - Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph) and release any shortfall so un-enqueued planned slots (e.g. graph with NEO4J off) don't strand the row. - Reparse paths reset pending_subtasks_count via an explicit column write since full-row Save now omits it. Question generation is also moved off the synchronous single-task path onto a batched async fan-out (windows of text chunks), each batch independently queued / retried / traced, grouped under a postprocess.question span. |
||
|
wizardchen
|
72272b2d62 |
feat(knowledge): count wiki ingest in the finalizing subtask counter
Wiki ingest runs asynchronously after the parse pipeline and was not counted in pending_subtasks_count, so a document flipped to "completed" while wiki generation was still minutes away (30s debounce + batch + retries). That hid the in-progress state and dropped the stop-parse affordance before wiki actually finished. Count wiki as a single enrichment subtask (when WikiEnabled and the doc has text chunks, matching the enqueue condition) so the row stays in "finalizing" until wiki is done. The batch worker drains the slot at the op's terminal state: once on successful map, and once when an op is dead-lettered after exhausting in-batch retries. Retract ops (deleted knowledge) are skipped. FinalizeSubtask guards both the decrement and the promote, so a wiki op enqueued before this accounting shipped is a harmless no-op on an already-completed row. A wiki op that never drains is bounded by the existing housekeeping finalizing sweep. |
||
|
wizardchen
|
44d6175559 |
feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.
State machine
pending -> processing -> finalizing -> completed
|
+-> cancelled (any of the three
in-flight states)
+-> failed
+-> deleting
`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.
Backend
- New ParseStatusFinalizing + pending_subtasks_count column with
migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
count up front, flip to finalizing (or completed when no
enrichment is enabled), and only then fan out subtasks. Subtask
handlers (summary, question, graph extract) defer-decrement on
terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
pending / processing / finalizing. Marks the row cancelled,
zeroes the counter, best-effort dequeues asynq tasks via a new
TaskInspector abstraction (asynq-mode walks pending/scheduled/
retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
for the attempt via a new repo.CancelAllOpenSpans helper so the
trace viewer's striped bars all flip to cancelled, even leaf
generations whose parent stage already EndSpan'd (multimodal
fan-out pattern). knowledge_post_process closes its postSpan
via SkipSpan on the cancel/deleting entry guard so a worker
that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
expensive enrichment (graph extract) bails immediately when the
parent knowledge is aborted.
Frontend
- New cancelKnowledgeParse API client + "Stop parsing" entry in
both list view and card view more menus, gated on
pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
that recognises `finalizing` (previously the doc list silently
stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
new isHardTerminal short-circuits LIVE for cancelled/failed/
completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
("增强中") from completed and shows the previous "生成摘要中"
copy when summary_status is still pending under finalizing.
Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
across zh-CN, en-US, ko-KR, ru-RU.
Docs / SDK
- docs/api/knowledge.md: documents the new finalizing state,
cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
the cancellable statuses.
|
||
|
wizardchen
|
bd029b6d19 |
feat(knowledge): instrument postprocess subspans + polish timeline UI
Backend: summary, question, and graph-extract async tasks now record real processing time as subspans hanging off the (closed) postprocess stage, so the trace viewer no longer caps the postprocess row at the ~10 ms enqueue duration. Carries Attempt through SummaryGeneration / QuestionGeneration / ExtractChunk payloads so cross-process workers can resolve the right parent attempt. Frontend: drawer now uses attach="body" so the secondary 820px detail drawer escapes the 654px container of the main drawer; timeline timestamps include date prefix (MM-DD, or YYYY-MM-DD across years); "updated X ago" caption only shows during live polling. Tests: 4 new cases covering postprocess subspan attaching under a closed parent, missing-parent fallthrough, and Attempt JSON round-trip on the three task payloads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
wizardchen
|
4db6e69b4e |
fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.
- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
closes the root row idempotently. Re-closing a terminal root no-ops
so success / cascade-fail / dead-letter paths can fire without
coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
on the success path. Async downstream work (summary/question/wiki/
graph) still records its own spans; their completion extends the
trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
(docreader / chunking / embedding / multimodal / postprocess).
Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
document-related task exhausts retries. The probe payload now
extracts the Attempt field that Document/Manual/PostProcess
payloads already thread through.
Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph
i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
|
||
|
wizardchen
|
c9941a6688 |
refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:
1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
are independent of each other, both downstream of Chunking, both
upstream of PostProcess. The flat (knowledge_id, stage) table
couldn't represent that, so a Chunking failure left dependents
stranded as "pending" forever instead of being marked as
impossible-to-run.
2. No history across attempts. A reparse erased the previous run's
status before the new run started, leaving operators with no way
to investigate "why did this fail twice?".
3. Stages had only status + duration. Operators want to know how big
the work was — pages parsed, chunks created, tokens embedded, VLM
calls made — to distinguish "slow because the file is huge" from
"slow because the docreader is wedged".
4. Multimodal fans out N image tasks; Embedding fans out M batches;
PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
is interesting on its own (Langfuse already captures this for
LLM calls). The flat model couldn't express it.
The redesign mirrors Langfuse's trace/span/generation hierarchy:
* Migration 000053 supersedes 000052: knowledge_processing_spans
table with (knowledge_id, attempt, span_id) primary key, plus
parent_span_id, kind ∈ {root, stage, subspan, generation},
status ∈ {pending,running,done,failed,skipped,cancelled}, and
JSONB input/output/metadata fields.
* SpanTracker (replacing StageTracker) exposes OpenAttempt /
BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
LookupStage. Cross-process workers (image_multimodal) get the
parent's attempt + span via payload + LookupStage so subspans
attach correctly.
* StageDependencies declares the DAG; FailSpan now cascades —
descendants of the failed span and dependent stages are flipped
to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
blast radius instead of orphan spinners.
* Reparse now calls OpenAttempt up front so the timeline reflects
"new attempt, all pending" instead of letting the previous run's
status linger until the worker picks up the task.
* Image_multimodal records each image as a generation subspan with
its own success/failure on the parent attempt's multimodal stage.
The finalize-on-last-attempt counter logic is preserved unchanged.
* GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
a tree shape with synthesized pending placeholders so the
frontend always renders five timeline segments. ?attempt=N
enables history navigation.
|
||
|
wizardchen
|
04f56f9cda |
feat(knowledge): track per-stage parsing progress with /stages API
Adds a five-segment progress model for the document parsing pipeline so the UI (PR③) can render a timeline showing where each document is (DocReader → Chunking → Embedding → Multimodal → PostProcess) and which stage failed with what error code. - New table `knowledge_processing_stages` (migration 000052) with one row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an attempt counter so re-parses don't lose history. - StageTracker service exposes Begin/Done/Fail/Skip; all calls are best-effort and never break the pipeline if persistence fails. - Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT / VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized remediation hints. - Tracker call sites added at the four meaningful failure points: convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding), enqueueImageMultimodalTasks (Multimodal start), KnowledgePostProcess.Handle (Multimodal close + PostProcess). - New endpoint `GET /api/v1/knowledge/:id/stages` returns the five canonical stages — missing rows are synthesized as "pending" so the timeline always renders five segments. Includes current_stage and last_error block. |
||
|
wizardchen
|
04e8105728 |
perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
|
||
|
wizardchen
|
beca2b89a3 |
feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests (chat / search / eval), so file uploads and every downstream asynq task (document parse, chunk embedding, OCR/VLM, summary / question gen, wiki ingest, datasource sync, etc.) produced either disconnected shallow traces or no observation at all. This change threads one trace end-to-end: - tracer: add SPAN observation type and StartSpan; add ResumeTrace so a worker can attach to an upstream trace without emitting a duplicate trace-create; StartGeneration now auto-picks parentObservationId from ctx so nested trace -> span -> generation trees render correctly. - types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq payloads so trace_id / parent_obs_id / user_id / session_id serialise into every job. - langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal so the HTTP-layer trace survives the Redis hop. - langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either resumes the upstream trace or opens a standalone asynq.<type> trace for scheduled jobs, and wraps the handler in a SPAN with task metadata (id / queue / retry / payload_bytes) plus ERROR level on failure. - GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation / FAQ / wiki / datasource endpoints so the root trace actually starts. - Tests: tracer_test.go covers span nesting, error status, and ResumeTrace no-trace-create guarantee; asynq_test.go covers InjectTracing round-trip, middleware resume path, and standalone trace fallback. - Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types and documents the cross-process propagation model. No behavioural change when Langfuse is disabled (all new code paths are no-ops and carriers serialise to empty strings with omitempty). |
||
|
wizardchen
|
a8a3d9e694 |
feat: Enhance wiki extraction and configuration options
- Introduced extraction granularity settings in the wiki configuration, allowing users to control the level of detail in entity and concept extraction (focused, standard, exhaustive). - Updated the UI to reflect these new options, including tooltips for better user guidance. - Refactored the knowledge base editor to support the new extraction granularity feature, ensuring a seamless user experience. - Improved the backend logic for wiki ingest tasks to utilize the new granularity settings, enhancing the accuracy of extracted content. - Added tests to validate the new extraction granularity functionality and ensure robust performance across different configurations. |
||
|
wizardchen
|
28deb8282f |
feat: Add comprehensive documentation for WeKnora knowledge base and wiki features
- Introduced multiple new documentation files, including a complete documentation index, quick reference guides, and comprehensive analyses of the knowledge base and wiki settings. - The new files provide detailed insights into the architecture, data models, API endpoints, and implementation strategies for the knowledge base and wiki functionalities. - Enhanced navigation and usability for developers and team members by summarizing key components, configurations, and data flows related to the knowledge base system. These additions significantly improve the clarity and accessibility of documentation, supporting better onboarding and reference for current and future team members. |
||
|
wizardchen
|
4b3f41812f |
feat: Update chat creation UI and enhance knowledge post-processing logic
- Modified the chat creation component to improve layout and responsiveness by adjusting the dialogue answers section. - Refined the logic in the KnowledgePostProcessService to conditionally spawn wiki ingest tasks based on wiki configuration settings, enhancing task management. - Added migration scripts for creating and dropping the `wiki_page_issues` table, facilitating better issue tracking within the wiki system. These changes enhance the user interface for chat creation and improve the backend processing of knowledge base tasks, contributing to a more efficient and user-friendly experience. |
||
|
wizardchen
|
99cc721140 |
feat: Enhance wiki API and repository with slug encoding and improved search functionality
- Introduced `encodeSlugPath` function to safely encode wiki slugs for API requests, preserving hierarchical routing. - Updated API functions to utilize the new slug encoding for `getWikiPage`, `updateWikiPage`, and `deleteWikiPage`. - Enhanced `ListBySourceRef` method in the repository to safely handle source knowledge IDs with JSON marshalling, improving query security. - Added `escapeLikePattern` function to ensure safe concatenation of LIKE patterns in SQL queries, preventing unintended matches. - Updated `KnowledgePostProcessService` to include Redis client for task management, improving performance in handling wiki ingest tasks. - Enhanced `WikiLintIssue` structure to include `TargetSlug` for better issue tracking and auto-fixing capabilities. These changes significantly improve the robustness and security of the wiki management system, enhancing user experience and data integrity. |
||
|
wizardchen
|
e147209e77 |
feat: Implement comprehensive wiki feature for knowledge bases
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes: **Backend Changes:** - Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log) - Database schema: wiki_pages table with full migration support - WikiPageService: CRUD operations and page management - WikiPageRepository: GORM-based persistence layer - Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents * Summary page generation using LLM * Entity and concept extraction in a single LLM call * Synthesis opportunity detection * Index page rebuilding * Log page maintenance - Wiki boost feature: Enhance chat retrieval with wiki context - Wiki linting: Maintenance and validation utilities - Agent wiki tools: Enable agents to query and interact with wiki pages - Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks - Language support: Reuse existing middleware language infrastructure for LLM prompts **Frontend Changes:** - Wiki browser UI: View all wiki pages with filtering and search - Wiki API client: Knowledge base wiki management endpoints - Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model) - i18n updates: Support for English, Korean, Russian, and Chinese interfaces **Configuration:** - Container DI: Wire up all wiki services - Router: Register wiki API endpoints - Task handling: Support async wiki ingest tasks **Testing:** - Unit tests for wiki page types - Service layer tests - Endpoint tests for wiki operations - Integration tests with LLM mocking **Documentation:** - Language refactoring analysis and guides - Implementation completion reports - Quick reference guides for developers **Key Features:** ✅ LLM-powered wiki page generation from documents ✅ Multi-language support (9+ languages) ✅ Automatic extraction of entities and concepts ✅ Synthesis opportunity detection ✅ Index and log page maintenance ✅ Progressive wiki building across multiple documents ✅ Agent-based wiki interaction ✅ Chat retrieval enhancement with wiki context ✅ Full frontend UI for wiki browsing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
wizardchen
|
a1a07e6818 |
Refactor knowledge post-processing into orchestrator task
Introduce KnowledgePostProcess task to decouple image multimodal processing from downstream operations like summary, question generation, and knowledge graph extraction. This allows unifying the processing logic across documents and images and ensures all text content, including from images, is properly extracted before post-processing. |