Commit Graph

28 Commits

Author SHA1 Message Date
wizardchen e76738d054 feat(wiki): implement folder management features in wiki API
Add new API functions for managing wiki folders, including creating, updating, deleting, and listing folders. Introduce a new WikiFolder interface to represent folder structure and enhance the WikiBrowser component with folder actions. Update internationalization strings for folder-related actions in English, Korean, Russian, and Chinese locales. Refactor existing code to accommodate folder hierarchy and improve user experience in the wiki browser.
2026-06-16 23:51:29 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00
jackson.jia c8645843c3 fix(knowledge): wrap locale code with LanguageLocaleName in wiki ingest prompts
The wiki ingest pipeline was passing raw locale codes (e.g. "zh-CN",
"en-US") directly into LLM prompts where a human-readable language name
is expected. The model would see "zh-CN" instead of "Chinese (Simplified)"
in its instructions, which could degrade output quality.

Two call sites were affected:

- SlugUpdate.Language (ProcessWikiIngest, retract path): consumed by the
  reduce phase to tell the page editor model what language to write in.
- lang variable (mapOneDocument): consumed by entity/concept extraction,
  chunk citation classification, and index page rebuild — all LLM prompts.

Wrap both with types.LanguageLocaleName() which maps locale codes to
descriptive names (e.g. "zh-CN" → "Chinese (Simplified)"). Unknown
locales fall through unchanged so no information is lost.
2026-05-30 16:56:32 +08:00
wizardchen 4a0a151d41 fix(knowledge): drain finalizing counter on all terminal subtask exits
The finalizing subtask counter (introduced when wiki ingest was counted)
could leak slots, leaving a fully-parsed doc stuck in "finalizing" until
the housekeeping sweep wrongly marked it "failed".

- wiki ingest: a doc skipped in map (knowledge deleted / no chunks /
  insufficient text) produced no docResult and was not a failedOp, so
  neither the success nor the dead-letter drain fired. Drain the slot on
  that terminal skip path.
- summary & question: the drain was keyed on the span-error variable,
  which assumes "err != nil => asynq will retry". Several branches set
  that variable yet `return nil` (insufficient text content, KB/knowledge
  fetch failures) - terminal, no retry - so the drain was skipped. Key
  the drain on the value actually returned to asynq (named retErr)
  instead, so terminal nil-returns drain and only retried errors wait
  for the final attempt.

Also fix the trace panel header flashing "已完成" mid-wiki: the latest
attempt's root span closes while async post-pipeline subspans keep
running, so trace.status read terminal while the row was still
"finalizing". Prefer parse_status on the latest attempt while it is
non-terminal so the panel header, LIVE badge and doc card agree, and add
the "finalizing" status label to all locales.
2026-05-29 19:32:13 +08:00
wizardchen 72272b2d62 feat(knowledge): count wiki ingest in the finalizing subtask counter
Wiki ingest runs asynchronously after the parse pipeline and was not
counted in pending_subtasks_count, so a document flipped to "completed"
while wiki generation was still minutes away (30s debounce + batch +
retries). That hid the in-progress state and dropped the stop-parse
affordance before wiki actually finished.

Count wiki as a single enrichment subtask (when WikiEnabled and the doc
has text chunks, matching the enqueue condition) so the row stays in
"finalizing" until wiki is done. The batch worker drains the slot at the
op's terminal state: once on successful map, and once when an op is
dead-lettered after exhausting in-batch retries. Retract ops (deleted
knowledge) are skipped. FinalizeSubtask guards both the decrement and the
promote, so a wiki op enqueued before this accounting shipped is a
harmless no-op on an already-completed row. A wiki op that never drains is
bounded by the existing housekeeping finalizing sweep.
2026-05-29 19:32:13 +08:00
wizardchen 797f55c567 feat(timeline): enhance knowledge processing spans and UI elements
This commit introduces several improvements to the knowledge processing timeline and related components. Key changes include:

1. Added a `gracePoll` prop to the `KnowledgeProcessingTimeline` component to manage polling behavior more effectively.
2. Enhanced the UI by displaying the document title in the drawer, improving user visibility of the current document context.
3. Implemented new CSS classes for better styling of the drawer title bar, ensuring a more polished appearance.
4. Updated the backend to support the new `WikiSpan` tracking, allowing for detailed monitoring of document processing stages.

These changes aim to improve user experience and provide better insights into the document processing workflow.
2026-05-28 15:14:45 +08:00
wizardchen 82947c726e fix(wiki-ingest,asynqdl): review fixups for PR #1241
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
  trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
  items" had two args), and harden validMerge against un-prefixed
  slugs whose strings.Index returned -1 and silently passed the type
  check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
  ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
  letter row instead of a hard-coded 0; tighten payloadProbe to the
  set of field names with consistent semantics across payloads
  (drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
  outside an asynq worker ctx so the semantics stay pinned.
2026-05-10 00:01:06 +08:00
wizardchen 04e8105728 perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.

This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.

  - migration 000041: task_pending_ops + task_dead_letters tables, plus
    three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
    source_refs text fallback, lower(title) trgm).
  - TaskPendingOpsRepository + TaskDeadLetterRepository in
    internal/application/repository/task_queue.go: cursor-paginated
    list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
    delete that refuses empty keys so a buggy caller can't wipe a KB's
    queue.
  - internal/middleware/asynqdl: writes a task_dead_letters row when an
    asynq task exhausts its retry budget. Payload-agnostic — a small
    probe struct extracts TenantID + scope hints across every existing
    payload type, so summary:generation / image:multimodal /
    faq_import / kb:clone / etc. all dead-letter without per-handler
    code. Best-effort: an insert failure never masks the underlying
    task error. Installed in router/task.go before the langfuse mw so
    it sees raw errors.

  - Pending queue moved Redis → PG (no TTL, restart-durable). Redis
    keeps just the active-batch lock and the delete tombstone.
  - In-batch retry budget (wikiMaxFailRetries=5) tracked via
    pendingRepo.IncrFailCount; over the cap the row is moved to
    task_dead_letters with task_type=wiki:ingest, related_id=knowledge
    id. Asynq retries (10) are handled by the new middleware.
  - Removed the per-batch ListAllPages. WikiBatchContext now carries
    lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
    mutex-protected caches; reduce reaches for titles / summaries on
    demand instead of pre-loading the whole KB.
  - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
    which the new GIN index on source_refs serves as a Bitmap Index
    Scan instead of a sequential text LIKE.
  - Dedup pre-filter uses pg_trgm via FindSimilarPages
    (idx_wiki_pages_title_trgm). For each new entity/concept (and each
    of its aliases) we ask the DB for the top-K trigram-similar
    existing pages and union the results — bounded prompt size, no Go-
    side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
  - cleanDeadLinks / injectCrossLinks are scoped to the batch's
    affected slugs (a few dozen) instead of every page in the KB. Both
    use the new lite ListBySlugs / ExistsSlugs repo methods so they
    pull only slug + title + outlinks, not full content.
  - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
    [[bad-slug|display]] the cleanup path now tries display-text
    reverse lookup → hyphen/case normalized equality → char-bigram
    Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
    break drift case ("shang-hai-tower" vs "shanghai-tower") in place
    instead of replacing the link with plain text.
  - rebuildIndexPage uses ListByTypeRecent(200) for the first-time
    intro and drops the full DocumentSummaries blob from the
    incremental update prompt, so its context stays bounded regardless
    of KB size.
  - Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
    IngestMapParallel / IngestReduceParallel, with sensible defaults
    via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
    follow-ups don't waste asynq retry slots bouncing on the active
    lock.

  - RunLint walks pages via the new ListPagesCursor in 200-page
    windows and computes the live-slug set with a one-column
    ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
    every node + edge. Memory is now bounded; 40k pages walks in
    constant ~4MB.

  - 14 GORM tests for both repos against an in-memory SQLite mirror of
    the production DDL (task_queue_test.go).
  - 6 tests for the dead-letter middleware covering retry budget
    detection, payload-agnostic scope inference, error truncation, and
    repo-failure isolation.
  - 7 tests for the slug fuzzy resolve helper covering the three
    resolution stages, display-text priority over normalized equality,
    bigram fallback acceptance, and rejection of unrelated slugs.
  - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
    new fetcher / cursor APIs.
2026-05-10 00:01:06 +08:00
wizardchen b7388f10e1 fix(wiki-ingest): reconcile summary links and log feed when reduce LLM fails
When a doc's entity/concept page generation hit a transient LLM error in
the reduce phase, the page never got written, but the doc's summary page
(generated in parallel with the slug list baked into its content) was
already persisted with [[entity/foo|name]] links pointing at the missing
page. The wiki log feed also surfaced the failed slugs as clickable
entries that 404'd. cleanDeadLinks only ran for retract batches, so the
debris persisted indefinitely.

reduceSlugUpdates now reports addition-path failures back to the batch
driver, which collects them and:

  - rewrites this batch's summary pages, replacing dead [[slug|display]]
    refs with plain display text (or a humanized slug tail);
  - filters dead slugs out of docResults[].Pages before the wiki log
    entries are flushed;
  - broadens the cleanDeadLinks trigger to fire whenever any page was
    touched or any slug failed, as a long-tail safety net.

Pure text replacement; no extra LLM calls. Failed slugs are picked up
naturally on the next ingest of the same document via the existing slug-
continuity rules, so the system self-heals without manual reingest.
2026-05-09 02:10:15 +08:00
wizardchen e66c42343b fix(wiki-ingest): harden conflict retry and right-size retry budget
Avoid treating Redis pending-length read failures as empty queues, reset stale fail-count keys on fresh ingest/drop paths, and reduce wiki ingest MaxRetry to a moderate shared constant to limit queue churn.
2026-05-08 21:58:51 +08:00
wizardchen d0144f3586 perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:48:52 +08:00
toy0116 bf12868923 fix(wiki): cap requeueFailedOps retry count to prevent queue pile-up
Root cause: requeueFailedOps unconditionally RPush'd every failed op back to
wiki:pending, causing unbounded growth when a document consistently triggers
LLM timeouts. Observed: 553 entries in wiki:pending:08134644 (one KB), mostly
duplicates for ~5 unique documents that each timed out on every batch cycle.

Fix:
- Add wikiFailCountKeyPrefix ("wiki:failcount:") and wikiMaxFailRetries (5)
  constants in wiki_ingest.go
- requeueFailedOps now atomically Incr(wiki:failcount:{kbID}:{knowledgeID})
  before each RPush; skips re-queue (drops op permanently) if count > 5
- Expire the fail-count key at wikiPendingTTL (24h) on every update
- wiki_ingest_batch.go: Del the fail-count key at ingestSucceeded++ so
  transient errors don't permanently burn through a document's retry budget

Evidence of problem severity: 5 persistently-failing docs × ~110 requeue
cycles each = 553 pending entries; had to use Lua atomic dedup to recover.

Behaviour after this fix:
- First 5 failures → normal retry (re-queued, follow-up batch scheduled)
- 6th failure → logged as WARN "dropping op … after 6 failures (limit 5)",
  not re-queued; prevents the feedback loop
- If document is fixed upstream and re-ingested via EnqueueWikiIngest, the
  op is a fresh entry with no existing fail-count key → full 5-retry budget

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 21:46:36 +08:00
toy0116 edbcfa80ab fix(wiki-ingest): prevent retry exhaustion on concurrent lock conflict
Root cause: when multiple wiki:ingest tasks queue up for the same KB,
they all hit ErrWikiIngestConcurrent and retry every 15 s. With
MaxRetry=10 (150 s window), any batch taking longer than ~2.5 min
caused pending tasks to exhaust retries and land in archived.

Confirmed via Redis inspection: archived wiki:ingest task had
retry=10, retried=10, error="concurrent wiki task active".

Two-part fix:

1. Early-exit at lock conflict when pending list is empty.
   If another batch is active and the pending list is already empty,
   there is nothing left to process — the concurrent task will handle
   everything. Return nil (success) immediately instead of burning
   through retry slots on a guaranteed no-op. Only retry when the
   pending list still has items, i.e., we have real work to do.

2. Increase MaxRetry 10 → 25 across all four enqueue sites.
   25 × 15 s = ~6 min window; accommodates large KB batches and the
   60 s orphan-lock expiry with generous headroom. Combined with the
   early-exit above, redundant tasks now consume 0 retries instead of
   10, so the remaining budget is entirely available for genuine
   concurrency conflicts.

No API or schema changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 21:46:36 +08:00
Claude 7ca4b2f53c chore(comment): update slug source to knowledge ID
Drive-by comment fix: extractEntitiesAndConceptsNoUpsert still claimed
summary slugs are code-generated from the document title. Since the
previous commit they are derived from the knowledge ID instead.
Pure documentation change, no behavior impact.
2026-05-06 11:23:32 +08:00
Claude 6744e1cdd5 fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks
Self-review of the prior commit surfaced one critical regression and
several smaller issues. This change addresses them.

Critical: the old stripImageMarkup stripped the WHOLE <image>...</image>
block, but searchutil.EnrichContentWithImageInfo wraps successful VLM OCR
output in exactly that shape — so a scanned PDF where OCR worked was
being rejected by hasSufficientTextContent and routed to the same
"failed" path as the OCR-empty case. The fix splits stripping into:
  * <image_original>...</image_original> blocks removed entirely
    (they only carry the redundant Markdown ref already stripped above)
  * <image>, <images>, <image_caption>, <image_ocr> tags removed at
    the tag level, INNER TEXT PRESERVED, so OCR / caption text reaches
    the threshold check and the LLM call.

A regression test pins this down with realistic enriched-image XML that
the previous code failed silently on.

Other fixes:
  * Threshold lowered from 50 to 10 runes — short legitimate documents
    (brief memos, single-line notes) must still pass.
  * Summary chunk content no longer prefixes "# Document\n<filename>";
    only the LLM-generated summary is indexed, so filenames cannot
    leak back into RAG retrieval context.
  * Wiki summary slug is now derived from the knowledge ID rather than
    the upload filename, eliminating filename-based slugs like
    "summary/mx5280-pdf" that downstream wiki-edit prompts can read.
  * Citation sourceRef likewise drops the "|<docTitle>" suffix.
  * Dead docTitle parameter removed from extractCandidateSlugs,
    extractEntitiesAndConceptsNoUpsert, classifyChunkCitations.
  * The redundant stripImageMarkup pass in the warning log is gone;
    realTextRuneCount caches the count once and uses
    utf8.RuneCountInString to avoid the rune-slice allocation.

Wiki-page display titles still flow through DocTitle (filename) into
later wiki-edit / index-rebuild prompts. That is a larger refactor of
the wiki pipeline and is intentionally left for a separate PR.
2026-05-06 11:23:32 +08:00
Claude 15550d14d4 fix(ingest): stop hallucinating summaries from filenames on empty PDFs
Scanned PDFs without a successful OCR pass (e.g. when no VLM is configured
on the KB, or when the VLM returned a "no text content" reply that the
sanitizer discarded) reach the summary and wiki-ingest pipelines with
nothing but markdown image references for content. The downstream LLM
prompts used to receive the filename, file type, and title alongside that
empty content, and the generate_summary template explicitly forbade the
model from refusing — so the model would invent a topic from the filename
alone, producing e.g. a "Canon scanner manual" summary for an MX5280.pdf
that is actually a scanned legal letter.

Two changes shut this down:

1. Drop filename / title / file-type from every LLM prompt input. The
   wiki summary, knowledge extract, candidate slug, and chunk citation
   prompts now see only the document content. The knowledge summary
   path no longer prepends the "Document Type: X / File Name: Y" intro.

2. Add a content-sufficiency guard in front of the LLM call. A new
   helper strips markdown image references and HTML image tags and
   refuses to invoke the model when fewer than 50 real-text runes
   remain. getSummary returns a sentinel error in that case and the
   caller marks the knowledge as SummaryStatusFailed instead of
   surfacing the first chunk's bare image reference. Wiki ingest skips
   LLM extraction with a clear log message.

The generate_summary.yaml prompt is also updated: the "NEVER output
refusal phrases" clause is removed and replaced with an explicit
empty-content rule that returns a deterministic placeholder line.

Real OCR (Tesseract / PaddleOCR) and filename-pattern sanitisation are
intentionally left for separate PRs.
2026-05-06 11:23:32 +08:00
wizardchen 1f03741462 fix(wiki): Lite ingest lock, failed-op requeue, sync task retry parity
- Add per-KB sync.Map lock in Lite mode to mirror Redis SetNX concurrency.
- Requeue failed wiki ops in Lite by enqueueing asynq tasks with delay/tracing.
- Teach SyncTaskExecutor to honor ProcessIn delay and MaxRetry like Redis asynq.
2026-04-28 18:27:00 +08:00
wizardchen c34f7b6254 Enhance wiki ingest process to handle failed operations
- Introduced a mechanism to track and requeue failed operations during the wiki ingest process.
- Added a new `requeueFailedOps` function to append failed operations back to the Redis pending list for retry in subsequent batches.
- Updated the `ProcessWikiIngest` method to collect failed operations and ensure they are retried after trimming the pending list.

This change improves the robustness of the ingest process by preventing data loss from transient failures.
2026-04-28 17:21:42 +08:00
wizardchen beca2b89a3 feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.

This change threads one trace end-to-end:

- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
  worker can attach to an upstream trace without emitting a duplicate
  trace-create; StartGeneration now auto-picks parentObservationId from
  ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
  payloads so trace_id / parent_obs_id / user_id / session_id serialise
  into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
  so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
  resumes the upstream trace or opens a standalone asynq.<type> trace
  for scheduled jobs, and wraps the handler in a SPAN with task metadata
  (id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
  FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
  ResumeTrace no-trace-create guarantee; asynq_test.go covers
  InjectTracing round-trip, middleware resume path, and standalone
  trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
  and documents the cross-process propagation model.

No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 13:16:47 +08:00
wizardchen 1d5b86ddf5 feat: Enhance wiki cleanup and reparse logic for improved consistency
- Updated the scrubWikiPendingIngest method to accept a reason parameter, improving logging clarity during wiki cleanup and reparse operations.
- Introduced prepareWikiForReparse to maintain the integrity of the pending ingest queue during reparse events, preventing stale operations from executing.
- Enhanced the mapping logic in the wiki ingest process to differentiate between stale and reparse updates, ensuring accurate handling of knowledge contributions.
- Improved logging to provide better insights into the number of reparse and stale slugs processed during wiki ingestion.
2026-04-23 10:47:53 +08:00
wizardchen a8a3d9e694 feat: Enhance wiki extraction and configuration options
- Introduced extraction granularity settings in the wiki configuration, allowing users to control the level of detail in entity and concept extraction (focused, standard, exhaustive).
- Updated the UI to reflect these new options, including tooltips for better user guidance.
- Refactored the knowledge base editor to support the new extraction granularity feature, ensuring a seamless user experience.
- Improved the backend logic for wiki ingest tasks to utilize the new granularity settings, enhancing the accuracy of extracted content.
- Added tests to validate the new extraction granularity functionality and ensure robust performance across different configurations.
2026-04-22 21:18:32 +08:00
wizardchen fdccb68ee4 feat: Improve wiki cleanup and ingest handling for deleted knowledge
- Enhanced the `cleanupWikiOnKnowledgeDelete` function to pass the full knowledge object, allowing for more accurate title and summary retrieval during wiki cleanup.
- Introduced a tombstone mechanism in Redis to mark recently deleted knowledge, enabling in-flight wiki ingest tasks to skip processing for deleted documents.
- Updated the `isKnowledgeGone` function to check for deleted knowledge using the tombstone, improving efficiency in handling concurrent delete and ingest operations.
- Added logic in the `wikiIngestService` to filter out updates for deleted knowledge, ensuring that no ghost references are created during the ingest process.
- Implemented lint checks for stale source references in the `WikiLintService`, allowing for automatic cleanup of pages referencing deleted knowledge.

These changes significantly enhance the robustness of the wiki system, ensuring consistency and accuracy in handling knowledge deletions and related wiki pages.
2026-04-22 21:18:20 +08:00
wizardchen 53ecd5df5d feat: Enhance wiki content processing with enriched image information and XML output
- Introduced `enrichChunkImageInfo` function to populate image information for text chunks by retrieving data from child chunks, ensuring that image-related content is not lost during processing.
- Updated `reconstructContent` to `reconstructEnrichedContent`, allowing for the inclusion of OCR text and captions from images, improving the quality of content sent to the LLM.
- Added `writeDedupItemXML` and `xmlEscape` functions to format entity and concept entries as structured XML, enhancing clarity and preventing nonsensical merges in deduplication prompts.
- Refined the `deduplicateExtractedBatch` method to utilize the new XML formatting for better output consistency.

These changes significantly improve the handling of image data and the overall clarity of deduplication prompts, contributing to a more effective wiki content management system.
2026-04-22 21:18:09 +08:00
wizardchen a75233c90f refactor: Enhance deduplication logic and logging in wiki ingestion service
- Updated the `WikiDeduplicationPrompt` to clarify the criteria for identifying exact duplicates, emphasizing strict matching of real-world entities and concepts.
- Improved the `appendLogEntry` method to include a summary parameter for better context in log entries, enhancing traceability of operations.
- Consolidated log entry creation for both retraction and ingestion operations, ensuring chronological clarity in the log.
- Implemented validation for slug merges to prevent mismatches based on entity types, improving the accuracy of deduplication.

These changes enhance the clarity and reliability of the wiki ingestion process, leading to more accurate content management.
2026-04-22 21:17:34 +08:00
wizardchen 28781785f7 refactor: Optimize wiki ingestion process by consolidating entity and concept deduplication
- Introduced a new `WikiBatchContext` to share data across Map and Reduce phases, reducing redundant database queries.
- Replaced separate deduplication calls with a single `deduplicateExtractedBatch` method, improving efficiency by leveraging pre-loaded page data.
- Enhanced JSON handling by implementing a `cleanLLMJSON` function to sanitize LLM-generated output, ensuring safe parsing.
- Updated relevant methods to utilize the new context and deduplication logic, streamlining the overall ingestion workflow.

These changes enhance the performance and clarity of the wiki ingestion service, leading to more efficient content processing.
2026-04-22 21:17:33 +08:00
wizardchen fba6d7d145 refactor: Update wiki prompts and ingestion logic for improved content handling
- Renamed `WikiPageUpdatePrompt` to `WikiPageModifyPrompt` to better reflect its functionality of adding new information and removing outdated content in a single operation.
- Enhanced the instructions for both `WikiPageModifyPrompt` and `WikiPageRetractPrompt` to clarify the handling of additions and retractions, ensuring accurate content updates.
- Consolidated the logic for enqueuing retraction tasks in the `cleanupWikiOnKnowledgeDelete` method to streamline the process of managing affected pages.
- Removed the `rebuildWikiIndexSimple` function to simplify the codebase, as its functionality is no longer needed with the new ingestion approach.

These changes improve the clarity and efficiency of the wiki content management process, enhancing overall system performance.
2026-04-22 21:17:31 +08:00