Commit Graph

37 Commits

Author SHA1 Message Date
wizardchen 82947c726e fix(wiki-ingest,asynqdl): review fixups for PR #1241
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
  trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
  items" had two args), and harden validMerge against un-prefixed
  slugs whose strings.Index returned -1 and silently passed the type
  check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
  ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
  letter row instead of a hard-coded 0; tighten payloadProbe to the
  set of field names with consistent semantics across payloads
  (drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
  outside an asynq worker ctx so the semantics stay pinned.
2026-05-10 00:01:06 +08:00
wizardchen 04e8105728 perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.

This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.

  - migration 000041: task_pending_ops + task_dead_letters tables, plus
    three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
    source_refs text fallback, lower(title) trgm).
  - TaskPendingOpsRepository + TaskDeadLetterRepository in
    internal/application/repository/task_queue.go: cursor-paginated
    list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
    delete that refuses empty keys so a buggy caller can't wipe a KB's
    queue.
  - internal/middleware/asynqdl: writes a task_dead_letters row when an
    asynq task exhausts its retry budget. Payload-agnostic — a small
    probe struct extracts TenantID + scope hints across every existing
    payload type, so summary:generation / image:multimodal /
    faq_import / kb:clone / etc. all dead-letter without per-handler
    code. Best-effort: an insert failure never masks the underlying
    task error. Installed in router/task.go before the langfuse mw so
    it sees raw errors.

  - Pending queue moved Redis → PG (no TTL, restart-durable). Redis
    keeps just the active-batch lock and the delete tombstone.
  - In-batch retry budget (wikiMaxFailRetries=5) tracked via
    pendingRepo.IncrFailCount; over the cap the row is moved to
    task_dead_letters with task_type=wiki:ingest, related_id=knowledge
    id. Asynq retries (10) are handled by the new middleware.
  - Removed the per-batch ListAllPages. WikiBatchContext now carries
    lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
    mutex-protected caches; reduce reaches for titles / summaries on
    demand instead of pre-loading the whole KB.
  - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
    which the new GIN index on source_refs serves as a Bitmap Index
    Scan instead of a sequential text LIKE.
  - Dedup pre-filter uses pg_trgm via FindSimilarPages
    (idx_wiki_pages_title_trgm). For each new entity/concept (and each
    of its aliases) we ask the DB for the top-K trigram-similar
    existing pages and union the results — bounded prompt size, no Go-
    side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
  - cleanDeadLinks / injectCrossLinks are scoped to the batch's
    affected slugs (a few dozen) instead of every page in the KB. Both
    use the new lite ListBySlugs / ExistsSlugs repo methods so they
    pull only slug + title + outlinks, not full content.
  - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
    [[bad-slug|display]] the cleanup path now tries display-text
    reverse lookup → hyphen/case normalized equality → char-bigram
    Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
    break drift case ("shang-hai-tower" vs "shanghai-tower") in place
    instead of replacing the link with plain text.
  - rebuildIndexPage uses ListByTypeRecent(200) for the first-time
    intro and drops the full DocumentSummaries blob from the
    incremental update prompt, so its context stays bounded regardless
    of KB size.
  - Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
    IngestMapParallel / IngestReduceParallel, with sensible defaults
    via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
    follow-ups don't waste asynq retry slots bouncing on the active
    lock.

  - RunLint walks pages via the new ListPagesCursor in 200-page
    windows and computes the live-slug set with a one-column
    ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
    every node + edge. Memory is now bounded; 40k pages walks in
    constant ~4MB.

  - 14 GORM tests for both repos against an in-memory SQLite mirror of
    the production DDL (task_queue_test.go).
  - 6 tests for the dead-letter middleware covering retry budget
    detection, payload-agnostic scope inference, error truncation, and
    repo-failure isolation.
  - 7 tests for the slug fuzzy resolve helper covering the three
    resolution stages, display-text priority over normalized equality,
    bigram fallback acceptance, and rejection of unrelated slugs.
  - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
    new fetcher / cursor APIs.
2026-05-10 00:01:06 +08:00
wizardchen b7388f10e1 fix(wiki-ingest): reconcile summary links and log feed when reduce LLM fails
When a doc's entity/concept page generation hit a transient LLM error in
the reduce phase, the page never got written, but the doc's summary page
(generated in parallel with the slug list baked into its content) was
already persisted with [[entity/foo|name]] links pointing at the missing
page. The wiki log feed also surfaced the failed slugs as clickable
entries that 404'd. cleanDeadLinks only ran for retract batches, so the
debris persisted indefinitely.

reduceSlugUpdates now reports addition-path failures back to the batch
driver, which collects them and:

  - rewrites this batch's summary pages, replacing dead [[slug|display]]
    refs with plain display text (or a humanized slug tail);
  - filters dead slugs out of docResults[].Pages before the wiki log
    entries are flushed;
  - broadens the cleanDeadLinks trigger to fire whenever any page was
    touched or any slug failed, as a long-tail safety net.

Pure text replacement; no extra LLM calls. Failed slugs are picked up
naturally on the next ingest of the same document via the existing slug-
continuity rules, so the system self-heals without manual reingest.
2026-05-09 02:10:15 +08:00
wizardchen e66c42343b fix(wiki-ingest): harden conflict retry and right-size retry budget
Avoid treating Redis pending-length read failures as empty queues, reset stale fail-count keys on fresh ingest/drop paths, and reduce wiki ingest MaxRetry to a moderate shared constant to limit queue churn.
2026-05-08 21:58:51 +08:00
wizardchen d0144f3586 perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:48:52 +08:00
toy0116 bf12868923 fix(wiki): cap requeueFailedOps retry count to prevent queue pile-up
Root cause: requeueFailedOps unconditionally RPush'd every failed op back to
wiki:pending, causing unbounded growth when a document consistently triggers
LLM timeouts. Observed: 553 entries in wiki:pending:08134644 (one KB), mostly
duplicates for ~5 unique documents that each timed out on every batch cycle.

Fix:
- Add wikiFailCountKeyPrefix ("wiki:failcount:") and wikiMaxFailRetries (5)
  constants in wiki_ingest.go
- requeueFailedOps now atomically Incr(wiki:failcount:{kbID}:{knowledgeID})
  before each RPush; skips re-queue (drops op permanently) if count > 5
- Expire the fail-count key at wikiPendingTTL (24h) on every update
- wiki_ingest_batch.go: Del the fail-count key at ingestSucceeded++ so
  transient errors don't permanently burn through a document's retry budget

Evidence of problem severity: 5 persistently-failing docs × ~110 requeue
cycles each = 553 pending entries; had to use Lua atomic dedup to recover.

Behaviour after this fix:
- First 5 failures → normal retry (re-queued, follow-up batch scheduled)
- 6th failure → logged as WARN "dropping op … after 6 failures (limit 5)",
  not re-queued; prevents the feedback loop
- If document is fixed upstream and re-ingested via EnqueueWikiIngest, the
  op is a fresh entry with no existing fail-count key → full 5-retry budget

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 21:46:36 +08:00
toy0116 edbcfa80ab fix(wiki-ingest): prevent retry exhaustion on concurrent lock conflict
Root cause: when multiple wiki:ingest tasks queue up for the same KB,
they all hit ErrWikiIngestConcurrent and retry every 15 s. With
MaxRetry=10 (150 s window), any batch taking longer than ~2.5 min
caused pending tasks to exhaust retries and land in archived.

Confirmed via Redis inspection: archived wiki:ingest task had
retry=10, retried=10, error="concurrent wiki task active".

Two-part fix:

1. Early-exit at lock conflict when pending list is empty.
   If another batch is active and the pending list is already empty,
   there is nothing left to process — the concurrent task will handle
   everything. Return nil (success) immediately instead of burning
   through retry slots on a guaranteed no-op. Only retry when the
   pending list still has items, i.e., we have real work to do.

2. Increase MaxRetry 10 → 25 across all four enqueue sites.
   25 × 15 s = ~6 min window; accommodates large KB batches and the
   60 s orphan-lock expiry with generous headroom. Combined with the
   early-exit above, redundant tasks now consume 0 retries instead of
   10, so the remaining budget is entirely available for genuine
   concurrency conflicts.

No API or schema changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 21:46:36 +08:00
Claude c3e53a3fc4 test(ingest): cover insufficient-content gate; allow threshold override
Addresses two pre-merge review notes:

1. Extract the sentinel-returning threshold check from getSummary into
   a small package-private helper (checkSufficientSummaryContent) and
   add a unit test for it. The new test pins the behavior the caller
   in ProcessSummaryGeneration relies on: the sentinel is returned for
   empty / image-only / sub-threshold / scanned-PDF-without-OCR inputs,
   and nil is returned for legitimate short notes and scanned PDFs
   where VLM OCR succeeded.

   The downstream "errors.Is(err, errInsufficientSummaryContent) →
   SummaryStatusFailed + skip summary chunk" branch is mechanical (no
   branching beyond the errors.Is check); covering it end-to-end would
   require mocking KnowledgeRepository / ChunkRepository / chat.Chat,
   for which there is no precedent in this package.

2. Switch minTextContentRunes from const to var so tests can override
   the threshold and so future config plumbing can adjust it at
   runtime without a rebuild. The new test exercises the override.
2026-05-06 11:23:32 +08:00
Claude 6744e1cdd5 fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks
Self-review of the prior commit surfaced one critical regression and
several smaller issues. This change addresses them.

Critical: the old stripImageMarkup stripped the WHOLE <image>...</image>
block, but searchutil.EnrichContentWithImageInfo wraps successful VLM OCR
output in exactly that shape — so a scanned PDF where OCR worked was
being rejected by hasSufficientTextContent and routed to the same
"failed" path as the OCR-empty case. The fix splits stripping into:
  * <image_original>...</image_original> blocks removed entirely
    (they only carry the redundant Markdown ref already stripped above)
  * <image>, <images>, <image_caption>, <image_ocr> tags removed at
    the tag level, INNER TEXT PRESERVED, so OCR / caption text reaches
    the threshold check and the LLM call.

A regression test pins this down with realistic enriched-image XML that
the previous code failed silently on.

Other fixes:
  * Threshold lowered from 50 to 10 runes — short legitimate documents
    (brief memos, single-line notes) must still pass.
  * Summary chunk content no longer prefixes "# Document\n<filename>";
    only the LLM-generated summary is indexed, so filenames cannot
    leak back into RAG retrieval context.
  * Wiki summary slug is now derived from the knowledge ID rather than
    the upload filename, eliminating filename-based slugs like
    "summary/mx5280-pdf" that downstream wiki-edit prompts can read.
  * Citation sourceRef likewise drops the "|<docTitle>" suffix.
  * Dead docTitle parameter removed from extractCandidateSlugs,
    extractEntitiesAndConceptsNoUpsert, classifyChunkCitations.
  * The redundant stripImageMarkup pass in the warning log is gone;
    realTextRuneCount caches the count once and uses
    utf8.RuneCountInString to avoid the rune-slice allocation.

Wiki-page display titles still flow through DocTitle (filename) into
later wiki-edit / index-rebuild prompts. That is a larger refactor of
the wiki pipeline and is intentionally left for a separate PR.
2026-05-06 11:23:32 +08:00
Claude 15550d14d4 fix(ingest): stop hallucinating summaries from filenames on empty PDFs
Scanned PDFs without a successful OCR pass (e.g. when no VLM is configured
on the KB, or when the VLM returned a "no text content" reply that the
sanitizer discarded) reach the summary and wiki-ingest pipelines with
nothing but markdown image references for content. The downstream LLM
prompts used to receive the filename, file type, and title alongside that
empty content, and the generate_summary template explicitly forbade the
model from refusing — so the model would invent a topic from the filename
alone, producing e.g. a "Canon scanner manual" summary for an MX5280.pdf
that is actually a scanned legal letter.

Two changes shut this down:

1. Drop filename / title / file-type from every LLM prompt input. The
   wiki summary, knowledge extract, candidate slug, and chunk citation
   prompts now see only the document content. The knowledge summary
   path no longer prepends the "Document Type: X / File Name: Y" intro.

2. Add a content-sufficiency guard in front of the LLM call. A new
   helper strips markdown image references and HTML image tags and
   refuses to invoke the model when fewer than 50 real-text runes
   remain. getSummary returns a sentinel error in that case and the
   caller marks the knowledge as SummaryStatusFailed instead of
   surfacing the first chunk's bare image reference. Wiki ingest skips
   LLM extraction with a clear log message.

The generate_summary.yaml prompt is also updated: the "NEVER output
refusal phrases" clause is removed and replaced with an explicit
empty-content rule that returns a deterministic placeholder line.

Real OCR (Tesseract / PaddleOCR) and filename-pattern sanitisation are
intentionally left for separate PRs.
2026-05-06 11:23:32 +08:00
wizardchen 1f03741462 fix(wiki): Lite ingest lock, failed-op requeue, sync task retry parity
- Add per-KB sync.Map lock in Lite mode to mirror Redis SetNX concurrency.
- Requeue failed wiki ops in Lite by enqueueing asynq tasks with delay/tracing.
- Teach SyncTaskExecutor to honor ProcessIn delay and MaxRetry like Redis asynq.
2026-04-28 18:27:00 +08:00
wizardchen c34f7b6254 Enhance wiki ingest process to handle failed operations
- Introduced a mechanism to track and requeue failed operations during the wiki ingest process.
- Added a new `requeueFailedOps` function to append failed operations back to the Redis pending list for retry in subsequent batches.
- Updated the `ProcessWikiIngest` method to collect failed operations and ensure they are retried after trimming the pending list.

This change improves the robustness of the ingest process by preventing data loss from transient failures.
2026-04-28 17:21:42 +08:00
wizardchen 13260b831c Fix wiki ingest silent data loss from malformed JSON in Redis queue
Two critical fixes to the peekPendingList function:

1. Add error logging for JSON unmarshal failures (line 303)
   - Previously: Malformed JSON items were silently skipped with no indication
   - Now: Each failure is logged with the error and raw item content (first 100 chars)
   - This makes data loss visible and debuggable

2. Fix trim count mismatch (line 325)
   - Previously: Returned len(result) - the count of raw Redis items peeked
   - Now: Returns len(ops) - the count of successfully parsed items
   - Impact: Malformed items now stay in Redis for retry instead of being discarded
   - This prevents silent data loss when JSON encoding issues occur

Root cause of the reported issue:
- When 8 files are uploaded in a batch, if one has encoding issues that cause
  JSON.Unmarshal to fail, it was silently dropped from processing
- The file was trimmed from Redis without being processed
- Result: Only 7 of 8 files appeared in the Wiki

Fix approach:
- Make unmarshal failures visible in logs
- Retry malformed items in the next batch instead of discarding them
- Admin can see the errors and take corrective action

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:42 +08:00
wizardchen beca2b89a3 feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.

This change threads one trace end-to-end:

- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
  worker can attach to an upstream trace without emitting a duplicate
  trace-create; StartGeneration now auto-picks parentObservationId from
  ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
  payloads so trace_id / parent_obs_id / user_id / session_id serialise
  into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
  so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
  resumes the upstream trace or opens a standalone asynq.<type> trace
  for scheduled jobs, and wraps the handler in a SPAN with task metadata
  (id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
  FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
  ResumeTrace no-trace-create guarantee; asynq_test.go covers
  InjectTracing round-trip, middleware resume path, and standalone
  trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
  and documents the cross-process propagation model.

No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 13:16:47 +08:00
wizardchen c5494f612b feat: Implement auto-linked content updates in wiki page management
- Introduced UpdateAutoLinkedContent method to persist changes from machine-only link decorators without incrementing the version number, ensuring user-facing revisions remain consistent.
- Updated existing methods to utilize the new auto-linked content update logic, enhancing the handling of cross-link injection and dead-link cleanup.
- Enhanced the KnowledgeBase and WikiBrowser components to improve user experience during wiki operations, including status tracking and indexing indicators.
- Added scheduling for wiki status probes to provide timely feedback on indexing processes after user-triggered actions.
2026-04-23 11:16:47 +08:00
wizardchen a8a3d9e694 feat: Enhance wiki extraction and configuration options
- Introduced extraction granularity settings in the wiki configuration, allowing users to control the level of detail in entity and concept extraction (focused, standard, exhaustive).
- Updated the UI to reflect these new options, including tooltips for better user guidance.
- Refactored the knowledge base editor to support the new extraction granularity feature, ensuring a seamless user experience.
- Improved the backend logic for wiki ingest tasks to utilize the new granularity settings, enhancing the accuracy of extracted content.
- Added tests to validate the new extraction granularity functionality and ensure robust performance across different configurations.
2026-04-22 21:18:32 +08:00
wizardchen fdccb68ee4 feat: Improve wiki cleanup and ingest handling for deleted knowledge
- Enhanced the `cleanupWikiOnKnowledgeDelete` function to pass the full knowledge object, allowing for more accurate title and summary retrieval during wiki cleanup.
- Introduced a tombstone mechanism in Redis to mark recently deleted knowledge, enabling in-flight wiki ingest tasks to skip processing for deleted documents.
- Updated the `isKnowledgeGone` function to check for deleted knowledge using the tombstone, improving efficiency in handling concurrent delete and ingest operations.
- Added logic in the `wikiIngestService` to filter out updates for deleted knowledge, ensuring that no ghost references are created during the ingest process.
- Implemented lint checks for stale source references in the `WikiLintService`, allowing for automatic cleanup of pages referencing deleted knowledge.

These changes significantly enhance the robustness of the wiki system, ensuring consistency and accuracy in handling knowledge deletions and related wiki pages.
2026-04-22 21:18:20 +08:00
wizardchen 53ecd5df5d feat: Enhance wiki content processing with enriched image information and XML output
- Introduced `enrichChunkImageInfo` function to populate image information for text chunks by retrieving data from child chunks, ensuring that image-related content is not lost during processing.
- Updated `reconstructContent` to `reconstructEnrichedContent`, allowing for the inclusion of OCR text and captions from images, improving the quality of content sent to the LLM.
- Added `writeDedupItemXML` and `xmlEscape` functions to format entity and concept entries as structured XML, enhancing clarity and preventing nonsensical merges in deduplication prompts.
- Refined the `deduplicateExtractedBatch` method to utilize the new XML formatting for better output consistency.

These changes significantly improve the handling of image data and the overall clarity of deduplication prompts, contributing to a more effective wiki content management system.
2026-04-22 21:18:09 +08:00
wizardchen 173547a17a feat: Refactor cross-link injection logic for wiki pages
- Simplified the `InjectCrossLinks` and `injectCrossLinks` methods by utilizing a shared `linkifyContent` function, ensuring consistent handling of code blocks, existing links, and word boundaries.
- Replaced manual reference collection and sorting with a dedicated `collectLinkRefs` function to streamline the process of gathering link references from wiki pages.
- Enhanced the overall readability and maintainability of the code by removing redundant structures and logic.

These changes improve the efficiency and clarity of the cross-linking functionality within the wiki service, contributing to a more robust content management system.
2026-04-22 21:18:09 +08:00
wizardchen a75233c90f refactor: Enhance deduplication logic and logging in wiki ingestion service
- Updated the `WikiDeduplicationPrompt` to clarify the criteria for identifying exact duplicates, emphasizing strict matching of real-world entities and concepts.
- Improved the `appendLogEntry` method to include a summary parameter for better context in log entries, enhancing traceability of operations.
- Consolidated log entry creation for both retraction and ingestion operations, ensuring chronological clarity in the log.
- Implemented validation for slug merges to prevent mismatches based on entity types, improving the accuracy of deduplication.

These changes enhance the clarity and reliability of the wiki ingestion process, leading to more accurate content management.
2026-04-22 21:17:34 +08:00
wizardchen 28781785f7 refactor: Optimize wiki ingestion process by consolidating entity and concept deduplication
- Introduced a new `WikiBatchContext` to share data across Map and Reduce phases, reducing redundant database queries.
- Replaced separate deduplication calls with a single `deduplicateExtractedBatch` method, improving efficiency by leveraging pre-loaded page data.
- Enhanced JSON handling by implementing a `cleanLLMJSON` function to sanitize LLM-generated output, ensuring safe parsing.
- Updated relevant methods to utilize the new context and deduplication logic, streamlining the overall ingestion workflow.

These changes enhance the performance and clarity of the wiki ingestion service, leading to more efficient content processing.
2026-04-22 21:17:33 +08:00
wizardchen fba6d7d145 refactor: Update wiki prompts and ingestion logic for improved content handling
- Renamed `WikiPageUpdatePrompt` to `WikiPageModifyPrompt` to better reflect its functionality of adding new information and removing outdated content in a single operation.
- Enhanced the instructions for both `WikiPageModifyPrompt` and `WikiPageRetractPrompt` to clarify the handling of additions and retractions, ensuring accurate content updates.
- Consolidated the logic for enqueuing retraction tasks in the `cleanupWikiOnKnowledgeDelete` method to streamline the process of managing affected pages.
- Removed the `rebuildWikiIndexSimple` function to simplify the codebase, as its functionality is no longer needed with the new ingestion approach.

These changes improve the clarity and efficiency of the wiki content management process, enhancing overall system performance.
2026-04-22 21:17:31 +08:00
wizardchen 8daf6b368c feat: Add pending tasks and active status to WikiStats
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.

These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
2026-04-22 21:17:31 +08:00
wizardchen 2b495c8232 refactor: Update wiki ingest service to support aliases in slug mapping
- Modified the `extractEntitiesAndConcepts` method to return a map of slugs to extracted items, including aliases for improved wiki-link generation.
- Updated the `processOneDocument` method to handle slug items with aliases, enhancing the clarity of generated summaries.
- Adjusted the `injectCrossLinks` method to utilize match text for cross-linking, ensuring accurate references in content.

These changes improve the accuracy and usability of wiki links and summaries by incorporating alias management.
2026-04-22 21:17:30 +08:00
wizardchen d9cf4b7aa9 feat: implement metadata-based alias management for wiki pages
Made-with: Cursor
2026-04-22 21:17:29 +08:00
wizardchen 98e2464831 refactor: Enhance wiki ingest service with knowledge service integration and content sorting
- Introduced a new `knowledgeSvc` interface to the `wikiIngestService`, allowing for improved document title retrieval based on knowledge ID.
- Updated the document processing logic to prioritize titles from the knowledge service, enhancing accuracy.
- Refactored content reconstruction to sort chunks by `StartAt` and `ChunkIndex`, improving the organization of the final output.
- Added functionality to append image information at the end of the content, ensuring a cleaner text flow.

These changes improve the efficiency and clarity of the wiki ingest process, enhancing overall content management.
2026-04-22 21:17:27 +08:00
wizardchen 5d5073c96b refactor: Update wiki link syntax and rendering in WikiBrowser component
- Enhanced the `renderMarkdown` function to support the new `[[slug|display name]]` format for wiki links, improving link clarity and usability.
- Adjusted CSS styles for wiki links to provide a more intuitive hover effect and visual distinction.
- Updated prompts in the `prompts_wiki.go` file to reflect the new wiki link syntax, ensuring consistency across the application.

These changes improve the overall user experience by making wiki links more informative and visually appealing.
2026-04-22 21:17:26 +08:00
wizardchen 7eea22d655 refactor: Enhance wiki link generation in summary processing
- Updated the WikiSummaryPrompt to clarify the wiki-link syntax, specifying the use of slugs for referenced names.
- Modified the extractEntitiesAndConcepts method to return a mapping of slugs to display names, improving the accuracy of wiki-link references in summaries.
- Adjusted the summary content generation to include display names alongside slugs, enhancing the readability and context of generated summaries.

These changes improve the clarity and usability of wiki summaries by ensuring accurate and consistent link references.
2026-04-22 21:17:25 +08:00
wizardchen ec8a5fc392 refactor: Enhance wiki page management and index rebuilding
- Introduced a new method to list all wiki pages without pagination, facilitating index rebuilding and cross-linking.
- Updated the wiki ingest and retract processes to include document summaries, improving clarity in log entries and index updates.
- Enhanced the cleanup process for deleted knowledge, ensuring accurate summary retention and index regeneration.
- Streamlined the handling of page updates and deletions, optimizing the overall wiki management workflow.

These changes improve the efficiency and reliability of wiki content management and indexing.
2026-04-22 21:17:24 +08:00
wizardchen f00b2a176b refactor: Update EnqueueWikiIngest to utilize Redis for pending document management
- Modified the EnqueueWikiIngest function to accept a Redis client, enabling the use of a Redis list for managing pending wiki ingest documents.
- Introduced a debouncing mechanism to optimize task processing, ensuring that multiple rapid uploads are handled efficiently without unnecessary locks.
- Enhanced the overall architecture for better performance and reliability in handling wiki ingest tasks.

These changes streamline the ingestion process and improve the handling of concurrent document uploads.
2026-04-22 21:17:24 +08:00
wizardchen 3edd7548f4 refactor: Update step numbering in wiki_ingest service process 2026-04-22 21:17:23 +08:00
wizardchen 5e113ddba3 refactor: Remove detectSynthesisOpportunities function from wiki_ingest service
- Eliminated the detectSynthesisOpportunities function, which was responsible for suggesting synthesis opportunities based on page counts and types. This function was deemed unnecessary as it relied on heuristic methods without LLM integration.
- The removal streamlines the codebase and focuses on essential functionalities within the wiki_ingest service.

This change simplifies the service and prepares it for future enhancements that may involve more sophisticated synthesis logic.
2026-04-22 21:17:23 +08:00
wizardchen 4dff109d1b fix: Update wiki index rebuilding and log entry formatting
- Modified the `rebuildWikiIndexSimple` function to include `knowledgeID` in the parameters, ensuring accurate logging of deleted pages.
- Enhanced the log entry format in `appendLogEntry` to include the action type and improved timestamp formatting, providing clearer context for actions taken on wiki pages.
- Adjusted the `ProcessWikiRetract` and `ProcessWikiIngest` methods to align with the new log entry structure, ensuring consistency across logging actions.

These changes improve the clarity and accuracy of wiki activity logs, facilitating better tracking of content changes.
2026-04-22 21:17:23 +08:00
wizardchen 0e28b2448c feat: Add arrow toggle functionality in WikiBrowser component
Implemented new features in the WikiBrowser component to allow users to toggle the visibility of arrows on graph edges. This includes:

1. **Localization Updates**: Added translations for "Show arrows" and "Hide arrows" in English, Korean, Russian, and Chinese.
2. **Graph Rendering Enhancements**: Introduced logic to dynamically update SVG line attributes based on the toggle state, improving visual clarity in graph representations.
3. **State Management**: Added reactive state management for arrow visibility, ensuring seamless user interaction.

These changes enhance the user experience by providing more control over graph visualizations.
2026-04-22 21:17:22 +08:00
wizardchen d4d005422d feat: Implement wiki_config updates in knowledge base management
Added functionality to enable updates for the `wiki_config` field in the `KnowledgeBaseConfig` struct. This includes:

1. **Type Definition Update**: Added `WikiConfig` field to `KnowledgeBaseConfig` to allow JSON unmarshaling during update requests.
2. **Service Layer Enhancement**: Updated the `UpdateKnowledgeBase` method to persist changes to `wiki_config` when provided in the request.
3. **Frontend TypeScript Adjustments**: Enhanced type definitions in the frontend to support `wiki_config`, improving IDE autocomplete and type checking.

This fix addresses the critical issue where `wiki_config` was not being updated due to its absence in the `KnowledgeBaseConfig` struct, ensuring proper handling of wiki configurations during knowledge base updates.
2026-04-22 21:17:21 +08:00
wizardchen 16a0bff7bd feat: Enhance wiki functionality with recent activity log
This commit introduces a new "Recent Activity" section in the WikiBrowser component, allowing users to view the latest changes made to the wiki. The implementation includes:

- Addition of a toggleable recent activity log displaying the last five entries.
- Backend support for fetching the activity log from the knowledge base.
- Updates to the wiki ingest process to ensure draft pages are published post-ingest.
- Improvements to the deduplication process for entities and concepts during wiki page creation.

These enhancements aim to improve user engagement and provide better insights into wiki updates.
2026-04-22 21:17:19 +08:00
wizardchen 095ccaed10 refactor: Replace hardcoded language logic with middleware infrastructure
- Refactor wiki_ingest.go to use types.LanguageLocaleName() instead of hardcoded if-else
- Supports 9+ languages (Chinese, English, Korean, Japanese, Russian, French, German, Spanish, Portuguese)
- Reduces 7 lines to 4 lines (43% reduction in code)
- Eliminates code duplication by reusing existing middleware infrastructure
- Consistent naming with middleware layer ("Chinese (Simplified)" format)
- Add comprehensive unit tests for LanguageLocaleName() (30 test cases)
- Add implementation guide, analysis, and completion documentation

Benefits:
- Single source of truth for language mappings (types.LanguageLocaleName)
- Better code maintainability and consistency
- No breaking changes or schema modifications required
- Performance: ~4.3ns per call, no measurable impact
- All tests passing, build successful

Files modified:
- internal/application/service/wiki_ingest.go: Refactored language determination (line 139)
- internal/types/context_helpers_test.go: Added 30 unit test cases (NEW)
- Documentation: Added LANGUAGE_MIDDLEWARE_ANALYSIS.md, LANGUAGE_REFACTORING_COMPLETED.md, IMPLEMENTATION_GUIDE.md, REFACTORING_SUMMARY.txt

Refs: LANGUAGE_MIDDLEWARE_ANALYSIS.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:17 +08:00