Commit Graph

169 Commits

Author SHA1 Message Date
wolfkill 8f4e5a459f fix(session): scope session access by user 2026-05-14 15:56:43 +08:00
wizardchen cacca049d9 feat(knowledge-base): document list filters and explicit batch-management UX
Add three new optional filters to the document list under a knowledge base
detail page — parse status, source/channel, and updated time range — and
rework multi-select to no longer cause the card title to jitter on hover.

Backend
- Introduce types.KnowledgeListFilter to aggregate optional filter dimensions
  (tag, keyword, file_type, parse_status, source, updated_from/to) and switch
  ListPagedKnowledgeByKnowledgeBaseID (repository/service/interface) to accept
  it instead of a growing positional parameter list.
- The ListKnowledge HTTP handler accepts new parse_status, source, start_time
  and end_time query params; time params accept RFC3339, "YYYY-MM-DD HH:MM:SS"
  and "YYYY-MM-DD". The repository routes source="manual"/"url" onto the type
  column to stay consistent with file_type semantics; other source values match
  the channel column.
- Update the four other callers (agent_service, initialization) to pass an
  empty filter struct, preserving prior behavior.

Frontend
- Add three controls in the doc-filter-bar (status select, source select,
  date-range picker with future-date disabled) wired through getKnowled /
  listKnowledgeFiles into the new backend params.
- Replace the hover-triggered card checkbox with an explicit "批量管理" mode
  (mirrors the session list UX): in card view the checkbox only renders while
  batch mode is on, entered via the per-card "..." menu; the list view keeps
  its leading checkbox column. Switching from list to grid auto-enables batch
  mode when something is already selected, so the selection stays visible.
- DocumentBatchBar now stays open whenever batch mode or selection > 0, and
  its "取消选择" button both clears the selection and exits batch mode.

API surface sync
- Regenerate Swagger artifacts (docs/docs.go / swagger.json / swagger.yaml).
- Update docs/api/knowledge.md with the new query parameters.
- Add backward-compatible ListKnowledgeWithFilter + KnowledgeListFilter to the
  Go SDK; the existing ListKnowledge keeps its signature.

i18n
- New filter labels in zh-CN / en-US / ko-KR / ru-RU; reuse existing
  menu.batchManage / batchManage.cancel for the multi-select strings.
2026-05-12 18:29:47 +08:00
nullkey 4ed7b90eac fix(faq): correct PostgreSQL cast precedence in FAQ search
`metadata->'similar_questions'::text` parses as
`metadata -> ('similar_questions'::text)` because `::` binds tighter
than `->`, so the expression yields jsonb instead of text. ILIKE on
jsonb then fails with "operator does not exist: jsonb ~~* unknown",
returning 1007 Internal server error for `search_field=similar_questions`
and `search_field=answers`. The `standard_question` branch was unaffected
because it uses `->>` (text) directly.

Wrap the json access in parens so the cast applies to the extracted
value: `(metadata->'similar_questions')::text ILIKE ?`. The MySQL
branch uses JSON_EXTRACT and is unchanged.

Fixes #1264
2026-05-12 16:42:21 +08:00
wizardchen a082c04d28 chore(deps): update dependencies in /docreader and adjust dependabot configuration
- Updated `pydantic` from 2.12.3 to 2.13.4 and `pypdfium2` from 5.0.0 to 5.8.0 in the `docreader` requirements.
- Modified the dependabot configuration to set `open-pull-requests-limit` to 0 and added an `ignore` rule for version updates across all ecosystems, allowing only security updates.
- Adjusted settings for `server-security`, `client-security`, `frontend-security`, and `miniprogram-security` groups to streamline security update handling.

This change aims to enhance dependency management and maintain security while reducing noise from version update PRs.
2026-05-12 14:58:52 +08:00
wizardchen 6d0f09d75f fix(agent): 加固 MCP 工具人审批的跨实例与并发语义
- 跨实例 Resolve 通过 Redis reply channel 接收 owning 实例的 ack
  (ok / not_found / tenant_mismatch / user_mismatch / already_resolved),
  HTTP 不再静默 200,超时退化为 NotFound。
- PendingRequest 增加 UserID,Resolve 校验会话所有者,避免同租户
  其他用户越权批准/篡改参数。
- 新增 ErrAlreadyResolved,timer/ctx 命中后再次 Resolve 返回 400 而
  不是静默成功;waiter.deliver 暴露胜出语义。
- NeedsApproval 默认 fail-close(DB 故障要求审批),可经
  WEKNORA_AGENT_TOOL_APPROVAL_FAIL_OPEN=true 回退。
- Pub/Sub 增加 OriginID 自过滤、退避重连、namespace 后缀,避免自回
  环噪声与多部署串扰。
- 工具执行超时常量统一通过 ToolExecContext.ExecTimeout 注入,
  mcp_tool.go 不再硬编码 60s。
- handler 修正 path 二次解码、错误用 errors.Is、modified_args=null 的
  静默清空、ListMCPToolApprovals 的 404/500 误判。
- repository 用 clause.OnConflict 做原子 upsert,消除并发 500。
- 增加 NotFound / TenantMismatch / UserMismatch / AlreadyResolved /
  Race 等用例。
2026-05-10 22:57:12 +08:00
wizardchen 5510ea8f5a feat(agent): human-in-the-loop approval for MCP tool calls (#1173)
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.

Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
  required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
  delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
  replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
  (without the per-tool 60s timeout) for the wait, and re-derives a
  fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
  EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
  WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
  duration).

Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
  args, validation feedback, mm:ss countdown that turns warning/danger
  near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.

Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
  considerations (Redis cross-instance, restart limitations).
2026-05-10 22:57:12 +08:00
langcaiye 02be05c095 feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
wizardchen 82947c726e fix(wiki-ingest,asynqdl): review fixups for PR #1241
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
  trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
  items" had two args), and harden validMerge against un-prefixed
  slugs whose strings.Index returned -1 and silently passed the type
  check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
  ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
  letter row instead of a hard-coded 0; tighten payloadProbe to the
  set of field names with consistent semantics across payloads
  (drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
  outside an asynq worker ctx so the semantics stay pinned.
2026-05-10 00:01:06 +08:00
wizardchen 04e8105728 perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.

This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.

  - migration 000041: task_pending_ops + task_dead_letters tables, plus
    three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
    source_refs text fallback, lower(title) trgm).
  - TaskPendingOpsRepository + TaskDeadLetterRepository in
    internal/application/repository/task_queue.go: cursor-paginated
    list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
    delete that refuses empty keys so a buggy caller can't wipe a KB's
    queue.
  - internal/middleware/asynqdl: writes a task_dead_letters row when an
    asynq task exhausts its retry budget. Payload-agnostic — a small
    probe struct extracts TenantID + scope hints across every existing
    payload type, so summary:generation / image:multimodal /
    faq_import / kb:clone / etc. all dead-letter without per-handler
    code. Best-effort: an insert failure never masks the underlying
    task error. Installed in router/task.go before the langfuse mw so
    it sees raw errors.

  - Pending queue moved Redis → PG (no TTL, restart-durable). Redis
    keeps just the active-batch lock and the delete tombstone.
  - In-batch retry budget (wikiMaxFailRetries=5) tracked via
    pendingRepo.IncrFailCount; over the cap the row is moved to
    task_dead_letters with task_type=wiki:ingest, related_id=knowledge
    id. Asynq retries (10) are handled by the new middleware.
  - Removed the per-batch ListAllPages. WikiBatchContext now carries
    lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
    mutex-protected caches; reduce reaches for titles / summaries on
    demand instead of pre-loading the whole KB.
  - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
    which the new GIN index on source_refs serves as a Bitmap Index
    Scan instead of a sequential text LIKE.
  - Dedup pre-filter uses pg_trgm via FindSimilarPages
    (idx_wiki_pages_title_trgm). For each new entity/concept (and each
    of its aliases) we ask the DB for the top-K trigram-similar
    existing pages and union the results — bounded prompt size, no Go-
    side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
  - cleanDeadLinks / injectCrossLinks are scoped to the batch's
    affected slugs (a few dozen) instead of every page in the KB. Both
    use the new lite ListBySlugs / ExistsSlugs repo methods so they
    pull only slug + title + outlinks, not full content.
  - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
    [[bad-slug|display]] the cleanup path now tries display-text
    reverse lookup → hyphen/case normalized equality → char-bigram
    Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
    break drift case ("shang-hai-tower" vs "shanghai-tower") in place
    instead of replacing the link with plain text.
  - rebuildIndexPage uses ListByTypeRecent(200) for the first-time
    intro and drops the full DocumentSummaries blob from the
    incremental update prompt, so its context stays bounded regardless
    of KB size.
  - Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
    IngestMapParallel / IngestReduceParallel, with sensible defaults
    via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
    follow-ups don't waste asynq retry slots bouncing on the active
    lock.

  - RunLint walks pages via the new ListPagesCursor in 200-page
    windows and computes the live-slug set with a one-column
    ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
    every node + edge. Memory is now bounded; 40k pages walks in
    constant ~4MB.

  - 14 GORM tests for both repos against an in-memory SQLite mirror of
    the production DDL (task_queue_test.go).
  - 6 tests for the dead-letter middleware covering retry budget
    detection, payload-agnostic scope inference, error truncation, and
    repo-failure isolation.
  - 7 tests for the slug fuzzy resolve helper covering the three
    resolution stages, display-text priority over normalized equality,
    bigram fallback acceptance, and rejection of unrelated slugs.
  - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
    new fetcher / cursor APIs.
2026-05-10 00:01:06 +08:00
langcaiye 74b1342440 feat: add Tencent VectorDB retriever backend 2026-05-09 13:14:01 +08:00
wizardchen fc6f160eff fix(retriever/doris): code review cleanup
针对 4cce6f2e(接入 Apache Doris)的 code review 修复,主要修正若干阻断性
问题与可读性问题,并剔除不应进入主仓的本地工作流文件。

阻断性修复:
- docker-compose: Doris 镜像由 2.1.0 升至 4.1.0。原 2.1.0 不支持 HNSW
  ANN、cosine_distance_approximate 与 Stream Load partial_columns,
  按当前 DDL 一启动就会失败。
- DSN 字面量拼接改用 mysql.Config.FormatDSN()。原 fmt.Sprintf 在用户名/
  密码包含 `@`/`:`/`/` 等字符时会跑偏。覆盖 health check 与 engine
  factory 两处。

健壮性修复:
- 新增 validateEmbedding,写入与查询前拒绝 NaN/±Inf;strconv.FormatFloat
  对非有限值会输出 "NaN"/"+Inf" 拼成无效 SQL。
- waitANNReady 改为后台 goroutine + 独立 context,避免新维度首次写入路径
  阻塞最长 30s;ANN 未就绪时 Doris 会自动退化为 brute-force。

清理:
- annIndexReady 移除最终两个分支都 return true 的冗余写法。
- Save 移除冗余的双重 toDorisVectorEmbedding。
- testDorisConnection 把 "5.7.99 Doris-4.1.0" 解析为裸 "4.1.0",与
  Postgres/ES 的版本格式对齐。

剔除(不应合入主仓):
- docs/wiki/集成扩展/Doris改动与上游同步.md:纯 fork 维护工作流文档。
- scripts/e2e-doris.sh:作者本地 E2E 验证清单。

测试:
- repository_test 用 require.Eventually 适配 ANN 异步轮询。
- 现有 doris 单测全部通过。
2026-05-09 00:31:03 +08:00
issunion 4cce6f2e99 feat(retriever): 接入 Apache Doris 4.1 作为向量数据库
为 RetrieveEngine 体系新增 Doris 后端,与现有 Qdrant/Milvus/Weaviate
等保持完整能力对齐:向量检索、关键词检索、健康检查、环境变量与多实例
DB 配置、前端类型注册、单元测试、Docker Compose 模板。

实现要点:
- 协议分工:主链路用 MySQL 协议(database/sql + go-sql-driver/mysql)
  做 DDL / 查询 / 删除;批量更新走 Stream Load HTTP API,并启用
  partial_update=true、merge_type=APPEND,自动按 1MiB 切分批次并处理
  307 重定向。
- 表结构:UNIQUE KEY(id) + enable_unique_key_merge_on_write=true 以
  支持 upsert/部分列更新;按维度分表(<base>_<dim>),每张表上建
  HNSW ANN 索引(metric_type=cosine_distance)和 INVERTED 索引
  (parser=chinese)。
- 分数语义:使用 cosine_distance_approximate,再以 1 - dist 转换为
  "越大越相似",与现有 KVHybridRetrieveEngine 约定一致。
- 异步索引:ANN 索引为后台构建,ensureTable 通过轮询 SHOW INDEX 等
  待索引就绪后再放行写入,避免首次检索召回为空。
- ARRAY<FLOAT> 序列化:go-sql-driver/mysql 不支持数组占位符,
  embeddingLiteral 将 []float32 转成 SQL 字面量字符串再拼接。

新增文件:
- internal/application/repository/retriever/doris/{structs,schema,
  query,repository,streamload,repository_test}.go
- scripts/e2e-doris.sh:E2E 验证清单
- docs/wiki/集成扩展/Doris改动与上游同步.md:fork-and-rebase 工作流
  与改动清单

修改文件(接线 + 文档):
- internal/types/{retriever,tenant,vectorstore}.go:新增
  DorisRetrieverEngineType、env 解析、表单 schema 与索引参数校验
- internal/container/{container,engine_factory}.go:环境变量驱动
  与 VectorStore 配置驱动两条路径都支持 Doris
- internal/application/service/vectorstore{,_healthcheck}.go:连接
  校验 + Ping/Version 健康检查
- docker-compose.yml:新增 doris-fe / doris-be 服务(profile=doris)
- .env.example:DORIS_* 环境变量与示例
- docs/{使用其他向量数据库,wiki/集成扩展/集成向量数据库}.md:
  使用说明与索引/分数行为说明

依赖:go.mod/go.sum 新增 github.com/go-sql-driver/mysql(运行时)和
github.com/DATA-DOG/go-sqlmock(测试)。

测试:repository 层 SQL 形状、Stream Load HTTP 行为、whereBuilder
逻辑、embeddingLiteral 往返、健康检查错误路径均有单测覆盖。
2026-05-08 21:59:35 +08:00
wizardchen d0144f3586 perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:48:52 +08:00
wizardchen df1e9a1bfd perf(wiki): subgraph API and interactive exploration for large graphs
Wikis with tens of thousands of pages used to crash the browser trying
to render the entire link graph at once (30MB+ JSON, 100k+ SVG elements).
This change moves the graph viewer from "fetch everything, render
everything" to "fetch a slice, expand on demand" with several matching
UX improvements.

Backend

* GET /wiki/graph now accepts `mode` (overview | ego), `center`, `depth`,
  `types`, `limit` query params. Default is overview top-500 by
  link_count, capped at 2000. Response includes a Meta object so the
  frontend can render a truncation hint and drive the UI.
* Pure helper `computeGraphSubset` extracted for testability; six unit
  tests cover overview truncation / type filter / ego BFS / missing
  center error.
* WikiLintService passes Limit=0 (uncapped) so link integrity checks
  still walk every page.
* Repo `Search` adds CASE-based relevance ranking (title 4, slug 3,
  summary 2, content 1) so full-text results put the most obvious
  matches first instead of whatever was updated most recently.

Frontend

* Graph viewer fetches overview on entry; ego pivot on double-click,
  search, wiki-link click, URL ?slug=, global issues jump.
* Shift+click or hover ⊕ button blooms neighbors onto the current
  canvas additively. Bloom tracks generations and LRU-evicts oldest
  when total exceeds 1500 nodes; ego center / selected node / latest
  anchor are always protected.
* "Grow frontier (N)" legend action expands every dashed-ring node in
  one click (6-way concurrency) while skipping Index/Log super-nodes
  that would otherwise dump the whole wiki onto the canvas.
* Node dashed expansion ring + drawer "X/Y neighbors shown" hint tell
  users which nodes still have neighbors to load.
* Search dropdown uses remote full-text search (debounced, sequence-
  numbered to drop stale responses) and falls back to the overview
  top-500 snapshot when the input is empty.
* Type filters now round-trip to the server so top-N is always
  computed from the user's active type set rather than hiding nodes
  client-side and shrinking the view.
* Status card replaces the cramped "centered on X · N hops · M nodes"
  line with a structured focus/overview summary, resolving slugs to
  page titles.
* Help popover in the legend lists every canvas shortcut.

i18n entries added for zh-CN / en-US / ko-KR / ru-RU.
2026-05-08 13:32:54 +08:00
wizardchen fe0d24ae87 fix(sessions): review fixes for keyword search / pinning / IM titles
- Apply the session-list additions to the SQLite init migration (Lite build)
  and make QueryPaged dialect-aware: LOWER(..) LIKE on SQLite, ILIKE on
  Postgres, drop NULLS LAST on SQLite. Escape LIKE wildcards in the keyword
  via the existing escapeLikeKeyword helper.
- SetPinned now returns rowsAffected so the handler can respond 404 on
  unknown / unauthorized session IDs instead of a misleading 200.
- GET /sessions always returns the enriched shape (pin state + IM origin
  fields) so the frontend never needs a second roundtrip; the dual-path
  legacy branch in the handler is gone.
- Register pin routes with the wildcard name that matches each verb's
  existing radix tree (POST :session_id, DELETE :id) and accept either
  param name in the handler; avoids gin's "wildcard conflicts" panic.
- Drop the redundant [platform] prefix from IM session titles now that
  the list renders a platform icon alongside the title; add unit tests
  for shortID / buildUserSessionTitle / buildThreadSessionTitle.
- Frontend: remove the submenu search input and its i18n/keyword wiring
  (search lives elsewhere in the app); pin icon uses the TDesign `pin`
  glyph and inherits color so the active session turns green; optimistic
  pin moves the item to the top of the list so it shows up at the top
  of the Pinned group; IM text badge replaced by the platform SVG icons
  from assets/img/im, desaturated by default and full-color on hover
  or when the session is active.
2026-04-30 16:23:00 +08:00
wizardchen dbd804d6e3 feat(sessions): add keyword search, user-scoped pinning, and IM source visibility
Sessions today are a flat list ordered by updated_at. Two gaps showed up in
practice:
- Users cannot find specific chats as the list grows beyond a screen.
- IM-created sessions (WeCom/Feishu/Slack/...) are indistinguishable:
  every title was "IM-<platform>" or "IM-<platform>-<username>" and the
  list API hid the underlying im_channel_sessions mapping, so admins had
  no way to tell which Feishu group a session came from.

Backend
- Migration 000039 adds sessions.user_id (owner), is_pinned, pinned_at
  plus a composite index for the list query. Existing rows keep user_id
  NULL and stay visible at the tenant level for backward compatibility.
- CreateSession now writes the caller's user_id from auth context.
- GET /sessions accepts keyword / source / agent_id. When any filter is
  set, the response switches to enriched items that LEFT JOIN
  im_channel_sessions and expose im_platform / im_chat_id / im_thread_id
  / im_user_id / im_agent_id / im_channel_id. No filters => legacy shape,
  existing clients unaffected.
- Ordering: is_pinned DESC, pinned_at DESC NULLS LAST, updated_at DESC.
- POST/DELETE /sessions/:id/pin for user-scoped pin/unpin.
- IM session titles: "[platform] <user|chat|thread>" with short ID
  suffixes so group/DM/thread sessions are visually distinct without
  needing a round-trip to fetch a display name from the IM adapter.

Frontend
- Search input debounced at 300ms drives the keyword filter.
- Pinned chats render in a dedicated group above the time-based groups,
  with a pin icon and a pin/unpin entry in the per-chat dropdown.
- IM chats get a short [platform] badge in the list.
- Pin toggle is optimistic and guards against double-clicks.
- zh/en/ko/ru i18n keys added for the new strings.
2026-04-30 16:23:00 +08:00
wizardchen c5494f612b feat: Implement auto-linked content updates in wiki page management
- Introduced UpdateAutoLinkedContent method to persist changes from machine-only link decorators without incrementing the version number, ensuring user-facing revisions remain consistent.
- Updated existing methods to utilize the new auto-linked content update logic, enhancing the handling of cross-link injection and dead-link cleanup.
- Enhanced the KnowledgeBase and WikiBrowser components to improve user experience during wiki operations, including status tracking and indexing indicators.
- Added scheduling for wiki status probes to provide timely feedback on indexing processes after user-triggered actions.
2026-04-23 11:16:47 +08:00
wizardchen ffc4eeaf57 feat: Refactor wiki page update logic for improved version control
- Enhanced the UpdatePage and UpdateMeta methods to differentiate between user-visible content changes and bookkeeping updates, ensuring the version number is only incremented for actual content modifications.
- Updated the WikiPage struct and associated interfaces to reflect the new versioning policy, improving clarity on when the version is modified.
- Improved documentation for methods to clarify their intended use and behavior regarding versioning and metadata updates.
2026-04-22 21:28:20 +08:00
wizardchen 0531700518 feat: Enhance suggested questions functionality with wiki integration
- Updated the SuggestedQuestion interface to include 'wiki' as a new source type, allowing for more diverse question origins.
- Implemented ListRecentForSuggestions method in the wikiPageRepository to retrieve recent wiki pages for fallback suggestions, improving the agent's ability to generate relevant questions when other sources are unavailable.
- Enhanced the customAgentService to utilize the new wiki page suggestions, ensuring a broader range of questions can be generated from wiki content.

These changes significantly improve the agent's suggestion capabilities by incorporating wiki content, enhancing user interaction and information retrieval.
2026-04-22 21:18:20 +08:00
wizardchen 99cc721140 feat: Enhance wiki API and repository with slug encoding and improved search functionality
- Introduced `encodeSlugPath` function to safely encode wiki slugs for API requests, preserving hierarchical routing.
- Updated API functions to utilize the new slug encoding for `getWikiPage`, `updateWikiPage`, and `deleteWikiPage`.
- Enhanced `ListBySourceRef` method in the repository to safely handle source knowledge IDs with JSON marshalling, improving query security.
- Added `escapeLikePattern` function to ensure safe concatenation of LIKE patterns in SQL queries, preventing unintended matches.
- Updated `KnowledgePostProcessService` to include Redis client for task management, improving performance in handling wiki ingest tasks.
- Enhanced `WikiLintIssue` structure to include `TargetSlug` for better issue tracking and auto-fixing capabilities.

These changes significantly improve the robustness and security of the wiki management system, enhancing user experience and data integrity.
2026-04-22 21:18:09 +08:00
wizardchen 5dd4b9f2ee feat: Introduce wiki_flag_issue tool for reporting wiki page issues
- Added a new tool, `wiki_flag_issue`, enabling users to report factual errors, mixed entities, or outdated information on wiki pages.
- Updated the agent system prompt and tool definitions to include the new tool, enhancing the agent's capabilities for maintaining wiki accuracy.
- Implemented backend functionality for creating, listing, and updating the status of flagged issues, ensuring effective tracking and resolution.

These changes significantly improve the agent's ability to manage and report issues within the wiki, fostering a more reliable knowledge base.
2026-04-22 21:17:38 +08:00
wizardchen 76c13c9bfe feat: Enhance wiki tools and agent system prompt for improved knowledge retrieval
- Updated the agent system prompt to clarify the workflow for knowledge retrieval, emphasizing the use of `wiki_search` and `wiki_read_page` tools for specific queries and general overviews.
- Modified the `wikiReadPageTool` to support reading multiple wiki pages simultaneously, improving efficiency in fetching content.
- Enhanced the `wikiSearchTool` to utilize PostgreSQL POSIX regular expressions for more effective search queries, allowing for complex pattern matching.
- Adjusted the `wikiPageRepository` to replace `ILIKE` with regex matching for search queries, increasing the precision of search results.

These changes significantly improve the agent's ability to retrieve and synthesize information from the wiki, enhancing user interactions and response accuracy.
2026-04-22 21:17:36 +08:00
wizardchen 8daf6b368c feat: Add pending tasks and active status to WikiStats
- Introduced `pending_tasks` and `is_active` fields in the `WikiStats` interface to track the number of tasks waiting for processing and the current activity status of the wiki ingestion.
- Updated localization files for English, Korean, Russian, and Chinese to include a new message for displaying the number of pending tasks in the wiki queue.
- Enhanced the `WikiBrowser` component to display the queue status, improving user awareness of ongoing tasks.
- Implemented polling logic to refresh wiki stats periodically when there are pending tasks or active ingestion, ensuring up-to-date information is presented.

These changes enhance the functionality and user experience of the wiki management system by providing real-time insights into task status.
2026-04-22 21:17:31 +08:00
wizardchen f07561f6b6 fix: resolve failing test and build errors
Made-with: Cursor
2026-04-22 21:17:29 +08:00
wizardchen d9cf4b7aa9 feat: implement metadata-based alias management for wiki pages
Made-with: Cursor
2026-04-22 21:17:29 +08:00
wizardchen 9be4152449 feat: Enhance search functionality in WikiPage repository and WikiBrowser component
- Updated the search logic in the WikiPage repository to use ILIKE for case-insensitive matching on title, content, summary, and slug fields, improving search accuracy.
- Added a watcher in the WikiBrowser component to reload all pages when the search query is cleared, enhancing user experience by ensuring relevant content is displayed immediately.

These changes collectively improve the search capabilities and responsiveness of the wiki interface.
2026-04-22 21:17:22 +08:00
wizardchen 0e28b2448c feat: Add arrow toggle functionality in WikiBrowser component
Implemented new features in the WikiBrowser component to allow users to toggle the visibility of arrows on graph edges. This includes:

1. **Localization Updates**: Added translations for "Show arrows" and "Hide arrows" in English, Korean, Russian, and Chinese.
2. **Graph Rendering Enhancements**: Introduced logic to dynamically update SVG line attributes based on the toggle state, improving visual clarity in graph representations.
3. **State Management**: Added reactive state management for arrow visibility, ensuring seamless user interaction.

These changes enhance the user experience by providing more control over graph visualizations.
2026-04-22 21:17:22 +08:00
wizardchen d4d005422d feat: Implement wiki_config updates in knowledge base management
Added functionality to enable updates for the `wiki_config` field in the `KnowledgeBaseConfig` struct. This includes:

1. **Type Definition Update**: Added `WikiConfig` field to `KnowledgeBaseConfig` to allow JSON unmarshaling during update requests.
2. **Service Layer Enhancement**: Updated the `UpdateKnowledgeBase` method to persist changes to `wiki_config` when provided in the request.
3. **Frontend TypeScript Adjustments**: Enhanced type definitions in the frontend to support `wiki_config`, improving IDE autocomplete and type checking.

This fix addresses the critical issue where `wiki_config` was not being updated due to its absence in the `KnowledgeBaseConfig` struct, ensuring proper handling of wiki configurations during knowledge base updates.
2026-04-22 21:17:21 +08:00
wizardchen 33da010605 feat: Implement deadlock retry mechanism for chunk creation
Added a new utility function, WithDeadlockRetry, to handle MySQL deadlock errors during chunk creation. This function retries the operation up to three times with exponential backoff. Updated the CreateChunks method to utilize this new function, ensuring robust handling of concurrent write scenarios and preventing potential deadlocks. Additionally, improved documentation for CreateChunks to clarify the use of Omit for avoiding auto-increment column issues.
2026-04-22 21:17:21 +08:00
wizardchen e147209e77 feat: Implement comprehensive wiki feature for knowledge bases
This commit introduces the complete wiki feature for WeKnora, enabling AI-powered wiki page generation and management. The implementation includes:

**Backend Changes:**
- Wiki data model: WikiPage type with support for multiple page types (Summary, Entity, Concept, Index, Log)
- Database schema: wiki_pages table with full migration support
- WikiPageService: CRUD operations and page management
- WikiPageRepository: GORM-based persistence layer
- Wiki ingest pipeline: Automated generation of wiki pages from knowledge documents
  * Summary page generation using LLM
  * Entity and concept extraction in a single LLM call
  * Synthesis opportunity detection
  * Index page rebuilding
  * Log page maintenance
- Wiki boost feature: Enhance chat retrieval with wiki context
- Wiki linting: Maintenance and validation utilities
- Agent wiki tools: Enable agents to query and interact with wiki pages
- Wiki prompts: Comprehensive LLM prompt templates for all wiki generation tasks
- Language support: Reuse existing middleware language infrastructure for LLM prompts

**Frontend Changes:**
- Wiki browser UI: View all wiki pages with filtering and search
- Wiki API client: Knowledge base wiki management endpoints
- Knowledge base editor: Configure wiki settings (language, auto-ingest, synthesis model)
- i18n updates: Support for English, Korean, Russian, and Chinese interfaces

**Configuration:**
- Container DI: Wire up all wiki services
- Router: Register wiki API endpoints
- Task handling: Support async wiki ingest tasks

**Testing:**
- Unit tests for wiki page types
- Service layer tests
- Endpoint tests for wiki operations
- Integration tests with LLM mocking

**Documentation:**
- Language refactoring analysis and guides
- Implementation completion reports
- Quick reference guides for developers

**Key Features:**
✅ LLM-powered wiki page generation from documents
✅ Multi-language support (9+ languages)
✅ Automatic extraction of entities and concepts
✅ Synthesis opportunity detection
✅ Index and log page maintenance
✅ Progressive wiki building across multiple documents
✅ Agent-based wiki interaction
✅ Chat retrieval enhancement with wiki context
✅ Full frontend UI for wiki browsing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 21:17:18 +08:00
ochan.kwon 127654ee78 feat: add vector_store_id column to knowledge_bases for per-KB vector store binding 2026-04-20 19:53:44 +08:00
ochan.kwon 927d3debff feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
wizardchen 092044b68f refactor(postgres): optimize vector retrieval and HNSW settings
- Adjusted the expanded TopK limit to a maximum of 200 candidates to enhance HNSW efficiency.
- Updated the vector retrieval query to ensure the ORDER BY expression matches the indexed expression, improving index usage.
- Implemented local settings for HNSW's ef_search and iterative_scan within a transaction to optimize candidate retrieval.
- Added fallback logic to handle unsupported GUCs gracefully, ensuring query execution continues even with older pgvector versions.

These changes improve the performance and reliability of vector retrieval in the PostgreSQL implementation.
2026-04-17 12:11:03 +08:00
wizardchen 4f55994266 feat(prompt): enhance question rewriting and intent classification in prompt templates
- Updated the prompt template to clarify the task of understanding and rewriting user questions, improving the instructions for coreference resolution and ellipsis completion.
- Renamed sections for better clarity, changing "Rewriting Goals" to "Query Understanding."
- Added critical notes for intent classification regarding image and document attachments, ensuring accurate processing based on the presence of attachments.
- Improved examples to reflect the updated intent classification logic, enhancing the overall user experience in query handling.
2026-04-16 23:15:02 +08:00
wizardchen 53e8e17df4 feat(logger): implement LLM debug logging functionality
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
2026-04-16 23:15:02 +08:00
wizardchen dcb7fd05b1 Fix compile error in weaviate repository 2026-04-16 18:37:31 +08:00
wizardchen af58e9cff4 fix(repository): ensure expandedTopK is at least params.TopK in VectorRetrieve
- Added a check to set expandedTopK to params.TopK if it is lower, ensuring the subquery limit is not less than the final limit.
- Removed unnecessary type casting in the SQL query for embedding distance calculations, simplifying the query structure.

This update improves the accuracy of vector retrieval operations in the PostgreSQL repository.
2026-04-16 18:13:20 +08:00
swim2sun d2c6814f22 fix: SQLite/Lite mode compatibility for FAQ import and chunk operations
Multiple fixes for WeKnora Lite mode (SQLite, no Redis):

1. Redis nil pointer panic on FAQ import (knowledge.go)
   - Add in-memory fallback (sync.Map) for FAQ import progress tracking
   - Add nil guards for running task locks and progress queries

2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
   - GORM autoIncrement on non-PK columns doesn't work in SQLite
   - Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
   - Add BeforeCreate hook for KnowledgeTag
   - Use Unscoped() to include soft-deleted records in MAX(seq_id) query

3. NOW() function not available in SQLite (chunk.go)
   - Replace raw SQL NOW() with time.Now() or datetime('now')
   - Use dialector check for raw SQL that must differ between PG and SQLite

4. FAQ KB should reject file uploads (knowledge.go)
   - Add kb.Type == "faq" check in CreateKnowledgeFromFile

5. Linux/macOS build compatibility (Makefile)
   - Conditionally apply macOS-only linker flag based on uname

6. Add SQLite integration tests (chunk_sqlite_test.go)
   - TestCreateChunks_SQLite_SeqIDAutoAssigned
   - TestCreateChunks_SQLite_SeqIDContinuesFromExisting
   - TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
   - TestCreateChunks_SQLite_SeqIDAfterSoftDelete
   - TestKnowledgeTag_SQLite_SeqIDAutoAssigned
   - TestUpdateChunk_SQLite_NoNOWError

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 20:37:10 +08:00
wizardchen 9bb4a12562 feat(repository): update keyword retrieval logic to use ParadeDB's matching operator
- Modified the SQL clause in the KeywordsRetrieve function to utilize ParadeDB's ||| operator for improved token matching.
- Enhanced the handling of retrieval results in the fuseOrDeduplicate function to maintain original scores for keyword-only results, ensuring better performance in FAQ scenarios.

This update optimizes the keyword retrieval process and improves the accuracy of search results.
2026-04-15 20:16:02 +08:00
ochan.kwon 9edf0839ed feat: add VectorStore service layer with types and connection testing
Add VectorStoreService with CRUD validation, duplicate checking (DB + env
stores), and engine-specific health checks for 6 vector database types.
Include VectorStoreResponse DTO, env store builder, engine type metadata,
and comprehensive unit tests.
2026-04-13 22:57:17 +08:00
wizardchen a655a25cf8 fix: revert to neo4j v6 and workaround Go 1.24 Windows compiler bug with -p=1
Go 1.24 has an internal compiler race condition on Windows that causes
'package without types' errors during parallel compilation. Setting
GOFLAGS="-p=1" serializes the build and avoids triggering the bug.
2026-04-11 22:08:02 +08:00
wizardchen 9525f08038 fix: downgrade neo4j driver from v6 to v5 to fix Go 1.24 Windows internal compiler bug 2026-04-11 21:59:03 +08:00
wizardchen 9a2b54536d fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
ochan.kwon 69785ad0a3 feat: add VectorStore entity, repository, and migrations
Introduce the VectorStore domain model as a foundation for multi-store
vector DB architecture (Phase 1, PR 1/4).

- VectorStore entity with ConnectionConfig (AES-GCM encrypted) and IndexConfig
- VectorStoreRepository interface and GORM implementation (CRUD + duplicate check)
- PostgreSQL migration (000031) and SQLite migration (000001)
- Unit tests for types, encryption, validation, and helpers
2026-04-09 01:39:51 +08:00
wizardchen 9d021a29b7 feat(elasticsearch): enhance ID field handling with dynamic suffix detection
- Added logic to determine whether to append the ".keyword" suffix to ID fields based on index mapping.
- Introduced `detectFieldTypes` method to inspect index mappings and set the `useKeywordSuffix` flag accordingly.
- Updated methods to utilize the new `idField` function for consistent ID field querying across the repository.
- Improved logging for better visibility into mapping detection and suffix usage.
2026-04-02 15:26:34 +08:00
ochan.kwon 33e919cc80 fix(knowledge): delete extracted images from storage when knowledge is removed
Previously, only the original uploaded file was deleted via knowledge.FilePath.
Images extracted by the builtin parser (stored as individual S3/MinIO/local files
via ImageResolver) were never cleaned up, causing orphan files to accumulate.

- Add ListImageInfoByKnowledgeIDs repository method (no chunk_type filter)
- Add collectImageURLs utility with URL deduplication and JSON parse logging
- Add deleteExtractedImages helper (standalone, best-effort with error logging)
- Fix DeleteKnowledge: collect image URLs before errgroup, delete in file goroutine
- Fix DeleteKnowledgeList: bulk collect with KB-level fileSvc mapping
- Fix cleanupKnowledgeResources: collect before chunk deletion, delete after
- Fix ProcessKBDelete: collect before chunk deletion for KB-wide cleanup
2026-04-02 12:04:04 +08:00
ochan.kwon 258a786129 Revert "refactor(elasticsearch): remove ".keyword" suffix from ID fields in Elasticsearch queries"
This reverts commit 3daaabfc4a.
2026-04-01 20:41:24 +08:00
wizardchen a5ca9a2784 feat(web-search): refactor web search provider
- Updated relevant files to include provider registration, implementation, and metadata.
- Enhanced frontend components to support provider management and configuration.
- Added localization for new provider settings and messages.
- Implemented backend repository methods for CRUD operations on web search providers.
2026-03-31 20:22:34 +08:00
wizardchen 1a340a8374 refactor(datasource): remove WeCom document integration and related components
- Deleted WeCom document connector and callback handling code to streamline the datasource management.
- Removed associated localization entries and UI elements related to WeCom integration.
- Updated data source management logic to eliminate references to WeCom, enhancing maintainability and clarity.
- Adjusted error handling and logging to reflect the removal of WeCom-specific functionality.
2026-03-30 20:46:07 +08:00
wizardchen 53bd412cf6 fix(datasource): enhance error handling in ProcessSync method
- Improved logging for missing data sources, providing clearer warnings when a data source is not found.
- Updated sync log status to 'canceled' when a data source is deleted, ensuring accurate tracking of sync operations.
- Adjusted error handling to prevent unnecessary error returns, enhancing overall process stability.
2026-03-30 20:46:07 +08:00