Commit Graph

1583 Commits

Author SHA1 Message Date
nullkey 54b6f2dc35 docs(api): regenerate swagger via make docs
由 make docs 一次性重新生成 docs/swagger.{yaml,json} 和 docs/docs.go,
包含前一 commit 中的:
- 24 个新增 swag 注解
- 12 处既有注解 bug 修正
- 几处 response 类型从 map[string]interface{} 收敛到具体 struct

机器生成,建议跳过逐行 review。

Refs #890 #1049
2026-05-12 13:16:58 +08:00
nullkey c753c67608 docs(api): add and fix swag annotations on handlers
将 swag 注解覆盖率从 ~88% 提升到 100%,并修复审计中发现的若干既有
注解 bug。

新增注解(24 个):
- im.go: 4 (UpdateIMChannel/DeleteIMChannel/ToggleIMChannel/IMCallback)
  其中 IMCallback 同时注册 GET 和 POST,两个方法都加了 @Router 声明。
- web_search.go: 1 (GetProviders)
- web_search_provider.go: 6
- wechat_qrcode.go: 2
- weknoracloud.go: 2
- knowledge.go: 2 (MoveKnowledge/GetKnowledgeMoveProgress)
- knowledgebase.go: 1 (ListMoveTargets)
- organization.go: 3 (ListSharedAgents/ListOrgAgentShares/RemoveAgentShare)
- mcp_service.go: 2 (SetMCPToolApproval/ResolveToolApproval)
- chunker_debug.go: 1 (PreviewChunking)

修复既有注解 bug(12 处):
- initialization.go: 10 个 @Router 路径与真实路由不符
  (e.g. /initialization/kb/{kbId}/config → /initialization/config/{kbId}
   /initialization/fabri/tag GET → /initialization/extract/fabri-tag POST)
- session/stream.go: ContinueStream 路径
  /sessions/{session_id}/continue → /sessions/continue-stream/{session_id}
- mcp_service.go: ResolveToolApproval 请求体 {approve: bool}
  → {decision: "approve"|"reject", reason?, modified_args?}

收敛响应 schema 到已有命名 struct(之前用 map[string]interface{}):
- knowledge.go MoveKnowledge → handler.MoveKnowledgeResponse
- knowledge.go GetKnowledgeMoveProgress → types.KnowledgeMoveProgress
- web_search_provider.go GetProvider/UpdateProvider → types.WebSearchProviderEntity
- initialization.go InitializeByKB → handler.InitializationRequest
- initialization.go 三个 model-test endpoint → handler.ModelTestRequest

修正 organization.go 中 errors.AppError 应使用 apperrors 别名(该文件
stdlib errors 未别名化)。

不补 /files 和 /api/v1/files/presigned 内联闭包路由(文件服务,非业务 REST)。

Refs #890 #1049 #1168
2026-05-12 13:16:58 +08:00
wizardchen 1f5970b67e refactor(agent): rebuild multi-turn history from DB, drop llmcontext layer
The Agent ran a parallel context cache (Redis/in-memory) on top of the
messages table to feed multi-turn history into the LLM. That dual-write
caused subtle drift (e.g. compression diverging from DB, system-prompt
swaps lost on restart) and required a separate ClearContext path on
session/IM clear.

Make the messages table the single source of truth:

- Add service.LoadAgentHistory: rebuilds chat.Message history per turn
  from the persisted messages, expanding AgentSteps into proper OpenAI
  assistant_with_tool_calls + tool messages and replaying the canonical
  final answer (with <think> blocks stripped). final_answer tool calls
  are filtered to avoid duplicating the trailing answer.
- Make AgentEngine stateless across turns: drop ContextManager / sessionID
  cache plumbing from the engine, agent service, and CreateAgentEngine
  signature. The engine only uses sessionID for logging/event emission.
- Wire AgentQA to load history from DB on demand using HistoryTurns
  (default 5) when MultiTurnEnabled, otherwise run with empty history.
- Delete the llmcontext package (ContextManager interface, Redis/memory
  storage, factory) and the SessionService.ClearContext API path; IM
  /clear and session message clear no longer need to invalidate cache.

Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with
empty AgentSteps just produce the single canonical assistant message),
and Agent-mode turns now consistently see prior tool calls and results.
2026-05-12 12:26:24 +08:00
wizardchen 0b2de5c412 chore(env): remove WEKNORA_SANDBOX_DOCKER_IMAGE from .env.example 2026-05-12 11:32:26 +08:00
wizardchen bd68a0c377 feat(cloud-image): support apt-based docker install for restricted-egress hosts
Mainland China cloud VMs (Tencent Lighthouse, Aliyun, etc.) frequently
cannot reach get.docker.com, github.com, or even community GitHub
mirrors like gh-proxy.com. The cloud-image bootstrap previously had no
escape hatch for this and failed at the very first curl.

This adds a new DOCKER_INSTALL_MIRROR env var to prepare.sh. When set,
it skips get.docker.com and installs docker-ce + compose-plugin from an
apt mirror of Docker's official repo (e.g. mirrors.tencent.com,
mirrors.aliyun.com).

README.md also gets:
- A GH_PROXY env var threaded through bootstrap methods A and B so the
  initial script pull can route through gh-proxy / ghfast.
- An explicit recommendation to prefer method C (scp from local) on
  mainland China VMs.
- A consolidated "三件套" table mapping WEKNORA_GH_PROXY /
  DOCKER_INSTALL_MIRROR / DOCKER_REGISTRY_MIRROR to per-cloud
  endpoints, so users hit one place to copy the full env.
2026-05-11 21:14:08 +08:00
wizardchen 6b812a54d2 fix(searxng): provide hardcoded default SEARXNG_SECRET for zero-config startup
`${SEARXNG_SECRET:?...}` made the variable mandatory at compose parse time,
which forced *any* compose command (default profile included) to fail when
SEARXNG_SECRET was unset, with a message confusingly claiming the searxng
profile was being started.

Switch to `${SEARXNG_SECRET:-weknora-default-searxng-secret-...}` so the
searxng profile starts zero-config. Default deployments bind searxng to
127.0.0.1 only, so a shared default secret is acceptable; .env.example
now explicitly warns to rotate it before flipping SEARXNG_BIND=0.0.0.0,
since secret_key signs image-proxy URLs.
2026-05-11 16:53:47 +08:00
wizardchen 5fb7e692e5 refactor(web-search): drop dead api_key path in SearXNG provider
The SearXNG provider forwarded `parameters.api_key` as
`Authorization: Bearer <key>` for reverse-proxy auth, but the frontend
never rendered the API key input for self-hosted providers
(`requires_api_key=false`), so the path was unreachable from the UI.

Rather than expose another UI knob for a niche reverse-proxy setup that
the project does not currently support, remove the Authorization header
branch and the `apiKey` field on `SearxngProvider`. Tenants who put
SearXNG behind an authenticating reverse proxy can still front it with
network-level auth (mTLS, IP allowlist) without changes here.

`WebSearchProviderParameters.APIKey` is generic and used by other
providers, so it stays.
2026-05-11 16:53:47 +08:00
wizardchen c2d60933c8 refactor(security_test): standardize struct field formatting in SSRF whitelist tests
- Adjusted the formatting of struct fields in the SSRF whitelist test cases for improved readability and consistency.
- No functional changes were made; this is purely a code style improvement to enhance maintainability.
2026-05-11 16:53:47 +08:00
wizardchen 0f5dc41f4e feat(searxng): enhance SearXNG configuration and validation
- Updated .env.example to clarify SEARXNG_SECRET generation and added SSRF_WHITELIST_EXTRA for improved security.
- Modified docker-compose files to bind SearXNG to localhost by default and introduced a one-time initialization service to set up settings.yml correctly.
- Enhanced SearxngProvider with stricter URL validation, ensuring no query or fragment is present in the base URL.
- Added unit tests for SearXNG validation and date parsing to ensure robustness.
- Updated frontend WebSearchSettings to reflect changes in SearXNG instance URL handling.

This commit improves the security and usability of the SearXNG integration, addressing potential misconfigurations and enhancing the developer experience.
2026-05-11 16:53:47 +08:00
wizardchen d2a1006beb fix(web-search): address review fixups for SearXNG provider
- utils: 合并 SSRF_WHITELIST 与 SSRF_WHITELIST_EXTRA,避免部署侧默认值
  (如 docker-compose 注入的 searxng 主机名)被用户的 SSRF_WHITELIST
  自定义值覆盖。
- docker-compose.yml: 把 searxng 默认值挪到 SSRF_WHITELIST_EXTRA。
- searxng: 抽出导出函数 ValidateSearxngBaseURL,让 service 层保存校验
  和 provider 构造校验完全一致;service 改为调用同一函数。
- searxng: language 由非法的 "auto" 改为 "all";移除强制 safesearch=1
  让实例 settings.yml 决定。
- searxng: publishedDate 增加多格式 fallback(RFC3339Nano/无时区/
  日期-only/RFC1123 等),无法解析时 debug 日志记录。
- searxng: 解析 unresponsive_engines;结果为空时打 warn 日志,便于排查
  "配置正确却搜不到结果"的情况。
- frontend: WebSearchSettings 的 Instance URL 标签/占位符走 i18n,
  zh-CN / en-US / ru-RU / ko-KR 四个 locale 补齐 baseUrlLabel /
  baseUrlPlaceholder。
2026-05-11 16:53:47 +08:00
wizardchen 1110615300 feat(web-search): add SearXNG provider (#1166)
支持对接自建/公共 SearXNG 实例作为网络搜索引擎,缓解免费搜索引擎在国内
网络环境下访问受限的问题。

- types: 新增 WebSearchProviderTypeSearxng 与 BaseURL 参数字段;
  类型元数据新增 RequiresBaseURL,前端可动态渲染 Instance URL 表单。
- infrastructure/web_search/searxng.go: 调用 /search?format=json,强制
  utils.ValidateURLForSSRF 校验 base_url,可选 api_key 透传给反代鉴权。
- service: isValidProviderType 与参数校验接入 searxng。
- container: 注册 NewSearxngProvider 工厂。
- frontend: WebSearchSettings 表单根据 requires_base_url 渲染 Instance
  URL 输入框;编辑回填、free 判定同步更新。
- docker: 新增可选 searxng 服务(profile=searxng/full),附带最小化
  settings.yml(启用 JSON 格式、关闭 limiter、关闭遥测),
  docker-compose 默认 SSRF_WHITELIST 包含 searxng 容器名。
- .env.example: 补充 SEARXNG_PORT / SEARXNG_SECRET 说明。

Closes #1166
2026-05-11 16:53:47 +08:00
wizardchen 9dcb61fb45 feat(frontend): enable gzip compression for static resources in nginx configuration
- Added gzip compression settings to improve loading times for static assets, significantly reducing the size of files served and enhancing performance for users, especially in low bandwidth regions.
2026-05-11 15:51:11 +08:00
wizardchen 0cfbad7f97 fix(cloud-image): enhance firstboot and cleanup scripts for improved security and functionality
- Updated cleanup.sh to avoid recreating .env during cleanup, preventing exposure of default passwords before firstboot.
- Modified firstboot.sh to create .env from .env.example only if it doesn't exist, ensuring no sensitive data is present before initialization.
- Added support for Docker Hub and GitHub tarball download acceleration via new environment variables WEKNORA_GH_PROXY and DOCKER_REGISTRY_MIRROR.
- Implemented a mechanism to prune old WeKnora images based on the current version, reducing image size and maintaining a clean environment.
- Enhanced README.md with instructions for using the new acceleration features and image pruning options.
2026-05-11 15:51:11 +08:00
qingfhuang 0eb95ccbd8 fix(client): correct GetChunkByIDOnly endpoint
Use /api/v1/chunks/by-id/{id} to match backend route.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-11 15:34:29 +08:00
wizardchen 1e3aa337db fix(cloud-image): refine cleanup process to prevent unwanted image deletions
Updated the cleanup script to replace the broad `docker system prune` command with more targeted commands. This change ensures that only unused containers, build cache, and dangling volumes are removed, preventing the accidental deletion of important images like `wechatopenai/weknora-*` that are pre-pulled for firstboot. The script now explicitly prunes containers, builders, and dangling volumes to maintain a clean environment without disrupting the intended image setup.
2026-05-11 12:25:19 +08:00
wizardchen cdfbf05524 fix(cloud-image): make firstboot idempotent and pin image versions
Address review feedback on PR #1249:

- prepare.sh: when WEKNORA_REF looks like a version tag (v*), write the
  matching WEKNORA_VERSION into .env so docker compose pulls images that
  match the compose YAML's git ref (previously stuck on :latest).
- prepare.sh: detect docker binary path via `command -v docker` and
  template it into weknora.service (replacing hardcoded /usr/bin/docker
  that fails when docker lives in /usr/local/bin).
- firstboot.sh: write a /opt/WeKnora/.firstboot.done marker immediately
  after rewriting .env, before `docker compose up -d`. If compose fails
  mid-run, the next boot is gated by ConditionPathExists=!marker so we
  never regenerate DB_PASSWORD against an already-initialized postgres
  volume (which previously bricked the database).
- firstboot.sh: stop deleting its own unit file / script while the
  oneshot is still executing; rely on the marker + `systemctl disable`
  instead, avoiding "job failed" markings from systemd.
- firstboot.sh: use detected docker path instead of /usr/bin/docker;
  add note in credentials file that .env is the source of truth.
- weknora-firstboot.service: add ConditionPathExists=!.firstboot.done.
- cleanup.sh: scope docker volume deletion to compose project label
  (com.docker.compose.project=<name>) instead of fuzzy substring match
  that could nuke unrelated postgres/redis volumes.
- cleanup.sh: also remove .firstboot.done marker, firstboot log, and
  any leftover /root/weknora-credentials.txt so the image is clean.
- README.md: clarify how to actually disable registration (edit the
  `replace` call list in firstboot.sh, not run that command in shell).
2026-05-11 12:25:19 +08:00
wizardchen 155f3b3e72 docs(cloud-image): clarify sudo + redirection pitfall in setup steps
Recommend `sudo -i` to avoid the classic `sudo cmd >> file` failure
where the shell redirection runs as the unprivileged user. Also document
the `sudo tee -a` workaround and add a scp option C.
2026-05-11 12:25:19 +08:00
wizardchen afd7d1fdf8 docs(cloud-image): add cloud-agnostic image packaging scripts
Add scripts and docs for packaging WeKnora into cloud images (AMI,
custom images, snapshots) so users can distribute one-click deployable
templates on any cloud provider.

- scripts/cloud-image/: cloud-agnostic prepare/cleanup/firstboot scripts
  plus systemd units. Downloads only the 4 runtime files needed by the
  compose stack (~100KB) instead of cloning the full repo, and pins to
  any git ref via WEKNORA_REF for reproducible builds.
- firstboot.sh randomizes DB/Redis/JWT/AES secrets on first boot,
  writes credentials to /root/weknora-credentials.txt and self-removes.
- docs/cloud-image/: per-platform packaging guides. Includes a guide
  for Tencent Cloud Lighthouse / CVM covering image creation, sharing,
  and marketplace listing.

Default-on services match the unprofiled compose stack (frontend, app,
docreader, postgres, redis); optional services (qdrant, milvus,
neo4j, langfuse, etc.) remain opt-in via compose profiles to keep the
image size small.
2026-05-11 12:25:19 +08:00
wizardchen 7c0964bb95 fix(agent): enhance user authorization and concurrency handling in tool approval
- Introduced a new test to ensure that empty user IDs are rejected when a waiter has a user, preventing unauthorized approvals.
- Added RequestNonce to resolve messages to uniquely identify Resolve calls, ensuring concurrent requests do not interfere with each other.
- Updated waiter structure to use atomic operations for the resolved state, improving thread safety.
- Enhanced error handling in the Resolve method to properly manage user mismatches and ensure accurate acknowledgment delivery across instances.
- Updated MCP service handler to reject unauthenticated requests early, providing clearer feedback on authorization requirements.
2026-05-10 22:57:12 +08:00
wizardchen 6d0f09d75f fix(agent): 加固 MCP 工具人审批的跨实例与并发语义
- 跨实例 Resolve 通过 Redis reply channel 接收 owning 实例的 ack
  (ok / not_found / tenant_mismatch / user_mismatch / already_resolved),
  HTTP 不再静默 200,超时退化为 NotFound。
- PendingRequest 增加 UserID,Resolve 校验会话所有者,避免同租户
  其他用户越权批准/篡改参数。
- 新增 ErrAlreadyResolved,timer/ctx 命中后再次 Resolve 返回 400 而
  不是静默成功;waiter.deliver 暴露胜出语义。
- NeedsApproval 默认 fail-close(DB 故障要求审批),可经
  WEKNORA_AGENT_TOOL_APPROVAL_FAIL_OPEN=true 回退。
- Pub/Sub 增加 OriginID 自过滤、退避重连、namespace 后缀,避免自回
  环噪声与多部署串扰。
- 工具执行超时常量统一通过 ToolExecContext.ExecTimeout 注入,
  mcp_tool.go 不再硬编码 60s。
- handler 修正 path 二次解码、错误用 errors.Is、modified_args=null 的
  静默清空、ListMCPToolApprovals 的 404/500 误判。
- repository 用 clause.OnConflict 做原子 upsert,消除并发 500。
- 增加 NotFound / TenantMismatch / UserMismatch / AlreadyResolved /
  Race 等用例。
2026-05-10 22:57:12 +08:00
wizardchen 5510ea8f5a feat(agent): human-in-the-loop approval for MCP tool calls (#1173)
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.

Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
  required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
  delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
  replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
  (without the per-tool 60s timeout) for the wait, and re-derives a
  fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
  EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
  WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
  duration).

Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
  args, validation feedback, mm:ss countdown that turns warning/danger
  near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.

Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
  considerations (Redis cross-instance, restart limitations).
2026-05-10 22:57:12 +08:00
wizardchen c1826030ab fix(frontend): 修正 LLM 调用超时文案与实际行为一致,上限放宽至 3600s
前端原文案'0 表示无限等待(不推荐)'与后端实际行为不符——
internal/agent/const.go 中 getLLMCallTimeout 在配置为 0 时回退到
defaultLLMCallTimeout (120s),并不会无限等待,文案误导用户。

- 同步 zh-CN/en-US/ru-RU/ko-KR 四种语言 hint 为'0 表示使用默认值(120s)'
- placeholder 推荐范围由 60-600 调整为 60-1800,与后端兜底超时上调保持一致
- AgentEditorModal 输入框 :max 由 600 放宽到 3600,避免长推理模型场景被卡住
2026-05-10 00:26:53 +08:00
wizardchen 24a010ff59 fix(chat): reduce default LLM timeout values to 300s for chat and 600s for stream 2026-05-10 00:16:12 +08:00
wizardchen e83378ecd1 fix(chat): 调大 LLM 兜底超时默认值(chat 600s / stream 1800s) 2026-05-10 00:16:12 +08:00
wizardchen 386df92582 fix(chat): LLM 兜底超时仅在上层无 deadline 时生效,并修复 ChatStream raw HTTP 路径的 cancel 泄漏
PR #1238 引入的 context.WithTimeout 会把上层 ctx 的 deadline 强制截断到
默认 120s/300s,导致调用方无法设置更长的合理超时(例如长推理模型场景)。
同时 ChatStream 在走 chatStreamWithRawHTTP 早返回路径时丢弃了 cancel
函数,timeoutCtx 要等 deadline 到期才释放,go vet/lostcancel 也会告警。

本次修复:
- 新增 withLLMTimeout 辅助函数:仅在上层 ctx 没有 deadline 时附加默认
  超时;上层若已显式设置 deadline(无论比默认更短还是更长),都原样
  尊重,把超时的最终决定权交还给调用方。
- 默认值通过环境变量可覆盖:
  - WEKNORA_LLM_CHAT_TIMEOUT_SECONDS    (默认 120s)
  - WEKNORA_LLM_STREAM_TIMEOUT_SECONDS  (默认 300s)
- 修复 ChatStream raw HTTP 早返回路径未调用 cancel 的泄漏:新增
  wrapStreamCancel,在底层 channel 关闭后统一执行 cancel。
- 补充单测覆盖三种 deadline 场景与环境变量解析。
2026-05-10 00:16:12 +08:00
杨明康 6b8dbaa25b fix(chat): 为 LLM 非流式/流式调用添加超时保护,防止 worker 被 hung 请求永久阻塞
IM 渠道(企微等)的消息处理走 KnowledgeQA 管道,该管道的 LLM 调用此前
未设置任何超时,依赖 context cancellation 控制。然而 QA 管道传入的 context
本身无 deadline,一旦上游 LLM API(如 mimo-v2.5-pro)的网络请求 hang 在
TCP 层面(连接未断但无数据返回),worker goroutine 会永久阻塞在
CreateChatCompletion / CreateChatCompletionStream 上。

随着时间推移,多个 worker 陆续卡死,队列中所有 worker 耗尽后机器人对任何
消息都不再回复。

本提交在 RemoteAPIChat.Chat 和 RemoteAPIChat.ChatStream 两个入口方法中
统一添加 context.WithTimeout:
- 非流式调用(Chat):120s 超时
- 流式调用(ChatStream):300s 超时

如果上层调用方已设置更短的 deadline,WithTimeout 自动遵循更短的那个,
不会覆盖已有的合理超时设置。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-10 00:03:53 +08:00
langcaiye 02be05c095 feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
wizardchen 82947c726e fix(wiki-ingest,asynqdl): review fixups for PR #1241
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
  trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
  items" had two args), and harden validMerge against un-prefixed
  slugs whose strings.Index returned -1 and silently passed the type
  check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
  ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
  letter row instead of a hard-coded 0; tighten payloadProbe to the
  set of field names with consistent semantics across payloads
  (drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
  outside an asynq worker ctx so the semantics stay pinned.
2026-05-10 00:01:06 +08:00
wizardchen 04e8105728 perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.

This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.

  - migration 000041: task_pending_ops + task_dead_letters tables, plus
    three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
    source_refs text fallback, lower(title) trgm).
  - TaskPendingOpsRepository + TaskDeadLetterRepository in
    internal/application/repository/task_queue.go: cursor-paginated
    list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
    delete that refuses empty keys so a buggy caller can't wipe a KB's
    queue.
  - internal/middleware/asynqdl: writes a task_dead_letters row when an
    asynq task exhausts its retry budget. Payload-agnostic — a small
    probe struct extracts TenantID + scope hints across every existing
    payload type, so summary:generation / image:multimodal /
    faq_import / kb:clone / etc. all dead-letter without per-handler
    code. Best-effort: an insert failure never masks the underlying
    task error. Installed in router/task.go before the langfuse mw so
    it sees raw errors.

  - Pending queue moved Redis → PG (no TTL, restart-durable). Redis
    keeps just the active-batch lock and the delete tombstone.
  - In-batch retry budget (wikiMaxFailRetries=5) tracked via
    pendingRepo.IncrFailCount; over the cap the row is moved to
    task_dead_letters with task_type=wiki:ingest, related_id=knowledge
    id. Asynq retries (10) are handled by the new middleware.
  - Removed the per-batch ListAllPages. WikiBatchContext now carries
    lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
    mutex-protected caches; reduce reaches for titles / summaries on
    demand instead of pre-loading the whole KB.
  - getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
    which the new GIN index on source_refs serves as a Bitmap Index
    Scan instead of a sequential text LIKE.
  - Dedup pre-filter uses pg_trgm via FindSimilarPages
    (idx_wiki_pages_title_trgm). For each new entity/concept (and each
    of its aliases) we ask the DB for the top-K trigram-similar
    existing pages and union the results — bounded prompt size, no Go-
    side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
  - cleanDeadLinks / injectCrossLinks are scoped to the batch's
    affected slugs (a few dozen) instead of every page in the KB. Both
    use the new lite ListBySlugs / ExistsSlugs repo methods so they
    pull only slug + title + outlinks, not full content.
  - Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
    [[bad-slug|display]] the cleanup path now tries display-text
    reverse lookup → hyphen/case normalized equality → char-bigram
    Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
    break drift case ("shang-hai-tower" vs "shanghai-tower") in place
    instead of replacing the link with plain text.
  - rebuildIndexPage uses ListByTypeRecent(200) for the first-time
    intro and drops the full DocumentSummaries blob from the
    incremental update prompt, so its context stays bounded regardless
    of KB size.
  - Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
    IngestMapParallel / IngestReduceParallel, with sensible defaults
    via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
    follow-ups don't waste asynq retry slots bouncing on the active
    lock.

  - RunLint walks pages via the new ListPagesCursor in 200-page
    windows and computes the live-slug set with a one-column
    ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
    every node + edge. Memory is now bounded; 40k pages walks in
    constant ~4MB.

  - 14 GORM tests for both repos against an in-memory SQLite mirror of
    the production DDL (task_queue_test.go).
  - 6 tests for the dead-letter middleware covering retry budget
    detection, payload-agnostic scope inference, error truncation, and
    repo-failure isolation.
  - 7 tests for the slug fuzzy resolve helper covering the three
    resolution stages, display-text priority over normalized equality,
    bigram fallback acceptance, and rejection of unrelated slugs.
  - Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
    new fetcher / cursor APIs.
2026-05-10 00:01:06 +08:00
dependabot[bot] 91cf237dd9 chore(deps): bump golang.org/x/crypto from 0.50.0 to 0.51.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.50.0 to 0.51.0.
- [Commits](https://github.com/golang/crypto/compare/v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 20:07:12 +08:00
wizardchen f016eff79a chore(deps): update github.com/go-openapi/swag to v0.26.0 and remove unused dependencies
Updated the go.mod and go.sum files to bump the version of github.com/go-openapi/swag and its related packages to v0.26.0. Removed the indirect dependency on github.com/josharian/intern as it is no longer needed.
2026-05-09 19:52:21 +08:00
dependabot[bot] 2fd2a139e0 chore(deps): bump github.com/jackc/pgx/v5 from 5.7.2 to 5.9.2
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) from 5.7.2 to 5.9.2.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v5.7.2...v5.9.2)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
  dependency-version: 5.9.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 19:42:26 +08:00
wizardchen ff950ef249 fix(ollama): adapt to ToolCallFunctionArguments struct API
The ollama Go SDK changed ToolCallFunctionArguments from
map[string]any to a struct backed by an ordered map. Use ToMap() for
read access and NewToolCallFunctionArguments()+UnmarshalJSON for
construction so the build matches the upgraded SDK.
2026-05-09 13:58:55 +08:00
dependabot[bot] 4bac6e30d2 chore(deps): bump github.com/milvus-io/milvus/client/v2
Bumps [github.com/milvus-io/milvus/client/v2](https://github.com/milvus-io/milvus) from 2.6.2 to 2.6.4.
- [Release notes](https://github.com/milvus-io/milvus/releases)
- [Commits](https://github.com/milvus-io/milvus/compare/v2.6.2...v2.6.4)

---
updated-dependencies:
- dependency-name: github.com/milvus-io/milvus/client/v2
  dependency-version: 2.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:36:17 +08:00
dependabot[bot] 46abae96f4 chore(deps): bump github.com/ollama/ollama from 0.11.4 to 0.23.2
Bumps [github.com/ollama/ollama](https://github.com/ollama/ollama) from 0.11.4 to 0.23.2.
- [Release notes](https://github.com/ollama/ollama/releases)
- [Commits](https://github.com/ollama/ollama/compare/v0.11.4...v0.23.2)

---
updated-dependencies:
- dependency-name: github.com/ollama/ollama
  dependency-version: 0.23.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:33:57 +08:00
dependabot[bot] bcd1700240 chore(deps): bump gorm.io/driver/postgres from 1.5.11 to 1.6.0
Bumps [gorm.io/driver/postgres](https://github.com/go-gorm/postgres) from 1.5.11 to 1.6.0.
- [Commits](https://github.com/go-gorm/postgres/compare/v1.5.11...v1.6.0)

---
updated-dependencies:
- dependency-name: gorm.io/driver/postgres
  dependency-version: 1.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:33:39 +08:00
dependabot[bot] 80731a2726 chore(deps): bump google.golang.org/grpc from 1.80.0 to 1.81.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.80.0 to 1.81.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.80.0...v1.81.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.81.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:33:28 +08:00
wizardchen d65e647f95 chore: bump Go to 1.26 and slim docreader dependencies
- Bump base image in docker/Dockerfile.app from golang:1.24 to golang:1.26
  to match `go 1.26` declared in go.mod (fixes CI build failure on
  `go mod download`).
- Drop unused docreader components and their dependencies:
  - Remove `docreader/ocr/` package (paddle/vlm/dummy backends are
    unreferenced by the main flow; OCR/VLM is handled by the Go App).
  - Remove `docreader/parser/storage.py` (dead code; image persistence
    happens in the Go App via inline ImageRef bytes).
  - Remove `docreader/scripts/download_deps.py` (PaddleOCR pre-download).
  - Drop deps: paddleocr, paddlepaddle, openai, ollama, minio,
    cos-python-sdk-v5, oss2, asyncio, pypdf2, markdown, mistletoe,
    goose3, markdownify, pdfplumber, antiword, urllib3.
- Re-lock uv.lock: 145 -> 79 packages.
- Update docreader/README.md to reflect that OCR/VLM/storage are no
  longer configured at the docreader level.
2026-05-09 13:32:40 +08:00
langcaiye 74b1342440 feat: add Tencent VectorDB retriever backend 2026-05-09 13:14:01 +08:00
dependabot[bot] eca060f3aa chore(deps): bump nltk from 3.9.2 to 3.9.4 in /docreader
Bumps [nltk](https://github.com/nltk/nltk) from 3.9.2 to 3.9.4.
- [Changelog](https://github.com/nltk/nltk/blob/develop/ChangeLog)
- [Commits](https://github.com/nltk/nltk/compare/3.9.2...3.9.4)

---
updated-dependencies:
- dependency-name: nltk
  dependency-version: 3.9.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:10:33 +08:00
dependabot[bot] 3293d5464b chore(deps): bump github.com/chromedp/chromedp from 0.14.2 to 0.15.1
Bumps [github.com/chromedp/chromedp](https://github.com/chromedp/chromedp) from 0.14.2 to 0.15.1.
- [Release notes](https://github.com/chromedp/chromedp/releases)
- [Commits](https://github.com/chromedp/chromedp/compare/v0.14.2...v0.15.1)

---
updated-dependencies:
- dependency-name: github.com/chromedp/chromedp
  dependency-version: 0.15.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:03:48 +08:00
dependabot[bot] ea4a2a39f2 chore(deps): bump github.com/aws/aws-sdk-go-v2/config
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.32.14 to 1.32.17.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.14...config/v1.32.17)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.32.17
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:03:25 +08:00
dependabot[bot] a842e2335b chore(deps): bump dompurify and @types/dompurify in /frontend
Bumps [dompurify](https://github.com/cure53/DOMPurify) and [@types/dompurify](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/dompurify). These dependencies needed to be updated together.

Updates `dompurify` from 3.4.1 to 3.4.2
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.1...3.4.2)

Updates `@types/dompurify` from 3.0.5 to 3.2.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/dompurify)

---
updated-dependencies:
- dependency-name: "@types/dompurify"
  dependency-version: 3.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: dompurify
  dependency-version: 3.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:00:58 +08:00
dependabot[bot] d2c741aeff chore(deps): bump github.com/weaviate/weaviate
Bumps [github.com/weaviate/weaviate](https://github.com/weaviate/weaviate) from 1.33.0-rc.1 to 1.37.2.
- [Release notes](https://github.com/weaviate/weaviate/releases)
- [Commits](https://github.com/weaviate/weaviate/compare/v1.33.0-rc.1...v1.37.2)

---
updated-dependencies:
- dependency-name: github.com/weaviate/weaviate
  dependency-version: 1.37.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:00:27 +08:00
dependabot[bot] f4c2934499 chore(deps): bump vue from 3.5.17 to 3.5.34 in /frontend
Bumps [vue](https://github.com/vuejs/core) from 3.5.17 to 3.5.34.
- [Release notes](https://github.com/vuejs/core/releases)
- [Changelog](https://github.com/vuejs/core/blob/main/CHANGELOG.md)
- [Commits](https://github.com/vuejs/core/compare/v3.5.17...v3.5.34)

---
updated-dependencies:
- dependency-name: vue
  dependency-version: 3.5.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:00:07 +08:00
dependabot[bot] c17a128298 chore(deps): update paddlepaddle requirement in /docreader
Updates the requirements on [paddlepaddle](https://github.com/paddlepaddle/paddle) to permit the latest version.
- [Release notes](https://github.com/paddlepaddle/paddle/releases)
- [Changelog](https://github.com/PaddlePaddle/Paddle/blob/develop/RELEASE.md)
- [Commits](https://github.com/paddlepaddle/paddle/compare/v3.0.0...v3.3.1)

---
updated-dependencies:
- dependency-name: paddlepaddle
  dependency-version: 3.3.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:59:05 +08:00
dependabot[bot] fb47d84d98 chore(deps): bump @types/papaparse from 5.5.0 to 5.5.2 in /frontend
Bumps [@types/papaparse](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/papaparse) from 5.5.0 to 5.5.2.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/papaparse)

---
updated-dependencies:
- dependency-name: "@types/papaparse"
  dependency-version: 5.5.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:58:48 +08:00
dependabot[bot] 9344907627 chore(deps): bump actions/checkout from 3 to 6
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:58:31 +08:00
dependabot[bot] 0455872111 chore(deps): bump actions/upload-artifact from 4 to 7
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:58:20 +08:00
dependabot[bot] f4fe86a842 chore(deps): bump docker/login-action from 3 to 4
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:58:11 +08:00