The Agent ran a parallel context cache (Redis/in-memory) on top of the
messages table to feed multi-turn history into the LLM. That dual-write
caused subtle drift (e.g. compression diverging from DB, system-prompt
swaps lost on restart) and required a separate ClearContext path on
session/IM clear.
Make the messages table the single source of truth:
- Add service.LoadAgentHistory: rebuilds chat.Message history per turn
from the persisted messages, expanding AgentSteps into proper OpenAI
assistant_with_tool_calls + tool messages and replaying the canonical
final answer (with <think> blocks stripped). final_answer tool calls
are filtered to avoid duplicating the trailing answer.
- Make AgentEngine stateless across turns: drop ContextManager / sessionID
cache plumbing from the engine, agent service, and CreateAgentEngine
signature. The engine only uses sessionID for logging/event emission.
- Wire AgentQA to load history from DB on demand using HistoryTurns
(default 5) when MultiTurnEnabled, otherwise run with empty history.
- Delete the llmcontext package (ContextManager interface, Redis/memory
storage, factory) and the SessionService.ClearContext API path; IM
/clear and session message clear no longer need to invalidate cache.
Behavior preserved: KnowledgeQA-mode replay is unchanged (turns with
empty AgentSteps just produce the single canonical assistant message),
and Agent-mode turns now consistently see prior tool calls and results.
Mainland China cloud VMs (Tencent Lighthouse, Aliyun, etc.) frequently
cannot reach get.docker.com, github.com, or even community GitHub
mirrors like gh-proxy.com. The cloud-image bootstrap previously had no
escape hatch for this and failed at the very first curl.
This adds a new DOCKER_INSTALL_MIRROR env var to prepare.sh. When set,
it skips get.docker.com and installs docker-ce + compose-plugin from an
apt mirror of Docker's official repo (e.g. mirrors.tencent.com,
mirrors.aliyun.com).
README.md also gets:
- A GH_PROXY env var threaded through bootstrap methods A and B so the
initial script pull can route through gh-proxy / ghfast.
- An explicit recommendation to prefer method C (scp from local) on
mainland China VMs.
- A consolidated "三件套" table mapping WEKNORA_GH_PROXY /
DOCKER_INSTALL_MIRROR / DOCKER_REGISTRY_MIRROR to per-cloud
endpoints, so users hit one place to copy the full env.
`${SEARXNG_SECRET:?...}` made the variable mandatory at compose parse time,
which forced *any* compose command (default profile included) to fail when
SEARXNG_SECRET was unset, with a message confusingly claiming the searxng
profile was being started.
Switch to `${SEARXNG_SECRET:-weknora-default-searxng-secret-...}` so the
searxng profile starts zero-config. Default deployments bind searxng to
127.0.0.1 only, so a shared default secret is acceptable; .env.example
now explicitly warns to rotate it before flipping SEARXNG_BIND=0.0.0.0,
since secret_key signs image-proxy URLs.
The SearXNG provider forwarded `parameters.api_key` as
`Authorization: Bearer <key>` for reverse-proxy auth, but the frontend
never rendered the API key input for self-hosted providers
(`requires_api_key=false`), so the path was unreachable from the UI.
Rather than expose another UI knob for a niche reverse-proxy setup that
the project does not currently support, remove the Authorization header
branch and the `apiKey` field on `SearxngProvider`. Tenants who put
SearXNG behind an authenticating reverse proxy can still front it with
network-level auth (mTLS, IP allowlist) without changes here.
`WebSearchProviderParameters.APIKey` is generic and used by other
providers, so it stays.
- Adjusted the formatting of struct fields in the SSRF whitelist test cases for improved readability and consistency.
- No functional changes were made; this is purely a code style improvement to enhance maintainability.
- Updated .env.example to clarify SEARXNG_SECRET generation and added SSRF_WHITELIST_EXTRA for improved security.
- Modified docker-compose files to bind SearXNG to localhost by default and introduced a one-time initialization service to set up settings.yml correctly.
- Enhanced SearxngProvider with stricter URL validation, ensuring no query or fragment is present in the base URL.
- Added unit tests for SearXNG validation and date parsing to ensure robustness.
- Updated frontend WebSearchSettings to reflect changes in SearXNG instance URL handling.
This commit improves the security and usability of the SearXNG integration, addressing potential misconfigurations and enhancing the developer experience.
- Added gzip compression settings to improve loading times for static assets, significantly reducing the size of files served and enhancing performance for users, especially in low bandwidth regions.
- Updated cleanup.sh to avoid recreating .env during cleanup, preventing exposure of default passwords before firstboot.
- Modified firstboot.sh to create .env from .env.example only if it doesn't exist, ensuring no sensitive data is present before initialization.
- Added support for Docker Hub and GitHub tarball download acceleration via new environment variables WEKNORA_GH_PROXY and DOCKER_REGISTRY_MIRROR.
- Implemented a mechanism to prune old WeKnora images based on the current version, reducing image size and maintaining a clean environment.
- Enhanced README.md with instructions for using the new acceleration features and image pruning options.
Updated the cleanup script to replace the broad `docker system prune` command with more targeted commands. This change ensures that only unused containers, build cache, and dangling volumes are removed, preventing the accidental deletion of important images like `wechatopenai/weknora-*` that are pre-pulled for firstboot. The script now explicitly prunes containers, builders, and dangling volumes to maintain a clean environment without disrupting the intended image setup.
Address review feedback on PR #1249:
- prepare.sh: when WEKNORA_REF looks like a version tag (v*), write the
matching WEKNORA_VERSION into .env so docker compose pulls images that
match the compose YAML's git ref (previously stuck on :latest).
- prepare.sh: detect docker binary path via `command -v docker` and
template it into weknora.service (replacing hardcoded /usr/bin/docker
that fails when docker lives in /usr/local/bin).
- firstboot.sh: write a /opt/WeKnora/.firstboot.done marker immediately
after rewriting .env, before `docker compose up -d`. If compose fails
mid-run, the next boot is gated by ConditionPathExists=!marker so we
never regenerate DB_PASSWORD against an already-initialized postgres
volume (which previously bricked the database).
- firstboot.sh: stop deleting its own unit file / script while the
oneshot is still executing; rely on the marker + `systemctl disable`
instead, avoiding "job failed" markings from systemd.
- firstboot.sh: use detected docker path instead of /usr/bin/docker;
add note in credentials file that .env is the source of truth.
- weknora-firstboot.service: add ConditionPathExists=!.firstboot.done.
- cleanup.sh: scope docker volume deletion to compose project label
(com.docker.compose.project=<name>) instead of fuzzy substring match
that could nuke unrelated postgres/redis volumes.
- cleanup.sh: also remove .firstboot.done marker, firstboot log, and
any leftover /root/weknora-credentials.txt so the image is clean.
- README.md: clarify how to actually disable registration (edit the
`replace` call list in firstboot.sh, not run that command in shell).
Recommend `sudo -i` to avoid the classic `sudo cmd >> file` failure
where the shell redirection runs as the unprivileged user. Also document
the `sudo tee -a` workaround and add a scp option C.
Add scripts and docs for packaging WeKnora into cloud images (AMI,
custom images, snapshots) so users can distribute one-click deployable
templates on any cloud provider.
- scripts/cloud-image/: cloud-agnostic prepare/cleanup/firstboot scripts
plus systemd units. Downloads only the 4 runtime files needed by the
compose stack (~100KB) instead of cloning the full repo, and pins to
any git ref via WEKNORA_REF for reproducible builds.
- firstboot.sh randomizes DB/Redis/JWT/AES secrets on first boot,
writes credentials to /root/weknora-credentials.txt and self-removes.
- docs/cloud-image/: per-platform packaging guides. Includes a guide
for Tencent Cloud Lighthouse / CVM covering image creation, sharing,
and marketplace listing.
Default-on services match the unprofiled compose stack (frontend, app,
docreader, postgres, redis); optional services (qdrant, milvus,
neo4j, langfuse, etc.) remain opt-in via compose profiles to keep the
image size small.
- Introduced a new test to ensure that empty user IDs are rejected when a waiter has a user, preventing unauthorized approvals.
- Added RequestNonce to resolve messages to uniquely identify Resolve calls, ensuring concurrent requests do not interfere with each other.
- Updated waiter structure to use atomic operations for the resolved state, improving thread safety.
- Enhanced error handling in the Resolve method to properly manage user mismatches and ensure accurate acknowledgment delivery across instances.
- Updated MCP service handler to reject unauthenticated requests early, providing clearer feedback on authorization requirements.
Add an opt-in human approval gate so Agent runs pause before executing
MCP tools that operators flag as dangerous, surface an approval card in
the chat UI, and only resume after the user approves (optionally with
edited args) or rejects.
Backend
- New mcp_tool_approvals table + repo/service to mark per-tool approval
required (PG migration 000042 + sqlite init).
- approval.Gate coordinates RequestAndWait / Resolve with sync.Once
delivery, configurable timeout, and Redis Pub/Sub fan-out so multi-
replica deployments work without sticky sessions.
- MCPTool.Execute integrates the gate; uses a round-level ApprovalCtx
(without the per-tool 60s timeout) for the wait, and re-derives a
fresh 60s exec ctx after approval so CallTool keeps a full window.
- New SSE response types (tool_approval_required / _resolved) and
EventBus events plumb approval state to AgentStreamDisplay.
- REST: list/set per-tool approval flag, resolve pending approval.
- Configurable via agent.tool_approval_timeout_seconds (yaml) or
WEKNORA_AGENT_TOOL_APPROVAL_TIMEOUT env (accepts seconds or Go
duration).
Frontend
- MCP settings: per-tool "require approval" switch on the test panel.
- Chat: ToolApprovalCard renders the pause point with editable JSON
args, validation feedback, mm:ss countdown that turns warning/danger
near deadline, and a resolved state that retains context.
- i18n strings added for zh-CN / en-US / ko-KR / ru-RU.
Docs
- docs/zh/mcp-approval.md covering behavior, config, API, deployment
considerations (Redis cross-instance, restart limitations).
- repo: drop r.db.Debug() from FindSimilarPages — it was dumping every
trigram probe's SQL+args (per-alias, per-item) into production logs.
- wiki_ingest dedup: fix Printf format string ("selected for %d new
items" had two args), and harden validMerge against un-prefixed
slugs whose strings.Index returned -1 and silently passed the type
check.
- wiki_ingest_batch: drop the duplicated loggedBatchSize/MapPar/
ReducePar assignments.
- asynqdl: record the real attempt count (retried + 1) on the dead
letter row instead of a hard-coded 0; tighten payloadProbe to the
set of field names with consistent semantics across payloads
(drop source_id/target_id/target_kb_id which differ by task type).
- asynqdl tests: update for the trimmed probe and assert FailCount=0
outside an asynq worker ctx so the semantics stay pinned.
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.
Updated the go.mod and go.sum files to bump the version of github.com/go-openapi/swag and its related packages to v0.26.0. Removed the indirect dependency on github.com/josharian/intern as it is no longer needed.
The ollama Go SDK changed ToolCallFunctionArguments from
map[string]any to a struct backed by an ordered map. Use ToMap() for
read access and NewToolCallFunctionArguments()+UnmarshalJSON for
construction so the build matches the upgraded SDK.
- Bump base image in docker/Dockerfile.app from golang:1.24 to golang:1.26
to match `go 1.26` declared in go.mod (fixes CI build failure on
`go mod download`).
- Drop unused docreader components and their dependencies:
- Remove `docreader/ocr/` package (paddle/vlm/dummy backends are
unreferenced by the main flow; OCR/VLM is handled by the Go App).
- Remove `docreader/parser/storage.py` (dead code; image persistence
happens in the Go App via inline ImageRef bytes).
- Remove `docreader/scripts/download_deps.py` (PaddleOCR pre-download).
- Drop deps: paddleocr, paddlepaddle, openai, ollama, minio,
cos-python-sdk-v5, oss2, asyncio, pypdf2, markdown, mistletoe,
goose3, markdownify, pdfplumber, antiword, urllib3.
- Re-lock uv.lock: 145 -> 79 packages.
- Update docreader/README.md to reflect that OCR/VLM/storage are no
longer configured at the docreader level.