Commit Graph

784 Commits

Author SHA1 Message Date
Mao Meng b48adde3e6 fix:attachment_document_failed 2026-04-29 22:34:13 +08:00
wizardchen b2975f7d1d refactor(im): move adapter factories into per-platform subpackages
registerIMAdapterFactories in internal/container/container.go had grown
to ~290 lines of platform-specific config parsing and runtime start/stop
logic across 7 IM platforms, plus three private credential helpers. The
container package is meant to wire dependencies, not own each platform's
credentials schema and websocket lifecycle.

Extract each platform's factory into its own subpackage file and expose
the credential helpers on the im package:

- New im.ParseCredentials / im.GetString / im.GetBool (formerly the
  unexported parseCredentials / getString / credentialBool in container).
- New im.ResolveMode helper collapses the six identical "default to
  websocket when Mode is empty" blocks.
- New {wecom,feishu,slack,telegram,dingtalk,mattermost,wechat}.NewFactory
  returns an im.AdapterFactory; container.go now calls each one in a
  single line.

Behavior is preserved: adapter construction, goroutine start/stop,
log messages, default modes (mattermost still defaults to "webhook";
others to "websocket"), and error strings are unchanged. The WeCom
corp_agent_id float64/int switch and the WeChat long-poll-only path
are preserved as-is.

Net: container.go drops ~320 lines and no longer needs to be touched
when adding or tuning an IM platform.
2026-04-29 20:08:22 +08:00
wizardchen 7fcd92994b feat(tenant): expose API Key reset from the API Info page
Add a danger-themed reset button next to the existing copy/eye buttons
on the API Info settings page. Confirmation dialog warns that the old
key is revoked immediately; on success the new plaintext key replaces
the displayed value and is auto-revealed so the user can copy it.

Backend: new POST /api/v1/tenants/:id/api-key handler that wraps the
existing TenantService.UpdateAPIKey; access is gated by the same
authorizeTenantAccess check as other tenant endpoints. The handler
returns the freshly generated plaintext key, while the encrypted form
is persisted to the database.

Frontend: new resetTenantApiKey API client, reset button + confirm
dialog wiring in ApiInfo.vue, plus matching i18n entries for zh-CN,
en-US, ko-KR, and ru-RU.
2026-04-29 20:07:38 +08:00
lyingbug cbdfba0075 Merge pull request #1086 from Windfarer/storage-allow-list
feat: add STORAGE_ALLOW_LIST env var
2026-04-29 19:49:46 +08:00
wizardchen c89fd5da98 fix(tenant): return plaintext API key after create/reset
CreateTenant and UpdateAPIKey both overwrote tenant.APIKey with the
AES-encrypted ciphertext before returning, causing the UI to display
"enc:v1:..." right after generating or resetting an API key. Save the
plaintext in a local variable, encrypt only the value handed to
UpdateTenant, and return the plaintext to callers.

Subsequent reads via GetTenantByID continue to rely on the AfterFind
hook for transparent decryption, so this only affects the initial
write-back path.
2026-04-29 19:46:21 +08:00
Windfarer 20be935fe0 Merge branch 'main' into storage-allow-list 2026-04-29 19:35:13 +08:00
Windfarer 6601a483be feat: add STORAGE_ALLOW_LIST env var 2026-04-29 18:58:09 +08:00
wizardchen da90867dd4 feat(ui): refactor IM channels list layout and form\n\n- Change IM channel form from dialog to drawer for better UX\n- Convert channel list from vertical layout to responsive grid cards\n- Move edit/delete actions into a dropdown menu for cleaner card design\n- Replace platform radio group with a select dropdown in the add channel form 2026-04-29 17:34:20 +08:00
nullkey 48a764b866 refactor(knowledge-base): address review feedback
Backend
- BatchDeleteKnowledge handler now enqueues an asynq
  TypeKnowledgeListDelete task instead of calling DeleteKnowledgeList
  synchronously, matching the pattern used by ClearKnowledgeBaseContents.
  Avoids long HTTP timeouts on large batches and shares the existing
  async cleanup pipeline.
- Extract a small enqueueKnowledgeListDelete helper on KnowledgeHandler
  so BatchDeleteKnowledge and ClearKnowledgeBaseContents share the
  payload/marshal/enqueue boilerplate.

Frontend (DocumentListView)
- softer outer border and lighter row separators
- header background switched to the page tone with placeholder-color
  text, reducing institutional gray
- selected row uses an inset left accent bar and a quieter brand tint
  instead of a saturated background; hover stays neutral
- consolidate the .row-more-btn { opacity: 1 } rule that was duplicated
  across the hover and selected blocks
2026-04-29 12:01:38 +08:00
nullkey be8e53bf08 feat(knowledge-base): add document batch delete and list view
Add a list-view alongside the existing card grid (issue #957) and
multi-select batch delete (issue #1045) for documents inside a knowledge
base. The two views are toggled from the toolbar and the preference is
persisted in localStorage. Selection state is shared between views with
shift-range support and a sticky batch-action bar.

Backend
- POST /api/v1/knowledge/batch-delete: validates KB scope and editor
  permission, then delegates to the existing DeleteKnowledgeList service
  which already cascades vector/file/graph cleanup. Caps batch size at
  200 and uses a single GetKnowledgeBatch call for membership validation.

Frontend
- DocumentListView: table layout with sticky header, checkbox column
  (with indeterminate select-all), file-type icons, status badges,
  per-row action menu.
- DocumentBatchBar: floating pill bar with selected count and delete
  action; appears when items are selected.
- KnowledgeBase.vue: view-mode toggle, selection state, batch-delete
  confirmation dialog, hover/active checkbox overlay on existing cards.
- Extracted formatFileSize and getFileIcon to utils/files.ts to share
  with the new list view and avoid further drift from the existing
  inline copies.
- i18n keys added for zh-CN, en-US, ko-KR, ru-RU.

Closes #1045
Closes #957
2026-04-29 12:01:38 +08:00
wizardchen d55b52652c fix(docparser): preserve standalone image uploads from icon filter
When the uploaded file is itself an image, the image reference now carries
an IsOriginal flag so ResolveAndStore skips the small-icon size filter.
Otherwise small standalone images (e.g. avatars below 64x64) were silently
dropped before reaching multimodal OCR/caption processing.
2026-04-29 11:58:02 +08:00
hjz 535aec87e3 feat: data_analysis sql validation & type processing 2026-04-28 23:47:34 +08:00
wizardchen d812770806 fix: inject KB document listing on fallback for broad queries (#959)
When users ask broad queries like "请整理知识库中的数据" in RAG mode,
vector/keyword search returns nothing because the query has no specific
content to match. The user needs to see what documents exist, not search
results.

- Add buildKBDocumentListing() to fetch document titles/filenames from
  the knowledge base and inject them into the fallback prompt via
  {{kb_documents}} placeholder
- Update fallback prompt templates (model_fallback, default_fallback_prompt)
  to include document listing context so the LLM can guide users
- Improve rewrite prompt: tighten summarize vs kb_search classification,
  expand kb_search to cover browse/organize/list operations, add examples

Closes #959
2026-04-28 23:31:04 +08:00
wizardchen 80a007cc06 fix(im): remove pipeline-level timeout that kills multi-round agent reasoning
The IM service applied a hard 120s deadline (`qaTimeout`) to the entire
QA pipeline context. Multi-round ReAct agents easily exceed this — when
the deadline fires mid-stream the LLM provider returns "context deadline
exceeded" which leaks into the user-visible answer (fixes #1000).

Changes:
- Replace `context.WithTimeout(ctx, qaTimeout)` with `context.WithCancel(ctx)`
  in both `handleMessageStream` and `runQA`, matching the web handler's
  approach. Each agent round still has its own per-call LLMCallTimeout
  (default 120s) which is sufficient.
- Filter `ResponseTypeError` stream chunks in `streamLLMToEventBus` so
  error messages (e.g. "context deadline exceeded") are never appended
  to the LLM content. Instead they are captured in `StreamError` and
  surfaced as a Go error when no usable content was produced.
- Bump Redis inflight mapping TTL from qaTimeout+30s to 10 minutes to
  accommodate longer agent runs.
- Remove the now-unused `qaTimeout` constant.
2026-04-28 21:17:56 +08:00
wizardchen c578fdbad6 fix(im): isolate IM sessions per agent and recover from deleted sessions
Three bugs fixed:

1. Session cross-contamination (#1066): resolveUserSession and
   resolveThreadSession looked up ChannelSession by (platform, user_id,
   chat_id, tenant_id) without agent_id. When the same user talked to
   two bots bound to different agents under the same tenant, they shared
   a session — mixing knowledge base context. Add agent_id to all four
   WHERE clauses and update the DB unique indexes to match.

2. Swapped parameters in resolveThreadSession: the SQL expected
   (chat_id, thread_id) but Go args passed (threadID, msg.ChatID).

3. Orphaned ChannelSession crash (#1046): deleting a session from the
   WeKnora UI soft-deletes it (GORM), but the ChannelSession row
   survives because soft-delete doesn't trigger SQL ON DELETE CASCADE.
   Subsequent IM messages hit "record not found" on GetSession and the
   bot becomes permanently unresponsive. Now detect this case, recycle
   the stale mapping, and transparently create a fresh session.
2026-04-28 20:53:06 +08:00
wizardchen a4c0832007 fix(search): align rerank priority between Execute and rerankResults
The outer if/else in Execute claimed to prefer chatModel (LLM-based
reranking), but both branches called rerankResults which internally
prefers rerankModel first. This made the outer branching dead code
with a misleading comment.

Consolidate into a single branch that delegates entirely to
rerankResults, whose actual priority is: rerankModel → chatModel → none.
2026-04-28 20:23:57 +08:00
goodnight ca1c8074bc fix(container): aggregate registration errors in connector registry initialization 2026-04-28 20:02:13 +08:00
wizardchen 1f03741462 fix(wiki): Lite ingest lock, failed-op requeue, sync task retry parity
- Add per-KB sync.Map lock in Lite mode to mirror Redis SetNX concurrency.
- Requeue failed wiki ops in Lite by enqueueing asynq tasks with delay/tracing.
- Teach SyncTaskExecutor to honor ProcessIn delay and MaxRetry like Redis asynq.
2026-04-28 18:27:00 +08:00
wizardchen dd524ffb1c feat: enhance Scan method in RetrieverEngines to support legacy and current formats
- Update Scan method to handle both legacy bare-array format (e.g. [{...}, {...}]) and current object-wrapped format (e.g. {"engines": [{...}, {...}]}).
- Improve error handling for unmarshalling failures, providing clearer feedback on format issues.

This change ensures backward compatibility while allowing for the new data structure.
2026-04-28 17:21:56 +08:00
wizardchen c34f7b6254 Enhance wiki ingest process to handle failed operations
- Introduced a mechanism to track and requeue failed operations during the wiki ingest process.
- Added a new `requeueFailedOps` function to append failed operations back to the Redis pending list for retry in subsequent batches.
- Updated the `ProcessWikiIngest` method to collect failed operations and ensure they are retried after trimming the pending list.

This change improves the robustness of the ingest process by preventing data loss from transient failures.
2026-04-28 17:21:42 +08:00
wizardchen 13260b831c Fix wiki ingest silent data loss from malformed JSON in Redis queue
Two critical fixes to the peekPendingList function:

1. Add error logging for JSON unmarshal failures (line 303)
   - Previously: Malformed JSON items were silently skipped with no indication
   - Now: Each failure is logged with the error and raw item content (first 100 chars)
   - This makes data loss visible and debuggable

2. Fix trim count mismatch (line 325)
   - Previously: Returned len(result) - the count of raw Redis items peeked
   - Now: Returns len(ops) - the count of successfully parsed items
   - Impact: Malformed items now stay in Redis for retry instead of being discarded
   - This prevents silent data loss when JSON encoding issues occur

Root cause of the reported issue:
- When 8 files are uploaded in a batch, if one has encoding issues that cause
  JSON.Unmarshal to fail, it was silently dropped from processing
- The file was trimmed from Redis without being processed
- Result: Only 7 of 8 files appeared in the Wiki

Fix approach:
- Make unmarshal failures visible in logs
- Retry malformed items in the next batch instead of discarding them
- Admin can see the errors and take corrective action

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:42 +08:00
hjz efae3d466f feat: correct data analysis errors using fileService. 2026-04-27 13:18:08 +08:00
goodnight a4fe86f828 fix(agent): improve tenant context handling and error reporting 2026-04-24 22:40:57 +08:00
wizardchen c12296aa88 feat(observability): instrument agent ReAct loop and tool calls in Langfuse
The existing Langfuse integration covered Chat / Embedding / Rerank / VLM /
ASR generations plus the HTTP + asynq spans, but the agent's own execution
tree was invisible: tool calls never appeared, multi-round ReAct iterations
were flat under the HTTP trace, and there was no single node representing
"one agent run".

This change adds three levels of agent-side spans:

  - agent.execute       — wraps AgentEngine.Execute, records query preview,
                           knowledge bases, allowed tools, final-answer length
                           and totals on finish.
  - agent.round.<N>     — wraps each ReAct iteration; records finish_reason,
                           tool-call count, token usage and duration.
  - agent.tool.<name>   — wraps each tool invocation; records arguments,
                           success, duration, output preview (rune-safe, 4KB
                           cap), error, data keys and image count.

To keep the loop's many exit paths (natural stop, stuck loop, empty-content
retry, final_answer, context cancellation) span-safe, the iteration body was
extracted into runReActIteration with a single defer span.Finish() and an
iterOutcome sentinel driving the outer loop. database_query arguments are
redacted (keys only) to avoid leaking raw SQL into the observability
backend, mirroring the existing UI hint policy.

Adds unit tests for the new helpers (truncateForLangfuse, argKeys, dataKeys,
finishToolSpan nil-safety, iterOutcome.String).
2026-04-24 19:58:08 +08:00
wizardchen 8103398fcf fix(agent): materialize knowledge files to temp path for DuckDB (#1007)
Dev-mode repro from the issue reporter's logs:

  [Tool][DataAnalysis] Failed to create table from Excel: IO Error:
  GDAL Error (4): Failed to open file
  'local://10000/.../1777030910246871000.xlsx': No such file or directory

The local file service returns a custom 'local://' URL from GetFileURL,
which the duckdb spatial/excel extensions can't resolve (they expect
plain paths or http(s)/s3:// style URIs). Presigned HTTPS URLs from
cloud backends worked by accident; the 'local://' dev path has always
been broken for the Data Analysis tool.

Instead of per-scheme adapters, stream the file through
FileService.GetFile into a temp file and hand DuckDB the resulting
filesystem path. This works uniformly across every backend (local,
OSS, S3, MinIO, COS) and survives future scheme changes without any
changes to the Data Analysis tool.

- Preserve the original file extension on the temp file so DuckDB's
  format auto-detection (csv / xlsx / xls) still kicks in.
- Clean up the temp file when LoadFromKnowledge returns, including on
  every error branch. Cleanup is idempotent (double-invoke safe).
- New tests stub FileService to lock in that (a) we never leak a
  provider:// URL to DuckDB, (b) GetFile errors propagate, (c) the
  extension survives case normalization.

Refs: https://github.com/Tencent/WeKnora/issues/1007
2026-04-24 19:57:56 +08:00
wizardchen 002d98a640 test(agent): add end-to-end DuckDB excel tests and fix sheet enumeration (#1007)
Catching up on a real-world check: when a multi-sheet .xlsx fixture is
fed through LoadFromExcel against an in-memory DuckDB with the spatial
and excel extensions, the first iteration's sheet enumeration quietly
fell back because st_read_meta doesn't expose a scalar 'layer_name'
column — it returns a LIST<STRUCT> column called 'layers'. We now
UNNEST(layers).name so enumeration actually works, and assert the full
data path with three DuckDB-backed tests:

  - multi-sheet workbook yields the sum of all rows and per-sheet
    breakdown via __sheet_name (including schema drift: columns only
    present in one sheet are NULL for the other)
  - single-sheet workbook still works and tags rows with its sheet name
  - sheet name containing a single quote (Q1'24) survives the SQL
    literal round-trip

excelize/v2 is added as a test-time helper to build the fixtures
deterministically. If DuckDB's spatial/excel extensions can't be
installed (offline CI), the tests skip rather than fail.

Refs: https://github.com/Tencent/WeKnora/issues/1007
2026-04-24 19:57:56 +08:00
wizardchen 11811f9bf8 fix(agent): load all sheets from Excel files in data analysis (#1007)
DuckDB's st_read (spatial) only reads the first layer/sheet of a .xlsx
workbook, so every sheet beyond Sheet1 was silently dropped from the
DuckDB table the Data Analysis tool builds. Users trying to analyse
multi-sheet workbooks could only see the first sheet.

Switch to DuckDB's dedicated 'excel' extension (read_xlsx) for the
actual data load, enumerate sheets via the spatial extension's
st_read_meta, and UNION ALL BY NAME the rows of every sheet into one
table. A synthetic __sheet_name column records the source so the LLM
can still filter/aggregate per sheet; schema drift between sheets is
tolerated via UNION BY NAME. If enumeration fails (older DuckDB, local
filesystem errors, …) we fall back to reading the first sheet so the
tool stays usable.

- Install & LOAD the 'excel' extension alongside 'spatial' both at
  startup (internal/container) and in the offline prefetch binary
  (cmd/download/duckdb).
- Harden sheet/path handling against single quotes.
- Update the tool description so the agent knows about __sheet_name.
- Add unit tests for the CREATE TABLE SQL builder covering 0 / 1 / N
  sheets and quote escaping.

Refs: https://github.com/Tencent/WeKnora/issues/1007
2026-04-24 19:57:56 +08:00
wizardchen beca2b89a3 feat(observability): extend Langfuse tracing across asynq pipeline
Previously the Langfuse integration only traced in-process HTTP requests
(chat / search / eval), so file uploads and every downstream asynq task
(document parse, chunk embedding, OCR/VLM, summary / question gen, wiki
ingest, datasource sync, etc.) produced either disconnected shallow
traces or no observation at all.

This change threads one trace end-to-end:

- tracer: add SPAN observation type and StartSpan; add ResumeTrace so a
  worker can attach to an upstream trace without emitting a duplicate
  trace-create; StartGeneration now auto-picks parentObservationId from
  ctx so nested trace -> span -> generation trees render correctly.
- types.TracingContext + LangfuseTracingCarrier: embed on all 17 asynq
  payloads so trace_id / parent_obs_id / user_id / session_id serialise
  into every job.
- langfuse.InjectTracing: injected at 28 enqueue sites before json.Marshal
  so the HTTP-layer trace survives the Redis hop.
- langfuse.AsynqMiddleware: mux.Use hook that peeks the payload, either
  resumes the upstream trace or opens a standalone asynq.<type> trace
  for scheduled jobs, and wraps the handler in a SPAN with task metadata
  (id / queue / retry / payload_bytes) plus ERROR level on failure.
- GinMiddleware.shouldTrace: whitelist ingestion / knowledge-mutation /
  FAQ / wiki / datasource endpoints so the root trace actually starts.
- Tests: tracer_test.go covers span nesting, error status, and
  ResumeTrace no-trace-create guarantee; asynq_test.go covers
  InjectTracing round-trip, middleware resume path, and standalone
  trace fallback.
- Docs: docs/Langfuse\u96c6\u6210.md now lists the covered task types
  and documents the cross-process propagation model.

No behavioural change when Langfuse is disabled (all new code paths are
no-ops and carriers serialise to empty strings with omitempty).
2026-04-24 13:16:47 +08:00
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
wizardchen b8c45b173e fix(agent-share): don't require rerank model for wiki-only agents
The share-time validation forced every agent bound to a knowledge base to
have a rerank model configured, which blocked Wiki-type agents
(`wiki_search` / `wiki_read_page` / …) from being shared even though they
never invoke the reranker at runtime.

Align the check with the runtime logic in `session_agent_qa.go`: only
require a rerank model when the agent's allowed_tools include
`knowledge_search` (or rely on the default tool set, which contains it).
Wiki-only agents can now be shared without a rerank model; hybrid / RAG
agents still enforce the requirement.

Also update the handler error message to reflect the new rule.
2026-04-24 10:21:05 +08:00
wizardchen fd3e2992b1 feat(models): support custom HTTP headers across all remote model calls
Introduce per-model `custom_headers` config (similar to OpenAI Python SDK's
`extra_headers`) so users can inject gateway auth tokens, trace IDs, etc.
into every outbound API request for chat / embedding / rerank / VLLM / ASR
models. Reserved headers like Authorization / Content-Type are always
preserved to avoid breaking auth or signing flows.

Along the way, unify production and "test connection" paths onto a single
`ConfigFromModel(*types.Model, appID, appSecret)` constructor in each
model package. Both `service.modelService.GetXxxModel` and the four
`handler.initialization.Check*Model` / `TestEmbeddingModel` endpoints now
go through the same field mapping, so new parameters only need to be
added in one place.

Backend:
- Add `CustomHeaders map[string]string` to `types.ModelParameters` with
  JSON/YAML `omitempty` tags for forward compatibility.
- New `internal/utils/extraheaders.go` providing `ApplyCustomHeaders`
  (for hand-rolled HTTP paths) and `WrapHTTPClientWithHeaders` /
  `CustomHeadersRoundTripper` (for SDK paths like go-openai). Reserved
  headers (Authorization, api-key, Content-Type, Accept, Host,
  Content-Length, User-Agent) are filtered out.
- Inject headers in every remote model path: chat (SDK + raw HTTP),
  embedding (openai/aliyun/jina/volcengine/nvidia/azure_openai), rerank
  (remote_api/aliyun/jina/nvidia/zhipu — via shared `customHeaderSetter`
  interface), VLM and ASR (via http.Client wrapping).
- Add `ConfigFromModel` to each of chat/embedding/rerank/vlm/asr,
  consolidating field mapping (ExtraConfig, CustomHeaders, InterfaceType
  defaulting, WeKnoraCloud credentials, etc.) and cover it with
  dedicated unit tests per package.
- Refactor `service/model.go`: replace ~100 lines of hand-written Config
  literals with one-line `ConfigFromModel` calls; drop the now-unused
  `stringMapToAnyMap` helper.
- Refactor `handler/initialization.go` test-connection endpoints: merge
  four ad-hoc request structs into one `ModelTestRequest`, extract
  `buildTestModel` and `resolveTenantWeKnoraCloudCreds` helpers, and
  route all four endpoints through `ConfigFromModel` + `NewXxx` so the
  test path is now behaviorally identical to production.

Frontend:
- Add `custom_headers` to the `ModelConfig` API type.
- `ModelEditorDialog.vue`: new Key-Value header editor (add/remove rows),
  auto-converts map <-> array when loading/saving, and passes the
  current header set to the `Test Connection` button so the preview
  reflects exactly what production will send.
- `ModelSettings.vue`: serialize the header array back into a map for
  the backend (drops empty rows).
- i18n: add custom-header labels / descriptions / placeholders to
  zh-CN, en-US, ru-RU, ko-KR.

Tests:
- `internal/utils/extraheaders_test.go` covers reserved-header filtering
  and round-tripper wrapping (including nil-client fallback).
- New `config_from_model_test.go` in chat / embedding / rerank / vlm /
  asr verifies all fields (CustomHeaders, ExtraConfig, InterfaceType
  defaulting, AppID/AppSecret) are propagated end-to-end.
2026-04-23 23:36:40 +08:00
wizardchen a6d4266c93 fix(agent): terminate ReAct loop on final_answer regardless of arg parse success
The agent loop previously kept running when the LLM emitted final_answer with
malformed JSON arguments: observe.analyzeResponse strictly json.Unmarshal'd the
raw args and returned isDone=false on failure, while act.runToolCall separately
repaired the JSON and executed the tool. The mismatch made final_answer
non-terminal, so the LLM saw its own answer in the next round's tool result
and re-emitted final_answer, surfacing near-identical answers multiple times
to the user (issue #1008).

Fix by unifying parsing in a new tools.ParseFinalAnswerArgs helper with three
fallbacks (strict -> RepairJSON -> regex extraction) and making final_answer
always terminal in analyzeResponse. When no answer can be recovered, emit a
user-visible fallback message and a Done=true event so the UI still resolves.

Also route final_answer.Execute through the same helper so tool execution and
loop termination stay consistent.

Adds regression tests in observe_test.go and final_answer_test.go covering
valid args, RepairJSON recovery, unrecoverable args, and garbage input.

Refs: Tencent/WeKnora#1008
2026-04-23 19:36:28 +08:00
wizardchen 1ec43b328d refactor: Simplify WeKnoraCloud request customization and message conversion
- Removed redundant struct definitions for WeKnoraCloud requests and messages, streamlining the customization process.
- Updated the weKnoraCloudRequestCustomizer function to directly modify the request object, enhancing clarity and reducing complexity.
- Improved the convertToWeKnoraCloudMessagesFromOpenAI function to maintain the original message structure while ensuring compatibility with WeKnoraCloud requirements.
2026-04-23 17:52:43 +08:00
goodnight 7bf09d2ac5 fix : mattermost handle io.ReadAll errors in client file 2026-04-23 17:51:07 +08:00
wizardchen 4c906e0431 feat: Enhance agent tool compatibility and knowledge base filtering
- Introduced a unified capability requirements system for agent tools, ensuring consistent filtering of knowledge bases based on their capabilities.
- Implemented derived `kb_filter` logic to streamline compatibility checks between agent tools and knowledge bases, reducing redundancy in configuration.
- Updated frontend components to reflect new filtering logic, including improved handling of empty states and user feedback when no compatible knowledge bases are available.
- Enhanced internationalization support by adding relevant messages for tool compatibility issues across multiple languages.
2026-04-23 17:28:00 +08:00
wizardchen 9f916181db refactor: Simplify logging in RemoteAPIChat HTTP requests
- Removed header printing from the chatWithRawHTTP method to streamline logging.
- Updated log message to focus on endpoint and model name, enhancing clarity while reducing verbosity.
2026-04-23 11:22:44 +08:00
wizardchen c5494f612b feat: Implement auto-linked content updates in wiki page management
- Introduced UpdateAutoLinkedContent method to persist changes from machine-only link decorators without incrementing the version number, ensuring user-facing revisions remain consistent.
- Updated existing methods to utilize the new auto-linked content update logic, enhancing the handling of cross-link injection and dead-link cleanup.
- Enhanced the KnowledgeBase and WikiBrowser components to improve user experience during wiki operations, including status tracking and indexing indicators.
- Added scheduling for wiki status probes to provide timely feedback on indexing processes after user-triggered actions.
2026-04-23 11:16:47 +08:00
wizardchen 1d5b86ddf5 feat: Enhance wiki cleanup and reparse logic for improved consistency
- Updated the scrubWikiPendingIngest method to accept a reason parameter, improving logging clarity during wiki cleanup and reparse operations.
- Introduced prepareWikiForReparse to maintain the integrity of the pending ingest queue during reparse events, preventing stale operations from executing.
- Enhanced the mapping logic in the wiki ingest process to differentiate between stale and reparse updates, ensuring accurate handling of knowledge contributions.
- Improved logging to provide better insights into the number of reparse and stale slugs processed during wiki ingestion.
2026-04-23 10:47:53 +08:00
wizardchen ffc4eeaf57 feat: Refactor wiki page update logic for improved version control
- Enhanced the UpdatePage and UpdateMeta methods to differentiate between user-visible content changes and bookkeeping updates, ensuring the version number is only incremented for actual content modifications.
- Updated the WikiPage struct and associated interfaces to reflect the new versioning policy, improving clarity on when the version is modified.
- Improved documentation for methods to clarify their intended use and behavior regarding versioning and metadata updates.
2026-04-22 21:28:20 +08:00
wizardchen 50c1c70334 feat: Update agent type presets and prompt templates for improved knowledge base handling
- Changed `kb_selection_mode` in agent type presets from "selected" to "all" to allow agents to access all knowledge bases by default.
- Revised prompt templates to clarify the handling of bound knowledge bases, replacing the `{{knowledge_bases}}` placeholder with a reference to the `<bound_knowledge_bases>` block in the user message's `<runtime_context>`.
- Enhanced internal logic to ensure that only searchable knowledge bases are considered during retrieval, improving the efficiency of knowledge searches.
- Added new functions for better formatting and handling of knowledge base metadata in the runtime context.
- Updated documentation and comments for clarity on the changes made to knowledge base interactions.
2026-04-22 21:18:34 +08:00
wizardchen a8a3d9e694 feat: Enhance wiki extraction and configuration options
- Introduced extraction granularity settings in the wiki configuration, allowing users to control the level of detail in entity and concept extraction (focused, standard, exhaustive).
- Updated the UI to reflect these new options, including tooltips for better user guidance.
- Refactored the knowledge base editor to support the new extraction granularity feature, ensuring a seamless user experience.
- Improved the backend logic for wiki ingest tasks to utilize the new granularity settings, enhancing the accuracy of extracted content.
- Added tests to validate the new extraction granularity functionality and ensure robust performance across different configurations.
2026-04-22 21:18:32 +08:00
wizardchen da8ba98622 feat: Refactor knowledge base service and router for improved wiki handling
- Updated the knowledge base service to ensure WikiConfig is created when wiki indexing is enabled, enhancing backward compatibility.
- Removed legacy syncing of WikiConfig.Enabled to IndexingStrategy.WikiEnabled, streamlining the logic for wiki enablement.
- Introduced a custom retry delay function for asynq tasks to handle wiki ingest lock conflicts more effectively, improving task management.
- Cleaned up the container file by removing unnecessary whitespace, contributing to better code readability.
2026-04-22 21:18:32 +08:00
wizardchen b3484592e7 feat: Remove RebuildKnowledgeBaseIndex and associated handler from knowledge service
- Deleted the RebuildKnowledgeBaseIndex function from the knowledge service, which was responsible for reprocessing all documents in a knowledge base.
- Removed the RebuildIndex handler from the KnowledgeBaseHandler, eliminating the endpoint that triggered the rebuild process.
- Updated the router to remove the route for rebuilding the index, streamlining the knowledge base management functionality.
2026-04-22 21:18:31 +08:00
wizardchen afa58a3610 feat: Update built-in agent display order and clarify internal agent usage
- Added comments to clarify the purpose of the `builtinAgentIDsOrdered` list, specifically noting the exclusion of `BuiltinWikiFixerID` to prevent clutter in the user-facing agent list.
- This change enhances the understanding of the built-in agent configuration and maintains a cleaner user interface for agent selection.
2026-04-22 21:18:30 +08:00
wizardchen 7e1632a393 feat: Improve knowledge deletion process by conditionally handling embedding models
- Updated the DeleteKnowledge and DeleteKnowledgeList functions to skip vector store cleanup for knowledge entries without an embedding model, preventing unnecessary errors.
- Enhanced logging to provide clear information when skipping cleanup for knowledge without embeddings, improving overall robustness and clarity in the deletion process.

These changes ensure that the deletion process is more efficient and user-friendly, particularly for knowledge bases that do not utilize embeddings.
2026-04-22 21:18:30 +08:00
wizardchen 586d5fda58 feat: Implement conditional indexing for multimodal chunks based on embedding model availability
- Added a check to skip vector and keyword indexing for knowledge bases that do not require an embedding model, preventing errors during the indexing process.
- Ensured that chunks are marked as indexed even when skipping the indexing step, maintaining a consistent state for downstream processes.
- Enhanced logging to provide clear information on the indexing status and any issues encountered while updating chunk statuses.

These changes improve the robustness of the indexing process in the ImageMultimodalService, ensuring better handling of knowledge bases without embedding capabilities.
2026-04-22 21:18:29 +08:00
wizardchen 37a517612d feat: Enhance agent retrieval strategies and regex handling
- Updated the agent system prompt to emphasize the use of regex for chunk searches, improving retrieval accuracy and efficiency.
- Introduced a new `Capabilities` field in the `KnowledgeBaseInfo` struct to define the retrieval surfaces available for each knowledge base, guiding the agent's strategy selection.
- Enhanced the `grep_chunks` tool to support regex queries, allowing for more flexible and powerful text pattern matching in knowledge base chunks.
- Improved JSON handling in the `RepairJSON` function to address invalid escape sequences, particularly for regex patterns, ensuring robust parsing and error handling.
- Updated documentation and comments across various files to clarify usage and expectations for regex and knowledge base capabilities.

These changes significantly enhance the agent's ability to retrieve relevant information while ensuring the integrity of input data handling.
2026-04-22 21:18:21 +08:00
wizardchen 4022f134bd feat: Introduce agent type presets for smart-reasoning mode
- Added a new configuration file `agent_type_presets.yaml` to define various agent type presets, including RAG Q&A, Wiki Q&A, Hybrid (Wiki + RAG), and Data Analysis.
- Updated the `builtin_agents.yaml` to associate built-in agents with the new agent type presets, enhancing their configuration options.
- Enhanced the agent system prompt templates to reflect the new agent types and their functionalities.
- Implemented API endpoints to retrieve agent type presets, allowing for dynamic loading in the frontend.
- Updated the frontend to support agent type selection in the agent editor modal, improving user experience by auto-filling relevant configurations based on selected presets.

These changes significantly enhance the flexibility and usability of the agent system, allowing users to easily configure agents based on predefined types.
2026-04-22 21:18:20 +08:00
wizardchen 0531700518 feat: Enhance suggested questions functionality with wiki integration
- Updated the SuggestedQuestion interface to include 'wiki' as a new source type, allowing for more diverse question origins.
- Implemented ListRecentForSuggestions method in the wikiPageRepository to retrieve recent wiki pages for fallback suggestions, improving the agent's ability to generate relevant questions when other sources are unavailable.
- Enhanced the customAgentService to utilize the new wiki page suggestions, ensuring a broader range of questions can be generated from wiki content.

These changes significantly improve the agent's suggestion capabilities by incorporating wiki content, enhancing user interaction and information retrieval.
2026-04-22 21:18:20 +08:00
wizardchen fdccb68ee4 feat: Improve wiki cleanup and ingest handling for deleted knowledge
- Enhanced the `cleanupWikiOnKnowledgeDelete` function to pass the full knowledge object, allowing for more accurate title and summary retrieval during wiki cleanup.
- Introduced a tombstone mechanism in Redis to mark recently deleted knowledge, enabling in-flight wiki ingest tasks to skip processing for deleted documents.
- Updated the `isKnowledgeGone` function to check for deleted knowledge using the tombstone, improving efficiency in handling concurrent delete and ingest operations.
- Added logic in the `wikiIngestService` to filter out updates for deleted knowledge, ensuring that no ghost references are created during the ingest process.
- Implemented lint checks for stale source references in the `WikiLintService`, allowing for automatic cleanup of pages referencing deleted knowledge.

These changes significantly enhance the robustness of the wiki system, ensuring consistency and accuracy in handling knowledge deletions and related wiki pages.
2026-04-22 21:18:20 +08:00