Commit Graph

1900 Commits

Author SHA1 Message Date
wizardchen f99bcb79e7 feat(timeline): scale axis to async tail, enrich span payloads, polish UI
The post-process stage closes in ~9ms (just enqueue work) but its async
subspans (postprocess.summary, postprocess.question, postprocess.graph)
keep producing rows for tens of seconds AFTER the root finalizes. The
old timeline used trace.duration_ms as the time axis maximum, which
clipped those subspan bars past the right edge.

Timeline:
- totalMs now always takes max(trace.duration_ms, observed-tail), so
  the axis stretches to fit the latest descendant end regardless of
  parse_status.
- Render a faint dashed wrapping outline behind a parent span when
  its descendants extend past its own finished_at, so the postprocess
  stage row visibly spans the full window without overloading the
  9ms self-time bar.
- Tree expand/collapse caret bumped from 10 to 14px in a 16x16 hit
  area; copy icons in detail panel bumped from 11/14 to 14/18px;
  .kp-kv-copy button grown from 18 to 22px.
- Short input/output payloads (<= 8 entries / <= 600 bytes JSON)
  auto-expand inline so users see the actual data without an extra
  click; longer payloads keep the click-to-expand summary.

Span payloads (subspans only - root keeps the canonical identity, no
duplicate knowledge_id/kb_id/tenant_id on every child):
- extract.go: graph subspan output gains chunk_chars, chunk_preview,
  sample_nodes, sample_relations.
- summary subspan output gains model_id, summary_preview.
- question subspan output gains model_id and a sample_question
  captured from the first non-empty LLM response.

i18n: new key knowledgeStages.detail.includingChildren for the
wrapping-bar tooltip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:14:45 +08:00
wizardchen bd029b6d19 feat(knowledge): instrument postprocess subspans + polish timeline UI
Backend: summary, question, and graph-extract async tasks now record
real processing time as subspans hanging off the (closed) postprocess
stage, so the trace viewer no longer caps the postprocess row at the
~10 ms enqueue duration. Carries Attempt through SummaryGeneration /
QuestionGeneration / ExtractChunk payloads so cross-process workers
can resolve the right parent attempt.

Frontend: drawer now uses attach="body" so the secondary 820px detail
drawer escapes the 654px container of the main drawer; timeline
timestamps include date prefix (MM-DD, or YYYY-MM-DD across years);
"updated X ago" caption only shows during live polling.

Tests: 4 new cases covering postprocess subspan attaching under a
closed parent, missing-parent fallthrough, and Attempt JSON round-trip
on the three task payloads.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:14:45 +08:00
wizardchen 5ffc755e76 fix(knowledge): infer synthesized stage status from parse_status
Pre-tracker historical knowledge has zero rows in
knowledge_processing_spans but parse_status correctly reads
"completed" or "failed". The /spans handler was synthesizing five
"pending" placeholders unconditionally, so legacy completed documents
rendered as if they were stuck waiting in the queue forever.

buildSpanTree now takes parse_status and chooses the placeholder
status accordingly:
- ParseStatusCompleted -> done
- ParseStatusFailed    -> failed
- everything else      -> pending (existing behaviour)

Real rows always take precedence; this only changes what we put in
the gaps. So healthy in-flight parses (parse_status=processing,
some real rows, some still pending) keep showing pending placeholders
exactly as before — the synthesized "completed" inference only fires
when the parse already hit terminal state.

Adds TestBuildSpanTree_LegacyCompletedRendersAsDone covering both
the completed-legacy and failed-legacy branches.
2026-05-28 15:14:45 +08:00
wizardchen 01d00c2a36 refactor(frontend): rebuild timeline UI with side drawer + attempt tabs
The previous Langfuse-style waterfall had concrete UX problems visible
in the field: the status text column wrapped Chinese characters across
two lines, the root span name truncated mid-identifier, the inline
detail expansion shifted sibling rows out of order, and the attempt
selector was a plain HTML <select>.

This commit rebuilds the visual layer around three corrections:

- Status column removed entirely. Status is conveyed by the 8 px dot
  and the bar color; the redundant Chinese label that was wrapping
  vertically is gone. Header subtitle now reads "总耗时 6.2s · ✓ 已完成"
  via knowledgeStages.total + knowledgeStages.status.<value>.
- Inline row-detail replaced by a right-side drawer absolutely
  positioned inside the timeline shell (overflow: hidden). Clicking a
  row no longer reorders later rows beneath it. The drawer slides via
  CSS transform, ESC and × close it, clicking the selected row toggles.
- Attempt <select> dropdown replaced by a horizontal pill tab strip
  with a per-attempt status glyph (✓ / ✗ / ●). attemptStatuses Map
  populates lazily — one /spans?attempt=n fetch per missing tab on
  mount — so multi-attempt history doesn't block the initial render.

Drawer body shows the rich span data the backend just started
emitting: timing (started / finished / duration), an error block when
failed/cancelled, and Input / Output / Metadata as a 2-column
humanized key/value table. Numbers use toLocaleString, booleans render
as ✓/✗, arrays/objects collapse to a summary with a per-key "Show
JSON" toggle. Existing copySpan retains the raw JSON export path.

Root span name is now localized via knowledgeStages.root ("Knowledge
processing" / "知识处理") instead of leaking the raw identifier
"knowledge_processing" — which previously truncated as
"knowledge_process…".

i18n: added knowledgeStages.total and knowledgeStages.detail.{started,
finished, duration, input, output, metadata, error, showJson, hideJson}
to en-US, zh-CN, ko-KR, ru-RU. Compact mode (used by the card hover
popover) is untouched.
2026-05-28 15:14:45 +08:00
wizardchen 4db6e69b4e fix(knowledge): close root span on terminal state, enrich stage metadata
The root span created by OpenAttempt was never closed: PostProcess only
ended the postprocess stage, so the root row stayed at status=running
forever even after parse_status flipped to completed/failed. The
timeline rendered "进行中" indefinitely on the root, defeating the
whole "is the document done" question the timeline is meant to answer.

- SpanTracker.FinalizeAttempt(kid, attempt, status, output, code, msg):
  closes the root row idempotently. Re-closing a terminal root no-ops
  so success / cascade-fail / dead-letter paths can fire without
  coordination.
- PostProcess.Handle calls FinalizeAttempt(done) after EndSpan(postprocess)
  on the success path. Async downstream work (summary/question/wiki/
  graph) still records its own spans; their completion extends the
  trace's wall-clock end-time but doesn't reopen the root.
- FailSpan auto-closes the root when a MAIN pipeline stage fails
  (docreader / chunking / embedding / multimodal / postprocess).
  Cascade-cancelled siblings stay closed-with-the-cascade as before.
- Dead-letter callback (router/task.go) accepts the SpanTracker via
  DI and calls FinalizeAttempt(failed, TASK_TIMEOUT) when a
  document-related task exhausts retries. The probe payload now
  extracts the Attempt field that Document/Manual/PostProcess
  payloads already thread through.

Stage spans were also being recorded with nil input/output, leaving
the new detail panel with timestamps only. Each Begin/End site now
emits useful work metrics:
- DocReader input: file_name, file_type, is_url, url
              output: text_length, images_found, is_audio, pages
- Chunking input: chunks_planned
            output: chunks_written, total_text_chars
- Embedding input: chunks_to_embed, model_id, dim
             output: vectors_written, storage_bytes
- Multimodal input: image_count, enable_ocr, enable_caption
- PostProcess output: chunks_total, enqueued_summary, _question, _wiki, _graph

i18n: add knowledgeStages.root ("Knowledge processing") so the UI
can render a localized name instead of the raw span identifier.
2026-05-28 15:14:45 +08:00
wizardchen c5722234ef refactor(frontend): redesign parsing timeline as Langfuse-style waterfall
The five-circle pipeline conveyed only "which stage we're on" — operators
asked for the actual span tree (root → stages → batches/images), the time
each span took relative to the whole run, and the recorded JSON in/out.
This commit rebuilds the visual layer to match Langfuse's trace view
without touching the data layer (polling, attempt selector, retry,
copy-span and watchers all unchanged).

- Three-column grid (name | status+duration | bar): each span is one
  row, indented by depth. Status dot 8px, name monospace, duration
  right-aligned. Bars are absolutely-positioned inside a shared
  timeline lane, left/width computed from each span's
  started_at / duration_ms relative to trace.started_at and the run's
  end (or "now" while running).
- Time ruler with five ticks (0% → 100%) above the bars; total
  duration label on the right.
- Sub-spans (embedding.batch[i], multimodal.image[i]) collapsed
  under each stage by default behind a chevron toggle. Click any
  row to expand an inline detail panel showing ISO timestamps,
  status, error_code/message when failed, and input/output/metadata
  JSON blocks plus a "Copy span JSON" action.
- Removed the parse_status gate in doc-content: the timeline is now
  always rendered when a knowledge detail is open, so completed
  documents also expose their recorded pipeline for audit. The
  KnowledgeBase hover popover stays compact-mode-only and still
  gates on parse_status (popover height matters there).
- Running bars grow visibly via a 1s ticker (cleared on unmount).
- Last-error block restricted to parse_status === failed.
- Added knowledgeStages.totalDuration to en-US / zh-CN / ko-KR / ru-RU.
2026-05-28 15:14:45 +08:00
wizardchen 1d8ac301c2 feat(frontend): visualize document parsing pipeline with span timeline
Renders a five-segment timeline (DocReader → Chunking → Embedding →
Multimodal → PostProcess) above each document so users can see exactly
which stage a document is in and which one failed, instead of a flat
"Parsing..." spinner.

- New KnowledgeProcessingTimeline component (full + compact mode):
  full mode lives at the top of doc-content for any document not yet
  in `completed`; compact mode replaces the parsing/failed lines in
  the card hover popover with a five-dot mini-progress.
- Polls GET /api/v1/knowledge/:id/spans every 2s while
  parse_status in {pending, processing}; stops on terminal status or
  unmount. Hover popover instances opt out of polling to avoid
  amplifying requests on grid hover.
- Failed / cancelled steps expand into an error card with the raw
  error_message, an error_code -> localized title + suggestion mapping
  for the twelve canonical codes (DOCREADER_TIMEOUT, EMBEDDING_RATE_LIMIT,
  ...), and a "Copy details" button that pastes the failed span JSON
  for support tickets.
- Attempt selector renders only when latest_attempt > 1 so reparse
  history is browsable; the latest attempt is selected by default.
- Last-error block under the timeline includes a Retry button that
  calls reparseKnowledge() and immediately re-fetches.
- i18n strings added under knowledgeStages for en-US, zh-CN, ko-KR,
  ru-RU. ja-JP locale file does not exist in this repository.
2026-05-28 15:14:45 +08:00
wizardchen 06f94a4811 feat(migrations): add knowledge_processing_spans table for tracking document parsing progress
- Introduced a new migration (000054) to create the knowledge_processing_spans table, which captures detailed progress of document parsing stages.
- The table includes fields for span hierarchy, status tracking, and metadata to enhance visibility into the parsing pipeline.
- Added indexes to optimize queries related to knowledge_id, attempt, and span relationships.
- Implemented rollback migration to safely drop the table and associated indexes if needed.
2026-05-28 15:14:45 +08:00
wizardchen c82b098f44 fix(knowledge): address span-tree review feedback
- Collapse migrations 000052 (flat stages) + 000053 (span tree) into a
  single 000052_knowledge_processing_spans migration; the flat stages
  table never escaped this branch and the create-then-drop sequence had
  no value.
- BeginStage: detect an existing (kid, attempt, stage) row before
  inserting and reuse its span_id with reset state, so re-entry from
  asynq retries or adjacent code paths no longer produces duplicate
  timeline segments.
- FailSpan: when sibling-stage cascade flips a dependent stage to
  cancelled, also CancelDescendants on its subtree so already-running
  subspans (embedding.batch[i] etc.) cannot remain as orphan running
  rows under a cancelled parent.
- Dead-letter callback: replace two sequential UpdateKnowledgeColumn
  writes with a single UpdateKnowledgeColumns map update so we cannot
  end up with parse_status=failed and stale error_message (or vice
  versa) when one of the two writes fails.
- touchKnowledgeHeartbeat: skip subspan/generation transitions; only
  root and stage transitions poke knowledge.updated_at. Spans-table
  MAX(updated_at) already covers subspan progress for housekeeping, so
  this avoids 2*N+ UPDATE bursts on the same hot row when a multimodal
  stage fans out to many images.
- Add regression tests for the BeginStage idempotency contract and the
  cascade-into-subspans behaviour.
2026-05-28 15:14:45 +08:00
wizardchen 414749681f test(knowledge): add coverage and protect housekeeping from false-killing long stages
Addresses three review concerns from the prior PR:

  1. No tests existed for any of the stuck-parsing fixes (PR① / PR②.5).
     This commit adds coverage for the four most regression-prone
     surfaces: span repo upsert/cascade/attempt isolation, span
     tracker cascade-cancel and cross-process LookupStage,
     housekeeping false-kill protection, and handler tree assembly.

  2. Housekeeping was using only knowledge.updated_at as its staleness
     signal, but knowledge.updated_at advances only at parse_status
     transitions — a long DocReader call (or large embedding batch)
     can run for an hour with no updated_at change, so a tight
     DocumentProcessTimeout setting would falsely flip an actively
     running parse to "failed".

     The sweep now does a two-stage check: candidates by knowledge
     updated_at, then filtered by MAX(spans.updated_at). Every
     SpanTracker Begin/End/Fail/Skip now also pokes
     knowledge.updated_at as a side-channel heartbeat, so the
     filter sees recent activity even when no parse_status
     transition has fired.

  3. parseHeartbeatTime accepts the timestamp formats both Postgres
     and SQLite emit for an aggregated MAX() column (the SQLite
     driver doesn't auto-cast aggregates to time.Time the way
     Postgres does), so the same code path works in Lite mode.

The new TestHousekeeping_NoFalseKill_ActiveSpan is the regression
test for the user-flagged scenario: a 3-hour-stale knowledge.updated_at
combined with a 2-minute-fresh span row must NOT be killed.
2026-05-28 15:14:45 +08:00
wizardchen c9941a6688 refactor(knowledge): replace flat stage table with langfuse-style span tree
Addresses review feedback that the PR② design had four shortcomings:

  1. The pipeline is a DAG, not a sequence — Embedding and Multimodal
     are independent of each other, both downstream of Chunking, both
     upstream of PostProcess. The flat (knowledge_id, stage) table
     couldn't represent that, so a Chunking failure left dependents
     stranded as "pending" forever instead of being marked as
     impossible-to-run.

  2. No history across attempts. A reparse erased the previous run's
     status before the new run started, leaving operators with no way
     to investigate "why did this fail twice?".

  3. Stages had only status + duration. Operators want to know how big
     the work was — pages parsed, chunks created, tokens embedded, VLM
     calls made — to distinguish "slow because the file is huge" from
     "slow because the docreader is wedged".

  4. Multimodal fans out N image tasks; Embedding fans out M batches;
     PostProcess fans out into Summary/Question/Wiki/Graph. Each unit
     is interesting on its own (Langfuse already captures this for
     LLM calls). The flat model couldn't express it.

The redesign mirrors Langfuse's trace/span/generation hierarchy:

  * Migration 000053 supersedes 000052: knowledge_processing_spans
    table with (knowledge_id, attempt, span_id) primary key, plus
    parent_span_id, kind ∈ {root, stage, subspan, generation},
    status ∈ {pending,running,done,failed,skipped,cancelled}, and
    JSONB input/output/metadata fields.

  * SpanTracker (replacing StageTracker) exposes OpenAttempt /
    BeginStage / BeginSubSpan / EndSpan / FailSpan / SkipSpan /
    LookupStage. Cross-process workers (image_multimodal) get the
    parent's attempt + span via payload + LookupStage so subspans
    attach correctly.

  * StageDependencies declares the DAG; FailSpan now cascades —
    descendants of the failed span and dependent stages are flipped
    to "cancelled" with a UPSTREAM_FAILED code. The UI sees a clear
    blast radius instead of orphan spinners.

  * Reparse now calls OpenAttempt up front so the timeline reflects
    "new attempt, all pending" instead of letting the previous run's
    status linger until the worker picks up the task.

  * Image_multimodal records each image as a generation subspan with
    its own success/failure on the parent attempt's multimodal stage.
    The finalize-on-last-attempt counter logic is preserved unchanged.

  * GET /api/v1/knowledge/:id/spans (also kept /stages alias) returns
    a tree shape with synthesized pending placeholders so the
    frontend always renders five timeline segments. ?attempt=N
    enables history navigation.
2026-05-28 15:14:45 +08:00
wizardchen 04f56f9cda feat(knowledge): track per-stage parsing progress with /stages API
Adds a five-segment progress model for the document parsing pipeline so
the UI (PR③) can render a timeline showing where each document is
(DocReader → Chunking → Embedding → Multimodal → PostProcess) and
which stage failed with what error code.

- New table `knowledge_processing_stages` (migration 000052) with one
  row per (knowledge_id, stage). UPSERT on Begin/Done/Fail bumps an
  attempt counter so re-parses don't lose history.

- StageTracker service exposes Begin/Done/Fail/Skip; all calls are
  best-effort and never break the pipeline if persistence fails.

- Stable error codes (DOCREADER_TIMEOUT / EMBEDDING_RATE_LIMIT /
  VECTORSTORE_WRITE_FAILED / ...) the UI can map to localized
  remediation hints.

- Tracker call sites added at the four meaningful failure points:
  convert (DocReader), CreateChunks (Chunking), BatchIndex (Embedding),
  enqueueImageMultimodalTasks (Multimodal start),
  KnowledgePostProcess.Handle (Multimodal close + PostProcess).

- New endpoint `GET /api/v1/knowledge/:id/stages` returns the five
  canonical stages — missing rows are synthesized as "pending" so
  the timeline always renders five segments. Includes current_stage
  and last_error block.
2026-05-28 15:14:45 +08:00
wizardchen 3ae3ea97c5 fix(knowledge): prevent documents from getting stuck in "processing"
Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.

- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
  WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
  CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
  (WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.

- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
  (default 30m). Without this, a hung docreader pinned a worker for the
  full DocumentProcessTimeout window.

- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
  failing single image no longer strands the parent — the asynq retry
  is allowed to run, but on the final attempt we count the image
  regardless of outcome. Redis DECR errors fall back to enqueuing the
  post-process task instead of returning silently.

- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
  ManualProcess exhausts retries, immediately mark the corresponding
  Knowledge as failed with the last error. This surfaces the failure
  in the UI without waiting for the housekeeping sweep.

- HousekeepingService: 5-minute cron that flips knowledge rows stuck
  in "processing" past DocumentProcessTimeout + 10m to failed, plus
  summary rows stuck > 1h. Catches anything the other safety nets
  miss (worker SIGKILL mid-handler, etc.). Disable with
  WEKNORA_HOUSEKEEPING_ENABLED=false.

- Distributed startup recovery: previously the post-restart sweep was
  skipped whenever REDIS_ADDR was set, even though Asynq does not
  reschedule the task that was actively running on the dead instance.
  Now the sweep runs in distributed mode too, but only against rows
  older than 30 minutes to avoid racing peer instances.
2026-05-28 15:14:45 +08:00
wizardchen ae4ec0cf06 fix(im): make presigned URL flow diagnosable end-to-end
When IM image rendering breaks, operators previously had no log line to
correlate against the IM platform's fetch attempt. Add observability
hooks on both ends and unblock HEAD probes so common IM previews work
at all.

- log rewriteStorageURLs success/failure/no-op with the full signed URL
  (operators can copy it from logs and verify public reachability)
- log presigned handler 4xx with client_ip + UA + tenant_id + file_path
  so failures correlate against IM platform fetch logs; use the request
  context so trace IDs are preserved
- accept HEAD on /api/v1/files/presigned: IM platforms (Feishu, Slack
  etc.) probe with HEAD before GET when rendering image previews, and a
  401 there is enough to break the inline image even when the GET would
  have succeeded
- add Admin-only GET /api/v1/files/presigned-preview that returns the
  exact URL an IM channel would embed for the calling tenant, for
  self-service verification without sending a real IM message
- clarify APP_EXTERNAL_URL and MINIO_ENDPOINT public-reachability rules
  in .env.example; misconfigured endpoints are the most common cause of
  "image broken in IM" reports
2026-05-28 08:03:57 +08:00
Ochan Kwon f5820c13ac feat(retriever): implement OpenSearch driver read/write paths (PR 2b of 3) (#1482)
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the
PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their
real implementations, drops the stubs from `stubs.go`, and adds the
27 corresponding test cases that were intentionally held back. The
driver remains gated dead code (no registry / factory / env path
mentions OpenSearch — PR 3 ships the activation switch); the
behaviour added here only fires once PR 3 lands.

Depends on PR 2a of #1440 (driver skeleton).

What lands in PR 2b
-------------------

Three new production files (+762 LoC):

- `query.go` (+154) — typed retrieveFilters struct as the only entry
  to query construction (closes the map[string]any injection surface;
  hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN,
  applies min_score when Threshold > 0) and buildKeywordQuery (BM25);
  native hybrid intentionally omitted (out of scope for Phase 3 —
  service-layer RRF stays in charge). Filter struct carries
  ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode
  use cases.
- `retrieve.go` (+204) — Retrieve resolves the per-dim alias from
  (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or
  (3) the cross-dim wildcard <base>_* for BM25-only callers. Always
  returns exactly one *RetrieveResult (the Phase 2 fan-out contract).
  Hit parsing caps response body at 16 MB and warns if _id !=
  _source.chunk_id. Score passes through raw in [0, 1] — the k-NN
  plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped
  (1 + cos) / 2 before the client sees it.
- `crud.go` (+405) — Save / BatchSave with _id = chunk_id for
  idempotency, dim-aware pre-marshal NDJSON size cap (rejects with
  ErrBatchTooLarge before allocating a 50 MB body for a too-big
  batch), and per-item error inspection that does NOT leak cluster-
  side Reason strings into wrapped public errors. additionalParams
  contract verified against elasticsearch/structs.go
  ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding
  is map[string][]float32 keyed by SourceID, chunk_enabled is
  map[string]bool keyed by ChunkID. Missing embedding routes Save /
  BatchSave to the dim-less <base>_keywords index. Sync DeleteBy*
  paths capped at 1000 IDs per call with ErrBatchTooLarge.

  toDoc writes the is_recommended boolean from IndexInfo. The
  matching mapping entry was added in PR 2a alongside is_enabled —
  the pair stays in sync between the vector and the keyword-only
  index so the field round-trips through CopyIndices (PR 3) and any
  future FAQ-priority filtering.

Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve /
DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList
no longer return ErrFeatureNotEnabled — they delegate to the
production implementations above. The remaining stubs (CopyIndices,
BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion,
EstimateStorageSize lower-bound) stay until PR 3.

Tests (+490 net to repository_test.go, 28 new cases):

- retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs,
  IncludeDisabled skip semantics, default is_enabled=true pin.
- buildKNNQuery: min_score applied when Threshold > 0; omitted at
  threshold zero.
- buildKeywordQuery: content match appended after typed filters.
- toDoc: embedding field omitted when empty, present when set,
  source_type serialised as integer, **is_recommended round-trips
  both true and false**.
- lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key
  contract; nil-params and wrong-type degrade paths.
- extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings
  yield dim=0.
- estimateBulkBodyBytes: size estimator boundary.
- effectiveTopK: clamping when caller leaves TopK at the magic-10
  default.
- resolveDim: AdditionalParams takes precedence; Embedding fallback;
  zero-zero triggers cross-dim multi-index search.
- inspectBulkResponse: all-succeeded fast path; total-count + leak-
  guard for cluster-side Reason strings.
- BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal);
  empty list is a no-op.
- DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty
  list is a no-op.

SDK quirk addressed in PR 2b
----------------------------

DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level
"wait_for" value documented in the OpenSearch REST API is not
directly expressible. deleteByTerms uses Refresh:&true which forces
immediate segment flush (read-your-writes guaranteed but more
expensive than wait_for). A follow-up PR can drop to
*opensearchapi.Client.Transport.Perform for wait_for if telemetry
shows the cost is material.

Backward compatibility
----------------------

- Additive within the opensearch package only. No file outside the
  package is touched.
- Driver still unreachable: PR 3 ships the activation switch.
- The PR 1 normalizer case for OpenSearch remains unreachable here
  (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-28 08:00:33 +08:00
caiyuze-cpu d9ba7716b5 fix(agent): harden tool parameter parsing against LLM type mismatches (#1505) 2026-05-28 07:50:26 +08:00
Sheep44044 3645477dba refactor(qdrant): optimize batch save, map iteration, and error wrapping (#1508) 2026-05-28 07:39:41 +08:00
lyingbug 3611a8607d feat(rbac): add multi-use share-link invitations for invite_only mode (#1504)
* feat(rbac): add multi-use share-link invitations for invite_only mode

When the auth.registration_mode toggle flips to invite_only, the public
/auth/register endpoint returns 403 — but until now there was no
channel for unregistered users to actually join a tenant. The existing
invitation table required a registered invitee, so new emails could
not enter the system at all.

This change introduces a share-link model:

- Owner generates a multi-use registration link with a per-tenant role
- Recipient opens the link, registers with their own email, and is
  added to the tenant on the same request
- Token is stored in plaintext (not hashed) so the management UI can
  re-display the URL on demand without a "copy now or revoke" trap.
  Threat model is bounded by 7-day TTL, revocability, and the fact
  that all the link grants is membership in one tenant.
- accepted_count tracks how many users joined through the link so the
  Owner can tell whether a link is fresh or has already been spread

Per-user invitations (registered email -> in-app inbox accept) are
unchanged. Share-link rows use empty invitee_user_id as discriminator;
the partial unique index on (tenant_id, invitee_user_id) was relaxed
to skip empty values so multiple share links can coexist per tenant.

Frontend reuses Login.vue for /register?token=xxx instead of a
parallel page, and adds icon-only revoke / remove actions plus a
distinct "active" status tag for share-link rows.

Migration 000054 adds the token column, the accepted_count column,
and the relaxed pending unique index in a single step.

* feat(auth): update invitation token handling and add rate limiting

- Refactor the invitation token retrieval to use POST instead of GET, enhancing security by preventing the plaintext token from appearing in logs and browser history.
- Update the API endpoint from `/auth/invitations/:token` to `/auth/invitations/lookup` to reflect the new method.
- Introduce a rate limiter for unauthenticated share-link endpoints to mitigate brute-force attacks and abuse, ensuring a maximum of 30 requests per minute per IP.
- Adjust the registration flow to maintain user experience while securing the invitation process.
2026-05-27 19:42:01 +08:00
wizardchen 6220667517 fix(im): recover from deleted session when GetSession returns app sentinel (#1499)
The recovery branch added in c578fdba (for #1046) checked
`errors.Is(err, gorm.ErrRecordNotFound)`, but the session repository
translates the GORM error into `apperrors.ErrSessionNotFound` before it
reaches the IM service. The predicate therefore never matched, the
stale ChannelSession mapping was never recycled, and the bot stayed
permanently unresponsive after the underlying session was deleted from
the UI — exactly the symptom reported in #1499.

Extract the predicate into `isSessionNotFound` that matches both
sentinels (application sentinel as actually returned today; GORM
sentinel kept as a safety net against future repository reverts), and
add a focused regression test guarding the invariant.

Refs: https://github.com/Tencent/WeKnora/issues/1499
2026-05-27 11:51:11 +08:00
wizardchen 39efcf02d2 feat(settings): replace monogram badges with brand logos
Drop the per-engine first-letter monograms on Vector Store / Storage /
Web Search cards in favor of real brand SVGs. Logos live under
`assets/img/providers/{color,mono}/<category>/<id>.svg`:

- color/: full-color vendor SVGs (gilbarbara/logos, svgl, vendor github)
  rendered directly via <img>; covers qdrant, milvus, elasticsearch,
  postgres, opensearch, sqlite, s3, minio, bing, duckduckgo, google,
  ollama, searxng.
- mono/: single-color marks (simple-icons / vendor) rendered via
  mask-image so the existing per-engine `color: <brand>` rule tints
  them; covers doris, cos, obs, oss, baidu.
- Anything without a logo falls back to the existing monogram badge.

A small `providerLogos.ts` helper auto-discovers SVGs with
`import.meta.glob` and returns `{ mode, url }`; the three settings
views read it to pick the render path. Badge styles stay backwards
compatible: a `--logo` modifier swaps the brand-tinted background for
a neutral white card with a thin stroke when a logo is present.
2026-05-27 11:21:23 +08:00
wizardchen 362c973aad refactor(settings): move vector store test into menu, toast result
The standalone "Test connection" button cluttered the title row
and the inline result bar made cards visually unstable as users
ran tests. Behavior change:

- Move "测试连接" into the more-actions dropdown as the first item.
  env-source stores now also get a dropdown (test-only) instead of
  showing a hidden placeholder button.
- Surface the result via MessagePlugin.success / .error toast and
  drop the per-card testResults state, getTestResult(),
  clearTestResult(), and the .store-card__result* SCSS rules.
- Keep an inline spinner in the title row while a test is in
  flight, so the user still sees progress after the menu closes.
2026-05-27 11:21:23 +08:00
wizardchen 272be9e8c5 feat(settings): redesign vector store cards
Bring vector stores in line with the rest of the settings list
redesign. The previous list-row layout (lock icon + 2 tags +
endpoint subtitle + buttons) carried the same "low" feel as the
old model/provider cards: flat, tag-heavy, no engine identity.

- Replace .store-list rows with a .store-grid card grid sharing
  the same skeleton used by Parser/Storage/Mcp: a 36x36 monogram
  badge tinted per engine_type (qdrant red, milvus blue, weaviate
  green, elasticsearch amber, postgres blue, opensearch / infinity
  purple, tencent_vectordb blue, doris orange, sqlite gray, others
  fall back to brand blue).
- Title row carries the name + an env pill for read-only stores +
  a low-contrast text-style "Test" button (admin-only) + the
  hover-only ellipsis menu (edit / delete, user stores only).
- Subtitle: engine_type · endpoint (monospaced, truncated).
- Test result keeps its inline success/error bar but rebuilt as
  store-card__result so it lives inside the card instead of
  hanging off a sibling row.

Drops the .store-list / .store-item / .item-* / .test-result-*
selectors that were only ever used here.
2026-05-27 11:21:23 +08:00
wizardchen 68d41ae7d9 feat(settings): redesign parser/storage engine cards
Bring the engine selectors in line with the rest of the redesigned
settings cards (Model / WebSearch / Mcp). Same skeleton: leading
36x36 monogram badge tinted per engine, clean title row, dot+label
status pill, two-line description, brand-color outline when the
config drawer is open for that card.

Parser engines:
- Replace the bespoke .engine-card markup with a button-based card
  carrying a badge derived from the localized display name.
- Keep available/unavailable as a status pill (green dot / red dot)
  instead of t-tags, retaining the unavailable-reason tooltip via a
  dedicated --help variant.

Storage engines:
- Collapse the eight hand-written cards into a single v-for over a
  STORAGE_PROVIDERS array; titles, monograms, and statuses funnel
  through providerTitle/providerInitial/providerStatus helpers so
  adding a future engine only needs an array entry plus translations.
- Status pill mirrors Parser/Mcp idioms: green for available /
  configurable, gray for "needs config" (currently MinIO-only).

Drops the now-unused .engine-card-header/.engine-card-desc/
.tag-with-tooltip rules.
2026-05-27 11:21:23 +08:00
wizardchen 48e1886089 refactor(settings): tighten provider card chrome
- Web search: drop the "default" and "free" pills from the card
  header; the provider type subtitle and the existing form-side
  free badge already cover this, and three rows of pills made the
  cards feel busy.
- MCP: collapse the inline switch and the on/off tag into a single
  on/off status pill (dot + label), and move the actual flip into
  the more-actions dropdown as the first item. Built-in services
  keep showing the "builtin" pill instead, since they're not
  togglable.

Drops the now-unused isEntityFree helper and the .provider-card__pill
SCSS variants.
2026-05-27 11:21:23 +08:00
wizardchen 260b3eec26 feat(settings): redesign websearch/mcp provider cards
Apply the model-card pattern to the other two provider lists in
the new "Data & Extensions" group so they share one visual idiom.

Web search providers:
- Drop the 3-tag row (provider type / default / free) for a 36x36
  monogram badge tinted per provider (duckduckgo, bing, google,
  tavily, baidu, searxng, ollama).
- Title row carries the entity name plus optional "default" /
  "free" pills; the more-actions button hides until hover/focus
  and opens on click.
- Subtitle reads "Provider type · description"; proxy URL drops
  to a muted monospaced last line, no leading icon.
- Grid is responsive: repeat(auto-fill, minmax(320px, 1fr)).

MCP services:
- Same skeleton with a transport-type badge (sse → cast / sse-green,
  http-streamable → link / brand-blue, stdio → code / amber).
- Keep the inline enable switch, but slot it into the title row so
  the card stops repeating "on/off" as a tag.
- Builtin services keep the muted background to signal read-only.
- Three-dot menu hides until hover/focus and opens on click.

Drops dead CSS (.provider-meta-item, .service-meta-item, the old
2-column grids) and the now-unused SettingCard imports. SettingCard
itself is left in place for future consumers.
2026-05-27 11:21:23 +08:00
wizardchen f10b83d92f feat(settings): redesign model card with type badge
The two-column model list looked low-res because every card
shared the same skeleton, with three saturated tags and a raw
baseUrl meta line carrying most of the visual weight.

Replace SettingCard usage on this page (other consumers — Mcp
and WebSearch settings — keep using it untouched) with an inline
ModelCard built around:
- A 36x36 leading badge per model type (chat, embedding, rerank,
  vllm, asr) with low-saturation tinted backgrounds and a
  matching tdesign icon, so users can scan by type at a glance
  instead of decoding tag colors.
- Title row with the model name and an optional muted "builtin"
  pill; the more-actions button only appears on hover/focus and
  opens the dropdown on click (not hover) to match the rest of
  the app.
- A muted subtitle (type · source · embedding dimension) and a
  monospaced, single-line truncated baseUrl as the last line —
  so secondary info reads as secondary.
- Responsive grid: repeat(auto-fill, minmax(320px, 1fr)) so
  wider modals get 3 columns and narrow viewports collapse to 1.

Drops dead CSS (.model-type-tag-* color rules, the page-wide
:deep(.t-tag) override, .model-meta-item) and the now-unused
SettingCard import.
2026-05-27 11:21:23 +08:00
wizardchen ad82692785 refactor(settings): regroup sidebar nav and pin header on scroll
Sidebar UX polish for the Settings modal:
- Regroup nav into 5 cleaner buckets: Account / Workspace /
  Models / Data & Extensions / Platform. Personal preferences
  (general) and credentials (api) move to Account; chathistory
  follows its scope into Workspace; integrations and engines
  merge to avoid two narrow groups.
- Pin the sidebar header (title) so it stays visible while the
  nav list scrolls; move overflow from the sidebar wrapper to
  the nav list itself.
- Tighten paddings/font sizes a notch for higher information
  density without sacrificing hit targets.
- Sync the four locale files (zh-CN as source of truth, plus
  en-US, ko-KR, ru-RU): drop workspaceAccount / integrations /
  knowledgeInfra; introduce account / workspace / dataExtensions.
2026-05-27 11:21:23 +08:00
nullkey b395db55de chore(cli): adapt ListKnowledgeChunks interface to variadic chunk_type filter
Upstream commit 5c0243cd ("feat: Support filtering chunks by multiple
chunk_type params") extended client.Client.ListKnowledgeChunks with a
trailing `chunkTypes ...string` variadic. The two narrow service
interfaces in cli/ — ListService in cli/cmd/chunk/list.go and
chunkListService in cli/internal/mcp/tools.go — plus the two fake
implementations in their *_test.go siblings, are updated to match so
*client.Client continues to satisfy them via duck typing.

No CLI flag surfaced yet; both call sites pass no filter and behavior
is unchanged. Exposing `--chunk-type` as a flag on `weknora chunk list`
and as an input field on the MCP `chunk_list` tool is a candidate for a
follow-up commit, per the broader SDK-required → CLI-required alignment
discussion.

Pre-flight: go build ./... clean; go test -count=1 ./... 28/28 packages
pass; go vet ./... clean; gofmt -l . empty.
2026-05-27 10:56:34 +08:00
nullkey 2ee9741fa1 refactor(cli): finish context→profile cascade + post-review hardening (BREAKING)
Post-review polish on the v0.7 wire / surface contract. Bundles five
follow-ups that landed after the main BREAKING feat commit:

1. Complete context→profile cascade (internal API + YAML schema)

The prior commit renamed only the user-visible surface (commands /
flags / env / project link / envelope field). The internal Go API
and on-disk config schema were still half-renamed — an L-25
self-consistency violation flagged by post-merge review. Closed here:

Internal Go API:
- config.Context           → config.Profile
- config.Config.CurrentContext → CurrentProfile
- config.Config.Contexts       → Profiles
- LoginOptions.Context     → LoginOptions.Profile
- clearContextSecrets()    → clearProfileSecrets()
- saveContextRef()         → saveProfileRef()
- secrets.Store: param name `context` → `profile` (interface +
  FileStore + KeyringStore + MemStore)
- cmdutil.LoadSecret(store, context, key) → LoadSecret(store, profile, key)
- cmdutil.RefreshAndPersist's ctxName → profileName
- Local var `ctx := &config.Profile{...}` → `prof := &config.Profile{...}`
  in auth/login.go to eliminate the visual collision with Go stdlib
  context.Context that motivated the whole rename in the first place.

On-disk config.yaml schema:
- current_context: → current_profile:
- contexts:       → profiles:
- Pre-1.0 break, no compat alias. Users on v0.6 dogfooded configs
  must delete ~/.config/weknora/config.yaml or hand-rename the two
  keys (CHANGELOG migration note added).

Tests / fixtures / golden files:
- factory_test.go YAML fixture + assertion updated.
- acceptance/e2e/e2e_test.go writeContextYAML → writeProfileYAML,
  fixture YAML keys updated.
- acceptance/testdata/wire/doctor.error_network.json golden updated
  ("active context" → "active profile" in hint string).

User-visible prose sweep:
- cmd/mcp/serve.go --help Long: "active context (or --context)" →
  "active profile (or --profile)" — most-visible miss.
- cmd/{kb/list, search/kb, session/list, api/api} Short/Long help.
- cmd/auth/login.go stdout: `(context=%s)` → `(profile=%s)`.
- cmd/auth/logout.go error: `"no current context"` → `"no current profile"`.
- cmd/doctor/doctor.go hint string (also the wire golden above).
- cmd/auth/refresh.go error: `"refresh token missing for context"` →
  `"refresh token missing for profile"`.
- README.md: `## Multi-context` H2 → `## Multi-profile`; code-block
  comment `# current context` → `# current profile`.

Code-comment / docstring sweep across cli/cmd/auth/ and
cli/internal/cmdutil/. Comments referencing Go stdlib context.Context,
the RAG / LLM "context window" concept, and historical CHANGELOG
entries for v0.4 / v0.5 were left alone.

CHANGELOG v0.7 BREAKING entry gains the on-disk-schema bullet under
the existing "context → profile" item.

2. Profile name validation (shell-injection guard)

`envelope.error.retry_command` is a single shell-string field. An
AI agent that exec()s it via `sh -c <retry_command>` was injectable
through a maliciously-named profile:

  weknora auth logout --name 'x; rm -rf ~'
  # would produce: retry_command = "weknora auth logout --name x; rm -rf ~ -y"

`cmd/profile/add.go` already enforced an alphanumeric + `-_.`
allowlist via `validateName`. The `auth login` and `auth logout`
paths bypassed it.

- Moved validation from `cmd/profile/add.go` to
  `cli/internal/cmdutil/profilename.go` as exported
  `ValidateProfileName` (cmdutil is the import-cycle-safe home;
  internal/config can't depend on cmdutil).
- `auth login` runs the validator before any persist call.
- `auth logout` runs the validator on `opts.Name` before
  constructing `retry_command`.
- Unit tests (`profilename_test.go`) cover the allowlist, empty
  rejection, path-traversal, shell metacharacters (`;`, `&`, `|`,
  `$()`, backticks, quotes, whitespace, glob, redirects), and the
  user-facing hint text. The shell-metachar test exists as a
  regression guard.

Wire shape (`retry_command` string → `retry_command_argv []string`)
remains a v0.8 additive change per ROADMAP — this fix removes the
practical exploit path without touching the wire contract.

3. AI-agent terminology disambiguation

"agent" has three referents in this codebase: (a) WeKnora's
server-side Custom Agent resource, (b) the removed `agent invoke`
verb, (c) external LLM/automation consumers. Per project memory
feedback_no_meta_disambiguation_in_docs, the fix is full-term
naming, not "X has N meanings" prose. Surgical changes at section
headers + ambiguous prose:

- AGENTS.md: "Agent decision shortcuts" → "AI agent decision
  shortcuts"; "agent-callable surface" → "AI-agent-callable
  surface".
- README.md: "Designed to be agent-first" → "AI-agent-first";
  "Other agent ergonomics" → "Other AI-agent ergonomics"; "in
  agent contexts" → "in AI-agent contexts"; "for CI / agents" →
  "for CI / AI agents".

Anaphoric "agents" inside paragraphs that already established
"AI agents" was left alone — full substitution everywhere would
have been prose noise without clarity gain.

4. Wire-contract review follow-ups

Real findings from a second-pass review of the v0.7 envelope /
streaming / surface design. Per project memory
feedback_check_in_domain_anchor_first, candidate findings were
first verified against the in-domain peer CLI explicitly cited as
the envelope anchor; two earlier-flagged issues turned out to be
in-pattern and were withdrawn.

Surviving fixes:

- AGENTS.md success-envelope example rewritten. The prior example
  showed `has_more: false` / `_notice: {}` as if they were always
  present, but both fields are `omitempty` and never serialize
  when zero / nil. Replaced with three realistic shapes (list /
  single resource / mutation with no payload) and added a note
  that optional fields are omitted when empty.

- cmd/chat/chat.go Args: MinimumNArgs(1) → ExactArgs(1).
  v0.6 silently joined `weknora chat hello world` into
  `"hello world"`. v0.7 now rejects multi-arg with exit 2,
  matching `weknora session ask`. BREAKING; CHANGELOG entry
  added under v0.7 BREAKING.

- internal/output/envelope.go extracts NewEnvelope(data, meta,
  profile) constructor. The jq-filter path in
  cmdutil.FormatOptions.Emit was manually rebuilding the
  envelope literal alongside the canonical WriteEnvelope path —
  drift risk when fields are added. Single construction point now.

- internal/cmdutil/factory.go adds AddKBFlag(cmd) helper.
  Five files (chat, doc/list, doc/upload, doc/create, doc/fetch)
  had verbatim-identical `cmd.Flags().String("kb", ...)`
  declarations. Centralised so flag name + help text stay
  in sync with Factory.ResolveKB. Docstring reordering + gofmt
  fixup landed in the same edit to keep ResolveKB's own godoc
  attached to its function.

5. OSS-readiness comment / doc sweep

Pre-publication scrub of code, comments, and shipped Markdown to
remove references that only make sense in the development repo:

- AGENTS.md "Deliberate deviations + mainstream alignments"
  section: removed peer-project name-drops from the comparison
  table; rewrote as five flagged design decisions with rationale
  but no specific competitor named. The four rows that previously
  contrasted against a named peer CLI now state WeKnora's choice
  + rationale directly. Section header renamed to "Design
  decisions worth flagging" since it is no longer a
  deviation/alignment matrix.

- CHANGELOG v0.7 BREAKING rationales: three references to a
  named peer CLI removed; the context→profile rationale now
  cites only mainstream multi-credential CLIs by category (AWS /
  Stripe / OpenAI / Anthropic), and the `api -d/--data` removal
  rationale cites only `gh api` / `curl`. `chat` BREAKING entry
  rationale similarly simplified.

- 35 cross-references to design-spec section numbers (§4.1 /
  §4.5 / §5.3 etc.) removed from Go doc comments and test
  comments across 13 files. The referenced spec lives outside
  the shipped tree; readers of the public repo cannot resolve
  them. Each reference replaced with a self-contained semantic
  description (e.g. "the batch envelope" / "AGENTS.md section
  on the success path").

- Mixed-language strings translated to English:
  - Four Go comments: internal/cmdutil/exit.go:213,215,
    internal/cmdutil/errors.go:156,
    internal/output/batch_test.go:90,
    internal/output/envelope_test.go:27.
  - One CHANGELOG section title:
    `v0.7 — Agent-first wire contract + 命令面集中清理` →
    `... + command-surface cleanup`.
  - CJK test fixtures (internal/text/truncate_test.go CJK
    truncation cases, cmd/session/list_test.go Chinese session
    title, acceptance/e2e/e2e_test.go Chinese RAG corpus)
    retained — they are intentional test inputs, not stray prose.

- Makefile help comment: `golangci-lint added in PR-9` →
  `golangci-lint planned`. Internal PR numbering should not
  surface in shipped Makefile prose.

Build green, 28/28 packages, +5 new ValidateProfileName tests.
go vet / gofmt / go mod verify / go mod tidy all clean.

Rationale for the cascade: pre-1.0 is the cheapest moment to close
L-25 self-consistency (L-26). The half-finished internal rename
would have perpetuated the very `context` vs `context.Context`
ambiguity that motivated v0.7's user-visible rename in the first
place.
2026-05-27 10:56:34 +08:00
nullkey 2ce348d020 feat(cli): --format json default + NDJSON event stream + context→profile cascade + help calibration + docs (BREAKING)
D1 — --format default flipped to json regardless of TTY:
- v0.6: smart default (text on TTY, json on pipe).
- v0.7: always json; TTY only affects indent (compact in pipe). Enum
  values unchanged (text | json | ndjson).
- Typed FormatMode enum replaces untyped string consts.
- --format / --jq promoted to persistent root flags so unknown-
  subcommand paths still reach the typed-envelope guard (per-command
  registration in v0.6 would have rejected --format on unknown
  commands as cobra-prose exit 2).
- WEKNORA_FORMAT env var added; precedence --format > env > default.
  Invalid env values silently ignored.

D2 — chat / session ask default to NDJSON event-stream:
- New cli/internal/output/ndjson_stream.go: InitEvent struct +
  EmitInit / EmitSDKEvent / WriteNDJSONLine helpers. EmitInit doc
  encodes the must-be-first-line invariant agents key on.
- chat / session ask: --format json AND --format ndjson both emit one
  JSON event per line (no envelope wrapping). CLI injects exactly one
  `init` event at stream head carrying session_id + optional kb_id /
  agent_id / profile. Subsequent events pass through verbatim from the
  SDK (passthrough discipline per spec §5.1).
- --format text keeps the SSE-style live renderer.

context → profile full cascade:
- Command group: cli/cmd/context/ → cli/cmd/profile/ (git mv;
  package contextcmd → profilecmd).
- Global flag --context → --profile. Factory.ContextOverride →
  ProfileOverride. WEKNORA_PROFILE env var honored
  (--profile flag > env > config.CurrentContext). When --profile or
  WEKNORA_PROFILE references a missing profile, the error is
  input.invalid_argument with hint "weknora profile list" — not the
  destructive local.config_corrupt path (which would have told users
  to delete their config file).
- Binding file .weknora/project.yaml field context: → profile:
  (no backwards-compat alias; re-run weknora link).
- profile use JSON fields current_context / previous_context →
  current_profile / previous_profile.
- weknora link JSON field context → profile.
- CodeLocalContextNotFound → CodeLocalProfileNotFound (typed code
  rename).
- Envelope top-level profile field populated via globalProfile (set
  by root PersistentPreRunE from Factory.ActiveProfile). chat /
  session ask NDJSON init event carries the same profile.
- Rationale: "context" collided with LLM context window / RAG context
  / Go context.Context; mainstream multi-credential CLIs (AWS /
  Stripe / OpenAI / Anthropic / lark) all use "profile".

H2/C1' help calibration:
- AgentHelp gains Warnings []string; single SetAgentHelp helper
  routes on WEKNORA_AGENT_HELP=1 (emits JSON blob including
  warnings) vs human help (appends "AI agents:" block from same
  source). Warnings surface as both a structured JSON field and
  visible help-text addendum without drift.
- 9 destructive commands carry warnings: kb / doc / agent / session /
  chunk delete; profile remove; kb / agent edit; auth logout.
- weknora doc wait dedups ids at entry; SIGINT mid-wait returns
  silently (root signal handler maps to exit 130) instead of being
  miscategorised as operation.timeout / operation.failed.

A4 — docs:
- cli/AGENTS.md gains four agent-facing sections: Wire contract for
  AI agents (stdout / stderr / NDJSON / _notice evolution / SDK
  contract boundary); Deliberate deviations + mainstream alignments;
  Pre-1.0 breaking policy; Exit-10 anti-patterns. ERROR_REFERENCE
  table extended.
- cli/README.md adds Agent quick start under Wire contract.
- cli/CHANGELOG.md v0.7 section: BREAKING entries with migration
  notes, Added (WEKNORA_FORMAT / WEKNORA_PROFILE / retry_command /
  retry_after_seconds / risk / _notice reserved infra / meta.count /
  meta.has_more / doc fetch / doc create / session ask / doc delete
  --all / NDJSON init), Changed (docs additions), Deprecated (none —
  pre-release one-shot breaking).

Spec: docs/superpowers/specs/2026-05-20-weknora-cli-v0.7-design.md §3 / §4 / §5 / §6 / §11
2026-05-27 10:56:34 +08:00
nullkey a2e368b1e8 refactor(cli): command-surface rename — session ask / doc fetch / doc create / doc delete --all
Three command renames consolidate the v0.7 verb table:

- `weknora agent invoke` → `weknora session ask --agent <id>`.
  Server route POST /sessions/{session_id}/agent-qa is session-
  anchored, so the verb moves with it. `weknora agent` subtree keeps
  CRUD only (list / view / create / edit / delete / status / check).
  SDK call (AgentQAStreamWithRequest) preserved verbatim; only the
  command surface + flag layout move.

- `weknora doc upload` split into three commands:
  - `weknora doc upload <file>`  — local file (only).
  - `weknora doc fetch <url>`    — server-side remote fetch (was
    `upload --from-url`). URL-only flags (--title / --file-type /
    --tag-id) move with the verb.
  - `weknora doc create --text`  — direct text knowledge entry via
    server CreateManualKnowledge.

- `weknora kb empty <id>` → `weknora doc delete --all --kb=<id>`.
  Atomic server ClearKnowledgeBaseContents (no list-then-delete
  race). Same exit-10 -y/--yes guard as other destructive verbs;
  unified through the extended ConfirmDestructive helper.

Parent commands (agent, kb, doc, chunk, session, auth, profile,
search) lose their explicit Args:NoArgs + Run:cmd.Help so the
unknown-subcommand guard fires correctly — `weknora agent invoke
ag_x q` now emits the typed input.unknown_subcommand envelope with
detail.available[] instead of cobra's free-form exit-2 prose.

Spec: docs/superpowers/specs/2026-05-20-weknora-cli-v0.7-design.md §3.4 / §10.7
2026-05-27 10:56:34 +08:00
nullkey ad150c7126 refactor(cli): envelope sweep — Emit shape, batch ops, MCP StructuredContent, api flag cleanup
Sweep every CLI output path through the v0.7 envelope contract.

FormatOptions.Emit signature changes to (w, data, meta *output.Meta)
so list commands surface meta.count / has_more in the wire envelope.
TTY indent decision plumbed via FormatOptions.TTY (set by
ResolveDefault). --format and --jq promoted to persistent root flags
so unknown-subcommand paths still reach the typed-envelope guard
(per-command registration would have rejected --format on unknown
commands as cobra-prose exit 2). WEKNORA_FORMAT env-var fallback
honored before TTY-resolved default. AddFormatFlag retained for the
per-command --jq field-hint Long-text addendum.

~55 production Emit call sites swept across cli/cmd/**: list paths
populate meta.count; mutation paths pass nil meta; envelope.profile
threaded via globalProfile. Existing JSON-shape tests migrated to
decode the envelope wrapper.

Multi-target batch operations get a unified shape via new helpers:
- output.BatchItem + output.WriteBatchEnvelope (§4.5 per-item ok
  pattern; data:[{id, ok, result?|error?}], meta:{count, successes*,
  failures*}; Successes/Failures *int so explicit zero survives
  omitempty in all-fail case).
- cmdutil.RunBatch + cmdutil.EmitBatch + cmdutil.DeletedAtNow +
  cmdutil.ClassifyContextErr collapse the three previously-duplicated
  multi-delete pipelines (doc / chunk / session) into one call site
  pattern. doc upload --recursive uses the same helpers with a per-
  file resultFn. Summary error sets Silent:true so the stderr
  envelope path doesn't duplicate the stdout batch detail (exit 1
  preserved via Code → ExitCode).

ConfirmDestructive / ConfirmDestructiveBatch extended with action +
retryCmd parameters. Every destructive command (kb delete, kb edit,
doc delete, doc delete --all, agent delete, agent edit, session
delete, chunk delete, auth logout, api -X DELETE) now attaches
error.risk.{level:"destructive", action:"<noun.verb>"} +
error.retry_command on the exit-10 envelope. kb edit / agent edit
also gain the destructive guard wiring (CHANGELOG listed them as
gated but they were silently executing without -y).

MCP tool handlers' 31 error paths return CallToolResult with
IsError + Content text fallback + StructuredContent (envelope.error
shape via cmdutil.ErrorToDetail). Handler Out type changed from
typed *sdk.X to any so the go-sdk auto-marshal doesn't overwrite
StructuredContent with a zero-struct on error returns. Success path
manually populates CallToolResult.StructuredContent via successResult
helper.

weknora api cleanup:
- -d/--data flag removed; body via --input <file> or --input -
  (stdin) only. Aligns with gh / curl convention.
- HTTP method whitelist removed; any non-empty method accepted.
- HTTP Retry-After header → cmdutil.Error.RetryAfterSeconds →
  envelope.error.retry_after_seconds.
- runAPIPaginated raw-passthrough fallback now wraps response in
  envelope so --paginate never emits bare JSON to stdout.

Test sweep: ~60 cmd/** tests migrated to decode envelope shape.
9 wire-contract goldens updated under cli/acceptance/testdata/wire/.

Spec: docs/superpowers/specs/2026-05-20-weknora-cli-v0.7-design.md §4 / §5
2026-05-27 10:56:34 +08:00
nullkey 733bb3aaa1 refactor(cli): symmetric envelope infrastructure (supersedes e623e820)
Re-introduce the agent-first symmetric envelope deleted in commit
e623e820 (2026-05-15). Under the v0.7 constraint that AI agents are
primary consumers, the bare-JSON shape can't carry protocol channels
(_notice / risk / meta / request_id / profile) that agents need.
Errors-on-stderr and typed exit codes from e623e820 are preserved;
the envelope wraps the success/error payloads on top.

- cli/internal/output/envelope.go (new): Envelope / ErrorEnvelope /
  Meta / ErrDetail / RiskDetail structs; WriteEnvelope +
  WriteErrorEnvelope writers; PendingNotice plumbing for the open-map
  _notice channel (reserved infra; producer wiring planned for v0.8).
- cli/internal/output/envelope_test.go (new): 7 tests covering
  success / error / TTY indent / Notice / Risk shapes.
- cmdutil.Error extended with RetryCommand (directly-executable
  argv distinct from prose Hint), RetryAfterSeconds (HTTP
  Retry-After), *RiskInfo (nested level+action; ErrInternalServer-
  vs-NotFound distinction), Detail (open structured payload),
  Silent (suppress stderr emit while preserving Code for ExitCode).
- ErrorToDetail single source of envelope.error construction; reused
  by stderr PrintError, MCP StructuredContent, and batch per-item
  error path.
- WithHint / WithRetryCommand / WithRetryAfter / WithDetail /
  WithRisk builders. IsCancelled helper. AsError unwrap helper.
- defaultHint covers ~21 codes; defaultRetryCommand symmetric
  counterpart for 6 keyway codes. DefaultHint / DefaultRetryCommand
  exported wrappers for cross-package callers.
- ClassifyHTTPError tightened: rescues HTTP 500 with structured
  server-side code 1003 (ErrNotFound) into resource.not_found.
  Server's generic 1007 (ErrInternalServer) bucket stays as
  server.error — including it would mis-route validation failures
  (e.g. SQLSTATE 22001) as not-found.
- PrintError dual-mode: text/human → prose with code:msg / hint /
  retry lines; json/ndjson → envelope on stderr. Mode pinned by
  root PersistentPreRunE via SetFormatMode. resolveFormatEarly()
  scans argv before cobra dispatch so cobra-side validators
  (unknown flag, arg-count) still surface as envelope when
  --format json is in effect. Silent typed errors short-circuit
  the stderr emit. FlagError mapped to input.invalid_argument for
  the envelope (exit code stays 2 via FlagError class).
- Unknown-subcommand guard installed recursively at the root:
  parents with subcommands but no Run/RunE get a typed RunE that
  emits input.unknown_subcommand envelope with
  detail.{unknown, command_path, available[]} and
  retry_command "<parent> --help". cobra.ArbitraryArgs bypasses
  legacyArgs validation so the guard receives unmatched argv.
- cmdutil.Error.Error() returns "<code>: <message>[: <cause>]" for
  chain debugging; ErrorToDetail strips the code prefix from
  envelope.message since the separate type field carries it
  (prevents the doubled-prefix "code: code: ..." that agents would
  see).

Spec: docs/superpowers/specs/2026-05-20-weknora-cli-v0.7-design.md §0 / §4
2026-05-27 10:56:34 +08:00
wizardchen 7d1fe6f11b chore(deps): add opensearch-go v4.6.0 to go.mod and update go.sum
- Added the opensearch-go v4.6.0 dependency to the go.mod file.
- Removed the indirect requirement for opensearch-go from go.sum as it is now a direct dependency.
2026-05-26 21:13:56 +08:00
wizardchen 2a117262bf test(system): cover system.* audit constants + emitAdminAudit contract
Two test surfaces, picked for cost/value:

internal/types/audit_log_test.go — extend the existing invariant suite
to include the system namespace:
- DotNamespaceConvention now covers system.setting_changed,
  system.admin_promoted, system.admin_revoked.
- NoCollisionsAcrossNamespaces guards against duplicates across all
  three new constants.
- New SystemNamespacePrefix test pins the shared "system." prefix —
  this is the contract by which GET /system/admin/audit-log filters
  out per-tenant rbac.* rows. Drift here would either leak per-tenant
  events into the platform feed or hide platform events from
  SystemAdmin.
- New SystemWireValues test pins the exact wire strings consumed by
  the new frontend audit drawer, Langfuse exporters, and future SIEM
  integrations; changes to these are a breaking change.

internal/handler/system_admin_audit_test.go — direct unit tests for
SystemHandler.emitAdminAudit, the helper that promote/revoke /
ApplyDefaultStorageQuotaToAllTenants all delegate to. Uses a
capturingAuditService stub (interface-embedded so any other method
call surfaces drift loudly) and a minimal SystemHandler with only
auditSvc wired — the helper deliberately doesn't touch other deps.

Coverage:
- NilServiceIsNoop: degraded-mode contract — a handler built without
  an audit service must not panic on the audit hook.
- PopulatesCanonicalFields: every responsibility of the helper —
  TenantID=0 (system scope), actor from ctx, role hard-pinned to
  "system_admin", action passed through, outcome=success,
  TargetType="user", TargetID/TargetUserID echoing user.ID, details
  round-tripping through JSON.
- NilDetailsLeavesEmptyPayload: nil details map must NOT fabricate a
  payload; the DB column defaults to '{}' and emitting an explicit
  null would muddle "no extra context" filters.
- NilTargetStillEmitsRow: guards the nil-target defensive branch —
  promote/revoke always supply one today, but the row still goes out
  with empty target ids rather than crashing.
- IdempotentBranchSurvivesMarshal: pins the two boolean discriminator
  flags (promote.idempotent, revoke.changed) so the audit reader can
  distinguish a real grant from a probe and a real revoke from a noop.
  Regression guard against accidentally swapping the payload to
  stringly-typed shapes.
- LogErrorIsSwallowed: best-effort contract — a failing audit write
  must NOT propagate, because the underlying privilege change has
  already succeeded and bubbling the error would force the caller to
  retry or roll back, both strictly worse than log-and-continue.
2026-05-26 21:13:56 +08:00
wizardchen 5b9542ebe6 test(system): cover ListSystemAuditLog handler contract
Mirrors the existing TestAuditLogHandler_* suite for the new
GET /system/admin/audit-log endpoint:

- AlwaysQueriesTenantZero: the defining contract — handler must call
  AuditLogService.List with tenant_id=0 unconditionally, regardless of
  any URL/header input. A regression here would leak per-tenant rbac.*
  rows into the platform feed (or hide system.* rows from SystemAdmin).
- PassesQueryFiltersThrough: every advertised query key (after_id,
  limit, action, outcome, actor) propagates exactly. Catches typos in
  the param-key list.
- EmptyResultProducesZeroCursor: an empty service response must
  collapse next_cursor to 0 so the drawer's infinite-scroll watcher
  stops paginating.
- GarbageCursorAndLimitTolerated: malformed after_id / non-positive
  limit fall back to defaults (matches ListTenantAuditLog) instead of
  hard-failing, so stale URL params never blank-screen the drawer.
- ServiceErrorReturns500: List() errors surface as 500 via
  errors.NewInternalServerError + ErrorHandler middleware, with a
  non-empty body so the drawer alert has something to render.
2026-05-26 21:13:56 +08:00
wizardchen acdc0a526f feat(system): expose platform audit log + polish audit drawers
The system_settings update, system admin promote/revoke, and
apply-default-storage-quota routes have been writing audit rows since
the prior commits, but with TenantID=0 (system-scope). The per-tenant
GET /tenants/:id/audit-log endpoint filters by tenant_id and never
returns them, so until now those rows existed only in the DB with no UI
surface. This commit closes the loop:

Backend:
- GET /api/v1/system/admin/audit-log: new SystemAdmin-gated endpoint
  reusing AuditLogService.List with tenant_id=0. Same cursor-paged
  shape and query params (after_id / limit / action / outcome / actor)
  as the per-tenant feed, so the frontend reuses the same client logic.
- Wired through RegisterSystemAdminRoutes (mounted on the existing
  adminRoutes group so it inherits the SystemAdmin() guard). The
  handler dependency is optional: nil auditLogHandler skips the route,
  mirroring RegisterTenantRoutes' /audit-log handling.

Frontend — new platform audit drawer in SystemSettings.vue:
- "审计日志" entry button in the section header opens a side drawer
  (880px) listing system-scope events. Lazy-loaded on first open;
  refresh is explicit via a button inside the drawer.
- Table columns: stacked date/time (so 50 events in the same minute
  remain distinguishable), stacked actor/role, action tag, structured
  target (subject key + diff line), outcome. The dead request column
  (system actions don't go through middleware path capture) is dropped
  in favour of richer target rendering.
- Per-action target formatters:
    * system.setting_changed: subject = registry key, diff = `old → new`
      (JSON-encoded, 80-char truncation). Reset shows `old → (空)`.
    * tenant_storage_quota bulk apply: subject = "批量同步", diff =
      "applied to N tenants (X GB)".
    * system.admin_promoted / revoked: subject = "name (email)", diff
      annotates idempotent / noop branches so an audit reader can tell
      a real grant from a probe.
- Click-to-expand row reveals the full audit context: actor UUID,
  target_user_id / target_type / target_id, and raw details JSON in
  monospaced scroll-capped block. No psql round-trip needed for
  forensic spot checks.
- Sticky thead pinned to the scroll container so column labels survive
  long scrolls. Cells vertical-aligned middle to keep single-line tag
  cells visually balanced against multi-line target cells. No zebra
  stripes — the stacked content already provides row separation, and
  stripes on top read as noise.

Frontend — same polish back-ported to TenantMembers.vue audit drawer:
- Same drawer width, stacked time / actor cells, structured target +
  diff layout, expandable raw-details row, sticky thead, vertical-
  align middle, no stripes. Refresh button reformulated as a text
  button with label (was an outlined square icon-only).
- request_path column kept (rbac.access_denied carries meaningful
  paths) but empty values render as a placeholder dash so they don't
  read as broken.
- Diff line now covers rbac.invitation_sent / invitation_revoked role
  in addition to the existing role_changed / access_denied details.

API:
- frontend/src/api/system/index.ts: listSystemAuditLog() reuses the
  AuditLog / ListAuditLogParams types from @/api/tenant/audit-log
  (re-exported) so consumers don't need to cross-import.

i18n (zh-CN / en-US / ko-KR / ru-RU):
- system.globalSettings.audit.*: full drawer copy + per-action labels
  (system.setting_changed / admin_promoted / admin_revoked) + target
  diff templates + expanded-row labels.
- tenantMember.audit.expanded.*: expanded-row labels added so the
  shared drawer treatment renders cleanly under tenant scope.
2026-05-26 21:13:56 +08:00
wizardchen 9cce0c8e5e feat(system): consolidate system admin and settings into one Settings panel
Replace the standalone /platform/system/* routes with a single
"系统设置" section inside the canonical Settings modal. The previous
SystemLayout.vue / SystemAdmins.vue surfaces are removed and their
functionality (system admin roster, global settings) is hosted directly
in SystemSettings.vue under the standard `.section-header` /
`.settings-group` skeleton. Legacy URLs redirect to the modal section
so external bookmarks don't 404.

Backend:
- SystemSettingService.Reset + repo.Delete: drop the DB override for a
  key so the 3-tier resolver falls back to ENV / built-in default.
  Idempotent (resetting a never-persisted key returns nil); emits an
  audit row only on real deletions, invalidates the local cache, and
  publishes to peers via the existing pubsub channel.
- TenantService.BulkSetStorageQuota: overwrite every tenant's
  storage_quota in one statement. Powers POST /system/admin/tenants/
  apply-default-storage-quota; bypasses the per-tenant whitelist on
  PUT /tenants/:id which intentionally forbids storage_quota edits.
- AuditAction{SystemAdminPromoted,SystemAdminRevoked} constants and
  emitAdminAudit() in SystemHandler — promote / revoke now leave an
  audit trail with TenantID=0 and {target_email,target_username,
  idempotent|changed} in details.
- SystemSetting.LastModifiedByName: derived per-request display label
  (username, email fallback) so the UI shows "wizardchen" instead of
  a UUID prefix without storing a denormalised column.

Frontend:
- SystemSettings.vue rewritten against the Settings modal skeleton with
  auto-persisting controls (switch/select @change, input/inputnumber
  @blur, tag-input + per-delta popconfirm for SSRF whitelist and admin
  roster). auth.registration_mode change goes through an inline
  popconfirm; cancel rolls back. Reset / bulk-apply also inline
  popconfirms — no dialog modal for these per-row affordances.
- Priority hint panel surfaces the DB > ENV > default resolver order
  so operators can reason about "I set the env but it doesn't show up".
- Router: /platform/system, /platform/system/settings, /platform/system/
  admins are now compatibility redirects to /platform/settings?section=
  system-global.
- Settings modal sized 900x700 → 1080x780 and content-wrapper 600 → 760
  so the wider tables (members, system settings) breathe; <1100px
  viewport still flexes to the screen.
- i18n: system.globalSettings.* keys for title / description / loading /
  empty / badges / priority hint / per-key labels / reset / bulkApply /
  admins (label, placeholder, save messages, popconfirm copy) across
  zh-CN, en-US, ko-KR, ru-RU.

Misc:
- internal/utils/filesize.go doc: clarify MAX_FILE_SIZE_MB is a
  deploy-time-only knob (nginx + docreader + frontend each cache the
  env at startup); a SystemAdmin UI override would mislead operators
  because nginx would still 413. Until all four layers can hot-reload
  the limit in lockstep, this stays env-only.
- internal/utils/security.go: SSRF whitelist parser/runtime now drives
  off SystemSettingService for live updates; ENV remains the fallback
  for never-overridden deployments.
2026-05-26 21:13:56 +08:00
wizardchen 311a033523 refactor(migrations): remove deprecated user system admin migration files and introduce new system settings migration
- Deleted the old migration files for user system admin (000052) as they are no longer needed.
- Added new migration files (000053) to implement system-level administrator flag and a new system settings table, consolidating related changes for improved management of platform-wide settings.
- Ensured that the new migrations include proper rollback procedures for both the system admin flag and the system settings table.
2026-05-26 21:13:56 +08:00
wizardchen 554ba9ef74 feat(settings): refactor system settings management and UI enhancements
- Consolidated global system settings into the standard Settings modal, improving accessibility and user experience.
- Updated routing to redirect system settings to the new modal structure, ensuring a seamless transition for users.
- Enhanced UI components for better responsiveness and visual clarity, including adjustments to layout and styling.
- Added visibility controls for system admin-specific settings, ensuring proper access management.
- Refactored related comments and documentation for improved clarity on the new settings structure.
2026-05-26 21:13:56 +08:00
wizardchen d074dc067a feat(system-admin): implement revocation of system admin privileges with safeguards
- Added RevokeSystemAdmin functionality to the user service and repository, ensuring atomic checks for self-revoke and last admin scenarios.
- Updated the system handler to utilize the new revocation method, improving error handling for various edge cases.
- Enhanced the bootstrap process to prevent unintended promotions when system admins already exist.
- Refactored related comments and documentation for clarity on the new behavior and safeguards in place.
2026-05-26 21:13:56 +08:00
wizardchen 47a183aa65 feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management
- Added WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL environment variable to promote a specified user to system admin on startup.
- Introduced a new bootstrap process in `bootstrap.go` to handle the promotion logic.
- Updated `.env.example` to document the new environment variable and its behavior.
- Created new views for managing system administrators and system settings, including listing, promoting, and revoking admin privileges.
- Enhanced the frontend to reflect the new system admin features, including UI elements for admin management and settings configuration.
- Updated API interfaces to support system admin functionalities, ensuring proper data handling and user management.
2026-05-26 21:13:56 +08:00
wizardchen e6ee87759d fix(agent): preserve agent intent prompt whitespace and add tests
The intent prompt override logic in query_understand applied
strings.TrimSpace to the value before assigning it as the system prompt
override, which silently stripped trailing newlines and intentional
formatting from agent-supplied prompts. Use TrimSpace only to detect
emptiness (so whitespace-only strings still fall back to the global
default) while passing the raw string through verbatim.

Extract the resolution into applyIntentPromptOverride and add unit tests
covering agent-wins, whitespace preservation, blank fallback, no-config,
and global-only paths.
2026-05-26 21:05:40 +08:00
ochan.kwon 11c3236e52 feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.

Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.

What lands in PR 2a
-------------------

Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):

- `repository.go` — Repository struct + NewRepository constructor
  that validates cluster reachability + OS version (2.4+ / 3.x;
  primary tested 3.3.2) + k-NN plugin presence on every cluster
  node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
  dimension index creation, with transient errors not cached so a
  momentary cluster blip does not permanently poison a dim.
  sanitizeIndexName enforces a strict OS-compatible name spec.
  probeVersion uses robust strings.Split/Atoi parsing for
  pre-release suffixes and missing-patch versions. EngineType
  returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
  (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
  only cipher list) and transport tuning for the driver. Caller
  exists only in PR 3 (container.go + engine_factory.go); PR 2a
  remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
  knn_vector + HNSW + content-analyzer mapping with every *_id
  field as an explicit keyword and source_type as integer.
  buildKeywordsMapping ships the dim-less keyword-only index
  mapping used by the no-embedding save path. createIndexAndAlias
  creates <alias>_v1 and aliases <alias> to it, with best-effort
  orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
  defaults (hnsw_m=16, ef_construction=100, ef_search=100,
  shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
  ErrDimensionMismatch, ErrAuth, ErrTransport,
  ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
  ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
  apperrors; PR 3's engine factory wraps these to typed AppError
  2200/2201.
- `stubs.go` — every behavioural method returns
  ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
  HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
  fails-closed for non-empty KBs.

Tests (~1115 LoC, 50 cases):

- `repository_test.go` — interface satisfaction, sentinel mapping,
  sanitizeIndexName positive/negative matrix, semver parsing
  (pre-release / missing-patch), buildIndexMapping JSON shape pin
  (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
  2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
  multi-node coverage, ensureReady concurrency + per-dim isolation
  + transient retry, NewRepository storeID validation, all 11
  stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
  SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
  wrapTransport sentinel mapping + leak guard, isNotFound /
  isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
  TLS 1.2 pinning / cipher list / transport tuning.

Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.

SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------

PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):

- AliasExists method passes dataPointer=nil to its internal do(),
  which means non-2xx responses come back as a plain
  *errors.errorString ("status: 404 Not Found") rather than as
  *opensearch.StructError. aliasExists therefore inspects
  resp.StatusCode directly (resp is returned even when err is
  non-nil) and only falls back to wrapTransport for the "no
  response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
  index uses a mutex + ready/err flag pattern so transient failures
  can be retried by the next caller. The per-dimension path uses
  the `once map[int]*sync.Once` delete-and-recreate trick.

Test fixes folded in
--------------------

While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:

(1) Follow-up to #1445 — fanout test missed the new normalizer policy
    (internal/application/service/knowledgebase_search_fanout_test.go,
    +46 / -6). #1445 changed EngineAwareNormalizer for ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
    Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
    passthrough (those engines surface non-negative cosine to the
    normalizer per Lucene script_score non-negative invariant for
    ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
    OpenSearch, engine-internal or IR-normalized conversions for
    the rest). Milvus is now the only engine that still surfaces
    raw signed cosine in [-1, 1].

    TestRetrieveFromStores_MixedEngine_Normalizes still asserted
    the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
    which under passthrough now becomes clamp01(-0.4) = 0. The
    normalizer's own _test.go was updated at #1445 time, but this
    fan-out integration test was not.

    Fix: rewrite the godoc to spell out the two engine groups;
    restate sub-case 2 as ES passthrough on a production-possible
    mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
    pinning the cosine-shift branch via Milvus -0.4 → 0.3.

(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
    (internal/utils/security.go + internal/utils/security_test.go +
    internal/infrastructure/web_search/searxng_test.go,
    +33 / -9). loadSSRFWhitelist in internal/utils/security.go is
    cached via sync.Once on first call. The internal reset helper
    resetSSRFWhitelistForTest was unexported, so tests in other
    packages could not reset and saw whatever whitelist was cached
    by the first sync.Once.Do() in the same test binary. In
    internal/infrastructure/web_search/, TestValidateProxyURL runs
    before TestValidateSearxngBaseURL alphabetically and exercises
    ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
    whitelist; the later setenv in searxng_test then has no effect
    and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
    Pre-existing on main; surfaced now because this PR was the
    first to do a full `go test ./...` run on top of #1445.

    Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
    ForTest suffix is the test-only contract); update in-package
    callers; in web_search/searxng_test.go import internal/utils
    and call ResetSSRFWhitelistForTest around the env mutation in
    both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
    No production code path changes.

Roadmap
-------

- PR 2b (next, depends on this PR) — read/write implementations:
  query.go + retrieve.go + crud.go land their real bodies; stubs
  for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
  removed; corresponding CRUD/retrieve/filter test cases (~430
  LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
  BatchUpdate*, large-batch async deletes) + i18n + docker-compose
  dev profile. After PR 3 merges, the OpenSearch driver becomes
  reachable via either `engine_type=opensearch` VectorStore or
  `RETRIEVE_DRIVER=opensearch` env.

Backward compatibility
----------------------

- New package — additive only. No existing file modified except
  go.mod / go.sum, the two test files in (1)/(2), and the
  test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
  here (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-26 20:54:58 +08:00
liuwei435 e0c6599c54 fix(datasource): support Yuque team token in connector
- Add Type field to v2User struct to distinguish personal/team tokens
- Route team tokens directly to ListGroupRepos (skip ListUserGroups)
- Gracefully handle ListUserGroups 404 for personal tokens without teams
- Add rate-limiting delay between GetDocDetail calls to avoid API throttling
2026-05-26 20:46:22 +08:00
ChenRussell 80a69ae6f3 fix(pipeline): fallback to raw retrieval results when rerank API fails
When the rerank model returns an error (e.g. 401 Unauthorized), the
  pipeline previously discarded all retrieved candidates and returned
  empty results to the caller.

  Now p.rerank returns ([]RankResult, error) to distinguish API failure
  from threshold-filtered empty results. On API error, the pipeline
  falls back to the original retrieval candidates (directLoad +
  candidatesToRerank) and continues to CHUNK_MERGE/FILTER_TOP_K,
  so users still get results even when the rerank model is misconfigured.
2026-05-26 20:45:17 +08:00
helloandyzhang e6a469631b feat(agent): add intent prompt customization in agent editor
Allow per-intent system prompt overrides for non-retrieval intents in normal mode.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-26 20:41:10 +08:00
Miles Lai 09892ef763 fix(doris): inline LIMIT/OFFSET as literals and enable parameter interpolation
Problem:
Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing
VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime
with SQL parse errors.

Root cause:
1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in
   MySQL/PostgreSQL but not in Doris's SQL dialect)
2. Doris MySQL driver was not configured to interpolate parameters, preventing
   automatic client-side placeholder replacement

Fix:
1. Enable parameter interpolation in Doris connection DSN:
   - Add &interpolateParams=true to mysql.Open() DSN in container.go
   - This enables driver-level parameter substitution at query time

2. Inline LIMIT/OFFSET as string literals in Doris queries:
   - VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK)
   - KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK)
   - CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset)
   - Remove inlined values from args slice

Result:
Queries are now built with literal LIMIT/OFFSET values instead of placeholders,
compatible with Doris's SQL parser.
2026-05-26 20:32:49 +08:00
jackson.jia dbdae65e9b fix(input-field): wrong toast when selecting built-in agents other than quick-answer / smart-reasoning
When selecting an agent from the chat page's agent selector dropdown,
built-in agents other than `builtin-quick-answer` and
`builtin-smart-reasoning` trigger the wrong toast. "Switched to
Intelligent Reasoning" pops up regardless of which agent was actually
picked — Wiki Questioner, Data Analyst, Wiki Fixer, etc.

Root cause: the toast logic checks only `agent.is_builtin`, then
branches on `agent_mode === 'smart-reasoning'` to pick between the
"Switched to Intelligent Reasoning" / "Switched to Quick Q&A" strings.
Those strings were designed for the two original built-ins
(`builtin-quick-answer` and `builtin-smart-reasoning`), which the
dropdown re-brands as "Normal Mode" / "Agent Mode". They don't make
sense for any other built-in.

Narrow the special-case branch to those two IDs. Other built-ins fall
back to the generic `agentSelected` toast (the path custom agents
already take), so they get a toast naming the agent that was actually
selected.
2026-05-26 20:26:31 +08:00
helloandyzhang af2f55c29f fix(frontend): apply default context template when switching to quick-answer mode
Start new agents with an empty context_template so mode-switch logic can
populate the system default instead of keeping the smart-reasoning placeholder.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-26 20:13:31 +08:00