Commit Graph

62 Commits

Author SHA1 Message Date
“jiangyijun” 6bc8e9ed49 fix(vectordb): 修复 question 数据复制到向量数据库的 ID 映射问题
主要改进:
1. CopyIndices 新增分页查询逻辑(每页500条),避免大批量数据查询失败
2. 修复 ID 字段优先级:优先使用 IndexInfo.ID,兜底使用 ChunkID
3. 新增 translateSourceID 函数,智能处理 question 类型数据的 SourceID 映射:
   - 当 SourceID == ChunkID 时,直接映射为目标 ChunkID
   - 当 SourceID 为 "ChunkID-QuestionID" 格式时,保留 QuestionID 部分
   - 其他情况使用 SHA256 生成稳定的映射 ID

影响范围:
- 修改文件:internal/application/repository/retriever/tencentvectordb/repository.go
- 新增函数:copySourceQueryParams、remapCopiedEmbeddings、translateSourceID
2026-06-17 05:08:50 +08:00
langcaiye 903b014992 fix(tcvdb): keep default replica number at one 2026-06-12 22:54:44 +08:00
langcaiye d07a62af06 fix(tcvdb): improve collection compatibility 2026-06-12 22:54:44 +08:00
ochan.kwon 40b74e2efa feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3)
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.

Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
  bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
  buildEnvStoreForDriver — every gated surface now recognises
  "opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
  HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
  serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
  createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
  in initRetrieveEngineRegistry; NewEngineFactory now closes over the
  AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
  testOpenSearchConnection case so CreateStore's connectivity probe
  accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
  NewOpenSearchClient is exported so the factory and env path can build
  the TLS-hardened client; healthcheck.go reuses the unexported
  probeVersion / probeKNNPlugin for the service-layer probe.

Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
  ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
  Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
  Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.

Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
  search → BatchSave with the source_id remap for generated questions —
  so dim/keyword routing and the source_id contract come from BatchSave
  for free. embeddingMap is keyed by the *target* SourceID because
  OpenSearch's BatchSave looks up embeddings by SourceID
  (lookupEmbedding), not by chunk_id (the ES driver's convention).
  Pagination is from/size; copies larger than max_result_window
  (default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
  group the input by target value and issue one _update_by_query per
  group over the cross-dim <base>_* pattern. Caller values flow through
  bound script params only — never string-interpolated into the Painless
  source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
  full failure reason goes to the debug log only; the returned error
  carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
  shape as DeleteByQuery's quirk), so refresh=wait_for is not
  expressible; we use refresh=true.

Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
  functional option) lets the driver emit opensearch.index_created and
  opensearch.reindex_executed events without importing any service
  package — the service layer implements the interface. NewRepository
  takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
  When the context carries no tenant (the env-path registration ctx
  during boot, for example) the adapter skips the emit with a warning
  rather than silently writing tenant_id=0, which would collide with the
  system-scope sentinel.

Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
  immutable. VectorStoreSettings.vue is now schema-driven: a closed
  `enum` renders a t-select; number inputs use the schema's `:min`/`:max`
  and fall back to the legacy replica-vs-shard heuristic only when the
  schema does not pin them; a danger-coloured warning fires when
  insecure_skip_verify is toggled on (the switch and warning are wrapped
  in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
  knn_engine / insecure_skip_verify plus the warning copy in en-US,
  ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
  security plugin disabled for dev only). OpenSearch Dashboards lives in a
  separate, opt-in opensearch-ui profile so the heavy UI container is not
  forced up alongside the cluster (the driver e2e is fully curl-verifiable
  against :9200). The new docs/dev/opensearch-integration-test.md covers the
  end-to-end exercise and the single-node guidance (set replicas=0 to keep
  the cluster Green).

Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
  env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
  their positive counterparts in this PR. The test suite was the
  activation checklist; the activation flip is its diff.

Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
  other engines' serialised config is byte-identical. Existing
  Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
  stores are untouched. No migrations.

Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
  endpoint), the docreader gRPC tests, and the doris SQL-shape tests
  fail; all three are pre-existing on upstream/main and untouched by
  this PR.
- New tests across internal/types, opensearch, service and container —
  including a full end-to-end env-path test that exercises
  initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
  httptest cluster.
2026-05-29 16:32:27 +08:00
Ochan Kwon f5820c13ac feat(retriever): implement OpenSearch driver read/write paths (PR 2b of 3) (#1482)
Second half of the gated OpenSearch k-NN driver (#1440). Replaces the
PR 2a stubs for Save / BatchSave / DeleteBy* / Retrieve with their
real implementations, drops the stubs from `stubs.go`, and adds the
27 corresponding test cases that were intentionally held back. The
driver remains gated dead code (no registry / factory / env path
mentions OpenSearch — PR 3 ships the activation switch); the
behaviour added here only fires once PR 3 lands.

Depends on PR 2a of #1440 (driver skeleton).

What lands in PR 2b
-------------------

Three new production files (+762 LoC):

- `query.go` (+154) — typed retrieveFilters struct as the only entry
  to query construction (closes the map[string]any injection surface;
  hostile chunk_id stays JSON-escaped). buildKNNQuery (pure k-NN,
  applies min_score when Threshold > 0) and buildKeywordQuery (BM25);
  native hybrid intentionally omitted (out of scope for Phase 3 —
  service-layer RRF stays in charge). Filter struct carries
  ExcludeKnowledgeIDs and an opt-in IncludeDisabled for admin-mode
  use cases.
- `retrieve.go` (+204) — Retrieve resolves the per-dim alias from
  (1) params.AdditionalParams["dim"], (2) len(params.Embedding), or
  (3) the cross-dim wildcard <base>_* for BM25-only callers. Always
  returns exactly one *RetrieveResult (the Phase 2 fan-out contract).
  Hit parsing caps response body at 16 MB and warns if _id !=
  _source.chunk_id. Score passes through raw in [0, 1] — the k-NN
  plugin's SpaceType.COSINESIMIL.scoreTranslation has already mapped
  (1 + cos) / 2 before the client sees it.
- `crud.go` (+405) — Save / BatchSave with _id = chunk_id for
  idempotency, dim-aware pre-marshal NDJSON size cap (rejects with
  ErrBatchTooLarge before allocating a 50 MB body for a too-big
  batch), and per-item error inspection that does NOT leak cluster-
  side Reason strings into wrapped public errors. additionalParams
  contract verified against elasticsearch/structs.go
  ToDBVectorEmbedding + keywords_vector_hybrid_indexer.go: embedding
  is map[string][]float32 keyed by SourceID, chunk_enabled is
  map[string]bool keyed by ChunkID. Missing embedding routes Save /
  BatchSave to the dim-less <base>_keywords index. Sync DeleteBy*
  paths capped at 1000 IDs per call with ErrBatchTooLarge.

  toDoc writes the is_recommended boolean from IndexInfo. The
  matching mapping entry was added in PR 2a alongside is_enabled —
  the pair stays in sync between the vector and the keyword-only
  index so the field round-trips through CopyIndices (PR 3) and any
  future FAQ-priority filtering.

Stubs removed (`stubs.go`, -86 LoC): Save / BatchSave / Retrieve /
DeleteByChunkIDList / DeleteBySourceIDList / DeleteByKnowledgeIDList
no longer return ErrFeatureNotEnabled — they delegate to the
production implementations above. The remaining stubs (CopyIndices,
BatchUpdateChunkEnabledStatus, BatchUpdateChunkTagID, swapToVersion,
EstimateStorageSize lower-bound) stay until PR 3.

Tests (+490 net to repository_test.go, 28 new cases):

- retrieveFilters: JSON injection guard, ExcludeKnowledgeIDs,
  IncludeDisabled skip semantics, default is_enabled=true pin.
- buildKNNQuery: min_score applied when Threshold > 0; omitted at
  threshold zero.
- buildKeywordQuery: content match appended after typed filters.
- toDoc: embedding field omitted when empty, present when set,
  source_type serialised as integer, **is_recommended round-trips
  both true and false**.
- lookupEmbedding / lookupChunkEnabled: SourceID / ChunkID key
  contract; nil-params and wrong-type degrade paths.
- extractBatchEmbeddings: mixed-dim rejection; all-empty embeddings
  yield dim=0.
- estimateBulkBodyBytes: size estimator boundary.
- effectiveTopK: clamping when caller leaves TopK at the magic-10
  default.
- resolveDim: AdditionalParams takes precedence; Embedding fallback;
  zero-zero triggers cross-dim multi-index search.
- inspectBulkResponse: all-succeeded fast path; total-count + leak-
  guard for cluster-side Reason strings.
- BatchSave cap: 1001 infos returns ErrBatchTooLarge (pre-marshal);
  empty list is a no-op.
- DeleteByChunkIDList cap: 1001 IDs returns ErrBatchTooLarge; empty
  list is a no-op.

SDK quirk addressed in PR 2b
----------------------------

DocumentDeleteByQueryParams.Refresh is *bool, so the wire-level
"wait_for" value documented in the OpenSearch REST API is not
directly expressible. deleteByTerms uses Refresh:&true which forces
immediate segment flush (read-your-writes guaranteed but more
expensive than wait_for). A follow-up PR can drop to
*opensearchapi.Client.Transport.Perform for wait_for if telemetry
shows the cost is material.

Backward compatibility
----------------------

- Additive within the opensearch package only. No file outside the
  package is touched.
- Driver still unreachable: PR 3 ships the activation switch.
- The PR 1 normalizer case for OpenSearch remains unreachable here
  (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-28 08:00:33 +08:00
Sheep44044 3645477dba refactor(qdrant): optimize batch save, map iteration, and error wrapping (#1508) 2026-05-28 07:39:41 +08:00
ochan.kwon 11c3236e52 feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.

Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.

What lands in PR 2a
-------------------

Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):

- `repository.go` — Repository struct + NewRepository constructor
  that validates cluster reachability + OS version (2.4+ / 3.x;
  primary tested 3.3.2) + k-NN plugin presence on every cluster
  node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
  dimension index creation, with transient errors not cached so a
  momentary cluster blip does not permanently poison a dim.
  sanitizeIndexName enforces a strict OS-compatible name spec.
  probeVersion uses robust strings.Split/Atoi parsing for
  pre-release suffixes and missing-patch versions. EngineType
  returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
  (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
  only cipher list) and transport tuning for the driver. Caller
  exists only in PR 3 (container.go + engine_factory.go); PR 2a
  remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
  knn_vector + HNSW + content-analyzer mapping with every *_id
  field as an explicit keyword and source_type as integer.
  buildKeywordsMapping ships the dim-less keyword-only index
  mapping used by the no-embedding save path. createIndexAndAlias
  creates <alias>_v1 and aliases <alias> to it, with best-effort
  orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
  defaults (hnsw_m=16, ef_construction=100, ef_search=100,
  shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
  ErrDimensionMismatch, ErrAuth, ErrTransport,
  ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
  ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
  apperrors; PR 3's engine factory wraps these to typed AppError
  2200/2201.
- `stubs.go` — every behavioural method returns
  ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
  HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
  fails-closed for non-empty KBs.

Tests (~1115 LoC, 50 cases):

- `repository_test.go` — interface satisfaction, sentinel mapping,
  sanitizeIndexName positive/negative matrix, semver parsing
  (pre-release / missing-patch), buildIndexMapping JSON shape pin
  (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
  2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
  multi-node coverage, ensureReady concurrency + per-dim isolation
  + transient retry, NewRepository storeID validation, all 11
  stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
  SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
  wrapTransport sentinel mapping + leak guard, isNotFound /
  isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
  TLS 1.2 pinning / cipher list / transport tuning.

Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.

SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------

PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):

- AliasExists method passes dataPointer=nil to its internal do(),
  which means non-2xx responses come back as a plain
  *errors.errorString ("status: 404 Not Found") rather than as
  *opensearch.StructError. aliasExists therefore inspects
  resp.StatusCode directly (resp is returned even when err is
  non-nil) and only falls back to wrapTransport for the "no
  response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
  index uses a mutex + ready/err flag pattern so transient failures
  can be retried by the next caller. The per-dimension path uses
  the `once map[int]*sync.Once` delete-and-recreate trick.

Test fixes folded in
--------------------

While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:

(1) Follow-up to #1445 — fanout test missed the new normalizer policy
    (internal/application/service/knowledgebase_search_fanout_test.go,
    +46 / -6). #1445 changed EngineAwareNormalizer for ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
    Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
    passthrough (those engines surface non-negative cosine to the
    normalizer per Lucene script_score non-negative invariant for
    ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
    OpenSearch, engine-internal or IR-normalized conversions for
    the rest). Milvus is now the only engine that still surfaces
    raw signed cosine in [-1, 1].

    TestRetrieveFromStores_MixedEngine_Normalizes still asserted
    the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
    which under passthrough now becomes clamp01(-0.4) = 0. The
    normalizer's own _test.go was updated at #1445 time, but this
    fan-out integration test was not.

    Fix: rewrite the godoc to spell out the two engine groups;
    restate sub-case 2 as ES passthrough on a production-possible
    mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
    pinning the cosine-shift branch via Milvus -0.4 → 0.3.

(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
    (internal/utils/security.go + internal/utils/security_test.go +
    internal/infrastructure/web_search/searxng_test.go,
    +33 / -9). loadSSRFWhitelist in internal/utils/security.go is
    cached via sync.Once on first call. The internal reset helper
    resetSSRFWhitelistForTest was unexported, so tests in other
    packages could not reset and saw whatever whitelist was cached
    by the first sync.Once.Do() in the same test binary. In
    internal/infrastructure/web_search/, TestValidateProxyURL runs
    before TestValidateSearxngBaseURL alphabetically and exercises
    ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
    whitelist; the later setenv in searxng_test then has no effect
    and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
    Pre-existing on main; surfaced now because this PR was the
    first to do a full `go test ./...` run on top of #1445.

    Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
    ForTest suffix is the test-only contract); update in-package
    callers; in web_search/searxng_test.go import internal/utils
    and call ResetSSRFWhitelistForTest around the env mutation in
    both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
    No production code path changes.

Roadmap
-------

- PR 2b (next, depends on this PR) — read/write implementations:
  query.go + retrieve.go + crud.go land their real bodies; stubs
  for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
  removed; corresponding CRUD/retrieve/filter test cases (~430
  LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
  BatchUpdate*, large-batch async deletes) + i18n + docker-compose
  dev profile. After PR 3 merges, the OpenSearch driver becomes
  reachable via either `engine_type=opensearch` VectorStore or
  `RETRIEVE_DRIVER=opensearch` env.

Backward compatibility
----------------------

- New package — additive only. No existing file modified except
  go.mod / go.sum, the two test files in (1)/(2), and the
  test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
  here (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-26 20:54:58 +08:00
Miles Lai 09892ef763 fix(doris): inline LIMIT/OFFSET as literals and enable parameter interpolation
Problem:
Doris does not support ? placeholders for LIMIT and OFFSET clauses, causing
VectorRetrieve, KeywordsRetrieve, and CopyIndices queries to fail at runtime
with SQL parse errors.

Root cause:
1. Query builders passed TopK/pageSize/offset as ? bind parameters (valid in
   MySQL/PostgreSQL but not in Doris's SQL dialect)
2. Doris MySQL driver was not configured to interpolate parameters, preventing
   automatic client-side placeholder replacement

Fix:
1. Enable parameter interpolation in Doris connection DSN:
   - Add &interpolateParams=true to mysql.Open() DSN in container.go
   - This enables driver-level parameter substitution at query time

2. Inline LIMIT/OFFSET as string literals in Doris queries:
   - VectorRetrieve: LIMIT ? → LIMIT %d (params.TopK)
   - KeywordsRetrieve: LIMIT ? → LIMIT %d (params.TopK)
   - CopyIndices: LIMIT ? OFFSET ? → LIMIT %d OFFSET %d (pageSize, offset)
   - Remove inlined values from args slice

Result:
Queries are now built with literal LIMIT/OFFSET values instead of placeholders,
compatible with Doris's SQL parser.
2026-05-26 20:32:49 +08:00
wolfkill 6331cce23d fix: skip empty milvus enabled status groups 2026-05-25 16:54:56 +08:00
MarionMa 0a9a920460 perf(repository): exclude embedding field from Elasticsearch search response to reduce size 2026-05-21 21:11:19 +08:00
Miles Lai e62c0563aa doris: add configurable compatibility modes and guard mode switches
Problem:
The hard-coded Doris vector function implementation (cosine_distance_approximate with
UNIQUE KEY ANN tables) fails on SelectDB 4.0.2-rc01 and other Doris builds lacking that
specific function support. Users had no way to adapt without code changes.

Root cause:
There is assumption all Doris deployments support the same vector function API, but different
builds (Doris OSS, SelectDB, Doris Cloud) ship with different function variants and table
key constraints. No capability detection or user configuration existed.

Solution:
Implement DORIS_COMPAT_MODE environment variable with three modes:

  * auto (default/recommended): probe Doris server on first use to detect available vector
    functions; prefer inner_product_duplicate (modern Doris 4.0+), fall back to legacy
    (older builds lacking inner_product_approximate)

  * legacy: hard-set to cosine_distance_approximate + UNIQUE KEY (for older Doris/SelectDB
    builds without inner_product_approximate support)

  * inner_product_duplicate: hard-set to inner_product_approximate + DUPLICATE KEY
    (for modern Doris 4.0+ and current SelectDB with normalized embeddings)

Implementation details:
- add compat.go with one-time mode resolution (sync.Once) and capability probing
- inspect existing weknora_embeddings_* table DDL via SHOW CREATE TABLE to detect and
  enforce schema compatibility; prevents silent mismatches
- fail fast with clear error message when configured mode does not match existing tables,
  with explicit remediation steps (recreate tables or change env var)
- branch all query paths (inner_product_approximate vs cosine_distance_approximate),
  DDL generation (DUPLICATE KEY vs UNIQUE KEY), write paths (embed normalization),
  and chunk updates (Stream Load vs read-modify-write) by resolved compat mode
- add comprehensive repository tests for mode selection, auto-detection, and mismatch
  scenarios; all tests pass
- expose DORIS_COMPAT_MODE in docker-compose.yml with auto as default
- document in .env.example with clear mode decision guidance
- log all mode decisions (requested, detected, probed, final) at INFO/WARN level

Key guarantee:
⚠️ DORIS_COMPAT_MODE is NOT interchangeable after embedding tables are created.
App will reject mode switches that conflict with existing table layout, preventing
silent data mismatches and query failures.
2026-05-19 17:15:59 +08:00
wizardchen a082c04d28 chore(deps): update dependencies in /docreader and adjust dependabot configuration
- Updated `pydantic` from 2.12.3 to 2.13.4 and `pypdfium2` from 5.0.0 to 5.8.0 in the `docreader` requirements.
- Modified the dependabot configuration to set `open-pull-requests-limit` to 0 and added an `ignore` rule for version updates across all ecosystems, allowing only security updates.
- Adjusted settings for `server-security`, `client-security`, `frontend-security`, and `miniprogram-security` groups to streamline security update handling.

This change aims to enhance dependency management and maintain security while reducing noise from version update PRs.
2026-05-12 14:58:52 +08:00
langcaiye 02be05c095 feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
langcaiye 74b1342440 feat: add Tencent VectorDB retriever backend 2026-05-09 13:14:01 +08:00
wizardchen fc6f160eff fix(retriever/doris): code review cleanup
针对 4cce6f2e(接入 Apache Doris)的 code review 修复,主要修正若干阻断性
问题与可读性问题,并剔除不应进入主仓的本地工作流文件。

阻断性修复:
- docker-compose: Doris 镜像由 2.1.0 升至 4.1.0。原 2.1.0 不支持 HNSW
  ANN、cosine_distance_approximate 与 Stream Load partial_columns,
  按当前 DDL 一启动就会失败。
- DSN 字面量拼接改用 mysql.Config.FormatDSN()。原 fmt.Sprintf 在用户名/
  密码包含 `@`/`:`/`/` 等字符时会跑偏。覆盖 health check 与 engine
  factory 两处。

健壮性修复:
- 新增 validateEmbedding,写入与查询前拒绝 NaN/±Inf;strconv.FormatFloat
  对非有限值会输出 "NaN"/"+Inf" 拼成无效 SQL。
- waitANNReady 改为后台 goroutine + 独立 context,避免新维度首次写入路径
  阻塞最长 30s;ANN 未就绪时 Doris 会自动退化为 brute-force。

清理:
- annIndexReady 移除最终两个分支都 return true 的冗余写法。
- Save 移除冗余的双重 toDorisVectorEmbedding。
- testDorisConnection 把 "5.7.99 Doris-4.1.0" 解析为裸 "4.1.0",与
  Postgres/ES 的版本格式对齐。

剔除(不应合入主仓):
- docs/wiki/集成扩展/Doris改动与上游同步.md:纯 fork 维护工作流文档。
- scripts/e2e-doris.sh:作者本地 E2E 验证清单。

测试:
- repository_test 用 require.Eventually 适配 ANN 异步轮询。
- 现有 doris 单测全部通过。
2026-05-09 00:31:03 +08:00
issunion 4cce6f2e99 feat(retriever): 接入 Apache Doris 4.1 作为向量数据库
为 RetrieveEngine 体系新增 Doris 后端,与现有 Qdrant/Milvus/Weaviate
等保持完整能力对齐:向量检索、关键词检索、健康检查、环境变量与多实例
DB 配置、前端类型注册、单元测试、Docker Compose 模板。

实现要点:
- 协议分工:主链路用 MySQL 协议(database/sql + go-sql-driver/mysql)
  做 DDL / 查询 / 删除;批量更新走 Stream Load HTTP API,并启用
  partial_update=true、merge_type=APPEND,自动按 1MiB 切分批次并处理
  307 重定向。
- 表结构:UNIQUE KEY(id) + enable_unique_key_merge_on_write=true 以
  支持 upsert/部分列更新;按维度分表(<base>_<dim>),每张表上建
  HNSW ANN 索引(metric_type=cosine_distance)和 INVERTED 索引
  (parser=chinese)。
- 分数语义:使用 cosine_distance_approximate,再以 1 - dist 转换为
  "越大越相似",与现有 KVHybridRetrieveEngine 约定一致。
- 异步索引:ANN 索引为后台构建,ensureTable 通过轮询 SHOW INDEX 等
  待索引就绪后再放行写入,避免首次检索召回为空。
- ARRAY<FLOAT> 序列化:go-sql-driver/mysql 不支持数组占位符,
  embeddingLiteral 将 []float32 转成 SQL 字面量字符串再拼接。

新增文件:
- internal/application/repository/retriever/doris/{structs,schema,
  query,repository,streamload,repository_test}.go
- scripts/e2e-doris.sh:E2E 验证清单
- docs/wiki/集成扩展/Doris改动与上游同步.md:fork-and-rebase 工作流
  与改动清单

修改文件(接线 + 文档):
- internal/types/{retriever,tenant,vectorstore}.go:新增
  DorisRetrieverEngineType、env 解析、表单 schema 与索引参数校验
- internal/container/{container,engine_factory}.go:环境变量驱动
  与 VectorStore 配置驱动两条路径都支持 Doris
- internal/application/service/vectorstore{,_healthcheck}.go:连接
  校验 + Ping/Version 健康检查
- docker-compose.yml:新增 doris-fe / doris-be 服务(profile=doris)
- .env.example:DORIS_* 环境变量与示例
- docs/{使用其他向量数据库,wiki/集成扩展/集成向量数据库}.md:
  使用说明与索引/分数行为说明

依赖:go.mod/go.sum 新增 github.com/go-sql-driver/mysql(运行时)和
github.com/DATA-DOG/go-sqlmock(测试)。

测试:repository 层 SQL 形状、Stream Load HTTP 行为、whereBuilder
逻辑、embeddingLiteral 往返、健康检查错误路径均有单测覆盖。
2026-05-08 21:59:35 +08:00
wizardchen f07561f6b6 fix: resolve failing test and build errors
Made-with: Cursor
2026-04-22 21:17:29 +08:00
ochan.kwon 927d3debff feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
wizardchen 092044b68f refactor(postgres): optimize vector retrieval and HNSW settings
- Adjusted the expanded TopK limit to a maximum of 200 candidates to enhance HNSW efficiency.
- Updated the vector retrieval query to ensure the ORDER BY expression matches the indexed expression, improving index usage.
- Implemented local settings for HNSW's ef_search and iterative_scan within a transaction to optimize candidate retrieval.
- Added fallback logic to handle unsupported GUCs gracefully, ensuring query execution continues even with older pgvector versions.

These changes improve the performance and reliability of vector retrieval in the PostgreSQL implementation.
2026-04-17 12:11:03 +08:00
wizardchen dcb7fd05b1 Fix compile error in weaviate repository 2026-04-16 18:37:31 +08:00
wizardchen af58e9cff4 fix(repository): ensure expandedTopK is at least params.TopK in VectorRetrieve
- Added a check to set expandedTopK to params.TopK if it is lower, ensuring the subquery limit is not less than the final limit.
- Removed unnecessary type casting in the SQL query for embedding distance calculations, simplifying the query structure.

This update improves the accuracy of vector retrieval operations in the PostgreSQL repository.
2026-04-16 18:13:20 +08:00
wizardchen 9bb4a12562 feat(repository): update keyword retrieval logic to use ParadeDB's matching operator
- Modified the SQL clause in the KeywordsRetrieve function to utilize ParadeDB's ||| operator for improved token matching.
- Enhanced the handling of retrieval results in the fuseOrDeduplicate function to maintain original scores for keyword-only results, ensuring better performance in FAQ scenarios.

This update optimizes the keyword retrieval process and improves the accuracy of search results.
2026-04-15 20:16:02 +08:00
wizardchen a655a25cf8 fix: revert to neo4j v6 and workaround Go 1.24 Windows compiler bug with -p=1
Go 1.24 has an internal compiler race condition on Windows that causes
'package without types' errors during parallel compilation. Setting
GOFLAGS="-p=1" serializes the build and avoids triggering the bug.
2026-04-11 22:08:02 +08:00
wizardchen 9525f08038 fix: downgrade neo4j driver from v6 to v5 to fix Go 1.24 Windows internal compiler bug 2026-04-11 21:59:03 +08:00
wizardchen 9a2b54536d fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
wizardchen 9d021a29b7 feat(elasticsearch): enhance ID field handling with dynamic suffix detection
- Added logic to determine whether to append the ".keyword" suffix to ID fields based on index mapping.
- Introduced `detectFieldTypes` method to inspect index mappings and set the `useKeywordSuffix` flag accordingly.
- Updated methods to utilize the new `idField` function for consistent ID field querying across the repository.
- Improved logging for better visibility into mapping detection and suffix usage.
2026-04-02 15:26:34 +08:00
ochan.kwon 258a786129 Revert "refactor(elasticsearch): remove ".keyword" suffix from ID fields in Elasticsearch queries"
This reverts commit 3daaabfc4a.
2026-04-01 20:41:24 +08:00
wizardchen 3daaabfc4a refactor(elasticsearch): remove ".keyword" suffix from ID fields in Elasticsearch queries
- Updated Elasticsearch repository methods to use ID fields without the ".keyword" suffix, aligning with the new index mapping requirements.
- Enhanced documentation for the VectorEmbedding struct to clarify expected index mapping and usage guidelines for ID fields.
- Ensured consistency across all repository methods that interact with ID fields, improving query performance and maintainability.
2026-03-25 22:08:29 +08:00
Dounx 8df12aeee2 fix: make Milvus vector metric type configurable via MILVUS_METRIC_TYPE 2026-03-25 21:20:49 +08:00
Dounx e11bd8c57d fix: use milvus result set length for empty search results 2026-03-20 18:22:40 +08:00
wizardchen bc2cb96d70 feat: enhance multimodal support and image handling
- Updated configuration to improve handling of multimodal queries, including clearer guidelines for when to use [NO_SEARCH] in responses.
- Enhanced the user interface to include dedicated sections for multimodal configuration, allowing users to manage image uploads and related settings more effectively.
- Added localization support for new multimodal features in both English and Chinese, improving accessibility for a broader audience.
- Implemented backend changes to support image analysis and extraction of OCR text, enriching the interaction experience for users.

These enhancements significantly improve the application's capability to process and respond to multimodal queries, fostering a more interactive and user-friendly environment.
2026-03-12 10:32:26 +08:00
wizardchen 33ccc41bea refactor: update storage size calculations for vector embeddings
- Revised the storage size calculation logic for Milvus, Qdrant, and Weaviate repositories to provide more accurate estimates based on the actual data structure and indexing methods.
- Updated comments to clarify the relationship between vector dimensions and index storage requirements, enhancing code readability and maintainability.
2026-03-09 13:23:16 +08:00
MaoMengww e898d62141 refactor: change Tokenization Method 2026-03-09 10:36:16 +08:00
MaoMengww c4a5a4d99b feat: support weaviate vectordb for knowledge retrieve 2026-03-09 10:36:16 +08:00
DaWesen 65743da754 fix: 修复 Milvus retriever 相关问题
- 修正 envMilvusCollection 字段名拼写错误
- 将 log.Errorf 中的 %w 改为 %v,修复日志格式化问题
- 保留错误包装功能,确保错误处理的完整性
2026-03-06 10:33:09 +08:00
wizardchen 931bd38138 feat(repository): add TagID and IsRecommended fields to vector embeddings
- Introduced `TagID` and `IsRecommended` fields across various repository implementations (Elasticsearch, Milvus, Postgres, Qdrant, SQLite) to enhance document categorization and recommendation capabilities.
- Updated the logic for handling `IsEnabled` to ensure it reflects the actual state from the source data.
- Adjusted the knowledge service to incorporate the new `IsRecommended` field, improving the indexing process for knowledge chunks.

These changes enhance the flexibility and functionality of document retrieval and indexing across the application.
2026-03-02 21:43:30 +08:00
wizardchen 7b8a78aa2a feat(repository): enhance document indexing with isEnabled and isRecommended fields
- Added `IsEnabled` and `IsRecommended` fields to the document indexing process in both ElasticFaiss and Elasticsearch repositories, improving the flexibility and functionality of document retrieval.
- Updated the data extraction process to include these new fields, ensuring they are correctly populated from the source data.
- Adjusted the indexing logic in the knowledge service to reflect these changes, maintaining consistency across the application.
2026-03-02 21:43:30 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
joeyczheng 1b56e99cac feat: support milvus vectordb for knowledge retrieve
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-27 09:51:17 +08:00
wizardchen da1dcb0cda feat: 批量更新tag时更新索引字段 2025-12-30 19:44:53 +08:00
wizardchen ac00206e77 feat: 支持FAQ两级优先级标签过滤
添加FAQ搜索支持两级优先级标签过滤功能,新增first_priority_tag_ids和second_priority_tag_ids字段,更新各检索器实现以支持按标签ID过滤,优化搜索结果排序确保优先级标签结果优先展示。
2025-12-30 19:44:53 +08:00
wizardchen de1ffa7f7c feat: 支持输入框内选择知识库和文件,优化选择交互体验 2025-12-22 13:11:31 +08:00
wizardchen 3977d90181 feat: 完善知识库复制时生成问题的SourceID转换逻辑和标签映射功能 2025-12-17 22:04:18 +08:00
wizardchen b70450ddd8 feat: 支持FAQ索引类型分离,新增is_enabled字段过滤和批量更新优化 2025-12-17 22:04:18 +08:00
wizardchen 1f6bb3f841 feat: 支持Qdrant多维度向量存储和全文检索优化,添加Docker Compose配置 2025-12-08 19:38:16 +08:00
wizardchen d23b917667 feat: Add chunk enabled status support in Qdrant repository with batch update and deletion by source ID functionality 2025-12-05 14:31:16 +08:00
wizardchen b1c79ccd4d Merge remote-tracking branch 'github_public/main' 2025-12-05 14:21:56 +08:00
wizardchen a50f53d205 feat: Implement delete functionality for generated questions in knowledge base, including API and UI updates 2025-12-04 01:26:30 +08:00
wizardchen 16562eebbb feat: 添加AI问题生成功能,支持配置生成数量并在前端展示生成的问题 2025-12-03 01:42:31 +08:00
wizardchen 93af4460d6 refactor: Remove agent_test command-line tool and associated files to streamline project structure and eliminate unused components 2025-12-01 17:24:57 +08:00