Commit Graph

21 Commits

Author SHA1 Message Date
wizardchen 1778c5ac7b refactor(tracing): remove Jaeger integration and related components
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
2026-06-08 21:01:04 +08:00
wizardchen 2241f9579d refactor(keywords_vector_hybrid_indexer): enhance image payload sanitization logic
- Added a check to skip regex processing for non-image content, improving performance when no inline base64 payload is present.
- Updated regex pattern to ensure it correctly matches base64 image data URIs.
2026-05-29 16:56:53 +08:00
wolfkill 6b3ad07543 fix embedding input image payload safety 2026-05-29 16:53:12 +08:00
ochan.kwon 744a367f16 feat(retriever): add OpenSearch type prep ahead of Phase 3 driver
Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.

Changes:

- OpenSearchRetrieverEngineType constant ("opensearch") in
  internal/types/retriever.go. Not added to validEngineTypes /
  GetVectorStoreTypes / retrieverEngineMapping yet (gated).

- ConnectionConfig.InsecureSkipVerify (bool, default false) in
  internal/types/vectorstore.go, placed inside the // Common section
  because it is a cross-driver TLS option. Distinct from the
  Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
  only controls verification of an already-TLS HTTPS connection.
  AES-GCM Value/Scan round-trips the field as plaintext (it travels
  alongside the encrypted Password / APIKey but is not sensitive itself).

- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
  Min/Max (*float64), Enum ([]string). All omitempty so existing UI
  schema entries serialize identically. The fields will drive the UI
  in Phase 3 PR 3 (read-only on Edit + range/enum constraints).

- Six new AuditAction constants under vector_store.* and opensearch.*
  namespaces. Definitions only — emission lands in Phase 3 PR 3.

- EngineAwareNormalizer.Normalize is restructured to group engines by
  the effective score range observed by the normalizer (not the
  theoretical raw cosine range):

    Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
      - Milvus (COSINE metric mode). Milvus docs explicitly state the
        COSINE metric range is [-1, 1].

    Range [0, 1] (passthrough — already on the target scale):
      - Elasticsearch v8 / ElasticFaiss. The driver issues a
        cosineSimilarity(...) script_score script, and Lucene rejects
        negative final scores ("Final relevance scores from the
        script_score query cannot be negative" — ES docs); the
        effective range observed by the normalizer is therefore [0, 1]
        for IR-normalized embeddings. ES was previously grouped with
        Milvus and over-corrected via (score + 1) / 2, which inflated
        every ES result by 50% in mixed-engine RRF fusion.
      - OpenSearch. The k-NN plugin's
        SpaceType.COSINESIMIL.scoreTranslation maps the underlying
        Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
        before the score reaches us. Source:
        github.com/opensearch-project/k-NN at
        src/main/java/org/opensearch/knn/index/SpaceType.java
        (COSINESIMIL enum, scoreTranslation method).
      - Weaviate. The driver requests `certainty`, defined by Weaviate
        as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
      - Postgres pgvector / SQLite sqlite-vec / Qdrant /
        TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
        or normalized inner_product, whose theoretical range is [-1, 1].
        The IR-normalized positive-component unit vectors WeKnora
        targets (BGE / OpenAI text-embedding-3 / Cohere /
        sentence-transformers) keep the observed range in [0, 1];
        negative-cosine embedding models would silently clamp to 0
        downstream — explicitly documented as the IR-normalization
        caveat in the struct godoc.

  Dead enum references (ElasticFaiss, Infinity) are flagged in the
  godoc with a pointer to internal/types/vectorstore.go's existing
  "legacy/experimental, no standalone deployable instance" annotation.
  Their case labels are kept for switch exhaustiveness.

Test coverage:

- OpenSearch constant wire value + collision check against existing
  10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
  field deserializes as false) + round-trip + AES-GCM coexistence
  with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
  + *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
  for vector_store.* and opensearch.*, no wire-string collisions, exact
  wire-value pins for the six new constants.
- EngineAwareNormalizer:
  - CosineRange retains its (score + 1) / 2 coverage but now only for
    Milvus.
  - UnitInterval now covers the full passthrough group (ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
    Infinity / TencentVectorDB / Doris).
  - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
    explicit regression guard for the score-range correction: cos=0.5
    maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
  - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
    (1.0001) + defensive negative + ±Inf / NaN edges + keyword
    passthrough + nil-ctx safety.

Backward compatibility is preserved at every layer:

- All new struct fields are omitempty / pointer-tagged so existing
  rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
  lands. The ES regrouping changes the post-normalization value for
  every ES vector search result (a correctness fix, not a feature) —
  ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).
2026-05-22 20:43:50 +08:00
ochan.kwon fd3d9f547c feat(search): fan-out KB retrieval across bound vector stores
Multi-KB hybrid search now groups KBs by their bound VectorStore (partition
key (storeID, owner_tenant_id)), retrieves in parallel via errgroup with a
SetLimit(4) cap and a per-group timeout (MULTI_STORE_RETRIEVE_TIMEOUT_SEC,
default 30s), and merges results. When the collected results span more than
one engine type, an EngineAwareNormalizer rescales vector scores to [0, 1];
keyword (BM25) scores pass through to the existing RRF fusion. Single-group
calls take the fast path with zero fan-out overhead, preserving today's
behavior for deployments where every KB has vector_store_id = NULL.

Embedding-model consistency is now enforced explicitly via
ResolveEmbeddingModelKeys. Multi-KB searches across KBs whose resolved
model identities differ return BadRequest instead of silently producing
incomparable scores. Cross-tenant Organization-shared KBs are preserved by
partitioning on KB.TenantID so the factory's ownership lookup runs against
the source tenant. Foreign-tenant KB UUIDs injected via the request body
are rejected via kbShareService.HasTenantKBPermission (Plan 3 of #1303,
3-D capped) before any retrieval; rejected scopes surface as 404 to avoid
leaking foreign KB existence.

Service-layer typed AppErrors (ErrVectorStoreBindingInvalid 2200 /
ErrVectorStoreUnavailable 2201) are mapped from PR2 sentinel hierarchy and
preserved end-to-end: the iterative FAQ path returns them rather than
swallowing, and the HybridSearch handler routes typed AppErrors to the
client unchanged instead of downgrading to 500.

Part of #993 (Phase 2: Per-KB VectorStore Binding).
Phase 2 roadmap item: PR 4 (Multi-store fan-out search).
Depends on #994, #1310, #1372.
2026-05-20 22:25:39 +08:00
ochan.kwon 0e8de6192c feat(knowledge-base): validate vector store bindings on create, copy, and delete
Wires KnowledgeBase.VectorStoreID and the ownership-aware retrieve factory
into the user-facing knowledge-base lifecycle:

- POST /knowledge-bases validates the requested vector_store_id against
  the caller's tenant scope and the engine registry. New error codes
  ErrVectorStoreBindingInvalid (2200) and ErrVectorStoreUnavailable (2201)
  distinguish the typed branches without echoing UUIDs to the client.
- GET / POST / PUT / PUT-pin responses embed the bound store's display
  metadata (name, source, engine_type, status) without exposing any
  connection credentials. Cross-tenant shared KBs receive a suppressed
  payload (vector_store_id stripped, source="shared") so operator-chosen
  store names cannot be enumerated across tenants.
- POST /knowledge-bases/copy synchronously rejects clones whose target
  has a different embedding model or vector store, before the async
  clone task is enqueued. The async clone worker re-applies the same
  checks for defense in depth.
- DELETE /vector-stores/:id refuses to remove a store with bound KBs,
  inside a transaction that row-locks the store on PostgreSQL and
  serializes via WAL on SQLite. unregister-from-registry is wrapped in
  defer/recover so a panic surfaces as a structured warning instead of
  silently leaking a stale engine.
- vector_store_id is immutable after creation. The GORM <-:create tag
  blocks every ORM update path; the service-layer DTO omits the field
  entirely; a reflection-based regression test catches any future
  maintainer who adds it back to either layer.
- Empty-string vector_store_id is normalized to nil at both the create
  path and inside SharesStoreWith, so rows persisted by callers that
  did not run Normalize first cannot trip false same-store comparisons.

Part of #993. Depends on #994 and #1310.
2026-05-18 15:58:46 +08:00
ochan.kwon 7478e3cddb refactor(retriever): introduce factory for KB-scoped retrieve engine resolution
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across
seven services into two factory functions with tenant-ownership
verification: CreateRetrieveEngineForKB (synchronous) and
CreateRetrieveEngineFromPayload (async task handlers). Promotes
GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry.
Extends KBDeletePayload / IndexDeletePayload with an omitempty
VectorStoreID snapshot.

Design highlights
- Factory verifies tenant ownership of the resolved store
  (defense-in-depth) so a gap in PR3 validation or a tampered Asynq
  payload cannot cross tenants. Cross-tenant attempts return
  ErrVectorStoreForbidden with a structured log entry.
- Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound,
  ErrVectorStoreForbidden) let async handlers classify non-retryable
  failures as asynq.SkipRetry.
- nil and empty-string pointers for vectorStoreID normalize to
  "unbound" so callers never send an empty UUID to GetByStoreID.
- resolveBoundEngine constructs engineInfos directly (with
  slices.Clone of Support()) instead of going through
  NewCompositeRetrieveEngine. The latter only reads from the
  byEngineType map and cannot reach DB stores in byStoreID -- multiple
  stores can share the same engine type. Unbound fallbacks still reuse
  NewCompositeRetrieveEngine so the tenant effective-engines path is
  byte-for-byte unchanged.
- DeleteKnowledgeBase now reads the KB before soft-delete so the
  enqueued KBDeletePayload carries a VectorStoreID snapshot; once the
  row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT
  per KB-delete request (a rare admin operation).
- enqueueIndexDeleteTask gains a vectorStoreID *string parameter that
  the single caller (tag.go) populates from the owning KB. The worker
  (ProcessIndexDelete) validates ownership via the factory.
- cleanupKnowledgeResources loads the KB so bound-store cleanup routes
  correctly. If the load fails it warns and falls back to tenant
  effective engines; orphan vectors become observable in logs instead
  of disappearing silently.
- DeleteKnowledges (batch) stays on the tenant effective-engines path
  -- batches can span stores and multi-store fan-out is PR4 scope.
  Noted inline.

Merge safety

All knowledge bases have VectorStoreID = NULL at merge time (introduced
nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to
NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()),
which is the existing code path for these KBs. Async payloads from
before this change decode to VectorStoreID = nil (omitempty) and take
the same fallback.

Testing

- go build ./...
- go vet ./...
- go test -race -count=1 ./internal/application/service/retriever/...
  covers unbound (nil/empty), bound, cross-tenant, unregistered store,
  ownership infra error, legacy payload shapes (missing field/null/
  empty string/nil), tampered payload, and parallel invocation.
- grep -r NewCompositeRetrieveEngine internal/application/service
  returns zero results outside factory.go/composite.go.

Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994.
2026-05-14 15:53:58 +08:00
Claude 0dc32cb0c4 feat(retriever): exponential backoff + safety truncation for embeddings
batchEmbedWithBackoff replaces the previous fixed 100ms × 5 retry loop
with proper exponential backoff (200/400/800/1600/3200 ms) and respects
context cancellation between attempts. sanitizeForEmbedding caps any
single embedding input at 20k runes — well above realistic chunk
budgets, so it only triggers on pathologically large inputs and logs
a warning when it does.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
ochan.kwon c5fc05f3d0 feat: add registry dual-map and engine factory for VectorStore
- Extend RetrieveEngineRegistry with byStoreID map for DB store engines
- Add StoreRegistry interface and EngineFactory function type
- Create engine factory with per-driver SDK client construction
- Load VectorStore records from DB into registry at startup
- Wire Create/Delete to dynamically register/unregister engines
- Add comprehensive unit tests for registry and service
2026-04-17 00:15:37 +08:00
wizardchen 17604c9b94 perf: 为批量索引添加并发保存优化 2026-01-14 17:21:42 +08:00
wizardchen da1dcb0cda feat: 批量更新tag时更新索引字段 2025-12-30 19:44:53 +08:00
wizardchen b70450ddd8 feat: 支持FAQ索引类型分离,新增is_enabled字段过滤和批量更新优化 2025-12-17 22:04:18 +08:00
wizardchen a50f53d205 feat: Implement delete functionality for generated questions in knowledge base, including API and UI updates 2025-12-04 01:26:30 +08:00
wizardchen eb8bfddfd1 refactor: Update function signatures across multiple files to improve readability by adding context parameters and enhancing code structure 2025-11-28 16:17:52 +08:00
wizardchen 135e33eca4 refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
wizardchen 64c44c9e27 refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
wizardchen 057cfd9eb0 feat: Enhance FAQ import functionality and UI improvements
- Added support for tracking the status of FAQ import tasks, including progress indicators for ongoing imports.
- Introduced new localization strings for processing tasks and batch operations in multiple languages.
- Updated the FAQ entry manager to display import progress and improved user feedback during the import process.
- Enhanced the KnowledgeBase component to reflect the processing state of knowledge items, improving user experience during document uploads.
- Implemented batch deletion functionality for chunks, streamlining the management of knowledge base content.
2025-11-23 17:32:50 +08:00
wizardchen fdf5d9d781 feat: Implement knowledge tag management and batch updates for FAQ entries
- Added API functions for listing, creating, updating, and deleting knowledge base tags.
- Introduced batch update functionality for enabling/disabling FAQ entries and updating their tags.
- Enhanced the KnowledgeBase.vue and FAQEntryManager.vue components to support tag management and batch operations.
- Updated localization files to include new strings related to tag management and batch operations.
2025-11-18 11:48:37 +08:00
wizardchen 25fdaed8ca feat: Implement FAQ functionality in Knowledge Base
- Added support for creating and managing FAQ knowledge bases, including new API endpoints for listing, upserting, updating, and deleting FAQ entries.
- Enhanced the frontend to allow users to create FAQ knowledge bases and manage FAQ entries with a dedicated interface.
- Introduced new types and configurations for FAQ handling in the backend, ensuring proper indexing and retrieval of FAQ entries.
- Updated localization files to include new strings related to FAQ management in multiple languages.
- Refactored existing components to accommodate the new FAQ features, improving user experience and functionality.
2025-11-15 01:30:17 +08:00
wizardchen 14d0e663cc feat: Add web search results handling and enhance knowledge retrieval features
- Introduced a new endpoint to fetch chunks by ID, improving access to specific knowledge base entries.
- Updated tool result types to include web search results, enhancing the display capabilities in the chat component.
- Enhanced the AgentStreamDisplay component to support rendering web search results with improved HTML formatting.
- Implemented logic for managing web search result summaries and citations, ensuring a cohesive user experience.
- Refactored session management to accommodate new web search functionalities, improving overall system integration.
2025-11-11 22:26:59 +08:00
wizardchen 56eb2bce33 init commit 2025-08-05 15:08:07 +08:00