Commit Graph

15 Commits

Author SHA1 Message Date
ochan.kwon 7478e3cddb refactor(retriever): introduce factory for KB-scoped retrieve engine resolution
Extracts the 25 repeated NewCompositeRetrieveEngine call sites across
seven services into two factory functions with tenant-ownership
verification: CreateRetrieveEngineForKB (synchronous) and
CreateRetrieveEngineFromPayload (async task handlers). Promotes
GetByStoreID from concrete-only to interfaces.RetrieveEngineRegistry.
Extends KBDeletePayload / IndexDeletePayload with an omitempty
VectorStoreID snapshot.

Design highlights
- Factory verifies tenant ownership of the resolved store
  (defense-in-depth) so a gap in PR3 validation or a tampered Asynq
  payload cannot cross tenants. Cross-tenant attempts return
  ErrVectorStoreForbidden with a structured log entry.
- Sentinel errors (ErrTenantInfoMissing, ErrVectorStoreNotFound,
  ErrVectorStoreForbidden) let async handlers classify non-retryable
  failures as asynq.SkipRetry.
- nil and empty-string pointers for vectorStoreID normalize to
  "unbound" so callers never send an empty UUID to GetByStoreID.
- resolveBoundEngine constructs engineInfos directly (with
  slices.Clone of Support()) instead of going through
  NewCompositeRetrieveEngine. The latter only reads from the
  byEngineType map and cannot reach DB stores in byStoreID -- multiple
  stores can share the same engine type. Unbound fallbacks still reuse
  NewCompositeRetrieveEngine so the tenant effective-engines path is
  byte-for-byte unchanged.
- DeleteKnowledgeBase now reads the KB before soft-delete so the
  enqueued KBDeletePayload carries a VectorStoreID snapshot; once the
  row is soft-deleted GORM's default scope hides it. Cost: +1 SELECT
  per KB-delete request (a rare admin operation).
- enqueueIndexDeleteTask gains a vectorStoreID *string parameter that
  the single caller (tag.go) populates from the owning KB. The worker
  (ProcessIndexDelete) validates ownership via the factory.
- cleanupKnowledgeResources loads the KB so bound-store cleanup routes
  correctly. If the load fails it warns and falls back to tenant
  effective engines; orphan vectors become observable in logs instead
  of disappearing silently.
- DeleteKnowledges (batch) stays on the tenant effective-engines path
  -- batches can span stores and multi-store fan-out is PR4 scope.
  Noted inline.

Merge safety

All knowledge bases have VectorStoreID = NULL at merge time (introduced
nullable in PR1, Tencent/WeKnora#994). Every factory call falls back to
NewCompositeRetrieveEngine(registry, tenantInfo.GetEffectiveEngines()),
which is the existing code path for these KBs. Async payloads from
before this change decode to VectorStoreID = nil (omitempty) and take
the same fallback.

Testing

- go build ./...
- go vet ./...
- go test -race -count=1 ./internal/application/service/retriever/...
  covers unbound (nil/empty), bound, cross-tenant, unregistered store,
  ownership infra error, legacy payload shapes (missing field/null/
  empty string/nil), tampered payload, and parallel invocation.
- grep -r NewCompositeRetrieveEngine internal/application/service
  returns zero results outside factory.go/composite.go.

Refs: Part of Tencent/WeKnora#993. Depends on Tencent/WeKnora#994.
2026-05-14 15:53:58 +08:00
Claude 0dc32cb0c4 feat(retriever): exponential backoff + safety truncation for embeddings
batchEmbedWithBackoff replaces the previous fixed 100ms × 5 retry loop
with proper exponential backoff (200/400/800/1600/3200 ms) and respects
context cancellation between attempts. sanitizeForEmbedding caps any
single embedding input at 20k runes — well above realistic chunk
budgets, so it only triggers on pathologically large inputs and logs
a warning when it does.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
ochan.kwon c5fc05f3d0 feat: add registry dual-map and engine factory for VectorStore
- Extend RetrieveEngineRegistry with byStoreID map for DB store engines
- Add StoreRegistry interface and EngineFactory function type
- Create engine factory with per-driver SDK client construction
- Load VectorStore records from DB into registry at startup
- Wire Create/Delete to dynamically register/unregister engines
- Add comprehensive unit tests for registry and service
2026-04-17 00:15:37 +08:00
wizardchen 17604c9b94 perf: 为批量索引添加并发保存优化 2026-01-14 17:21:42 +08:00
wizardchen da1dcb0cda feat: 批量更新tag时更新索引字段 2025-12-30 19:44:53 +08:00
wizardchen b70450ddd8 feat: 支持FAQ索引类型分离,新增is_enabled字段过滤和批量更新优化 2025-12-17 22:04:18 +08:00
wizardchen a50f53d205 feat: Implement delete functionality for generated questions in knowledge base, including API and UI updates 2025-12-04 01:26:30 +08:00
wizardchen eb8bfddfd1 refactor: Update function signatures across multiple files to improve readability by adding context parameters and enhancing code structure 2025-11-28 16:17:52 +08:00
wizardchen 135e33eca4 refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
wizardchen 64c44c9e27 refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
wizardchen 057cfd9eb0 feat: Enhance FAQ import functionality and UI improvements
- Added support for tracking the status of FAQ import tasks, including progress indicators for ongoing imports.
- Introduced new localization strings for processing tasks and batch operations in multiple languages.
- Updated the FAQ entry manager to display import progress and improved user feedback during the import process.
- Enhanced the KnowledgeBase component to reflect the processing state of knowledge items, improving user experience during document uploads.
- Implemented batch deletion functionality for chunks, streamlining the management of knowledge base content.
2025-11-23 17:32:50 +08:00
wizardchen fdf5d9d781 feat: Implement knowledge tag management and batch updates for FAQ entries
- Added API functions for listing, creating, updating, and deleting knowledge base tags.
- Introduced batch update functionality for enabling/disabling FAQ entries and updating their tags.
- Enhanced the KnowledgeBase.vue and FAQEntryManager.vue components to support tag management and batch operations.
- Updated localization files to include new strings related to tag management and batch operations.
2025-11-18 11:48:37 +08:00
wizardchen 25fdaed8ca feat: Implement FAQ functionality in Knowledge Base
- Added support for creating and managing FAQ knowledge bases, including new API endpoints for listing, upserting, updating, and deleting FAQ entries.
- Enhanced the frontend to allow users to create FAQ knowledge bases and manage FAQ entries with a dedicated interface.
- Introduced new types and configurations for FAQ handling in the backend, ensuring proper indexing and retrieval of FAQ entries.
- Updated localization files to include new strings related to FAQ management in multiple languages.
- Refactored existing components to accommodate the new FAQ features, improving user experience and functionality.
2025-11-15 01:30:17 +08:00
wizardchen 14d0e663cc feat: Add web search results handling and enhance knowledge retrieval features
- Introduced a new endpoint to fetch chunks by ID, improving access to specific knowledge base entries.
- Updated tool result types to include web search results, enhancing the display capabilities in the chat component.
- Enhanced the AgentStreamDisplay component to support rendering web search results with improved HTML formatting.
- Implemented logic for managing web search result summaries and citations, ensuring a cohesive user experience.
- Refactored session management to accommodate new web search functionalities, improving overall system integration.
2025-11-11 22:26:59 +08:00
wizardchen 56eb2bce33 init commit 2025-08-05 15:08:07 +08:00