Commit Graph

30 Commits

Author SHA1 Message Date
ochan.kwon e9980c6011 fix: deep-copy stored files and images when cloning a knowledge base
Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.

Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.

Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.

Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
2026-06-03 14:45:59 +08:00
wizardchen e0823eff31 feat: enhance IM file service tests and refactor storage URL handling
This commit introduces a new test suite for the IM file service, including a stub implementation for testing purposes. It adds tests for resolving file services based on storage providers and ensures proper fallback mechanisms for MinIO URLs. Additionally, the `rewriteStorageURLs` and `cleanIMContent` functions have been refactored to utilize a resolver for improved caching and efficiency. These changes enhance the robustness of file service handling and improve test coverage for various storage scenarios.
2026-05-29 13:47:32 +08:00
wizardchen 5d02404fdd feat: add unit tests for COS object name parsing and enhance error handling
This commit introduces unit tests for the `parseCosObjectName` method in the `cosFileService`, ensuring it correctly rejects local scheme URLs and properly parses COS scheme URLs. Additionally, the `parseCosObjectName` method has been updated to return an error for unsupported schemes, improving error handling in the `GetFile` and `DeleteFile` methods. This enhancement ensures more robust handling of file paths in the application.
2026-05-29 13:47:32 +08:00
Li Xianggang b3dbf86bf5 feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
ChenRussell 42348eb40d feat(ks3): add KS3 object storage support
Add Kingsoft Cloud KS3 as a supported storage provider, including
  backend service implementation, storage config UI, and provider
  scheme recognition throughout the codebase.
2026-05-05 00:22:27 +08:00
wizardchen 097c9d0ad5 fix(im): revert tenant-from-context for presigned URL
Reverts the tenant-ID-from-context change in da5fcd3. The storage path
encodes the resource owner's tenant, and the presigned-URL verifier
uses that ID to look up the owning tenant's StorageEngineConfig. Using
the caller's tenant ID from context would break cross-tenant shared
resources — e.g. when tenant Y reads an image from a KB shared by
tenant X, signing with Y would cause the verifier to open Y's storage
backend and 404 on X's file.

Keep:
- presignDefaultTTL shortened from 24h to 2h (independent improvement).
- godoc note on ParseTenantIDFromStoragePath flagging the ambiguity
  for cloud paths with numeric bucket/region names.
- Unit tests for path-based tenant extraction and the no-external-URL
  backward-compat path.
2026-04-30 11:40:30 +08:00
wizardchen f27a1e083c fix(im): prefer tenant ID from context, shorten presigned URL TTL
Addresses review feedback on the IM storage URL rewrite:

- localFileService.GetFileURL now reads tenant ID from request context
  first, falling back to ParseTenantIDFromStoragePath only when context
  is absent. Fixes ambiguity for cloud providers whose paths embed
  numeric bucket/region names before the tenant segment, which could
  mint presigned URLs bound to the wrong tenant ID.
- Shorten presigned URL default TTL from 24h to 2h. A leaked HMAC key
  authorizes cross-tenant file reads, so URLs should expire quickly;
  IM clients fetch referenced images within seconds anyway.
- Document ParseTenantIDFromStoragePath as a best-effort fallback.
- Add unit tests covering context-first, path-fallback, and the
  no-external-URL backward-compat path.
2026-04-30 11:40:30 +08:00
wizardchen 7fd566bc15 fix(im): rewrite private storage URLs to HTTP in IM channel replies
IM platforms (Feishu, Slack, Telegram, DingTalk, Mattermost, WeCom) cannot
render provider:// URLs (local://, minio://, s3://, etc.) that appear in
LLM answers containing knowledge base images. The web frontend handles
these via the authenticated /files endpoint, but IM clients need publicly
resolvable HTTP URLs.

Changes:
- Add HMAC-SHA256 presigned URL utility (internal/utils/presign.go) for
  generating time-limited, signature-verified file access URLs
- Add GET /api/v1/files/presigned endpoint that serves files without
  session auth, verified by HMAC signature and expiry
- Update localFileService.GetFileURL() to return presigned HTTP URLs
  when APP_EXTERNAL_URL is configured (cloud backends already return
  presigned URLs via their SDKs)
- Add IM content rewriting pipeline: strip <image> XML tags, strip
  citation tags, rewrite storage URLs to HTTP — applied at all IM
  output points (streaming flush, non-streaming reply, fallback)
- Add holdback buffer in streaming flush to prevent URL/tag splitting
  across chunk boundaries

Closes #1058
2026-04-30 11:40:30 +08:00
jw_mac 09e1642335 fix(storage): auto-detect content-type to resolve image rendering issues
Fixes #995. Added mime type detection in SaveBytes for MinIO, OSS, S3, and TOS to prevent images from being incorrectly saved as text/csv.
2026-04-20 21:59:54 +08:00
bingxiang.cheng a2900d5540 chore(deps): 更新并新增go.mod依赖项
- 新增aliyun alibabacloud-oss-go-sdk-v2依赖
- 删除部分冗余依赖,调整依赖列表结构
- 添加jackc/pgx/v5作为直接依赖
- 统一和规范间接依赖声明顺序
- 移除go.opentelemetry.io/contrib的版本替换声明
2026-04-14 17:33:12 +08:00
bingxiang.cheng c21d1edd80 fix(oss): 更新OSS S3兼容模式的注释,修复路径样式描述并禁用自动校验和计算 2026-04-13 15:37:33 +08:00
bingxiang.cheng 12001f23bf fix(oss): 禁用OSS S3兼容API的自动校验和计算,修复签名不匹配问题并调整路径样式设置 2026-04-13 15:37:33 +08:00
bingxiang.cheng 00a09d6c56 fix(oss): 禁用OSS S3兼容API的自动校验和计算,以防止签名不匹配 2026-04-13 15:37:33 +08:00
bingxiang.cheng 3bd56c17a8 feat(oss): register OSS in factory and container initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:34:28 +08:00
bingxiang.cheng a584df052d feat(oss): implement OSS FileService using S3-compatible mode
Add oss.go with full FileService implementation (SaveFile, SaveBytes,
GetFile, DeleteFile, GetFileURL, CheckConnectivity) using aws-sdk-go-v2
with path-style addressing for Aliyun OSS. Includes temp bucket support
and unit tests for parseOssFilePath and client creation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:34:28 +08:00
wizardchen 624e24ba49 feat(asr): integrate Automatic Speech Recognition (ASR) support
- Added ASR configuration options to the knowledge base and model settings.
- Implemented ASR model selection and transcription capabilities in the knowledge processing pipeline.
- Enhanced file upload support to include audio formats for ASR processing.
- Updated localization files to include ASR-related labels and descriptions.
- Refactored model handling to accommodate ASR as a new model type.
2026-04-02 21:27:27 +08:00
ochan.kwon 5c0cda3630 fix(storage): add S3 provider scheme support to frontend and backend allowlists
The S3 storage adapter (s3.go) was added in e309e0b but the provider
scheme integration was incomplete — ParseProviderScheme() and frontend
security.ts were not updated, causing s3:// URLs to fail in the /files
proxy, VLM image processing, and frontend rendering.

Additionally, S3-compatible services (MinIO, Ceph, etc.) require
path-style addressing (endpoint/bucket/key) but the SDK client only
used virtual-hosted style. The deprecated WithEndpointResolver API
is replaced with BaseEndpoint, and path-style is auto-enabled for
non-AWS endpoints (detected by amazonaws.com in the URL).

Changes:
- types/knowledgebase.go: add "s3" to ParseProviderScheme()
- file/s3.go: auto path-style for S3-compatible endpoints + replace
  deprecated WithEndpointResolver with BaseEndpoint
- frontend/security.ts: add s3 to 5 provider scheme allowlists
- knowledgebase_test.go: add ParseProviderScheme + InferStorage tests
- s3_test.go: add path-style detection + parseS3FilePath tests
2026-03-30 16:57:27 +08:00
DaWesen e309e0bed8 feat(storage): 集成S3存储适配器
添加对AWS S3及兼容存储服务的支持:
- 实现完整的S3FileService接口
- 支持文件上传、下载、删除功能
- 添加配置支持和环境变量检查
- 实现连接测试功能
- 遵循与其他存储适配器相同的代码风格
2026-03-09 10:39:46 +08:00
wizardchen 4192cfd072 refactor: simplify image handling in _resolve_images function
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.

These changes improve the clarity and efficiency of image processing and API interactions within the application.
2026-03-03 22:23:23 +08:00
wizardchen 4a9977b11f feat: enhance document loading and state management in doc-content component
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.

This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
2026-03-02 21:21:49 +08:00
wizardchen 393ebf9f92 feat: standardize storage configuration and enhance backward compatibility
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.

This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
2026-03-02 21:21:49 +08:00
wizardchen 469f320d10 feat: enhance storage engine configuration and UI updates
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.

This update enhances the flexibility of storage options and improves user experience with better UI interactions.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
wizardchen 5241dbc39e feat(security): implement path and filename validation utilities
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.

This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.
2026-03-02 21:21:49 +08:00
Dounx 6adecdb30e feat: add volcengine tos support 2026-02-25 14:24:50 +08:00
wizardchen 100484f13d feat: 新增FAQ批量导入dry run功能 2026-01-14 17:08:49 +08:00
wizardchen 135e33eca4 refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
wizardchen 64c44c9e27 refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
begoniezhao c2d52a9374 feat: Modify COS file service initialization parameters and URL processing logic 2025-09-16 10:15:50 +08:00
wizardchen 56eb2bce33 init commit 2025-08-05 15:08:07 +08:00