Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.
Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.
Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.
Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
This commit introduces a new test suite for the IM file service, including a stub implementation for testing purposes. It adds tests for resolving file services based on storage providers and ensures proper fallback mechanisms for MinIO URLs. Additionally, the `rewriteStorageURLs` and `cleanIMContent` functions have been refactored to utilize a resolver for improved caching and efficiency. These changes enhance the robustness of file service handling and improve test coverage for various storage scenarios.
This commit introduces unit tests for the `parseCosObjectName` method in the `cosFileService`, ensuring it correctly rejects local scheme URLs and properly parses COS scheme URLs. Additionally, the `parseCosObjectName` method has been updated to return an error for unsupported schemes, improving error handling in the `GetFile` and `DeleteFile` methods. This enhancement ensures more robust handling of file paths in the application.
Add Kingsoft Cloud KS3 as a supported storage provider, including
backend service implementation, storage config UI, and provider
scheme recognition throughout the codebase.
Reverts the tenant-ID-from-context change in da5fcd3. The storage path
encodes the resource owner's tenant, and the presigned-URL verifier
uses that ID to look up the owning tenant's StorageEngineConfig. Using
the caller's tenant ID from context would break cross-tenant shared
resources — e.g. when tenant Y reads an image from a KB shared by
tenant X, signing with Y would cause the verifier to open Y's storage
backend and 404 on X's file.
Keep:
- presignDefaultTTL shortened from 24h to 2h (independent improvement).
- godoc note on ParseTenantIDFromStoragePath flagging the ambiguity
for cloud paths with numeric bucket/region names.
- Unit tests for path-based tenant extraction and the no-external-URL
backward-compat path.
Addresses review feedback on the IM storage URL rewrite:
- localFileService.GetFileURL now reads tenant ID from request context
first, falling back to ParseTenantIDFromStoragePath only when context
is absent. Fixes ambiguity for cloud providers whose paths embed
numeric bucket/region names before the tenant segment, which could
mint presigned URLs bound to the wrong tenant ID.
- Shorten presigned URL default TTL from 24h to 2h. A leaked HMAC key
authorizes cross-tenant file reads, so URLs should expire quickly;
IM clients fetch referenced images within seconds anyway.
- Document ParseTenantIDFromStoragePath as a best-effort fallback.
- Add unit tests covering context-first, path-fallback, and the
no-external-URL backward-compat path.
IM platforms (Feishu, Slack, Telegram, DingTalk, Mattermost, WeCom) cannot
render provider:// URLs (local://, minio://, s3://, etc.) that appear in
LLM answers containing knowledge base images. The web frontend handles
these via the authenticated /files endpoint, but IM clients need publicly
resolvable HTTP URLs.
Changes:
- Add HMAC-SHA256 presigned URL utility (internal/utils/presign.go) for
generating time-limited, signature-verified file access URLs
- Add GET /api/v1/files/presigned endpoint that serves files without
session auth, verified by HMAC signature and expiry
- Update localFileService.GetFileURL() to return presigned HTTP URLs
when APP_EXTERNAL_URL is configured (cloud backends already return
presigned URLs via their SDKs)
- Add IM content rewriting pipeline: strip <image> XML tags, strip
citation tags, rewrite storage URLs to HTTP — applied at all IM
output points (streaming flush, non-streaming reply, fallback)
- Add holdback buffer in streaming flush to prevent URL/tag splitting
across chunk boundaries
Closes#1058
Add oss.go with full FileService implementation (SaveFile, SaveBytes,
GetFile, DeleteFile, GetFileURL, CheckConnectivity) using aws-sdk-go-v2
with path-style addressing for Aliyun OSS. Includes temp bucket support
and unit tests for parseOssFilePath and client creation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Added ASR configuration options to the knowledge base and model settings.
- Implemented ASR model selection and transcription capabilities in the knowledge processing pipeline.
- Enhanced file upload support to include audio formats for ASR processing.
- Updated localization files to include ASR-related labels and descriptions.
- Refactored model handling to accommodate ASR as a new model type.
The S3 storage adapter (s3.go) was added in e309e0b but the provider
scheme integration was incomplete — ParseProviderScheme() and frontend
security.ts were not updated, causing s3:// URLs to fail in the /files
proxy, VLM image processing, and frontend rendering.
Additionally, S3-compatible services (MinIO, Ceph, etc.) require
path-style addressing (endpoint/bucket/key) but the SDK client only
used virtual-hosted style. The deprecated WithEndpointResolver API
is replaced with BaseEndpoint, and path-style is auto-enabled for
non-AWS endpoints (detected by amazonaws.com in the URL).
Changes:
- types/knowledgebase.go: add "s3" to ParseProviderScheme()
- file/s3.go: auto path-style for S3-compatible endpoints + replace
deprecated WithEndpointResolver with BaseEndpoint
- frontend/security.ts: add s3 to 5 provider scheme allowlists
- knowledgebase_test.go: add ParseProviderScheme + InferStorage tests
- s3_test.go: add path-style detection + parseS3FilePath tests
- Removed the image storage handling from the `_resolve_images` function, which now only decodes images and returns them as inline bytes.
- Updated the function's docstring to clarify that image persistence is managed by the Go App, and the return value for `image_dir_path` is always empty.
- Adjusted related gRPC and protobuf files to remove the `ConvertToPDF` method and its associated types, streamlining the API.
- Enhanced security measures in the frontend by implementing a placeholder for provider images and ensuring proper hydration of protected file images.
These changes improve the clarity and efficiency of image processing and API interactions within the application.
- Introduced a loading state (`loadingChunks`) to manage document chunk loading more effectively.
- Updated the scroll handling logic to prevent multiple requests while loading chunks.
- Adjusted the pagination logic to ensure correct page increments based on the total document count.
This update improves the user experience by ensuring smoother document loading and better state management during scrolling interactions.
- Updated the `KnowledgeBase` struct to rename `cos_config` to `storage_config` for consistency across the application.
- Implemented a custom `UnmarshalJSON` method to maintain backward compatibility with legacy responses that still use `cos_config`.
- Modified various frontend components to reflect the new `storage_config` naming, ensuring seamless integration with the updated backend structure.
- Enhanced Dockerfiles and service configurations to support the new storage engine setup.
This update improves the clarity of storage configurations and ensures compatibility with existing data structures.
- Updated `.air.toml` to include additional CGO flags for improved build settings.
- Expanded `StorageEngineConfig` interface to support "tos" (火山引擎 TOS) as a new storage provider.
- Modified related components and views to accommodate the new storage engine, including updates to `ListSpaceSidebar`, `AgentList`, `KnowledgeBaseList`, and `OrganizationList`.
- Improved UI elements for batch management in the menu and sidebar components.
- Added internationalization support for new memory features and storage engine descriptions across multiple languages.
This update enhances the flexibility of storage options and improves user experience with better UI interactions.
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.
This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
- Added `SafePathUnderBase` to prevent path traversal by ensuring file paths remain within a specified base directory.
- Introduced `SafeFileName` to validate and sanitize file names, disallowing path traversal and empty names.
- Implemented `SafeObjectKey` to validate object keys for storage, ensuring they do not contain path traversal sequences.
- Updated file handling methods in `cos.go`, `local.go`, and `minio.go` to utilize these new validation utilities, enhancing security against invalid file paths and names.
This update improves the robustness of file operations by enforcing strict validation rules, thereby mitigating potential security risks.