fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117)

knowledgeService.buildStorageConfig had hardcoded switches in two places that
only handled local/minio/cos. Knowledge bases configured for tos, s3, oss, or
ks3 reached the docreader with a DocParserStorageConfig containing only the
Provider tag — bucket, endpoint, and credentials were silently dropped, and
extraction stalled or fetched from the wrong location.

- Tenant-merge switch: add tos/s3/oss/ks3 cases mapping
  StorageEngineConfig.{TOS,S3,OSS,KS3} fields onto DocParserStorageConfig.
  Same shape as the existing minio/cos branches; AccessKey -> AccessKeyID,
  SecretKey -> SecretAccessKey, no AppID.
- Legacy compat switch: list tos/s3/oss/ks3 alongside local with hasKBFull=false
  so fall-through to the tenant-merge path is intentional rather than an
  unrecognised provider sliding past the switch. The legacy `cos_config` column
  predates these providers and they should never resolve via that path.
- Doc drift: types/custom_agent.go ImageStorageProvider comment and
  handler/system.go StorageEngineStatusItem.Name comment now list all 7
  providers (matched against types.StorageEngineConfig and ParseProviderScheme).

Tests (knowledge_storage_config_test.go, +268 LOC, 17 cases):
- TestBuildStorageConfig_TenantMergeAllProviders pins per-provider field
  population for all 7 providers; the tos/s3/oss/ks3 sub-tests are the
  regression guard for #1117.
- TestBuildStorageConfig_LegacyPathOnlyForCOSAndMinIO pins that legacy
  StorageConfig data is only consumed for cos/minio and skipped for the
  others (intentional fall-through).
- TestBuildStorageConfig_NoTenantFallsThroughToEmpty pins the no-tenant case.

Out of scope (called out in the linked issue):
- The systemic refactor (single source-of-truth provider registry) — left as
  a follow-up; this PR keeps changes local to the buggy code path.
- initialization.go validateStorageEngineConfig multimodal validation gap —
  the InitializationRequest struct does not carry tos/s3/oss/ks3 fields, so
  fixing it requires API surface additions; can ship separately.

Closes #1117 (the buildStorageConfig switches; remaining items deferred).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
voidborne-d
2026-05-06 16:47:59 +08:00
committed by lyingbug
parent 0d8540c93c
commit 46cab52fb0
4 changed files with 341 additions and 3 deletions
+45 -1
View File
@@ -7642,6 +7642,10 @@ func (s *knowledgeService) buildStorageConfig(ctx context.Context, kb *types.Kno
}
// Backward compatibility: if legacy cos_config has full params for the chosen provider, use them.
// Note: legacy StorageConfig predates tos/s3/oss/ks3, so those providers always
// resolve via the tenant-merge path below. Listing them here keeps the fall-through
// intentional (instead of an unrecognised provider silently sliding past the switch).
// See issue #1117: provider enum was missing tos/s3/oss in this switch.
sc := &kb.StorageConfig
hasKBFull := false
switch provider {
@@ -7649,7 +7653,7 @@ func (s *knowledgeService) buildStorageConfig(ctx context.Context, kb *types.Kno
hasKBFull = sc.SecretID != "" && sc.BucketName != ""
case "minio":
hasKBFull = sc.BucketName != ""
case "local":
case "local", "tos", "s3", "oss", "ks3":
hasKBFull = false
}
@@ -7678,6 +7682,10 @@ func (s *knowledgeService) buildStorageConfig(ctx context.Context, kb *types.Kno
provider = strings.ToLower(strings.TrimSpace(sec.DefaultProvider))
out.Provider = strings.ToUpper(provider)
}
// Provider list must match types.StorageEngineConfig + ParseProviderScheme.
// Missing a case here causes DocParserStorageConfig to be returned with only
// Provider set — bucket/endpoint/credentials are silently dropped, and the
// docreader then fails or fetches from the wrong location. See issue #1117.
switch provider {
case "local":
if sec.Local != nil {
@@ -7706,6 +7714,42 @@ func (s *knowledgeService) buildStorageConfig(ctx context.Context, kb *types.Kno
out.AppID = sec.COS.AppID
out.PathPrefix = sec.COS.PathPrefix
}
case "tos":
if sec.TOS != nil {
out.Endpoint = sec.TOS.Endpoint
out.Region = sec.TOS.Region
out.AccessKeyID = sec.TOS.AccessKey
out.SecretAccessKey = sec.TOS.SecretKey
out.BucketName = sec.TOS.BucketName
out.PathPrefix = sec.TOS.PathPrefix
}
case "s3":
if sec.S3 != nil {
out.Endpoint = sec.S3.Endpoint
out.Region = sec.S3.Region
out.AccessKeyID = sec.S3.AccessKey
out.SecretAccessKey = sec.S3.SecretKey
out.BucketName = sec.S3.BucketName
out.PathPrefix = sec.S3.PathPrefix
}
case "oss":
if sec.OSS != nil {
out.Endpoint = sec.OSS.Endpoint
out.Region = sec.OSS.Region
out.AccessKeyID = sec.OSS.AccessKey
out.SecretAccessKey = sec.OSS.SecretKey
out.BucketName = sec.OSS.BucketName
out.PathPrefix = sec.OSS.PathPrefix
}
case "ks3":
if sec.KS3 != nil {
out.Endpoint = sec.KS3.Endpoint
out.Region = sec.KS3.Region
out.AccessKeyID = sec.KS3.AccessKey
out.SecretAccessKey = sec.KS3.SecretKey
out.BucketName = sec.KS3.BucketName
out.PathPrefix = sec.KS3.PathPrefix
}
}
}