Files
Enterprise-Knowledge-Base/internal/handler/web_search_provider.go
T
ochan.kwon d869cc1ff6 fix: redact sensitive fields and prevent credential loss on edit
Closes #988

Unify the "write-only secrets" pattern across MCP services, Models,
WebSearch providers, and DataSources. Fix two related defects:

1. GET responses leaked plaintext credentials — GetMCPServiceByID
   while List masked, all Model / WebSearchProvider / DataSource GET
   responses unconditionally.
2. When the frontend echoed a masked placeholder back on save, the
   update path overwrote the stored secret with the mask — silently
   wiping the real credential. MCP surfaced this most visibly:
   ListMCPServices masked Token to "first4****last4" and
   McpServiceDialog round-tripped that masked string back into the DB.

- internal/types/secret.go introduces the shared sentinel
  RedactedSecretPlaceholder = "***" and two helpers:
  - IsRedactedOrEmpty(s): true for "" or the sentinel
  - PreserveIfRedacted(incoming, existing): merge predicate
- Each resource adds a RedactSensitiveData (mutates-in-place) or
  Redacted (returns-a-copy) method and a MergeUpdate contract:
  empty or placeholder preserves, any other value replaces, and a
  write-only Clear* bool flag (gorm:"-", json omitempty) wipes with
  precedence over preserve / replace. Clear* never persists.
- Every GET / List / Create / Update response path now redacts.
  VectorStore's pre-existing "***" literal is refactored onto the
  shared constant for consistency.

All four dialogs — McpServiceDialog, ModelEditorDialog,
WebSearchSettings, DataSourceEditorDialog — never pre-fill secret
inputs. The stored-state signal is collapsed into the input
placeholder: "•••••••• (Enter new value to replace)" when the server
reports a stored value, falling back to the connector-specific
placeholder otherwise. The input itself carries the "something is
there" cue, matching how password dots render.

An opt-in "Remove this credential" checkbox appears in edit mode when
a value is stored; checking it disables the input and triggers a
danger-themed confirmation dialog before submit. DataSource rolls the
clear up to a connector-level "Remove all credentials" flag since its
credentials are per-connector atomic bundles (e.g. Feishu app_id +
app_secret).

i18n updates across en-US, zh-CN, ko-KR, ru-RU:
secret.storedPlaceholder / clearHint / confirmClearTitle /
confirmClearBody.

- MCP update handler's manual map-to-struct projection was silently
  dropping clear_api_key / clear_token / custom_headers on partial
  updates. Pull them off the map explicitly.
- MCP configChanged detection now snapshots URL / StdioConfig /
  TransportType before the merge so connection recycling still
  triggers on real transport changes (not just whenever AuthConfig
  is present).
- MCP CustomHeaders: nil preserves existing, non-nil replaces. Was
  always overwriting, silently wiping headers on any auth-bearing
  update.
- GetMCPServiceByID now redacts (was plaintext while List masked).
- WebSearchProvider update preserves Name / Description when empty
  instead of clobbering them.
- WebSearchProvider TestProviderRaw rejects the "***" placeholder up
  front with an actionable message rather than forwarding it to the
  upstream provider API as a literal key.
- Model builtin hideSensitiveInfo now preserves non-secret metadata
  (SupportsVision, Provider, ExtraConfig, InterfaceType,
  ParameterSize, EmbeddingParameters) that were previously zeroed
  out for builtin model cards.
- Replace local equalStringSlices / equalStringMaps helpers with
  stdlib slices.Equal / maps.Equal (Go 1.21+).

- internal/types/{secret,mcp,model,web_search_provider,datasource}_test.go:
  table-driven coverage of merge, redact, clear, and write-only flag
  invariants for every resource.
- internal/application/service/mcp_service_test.go: full service layer
  with a fake repo; covers ListMCPServices / GetMCPServiceByID
  redaction, UpdateMCPService partial-update honoring of Clear*,
  CustomHeaders preserve-on-nil / replace-on-non-nil, and
  clear-flag precedence over submitted values.

Additive only. No changes to request or response JSON shape beyond
the write-only Clear* flags (omitted by default via omitempty). No DB
schema changes. No SDK struct field removals. Existing clients that
don't know about the new flags continue to function unchanged.

- DataSource.Config jsonb at-rest encryption. Pre-existing residual
  risk; tracked as a follow-up.
- Renaming VectorStore.MaskSensitiveFields to match the new
  RedactSensitiveData / Redacted naming convention. Left alone to
  minimize diff; a follow-up rename PR can unify.
2026-05-17 11:37:53 +08:00

432 lines
15 KiB
Go

package handler
import (
"context"
"fmt"
"net/http"
"github.com/Tencent/WeKnora/internal/errors"
infra_web_search "github.com/Tencent/WeKnora/internal/infrastructure/web_search"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
secutils "github.com/Tencent/WeKnora/internal/utils"
"github.com/gin-gonic/gin"
)
// WebSearchProviderHandler handles HTTP requests for web search provider CRUD
type WebSearchProviderHandler struct {
repo interfaces.WebSearchProviderRepository
service interfaces.WebSearchProviderService
registry *infra_web_search.Registry
}
// NewWebSearchProviderHandler creates a new handler
func NewWebSearchProviderHandler(
repo interfaces.WebSearchProviderRepository,
service interfaces.WebSearchProviderService,
registry *infra_web_search.Registry,
) *WebSearchProviderHandler {
return &WebSearchProviderHandler{repo: repo, service: service, registry: registry}
}
// --- request DTOs ---
// CreateProviderRequest defines the request body for creating a provider
type CreateProviderRequest struct {
Name string `json:"name" binding:"required"`
Provider types.WebSearchProviderType `json:"provider" binding:"required"`
Description string `json:"description"`
Parameters types.WebSearchProviderParameters `json:"parameters"`
IsDefault bool `json:"is_default"`
}
// UpdateProviderRequest defines the request body for updating a provider
type UpdateProviderRequest struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters types.WebSearchProviderParameters `json:"parameters"`
IsDefault bool `json:"is_default"`
}
// --- helpers ---
// getTenantID extracts tenant ID from gin context (set by auth middleware).
func (h *WebSearchProviderHandler) getTenantID(c *gin.Context) uint64 {
return c.GetUint64(types.TenantIDContextKey.String())
}
// getOwnedProvider loads a provider and verifies it belongs to the given tenant.
// Returns (nil, status, msg) on failure so callers can respond immediately.
func (h *WebSearchProviderHandler) getOwnedProvider(
ctx context.Context, tenantID uint64, id string,
) (*types.WebSearchProviderEntity, int, string) {
provider, err := h.repo.GetByID(ctx, tenantID, id)
if err != nil {
return nil, http.StatusInternalServerError, "failed to query provider"
}
if provider == nil {
return nil, http.StatusNotFound, "web search provider not found"
}
return provider, http.StatusOK, ""
}
// --- endpoints ---
// CreateProvider creates a new web search provider
func (h *WebSearchProviderHandler) CreateProvider(c *gin.Context) {
ctx := c.Request.Context()
tenantID := h.getTenantID(c)
if tenantID == 0 {
c.JSON(http.StatusUnauthorized, gin.H{"success": false, "error": "unauthorized: tenant context missing"})
return
}
var req CreateProviderRequest
if err := c.ShouldBindJSON(&req); err != nil {
logger.Warnf(ctx, "Invalid create provider request: %v", err)
c.Error(errors.NewBadRequestError(err.Error()))
return
}
logger.Infof(ctx, "Creating web search provider: tenant=%d, name=%s, type=%s",
tenantID, secutils.SanitizeForLog(req.Name), secutils.SanitizeForLog(string(req.Provider)))
provider := &types.WebSearchProviderEntity{
TenantID: tenantID,
Name: secutils.SanitizeForLog(req.Name),
Provider: req.Provider,
Description: secutils.SanitizeForLog(req.Description),
Parameters: req.Parameters,
IsDefault: req.IsDefault,
}
if err := h.service.CreateProvider(ctx, provider); err != nil {
logger.Warnf(ctx, "Failed to create web search provider: %v", err)
c.Error(errors.NewInternalServerError(err.Error()))
return
}
provider.RedactSensitiveData()
c.JSON(http.StatusCreated, gin.H{
"success": true,
"data": provider,
})
}
// ListProviders lists all web search providers for the current tenant
func (h *WebSearchProviderHandler) ListProviders(c *gin.Context) {
ctx := c.Request.Context()
tenantID := h.getTenantID(c)
if tenantID == 0 {
c.JSON(http.StatusUnauthorized, gin.H{"success": false, "error": "unauthorized: tenant context missing"})
return
}
providers, err := h.repo.List(ctx, tenantID)
if err != nil {
logger.Warnf(ctx, "Failed to list web search providers: %v", err)
c.Error(errors.NewInternalServerError(err.Error()))
return
}
for _, p := range providers {
p.RedactSensitiveData()
}
c.JSON(http.StatusOK, gin.H{
"success": true,
"data": providers,
})
}
// GetProvider retrieves a single web search provider by ID.
//
// GetProvider godoc
// @Summary 获取网络搜索 Provider 详情
// @Description 根据 ID 获取指定 provider 配置
// @Tags 网络搜索
// @Produce json
// @Param id path string true "Provider ID"
// @Success 200 {object} types.WebSearchProviderEntity "Provider 详情"
// @Failure 404 {object} map[string]interface{} "Provider 不存在"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /web-search-providers/{id} [get]
func (h *WebSearchProviderHandler) GetProvider(c *gin.Context) {
ctx := c.Request.Context()
tenantID := h.getTenantID(c)
if tenantID == 0 {
c.JSON(http.StatusUnauthorized, gin.H{"success": false, "error": "unauthorized: tenant context missing"})
return
}
id := c.Param("id")
provider, status, msg := h.getOwnedProvider(ctx, tenantID, id)
if status != http.StatusOK {
c.JSON(status, gin.H{"success": false, "error": msg})
return
}
provider.RedactSensitiveData()
c.JSON(http.StatusOK, gin.H{
"success": true,
"data": provider,
})
}
// UpdateProvider updates a web search provider.
//
// UpdateProvider godoc
// @Summary 更新网络搜索 Provider
// @Description 更新指定 provider 的名称/描述/参数/是否默认
// @Tags 网络搜索
// @Accept json
// @Produce json
// @Param id path string true "Provider ID"
// @Param request body handler.UpdateProviderRequest true "更新字段"
// @Success 200 {object} types.WebSearchProviderEntity "更新后的 Provider"
// @Failure 400 {object} map[string]interface{} "请求参数错误"
// @Failure 404 {object} map[string]interface{} "Provider 不存在"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /web-search-providers/{id} [put]
func (h *WebSearchProviderHandler) UpdateProvider(c *gin.Context) {
ctx := c.Request.Context()
tenantID := h.getTenantID(c)
if tenantID == 0 {
c.JSON(http.StatusUnauthorized, gin.H{"success": false, "error": "unauthorized: tenant context missing"})
return
}
id := c.Param("id")
// Ownership check
existing, status, msg := h.getOwnedProvider(ctx, tenantID, id)
if status != http.StatusOK {
c.JSON(status, gin.H{"success": false, "error": msg})
return
}
var req UpdateProviderRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.Error(errors.NewBadRequestError(err.Error()))
return
}
// Merge incoming parameters into the existing ones using write-only
// secret semantics: empty / redacted APIKey preserves the stored value,
// ClearAPIKey explicitly wipes it, any other value replaces. Non-secret
// fields flow through from the request directly.
mergedParams := req.Parameters.MergeUpdate(existing.Parameters)
if req.Parameters.ClearAPIKey {
logger.Infof(ctx, "WebSearchProvider API key cleared by user: id=%s",
secutils.SanitizeForLog(id))
}
// Preserve existing values for top-level metadata fields when the
// request omits them (empty string from the JSON decoder). Without this,
// a partial update that only flips IsDefault would clobber Name and
// Description on the stored record.
mergedName := req.Name
if mergedName == "" {
mergedName = existing.Name
}
mergedDescription := req.Description
if mergedDescription == "" {
mergedDescription = existing.Description
}
// Build updated entity, keeping immutable fields from existing
provider := &types.WebSearchProviderEntity{
ID: id,
TenantID: tenantID,
Name: secutils.SanitizeForLog(mergedName),
Provider: existing.Provider, // Provider type is immutable after creation
Description: secutils.SanitizeForLog(mergedDescription),
Parameters: mergedParams,
IsDefault: req.IsDefault,
}
if err := h.service.UpdateProvider(ctx, provider); err != nil {
logger.Warnf(ctx, "Failed to update web search provider %s: %v", id, err)
c.Error(errors.NewInternalServerError(err.Error()))
return
}
// Re-fetch to get the full stored state
updated, _ := h.repo.GetByID(ctx, tenantID, id)
if updated != nil {
updated.RedactSensitiveData()
c.JSON(http.StatusOK, gin.H{"success": true, "data": updated})
} else {
c.JSON(http.StatusOK, gin.H{"success": true})
}
}
// DeleteProvider deletes a web search provider.
//
// DeleteProvider godoc
// @Summary 删除网络搜索 Provider
// @Description 删除指定 provider 配置
// @Tags 网络搜索
// @Produce json
// @Param id path string true "Provider ID"
// @Success 200 {object} map[string]interface{} "success: true"
// @Failure 404 {object} map[string]interface{} "Provider 不存在"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /web-search-providers/{id} [delete]
func (h *WebSearchProviderHandler) DeleteProvider(c *gin.Context) {
ctx := c.Request.Context()
tenantID := h.getTenantID(c)
if tenantID == 0 {
c.JSON(http.StatusUnauthorized, gin.H{"success": false, "error": "unauthorized: tenant context missing"})
return
}
id := c.Param("id")
// Ownership check
if _, status, msg := h.getOwnedProvider(ctx, tenantID, id); status != http.StatusOK {
c.JSON(status, gin.H{"success": false, "error": msg})
return
}
if err := h.service.DeleteProvider(ctx, tenantID, id); err != nil {
logger.Warnf(ctx, "Failed to delete web search provider %s: %v", id, err)
c.Error(errors.NewInternalServerError(err.Error()))
return
}
c.JSON(http.StatusOK, gin.H{"success": true})
}
// ListProviderTypes returns available provider types and their parameter requirements.
//
// ListProviderTypes godoc
// @Summary 获取网络搜索 Provider 类型元数据
// @Description 返回 UI 表单需要的 provider 类型及参数定义
// @Tags 网络搜索
// @Produce json
// @Success 200 {object} map[string]interface{} "provider 类型列表"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /web-search-providers/types [get]
func (h *WebSearchProviderHandler) ListProviderTypes(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{
"success": true,
"data": types.GetWebSearchProviderTypes(),
})
}
// TestProviderByID tests an existing saved provider by performing a sample search.
//
// TestProviderByID godoc
// @Summary 测试已保存的 Provider
// @Description 使用数据库中已保存的凭证测试连通性
// @Tags 网络搜索
// @Produce json
// @Param id path string true "Provider ID"
// @Success 200 {object} map[string]interface{} "测试结果"
// @Failure 404 {object} map[string]interface{} "Provider 不存在"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /web-search-providers/{id}/test [post]
func (h *WebSearchProviderHandler) TestProviderByID(c *gin.Context) {
ctx := c.Request.Context()
tenantID := h.getTenantID(c)
if tenantID == 0 {
c.JSON(http.StatusUnauthorized, gin.H{"success": false, "error": "unauthorized: tenant context missing"})
return
}
id := c.Param("id")
provider, status, msg := h.getOwnedProvider(ctx, tenantID, id)
if status != http.StatusOK {
c.JSON(status, gin.H{"success": false, "error": msg})
return
}
if err := h.doTestSearch(ctx, string(provider.Provider), provider.Parameters); err != nil {
logger.Warnf(ctx, "Web search provider test failed: %v", err)
c.JSON(http.StatusOK, gin.H{"success": false, "error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"success": true})
}
// TestProviderRequest defines the body for testing raw credentials
type TestProviderRequest struct {
Provider string `json:"provider" binding:"required"`
Parameters types.WebSearchProviderParameters `json:"parameters"`
}
// TestProviderRaw tests a provider with raw credentials (no persistence).
//
// TestProviderRaw godoc
// @Summary 使用原始凭证测试 Provider(不落库)
// @Description 使用前端表单中尚未保存的凭证测试连通性,用于"测试连接"按钮
// @Tags 网络搜索
// @Accept json
// @Produce json
// @Param request body handler.TestProviderRequest true "{provider, parameters}"
// @Success 200 {object} map[string]interface{} "测试结果"
// @Failure 400 {object} map[string]interface{} "请求参数错误"
// @Security Bearer
// @Security ApiKeyAuth
// @Router /web-search-providers/test [post]
func (h *WebSearchProviderHandler) TestProviderRaw(c *gin.Context) {
ctx := c.Request.Context()
var req TestProviderRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.Error(errors.NewBadRequestError(err.Error()))
return
}
if err := h.doTestSearch(ctx, req.Provider, req.Parameters); err != nil {
logger.Warnf(ctx, "Web search provider test failed: %v", err)
c.JSON(http.StatusOK, gin.H{"success": false, "error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"success": true})
}
// doTestSearch creates a temporary provider and runs a simple test query.
//
// The provider would otherwise try to authenticate against the upstream API
// with the redacted placeholder (which is guaranteed to fail with a
// confusing error). Reject it up front with an actionable message so the
// user knows they should type a real key or test against the saved config
// via /test instead.
func (h *WebSearchProviderHandler) doTestSearch(ctx context.Context, providerType string, params types.WebSearchProviderParameters) error {
if params.APIKey == types.RedactedSecretPlaceholder {
return fmt.Errorf("api_key looks like the redacted placeholder; enter your real key or call POST /api/v1/web-search-providers/{id}/test to exercise the saved credential")
}
logger.Infof(ctx, "[WebSearch][Test] testing provider type=%s", providerType)
searchProvider, err := h.registry.CreateProvider(providerType, params)
if err != nil {
logger.Warnf(ctx, "[WebSearch][Test] failed to create provider: %v", err)
return fmt.Errorf("failed to create provider: %w", err)
}
results, err := searchProvider.Search(ctx, "test", 1, false)
if err != nil {
logger.Warnf(ctx, "[WebSearch][Test] search failed: %v", err)
return err
}
if len(results) == 0 {
logger.Warnf(ctx, "[WebSearch][Test] search returned 0 results — API key or configuration may be invalid")
return fmt.Errorf("search returned 0 results, please verify your API key and configuration")
}
logger.Infof(ctx, "[WebSearch][Test] succeeded: type=%s, results=%d", providerType, len(results))
return nil
}