nullkey
c695125dce
Large Feishu wikis stopped syncing after the first batch (Tencent/WeKnora#2136: 5000+ docs, only ~150 ingested; #1262: a child-list failure aborted the run). Three compounding causes, fixed in layers: 1. Rate-limit / transient failures had no retry. Feishu's wiki/drive/export APIs are aggressively rate limited; a thousand-doc sync issues tens of thousands of calls, and a single 429/5xx burst failed whole swathes of docs. Add exponential backoff honouring Retry-After to doRequest (JSON) and downloadRawBytes (files): 429 up to 3x, 5xx once, transport errors back off. 2. The batch path fetched the entire tree into memory before ingesting, so a sync that hit the 2h task timeout lost all progress and restarted from scratch, and a transient per-document export failure advanced the cursor and skipped that doc permanently. Add an optional StreamingConnector interface, implemented for Feishu: FetchStream interleaves fetch -> ingest -> checkpoint, bounding memory to one item and persisting a per-node edit-time cursor at page boundaries. The service dispatches to it via type assertion; connectors without it fall back to FetchAll / FetchIncremental unchanged (non-breaking). A resumed/incremental run skips nodes already recorded at their current edit time (converges), a failed fetch does not advance the cursor (retried next run instead of dropped), and partial listing failures emit error items and carry prior edit times forward instead of aborting the sync. 3. Failures and skips were invisible. Unsupported node types (mindnote/slides/…) were skipped with no item, error or log ("13 docs synced only 3"). FetchAll now tallies discovered/fetched/failed/skipped per resource with a per-type skip breakdown and a progress line every 100 nodes. The backend already persisted per-item errors in SyncLog.result.errors but the frontend type omitted the field; declare `result` on SyncLog and render up to 50 failed documents in the sync-log drawer (#1262). i18n for zh-CN/en-US/ko-KR/ru-RU. result.errors is a jsonb column shipped in every list response, so it is capped at 100 samples server-side (recordSyncError); the exact failure total comes from the bounded items_failed counter, which the drawer uses for its count and "N more" line so a sync that fails thousands of docs stays bounded. Each sample is a structured, localisable SyncItemError instead of a raw "title: ...api error: status=500 body={json}" dump that repeated the title and leaked status/body/log_id. The connector (feishuFailure) classifies the error into a stable i18n code (+ the feishu code as a param); the service builds {title, code, params, message}; the drawer localises via vue-i18n to the viewer's language (zh/en/ko/ru), falling back to message. A custom UnmarshalJSON keeps old string-form logs readable. The banner is localised from items_failed too. Raw errors stay in the server logs. Follows the two-tier norm of Airbyte/Fivetran/Onyx: human, actionable, localised summaries in the UI; raw detail in logs. Transient errors say "will retry" (the cursor is retained); auth/permission errors point at the fix instead. Tests: client_retry_test (backoff), tally_test (accounting), connector_stream / datasource_stream (streaming dispatch + checkpoint), connector_convergence (timeout + transient-failure resume converges to full coverage; proven to catch #2136 via bug injection), datasource_result_cap (error-sample cap + structured localisable SyncItemError, no raw body), connector_error_reason (error→i18n-code classification), sync_item_error (legacy-string + structured back-compat) — all in the normal suite. connector_realapi / connector_seed are an opt-in real-Feishu harness gated by the feishu_integration build tag AND env vars, never run in CI.
68 lines
2.8 KiB
Go
68 lines
2.8 KiB
Go
package service
|
|
|
|
import (
|
|
"context"
|
|
"testing"
|
|
|
|
"github.com/Tencent/WeKnora/internal/types"
|
|
"github.com/stretchr/testify/assert"
|
|
)
|
|
|
|
// A sync that fails thousands of documents must not accumulate an unbounded
|
|
// result.Errors slice: that slice is persisted as jsonb and shipped in every
|
|
// sync-log list response, so an uncapped list means multi-MB rows and payloads.
|
|
// The accurate count lives in result.Failed (a bounded int), so Errors only
|
|
// needs to retain a sample for display (Tencent/WeKnora#2136 / #1262).
|
|
func TestApplyFetchedItem_CapsErrorSampleButKeepsAccurateFailedCount(t *testing.T) {
|
|
s := &DataSourceService{}
|
|
ds := &types.DataSource{}
|
|
result := &types.SyncResult{}
|
|
ctx := context.Background()
|
|
|
|
const failures = 5000
|
|
for i := 0; i < failures; i++ {
|
|
s.applyFetchedItem(ctx, ds, &types.FetchedItem{
|
|
ExternalID: "node",
|
|
Title: "doc",
|
|
Metadata: map[string]string{"error": "export failed: rate limited"},
|
|
}, nil, result)
|
|
}
|
|
|
|
assert.Equal(t, failures, result.Failed,
|
|
"Failed must count every failure regardless of the error-sample cap")
|
|
assert.LessOrEqual(t, len(result.Errors), maxSyncResultErrors,
|
|
"Errors sample must be capped so the persisted jsonb stays bounded")
|
|
assert.Equal(t, maxSyncResultErrors, len(result.Errors),
|
|
"with more failures than the cap, the sample should be exactly the cap")
|
|
}
|
|
|
|
// The user-facing sample must be a structured SyncItemError carrying the
|
|
// connector's stable i18n code + params (so the frontend localises it), the
|
|
// title as a separate field, and no raw API body/log_id — that stays in logs.
|
|
func TestApplyFetchedItem_ProducesLocalisableStructuredError(t *testing.T) {
|
|
s := &DataSourceService{}
|
|
ds := &types.DataSource{}
|
|
result := &types.SyncResult{}
|
|
|
|
s.applyFetchedItem(context.Background(), ds, &types.FetchedItem{
|
|
ExternalID: "nt1",
|
|
Title: "季度报告",
|
|
Metadata: map[string]string{
|
|
// raw, for logs only
|
|
"error": `export 季度报告 (docx): feishu api error: status=500 body={"code":1663,"error":{"log_id":"20260"}}`,
|
|
// classification, for a localisable UI sample
|
|
"error_reason_code": "feishu_api_error",
|
|
"error_reason_code_value": "1663",
|
|
"error_reason": "Feishu API error (code=1663); will retry automatically",
|
|
},
|
|
}, nil, result)
|
|
|
|
assert.Equal(t, 1, len(result.Errors))
|
|
got := result.Errors[0]
|
|
assert.Equal(t, "季度报告", got.Title, "title is a separate field, not embedded in the message")
|
|
assert.Equal(t, "feishu_api_error", got.Code, "stable i18n code drives frontend localisation")
|
|
assert.Equal(t, "1663", got.Params["code"], "feishu error code is passed as an interpolation param")
|
|
assert.NotContains(t, got.Message, "body=", "raw API body must not reach the UI sample")
|
|
assert.NotContains(t, got.Message, "log_id", "raw log_id must not reach the UI sample")
|
|
}
|