Files
Enterprise-Knowledge-Base/internal/application
wizardchen 13260b831c Fix wiki ingest silent data loss from malformed JSON in Redis queue
Two critical fixes to the peekPendingList function:

1. Add error logging for JSON unmarshal failures (line 303)
   - Previously: Malformed JSON items were silently skipped with no indication
   - Now: Each failure is logged with the error and raw item content (first 100 chars)
   - This makes data loss visible and debuggable

2. Fix trim count mismatch (line 325)
   - Previously: Returned len(result) - the count of raw Redis items peeked
   - Now: Returns len(ops) - the count of successfully parsed items
   - Impact: Malformed items now stay in Redis for retry instead of being discarded
   - This prevents silent data loss when JSON encoding issues occur

Root cause of the reported issue:
- When 8 files are uploaded in a batch, if one has encoding issues that cause
  JSON.Unmarshal to fail, it was silently dropped from processing
- The file was trimmed from Redis without being processed
- Result: Only 7 of 8 files appeared in the Wiki

Fix approach:
- Make unmarshal failures visible in logs
- Retry malformed items in the next batch instead of discarding them
- Admin can see the errors and take corrective action

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:42 +08:00
..