Files
Enterprise-Knowledge-Base/.gitignore
T
wizardchen 13260b831c Fix wiki ingest silent data loss from malformed JSON in Redis queue
Two critical fixes to the peekPendingList function:

1. Add error logging for JSON unmarshal failures (line 303)
   - Previously: Malformed JSON items were silently skipped with no indication
   - Now: Each failure is logged with the error and raw item content (first 100 chars)
   - This makes data loss visible and debuggable

2. Fix trim count mismatch (line 325)
   - Previously: Returned len(result) - the count of raw Redis items peeked
   - Now: Returns len(ops) - the count of successfully parsed items
   - Impact: Malformed items now stay in Redis for retry instead of being discarded
   - This prevents silent data loss when JSON encoding issues occur

Root cause of the reported issue:
- When 8 files are uploaded in a batch, if one has encoding issues that cause
  JSON.Unmarshal to fail, it was silently dropped from processing
- The file was trimmed from Redis without being processed
- Result: Only 7 of 8 files appeared in the Wiki

Fix approach:
- Make unmarshal failures visible in logs
- Retry malformed items in the next batch instead of discarding them
- Admin can see the errors and take corrective action

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 17:21:42 +08:00

55 lines
674 B
Plaintext

# 忽略所有隐藏文件和目录
.*
# 但不忽略特定隐藏文件和目录
!.env.example
!.gitignore
!.github/
# 敏感文件
*.pem
*_key
*_secret
*.key
*.crt
# IDE和编辑器文件
*.swp
*.swo
# 构建和依赖文件
node_modules/
/dist/
/build/
*.log
# 临时文件
tmp/
temp/
logs/
*.pid
WeKnora
WeKnora-lite
cmd/desktop/build/bin/
/models/
test/data/mswag.txt
data/files/
data/weknora.db
data/weknora.db-wal
data/weknora.db-shm
web/
**/__pycache__
/scripts/scale_dev_jobs.sh
/server
frontend/.vite
frontend/chrome-extension/
WeKnora-Chrome-Extension
packages/protoc-3.19.4-linux-x86_64.zip
Claude.md
# 本地测试脚本
configure-ollama.sh
fix-docker.sh