28781785f7
- Introduced a new `WikiBatchContext` to share data across Map and Reduce phases, reducing redundant database queries. - Replaced separate deduplication calls with a single `deduplicateExtractedBatch` method, improving efficiency by leveraging pre-loaded page data. - Enhanced JSON handling by implementing a `cleanLLMJSON` function to sanitize LLM-generated output, ensuring safe parsing. - Updated relevant methods to utilize the new context and deduplication logic, streamlining the overall ingestion workflow. These changes enhance the performance and clarity of the wiki ingestion service, leading to more efficient content processing.