How This Works

Architecture

• Workers - 7 parallel workers, each with unique API key and IPv6
• Rate Limiting - 25 req/s per key per IP (Bungie's limit)
• Redis Queue - BullMQ queues all fetched records for batch processing
• Batch Consumer - Processes 1000 records at a time before writing
• ClickHouse - Columnar database with 70-90% compression
• Compression - ZSTD compression on JSON payloads

Data Flow

API Endpoints

• GET / - This dashboard
• GET /api/:pgcrId - Retrieve specific PGCR (decompressed JSON)
• GET /health - Health check endpoint

Storage

All PGCRs stored in d2_pgcrs.all_activities, while Gambit matches (modes 63 & 75) are duplicated to d2_pgcrs.gambit_activities for quick Edge of Fate analysis.