What it costs

Faster and cheaper

ClickHouse cost of the journey worker, every 5 minutes around the 11:43 UTC switch. The old job was busy all the time and still losing ground; after a one-time catch-up (amber), the new worker keeps up live for about half the CPU and a seventh of the reading.

ClickHouse CPU-seconds per 5 minutes −47%06012011:20 UTC · 80.3 cpu8011:2011:25 UTC · 71.3 cpu7111:30 UTC · 74.2 cpu7411:3011:35 UTC · 73.5 cpu7411:40 UTC · 84.2 cpu8411:4011:45 UTC · 106.6 cpu10711:50 UTC · 39.5 cpu4011:5011:55 UTC · 40.3 cpu4012:00 UTC · 39.2 cpu3912:0012:05 UTC · 41.8 cpu4212:10 UTC · 41.7 cpu4212:10
Million rows read per 5 minutes −85%016533011:20 UTC · 255.1 rows25511:2011:25 UTC · 264 rows26411:30 UTC · 264.6 rows26511:3011:35 UTC · 232.8 rows23311:40 UTC · 262.7 rows26311:4011:45 UTC · 313.3 rows31311:50 UTC · 36.9 rows3711:5011:55 UTC · 37.7 rows3812:00 UTC · 37.9 rows3812:0012:05 UTC · 40.9 rows4112:10 UTC · 40.4 rows4012:10
1.7%of prod ClickHouse
Production has 8 cores (2 replicas × 4). The worker's pod itself runs at 0.025 CPU and 54 MiB.
ClickHouse CPU-seconds by user, over 20 minutes (11:55–12:15 UTC)
Journey worker: 163 s in 20 min ≈ 8 CPU-seconds a minute ≈ 0.14 of one core on average ≈ 3.3 CPU-hours a day.
Acquisition reconciler
211s
Journey worker
163s
Monitoring
76s
Ads egress
45s
Jitsu
31s
Observability
27s
Grafana
11s
old scheduled jobcatch-up: 7.5 h of backlog clearedcontinuous worker