← Back to workCase study 02 / 04
Engati · Event-driven data
RCS billing pipeline
Every RCS message event ends up on a customer’s bill. This pipeline counts each one once, even when a job runs again.
~4Mconversations a day
~8Mbilling events a day, all in working hours
~500 RPSat peak; about 250 RPS on average
3 keysbotRef · customerId · metric type
(01) The problem
RCS usage is billed from webhook events, so the numbers have to be exact. About 8 million of them arrive across the working day, and a billing job that fails halfway, or runs twice, must never change what a customer pays.
(02) How it works
- Every RCS message event arrives as a webhook and is published to Apache Kafka.
- Events are persisted to AWS S3, so the raw data is always kept.
- Spark jobs aggregate them by botRef, customerId and metric type.
- The totals feed customer billing.
- The jobs are idempotent and replay-safe: re-running any period gives the same totals.
(03) Key decisions
Kafka in front
Publishing webhooks to Kafka separates receiving events from processing them, so a burst of traffic doesn’t hold billing up.
Keep the raw events
Everything lands in S3 first, so any period can be recomputed from the source if something goes wrong.
Idempotent by design
Running a job twice gives the same totals as running it once, so a replay can never double-bill a customer.
(04) Results
- Takes in ~8M billing events a day from ~4M conversations, all inside working hours: about 250 requests a second on average and ~500 at peak.
- Accurate customer billing for RCS, aggregated per bot, customer and metric.
- Failed or repeated jobs are safe to re-run: the totals stay the same.