← Back to workCase study 01 / 04
Engati · Resilience
Auto-retry framework
Meta can’t deliver every message the first time. This framework gives failed deliveries another chance, for about 2 million API triggers a day.
35% → 12%failure rate
~2MAPI triggers a day
50K–100Kretries a day
300K–400Kretries a day at peak, during Meta failures
(01) The problem
Marketing and CRM platforms like LeadSquared and MoEngage trigger messages through Engati’s API. When Meta failed to deliver one, it stayed failed: 35% of triggers were failing before this framework.
(02) How it works
- LeadSquared or MoEngage sends an API trigger.
- It enters through the API gateway and reaches trigger-mvc, the action trigger management service.
- trigger-mvc hands it to the messaging pipeline, which keeps its trackerId in Redis and sends it through Meta to the user.
- Meta sends a webhook back to the webhook receiver: delivered, or failed with a status code.
- The analytics pipeline records why it failed and passes the failure on to trigger-mvc.
- trigger-mvc checks the status code. Retryable failures wait in RabbitMQ, with fixed-interval or exponential back-off.
- When the wait is over, trigger-mvc reads the trackerId from Redis, fetches the original payload from MongoDB and sends it out through messaging again.
(03) Key decisions
trackerId as the idempotency key
Every retry carries the trackerId of the original message, so it finds the right payload and the same message isn’t sent twice.
Retry only what can succeed
Status codes separate temporary failures from permanent ones, so the system doesn’t keep hammering messages that will never go through.
Back off, don’t pile on
Fixed-interval and exponential back-off spread retries out, so a Meta outage doesn’t turn into a retry storm.
(04) Results
- Failure rate cut from 35% to 12%.
- Handles ~2M API triggers and 50K–100K retries a day.
- Absorbed peaks of 300K–400K retries a day during Meta delivery failures.