Event-driven microservices that stay consistent

Teams could ship services independently without breaking each other or losing events.

Software architecture / Hands-on engineering / Engineering leadership

Problem

Services called each other synchronously. One slow dependency stalled every request behind it, and each team’s release had to be coordinated with the others.

Constraints

No lost or double-applied business events
Teams migrating at different speeds
Existing PostgreSQL data as the source of truth

Architecture & approach

Services write state and an outbox record in one transaction. A relay publishes the outbox to Kafka, and consumers are idempotent, so retries are safe. Redis handles hot read paths. Synchronous calls remain only where a user is actually waiting.

Fig. 305-2Outbox to Kafka with idempotent consumers Critical path

Key decisions

How do we publish events without losing any? 305-3.1

RejectedWrite to the database, then publishA crash between the two steps loses the event.
RejectedDistributed transaction across DB and brokerCorrect, but slow and operationally fragile.
ChosenTransactional outbox plus idempotent consumersOne local transaction. Replays are safe by design.

Outcome

Services release independently
A slow dependency no longer stalls unrelated requests
Event delivery is safe to retry end to end

Technology

Kafka, PostgreSQL, Redis, Microservices, REST APIs, Docker

Lessons & tradeoffs

Exactly-once is a property you design with idempotency, not a setting you buy.
Keep synchronous calls for where a person is waiting. Everything else can be an event.