How Kurly's Inbound System Safely Synchronizes External Inbound Data

Kurly adopted outbox pattern and retry topic to synchronize external inbound data, sCM product team described synchronization of inbound data from external channels.

External channels provide inbound schedule data either by calling Kurly's creation API or by exposing a retrieval API Kurly polls, partners may call create API or Kurly may poll partner APIs.

Legacy integration flow grouped interface DB write and Kafka message publish in one transaction, dB write and Kafka produce were previously in a single transaction.

Failure modes commit failure or missing dependent data can break atomicity between interface table and Kafka message or prevent creation of inbound records mentioned commit failures and missing dependent data as causes.

Outbox pattern store outbound message in an outbox table and use a poller to publish Kafka messages and mark records after successful publish described sequence of storing message in outbox table then polling to publish.