Two-phase commit gives you real ACID guarantees across multiple databases, and almost nobody uses it at scale for a simple reason: it requires every participant to hold locks and stay available for the entire duration of the transaction, coordinated by a single component whose failure blocks the whole operation. In a microservices architecture where “reserve inventory,” “charge payment,” and “create shipment” are three different services with three different databases, none of which want to hold a lock while waiting on the others, 2PC’s coordination cost is usually worse than the problem it solves.
The Saga pattern trades strict atomicity for something more workable: a sequence of local transactions, each of which commits independently, with an explicit compensating transaction defined for each step to undo it if a later step fails. Nothing is ever locked waiting on a remote party — you accept a temporary inconsistent state and guarantee you can always walk it back to consistent.
The core model: steps with paired compensations
Every saga step needs exactly two things: an action, and the compensation that undoes it. Model that explicitly instead of scattering rollback logic across exception handlers:
from dataclasses import dataclass
from typing import Callable, Any
from enum import Enum
class StepStatus(Enum):
PENDING = "pending"
COMPLETED = "completed"
COMPENSATED = "compensated"
FAILED = "failed"
@dataclass
class SagaStep:
name: str
action: Callable[[dict], Any]
compensation: Callable[[dict], Any]
status: StepStatus = StepStatus.PENDING
result: Any = None
The dict context passed to both action and compensation is deliberate — a compensation often needs data the action produced (you can’t release a specific inventory reservation without knowing the reservation ID the action created), so state has to flow forward through the saga, not just errors flowing backward.
An order fulfillment saga, concretely
def reserve_inventory(context: dict) -> dict:
reservation = inventory_service.reserve(context["order_id"], context["items"])
return {"reservation_id": reservation.id}
def release_inventory(context: dict):
inventory_service.release(context["reservation_id"])
def charge_payment(context: dict) -> dict:
charge = payment_service.charge(context["customer_id"], context["total_amount"])
return {"charge_id": charge.id}
def refund_payment(context: dict):
payment_service.refund(context["charge_id"])
def create_shipment(context: dict) -> dict:
shipment = shipping_service.create(context["order_id"], context["address"])
return {"shipment_id": shipment.id}
def cancel_shipment(context: dict):
shipping_service.cancel(context["shipment_id"])
ORDER_SAGA_STEPS = [
SagaStep("reserve_inventory", reserve_inventory, release_inventory),
SagaStep("charge_payment", charge_payment, refund_payment),
SagaStep("create_shipment", create_shipment, cancel_shipment),
]
Step ordering here isn’t arbitrary — inventory is reserved before payment is charged, so a failed payment never leaves inventory over-committed for longer than necessary, and shipment creation happens last, after the two steps that can actually fail for business reasons (out of stock, declined card) have already succeeded.
The orchestrator: run forward, unwind backward on failure
Orchestration — a single component explicitly driving the saga forward and deciding when to compensate — is the more testable and debuggable choice over choreography (each service reacting to the previous one’s event for most teams, because the full flow lives in one place instead of being implicit across N services’ event handlers:
class SagaExecutionError(Exception):
def __init__(self, failed_step: str, original_error: Exception):
self.failed_step = failed_step
self.original_error = original_error
super().__init__(f"Saga failed at step '{failed_step}': {original_error}")
class SagaOrchestrator:
def __init__(self, steps: list[SagaStep]):
self.steps = steps
def execute(self, initial_context: dict) -> dict:
context = dict(initial_context)
completed_steps: list[SagaStep] = []
for step in self.steps:
try:
result = step.action(context)
step.status = StepStatus.COMPLETED
step.result = result
if result:
context.update(result)
completed_steps.append(step)
except Exception as exc:
step.status = StepStatus.FAILED
self._compensate(completed_steps, context)
raise SagaExecutionError(step.name, exc) from exc
return context
def _compensate(self, completed_steps: list[SagaStep], context: dict):
# Unwind in reverse order — last completed step gets compensated first
for step in reversed(completed_steps):
try:
step.compensation(context)
step.status = StepStatus.COMPENSATED
except Exception as comp_exc:
# A failed compensation is a genuine incident — surface it loudly, don't swallow it
emit_alert(f"Compensation failed for step '{step.name}': {comp_exc}", severity="critical")
raise
The reverse-order unwind matters: if payment charging fails after inventory was reserved, you compensate inventory (the only completed step) — you never call a compensation for a step that never ran. And a failed compensation is treated as a different, more serious class of problem than a failed action — the saga’s whole safety guarantee depends on compensations succeeding, so a compensation failure needs a human, not a silent retry-and-move-on.
Compensations must be idempotent
A compensation might be retried — the orchestrator process could crash mid-unwind and resume, or a compensation call might time out without the caller knowing if it actually succeeded. If release_inventory isn’t idempotent, a retry could release inventory that a different order’s reservation now occupies:
def release_inventory(context: dict):
# Idempotent: releasing an already-released reservation is a no-op, not an error
inventory_service.release_if_reserved(context["reservation_id"])
def refund_payment(context: dict):
# Idempotent: check for an existing refund against this charge before issuing a new one
if not payment_service.has_refund(context["charge_id"]):
payment_service.refund(context["charge_id"])
This is non-negotiable for production sagas — every compensation function should be safe to call twice with the same context and produce the same end state as calling it once.
Surviving a crash mid-saga: persist state, don’t just hold it in memory
The SagaOrchestrator above loses all state if the process crashes between steps — an order could be left with inventory reserved and no payment charged, forever, if nothing resumes it. A durable saga log fixes this:
class DurableSagaOrchestrator(SagaOrchestrator):
def __init__(self, steps: list[SagaStep], saga_log):
super().__init__(steps)
self.saga_log = saga_log # e.g. a Postgres table or Redis-backed log
def execute(self, saga_id: str, initial_context: dict) -> dict:
context = dict(initial_context)
completed_steps: list[SagaStep] = []
for step in self.steps:
self.saga_log.record_step_started(saga_id, step.name, context)
try:
result = step.action(context)
step.status = StepStatus.COMPLETED
if result:
context.update(result)
self.saga_log.record_step_completed(saga_id, step.name, result)
completed_steps.append(step)
except Exception as exc:
self.saga_log.record_step_failed(saga_id, step.name, str(exc))
self._compensate(completed_steps, context)
raise SagaExecutionError(step.name, exc) from exc
self.saga_log.record_saga_completed(saga_id)
return context
A background recovery job then scans the saga log on startup (or periodically) for sagas stuck in a non-terminal state, and either resumes forward execution or runs compensation for whatever completed before the crash — the log is what makes “resume after a crash” possible instead of “manually investigate which orders are in a broken state.”
Testing: assert the exact compensation sequence, not just the failure
The valuable test for a saga isn’t “does it raise an exception when a step fails” — it’s “does it compensate the right completed steps in the right order”:
def test_payment_failure_compensates_only_inventory():
calls = []
steps = [
SagaStep("reserve_inventory", lambda ctx: calls.append("reserve") or {"reservation_id": "r1"},
lambda ctx: calls.append("release")),
SagaStep("charge_payment", lambda ctx: (_ for _ in ()).throw(PaymentDeclinedError()),
lambda ctx: calls.append("refund")),
SagaStep("create_shipment", lambda ctx: calls.append("ship"),
lambda ctx: calls.append("cancel_ship")),
]
orchestrator = SagaOrchestrator(steps)
with pytest.raises(SagaExecutionError):
orchestrator.execute({"order_id": "o1"})
assert calls == ["reserve", "release"] # shipment never ran, so it's never compensated
That last assertion — create_shipment never appears in calls at all, not even its compensation — is the behavior that actually matters. A test that only checks “an exception was raised” would pass even if the orchestrator incorrectly tried to compensate a step that never executed.
Related Reading
- The Circuit Breaker pattern in Python
- The Bulkhead pattern for isolating AI inference failures
- microservices.io’s reference on the Saga pattern
The takeaway
The Saga pattern doesn’t give you the atomicity 2PC promises — it gives you something more honest about how distributed systems actually fail: a guaranteed path back to a consistent state through explicit, idempotent compensations, persisted durably enough to survive a crash mid-flight. That’s a weaker guarantee on paper and a meaningfully more operable one in practice, which is why it’s the pattern behind order fulfillment at Amazon-scale systems rather than a distributed lock held across three services and a network partition waiting to happen.
Deepak Balasubramaniam — Technical Manager, 14 yrs full-stack (Django/React/AWS), Writes on system design & AI-assisted dev