Two Databases Must Agree or Neither Changes. How Do You Arrange That?
By the BrainSnail editorial team. How these articles are written and checked, and how to tell us when one is wrong.
Making several independent systems commit a change together, or none of them commit at all, requires asking them all first and only then telling them to proceed.
The problem
A single action may need to update two separate systems, such as taking money from an account in one and adding it to an account in another. Each can commit its own part reliably. What neither can do alone is guarantee that the other also committed, so a failure between the two updates leaves the world inconsistent in a way that is difficult to detect and worse to repair. The requirement is that either both changes take effect or neither does, across systems that fail independently and cannot read each other's minds.
How the protocol runs
A coordinator drives the participants through two rounds:
- •Round one asks every participant whether it can commit
- •Each does all the work, locks what it needs and answers yes or no
- •A yes is a binding promise that it can still commit later
- •If every answer is yes, round two tells them all to commit
- •If any answer is no or is missing, round two tells them to abort
- •Participants must obey round two whatever has happened meanwhile
Why it blocks
The protocol has a well known and unfixable weakness. A participant that has promised to commit and then loses contact with the coordinator cannot decide anything on its own, because it does not know whether the others said yes. It must hold its locks and wait, and anything needing those resources waits with it. If the coordinator fails permanently at that moment, the participants are stuck indefinitely. Extensions add a third round to reduce the exposure, at the cost of more messages, and no arrangement removes the problem entirely.
Why the promise is the hard part
Answering yes in the first round commits a participant to something unusually demanding, and understanding that explains why the protocol is expensive. Having promised, it must be able to commit no matter what happens next, including its own crash and restart, so the prepared state and everything needed to complete has to be written durably before the answer is sent. Locks taken during the first round must be held until the second, which may be a long time. A participant that could not honour a promise after a power failure would break the whole guarantee.
What is used instead
Large systems mostly avoid the protocol rather than fixing it, and the alternatives trade strictness for availability. A saga performs each step as an independent transaction and defines a compensating action to undo each one, so a failure halfway through is corrected by running the compensations rather than by holding locks. Messages recorded in the same transaction as the data change and delivered afterwards achieve a similar effect. Both accept that the world is briefly inconsistent and guarantee that it converges, which is usually acceptable and is occasionally not.
The takeaway
Asking every participant whether it can commit, and only telling them to proceed once all have promised, makes several systems change together or not at all. A participant that has promised and then loses the coordinator must hold its locks and wait, which is the unfixable weakness. Large systems mostly use compensating actions instead and accept brief inconsistency.