How Do You Test on Real Users Without Breaking Everything? Send in One Bird
By the BrainSnail editorial team. How these articles are written and checked, and how to tell us when one is wrong.
Releasing new software to a small slice of real traffic first, watching what happens and only then continuing, converts a large risk into a small and reversible one.
How the process works
New software is deployed alongside the existing version rather than replacing it. A small proportion of live traffic, perhaps one percent, is routed to the new version while everything else continues on the old one. Metrics from both are compared directly, and if the new version behaves acceptably the proportion is increased in stages until it carries everything. If anything goes wrong, traffic is switched back and the affected proportion was small. The old version is only removed once the new one has carried full traffic for a while.
What it catches
The faults it finds are the ones testing cannot:
- •Behaviour under real traffic patterns rather than simulated ones
- •Interactions with real data, which is always stranger than test data
- •Performance problems that only appear at scale
- •Failures in dependencies that test environments stub out
- •Configuration differences between environments
- •Things nobody thought to write a test for
What has to be watched
The technique is only as good as the comparison being made, which is where most of the difficulty lies. Error rates and response times are the obvious measures and they are not sufficient, since a change can be technically healthy and commercially disastrous. Business measures matter as much, including completed purchases, sign-ups and whatever the system exists to produce. The comparison must be between the two versions at the same moment rather than against yesterday, because traffic varies by hour and by day. And somebody or something has to be watching, since a gradual rollout with nobody looking is just a slow deployment.
The related rollout strategies
Several deployment approaches are frequently confused with each other and the distinctions are practical. Running two complete environments and switching all traffic between them at once gives an instant reversal and no gradual exposure, so a fault reaches everybody before anybody notices. A rolling update replaces servers one at a time, which limits blast radius without deliberately comparing versions. Deploying a new version and sending it a copy of live traffic without using its responses tests behaviour at full scale with no user risk at all, at the cost of doubling load. Serious systems combine several.
Where it does not fit
Several situations defeat the approach and recognising them avoids false confidence. A change to a database schema cannot easily run in two versions at once, so such changes are split into separately deployable steps that are each compatible with both. Features that users notice cause confusion if some see them and others do not, particularly where those users interact. Low traffic systems take too long to accumulate meaningful evidence at one percent. And a fault that corrupts data rather than failing visibly is not caught by watching error rates, since the damage is silent and already done.
The takeaway
Routing a small share of real traffic to a new version alongside the old one exposes faults that testing cannot reproduce, at a cost proportional to the share. The comparison must include business measures and must be against the old version at the same moment. Schema changes, visible features, low traffic and silent data corruption all defeat it.