Checkout Outage on October 18
Incident report: 47 minutes on Sunday morning
In plain terms
Every time a customer clicks Pay, our checkout service borrows one of a fixed number of shared lines to the orders database, saves the order, and hands the line back. We normally allow 200 lines at once. A routine settings change on Sunday morning cut that number to 20. Sunday morning is one of our busiest times, so the 20 lines were used up within minutes and everyone else got an error.
The change was meant for our test environment, which has very little traffic. It was copied into the live settings by mistake. Fixing it took 47 minutes because our usual way of undoing a change refused to run while checkout was unhealthy.
Impact
| measure | value | note |
|---|---|---|
| Checkout attempts during the outage | 41,800 | Normal Sunday-morning volume |
| Attempts that failed | 27,900 | 67%; the rest got through as lines were freed up |
| Customers who gave up | 10,500 | Did not complete an order within 24 hours |
| Estimated lost sales | $672,000 | 10,500 orders at the $64 average order |
| Support tickets opened | 1,860 | Mostly 'my payment failed'; 94% answered within 4 hours |
| Customers charged twice | 0 | Failed attempts never reached the payment step |
| Payment card data exposed | 0 | Not involved in this incident |
The chart shows the share of checkout attempts that succeeded. It dropped from 99% to about 30% within minutes of the change and stayed there until engineers fixed the setting by hand.
Data
| time | rate | Event |
|---|---|---|
| 10:00 | 99.4% | |
| 10:04 | 99.3% | |
| 10:08 | 99.4% | |
| 10:12 | 99.2% | Change live |
| 10:16 | 42% | |
| 10:20 | 31% | |
| 10:24 | 30% | |
| 10:28 | 32% | |
| 10:32 | 31% | |
| 10:36 | 33% | |
| 10:40 | 30% | |
| 10:44 | 32% | |
| 10:48 | 31% | |
| 10:52 | 34% | Fix applied |
| 10:56 | 81% | |
| 11:00 | 99.1% | |
| 11:04 | 99.4% |
Timeline
All times are Eastern, Sunday October 18.
| time | what happened | who |
|---|---|---|
| 10:12 AM | A routine settings change is released to all customers at once | Automated release system |
| 10:16 AM | Checkout failures climb as Sunday-morning shoppers arrive | n/a |
| 10:19 AM | Alert fires after failures stay high for 5 minutes; the on-call engineer is paged | Nia Osei |
| 10:24 AM | Incident declared; an incident commander and 4 engineers join | Rafael Mendes |
| 10:31 AM | First fix, undoing the change, is blocked by the release system's safety check | Rafael Mendes |
| 10:38 AM | Cause found: the database line limit is 20, not 200 | Nia Osei |
| 10:52 AM | Engineers set the limit back to 200 by hand, bypassing the release system | Nia Osei and Rafael Mendes |
| 10:59 AM | Checkout success is back to 99%; incident closed at 11:30 AM after monitoring | Rafael Mendes |
Data
| phase | start | end |
|---|---|---|
| Not yet noticed | 0 | 7 |
| Finding the cause | 7 | 26 |
| Fixing | 26 | 40 |
| Recovering | 40 | 47 |
Why it happened
Three things lined up. Any one of them alone would likely have limited the damage.
- A test-environment value was copied into the live settings file. The value, 20, is correct for the test environment but far too small for live traffic.
- Review did not catch it. The reviewer saw a one-line change and the tool did not show what the live value had been before.
- There was no gradual rollout, and no easy undo. The change reached every customer at once, and the release system will not undo a change while checkout is unhealthy, which is exactly when an undo is needed.
Detection was also slower than it should be: the alert waits for 5 minutes of sustained failures, so 7 minutes passed before anyone was paged.
For engineers: the change
# config/prod/checkout-service.yaml
database:
- pool_max_connections: 200
+ pool_max_connections: 20# release gate that blocked the rollback
post_deploy_gate:
require: checkout_success_rate >= 95
applies_to: [deploy, rollback]What we are changing
| action | owner | due | status |
|---|---|---|---|
| Live setting restored to 200 and locked against edits outside the release system | Rafael Mendes | Oct 18 | Done |
| Page the on-call engineer after 2 minutes of failures instead of 5 | Nia Osei | Oct 19 | Done |
| Show old and new live values side by side in every settings review | Tessa Lindqvist | Oct 23 | In progress |
| Automatically block any setting that cuts a live limit by more than half unless a second person approves | Tessa Lindqvist | Oct 30 | In progress |
| Allow an undo even when checkout is unhealthy | Rafael Mendes | Oct 27 | In progress |
| Roll out settings to 5% of customers first and to everyone after 10 healthy minutes | Jun Park | Nov 13 | In progress |
| Email and a $10 credit to the 10,500 customers who did not complete an order | Customer Support | Oct 22 | In progress |
Words used in this report
- Database lines (connection pool): the fixed number of simultaneous links between checkout and the orders database.
- Release system: the tool that delivers code and settings changes to the live site.
- Rollback (undo): returning to the previous version of a change.
- On-call engineer: the person who is responsible for answering alerts outside office hours.
- Alert: an automatic message that tells on-call something is wrong.