Ready for a sale rush

We tested what happens when many shoppers arrive at once, as on Black Friday, with a load testing tool that runs hundreds of shoppers in parallel: browsing the catalogue, opening products, adding to cart, details and checkout. At 300 shoppers the system jammed: 90 requests per second, 95% of responses within 15 seconds, and 6% errors. The cause was code that asked for another database connection in the middle of an open order, so under load all the connections waited for each other. We fixed two such places (the gift offer in the cart and the customer's account page), and added a check that runs all the tests with a single connection, so a bug of this kind cannot come back quietly. After the fix, with only 4 connections: 2,093 requests per second, 95% of responses within 2 milliseconds, and 0.01% errors. During the load, inventory never went below zero, and a product that sold out was blocked at checkout instead of being sold twice. The server now moves to 20 connections per process.

New 1

  • A load testing tool that runs hundreds of shoppers in parallel and fails when the targets are not met.

Improved 1

  • 20 database connections per server process, and a check that catches this bug in advance.

Fixed 1

  • A store that jammed under load because of another database connection in the middle of an order.