From zero automated checks to 918 in under nine weeks

When we took stock of an AI sales-coaching product we were building, the code from its first sixteen weeks had no automated checks at all. It worked, because people checked it by hand. That does not scale, and it does not survive the next change.

Three layers

• Automated checks on the logic and the database: from 0 to 287 in the first week of the effort, and 918 by the release on 25 September.
• Full customer journeys run in a real browser on every change: sign up, connect, score, review. Ten of them.
• Hands-on review by people using the product as a customer would: 110 fixes across 5 rounds, the problems automation does not think to look for.

A gate, not a suggestion

Checks only help if a failure stops the release. A release is blocked whenever any check fails, is still running, or ran on a different version of the code.

We check the safety net too

A safety net is only as good as its own checks. Two examples of tightening it:

• Database and code always in step. One release went out missing a database change; it was fixed within minutes, and a new check now compares the code’s expectations with the live database before every release.
• A gate that cannot be fooled. After a credential expired, the release gate could not read the check results and let every release pass. We found it and fixed it, so an unreadable result now blocks the release.