AI, in plain words

Blog / Custom software

Tests, explained for people who never wrote them

What an automated test is, why it matters more when AI writes the code, which few tests to start with, and how to tell whether the tests an AI tool wrote actually check anything.

Every time someone changes software, something that used to work can quietly stop working. A new discount rule breaks the checkout. A faster search drops half the results. Nobody notices until a customer does.

Tests are how a team notices first. You don’t need to write them to understand them, and if you’re building with an AI tool, understanding them matters more than ever.

What a test is

A test is a small piece of code that checks one thing and answers yes or no. “If a customer orders two items at $10, the total is $20.” “If the password is wrong, the login is refused.” The point is that it runs automatically, every time the code changes, in seconds.

Think of a pilot’s pre-flight checklist. The pilot knows how to fly. The checklist exists because people forget, and forgetting one item is expensive. Tests are the checklist your software runs on itself before every release.

Why they matter more when AI writes the code

AI tools change code fast, and often in many places at once. Ask for a small fix and the tool may also “tidy up” three other files. A person reviewing that change can easily miss what was touched. Tests don’t get tired and don’t skim. If the tidy-up broke the invoice total, the invoice test fails, and you find out before your customers do.

Tests also give the AI tool feedback. Many coding assistants can run the tests themselves, see what failed and fix it. Without tests, the tool is guessing whether its change worked, and so are you.

The kinds, in plain words

  • Does this piece work? Checks one small part on its own, like the function that calculates tax. Engineers call these unit tests. They are fast and cheap, so you can have hundreds.
  • Do these pieces work together? Checks that two parts talk to each other correctly, like your app saving an order to the database. These are integration tests.
  • Does the whole flow work? Acts like a real user: opens the site, signs up, buys something, gets the email. These are end-to-end tests. They are slower and break more easily, so keep a few for the flows that matter most.

The few tests to start with

If a project has no tests, don’t aim to cover everything. Aim at the places where a failure costs money or trust:

  1. The money path. Prices, totals, payments, invoices.
  2. The way in. Sign-up, login, password reset.
  3. Who sees what. One customer must never see another customer’s data.
  4. The main job. The one thing users come for, from start to finish.
  5. Every bug you fix. When something breaks, add a test that would have caught it, so it can’t come back unnoticed.

Then make the tests run automatically on every change, and block a release when one fails. A test nobody runs protects nothing.

How to ask an AI tool to write them

Be specific. “Write tests” gets you tests that pass. That is not the same as tests that check.

  • “Write tests for the order total: a normal order, an empty cart, a discount, a refund. Include cases that should be rejected.”
  • “Test that a user from company A can never read company B’s records.”
  • “Before writing any tests, list what this code is supposed to do, so I can confirm it.”

That last one matters most. If the tool writes tests from the code alone, it will test what the code does, bugs included. Tests should describe what the code is supposed to do, and only you can confirm that.

How to check the tests aren’t fake

AI tools sometimes write tests that look busy but check nothing, or quietly change a test until it passes. Here is what to look for:

  • Break it on purpose. Change one number in the code, like a tax rate, and run the tests. If nothing fails, no test is watching that number.
  • Read the names. Test names should read like sentences about behavior: “refuses login with a wrong password.” Vague names like “test 1” usually hide vague checks.
  • Look for the yes-or-no. Every test should compare a result with an expected value. A test that only runs the code without checking the result will always pass.
  • Watch for edited expectations. If a change made a test fail and the “fix” was to change the expected answer, ask why. Sometimes that is right. Often it hides a bug.
  • Count skipped tests. Tests marked “skip” don’t run. A growing skip list is a growing blind spot.

Checklist: tests you can rely on

  • Payments, login and data separation have tests.
  • Tests run automatically on every change.
  • A failing test blocks the release.
  • You broke something on purpose, and a test caught it.
  • Test names describe behavior in plain sentences.
  • No test was changed just to make it pass.
  • Every fixed bug has a test that keeps it fixed.

Written from our engineers’ work on production systems. Want a second opinion on your project? Talk to an engineer.

See the work →

Want us to look
at your site?

Tell us where traffic, revenue or your numbers stopped making sense. We will tell you what we would check first.

Prefer to write directly? enable JavaScript to see the address

Talk to an engineer

No sales theater. Tell us where your operation feels slow, repetitive or difficult. An engineer reads every message and replies by email.

Prefer to talk? Pick a 15-minute slot →

Your message goes straight to our engineers at our address.