Use your product as a customer with nothing in it

The team that builds a product almost never sees it the way a new customer does. Our accounts are full of data, our habits route us around the rough parts, and the demo account was set up to look good. A customer on their first day has an empty screen and no habits.

Before a recent release of a sales management tool we build, we walked every customer page twice: once as a brand-new company with nothing in it, and once as an established one. Then both again on a phone-sized screen. All the automated tests had passed. The walk still found more than thirty things to fix.

Results shown over no data

These are the ones a new customer meets in the first minute.

• A new account’s summary showed a best weekly score of minus infinity percent.
• A weekly report awarded a green trophy for a count of zero, and warned a company on its first day that its business activity had stopped.
• A page that compares two sets of scores graded how well they agreed before anything had been measured.

Nothing here is a crash. Each calculation ran correctly on the numbers it was given. It was given nothing, and nobody had decided what the screen should say then. Each one is now a plain statement of what is actually there.

Numbers that did not agree with themselves

• Dates on different pages read an hour apart, because some pages used the server’s clock and some used the browser’s.
• A “previous week” figure counted everything the person had ever done.
• One internal page mixed figures for the whole company with figures for one account, without saying which was which.

A user who sees two dates disagree stops trusting both, and then the rest of the page. Every date now goes through one formatter, and the build fails if a new page formats its own.

Copy that sent people the wrong way

• One page pointed to a setting that does not exist.
• Another called a nightly check clean when it had not run.
• A third told a team that had already done the work to go and do it.

Text like this gets written early, when the feature is a plan, and nobody rereads it once the feature changes.

Why the tests did not catch any of it

Automated tests answer the question they were asked: given this input, does the code return that output? They did, every time. Nobody had asked “what should a person with nothing see here?” or “does this sentence still describe the product?”. Those questions only come up when someone looks at the screen.

So the walk is now a required stage of every release, next to the automated ones: every page as an empty account, every page as a full one, on a desktop and on a phone. Where a finding can become an automatic check, it does. The date formatter is one example. The looking still has to be done by someone.

Then do it again on the live product

Straight after the release we signed in to production and walked it there. Every page loaded and nothing was broken. The walk still found the most important thing of the week: some scheduled jobs were not doing their work on the live system. One summary had not been produced for weeks. A sync with another system had never been scheduled there at all, while its settings page said it ran every few hours.

No test environment could have shown that. The code was right and the pages rendered. Only the live system knew that the jobs had not run. It is the same family of problem as a check that reports “nothing wrong” when it could not look.

How to run this on your own product

• Create a new account the way a customer would. Not a copy of a test account. Sign up, and touch nothing else first.
• Open every page before adding any data. Write down every number, badge, chart and sentence that claims something. With no data, each of those claims is suspect.
• Add one record and look again. One is the other case nobody designs for: averages, trends and “compared with last week” all behave oddly.
• Do it on a phone. Use a real one if you can.
• Read every instruction and follow it literally. If it names a button or a setting, go and find it.
• After each release, repeat the short version on the live product, and check that anything scheduled has a recent run.

It takes a few hours. On this release it found more problems than any other stage.

Where this fits

The empty state is one of the things a prototype skips, because a prototype is always shown with data in it. Finding these before customers do is a large part of taking a product from prototype to production. An earlier round of the same kind of hands-on review on this product is in Test the paths that have never run.

See how we take a prototype to production