Decide what success means before the test starts
One way of wording a title matches 28,413 searches a month; the other, 24. Before we test which wins, we write down how we will judge it. Why that order matters.
This month, on our current engagement, we start testing article titles. The question is simple: should titles use the exact words people type into search? For one topic, one wording matches about 28,413 searches a month; the other, 24. The answer seems obvious. That is exactly why we write down the rules first.
Why decide first
Once results start arriving, everyone finds the number that agrees with what they hoped. Clicks up but positions down? Pick clicks. One week good, the next bad? Pick the good week. Deciding the measure in advance is the only defence against fooling yourself, and it is free.
What we write down before any test
- The change: exactly what differs, and on which pages.
- The measure: one main number, such as clicks from search per page, and where it comes from.
- The comparison: similar pages left unchanged over the same weeks, so seasons and Google updates hit both groups.
- The window: how long the test runs before anyone reads the result.
- The bar: how big a difference counts as a win, and what happens if it is smaller.
- What would stop it early: for example, a clear drop on pages that earn.
What we will report
Whatever happens. A test that shows no difference is a result: it means the effort can go somewhere else. We will label the outcome proven, likely or unknown, like everything else in our weekly reports.
A goal, not a forecast
For search, the target is simple: the first page of Google for many searches, and the top three for the few that earn. Each test is judged against that, not against hope.
The examples in this article come from real engagements. Client details withheld; every figure comes from the client’s own data.
Read the case study →