paidAffiliate disclosure. The partner link in the masthead and in the band beside the copy on this page is a sponsored link to a partner operator, and this site may be paid if you open an account through it, at no extra cost to you. It carries rel="sponsored noopener" and opens in a new tab. A desk about how a screen is measured and tested should not leave its own funding unsaid: one link funds the site, no operator and no product is named, rated or recommended anywhere on it, and this site runs no analytics of its own on its readers.
The Test Group / The limits
The changes made to a step whose cause was somewhere else
What the loop cannot fix
A funnel gap looks like a page problem because the page is where it is recorded. Often it is not. On the samples 11 of the 24 shipped changes, or 45.8%, were aimed at a step whose real cause lay outside the interface: an identity check, a bank's decision, a charge category, a device that could not load at all.
Desk spec
- experiments reviewed
- 40
- shipped
- 24
- aimed outside the page
- 11
- share that could not work
- 45.8%
the eventOne interaction, written down. A click carries a name, a time, an account, a session and what was on the screen, and it is kept whether or not the reader chose to be measured.
the funnelThe order the steps happen in. A landing visit becomes an account, an account becomes a deposit page, and 100,000 visits end in 2,074 first bets - a 2.1% path the whole loop is aimed at.
the armsTwo versions of one screen shown at the same time, and a rate for each. 4.30% against 5.16% is a 0.86-point lift, and the interval decides whether it is a result or a coincidence.
Direct answerAn interface experiment can only move what the interface controls. On the samples 40 experiments produced 24 shipped changes, and 11 of those 24 - 45.8% - were aimed at a funnel step whose real cause was outside the page: an identity check, a bank's decline, a charge category or a page a device could not load.
Four causes the page cannot reach
Each of these appears in the funnel as an abandoned step, and each is produced by something the reader is doing with a third party. The page can be rewritten a hundred times and the number will not move.
Sample I - the four off-page causes behind the 11 wasted changes
| Cause | Experiments aimed at it | Where the decision is actually made |
|---|
| an identity or document check | 4 | the operator's verification process, not the screen |
| a payment declined by the bank | 3 | the issuer, with no message the page can change |
| a charge the reader did not recognise | 2 | the descriptor on a statement, after the fact |
| a device or connection that failed | 2 | the reader's side of the connection |
| shipped but unable to work | 11 of 24 | 45.8% |
sample I - the cost of optimising the wrong object
experiments reviewed = 40
shipped = 24 -> 60.0%
stopped early = 9 -> 22.5%
null, nothing moved = 7 -> 17.5%
of the 24 shipped, aimed at a step whose cause was elsewhere:
11 -> 11 / 24 = 45.8%
at 1,612.00 of engineering each:
11 x 1,612.00 = 17,732.00 spent in a year on changes that
could not move the number they were measured on
and each of the 11 also consumed a slot a reachable change
could have used, against a programme of 120 shipments a year.
The limits that are choices, not accidents
Two further limits are structural. The loop cannot measure the readers it has excluded, and it cannot see the harm it causes after the deposit, because the metric stops at the step.
Sample I - the loop's hard edges
| Limit | Why | On the samples |
|---|
| the unmeasured readers | they refused, so they are not in the denominator | 31.6% |
| the off-page cause | the decision is made by a third party | 45.8% of ships |
| the harm after the step | the metric ends at the deposit | +11.8% guardrail |
| the long-run effect | only a holdout sees the whole series | 3.6% realised |
| what remains inside the loop's reach | the screen, the order of the steps and the copy | - |
sample I - what a page can and cannot do, stated in one list
inside the page's reach:
where a step sits, what the button says, how many fields are
asked for before the step, what the page shows while it waits
outside it:
whether a document is accepted, whether an issuer approves,
what the statement shows afterwards, whether the connection
holds, and whether the reader wanted to deposit at all
a funnel gap does not distinguish them, and on the samples
45.8% of shipped changes were spent on the second list.
The counts are invented. The desk's claim is structural: a recorded drop at a step is evidence about the reader's path and not a diagnosis of the screen, and an operator that treats it as a diagnosis will spend about half its effort on changes that cannot work. That is a limit of the method, not of any one operator.
What this means for a reader
- A page that keeps changing around a step you are stuck on is a sign the cause is elsewhere.
- If a deposit is declined, the message comes from your bank and no page rewrite changes it.
- If a check is running, the page cannot shorten it; only the documents can.
- If the interface changes constantly without helping, the loop is working on the wrong object.
- None of this excuses a change made to speed a decision up; it only explains why so many of them fail.
Read next