One experiment in money

A lift of 0.86 of a point is hard to argue with because it is small and abstract. Turned into people it is 860 extra first deposits a month; turned into money it is 29,412.00 of net revenue over 90 days for the operator and 6,192.00 of extra first-week loss for the readers it worked on. Same change, one arithmetic.

Desk spec

lift: 0.86 point
extra deposits a month: 860
revenue over 90 days: 29,412.00
reader cost a month: 6,192.00

the eventOne interaction, written down. A click carries a name, a time, an account, a session and what was on the screen, and it is kept whether or not the reader chose to be measured.

the funnelThe order the steps happen in. A landing visit becomes an account, an account becomes a deposit page, and 100,000 visits end in 2,074 first bets - a 2.1% path the whole loop is aimed at.

the armsTwo versions of one screen shown at the same time, and a rate for each. 4.30% against 5.16% is a 0.86-point lift, and the interval decides whether it is a result or a coincidence.

Direct answer

One experiment's 0.86-point lift, applied to 100,000 monthly visitors, produces 860 extra first deposits. At an average first deposit of 40.00 that is 34,400.00 of deposits, and at 34.20 of net revenue per depositor over 90 days it is 29,412.00 - the same change that costs the 860 readers 6,192.00 of extra first-week loss at 7.20 each.

From a point of rate to people and money

The first conversion is people, and it is the one that makes the number real. A lift of 0.86 of a point on 100,000 visitors is 860 readers who would not otherwise have deposited, and every figure after this one is a price multiplied by 860.

Sample J - the lift converted, step by step
Step	Value	How it is derived
monthly visitors	100,000	the traffic the experiment runs inside
lift	0.86 point	from sample A
extra first deposits	860	0.0086 x 100,000
average first deposit	40.00	the sample's own average
deposits moved	34,400.00	860 x 40.00
net revenue per depositor, 90 days	34.20	the sample's own margin, not the deposit
net revenue over 90 days	29,412.00	860 x 34.20

sample J - the lift, turned into money extra first deposits = 0.0086 x 100,000 = 860 deposits moved = 860 x 40.00 = 34,400.00 but a deposit is not revenue: it is the reader's money net revenue per depositor = 34.20 over 90 days net revenue from the change = 860 x 34.20 = 29,412.00 reader cost of the change = 7.20 more net loss in week one = 860 x 7.20 = 6,192.00 a month so one change moves 6,192.00 a month out of the readers it worked on and 29,412.00 of revenue into the operator's quarter, and both figures come from the same 0.86 point.

What the programme costs to run

The 29,412.00 is not free: it is produced by a machine with a staff, a stack and a slot for every test. The samples run 120 shipped experiments a year, and the measurement stack runs whether or not an experiment is in flight.

Sample J - the annual cost of the loop, and what it returns
Item	Value	Note
experiments shipped a year	120	from samples E and I
engineering per experiment	1,612.00	26 hours at 62.00
measurement stack a year	57,600.00	4,800.00 a month
total cost	251,040.00	120 x 1,612.00 + 57,600.00
changes worth a full 29,412.00	48	40.0% of the 120
revenue those changes produce	1,411,776.00	48 x 29,412.00
return on the cost	5.6 times	1,411,776.00 / 251,040.00

sample J - the cost, the return and the floor beneath it engineering = 120 x 26 hours x 62.00 = 193,440.00 measurement stack = 4,800.00 x 12 = 57,600.00 total = 193,440.00 + 57,600.00 = 251,040.00 revenue, if 40.0% of the 120 changes are worth a full 29,412.00: 48 x 29,412.00 = 1,411,776.00 return = 1,411,776.00 / 251,040.00 = 5.6x and the floor: from sample I, 11 of 24 shipped changes could not move their metric at all, so the honest yield is 1,411,776.00 x (13 / 24) = 764,715.00, a return of 3.0x the loop pays, and it pays less than the scoreboard says.

The settled experiment

At the end of the year the loop has to be read as a book rather than as a series of wins: what was run, what shipped, what survived, and what the group that never changed shows.

Samples E, F and I - the year, settled
Line	Value	Source
experiments shipped	120	sample E
individual lifts, added up	22.4%	the scoreboard
realised effect against the holdout	3.6%	the holdout, a year later
failed to reproduce	41 of 120	34.2%
aimed outside the page	11 of 24	45.8%, sample I
guardrail movement that shipped anyway	+11.8%	sample F
the book's honest summary	3.6%	a 6.2-fold gap between the scoreboard and the outcome

samples E and F - the year as one book revenue the scoreboard implies = 120 changes on a 22.4% path revenue the holdout shows = one 3.6% effect per depositing reader a month: holdout = 41.20 treated = 42.76 difference = 1.56 across 4,215 depositors a month = 6,575.40 of monthly difference the loop can actually be credited with and against that, the guardrail cost of the one change measured here: 6,192.00 a month of extra first-week loss so the measured effect and the measured cost are the same order of magnitude, which is the single most useful thing this arithmetic shows and the thing no dashboard reports.

Every figure here is invented and derives from samples A to I. No real operator's revenue, cost, staff or experiment count is described. The arithmetic is the point: a change that moves a rate by a fifth of a point and a change that moves a reader's first week by seven pounds are the same event, and only one of them is on the dashboard.

What to take from an arithmetic like this

A small lift is a large number of people as soon as it meets real traffic.
A deposit is not revenue, and a revenue figure is not a reader's outcome.
Every experiment has a cost, and the cost is paid whether the test wins or not.
A scoreboard of wins is not the same thing as a measured improvement.
The number the dashboard shows is the operator's, so read the guardrail beside it or assume there is none.

One experiment in money

From a point of rate to people and money

What the programme costs to run

The settled experiment

Read next