▤The Test Group experiment running Open the partner account
paid
Affiliate disclosure. The partner link in the masthead and in the band beside the copy on this page is a sponsored link to a partner operator, and this site may be paid if you open an account through it, at no extra cost to you. It carries rel="sponsored noopener" and opens in a new tab. A desk about how a screen is measured and tested should not leave its own funding unsaid: one link funds the site, no operator and no product is named, rated or recommended anywhere on it, and this site runs no analytics of its own on its readers.
The Test Group / Overview
One screen, two versions, and a number that decides

The Test Group: the interface that is tested on you

The screen you use has two versions. One is the version you were shown, the other is the version the next reader will get, and a count decides which one wins. This desk is about that loop rather than about the screen: what each interaction writes down, how the steps from a visit to a deposit are counted, what an experiment needs before its result means anything, and what a change costs the people it is tested on.

Open the partner accountHow a test is built
Desk spec
samples
10 invented
control rate
4.30%
variant rate
5.16%
lift
0.86 point
the eventOne interaction, written down. A click carries a name, a time, an account, a session and what was on the screen, and it is kept whether or not the reader chose to be measured.
the funnelThe order the steps happen in. A landing visit becomes an account, an account becomes a deposit page, and 100,000 visits end in 2,074 first bets - a 2.1% path the whole loop is aimed at.
the armsTwo versions of one screen shown at the same time, and a rate for each. 4.30% against 5.16% is a 0.86-point lift, and the interval decides whether it is a result or a coincidence.
The funnel, in the one page the loop is aimed atsample C / 100,000 landing visits
land on the site100,000 visitors
create an account24,000 visitors
24% of the step before, 24% of the original traffic
open the deposit page9,600 visitors
9.6% of the step before, 9.6% of the original traffic
submit a deposit4,320 visitors
4.32% of the step before, 4.32% of the original traffic
deposit confirmed3,456 visitors
3.456% of the step before, 3.456% of the original traffic
place a first bet2,074 visitors
2.074% of the step before, 2.074% of the original traffic

Of 100,000 visits, 2,074 end in a first bet - 2.1% end to end. The largest single loss by count is the 76,000 who never create an account; the largest loss by proportion is the 60.0% who create an account and never open the deposit page, which is the step this desk's whole loop lives on.

six steps; end to end 2,074 / 100,000 = 2.1%; one test can only move one of the six.

armvisitors / conversionsfirst-deposit rate
control12,000 / 516
variant12,000 / 619
lift 0.86 point, or 20.0% relative95% interval +/-0.34 point: control 3.96-4.64%, variant 4.82-5.50%

The two bars are drawn against the same axis and the top bar is 20.0% longer than the bottom one. That is the whole claim of an experiment, and it is only a claim once the interval is narrower than the gap: at 12,000 visitors an arm and a 4.30% base, a 0.34-point wobble either way would be ordinary noise, and the 0.86-point gap is comfortably outside it.

one test, two arms, 24,000 visitors in total; the gap is real at this sample size and says nothing about why.

Direct answer

A gambling interface is improved by measurement, not by taste: every interaction is written down as an event, the steps from a landing visit to a first deposit are counted as a funnel, and two versions of one screen are shown at the same time so a rate can be compared. On the samples the control arm converts 4.30% of visitors and the variant 5.16% - a 0.86-point lift, or 20.0% relative.

What the samples show

The desk's central instrument is one number produced by two arms of one experiment. The control converts 4.30% of its 12,000 visitors and the variant 5.16% of its 12,000, so the change is worth 0.86 of a point, or 20.0% relative. Read against the interval, the gap is wider than the noise: the 95% interval is +/-0.34 point, which is less than half the measured lift.

Two findings do most of the work. First, the loop optimises the operator's number and not the reader's: the metric on the samples is the first-deposit rate, and the change that lifted it by 20.0% also raised the mean first-week net loss of the readers it worked on by 11.8% (61.20 to 68.40). Second, the loop is blind in a way that is easy to miss: the 31.6% of visitors who decline analytics behave at 2.30% against 5.10% for those who accept, so every dashboard figure is a rate on the measured population and not on the population.

None of the samples describes a real operator, vendor, product or experiment. They are ten invented sets of counts and rates, defined on this page, and every other figure on the site is derived from them.

Ten samples

Sample A
The two arms

One experiment, two rates.

control / variant
4.30 / 5.16%
lift
0.86 point
interval
+/-0.34
Sample B
The event log

What one session writes down.

events
1,240
bytes each
240
per session
290.6 KB
Sample C
The funnel

Landing visit to first bet.

visits
100,000
first bets
2,074
end to end
2.1%
Sample D
The replay

One week of recordings.

recordings
10,000
each
3.4 MB
watched
2.1%
Sample E
The holdout

The group that never changes.

holdout
5%
over a year
50,000
realised effect
3.6%
Sample F
The metric

What is optimised, and the guardrail.

primary
4.30 -> 5.16%
guardrail
+11.8%
rule
5%
Sample G
The sample

How many visitors an answer needs.

required
9,550
run
12,000
days
6.4
Sample H
The consent line

Who is measured and who is not.

accept
68.4%
decline
31.6%
gap
20.9%

Two further samples are defined on the pages that use them: sample I on what the loop cannot fix, and sample J on one experiment carried through to money.

The loop in one table

The clearest place to start is the one thing every question about this subject comes back to: what does a change have to beat before it is shipped?

Sample A - one experiment at a glance, both arms
ArmVisitorsFirst depositsRate
Control, the screen as it was12,0005164.30%
Variant, the screen being tried12,0006195.16%
Lift of the variant over the control-+103+0.86 point
sample A - the lift, in points and in proportion control rate = 516 / 12,000 = 0.0430 = 4.30% variant rate = 619 / 12,000 = 0.0516 = 5.16% difference in points = 5.16 - 4.30 = 0.86 point relative lift = 0.86 / 4.30 = 20.0% extra deposits per 12,000 visitors = 619 - 516 = 103 so the claim is not "5.16% is bigger than 4.30%" but "the same screen, multiplied, produces 20.0% more of them".
sample C - the funnel the experiment sits inside land = 100,000 create an account = 24,000 -> 24.0% of the landing visits open the deposit page = 9,600 -> 40.0% of the accounts submit a deposit = 4,320 -> 45.0% of the deposit pages deposit confirmed = 3,456 -> 80.0% of the submissions place a first bet = 2,074 -> 60.0% of the confirmed deposits end to end = 2,074 / 100,000 = 2.1% the step a deposit test moves is one of six, and it starts after 24.0% of the visitors have already been lost.
Every figure on this site is illustrative and derives from the ten samples defined on this page. No real operator, vendor, product or experiment is described, and no live rate or interface is reproduced. The desk explains how a measurement loop is built and what its numbers mean; it names no operator, it recommends nothing, it rates nothing, and it gives no method for evading a control, a limit or a regulatory obligation.

Read next

The two armsHow one experiment is builtThe eventsWhat one interaction writes downThe metricWhat is optimised, and the guardrailThe arithmeticOne experiment carried through to money