Understanding the mathematics behind Gleef
Confidence Level | Critical Value, ππ |
---|---|
99% | 2.575 |
95% | 2.33 |
90% | 1.645 |
80% | 1.28 |
baseline
, which represents the original wording being tested.
n
can be calculated using the following formula for two proportions:
n
= Z_alpha
Z_beta
p
p
delta
Where:
Z_alpha/2
is the Z-value
for the desired confidence level (e.g., 1.96 for 95% confidence).Z_beta
is the Z-value
for the desired power (e.g., 0.84 for 80% power).p
is the estimated overall conversion rate.delta
is the minimum detectable effect size (difference between conversion rates of the control and variant).Example Calculation for a 1% difference
p
) = 5% (0.05)Z-value
= 1.96)Z-value
= 0.84)delta
) = 1% (0.01)n
= n
= Example Calculation for a 10% difference
p
) = 5% (0.05)Z-value
= 1.96)Z-value
= 0.84)delta
) = 10% (0.1)n
= n
= p-value
and Z-value
for all experiments, as well as extensive data points we use to compute significance.