Understanding the mathematics behind Gleef
Statistical significance is based on various parameters. Gleef uses an automatic calculation to determine whether your results are significant. For detailed insight on how this is done, refer to our statistics Q&A.
Continue running the experiment until you achieve statistical significance. Ending the experiment prematurely could hinder the reliability of your results. Remember, significance is closely related to the magnitude of the conversion difference between variations. For example, if you have a 10% conversion difference between two variations, your experiment may reach significance much faster than if the difference were only 1%.
As a reminder, the confidence interval is calculated as follows:
Confidence Level | Critical Value, 𝑍𝑐 |
---|---|
99% | 2.575 |
95% | 2.33 |
90% | 1.645 |
80% | 1.28 |
The console displays the critical Z value and the p-value, if needed.
The following default values are recommended for a fully significant experiment:
At any time, you can access and monitor the results of your experiment, observing the conversion rates of each variation, and the improvement compared to the baseline
, which represents the original wording being tested.
Hover over the conversion or growth metrics for detailed figures.
Traffic Volume:
Test Duration:
Multiple Variants:
Any website can run A/B tests; the key factor is how much time you allow the test to run without making changes to your website. It’s essential to ensure that your website has sufficient traffic to yield reliable and statistically significant results.
Key Considerations:
Sample Size:
Effect Size:
Confidence Level:
Statistical Formula for Sample Size
The sample size n
can be calculated using the following formula for two proportions:
n
= Z_alpha
Z_beta
p
p
delta
Where:
Z_alpha/2
is the Z-value
for the desired confidence level (e.g., 1.96 for 95% confidence).Z_beta
is the Z-value
for the desired power (e.g., 0.84 for 80% power).p
is the estimated overall conversion rate.delta
is the minimum detectable effect size (difference between conversion rates of the control and variant).Example Calculation for a 1% difference
statistical-significance Assume you have the following:
p
) = 5% (0.05)Z-value
= 1.96)Z-value
= 0.84)delta
) = 1% (0.01)Plugging these values into the formula:
n
=
n
=
Example Calculation for a 10% difference
statistical-significance Assume you have the following:
p
) = 5% (0.05)Z-value
= 1.96)Z-value
= 0.84)delta
) = 10% (0.1)Plugging these values into the formula:
n
=
n
=
High-traffic website:
Conclusion: This test can be completed in a few days.
Low-traffic website:
Conclusion: This test can be completed within hours.
By ensuring you have enough traffic and using the correct calculations, you can run A/B tests that provide reliable and actionable insights.
Currently, variations are equally distributed among visitors. For example, if you have three different variations in an experiment (including the baseline), one out of every three visitors will randomly see each variation.
You can contact the team at support@gleef.eu.
Currently, we do not provide built-in segmentation capabilities. However, you can conduct your own segmentation by directing visitors to different URLs and implementing experiments on each specific URL.
To conduct effective A/B testing, you should keep the text you’re experimenting with as the baseline. This serves as the reference point for all other variations tested.
The type of success event depends on the experiment you’re running:
Running an experiment on a Call to Action (CTA) you can define the success event as either:
Running an experiment on regular non-clickable text you can specify the relevant visit to a specific URL (a page view later in the funnel).
We strive to provide all the essential information directly on your dashboard.
Compare the performance metrics (e.g., conversion rates) of each variation against the control to determine the most effective copy. Gleef highlights the best-performing variation directly below the ‘baseline’, which represents the text you selected for the experiment.
p-value
and Z-value
for all experiments, as well as extensive data points we use to compute significance.While we currently do not offer a built-in option for downloading complete experiment data, we can send you a comprehensive data export file with all necessary details upon request. Simply email us at support@gleef.eu.
Gleef does not currently offer the capability to directly implement changes on your website. Therefore, you will need to follow your internal processes to submit the wording changes based on the best-performing variation.
Understanding the mathematics behind Gleef
Statistical significance is based on various parameters. Gleef uses an automatic calculation to determine whether your results are significant. For detailed insight on how this is done, refer to our statistics Q&A.
Continue running the experiment until you achieve statistical significance. Ending the experiment prematurely could hinder the reliability of your results. Remember, significance is closely related to the magnitude of the conversion difference between variations. For example, if you have a 10% conversion difference between two variations, your experiment may reach significance much faster than if the difference were only 1%.
As a reminder, the confidence interval is calculated as follows:
Confidence Level | Critical Value, 𝑍𝑐 |
---|---|
99% | 2.575 |
95% | 2.33 |
90% | 1.645 |
80% | 1.28 |
The console displays the critical Z value and the p-value, if needed.
The following default values are recommended for a fully significant experiment:
At any time, you can access and monitor the results of your experiment, observing the conversion rates of each variation, and the improvement compared to the baseline
, which represents the original wording being tested.
Hover over the conversion or growth metrics for detailed figures.
Traffic Volume:
Test Duration:
Multiple Variants:
Any website can run A/B tests; the key factor is how much time you allow the test to run without making changes to your website. It’s essential to ensure that your website has sufficient traffic to yield reliable and statistically significant results.
Key Considerations:
Sample Size:
Effect Size:
Confidence Level:
Statistical Formula for Sample Size
The sample size n
can be calculated using the following formula for two proportions:
n
= Z_alpha
Z_beta
p
p
delta
Where:
Z_alpha/2
is the Z-value
for the desired confidence level (e.g., 1.96 for 95% confidence).Z_beta
is the Z-value
for the desired power (e.g., 0.84 for 80% power).p
is the estimated overall conversion rate.delta
is the minimum detectable effect size (difference between conversion rates of the control and variant).Example Calculation for a 1% difference
statistical-significance Assume you have the following:
p
) = 5% (0.05)Z-value
= 1.96)Z-value
= 0.84)delta
) = 1% (0.01)Plugging these values into the formula:
n
=
n
=
Example Calculation for a 10% difference
statistical-significance Assume you have the following:
p
) = 5% (0.05)Z-value
= 1.96)Z-value
= 0.84)delta
) = 10% (0.1)Plugging these values into the formula:
n
=
n
=
High-traffic website:
Conclusion: This test can be completed in a few days.
Low-traffic website:
Conclusion: This test can be completed within hours.
By ensuring you have enough traffic and using the correct calculations, you can run A/B tests that provide reliable and actionable insights.
Currently, variations are equally distributed among visitors. For example, if you have three different variations in an experiment (including the baseline), one out of every three visitors will randomly see each variation.
You can contact the team at support@gleef.eu.
Currently, we do not provide built-in segmentation capabilities. However, you can conduct your own segmentation by directing visitors to different URLs and implementing experiments on each specific URL.
To conduct effective A/B testing, you should keep the text you’re experimenting with as the baseline. This serves as the reference point for all other variations tested.
The type of success event depends on the experiment you’re running:
Running an experiment on a Call to Action (CTA) you can define the success event as either:
Running an experiment on regular non-clickable text you can specify the relevant visit to a specific URL (a page view later in the funnel).
We strive to provide all the essential information directly on your dashboard.
Compare the performance metrics (e.g., conversion rates) of each variation against the control to determine the most effective copy. Gleef highlights the best-performing variation directly below the ‘baseline’, which represents the text you selected for the experiment.
p-value
and Z-value
for all experiments, as well as extensive data points we use to compute significance.While we currently do not offer a built-in option for downloading complete experiment data, we can send you a comprehensive data export file with all necessary details upon request. Simply email us at support@gleef.eu.
Gleef does not currently offer the capability to directly implement changes on your website. Therefore, you will need to follow your internal processes to submit the wording changes based on the best-performing variation.