Data & Statistics

Statistical significance is based on various parameters. Gleef uses an automatic calculation to determine whether your results are significant. For detailed insight on how this is done, refer to our statistics Q&A.

Continue running the experiment until you achieve statistical significance. Ending the experiment prematurely could hinder the reliability of your results. Remember, significance is closely related to the magnitude of the conversion difference between variations. For example, if you have a 10% conversion difference between two variations, your experiment may reach significance much faster than if the difference were only 1%.

For the moment, Gleef does not offer the possibility of managing the significance parameters of experiments. This is automatically triggered when the 95% confidence level is reached (p-value of 0.05). You can see the critical Z value directly in the console when you open the dashboard on a specific experiment. As a reminder, the values are as follows:

The console displays the critical Z value and the p-value.

Value to look at on top of statistical significance, before taking any decision on your wording:

• At least 5,000 unique visitors per variation;
• To run the test for at least 14 days (two business cycles);
• Achieve 250 conversions.

​

Conversion & growth

At any time, you can access and monitor the results of your experiment, observing the conversion rates of each variation, and the improvement compared to the baseline, which represents the original wording being tested.

​

Common Questions About Statistics

​

What do I need to consider for a successful test?

1. Traffic Volume:

   • High-traffic websites can quickly achieve the necessary sample sizes.
   • Low-traffic websites may need to run tests for extended periods to gather sufficient data.

2. Test Duration:

   • Ensure the test runs long enough to collect the necessary sample size for accurate results.
   • Account for seasonal or time-based variations in traffic and user behavior.

3. Multiple Variants:

   • If testing more than two variants, make sure to adjust the sample size calculation accordingly to maintain statistical validity.

​

Do I have enough traffic on my website to run A/B tests?

Any website can run A/B tests; the key factor is how much time you allow the test to run without making changes to your website. It’s essential to ensure that your website has sufficient traffic to yield reliable and statistically significant results.

Key Considerations:

1. Sample Size:

   • The number of visitors required to detect a meaningful difference between variants.
   • Calculated using statistical formulas based on your desired confidence level and effect size.

2. Effect Size:

   • The minimum detectable change in conversion rate or other metrics.
   • Smaller effect sizes necessitate larger sample sizes.

3. Confidence Level:

   • The probability that the test results are not due to random chance.
   • Commonly used confidence levels are 95% and 99%.

Statistical Formula for Sample Size The sample size n can be calculated using the following formula for two proportions: n = $(2 * ($Z_alpha$/2 +$Z_beta$)^2 *$p$* (1 -$p$)) / ($delta$^2)$

Where:

• Z_alpha/2 is the Z-value for the desired confidence level (e.g., 1.96 for 95% confidence).
• Z_beta is the Z-value for the desired power (e.g., 0.84 for 80% power).
• p is the estimated overall conversion rate.
• delta is the minimum detectable effect size (difference between conversion rates of the control and variant).

​

What are some common examples of traffic and A/B testing statistics?

1. High-traffic website:

   • Daily visitors: 50,000
   • Estimated conversion rate: 5%
   • Desired effect size: 1%
   • Sample size per variant: 7,448

   Conclusion: This test can be completed in a few days.

2. Low-traffic website:

   • Daily visitors: 500
   • Estimated conversion rate: 5%
   • Desired effect size: 10%
   • Sample size per variant: 75

   Conclusion: This test can be completed within hours.

By ensuring you have enough traffic and using the correct calculations, you can run A/B tests that provide reliable and actionable insights.

​

How are variations split among visitors?

Currently, variations are equally distributed among visitors. For example, if you have three different variations in an experiment (including the baseline), one out of every three visitors will randomly see each variation.

​

How can I change the significance level of my experiment?

At the moment, you cannot change the significance level of your experiment. It is fixed at 95%. This means the confidence level for determining the statistical significance of your results is set at 95%, ensuring robustness and reliability for your experiments.

​

How can I segment the traffic and determine who sees an experiment?

Currently, we do not provide built-in segmentation capabilities. However, you can conduct your own segmentation by directing visitors to different URLs and implementing experiments on each specific URL.

​

Can I remove the default text?

To conduct effective A/B testing, you should keep the text you’re experimenting with as the baseline. This serves as the reference point for all other variations tested.

​

What kinds of success events can be set up?

The type of success event depends on the experiment you’re running:

1. Running an experiment on a Call to Action (CTA) you can define the success event as either:

   1. A click on the specific item.
   2. A visit to a specific URL (a page view later in the funnel).

2. Running an experiment on regular non-clickable text you can specify the relevant visit to a specific URL (a page view later in the funnel).

​

Retrieving Data from Experiments

We strive to provide all the essential information directly on your dashboard.

​

How do I interpret the results of an A/B experiment?

Compare the performance metrics (e.g., conversion rates) of each variation against the control to determine the most effective text. Gleef highlights the best-performing variation directly below the ‘baseline’, which represents the text you selected for the experiment.

​

Does Gleef provide detailed statistical information on experiments?

We aim to deliver all necessary statistical data to provide a comprehensive overview of your experiments. Our intention is to keep Gleef smooth and easy to operate by avoiding information overload. However, if you need more detailed statistics, please contact us at support@gleef.eu, and we will be happy to provide additional information.

​

Is it possible to download full experiment data?

While we currently do not offer a built-in option for downloading complete experiment data, we can send you a comprehensive CSV file with all necessary details upon request. Simply email us at support@gleef.eu.

​

How do I use the results to optimize my website?

Gleef does not currently offer the capability to directly implement changes on your website. Therefore, you will need to follow your internal processes to submit the wording changes based on the best-performing variation.