Running Experiments
Running your A/B test correctly ensures that you gather reliable, actionable data.
Duration of Your A/B Test
How Long Should You Run Your Experiment?
Run your test for a duration that allows you to collect enough data to reach statistical significance. Typically, you should aim for at least 1-2 weeks, depending on your traffic volume.
Can You Stop Early?
Avoid stopping your experiments too early. Even if you see promising results after a few days, stopping before reaching significance can lead to unreliable conclusions. Always wait for the experiment to complete its cycle. The longer your experiment, the more reliable the results will be. Consider adjusting your significance parameters by contacting us contact us. By default, Gleef recommand the following metrics to be significant:
- Duration: 14 days
- Confidence: 95%
- Sample size: 1000 visitors per variation (including the baseline)
- Success events: 50 success events per variation (including the baseline)
Avoiding Bias in Your Test
How to Avoid Bias
- Timing: Run the test for more than a week to account for variations in traffic patterns (e.g., weekdays vs. weekends).
- Consistent Conditions: Ensure that no changes are made to the website during the test, as this could influence the results.
- Overlapping Tests: Avoid running multiple experiments on the same page simultaneously, as this can skew the results.
Gleef helps you manage multiple experiments on the same page. Learn how here.
Running Simultaneous Experiments
You can run multiple A/B tests across different pages simultaneously. However, to ensure the accuracy of each test, avoid running overlapping tests on the same page.