A Distinction That Generates More Confusion Than It Deserves
The difference between one-tailed and two-tailed tests is one of the most frequently asked questions in A/B testing, and ironically one of the least consequential for most practitioners. The distinction matters theoretically, but in practice, it rarely changes the decision you make. This guide will explain what the difference is, when it matters, and why you probably should not lose sleep over it.
What One-Tailed and Two-Tailed Mean
The terms refer to how you set up your hypothesis test, specifically what kind of difference you are looking for.
A two-tailed test asks: is there a difference between A and B in either direction? Your alternative hypothesis is that B is different from A. It could be better or worse. You are testing for any change at all.
A one-tailed test asks: is B better than A in a specific direction? Your alternative hypothesis is that B is better (or worse, depending on which tail you choose). You only care about changes in one direction.
Visually, imagine the bell curve of possible outcomes under the null hypothesis. A two-tailed test places your rejection region in both tails of the distribution (both extremes). A one-tailed test places the entire rejection region in just one tail.
The Practical Difference in Numbers
Because a one-tailed test concentrates all of the rejection region in one tail, it has more power to detect effects in that specific direction. In practical terms, a one-tailed test at the 5% significance level is equivalent to a two-tailed test at the 10% level, looking at just one side.
The conversion between them is simple arithmetic:
To convert a two-tailed p-value to one-tailed: divide by 2. A two-tailed p-value of 0.08 becomes a one-tailed p-value of 0.04.
To convert a one-tailed p-value to two-tailed: multiply by 2. A one-tailed p-value of 0.03 becomes a two-tailed p-value of 0.06.
This means that if you use a one-tailed test, you will find significance more easily for effects in the predicted direction, but you will completely ignore effects in the opposite direction.
When to Use Each Type
When a Two-Tailed Test Is Appropriate
Use a two-tailed test when you genuinely want to know if your variation is different from the control in either direction. This is the more conservative choice and is appropriate in most A/B testing contexts because:
You should care if your change makes things worse. In A/B testing, a variation could hurt performance. Knowing this is valuable because it tells you to definitely not implement the change.
It is more conservative. You are less likely to declare significance when the effect is marginal, which protects you from false positives.
It is the default in most statistical software and testing platforms. If you are not explicitly choosing a one-tailed test, you are almost certainly running a two-tailed test.
When a One-Tailed Test Is Appropriate
Use a one-tailed test when you only care about effects in one direction and have a strong theoretical reason to believe the effect can only go one way. For example:
You are testing whether adding a trust badge increases conversion. You have strong theoretical reasons to believe it can only help or have no effect, and you do not care about detecting harm. (Though in practice, even trust badges can hurt if they make visitors question why the badge is needed.)
You are testing a major site overhaul and your only question is whether the new design is better than the old one. If it is worse, you are keeping the old design regardless, so you do not need the extra precision of detecting negative effects.
However, the argument for one-tailed tests is weaker than it appears. In the second example, you would still want to know if the new design is significantly worse, because that is useful information about the direction of your design choices. A two-tailed test gives you this information; a one-tailed test throws it away.
Why Most Practitioners Should Not Worry About This
Here is the practical reality: the choice between one-tailed and two-tailed tests rarely changes the actual decision. If your two-tailed p-value is 0.03, the one-tailed p-value would be 0.015. Both are significant at the 0.05 level. If your two-tailed p-value is 0.15, the one-tailed p-value would be 0.075. Neither is significant at 0.05.
The only scenario where the choice matters is when your two-tailed p-value falls between 0.05 and 0.10. In this narrow range, a one-tailed test would call the result significant while a two-tailed test would not. But results in this range are borderline regardless, and making major business decisions based on borderline evidence is risky no matter which test you use.
If you are spending significant energy debating one-tailed versus two-tailed tests, you are optimizing the wrong thing. Your time is better spent ensuring adequate sample size, predetermining test duration, and running for complete business cycles. These factors have a much larger impact on the reliability of your results.
The Simple Rule for Converting Between Them
If you ever need to convert between one-tailed and two-tailed results, the rule is straightforward:
One-tailed to two-tailed: Multiply the significance threshold by 2. A one-tailed test at 5% is equivalent to a two-tailed test at 10%.
Two-tailed to one-tailed: Divide the significance threshold by 2. A two-tailed test at 5% is equivalent to a one-tailed test at 2.5%.
This works because in a two-tailed test at 5%, you are placing 2.5% in each tail. A one-tailed test at 2.5% puts all 2.5% in one tail, giving you the same critical value.
A Note on Intellectual Honesty
One important ethical consideration: you should never choose between one-tailed and two-tailed tests after seeing your results. This is a form of p-hacking. If your two-tailed test comes back with p = 0.07, you cannot retroactively switch to a one-tailed test to get p = 0.035 and declare significance.
The choice must be made before the test begins, as part of your test design. It should be based on your research question and hypothesis, not on what makes the results look better. This is part of the broader principle that all analysis decisions should be predetermined to prevent data-dependent decision-making.
Key Takeaways
One-tailed tests look for effects in a specific direction while two-tailed tests detect effects in both directions. Converting between them is simple arithmetic. The choice rarely changes the practical decision. Most A/B testers should default to two-tailed tests and focus their energy on more impactful methodological choices like sample size calculation and test duration. And the choice must always be made before the test begins, never after seeing results.