What is a multiverse analysis?
A multiverse analysis runs every reasonable version of a data analysis instead of only one, and then shows all the results together. If most versions give the same answer, you can trust it more. If the answer flips, you know it depends on the choices someone made.
5 minute read · Updated 13 October 2026
One dataset, many answers
Imagine you want to know if studying more gives higher exam scores. You have a table with study hours, sleep, attendance and exam scores for 400 students. Before you get an answer, you have to make many small decisions:
- Control variables: should you also take sleep or attendance into account?
- Outliers: one student wrote 70 study hours a week. Keep the row or remove it?
- Missing values: delete rows with empty cells, or fill them with the middle value?
- The model: a straight line, a curve, or something else?
Every option is reasonable. Two careful people can choose differently. With only four questions and two or three options each, there are already dozens of possible analyses. Scientists call this the garden of forking paths, and the freedom to choose is called researcher degrees of freedom.
A normal study shows you one path through the garden. A multiverse analysis walks every path and shows you where they lead.
Why it matters: the 29 teams study
In 2018, a group of researchers led by Raphael Silberzahn gave the same football dataset to 29 teams of scientists. Every team got the same question: do referees give more red cards to players with darker skin? The teams chose different methods. Twenty found a significant effect and nine did not, and the size of the effect changed a lot from team to team.
Nobody cheated. They just made different reasonable choices. This study showed that a single published result can hide how fragile it really is.
How a multiverse analysis works
- List the choices. Write down every decision where more than one option is reasonable.
- Build every combination. Each combination of options is one "universe".
- Run them all. Calculate the same result, for example the effect of X on Y, in every universe.
- Look at all results together. Usually with a specification curve, a chart with one dot per analysis.
- Judge the pattern. Do the results agree? Which choice moves them the most?
The idea was described by Sara Steegen and colleagues in 2016. A close cousin, the specification curve analysis, was described by Uri Simonsohn, Joseph Simmons and Leif Nelson in 2020.
Multiverse analysis with AI analysts
Listing every reasonable choice is hard work, and it takes coding skills to run hundreds of regressions. Forkwise does this for you. It sends your question to many AI analysts. Each analyst picks its own control variables, outlier rule and missing value rule, like the 29 teams did. Then Python runs every analysis, and a judge gives one of four verdicts:
| Verdict | What it means |
|---|---|
| Robust | Almost every analyst found the same direction and a real effect. |
| Fragile | They agree on the direction, but many could not show the effect is real. |
| Choice dependent | The answer changes with the choices, sometimes even the direction. |
| No effect | Almost no analyst found a real effect. |
The AI never calculates the numbers itself. It only decides how to analyze. You can read the exact rules on the how it works page.
What a multiverse analysis can and cannot tell you
It can show if a result survives many reasonable choices, and which choice changes it the most.
It cannot prove that X causes Y. If a hidden variable that is not in your data drives both, every analysis can agree and still be wrong. See the coffee example for how a missing control can fool you.
Sources
- Silberzahn, R. et al. (2018). Many analysts, one data set. Advances in Methods and Practices in Psychological Science.
- Steegen, S., Tuerlinckx, F., Gelman, A. and Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science.
- Simonsohn, U., Simmons, J. P. and Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour.
- Gelman, A. and Loken, E. (2014). The statistical crisis in science. American Scientist.