Same data, different answer? A new Nature study counts the gap

The study has looked into how much a finding depends on the particular person who happens to analyse the data.

ImageJohnér Bildbyrå AB

A new study in Nature has put a number on something the published literature usually leaves unmeasured: how much a finding depends on the particular person who happens to analyse the data. Across 100 social and behavioural science studies reanalysed by 457 independent researchers, only a third of the new effect-size estimates fell close to the original. Marcus Kubsch, recently appointed Associate Professor at the Department of Science and Mathematics Education (NMD) at Umeå University, contributed three of those reanalyses.

The project, known as Multi100 and led by Balazs Aczel and Barnabas Szaszi at Eötvös Loránd University in Budapest, Hungary, gave each of 100 published claims to at least five independent analysts. Each analyst received the data and the claim, but no instructions about how to test it. The question was simple: if competent researchers are left to make their own justifiable choices about how to analyse the same data, do they reach the same answer?

One third of the numbers, three quarters of the conclusions

Results often differed — at least when it came to the exact numbers. In just over a third of cases, the new estimates were close to the original, and with a wider margin this rose to just over half. However, the overall conclusions were more consistent: in nearly three out of four cases, researchers reached the same general conclusion as in the original study. The differences could not be explained by the analysts’ statistical expertise, and larger datasets did not make the results more consistent.

Marcus Kubsch, associate professor at the Department of Science and Mathematics Education.

ImageHans Karlsson

The implication is not that the published literature is broadly wrong but that a single analytical path through a dataset is not, by itself, enough to settle a question. The authors behind the study recommend that high-stakes findings be accompanied by structured robustness reports or multi-analyst checks.

A question that predated the project

Marcus Kubsch took three of the 100 datasets, built his own analysis from the claim and the raw data, and submitted code and results to the lead team. His interest in the subject actually predates the paper. As a master's student he read John Ioannidis's Why most published research findings are false, and the argument stayed with him. As a PhD student, while analysing data for the project he was working on, he found a quiet bug in an R script that had been inflating effect sizes.

"Once you have seen one of those errors, you start asking how often the same kind of thing happens and never gets caught” he says. “Multi100 is the systematic version of that question."

The work travelled with him geographically. He began the reanalyses as a postdoctoral research group leader, continued them through his time as assistant professor, and saw the paper appear shortly after taking up his Umeå University position in May 2026.

A global effort

Multi100 is part of the Systematizing Confidence in Open Research and Evidence (SCORE) programme, funded by DARPA and coordinated by the Center for Open Science. The wider SCORE collaboration drew on 865 researchers across the social and behavioural sciences.

"Several hundred people on every continent were willing to spend real hours on a problem whose only payoff is a clearer picture of how science actually works" Kubsch says. "That tells you something about the field as well as about the finding."