Skip to content
Go back

Data Analysis: 29 Teams, the Same Data, Different Answers

Illustration of charts and data being analyzed on a dashboard

The most common advice for someone starting with data is to pick the tool: Excel, SQL, Python, a dashboard course. It’s advice that ignores the most important experimental result in the field.

In 2018 Raphael Silberzahn and Eric Uhlmann published, in Advances in Methods and Practices in Psychological Science, an experiment worth telling in full. They gave 29 independent research teams, 61 analysts in total, the same dataset and the same question: do football referees give more red cards to darker-skinned players?

The results:

Same data, same question, different answer. Not through incompetence: these were all qualified researchers, and each choice taken on its own was reasonable.

It isn’t an isolated case

In 2020 Rotem Botvinik-Nezer and colleagues replicated the same design on far more technical ground, publishing in Nature: 70 independent teams analysed the same functional MRI dataset, testing nine predefined hypotheses. No two teams used the same pipeline. For most hypotheses, the teams’ conclusions disagreed with each other.

The phenomenon has a name: researcher degrees of freedom. Joseph Simmons, Leif Nelson and Uri Simonsohn demonstrated it in 2011 in Psychological Science, in a paper provocatively titled False-Positive Psychology. They showed, with simulations and a real experiment, that if an analyst allows themselves flexible choices (when to stop collecting, which control variables to include, which observations to exclude) the probability of finding a “significant” result by pure chance rises well above the stated 5%. With four flexible decisions combined they got above 60%.

The risk in data analysis isn’t getting a calculation wrong. It’s making a sequence of individually reasonable choices that leads to a wrong conclusion.

The chart that lies to you, demonstrated

In 1973 Francis Anscombe constructed four datasets with virtually identical means, variances, correlation and regression line, and completely different appearances once plotted: one linear, one curved, one where a single outlier runs everything.

Justin Matejka and George Fitzmaurice extended the idea in 2017, in a paper presented at the CHI conference: they algorithmically constructed families of datasets with the same descriptive statistics to two decimal places, one of which draws a dinosaur. The message is Anscombe’s, with a demonstration that’s harder to ignore: summary statistics alone don’t tell you what shape the data has.

Then there’s the most famous case of a conclusion reversed by aggregation. In 1975 Peter Bickel, Eugene Hammel and William O’Connell published in Science an analysis of Berkeley graduate admissions: in aggregate, women were admitted at a lower rate, which looked like discrimination. Looking department by department, most departments admitted women at an equal or higher rate. Women applied in greater numbers to departments with lower admission rates for everyone. It’s Simpson’s paradox, and it isn’t a curiosity: it’s why a company-wide average can say the opposite of every one of its components.

So what does “being good with data” mean

From the three findings above you can extract a usable definition, and it has nothing to do with the tool:

SkillWhat it protects against
Defining the question before opening the fileResearcher degrees of freedom
Looking at the distribution, not just the meanAnscombe’s quartet
Disaggregating by subgroupSimpson’s paradox
Declaring analytic choices in advanceUnintentional p-hacking
Knowing where the data came fromUpstream measurement error

None of these requires knowing how to code. All of them change the outcome more than any library.

On the presentation side there’s an experimental finding often ignored: William Cleveland and Robert McGill measured in 1984, in the Journal of the American Statistical Association, how accurately people read different graphical encodings. Position on a common scale, which is what a bar chart does, is read far more accurately than angles and areas, which is what a pie chart does. “Don’t use pie charts” isn’t a matter of taste: it’s a measured result.

Four habits before you touch the numbers

  1. Write the question before opening the dataset. One sentence, with the criterion for an answer. It’s the cheapest defence against degrees of freedom.
  2. Always look at the shape, not just the summary. A histogram and a scatterplot before any mean. Anscombe is from 1973 and is still ignored.
  3. Disaggregate. If the aggregate says something surprising, break it down by subgroup before believing it.
  4. Learn SQL before Python. Most real work is selecting, filtering and aggregating, and SQL does exactly that with a week of study.
  5. Learn to read a confidence interval. Knowing how uncertain an estimate is matters more than knowing how to compute it.
  6. Use AI as a syntax assistant, not a judgement one. It generates the query well; it doesn’t decide whether the question makes sense. I’ve written about this in how to use ChatGPT at work.
  7. Present the uncertainty, don’t hide it. Delivering a number without a margin is delivering a decision disguised as a measurement. The same applies to how you give feedback.

My take: write the question before you open the file

I think the most expensive mistake I see isn’t technical but sequential: you open the file, look at the numbers, and the question forms afterwards. From then on any analysis is a search for confirmation, even in good faith. The 29-team experiment says exactly this: when the road is open, the analyst walks it in a way consistent with their own expectations without noticing.

I also suspect introductory courses have the order wrong. They teach the tools first and, if there’s time left, interpretation. But the wrong tool produces a visible error; the wrong interpretation produces an error nobody notices and that ends up in a decision.

I have no study to prove it, but I believe it: the skill that pays most inside a company isn’t building the model, it’s explaining to decision makers why a number doesn’t answer the question being asked.

Frequently asked questions

Do I need to learn Python? Not to start. SQL and a good spreadsheet cover most corporate analytical work. Python matters when volume or repetition demands it.

Is Excel still enough? For analyses up to a few hundred thousand rows, yes. Excel’s limit isn’t power, it’s traceability: it’s hard to reconstruct which choices you made, which is precisely the 29-team problem.

How long does it take to be operational? For basic SQL and reading charts correctly, a few weeks of regular practice. For analytical judgement, much longer: it’s a skill built on mistakes made.

Does AI make learning analysis pointless? It speeds up the execution part and doesn’t touch the judgement part. If anything, by lowering the cost of producing analyses it raises the value of telling good ones apart. I’ve written about this in critical thinking.

Is it true that 80% of the time is data cleaning? It’s a figure repeated everywhere without a solid source. The professional surveys available give lower numbers, around 40-45%. It remains the longest part of the job, but the 80% is a figure with no parent.

Sources

This article was written following the process described in how we write. If you find a figure reported incorrectly, write to me: I’ll correct it and flag it at the bottom of the article.

Founder of UpSkillLab, with a Master's in Digital Marketing from the University of Florence. Every article starts from public sources cited in the text and is drafted with the help of AI tools, under my review and editorial responsibility: no purchased content, nothing published without human checking. Read the editorial method or report an error.


Share this post:

See all articles in Skills & Competencies

Previous Post
Free AI Tools: The Real Cost Isn't in the Price
Next Post
Hard vs Soft Skills: The 85% Has No Source, and the Question Is Wrong