The most common advice for someone starting with data is to pick the tool: Excel, SQL, Python, a dashboard course. It’s advice that ignores the most important experimental result in the field.
In 2018 Raphael Silberzahn and Eric Uhlmann published, in Advances in Methods and Practices in Psychological Science, an experiment worth telling in full. They gave 29 independent research teams, 61 analysts in total, the same dataset and the same question: do football referees give more red cards to darker-skinned players?
The results:
- 20 teams found a statistically significant effect, 9 did not.
- Effect estimates ranged from about 0.89 (no effect, or slightly inverse) to 2.93 (strong effect).
- The teams used twenty-one different analytic strategies, all defensible.
- Even peer review among the teams didn’t make the answers converge.
Same data, same question, different answer. Not through incompetence: these were all qualified researchers, and each choice taken on its own was reasonable.
It isn’t an isolated case
In 2020 Rotem Botvinik-Nezer and colleagues replicated the same design on far more technical ground, publishing in Nature: 70 independent teams analysed the same functional MRI dataset, testing nine predefined hypotheses. No two teams used the same pipeline. For most hypotheses, the teams’ conclusions disagreed with each other.
The phenomenon has a name: researcher degrees of freedom. Joseph Simmons, Leif Nelson and Uri Simonsohn demonstrated it in 2011 in Psychological Science, in a paper provocatively titled False-Positive Psychology. They showed, with simulations and a real experiment, that if an analyst allows themselves flexible choices (when to stop collecting, which control variables to include, which observations to exclude) the probability of finding a “significant” result by pure chance rises well above the stated 5%. With four flexible decisions combined they got above 60%.
The risk in data analysis isn’t getting a calculation wrong. It’s making a sequence of individually reasonable choices that leads to a wrong conclusion.
The chart that lies to you, demonstrated
In 1973 Francis Anscombe constructed four datasets with virtually identical means, variances, correlation and regression line, and completely different appearances once plotted: one linear, one curved, one where a single outlier runs everything.
Justin Matejka and George Fitzmaurice extended the idea in 2017, in a paper presented at the CHI conference: they algorithmically constructed families of datasets with the same descriptive statistics to two decimal places, one of which draws a dinosaur. The message is Anscombe’s, with a demonstration that’s harder to ignore: summary statistics alone don’t tell you what shape the data has.
Then there’s the most famous case of a conclusion reversed by aggregation. In 1975 Peter Bickel, Eugene Hammel and William O’Connell published in Science an analysis of Berkeley graduate admissions: in aggregate, women were admitted at a lower rate, which looked like discrimination. Looking department by department, most departments admitted women at an equal or higher rate. Women applied in greater numbers to departments with lower admission rates for everyone. It’s Simpson’s paradox, and it isn’t a curiosity: it’s why a company-wide average can say the opposite of every one of its components.
So what does “being good with data” mean
From the three findings above you can extract a usable definition, and it has nothing to do with the tool:
| Skill | What it protects against |
|---|---|
| Defining the question before opening the file | Researcher degrees of freedom |
| Looking at the distribution, not just the mean | Anscombe’s quartet |
| Disaggregating by subgroup | Simpson’s paradox |
| Declaring analytic choices in advance | Unintentional p-hacking |
| Knowing where the data came from | Upstream measurement error |
None of these requires knowing how to code. All of them change the outcome more than any library.
On the presentation side there’s an experimental finding often ignored: William Cleveland and Robert McGill measured in 1984, in the Journal of the American Statistical Association, how accurately people read different graphical encodings. Position on a common scale, which is what a bar chart does, is read far more accurately than angles and areas, which is what a pie chart does. “Don’t use pie charts” isn’t a matter of taste: it’s a measured result.
Four habits before you touch the numbers
- Write the question before opening the dataset. One sentence, with the criterion for an answer. It’s the cheapest defence against degrees of freedom.
- Always look at the shape, not just the summary. A histogram and a scatterplot before any mean. Anscombe is from 1973 and is still ignored.
- Disaggregate. If the aggregate says something surprising, break it down by subgroup before believing it.
- Learn SQL before Python. Most real work is selecting, filtering and aggregating, and SQL does exactly that with a week of study.
- Learn to read a confidence interval. Knowing how uncertain an estimate is matters more than knowing how to compute it.
- Use AI as a syntax assistant, not a judgement one. It generates the query well; it doesn’t decide whether the question makes sense. I’ve written about this in how to use ChatGPT at work.
- Present the uncertainty, don’t hide it. Delivering a number without a margin is delivering a decision disguised as a measurement. The same applies to how you give feedback.
My take: write the question before you open the file
I think the most expensive mistake I see isn’t technical but sequential: you open the file, look at the numbers, and the question forms afterwards. From then on any analysis is a search for confirmation, even in good faith. The 29-team experiment says exactly this: when the road is open, the analyst walks it in a way consistent with their own expectations without noticing.
I also suspect introductory courses have the order wrong. They teach the tools first and, if there’s time left, interpretation. But the wrong tool produces a visible error; the wrong interpretation produces an error nobody notices and that ends up in a decision.
I have no study to prove it, but I believe it: the skill that pays most inside a company isn’t building the model, it’s explaining to decision makers why a number doesn’t answer the question being asked.
Frequently asked questions
Do I need to learn Python? Not to start. SQL and a good spreadsheet cover most corporate analytical work. Python matters when volume or repetition demands it.
Is Excel still enough? For analyses up to a few hundred thousand rows, yes. Excel’s limit isn’t power, it’s traceability: it’s hard to reconstruct which choices you made, which is precisely the 29-team problem.
How long does it take to be operational? For basic SQL and reading charts correctly, a few weeks of regular practice. For analytical judgement, much longer: it’s a skill built on mistakes made.
Does AI make learning analysis pointless? It speeds up the execution part and doesn’t touch the judgement part. If anything, by lowering the cost of producing analyses it raises the value of telling good ones apart. I’ve written about this in critical thinking.
Is it true that 80% of the time is data cleaning? It’s a figure repeated everywhere without a solid source. The professional surveys available give lower numbers, around 40-45%. It remains the longest part of the job, but the 80% is a figure with no parent.
Sources
- Silberzahn, R., Uhlmann, E. L., et al. (2018). Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356. The 29-team experiment.
- Botvinik-Nezer, R., et al. (2020). Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams. Nature, 582, 84-88. The 70-team replication.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359-1366. Researcher degrees of freedom.
- Anscombe, F. J. (1973). Graphs in Statistical Analysis. The American Statistician, 27(1), 17-21. The quartet.
- Matejka, J., & Fitzmaurice, G. (2017). Same Stats, Different Graphs. Proceedings of the CHI Conference on Human Factors in Computing Systems, 1290-1294. Algorithmic generation of datasets with identical statistics.
- Bickel, P. J., Hammel, E. A., & O’Connell, J. W. (1975). Sex Bias in Graduate Admissions: Data from Berkeley. Science, 187(4175), 398-404. Simpson’s paradox in its best-known case.
- Cleveland, W. S., & McGill, R. (1984). Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods. Journal of the American Statistical Association, 79(387), 531-554. Which graphical encodings are read most accurately.
- Gelman, A., & Loken, E. (2013). The Garden of Forking Paths. Working paper, Columbia University. Why the problem exists even without intent to cheat.
- Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124. The general picture on analytic flexibility and reliability.
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129-133. The official position on the use and abuse of significance.
This article was written following the process described in how we write. If you find a figure reported incorrectly, write to me: I’ll correct it and flag it at the bottom of the article.



