Skip to content
Go back

Project Management: The 0.5% That Succeed, and Why Your Estimates Are Wrong

Illustration of a project board with organized phases and tasks

The figure that opens almost every project management course is that around 70% of projects fail. It comes from the Standish Group’s CHAOS Report, published since 1994 and cited everywhere.

In 2010 Jan Eveleens and Chris Verhoef published an analysis of that report in IEEE Software, titled without much diplomacy the rise and fall of the CHAOS figures. Their main objection concerns definitions. Under Standish criteria a project is “successful” only if it meets initial estimates of time, cost and functionality. Which means a project delivered under budget is classified as challenged, and that improving your estimating ability lowers your score if your initial estimates were padded. They add that the sample is self-selected and unrepresentative, and that the raw data has never been made verifiable.

Which doesn’t mean projects are going fine. It means that to know how they’re going you have to look elsewhere.

The real number: 16,000 projects

Bent Flyvbjerg, who has studied large projects for thirty years, first at Oxford and then in Copenhagen, built the largest database available: over 16,000 projects across more than twenty sectors and over a hundred countries, from construction to IT to the Olympics.

The numbers he draws from it, published in 2023 in the book written with Dan Gardner, are these:

CriterionShare of projects meeting it
On budget~48%
On budget and on time~8.5%
On budget, on time and delivering expected benefits~0.5%

One in two hundred. That’s a number worth keeping in mind before deciding your own delay is a personal failing.

Why your estimates are wrong, and by how much

The phenomenon has a precise name, coined by Kahneman and Tversky: the planning fallacy. It’s the systematic tendency to underestimate the time and cost of a task even when you know similar tasks in the past ran over.

The cleanest demonstration is by Roger Buehler, Dale Griffin and Michael Ross, published in 1994 in the Journal of Personality and Social Psychology. They asked university students when they would finish their thesis. The average estimate was about 34 days. The average actual time was about 55. Only around 30% finished by the date they had given as most likely. And the most interesting part: when asked to construct a worst-case scenario, fewer than half finished by that date.

This isn’t generic optimism. It’s a specific distortion: when we plan, we build a scenario from the inside, the steps of this project, and ignore the statistics of how similar projects went.

At scale the numbers are documented. Flyvbjerg, Holm and Buhl analysed in 2002, in the Journal of the American Planning Association, 258 transport infrastructure projects across 20 countries, covering seventy years:

That last point is the most important and the most ignored: the problem isn’t solved by better software.

The one remedy with evidence behind it

The remedy Flyvbjerg proposes is called reference class forecasting, and it derives directly from Kahneman’s distinction between the inside view and the outside view. The procedure is trivial to describe and uncomfortable to apply:

  1. Identify the class of projects similar to yours.
  2. Find the real distribution of outcomes for that class (times, costs).
  3. Locate your project in that distribution, and correct your estimate accordingly.

The estimate no longer comes from the plan, it comes from historical data. The method has been adopted institutionally: the UK Department for Transport and the Treasury’s Green Book prescribe mandatory uplifts for “optimism bias” calibrated precisely on historical distributions.

Your project isn’t special. It’s a case in its class, and the class has a known distribution.

Does agile fix it?

Here it’s honest to say the evidence is weaker than the marketing suggests. Many of the most-cited agile-versus-waterfall comparisons come from the same reports we’ve just called into question.

The most solid independent quantitative work is by Pedro Serrador and Jeffrey Pinto, published in 2015 in the International Journal of Project Management, on a sample of 1,002 projects across various sectors and countries. The result: the use of agile and iterative approaches was positively correlated with both efficiency and stakeholder satisfaction. The effect was positive and statistically significant, but moderate, far from the order of magnitude that gets told in presentations.

There’s a theoretical reason iteration helps anyway: it shortens the time between decision and verification. It doesn’t eliminate the planning fallacy, but it reduces the cost of each individual estimating error.

Four rules for estimating and finishing

  1. Write down what “done” means before you start. A verifiable definition, not an adjective. Most projects that never end never had an ending condition.
  2. Use the outside view to estimate. How long did the last similar thing take? If you don’t know, the first job is measuring it, not planning better.
  3. Apply an explicit uplift, not a hidden margin. Saying “I’m adding 30% because the reference class says so” is defensible. Quietly padding individual tasks isn’t.
  4. Reduce batch size. Shipping something every two weeks doesn’t make estimates right, it makes errors small.
  5. Keep one priority list. If there are three “top” priorities, there are none. This is where project management meets time management.
  6. Make status visible without asking for updates. An up-to-date board eliminates half the meetings. It matters even more working remotely.
  7. Run retrospectives on data, not opinions. Estimated versus actual, for every block. That’s how you build your own reference class for next time.

My take: the most neglected skill is saying no

I think the most underrated part of the role is saying no, and that it’s also the least taught. Courses spend a lot of time on Gantt charts and responsibility matrices, and almost none on how you tell someone their request doesn’t fit this quarter.

I also suspect the obsession with tools is partly a form of deferral. Flyvbjerg’s seventy-years-without-improvement figure suggests the bottleneck was never technological. But it’s easier to evaluate software than to revisit how you make commitments.

Finally, a hypothesis I can’t verify: many projects don’t go wrong on execution. They go wrong because nobody had agreed on what the result was, and the ambiguity only surfaces at the end, when it costs the most.

Frequently asked questions

Is the 70% failure rate false? It’s a figure that depends on contested definitions and an unverifiable sample. Flyvbjerg’s data, on a much larger database, gives a different picture but not a more reassuring one.

Do I need a certification like PMP? In some sectors and countries it’s a formal requirement and opens doors. As a predictor of project success, the evidence is much weaker than the price suggests. I’ve written about this in which certifications are worth it.

How do I build a reference class if I have no historical data? Start collecting it now, even crudely: planned date, actual date, for every delivery. After ten items you already have more information than any intuitive estimate.

Does adding people speed up a late project? Fred Brooks argued no back in 1975, because of communication and training costs. It’s a conceptual argument rather than an experiment, but it has stood for fifty years.

Agile or waterfall? The question is as badly posed as the one about hard and soft skills. In Serrador and Pinto’s data the iterative approach helps, with a moderate effect. Having a definition of done and an honest estimate matters far more.

Sources

This article was written following the process described in how we write. If you find a figure reported incorrectly, write to me: I’ll correct it and flag it at the bottom of the article.

Founder of UpSkillLab, with a Master's in Digital Marketing from the University of Florence. Every article starts from public sources cited in the text and is drafted with the help of AI tools, under my review and editorial responsibility: no purchased content, nothing published without human checking. Read the editorial method or report an error.


Share this post:

See all articles in Skills & Competencies

Previous Post
Personal Branding: A Signal Anyone Can Copy Isn't a Signal
Next Post
Reskilling: What Changing Industry Really Costs (and How to Cut It)