Skip to content
Go back

Generative AI: Experts Get Slower, and Don't Notice

Illustration of generative artificial intelligence creating content

In 2025 the research group METR ran an experiment that nobody expected to turn out the way it did. They took 16 open-source developers with years of experience on their own projects, assigned them 246 real tasks, and randomised which ones they would do with AI tools and which without.

Before starting, the developers estimated AI would make them about 24% faster. When actual times were measured, they had been 19% slower with AI than without. And here is the part that matters: after finishing, they still believed they had been about 20% faster.

They hadn’t just mispredicted. They had misremembered.

What a generative model actually does

Before the numbers, the mechanism, because it explains everything else.

A language model does not look information up and does not reason over an archive. It predicts the next piece of text given everything that came before. The architecture that makes this workable at scale is the transformer, described by Vaswani and colleagues in 2017 in a paper titled, with some cheek, Attention Is All You Need. The attention mechanism lets the model weigh which parts of the context matter for the next word.

In 2020 Kaplan and colleagues showed that these models’ performance follows regular scaling laws: it grows predictably with data, parameters and compute. That is the economic reason the industry spent what it spent.

From this follows the most misunderstood property. The model produces the most plausible continuation, not the true one. Emily Bender and colleagues, in 2021, used the image of stochastic parrots: systems that stitch together linguistic sequences seen in training with no access to meaning. The image has been contested, but the technical point holds. Plausibility and truth are different criteria, and the model optimises the first.

In 2025 a group of OpenAI researchers added a piece: hallucinations aren’t only an architectural limit, they are also incentivised by how models are evaluated. If a benchmark awards one point for a correct answer and zero for both an error and “I don’t know”, then guessing is the optimal strategy. We trained systems never to admit ignorance.

The real numbers, and why they differ so much

There are now several serious randomised experiments. All find positive effects. But the size varies in a way that says more than the numbers themselves.

StudySettingEffect
Noy & Zhang 2023 (Science)453 professionals, office writing-40% time, +18% quality
Peng et al. 202395 developers, GitHub Copilot-55.8% time on one specific task
Brynjolfsson et al. 2025 (QJE)5,179 customer support agents+14% issues resolved/hour
Dell’Acqua et al. 2023758 BCG consultants+12.2% tasks, +25.1% speed
METR 202516 expert developers, own projects-19% speed

How do a -55.8% and a -19% coexist in the same trade? By looking at who was in the room and what they had to do.

In Peng’s study the task was writing an HTTP server in JavaScript from scratch: a closed, standard problem with millions of similar examples in the training data. In the METR study the tasks were changes to enormous codebases the participants had known for years, full of implicit conventions the model could not know.

The first case is where AI excels. The second is where the cost of reviewing and correcting the output exceeds the time saved.

The real finding: the distribution compresses

The most solid result, and the least discussed, is about who gains.

Brynjolfsson, Li and Raymond followed 5,179 customer support agents through a staggered rollout, the closest thing to a natural experiment you can get inside a company. The average increase was 14% more issues resolved per hour. But disaggregated:

Noy and Zhang find the same pattern in writing: inequality between participants narrows, because those who started worse improve far more.

The authors’ interpretation is that the model encodes and diffuses the practices of the best workers. Those who already had them get nothing new. Those who didn’t get them for free.

This inverts the current narrative. Generative AI is not a multiplier for people who are good. It is a levelling-up device for people who aren’t yet, and close to nothing for people who already are.

The jagged frontier

Dell’Acqua and colleagues gave their experiment on 758 BCG consultants a useful name: the jagged technological frontier. AI is not good at “hard” tasks and bad at “easy” ones. It is excellent at some tasks and terrible at very adjacent ones, and the border between them is neither intuitive nor visible.

Their experiment had two task types. On the one inside the frontier (generating product ideas), those using AI produced more, faster, with quality ratings over 40% higher. On the one outside the frontier, a recommendation requiring quantitative data to be integrated with qualitative interviews in which the numbers were misleading, those using AI were about 19 percentage points less likely to reach the correct answer.

The authors also observed two usage strategies. Some participants acted as centaurs: dividing the work, keeping for themselves the parts where they knew they were better. Others acted as cyborgs: interweaving their work with the model’s at every step. Both beat passive use, which the authors describe as falling asleep at the wheel.

The problem isn’t that the model gets things wrong. It’s that it gets them wrong in exactly the tone it uses when it’s right.

And the economics?

At the macro level one dissenting voice is worth keeping in mind. Daron Acemoglu estimated in 2024 that, under reasonable assumptions about what share of tasks is actually exposed and how much is saved on each, AI’s effect on total factor productivity is on the order of 0.5-0.7% cumulative over ten years. A tenth or less of what investment-bank projections suggest.

The estimate is contested and Acemoglu himself calls it conservative. But it rests on a solid point: the gains measured in experiments concern isolated tasks, not whole processes. And whole processes contain waiting, approvals, meetings and rewrites that no model shortens.

Three habits for using it without being fooled

  1. Measure, don’t remember. The METR experiment shows that perceived time saved is unreliable even among competent people. For two weeks, log actual time on similar tasks, with and without.
  2. Map your frontier. Keep a concrete list of what the model does well for you and what it does badly. It’s the one skill specific to this tool, and it doesn’t transfer from anyone else.
  3. Use it where you’re weak, not where you’re strong. That is literally Brynjolfsson’s finding. On your core trade the gain will be near zero; on the spreadsheet you hate it will be enormous.
  4. Don’t delegate tasks where the data is misleading. That’s exactly where the BCG consultants lost 19 points. If the task requires noticing that a number is lying, keep it.
  5. Ask for the reasoning, not just the answer. Not because the model “reasons”, but because explicit reasoning gives you something to check. More on this in writing effective prompts.
  6. Always verify names, figures and citations. They are the category where the model is most fluent and least reliable, for the structural reason explained above.
  7. Keep your judgement in training. If you stop producing first drafts, you also stop recognising a bad first draft. The point is developed in critical thinking.

My view: judgement matters, not output

I think “will AI replace me” is the wrong question, and I’ve written about that in AI and work. The useful question, given this research, is a different one: does my value lie in production or in judgement? Because production is levelling out fast, and judgement isn’t.

I also suspect the bigger short-term risk isn’t unemployment but diffuse mediocrity: lots of acceptable output, produced by people who wouldn’t recognise a wrong one. The 19 points lost by the BCG consultants strikes me as the single most important number here, and almost nobody cites it.

I’ll close with a personal bet I can’t prove: in a few years the distinctive skill won’t be writing better prompts. It will be knowing when to close the tool.

Frequently asked questions

Does generative AI understand what it writes? In the sense we mean, no: it predicts plausible continuations. Whether that suffices to be called understanding is an open and legitimate debate, but in practice it’s safer to assume it doesn’t.

Why does it invent sources that don’t exist? Because a plausible citation and a real one have the same statistical shape. And because, as Kalai and colleagues show, evaluation systems have rewarded guessing over abstaining.

If I’m already good at my job, do I need it? The data says the gain will be small on your core competence and large on everything else. Which is still a good reason to use it.

These studies are already old, the models have improved. True, and worth keeping in mind. But the pattern the various studies find (gains concentrated among the less experienced, an unpredictable frontier, inflated perception) doesn’t depend on the model generation.

Is it worth learning to use it well? Yes, but with calibrated expectations. The gap between using it badly and using it well exists and is documented. The gap between using it well and not using it at all is smaller than the story suggests.

Sources

This article was written following the process described in how we write. If you find a figure reported wrongly, write to me: I’ll correct it and note the correction at the bottom of the article.

Founder of UpSkillLab, with a Master's in Digital Marketing from the University of Florence. Every article starts from public sources cited in the text and is drafted with the help of AI tools, under my review and editorial responsibility: no purchased content, nothing published without human checking. Read the editorial method or report an error.


Share this post:

See all articles in AI at Work

Previous Post
AI in Hiring: What Happens to Your Data (and Your Rights)
Next Post
AI at Work: People Who Use It Are Judged Lazier (There's a Study)