Skip to content
Go back

Microsoft Copilot: What the Controlled Trials Say (and Don't)

Illustration of the Copilot AI assistant integrated into work tools

Copilot demos always show the same moment: a meeting ends, the assistant produces a recap with the decisions taken and the actions assigned, everyone nods. It’s the right scene, for a reason the demos don’t mention. That specific task is also the only one where controlled experiments find a large, stable effect.

On the others, much less.

First: which Copilot

The name covers three different products that get confused constantly, and the confusion leads people to judge one using another’s data.

NameWhat it isWhere it lives
Copilot (consumer)Generic conversational assistantWebsite, app, Windows bar
Microsoft 365 CopilotIntegrated assistant with access to your company dataWord, Excel, Outlook, Teams, PowerPoint
GitHub CopilotCode completion and generationCode editor

The line that matters is the second: Microsoft 365 Copilot sees your organisation’s content (messages, files, calendar) through the enterprise search index. That’s where its real usefulness comes from, and also why its quality depends on how tidy your company’s data is, not only on the model.

The number Microsoft itself measured

In 2023 a Microsoft research group published a randomised study on knowledge workers using an early version of these tools. It’s worth reading precisely because it comes from the company selling the product: the results are more restrained than a marketing communication would suggest.

The picture that emerges:

The pattern makes sense. Summarising is a task where all the necessary material is already given, the correct answer is verifiable in seconds, and the tedious part is purely mechanical. Writing a new document has none of those three properties.

The code case: two opposite numbers

The sharpest data exists for GitHub Copilot, and it’s contradictory only in appearance.

In 2023 Peng and colleagues ran a randomised experiment on 95 developers asked to write an HTTP server in JavaScript. The Copilot group finished 55.8% faster. That’s the number everyone cites.

In 2025 the METR group measured 16 expert developers on 246 real tasks in their own open-source projects. With AI tools they were 19% slower, while believing they had been faster.

The difference isn’t in the tool’s quality, it’s in the task. The first was a closed, standard problem with millions of analogous examples in the training data. The second was changes to large codebases full of implicit conventions. A 2022 study by Vaithilingam and colleagues had already observed something consistent: participants with the assistant did not complete tasks significantly faster, but struggled considerably to understand and repair generated code when it was wrong.

The uncomfortable part: security

Here the numbers are specific and worth knowing, even if you don’t write code yourself, because they describe a general mechanism.

Hammond Pearce and colleagues published in 2022, at the IEEE Symposium on Security and Privacy, a study titled Asleep at the Keyboard. They built scenarios targeting known weaknesses and had the assistant complete the code. About 40% of the programs generated in security-relevant situations contained a vulnerability.

The following year Neil Perry and colleagues published at ACM CCS a controlled study with human participants. Those with the assistant wrote significantly less secure code and, at the same time, were more confident that they had written secure code than the group without it.

It’s the same pattern as the METR speed result: performance drops and confidence rises. It isn’t a defect of one product; it’s a predictable effect of any system producing plausible output in a uniform tone.

The risk isn’t that Copilot gets things wrong. It’s that it convinces you not to check.

Where to use it, and with what caution

  1. Use it above all for summaries. Meetings, long email threads, documents you have to read but not study. It’s the use with the best evidence, and as it happens also the least risky.
  2. Give it the context you have and it doesn’t. Who’s reading, what for, what has already been decided. General guidance is in writing effective prompts.
  3. Tidy your files before expecting magic. Microsoft 365 Copilot draws from the enterprise index: if the right document has a useless name in a randomly shared folder, it won’t find it.
  4. Don’t trust numbers in Excel without rechecking. Ask for the formula and verify one case by hand. That’s where a plausible error slips through most easily.
  5. On generated code, review the security and external-input parts first. That’s exactly where Pearce and Perry find the problems.
  6. Verify attributions in meeting recaps. “Who said what” and “who took which action” are the two fields where a wrong summary creates real organisational consequences.
  7. Measure your times, don’t trust the feeling. True in general, but here there’s a direct measurement saying the feeling can be wrong in sign.

My take: tidy data matters more than the model

I think Microsoft 365 Copilot’s value depends far more on the state of company data than on the model, and that this is why people’s reported experiences differ so wildly. In a company with tidy documents and sensible channels it works; in one with fifteen folders called “Final” it doesn’t. Neither of those things is about artificial intelligence.

I also find the “hours saved every week” promise the least verifiable piece of communication in the sector, because it almost always comes from self-assessment. The only numbers I would rely on come from randomised experiments, and those numbers are lower and more selective.

A personal conviction, to be taken as such: the competitive advantage won’t lie in having Copilot, which everyone will, but in knowing which three tasks to give it and which twenty not to.

Frequently asked questions

Does Copilot read my company documents? Microsoft 365 Copilot accesses what you already have access to, via the organisation’s search index. That’s why badly configured permissions become visible fast once it’s switched on.

Is it included in my Office subscription? The versions integrated into the applications are generally a separate paid licence, distinct from the free consumer Copilot. Terms change often: check your organisation’s contract.

How much time does it really save? A lot on summarisation tasks, little or nothing on creation tasks. That’s the result emerging from randomised studies, including Microsoft’s internal one.

Is Copilot better than a generic conversational assistant? Depends on the task. Copilot’s specific advantage is access to your data; if the task doesn’t need it, the advantage shrinks considerably.

Can I trust the code it generates? As you would a first draft written by someone who doesn’t know your context. The security studies cited above say review is not optional.

Sources

This article was written following the process described in how we write. If you find a figure reported wrongly, write to me: I’ll correct it and note the correction at the bottom of the article.

Founder of UpSkillLab, with a Master's in Digital Marketing from the University of Florence. Every article starts from public sources cited in the text and is drafted with the help of AI tools, under my review and editorial responsibility: no purchased content, nothing published without human checking. Read the editorial method or report an error.


Share this post:

See all articles in AI at Work

Previous Post
Jobs of the Future: 60% of Today's Didn't Exist in 1940
Next Post
Online Portfolio: It Works Because It's a Work Sample, Not Because It's Pretty