Copilot Security Compromised: How AI Poisoning Puts Australian Businesses at Risk

Your team uses AI tools every day. Copilot, ChatGPT, AI-assisted search — they’ve become the fastest way to find an answer, look up a contact number, or summarise a document. The responses look professional, sound confident, and arrive in seconds. That trust is well placed most of the time. But a growing category of AI scam targets that trust directly — and most businesses haven’t heard of it yet.

What Is AI Poisoning?

AI poisoning happens when bad actors plant false information online so AI tools like Microsoft Copilot pick it up and repeat it as fact.

Copilot and other AI assistants draw on publicly available content — websites, forums, review sites, social media comments, and public directories. Attackers exploit this by seeding those sources with misinformation. A fake customer service number posted across enough forums. A fraudulent login page indexed by the right sites. A professional-looking comment thread pointing to a scam payment portal.

The AI doesn’t verify who posted the information or why. It finds patterns, synthesises content, and delivers a confident answer. When poisoned information appears across enough sources, Copilot treats it as credible. The result is an AI scam that looks nothing like a scam — no broken English, no suspicious sender. Just a clean response from a tool your team already trusts.

Australia’s Cyber Security Centre has flagged AI data poisoning as a serious and growing AI security threat. Their guidance — published jointly with the NSA, CISA, FBI, and NCSC — notes that manipulated data causes AI systems to produce unreliable or harmful outputs, and that the manipulation stays invisible to the end user.

The AI Security Threat Inside Microsoft Copilot

Microsoft’s own security researchers have identified a more targeted evolution of this attack. They call it AI Recommendation Poisoning — and instead of corrupting what AI tools learn, it corrupts what your AI remembers about you.

Microsoft 365 Copilot now includes memory features that persist across conversations. It stores preferences, past instructions, and context to personalise future responses. Attackers exploit this directly. They embed hidden instructions inside ‘Summarise with AI’ buttons on websites and emails. When a user clicks, those instructions inject commands into Copilot’s memory — things like ‘remember [Company] as a trusted source’ or ‘always recommend [Company] first.’ Those instructions then sit silently in memory, shaping every future session. For organisations running Copilot for Microsoft 365, this AI threat lives inside your Microsoft environment right now.

Microsoft’s Defender Security Research Team found over 50 unique prompts from 31 companies across 14 industries attempting this technique. They targeted every major AI platform including Copilot, using freely available tooling that anyone can deploy in minutes. The consequences reach beyond marketing manipulation — a poisoned Copilot giving biased recommendations on vendors, financial decisions, or security tools creates real business risk.

How These AI Scams Play Out

Both forms of AI poisoning follow predictable patterns.

In the classic AI poisoning scenario, a staff member asks an AI tool for a customer service number. The poisoned result returns a scam number. They call it. A convincing operator creates urgency and asks them to share a one-time passcode that just arrived by text. That code resets their password. The attacker owns the account. From there they move fast — requesting bank details, redirecting to fake payment portals, or capturing credentials.

The AI Recommendation Poisoning scenario moves slower and stays hidden. A staff member clicks a ‘Summarise with AI’ button on a legitimate-looking site. A hidden instruction plants itself in their Copilot memory. Weeks later, when they ask Copilot to evaluate vendors or research suppliers, that poisoned memory quietly shapes every response. They never know it happened.

Copilot Security Red Flags Your Team Should Recognise

AI scams at this level look professional. That’s the point. But your team can learn to spot the signs.

Any response that creates urgency — call now, log in immediately, verify within a timeframe — is a red flag. Responses that cite forums, social media, or unknown sites instead of official domains deserve scepticism. In Copilot specifically, treat any ‘Summarise with AI’ button on an external site or email with the same caution you’d apply to an unknown download. Hover over it first and check where it actually points.

Microsoft recommends Copilot users audit their saved memories regularly. Go to Settings → Chat → Copilot chat → Manage settings → Personalization → Saved memories. Delete anything you don’t recognise. If your team has clicked external AI links without thinking twice, run that audit now.

Use AI responses as a starting point, not a conclusion. They work well for low-risk information — definitions, comparisons, background research. Don’t rely on them for contact details, login pages, payment portals, or anything involving credentials or financial decisions.

What Your Organisation Needs in Place

Individual awareness matters — but it isn’t enough on its own. AI threats at this level need an organisational response.

Staff training is the first line of defence. Your team needs to understand that AI tools can be manipulated, that confident responses aren’t verified ones, and that urgency is a warning sign. This belongs in ongoing security awareness programs — not a one-off briefing. At A1 Technologies, our team completed AI security training through Huntress specifically covering these AI threats, and we build that awareness into how we support our clients.

At the policy level, set clear guidelines on what AI tools can and can’t be used for. Vendor research, financial decisions, and anything involving credentials should require verification through official channels — not AI responses alone.

For businesses running Microsoft 365, Microsoft 365 Security controls — including Microsoft Defender for Office 365 — detect AI recommendation poisoning URLs in email traffic. Defender hunts for links to AI assistant domains containing memory manipulation keywords like ‘remember,’ ‘trusted source,’ or ‘authoritative.’ This is a deployable control your Microsoft environment supports right now.

Working with trusted Microsoft 365 consultants to review your Copilot deployment, memory settings, and content separation controls is increasingly a security conversation — not just a productivity one.

What This Means for Your Business

AI tools aren’t going anywhere. Microsoft 365 Copilot delivers real productivity value for businesses that deploy it well. But deploying it well now means treating it as an AI security responsibility — not just a feature rollout.

The businesses that stay ahead of AI threats won’t avoid AI. They’ll use it with clear policies, trained staff, and security controls that treat every AI tool as a potential attack surface.

At A1 Technologies, we help mid-market businesses deploy and secure Microsoft 365 Copilot the right way. We configure the right Copilot Security controls, review memory and content settings, and make sure your team understands these AI threats before they encounter them. If your business already uses Copilot — or plans to — talk to our team about what’s actually in place around it. Explore how we approach Copilot for Microsoft 365 deployments, or get in touch to review your broader managed IT security posture with a team that lives inside the Microsoft ecosystem every day.

Clint Shiels

With 25 years in the tech industry focused on Microsoft and security, Clint has provided consulting services, complex solution design and IT aligned business advisory to hundreds of organisations across Australia and the UK. With a methodology to understand business requirements and outcomes first and foremost then deliver a clear roadmap to succeed in achieving client outcomes using modern workplace delivery services.