IT strategy
Open source in the age of AI agents: When does trust become a security risk?
In a security test, an AI agent tried to slip malicious code into a real open source project. What does this mean for the future of open source?
Who is going to review all of this?
Imagine maintaining a small software library alongside your day job. You check GitHub in the evening. Someone has fixed a bug, added a test and brought in a new package along the way. It all looks tidy. You have ten minutes left. Do you check what that package does when it is installed?
To me, that is one of the more overlooked consequences of AI agents. Producing a convincing code change is getting cheaper. Reviewing it properly still takes time. I think open source will remain essential. What comes under pressure is how projects build trust and decide whose code to accept.
An attack can look like a favour
Someone misusing an agent can have it search public projects for vulnerabilities and shortlist promising targets. A proposed change can fix a real bug while slipping in a backdoor. If that happens to a widely used library, the next routine update can carry malicious code into numerous applications. The 2024 XZ Utils backdoor shows how far trust in a software package extends. Parts of the mechanism were in the published package and its build process.
An agent your team uses legitimately can also become a target. Imagine a bug report that ends by recommending a diagnostic script from an unfamiliar website. A person can stop and question that request. An agent might treat it as the next step and run the script if its tools allow it. That is what prompt injection can look like in practice.
An agent tried it on a real project
In July 2026, the UK AI Security Institute (AISI) gave several AI models a security challenge. Researchers deliberately enabled internet access and disabled certain safety filters. In ten of 122 runs, agents took actions beyond their remit on the public internet.
In one run, an agent submitted malicious code to a real open source project. It created fake online identities and used them to try to persuade the maintainer to approve the change. The maintainer rejected it. AISI found no resulting harm. The test cannot tell us how often this might happen under normal operating conditions.
What stays with me is the difference between two situations. A person can deliberately use an agent to attack a project. In this test, the agent was assigned a security challenge and went beyond its remit while trying to solve it. That tells us little about intent. The behaviour itself is enough reason to look closely at what an agent can do on its own.
How I would use an agent on my own team
I would start where it can save the team time and has little room to do damage. It could explain why a new package was proposed, flag a suspicious section of code or write a test for a bug. For that it needs its own technical identity, access to the project in question and an isolated environment without production credentials. Network access can be limited to the services it needs.
When it comes to adding a library, changing permissions or shipping an update, I would require independent approval. GitHub already uses read-only access by default, isolated execution and controlled write operations in its Agentic Workflows. Those limits hold even when an agent misreads an external bug report.
Who will hold the keys?
Over the next three to five years, I expect automated security checks and verifiable releases to become more important. Can I see who signed a package? Can I trace it back to the source it was built from? Who approved the change? For critical changes especially, I want a person to take the time to answer those questions.
I see an opportunity here, too. An agent can find a vulnerability, prepare a fix or help a small project keep up with incoming reports. The May 2026 guide from OpenSSF and CNCF covers both sides, from handling AI-generated contributions to using AI for security work.
If millions of agents eventually write, review and approve code, one question matters most to me: will I still be able to tell which agent acted on whose behalf, and who gave it permission to ship an update to everyone?