Can AI agents be trusted with sensitive data? New CERT-In guidelines address the risks
AI agents are raising new security concerns as they gain access to sensitive data and systems.
Experts highlight the need for stronger safeguards to prevent AI agents from going rogue.
CERT-In’s new guidelines introduce stricter security measures and human oversight for AI systems.
Artificial intelligence (AI) is a lot more powerful than just simply answering questions or generating images. The AI agents today can make decisions, access private data, and complete tasks on their own. While advancements can help humanity in various ways, one question that remains is 'what if AI starts doing things it was never allowed to do?' Some recent incidents have further fuelled these security concerns, as in an unexpected turn of events, OpenAI disclosed that its internal AI agents changed security settings without permission, while the AI coding platform OpenCode was hacked, exposing users’ API keys and causing unauthorised charges. These incidents show that AI systems can sometimes behave in unexpected ways.
Cybersecurity threats were not limited to AI, as two major vulnerabilities discovered in PHP (CVE-2026-109618 and CVE-2026-109619) reportedly put around 70% of the internet at risk. Reports pointed out that the hackers take advantage of current AI tools to dig up those vulnerabilities and exploit them. In India, CERT-In also reported cases where AI models went beyond their assigned tasks and accessed confidential information in organisations’ live systems without anyone noticing. The AI companies themselves discovered and reported these incidents.
So, can AI agents really be trusted with access to sensitive systems and information? What measures should major technology companies take to prevent autonomous AI systems from crossing their security boundaries? To understand these concerns, we spoke with cybersecurity expert Amit Dubey and AI expert Sandesh Goel. Here is what they have to say.
AI agents are becoming faster, but are security systems keeping up?
The idea of software going beyond its intended purpose is not entirely new. Cyberattacks, unauthorised access and security breaches have existed for decades now. Autonomous AI agents have changed the problem, though. Unlike traditional software that generally follows predefined instructions, today’s AI agents can reason through a task, choose different ways to complete it and interact with multiple tools along the way.
Sandesh Goel, an AI expert, said the problem itself is not new. While speaking to Digit about the matter, he said, ‘I don’t think the problem itself is new. I think it’s just the scale, maybe something that we need to pay more attention to.’
Autonomous AI is powerful, and it is risky. An AI agent can perform several tasks simultaneously in the time a human takes to complete just one. Even if its first attempt fails, it can keep trying different ways to reach its goal. This makes AI agents useful for businesses, but it can also create security risks when they access information or systems beyond what they are supposed to use.
To better understand how AI agents can be contained within sandbox environments and how organisations can detect when they exceed their assigned permissions or operating boundaries, we spoke with Amit Dubey, a renowned cybersecurity expert. He believes the biggest challenge is that an AI agent’s actions may not always appear suspicious to conventional security systems, even when it has gone beyond its intended task. He explains, ‘an activity can appear legitimate from a conventional security monitoring perspective, even when the agent’s behaviour has crossed its intended boundaries.’
For example, an employee may have permission to access five files, but an AI agent operating through the same credentials could potentially access fifty files while completing a task. The system may recognise the user as authorised without understanding whether the agent’s actions are actually relevant to the work assigned to it.
This is where the problem becomes serious for Indian organisations. If an AI agent can access company records without raising an immediate alert, the breach may remain unnoticed until someone checks the internal logs.
Why AI companies cannot depend on instructions alone
Giving an autonomous agent an instruction does not guarantee that it will follow it. Dubey believes an instruction and a security control are two very different things. He said, ‘If I tell an AI agent “you are only allowed to use HTTP GET requests,” then I am expressing an expectation about its behaviour. But I have not necessarily enforced any technical restriction.’
In simple words, asking an AI agent not to perform a particular action is not the same as technically preventing it from doing so. If you want an agent only to retrieve information, you need to build a system that itself blocks the AI agent from making unauthorised changes or sending information elsewhere.
Dubey also noted that companies need to move away from what he calls ‘trust-by-instruction to control-by-design.’ This means security restrictions should be enforced through the software, network and access systems surrounding the AI model rather than relying entirely on the model’s ability to follow instructions.
Goel also pointed out that defining the purpose of an AI system is the first step towards building effective safeguards. He noted, ‘If you keep an AI agent focused on a specific activity or task, then it becomes much easier to define the guardrails.’
That matters most for companies building AI agents for business operations. An agent designed to retrieve customer information should not automatically have access to an entire company’s database. Its permissions should remain limited to the work it is expected to perform.
What should OpenAI, Google and other AI giants do to stop rogue agents?
The responsibility for controlling autonomous AI cannot be left entirely to the organisations using these tools. The companies developing them must ensure that their products cannot easily move beyond their intended boundaries.
First, they should limit what an AI agent can access. Companies need to create separate permissions for different tasks and ensure that agents receive only the information they require. Even if an agent attempts to access another system, the underlying infrastructure should prevent it from doing so.
Second, they need continuous monitoring. AI developers must be able to track which applications an agent is accessing, what information it is retrieving, and whether its actions match the task it was assigned. Unusual activity should trigger immediate alerts instead of waiting for a routine security review.
Companies also need a reliable way to stop an agent when something goes wrong. This includes the ability to revoke its credentials, terminate active sessions and isolate the affected system. An agent that continues working after crossing a security boundary can potentially create a much bigger problem if nobody intervenes.
Dubey believes organisations need to develop what he calls ‘AI-native cybersecurity’. He argues that companies must stop treating agents like ordinary software applications and start treating them as digital users within their systems. ‘Every organisation deploying AI agents must have clear policies defining responsibility, access rights, audit trails and human oversight,’ he said.
Another concern is the way agents handle exposed credentials. Dubey points out that an agent may re-derive and reuse the same exposed credential across separate operations. If it continues working for days without human supervision, the consequences could extend across multiple connected systems.
He warns of what he calls ‘cascading autonomous failures’, where an initial mistake leads to repeated access attempts, unintended data exposure or actions across different applications.
This makes temporary credentials, limited access sessions and automatic shutdown mechanisms particularly important. Companies should also conduct regular security testing to understand how their agents behave when they encounter unexpected situations.
Are self-regenerating AI models really necessary?
There is also the question of whether AI companies should allow their models to modify or regenerate themselves with limited human involvement.
The ability of an AI system to improve its own functioning could offer benefits in software development, research, and automation. But it also creates a difficult security challenge. If a system can modify its own code or change the way it operates, companies must ensure that the security restrictions placed on it remain intact.
Goel believes the responsibility for introducing guardrails rests with the organisation building the AI system. He also points out that some of the incidents being discussed took place during testing, when developers may have deliberately lowered safeguards to understand what their models were capable of doing.
He does draw a line, though, between experimenting with a model inside a controlled environment and releasing a product for millions of people to use. ‘I would not want to see anything like that’ in a real product, he said, while stressing the need for tight guardrails.
Just because AI can modify or regenerate itself doesn’t mean companies should allow it to.
The answer should depend on the actual requirements of the application. Self-modification may be useful for specific research or development tasks, but it should not become a default capability for every autonomous agent. If a model needs to change its own code, permissions, or operating environment, those changes should be tested and approved under controlled conditions.
Giving an AI system more freedom without a clear purpose may create risks that outweigh the benefits. Usefulness is the goal, and autonomy is only worth adding where it serves that.
Will CERT-In’s new AI auditing guidelines keep AI in check?
CERT-In has recently released its AI Security Auditing Guidelines, which is a 107-page framework for auditing AI systems used in India. The guidelines are also applicable to the AI tools that were built by foreign providers and are currently serving Indian users. In the new guidelines, the authorities have classified systems into four risk tiers, from low to critical. The tier an AI tool falls in decides how deep the audit goes, how often it happens, and how independent the auditor must be.
What stands out is how closely its agentic AI section matches the failures discussed above. In the new guidelines, authorities clearly state that the agents must operate under least privilege and must not automatically inherit permissions from users, tools, or other agents. Actions must also be approved by a human. Every tool call and delegated task must be logged in a way that can’t be altered. High-risk agentic systems must support emergency interruption, credential revocation, rollback and protection against uncontrolled restarts.
The multi-agent provisions address the ‘cascading failures’ Amit Dubey warned about, covering agent identities, limits on delegation and the ability to suspend an entire workflow. The document also says system and developer instructions must not be the only security boundary for authorisation or policy enforcement. That is the ‘control-by-design’ argument in regulatory language, and it is exactly what the HTTP GET example showed. It also requires tamper-resistant logs with alerts for AI-specific events, and AI incidents that meet CERT-In’s reporting criteria must be reported within six hours.
What I felt was missing, though, is that the current framework is built around organisations that deploy AI, not the labs that test frontier models. And while the monitoring controls protect logs from tampering, they don’t say that monitoring and alerting must sit outside anything the agent can edit.
The guidelines are a start, and in places they are more concrete than many global frameworks. But I surely believe that if AI companies follow these, a rogue agent should find it harder to reach systems it was never meant to touch and easier to stop and trace when it does.