There is a company policy stating that you can’t authorize a wire transfer as an intern. But what policy applies when your AI agent with read permissions for your whole hard drive does that? According to Satya Nadella, there is no good answer for most organizations. In his article on X, he states that “frontier” models actually operate with real power within companies while their actions cannot be tracked down to a single line of code or piece of training data. His solution looks like traditional security applied to modern intelligence.
Also read: Want to pre-order the Temple wearable? Here are 5 things to know
It all begins with a confession on the part of Nadella. When software that was traditionally used was integrated into the world economy, it was possible to find out which code path was responsible for a certain behavior. It’s impossible with frontier software. It’s impossible to say which output was caused by some training dataset and by a certain set of weights. Still, companies are using these programs and giving them access to private data and rights to perform actions on their behalf.
He comes to a rather clear conclusion. You are still responsible, and the guarantee given by the provider of the model does not relieve you of this responsibility. It’s pretty significant coming from the CEO of a company that sells lots of AI.
The solution may seem like a cliche but is an intelligent repackaging of the issue. Nadella argues that frontier models, whether closed or open, should be approached as an insider risk. This is not because the model is malicious. But simply because any competent entity with access to the critical systems could commit errors or fall victim to compromise.
Also read: Best 55-inch Mini LED TVs under Rs 50000 in India to buy in this festive sale
This approach dodges the unresolved alignment problem. Rather than wondering about the morality of the system, the questions are about its reach and the possible consequences. Enterprises have years of experience dealing with such issues: define who you are, limit privileges, monitor, draw containment boundaries. Nadella seeks similar strategies for AI.
Transparency about the chain of thought is non-negotiable to him, and he dismisses the term “neuralese” as a justification for opacity. Yet it is not sufficient because currently there is no way to ensure consistent fidelity of model output. The process of using one model to check the output of another is useful, but it comes at a cost. You get a black box nested inside an orchestration layer which is itself a black box, supervised by yet another black box.
And the controls have to be outside the model. Here he draws an analogy from the security principles of the 1970s, which stipulate that a program must not be able to bypass or interfere with the permission enforcement mechanisms. Essentially, the model has to be separated from both its harness and its action space.
The foundation of the design by Nadella is observability, which contains seven principles. The principle of model diversity states that there should be no single model dependency on some result or self-verification of its workings. All significant actions should leave non-tampered evidence in human-readable form, and all outcomes should be re-created without help from the model. The whole process should be tested continually for failures, attacks, and other possible cases when something goes wrong, and not only for successful execution of some task. The organizations should establish control on access and actions independently, while validation should be independent of the intelligence.
Then comes containment, which assumes that the model has been compromised, and there should always be an emergency brake under control of an authorized person with the standard of behavior of advanced models. The last principle is incident disclosure, which should include information about what has broken, what controls have been broken, and how to avoid recurrence of an incident with changed agent’s runtime behavior.
For Indian businesses deploying agents in the sectors of banking, IT services and customer support, this checklist could serve as a good auditing tool. While all pilot studies start by asking what the agent can do, the more important question is who can stop it from doing whatever it is doing. In Nadella’s words, “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.”
Also read: Pixel 11 vs iPhone 17: Google’s AI ambitions meet Apple’s raw power