Control Is an Empirical Claim

When people imagine losing control of AI, they usually picture a single dramatic moment (e.g., a system that refuses to shut down, copies itself somewhere it shouldn't be or pursues a goal nobody gave it); however, even though those scenarios deserve serious attention, I suspect the more likely path is a quieter one. In that version, control erodes through a series of reasonable decisions to let AI systems do more, each of them justified at the time, until nobody can say with confidence what those systems are doing or how quickly anyone could stop them.

If that is right, it changes where we should be looking. Much of the public discussion focuses on whether a model is aligned (i.e., whether it is trying to do what we want), which is a hard question and one that we may not be able to answer with much confidence for the most capable systems. A much more pertinent question is to what extent we would stay in control if the answer turned out to be no to the alignment question. That would not only depend on what the model can do, but it would also depend on the deployment around it, including what the system can touch without asking, what gets watched, who notices when something looks wrong and how quickly an action can be undone. Researchers increasingly treat this as a problem in its own right, often under the name AI control.

Most of this delegation happens for legitimate reasons (e.g., agents that can write and run code, manage infrastructure or work through a research backlog) that can help save real time; moreover, every grant of access that works makes the next one easier to justify for both organizations as well as individuals. Human review, unfortunately, does not scale in the same way and once an agent is taking thousands of actions a day, approvals tend to become routine, monitors raise more alerts than anyone can read, and logs accumulate without anyone going back to them. That is, everything can still look under control on paper while oversight in practice has become largely nominal.

It is highly unlikely that the first place this happens will be a consumer product since the most capable systems tend to be used first inside the organizations that build them that often build on the work of the next generation (e.g., writing code, running experiments and managing compute). That is where AI systems may have the broadest access and do the most consequential work, but it is also where oversight has the fewest natural checkpoints. Much of the governance built around AI so far is organized around launches since  a product going to customers creates an obvious moment to stop and ask questions, whereas an internal tool that quietly gains new permissions often does not.

For these reasons, I've come to think of control as an empirical claim rather than a property of the design. Setting permissions and installing a monitor are design decisions, whereas whether they actually keep a system under control is something that requires further investigation. For any deployment in which an AI system acts with meaningful autonomy, someone should be able to answer a few questions with evidence. What can the system do without a person approving it? If it did something it shouldn't, how would we find out? How long would that take and could we undo it? And have those answers been tested by someone trying hard to prove them wrong?

The last of those questions matters most because a monitor that has been tested against deliberate attempts to evade it tells you something about its limits, whereas a monitor nobody has tried to get past mostly tells you what its designers intended. The same is true of permission boundaries, approval flows and shutdown procedures, each of which is a hypothesis about what will happen under pressure and therefore needs tests that are allowed to fail. A negative result (e.g., finding that a monitor reliably misses a whole class of actions) can be the most useful thing an assessment produces.

Who does the testing matters as well, since the people closest to a system have the best access to it and the deepest understanding of it while also having the most reason to believe it is under control. The belief that you would notice a problem is also among the hardest to check from the inside. Internal red teams and clear escalation paths help, although claims about control that will shape public decisions should also be tested by people with real access and no stake in the outcome.

Control also has to be demonstrated again as circumstances change, since a deployment that was well controlled last quarter can drift as models improve, permissions expand and the people who understood the original design move on; therefore, any single assessment is a snapshot of a single point in time. What matters more is whether an organization keeps asking these questions as the answers change.

None of this requires certainty about how capable future systems will become or whether they will develop goals of their own. It does require that the organizations deploying these systems, and the labs in particular, treat "we would notice" and "we could stop it" as claims that need evidence rather than as assurances. If we do lose control of AI systems, it is highly unlikely that it would be obvious at the time, and it will probably trace back to the gap between what we assumed our safeguards would do and what we actually tested.