Command, not control

Almost every organisation building autonomous systems says that a human stays in the loop. The phrase has become so common that it has stopped carrying much information. It is worth being precise about what it can actually mean, because the versions differ enormously in their consequences.

Three arrangements, often confused

• Human in the loop. The system cannot take a significant action until a person approves it. Maximum control, and it does not scale — the operator becomes the bottleneck, which defeats much of the purpose.
• Human on the loop. The system acts within set limits while a person supervises continuously and can intervene or stop it at any moment.
• Human out of the loop. The system acts without a person available to intervene within the relevant timeframe.

These are frequently discussed as if they were points on a single dial of trust. They are not. They describe different distributions of responsibility, and the third one is where the serious ethical weight sits.

Why we say command rather than loop

Our own framing is deliberately different: a person is always in command. The distinction is not pedantry. 'In the loop' describes a mechanism — where a human sits relative to a decision cycle. 'In command' describes responsibility, which is the thing that actually matters.

Machines carry out tasks. People set the rules, define the limits, and remain answerable for the outcome.

A commander does not personally perform every action taken under their authority. They set the objective, define what is and is not permitted, retain the ability to intervene, and answer for the result. That relationship is well understood in military organisations, and it transfers to autonomous systems more usefully than a diagram about loops.

What it requires in engineering

Stating the principle is easy. Building for it changes the architecture in specific ways:

• Bounded discretion. The system operates freely inside limits a person has set, and stops at their edge rather than improvising past it.
• Legible behaviour. An operator must be able to anticipate what the system will do. A system that is brilliant but surprising cannot be commanded, only watched.
• Clean hand-off. Transferring control between autonomy and operator without ambiguity about who currently holds it. Confusion here is a known cause of accidents in every automated domain.
• Recognising its own limits. Knowing when a situation has moved outside what it should handle alone, and escalating rather than continuing.
• Reconstructable decisions. Recording enough that any action can be examined afterwards and explained. Accountability without a record is an aspiration, not a practice.

Scale is the hard case

The principle is straightforward with one platform and one operator. It gets harder when a single person directs many systems, which is precisely the situation autonomy is meant to enable.

Approving every action does not survive contact with numbers. Our position is that command operates at the level of the task and its boundaries: a person sets what is to be achieved and what must not happen, retains the ability to intervene or recall at any point, and remains responsible. The system's job is to make that supervision tractable — reporting clearly, escalating honestly, and never quietly expanding its own remit.

A commitment, not a feature

We treat this as a constraint on what we will build rather than an option a customer can configure away. It is stated in our responsible autonomy commitments and it shapes what we agree to work on.

There is a commercial cost to holding a line like this, and we would rather absorb it than discover later that we built something we cannot defend.