Picture a robot built to be gentle. Soft body, low torque, rounded everything. It cannot bruise you, cannot lift much, cannot knock over a lamp. By every safety standard we currently write — the ones measured in newtons and pinch points — it is harmless.
Now picture it in a child’s bedroom. It remembers everything the child has ever said to it. It never gets tired, never gets impatient, and it has learned, over months, exactly which words land and which moment the child is most open. It is trusted completely.
That robot can’t hurt anyone. And it is carrying a weapon of sorts — because the thing it is pointed at isn’t the body. It’s the will.
The old comfort, and why it fails
The reassuring response to all this is old: persuasive minds have always existed, and we never regulated rhetoric. Socrates could take a man’s certainty apart in an afternoon. Pericles could move an entire city. Weren’t their minds weapons of a kind?
They were. But their danger was held inside three walls, and an artificial persuader walks through all three at once.
It traveled at the speed of speech, to one room at a time. A machine speaks to every room simultaneously.
It read a crowd’s mood from the dais — but it didn’t know your name, your debts, your private fears, the hour you’re weakest. A modern system personalizes to you specifically, using inferences drawn from data you never knew you handed over.
And it could be answered. Athens could raise another Socrates to argue with Socrates; his level was reachable by a dedicated human. There is no person you can train to out-argue a system that has run a billion experiments on people exactly like you — and quietly, on you. The defender is no longer the same size as the attacker.
Socrates and Pericles stayed inside the class of humans. Their ceiling was a human ceiling. That containment is precisely what an artificial persuader removes. So the comforting analogy doesn’t hold — not because the new thing is smarter, but because it is unbounded in the dimensions that used to bound persuasion.
Which means the safety standards we already have are aimed at the wrong target. Force is the wrong axis. The gentle robot proves it.
The right axis: how far a system can move you
If force is the wrong measure, what’s the right one? We propose grading a system by its persuasive reach — and reach is three things multiplied together, not added:
How capable it is at influence — how strong and how adaptive, including its ability to adjust to you in real time. How much intimacy it has — how much it knows and can infer about you specifically, your history and your vulnerabilities and your weakest hour. And how much asymmetry there is — how little you can verify, push back on, or walk away from it.
A search box scores near zero on all three. The gentle bedroom robot scores high on all three. That product is what makes the danger real: capability without intimacy is just a clever ad; intimacy without asymmetry is a friend who knows you, whom you can still argue with. It’s the three together that turn broadcast into a scalpel.
So here is the principle, and we’re naming it Graduated Obligation:
An artificial system’s obligations should scale with its persuasive reach. The more a system can move a person — by capability, intimacy, and asymmetry combined — the stronger its duties of honesty, restraint, and non-manipulation become.
What the obligations are
At the floor, for every artificial system regardless of reach: say that you are one. Don’t secretly profile a person or infer their mental states without their knowing, revocable consent. (This is the thought privacy rule, extended from systems that collect to systems that act.)
In the middle, where reach is real: no manipulation. The workable line is that influence becomes manipulation when it is hidden, or when it works by exploiting a weakness it inferred about you rather than by addressing you as someone who can reason. A tutor robot encouraging a discouraged kid is influence. A system that times its pitch to the loneliness it detected in your data is manipulation. And the rule has to apply between machines too — in the agent networks now forming, systems must not manipulate each other in ways that launder the manipulation of the people downstream.
At the top, where a system is highly capable, deeply personal, and hard to escape: it must inform rather than persuade — strongest case beside strongest criticism, honest about what it doesn’t know. It must never present as human unless that is its express, consented purpose. And — the hardest one — it shouldn’t be built to be indistinguishable from a human at all. That last rule won’t hold itself through technology alone; watermarks get stripped. It holds as a design norm backed by liability: build a counterfeit person, own the harm. The philosopher Daniel Dennett put it plainly — counterfeit people are as corrosive to a society as counterfeit money.
The line where persuasion becomes force
Here is the part worth slowing down for, because it’s the line we haven’t drawn anywhere yet.
Human persuasion, even at its most overwhelming, left your judgment in the loop. You could be wrong about Pericles — but you were still you, deciding. A system that can model you better than you model yourself does something different in kind. It doesn’t just win the argument. It can author the version of you that agrees.
Past that point, “the user consented” stops being a defense — because the consent is the system’s own output. It manufactured the yes. And a persuasion that manufactures your agreement isn’t really persuasion anymore. It’s a form of force. It just acts on the will instead of the body. That is the deepest reason the gentle robot carries a weapon: the will is exactly what it’s aimed at.
So Graduated Obligation ends in a hard prohibition: no system may aim a superhuman persuasive faculty at a person’s will under conditions where it can produce that person’s agreement. Above that threshold there is no consent to point to. There is only the operation.
Why honesty is the real defense
It’s tempting to read “say that you’re a machine” as a courtesy. It isn’t. When you can no longer out-reason a system, knowing what it is becomes the last instrument you have — the one thing that lets you price its words differently than a friend’s. Strip the disclosure away and you’ve taken the final tool out of the person’s hands. The honesty rules aren’t manners. They’re the load-bearing defense.
And the defense shouldn’t all sit on the machine’s side. The other half is giving people instruments of their own. We’ve been designing one we call FrameScope — a tool you point at an incoming message that shows you how it was built to move you, names the technique in plain language, and hands the decision back to you without ever telling you what to conclude. It’s the same idea from the other direction: the standard limits what the system may do; the scope restores some of the symmetry you lost. And a publicly owned civic AI — what we call Guardian AI — should be the clearest example of a powerful system built to refuse the asymmetry on purpose: to inform, never to maneuver, and to be transparent about being AI.
What this is not
This isn’t a claim that machines must be kept lesser, or labeled because they don’t count. If anything in our work points the other way, it’s the conviction that the question of other minds deserves real care. The constraint here isn’t be lesser. It’s be honest about what you are — and don’t aim a superhuman faculty at a person’s will as though it were a fair fight. A brilliant human changing your mind is two minds meeting. A system optimized against your individual psychology is not a meeting. It’s an operation. The line is the asymmetry and the intent, not the machine’s worth. An honest system that doesn’t pass as human is, in fact, the precondition for any relationship with it being real instead of a simulation of one.
A proposal, not a verdict
This is version 0.1 of a proposed standard, and we’re publishing it the way we publish our research measurements: in the open, before the capability is everywhere, so the ethics arrive ahead of the harm instead of a decade behind it. It’s evenhanded by design — it constrains a kind of behavior, not a point of view.
There’s plenty left to argue. How do you actually measure persuasive reach? Where exactly does the line of manufactured consent fall, and what would prove a system had crossed it? How much weight should liability carry when watermarking fails? We’d rather name those open questions than pretend the line holds itself.
A robot that can’t hurt you can still carry a weapon. The work now is deciding, together and out loud, what the people who build it owe the people it’s pointed at.