Designing AI Systems People Actually Trust

There is a bakery near where I used to live that opened at six and closed when the bread ran out, which was sometimes noon and sometimes four. The bread was extraordinary. I stopped going. Not because of the quality, which never wavered, but because I could never build the trip into my day. A friend of mine went for years. She had worked out the pattern — Tuesdays it lasted, Saturdays it never did — and for her the bakery was reliable. Same shop, same bread, two entirely different relationships, decided not by quality but by predictability.

I think about that bakery constantly when building systems with AI in them. We spend enormous effort raising the ceiling of what a system can produce, and comparatively little on making its behavior legible. Then we are surprised when people who were shown an impressive demo quietly stop using the thing.

People do not trust systems that are usually right. They trust systems that are predictable about being wrong.

Confidence without calibration

The specific failure that erodes trust fastest is uniform confidence. A system answers a question it knows cold, and a question it is guessing at, in exactly the same tone, at exactly the same length, with exactly the same composure. The user has no signal. They must verify everything or verify nothing, and since verifying everything defeats the purpose, most people eventually choose nothing and then get burned once, publicly, and never come back.

A human expert does not behave this way. Ask a good accountant something outside their specialty and you will hear it in their voice before you hear it in their words. They hedge. They say they would want to check. That hedging is not weakness; it is the mechanism by which you calibrate how much of your own attention to spend.

Designing that behavior in is unglamorous work. It means deciding, in the architecture, what the system does when it is operating outside its grounded material — and then making the difference visible in the interface, not just in the text. It means resisting the pressure to make every answer feel authoritative, because authority that is not earned is a liability the first time it is wrong.

Show the seams

The instinct in product design is to hide machinery. Usually correct. With AI systems it is often wrong, because the machinery is precisely what the user needs in order to judge the output.

When a system tells someone what their contract says, the useful design is not a clean paragraph of summary. It is a paragraph with the source visible beside it, so the person can spend two seconds confirming rather than ten minutes worrying. Attribution is not clutter. It is the thing that converts an interesting output into a usable one.

The same applies to scope. Systems that state plainly what they can and cannot see are trusted more than systems that let users discover the boundary by accident. A line that says this covers documents uploaded before March costs nothing and prevents the particular kind of betrayal that ends adoption.

Reversibility beats accuracy

If I could give one piece of design advice to teams building anything that acts rather than merely answers, it would be this: invest in undo before you invest in accuracy.

A system that is ninety percent accurate and fully reversible feels safe. Users will explore it, push it, learn it. A system that is ninety-eight percent accurate and irreversible feels dangerous, and people use dangerous tools tentatively, in the narrow band where they are certain, which is exactly the band where the tool adds least value.

Reversibility also changes the emotional character of a mistake. An error you can undo is a small annoyance and a piece of learning. The same error, permanent, is a story the person tells their colleagues, and that story does more damage to your product than a hundred good sessions repair.

Invest in undo before you invest in accuracy. Safety is what makes exploration possible.

Consistency is a feature

Users build mental models whether or not we intend them to. They learn that phrasing a request one way works better, that the third result is usually the good one, that it struggles with tables. These models are how people become fast with a tool.

Every silent change we ship invalidates some of them. This is the hidden cost of continuous improvement: an upgrade that raises average quality while changing behavior can make an experienced user slower for weeks. They have to relearn a colleague they thought they knew.

This does not argue against improving. It argues for treating behavioral change as a communicated event rather than a background detail. Tell people what changed. Tell them what to expect. The version note that engineers consider trivia is, to a daily user, the difference between a tool that evolves and a tool that becomes unreliable.

The slow accumulation

None of the above is a trick, and none of it produces a demo moment. Trust is not delivered; it is accumulated, in small consistent behaviors over months. It is also asymmetric — built slowly, lost in a single well-timed failure, and rebuilt more slowly than it was built the first time.

Which is why the design decisions that matter most are the ones about failure. What happens when the system does not know. What happens when it is wrong and the user notices. What happens when it is wrong and the user does not. A team that has good answers to those three questions will build something people rely on. A team that has only optimized the happy path will build something people admire briefly and abandon.

If you are building something people will need to rely on, the failure paths are worth designing first. I am glad to think through them with you.