How to tell a demo from a system

A demo is designed to produce one feeling: that is impressive. It succeeds more often than not, because the technology is genuinely impressive, and because demos are built to show the cases that work.

A system is designed to keep working when nobody is watching. It is much rarer, and it is the only thing worth paying for. Here are the questions we ask in any vendor pitch, including pitches for our own work, to tell the two apart.

What happens when it is wrong?

This is the question, and everything else is a variation of it. A demo answers by changing the subject to how good the output is when it is right. A system answers with a mechanism: here is the log, here is who gets notified, here is what stops it from being wrong at scale, here is who fixes it and by when.

If the answer is a redirect, you have seen a demo.

Show us the evaluation

Ask how they know it works. Not how they know it can work, which the demo just showed, but how they know it is working today, on your kind of input. A system has an evaluation harness: real cases, expected outcomes, a score, run automatically. Ask to see the score and the list of cases it gets wrong. A vendor who will not show the failures does not know them or does not want you to.

What does it cost per run, and who watches that?

Demos are free. Systems have a cost per run that moves when the model changes, when inputs grow, or when something starts retrying. A system has that number on a dashboard someone looks at. Ask what happened to the cost line the last time the vendor changed models, and who noticed.

Where does our data go, and can we leave?

Ask for the data export on day one. Ask what leaves your environment and what stays. Ask what you would have if you cancelled tomorrow: your prompts, your evaluation cases, your history. A system answers these plainly. A demo has not thought about the end.

Who owns it after launch?

On your side, who will be paged, who will make the next change, who will explain it to their successor? On theirs, what is the ongoing arrangement, and what does it include? A system comes with an operating model. A demo comes with a launch date.

What does it refuse to do?

An agent that can do anything is an agent that will eventually do something you did not want. Ask what the system will not do, what needs a human, and how those limits are enforced. The right answer involves the words "as code," not "by policy." A limit the system cannot exceed is a guardrail. A limit in a document is a hope.

What did the last one fail at?

Every real system has failed at something. Ask the vendor to describe a failure from a previous deployment and what they changed because of it. A system builder will have a story and a fix. A demo builder will have a testimonial.

What to do with the answers

You do not need all seven answered well. You need to notice which ones were answered with a mechanism and which with a redirect. Two or more redirects on the first two questions, and you are looking at a demo, however impressive.

This applies to us. Ask us these questions. The evaluation harness, the audit trail, the operating model, and the handoff plan are in every engagement precisely because we have been on the other side of this table and know which answers we would want to hear.

Launch week is the easy part. The questions are about everything after.