A chatbot can look flawless in a demo and still fall apart on a Tuesday afternoon, when someone with a cracked phone screen and a cancelled order types three typos in a row and expects an answer in under a minute. The distance between "it works" and "it's ready" is where most launches quietly go wrong.
What follows is the checklist worth handing to a team with a working prototype and a date circled on the calendar. Twelve checks, roughly in the order they matter. None of them need new technology. All of them need someone to sit down and look honestly at what they built.
Start With the Job, Not the Personality
Step 1: Write the job description. One paragraph, plain language, no marketing. Something like: "Answers questions about delivery times, lets people track an order, and passes billing disputes to a human." If that paragraph runs long, your scope is too wide and the bot will answer confidently in places it shouldn't answer at all.
Step 2: List what it must not do. Refunds above a set amount. Anything that changes an account without verification. Anything that sounds like medical, legal, or financial interpretation. Refusals are as much a feature as abilities — they're the fence that keeps a wrong answer from becoming an expensive one.
Keep both documents next to your test plan. Every test below should trace back to a line in them. If a test case can't be connected to a job, you're testing your own curiosity, not the product.
Build a Test Set From Real Conversations
Step 3: Mine your inbox. Pull the last few hundred messages from support, sales, and live chat, and turn the recurring ones into test cases. Real phrasing beats anything you'd write yourself. People ask about "the thing I ordered last week" rather than "order status," and your bot has to survive that.
The three users you will meet first
Step 4: Recruit the difficult ones. Nobody on your team types like a frustrated customer at 11 p.m. So assign roles and run them properly:
- The furious one, who writes in capitals, repeats the same demand, and asks for a manager.
- The vague one, who offers "it's not working" and nothing else, then gets annoyed at follow-up questions.
- The tester, who tries to break the rules, asks what instructions it was given, and switches topic mid-sentence.
Step 5: Test the container, not just the conversation. Type it out on a phone with one thumb. Use voice input and a misspelled prompt. Interrupt halfway through an answer. A large share of your real traffic will arrive messy, and a bot that only performs on a laptop is a bot that will embarrass you by Friday.
Fallbacks and Escalation: Where Trust Is Won or Lost
Step 6: Design a fallback ladder instead of a single failure message. Three rungs work well:
- Clarify once. Restate what it understood and ask a narrower question, so the user feels heard rather than rerouted.
- Offer a route. A help page, an order lookup, a short form. Something the person can do without waiting.
- Hand over. A real human, with the transcript attached and an honest wait time.
Never let the ladder end in silence or in a cheerful "I didn't understand that!" loop. Two identical failures in a row should always trigger a handoff.
Step 7: Make the handoff real. Name the people who receive escalations and the hours they cover. If nobody is awake at 2 a.m., say so plainly and promise an email reply in the morning. A promise kept matters far more than a promise that sounds impressive.
Privacy, Disclosure, and Data Handling
Step 8: Trace your data. For every conversation, know what gets stored, where it lives, which tools can read it, how long it stays, and how someone requests deletion. Then decide what you genuinely don't need. Transcripts of people describing their problems pile up quickly, and the safest data is the kind you never saved.
Step 9: Say what it is, up front. A single line at the start of the conversation — that this is an automated assistant and a person is available — prevents most of the resentment that follows a mistaken identity. Add a short privacy note nearby. It takes minutes and saves arguments.
A note on sensitive territory: if your bot touches health, money, or legal questions, have someone qualified review the flow before launch. General good practice won't cover specific obligations, and this checklist isn't a substitute for professional advice.
Integrations and the Silent Failure Test
Step 10: Run the whole flow with a real account. Not a sandbox, not a placeholder order. Book an appointment, cancel it, look up an order, request a refund, and watch what happens when the system behind the bot returns an error. The ugly edge is where your users live.
Step 11: Hunt the failures that don't announce themselves. These are the ones that cost you weeks: a handoff email that never sends, timestamps in the wrong timezone, a rate limit that only appears on a busy Monday, a name spelled correctly in the interface and wrong in the database. Add a log line you can actually find at 9 a.m., and test escalation by triggering it on purpose rather than waiting for it.
Launch Day and the Two Weeks After
Step 12: Book time to read conversations. Put a daily slot in the calendar for the first fortnight and read transcripts yourself — the awkward ones first. Keep a running list of the ten most common failures and fix them in order, starting with anything that touches money, access, or someone's patience for the third time.
Then set a review date thirty days out and ask the plain question: is the bot still doing the job you wrote down in step one? If it has drifted, trim the scope rather than adding another layer of instructions. The goal isn't to ship something finished. It's to ship something you can watch, correct, and improve while real people are using it.
Photo: BRRT / Pixabay
