We route 70% of support tickets through AI drafts now. Here is the setup

We run Replybase on our own docs plus the last 400 resolved tickets. The draft appears inside the help desk, the human still hits send. Numbers after 90 days: median first response 6h to 22min, CSAT flat at 4.6, escalations up slightly because agents now handle the hard ones.

Three things that mattered more than the model: quoted sources on every draft (agents reject anything they cannot verify in one click), a per-customer tone override (two enterprise accounts get formal drafts, everyone else casual), and a kill switch per category (billing drafts go back to humans during invoice week).

What broke: multilingual tickets. Arabic drafts were fine, French ones hallucinated policy. We now force French drafts to cite exact help-center paragraphs or stay silent.

Ask me about evals, the prompt is boring, the eval set is everything.

The cited-sources rule is the whole game. We do the same with Sahayi: no quote, no draft. Support teams do not trust fluency, they trust receipts. How big is your eval set?

340 tickets, hand-labelled, re-run weekly. Caught two regressions before customers did. Boring and worth it.

Arabic-first here too and the dialects will humble you fast. Egyptian Arabic drafts versus Gulf Arabic tickets was our worst week. We now detect dialect and fall back to Modern Standard when unsure. Do you do anything per-dialect?

Not yet and you just gave me next week's project. Our Arabic volume is mostly Levantine and Gulf, and I have been pretending that is one thing. It is not.

The per-category kill switch is smart. We do the opposite: AI handles everything except the categories we burned on. Same idea, worse branding. Stealing the kill-switch framing for our next agency call.

Evals question from the cheap seats: who labels them? Founders or agents?

Agents, paid, one hour a week each. Founder-labelled evals measure what the founder wishes customers asked. Learned that the embarrassing way.