We let the model answer support tickets unsupervised for a week. Numbers inside

Replybase drafts answers for support teams. Normally a person approves each one. A customer asked to turn approval off for one category (password and login questions), and agreed to let me share what happened.

• 412 tickets auto-answered in 7 days
• 371 closed without a follow-up (90%)
• 34 reopened because the answer missed the point
• 7 were wrong in a way that mattered, including telling one user a feature existed that had been removed

Their human agents on the same category last quarter: 94% closed without follow-up.

Thank you for publishing the 7. Most posts like this stop at the 90%.

The removed-feature error is the interesting one. Was the old help article still in the index?

Yes. The article was unpublished but our sync only added and never removed. Fixed on day 4.

Ten years in support. 90% against 94% sounds close, but the 4 points are not the same kind of ticket. A person who gets it wrong says "let me check". The model got 7 confidently wrong. I would keep approval on for anything that states a fact about the product, and take it off for "here is the reset link".

That is the split the customer landed on too. Links and steps go out on their own, product facts need a person.

Unpopular opinion: in two years nobody will run the human comparison at all and we will just accept 90.

Not in health. One wrong appointment answer and I am on the phone with a clinic director.

What did the week cost in model usage for 412 tickets?

About $6. The cost was never the question. The 7 are the question.