Drafting first replies to routine tickets
The first reply is where the support queue backs up. This use case drafts that reply from your knowledge base, cites the articles it relies on, and hands it to an agent for approval — so the human stays in the loop and the queue stops stalling on typing speed.
AI ticket drafting writes the first response to a support ticket, cites the knowledge-base articles the reply relies on, and hands the draft to a human agent for approval — the draft is a suggestion, not an automation.
Every reply traces to a source. If the retriever can't cite, the draft doesn't appear. The agent still decides whether it sends.
Data in, transforms, integrations, outputs
Map the workflow end to end before you automate any part of it — including the mid-stage inputs most pilots forget. That's the artifact every stage after this one depends on.
Data in
Start
- Incoming ticket (subject + body + attachments)
- Customer account context — tier, contract, history
- Ticket metadata — product, priority, category
- Current knowledge-base articles on the topic
- Active incident and policy notes
Mid-stage
- Agent edits and the reason (tone / policy / factual)
- Whether the draft sent as-is or was revised
- Customer's reply — resolution signal
- CSAT if the survey fires
- Escalation decision and reason
Transforms
- Classify intent + sensitivity
- Retrieve relevant KB articles
- Draft reply with citations
- Check policy vs. account tier
- Suggest next action — macro, credit, escalate
- Hand to agent with confidence note
Integrations
In
- Helpdesk (Zendesk / Front / Intercom)
- KB / CMS (Confluence, Notion, help center)
- CRM for account context
- Billing system for credit / refund actions
Out
- Helpdesk draft composer
- Agent-facing confidence UI
- Event stream to eval pipeline
- Feedback loop from agent edits
Outputs
- Draft reply in queue, with citations
- Agent decision (approve / edit / reject)
- Customer response and CSAT
- Weekly eval set from edits
- Daily metrics: TFFR, handle time, FCR, CSAT, intervention rate
Why Assist, not Automate
Assist is the right posture for weeks 1 through 6 of this pilot — and for most teams, longer.
Three reasons. First, the blast radius of a wrong answer is customer-facing — a mis-cited policy on a live ticket creates a new support escalation, not just a metrics dip. The human gate is the safety system.
Second, the last 10% of a reply is where brand and tone live. A drafted reply a human approves in 15 seconds is worth more than an auto-sent reply that misses the customer's tone in 3.
Third, the agent edits are your training data. Auto-sending eliminates the loop that makes automation safe later.
What changes when you move to Automate
- Automate becomes right when
- Intervention rate is below 20% across 500+ drafts, gold-set accuracy is above 95% (citations match current KB), and the exclude-list has survived a red-team pass.
- Agentic becomes right
- Rarely for this workflow. An agent that sends replies without approval is an escalation machine — the shape is wrong even when the model is good.
- Co-designed becomes right
- If you're redesigning support entirely — e.g., an AI that owns a book of accounts end to end. That's a Transform-stage redesign, not a Pilot-stage posture for this workflow.
The baseline
Two weeks of measurement, before any drafting is turned on. Without this, the pilot produces a number nobody believes.
Measure these six things for two weeks
Rolling median is fine — you don't need a research study, you need a defensible starting point.
- Median handle time on routine tickets (category-filtered, not all tickets)
- First-contact resolution % on routine tickets — the quality measure speed can't replace
- CSAT on routine tickets (your existing survey is fine)
- Escalation rate — routine → specialist, as a proxy for how often "routine" wasn't
- Rework rate — if drafting is already partially in place via templates or macros
- Ticket volume breakdown by topic — so "routine" can be an explicit list, not a classifier guess
The metrics, with fail-ifs
Speed alone is not a result. Every row has a fail-if for a reason — a metric without one is a number that looks positive while the quality it depends on rots.
| Metric | Baseline | Target after pilot | Fail if | How measured |
|---|---|---|---|---|
| Handle time (routine) | [median from baseline] | −25% to −40% | Improvement < 15% | Helpdesk native |
| Time to first response | [median] | −40% to −60% | Improvement < 25% | Helpdesk native |
| First-contact resolution | [%] | Flat or +5 pts | Any drop ≥ 3 pts | Helpdesk native |
| CSAT | [score] | Flat or +5 pts | Any drop | Existing survey |
| Intervention rate | N/A | < 50% | > 65% | Edit tracking in draft UI |
| Hallucination audit | N/A | < 3% | > 5% | Weekly spot-audit of 50 sent replies against the KB |
What usually goes wrong
Four failure modes that recur across most pilots. Each has a name, a diagnosis, and a prescription — because a trap without a prescription is just a complaint.
-
"Routine" defined by classifier confidence
When you let the classifier decide which tickets get drafts, "routine" becomes "whatever the classifier is confident about" — which drifts as you improve the model. The scope quietly widens, and quality quietly drops.
Prescription: Define routine as an explicit list of 15–25 named topic categories. Apply the classifier only to route within that list, never to expand it. The list is owned by the support lead, reviewed quarterly.
-
Speed without quality
Teams celebrate a 40% handle-time drop without checking FCR or CSAT. That's regression dressed as progress — you've automated the fast replies and left the hard ones to a queue that's still slow.
Prescription: No dashboard without all three — handle time, FCR, CSAT — in the same view. If handle time improves and FCR drops, you've automated complaints, not resolution.
-
Drafts without citations
A draft that says "per our policy on late deliveries" without linking the actual article teaches the customer to ask again. It also teaches the agent not to trust the draft, which collapses intervention rate over time.
Prescription: The drafting prompt fails closed. If the retriever can't cite, the draft doesn't appear — the agent sees a note instead. Citation is not a nice-to-have, it's the credibility system.
-
Losing the agent edits
Every agent edit is a free training signal. If you don't instrument what they changed and why, you're rebuilding the eval set from scratch every quarter — and every quarter it gets harder.
Prescription: Every edit gets a structured tag — tone, policy, factual, length. Feed them into a weekly review. Six weeks of edits becomes a 300-item gold eval set you couldn't have written by hand.
Scale, iterate, or stop
These conditions are cumulative, not weighted. Any one "stop" condition overrides all "scale" conditions.
Scale
- Intervention rate < 50% for 3 consecutive weeks
- FCR and CSAT flat or up across those weeks
- Daily hallucination audit below 3%
- Exclude-list caught a real churn-threat case in a red-team pass
Iterate
- Intervention rate 50%–65%
- Any quality metric moved by 3–5 pts
- Hallucination audit 3%–5%
- Classifier drift observed within the routine list
Stop
- Intervention rate above 65% for 4 consecutive weeks
- FCR drops ≥ 5 pts, or CSAT drops at all
- Any publicly-known hallucination incident
- Any excluded category reached a customer
Two scenarios
The routine case — where the workflow operates as designed — and the trap case — where it doesn't and why. Illustrative numbers; real customer and vendor data would replace them.
Scenario A — the routine case
- Customer writes
- Enterprise tier, order #4821, shipping 3 days late — "Where is my order 4821? This is the second time this quarter."
- System retrieves
- Order status (delivered to hub, not final mile), SLA tier for enterprise, KB article SLA-003 (ships-in-48h credit rule), prior ticket on this account (2 months ago, same hub).
- Draft shown
- Reply linking the tracking status, citing SLA-003, applying the €50 credit per the SLA tier, with a suggested action already queued. Confidence: high. Three citations.
- Agent reviews
- Reads in 12 seconds, changes one phrase — "we apologize" → "I apologize" — approves.
Scenario B — the trap case (and the fix)
- Week 2
- Drafting was applied to all tickets by classifier confidence. Scope = "whatever the model is confident about."
- Customer writes
- Mid-market account, churn threat on the ticket: "This is the third time. Our legal team has been asked to review the contract. I want to speak to someone senior."
- What happened
- The classifier was confident — the ticket looked like a "shipping delay + escalation" pattern. Draft went to an agent who rewrote it completely. Intervention rate for the week: 92%.
- Root cause
- No explicit exclude-list. "Routine" was defined by the classifier, not by policy.
- Fix
- Explicit topic list (18 categories). Explicit exclude-list applied at ticket intake: churn threats, legal, PR-sensitive, security incidents, vulnerable-customer flagged. All decided before the classifier runs.
What good looks like
Four things in one view. If any is off, the pilot isn't ready for the next stage.
The dashboard you should be able to open in 60 seconds
Not a vanity dashboard. Four numbers with four thresholds.
- Intervention rate trending toward 30% (from a starting point; not a hard number until week 6)
- FCR flat or up versus baseline
- CSAT flat or up versus baseline
- Weekly hallucination audit below 2% by week 12
Related
Frequently asked questions
Should AI-drafted replies disclose that AI assisted?
There is no universal legal requirement, but there is a strong operational one: disclose. A one-line note at the bottom of the reply — "Drafted with AI, reviewed by our team" — sets trust expectations correctly, and if a draft ever goes out wrong, disclosure is the difference between a mistake and a deception. Most enterprise customer-success teams now disclose by default. Consumer-facing teams often do not, because the review is human. Either way, decide this deliberately before launch, not after the first incident.
How do we define "routine" if we cannot rely on the classifier?
Define routine as an explicit list of 15–25 named topic categories — password reset, order status, shipping delay, returns eligibility, plan change, and so on. Apply the classifier only to route within this list, not to expand it. The list is a business decision, owned by the support lead, reviewed quarterly. Classifier confidence is a signal about the classifier, not a definition of what is safe to draft.
What kinds of tickets should never get a draft?
At minimum: churn threats, legal complaints, PR-sensitive topics, security incidents, vulnerable-customer flags, and anything mentioning an active regulator or law firm. Apply the exclude-list at ticket intake, not at draft time — deciding not to draft after the classifier has already burned tokens is slower and leaks information about the topic into the pipeline. The exclude-list is a hard filter, not a soft recommendation.
When do we move from Assist to Automate?
Three gates, all required: intervention rate below 20% sustained across 500+ drafts, a gold eval set of 100+ tickets above 95% accuracy (citations match current knowledge base), and an explicit exclude-list verified through at least one red-team pass. Speed is not a gate — handle time dropping is the expected result of Assist, not a reason to move to Automate. The gate is whether a wrong draft would be safe to send without review. If it would not, you stay in Assist.