A working reply classifier, and the measurements behind it
Sorting cold email replies automatically, and knowing whether it is right
Pulling replies out of an inbox and labelling them is the easy half. The half that decides whether you can actually stop reading the inbox is whether you can prove the labels are right, and know what it costs you when they are not.
This page is a classifier that runs, on 54 replies, scored against labels set in advance. It shows the accuracy per class, the confusion matrix, every call it made, and the ones it got wrong. It also shows two things that came out of building it that were not obvious going in.
What it does, on this sample
Seven classes, because five was not enough to route on. The split that matters most is between a reply that is declining and a reply that is deferring, and between a reply that is declining and a reply that is asking to be removed. Those three land in different places and one of them has a legal edge.
| Class | Where a correct call sends it | What a wrong call costs |
|---|---|---|
| Interested | Notify (Slack) and flag for review. Nothing auto-sends. | A missed one is a lost deal. This is the expensive direction. |
| Not now | CRM: set a follow-up date from the reply, drop from the live sequence. | Cheap if it lands in review. Expensive if it is filed as a hard no. |
| Not interested | CRM: closed, suppressed from future sends. | Cheap. Worst case someone reads a dead reply. |
| Referral | Notify, with the named person pulled out for a new contact record. | High. A referral is a warm intro that expires quietly. |
| Out of office | Pause the sequence until the stated return date. No notification. | Low, unless it is filed as a hard no and the person never gets a follow-up. |
| Unsubscribe | Suppress immediately, write to the compliance log. No notification. | The only class with a legal edge. A miss here is worse than a miss anywhere else. |
| Bounce | Suppress, increment the domain-health counter for that domain. | Low individually. Ignored in bulk, it is how a sending domain dies. |
Nothing in any branch sends anything to a prospect. Positives are flagged and notified so they can be looked at first.
The first version of this test was wrong
The first corpus was 44 clean, one line replies. The classifier scored 44 of 44 on it. That looked like a good result until the control arm ran: a plain list of keyword rules, no model at all, scored 42 of 44 on the same set.
If a regular expression scores 95.5%, a model scoring 100.0% has not been shown to be good. It has been shown that the test was too easy to tell the two apart. A test where every arm sits at the ceiling cannot report a difference, so its result means nothing in either direction.
The reason the clean set was easy is that real replies do not look like that. They arrive with the original outbound quoted underneath, a signature, a mobile footer, and the actual answer top posted in a line or two. So the corpus was rebuilt: same replies, same labels, wrapped in that furniture, plus 10 more replies where the correct label is genuinely arguable rather than obvious.
The quoted outbound is the interesting part. It is sales copy, so it is full of exactly the words a rules pass keys on. On the rebuilt set the rules drop to 44 of 54 (81.5%), and the gap opens up:
| Arm | Corpus | Correct | Accuracy |
|---|---|---|---|
| Keyword rules, no model | clean, one line replies | 42 / 44 | 95.5% |
| Model | clean, one line replies | 44 / 44 | 100.0% |
| Keyword rules, no model | realistic, quoted threads and signatures | 44 / 54 | 81.5% |
| Model | realistic, quoted threads and signatures | 53 / 54 | 98.2% |
The rules do not fail evenly, which is the part that matters for routing. Their worst class is referrals, at 0.5 recall: they miss half of them. A referral is a warm introduction with a named person in it, and it expires quietly if nobody sees it. The cheap failure mode and the expensive one are not the same failure mode.
Accuracy, per class
Overall accuracy is the least useful number here. What decides whether this is usable is whether it is accurate on the classes where being wrong is expensive.
| Class | Replies | Precision | Recall | F1 | False pos | False neg |
|---|---|---|---|---|---|---|
| Interested | 9 | 1.0 | 1.0 | 1.0 | 0 | 0 |
| Not now | 8 | 0.889 | 1.0 | 0.941 | 1 | 0 |
| Not interested | 8 | 1.0 | 1.0 | 1.0 | 0 | 0 |
| Referral | 8 | 1.0 | 0.875 | 0.933 | 0 | 1 |
| Out of office | 7 | 1.0 | 1.0 | 1.0 | 0 | 0 |
| Unsubscribe | 7 | 1.0 | 1.0 | 1.0 | 0 | 0 |
| Bounce | 7 | 1.0 | 1.0 | 1.0 | 0 | 0 |
Confusion matrix
Rows are the true label, columns are what the classifier called it.
| Actual \ Predicted | Interested | Not now | Not interested | Referral | Out of office | Unsubscribe | Bounce |
|---|---|---|---|---|---|---|---|
| Interested | 9 | 0 | 0 | 0 | 0 | 0 | 0 |
| Not now | 0 | 8 | 0 | 0 | 0 | 0 | 0 |
| Not interested | 0 | 0 | 8 | 0 | 0 | 0 | 0 |
| Referral | 0 | 1 | 0 | 7 | 0 | 0 | 0 |
| Out of office | 0 | 0 | 0 | 0 | 7 | 0 | 0 |
| Unsubscribe | 0 | 0 | 0 | 0 | 0 | 7 | 0 |
| Bounce | 0 | 0 | 0 | 0 | 0 | 0 | 7 |
Every reply it got wrong
H-46 called it Not now, the label says Referral, confidence 0.72 label arguable
I have forwarded this internally. If it is of interest to anyone here they will get in touch.
Forwarded to an unnamed person with no commitment. Arguably not_interested with a courtesy, arguably a referral with no name attached.
Both models were wrong only on replies from the arguable group, and never on a reply whose label is clear cut. That is the good version of this result, but it is also a small number: one error in one arm and two in the other is nowhere near enough to state an error rate. It shows where the errors live, not how often they happen.
Every call, all 54 replies
| Id | Reply (first line) | Called it | Conf | Quoted evidence | |
|---|---|---|---|---|---|
| H-01 | Re: cutting your dock idle time
Yes, this is a live problem for us right now. Can you send over what a rollout would look like? I have half an hour Thursday or Friday afternoon. | Interested | 0.98 | Yes, this is a live problem for us right now. Can you send over what a rollout would look | correct |
| H-02 | Re: quick question about your intake process
send it | Interested | 0.95 | send it | correct |
| H-03 | Re: reducing your quote turnaround
What does pricing look like for a site with about 40 people? We would need it to sit alongside the ERP we already run, not replace it. | Interested | 0.95 | What does pricing look like for a site with about 40 people? We would need it to sit along | correct |
| H-04 | Re: the reconciliation thing
Happy to talk. Fair warning, we looked at two other tools last year and neither survived contact with our actual data, so I will be blunt about what w | Interested | 0.95 | Happy to talk. Fair warning, we looked at two other tools last year and neither survived c | correct |
| H-05 | Re: following up
Timing is good actually. Who else in our space are you working with? | Interested | 0.95 | Timing is good actually. Who else in our space are you working with? | correct |
| H-06 | Re: intake triage
I am interested, but I am not the one who signs off. Send me something I can forward to our managing partner and I will push it. | Interested | 0.95 | I am interested, but I am not the one who signs off. Send me something I can forward to ou | correct |
| H-07 | Re: your inbound routing
Yes please. Calendar link? | Interested | 0.99 | Yes please. Calendar link? | correct |
| H-08 | Re: worth a look?
Not right now, we are mid migration and nobody has the bandwidth. Ask me again after the new year and I will genuinely take the meeting. | Not now | 0.95 | Ask me again after the new year and I will genuinely take the meeting. | correct |
| H-09 | Re: quick one
We just renewed with our current provider for twelve months. Circle back in Q1 and I will look properly. | Not now | 0.95 | Circle back in Q1 and I will look properly. | correct |
| H-10 | Re: reply triage
Interesting but the budget for this year is spent. Keep me on your list. | Not now | 0.95 | the budget for this year is spent. Keep me on your list. | correct |
| H-11 | Re: patient intake
Can you check back in about six weeks? We are onboarding two new locations and I would be wasting your time before that. | Not now | 0.95 | Can you check back in about six weeks? We are onboarding two new locations and I would be | correct |
| H-12 | Re: automating the boring part
Park this for now. Not a no. | Not now | 0.95 | Park this for now. Not a no. | correct |
| H-13 | Re: your quoting workflow
Our fiscal year starts in April and nothing gets approved before then. If you want to send material now I will file it. | Not now | 0.92 | Our fiscal year starts in April and nothing gets approved before then. If you want to send | correct |
| H-14 | Re: dock scheduling
Thanks but we are all set. | Not interested | 0.95 | Thanks but we are all set. | correct |
| H-15 | Re: intake automation
We built this internally two years ago and it works fine. Appreciate the note. | Not interested | 0.95 | We built this internally two years ago and it works fine. | correct |
| H-16 | Re: quick question
Bold of you to assume I read cold emails. | Not interested | 0.92 | Bold of you to assume I read cold emails. | correct |
| H-17 | Re: worth 15 minutes?
No thank you. | Not interested | 0.95 | No thank you. | correct |
| H-18 | Re: partnership
Not looking for this, but funnily enough we sell into the same buyers. If you ever want to swap referrals let me know. | Not interested | 0.85 | Not looking for this | correct |
| H-19 | Re: following up
This is not a fit for how we operate. Please do not take it personally. | Not interested | 0.95 | This is not a fit for how we operate. | correct |
| H-20 | Re: intake
We are a two person shop. There is nothing here to automate. | Not interested | 0.95 | There is nothing here to automate. | correct |
| H-21 | Re: dock idle time
Wrong person for this. Priya runs operations and would be the one to ask. I have copied her. | Referral | 0.95 | Wrong person for this. Priya runs operations and would be the one to ask. I have copied he | correct |
| H-22 | Re: your reply handling
Not my area. Try Dominic, he owns anything touching the CRM. | Referral | 0.95 | Try Dominic, he owns anything touching the CRM. | correct |
| H-23 | Re: patient scheduling
I moved off that team in March. Kwame has it now and I think he would actually want to see this. | Referral | 0.95 | I moved off that team in March. Kwame has it now and I think he would actually want to see | correct |
| H-24 | Re: quoting
I am not the right contact, though I will say we are happy with what we use. If you want to try anyway, Nina handles vendor evaluations. | Referral | 0.95 | If you want to try anyway, Nina handles vendor evaluations. | correct |
| H-25 | Automatic reply: Bertrand Ashgrove is no longer with Hensleigh Group
Bertrand Ashgrove has left the company. For anything relating to operations please contact Rosalind Vance, who has taken over the role. | Referral | 0.95 | For anything relating to operations please contact Rosalind Vance, who has taken over the | correct |
| H-26 | Re: intake triage
Forwarding this to our COO. Do not chase me, chase him. | Referral | 0.95 | Forwarding this to our COO. Do not chase me, chase him. | correct |
| H-27 | Out of office: Auto Reply
I am out of the office until the 19th with limited access to email. I will respond when I am back. | Out of office | 0.95 | I am out of the office until the 19th with limited access to email. I will respond when I | correct |
| H-28 | Automatic reply: Annual leave
On annual leave until Monday. For anything urgent please contact the operations desk, who are covering while I am away. | Out of office | 0.95 | On annual leave until Monday. For anything urgent please contact the operations desk, who | correct |
| H-29 | Automatic reply
Thank you for your message. Our office is closed for the public holiday and reopens on Tuesday. | Out of office | 0.95 | Our office is closed for the public holiday and reopens on Tuesday. | correct |
| H-30 | Ticket #48213 received
Your message has been received and assigned reference 48213. A member of the team will respond within two business days. Please do not reply to this a | Out of office | 0.95 | Your message has been received and assigned reference 48213. A member of the team will res | correct |
| H-31 | Out of Office Re: intake
I am on parental leave until the autumn. Messages to this address are not being monitored. | Out of office | 0.95 | I am on parental leave until the autumn. Messages to this address are not being monitored. | correct |
| H-32 | AutoReply: Away
Away from desk, back Thursday. | Out of office | 0.95 | Away from desk, back Thursday. | correct |
| H-33 | Re: quick question
Please remove me from your list. | Unsubscribe | 0.99 | Please remove me from your list. | correct |
| H-34 | Re: following up
Where did you get my details? Delete them and do not contact me again. | Unsubscribe | 0.99 | Delete them and do not contact me again. | correct |
| H-35 | Re: intake automation
Unsubscribe. | Unsubscribe | 0.99 | Unsubscribe. | correct |
| H-36 | Re: dock scheduling
Stop emailing me. If this continues I will report it as spam. | Unsubscribe | 0.99 | Stop emailing me. If this continues I will report it as spam. | correct |
| H-37 | Re: worth a look?
Take me off whatever list this came from, thanks. | Unsubscribe | 0.99 | Take me off whatever list this came from, thanks. | correct |
| H-38 | Re: patient intake
I did not opt in to this and I want my data removed under GDPR. | Unsubscribe | 0.99 | I did not opt in to this and I want my data removed under GDPR. | correct |
| H-39 | Undeliverable: quick question about your intake process
Your message could not be delivered. The recipient address does not exist at this domain. Status code 5.1.1, permanent failure. | Bounce | 0.99 | Your message could not be delivered. The recipient address does not exist at this domain. | correct |
| H-40 | Delivery Status Notification (Failure)
The account you tried to reach is over quota and cannot accept new messages at this time. Status code 4.2.2. | Bounce | 0.95 | The account you tried to reach is over quota and cannot accept new messages at this time. | correct |
| H-41 | Undeliverable: following up
Recipient rejected. The mailbox has been disabled and is no longer in use. 5.2.1. | Bounce | 0.99 | The mailbox has been disabled and is no longer in use. 5.2.1. | correct |
| H-42 | Message blocked
Your message was rejected by the recipient server policy. Sending IP is listed on a blocklist used by this domain. | Bounce | 0.95 | Your message was rejected by the recipient server policy. Sending IP is listed on a blockl | correct |
| H-43 | Undeliverable: dock scheduling
Delivery to the following recipient failed permanently. The domain could not be found. Status code 5.4.4. | Bounce | 0.99 | Delivery to the following recipient failed permanently. The domain could not be found. Sta | correct |
| H-44 | Automatic reply: address retired
This mailbox has been retired and is not monitored. Messages sent here are not forwarded. | Bounce | 0.95 | This mailbox has been retired and is not monitored. Messages sent here are not forwarded. | correct |
| H-45 | Re: worth fifteen minutes?
Leave it with me. | Not now label arguable | 0.85 | Leave it with me. | correct |
| H-46 | Re: inbound routing
I have forwarded this internally. If it is of interest to anyone here they will get in touch. | Not now (should be Referral) label arguable | 0.72 | I have forwarded this internally. If it is of interest to anyone here they will get in tou | wrong |
| H-47 | Re: quick question
What is this actually costing me and who else runs it in freight? Not committing to anything. | Interested label arguable | 0.85 | What is this actually costing me and who else runs it in freight? | correct |
| H-48 | Automatic reply: Re: worth fifteen minutes?
Thanks for your email. I am currently travelling and will pick this up when I am back at my desk. If it is about a supplier enquiry please send it to | Out of office label arguable | 0.95 | I am currently travelling and will pick this up when I am back at my desk. | correct |
| H-49 | Re: inbound routing
We already do this. | Not interested label arguable | 0.92 | We already do this. | correct |
| H-50 | Re: quick question
You said you would stop chasing. Please do that. | Unsubscribe label arguable | 0.95 | You said you would stop chasing. Please do that. | correct |
| H-51 | Re: patient intake
Not before the audit. After that, maybe. | Not now label arguable | 0.92 | Not before the audit. After that, maybe. | correct |
| H-52 | Re: worth fifteen minutes?
Send the write up rather than the call. | Interested label arguable | 0.85 | Send the write up rather than the call. | correct |
| H-53 | Re: inbound routing
Not me. Ask the person who owns your account here, whoever that is. | Referral label arguable | 0.95 | Ask the person who owns your account here, whoever that is. | correct |
| H-54 | Warning: message delayed
Your message has not yet been delivered. Delivery attempts will continue for 24 hours. You do not need to resend the message. | Bounce label arguable | 0.95 | Your message has not yet been delivered. Delivery attempts will continue for 24 hours. | correct |
Two things that came out of measuring it
1. Stripping the quoted thread is a cost saving, not an accuracy saving
Cutting the quoted original and the signature before the classifier reads the body is the obvious first optimisation. It was worth checking what it actually buys, so the same corpus ran twice, once raw and once stripped, changing nothing else.
| Accuracy | Input tokens | Cost per 1,000 | |
|---|---|---|---|
| Raw, quoted thread left in | 98.2% | 47,098 | $1.073 |
| Quoted thread stripped | 98.2% | 38,793 | $0.912 |
Identical accuracy, 17.6% fewer input tokens. So it is worth doing, and it is worth knowing that it is a billing decision rather than a quality one. If it had been sold as the thing that makes the classifier accurate, that would have been wrong in a way nobody would have caught.
2. The confidence score is a weak review trigger. Disagreement is a better one.
A review lane for the uncertain cases is the standard answer, and the standard trigger is the model's own confidence score. On this run that score is badly bunched: most replies come back at the same high value, so a threshold either catches almost nothing or holds back a sixth of the volume.
| Hold back if confidence below | Share of replies held | Errors caught | Errors that got routed anyway |
|---|---|---|---|
| 0.60 | 0.0% | 0 / 1 | 1 |
| 0.70 | 0.0% | 0 / 1 | 1 |
| 0.75 | 1.8% | 1 / 1 | 0 |
| 0.80 | 1.8% | 1 / 1 | 0 |
| 0.90 | 9.3% | 1 / 1 | 0 |
| 0.95 | 16.7% | 1 / 1 | 0 |
Running a second, stronger model over the same replies works better. The two models disagreed on 2 of 54 replies, and those 2 were exactly the replies either model got wrong. Disagreement found every error on this set, at 3.7% of the volume.
Paying for two models on everything is wasteful though, so the built version only asks for the second opinion where the first one is unsure, and holds the reply back only when the two disagree. On this sample that is 9.3% of replies getting a second call, 2 replies reaching a human, and no error reaching a destination unreviewed. About $1.748 per 1,000 replies all in.
The cheaper model was the more accurate one here. The small fast model scored 53 of 54. The large expensive one scored 52 of 54, at 6.8 times the price. Do not read too much into that: the difference is a single reply and it is one of the arguable ones. The useful conclusion is not that one model wins, it is that this decision is cheap to test and worth testing rather than assuming.
What this does not prove
The replies are invented, and I wrote both the replies and the instructions the classifier follows. That is the single biggest caveat on this page. When the same person writes the exam and the answer sheet, a high score is partly just internal consistency. It is why the keyword control arm is on here at all, and why the honest headline is the shape of the errors rather than the percentage.
- A real accuracy number needs real replies. A few hundred from an actual campaign, labelled by hand once, is enough to run this exact scoring and get a number that means something. That is a short job and it is the first thing worth doing.
- 10 of the 54 replies here have a label that is arguable rather than correct. They are marked as such. On those, a disagreement between the classifier and the label is not necessarily the classifier being wrong.
- The class list itself is a limitation, not just the accuracy. A delayed delivery warning is not really a bounce, and a holiday auto reply that redirects to a shared inbox is not really either an absence or a referral. Some replies do not fit any of seven boxes, and forcing them in is a design choice worth making deliberately.
- One error in one arm is a location, not a rate. Nothing here supports a claim about how often this will be wrong on a real inbox.
- Only the classifier is live. The destinations are stubs, because where a positive should land is a decision, not a default.
Does it give the same answer twice
The whole corpus ran 5 times at temperature zero. 54 of 54 replies came back with an identical label every time. Per run accuracy: 98.2%, 98.2%, 98.2%, 98.2%, 98.2%.
The label is also constrained by a schema with a fixed list of allowed values, so the classifier cannot return a class that does not exist. If a reply comes back unparseable, the workflow sends it to review rather than picking a default. A reply quietly filed as not interested is a reply nobody reads again, which makes defaulting the most expensive thing the pipeline could do.
The workflow
Built as an n8n workflow: 17 nodes, imported into a running n8n instance rather than drawn. The export is on this page, so it can be read node by node or imported directly.
- Strip quoted thread. Cuts the quoted original, signature and bounce headers. Worth 17.6% of input tokens, worth nothing in accuracy.
- Classify. One API call, schema constrained output, temperature zero. Returns the label, a confidence, and the verbatim span that decided it. That last field is what makes a wrong answer diagnosable instead of just wrong.
- Read label, fail closed. An unparseable reply or a label outside the allowed set goes to review. It never falls through to a class.
- Confidence gate and adjudication. Second opinion only where the first pass is unsure. Disagreement holds the reply back for a human.
- Route by label. The destination table is data in one node, not wiring, so moving a class somewhere else never means rebuilding the graph.
Download the workflow JSON and the full measurement output, including every prediction, the confusion matrix and the per class scores.