We Scored Every Support Ticket Instead of a Sample. Here's What the Audit Missed

We took 20 customer support tickets that a trained QA analyst had already audited and scored the same conversations with AI-driven QA scoring, using the analyst's own scorecard. The automated scoring agreed with the human on 96% of individual checks (192 of 200). Where it disagreed, it was stricter more often than not. Three of the 20 tickets kept developing after the audit date, and one that the analyst scored 90% scored 30% once the full thread was included. The analyst scored it correctly on the day. The sample couldn't see what came after.

If you run a support team, the finding is that a sampled QA audit is a photograph of a ticket on one day, and the ticket keeps moving after the photo is taken.

Why Sample QA Feels Fine, and Isn't

If you run a support team, you probably QA a sample. Most teams do. A QA analyst pulls a handful of tickets a month, scores them against a form, coaches the agents, and reports a number. The number is usually good. Everyone feels fine.

Here is the math on the queue we studied. The support team handled roughly 900 tickets a month. The QA analyst audited about 25 of them. That's under 3% coverage. The other 97% of conversations got no score, no review, and no coaching, and nobody was doing anything wrong. That's just what one analyst can get through.

Most people already know about the coverage problem. There's a second one. A QA audit happens on a date. The analyst opens the ticket, reads what's there, scores it, and moves on. But support tickets don't stop when the audit does. The customer replies. The agent replies. The thread runs another week. The score on file still says 90%.

So you have two blind spots stacked on top of each other: you only look at 3% of tickets, and you only look at each of those once.

What We Did

We wanted a like-for-like comparison, not a demo. So we didn't invent a scorecard. We used the one the team already had.

• The queue. An email-based customer support queue for a logistics business, handling roughly 900 tickets a month across a small team of agents.
• The baseline. 20 tickets a trained QA analyst had already audited on his standard form: 10 pass/fail checks per ticket covering first-response time, accuracy, tone, commitments kept, proactive updates, and closing.
• The comparison. We scored the same 20 email threads with RubriCore, Booth's AI QA scoring agent, using the analyst's form and his definitions of pass and fail rather than a scorecard of its own. RubriCore scores every ticket in a queue against a team's existing rubric, which is what made a like-for-like comparison possible. Same tickets, same checks, same rules.
• The second pass. For each ticket, we also scored the full thread as it stood weeks later to see whether the audit-day score still held.

Two caveats. Twenty tickets is a test run, not a study. And one check on the form depends on internal systems the automated scoring couldn't see, so we excluded it from the agreement figures.

What We Found

96% agreement is high enough to trust the scoring. It's also what you'd expect from two people scoring the same ticket against the same form. The last two rows are the findings.

The Ticket that Scored 90% and Deserved 30%

A customer disputed an invoice. The agent's first reply was fast, accurate, and polite. It acknowledged the problem, explained what would happen next, and promised a follow-up. On the day the analyst audited it, that ticket was a 90%. Looking at the same messages, the automated scoring gave it a 90% too. The human and the machine agreed. It was a good ticket.

Then the thread kept going. The customer wrote back asking for the update they'd been promised. No reply. They wrote again. The account manager on the client side stepped in and escalated. Then escalated again. Requests in the thread went unanswered. By the time the conversation actually ended, it was the kind of ticket that loses a customer. Scored on the full thread, it was a 30%.

Nothing in the audit process failed. The analyst read what was in front of him and scored it correctly. The problem is that the audit was a snapshot, and the snapshot was taken on the ticket's best day.

The other two tickets that kept developing changed less, but they changed. One was a shipment held at customs: after the audit, there was a 5-day gap with no update to the customer while the shipment sat, and it was eventually returned. One was a missing laptop: the laptop was found, which is the outcome everyone wanted, but one reply in the thread took 21 hours.

Three of 20 sampled tickets changed materially after their audit. If that rate holds across the 97% of tickets nobody sampled at all, the QA report is describing a queue that doesn't exist.

Where the Human Was Right

The automated scoring was stricter on 5 tickets. It failed two for accuracy issues, two for commitments that were made and not kept ("we'll let you know," "we'll keep monitoring this"), and one for a first reply that took 16 minutes against a 15-minute standard. All five are defensible. They're the kind of miss a tired human skips at the end of a shift.

The human was stricter on 3 tickets. On two of them, the analyst failed the agent for not sending a proactive update. The thread itself didn't show anything wrong. The analyst knew, from working alongside the team, that an update should have gone out. He was scoring with context that wasn't in the ticket.

That's the limit of automated QA. It sees the conversation. It doesn't see the CRM note, the internal tracker, or the thing the team lead said in standup. One check on the form in this study depended entirely on internal systems, which is why we left it out of the agreement figures.

It also caught something nobody was looking for: one of the 20 audits had almost certainly been logged against the wrong ticket number. The agent it failed for a slow reply had no messages in that ticket at all. That's a bookkeeping error, and 100% coverage surfaces those as a side effect.

So the model that works is the machine on every ticket, every day, and the analyst on the judgment calls, the coaching, and the context the machine can't see.

What to Do About It if You Run a Support Team

The first thing on this list doesn't need a tool.

Re-score your last audit batch today. Take the tickets your QA team scored last month, open them now, and read what happened after the audit date. Count how many changed. If it's one in seven, like ours, you've just learned something about your QA report that your QA report can't tell you.

Separate coverage from judgment. Ask your QA lead two questions: what percentage of tickets get scored, and how long after the ticket closes does the score get finalized? Most teams can't answer the second one, because the answer is "it doesn't." An audit should have a close-out date, not just an audit date.

Write down the rules the form assumes. Before we ran this comparison, nobody had decided whether the 15-minute response standard applied to every reply or only the first one, whether an unanswered "thanks!" from a customer fails the closing check, or whether a promise kept in a different ticket counts. The analyst had answers in his head. The form didn't. Those four questions came up in a day once a second scorer entered the picture, and they'd come up for a new human hire too.

Decide which checks need context the ticket doesn't contain. Mark them. Those stay human. Everything else is a candidate for scoring on every ticket, not a sample.

If you do bring in automated scoring, run it against your existing form first. Don't let a vendor hand you a new scorecard. Score the same tickets your analyst already scored, on your rules, and look at where they disagree. The disagreement list is what you're buying. Ours had 8 tickets on it, and working through them surfaced 4 scoring rules the form had never written down.

If you'd rather have someone run that comparison with you, that's the kind of help we offer.

Frequently Asked Questions

What is QA sampling in customer support?

QA sampling is the practice of auditing a small subset of support tickets, typically 1% to 5% of monthly volume, against a scorecard, and treating the results as representative of the whole queue. It's the default for most support teams because a human analyst can only read so many conversations.

What is 100% QA coverage?

100% QA coverage means every ticket in the queue is scored against the scorecard, not a sample. In practice this requires automated scoring, because the volume is beyond what a QA team can read. The human role shifts from scoring to reviewing exceptions, coaching, and handling checks that need context outside the ticket.

How accurate is AI QA scoring compared to a human analyst?

In this test, AI scoring agreed with a trained analyst on 96% of individual pass/fail checks across 20 tickets, and was stricter where it differed in 5 of 8 cases. Results depend on how clearly the scorecard defines pass and fail, which is why we recommend running any automated scoring against your existing form before adopting a new one.

What does automated QA miss?

Anything outside the conversation. Internal CRM notes, tracker updates, verbal context from the team, and promises kept in a different ticket are invisible to a scorer that only reads the thread. Those checks should stay with a human or be given system access explicitly.

Why do QA scores change after the audit?

Because tickets keep going. An audit scores a conversation as it stands on the audit date. If the customer replies, chases, or escalates afterward, the score on file no longer describes the ticket. In our sample, 3 of 20 tickets changed materially after their audit, one from 90% to 30%.

Does this mean the QA analyst did a bad job?

No. On every ticket where the audit-day score later dropped, the analyst and the automated scoring agreed on the audit-day score. The analyst scored accurately. The sampling method couldn't see what happened next.