The results at a glance
Jev is TypeSafe’s AI model for tasks such as choosing a category, estimating a score, or answering a yes-or-no question. We tested it in a support application that routes customer messages to billing, technical support, or a team handling escalations. GPT-4o mini performed the same task for comparison.
The project is called Confidence Gate because it tests a simple routing rule: send a ticket for human review when the model’s confidence falls below 55%. For this rule to help, incorrect answers need to fall below that threshold often enough to be caught.
The same tickets, questions, and rules
Each model processed the same 24 tickets. For each ticket, it answered questions about the right support team, urgency, need for human help, refund eligibility, and customer frustration. We compared its answers with labels written by a reviewer.
The application uses LangGraph, a Python library for connecting steps in a workflow. Both models ran through the same steps and the same routing and approval rules. The question wording and evaluation criteria were shared; each model’s adapter converted them into the format its API accepts.
flowchart TD
Q["Shared questions and ticket"]
subgraph ENG["Run once with each model"]
direction LR
J["Jev"]
O["GPT-4o mini"]
end
N["Answers and confidence scores"]
G["Shared routing and approval rules"]
Q --> J
Q --> O
J --> N
O --> N
N --> G
Each adapter exposes ask(state, questions). It sends the ticket and questions to its model, then returns answers and confidence scores in a shared format. The rest of the application reads that format.
From a customer message to a support action
The first model request assesses the ticket and chooses a support team. The application then checks the confidence score. If the score is below 55%, it sends the ticket to the escalation workflow for human review. It also escalates urgent cases that meet the human-help threshold. Other tickets go to the team the model selected.
flowchart TD
S(["Customer ticket"]) --> T["Assess the ticket<br/>First model request"]
T --> GT{"Apply routing rules<br/>including the confidence check"}
GT -->|"Billing"| B
GT -->|"Technical"| TE
GT -->|"Escalation or human review"| E
subgraph B["Billing workflow"]
B1["Assess the charge"] --> B2["Choose a refund or response<br/>Apply approval rules"]
end
subgraph TE["Technical workflow"]
T1["Assess the problem"] --> T2["Choose a response<br/>Check whether to alert on-call"]
end
subgraph E["Escalation workflow"]
E1["Assess the risks"] --> E2["Choose a specialist<br/>Check whether legal review is needed"]
end
B2 --> F["Record the proposed action,<br/>response deadline, and API usage"]
T2 --> F
E2 --> F
Each team has its own two-step workflow: a second model request assesses the details, then application rules produce a proposed action. The benchmark records that proposed action, a target response time, and API usage for evaluation.
How the application uses the answers
For example, a proposed refund above $200 requires supervisor approval. That limit is set in the application and applies to both models. Keeping these rules consistent lets us compare how the models’ answers affect the outcome.
How long routing took
163 ms vs 1,164 ms, averaged over 24 tickets
We measured two intervals. Time to decision starts when a ticket enters the application and ends when the routing rules choose its destination. End-to-end time also includes the team’s assessment and the final proposed action. Each model made two API requests per ticket.
The application itself added an average of 3.7 ms with Jev and 3.6 ms with GPT-4o mini. Most of the measured time was spent waiting for the model APIs.
What these timings cover
Jev chose a route in 163 ms on average, compared with 1,164 ms for GPT-4o mini. The complete workflow averaged 316 ms and 2,273 ms respectively. These measurements include network time from the machine running the test, so timings may change with location, service load, and ticket content.
What processing the tickets cost
$0.06 vs $0.37 per 1,000 tickets
Using the prices recorded for this experiment, all 24 tickets cost about $0.00144 with Jev and $0.00880 with GPT-4o mini. Scaling those totals to 1,000 tickets gives about $0.06 and $0.37. These are estimates based on API token usage.
Tokens are the units the APIs use to measure input and output. The calculation prices Jev input at $0.042 per million tokens and output at zero. For GPT-4o mini, it uses $0.15 for input and $0.60 for output. Jev used fewer input tokens, while output counts were similar: 4,573 for Jev and 4,487 for GPT-4o mini. The different prices account for much of the cost gap.
Full results
Both columns cover the same 24 tickets. An upward arrow means a higher value is better; a downward arrow means a lower value is better. The definitions below explain the less familiar measures.
| Metric | jev-latest | gpt-4o-mini |
|---|---|---|
| Quality | ||
| Department accuracy ↑ | 0.875 | 0.750 |
| Average urgency error ↓ | 0.549 | 0.646 |
| Human-help classification accuracy ↑ | 0.583 | 0.750 |
| Confidence and human review | ||
| Calibration error ↓ | 0.199 | 0.150 |
| Confidence separation ↑ | −0.010 | −0.000 |
| Share sent for low-confidence review | 0.083 | 0.000 |
| Share of errors flagged by confidence check ↑ | 0.000 | 0.000 |
| Speed | ||
| Time to decision, mean ↓ | 163.3 ms | 1164.4 ms |
| Time to decision, p95 ↓ | 283.6 ms | 1345.5 ms |
| End to end, mean ↓ | 316.1 ms | 2273.2 ms |
| End to end, p95 ↓ | 429.8 ms | 2996.1 ms |
| Average time in model API calls | 312.4 ms | 2269.6 ms |
| Average application overhead | 3.7 ms | 3.6 ms |
| Cost | ||
| Input tokens ↓ | 34,226 | 40,750 |
| Output tokens | 4,573 | 4,487 |
| Estimated cost per 1,000 tickets ↓ | $0.060 | $0.367 |
| Requests per ticket | 2.00 | 2.00 |
The models chose the same department for 21 of the 24 tickets (about 88%). Values such as 0.875 are proportions: 0.875 means 87.5%. Confidence measurements are especially sensitive to this small sample.
How to read the measurements
- Department accuracy
- The share of tickets where the chosen team matches the reviewer’s label, measured before the confidence check changes the route.
- Average urgency error
- The average distance between the model’s urgency score and the reviewer’s score on a 0–3 scale. A value of 0.549 means the scores differed by about half a level.
- Human-help classification accuracy
- How often the application agrees with the reviewer about whether a person is needed. It treats a model probability of 65% or more as a yes.
- Calibration error
- Predictions are grouped by confidence. For each group, we compare average confidence with the share of correct answers. The reported value is the weighted average of those gaps.
- Confidence separation
- Average confidence on correct answers minus average confidence on incorrect answers. A positive value means correct answers received higher confidence on average.
- Low-confidence review and errors flagged
- The review rate counts all tickets sent for review because confidence was below 55%. The errors-flagged rate counts how many incorrect department choices were among those tickets.
- Mean and p95
- The mean is the average time. The 95th percentile, or p95, describes the slower end of the run: roughly 95% of tickets finished within that time.
When the models chose different teams
The models chose different departments on three tickets. Jev matched the reviewer’s label in each case. The percentages below are the models’ confidence in their chosen department. These examples show why a request can be difficult to route: a customer may ask for a refund while describing a technical problem.
"Two hours of edits vanished when the editor crashed. I want my money back for this month."
"I've asked twice about the duplicate charge and got a template reply both times. I'd like an actual person to look at this."
"I got a login alert from a country I've never been to, and there are records in my account I didn't create."
Did confidence help identify wrong answers?
A model is calibrated when its confidence matches how often it is right. For example, among many answers given with 80% confidence, about 80% should be correct. A support application could use confidence to decide which tickets need a person to check the routing. Here is what happened with Jev in this test.
Jev chose the wrong department for three tickets. The confidence check sent two tickets for human review, but both had correct department choices. All three incorrect choices scored at least 55% confidence and passed the check. At the threshold used here, the check caught none of Jev’s routing mistakes.
There were only 24 tickets and three incorrect Jev answers. That gives us little evidence about how its confidence behaves across a wider range of mistakes. A larger dataset is needed to estimate calibration reliably and choose a useful review threshold.
For this run, the practical result is clear: two correct department choices were flagged for review, and three incorrect choices passed. Anyone using this approach would need to measure both the mistakes caught and the extra review work on their own tickets.
GPT-4o mini had a lower calibration error: 0.150, compared with Jev’s 0.199. It also reported about 90% confidence on nearly every ticket. A nearly constant score gives the application little help in choosing which individual answers to review, even when its average calibration error is lower.
Jev gave a wider range of confidence scores, including 32%, 42%, and 55% on the examples above. Across the whole dataset, however, its average confidence on correct answers was 0.010 lower than on incorrect answers. GPT-4o mini’s difference was approximately zero. Neither model showed a useful average separation in this run.
Test the review rule on more tickets
The next step is to evaluate a larger set of representative customer tickets, with reviewed labels and enough incorrect answers to study confidence. The review threshold should be chosen on a separate development set, then evaluated on new tickets. That would show how many mistakes it catches and how much work it sends to people.
On a set of 24 tickets, one answer changes accuracy by about four percentage points. Repeat runs can also change which scores cross a threshold. Across two runs, Jev chose the same departments, while human-help classification accuracy changed from 66.7% to 58.3% and the low-confidence review rate changed from 12.5% to 8.3%.
The tickets were invented for this experiment, and one reviewer assigned the labels. About a third were deliberately ambiguous. The routing and approval thresholds need evaluation for any intended use. Cost estimates use manually recorded prices, and timings include network conditions during the run. Gemini was also attempted, but its free-tier quota ran out after seven tickets, so it is excluded from this 24-ticket comparison.