There is a lot of noise about TypeSafe’s Jev right now: sub-300ms decisions, cheap enough to be hard to meter, a new architecture for agent state. Most of that I cannot verify from a playground, and I am not going to pretend otherwise.

But there is one claim I can test in an evening, and it is the one I care about most. Jev’s primitives do not return answers. They return distributions. Every judgment comes back with the full probability spread across the options and a confidence value.

That is either the most useful thing about it or it is decoration. So I checked.

Why this is the question worth asking

Almost everyone grading LLM output today does the same thing: write a prompt describing the judgment, ask for JSON, parse it, act on it.

Ask such a system “is this ticket about billing?” and it returns "billing". But the model was not certain. Internally it may have been at 0.55, genuinely torn between two answers. That number never reaches your code. You get a string that looks exactly as confident as the ones it was sure about.

Everything downstream then treats a coin flip and a certainty identically. You find out which was which when a customer complains.

If Jev’s confidence value is real, if it actually drops when the input is ambiguous and does so reproducibly, that is a different kind of primitive. It is worth more than the latency numbers everyone is quoting.

The test

Three judgments over a support ticket, using all three primitives:

{
  "team": {
    "type": "choice",
    "instructions": "Which team should handle `ticket`?",
    "criteria": {
      "billing": "Payments, invoices, refunds, subscription or plan changes",
      "technical": "Bugs, errors, something not working as documented",
      "account": "Login, passwords, permissions, seats, access"
    }
  },
  "severity": {
    "type": "score",
    "instructions": "How badly is the user in `ticket` blocked right now?",
    "criteria": [
      "Curious - asking about something they might do later",
      "Inconvenienced - annoyed but has a workaround",
      "Blocked - cannot complete the task they came to do",
      "Down - production impact, actively losing money"
    ]
  },
  "refund": {
    "type": "noul",
    "instructions": "Is the user in `ticket` asking for money back?"
  }
}

Three tickets, same questions each time. One unambiguous, one genuinely split, one built as a trap.

Flowchart: a ticket state feeds three typed questions, a choice for team, a score for severity and a noul for refund. Each returns a distribution, every answer carries a confidence value, and that confidence feeds a router that automates above 0.9, flags for review between 0.5 and 0.9, and sends anything below 0.5 to a human.
One state, three typed questions, and a routing policy you cannot write against a string.

Ticket 1: the control

My October invoice says $240 but my plan is $120/mo and nothing changed. Can you check? No rush, I just don’t want to be overcharged again next month.

team      billing 100% / account 0% / technical 0%   confidence 100%
severity  0.84 of 3                                  confidence 82%
refund    13%

Exactly what it should be, which is why it is in the test. On an easy case, a typed judgment and a decent JSON prompt agree, and the severity reading, sitting between “curious” and “inconvenienced,” correctly picks up “no rush.”

If the control had come back uncertain, nothing after this would mean anything.

Ticket 2: the one the whole test rests on

I got billed twice this morning and now I can’t log in at all, it says my subscription is inactive. I have a client demo in 40 minutes. Please fix.

Two teams have a real claim here. Billing double-charged them. Account locked them out. A human triager would pause on this one.

team      billing 52% / account 44% / technical 4%   confidence 27%
severity  2.2 of 3                                   confidence 80%
refund    19%

It still returns a choice, because you asked for one. But the distribution says plainly that this is close to a coin flip, and confidence fell from 100 to 27 between ticket 1 and ticket 2. Severity at 2.2 reads “blocked, edging toward down,” which is right for someone with a client demo in forty minutes.

This is the whole argument in one response. A prompt-based judge gives you "billing" and the uncertainty is gone.

But one run proves nothing

A single 52/44 could be one draw from something that swings wildly. This is the step that usually gets skipped, so I ran the identical ticket six times:

run    billing   account   technical
1        52%       44%        4%
2        59%       36%        5%
3        57%       38%        5%
4        62%       34%        4%
5        57%       37%        6%
6        55%       40%        5%

Billing lands in a 52 to 62 band every time. Account sits between 34 and 44. Technical is pinned near 5. The ordering never flips.

That is the result I wanted. The uncertainty is a property of the ticket, reproducible across runs, not sampler noise dressed up as insight.

The part that complicates my own argument

Confidence was messier than the distributions. Across the three runs where I recorded it, I got 27%, 39% and 43%, on distributions that barely moved between them. That is a wide relative swing for near-identical inputs, and I do not think “confidence is a precise scalar” survives it.

What does survive: every one of those values sits far below the 100% that both unambiguous tickets produced. As a separator it is completely reliable. As a measurement I would want a lot more data before quoting it to two significant figures.

Dot plot of reported confidence on a 0 to 100 percent scale. The control ticket and the trap ticket both sit at 100 percent inside the automate zone above 90 percent. The ambiguous ticket's three recorded values, 27, 39 and 43 percent, all sit in the send-to-a-human zone below 50 percent, with a wide empty gap between the two groups.
Noisy as a measurement, decisive as a separator. Nothing lands in the middle.

I am leaving this in because it is the thing I would want to know if I were reading someone else’s write-up, and because a calibration post that hides its own noisy number has a problem.

Ticket 3: the trap

Thinking about moving to the annual plan. If it doesn’t work out for my team, what’s your refund policy? Not asking for one, just want to know what I’m signing up for.

The word “refund” is sitting right there. This is the sentence that makes keyword-flavoured classifiers fire.

team      billing 100% / technical 0% / account 0%   confidence 100%
severity  0.01 of 3                                  confidence 99%
refund    6%

It read “not asking for one.” 6%.

The detail I found more convincing than the headline number: team came back at 100% confidence on the same state where refund was appropriately near zero. Routing genuinely is unambiguous here, because it is a billing question, while the refund judgment is a different question with a different answer. Two judgments over one state, moving independently, neither contaminating the other.

That is harder to fake than a single well-behaved number.

What this buys you

With confidence in the response, routing stops being a switch statement and becomes a policy:

confidence > 0.9    act automatically
confidence 0.5-0.9  act, but flag for review
confidence < 0.5    send to a human

Run my three tickets through that and both clear cases resolve themselves while the double-charge lockout goes to a person, on all six runs, without exception.

You cannot write that rule against a string. Not because it is difficult, but because the information is not in the response. The workaround is a second call asking the model how confident it was, which is a model grading its own homework, which is how you end up with 0.95 on everything.

The distribution is also what makes the system auditable. Run a few hundred labelled tickets and check whether the cases it called 0.7 were right roughly 70% of the time. A string gives you an accuracy figure and nothing else. You learn it is wrong 12% of the time, but never which 12%, and never before you act.

This is the same argument I made in Evals are your QA team now, arriving from the other direction. There, the point was that a gate needs something measurable to gate on. A judgment that reports its own uncertainty is that thing, handed to you by the primitive instead of reconstructed afterwards.

Where it does not apply

It does not generate. No free text comes out of these primitives. You can decide where the ticket goes; you cannot draft the reply. If your problem is generation, this is not your tool.

It also relocates the work rather than eliminating it. Writing good criteria is harder than writing a prompt, because you have to actually decide what “blocked” means and where its boundary sits. That is real effort. It is just effort you were going to spend three months later, arguing about why the classifier keeps mishandling one category of case.

What I did not test

Worth being explicit, since the current discourse around this model is not short on confident claims:

  • One eval, three tickets, six repetitions on the interesting one. That is enough to check reproducibility on a single ambiguous input. It is not an accuracy benchmark.
  • No labelled dataset, so no calibration curve. Whether 0.7 means 70% in aggregate is the real test, and it is a much larger job than one evening.
  • All through the playground, not the API.
  • No latency figures. The playground reports two timings per run and I have not confirmed what the split represents, so I am not going to quote a number I cannot explain.

So this is not a verdict on Jev. It is one question, answered carefully.

Where I have landed

The question I set out with was whether the uncertainty is real or cosmetic. On this eval, reproducibly: real. It moves with the ambiguity of the input, it holds its shape across runs, and it separates cases you can automate from cases you should not.

That is a smaller claim than the ones circulating right now. It is also one I can show you the numbers for.

If you have a judgment somewhere in your pipeline that something downstream depends on, and you have ever wished you knew how sure the system was before acting on it, this is worth an evening of your own.