Consider an invented return policy: unused products can be returned within 30 days, with an exception for personalized items. An AI summary that promises 30-day returns on every product has changed what a customer is being told. The summary has dropped both the unused-product condition and the exception.
That is a useful kind of problem to give a cheaper evaluation model. Supply the policy and the proposed answer, then ask whether the answer preserves the restrictions. A flag could send the writer back to the relevant passage before the answer reaches a customer.
Jev, the model TypeSafe AI introduced in September, makes that possibility interesting because it is built for inexpensive, structured judgments. It returns choices and scores. For anyone thinking about using it to help detect hallucinations, the first job is to define what the evaluator should look for.
Give the evaluator a specific comparison
A broad request to rate an answer mixes several judgments together. The answer may be readable and relevant while making a promise the source does not support. A high overall score would be difficult to act on without knowing which part of the work earned it.
The return-policy example gives the evaluator something more precise. Does the proposed answer keep the unused-product condition? Does it preserve the exception? These are separate questions about supplied material, which fits TypeSafe's guidance to ask narrow questions. The example below is invented to show the comparison.
An invented example
Unused products may be returned within 30 days. Personalized items are excluded.
Every product can be returned within 30 days.
Does the summary preserve the policy’s restrictions?
The same idea could help with research summaries. We could ask whether a claim has support in the cited passage, or whether the summary makes the source sound more certain than it is. A separate score could help compare which answer addresses the original question more directly. Those checks describe different qualities, and keeping them separate would make their results easier to use.
A claim can be true elsewhere and lack support in the supplied material. The source itself might be wrong. Comparing the texts helps find a mismatch; establishing the underlying fact can require more reporting.
Learn what the score tells you
Jev's Choice and Score responses include a confidence value. TypeSafe's documentation says it is calculated from how concentrated the model's probability distribution is across the available answers. A confident result means the model strongly favors an answer. It does not, by itself, establish how often answers like that are correct in your material.
To find that out, someone has to compare the results with examples whose answers have been checked. Include summaries with omitted conditions, along with correct summaries that use different wording. Otherwise, an evaluator might appear successful by flagging anything that departs from the original sentence.
The missed errors deserve particular attention. If a reassuring score causes someone to stop checking, the evaluator could make a bad answer easier to accept. Too many false alarms create a different cost: people spend their time reopening passages that were already fine, and eventually stop paying attention to the flags.
TypeSafe's launch material guarantees that Jev's outputs match the requested format. An allowed choice can still be wrong, so hallucination checks need to be tested against known examples.
Make the result easy to follow up
For work like ours, I would want a flag to leave the draft and its source easy to compare. Someone reviewing a low score for source support should be able to see the claim and the passage behind it without starting the search again. Jev's numeric output supplies a judgment; the material around that judgment is what lets a person investigate it.
Cheaper evaluation could make this practical for more drafts and more revisions. A useful first experiment would keep the question narrow, measure both missed errors and false alarms, and see whether following the flags helps someone make corrections sooner. That would give us a basis for deciding where the scoring helps and where a closer reading is still needed.