Jev-X

Millin Gabani

@trillhause_ · 2K
Opinions

We have found the same thing for creating cheap and highly accurate evals. LLMs as Judges are fine, but they are expensive, slow and still shit at scoring. What works best is to perform multiple binary classifications using cheap decision models like JEV and then create deterministic heuristics on top of these classification results to compute scores, or even higher level pass or fail decision using different thresholds. you can run evals across millions of traces with higher accuracy at almost negligible costs.

Nathan Flurry 🔩@NathanFlurry · 2026-11-18

quick tip we learned the hard way @ rivet: jev does not reason ...which is by design ...and we should have rtfm first ~ we started by prompt-engineer to ask something like: ``` 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 = 𝚊𝚜𝚔( "𝚐𝚒𝚟𝚎 𝚖𝚎 𝚝𝚑𝚎 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 𝚘𝚏 𝚝𝚑𝚒𝚜 𝚒𝚜𝚜𝚞𝚎. 𝚏𝚘𝚛 𝚎𝚡𝚊𝚖𝚙𝚕𝚎, 𝚜𝚎𝚟-𝟷 𝚒𝚜 𝚇, 𝚜𝚎𝚟-𝟸 𝚒𝚜 𝚈, 𝚜𝚎𝚟-𝟹 𝚒𝚜 𝚉, 𝚎𝚝𝚌", 𝚘𝚞𝚝𝚙𝚞𝚝𝚜=[𝚂𝙴𝚅𝟷, 𝚂𝙴𝚅𝟸, 𝚂𝙴𝚅𝟹, 𝚂𝙴𝚅𝟺] ) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 ``` normally, an llm would spend reasoning tokens to come to a descent conclusion but doing this with jev gives meh results, and defeats what makes jev awesome ~ after rtfm, we split it up in to tiny prompts like this: ``` 𝚒𝚜_𝚍𝚘𝚠𝚗 = 𝚊𝚜𝚔("𝙳𝚒𝚍 𝚝𝚑𝚎𝚒𝚛 𝚙𝚛𝚘𝚍𝚞𝚌𝚝𝚒𝚘𝚗 𝚜𝚎𝚛𝚟𝚒𝚌𝚎 𝚐𝚘 𝚍𝚘𝚠𝚗 𝚘𝚛 𝚋𝚎𝚌𝚘𝚖𝚎 𝚞𝚗𝚞𝚜𝚊𝚋𝚕𝚎?") 𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍 = 𝚊𝚜𝚔("𝙰𝚛𝚎 𝚝𝚑𝚎𝚢 𝚋𝚕𝚘𝚌𝚔𝚎𝚍 𝚠𝚒𝚝𝚑 𝚗𝚘 𝚠𝚘𝚛𝚔𝚊𝚛𝚘𝚞𝚗𝚍?") 𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗 = 𝚊𝚜𝚔("𝙸𝚜 𝚜𝚘𝚖𝚎𝚝𝚑𝚒𝚗𝚐 𝚒𝚗 𝚁𝚒𝚟𝚎𝚝 𝚋𝚛𝚘𝚔𝚎𝚗

Replies1Reposts0Likes3Bookmarks3Views300
Picked for 11/18 daily picks · ranked by quality that daySee that day →
View on X ↗

Engagement counts are snapshots taken when collected.