We have found the same thing for creating cheap and highly accurate evals.
LLMs as Judges are fine, but they are expensive, slow and still shit at scoring.
What works best is to perform multiple binary classifications using cheap decision models like JEV and then create deterministic heuristics on top of these classification results to compute scores, or even higher level pass or fail decision using different thresholds.
you can run evals across millions of traces with higher accuracy at almost negligible costs.