Show HN: JevBench, a reproducible benchmark for typed decision models

hckrnws

Show HN: JevBench, a reproducible benchmark for typed decision models

148

by florianstandhar

Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.

Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.

JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.

A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.

Leaderboard right now:

  #1 - Jev            74.4
  #2 - SemIf          73.1
  #3 - djev           73.0
  #4 - Winnow-12B Q8  71.2
  #5 reflex 4B        70.3.

MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes:

https://github.com/fstandhartinger/jevbench

Two no-signup demos:

https://who-is-right.app.mintapis.com

https://is-it-ai-slop.app.mintapis.com

Limitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.

Wdyt?

hbrn
63d
$40m in funding, 2 years in stealth.
Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
1. vessenes
  33d
  I think it’s much more likely that these benchmarks are not good, in that they do not explore much of the space that jev was (likely) trained to cover. The doom demo is a good example — I’d like to see a wide variety of things like that included in any benchmark, not just ‘email classification’ or what have you.
  Think of it this way: there’s some time needed to optimize / design an architecture, and the world gets that for free when it’s described. As to the rest of the last two years spent, is it more likely a former oAI lead spent them fucking around, or adding as many RL environments as possible to its model that is supposed to be a generalized classifier?
  Right now my prior is that jev is probably better than these rando weekend models, whether or not we know how to test and demonstrate that in a benchmark. It’s also super cheap, so I don’t think there’s a strong reason not to try out building with it first, then walk down the ladder to an open model if you need to for some reason.
  1. hbrn
    13d
    So if benchmark says Jev sucks, it's benchmark's fault. And yet Typesafe doesn't share anything about their internal benchmark that apparently proves how amazing Jev is.
    Funny enough, when I tell people that Jev cannot play tic-tac-toe I hear a similar argument - it's not what Jev was built for. Jev has this elusive use case that noone can describe, so when Jev fails, it's just because it wasn't built for it. Convenient.
    Don't you find it suspicious that Jev cannot play tic-tac-toe or checkers, but can play Doom? Don't you find it suspicious that nothing of the Doom demo was shared: no harness, no control loop, no state encoding, no prompts - nothing.
    And the explanation is obvious - Jev isn't playing doom. The harness is. They essentially built a Doom bot, dumbed down it's control loop for Jev, and gave reins to Jev. Look mom, Jev is playing Doom!
    The state says "you're pointing at the cacodemon" or "you're not pointing at the cacodemon" and Jev has to decide whether to press fire. Frontier intelligence!
    You can build a harness where a coin flip is playing Doom.
    vessenes
2. Retro_Dev
  03d
  Isn't the entire deal with jev that it is fast? I'd be interested to know how the energy cost of the Qwen-based model compares with Jev. Of course, Jev is currently locked up so we don't know... "Trust me bro Jev is revolutionary and amazing, pay more money for our inferior product which costs more to run, and of which you need to access by sending us the data"
3. flashnik86
  02d
  [dead]
dmix
93d
You can spot vibecoded websites by how they include the prompt or commit-style comments into the literal interface, instead of communicating it via visual context (or simply excluding it)
> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.
A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
1. spiderfarmer
  03d
sean_pedersen
13d
Good project but this one also exists https://huggingface.co/spaces/multimodalart/jev-decision-ind... and the results do not seem to add up and also model sets are different... still needs time to mature likely
1. florianstandhar • OP
  03d
  [dead]

ks2048

13d

I was trying to figure out what exactly the tests here are. I guess I found some of the questions (here: https://github.com/fstandhartinger/jevbench/blob/main/datase...)

e.g.,

  "instructions": "Which intent does the user's message express?",
  "labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"],
  "state": "Play some Taylor Swift.",
  "expected": "play_music"

florianstandhar • OP
03d

swyx
63d
jev ceo on why he eschewed benchmarking: https://www.latent.space/i/216783460/privacy-benchmarking-an...
1. arbot360
  03d
  Many SaaS vendors forbid benchmarking, I find it crazy that such anti-competitive terms are standard across the industry but they are. Generally the goal of such terms is to "control the narrative" around the product, regardless of the truth of performance being better or worse than competitors.
joknoll
01d
Are there any "System One" models or jev like models that run on ollama?
faangguyindia
13d
Can you please add our model to benchmark? https://gambler-relay-us-west1.leo-fish.ts.net/demo
1. florianstandhar • OP
  03d
  Happy to — please open an issue at https://github.com/fstandhartinger/jevbench/issues with an endpoint or runnable code, the model and licence, and whether it was trained on the public items. Every entrant runs through the same harness, including the sealed set. I definitely prefer open models I can run locally, because sending our private held-out set of tasks to an external API produces some headaches on ur side and basically means we have to rotate the test set a lot, to avoid contaimination. We can call API hosted models though, we'll just flag the leaderboard entries appropriately.
[deleted]
03d
[deleted]
adityamishra241
13d
How do you handle task distribution and prevent the benchmark from favoring models that are tuned specifically to these 534 questions?
1. florianstandhar • OP
  03d
  [dead]
tamimio
02d
For something that was announced a week ago, and now there are ~80 model competing is crazy!
janalsncm
13d
It is strange that you put the BGE reranker in the list but not BART which is an actual zero shot classifier.
1. florianstandhar • OP
  03d
  Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.
DylanMerigaud
03d
534 English decisions in one full run sounds substantial.
nzoschke
33d
https://is-it-ai-slop.app.mintapis.com/ is a fun tool. Is the source or methodology for that in the github repo? I couldn't find it immediately.
We've been experimenting with Jev for classifying email, some thoughts here: https://housecat.com/blog/classifying-email
Flagging AI written email is a much requested feature too.
1. 542458
  03d
pushpendraw
13d
the slop detector giving 86% confidence on a keysmash is the real finding here, not the leaderboard score. confident and wrong is worse than an LLM that just hedges.
1. TheWayWithin
  02d
  [dead]
MisterMunchkin
02d
Lmao this just proves Jev is being shilled for money. Nobody would speak like this for fun.
visarga
03d
[dead]
flashnik86
02d
[dead]
kevinbaiv
03d
[dead]
[deleted]
04d
[deleted]
joserobles84
03d
[dead]
rg1992
03d
[dead]

"instructions": "Which intent does the user's message express?", "labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"], "state": "Play some Taylor Swift.", "expected": "play_music"