hckrnws

It’s definitely not in the community’s interest to have auto posting bots also commenting, and I am not advocating for that

But in this case I appreciated the summary so i didn’t have to waste any time on the article.

Hopefully next time, my agent will read it all for me and make this comment.

Using Reddit as the ground truth here feels like evaluating on training data

Too much text. The only question I have and want to see at the top is whether it's more human aligned than LLMs (or, potentially, overfit)

TypeSafe's new model, Jev, doesn't write text. You give it a situation and a question, and one quick call returns a probability for each answer.

The test: 770 posts from Reddit's r/AmItheAsshole. Each model had to predict the verdict Reddit actually gave.

Jev came second of seven setups. Sonnet 5 was a bit more accurate, though the lead is borderline once you account for how many comparisons were made.

Jev's median call was 6.3× faster than Sonnet's. Fast, but not 40–200×. That was measured on one laptop in one evening, so treat the exact number with some caution.

I have no affiliation with TypeSafe. Code, logs and method are open, so you can rerun it yourself.