I can't see any benefits that a typical ML classifier would not be better at.
This is a transformer based classifier with massive pretraining on synthetic datasets and it outperforms boosting classifiers on many benchmarks without the need of more gradient descent steps (the forward pass on X_train, y_train IS the training).
I understand that jev focus on text entry. But I feel that it is a similar kind of model but trained on text. Did someone test it on tabular data as well ?
The good news is it’s fun to see people discover and get excited about things that I like as well.
I would have never considered importing pytorch for filtering through log files before even knowing my way around it. But if i can type a filtering condition by text and hit enter; i may actually use that to save some time.
Id want something local though, but thats hardly a difficult demand for what it is.
Agreed. This isn't new. I led a research team at a Fortune 500 that used a transformer based classifier approach in a commercial product as far back as 2022 and we didn't come up with it. It was already common enough that we found the inspiration for our implementation on some web forum. Models like RouteLLM have been around for a long time. The news here isn't that a new model type came about, its that a large percentage of people messing around with this stuff that are new to AI just learned that not all transformer based implementations need to be autoregressive.
I think building generalist classifier is some open ended research task, where frontier labs can contribute: different internal reasoning, instruction tuning, building datasets and benchmarks, building and distilling super large models.
and then whatever tech it is will be absorbed/assimilated/Sherlocked into the leading products anyway
[deleted]
Also, moat discussion is the lowest form of discussion. I don’t care if jev has a moat. Did it get the interface right? What other past ideas have we overlooked that if given some love, could kick the door down like jev did?
Really silly stuff.. people wanting to talk about moats when there’s no castle. Moat talk merely projects the illusion of being engaged but, much more often than not, it’s hollow engagement.
> My main assumption is that Jev is using something quite close to a conventional large language model. As evidence of this, Latent Space reports that many of the early clones are indeed LLM-based.
Not proof that this is the case with Jev though. It might use non causal text encoder for the state, which could make sense given that it's very good for its price.
For now. Any company that grows to OpenAI/Anthropic's size and gets VC money is ought to become greedy.
so maybe typesafe's real plan is to front run and releasing their own new models for some time until they can get acquired which seems to be the only rational objective
If the headline said "Frontier labs are about to eat Jev's lunch" it might be an easier sell. But if we're gonna include Anthropic, I think part of their success is actually making products for which there is demand. It will take time for something like that to come out of this new "decision model" paradigm.
But if they had to show how well their product worked they might give away the whole game... because they'd have to compare their "noul" class against an NLI benchmark for instance, and possibly show they're losing to cross encoders and give away the fact that they are just rebranding NLI. Or rerankers (choice) or zero-shot classifiers.
There are many automation pipelines that use LLMs because there was no choice, but the multi-way classification that Jev provides is exactly what they need, and is going to be way faster and cheaper, as well as having the benefit of calibrated probabilities and structured output that can be relied on.
Or they can even offer it as a standalone API if deemed worth it.
Hopefully there will be some decent benchmarks and gateways for switching between providers easily.
The ability to use classifiers under the hood for the larger models has the potential to dramatically improve cost and throughput, allowing them to increase margin on a very similar service.
vLLM has a PR very close to merging: https://github.com/vllm-project/vllm/pull/57250
Kev is an open Jev: https://github.com/jaredpalmer/kev
The right part: autoregressive LLMs are indeed generating “probabilities” (scare quotes very much intentional). During pre-training and any SFT steps, those probabilities are nudged toward the probabilities, over the training distribution, of the next token conditioned on the previous tokens. (This is an explicit property of most training recipes: KL divergence is a “proper scoring function”.)
So if you prompt with “Paris is a city in ”, the next token probabilities estimate the probabilities over the input distribution that the next token in the sentence is the first token of France or of something else.
But there are huge caveats:
1. That is not at all the same thing as the probability that Paris is France under any distribution that you care about (the population of the various Parises, for example).
2. None of this necessarily usefully applies to RL or, as the article discusses, tool calling. The output probability of a tool call is not some Platonic idea of a probability that the input is worthy of a tool call. It’s a the result of a training process that tried to teach the model to be useful and to achieve its goals.
3. I suspect that reasoning makes this all much worse. Suppose that you prompt with “a help desk user with IP=a.b.c.d says they’re ‘in Paris’. What country are they in?” The model has been trained to generate a reasoning trace, which may well start with “let me think of where Paris could be. It could be in France or in Texas etc. The user was speaking English…” See the problem? The model is reasoning well, but it reasoned “France” before “Texas”, so the logprob for France was probably higher than “Texas”. At the end of the reasoning trade there will be an answer, but the logprobs for that answer are, at best, some representation of the probabilities of the answer conditioned on the sampled reasoning trace. And that is not the probability distribution that a Jev user wants.
And in this case I hope its true. I've been classifying a lot of email and while OpenAI `text-embedding-3-small` has been very helpful for fast and cheap embeddings, initial tests with Jev are very promising and much more ergonomic.
I put more thoughts here: https://housecat.com/blog/classifying-email
They certainly have the token budget for it.
I did so a month ago, I developed a deterministic agent framework that works with a set of predefined intents, it is instantaneous and fully deterministic.
It works thanks to FlintParser (https://github.com/gioblu/NPC-Forge/blob/main/src/FlintParse...) which can transpile plain English to any Programming Language. IMHO this is the future of AI.
Check it out: https://github.com/gioblu/NPC-Forge
Open weight classifiers and open weight LLMs will be burned onto silicon cards in a few years after the models begin to stabilize. They'll be in PCs and laptops. That's going to capture a HUGE chunk of the market.
If you need more horsepower, you'll rent the same silicon safely from AI services cloud providers without handing your data over to Anthropic and OpenAI.
That being said- hard to believe OpenAI etc couldn't build a frontier classifier too.
I love this!!
Would it be intesting/useful to use jev to generate a block of text like LLMs do ?
Like asking it to pick the n + 1 word given the starting text (using it's choice primitive), but also asking n + 2,n+3 and so on at the same time.
Would it give coherent or useful results ? Or does the fact that it computes it "all at once" means it cannot make one of it's answer influence the other ones ?
https://github.com/kyle-pena-nlp/jevchat
https://www.reddit.com/r/LLM/comments/1winnju/jev_the_new_ai...
Despite Typesafe claims that Jev is not an LLM, it obviously is.
Reading logits is the cornerstone of ML. It’s almost like many of the people reporting on and fawning over AI have no technical background and never knew about ML classifiers or calibration.
Which is likely what all the VC, hype machine, and overinflated claims are really about anyways.
The tech etc is easily replicated. The hype / name, not.
I seem to remember reading that the Jev-founder-guy is ex-OpenAI anyways. So that's how these things often roll.
Wouldn’t be surprised if every single AI house spins up a copy
But like they usually also have an embeddings endpoint
In reality, none of these really matter. The frontier labs can easily do something like this but likely havent because the size of this market is too small and it is not on the critical path to AGI.
When you have as many resources as OpenAI and Anthropic, theres basically no point in putting compute towards random bets that don't have a predictable return. And at this point, scaling up transformers is almost a surefire way ot putting money in via training and getting money out via increased capabilities AND it speeds up your own business by factors of X. Sidequesting a Jev like product is falling for twitter hype and is likely not going to happen, definitely not by Anthropic, and I'd bet probably not by OpenAI either.
I wrote a function pretty much exactly like Jev about a year ago... I would provide my function an array with a bunch of labels as first argument and a string of text/content as second argument and it would give me a relevance score for each label in the array against the provided text/content.
It's really easy to implement. Took me like a day or so by hand. You just compute the embedding vector of each label in the array then compare each one against the embedding vector of the text/content and it then return an object which maps each label to its similarity score.
I used it to allow users to tag employee data from LinkedIn. I had essentially forgotten about that code until now...
Fed Claude an api key from typesafe and a link to documentation, and within about 10 minutes I had a view of HN that was populated with a little ranking as to sloppiness of each comment.
When your mind has been wired a bit to LLM latency, it feels extremely fast, and for such a subjective rating I think it did a good job.
Feels like it sits in a space between traditional ML classification and the frontier models. I can't think of a 'real' production use case for it in my sphere of influence, but certainly some will. And of course there will be five Jev competitors by the end of the year.
To add to this the more difficult problems in classification done on scale have always been about collecting "good" -ve examples, enough data to calibrate on every confidence interval and debugging outliers. And those are solved on a case by case basis by the company pursuing its own peculiar version of the problem.
Am I the only one who thinks this is just all hype?
As there have been a lot of Jev related submissions, can someone point me to a simple guide on how I can use it? For example, say I have a script/workflow where I use OpenRouter for LLM calls, and at some point I want to do a simple classification. Can I still use OpenRouter with some Jev model...?
[dead]
[dead]
I feel like good engineering doesn't just ignore those things, or at least it didn't before recently. Now I guess social media has added a pressure to reduce everything to a hot take.
Following AI from the academic papers side; jev really feels silly. They one-pass the LLM tranformer stack and tune the output network for a probability value.
(some clever pararellization optimisations to make it viable to offer as an api, since the normal kv cashing no longer works if you oneshot the tranformer)
The largest change is the packaging; An api with a tolken based pricing, and a schema to define the output structure for quick setup.
Previous projects would probably involve installing pytorch, running a converter script on Qwen, and write a fair bit of matrix math to change the output shape.
I'm kinda amused that it took this long though.
I agree with you on the "ease of use" business though. No one thought to make this sort of thing commercially available.
But there is no hot take here. Jev is not some new paradigm; engineering-wise, it is a trivial modification to the existing pipeline. That doesn't mean it isn't commercially viable.
Classic classifiers are regularly just tuned general models; Training a CCN on ImageNet and tune it for cats and dogs gives better results than just training it on cats and dogs.
There is likley a small network used to tranform model output vector to probabilities, but that wouldn't be massive. Retraining that small network for specific task may beat jev; but that's bairly considered training by modern standards.
LLM were used as classifiers because they solved that.
Jev have the flexibility of LLM and the perf and api of classifiers.
As a DevOps Engineer, I never once saw before the advantage of using a classifier. Now I see multiple parts of the stack where a better level of expressiveness will be useful (PR validations, Blue/Green validation, notification router for alerts, quick smoke tests, etc).
Nobody will give us the time and budget to build a custom classifier for these use cases, but a simple API call yes.
Being able to route prompt to features that then route to special models would be a really solid implementation.
I saw an article about 2+ years ago of a researcher using a small local AI strapped into excel to evaluate the abstract and intro of 10000 papers for "papers that research X in domain of Y", and let it loose.
jev is probably more capable avd faster than that workflow was, but saved one dude a few very grindy weeks for a litteratur review.
It's amusing how long it took, and much hype it gets for someone releasing the least revolutionary ML architecture in a new package. But i can see a fair few uses.
A LLM agent could be trained to use jev effectively as a tool call, even (but even without specific RL they do a good job already)
It doesn't have to be new, it just has to be consumable by devs.
You could send text before Twilio. You could process credit cards before Stripe.
Jev, at the end of the day is an easy to use API.
Everyone seems to forget that usability is a thing.
pangram says... 20% of content likely AI written, 80% of content likely human written.
Eventually humans are going to start writing like AI if we read enough of it.
Process: First, actually have ideas :D Then, I write an outline for what I want to talk about at basically a sentence-by-sentence level. (This is me yelling things at my computer.) And then I have the AI convert a chunk at a time into prose. I reread it and rework it to be my voice.
Then I have the AI help with things like subject titles and social posts.
¯\_(ツ)_/¯
I am not personally like philosophically or ethically opposed to having LLMs help or even write text... the issue is that I see so much LLM-written text that is just _bad_, and very hard for me to read or extract meaning out of, especially relative to it's often long length.
People think they are bad writers, but usually LLMs are actually worse (although they are great writers of catchy slogans and phrases, and then put together an article out of them, which I find just exhausting to try to get more than a "vibe" out of).
What you describe sounds like a fairly reasonable approach, but I suspect the parts the person I was replying to were reacting to was areas were you had not been as succesful at reworking it to be your voice. Which are probably also the parts pangram flagged as likely LLM written.
Pangram gives you a handful of free tokens, it would be interesting if you wanted to see what parts are the 20% pangram is flagging as LLM, and reflect on if they went through your process differently. Perhaps they were the parts you didn't spend quite as much time reworking it to be your voice. (I 100% believe you, because I've been running things through pangram a lot lately, and it's actually pretty rare for it to flag mixed content, instead of 100% likely AI or 100% likely human).
There was recently a post on HN that said if you want to avoid this, you really can't use any words at all that are written by the LLM, you can use it for suggesgting structure or points, or reviewing your work in various ways, but if you accept even a single phrase it provides... it's not going to be "reworked into your voice", it's going to be picked up by people (at least those of us who have become sensitive to it) as AI, because it's like, headline-speak.
(I can't find the article now, because I'm trying to quit facebook so can't log in to find my own post of it there, have to stop using that as bookmarks substtitue!)
Of course, that's not welcome advice if what you want AI for is "phrase choice".
I'm just here to say, LLMs are not good at phrase choice either. Although they may be quick at it. I feel like it's asking the reader to do the work of trying to extract meaning from slop that the author didn't have the energy to use to encode it well in the first place. I don't have time to try to read sentences that the writer didn't have time to write, i find myself bailing out quicker and quicker at signs of AI slop buzzword headline-speak.
prodigycorp is entitled to their opinion
[deleted]
LLMs already shell out and write code to solve certain problems. This is just a special case of that.
Note that I don't think OpenAI is incapable of doing it, but I just don't think they will bother with it.
And for Jev, everyone has a price, and OpenAI's raised an historical amount of funding.
They are all very open about it. It's likely the reason why the frontier labs are ok paying many more dollars a person in compute than they receive as revenue on subscription users for now.
Here is OpenAIs page
https://openai.com/policies/how-your-data-is-used-to-improve...
>When you share your content with us, it helps our models become more accurate and better at solving your specific problems
And as a bonus here is anthropics saying the same thing
https://privacy.claude.com/en/articles/10023580-is-my-data-u...
Especially if you're doing something in a very sparsely populated part of their latent space it just makes sense that it would get used. Their biggest problem getting solved at the moment is going beyond what common crawl enables.
I await the standard goal post moving.
[dead]
But Jev is works pretty much out of the box without any fine tuning.
Why build codex if AGI will replace SWEs?
Why build excel integrations if AGI will replace spreadsheets?
You have to run a legitimate business and make money. After the "takeoff," anything is fair game. It could result in value accruing to capital (lab shareholders) and everyone else is screwed, it could result in the status quo being maintained but people do way more and GDP goes up by a lot, it could result in a post scarcity society etc. Theres a reason why its called the singularity - you can't see past the event horizon.
In any case it is optimal for OpenAI to create good products that generate revenue NOW vs going for some straight shot pie in the sky model that is "AGI." SSI is attempting to do this and I suspect they are about as close as anyone else is, and I'd honestly assume they are likely further away than OpenAI or Anthropic at the moment. But one of these classes of companies has actual revenue that is arguably good for the economy, the other is just a glorified research lab that has produced nothing of value.
But you're completely right: our current APIs were optimized for humans, and some like Excel date from the 80s. So in the long term, agents will highly likely create their own interfaces optimized for when no human is in the loop.
Which is exactly why companies stick with current APIs for now. Fable 5.1 already emits alien-like reasoning traces, and the recent OpenAI agent swarm accident just proved this.
[deleted]
Is it? If AGI is here then by the time I test and deploy that the AGI will be most likely cheaper and smarter because it improved itself (for example by implementing it's own Jev for stupid prompts like this), so why invest into a more complex solutions?
Though for tasks where you are trying to search through billions of documents, social media posts, etc. and extract certain information, where each individual post is of low value and only the data in aggregate is valuable, then that’s where you’d want something cheaper and faster.
Such as if you want to look at all posts on X in the last few months and find how many have a negative or positive sentiment about the economy (or are unrelated).
Of course you could use a special-purpose model for this, but the whole point of something like Jev is to ask whatever questions you want without having to train something new.
1) It's very cheap and fast - you provide one input and many potential classifications, and the compute to ingest the input is shared.
2) It generates structured output natively - guaranteed to be correct
3) It's output probabilities are calibrated to actually mean something
OpenAI, or anyone else, could certainly replicate it - there are already articles guessing how Jev achieves its "parallel" classifications, but it seems the AI companies need to decide are they in the business of providing intelligence/tokens, or are they in the application business trying to compete with all their customers (not that Jev uses OpenAI).
Don't fall for marketing BS so easily.
Jev can output drastically different probabilities if you simply reorder the list of choices. And Jev's "confidence" output is fake/redundant - it's just a formula applied to probabilities, it conveys no additional information.
I bet they will eventually "fix" (read hide under the rug) the ordering problem by ordering the list on the backend before feeding to the model.
If it really matters to you whether whether some business-specific classification confidence is above/below some specific threshold (vs just relative order), then you'd be better off training or fine tuning a custom model for that. Maybe that is something that TypeSafe are planning to also provide?
> 2) It generates structured output natively - guaranteed to be correct
It's not guaranteed to be correct: it's guaranteed to be _formatted in a particular way_. You can get the same thing with grammars on any LLM.
Jev and Jev-like models have other advantages, but I feel like people forget grammars exist for LLMs.
Is this actually true ?
It has the advantage of speed and the confidence not being hallucinated.
But LLMs start to generalise on the pattern, rather than the classification that you want the more examples you have to train on.
LLMs start to break down as well the more classifications you have. Laya (Open source paper Jev is based on) even mentions that over 20 classifications and it starts to fail rapidly.
20 is around the level of sentiment analysis or minor intent routing. There are cheaper, smaller and easier ML models for that level of classification.
That is, if you force any llm to return json and a confidence it can also do that too and mostly likely it will he better at any one shot classification task than Jev.
LLMs have the great quality of knowing more due to the depth and richness of the training data. If Jev is trying to classify anything outside of its training data, it’s going to do a terrible job.
There are plenty of other ways to do zero shot classification that would result in more "token usage" (really just having to reprocess everything for each class), but the pricing and the way they describe it narrows it down somewhat.
I'm not following. There are way more tools in the world, and likely available on every machine, than can fit in any model context. One can create a small list, but that will be very org dependent. Programming is very open ended, I struggle to see the 'deterministic set of choices' here.
I'm looking at Jev/Kev like models for some control over tool calling and similar, but I also think that fine-tuning out behaviors (like qwen 3.8 wanting to use rkt all the time) may be better in the long run. Instead of having to check/change every tool call, we can refine the main LLM to have high enough likelihood of using the tools we actually have, that we won't need to check every call.
Jev is definitely useful, but also being held like a hammer in search of a nail right now, approaching OpenClaw like fanfare and hype levels. No one talks much about OpenClaw these days, I would not be surprised if Jev has a similar trajectory.
People don't want to believe something as unsatisfying as "Scaling up LLMs" can yield something as profound as AGI/be useful, and just hope that literally anything else can take their mindshare away, and this just happens to be the new rage. Along with clearly-not-frontier-level open source models, non-transformer based architectures, etc.
I think demand for tools that are more tailored for this type of integration is high. I don't really understand why Jev is supposed to get my company's decisions right more than an LLM, but regardless of the tech I think people are just excited about the possibility of iterating faster, more explainability, higher-level tools that are specifically created to help hone classifiers etc.
I find it slightly more helpful to say they generate plausibility
[deleted]
Yeah, that was the original idea with GPT, Generative Pre-trained Transformer, and earlier open pre-trained transformers.
Today people use AI via APIs rather then fine-tuning models by themselves and when someone provides this as an API they got excited.
(also most signs point to this being LLaDA 2.0-adjacent so throw in solving some substantial mid-training)
I think it's 100% a hot take to call what they built trivial. Or at least it used to be.
There was a time when that kind of stuff was something between sour grapes and cluelessness about the gap between an idea and an actual commercial product deployed at scale, but now that's just weirdly normalized.
In fact, if anything I'm the weirdo for repeatedly taking issue with the way people are trivializing it ¯\_(ツ)_/¯
https://benchmarkheaven.com/jev-models?w=100-0-0-0#jevc-weig...
Jev actually isn’t anywhere near the top. It even loses to open weight clones. This tells me that whatever their “calibration” dataset is, it doesn’t seem to be anything special.
https://benchmarkheaven.com/jev-models
You linked to some weird subtable that labeled: " Not the default — not the JevBench Score", that can only be reached after you see what I just linked... lmao are you really this hard up about things?
Also every single question (even in the hard set) is single dimensional?: https://github.com/fstandhartinger/jevbench/blob/main/datase...
Jeeze, this is getting sad. I guess after all the mass-psychoses where people thought pointless things are going to change the world, we were due for a mass-psychosis where something interesting just has to be pointless?
I see your point, but Jev doesn't exist in a vacuum. When one (like me) says "trivial", they mean it relative to other attempts and developments in the field, all of which require everything you've mentioned at minimum. Commercialising any product, and doing it well, is hard. But the R&D factor here is substantially more straightforward than almost any other product in its category, because there is no architectural breakthrough here.
That only makes sense if you try to rope in data previously used to establish the model's priors, but that wouldn't make sense in this context. That same additional data is what enables things like...
> use generalized models to generate ad hoc specialized classifiers.
Isn't this exactly what the bitter lesson is about?
This probably just means that I could have been reaching for that tool more often already. But in practice I wasn't, and this has opened my eyes to the potential opportunities there.
But makes issues for someone (or everyone) else?
One advantage of using generalist models is that the generalists are improving - regardless of whether you're doing anything about it.
Creative writing and Claude - amusing that you say that, given that Anthropic just went and tried to unfuck it in Opus 5.5 specifically. It is an example of a capability no one typically cares about, yes. No money in creative writing. But even there, we had gains in newer models.
Or use it through Vercel [0] with per call zdr.
https://vercel.com/changelog/typesafe-ai-jev-now-available-o...
Its quite obviously about good/bad faith use of user data.
> so why invest into a more complex solutions
In the meantime, today, in the real world, there are businesses wanting to automate well-defined business flows, who don't want some stroppy AGI with a mind of it's own to instead decide to hack into something, or reward hack and make the customer happy by just wire transferring $1M of company money into their account.
It does not change potential distribution in any means. It DROPS part of answer model returned directly.
The text generation model go wild because model relies on previous section it answered to continue later section. And because now it contain item model have no idea, it is completely screwed.
In the case you only require model to answer one of a,b,c,d and don't care about later segment at all. It don't really matter.
the underlying data set needs to be representative
The whole point of Jev/Kev is efficiency. If you can make decisions faster, cheaper, and more reliably that's the benefit. Even within the model architecture itself, you could use a classifier as a router within MoE to pick the expert that's likely to give the best outcome.
Classifiers like classical NNs require:
- annotated data, potentially a lot of it - training - inference
#2 and #3 aren’t a big deal if you have an ML engineer, but #1 will always be a potential headache no matter who you are. The tradeoff is that they could be quite fast, cheap, and you can get probabilities, not just classes.
With LLMs you get:
- zero shot classification (no dataset or training required) - potentially can use third party model providers like OpenAI off the shelf. Don’t even need to host your own model.
The downside to LLMs is that they are comparatively slow and expensive to traditional classifiers. Historically they also were prone to hallucination or malformed responses, though not as much these days. You also can technically get log-probs back, but these aren’t equivalent to the classifier probabilities.
Jev gets you the zero-shot, zero-infra benefits of LLMs, while being closer to the speed and cost of traditional ML classifiers, as well as both classification and probability responses.
Jev is ranked higher than others on the overall benchmark due to speed and/or cost, not accuracy.
Obviously it’s the speed and cost that make it compelling. The tradeoff is accuracy.
Enough to matter? Maybe, maybe not. It’s not like it’s way down the chart. It’s probably good enough for a lot of tasks.
[dead]
Sorry who else did everything I mentioned? I think the guy behind Laya tried after noticing Jev's traction... but the site's auth went down and has stayed down for a day now.
"substantially more straightforward than almost any other product in its category"
More straightforward than the spite projects based on constrained decoding? Or Laya with it's couple of days post-training ModernBERT?
-
I have no doubt other teams can build models like this and I've love for a frontier lab to give us an even smarter model with these ergonomics... but in the rush to show Jev what's up, we're mostly getting slop.
PS: I don't know anyone who's done anything of note who uses trivial like that. The commentariat do, and the "I could have done that" crowd do, but I don't pay much attention to them until they actually do the thing.
Let me put it this way. OpenAI and Anthropic have a slight moat over the Chinese labs because they have strong training data and the most advanced RL strategies. It will take the Chinese labs significant R&D effort to bridge that, especially in math (and there is a good chance they will, provided they want to).
Jev has no moat other than the fact that no one else has bothered to package a model in this way. Another lab could build a strong competitor very quickly if they want to put the effort in. That's the point of this post. There is no uncertainty about what they have done, nothing to figure out. Someone just needs to do it. I'm not sure what to say if you can't see the difference between the two. Jev is worth celebrating because of the idea to package it in this way. But it is not a paradigm shift and that is likely a problem for them.
Your question is something like
anwser only a,b,c,d for following question a. b. c. d....
the model output possibility of next character a: 0.8 b: 0.7 c: 0.3 f: 0.2 d: 0.1
If the list contains option you did not provide. The model is confused anyway, it don't matter if you use grammer to filter out the bad option or not, the answer is screwed already.
In your example, I would expect an LLM to do fine and if you have access to the raw logits you can measure whether or not it was confused and assign a confidence to the answer it gave.
I do think that Jev handles more than this though and, in my early testing, does things that are not easily accomplished with guided decoding techniques.
I ran it through MMLU a few days ago and it scored ~90% so seems to have a lot of general world knowledge trained in. Makes me think your speculation is right. I have some credits left, might try and think of an experiment. I saw a gist where someone was asking it which model it was and it was picking qwen a lot, but who knows...
Anyway, thank you for the interesting discussion!
and then you are going to ignore all the research and results that clearly show otherwise? why?
what might we infer about the importance of data from a learning algorithm like decision trees?
Existing datasets, different reward function.
I did, in the first days Jev came out, when people were bringing it up. Another assumption. Please review the HN commenting guidelines, the one which starts with "Please don't comment on whether someone read an article." is relevant here.
Nothing in that paper changes that ML algorithms are dependent on the training data. We can step back from Jev and algos to consider Bayes Theorem. If your sample is not representative of the population, your resulting statistics will be off. The same is true here. If the data you train a model like Jev with is not representative, the probabilities and confidences it outputs will not be representative.
What makes Jev interesting is that it works well out of the box across domains. What people who are well known in the field believe is that this is the result of Typesafe having a really good training data set. People are saying similar of MiMo-2.6 today.
https://news.ycombinator.com/item?id=49816899
https://www.alexmolas.com/2026/09/23/jev-cant-be-calibrated....
[deleted]
We both know who is
> just acting in bad faith at this point.
Take RL 101. This is a common pattern.
Another that uses dice rolling, coin flips, and an inventory level example to drive home the point that Jev's output are not real probabilities for outcomes.
https://news.ycombinator.com/item?id=49830385
> Take RL 101
I taught it (ML course; a day on RL, at a university), you should really stop making assumptions friend. Data quality and coverage matters in learning algorithms.
Here's one of the books used in that course https://amlbook.com/
Thinking blocks are not a place you can derive real confidence scores in LLMs
You are out of your depth and grasping at straws.
Do you have any credentials or evidence that others can use to determine if this statement is not more accurately describing the author who wrote it?
Perhaps a PhD in ML, research output like published papers, or teaching/professional experience - all things I have
We could debate the merits of the paper contents, but I suspect you have intentionally moved on to personal attacks. Regardless, nothing you have said (nor can be found in this paper) has been a counter argument that learning algorithms are sensitive to training data, where the measured output difference is used by the optimization algorithm when updating the parameters. Garbage in, garbage out is a saying for a reason. No algorithm fixes non-representative data.