hckrnws

IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.

IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.

I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)

I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.

However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.

[1] https://openai.com/index/clip/

I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).

That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.

You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.

The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).

https://benchmarkheaven.com/jev-models

https://huggingface.co/spaces/multimodalart/jev-decision-ind...

A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.

Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."

There does definitely seem to be something there with respect to Contrastive Language Models.

They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...