I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Not reliably - look at microsoft's guidance project that did some of this.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma 4 26A4. That makes me optimistic for this release, despite the high cost.
Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
What got me into Claude Code started with Google Antigravity. Somewhere along the way Antigravity took a dive. The rollout was great, then it started feeling like other G-products, the dreaded tail of lackluster developer energy after a peak.
Startup / Founder focus vs Large Organizations with no vision.
With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
However, it can accept Audio and Video as input as well and for those specific cases it can be extremely useful.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
https://developers.google.com/edge/mediapipe/solutions/decis...
Standard practice by the big G.
It seems unlikely this will be around in 6 months at this rate.
Google is maintaining the AI models and APIs for much longer, even if many people don't know about them unless they use them to build things
I say plant a thousand flowers and cull the ones that don't bloom.