Gemma 4 31B

Amazing! shrinking size while high intelligence, Apache 2, works on Phones. This is the 2026 news we need. Keep going Google.

I installed Gemma 4 (gemma4:e2b 7.2GB) running locally derived a Constraint-Dynamical Hamiltonian for a Clifford algebra project I am working on. And you have to trust me what it provided was amazing... So I have a thinking model running locally using RAG to read files and can do quite advanced math... that is the absolute bomb.. :)

Gemma 4 welcome! 🎉 And thanks to everyone behind Gemma 4's development. We all appreciate the incredible work you all do.

Not surprised. Gemma is just a mini Gemini, it's good with that stuff. Where GLM 5.1 shines is coding.

I don't know how you ran it, if you're running it locally using llama.cpp, use the b8660 llama.cpp build (more recent versions have a regression, another tokenization issue) and use --temp 0.3 --top-p 0.9 --min-p 0.1 --top-k 20 I am sure the 26B will do much better. Also, Claude might favor better formatting etc., a boolean test is not good. Try the below prompt for the judge: I am benchmarking many AIs in many tasks. You are a judge. Go through them question by question, not LLM by LLM. Go through each question and, for every question, give all AIs a score out of 10, and be sure to be fair with them. Later, rank them all by their total score. MAKE SURE to evaluate them correctly, not based on vibe alone (check for misinformation, hallucinations, if they are useful or not, and not on formatting). PROMPT= AI 1: ... AI 2: ....

LLM as judge = no thanks. It also depends how you're running Gemma 4 for the test. The new custom parser for gemma 4 in llama.cpp b8665 has fixed it for me. Before, it failed the test of just being given the image below. Now it solves it.

Super excited about the direction things are going. Next generation will be frontier quality for most daily uses and fit on a single solid GPU like the Intel B70. A couple more turbo quant type advances and we're there on SOTA phones, prob two generations. Genuinely concerned about the economy if the AI takeoff is entirely agents running on edge devices and the major labs' trillions in capital goes stale, but very glad we're leaning towards the good path where AI won't be controlled by the few.

Gemma 4 is the first actual leap AI did in a "long" time. It makes it smaller but also use less computing power. I am running it on my PC and while it takes up 20gb its equivalent to a 400gb model… insane and on Apache 2.0 so you can make and sell any product you make with it.

Even since Gemma 2 it's been useful for being good at interacting instead of being a 'yes man' (girl). Agreeableness is a flaw and I don't like it in Qwen. (I'm absolutely right)

qwen3 coder next losing to the 4b at actual game logic is the most demoralizing benchmark result i've seen this week, playwright mcp doing the heavy lifting probably explains a lot of the variance here.

>Even Q8_0 shows a KL of 0.45 on long documents and 0.24 on non-Latin scripts. All categories roughly double from Q8_0 to Q5_K_S, but science and tool use remain the lowest throughout (0.07 and 0.08 at Q8_0). This looks like it's a significant finding. Most people assume Q8_0 to be virtually the same as BF16.

So what's the point of the post if you're not comparing it with anything? you can take a Q2-Q3 quant (still usable for some use!) and q4 cache and it would probably fit as well. ^ Nobody cares if it fits, everybody cares if the quality is there And if it fits = it's fast. Surprise.

the real test isn't tokens-per-second. it's whether the model still reads back its own output reliably after 256k. that's where these quants break.

Seems odd that the Q8 would perform worse than the Q4. Can you link the exact quants you tested?

Speeds all well and good but how badly does it suffer from the KV quant?

i know that on my RTX PRO 4500 32GB GPU, I get more context length (115K) with qwen3.5-27b, because its weight occupy less VRAM. Which is sorta important when you have a lobster living in your server.

The 'car wash test' is not very good, because it's the best known example of a nearly infinite number of embodied reasoning/common sense fails an LLM can make. Model makers can patch one such example in training, they cannot patch them all.

I've tried both MOEs - 26B and 35B. Both are significant downgrades compared to the dense models and loaded with SLOP. 26B is the worst culprit.

Has anyone else run the OPs prompt to check on their own? Not saying they DIDN'T get that answer, but it's just one data point. My experience has been that it is suddenly effing SLOW. Still smart, but damn, feels like walking when we used to be a racecar team.

For those interested, i ran Gemma 4 31B on my DGX, Q8 was doing 6tps and the Q4 was doing 10tps. It was loaded with full context window, but only using a tiny bit since i did a new chat with a few messages. Also to fully load the Q8 with full context, it was 101gb O_o