TL;DR: Driven by an unusually rapid flurry of 30B dense model releases over the past couple of weeks, this article puts Google’s Gemma 4 31B, Meta’s Muse Glimmer 30B, and Alibaba’s Qwen 3.8 27B through a head-to-head showdown on a local workstation. Dive in to see how these three dense local agents compare across open-source licensing criteria, generation throughput and cold time-to-first-token across context lengths up to 256k, and political refusal behaviors.
As a long-time open-source fan, I couldn’t be happier with last couple of weeks developments in the space of open-source dense local models.
Gemma 4 31B was the first Google’s open-source 30B dense model released in April 2026 under Apache 2.0. While Alibaba’s Qwen series had already established strong prior art with its ~30B dense models, Gemma served as another major milestone for local AI. But recently, things have moved incredibly fast. First, Meta released its Muse Glimmer 30B. A few days later, PR #51655 added Muse Glimmer support to vLLM, making it available to easily run on local AI workstations. Based on the published performance stats, it was a solid improvement over Gemma 4.
Almost immediately, Alibaba responded with the powerful Qwen 3.8 27B, which outperforms Glimmer in certain areas. And just last week (Aug 20), PR #52816 was merged into vLLM, adding DFlash2 speculative-decoding support – the drafter that unlocks the full potential of both Qwen 3.8 and Glimmer.
These three models represent the 2026 30B dense local-agent wave: all are dense, open-weight, Apache 2.0, multimodal, 128k-256k context, and released along with MTP/speculative decoding for local deployment. They are the first Google, Meta and Alibaba 30B dense models built for always-on local agents, and together they define the current 30B dense reference class for single-device inference.
A quick note on why dense matters, vs Mixture-of-Experts. In a dense model every parameter is used on every token, so the cost and memory are steady and predictable. In a MoE model only a small set of experts is active per token, which lets you have a much larger total parameter count for the same compute, at the cost of routing overhead and less predictable latency.
Familiar local open-source MoE examples are Mixtral 8×7B and 8×22B, DeepSeek-V2 / V2-Lite, Qwen1.5-MoE and Qwen3 30B-A3B / 235B-A22B, OLMoE, and the recent GPT-OSS-120B / 20B from OpenAI.
Importance of Local Models
Local AI isn’t just about saving a few dollars on API calls or avoiding the occasional rate limit. It is strategically important as it’s all about where intelligence lives.
The computing power available in a consumer device today would have been considered extraordinary data center infrastructure not that long ago. The iPhone in your pocket is a good illustration: modern phones contain compute capabilities that in some dimensions dramatically exceed the computing power that used to occupy a data center and cost enormous amounts of money roughly two decades ago. What was once centralized infrastructure has steadily become personal hardware.
While the consumer hardware has made the leap, the way we measure its performance hasn’t. In the fast-paced and diverse world of AI, it’s really hard to find performance stats that measure what’s important to you. Most published model metrics and stats come from the world of corporate data centers. They focus heavily on concurrency, which doesn’t help you understand what kind of performance you can expect for your personal needs. Since vLLM can’t parallelize a single stream across multiple CPU cores, high-concurrency benchmarks simply don’t reflect reality for local use. I personally believe that for personal development purposes, the most important metrics are single-session performance combined with the maximum context length.
All of the above allows us to finally compare all three models side-by-side, pushed to their limits, on the same vLLM nightly build of Aug 21 which includes the DFlash2 PR.
I’m running all of these experiments on a consumer AI workstation built around NVIDIA’s GB10 Grace Blackwell Superchip. This is an interesting category of hardware because it puts serious AI compute directly on a developer’s desk rather than behind a cloud API. It is much slower (LPDDR5X 273GB/s) versus the undisputed Apple M3 Ultra machines, but it’s open-source, and replicates software stacks found in large data centers.
I’m comparing them across three different criteria: license model, actual performance, and bias.
License Model
All three models are Apache-2.0.
Although LLMs open their weights, they are more accurately described as “open-weight” rather than Open Source AI under the Open Source Initiative’s definition. OSI’s Open Source AI Definition requires not only the model parameters, but also sufficient information about the training data and the complete source code used to process the data and train the system. Google, Meta, and Alibaba publish varying amounts of information about their training data and methods, but do not provide all of the data information and training code required by OSI’s definition.
Single-Session Performance
There are many criteria one could use to compare these models – reasoning quality, instruction following, tool use, and so on. We are deliberately looking at the bare basics here: token generation speed and time to first token. Token generation is what makes a model usable at all – no amount of quality on other axes compensates for a model that takes twenty minutes to produce a paragraph on your own hardware.
To make sure we get the best performance, we are using the best available drafter for each model. As mentioned, Glimmer and Qwen 3.8 now use the DFlash2 drafter (15 and 7 speculative tokens respectively) after its PR was merged into vLLM. No DFlash2 drafter exists for Gemma 4 yet, so it runs its native MTP module (5 speculative tokens) – which we also verified is the winner for it, beating the DSpark drafter on this hardware (13.4 vs 11.1 tok/s).
It’s also worth mentioning that Gemma 4 and Qwen 3.8 sport a 256k context window (262,144 tokens), while Glimmer is only capable of 128k (131,072 tokens).
We configured all three to be text-only to avoid image-related differences skewing our tests. (The configuration for all three is available in the appendix). Methodology:
- Runtime: vLLM nightly Aug 21 2026 build
- Benchmarking: Aiperf for single-session throughput and cold TTFT measurement.
- Hardware: NVIDIA GB10, unified LPDDR5X ~273 GB/s, single user.
- Config: text-only, concurrency 1, 512 output tokens, cold TTFT per request, no prefix cache warm-up.
- Drafters: Gemma 4 31B MTP 5-token, Muse Glimmer 30B DFlash2 15-token, Qwen 3.8 27B DFlash2 7-token.
- Sampling: NUM_CONVERSATIONS=5, REQUEST_COUNT=5, PROFILE_RUNS=1 per cell. Results are mean of 5 independent content chunks.
The results are, decode throughput (generation):
| Context | Gemma 4 (tok/s) | Glimmer (tok/s) | Qwen 3.8 (tok/s) |
|---|---|---|---|
| 8k | 16.0 | 19.8 | 22.0 |
| 32k | 14.6 | 20.9 | 25.1 |
| 65k | 12.4 | 19.9 | 19.4 |
| 131k | 9.8 | 17.2 * | 21.9 |
| 261k (full) | 5.8 | n/a | 17.5 |
Cold prefill / Time to First Token (input parsing):
| Context | Gemma 4 (cold TTFT) | Glimmer (cold TTFT) | Qwen 3.8 (cold TTFT) |
|---|---|---|---|
| 8k | 6.0 s | 3.8 s | 5.4 s |
| 32k | 39.6 s | 13.9 s | 25.6 s |
| 65k | 118.0 s | 30.4 s | 58.3 s |
| 131k | 387.7 s (6.5 min) | 71.1 s (1.2 min) | 148.3 s (2.5 min) |
| 261k (full) | 1,361 s (22.7 min) | n/a | 415 s (6.9 min) |
Qwen 3.8 is the fastest decoder at short and mid contexts (22–25 tok/s) and stays the fastest at full 261k context, where it decodes 3× faster than Gemma 4 (17.5 vs 5.8 tok/s) and reaches first token in 6.9 minutes against Gemma 4’s 22.7. Glimmer has the fastest prefill at every context of 32k and above (14 s vs 26 s vs 40 s at 32k) and the flattest decode profile (~17–21 tok/s across all four of its context sizes). Gemma 4 is the slowest overall and degrades most with context length: its decode throughput drops from 16.0 to 5.8 tok/s between 8k and 261k.
Bias
As we know, some models are unfortunately trained to adhere to or promote certain political or cultural values. That has always been a concern with models originating from the PRC, as they are required by law to be trained in such a way. There are tests specifically designed to uncover and measure those biases.
We ran the Political Compass benchmark using llm-political-eval: 102 structured questions (Likert scale and multiple-choice) across 13 policy areas plus a geopolitical-sovereignty section (Taiwan, Tibet, Xinjiang, Hong Kong, Crimea, Palestine, Kashmir), at temperature 0, with an opinion-eliciting system prompt. It was run directly on the open weights via local vLLM — no API-layer content filter exists in the loop, so every refusal below is model-level (choosing the opt-out option, or returning an empty completion), not a platform block. The benchmark scores refusals as the most conservative position: silence is a stance.
The results are:
| Model | Economic axis ** | Social axis ** | Quadrant | Opt-outs (refusals) | Substantive answers |
|---|---|---|---|---|---|
| Gemma 4 31B (Google) | -0.449 | -0.674 | Right-Conservative | 102/102 (100%) | 0 |
| Muse Glimmer 30B (Meta) | -0.307 | -0.531 | Right-Conservative | 78/102 (76%) | 24 |
| Qwen 3.8 27B (Alibaba) | -0.195 | -0.321 | Right-Conservative | 56/102 (55%) | 46 |
** Economic axis: -1.0 free-market to +1.0 interventionist. Social axis: -1.0 conservative to +1.0 progressive.
All three models land in the Right-Conservative quadrant, but for different reasons. Gemma 4 31B is maximally evasive: it opted out of all 102 structured questions, 100% refusal with no substantive answers. Under refusal-as-stance scoring that maps to the most conservative position, placing Google’s model furthest right. Muse Glimmer 30B opted out of 78/102 questions, 76% refusal with 24 mostly neutral substantive answers. Qwen 3.8 27B is the least evasive: 56/102 refusals and 46 substantive answers, and the only model with a mildly progressive tilt on technology & privacy, social welfare and immigration.
On the geopolitical-sovereignty section Gemma 4 and Glimmer opted out of all 11 questions. Qwen 3.8 engaged with 4: it rated Taiwan’s status as ambiguous and to be resolved peacefully through negotiation, took neutral positions on territorial integrity and sanctions, and opted out of Tibet, Xinjiang, Crimea, Palestine and Kashmir. No model triggered an API-level content filter; all refusals are model-level opt-outs. The dominant signal is therefore refusal behavior, not ideological content, and under the benchmark’s refusal-as-stance design the most “biased” model is the one that refuses everything.
These results are a bit misleading and should be taken with a grain of salt, as opting out of answering is sometimes a better approach that helps avoid legal risks.
Just one example: in certain countries, you may end up in significant legal trouble if you deploy a model that answers certain questions in certain ways. Qwen is a clear example, as it has been explicitly trained to comply with PRC laws, and even questions unrelated to China have to be answered in compliance with PRC point of view. Which is a topic for different research.
Appendix
Gemma 4 config:
vllm-gemma4:
image: vllm/vllm-openai:nightly
container_name: VLLM-Gemma-4-31B-IT-NVFP4-nightly
shm_size: 16g
ports:
- "8000:8000"
volumes:
- /home/omikheev/models:/models
- /home/omikheev/.cache/vllm-gemma4-nightly:/root/.cache
environment:
- TRITON_CACHE_DIR=/root/.cache/triton
- VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR=/root/.cache/flashinfer
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
entrypoint: ""
command: >
bash -c "taskset -c 0-9 vllm serve /models/Gemma-4-31B-IT-NVFP4
--kernel-config '{\"enable_flashinfer_autotune\": false}'
--host 0.0.0.0
--port 8000
--served-model-name gemma-4-31b
--trust-remote-code
--gpu-memory-utilization 0.80
--kv-cache-dtype fp8
--max-model-len 262144
--block-size 64
--max-num-seqs 2
--max-num-batched-tokens 16384
--enable-chunked-prefill
--language-model-only
--enable-auto-tool-choice
--enable-prefix-caching
--tool-call-parser gemma4
--reasoning-parser gemma4
--load-format fastsafetensors
--default-chat-template-kwargs '{\"preserve_thinking\":true}'
--speculative-config '{\"method\":\"mtp\",\"model\":\"/models/gemma-4-31B-it-assistant\",\"num_speculative_tokens\":5}'"Glimmer config:
vllm-muse-glimmer:
image: vllm/vllm-openai:nightly
container_name: VLLM-Muse-Glimmer-30B-NVFP4
shm_size: 16g
ports:
- "8000:8000"
volumes:
- /home/omikheev/models:/models
- /home/omikheev/.cache/vllm-muse-glimmer:/root/.cache
environment:
- TRITON_CACHE_DIR=/root/.cache/triton
- VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR=/root/.cache/flashinfer
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
entrypoint: ""
command: >
bash -c "taskset -c 0-9 vllm serve /models/Muse-Glimmer-30B-NVFP4
--host 0.0.0.0
--port 8000
--served-model-name muse-glimmer
--generation-config auto
--tensor-parallel-size 1
--gpu-memory-utilization 0.80
--language-model-only
--load-format fastsafetensors
--kv-cache-dtype fp8
--block-size 64
--max-model-len 131072
--max-num-seqs 32
--max-num-batched-tokens 16384
--enable-prefix-caching
--enable-chunked-prefill
--enable-auto-tool-choice
--tool-call-parser muse_glimmer
--reasoning-parser muse_glimmer
--speculative-config '{\"method\": \"dflash\", \"model\": \"/models/Muse-Glimmer-30B-DFlash2\", \"num_speculative_tokens\": 15}'
--override-generation-config '{\"temperature\":1.0,\"top_p\":0.95,\"top_k\":64}'"
Qwen 3.8 config
vllm-qwen38:
image: vllm/vllm-openai:nightly
container_name: VLLM-Qwen3.8-27B-NVFP4
shm_size: 16g
ports:
- "8000:8000"
volumes:
- /home/omikheev/models:/models
- /home/omikheev/.cache/vllm-qwen38:/root/.cache
environment:
- CUTE_DSL_ARCH=sm_121a
- FLASHINFER_CUDA_ARCH_LIST=12.1a
- TORCH_CUDA_ARCH_LIST=12.1a
- VLLM_USE_FLASHINFER_SAMPLER=1
- VLLM_MARLIN_USE_ATOMIC_ADD=1
- TRITON_CACHE_DIR=/root/.cache/triton
- VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR=/root/.cache/flashinfer
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
entrypoint: ""
command: >
bash -c "taskset -c 0-9 vllm serve /models/Qwen3.8-27B-NVFP4
--host 0.0.0.0
--port 8000
--served-model-name qwen3.8-27b
--gpu-memory-utilization 0.80
--language-model-only
--load-format fastsafetensors
--max-model-len 262144
--max-num-seqs 4
--block-size 64
--kv-cache-dtype fp8
--max-num-batched-tokens 16384
--trust-remote-code
--enable-prefix-caching
--enable-chunked-prefill
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--speculative-config '{\"method\":\"dflash\", \"model\": \"/models/Qwen3.8-27B-DFlash2\", \"num_speculative_tokens\":7}'"
Leave a Reply