> ## Content Index
> Fetch the complete content index at: https://www.tao.media/llms.txt
> Use this file to discover other available public pages before exploring further.

# NVIDIA Shows How Compact Reasoning Models Can Run Locally on Jetson
- URL: https://www.tao.media/nvidia-shows-how-compact-reasoning-models-can-run-locally-on-jetson/
- Published: 2026-09-11T14:37:41.000Z
- Updated: 2026-09-11T14:37:41.000Z
- Description: NVIDIA says compact reasoning models are now practical to run fully on Jetson, so agentic AI can make decisions on-device instead of round-tripping to the cloud. In a new guide, it walks through picking dense vs. MoE models and speeding inference with NVFP4 quantization plus speculative decoding.
- Author: Antonio Verrico
- Tags: Nvidia, AI

[NVIDIA*,* ](https://www.nvidia.com/en-us/?ref=tao.media)an AI system architecture, has published new deployment guidance for running compact reasoning models locally on NVIDIA Jetson, a small GPU module built for edge inference. Smaller open models are starting to support workloads that previously depended on data center infrastructure.

The [technical blog](https://developer.nvidia.com/blog/frontier-reasoning-reaches-the-edge-how-to-deploy-and-optimize-models-on-nvidia-jetson/?ncid=so-twit-866469-vt48&linkId=100000437953386&ref=tao.media), published Sept. 4, focuses on how developers can deploy and optimize reasoning and agentic AI models on Jetson systems. NVIDIA uses Nemotron 3.5 Lightning and Qwen3.8-27B as its main examples and explains how architecture choices, quantization, and speculative decoding affect performance on edge hardware.

The guidance is relevant for robotics, industrial systems, remote monitoring, and other physical AI applications where sending every request to the cloud can add latency, cost, reliability constraints, or data exposure. Newer compact open models are making it more practical for agents to reason and act close to the sensors, machines, and environments they support.

![](https://storage.ghost.io/c/78/0b/780ba906-b1a7-4bf0-873c-bdd5c32e5331/content/images/2026/09/image-3.png)

**Edge-capable 2026 models (green) reach intelligence scores comparable to 2025 frontier models at a fraction of the parameter count*

## Why Edge Reasoning Matters

Reasoning models are useful because they can handle multi-step tasks, evaluate intermediate results, and make more structured decisions than simple text-generation systems. That capability matters for edge applications where connectivity is limited or where decisions need to happen near real-world equipment.

NVIDIA points to use cases such as in-cab assistants, real-time anomaly detection, and robots operating in harsh or remote environments. In those scenarios, a local reasoning loop could monitor sensor data and device logs, take approved corrective actions, verify outcomes against predefined tests, and escalate only when needed.

This local-first approach changes an AI system's architecture. Instead of treating the edge device as a thin client connected to a remote model, developers can run the model next to the operational data source. This can reduce network dependency and improve responsiveness, while keeping more information on device.

The tradeoff especially matters for robotics and industrial AI. A robot or field system may need to respond when connectivity is degraded, and some environments cannot afford to wait for round trips to a data center. Edge reasoning does not remove the need for cloud systems, but it gives developers another deployment option for latency-sensitive or privacy-sensitive workloads.

## Choosing Between Dense and Mixture-of-Experts Models

NVIDIA's guide compares two model architectures, Qwen3.8-27B and Nemotron 3.5 Lightning.

Qwen3.8-27B is a dense model, meaning it activates all 27 billion parameters for each token. Dense models can be attractive for tasks that require fewer but more difficult decisions, where generation speed is not the only priority.

Nemotron 3.5 Lightning uses a mixture-of-experts architecture. It has 30 billion total parameters but activates only 3 billion per token, which reduces active compute while retaining a larger overall parameter pool. NVIDIA describes it as a better fit for response-heavy workflows where faster token generation can shorten the overall agent loop.

The distinction matters because edge AI is constrained by memory, power, and throughput. A model that performs well in a general benchmark may not be the right choice for a particular deployed system. NVIDIA recommends benchmarking models against the decisions, tools, and response patterns required by the target application before selecting one.

The company also names different Jetson starting points for different hardware classes. Gemma 4 E4B is presented as a strong starting point for Jetson Orin Nano, while Nemotron 3.5 Lightning and Qwen3.8-27B are positioned for Jetson AGX Orin and Jetson AGX Thor.

## Quantization and Speculative Decoding Improve Throughput

The core optimization section focuses on two techniques: NVFP4 quantization and speculative decoding.

Quantization lowers the precision used in model operations and reduces the amount of data the GPU must move and process. NVFP4 can improve generation speed and reduce memory use while keeping quality close to BF16, the baseline format used in the comparison.

Speculative decoding improves performance differently. A smaller draft model proposes multiple tokens, and the main model verifies them together. If the main model accepts several proposed tokens, generation can move forward by more than one token in a single verification step.

NVIDIA's test results show that the combination can produce substantial performance gains on Jetson. When comparing, Nemotron 3.5 Lightning reached a 2.2x speedup with NVFP4 and a 3.37x speedup when NVFP4 was combined with DSpark speculative decoding. Qwen3.8-27B reached a 2.33x speedup with NVFP4 and a 6.28x speedup when combined with DFlash2.

![](https://storage.ghost.io/c/78/0b/780ba906-b1a7-4bf0-873c-bdd5c32e5331/content/images/2026/09/image-2.png)

**NVFP4 quantization and speculative decoding together deliver up to 6.28x decode throughput speedup over BF16 on Jetson*

The best speculative decoding method was different depending on the model used. Nvidia found that Nemotron 3.5 Lightning performed best with DSpark, while Qwen3.8-27B performed best with DFlash2, which was a major finding from the guide. Developers should not assume a single acceleration method will be optimal across model families.

The guide also provides example vLLM serving commands for both models, including configuration details such as reasoning parsers, tool-call parsers, model length, GPU memory utilization, KV cache settings, and speculative decoding parameters.

## Application Testing Still Determines Deployment Quality

NVIDIA emphasizes that model-level benchmarks are not enough to validate a production configuration. Performance can change based on workload type, even when the model and optimization method stay fixed.

To test that effect, NVIDIA measured the fastest configuration for each model across SpeedBench categories including writing, reasoning, summarization, and retrieval-augmented generation. Nemotron 3.5 Lightning with DSpark ranged from 123.01 to 138.02 output tokens per second, while Qwen3.8-27B with DFlash2 ranged from 27.69 to 34.44 output tokens per second.

Those results suggest that application-level validation is still necessary. A robotics assistant, industrial troubleshooting agent, or edge RAG system may produce different prompt patterns than a public benchmark. Developers need to test with representative prompts and evaluate whether the model preserves the behaviors that matter for their own data.

NVIDIA also outlines when custom training may be useful. Most applications can start with existing quantized checkpoints and draft models. If quantization reduces accuracy, developers can use quantization-aware training or quantization-aware distillation through NVIDIA Model Optimizer. If speculative decoding underperforms because the main model rejects too many draft tokens, teams can train a compatible speculator with representative application data.

Compact models can now run locally on Jetson. The harder question is how developers tune the full serving stack for a specific workload. Jetson now supports newer open models, optimized runtimes, quantized checkpoints, and speculative decoding methods, which give developers a more complete foundation for local agentic AI.

That foundation could make local reasoning a more realistic design choice for physical AI systems. How much it helps will depend on the workload, but NVIDIA's guide shows that model architecture and inference optimization are now central to bringing agentic AI closer to the machines and environments where it operates.