smolbenchmark · edge inference

Models that fit in 8 GB

Ranked by decode speed, tokens per joule, and heat — on your own hardware: tablets, phones, Macs, Jetsons, and Pis. Jetson is live; Pi, phones, and Mac minis are still cooking. The chart is decode tok/s vs output tok/J: fast isn’t enough if every token costs too much energy.

0 families 0 configs 0 live

Families

Detailed reports

Full analysis and reports are available on the blogs:

#	Model	Quant	Backend	Mode	tok/s	tok/J	TTFT	ITL	E2E	prefill	W	TJ°C	ctx	gen

How it’s done

Same shape on every board: load the model, lock the power mode, sweep fixed prompt × gen cells with aiperf, then align power/thermal logs. Details live in each device README; this page is the short map.

Load generator: aiperf, concurrency 1, synthetic prompts at fixed token counts (usually 20 reqs per cell).
Headline metric: output tok/J - formula differs by hardware (decode-phase energy on Jetson/Mac posts; throughput ÷ average power on Pi). Do not cross-compare boards on tok/J alone.
Artifacts under artifacts/<backend>/<model>/gen<G>/ctx<P>/; --resume skips finished cells.
Jetson is live here; Mac and Pi use the same procedure in-repo and land as those runs publish.

General flow

01Prepare Build or install the backend(s). Pull GGUFs (and MLX packs on Mac) via hf. Activate the device’s aiperf venv.
02Stabilize Lock power mode / clocks where the board supports it. Start power sampling. Relaunch inside tmux for multi-hour runs.
03Serve One server process per model (llama.cpp, Ollama, or MLX-LM). Smoke a short request before the sweep.
04Sweep aiperf drives prompt × gen cells (typically 20 reqs each, concurrency 1). Failed cells are logged; --resume skips done work.
05Record Align power/thermal logs to each cell. Write profile_export_aiperf.json under artifacts/<backend>/<model>/gen<G>/ctx<P>/, then charts/report.

By machine

Open a board for the concrete stack, sweep, and how to run it from that folder’s README / .sh.

Jetson Orin Nano Super

Folders: benchmark-jetson-nano-orin-super/single-node/non-reasoning-models/ (bench-non-reasoning.sh), bonsai-models/, plus multi-node MoE RPC. Hardware: Orin Nano Super 8 GB, JetPack R36.x / CUDA 12.x, Ampere SM_87.

Stack

llama-server with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87, and/or Ollama on the same local GGUFs.
aiperf from ~/venv or ~/aiperf-env.
sudo nvpmodel / jetson_clocks for modes 7W · 15W · 25W · MAXN; tegrastats for VDD_CPU_GPU_CV and junction temps.

Sweep

Non-reasoning: prompt ∈ {128, 512, 1024, 2048} × gen ∈ {64, 128, 256}, 20 requests per cell. Bonsai: prompt ∈ {256..2048} × gen ∈ {128, 256, 512}. Tiny instruct models (~135M-1.2B) plus Bonsai / Ternary-Bonsai sizes.

Tok/J

Published Jetson posts use decode-phase energy: OSL ÷ (decode power × decode time). Scripts also log whole-window rail averages from tegrastats.

Run

bash bench-non-reasoning.sh --power-mode 1 (25W sweet spot). Flags: --backend llamacpp|ollama|both, --only, --reqs, --resume, --dry-run. Auto-tmux session non-reasoning-bench.

Canonical leaderboard cells (ctx 2048 / gen 256 or 512) match the SmolHub Jetson post tables for tok/s and tok/J.

Mac Mini M4

Folder: benchmark-mac-mini-m4/non-reasoning-models/ (benchmark_non_reasoning.sh); MoE suite under mixture-of-experts/. Hardware: Mac Mini M4, 16 GB unified memory, macOS Sequoia+.

Stack

llama.cpp Metal (-DGGML_METAL=ON) serving Q4_K_M GGUFs, or MLX-LM 4-bit via mlx_lm.server (--backend mlxlm).
Shared venv at ~/Desktop/smolbenchmark/venv/ with aiperf installed from git+https://github.com/ai-dynamo/aiperf.git (PyPI aiperf is a yanked stub).
Power via sudo powermetrics (CPU + GPU + ANE); skip with --no-power. Peak RSS sampled per combo.

Sweep

prompt ∈ {256, 512, 1024, 2048, 4096} × gen ∈ {256, 512, 1024} (15 cells), context 6144, ~10 dense instruct models (Granite / Nemotron / Qwen / Gemma 3 class). Same aiperf methodology for both backends so rows stay comparable.

Tok/J

Decode-only: OSL_p50 / (decode_power_W × p50_decode_s), using per-request phase timestamps from the aiperf export to classify powermetrics samples.

Run

bash benchmark_non_reasoning.sh (bash 4+; auto-tmux). Flags: --backend llamacpp|mlxlm, --stream (download/bench/delete one model for disk), --only, --resume, --no-power. Unlike Jetson/Pi, a failed aiperf combo aborts the run - fix and --resume.

Gated HF models (e.g. Gemma 3) need hf auth login and license acceptance first.

Raspberry Pi 5

Folder: benchmark-raspberrypi5/ (benchmark_all_cpu.sh). CPU-only tiny LLM sweep with llama.cpp and/or Ollama.

Stack

llama.cpp built with BLIS (-DGGML_BLAS_VENDOR=FLAME -DGGML_NATIVE=ON). Thread env at server launch: BLIS/OpenBLAS/OMP = 1 thread, LLAMA_ARG_THREADS=4.
Ollama pinned to v0.24.0 (later 0.30.x rejects several benchmark GGUFs). Script sets OLLAMA_NUM_PARALLEL=1, flash attention, single loaded model.
aiperf from ~/venv or ~/aiperf-env.

Stabilize clocks

Manual (not done by the script): lock 2400 MHz via force_turbo=1 / arm_freq=2400 in config.txt and performance governor. Check vcgencmd get_throttled / measure_temp before and after - soft throttle ~80 °C.

Sweep

Same shape as Jetson non-reasoning: prompt ∈ {128, 512, 1024, 2048} × gen ∈ {64, 128, 256}, 20 requests, concurrency 1. Backends: llamacpp|ollama|both.

Tok/J / power

PMIC via vcgencmd pmic_read_adc every 500 ms (sum of measurable rails; undercounts wall power ~10-15%). Per-combo power.log. Report tok/J uses throughput over average power for the cell window.

Run

bash benchmark_all_cpu.sh. Flags: --backend, --only, --reqs, --resume, --dry-run. Auto-tmux. Artifacts under artifacts/blog-all-YYYYMMDD-HHMM/.

Models that fit in 8 GB

Coming soon

Detailed reports

Problem

Vision

What we do

How it’s done

General flow

By machine

Stack

Sweep

Tok/J

Run

Stack

Sweep

Tok/J

Run

Stack

Stabilize clocks

Sweep

Tok/J / power

Run