On-Device LLMs: The 7B Sweet Spot

By 2026, a 7B-parameter quantised LLM runs comfortably on flagship phones and competently on laptops. The sweet spot for ‘local AI that works.’

Why 7B

The 7-8B parameter range hits a useful equilibrium. Small enough to fit in 4-8GB after quantisation (laptop or modern phone). Large enough to handle the long tail of basic tasks (Q&A, drafting, summarising, code completion at modest scale). At 3B and below, capability falls off sharply for general-purpose use; at 13B and above, memory and latency become problems for client devices.

The capability cliff at smaller sizes. 1B-3B models are strong at narrow tasks (translation, basic classification) but weak at multi-step reasoning and instruction following. The capability gap to 7B is bigger than the parameter gap suggests; 7B is roughly 2-3x more capable, not 2-3x.

The memory math at 7B. 7B in Q4 is ~3.5-4GB. Modern phones have 6-12GB RAM; modern laptops have 8-32GB. The model fits with room for OS and apps. At 13B (Q4 ~6.5GB), the fit is tighter; the OS sometimes evicts the model under memory pressure.

The latency math at 7B. On modern phone CPU/NPU, 7B runs at 5-15 tokens/second. On laptop CPU, 15-40 tokens/second. Both are interactive, first token in under 1 second; full response in seconds. Acceptable for chat-style UX.

The forecast. The 7-8B sweet spot is durable. Hardware improves; models improve at the same pace; the relative position of 7B as "fits everywhere usefully" stays stable. Plan around 7B as the current, two-year, and probably five-year edge target.

The model lineup

As of 2026, the strong 7B class includes Llama 3.1 8B, Mistral 7B v0.3, Phi-3 Mini (3.8B punching above weight), Qwen 2.5 7B, Gemma 2 9B (slightly larger but in the bucket). Each has slightly different strengths, Llama is general-purpose, Mistral is solid-mid, Phi punches above weight on reasoning, Qwen has multilingual edge.

The Llama 3.1 8B case. The default for most teams. General-purpose, well-supported, strong instruction following. License permits commercial use with restrictions. Works as a drop-in for most edge use cases.

The Phi-3 Mini case. Microsoft's small-model bet. 3.8B parameters punching at 7B-class quality on benchmarks. Excellent for severe size constraints (older phones, embedded). Quality is real but the small size shows on long-context or complex reasoning.

The Qwen 2.5 case. Alibaba's open-weights series. Strong multilingual capability, better Chinese/Japanese performance than Llama or Mistral. For non-English-primary use cases, Qwen often wins.

The Gemma 2 case. Google's open weights. 9B is slightly large for tight memory constraints but competitive on capability. Strong for code-related tasks; integrates well with the broader Google ML ecosystem.

The choice criterion. Test all four on YOUR task. Capability rankings on general benchmarks don't always translate to specific use cases. The first 30 minutes of evaluation tells you which model is best for your application.

Runtime stacks

Multiple options for running 7B on the edge:

• llama.cpp, the gold standard for CPU inference. Cross-platform, well-optimised, supports GGUF format. Most ports start here.
• MLX (Apple), optimised for Apple Silicon. Faster than llama.cpp on M-series chips. Limited to Apple platforms.
• ONNX Runtime, cross-platform, good for non-LLM models. Less LLM-specific optimisation than llama.cpp.
• TensorFlow Lite, strong on Android. Edge-deploy story improves yearly.
• Mobile-specific, CoreML (iOS), MLC-LLM (cross-platform with WebGPU). Fast-moving space.

The llama.cpp default. For most edge use cases, llama.cpp is the right starting point. CPU and GPU backends; mature; large user base. The ecosystem of pre-quantised GGUF models means you rarely need to quantise yourself.

The Apple Silicon edge. MLX achieves 30-60% better tokens/second than llama.cpp on M-series. For Mac/iPad/iPhone deployment, MLX is the right choice; the platform-specific optimisation is worth the platform constraint.

The Android landscape. TensorFlow Lite, MediaPipe, MLC-LLM all target Android. None has dominated yet. Android NDK + llama.cpp is also viable. Choose based on your team's existing Android tooling familiarity.

The Web case. WebGPU enables in-browser LLM inference. MLC-LLM and Transformers.js support this. Latency and capability are below native; the deployment simplicity (no app install) makes it valuable for some use cases.

Where on-device wins

Privacy-sensitive workloads (personal data never leaves the device). Offline scenarios (no connectivity). Cost-sensitive scaling (no API fees per request). Latency-sensitive UI (no network round-trip). Plus political/regulatory cases where "no data leaves the device" is the entire value proposition.

The privacy case. Personal assistants that read user emails, photos, messages can run fully on-device. Data never leaves; no cloud trust required. The UX is identical to cloud-backed assistants; the privacy story is dramatically better.

The offline case. Field workers without reliable connectivity. Travelers without roaming. Anyone in a connectivity-constrained environment. On-device LLMs work; cloud-backed ones don't.

The cost case. Free users who can't be served at API cost. High-volume features where per-call cost would dominate. Educational apps in cost-sensitive markets. On-device shifts cost from operating expense (per call) to one-time (model download + device compute).

The latency case. UI features needing sub-100ms response (autocomplete, real-time grammar). Network round-trip alone exceeds 100ms; cloud-backed isn't viable. On-device first-token latency on modern hardware is 50-200ms.

The regulatory case. Some jurisdictions require data to stay in-country or on-device. On-device LLMs satisfy the requirement structurally; no cloud routing decisions needed.

Common antipatterns

Targeting 13B for "better quality" without measuring fit. Many devices won't fit 13B comfortably. Validate target hardware before committing to model size.

Using FP16 on-device. Wasted memory and slower inference. Quantise to Q4 or Q5 for production.

One-time model installs without update path. Models improve; you'll want to update. Build the update mechanism into the app from day one.

Skipping tokens-per-second testing on real devices. Specs lie. Measure on the actual hardware you'll deploy to. Mid-range Android performance often surprises.

What to do this week

Three moves. (1) Pick your target device tier (top, mid, low). Measure tokens/second of Q4 7B on representative hardware. The number determines whether 7B is your target or if you need to drop to 3B. (2) Build a "model swap" mechanism into your app from day one. The model you ship with isn't the one you'll ship in 12 months. (3) Decide on cloud fallback. For complex tasks beyond 7B capability, falling back to cloud preserves the privacy story for routine tasks while delivering full capability when needed.