Self-Hosting AI on Apple Silicon (M4, 2026)

Apple Silicon (Apple's own M-series chips, which put the CPU, GPU and memory on one package) shares a single high-bandwidth pool of memory across the whole chip. That one design choice is why a Mac can run large models that would need an expensive dedicated GPU on a PC - and why the buying decision comes down to memory, not benchmarks.

Two things, in this order. First, unified memory sets the ceiling on model size. Because the CPU, GPU and Neural Engine all share one pool of RAM, the GPU can use almost all of it to hold model weights - no copying between separate memory banks. So the biggest model you can run is roughly your total RAM minus a few gigabytes for the system and your context. A 32GB Mac comfortably holds a model of around 28GB.

Second, memory bandwidth sets the speed. Local LLM inference is bandwidth-bound, not compute-bound: the chip spends its time reading weights from memory, so how fast it can do that - not how many GPU cores it has - decides your tokens per second. This is why the headline GPU-core counts matter far less than the memory specs when you are buying for AI.

Model files are sized by their parameter count and how aggressively they are quantized (compressed - 4-bit is the common sweet spot, trading a little quality for a much smaller footprint). These are practical, real-world tiers, leaving headroom for the OS and a reasonable context window.

For local AI the chips line up by memory bandwidth and maximum RAM rather than by name. An M4 or M4 Pro Mac mini with 32 to 64GB is the value sweet spot for most people - quiet, cheap to run, and capable of genuinely useful models. An M4 Max (around 546 GB/s of bandwidth) pushes speed and RAM higher for heavier use. The M3 Ultra sits at the top, with around 800 GB/s of bandwidth and up to 192GB of unified memory - that extra bandwidth translates fairly directly into faster generation on the same model, and the huge RAM ceiling unlocks the largest models.

For most home setups, a Mac mini with as much RAM as you can justify beats a faster chip with less memory. If you are weighing a Mac against a small PC for this job, the best mini PCs for local LLMs guide covers the other side of that decision.

This is the choice that most affects speed for free. llama.cpp is the long-standing cross-platform engine; MLX is Apple's own framework, tuned for Apple Silicon. On mid-sized models, MLX generates noticeably faster - commonly 20 to 50 percent quicker than llama.cpp - because it is built for the hardware. Above roughly 27B parameters the gap narrows, because memory bandwidth becomes the bottleneck for both and the runtime matters less.

The good news is you no longer have to choose a harder tool to get MLX speed: recent versions of Ollama added an MLX backend, so you can keep the easy Ollama workflow and still get the Apple-optimised speed. One honest caveat - MLX's decode-speed lead does not always carry over to very long contexts, where the initial prompt-processing step can dominate the wall-clock time. For everyday chat and coding at normal context lengths, MLX is the faster default.

The fastest path to a working local AI on a Mac is Ollama for the engine and Open WebUI for the interface. The steps below get you from nothing to a private chat assistant.

For a fuller walkthrough of the Ollama side, the Ollama setup guide goes step by step. And if you are weighing whether to keep AI local at all versus using the cloud, cloud LLM vs local LLM lays out the cost and privacy trade-offs.

Speeds vary a lot by model size, quantization and context length, so treat any single tokens-per-second figure with caution. The pattern that holds: smaller models on higher-bandwidth chips feel instant, and speed drops as the model grows because you are reading more weights per token. A well-matched setup - say an 8B model on a 16GB-plus Mac with MLX - is comfortably fast for interactive chat; push to a 70B model and you trade speed for capability.

The common mistakes are predictable. Buying for GPU cores instead of memory. Running the default backend and leaving MLX's free speed on the table. Loading a model too large for your RAM, which forces swapping and tanks performance. And maxing out the context window when you do not need it, which eats memory and slows the first response. Match the model to your RAM, use MLX for everyday work, and a Mac is one of the most pleasant ways to run AI privately at home.