Workloads
Local models
Size unified memory for a model, then run it with MLX or llama.cpp.
On Apple silicon the CPU and GPU share one pool of memory. A model runs well when its weights, context, and cache fit in that pool with room to spare.
Rough sizing
Weights take about the parameter count times the bytes per parameter. Leave 20 percent on top for context and the runtime.
| Model | 4-bit | 8-bit | 16-bit |
|---|---|---|---|
| 8B | 5 GB | 10 GB | 19 GB |
| 32B | 19 GB | 38 GB | 77 GB |
| 70B | 42 GB | 84 GB | 168 GB |
| 405B | 243 GB | 486 GB | 972 GB |
The model fit tool on the home page checks every config in the fleet for you.
Run with MLX
python3 -m venv .venv && source .venv/bin/activatepip install mlx-lmmlx_lm.generate --model mlx-community/Meta-Llama-3.1-70B-Instruct-4bit --prompt "Hello"