• Guides & Tutorials
• Run Local LLMs on Mac

Run Local LLMs on Mac

This guide walks you through running a local LLM server on macOS with an OpenAI-compatible API. You get full privacy, zero API costs, and surprisingly good performance on Apple Silicon.

We cover two backends:

brew install llama.cpp

Both expose an OpenAI-compatible/v1/chat/completionsendpoint. Hermes works with either one — just point it athttp://localhost:8080orhttp://localhost:8000.

/v1/chat/completions http://localhost:8080 http://localhost:8000

This guide targets Macs with Apple Silicon (M1 and later). Intel Macs will work with llama.cpp but without GPU acceleration — expect significantly slower performance.

Choosing a model​

For getting started, we recommendQwen3.5-9B— it’s a strong reasoning model that fits comfortably in 8GB+ of unified memory with quantization.

Memory rule of thumb:model size + KV cache. A 9B Q4 model is ~5 GB. The KV cache at 128K context with Q4 quantization adds ~4-5 GB. With default (f16) KV cache, that balloons to ~16 GB. The quantized KV cache flags in llama.cpp are the key trick for memory-constrained systems.

For larger models (27B, 35B), you’ll need 32 GB+ of unified memory. The 9B is the sweet spot for 8-16 GB machines.

Option A: llama.cpp​

llama.cpp is the most portable local LLM runtime. On macOS it uses Metal for GPU acceleration out of the box.

Install​

brew install llama.cpp

This gives you thellama-servercommand globally.

llama-server

Download the model​

You need a GGUF-format model. The easiest source is Hugging Face via thehuggingface-cli:

huggingface-cli

brew install huggingface-cli

Then download:

huggingface-cli download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-Q4_K_M.gguf --local-dir ~/models

Some models on Hugging Face require authentication. Runhuggingface-cli loginfirst if you get a 401 or 404 error.

huggingface-cli login

Start the server​

llama-server -m ~/models/Qwen3.5-9B-Q4_K_M.gguf \  -ngl 99 \  -c 131072 \  -np 1 \  -fa on \  --cache-type-k q4_0 \  --cache-type-v q4_0 \  --host 0.0.0.0

Here’s what each flag does:

-ngl 99 -c 131072 -np 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --host 0.0.0.0 127.0.0.1

The server is ready when you see:

main: server is listening on http://0.0.0.0:8080srv  update_slots: all slots are idle

Memory optimization for constrained systems​

The–cache-type-k q4_0 –cache-type-v q4_0flags are the most important optimization for systems with limited memory. Here’s the impact at 128K context:

--cache-type-k q4_0 --cache-type-v q4_0 | KV cache type | KV cache memory (128K ctx, 9B model) | | — | — | | f16 (default) | ~16 GB | | q8_0 | ~8 GB | | q4_0 | ~4 GB |

On an 8 GB Mac, useq4_0KV cache and choose a smaller model that can still fit Hermes’ 64K minimum context. On 16 GB, you can comfortably do 128K context. On 32 GB+, you can run larger models or multiple parallel slots.

q4_0

If you’re still running out of memory, reduce context only while staying at or above Hermes’ 64K minimum; otherwise switch to a smaller model or smaller quantization (Q3_K_M instead of Q4_K_M).

Test it​

curl -s http://localhost:8080/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{    "model": "Qwen3.5-9B-Q4_K_M.gguf",    "messages": [{"role": "user", "content": "Hello!"}],    "max_tokens": 50  }' | jq .choices[0].message.content

Get the model name​

If you forget the model name, query the models endpoint:

curl -s http://localhost:8080/v1/models | jq '.data[].id'

Option B: MLX via omlx​

omlxis a macOS-native app that manages and serves MLX models. MLX is Apple’s own machine learning framework, optimized specifically for Apple Silicon’s unified memory architecture.

Install​

Download and install fromomlx.ai. It provides a GUI for model management and a built-in server.

Download the model​

Use the omlx app to browse and download models. Search forQwen3.5-9B-mlx-lm-mxfp4and download it. Models are stored locally (typically in~/.omlx/models/).

Qwen3.5-9B-mlx-lm-mxfp4 ~/.omlx/models/

Start the server​

omlx serves models onhttp://127.0.0.1:8000by default. Start serving from the app UI, or use the CLI if available.

http://127.0.0.1:8000

Test it​

curl -s http://127.0.0.1:8000/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{    "model": "Qwen3.5-9B-mlx-lm-mxfp4",    "messages": [{"role": "user", "content": "Hello!"}],    "max_tokens": 50  }' | jq .choices[0].message.content

List available models​

omlx can serve multiple models simultaneously:

curl -s http://127.0.0.1:8000/v1/models | jq '.data[].id'

Benchmarks: llama.cpp vs MLX​

Both backends tested on the same machine (Apple M5 Max, 128 GB unified memory) running the same model (Qwen3.5-9B) at comparable quantization levels (Q4_K_M for GGUF, mxfp4 for MLX). Five diverse prompts, three runs each, backends tested sequentially to avoid resource contention.

Results​

What this means​

• llama.cppexcels at prompt processing — its flash attention + quantized KV cache pipeline gets you the first token in ~66ms. If you’re building interactive applications where perceived responsiveness matters (chatbots, autocomplete), this is a meaningful advantage.
• MLXgenerates tokens ~37% faster once it gets going. For batch workloads, long-form generation, or any task where total completion time matters more than initial latency, MLX finishes sooner.
• Both backends areextremely consistent— variance across runs was negligible. You can rely on these numbers.

llama.cppexcels at prompt processing — its flash attention + quantized KV cache pipeline gets you the first token in ~66ms. If you’re building interactive applications where perceived responsiveness matters (chatbots, autocomplete), this is a meaningful advantage.

MLXgenerates tokens ~37% faster once it gets going. For batch workloads, long-form generation, or any task where total completion time matters more than initial latency, MLX finishes sooner.

Both backends areextremely consistent— variance across runs was negligible. You can rely on these numbers.

Which one should you pick?​

Connect to Hermes​

Once your local server is running:

hermes model

SelectCustom endpointand follow the prompts. It will ask for the base URL and model name — use the values from whichever backend you set up above.

Timeouts​

Hermes automatically detects local endpoints (localhost, LAN IPs) and relaxes its streaming timeouts. No configuration needed for most setups.

If you still hit timeout errors (e.g. very large contexts on slow hardware), you can override the streaming read timeout:

# In your .env — raise from the 120s default to 30 minutesHERMES_STREAM_READ_TIMEOUT=1800

HERMES_STREAM_READ_TIMEOUT HERMES_STREAM_STALE_TIMEOUT HERMES_API_TIMEOUT

The stream read timeout is the one most likely to cause issues — it’s the socket-level deadline for receiving the next chunk of data. During prefill on large contexts, local models may produce no output for minutes while processing the prompt. The auto-detection handles this transparently.