Run Hermes Locally with Ollama — Zero API Cost

The Problem​

Cloud LLM APIs charge per token. A heavy coding session can cost $5–20. For personal projects, learning, or privacy-sensitive work, that adds up — and you’re sending every conversation to a third party.

What This Guide Solves​

You’ll set up Hermes Agent running entirely on your own hardware, usingOllamaas the model backend. No API keys, no subscriptions, no data leaving your machine. Once configured, Hermes works exactly like it does with OpenRouter or Anthropic — terminal commands, file editing, web browsing, delegation — but the model runs locally.

By the end, you’ll have:

• Ollama serving one or more open-weight models
• Hermes connected to Ollama as a custom endpoint
• A working local agent that can edit files, run commands, and browse the web
• Optional: a Telegram/Discord bot powered entirely by your own hardware

What You Need​

Ollama runs on CPU-only servers. A 9B model on a modern 8-core CPU gives ~10 tokens/sec. A 31B model on CPU is slower (~2–5 tokens/sec) — each response takes 30–120 seconds, but it works. A GPU dramatically improves this. For CPU-only setups, widen the API timeout via the env var (it’s not aconfig.yamlkey):

config.yaml

# ~/.hermes/.envHERMES_API_TIMEOUT=1800   # 30 minutes — generous for slow local models

Step 1: Install Ollama​

curl -fsSL https://ollama.com/install.sh | sh

Verify it’s running:

ollama --versioncurl http://localhost:11434/api/tags   # Should return {"models":[]}

Step 2: Pull a Model​

Choose based on your hardware:

gemma4:31b gemma2:27b gemma2:9b llama3.2:3b

Hermes is anagenticassistant — it edits files, runs commands, and browses the web through tool calls. Models without tool-call support can only chat; they can’t take actions. For the full Hermes experience, use a model that supports tools (likegemma4:31b).

gemma4:31b

Pull your chosen model:

ollama pull gemma4:31b

You can pull several models and switch between them inside Hermes with/model. Ollama loads the active model into memory on demand and unloads idle ones automatically.

/model

Verify the model works:

curl http://localhost:11434/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{    "model": "gemma4:31b",    "messages": [{"role": "user", "content": "Say hello"}],    "max_tokens": 50  }'

You should see a JSON response with the model’s reply.

Step 3: Configure Hermes​

Run the Hermes setup wizard:

hermes setup

When prompted for a provider, selectCustom Endpointand enter:

• Base URL:http://localhost:11434/v1
• API Key:Leave empty or typeno-key(Ollama doesn’t need one)
• Model:gemma4:31b(or whichever model you pulled)

http://localhost:11434/v1 no-key gemma4:31b

Alternatively, edit~/.hermes/config.yamldirectly:

~/.hermes/config.yaml

model:  default: "gemma4:31b"  provider: "custom"  base_url: "http://localhost:11434/v1"

Step 4: Start Using Hermes​

hermes

That’s it. You’re now running a fully local agent. Try it out:

You: List all Python files in this directory and count the lines of code in eachYou: Read the README.md and summarize what this project doesYou: Create a Python script that fetches the weather for Ho Chi Minh City

Hermes will use the terminal tool, file operations, and your local model — no cloud calls.

Step 5: Pick the Right Model for Your Task​

Not every task needs the biggest model. Here’s a practical guide:

gemma4:31b gemma2:9b llama3.2:3b

For full agentic work (editing files, running commands, browsing),gemma4:31bis currently the best local option with tool-call support. CheckOllama’s model libraryfor newer models — tool-calling support is expanding rapidly.

gemma4:31b

Switch models on the fly inside a session:

/model gemma2:9b

Step 6: Optimize for Speed​

Increase Ollama’s Context Window​

By default, Ollama uses a 2048-token context. Hermes requires at least 64,000 tokens for agentic work with tools:

# Create a Modelfile that extends contextcat > /tmp/Modelfile << 'EOF'FROM gemma4:31bPARAMETER num_ctx 64000EOFollama create gemma4-64k -f /tmp/Modelfile

Then update your Hermes config to usegemma4-64kas the model name.

gemma4-64k

Keep the Model Loaded​

By default, Ollama unloads models after 5 minutes of inactivity. For a persistent gateway bot, keep it loaded:

# Set keep-alive to 24 hourscurl http://localhost:11434/api/generate \  -d '{"model": "gemma4:31b", "keep_alive": "24h"}'

Or set it globally in Ollama’s environment:

# /etc/systemd/system/ollama.service.d/override.conf[Service]Environment="OLLAMA_KEEP_ALIVE=24h"

Use GPU Offloading (If Available)​

If you have an NVIDIA GPU, Ollama automatically offloads layers to it. Check with:

ollama ps   # Shows which model is loaded and how many GPU layers

For a 31B model on a 12 GB GPU, you’ll get partial offload (~40 layers on GPU, rest on CPU), which still gives a significant speedup.

Step 7: Run as a Gateway Bot (Optional)​

Once Hermes works locally in the CLI, you can expose it as a Telegram or Discord bot — still running entirely on your hardware.

Telegram​

1. Create a bot via@BotFatherand get the token
2. Add to your~/.hermes/config.yaml:

~/.hermes/config.yaml

model:  default: "gemma4:31b"  provider: "custom"  base_url: "http://localhost:11434/v1"platforms:  telegram:    enabled: true    token: "YOUR_TELEGRAM_BOT_TOKEN"

1. Start the gateway:

hermes gateway

Now message your bot on Telegram — it responds using your local model.

Discord​

1. Create a Discord application atdiscord.com/developers
2. Add to config:

platforms:  discord:    enabled: true    token: "YOUR_DISCORD_BOT_TOKEN"

1. Start:hermes gateway

hermes gateway

Step 8: Set Up Fallbacks (Optional)​

Local models can struggle with complex tasks. Set up a cloud fallback that only activates when the local model fails:

model:  default: "gemma4:31b"  provider: "custom"  base_url: "http://localhost:11434/v1"fallback_providers:  - provider: openrouter    model: anthropic/claude-sonnet-4

This way, 90% of your usage is free (local), and only the hard tasks hit the paid API.

Troubleshooting​

“Connection refused” on startup​

Ollama isn’t running. Start it:

sudo systemctl start ollama# orollama serve

Slow responses​

• Check model size vs RAM:If your model needs more RAM than available, it swaps to disk. Use a smaller model or add RAM.
• Checkollama ps:If no GPU layers are offloaded, responses are CPU-bound. This is normal for CPU-only servers.
• Reduce context:Large conversations slow down inference. Use/compressregularly, or set a lower compression threshold in config.

ollama ps /compress

Model doesn’t follow tool calls​

Smaller models (3B, 7B) sometimes ignore tool-call instructions and produce plain text instead of structured function calls. Solutions:

• Use a bigger model—gemma4:31borgemma2:27bhandle tool calls much better than 3B/7B models.
• Hermes has auto-repair— it detects malformed tool calls and attempts to fix them automatically.
• Set up a fallback— if the local model fails 3 times, Hermes falls back to a cloud provider.

gemma4:31b gemma2:27b

Context window errors​

The default Ollama context (2048 tokens) is too small for agentic work. SeeStep 6to increase it.

Cost Comparison​

Here’s what running locally saves compared to cloud APIs, based on a typical coding session (~100K tokens input, ~20K tokens output):

Your only cost is electricity — roughly $0.01–0.05 per session depending on hardware.

What Works Well Locally​

• File editing and code generation— models 9B+ handle this well
• Terminal commands— Hermes wraps the command, runs it, reads output regardless of model
• Web browsing— the browser tool does the fetching; the model just interprets results
• Cron jobs and scheduled tasks— work identically to cloud setups
• Multi-platform gateway— Telegram, Discord, Slack all work with local models

What’s Better with Cloud Models​

• Very complex multi-step reasoning— 70B+ or cloud models like Claude Opus are noticeably better
• Long context windows— cloud models offer 100K–1M tokens; local runtimes often default below Hermes’ 64K minimum unless you configure them
• Speed on large responses— cloud inference is faster than CPU-only local for long generations

The sweet spot: use local for everyday tasks, set up a cloud fallback for the hard stuff.