Hermes llama.cpp: One-Click Local Models in Hermes Agent

Hermes llama.cpp support means you can now run AI models fully on your own computer from inside Hermes Agent, with no account, no API key and no internet connection once the model is downloaded. Hermes installs and maintains llama.cpp for you, picks the right engine for your hardware, and rates every model in its catalog against your GPU and RAM so you know what will actually run. In this guide I cover how the one-click setup works, how Hermes picks a model, the settings worth knowing and when local models make sense. Watch the video below, then try it.

What is llama.cpp, and why does it matter for Hermes?

llama.cpp is the open-source engine that runs AI models on ordinary computers. It’s what a lot of local AI tools use under the hood. Until now, using it with Hermes meant installing it yourself, choosing the right build for your graphics card, downloading a model file and pointing Hermes at the server.

According to Hermes’ Local Models docs, Hermes now does all of that for you. It installs and maintains llama.cpp with the right engine for your machine, rates every catalog model against your GPU and memory, and starts and stops the model server along with Hermes.

The headline benefit is privacy and independence: the model runs on your machine, so there’s no account, no API key and no network needed after the download.

How to set up local models in Hermes

There are two ways in, and both work on Windows, macOS and Linux:

• Hermes Desktop: Settings → Providers → Local Models.
• The terminal: run hermes model and choose Local models.

From there it’s three steps: Install runtime, Download a model, then Use it. Downloads can pause and resume, so a big model on a slow connection isn’t a problem.

Hermes picks the engine for you. Every backend runs the same llama.cpp build (b11370 at the time of writing). On Linux, NVIDIA cards get CUDA, while AMD and Intel cards get Vulkan. macOS uses Metal, and every platform has a CPU fallback. If you want to force a backend, the config has a local_runtime.backend setting that accepts auto, cuda, metal, vulkan, hip or cpu.

How Hermes picks a model for your machine

This is the part I like most. Instead of guessing which model your computer can handle, Hermes rates every model in its catalog against your hardware and colour-codes it:

Hermes recommends the highest-quality model that fits fully in GPU memory at a predicted 20 tokens per second or more. It never goes below 4-bit quantisation, and every recommended model gets at least 64K of context, which grows as needed.

The catalog includes models such as Qwen3.8 27B, Qwen3.8 Flash Next, Qwen3.6 35B-A3B and DeepSeek V4 Flash. You’re not limited to the list: “Find more models” searches any GGUF model on Hugging Face, and “Add model file” loads a .gguf you already have on disk.

Settings worth knowing

• Idle unload. Models unload after 15 minutes of no use, so they don’t hold your GPU memory all day.
• Engine updates. When a new llama.cpp build is ready, Hermes shows an “Update now” prompt.
• Check your resources. In Hermes Desktop, right-click the status bar and choose System resources to see GPU and RAM use.
• Fallback. If your main provider fails, Hermes can fall back to the desktop-managed local model.

If you already run your own llama-server, Hermes detects it too. By default it checks port 8080; to check other ports, add them under local_runtime.detect_ports. A server you started yourself shows up in /model.

local_runtime:
  enabled: false
  backend: auto   # auto|cuda|metal|vulkan|hip|cpu
  detect_ports: [8081]

You can also point Hermes at a llama.cpp server directly with the llamacpp provider (aliases llama.cpp and llama-cpp):

providers:
  llamacpp:
    base_url: http://127.0.0.1:8081/v1
    model: my-model

LM Studio users get their own provider profile as well, and the Ollama model picker no longer flickers.

System requirements

You don’t need a top-end graphics card to try it. The colour ratings tell you honestly what your machine can run, and a smaller green model will usually feel better than a bigger amber one.

When local models make sense (and when they don’t)

Local models are a great fit when:

• You handle private or client data that you don’t want leaving your machine.
• You want Hermes to keep working offline or on a poor connection.
• You run lots of simple, repetitive jobs — sorting, summarising, tagging — where a small fast model is enough.

They’re a weaker fit for the hardest coding and reasoning work, where the biggest hosted models are still ahead. The smart setup is both: a local model for everyday and private work, and a hosted model for the heavy lifting. Hermes’ fallback option makes that easy.

If you want to compare specific local models I’ve tested, see my guides to the Flint local coding model, Spark X2 6.4B and Nemotron 3.5 Bolt in Hermes.

My verdict on Hermes llama.cpp

Running models locally used to be a weekend project. Hermes turns it into a few clicks: it installs the engine, tells you what fits, downloads the model and manages the server. If you’ve wanted a private, offline agent, Hermes llama.cpp support is the easiest way I’ve seen to get one.

About Julian Goldie

I’m Julian Goldie — founder of Goldie Agency and the AI Profit Boardroom. I test new AI agents almost every day for 432K+ YouTube subscribers and run Hermes, OpenClaw, OpenCode, Claude and Codex side by side in my own Agent OS. Claim your free strategy session or join the AI Profit Boardroom.

Want another angle? Read the AI Profit Boardroom guide to Hermes llama.cpp.

FAQ: hermes llama.cpp

Does Hermes Agent support llama.cpp?

Yes. Hermes installs and maintains llama.cpp for you from Settings, Providers, Local Models or hermes model, and you can also point it at your own llama-server with the llamacpp provider.

Do I need an API key to run local models in Hermes?

No. Local models run fully on your machine with no account, no API key and no network once the model is downloaded.

Which GPUs work with Hermes local models?

NVIDIA cards use CUDA, AMD and Intel cards use Vulkan on Linux, Macs use Metal, and every platform can fall back to CPU.

How does Hermes choose which local model to recommend?

It rates each model against your GPU and RAM and recommends the highest-quality one that fits fully in GPU memory at a predicted 20 tokens per second or more, never below 4-bit.

Can I use my own GGUF model in Hermes?

Yes. Use Find more models to search GGUF models on Hugging Face, or Add model file to load a .gguf file you already have.

Related reading

• Flint Local Coding Model
• Spark X2 6.4B
• Nemotron 3.5 Bolt in Hermes

Last updated October 2026. This page is a living guide to hermes llama.cpp — the facts here move fast and it is updated as they do.