Running Hermes on a local model means no API bills, no data leaving your machine, and an agent that works on a plane. Here’s the setup, which model to pick, and the one mistake that makes people think local models are slow.
Short answer
- Install a runtime, download a model, serve it locally, point Hermes at the endpoint.
- Pick small over big — an oversized model in an agent loop is unusable.
- Test tool calls, not chat. Plenty of models chat fine and fail at tools.
- Best setup is hybrid: frontier brain, local model for the grunt work.
What this actually involves
Setting up a local model with Hermes is two separate jobs, and people conflate them.
- Installing Hermes — getting the agent itself running. That’s covered in the Hermes agent local setup guide.
- Wiring a local model into it — choosing a model, serving it on your machine, and pointing Hermes at it instead of a paid API. That’s this page.
The second part is where most people stall, usually because they pick a model that’s too big and conclude that local models are just slow.
Pick the model before anything else
This is the decision that determines whether the whole thing feels good or terrible. Match the model to the job.
| Job | What to run |
|---|---|
| Agent work, tool calls, memory lookups | LFM2.5-2.6B — trained through the Hermes harness |
| Fast local building and quick generation | Maple Preview — ternary weights, very quick |
| General local use on decent hardware | Gemma 4 and similar mid-size models |
| Anything you’d bill a client for | A frontier model — be honest about the gap |
The mistake is reaching for the biggest model your machine can technically load. A 27B model that takes minutes to answer a simple question is worse than useless in an agent loop, because the agent makes many calls per task.
The setup, step by step
- Install a local runtime. LM Studio is the easiest starting point. Hugging Face works too if you prefer pulling weights directly.
- Download your model. Start with something in the 2-3B range if you’re unsure — you can always go bigger once it’s working.
- Serve it locally. Your runtime exposes a local endpoint. That’s what Hermes will talk to.
- Point Hermes at that endpoint instead of a cloud API, and select the model you’ve loaded.
- Test tool use, not chat. Ask it to run a skill or search the web. Plenty of models chat fine and fall over the moment they have to call a tool.
- Wire in your memory vault so it answers from your own context rather than generically.
The whole thing is a few commands. What takes the time is testing, and testing the right thing — tool calls, not conversation.
The test that actually tells you something
Don’t judge a local model by whether it sounds clever. Judge it on whether it can do the job.
- Can it call a tool correctly and use the result?
- Can it learn a skill and then actually run it?
- Can it read your memory vault and answer from what’s in there?
- Does it hold state across a multi-step task?
- Does it slow the rest of your machine to a crawl while it works?
That last one is the killer nobody mentions. A model that technically works but bogs your whole setup down is a model you’ll stop using within a week.
Why bother going local at all
| Cloud API | Local model |
|---|---|
| Every message costs money | Free once it’s downloaded |
| Your data goes to someone else’s server | Nothing leaves your machine |
| Rate limits and token anxiety | Run as much as you like |
| No Wi-Fi means no agent | Works offline, on a plane |
| Provider decides what you can do | You decide |
Privacy is the one people underrate. If your agent is reading client notes or business context, a local model means that context never leaves your hardware.
The hybrid setup I’d actually recommend
You don’t have to choose. The best setup runs both.
Keep a frontier model as the brain for the decisions that matter, and delegate the time-consuming, token-heavy, non-frontier subtasks to a local model. You get quality where it counts and stop paying for the grunt work.
In my agent OS I keep a local section where models swap in and out, so I can run one local model as the builder, another for agent tasks, and reach for a frontier model only when the job needs it. Details in the Agentic OS dashboard and best OS for Hermes Agent.
Want the local engine ready-made? The Agent OS in the AI Profit Boardroom ships with the local model section, swap-in-swap-out model switching, multiple agent profiles and the memory system — plus a full course on running the whole thing free on local models, and weekly coaching calls if you get stuck.
FAQ
How do I set up a local model with Hermes?
Install a local runtime like LM Studio, download a model, serve it locally, then point Hermes at that local endpoint instead of a cloud API. It’s a few commands.
Which local model should I start with?
For agent work, LFM2.5-2.6B — it was post-trained through the Hermes harness. For fast local building, Maple Preview. Start small rather than big.
Why is my local model so slow?
You’ve almost certainly picked one that’s too large for your hardware. A 27B model in an agent loop can take minutes per step. Drop to something in the 2-3B range.
Do I need a powerful machine?
Not for the smaller models. Something in the 2-3B class runs on around 8GB, which covers most modern laptops.
LM Studio or Hugging Face?
LM Studio if you want the easiest path, Hugging Face if you’d rather pull weights directly. Both work fine with Hermes.
Is a local model private?
Yes — nothing is sent to the cloud. That’s the main reason to run one if your agent touches client or business data.
Can I still use paid models too?
Yes, and you should. Keep a frontier model as the brain and delegate the routine token-heavy work to the local one.
How do I know if it’s actually working?
Test tool use, not chat. Ask it to run a skill, search the web, or read your memory vault. Many models chat well and fail the moment they have to call a tool.
The bottom line
A Hermes local model setup is a runtime, a sensibly sized model, and one endpoint change — then testing tool calls rather than conversation. Pick small over big, run a local model for the grunt work with a frontier brain on top, and you get a private, offline, free agent that doesn’t grind your machine to a halt.
About Julian Goldie
I run Goldie Agency, a 7-figure SEO agency, and teach this stuff daily on a 394K+ subscriber YouTube channel. I’ve delivered 240+ client projects on Upwork at a 100% job-success score over 10+ years of ranking sites through every major Google update. The systems I actually run are inside the AI Profit Boardroom, and my link building book is free here.
