I benchmarked two well-reviewed 9B models as coding agents. Both failed the easiest tier — and native tool-calling made them worse.

I've been building QuantaMind, an open-source tool that tests whether a local model can actually drive a coding agent on your machine — not "is it smart in a chat window," but "does it survive a multi-step tool-calling loop." It's free, fully offline, and breaks failures down by type instead of hiding them behind one score. Repo's at the bottom. But I'd rather show you a result than pitch it.

I ran Qwen 3.5 9B and Ornith 1.0 9B — both well-regarded, both at Q8, same 16GB Mac, same llama.cpp backend, same agent tasks, each repeated and counted as passed only if it succeeded every run (pass^k).

Three things I didn't expect:

1. Neither cleared even the Easy tier. Both came back NOT READY. These answer fine in a single back-and-forth — but "answers one question well" and "survives a tool-calling loop" turn out to be nearly unrelated skills, and the loop is where both fell apart.

2. Native tool-calling scored worse than plain prompting. On easy tasks, describing the tools in the prompt and parsing the text beat the native function-calling API (25/25 vs 15/25) — for both models. 100% of those native failures were the model doing the work correctly but answering in prose instead of emitting a tool call. Right answer, wrong channel. Everyone's default advice is "use the native API." Here it was backwards.

3. The cold-start tax nobody reports. Both models are ~8.9GB at Q8; a run pushes ~15.6GB on a 16GB machine — right at the ceiling. So the model gets evicted after each call and reloaded from disk before the next one: ~20s time-to-first-token, every single call. In chat you don't notice. In a 20-step agent loop that's ~7 minutes of pure reload before any real reasoning. The fix is counterintuitive — "largest quant that fits" is wrong when fitting exactly means reloading every step. Leave headroom.

Full write-up with the failure-mode breakdown (hallucinated-done vs infinite-loop vs prose): [Medium link]

Honest caveats: this is 9B at Q8 on a 16GB Mac, one backend. Different hardware/quant/harness may land differently — which is the whole reason I built a tool to test your own combo instead of trusting a recommendation thread.

Repo: github.com/QuantaMinds/QuantaMind

Question for the room: have you seen native tool-calling underperform plain prompting on a small model? Or does that gap close at 27B+? I'm collecting data points and would like to know where the line is.