Asking the user to choose is the wrong design

Most local-AI setups expose a model picker. That pushes an engineering decision onto someone who cannot make it. The right model depends on how hard the question is, and you only know that after answering it. A picker also fails in the expensive direction silently . A user who leaves it on the small model gets worse answers forever and has no way to notice.

The alternative is to make it the system's decision and let the model that is cheapest to run be the one that raises its hand. A 0.8B model costs little enough to run on every turn that using it as a triage step is close to free, and it is capable enough to recognise when a question is beyond it even when it cannot answer it.

What the ladder actually looks like

Four rungs, climbed automatically. Each one is entered only when the rung below signals it cannot finish the job:

| rung | model | entered when | |---|---|---| | draft | 0.8B, local | every turn, always | | local | 4B, local | draft refuses, hedges, or scores low confidence | | hive | expert on a peer machine | draft self-delegates, or the local tier is insufficient | | fusion | MoE consult across peers | explicitly preferred by the user |

Which tier a machine can host comes off available VRAM, using thresholds of 24, 10, 4 and 0 GB, with speculative decoding enabled around 10 GB to leave headroom for speech. Those four numbers were chosen by judgement, not measurement, and we say so on the open-problems list rather than presenting them as tuned.

The failure this design does not cover

Escalating to a bigger model only helps for problems a bigger model fixes. Some problems are hard because they need decomposition, several attempts, verification and more minutes than a request should hold open. That machinery exists goals that run over time, a task protocol carrying a quality score, offload to a peer's hardware and none of it can be reached from difficulty. Sustained work begins only when the caller sets a flag, which is a person deciding rather than the system concluding.

So the autopilot stops exactly where autonomy would start being worth something. Ask something trivial and the system decides on its own to answer it cheaply. ask something needing an hour across three machines and it returns its best single paragraph, because nothing is watching for “this one deserves more than a turn”.

What the running system reports

Measured on the live central node over a six-hour window on 26 July 2026, read from the running container's own logs rather than a benchmark:

| observation | count | |---|---| | goals completed | 27 | | budget-gate blocks | 717 | | distinct goals blocked | 5 |

The second row is a defect, not a throughput figure. Five goals that could not afford their next step were re-evaluated roughly 143 times each instead of waiting for budget to accrue about one check every 2.5 minutes, indefinitely. It is filed as an open issue with that measurement attached. We publish it because a page that only carries the flattering number is not worth trusting on the others.

Arguing with this

The framing above may be wrong, and that is worth more to us than agreement. The questions we cannot answer are written up with the files that implement today's inadequate versions, including what a system may commit on your behalf, how a 40-minute answer can be the same interaction as a 400-millisecond one, and how it would know it was finished.

The list is at github.com/hertz-ai/HARTOS → OPEN_PROBLEMS.md, and the discussion on this specific problem is open. HART OS is Apache 2.0 and in public alpha. disagreement needs no setup at all.