By Hevolve AI · Updated 2026-07-23
It is computed, not guessed. Four GPU tiers decide the language model: 24GB+ runs 70B-class, 10GB+ turns on speculative decoding (a draft model replies in ~300ms while the main one verifies), 4GB+ runs a heavy model without it, below that is CPU. Speech engines tier separately and far lower — usable English speech and transcription fit inside 1GB. And because a single model can serve several purposes at once (Qwen3.5-0.8B does drafting, captioning and GUI grounding), the question is not how many models fit, but which jobs are still unclaimed after the ones that fit have taken everything they can do.
Free VRAM, not total. Your desktop compositor and browser are already holding some.
UNDER 1 GB — usable, not a consolation prize whisper-base (STT) 0.5 GB floor, 0.2 GB model Kokoro (TTS, English) 0.5 GB floor, 0.2 GB model NeuTTS Air (TTS) 0.7 GB floor, Q4 GGUF ~600 MB MMS-TTS (~50 languages) 1.0 GB floor
1–3 GB — the range most laptops actually sit in MeloTTS (en/es/fr/zh/ja/ko) 1.5 GB Indic Parler (21 Indic + en) 2.0 GB whisper-medium (STT) 2.0 GB F5 (voice cloning, en+zh) 2.5 GB XTTS-v2 (17 langs, cloning) 2.5 GB
3–6 GB OmniVoice (646 languages) 3.5 GB CosyVoice3 (zh/ja/ko/de/es/fr/it/ru) 4.0 GB whisper-large-v3-turbo 4.0 GB Chatterbox Turbo (expressive English) 5.6 GB
6 GB AND UP MiniCPM (vision) 6.0 GB LTX2 / ACEStep / DiffRhythm 6.0 GB WAN2GP 8.0 GB Chatterbox multilingual (23 langs) 14.0 GB — wants a 16 GB card
These are the actual budgets the runtime allocates against, from VRAM_BUDGETS in vram_manager.py. Each entry is a pair: the floor below which the engine is skipped, and the size it is expected to occupy once loaded.
Speech engines tier per-engine. The language model tiers differently — on four VRAM thresholds, and what changes at each is not simply 'a bigger model'.
ULTRA 24 GB+ 70B-class viable, speculative decoding, full voice headroom FULL 10 GB+ draft + main speculative decoding active STANDARD 4 GB+ heavy model only, no speculative decoding CPU under 4 no CUDA GPU, or under 4 GB — chat runs on CPU
The interesting boundary is 10 GB, and it is not where a bigger model fits. It is where **speculative decoding** turns on: a small draft model answers in roughly 300ms while the main model verifies behind it, which makes a local machine feel like a hosted API. Measured at about 40% faster replies than Standard.
And the reason the threshold is 10 and not 6 is a trade nobody would guess from the outside: running draft and main together fits well below 10 GB, but then there is no room left for a voice engine. The threshold is set where speculative decoding can run *and* the machine can still speak. A number chosen for the whole system rather than for the model.
The usual mental model is one model per task: an LLM, plus a vision model, plus a captioner. That is what makes the hardware question feel unanswerable, because you are summing four models against one budget.
Nunba assigns *purposes*, not model types. A single model can serve any combination of them, and capability decides — not a label:
draft · main · vision · caption · grounding · tts · stt · diarization vad · embedding · rerank · ocr · music · image-gen · video-gen · translate
So Qwen3.5-0.8B covers *draft*, *caption* and *grounding* at once — speculative-decode drafting, image captioning, and locating a click target on screen for GUI automation. Qwen3-4B-Omni covers *main*, *tts* and *stt*: chat, speech out and speech in from one load.
That changes the arithmetic. Two models can cover six jobs, so 'can I run agentic workloads on 8 GB' is not answered by adding up six model sizes. It is answered by which purposes are still unassigned once the models that fit have claimed everything they can do.
The comment in the source states the principle plainly: purposes are not gated by model_type — model capabilities drive it, not an artificial type label.
The point of all of this is that the answer is computed, not guessed. The machine is probed on launch, engines whose floor exceeds free VRAM are skipped, and purposes are assigned from what actually loaded. The same hardware produces the same configuration every time.
That property is fragile in one specific way, and it is worth saying because it already went wrong here. The 24 / 10 / 4 thresholds were once duplicated in two places — the backend tier logic and the frontend badge. The failure mode is not a crash: one side ships, the other does not, and the interface now reports a tier the runtime does not agree with. A user is told Full and gets Standard behaviour, with nothing in any log to say so.
They are now one table, exported over an API the frontend reads, precisely so the label cannot drift from the behaviour. Determinism is not a property of having thresholds; it is a property of having exactly one copy of them.
This is the one that wastes an afternoon.
A CUDA-12 build will not load on an older driver. The floors are 527.41 on Windows and 525.60 on Linux. Below those, the GPU is real, `nvidia-smi` reports it happily, and the runtime is still correct to treat the machine as CPU-only — because the build it would have to load cannot run there.
So 'my 3060 is not being used' is usually a driver problem, not a VRAM problem, and no amount of changing model settings will fix it. Check `nvidia-smi` for driver_version first.
One design note worth stealing: when the driver version cannot be parsed at all, the check returns True rather than False. Fail-safe, deliberately — never demote a GPU you failed to measure. An unreadable version string is ignorance, not evidence of an old driver, and treating the two the same silently downgrades working hardware.
Budgeting against total VRAM instead of free. The compositor, the browser and any other CUDA process are already resident. A 6 GB card with a browser open is not a 6 GB budget.
Assuming bigger is better for speech. Kokoro at 82M parameters is 0.2 GB and sounds good in English. Chatterbox multilingual is 12 GB. If you only need English, the 60x smaller model is the right answer, and it runs on CPU.
Treating STT and TTS as one budget. They load separately and at different moments. whisper-base plus Kokoro is a complete listen-and-speak loop inside 1 GB.
Silent GPU-detection failures. If the detection path swallows its error, tiering falls back and never tells you. We shipped exactly that: `detect_gpu: swallowed FileNotFoundError` in a log nobody reads, on a machine where nvidia-smi was not on PATH. The fallback was correct; the silence was not.
They come from the table a running system allocates against, not from a benchmark or a spec sheet. Nunba probes the machine on launch and skips engines whose floor exceeds free VRAM, so these thresholds decide what real installs load every day.
Two honest limits. Several entries are marked as stub budgets that auto-tighten on first successful load — OmniVoice and NeuTTS Air among them — so their published floor is conservative until the runtime has measured the engine once. And a floor is not a quality claim: it says the engine will load and run, not that it will sound better than a larger one.
The numbers are also specific to CUDA builds on NVIDIA. Apple Silicon and CPU-only paths have different characteristics and are not what this table describes.
A published budget is an estimate until something measures it. This one measures.
Each engine loads in a worker subprocess, and after a successful load the worker self-reports its actual GPU usage back to the parent, which stores the value and uses it in preference to the declared estimate on every subsequent load. The measurement is persisted, so it survives a restart.
That is what lets a new engine ship with a deliberately conservative stub budget and tighten itself after one real load, with no code change. OmniVoice and NeuTTS Air are both in that state right now — their published floors are placeholders that reality will lower.
The rails around it are the part worth copying:
* Non-positive readings are ignored. A worker emits 0.0 when it cannot measure — CPU-only, Metal, broken nvidia-smi — and a 0 that means 'I don't know' must never be recorded as 'this model is free'. * Values are clamped to 0.1–64 GB, which catches negative deltas from concurrent workers and runaway leaks. * If measured exceeds declared by more than 50%, it logs a prominent warning rather than quietly accepting it: the declared budget is wrong and will not fit on the GPU class it claims to target.
Allocation is also lock-guarded, for a failure that is easy to miss: two tools checking free VRAM at the same moment both see 5 GB, both conclude 4 GB fits, both allocate, and the device is asked for 8 GB it does not have. A check-then-act race with CUDA OOM as the outcome.
So the honest description of the table above is not 'these are the requirements'. It is: these are the current best estimates, several of them already replaced by measurements from real loads, and each one is checked against reality the next time it runs.
3 papers, each explained in plain language.