Short answer: yes, within limits that are easy to work out for yourself. This is the arithmetic that decides whether a model fits your machine, why the speed is what it is, and how to test it without trusting anybody’s benchmark — including ours.
This question gets answered badly in two directions. One camp says local models are toys. The other posts a screenshot of tokens streaming on a high-end workstation and calls it “runs on a laptop”.
Both skip the part you actually need, which is how to work out the answer for your machine. So here it is, from first principles.
A model’s size is set by how many parameters it has and how many bits each one is stored in. That gives you arithmetic rather than a recommendation:
So the rule of thumb worth memorising is: at 4-bit, a model needs roughly half a gigabyte of memory per billion parameters, before overhead.
Then add the overhead, which people routinely forget:
Work from free memory, not installed memory, and leave real headroom. If the weights plus context do not comfortably fit, the system will start swapping to disk and the experience collapses — not gradually, but off a cliff.
Here is the thing most guides miss. Generating each token requires reading the model’s weights out of memory. Not some of them — effectively all of them, for every token produced.
That makes text generation a memory-bound problem rather than a compute-bound one. The ceiling is set by how fast your machine can move data out of RAM, and adding processor cores does not lift it. It is why two laptops with similar-sounding processors can perform noticeably differently if their memory configurations differ, and why a smaller model is faster in a way that feels disproportionate: half the weights means half the reading, every single token.
It also explains the single most reliable tuning move available to you: if it is too slow, use a smaller model before you use a more aggressive compression setting.
Quantisation is storing the model’s numbers at lower precision. It is the reason local models are practical at all — it shrinks what has to be held and what has to be read, which helps both constraints at once.
It is not free, and two findings from recent research are worth knowing before you pick a setting.
First, it does not fully solve the problem. A study of language model inference on edge devices found that while “quantization helps mitigate memory overhead, it does not fully eliminate resource bottlenecks, especially for larger models”. Compression buys headroom; it does not remove the ceiling.
Second — and this is the counter-intuitive one — the recipes that work for large models do not simply carry over to small ones. A benchmark of quantisation applied specifically to small language models reported “fundamental disparities between SLMs and LLMs in quantization sensitivity”, concluding that “direct transfer of LLM-optimized techniques leads to suboptimal results due to SLMs’ unique architectural characteristics”.
The practical consequence: advice of the form “always use this quantisation level” is worth very little, because where quality falls off depends on the model and on your task. Which brings us to the only reliable method.
Published benchmark numbers are close to useless for this decision. They were measured on different hardware, with a different memory configuration, a different runtime, a different context length and a different prompt. Any of those can move the result substantially.
Do this instead, and it takes an afternoon:
| Works well locally | Still better hosted |
|---|---|
| Summarising a document you already have | Long multi-step reasoning |
| Extracting fields into a structure | Broad world knowledge and recall |
| Classifying and routing | Unusual or highly open-ended requests |
| Reformatting and rewriting to a pattern | Work where the best available quality matters most |
| Anything you run hundreds of times | One-off tasks where cost is irrelevant |
| Anything on material that should not leave the machine | Tasks needing current information from the web |
The pattern is clear enough. Local models are good at bounded work close to your data, and that describes a surprising amount of ordinary business activity. They are weaker where the task is open-ended and the quality ceiling matters, which is exactly where a large hosted model earns its cost.
Most sensible setups are not one or the other. They route the high-volume, well-defined work to whatever is cheapest and closest, and reserve the expensive model for the work that needs it.
The reason this question keeps getting more interesting is that the small end is improving faster than the hardware requirement is growing. Every year, the size of model that does genuinely useful work on an ordinary machine gets smaller.
That trend is the reason we are working on it ourselves. Oppermind already develops and runs its own language models — they power parts of the workspace, and we deploy custom versions of them for corporate partners. The next step is a model you can install on your own PC without a datacentre GPU, and why we are building it covers what it is for and why we intend to publish it openly.
If what you want is the work done rather than the infrastructure, the Oppermind workspace does it today: documents, spreadsheets, slides, PDFs, designs, images, video and code in one subscription, with the AI inside each editor. Free with no card, then Starter A$9.95, Pro A$29.99 and Pro Plus A$59.99 a month, with unlimited documents, spreadsheets, decks and designs on every plan.
Running a model yourself is a genuinely interesting project and a real skill. It is also, for most people most of the time, not the shortest path to the thing they were trying to make.
Whatever you’re here to make, make more of it.
Start with arithmetic rather than a recommendation. At 4-bit precision each parameter occupies roughly half a byte, so a model’s weights need approximately half its parameter count in gigabytes — a 3-billion-parameter model is roughly 1.5 GB of weights. Then add overhead for the context window and the runtime, and leave room for the operating system and whatever else you actually run.
Not for small models. A capable GPU with enough video memory is faster, sometimes dramatically, but it is not a requirement for models in the small size ranges. What matters more on a CPU-only machine is how much memory you have and how fast it can be read.
Memory bandwidth, usually before processing power. Generating each token requires reading the model’s weights out of memory, so the rate at which memory can be read sets a ceiling that extra processor cores cannot lift. This is why two machines with similar processors can differ noticeably if their memory configurations differ.
Storing the model’s numbers at lower precision so there is less to read and hold. It is the main reason local models are practical at all. It is not free: quality degrades as precision falls, the point at which it becomes noticeable varies by model and task, and research finds that compression techniques tuned for large models transfer poorly to small ones.
Good at bounded, repetitive work close to your data: summarising, extracting fields, classifying, reformatting, drafting from a template, and anything you would run many times. Weaker at long multi-step reasoning, broad world knowledge and unusual requests, where larger hosted models remain clearly ahead. Match the task to the tool rather than expecting one to replace the other.
The memory figures in this article are arithmetic from parameter counts and storage precision, not measurements; actual requirements vary with the model, the runtime, the context length and the quantisation scheme, and you should verify on your own hardware before relying on them. No throughput figures are quoted here deliberately, because published benchmarks are highly sensitive to hardware and configuration and rarely transfer to another machine. This article is general technical information, not advice, and does not take account of your circumstances. Third-party research is cited with its date and is the work of its authors. Oppermind platform prices are current as at 15 September 2026, are in Australian dollars, and are subject to the plan terms at checkout.
A full suite of AI tools and editors, with the autonomous worker included in every plan.