Local LLM: run a large language model on your own machine

How a local LLM works

A local LLM setup has three pieces: the model weights, an inference engine that turns those weights into generated text, and an interface or API. Aspen bundles those pieces so choosing and running a model feels like using a regular application.

The model generates text token by token. The speed and maximum model size depend mainly on available memory and compute.

Model size and memory

Model names often include a parameter count such as 3B, 8B, 14B or 32B. Larger models generally need more memory. A practical rule of thumb is: 8GB for small models, 16GB for many 7–8B models, 32GB for many 13–14B models, and 64GB or more for larger models. Exact requirements vary with architecture and quantization.

Quantization stores model values at lower precision, shrinking memory use and often improving speed with a modest quality tradeoff. It is a major reason capable models can run on consumer hardware.

CPU, GPU and Apple Silicon

You can run an LLM without a dedicated GPU, although generation is usually slower. A recent discrete GPU improves throughput and makes larger models practical. Apple Silicon is a strong local-LLM platform because its unified memory can be used efficiently by the GPU rather than splitting system RAM and VRAM into separate pools.

Which local model should you choose?

There is no universal best model. The best choice is the strongest model that fits your hardware comfortably and performs well on your task. Families such as Llama, Qwen, DeepSeek, Mistral and Gemma offer different size and capability tradeoffs.

Aspen detects your hardware and recommends a sensible default. You can switch models later instead of locking your workflow to one provider.

When local LLMs make sense

Local LLMs are especially attractive for private documents, offline work, repeated high-volume use, development against an AI API, and workflows where you want predictable ownership of the model and data path. For the hardest frontier reasoning, a cloud model may still be the better tool.

For a broader introduction, read the local AI guide. If privacy is the primary reason you are going local, see private AI.

Frequently asked questions