Small models, big consequences

The headlines about artificial intelligence usually concern giants: models with hundreds of billions of parameters trained on warehouses of hardware. Quietly, a different story has been unfolding at the other end of the scale.

Small models, with a few billion parameters or fewer, have become surprisingly capable. They run on a laptop, sometimes on a phone, with no network connection at all.

How small became good

Three techniques did most of the work. Better training data, curated rather than merely vast, lets a small model learn more per example. Distillation trains a compact student to imitate a larger teacher. Quantisation stores the weights with fewer bits, shrinking memory use dramatically with a modest loss in quality.

The result is a model that is not the best at everything, but is excellent at a narrow job.

Why that matters

Running a model locally changes the economics and the politics of the technology.

• Responses arrive instantly, with no network latency.
• Private text never leaves the device.
• There is no per-request fee, only the electricity you already pay for.
• The tool keeps working offline or after a service shuts down.

The trade-offs

Small models are more brittle. They make more factual errors, handle long documents less gracefully and are easier to confuse. For high-stakes questions, a larger model or a human check is still wise.

Hardware is also uneven. A recent laptop runs a compact model comfortably, while a five-year-old phone may struggle and drain its battery quickly.

A hybrid future

The likely outcome is a blend. A small local model handles quick, private tasks such as summarising a note or fixing grammar, and calls a larger remote model only when it must. The user sees one assistant; underneath, a router quietly weighs cost, speed and privacy.

What to try

If you are curious, download an open compact model and run it on your own machine. Ask it about something you know well. You will quickly learn where it shines, where it stumbles and why so many engineers are excited.

The next wave of AI may be less about astonishing demos and more about dependable, modest tools.