What Can You Actually Run Locally?

Running AI models on your own machine is a lot easier than it used to be, and you don't need a huge graphics card to do it. But how far can a normal laptop actually go? Let's install a model, then give it harder and harder jobs until it starts to struggle.

Everything here was run on my own laptop, and you can follow along on yours. Let's dive in.

The machine

I'm using a MacBook Pro with an M4 Pro and 24 GB of memory. It's a pretty standard developer laptop, and I left my usual apps open (PhpStorm, Safari and a few others) while testing, since that's how you'd use it day to day.

If your machine has less memory than this, don't worry. Most of what we'll do runs on models that fit in 16 GB, and we'll cover what you can run on 8, 16, 32 and 64 GB a bit later on.

What actually matters for running models

When you run a model, the whole thing has to be loaded into memory, and the processor doing the work (ideally your GPU) needs fast access to it. So the most important number is how much memory your GPU can use, even more than how powerful it is.

On a Mac with Apple Silicon, the CPU and GPU share the same memory (unified memory), so the GPU can use most of your RAM, but not all of it. On my 24 GB machine, Ollama reports that the GPU can use 17.8 GiB, and macOS keeps the rest for itself.

On a Windows or Linux machine with a graphics card, the number that matters is the card's own memory (VRAM). If a model doesn't fit, part of it gets pushed onto the CPU and system RAM, and as we'll see later, that's where things slow down dramatically.

What makes it fast

Once a model fits, speed mostly comes down to memory bandwidth. Behind the scenes, generating each token means reading through all of the model's parameters, so a 6 GB model generating 30 tokens a second is reading around 180 GB every second.

My M4 Pro has 273 GB/s of memory bandwidth, which is why that's roughly the speed we'll see. It also means a model twice the size will run at roughly half the speed.

There's also a second speed. Before a model writes anything, it has to read your prompt, and on this machine that ran at around 300 to 400 tokens a second. That's nothing for a quick question, but it adds up when you hand it a long document (or, as we'll see, when an agent hands it 20,000 tokens of instructions).

Parameters and quantisation, briefly

Model names usually include a size, like 9b or 27b. That's the number of parameters (the numbers the model learned during training), in billions. More parameters generally means a more capable model, but also a bigger download and more memory.

If you stored each parameter at full 16-bit precision, a 9B model would need about 18 GB. Quantisation stores them at lower precision instead, usually around 4 bits each, which shrinks the model a lot without losing much quality.

Here's what that looks like for qwen3.5:9b, one of the models we'll use:

You'll see tags like q4_K_M and q8_0 on every model page. If you pull a model without picking one, you'll usually get a 4-bit version, and for most people that's the right choice.

As a rough rule of thumb, at 4-bit, a model needs roughly 0.6 to 0.7 GB per billion parameters, plus a GB or two for your conversation and some overhead.

Mixture of experts

Some models have two sizes in their name, like qwen3.6:35b-a3b or gemma4:26b-a4b. These are mixture of experts (MoE) models. All 35 billion parameters have to sit in memory, but only around 3 billion of them are used for each token.

That means they need the memory of a big model but can run closer to the speed of a small one. They're great if you have the memory for them, and we'll see what happens when you don't.

What can I run with my hardware?

I've only tested on 24 GB, so treat these as rough guides based on model sizes and how memory behaved on my machine. Remember to leave room for your operating system and whatever else you've got open.

8 GB

Stick to models around 2B to 4B, like qwen3.5:4b (3.3 GB) or qwen3.5:2b (2.7 GB). Google's gemma4:e2b (4.6 GB) is designed for this kind of device, but it'll be tight.

These won't be much fun for long conversations, but they're fine for embeddings, pulling data out of text and simple questions. Close what you can before you start.

16 GB

This is where local models get properly useful. Models from around 4B to 12B at 4-bit fit comfortably, which covers qwen3.5:9b, gemma4:e4b and gemma4:12b, the three main models I tested with.

A 16 GB machine should be able to follow along with everything in this article apart from the deliberately ambitious section near the end.

32 GB

Now you can run models in the 27B to 35B range at 4-bit, like qwen3.6:27b (17 GB), gemma4:26b (18 GB) or gemma4:31b (20 GB). You can also run the smaller models with much longer context, or at 8-bit.

64 GB and up

This is where you can run the really large open models, keep a couple of models loaded at once, or use very long contexts. Big mixture of experts models like qwen3.5:122b-a10b (81 GB) need 96 GB or more.

Graphics cards

On a PC with a graphics card, go by your VRAM rather than your system RAM. As a rough guide, an 8 GB card will handle models up to around 9B, 12 to 16 GB will get you up to around 14B, and a 24 GB card can run models in the 27B to 32B range.

You can run models on the CPU alone, but it's slow for anything bigger than the small models.

Installing Ollama

There are a few ways to run models locally (LM Studio and llama.cpp are both great), but we'll use Ollama. It's free, it runs on macOS, Windows and Linux, and it gives you a command line and a local API that we can write code against.

You can download it from ollama.com/download, or install it from your terminal:

curl -fsSL https://ollama.com/install.sh | sh

Once it's installed, let's pull our first model and start chatting with it:

ollama pull qwen3.5:9b
ollama run qwen3.5:9b

The download is 6.6 GB and took about nine minutes on my connection. Once it's done, you'll get a prompt where you can start typing, and /bye gets you out again.

What's happening behind the scenes

Ollama runs a server in the background on http://localhost:11434. When you send a message, it loads the model into memory (which took a couple of seconds here), runs it, and then keeps it loaded for five minutes in case you send another one.

You can see what's loaded with ollama ps:

NAME          ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen3.5:9b    56671c2ab938    6.2 GB    100% GPU     4096       4 minutes from now

100% GPU is what you want to see. If that column shows a split like 30%/70% CPU/GPU, the model didn't fit, and it's going to be slow.

Keep an eye on that CONTEXT column, too. We'll come back to it when we get to summarising long documents.

ollama list shows the models you've downloaded, and ollama rm deletes one when you need the disk space back.

Switching off thinking

My very first question was "Why is the sky blue? Answer in two sentences." I got a perfectly good answer, but it took 44 seconds, because the model spent 1,220 tokens thinking before writing two sentences.

Both of the main model families I used have thinking switched on by default. You won't need it for a lot of everyday jobs, so you can switch it off:

ollama run qwen3.5:9b --think=false

With thinking off, the same kind of question takes a few seconds. We'll look at whether thinking is worth the wait as we go.

Chat and general questions

Let's start with what most people try first. I asked qwen3.5:9b five questions with thinking switched off: a beginner explanation, a niche framework question, a question about recent versions, a timetable puzzle and a short email to write.

The explanation of processes and threads was clear, accurate and took five seconds. It also worked out the timetable puzzle (would I make a 17:40 meeting after a delayed train and a 20 minute walk?) correctly, step by step, and the client email was ready to send.

The other two didn't go so well. I asked about the difference between Laravel's defer() helper and a queued job, and got a long, detailed, nicely formatted answer about database transactions and deferrable constraints in PostgreSQL. None of it is true, since defer() runs a closure after the response has been sent to the user.

When I asked for the latest versions of Laravel and PHP, it told me Laravel 11 and described PHP 8.3 as an LTS release (PHP doesn't do LTS releases). Small models only know what was in their training data, and there's much less room in 9 billion parameters for niche facts.

Does thinking help?

I ran the same five questions again with thinking on. Each answer took between 30 seconds and two minutes, compared to five to ten seconds without it.

Anything it had already got right stayed right. The defer() answer changed from made up to "there's no defer() helper in Laravel", which is also wrong.

How it did

Generation ran at about 31 tokens a second with 6.2 GB of memory in use, and for explanations, writing and working through a problem, it felt quick and capable. I'd put this in works, with compromises. Use it for things you can check, and don't trust it on niche or recent facts.

Summarisation and extraction

This is where I expected local models to be useful, since you're giving the model everything it needs and asking it to reorganise it. It also tends to be the kind of data (emails, notes, documents) you'd rather not send anywhere.

Pulling structured data out of an email

Here's a slightly messy customer email, the kind a support inbox gets every day:

Subject: Re: Re: order still not here??

Hi there,

Sorry to email again but I'm getting a bit fed up now. I ordered the standing desk (the oak one, 140cm) on the 3rd of September, order number #48213, and it was meant to arrive within 5 working days. It's now been nearly three weeks. I got a tracking link (DPD, 1550 3392 8812) but it's said "at depot" since the 12th.

On top of that I was charged twice!! £389.00 came out on the 3rd and again on the 4th. My partner also ordered a monitor arm separately (#48240) which arrived fine, so I know it's not my address.

I'd like either the desk delivered by this Friday or a full refund, and the duplicate payment refunded either way. If I don't hear back by Wednesday I'll be raising it with my bank.

You can reach me on 07700 900123 after 5pm, I'm in meetings all day.

Thanks,
Priya Shah

Save that as email.txt. Then let's set up a Python environment with Ollama's Python library, Pydantic and NumPy (we'll need NumPy later). You'll need Python 3.10 or newer:

python3 -m venv .venv
source .venv/bin/activate
pip install ollama pydantic numpy

We'll describe the data we want back as a Pydantic model, then pass its JSON schema to Ollama with format. Create extract.py:

from typing import Literal

from ollama import chat
from pydantic import BaseModel


class SupportTicket(BaseModel):
    customer_name: str
    order_numbers: list[str]
    products: list[str]
    issues: list[str]
    amount_charged_twice: float | None
    tracking_number: str | None
    requested_outcome: str
    deadline: str | None
    sentiment: Literal["happy", "neutral", "frustrated", "angry"]
    urgent: bool


response = chat(
    model="qwen3.5:9b",
    messages=[
        {"role": "system", "content": "Extract the support ticket details from the customer's email."},
        {"role": "user", "content": open("email.txt").read()},
    ],
    format=SupportTicket.model_json_schema(),
    options={"temperature": 0},
    think=False,
)

ticket = SupportTicket.model_validate_json(response.message.content)
print(ticket.model_dump_json(indent=2))

Run it with python extract.py, and here's what came back:

{
  "customer_name": "Priya Shah",
  "order_numbers": ["#48213", "#48240"],
  "products": ["Standing desk (Oak, 140cm)", "Monitor arm"],
  "issues": [
    "Order delayed significantly (ordered Sept 3rd, expected within 5 days, now nearly 3 weeks)",
    "Tracking status stuck at 'at depot' since the 12th",
    "Duplicate charge of £389.00 on consecutive days (3rd and 4th)"
  ],
  "amount_charged_twice": 389.0,
  "tracking_number": "DPD 1550 3392 8812",
  "requested_outcome": "Delivery by this Friday OR full refund (including duplicate charge)",
  "deadline": "Wednesday",
  "sentiment": "frustrated",
  "urgent": true
}

Behind the scenes, Ollama uses the schema to restrict what the model is allowed to generate, so the output matches it. That means model_validate_json won't fall over because of a missing field or a chatty "Sure, here's your JSON!" at the start.

I ran it three times and got the same result each time, in about seven seconds. The only slip was including the partner's monitor arm order, which arrived fine and isn't really part of the complaint.

Summarising a long document

Next, I gave it a 2,500 word article (about 3,500 tokens) and asked for a five point summary and a list of every file the article tells you to create. The summary was accurate, the file list was complete, and it took 21 seconds.

Then I made the document longer by adding a content plan to the start (about 5,100 tokens in total), and asked it to list three ideas from the plan along with the files from the article.

It came back with three "ideas" that weren't in the plan at all, and missed one of the files. There was no error and no warning in my code. The only sign anything had gone wrong was this line in Ollama's server log:

msg="truncating input prompt" limit=2050 prompt=5133

The context window

Remember that CONTEXT column showing 4096? That's the context window, the maximum number of tokens the model can work with at once (your prompt and its reply together). Ollama picks a default based on how much memory your GPU has, and on anything under 24 GiB, you get 4,096 tokens.

When your prompt doesn't fit, Ollama cuts it down to make room for the reply, and the model answers based on whatever's left. In my case, most of the content plan had gone, so it made up some ideas to fill the space.

The fix is to ask for a bigger context window with num_ctx. Here's a small summarising script, summarise.py:

import sys

from ollama import chat

document = open(sys.argv[1]).read()

response = chat(
    model="qwen3.5:9b",
    messages=[{"role": "user", "content": f"{document}\n\n---\n\nSummarise the document above in 5 bullet points."}],
    options={"num_ctx": 16384},
    think=False,
)

print(response.message.content)
print(f"\n({response.prompt_eval_count} tokens in)")

Run it with python summarise.py notes.md. With a 16k context window, the same long document question came back completely right in 21 seconds, and memory use only went up from 6.2 GB to 6.7 GB.

It's worth printing prompt_eval_count, too. If it's much lower than you'd expect for the document you sent, your prompt got cut. If you'd rather change the default for everything, set the OLLAMA_CONTEXT_LENGTH environment variable before starting the server, or use /set parameter num_ctx 16384 inside ollama run.

How it did

Once the context window was sorted, this was the most impressive test so far. Structured extraction gave the same correct result every time, and summaries were accurate. This one goes in works well.

Coding

Let's give it some real code. I set two tasks and ran each one three times per model, with a 16k context window and thinking off. Then I ran actual tests against the code that came back.

The first task was writing something new from a written spec: a parse_duration function that turns strings like "1h30m" and "2h 5m 10s" into seconds, with rules for spacing, ordering, case and invalid input. I checked each attempt against 20 hidden test cases.

The second task was fixing a bug. Here's a small module for merging calendar bookings, bookings.py:

def merge_bookings(bookings):
    """Merge overlapping bookings into combined busy periods.

    Each booking is a (start, end) tuple of datetimes. Returns a list of
    (start, end) tuples with any overlapping or touching bookings merged.
    """
    merged = []
    for start, end in bookings:
        if merged and start < merged[-1][1]:
            merged[-1] = (merged[-1][0], end)
        else:
            merged.append((start, end))
    return merged


def free_slots(bookings, day_start, day_end):
    """Return the free periods between day_start and day_end."""
    slots = []
    cursor = day_start
    for start, end in merge_bookings(bookings):
        if start > cursor:
            slots.append((cursor, start))
        cursor = end
    if cursor < day_end:
        slots.append((cursor, day_end))
    return slots

And the tests, test_bookings.py:

from datetime import datetime

from bookings import free_slots, merge_bookings


def t(h, m=0):
    return datetime(2026, 10, 14, h, m)


def test_merges_overlapping():
    assert merge_bookings([(t(9), t(10)), (t(9, 30), t(11))]) == [(t(9), t(11))]


def test_merges_touching():
    assert merge_bookings([(t(9), t(10)), (t(10), t(11))]) == [(t(9), t(11))]


def test_unsorted_input():
    assert merge_bookings([(t(13), t(14)), (t(9), t(10)), (t(9, 30), t(9, 45))]) == [(t(9), t(10)), (t(13), t(14))]


def test_booking_inside_another():
    assert merge_bookings([(t(9), t(12)), (t(10), t(11))]) == [(t(9), t(12))]


def test_free_slots():
    bookings = [(t(13), t(14)), (t(9), t(10)), (t(9, 30), t(11))]
    assert free_slots(bookings, t(8), t(17)) == [(t(8), t(9)), (t(11), t(13)), (t(14), t(17))]

Four of those five tests fail, because there are three bugs. The bookings aren't sorted first, a booking inside another one shrinks the merged period (it should use max()), and touching bookings don't merge because of < instead of <=.

If you're following along, paste both files into ollama run (with pip install pytest to run the tests yourself) and ask it to find and fix the bugs. Here's how the three models did:

gemma4:12b was the clear winner. Its one miss on the spec was forgetting that "1H" should be allowed, which is the kind of thing a quick test run catches.

The other two wrote code that looked right and mostly worked, but failed on basic cases like "1h30m" or "90m". And qwen3.5:9b found the sorting and max() bugs every time, but missed <= all three times, even though the failing test for touching bookings was right there in the prompt.

I also tried qwen3.5:9b with thinking switched on. It was slower and didn't pass any more tests.

Before I'd set a bigger context window, a couple of attempts also just stopped halfway through the code. The model had used up all 4,096 tokens rambling, so the answer was cut off.

How it did

Most attempts took about a minute. With the right model, small, well defined jobs with tests to check the result went well, but every model got something subtly wrong at some point. I'd call this works, with compromises, and only for code you're going to test.

Vision

A lot of newer local models can read images too. Both qwen3.5:9b and the Gemma 4 models are multimodal, so there's nothing extra to download.

I tried three images. First, a screenshot of a café receipt, extracted into structured data with the same Pydantic approach as before. Create receipt.py:

from ollama import chat
from pydantic import BaseModel


class Item(BaseModel):
    quantity: int
    name: str
    price: float


class Receipt(BaseModel):
    merchant: str
    date: str
    items: list[Item]
    subtotal: float
    service_charge: float
    total: float


response = chat(
    model="qwen3.5:9b",
    messages=[{"role": "user", "content": "Extract the details from this receipt.", "images": ["receipt.png"]}],
    format=Receipt.model_json_schema(),
    options={"temperature": 0, "num_ctx": 8192},
    think=False,
)

print(Receipt.model_validate_json(response.message.content).model_dump_json(indent=2))

The only change from text is the images key on the message, which takes a list of file paths. Images use up a fair chunk of the context window, so I've bumped num_ctx up a bit.

The other two images were a bar chart of monthly signups with a churn line on a second axis, and a photo of the inside of Kirkgate Market in Leeds (photo by Hullian111, CC BY-SA 4.0). For the photo, I asked for a short description, a guess at the location and any shop signs it could read.

What came back

qwen3.5:9b read the receipt perfectly (every item, price and total) in under nine seconds. It also read every value off the chart correctly and spotted September's 2.2% as the lowest churn.

Its description of the photo was spot on too, from the Victorian iron and glass roof to the flower stalls and the person in the blue jacket. It just decided this was Covent Garden in London.

The small gemma4:e4b did surprisingly well. It got the receipt and chart right, and described the photo well, but misread or made up a couple of the shop names.

gemma4:12b, the best of the bunch at coding, was the worst here. It got the date wrong on the receipt, dropped an item and shuffled the prices around. On the photo, it read French signs that aren't there and placed it in Paris, and on one run (with temperature set to 0) it got stuck repeating the same line about a "Flowers" sign for nearly five minutes.

How it did

For documents, screenshots and charts, the right model is fast and accurate, and reading receipts or screenshots locally is something I'd happily rely on. Photos come with more guesswork. With qwen3.5:9b, this is works well. Just don't assume the model that's best at coding will be best at images too.

Embeddings and search

Local AI is useful for more than chat. Embedding models turn text into a list of numbers (a vector) that represents its meaning, so text about similar things ends up with similar numbers, even when it uses completely different words.

That's what powers semantic search, so let's build one. Pull Google's EmbeddingGemma, which is a tiny 621 MB download:

ollama pull embeddinggemma

Put some Markdown files in a notes folder. I used 157 articles I've written over the years, but your own notes or project docs work just as well.

Create index.py, which splits each file into paragraphs and embeds them in batches:

import json
from pathlib import Path

import ollama

chunks = []
for path in Path("notes").glob("*.md"):
    for paragraph in path.read_text().split("\n\n"):
        if len(paragraph) > 80:
            chunks.append({"file": path.name, "text": paragraph.strip()})

for i in range(0, len(chunks), 64):
    batch = chunks[i:i + 64]
    response = ollama.embed(
        model="embeddinggemma",
        input=[f"title: {c['file']} | text: {c['text']}" for c in batch],
    )
    for chunk, embedding in zip(batch, response.embeddings):
        chunk["embedding"] = embedding

Path("index.json").write_text(json.dumps(chunks))
print(f"Embedded {len(chunks)} chunks")

The title: ... | text: ... format is what EmbeddingGemma's documentation recommends for documents you're searching through. Next, search.py embeds your query and compares it against every chunk:

import json
import sys

import numpy as np
import ollama

chunks = json.load(open("index.json"))
vectors = np.array([c["embedding"] for c in chunks])

query = ollama.embed(model="embeddinggemma", input=f"task: search result | query: {sys.argv[1]}")
q = np.array(query.embeddings[0])

scores = vectors @ q / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q))
for i in scores.argsort()[::-1][:3]:
    print(f"{scores[i]:.2f}  {chunks[i]['file']}")
    print(f"      {chunks[i]['text'][:100]}...")

That scores line is cosine similarity, which measures how closely two vectors point in the same direction. Then we sort the scores and print the top three.

Let's run them:

python index.py
python search.py "how do I make a slow page respond faster without a queue"

Indexing all 157 articles (2,244 chunks) took between 20 and 35 seconds, and each search took about a third of a second. Here's what came back:

0.52  deferred-props-in-inertia.md
      One way to speed up your applications is to defer data loading until the page has been rendered. Whe...
0.47  dive-into-laravels-new-defer-helper.md
      In this example, when hitting the / route, users will instantly see the This is a response response....
0.42  prefetching-in-inertia.md
      Triggering prefetching on mount is a good way to prefetch data that you know a user will visit. This...

The top result doesn't use the words "slow" or "queue" at all, but it's about the right thing, and the second result is the defer() article I'd have pointed someone to. Searching "which laptop and microphone do you record with" went straight to my post about the gear I use, too.

Searching for "stop people spamming my signup form with bots" did miss my article on Cloudflare Turnstile, though, because that article never mentions bots or spam.

Without the recommended prefixes, results were still decent, but the defer() article dropped out of the top three. It's worth checking the docs for whichever embedding model you use.

How it did

The model is tiny and runs comfortably alongside everything else, and since search runs over your own files, keeping it local makes a lot of sense. This is works well, and it's probably the easiest win in this whole article.

Agents and tool use

Can a local model drive tools by itself, deciding what to call, reading the results and carrying on until the job's done?

Ollama supports tool calling, and the Python library can turn plain Python functions into tool definitions using their type hints and docstrings. Let's give a model three tools for working with files in a project, and ask it to find every TODO comment and write them to a checklist.

I made a tiny Flask project with five TODO comments spread across different folders (plus a variable called todo_list to try and trip it up). Any small project of yours with a few TODOs will do. Create agent.py next to a project folder:

from pathlib import Path

from ollama import chat

ROOT = Path("project").resolve()


def safe_path(path: str) -> Path:
    full = (ROOT / path).resolve()
    if not full.is_relative_to(ROOT):
        raise ValueError("Path is outside the project")
    return full


def list_files(directory: str = ".") -> str:
    """
    List the files and folders inside a directory of the project. Folders end with a slash.

    Args:
      directory: The directory to list, relative to the project root
    """
    entries = sorted(safe_path(directory).iterdir())
    return "\n".join(e.name + ("/" if e.is_dir() else "") for e in entries)


def read_file(path: str) -> str:
    """
    Read the contents of a file in the project.

    Args:
      path: The path to the file, relative to the project root
    """
    return safe_path(path).read_text()


def write_file(path: str, content: str) -> str:
    """
    Create or overwrite a file in the project.

    Args:
      path: The path to the file, relative to the project root
      content: The full contents to write to the file
    """
    safe_path(path).write_text(content)
    return f"Wrote {len(content)} characters to {path}"


tools = {"list_files": list_files, "read_file": read_file, "write_file": write_file}

messages = [{
    "role": "user",
    "content": "Find every TODO comment in this project (check every folder) and write them "
               "to TODO.md as a Markdown checklist, grouped by file.",
}]

for step in range(25):
    response = chat(model="qwen3.5:9b", messages=messages, tools=list(tools.values()))
    messages.append(response.message)

    if not response.message.tool_calls:
        print(response.message.content)
        break

    for call in response.message.tool_calls:
        name, args = call.function.name, call.function.arguments
        print(f"→ {name}({args})")
        try:
            result = tools[name](**args)
        except Exception as e:
            result = f"Error: {e}"
        messages.append({"role": "tool", "content": str(result), "tool_name": name})

Let's run over what's happening here:

1. We send the conversation and the list of tools to the model.
2. If the model wants to use a tool, it replies with one or more tool calls instead of text.
3. We run each tool ourselves, and add the result to the conversation as a tool message.
4. We go round again, until the model replies without any tool calls (or we hit 25 steps).

The model never touches your files directly. It can only ask for one of our functions to be run, and safe_path makes sure those functions can't reach outside the project folder.

Run it with python agent.py, and you'll see each step as it happens:

→ list_files({'directory': '.'})
→ list_files({'directory': 'app/'})
→ list_files({'directory': 'templates/'})
→ list_files({'directory': 'tests/'})
→ list_files({'directory': 'app/services/'})
→ read_file({'path': 'app/__init__.py'})
→ read_file({'path': 'app/models.py'})
...
→ write_file({'path': 'TODO.md', 'content': '# TODO List\n\n## app/__init__.py\n\n- [ ] load config from...'})

It explored every folder, read every file, ignored the todo_list variable and wrote all five TODOs to the checklist. I ran this five times each on qwen3.5:9b, gemma4:e4b and gemma4:12b, and all 15 runs got it right, each taking between 30 seconds and a minute and a half, with around 13 tool calls.

A harder agent task

Collecting TODOs is fairly simple, so I added a run_tests tool and asked for something closer to real work. The model had to resolve the TODOs in two files, write tests for its changes and keep going until the tests passed. I then checked the results against my own hidden tests.

At the default 4k context, qwen3.5:9b succeeded once in three runs. In the other two, the conversation filled up the context window after a dozen or so steps, and the model stopped with an empty reply, leaving failing tests behind.

With options={"num_ctx": 32768} added to the chat call, it succeeded five out of five times. But each run took between two and five and a half minutes, and needed anywhere from 17 to 38 steps.

Looking through the logs, the longest run ran the tests 12 times, rewrote its test file six times, and edited a file it wasn't asked to touch to work around an import problem. It got there, but you'd want to read the changes before keeping them.

How it did

For simple tool use with a handful of steps, small local models were completely reliable, so that part's works well. For longer, multi-step jobs that change code, it's works, with compromises. Give it plenty of context, and check what it changed afterwards.

Something too ambitious

Let's try two things this laptop really isn't built for.

A real coding agent

Ollama can stand in for a cloud API, which means you can point coding agents like Claude Code, Codex or OpenCode at a local model. Ollama's docs recommend at least a 64k context window for this, so let's make a version of our model with that built in, using a Modelfile:

FROM qwen3.5:9b
PARAMETER num_ctx 65536

ollama create qwen3.5-64k -f Modelfile
ollama launch claude --model qwen3.5-64k

ollama launch sets up the connection for you. I gave Claude Code the same "resolve the TODOs and make the tests pass" task as above.

Before the model could do anything, it had to read Claude Code's instructions and tool definitions, which came to 21,670 tokens. At around 350 tokens a second, that's roughly a minute before the first word of the first reply.

It did finish the job, and the result passed my hidden tests. That took 37 turns and nine and a half minutes, the context got up to 36,665 tokens, and it added some extra handling to a function that nobody asked for.

So it works, which is impressive for a 9B model on a laptop. But it's too slow to work with interactively, and this was a tiny project with two files to change.

A model that doesn't fit

Next, I pulled qwen3.6:27b, a 17 GB download. Remember, the GPU on this machine can use 17.8 GiB, and the model needs some room for context on top of its own size.

Ollama loaded it with 12% on the CPU and 88% on the GPU, and every request failed with an out of memory error from the GPU. My Python script got back empty responses, so if I hadn't been watching the logs, I wouldn't have known why.

To get it running at all, I had to tell Ollama to put fewer layers on the GPU with the num_gpu option. It worked, but here's what happened to the speed:

At 0.8 tokens a second, a short paragraph about processes and threads took over four minutes.

The 22 GB qwen3.6:35b, a mixture of experts model, loaded with 30% on the CPU and ran at 2.1 tokens a second. Meanwhile, macOS's swap usage climbed to nearly 20 GB as it moved everything else out of the way.

The answers themselves were good, for what it's worth. On a machine with enough memory to hold them, these bigger models would be well worth a look.

How it did

Running a coding agent on a small local model works in a technical sense, but it isn't practical yet. And once a model doesn't fit in GPU memory, it drops to a couple of tokens a second. On this laptop, both of these are stick to cloud.

The scorecard

Here's how everything came out on a 24 GB laptop:

What you get from running locally

Now that we've seen what it can do, let's look at why you'd bother.

Privacy

Your data never leaves your machine. For things like customer emails, receipts, private notes or a client's codebase, that can decide whether you're allowed to use AI on it at all.

No per token cost

Once you've got the hardware, every request is free. You can run extraction over thousands of emails or re-embed all your notes every night without watching a bill.

Offline use

It all works on a train or when your internet goes down. Once the models are downloaded, nothing needs a connection.

Control

The model only changes when you decide to change it. A prompt that works today will behave the same next month, which is handy for anything you've built a process around.

What it costs you

Hardware

You need enough memory to hold the model, and as we saw, it gets very slow when you don't have it. If you're buying a machine with local AI in mind, more memory will help more than anything else.

Speed

30 tokens a second is perfectly usable for chat, but it's slower than most cloud models, and long prompts take a while to read in. Agents feel it the most, since they send the whole conversation back on every step.

Downloads and disk space

The models I used for this article added up to about 61 GB and took around an hour and a half to download. Check ollama list now and then, and use ollama rm for anything you're not using.

Capability

Small models know less and make more things up, especially about niche or recent topics. The best cloud models are still well ahead for complex coding and long, multi-step work.

Maintenance

You're responsible for the settings that a cloud provider would handle for you. Defaults like a 4k context window or thinking being switched on had a bigger effect on my results than the choice of model, and new models come out constantly, so there's always something to update or re-test.

So, should you run AI locally?

Local AI is already good enough for more than you might think, but the sweet spot depends a lot on the job and the machine.

On a normal laptop today, I'd happily use a local model for pulling structured data out of documents, reading receipts and screenshots, summarising private notes, semantic search over my own files, and small scripts that use tools. They're fast enough, and nothing leaves my machine.

I'd stick with a cloud model for coding agents, questions where I need the facts to be right and up to date, and anything long and multi-step.

If you've got 16 GB or more, pull qwen3.5:9b and embeddinggemma, set a sensible context window and try them on something from your own work. You'll find out pretty quickly which of your jobs a local model can handle.