Four ways to run AI on your own hardware, and what each costs

In the last post we covered open-weight AI models: models you can download and run on hardware you control, which means no vendor sees what you put into them. The obvious follow-up question is what that actually involves, and what it costs.

There are four realistic answers. They range from nothing upfront to several thousand dollars, and the right one depends on what you're running and how sensitive the data is. Before any of them, though, there's a question worth asking honestly.

Start with the honest question: do you need this?

Most businesses don't.

If you're not handling client-confidential, health, legal or otherwise regulated information, a Business or Team tier account with a mainstream vendor is cheaper, faster to set up, and less work to maintain than anything below. The protections you actually need are a written commitment that your data isn't used for training, a written commitment that it isn't sold, and an account tier where those commitments apply by default rather than after you've found the right setting.

That covers most businesses. We'd rather tell you that than sell you infrastructure.

The rest of this post is for the cases where it doesn't: regulated data, contractual obligations to clients about where their information lives, or a volume of AI use where the per-token bill has started to look like a salary.

Path 1: your own computer

The cheapest option is the machine on your desk.

Models come in sizes. The ones that beat the frontier labs on benchmarks are enormous, but the smaller ones are surprisingly capable and they run on ordinary hardware. The constraint is video memory (VRAM), and the arithmetic is simple: a model at 4-bit quantisation needs roughly half a gigabyte of VRAM per billion parameters, plus a bit of headroom.

In practice, on hardware you might already own:

• 8 GB VRAM (RTX 4060, RTX 5060 Ti): 7-8B models. Llama 3.1 8B runs at around 70 tokens per second.
• 16 GB: models up to about 24B.
• 24 GB (RTX 3090, RTX 4090): 32B-class models.
• 40 GB+: 70B models, which realistically means two 24 GB cards.

Macs work differently. Apple silicon shares one pool of unified memory between the system and the graphics chip rather than having dedicated VRAM, so the numbers don't map across directly: a Mac needs 64 GB or more of unified memory to sit comfortably in 70B territory.

A build for the 7-13B range runs around USD $1,500; around $2,500 gets you into 13-30B territory. If a model doesn't fit in VRAM it will still run by offloading to system RAM, but a 70B model on a 24 GB card drops to roughly 3-8 tokens per second instead of 30+, which is the difference between useful and unusable.

What that tier is genuinely good at: drafting, summarising, classifying incoming enquiries, pulling fields out of invoices and forms. Well-defined jobs with a clear input and output. What it isn't good at: the hard reasoning end, and anything where you'd notice the gap between a 14B model and Claude or GPT.

Tools: Ollama and LM Studio. Both are free and neither requires you to write code to get started.

Cost: whatever the hardware costs, once. No usage bill. No data leaving the building.

Path 2: rented cloud GPU

Rent a GPU by the hour, run the model on it, shut it down when you're done. In theory this is the flexible middle ground. In practice there are two things nobody mentions.

The first is that getting capacity isn't a given. Our own attempt to rent serious GPU capacity from Azure went nowhere: the answer that came back was that meaningful allocation would only be considered on a long-term commitment, which changed the economics entirely. That isn't unique to us. Microsoft's own support forums are full of the same experience, including from businesses on Microsoft's startup programs, and Microsoft's engineers state the mechanism plainly: quota approval is not a capacity guarantee, and allocation depends on real-time hardware availability in that region.

The second is that the hyperscalers are the expensive option. Azure's ND96isr H100 v5 lists at around $98 per hour for the eight-GPU instance, roughly $12.29 per GPU-hour. Independent trackers put the same H100 hardware at a median of about $2.99 per GPU-hour across the wider market, with specialist providers (Lambda, Runpod, Nebius and others) generally sitting well below the hyperscalers. Azure's own three-year reserved rate of about $5.47 per GPU-hour is still above what several specialist providers charge on demand with no commitment at all.

The honest maths: at $2.99 an hour, running one H100 continuously is roughly $2,150 a month. That's the number that catches people out. Rented GPU looks cheap because the hourly rate is small, and it stops looking cheap the moment the machine is on all the time. It works well for bursty or scheduled work. It works badly as a substitute for owning hardware.

If data residency matters to you, check that the provider actually offers an Australian region before committing to anything.

Path 3: an AI workstation on site

Buy the hardware, put it in your office, run everything locally at full speed.

This is path 1 with a bigger budget: same principle, better hardware. As of mid-2026 the RTX 5090 is the only consumer card that holds a quantised 70B model entirely in VRAM with 32 GB, at roughly 40-50 tokens per second, at around USD $2,000 RRP and 575W of power draw. A two-card build lands you comfortably in 70B territory for something in the region of $4,000-6,000 depending on the rest of the specification.

Compare that to path 2 honestly. At $2.99 per GPU-hour with the machine running around the clock, which almost no small business actually does, a $6,000 workstation pays for itself against continuous cloud rental in roughly three months. Against occasional use, it never does.

The upfront cost is the visible part. The parts people forget: power, cooling, someone to maintain drivers and updates, and the fact that hardware you bought in 2026 is hardware you still own in 2029 when the models have moved on.

Worth being clear that this tier is not the same thing as running a frontier-class open-weight model. Those need data-centre hardware, as we covered last week. A workstation runs the useful middle of the range, not the top of it.

Path 4: a hosted inference API

This one breaks the pattern. It isn't hardware you control, it's someone else's, and it's here because for a lot of businesses it's the honest answer.

Someone else runs the open-weight model on their infrastructure. You send requests to it and pay per token. Cheapest to start, nothing to maintain, and no hardware to buy.

The trade is obvious once you say it out loud: you're back to trusting a vendor's policy. Just a different vendor. Which means the same question we asked about ChatGPT, Claude and Gemini applies here too, and the answer varies more than you'd expect.

Some concrete examples, all from the providers' own documentation:

• DigitalOcean states that customer data submitted for inference is not used to train, retrain or fine-tune models on its platform, and is not shared with third parties for training.
• Fireworks AI states it does not use prompts or API inputs to train its models without explicit opt-in, and operates zero data retention for open models without opt-in.
• Together AI stores prompts and responses by default. Zero data retention is available but has to be turned on in the privacy settings.

That last one is the pattern to watch for, and it's the same pattern as ChatGPT and Gemini: the protection exists, but it's off until you go and find it. A comparative review by DigitalOcean's own team found meaningful differences in default behaviour across six inference platforms, including retention windows that only apply unless a specific parameter is set.

Read the policy of whichever provider you're considering, and check what happens by default rather than what's possible.

Which one fits

The pattern: if your data is sensitive and your volume is steady, owning hardware wins. If your volume is bursty, rent. If neither, you probably didn't need any of this and the first row is your answer.

Where to start

Pick the row that matches your actual situation, not the one that sounds most impressive. Most businesses land on row one, and a good number of the rest land on row two with a $2,000 machine doing one specific job very well.

If you're not sure which row you're in, that's a 30-minute conversation rather than a research project. We'll tell you straight if the answer is that you don't need any of this.