What it actually costs to run AI inside your company: three builds with prices and real throughput

Search for "on-premise AI" and you will find dozens of pages promising your own private, secure, compliant artificial intelligence, running on your own servers.

Now try to find, on those pages, the name of a graphics card. A price. A tokens-per-second figure. How many gigabytes of memory you need.

You will not. The topic is crowded with claims and empty of numbers.

This article tries to fill that gap. It is long, it has tables, and at the end it also says when you should not do it — the part a vendor normally leaves out.

The rule that decides everything: video memory

Before prices, there is one thing to understand, because every later decision follows from it.

To run a language model you have to hold it entirely in memory. Not on disk: in memory, either the graphics card's (VRAM) or, on a Mac, unified memory. If the model does not fit, it does not run — or it runs off disk at unusable speed.

How much memory you need depends on the parameter count and how much the model has been compressed. That compression is called quantisation: at "Q4", the compromise most used in business settings, you need roughly half a gigabyte per billion parameters, plus headroom for context.

There is a trap in that table, and it catches technical people too. Recent models are often MoE, mixture of experts: they have many total parameters but activate only a fraction per response. OpenAI's gpt-oss-120b, for instance, has 117 billion total parameters and activates 5.1 billion.

It sounds like the answer to everything. It is not: memory is driven by the total, not the active count, because every expert still has to be loaded. The advantage of MoE is speed, not footprint. Anyone telling you "it is only 5 billion active, it runs anywhere" has misunderstood — or is hoping you will.

Three real builds

The prices below come from an April 2026 review with published benchmarks, based on the US market. We converted them to euro at the 12 August 2026 rate (1.1525 dollars to the euro) and rounded, and they are excluding VAT. In Europe, between distribution margins and actual availability, list prices tend to run higher: treat them as orders of magnitude that tell you which league you are playing in, not as a quote.

Build A — the departmental machine (~€4,300-7,000 complete)

An RTX 5090: 32 GB of memory, 1,792 GB/s of bandwidth. The card alone runs €3,000-4,200; a complete system €4,300-7,000.

What it handles: models up to roughly 30 billion parameters, at 60-90 tokens per second. That is a comfortable speed, comparable to an online service.

What companies actually do with it: classifying incoming documents, extracting data from invoices and delivery notes, semantic search across an internal archive, first drafts of standard replies, summarising tender documents. Repetitive, vertical, high-volume tasks.

This is the build that solves the real problem for an SME nine times out of ten. And almost nobody proposes it, because it is cheap.

Build B — the Mac that looks like a bargain (€3,000-5,200)

A Mac Studio M4 Max with 128 GB of unified memory costs around €3,000 and will hold a 70-billion-parameter model in memory. On paper it is unbeatable: half the price of a professional GPU, four times the memory of a 5090.

The catch is memory bandwidth: 546 GB/s against the 5090's 1,792. And bandwidth is what governs generation speed. Measured result: 8-15 tokens per second on a 70-billion model.

In plain terms, that is about ten words a second: readable, but ask for a two-page summary and you will wait the better part of a minute watching a cursor. For interactive use — a person asking questions and waiting — it is frustrating. For batch work — three hundred documents to process overnight — it is entirely adequate.

The M3 Ultra with 256 GB (around €5,200) reaches 819 GB/s and holds enormous models. It remains the cheapest machine in the world for running something with 400 billion parameters, with the same caveat about speed.

Build C — the professional workstation (~€19,000)

An RTX PRO 6000 Blackwell: 96 GB at 1.8 TB/s. It runs 70-billion models on the CUDA stack, so with the whole mature software ecosystem, and it handles fine-tuning.

But here is the news that justifies the reading time on its own: this card went from roughly €7,400 at launch to roughly €11,500 on NVIDIA's marketplace in August 2026. Up 55% in a matter of months.

And it is not an isolated case.

The quote in your drawer is no longer valid

If you are planning an infrastructure investment on figures gathered in 2025, stop.

Between September and December 2025 enterprise DRAM rose by an average of 60%. Server components bought by manufacturers rose 25%. General purpose servers were expected up 10-15% by March 2026. And according to IDC, prices are not expected to stabilise before the end of 2027.

The cause is upstream and no amount of negotiating will fix it: memory fabrication capacity has been redirected to HBM for the AI accelerators going into hyperscale data centres. Anyone buying memory for a company server is queuing behind buyers taking tens of thousands of units.

One side effect says it all: Apple withdrew the 512 GB option for the Mac Studio for the same reason.

There are three practical consequences for decision-makers. Hardware quotes should carry an explicit validity date and be treated as stale after a few weeks. If you already have a migration planned — and support for SQL Server 2016 and Windows Server 2016 ends across 2026-2027 — pulling it forward has measurable financial value. And third: over-specifying memory "to be safe" costs far more than it used to, so size for the real workload rather than the theoretical maximum.

The cost line nobody puts in the business case: electricity

It never appears in the quotes we see. And yet a machine like this, if you actually use it, stays on.

You can do the arithmetic yourself, and we suggest doing it with your own bill rather than our numbers:

kWh per year = average watts × hours on per day × 365 ÷ 1,000

For orders of magnitude: a workstation with a professional GPU under sustained load draws around 800-1,000 W across the whole system; on but idle, around 150-250 W. A machine running around the clock with realistic mixed use lands somewhere between 2,500 and 7,000 kWh a year.

On European industrial tariffs that is an annual cost ranging from a few hundred to over a thousand euro. It is not the figure that kills a project, but it is the figure that costs a business case its credibility when it surfaces afterwards, at reconciliation. Put it in — and put in the fact that the machine needs cooling, in a room that also needs cooling.

Fine-tuning: what you can do in-house and what you cannot

A recurring question: "can we train it on our data?"

Let us separate two things. Light fine-tuning with techniques such as QLoRA on a 70-billion model needs about 38 GB of peak memory and runs in a few hours on a professional workstation. It is feasible in-house and it is almost always what is actually needed.

Full training of a 70-billion model inside a company does not happen. This is not about ambition: it is about orders of magnitude. If someone proposes it, ask on what hardware.

It is worth adding that in most business cases fine-tuning is not needed at all: what is needed is a good document retrieval system (RAG) that hands the model your documents at question time. It is cheaper, easier to keep current, and easier to correct when it gets something wrong.

When it makes sense, and on what reasoning

If the criterion is cost per token, on-premise almost never beats cloud provider APIs for an SME. Market analyses put the economic break-even around a billion tokens a month, with payback periods near thirty months; at ten billion a month, payback drops to two. A mid-sized European manufacturer does not generate those volumes, and anyone presenting local AI as a saving is running a calculation that does not add up.

There are, however, four reasons on-premise holds up very well — and they are reasons other than price.

Where the data goes. If you have to process contracts, designs, price lists, clinical records or trade-secret material, knowing that content never leaves your network is not an aesthetic preference: it is a contractual or regulatory condition.

High, predictable volume. A repetitive task chewing through thousands of documents a day changes the arithmetic completely compared with occasional conversational use.

Working offline. Plants with poor connectivity, deliberately isolated environments, business continuity that must not depend on an external provider.

Small, specialised models. An 8 or 27-billion model that does one thing well, on a €4,000 machine, is a solid project. A "generalist company ChatGPT" at 10 tokens per second competing with an online service ten times faster is a weak one, and you will find out three months in, when nobody uses it any more.

How to size it, in practice

The right order of questions is this, and it is not the order most people follow.

First: what is the task, in one sentence, and how many documents or requests a day. Second: does it need to be interactive or is batch fine — that single answer moves the hardware decision more than anything else. Third: what data does it need to touch and where does that data live. Fourth: what is the smallest model that does the job — and that answer is found by testing, not estimating. Only then do you choose the machine.

In our experience, those who start from hardware buy too much, and those who start from the use case often discover Build A is enough.

A note on model names

We named exactly one model in this article, OpenAI's gpt-oss, because its requirements are verifiable on the official model card: 117 billion total parameters and 5.1 billion active for the large version, running on a single 80 GB GPU, and 21 billion total with a 16 GB footprint for the small one. Apache 2.0 licence, so commercially usable.

Everywhere else we reasoned in size classes rather than names. That is deliberate: the "best open models of 2026" rankings in circulation contradict each other even two weeks apart, and citing a version that will be superseded next month is the fastest way to make an article age badly. When you choose, verify requirements on the official model card the day you install it.

Questions we get asked

Can I run AI on the server I already have?

Almost certainly not, if it is an ordinary business server: without a suitable graphics card the model runs on the CPU at unusable speed. The right question is not whether the server is enough, but which card you can fit and whether the power supply and cooling will take it.

How much memory does a 70-billion-parameter model need?

Between 35 and 40 GB at Q4 compression. That means a professional GPU or a Mac with 128 GB of unified memory: a 32 GB consumer card does not get there.

Mac or NVIDIA?

It comes down to one thing: interactive or batch. The Mac costs less and holds larger models, but generates at 8-15 tokens per second. If somebody is sitting in front of the screen waiting, buy NVIDIA; if the work runs overnight against a queue of documents, the Mac is the economically sensible choice.

What does it cost to run?

Between a few hundred and over a thousand euro a year, depending on how long it stays on and under load. The formula for working it out on your own tariff is earlier in this article: do it before approving the spend, not after.

Can we train it on our own data?

Light fine-tuning, yes, in-house, in a few hours. Full training of a large model, no — and that is not a budget question. But in most cases neither is needed: what is needed is a good document retrieval setup, which costs less and updates itself when your documents change.

If I wait, will prices come down?

Not in the near term. IDC does not expect memory prices to stabilise before the end of 2027, and HBM demand from data centres keeps taking capacity away from everything else. If you have a real need now, waiting probably costs you more.

In short

A departmental machine for vertical tasks sits around €4,300-7,000 and delivers 60-90 tokens per second. A Mac with lots of memory costs less and holds bigger models, but at 8-15 tokens per second: excellent in batch, awkward interactively. A professional workstation sits around €19,000 and is the only one that handles fine-tuning and large models at full speed. Hardware prices are rising and will not stabilise before the end of 2027. And electricity belongs in the calculation up front, not afterwards.

If you want to work out which build your case falls into — or whether you should skip it entirely — let's talk. We ask the first two questions, and they are about the task, not the machine.

Related articles:

• When local AI is not worth it: the arithmetic no vendor shows you
• How much does enterprise AI integration cost?
• Data Lake in retail: how 120 stores stopped searching for information