What hardware does local AI need in a company?

In short: For a first test, the hardware already in your office is often enough: a small language model needs around 5 GB of memory in a 4-bit version, a large 70B model around 43 GB. For everyday team use, two things matter most: how much system or graphics memory the device has, and how many requests arrive at the same time. A compact AI system with 128 GB that can serve a small team cost from around €5,000 excluding VAT at resellers in September 2026. If you want to process many requests in parallel or automatically, you need more hardware.

Prices and model details: as of November 2026.

"Is our server good enough for this?" It is one of the first questions when a law firm, a medical practice or a small office considers running AI of its own. The answer depends less on the processor than many people assume. With local AI hardware, memory is what counts most, and it can be estimated fairly precisely once the model and the number of simultaneous users are known.

What "local AI" means in terms of hardware: The language model, also called an LLM (large language model), runs on a computer or server in your own company, not in a provider's data center. This is also known as a self-hosted or on-premise setup. Inputs are processed where the device is located. Why that matters for confidential data is covered in the article Where does your data end up when employees use ChatGPT?

What determines the hardware requirements for local AI?

Mainly two things: how much fast memory is available and how many requests run at the same time. Behind them are five factors.

1. Model size. Language models are named after the number of their parameters, the numerical values they have learned. "8B" stands for 8 billion, "27B" for 27 billion, "70B" for 70 billion. Larger models can handle more demanding tasks, but they need more memory and run more slowly.

2. Quantization. A model can be made smaller by storing each parameter with fewer bits, for example 4 instead of 16. That saves around three quarters of the memory. A study with more than 500,000 evaluations shows that well-implemented 4-bit versions come close to full accuracy on many tasks (Kurtic et al., ACL 2025). For particularly sensitive tasks, it is still worth comparing against a more precise version.

3. System memory and graphics memory (RAM/VRAM). This is the main bottleneck. For a model to respond quickly, it has to fit entirely into memory that the processing chip can access quickly: on a graphics card, that is its own graphics memory (VRAM); on devices with unified memory, such as Macs or NVIDIA's GB10 systems, it is the shared system memory.

4. Context length. The context is the amount of text the model takes into account at once: the question, the conversation so far and any uploaded documents. More context needs more memory, which is why the Ollama software adjusts its default to the available graphics memory, from around 4,000 tokens with less than 24 GB of graphics memory to around 256,000 tokens from 48 GB (Ollama documentation). A token roughly corresponds to part of a word. Modern models, however, are built for long documents: in Qwen3.8-27B, only 16 of 64 layers use classic attention, whose memory grows with the text, and they do so with only 4 key-value heads. A request with 32,000 tokens of context therefore needs only around 2 GB extra (our own calculation based on the model card).

5. Concurrency. This is where the most common misunderstanding lies. Ten regular users are not the same as ten simultaneous requests. Someone who asks three questions in the morning and has a document summarized in the afternoon only occupies the device for the seconds or minutes in which the answer is generated. Requests become simultaneous only when several are being processed at exactly the same moment. Each simultaneous request needs its own memory for its context. In Ollama, for example, memory requirements grow with the number of parallel requests times the context length; by default it processes one request per model at a time and queues the rest (Ollama FAQ). Server software such as vLLM, by contrast, is designed to batch many requests together (vLLM documentation).

How much memory does a local LLM need?

In the common 4-bit version, a small 8B model needs around 5 GB, a mid-sized 27B model around 19 GB and a 70B model around 43 GB, each for the model file without context.

Rule of thumb: memory needed for the model file in GB ≈ parameters in billions × bits per parameter ÷ 8. The widely used 4-bit format Q4_K_M actually uses just under 4.9 bits per parameter, because some parts are kept at higher precision. An 8B model therefore needs around 4.9 GB, a 70B model around 43 GB (llama.cpp). On top of that comes memory for context and the operating system.

Two examples from the official file listings show how much quantization reduces the requirement:

Sources: llama.cpp, Hugging Face, as of November 2026. Values for the model file, without context and operating system.

Alibaba's Qwen3.8-27B is released under the Apache 2.0 license, so it may be used commercially, and it handles up to 262,144 tokens of context (model card). OpenAI's open model gpt-oss takes a different approach: according to the developer, the large version with 117 billion parameters fits on a single 80 GB graphics card, and the small one with 21 billion parameters runs within 16 GB of memory (OpenAI model card).

This gives a rough guide to which model size fits into how much memory:

A guide for the model file, without context. On devices with unified memory, the operating system also needs part of it.

Why large models still run smoothly: Many current models are so-called mixture-of-experts models. gpt-oss-120b, for example, has 117 billion parameters but uses only 5.1 billion of them for each token (OpenAI model card). Memory requirements depend on all parameters, speed on the active ones. That is why such models run smoothly on devices with plenty of memory.

Can I test local AI on existing hardware?

Yes, for a first impression an existing computer is usually enough. Free programs such as Ollama, LM Studio or the chat interface Open WebUI let you run a model on a single computer.

• A current laptop or office PC with 16 GB of RAM is enough for small models up to around 8B in 4-bit. Without a graphics card, the model responds more slowly, but it is enough to get an impression.
• A Mac with Apple silicon and plenty of memory makes a good test machine, because its unified memory is also available to the graphics cores. From around 36 GB, mid-sized models such as Qwen3.8-27B in 4-bit are possible (model file around 19 GB).
• A PC with a graphics card is fast as long as the model fits into graphics memory. With 16 GB of VRAM, for example, gpt-oss-20b runs.

A test like this shows whether the answers are good enough for your tasks. It does not yet show how the system behaves when several employees use it at the same time.

Three scenarios: from a test to an AI server

Existing hardware is enough for a test, a compact AI system or a workstation for a small team, and an AI server with professional graphics cards for many simultaneous requests. How many people a device can actually serve well can only be determined reliably by measuring.

Prices: retailer figures excluding VAT, as of September to November 2026. *Graphics cards: lowest retail price according to a price comparison site, which does not state whether VAT is included. Because memory chips are scarce, prices rose significantly in 2026 (Hardwareluxx); check the current price before buying.

Compact AI systems. NVIDIA's DGX Spark with the GB10 chip fits on any desk and has 128 GB of unified memory with 273 GB/s of bandwidth and a 240-watt power supply. NVIDIA states that models with up to 200 billion parameters are possible (according to NVIDIA). At resellers, 128 GB devices cost around €5,000 to €8,100 excluding VAT in September 2026, depending on the manufacturer and SSD size, and a 64 GB version starts at around €3,150 excluding VAT (pi3g). Devices with AMD's Ryzen AI Max+ 395 offer up to 128 GB, of which up to 96 GB can be used as graphics memory on Windows (AMD). Apple's Mac Studio is available with the M5 Max and up to 128 GB (up to 614 GB/s) or with the M5 Ultra and up to 512 GB (1.2 TB/s) (Apple); in Germany, the entry model with M5 Max and 36 GB costs €2,999 including VAT, or around €2,520 excluding VAT (Macwelt), and configurations with a lot of memory cost considerably more.

Which device for which work. When reading in long documents, compute matters most; when writing the answer, memory bandwidth matters most, meaning how fast the device reads the model data from memory (Davies et al., 2025). GB10 systems such as the DGX Spark bring plenty of compute, up to 1,000 trillion operations per second according to NVIDIA (NVIDIA); they are therefore strong at reading in documents and batch many simultaneous requests well. The Mac Studio has very high bandwidth of 614 GB/s or 1.2 TB/s and writes individual answers quickly (Apple). Which strength matters more depends on whether your team mainly analyzes long documents or asks many short questions.

Workstations and servers with graphics cards. Professional graphics cards have their own very fast memory. The RTX PRO 5000 Blackwell offers 48 GB, with the lowest retail price in Europe at just under €7,700 in early November 2026 (GPU Prix). The RTX PRO 6000 Blackwell has 96 GB; its lowest retail price in Europe at the same time was around €11,000 (GPU Prix). Case, processor, RAM and power supply come on top. Systems like these play to their strengths when many requests run at the same time, for example in automated document processing.

What else matters besides the hardware?

Besides the device, what matters most is operation, updates and software, and with local AI all of this is in your hands. Treat an AI system under continuous load like a server: a location with enough airflow, not inside a closed cabinet, and a suitable limit for simultaneous requests in the server software. This is how local AI stays reliable:

• Economical continuous operation: Compact AI systems get by with a moderate power draw. The DGX Spark, for example, has a power supply rated at no more than 240 watts.
• Updates on your schedule: You decide when an update or a new model is installed. A proven model stays in use for as long as it serves you; no provider swaps it out overnight.
• Backups by your rules: You back up settings and conversations just like the rest of your company data.
• Support like any other IT: Someone in-house or a service provider looks after the system, much like a server or phone system.
• Software that grows with you: Chat interfaces and user management are freely available as software. Connected to your documents, the chat becomes a tool that knows your own knowledge.

In return, you get fixed, predictable costs with no usage-based fees, and your data stays in the building. How the costs add up is covered in the article What does local AI really cost?

Frequently asked questions

Is a normal office PC enough for local AI?
For a first test, often yes. With 16 GB of RAM, small models up to around 8 billion parameters run in a 4-bit version, though slowly without a graphics card. For everyday team use, a device with considerably more fast memory makes sense.

Do I need a graphics card for local AI?
Not necessarily. What matters is fast memory that an AI-capable chip can access. A graphics card with its own VRAM provides this, but so does a device with unified memory, such as a Mac with Apple silicon or a system with NVIDIA GB10 or AMD Ryzen AI Max.

How much VRAM does a 70B model need?
In the common 4-bit format Q4_K_M, the model file of a 70B model is around 43 GB (llama.cpp). Memory for the context comes on top. In practice, that means a graphics card with 48 GB, with little headroom, or a device with 64 GB or more of unified memory.

What does an on-premise AI server cost for a small business?
For a small team, a compact AI system is often enough: devices with NVIDIA's GB10 chip and 128 GB cost around €5,000 to €8,100 excluding VAT at resellers in September 2026 (pi3g). Servers for many simultaneous requests cost considerably more; a single professional graphics card with 96 GB alone was around €11,000 in early November 2026 (GPU Prix). What power and support add is covered in the article What does local AI really cost?

How many employees can one device serve?
That depends less on the number of employees than on the number of simultaneous requests, the model and the length of the documents. It can only be answered reliably by measuring under your own conditions.

Mac or PC for local AI?
Both work. A Mac with plenty of unified memory loads large models too and writes individual answers quickly. A PC or server with a professional graphics card is particularly well suited to many simultaneous requests and is the usual platform for server software such as vLLM.

Can I upgrade later?
Workstations and servers can often take an additional graphics card. On compact devices with unified memory, the memory is usually soldered in, so you choose the size when you buy. According to NVIDIA, two DGX Spark units can be linked to run larger models.

Does a local LLM work without the internet?
Yes. Once the model has been downloaded, it answers requests on your own device. A connection is only needed for updates and new models.

Conclusion: memory and concurrency decide

What hardware local AI needs depends mainly on memory and the number of simultaneous requests. It does not require a server rack: an existing computer is often enough to get started, and a compact system with plenty of fast memory can serve a small team. A server with professional graphics cards only becomes necessary when many requests need to run at the same time.

Alae sets up local AI on companies' own hardware and connects it to their knowledge and workflows. Find out more on our Local AI page.

We measure together with you, under your conditions, how many simultaneous requests your team actually needs and how fast a device responds. Would you like to know whether your existing hardware is enough? Let's check it together.

Request a hardware check

Sources

• NVIDIA: DGX Spark Hardware Overview (last updated 21 Oct 2026)
• Hardwareluxx: Memory shortage drives DGX Spark to €4,800 (27 Feb 2026, German)
• pi3g: NVIDIA DGX Spark Price, reseller prices excl. VAT (24 Sep 2026)
• Apple: Mac Studio technical specifications
• Macwelt: Mac Studio M5 Max and M5 Ultra (German)
• AMD: Ryzen AI Max+ Upgraded, up to 96 GB of graphics memory on Windows (29 Jul 2025)
• GPU Prix: RTX PRO 5000 Blackwell, prices in Europe (as of 2 Nov 2026)
• GPU Prix: RTX PRO 6000 Blackwell, prices in Europe (as of 2 Nov 2026)
• llama.cpp: quantization, bits per weight and file sizes
• Hugging Face: Qwen3.8-27B model card
• Hugging Face: Qwen3.8-27B-GGUF file sizes
• Hugging Face: OpenAI gpt-oss-120b model card
• Kurtic et al.: "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization (ACL 2025)
• Davies et al.: LIMINAL: Exploring The Frontiers of LLM Decode Performance (2025)
• Ollama: FAQ
• Ollama: Context length
• vLLM: documentation