A self-hosted LLM is a language model running on GPUs you control: in your own server room, on a rented server or in colocation. Your data never goes to a third-party API provider, and the bill does not grow with every token. This guide walks through the four decisions you need to make before the first user sends the model a prompt.
We write this as the team that runs GPU servers every day in a Tier III data centre in Poland. As of November 2026, the open-weight model market changes monthly and hardware every few quarters, but the selection logic stays the same. Each step links to a detailed article in this series, so treat this page as the map.
Why run a self-hosted LLM when APIs exist?
A public API is the fastest way to start: an API key, a few lines of code and you have a prototype. The problems begin when that prototype has to work on real company data in production. Running your own model solves several of them:
- Your data stays with you. Contracts, medical records, source code and customer data never leave infrastructure you control. In our case that is an ISO 27001-certified data centre in Poland, inside the EU.
- Predictable cost. You pay for hardware (a monthly fee or a purchase), not per request. Another user or another workflow does not raise the bill until the GPU runs out of capacity.
- Control over the model version. An API provider can change or retire the model your application depends on. A model on your own server changes only when you decide.
- Customisation. You can fine-tune the model on your own data and connect it to internal knowledge bases (RAG) and systems you would never expose to the internet.
There are costs too. Someone has to deploy, monitor and update the model. The strongest closed models are still better than open ones at some tasks. And you have to size the hardware up front; getting it wrong is expensive. If you process little data and none of it is sensitive, an API may well be the better choice. We cover the risks of sending data to public services in our article on deploying local LLMs without data leaks.
Step 1: choose the model for the task, not the other way round
The most common mistake is starting with whichever large model is in the headlines that month. Start with the task instead: summarising documents, answering questions from a knowledge base, triaging tickets, writing code, transcription. Then look for the smallest model that does it well enough. As of November 2026, open-weight models fall roughly into three groups:
- Small, 7–31B parameters: for example Gemma 4 (12B and 31B), Qwen3.8-27B, gpt-oss-20b, Devstral Small 2 for code (see our self-hosted AI coding assistant guide), and the Polish-language Bielik and PLLuM. They fit on a single GPU and work well for narrow, well-defined tasks.
- Medium, around 100–200B: for example gpt-oss-120b, Mistral Small 4 (119B), Mistral Medium 3.5 (128B), Llama 4 Scout (109B) and Qwen3.8-Flash-Next (180B). A sensible balance between quality and hardware cost.
- Large MoE models, from about 300B to over 2T: for example GLM-5.3-Flash (~321B), DeepSeek-V4.1-Flash (~763B), Mistral Large 4 (~1T), DeepSeek-V4-Pro (~1.65T) and Kimi K3 (~2.78T). The highest quality, but they need a multi-GPU server or a cluster.
The second criterion is the licence. Apache 2.0 and MIT allow unrestricted commercial use. Vendor-specific licences (Llama, Mistral, Kimi, some Qwen and GLM models) need reading before deployment because they can contain restrictions. A full comparison with licences and minimum hardware is in the best open-source LLMs to self-host in 2026. If you work in Polish, also see self-hosting the Polish LLMs Bielik and PLLuM: they are small, cheap to run and tuned for Polish, although large global models usually score higher in general benchmarks.
Step 2: calculate memory and choose the GPU
What hardware you need is decided mainly by GPU memory (VRAM). The rule of thumb: weights ≈ number of parameters × bytes per parameter. That is 2 bytes in FP16/BF16, 1 byte in FP8 and about 0.5 bytes with 4-bit quantisation. On top of that comes the context memory (KV cache), which grows with context length and the number of concurrent users, plus 10–20% overhead. MoE models need memory for all their parameters, even though only a fraction of them is computed for each token.
| Model size | FP16/BF16 | FP8 | 4-bit |
|---|---|---|---|
| 12B (e.g. Gemma 4 12B) | ~24 GB | ~12 GB | ~6 GB |
| 31B (e.g. Gemma 4 31B) | ~62 GB | ~31 GB | ~16 GB |
| 117B (gpt-oss-120b) | ~234 GB | ~117 GB | ~59 GB |
| 321B (GLM-5.3-Flash) | ~642 GB | ~321 GB | ~161 GB |
| 763B (DeepSeek-V4.1-Flash) | ~1,526 GB | ~763 GB | ~382 GB |
The table shows weights only, so real requirements are higher. Quantisation is the cheapest way to shrink the hardware you need, as we explained in cutting AI infrastructure costs with quantisation. With many concurrent users, memory bandwidth matters as much as capacity, because it determines how many tokens per second each user gets. Fine-tuning needs far more memory than inference.
The full method with worked examples is in how much VRAM does an LLM need, and a comparison of specific cards in the best GPU for LLMs: B300, B200, H200, RTX PRO 6000 and Gaudi 3.
Step 3: decide where the model will run
Once you know the memory requirement, pick the form. All options below run in our data centre, so you can start small and grow without changing provider.
| Option | Best for | Strengths | Limitations |
|---|---|---|---|
| VPS with GPU / ready-made GPU package | getting started, small models, one team | fast setup, fixed monthly fee | 12 to 96 GB of GPU memory |
| Bare-metal GPU server (up to 8 GPUs) | production, many users, large models | the whole machine is yours, choice of GPUs | higher cost, needs careful sizing |
| DGX Spark as a Service | R&D, prototypes, fine-tuning up to 70B | 128 GB unified memory, NVIDIA software stack | 273 GB/s bandwidth, not for many concurrent users |
| Cluster of 2 or 4 DGX Spark units | working with ~400B or ~700B models | 256 or 512 GB of combined memory | same bandwidth limitation |
| Multi-node GPU cluster | training, the largest models, heavy traffic | scale-out, fast interconnect | complexity, needs liquid cooling |
| Colocation of your own hardware | long horizon, steady workload | you own the hardware; we provide power, cooling, network and operations | capital expense up front |
Ready-made packages and larger configurations are on our DeepSeek GPU packages and GPU servers pages. The packages run not only DeepSeek but also Llama, Qwen, Mistral, Gemma or Whisper. How the subscription model works and what the fee covers is explained in GPU as a Service: GPU server rental.
DGX Spark is a category of its own: an NVIDIA GB10 Grace Blackwell chip with 128 GB of unified memory in one unit, and 256 or 512 GB in a cluster of two (direct QSFP cable) or four units (200 GbE switch). The limitation is memory bandwidth (273 GB/s), far below HBM-based GPUs, so it is hardware for development, prototyping and private inference for a small team, not for serving hundreds of users. Details: NVIDIA DGX Spark as a Service and DGX Spark clusters for 400B–700B models.
If you would rather buy, as a Supermicro and NVIDIA partner we supply servers with NVIDIA GPUs or Intel Gaudi 3 and install them in colocation in our data centre. An office server room is rarely suitable: an NVIDIA B300 GPU draws up to 1,100 W, and according to Supermicro an air-cooled rack holds up to 32 B200 GPUs versus up to 96 in a liquid-cooled one. More in GPU server cooling: air vs liquid.
Step 4: operations, monitoring and security
Getting a model running takes a few hours. Keeping it running in production is ongoing work. It starts with the serving engine: Ollama is fine for testing and a single user, vLLM or SGLang are better for multi-user production, and in the NVIDIA ecosystem there is also NIM. Most of them expose an OpenAI-compatible API, so applications do not need to know which model or hardware sits underneath. See Ollama vs vLLM vs SGLang vs NVIDIA NIM.
What you need to keep under control:
- Monitoring: GPU utilisation and memory, temperatures, request queue length, latency, errors.
- Updates: drivers, CUDA, the inference engine and the model itself, always on a test environment first.
- Access: the model should not have a public endpoint. Use authentication, a VPN or reverse proxy, and per-team permissions.
- Logs: prompts can contain personal data, so logs fall under GDPR as well. Decide what you log and how long you keep it.
- Backups: fine-tuned weights, the vector database and configuration.
- Supply chain: download models from the vendors’ official repositories and verify what you run.
You can do this yourself with root access or hand it to us as 24/7 server administration. What that looks like day to day is described in MLOps in practice.
The hardware ladder: from prototype to production
A condensed path from pilot to production. You do not have to start at the bottom, but it helps to know where the next rung is.
| Rung | Hardware | Memory for the model | Typical use |
|---|---|---|---|
| 1 | Ready-made GPU package (RTX A2000 up to 2× A40) | 12–96 GB | pilot, small models, one team |
| 2 | NVIDIA DGX Spark | 128 GB unified | prototypes, fine-tuning up to 70B, inference up to 200B |
| 3 | Cluster of 2 or 4 DGX Spark units | 256 / 512 GB | development on ~400B / ~700B models |
| 4 | Server with RTX PRO 6000 Blackwell (up to 8 cards) | up to 768 GB GDDR7 | production inference, RAG, multi-tenant |
| 5 | HGX B200 / HGX B300 (8 GPUs) | ~1.4 TB / ~2.3 TB HBM3e | large MoE models, heavy traffic, training |
| 6 | HGX cluster or GB300 NVL72 | multiple nodes / 72 GPUs in one NVLink domain | training and the largest models at scale |
An alternative to NVIDIA on rungs 4–5 is Intel Gaudi 3: eight accelerators with 128 GB HBM2e each and built-in 800 GbE networking, supported by PyTorch and vLLM. We look at it in Intel Gaudi 3 for LLM inference. Detailed criteria for moving between rungs are in DGX Spark vs RTX PRO 6000 server vs HGX B200.
What a self-hosted LLM costs: how to work it out
With an API, cost is the number of tokens multiplied by the list price. It grows linearly with usage and is hard to forecast, especially with RAG and agents, where every request carries a lot of context (see tokens in large language models). With your own GPU, cost is fixed: a monthly fee, or hardware purchase plus colocation. Each additional request costs practically nothing until you reach the machine’s capacity.
So the break-even point comes from comparing your projected monthly API spend with the fixed cost of hardware and its operation. Our ready-made GPU packages are the entry point with a published monthly price; DGX Spark as a Service and Blackwell servers are quoted individually. The method, with a worked example, is in how to calculate the ROI of an AI deployment, and renting versus buying with colocation is compared in our cloud vs GPU colocation TCO analysis.
Compliance: GDPR, the AI Act, NIS2 and DORA
A self-hosted model in an EU data centre does not remove your legal obligations, but it simplifies them considerably. Under GDPR, the questions about transfers outside the EEA and about what an API provider does with your prompts disappear. The standard controller duties remain: legal basis, data minimisation and log retention.
The AI Act regulates use cases, not hosting location. Recruitment or employee evaluation are high-risk areas wherever the model runs. The EU has announced that some deadlines may be postponed, so check the current timetable. For organisations within the scope of NIS2 or DORA, an AI system becomes part of the ICT estate: it needs risk assessment, supplier oversight and audit-ready documentation. We go deeper in self-hosted AI and EU law: GDPR, the AI Act, NIS2 and DORA. We also run NIS2 audits.
FAQ: frequently asked questions
How much does it cost to self-host an LLM?
It depends on model size and the number of users. Our ready-made GPU packages, which run models up to a few tens of billions of parameters, have fixed monthly prices listed on the packages page. Blackwell servers and DGX Spark as a Service are quoted individually. Add operations if you do not run them yourself.
What server do I need to run an LLM?
For a pilot with a small model, a package with a single 24–48 GB card is enough. For prototypes on larger models and fine-tuning, DGX Spark works well. A multi-GPU server makes sense once you know the model and the expected number of users.
Is a self-hosted LLM as good as commercial cloud models?
For narrow tasks on your own data, such as answering from a knowledge base or summarising documents, a well-chosen open model is usually enough. For the hardest general tasks, the strongest closed models can still be better. Test several models on your own examples before deciding.
How much VRAM does a 70B model need?
The weights alone take about 140 GB in FP16, about 70 GB in FP8 and about 35 GB with 4-bit quantisation. Add context memory, which depends on the number of users, plus 10–20% overhead.
Can I run DeepSeek, Llama or Qwen on my own server?
Yes, all of them are available as open-weight models. You need enough GPU memory and a licence that fits your use: DeepSeek is MIT-licensed, while Llama and some Qwen variants use vendor-specific licences.
Is a self-hosted LLM GDPR compliant?
Hosting alone does not make you compliant, but it helps: data does not leave your infrastructure in the EU, so there is no transfer to an outside provider. You still need a legal basis, access control and a log retention policy.
Want hardware sized for a specific model and number of users? Take a look at our GPU servers and ready-made GPU packages, then get in touch: we will propose a configuration, run the model on a test setup and quote the target setup.