Intel Gaudi 3 is an AI accelerator that Intel and Supermicro position as a more cost-efficient alternative to NVIDIA GPUs for training and LLM inference. It offers 128 GB of HBM2e per accelerator, built-in 800 GbE networking, and runs PyTorch and vLLM. Here is where it is a genuine option and where the breadth of the CUDA ecosystem still decides the matter.
Most hardware discussions about LLMs boil down to “which NVIDIA part”. Gaudi 3 deserves a look if you are planning inference or training at scale, you run mainstream open-weight models and you care about the cost of every accelerator you deploy. Below we cover the system architecture, the state of the software, how it compares with NVIDIA HGX B200, and how to test Gaudi 3 on your own workload before committing to a purchase.
Intel Gaudi 3 architecture: an 8-accelerator system
Gaudi 3 is not a PCIe card you can drop into any server. The accelerators use the OAM form factor and are mounted eight at a time on a shared baseboard. Supermicro ships them in the SYS-822GA-NGR3 system, with two baseboard options:
- HL-325L – air-cooled,
- HL-335 – liquid-cooled, for denser deployments.
Key system specifications:
- 8 Gaudi 3 accelerators, each with 128 GB of HBM2e, giving roughly 1 TB (1,024 GB) of accelerator memory per server,
- 6 OSFP 800 GbE ports on the baseboard – scale-out networking for clustering servers is part of the platform rather than an add-on,
- dual Intel Xeon 6 host processors.
Built-in 800 GbE networking
This is the most interesting part of the design. Gaudi 3 relies on standard Ethernet, both between accelerators and between servers. In practice, a multi-node cluster is built on Ethernet switches your network team already understands, not on a separate specialised fabric. Fewer network cards to buy and a single type of cabling add up to real savings across several servers. For a single inference server the advantage matters much less.
Cooling and power
An eight-accelerator system draws a lot of power and produces a lot of heat. The air-cooled variant needs a facility with enough power and cooling headroom per rack; the liquid-cooled one needs liquid cooling infrastructure. This is not hardware for an office server room, which is why it usually ends up in colocation at a professional data centre.
Software: PyTorch, vLLM and the Intel Gaudi software suite
Hardware is only half of the decision. Whether Gaudi 3 works for your project depends largely on software.
Intel provides the Intel Gaudi software suite: drivers, a graph compiler, communication libraries and PyTorch integration. PyTorch code can run on Gaudi with modest changes, as long as it does not contain code written directly for CUDA. For model serving, the key point is that vLLM supports Intel Gaudi. You get the same inference server, with continuous batching and an OpenAI-compatible API, that you would use on NVIDIA GPUs. An application talking to the model through that API does not need to know what hardware sits underneath. We cover inference servers in more depth in Ollama vs vLLM vs SGLang vs NVIDIA NIM.
Where to be honest about the limits
The CUDA ecosystem is simply broader. It has been built up for well over a decade, and most AI tooling is written for NVIDIA GPUs first. What that means in practice:
- New models. Popular architectures run on Gaudi, but a freshly released model may take time to be optimised and validated. On NVIDIA GPUs it usually works on day one.
- Custom CUDA kernels. If your code or its libraries include kernels written for CUDA, they have to be ported or replaced. That is real engineering work.
- Quantisation formats. Some quantisation formats and kernels were designed with NVIDIA GPUs in mind. Before choosing a model, check that the variant you want (for example FP8 or 4-bit) is supported on Gaudi.
- NVIDIA tooling. TensorRT-LLM and NVIDIA NIM run only on NVIDIA GPUs. If your team depends on them, moving to Gaudi means changing your stack.
For typical inference of mainstream models through vLLM, these limits may not matter at all. For a research team testing new architectures every week, they will.
Gaudi 3 vs NVIDIA HGX B200: a qualitative comparison
The closest reference point for an 8-accelerator Gaudi 3 system is NVIDIA’s 8-GPU HGX B200 platform. We do not quote performance figures here, because they depend on the model, quantisation, context length and number of concurrent users. We compare what can be compared fairly.
| Feature | Gaudi 3 (Supermicro SYS-822GA-NGR3) | NVIDIA HGX B200 |
|---|---|---|
| Accelerators per system | 8 (OAM) | 8 GPUs |
| Memory per accelerator | 128 GB HBM2e | 180 GB HBM3e |
| Memory per system | approx. 1 TB | approx. 1.44 TB |
| Memory generation | HBM2e – older, lower bandwidth | HBM3e – newer, higher bandwidth |
| Accelerator-to-accelerator link | Ethernet integrated into the accelerators | NVLink at 1.8 TB/s |
| Server-to-server networking | 6× OSFP 800 GbE on the baseboard | separate network cards chosen per configuration |
| Host CPUs | 2× Intel Xeon 6 | 2× Intel Xeon 6 or AMD EPYC 9005/9004 |
| Cooling | air (HL-325L) or liquid (HL-335) | air (8U/10U) or liquid (4U) |
| Software | Intel Gaudi software suite, PyTorch, vLLM | CUDA, TensorRT-LLM, NIM, vLLM, SGLang and practically the whole AI ecosystem |
| New models | need validation | usually work immediately |
| Price positioning | cost-efficient alternative, custom quote | top tier, custom quote |
The takeaway: B200 has more, faster memory and a more mature ecosystem. Gaudi 3 competes on cost and on simpler networking. For a full comparison with B300, H200 and RTX PRO 6000, see Best GPU for LLMs in 2026.
What fits into a Gaudi 3 system’s memory
Around 1 TB per server is a lot, but it is not unlimited. The rule of thumb: weights equal parameters × bytes per parameter (BF16 = 2 bytes, FP8 = 1 byte, 4-bit ≈ 0.5 bytes), plus a KV cache that grows with context length and concurrent users, plus 10–20% overhead. MoE models need memory for all parameters, not only the active ones. Examples using models available in November 2026:
| Model | Parameters | Weights only | Gaudi 3 accelerators needed |
|---|---|---|---|
| Qwen3.8-27B | 27.8B | approx. 56 GB in BF16 | 1, with room for KV cache |
| Mistral Small 4 | 119B | approx. 238 GB in BF16, approx. 119 GB in FP8 | 2–4, depending on precision and context |
| Llama 4 Maverick | approx. 400B (MoE) | approx. 400 GB in FP8 | 4–8 |
| DeepSeek-V4.1-Flash | approx. 763B (MoE) | approx. 763 GB in FP8 | the whole system, little room for KV cache |
These are memory estimates, not confirmation that a given model at a given precision has been validated on Gaudi 3. We walk through the calculation in How much VRAM does an LLM need.
Who Intel Gaudi 3 makes sense for
Gaudi 3 is a sensible choice when most of the following apply:
- Scale. You are planning several servers or more, and hardware and power are a significant line in the project budget.
- Mainstream models. You serve or fine-tune well-known open-weight models through PyTorch and vLLM, not experimental architectures.
- Stable workload. The same model, or a few models, run in production for months, so a one-off validation effort pays back.
- Skills. You have ML engineers, or a partner, who can validate and maintain the stack.
- Ethernet. You want to build the cluster on standard network infrastructure.
NVIDIA is likely the better fit when:
- you need the newest models from launch day,
- your code relies on custom CUDA kernels, TensorRT-LLM or NIM,
- you have a small team with no time to validate a platform,
- you need one or two accelerators – in that case an RTX PRO 6000 server or a DGX Spark is usually simpler and cheaper to start with than a full 8-accelerator system.
How to evaluate Gaudi 3: a step-by-step proof of concept
Do not decide on the basis of vendor slides – Intel’s or NVIDIA’s. A fair comparison requires a test on your own workload. We suggest this plan:
- Define the workload. A specific model and precision, typical and maximum context length, target number of concurrent users, required time to first token.
- Check compatibility. Does the model, in your chosen quantisation format, run on Gaudi 3 with the current software and vLLM releases?
- Measure performance. Time to first token, tokens per second per user and aggregate throughput at your target concurrency. Run the same tests on the NVIDIA platform you are comparing against.
- Measure quality. Use the same test prompts and compare outputs with a reference. Different kernels and quantisations can shift results slightly.
- Work out the cost. Hardware or service price, power, colocation and staff time, per user or per million tokens.
- Plan your exit. If the application uses an OpenAI-compatible API, changing hardware later does not mean rewriting the application.
Gaudi 3 in an EU data centre: purchase, colocation and operations
As a Supermicro partner, we supply systems with Gaudi 3 accelerators as well as with NVIDIA GPUs, install them and run them in our Tier III, ISO 27001-certified data centre in Poland. Your data stays in Poland and the EU. For dense configurations we offer liquid-cooled racks. You can buy a server and place it in server colocation, or start with a GPU server as a service. We take over 24/7 administration completely, or leave you with root access. When buying and colocating beats the cloud is covered in our cloud vs GPU colocation TCO analysis.
FAQ: frequently asked questions
Does Intel Gaudi 3 support vLLM?
Yes. vLLM supports Intel Gaudi, so you can serve models through the same inference server and the same OpenAI-compatible API as on NVIDIA GPUs. Before deploying, check that your specific model and quantisation format are supported in the current release.
How much memory does Intel Gaudi 3 have?
Each accelerator has 128 GB of HBM2e. A Supermicro SYS-822GA-NGR3 system with eight accelerators has roughly 1 TB of accelerator memory in total. For comparison, NVIDIA B200 has 180 GB of HBM3e per GPU.
Gaudi 3 vs H100 or H200: which is better for inference?
There is no universal answer, and generic comparison charts are not worth much. H200 has 141 GB of HBM3e per GPU and the full CUDA ecosystem; Gaudi 3 has 128 GB of HBM2e and lower price positioning. A test of your model at your concurrency will decide.
Does CUDA run on Gaudi 3?
No. Gaudi 3 uses the Intel Gaudi software suite and PyTorch integration. PyTorch code usually ports with modest changes, but custom CUDA kernels need to be rewritten or replaced.
Is Gaudi 3 suitable for training models?
Yes, Gaudi 3 is designed for both training and inference, and the built-in 800 GbE networking makes clustering easier. For training, though, the ecosystem matters even more, so validating your training pipeline should be part of the test.
Wondering whether Gaudi 3 would lower the cost of your deployment? We will prepare a comparison with an NVIDIA platform and a custom quote. See our GPU servers or get in touch.