A self-hosted AI coding assistant works much like the cloud tools – it suggests code, answers questions about your repository and helps with refactoring – but your source code never leaves your infrastructure. We explain which open-weight models are suitable, how to connect them to your IDE and how much hardware a team of 10, 50 or 200 developers needs.
AI coding assistants have become an everyday tool for many teams. The catch is that every completed line and every question about a function means sending a piece of code to an external provider. For a software house working on client code, a financial institution or a manufacturer with its own know-how, that is often unacceptable. A self-hosted AI coding assistant on your own GPUs solves the problem, but it requires careful choices about the model, the architecture and the hardware. We cover all three.
Why source code should stay inside the company
Code is often a technology company’s most valuable asset. There are several reasons not to send it outside:
- Intellectual property. Algorithms, business logic and system architecture are a competitive advantage. Even if a provider promises not to train on your data, the code still physically lands on its servers.
- Client contracts and NDAs. Software houses often work on code owned by the client. The contract may explicitly forbid sharing it with third parties – and the provider of a cloud assistant is a third party.
- Regulation. In financial services (DORA), for entities covered by NIS2 and in the public sector, using external services that process code and data requires vendor risk assessment, contracts and documentation. Your own infrastructure in the EU makes this much simpler.
- Secrets in code. Despite good practice, repositories still contain the odd API key, password in a config file or test data copied from production. An assistant sends context automatically, often without a conscious decision by the developer.
- Control over retention. With an external service, how long prompts and logs are kept is decided by the provider and its terms, which can change.
We look at the wider risks of sending data to public models in Local LLM deployment: how to avoid data leaks to public AI.
Open-weight models for coding (as of November 2026)
The open model market changes every month, so the list below reflects November 2026. We do not quote benchmark results: quality on your code, in your languages and frameworks, has to be checked in a pilot. The memory column is an approximate size of the weights alone, excluding KV cache and overhead.
| Model | Parameters | Licence | Weights (approx.) | Role in the team |
|---|---|---|---|---|
| Devstral Small 2 (24B) | 24B, coding model | Apache 2.0 | 24 GB in FP8, 12 GB in 4-bit | a good starting point; fits on one GPU with room for many users |
| Devstral 2 (123B) | approx. 125B, coding model | Mistral custom licence – check commercial terms | 125 GB in FP8, 63 GB in 4-bit | a larger coding model for a multi-GPU server |
| Kimi K2.7-Code | approx. 1.03T, coding model | modified MIT – read the additional terms | approx. 1 TB in FP8, approx. 515 GB in 4-bit | the largest on the list; needs an HGX-class server |
| Qwen3.8-27B | 27.8B, multimodal | Apache 2.0 | 28 GB in FP8, 14 GB in 4-bit | a versatile general model; also understands images such as screenshots |
| Qwen3.8-Flash-Next | 180B | Qwen Community License (custom) | 180 GB in FP8, 90 GB in 4-bit | a larger general model from the Qwen family |
| gpt-oss-120b / gpt-oss-20b | 117B / 21B | Apache 2.0 | approx. 59 GB / 11 GB in 4-bit | OpenAI’s general models with a permissive licence; 20b for smaller hardware |
Two practical notes. First, licensing: Apache 2.0 allows commercial use without extra conditions, while custom licences need to be read before deployment. Second, you do not have to pick a single model. A typical setup pairs a larger model for chat and agent mode with a smaller, fast model for autocomplete. We show how to calculate memory including context in How much VRAM does an LLM need.
Self-hosted AI coding assistant architecture
The solution has three layers, each of which can be swapped independently:
- An inference server with an OpenAI-compatible API. In production this is usually vLLM or SGLang, which serve many users at once. We compare the options in Ollama vs vLLM vs SGLang vs NVIDIA NIM.
- An API gateway. SSO or per-developer keys, rate limits, routing requests to the right model, and metadata logging without storing code.
- IDE extensions that support a custom OpenAI-compatible endpoint. Open-source examples include Continue (VS Code and JetBrains), where you point the extension at your own server, and Cline for VS Code. Developers keep working in their editor; requests go to your data centre.
Optionally, add a repository index: an embedding model running on the same server plus a vector database, so the assistant can find the right parts of a large codebase.
Chat, agents and autocomplete are different workloads
- Autocomplete sends short requests very often, at every pause in typing. Latency is everything: a suggestion that arrives late is useless. You need a small, fast model that supports FIM (fill-in-the-middle) – check the model card.
- Chat covers questions about code, explanations and test generation. Medium context length; a few seconds of waiting is acceptable.
- Agent mode runs multi-step tasks: the model reads many files, proposes changes and runs commands. Contexts are long and the KV cache grows the most. This is where the largest model your hardware allows pays off.
Security in practice
Keep the server reachable only from the company network or via VPN, log metadata instead of request content, exclude files containing secrets from the context the extension sends, and keep the inference server patched. These are the same rules we apply to every production LLM deployment.
How much hardware for a team of 10, 50 or 200 developers
The most common sizing mistake is counting “GPUs per developer”. What drives hardware is the number of concurrent requests and their context length, not headcount. Developers do not query the model non-stop. On the other hand, a single agent working across a large repository can take up more KV cache than several people using chat. So treat the table below as a starting point to be verified by measurement in a pilot.
| Team | Traffic profile | Suggested hardware | Example model setup |
|---|---|---|---|
| 10 developers | a few concurrent requests, short peaks | DGX Spark, a GPU VPS or a ready-made GPU package (e.g. A40 48 GB), or a server with 1× RTX PRO 6000 | Devstral Small 2 or gpt-oss-20b, quantised; on DGX Spark also gpt-oss-120b or Devstral 2 in 4-bit, at the cost of generation speed |
| 50 developers | dozens of requests at peak, steady autocomplete traffic | a server with 2–4 RTX PRO 6000 GPUs (96 GB each) | Devstral 2 or gpt-oss-120b for chat and agents, a small autocomplete model on a separate GPU |
| 200 developers | continuous traffic, many long agent contexts at once | a server with 8 RTX PRO 6000 GPUs or HGX B200; for the largest models HGX B200 or B300 | several replicas of a mid-size model, or Kimi K2.7-Code (approx. 1 TB of weights in FP8) on HGX B200 with little KV cache headroom, or on HGX B300 with more |
A few notes on the table. DGX Spark has 128 GB of unified memory, so it can hold even large models, but its memory bandwidth (273 GB/s) is far below that of HBM-based GPUs. It works well for a small team and for a pilot, not for dozens of concurrent users – see NVIDIA DGX Spark as a Service. To get started, you can also use our ready-made GPU packages, which run any open model, not only DeepSeek. With 200 people it is usually better to run several replicas of a mid-size model than one instance of the largest, so that a failure or update of one replica does not stop the whole team. The full hardware ladder from prototype to production is in DGX Spark vs RTX PRO 6000 server vs HGX B200.
Rolling out the assistant: from pilot to the whole team
- Pilot with 5–10 people. Smaller hardware, two or three models to compare, tasks from real projects.
- Measure. The API gateway records concurrent requests, context lengths, response times and how each mode is used. This is the basis for sizing the target hardware.
- Assess quality. Developers rate answers on their own code. Their judgement is worth more than generic leaderboards.
- Choose the model and hardware. Based on measurements, not gut feeling.
- Roll out to everyone. IDE extension configuration, a short guide for the team, usage rules (such as what not to paste), monitoring.
Limitations worth knowing
To be fair: on the hardest tasks, the best commercial cloud models may still outperform the open models you can fit on your own hardware. How big that gap is for your projects is worth testing on your own cases. A self-hosted assistant is also infrastructure to run: model and inference server updates, monitoring, incident response. If you would rather not tie up your own team, we can take over operations entirely. In return you get control over your code, a predictable cost that does not depend on token volume, and the freedom to adapt the model to your own standards.
FAQ: frequently asked questions
Can you run an AI coding assistant locally?
Yes. An open-weight coding model runs on an inference server with an OpenAI-compatible API, and developers connect to it through IDE extensions that support a custom endpoint, such as Continue. The code never leaves your infrastructure.
What is the best open-source model for coding?
As of November 2026, models worth testing include Devstral Small 2 and Devstral 2, Kimi K2.7-Code, the Qwen family and gpt-oss. The best one is the model that handles your code well and fits your hardware – a pilot will tell you.
How many GPUs does a self-hosted AI coding assistant need?
For around 10 developers, a DGX Spark or a single GPU is enough; for 50, a server with 2–4 RTX PRO 6000 GPUs; for 200, an 8-GPU server or HGX B200. Concurrent requests and context length are what matter, so measure them in a pilot.
Does a self-hosted coding assistant work in VS Code and JetBrains IDEs?
Yes, through extensions that let you set a custom OpenAI-compatible endpoint. Continue works in both, Cline in VS Code.
Can a local model provide inline code completion?
Yes, but autocomplete needs a small, fast model that supports FIM and responds quickly. It usually runs alongside a larger model used for chat and agent mode.
Want to give your team an AI assistant without sending code outside? We will choose the model and hardware, run a pilot and take over operations. See our GPU servers or get in touch.