A company knowledge assistant is a chat tool that answers employees’ questions from your own procedures, policies, manuals and documentation, and shows the source behind every answer. The technical name is RAG (Retrieval-Augmented Generation). When the model runs on your own server, your documents never reach a public AI cloud.
This article looks at it from the business side: what problem it solves, how it works in plain terms, how it respects permissions, what hardware it needs and where deployments usually go wrong.
The problem: knowledge scattered across drives, inboxes and heads
In most organisations, knowledge lives in several places at once: a network drive, a wiki, the ticketing system, email threads and the heads of a few experienced people. The effects show up every day:
- new hires spend their first weeks asking colleagues about things that are documented somewhere,
- experts lose time answering the same questions again and again,
- people follow an outdated version of a procedure because it was the first one they found,
- file search matches words, not the meaning of the question.
A company knowledge assistant does not replace well-maintained documentation, but it makes that documentation fast to use: an employee asks in plain language and gets a specific answer with a link to the source.
How a company knowledge assistant (RAG) works, in plain terms
The language model does not memorise your documents. Instead, for every question it is handed the few most relevant passages to read. The process looks like this:
- Preparation: documents are split into chunks, and each chunk is turned into a vector, a numerical representation of its meaning. The vectors are stored in a vector database.
- Retrieval: the employee’s question is also turned into a vector, and the system finds the chunks closest in meaning.
- Answer: the language model receives the question and the retrieved chunks with an instruction to “answer only from these materials and cite your sources”.
That means no training on company data, and when a document changes you simply re-index it. For the technical side (choosing a vector database, chunking strategy, search optimisation), see our article on vector database optimisation for enterprise RAG.
Permissions: the assistant must not know more than the person asking
This is the most important element and the one most often skipped in prototypes. If board contracts, HR records and warehouse procedures all go into the index and the assistant does not check permissions, any employee can ask about them. A sound deployment:
- stores, with each chunk, who may access it, carried over from the source system (file share, wiki, document management system),
- filters search results by the logged-in user’s permissions before anything reaches the model,
- logs questions and cited sources so the system can be audited,
- re-syncs permissions regularly, so revoking access also takes effect in the assistant.
The second aspect is where processing happens. With public AI services, document fragments leave your organisation with every question. A model on your own server in an EU data centre removes that problem. We cover this in more depth in how to deploy local LLMs and avoid data leaks.
What hardware you need
A RAG assistant does not need the largest models, because the answer is grounded in the supplied passages rather than in the model’s own knowledge. Models in the 20–120B range work well. As a rough guide:
| Scale | Example models | Hardware |
|---|---|---|
| Pilot in one department | Qwen3.8-27B, gpt-oss-20b, Gemma 4 31B (4-bit) | 1 GPU with 24–48 GB, e.g. our A10, A30 or A40 packages |
| Prototype with a larger model, small team | gpt-oss-120b, Mistral Small 4 (4-bit, ~67–68 GB) | DGX Spark (128 GB) or 1× RTX PRO 6000 96 GB |
| Company-wide, many concurrent users | as above, with memory headroom for context | a server with 2 GPUs, e.g. 2× RTX PRO 6000, running vLLM |
The embedding model and the vector database are small compared with the language model and usually run on the same server. DGX Spark is a good fit for a prototype or a small team, but with many concurrent users its memory bandwidth becomes the bottleneck; more in NVIDIA DGX Spark as a Service. To size memory for a specific model and user count, see How much VRAM does an LLM need, and to pick a model, our comparison of open-source LLMs.
Rolling it out step by step
- Pick one area. For example HR procedures, technical product documentation or the customer service knowledge base. Fewer documents, but well maintained.
- Collect 30–50 real questions from employees, together with the correct answers. This is your test set.
- Clean up the sources. Remove duplicates and old versions, and assign an owner to every document.
- Run a pilot on a single GPU or a DGX Spark, with permission mapping from day one.
- Measure quality against the test set and collect user feedback (a simple “helpful / not helpful” button).
- Expand to other departments only when the pilot answers accurately and people actually use it.
Pitfalls: stale documents and hallucinations
Stale documents
The assistant is only as good as the documents it is given. If the index holds three versions of a procedure, the answer may be based on the oldest. The fix is organisational, not technical: every document has an owner and a review date, indexing runs automatically after every change, and archived versions are excluded from search.
Hallucinations
A model can give a convincing answer that contradicts the document, or “fill in” something the sources do not say. Three practices keep this in check: citing sources with every answer (linking to the document and passage), instructing the model to say “I don’t know” when the retrieved passages do not contain the answer, and regular testing against your question set. Citations have a second benefit: employees can verify an answer in seconds, and trust in the tool grows over time.
FAQ: frequently asked questions
What is RAG in AI?
RAG (Retrieval-Augmented Generation) is an approach where the language model first searches your documents for relevant passages and then answers based on them. It avoids having to train the model on company data.
Can an AI assistant use company documents without sending them to the cloud?
Yes. The language model, vector database and documents can all run on your own server or on a GPU server in an EU data centre, with no connection to public AI services.
How do you control access to confidential documents in an AI assistant?
Carry permissions over from the source system into the vector database and filter results by the logged-in user before they reach the model. Add query logging and regular permission syncs.
How much does a company knowledge assistant cost?
It depends on the number of users and documents. A single-department pilot fits on one GPU from our ready-made packages; larger setups and DGX Spark are quoted individually.
Do I need to train an AI model on my company data?
Usually not. With RAG, the model reads the retrieved passages for every question. Fine-tuning only makes sense for a very specific answer style or highly specialised vocabulary.
Want to pilot a knowledge assistant on your own documents? We will choose the model and hardware, from a single GPU to DGX Spark and GPU servers in our Tier III data centre in Poland, and run it for you 24/7. Get in touch.