"We want AI that actually understands our company. Do we need to train our own model on our company data?"
This is what most owners picture when they start an on-premise AI implementation. It sounds reasonable: your own model, trained on your own data, should know you best. In practice, most companies that take this road spend a lot of money and get less than they expected.
I'm Johnny Yang, founder of Brightstream Technology in Taiwan. I've led my team through seven years of systems implementation, more than 252 projects across over 32 industries, and I've adjusted the behavior of open-source models myself.
My answer: if you want AI to "know" your company, build a knowledge base first. If you want AI to "talk" like your company, then consider fine-tuning. A knowledge base turns company documents into data the AI can look up. Change the data and the change takes effect immediately, and answers can cite their source. Fine-tuning changes the model's own behavior. It suits tone, format, and industry terminology. It does not suit knowledge that keeps changing.
What's the difference between a knowledge base and fine-tuning?
Here's an analogy. A knowledge base is like handing a new hire an organized manual. Before every answer, he looks it up, so the answer has a basis, and when the manual is updated he knows the new version. Fine-tuning is like sending him to three months of training. What changes is how he talks and how he works.
| Comparison | Knowledge base (RAG) | Fine-tuning |
|---|---|---|
| What it changes | Gives the AI data it can look up | Changes the model's own behavior |
| Updating data | Edit the data and it takes effect | Requires retraining |
| Can it cite sources | Yes, answers include the source passage | Hard |
| Cost | Lower, mostly organizing data | Higher, needs training data and compute |
| Good for | Product info, SOPs, rules, quotation records | Tone, format, terminology, refusal behavior |
According to Largitdata's recommendation, when knowledge changes or needs a citation, build the knowledge base first and consider fine-tuning later. Our experience matches: for most companies, the knowledge base alone solves 80% of the problem.

Why most companies shouldn't fine-tune first
First, an admission: fine-tuning really does make a model feel more like "yours." That instinct isn't wrong.
The problem is that what companies most want AI to know usually changes: product prices, specs, processes, regulations. If you use fine-tuning to "memorize" these into the model, every change means retraining. And what the model memorizes can't cite a source, so your staff can't tell whether it's right.
The other problem is data. Fine-tuning needs a large set of organized "question plus correct answer" pairs, and most companies haven't even finished writing their SOPs. Fine-tuning in that situation is building the second floor before the foundation is poured. I wrote about the data cleanup problem in When your Excel data is too messy for AI to fix.
When do you actually need fine-tuning?
Fine-tuning isn't useless. It just has to be used in the right place. These situations are worth considering:
- Tone and format: You want AI answers in a fixed format, such as fixed sections in an inspection report or a fixed tone in customer service replies.
- Industry terminology: Your industry has a lot of specialized terms and the model keeps getting them wrong.
- Fixed workflows: A task follows the same steps every time, such as sorting customer emails into ten categories. Fine-tuning makes it more consistent.
- Refusal behavior: The model over-refuses certain legitimate professional questions and needs adjusting.
The first three can be handled with standard fine-tuning. The fourth is a special case, covered separately below.
What fine-tuning methods are there, and how much hardware do they need?
There are many fine-tuning methods. Owners don't need the details, just a few terms:
| Method | What it is | Hardware needs |
|---|---|---|
| LoRA/QLoRA | Trains only a small set of extra parameters and leaves the original model untouched; QLoRA also compresses the original model to save memory | Lowest. NVIDIA says DGX Spark can fine-tune models up to 70 billion parameters |
| Full-parameter fine-tuning | Retrains every parameter in the model | Most expensive. Above 7 billion parameters you usually need data-center-grade GPUs |
| SFT | Trains on "question + correct answer" pairs | Used with LoRA |
| DPO | Trains on "good answer, bad answer" pairs to teach the model which one to prefer | Used with LoRA |
According to Hugging Face's PEFT documentation, methods like LoRA fine-tune only a small number of extra parameters and are what companies use most. How much memory does full-parameter fine-tuning take? The same documentation gives an example: a 1-billion-parameter model needs 1.86 GiB for the weights alone, plus about 5.59 GB more for gradients and optimizer states during training (source). The bigger the model, the more the requirements multiply.
As for how much training data to prepare, none of the sources I checked gave a reliable number. It depends on the task. Our approach is to start with a few hundred high-quality examples, look at the results, then decide whether to add more.

Special use: adjusting a model's refusal behavior
This is a very useful capability of open-source models that few people talk about.
Mainstream cloud models have strict usage policies on certain topics. For example, Anthropic's usage policy explicitly prohibits sexual content (source). Most of these rules are reasonable, but sometimes they cause the model to refuse legitimate medical questions too. One case we ran into was a medical aesthetics clinic: when clients asked about intimate-area treatments or body parts, the model often refused outright or gave vague answers. For the clinic's customer service, that was a real problem.
Open-source models let you adjust this kind of over-refusal. One method used in the community works by finding the direction inside the model responsible for "refusing to answer" and removing it (source).
But this has costs, and they need to be stated plainly:
- Quality drops: The same article notes that removing refusals also lowers model quality, which then has to be recovered with DPO.
- The responsibility is yours: If the model no longer refuses, the job of gatekeeping falls back on the company using it. You need to add your own rules and logs.
- Check the license: Each model's license handles removing safety mechanisms differently. Read the original text for each one.
- Check the law: Adult content and anything involving minors carry legal liability in Taiwan. Have a lawyer review before going live. We never do content involving minors.
So our approach is to adjust the refusal scope only for legitimate professional needs, such as letting a medical aesthetics AI answer medical questions normally, while adding our own rules and logs. We do not make the model answer everything.
FAQ
To make AI understand our company, do we have to train our own model?
No. In most cases a knowledge base is enough: organize your SOPs, rules, and product data into something the AI can look up. Changes take effect immediately, answers can cite sources, and it's cheaper and easier to maintain than fine-tuning.
How much does one round of fine-tuning cost?
It mainly depends on model size and method. Fine-tuning a mid-sized model with LoRA can be done on a small AI box or a workstation with a professional GPU. Full-parameter fine-tuning of a large model needs data-center-grade GPUs. The biggest cost usually isn't hardware. It's organizing the training data.
Can a fine-tuned model still be updated?
Yes, but it has to be retrained. That's why knowledge that changes doesn't belong in fine-tuning. It belongs in the knowledge base. Fine-tuning is for stable behavior, such as tone, format, and terminology.
Can an open-source model be changed to stop refusing?
Technically yes, but it lowers model quality and puts the gatekeeping responsibility back on you. We only recommend adjusting for legitimate professional needs, such as medical questions, with rules and logs added. For adult content or anything involving minors, confirm the legal liability first.
Conclusion: teach it to look things up first, then change how it talks
The most effective first step to making AI understand your company is building a knowledge base, not training a model. Once the knowledge base is stable, if the tone, format, terminology, or refusal behavior is still off, then use fine-tuning.
If you want AI that can actually use your company's data, see how we handle AI system development, or read further: How to choose an on-premise AI model and Why teaching employees to write prompts doesn't work.




