Should you fine-tune an AI model? Most companies only need a knowledge base first

Many owners assume that getting AI to understand their company means training their own model. When knowledge changes and answers need sources, a knowledge base pays off better than fine-tuning. This post covers what each one is good for, how much hardware fine-tuning takes, and one special use: adjusting how a model refuses questions.

Johnny Yang7 min read
Should you fine-tune an AI model? Most companies only need a knowledge base first

"We want AI that actually understands our company. Do we need to train our own model on our company data?"

This is what most owners picture when they start an on-premise AI implementation. It sounds reasonable: your own model, trained on your own data, should know you best. In practice, most companies that take this road spend a lot of money and get less than they expected.

I'm Johnny Yang, founder of Brightstream Technology in Taiwan. I've led my team through seven years of systems implementation, more than 252 projects across over 32 industries, and I've adjusted the behavior of open-source models myself.

My answer: if you want AI to "know" your company, build a knowledge base first. If you want AI to "talk" like your company, then consider fine-tuning. A knowledge base turns company documents into data the AI can look up. Change the data and the change takes effect immediately, and answers can cite their source. Fine-tuning changes the model's own behavior. It suits tone, format, and industry terminology. It does not suit knowledge that keeps changing.

What's the difference between a knowledge base and fine-tuning?

Here's an analogy. A knowledge base is like handing a new hire an organized manual. Before every answer, he looks it up, so the answer has a basis, and when the manual is updated he knows the new version. Fine-tuning is like sending him to three months of training. What changes is how he talks and how he works.

ComparisonKnowledge base (RAG)Fine-tuning
What it changesGives the AI data it can look upChanges the model's own behavior
Updating dataEdit the data and it takes effectRequires retraining
Can it cite sourcesYes, answers include the source passageHard
CostLower, mostly organizing dataHigher, needs training data and compute
Good forProduct info, SOPs, rules, quotation recordsTone, format, terminology, refusal behavior

According to Largitdata's recommendation, when knowledge changes or needs a citation, build the knowledge base first and consider fine-tuning later. Our experience matches: for most companies, the knowledge base alone solves 80% of the problem.

Knowledge base vs fine-tuning: what changes, how updates work, whether sources can be cited, cost, and best uses

Why most companies shouldn't fine-tune first

First, an admission: fine-tuning really does make a model feel more like "yours." That instinct isn't wrong.

The problem is that what companies most want AI to know usually changes: product prices, specs, processes, regulations. If you use fine-tuning to "memorize" these into the model, every change means retraining. And what the model memorizes can't cite a source, so your staff can't tell whether it's right.

The other problem is data. Fine-tuning needs a large set of organized "question plus correct answer" pairs, and most companies haven't even finished writing their SOPs. Fine-tuning in that situation is building the second floor before the foundation is poured. I wrote about the data cleanup problem in When your Excel data is too messy for AI to fix.

When do you actually need fine-tuning?

Fine-tuning isn't useless. It just has to be used in the right place. These situations are worth considering:

  1. Tone and format: You want AI answers in a fixed format, such as fixed sections in an inspection report or a fixed tone in customer service replies.
  2. Industry terminology: Your industry has a lot of specialized terms and the model keeps getting them wrong.
  3. Fixed workflows: A task follows the same steps every time, such as sorting customer emails into ten categories. Fine-tuning makes it more consistent.
  4. Refusal behavior: The model over-refuses certain legitimate professional questions and needs adjusting.

The first three can be handled with standard fine-tuning. The fourth is a special case, covered separately below.

What fine-tuning methods are there, and how much hardware do they need?

There are many fine-tuning methods. Owners don't need the details, just a few terms:

MethodWhat it isHardware needs
LoRA/QLoRATrains only a small set of extra parameters and leaves the original model untouched; QLoRA also compresses the original model to save memoryLowest. NVIDIA says DGX Spark can fine-tune models up to 70 billion parameters
Full-parameter fine-tuningRetrains every parameter in the modelMost expensive. Above 7 billion parameters you usually need data-center-grade GPUs
SFTTrains on "question + correct answer" pairsUsed with LoRA
DPOTrains on "good answer, bad answer" pairs to teach the model which one to preferUsed with LoRA

According to Hugging Face's PEFT documentation, methods like LoRA fine-tune only a small number of extra parameters and are what companies use most. How much memory does full-parameter fine-tuning take? The same documentation gives an example: a 1-billion-parameter model needs 1.86 GiB for the weights alone, plus about 5.59 GB more for gradients and optimizer states during training (source). The bigger the model, the more the requirements multiply.

As for how much training data to prepare, none of the sources I checked gave a reliable number. It depends on the task. Our approach is to start with a few hundred high-quality examples, look at the results, then decide whether to add more.

Decision steps for choosing between a knowledge base and fine-tuning in an AI implementation

Special use: adjusting a model's refusal behavior

This is a very useful capability of open-source models that few people talk about.

Mainstream cloud models have strict usage policies on certain topics. For example, Anthropic's usage policy explicitly prohibits sexual content (source). Most of these rules are reasonable, but sometimes they cause the model to refuse legitimate medical questions too. One case we ran into was a medical aesthetics clinic: when clients asked about intimate-area treatments or body parts, the model often refused outright or gave vague answers. For the clinic's customer service, that was a real problem.

Open-source models let you adjust this kind of over-refusal. One method used in the community works by finding the direction inside the model responsible for "refusing to answer" and removing it (source).

But this has costs, and they need to be stated plainly:

  • Quality drops: The same article notes that removing refusals also lowers model quality, which then has to be recovered with DPO.
  • The responsibility is yours: If the model no longer refuses, the job of gatekeeping falls back on the company using it. You need to add your own rules and logs.
  • Check the license: Each model's license handles removing safety mechanisms differently. Read the original text for each one.
  • Check the law: Adult content and anything involving minors carry legal liability in Taiwan. Have a lawyer review before going live. We never do content involving minors.

So our approach is to adjust the refusal scope only for legitimate professional needs, such as letting a medical aesthetics AI answer medical questions normally, while adding our own rules and logs. We do not make the model answer everything.

FAQ

To make AI understand our company, do we have to train our own model?

No. In most cases a knowledge base is enough: organize your SOPs, rules, and product data into something the AI can look up. Changes take effect immediately, answers can cite sources, and it's cheaper and easier to maintain than fine-tuning.

How much does one round of fine-tuning cost?

It mainly depends on model size and method. Fine-tuning a mid-sized model with LoRA can be done on a small AI box or a workstation with a professional GPU. Full-parameter fine-tuning of a large model needs data-center-grade GPUs. The biggest cost usually isn't hardware. It's organizing the training data.

Can a fine-tuned model still be updated?

Yes, but it has to be retrained. That's why knowledge that changes doesn't belong in fine-tuning. It belongs in the knowledge base. Fine-tuning is for stable behavior, such as tone, format, and terminology.

Can an open-source model be changed to stop refusing?

Technically yes, but it lowers model quality and puts the gatekeeping responsibility back on you. We only recommend adjusting for legitimate professional needs, such as medical questions, with rules and logs added. For adult content or anything involving minors, confirm the legal liability first.

Conclusion: teach it to look things up first, then change how it talks

The most effective first step to making AI understand your company is building a knowledge base, not training a model. Once the knowledge base is stable, if the tone, format, terminology, or refusal behavior is still off, then use fine-tuning.

If you want AI that can actually use your company's data, see how we handle AI system development, or read further: How to choose an on-premise AI model and Why teaching employees to write prompts doesn't work.

Found this useful? Pass it on
30 minutes · free

Does this sound like your company?

Spend 30 minutes telling Johnny Yang how you do it now, and you get told where hours are leaking, where the risk is buried, and whether this is worth paying to fix. No pitch, no course to sell — if it is not a fit, that is fine too.

Pick a slot online · with Johnny Yang himself · 252+ projects behind it

Johnny Yang (熊董)
Johnny Yang (熊董)
Founder, Brightstream Technology

Serial founder leading a small team that builds systems into an owner's everyday work.

As a founder
7 yrs
Projects delivered
252+
Industries served
32+
About Johnny