“Large models are already that smart. If we install a small model ourselves, no amount of tuning will catch up, right?”
That's the objection I hear most when I talk about on-premise AI. Honestly, on general benchmark questions, it's mostly right. But owners don't want an AI that's good at exams. They want an AI that does “that one thing at our company” well.
I'm Johnny Yang, founder of Brightstream Technology in Taiwan. I've led my team through system implementation for seven years, across more than 252 projects in over 32 industries. On June 2 this year, I attended the full-day hands-on workshop “Adding New Knowledge to Large Language Models” at NVIDIA GTC Taipei 2026, held at the Taipei International Convention Center, from 9:30 a.m. to 5:30 p.m. The workshop used one complete case to show, from scratch, how to tune an open-source small model into a specialized model. This post breaks down the most valuable numbers from that workshop.
My answer: a small model won't beat a large model at everything, but it can at “that one thing at your company.” In NVIDIA's case, a fine-tuned 8-billion-parameter model, without the full rule set stuffed into the prompt, hit 77% average classification accuracy, beating a 120-billion-parameter model (75%) and Claude Opus 4.6 (73%). The key isn't model size. It's measuring exactly what needs to be done well first, then filling only the missing piece, one step at a time.
The case: an organization that has to route disaster reports
The workshop used a fictional humanitarian organization called Meridian. Hundreds of field reports pour in from disaster zones every day. Someone has to read each one, judge it, and send it to the right team. Right now it's all manual.
They want the AI to read each report and output four judgments:
| Judgment | What it is | What makes it hard |
|---|---|---|
| Category | 8 categories, e.g. water and sanitation, medical, logistics | 10 priority rules; e.g. a contaminated well counts as sanitation, but if people already have diarrhea it's medical |
| Urgency | P1 to P4 | Based on objective evidence, not the reporter's tone; marked “urgent” but nothing is actually broken means the lowest level |
| Assigned team | 15 team codes | Not written as rules; can only be learned from examples |
| Flags | Outbreak, access blocked, funding, partner needed | Mentioning a “road” doesn't mean flagging access blocked; if the road is fine, don't flag it |
This problem looks a lot like what Taiwanese companies do every day: which department gets this customer service ticket, who to call about an order exception, whether an inspection result needs to be reported. The rules can be written down, but people judge a little differently every time.
Step one: write the exam before any training
Many companies start AI implementation by hunting for data and models. Meridian's first step was writing an exam.
They hand-wrote 25 reports, each with a reference answer, and each answer noted which rule it was based on. The exam was finished before any training data was generated. The workshop was frank: the exam took three revisions before it actually measured what it needed to measure.
I like this step a lot because it makes AI investment auditable: the score before, the score after, which rule the gap is in. All of it is clear. When the owner asks “was this money worth it,” there are numbers to answer with.
Scores across four stages
Below are the scores on the same 25-question exam with the same 8-billion-parameter model (Mistral-NeMo-Minitron-8B-Instruct). Style is a separate 20-point rubric that checks whether answers are concise and grounded.
| Stage | Category | Urgency | Assigned team | Flags | Average | Style |
|---|---|---|---|---|---|---|
| Base model, category list only | 84% | 24% | 44% | 48% | 50% | 14.3 |
| All rules written into the prompt | 96% | 64% | 48% | 88% | 74% | 13.7 |
| Fine-tuning (LoRA) | 84% | 68% | 72% | 84% | 77% | 11.3 |
| Fine-tuning + preference training (DPO) | 84% | 76% | 68% | 72% | 75% | 15.5 |
Source: NVIDIA GTC Taipei 2026 workshop “Adding New Knowledge to Large Language Models” course materials.

Each stage fills something the previous stage couldn't:
Writing the rules into the prompt (50% → 74%). Just writing the rules clearly and putting them in raised the average by 24 percentage points. Category, flags, and urgency all improved sharply. But “assigned team” barely moved (44% → 48%), because that part wasn't written as rules and the model had nothing to read. The part that didn't move is the reason to train. If writing the rules in already meets the bar, the workshop's conclusion is: go live, skip training.
Fine-tuning (74% → 77%). The average gained only 3 points, but “assigned team” jumped from 48% to 72%. And the prompt got about 2,000 tokens shorter, so every call is cheaper and faster. The workshop put it this way: fine-tuning makes the model make the same judgment the same way every time. It doesn't teach it new knowledge.
Preference training (style 11.3 → 15.5). After fine-tuning, the model got verbose: 55 to 80 words every time, plus a full list of “why I didn't flag this, why I didn't flag that.” After preference training, it became a grounded 10-to-15-word summary a supervisor can read in one second. Accuracy stayed roughly flat (77% → 75%), but this is the difference between “gets used” and “doesn't get used.”
Small model vs. top cloud models
Before investing in training, the workshop also gave the same exam to two top cloud models:
| Model | Prompt | Average accuracy |
|---|---|---|
| 8B base model (on-premise) | Category list only | 50% |
| Claude Opus 4.6 (cloud) | Category list only | 73% |
| 120B teacher model (cloud) | Category list only | 75% |
| 8B fine-tuned (on-premise) | Category list only | 77% |
| Claude Opus 4.6 (cloud) | Plus all rules | 81% |
| 120B teacher model (cloud) | Plus all rules | 81% |
Read this table honestly:
- Under the same category-list-only condition, the fine-tuned small model won. The rules are already learned into the model, so they don't have to be attached every time.
- With all rules attached, the large models still beat the small model by 4 points. The cost is sending about 2,000 extra tokens per call, and the data has to leave the company.
- The large models are stronger on “flags,” which need nuanced judgment (92% to 96% with rules, vs. 84% for the small model). If that kind of judgment is what you care about most, the conclusion may be different.
The workshop's conclusion was direct: fine-tuning a small model isn't about beating large models on a leaderboard. It's about winning at your own job when three things matter: cost, speed, and data that can't leave the company. For a relief organization, field reports can't be sent outside. That's the deciding reason. The test is simple: first ask whether the data can leave. Details in open-source models or cloud LLMs.
How much hardware and time does it take?
This is what owners care about most. The workshop environment was two NVIDIA H100s:
| Step | Scale | Time |
|---|---|---|
| Generate training data | 2,193 examples (a large model writes report text from the reference answers) | Depends on API speed |
| Fine-tuning (LoRA) | One H100, 5 epochs | About 23 minutes (tutorial version: about 3 minutes per epoch) |
| Preference training (DPO) | 263 comparison pairs, two H100s | About 5 minutes |
| Production serving | vLLM, one GPU | Can switch between fine-tuned versions on the fly |
What actually takes time isn't training. It's the work before it: writing the exam, organizing the rules, generating and checking training data, and trial and error in the wrong direction. For hardware tiers and prices, see what hardware to buy for on-premise AI.

What this means for Taiwanese companies
The thing worth taking from this case is its order, not its numbers:
- Start from an operational decision. Meridian needed “which team handles it first.” What's yours? Who gets this customer service ticket, who to call about an order exception, whether this inspection sheet needs to be reported.
- Write the exam first. 25 questions with reference answers and the rule behind each are more useful than any model benchmark.
- Try writing the rules into the prompt first. Often that's enough. No training needed.
- Train only the part that didn't move. Where the rules are in the prompt and it still fails, that's what fine-tuning is for.
- Measure accuracy and usability separately. A correct answer nobody wants to read still means a system nobody uses.
This is also how we do AI implementation. Most companies solve most of their problem by step three. When fine-tuning is really needed, we use the exam to prove it's worth it before we recommend the investment. For what fine-tuning and knowledge bases are each suited to, see should you fine-tune an AI model.
FAQ
After fine-tuning, will a small model really be better than ChatGPT or Claude?
Possibly at the specific thing your company does. Not at general ability. In NVIDIA's case, the fine-tuned 8B model beat top cloud models when no rules were attached, but with the full rules attached the cloud models still scored 4 points higher. Whether to fine-tune comes down to cost, speed, and whether the data can leave the company.
Do I have to fine-tune? Can't I just write the rules into the prompt?
You should try the prompt first. In this case, just writing the rules into the prompt took average accuracy from 50% to 74%. If that step gets you to the level you need, skip training. Only the parts the prompt can't fix need fine-tuning.
How much data do I need for fine-tuning?
This case used about 2,200 training examples. Research shows that about 1,000 good-quality examples is enough to teach a model a fixed response format (LIMA study). What matters is quality and coverage, not volume. In this case, 60% more data added only 2 points of accuracy.
Does the training data have to be real company data?
Not necessarily. All the training data in this case was generated by a large model from the reference answers. The real field reports never left the organization. This approach suits companies with sensitive data, but the generated data must be spot-checked. If the teacher model is wrong, the student will be wrong too.
Closing: measure first, then fill only the gap
What NVIDIA's case proves isn't “small models beat large models.” It's an auditable method: write the exam first, see how far the prompt gets you, then train only the piece the prompt can't fix. Every step has a score. Every investment can say what it bought.
If your company has a task where “the rules can be written down, but everyone judges it differently,” take a look at how we do AI system development, or bring that task to me and we'll write the exam together first.




