“Johnny, there are so many open-source models now. Qwen, DeepSeek, Llama, Gemma. Which one is the best? We want to install it inside the company.”
This is one of the questions business owners asked me most this year. The person asking has usually already read a few leaderboard articles. Each one recommends something different. The more they read, the more confused they get.
I'm Johnny Yang, founder of Brightstream Technology in Taiwan. My team and I have done systems implementation for seven years, across more than 252 projects in over 32 industries. We have built an AI customer service system that runs offline, and we built an on-premise model for a consulting firm in their own server room.
My answer: there is no “best” model. There is only a model you are allowed to use, that fits on your hardware, and that passes your test. The order of selection is: use your constraints to eliminate what you can't use, test the rest on your company's own questions, and only then look at speed and price. Leaderboards are a reference. They are not the answer.
Why can't you just pick from the leaderboard?
To be fair, checking a leaderboard is a reasonable instinct. You look at reviews when you buy a car, too. And leaderboards are useful. They tell you which models deserve a place on the candidate list.
The problem is that leaderboards don't test your questions. They test math, coding and English general knowledge. Your staff will ask things like “What discount did we give this customer on the last quotation?”, “Which item on this inspection report is over the limit?” and “How does this contract differ from our template?” Those questions involve Traditional Chinese, Taiwanese phrasing and your company's own terminology. The leaderboard tests none of them.
There is a more practical reason, too. Many top-ranked models you simply cannot use. The country of origin is a problem, the license doesn't allow it, or your hardware can't hold it. So the correct order is reversed. Eliminate first, then test.
Before choosing a model, confirm you really need on-premise
Before the four questions, there is an earlier one: can your data go to the cloud?
If it can, in most cases I'll recommend using a cloud AI service directly. The top cloud models are usually smarter. You don't buy machines, you don't hire anyone to maintain them, and you pay for what you use. For a company whose data isn't sensitive and whose usage is low, on-premise is money wasted.
On-premise is worth doing when data cannot leave the company: medical records at clinics and hospitals, client sample data at testing labs, formulas and process parameters at manufacturers, client files held by accountants and lawyers, or cases where a contract, an audit or a tender explicitly requires that data stay in-house. In those cases it doesn't matter how smart the cloud is. You can't use it. On-premise is the only way to use AI at all.
Many companies run a mix: sensitive data goes to the on-premise model, routine paperwork goes to the cloud. If you go this route, there is one more thing to consider during selection. The answer style and format on both sides should be as consistent as possible, so staff don't feel like they're using two different systems.
Once you've confirmed you really need on-premise, move on to the four questions below.
Question 1: Are there restrictions on the model's country of origin?
Ask this one first. If the answer is “not allowed,” nothing else matters.
If your company bids on government tenders, supplies the public sector, or sits in a major customer's supply chain that gets audited, first confirm whether you can use models from Chinese vendors. Among today's mainstream large open-source models, more than half come from Chinese vendors, including Alibaba's Qwen, DeepSeek, Moonshot AI's Kimi, Zhipu's GLM, MiniMax and Xiaomi's MiMo.
According to Largitdata's summary, Taiwanese government agencies may not use DeepSeek's services, and that includes models downloaded to run locally. This is the policy as reported in that article. Check the competent authority's official announcements for the actual scope.
If you can't use Chinese models, start your candidate list with these: OpenAI's gpt-oss, Google's Gemma, IBM's Granite, France's Mistral, and Taiwan's own TAIDE.
Question 2: Does the license allow commercial use?
Open source does not mean you can use it however you like. Every model, and even different versions within the same family, can carry a different license.
| License type | Example models | What it means for a business |
|---|---|---|
| Apache 2.0, MIT | gpt-oss, Gemma 4, Qwen3.8-27B, DeepSeek, Phi, Granite | Least burden for commercial use. Choose these first |
| Revenue or user thresholds | Llama (separate agreement above 700 million monthly active users), MiniMax (revenue threshold), some large Qwen versions | SMEs usually won't hit the threshold, but keep a record |
| “Built with” attribution required | MiniMax, Meta Muse Spark Open | You must display attribution on the product. Watch this for external-facing services |
| Non-commercial | Cohere Command A | Internal research only. You can't use it to do business |
| No commercial use above a revenue cap | Liquid AI's LFM | Becomes a blocker once your company grows |
The above is compiled from ComputingForGeeks' model comparison table. Licenses get revised. Before going live, read the original model card for that exact version, and write the license into your delivery documents so you can produce it when an auditor or customer asks.
Question 3: Can your hardware hold it?
This question decides how much you spend on machines. There is a simple formula for roughly estimating how much memory the model itself needs:
Parameter count × bits per parameter ÷ 8 = GB required
Hugging Face's documentation gives an example: an 8B (8 billion parameter) model takes about 16 GB stored at 16 bits, and about 4 to 5 GB compressed to 4 bits. This “compression” is called quantization. You trade a little accuracy for a lot of memory.
But that only covers the model itself. When many people use it at once, or the documents are long, you need extra headroom. So the right approach here is: estimate first, then stress-test with real usage. Don't place the order the moment you finish the math.
There's another common misunderstanding. Some models are labeled “1 trillion parameters total, only 30 billion used per pass.” This is called an MoE (mixture of experts) architecture. It runs faster, but memory has to be calculated on the total parameter count, not on the part used per pass.
How to pick hardware and what it costs, I'll lay out in a full table in the next post in this series, “What hardware to buy for on-premise AI.”
Question 4: Has it passed a test with your own questions?
After the first three questions, you usually have three to five candidates left. Only now does the testing start, and the test uses your own questions.
The method is simple:
- Write the questions: Pick 30 to 50 questions your staff would actually ask. Include Traditional Chinese, Taiwanese phrasing, and your company's terminology and product names.
- Prepare reference answers: Ask the colleague who knows best to write what they consider the correct answer, or at least list the key points the answer must include.
- Compare under identical conditions: Run 2 to 3 candidate models on the same questions, the same knowledge base and the same settings.
- Find a ceiling: Run the strongest cloud model once using dummy data, so you know the best result that's possible.
- Check three things: How many it answered correctly, whether it made things up, and whether Simplified Chinese characters or mainland Chinese phrasing showed up.
The last point matters. According to NSS's deployment notes, some models output Simplified Chinese characters or mainland phrasing. For Taiwanese customers and staff, that's not a minor flaw. It makes people feel the system “wasn't built for us.”

Seven scenarios: which models to try first?
Before you've finished eliminating and testing, you still need a starting point. Below is our shortlist for common scenarios. Note that it's a list to try first, not a conclusion.
| Scenario | Models to try first |
|---|---|
| Owner or a small team of 1 to 5, starting with document Q&A | gpt-oss-20b, Gemma 4 12B, TAIDE 12B |
| 20 to 80 people, multiple departments using a company knowledge base | Qwen3.8-27B, Gemma 4 31B |
| Medical or testing work that needs to read images (report photos, forms) | Qwen3.8-27B, MiniCPM-V, Muse Glimmer 30B |
| Government, public sector, major-customer supply chains that can't use Chinese models | gpt-oss-120b, Gemma 4 31B, TAIDE 12B |
| Need near-cloud strength (reasoning, coding) with a small team | gpt-oss-120b, DeepSeek V4-Flash |
| Traditional Chinese official documents, regulations, internal documents | TAIDE 12B, Qwen3.8-27B, Gemma 4 31B |
| Hundreds of concurrent users, multiple uses | Any of the above, paired with data-center-grade servers |
Model names and sizes are compiled from Sysart and other sources. Every vendor ships a new version every few months, so go by the current version when you choose.

Three mistakes business owners make most often
We run into these three often during assessments. Each comes from perfectly reasonable thinking. The direction is just off.
First: picking the biggest model, then nobody can afford to use it. An owner sees that a model has the top score and wants to deploy it straight away. But the biggest models often need several data-center-grade GPUs. The hardware alone starts at several million NT dollars, and responses are slower. If staff wait more than ten seconds per question, they stop using it after two days. For most companies' daily work, looking things up, summarizing and drafting, a mid-sized model is enough.
Second: looking only at English benchmarks, then finding out after launch that the Chinese is bad. Many benchmarks use English questions. Switch to Traditional Chinese with Taiwanese phrasing, and the same model can perform very differently. We've seen it in our own testing: beautiful answers to English questions, and a pile of Simplified Chinese characters in the Chinese answers. You must find this out before you buy hardware.
Third: not checking the license, then discovering halfway through that commercial use isn't allowed. Some models are easy to download and look free. But the license says non-commercial, or requires a separate agreement once revenue passes a certain amount. A small company doesn't hit the threshold at first. Then business picks up, customers start auditing, and you discover the whole system is built on a model you can't use. Switching at that point is very expensive.
What these three have in common: they all skipped the elimination steps and started from “which one is strongest.”
What should the selection result be written up as?
Model selection isn't something you just agree on verbally. Every time we finish one, we hand the client a simple record containing:
| Item | What to write |
|---|---|
| Candidate models and versions | Name, version number, download source |
| License | License name, whether commercial use is allowed, any thresholds or attribution requirements |
| Country of origin | Developer's country, and whether it meets customer or tender requirements |
| Hardware requirements | Quantization method, estimated memory, tested number of concurrent users |
| Test results | Number of questions, accuracy rate, number of fabrications, Simplified Chinese or phrasing issues |
| Reason for selection | Why this one and not the others |
This record serves two purposes. One, when the model needs a version upgrade, you have the same question set and criteria to compare against. Two, when the client gets audited, or a major customer asks which AI model they use, they can produce it. Many owners think this is unnecessary paperwork. The day someone actually asks, they find out what it's worth.
After choosing the model, what actually decides success?
Many owners think choosing the right model gets them halfway there. In my experience, the model is about 30% of it.
The other 70% is two things. The first is the knowledge base: how well your SOPs, specifications and quotation records are organized decides whether the AI can find things and answer correctly. I've covered this in Is it safe to give company data to AI and When your Excel data is too messy even for AI. The second is wiring it into the workflow: the AI has to live inside the systems and LINE your staff use every day, not on a separate web page everyone has to remember to visit.
So the goal of model selection isn't to find the strongest model in the world. It's to find one that's good enough, legal, stable and runs on your machines, and then put your effort into making sure it actually gets used.
Frequently asked questions
Does on-premise AI need the biggest model to be useful?
No. SMEs most commonly use models of around 20 to 30 billion parameters, which run on a single workstation. Bigger models are more expensive and slower. A mid-sized model plus a well-organized knowledge base is usually better value than a big model on its own.
Can we use Qwen and DeepSeek?
Technically yes, and most of their licenses are permissive. But if you bid on government tenders, supply the public sector, or have major customers who audit you, first confirm whether models from Chinese vendors are allowed. If you're unsure, gpt-oss, Gemma or TAIDE are the safer choice.
Is the Traditional Chinese ability of open-source models good enough?
Most mainstream models can read and write Traditional Chinese, but some mix in Simplified characters or mainland phrasing. You must test with your own Traditional Chinese questions. For official documents and regulatory work, try Taiwan's TAIDE first.
Once we've chosen a model, how often should we switch?
Open-source models get a new version roughly every three to six months. You don't need to switch every time. But during maintenance, you should regularly run the same question set against new versions and only switch if the new one is better. That's how you avoid falling further behind the longer you use it.
Conclusion: eliminate first, test second, compare speed last
The correct order for choosing an on-premise AI model is: country of origin, license, hardware, real-world testing. The first three questions eliminate what you can't use. Only the fourth tells you which one fits you best.
If you're evaluating on-premise AI and aren't sure which scenario to start from, take a look at how we do AI system development, or bring the five questions your team gets asked most and come talk to me.




