Open-Weight LLMs: Making the Right Choice
AICloud models are convenient. They are also expensive, dependent on US providers, and often bound to rate limits. The debate has intensified in recent weeks. Frontier models with open weights now reach a quality that until recently was reserved for closed cloud APIs.
On Oxide and Friends, Simon Willison speaks of an “Open Weight Revolution.” He asks the right questions: who is for it, who is against it — and do open models make our future safer or more dangerous?
So over the past few months I have tested open-weight models on my own hardware. The goal was not to replace the cloud entirely, but to find out when the local model is the better choice.
The central insight: not every model fits every piece of hardware. And quantization is a deliberate trade-off between quality and resource consumption.
Two Axes That Often Get Confused
Before getting into practice, a clarification of terms is worthwhile. Open-Weight and Frontier describe the accessibility of a model. Cloud and Local describe where it runs. The two axes are independent, but they get mixed up constantly in discussions.
Open-weight models make their weights available for download. Whoever has them can run them anywhere: on their own hardware, on rented cloud GPUs, in an on-premise data center. Frontier models are the most capable class at any given time — so far mostly closed and reachable only via API.
| Axis | Frontier | Open-Weight |
|---|---|---|
| Definition | Best class, current state of the art | Weights publicly downloadable |
| Access | usually API-only, closed weights | anywhere, local or in the cloud |
| Quality | top tier | often lags frontier by 6-12 months |
| Control | provider decides on version, limits, data | operator decides on everything |
Then comes the second axis. Using a model in the cloud does not mean it is a cloud model in the proprietary sense. Many providers offer open-weight models as a service. Conversely, you can run the same open-weight model on-premise and then have local control without any cloud connection.
| Axis | Cloud | Local / On-Premise |
|---|---|---|
| Infrastructure | rented, at the provider | your own hardware |
| Data | leaves your own network | stays in your own network |
| Latency | network round-trip | local, predictable |
| Cost | usage-based, ongoing | one-time hardware, power, maintenance |
| Dependency | provider, rate limits, roadmap | your own skills, your own drivers |
The combination that interests me: open-weight models on my own hardware. That combines the freedom of open weights with the control of local infrastructure. This is exactly the combination I tested.
What I Tested
On an NVIDIA GB10 Grace Blackwell with 128 GB of RAM, I have been running various models in production since April 2026.
The workhorse is Qwen 3.6 35B-A3B with FP8 quantization, 256K context, on vLLM.
It is stable in 24/7 use as an agent backend. For coding tasks I use Qwen 3 Coder Next, also FP8, with 131K context. Gemma 4 26B-A4B with 262K context runs for instruction-following tasks.
Open Weights: Revolution or Risk?
The current debate around open weights revolves around two camps.
Proponents argue: transparency, independent audits, less dependency on a handful of cloud providers.
Critics warn: whoever has the weights can run the model without filters, without oversight, and without any way to recall it.
Both sides have arguments. On the podcast Oxide and Friends: The Open Weight Revolution with Simon Willison, Willison posed the right counter-question: do closed models actually make the world safer?
The Hugging Face security incident of July 2026 showed that the evaluation and distribution infrastructure of the large providers is vulnerable too.
Security through obscurity is a weak concept when the infrastructure itself has holes.
My position is pragmatic: open weights shift control from the provider to the operator.
Whoever self-hosts decides on quantization, prompt filters, network isolation, and auditing. For companies that do not want to hand their data to foreign clouds, that is a concrete advantage.
At the same time it is clear: not every open-weight model is the same. Some come with restrictive licenses, some with unclear training data, some are unsuitable for commercial use.
Choosing a fitting model is also a license and compliance decision.
What Works, What Surprises
After roughly two and a half months of production use since April 2026, a few insights have solidified. FP8 is my new standard: 50 percent memory reduction at under five percent quality loss.
Qwen 3.6 35B-A3B runs around the clock without outages. For agentic use cases, context length mattered more than parameter count. In my tests, 256K context beats a larger model when it comes to multi-stage tasks — it is far more responsive, with no noticeable quality loss. In exceptional cases, problems came up when tasks ran long (> 4h).
Two things surprised me.
First: driver support weighs more than hardware specs. On consumer GPUs, a missing Vulkan driver got me only 8.7 tokens per second instead of the 30-plus the hardware could have delivered.
Second: without observability you find no bottlenecks. Only the 379 vLLM metrics I collect through Langfuse made performance bottlenecks visible that I had not seen before.
Migration as a Three-Stage Process
For decision-makers, I recommend an evaluation in three phases.
- First, measure your current cloud provider as a baseline (e.g. with lightLLM as a MITM proxy) and set up comparable hardware.
- Then comes pilot testing with real workloads. In phase two, identical prompts run in parallel on the cloud model and the local model, to compare quality, latency, and cost.
- Phase three is the go-live decision, which needs an ROI calculation, a fallback strategy, and possibly a hybrid approach.
I personally rely on a hybrid stack today: standard tasks run locally, edge cases and tests against new models still go to the cloud. A deliberate weighting by risk and cost.
What Remains
- Open-weight LLMs pay off if you deploy them correctly.
- They give you control over data and costs, decouple you from provider roadmaps, and enable deep observability. But they cost time, know-how, and hardware.
- INT4 does not automatically work well, drivers can throw a wrench in the works, and larger models are often economically pointless.
The current open-weight revolution changes the starting position. Frontier models with open weights are no longer experiments. Whoever wants to migrate should start with a realistic test setup. The right (sufficient) model is the starting point, not the biggest one.
How does it look on your end: do you run open-weight models in production, or do you stick with the cloud? I am curious about your experiences with hardware, quantization, and cost — feel free to share them in the comments.
Sources
- Oxide and Friends: The Open Weight Revolution with Simon Willison — discussion on open weights, security, and the Hugging Face security incident
- Hugging Face Blog: Security Incident July 2026
- OpenAI: Hugging Face model evaluation security incident
Leave a Reply