After the open source model is made public, the first thing that becomes fragile is the version lock.

After an open source model is made public, the fixed version is usually the first to be loosened, while the weight file is often the most stable.

The warehouse name is still there, and the model name is still there, but the actual input used to run evaluations, do regressions, and connect traffic is often not the same thing. What we pulled today is main, and it will still be the same path next week, but the tokenizer, template, quantization package, default dtype, and even the recommended parameters in the README have been changed. What you see online is not “the model disappears”, but “the same model name becomes another deliverable”.

After taking over the model access several times, you will find that the most easily ignored value is the default value. The weight file is usually watched by someone, but the default tag, default image, default template, and default cache directory are often left unattended. When a synchronization window is stuck, or the mirror station only synchronizes the weights but not the configuration, the team will suddenly find that what they are holding is not a reproducible version, but a string of drifting names.

The default entrance is easier to drift than the weight.

The weight file is static, the entry is not.

model-name:latest This way of writing seems easy, but the problem is that it leaves the matter of “when to update” to the outside. If the upstream changes the tokenizer, adds a chat template, or re-types the quantification package, the behavior of the access party will change accordingly. The evaluation scores may only fluctuate a little, but the online output will quietly change its tone. The most torturous part when troubleshooting is here: everything looks normal in the log, but in fact, another input is being called.

What is really uncomfortable is not the change itself, but the fact that the change leaves no boundaries. As long as the name remains unchanged, regression, grayscale, and accident review can only focus on the results and guess the reasons. The model name continues to hang there, but the team has no way to confirm whether the one they got today is the same product as the one that ran the baseline last week.

Images, templates and quantization packages should be frozen together

Just locking the weight is not enough.

After an open source model actually enters the workflow, it usually contains more than just one .bin or .safetensors. It will also bring tokenizer, chat template, inference framework parameters, quantification files, download images, startup scripts and cache paths. If any one of them drifts, it may eventually show that “the model has become worse”. In fact, it is often not the model that changes, but the delivery package.

model:
  repo: example/model
  revision: 8f3c1a2
  tokenizer_revision: 8f3c1a2
  cache_dir: /opt/model-cache

This type of configuration seems verbose, but it results in three things: the image is responsible for availability, the version number and hash are responsible for reproducibility, and the startup script is responsible for consistent inference parameters. Without any of these layers, the so-called “already public” model can only be considered a semi-finished product. For the access party, the most important thing is not whether it can be downloaded, but whether the downloaded content can still produce the same set of results three weeks later.

What really needs to be preserved is the ability to reproduce

After the open source model is made public, what the team needs to protect is not a mysterious entrance, but the reproducibility.

Once the reproducibility is interrupted, the evaluation baseline will drift, the A/B results will be distorted, and the accident playback will lose reference. By the time everyone is discussing the problem using the saying “it was normal last week”, version management has basically failed. It doesn’t make much sense to talk about model capabilities at this point; only by tightening the fixed version, image synchronization, hash verification and rollback path can we be qualified to continue talking about the effect.

This type of model is more like a software artifact than a web service. If the web page is broken, you can still see 500, and the model version is drifting, and in many cases it will just slowly change in the output. On the surface it still has the same name, but in fact half the system has been changed.