For now, I tried a few things myself:
The short version is: the determinism claim held up surprisingly well in the external checks I could run, including a Tesla T4 path and a one-step Qwen3-4B test using the same Q4_K_M base SHA and ToolUse training-data SHA recorded in the published adapter provenance.
There was also one useful boundary condition: holding the binary fixed and changing the host was not enough to break the tiny test, but changing the build profile was enough to change the adapter SHA. So I think there are really a few different reproducibility contracts hiding under the word “deterministic”, and separating them makes the results easier to interpret.
The strongest 4B check I ran was:
published base GGUF SHA == probe base GGUF SHA
published training SHA == probe training SHA
same Runner binary
same rank / alpha / lr / ctx / seed semantics
same one-step training job
CPU path
vs
Tesla T4-assisted backward
The CPU and T4-assisted runs both reported the same one-step loss (0.676138) and produced the same adapter SHA-256:
79c5f136486bcad288f0ad2f4f66345dab07db668f0cfe3de9d757bbfdb48e8f
The adapter files were byte-for-byte identical.
The T4 side was not a silent CPU fallback; Runner reported:
train-gpu: Tesla T4 — backward matvec on device
The exact base was the bartowski Qwen3-4B Q4_K_M, whose file SHA is:
fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3
and the regenerated ToolUse training set also matched the SHA recorded in the published adapter provenance.
I would still phrase this as an exact-base/data one-step bridge, not a reproduction of the full published training run: the published adapter provenance records Runner 0.1.20-alpha and 316 steps, while my probe used a pinned 0.2.0-era commit (8652a7f...) and deliberately stopped after one update.
One distinction that may be useful for future reproduction reports
After doing these checks, I would probably separate at least three things:
1. Artifact determinism
same binary + same model/data/config
-> same adapter bytes?
2. Build/toolchain reproducibility
same source + independently built binary
-> same adapter bytes?
3. Behavioral reproducibility
same predictions / probabilities / task behavior?
Those did not behave identically in my small tests.
With the same binary, I could replay the tiny fixture on a second Linux x86_64 host with a different Xeon generation, kernel and glibc version and still get the same adapter SHA. Thread counts 1, 2, and 4 also gave the same adapter.
But on the same Colab machine, rebuilding the same source with a different ISA/build profile changed the tiny adapter SHA.
Interestingly, the resulting parameter difference was extremely small despite the file SHA changing: the adapter cosine was effectively 1 and the maximum F32 difference was only on the order of 1e-7.
So I would treat SHA-256 as a very sensitive artifact-identity oracle, but not as a behavioral-distance metric:
different SHA
does not necessarily mean
meaningfully different behavior
That distinction may be useful if people start reporting results from ARM, Windows, different compilers, different release binaries, etc.
For that reason, a third-party reproduction record might be easier to interpret if it includes something like:
Runner commit
Runner binary SHA-256
compiler + version
build profile / ISA
OS + architecture
base GGUF SHA-256
training-data SHA-256
seed / rank / alpha / optimizer config
adapter SHA-256
loss trajectory
rather than treating “same source checkout” and “same executable” as interchangeable.
This is also a stronger reproducibility target than the one normally assumed by ML frameworks. For comparison, PyTorch’s reproducibility documentation explicitly does not promise complete reproducibility across releases/platforms or between CPU and GPU execution.
For the BF16 / Q8_0 / Q4_K_M part, I think there is a relatively cheap next measurement before constructing a substantially harder benchmark.
Right now the interesting result is:
adapter weights:
BF16 ≈ Q8, Q4 somewhat farther away
held-out exact-call score:
BF16 = Q8 = Q4 = 1.00
That does not necessarily mean the learned functions are equally far apart or equally close. It may just mean the current exact-call metric is saturated.
My default next step would be:
existing held-out prompts
|
v
teacher-force the gold tool call
and compare its token log-prob / NLL
|
+-- differences visible
| -> the precision-dependent path is already
| behaviorally visible below top-1
|
+-- still almost identical
|
v
compare earlier checkpoints
|
+-- trajectories separate
| -> different learning path,
| similar final endpoint
|
+-- still insensitive
-> move to a harder / OOD /
near-confusable tool task
That seems cheaper than immediately building a large new evaluation suite, and it keeps the claim narrow: first ask whether the existing test hides probability-margin differences; only then ask for a harder task.
What I actually tested
Tiny deterministic fixture
I first used Runner’s tiny deterministic training fixture.
On CPU, these all produced the same adapter SHA:
repeat A, t=2
repeat B, t=2
t=1
t=2
t=4
Then I set:
RUNNER_TRAIN_GPU=1
on a Colab Tesla T4.
The CUDA-assisted backward path engaged, and the resulting adapter was byte-identical to the CPU adapter.
I also checked that this was not merely producing an unchanged/no-op adapter: teacher-forced scoring improved from roughly
base:
NLL 5.5785
PPL 264.68
trained:
NLL 5.1719
PPL 176.24
and the CPU and T4-trained adapters produced the same score.
The pinned repository’s training tests also passed in that environment.
Same binary, second Linux host
I then took the actual binary built in Colab, without recompiling it, and ran the same tiny fixture on another Linux x86_64 host.
The two environments differed in CPU generation, kernel, glibc and CPU count.
The adapter SHA remained identical.
That does not establish x86/ARM or Linux/macOS/Windows identity, but it does suggest that, at least for this fixture, host/runtime variation alone was not sufficient to break the result once the executable itself was fixed.
Build profile boundary
On the other hand, rebuilding on the same machine changed the exact bytes.
A normal local build and an explicit x86-64-v3 build produced different tiny adapter SHAs.
That is why I think “same binary on other hardware” and “independent rebuild from the same source” are worth treating as two separate tests.
I would not attribute this to one specific compiler transformation without a smaller isolation test; the observation I can actually support is just that changing the build/code-generation profile changed the exact training artifact.
Real Qwen3-4B bridge
I first repeated the CPU/T4 check with the first-party Qwen3-4B Q4_K_M GGUF.
That one-step CPU/T4 comparison also produced byte-identical adapters.
That was useful as a real-model sanity check, but that GGUF did not have the same file SHA as the base recorded by the published ToolUse adapter.
So I repeated the experiment a second time using the exact Q4_K_M file recorded in the published provenance.
For the final run:
base:
bartowski/Qwen_Qwen3-4B-GGUF
Qwen_Qwen3-4B-Q4_K_M.gguf
base SHA:
fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3
ToolUse train examples:
158
train-data SHA:
4c3feca6afa9d776b0e5d08afb6ac4b134bee93a94548dd934a74b2c944d6a5a
Both SHAs matched the published adapter provenance before the training comparison was allowed to run.
The one-step configuration used the published rank/alpha/lr/context settings:
rank = 8
alpha = 16
lr = 1e-4
ctx = 128
One small reproducibility footgun I ran into: the provenance records seed 0, but explicitly passing:
-s 0
is not equivalent — Runner correctly rejects that because RNG state zero is a fixed point. For the corrected probe I omitted -s, which invokes Runner’s deterministic training-init default that is recorded as seed 0 in the training provenance.
That seems worth documenting for anyone trying to reproduce the sidecar mechanically.
What this test does not establish
I would not generalize the result to any of these yet:
- all hardware;
- ARM64;
- Windows or macOS;
- arbitrary independently compiled binaries;
- every CUDA training path;
- the entire 316-step trajectory;
- byte reproduction of the historical
0.1.20-alpha run;
- later Runner revisions;
- behavioral equivalence whenever two adapter SHAs differ.
The positive result is narrower:
for the pinned Runner build and the tested Qwen3-4B Q4_K_M base/data/config, the first CPU update and the Tesla T4-assisted update produced the same adapter bytes.
Where I think this sits relative to nearby GGUF / training work
I would separate three ideas that can otherwise sound like the same thing:
loading a GGUF for further training
!=
keeping the quantized GGUF weights as the frozen training base
!=
using the serving forward path itself as the training forward path
For example, Transformers supports loading GGUF for further training, but its documented path dequantizes the GGUF checkpoint to FP32 PyTorch weights.
There is also already nearby work on training LoRA directly over GGUF quantized bases. In particular, this Unsloth discussion / proof of concept explicitly explores LoRA training over GGUF rather than a bitsandbytes 4-bit base.
So I would not frame “LoRA over GGUF” by itself as the unusual part here.
The part I find more interesting in Runner is the systems boundary:
the forward used while adapting the LoRA is the same forward implementation used to serve that quantized GGUF.
There is a useful analogy in another domain: the vLLM + TorchTitan bitwise-consistent train/inference work audited training/inference forward kernels for numerical equivalence and reused inference-side forward operations while supplying custom backward implementations.
That project is solving an on-policy RL problem rather than GGUF LoRA adaptation, so I would not treat them as the same system. But the design pressure is similar: reducing or eliminating the numerical seam between the model that produces outputs and the model through which gradients are computed.
That seems like a useful existing vocabulary for this part of Runner.
One related branch: merging the adapter back into low-bit weights
The later merge/requantization result also seems worth keeping as a separate branch rather than mixing it into the determinism claim.
There is a very close open research question in llama.cpp issue #13485, which explicitly proposes comparing:
1. train LoRA on full precision
-> merge
-> quantize
2. train LoRA on quantized model
-> merge
3. train LoRA on quantized model
-> keep adapter separate
That is almost exactly the comparison I would want around the Q4 merge result.
There is also a practical reason to distinguish quantization from requantization. The current llama-quantize documentation warns that requantizing tensors that are already quantized can severely reduce quality compared with quantizing from 16- or 32-bit weights.
And PEFT’s 4-bit LoRA merge implementation likewise explicitly warns that merging into a 4-bit linear layer can produce different generations because of rounding errors: PEFT bnb.py.
So if this branch gets explored further, one fairly clean control would be:
A. Q4 base + separate LoRA
vs
B. high-precision base
+ merge LoRA
+ one-shot quantize to Q4
vs
C. already-Q4 base
+ merge delta
+ requantize to Q4
That would help separate:
"the Q4 grid is too coarse for this delta"
from:
"the second projection/requantization step is the damaging part"
I would still keep this result local to the tested adapter/model until more cases are measured; I do not think the current evidence supports a general statement like “merging LoRA into Q4 destroys the adaptation.”
So my current read is:
- the external determinism check is positive for the paths I could test;
- binary identity looks like an important part of the reproducibility contract;
- adapter SHA and behavioral distance should probably be reported separately;
- and, for the precision experiment, gold-sequence probability / NLL looks like the cheapest next discriminator before moving to a harder benchmark.
At least from these checks, I would probably spend the next experiment budget on the BF16/Q8/Q4 behavioral-separation question rather than on another small determinism repetition.