The design direction looks promising:
My short answers to the three things you asked about would be:
-
I would keep the LoRA/PEFT pre-flight as a first-class part of the tool. In fact, I think it may be one of the more practically distinctive parts of the project. Before expanding mainly by model family, though, I would test it against a small matrix of semantic failure modes — truncation, completion/assistant masking, packing, chat-template masks, trainable embeddings/tokens, tied weights, etc. Those seem more likely to expose false “all clear” paths than simply trying more architectures.
-
100 members / 200 controls and {1,4,16} look reasonable to me as a powered stress-test profile, but I would hesitate to call them a universal default. The useful operating point depends on canary construction, repetition/exposure, the training recipe, the membership score, and the FPR you want to resolve. I would probably expose something like smoke / routine / powered profiles rather than searching for one magic count.
-
There are real Trainer/TRL/PEFT-style silent-failure classes worth turning into regression fixtures. The highest-information ones look like “the record survived, but the intended tokens did not receive loss” rather than exotic model-specific failures.
I tried a few small CPU probes against the current repo, and they mostly reinforced that direction rather than changing it.
One especially encouraging result: when I deliberately constructed a canary whose secret completion would not fit inside a very small max_length, the pre-flight stopped before training with a fatal “record exceeds max_length” finding. That is exactly the kind of failure I would want an audit tool to catch before interpreting a later zero-leak result.
The main thing I would tighten is the meaning of each pre-flight success state.
A useful progression might be something like:
record observed → token span aligned → labels inspected → secret span directly supervised
rather than allowing all of those to collapse into a single “found/trainable” concept.
That distinction matters because the current TRL SFT pipeline can transform the original example substantially: it constructs completion/assistant masks, converts excluded labels to -100, truncates to max_length, and may pack examples into fixed-length sequences. So “I can still find the string” is useful evidence, but it is not necessarily the same evidence as “I verified that these exact secret tokens participate in the training loss.”
A compact default route I would probably use is:
- Keep the static/config pre-flight.
- Add a few known-bad semantic regression fixtures.
- Report the level of verification achieved for each canary.
- Report scan coverage explicitly.
- Then widen across representative PEFT/model configurations.
That seems relatively cheap while directly protecting the central goal: avoiding a falsely reassuring audit caused by the audit probe itself not participating in training.
Why I would test semantic failure modes before adding lots of model families
There are several useful historical/current TRL examples here.
A recent SFTTrainer issue #6668 demonstrates a particularly relevant failure shape: with prompt-completion data and completion_only_loss=True, truncation can remove every completion token. The resulting batch has every label set to -100; the underlying model forward loss becomes NaN, while the training result can appear as loss: 0.0.
I would not treat that issue as proof that the same thing happens inside memaudit, and library behavior can change. But it is an excellent known-bad fixture for a privacy pre-flight:
if an audit canary’s supervised span disappears, can the auditor fail loudly before training?
Similarly, older TRL regressions show why configuration-space tests can be more informative than a list of model names:
Some of these are fixed historical bugs, not claims about current TRL. That is actually what makes them useful here: they can become permanent regression fixtures without requiring the bugs to still exist upstream.
Current TRL also has several semantics that are worth making explicit in the matrix:
| Axis |
Why it matters for a canary audit |
completion_only_loss |
Presence in the prompt does not imply direct loss on the secret |
assistant_only_loss |
Depends on correct assistant-token masking |
| chat template |
Assistant-only training depends on generation markers / supported templates |
max_length |
Can remove the intended supervised span |
| packing |
Changes sequence boundaries and mask handling |
packing_strategy |
bfd, bfd_split, and wrapped have different overflow semantics |
| pre-tokenized datasets |
User-supplied labels may bypass some preparation assumptions |
skip_prepare_dataset |
Moves responsibility from SFT preparation to the supplied collator |
| padding-free / optimized kernels |
Adds another route by which masks and representations can differ |
The current SFTTrainer documentation is unusually helpful here because it spells out most of these transformations.
On the PEFT side, I would similarly choose representative configuration semantics, not just “Llama / Qwen / Gemma”.
The current PEFT LoRA documentation has fairly detailed behavior around:
modules_to_save,trainable_token_indices,- tied embeddings /
lm_head,
ensure_weight_tying,- targeting embeddings or
lm_head,
- whether the same or different token indices are trained on tied layers.
It even notes that some of the tying logic depends on conventional layer names such as embed_tokens and lm_head, so correct behavior cannot be guaranteed for arbitrary naming schemes.
That suggests a compact matrix like:
plain LoRA
LoRA + modules_to_save
LoRA + trainable_token_indices
tied embedding/lm_head + ensure_weight_tying
untied embedding/lm_head
quantized LoRA/QLoRA
one MoE/target_parameters configuration
You probably do not need every combination. A few deliberately adversarial representatives would tell you more than a large compatibility badge table.
One small control I tried was the ordinary LoRA case: train a tiny adapter for a few steps, compare active logits, then enter disable_adapter(). In that ordinary path, adapter-disabled logits returned exactly to the pre-LoRA base logits and the active adapter state was restored afterward.
So I would not treat disable_adapter() itself as suspicious in normal LoRA. If desired, a few-token “base-equivalence” doctor check could simply be a cheap guard for unusual PEFT configurations rather than another required feature.
Pre-flight reporting: presence and supervision are slightly different questions
This is the part I would most strongly consider separating in the report schema.
In the current preflight.py, token-level matches can support stronger checking than the string fallback. A token-level path can inspect the relevant token positions and their labels; a string fallback mainly establishes that the textual material is still observable in the processed record.
In my small truncation probe, the useful combination was:
record found: yes
token-level hit: no
string-level hit: yes
pre-flight fatal: yes (sequence exceeds max_length)
That is actually a nice demonstration of why separate evidence levels help. The tool correctly stopped the run, but “found” by itself would not tell the whole story.
I might make the report terminology something like:
| Status |
Meaning |
record_observed |
The canary can still be located after preprocessing |
secret_token_aligned |
The expected token span can be located unambiguously |
loss_mask_checked |
Labels for the span were actually inspected |
directly_supervised |
At least the intended secret span receives training loss |
verification_unknown |
The pipeline representation does not allow a stronger conclusion |
The names can obviously differ; the useful part is the separation.
One terminology caution: I would avoid turning
“the secret span has labels -100”
into
“the information is untrainable.”
Those are not equivalent in the general case.
There is now work specifically on input-only PII memorization, where private information appears in fine-tuning inputs but is not an intended training target, yet can still become extractable after fine-tuning. See the EACL 2026 study on unintended input-only PII memorization.
So for this tool I would phrase the property narrowly:
“the canary protocol expected this secret span to be directly supervised, and pre-flight verified/did not verify that condition.”
That gives the pre-flight a crisp contract without claiming that all other forms of memorization are impossible.
One concrete low-cost boundary I also noticed is the survival-scan window.
Using the current injection/scan logic on a synthetic 80k-row host dataset with 100 canaries, the default 50k-row scan did not necessarily observe every inserted canary. On one seed I got 83/100 found in the default scan versus 100/100 with a full scan. Repeating the placement simulation over 100 seeds gave a mean of about 86.4/100 observed in the first 50k.
The misses were overwhelmingly the low-repetition probes, which is exactly what one would expect: a 16x canary has many opportunities to land inside the scan window; a 1x canary has only one.
I therefore would not necessarily remove the limit — a bounded pre-flight may be a sensible performance choice — but I would distinguish:
missing after full/known-complete inspection
from
not observed within pre-flight scan window
and add something like:
rows_scanned: 50000
rows_total: 80100
scan_complete: false
That is a very small reporting change with a fairly large interpretability payoff.
On the 100 members / 200 controls / {1,4,16} question, I think the repetition tiers are useful, but I would show them as a stress-response curve in addition to the pooled headline.
Using the same global threshold from the checked-in powered report, the 18 detected members decompose as:
| Repetitions |
Detected |
| 1x |
0 / 34 |
| 4x |
2 / 33 |
| 16x |
16 / 33 |
| pooled |
18 / 100 |
I do not think this weakens the case-study result. If anything, it makes the experiment more informative.
It tells the reader that the pooled 18% result is substantially a duplication/exposure stress signal, not evidence that an ordinary single-exposure record in that run had an 18% detection probability.
That is consistent with the broader memorization literature: repetition/duplication can strongly increase extractability and memorization. A useful classic reference is Kandpal et al., “Deduplicating Training Data Mitigates Privacy Risks in Language Models”, which found a strong relationship between sequence duplication and regeneration.
So I would probably keep {1,4,16}, but make the semantics explicit:
1x = closer to a single-exposure probe
4x = moderate stress
16x = high-exposure stress
pooled = overall powered-audit headline
Then report both the pooled result and the tier curve.
How I would think about 100/200 as defaults
I would avoid choosing the counts independently of the rest of the audit design.
The power of a canary audit depends on at least:
canary construction
× repetition/exposure
× training recipe
× model/adaptation method
× membership score
× target FPR
× number of members
× number of controls
The ICLR 2025 paper “Privacy Auditing of Large Language Models” is relevant here. One of its main results is that canary construction itself can change low-FPR detection substantially. Their stronger canaries achieved much higher TPR at 1% FPR than earlier canary designs under the same broad auditing objective.
So “200 controls” cannot really be classified as conservative/aggressive by itself.
At a 1% operating point, 200 controls also means you are estimating a fairly extreme tail from a small number of observations. Your existing refusal to produce a headline when controls are clearly insufficient is a good design choice, and the bootstrap/stability information already present in the project points in the right direction.
I might turn this into user-facing profiles instead:
| Profile |
Purpose |
Example behavior |
smoke |
catch obvious memorization/integration failures cheaply |
fewer probes, clearly marked exploratory |
routine |
recurring fine-tune audit |
moderate members/controls |
powered |
publishable/internal review stress run |
larger controls, repetition tiers, calibration stability |
The actual numbers can evolve without changing the conceptual contract.
For the powered profile, I would expose two separate uncertainties:
- member-side detection uncertainty — e.g. the CI around the detected fraction;
- threshold/calibration stability — how much the low-FPR decision boundary moves under plausible resampling of controls.
Those answer different questions, so showing both makes the headline easier to interpret.
I also agree with the decision to keep membership and regurgitation as separate outputs.
That is not just a presentation choice: they are different attack surfaces.
The EDPB’s Opinion 28/2024 explicitly lists membership inference and training-data regurgitation separately, and makes the broader point that successful testing is evidence about the attacks actually tested rather than a universal privacy guarantee.
There is also fine-tuning-specific empirical work using both signals. For example, “Memorization in Fine-Tuned Large Language Models” uses membership inference and prompted-prefix verbatim reproduction as separate measurements.
So the disagreement in your case study is interesting in its own right.
I would only scope the wording of the second result fairly tightly:
0/100 regurgitation under this prefix / decoding / exact-match protocol
rather than “no extraction risk”.
Other prompting strategies, approximate matches, longer/shorter prefixes, sampling, or paraphrased variants are different tests. Those could become optional attack profiles later, but I would not expand v0.1 just to cover them.
There is one additional branch I would keep explicitly separate from the controlled-canary audit: the real-record exploratory path.
Real-record set-level analysis: I would separate this contract from the canary audit
In a small check of the current path, I found a semantic distinction that may be worth making explicit.
run_audit() can work with a real held-out set, but in the path I tested, when no explicit held-out dataset was supplied, the real-record sampling helper split sampled training records and used one side as the comparison side.
At the same time, the standard MemorizationAuditCallback did not expose a held_out= argument in that checkout.
If that remains the intended API, I would avoid describing the fallback result as “training records versus held-out non-members”, because both sides originate from the training dataset.
This does not affect the TinyLlama powered canary result you posted — that case uses real_sample=0.
So I would treat it as a separable future-facing contract:
controlled-canary audit
-> has known member/non-member assignment by construction
-> headline-capable
real-record ranking
-> exploratory score/ranking
real-record set-level inference
-> only inferential when a genuine held-out/non-member population is supplied
A minimal implementation option would be:
if true held-out is supplied:
run set-level member-vs-nonmember comparison
otherwise:
return ranking/descriptive scores
skip or rename the inferential set-level test
This would actually strengthen your own distinction between “trust the set-level result more” and “per-record flagging is exploratory”, because the set-level result would then have an explicit population contract.
There is a useful conceptual reason for keeping controlled canaries separate from arbitrary real-record membership claims.
Zhang et al., “Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data” points out the difficulty of using arbitrary-record MIA as a training-data proof when the required null distribution cannot be sampled. Importantly, the paper identifies special pre-registered canaries as one path around that problem.
That seems very compatible with your architecture: the canary audit can stay the controlled, interpretable core, while real-record scoring can remain a separately labelled exploratory layer.
A much smaller provenance detail: when I tested family="high_ppl" without a model or corpus, the generated canaries reported an actual source of uniform_vocab.
Again, I do not think this invalidates the audit. I would just record both:
requested_family: high_ppl
actual_generator: uniform_vocab
because canary construction can materially affect audit power.
That makes reports easier to compare across versions and prevents a fallback implementation from silently changing the meaning of a named audit profile. The relevant implementation is in canaries.py.
A possible compact report/provenance schema
Something along these lines would make the report fairly self-describing:
audit_profile:
name: powered
target_fpr: 0.01
canaries:
requested_family: high_ppl
actual_generator: uniform_vocab
repetitions: [1, 4, 16]
requested_members: 100
controls: 200
preflight:
rows_total: ...
rows_scanned: ...
scan_complete: ...
record_observed: ...
token_aligned: ...
loss_mask_verified: ...
directly_supervised: ...
fatal: ...
membership:
scorer: ...
overall_tpr: ...
ci: ...
calibration_stability: ...
by_repetition:
1: ...
4: ...
16: ...
regurgitation:
prefix_policy: ...
decoding: ...
match_rule: exact
detected: ...
Not all of those need to be mandatory fields. The useful design property is that a future reader can reconstruct what was actually tested, not just see a single privacy number.
That also lines up with the EDPB emphasis on documenting the threat model, tests and controls rather than treating one successful test as a certificate.
One more architectural choice I would preserve is keeping the membership scorer replaceable.
Min-K%++ is a perfectly reasonable backend to have, but the MIA literature is moving quickly, especially for fine-tuned models. For example, the ACL 2026 paper EZ-MIA reports substantially stronger low-FPR detection than previous approaches in several fine-tuning settings while still requiring only a small number of forward passes.
I would not interpret that as “replace Min-K%++ now”.
I would interpret it as:
memaudit
├── audit orchestration
├── canary construction/injection
├── pre-flight validity checks
├── reporting/calibration
└── membership scorer backend
If the last piece is pluggable, the valuable Trainer/TRL/PEFT integration survives changes in the attack literature.
That separation also resembles the design of broader audit libraries such as Privacy Meter, which separates the auditing workflow from particular membership signals/attack variants.
Again, I would keep this as a design boundary rather than adding multiple sophisticated attacks to v0.1.
So if I were prioritizing by implementation cost × information gain, my order would be roughly:
-
Add known-bad preprocessing/masking fixtures to pre-flight tests.
- truncation removing the supervised canary span;
- assistant/completion mask disappearance;
- one packing case.
-
Separate pre-flight evidence levels.
- observed;
- token-aligned;
- label-checked;
- directly supervised.
-
Expose scan coverage / distinguish scan-window misses from true misses.
-
Report repetition tiers alongside the pooled MIA headline.
-
Keep true-held-out real-record inference separate from exploratory real-record ranking.
-
Record requested versus actual canary-generator provenance.
-
Then broaden across PEFT/model configurations and, later, alternative MIA scorers.
That feels like a fairly small amount of work relative to the amount of ambiguity it removes.
Most importantly, I would not broaden the scope into inversion, shadow models, approximate extraction, DP auditing, paraphrase/range membership, etc. just because those things exist. You already stated that v0.1 is not a compliance certificate and deliberately has a narrower attack surface. That seems like the right boundary.
The strongest part of the project, to me, is not that any single MIA score is final; it is that the audit is being treated as an instrument whose own validity has to be checked before its result is trusted.
The small truncation test was a good example: the pre-flight really did stop a case where the planned canary supervision would not survive the configured sequence length.
If that same fail-closed philosophy is carried through the mask-verification levels, scan coverage, and true-held-out distinction, I think the tool becomes easier to reason about without needing to become much larger.
, so I’ll put this here on the forum first: