If you SFT on private or customer data, at some point legal, security, or a customer asks: “Show me the model didn’t memorize any of it.” Most teams have nothing — no test, no report, a shrug.
memaudit turns your existing TRL job into that evidence. Two lines: inject() plants calibrated decoy secrets in your raw dataset, MemorizationAuditCallback does the rest. When training ends you get a local JSON report you can forward — versioned, threat-model table mapped to EDPB Opinion 28/2024, checksums, no phone-home. Apache-2.0.
It’s built for how SFTTrainer actually works. Your trainer tokenizes, masks, truncates, and packs before callbacks fire — which is exactly how naive memorization tests silently break: the probe lands in the prompt with completion_only_loss=True, or truncation eats it, or LoRA never touches the layer it lives in. Result: a test that says “clean” about a model that never saw the test. memaudit verifies at train start that every decoy actually receives loss under your real config — masking, packing, max_length, chat templates, LoRA/QLoRA, tied embeddings — and fails the run instead of printing a fake all-clear.
What the report answers — two questions regulators treat separately, never blended into one score: can an attacker with model access tell which records were trained on (membership, at a 1% false-alarm point with confidence intervals), and does the model complete a training record from its prefix (regurgitation, under a stated protocol). It refuses to print headlines when your canary/control counts are too thin, and anything that didn’t run says “not run” — never 0.
Public proof run (reproduces on a laptop): TinyLlama-1.1B-Chat + 20k Alpaca, LoRA r=8, 1 epoch, 100 decoys / 200 controls. Membership: 10/100 at 1% false alarm, CI [0.049, 0.176], AUC 0.837. The leakage curve is the useful part: decoys seen once — 0/34; seen 16× — 9/33. Duplication is what leaks, and the report shows that curve for your dataset. Regurgitation: 0/100 under the exact-match protocol. Full tables: case study.
Repo: https://github.com/mem-audit/memaudit
Site: https://ansh200516.github.io/memaudit-site/
Case study: https://github.com/mem-audit/memaudit/blob/main/docs/case-study-alpaca.md
The ask: I’d like PEFT/TRL eyes on the pre-flight — especially where only the assistant reply gets loss, packing drops tokens, or LoRA skips the layer you thought you trained. If a canary never receives loss, the audit must refuse the false all-clear; tell me where that check can still be fooled.
Put it on one training job this week. memaudit produces test evidence, not a compliance certificate.