Revisiting the LiRA Membership Inference Attack Under Realistic Assumptions
Authors: Najeeb Jebreel (Universitat Rovira i Virgili), Mona Khalil (Universitat Rovira i Virgili), David Sánchez (Universitat Rovira i Virgili), Josep Domingo-Ferrer (Universitat Rovira i Virgili)
Volume: 2026
Issue: 3
Pages: 765–786
DOI: https://doi.org/10.56553/popets-2026-0105
Abstract: Membership inference attacks (MIAs) have become the standard tool for evaluating privacy leakage in machine learning (ML). Among them, the Likelihood-Ratio Attack (LiRA) is widely regarded as the state of the art when sufficient shadow models are available. However, prior evaluations have often overstated the effectiveness of LiRA by attacking models overconfident on their training samples, calibrating thresholds on target data, assuming balanced membership priors, and/or overlooking attack reproducibility. We re-evaluate LiRA under a realistic protocol that (i) trains models using anti-overfitting (AOF) (and transfer learning (TL), when applicable) to reduce overconfidence as it would be desirable in production models; (ii) calibrates decision thresholds from shadow models and data rather than (usually unavailable) target data; (iii) measures positive predictive value (PPV, a.k.a. precision) under shadow-based thresholds and skewed - rather than unrealistically balanced - membership priors (pi <= 10%); and (iv) quantifies per-sample membership reproducibility across different seeds and training variations. In this setting, we find that (a) AOF significantly weakens LiRA and TL further reduces the effectiveness of the attack, while improving model accuracy; (b) with shadow-based thresholds and skewed priors, LiRA's PPV often drops from near-perfect to substantially lower levels, especially under AOF/AOF+TL and for pi <= 10%; and (c) LiRA's thresholded vulnerable sets at extremely low FPR exhibit poor reproducibility across runs, while likelihood ratio-based rankings are more stable. These results suggest that (i) LiRA, and likely weaker MIAs, are less effective than previously suggested, and their positive inferences can be less reliable under realistic settings; and (ii) for MIAs to serve as meaningful privacy auditing tools, their evaluation must reflect pragmatic training practices, feasible attacker assumptions, and reproducibility considerations. We release our code at: https://github.com/najeebjebreel/lira_analysis.
Keywords: machine learning, privacy, membership inference attacks, attack reliability, reproducibility
Copyright in PoPETs articles are held by their authors. This article is published under a Creative Commons Attribution 4.0 license.
