27/03/2025 - 14:30 - ISBA C115 - 
 

Clement Berenfeld 

(Institut national de recherche en sciences et technologies du numérique) 

Will give a presentation on : 

A causal framework for reliable membership inference attack evaluation

Abstract :  
Quantifying memorization is central for assessing privacy risks in machine learning. The standard tool for this purpose, membership inference attacks (MIAs), traditionally re-quires multi-run evaluations (repeated retraining) that are computationally prohibitive for modern large-scale models. This has led to the adoption of one-run methods (training once with a randomized subset of points) and zero-run methods (evaluating models “as-is”), though their statistical soundness remains unclear. We address this by reframing MIA evaluation as a causal inference problem, defining memorization as the causal effect of a data point’s inclusion in the training set. Our work reveals systematic issues: one-run regimes introduce interferences between jointly inserted points, and zero-run regimes introduce confounding from non-random membership assignments. We formalize these challenges by introducing a new interference model for treatment effect estimation, derive causal counterparts to standard MIA evaluation metrics, and propose estimators that are provably consistent by leveraging learning-theory properties.