FedMIA against DP-SGD

Federated learning (FL) has emerged as a promising paradigm for training machine learning models in privacy-sensitive domains such as healthcare, finance, and cross-institutional collaborations. Instead of centralizing data, multiple clients collaboratively train a shared model while keeping their raw data local. At each communication round, clients compute updates based on their private data and send them to a central server, which aggregates them into a global model. This decentralized approach is often perceived as inherently privacy-preserving, since no raw data is directly exchanged.

However, this assumption has been repeatedly challenged. Even when only model updates are shared, sensitive information about the underlying training data can still leak. Among the most prominent threats are membership inference attacks (MIAs)[2,3,4,5], where an adversary attempts to determine whether a specific data point was used during training. Formally, let $\mathcal{M}$ denote a learning algorithm used to train a classifier $f(\mathbf{x}; \theta)$ on a dataset $D_{\text{train}} = {(x^{(n)}, y^{(n)})}^N_{n=1}$. After training, the model $f(\mathbf{x}; \theta^*)$ can be queried on arbitrary inputs. A membership inference attack aims to infer, given an input $\mathbf{x}$ and access to the trained model, whether $\mathbf{x} \in D_{\text{train}}$. Such attacks are particularly concerning in high-stakes applications, where the mere presence of a record (e.g., a patient in a medical dataset) is already sensitive.

A recent and particularly powerful attack in federated settings is FedMIA [1], which exploits the temporal dynamics of model updates across communication rounds to perform sample-level membership inference against individual clients. This makes it significantly more powerful than classical MIAs, especially in cross-silo settings where each client holds a substantial and semantically meaningful dataset.

To counter such threats, differential privacy (DP)[6,7,10] has become a standard defense mechanism, offering formal guarantees on the leakage of individual data points. In this work, we investigate the interplay between FedMIA and Differentially Private Stochastic Gradient Descent (DP-SGD) [9] in a horizontal federated learning setting. Our goal is twofold: (i) to evaluate how effective FedMIA remains under realistic training conditions, and (ii) to quantify the trade-off between privacy and model utility when DP-SGD is applied.

FedMIA Setup and Method

We consider a horizontal federated learning setting with one central server and $K$ clients. Each client $k$ holds a local dataset

and all client datasets are assumed to be pairwise disjoint. Training proceeds over $T$ communication rounds, during which each client sends updates, such as gradients or model parameters, to the server.

The adversary is assumed to be a semi-honest server that faithfully follows the protocol but attempts to infer whether a specific sample $(x,y)$ belongs to a target client’s dataset $D_{\text{tar}}$. The server observes the sequence of client updates

A membership inference attack can then be formulated as the following hypothesis test:

A natural approach is to use a likelihood-ratio test,

where \(\mathbb{Q}_{\mathrm{in}}(I \mid x,y)\) and \(\mathbb{Q}_{\mathrm{out}}(I \mid x,y)\) denote the distributions of updates produced when the training data include or exclude the target sample \((x,y)\), respectively. The corresponding probability density functions are denoted by \(p_{\mathrm{in}}\) and \(p_{\mathrm{out}}\).

In practice, however, estimating \(\mathbb{Q}_{\mathrm{in}}(I \mid x,y)\) is difficult because the attacker does not know which observed updates were influenced by the target sample. FedMIA therefore avoids estimating the in-distribution and instead constructs a one-tailed test using only \(\mathbb{Q}_{\mathrm{out}}(I \mid x,y)\):

which represents the probability, under the null hypothesis, of observing an update at least as extreme as $I_{\text{tar}}$.

To leverage temporal information, FedMIA aggregates evidence across communication rounds through

where $\mathbb{Q}^t_{out}(I|x,y)$ are the distribution of updates trained on datasets without $(x,y)$ in the $t$th communication round. This formulation captures the intuition that membership leakage may occur in any round, and aggregating across rounds increases the attack’s effectiveness. In practice, FedMIA first computes low-dimensional measurements $M(I \mid x,y)$ from updates. Two variants are commonly used: FedMIA-I relies on loss values, while FedMIA-II uses gradient cosine similarity, which is typically more informative in federated settings. The attacker then estimates the distribution $\mathbb{Q}_{\text{out}}(M(I \mid x,y))$ and applies the likelihood-based test to infer membership.

DP-SGD

Differential privacy provides a rigorous framework for limiting the information that can be inferred about individual data points. A mechanism $\mathcal{M}$ is said to satisfy $(\varepsilon, \delta)$-differential privacy, also called approximated differential privacy if, for any two neighboring datasets $D$ and $D’$ differing in a single sample, and for any measurable set $S$ [6,10]: \(\Pr[\mathcal{M}(D) \in S] \leq \exp(\varepsilon)\Pr[\mathcal{M}(D') \in S] + \delta.\)

Here, $\varepsilon$ controls the privacy loss, with smaller values indicating stronger privacy, while $\delta$ represents a small probability of failure. If $\delta=0$ we referring to pure DP.

In the context of federated learning, we focus on sample-level differential privacy (SL-DP) [7], where neighboring datasets differ by one record. The goal is to ensure that the contribution of any single sample has a limited effect on the distribution of model updates.

A widely used method to achieve SL-DP is Differentially Private Stochastic Gradient Descent (DP-SGD) [9]. This algorithm modifies standard SGD in two key ways. First, per-sample gradients are clipped to a fixed norm $C$ to bound their influence. Second, Gaussian noise is added to the aggregated gradient before updating the model parameters. These two steps ensure that the training process satisfies differential privacy guarantees.

The overall privacy budget depends on factors such as the noise multiplier, the number of training steps, and the sampling rate. In practice, tools such as Opacus [10] are used to track and compute the resulting $(\varepsilon, \delta)$ guarantees.

A note to add is that FedMIA was tested against some defenses. One of them claiming to be client-level DP[1]. Looking through their repository we can observe that their DP is adding independent gaussian noise with zero mean and a given standard deviation to each model parameter each communication round. As a result, each client perturbs its entire local model before uploading it to the server. The noise is added directly to model parameters rather than to per-example gradients, and no gradient clipping or per-sample sensitivity control is applied. Consequently, this mechanism does not implement sample-level DP which is much needed in such context. Their method does protect the client as a single unit, but does not provide formal guarantees for individual training records within a client’s dataset. Therefore it has no privacy guarantee ($\varepsilon,\delta$). After an email echange the authors did say it should say local DP instead of client-level DP, however, depending on which resources one cites, this can still be debated but won’t be further discussed in this post.

Experimental Setup

Federated setting. We evaluate FedMIA and DP-SGD in a cross-silo horizontal federated learning scenario using FedAvg [12]. In this setup, multiple clients train locally on disjoint data and send model updates to a central server for aggregation. We study federations of 5 and 10 clients to measure how participant count affects model utility and attack vulnerability.
Datasets & model. Experiments use CIFAR-10 and CIFAR-100 (each with 60,000 images and standard train/test splits). We use an AlexNet-inspired convolutional architecture [11] adapted for CIFAR images as the original FedMIA setup, but also considering some changes which better suits the smaller image size, as well as using a dropout rate of 0.5.
Training protocol. Each client performs one local epoch per communication round. Models are trained for up to 200 rounds with early stopping on a validation split. Optimization uses SGD with initial learning rate 0.1 and a cosine learning-rate schedule. Local batches use size 100 and client data are assumed IID.
Defense (DP-SGD). We apply DP-SGD with per-example gradient clipping and Gaussian noise addition. We sweep the noise multiplier over {0.1, 0.5, 1.0} and the clipping norm over {1.0, 1.5, 2.0} to examine a range of sample-level privacy strengths.
Attacks evaluated. We evaluate FedMIA-I (loss-based) and FedMIA-II (gradient-similarity-based), which extract low-dimensional measurements from client updates and aggregate evidence across rounds.
Evaluation metrics. Attack performance is reported with AUC and TPR@FPR [13]. AUC gives an overall ranking quality, while TPR@FPR (e.g., TPR at FPR = 0.1%) highlights how many true members the attack recovers at a very low false-positive rate — a useful indicator of worst-case leakage.

Results

We begin by revisiting the baseline configuration used in the original FedMIA work. Using CIFAR-100 with a standard AlexNet architecture, we observe extremely high training accuracy (≈ 0.999) but very low test accuracy (≈ 0.39), indicating severe overfitting. While the original paper does not explicitly report these metrics, our reproduction highlights that the baseline model memorizes the training data rather than generalizing. This observation is crucial, as overfitting is strongly linked to vulnerability to membership inference attacks. When a model memorizes training samples, its behavior on seen versus unseen data becomes more distinguishable, providing a signal that attacks like FedMIA can exploit.

To better reflect realistic training conditions, we introduce architectural and regularization improvements, including dropout and adjusted weight decay, and adapt the model to the CIFAR input resolution (AlexNetCIFAR). These changes significantly improve generalization performance. On CIFAR-100, test accuracy increases from roughly 0.39 to 0.59, while on CIFAR-10 it reaches nearly 0.90. This improvement in utility has a direct impact on privacy. As the model generalizes better, the gap between training and test accuracy shrinks, and the effectiveness of both FedMIA-I and FedMIA-II decreases substantially. This confirms a key insight: attack success is strongly correlated with overfitting.

Comparing datasets, CIFAR-100 remains more challenging than CIFAR-10 due to its larger number of classes. This results in lower absolute accuracies and, interestingly, relatively stronger attack performance. The increased difficulty appears to amplify subtle differences between training and non-training samples, making inference easier. We also observe that the number of clients influences attack success. With fewer clients (e.g., 5 instead of 10), each client’s contribution to the aggregated update is larger, making it easier for the server to attribute changes to a specific client. Consequently, attack performance is higher in smaller federations.

When introducing DP-SGD, a clear privacy–utility trade-off emerges. Adding noise and clipping gradients reduces the distinguishability of updates, thereby weakening the attack. In this post we only consider the original model AlexNet since our server did not manage to let a big model like AlexNetCIFAR run with DP-SGD, CIFAR10, we use 5 clients and a droprate of $0.5$ as well as a weight decay of $0.001$. For small noise multipliers (e.g., $0.1$), we observe a moderate drop in test accuracy alongside a noticeable reduction in attack metrics such as AUC and TPR@low-FPR. As the noise increases (e.g., multiplier $1.0$), attack performance drops further, but at the cost of severe degradation in model utility.

As for the privacy guarantees, we fix the failure probability to $\delta = 10^{-5}$. Using the privacy accountant provided by Opacus, the resulting privacy budgets vary substantially depending on the chosen noise multiplier. For a noise multiplier of $1.0$, the training achieves a strong privacy guarantee of approximately $\varepsilon \approx 0.1$. Reducing the multiplier to $0.5$ increases the privacy loss to $\varepsilon \approx 8.9$, while a multiplier of $0.1$ results in extremely weak guarantees with $\varepsilon > 900$. However, in federated learning the effective privacy guarantee also depends on the information available to the attacker. An external eavesdropper observing only the aggregated global model benefits from parallel composition, since client datasets are disjoint. In this case, the overall privacy guarantee in each communication round is bounded by the maximum client privacy loss, i.e., $\max_i \varepsilon_{i,r}, \max_i \delta_{i,r}$. In contrast, a stronger adversary such as the central aggregator observes each client update individually across all communication rounds. The privacy loss therefore accumulates sequentially over rounds, leading to the guarantee \(\left(\sum_{r=0}^{E-1}\varepsilon_{i,r}, \sum_{r=0}^{E-1}\delta_{i,r}\right).\) An insider client observing the global model at every round experiences an intermediate setting, where the privacy loss accumulates over rounds while still depending on the maximum leakage among participating clients. Consequently, the practical privacy protection in federated learning is not solely determined by the DP-SGD parameters, but also by the attacker’s visibility into the training process and the communication protocol. In extreme cases, strong privacy settings render the model nearly unusable for the underlying classification task. This highlights a fundamental tension: while DP-SGD is effective in mitigating membership inference, achieving strong privacy guarantees can significantly impair learning performance.

Comparing the two FedMIA variants, FedMIA-II (gradient similarity) tends to be more robust across settings because it leverages directional information from gradients. FedMIA-I (loss-based) is easier to blunt with DP noise and clipping.

Discussion and Future Work

Our findings highlight several important insights into the relationship between model utility, training dynamics, and privacy leakage in federated learning.

First, the degree of overfitting emerges as a central factor governing vulnerability to membership inference. Models that generalize well are inherently more resistant to attacks like FedMIA, even in the absence of explicit privacy mechanisms. This suggests that standard regularization techniques—such as dropout, weight decay, and architectural tuning—play an indirect but important role in improving privacy.

Second, while DP-SGD provides a principled defense with formal guarantees, its practical deployment remains challenging. The trade-off between privacy and utility is substantial, especially in complex tasks or resource-constrained settings. In our experiments, achieving strong privacy often led to significant drops in accuracy, limiting the usefulness of the resulting models. This raises the question of how to design more efficient privacy-preserving training methods that maintain high utility.

Third, our analysis indicates that the federated setting itself influences privacy risks. Factors such as the number of clients, data distribution, and model architecture all affect the attack surface. In particular, smaller federations and poorly generalized models amplify leakage, suggesting that system-level design choices are as important as algorithmic defenses.

Several directions for future work arise from these observations:

• Adaptive privacy mechanisms. Instead of applying uniform noise throughout training, future approaches could dynamically adjust privacy parameters based on training dynamics or model convergence.
• Alternative defenses. Beyond DP-SGD, techniques such as secure aggregation, regularization-based defenses, or adversarial training against inference attacks may offer improved privacy–utility trade-offs.
• Non-IID data settings. Our experiments assume IID data across clients, which simplifies analysis. In real-world federated learning, data is often highly heterogeneous, which may significantly affect both attack effectiveness and defense mechanisms.
• Scalability and model complexity. Applying DP-SGD to larger models remains computationally expensive. Exploring lightweight architectures or more efficient privacy accounting methods could make privacy-preserving FL more practical.

In summary, while differential privacy provides a strong theoretical foundation for mitigating membership inference, our results show that achieving both high utility and strong privacy in federated learning remains an open challenge. Bridging this gap will require advances not only in privacy mechanisms, but also in model design, training strategies, and system-level considerations.

Sources

[1] Zhu Gongxi et al. FedMIA: An Effective Membership Inference Attack Exploiting “All for One” Principle in Federated Learning. arXiv preprint arXiv:2402.06289, 2024. arXiv.

[2] Reza Shokri et al. “Membership Inference Attacks Against Machine Learning Models.” In: IEEE Symposium on Security and Privacy, 2017. arXiv.

[3] Milad Nasr, Reza Shokri, and Amir Houmansadr. “Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-Box Inference Attacks Against Centralized and Federated Learning.” In: IEEE Symposium on Security and Privacy, 2019. IEEE Xplore.

[4] Liwei Song et al. “Source Inference Attacks in Federated Learning.” In: IEEE International Conference on Data Mining (ICDM), 2021. IEEE Xplore.

[5] Zheng Xu et al. Subject Membership Inference Attacks in Federated Learning. arXiv preprint arXiv:2206.03317, 2022. arXiv.

[6] Cynthia Dwork. “Differential Privacy.” In: International Colloquium on Automata, Languages and Programming (ICALP), 2006. DOI.

[7] Fu Jie et al. Differentially Private Federated Learning: A Systematic Review. arXiv preprint arXiv:2405.08299, 2024. arXiv.

[8] Cynthia Dwork et al. “Our Data, Ourselves: Privacy via Distributed Noise Generation.” In: Advances in Cryptology – EUROCRYPT 2006. Lecture Notes in Computer Science. St. Petersburg, Russia: Springer, 2006, pp. 486–503.

[9] Martin Abadi et al. “Deep Learning with Differential Privacy.” In: ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016. arXiv.

[10] Meta Platforms, Inc. Opacus: Train PyTorch Models with Differential Privacy. Website.

[11] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” In: Advances in Neural Information Processing Systems (NeurIPS), vol. 25, 2012.

[12] H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. Federated Learning of Deep Networks Using Model Averaging. arXiv preprint arXiv:1602.05629, 2016. arXiv.

[13] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. “Membership Inference Attacks from First Principles.” In: 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE, 2022.

Author: Clara Pichler