DPlis: Boosting Utility of Differentially Private Deep Learning via Randomized Smoothing
Authors: Wenxiao Wang (Tsinghua University.), Tianhao Wang (Harvard University.), Lun Wang (University of California, Berkeley.), Nanqing Luo (Huazhong University of Science and Technology.), Pan Zhou (Huazhong University of Science and Technology.), Dawn Song (University of California, Berkeley.), Ruoxi Jia (Virginia Tech.)
Volume: 2021
Issue: 4
Pages: 163–183
DOI: https://doi.org/10.2478/popets-2021-0065
Abstract: Deep learning techniques have achieved remarkable performance in wide-ranging tasks. However, when trained on privacy-sensitive datasets, the model parameters may expose private information in training data. Prior attempts for differentially private training, although offering rigorous privacy guarantees, lead to much lower model performance than the non-private ones. Besides, different runs of the same training algorithm produce models with large performance variance. To address these issues, we propose DPlis– Differentially Private Learning wIth Smoothing. The core idea of DPlis is to construct a smooth loss function that favors noise-resilient models lying in large flat regions of the loss landscape. We provide theoretical justification for the utility improvements of DPlis. Extensive experiments also demonstrate that DPlis can effectively boost model quality and training stability under a given privacy budget.
Keywords: differential privacy, stochastic gradient descent, deep learning, randomized smoothing
Copyright in PoPETs articles are held by their authors. This article is published under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 license.
