文章速览 | 联邦学习 x AAAI'2023 （上）

本文是由白小鱼博主整理的AAAI 2023会议中，与联邦学习相关的论文合集及摘要翻译。

• Authors: Jie Zhang; Bo Li; Chen Chen; Lingjuan Lyu; Shuang Wu; Shouhong Ding; Chao Wu

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26331

Abstract: In Federated Learning (FL), models are as fragile as centrally trained models against adversarial examples. However, the adversarial robustness of federated learning remains largely unexplored. This paper casts light on the challenge of adversarial robustness of federated learning. To facilitate a better understanding of the adversarial vulnerability of the existing FL methods, we conduct comprehensive robustness evaluations on various attacks and adversarial training methods. Moreover, we reveal the negative impacts induced by directly adopting adversarial training in FL, which seriously hurts the test accuracy, especially in non-IID settings. In this work, we propose a novel algorithm called Decision Boundary based Federated Adversarial Training (DBFAT), which consists of two components (local re-weighting and global regularization) to improve both accuracy and robustness of FL systems. Extensive experiments on multiple datasets demonstrate that DBFAT consistently outperforms other baselines under both IID and non-IID settings.

abstractTranslation: 在联邦学习（FL）中，模型与针对对抗性示例的集中训练模型一样脆弱。然而，联邦学习的对抗鲁棒性在很大程度上仍未得到探索。本文揭示了联邦学习的对抗鲁棒性挑战。为了更好地理解现有 FL 方法的对抗漏洞，我们对各种攻击和对抗训练方法进行了全面的鲁棒性评估。此外，我们还揭示了在 FL 中直接采用对抗性训练所带来的负面影响，这严重损害了测试准确性，尤其是在非独立同分布环境中。在这项工作中，我们提出了一种称为基于决策边界的联邦对抗训练（DBFAT）的新颖算法，它由两个部分组成（局部重新加权和全局正则化），以提高 FL 系统的准确性和鲁棒性。对多个数据集的大量实验表明，在 IID 和非 IID 设置下，DBFAT 始终优于其他基线。

Notes:

[PDF](https://arxiv.org/abs/2302.09479)

• Authors: Jianqing Zhang; Yang Hua; Hao Wang; Tao Song; Zhengui Xue; Ruhui Ma; Haibing Guan

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26330

Abstract: A key challenge in federated learning (FL) is the statistical heterogeneity that impairs the generalization of the global model on each client. To address this, we propose a method Federated learning with Adaptive Local Aggregation (FedALA) by capturing the desired information in the global model for client models in personalized FL. The key component of FedALA is an Adaptive Local Aggregation (ALA) module, which can adaptively aggregate the downloaded global model and local model towards the local objective on each client to initialize the local model before training in each iteration. To evaluate the effectiveness of FedALA, we conduct extensive experiments with five benchmark datasets in computer vision and natural language processing domains. FedALA outperforms eleven state-of-the-art baselines by up to 3.27% in test accuracy. Furthermore, we also apply ALA module to other federated learning methods and achieve up to 24.19% improvement in test accuracy. Code is available at https://github.com/TsingZ0/FedALA.

abstractTranslation: 联邦学习 (FL) 的一个关键挑战是统计异质性，它会损害每个客户端的全局模型的泛化能力。为了解决这个问题，我们提出了一种采用自适应本地聚合（FedALA）的联邦学习方法，通过捕获个性化 FL 中客户端模型的全局模型中所需的信息。FedALA 的关键组件是自适应本地聚合（ALA）模块，它可以针对每个客户端上的本地目标自适应聚合下载的全局模型和本地模型，以便在每次迭代训练之前初始化本地模型。为了评估 FedALA 的有效性，我们使用计算机视觉和自然语言处理领域的五个基准数据集进行了广泛的实验。FedALA 的测试准确度比 11 个最先进的基线高出 3.27%。此外，我们还将ALA模块应用到其他联邦学习方法中，测试精度提高了24.19%。代码可在 https://github.com/TsingZ0/FedALA 获取。

Notes:

[PDF](https://arxiv.org/abs/2212.01197) 

[code](https://github.com/tsingz0/fedala)

• Authors: Yang Yu; Qi Liu; Likang Wu; Runlong Yu; Sanshi Lei Yu; Zaixi Zhang

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/25611

Abstract: Federated recommendation (FedRec) can train personalized recommenders without collecting user data, but the decentralized nature makes it susceptible to poisoning attacks. Most previous studies focus on the targeted attack to promote certain items, while the untargeted attack that aims to degrade the overall performance of the FedRec system remains less explored. In fact, untargeted attacks can disrupt the user experience and bring severe financial loss to the service provider. However, existing untargeted attack methods are either inapplicable or ineffective against FedRec systems. In this paper, we delve into the untargeted attack and its defense for FedRec systems. (i) We propose ClusterAttack, a novel untargeted attack method. It uploads poisonous gradients that converge the item embeddings into several dense clusters, which make the recommender generate similar scores for these items in the same cluster and perturb the ranking order. (ii) We propose a uniformity-based defense mechanism (UNION) to protect FedRec systems from such attacks. We design a contrastive learning task that regularizes the item embeddings toward a uniform distribution. Then the server filters out these malicious gradients by estimating the uniformity of updated item embeddings. Experiments on two public datasets show that ClusterAttack can effectively degrade the performance of FedRec systems while circumventing many defense methods, and UNION can improve the resistance of the system against various untargeted attacks, including our ClusterAttack.

abstractTranslation: 联邦推荐（FedRec）可以在不收集用户数据的情况下训练个性化推荐器，但去中心化的性质使其容易受到中毒攻击。之前的大多数研究都集中在促进某些项目的有针对性的攻击上，而旨在降低 FedRec 系统整体性能的非针对性攻击仍然很少被探讨。事实上，无针对性的攻击可能会破坏用户体验并给服务提供商带来严重的经济损失。然而，现有的非针对性攻击方法对于 FedRec 系统要么不适用，要么无效。在本文中，我们深入研究了 FedRec 系统的非针对性攻击及其防御。(i)我们提出ClusterAttack，一种新颖的非目标攻击方法。它上传有毒梯度，将项目嵌入收敛到几个密集的集群中，这使得推荐器为同一集群中的这些项目生成相似的分数，并扰乱排名顺序。(ii) 我们提出了一种基于一致性的防御机制（UNION）来保护 FedRec 系统免受此类攻击。我们设计了一个对比学习任务，将项目嵌入规范化为均匀分布。然后，服务器通过估计更新项目嵌入的均匀性来过滤掉这些恶意梯度。在两个公共数据集上的实验表明，ClusterAttack可以有效降低FedRec系统的性能，同时规避许多防御方法，而UNION可以提高系统对各种非针对性攻击的抵抗力，包括我们的ClusterAttack。

Notes:

[PDF](https://arxiv.org/abs/2212.05399) 

[code](https://github.com/yflyl613/fedrec)

• Authors: Gang Yan; Hao Wang; Xu Yuan; Jian Li

• Journal : Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26271

• Authors: Zuobin Xiong; Wei Li; Zhipeng Cai

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26252

Abstract: The study of generative models is a promising branch of deep learning techniques, which has been successfully applied to different scenarios, such as Artificial Intelligence and the Internet of Things. While in most of the existing works, the generative models are realized as a centralized structure, raising the threats of security and privacy and the overburden of communication costs. Rare efforts have been committed to investigating distributed generative models, especially when the training data comes from multiple heterogeneous sources under realistic IoT settings. In this paper, to handle this challenging problem, we design a federated generative model framework that can learn a powerful generator for the hierarchical IoT systems. Particularly, our generative model framework can solve the problem of distributed data generation on multi-source heterogeneous data in two scenarios, i.e., feature related scenario and label related scenario. In addition, in our federated generative models, we develop a synchronous and an asynchronous updating methods to satisfy different application requirements. Extensive experiments on a simulated dataset and multiple real datasets are conducted to evaluate the data generation performance of our proposed generative models through comparison with the state-of-the-arts.

abstractTranslation: 生成模型的研究是深度学习技术的一个有前途的分支，已成功应用于人工智能和物联网等不同场景。而在大多数现有工作中，生成模型都是作为中心化结构实现的，这增加了安全和隐私的威胁以及通信成本的负担。人们很少致力于研究分布式生成模型，特别是当训练数据来自现实物联网设置下的多个异构源时。在本文中，为了解决这个具有挑战性的问题，我们设计了一个联邦生成模型框架，可以为分层物联网系统学习强大的生成器。特别是，我们的生成模型框架可以解决特征相关场景和标签相关场景两种场景下多源异构数据的分布式数据生成问题。此外，在我们的联邦生成模型中，我们开发了同步和异步更新方法来满足不同的应用需求。对模拟数据集和多个真实数据集进行了广泛的实验，通过与最先进的技术进行比较来评估我们提出的生成模型的数据生成性能。

• Authors: Jiahao Xie; Chao Zhang; Zebang Shen; Weijie Liu; Hui Qian

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26246

Abstract: Minimax problems arise in a wide range of important applications including robust adversarial learning and Generative Adversarial Network (GAN) training. Recently, algorithms for minimax problems in the Federated Learning (FL) paradigm have received considerable interest. Existing federated algorithms for general minimax problems require the full aggregation (i.e., aggregation of local model information from all clients) in each training round. Thus, they are inapplicable to an important setting of FL known as the cross-device setting, which involves numerous unreliable mobile/IoT devices. In this paper, we develop the first practical algorithm named CDMA for general minimax problems in the cross-device FL setting. CDMA is based on a Start-Immediately-With-Enough-Responses mechanism, in which the server first signals a subset of clients to perform local computation and then starts to aggregate the local results reported by clients once it receives responses from enough clients in each round. With this mechanism, CDMA is resilient to the low client availability. In addition, CDMA is incorporated with a lightweight global correction in the local update steps of clients, which mitigates the impact of slow network connections. We establish theoretical guarantees of CDMA under different choices of hyperparameters and conduct experiments on AUC maximization, robust adversarial network training, and GAN training tasks. Theoretical and experimental results demonstrate the efficiency of CDMA.

abstractTranslation: 极小极大问题出现在广泛的重要应用中，包括鲁棒的对抗性学习和生成对抗性网络（GAN）训练。最近，联邦学习（FL）范式中的极小极大问题的算法引起了相当大的兴趣。针对一般极小极大问题的现有联邦算法需要在每轮训练中进行完全聚合（即来自所有客户端的本地模型信息的聚合）。因此，它们不适用于 FL 的重要设置（称为跨设备设置），该设置涉及大量不可靠的移动/物联网设备。在本文中，我们针对跨设备 FL 设置中的一般极小极大问题开发了第一个名为 CDMA 的实用算法。CDMA 基于“立即启动足够的响应”机制，其中服务器首先向客户端的子集发出信号以执行本地计算，然后一旦收到每个客户端中足够多的客户端的响应，就开始聚合客户端报告的本地结果。圆形的。通过这种机制，CDMA 能够适应低客户端可用性。此外，CDMA在客户端本地更新步骤中加入了轻量级全局校正，从而减轻了网络连接速度慢的影响。我们在不同超参数选择下建立了 CDMA 的理论保证，并对 AUC 最大化、鲁棒对抗网络训练和 GAN 训练任务进行了实验。理论和实验结果证明了CDMA 的效率。

Notes:

[PDF](https://arxiv.org/abs/2105.14216) 

[CODE](https://github.com/xjiajiahao/federated-minimax)

• Authors: Peng Xiao; Samuel Cheng

• Journal: Proceedings of the AAAI Conference on Artificial Intelligence

• Url: https://ojs.aaai.org/index.php/AAAI/article/view/26245

Abstract: Federated learning is a contemporary machine learning paradigm where locally trained models are distilled into a global model. Due to the intrinsic permutation invariance of neural networks, Probabilistic Federated Neural Matching (PFNM) employs a Bayesian nonparametric framework in the generation process of local neurons, and then creates a linear sum assignment formulation in each alternative optimization iteration. But according to our theoretical analysis, the optimization iteration in PFNM omits global information from existing. In this study, we propose a novel approach that overcomes this flaw by introducing a Kullback-Leibler divergence penalty at each iteration. The effectiveness of our approach is demonstrated by experiments on both image classification and semantic segmentation tasks.

abstractTranslation: 联邦学习是一种当代机器学习范式，其中本地训练的模型被提炼为全局模型。由于神经网络固有的排列不变性，概率联邦神经匹配（PFNM）在局部神经元的生成过程中采用贝叶斯非参数框架，然后在每次替代优化迭代中创建线性和分配公式。但根据我们的理论分析，PFNM 中的优化迭代忽略了现有的全局信息。在这项研究中，我们提出了一种新颖的方法，通过在每次迭代中引入 Kullback-Leibler 散度惩罚来克服这一缺陷。我们的方法的有效性通过图像分类和语义分割任务的实验得到了证明。

Abstract:

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, the federated learning in practice still faces numerous challenges, such as the large training iterations to converge since the sizes of models and datasets keep increasing, and the lack of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive methods in federated learning is scarce and existing works either lack a complete theoretical convergence guarantee or have slow sample complexity. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on the momentum-based variance reduced technique in cross-silo FL. We first explore how to design the adaptive algorithm in the FL setting. By providing a counter-example, we prove that a simple combination of FL and adaptive methods could lead to divergence. More importantly, we provide a convergence analysis for our method and prove that our algorithm is the first adaptive FL algorithm to reach the best-known samples O(epsilon(-3)) and O(epsilon(-2)) communication rounds to find an epsilon-stationary point without large batches. The experimental results on the language modeling task and image classification task with heterogeneous data demonstrate the efficiency of our algorithms.

abstractTranslation: 随着分布式数据的出现，联邦学习越来越受到人们的关注。虽然针对非凸分布式问题提出了广泛的联邦学习算法，但联邦学习在实践中仍然面临着许多挑战，例如由于模型和数据集的大小不断增加而需要大量的训练迭代才能收敛，以及联邦学习缺乏适应性。基于 SGD 的模型更新。同时，联邦学习中自适应方法的研究很少，现有的工作要么缺乏完整的理论收敛保证，要么样本复杂度较低。在本文中，我们提出了一种基于跨筒仓 FL 中基于动量的方差减少技术的高效自适应算法（即 FAFED）。我们首先探讨如何在 FL 设置中设计自适应算法。通过提供一个反例，我们证明了 FL 和自适应方法的简单组合可能会导致发散。更重要的是，我们为我们的方法提供了收敛分析，并证明我们的算法是第一个达到最著名样本 O(epsilon(-3)) 和 O(epsilon(-2)) 通信轮数的自适应 FL 算法，以找到没有大批量的 epsilon 驻点。在异构数据的语言建模任务和图像分类任务上的实验结果证明了我们算法的效率。

Notes:

[PDF](https://arxiv.org/abs/2212.00974)

Abstract: As a novel distributed learning paradigm, federated learning (FL) faces serious challenges in dealing with massive clients with heterogeneous data distribution and computation and communication resources. Various client-variance-reduction schemes and client sampling strategies have been respectively introduced to improve the robustness of FL. Among others, primal-dual algorithms such as the alternating direction of method multipliers (ADMM) have been found being resilient to data distribution and outperform most of the primal-only FL algorithms. However, the reason behind remains a mystery still. In this paper, we firstly reveal the fact that the federated ADMM is essentially a client-variance-reduced algorithm. While this explains the inherent robustness of federated ADMM, the vanilla version of it lacks the ability to be adaptive to the degree of client heterogeneity. Besides, the global model at the server under client sampling is biased which slows down the practical convergence. To go beyond ADMM, we propose a novel primal-dual FL algorithm, termed FedVRA, that allows one to adaptively control the variance-reduction level and biasness of the global model. In addition, FedVRA unifies several representative FL algorithms in the sense that they are either special instances of FedVRA or are close to it. Extensions of FedVRA to semi/un-supervised learning are also presented. Experiments based on (semi-)supervised image classification tasks demonstrate superiority of FedVRA over the existing schemes in learning scenarios with massive heterogeneous clients and client sampling.

abstractTranslation: 作为一种新颖的分布式学习范式，联邦学习（FL）在处理具有异构数据分布以及计算和通信资源的海量客户端时面临着严峻的挑战。分别引入了各种客户端方差减少方案和客户端采样策略来提高 FL 的鲁棒性。其中，方法乘数交替方向 (ADMM) 等原始对偶算法被发现对数据分布具有弹性，并且优于大多数仅原始 FL 算法。然而，背后的原因仍然是个谜。在本文中，我们首先揭示了联邦 ADMM 本质上是一种客户端方差减少算法。虽然这解释了联邦 ADMM 固有的稳健性，但它的普通版本缺乏适应客户端异构程度的能力。此外，客户端采样下服务器端的全局模型存在偏差，这会减慢实际收敛速度。为了超越 ADMM，我们提出了一种新颖的原始对偶 FL 算法，称为 FedVRA，它允许人们自适应地控制全局模型的方差减少水平和偏差。此外，FedVRA 统一了几种具有代表性的 FL 算法，因为它们要么是 FedVRA 的特殊实例，要么与其接近。还介绍了 FedVRA 对半监督/无监督学习的扩展。基于（半）监督图像分类任务的实验证明了 FedVRA 在具有大量异构客户端和客户端采样的学习场景中相对于现有方案的优越性。

Notes:

[pdf](https://arxiv.org/abs/2212.01519)

Abstract: Federated learning aims to collaboratively train models without accessing their client's local private data. The data may be Non-IID for different clients and thus resulting in poor performance. Recently, personalized federated learning (PFL) has achieved great success in handling Non-IID data by enforcing regularization in local optimization or improving the model aggregation scheme on the server. However, most of the PFL approaches do not take into account the unfair competition issue caused by the imbalanced data distribution and lack of positive samples for some classes in each client. To address this issue, we propose a novel and generic PFL framework termed Federated Averaging via Binary Classification, dubbed FedABC. In particular, we adopt the ``one-vs-all'' training strategy in each client to alleviate the unfair competition between classes by constructing a personalized binary classification problem for each class. This may aggravate the class imbalance challenge and thus a novel personalized binary classification loss that incorporates both the under-sampling and hard sample mining strategies is designed. Extensive experiments are conducted on two popular datasets under different settings, and the results demonstrate that our FedABC can significantly outperform the existing counterparts.

abstractTranslation: 联邦学习旨在协作训练模型，而无需访问客户的本地私有数据。对于不同的客户端，数据可能是非独立同分布的，从而导致性能不佳。最近，个性化联邦学习（PFL）通过在本地优化中强制正则化或改进服务器上的模型聚合方案，在处理非独立同分布数据方面取得了巨大成功。然而，大多数PFL方法没有考虑到由于数据分布不平衡以及每个客户端中某些类别缺乏正样本而导致的不公平竞争问题。为了解决这个问题，我们提出了一种新颖且通用的 PFL 框架，称为通过二元分类进行联邦平均，称为 FedABC。特别是，我们在每个客户端中采用“一对一”训练策略，通过为每个类别构建个性化的二元分类问题来缓解类别之间的不公平竞争。这可能会加剧类别不平衡的挑战，因此设计了一种结合了欠采样和硬样本挖掘策略的新颖的个性化二元分类损失。在不同设置下对两个流行的数据集进行了广泛的实验，结果表明我们的 FedABC 可以显着优于现有的同行。

Notes:

[PDF](https://arxiv.org/abs/2208.12294) 

[code](https://github.com/bytedance/fedlearner)