文章速览 | 联邦学习 x CVPR'2023 （下）

本文是由白小鱼博主整理的CVPR 2023会议中，与联邦学习相关的论文合集及摘要翻译，上一次的整理详情请见文章速览 | 联邦学习 x CVPR'2023 （上）。

• Authors: Ming Li; Qingli Li; Yan Wang

• Conference: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url:https://openaccess.thecvf.com/content/CVPR2023/html/Li_Class_Balanced_Adaptive_Pseudo_Labeling_for_Federated_Semi-Supervised_Learning_CVPR_2023_paper.html

Abstract: This paper focuses on federated semi-supervised learning (FSSL), assuming that few clients have fully labeled data (labeled clients) and the training datasets in other clients are fully unlabeled (unlabeled clients). Existing methods attempt to deal with the challenges caused by not independent and identically distributed data (Non-IID) setting. Though methods such as sub-consensus models have been proposed, they usually adopt standard pseudo labeling or consistency regularization on unlabeled clients which can be easily influenced by imbalanced class distribution. Thus, problems in FSSL are still yet to be solved. To seek for a fundamental solution to this problem, we present Class Balanced Adaptive Pseudo Labeling (CBAFed), to study FSSL from the perspective of pseudo labeling. In CBAFed, the first key element is a fixed pseudo labeling strategy to handle the catastrophic forgetting problem, where we keep a fixed set by letting pass information of unlabeled data at the beginning of the unlabeled client training in each communication round. The second key element is that we design class balanced adaptive thresholds via considering the empirical distribution of all training data in local clients, to encourage a balanced training process. To make the model reach a better optimum, we further propose a residual weight connection in local supervised training and global model aggregation. Extensive experiments on five datasets demonstrate the superiority of CBAFed. Code will be released.

abstractTranslation: 本文侧重于联邦半监督学习（FSSL），假设很少有客户端具有完全标记的数据（标记的客户端）并且其他客户端中的训练数据集完全未标记（未标记的客户端）。现有方法试图应对非独立同分布数据（Non-IID）设置带来的挑战。虽然已经提出了子共识模型等方法，但它们通常对未标记的客户端采用标准伪标记或一致性正则化，这些客户端很容易受到不平衡类分布的影响。因此，FSSL 中的问题仍然有待解决。为了寻求解决这个问题的根本方法，我们提出了类平衡自适应伪标签（CBAFed），从伪标签的角度研究 FSSL。在 CBAFed 中，第一个关键要素是用于处理灾难性遗忘问题的固定伪标记策略，我们通过在每一轮通信中未标记的客户端训练开始时让未标记数据的信息传递来保持固定集。第二个关键要素是我们通过考虑本地客户端所有训练数据的经验分布来设计类平衡自适应阈值，以鼓励平衡的训练过程。为了使模型达到更好的最优，我们进一步提出了局部监督训练和全局模型聚合中的残差权重连接。对五个数据集的大量实验证明了 CBAFed 的优越性。代码将被释放。

Notes: https://github.com/minglllli/CBAFed

• Authors: Bo Li; Mikkel N. Schmidt; Tommy S. Alstrøm; Sebastian U. Stich

• Conference : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url:https://openaccess.thecvf.com/content/CVPR2023/html/Li_On_the_Effectiveness_of_Partial_Variance_Reduction_in_Federated_Learning_CVPR_2023_paper.html

Abstract: Data heterogeneity across clients is a key challenge in federated learning. Prior works address this by either aligning client and server models or using control variates to correct client model drift. Although these methods achieve fast convergence in convex or simple non-convex problems, the performance in over-parameterized models such as deep neural networks is lacking. In this paper, we first revisit the widely used FedAvg algorithm in a deep neural network to understand how data heterogeneity influences the gradient updates across the neural network layers. We observe that while the feature extraction layers are learned efficiently by FedAvg, the substantial diversity of the final classification layers across clients impedes the performance. Motivated by this, we propose to correct model drift by variance reduction only on the final layers. We demonstrate that this significantly outperforms existing benchmarks at a similar or lower communication cost. We furthermore provide proof for the convergence rate of our algorithm.

abstractTranslation: 跨客户端的数据异构性是联邦学习中的一个关键挑战。先前的工作通过对齐客户端和服务器模型或使用控制变量来纠正客户端模型漂移来解决这个问题。虽然这些方法在凸或简单的非凸问题中实现了快速收敛，但在深度神经网络等过度参数化模型中的性能不足。在本文中，我们首先回顾了深度神经网络中广泛使用的 FedAvg 算法，以了解数据异质性如何影响神经网络各层的梯度更新。我们观察到，虽然 FedAvg 有效地学习了特征提取层，但客户端最终分类层的实质性差异阻碍了性能。受此启发，我们建议仅在最后一层通过方差减少来校正模型漂移。我们证明，这在类似或更低的通信成本下显着优于现有基准。我们进一步证明了我们算法的收敛速度。

Notes: [PDF] (https://arxiv.org/abs/2212.02191)

• Authors: SangMook Kim; Sangmin Bae; Hwanjun Song; Se-Young Yun

• Conference : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url: https://openaccess.thecvf.com/content/CVPR2023/html/Kim_Re-Thinking_Federated_Active_Learning_Based_on_Inter-Class_Diversity_CVPR_2023_paper.html

Abstract: Although federated learning has made awe-inspiring advances, most studies have assumed that the client's data are fully labeled. However, in a real-world scenario, every client may have a significant amount of unlabeled instances. Among the various approaches to utilizing unlabeled data, a federated active learning framework has emerged as a promising solution. In the decentralized setting, there are two types of available query selector models, namely 'global' and 'local-only' models, but little literature discusses their performance dominance and its causes. In this work, we first demonstrate that the superiority of two selector models depends on the global and local inter-class diversity. Furthermore, we observe that the global and local-only models are the keys to resolving the imbalance of each side. Based on our findings, we propose LoGo, a FAL sampling strategy robust to varying local heterogeneity levels and global imbalance ratio, that integrates both models by two steps of active selection scheme. LoGo consistently outperforms six active learning strategies in the total number of 38 experimental settings.

Notes: 

[pdf](http://arxiv.org/abs/2303.12317) 

[code](https://github.com/raymin0223/LoGo)

• Authors: Meirui Jiang; Holger R. Roth; Wenqi Li; Dong Yang; Can Zhao; Vishwesh Nath; Daguang Xu; Qi Dou; Ziyue Xu

• Conference : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url: https://openaccess.thecvf.com/content/CVPR2023/html/Jiang_Fair_Federated_Medical_Image_Segmentation_via_Client_Contribution_Estimation_CVPR_2023_paper.html

• Authors: Fatih Ilhan; Gong Su; Ling Liu

• Conference : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url: https://openaccess.thecvf.com/content/CVPR2023/html/Ilhan_ScaleFL_Resource-Adaptive_Federated_Learning_With_Heterogeneous_Clients_CVPR_2023_paper.html

Abstract: Federated learning (FL) is an attractive distributed learning paradigm supporting real-time continuous learning and client privacy by default. In most FL approaches, all edge clients are assumed to have sufficient computation capabilities to participate in the learning of a deep neural network (DNN) model. However, in real-life applications, some clients may have severely limited resources and can only train a much smaller local model. This paper presents ScaleFL, a novel FL approach with two distinctive mechanisms to handle resource heterogeneity and provide an equitable FL framework for all clients. First, ScaleFL adaptively scales down the DNN model along width and depth dimensions by leveraging early exits to find the best-fit models for resource-aware local training on distributed clients. In this way, ScaleFL provides an efficient balance of preserving basic and complex features in local model splits with various sizes for joint training while enabling fast inference for model deployment. Second, ScaleFL utilizes self-distillation among exit predictions during training to improve aggregation through knowledge transfer among subnetworks. We conduct extensive experiments on benchmark CV (CIFAR-10/100, ImageNet) and NLP datasets (SST-2, AgNews). We demonstrate that ScaleFL outperforms existing representative heterogeneous FL approaches in terms of global/local model performance and provides inference efficiency, with up to 2x latency and 4x model size reduction with negligible performance drop below 2%.

abstractTranslation: 联邦学习 (FL) 是一种有吸引力的分布式学习范式，默认支持实时持续学习和客户端隐私。在大多数 FL 方法中，假定所有边缘客户端都具有足够的计算能力来参与深度神经网络 (DNN) 模型的学习。然而，在实际应用中，一些客户端的资源可能非常有限，只能训练一个小得多的本地模型。本文介绍了 ScaleFL，这是一种新颖的 FL 方法，具有两种独特的机制来处理资源异质性并为所有客户端提供公平的 FL 框架。首先，ScaleFL 通过利用早期退出找到最适合分布式客户端资源感知本地训练的模型，沿宽度和深度维度自适应地缩小 DNN 模型。通过这种方式，ScaleFL 提供了一种有效的平衡，既可以在具有各种大小的局部模型分割中保留基本特征和复杂特征以进行联邦训练，又可以为模型部署实现快速推理。其次，ScaleFL 在训练期间利用退出预测之间的自蒸馏，通过子网络之间的知识转移来改进聚合。我们对基准 CV（CIFAR-10/100、ImageNet）和 NLP 数据集（SST-2、AgNews）进行了大量实验。我们证明，ScaleFL 在全局/局部模型性能方面优于现有的代表性异构 FL 方法，并提供推理效率，延迟高达 2 倍，模型大小减少 4 倍，性能下降可忽略不计，低于 2%。

Notes:

[code] (https://github.com/git-disl/scale-fl)

• Authors: Wenke Huang; Mang Ye; Zekun Shi; He Li; Bo Du

• Conference : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url:https://openaccess.thecvf.com/content/CVPR2023/html/Huang_Rethinking_Federated_Learning_With_Domain_Shift_A_Prototype_View_CVPR_2023_paper.html

Abstract: Federated learning shows a bright promise as a privacy-preserving collaborative learning technique. However, prevalent solutions mainly focus on all private data sampled from the same domain. An important challenge is that when distributed data are derived from diverse domains. The private model presents degenerative performance on other domains (with domain shift). Therefore, we expect that the global model optimized after the federated learning process stably provides generalizability performance on multiple domains. In this paper, we propose Federated Prototypes Learning (FPL) for federated learning under domain shift. The core idea is to construct cluster prototypes and unbiased prototypes, providing fruitful domain knowledge and a fair convergent target. On the one hand, we pull the sample embedding closer to cluster prototypes belonging to the same semantics than cluster prototypes from distinct classes. On the other hand, we introduce consistency regularization to align the local instance with the respective unbiased prototype. Empirical results on Digits and Office Caltech tasks demonstrate the effectiveness of the proposed solution and the efficiency of crucial modules.

abstractTranslation: 联邦学习作为一种保护隐私的协作学习技术显示出光明的前景。然而，流行的解决方案主要关注从同一域采样的所有私有数据。一个重要的挑战是当分布式数据来自不同的域时。私有模型在其他域上呈现退化性能（域转移）。因此，我们期望联邦学习过程后优化的全局模型在多个领域稳定地提供泛化性能。在本文中，我们提出了联邦原型学习（FPL）用于域转移下的联邦学习。核心思想是构建集群原型和无偏原型，提供丰富的领域知识和公平的收敛目标。一方面，与来自不同类的聚类原型相比，我们将样本嵌入更接近属于相同语义的聚类原型。另一方面，我们引入一致性正则化以使本地实例与相应的无偏原型对齐。Digits 和 Office Caltech 任务的实证结果证明了所提出解决方案的有效性和关键模块的效率。

Notes: https://github.com/WenkeHuang/RethinkFL

• Authors: Chun-Mei Feng; Bangjun Li; Xinxing Xu; Yong Liu; Huazhu Fu; Wangmeng Zuo

• Conference : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

• Url:https://openaccess.thecvf.com/content/CVPR2023/html/Feng_Learning_Federated_Visual_Prompt_in_Null_Space_for_MRI_Reconstruction_CVPR_2023_paper.html

Abstract: Federated Magnetic Resonance Imaging (MRI) reconstruction enables multiple hospitals to collaborate distributedly without aggregating local data, thereby protecting patient privacy. However, the data heterogeneity caused by different MRI protocols, insufficient local training data, and limited communication bandwidth inevitably impair global model convergence and updating. In this paper, we propose a new algorithm, FedPR, to learn federated visual prompts in the null space of global prompt for MRI reconstruction. FedPR is a new federated paradigm that adopts a powerful pre-trained model while only learning and communicating the prompts with few learnable parameters, thereby significantly reducing communication costs and achieving competitive performance on limited local data. Moreover, to deal with catastrophic forgetting caused by data heterogeneity, FedPR also updates efficient federated visual prompts that project the local prompts into an approximate null space of the global prompt, thereby suppressing the interference of gradients on the server performance. Extensive experiments on federated MRI show that FedPR significantly outperforms state-of-the-art FL algorithms with < 6% of communication costs when given the limited amount of local data.

abstractTranslation: 联邦磁共振成像 (MRI) 重建使多家医院能够在不聚合本地数据的情况下进行分布式协作，从而保护患者隐私。然而，不同 MRI 协议导致的数据异构性、局部训练数据不足和通信带宽有限不可避免地会影响全局模型的收敛和更新。在本文中，我们提出了一种新算法 FedPR，用于在 MRI 重建的全局提示的零空间中学习联邦视觉提示。FedPR 是一种新的联邦范式，它采用强大的预训练模型，同时仅学习和交流具有少量可学习参数的提示，从而显着降低通信成本并在有限的本地数据上实现有竞争力的性能。此外，为了应对数据异构性导致的灾难性遗忘，FedPR 还更新了高效的联邦视觉提示，将本地提示投影到全局提示的近似零空间中，从而抑制梯度对服务器性能的干扰。联邦 MRI 的大量实验表明，在给定有限数量的本地数据时，FedPR 的通信成本明显优于最先进的 FL 算法，通信成本低于 6%。

Notes:

[pdf](http://arxiv.org/abs/2303.16181) 

[code](https://github.com/chunmeifeng/FedPR)

Abstract:

abstractTranslation: 基于联邦学习的语义分割（FSS）通过对本地客户端的分散培训引起了广泛关注。然而，大多数 FSS 模型假设类别是预先固定的，因此在实际应用中本地客户端逐渐接收新类别而没有内存存储来访问旧类别时，会严重遗忘旧类别。此外，收集新课程的新客户可能会加入 FSS 的全球培训，这进一步加剧了灾难性遗忘。为了克服上述挑战，我们提出了一种遗忘平衡学习 (FBL) 模型，从客户端内和客户端间两个方面解决旧类的异构遗忘问题。具体来说，在通过自适应类平衡伪标签生成的伪标签的指导下，我们开发了遗忘平衡语义补偿损失和遗忘平衡关系一致性损失，以纠正具有背景偏移的旧类别的客户端内异构遗忘。它在本地客户端中执行平衡梯度传播和关系一致性蒸馏。此外，为了从客户端间的角度解决异构遗忘问题，我们提出了一个任务转换监视器。它可以识别隐私保护下的新类，并存储最新的旧全局模型以进行关系蒸馏。定性实验表明我们的模型相对于比较方法有了很大的改进。代码可在 https://github.com/JiahuaDong/FISS 获取。

Abstract: Federated Learning (FL) has emerged as a de facto machine learning area and received rapid increasing research interests from the community. However, catastrophic forgetting caused by data heterogeneity and partial participation poses distinctive challenges for FL, which are detrimental to the performance. To tackle the problems, we propose a new FL approach (namely GradMA), which takes inspiration from continual learning to simultaneously correct the server-side and worker-side update directions as well as take full advantage of server's rich computing and memory resources. Furthermore, we elaborate a memory reduction strategy to enable GradMA to accommodate FL with a large scale of workers. We then analyze convergence of GradMA theoretically under the smooth non-convex setting and show that its convergence rate achieves a linear speed up w.r.t the increasing number of sampled active workers. At last, our extensive experiments on various image classification tasks show that GradMA achieves significant performance gains in accuracy and communication efficiency compared to SOTA baselines. We provide our code here: https://github.com/lkyddd/GradMA.

abstractTranslation: 联邦学习 (FL) 已成为事实上的机器学习领域，并受到社区快速增长的研究兴趣。然而，由数据异质性和部分参与引起的灾难性遗忘给 FL 带来了独特的挑战，这对性能是不利的。为了解决这些问题，我们提出了一种新的 FL 方法（即 GradMA），它从持续学习中汲取灵感，同时纠正服务器端和工作端更新方向，并充分利用服务器丰富的计算和内存资源。此外，我们详细阐述了一种内存减少策略，使 GradMA 能够适应具有大量 worker 的 FL。然后，我们从理论上分析了 GradMA 在平滑非凸设置下的收敛性，并表明其收敛速度实现了线性加速 w.r.t 随着采样活跃工人数量的增加。最后，我们对各种图像分类任务的广泛实验表明，与 SOTA 基线相比，GradMA 在准确性和通信效率方面取得了显着的性能提升。我们在这里提供我们的代码：https://github.com/lkyddd/GradMA

Notes:

[pdf] (http://arxiv.org/abs/2302.14307) 

[code] (https://github.com/lkyddd/gradma)

Abstract: Federated learning enables the privacy-preserving training of neural network models using real-world data across distributed clients. FedAvg has become the preferred optimizer for federated learning because of its simplicity and effectiveness. FedAvg uses naive aggregation to update the server model, interpolating client models based on the number of instances used in their training. However, naive aggregation suffers from client-drift when the data is heterogenous (non-IID), leading to unstable and slow convergence. In this work, we propose a novel aggregation approach, elastic aggregation, to overcome these issues. Elastic aggregation interpolates client models adaptively according to parameter sensitivity, which is measured by computing how much the overall prediction function output changes when each parameter is changed. This measurement is performed in an unsupervised and online manner. Elastic aggregation reduces the magnitudes of updates to the more sensitive parameters so as to prevent the server model from drifting to any one client distribution, and conversely boosts updates to the less sensitive parameters to better explore different client distributions. Empirical results on real and synthetic data as well as analytical results show that elastic aggregation leads to efficient training in both convex and non-convex settings, while being fully agnostic to client heterogeneity and robust to large numbers of clients, partial participation, and imbalanced data. Finally, elastic aggregation works well with other federated optimizers and achieves significant improvements across the board.

abstractTranslation: 联邦学习可以使用跨分布式客户端的真实数据对神经网络模型进行隐私保护训练。FedAvg 因其简单有效而成为联邦学习的首选优化器。FedAvg 使用朴素聚合来更新服务器模型，根据训练中使用的实例数量对客户端模型进行插值。然而，当数据是异构的（非独立同分布）时，朴素聚合会受到客户端漂移的影响，从而导致不稳定和缓慢的收敛。在这项工作中，我们提出了一种新的聚合方法，即弹性聚合，以克服这些问题。弹性聚合根据参数敏感度自适应地插值客户端模型，参数敏感度通过计算每个参数变化时整体预测函数输出的变化量来衡量。该测量以无人监督的在线方式进行。弹性聚合减少了对更敏感参数的更新幅度，以防止服务器模型漂移到任何一个客户端分布，并反过来增加对不太敏感参数的更新，以更好地探索不同的客户端分布。真实数据和合成数据的实证结果以及分析结果表明，弹性聚合可以在凸和非凸设置中实现高效训练，同时完全不可知客户端异质性，并且对大量客户端、部分参与和不平衡数据具有鲁棒性.最后，弹性聚合与其他联邦优化器配合良好，并在各方面实现了显着改进。

作者: 白小鱼（上海交通大学计算机系博士生）