Information entropy-based scheduling for communication-efficient decentralized learning

This paper addresses decentralized stochastic gradient descent (DSGD) over resource-constrained networks by introducing nodebased and link-based scheduling strategies to enhance communication efficiency. In each iteration of the D-SGD algorithm, only a

few disjoint subsets of nodes or links are probabilistically activated, subject to a given communication cost constraint. We propose a novel importance metric based on information entropy to determine node and link scheduling probabilities. We validate the effectiveness of our approach through extensive simulations, comparing it against state-of-the-art methods, including betweenness centrality (BC) for node scheduling and MATCHA for link scheduling. The results show that our method consistently outperforms the BC-based method in node scheduling, achieving faster convergence with up to 60% lower communication budgets. At higher communication budgets (above 60%), our method maintains comparable or superior performance. In link scheduling, our method delivers results that are either superior or on par with those of MATCHA.