Binary domain generalization for sparsifying binary neural networks

Schiavone, Riccardo; Galati, Francesco; Zuluaga, Maria A.
ECML-PKDD 2023, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 18-22 September 2023, Torino, Italy

Binary neural networks (BNNs) are an attractive solution for developing and deploying deep neural network (DNN)-based applica-tions in resource constrained devices. Despite their success, BNNs still suffer from a fixed and limited compression factor that may be explained by the fact that existing pruning methods for full-precision DNNs cannot be directly applied to BNNs. In fact, weight pruning of BNNs leads to performance degradation, which suggests that the standard binarization domain of BNNs is not well adapted for the task. This work proposes a novel more general binary domain that extends the standard binary one that is more robust to pruning techniques, thus guaranteeing improved compression and avoiding severe performance losses. We demonstrate a closed-form solution for quantizing the weights of a full-precision network into the proposed binary domain. Finally, we show the flexibility of our method, which can be combined with other pruning strategies. Exper-iments over CIFAR-10 and CIFAR-100 demonstrate that the novel ap-proach is able to generate efficient sparse networks with reduced memory usage and run-time latency, while maintaining performance.

Data Science
Eurecom Ref:
© 2023 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.