Sparse Training Algorithms based on Compressed Sensing for Accelerating Large-Scale Neural Networks

Abstract

The exponential growth in the parameter count of modern deep neural networks has  precipitated a significant computational bottleneck, necessitating the development of efficient  training methodologies. This paper explores the integration of Compressed Sensing theories into  the training dynamics of large-scale neural networks to achieve acceleration through enforced  sparsity. Unlike traditional pruning methods that operate post-training, the proposed algorithmic  framework introduces sparsity constraints during the initialization and optimization phases, 
effectively reducing the memory footprint and floating-point operations required for convergence.  We leverage the Restricted Isometry Property to guarantee that the sparse representations learned  during the training process retain sufficient information to reconstruct the underlying mapping  functions of the network. By treating the weight matrices as sparse signals and the gradient  updates as measurements, we formulate a recovery algorithm that allows the network to learn  optimal sparse topologies dynamically. Extensive empirical analysis demonstrates that this  approach not only accelerates the training phase by reducing computational complexity but also  produces models that are robust and generalizable. The findings suggest that Compressed Sensing  offers a rigorous theoretical foundation for sparse training, bridging the gap between  mathematical signal processing and empirical deep learning optimization.