CNN Spatial Optimizations

Convolutional neural networks (CNNs) compute their output using weighted-sums of adjacent input elements. This method enables CNNs to achieve state-of-the-art results in a wide range of applications such as computer vision and speech recognition. However, it also comes with the cost of high computational intensity. Shomron et al purposed exploiting the spatial correlation inherent in CNNs and predict activation values, thus reducing the needed computations in the network. They introduced a heuristic that predicts which activations are zero-valued according to nearby activation values, in a scheme they call cross-neuron prediction:

In this work, we further generalize the work done by Shomron et al, with the following steps:

  1. Implement a custom CNN layer which we dub a “Spatial” layer, allowing for a fast and efficient prediction statistics collection on any chosen CNN architecture.
  2. Create a hierarchal framework that allows for the cross-neuron predictions on any generic CNN architecture and with any generic dataset chosen.
  3. Implement the class structure into a few chosen networks: AlexNet, ResNet18, ResNet34
  4. Implement a greedy optimization algorithm to choose the best Mask Configuration, allowing for a maximal number of saved operations, under a certain loss in accuracy and based on changing optimization granularity.

The algorithm was implemented in 4 modes: Max Granularity, Uniform Layer, Uniform Filter and Uniform Patch.

  1. Test framework on the chosen networks with varying datasets

Main Contribution

We bring forth a complete end-to-end framework enabling testing of cross-neuron predictions on any generic CNN network and dataset. We establish a “knob”, controlling prediction performance vs accuracy degradation. We show promising results of an up to 27.15% reduction in convolution operations on AlexNet and the CIFAR10 dataset, and up to 17.51% on ResNet18 and the MNIST dataset, with no decrease in accuracy.

We further show on the AlexNet and CIFAR10 setup a possible increase in accuracy after retraining, while keeping the same amount of operation savings. We demonstrate a 47.77% decrease in convolution operations under a mere 0.5% accuracy loss is achievable under retraining.

System Architecture

Spatial Layer

We build upon the work done by Shomron et al, and purpose a generalization of its idea. We propose a custom CNN layer which we dub a “Spatial” layer which computes the cross-neuron prediction for any given prediction mask. We recognize the fact that the flat prediction mask (pmask) used by Shomron et al may not be optimal, and other, more varied masks may achieve superior results, while supplying lower accuracy degradation. In general, for a patch of size p x p, may receive as many as  possible singleton masks (relevant for a single patch in the ofmap), removing the degenerate all 1s or all 0s mask.

The 2×2 prediction mask used by Shomron et al (left), versus all possible masks (right). The degenerate all ones or all zeros masks were removed, seeing they define deterministic and degenerate predictions

For a given mask, the activations positioned with “1” in each patch are calculated. If these activations are zero-valued, the remaining activations within that patch are predicted to be zero-valued as well, thereby saving their MAC operations.

This custom layer was implemented with Python utilizing the well-known PyTorch module, and runs efficiently with a GPU compatible implementation. The layer receives a given full pmask (combined from singleton patch-size pmasks, targeted to be in the shape of the ofmap), and collects statistics such as the number of MAC and convolution operations saved via the supplied pmask.

Spatial Network Framework

Recognizing the difference between different network architectures, such as VGG16 or AlexNet, we model a new class dubbed “SpatialNet”. This class inherits from the PyTorch neural network class, and implements generic functions that deal with the control and statistical analysis of all Spatial layers placed inside the network’s architecture definition. This allows an easy and simple way to implement Spatial layers into any desired CNN. We implement the framework on the AlexNet, ResNet18 and ResNet34 classes.

Greedy Optimization Algorithm

We devise a greedy hierarchal optimization algorithm to find the best network prediction mask, in terms of MAC operation savings. The overall flow diagram is as follows:

Results

Top Results

Network & Dataset AlexNet,CIFAR10 ResNet18, CIFAR10 ResNet18,

MNIST

Prediction Window 3×3 2×2 2×2 2×2
Op Reduction

 

27.15% 8.91% 13.69% 17.51%
Op Reduction

 

31.24% 23.4% 25.32% 20.78%
Op Reduction 33.5% 23.53% 27.12% 21.01%
Baseline Op Reduction 34.02% 24.9% 23.5% 16.76%
Baseline Accuracy

Drop [%]

14.1 13.2 3.5 1.2
Our Op Reduction

for baseline accuracy

 

42.42% 33.3% 25.32% 20.61%

Table 1: Top results over all optimization modes and under different network, dataset and patch size setups

We note a substantial increase in convolution operation savings from the baseline results. Some networks and datasets allow for a more substantial optimization than others.

Top Retraining Results:

Results after retraining – AlexNet, CIFAR10, Prediction Window 3×3, Initial Accuracy 75.8

Baseline convolution Reduction: 34.02%, Accuracy Drop: 14.1

 

  Before Retraining After Retraining Raw Increase in Accuracy [%] Final Accuracy [%]
Op Reduction

 

27.15% 27.16% +0.6 76.4

(Higher than initial)

Op Reduction

 

31.24% 35.79% +0.8 74
Op Reduction 33.5% 37.2% +1.4 73.2
Our Op Reduction

for baseline accuracy

 

42.42% 47.77%  

+13.2

 

 

75.3

Table 2: Top results after retraining for 50 epochs

We note a vast increase in accuracy under retraining (almost returning to the very same initial accuracy of the network) while increasing the number of convolution operations saved. With this retraining, we achieve a close to 50% decrease in convolution operations with a mere 0.5% drop in accuracy.