Abstract: The goal of this invention is to develop smart and fast data processing scheme for more computational efficient deep learning to support adaptive and real-time applications. We propose to apply Singular-Value Decomposition (SVD)-QR algorithm to preprocessing of deep learning for large scale data input. For the mass data input, we apply Limited Memory Subspace Optimization for SVD (LMSVD)-QR algorithm to increase the data processing speed. Simulation results in automated handwritten digit recognition show that SVD-QR and LMSVD-QR can tremendously reduce the number of input to deep learning neural network without losing its performance, and both can tremendously increase the data processing speed for deep learning.

Abstract: The deep Convolutional Neural Networks (CNN) has vast amount of parameters, especially in the Fully Connected (FC) layers, which has become a bottleneck for real-time sensing where processing latency is high due to computational cost. In this invention, we propose to optimize the FC layers in CNN for real-time sensing via making it much slimmer. We derive a CNN Design and Optimization Theorem for FC layers from information theory point of view. The optimization criteria is eigenvalues-based, so we apply Singular Value Decomposition (SVD) to find the maximal eigenvalues and QR to identify the corresponding columns in FC layer. Further, we propose Efficient Weights for CNN Design Theorem, and show that weights with colored Gaussian are much more efficient than those with white Gaussian. We evaluate our optimization approach to AlexNet and apply the slimmer CNN to ImageNet classification. Testing results show our approach performs much better than random dropout.