ShiDianNao: shifting vision processing closer to the sensor: ACM SIGARCH Computer Architecture News: Vol 43, No 3S
- ️TemamOlivier
- ️Sat Jun 13 2015
ShiDianNao: shifting vision processing closer to the sensor
Abstract
In recent years, neural network accelerators have been shown to achieve both high energy efficiency and high performance for a broad application scope within the important category of recognition and mining applications.
Still, both the energy efficiency and performance of such accelerators remain limited by memory accesses. In this paper, we focus on image applications, arguably the most important category among recognition and mining applications. The neural networks which are state-of-the-art for these applications are Convolutional Neural Networks (CNN), and they have an important property: weights are shared among many neurons, considerably reducing the neural network memory footprint. This property allows to entirely map a CNN within an SRAM, eliminating all DRAM accesses for weights. By further hoisting this accelerator next to the image sensor, it is possible to eliminate all remaining DRAM accesses, i.e., for inputs and outputs.
In this paper, we propose such a CNN accelerator, placed next to a CMOS or CCD sensor. The absence of DRAM accesses combined with a careful exploitation of the specific data access patterns within CNNs allows us to design an accelerator which is 60× more energy efficient than the previous state-of-the-art neural network accelerator. We present a full design down to the layout at 65 nm, with a modest footprint of 4.86mm2 and consuming only 320mW, but still about 30× faster than high-end GPUs.
References
[1]
Berkeley Vision and Learning Center, "Caffe: a deep learning framework." Available: http://caffe.berkeleyvision.org/
[2]
S. Chakradhar, M. Sankaradas, V. Jakkula, and S. Cadambi, "A dynamically configurable coprocessor for convolutional neural networks," in Proceedings of the 37th annual international symposium on Computer architecture (ISCA). New York, USA: ACM Press, 2010, pp. 247--257.
[3]
T. Chen, Z. Du, N. Sun, J. Wang, and C. Wu, "DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning," in Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Salt Lake City, UT, USA, 2014, pp. 269--284.
[4]
Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, "DaDianNao: A Machine-Learning Supercomputer," in Proceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2015, pp. 609--622.
[5]
D. C. Cires, U. Meier, J. Masci, and L. M. Gambardella, "Flexible, High Performance Convolutional Neural Networks for Image Classification," in Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCNN), 2003, pp. 1237--1242.
[6]
A. Coates, B. Huval, T. Wang, D. J. Wu, and A. Y. Ng, "Deep learning with COTS HPC systems," in Proceedings of the 30th International Conference on Machine Learning (ICML), 2013, pp. 1337--1345.
[7]
G. Dahl, T. Sainath, and G. Hinton, "Improving deep neural networks for LVCSR using rectified linear units and dropout," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'13), 2013, pp. 8609--8613.
[8]
S. A. Dawwd, "The multi 2D systolic design and implementation of Convolutional Neural Networks," in 2013 IEEE 20th International Conference on Electronics, Circuits, and Systems (ICECS). IEEE, Dec. 2013, pp. 221--224.
[9]
M. Delakis and C. Garcia, "Text Detection with Convolutional Neural Networks," in International Conference on Computer Vision Theory and Applications (VISAPP), 2008, pp. 290--294.
[10]
Z. Du, A. Lingamneni, Y. Chen, K. Palem, O. Temam, and C. Wu, "Leveraging the error resilience of machine-learning applications for designing highly energy efficient accelerators," 2014 19th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 201--206, Jan. 2014.
[11]
S. Duffner and C. Garcia, "Robust Face Alignment Using Convolutional Neural Networks," in International Conference on Computer Vision Theory and Applications (VISAPP), 2008, pp. 30--37.
[12]
H. Esmaeilzadeh, P. Saeedi, B. Araabi, C. Lucas, and S. Fakhraie, "Neural Network Stream Processing Core (NnSP) for Embedded Systems," 2006 IEEE International Symposium on Circuits and Systems (ISCS), pp. 2773--2776, 2006.
[13]
H. Esmaeilzadeh, A. Sampson, L. Ceze, and D. Burger, "Neural Acceleration for General-Purpose Approximate Programs," 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 449--460, Dec. 2012.
[14]
K. Fan, S. Mahlke, and A. Arbor, "Bridging the Computation Gap Between Programmable Processors and Hardwired Accelerators," in IEEE 15th International Symposium on High Performance Computer Architecture (HPCA), 2009, pp. 313--322.
[15]
C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello, "Hardware accelerated convolutional neural networks for synthetic vision systems," in Proceedings of 2010 IEEE International Symposium on Circuits and Systems (ISCS). IEEE, May 2010, pp. 257--260.
[16]
C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, and Y. LeCun, "NeuFlow: A runtime reconfigurable dataflow processor for vision," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, Jun. 2011, pp. 109--116.
[17]
C. Garcia and M. Delakis, "Convolutional face finder: a neural architecture for fast and robust face detection." IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 26, no. 11, pp. 1408--23, Nov. 2004.
[18]
V. Gokhale, J. Jin, and A. Dundar, "A 240 G-ops/s mobile coprocessor for deep neural networks," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2014, pp. 682--687.
[19]
Google, "Google image search." Available: http://www.google.com/insidesearch/features/images/searchbyimage.htm
[20]
R. Hameed, W. Qadeer, M. Wachs, O. Azizi, A. Solomatnikov, B. C. Lee, S. Richardson, C. Kozyrakis, and M. Horowitz, "Understanding sources of inefficiency in general-purpose chips," in Proceedings of Annual International Symposium on Computer Architecture (ISCA). New York, USA: ACM Press, 2010, p. 37.
[21]
S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed. Upper Saddle River, NJ, USA: Prentice Hall PTR, 1998.
[22]
V. Hecht, K. Ronner, and P. Pirsch, "An Advanced Programmable 2D-Convolution Chip for Real Time Image Processing," in IEEE International Sympoisum on Circuits and Systems (ISCS), 1991, pp. 1897--1900.
[23]
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors," in arXiv: 1207.0580 (2012), pp. 1--18.
[24]
P. S. Huang, X. He, J. Gao, and L. Deng, "Learning deep structured semantic models for web search using clickthrough data," in International Conference on Information and Knowledge Management (CIKM), 2013, pp. 2333--2338.
[25]
P. Ienne, T. Cornu, and G. Kuhn, "Special-purpose digital hardware for neural networks: An architectural survey," Journal of VLSI Signal Processing, vol. 13, no. 1, pp. 5--25, 1996.
[26]
K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun, "What is the best multi-stage architecture for object recognition?" 2009 IEEE 12th International Conference on Computer Vision (ICCV), pp. 2146--2153, Sep. 2009.
[27]
S. Kamijo, Y. Matsushita, K. Ikeuchi, and M. Sakauchi, "Traffic monitoring and accident detection at intersections," IEEE Transactions on Intelligent Transportation Systems, vol. 1, no. 2, pp. 108--118, Jun. 2000.
[28]
S. W. Keckler, W. J. Dally, B. Khailany, M. Garland, and D. Glasco, "GPUs and the future of parallel computing," IEEE Micro, pp. 7--17, 2011.
[29]
A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Advances In Neural Information Processing Systems, pp. 1--9, 2012.
[30]
B. Kwolek, "Face detection using convolutional neural networks and Gabor filters," Artificial Neural Networks: Biological Inspirations (ICANN), pp. 551--556, 2005.
[31]
D. Larkin, A. Kinane, V. Muresan, and N. E. O'Connor, "An Efficient Hardware Architecture for a Neural Network Activation Function Generator," in Advances in Neural Networks, ser. Lecture Notes in Computer Science, vol. 3973. Springer, 2006, pp. 1319--1327.
[32]
H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio, "An empirical evaluation of deep architectures on problems with many factors of variation," in International Conference on Machine Learning (ICML). New York, New York, USA: ACM Press, 2007, pp. 473--480.
[33]
S. Lawrence, C. L. Giles, a. C. Tsoi, and a. D. Back, "Face recognition: a convolutional neural-network approach." IEEE Transactions on Neural Networks, vol. 8, no. 1, pp. 98--113, Jan. 1997.
[34]
Q. V. Le, M. A. Ranzato, M. Devin, G. S. Corrado, and A. Y. Ng, "Building High-level Features Using Large Scale Unsupervised Learning," International Conference on Machine Learning (ICML), pp. 8595--8598, 2012.
[35]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-Based Learning Applied to Document Recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278--2324, 1998.
[36]
Y. LeCun, K. Kavukcuoglu, and C. Farabet, "Convolutional networks and applications in vision," Proceedings of 2010 IEEE International Symposium on Circuits and Systems (ISCS), pp. 253--256, May 2010.
[37]
J.-J. Lee and G.-Y. Song, "Super-Systolic Array for 2D Convolution," 2006 IEEE Region 10 Conference (TENCON), pp. 1--4, 2006.
[38]
S. Y. Lee and J. K. Aggarwal, "Parallel 2D convolution on a Mesh Connected Array Processor," IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. PAMI-9, no. 4, pp. 590--594, 1987.
[39]
B. Liang and P. Dubey, "Recognition, Mining and Synthesis," Intel Technology Journal, vol. 09, no. 02, 2005.
[40]
D. Liu, T. Chen, S. Liu, J. Zhou, S. Zhou, O. Temam, X. Feng, X. Zhou, and Y. Chen, "Pudiannao: A polyvalent machine learning accelerator," in Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ), 2015, pp. 369--381.
[41]
V. Mnih and G. Hinton, "Learning to Label Aerial Images from Noisy Data," in Proceedings of the 29th International Conference on Machine Learning (ICML), 2012, pp. 567--574.
[42]
N. Muralimanohar, R. Balasubramonian, and N. Jouppi, "Optimizing NUCA Organizations and Wiring Alternatives for Large Caches with CACTI 6.0," The 40th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 3--14, Dec. 2007.
[43]
J. Nagi, F. Ducatelle, G. A. D. Caro, D. Cires, U. Meier, A. Giusti, and L. M. Gambardella, "Max-Pooling Convolutional Neural Networks for Vision-based Hand Gesture Recognition," in IEEE International Conference on Signal and Image Processing Applications (ICSIPA), 2011, pp. 342--347.
[44]
C. Nebauer, "Evaluation of convolutional neural networks for visual recognition." IEEE Transactions on Neural Networks, vol. 9, no. 4, pp. 685--96, Jan. 1998.
[45]
M. Peemen, A. a. a. Setio, B. Mesman, and H. Corporaal, "Memory-centric accelerator design for Convolutional Neural Networks," in International Conference on Computer Design (ICCD). IEEE, Oct. 2013, pp. 13--19.
[46]
C. Poulet, J. Y. Han, and Y. Lecun, "CNP: An FPGA-based processor for Convolutional Networks," in International Conference on Field Programmable Logic and Applications (FPL), vol. 1, no. 1, 2009, pp. 32--37.
[47]
M. Ranzato, F. J. Huang, Y. L. Boureau, and Y. LeCun, "Unsupervised learning of invariant feature hierarchies with applications to object recognition," in Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2007, pp. 1--8.
[48]
R. Salakhutdinov and G. Hinton, "Learning a nonlinear embedding by preserving class neighbourhood structure," in AI and Statistics, ser. JMLR Workshop and Conference Proceedings, vol. 3, no. 5. Citeseer, 2007, pp. 412--419.
[49]
SAMSUNG, "SAMSUNG Gear2 Tech Specs," Samsung Electronics, 2014.
[50]
M. Sankaradas, V. Jakkula, S. Cadambi, S. Chakradhar, I. Durdanovic, E. Cosatto, and H. P. Graf, "A Massively Parallel Coprocessor for Convolutional Neural Networks," 2009 20th IEEE International Conference on Application-specific Systems, Architectures and Processors (ASAP), pp. 53--60, Jul. 2009.
[51]
D. Scherer, H. Schulz, and S. Behnke, "Accelerating large-scale convolutional neural networks with parallel graphics multiprocessors," in Artificial Neural Networks (ICANN), ser. Lecture Notes in Computer Science, K. Diamantaras, W. Duch, and L. Iliadis, Eds. Springer Berlin Heidelberg, 2010, vol. 6354, pp. 82--91.
[52]
P. Sermanet and Y. LeCun, "Traffic sign recognition with multi-scale Convolutional Networks," in International Joint Conference on Neural Networks (IJCNN). Ieee, Jul. 2011, pp. 2809--2813.
[53]
P. Y. Simard, D. Steinkraus, and J. C. Platt, "Best practices for convolutional neural networks applied to visual document analysis," in Seventh International Conference on Document Analysis and Recognition 2003 Proceedings (ICDAR), vol. 1. IEEE Comput. Soc, 2003, pp. 958--963.
[54]
T. Starner, "Project Glass: An Extension of the Self," IEEE Pervasive Computing, vol. 12, no. 2, pp. 14--16, Apr. 2013.
[55]
STV0986, 5 Megapixel mobile imaging processor (Data Brief), STMicroelectronics, Jan. 2007.
[56]
STV0987, 8 Megapixel mobile imaging processor (Data Brief), STMicroelectronics, Mar. 2013.
[57]
O. Temam, "A defect-tolerant accelerator for emerging high-performance applications," 2012 39th Annual International Symposium on Computer Architecture (ISCA), vol. 00, no. c, pp. 356--367, 2012.
[58]
H. Tkung and D. Wl, "Two-level pipelined systolic array for multidimensional convolution," Image and Vision Computing, vol. 1, no. 1, pp. 30--36, 1983.
[59]
V. Vanhoucke, A. Senior, and M. Z. Mao, "Improving the speed of neural networks on CPUs," in Deep Learning and Unsupervised Feature Learning Workshop, Neural Information Processing Systems Conference (NIPS), 2011.
[60]
G. Venkatesh, J. Sampson, N. Goulding-hotta, S. K. Venkata, M. B. Taylor, and S. Swanson, "QSCORES: Trading Dark Silicon for Scalable Energy Efficiency with Quasi-Specific Cores Categories and Subject Descriptors," in Proceedings of the 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2011, pp. 163--174.
[61]
S. Yehia, S. Girbal, H. Berry, and O. Temam, "Reconciling specialization and flexibility through compound circuits," in IEEE 15th International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2009, pp. 277--288.
Information & Contributors
Information
Published In
ACM SIGARCH Computer Architecture News Volume 43, Issue 3S
ISCA'15
June 2015
745 pages
ISCA '15: Proceedings of the 42nd Annual International Symposium on Computer Architecture
June 2015
768 pages
Copyright © 2015 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 June 2015
Published in SIGARCH Volume 43, Issue 3S
Check for updates
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- Downloads (Last 12 months)413
- Downloads (Last 6 weeks)39
Reflects downloads up to 20 Feb 2025
Other Metrics
Citations
- S. C. VSabareeshwari VK. C. S(2024)Artificial Intelligence in CommunicationsConvergence of Antenna Technologies, Electronics, and AI10.4018/979-8-3693-3775-2.ch008(209-238)Online publication date: 27-Sep-2024
- Liu BHuang XZhang YYang GYan HZhang CLi ZWang YCai H(2024)Layer-Wise Mixed-Modes CNN Processing Architecture With Double-Stationary Dataflow and Dimension-Reshape StrategyIEEE Transactions on Circuits and Systems I: Regular Papers10.1109/TCSI.2024.343470671:10(4652-4664)Online publication date: Oct-2024
- Zhang LLi HZhang XLou X(2024)RAW Images-Based Motion-Assisted Object Detection Accelerator Using Deformable Parts Models Features on 1080p VideosIEEE Transactions on Circuits and Systems I: Regular Papers10.1109/TCSI.2024.342575171:11(5054-5066)Online publication date: Nov-2024
- Schmid PPalomero Bernardo PGerum CBringmann O(2024)GOURD: Tensorizing Streaming Applications to Generate Multi-Instance Compute PlatformsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2024.344581043:11(4166-4177)Online publication date: Nov-2024
- Lu LLuo ZZheng SYin JCong JLiang YYin J(2024)Rubick: A Unified Infrastructure for Analyzing, Exploring, and Implementing Spatial Architectures via Dataflow DecompositionIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.333720843:4(1177-1190)Online publication date: Apr-2024
- Genc HKim HGanesh PShao Y(2024)Stellar: An Automated Design Framework for Dense and Sparse Spatial Accelerators2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO)10.1109/MICRO61859.2024.00038(409-422)Online publication date: 2-Nov-2024
- Wei XWang CGuan ZLi FYue H(2024)HAp-FT: A Hybrid Approximate Fault Tolerance Framework for DNN Accelerator2024 IEEE 42nd International Conference on Computer Design (ICCD)10.1109/ICCD63220.2024.00065(381-388)Online publication date: 18-Nov-2024
- Wang SXu HMamandipoor AMahapatra RAhn BGhodrati SKailas KAlian MEsmaeilzadeh H(2024)Data Motion Acceleration: Chaining Cross-Domain Multi Accelerators2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA57654.2024.00083(1043-1062)Online publication date: 2-Mar-2024
- Wu ZHu YLi NLu WLiu Y(2024)HiEval: A scheduling performance estimation approach for spatial accelerators via hierarchical abstractionJournal of Systems Architecture10.1016/j.sysarc.2024.103079148(103079)Online publication date: Mar-2024
- NGUYEN VNAKASHIMA Y(2023)Implementation of Fully-Pipelined CNN Inference Accelerator on FPGA and HBM2 PlatformIEICE Transactions on Information and Systems10.1587/transinf.2022EDP7155E106.D:6(1117-1129)Online publication date: 1-Jun-2023
- Show More Cited By
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.