Deep learning processor

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Deep_learning_processor 

A deep learning processor (DLP), or a deep learning accelerator, is an electronic circuit designed for deep learning algorithms, usually with separate data memory and dedicated instruction set architecture. Deep learning processors range from mobile devices, such as neural processing units (NPUs) in Huawei cellphones, to cloud computing servers such as tensor processing units (TPU) in the Google Cloud Platform.

The goal of DLPs is to provide higher efficiency and performance for deep learning algorithms than general central processing unit (CPUs) and graphics processing units (GPUs) would. Most DLPs employ a large number of computing components to leverage high data-level parallelism, a relatively larger on-chip buffer/memory to leverage the data reuse patterns, and limited data-width operators for error-resilience of deep learning. Deep learning processors differ from AI accelerators in that they are specialized for running learning algorithms, while AI accelerators are typically more specialized for inference. However, the two terms (DLP vs AI accelerator) are not used rigorously and there is often overlap between the two.

History

The use of CPUs/GPUs

At the beginning, general CPUs were adopted to perform deep learning algorithms. Later, GPUs are introduced to the domain of deep learning. For example, in 2012, Alex Krizhevsky adopted two GPUs to train a deep learning network, i.e., AlexNet, which won the champion of the ISLVRC-2012 competition. As the interests in deep learning algorithms and DLPs keep increasing, GPU manufacturers start to add deep learning related features in both hardware (e.g., INT8 operators) and software (e.g., cuDNN Library). For example, Nvidia even released the Turing Tensor Core—a DLP—to accelerate deep learning processing.

The first DLP

To provide higher efficiency in performance and energy, domain-specific design starts to draw a great attention. In 2014, Chen et al. proposed the first DLP in the world, DianNao (Chinese for "electric brain"), to accelerate deep neural networks especially. DianNao provides the 452 Gop/s peak performance (of key operations in deep neural networks) only in a small footprint of 3.02 mm2 and 485 mW. Later, the successors (DaDianNao, ShiDianNao, PuDianNao) are proposed by the same group, forming the DianNao Family.

The blooming DLPs

Inspired from the pioneer work of DianNao Family, many DLPs are proposed in both academia and industry with design optimized to leverage the features of deep neural networks for high efficiency. Only at ISCA 2016, three sessions, 15% (!) of the accepted papers, are all architecture designs about deep learning. Such efforts include Eyeriss (MIT), EIE (Stanford), Minerva (Harvard), Stripes (University of Toronto) in academia, and TPU (Google), MLU (Cambricon) in industry. We listed several representative works in Table 1.

Table 1. Typical DLPs
Year	DLPs	Institution	Type	Computation	Memory Hierarchy	Control	Peak Performance
2014	DianNao	ICT, CAS	digital	vector MACs	scratchpad	VLIW	452 Gops (16-bit)
	DaDianNao	ICT, CAS	digital	vector MACs	scratchpad	VLIW	5.58 Tops (16-bit)
2015	ShiDianNao	ICT, CAS	digital	scalar MACs	scratchpad	VLIW	194 Gops (16-bit)
	PuDianNao	ICT, CAS	digital	vector MACs	scratchpad	VLIW	1,056 Gops (16-bit)
2016	DnnWeaver	Georgia Tech	digital	Vector MACs	scratchpad	-	-
	EIE	Stanford	digital	scalar MACs	scratchpad	-	102 Gops (16-bit)
	Eyeriss	MIT	digital	scalar MACs	scratchpad	-	67.2 Gops (16-bit)
	Prime	UCSB	hybrid	Process-in-Memory	ReRAM	-	-
2017	TPU	Google	digital	scalar MACs	scratchpad	CISC	92 Tops (8-bit)
	PipeLayer	U of Pittsburgh	hybrid	Process-in-Memory	ReRAM	-
	FlexFlow	ICT, CAS	digital	scalar MACs	scratchpad	-	420 Gops ()
2018	MAERI	Georgia Tech	digital	scalar MACs	scratchpad	-
	PermDNN	City University of New York	digital	vector MACs	scratchpad	-	614.4 Gops (16-bit)
2019	FPSA	Tsinghua	hybrid	Process-in-Memory	ReRAM	-
	Cambricon-F	ICT, CAS	digital	vector MACs	scratchpad	FISA	14.9 Tops (F1, 16-bit) 956 Tops (F100, 16-bit)

DLP architecture

With the rapid evolution of deep learning algorithms and DLPs, many architectures have been explored. Roughly, DLPs can be classified into three categories based on their implementation: digital circuits, analog circuits, and hybrid circuits. As the pure analog DLPs are rarely seen, we introduce the digital DLPs and hybrid DLPs.

Digital DLPs

The major components of DLPs architecture usually include a computation component, the on-chip memory hierarchy, and the control logic that manages the data communication and computing flows.

Regarding the computation component, as most operations in deep learning can be aggregated into vector operations, the most common ways for building computation components in digital DLPs are the MAC-based (multiplier-accumulation) organization, either with vector MACs or scalar MACs. Rather than SIMD or SIMT in general processing devices, deep learning domain-specific parallelism is better explored on these MAC-based organizations. Regarding the memory hierarchy, as deep learning algorithms require high bandwidth to provide the computation component with sufficient data, DLPs usually employ a relatively larger size (tens of kilobytes or several megabytes) on-chip buffer but with dedicated on-chip data reuse strategy and data exchange strategy to alleviate the burden for memory bandwidth. For example, DianNao, 16 16-in vector MAC, requires 16 × 16 × 2 = 512 16-bit data, i.e., almost 1024GB/s bandwidth requirements between computation components and buffers. With on-chip reuse, such bandwidth requirements are reduced drastically. Instead of the widely used cache in general processing devices, DLPs always use scratchpad memory as it could provide higher data reuse opportunities by leveraging the relatively regular data access pattern in deep learning algorithms. Regarding the control logic, as the deep learning algorithms keep evolving at a dramatic speed, DLPs start to leverage dedicated ISA (instruction set architecture) to support the deep learning domain flexibly. At first, DianNao used a VLIW-style instruction set where each instruction could finish a layer in a DNN. Cambricon introduces the first deep learning domain-specific ISA, which could support more than ten different deep learning algorithms. TPU also reveals five key instructions from the CISC-style ISA.

Hybrid DLPs

Hybrid DLPs emerge for DNN inference and training acceleration because of their high efficiency. Processing-in-memory (PIM) architectures are one most important type of hybrid DLP. The key design concept of PIM is to bridge the gap between computing and memory, with the following manners: 1) Moving computation components into memory cells, controllers, or memory chips to alleviate the memory wall issue. Such architectures significantly shorten data paths and leverage much higher internal bandwidth, hence resulting in attractive performance improvement. 2) Build high efficient DNN engines by adopting computational devices. In 2013, HP Lab demonstrated the astonishing capability of adopting ReRAM crossbar structure for computing. Inspiring by this work, tremendous work are proposed to explore the new architecture and system design based on ReRAM, phase change memory, etc.

GPUs and FPGAs

Despite the DLPs, GPUs and FPGAs are also being used as accelerators to speed up the execution of deep learning algorithms. For example, Summit, a supercomputer from IBM for Oak Ridge National Laboratory, contains 27,648 Nvidia Tesla V100 cards, which can be used to accelerate deep learning algorithms. Microsoft builds its deep learning platform using FPGAs in its Azure to support real-time deep learning services. In Table 2 we compare the DLPs against GPUs and FPGAs in terms of target, performance, energy efficiency, and flexibility.

Table 2. DLPs vs. GPUs vs. FPGAs

	Target	Performance	Energy Efficiency	Flexibility
DLPs	deep learning	high	high	domain-specific
FPGAs	all	low	moderate	general
GPUs	matrix computation	moderate	low	matrix applications

Atomically thin semiconductors for deep learning

Atomically thin semiconductors are considered promising for energy-efficient deep learning hardware where the same basic device structure is used for both logic operations and data storage. In 2020, Marega et al. published experiments with a large-area active channel material for developing logic-in-memory devices and circuits based on floating-gate field-effect transistors (FGFETs). They use two-dimensional materials such as semiconducting molybdenum disulphide to precisely tune FGFETs as building blocks in which logic operations can be performed with the memory elements. 

Integrated photonic tensor core

In 2021, J. Feldmann et al. proposed an integrated photonic hardware accelerator for parallel convolutional processing. The authors identify two key advantages of integrated photonics over its electronic counterparts: (1) massively parallel data transfer through wavelength division multiplexing in conjunction with frequency combs, and (2) extremely high data modulation speeds. Their system can execute trillions of multiply-accumulate operations per second, indicating the potential of integrated photonics in data-heavy AI applications.

Benchmarks

Benchmarking has served long as the foundation of designing new hardware architectures, where both architects and practitioners can compare various architectures, identify their bottlenecks, and conduct the corresponding system/architectural optimization. Table 3 lists several typical benchmarks for DLPs, dating from the year 2012 in time order.

Table 3. Benchmarks.

Year	NN Benchmark	Affiliations	# of microbenchmarks	# of component benchmarks	# of application benchmarks
2012	BenchNN	ICT, CAS	N/A	12	N/A
2016	Fathom	Harvard	N/A	8	N/A
2017	BenchIP	ICT, CAS	12	11	N/A
2017	DAWNBench	Stanford	8	N/A	N/A
2017	DeepBench	Baidu	4	N/A	N/A
2018	MLPerf	Harvard, Intel, and Google, etc.	N/A	7	N/A
2019	AIBench	ICT, CAS and Alibaba, etc.	12	16	2
2019	NNBench-X	UCSB	N/A	10	N/A