An AI accelerator is a class of microprocessor or computer system designed as hardware acceleration for artificial intelligence applications, especially artificial neural networks, machine vision and machine learning. Typical applications include algorithms for robotics, internet of things and other data-intensive or sensor-driven tasks. They are often manycore designs and generally focus on low-precision arithmetic, novel dataflow architectures or in-memory computing capability. A number of vendor-specific terms exist for devices in this category, and it is an emerging technology without a dominant design. AI accelerators can be found in many devices such as smartphones, tablets, and computers all around the world. See the heading titled ¨Examples" for more examples.
History of AI acceleration
Computer systems have frequently complemented the CPU with special purpose accelerators for specialized tasks, known as coprocessors. Notable application-specific hardware units include video cards for graphics, sound cards, graphics processing units and digital signal processors. As deep learning and artificial intelligence workloads rose in prominence in the 2010s, specialized hardware units were developed or adapted from existing products to accelerate these tasks.
Early attempts
As early as 1993, digital signal processors were used as neural network accelerators e.g. to accelerate optical character recognition software.
In the 1990s, there were also attempts to create parallel
high-throughput systems for workstations aimed at various applications,
including neural network simulations. FPGA-based accelerators were also first explored in the 1990s for both inference and training. ANNA was a neural net CMOS accelerator developed by Yann LeCun.
Heterogeneous computing
Heterogeneous computing
refers to incorporating a number of specialized processors in a single
system, or even a single chip, each optimized for a specific type of
task. Architectures such as the cell microprocessor have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic, dataflow architecture, and prioritizing 'throughput' over latency. The Cell microprocessor was subsequently applied to a number of tasks including AI.
In the 2000s, CPUs also gained increasingly wide SIMD units, driven by video and gaming workloads; as well as support for packed low precision data types.
Use of GPU
Graphics processing units
or GPUs are specialized hardware for the manipulation of images and
calculation of local image properties. The mathematical basis of neural
networks and image manipulation are similar, embarrassingly parallel tasks involving matrices, leading GPUs to become increasingly used for machine learning tasks. As of 2016, GPUs are popular for AI work, and they continue to evolve in a direction to facilitate deep learning, both for training and inference in devices such as self-driving cars. GPU developers such as Nvidia NVLink are developing additional connective capability for the kind of dataflow workloads AI benefits from. As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated neural network specific hardware to further accelerate these tasks. Tensor cores are intended to speed up the training of neural networks.
Use of FPGAs
Deep learning frameworks are still evolving, making it hard to design custom hardware. Reconfigurable devices such as field-programmable gate arrays (FPGA) make it easier to evolve hardware, frameworks and software alongside each other.
Microsoft has used FPGA chips to accelerate inference. The application of FPGAs to AI acceleration motivated Intel to acquire Altera with the aim of integrating FPGAs in server CPUs, which would be capable of accelerating AI as well as general purpose tasks.
Emergence of dedicated AI accelerator ASICs
While GPUs and FPGAs perform far better than CPUs for AI related tasks, a factor of up to 10 in efficiency may be gained with a more specific design, via an application-specific integrated circuit (ASIC). These accelerators employ strategies such as optimized memory use and the use of lower precision arithmetic to accelerate calculation and increase throughput of computation. Some adopted low-precision floating-point formats used AI acceleration are half-precision and the bfloat16 floating-point format.
In-memory computing architectures
In June 2017, IBM researchers announced an architecture in contrast to the Von Neumann architecture based on in-memory computing and phase-change memory arrays applied to temporal correlation detection, intending to generalize the approach to heterogeneous computing and massively parallel systems. In October 2018, IBM researchers announced an architecture based on in-memory processing and modeled on the human brain's synaptic network to accelerate deep neural networks. The system is based on phase-change memory arrays.
Nomenclature
As
of 2016, the field is still in flux and vendors are pushing their own
marketing term for what amounts to an "AI accelerator", in the hope that
their designs and APIs will become the dominant design.
There is no consensus on the boundary between these devices, nor the
exact form they will take; however several examples clearly aim to fill
this new space, with a fair amount of overlap in capabilities.
In the past when consumer graphics accelerators emerged, the industry eventually adopted Nvidia's self-assigned term, "the GPU",
as the collective noun for "graphics accelerators", which had taken
many forms before settling on an overall pipeline implementing a model
presented by Direct3D.
Examples
Stand alone products
- Google Tensor processing unit is an accelerator specifically designed by Google for its TensorFlow framework, which is extensively used for convolutional neural networks. It focuses on a high volume of 8-bit precision arithmetic. The initial first generation from 2015 focused on inference, while the second generation announced in May 2017 increased capability for neural network training also. The third-generation TPU was announced on 8 May 2018. On July 2018 the Edge TPU was announced. Edge TPU is Google’s purpose-built ASIC chip designed to run its TensorFlow Lite machine learning (ML) models at the edge.
- Adapteva epiphany is a many-core coprocessor featuring a network on a chip scratchpad memory model, suitable for a dataflow programming model, which should be suitable for many machine learning tasks.
- Intel Nervana NNP (Neural Network Processor) (a.k.a. ”Lake Crest”), which Intel claims is the first commercially available chip with a purpose built architecture for deep learning. Facebook was a partner in the design process.
- Movidius Myriad 2 is a many-core VLIW AI accelerator complemented with video fixed function units.
- Mobileye's EyeQ is a processor specialized for vision processing for self-driving cars
- NM500 is the latest as of 2016 in a series of accelerator chips for radial basis function neural nets from General Vision.
- Kendryte K210 contains 64-bit RISC-V CPU and KPU, a general-purpose neural network processor with built-in convolution, batch normalization, activation, and pooling operations.
GPU based products
- Nvidia Tesla is Nvidia's line of GPU derived products marketed for GPGPU and AI tasks.
- Nvidia Volta is a microarchitecture which augments the Graphics processing unit with additional 'tensor units' targeted specifically at accelerating calculations for neural networks
- Nvidia GeForce 20 series is the first series based on the Turing microarchitecture and features built in "Tensor Cores".
- Nvidia DGX-1 is a Nvidia workstation/server product which incorporates Nvidia brand GPUs for GPGPU tasks including machine learning.
- Nvidia Tegra Xavier SoC features their Deep Learning Accelerator (DLA) and Programmable Vision Accelerator (PVA).
- Radeon Instinct is AMD's line of GPU derived products for AI acceleration.
- Qualcomm's Adreno GPUs since the Snapdragon 820 released in March 2015 using their Qualcomm Snapdragon Neural Processing Engine SDK.
- NEC SX-Aurora TSUBASA is NEC's product line for AI applications and machine learning.
AI accelerating co-processors
- Qualcomm's Hexagon DSPs since the Snapdragon 820 released in March 2015 using their Qualcomm Snapdragon Neural Processing Engine SDK.
- Qualcomm's Snapdragon 855 contains their 4th generation on-device AI engine, including a dedicated Tensor Accelerator.
- Cadence's Tensilica IP is a family of neural network processor and neural network-optimized digital signal processor IP core. Such as the Tensilica Vision C5 DSP released in May 2017 and Tensilica Vision Q6 DSP released in April 2018. The Tensilica DNA 100 Processor was announced in September 2018.
- Imagination Technologies' PowerVR 2NX NNA (Neural Net Accelerator) is an IP core from NEC (now Renesas) licensed for integration into chips, first announced September 2017. On December 2018 PowerVR 3NX NNA was announced.
- Apple's Neural Engine is an AI accelerator core within Apple-designed processors. The Apple A11 Bionic SoC released on September 2017 featured a dual core Neural Engine. The Apple A12 Bionic SoC released on September 2018 featured an octa core Neural Engine.
- Samsung's Exynos 9820 has an integrated Neural Processing Unit (NPU). It allows the processor to perform AI-related functions seven times faster than its predecessor. From enhancing photos to advanced AR features, the Exynos 9820 with NPU expands AI capabilities of mobile devices.
- Cambricon Technologies's Machine Learning Unit (MLU) family of neural processors such as the MLU-100 and MLU-200.
- HiSilicon's Neural Processing Unit is a neural network accelerator within HiSilicon's Kirin SoCs. The Kirin 970 with a NPU from Cambricon Technologies was released in October, 2017. The Kirin 980 with a dual core NPU from Cambricon Technologies was released in October, 2018.
- Google's Pixel Visual Core (PVC) is a fully programmable Image, Vision and AI processor for mobile devices. First featured in the Google Pixel 2 released in October, 2017.
- Arm's ML Processor is dedicated IP for neural network model inferencing acceleration. First announced as Project Trillium in January 2018.
- CEVA's NeuPro family of AI processors. The NP500, NP1000, NP2000 and NP4000 were first announced on January 2018. Each containing one programmable vector DSP and one hardwired implementation of 8-bit or 16-bit neural network layers supporting neural nets with performances ranging from 2 TOPS thru 12.5 TOPS.
- Universal Multifunction Accelerator (UMA) by Manjeera Digital Systems in Hyderabad is an accelerator in a proprietary architecture based on Middle Stratum Operations.
Research and unreleased products
- In December 2017 Tesla Motors confirmed a rumour that it is developing an AI chip for autonomous driving. Jim Keller worked on this project between at least early 2016 and early 2018.
- MIT Eyeriss is an accelerator design aimed explicitly at convolutional neural networks, using a scratchpad memory and network-on-chip architecture.
- Georgia Tech has designed a neuro-inspired processor for performing online reinforcement learning for ultra-low power robotics. It employs mixed-signal design techniques to reduce the operating power.
- Nullhop is an accelerator designed at the Institute of Neuroinformatics of ETH Zürich and University of Zürich based on sparse representation of feature maps. The second generation of the architecture is commercialized by the university spin-off Synthara Technologies.
- Kalray is an accelerator for convolutional neural nets.
- SpiNNaker is a many-core design specialized for simulating a large neural network.
- Graphcore IPU is a graph-based AI accelerator.
- DPU, by Wave Computing, a dataflow architecture
- STMicroelectronics at the start of 2017 presented a demonstrator SoC manufactured in a 28 nm process containing a deep CNN accelerator.
- TrueNorth is a manycore design based on spiking neurons rather than traditional arithmetic.
- Intel Loihi is an experimental neuromorphic chip.
- BrainChip in September 2017 introduced a commercial PCI Express card with a Xilinx Kintex Ultrascale FPGA running neuromorphic neural cores applying pattern recognition on 600 video images per second using 16 watts of power.
- IIT Madras is designing a spiking neuron accelerator for big-data analytics.
- Several memristor-based AI accelerators have been proposed which leverage in-memory computing capability of memristor.
- AlphaICs is designing an agent-based coprocessor called Real AI Processor (RAP) to enable perception and decision making in a chip.
Potential applications
- Autonomous vehicles: Nvidia has targeted their Drive PX-series boards at this space.
- Military robots
- Agricultural robots, for example pesticide-free weed control.
- Voice control, e.g. in mobile phones, a target for Qualcomm Zeroth.
- Machine translation
- Unmanned aerial vehicles, e.g. navigation systems, e.g. the Movidius Myriad 2 has been demonstrated successfully guiding autonomous drones.
- Industrial robots, increasing the range of tasks that can be automated, by adding adaptability to variable situations.
- Health care, to assist with diagnoses
- Search engines, increasing the energy efficiency of data centers and ability to use increasingly advanced queries.
- Natural language processing
