Contributions to the realization of DNN-based visual inference on embedded systems

Contributions to the realization of DNN-based visual inference on embedded systems Delia Velasco Montero Tesis depositada para optar al grado de Doctor en Ciencias y Tecnologías Físicas Universidad de Sevilla Directores: Jorge Fernández Berni Ángel Rodríguez Vázquez Sevilla, Septiembre de 2022 A cada persona que ha mostrado interés por el desarrollo de mi tesis, especialmente a mi familia. Agradecimientos Enprimerlugar, a misdirectores. Enparticular aÁngel, por darme la oportunidad deintroducirme en el mundo de la investigación; a Jorge, por buscar temas de interés común, por crear un ambiente en el que es un placer trabajar, y por la confianza mutua que ha guiado todo el proceso; a Bart, por la exhaustiva colaboración realizada; y a Ricardo, por su apoyo. En segundo lugar, a todos los que han hecho más fácil y ameno este camino, desde compañeros del IMSE hasta cada persona que ha aportado su granito de arena de cualquier manera. Contents List of Figures ix List of Tables xiii Abbreviations xv 1 Introduction 1 2 Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models 5 2.1 Networks models ...................................... 6 2.1.1 CNN-based tasks .................................. 9 2.1.2 Metrics on CNN complexity and performance ................... 10 2.1.3 Popular convolutional neural networks ....................... 12 2.1.3.1 Classification networks ......................... 12 2.1.3.2 Object-detection networks ........................ 14 2.1.4 Training convolutional neural networks ...................... 15 2.2 Software ........................................... 15 2.2.1 Software implementation strategies ........................ 16 2.2.2 Additional software tools .............................. 17 2.3 Hardware devices and accelerators ............................. 18 2.4 Summary and Related Topics ................................ 20 3 Optimal selection of software and models for visual inference 23 3.1 Methodology Overview ................................... 23 3.2 Benchmarking ........................................ 24 3.2.1 Performance Evaluation Metrics .......................... 24 3.2.2 Selected Components ................................ 25 3.2.2.1 Embedded Platform ........................... 25 3.2.2.2 Convolutional Neural Networks ..................... 25 3.2.2.3 Software Frameworks .......................... 25 3.2.3 Benchmarking Results ............................... 26 3.3 Optimum Selection ..................................... 28 3.3.1 Pairwise metric comparison ............................ 29 3.3.1.1 Results and Analysis ........................... 29 3.3.2 Figure of Merit ................................... 29 3.3.2.1 Weighted FoM .............................. 31 3.3.2.2 Results .................................. 32 3.3.2.3 Analysis ................................. 34 3.4 Summary and Related Topics ................................ 35 4 Relevant hardware metrics for performance evaluation 39 4.1 Introduction ......................................... 39 v vi CONTENTS 4.2 Procedure Overview .................................... 40 4.3 Hardware-aware Analysis .................................. 41 4.3.1 Results and Analysis ................................ 43 4.4 High-level Performance Analysis .............................. 45 4.4.1 Results ....................................... 45 4.4.2 Analysis ....................................... 47 4.5 Qualitative Performance Explanation ............................ 47 4.5.1 From Aggregated Statistics to Inference Performance ............... 47 4.5.2 From Instantaneous Statistics to Inference Performance .............. 48 4.6 Graphical Comparison of Metrics ............................. 49 4.7 Summary and Related Topics ................................ 51 5 Prediction of visual inference performance 53 5.1 Introduction ......................................... 53 5.2 Fundamentals of CNNs ................................... 54 5.3 Modeling and Prediction of Inference Performance .................... 57 5.3.1 General Description ................................ 57 5.3.2 Selected System .................................. 57 5.3.3 Network Profiling .................................. 58 5.3.4 Model Construction ................................ 59 5.3.4.1 Model formulation ............................ 59 5.3.4.2 Observation dataset ........................... 60 5.3.4.3 Model Tuning. Feature Selection and Extraction ............ 60 5.4 PreVIousNet: a Network for Fine-Grained Characterization ................ 62 5.4.1 Layer Parameters .................................. 63 5.4.2 Architecture ..................................... 63 5.4.3 Network Configuration for the Modeling Stage .................. 65 5.5 Experimental Tests ..................................... 65 5.5.1 Embedded System ................................. 65 5.5.2 Networks ...................................... 66 5.5.3 Experimental Results: Layer-wise Predictions ................... 66 5.5.4 Experimental Results: Network Predictions .................... 69 5.5.5 Analysis ....................................... 70 5.6 Summary and Related Topics ................................ 71 6 Exploiting heterogeneous edge platforms for high-throughput or low-energy visual inference 75 6.1 Introduction ......................................... 75 6.1.1 Motivation ..................................... 76 6.2 Proposed approach ..................................... 77 6.2.1 Problem formulation ................................ 78 6.2.1.1 Scheduling space ............................ 79 6.2.1.2 Scheduling search problem ....................... 80 6.2.1.3 Scheduling on TX2 using TVM ..................... 80 6.3 Experimental Procedure .................................. 82 6.3.1 Hardware and Software ............................... 82 6.3.2 Neural Networks .................................. 83 6.3.3 Layer-wise Performance Profiling ......................... 83 6.4 Performance Evaluation ................................... 85 6.4.1 Single-device execution .............................. 85 6.4.2 Two-stage execution ................................ 85 6.4.3 K-stage execution .................................. 88 6.5 Discussion .......................................... 88 6.6 Summary and Related Topics ................................ 89 Contents vii 7 A case study: remote animal monitoring 91 7.1 Introduction: Target Application .............................. 91 7.2 Methodology – Overview of DL Model Deployment ................... 92 7.3 Data Collection ....................................... 93 7.3.1 Leveraging the dataset ............................... 94 7.4 Model Training ....................................... 95 7.4.1 Training Classification Networks .......................... 95 7.4.1.1 Results .................................. 96 7.4.2 Training Object-Detection Networks ........................ 97 7.4.2.1 Results .................................. 98 7.4.3 Discussion ..................................... 98 7.4.4 Explainable Deep Learning ............................ 99 7.5 Visual System Implementation ............................... 100 7.5.1 Hardware ...................................... 100 7.5.2 Software Libraries ................................. 101 7.5.3 Application Algorithm ............................... 103 7.5.3.1 Motion detection – camera trigger ................... 103 7.5.3.2 Camera capture ............................. 103 7.5.3.3 CNN classification ............................ 103 7.5.3.4 Send notification – alarm ........................ 104 7.5.3.5 Application Configuration ........................ 104 7.6 Experimental Tests ..................................... 105 7.6.1 System Setup .................................... 105 7.6.2 Performance Metrics ................................ 106 7.7 Summary and Related Topics ................................ 107 Conclusions 109 Bibliography 121 Brief Resume 123 List of Figures 2.1 Hierarchical stack of technological components involved in embedded realizations of DL. Numerous options are available at each level ........................ 5 2.2 Generic CNN architecture. Its core operation is the convolutional layer, but other operations, such as pooling or normalization, are also included in the network architecture. Fully-connected layers are usually inserted at the end of the CNN ............. 6 2.3 Processing based on a sliding window. The green array is the operation kernel (implementing filtering, pooling, etc.), whereas the RGB array is the input image. The process of sliding the kernel along the horizontal dimension of the input is depicted with light areas. The kernel will be eventually applied to the whole image by sliding in the vertical dimension as well ...................................... 7 2.4 Convolutional layers constitute the core operation of CNNs. Each kernel filter – depicted in green – operates on sliding local regions of the input fmaps – receptive fields in light blue – to produce the corresponding output fmaps ..................... 8 2.5 Intersection over Union. Intersection is the overlap region between areas A (green) and B (blue), whereas union is the overall region including both boxes. IoU is the fraction between intersection and union ............................... 11 2.6 Data reshaping on input data and kernel weights – so-called im2col and im2row. Each receptive field on an input fmap sized kh×kw×Cin becomes a column of an enlarged matrix. Similarly, weights on each 3D filter are flattened into a row. Thus, convolution becomes a matrix-to-matrix product ............................ 17 2.7 Left: FC layer operating on 9 input activations and producing a 5-neuron output vector. One weight connection is applied between each input-output pair (these weights are not included to avoid clutter in the graph). Right: implementation of this layer as the matrixproduct of an array of weights (green) and the input vector (blue). For instance, the dot product of the weights highlighted in the second row and the column of input activations produces the second output of this FC layer ........................ 17 3.1 Inference performance metrics for each combination of DNN model and DL framework. Numerical results are also reported in Table 3.3 ...................... 28 3.2 Throughput performance versus network accuracy. Each sample point is the performance of a combination network–framework from Figs. 3.1a and 3.1c. The black curve represents the Pareto frontier, from which one metric cannot be improved without worsening the other. 29 3.3 Throughput performance versus power consumption for each network–framework combination and Pareto frontier of optimal solutions ...................... 30 3.4 Network accuracy versus power consumption for each network–framework combination and Pareto frontier of optimal solutions .......................... 30 3.5 Figure of Merit defined in Eq. (3.1). It is expressed as ‘number of correctly inferred images per joule’ ...................................... 31 3.6 Normalized weighted FoM for Network in Network .................... 32 3.7 Normalized weighted FoM for GoogLeNet ......................... 33 3.8 Normalized weighted FoM for ResNet-50 ......................... 33 3.9 Normalized weighted FoM for SqueezeNet ......................... 33 3.10 Normalized weighted FoM for MobileNet ......................... 33 3.11 Normalized weighted FoM for SimpleNet ......................... 34 ix xvi ABBREVIATIONS L1 Level 1 memory system L2 Level 2 memory system MAC Multiply–Accumulate Operation mAP Mean Average Precision MAPE mean absolute percentage error MCU Microcontroller Unit MKL Math Kernel Library ML Machine Learning MMU Memory Management Unit MNIST Modified National Institute of Standards and Technology database MS COCO Microsoft Common Objects in Context NAS Neural Architecture Search NCS Neural Compute Stick ONNX Open Neural Network Exchange OLS Ordinary Least Squares OS Operating System OSD On-screen display interface PASCAL VOC Pattern Analysis, Statistical Modelling and Computational Learning, Visual Object Classes challenge PIR Passive Infrared Sensor PMU Performance Monitoring Unit POOL Pooling layer RAM Random Access Memory R-CNN Region CNN ReLU Rectified Linear Unit RMSprop Root Mean Square Propagation RoI Region of Interest RPC Remote Procedure Call RPi Raspberry Pi RSS Resident Set Size SD Secure Digital SGD Stochastic Gradient Descent SIMD Single Instruction Multiple Data SoC System-on-a-Chip SPP Spatial Pyramid Pooling SSD Single-Shot Detector SVHN Street View House Numbers dataset SWA Scottish Wildcat Action TF TensorFlow TLB Translation Lookaside Buffer TN True Negative TP True Positive TPU Tensor Processing Unit URL Uniform Resource Locator USS Unique Set Size VPF Vector Floating-Point register Abbreviations xvii VPU Vision Processing Unit YOLO You Only Look Once architecture Chapter 1 Introduction Back in 2012, AlexNet [1], a DNN fundamentally based on trainable convolutions, accomplished an unprecedented boost in accuracy in the ImageNet Large-Scale Visual Recognition Challenge [2]. Prior to that milestone, vision algorithms were ad-hoc pieces of engineering demanding painstaking efforts from senior practitioners to achieve moderate performance in real-world scenarios. AlexNet proved that highly accurate visual pipelines were possible. Since then, both academia [3] and industry [4] have been successfully pushing the limits of DNNs and creating a vast technological ecosystem to make the most of them. Actually, the number of hardware platforms, software frameworks, and model architectures tailored for DNN-based vision is overwhelming by now. From the point of view of application developers, the optimal selection of suitable components for prescribed specifications can be a really daunting task. One of the main objectives of this thesis is precisely to assist with this task. Neural networks are not a new concept. They were proposed back in the 1960s. In 1989, a major breakthrough based on these networks was accomplished: accurate visual recognition of hand-written digits [5]. However, no remarkable advances were reported for the next couple of decades until the irruption of AlexNet. Four aspects are commonly identified as the joint reason for the renaissance of neural networks, with special relevance in the field of computer vision: •Massive data available for training: DNNs require extensive datasets to learn feature maps at different abstraction levels that make accurate inference possible. Efforts in academia to create such datasets in addition to the potential of companies such as Facebook or Google to collect information from millions of users produced enough training material to improve the performance of DNNs significantly. •Massive compute capacity: advances in semiconductor technologies [6] and processing architectures [7] provide the computing horsepower to conduct training and inference in reasonable time scales. •Algorithmic techniques: new types of layers, operations, and interlayer connections have been progressively explored and incorporated in DNNs to increase their performance [8]. •Vast technological ecosystem for prototyping: in parallel with the three previous aspects, an extensive ecosystem of open-source tools and platforms has been developed by both academia and industry, enabling rapid prototyping of ideas and products. This in turn has spurred significant 1 2Introduction efforts in benchmarking [9] for fair comparison among the countless DNN-based implementations proposed in the literature. Early successes with DNN solutions (e.g., AlexNet for image recognition) encouraged the study of their application to other vision tasks. Gradually, DNNs have become the reference approach for a number of tasks – object detection and tracking, pixel segmentation, image enhancement, etc. – in contrast with the myriad of distinct options available for every task in classical computer vision. This convergence of procedures around DNNs together with the boost in accuracy are the major assets fueling this technological revolution. A common processing scheme for many tasks facilitates the exploitation of a particular software-hardware infrastructure across multiple scenarios, which in turn decreases the risk of product development, creating a virtuous cycle of innovation [10]. But everything has a cost. DNNs are computationally heavy and memory-hungry, not only during training but also when it comes to inferring in real deployments. This significant increase in consumption of hardware resources in turn leads to an energy consumption several orders of magnitude greater than that of classical algorithms [11]. Therefore, on the one hand, we have that DNNs constitute a highly accurate and flexible approach for artificial vision; on the other hand, DNNs are a computational burden for systems that can quickly deplete resources for other concurrent processes, especially in embedded devices. The challenge is clear: How can we lighten DNNs while keeping their accuracy and versatility? Pruning stands out as one of the most efficient techniques to address this challenge. It aims at removing unnecessary, or at least non-critical, operations within a DNN in order to reduce its computational complexity while ideally keeping accuracy unchanged or even increasing it. However, this technique must be jointly conducted with energy measurements to be effective [12]. Energy-aware quantization [13] is another technique with similar objective based on data precision scaling, having a major impact on memory management as well. In addition to these approaches, which act on the network models, solutions at system [14] or chip [15] level have also been proposed. In this thesis, we describe how bottlenecks in DNN processing can be identified by monitoring specific hardware metrics. We also present a methodology to predict the performance of a network layer by layer on a specific combination of hardware and software. Both tools provide meaningful information for direct action on those aspects of the system and network architecture that exert the greatest impact on the global operation, with a focus on runtime and energy consumption. We are just starting to envisage the possibilities that pervasive DNN-enabled artificial vision will bring about [16]. From permanent assistance to blind and visually impaired people [17], to drone autonomous navigation [18] or unattended shopping [19], deep learning is becoming a silent background technology in all kinds of practical scenarios. We also cover the application level in this thesis by exploring the field of remote animal monitoring. In collaboration with conservationists, the specifications for a smart camera trap were defined. Every step in the design and implementation process is carefully analyzed, from data curation to the final system realization, including DNN training parameterization and inference tuning. All in all, artificial intelligence, and its embodiment in the form of DNNs, will play – is already playing – an instrumental role in people’s daily lives, and the implications of this are being evaluated at all levels [20]. In particular, this thesis presents contributions to the practical realization of DNN-based embedded vision systems. The thesis is organized in seven chapters. Chapter 1introduces the context within which the rest of chapters elaborate on. Chapter 2surveys state-of-the-art DL components at both hardware and software levels, as well as a diversity of neural-network models. Chapter 3describes a benchmarkingbased methodology to select the optimum combination of deep neural network and software framework 3 for visual inference on embedded systems. In Chapter 4, we analyze how the hardware resources in a low-power processor are differently exploited by DL frameworks. We also determine key metrics to identify limitations and bottlenecks in particular realizations. Chapter 5demonstrates that it is possible to predict the performance of convolutional neural networks on embedded vision devices with high accuracy through the characterization of a single ad-hoc model. Chapter 6introduces a methodology which benefits from CPU-GPU heterogeneous hardware to execute neural-networks models while boosting inference performance. Finally, Chapter 7details a case study: remote animal monitoring. In this conclusive chapter, we apply some of the concepts and results reported in previous chapters to a real application scenario. Chapter 2 Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models The adoption of DL-based computer vision algorithms in IoT devices enables the development of accurate vision-enabled applications. This requires the integration of network models in embedded hardware and software. In this chapter, we review core technological components to achieve such integration in practice. Within the broad and ever-growing ecosystem of DL technological elements, we describe representative CNNs, libraries, and hardware architectures currently available. A global overview of this ecosystem is depicted in Fig. 2.1. Each application scenario is intimately related to a particular dataset to train a CNN architecture, which will be deployed on a framework supported by underlying libraries that interact with the hardware through the operating system. Image Dataset CNN Architecture Application Scenario Supporting Libraries Operating System Deep Learning Framework Hardware Platform Robotics Surveillance Industry ImageNet MS COCO CIFAR AlexNet MobileNet ResNet Caffe TensorFlow PyTorch OpenBLAS TensorRT cuBLAS Linux Android RTOS CPU GPU MCU Figure 2.1: Hierarchical stack of technological components involved in embedded realizations of DL. Numerous options are available at each level 5 6Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models 2.1 Networks models Convolutional Neural Networks (CNNs) mainly rely on the convolution operation to extract features. Their architecture is inspired by the processing flow inherent to visual perception in biological systems. The generic architecture of a CNN is illustrated in Fig. 2.2. Network training refers to the optimization process through which the learnable parameters (weights) of the CNN are adjusted to perform a visual task, e.g., classification, object detection, or segmentation. A large dataset and high computational power are required for training CNNs. Network inference (also called network execution or CNN forward) refers to the process through which a previously trained network infers on an input image. This inference produces a numerical output resulting from the layers’ operations and learnt weights. For instance, classification networks yield an output vector in which each element represents the probability of its corresponding category. CNNs are composed of a large number of layers operating on their corresponding input data. Various architectural principles are followed: (i) local connectivity between neurons, (ii) weight sharing for groups of connections, which reduces the number of parameters, and (iii) dimensionality reduction to retain only useful information. Each layer’s output constitutes the next layer’s input. Relevant aspects of CNN layers will be outlined in this section. The mathematical notation employed throughout the thesis will also be introduced. Input CONV1 POOL1 CONVn POOLn FC … Figure 2.2: Generic CNN architecture. Its core operation is the convolutional layer, but other operations, such as pooling or normalization, are also included in the network architecture. Fully-connected layers are usually inserted at the end of the CNN Feature maps The input and output data of each network layer comprise several channels of two-dimensional arrays; these are the so-called feature maps, fmaps, arranged in a 3D tensor. We will denote fmap dimensions as H×W×C, meaning a set of Hrows of activations1, with Wactivations per row, and Cchannels in the third dimension. Sub-indexes in and out will refer to input and output data from network operation layers. Most CNNs are fed with a 3-channel input image (in RGB format), which is the input for the first network layer. Then, the CNN progressively reduces the spatial dimensions (H×W) of the successive fmaps, while increasing the number of channels C(dimensionality reduction is thus achieved). Input and output fmaps will be denoted as Iand O, respectively, whereas Wdenotes the learnt network weights. Therefore, each layer takes a 3D input tensor I, performs operations that could involve a set of parameters W, and generates output data Ofor the next layer. 1Although the input of a network is given by the original image composed of pixels, subsequent output data from network layers are called activations and arranged into fmaps. 2.1. Networks models 7 These layer operations can be locally performed on sub-regions of the Ivolume called receptive fields (local connectivity). In this case, usually a kernel kh×kw×kcoperates over some inputs composing the receptive field (kcusually coincides with Cin). The layer will eventually operate over the whole input data if the kernel is progressively slid on I– so-called sliding windows with a stride of spixels (weight sharing). Fig. 2.3 exemplifies how a 3×3×3kernel filter (green array) is progressively applied on receptive fields of the H×W×3(RGB array) input image. In particular, this figure illustrates four positions of the sliding window (light areas) resulting from a unit stride s=1(to save space, this figure only shows a set of horizontal steps; the filter will eventually travel over the entire image). Therefore, the size of the output fmap Ois expressed as follows2: Hout =$Hin −kh+2p s +1%(2.1) where pdenotes an optional zero-padding that can be applied to enlarge the input in order to keep prescribed spatial dimensions. For instance, if the input contains Hin =224 rows, and the receptive field is characterized by k=3,s=1, without padding, then we will obtain an output with Hout =222. However, if a border with p=1is applied, the input dimensions will be kept constant, i.e., Hout =Hin =224. This is the so-called ‘same’ padding, given by p=(k−1)/2, which keeps the dimensions unmodified, as opposed to the so-called ‘valid’ padding, which adds no borders (p=0). Table 2.1 summarizes the explained parameters. Table 2.1: Notation used throughout this thesis for data and operations involved in CNNs Variable name Description I,O,WInput, output, and weight tensors Xi,j,kElement located at the (i,j,k)-th position of the 3D tensor X n(X)Number of elements in the tensor X H,W,CHeight and weight of a feature map, and number of channels kh,kh,kcKernel size: height, width, and number of channels s,pStride and padding used for sliding operations Figure 2.3: Processing based on a sliding window. The green array is the operation kernel (implementing filtering, pooling, etc.), whereas the RGB array is the input image. The process of sliding the kernel along the horizontal dimension of the input is depicted with light areas. The kernel will be eventually applied to the whole image by sliding in the vertical dimension as well Common layers Three types of layers are usually included in CNNs: convolutional, pooling, and fully-connected. They are briefly described next together with other types that can also be found in modern networks. •Convolutional (CONV). A set of Nkernels of dimensions kh×kw×Cin filters the input fmap. In the 4D tensor W, the n-th kernel W(n),n=1, ..., Nyields the n-th 2D output feature map in which 2A similar equation for the horizontal dimension is obtained by replacing Hfor W. 14 Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models resolution multiplier ρ∈(0,1] – which reduces the height and width of the squared input image. These multipliers have the effect of reducing the computational cost by α2and ρ2, respectively. The baseline MobileNet architecture sets these factors to 1. ShuffleNet [39, 40] uses two techniques to reduce the computational load, namely pointwise group CONV (as introduced in AlexNet) and channel shuffle. Its main building block, i.e., the ShuffleNet unit, is composed of a group pointwise CONV, a channel shuffle layer, a 3x3 deptwise CONV, and another pointwise group CONV Therefore, each group CONV splits the fmaps into G groups and applies a depthwise kernel to each group. In addition, the channel shuffle operation is performed to randomly combine information between groups. 2.1.3.2 Object-detection networks Concerning object-detection networks, their architectures can be classified into two-stage and one-stage approaches. These networks are usually trained on two popular datasets for object detection: PASCAL VOC [27] (with bounding boxes belonging to 20 object categories), or MS COCO dataset [29] (including 80 classes). •Two-stage object detectors first generate class-agnostic regions of interest (ROI) and then classify them into object categories. Relevant two-stage architectures include Region CNN (R-CNN) [41], fast-RCNN [42], faster R-CNN [43], region-based fully convolutional network (R-FCN) [44], and spatial pyramid pooling (SPP-net) [45]. These networks usually achieve higher accuracy than one-stage approaches, but with slower inference. Many architectures were inspired by the SSD [46] one-stage detector. This network searches for bounding boxes with a series of established aspect ratios and scales. The object-size variability is handled by using multiple-resolution fmaps as prediction inputs. The architecture includes (i) a backbone network for extracting image features, and (ii) extra feature layers for predicting the bounding boxes and their associated confidences (i.e., the ‘SSD head’ located at the end of the network). For instance, MobileNet-SSD uses MobileNet as the backbone, thus becoming a high-performance model. •One-stage approaches combine the two aforementioned stages into a single end-to-end trainable network. Therefore, they are more computationally efficient models, although some accuracy is traded off. Relevant one-stage models include the single-shot detector (SSD) [46], YOLO [47, 48, 49], CornerNet [50, 51], CenterNet [52], ExtremeNet [53], RetinaNet [54], and EfficientNet [55]. YOLO series (‘You Only Look Once’) [47, 48, 49] was the pioneer one-stage model, becoming a widely-extended real-time detector. It divides the input image into an SxS grid of cells; the output is given in terms of confidences for multiple categories and bounding boxes – each cell is responsible for detecting objects centered in it. Using different stride sizes, YOLO can detect objects at three scales: 13x13 (small), 26x26 (medium) and 52x52 (large). Its backbone network for feature extraction is the so-called DarkNet-53 architecture (53 CONV layers) [56] – plus detection modules. Tiny-YOLO [56] and YOLO-LITE are light-weight versions of YOLO. 2.2. Software 15 2.1.4 Training convolutional neural networks Given a set of labeled samples (images), DL training based on supervised learning is an optimization process that minimizes a function. For instance, classification networks usually minimize the differences between predictions and true labels calculated with the cross-entropy function. It is common to use the Stochastic Gradient Descent (SGD) for this optimization problem. This method iteratively takes subsets of samples (batch) to update the parameters (network weights) in the opposite direction of the gradient of the objective function w.r.t. to the parameters. The initialization of the network parameters can be randomly performed, or may be taken from a pre-trained network – this is the so-called transfer learning approach. Then, a weight update rule is employed to adapt the weights in each iteration. This optimizer rule [57] usually depends on training parameters such as learning rate and momentum, which control the speed in the gradient descend. The gradient of the loss function w.r.t the weigths  ∇f(W) is calculated for each layer in the network using backpropagation – from top layers to initial ones in the CNN. For instance, two simple weight update rules are defined as follows: W(i+1) =W(i)−η ∇f(W(i))(2.16) W(i+1) =W(i)+µ(W(i)−W(i−1))−η ∇f(W(i))(2.17) where W(i)represents the weights at iteration i, and ηand µare the learning rate and momentum, respectively. Note that at each iteration i, only a subset of the dataset is employed to calculate  ∇f(W). More refined weight optimization algorithms are Nesterov, Adagrad, Adadelta, Adam, RMSProp, etc. [57]. The majority of this thesis will focus on pre-trained neural networks ready for embedded inference. Notwithstanding, specific application scenarios require CNN training on particular datasets. In this regard, network training will be covered in Chap. 7, which comprehensively elaborates and exemplifies the process. 2.2 Software We can find a notable number of software libraries that facilitate the design, training, and execution of CNNs. Some of the most popular ones are summarized next. Caffe [58] was developed by the Berkeley Vision and Learning Center (BVLC). This is a very popular tool that indeed offers the network implementation of an extensive number of state-of-the-art CNNs. Pre-trained weights for these architectures are also provided and supported by developers [59]. Its core code is written in C++, but it also provides Python and Matlab APIs. TensorFlow [60] (TF) was released by Google as an open-source library for developing ML applications. The set of computational operations (layers) is expressed as a stateful dataflow graph based on tensors (activations). Differentiable programming on that graph allows model training, whereas graph execution enables image inference. C++ and Java interfaces are available. However, most of TF’s open-source code is provided in Python. This framework has become a reference tool for ML development. In addition, Google also developed TensorFlow Lite for model implementation on embedded devices. This variant specifically targets network inference, reducing the latency and model size. 16 Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models Caffe2 [61], released by Facebook AI Research group, was based on Caffe code but with a focus on mobile deployment. The network definition is expressed as computation graphs. This light-weight and modular framework specifically supports embedded operating systems such as Android, Raspbian, and iOS. PyTorch [62, 63] was also developed by Facebook AI. It is built on the Torch library. A software package called PyTorch Mobile was also released to deploy models on embedded devices using Android or iOs. In 2018, Facebook AI merged Caffe2 and PyTorch projects, so Caffe2 is now part of PyTorch. OpenCV [64] (Open Source Computer Vision Library) is a open-source library that includes a large number of functions and modules for CV mainly supporting classical algorithms. Concerning DL, the library implements network inference on its ‘DNN module’. CNNs can be loaded from a diversity framework formats – e.g., models trained on Caffe, TF, Torch, etc. – and then executed on OpenCV just to run inference. This library offers C++, Python, and Java APIs, and is compatible with several platforms – Linux, Windows, iOS, Android, etc. Other alternatives to the above frameworks can be pointed out, such as MXNet [65], TVM [66], Paddle / Paddle-Lite (PArallel Distributed Deep LEarning) [67, 68], Theano [69], Mobile AI Compute Engine (MACE) [70], MNN (Mobile Neural Network) [71, 72], and Tengine [73]. 2.2.1 Software implementation strategies Althoughtheinputsandoutputsofconvolutional networks aretensors, their computationalimplementation and memory allocation can be performed in several ways. For instance, a usual approach for storing image data is using a row-major C×H×Wlayout, in which activations from the same row ([0,1,2, ..., W−1]) are physically adjacent in memory, whereas fetching channel information implies discontinuous memory accesses. By contrast, column-major layout allocates elements from the same column adjacent in memory. In addition, there are different ways to implement each CNN layer. For instance, a simple six-level nested loop overCin −Hin−Win−Cout −Hout −Wout can perform the CONV operation defined in Eq. (2.2). However, this naive implementation can be enhanced by restructuring the data or calling a matrix-multiplication subroutine. For instance, vectorizing input data or changing the loop order can exploit local data reuse on the registers and the cache hierarchy. A common approach to implement convolutions involves the use of highly-optimized general matrix-to- matrix multiplication (GEMM) functions, such as those provided by Basic Linear Algebra Subroutines (BLAS) [74, 75, 76, 77] libraries: ATLAS [78], OpenBLAS [76], MKL [79], Eigen [80], cuBLAS [81], cuDNN [82], etc. To leverage such routines, image-to-column transformation, known as im2col or matrix lowering, is first performed [83, 84]. This data re-order operation expands the 3D input volume into a large 2D matrix, containing one receptive field of khkwCin elements per column (i.e., an array with shape n(W)×(HoutWout)). Convolution weights are also organized into 2D arrays Cout×n(W)– see Fig. 2.6. Thus, the convolution becomes a large matrix-to-matrix multiplication expressed as Ounrolled =WunrolledIunrolled. Reading the ‘unrolled’ input matrix involves sequential accesses to memory, which is more efficient and also essential for exploiting SIMD and parallel execution. However, note that this approach inherently leads to a trade-off between memory consumption and computational efficiency, because oversized arrays that includes duplicate data are built. These arrays are larger than the original input tensor. 2.2. Software 17 Likewise, FC layers can also be implemented with GEMM routines. Layer weights Wcan be arranged into a matrix sized n(O)×n(I), then FC operation becomes a matrix-to-vector multiplication expressed as O=WI (see Fig. 2.7). Input data Kernel weights Figure 2.6: Data reshaping on input data and kernel weights – so-called im2col and im2row. Each receptive field on an input fmap sized kh×kw×Cin becomes a column of an enlarged matrix. Similarly, weights on each 3D filter are flattened into a row. Thus, convolution becomes a matrix-to-matrix product Figure 2.7: Left: FC layer operating on 9 input activations and producing a 5-neuron output vector. One weight connection is applied between each input-output pair (these weights are not included to avoid clutter in the graph). Right: implementation of this layer as the matrix-product of an array of weights (green) and the input vector (blue). For instance, the dot product of the weights highlighted in the second row and the column of input activations produces the second output of this FC layer Other approaches to accelerate matrix multiplication in CNNs include algorithms based on the Fast Fourier Transform (FFT) [85], Winograd [86], and Strassen [87]. 2.2.2 Additional software tools There exist open-source tools that facilitate the process of designing, training, and integrating CNNs in embedded systems. •Network visualization tools, such as Netscope [88] and Netron [89], help to graphically understand the architecture of the networks. •Image annotation and labeling software, such as LabelImg [90] and LabelMe [91], among others [28]. These tools allow, for instance, drawing bounding boxes to create object-detection training datasets. 18 Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models •Framework format conversion tools. For example, ONNX [92] enables library format conversion – involving some of the libraries described in Sect. 2.2 – through an intermediate representation. This is convenient when different frameworks are used during the training and deployment stages. •The operating system (OS) installed on an embedded device can significantly simplify the management of platform resources. Linux is popular for embedded platforms owing to its open-source code, which allows creating variants tailored for specific hardware. Mobile and wearable devices commonly run Android or iOS. Microcontrollers can also employ OS specifically designed for their architectures. 2.3 Hardware devices and accelerators The heterogeneity in computing architectures is translated into a variety of hardware features. Indeed, each architecture can be particularly exploited for prescribed workloads on which optimal performance is achieved [93]. CPU-based devices Central processing units (CPU) constitute a ubiquitous compute architecture with broad library support. Although the unit is designed to process instructions sequentially, more efficient program execution is possible by exploiting instruction-level parallelism. Multi-thread processors and CPUs supporting single-instruction multiple-data (SIMD) pipelines speed up program execution by performing several instructions per clock cycle. To this end, accurate branch prediction mechanisms are incorporated – once the dependency graph of the program is analyzed, various instructions are fetched and executed in parallel. Finally, an evident advantage of this architecture is the reduced data-transfer latency when compared to external accelerators or GPUs. They are also easy to program, which make them attractive for non-expert users. This thesis will particularize various inference studies on Raspberry Pi (RPi) platforms [94], a series of low-cost general-purpose embedded devices. For instance, the popular RPi model 3B [95] features a quad-core ARM Cortex-A53 1.2GHz 64-bit CPU, with 1GB RAM LPDDR2 volatile memory, enough for loading and running most light-weight CNNs. Note that this is a low-power device, with a consumption below 5 W when the four cores are at full operation. There is a myriad of providers in the market offering general-purpose CPU-based embedded devices, such as the Odroid series [96], LattePanda [97], Orange Pi series [98], Beagleboard series [99], etc. In this regard, Cloutier et al. [100] provides an overview of ARM-based embedded systems including computational-performance and power-related metrics. GPU-based devices Graphics processing units (GPU) are specialized processors optimized for operation on data vectors. Data processing on these units relies on single-instruction multiple-threads (SIMT) execution. GPUs are composed of numerous parallel ‘execution units’ (small and efficient cores that can process SIMD instruction streams), which independently process massive data. In general, GPUs offer high processing performance on large workloads – they are not designed for irregular workflows. Notwithstanding, these high computing capabilities usually come at the cost of higher power consumption. 2.3. Hardware devices and accelerators 19 A popular embedded GPU-based platform is Nvidia Jetson Nano [101], which comprises a 128-core Maxwell GPU and a volatile 4 GB 64-bit LPDDR4 memory (in addition to a quad-core ARM A57 1.43 GHz CPU). More powerful Nvidia devices include Nvidia Jetson Xavier NX [102] – which is a high-power Nvidia CUDA GPU edge platform – and Nvidia Jetson TX2 series [103] – which trade off computational capacity (256-core Nvidia Pascal GPU) for a higher power demand (from 7.5 W to 20 W). These platforms rely on the TensorRT library to perform CNN inference. In Chap. 6, we will focus on embedded vision exploiting GPU-based heterogeneous platforms. MCU devices Microcontroller units (MCU) constitute a valid option for building IoT systems. They enable applications with very low energy budgets but at the cost of extremely scarce computational and memory resources. Still, these simple computing architectures can be effectively exploited by ML libraries, such as CMSISNN [104], which leverages hardware-optimized kernels on an MCU. Light-weights CNNs, such as LeNet, can run on these processors [105, 106]. Examples of edge devices incorporating the ubiquitous ARM Cortex-M architecture include: Arduino Nano 33 BLE [107, 108] (featuring a 32-bit ARM Cortex-M4 CPU 64MHz), STM32 microcontrollers [109] (also providing software libraries for embedded DNNs [110]), or SparkFun Edge [111] and AdaFruit Edge Badge [112] (both including a single-core ARM Cortex-M4F processor compatible with TensorFlow Lite). Other Architectures and Accelerators DL pipelines can leverage other existing computing architectures. For instance, field-programmable gate arrays (FPGAs) are specific hardware components composed of processing units that can be reprogrammed via software. This technology offers high flexibility, allowing custom instructions and data types. DL accelerators arise as a hardware solution applicable to multiple IoT scenarios. These specific circuits provide high performance on DL workloads, although they may not support all operations – programmability is limited. For instance, we briefly describe two popular DL accelerators – compatible with CNN visual inference – which can be integrated in edge systems: •Google Coral Accelerator [113]. This device features a Google edge TPU co-processor (tensor processing unit, i.e., an array of processors), designed to provide high-performance inference on mobile CNN models. The device can be attached to any system running Debian Linux OS (e.g., RPi); it is compatible with TensorFlow Lite sofware (also from Google). •Intel Neural Compute Stick (NCS). This USB accelerator includes a dedicated Myriad X vision processing unit. The Intel NCS2 release [114] outperforms its predecessor Intel-Movidius NCS [115]. This company also provides the OpenVINO Toolkit, a library compatible with this device that supports a variety of DL operations. Moreover, some platforms include both general-purpose CPU and AI accelerators: •Google Coral Dev Board [116]. In addition to its quad-core Cortex-A53 CPU and its integrated GC7000 Lite Graphics GPU, this device features a tensor processing unit (TPU), which is an application-specific integrated circuit (ASIC) for accelerating DNNs. It is compatible with Google’s TensorFlow framework as well. 20 Embedded vision for the Internet of Things: a survey on state-of-the-art hardware, software, and deep-learning models •Tinker Edge R board [117]. Its ARM big.LITTLE technology comprises a hexa-core CPU (dualcore ARM Cortex-A72 1.8 GHz plus quad-core ARM Cortex-A53 1.4 GHz). It also integrates a neural processing unit, the Rockchip NPU, a ML accelerator that supports various software frameworks. 2.4 Summary and Related Topics Many commercial applications require complex DL-based solutions. Deeper networks usually offer more accuracy, but challenge – or even prevent – their embedded realization. Therefore, understanding the trade-offs between different CNN architectures and implementations is critical. This thesis addresses the challenges associated with the heterogeneity of existing CNN architectures, software, and hardware components. Research on component benchmarking The extensive ecosystem of architectures, datasets, DL frameworks, software libraries and tools, implementation strategies, and hardware alternatives calls for benchmarking and comparison. For instance, concerning embedded platforms running open-source or vendor-provided DL software, experimental evaluations have been conducted. Thus, we can find benchmarking and characterizations for specific DL accelerators [118], smartphones [119, 120, 121], GPU-based devices [122], and other hardware devices [123, 124, 125, 126, 127, 128, 129, 130]. With respect to CNNs, both image classification [30, 131] and object detection [132] networks have been analyzed in terms of accuracy, runtime, computational complexity, etc. Research on Optimization The resource and memory constraints of edge devices set the need for optimizing and compressing CNN architectures. Optimization strategies can be globally categorized into four approaches: •Network pruning. This technique identifies the parameters which less contribute to the output to remove them, in principle with a slight accuracy loss [133, 134, 135, 136, 137, 138, 139]. Therefore, the model size and memory footprint are reduced. A common and simple way to apply pruning is by removing weights below a threshold. Then, a dense network may become a sparse one. However, irregular sparsity of kernels is difficult to accelerate on most hardware. •Parameter quantization. The objective is to reduce the memory requirements by constraining the number of bits to represent each weight, i.e., by reducing the range of possible unique values (2n for nbits). Several data precisions (float32, int8, 4 bits, 2 bits, or binary) can be used to reduce the model size [140, 141, 142, 143, 144, 145]. Note that a smaller data size implies greater information loss. In these quantization approaches, two key choices are (i) the number of bits, and (ii) how to allocate bits to cover the quantization range. For instance, the popular uniform-scale quantization technique – implemented in frameworks such as TensorFlow and PyTorch – first scales/shifts the values to fit into the [−2n−1,2n−1]range, and then rounds each value to its nearest quantization step. Owing to the limited number of quantization steps (2n), an inherent quantization error emerges in the process. 2.4. Summary and Related Topics 21 •Low-rank approximation, or tensor decomposition. This approach divides large weight arrays into several smaller matrices, aiming at reducing the information loss as much as possible. Different methods have been proposed, such as Tucker and CP decomposition of 4D kernel tensors [146, 147], structured transformation into circulant or Toeplitz-like matrices [148], or singular value decomposition [149]. •Network architecture search (NAS) [150]. Instead of manual design of networks, current research is moving towards neural-architecture-search algorithms [151, 152, 153, 154, 155, 156, 157]. The NAS approach treats model design or optimization as a search task. The objective is to automatically find optimal architectures according to specific performance criteria by systematically exploring the design space while using a performance estimation strategy. The search space, whose variables could be for instance the filter size or the number of kernels, might be extremely large. Therefore, this technique is resource-intensive and time-consuming. Note that the implementation of layers provided by DL frameworks such as those described in Sect. 2.2 does not support specific optimization methods (as listed above) in most cases. Indeed, leveraging the underlying hardware with optimized CNNs may be challenging. For instance, some bit-widths could not be supported by a particular device. Chapter 3 Optimal selection of software and models for visual inference The vast technological ecosystem of components for visual inference on edge devices makes it difficult to determine the best option according to the target application specifications. For instance, power consumption is crucial for edge smart cameras whereas real-time processing may be a requisite for other applications. This chapter describes two graphical methodologies to select the optimum combination of convolutional neural network and software framework for boosting visual performance on embedded systems. Three indicators of inference performance – accuracy, throughput and power consumption – will represent the decision variables for optimization. Both single-objective and multi-objective problems regarding those metrics are introduced in this chapter. Prior to conducting optimization, real values of inference performance must be obtained by benchmarking a diversity of components. In particular, the proposed methodologies are illustrated through a study comprising six popular network models running on four Deep Learning frameworks implemented on a low-cost embedded platform. In the first methodology, Pareto-optimal solutions – i.e., network–framework combinations – enable pairwise metric assessment. When more than two application-level parameters come into play, we propose a second methodology based on the definition of a Figure of Merit that will facilitate optimal decisions. In fact, several application scenarios can be evaluated by sweeping the relative importance of performance variables. These procedures reveal that only a reduced subset of network–software pairs should actually be considered for real deployment when targeting a specific embedded application. 3.1 Methodology Overview As stated in Chap. 2, there is a growing spectrum of Deep Learning technological components aiming at enhancing aspects such as training, inference performance, ease of use, and software compatibility. In practice, practitioners must choose within an extensive set of network architectures, software frameworks, acceleration libraries, hardware accelerators, image sensors, etc. Although this variety certainly covers a wide range of specifications, meeting stringent application-level requirements in embedded platforms can be difficult. The adoption of non-optimal components could lead to the apparent impossibility of 23 30 Optimal selection of software and models for visual inference 2.8 3.0 3.2 3.4 3.6 3.8 Mean Power Consumption (W) 0 1 2 3 4 5 Mean Throughput (fps) Network in Network GoogLeNet ResNet-50 SqueezeNet MobileNet SimpleNet Caffe TensorFlow OpenCV Caffe2 Figure 3.3: Throughput performance versus power consumption for each network–framework combination and Pareto frontier of optimal solutions 2.8 3.0 3.2 3.4 3.6 3.8 Mean Power Consumption (W) 0.78 0.80 0.82 0.84 0.86 0.88 0.90 0.92 Top-5 Accuracy (%) Network in Network GoogLeNet ResNet-50 SqueezeNet MobileNet SimpleNet Caffe TensorFlow OpenCV Caffe2 Figure 3.4: Network accuracy versus power consumption for each network–framework combination and Pareto frontier of optimal solutions Table 3.4: Optimum single-objective selection, and Pareto-optimal dual-objective selections Variables Optimal Combinations Reference Figure Throughput SqueezeNet–OpenCV Fig. 3.1a Power Consumption MobileNet–Caffe2 Fig. 3.1b Top-5 Accuracy ResNet-50–Caffe/OpenCV/TF Fig. 3.1c Throughput – Accuracy SqueezeNet–OpenCV Fig. 3.2 MobileNet–OpenCV MobileNet–TensorFlow ResNet-50–TensorFlow Throughput – Power Consumption SqueezeNet–OpenCV Fig. 3.3 SqueezeNet–TensorFlow MobileNet–Caffe2 Accuracy – Power Consumption MobileNet–Caffe2 Fig. 3.4 ResNet-50–Caffe2 We can reduce the multi-objective optimization problem to a single-objective one by introducing a combination of the variables. Definition. The ‘number of correctly inferred images per joule’ can be obtained from a Figure of Merit aggregating the three performance parameters under study: 3.3. Optimum Selection 31 FoM =Accuracy ·Throughput Power (3.1) This meaningful metric allows the joint assessment of the computational and energy efficiency of networks and frameworks on the considered hardware platform. The FoM in Eq. (3.1) calculated for values in Table 3.3 – or Figs. 3.1a–3.1c – enables a single and quick comparison between solutions. Fig. 3.5 depicts the single-criteria evaluation of networks and frameworks accomplished by defining this FoM. Caffe TensorFlow OpenCV Caffe2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 # correctly inferred images (top-5) per joule 0.44 0.19 0.10 0.70 0.29 0.26 0.59 0.44 0.19 1.16 0.67 0.38 0.37 0.32 0.13 1.21 0.71 0.15 0.59 0.27 0.17 0.96 0.56 0.34 Network in Network GoogLeNet ResNet-50 SqueezeNet MobileNet SimpleNet Figure 3.5: Figure of Merit defined in Eq. (3.1). It is expressed as ‘number of correctly inferred images per joule’ According to the FoM in Eq. (3.1), SqueezeNet running on OpenCV performs the best. The superiority of this solution arises from achieving a superior frame rate in a power-efficient way, while trading off some accuracy in comparison with other networks. The question is: would this solution still be the best choice for a particular application where accuracy is more important, in relative terms, than throughput and/or power consumption? 3.3.2.1 Weighted FoM In order to answer the question above, the relative importance of each performance metric on high-level specifications will define a possible application scenario. Definition. Relative weights of evaluation parameters – accuracy, throughput, power – can be specified by a vector [αA, αT, αP]satisfying X i αi=1, αi∈[0,1] (3.2) where icorresponds to A,T,P, which stand for accuracy, throughput and power consumption. Itfollows fromEq.(3.2)that, for instance, the triplet [αA, αT, αP]=[0.35,0.05,0.60] defines an application scenario where power consumption presents a relative preeminence of 60% on the targeted performance, followed by accuracy with a weight of 35% and throughput with very little importance, only 5%. Definition. Each specific weighted combination of the variables (i.e., a vector of αi) must produce a single metric. A new FoM parameterized by the set of αican be defined as follows: FoM(αA, αT, αP)=Accuracy3αA·Throughput3αT Power3αP(3.3) 32 Optimal selection of software and models for visual inference Therefore, Eq. (3.1) is just a particular case of the generalized FoM introduced in Eq. (3.3): when the three performance parameters have the same impact on application-level requirements (αi=1/3∀i). Any other distribution of αiwill emphasize or undermine the specific performance variable by setting its exponent to a value greater or less than unity. Extreme cases occur when any αiis set to 1, boosting just one performance parameter on the FoM at the cost of completely dismissing the other ones. These singular scenarios have already been identified as single-objective selections in Table 3.4. Hence, the choice of αivalues highly influences the final decision. Now, the objective is to find the optimum network–framework choice per application scenario, that is, per possible combination of relative weights for accuracy, throughput and power consumption. 3.3.2.2 Results The parameterized FoM(αA, αT, αP)can be obtained for each technological combination per application scenario by sweeping αi. Note that, for these calculations, we make use again of the baseline experimental measurements presented in Sect. 3.2.3. Graphical representations of FoM values for each network–framework pair as a function of αiare depicted in Figs. 3.6–3.114. In fact, for each combination of αi, FoM values are normalized with respect to its maximum value obtained for the optimum network–framework pair. Thus, the best selection exhibits aFoM =1whereas the range of FoM values allows to easily evaluate how far each solution is from the optimum one through color degradation from blue (representing the maximum, 1) towards lighter tones (solutions getting far from this maximum). For a particular application scenario, that is, for a specific point within the exploration domain of αi, a single-variable comparison similar to that of Fig. 3.5 can be extracted. For instance, the scenario defined by the triplet previously mentioned, [αA, αT, αP]= [0.35,0.05,0.6], is characterized by values in Fig. 3.12. In contrast to Fig. 3.5 where all metrics were equally important, MobileNet on Caffe2 is the best choice in this case. 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T TensorFlow 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T OpenCV 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Figure 3.6: Normalized weighted FoM for Network in Network The graphical representation that summarizes the methodology for optimal selection based on the proposed weighted FoM in multiple application scenarios is shown in Fig. 3.13. Specifically, Fig. 3.13 shows the best selection, i.e., the network–framework pair reaching the maximum FoM, for the whole domain of αi. On the opposite side, Fig. 3.14a overviews which combinations perform the worst across the domain of application scenarios. The corresponding lowest values of FoM are shown in Fig. 3.14b. Light tones in this plot reveal that there is a wide range of FoM across technological solutions, highlighting that some network–framework combinations can be really far from optimum performance. 4Note that the plot axes represent αAand αTsince, according to Eq. (3.2), αP=1−αA−αT. 3.3. Optimum Selection 33 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T TensorFlow 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T OpenCV 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Figure 3.7: Normalized weighted FoM for GoogLeNet 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T TensorFlow 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T OpenCV 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Figure 3.8: Normalized weighted FoM for ResNet-50 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T TensorFlow 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T OpenCV 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Figure 3.9: Normalized weighted FoM for SqueezeNet 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T TensorFlow 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T OpenCV 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Figure 3.10: Normalized weighted FoM for MobileNet For all the domain of scenarios, the percentages of solutions achieving optimum and worst performance in Figs. 3.13 and 3.14a are reported in Table 3.5. 34 Optimal selection of software and models for visual inference 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T TensorFlow 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T OpenCV 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T Caffe2 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Figure 3.11: Normalized weighted FoM for SimpleNet Caffe TensorFlow OpenCV Caffe2 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Weighted FoM 0.57 0.53 0.5 0.68 0.65 0.54 0.670.68 0.64 1.0 0.86 0.63 0.74 0.63 0.59 0.890.87 0.56 0.69 0.82 0.69 0.88 0.99 0.63 Network in Network GoogLeNet ResNet-50 SqueezeNet MobileNet SimpleNet Figure 3.12: Normalized values of the FoM defined in Eq. (3.3) for a scenario defined by relative preeminence of power consumption, throughput and accuracy of 60%, 35% and 5%, respectively. In this particular case, MobileNet on Caffe2 is the best choice Table 3.5: Left: optimum network–framework selections according to FoM in Eq. 3.3 when sweeping αi, and corresponding percentage of scenarios (area covered by optimum solutions in Fig. 3.13). Right: worst selections and corresponding areas in Fig. 3.14a Optimum Network–framework Cases (%) Last-place Network–Framework Cases (%) ResNet-50–Caffe/OpenCV/Caffe2 0.00aSimpleNet–Caffe 0.04 ResNet-50–TensorFlow 0.04 Network in Network–OpenCV 0.43 ResNet-50–Caffe2 0.25 GoogLeNet–Caffe 1.76 MobileNet–OpenCV 1.27 SimpleNet–OpenCV 5.77 MobileNet–TensorFlow 2.58 Network in Network–Caffe 5.95 MobileNet–Caffe2 9.61 ResNet-50–Caffe 86.05 SqueezeNet–TensorFlow 28.17 SqueezeNet–OpenCV 58.09 aSingular case just for αA→1. These three combinations achieve the highest accuracy and thus maximize FoM(αA=1, αT= 0, αP=0) – see Table 3.3. 3.3.2.3 Analysis The corners on the graphical representation in Fig. 3.13 represent single-objective optimization scenarios. Any other vector of αidefining the application is assessed through the FoM in Eq. (3.3). These results reveal that SqueezeNet performs the best for the majority of the explored scenarios – in conjunction with OpenCV for high frame rates and with TensorFlow for energy efficiency. For relaxed frame-rate requirements, MobileNet is the best option on combining high accuracy and reduced power consumption. In practical terms, only 3 combinations – SqueezeNet on OpenCV and TensorFlow; MobileNet on Caffe2 – cover more than 95% of cases. On the other hand, Network in Network, GoogLeNet and SimpleNet 3.4. Summary and Related Topics 35 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 A 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 T P = 0.9 P = 0.8 P = 0.7 P = 0.6 P = 0.5 P = 0.4 P = 0.3 P = 0.2 P = 0.1 ResNet-50 - Caffe ResNet-50 - TensorFlow ResNet-50 - Caffe2 SqueezeNet - TensorFlow SqueezeNet - OpenCV MobileNet - TensorFlow MobileNet - OpenCV MobileNet - Caffe2 ResNet-50 - Caffe/OpenCV/Caffe2 Figure 3.13: Optimal network–framework selection according to the weighted FoM defined in Eq. (3.3). Only a subset of possible solutions perform the best in at least one point of the exploration domain are discarded in all scenarios, no matter the software framework considered. Actually, they are the worst solutions in several cases, as shown in Fig. 3.14a. Network in Network features the lowest accuracy, a moderate frame rate, and also high power consumption, specially in combination with Caffe. SimpleNet also offers a deficient performance in the three considered metrics. Despite the notable accuracy of GoogLeNet, it exhibits a poor performance in the other aspects, being the most energy-demanding model when running on Caffe. Concerning ResNet-50, this model performs with high accuracy but presents the lowest throughput and high power demand. This is why it is the worst model in more than 85% of the scenarios – see Fig. 3.14a. In particular, its notably low throughput (0.41 pfs) compared with the best one (4.79 fps for SqueezeNet–OpenCV) makes the normalized FoM in Fig. 3.14b degrade for increasing values of αT. 3.4 Summary and Related Topics This chapter exemplifies two graphical methodologies for optimal selection of software tools and networks for embedded inference. First, measured metrics for several networks and software tools on the popular Raspberry Pi 3B demonstrate the capability of low-cost embedded platforms for implementing CNN image classification. Second, the presented methodologies filter out most of the studied alternatives, thus significantly simplifying informed system decisions. Certainly, only 9 network–software selections perform the best in practical scenarios. In fact, this reduced number of cases has been pointed out by both methodologies, i.e., Pareto-optimal choices and weighted FoM maximization in Sects. 3.3.1 and 3.3.2. Other methodologies can be applied. For instance, additional parameters, such as temperature and memory, can be taken into account in the optimization decision as problem restrictions. Note also that the presented approaches are strictly quantitative since they are exclusively based on measurable performance. However, practitioners must also deal with qualitative parameters encoding the suitability 36 Optimal selection of software and models for visual inference 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 A 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 T P = 0.9 P = 0.8 P = 0.7 P = 0.6 P = 0.5 P = 0.4 P = 0.3 P = 0.2 P = 0.1 Network in Network - Caffe Network in Network - OpenCV GoogLeNet - Caffe ResNet-50 - Caffe SimpleNet - Caffe SimpleNet - OpenCV (a) Worst network–framework selection according to the weighted FoM 0.0 0.2 0.4 0.6 0.8 1.0 A 0.0 0.2 0.4 0.6 0.8 1.0 T 0.00 0.09 0.18 0.27 0.36 0.45 0.54 0.63 (b) Corresponding lowest values of the normalized FoM Figure 3.14: Lowest-FoM network–framework combinations of a particular framework or hardware platform according to, for instance, a long-term technological strategy in a company. Platform size, weight or cost are also relevant. These extra restrictions would mainly affect the benchmarking process. In fact, instead of manually evaluating each network–software–platform, available public benchmarks can assist this step – see Chap. 2. Alternative definitions of Figures of Merit can also be beneficial. For example, most public benchmarks only include inference time and network accuracy as measured metrics. This allows to define a FoM as 3.4. Summary and Related Topics 37 ‘number of correct images per second’ [181]: FoM =Accuracy ·Throughput (3.4) or ‘number of valid FLOPs5per second’: FoM =Accuracy ·FLOPs ·Throughput (3.5) Note. This chapter is partially based on a previous authors’ publication [182]. 5FLOPs stands for the number of floating-point operations per inference, reflecting the computational complexity of the network. Chapter 4 Relevant hardware metrics for performance evaluation This chapter presents a dual approach to assess embedded network inference performance. On the one hand, low-level hardware exploitation metrics reveal how the resources in the processor are exploited during visual inference. On the other hand, several performance parameters – throughput, power consumption, memory footprint, or CPU utilization – describe the actual system performance. More specifically, both CNN architectures – with diverse complexity and accuracy – and deep learning libraries – which adopt various coding strategies – show notable differences in terms of microarchitecture metrics when experimentally assessed. In addition, these software–hardware combinations also present a dissimilar behaviour revealed by heterogeneous high-level performance metrics. Therefore, the proposed hardware-aware analysis is a powerful tool to understand software performance and find limitations and bottlenecks. In sum, a bottom-up approach for identification of correlation between measurable hardware statistics and performance figures provides insight to improve the performance of software components for network inference. 4.1 Introduction The variety of existing DL frameworks emerges from attempts to enhance certain system aspects while globally targeting the same functionality. For instance, frameworks tailored for developing DL models aim tospeedup thetraining processof CNNs, particularly onhigh-end GPU-basedsystems. Likewise, software tools designed to boost edge inference at deployment stage must make the most of embedded devices, always paying attention to their CPUprocessors and memory hierarchy. Indeed, that ecosystem of software tools exploits several optimization libraries to deal with the demanding computational requirements of CNNs. As explained in Chap. 2,unrolled convolution is a quite common computational approach that implements the convolution operation through im2col data re-order followed by GEMM. In short, CONVs and FCs composing CNNs can be expressed as matrix-matrix or matrix-vector operations, respectively. These operations can be optimally calculated using BLAS methods [74, 75], which are efficiently implemented by several libraries underlying the frameworks, e.g., ATLAS [78], OpenBLAS 39 46 Relevant hardware metrics for performance evaluation Temporal behaviour of the CPU when running continuous inference and its impact on frame rate are exemplified in Fig. 4.2 for SqueezeNet network. The average values of the performance indicators for all the network–framework pairs under study are reported in Fig. 4.3. Some aspects must be emphasized: •Thermal throttling. Fig. 4.2 illustrates the fact that when the temperature of the quad-core ARM processor exceeds a critical value of 80◦C, the clock frequency is progressively reduced, which in turn decreases the throughput. •The particular acceleration libraries exploited by the tools make them allocate different amounts of memory (Fig. 4.3(d))7and also explain the differences in CPU utilization and throughput (Fig. 4.3(a)–(b)). •Both framework coding and network layers composing the CNN architecture affect the instantaneous power consumption (averaged in Fig. 4.3(c)). Detailed power profiling of GoogLeNet–Caffe inference is depicted in Fig. 4.4. Note that some layers require higher power than others, emphasizing that the network architecture has an influence on the overall power demand. This topic will be covered in Chap. 5. GoogLeNet ResNet-50 SqueezeNet (a) 50 60 70 80 90 100 Average CPU utilization (%) GoogLeNet ResNet-50 SqueezeNet (b) 0 1 2 3 Average Throughput (fps) GoogLeNet ResNet50 SqueezeNet (c) 2.6 2.8 3.0 3.2 3.4 3.6 3.8 Power (W) GoogLeNet ResNet50 SqueezeNet (d) 0 100 200 300 400 Allocated Memory (MiB) Caffe TensorFlow OpenCV Caffe2 Figure 4.3: Average values of (a) CPU utilization, (b) frame rate, (c) power consumption, and (d) allocated memory, during a period of continuous inference with performance degradation due to temperature 7MiB refers to Mebibyte, which is equivalent to 220 bytes. 4.5. Qualitative Performance Explanation 47 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Time (seconds) 3.0 3.5 4.0 4.5 Instantaneous Power Consumption (W) Figure 4.4: Power profiling of GoogLeNet running on Caffe (black), and average value (blue). Measurement data come from an external power analyzer 4.4.2 Analysis Performance metrics greatly differ among software tools. In this embedded-system case study, Caffe shows the highest CPU utilization, what makes the system demand high power and quickly increases its temperature. However, in spite of apparently making the most of the CPU, Caffe’s throughput is the lowest. Actually, the same behavior has been observed for other CNNs running on this framework, such as those characterized in Chap. 3. The earlier hardware exploitation analysis carried out can explain the underlying reasons for these outcomes. Firstly, highest processing workload and memory traffic achieved by Caffe framework (Figs. 4.1(a) and (c)) justify its high CPU utilization and power consumption measured. However, this DL library also showed a good cache and TLB exploitation, which contrast with its poor frame rate. These outcomes on Caffe suggests a poor mapping between source code and the RPi’s ARM instruction set. Regarding TensorFlow, the achieved high throughput is explained from its lowest computational and memory workloads, which are indeed executed at high operation rates (Figs. 4.1(a)–(d)). Great cache exploitation efficiency of OpenCV renders notable frame rates on this framework – note that external memory accesses take several CPU cycles, which OpenCV saves upon its efficient use of the last level cache (Fig. 4.1(h)). Finally, the lowest power consumption of Caffe2 could result from its reduced memory access and instruction execution rates (Figs. 4.1(b) and (d)). 4.5 Qualitative Performance Explanation Numerical results presentedinSects.4.3 and 4.4areconsistent, aspreviouslydiscussed. In fact, identifying correlations among these data establishes a bottom-up approach to trace visual inference performance. 4.5.1 From Aggregated Statistics to Inference Performance To start with, results above outline how each network architecture leverages available hardware resources for performing inference. For each DL network under consideration, Fig. 4.5(a) shows that the per-image processor workload (shown in Fig. 4.1(a)) is clearly consistent with the amount of MAC operations required for inference (Sect. 4.2) – with each framework exhibiting a distinctive relationship due to its coding strategy, as previously discussed. Similarly, in Fig. 4.5(b), data memory write and read operations are correlated with the number of weights learnt by the network. Moreover, the specific acceleration 48 Relevant hardware metrics for performance evaluation libraries underlying each framework give rise to a range of memory requirements on the system. Indeed, that amount of memory resources per framework roughly follows a linear pattern according to the network architecture, as depicted in Fig. 4.6. 1.6 3.90.4 billions of MACs (a) 0.0 0.2 0.4 0.6 0.8 Instructions architecturally executed 1e10 Caffe TensorFlow OpenCV Caffe2 7.0 25.61.2 millions of weights (b) 0.0 0.2 0.4 0.6 0.8 Data memory accesess 1e10 Caffe TensorFlow OpenCV Caffe2 Figure 4.5: Each software shows a correlation between PMU hardware statistics (y-axis) and parameters defining the network architecture (x-axis), for the three CNNs 0 5 10 15 20 25 millions of weights 100 200 300 400 Memory (MiB) Caffe TensorFlow OpenCV Caffe2 Figure 4.6: Memory allocation vs. network weights Concerning visual inference applications on resource-constrained embedded platforms, optimizing hardware resources – instructions and memory traffic – is mandatory in order to boost performance. The greater the number of CPU cycles employed for memory access or execution of CPU operations, the lower the overall frame rate. This is observed for the RPi3B, where a nearly linear pattern between frame rate and data memory accesses can be identified (Fig. 4.7), making it possible to obtain good estimates of performance from this aggregated hardware metric. 4.5.2 From Instantaneous Statistics to Inference Performance This subsection departs from the aggregated statistics analyzed in Sect. 4.3 and studies hardware metrics sampled over time. Specifically, a sampling period of 10 milliseconds was set when using perf. 4.6. Graphical Comparison of Metrics 49 109 Data memory accesses 0 1 2 3 Average Throughput (fps) Figure 4.7: Alignment between throughput and registered memory-access events for the 12 accessed framework– network pairs. Pentagonal, triangular, and hexagonal points represent SqueezeNet, GoogLeNet, and ResNet-50, respectively 012345 Time (seconds) 0 1 2 3 4 5 #counts per second 1e9 Hardware Measurements L1 d-cache loads /sec L2 d-cache loads /sec Instructions /sec 2.0 2.5 3.0 3.5 4.0 4.5 Power Consumption Power (W) Figure 4.8: Instantaneous hardware statistics and power consumption for four inferences of GoogLeNet on Caffe. High correlation can be visually identified Fig. 4.8 profiles instantaneous power consumption (right y-axis) versus three hardware metrics simultaneously measured (left y-axis) – namely L1 and L2 data-cache loads per second, and instructions per second – during four consecutive inferences. The different measurement sources employed, i.e., (1) PMU event counters and (2) the external power analyzer mentioned in Sect. 4.4 make temporal alignment necessary. Although not included here, similar plots to that in Fig. 4.8 were obtained for all the analyzed networks running on each DL library. Note the high correlation visually identifiable in that plot. Therefore, the corresponding Pearson correlation coefficients of these aligned temporal signals of hardware counts and power consumption are given in Table 4.3. Note that these coefficients are in the range 0.54 – 0.95, being greater than 0.80 in most cases. Taking into account how difficult measuring power consumption is in most embedded platforms – supply pins must be accessible and special lab equipment is required – and its importance for the operation lifetime of edge vision systems, the established hardware-aware analysis constitutes a simple way to characterize the overall system – embedded platform, network, software libraries, vision application – in terms of energy. 4.6 Graphical Comparison of Metrics Outcomes from this analysis can be properly represented to assist in evaluating application requirements. 50 Relevant hardware metrics for performance evaluation Table 4.3: Pearson correlation coefficient between three hardware statistics and instantaneous power consumption L1-d-cache L2-d-cache Instructions loads/sec loads/sec /sec GoogLeNet–Caffe 0.85 0.72 0.94 GoogLeNet–TensorFlow 0.95 0.88 0.92 GoogLeNet–OpenCV 0.89 0.66 0.89 GoogLeNet–Caffe2 0.82 0.79 0.80 ResNet-50–Caffe 0.79 0.61 0.78 ResNet-50–TensorFlow 0.76 0.68 0.73 ResNet-50–OpenCV 0.86 0.54 0.85 ResNet-50–Caffe2 0.67 0.73 0.61 SqueezeNet–Caffe 0.94 0.82 0.95 SqueezeNet–TensorFlow 0.80 0.80 0.80 SqueezeNet–OpenCV 0.94 0.70 0.94 SqueezeNet–Caffe2 0.86 0.81 0.86 To simplify this task, a quick visual comparison of benchmarked scenarios is provided in Figs. 4.9a–4.9b. These plots also summarize the two-fold experimental data reported in Sects. 4.3 and 4.4 for the bottom-up approach followed in this chapter. Instructions L1 d-cache miss ratio L2 d-cache miss ratio d-TLB miss ratio Branch misspredict ratio 0.2 0.4 0.6 0.8 1.0 8.2e9 3% 19.8% 4.7% 7.5% GoogLeNet Instructions L1 d-cache miss ratio L2 d-cache miss ratio d-TLB miss ratio Branch misspredict ratio 0.2 0.4 0.6 0.8 1.0 8.2e9 3% 19.8% 4.7% 7.5% ResNet-50 Instructions L1 d-cache miss ratio L2 d-cache miss ratio d-TLB miss ratio Branch misspredict ratio 0.2 0.4 0.6 0.8 1.0 8.2e9 3% 19.8% 4.7% 7.5% SqueezeNet Caffe TensorFlow OpenCV Caffe2 (a) Hardware point of view Inference Time Power CPU utilization Memory Top-5 Error 0.2 0.4 0.6 0.8 1.0 3.2s 3.8W 95% 439MiB 9% GoogLeNet Inference Time Power CPU utilization Memory Top-5 Error 0.2 0.4 0.6 0.8 1.0 3.2s 3.8W 95% 439MiB 9% ResNet-50 Inference Time Power CPU utilization Memory Top-5 Error 0.2 0.4 0.6 0.8 1.0 3.2s 3.8W 95% 439MiB 9% SqueezeNet Caffe TensorFlow OpenCV Caffe2 (b) Sytem point of view Figure 4.9: Polar charts highlighting the main factors that critically impact the system performance From the point of view of hardware resource exploitation, Fig. 4.9a assesses five selected hardware events, for each software package. Three diagrams also help to compare among CNNs. Colored charts represent 4.7. Summary and Related Topics 51 the hardware metrics for the four studied software tools. The external circumference on the circular plots correspond to the maximum measured value among all the frameworks and networks, whereas the radial axis scales values from that maximum. Both maximum and scale factors are indicated on the plots. Thus, for example, the branch mispredict ratio of GooLeNet–TensorFlow (Fig. 4.9a–left, orange line) is approximately 0.6 times 7.5%, which is the maximum obtained for this metric. Fig. 4.9a immediately suggests the bottlenecks of each software library on this CPU-based platform: the higher the value of a metric, the worse the performance on that aspect. For instance, Caffe is characterized for the highest number of executed instructions and branch mispredictions in the three network cases. On the opposite side, TensorFlow stands out for requiring fewer instructions, OpenCV for reduced L2 cache misses, and Caffe2 for a very efficient use of branch prediction resources. Fig. 4.9b compares high-level performance metrics of the system. There is an unequivocal correspondence between reduced latency and TensorFlow, minimum memory allocation and OpenCV, and low power consumption and Caffe2. Comparing the networks on each circular diagram, note that SqueezeNet trades off classification accuracy with low power, memory resources, and latency (Fig. 4.9b–right). 4.7 Summary and Related Topics This chapter establishes guidelines to investigate hardware-resource utilization in order to explain system performance and identify critical factors. These low-level metrics effectively characterize the performance of CNN software frameworks running on the CPU of embedded systems. The identified correlations between such parameters and performance metrics highlight bottlenecks and limitations in the interaction between hardware and software. The companion graphical representations prove the validity of these considerations in practical terms. Results such as those presented in Sect. 4.5.2 suggest the use of analytical models for instantaneous power consumption estimation, whereas outcomes in Sect. 4.5.1 also hypothesize the idea of modeling the system performance from network architecture parameters. Accordingly, a comprehensive approach including analytical models to assess embedded systems will be presented in the next chapter. Actually, this approach characterizes the system performance prior to actually running inference. Concerning event counters, additional metrics can be gathered; for instance, those selected as relevant in the hierarchical bottleneck analysis proposed by Yasin [191]. Other tools can also be employed to monitor hardware events in specific embedded hardware, such as toplev tool specific to Intel’s CPUs, among others. All in all, further information about what hardware performance counters are actually measuring and how these statistics explain software performance is comprehensively described in [192]. Note. This chapter is partially based on authors’ publication [193]. Chapter 5 Prediction of visual inference performance The implementation of CNN-based applications on resource-constrained devices operating at the edge constitutes a major challenge. Once the network chosen for a specific edge application is trained, its actual implementation on a particular edge platform may not fulfill prescribed application requirements. Therefore, selecting or designing a CNN able to deliver acceptable execution time or energy consumption on the embedded system is crucial – even before training the network on the target dataset. In this chapter, a methodology for estimating CNN performance on edge devices is presented. It allows evaluating the execution latency and energy demand of CNNs prior to their actual deployment. Furthermore, this procedure also provides layer-wise performance analysis that can be exploited by NAS engines or guide the design of ad-hoc networks optimally adapted to the system. The resulting performance predictive models for both inference time and energy consumption have been tested on several well-known CNNs running on three different software–hardware systems. 5.1 Introduction When it comes to implementing embedded vision systems, practitioners must focus on a particular realization defined in terms of a complex algorithm–software–hardware stack. For such design labour, a preliminary benchmark methodology such as the one presented in Chap. 3can guide the technological choicesaccording toapplication requirements. Inaddition, software codingenhancement canfurtherboost the system performance. To this end, the recommendations provided in Chap. 4are helpful. Furthermore, oncethesoftwarelibrariesunderlying theoperationofa particular devicehave beenselectedandoptimized, edge inference performance only depends on the network architecture. In this regard, designing networks optimally adapted to the hardware to fulfill stringent application requirements is not easy. Well-known lightweight models, as those listed in Chap. 2, were designed to reduce the number of parameters and thus the computational load as a whole. Actually, as they did not specifically target a particular embedded hardware, their performance significantly varies depending on the host system [194]. In this regard, a preliminary evaluation is possible by simply comparing the number of operations performed by each network architecture for inference. However, this straightforward evaluation does 53 54 Prediction of visual inference performance not usually produce reliable values of performance metrics, in particular power consumption [195]. Additionally, manual design of computationally efficient CNNs is currently moving towards automatic algorithms such as NAS. To incorporate specific platform constraints to such approaches, a previous step is modeling how both the network architecture and the software–hardware system impact the optimization target function. The main objective when implementing CNNs on resource-constrained devices is maximizing throughput and inference accuracy while minimizing energy consumption. Concerning architectural design or network selection, there exist a vast space of possibilities that perform differently on the same embedded system. Benchmarking is non-scalable concerning this diversity of network models. To address this issue, a methodology that removes the need of comprehensive benchmarking is presented in this chapter. This methodology builds a performance prediction model based on the single characterization of a CNN specifically designed to cover most of the common layers configurations in state-of-the-art CNNs. The resulting model targets a particular software–hardware combination and provides accurate per-layer estimation of the expected performance for any network architecture running on that combination. Such a priori prediction is key for rapid exploration and optimal implementation of visual inference. In summary, this chapter describes: •A procedure to accurately estimate the performance of CNN architectures – in terms of throughput and energy consumption – layer by layer without the need of actually running them. •A neural network whose fine-grained characterization enables the construction of the aforementioned prediction model. This network incorporates a large variety of CNN layers and parameters to achieve CNN layer-wise characterization. •Fast identification of layers incurring the most inference time or demanding distinctively higher energy consumption for a network running on a particular software–hardware combination. Such layers would be the first ones to be modified by an optimization procedure or a NAS engine. In order to guide the reader through the methodology, an overview of CNN parameters and metrics is first reviewed. Then, we comprehensively describe the methodology for building performance estimation models. This procedure relies on an ad-hoc CNN specifically designed for performance characterization, whose details are reported later on. Finally, the capability of such predictive models to estimate the performance of any CNN on the targeted system is demonstrated through experimental tests. 5.2 Fundamentals of CNNs The most relevant aspects related to the proposed methodology are described in this Section. Mathematical notationin this chapterwill conform toChap.2– remember, we will use in/out sub-indexes for input/output 3D data volumes sized (H×W×C). In addition, n(X)will denote the number of elements in the tensor X. Thus, for example, n(I)for an input fmap Icontaining Cin channels sized Hin ×Win will be equal to HinWinCin. Common CNN Layers and Parameters From an architectural point of view, common CNN layers composing state-of-the-art CNNs and their related metrics are summarized in Table 5.1 and explained below. 5.2. Fundamentals of CNNs 55 Table 5.1: Operation metrics for most usual layers composing CNNs. The number of weights and operations assume no biases added Layer type Output dimensions # weights # operations Convolutional Hout =⌊Hin−kh+2p s+1⌋khkwCinCout khkwCinn(O) layer Wout =⌊Win−kw+2p s+1⌋ Cout =#filters Fully-connected 1×1×Cout n(I)Cout n(I)Cout Pooling Hout =⌊Hin−kh+2p s+1⌋—khkwn(O) Wout =⌊Win−kw+2p s+1⌋ Cout =Cin Activation layer Hin ×Win ×Cin ∝Cin ∝n(O) Multiple-input Hin ×Win ×C′∗ ∝Cin ∝n(O) layer ∗For concatenation layers, C′=P inputs Cin; whereas simple element-wise operations keep the same channel dimension as input volumes, i.e., C′=Cin •CONV layers apply a total of Cout filters sized kh×kw×Cin on sliding windows in the input volume. The dot product is performed between kernel weights and the considered receptive field, thus executing khkwCin MACs per output activation. In case of adding biases to each output, the computation increases with n(O)extra MACs. •FC layers yield a vector of activations after applying a set of weights to each input. Generally assuming n(I)inputs that produce Cout outputs, this layer involves a computational cost of n(I)Cout MACs. •Pooling operation is applied on sliding windows sized kh×kw, performing a total of khkwHoutWoutCout operations. Note that different functions can be applied – maximum, average, etc. •Activation layers perform a specific operation on each input activation, thus keeping their input dimensions unmodified. Examples are non-linear layers (ReLU, sigmoid, etc), normalization layers, scale, softmax, etc. These layers may require learning weights – as in the case of adding biases or normalizing inputs. •Multiple-input layers combine activations from various branches. This is usually implemented by directly concatenating fmaps (Concat), or applying element-wise operations (Eltwise). Overall, the approach presented in this chapter covers nine types of layers: CONV, FC, Pooling, BN, ReLU, Scale, Eltwise, Concat, and Softmax. Therefore, the most popular normalization (BN) and nonlinearities (ReLU, Scale) are studied, in addition to branch-combining layers (Eltwise, Concat) and the loss function included in most classification networks (Softmax). Intrinsic CNN Metrics Trained CNNs present the following parameters, relevant for practical deployment of embedded applications: •Computational Complexity (#OPs). The amount of floating-point operations required for network inference – or, at least, required by CONV and FC layers – is a widely-used metric to measure 62 Prediction of visual inference performance our analysis, n(W),#OPs, and #memOPs are a priori the best candidates. In fact, these selected features show correlation with the prediction variable, as demonstrated in Fig. 5.3 and also exemplified in Fig. 5.4, which plots #memOPs versus layer runtime in the baseline system for the data gathered from PreVIousNet profiling during the modeling stage. Indeed, this inherent linear relation supports the decision of applying linear regression models rather than more complex approaches such as Support Vector Machines, neural networks, or Gaussian process regression. Furthermore, derivation of per-layer regressors also makes sense, as observed in the different linear trends followed by the observation data (Fig. 5.4). In contrast, basic metrics such as first-order variables (i.e., H,W,C, etc.) would hardly predict the output in a linear model – see, as an example, Fig. 5.5, in which no linear relationships can be identified for Hin. 0 1 2 3 4 5 # memOPs 1e6 5 10 15 20 25 30 35 40 Time (ms) CONV FC Pooling ReLU BN Scale Concat Eltwise Softmax Figure 5.4: Inference runtime vs. architectural metric #memOPs (variable selected for the prediction model). Per-layer linear correlations can be identified 012345 H in 1e1 5 10 15 20 25 30 35 40 Time (ms) Figure 5.5: Inference runtime vs. the architectural metric Hin (which was finally excluded from the set of model input variables) 5.4 PreVIousNet: a Network for Fine-Grained Characterization In order to define a systematic modeling stage, a full-custom neural network, PreVIousNet, is proposed. It is specifically designed for modeling the performance of a diversity of layers with different settings when running on a selected system. Therefore, it neither needs to be trained nor is applicable for vision applications. Common architectural settings and layer parameters included in CNNs are contemplated in PreVIousNet, as described below. 5.4. PreVIousNet: a Network for Fine-Grained Characterization 63 5.4.1 Layer Parameters Concerning convolutions, common strategies implemented in lightweight CNNs are included in PreVI- ousNet: •Standard CONV. Adjustable layer parameters include: –Kernel sizes (kh,kw): conventionally, square odd-sized kernel are used. In addition, state-of- the-art CNNs employ small kernel sizes in ordertoreducethecomputationalload (#OPs ∝k2). –Number of kernels N=Cout: frequently, N>Cin kernels are applied to progressively expand the channel dimension. –Stride s: if the sliding step is greater than 1, output fmap spatial dimensions H,Ware accordingly reduced. The most common value for strided convolutions is s=2. •Depthwise CONV. This type of layer operates independently on each input channel by applying one kh×kw×1kernel filter per 2D fmap. Thus, the number of weights and computational load are reduced by a factor of Cin. •Pointwise CONV. If the kernel only operates along the channel dimension, i.e., filters are sized 1×1, then we have a non-spatial convolution that reduces the computational load by k2. Two variants are possible: –Bottleneck pointwise CONV, which aims to reduce the computational load of subsequent layers by cutting down the number of channels Cout <Cin. –Expansion pointwise CONV, which increments the channel dimension. This type of layer is applied either to revert the bottleneck channel-shrinking effect [35], or to build separable convolutions in conjunction with depthwise CONV (note that pointwise CONV does not operate along width or height axes; however, depthwise does) [37]. Concerning other layers, only Pooling layers apply sliding windows on adjustable (kh,kw,s)parameters. The usual operation performed is the maximum function, which is typically applied over 2×2patches. Additionally, the so-called global average pooling is introduced in state-of-the-art CNNs in order to replace the high memory-consuming FC layers. This type of pooling averages over each entire input 2D fmap. For the rest of layers without parameters, input data dimensions are parameters to considerate for performance modeling. 5.4.2 Architecture For each layer composing a particular network, both the input data dimensions and layer parameters determine the computational load and memory requirements, thus affecting the execution performance. Based on this fact, PreVIousNet aims at covering a wide range of settings within the architectural design space. For the main architecture of PreVIousNet – denoted as PreVIousNet-01 in Fig. 5.6(a) – this is achieved as follows: 64 Prediction of visual inference performance Figure 5.6: Macroarchitecture of PreVIousNet used in the modeling stage of the proposed method. (a) The first architecture encompasses seven types of layers with different settings (PreVIousNet-01), whereas (b) the second structure (PreVIousNet-02) covers the vectorized FC and Softmax layers. The network input dimensions at the first level – H,W, and C– are adjustable variables of the network 1. Data dimensions and thus the computational load progressively increase as the network goes deeper – i.e., moving rightwards in Fig. 5.6(a) through the stacked levels of the network. The network input dimensions at the first level – H,W, and C– are adjustable variables allowing covering different volume data sizes. 2. Diverse layers and parametersarecontemplatedinparallelbranchesinsertedat eachlevel –vertically arranged in Fig. 5.6(a). Based on these key points, PreVIousNet-01 was designed as follows. At each level, a set of parallel CONV layers covers the aforementioned strategies: standard convolutions, pointwise, depthwise, and bottlenecks. Concerning layers individually operating on activations (BN,Scale,ReLU), they are introduced at different levels and network branches in order to operate on diversely shaped intermediate fmaps. Likewise, Pooling layers with diverse settings are introduced at different levels. Finally, multiple-input layers (Eltwise and Concat) operate on equally sized pairs of tensors coming from previous branches of the network. Note that FC and Softmax deal with 1D data vectors instead of 3D tensors; in addition, FCs are memoryintensive. Therefore, these layers are not included in PreVIousNet-01 but in a different architecture, PreVIousNet-02 depicted in Fig. 5.6(b). Following the same design guidelines, various sets of parallel FC layers deal with a diversity of input and output vector sizes. The network input is an adjustable 1×1×C vector. Then, vectors of various sizes are processed by the layers: either resulting from applying an expansion factor to the input (2C,4C, etc.), or using customized vector lengths Ki. For instance, K1=10 and K2=1000 are common output sizes in classification networks trained on ImageNet , CIFAR, and MNIST. Similarly, Softmax layers operate on differently sized vectors. All in all, PreVIousNet comprises two compact specialized networks that include 52 layers – 15 CONV, 7BN, 7 Scale, 7 ReLU, 6 Pooling, 5 Eltwise, and 5 Concat – plus 44 1D-data layers – 32 FC and 12 Softmax. Further details about the architecture are included in its model definition, publicly available in Caffe format [201]. For instance, although no weight file is provided, loading the network in Caffe will automatically initialize the weights according to the “MSRA” initialization scheme [202]. Interestingly, 5.5. Experimental Tests 65 the proposed architecture can be adjusted according to the features of the networks to be characterized in the performance prediction stage. Finally, note that PreVIousNet constitute a singular approach to characterize CNN layers, but alternative networks or systematic benchmarking methods would also be valid. For instance, the characterization of recurrent building blocks of highly optimized architectures as a whole, e.g., the Fire module of SqueezeNet or separable convolutions of MobileNets, could constitute an alternative. 5.4.3 Network Configuration for the Modeling Stage As previously mentioned, the adjustable input size of PreVIousNet-01 (H×W×C) can be set according to the most common tensor sizes handled by CNNs. For instance, consider SqueezeNet [36], which is a popular lightweight CNN. Input volumes in the layers composing this network range from 3to 512 channels, whereas both their height and width decrease following the sequence 227 −113 −56 −28 −14. Accordingly, a characterization of PreVIousNet with varied input tensor sizes will allow collecting enough data to build accurate prediction models. In the modeling stage, we empirically set the following four input dimensions for PreVIousNet-01: (i) 56 ×56 ×32, (ii) 28 ×28 ×64, (iii) 14 ×14 ×64, and (iv) 7×7×64. Thus, we are particularly sampling11 CONV layers with Hin =Win ={56,28,14,7}and Cin ={32,64,128,256}. The regression models will accurately interpolate or extrapolate performance values from these ranges, as will be shown in Sect. 5.5. Concerning PreVIousNet-02, the particular realization characterized uses an input vector sized 1×1×256. Consequently, thecommon vector lengthscovered areCin ={256,512,1024,2048,4096}, pluscustomized sizes {K1,K2}={10,1000}. Note that for specific architectures of interest, different input configurations could be used. 5.5 Experimental Tests 5.5.1 Embedded System To test the performance of the proposed prediction methodology, the baseline system consisted of the low-cost RPi3B platform running CNNs on Caffe. Once the effectiveness of PreVIous on this baseline case was verified, experimental tests were extended by changing first the software – OpenCV 4.0.1 on RPi3B – and then the hardware – Caffe on Odroid-XU4. This second hardware device is more suitable for high-performance IoT applications, presenting a multi-core architecture arranged into two clusters that can be tuned for higher performance or power efficiency. Further details about these embedded devices are included in Table 5.2. All the experiments were performed on the quad-core CPU included in the RPi3B or on the ‘big’ cluster at 2 GHz in Odroid-XU4. The operating systems were booted in console mode to boost inference performance and reduce power consumption. 11Note that in PreVIousNet-01, the channel dimension increases over network levels, whereas the input fmap resolution remains constant. The purpose is to design a simplified network to be characterized under different configurations. 66 Prediction of visual inference performance Table 5.2: Features of the embedded platforms included in the studied IoT vision systems Raspberry Pi 3B Odroid-XU4 SoC Broadcom BCM2837 Octa-core Exynos 5422 big.LITTLE Processor QuadCore ARMCortex-A53@ 1.2GHz Quad CoreARM Cortex-A15 @2 GHz+ Quad Core ARM Cortex-A7 @ 1.4 GHz Accelerator VideoCore IV GPU Mali-T628 MP6 GPU Memory 1 GB RAM LPDDR2 @ 900 MHz 2 GB LPDDR3 RAM @ 933 MHz OS Raspbian v9.4 Linux Kernel v4.14 Ubuntu v16.04 Linux Kernel v4.14 GPIO 40-pin GPIO 40-pin GPIO Dimensions 85 ×56 ×20 mm383 ×58 ×20 mm3 Power MicroUSB socket 5V/2.5A Barrel jack 5V/4A Cost ∼30= C∼43= C 5.5.2 Networks We applied the PreVIous methodology to predict the runtime and energy consumption of CNNs when running on the aforementioned systems. ThemodelingstageincludedtheperformanceprofilingofPreVIousNetunder the 5 configurations specified in Sect. 5.4.3 — four for PreVIousNet-01 and one for PreVIousNet-02. Then, vector yin Eq. (5.2) was built for both runtime and energy consumption. Likewise, observation matrix Xin Eq. (5.2) was obtained for each type of layer according to their corresponding architectural parameters. The prediction stage was conducted on seven popular CNNs, most of them designed for embedded inference: AlexNet [1], All-CNN-C [200], MobileNet [37], ResNet-18 [35], SimpleNet [160], SqueezeNet [36], and Tiny YOLO [203] – see Table 5.312. These networks cover classification on ImageNet and CIFAR-10, or object detection on COCO. All in all, 399 CNN layers were assessed in this extensive study. Table 5.3: Popular CNN architectures employed for testing the performance prediction capability of the PreVIous. Built regression models are applied on networks’ constituent layers, here summarized Network Dataset Layers AlexNet ImageNet 5 CONV, 3 FC, 3 Pooling, 7 ReLU, 1 softmax, 2 LRN All-CNN-C CIFAR-10 9 CONV, 1 Pooling, 9 ReLU, 1 softmax MobileNet ImageNet 27 BN, 28 CONV, 1 Pooling, 27 ReLU, 27 scale, 1 softmax ResNet-18 ImageNet 21 BN, 21 CONV, 8 eltwise, 1 FC, 2 Pooling, 17 ReLU, 21 scale, 1 softmax SimpleNet CIFAR-10 10 BN, 13 CONV, 1 FC, 5 Pooling, 13 ReLU, 10 scale, 1 softmax SqueezeNet ImageNet 8 concat, 26 CONV, 4 Pooling, 26 ReLU, 1 softmax Tiny YOLO COCO 8 BN, 9 CONV, 6 Pooling, 8 ReLU, 8 scale 5.5.3 Experimental Results: Layer-wise Predictions The precision of the per-layer prediction models resulting from PreVIous can be assessed by comparing layer-wise predictions with actual experimental profiling for the considered networks under test. For instance, fine-grained layer-level performance characterization of a particular CNN is shown in Fig. 5.7. It compares the actual profiling of All-CNN-C [200] with model estimations, for the RPi–Caffe implementation. The prediction model correctly identifies ‘conv2’ and ‘conv5’ as the layers consuming most of the inference time and energy. This is an example of how useful this methodology can be for optimization algorithms or NAS methods. 12AlexNet also includes two Local Response Normalization (LRN) layers that have not been included in this layer-wise prediction because they were deprecated in favor of other normalization layers such as BN. 5.5. Experimental Tests 67 conv1 relu1 conv2 relu2 conv3 relu3 conv4 relu4 conv5 relu5 conv6 relu6 conv7 relu7 conv8 relu8 conv9 relu9 pool prob 0 5 10 15 20 25 30 35 Time (ms) Measured Predicted (a) Runtime conv1 relu1 conv2 relu2 conv3 relu3 conv4 relu4 conv5 relu5 conv6 relu6 conv7 relu7 conv8 relu8 conv9 relu9 pool prob 0 20 40 60 80 100 120 140 Energy (mJ) Measured Predicted (b) Energy Figure 5.7: Layer-wise measured profiling vs. predictions for All-CNN-C running on the baseline system, i.e., RPi–Caffe PreVIous is designed for prediction of inference time and energy consumption; their combination also allows estimating the average power consumption as follows: ˆ Pl=ˆ El ˆ tl (5.7) where ˆ El,ˆ tl, and ˆ Plare the energy, runtime and power predictions for layer l, respectively. Fig. 5.8 exemplifies the per-layer power consumption predictions ˆ Plas a function of the predicted runtime ˆ tlfor the complete model All-CNN-C. Note that this average power estimation is similar to the actual measured profiling outlined in Fig. 5.2. 0 20 40 60 80 100 120 Time (ms) 2.0 2.5 3.0 3.5 4.0 Predicted Power (W) Figure 5.8: Layer-wise power consumptionpredictions, expressedas ˆ Pl=ˆ El ˆ tl, during the All-CNN-Cnetwork inference. The temporal duration of each layer (x-axis) is also a prediction (Fig. 5.7(a)). Note that the power, represented in the y-axis, keeps constant for each layer owing to the average estimation derived from the aforementioned equation. The predicted energy demand (Fig. 5.7(b)) corresponds to the colored area Applying the methodology to all the considered networks, we obtain per-layer predictions, as shown in Fig. 5.9 for the inference time in the baseline test system. Note that the runtime predictions are accurate when compared to empirical measurements. 68 Prediction of visual inference performance Fig. 5.9 highlights the non-negligible contribution of certain layers, even dominant in some cases, e.g., Pooling in SqueezeNet or BN in MobileNet and ResNet, to the global inference runtime. This proves the importance of their consideration for performance modeling, as opposed to approaches exclusively focused on CONV and FC layers. 0 50 100 150 200 Measured (ms) 0 25 50 75 100 125 150 175 200 Predicted (ms) AlexNet 0 10 20 30 Measured (ms) 0 5 10 15 20 25 30 35 Predicted (ms) All-CNN-C 0 10 20 30 40 Measured (ms) 0 10 20 30 40 Predicted (ms) MobileNet 0 20 40 60 Measured (ms) 0 10 20 30 40 50 60 Predicted (ms) ResNet-18 0 10 20 30 40 50 Measured (ms) 0 10 20 30 40 50 Predicted (ms) SimpleNet 0 10 20 30 40 Measured (ms) 0 5 10 15 20 25 30 35 40 Predicted (ms) SqueezeNet 0 100 200 300 400 Measured (ms) 0 50 100 150 200 250 300 350 400 Predicted (ms) Tiny YOLO CONV FC Pooling ReLU BN Concat Eltwise Scale Softmax Figure 5.9: Per-layer inference runtime predictions (y-axis) compared with actual measurements (x-axis) for the 7 networks running on the baseline combination RPi–Caffe. Similar results are obtained for RPi–OpenCV and XU4– Caffe. The dashed line depicts an ideal estimation in which predictions exactly match actual measurements When adding up layer-level predictions for all layers composing each particular network, the total runtime can be estimated, as shown in Table 5.4 for RPi–Caffe. Each row reports the total measured (t) and predicted (ˆ t) runtime, as well as the prediction error, according to the following equations: t= Nl X l=1 tlˆ t= Nl X l=1 ˆ tlerror =ˆ t−t t(5.8) where tldenotes per-layer measurements, ˆ tldenotes per-layer predictions, and Nlis the number of layers in the particular CNN. The last row in the table shows the mean absolute percentage error (MAPE) over the seven considered networks: MAPE =100% n n X i=1 ˆyi−yi y (5.9) where y,ˆyare the actual and predicted values, and nis the number of test samples considered – seven in this case. Layer-wise measurements, predictions, and errors can be obtained for energy consumption: E= Nl X l=1 Elˆ E= Nl X l=1 ˆ Elerror =ˆ E−E E(5.10) where Eland ˆ Elrepresent per-layer energy measurements and predictions, respectively. 5.5. Experimental Tests 69 All in all, Table 5.5 shows the accurate prediction results achieved for all tested systems, CNNs, and measured metrics. For each vision system, this table presents the MAPE, given by Eq. (5.9), over the seven considered CNNs resulting from applying Eqs. (5.8) and (5.10). Table 5.4: Predictions of total runtime on the RPi–Caffe system based on per-layer values. The corresponding profiling data are depicted in Fig. 5.9 Measured Predicted Error t(ms) ˆ t(ms) (%) AlexNet 561.64 526.75 -6.21% All-CNN-C 115.68 123.22 6.52% MobileNet 943.73 908.74 -3.71% ResNet-18 1032.84 1049.10 1.57% SimpleNet 347.59 349.22 0.47% SqueezeNet 348.15 343.54 -1.32% Tiny YOLO 1691.37 1740.03 2.88% mean absolute error (MAPE) 3.24% Table 5.5: MAPE for predictions based on per-layer values on the seven tested CNNs Runtime Energy RPi3B – Caffe 3.24% 5.30% RPi3B – OpenCV 4.08% 3.63% XU4 – Caffe 3.82% 5.01% 5.5.4 Experimental Results: Network Predictions Equations (5.8) and (5.10) in Sect. 5.5.3 represent a widely-used procedure to estimate the global forwardpass performance of a CNN: adding up per-layer metrics. Indeed, this is an assumption underlying several works on network optimization or NAS [204, 155, 197, 205]. Per-layer aggregation should be valid to estimate the global metrics given that layers are sequentially executed during network inference in many realizations. However, in practice, this direct addition of per-layer measurements independently taken may not reflect actual network performance metrics [206, 207, 208]. This mismatch arises from computational optimizations that can be applied during forward-pass, such as those related to software algorithms (e.g., layer fusion or constant folding) or processor strategies (e.g., data prefetching or data re-utilization within the memory hierarchy). For further accuracy in prediction taking into account this identified mismatch, additional experimental data were collected during complete network inference without layer separation. From these forward-pass characterizations and the previous per-layer profiling, we can find a simple expression to predict network inference, thus replacing Eq. (5.8) or (5.10) for: ˆy=c Nl X l=1 ˆyl(5.11) where ˆystands for the total predicted runtime or energy, ˆylrepresents the per-layer predictions either for runtime or energy, and cis a mismatch-fitting coefficient. This factor cresults from a linear regression between the aggregated per-layer predictions (Pˆyl) and the corresponding actual forward-pass measurement (y), adjusted from the 5 configurations of PreVIousNet on each visual system. The resulting coefficients cfor each system case are shown in Table 5.6. 70 Prediction of visual inference performance Table 5.6: Empirical coefficient cin Eq. (5.11) for the three studied systems Runtime Energy RPi3B – Caffe 0.88 1.08 RPi3B – OpenCV 0.85 0.89 XU4 – Caffe 0.93 1.09 The proposed performance prediction framework allows evaluating complete network inference by using Eq. (5.11). Both performance inference prediction and experimental forward-pass measurements are compared in Figs. 5.10(a) and 5.10(b) for runtime and energy, respectively13. Note that complete forward-pass estimations are also accurate, with deviations below 10% in 18 out of 21 studied cases for runtime and 15 out of 21 cases for energy. Table 5.7 summarizes the MAPE on forward-pass performance prediction. Table 5.7: MAPE for forward-pass predictions on the seven testing CNNs Runtime Energy RPi3B – Caffe 5.02% 8.52% RPi3B – OpenCV 7.92% 7.24% XU4 – Caffe 3.25% 10.46% 5.5.5 Analysis Let us next analyze the reasons for the less accurate predictions in some networks – for instance, AlexNet, specially when modeling energy on Odroid XU4; in fact, runtime and energy consumption were underestimated in all cases involving AlexNet, note the negative errors shown in Fig. 5.10. This behaviour can be explained in terms of the mismatch between per-layer and complete-CNN inference performance, characterized with the cterm through the simple Eq. (5.11). This single coefficient cper software–hardware system may not accurately describe empirical behavior in all cases. For example, delving into the worst prediction case (energy estimation of AlexNet14 on XU4–Caffe), the actual ratio between per-layer and complete-CNN measurements is 1.60, but it is characterized applying a factor of 1.09 (Table 5.6), notably lower. This same issue is identified for MobileNet’s energy consumption in RPi–Caffe and XU4–Caffe. Note that, as a whole, Fig. 5.10 reports absolute errors below 10% for most of the 42 studied cases (although most network prediction errors exceed 5%). By contrast, per-layer predictions of PreVIous summarized in Table 5.5 were certainly more accurate, with absolute network errors below 5% in the vast majority of the 42 cases, and below 10% in the 95% of the cases. Finally, it is worth noting that performance metrics can slightly fluctuate between network executions according to the state of the system at the moment of running the CNN. Indeed, this is why measurement data were averaged over N=50 executions. Therefore, conducting the modeling stage again will produce slightly different predictions and errors. Interestingly, the prediction errors from PreVIous are in the order of the experimental fluctuations when repeating executions. This suggests that much better accuracy in prediction may not be possible. 13For a fair comparison on AlexNet, actual profiling measurements of the deprecated LRN layers were added in the summation in Eq. (5.11). 14AlexNet is a starting point in the CNN paradigm. Therefore, its architecture includes deprecated layers and a high number of weights, somehow differing from state-of-the-art architectures. 5.6. Summary and Related Topics 71 0 250 500 750 1000 1250 1500 1750 Inference Time (ms) -15.59% 2.43% -2.68% -0.27% 3.21% 3.04% -7.90% -18.09% -6.43% 9.52% -3.08% -6.21% 10.64% -1.49% -4.29% 4.38% 0.80% 1.58% 6.33% 0.47% -4.92% Tiny YOLO SqueezeNet SimpleNet ResNet-18 MobileNet All-CNN-C AlexNet RPi-Caffe (measured) RPi-Caffe (predicted) RPi-OpenCV (measured) RPi-OpenCV (predicted) XU4-Caffe (measured) XU4-Caffe (predicted) (a) Runtime 0 2000 4000 6000 8000 10000 12000 Energy (mJ) -13.75% 7.74% -14.21% -5.25% 1.85% -7.24% -9.56% -10.03% -2.63% 8.67% -5.54% -3.72% 15.40% -4.70% -27.05% 9.90% -17.35% 1.88% 6.96% -8.91% -1.15% Tiny YOLO SqueezeNet SimpleNet ResNet-18 MobileNet All-CNN-C AlexNet RPi-Caffe (measured) RPi-Caffe (predicted) RPi-OpenCV (measured) RPi-OpenCV (predicted) XU4-Caffe (measured) XU4-Caffe (predicted) (b) Energy Figure 5.10: Network predictions vs. actual measurements for all the assessed CNNs and software–hardware systems 5.6 Summary and Related Topics The methodology developed in this chapter offers accurate predictions on the performance of CNNs on CPU-based embedded vision devices. It is a simple procedure that (i) requires a single systematic network characterization, (ii) relies on linear regression models, and (iii) is agnostic with respect to the selected hardware–software system. Moreover, a greater diversity of network layers has been considered with respect to previous studies, which mostly focused on CONV layers. The application of this methodology to IoT vision systems is two-fold. First, fine-grained layer-level performance estimation assists network architecture design and optimization in order to create ad-hoc networks optimally adapted to the system. Second, network performance prediction facilitates the task of selecting CNNs according to prescribed 78 Exploiting heterogeneous edge platforms for high-throughput or low-energy visual inference CNN graph (img #0) CNN graph (img #1) CNN graph (img #2) time (a) 72% 22% Subgraph1 (img #0) Subgraph0 (img #0) Subgraph0 (img #1) Subgraph1 (img #1) Subgraph0 (img #2) Subgraph1 (img #2) time (b) Figure 6.2: CNN inference on input stream based on (a) single-device execution, or (b) pipeline scheme This unified scheme of co-execution can simultaneously exploit all processors on a board. Throughput acceleration depends on the slowest sub-graph T=1/max i{tMi}(imgs/s). Next, we define a generic graphlevel partitioning problem to optimize inference performance. Additionally, kernel-level optimization is automatically achieved using TVM auto-tuning, which renders highly efficient ML graphs [238]. 6.2.1 Problem formulation Deep neural networks are computational dataflows commonly expressed as directed acyclic graphs (DAGs). In DAGs, nodes represent operators/layers, whereas edges represent their dependencies. For instance, nodes in CNNs embody convolutional, fully-connected, or softmax layers, as well as their intrinsic parameters – kernel size, feature-map size, etc. Given a network model Mand a set of available hardware devices hi∈ H, the objective is to split M into sub-graphs Mi∈ M running on heterogeneous devices to optimize particular performance metrics. After network mapping, each sub-graph Miwill contain a set of nodes from the original graph Mwithout modifying it, i.e., Mis unambiguously expressed as the concatenation of all Mi. In this work we will assume that networks are sequential – or that they contain a limited number of branches with parallelism. This is indeed the case of many widely-used classification networks such as AlexNet [1], ResNets [239], MobileNets [37, 240], and VGG [34]. Many networks containing branches can be expressed as a chain; otherwise, the methodology proposed in this chapter can still be applied, although the scheduling algorithm needs to be replaced by a more complex algorithm – scheduling algorithms are well researched [241, 242, 243]. In this chapter, we will show the advantages of the heterogeneous scheduling of neural network chains (models detailed in Sect. 6.3.2). As an example, the two-stage network partition shown in Fig. 6.2b schedules a model Mwith Nsequential nodes over two devices {h0,h1}. Layers with indexes {0,1,2, ..., n−1}are assigned to graph0 and mapped 6.2. Proposed approach 79 into h0, whereas graph1 includes operators {n,n+1, ..., N−1}running on h1. Inference throughput optimization in this case involves finding the point n∈[0,N−1] that minimizes the maximum runtime among all stages: ˆn=min n[max{th0 M0(n),th1 M1(n)}](6.1) In this equation, runtimes are collected in advance through network profiling – see data collection and related issues in Sect. 6.3.3. Fig. 6.3 illustrates CNN inference on three consecutive frames in two scenarios: (i) single-stage graph execution, and (ii) two-stage network partition, exemplified with n=5,N=8. Intermediate data at layer n−1must be transferred from the first processing device to the second one. These two scenarios of network workload distribution will be thoroughly studied in Sects. 6.3–6.4. (a) (b) Figure 6.3: Network workload distribution for single- and two-stage approaches. CNN classification over three frames is shown for (a) single-stage execution on a particular hardware, or (b) two-stage concurrent execution on two hardware devices 6.2.1.1 Scheduling space In a general case with multiple stages (s), the network partition is defined by a vector nof s−1splitting points producing the sub-graphs Mi. In total, an s-stage workload distribution with N−1possible partition points renders up to N−1 s−1possible distributions16. Once defined the s-stage distribution, and given nhavailable hardware devices, we can assign the s processing workloads to these processors in (nh)spossible sequential processing orders. For instance, h2→h0→h1,h0→h2→h1, or h1→h0→h1are examples of device processing orders for a three-stage pipeline on three available devices. All in all, the total number of possible configurations to distribute the workload over sstages is: N−1 s−1!(nh)s(6.2) Starting from s=1, the greater the number of stages s, the greater the total number of possible configurations. To avoid combinatorial explosion in the number of configurations to be analyzed, in the next section we will show how to guide the search taking the memory transfer cost into account. 16This binomial coefficient is obtained assuming Nsequential nodes {l0,l1, ..., lN−1}in the DAG, and s−1partition points which can be located at any of the N−1intermediate positions. 80 Exploiting heterogeneous edge platforms for high-throughput or low-energy visual inference 6.2.1.2 Scheduling search problem In general, the problem is to find the set of partition points ˆ nthat optimize at least one of the following target metrics: Throughput: ˆ n=min n[max i{thi Mi(n)}](6.3) Energy: ˆ n=min nX i Ehi Mi(n)(6.4) Latency: ˆ n=min nX i thi Mi(n)(6.5) Note that in practice there will be a transfer cost due to copying data between the devices – symbolically represented in Fig. 6.3b with diagonal dashed arrows. So that, in the equations above, we are assuming that this transfer cost is already included in the stage cost: thi Miis the runtime of stage Mirunning on device hiwhich also includes the transfer cost to copy data to hi. This extra cost thi trans f er depends on the transferred data, which will be different for each partition point and network architecture. To incorporate data-transfer cost in the problem, we can experimentally measure or estimate it – see Sect. 6.3.3. For such an optimization problem, an intuitive general approach consists of starting with s=1and finding the optimal solution. Then, we set s=2, consider all possibilities, and we find the optimal solution again. Next, we successively minimize the function with increasing number of stages. We can introduce a heuristic stopping rule for the search. Given that the transfer cost is usually significant and given that the number of devices nhis usually limited, eventually no better solution can be found for s=ˆs+1than for s=ˆs. At this point, it makes no sense to look for other solutions with more stages. In case of our heterogeneous GPU-CPU device, this will likely be the case for 2 stages (as exemplified in Sect. 6.4.3). 6.2.1.3 Scheduling on TX2 using TVM Image-classification neural networks are composed of multiple layers that sequentially process data to produce an output vector from an input image. In most architectures, the first layers operate on data with higher dimensionality, whereas smaller feature maps are processed at ending layers. As a result, the computational demand varies among layers. This is why the performance profile strongly depends on the network architecture. As an example, Fig. 6.4 depicts the per-layer processing time for MobileNet [37] entirely running on one of the two processing components available in Jetson TX2, i.e., H={GPU,CPU} (nh=2). Note that layer performance also depends strongly on the selected hardware back-end. Indeed, we will demonstrate that layer-wise operation distribution approach can benefit from high-performance computation on the GPU while offloading less computationally demanding layers to the CPU. For instance, the simplest pipeline case is a two-stage network execution in which case the neural network is pipelined on CPU and GPU with only one memory transfer. In this case, once we know the layer-wise execution runtime profile {thi 0,thi 1, ..., thi N−1}of the CNN on both processors h0=GPU,h1=CPU (Fig. 6.4), we can estimate sub-graph latencies: 6.2. Proposed approach 81 0.00 500.00 1,000.00 1,500.00 runtime (us) (a) 0 1,000 2,000 3,000 4,000 5,000 runtime (us) (b) Figure 6.4: Layer-wise runtimes for MobileNet running entirely either on (a) the GPU, or (b) CPU of Jetson TX2 th0 M0(n)= n−1 X l=0 th0 l+th0 trans f er (6.6) th1 M1(n)= N−1 X l=n th1 l+th1 trans f er (6.7) We maximize throughput by solving Eq. (6.1). Then, the expected throughput (Tin img/s) and latency (L in seconds), respectively, of the optimized 2-stage configuration running on h0→h1are: T=1 max{th0 M0(ˆn),th1 M1(ˆn)}(6.8) L=th0 M0(ˆn)+th1 M1(ˆn)(6.9) This throughput optimization problem for s=2can be solved with Algorithm 1. This algorithm assumes that we have collected runtime profiling and we have estimated transfer costs. It also returns the optimal device order ˆ h0→ˆ h1, e.g., GPU →CPU. 82 Exploiting heterogeneous edge platforms for high-throughput or low-energy visual inference Algorithm 1 Two-stage network partitioning Input: {tl}h0,{tl}h1,{ttrans f er,l}h0,{ttrans f er,l}h1 Output: ˆn,ˆ t,ˆ h0,ˆ h1 ˆ t← ∞ for (h′ 0,h′ 1)in all combinations({h0,h1})do tM0←0 tM1←P i th′ 1 i for i←0to N−1do tM0←tM0+th′ 0 i/* Partition 0: [0,i] */ tM1←tM1−th′ 1 i/* Partition 1: [i+1,N-1] */ m[i]←max{tM0+th′ 0 trans f er,i,tM1+th′ 1 trans f er,i} end for if min im[i]<ˆ tthen ˆ t←min im[i]/* new optimal found for h′ 0→h′ 1*/ ˆn←arg min im[i] ˆ h0←h′ 0 ˆ h1←h′ 1 end if end for return ˆn,ˆ t,ˆ h0,ˆ h1 6.3 Experimental Procedure 6.3.1 Hardware and Software The selected system, Nvidia Jetson TX2, was designed to jointly achieve power efficiency and high performance for different embedded applications. It is a popular commercial board that integrates a 256-core Nvidia Pascal GPU and a 64-bit ARMv8 general-purpose multi-processor. The memory and storage included in the platform are 32-bit 8GB LPDDR4 and 32GB eMMC 5.1, respectively. A camera expansion connector is included to endow the system with on-board video analysis capabilities. The Jetson TX2 platform was selected due to the (compared to a desktop CPU and GPU) relatively low power consumption (7.5W – 15W), while still providing sufficient throughput for many deep learning applications. Using TVM on this platform, we can (1) define, (2) tune, (3) compile, and (4) run ML models: 1) ML models are internally represented in TVM as computational graphs in a so-called Tensor Expression language. Users can define models in an intermediate representation on TVM, or alternatively networks can be loaded from files compatible with other ML frameworks, such as TensorFlow, Keras, MXNet, PyTorch, or DarkNet. 2) TVM integrates auto-tuner tools to generate efficient code for the targeted platform [66, 238]. Specifically, the results presented in this chapter regarding auto-tuned models were compiled with Ansor [238, 244]. This tool finds the best compiling options for each back-end to reduce the corresponding runtime – see Sect. 6.4.1. 6.3. Experimental Procedure 83 3) TVM compiles graphs to generate the C++ code that will run on the target hardware. We used the TVM Python interface, which greatly facilitates network definition and execution (other available interfaces are TVMC [245] and C++). 6.3.2 Neural Networks Table 6.1 summarizes the architecture of the CNNs considered in this chapter. We used the TVM definitions of these networks [246]. TVM can process them using two types of data orders called NHWC and NCHW. Experiments in this work refer to channel-first (NCHW) layout. Layers in the network are represented by nodes in the graph, sequentially connected to compose the overall architecture. Although the networks may contain modules with parallel branches – as in the Fire module of SqueezeNet and Inception of Inception-v3 –, they are treated as individual nodes in the proposed procedure. Therefore, ‘# modules’ in Table 6.1 refers to the number of sequential nodes Nin the graph. Note that even if these models contain layers other than those listed in Table 6.1, such as ReLU, batch normalization, or dropout, TVM automatically optimizes the computational graph by fusing these operations with the previous layer. Table 6.1: Assessed neural networks Network # TVM layers # modules MobileNet [37] 31 27 conv + 1 pooling + 1 flatten + 1 FC + 1 softmax SqueezeNet-v1.1 [36] 40 2 conv + 4 pooling + 8 Fire + 1 flatten + 1 softmax Inception-v3 [24] 122 5 conv + 11 Inception + 3 pooling + 1 flatten + 1 FC + 1 softmax VGG-19 [34] 26 16 conv + 5 pooling + 1 flatten + 3 FC + 1 softmax 6.3.3 Layer-wise Performance Profiling In short, the proposed method comprises a first stage of layer-level network performance profiling and a second stage of operator mapping to the available hardware components. Concerning the first stage, for each hardware component hi, we must obtain a performance profile of the Nnodes in the model. Then, in the second stage, we solve the network partition problem for any of Eqs. (6.3)–(6.5). We assume that the neural-network layer execution performance is independent of the input data, that is, we do not consider dynamic neural networks and influences of L1/L2 caching are ignored. •Runtime. Throughput optimization in Jetson TX2 means solving Eq. (6.1) (or Eq. (6.3), more generally) with the profiling of the network expressed as {tl}GPU ,{tl}CPU ∀l∈[0,N−1]. An example was depicted in Fig. 6.4 for MobileNet; this figure clearly exemplifies how a pipelined approach distributing operations over the GPU and CPU in TX2 benefits from their dissimilar performance. The operation of Algorithm 1solving Eq. (6.1) for a cooperative GPU →CPU execution is illustrated in Fig. 6.5, which shows the runtime tMiof the stage as a function of the selected partition point n. Up to N+1solutions are possible, being the optimal one ˆn=23, which means that 23 nodes run on the GPU and the rest on the CPU. –Aspreviously stated, data-transfer latencycan beincorporated intothe scheduling model. Even though the GPU memory is integrated in the CPU memory in TX2 through CUDA, the TVM implementation (and many other implementations) uses distinct memory regions for CPU and GPU, still requiring a transfer between both. Several techniques exist to mitigate or reduce the 84 Exploiting heterogeneous edge platforms for high-throughput or low-energy visual inference 0 5 10 15 20 25 30 35 40 45 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 sub-graph time (ms) network partition point, n t_M0 t_M1 max{t_M0, t_M1} Figure 6.5: Throughput optimization for a two-stage pipeline on MobileNet running in Jetson TX2 with order GPU → CPU. Sub-graph execution runtimes tMi(n)depend on the selected partition point n data transfer latency, but none of them totally eliminates the cost. For example, the transfer latency can be improved by using CUDA pinned host memory, but due to the lack of hardware support for cache coherence between the CPU and GPU caches, a costly synchronization step (in principle, a call to the cudaDeviceSynchronize or cudaStreamSynchronize function) is needed, leading again to a performance cost. Another solution is to use CUDA unified memory; in this case, the transfer is handled transparently by the GPU driver, but there is still a cost (which manifests as a slower execution of the kernel function of the particular neural network layer(s), as well as the need for calling the cudaDeviceSynchronize function). In addition, by using multiple streams, CUDA allows overlapping of memory transfers with the actual computations, reducing the transfer costs. At the time of writing, the TVM support of this feature is in an initial stage (see e.g., [247]). However, it is generally known that this technique does not reduce the transfer cost significantly in most practical cases with a high GPU kernel occupancy. To experimentally characterize this transfer overhead, we could assume linear dependence w.r.t. the size of the transferred data ttrans f er =k·B(where kis an arbitrary constant that depends on the considered hardware, and Bis the number of bytes). We thoroughly measured ttrans f er in TX2, including several batch sizes and CNN models in experimental data. In particular, we measured the time cost of the set_input function of the GraphModule TVM python class, which transfers data to the CNN and incurs high costs. The results confirmed this linear dependence – see Fig. 6.6. Indeed, the methodology assessment presented in Sects. 6.4–6.5 further validated this assumption. 1.3 GB/s Figure 6.6: Data transfer time overhead on TVM-Python 6.4. Performance Evaluation 85 •Energy. The main power supply to the TX2 module is provided by a single DC input. According to its documentation [248], the voltage of this DC input ranges from 5.5 V to 19.6 V, with a maximum current of 4000 mA. Two INA3221 power monitors are included in the TX2 carrier board. We measured the per-layer power demand of CNNs, {Pl}hi, using the TVM debugger and software included in Tegra Linux Driver Package [249]. –When it comes to taking accurate power consumption measurements, the limited time resolution of the INA3221 sensors plays a major role. Therefore, mismatches between per-layer and overall CNN power consumption are common [250, 251, 252]. For instance, Cai et al. [251] showed that per-layer energy profiling is lower than actual CNN energy consumption in all of their study cases. An interesting aspect highlighted in [252] is that monitoring systemlevel power consumption (instead of taking separate GPU and CPU measurements) renders better estimations because, among other aspects, the energy consumed by memory accesses is also taken into account. This is why we measured power at system level. Interestingly, the aforementioned approaches reported in the literature do not try to correct measurement errors because estimating the energy-consumption trend is enough, instead of finding exact energy values. As we will discuss in Sect. 6.5, the proposed scheduling model is indeed able to find the optimal cases. 6.4 Performance Evaluation 6.4.1 Single-device execution The baseline performance of network execution on each hardware component of the platform (i.e., on each TVM target) is shown in Fig. 6.1. Notwithstanding, as previously pointed out, one of the advantages of TVM is its open-source auto-tuning tool, which generates highly efficient code and boosts inference performance. In this regard, we measured a performance improvement (up to 87%) in terms of runtime between (i) default TVM network execution (Fig. 6.1), and (ii)17 tuned network execution (using Ansor [238]), as reported in Fig. 6.7. Furthermore, for a baseline performance reference on our platform, Fig. 6.7 also presents a inference runtime comparison that includes other software frameworks. The performance of each library depends on the architecture of the DL model [253, 254]. TVM definitively constitutes as a promising general framework. 6.4.2 Two-stage execution Once the layer-wise performance measurements were taken (as explained in Sect. 6.3.3), performance optimization was conducted according to Eqs. (6.3)–(6.5). To define network sub-graphs, we edited the pre-defined networks [246] in TVM. The proposed co-execution pipeline scheme was enabled by developing TVM-Python code to compile network sub-graphs and continuously offload the corresponding processing stage to each back-end. This code of ours is available in [255]. 17We will use “(i)” to denote networks compiled with TVM by default and “(ii)” for models compiled with compiling optimizations produced by Ansor auto-tuning tool. 86 Exploiting heterogeneous edge platforms for high-throughput or low-energy visual inference 116.9 102.6 26.1 29.8 68.8 37.8 3.4 6.6 72.7 46.4 6.1 7.0 72.7 112.0 46.9 13.1 0.0 20.0 40.0 60.0 80.0 100.0 VGG-19 Inception-v3 SqueezeNet MobileNet runtime (ms) TVM-default TVM-tuned TensorRT MXNet (a) 3443.5 11729.1 (b) Figure 6.7: Inference performance of various software frameworks for running CNNs on TX2, and comparison with TVM. (a) CNN execution on GPU, and (b) CNN execution on CPU Throughput-optimized configurations. Tables 6.2–6.3 report the resulting device mapping with highest inference rate. The last two columns confirm that the ‘expected’ performance (from the optimization problem defined by Eq. (6.3) and network profiling data) is remarkably close to the experimental measurements. ‘Measured’ experimental performance is also reported in Fig. 6.8, with additional comparison with single-device execution. Interestingly, heterogeneous hardware exploitation achieves throughput improvements w.r.t. the best single-device case ranging from 22% to 150% for models compiled by default; and up to 22% for auto-tuned models. Only for tuned MobileNet and SqueezeNet, the performance resulting from the proposed two-stage approach remains similar to GPU-only execution, as otherwise analytically expected. Table 6.2: Throughput-optimized configurations – (i) default TVM models Network Configuration (# layers) Expected T(img/s) Measured T(img/s) MobileNet 23 GPU →8 CPU 100.0 img/s 98.0 ±2.3 img/s SqueezeNet-v1.1 32 GPU →8 CPU 100.2 img/s 101.3 ±6.6 img/s Inception-v3 88 GPU →34 CPU 9.5 img/s 10.9 ±0.2 img/s VGG-19 20 GPU →6 CPU 17.9 img/s 16.0 ±0.2 img/s Energy-optimized configurations. GPU-only execution is the most energy-efficient configuration for all the networks except for default SqueezeNet. The underlying reason is the combination of reduced runtime on GPU with reduced data-transfer energy cost. In the particular case of default SqueezeNet, it 6.4. Performance Evaluation 87 Table 6.3: Throughput-optimized configurations – (ii) tuned TVM models Network Configuration (# layers) Expected T(img/s) Measured T(img/s) MobileNet 4 CPU →27 GPU 167.9 img/s 139.9 ±2.8 img/s SqueezeNet-v1.1 6 CPU →34 GPU 175.0 img/s 161.2 ±11.9 img/s Inception-v3 5 CPU →117 GPU 20.9 img/s 24.5 ±0.9 img/s VGG-19 20 GPU →6 CPU 17.9 img/s 16.8 ±0.1 img/s 0 20 40 60 80 100 GPU CPU Pipeline (a) MobileNet (i) 0 20 40 60 80 100 GPU CPU Pipeline (b) SqueezeNet (i) 0 2 4 6 8 10 12 GPU CPU Pipeline (c) Inception-v3 (i) 0 5 10 15 GPU CPU Pipeline (d) VGG-19 (i) 0 50 100 150 GPU CPU Pipeline (e) MobileNet (ii) 0 50 100 150 GPU CPU Pipeline (f) SqueezeNet (ii) 0 5 10 15 20 25 GPU CPU Pipeline (g) Inception-v3 (ii) 0 5 10 15 GPU CPU Pipeline (h) VGG-19 (ii) Figure 6.8: Throughput (img/s) comparison of one-device execution vs. throughput-optimized cases. Four CNN models were assessed (i) without and (ii) with TVM auto-tuning 0 20 40 60 80 100 120 0 200 400 600 Throughput (img/s) Energy (mJ) (a) MobileNet (i) 0 20 40 60 80 100 120 0 200 400 600 Throughput (img/s) Energy (mJ) (b) SqueezeNet (i) 0 2 4 6 8 10 12 0 2000 4000 6000 Throughput (img/s) Energy (mJ) (c) Inception-v3 (i) 0 5 10 15 20 0 10000 20000 30000 Throughput (img/s) Energy (mJ) (d) VGG-19 (i) 0 20 40 60 80 100 120 140 160 0 200 400 Throughput (img/s) Energy (mJ) (e) MobileNet (ii) 0 50 100 150 200 0 100 200 300 Throughput (img/s) Energy (mJ) (f) SqueezeNet (ii) 0 5 10 15 20 25 30 0 2000 4000 Throughput (img/s) Energy (mJ) (g) Inception-v3 (ii) 0 5 10 15 20 0 5000 10000 15000 Throughput (img/s) Energy (mJ) (h) VGG-19 (ii) Figure 6.9: Performance trade-offs for GPU-only (green), CPU-only (blue), and throughput-optimized pipeline (black) for the CNNs considered in this chapter; “(i)” denotes networks compiled with TVM by default whereas “(ii)” denotes models compiled with compiling optimizations produced by Ansor auto-tuning tool is optimized in terms of both frame rate and energy when using a heterogeneous GPU →CPU pipeline with ˆn=32. Experimental energy-consumption measurements agree with these analytical expectations, as will be demonstrated in Sect. 6.5. Latency-optimized configurations. Similarly, GPU-only execution is the best configuration for fastresponse applications. Optimal latency metrics correspond to those shown in Fig. 6.7a. General Trade-offs. Finally, Fig. 6.9 shows performance metrics for each CNN under various configurations. Single-device execution is presented with green markers for GPU, and blue ones for CPU. Energy-optimized and latency-optimized configurations match with GPU execution – in green – (except for default SqueezeNet). Two-stage scheduling for optimal throughput is shown in black. A particularly remarkable case is precisely default SqueezeNet, whose throughput increases 150% when running upon a heterogeneous scheme, with an energy reduction of 66%. 94 A case study: remote animal monitoring access includes thousands of camera-trap images. We also extracted frames from the video sequences. Figure 7.2 shows examples of images in the dataset. Note how challenging the visual analysis required in this scenario is, with images that might picture individuals far away or close to the camera location, varying weather/illumination conditions, blurring, etc. Overall, the available dataset comprises a variety of park locations, environmental conditions, and species. Figure 7.2: Training images belonging to three categories: (i) wildcat, (ii) other species, and (iii) blank 7.3.1 Leveraging the dataset Before training, the dataset must be curated to boost training performance and ensure convergence. •Data variability and data augmentation. As a rule of thumb, 1000 images per category could be deemed suitable to train a network from scratch. Indeed, the larger and more varied the training dataset is (in our case study, this means including as many camera locations as possible19), the better the model generalization with respect to unseen contents, and the more reliable the accuracy rate resulting from training. Data augmentation techniques – including random cropping, horizontal flipping, color distortion, brightness/contrast modification, etc – provide more variability in the available data, which is crucial in datasets with a low number of samples. •Data balancing, i.e., an even distribution of images among classes, is required to minimize a classification bias towards majority classes. If the dataset is not properly balanced, it has been proved that categories with fewer samples, i.e., rare species in our case, are classified with low accuracy – even if the global model accuracy meets the corresponding specification. •Data quality. Avoiding confusing data in the training dataset is essential. In our case, very close or far away individuals, occluded animals, or blurred images lie in this category. Moreover, manual image segmentation by cropping out the animal region can facilitate the training procedure. Following the notes above, captures depicting individuals mostly occluded as well as fuzzy images were removed from our original dataset. Class balance was ensured by selecting proportionate sub-sets for each category. The final dataset resulting from the original data collection contains more than 4000 images/frames, balanced among the three considered classes. This dataset was augmented – rotated, flipped, etc. – during model training. 19It has been demonstrated that introducing new camera-trap locations in the test dataset – not seen by the network during training – produces an accuracy loss [280]. Thus, scene and location variability in the training data will enhance the network accuracy. 7.4. Model Training 95 7.4 Model Training 7.4.1 Training Classification Networks For the targeted visual task, i.e., identification of Scottish-wildcat individuals, two classification approaches were explored according to the two category hierarchies defined in Section 7.3: •Atwo-stage workflow, in which a first CNN trained on images containing animals vs. those without any individuals – i.e., category (I) vs. category (II) – filters out blanks. A second network identifies the species – either category (i), wildcat; or category (ii), other species – within category (I). Therefore, wildlife detection (first stage) is followed by species identification (second stage): detection (T1) →species identification (T2) •Aone-stage method that employs a CNN for 3-class classification. The network must be trained on images belonging to the three categories: (i) Scottish wildcats, (ii) other species, and (iii) blank images. Therefore, it will detect animals and distinguish the Scottish wildcat against other species in a single step. Thus, we define three networks tasks, for which three corresponding training datasets had to be created from the original one: T1) Filter out empty images, for the first step of the two-stage approach. The training dataset must contain images belonging to categories (I)–(II), i.e., animal vs. blank. T2) Identify which species are contained on the ‘animal’ images, for the second step of the two-stage approach. The network must be trained on categories (i)–(ii), i.e., wildcat vs. other species. T3) Identify species and remove blanks, for the one-stage approach. The CNN must be trained on the complete dataset, which encompasses categories (i) to (iii). According to the graphical methodology for optimal selection developed in Chap. 3,SqueezeNet was here selected as the baseline CNN for this application. We demonstrated that this model performs the best for the majority of edge application scenarios. It is particularly suitable for scenarios in which power consumption or frame rate are critical. Furthermore, its reduced number of parameters makes it more robust to overfitting – although it may be at the cost of losing some accuracy. In each case, the training data were split into 80%–20% fractions for training and validation datasets, respectively. Data augmentation included zooming, rotation (< 5◦), width/height shifting, and horizontal flipping. Given our relatively small dataset and to facilitate training convergence, we applied transfer learning from weights trained on ImageNet. Before training SqueezeNet, the last 1000-category layer was replaced by a reduced classifier with the desired number of output classes (two for tasks T1-T2, and three for T3). Network training was carried out by minimizing a categorical cross-entropy function using the Root Mean Square Propagation (RMSprop) optimizer [57]. We used the TensorFlow-Keras high-level API for training – TensorFlow version 2.1.0 in particular. 96 A case study: remote animal monitoring 7.4.1.1 Results We next analyze the training results achieved on the three classification networks for tasks T1–T3. T1 – Filter out blanks. This task provides animal identification within the entire dataset. Therefore, as usually done for this type of networks, i.e., detectors, we can associate the ‘animal’ class to positive samples and the ‘blank’ class to negative ones. Thus, correct classifications into the animal/blank classes become true positives/negatives (TP, TN), respectively. Likewise, incorrect classifications become false positives (FP) or false negatives (FN). •The same metrics as for detectors can be used. Precision (P)contemplates the percentage of correct detections. Recall (R), also known as sensitivity, is the rate of correct detections within actual positive (animal) samples. P=T P T P +FP =T P #Detections R=T P T P +FN =T P #Positives (7.1) •Accuracy is calculated as usually done for classification tasks20: Acc =T P +T N T P +T N +FP +FN =T P +T N #Images (7.2) The prediction confusion matrix on the validation dataset after training is shown in Table 7.1(a). The precision rate is 0.74, whereas the recall is equal to 0.77. The network attains an accuracy of 73.8% on the validation dataset. Note that all the results on training metrics presented in this chapter were calculated on the validation dataset. Thus, the aforementioned confusion matrix only refers to 20% of the samples. Table 7.1: Two-stage approach – confusion matrices (a) T1 – Presence (I) vs. Absence (II) Pred. (I) Pred. (II) (I) Presence 215 63 (II) Absence 76 177 (b) T2 – two-category species classification Pred. (i) Pred. (ii) (i) Wildcat 240 27 (ii) Mixed 10 294 T2, T3 – Species classification. In task T2, to identify wildcats vs. other species, the trained architecture presents the confusion matrix shown in Table 7.1(b). Its corresponding network accuracy is 93.5%. In this case, considering ‘wildcat’ detections as positive samples, the precision and recall rates are 0.96 and 0.90, respectively. Regarding the three-category network for the one-stage approach (T3), its confusion matrix is shown in Table 7.2. Table 7.2: One-stage approach – confusion matrix Pred. (i) Pred. (ii) Pred. (iii) (i) Wildcat 199 31 37 (ii) Mixed 6 285 13 (iii) Blank 72 21 160 20Note that task T1 requires classification models for detecting individuals. Thus, the network cannot produce more than one detection per image, and T P +T N +FP +FN equals the number of test images. 7.4. Model Training 97 The results for the three tasks are summarized in Table 7.3. Some examples of T3 are shown in Figs. 7.3– 7.4. Note that confidence thresholding can improve accuracy once the network is trained. It implies that network predictions with low certainty (CNN output probability below a prescribed threshold) are not taken into account. For instance, in the case of a network classifying ‘animal’ images vs. ‘blank’ ones (T1), only animal detections with high outcome probability will be considered, thus reducing false-alarm cases. While this technique certainly improves accuracy, it also reduces the classification coverage [270]. Table 7.3: Network accuracy metrics for wildlife monitorization. Results on fine-tuned SqueezeNet Task Precision Recall Accuracy T1) Filter out blanks 0.74 0.77 73.8% T2) Identify wildcats 0.96 0.90 93.5% T3) Filter out blanks & Identify wildcats 0.85 0.91 78.2% Figure 7.3: Examples of correct classifications with the three-category classification network: (i) wildcat, and (ii) other mixed species, and (iii) empty image Figure 7.4: Examples of incorrect classifications with the three-category classification network. They are challenging cases even for humans. The first two images contain a wildcat whereas the last one is blank. Note that confidence thresholding can improve the accuracy rate 7.4.2 Training Object-Detection Networks In addition to the two proposed approaches above based on classification CNNs, object-detection networks are also contemplated in this section. Concerning object detection, network training requires a collection of images annotated with bounding boxes. To assist on image bounding-box annotation, the LabelImg [90] tool was employed so that our data format became consistent with that of the PASCAL VOC dataset [27]. We manually annotated bounding boxes of approximately one thousand images. Two categories were assigned: (i) wildcat, and (ii) mixed species. Model definition and training were facilitated by a tool [281] relying on Keras to train object detectors. In particular, we employed the lightweight YOLOv3 [49] architecture. 98 A case study: remote animal monitoring 7.4.2.1 Results Precision and recall of object – i.e., animal – detection are summarized in Table 7.4 for IoU=0.5, as employed on PASCAL VOC [27]. The Average Precision (AP) for each category is the area under the precision–recall curve, which considers the accumulated TP and FP samples [27]. Overall, the network achieves a mean AP (mAP) equal to 98.5% averaged over the two species. According to the COCO criteria, the calculation of the mAP (AP averaged over 10 IoU thresholds [29]) results in 66.3%. Table 7.4: Detection metrics on fine-tuned YOLO-v3 (IoU=0.5) Precision Recall AP Wildcat 0.98 1 99.9% Mixed Species 0.98 0.97 97.1% Two images exemplifying animal detection are shown in Fig. 7.5. Figure 7.5: Test examples on YOLO-v3 network trained for the detection of (i) wildcat, and (ii) other species. The high accuracy proves the applicability of CNN-based object detection for this wildlife dataset – once annotated 7.4.3 Discussion Some further comments can be made on the proposed methods according to their intrinsic aspects and the measured evaluation metrics. First, CNN-based object detection is an accurate approach that improves generalization with respect to classification methods, and also enables the detection of more than one individual per image. However, its computational load is notably high, causing a strong impact on the energy budget, CPU usage, and response time in embedded implementations. In addition, manual image annotation of bounding boxes is a tedious and not always addressable task that must be realized in advance. Finally, not all operations involved in object detectors are implemented in most CNN libraries, which may make them incompatible for execution on general-purpose embedded platforms. Concerning classification networks, their lower complexity make them more suitable for efficient implementation on embedded systems. The training metrics obtained for SqueezeNet are satisfactory and validate the applicability of this DL-based approach to our wildlife application scenario. According to the presented results, we made the design choice of employing a classification network (simpler than object-detection networks) within the one-stage DL approach. 7.4. Model Training 99 7.4.4 Explainable Deep Learning Can we ensure that our CNN have correctly learnt relevant features? CNNs are complex models typically composed of many layers. In practical terms, they behave as a black box trainable to perform a target task. The internal network operation is not easy to uncover, and the learnt features which a CNN relies on to output a classification vector cannot be extracted in a simple manner. The field of Explainable Deep Learning [282] provides various approaches to reveal DL operation in an interpretable way. For instance, locally interpretable model-agnostic explanations (LIME) [283] is a popular technique that allows understanding how any black-box ML model (e.g., a classification CNN) makes its prediction. This method defines a model (g), simpler than the unexplainable model f(x),x∈Rd, in order to “explain” the importance which fgives to each input feature. In our case, model fis the classification CNN and f(x)is the probability that xbelongs to a class – for simplicity, multiple classes are explained separately. The underlying LIME’s explanation basis is the fact that removing features which are relevant for the CNN classification decision significantly affects the classification score f(x). Then, the explainable model gwill also assign the same importance to that feature. Mathematically, a model g:{0,1}d′→Ris defined to explain the black-box model f:Rd→R. This explanation model g(z)operates on interpretable components (e.g., a binary vector denoting the absence/presence of a certain pattern) and outputs an interpretable explanation. LIME tends to keep the model gsimple by minimizing its complexity Ω(g). To build g, the idea is to feed the black-box model with data points x′sampled in a vicinity of x, denoted as Πx, and observe the model outputs f(x′). The model gis then adjusted with such local points and their associated output decisions in order to locally explain them. That is, the model gminimizes a locality-aware loss L(f,g,Πx)that measures “how unfaithful g approximates the black-box model fin its vicinity Πx” (while keeping the model complexity low): arg min gL(f,g,Πx)+ Ω(g)(7.3) For image classification networks, a standard algorithm first divides the image into super-pixels. Then, LIME produces an explanation for each one. The output decision of the network constitutes the interpretable components z∈ {0,1}, whereas the produced interpretable explanation g(z)will indicate whether that super-pixel “explains” the corresponding network output z. For simplicity, we will only consider the CNN output for the most relevant class (i.e., the greater item in the output classification vector), and then LIME’s model gwill explain the assigned class. We employed the Python implementation of this method [284] – compatible with Keras – to understand the operation of our three-category CNN. Fig. 7.6 shows examples of correct classifications with this network. Note that the considered CNN have correctly learnt to identify animal features in order to produce its classification output. By contrast, Fig. 7.7 illustrates the network decision of a CNN whose training process must be enhanced; note that the classification decision was made on the basis of inappropriate image features, as shown in its corresponding LIME’s explanation map. Alternative training settings (‘Evaluation’ step in Sect. 7.2) and monitoring training metrics are required to delve into possible issues, such as overfitting or lack of convergence. In particular, the CNN whose decision is analyzed in Fig. 7.7 was trained with a too high learning rate. 100 A case study: remote animal monitoring Figure 7.6: Left: test input images; right: their corresponding interpretable representations. The explanation map indicates whether each super-pixel “explains” the classification output (green tones) or it does not (purple tones). Concerning the first image, it was classified by the network as (i) wildcat. On the right, LIME’s best explanation of this output is represented by the darkest green area, where gis greater. Regarding the second image, the CNN output category was (ii) mixed species, and its primary explanation is also highlighted in green Figure 7.7: Example of output interpretation for a CNN inadequately trained. The classification decision is not based on visually understandable key features, e.g., animal shapes. Further training adjustments must be made to achieve classification decisions on proper information and hence higher classification accuracy Explainable DL uncovers CNN operation and allows identifying issues such as model bias and causality in network output decisions. 7.5 Visual System Implementation Once the decision about the DL-based method to be implemented has been made, and the selected network has been properly trained and evaluated, it is time to address the embedded realization. To this end, Chaps. 3and 4can assist with the challenge of achieving real-time processing while minimizing energy demand. This section summarizes the hardware–software design choices made in our application context. The final performance indicators will be presented in Sect. 7.6. 7.5.1 Hardware To make the most of the experiments and code developed along this thesis, the core platform selected to build a DL-based smart camera trap was a RPi. In addition, for a complete implementation of the vision system, external sensors and peripherals were integrated. 7.5. Visual System Implementation 101 Edge Platform. It is a RPi model 4B [285]. Its Broadcom BCM2711 system-on-chip includes a quad-core ARMv8 Cortex-A72 64-bit CPU and a Broadcom VideoCore VI GPU. Among the three available RAM options, we selected a RPi4B model featuring 8 GB LPDDR4-3200 SDRAM. Nevertheless, alternative edge devices could be used, such as Raspberry Pi Zero [286], NCS [114], or Google Coral USB accelerator [113] – see hardware review in Chap.2. Camera Module. Different camera modules can be integrated with RPi4B. Among them, the Raspberry Pi camera module [287] is the most popular one. For specific operation environments, we can find alternative devices in the market, such as NoIR cameras (without infrared filter, for night or poorly illuminated scenarios) [288], high-quality cameras (with increased resolution, and better sensitivity and low-light performance) [289], and even third-party lens adapters to accommodate other lenses [290]. PIR Sensor. Most commercial camera traps are equipped with a passive infrared (PIR) sensor to detect individuals before triggering the camera capture. These sensors are designed to detect moving objects whose surface temperature differs from that of the background environment. In particular, we explored high-performance PIR sensors from Panasonic series EKMB (high-end), EKMC (economy), and AMN (second generation) [291]. Battery Supply. Autonomous remote operation must be enabled with the use of an external battery providing 5.1 V to the RPi-based system [292]. In addition, solar panels could also be attached in order to prolong the battery lifetime. Internet Peripheral. To provide wildlife conservationists with real-time information, the camera must be able to remotely send processed data21. Different types of notification could be sent: an SMS, e-mail, etc. Camera status or low-battery indicators could also be informed. A further advantage of camera connectivity is the possibility of remotely adjusting camera settings. In particular, RPi4B platform includes 802.11 Wi-Fi, Gigabit Ethernet, and Bluetooth technologies for node communication. We will assume that a wireless local area network (WLAN) provides connectivity with a central server deployed in the park. In practical scenarios in which this connectivity is not available, GSM, 4G/5G, ZigBee, or other IoT communication alternatives must be explored. 7.5.2 Software Libraries A variety of libraries and tools can be leveraged for developing the targeted embedded vision application. Here, we summarize some of them for inclusion in the software stack. Operating system. Specifically designed for the RPi series platforms, Raspberry Pi Foundation officially provides an OS formerly called Raspbian [159]. However, there are other Linux-based alternatives, such as piCore Linux [293], Arch Linux [294], or Ubuntu MATE [295]. GPIO. Accessing the general-purpose input/output (GPIO) pins on the RPi is facilitated by the pigpio library [296]. Therefore, we can easily control external sensors and/or actuators. RPi camera libraries. Code to access the camera can leverage the Broadcom-specific Multimedia Abstraction Layer (MMAL) API, intended for using the VideoCore IV GPU on RPi. It runs upon 21In the case of non-networked cameras, which nowadays constitute the majority of the camera-trap market, processed data must be stored in the camera system, i.e., information is not accessible in real time. Camera keepers have to eventually retrieve their storage resource containing both raw data and processed information. 102 A case study: remote animal monitoring other C library called OpenMAX. This API facilitates the access to multimedia components running on VideoCore, such as the RPi camera [297, 287]. The camera operation can be tuned by setting several parameters, depending on the surrounding conditions and the target application [287]. Raspbian OS also offers some command-line applications to capture images and video streams, such as raspistill and raspivid [287]. In addition, other libraries may be employed for accessing the camera, including Raspicam [298] or the widely used OpenCV library. Motion-detection software. Exploiting temporal information in video sequences can facilitate animal detection. Automatic motion detection constitutes an alternative to PIR sensors. These motion-detection algorithms rely on continuous video streams and classical computer-vision approaches. There are some open-source libraries that implement these motion-detection techniques on camera sequences. For example, two popular and useful software tools, which could be indeed applied to any embedded visual application, are here reviewed: •Motion software [299, 300]. This is a general motion-detection Linux library, compatible with several cameras and platforms. It provides an on-screen display (OSD) web interface to remotely monitor the camera stream. Its core algorithm for motion detection is based on finding significant changes on the captured scene. A number of options can be configured in order to establish camera operation settings, tune the algorithm parameters, run scripts when motion is detected, set the configuration for saving motion-detected images/videos, control web interface, etc. •PiKrellCam software [301, 302]. As opposed to the Motion software, this tool particularly targets RPi hardware and camera. This motion-detection software, whose algorithm is based on motion vectors, also offers an OSD web interface. Its algorithm allows detecting motion only on specified image regions. Images and videos can be automatically recorded when motion is detected, or manually triggered via web interface. CNN inference framework. Neural networks are composed of a set of weights and operations. The computational graph of operations can be coded in multiple ways, as demonstrated by the diversity of implementations available in DL frameworks. Concerning weights, the values adjusted during training must be stored in a file with a particular data format – in fact, this file can also keep CNN graph information. In our case study, after training the networks and achieving a satisfactory accuracy, a TensorFlow model file was rendered that may be transformed to other format compatible with the DL framework finally employed during the deployment stage. In general, translating trained networks into other formats is not easy at all. A counterexample of this is the conversion from TensorFlow to TensorFlow Lite, which is simple [303]. However, it is not the case for the majority of possible format conversions, such as from/to Caffe, Caffe2, Torch, etc. In this regard, it is worth mentioning the open-source ONNX package, which was released to facilitate library format conversion – through an intermediate representation. The library objective is to enable possible conversions involving well-known DL frameworks such as TensorFlow, Caffe, and PyTorch. However, not all conversion cases are possible yet. We made the decision of selecting SqueezeNet network running on optimized C++ OpenCV. According to Chap. 3, this selection will provide a high inference speed, which determines energy consumption as well. Note that although model training and tests for developing the application can be carried out on the easy-to-use Python language, the embedded application will be coded on C++ and compiled in order to improve the system performance. 7.5. Visual System Implementation 103 7.5.3 Application Algorithm The embedded application for this case study must combine various libraries explained above to implement the targeted functionality. A basic workflow is presented in Fig. 7.8. The operation and particular realization of each step are described below. SEND ALARM?TRIGGER CAMERA CAPTURE CNN Figure 7.8: Basic workflow for the targeted smart-camera application, i.e., remote animal monitoring 7.5.3.1 Motion detection – camera trigger Conventionally, camera traps are automatically triggered by the presence of an animal – ideally – or by spurious motion in the scene. No discrimination is made on the actual triggering cause, and images and/or video sequences are usually recorded straightaway. In our case, we aim at incorporating smartness in such discrimination, but triggering must still be implemented. Among the aforementioned alternatives, we dismissed running a motion-detection algorithm. Continuously capturing and processing a video stream implies a high energy demand. In contrast, PIR sensors have been largely used for power-efficient motion detection. In fact, commercial camera traps usually include PIR sensors to trigger the camera. Therefore, a PIR sensor was integrated, in particular a high-end sensor from the Panasonic EKMB 6 uA series, which is advertized for security cameras as well as intrusion alarm systems. Its specifications report a detection distance up to 12 m with a horizontal/vertical field of view of 108◦×99◦. 7.5.3.2 Camera capture Once motion is detected, the camera must confirm the presence of an individual and eventually proceed with information recording. The camera operation can be variably configured: image vs. video recording, capture time, frames per minute, etc. Regarding commercial camera traps, not all of them can record video. Video sequences demand heavier processing for storage and the resulting files require much more non-volatile memory. Instead, camera traps typically take multiple pictures (ranging usually from 1 to 10) after each trigger event – “burst” mode. Capture speeds vary from few seconds between frames to faster rates (>1fps). To reduce energy consumption, our smart camera will capture and process images instead of video sequences. Capture parameters can be configured by the user. 7.5.3.3 CNN classification A CNN architecture constitutes the core intelligent algorithm to identify the species of interest, thus giving rise to a smart DL-based camera trap. As comprehensively explained in previous sections, the trained three-category network implicitly filters out blank images resulting from false-positive motion-detection A case study: remote animal monitoring •Related to the previous point, we also provide an extensive characterization of DL components expressed through a wide set of metrics under varying operation conditions. Together, these two contributions provide a faithful global picture of modern DNN-based embedded vision systems. •We propose strategies to leverage the limited resources of edge devices. They encompass from monitoring key hardware parameters that warn against processing and memory bottlenecks to experimentally characterizing a single ad-hoc network architecture to speed up CNN selection without further modeling of the toolchain. These strategies can be exploited by automatic optimization algorithms or to make informed low-level decisions for network design, model compression, network distribution, etc. •Meaningful metrics and FoMs are introduced to evaluate diverse system configurations and determine those that meet application requirements in a rapid and effective manner. •We propose a procedure to treat the numerical values of FoMs and graphically represent them for direct identification of both optimal and worst model/framework combinations. We demonstrate that this procedure is able to filter out most of such combinations after sweeping all possible application scenarios, which are compactly encoded through a small set of parameters. •An easy-to-implement and general method to exploit heterogeneous hardware available in edge platforms is proposed. This method is especially pertinent given the current trend in which both a CPU and a GPU are integrated even in low-end processing devices, e.g., RPi. •Based on the results and conclusions resulting from the realization of the above contributions, a specific real-world application scenario has been addressed. We demonstrate the current possibilities and limitations of embedded vision based on DNN inference. Bibliography [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, NIPS, 2012, pp. 1097–1105. [Online]. Available: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf [2] O. Russakovsky et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision (IJCV), vol. 115, no. 3, pp. 211–252, 2015. [3] M. Alam et al., “Survey on deep neural networks in speech and vision systems,” Neurocomputing, vol. 17, pp. 302–321, 2020. [4] “Edge AI and vision alliance, computer vision developer survey,” https://www.edge-ai-vision.com/2020/02/ edge-ai-and-vision-alliance-november-2019-computer-vision-developer-survey-white-paper/. [5] Y. Le Cun et al., “Handwritten digit recognition: applications of neural network chips and automatic learning,” IEEE Communications Magazine, vol. 27, no. 11, pp. 41–46, 1989. [6] “IEEE international roadmap for devices and systems (IRDS),” https://irds.ieee.org/. [7] T. Perry, “David Patterson says it’s time for new computer architectures and software languages,” IEEE Spectrum, 2018. [Online]. Available: https://spectrum.ieee.org/ david-patterson-says-its-time-for-new-computer-architectures-and-software-languages [8] X. Wang, Y. Zhao, and F. Pourpanah, “Recent advances in deep learning,” Int. Journal of Machine Learning and Cybernetics, vol. 11, pp. 747–750, 2020. [9] V. Reddi et al., “MLPerf inference benchmark,” in Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020, pp. 446–459. [10] “Jeff Bier, computer vision 2.0: Where we are and where we’re going, Embedded Vision Summit, 2016,” https://www.edge-ai-vision.com/2016/05/ computer-vision-2-0-where-we-are-and-where-were-going-a-presentation-from-the-embedded-vision-alliance/. [11] A. Suleiman et al., “Towards closing the energy gap between HOG and CNN features for embedded vision,” in IEEE International Symposium on Circuits and Systems (ISCAS), 2017. [12] T. Yang, Y. Chen, and V. Sze, “Designing energy-efficient convolutional neural networks using energy-aware pruning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6071–6079. [13] V. Peluso and A. Calimera, “Energy-driven precision scaling for fixed-point convNets,” in IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), 2018, pp. 113–118. [14] A. Moreno et al., “Analysis of a pipelined architecture for sparse DNNs on embedded systems,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 28, no. 9, pp. 1993–2003, 2020. [15] J. Wang et al., “Enabling energy-efficient and reliable neural network via neuron-level voltage scaling,” IEEE Transactions on Computers, vol. 69, no. 10, pp. 1460–1473, 2020. [16] “Qualcomm, emerging vision technologies: Enabling a new era of intelligent devices, technical report, 2016,” https: //www.qualcomm.com/documents/emerging-vision-technologies-enabling-new-era-intelligent-devices. [17] “OrCam MyEye,” https://www.orcam.com/en/. [18] “Skydio Autonomy,” https://www.skydio.com/skydio-autonomy. [19] “Amazon Go,” https://www.computerweekly.com/feature/Amazon-Go-is-now-the-right-time. [20] “Artificial intelligence, a european perspective,” \https://publications.jrc.ec.europa.eu/repository/handle/JRC113826. 111 BIBLIOGRAPHY [21] D. Mishkin, N. Sergievskiy, and J. Matas, “Systematic evaluation of convolution neural network advances on the imagenet,” Computer Vision and Image Understanding, vol. 161, pp. 11 – 19, 2017. [22] S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating deep network training by reducing internal covariate shift,” in Proceedings of the 32nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 37, 2015, pp. 448–456. [23] C. Szegedy et al., “Going deeper with convolutions,” arXiv, no. 1409.4842, 2014. [24] ——, “Rethinking the inception architecture for computer vision,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 2818–2826. [25] C. Szegedy, S. Ioffe, and V. Vanhoucke, “Inception-v4, Inception-ResNet and the impact of residual connections on learning,” arXiv, no. 1602.07261, 2016. [26] M. Everingham et al., “The Pascal Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, pp. 303–338, 2009. [27] ——, “The Pascal Visual Object Classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, pp. 303–338, 2009. [28] R. Padilla et al., “A comparative analysis of object detection metrics with a companion open-source toolkit,” Electronics, vol. 10, no. 3, 2021. [Online]. Available: https://www.mdpi.com/2079-9292/10/3/279 [29] T.-Y. Lin et al., “Microsoft COCO: Common objects in context,” in Computer Vision – ECCV 2014, D. Fleet et al., Eds. Springer International Publishing, 2014, pp. 740–755. [30] A. Canziani, A. Paszke, and E. Culurciello, “An analysis of deep neural network models for practical applications,” arXiv, no. 1605.07678, 2017. [31] Y. Lecun et al., “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. [32] Y. LeCun, C. Cortes, and C. J. Burges, “The MNIST database of handwritten digits,” http://yann.lecun.com/exdb/mnist/. [33] M. Lin, Q. Chen, and S. Yan, “Network in network,” arXiv, no. 1312.4400, 2013. [34] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv, no. 1409.1556, 2014. [35] K. He et al., “Deep residual learning for image recognition,” arXiv, no. 1512.03385, 2015. [36] F. Iandola et al., “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size,” arXiv, no. 1602.07360, 2016. [37] A. Howard et al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv, no. 1704.04861, 2017. [38] M. Sandler et al., “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” arXiv, no. 1801.04381, 2018. [39] X. Zhang et al., “ShuffleNet: An extremely efficient convolutional neural network for mobile devices,” June 2018, pp. 6848–6856. [40] N. Ma et al., “ShuffleNet V2: Practical guidelines for efficient cnn architecture design,” in Proceedings of the European Conference on Computer Vision (ECCV), September 2018. [41] R. B. Girshick et al., “Rich feature hierarchies for accurate object detection and semantic segmentation,” 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587, 2014. [42] R. Girshick, “Fast R-CNN,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448. [43] S. Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, June 2015. [44] J. Dai et al., “R-FCN: Object detection via region-based fully convolutional networks,” in Advances in Neural Information Processing Systems, D. Lee et al., Eds., vol. 29. Curran Associates, Inc., 2016. [45] K. He et al., “Spatial pyramid pooling in deep convolutional networks for visual recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1904–1916, 2015. [46] W. Liu et al., “SSD: Single shot multibox detector,” in Computer Vision – ECCV 2016, B. Leibe et al., Eds. Springer International Publishing, 2016, pp. 21–37. [47] J. Redmon et al., “You Only Look Once: Unified, real-time object detection,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788, 2016. 113 [48] J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. [49] ——, “YOLOv3: An incremental improvement,” arXiv, no. 1804.02767, 2018. [50] H. Law and J. Deng, “CornerNet: Detecting objects as paired keypoints,” International Journal of Computer Vision, vol. 128, pp. 642–656, 2020. [51] H. Law et al., “CornerNet-Lite: Efficient keypoint based object detection,” in 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020. [52] K. Duan et al., “CenterNet: Keypoint triplets for object detection,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 6568–6577. [53] X. Zhou, J. Zhuo, and P. Krahenbuhl, “Bottom-up object detection by grouping extreme and center points,” June 2019, pp. 850–859. [54] T.-Y. Lin et al., “Focal loss for dense object detection,” in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999–3007. [55] M. Tan, R. Pang, and Q. V. Le, “EfficientDet: Scalable and efficient object detection,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10 778–10 787, 2020. [56] J. Redmon, “Darknet: open source neural networks in C,” http://pjreddie.com/darknet/. [57] S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv, no. 1609.04747, 2016. [58] Y. Jia et al., “Caffe: Convolutional architecture for fast feature embedding,” arXiv, no. 1408.5093, 2014. [59] “GitHub – Caffe. Model Zoo,” https://github.com/BVLC/caffe/wiki/Model-Zoo. [60] Abadi, M. et al., “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 2016, pp. 265–283. [Online]. Available: https: //www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf [61] “Caffe2,” https://caffe2.ai/. [62] A. Paszke et al., “Automatic differentiation in pytorch,” 2017. [63] “PyTorch,” https://pytorch.org/. [64] “OpenCV,” https://opencv.org/. [65] The Apache Software Foundation (ASF), “GitHub – MXNet,” https://mxnet.apache.org/. [66] T. Chen et al., “TVM: An automated end-to-end optimizing compiler for deep learning,” in Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’18. USA: USENIX Association, 2018, p. 579–594. [67] “GitHub – Paddle,” https://github.com/PaddlePaddle/Paddle. [68] “GitHub – Paddle Lite,” https://github.com/PaddlePaddle/Paddle-Lite. [69] “GitHub – Theano,” https://github.com/Theano/Theano. [70] Xiaomi, “GitHub – Mobile AI Compute Engine (MACE),” https://github.com/XiaoMi/mace. [71] X. Jiang et al., “MNN: A universal and efficient inference engine,” arXiv, no. 2002.12418, 2020. [72] “GitHub – Mobile Neural Network (MNN),” https://github.com/alibaba/MNN. [73] OPEN AI LAB, “GitHub – Tengine,” https://github.com/OAID/Tengine. [74] C. L. Lawson et al., “Basic linear algebra subprograms for Fortran usage,” ACM Trans. Math. Softw., vol. 5, no. 3, pp. 308–323, Sep. 1979. [75] T. e. a. Kielmann, Basic Linear Algebra Subprograms (BLAS), 2011. [76] Goto, K. et al., “Anatomy of high-performance matrix multiplication,” ACM Trans. Math. Softw., vol. 34, no. 3, pp. 12:1– 12:25, May 2008. [77] K. Goto and R. van de Geijn, “On reducing TLB misses in matrix multiplication,” 2002. [78] “Automatically tuned linear algebra software (ATLAS),” http://math-atlas.sourceforge.net. [79] “Intel Math Kernel Library,” https://software.intel.com/en-us/mkl. [80] Guennebaud, G. et al., “Eigen,” http://eigen.tuxfamily.org/. [81] “Dense linear algebra on GPUs,” https://developer.nvidia.com/cublas. [82] “NVIDIA cuDNN,” https://developer.nvidia.com/cudnn. BIBLIOGRAPHY [83] A. Anderson et al., “Low-memory GEMM-based convolution algorithms for deep neural networks,” ArXiv, vol. abs/1709.03395, 2017. [84] A. Vasudevan, A. Anderson, and D. Gregg, “Parallel multi channel convolution using general matrix multiplication,” in 2017 IEEE 28th International Conference on Application-specific Systems, Architectures and Processors (ASAP), 2017, pp. 19–24. [85] M. Mathieu, M. Henaff, and Y. LeCun, “Fast training of convolutional networks through FFTs,” in International Conference on Learning Representations (ICLR2014), 2014. [86] A. Lavin and S. Gray, “Fast algorithms for convolutional neural networks,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 4013–4021. [87] J. Cong and B. Xiao, “Minimizing computation in convolutional neural networks,” in Artificial Neural Networks and Machine Learning - ICANN 2014. Springer International Publishing, 2014, pp. 281–290. [88] “Netscope CNN analyzer,” http://dgschwend.github.io/netscope/quickstart.html. [89] “GitHub – NETRON,” https://github.com/lutzroeder/netron. [90] “LabelImg,” https://tzutalin.github.io/labelImg/. [91] “GitHub – LabelMe. Image Polygonal Annotation with Python,” https://github.com/wkentaro/labelme. [92] “ONNX,” https://onnx.ai/. [93] A. Shahid and M. Mushtaq, “A survey comparing specialized hardware and evolution in TPUs for neural networks,” in 2020 IEEE 23rd International Multitopic Conference (INMIC), 2020, pp. 1–6. [94] “Raspberry Pi. Products,” https://www.raspberrypi.org/products/. [95] “Raspberry Pi 3 Model B,” https://www.raspberrypi.org/products/raspberry-pi-3-model-b/. [96] “Odroid Board,” https://www.hardkernel.com/product-category/odroid-board/. [97] “LattePanda. A Windows 10 development board for everything,” https://www.lattepanda.com/. [98] “Orange Pi. New generation mini pc,” http://www.orangepi.org/index.html. [99] BeagleBoard.org, “BeagleBone Black,” https://beagleboard.org/black. [100] M. F. Cloutier, C. Paradis, and V. M. Weaver, “A Raspberry Pi cluster instrumented for fine-grained power measurement,” Electronics, vol. 5, no. 4, 2016. [Online]. Available: https://www.mdpi.com/2079-9292/5/4/61 [101] “Nvidia Developer. Jetson Nano Developer Kit,” https://developer.nvidia.com/embedded/jetson-nano-developer-kit. [102] “Nvidia Developer. Jetson Xavier NX Developer Kit,” https://developer.nvidia.com/embedded/jetson-xavier-nx-devkit. [103] “Nvidia Developer. Jetson Modules,” https://developer.nvidia.com/embedded/jetson-modules. [104] “CMSIS NN software library,” https://www.keil.com/pack/doc/CMSIS/NN/html/index.html. [105] L. Lai, N. Suda, and V. Chandra, “CMSIS-NN: Efficient neural network kernels for Arm Cortex-M CPUs,” arXiv, no. 1801.06601, 2018. [106] L. Heim et al., “Measuring what really matters: Optimizing neural networks for TinyML,” arXiv, no. 2104.10645, 2021. [107] Arduino, “Arduino Nano 33 BLE,” https://store.arduino.cc/arduino-nano-33-ble. [108] Nordic-Semiconductor, “nRF52840. Product Specification v1.1,” https://content.arduino.cc/assets/Nano_BLE_ MCU-nRF52840_PS_v1.1.pdf, 2019. [109] “STM32 32-bit Arm Cortex MCUs,” https://www.st.com/en/microcontrollers-microprocessors/ stm32-32-bit-arm-cortex-mcus.html. [110] “STM32 solutions for artificial neural networks,” https://www.st.com/content/st_com/en/ecosystems/stm32-ann.html# stm32sann-overview. [111] “SparkFun edge development board,” https://www.sparkfun.com/products/15170. [112] “Adafruit EdgeBadge – TensorFlow Lite for microcontrollers,” https://www.adafruit.com/product/4400. [113] Google-LCC, “Coral USB Accelerator datasheet. Version 1.4,” https://coral.ai/static/files/Coral-USB-Accelerator-datasheet. pdf, 2019. [114] Intel, “Data sheet. Intel Neural Compute Stick 2,” https://software.intel.com/content/dam/develop/public/us/en/documents/ ncs2-data-sheet.pdf, 2017. [115] ——, “Intel Movidius Neural Compute Stick,” https://software.intel.com/content/www/us/en/develop/articles/ intel-movidius-neural-compute-stick.html. 115 [116] Google-LCC, “Coral Dev Board datasheet. Version 1.6,” https://coral.ai/static/files/Coral-Dev-Board-datasheet.pdf, 2020. [117] “Tinker Edge R,” https://tinker-board.asus.com/product/tinker-edge-r.html. [118] H. Kong et al., “EDLAB: A benchmark for edge deep learning accelerators,” IEEE Design Test, pp. 1–1, 2021. [119] A. Ignatov et al., “AI benchmark: Running deep neural networks on android smartphones,” arXiv, no. 1810.01109, 2018. [120] ——, “AI benchmark: All about deep learning on smartphones in 2019,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019, pp. 3617–3635. [121] C. Luo et al., “Comparison and benchmarking of AI models and frameworks on mobile devices,” arXiv, no. 2005.05085, 2020. [122] J. Jo, S. Jeong, and P. Kang, “Benchmarking GPU-accelerated edge devices,” in 2020 IEEE International Conference on Big Data and Smart Computing (BigComp), 2020, pp. 117–120. [123] V. J. Reddi et al., “MLPerf inference benchmark,” no. 1911.02549, 2019. [124] P. Miliadis, C. Bouganis, and D. N. Pnevmatikatos, “Performance landscape of resource-constrained platforms targeting DNNs,” arXiv, no. 2107.10047, 2021. [125] B. Ulker et al., “Reviewing inference performance of state-of-the-art deep learning frameworks,” in Proceedings of the 23th International Workshop on Software and Compilers for Embedded Systems. Association for Computing Machinery, 2020, p. 48–53. [126] Y. Wang, G. Wei, and D. Brooks, “Benchmarking TPU, GPU, and CPU platforms for deep learning,” arXiv, no. abs/1907.10701, 2019. [127] P. KANG and J. JO, “Benchmarking modern edge devices for AI applications,” IEICE Transactions on Information and Systems, vol. E104.D, pp. 394–403, March 2021. [128] X. Rancaño et al., “Performance evaluation of state-of-the-art edge computing devices for DNN inference,” in IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, 2020, pp. 2286–2291. [129] “Jetson Nano: Deep learning inference benchmarks,” https://developer.nvidia.com/embedded/ jetson-nano-dl-inference-benchmarks. [130] E. Ostrowski and S. Kaufmann, “Survey of alternative hardware for neural network computation in the context of computer vision,” May 2020. [131] S. Bianco et al., “Benchmark analysis of representative deep neural network architectures,” IEEE Access, vol. 6, pp. 64 270– 64 277, October 2018. [132] J. Huang et al., “Speed/accuracy trade-offs for modern convolutional object detectors,” arXiv, no. 1611.10012, 2016. [133] S. Han et al., “Learning both weights and connections for efficient neural networks,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, ser. NIPS’15. Cambridge, MA, USA: MIT Press, 2015, p. 1135–1143. [134] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural network with pruning, trained quantization and Huffman coding,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y. Bengio and Y. LeCun, Eds., 2016. [135] H. Mao et al., “Exploring the regularity of sparse structure in convolutional neural networks,” arXiv, no. 1705.08922, 2017. [136] P. Molchanov et al., “Pruning convolutional neural networks for resource efficient transfer learning,” in International Conference on Learning Representations, November 2017. [137] H. Hu et al., “Network trimming: A data-driven neuron pruning approach towards efficient deep architectures,” arXiv, no. 1607.03250, 2016. [138] J. Zou et al., “Convolutional neural network simplification via feature map pruning,” Computers & Electrical Engineering, vol. 70, 2018. [139] Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in 2017 IEEE International Conference on Computer Vision (ICCV), October 2017, pp. 1398–1406. [140] M. Rastegari et al., “XNOR-Net: ImageNet classification using binary convolutional neural networks,” in European Conference on Computer Vision, October 2016, pp. 525–542. [141] Z. Cai et al., “Deep learning with low precision by half-wave gaussian quantization,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5406–5414, 2017. [142] S. Zhou et al., “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv, no. 1606.06160, 2016. BIBLIOGRAPHY [143] M. Courbariaux et al., “Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1,” arXiv, no. 1602.02830, 2016. [144] A. Zhou et al., “Incremental network quantization: Towards lossless cnns with low-precision weights,” in International Conference on Learning Representations (ICLR 2017), 2017. [145] J. Martinez et al., “Permute, quantize, and fine-tune: Efficient compression of neural networks,” arXiv, no. 2010.15703, 2020. [146] Y.-D. Kim et al., “Compression of deep convolutional neural networks for fast and low power mobile applications,” arXiv, no. 1511.06530, 2015. [147] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” arXiv, no. 1405.3866, 2014. [148] V. Sindhwani, T. Sainath, and S. Kumar, “Structured transforms for small-footprint deep learning,” in Advances in Neural Information Processing Systems, C. Cortes et al., Eds., vol. 28. Curran Associates, Inc., 2015. [149] E. Denton et al., “Exploiting linear structure within convolutional networks for efficient evaluation,” in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1, ser. NIPS’14. MIT Press, 2014, p. 1269–1277. [150] T. Elsken, J. H. Metzen, and F. Hutter, Neural Architecture Search. Springer International Publishing, 2019, pp. 63–77. [Online]. Available: https://doi.org/10.1007/978-3-030-05318-5_3 [151] B. Zoph et al., “Learning transferable architectures for scalable image recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8697–8710. [152] M. Tan et al., “MnasNet: Platform-aware neural architecture search for mobile,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. [153] H. Pham et al., “Efficient neural architecture search via parameters sharing,” in Proceedings of the 35th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, July 2018, pp. 4095–4104. [154] H. Cai, L. Zhu, and S. Han, “ProxylessNAS: Direct neural architecture search on target task and hardware,” in International Conference on Learning Representations, 2019. [155] B. Wu et al., “FBNet: Hardware-aware efficient ConvNet design via differentiable neural architecture search,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. [156] D. Stamoulis et al., “Single-Path NAS: Designing hardware-efficient convnets in less than 4 hours,” ArXiv, no. 1904.02877, 2019. [157] A. Gordon et al., “MorphNet: Fast & simple resource-constrained structure learning of deep networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. [158] Velasco-Montero, D. et al., “Performance analysis of real-time DNN inference on Raspberry Pi,” in SPIE Commercial + Scientific Sensing and Imaging, April 2018. [159] “Raspberry Pi. Operating system images,” https://www.raspberrypi.org/software/operating-systems/. [160] S. Hasanpour et al., “Lets keep it simple, using simple architectures to outperform deeper and more complex architectures,” arXiv, no. 1608.06037, 2016. [161] “GitHub – OpenBLAS, optimized BLAS library based on GotoBLAS2 1.13 BSD version,” https://github.com/xianyi/ OpenBLAS. [162] “GitHub – A Docker image for Tensorflow,” 2017 https://github.com/DeftWork/rpi-tensorflow. [163] “GitHub – TensorFlow-Slim image classification model library,” https://github.com/tensorflow/models/tree/master/research/ slim. [164] “GitHub – OpenCV. Open Source Computer Vision Library,” https://github.com/opencv/opencv. [165] “GitHub – Caffe2. Model Zoo,” https://github.com/caffe2/caffe2/wiki/Model-Zoo. [166] “Caffe to Caffe2,” https://caffe2.ai/docs/caffe-migration.html#caffe-to-caffe2. [167] “GitHub – Caffe to TensorFlow,” 2017 https://github.com/ethereon/caffe-tensorflow. [168] “GitHub – Model Zoo. Network in Network model,” https://github.com/BVLC/caffe/wiki/Model-Zoo# network-in-network-model. [169] “GitHub – BAIR/BVLC GoogLeNet Model,” 2017 https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet. 117 [170] “GitHub – Deep Residual Learning for Image Recognition,” 2016 https://github.com/KaimingHe/deep-residual-networks. [171] “GitHub – SqueezeNet_v1.1,” 2016 https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.1. [172] “GitHub – Caffe Implementation of Google’s MobileNets (v1 and v2),” 2018 https://github.com/shicai/MobileNet-Caffe. [173] “GitHub – SimpleNet,” 2018 https://github.com/Coderx7/SimpleNet. [174] “Inception V1,” 2016 http://download.tensorflow.org/models/inception_v1_2016_08_28.tar.gz. [175] “ResNet V1 50,” 2016 http://download.tensorflow.org/models/resnet_v1_50_2016_08_28.tar.gz. [176] “MobileNet_v1_1.0_224,” 2018 http://download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_ 224.tgz. [177] “GitHub – Caffe2 Models. BVLC GoogLeNet,” https://github.com/caffe2/models/tree/master/bvlc_googlenet. [178] “GitHub – Caffe2 Models. SqueezeNet,” https://github.com/caffe2/models/tree/master/squeezenet. [179] “Raspberry Pi Documentation. Frequency management and thermal control,” https://www.raspberrypi.org/documentation/ computers/raspberry-pi.html#frequency-management-and-thermal-control. [180] G. Chiandussi et al., “Comparison of multi-objective optimization methodologies for engineering applications,” Computers & Mathematics with Applications, vol. 63, no. 5, pp. 912–942, 2012. [181] C. Luo et al., “Comparison and benchmarking of ai models and frameworks on mobile devices,” ArXiv, no. 2005.05085, 2020. [182] D. Velasco-Montero et al., “Optimum selection of DNN model and framework for edge inference,” IEEE Access, vol. 6, pp. 51 680–51 692, 2018. [183] ARM, ARM®Cortex®-A53 MPCore Processor. Technical reference manual. [184] “ARM Processors.Cortex-A53,” https://developer.arm.com/products/processors/cortex-a/cortex-a53. [185] ARM, ARM®Architecture Reference Manual. ARMv8, for ARMv8-A architecture profile, 2017. [186] ——, ARM Cortex-A53 MPCore Processor Advanced SIMD and Floating-point Extension. Technical Reference Manual. [187] “perf: Linux profiling with performance counters,” https://perf.wiki.kernel.org/index.php/Main_Page. [188] “Raspberry Pi Documentation. vcgencmd,” https://www.raspberrypi.org/documentation/computers/os.html#vcgencmd. [189] “RPi vcgencmd usage,” https://elinux.org/RPI_vcgencmd_usage. [190] “CPU frequency and voltage scaling code in the Linux(TM) kernel,” https://www.kernel.org/doc/Documentation/cpu-freq/ governors.txt. [191] A. Yasin, “A top-down method for performance analysis and counters architecture,” in 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2014, pp. 35–44. [192] M. Becker and S. Chakraborty, “Measuring software performance on linux,” arXiv, no. 1811.01412, 2018. [193] D. Velasco-Montero et al., “Performance assessment of deep learning frameworks through metrics of CPU hardware exploitation on an embedded platform,” International journal of electrical and computer engineering systems, vol. 11, no. 1, 2020. [194] D. Li et al., “Evaluating the energy efficiency of deep convolutional neural networks on CPUs and GPUs,” in 2016 IEEE Int. Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom), October 2016, pp. 477–484. [195] A. Canziani, A. Paszke, and E. Culurciello, “An analysis of deep neural network models for practical applications,” arXiv, no. 1605.07678, 2016. [196] T.-J. Yang, Y.-H. Chen, and V. Sze, “Designing energy-efficient convolutional neural networks using energy-aware pruning,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6071–6079, 2016. [197] T.-J. Yang et al., “NetAdapt: Platform-aware neural network adaptation for mobile applications,” in European Conference on Computer Vision (ECCV), September 2018. [198] V. Sze et al., “Efficient processing of deep neural networks: A tutorial and survey,” Proceedings of the IEEE, vol. 105, no. 12, pp. 2295–2329, 2017. [199] “Odroid XU4,” https://wiki.odroid.com/odroid-xu4/odroid-xu4. [200] J. T. Springenberg et al., “Striving for simplicity: The All Convolutional Net,” arXiv, no. 1412.6806, 2014. [201] D. Velasco-Montero and J. Fernández-Berni and R. Carmona-Galán and Á. Rodríguez-Vázquez, “GitHub – PreVIous,” https://github.com/DVM000/PreVIous.git. BIBLIOGRAPHY [202] K. He et al., “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in International Conference on Computer Vision (ICCV), 2015. [203] J. Redmon and A. Farhadi, “YOLO9000: better, faster, stronger,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. [204] X. Dai et al., “ChamNet: Towards efficient network design through platform-aware model adaptation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. [205] S. Han et al., “Design automation for efficient deep learning computing,” arXiv, no. 1904.10616, 2019. [206] Z. Lu et al., “Augur: Modeling the resource requirements of ConvNets on mobile devices,” IEEE Transactions on Mobile Computing, 2019. [207] E. Cai et al., “NeuralPower: Predict and deploy energy-efficient convolutional neural networks,” arXiv, no. 1710.05420, 2017. [208] A. E. Eshratifar and M. Pedram, “Energy and performance efficient computation offloading for deep neural networks in a mobile cloud computing environment,” in Proceedings of the 2018 on Great Lakes Symposium on VLSI, 2018, p. 111–116. [209] C. F. Rodrigues, G. D. Riley, and M. Luján, “Fine-grained energy profiling for deep convolutional neural networks on the Jetson TX1,” arXiv, no. 1803.11151, 2018. [210] C. F. Rodrigues, G. Riley, and M. Luján, “SyNERGY: An energy measurement and prediction framework for convolutional neural networks on jetson tx1,” in International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA), October 2018. [211] C. F. Rodrigues, G. D. Riley, and M. Luján, “Energy predictive models for convolutional neural networks on mobile platforms,” arXiv, no. 2004.05137, 2020. [212] H. Bouzidi et al., “Performance prediction for convolutional neural networks on edge GPUs,” in Proceedings of the 18th ACM International Conference on Computing Frontiers, ser. CF ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 54–62. [213] Y. Wu and V. Sze, “Accelergy: An architecture-level energy estimation methodology for accelerator designs,” in Int. Conference On Computer Aided Design, 2019. [214] Y. Chen et al., “Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks,” IEEE Journal of Solid-State Circuits, vol. 52, no. 1, pp. 127–138, January 2017. [215] L. Heim et al., “Measuring what really matters: Optimizing neural networks for TinyML,” arXiv, no. 2104.10645, 2021. [216] T.-J. Yang, Y.-L. Liao, and V. Sze, “NetAdaptV2: Efficient neural architecture search with fast super-network training and architecture optimization,” in Conference on Computer Vision and Pattern Recognition (CVPR), June 2021. [217] D. Stamoulis et al., “HyperPower: Power- and memory-constrained hyper-parameter optimization for neural networks,” in 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017, pp. 19–24. [218] E. Gianniti and L. Zhang, “Performance prediction of gpu-based deep learning applications,” in International Symposium on Computer Architecture and High Performance Computing SBAC-PAD, September 2018. [219] H. Qi, E. R. Sparks, and A. Talwalkar, “Paleo: A performance model for deep neural networks,” in International Conference on Learning Representations (ICLR), 2017. [220] K. Singh, M. Bhadauria, and S. A. McKee, “Real time power estimation and thread scheduling via performance counters,” SIGARCH Comput. Archit. News, vol. 37, no. 2, p. 46–55, 2009. [221] G. Contreras and M. Martonosi, “Power prediction for intel xscale/spl reg/ processors using performance monitoring unit events,” in ISLPED ’05. Proceedings of the 2005 International Symposium on Low Power Electronics and Design, 2005., 2005, pp. 221–226. [222] “Program power profiling based on phase behaviors,” Sustainable Computing: Informatics and Systems, vol. 19, pp. 341–350, 2018. [223] D. Velasco-Montero et al., “PreVIous: A methodology for prediction of visual inference performance on IoT devices,” IEEE Internet of Things Journal, vol. 7, no. 10, pp. 9227–9240, 2020. [224] A. Yazdanbakhsh et al., “An evaluation of edge TPU accelerators for convolutional neural networks,” arXiv, no. 2102.10423, 2021. [225] S. S. Ogden and T. Guo, “Characterizing the deep neural networks inference performance of mobile applications,” arXiv, no. 1909.04783, 2019. 119 [226] Y. Kang et al., “Neurosurgeon: Collaborative intelligence between the cloud and mobile edge,” in Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, ser. ASPLOS ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 615–629. [Online]. Available: https://doi.org/10.1145/3037697.3037698 [227] S. Zhang et al., “Towards real-time cooperative deep inference over the cloud and edge end devices,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 4, no. 2, June 2020. [228] A. E. Eshratifar and M. Pedram, “Energy and performance efficient computation offloading for deep neural networks in a mobile cloud computing environment,” in Proceedings of the 2018 on Great Lakes Symposium on VLSI, ser. GLSVLSI ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 111–116. [229] L. Zeng et al., “Boomerang: On-demand cooperative deep neural network inference for edge intelligence on the industrial Internet of Things,” IEEE Network, vol. 33, no. 5, pp. 96–103, 2019. [230] E. Li, Z. Zhou, and X. Chen, “Edge intelligence: On-demand deep learning model co-inference with device-edge synergy,” arXiv, no. 1806.07840, 2018. [231] I. Prakash et al., “SmartSplit: Latency-energy-memory optimisation for CNN splitting on smartphone environment,” arXiv, no. 2111.01077, 2021. [232] F. Martins Campos de Oliveira and E. Borin, “Partitioning convolutional neural networks to maximize the inference rate on constrained IoT devices,” Future Internet, vol. 11, no. 10, 2019. [Online]. Available: https://www.mdpi.com/1999-5903/11/10/209 [233] C. Hu et al., “Dynamic adaptive DNN surgery for inference acceleration on the edge,” in IEEE INFOCOM 2019 - IEEE Conference on Computer Communications, 2019, pp. 1423–1431. [234] R. Hadidi et al., “Toward collaborative inferencing of deep neural networks on Internet-of-Things devices,” IEEE Internet of Things Journal, vol. 7, no. 6, pp. 4950–4960, 2020. [235] X. Wang et al., “In-edge AI: intelligentizing mobile edge computing, caching and communication by federated learning,” arXiv, no. 1809.07857, 2018. [236] Z. Zhao, K. M. Barijough, and A. Gerstlauer, “DeepThings: Distributed adaptive deep learning inference on resourceconstrained IoT edge clusters,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 37, no. 11, pp. 2348–2359, 2018. [237] The Apache Software Foundation, “Apache TVM,” https://tvm.apache.org/, 2020. [238] L. Zheng et al., “Ansor: Generating high-performance tensor programs for deep learning,” in Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation. USA: USENIX Association, 2020. [239] K. He et al., “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. [240] M. Sandler et al., “MobileNetV2: Inverted residuals and linear bottlenecks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510–4520. [241] B. W. Kernighan and S. Lin, “An efficient heuristic procedure for partitioning graphs,” The Bell System Technical Journal, vol. 49, no. 2, pp. 291–307, 1970. [242] A. Grama, V. Kumar, and A. Sameh, “Parallel hierarchical solvers and preconditioners for boundary element methods,” SIAM Journal on Scientific Computing, vol. 20, no. 1, pp. 337–358, 1998. [243] T. Verbelen et al., “Graph partitioning algorithms for optimizing software deployment in mobile cloud computing,” Future Generation Computer Systems, vol. 29, no. 2, pp. 451–459, 2013, special section: Recent advances in e-Science. [244] L. Zheng et al., “Introducing TVM Auto-scheduler (a.k.a. Ansor),” https://tvm.apache.org/2021/03/03/intro-auto-scheduler, March 2021. [245] The Apache Software Foundation, “TVM. User Tutorial. Compiling and Optimizing a Model with TVMC,” https://tvm. apache.org/docs/tutorial/tvmc_command_line_driver.html, 2020. [246] ——, “TVM Python API. tvm.relay.testing,” https://tvm.apache.org/docs/reference/api/python/relay/testing.html, 2020. [247] Y. Ding, “Multi-stream Execution in Meta VM,” https://www.tvmcon.org/wp-content/uploads/2022/01/16-15_ 40LT-Yaoyao_Ding-Multi-stream_Support_for_Virtual_Machine_Executor_in_Meta.pdf, 2022. [248] N. Corporation, NVIDIA Jetson TX2 NX System-on-Module, 2022. [249] Nvidia Corporation, “Tegra Linux Driver Package Release Notes,” http://developer2.download.nvidia.com/embedded/L4T/ r27_Release_v1.0/Docs/Tegra_Linux_Driver_Package_Release_Notes_R27.1.pdf.