Optimizing deep neural networks for resource-limited embedded systems

id192731   OPTIMIZING DEEP NEURAL NETWORKS FOR RESOURCE-LIMITED EMBEDDED SYSTEMS ESTEBAN GATEIN Thesis supervisor MARINAZAPATERSANCHO(HEIG-VDSchoolofEngineeringandManagement) Tutor:JORDIDELGADOPIN(DepartmentofComputerScience) Degree Bachelor'sDegreeinArtificialIntelligence Bachelor's thesis Facultat d'Informàtica de Barcelona (FIB) Universitat Politècnica de Catalunya (UPC) - BarcelonaTech 22/01/2025  Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Abstract In recent years, applications based on artificial intelligence (AI) techniques have become increasingly common, to the point where deep neural networks (DNNs) have been applied across almost any field, providing solutions to a wide range of problems. Simultaneously, embedded systems have evolved to meet the growing demand for AI integration. While most conventional microprocessors have remained unable to process DNNs efficiently, many researchers have believed that the future of embedded technology lies in increasing the autonomy of edge devices, enabling them to run AI-based applications. This bachelor thesis focused on the implementation of DNNs for autonomous drone navigation, deploying these networks on an ultra-low-power microprocessor specifically designed for deep learning applications. Furthermore, it explored cutting-edge model compression techniques to optimize the performance of the DNNs and investigated the limitations of this emerging technology in the context of embedded systems. The research examined various approaches, including model quantization, knowledge distillation, the implementation of early exits, and edge-to-cloud strategies, aiming to balance computational efficiency and accuracy while respecting the constraints of ultra-lowpower hardware. 2 Resum En els darrers anys, les aplicacions basades en t` ecniques d’intel·lig` encia artificial (IA) s’han tornat cada vegada m´ es habituals, fins al punt que les xarxes neuronals profundes s’han aplicat a gaireb´ e qualsevol camp, proporcionant solucions a una ` amplia gamma de problemes. Al mateix temps, els sistemes integrats han evolucionat per satisfer la creixent demanda d’integraci´ o de la IA. Tot i que la majoria de microprocessadors convencionals continuen sent ineficients per processar xarxes neuronals, molts investigadors creuen que el futur de la tecnologia incrustada passa per augmentar l’autonomia dels dispositius edge, permetent-los executar aplicacions basades en IA. Aquest treball de fi de grau s’ha centrat en la implementaci´ o de xarxes neuronals per a la navegaci´ o aut` onoma de drons, desplegant aquestes xarxes en un microprocessador ultra eficient dissenyat espec´ ıficament per a aplicacions d’aprenentatge profund. A m´ es, s’han explorat t` ecniques innovadores de compressi´ o de models per optimitzar el rendiment de les xarxes neuronals i s’han investigat les limitacions d’aquesta tecnologia emergent en el context dels sistemes incrustats. La investigaci´ o ha examinat diversos enfocaments, incloent-hi la quantitzaci´ o de models, la destil·laci´ o de coneixement, la implementaci´ o de sortides anticipades i estrat` egies edge-to-cloud, amb l’objectiu d’equilibrar l’efici` encia computacional i la precisi´ o respectant les limitacions del maquinari ultra eficient. 3 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Resumen En los ´ ultimos a˜ nos, las aplicaciones basadas en t´ ecnicas de inteligencia artificial (IA) se han vuelto cada vez m´ as comunes, hasta el punto de que las redes neuronales profundas se han aplicado en casi cualquier campo, proporcionando soluciones a una amplia gama de problemas. Al mismo tiempo, los sistemas embebidos han evolucionado para satisfacer la creciente demanda de integraci´ on de la IA. Aunque la mayor´ ıa de los microprocesadores convencionales siguen siendo ineficientes para procesar redes neuronales, muchos investigadores creen que el futuro de la tecnolog´ ıa embebida radica en aumentar la autonom´ ıa de los dispositivos edge, permiti´ endoles ejecutar aplicaciones basadas en IA. Este trabajo de fin de grado se ha centrado en la implementaci´ on de redes neuronales para la navegaci´ on aut´ onoma de drones, desplegando estas redes en un microprocesador de ultra bajo consumo espec´ ıficamente dise˜ nado para aplicaciones de aprendizaje profundo. Adem´ as, se han explorado t´ ecnicas de vanguardia de compresi´ on de modelos para optimizar el rendimiento de las redes neuronales y se han investigado las limitaciones de esta tecnolog´ ıa emergente en el contexto de los sistemas embebidos. La investigaci´ on ha examinado varios enfoques, incluidos la cuantizaci´ on de modelos, la destilaci´ on de conocimiento, la implementaci´ on de salidas tempranas y estrategias edgeto-cloud, con el objetivo de equilibrar la eficiencia computacional y la precisi´ on respetando las limitaciones del hardware de ultra bajo consumo. 4 CONTENTS Contents 1 Introduction 7 2 Context and State of the Art 9 2.1 Low-Power Devices for Edge Computing . . . . . . . . . . . . . . . . . . . . 9 2.2 Edge-to-Cloud Strategies for Efficiency . . . . . . . . . . . . . . . . . . . . . 10 2.3 Multi-Task Learning in Convolutional Neural Networks . . . . . . . . . . . . . 11 2.4 Optimization for Low-Power Devices: Model Compression . . . . . . . . . . . 12 2.5 Quantization in Convolutional Neural Networks . . . . . . . . . . . . . . . . . 12 2.5.1 Post-Training Quantization (PTQ): . . . . . . . . . . . . . . . . . . . . 13 2.5.2 Quantization-Aware Training (QAT): . . . . . . . . . . . . . . . . . . 13 2.6 Early-Exit Strategies in DNNs for Edge Computing . . . . . . . . . . . . . . . 14 2.7 Knowledge Distillation in DNNs . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.8 Established Work and Project Context . . . . . . . . . . . . . . . . . . . . . . 16 3 Data Acquisition 18 3.1 CollisionDataset ................................. 18 3.2 UdacityDataset.................................. 18 3.3 Fire and Smoke Detection Dataset (FASDD) . . . . . . . . . . . . . . . . . . . 20 4 Training for Resource-Limited Platforms 21 4.1 Quantization-Aware Training (QAT) . . . . . . . . . . . . . . . . . . . . . . . 21 4.1.1 QAT for the DroNet model . . . . . . . . . . . . . . . . . . . . . . . . 21 4.1.2 Design of the adapted model . . . . . . . . . . . . . . . . . . . . . . . 23 4.2 Smoke and Fire Detection Neural Network . . . . . . . . . . . . . . . . . . . . 26 4.3 Testing on the GAP9 Platform: Limitations and Deployment Process . . . . . . 27 4.3.1 Setup of the GAP9 Platform . . . . . . . . . . . . . . . . . . . . . . . 27 4.3.2 NNTool and Autotiler for DNN Deployment . . . . . . . . . . . . . . 27 5 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems 4.3.3 Challenges During Deployment . . . . . . . . . . . . . . . . . . . . . 28 4.3.4 Transition to PyTorch and ONNX . . . . . . . . . . . . . . . . . . . . 29 4.3.5 Adaptations for GAP9 Deployment . . . . . . . . . . . . . . . . . . . 29 5 Reducing DNNs for Resource-Limited Platforms 32 5.1 Early Exit Strategies Investigation . . . . . . . . . . . . . . . . . . . . . . . . 32 5.2 Multi-Objective DNNs with Early Exits . . . . . . . . . . . . . . . . . . . . . 33 5.2.1 Model with Early Exit for Fire and Smoke Detection . . . . . . . . . . 33 5.2.2 Model with Early Exits for Collision Prediction and Steering Angle . . 35 5.3 Knowledge Distillation for DNNs . . . . . . . . . . . . . . . . . . . . . . . . 36 5.4 Compression of the Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6 Demonstrator 40 6.1 ModelGeneralization............................... 40 6.1.1 ContextVectors.............................. 40 6.1.2 Model Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . 41 6.2 Deployment with the Edge-to-Cloud setup . . . . . . . . . . . . . . . . . . . . 42 6.2.1 Server Design and Implementation . . . . . . . . . . . . . . . . . . . . 42 6.2.2 DroneBandit on GVSoC . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.2.3 Deployment of a Knowledge Distilled Model . . . . . . . . . . . . . . 47 7 Evaluation and Results 48 8 Sustainability Analysis and Ethical Implications 53 9 Conclusions 55 Appendices 56 6 1 Introduction 1 Introduction The integration of artificial intelligence into embedded systems has brought important advancements to many applications, making solutions more efficient and smarter. However, working with resource-constrained embedded systems makes it very difficult to deploy complex deep neural networks. This thesis explores methods for overcoming these challenges in optimizing DNNs for ultra-low-power devices, with a focus on autonomous navigation for drones. The project serves as a continuation of the DroneBandit project, which uses the GAP8 processor integrated into a nanodrone for real-time neural inference, presenting at the same time an algorithm to define a dynamic edge-to-cloud strategy for DNNs, with the objective of iteratively choosing the optimal cutting point of the model. Based on this project, this research aims to explore the capabilities of the GAP9 processor to implement a multi-task learning model capable of predicting collisions, steering angles, and fire or smoke detection. The project has the objective of compressing such DNNs to make the deployment feasible while maintaining a balance between optimization and performance loss. Additionally, the DNNs have to be designed with the intention of working in a real-world use case defined by a nanodrone employing an unstable edge-to-cloud setup, with the assumption that the GAP9 can be integrated into a drone similarly to the GAP8. The optimization techniques considered and experimented with have the objective of solving the problem of the connection by the implementation of an alternative DNN working only on the edge device. The GAP9 processor is to be pushed to its limits to understand its limitations and its capabilities, and the deployment of DNNs on this chip has as a goal demonstrating that compression techniques are a solution for resource-constrained embedded systems. From experimenting with the GAP8 processor to the deployment of models trained with cuttingedge compression techniques, this bachelor thesis will answer the following questions: • How can quantization and early-exit techniques improve the efficiency of neural networks for deployment on the GAP9 microprocessor? • How effectively can a compressed neural network predict collision detection, steering angles, and fire/smoke detection on ultra-low-power embedded systems? • What is the maximum compression ratio that can be achieved for neural networks on GAP9 without compromising task-specific accuracy? By the integration and the support of a research group at the Reconfigurable & Embedded Digital Systems (REDS) institute at the HEIG-VD, this project is carried out in a research framework that combines theoretical exploration with practical implementation. This document starts, therefore, with a brief exposition and definition of the concepts used during the research project. Then, the methodology is defined by exposing the design and decision-making 7 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems processes, followed by sections dedicated to the practical demonstration and the evaluation of results. The results details, including the training processes of neural network models, are available in the public GitHub repository: Embedded & Optimized DNNs. This repository provides access to the codebase and scripts used to replicate the results discussed in this thesis, showing transparency and reproducibility of the work. The practical outcome of the project includes the possibility of using the GAP9 processor by the REDS institute with the limitation of the processors already known, with the solutions that can be applied to DNNs to make their deployments feasible. Nevertheless, as this project is a continuation of the DroneBandit project, the more important outcome is the design of the tools that make the use of the algorithm less complicated. While the range of fields in which this algorithm can be used to design innovative solutions is vast, its implementation can sometimes be challenging due to resource limitations. Therefore, the solutions proposed in this work must be understood as a way of minimizing the impact of these challenges by making effective deployments more convenient. 8 2 Context and State of the Art During the knowledge distillation process, the teacher model is used to generate soft targets or probability distributions over the output classes. These soft targets encode additional information about the relationships between classes, which the student model learns during training. This auxiliary training signal allows the student model to achieve higher accuracy than it would using traditional training methods based on ground-truth labels. By using this approach, it is possible to train models that meet the strict memory and computational requirements of edge devices like the GAP9. Another critical advantage of knowledge distillation in edge scenarios is its ability to reduce the student model to specific tasks or environments. For example, the distilled model can be fine-tuned to prioritize the most relevant features for the deployment context, further reducing computational complexity without sacrificing performance. In practice, the success of knowledge distillation relies on effectively combining the loss functions associated with the soft targets generated by the teacher model and the hard labels. The soft targets are typically represented as a probability distribution over the output classes, smoothed by a temperature parameter Tto amplify the relative differences between class probabilities. The Kullback-Leibler (KL) divergence is commonly used as the loss function for aligning the student model’s output distribution qwith the teacher’s softened output distribution p, defined as: LKD =T2KL(p||q) = T2∑ i pilog pi qi , where piand qidenote the probabilities of class ipredicted by the teacher and student models, respectively, and T2scales the loss for the effect of temperature T. This loss is combined with the standard cross-entropy loss LCE calculated using the labels yto guide the student model’s learning. The overall objective function is a weighted sum of the two components: Ltotal =αLCE +(1−α)LKD, where α∈[0,1]is a hyperparameter that controls the balance between the hard label supervision and the knowledge distillation from the teacher. By tuning αand T, the student model can learn to imitate the teacher’s behavior while also being restricted to the true labels, resulting in a model that achieves an optimal balance between accuracy, model size, and computational efficiency. 15 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Figure 2: Crazyflie drone compared to a 0.01 C coin. [Palossi et al., 2019] 2.8 Established Work and Project Context It was before starting this thesis work and joining the research collective that the main project on which this work is based was already in motion. The last attempt was directed to the formation of a framework for using an unmanned aerial vehicle (UAV) to apply real-time neural image processing. It was more about developing a strategy for getting information from the edge to the cloud that would make the drone avoid these barriers dynamically and in real time. This task was to satisfy various constraints of limited computational resources and energy availability, which are usually attributed to nanodrones. The project uncovered the fact that the lightweight convolutional neural networks based on the use of the GAP8 processor, an SoC system, had been deployed. The processor is a cuttingedge piece of hardware specially developed to put low-power AI applications into action. The chip was integrated into the Crazyflie drone, which is a modular and versatile nanodrone built by Bitcraze and to be used with the AI deck, the carry-on module that is specifically made to improve the drone’s processing speed in machine learning tasks. The size of the drone can be appreciated in Figure 2. This achievement revealed the possibility of running a neural network inference on a resource-constrained nanodrone, which is a tough project because of the drone’s strict size, weight, and power limitations. Related to the technical problems with the use of a CNN-based application performed on a nanodrone, problems with the edge-to-cloud inference process were also dealt with. All of this was made possible by the development of DroneBandit, which is an online decision-making algorithm operating in the area of multi-armed contextual bandits. The primary focus of DroneBandit, as presented in [Chacun et al., 2024], is to find the most optimal part to cut so as the edgeto-cloud strategy can be realized most efficiently during every inference iteration. To do this, the algorithm dynamically assumes the most suitable distribution of the computational tasks 16 2 Context and State of the Art between the drone (edge) and the server cloud in the process by estimating the data amount and time or latency of the connection. Delays caused by these factors are minimized. The predictive modeling by DroneBandit estimates inference time on the edge and the cloud, enabling it to dynamically adjust the strategy in real time, based on each specific context and adapting to the WiFi conditions. This not only ensures latency reduction but also ensures that the limited drone resources are efficiently utilized to provide smoother and more reliable obstacle avoidance during flight. This paper has pursued the integration of edge computing, cloud processing, and advanced decision algorithms in order to establish a solid basis for future research on collaborative inference strategies in resource-constrained environments. Given the description of the established work, this bachelor thesis will continue the DroneBandit project by proposing a more complex use case fitting to the restrictions of the edge-to-cloud framework and propose solutions to overcome the encountered limitations. 17 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems 3 Data Acquisition The first step of the project has been obtaining the data for the models. Three different datasets have been used, all of them containing thousands of images with their corresponding labels for each task. In this section a description of each dataset is given, as well as the process followed to obtain them. Figure 3: Image examples of the collision dataset. [Loquercio et al., 2018] 3.1 Collision Dataset The first dataset collected is the one created by the DroNet project, the collision dataset, which has also been used for the DroneBandit project. The data is composed of different folders that contain images of videos taken by a camera mounted on a bike filming through a city. The data has been labeled to indicate whether there is an imminent collision or not in each frame and has been structured to fit the training pipeline of DroNet. A data sample with its corresponding labels can be observed in Figure 3. The data can be downloaded through the webpage of the project, with the structure needed to run the model, which can be found in [Loquercio et al., 2018]. The quality of the data has not been analyzed, as satisfactory results are reached in both projects using this data. 3.2 Udacity Dataset The second dataset collected is the Udacity Dataset, which contains images and sensor data from an autonomous car, as it can be observed in Figure 4. Concretely, the images are collected from three different cameras disposed of in the center and on both sides of the car and files containing information about the control of the car, including the angle of the steering wheel. The instructions to obtain this publicly available data are described in the DroNet project. 18 3 Data Acquisition Figure 4: Sample images from the udacity dataset. [Loquercio et al., 2018] The first step has been to download the data files, which are available in a torrent format, the files to download can be found as explained in [Loquercio et al., 2018]. Then, a torrent client has been used to download the data itself; in this case, open-source software has been used, qBittorrent. After downloading the data, the format obtained is compressed images in the Robot Operating System (ROS) bag2format. These bags contain the sequences of images and data extracted directly from the car and need to be preprocessed in order to be able to feed the training pipeline of a model. In the DroNet project, it is recommended to use a concrete repository, which can be found in [Rwightman, 2019]. After trying to install one version from ROS in the operating system that was used, it has been observed that the container that has to be used according to Rwightman was not able to work. A virtual machine has been installed running Ubuntu 18.04, as the container given by the repository and the ROS version needed have been tested on this version of Ubuntu. Even having the compatible software, the containers needed to be updated, concretely in the Dockerfile3where the commands to download the requirements were outdated. Also, the structure of the directories containing the ROS bags has been changed, as the container is sensitive to the number of files in each directory. Then, executing the container produced, for each ROS bag, the three sets of images of the cameras of the car and the files containing the information for the sensors. Additionally, to obtain the useful data for training a model, another script provided by DroNet has to be executed, which maps data from the car sensors and matches the angle of the steering wheel with the images of the central camera, using the timestamps of the data. Finally, the directory structures of the data have been organized in three main sets: training, validation, and test. In each one of them, it can be seen that different directories are contained, corresponding to different video sequences and referred to from now on as experiments, matching with the structure of the collision dataset. Therefore, both datasets have been merged without any further analysis, for the same reason as in the collision dataset. 2More information can be found on http://wiki.ros.org/Bags. 3Dockerfiles are files used by Docker, an open-source platform used to deploy applications in containers, that creates and initializes the environment. 19 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems 3.3 Fire and Smoke Detection Dataset (FASDD) The fire and smoke detection dataset (FASDD) contains thousands of images of fire, smoke, fire and smoke, or neither of them in different setups, as shown in the sample in Figure 5. This openaccess dataset is constructed to be able to train very accurate fire detection models and contains labels constructed for concrete models used in the state of the art computer vision context, such as YOLOv9. No deep analysis has been realized to measure the quality of the data, as the authors of the dataset provide all of the needed information with some benchmarks testing the data and making some experiments with it in [Wang et al., 2024]. Even so, the data structure has been reorganized in order to only keep the data from the classes of each image and match the structure of the other two datasets, so generating simulated experiments with balanced classes by randomly choosing the images from the different classes. Figure 5: Sample images from the FASDD. [Wang et al., 2024] 20 4 Training for Resource-Limited Platforms 4 Training for Resource-Limited Platforms GAP8 and GAP9 are examples of resource-constrained platforms designed to work under processing constrictions, limited memory, and low energy consumption. These constraints trigger new approaches for adaptation of a model, such as quantization, which makes low-bit format types of model parameters without making much difference to accuracy. The purpose is to create lightweight and efficient models that meet the edge-to-cloud requirements while achieving high-quality performance. This section first elucidates the adaptation of DroNet to quantization-aware training for achieving effective inference using integers. A comprehensive overview is made on the steps of designing and adapting the model for the latest frameworks’ and the specific needs of resourceconstrained environments. It also outlines the progression towards developing a low-weight neural smoke detection network that focuses on edge devices with less architecture and efficient implementation. These challenges pave the journey for model deployment on the GAP9 platform, in addition to putting into perspective some limitations with GAP9 as well as solutions put in place. The outcome of such an experiment offers an insight on how the design of neural networks optimized for resource-constrained platforms is produced. In the case of this project, even if it is presented as a continuation of the DroneBandit project deployed on GAP8, only the GAP9 processor will be considered. The reason for this decision is based on the upgrades given by the GAP9 processor compared to its previous version, and starting a project with this new processor unlocks the possibility of transitioning the continuing ongoing projects working with the GAP8 to the GAP9 processor in order to take advantage of the new features of the processor. The difference between processors can be consulted in [Sizova, 2024]. 4.1 Quantization-Aware Training (QAT) The initial part of the project has been to investigate the possibility of training the model that was deployed on the GAP8 using QAT in order to avoid the loss of precision induced by the PTQ. Additionally, with the objective of improving the use case given by the project, which was the collision avoidance, one of the main objectives has been adding a task to be solved by the drone for a gain of autonomy. 4.1.1 QAT for the DroNet model The first model considered in this work is the one designed in the DroNet project, a residual neural network formed by blocks of convolutional layers that outputs two different results: the 21 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems probability of collision of the drone with an obstacle and the steering angle needed to avoid it, trained with the datasets previously introduced in 3.1 and 3.2. Deploying a model in a quantized form is a restriction given by the GAP8, as it is impossible for the processor to handle floating point operations and numbers. It is important to remark that this is not a restriction for the GAP9, as it can work with floating point operations with 32-bit precision, but this option limits the possible size of the model and won’t be aligned with the objective of maximizing the compressions of the models to the maximum; therefore, this alternative is not considered in this work. Given the possibility of improving the results observed in the DroNet project and the fact that the model could be deployed resolving more than one task in the GAP8, the first step has been about preparing the original model for a training pipeline in QAT. As the DroNet project contains the requirements of the environment needed to evaluate a pretrained version of the model or to train the model again. The first problem that has been encountered with the code of the project is about the version compatibility of the deep learning library used for the training of the models. In fact, the TensorFlow version used for the project is 1.5.0, released in 2018, which is not able to work with the QAT tools from TensorFlow, as they are way more recent. Other problems that have been observed are the lack of possibility of making inferences with a single image, which can be useful to observe the punctual results and evaluate models on a given set of images; this has been solved by creating a script that would read an image, load a pretrained model with the required architecture, and run the inference; and the impossibility to modify the model easily for the introduction of the alternative net that has to be able to run on the edge exclusively. Given these limitations, the decision has been taken to not use the model as is but to recreate a similar version using the latest version of TensorFlow to train the model via QAT and include the solution for the alternative network. Even though the use of the original code of the project is not considered, the model has been run for inference, so a brief description of the architecture is provided, which can also be observed in Figure 6. The model is composed of three residual blocks, each one of them formed by two convolutional layers. Before the first convolutional block, the first convolutional layer can be found, and the two objectives of the net are preceded by a single fully connected layer for each objective. 22 4 Training for Resource-Limited Platforms Figure 6: Architecture of the DroNet model. [Loquercio et al., 2018] 4.1.2 Design of the adapted model With the objective of running an alternative network exclusively on the edge network for the moments in which the connection with the server is malfunctioning, a solution has been proposed. This solution combines adapting the DroNet model to the new version of the framework to train the model with QAT and adding the previously mentioned alternative net. After an initial approach not considered in this work exposed in Appendix A, the solution evolved towards including another output to the DroNet model. The idea relies on having a model capable of predicting the probability of collision, the steering angle to avoid it, and another classifier, with the objective of predicting whether the zone is safe to land in or not. This last classifier has to give the output on the edge, so its branch has to be defined after the first layers, and the algorithm deciding where the cutting point is for the edge-to-cloud strategy has to be restricted for making decisions considering only the cutting points available after this output. The first problem that has been considered for a task like this is the data, as no public datasets were available with landing zone images for drones or UAV camera images with labels indicating if it could land or not, avoiding any danger. To overcome this, FASDD has been used, given that the images contain four different classes (as exposed in Section 3.3), this dataset is considered as an alternative even more complex than a binary classifier. Additionally, the possibility of predicting whether there is smoke, fire, both, or neither with an interior nanodrone is a use case useful enough to be considered. The model designed is similar to the one presented in the DroNet project, removing the residual connections as it is a restriction from DroneBandit, the algorithm used for the edge-to-cloud strategy. It is composed of three convolutional layers, followed by the branch with the output for the fire and smoke detection and a branch with four more convolutions and the other two outputs. A schema of the model can be observed in Figure 7, however, note that some convolution layers include batch normalization layers, and activations are not represented. 23 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Input Convolution Pooling Convolution Convolution Pooling Fully Connected FASDD Convolution Convolution Convolution Convolution Fully Connected Collision Fully Connected Steering Angle Figure 7: Architecture of the adapted model. The main challenge to train a model like this one is about the labels, as for each image, the network outputs three labels (one for each task) but is trained with only one label per image at the time. At first, it was expected to create a lot of problems during the training, but it has been observed that this also was a problem overcome in the DroNet project by ignoring the loss of the tasks when the label was not available. Even if the theoretical solution was available, the customized loss functions had to be designed from scratch, as well as the data loaders, due to the deprecation of the code and the impossibility to train the model by using masks in the training process of a TensorFlow model. To make everything work, the data loaders have had to be redesigned to read and preprocess the images from the different experiments. The preprocessing is only about reading the images in grayscale, resizing the images to a 200x200 pixel definition, as the camera of the drone considered has this resolution, and scaling the values between 0 and 1. The missing labels have been replaced by NaN (Not a Number) values in order to have the same structure for each image to feed the trainer of the model. In the case of the loss, it has been observed that training the model with NaN values was introducing errors and producing NaN values back, which would make the loss impossible to compute. Because of that reason, these values have been replaced by a numerical symbolic value that could be detected and ignored by the personalized loss functions, in this case, 999. Three different loss functions have been defined: one for the binary classifier for the collision task computing the binary cross-entropy, one for the multi-class classifier for the FASDD, and one for the regression task corresponding to the steering angle and computing the mean squared error (MSE). Each one of them has been defined using the TensorFlow operators, as they have to work on tensors. Find each loss function defined as follows: 24 4 Training for Resource-Limited Platforms Cycles Time (ms) Energy (µJ) Inferences per sec. Inference 1 6255513 16.90679189 0.1892103732 59.14782688 Inference 2 6253756 16.90204324 0.1891572293 59.16444454 Inference 3 6254906 16.90515135 0.1891920133 59.15356682 Inference 4 6254092 16.90295135 0.1891673923 59.16126594 Inference 5 6254313 16.90354865 0.1891740768 59.15917544 Inference 6 6253601 16.90162432 0.189152541 59.16591097 Inference 7 6254643 16.90444054 0.1891840583 59.15605415 Inference 8 6253591 16.9015973 0.1891522385 59.16600558 Inference 9 6253762 16.90205946 0.1891574108 59.16438777 Inference 10 6253501 16.90135405 0.1891495163 59.16685709 Average 6254167.8 16.90315622 0.189169685 59.16054891 Table 1: Time and energy consumption on single inference for the FASDD task. the computer is limited, but thanks to the evaluation kit, some data about the execution can be retrieved. The results are shown in Table 1, where it can be seen that the number of cycles is not always the same, even if the number of operations to compute is, for realizing this inference, exactly 33913280 operations are realized. The time is calculated by dividing the number of cycles by the frequency of the board, in this case, 370 MHz, and the energy is calculated by multiplying the number of operations by the efficiency and by the time, and the efficiency of the board is known to be about 3.3×10−13 W/operation, according to [Sizova, 2024]. Finally, it is important to remark on the last column of Table 1, where the inferences per second that could be reached are shown; almost 60 inferences per second is an acceptable value for the use case of a nanodrone flying in an interior setup. 31 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems 5 Reducing DNNs for Resource-Limited Platforms 5.1 Early Exit Strategies Investigation Inspired by the way the auxiliary network has been designed as an exit that will always give an output in the edge device, early exits are considered in this work to lower the workload of the data transfer between the edge device and the server. These early exits have the objective to resolve one or more than one task that is already resolved by the whole network. The output of these branches has to be given inside the edge device, so the cutting point will have to be defined after these exits. The main idea is to have an output on which the confidence of the prediction can be estimated, given a threshold of confidence. If the predictions given by the early exit are confident enough, the model has to stop the execution there, so there won’t be data transfer for that iteration; otherwise, the model has to keep with the execution and send the intermediate outputs of the optimal cutting point for the iteration to the server in order to end the execution. For the confidence measurement, it is easy to define for the classifiers, as the probability of predicted classes will indicate how much the model believes an image belongs to one of the classes. It has to be noted that the outputs from the models on the GAP9 processors are in a range from -128 to 127, and that the softmax function is sensitive to the scale of the outputs, so the results will always show high confidence in the outputs. To overcome this, the outputs of the classifiers have to be normalized, which is not a hard task as the range of the values is known. Once the outputs are normalized, a softmax function can be applied (not usually present in the model as PyTorch applies it internally in the loss functions), and a decision can be taken. Confidence measure cannot rely on class probabilities as in the case of classifiers for regression tasks. Instead, an approach has been designed to estimate the confidence of the early exit predictions. A Lasso regression model is trained to predict the error of the main regression model based on the intermediate outputs of the early exit. This error prediction is a proxy for confidence, in that the lower the predicted error, the higher the confidence in the early exit’s prediction. In this way, the system sets a threshold on acceptable error and decides whether the early exit is confident enough to stop execution or needs to continue the processing of the input further through the main network. The training and application of the Lasso regression model will be elaborated in later sections. A few models were developed to test the early exit strategy, using a number of different configurations and strategies: some by using different numbers of early exits, others by varying their placement within the network, and still others by testing alternate methods for confidence estimation. Besides, multiple confidence thresholds have been tried to balance the trade-off between the accuracy of predictions and reduction in data transfer and computational workloads. In fact, all these strategies are evaluated against their impact on performance metrics such as 32 5 Reducing DNNs for Resource-Limited Platforms overall accuracy, latency, and energy consumption. The results of those experiments are included later in this work and present several views on how early exits could be implemented and optimized. 5.2 Multi-Objective DNNs with Early Exits Three different DNNs have been developed with early exits, one of them already presented in Section 4.2, working with an early exit to evaluate the impact of having an early output like the one presented in Section 4.1.2 for the GAP9 microprocessor. Two other models have been developed with early exits and integrating one or more than one strategy with the objective of reducing the computational charge on the chip. For the model resolving the FASDD task, the strategy is not implemented nor considered in this section, as it is not resolving the primary tasks that have to be solved to make a drone autonomous. Consequently, the other two models are the ones that are presented in this section. 5.2.1 Model with Early Exit for Fire and Smoke Detection The first model developed with an early exit strategy has been a model resolving the three tasks considered in this work: the object collision prediction, the steering angle regression to avoid collision, and the classifier to indicate whether there is fire, smoke, both, or neither in the image. The early exit for this model is considered to be only on the fire and smoke detection, as it is a multi-class classifier and it is easy to obtain a confidence measure from its outputs, as it can be done considering the probability of each class given by a softmax function applied to the outputs as a form of confidence. Considering the outputs of the softmax function is a first approach to obtain a form of confidence of the model; in future implementations, a deeper analysis on how to extract confidence on quantized output logits should be done. Input Convolution Convolution Pooling Convolution Convolution Pooling Convolution Fully Connected FASDD Convolution Convolution Pooling Fully Connected Collision Fully Connected FASDD Fully Connected Steering Angle Figure 10: Architecture of the model with an early exit for the FASDD task. 33 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems The model is based on the one developed in Section 4.1.2, as it has to resolve the same tasks in the same way but add the final output for the FASDD task. The resulting model is formed by four convolutional layers, followed by two branches, one with the early exit with one more convolutional layer and a fully connected layer and the other branch with two more convolutional layers and the three exits, as displayed in Figure 10. The design and the training of the model are performed using PyTorch, by using similar techniques for handling the labels as done for the model on which it is based. The data loaders have been designed to load the data, scale it between -1 and 1, and create the labels structure by filling the missing values with a symbolic number that can easily be detected in the personalized loss functions, in this case -9999. In the case of the classifier, zeros are used on the full vector of the labels, which is also easy to detect by summing up all the values of the vector, resulting in one if the label is valid and zero otherwise. For the training loop, the losses from the PyTorch framework can be used by sending the valid labels and the corresponding predictions for each picture. In this case, the model is trained by counting the loss of the valid labels in each batch, without considering that no valid labels could be found on a concrete batch. Also, for the FASDD task, the loss is computed two times, as one will count for the early exit and one for the final output, which can give two different values. The training is also monitored by computing some metrics that are printed and saved to observe its evolution, as it is computationally expensive to train a model as this one. The metrics using PyTorch have to be computed manually, so different counters have been defined to count how many labels are right and how many labels are considered in total, which allows to extract metrics as the accuracy for the collision prediction and the FASDD task, separating the early exit from the final output, and the MSE for the regression task. The monitoring of the model has been done both on the training and the validation partitions of the data, allowing to observe whether overfitting4is present or not. The strategy itself is designed in order to make the early exit resolve the task only when the model is confident enough to give an answer without running all of the network. To do that, a softmax function is applied to the logits of the classifier in the early exit, and a threshold has to be defined in order to decide whether the confidence is enough to stop the execution of this task or not. The definition of the threshold has been done by iterating over possible threshold values, making inferences over a testing partition of the data, and observing the accuracy with each value, with the objective of maximizing the accuracy of the early exit. By following this process, the best value observed as a confidence (and therefore, probability for one of the classes) threshold is 0.95, which increases the overall accuracy of the task, as the only samples treated by the final output of the model are the ones where the early exit is not sure. In Section 7, a more detailed analysis of the computational savings is given when deploying this strategy. Additionally, the early exit strategy has to be implemented in the ONNX graph to make it 4Observed during the training of the model by watching the loss of the validation dataset, as an explosion of its value would indicate that the model is learning to resolve the concrete samples of the training partition. 34 5 Reducing DNNs for Resource-Limited Platforms work on the GAP9 processor. As this early exit is not stopping the full execution of the DNN, the execution graph needs to be modified in order to verify if the confidence criteria are met by the early exit and, in the case that they are, execute only the two other tasks, saving the computational charge of the final output of the FASDD task. To effect this change, the ONNX model is first loaded through the ONNX Python library. An early exit strategy is added by inserting an If node in the ONNX graph that checks the confidence criteria of the early exit output. Confidence is calculated with the normalized outputs of the classifier and their softmax probabilities. This means that if the confidence of the prediction is greater than the set threshold, then the flow of execution will skip the computation for the final output of the FASDD task and proceed with executing only the other two tasks. The described conditional logic was programmatically introduced into the ONNX graph. The confidence is computed using the outputs of the early exit classifier, and the comparator node checks whether the confidence exceeds the threshold. The If node then branches the execution graph into two paths, where one path bypasses the final layers responsible for the FASDD task when confidence is enough and another path if the confidence does not meet the bar. Adding the several ONNX initializers and tensors to maintain confidence evaluation and conditional execution logic extends this further. This modification allowed turning the early exit strategy with the GAP9 processor compatible and efficient. 5.2.2 Model with Early Exits for Collision Prediction and Steering Angle A second model has been defined with the objective of implementing an early exit strategy that stops the full execution of the model. As it is not a complex task to measure confidence on a multi-class classifier, it is not the same case for a regression resolved with a DNN. Therefore, a model resolving only the collision prediction and the steering angle regression, integrating an early exit for each one of them, and with the objective of fully stopping the execution of the model after the early exit. The model, as shown in Figure 11, is formed by four convolutional layers, followed by the two early exit branches that include two fully connected layers, each one, and the main branch with two more shared convolutional layers and the last bifurcation, including the two outputs with a fully connected layer for each task. The training code has been implemented similarly to the previously presented model, by considering the loss of the valid labels only, using symbolic values to identify these missing values while respecting the structure of the data, and creating binary masks to avoid calculating the loss over these missing values. Two strategies have been considered for the implementation of the early exits. The first one is based on the confidence of the binary classifier indicating whether the model is confident 35 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Input Convolution Convolution Pooling Convolution Convolution Pooling Convolution Convolution Pooling Convolution Fully Connected Collision Convolution Fully Connected Steering Angle Fully Connected Collision Fully Connected Steering Angle Figure 11: Architecture of the model with two early exits resolving two tasks. enough to take the output as an answer or not. This first strategy relies on stopping all of the execution if the early exit of the collision task is taken, also taking the result given by the task of the steering angle collision, even if there are no present clues that the model is confident about the given value of the steering angle. This could be a problem, as the steering angle is needed to avoid the collision, but it is also not needed in most of the cases, as if the binary prediction indicates no collision, it is not considered. Also, in order to implement a strategy like this, the performance of the steering angle regression has to meet acceptable criteria, as it would not make sense to take the output as a solution if it produces random numbers. The quality of the early exit is analyzed with more detail in Section 7. The second strategy is based on the creation of a second model based on the possibility of understanding the error of the regression given by the DNN. The principal idea is to take the early exit of the binary classifier only when the model is sure enough about the quality of the answer, and then, if the output of the collision prediction indicates that a collision is about to happen, an evaluation of the quality of the steering output is executed in order to know if the model is confident enough about the regression output. In case it meets a confidence threshold, the execution is stopped; otherwise, only the regression task needs to be executed in its full form, so the execution of the branch of the final output for the collision task can be deactivated. The design of a strategy like that allows a huge reduction of the computational complexity of the model, and the implementation of it can be done using the ONNX library, as seen in the previous section. 5.3 Knowledge Distillation for DNNs As knowledge distillation is becoming a technique used to reduce DNNs size before deploying a model, it is considered in this work as a way to reduce the dimensionality of a model to make it 36 5 Reducing DNNs for Resource-Limited Platforms Input Convolution Pooling Convolution Pooling Fully Connected FASDD Convolution Pooling Convolution Pooling Pooling Fully Connected Collision Fully Connected FASDD Fully Connected Steering Angle Figure 12: Architecture of the student model. fit in the GAP9, even if its original form would not be able to fit. As when using quantization to reduce the data precision of the model, knowledge distillation aims to reduce the computational cost of a model while preserving its performance. In order to apply knowledge distillation and demonstrate its power, the DNNs developed until this point of this work have been designed with millions of parameters, concretely about 150 million. It is taken into account that a model this size could already have been reduced without losing accuracy, but the goal of applying KD in this case will be to reduce the number of parameters to approximately one million. The process of distilling knowledge starts by defining a teacher model, so a large model with results good enough that will be used in order to train a student model by teaching it how to learn. Similarly to the early exits and the confidence measure, this is a process very well adapted to multi-class classifiers, as the raw outputs of the model (the logits) are directly used as the way target to make the student learn; these logits are the soft labels. Of course, hard labels, so the true ones corresponding to data, are also used in order to train the model as it would be usually done. The teacher model used for applying this technique is the model with an early exit for fire and smoke detection, presented in Section 5.2.1. As the problem with KD is to apply it to other types of tasks than multi-class classifiers, this section only considers the fact of solving the FASDD task by adding KD and that the consequence will be that the pressure of this task over the full DNN will be released, making it possible to resolve the other two tasks with not that many weights. The student model is formed by two convolutional layers, followed by the early exit with fewer weights in its fully connected layer, and two more convolutional layers for the main branch, ending with the three outputs with high savings in the fully connected weights by using two pooling layers before them, as it can be observed in Figure 12. Also, each one of the convolutional layers is followed by batch normalization and pooling layers, always respecting the 37 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems sequence of layers compatible with the GAP9, which forces the batch normalization layers to be between the convolution and the activation layer. The training of the model is done with PyTorch; the dataloaders are the same as in the training of the model in 5.2.1, but the loss function has a little difference, as it has to compute the KD loss for the soft targets for the classifier. This loss is based on the Kullback-Leibler divergence, which defines a distance between two probability distributions. Therefore, the loss for the distillation is defined as follows: Ldistill =T2·KL(PS,PT) where: KL(PS,PT) = N ∑ i=1 pS(i)logpS(i)+ε pT(i)+ε Here: •Tis the temperature scaling parameter. •PSand PTare the softened probability distributions of the student and teacher networks, respectively, given by: PT=softmaxzT T,PS=softmaxzS T where zTand zSare the logits of the teacher and student models. •pS(i)and pT(i)denote the probabilities for class iin the student and teacher distributions. •εis a small constant added for numerical stability, set to 1 ×10−10 in the training fase. The T2term scales the KL divergence to account for the effect of temperature on the logits. During the training, it is defined as T=2. The loss encourages the student network to imitate the softened outputs of the teacher network, facilitating knowledge distillation. It is important to remark that this loss needs to be combined with the cross-entropy loss in order to fully train the multi-class classifier. In this case, the distillation loss is added to the total loss in the same way as the hard targets loss. This could be adapted depending on the importance KD needs to take in order to obtain better results, as well as the losses of the other two tasks, which could be weighted with the objective of not losing importance when the model is being trained. 38 5 Reducing DNNs for Resource-Limited Platforms 5.4 Compression of the Model Finally, after having tried different compression and optimization methods, one final model could be defined. As the perfect model depends on the use case that needs to be resolved, the compression of a model has as its only limit the performance loss that one could assume to lose. As the compression techniques presented in this work are all compatible with each other, a model combining all of them is not hard to create. For example, the model presented with three tasks and one early exit in Section 5.2.1, has been quantized following PTQ and evaluated, as the number of operations can be lowered thanks to the early exit at the same time as the size of the weights thanks to quantization. The results of such a combination of techniques are presented with more detail in Section 7. Nevertheless, compressing DNNs has a limit. It is observed in this work and exposed with more detail in the evaluation and results sections, Section 7, that when using KD for the FASDD task, there is an important accuracy loss for the early exit. This accuracy loss is totally canceling the effect and benefit that having an early exit represents, and the reason why this happens is unknown. One solution would be to run a deeper analysis over the training in order to understand if the distillation loss is having an impact on the early exit or on the overall loss of the model, the one which is used by the stochastic gradient descent optimizer to train the model. The fact of having multiple objectives is destabilizing enough for the training process, so adding KD, early exits, and then applying quantization to the full model is introducing a lot more parameters that have to be taken in account before deployment. Additionally, the restrictions given by the GAP9 processor and the edge-to-cloud setup make some of the problems impossible to resolve, or even impossible to reduce their impacts. For example, skip connections would have been an important feature to make the training stable, as recently demonstrated in [MacDonald et al., 2022]. One other issue that could have been solved with residual connections is the results of QAT and the fact that the early exit was giving better results than the final output, even if no explanation can be given on why this happens. This is considered to be a possible further work of this project. 39 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems 6 Demonstrator 6.1 Model Generalization As one of the objectives of this bachelor thesis is deploying DNNs in a concrete edge-to-cloud setup, it is important to have all the necessary tools to do so. In order to improve the experience of using DroneBandit, a few scripts have been developed, with the aim of facilitating the integration of a model with DroneBandit. 6.1.1 Context Vectors Context vectors are needed by DroneBandit to calculate the complexity of the model depending on the cutting point. Those vectors are similar to those used in the ANS algorithm, presented in [Zhang et al., 2021], but DroneBandit normalizes the vectors to obtain better results. These vectors define the number of multiply–accumulate operations (MACs) per type of layer, as well as the number of layers of each type and the size of the intermediate output given the cutting point. One vector can be defined for the full model but also for a part of the model, which is interesting for these optimization-based algorithms because of the computational complexity information they give. Note that ReLU layers do not really have MAC operations, so the computation time is different than for the other layer types; this is why the number of MACs is considered depending on the layer type. The size of the intermediate output corresponds to the number of bytes occupied by the data that has to be sent to the server, given the cutting point. Note that the models are quantized for the deployment, so this number of bytes corresponds exactly to the number of weights, as the data type is int8. To extract these context vectors, a code was provided that was used for the DroneBandit project, but the information extracted depended too much on the type of the model and was unable to calculate the MACs for the early exits. Given that issue, it has been added as an objective the development of a function capable of extracting those context vectors for any model. The developed code starts by extracting a summary of the model, with similar information to the one given by the summary function of the torchinfo library, but returning a dictionary with the information. Given the summary of the model, the function iterates over the possible cutting points and calculates the MACs and the intermediate output size from the cutting point to the end of the model. It also handles an early exit in the case this early exit does not present more layers than the output one. This function works for models made out of sequential blocks, as they are the models compatible with the DroneBandit algorithm. The MACs are calculated as follows: 40 6 Demonstrator Cycles Time (ms) Energy (µJ) Inferences per sec. Inference 1 24392955 65.92690541 0.7378132088 15.16831397 Inference 2 24394882 65.93211351 0.7378714948 15.16711579 Inference 3 24393346 65.92796216 0.7378250354 15.16807083 Inference 4 24393202 65.92757297 0.7378206798 15.16816038 Inference 5 24392439 65.92551081 0.7377976014 15.16863484 Inference 6 24388213 65.91408919 0.7376697777 15.17126327 Inference 7 24393120 65.92735135 0.7378181996 15.16821136 Inference 8 24391777 65.92372162 0.7377775779 15.16904652 Inference 9 24393292 65.92781622 0.7378234021 15.16810441 Inference 10 24395841 65.93470541 0.7379005016 15.16651957 Average 24392906.7 65.92677486 0,7378117479 15.168344 Table 2: Time and energy consumption on single inference for the model resolving the three tasks, one of them trained using KD. be observed as DroneBandit finds a balance by cutting the model on the second cutting point, minimizing both latencies and sending the most little intermediate output possible. Given that all the features of DroneBandit have been tested and successfully deployed on the simulation of the GAP9, the project is considered functional. On one hand, the server is able to read and obtain information running the inference from a given point, while on the other hand, DroneBandit is able to correctly choose a cutting point, adapting the decision on the latency and the context vectors. The limitations of the edge device made it impossible to join the two parts of the demonstration, but having all of its elements working also demonstrates that the use case can be solved in a fictional case in which the GAP9 was integrated into a drone with a WiFi module and a compatible camera. 6.2.3 Deployment of a Knowledge Distilled Model As all of the deployment has been made with the model resolving the FASDD task, and given that time was too short in order to follow the full process of the deployment on the model using KD, a deployment of this model has been done separately in order to demonstrate that the full model can run on the edge. This is important as DroneBandit needs to have the possibility of making the full inference on edge to evaluate where the model needs to be cut. After seeing that the model is able to fit in the GAP9, a similar extract of metrics has been done, exactly as the one done for the model resolving the FASDD task. The results, shown in Table 2, show that the number of inferences per second drops significantly. The fact that the initial number of inferences per second was almost 60 and it dropped to 15 indicates that the more complex the model is for the drone, the fewer images it can process. This can be a problem depending on the use case, and given that this is developed for a drone, the speed of the UAV has to be adapted to its processing time, as otherwise it may collide or change of direction mistakenly. 47 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems 7 Evaluation and Results This section has the objective of exposing the performance of the models developed, relating the performance loss to each compression technique that has been considered. The first model trained for this work is the one resolving three tasks, with a single output for each one of them, presented in Section 4.1.2. In the following plot, Figure 15, it can be observed the difference of accuracies between tasks. For the FASDD task, the QAT model presents a decrease in accuracy of a bit more than 5%, while the MSE is also slightly increasing. These could be acceptable results considering the loss of performance of the quantized model, if it were not for the problem found on the collision task. It is observed that the collision task is underfitting on the original model, probably because of the number of epochs that is automatically chosen by an Early Stop object, which has a condition to stop the training after seeing that the loss is not improving over the validation partition of the data. As this is a problem for the evaluation of the models and comparing them between them, this is the only model trained with an early stop condition. Figure 15: Comparison of performance metrics between the Original Model and QAT Model for the model resolving three tasks without early exits. Then, the size of the model and the time needed to make an inference have been retrieved in order to be able to understand the power of quantization. The original model weights 119.30 MB in TFLite format, saving this way the weights and the architecture in the same file, while the QAT model weights 9.91 MB, also in TFLite format. It is impressive to see how the same model with not big performance differences is capable of being compressed until less than one 48 7 Evaluation and Results tenth of its original form by reducing the precision of the weights and activations. Additionally, both models have been executed on a computer using CPUs; this is not representative of the time needed for a QAT model to do an inference, as edge devices are optimized for executing operations of this kind. Nevertheless, comparing the computing times on the same computer can give an idea of the savings that can be achieved: in this case, the inference with the original model needed 30.58 ms, while the quantized one only needed 13.99 ms. Just after the development of this model, the one resolving the FASDD task has been designed. As this model integrated an early exit, the evaluation of the accuracy aims to understand how the type of quantization can affect this one. The results of the performance of each exit are shown depending on the quantization type in Table 3. As it can be observed in the table, the model is not presenting higher accuracy with the QAT process, and the results show a better performance on the early exit than with the full model in QAT. These results are not satisfying as there is no logical explanation for the accuracy difference. The main reason that can be considered for this to happen is the instability of training a little model like the one used for FASDD and introduced by the fake nodes to simulate the quantization. Given these results, the decision taken has been to proceed with PTQ for the deployment without worrying about a big loss of performance with QAT. The model weights 0.42 MB, both in PTQ and in QAT, which makes sense as the execution graph resulting from both quantization processes is the same. QAT PTQ Early Exit 69.80 73.69 Full model 68.71 76.38 Table 3: Percentage of accuracy depending on the quantization method. When coming to the analysis of the early exits, two models have been presented, one resolving the original tasks of DroNet, the collision prediction and the steering angle regression, with an early exit for each one of them. The other one presented resolved three tasks, the two same as the previous one, and the FASDD task, with an early exit on this last task. As the second of these two models has been implemented with KD also, the first one evaluated here is the one resolving two tasks. As the first strategy relied on choosing the best confidence threshold for the collision task, different values of the possible threshold were tried in order to maximize the accuracy of the early exit, resulting in a threshold of 0.95. In Table 4, it can be observed how the accuracy of the collision task is changing when calculating it based on the strategy, so stopping the execution when the early exit confidence exceeds 0.95. Even if for the final output the accuracy decreases, the overall accuracy resulting from the evaluation is 0.9274, given that the model takes the output of the early exit 89.48% of the time. Also, the MSE is not considered because it is not a parameter of the decision for the strategy, but given the times the early exit is taken, 49 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems the overall MSE can be computed for the steering angle decision, resulting in 0.0088, which is a better result than taking the final output of the original model as a decision. Original Model Applying the Strategy Accuracy early exit 0.9036 0.9315 Accuracy final output 0.9287 0.8928 MSE early exit 0.0082 - MSE final output 0.0141 - Table 4: Performance metrics on the model resolving two tasks with two early exits, comparison between model with and without the early exit strategy, with the strategy considering only the collision task accuracy. As almost half of the model is not computed when the early exit is taken as the result of an inference, the number of MACs that can be saved has been computed. In fact, the experiment has been done over a hundred inferences, and it has been observed that about 1759721592.99 MACs are saved per inference. This represents a 89.59% of the MACs of the model. Then, a second strategy has been defined for the same model, taking also the decision of the steering angle into account for the early exit strategy. The strategy relies on predicting the error that can be given by the DNN with a pretrained and validated lasso regression to define a measure of confidence similar to the one of the collision binary task. The lasso regression is trained in order to predict the error of the DNN, and the initial hypothesis explaining why it might work is because DNN errors are often not random; they might depend on certain input features, edge cases, or regions where the model’s capacity is insufficient. Therefore, the lasso regression is trained with the same input given to the DNN, the output of the DNN for the regression, and has the objective of predicting the error done by the DNN. Then, the confidence is computed for any sample as follows: X=hFlatten(Image)ˆypredi σ=rLassoPrediction(X) 100 Confidence =1−σ MaxError where: • Image represents the input images for testing the DNN, flattened into a vector. • ˆypred is the DNN’s predicted output for the corresponding input images. 50 7 Evaluation and Results •Xis the input to the Lasso regression model, which is a concatenation of the flattened test images and the corresponding DNN predictions. • LassoPredict(X)is the Lasso regression model’s prediction of the error variance, given X, divided by 100 for better performance and results between 0 and 1. •σis the standard deviation of the predicted error. • MaxError is the maximum observed error in the dataset. This model has been tested on a testing partition of the dataset, and the results given by the model are an MSE of 0.0029300626 and an R2of 0.9979, which are excellent results and allow to state that the lasso regression is explaining the error of the DNN. Given this regression, and applying it to the strategy by calculating the confidence of the regression early exit, it has been seen that the best threshold for the regression task is 0.99, which is apparently not hard to reach as when the confidence is high enough for the collision task, the regression task is exceeding this confidence 100% of the time, resulting in the results exposed in Table 5. The performance of the collision task is the same as in the previous task, as it is the first parameter of the decision taken into account; only if the strategy is still being evaluated is the regression considered. Applying this strategy on real data results in taking the decision in the same cases as in the previous case, so no further results are given about the MACs savings, as they are the same as before. Original Model Applying the Strategy Accuracy early exit 0.9036 0.9315 Accuracy final output 0.9287 0.8928 MSE early exit 0.0082 0.0013 MSE final output 0.0141 0.0148 Table 5: Performance metrics on the model resolving two tasks with two early exits, comparison between model with and without the early exit strategy, with the strategy considering both tasks. Still in the topic of early exits, the second model that had an implementation of an early exit is the one resolving three tasks with the early exit on the FASDD task, presented in Section 5.2.1. In Table 6, it can be observed how the accuracy of the early exit is increasing, with a threshold of 0.95, chosen the same way as for the previous case. Even if the accuracy of the final output is considerably dropping, the threshold is exceeded 77.77% of the time, making the overall accuracy of the FASDD task 80.29%, which is better than what it could be done without the strategy, indicating that there is a good balance between the resolution of the tasks on the early exit when the model is confident with the tasks resolved by the full model when the model needs more computation. For this case and for the times the threshold is exceeded, it has been observed that 28456464.77 MACs can be saved for each inference, way less than for the previous model, but the reason is 51 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Original Model Applying the Strategy Accuracy FASDD early exit 0.7819 0.8608 Accuracy FASDD final output 0.8083 0.6 Accuracy collision task 0.9499 0.9499 MSE steering angle task 0.0017 0.0017 Table 6: Performance metrics on the model resolving three tasks with one early exit on the FASDD task, comparison between model with and without the early exit strategy. that in this strategy only the FASDD task is deactivated, but the full net still has to be run for the other two tasks. Even so, considering only the FASDD task, 44.76% of the MACs are saved. Ending with the same model, the last technique considered to compress a neural network is knowledge distillation, which has been applied to both the early exit and the final exit of the FASDD branch of the previous model. The student model, presented in Section 5.3, was shown to be defined with only one million parameters, compared to the 160 million of the teacher model. This way, the student model size is about 3.916 MB, compared to the 650.883 MB of the original model, and can even be reduced to 0.9965 MB if the model is quantized after the training. KD Model KD + PTQ model Accuracy FASDD early exit 0.1977 0.2065 Accuracy FASDD final output 0.8112 0.8154 Accuracy collision task 0.8455 0.8374 MSE steering angle task 0.0145 0.0155 Table 7: Performance metrics on the model resolving three tasks with one early exit on the FASDD task, with knowledge distillation integrated to the training and compared to the quantized version of the same model. In Table 7, the performance results are shown, where it can be observed that the early exit is giving totally random results, probably due to the compression level achieved. What is really interesting to remark is that the results of the PTQ model are really close to the ones of the KD model, and the results are more than acceptable for the tasks they have to resolve, having achieved an incredible compression of the size of the model. 52 8 Sustainability Analysis and Ethical Implications 8 Sustainability Analysis and Ethical Implications With the quick development in the field of technology, there is a need to think about the ethical and sustainable implications of artificial intelligence. While there’s huge potential with AI technologies, their developments also bring questions of privacy, misuse, and resource consumption to the front. This section is important in framing the limited ethical considerations relevant to this project and highlighting its alignment with sustainable practices. The ethical risks of this project are very minimal. Compared to applications handling sensitive or personal data, the datasets used in this study are publicly available and do not contain identifiable information. The case at hand deals mainly with the use of drones that can analyze visual information in real time to perform autonomous decision-making actions, like collision avoidance or fire detection, with the objective of minimizing data transfers between servers. This ultimately ensures that decisions made are only based on incoming data and context, not on any underlying data misuse or bias. Furthermore, the project explicitly avoids any misuse or even enabling it; the focus is firmly on safe and transparent applications, with no tolerance for deployment in contexts that could compromise ethical standards, such as surveillance or harmful automation. The focus of this project has been on the deployment of AI models in microprocessors for tasks such as fire and smoke detection or collision avoidance with the use of drones. These applications are thought to increase safety and efficiency in resource-constrained environments, like disaster scenarios. By limiting the scope of this project to this beneficial use, misuse is being minimized while maximizing the value of the technology. This will be a foundation of sustainability in the project of AI model compression. Optimizing models for resource-constrained devices not only leads to energy efficiency but also lowers the environmental impact of deploying AI in embedded systems. The requirement for compact and computationally efficient models becomes very critical when AI systems start working in vastly different, resource-constrained environments. This project used 60.5 hours of GPU training with an Nvidia GeForce RTX 3090. This is a significantly long training time, but in fact, many model compression methods have been performed to reduce computation requirements for both training and inference, which helped in reducing the potential environmental impact that a model like the ones developed could have. These efforts align with sustainable AI practices by prioritizing energy efficiency. Model compression also fulfills the goal of the responsible use of AI resources, as smaller and optimized models use fewer computational resources when deployed. This approach can also be considered a way to minimize the carbon footprint generated by the training processes of these models, as if deployed and used for long periods, their environmental impact is way lower than it could be with non-optimized models in the long term. 53 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Lastly, it is important to highlight the alignment of the project with the European Union’s Sustainable Development Goals (SDGs), particularly the following: •SDG 9: Industry, Innovation, and Infrastructure –By developing and optimizing AI models for deployment in resource-constrained environments, the project promotes innovation and the sustainable use of advanced technologies in industrial and operational contexts. •SDG 11: Sustainable Cities and Communities –The use case of autonomous drones for fire and smoke detection directly contributes to safer communities by offering the possibility of responding to environmental emergencies in real-world scenarios. •SDG 12: Responsible Consumption and Production –The project exemplifies responsible resource usage by prioritizing efficiency in AI model training and inference. •SDG 13: Climate Action –The energy-efficient design of AI models supports climate action by reducing the environmental impact of deploying and training AI systems, particularly in energyintensive scenarios. 54 9 Conclusions 9 Conclusions During this project, different DNNs have been designed in order to fulfill the given objectives. The bachelor thesis explored the different possibilities of compressing models in order to deploy them on an embedded system using a GAP processor as the edge device. The techniques that have been experimented with include quantization, early exits, and knowledge distillation, all of them respecting the restrictions of the given edge-to-cloud setup. The use case of an autonomous nanodrone has been the base of the project, making the goal of each developed model to solve the basic tasks usually operated by humans when working with UAVs. The compression techniques demonstrated that DNNs can be optimized with minimal impacts on their performance, while the successful deployments on the edge device show that the implementation on a drone is perfectly possible. This research has been carried out in the context of a research group of the REDs Institute at the HEIG-VD, so the outcome of the project will allow a democratization of the use of the GAP9 microprocessor in the institute, also opening a door to the use of such embedded systems for the laboratories of the subjects carried by the professors working at this institute. More importantly, this work included the development of many tools that make the use of DroneBandit, the project to which this thesis is linked, easier. By providing the necessary tools to sequentialize a complex model and the automatic extraction of context vectors, DroneBandit becomes the perfect algorithm to be chosen when defining a dynamic edge-to-cloud strategy for DNNs. On the other hand, the limitations encountered with the GAP9 processor limited the possibility of deployment and demonstration of the project. The challenges presented in this thesis have been the main pain points of the work and have been overcome by proposing alternatives that were not planned on a first time. Using a cutting-edge device like the GAP9 allowed experimentation with uncommon and inaccessible technology, but at the same time meant an increase in the workload of the project. The future work of this thesis relies on the integration of the GAP9 in a real nanodrone and the testing of the models in a real-world scenario by experimenting on the effect that the different presented compression techniques can have. On a final note, a reflection about the modern form of working with AI is made. The unstoppable pursuit of training larger and larger models has led us to a critical point where the sustainability of AI must take precedence over its scale. As demonstrated in this thesis, the deployment of DNNs cannot, and should not, mirror their training phase. The availability of advanced compression techniques has proven that remarkable efficiency can be achieved without compromising performance. It is time to change from a mindset that glorifies size and complexity to one that values innovation, resources, and responsibility. The true potential of AI lies not in how big it can become but in how thoughtfully it can be developed and deployed. 55 Optimizing Deep Neural Networks for Resource-Limited Embedded Systems Appendices A Hopfield Network as a Solution for the Image Classifier One of the proposed goals to initiate the project was to find a new use case for the drone other than its capability to circumvent obstacles through the edge-to-cloud processing approach. The concept concentrated on making the inference process autonomous on the processor of the drone without the requirement of server connectivity. This then led to exploring object detection as a potential new application, which is one of the classic tasks falling under computer vision. The approach envisaged the use of a Hopfield Network, a network well known as capable of reconstructing patterns from binary images. In the proposal, that meant training the network on the binary images of an object and then running inference continuously to determine how the network energy state was behaving. If the network converged to a trained pattern based on a predefined threshold, that would signal that an object had been detected. However, implementing this solution revealed some challenges: 1. Binary Input Requirement: The classical Hopfield Network requires binary input data. To satisfy this requirement, a Canny edge detection filter was applied to preprocess images, producing binary contours of objects. 2. High Dimensionality: The number of neurons in the Hopfield Network must match the dimensionality of the input data. For a 200x200-pixel image, this meant 40,000 neurons, leading to a network with over 1.6 billion parameters due to its full connectivity. This made the model computationally impossible to train or deploy. 3. Dimensionality Reduction Attempts: To mitigate the high dimensionality, several strategies were tested: • Applying PCA after binarization reduced the dimensionality but produced nonbinary outputs, which could not serve as inputs to the Hopfield Network. • Applying PCA before binarization failed to preserve local features, making edge detection ineffective. • Cropping regions with maximum information succeeded in reducing image size by half while retaining critical object features. Still, the results were limited. Experiments on the CIFAR-10, CIFAR-100, Imagenette, and Himax datasets demonstrated that the network was only able to identify objects when the input image was very close to the training data. Furthermore, because of the high dimensionality, the network would often converge to the same energy state for the object and without. With these limitations in mind, the Hopfield Network was not suited for this task. 56