Safety check in critical safety scenario for self-driving vehicles

Equation Chapter 1 Section 1 Master´s Thesis Master in Electronics, Robotics and Automation Engineering Safety check in critical safety scenario for self-driving vehicles Autor: Raúl López Martín Tutor: Daniel Limón Marruedo Department of Systems and Automation Higher Technical School of Engineering University of Seville Seville, 2020 iii Master´s Thesis Electronics, Robotics and Automation Engineering Safety check in critical safety scenario for selfdriving vehicles Autor: Raúl López Martín Tutor: Daniel Limón Marruedo Department of Systems and Automation Higher Technical School of Engineering University of Seville Seville, 2020 v Proyecto Fin de Carrera: Safety check in critical safety scenario for self-driving vehicles Autor: Raúl López Martín Tutor: Daniel Limón Marruedo El tribunal nombrado para juzgar el Proyecto arriba indicado, compuesto por los siguientes miembros: Presidente: Vocales: Secretario: Acuerdan otorgarle la calificación de: Sevilla, 2020 El Secretario del Tribunal vii Appreciations My special thanks to Hitachi Company and Vulcanus Programme for allowing me to what can only be describe as the best experience any engineer may wish for. To my family for always trust in me and always keep supporting me. Special thanks to Daniel Limon, tutor of this project, for trusting in me and all the lessons he gave me. Special mention to Rodrigo Alcubilla for making the whole stay in Japan a year I will never forget. ix Resumen En un Mercado emergente como es el de los coches autonómos una de las funciones esenciales es asegurar la seguridad del funcionamiento de dichos sistemas. La mayoría de los comportamientos del estado del arte son capaces de manejarse en un escenario sin anomalías. Sin embargo, las dinámicas del entorno, tales como las condiciones metereológicas o las oclusiones de los sensores, pueden comprometer el funcionamiento de estos. Para ampliar los escenarios en los cuales estos sistemas son capaces de funcionar, es necesario incluir nuevas funciones de seguirdad para una conducción segura en cualquier entorno. Esta contribución demuestra una validación para vehículos autónomos basada en las condiciones del entorno. Se propone una comprobación de los obstaculos y limitaciones de los sensores. Para ellos se define una Region de Interes (RoI). Combinando ambos conceptos se obtiene un valor cuantitativo del conocimiento de los entornos del sistema. La idea propuesta se basa en modificar el plan de actuación según dicho valor, mejorando el tiempo de reacción ante situaciones imprevistas. Los resultados de la simulación e implementación física en el coche autónomo muestran una mejora en los tiempos de reacción ante situaciones fuera del dominio operacional designado. Se considera que este proyecto resuelve una de las condiciones obligatorias para conseguir un coche autónomo con un nivel de automatización de nivel cuatro. xvii FIGURE INDEX Figure 1 Similarity between biological and artificial Neurons 2 Figure 2 FSM Scheme 6 Figure 3 Behavior tree Scheme 6 4 MDP Scheme 7 Figure 5 NN Scheme 8 Figure 6 MPC Time representation 9 Figure 7 LiDAR Working 11 Figure 8 LiDAR Example of data 12 Figure 9 Scheme for CMOS and CCD cameras 12 Figure 10 Image classification, Object detection and Instance Segmentation 14 Figure 11 Safety check scheme propose 14 Figure 12 Region of interest required. Intersection scenario 15 Figure 13 Region of interest feasible. Intersection scenario 16 Figure 14 RoI Definition 18 Figure 15 Matlab program blueprint 21 Figure 16 Reynolds' steering behavior 22 Figure 17 Egocartesian representation 23 Figure 18 Seeking destination without mid-way targets 23 Figure 19 Seeking with few ahead vision 24 Figure 20 Seeking target looking ahead 24 Figure 21 Ideal target seeking 25 Figure 22 Realistic target seeking 25 Figure 23 first try: Total failure 26 Figure 24 Map of Oomika Station surrounds 26 Figure 25 Real map based consecutive target seeking 27 Figure 26 Multiple vehicle path following. Steering Behavior. 27 Figure 27 Incorporation experiment. Steering Behavior 28 Figure 28 ADST: Menu Bar 29 Figure 29 ADST: Actor creator Menu 29 Figure 30 ADST: Trajectory Definition 30 Figure 31 ADST: Sensor placement 31 Figure 32 ADST: Sensor Data Acquaintance 31 Figure 33 ADST: Representation of experiments 32 Figure 34 ADST: Collision detection 33 Figure 35 LiDAR-like radar configuration 34 Figure 36 Big Intersection initial trajectories 35 Figure 37 T section initial trajectories 35 Figure 38 Display of state of the car 36 Figure 39 Small intersection. RoI 37 Figure 40 Small intersection. RoI bug 38 Figure 41 Big intersection. RoI parametrized 39 Figure 42 Big intersection. RoI feasible Y/N 40 Figure 43 Sensor occlusion basis. Matlab 41 Figure 44 Object occlusion measurement. Matlab 42 Figure 45 Sensor capacity after occlusion. Matlab 42 Figure 47 Small intersection. RoI feasible 43 Figure 48 T section. Occlusion in sensors 43 Figure 46 T section. Crash due to lack of visibility 44 Figure 49 T section road. RoI Feasible 44 Figure 50 Speed and distance check with and without our proposed safety layer implemented. We can see how the accident is prevented by improving situational awareness 45 Figure 51 Comparison between FSM with and without our proposed idea, in detail. On the first picture there is the result from the Figure 8 with the accident, on the second picture the speed using our safety assessment management. 46 Figure 52 RQT from rerunning experiment 49 Figure 53 Autoware graphic representation of experiment. Moriyama ROSBAG. 50 Figure 54 RoI Waypoints. Autoware 51 Figure 55 RoI Intersection. Autoware 52 Figure 56 Polygon range. Figure 1 53 Figure 57 Polygon range. Figure 2 54 Figure 58 RoI Feasible. Third Person View Plot 55 Figure 59 RoI Feasible. Birds Eye Plot 56 Figure 60 Speed comparison based on feasible RoI values 57 Figure 59 RQT from Autoware simulation 64 xix List of abbreviation Abbreviation Meaning GPU Graphic Processing Unit CPU Computational Processing Unit NPU Neural Processing Unit NN Neural Network CNN Convolutional Neural Network DNN Deep Neural Network OD Object Detection SSD Single Shot multibox Detection. Framework for OD YOLO You Only Look Once. Framework for OD DoG Difference of Gaussian LiDAR Light Detection And Ranging 3D 3 Dimensions Laser Light Amplification by Stimulated Emission of Radiation FPS Frames per second m Meters. (cm=centimeter… etc.) CCD Charged Couple Device CMOS Complementary Metal-oxide-semiconductor Gb Gigabyte GDDRx Double data rate, a type of dynamic random access memory GHz Giga Hertz CUDA Compute Unified Device Architecture AI Artificial Intelligence POMDP Partially Observable Markov Decision Process MPC Model Predictive Control FSM Finite State Machine RoI Region of Interest RT Real Time App Application ADST Autonomous Driving System Toolbox. From Matlab. BEP Bird Eye Plot ROS Robotics Operating System ID Identification Digits Table 1 Abbreviation table 1 Introduction 2 1 INTRODUCTION How do we understand the environment surrounding us and how do we act within it? For humans, and rest of the animals; once they reached their maturity, being able to extract information from the environment and react in consequence is a process that usually happens without much effort. A human does not have to think how to steep his feet to not fall while walking, like a lion does not have to think he has to turn right while running to not bump to a tree. But this process are not innate for us, yet we develop these skill since the moment we are born. We learn how to climb stairs, how to stand up, how to talk and how to read the surroundings to extract the key information from it. Still, the functionality of the brain is not fully understood, yet; as humans always do to create things, we can inspire ourselves in the nature to try to reproduce an artificial brain. If brain were fully understood and a perfect model of it could be simulate, then maybe a complete artificial brain could be created. As there is no complete model for the whole brain, we just focus on all members that are part of it. That is why using a similar structure than the one presented in our brains such a connection on simple links in a complex system, still much different from the one in human brains, can be the base to found an artificial brain. This artificial brain will undergo to a series of process to learn how to behave, depending on the applications which we want to carry out. Figure 1 Similarity between biological and artificial Neurons But this is not the only way to develop artificial intelligence. Since early in the 50s several attempts to design and develop intelligence have been tried out. From large states machines and tree behaviors that compiles all the possible situations that can happen, to mimic chatbots that tries to copy certain activity by being exposed for a long time to the correct answers. Any process that allows a machine to learn from its errors can be considered as an artificial brain. What it would be use this artificial brain for? There are plenty of uses for artificial intelligence, every action that humans and animals performs can be imitate by a machine using AI, from how to talk to people, how to drive a car, how to behave in situations or even how to play board games. Of course, not all machines have to be humanoids or big servers 1 and most of them can make the simplest action and still use an artificial neural network. Not all the machines tries to reproduce the whole brain itself, but just a part of it able to perform the desired action. There are plenty of choices when these artificial brain become pretty handful, as early detection on cancer or diabetes related sight problems. Some of the functions have a road long yet to be traveled, as it 1 As most Sci-fi movies and books like to imagine. 3 3 Safety check in critical safety scenario for self-driving vehicles comes, for instance, to creativity process such as painting or coming up with jokes. Others are rather advanced and the applications are already in the society or soon to be released. One of the target for these intelligent machines is to perform a self-driving car where the driver is not needed anymore. And that is the topic we will be discussing over this document, how to make a car move by itself, which information should be recollected from the surroundings and how to react about that information. How a car does drives itself? Or, another way of asking ourselves this, what kind of actions has to do a machine in order to be able to drive a vehicle? Of course it will have to connect with a GPS system and be able to trace a route from the initial place to the destination, but mostly, it has to make decisions from the information obtained from the sensors. One of the task we face in this problem is not only to be able to record the environment that surrounds the cars but also to be able to understand and to make choices from that information. Even for humans, get use to recognize dangers, pedestrians, animals and traffic lights, the first time we drive a vehicle can be overwhelming. Therefore, training a machine to be able to check the surroundings and recognize all the things that are going on around it, and with that information make a decision, it is a wide task. When it comes to recognize and make a decision, one of the most common way to perform this purpose is to use an Artificial Neural Network and train it to focus its optimality for the task itself. Although there are other ways to accomplish this task, as we will discuss later on this document, we shall focus on the NN behavior. Once our NN is already trained for this performance, it could be applied in real-life actuation. Once all of this has been done, the car would have a perfect understanding of the surroundings in that instant. But when it comes to human behavior not everything can be guessed by just one image and probably information from previous state is also required to make a prediction on their future movements. With all this information acknowledged, the system should make the optimum action. Why to make a car drive itself? Surveys made among car users proves that around half of the population, in surveyed countries, is still feeling insecure about the fact of using an autonomous vehicle. Users and pedestrian are concern about traffic congestion, inefficient driving, but mostly, about the safety while using a car or walking on the streets. As a matter of fact, in early stages of NN, performance was not enough to these kind of applications. Lately, a lot of research have been made in NN and other machine learning techniques and its development, expanding fields of application more and more in the past few years. From the start of the first capable-to-learn systems in the early 60s, until the techniques for face-detection in your smartphone; the structure, training and computability skill for artificial intelligence have changed widely. In experiments performed in recent years, AI accuracy has overtaken human capacity for some applications, as it is image classification or detection for small spots in cancer detection, among many others. Nevertheless, most of the applications for AI in those cases are just used as a filter, to underline the cases that requires an expert attention. Same happens with the self-driving cars which are being sold at this time. Those cars, although they can drive itself, they require a driver placed in its position and focused on what is going on the street, able to react in case that some anomaly happens or a wrong operation has or is going to be taken by the car. What have the future ahead waiting for us? Not all the scientific community are really willing to use or comfortable with NN and some of the applications which these are being used for. The way NN are used make of them as a black box where you cannot know what is happening there made them unpredictable, and that is the main reason not all scientist are wager to use this technology. Other technologies have still to grow to be able to perform a robust and fast performance. However, self-driving vehicles are already a fact, they are being sold already and improving their skills with their year. Will it come a future when fully autonomous vehicles can be used safely? Only time knows. 1.1 Context of this Project This Project was developed during an internship in Hitachi Research Lab, placed in Oomika, Japan. Such internship was part of Vulcanus European Program that aims to merge Europe and Japan´s industries by funding bridges between companies and graduate engineer and scientist students. For the development of this project there were several resources available from the company. Starting with a high-technology computer able to create and computes NN as fast as possible, following an Autonomous Driving Vehicle, patent of the company. As this idea is meant to be used as an upper layer algorithm, ready to be included in any decision making module, none of the previously developed code were modify during the course of this thesis, only new modules were added to the previous ones. Introduction 4 5 2 GOAL The first thing to do is to specified goals and search for the state of art in the desired field. The main purpose of this project is to create an additional safety layer to the decision making in autonomous vehicles. To accomplish that task it is needed to develop an Artificial Intelligence for decision making for a self-driving car. This car must be able to perform some task that also are crucial to be defined. These are the main tasks to perform by self-driving car: - Being able to follow a route and reach a goal point in an optimum method. - Detect other dynamic objects present in the environment and react safely for them and the car. - Keep the track inside the lanes. - Respect traffic rules. - Perform the movement as smooth as possible In order to achieve all those task, there are different ways to adapt a machine to learn how to safely drive and optimizing the route, reach from a starting point until the destination point (or at least one place nearby that last desired position). As we discuss in the previous section there are several ways to recollect data from the environment to use it on the car, so the ways to control the car may vary depending on how they take down all data. Likewise, some of the implementation were not aiming to create a full driving model for autonomous vehicles but few implementations as lane keeping and overtaking some slower cars, for instance. These are some of the different methods used: - FSM: A machine with a finite number of state was one of the first ideas used to design autonomous vehicles. This method is as various and wide as the programmer wants to go. It can be used for a simple action as is parking the car, by riding the steps one by one; or it could implement big actions as a control for a whole plant. One of the biggest load it has the finite state machine is the need to program each one of all the possible situations that may happen in the road, which lays between a hard and risky thing to do. If one scenario is no declared in the FSM the car would not be able to react, but it is needed to be said that most of the other methods does not have neither the ability to react against undefined situations. This method have a bad reusability and the process must be rewritten for each situation that it want to be done. Vision in Vehicles 12 In the previous picture a typical disposure of sensors in an autonomous driving is dispatch. As it is better to have reliable data from surroundings, it is always an improvement to obtain data from more than one source in more than one method. LiDAR has some really good specifications, which makes it really valuable when obtaining visual data for these kind of applications. One of its advantages is that it doesn’t rely on the light ambience, making easier to detect object in the map created. It also doesn’t have a range limitation when it comes to angles, it can cover all around the car, except from those angles pretty near the car that are not caught by the perspective of the LiDAR, if placed on the top of the car. Length range is also pretty good, covering up to 250m and with an accuracy of 0.5cm. Ahead, an example of a map created by LiDAR is shown. Figure 8 LiDAR Example of data Once you get this image, you need to train your vehicle to be able to recognize other vehicles, pedestrian and other obstacles that may exist in the road. Also, lately, lines for keeping track of the road and the direction and orientation of objects have been also detected by this technology as the power of the laser steam changes depending on the surface they reflect on. 3.1.2 Camera For the visual recognition with the camera, there are two main technologies that are being used. CCD and CMOS. Both of them works mostly in the same way, working by MOS transistors technology, but with a different way of recollecting the light from the target. The main, and simplified, scheme is the following: Figure 9 Scheme for CMOS and CCD cameras Keeping the transistor functioning in the inversed zone, when the protons from the light enters in the well of the transistor, the energy they are carrying through their movement is transmitted to the electrons, creating a current in the device. By making an array of transistor for each pixel we will be able to distinguish how strong is the energy in each one of the pixel. For each pixel, one transistor is needed in case the picture is only in black and white. To make colored images a prefilter is needed in the transistors. Each one of the will only 13 Safety check in critical safety scenario for self-driving vehicles allow one the 3 primary colors to go in. 3 With that made, the system is allow to take a picture, and by repeating this process in a periodic way, videos are taken. This is the technology included in the cameras from selfdriving cars. About which technology to use is up to the company and both of them have their unique weakens and strengths, which made them fit better for some applications or for others. The big challenge when it comes to vision by cameras is not the fact of obtaining the value of each pixel but to interpret them. The information that is available from a camera is wider than from any other sensor nowadays, although the process of that information is the hard task to accomplish. There are several ways to solve this, as CNN, and, even though the accuracy is quite well, they still have some room to improve. 3.1.3 Radar Same technology that has been used since the 30s for detecting incoming vehicles, mostly flying, can also be used for autonomous vehicles. By sending an electromagnetic wave in a direction, and by catching the reflection from when this same wave hits with some object, you can determine the position and velocity of the objects: 𝑓`=𝑓∗𝑣𝑤𝑎𝑣𝑒+𝑣𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑟 𝑣𝑤𝑎𝑣𝑒−𝑣𝑠𝑜𝑢𝑟𝑐𝑒 [1] Doppler Effect 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒=𝑣𝑤𝑎𝑣𝑒∗𝑡𝑟𝑜𝑢𝑛𝑑𝑎𝑏𝑜𝑢𝑡 2 [2] For EM waves, the velocity will be equal to the speed of light, 300.000 km/s. Based on these two physical effects, measuring the frequency of the wave and the time it takes for the wave to come back: knowing at which speed the cars is currently moving, it is possible to calculate those two values. Distance and velocity of obstacles ahead in the road. Different sensors are placed all around the car to add some safety to the vehicle by measuring distance and velocity from the obstacles in various angles. With a similar effect but for a closer range (up to 6 meters) ultrasounds can be also be used, perceiving the same effects than the radar. Usually radars are placed in the front part of the car while ultrasonic sensor are placed for backwards movement. 3.2 Visual Recognition Each autonomous vehicles will use assorted sensors, with at least one of the previous list, depending on how they would like to proceed with the visual recognition problem. Once the data is collected, a process of those data is needed. When looking to a picture and trying to figure out important information there are different grades on how accurate you can determine the objects present in the frame. Depending on those grades there are different methods to extract the features and applications to be used. 3 The three main colors for additive systems are Red, Green and Blue (RGB). Vision in Vehicles 14 Figure 10 Image classification, Object detection and Instance Segmentation The first step you can do with an image is to recognize the objects presented on it, even when there are more than one, and export that label outside. On the example we can see, how we got an answer of “bunny”. This is called Image classification; this is not enough for the application we are trying to use. The second step would be to surround the object with a square inside the image. As we are looking for more than one object and bounding this object with a box, usually the prediction of an object with its box comes with a probability of existence and probability of what kind of object it is. One of the things to take into account when carrying out this is when bounding boxes overlap, matching one object with more than one object. This is not the most precise one, but is enough for detection pedestrian, traffic lights, bikes, cars and others objects typical from the roads. The third step, is to not only to select which kind of object it is, and where it is placed, but to limit its shape and differentiate each one of the objects, where are the boundaries and connections with other objects. It gives the biggest accuracy for problems of location and detection of objects, but takes much longer times to make this process. 11 4 SELF-DRIVING CARS NOWADAYS The fever of self-driving vehicles is at its peak currently, as many of the main developer of automobile companies are investing considerable amounts of money on research on those topics to achieve as fast as possible a dully autonomous vehicle. Being a technology developed since early 80s, there has been many steps to reach where we are now. Self-driving cars improve in the way daily vehicles works, and is only the first one among several modifications until an SmartDrive system can be fully developed, Inter-connected (where it relies one of the main problems in the present, due to the funnel in wireless communications) cars with the capacity of reading the environment. This could reduce the risk of accidents in the roads, up to 60% by 2030 with the use of self-driving cars. However, this growth is unequable over the world, depending on the legislation and presence of R&D industry in each country. All those countries trying to get self-driving cars in the present world started a race to win the market. The race for self-driving vehicles has already started, but there are some struggles they have to overcome. Besides the technological problems to be solved, as it is a real change on the way transports are used, society needs to get used to these changes, therefore, as Violeta Bulc said: “Autonomous mobility will only real take off to the extent that is accepted by passengers and citizens as human beings and by society at large”. You cannot get ahead the market, so these vehicles will only be profitable as long as the target group is big enough, and that start with optimizing the performance of them. Other problem to deal with is the ethical behind the programing in decision-making and the responsibilities in accidents, which are deeply different depending on the country. There are different opinions about who should be accountable for the accident (in case there was one), creating a gap in the law about insurance. Most of the cars released into the market by now are Level Three automated driving2, so pilots should never distract themselves from the road, and they have to react in case an incoming accident. This means that responsibilities from crash and pedestrian injuries are still pilots’ responsibilities. This could be one of the main reasons that development of self-driving techniques are so uneven, as the laws adopted by each country varies, some circumstances does not support the implement of this raising upturn. From now on, the driver will have helps on the direction, speed, and maneuver, which implies a change in mentality of the drivers to be willing to take this change. Alongside with this problem, comes the moral implied in programming death near experiences reactions for the autonomous vehicle. “What to allow to do to preserve driver and passenger’s life?”, “Has the car save a pedestrian or the driver?”, “What if there were more than one pedestrian that could be injured?”, “How many life has to be put in danger to sacrifice drivers´ life?” are some of the question being held by moral professionals. American Automobile Association studied that around 70% of American would be scared of to ride a fully self-driving vehicle. This fact states that society is still yet to adopt this new technology as a reliable one. The three main zones were this technology is researched and they are making progress are Europe, Japan and USA. This does not mean that other zones are not working currently on this technology, but their short time dedicated to it or lack of resources make them to be outside the highlights on this reinvention of driving as we know in the present. As for Europe, there is a different horizon for this technique. As Europe cannot be defined as an unique country when it comes to laws and insurances framework, all the efforts to try to create an standard law it has been harder. There is not a clear frame for all these situations, which involve self-driving vehicles. That makes uneven develop inside Europe, with each country applying laws separately. There were a restriction until midterm of 2017 in Europe that forbid any autonomous vehicle to move faster than 10 kilometers per hour [6]. This lack of laws and support to this researching, makes the market less attractive for the companies who are already working on this, such thing happens for Volkswagen. They have to release the new technology in other part of the world, because Europe is not the right market nowadays. As Johann Jungwirth, Volkswagen Self-driving cars Nowadays 12 12 Group chief, said in 2017 “And then comes Europe. We would love to come earlier since it’s our home market, but the legislation just isn’t there”, pinpointing that Europe´s laws are not yet created to accept this technology in the society. When it comes to this field, the variations made to the laws has been really slow during the last decade, delaying almost 5 years to catch up with other countries on the head of this race . All the procedures are trying to be done really slow and with the right terms so they can last longer, that is why they are doing some evaluations about ethical questions and how they should be answered. A team of experts that will discuss all these problems carries that study. Nonetheless, Europe is trying to catch up with the rest of the industry [8], by supporting a development of these vehicles. Several countries, such as German and England, they have changed the law in order to agree with the development already carried out in those countries. For example, Audi has already created several self-driving cars, which has been tested in closed circuits and they are expecting to release them to the market by 2020’s. As for Japan, it has to been said that they raised a lot of positions on the “race” for the autonomous vehicle integration in the roads, having several companies researching in this field, like Toyota, Nissan and General Motors (which is one of the companies who has spent more money on this matter, with a budget of 2.25 billion $ for 2018) among others. Between Japan and USA are competing for the first spot on this race, both of them creating new vehicles with Level 3 Autonomous Driving for the market, and currently working in Level 4. Yet, not a single company has finalized developing a Level 5 Autonomous Driving vehicle. Waymo (USA) and General Motors (Japan) are the companies with the biggest amount of self-driving cars currently working in the world. Japan is now outlining laws about these vehicles, and how the responsibilities is hold by the pilot in all Level 3 vehicles. There are new policies for this incoming fleet of cars, which covers the main scenarios that could happen [1]. In addition, Japan is experiencing an uprising economy for the last decades that has position Japan as the second largest economy nowadays, which implies a larger amount of resources available for the market. Incoming large event such as Olympic Games is also a notable impulse for Japan to release this new technology as soon as possible. Prime Minister of Japan, Shinzo Abe, said this year that by 2020 (year of the event) self-driving cars will be try to be released to the market, including a fleet of self-driving cars dedicated for public transport . With this goal set in the future for Japan, all the research and development about this field is being speed up, with a lot of companies trying to get in the head of the revolution of the selfdriving vehicles. As a sum up for this section, a list of the biggest companies participating on AV technology nowadays: Zone USA Europe Japan Companies MobilEye General Motors Microsoft Nvidia Waymo (Google) Audi BMW Volkswagen Delphi Renault Nissan Toyota Honda NTT Data Fuji Heavy Industries 13 Safety check in critical safety scenario for self-driving vehicles Region of Interest 14 14 5 REGION OF INTEREST This idea´s main contribution is a new layer, easy to implement and scalable, on ADAS, to increase safety in decision making, especially for level 4 autonomous systems. Up until now the decision making was based in 3 main process: Sensor information, recollecting and processing all the data from the sensor obtaining a clear scene of the surroundings; Plan planning, with all the information from the sensor to decide an optimum path to follow; and lastly to convert those decisions from the plan planning module to actuation signals, which are usually acceleration, braking and steer angle. With this technique we included two more steps in the process. In sensor information it would need not to only get all the data from environment and to detect all the objects present in the scene, but also to check the occlusion those objects are carrying out in our vision. With that information we create an ideal plan the same way we were doing without safety check module. With that ideal plan we can enter in the safety validation module, where will need information from the previous two modules. In this module we will calculate the Region of interest required to check for the selected path in the plan planning. Obtaining this region of interest is hold on debate and can be determined differently by each user. Sensor information Ideal plan planning Safety validation Real plan planning Actuation Figure 11 Safety check scheme propose 15 Safety check in critical safety scenario for self-driving vehicles To explain what a region of interest is and how we would check that space, is better to establish some background scenario. One of the most interesting scenarios to apply the region of interest is those where the visibility is reduced. We shall take an intersection with some obstacles in the corner that blocks our sight for the incoming cars or the next step of the route. Within this scenario we shall restrain from the whole map which are the areas of this zone we need to check before taking any action into course. From the map of the area we can check where are the cars allow to come to our position, which is maximum speed those car can be travelling at and other features from near stances that may be useful for accident prevention actions. Figure 12 Region of interest required. Intersection scenario From that scenario, with the new information obtained by the sensors in the first module (their range of vision and occlusions) we can establish how sure the ego vehicle is about its surroundings. Not only to check whether we can see a car or other vehicle coming to the intersection, but also to see how much of the scene we are being able to check. With all that information we can obtain the region of interest feasible. In other words, the part of environment the car is actually checking in that moment. Looking at Figure 13 Region of interest feasible. Intersection scenario there are three different colored areas able to see. The yellow one, as it was already written in Figure 12 Region of interest required. Intersection scenario is the RoI required for that moment. The green area in the figure represents the area of the RoI that has no occlusion, therefore, it can be checked. That is going to be called the RoI feasible. As the green represents the RoI feasible, the areas that remains yellow in the figure are not only areas that are required for the action in that instant yet areas that are not covered by the sensors in that current moment. The salmon color areas in the figure are the spaces that are being covered by the sensors, but the information from them is no longer required by the decision making module. How to proceed with this areas is up to the user, as they can contain information that may be worth in the future such as tracking objects in those areas that might be moving into the RoI. Another action to be perform due to this information is the selection of the most important sensors, putting to sleep those sensors which are no pointing to the RoI of that moment, speeding up the processing of data. Region of Interest 16 16 Figure 13 Region of interest feasible. Intersection scenario After obtaining this feasible RoI, it is time to set up certain constrains as the last step in Safety validation. Based on the areas of the RoI which are in the sight of the ego vehicle and the initial plan planning done, there are certain actions to be taken. These actions can be chosen at ease, for instance, in case we have not enough visibility the speed can be reduced, or when doing a turn or an incorporation, when there is not enough area in sight, we can carry out a small move to increase our feasible RoI. One example of this situation would be a traffic jam, where the cars in front and in our back blocks the sight. In this situation is sometimes recommended to move the car slightly to the side to increase the visibility before starting any action. Once this constrains are settled, a new plan planning can be carried out. This new plan increased the safety of the behavior of the car. Then, we send the information of the desired path to the actuators. Something crucial to emphasize is the selection of this region of interest. One of the basic selection, based on the scenario, which can be done for the RoI is to check ahead for the maximum braking distance, based on the comfortable braking deceleration and current speed. On a more complex way, the RSS model [5][25] checked all the possible incorporations in the road, always calculating the needed braking distance, 𝑑𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥 𝑒𝑔𝑜 . That distance can be obtained by using a comfortable braking deceleration, which was established to be under 3.4𝑚 𝑠2 [15], 𝑎𝑏𝑟𝑎𝑘𝑒 𝑚𝑎𝑥 . If the car started braking with an initial speed of 𝑣=𝑣𝑜𝑒𝑔𝑜𝑚 𝑠 then: 𝑡𝑏𝑟𝑎𝑘𝑖𝑛𝑔 𝑒𝑔𝑜 =𝑣𝑜 𝑎𝑏𝑟𝑎𝑘𝑒 𝑚𝑎𝑥 𝑑𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥 𝑒𝑔𝑜 =𝑣𝑜𝑒𝑔𝑜∗𝑡𝑏𝑟𝑎𝑘𝑖𝑛𝑔 𝑒𝑔𝑜 −1 2∗𝑎𝑏𝑟𝑎𝑘𝑒 𝑚𝑎𝑥 ∗𝑡𝑏𝑟𝑎𝑘𝑖𝑛𝑔 𝑒𝑔𝑜 2 To perform all the test in a safety way and to endow the system with a loose threshold to detect incoming dangers, we enlarged the distance by 25% to always stay on the safety side, 𝑑𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥 𝑒𝑔𝑜 . Within this Region of interest we may encounter static obstacles in the ego vehicle path, other vehicles following the same lane than us, and vehicles and pedestrian crossing through our region of interest, yet, cars circulating on the opposite direction are not considered in this approach as it is an illegal behavior. With a definition of speed as following 𝑣=𝑣𝑜∗cos (𝛾), with 𝑉𝐿𝐼𝑀𝐼𝑇 as the maximum speed allowed in the current scenario, 𝛾 from the system of reference the road, being zero radians circulate on the direction of the road. 𝑣𝑜∈[0,𝑉𝐿𝐼𝑀𝐼𝑇]⊂ ℝ 𝑎𝑛𝑑 𝛾∈[𝜋 2,−𝜋 2]⊂ℝ. What it is widely defined in [5] about safe distance and keeping AV cars from dangerous situations, is deeply 17 Safety check in critical safety scenario for self-driving vehicles touched in [25]. Using the same nomenclature as of [25], 𝜌 is the response time, and 𝑎𝑎𝑐𝑐𝑒𝑙,𝑚𝑎𝑥 𝑒𝑔𝑜 as the maximum acceleration for the car during the response time. As it was already presented in [25], to assure safety for the ego vehicle is needed to keep a safe distance, both longitudinal and transversal, enough to avoid any dangerous situation that may occur. We applied this same definition for safety to calculate our RoI. With other vehicle on the scene with 𝑣𝑒𝑥𝑡 in our same road, 𝑟𝑚𝑎𝑖𝑛 with the same speed limit than the ego vehicle, 𝑉𝐿𝐼𝑀𝐼𝑇 𝑒𝑔𝑜 , as defined in [25] a safety distance for braking is: 𝑑𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥 𝑟𝑚𝑎𝑖𝑛 =[𝑣𝑒𝑔𝑜𝜌+ 1 2𝑎𝑎𝑐𝑐𝑒𝑙,𝑚𝑎𝑥 𝑒𝑔𝑜 𝜌2+ (𝑣𝑒𝑔𝑜+𝜌𝑎𝑎𝑐𝑐𝑒𝑙,𝑚𝑎𝑥 𝑒𝑔𝑜 )2 2(𝑎𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥 𝑒𝑔𝑜 )∗−𝑣𝑒𝑥𝑡2 2𝑎𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥] *Note that we use the maximum braking for this formulation, differing from [25] 𝑅𝑜𝐼𝑚𝑎𝑖𝑛is the extension of the road topology, with a width 𝛿𝑅𝑜𝐼 ∈[−𝛿𝑙𝑎𝑛𝑒 2,𝛿𝑙𝑎𝑛𝑒 2], with a longitude of 𝑑𝑏𝑟𝑎𝑘𝑒,𝑚𝑎𝑥 𝑟𝑚𝑎𝑖𝑛 . For roads incorporating to 𝑟𝑚𝑎𝑖𝑛 , 𝑟𝑖𝑛𝑡 1,𝑟𝑖𝑛𝑡 2… 𝑟𝑖𝑛𝑡 𝑛, with an incorporation angle 𝛾𝑛, and a speed limitation of 𝑉𝐿𝐼𝑀𝐼𝑇 𝑛, with a region of interest of 𝑅𝑜𝐼𝑛, and a time for the ego vehicle to cross safely the intersection of 𝜌𝑐𝑟𝑜𝑠𝑠 𝑟𝑖𝑛𝑡 𝑛, and intersection point of roads 𝜉𝑠𝑎𝑓𝑒 𝑛, the region of interest was described as an extension of the road topology with the same width of the lane, 𝛿𝑅𝑜𝐼𝑛∈[−𝛿𝑙𝑎𝑛𝑒𝑛 2,𝛿𝑙𝑎𝑛𝑒𝑛 2], longitude from the intersection to the look ahead needed distance with orientation 𝛾𝑛, 𝑅𝑜𝐼𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑛=[0,𝑑𝑐𝑟𝑜𝑠𝑠 𝑟𝑖𝑛𝑡 𝑛], defining such distance as: 𝑑𝑐𝑟𝑜𝑠𝑠 𝑟𝑖𝑛𝑡 𝑛=𝑉𝐿𝐼𝑀𝐼𝑇 𝑟𝑖𝑛𝑡 𝑛𝜌𝑐𝑟𝑜𝑠𝑠 𝑟𝑖𝑛𝑡 𝑛 With 𝜌𝑐𝑟𝑜𝑠𝑠 𝑟𝑖𝑛𝑡 𝑛 as function of the speed of the ego vehicle, 𝑣𝑜𝑒𝑔𝑜, and the distance to the first space in 𝑟𝑚𝑎𝑖𝑛 for the ego car to be safe after crossing the intersection or incorporation, 𝑑𝑠𝑎𝑓𝑒 𝑝𝑙𝑎𝑐𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 : 𝜌𝑐𝑟𝑜𝑠𝑠 𝑟𝑖𝑛𝑡 𝑛= 𝑑𝑠𝑎𝑓𝑒 𝑝𝑙𝑎𝑐𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑣𝑜𝑒𝑔𝑜 ⁄ As a result for our region of interest we merged all the results hitherto 𝑍≡𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 =[𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑚𝑎𝑖𝑛∪ 𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑛]⇔∃𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑚𝑎𝑖𝑛 ∋𝜉𝑐𝑟𝑜𝑠𝑠 𝑛∀𝑛 with 𝑛 being all the roads that have an incorporation or intersection on main road in the nearby states. If ∄𝑅𝑜𝐼𝑚𝑎𝑖𝑛∩𝜉𝑐𝑟𝑜𝑠𝑠 𝑛∀𝑛⇒𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑐𝑢𝑟𝑟𝑒𝑛𝑡≡𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑚𝑎𝑖𝑛. For the feasible RoI, we need to check the occlusions from the different dynamics ODD that block the perception for the sensor, for a sensor 𝑖 with a range 𝜎𝑟𝑎𝑛𝑔𝑒,𝑚𝑎𝑥 𝑖 exposed to some occlusions from different natures,𝜎𝑜𝑐𝑐𝑙𝑢𝑠𝑖𝑜𝑛𝑠 𝑖, the true range for that sensor is 𝜎𝑟𝑎𝑛𝑔𝑒,𝑟𝑒𝑎𝑙 𝑖=𝜎𝑟𝑎𝑛𝑔𝑒,𝑚𝑎𝑥 𝑖−𝜎𝑜𝑐𝑐𝑙𝑢𝑠𝑖𝑜𝑛𝑠 𝑖. Thus, 𝜎𝑟𝑎𝑛𝑔𝑒,𝑟𝑒𝑎𝑙 𝑒𝑔𝑜 =⋃𝜎𝑟𝑎𝑛𝑔𝑒,𝑟𝑒𝑎𝑙 𝑖∀𝑠𝑒𝑛𝑠𝑜𝑟 𝑖𝑛 𝑡ℎ𝑒 𝑒𝑔𝑜 𝑣𝑒ℎ𝑖𝑐𝑙𝑒is the addition of all the perception of the ego vehicle. Matlab Simulation 24 24 Figure 19 Seeking with few ahead vision As you can see in the previous image, with a close target in the trajectory, our movement becomes unstable and takes longer for the trajectory to adapt to the route. Also reduces the smoothness of the movement. By choosing the right parameters a smooth and stable path can be followed through the route, and depending on the situation those parameters can be change to optimize the behavior. In the following picture there a current speed linear variable was used to keep the route. Figure 20 Seeking target looking ahead With that all the bases to follow a path between a starting point to a destination, and the way to do it is by using the equation of Reynolds for autonomous particles. Looking in the next picture all the important values can be seen with a graphic demonstration. From those values, performing some operations, the acceleration can be calculated. 𝑇𝑎𝑟𝑔𝑒𝑡 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 =𝑂𝑃 󰇍 󰇍 󰇍 󰇍 󰇍 +𝑉𝑀𝐴𝑋∗𝑇𝑆𝑎𝑚𝑝𝑙𝑒∗𝑡𝑟𝑎𝑗 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 |𝑡𝑟𝑎𝑗 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 | 𝑑𝑖𝑟𝑑𝑒𝑠𝑖𝑟𝑒𝑑 𝑒𝑔𝑜 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 =𝑇𝑎𝑟𝑔𝑒𝑡 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 −𝑝𝑒𝑔𝑜 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 𝑣𝑑𝑒𝑠𝑖𝑟𝑒𝑑 𝑒𝑔𝑜 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 =𝑉𝑀𝐴𝑋∗𝑑𝑖𝑟𝑑𝑒𝑠𝑖𝑟𝑒𝑑 𝑒𝑔𝑜 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 𝑆𝑡𝑒𝑒𝑟 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 = 𝑣𝑑𝑒𝑠𝑖𝑟𝑒𝑑 𝑒𝑔𝑜 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 −𝑣𝑒𝑔𝑜 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 25 Safety check in critical safety scenario for self-driving vehicles 𝐴𝑐𝑐𝑒𝑙𝑒𝑟𝑎𝑡𝑖𝑜𝑛 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 =𝑇𝑜𝑟𝑞𝑢𝑒𝑀𝐴𝑋∗𝑆𝑡𝑒𝑒𝑟 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 |𝑆𝑡𝑒𝑒𝑟 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 󰇍 | With these simples operations we can calculate the acceleration to apply for the vehicle. In the last operation we use a parameter called maximum torque for the vehicle. With that we can restrict the speed increase and the running of the car. This is not applicable for cars when it comes to lateral sliding, but with a simple restriction in the code we can keep this movement as one similar for vehicles. If we choose a big value for the maximum torque, we would get an ideal movement. Figure 21 Ideal target seeking In the previous picture, a vehicle with an starting velocity, represented by the blue arrow in the opposite direction than the destination, with a big value of maximum torque can turn with a really obtuse angle, unlike a vehicle would do it. To accurate this movement to one more realistic, an experimental parameter was chosen by submitting the vehicle to several experiment. Even when in a static picture may seem that the following picture turn is more obtuse, the speed is lower in the moment previous to the turning. Figure 22 Realistic target seeking From this point, the next step to be performed by the car is not to follow only a line from to points, but to keep going through several waypoints, and as we cross them, change for the next one. One of the main things to take into account here is that this technique is good to approach to the surroundings of the point, but after we reach the point, there is nothing that make car stop in that point. For the previous experiments the simulation Matlab Simulation 26 26 was stopped when reaching some area nearby the destination point, but when working with different points, these event may trigger instabilities in the corners, as it can be seen in Figure 23 first try: Total failure. Figure 23 first try: Total failure To avoid this phenomenon, once our prediction in the trajectory, target, have reached the destination point of the first stretch of the route, we start looking to the next stretch of the route, so the car star turning before reaching the corner and not once it is on the corner, which indices oscillations on the movement. Another arrangement done for the improvement of this movement is to reduce the speed near the corners, as a human driver would do, and not to take turn at maximum speed, but at 25% of it. Once we have one particle/vehicle capable of going through different waypoints, instead of choosing random spots in a blank space and make the vehicle to run them, we decided to use one map from the surroundings of the Oomika Station, in Ibaraki, Japan. But to use the map in Matlab as a series of points there is a need for another package from internet. Using the Open Source Package of functions from OpenStreetMap, the following map could be turned into various nodes, being aware of cross and lane directions, obtaining a route. Figure 24 Map of Oomika Station surrounds From those points, after doing a transformation to be used by the vehicle (as the points were measured in latitude and longitude, some of the accuracy was lost in calculations by the declaration of the variables), the vehicle followed the route, represented by the discontinuous red line, moving as show in the black line. The sample times of these experiments were no chosen accordingly to the real one needed. 27 Safety check in critical safety scenario for self-driving vehicles Figure 25 Real map based consecutive target seeking From the same map, other routes may be taken for different vehicles in a scenario. When setting up the scenario, all the car or vehicles must have a behavior determined, but it does not have to be the same. Therefore, two demeanor have been chosen. One of them, aggressive behavior, is avoiding bumping into other actors in the scene by turning the car to the opposite side of the incoming car. The other one, passive behavior, will yield when crossing with another actor. All the vehicles in the scene are suppose as all-knowing actors, hence, they all know at all tie when are the rest of vehicles. As showed in the Figure 26 Multiple vehicle path following. Steering Behavior. a test was run, were all the vehicles were repeating in loops their routes. All the routes have a various length, creating differences in the scenario in each loop. This would repeat for a fixed amount of time for the purpose of the testing. 4 Figure 26 Multiple vehicle path following. Steering Behavior. In this experiment, the vehicles 1 and 5 have to cross path, therefore both of them are using aggressive behavior as there is only one lane to drive through. The rest of the vehicles are using the passive behavior. For reality purpose, there is an invisible area surrounding the vehicles that acts as their bodies, and if two of them get in touch, a collision error pops up and stop the reproduction of the experiment. Without this feature is not possible to debug problems in the behavior as there is no way to check if two vehicles have bumped into each other. 4 The whole video of this experiment is disposable in the attached files by the name of “”. Matlab Simulation 28 28 Besides that experiment focused on intersections, another test were run in diverse scenarios. For example, in Figure 27 Incorporation experiment. Steering Behavior an incorporation to a highway from a secondary road, including some time to check the lane to see if there is any car coming toward our designed spot and some prediction of the car to see if there is time to incorporate the road when a car is coming slow or from a far distance. In these experiments, we will be focusing on a single car, denominated ego vehicle, and the other actors in the scene will be attach to a fixed route. In the same way we did in the previous experiment, in this experiment also all the vehicles return to their initial state after reaching the destination point, creating anomalies in the original situation that allows us to prove extra situations. Figure 27 Incorporation experiment. Steering Behavior In this experiment there are several car going through a highway (1, 2, 4 and 5) and there is road that incorporates in that highway. From that secondary road there is our ego vehicle coming. Our ego vehicle is going to slow down as he is approaching the incorporation to the road. If when doing so captures any car incoming from the right that might enter the cross before him, will stop. To model the lack of visibility of the sensor, a variable was created. This variable was increase as close the car were to the corner. In case this variable was not beyond a certain threshold the car will not incorporate into the main road. In this experiments we can see how the car stops for the first two loops (upper images) as there is no space enough, but on third one it enters between the vehicles 4 and 5 (lower image). In case this maneuver was considered too risky, by changing the threshold and prediction of other vehicles, it could be reduce the aggressiveness of condition of our ego vehicle. With this it finish the first part of the structure in the development of the program. Until now we have manage to obtain an autonomous particle seeking a target, which is able to evaluate the situation in the surroundings and react in consequence in a simple basis. Starting from that a much more developed and critical safety environment focus system can be researched. 29 Safety check in critical safety scenario for self-driving vehicles 6.2 Autonomous Driving System Toolbox For deeper studies in this area a special Toolbox of Matlab is going to be used. Autonomous Driving System Toolbox is specially designed for self-driving vehicle, proving a wide range of scenarios thank to all the utilities, functions and applications. For further information check the previous section were all the links to the toolbox are placed. The first thing were this toolbox come in very handy is when it comes to scenario making. Along with all the functions included in the toolbox there is a scenario maker with a series of features in it. In the following picture, Figure 28 ADST: Menu Bar, we can see all the objects and actions available in the App. It allows us to design roads and different actors, include sensors on those actors, run simulations on the own application and export data. The exportation data may be two different nonexclusive sets, one of them give us a script in Matlab with the declaration of all the actors, objects, roads and trajectories defined in the application. The other set can only be obtained after adding some sensors to the ego vehicle and running one simulation. After that all the data taken from the sensors can be exported to be used for later processing. It also allow to change some stings of the simulation such as sample time or simulation time among others. It can also be executed stopping in each sample time to observe more in detail Figure 28 ADST: Menu Bar As for the actors is not only made by one ego vehicle, there are various kind of vehicles and other objects to be set in the scenario, granting more scenarios to be created with miscellaneous characteristics. Figure 29 ADST: Actor creator Menu All of the objects shown in Figure 29 ADST: Actor creator Menu can be modified in several ways. In nonmoving objects such as barrier you can only modify the shape. In other objects alongside with the shape you can also change its velocity, trajectory, starting point and radar and camera errors when being detected. Apart from this you can also create your own actor with unique properties. To create roads and trajectories for actors the same process is carried out. You select the waypoints or road centers and then an interpolation among them is done to create a smooth line to be followed for the vehicle. All the position, velocity and space values in this simulator uses a 3D axis, but to simplify all control signals, in the test only x and y axis were used (as explained in the section Reynolds Steering behavior). When choosing Matlab Simulation 30 30 the same points for both the road and the ego vehicle, the trajectory of the car fits with the road center for the whole path. Figure 30 ADST: Trajectory Definition Once the car and other objects are settled, with their trajectories define and road is fit for the path they are going to follow (in this simulator there is no possibility of adding some control to the vehicles such as lane keeping algorithm, hence the trajectory needs to be adjusted to fit in the road center, otherwise experiments would look strange) it is needed to place and define the parameters of the sensors. In this Toolbox there are two types of sensors accessible: radars and camera. The camera works as a real camera with the data from the experiments and situates the objects in the middle of it. Techniques for the camera can be also be used in real camera data to detect objects and obtain bird eye plot. The radars in turn detects in discrete ranges among a certain angle. Thereupon same object may obtain multiple detections from a single radar, each one separate from the others a certain minimum angle that the radar is capable to disclose. Each sensor can be place in any part of the car (even some part outside the range of the own car) pointing to any direction desired. Hence, azimuth, yaw, maximum angle, range and position can be changed at ease. In Figure 31 ADST: Sensor placement a basic two front radar and one camera structure can be seen, and can be tested in early stages of this simulations. 31 Safety check in critical safety scenario for self-driving vehicles Figure 31 ADST: Sensor placement With those sensors placed as shown, a scenario with other actors inside a road can be structured, to start testing how data are collected by those sensors. In Figure 32 ADST: Sensor Data Acquaintance there is a simple test being done with this sensor configuration. Two vehicles, ego vehicle and a fixed-trajectory actor, were arranged in a circular loop road, having the ego vehicle running in the opposite direction than the other vehicle. Figure 32 ADST: Sensor Data Acquaintance In the figure there are 8 detections from the radar and one made by the camera. From now on, all the experiments will use the same color pattern, all the detections from radars and range of them will be painted by a light red and for cameras will be painted by a navy blue. The camera is only one detection allocated in the middle of the object as it is seen by the ego vehicle and does not give us any other information but the coordinates of that objects at first. It can be changed to be able to detect type of object, lanes boundaries and also speed. For radars, in the other hand, there are multiple detections and there are lanes coming from them. Those lanes are graphic representations of the speed of those points by the egocartesian axis. As we have Matlab Simulation 32 32 multiple data from the same object but the radar repeatability is not reliable, we can conciliate all data from sensors to have one more accurate acquisition. With all this information, we are ready to start developing a fully equipped environment for test and behavior planning. Starting for the easiest one, to increase gradually its difficult. We have chosen highway driving as a starting point as it is reduced the directions from where the danger may come from. Also, to reduce difficulties in programming the behavior a straight lane have been chose so the first steps were easy to reach. With this a full directional sensor equipment was allocated in the ego vehicle to start the testing. Exporting all this information to a script in Matlab made possible to start working and modify some aspects in order to create more experiments to test the algorithm in. In Figure 33 ADST: Representation of experiments you can see a preview of how the sensor were placed and how the experiments were represented during testing and debugging of the code. Figure 33 ADST: Representation of experiments 33 Safety check in critical safety scenario for self-driving vehicles 6.3 Highways In the beginning of the experiments, a double lane straight road was designed as to move only in one axe in the absolute reference system, so all the movements were easier to program, as to lane change and acceleration and brake. In this moment one first challenge, not problem but lack of feature, was found. In the ADAS Toolbox there were no feature designed for collision among objects, hence, while doing the experiments everything could happen and the program will still keep running as usual, with the only exception of some measures in the sensors. While the sensors were inside other object a warning was displayed by Matlab but did not stop the execution of the code. Thus, a collision detection function was added to the code. As the only object without a fixed trajectory in the scene is the ego vehicle, only the collisions between this actor and the other objects have to be taken into account, as if the other actors collapsed is not a code problem but a scenario bad design. In a first instance an exhaustive point by point checker was designed but the simulation became slow as there were too many comparisons to be done in each sample time. Therefore a simpler check was done. From the center of the vehicle, a radius was traced with the highest distance inside the vehicle. Inside that area there were check all the position of the other objects to see if there were inside. One problem with this coding was the lateral crash, as they were obtained even when the ego car and the other object did not touch. For that, the radius were cropped to a certain percentage. This give us the collision warning that stop the running of the code for the situations, allowing a faster and more accurate debug of the code. As it can be seen in Figure 34 ADST: Collision detection, keeping the trajectory of the ego car and placing some other objects in that trajectory it can checked that the code worked for collisions from different angles and in various heights. Also, to determine what kind of crash happened in the simulation, as sometimes can be varied reasons for a crash, also the class of the object that enter the collision radius is also displayed. This thing too helps to explain crash from blind spot as it happened in the right top picture in that figure. One of the main changes made in the code while performing this development is the change of the sensor disposition. Up until now, the sensor were placed as explained in the section Autonomous Driving System Figure 34 ADST: Collision detection Matlab Simulation 40 40 Figure 42 Big intersection. RoI feasible Y/N Therefore, a new method was implemented. This new method, how it was implemented and the characteristics it adds to the program functions are discussed in the next section, Sensor Occlusion. 41 Safety check in critical safety scenario for self-driving vehicles 6.5 Sensor Occlusion The previous method was not working for a real implementation, therefore, a new method had to be develop. In this case, as we want to see the occlusions happening inside our sensor range, and our sensor are reduced to just a 360 degrees radar and a camera, our initial range is a circle with the area that the LiDAR covers. When it comes to create this implementation with a real LiDAR obtaining data it would be as simple, a priori, as joining all the point cloud obtained from the sensor by layers. By doing so we will obtain a 3D map of occlusions that could be overlapped on the map so it would obtain a range of sensors capacity. As for the simulation, as we do not obtain a LiDAR information but radars data, there is a need to simulate additionally the occlusion from the sensors. One first idea that may come to your mind is to apply the same reasoning that was carried out with the LIDAR data. To create an area that surrounds the vehicle in a circular shape, with the radius of the sensor range. Then connect all those points with the ones obtained from the sensors. But as the sensor data is not recorded by some order, yet all the information comes and is saved as soon as it is received, it is not possible to carry out this technique. Another way to perform this simulation of the occlusion from the sensors is the one that end up being implemented. In this technique we make a circular shape and place all the points obtained from the sensors. Starting from the center of the circle, which is the same position where the ego vehicle is positioned, we obtain the direction of the vector to each one of the detections. Then, we scale that vector to point in that direction up till the end of the circle. In the ending point, the one laying on the circumference, we trace a perpendicular line to the direction from the center to the detection. In that perpendicular line we place two points separated by a distance δ. To determine δ we used a simple linear equation. The longer is the distance from the circumference to the detection, the bigger the value of δ becomes. Making a triangle with these two points separated δ with the detection, we obtain the blind side made by that detection. In the next figure, Figure 44 Object occlusion measurement, there is a representation of this effect. Figure 43 Sensor occlusion basis. Matlab Within a recreation of the range of the sensor in a circular shape, we place the ego vehicle,𝐸, the radar detection, 𝑅, and the projection of this detection at the end of the circumference, 𝑃; and with a sensor resolution of 2𝛿 we simulate a square shaped object in 𝑅 with a side size of 2𝛿 and facing 𝐸. We calculate 𝛿𝑝𝑟𝑜𝑗 for both directions. 𝛼=tan(𝛿 𝐸𝑅 󰇍 󰇍 󰇍 󰇍 󰇍 )−1 𝛿𝑝𝑟𝑜𝑗 =𝐸𝑃 󰇍 󰇍 󰇍 󰇍 󰇍 tan(𝛼) Matlab Simulation 42 42 Figure 44 Object occlusion measurement. Matlab In the previous figure a red area represent the range of the sensors (LiDAR and camera, both mixed) and the rest of the areas painted in various colors are the occlusion from the sensors. Merging all the areas and subtracting the occlusions from the range of the sensors we obtain the feasible range of the sensors, as it is shown in Figure 45 Sensor capacity after occlusion. This is the results from applying the previous idea into Matlab simulation. Figure 45 Sensor capacity after occlusion. Matlab This figure represents how far the sensor are able to see, starting with the assumption that we have a circular 43 Safety check in critical safety scenario for self-driving vehicles ideal surface as the capacity of the sensor. With an ideal sensor range 𝜎𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 from where we subtract the occlusions, 𝜎𝑜𝑐𝑐𝑙𝑢𝑠𝑖𝑜𝑛𝑠 as the area between 𝛿 𝑎𝑛𝑑 𝛿𝑝𝑟𝑜𝑗 we obtain 𝜎𝑟𝑒𝑎𝑙 shown in Figure 45 Sensor capacity after occlusion. Matlab. If we take into reference the first sensor placement we discussed in this research, Figure 31 ADST: Sensor placement, and we merge the areas of occlusion obtained with a circular shape with the area that all the sensors are covering, we can obtain a simple but reality likewise simulation of the occlusion for the objects. In the next figure, Figure 47 T section. Occlusion in sensors, a bird’s eye plot representation with the occlusion of the sensor can be seen, checking how it works during the test. When we have an area that covers all the space that the sensors are able to cover, and we have already obtained the Region of Interest that we need, we can check the percentage of that region that we are able to see. By merging those two areas into the intersection of both of them and comparing both the area of the intersection, Zf, and the area of region we need to check, Z, we can value how sure we are about the environment that surrounds our ego vehicle. Once we reached this point, the last step is to test this algorithm in the critical safety scenario. We designed a T section of a road, where a high speed road incorporation is need to be done. Besides that initial problem, we added obstacles in both sides blocking the sight for the road. As a human we would act in the following way, we would reduce speed as we reach the intersection, until we stop completely the car, and as we see there is not enough visibility, we would move forward in small steps until we find ourselves with enough visibility to decide to incorporate. We designed a code that would decide the same as the human should without declaring step by step how to proceed. We declared a vehicle that arrives to the intersection from the blind side right in the moment to crash with our ego vehicle if he moves reckless. The result of the experiments before adding the feasible region of interest can be seen in Figure 48 T section. Crash due to lack of visibility. Figure 47 T section. Occlusion in sensors Figure 46 Small intersection. RoI feasible Matlab Simulation 44 44 After implementing the code for the feasible RoI, we need to establish a threshold. When the car travels with a value of their surroundings below that threshold will slow down its speed so it have more time to react for unforeseen situations. Also when decision making time such as intersections or overtaking in highways, when the value is under that threshold, smooth movements to acquire more vision would be taken until being sure up to that verge value to make the action. As it can be seen in Figure 49 T section road. RoI Feasible, the ego vehicle is moving slow, 0.6 m/s, as the percentage of RoI feasible is not enough due to the obstacles that are blocking the sight. We calculate the percentage of area visible,𝑅𝑜𝐼%=𝑅𝑜𝐼𝑓𝑒𝑎𝑠𝑖𝑏𝑙𝑒 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑅𝑜𝐼𝑖𝑑𝑒𝑎𝑙 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 =𝑍𝑓 𝑍, and based on that value perform the proper safe reaction for the ego vehicle. We implemented this idea in the critical safety scenario with a simple FSM for decision making, adding a behavior to improve the 𝑅𝑜𝐼% value if there is not enough visibility on the turn. Choosing the correct value for the threshold we manage to make our ego car to wait for the car in the blind spot to cross before taking the intersection, or to go first if there is time enough to merge into the new lane. Figure 49 T section road. RoI Feasible Figure 48 T section. Crash due to lack of visibility 45 Safety check in critical safety scenario for self-driving vehicles Figure 50 Speed and distance check with and without our proposed safety layer implemented. We can see how the accident is prevented by improving situational awareness When using our proposed idea we not only avoided the crash, yet performed a smoother action around an Matlab Simulation 46 46 intersection with low visibility, as we show in Figure 50 and Figure 51. In the following picture we deepened in details explaining the car state in each moment during the simulation. We show that the dangerous situation for the car is avoided by slowing down when visibility is not enough. We also limited the actions on the decision making module based on the value 𝑅𝑜𝐼%, giving a safer and more consistent behavior. Figure 51 Comparison between FSM with and without our proposed idea, in detail. On the first picture there is the result from the Figure 8 with the accident, on the second picture the speed using our safety assessment management. 47 Safety check in critical safety scenario for self-driving vehicles Autoware Implementation 48 48 7 AUTOWARE IMPLEMENTATION Autoware is an open source package for ROS for autonomous vehicles, which includes several packages for the processing of data, motion and mission planning for the movement and several algorithms for obstacle avoidance. ROS can be worked either in Python or C/C++ languages. As a basic of ROS is needed to understand the whole program, a simple explanation of ROS performance is giving ahead. There are several terms required to understand in depth how ROS works, besides the relation among them: - Node: are the backbone of ROS architecture. In ROS there is no one big program that handles all the functions on it, but there are several nodes that connects through topics to perform defined task each one. There is one main core node that is going to be the handler of all the other nodes address and topics links. Each node has to have a unique name so the core node is able to identify it from the list. Each node can be program in different languages and the it has to be built so the program know all the dependencies needed in order to create an executable that will work as one or more node. One of the main differences between the two languages available in ROS is that when programming in Python the executable just point to the .py file so all the changes made in the code does not need to be rebuilt in the program. Just the opposite than with C/C++ where each change has to be applied and rebuilt to create a new executable that will launch a node. All the code in the nodes can be launch more than one time as long as the nodes have different ID. - Packages: is the connection of one or more nodes with other packages and among them. All and each one of the nodes needs to belong to one package. By building all the packages in your projects is the only method to launch the different nodes in a successful way. - Topic: reserved space for nodes to publish messages and receive the data. Instead of sending the information to one of each of the nodes in the system, the message are published in the topics where other nodes can request to obtain that information. Each topic has to define a message type for the topic, so all the nodes can know how the information is being uploaded. Same as with nodes, each topic has to have a distinct name. - Message: are the information published on each topic. One type of message can be published in various topics, yet, each topic must always publish the information in the same type of messages. Inside the own ROS libraries there are several messages types, which are the most common. - Publisher/subscriber: as it was mentioned in the previous terms, each topic has messages published on it. When a node wants to upload information to a topic need to sign up as a publisher in the core node. When a node wants to receive the information from a topic needs to sign up as a subscriber in the core node. Each node can subscribe and publish as many topic as it needs. In the next picture Figure 52 RQT from rerunning experiment, there is a simple representation (carried out by one of the different nodes in ROS libraries called RQT-Graphs). The rounded text box are representing nodes and squared ones represent the topics of the program actually running. Not all the nodes has to be link among others, but may be isolated zones for specific functions, but as long they all connected to the core node, even when the node are not publishing or subscribing, they are going to be placed with this functionality. 49 Safety check in critical safety scenario for self-driving vehicles Figure 52 RQT from rerunning experiment Autoware works on top of this structure and it can be used for different purposes. The tree main purposes that we will be using are to rerun simulation from a ROSbag 5 with data from a previous experiment recorded, simulate the behavior of synthesized experience based on a real map and, lastly, to run a whole autonomous system on a real car. The complexity of the process increases as we move toward the final destination of real life implementation. In Figure 52 RQT from rerunning experiment there is a representation of all nodes while rerunning an experiment, adding some features, later discussed. In Annex A there is a representation of nodes when running a simulation in Autoware environment. The step between this and real-life implementation would be including all the read from the sensors in real time, and publishing this information on the different topics. As a method of visualization for the experiments there is one module of ROS that come in handy, RViz. With RViz there a various type of figures, lines and models that can be represented in a 3D space, with different represent axis. As it is displayed in Figure 53 Autoware graphic representation of experiment. Moriyama ROSBAG., we can see all the rings (layers) from the LiDAR, a model for the car, and object distinction in the image, by varying the color of the points from the LiDAR. This image belongs to one of the maps distributed by the same company from Autoware, and it is created from a place in Japan, Moriyama. From Moriyama ROSbag we do not only get the map, but also a crucial element inside Autoware conception, a vector map. In order to perform the task of the autonomous driving faster, Autoware relays on a vector map loaded offline with the key points from the map, such as lane centers, lane width, light traffic position, stop signals among others. 5 A ROSbag is a type of storage data for ROS environment. All the data from a certain experiment or for a database can be saved in it, and the access it publishing its data in the attached topics. Autoware Implementation 56 56 Figure 59 RoI Feasible. Birds Eye Plot With all this information obtained, and all the nodes already working we need to increase the robustness of the program. When starting the program we need to have a series of Waypoints in a folder, we need to have access to the RoI boundaries files and we need to have a certain amount of data to work. As for simulations if there were fails obtaining one of those data, we just needed to rerun the node and the problem would disappear, but when wanting to implement the code in real life there is no room for those mistakes. For that reason, before starting there is a check of whether all the information were collected or not, and if not, the node would stop its process and not publish any data until it receives all the required data. Other problem may appear from synchronization problems between the polygons, as in some times, the program would try to check the intersection between two polygons that were not completely defined. In those cases just a warning appeared and the program kept running, but to avoid further due with these errors, there were check for both validation of polygon shape and intersection of areas. If one of those were incorrect or the area of intersection were founded as zero percent, there would be no new publication, as it is consider to be a problem of processing, not the presence of uneven cases. By letting the topic unpublished we will keep the last value it was published for other node to read, that could causes some troubles. To avoid reading an outdated information about the feasible RoI, there is a header with time stamps for both RoI and Feasible RoI to check if there is so much delay from real time. As the feasible RoI might not be a variable we need to update that often. Yet, we could change the update statement to a periodically or by request in case the module failed on the work. If the safety check is not working a warning is sent to warn the user that the car is not able to check its surroundings. With the feasible RoI we obtained a quantitative value for the safety validation check, and defined an action contract that improved the safety state of the vehicle. By reducing the speed of the vehicle on those situations we become able to react properly to unforeseen events that may occur. 57 Safety check in critical safety scenario for self-driving vehicles Figure 60 Speed comparison based on feasible RoI values Autoware Implementation 58 58 59 Safety check in critical safety scenario for self-driving vehicles 8 CONCLUSIONS AND FUTURE WORK We developed a novel system that guarantee safety for AV in scenarios where dynamics such as weather conditions or limitations in the sensors can be overcame by measuring the percentage of surroundings can be checked by all the sensors. We proved this idea in several simulations and toolbox (Matlab and Autoware) with highly positive results. We also took the next step and implemented this idea in an autonomous vehicle and tested its results in a controlled scenario with also positive response. This novel idea can be a required step for a level 4 automation in AV. This idea also works as an upper level addition for the machine behavior, allowing it to be implemented in any kind of decision making plan with just a few arrangements. As for future work relating this idea, there are several ways to increase its effectiveness. Firstly it would be mandatory to test this idea in a larger amount of samples, different scenarios to assure the effectiveness of this idea no matter the scenario. This would be a mandatory step in the development of this idea. As for any additional content that can be included in this idea there a different approaches:  A standardization of the region of interest (RoI): To standardize all the region of interest for the whole possibilities in the road will allow this idea to be implemented in all the current state-of-the-art AV behavior. As of the moment multiple lanes scenarios and intersections scenarios are defined, but other unique elements, such round wheels are not yet defined. There is a necessity of adding this definition for this scenario to the idea in order to work in countries that includes this elements in their road infrastructure.  Map reading by the machine: Being able to adapt for all the scenarios includes a necessity for this idea to be able to read maps of the surroundings in order to create an accurate region of interest of the surroundings. This could be done by including a map of the whole country where the system might be tested or, rather, to download the maps from a cloud and being able to turn those into information the machine can check for occlusions.  Other sensor´s occlusion: The project showed the occlusion in sensors such as LiDAR and radar, which works on most the AV vehicles, yet, those are not the only one that can be used for measuring the environment. If we take, for example, Tesla´s car only uses camera for a safe autonomous driving. Therefore, to include a program that can check the occlusion of the camera based on the variables read by itself, will increase the acknowledge of the machine about its surroundings and all the occlusions and limitations happening around it, and therefore, will increase the security of the system itself. Conclusions and Future Work 60 60 61 Safety check in critical safety scenario for self-driving vehicles REFERENCES [] Lemaignan, S., Warnier, M., Sisbot, A., Clodic, A. & Rachid, A. (2017). Artificial cognition for social human–robot interaction: An implementation [] Center for Educational Research and Innovation. (2007) Understanding the Brain: The Birth of a Learning Science. [] Boden, M. A. (1997).Creativity and artificial intelligence. Sussex: School of Cognitive and Computing Sciences, University of Sussex. [] Nothman, J., Ringland, N., Radforda, W., Murphya, T. & Curran, J.R. (2012).Learning multilingual named entity recognition from Wikipedia. [] Serj, M.F., Lavi, B. & Hoff, G. (2018). A Deep Convolutional Neural Network for Lung Cancer Diagnostic [] Verma, M. (2014). Medical Diagnosis using Back Propagation Algorithm in ANN [] Schoettle, B. & Sivak, M. (2014) A survey of public opinion about autonomous and self-driving vehicles in the U.S., the U.K. and Australia. University of Michigan, Transportation Research Institute. [] Alcantarilla, P.F., Bartoli, A. & Davison, A.J. (2012). KAZE Features. [] Andersson, O. & Marquez, S.R. (2016). A comparison of object detection algorithms using unmanipulated testing images Comparing SIFT, KAZE, AKAZE, and ORB. [] Bay, H., Tuytelaars, T. & Gool, L.V. (2008). SURF: Speed Up Robust Features. [] Ahmed Khan Tareen, S. & Saleem, Z. (2018). A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. International Conference on Computing, Mathematics and Engineering Technologies iCoMET 2018. [] Krizhevsky, A. & Sutskever, I. (2012). ImageNet Classification with Deep Convolutional Neural Networks. [] Kumar, C., Chawla, K. & Arora, S. (2018). The comparison between various object detection algorithms. Delhi: Maharaja Agrasen Institute of Technology. [] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Jia, Y.... Rabinovich, A. (2014). Going Deeper with Convolutions. Michigan: Ann Arbor Magic Leap Inc. & University of North Carolina. [] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. & Berg, A.C. (2016). SSD: Single shot multibox detection. [] Redmon, J. & Farhadi, A. (2018). YOLOv3: An Incremental Improvement. University of Washington. [] Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., ... Murphy, K. (2017). Speed/accuracy trade-offs for modern convolutional object detectors. Google Research. [] Girshick, R. (2015). Fast R-CNN. Microsoft Research. [] Girshick, R., Ren, S., He, K. & Sun, J. (2016). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. [] Dalal, N. & Triggs, B. (2005). Histograms of Oriented Gradients for Human Detection. INRIA [] Pavani, S., Delgado, D. & Frangia, A.F. (2010). Haar-like features with optimally weighted rectangles for rapid object detection. [] Simonyan, K. & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. Visual Geometry Group, [] Lowe, D.G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. Canada: Computer Science Department, University of British Columbia. [] Chao, J., Al-Nuaimi, A., Schroth, G. & Steinbach, E. (2013). Performance Comparison of Various Features Detector-Descriptor Combinations for Content-based Image Retrieval with JPEG-encoded Query Images. Munich: Institute for Media Technology. [] Dai, A.M. & Le, Q.V. (2015).Semi-supervised Sequence Learning. Google Inc. [] Rojas, R. (1996). Neural Networks - A Systematic Introduction. [] Argall, P. S. & R. J. Sica. (2007). Lidar (Laser Radar). The Optics Encyclopedia [] Schwarz, B. (2010). Mapping the world in 3D. Nature Photonics, volume 4, pages 429–430 [] (2018). AR0820AT: CMOS Image Sensor, Datasheet. References 62 62 [] Frost & Sullivan. (2016). LiDAR: Driving the Future of Autonomous Navigation. Analysis of LiDAR technology for advanced safety. [] Wind, Intel. (2015). Autonomous driving: An eye on the road ahead. [] NVIDIA. (2018). GEFORCE RTX 2080 ti. Retrieved from https://www.nvidia.com/en-us/geforce/graphicscards/rtx-2080-ti/ [] NVIDIA. (1999). GEFORCE 256. Retrieved from https://www.nvidia.com/page/geforce256.html [] Brechtel, S., Gindele, T. & Dillmann, R. (2014). Probabilistic Decision-Making under Uncertainty for Autonomous Driving using Continuous POMDP. Humanoids and Intelligence System Laboratories, Karlsruhe Institute of Technology. [] Galceran, E., Cunningham, A.G., Eustice, R.M. & Olson, E. (2017). Multipolicy Decision-Making for Autonomous Driving via Changepoint-based Behavior Prediction: Theory and Experiment. [] Bansal, M., Krizhevsky, A. & Ogale, A. (2018). Chauffeur Net: Learning to Drive by Imitating the Best and Synthesizing the Worst. Waymo Research. [] Heckerman, D. (1995) A tutorial on Learning with Bayesian Networks. Microsoft Research, Advanced Technology Division. [] Web source information: https://ch.mathworks.com/products/matlab.html [] Orzechowski, P.F., Meyer, A. & Lauer, M. (2019) Tackling occlusions & limited sensor range with set-based safety verification. Karlsruhe, Germany [] Schratter, M., Bouton, M., Kochenderfer, M.J. & Watzenig, D. (2019) Pedestrian collision avoidance system for scenarios with occlusions. Stanford University & Graz University of Technology [] Chung, W., Kim, S., Choi, M., Choi, J., Kim, H., Moon, C. & Song, J. (2009) Safe navigation of a mobile robot considering visibility of environment. [] Reynolds, C.W. (1999) Steering Behaviors for Autonomous Characters. Sony Computer Entertainment America [] Autoware papers [] AASHTO (2004). Geometric Design of Highways and Streets. [] “Toyota invest $500 million in Uber for development of self-driving cars” The Japan News Webpage: TMC & Uber, August 28th, 2018. [] “Autonomous taxi trials carrying passengers begin in Tokyo” Kyodo, August 27th, 2018. [] “First self-driving car hub in Japan set for Tokyo's Shinagawa” Nikkei Assian Review Webpage. September 14th,2018. [] “Uber self-driving car crash: Vehicle detected Arizona pedestrian 6 seconds before accident” USA Today Article, Nathan BOMEY, May 24th, 2018. [] “Self-driving cars test won’t need permission to use public, say NPA guidelines” The Japan News Webpage: April 11th, 2016. [] “Driver Shortage in Japan Speeds Up Commercial Use of Self-Driving Tech” Transport Topics Website: August 28th, 2018. [] “Open Innovation for Fully Automated Driving” Japan Government Webpage: Winter 2018. [] “Who’s Winning the Self-Driving Car Race?” Elisabeth BEHRMANN & David WELCH. Bloomerg Hyperdrive [] “Which companies are making driverless cars?” Christina MERCER & Tom MACAULAY. 2019. Techworld Webpage. [] “Japan: Autonomous Driving Patents Rated; American And Japanese Top Competitors” Mondaq Webpage. [] [] Nikkei Asian Review. Website. Online resource available here: “https://asia.nikkei.com/Business/Business-Trends/Self-driving-cars-in-Japan-clear-insurance-hurdle” [] “The economic development of Japan” (Original name: “Tojokoku Nippon no Ayumi: Edo kara Heisei madeno Keizai Hatten”) Kenichi OHNO. 2005, Tokyo. [] “Japan works to launch self-driving car system by 2020” Sean KEANE. June 4th, 2018 [4] “Japan looks to launch driverless car system in Tokyo by 2020” Stanley WHITE. June 4th, 2018 [] IOT for all, Website. “The 5 Autonomous Driving Levels Explained” Isabel HERNER. October 23th, 2017. Online resource available here: “https://www.iotforall.com/5-autonomous-driving-levels-explained/” 63 Safety check in critical safety scenario for self-driving vehicles [] “Self-driving cars hit European speed bump. The future is now, but the EU could find itself left behind.” Joanna PLUCINSKA, Joshua POSANER. August 25th, 2016. [] Allen & Overy. “Autonomous and connected vehicles: navigating legal isues”. 2017. [] “EU motors ahead with rules for self-driving cars Bid to catch up with China and US and become world leader in autonomous vehicles” Peter CAMPBELL. May 14th, 2018 [] “Psychological roadblocks to the adoption of self-driving vehicles” Azim SHARIFF, J.F. BONNEFON, Iyad RAHWAN. September 2017. []“Breaking the psychological barrier to autonomous vehicle adoption” Frederic WEILLER, 15th August, 2018. [] “Trust in driverless cars: Investigating key factors influencing the adoption of driverless cars” Kanwaldeep KAUR, Giselle RAMPERSAD. Journal of Engineering and Technology Management, Volume 48, 2018, Pages 87-96. [] The Electronic Chronicles: “Electric Motor Power and HP Ratings” Ted DILLAR, July 21st, 2014. [] “The cost of keeping a car in Japan” City-cost webpage. Jan 23rd, 2017. Based on (https://www.keikenkyo.or.jp/) [] The World Bank Data. Avalaible here: https://web.archive.org/web/20140209085318/http://data.worldbank.org/indicator/IS.VEH.NVEH.P3?or der=wbapi_data_value0+wbapi_data_value+wbapi_data_value-last&sort=desc [] “Motor Vehicle Statics of Japan” JAMA, 2016. [] Japan census 2015. Data from Website PopulationPyramid. [] “Where are the world’s best metro system?” CNN Travel, Edward FALZON, July 12th, 2017. [] “Statics Brief: Urban Public Transport in the 21st Century” Advancing Public Transport, October 2017. [] “NTT Data, Gunma University to test self-driving ride services on publics road near Yoyosu” Kazuaki NAGATA, September 13th, 2018. Annex A 64 64 ANNEX A RQT Graph from Autoware simulation: Figure 61 RQT from Autoware simulation 65 Safety check in critical safety scenario for self-driving vehicles