Procedural modelling techniques to configure driving serious game scenes

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Procedural Modelling Techniques to Configure Driving Serious Game Scenes Pedro Miguel Cesário Rosa Mestrado Integrado em Engenharia Informática e Computação Supervisor: Rosaldo J. F. Rossetti, PhD (Assistant Professor) July 2015 © Pedro Miguel Cesário Rosa, 2015 Procedural Modelling Techniques to Configure Driving Serious Game Scenes Pedro Miguel Cesário Rosa Mestrado Integrado em Engenharia Informática e Computação Approved in a public examination by the Jury: President: Ana Paula Rocha, PhD (Assistant Professor) External Member: Luis Paulo Reis, PhD (Associate Professor) Supervisor: Rosaldo J. F. Rossetti, PhD (Assistant Professor) ____________________________________________________ 14 of July 2015 Summary This dissertation has the objective of tackling the road safety problem in order to further prevent accidents and casualties for both drivers and pedestrians by creating a tool that is capable of loading real world data from user selected locations and render them in 3 dimensional models for further use in serious driving games. These models are then populated with pedestrians that walk around and the ability to drive a vehicle is given to the user. Also the tool should be flexible enough to allow the users to configure the different conditions such as weather, time of day, rendering options, vehicle damage and pedestrian density, in order to conduct studies on different conditions. The project should also be open source, so anyone can edit it and expand it their own way to suit their needs and conduct specific studies. It features Oculus Rift integration, which further extends the possibility of conducting studies to the human driver by giving the possibility to expand this integration to evaluate the driver’s behaviours. Another important aspect is the possibility to export the entire procedurally generated scene to a 3D file format that can be edited by an external application. By providing such tool for free, not only marks the beginning of an open source world 3D generator, but also a framework capable of allowing multiple different usages, such as conducting studies, building video game scenarios or be used as a learning tool by a driving school for instance. Hopefully this will increase road safety if used carefully as a serious tool. To do so we’ll use the game engine Unity 5 to develop the project, CGIAR-CSI to download elevation data, Google Static Maps for the satellite imagery, OpenStreetMap for the location data and everything else is built inside Unity. Also UnitySlippyMap, which is a world map that works with various tile providers was used and integrated on the context of this project to allow users to select a location inside Unity. Along this document you will find a literature review on the topic, including related work and attempts to do similar projects followed by a comparison between other project and this one. The reader will also find details about development and the system’s architecture as well as deep details about implementation. On the end you can find a few screen shots of the results of this project. Keywords: Procedural Modelling, Driving Simulation, World Locations, Serious Games, Road Safety Resumo Técnicas de Modelação Procedimental para Configurar Cenas de Jogos Sérios de Condução Esta dissertação tem o objetivo de tratar do problema da segurança rodoviária, a fim de evitar mais acidentes e mortes tanto de motoristas como de pedestres através da criação de uma ferramenta que é capaz de carregar dados do mundo real a partir de localizações selecionadas pelo utilizador e transformá-los em modelos tridimensionais para uso posterior em jogos de condução sérios. Estes modelos são então povoados com os pedestres que andam nos passeios e a capacidade de conduzir um veículo é dada ao utilizador. Além disso, a ferramenta deve ser flexível o suficiente para permitir que os utilizadores configurem diferentes condições, tais como o tempo, hora do dia, opções de renderização, os danos do veículo e densidade de pedestres, a fim de realizar estudos em diferentes condições. O projeto também deve ser de código aberto, para que qualquer pessoa pode editá-lo e expandi-lo para atender as suas necessidades e realizar estudos específicos. Possui integração Oculus Rift, que estende ainda mais a possibilidade de realização de estudos para o motorista humano, através da expanção dessa integração para avaliar os comportamentos do motorista. Outro aspeto importante é a possibilidade de exportar toda a cena procedimentalmente gerada para um formato de ficheiro 3D que pode ser editado numa aplicação externa. Ao fornecer esta ferramenta de forma gratuita, não só marca o início de um gerador do mundo em 3D de código aberto, mas também uma ferramenta capaz de permitir diversos usos diferentes, tais como a realização de estudos, a construção de cenários de jogos de vídeo ou ser usado como uma ferramenta de aprendizagem por uma escola de condução. Esperemos que isto seja capaz de aumentar a segurança rodoviária, se usado com cuidado como uma ferramenta séria. Para fazer isto vamos usar o motor de jogo Unity 5 para desenvolver o projeto, CGIAR-CSI para baixar dados de elevação, o Google Static Maps para as imagens de satélite, OpenStreetMap para os dados de localização e tudo o mais é construído dentro do Unity. Também é usado o UnitySlippyMap, que é um mapa do mundo que trabalha com vários fornecedores de “tiles” que foi integrado no contexto deste projeto para permitir que os utilizadores selecionem um local dentro do Unity. Ao longo deste documento, irá encontrar uma revisão da literatura sobre o tema, incluindo o trabalho relacionado e tentativas de fazer projetos semelhantes, seguido de uma comparação entre outros projetos e este. Irá também encontrar detalhes sobre o desenvolvimento e a arquitetura do sistema, bem como detalhes profundos sobre a implementação. No final pode encontrar algumas capturas de ecrã dos resultados deste projeto e a conclusão que refere a satisfação dos objetivos e trabalho futuro. Palavras-chave: Modelação Procedimental, Simulação de Condução, Locais do Mundo, Jogos Sérios, Segurança Rodoviária Acknowledgment I would like to thank Professor Rosaldo Rossetti, my supervisor, for all orientation and help along this dissertation. I also thank him for giving me the freedom to elaborate the main goals of this dissertation along him. This dissertation’s idea began on a conversation with him about investigation tastes. We figured out that we had some in common, so he suggested me to create a dissertation project along him based on our ideas. Being so I could arrange the tools to use as well as add details and ask for possible and useful alterations. I would also like to thank Professor Rui Rodrigues for letting me use his Oculus Rift while being supervised by his personnel. xvi SRTM Satellite Radar Topology Mission SUMO Simulation of Urban Mobility TCP Transmission Control Protocol TraCI Traffic Control Interface UV U and V axis XML EXtensible Markup Language Chapter 1 Introduction Along this document, the reader will find the development of this dissertation step by step and in an understandable and simple way, so the reader can more easily read and follow this dissertation. Every significant step is organized in chapters with subchapters. Each subchapter is a specification of the subject about the chapter it belongs to. This chapter introduces the reader to this project and to which its motivations and main goals are. In the end you can find a small overview on the structure of this document. 1.1 Motivations and Goals Road safety is a very important problem needing further appreciation in what concerns prevention, especially when drivers are distracted and not paying attention to the road. Thus, any help attempting to study human drivers and their behaviour on important aspects will greatly contribute to road safety. In order to tackle this problem we propose a framework capable of loading a real-world location in 3-dimension scenes that will be flexible enough to help engineers as well as practitioners to conduct different kinds of studies. From this framework it is expected that it can import real-world data and render it in a virtual environment in 3-dimensions by means of procedural modelling techniques that will then be populated by pedestrians that walk around and vehicles traversing the streets of a network. When all the location generation is finished this framework should allow the user to drive a virtual vehicle and collect metrics while the user drives around. It is possible to configure such metrics on the main menu. Using these metrics, for instance the time the user is not paying attention to the road should be a great contribution to study human drivers and their behaviour towards improving road safety somehow. Since the scene is going to be procedurally generated, it should Introduction 2 be a good feature to allow the possibility to export the entire scene, so it can be used by an external 3D application. The main contribution of this work is to introduce procedural modelling techniques to speed up the process of setting up driving simulation scenes to be used in serious game environments. Contrary to traditional driving simulation setups, which are generally very expensive to run and maintain, although much more limited in terms of experimentation capability, serious games have been introduced as an important resource to the study of subject players. Having so, this tool allows to save modelling time, because of the use of procedural modelling and to save money, because anyone will be able to run this tool on their own personal computers. 1.2 Structure of the Dissertation Apart the introduction this document contains 4 more chapters. On Chapter 2 you can find the literature review on the subjects related to this dissertation exploring different approaches and theories, as well as explaining concepts and related work. At the end of this chapter you can find the tools that are going to be used, explaining why and their relation. On Chapter 3 there are further details about the project such as the problem to solve and how to, the system architecture and module relation and the work planning. Chapter 4 describes the details about the implementation regarding algorithms with example diagrams and included features and how they relate with each other as well as why they are needed and how they interact with the user. On Chapter 5, we can see the achieved and completed goals, results and future work, including possible enhancements and extensions. Chapter 2 Literature Review This chapter contains a few descriptions about the current state of the art. There are different approaches of the same problem by different authors. Starting by the root of this dissertation, the literature review starts by describing computer technologies that enhance user experience, covering driving simulators and their ultimate software relating them to their usages and peripherals that contribute to the immersion of the user. After, there will be covered traffic simulators and their types so the reader can have a notion on what they are and where to apply. There is also a coverage of procedural modelling applications before explaining how it happens so the reader can understand the possibilities before knowing what is behind them. There are also descriptions of the main applications of procedural modelling as well as existing tools to generate any kind of procedural mesh. In the end are presented the tools that will be used to elaborate this project and why they were chosen. 2.1 Game Engines A game engine [1] is a software frame designed to create and develop video games. They provide rendering engines for both 2D and 3D graphics, a physics engine, collision detection algorithms, that might be included with the physics engine, collision response system, sound manipulation, multiple language scripting, animation engine, AI algorithms, networking capabilities, usually any of those are found in the language libraries, a good memory management, enables threading and a scene graph. Using a game engine can reduce a lot the development time and cost, because the main core of the game is already implemented by the game engine’s package, which mean that allows the creation of multiple game types as well as exporting them to different platforms with just a click. Literature Review 4 So, a game engine is a tool that integrates multiple different engines (render, physics and so on) in a single tool. There are a lot of game engines to choose from. Some of them only have a 2D rendering system, others have just the 3D and others have both. Different game engines often have different characteristics, performances, pros, cons and ease of use. The most evolved game engines are usually paid, but there are a few that are free, or at least have a free license like Unity3D. 2.2 Simulation Engines A simulation engine can be almost any software that tries to recreate something that happens in real life, usually physically quantified. They are also very specific, providing only the tools meant to simulate a certain range of conditions, for instance a driving simulator is not meant to simulate water behaviour, but is meant to accurately simulate a vehicle’s behaviour. That’s why they are usually integrated into game engines, so that it can simulate multiple different and distinct behaviours. Having so, the classification on what can be a simulation engine or not is quite big. There are thousands of software tools that recreate real life experiences. 2.3 Physics Engine A physics engine [2] is a computer software that provides an approximate simulation of many physical systems, such as gravity, rigid body, soft body, fluid dynamics, collision detection and collision response. Having so they often integrate part of a game engine as a middleware to run the game with a physical simulation in real time. They have use beyond game engines, such as realistic simulation in serious software to study bridges, car deformation, buildings, tyre models, suspension models and the list goes on. Basically, any kind of real life that can be physically quantified can be done computationally with a physics engine. The most used physics engines are Havok and PhysX 2.3.1 PhysX PhysX is a scalable multi-platform game physics solution supporting a wide range of devices, from smartphones to high-end multicore CPUs and GPUs. PhysX is already integrated into some of the most popular game engines, e.g. UE3/UE4. PhysX also enables simulation - driven effects like Clothing, Destruction and Particles[3]. Literature Review 5 2.3.2 Havok Havok's modular suite of tools and technologies put power in the hands of creators, making sure they can reach new standards of believability and interactivity, while mitigating the overall cost and risks associated with creating today's leading video games and movies. Fully multithreaded and cross-platform optimized, Havok’s technologies offer full support for leading game platforms[4]. 2.4 Render Engine A render engine [5] generates an image from a 2D or 3D model, by means of computer programs and algorithms. A scene file, also called scene graph contains object in a strictly defined language, that contains geometry, viewpoint, textures, lights, and shading information, that describes the scene. The renderer loads that information and calculates every scene polygon, with respective lighting and textures. Some render engines also have a few algorithms that recreate light behaviour. One example is RayTrace, where the light is calculated through a ray that is casted from a pixel of the screen into the scene, hitting and reflecting on objects until its energy is near zero. This algorithm can create extremely realistic images. A good simulation of the light can be used to conduct studies on object materials or even create after effects on a movie. 2.5 Driving Simulators A driving simulator is a very refined and complex system that recreates a real vehicle driving experience in a computerized environment that uses a graphical library to represent the graphical environment and code that is divided in a few modules to simulate the behaviours. Some of this modules may act inside a general physics engine such as the tire module that recreates the behaviour of a real tire, suspensions, engine and braking system, real-time telemetry that couples all the information and displays in a graphical way how the parameters are working, visualization module and so on. The content of a driving simulator could be a thesis on its own[6]. There are also three types of driving simulators. One is for training, the other is for entertainment and the third for research. All can be used to analyse people behaviour and create statistics on certain conditions[7]. Literature Review 6 2.5.1 Entertainment This kind of driving simulator is meant for amusement, to present the player a challenge on beating a lap time or following a car or even just driving around on a big fantasy city making jumps and escaping the police[8]. Diving simulators for entertainment should not be confused as just driving or racing games. Nowadays there are a few platform dominant driving simulators. Although they are video games, they recreate real environments and the car’s behaviour so well that they are used by real race drivers as a training tool at the comfort of their homes. This kind of driving simulators are widely commercialized and sold on an available disk to anyone. 2.5.2 Research A driving simulator can replicate millions of controlled and random situations to challenge the user to test his abilities behind the wheel as well as his reaction to a certain situation. Being so, it can have a massive impact on road safety by conducting studies on real drivers and retrieve information about their reactions. There are hundreds of companies and industries that use these simulators. One good example are the vehicle manufacturers, which build and recreate the planned car on the simulation in order to test it so it can be approved for the production line. By doing so they can measure and analyse the car’s behaviour and crash test safety in many possible conditions in a cheap, fast and safe way. Another example is their usage on civil engineering, where a specific road can be tested before making any changes to the real area. Instead they alter the environment on the simulation and study how the driver reacts to the changes made [9]. 2.5.3 Training Training driving simulators are really serious simulators that can’t be considered video games, they are very expensive and meant to just one purpose, to train a certain topic, offering the user a precise goal. That’s why they are used on driving schools, police academies and so on. As we can see they are mainly used by entities that require serious and sometimes flat-out driving, as well as data retrieval and person behaviour analysis. The main difference between the entertainment simulators is that in some cases this simulators are implemented on real cars that have their wheels attached to hydraulics that react to the simulation, the steering wheel, pedals and gear box are all connected to the simulator and in front of them a big curved screen. Literature Review 7 This simulators are mainly for training and behaviour analysis to collect data on how people react in some driving situations[10]. 2.5.4 The Latest Driving Simulators The following subsections describe a few of the latest driving simulators. Although there are a lot more, these are just an example of the very best in the market. IRacing is the most advanced driving simulator on PC. In what comes to graphics it might not be the most beautiful game, but it sure does is perfectly when it comes to driving. Players can experience an extremely realistic and detailed racetrack, modelled by professionals with data acquired through location laser scanning, which means that every bump on the real world is also on the game’s racetrack, as well as an extremely realistic vehicle behaviour and damage system [11]. It is available to anyone, although it is paid. This racing game simulates so well the real experience, that real race drivers use it to train. “iRacing is the most modern racing simulation ever created. Every inch of every track is modelled perfectly. I’ve used iRacing to learn new courses such as Virginia International Raceway, or to keep the rust off at tracks such as Infineon Raceway. For the hardcore sim racer, this is your dream simulation. For the actual real-world racer, this is your ‘at-home’ test vehicle. Acclimate yourself with a new course or learn something new on an old favourite. Even a rookie Figure 1iRacing gameplay [83] Literature Review 8 in the sim racing world can get up to speed fast within the iRacing system” – says Dale Earnhardt, Jr. 2-time NASCAR Nationwide Series Champion (1998, 1999)[12]. Project CARS is the most recent racing game presenting innovative features. One of them is the capability to simulate a race day on a certain date, this is, the game can retrieve weather information [13], event type, time of day and the cars that raced that day, from a database recreating those conditions in the actual gameplay. The game also has an extremely precise physics engine, including an extremely advanced tire model [14], suspension model, car handling, it recreates almost any possible racing conditions, such as heavy storm at night or day and any other endless combinations. The car also behaves differently depending on the track’s condition, rain, haze, water poodles, small debris affect the vehicle, forcing the player to adopt a different racing strategy. This game does not only simulates a vehicle’s handling on a certain weather, it also simulates a real driver’s career making the player feel like a real driver, by reading e-mails, signing contracts and moving on up to the top classes [15]. Figure 2 - Project CARS Gameplay [84] Literature Review 9 City Car Driving was designed to help users feel they are actually driving a car in a big city or in a country under varying conditions. It is also a traffic simulator, because the vehicles behave themselves in a very realistic way respecting each other and real traffic rules. Some AI vehicles are more aggressive than others, they overtake, they have a target destination, they sometimes crash and the other vehicles have to respond to that situation. This simulator is specially meant to be played with a gaming steering wheel, it is available to anyone, being an exceptional training tool for people who are attending driving lessons or even for regular day drivers, since it allows the training of the basic handling and feeling of a car, implements road rules such as road signals, simulates driving on different conditions of weather and time of day, it allows users to train parking manoeuvres and a wide set of situations by defining specific goals for the user to achieve [16]. Although City Car Driving is both a driving and traffic simulator some people use it as a game, because it is possible to turn off traffic rules and drive flat-out just for amusement. Comparing to SUMO, City Car Driving has ad-hoc well defined cities and traffic routes, so the AI does not need to adapt to the current scene because they are already programmed to behave properly in it. SUMO on the other hand is able to load any kind of road network, which makes it possible to go to OpenStreetMap and download a road network and then import it to SUMO [17]. So the driver agents have to adapt to the current conditions of the road, the traffic ahead and sideways. Figure 3 - City Car Driving [85] Literature Review 16 This model describes the traffic entities at a higher level of detail, but their behaviour and interactions are presented at a lesser level. It shows information about traffic density, flow, velocity, average velocity per individual, the number of vehicles on a lane, but never information about a specific individual, so we can say that a vehicle is on a road, but we don’t know on which lane, it’s speed, it’s direction, its own information, and this is because the vehicles are clustered in groups. The types of groups vary from simulator to simulator and they all have their strengths and weaknesses. The main application of the mesoscopic model is where the detail of the microscopic model is needed, but not all of it. So instead of having a well detailed and computationally heavy road network on a microscopic simulator, the mesoscopic simulators [29] can be used. 2.6.4 Hybrid A hybrid traffic model is a combination of other models. It combines two or more models into a single model with the advantages and disadvantages of each, but with special focus on the limits of each one individually in order to take maximum profit of each model. The most used hybrid simulators are mesoscopic-microscopic simulators that allow detailed microscopic simulation of specific areas of interest while simulating the remaining areas in a lesser level of detail, on a mesoscopic level. This approach comes in handy when there is the need to study a very large road network that is very heavy for the microscopic traffic simulator and requires constant calibration. For instance, if we have 3 districts in a row and we want to study the traffic behaviour in the central one, both the left and right side districts can be simulated on a mesoscopic detail, while the central one can be simulated in a microscopic detail, because it is the main focus and the other districts don’t matter so much in a microscopic way [30]. 2.6.5 Nanoscopic A nanoscopic traffic simulator is even more detailed than a microscopic simulator. It is only capable of supporting small road networks and represent them with a huge amount of detail and information. It is the most suitable model to simulate accidents due to its large level of detail [31]. It can also be integrated with other traffic simulators. Since this is just a model it can be programmed separately from the traffic simulators. These models can be applied in building a nanoscopic simulation based on the four wheels of a car taking into account angular velocity, drag, traction and slip, for instance. Having this, the microscopic simulator can create cars that integrate nanoscopic model making them acting as desired [32]. Literature Review 17 2.7 Procedural Modelling Procedural modelling is the name for a computer graphics techniques that creates a 3D model or texture by means of a computational procedure that implements a computer graphics algorithm to generate a model. This is usually performed in runtime of an application or done a priori to generate a model that is stored for future use. The resulting model depends on the inputs of the procedure, on the sets of rules the algorithm obeys and also on the goal of the model [33]. The biggest advantage of procedural modelling is the fact that it is a procedure that generates the content with no human intervention and, if needed, in runtime. Being so, it is possible to generate an endless terrain that keeps being created as the user moves around (in the case of a game). Another advantage is that it saves storage space and, since the content is runtime generated, in some cases, there is no need to store the generated content, because it can still be generated again by the same rule set. This allows newly generated data in runtime and no data at all on the storage. A disadvantage is the fact that sometimes the content generation might not be perfect and it still needs human intervention. For instance, when loading data from the real world to represent it on 3D by means of a procedural modelling algorithm sometimes there is lack of information and the algorithm cannot find a consistent solution for the problem. 2.7.1 Procedural Modelling and Games Procedural modelling has its main usage on game scenario generation. It is a resource of game scene generation since the first games appeared and by the years it is getting more and more complex, capable of achieving extremely realistic and detailed scenes. Figure 10 - Fuel Gameplay [92] Literature Review 18 Fuel is an open world racing game set on a post-apocalyptic world featuring extreme weather conditions due to the global warming and day-night cycle. The distinctive feature in this game is the huge map, extending over 14400 square kilometres in size, allowing the player to explore it freely without loading times. The Executive Producer David Brickley explain that “The technology behind FUEL is based on a concept to procedurally generate data on the fly instead of just loading and decompressing it. Today’s hardware now has enough parallel processing power to procedurally create high quality environments, in real-time, while the player is moving around the world. Generated environments are optimized and arranged to get the best performance out of the machine’s hardware. As a result, the player gets the maximum amount of detail and quality of environments displayed on screen and a 40km draw distance. Generated doesn’t mean random though, it’s the same result every time, and owing to the locations we chose is in many cases accurate to satellite data, we just took some liberties for the sake of design, i.e. putting all our favourite locations in one map.”[34]. Minecraft is a sandbox open world video game that offers players the ability to build anything they can think of by simply collecting cubes from the map. The generated world is entirely composed by cubes, called voxels. A voxel is a 3D cube that is usually on discrete positions of a 3D grid and is used to create procedural modelled scenes. By clustering many cubes one can create objects such as mountains, lakes, trees, buildings and so on with extreme ease. These cubes act as large scale atoms, enabling the generation of procedural terrains with holes and caves with no gaps between cubes[35]. In Minecraft the voxels are not smoothen, making the scene look cubic. In other cases the voxels can be smoothed to create a continuous shape. Minecraft is an award winner because of its usage of procedural modelling. Whenever the game has to generate a new world, it calls upon an algorithm. This algorithm will output a pseudorandom value that is then used to determine what the world will look like. However, the algorithm Figure 11 - Minecraft Gameplay [93] Literature Review 19 will always end up with the same value if the starting point (seed) used is the same. Seeds exist, to easily generate entirely different worlds from a single value [36]. In fact, Minecraft can generate a map without size limitations. This is possible, because the map is generated in chunks, when the player explores further a new chunk is generated using the same seed and the relative position to other chunks. This guarantees that if the player moves too far from a chunk it is destroyed, but when the player moves close enough to it that chunk is generated again with the same seed, meaning that chunk is equal to the one destroyed before. Having these there are no memory concerns, but it needs to be kept in mind that any alteration made to a chunk’s terrain by a player will be destroyed with the chunk and not recovered. No Man’s Sky is an exploration game that allows the player to travel through a galaxy flying seamlessly between different planets and stars, seen on the sky. It is about exploring and cataloguing unique forms of life from planet to planet, that the player can share with other players. But of course, there are dangers, both in the open space an enemy player spaceship can appear to kill the player and even on land wild creatures can appear. The player can also evolve by acquiring new and better equipment by trading resources gathered on planet’s surfaces [37]. There is an infinite number of planets, creatures, spaceships and every planet is different from the others. This is done using only procedural modelling technologies. Everything the player sees is procedurally generated in runtime, every creature, rock, grass field, texture, ships and planets (there are no pre-defined 3D models) [38]. The developer states that the world is generated in runtime using voxels. These voxels are smoothed to get a smooth surface and are merged with each other. This mesh is showed with different levels of detail according to the speed the player is traveling and his distance to the surface. Every level of detail is also procedurally generated to ensure a higher frame rate and the usage of less memory. The developer also states that the player will never be able to travel faster Figure 12 - No Men's Sky in game footage [94] Literature Review 20 than the algorithm’s generation time, and the player travels really fast sometimes, which makes these algorithms very efficient. Every planet is generated using a unique seed, in a similar way than Minecraft. This ensures that every planet is different from the others. On land (planet surface), the player will find and catalogue creatures. Creatures are procedurally generated using a reference prototype model. This prototype does not have textures and it is a more simplified mesh. The creature generation algorithm receives several parameters as input, including the prototype mesh and generates a new mesh derived from the prototype. This guarantees creature specie similarity, but at the same time uniqueness within the same race and ensures that all races derived from the same prototype are different from each other. Also the prototypes range from very small creatures to colossal beasts taller than trees. The same happens with the spaceships. There is a prototype that is used to generate more complex spaceship models different from each other. The trees, the rocks, the grass, and even the smallest details are all generated in runtime, procedurally. 2.7.2 Procedural Modelling Techniques Procedural modelling can be achieved by any kind of polygon generation algorithm. Nevertheless, there are a few techniques that provide a decent blend of realism and performance, being used with great frequency by professional developers. The sections below detail the most common techniques used on procedural modelling. a. L-Systems An L-System, also called Lindenmayer System, was developed as a mathematical theory of plant development. The original focus was on plant topology by analysing neighbourhood relations between plant cells or higher structures. They consist on a finite set of characters belonging to an alphabet that are arranged in arbitrary length sequences to form strings. Each character is associated with a rewriting rule that is defined by a grammar. These rules mean that any occurrence of “a” in a string will be replaced by “ab” and “b” by “a”. Now the string needs to be converted to a three dimensional model using a technique called turtle interpretation. This technique is based on an imaginary turtle that walks, turns and draws Figure 13 - Example on how L-Systems work Literature Review 21 according to the given instructions. The turtle has a well-known and defined 3D position and a heading vector that points towards the direction of the movement. Each character of the string is interpreted as a command by the turtle. There is a variant of the L-Systems called bracketed LSystems that provides two extra characters that are usually square brackets (“[“,”]”) that allows the push and pop of the turtle’s direction, position and other information in order to enable the generation of branching structures [39]. More information regarding L-System can be accessed in the work performed by Przemyslaw Prusinkiewicz [40]. b. Fractals A fractal is a natural phenomenon that can be represented by a mathematical function capable of generating a repeating pattern. This pattern can be displayed at every scale. So no matter the scale, the pattern will still be noticeable, making the fractal scale invariant. When the replication is the same no matter the scale the fractal is called a self-similar pattern. It generates the same pattern at smaller and smaller scales as the recursion progresses. Another type of fractals are the Statistical fractals where the pattern repeats stochastically so numerical or statistical data are preserved across scales. This last one can add a little randomness to the fractal meaning that it is not mandatory that the fractal pattern strictly repeats. This is useful to generate flora. This fractals can generate a tree by adding the log and fraction it on each iteration until the leaves are reached (usually a predefined depth). As we can see on the above picture, we have a hand, and on the top of each finger the fractal generates another hand and so on recursively until a defined limit/depth is reached. In this case it is a self-similar algorithm, because the pattern repeats iteration after iteration getting smaller and smaller at each one. Now that we know what fractals are, there must be a way to generate them. This way is the mathematical function mentioned before. L-Systems can create the exact same patterns in some cases, meaning they are fractal generators. They use a grammar to modify the string that will be processed by the turtle interpreter, but since each character represents a command and is disposed Figure 14 - Hand fractal, just an example to better understand them [95] Literature Review 22 on a pattern style, the turtle will build that same pattern as well. On the image of the hand the grammar should be specified to determine where to fraction the pattern by specifying where the pattern should be continued, in this case the finger tops. c. Generative Modelling Language Nowadays most of the 3D file extensions describe its objects in terms of geometric primitives and respective geometric operations. The term Generative Modelling brings a different approach. Basically the main idea is to replace the 3D objects descriptions on the file by operations that will generate those objects when executed. These operations may be called rules. This means that an object is described by a sequence of processing steps rather than vertices and geometric operations. Other properties of generative modelling are the fact that it is possible to group operations in higher-level operations leading us to Style Libraries. A SL is a library that contains more or less complex operations that differentiate one type of 3D model from another. For instance, one can have a library for generating trees (it contains operations that lead to the generation of a tree) and another for generating buildings [41]. The advantage is making possible to implement a programing language that allows the generation of 3D models by just executing a sequence of commands called Generative Modelling Language. GML provides a set of operations that the programmer can combine to generate 3D models. This not only is a key for procedural modelling, but it also allows simple manipulation of 3D objects in run time by simply adjusting parameters and running the commands again[42][43]. 2.7.3 Procedural Modelling Main Usage The sections below describe the main usage of procedural modelling techniques, including what kind of techniques are used and fit best to generate a specific kind of 3D mesh. Also there are a few references about the evolution of some applications. Figure 15 - GML changing the parameter that controls the thickness [96] Literature Review 23 Terrain Terrain procedural modelling might be the main usage for procedural modelling, because as seen above most of the applications that need a huge terrain, in order to save storage and development time it is more efficient to generate it in real time recurring to terrain generation algorithms rather than storing the entire terrain [44]. A height map is a two dimensional grid that represents the elevation values of a terrain, generally used as a basis. There are many procedural algorithms for creating height-maps. Among the oldest algorithms are the subdivision based methods. A coarse height-map is iteratively subdivide each iteration using controlled randomness to add detail. Another algorithm is a variant of the mid-point displacement method, in which a new point’s elevation is set to the average of its corners in a triangle or diamond shape and then a random offset is added. The offset’s range decreases each iteration according to a parameter that controls the roughness of the resulting height-map. Nowadays, height map generation is often based on fractal noise generators such as Perlin noise that is capable of generating noise by sampling and interpolating points in a grid of random vectors. Rescaling and adding several levels of noise to each point in the height-map results in natural, mountainous-like structures. These height-maps can be transformed further based on common imaging filters or on simulations of physical phenomena, for instance erosion, because a height-map can be displayed on a grey-scale image. Thermal erosion diminishes sharp changes in elevation, by iteratively distributing material from higher to lower points, until the talus angle. The talus angles is, for instance, the maximum angle of stability for a material such as rock or sand. Erosion caused by rainfall can be simulated using, for example, cellular automata, where the amount of water and dissolved material that flows out to other cells is calculated based on the local slope of the terrain surface. Benes and Forsbach terrain model consists of stacked horizontal slices of material, each having an elevation value and material properties like density. It’s a trade-off between the limited, but efficient height map structure and a full voxel terrain. The model also allows for air layers, thereby it supports cave structures. While these erosion algorithms add much to the believability of mountainous terrain, they are also very slow, having to run for hundreds to thousands of iterations. Recent research has focussed on interactive erosion algorithms, often by porting algorithms to the GPU. Usually on video game, when the terrain needs to be very big and detailed, the developer adds a progressive LOD (level of detail). The goal is to generate the terrain detail according to the distance to the camera. When the camera is close, a portion of terrain gets detailed to the maximum procedurally. When the camera starts getting further away, the terrain progressively decreases its detail until only the greatest elevations, such as mountains, are visible. This technique increases performance, by not processing irrelevant information while it is not visible and only processing it when it is needed. The basic noise-based height-map generation delivers results that are fairly random and natural. Usually, users control the outcome only on a global level, often using unintuitive parameters. To overcome this problem and allow a more easy terrain configuration several researchers have addressed this issue, such as Stachniak and Stuerzlinger [45] that proposed a method that Literature Review 24 integrates constraints into the terrain generation process. It uses a search algorithm to find an acceptable set of deformation operations that conform to the constraints to apply to the random terrain. Gamito and Musgrave propose a terrain warping system that results in regular, artificial overhangs. A recent method by Peytavie provides a more complex structure that considers the existence of different material layers that support rocks, arches, overhangs and caves. In most of the cases their resulting terrain models are usually visually very appellative and natural [46]. a. Height Maps A height map is, in terms of computer graphics a raster image where each pixel of this image has a colour between black and white inclusive that represents the height of that point. The whiter the pixel the higher the point. The image is used to create a 3D mesh where each vertex has the same height of the pixel or average of pixel heights for that corresponding mesh point. Being so, and repeating the procedure for every mesh vertex one obtains a 3D mesh corresponding to the height map used. The real height of the 3D mesh is scaled, because since the height map is a black and white image the pixel’s values are just between 0 and 1, so the values need to be scaled. Height maps are not just for creating terrains, they also apply on computer graphics materials by acting as a displacement map for a texture to displace the position of the texture points in order to give the sense of depth. There are a lot of tools that can generate height maps, because they are simple grey-scale images. For terrain height generation developers usually use Perlin Noise, fractals and L-System to paint the height map’s image. This produces a pseudo-random height map with more or less smooth black and white transitions in order to create mountains or rough planes. Basically the result of the previous techniques give a realistic feeling to the terrain. Height maps do not always need to be generated using the referred techniques, they can also be calculated by physical means such as LIDAR, stereo photogrammetry and so on. Data obtained by these means is very expensive and extremely precise, being so it is used by NASA to create a full earth scale height map called Digital Elevation Model (DEM). This model has different resolutions for the images, in other words the data contained on a pixel of the image can have different corresponding distances on the real world. Therefore one pixel can average and cover a certain number of square meters. CGIAR-CSI distributes this kind of data for free, but of course it is not the best data that exists, but it still manages to cover around 80% of the entire world with a resolution of 90 meters [47]. It also provides different downloadable files such as GeoTIFF and ArcInfo ASCII. GeoTIFF is an image format that contains additional information for each pixel, but it needs a special interpreter to open and manage it. On the other hand ArcInfo ASCII is a text file that contains the elevation of each point. It makes it extremely simple to process and manage. Having so, it is possible to process real world data and create height maps with it, to apply to terrains. Literature Review 25 b. World Machine World Machine [48] allows users to generate extremely realistic and detailed terrains using a graph based interface where the user can add nodes that act as modifiers. The user can add as much nodes as he wants to achieve an endless combination of effects. It also implements very powerful fractal generators with the addition of Perlin Noise. Perlin noise is a procedural texture primitive that generates a gradient noise to increase the appearance of realism in computer graphics. Its powerful erosion system allows to naturally erode the surface of the terrain by realistically simulating natural effects that acted on that terrain for millions of years, such as water, wind and others in just a few seconds. It also allows the user to insert shapes and roads on the terrain with realistic adaptation to the terrain’s surface. For the terrains to look as the user wants them to, it comes with a built-in texture creator that with just a few steps creates very complex textures according to height, erosion, vegetation and so on. Figure 16 - World Machine - Mountain Terrain Example [97] Literature Review 32 relative to the imagery quality and contrast. This solution was developed by Mohammad Izadi and Parvaneh Saeedi that attempt to detect building’s height using satellite images by detecting the building’s shadow and creating an estimate of a projected shadow for each given height. It uses a genetic algorithm that generates different heights for the buildings. Then the projected shadow is calculated and compared to the actual shadow and when they match we have the height [61]. They claim that have achieved a solution that has a mean error of 27cm for satellite images and just 15cm for aerial images. This is because the image resolution and detail as said before can be a problem. 2.8 Tools to be Used To implement this framework I was free to choose all of my tools, but with the condition that it had to have both a game engine and a microscopic traffic simulator. 2.8.1 Unity 5 Unity 5 is the game engine I choose, because I am familiar with it and have used it to do other projects. It is also very intuitive and simplistic allowing the user to do very complex tasks with just a little effort. There is also a huge amount of very well made documentation, tutorials and even specialized forums. The Unity ecosystem is available to anyone who downloads the Unity engine. The Unity engine integrates into one unparalleled platform the tools to create 2D and 3D interactive content, collaboration solutions, rapid multiplatform deployment, and retention, advertising and analytics services to grow your business. 2.8.2 UnitySlippyMap UnitySlippyMap [62] is an open source project developed by Jonathan Derrough that aims at helping developers create maps working with a variety of online tile providers including OpenStreetMap. It can create tiles on Unity’s 3D space along the XZ plane. The map can be zoomed and dragged using the mouse and new tiles appear or disappear as needed. In order to include OpenStreetMap maps in my application, UnitySlippyMap needs a few changes in the code. One of them was disabling all unnecessary features to make the most simple and easy to use interface. The second was to allow to place a marker on the map using the mouse and when the marker is placed the second maker will define a diagonal that will be used to create a bounding box. This bounding box limits the area to be downloaded. Literature Review 33 2.8.3 Open Street Map OpenStreetMap [63] is an editable map that is a trusty representation of the whole world built and enhanced by the community [64]. Being so, it is almost an open source project with the main goal of sharing with everyone real world map data so people can use it for whatever they want. Being an open source tool some researchers start thinking about solutions to solve real world problems such as Moritz Göbelbecker and Christian Dornhege that attempted to parse the OSM data in order to use it on a robocup simulator for rescue missions [65]. 2.8.4 Google Static Maps API Google Static Maps API allows users to download satellite images based on the parameters sent on a URL by HTTP. These parameters can be the geographic coordinates, the zoom, the address of the location and pretty much anything that Google Maps uses to refer to its locations. Having this, it is possible to download satellite images from Google Maps. 2.8.5 SUMO Simulation of Urban MObility SUMO [66] is a microscopic traffic simulator, which consists on a platform developed by the German Aerospace Centre (DLR) that has all the needed tools to implement a traffic simulation. It can import any kind of road network, even from OSM (OpenStreetMap) and generate a road network according to that data. Having the network it can generate traffic, traffic routes, traffic lights, pedestrians and different types of vehicles with different characteristics. A great advantage of this software is that it is extremely portable and can be integrated in any external application, because due to TraCI, which is a middleware layer that creates a TCP channel and has a very well defined protocol, SUMO can act like a server and be controlled by the external application [67]. 2.8.6 CGIAR-CSI CGIAR-CSI [47] is a free SRTM digital elevation data provider where all that data has been processed to fill data voids, and to facilitate its ease of use. Their mission is to provide DEM data in an effort to promote the use of geospatial science and applications for sustainable development and resource conservation in the developing world. The free data provided is from SRTM 90m DEM that has a resolution of 90m at the equator, and are downloadable in seamless mosaicked 5 deg x 5 deg tiles. All this data is available in both ArcInfo ASCII and GeoTIFF formats to facilitate their ease of use in a variety of image processing and GIS applications and cover over 80% of the entire globe. Literature Review 34 Also, all the data has been processed to remove possible “holes” on it caused by bad height reading on certain locations due to water basins or heavy shadows. 2.8.7 7zip 7zip [68] is an open source compression and decompression tool capable of high compression and ease of use even by command line. Being open source it can be adapted to integrate multiple compression and decompression algorithms as well as cryptographic algorithms to encipher files in a secure way. The need to use 7zip is because the data downloaded from CGIAR-CSI comes compressed in a zip file and in order to be decompressed automatically by the application, an open source decompressor was needed to be integrated in the application. Since 7zip can run in command line it is easy to execute it and extract the needed files into a specified folder by command line arguments. 2.8.8 Blender Blender is an open source software aiming to help artists create 3D models by means of free to use tools and software, as well as an active community and development team. On the context of this project, Blender is used to take care of any manual modifications to meshes, such as enhancing the vehicles mesh, create simple collision meshes for the objects, more easily mapping the UV coordinates and 3D model simplification [69]. 2.9 Related Work There are a few attempts to do a 3D reconstruction of real-world data using OpenStreetMap, but none of them do it so deeply and are far away from having any kind of population of vehicles or walkers. M. Goetz and A. Zipf transform the OpenStreetMap buildings into polygons with no windows and no texture, and there are no roads and no vegetation [70]. Two more examples are [71] and the work by Neubauer, et al. [72]. Comparing our approach to these examples, this project advances a step further as it merges procedural modelling of real-world data and driver behaviour analysis with the possibility to generate a scene that can be exported as a 3D object file, which can be opened by 3D modelling applications. Of course there are other alternatives mentioned above, such as World Composer that can be integrated with multiple tools to generate the buildings and the roads as well as the vegetation in an extremely optimized way. But on the other hand, each one of these applications is paid and optimized to do a very strict and specific task, making it very hard to expand in terms of functionality. Another problem is that when using a paid application to develop another, the final version of the code can never be open source, because of legal rights, it is obvious, because the Literature Review 35 paid code will be available to anyone for free. Having so this tool will only use free and open source software, in order to be freely distributed and modified by everyone. This is the major difference relative to any other existent application/project. Although not so immersive as real scale driving simulators, serious driving simulators can give important contributions to the analysis and proper understanding of driving behaviour, such as what has been carried out in previous projects [73][74][75][76][77]. 2.10 Summary So far, this dissertation presented a detailed review on some of the existing technologies and techniques. First of all there were some descriptions on simulation engines followed by driving simulators, their types and top simulators, as well as detailed functionality of their possible peripherals. There is then a change of topic to cover procedural modelling latest and best applications followed by the technologies behind the content generation such as terrains, flora, roads, buildings and cities with a simple explanation and a few software capable of generating them. In the end the reader finds the technologies that are going to be used and an explanation on why they are going to be used. There are a lot of techniques and technologies to choose from, every of them with their own positive and negative sides. For this project most of them will be needed, such as terrain generation, height map download, procedural buildings, roads, vegetation, textures, traffic simulation forcing procedural roads to have lanes and traffic rules and microscopic traffic simulator integration as well as a fully controllable and interactive vehicle. Methodological Approach and System Architecture 36 Chapter 3 Methodological Approach and System Architecture This chapter starts by giving a detailed presentation of the problem to solve. It presents the system’s architecture with explanations on why and how it is going to work, on a high level without too much detail regarding algorithms or techniques. This project follows up and extends previous attempts to implement an integrated platform for traffic and transportation engineering and research, focusing on driver behaviour modelling and the interactions of drivers with the surrounding environment [78][79]. It will underlie the integration of concepts such as artificial transportation systems [80] and serious games applied to transportation [81]. 3.1 Problem to Solve Nowadays the configuration of driving simulation environments takes a very long time, because the modelling of 3D scene by hand is a very laborious and time consuming task. If one wants to conduct an experiment on a real world location, the artist that is going to model the scene needs to gather information relative to the elevation, the roads, the buildings and so on, making this a very expensive task. Therefore the driving simulation experiments should be faster, easier and cheaper to conduct, as well as real world generation. There are multiple free world data provider, so it should be possible to procedurally generate real world locations automatically to tackle the production time and cost problems. Serious games provide a cheap, portable and intuitive way to create any kind of experiment, so this is the right way to go. Methodological Approach and System Architecture 37 Therefore, this dissertation has the objective of exploring different techniques and approaches to tackle this problem by developing a framework based on serious games that is capable of:  Procedurally generate real world locations in 3 dimensions automatically;  Simulate a virtual vehicle that can be controlled by the user;  Simulate and give the user the freedom to configure different simulation conditions that directly or indirectly affect the results of the simulation. To accomplish this goals, the framework should:  Gather real world data and render it in 3 dimensions;  Allow the user to configure different simulation conditions;  Allow the user to drive a virtual vehicle;  Be completely open source;  Export the generated scene;  Simulate pedestrians;  Integrate Oculus Rift;  Simulate traffic; Methodological Approach and System Architecture 38 3.2 System Architecture The image below is a diagram that shows all the system architecture including all the interactions between the system’s modules. In order to accomplish all goals, the system needs to integrate multiple existing tools. For starters, Unity 5 will be the main application. The user runs the Unity executable and is prompted with the main menu of the application. Here, the user can select options and the location he wants to download, either by geographical coordinates or by address. This information is kept in a class with static variables called “Manager”. The “Manager” is responsible for storing key values such as all the options the user selects and the current vehicle. On the menu the user can travel to the vehicle selection scene. There is also an input box for the user to type the desired location. When this box is filled with a valid location, its value is sent to OSM API by http protocol and UnitySlippyMap opens up a map on the specified location. The user can zoom in and out and drag this map. On it, the user can select the bounding box he wants to download. Once it is selected the corner limits of the bounding box are sent to the OSM API and it responds by sending an XML file to the requester with the desired location. This XML file contains all the information about the location the user downloaded. When it arrives to Unity it is then processed by a parser on a new scene where all the 3D construction will happen. Initially, the parser will read the bounding box’s limits of the downloaded area as geographic coordinates that will be used to find and download the corresponding tiles of DEM from CGIAR-CSI. When the DEM finishes loading and the terrain is configured with a height and size to fit the mentioned bounding box, the parser will download all the satellite images for that location by making requests to Google Static Figure 21 - System Architecture and Module Interaction Methodological Approach and System Architecture 39 Maps API. As the images arrive to Unity they start being placed on the respective terrain tiles as a texture. While the XML is being parsed, 3D models start being generated by making use of the parsed information and the classes on the Procedural Modelling module. When the meshes are all generated the user is allowed to control the selected vehicle. If Oculus Rift integration is enabled, then the vehicle will only have the cockpit camera active, not allowing to switch to any other camera. 3.3 Unity3D + SUMO (Simulation of Urban MObility) First of all, the basis of this integration takes in account the great capability of SUMO that is acting like a server when launched by command line using a specific argument that allows to create and open a TCP port, for an external application to connect. While acting as a server, SUMO responds to commands from the connected application. On the other hand, these commands are very complex and hard to compose, because they have to be managed at byte level following a well-defined protocol. The same happens to the answers received from SUMO. Also, the SUMO’s documentation explains all the possible command-answer combinations, but there is no information at all on how to compose a specific command making it very hard to test, since SUMO may not answer or say what is happening with the commands received. There is a platform, created at FEUP’s LIACC, written in Java, called TrasMAPI [82] that has a good list of predefined commands where the developer just has to insert the arguments, but it still does not cover this project’s requirements since the commands needed to extract all vehicles do not exist yet, which brings us to stack zero. Another problem of this integration is the fact of being a TCP connection. Unfortunately SUMO can’t be part of Unity, it can only communicate by TCP connections. A goal of this project is to allow the user to drive a virtual vehicle, which needs to be in real time with the highest frame rate possible. In order to have a smooth driving experience it needs to feel seamless to the user and this only happens over 25 FPS. The problem is, that to recreate the vehicles of the simulation on Unity the information of their positions needs to be sent to it constantly, otherwise the vehicles movement won’t feel seamless, but the information takes time to be written to and read from the TCP channel. This is inevitable. A solution to this problem is, for instance, instead of asking for and receiving the information every update, do it as soon the last information arrives to Unity, asynchronously. Of course the movement will not be seamless like this, but it can be corrected by position interpolation over time. SUMO can send a specific vehicle’s speed at the moment and using it, it is possible to interpolate between every frame. Unfortunately this will not work perfectly, because the interactions between the drivable vehicle and the simulation vehicles will not happen in real time, but only when Unity sends the drivable vehicle positions, making the vehicles run over each other, because of the interpolation between SUMO answers. In addition, Methodological Approach and System Architecture 40 this approach adds even more overhead to the TCP connection, because the speed of each vehicle needs to be sent in a different command from the positions commands. So to render the vehicles correctly there are needed 2 commands and 2 answers. The overhead problem gets even worse when lots of vehicles are running, making it impractical. There can be found on the internet a few tools and tutorials that can help building a traffic simulation inside Unity, which is a very plausible idea to overcome the TCP overhead problem. In addition it makes possible to simulate full physics and collision interactions between all vehicles and also each vehicle can have different characteristics determined by their shape and mechanics and can adapt to the road inclination instead of a 2D model of the network that always considers the roads flat surfaces. Another problem with SUMO is that it does not provide a seamless simulation. The simulation time step is too big by default and in order to enhance it, some changes need to be done to the source code. Finally, SUMO’s lane changing model is not continuous, it is discrete. This means that the transition of one lane to another is not seamless. On other words, the vehicle simply disappears from its current lane and appears on the lane it wants to go to. The vehicles do not overlap, because they know about each other’s positions, but when a human is driving a virtual vehicle on this simulation it will not have good results, because vehicles simply appear in front of him whenever they want, and there is no signal of the vehicles intentions except for signal lights. Having all these problems definitely makes the SUMO connection a bad way to go, because most of the time would be spent attempting to establish a good connection speed. So the supervisor told me to forget about SUMO and to focus harder on the procedural modelling algorithms and all the other functionalities. Therefore this project is supposed to do everything even better than what it was expected on the beginning, with the exception of the SUMO integration that fortunately was abandoned. 3.4 Development This section intends to clarify the reader on development matter, because this is a very ambitious project with lots of features to implement. The architecture described above might seem simple, but the underlying implementation is very complex, because it requires the study and knowledge of the tools to be used and every component that interacts with them needs to be carefully designed and tested to grant no space for errors, malfunctions and bugs. The development was based on the implementation and merging of different functional prototypes, each with a different specific functionality. What this means is that the development was done by successive iterations, each one consisting on a functional prototype construction and respective testing. The validation of each one was done simultaneously by half of the time together with the supervisor. A functional prototype is an application that can perform a very strict Methodological Approach and System Architecture 41 and specific task and are usually a separate scene on the Unity project, isolated from the other scenes. By creating a different scene for each prototype it is possible to easily merge them with the main scene, because they belong to the same project and automatically share the same logic. Some prototypes where implemented along the 1st semester, explaining the main reason for this project to be so ambitious. The first prototype developed is able to load and parse a file from OSM. While loading the file, Unity draws lines that correspond to the roads and buildings. Meanwhile it starts calling the Google API to download satellite images and display them on tiles to cover the entire region, with the best resolution as possible. The second one allows the user to control a vehicle. This one is meant to fine tune the handling and physics of the vehicle to control. Now, it was time to build the main scene. Therefore, the first and the second ones were merged into the main scene. The third prototype generates procedural roads along a predefined path. Next, another prototype was developed to create the day and night cycle. This prototype is very simple, it is just a script, so it was merged with the road generation prototype to make it easier to build the illumination system. The street illumination is generated on this prototype. The fourth prototype can load an OSM file and download a DEM, corresponding to that location, from CGIAR-CSI and unzip it using 7zip by command line. After the download and the unzipping are complete the corresponding region of that OSM is extracted from the DEM file and applied to a terrain to generate heights from the real world. Having these prototypes it was time to merge them all into the main scene. By this time the work was stopped for a time period of around a month due to personal setbacks. When the work was resumed a new version of Unity had been released. The work was started on Unity 4.6, but then Unity 5 came out, introducing a lot of new features as well as API modifications. After the update a lot of scripts were needing an API update, both automatic and manual. Unity 5 includes the new Physics 3.3, which provides major modification to the vehicle’s physics. After the update the current vehicle physical configurations and scripts didn’t work at all. Not only the work was delayed a month, but also the vehicle physics had to be remade from scratch. The first step after the update of Unity and the project was to quickly fix the vehicle physics in order to at least have a more or less drivable vehicle. Next the buildings prototype was created. This prototype includes everything about the building generation and consists on a script that generates a building based on a path. Then it was time to generate the vegetation. This took a bit more than the buildings, because there was a need to find the balance between the number of vegetation elements and the performance of rendering such objects. After integrating the vegetation on the main scene, a new prototype was started, this time to generate the weather conditions. The weather conditions depend on the duration of the day, therefore this script depends on the script that controls the day and night cycle. Methodological Approach and System Architecture 48 If there needs to be more than one satellite image to cover the entire area, then more than one tile needs to be generated. Knowing the bounding box limits and that the satellite images cover a constant distance of 382 meters, the algorithm iterates by 382 meters at 382 meters, starting from the minimum latitude and longitude, until it reaches the maximum latitude and longitude and for each 382 meters, a tile is created and positioned on the respective distance from the centre of the locations bounding box. Then, a request is made to Google Static Maps to download the image. Now, if the tiles were placed on their exact position, corresponding to the real Earth’s position, Unity would throw an overflow exception, because the position values are all floating point values and the desired position’s value may not fit on a float, so the centre of the location is translated to the origin of the Unity’s referential and the tiles are placed on their correct real world scale position, but relative to the translated centre, preventing the overflow exception, since the position values are never that large this way. As soon as the download of the image finishes, a material is created using this image as a texture and it is applied to the tile. This tiles are simple flat planes and do not have any information regarding heights. Each elevation tile, is generated right after the respective flat tile, but in a much complex way. The plane is only used to hold the texture when it arrives and to simplify the positioning of the elements that constitute the elevation tile. 3.5.7 Terrain Elevation To generate terrain elevation, in this case, elevation tiles, the algorithm uses the data gathered from CGIAR-CSI and creates tiles that have the correct elevation, for that specified location. Each of this tiles, is generated using the same conditions as the ones in the section above, such as the scale, position and translation and are generated right after the respective plane. For starters, each tile is composed by 16 square polygons, which is the result of the following calculation:  A tile has to cover around 382 meters on x and y axis to be in real world scale, meaning it will cover the satellite image completely and we also know that each elevation point is separated by 90 meters from the others. If each polygon that makes up the tile covers 90 meters on both axis, then we will have 382/90 = 4.244 polygons. The result is a decimal number, but it does not make sense to have 0.244 of a polygon, therefore to cover the area, the value e rounded making 4 polygons. To cover the elevation along x and y axis we need 4x4 = 16 polygons in total. Although, this approach introduces an error of around 5.5 meters per polygon, meaning that the vertices that contain the height values will be 5.5 meters offset from their real world position, but since the data resolution is 90 meters the error is almost insignificant. Methodological Approach and System Architecture 49 The image above show the order each polygon is processed and the order the vertices are used, following the left hand rule, so the front face is facing upwards. Each red dot is a vertex that contains a height value. Every time two black lines cross, means that vertex is shared by other polygons. To apply the whole texture on the entire tile, the UV coordinates need to divide the texture in 16 smaller textures, each one covering the respective polygon. Since the range of the UVs is from zero to one, where (0,0) represents the bottom left corner of the texture and the (1,1) the top right corner, then the UV’s for each polygon are set based on its relative position to the others. For instance, the UV coords of the first polygon will be (0, 0), (0, 1/4), (1/4, 1/4), (1/4, 0), following the order of the vertices and for the second (0, 1/4), (0, 2/4), (1/4, 2/4), (1/4, 1/4). The image above, attempts to illustrate the basic idea behind the UV mapping. The algorithm takes the image on the right and slices it in 16 pieces that will fit the respective polygon. If the UV coordinates were not set, the image on the right would repeat on each polygon, so the UVs of each one are mapped to a percentage of the texture. When the tile is generated, it is composed by 16 separate polygons. This is not a bad thing for one tile, but when the scene has a big number of tiles the processing power to draw those increases rapidly. Unity renderer can process a mesh with thousands of vertices much faster than processing thousands of meshes with only 4 vertices. Having this principle, the tile’s polygons Figure 22 - Elevation tile vertices and polygon processing order Figure 23 - Elevation Tile UV Mapping Methodological Approach and System Architecture 50 are combined by a combining function, found on Unity Community Wiki and this clearly improves performance by reducing the number of game objects on the scene, thus the draw calls. This process repeats for each tile the algorithms needs to generate, so it can cover the desired location. To do so, the algorithm iterates 382 meters at 382 meters, until the location is all covered, just like the flat planes explained in the section above. 3.5.8 Procedural Roads Having all the “ways” read from the .osm and organized, it is time to render all the roads. For starters, let us remember what was explained in section 3.5.5. All the Ways are separated by their tags and classified, so they can be stored in the correct lists. Also, remember that each Way has a reference to the game object it represents. For each Way on the list, the algorithm calls a function that adds the script responsible for generating the road’s mesh and the tags are processed once more, to configure the values of the added script. This values are information that allows the generation procedure to know about the existence of sidewalks on the left or on the right, the road and the sidewalks width, the surface type, the road type and whether it is lit or not. OpenStreetMap calls “highway” to any kind of road. So, the algorithm that searches for roads, looks for the tag “highway”, in order to add the way to the roads list. Also, OSM considers the existence of around 27 different types of roads, but we only consider 14 for simplification and some of them have predefined values, because they vary from country to country. Some types of road require a very specific mesh, such as motorway nodes, bridges, tunnels or have less relevant and frequent usage, such as bridleway (specific for horses). So, due to the short delivery deadline, those kinds of road types were not taken in account. The considered ones are:  Primary roads – usually 6 meters width, no sidewalks, two lanes, not lit  Motorways – same as Primary roads  Secondary and Tertiary – same as primary roads, but around 4 to 5 meters width  Residential and Living Streets – inside city roads, so they are always lit, they have sidewalks on both sides, unless a tag defines it as sidewalks on the left or on the right or none and they have two lanes.  Trunks – like motorways, but do not meet the same requirements, such as the speed limit or safety measures.  Tracks – usually a dirt road used for walking or off-roading.  Not clearly defined roads are considered non-laned asphalt roads Methodological Approach and System Architecture 51  Paths – similar to tracks, but for undefined roads, dirt surface usually  Raceways – usually over 6 meters width, not lit, no lanes, and race track surface.  Footways – not over 2 meters width, not lit, no lanes, tartan surface (red to better distinguish), may have sidewalks of not  Cycleways – same as footways  Road Construction – usually around 5 meters width, may be lit or not, may have sidewalks or not, usually have a concrete or dirt surface When all the parameters are configured and assigned, it is time to generate the road mesh. There are a few techniques to generate the road mesh, such as Bezier curves, round splines or grid layouts. These are all good techniques for completely procedural scenes, but this project is not the case, because we already have the paths a road has to follow, on other words its nodes. Having these nodes restricts the road’s shape and roundness. With Bezier curves, we could not have hard bends or non-smooth corners, due to the control point. Regarding the round splines, the paths are always straight lines that connect nodes and are never round. So, the solution found to create the best roads possible, was actually very simple. To generate the road mesh, the procedure divides the distance between 2 consecutive nodes by a predefined value, to determine the number of segments this piece, of the entire road, will have. Then, the length of the segments is calculated. All the segments have the same size, so to determine their length, the algorithm simply divides the distance between the two nodes by the number of segments discovered before. When the length is found, the procedure iterates from the initial node to the next by the length of each segment and creates a control point on that position, storing it on a list. When this control point is created, the procedure create 2 vertices that will belong to the road’s mesh. Each one, is placed on each side of the road at a distance from the control point equal to half the road’s width and perpendicularly to the path’s current direction. The Y coordinate of these points, which is the height, is determined by casting a ray down, from the highest Y possible on that point’s position and getting the Y coordinate of the hit point with the ground. This procedure is repeated for each pair of nodes. Methodological Approach and System Architecture 52 Now comes the solution found. When the entire path is processed and the points stored, the next step is to smooth the path, to make it look more realistic and to avoid sharp square corners and possible overlaps caused by them. As explained before, a different method had to be used. This method attempts to smooth a road by making an average of its points, with some rules. The first set of points to be smoothen are the control points, because they represent the middle of the road. The procedure picks the first and the third control points and calculates their average position. The second control point’s position, will be equal to this average and its height is calculated using the ray cast method, mentioned above. Then, the procedure goes on to the next point and the process repeats. For the road vertices, the average is calculated the same way, the only difference is that the point needs to maintain its distance from the control points (half the width of the road). If the distance is smaller or greater, then the point is translated by a value equal to the difference between the road’s width and the point’s position and once more the height is calculated. This process is repeated a determined number of times, called “smoothing level”. After a few testing, it was found that the smoothing level that offers the best results is 8. In case the path is a closed one, for instance, a roundabout, there is an extra step, which is to include the smoothing of the first and the last vertices of the road. Figure 24 - Procedural Road Vertices Placement Figure 25 - Procedural Road Before and After Smoothing Operation Methodological Approach and System Architecture 53 After the smoothing operation, the vertices are ready to be used to build a mesh. The mesh is built by getting 2 vertices from the vertices list that belong to the left side, 2 control points and 2 more vertices that belong to the right side. Unity uses the left hand rule, meaning that the vertices should be ordered clockwise. This grants that the normal of the generated polygon points up and the front face is visible. Having this in mind, the road is generated and the front face is visible. The road is composed by multiple polygons, all well-arranged, well positioned and dependent from each other, but, to Unity, they have no relation with each other, they are simply separate pieces of a puzzle. This is very bad, because there are so much separate pieces in the scene that the frame rate will suffer a massive penalty. To fix this, the road mesh is merged into a single mesh, using a function from Unity Community Wiki. For Unity, processing thousands of very small meshes is much slower than processing a huge and complex mesh with thousands of vertices. Regarding the sidewalks, they are generated exactly the same way as the roads, the only difference is that they have an offset equal to the road’s width and do not have control points. Therefore, to generate a sidewalk the algorithm gets 2 vertices from the sidewalk vertex list and 2 more from the road, belonging to the respective side. Having these 4 vertices, the sidewalk height is added to each vertex’s Y coordinate, generating a polygon with 4 vertices. There is also a step that closes the gap between the road and the sidewalk. This step’s vertices are equal 2 by 2 to the vertices of the sidewalk and the road they are going to join, respectively. As seen on the image above, the points that are used to generate the sidewalk step, all belong both to the sidewalk and the road, so its generation is very simple and easy. To follow the left hand rule, the vertices are processed on the following order: S3, S2, R1, R0 for the left sidewalk step. And for the other side: R3, R2, S1, S0, where S1 and S0 are not the ones represented, but instead the ones that belong to the right sidewalk step. Figure 26 - Sidewalk and Step Generation - Vertices relation with each other Methodological Approach and System Architecture 54 a. Road Enlightenment Road enlightenment is a key feature on road safety, both for pedestrians and drivers and its presence and type depend on the type of road they are illuminating. Residential areas always have some sort of lighting, on the other hand, highway/motorways are almost never lit, such as city links, very rural areas, secondary and tertiary roads. Due to its importance, it is fundamental to have illumination on the procedural roads. The illumination is generated by two types of street lights. The first ones are regular street lamps that can be found on city locations. The second ones are out of city illumination, therefore they are street lights like the ones found on highway/motorway links. Each one of these lights can cast shadows, and be displayed with different levels of detail, depending on the distance to the main camera. The lights are placed close to the road, usually on the sidewalk step, but they can be offset, to be placed further away from the road itself. They are placed on a regular interval, on both sides of the street, and are generated as soon as the road they are going to be placed on is finished rendering. To place the lights, the algorithm simple gets vertices of the inner part of the sidewalk on a regular predefined interval until reaching the end of it and places a street light on that vertices position. When all the roads are rendered, there is an algorithm that gets all the street lights and, for each one, it casts a ray down from the position of the light and if this ray crosses more than one road, then probably this is a zone where two or more roads overlap, which means that the street light needs to be removed from the scene. This prevents having a street light in the middle of an intersection for instance. This project also features a full day-night cycle, meaning that the lights turn on and off when needed. When the sun is rising, a messaging system sends a message to all the lights to turn “off”, when the sun is setting, the process repeats, but sending the message “on”. The reason for sending a message, instead of a polling operation, is that every single light would have this polling operation on every frame, this would cause a massive drop down on the frame rate specially when Figure 27 - Street Illumination 3D Models City Lamp on the left and out of city light pole on the right Methodological Approach and System Architecture 55 there are a big number of lights, so instead all the street lights have an event listener that reacts only to “on” and “off” messages. 3.5.9 Pedestrians When driving a vehicle, pedestrians usually have priority over it, when it is allowed. They interact with the roads and some of them may be too distracted to pay attention to the surrounding environment, for instance, if they have headsets or are on the phone. Another thing, is that some pedestrians may not have a driving license, therefore they do not know the traffic rules, so they think they can cross a road anywhere knowing that there should be a road crossing somewhere. For last, pedestrians are very unpredictable, especially children. A driver should be paying attention to the road, to the surrounding vehicles and to the pedestrians, but contradictorily, he should be capable of predicting other drivers and pedestrians behaviours, to increase road safety. This project implements a simple pedestrian system that attempts to simulate small crowds of different persons with different behaviours within some limits. When the road finishes being generated, a script is attached to every walkable element, in this case the sidewalks and the footways. This script, stores their vertices, so they are easily accessible. This is done like this, because all the information regarding the roads is destroyed as soon as the road finishes generating, to save resources. So, there is the need to store the relevant information elsewhere, in this case, on the attached script. When all the roads are generated and all the walkable areas contain their own vertices stored, an algorithm that will generate the pedestrians, is called. This algorithm generates pedestrians based on the pedestrian density, configured on the main menu. This density is relative to the size of the location selected, to prevent an extremely large crowd that would kill the frame rate. If the map is made just by one tile, the amount of pedestrians compared to a 10 tile map, for the same density, would be 10 times smaller. The pedestrians are not distributed on equal number per tile, but instead, they are randomly generated along the scene. Although the density is relative to the location’s size, there may be smaller or larger crowds on some places. The objective is to have harder places to drive on and some other easy, so the driver can relax. The generator picks a random walkable area, a random vertex belonging to that area to define the position where the pedestrian is created and a random human character. Each pedestrian has an AI script attached that receives, as parameters, the game object the pedestrian is walking on. Also, this walker is added to a level of detail controller explained below. There are six different walker 3D models, three female and three male, with three animation states each. There are walking, running and idle animations. Methodological Approach and System Architecture 56 Each one has its own AI script, whose values are configured automatically and randomly, within certain limits. These values are the walking and running speeds, the force the walker does to avoid obstacles, the ahead visibility and the acceptable node distance that indicates the limit distance that the pedestrian has to pass from its next node on its route. The walker can be in one of four possible states that identify whether the pedestrian is idle, walking, running or has been hit by the vehicle. If the pedestrian does not have a path defined yet, then it will calculate its own path over a walkable surface. A pedestrian always has a walkable game object associated, but that does not mean that its path along that walkable surface is calculated. If that path is not defined yet, the algorithm iterates along the entire walkable surface, calculating an average of 4 vertices of the walkable area (2 from the inside and 2 from the outside), adding this average point to a list, until the last set of 4 vertices is reached. By this method, every pedestrian on the same walkable area has the exact same path, so to avoid this and to make their movement feel more natural a random offset is added to the average points of their path. This offset is a random value between the sides of the walkable area and since this area’s width is constant along the entire mesh the offset value can and should be constant, therefore the first two vertices of the walkable area can be used to calculate this offset (vertices 0 and 1). When the path finishes calculating, the pedestrian decides randomly to go left or right and finds the closest point to his position, which will be his first target. Then it can walk from node to node. To accomplish this, the pedestrian selects the next node he needs to go to depending whether it is going left or right. Having the next node selected, it analyses the scene ahead of it, using a defined distance of vision and attempts to calculate the force it needs to avoid the objects in front. The force is calculated as a vector on space, equal to the difference between the detected object’s position and the pedestrian’s position. To detect an object, the pedestrian casts a ray forward with a distance equal to the ahead visibility. If the avoiding force isn’t enough, then its value is doubled. Another thing the pedestrians do, is avoid being out of a walkable area by running from non-walkable areas to walkable ones. As soon as the walker detects it is on a road, it runs to the next point of its old path. A non-walkable area is always a road where traffic moves, other areas that are not considered walkable do not interfere, Figure 28 - Pedestrian Animation Skeleton Walking – Running – Walking 2 (Jogging) - Idle Methodological Approach and System Architecture 57 so the pedestrian does not feel the necessity to run. Having analysed the environment to calculate the avoidance force and determined whether to run or not, the movement force is ready to be applied to the character. The walker is always looking to his next point of the path and moves forward with walking or running speed, depending on his state. After applying movement to it, the avoidance force is applied, to make the walker move left or right depending of the force signal. When the pedestrian is over more than one walkable area, it has the freedom to choose whether he wants to change to another walkable area or keep on the same one. If it changes to another one, the walkable area vertices are retrieved from the identified game object and the old walkable vertices that the walker had stored are replaced by the new ones, then the path along the new walkable area is calculated and the walker changes its route. Till the pedestrians are hit by the vehicle, they do not have any kind of physics applied. They are just game objects that move at a certain speed and fall over heights, because there is a programmed constant velocity (the move function receives velocities, not forces) pulling them down. When they get hit by the vehicle, their movement characteristics and animations are disabled and they turn into a ragdoll. A ragdoll is a physics puppet, where each member of the body has physical properties such as weight and joints to other parts that can break and have torque limits, making the character a free physics object, where each part of its body can interact with any other physics object. This changes their state to the “hit by car” state and counts a casualty for the statistics. The ragdoll cannot be enabled or disabled, it can simply be disguised by the animations, so at each Unity’s physics update the physics calculations for each pedestrian are done, but not applied. When the number of pedestrians reaches hundreds on the entire scene, the frame rate drops a lot. To fix this, a level of detail (LOD) controller was created. This LOD controller has a reference to all the pedestrians in the entire scene, stored in a list and at each frame the algorithm runs through the list and for each one, it calculates the distance between it and the main camera, if the distance is bigger than a certain value (in meters), then the pedestrian is disabled, otherwise the pedestrian is enabled. Disabling a game object removes that object from any processing routines or scripts, so there are no calculations at all for the disabled, it is like the object does not exist, although it is not removed from the scene. By doing this, only the pedestrians at a certain range are processed and visible, but when they are outside that range they stop being processed. By removing the ragdoll from the characters, a much greater number of visible walkers could be achieved, but then there would be no kind of interaction between them and the vehicle. Methodological Approach and System Architecture 64 There are only being used two models, one for the trees and one for the bushes, because, first of all, it is not important to procedurally generate the tree or bush meshes in this project, since its goal is to procedurally generate a world location, not necessarily all the location’s assets and secondly, it is just a detail, it is more important to have quantity instead of variety. For starters, just like for the buildings, the perimeter of the path is calculated and the seed of the random generator is set to the perimeter value, which guaranties that the vegetation will be distributed randomly, but at each execution, for the same path, the vegetation objects will always be in the same place. To generate the vegetation, the algorithm gets the bounds of the area in order to know what distance needs to be covered on X and Z axis. Then, a cycle starts from the minimum bound until the maximum bound iterating at a value that depends on the density. The bigger the density, the lower this value and the more vegetation objects will be rendered. On each iteration, the algorithm generates a random value that identifies the probability of the object being generated at this position. This probability depends on the seed of the random generator. If the object is going to be placed, a random value is added to its position and then an instance of a vegetation object is created. This object needs to be placed on the ground taking in account the terrain elevation, so a ray is casted downward from the objects position X and Z and the highest Y value possible. If the ray hits the ground, the position of the vegetation object is equal to the hit point. Another condition mandatory to place a vegetation object, is that when casting the ray, it cannot hit any road or sidewalk, therefore the land use generation procedure needs to execute after all the roads are complete. When the area is covered, the FloorPlant can be destroyed, because it is no further needed. Figure 34 - Tree 3D model on the left and bush 3D model on the right Methodological Approach and System Architecture 65 3.5.13 Procedural Amenities or City Furniture An amenity is a community facility such as, garbage bins, bus stops, health facilities, education facilities, monuments, banks, toilets and any other kind of urban furniture. The amenity tag, is the tag that has the most keys on the OpenStreetMap system. That is why this project does not cover almost any of those amenities. There are hundreds of different objects possible and lots of combinations, making it a very complex task, with a large number of 3 dimensional models to group with each other, get the entire coverage of all the amenities. This cannot be accomplished on the context of this project, because hundreds of 3 dimensional models need to be found for free, with relative good quality and low polygon count and then, they all need to be processed, recognized, and merged with each other based on “if clauses”. All of this work, would require almost the same time this project took to be accomplished, so only very little different objects were considered, in this case, only the trash and recycle bins. This objects are placed on the map, but they do not correspond to a path, they correspond to single separate nodes. So, the algorithm just needs to identify the tag value and instantiate the corresponding object on the position of the node. Both of this objects, contain a rigid body and have colliders attached. The rigid body allows the object to behave physically, applying the gravity force and external forces upon collision. 3.5.14 Day and Night Cycle Driver awareness and reaction time are often affected by the global illumination and sun position relative to the driver’s eyes. When the sun is low, it can strike the driver directly in the eyes causing temporary blindness, making the driver not able to see, due to the intensity of the sun’s light, therefore there is an extremely increased difficulty in seeing pedestrians, traffic and objects. This project attempts to recreate such conditions. This can be found in more detail in the Visual Effects section, while this section’s objective is to describe the algorithms behind the daynight cycle and their capabilities. Figure 35 – Garbage Bin 3D models Methodological Approach and System Architecture 66 The sun is a game object that rotates on the X axis of the Unity’s scene. It rotates a percentage of the day’s duration every frame. The day’s duration is configurable on the main menu and its units are minutes. Having this in mind, it is possible to calculate the rotation of the sun on each frame: DayCycleInSeconds = dayCycleInMinutes (from the menu) * 60(minutes) RotationEachFrame = (360º/DayCycleInSeconds)*Time between 2 consecutive frames The RotationEachFrame is applied on the X axis every frame, making a smooth sun movement along the entire day. Unity has a feature called procedural skybox. This skybox is generated by a shader, whose values can be configured, such as the sky’s colour, atmosphere thickness, sun size and exposure and it can detect the main directional light on the scene (the sun), to create the sun’s representation on the it. As the sun rotates, the skybox shader blends its colours to recreate the actual sun effect on the atmosphere, meaning that, for instance, starting from noon, the skybox gets more orange and darker as the sun goes down. This algorithm also includes “events”. This events occur at a defined time of the day. There are two events, sunrise and sunset, determined by a percentage of the day’s duration from 0 to 1. The sun starts always at midnight, so this percentage is zero, at noon the percentage is 0.5, so from there it is possible to configure the time when this events happen by just changing the percentage, to when we want it to happen. These two event are used to send a message to the street lights. When the sunrise event occurs, the algorithm sends a message containing “off” to all the street lights, this will make all of them turn off. On the other hand, when the sunset event occurs, the algorithm sends the message “on” to all the street lights. Another important feature is the global illumination. In Unity 5, global illumination is not generated by any light source, instead it is generated by the skybox, by a colour or a predefined gradient and its intensity can be configured. In this case, the global illumination is being generated by the skybox. So, while the sun goes up, the global illumination starts going up to the maximum to create a nice and bright day, when the sun starts coming down, the algorithm does the opposite and the global illumination starts going down till the minimum, to create a really dark night. At night, the illumination depends only on the street lights and on the intensity of the moon’s light. This algorithm supports more than just a single sun, in fact it supports an unlimited number of suns. A sun can also act as a moon. This is because a sun is simply a directional light that is positioned at infinity, so its intensity, colour and flare can be configured to make it look like a moon. The moon is positioned on the opposite side of the sun and rotates at the same speed and axis as the sun, meaning that there will never be an eclipse. Another important feature is the starting time. On the main menu the user can configure the starting hours of the day, ranging from 00:00 to 24:00. When the user configures the starting time, Methodological Approach and System Architecture 67 the sun is rotated to the correct position, such as the moon, and the global illumination is configured accordingly as well. 3.5.15 Weather Cycle Weather conditions are the main reason of road accidents, right after driver neglects, such as excessive speed, driving drunk or lack of attention, so it is important to recreates a few weather conditions that affect visibility, pedestrian behaviour, road conditions and vehicle tire grip, causing the driver to increase the reaction time and the vehicle to increase braking distance and decrease its stability. The featured weather conditions are sunny, cloudy, rainy, and stormy and any of these conditions can have fog added to the mix. On the main menu, the user can select what weather condition he wants to happen on the simulation, as well as setting the weather changing probability. The weather changing probability, is a number ranging from 0 to 0.9 and controls for how long a condition is present until changing to another. The lower the change probability, the longer a condition will stay on the scene. The duration and the transition between conditions, also depend on the duration of the day, meaning the longer the day’s duration, the longer the conditions will maintain and the longer will be the transition between each one of them. On the other hand, let us say, for a full day cycle that lasts one minute and for a changing probability of 0.9, everything will happen very fast and for a changing probability of 0.1 the conditions will last longer, but the transitions will be as fast. Another less important aspect is the sound of the rain and the lightning, which contributes for a more immersive experience. There is also a flashing light when a lightning strikes. Regarding the vehicle, it is important to have different behaviours and responses on different weather conditions. When it is raining, the grip of the tires is reduced almost by half, like in real life, making the vehicle harder to drive. Having different tire responses, the vehicle’s response, acceleration and braking distance will be affected as well. The vehicle’s response will also depend on its wheel base, axel distance and drivetrain. To adjust the tire grip, instead of changing the tire friction values, it is easier to change the stiffness of the friction curves. The stiffness is basically a friction curve multiplier that increases or decreases all the values of the curve. The fog is one important factor, and Unity has a limitation in what comes to it. Unity renders fog as a post process image effect, it is rendered after everything else and in image space, so it picks every element of the scene, calculates its distance to the main camera and applies the fog according to the distance. The problem is that the skybox is not a game object, so no fog effect is calculated on it. The algorithms attempt to blend the skybox’s colour to a more or less uniform grey tone, to overcome this problem. Methodological Approach and System Architecture 68 3.5.16 Vehicle The idea is to have different types of vehicles with different characteristics that the user can select, on the main menu. For now, this project has a sports car, with rear wheel drive and a 4x4 pickup truck, with lifted suspension and larger wheels, in order to explore different wheel bases, axel distances, weights, braking distances, accelerations, drivetrains and sizes. a. Vehicle Selection In the main menu, the user has a button that sends him to the vehicle selection screen. This is a new scene, where the user can select the vehicle using the navigation arrows, left or right, to cycle through all the vehicles available. On the left bottom corner, there is a button to select the currently visible vehicle and go back to the main menu. There is also a small panel that shows the currently visible vehicle’s characteristics, such as dimensions, power and drivetrain. b. Vehicle Handling The vehicle handling depends on many parameters, especially the tire friction curves. These curves determine the vehicle’s tires capabilities and limits, in what comes to grip by identifying the slip values and force needed to make the wheels spin/slide, both sideways and forward. Each wheel has two friction curves, corresponding to the sideways and the forward grip. The suspension system is also part of the wheels. Each wheel has a suspension component that can be fully configured, including the suspension height, suspension damping and spring force. By adjusting these values, the vehicle’s stability and response to bumps changes, making it stiffer of softer on a bumpy road. The vehicle’s drivetrain is also a very important part, because it determines whether the vehicle is front wheel drive, rear wheel drive, 4x4 or all-wheel drive and depending on its differential, the vehicle assumes different driving strategies and behaviours. For instance, the front wheel drive (FWD) vehicles tend to suffer from understeer, while the rear wheel drive (RWD) tend to suffer from oversteer. The 4x4 or 4WD may suffer from both, but can easily find a balance to maintain maximum performance on corners and also have smaller acceleration times, due to the fact that the 4 wheels are pushing the vehicle. Just for reference, understeer is when the front wheels of a vehicle exceed the slip that provides maximum friction. When this value is exceeded, the friction values decrease meaning the vehicle starts sliding instead of cornering. On oversteer the same happens, but on the rear wheels, making the rear end lose forcing the driver to counter steer, to get back to control. The 4x4 is the midterm, because usually the differentials that connect the front and rear axles, attempt to distribute engine torque to the axel that needs it most and, therefore increase stability. A good example of a mechanical differential, is the Audi Quattro system that has a clutch that moves forward or backward, depending on the rotations of the axels. Methodological Approach and System Architecture 69 If the front wheels are spinning faster than the rear wheels, the rotational force of the differential pushes the clutch towards the rear axle, making the torque provided from the engine, go in greater percentage to the rear wheels. The inverse happens when the rear wheels have a greater rotation then the front wheels. c. Vehicle Engine The engine is responsible for producing the torque that is applied to the wheels, by means of a mechanical system. First of all, the engine has to be turned on. To turn on the engine the driver rotates the key that activates an electric system, capable of rotating the engine pistons and creating a spark on the ignition chamber. When the pistons start rotating, they force the intake valves and the fuel valves to open and close, depending whether the piston is up or down respectively. With the piston up and the valves open, the fuel enters the ignition camber and ignites due to the spark and the pressure, making the engine start running on its own. It runs on its own, because the force of the fuel explosion/combustion forces the piston down, making other pistons go up and the cycle repeats. This is a 2 stroke engine, because it only has 2 phases, the ignition and the exhaust phase. A four stroke engine, which is the most regular on vehicles, is capable of producing more power and efficiently, because it introduces 2 more phases, the admission and the compression. A four stroke engine, starts by sucking fresh air into the pistons (admission), while the piston goes down and, when it comes up, the compression phase occurs, because the air gets compress. When the compression is maximum, the fuel is ignited (ignition) and the piston goes down with the force of the explosion. When it goes down, the dirty air inside the camber, which is the result of the explosion, is decompressed and, due to the other pistons ignition, this piston returns up forcing the dirty air into the exhaust system (exhaust phase). The valves are mechanical systems that are connected to the engines camshaft and are fine tuned to perfectly control the entrance and exit of air and fuel. Some vehicles also have a turbo. A turbo is a system connected to the exhaust system that receives and takes advantage of the dirty air, to rotate a fan that has a tube connected to the admission system, to eject fresh air at a much larger rate. The more fresh air inside the chamber, the best the force of the explosion, therefore the better the performance. Another system to eject air into the cambers is the supercharger. A supercharger is usually found on American muscle cars and is outside the front bonnet. Inside, a supercharger has a fan that is directly connected to the camshaft, rotating at the same speed than the engine and sucking air from the outside at a rate as large as the engine’s RPMs. This kind of systems, may not be adequate for non-muscle cars, because a muscle car is known for not being fuel efficient, and since the supercharger is connected directly to engine, it drains out a bit of horse power, making the engine push harder, increasing fuel consumption. On the other hand, the turbo rotates due to the pressure of the air that comes out of the engine and, therefore does not drain any power from the engine. With the pistons moving, the camshaft rotates. To move the vehicle, there needs to be a drivetrain connected to the camshaft. This is done by using a belt that is connected to the camshaft Methodological Approach and System Architecture 70 and the drivetrain. The power is then transferred to the gear box. At the beginning of the gear box, right before the gears, there is the clutch and the fly wheel. The flywheel is always rotating at the same rate has the engine, while the clutch is controlled by the clutch pedal. When the clutch is pressed, it makes the engine run freely, by separating the connection on the flywheel. When the clutch is released, it makes contact with the fly wheel and starts skidding until it reaches the same speed. This is why vehicles stall. If the clutch pedal is released to soon, the clutch does not have time to gain enough rotation and the system blocks, because the transferred power is not enough to rotate the wheels. Since the engine is always running, just as the flywheel, the pistons are forced to slow down to a rate that the engine cannot enter in a cycle, therefore the vehicle stalls. With the clutch engaged, the power starts being transferred to the gear box. A gear box, is a complicated and complex system that consists on multiple gears and can be manual or automatic with double or single clutch. Inside the cockpit, the driver selects the gears. Each gear, has its own ratio. This ratio is used to multiply the torque that comes from the engine. By multiplying the torque, it divides the maximum speed the vehicle can reach. This is, the bigger the ratio, the bigger the torque, but the smaller the speed the vehicle can reach. By this logic, the first gear is the one with a bigger ratio, therefore it is used to start moving the vehicle. After the gearbox, the torque is transferred to the differentials of the traction wheels. The differential, makes possible that the wheels of an axel, spin at different velocities, making cornering and getting out of slippery surfaces much easier. The differential selects the power that needs to be transferred to each wheel, depending on the type of differential. This was just to explain the basis on how a vehicle works and to better understand how the vehicle scripts is coded, because it attempts to follow the closest as possible the above procedure. Figure 36 - Triumph Daytona 675 - Naturally Aspirated Engine (Adapted from [101]) 675cc 12v in line triple four-stroke capable of 123 horse power Methodological Approach and System Architecture 71 Fuel engines have multiple parts creating frictions and have recovery times from low to high throttle, or vice versa, and additional torque created by the turbo system that enters the mix, in a non-seamless way, therefore these engines generate a torque curve that is not constant. On the other hand, electric engines produce almost a constant torque curve, providing almost full torque at the entire RPM range, because ,not only they are made of a single rotating piece, but also because the system generates maximum electric intensity and voltage almost instantly when the pedal is fully pressed. The torque curve, in this project, is represented on a file inside “CarCurves” and consists on an association RPM-Torque. What this means, is that for a certain RPM of the engine, it generates a determined amount of torque. The intermediary values of torque are interpolated. The vehicle script loads this torque file and stores this information on lists. It has also defined the gear box ratios, max and idle engine RPM, brake distribution, brake force, handbrake force, drag constants for air and roll resistance, centre of mass, clutch engage time, maximum steering speed, maximum steer angle and so on. First, the algorithm calculates the drag forces that depend on the velocity, to further apply. Then, it manages the gear box. If the engine’s RPM are close to the maximum, then it is time to shift up, on the other hand, if the RPMs are close to the idle, then it is time to shift down for increased power. When the gear shifts, the clutch is applied and the vehicle stops accelerating for as long as the clutch needs to fully engage. After the gearbox, the algorithm calculates the engine’s RPMs based on the motor wheels rotation. It makes an average of the motor wheels RPMs and multiplies it by the ratio of the current gear and by the differential ratio. If these rotations are below the idle RPM value of the engine, then they must be forced to this value, otherwise the vehicle stalls. When the algorithm knows the RPMs, it calculates the torque for the given RPMs. Remember that the torque values were loaded into lists, so the algorithm interpolates between the RPM and torque values, to get the current torque. Known the torque, it is time to apply the drag forces that should point backwards the vehicle. The RPMs should also be below the maximum engine RPM. If they are above this value, then their value has to be forced down. On real life, when this happens the engine struggles to keep pushing, but then, the electronic system comes in and drops them down to avoid any damage. To simulate this, a few RPMs are decreased (around 500) from the actual RPMs. As the driver presses the accelerator pedal, the valves open accordingly and the engine starts producing power. In Unity, the pedal input is a value between zero and one, therefore the power applied on the wheels is equal to the current torque, times the pedal input. The torque is only applied to the motor wheels. Methodological Approach and System Architecture 72 d. Vehicle Brakes The vehicle brakes have an important role in stopping the vehicle, in the shortest distance possible. To decrease the stopping time, the brake system has to be well balanced, taking in account the vehicle’s mass and the tires characteristics. Nowadays, there are electronic systems that allow to reduce the braking distance on different road conditions, such as ABS. Anti-lock braking system, makes sure the wheels do not lock. When the wheels lock, the vehicle starts skidding, increasing the stopping time and not allowing the vehicle to corner. With this system installed and active, the wheels will never lock, regardless of the road conditions, improving safety, stability and cornering. A braking system is operated by the brake pedal, which compresses a fluid that circulates inside pipes that are connected to the brakes. Most of the cars have disk brakes, but some of them still have drum brakes in the rear wheels. The disk brakes, have a calliper that when the fluid pressure increases, pushes the pistons inside the callipers against the disk, creating a massive friction. The drum brakes work differently. They have and external part, called drum and inside it there are pedals and a spring. When the fluid pressure increases, the pedals are pushed against the walls of the drum. The disk brakes are better, because the disk is refreshed by the cold air coming from the front when the car moves, on the other hand the drum brakes do not breathe so much, so their temperature increases more. The temperature directly influences the way the materials generate friction. Usually, the better balance of temperature and performance is in the mid-range of the minimum brake temperature and the maximum temperature they can reach. Therefore, when using the brakes too much, they can overheat and lose performance, increasing the braking distance. On the other hand, if the brakes are cold, their efficiency is reduced, but not so much as if they were hot. Figure 37 - Drum Brake Parts (Adapted from [102]) No brake pressure on the left, almost full brake pressure on the right Methodological Approach and System Architecture 73 Also, to further decrease the braking distance, it is possible to configure the brake distribution. Brake distribution allows to distribute the pressure of the fluid on the front and rear axles. For better efficiency, the brake distribution is usually 60% in the front wheels and 40% on the rear. When the vehicle brake the suspension in the front gets compressed due to the inertia, while the rear suspension gets decompressed, making the centre of mass move forward. To compensate the movement of the centre of mass, the front wheels need to have more braking power sense they are sustaining most of the mass of the vehicle while braking. When the rear wheels brake more than the front wheels, the vehicle may suffer oversteer and spin. This project does not feature the factor of the braking temperature, but features the brake distribution, which is predefined to 60 in the front and 40 in the rear. The total braking force is defined by multiplying the brake force by the brake pedal input that ranges from zero to one. Then, this force is applied to the front and rear wheels, multiplying by the respective distribution in percentage. e. Vehicle Steering Vehicle steering is responsible for steering the steerable wheels on the direction the driver wants to go. To do so, there is a shaft connected to the steering wheel that rotates a gear. This gear is connected to the steerable wheels axel that moves on the opposite direction of the wheel rotation. By moving in the opposite direction, the wheels will turn to the side the steering wheel is pointing. This system is usually very heavy, requiring a lot of effort from the driver, especially with the vehicle stopped. To overcome this problem, the vehicles come with a systems installed called power steering that can be hydraulic or electric. This system aids the driver by generating the necessary force to rotate the wheels, while the driver rotates the steering wheel. Figure 38 - Disk Brake Parts (Adapted from [103]) Methodological Approach and System Architecture 80 The implementation is fully detailed, in order to help any one that wants to continue extending and enhancing this project by providing extremely detailed information on how the modules interact and how the algorithms work. Thanks to the first semester it was possible to start making prototypes and exploring possibilities that leaded to the abandonment of the SUMO integration, because it is impractical on this context. Therefore, regardless of any delays it was much easier to develop the rest of the project by following regular and fixed work hours and methods. It was not an easy task, there were moments of victory, failure, happiness, frustration, sadness and so on, but it was a journey where a lot was learned, ties where created and develop and the human factor was enriched. Results 81 Chapter 4 Results This chapter describes all the information related to the results of the implementation section in Chapter 3. It is divided in multiple sections, each one corresponding to the respective section of the implementation. All the imagery was gathered from the final version of the project and represent the actual quality of the final product. Nevertheless, the images were gathered on the Unity Editor, therefore they have reduced graphical quality compared to the exported executable. Although this was a very ambitious project, all the proposed goals were completed and the results are very much promising. They might not be perfect, but they do the job and establish the beginning of an important tool that can be further enhanced and expanded as mentioned before. The images only show a few possible scenarios, but there is a lot more content to see than what these images suggest. Results 82 4.1 Main Menu The main menu is simple and easy to use as seen on the picture above. It provides access to the options, the vehicle selection screen and the start button that only works if the input box is filled with a valid world location by address or coordinates. 4.2 Main Menu Possible Configurations Figure 41 - Main Menu Figure 42 - Advanced Options Screen Results 83 This screen is the advanced options menu, accessible on the main menu. It provides a decent amount of possible configurations on a stylish layout. It allows the user to select the render options, the type of vehicle damage, the weather conditions that can occur, the weather changing probability, the pedestrian density, the time of day and its duration and on the right all the controls are displayed. The controls can be changed before the application start, although any alteration will not affect the current controls displayed. 4.3 UnitySlippyMap This interface is a completely new scene that display on location typed on the input box of the main menu. This interface allows the user to navigate the world map by zooming in or out and dragging the map with the left button of the mouse and select a bounding box with the right button of the mouse. To select a location the user simply uses the target in the middle as a reference to place the initial marker by right clicking. The second right click places the second marker and two more markers that will close the bounding box. The first and the second markers represent the diagonal of the box. If the user is not satisfied the next right click will place a new first maker and overwrite the previous selection. On the top of the image there are four buttons. The first gets the user back to the main menu, the second selects the location inside the bounding box and advances to the next step, the 2D/3D button allow to switch between the top-down view and the 45º view and the last button changes between different tile providers. Although the interface is simple the bounding box selection may not be very intuitive. But this is because all the code was adapted from an initial code without any documentation, making Figure 43 – UnitySlippyMap Four points define de bounding box around Suzuka Race Tack Results 84 it hard to understand the flow of the scripts and how to integrate it on the context of this project. Nevertheless, the integration made, is solid and functional, providing the possibility to select any world location. 4.4 Elevation Data The elevation data used has a 90 meter resolution on both longitude and latitude, but this has a disadvantage, because the elevation values belong to an exact point of the Earth. Also, the Earth is not a perfect sphere, it is a spheroid, which means that the distance covered by 5º will not be the same across the entire surface. This means that the elevation data and the real world data have different scales, therefore the real elevation is not apart 90 meters across the entire surface of the Earth. This introduces an offset, causing the elevation to be misaligned with the real elevation. As you can see on the image above, the river starts climbing the mountain. Obviously, this offset is noticeable, because rivers always go down, otherwise they make their way through. This couldn’t be corrected, because there was no time to, but in certain locations the elevation offset is not noticeable, rather because it is too small or there is not a great elevation variance. Regardless of the offset problem, the elevation model looks fairly realistic and approximate to the real world. Figure 44 - Elevation Data Small portion of the Grand Canyon, USA Results 85 4.5 Location Data Parsing and Satellite Image Gathering The location data parser is responsible for organizing all the information so it can further be procedurally generated. By the image above, it is possible to see all the generated models such as roads, footways (red), paths, vegetation and the walls of the buildings between others. The satellite imagery is gathered from Google Maps, as it contains the Google mark on the bottom right corner. In most of the cases the imagery and the generated meshes are well aligned. On other cases there may be an error of around 5 meters, which is not major. 4.6 Terrain Elevation Figure 45 - Bird Eye View of the Suzuka Race Track, Japan Figure 46 - Highest peak of the Mount Everest, Himalayas Results 86 The terrain elevation suffers from the problem of the differences of scale between the location data and the elevation data, causing a more or less small offset between the position of the elevation points and the real world. Despite this problem, the elevation system can generate realistic results especially having all the satellite imagery aligned. 4.7 Procedural Roads The procedural road produce a very smooth and satisfying result. They perfectly adapt to the ground, always following the path and not producing any kind of sharp corners that are not smooth or seamless. Also, they can preserve their width along the entire path. The sidewalks and the sidewalk steps go along with the road, always keeping the correct height, width and shape. The only problem with the roads is the fact that different roads overlap. What this means is that on the intersections, two or more roads take the same place and the sidewalks go into the middle of the other roads. This project does not support intersection handling, for now. Figure 47 - Procedural Roads with Street Lights Figure 48 - Procedural Roads Overlap Problem Results 87 4.7.1 Road Enlightenment As expected, all the street illumination is placed along the enlightened road, at a fixed distance on each side, providing enough light to see the pedestrians and the road itself. Without the illumination the user could only rely on the vehicle’s lights, which may not be enough for an environment where pedestrians are walking. They turn on and off when needed, in this case when it gets darker or brighter respectively. Regarding performance, having dozens of lights casting dynamic shadows at the same time causes the frame rate to drop. To counterbalance this fact, it is possible to remove the shadow casting on the initial screen before starting the project by reducing the graphical quality. Figure 49 - Procedural Roads Night Illumination Results 88 4.8 Pedestrians Pedestrians populate the procedural scene making it harder to drive around without making any casualties. As expected, the pedestrians are capable of walking around on the sidewalks and on the footways and are physically interactive with the vehicle by applying rag-doll physics. Also, they attempt to avoid obstacles in front of them, including each other. When they are off a sidewalk or a footway, they try to run to the safety of the closest sidewalk or footway. In addition, when they cross over another walkable surface, they decide whether to change their path or not. After implementing the LOD controller, there was a massive improvement on the performance of the application. By using a LOD controller, the pedestrians disappear according to the distance between them and the main camera. By just showing a few pedestrians instead of showing the further away, the frame rate gets a very big improvement. Figure 50 - Pedestrians Walking and Getting Hit By Vehicle Results 89 When the pedestrians get hit, they never damage the vehicle, regardless of the force of impact. This is because they are careless, although they analyse the path in front of them, they have no notion of what surrounds them, so they do not react or predict the vehicle’s movement in order to avoid it. Although the AI is simple and functional, it still needs a lot more work in order to achieve a much more complex and smart intelligence. 4.9 Procedural Buildings As mentioned before, buildings have no rooftops. This is a driving simulator, everything happens on ground level, so, to save resources, there is no need to implement the rooftops. This picture was taken on the editor, with a free camera and, therefore, it can be moved anywhere. To show the potential of the building generation procedure, the picture was taken from a bird eye view. By analysing the picture, the results seem very good, each building is well suited to the ground with a basement, multiple floors, a large number of windows and a different material selected depending on its perimeter. It is also possible to see different kinds of buildings, the regular ones and the skyscrapers. Skyscrapers are buildings over 15 floors and do not have window meshes, instead they have a texture of glassy windows. Unfortunately there is lack of information to generate accurate buildings. The file taken from OpenStreetMap barely contains information regarding the existence of buildings or their characteristics. This is why it is possible to see buildings on the satellite images that were not generated. Figure 51 - Procedural Buildings, Reconstructing Avenida dos Aliados, Porto, Portugal Results 96 4.15.5 Vehicle Damage The image on the left, is an example of the mechanical damage, the deformation is the same as on the visual damage, but the wheels flew off not allowing the vehicle to move anymore. When the vehicle hits something, collision debris come out on the colour of the vehicle and a collision sound is produced. There are 5 different collision sounds, and the sound that plays corresponds to the respective collision force. The debris amount depends on the force of the collision as well. Also, when the vehicle scraps objects, sparks are produced on the collision point and their emission rate depends on the velocity of the vehicle. The damage looks extremely good, because the vehicle has no deformation cap, meaning that it can fully deform without any limitation. Figure 59 - Vehicle Damage System Mechanical Damage on the Left and Visual Damage Only on the Right Results 97 4.16 Visual Effects Visual effects massively contribute to the user immersion on the experience, but they come with a cost, the drop on the frame rate. Nevertheless, Unity includes some extremely optimized visual effects that work as a post process effect applying their calculations to the final image gathered by the camera. As seen along this chapter, on the pictures there are multiple visual effects taking place. For instance, on the image above the sun is setting behind the trees, creating a very bright bloom that spreads across the dirty lens of the camera. As the sun light is right in front of the user’s eyes he can barely see the road ahead, just like in real live he gets flared. When running with the high performance graphics card, the drop on the frame rate is almost unnoticeable, but with the integrated graphics card the visual effects drop the frame rate a bit. When decreasing the visual quality down, these effects turn off granting a much smoother experience, especially with Oculus Rift where the frame rate needs to be high to avoid any motion sickness. The visual effects used in this project, make it look like a high end product with AAA graphical quality. Figure 60 – Beautiful Effect of the Sun Setting Behind the Trees Using Cockpit View Results 98 4.17 Procedural Scene Export To export a scene, the user simply has to press Escape button to pause the application and click on export button. The exported scene goes under “ExportedObj” on the project folder. The resultant file of the export operation is a .obj 3D file that can be opened and edited by an external application, such as blender seen on the image above. Then, the artist can modify any structure on the scene, since they are grouped on a hierarchy and separated by their types and from each other, any existent object is accessible independently from the others. The 3D model can then be exported to a different file type, or not, and be used on any other application for any kind of use. The .obj file exported, is usually a very big file for a small scene (around 1GB), and even larger for a bigger scene, obviously. The problem is that the application takes a huge amount of time to export the entire scene and the user may think that the application is blocked. This type of file format is usually very large for even simple meshes. The application could export the scene to a different smaller extension, but the .obj extension is much easier and faster to implement, due to the time available to develop all the features of this project. Still, the application never blocked so far on the export process, so if the user is patient the scene is successfully exported and can then be used as desired. Figure 61 - Exported Scene Edition in Blender Results 99 4.18 Summary From this chapter, we can conclude that the results look very appellative and adequate, especially taking in account the time available to develop all the features. All the features of the project are solid and work well with no major bugs or glitches, meaning that the application is functional. About practicality, there are parts that could be more accessible to the user in terms of ease of use, such as the UnitySlippyMap bounding box selection and the export process that makes the user think that the program blocked while exporting. Although it is solid and functional it is not perfect, there is always room for further improvement and expansion, but that is also a goal of this project, distribute freely the source code so any one can expand and modify the application their own way, to suit their needs. The next chapter describes the possible future work and enhancements. In conclusion, this project creates the beginning of a new open source tool that is capable of reconstructing real world locations from anywhere, populating them with pedestrians, allowing the user to drive a vehicle, simulating different day and night conditions and weather cycles and allowing to export the entire scene to be externally edited by another program. Conclusion and Future Work 100 Chapter 5 Conclusion and Future Work Procedural modelling is a very easy concept, with understandable and easy to implement algorithms. The hard part is the vertex manipulation, because the magic of generating a beautiful mesh is not the algorithm itself, but the way it manipulates vertices. To create such meshes, the vertices need to be merged and used by each other, making this a very hard procedure to implement. This project attempts to create the better meshes possible taking in account the time available to develop all the features. The procedural modelling algorithms can still be extensively expanded and enhanced. Nevertheless, the results are very good and convincing, generating an appellative and realistic environment, populated with pedestrians, which can be exported and edited by an external application. Having so, it is possible to configure many parameters that will allow to create different driving conditions, to conduct different kinds of studies. Also, if the user is not satisfied, the project can be easily extended to suit any kind of needs. 5.1 Goal Satisfaction All the main goals of this project are fully satisfied, with the exception of the traffic simulator that was abandoned, because it is impractical to integrate an application that runs in real time with an average of 30 fps with an application that does not have a continuous time step, through a TCP channel. The overhead would be so large that the application would be unusable. On the other hand, the road system allows the implementation of a traffic simulator within Unity, without recurring to external applications. Although, this was not done, because it would take too much time. Only the vehicle AI would take around 2 months to be completed, in order to be fully Conclusion and Future Work 101 functional and intractable with the physics engine. This project has too much content to care of that there was no time at all to create an internal traffic system. On the other hand, it includes a pedestrian system that makes pedestrians move around on the sidewalks and footways, avoiding objects and each other and allows them to interact with the vehicle, using ragdoll physics. What this means is that when the vehicle hits them, they are thrown away like a puppet. Every other proposed feature is fully implemented and functional. There is the procedurally generated terrain with satellite images as a texture, the satellite images are downloaded in runtime, there are procedural roads with sidewalks that adapt to the terrain elevation, the roads have different materials, different types and characteristics such as width, surface, the existence of sidewalks and street illumination, the procedural buildings have varying textures, heights, windows and a basement that adapts to the terrain, the vegetation spreads around the entire scene, a full day and night cycle simulates different light conditions, street illumination that turns on or off depending on the time of day, the full weather cycle that simulates sun, clouds, rain, fog and storm with lightning effects that cast a very bright light and noise, two different vehicles that can be selected on the selection screen with different torque curves, dimensions, suspension setting and drivetrains, the vehicle selection screen is a well detailed warehouse, full vehicle damage that can be switched off, visual only or mechanical where the wheels fly off and get misaligned or the suspension breaks, vehicle collision particles including sparks and flying debris, vehicle collision sounds depending on the force of the impact, there is also a map on the main menu where the user can select the location he wants to download, the elevations data is downloaded in runtime, if not already existent in the database, the project offers the possibility to export the generated scene (by pressing Escape for the in-game pause menu) to .OBJ 3D file, to modify the scene on an external 3D application, there are also a few amenities such as garbage bins around the map, procedural guard rails, walls and fences that enrich the scene, there is a water simulation system that simulates the waves on the ocean and provides the best looking water as possible, there are a good amount of image effects applied to every camera such as bloom, vigneting, chromatic aberration, lens dirtiness, HDR camera rendering and linear colour space, there is the Oculus Rift integration that allows the user to drive the vehicles using the cockpit camera with the Oculus Rift on extended mode and finally there is an options menu where the user can configure parameters such as the pedestrian density, the weather changing probability, the weather conditions that can occur, the rendering options that define what objects are going to be procedurally generated, the vehicle damage type and the day and night cycle duration and starting time. This is a very complex project that attempts to generate serious driving games scenes and possibilities of configurations, so the user can simulate any kind of conditions and situations possible. It marks the beginning of a new tool to quickly generate scenarios of the real world that can be exported to be used in other projects or applications and it can be enhanced and further expanded to fulfil the desired needs. All the features are fully implemented and functional, providing a complete ecosystem of world location generating tools. By means of procedural Conclusion and Future Work 102 modelling, the process of generating an entire scene is made fast, simple and free, because everything is available to anyone who wants to expand or enhance the project. 5.2 Future Work Although all the goals have been implemented, it is possible to improve some of them and implement new ones. Regarding the procedural elevation tiles, maybe not now, but in the future, the source of the elevation data can be changed to a source that offers better data resolution. The same applies to the satellite imagery, for now the zoom level used is the maximum possible, but in the future eventually Google will have HD imagery. Another thing about the elevation data, is that the elevation data provided from CGIAR-CSI follows a square grid that is constant around the entire earth. On the other hand, the earth is an ellipsoid and both OpenStreetMap and Google Maps use an ellipsoid model. Therefore, there is an offset between the elevation presented on the elevation tiles and the real elevation. The solution is to somehow find the relation between the different sources and apply a negative offset to compensate the differences. Procedural roads were a complicated task, due to the fact that they have to follow a predefined path, so they have to be built along them. They still have a problem, which is road overlap. When two or more roads intersect with each other, they overlap, making the sidewalks go over the intersection. It is possible to avoid this, but it would require a lot of effort and time. Procedural buildings do not have neither a roof nor interior. Since this is a driving simulator, everything happens on ground, but it can still be expanded to a flying simulator and then, the rooftops will be needed. But once again, for the purpose of this project the rooftops are not needed, although they can easily be implemented. Regarding the interiors, it comes to the same basis, the interiors are not required, but let’s say other developer needs to expand the project for firefighting simulation. Then the doors need to open, there needs to be interior decoration, stairs and functional windows. Unfortunately, OpenStreetMap does not have information of all the buildings of a location, only the most relevant. So, an interesting improvement could be to attempt to extract the rooftops from the satellite imagery to then render the missing buildings. The vegetation itself is very simple, there are just 2 different models that are instantiated. It was done like this to save the maximum computer resources as possible. Nevertheless, the project is configured to work with an unlimited number of different 3D models for the vegetation, this function is only deactivated for the resource saving. Although, it can still be activated easily. There are shaders and other workarounds that can do this, but more time was needed to fully investigate. Conclusion and Future Work 103 The number of different amenities can also be increased by finding different 3D models for the different OSM tags that constitute the amenities and somehow group those when needed. The pedestrians are not very smart, although they run from the road to the sidewalk, change paths, avoid each other’s and the obstacles and follow along the sidewalks, they do not have much awareness of the world around, they only see in front of them, meaning that they do not attempt to run away from the car. So, their AI still needs a few weeks of work, but it is not bad at all. The project features no traffic system. One major improvement would be to take advantage of the well-known vertices of the road and create a traffic system. This may seem easy, if we want a simple block that represents the vehicle to follow a path, but when the vehicles have wheels and follow a full physics simulation, the work is much harder, because the input for the brakes and gas pedal needs to be controlled based on the surrounding environment, as well as the steering input. Although it is not impossible, only very complex, so, maybe more than a month is needed. The list of future work above, is just a small example of maybe the most important features, but this kind of project has the possibility to be expanded in millions of ways, because, not only it is supposed to be a tool to help conduct experiments, but also because it was built on a powerful game engine. 104 References [1] J. Gregory, Game engine architecture. 2009. [2] I. Millington, “Game Physics Engine Development,” J. Vis. Commun. Image Represent., vol. 18, p. 481, 2007. [3] GameWorks, “GameWorks PhysX Overview,” 2014. [Online]. Available: https://developer.nvidia.com//gameworks-physx-overview. [Accessed: 26-Nov-2014]. [4] Havok, “Product Overview | Havok,” 2014. [Online]. Available: http://www.havok.com/products. [Accessed: 26-Nov-2014]. [5] H. Bao, Real-time graphics rendering engine. Springer Science & Business Media, 2011. [6] Mechanical Simulation Corporation, “Driving Simulators (CarSim and TruckSim),” 2014. [Online]. Available: http://www.carsim.com/products/ds/index.php. [Accessed: 13-Nov-2014]. [7] Toyota Motor Corporation, “Toyota Global Site History of Toyota,” 2014. [Online]. Available: http://www.toyotaglobal.com/innovation/safety_technology/safety_measurements/driving_simulator.html. [Accessed: 13-Nov-2014]. [8] I. 2007: 6th I. Conference and ... S., Entertainment Computing - ICEC 2007: 6th International Conference, Shanghai, China, September 15-17, 2007, Proceedings. Springer Science & Business Media, 2007. [9] J. Slob, “State-of-the-Art driving simulators, a literature survey,” DCT Rep., no. August, 2008. [10] I. Doron Precision Systems, “Doron Precision Systems |,” 2011. [Online]. Available: http://www.doronprecision.com/. [Accessed: 28-Dec-2014]. [11] IRacing, “Online Racing - What is iRacing | iRacing.com,” 2014. [Online]. Available: http://www.iracing.com/. [Accessed: 11-Nov-2014]. References 105 [12] iRacing, “See what the profesionals are saying | iRacing.com,” 2014. [Online]. Available: http://www.iracing.com/testimonials/. [Accessed: 11-Nov-2014]. [13] Slightly Mad Studios, “New Project CARS Weather Previews – WMD Portal,” 2014. [Online]. Available: http://www.wmdportal.com/projectnews/new-project-cars-weatherpreviews/. [Accessed: 12-Nov-2014]. [14] Slightly Mad Studios, “Inside Project CARS Seta Tire Model – WMD Portal,” 2014. [Online]. Available: http://www.wmdportal.com/projectnews/inside-project-cars-setatire-model/. [Accessed: 12-Nov-2014]. [15] Slightly Mad Studios, “Game Info - Project CARS,” 2014. [Online]. Available: http://www.projectcarsgame.com/game-info.html. [Accessed: 12-Nov-2014]. [16] Forward Development, “City Car Driving 1.3 Description.” [Online]. Available: http://citycardriving.com/products/citycardriving. [Accessed: 12-Nov-2014]. [17] D. Krajzewicz, G. Hertkorn, C. Rössel, and P. Wagner, “Sumo (simulation of urban mobility),” in Proc. of the 4th Middle East Symposium on Simulation and Modelling, 2002. [18] D. Toffin, G. Reymond, a. Kemeny, and J. Droulez, “Influence of steering wheel torque feedback in a dynamic driving simulator,” … Driv. Simul. …, vol. 2003, no. 1, pp. 1–11, 2003. [19] T. R. M. Coeckelbergh, W. H. Brouwer, F. W. Cornelissen, P. Van Wolffelaar, and A. C. Kooijman, “The effect of visual field defects on driving performance: a driving simulator study.,” Arch. Ophthalmol., vol. 120, no. 11, pp. 1509–1516, Nov. 2002. [20] P. R. Desai, P. N. Desai, K. D. Ajmera, and K. Mehta, “A Review Paper on Oculus RiftA Virtual,” Int. J. Eng. Trends Technol., vol. 13, no. 4, pp. 175–179, 2014. [21] E. Zeeb, “Daimler ’ s New Full-Scale , High-dynamic Driving Simulator – A Technical Overview,” Driv. Simul. Conf. 2010 Eur., pp. 157–165, 2012. [22] G. Kotusevski and K. Hawick, “A review of traffic simulation software,” vol. 13, pp. 35–54, 2009. [23] L. S. Passos, R. J. F. Rossetti, and Z. Kokkinogenis, “Towards the next-generation traffic simulation tools: a first appraisal,” 6th Iber. Conf. Inf. Syst. Technol. (CISTI 2011), pp. 1–6, 2011. [24] T. V. Mathew and K. Rao, “Microscopic traffic flow modeling,” Mumbai India, 2007. [25] T. V. Mathew, “Lecture notes in Traffic Engineering And Management,” 2014. [Online]. Available: http://www.civil.iitb.ac.in/tvm/1111_nptel/535_TrSim/plain/plain.html. [26] A. Wegener, M. Piórkowski, M. Raya, H. Hellbrück, S. Fischer, and J.-P. Hubaux, “TraCI,” Proc. 11th Commun. Netw. Simul. Symp. - CNS ’08, p. 155, 2008. Table of Contents 112 Table of Contents Introduction ....................................................................................................................... 1 1.1 Motivations and Goals .................................................................... 1 1.2 Structure of the Dissertation ............................................................ 2 Literature Review .............................................................................................................. 3 2.1 Game Engines ................................................................................. 3 2.2 Simulation Engines ......................................................................... 4 2.3 Physics Engine ................................................................................ 4 2.3.1 PhysX .............................................................................................. 4 2.3.2 Havok .............................................................................................. 5 2.4 Render Engine ................................................................................. 5 2.5 Driving Simulators .......................................................................... 5 2.5.1 Entertainment .................................................................................. 6 2.5.2 Research .......................................................................................... 6 2.5.3 Training ........................................................................................... 6 2.5.4 The Latest Driving Simulators ........................................................ 7 2.5.5 Driving Simulator Peripherals ....................................................... 10 a. Steering Wheels 10 b. Vibration Chairs 11 c. Force Dynamic Simulator 11 d. Big Round Screen 12 e. Surround Sound System 13 f. Head Tracking 13 g. Real Vehicles 14 h. Extreme Experience 14 2.6 Traffic Simulators ......................................................................... 14 Table of Contents 113 2.6.1 Microscopic ................................................................................... 15 2.6.2 Macroscopic .................................................................................. 15 2.6.3 Mesoscopic .................................................................................... 15 2.6.4 Hybrid ........................................................................................... 16 2.6.5 Nanoscopic .................................................................................... 16 2.7 Procedural Modelling .................................................................... 17 2.7.1 Procedural Modelling and Games ................................................. 17 19 2.7.2 Procedural Modelling Techniques................................................. 20 a. L-Systems 20 b. Fractals 21 c. Generative Modelling Language 22 2.7.3 Procedural Modelling Main Usage................................................ 22 Terrain 23 a. Height Maps 24 b. World Machine 25 c. World Composer 26 Flora 27 Buildings 28 a. Geometric Primitives 28 b. L-Systems 28 c. Building Generation Tools 29 Roads 29 a. Grid Layout 29 b. L-Systems 29 Cities 30 a. Esri CityEngine 30 2.7.4 Extracting Rooftops....................................................................... 31 2.8 Tools to be Used ............................................................................ 32 2.8.1 Unity 5 ........................................................................................... 32 2.8.2 UnitySlippyMap ............................................................................ 32 2.8.3 Open Street Map............................................................................ 33 2.8.4 Google Static Maps API ................................................................ 33 2.8.5 SUMO Simulation of Urban MObility .......................................... 33 2.8.6 CGIAR-CSI ................................................................................... 33 2.8.7 7zip ................................................................................................ 34 2.8.8 Blender .......................................................................................... 34 2.9 Related Work ................................................................................ 34 2.10 Summary ....................................................................................... 35 Methodological Approach and System Architecture ................................................... 36 Table of Contents 114 3.1 Problem to Solve ........................................................................... 36 3.2 System Architecture ...................................................................... 38 3.3 Unity3D + SUMO (Simulation of Urban MObility) ..................... 39 3.4 Development ................................................................................. 40 3.5 Implementation.............................................................................. 42 3.5.1 Main Menu .................................................................................... 42 3.5.2 Main Menu Possible Configurations ............................................. 42 3.5.3 UnitySlippyMap ............................................................................ 43 3.5.4 Elevation Data ............................................................................... 44 3.5.5 Location Data Parsing ................................................................... 45 3.5.6 Satellite Image Gathering .............................................................. 47 3.5.7 Terrain Elevation ........................................................................... 48 3.5.8 Procedural Roads........................................................................... 50 a. Road Enlightenment 54 3.5.9 Pedestrians ..................................................................................... 55 3.5.10 Procedural Buildings ..................................................................... 58 3.5.11 Procedural Barriers ........................................................................ 61 3.5.12 Procedural Land Uses.................................................................... 62 3.5.13 Procedural Amenities or City Furniture ........................................ 65 3.5.14 Day and Night Cycle ..................................................................... 65 3.5.15 Weather Cycle ............................................................................... 67 3.5.16 Vehicle .......................................................................................... 68 a. Vehicle Selection 68 b. Vehicle Handling 68 c. Vehicle Engine 69 d. Vehicle Brakes 72 e. Vehicle Steering 73 f. Vehicle Illumination 74 g. Vehicle Cameras 75 h. Oculus Rift Integration 76 i. Vehicle Damage 77 3.5.17 Visual Effects ................................................................................ 77 3.5.18 Procedural Scene Export ............................................................... 79 3.6 Summary ....................................................................................... 79 Results .............................................................................................................................. 81 4.1 Main Menu .................................................................................... 82 4.2 Main Menu Possible Configurations ............................................. 82 4.3 UnitySlippyMap ............................................................................ 83 4.4 Elevation Data ............................................................................... 84 4.5 Location Data Parsing and Satellite Image Gathering .................. 85 Table of Contents 115 4.6 Terrain Elevation ........................................................................... 85 4.7 Procedural Roads........................................................................... 86 4.7.1 Road Enlightenment ...................................................................... 87 4.8 Pedestrians ..................................................................................... 88 4.9 Procedural Buildings ..................................................................... 89 4.10 Procedural Barriers ........................................................................ 90 4.11 Procedural Land Uses.................................................................... 90 4.12 Procedural Amenities or City Furniture ........................................ 91 4.13 Day and Night Cycle ..................................................................... 91 4.14 Weather Cycle ............................................................................... 92 4.15 Vehicle .......................................................................................... 92 4.15.1 Vehicle Selection........................................................................... 93 4.15.2 Vehicle Dynamics ......................................................................... 94 4.15.3 Vehicle Illumination ...................................................................... 95 4.15.4 Oculus Rift Integration .................................................................. 95 4.15.5 Vehicle Damage ............................................................................ 96 4.16 Visual Effects ................................................................................ 97 4.17 Procedural Scene Export ............................................................... 98 4.18 Summary ....................................................................................... 99 Conclusion and Future Work....................................................................................... 100 5.1 Goal Satisfaction ......................................................................... 100 5.2 Future Work ................................................................................ 102 References ...................................................................................................................... 104 Table of Contents .......................................................................................................... 112