The Smartphone-Based Offline Indoor Location Competition at IPIN 2016: Analysis and FutureWork

sensors Article The Smartphone-Based Offline Indoor Location Competition at IPIN 2016: Analysis and Future Work Joaquín Torres-Sospedra 1,†,*, Antonio R. Jiménez 2,†, Stefan Knauth 3,†, Adriano Moreira 4,†, Yair Beer 5,†, Toni Fetzer 6,†, Viet-Cuong Ta 7,†, Raul Montoliu 1, Fernando Seco 2, Germán M. Mendoza-Silva 1, Oscar Belmonte 1, Athanasios Koukofikis 3, Maria João Nicolau 4, António Costa 4, Filipe Meneses 8, Frank Ebner 6, Frank Deinzer 6, Dominique Vaufreydaz 9, Trung-Kien Dao 7and Eric Castelli 7 1Institute of New Imaging Technologies, Universitat Jaume I, 12071 Castelló, Spain; [email protected] (R.M.); [email protected] (G.M.M.-S.); [email protected] (O.B.) 2Centre for Automation and Robotics (CAR), CSIC-UPM, 28500 Arganda del Rey, Spain; [email protected] (A.R.J.); [email protected] (F.S.) 3 Faculty for Geomatics, Computer Science and Mathematics, HFT Stuttgart—University of Applied Sciences, 70174 Stuttgart, Germany; [email protected] (S.K.); [email protected] (A.K.) 4Algoritmi Research Centre, University of Minho, 4800-058 Guimarães, Portugal; [email protected] (A.M.); [email protected] (M.J.N.); [email protected] (A.C.) 5BlockDox, 129 Finchley Road, NW3 6HY London, UK; [email protected] 6Faculty of Computer Science and Business Information Systems, University of Applied Sciences Würzburg-Schweinfurt, 97070 Würzburg, Germany; [email protected] (T.F.); [email protected] (F.E.); [email protected] (F.D.) 7MICA Institute (HUST-CNRS/UMI2954-Grenoble INP), Hanoi University of Science and Technology, 100000 Hanoi, Vietnam; [email protected] (V.-C.T.); [email protected] (T.-K.D.); [email protected] (E.C.) 8Centro de Computação Gráfica (CCG), 4800-058 Guimarães, Portugal; [email protected] 9Pervasive Interaction/LIG, CNRS, Université Grenoble Alpes, Inria, LIG, F-38000 Grenoble, France; Dominique.V[email protected] *Correspondence: jtorr[email protected]; Tel.: +34-964-387-683 † These authors contributed equally to this work. Academic Editor: Vittorio M. N. Passaro Received: 7 February 2017; Accepted: 7 March 2017; Published: 10 March 2017 Abstract: This paper presents the analysis and discussion of the off-site localization competition track, which took place during the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016). Five international teams proposed different strategies for smartphone-based indoor positioning using the same reference data. The competitors were provided with several smartphone-collected signal datasets, some of which were used for training (known trajectories), and others for evaluating (unknown trajectories). The competition permits a coherent evaluation method of the competitors’ estimations, where inside information to fine-tune their systems is not offered, and thus provides, in our opinion, a good starting point to introduce a fair comparison between the smartphone-based systems found in the literature. The methodology, experience, feedback from competitors and future working lines are described. Keywords: indoor localization technology; indoor navigation; smartphone applications; evaluation and benchmarking Sensors 2017,17, 557; doi:10.3390/s17030557 www.mdpi.com/journal/sensors Sensors 2017,17, 557 2 of 17 1. Introduction Personal indoor localization is still an open problem, for which many different approaches and technologies have been proposed, with the aim of obtaining a usability similar to that achieved outdoors by the GPS system [ 1 – 3 ]. Some indoor positioning techniques are based on specifically designed beacons for localization, while others restrict themselves to the already existing infrastructure in buildings, such as Wi-Fi Access Points (APs), or other so-called signals of opportunity (magnetic field, pressure, light and sound intensity, Global Navigation Satellite Systems (GNSS), among others). New challenging applications, such as ubiquitous indoor positioning (e.g., [ 4 ]), may be enabled by the data generated by smartphone sensors and the crowd’s movement [5,6]. Many indoor positioning designs found in the literature are accompanied by physical experimentation and evaluation of the system (about 77% of works, according to [ 7 ]). However, most of these evaluations are carried out in rather restricted areas; usually, a few university departments easily accessible to the researcher [ 8 ]. A second related problem is that the raw data collected and used during the evaluation of the positioning system is not made available to the research community, making it impossible to confirm the claimed results. Both issues make a critical comparison of Indoor Positioning Systems (IPS) developed by different groups not feasible in a rigorous manner, since collected signals, context and scenarios may change in an uncontrolled way. A better way to compare positioning algorithms under the same experimental conditions would be using a repository of prerecorded data in a large variety of buildings and contexts. An inspiration for this approach is the Machine Learning Repository of the University of California Irvine (UCI) [ 9 ] and Kaggle [ 10 ], both created for evaluating machine learning algorithms with common databases. Although some related databases [ 11 – 22 ], interesting initiatives for benchmarking [ 23 ] and even competitions (Microsoft-IPSN [ 24 – 27 ], EvAAL [ 28 – 32 ] and EVARILOS [ 33 , 34 ]) for better evaluation of IPS already exist, no one encompasses the wide diversity in technologies, contexts and scenarios currently found in this field. A competition is a convenient way to initiate IPS comparisons and reach a worldwide recognition of datasets for fair evaluation. The first off-site indoor location competition was the EvAAL-ETRI Indoor Location [ 35 , 36 ], which was held during the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015). The third track, called “Wi-Fi fingerprinting in large environments” was off-site, and the competitors had access to a large Wi-Fi fingerprint database, the UJIIndoorLoc [ 15 ], to which they had to apply their positioning estimation approaches offline. In the current edition, the organizers decided to make the off-site competition more challenging by creating a new database including the data provided by all sensors embedded in typical smartphones, acquired by different people moving in different types of buildings. The aim of this paper is to contribute to the definition and creation of a new framework for the evaluation of smartphone-based IPS based on our experience at the IPIN2016 off-site competition. We will: 1. describe the database and the evaluation criteria used 2. analyze and compare the competing IPS under equal evaluation conditions 3. present the experiences and suggestions from the competitors to enhance the evaluation framework 4. discuss directions to improve a repository that could be used as a universal reference for testing smartphone-based IPS The remaining of this paper is organized as follows. Section 2describes the IPIN 2016 off-site competition track. Section 3introduces the competing teams and their competing systems. Section 4 shows the results of the competition and their analysis. Section 5contains the discussion about the current competition and the future lines to improve it. Finally, the conclusions are given in Section 6. Sensors 2017,17, 557 3 of 17 2. The IPIN 2016 Off-Site Competition The goal of the competition was to evaluate the performance of different indoor localization solutions based on the signals available in a smartphone (not augmented by any additional hardware) and received while a person was walking in a realistic multi-building, multi-floor indoor environment. Any kind of positioning estimation method based on the provided data was admitted to the competition, for example: •Wi-Fi or magnetic-based fingerprinting. •Multi-sensor fusion algorithms trying to exploit dynamic time-correlated information. • Innovative approaches using map information or activity recognition to complement the above-mentioned ones. This section introduces the main features of the off-site competition, the description of the data provided to competitors and the evaluation criteria. 2.1. Main Features of the Competition The main features of the competition were: • Off-site competition approach. This track was done off-site and offline, so all data processing for calibration and evaluation had to be done before the date of the IPIN conference. Competitors were provided with sensor data logfiles acquired with different mobile phones, including ground truth (known trajectories), that could be used by the teams for tuning their models, as well as data for evaluation (all sensors’ data without ground truth, unknown trajectories). • Multiple sources of information. The provided data logfiles were captured by using several conventional modern smartphones and a dedicated Android application named GetSensorData [ 37 ]. The logfiles contained all of the available signals that were captured in real time with a smartphone: Wi-Fi Received Signal Strengh (RSS), inertial data, magnetic field, GPS, and pressure, among others. • Continuous motion and recording process. While recording the logfiles with the smartphone, the person moved along a continuous trajectory passing by some known landmarks that were recorded in the logfiles. • Realistic walking style. The person recording the data moved in a natural way: most of the time walking forward at normal speed, but occasionally taking 90 or 180 degree turns (e.g., at corridor ends), moving backward or laterally at certain points (e.g., if giving way at door accesses), changing floors through elevators and stairs, etc. The user speed was approximately constant when recording the data with eventual stops at some positions. • Phone holding. The phone was hand held at all times by the user, either stable in front of his face or chest (typical position for reading or typing with the phone), or with the arm downwards while holding the phone in his hand (producing a natural arm swing if walking). No pocket use, calling or any strange handling conditions were considered while collecting data. • Realistic environment and diversity The competition took place in four different buildings (see Figure 1) that were not modified by installing any additional hardware. Moreover, different smartphones were used to gather the data so the competition was not attached to the features of any particular smartphone. With respect to the 2015 EvAAL-ETRI off-site competition, the competition organizers introduced several major changes, with the goal of improvement of the indoor location systems’ performance and increasing the interest for the competition itself. Those changes were: •Data came from multiple sensors in 2016 competition. •Database is now provided in logfiles, as sequences of readings from multiple sensors. •Data has been gathered while the user is moving, whereas data were statically captured in 2015. Sensors 2017,17, 557 4 of 17 • The reference database is not explicitly divided into training and validation sets , including only data with ground truth and data without ground truth. Data without ground truth was used for the evaluation of the different IPS. • Additional information about the reference dataset was provided: floorplan maps, map-based reference trajectories and videos. •The testing scenario is comprised of heterogeneous buildings at very different locations. 2.2. Testing Buildings The competition environment comprised a total of four buildings. One of the buildings was the venue of the on-site IPIN 2016 Conference competition, the Polytechnic School at the University of Alcalá (UAH), Alcalá de Henares, Spain. In order to avoid possible interference between the on-site and off-site tracks, they took place in non-overlapping sectors of the building. The other three evaluation buildings (see Figure 1) correspond to CAR (CSIC Arganda, Madrid, Spain), UJIUB (Universitat Jaume I, Castellón, Spain) and UJITI (Universitat Jaume I, Castellón, Spain). CAR (ID 10) 1 floor UAH (ID 20) 4 floors UJIUB (ID 30) 6 floors UJITI (ID 40) 4 floors Figure 1. Satellite view of the four buildings used in the Indoor Positioning and Indoor Navigation (IPIN) 2016 competition. The building identifiers are also included in the figure (see Section 2.3). 2.3. Description of Datasets (Logfiles) The stream of sensor data generated in the phone is sequentially registered at the time it is obtained in a plain text file (called a logfile). Each line in the logfile corresponds to a single sensor reading, beginning with an initial header identifier (such as WIFI, ACC, or MAGN, among others) that determines the kind of sensor reading, and several fields separated by semicolons with different values. Additionally, those lines starting with the POSI header identifier does not belong to a sensor reading and they contain the ground truth location provided by the users. A parser in Matlab code (version 1, CSIC, Arganda del Rey, Madrid, Spain) [ 37 ] was provided to competitors in order to facilitate data rearrangement in the preferred format. Figure 2shows an excerpt of a logfile. The sampling rate of each sensor varied from logfile to logfile, depending on the embedded sensor chips used by a particular phone and version of the operating system. Typical sampling frequency values for the inertial data is about 50 Hz, while pressure, sound and light sensors have a much lower update rate ( < 10 Hz). Wi-Fi scans are available every 4.7 s in average according to the analysis done by the UMinho team. Wi-Fi scans depend on the device and they are generally available every 4 to 6 s (0.25 to 0.17 Hz). Although multiple Wi-Fi access points were registered in the logfiles, their positions were unknown to the competitors. This is to emulate real world cases, where the positions of the access points is often unavailable. Nevertheless, several geo-referenced floor-map images for each building and some videos were provided to the competitors as additional data. Sensors 2017,17, 557 5 of 17 Sensors 2017,17, x 4 of 16 • Additional information about the reference dataset was provided: floorplan maps, map-based reference trajectories and videos. •The testing scenario comprises heterogeneous buildings at very different locations. 2.2. Testing Buildings The competition environment comprised a total of 4 buildings. One of the buildings was the venue of the on-site IPIN 2016 Conference competition, the Polytechnic School at the University of Alcalá (UAH), Alcalá de Henares, Spain. In order to avoid possible interference between the on-site and off-site tracks, they took place in non-overlapping sectors of the building. The other three evaluation buildings (see Figure 1) correspond to CAR (CSIC Arganda, Madrid), UJIUB (Universitat Jaume I, Castellón) and UJITI (Universitat Jaume I, Castellón). CAR (ID 10) 1 floor UAH (ID 20) 4 floors UJIUB (ID 30) 6 floors UJITI (ID 40) 4 floors Figure 1. Satellite view of the buildings used in the IPIN 2016 competition. 2.3. Description of Datasets (Logfiles) The stream of sensor data generated in the phone is sequentially registered at the time it is obtained in a plain text file (called a logfile). Each line in the logfile corresponds to a single sensor reading, beginning with an initial header identifier (such as WI-FI, ACC, or MAGN, among others) that determines the kind of sensor reading, and several fields separated by semicolons with different values. A parser in Matlab code was provided to competitors (http://indoorloc.uji.es/ipin2016track3/) to facilitate data rearrangement in the preferred format. Figure 2shows an excerpt of a logfile. LIGH;8.264;4301.053;14375.0;3 MAGN;8.273;21.059;-9.30000;-40.74000;-15.54000;3 PRES;8.274;21.061;957.6104;3 WIFI;8.276;22204.687;test-CAR;00:0b:86:27:36:c2;-84 WIFI;8.276;22204.687;test-CAR;00:0b:86:27:32:e2;-69 WIFI;8.276;22204.687;portal-csic;00:0b:86:27:32:e1;-69 WIFI;8.276;22204.687;eduroam;00:0b:86:27:36:c0;-84 WIFI;8.276;22204.687;test-CAR;00:0b:86:27:35:92;-87 WIFI;8.276;22204.687;portal-csic;00:0b:86:27:35:91;-89 WIFI;8.276;22204.687;eduroam;00:0b:86:27:35:80;-92 WIFI;8.276;22204.687;WiFiArgandaParque;00:0d:97:10:c5:f3;-94 WIFI;8.276;22204.687;WLAN_3F;e0:91:53:1b:36:f3;-92 WIFI;8.276;22204.687;portal-csic;00:0b:86:27:35:81;-93 WIFI;8.276;22204.687;WiFiArganda;00:0d:97:00:c5:f3;-92 POSI;8.302;2;40.51296307;-3.34844;0;20 LIGH;8.310;21.073;14375.0;3 GYRO;8.311;21.075;0.19884;-0.05253;0.04276;3 AHRS;8.311;21.07;33.297;-1.474;-6.154;0.034;-0.286;-0.95;3 Figure 2. An excerpt from a logfile, as provided to the competitors. Note that example includes a POSI entry (highlighted in red color), but they are not included in the evaluation files. Figure 2. An excerpt from a logfile, as provided to the competitors. Note that the example includes a POSI entry (ground truth location highlighted in red color in the excerpt), but these entries are not included in the evaluation files since they do not contain ground truth locations. Dataset Types As mentioned earlier, logfiles provided to the competitors were divided into training (known trajectories) and evaluation (unknown trajectories) logfiles. The training logfiles contained geo-referenced positions at some landmarks, in the form of additional records with a POSI header followed by the ground truth coordinates: latitude, longitude, floor and building identifiers (see highlighted line in Figure 2). The latitude and longitude coordinates are in WGS84 format, the floor corresponds to the floor number, and the building identifiers were introduced in Figure 1. Table 1. Description of the training logfiles. # Building * Route Floors Landmarks Duration (s) Smartphone 01 CAR 1 1 75 1257 S3 02 CAR 1 1 75 1260 S3mini 03 CAR 2 1 52 888 S3 04 CAR 2 1 52 887 S4 05 UAH 1 3 67 1101 S3 06 UAH 1 3 67 1101 S4 07 UAH 2 4 64 1192 S3 08 UAH 2 4 64 1188 S4 09 UAH 4 2 29 508 S3 10 UAH 4 2 29 508 S4 11 UJIUB 1 6 58 529 S3 12 UJIUB 1’ 6 58 467 S3 13 UJIUB 2 6 59 397 S3 14 UJIUB 2’ 6 59 375 S3 15 UJIUB 3 6 60 516 S3 16 UJITI 1 3 360 1134 GN5 17 UJITI 2 3 291 590 GN5 * See Figure 1; S3: Samsung Galaxy S3, Samsung, South Korea; S3mini: Samsung Galaxy S3 mini, Samsung, South Korea; S4: Samsung Galaxy S4, Samsung, South Korea; GN5: Google Nexus 5, LG Electronics, South Korea. Sensors 2017,17, 557 6 of 17 The competition organizers published the training logfiles in the official website, including geo-referenced positions at some landmarks. On 25 July, the organizers provided the evaluation logfiles to the competitors, which contained measurements taken following the same procedure of the training files (although possibly by different users or phone models), but without geo-referenced positions (POSI data). Thus, the competitors had to provide their estimations of the unknown trajectories by using only the phone sensor data without any landmark information. Tables 1and 2show a summary of the training and evaluation logfiles. Experimental diversity comes from the variation of trajectories, buildings, users and devices used to collect data. Table 2. Description of the evaluation logfiles. # Building * Route Floors Landmarks Duration (s) Smartphone 01 UJITI 3 3 46 241 HW 02 UJIUB 4 6 91 730 S3 03 UAH 3 4 65 1476 S3 04 UJITI 4 3 75 430 SP 05 UAH 5 3 42 899 S4 06 CAR 3 1 76 1223 S3 07 UAH 3 4 65 1477 S4 08 UAH 5 3 42 899 S3 09 CAR 3 1 76 1223 S4 * See Figure 1; S3: Samsung Galaxy S3, Samsung, South Korea; S4: Samsung Galaxy S4, Samsung, South Korea; SP: Sony Xperia SP, Sony, Japan; HW: Huawei G630, Huawei, China. 2.4. Submission of Results and Evaluation After processing the evaluation logfiles, participants had the opportunity to submit up to three different sets of indoor location estimates. The estimates had to be provided at a pace of 0.5 s and starting from the first timestamp available in each logfile. Each submission had to be independently provided to the organizers for its evaluation. The estimations had to adhere to the following format: timestamp, latitude, longitude, floor, and building. Each submission was evaluated considering the estimates provided for the nine evaluation logfiles. The base positioning error was defined as the geometric distance between the two-dimensional (latitude and longitude) real position as recorded by competition organizers and the estimated position provided by the competing IPS. Furthermore, we took into account floor and building mis-identifications during the trajectories. In this case, 15 and 50 meter penalties were added to the geometric error if the competitors did not guess the correct floor and building, respectively. The final metric used for the competition results was the third quartile (75%) of this joint positioning error. Of the three alternative sets of estimations submitted by the competition teams to the final tests, only the one with the best results was considered for the ranking of competitors. 3. Description of the Competing IPS This section highlights the main features of the competing teams and references their work in the IPIN 2016 proceedings: • The HFTS team: S. Knauth and A. Koukofikis. Stuttgart University of Applied Sciences, Stuttgart, Germany [38]. • The UMinho team: A. Moreira, M.J. Nicolau, A. Costa and F. Meneses. University of Minho and Centro de Computação Gráfica, Guimarães, Portugal [39] •The BlockDox (BD) team: Y. Beer. BlockDox, London, United Kingdom (This competing team did not submit a paper to the conference). • The FHWS team: T. Fetzer, F. Ebner and F. Deinzer. University of Applied Sciences Würzburg-Schweinfurt, Würzburg, Germany [40]. Sensors 2017,17, 557 7 of 17 •The Marauder team: V.C. Ta, D. Vaufreydaz, T.K. Dao, and E. Castelli. Université Grenoble Alpes, CNRS, Inria, LIG, Grenoble, France and Hanoi University of Science and Technology, Hanoi, Vietnam [41]. 3.1. The HFTS Team The HFTS system employed GNSS, Wi-Fi, accelerometer, compass, and gyroscope data [ 42 ]. Accelerometer data was used to detect step events, and compass and gyroscope were used for heading estimation. Drift compensation and step length estimation were performed by a particle filter using the information of floor plans to detect the most likely path (see also [ 43 , 44 ]). Floor detection was based on position and received signal strength indicator (RSSI) evaluation. For the RSSI positioning, the scalar product correlation fingerprinting algorithm [45] was applied. Heading detection was performed by combining gyro and magnetic information: the gyroscope is able to detect heading changes quite accurately on a short timescale, but will drift in the long-term. Therefore, the algorithm stabilized the gyro heading with the compass heading. Compass heading is subject to strong local magnetic perturbations but shows no drift on a long-term scale. The estimated heading was calculated by summing gyro heading changes immediately, but relaxing back to magnetic heading with a configurable constant. Step detection was obtained by filtering the accelerometer data and detecting maximal and minimal values of the result. Each time a step was detected, the particle filter updated its state. The particle filter contained a constant number of particles. Besides the position, a particle state also comprised individual step length and heading offset values. On each step, all particles were moved according to the estimated heading, individually modified by the particle specific offsets. Each time a particle collided with, for example, a wall, it was replaced by a new one. For collision detection, the provided floor plans were used. This new particle was seeded at the position of an existing particle, but had its own step length and heading offset value. The new seeded particle data contained also a reference to the particle where it was created. A particular feature of the algorithm was its backtracking capability. By recursively tracking back the history of a particle, a continuous track estimation for each individual particle was possible, allowing a posteriori elimination of failed particles. This could lead to a significant accuracy increase for cases where the position information was not needed in real time. This method is related to particle backward smoothers (for example [46]), as well as loop closing. 3.2. The UMinho Team The UMinho approach is mostly based on Wi-Fi fingerprinting enhanced with information extracted from the accelerometers and atmospheric pressure data. Thus, the UMinho team used only the POSI, Wi-Fi, accelerometer and atmospheric pressure records, ignoring the remaining data. The strategy involved three steps: creation of additional POSI records to increase the ground truth information density, segmentation of the data in periods of movement and immobility, and association between positions and Wi-Fi fingerprints. Building the radio map raises a few challenges. The POSI records are sparse in time and the user movement cannot be modelled by straight lines between adjacent POSI records since they involve crossing walls or floors in the provided building blueprints. Thus, intermediate POSI records were created based on the videos of the data collection process, or, in cases where video was not available, by our best guesses of the most likely path compatible with the building blueprints. With the new additional records, the sequence of POSI records represents the estimated real trajectory. The user movement was obtained from the accelerometer readings. The team removed the DC component of the acceleration magnitude signal and applied a one-dimensional median filter and a Butterworth filter to clean the data. Using the peaks of the resulting data, it was possible to count the steps and step times and determine if the user was moving or not. Sensors 2017,17, 557 8 of 17 As previously mentioned, the UMinho team approach is mainly based on the Wi-Fi fingerprint data; this requires building a Wi-Fi radio map, where each Wi-Fi fingerprint has to be associated to a geographic position. The entire set of fingerprints was used to create a radio map. Since Wi-Fi records were not created simultaneously with the POSI records, different strategies were tried to identify the coordinates for each fingerprint. In the end, the radio map was built by associating fingerprints to the linearly interpolated coordinates between known positions, combined with the movement information extracted from the accelerometer data. Once the radio map is built, it is possible to estimate the trajectories in each of the evaluation dataset. For each fingerprint in the evaluation dataset, it was necessary to estimate the corresponding location. To estimate the building, a majority rule was used. To estimate the floor, the team used a filtering function combined with another majority rule [ 47 ]. Additionally, the pressure data was also used to estimate the floor. Having estimated the building and the floor, a k-Nearest Neighbors (kNN) [ 48 ] based method was used to estimate the coordinates using the fingerprints contained in the radio map that are most similar to the one being processed [49]. With the previously described method, the UMinho team was able to estimate the position (building, floor and coordinates) for the provided evaluation datasets. However, Wi-Fi fingerprints were collected with an average sampling period of 4.7 s and, as defined in the competition rules, the final trajectory estimation must be provided at discrete time intervals of 0.5 s. For this purpose, different strategies were considered, based on using linear interpolation and the mobility profile along each trajectory built from the accelerometer data. In the end, with these different strategies, three different solutions were built [39]. 3.3. The BlockDox Team The algorithm proposed by the BD team was on Random Forest fingerprinting assisted by GPS in order to detect outdoor location. Location was solved hierarchically in order to reduce the number of features in each step. Finding the correct building, then the floor and finally the location, all steps are done by using Wi-Fi fingerprinting. The first step was to associate a ground truth to the sensors’ records. The sensors’ records were not synchronized with the ground truth points. In order to associate the sensors’ records with ground truth, linear interpolation was used between ground truth locations and the floor and building identifiers of the point with the closest time. Wi-Fi fingerprinting only relays on power without a direction of arrival; therefore, this method depends on different Radio Frequency (RF) blockage from different Wi-Fi APs in order to get distinct fingerprints for different cells. Change in loss due to distance is hard to determine due to reflections. Using it in large halls or outside the building could lead to inaccuracy. GPS location was used to complete the Wi-Fi fingerprinting method. 3.3.1. Hierarchical Algorithm The primary classification was the building classification. The media access control (MAC) addresses of the visible Wi-Fi access points were orthogonal between buildings; therefore, it was possible to count known MAC addresses from each building and reducing the number of features from 730 unique addresses to four counters of known addresses, one per building, getting 100% correct results. The secondary classification was floor estimation. The MAC addresses were not orthogonal between floors, and the station receives access points from multiple floors. MAC address fingerprinting was used to predict the floor because it is a finer approach and the number of MAC addresses is reduced. The final classification was based on Wi-Fi fingerprinting. The classification used only the MAC addresses of the already predicted building and floor. The tested area was split into multiple cells and Wi-Fi RSSI features were used in a Random Forest Classifier algorithm in order to decide in which cell the device is located in. Time series attributes were applied in order to filter bogus predicted locations and smooth out the result. It allowed a drastic reduction in features. Sensors 2017,17, 557 9 of 17 3.3.2. Cross Validation and Optimization The training files were divided into different routes. To avoid overfitting, each route was evaluated using the other routes for training. The algorithm hyper parameters were hierarchically and greedily optimized in order to reduce the complexity and time of solving the problem—first optimizing the parameters of the floor classification, and only then the location parameters. When the optimum of each parameter was found, it was turned into a constraint and the optimization procedure continued to optimize the next parameter. 3.4. The FHWS Team The FHWS localization system is based on previous works, primarily on the approach presented in [ 50 ]. Since then, this technique has been extended by including prior navigation knowledge using realistic human walking paths [ 51 ] and adding smoothing methods [ 40 ]. Additionally, a self-developed map editor allows for creating advanced 3D maps and realistically shaped stairs. This proposal avoids time-consuming fingerprinting and calibration processes. Furthermore, it does not need any prior information on the starting position. The system has been implemented in C++ using the Qt cross-platform application framework. The smartphone provides all necessary measurements and no additional devices are needed. The readings of all smartphone sensors are fused using recursive density estimation, implemented by a particle filter with the state transition as proposal density. This transition is based on random walks along a 20 cm -grid graph, derived from the building’s floorplan. By removing edges intersecting with walls, it is ensured that only valid movements can be sampled. The random walk along the graph’s edges is constrained by the measured heading change and detected number of steps. Additionally, each node within the graph contains a weight, denoting the likelihood of it being visited by the pedestrian. Using this approach, nodes near walls receive a lower likelihood, while nodes marked as staircases, elevators and doors are more likely. During the state transition, this importance is used to increase the weight between two nodes. The probability density of the state evaluation is provided by two probabilistic sensor models. First, the smartphone’s barometer, if available, is used to infer the likeliness of the current z location. As ambient pressure is dependent on environmental conditions like weather, time of day and others [ 50 ], relative pressure values were used instead of absolute ones. Due to noisy sensors, multiple readings are used to estimate this relative base. A probability is then obtained by comparing the measured relative pressure with a pressure prediction. This relative prediction is adjusted whenever a z change happens within the transition. Secondly, the Wi-Fi component provides absolute location estimates by measuring the RSS of nearby APs and comparing them with the expected signal strengths, determined with a wall attenuation factor model [ 50 ]. Thus, no fingerprinting is required. To reduce the setup time, the same parameters are used for all transmitters at the expense of a worse location estimation performance and higher uncertainty. Since the AP positions were not provided, a genetic algorithm is used to approximate the model based upon the RSS measured at the ground truth positions. Finally, a fixed interval smoother, using backward simulation, is deployed for further optimisation and to reduce multimodalities [ 40 ]. Here, a smoothing transition model compares the distance, angle and height between some future and the current state. The resulting likelihood is then used for reweighting the particles. 3.5. The Marauder Team The Maraduder team methodology for solving the challenge is to try as many indoor positioning methods as possible. The methods were evaluated by using a cross-validation scheme on the provided data. Based on the observation, several adjustment were applied in order to decrease the overall distance errors. Sensors 2017,17, 557 16 of 17 35. Potortì, F.; Barsocchi, P.; Girolami, M.; Torres-Sospedra, J.; Montoliu, R. Evaluating indoor localization solutions in large environments through competitive benchmarking: The EvAAL-ETRI competition. In Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015), Banff, AB, Canada, 13–16 October 2015. 36. Torres-Sospedra, J.; Moreira, A.; Knauth, S.; Berkvens, R.; Montoliu, R.; Belmonte, O.; Trilles, S.; João Nicolau, M.; Meneses, F.; Costa, A.; et al. A realistic evaluation of indoor positioning systems based on Wi-Fi fingerprinting: The 2015 EvAAL–ETRI competition. J. Ambient Intell. Smart Environ. 2017,9, 263–279. 37. Zampella, F.; Jiménez, A.R.; Seco, F. LOPSI repository: GetSensorData application. Available online: http://lopsi.weebly.com/downloads.html (accessed on 8 March 2017). 38. Knauth, S.; Koukofikis, A. Smartphone positioning in large environments by sensor data fusion, particle filter and FCWC. In Proceedings of the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016), Alcalá de Henares, Spain, 4–7 October 2016. 39. Moreira, A.; Nicolau, M.J.; Costa, A.; Meneses, F. Indoor tracking from multidimensional sensor data: UMINHO at the IPIN 2016 Competition. In Proceedings of the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016), Alcalá de Henares, Spain, 4–7 October 2016. 40. Fetzer, T.; Ebner, F.; Deinzer, F.; Köping, L.; Grzegorzek, M. On Monte Carlo Smoothing in Multi Sensor Indoor Localisation. In Proceedings of the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016), Alcalá de Henares, Spain, 4–7 October 2016. 41. Ta, V.C.; Vaufreydaz, D.; Dao, T.K.; Castelli, E. Smartphone-based User Location Tracking in Indoor Environment. In Proceedings of the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016), Alcalá de Henares, Spain, 4–7 October 2016. 42. Google: Develop—API Guides—Location and Sensors. Motion Sensors. Available online: https://developer. android.com/guide/topics/sensors/sensors_motion.html (accessed on 8 March 2017). 43. Kang, W.; Han, Y. SmartPDR: Smartphone-Based Pedestrian Dead Reckoning for Indoor Localization. IEEE Sens. J. 2015,15, 2906–2916. 44. Willemsen, T.; Keller, F.; Sternberg, H. A topological approach with MEMS in smartphones based on routing-graph. In Proceedings of the 2015 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Banff, AB, Canada, 13–16 October 2015; pp. 1–6. 45. Knauth, S.; Storz, M.; Dastageeri, H.; Koukofikis, A.; Mähser-Hipp, N. Fingerprint Calibrated Centroid and Scalar Product Correlation RSSI Positioning in Large Environments. In Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015), Banff, AB, Canada, 13–16 October 2015. 46. Nurminen, H.; Ristimäki, A.; Ali-Löytty, S.; Piché, R. Particle filter and smoother for indoor localization. In Proceedings of the 2013 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Montbéliard, France, 28–31 October 2013; pp. 1–10. 47. Marques, N.; Meneses, F.; Moreira, A. Combining similarity functions and majority rules for multi-building, multi-floor, WiFi positioning. In Proceedings of the 3th the International Conference on Indoor Positioning and Indoor Navigation (IPIN’2012), Sydney, Australia, 13–15 November 2012. 48. Cover, T. and Hart, P. Nearest Neighbor Pattern Classification IEEE Trans. Inf. Theor. 1967,131, 21–27. 49. Moreira, A.; Nicolau, M.J.; Meneses, F.; Costa, A. Wi-Fi fingerprinting in the real world—RTLS@UM at the EvAAL competition. In Proceedings of the International Conference on Indoor Positioning and Indoor Navigation (IPIN), Banff, AB, Canada, 13–16 October 2015; pp. 1–10. 50. Ebner, F.; Fetzer, T.; Köping, L.; Grzegorzek, M.; Deinzer, F. Multi Sensor 3D Indoor Localisation. In Proceedings of the Sixth International Conference on Indoor Positioning and Indoor Navigation (IPIN 2015), Banff, AB, Canada, 13–16 October 2015. 51. Ebner, F.; Fetzer, T.; Grzegorzek, M.; Deinzer, F. On Prior Navigation Knowledge in Multi Sensor Indoor Localisation. In Proceedings of the International Conference on Information Fusion (FUSION 2016), Heidelberg, Germany, 5–8 July 2016. 52. Breiman, L. Random forests. Mach. Learn. 2001,45, 5–32. 53. Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016. Sensors 2017,17, 557 17 of 17 54. Kjaergaard, M.; Munk, C. Hyperbolic Location Fingerprinting: A Calibration-Free Solution for Handling Differences in Signal Strength. In Proceedings of the Sixth Annual IEEE International Conference on Pervasive Computing and Communications, Hong Kong, China, 17–21 March 2008; pp. 110–116. 55. Madgwick, S. An Efficient Orientation Filter for Inertial and Inertial/magnetic Sensor Arrays; University of Bristol: Bristol, UK, 2010. c  2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).