Off-line evaluation of indoor positioning systems in different scenarios: the experiences from IPIN 2020 competition

IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 5011 Off-Line Evaluation of Indoor Positioning Systems in Different Scenarios: The Experiences From IPIN 2020 Competition Francesco Potortì , Member, IEEE , Joaquín Torres-Sospedra , Darwin Quezada-Gaibor, Antonio Ramón Jiménez , Fernando Seco , Antoni Pérez-Navarro , Member, IEEE , Miguel Ortiz , Ni Zhu , Valerie Renaudin , Member, IEEE , Ryosuke Ichikari , Ryo Shimomura, Nozomu Ohta, Satsuki Nagae, Takeshi Kurata , Dongyan Wei , Member, IEEE , Xinchun Ji, Wenchao Zhang , Sebastian Kram , Maximilian Stahlke , Christopher Mutschler , Antonino Crivello , Paolo Barsocchi , Michele Girolami , Filippo Palumbo ,RuizhiChen , Yuan Wu, Wei Li, Yue Yu , Shihao Xu , Lixiong Huang , Tao Liu, Jian Kuang , Xiaoji Niu , Takuto Yoshida , Yoshiteru Nagata , Yuto Fukushima , Nobuya Fukatani, Nozomi Hayashida , Yusuke Asai, Kenta Urano, Graduate Student Member, IEEE , Wenfei Ge , Nien-Ting Lee, Shih-Hau Fang , Senior Member, IEEE , You-Cheng Jie , Shawn-Rong Young , Ying-Ren Chien , Senior Member, IEEE , Chih-Chieh Yu , Chengqi Ma , Bang Wu , Wei Zhang , Yankun Wang, Yonglei Fan ,StefanPoslad , David R. Selviah , Member, IEEE , Weixi Wang , Hong Yuan , Yoshitomo Yonamoto, Masahiro Yamaguchi, Tomoya Kaichi , Baoding Zhou ,XuLiu , Zhining Gu, Chengjing Yang, Zhiqian Wu, Doudou Xie, Can Huang, Lingxiang Zheng, Ao Peng , Member, IEEE ,GeJin , Qu Wang , Haiyong Luo , Member, IEEE , Hao Xiong, Linfeng Bao, Pushuo Zhang, Fang Zhao ,Chia-AnYu , Chun-Hao Hung , Leonid Antsfeld, Boris Chidlovskii , Member, IEEE , Haitao Jiang, Ming Xia ,DayuYan , Yuhang Li, Yitong Dong, Ivo Silva , Cristiano Pendão , Filipe Meneses , Member, IEEE , Maria João Nicolau , António Costa , Member, IEEE , Adriano Moreira , Member, IEEE , Cedric De Cock ,DavidPlets , Member, IEEE , Miroslav Opiela , Jakub Džama, Liqiang Zhang , Graduate Student Member, IEEE , Hu Li, Boxuan Chen, Yu Liu , Seanglidet Yean , Bo Zhi Lim , Wei Jie Teo , Bu Sung Lee , Senior Member, IEEE , and Hong Lye Oh Abstract —Every year, for ten years now, the IPIN competition has aimed at evaluating real-world indoor localisation systems by testing them in a realistic environment, with realistic movement, using the EvAAL framework. The competition provided a unique overview of the state-of-the-art of systems, technologies, and methods for indoor positioning and navigation purposes. Through fair comparison of the performance achieved by each system, the competition was able to identify the most promising approaches and to pinpoint the most critical working conditions. In 2020, the competition included 5 diverse off-site off-site Tracks, each resembling real use cases and challenges for indoor positioning. The results in terms of participation and accuracy of the proposed systems have been encouraging. The best performing competitors obtained a third quartile of error of 1 m for the Smartphone Track and 0.5 m for the Foot-mounted IMU Track. While not running on physical systems, but only as algorithms, these results represent impressive achievements. Index Terms —Indoor positioning and navigation, evaluation, smartphone-based positioning, foot-mounted IMU, positioning in industrial scenarios and factories, vehicle-positioning. Manuscript received April 2, 2021; accepted May 6, 2021. Date of publication May 24, 2021; date of current version March 14, 2022. The associate editor coordinating the review of this article and approving it for publication was Prof. Chan Gook Park. (Corresponding authors: Joaquín Torres-Sospedra; Antonino Crivello.) Please see the Acknowledgment section of this article for the author supporting information and affiliations. Digital Object Identifier 10.1109/JSEN.2021.3083149 I. INTRODUCTION THE International conference on Indoor Positioning and Indoor Navigation (IPIN), born in 2010, has been a reference for researchers and practitioners interested in systems, methods, techniques and technologies for indoor positioning and indoor navigation. In fact, estimating the location of a mobile target still represents a challenging task in indoor This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 5012 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 environments. While solutions based on Global Navigation Satellite System (GNSS) are successfully used outdoor, pinpointing the location of an indoor target requires the adoption of technologies that most often cannot exploit satellites because indoor obstacles, walls and, most of all, ceilings are all factors that significantly reduce the strength of satellite signals. Indoor localisation systems, be they targeted at personal navigation or other purposes, heavily rely on the use of a wide variety of sensors. This is in sharp contrast with outdoor localisation, which relies only on GNSS radio signals, at least as far as consumer-grade applications are concerned. Since its inception, IPIN’s core topics have been low-level hardware and software techniques for positioning and navigation. In the last few years a growing interest has been observed in topics regarding system evaluation, standardisation and interoperability. In fact, reaching a wide consensus on the evaluation metrics for these systems is a fundamental step towards filling the gap between prototypes and commercial systems. In this paper we are particularly interested in testing and evaluation of systems and, as a showcase, we fully describe the IPIN 2020 competition. While the competition usually benefits from the attendance of the congress, in 2020 the competition was a solo event which nonetheless attracted 95 attendees to the final event, which was held online. Past editions of the IPIN competitions were organised by hosting two different kinds of Tracks, namely on-site and off-site. In the on-site Tracks, competitors demonstrate their system by performing an assigned test in a given place. An actor carries the competing system while walking in and between multi-floor buildings. The system shall provide position estimates in real time using local data processing on opportunistic signals, without any ad hoc infrastructure. In off-sites Tracks competitors calibrate their algorithms in advance using a ground-truth reference database provided by the committee, and compete using new unreferenced data. Due to worldwide travel restrictions, the 2020 competition only hosted off-site tracks for active indoor positioning systems. This paper contains organisational aspects and highlights the choices taken by the organisers. The core part of the paper is the description of the competing systems. This edition provided five off-site Tracks: Smartphone,Foot-mounted IMU, xDR in manufacturing,On-vehicle smartphone and Channel impulse response. Each Track is explained in a dedicated section which also contains contributions and system descriptions authored by the competitors. As a follow-up to [1], [2], this work provides a unique overview on the state of the art of systems, technologies and methods for indoor positioning and navigation purpose. Through a fair comparison, the performance achieved by each system in a real-world scenario helps understanding which are the most promising approaches, under which working conditions. Comparison is performed according to the Evaluating Ambient Assisted Living (EvAAL) framework [3]. The paper is structured as follows. Section II summarises the history of the IPIN competition and highlights possible future directions for the next editions. Section III is an overview of the five Tracks and their commonalities, which are founded on the EvAAL framework. Sections IV to VIII report the characteristics and final results for each Track and the detailed descriptions of most competing systems. Section IX is an attempt at identifying lessons to be learned from the practical experience of competitors: even if the competition was off-site, the algorithms used were stressed in a challenging and competitive environment, which offered insight to both competitors and attendees. II. PAST,PRESENT AND FUTURE DIRECTIONS Research in the area of indoor positioning and navigation in the last decade has elicited a strong interest from both academic and industrial communities. We expect indoor Location-based services (LBS) to experience significant growth and evolution and to be commonly available on commercial devices in the future Although impressive advances in the field of algorithms for indoor localisation and tracking have been achieved, evaluation frameworks are missing. Under this respect, the EvAAL framework was a pioneer initiative devoted to compare, with a rigorous methodology, the performance of indoor localisation systems in real-world, non-trivial settings. Here we summarise the 10-year-long journey of the EvAAL framework, from the first EvAAL competition in 2011 to the recent IPIN 2020 competition, and we give a look at the next edition of IPIN scheduled for late November 2021. The EvAAL framework has been designed to test and compare the performance of indoor localisation systems, following a rigorous approach. It consists of four core criteria plus four extended criteria, the latter being desirable ones which should be applied as far as possible [3]. The core criteria, which are necessary to define a competition as conformant to the EvAAL framework, are: 1) Natural movement of an actor: an actor walks with natural speed and attitude. 2) Realistic environment: the walking path is set in a realistic setting; EvAAL competitions were done in a living lab, IPIN competitions in wider settings, like a congress centre, a university building, a shopping mall. 3) Realistic measurement resolution: final error measurements below 50 cm in space and 0.5 s in time should be considered as null, when indoor people’s movement are considered; when the actor walks, the test should be considered adequate if his/her time and space errors when passing on the test points are not greater that the above figures, which is easy for a trained person. 4) Third quartile of point Euclidean error: the accuracy score is based on the third quartile of the point error. Applicability of extended criteria to IPIN 2020 is discussed in Section III. Table I is an overview of the size of past competitions. While the IPIN competitions aim to compare systems based only on their accuracy performance, the early EvAAL editions were characterised by a richer set of goals, including the deployment complexity of the solution; the time required to calibrate and configure the system; the impact of the system in terms of the end-user’s perception. These indicators were mainly driven by the Ambient Assisted Living (AAL) application scenarios to which EvAAL was inspired [4]. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5013 TABLE I NUMBER OF TRACKS AND NUMBER OF COMPETITORS FOR ON-SITE AND OFF-SITE INDOOR LOCALISATION COMPETITION TRACKS In 2014, the EvAAL competition met the IEEE IPIN conference, giving birth to the IPIN competition [5]. Such partnership was the result of two complementary communities: on the one hand, experience from the EvAAL competitions provided a well-established evaluation framework; on the other hand the IPIN provided a vibrant community composed of academic and industrial attendants who every year share advances in the area of indoor navigation and positioning. The birth of the IPIN competition series extended the range of potential competitors. Indeed, the IPIN competition has seen a consistent increased of the number of competition Tracks, each of which focused on specific constraints and objectives, as shown in Table I. Tracks are split between on-site and off-site. The on-site Tracks take place during the IPIN conference and competitors do a live test of their solutions. Off-site Tracks are performed remotely. For the latter ones, competitors are required to test their solution by following rules and data sets provided by the organisers. During the last 7 IPIN competition editions, competitors have had the opportunity to test their systems in shopping malls, conference halls, university campus and large research centres. Such a variety of locations is the distinguishing feature of IPIN competitions with respect to similar initiatives. In fact, the confluence of EvAAL into the IPIN conference refined the methodology adopted to assess the performance, adding the following characteristics: no additional instrumentation allowed, non-overlapping competition Tracks, highly representative competition areas, easy-to-understand measurement statistics to define the final ranking of the tested systems. Appreciation of this format by competitors (both from academy and industry) and sponsors is reflected in the consistently growing attendance to the competition. So far so good, but what’s next for the IPIN competition? Organisers are looking at two growing trends: •the increasing performance and diffusion of sensing units available with commercial devices; •the wide adoption of learning methodologies with a never-seen-before statistical power. As far as sensing is concerned, new short-range RadioFrequency (RF) technologies such as Wi-Fi Time-of-Flight (TOF) measurements, Ultra-Wide Band (UWB) and Bluetooth 5.0 are the next obvious target to include in testing by augmenting the existing Tracks or creating new ones. In the future, mediumand long-range RF technologies 5G and 6G may become drivers for localisation technology, but currently it is not easy to set up a representative testbed: telecommunication providers might play a crucial role for indoor localisation; the IPIN competition is open to testing and experiencing such disrupting technologies. As far as the increasing pervasiveness of machine learning is concerned, our prospect is to support such evolution by offering always-more challenging data sets to the competitors, in order to assess the performance of their systems, as it has been done with the off-site Tracks. Under this respect, we consider heterogeneity as one of the most challenging properties of such data sets. Heterogeneity refers to the different nature of data that can be simultaneously analysed, to improve the performance of ML-based algorithms. Fingerprint data sets, based on of Wi-Fi Received Signal Strength Indicator (RSSI) readings, can be enriched with context information derived from Bluetooth beacons, environmental or physiological sensor readings, giving rise to unexpected possible correlations. In turn, such data sets can be used as non-structured inputs to multi-layer neural networks (e.g. Recurrent Neural Network (RNN) based on Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) layers) to solve classification and regression problems applied to indoor localisation. This trend has been in place in both on-site and off-site Tracks in the last years, and it is going to continue. Another interesting topic where the IPIN competition could promote new challenges is the adoption of a more accurate and meaningful metric for computing the positioning error. In fact, one of the objectives of IPIN is to define standard procedures for evaluating positioning systems, in an effort to improve over the recent ISO/IEC 18305 standard [6], [7]. Discussion is underway about using an alternative or additional criterion for computing the error, that is, the distance from each reference point in the ground truth to the position estimated by the competing system. Currently, the IPIN competition series defines the point error as the horizontal distance plus a fixed penalty of 15 m per each wrong floor. Now times appear to be mature for the adoption of a “real-world” distance, that is, the length of the path that a person would need to travel from the reference point to the estimated point. This is the same as the Euclidean distance if the two points are in line of sight, that is on the same floor and in the same room, but can be very different otherwise. A complete discussion on the benefits of this new method and of the possible algorithms to use, complete with code, is available at [8]. A final consideration about the future of the IPIN competition refers to the integration of multiple indoor localisation systems. More specifically, we envision a future where different indoor localisation services coexist in the same area. Such systems will require to be integrated and orchestrated so that to reproduce, as much as possible, the well-assessed user experience of navigation in outdoor environments [9]. We consider two key challenges: •to standardise Application Programming Interfaces (APIs) designed to discover, access and use an indoor localisation systems with a commercial device; •to regulate the privacy consents asked of end-users in order to provide location-based services in accordance 5014 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 with the EU General Data Protection Regulation (GDPR) regulation framework. III. ORGANISING AN OFFSITE COMPETITION WITH MULTIPLE TRACKS In the 2014–2019 editions the systems competing in on-site Tracks were tested during the IPIN conference, in the same or a nearby site. This way, competitors were able to both compete with their system and attend the conference, and conference attendees could come and look at how the competition was done. Since the conference was cancelled in 2020, only off-site Tracks were organised for this edition, under the supervision of competition chairs Francesco Potortì and Sangjoon Park. The institutions involved were the Institute of Information Science and Technologies (ISTI) of the National Research Council (CNR, IT), UBIK Geospatial Solutions S.L. (ES), the GEOTEC laboratory of Universitat Jaume I (UJI, ES), Consejo Superior de Investigaciones Científicas (CSIC, ES), the IN3 of Universitat Oberta de Catalunya (UOC, ES) the GEOLOC Team, University Eiffel (FR), the National Institute of Advanced Industrial Science and Technology (AIST, JP), University of Tsukuba (JP), Aerospace Information Research Institute, (CAS, CH), IIS Fraunhofer (DE) and the Electronics and Telecommunications Research Institute (ETRI, KR). A. Preparing the Competition Areas In contrast with the two previous editions where most on-site and off-site Tracks took place in the same large area (a shopping mall in 2018 [1] and aresearch centre in 2019 [2]), in 2020 travel restrictions made it impractical to gather together and take measurements in the same place, so all Tracks were independent. The set of evaluation scenarios cover a university library building, the shopping mall from IPIN 2018 competition [1], a manufacturing site, road-based tracks with different satellite view conditions (including indoors) and an environment resembling an industrial setting. All the Tracks complied with the EvAAL framework [3] by adopting its four distinguishing core criteria (described in Section II): 1) Natural movement of an actor 2) Realistic environment 3) Realistic measurement resolution 4) Third quartile of point Euclidean error Additionally, all the Tracks were compliant with most of the extended criteria defined by the EvAAL framework, as detailed below. 1) Secret Path: The final path is disclosed immediately before the test starts, and only to the competitor whose system is under test. This prevents competitors to design systems exploiting specific features of the path. This criterion is always respected in all Tracks given the way the off-site competition Tracks are set up: competitors are provided with training sets, ground truth and, in some Tracks, a map. When they have finished tuning their systems, they ask the organisers for a path without ground truth, and submit their estimate. The ground truth is published only after the competition is finished. 2) Independent Actor: The actor is an agent not trained to use the localisation system. This criterion is always respected, given the way the off-site Tracks are set up. 3) IndependentLoggingSystem: The competitor system estimates the position at a rate of twice per second....This criterion is respected or exceeded in all Tracks. ... and sends the estimates on a radio network provided by the committee. This prevents any malicious actions from the competitors. The source code of the logging system is publicly available. This criterion is not respected, because the competitors may retry and further tune their systems while trying to guess the correct ground truth. To avoid this, the committee should ask the competitors to provide their code, and run it locally in a real-time fashion, or provide a real-time APIs. This is feasible, in principle, but would require a non-trivial software infrastructure to be in place, and a non-trivial additional effort from the competitors to comply with it. 4) IdenticalPathandTiming: The actor walks along the same identical path with the same identical timing for all competitors, within time and space errors smaller than the above defined measurement resolution. This is a natural consequence of the fact that the same data are provided to all competitors. B. Competition Results For each submitted trial, the error was computed by comparing the estimated coordinates with the ground truth, that is, reference coordinates of the key points marked on the ground along the path. This metric combines the floor detection accuracy and the horizontal positioning error. ε=PR−PE+p·|fR−fE|(1) where •PRis the vector with the ground truth horizontal (2-D) coordinates •PEis the vector with the horizontal coordinates estimated by the competitors •PR−PEis the horizontal error, and it is computed as the Euclidean distance between the ground truth and the estimated position provided by the competitor in the 2D space. •pis the base floor estimation error penalty and is set to 15 m. •|fR−fE|is the absolute difference between the actual floor number and the estimated one. The point error εis computed for all key points marked on the ground that define the path of a specific challenge. The “accuracy score” sis given by the third quartile of ε: s=3rdquartile {ε}(2) The team with the lowest score wins the challenge. Note that each competitor had the opportunity to submit the results for multiple trials. Table II shows the scores for all the five Tracks. Some additional metrics included in the ISO/IEC 18305 standard are also reported in the table. Fig. 1 depicts the cumulative distributions of the accuracy score sfor the winners and runners-up of the five Tracks. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5015 TABLE II RESULTS FOR ALL TRACKS.THE FIRST COLUMN ISTHECOMPETITION SCORE (EQUATION 2), WHILE THE REMAINING COLUMNS SHOW OTHER COMPLEMENTARY RELEVANT METRICS (MEAN, RMSE. MEDIAN,95 th PERCENTILE AND FLOOR HIT RATE (IFAVAILABLE). WEALSO INCLUDE A REFERENCE TO THE SECTION WHERE THE SYSTEM ISDESCRIBED Fig. 1. Cumulative distributions of point errors (Equation 1). IV. TRACK 3: SMARTPHONE A. Track Description The goal of Track 3 is to evaluate the performance of different integrated navigation solutions based on a regular smartphone sensor fusion (magnetometer, barometer, wireless communications, Attitude and Heading Reference System (AHRS) or micro-electro-mechanical systems (MEMS), among others) in an off-site context. As done in the 2016–2019 editions [1], [2], [10], [11], the same data collection strategy and evaluation procedure has been followed. 5016 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 TABLE III INFORMATION OF THE SENSORS IN THE SAMSUNG GALAXY A5 2017 (SM-A520F) The competition data set was collected by the same actor using a Samsung Galaxy A5 2017 (SM-A520F) phone with Android 8.0. Despite being three years old, this model has the advantage of having being used in Track 3 for 2018 and 2019 competitions. The main features of the embedded sensors, including the maximum sampling frequency, are summarised in Table III. As in the previous competitions, we have used the Android app “GetSensorData” [12] to record and store the smartphone sensors data into a single text file, i.e. into a logfile. The data set is split into three subsets, namely training, validation and evaluation: •The first set is devoted to calibration purposes and covers most of the evaluation area; it contains 18 short single-floor tracks (collected 4 times each), 4 long trajectories across bookshelves and 2 floor transition tracks. We placed key points at every relevant location, i.e. initial/final locations, significant turns in the tracks and the last step to arrive at a new floor. A total of 78 training logfiles were provided to competitors. •The second set is devoted to validation purposes, allowing competitors to have an initial assessment of the positioning system, and contains 13 multi-floor long tracks. The number of key points is arbitrary and significantly lower than in the training set. A total of 13 validation logfiles were provided to competitors. •The last set is devoted to evaluation purposes, allowing competitors to have an independent evaluation without ground truth data, and contains just 1 multi-floor very long track. In contrast to the systematic data collection done in the training files, the evaluation logfile included realistic movements (e.g. simulating a user that was messaging or attending a phone call) and stops. Only 1 unlabelled logfile was provided to competitors. We set the maximum allowed sampling frequency in “GetSensorData” for all sensors to record as much as possible data. Additionally, the smartphone was not connected to any cellular or Wi-Fi network as, for instance, the Wi-Fi sampling frequency significantly drops when the phone is connected to a Wi-Fi network. The logfiles and supplementary materials are available in [13]. This package complements the ones from the previous editions [14]–[17]. B. Competition Area The environment selected for Track 3 is a modern multi-storey library building located at Universitat Jaume I Fig. 2. Floor plan of UJI’s Library. (Castellón, Spain) and includes a small outdoor area near the main entry. This environment covers the use case for a smartphone application guiding students and staff to find the location of a book. Before collecting data, the library building was visually inspected to identify the most challenging parts where competitors could find it difficult to obtain accurate positioning. We finally selected the main hall entry (≈300 m2), the second floor (≈1000 m2), the third floor (≈900 m2), the floating fourth floor (≈200 m2) and the fifth floor (≈700 m2) to collect data. We discarded common areas on the first floor and zones with restricted access. The library is composed by two interconnected blocks, we mainly collected data in the first block, except for the fifth floor, where part of the second block was finally surveyed. For the evaluation path, we considered a walk done by a student that was doing some homework in the library. The student starts sitting in his/her work place (on the third floor), the student stands up and looks for a book, attends a phone call –despite it not being allowed–, comes back to the main workplace and stops for a while. Then, the student needs additional materials that is on the fifth floor, and goes directly there. The book is not in the place it was supposed to be, and the student asks a mate. The book seems to be in the new bookshelves located in the same floor but in the second block (left side in Fig. 2). On the way to the second block, the student meets a friend in the floating fourth floor (which was not mapped). Our student gets the book, returns to the work place, but the computer and other materials are gone. The student has not realised being instead on the second floor and starts to look around desperately. The student goes out to notify the security staff about this event. When the student goes back to the workplace on the third floor, he/she realises everything is there and sits to continue working after 20 minutes walking. The path goes through 82 key points for a total length of approximately 1000 m. We have used geo-referenced indoor maps (ArcGIS engine) and the ArcMap tool to calibrate all the reference points used in the data set. We performed on-site local measurements using a laser distance meter with respect to representative points, such as walls, pillars and doors, which were already well represented in the indoor maps. The inaccuracies deriving from this procedure might be considered irrelevant as all the information and indoor maps are provided to all competitors. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5017 Fig. 3. Tasks during the training stage of WHU-FIVE team system. C. Indoor Positioning Solutions Provided by Competitors 1) Team WHU-FIVE: Just like many indoor positioning systems, the WHU-Five system includes two stages, the training stage and the testing stage. In the training stage, the system attempts to build fingerprints, train the models and extract some information about the positioning environment. In the testing stage, it deals with the testing data to form the final trajectory, leveraging the trained information and fusion algorithm. a) Training stage: In the training stage, four tasks are performed that can be seen in Fig. 3. The first one is the feature map construction. In this task, the Wi-Fi fingerprint and geo-magnetic fingerprints map is built, and extract the book-shelf layout from the book-shelf training data set. The second task is the training of the step length model. The third one is the extraction of the stair-steps’ number of each two layers from the Floor-Transition data set. The last task is training the motion pattern model to recognize some motion types, such as up/down stairs, in/out doors, in-situ steering motion and phone-holding posture. In order to build the feature maps, the position of every sampled signal features must be known. The reference points are used in the training sets to optimize the Pedestrian Dead Reckoning (PDR) algorithm to obtain a highly accurate trajectory estimations. The model between PDR and reference points can then be build. The model is optimized with the LevenbergMarquardt (LMA) optimal algorithm. With the optimal PDR, the feature maps for Wi-Fi and geomagnetic can be build. Also, the book-shelf layouts can be extracted. As for the step length model, a leverage linear regression is used to train the model parameters. For the stair-step number between each layer, the average steps of each stair on the stair training data set is used. The motion recognition is performed by training a multi-layer perceptron neural network model. The time domain and frequency domain characteristics are extracted from the original data of the Inertial Measurement Unit (IMU) and barometer sensors, and then are input to the neural network. The output of neural network is the motion types set. b) Testing stage: In the testing stage, first, a Wi-Fi fingerprint positioning is used to find out the initial 3-dimensional position. Then, PDR is fused with the building map and geomagnetic fingerprint positioning result to estimate the 2-dimensional trajectory. Meanwhile the motion recognition Fig. 4. Main algorithm of WHU-FIVE system. Fig. 5. IOT2US system overview. result is used to revise some error estimation of PDR. Lastly, the motion on stairs is used to estimate the layer id. Combined with the layer id, the 2-dimensional trajectory can be built up to 3-dimensional trajectory. For the final testing trajectory estimation, the IMU is used to provide the original PDR. The recognized in-situ steering motion is used to weed out the corresponding steps and the phone-holding posture is used to revise the heading of PDR. Then the revised PDR is fused with the building map and book-shelf layout in particle filtering algorithm to obtain further trajectory estimation. The geomagnetic fingerprint matching positioning result is then fused with the previously estimated trajectory in Kalman Filter (KF) algorithm to obtain the final 2-dimensional trajectory estimates. Combining the layer estimation result and the 2-dimensional estimates, the final 3-dimensional trajectory is obtained. The algorithm of testing stage of the system as shown in Fig. 4. 2) Team IOT2US: IOT2US team system includes four main stages: 1) the floor and region decisioning based on Wi-Fi; 2) the mobility mode detection; 3) the landmark detection; and 4) the PDR algorithm and information fusion. Six types of sensor data were fused in the whole processing of the track reconstruction and each part involved some of them, as can be seen in Fig. 5. In the following paragraphs every stage is explained in detail. a) Wi-Fi: According to the training data, there are five floors involved in this Track. To determine the floor information and get the rough location of the user, Wi-Fi Received Signal Strength (RSS) information is used to decide the floor and region. Here region is defined as the area that each training logfile covers. 5018 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 There are two phases in the Wi-Fi process. The RSS fingerprints which contains time, floor, region and key-value pair of MAC and RSS, are built from the training data set during the offline phase. The RSS fingerprint can be represented as a vector of (time,floor,region,mac,rss). And during the online phase, the Wi-Fi RSS information in the evaluation logfile is compared with the RSS fingerprints to compute the most suitable location. More specifically, there are two steps in the online phase. First, the floor and the region of estimation of the point are determined with respect to the wireless Access Point (AP) availability. A coefficient (λ) is defined to indicate the possibility to estimate a point appearing in the region to which the RSS fingerprint belongs, λ=1−n ne+nr (3) where, the neis the number of APs detected at the estimation point, the nris the number of APs detected at the RSS fingerprint. The nis the number of APs detected at the estimation point and the RSS fingerprint at the same time. This coefficient λis calculated for every RSS fingerprint that belongs to the same region. The region with the minimum sum λis the estimated region to which the estimation point belongs to. Then the floor based on this estimated region can be obtained. Second, to get the rough location of the estimation point, the RSS information is compared against all RSS fingerprints. The Euclidean distance in the signal space between the estimation point and every RSS fingerprint is calculated as: d=    m  i=1rssi e−rssi r(4) where, the rssi eand rssi rare the RSS values of the i-th AP detected at the estimation point and the RSS fingerprint, respectively. b) Movement modes recognition: Different modes of mobility can be detected using machine learning or deep learning algorithms using multi-sensors data. For this Track, four categories of motion modes were introduced: normal walking, turning, climbing (stairs), descending (stairs). The process chain mainly includes data segmentation, labelling, feature extraction and classification. In IOT2US team system, accelerometer, gyroscope, magnetic field, and pressure are used for motion modes classification. Some statistical characteristics (e.g., mean, max, derivative) of these time series in time-domain are extracted as features. Decision tree and Support Vector Machine (SVM) are investigated to classify these motion modes. c) Landmark detection: Map information is one of the most important clues that can help to correct the trajectory. Traditional map matching trends to make use of the structure of the rooms, corridors and tunnels to restrict the trajectory. However, this request too many details of the map and sometimes to measure the building structure in detail is a heavy workload. Hence, some landmarks are identified that activities can only happen at certain places as Correction Reference Points (CRP) to correct the trajectory. Fig. 6. Light intensity determined CRPs in IOT2US system. Through analyzing the relationship between the real environment and sensors data, three types of CRP can be identified, which are determined by Barometer, Ambient light sensor, and door-crossing activity: 1) the activity of climbing and descending stairs can only happen at stairs, and hence, with a rough location using Wi-Fi can give at least two CRPs. 2) Light intensity measured by Ambient light sensor also has relationship with activities. As an example of walking across bookshelf activity, as shown in Fig. 6, the activity of walking out of the bookshelf and turning backing is recorded as a peak in light intensity. The light determined CRPs also can be determined at place of crossing doors, enter/leave the building and approaching to a window. 3) The third type of CRP determined by door-crossing is similar with the first one that requires recognize the door-crossing activity and Wi-Fi to find a rough location to determine the position of a door. d) PDR and Information Fusion: Step counting, step/stride length estimation and heading determination are three crucial processes for PDR positioning systems. Peak detection [18] is used to count steps, the Weinberg method [19] to estimate step length and complementary filter [20] for heading determination. To overcome this the traditional PDR algorithm issue of error accumulation over elapsed time, information is fused, including movement modes, floors, regions and landmarks. First, as the sensor data may show distinctive characteristics when a user performs different activities, movement modes are used to finely tune the parameters of step counting and step length estimation. For example, when a user is climbing or descending stairs, the parameters of step counting and step length estimation algorithms is accordingly adjusted to improve their performance. Second, before calculating locations using PDR, Wi-Fi is used to determine the absolute floor and the region information where the user roughly locates. This process assisted IOT2US to provide absolute location to help calculating the PDR trajectory and determining a CRP. And third, if the user is detected as reaching at a CRP, but the calculation result diverges from it, the heading is adjusted, step length and the previous track, to drive the trajectory back to the CRP. 3) Team XMU_ATR: XMU_ATR proposed XMU_PDR system. It is a multi-source indoor positioning system using information obtained from a Inertial Measurement Unit (IMU), POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5019 Fig. 7. Framework of XMU_ATR team system: XMU_PDR. Wi-Fi, magnetometer, barometer and indoor maps, jointly. Fig. 7 shows the framework of the proposed system. There are four main functional modules described as in the following paragraphs. a) Inverted pendulum model based Pedestrian Dead Reckoning (PDR): A raw trajectory is obtained from the inertial data using an inverted pendulum model based PDR algorithm [21] by estimating every pedestrian step length and heading angle directly. The proposed system uses an inverted pendulum model to calculate the step length. To achieve 3-D positioning, the barometer is used to estimate the transition of floors. In the PDR system, heading errors are one of the main factors when estimating positions which will lead to a decrease in positioning accuracy over time, other information needs to be introduced to correct the trajectory. b) Wi-Fi fingerprints matching: Wi-Fi fingerprints matching is a reliable way to obtain absolute indoor position. Since the ground truth of some reference points in the training set is provided, the Wi-Fi Received Signal Strength (RSS) is extracted at the reference points as fingerprints to build up the fingerprints database. Weighted k-Nearest Neighbor (WKNN) algorithm is used to match the Wi-Fi Access Point (AP) between fingerprints database and the RSS from evaluation data. Since the initial point is unknown, the trajectory obtained by PDR can only be presented in a temporary navigation frame automatic defined by the dead-reckoning system. Therefore, the result of Wi-Fi positioning can be used to estimate the transformation relationship between the navigation frame and the geographic frame, including coordinate translation and rotation. c) Magnetic fingerprints matching: The indoor magnetic field can be treated as a time invariance distribution in spatiality. The accuracy of spatial resolution can achieve centimeter level in a small area. The magnetic fingerprints matching is used for trajectory refinement. Since the trajectory to be evaluated may have some overlap with the training set, the observations of magnetometer in the training set are also used as labeled fingerprints. A modified dynamic time warping algorithm is used in this part which can deal with the matching problem between two sequences with different directions. A matching threshold is set to decide whether the matching is successful. If there is a trajectory in the evaluation set match to part of trajectories from the training set, this track can be located on the map. d) Map matching: To reduce positioning error accumulated from the noise of inertial observations, the map information is used for trajectory calibration. The optimal estimation under the map constraint especially the track at specific locations such as walls, doors, and stairs are realized. The floorplan is presented in the form of grids, and a loss function is defined to adjust trajectory actions referring to walls and some specified behavior patterns. 4) Naver Labs Europe (NLE) Team: Naver Labs Europe (NLE) Team system is based on extending the localization pipeline developed for IPIN 2019 [1] challenge with new components. The pipeline is a sensor fusion framework deploying smartphone inertial sensors, Wi-Fi measurements, magnetic field data and landmarks. The main components are described in the following paragraphs: a) Floor detection: Floor is detected using a standard k-Nearest Neighbor (KNN) classifier trained on barometer and Wi-Fi data. b) Activity detection: User’s activity (walking, standing, going up or down the stairs) is identified by applying spectral analysis on the accelerometer data and simple thresholds. c) PDR: User steps are first identified by applying peaks detection to accelerometer data.Then, the acceleration features are extracted and a model is trained for the step length/speed in a given sliding window. Together with the orientation sensor, a first order approximation PDR of user’s track is built. 5026 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Fig. 17. PERSY and description of Track 4 over Atlantis shopping mall. TABLE V STEPS DESCRIPTION COMPOSING Data Set Num 2 Fig. 18. Temporal view of steps composing Data set num 2 . The competitors’ objective was to re-build the trajectory realised by the Track chairs. The evaluation was done by comparing 2D position and floor level estimated by each team to the coordinates of 67 reference points (key points). To do so, a Table containing timestamps of expected key points was shared, and competitors had to provide the corresponding coordinates. TABLE VI INFORMATION ABOUT EMBEDDED SENSORS INSIDE PERSY Data Set and supplementary materials –e.g. data sheet of sensors embedded in PERSY– were provided to competitors of Track 4. These contents and the ground truth location for evaluation are now available for further benchmarking in [38]. This package complements the ones from the previous editions [39] and [40]. B. Competition Area For IPIN 2020, due to Covid-19 situation, Track4 competition was held in “Atlantis Le centre”, a large shopping mall close to Nantes - France. This site has already been used in IPIN 2018 for all competition Tracks, and a very accurate survey was realised. This has eased the design of the ground truth (see [1] for details on the survey). There were multiple difficulties when surveying such a big shopping mall: wide areas, lifts, escalators, and even a carousel, as illustrated in Fig. 19. Complexity related to the Covid-19 also led the Track chairs to make loops on the path in order to respect the direction of travel, as shown in Figure 20. C. Indoor Positioning Solutions Provided by Competitors 1) Team WHUGNSS: The classic zero-velocity update algorithm (Zero-velocity update (ZUPT)) based foot-mounted pedestrian dead reckoning consists of a strap-down inertial navigation algorithm, a stance phase detection algorithm, and an error state Kalman filter. However, the classic ZUPT-based Foot-PDR [41], [42] cannot overcome the influence of the complex motion of the pedestrian. The WHU-GNSS team system is based on several schemes designed to improve navigation performance, as shown in Fig. 21. The core algorithm is the strap-down inertial navigation algorithm. On this basis, a zero-speed detection method with adaptive threshold setting is used to adapt to different users. Next, the motion pattern recognition algorithm is used to distinguish whether the user is walking normally or taking the escalator and elevator, and uses constant speed, Zero-velocity update (ZUPT), Zero angular rate update (ZARU), Improved heuristic drift elimination (iHDE), linear trajectory and height constraints to improve the position estimation accuracy according to the discrimination results. In addition, the magnetic field will be used to detect whether the user has returned to the place where they have walked, so as to correct the current navigation state with the historical estimated position. And when the user comes to an outdoor scene, the GNSS signal will be used to improve the final positioning performance. a) The multi-constraint algorithms: The classic Generalized Likelihood Ratio Test (GLRT) method is one of the most common algorithms for detecting the stance phase [43], [44]. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5027 Fig. 19. Track4 difficulties over the path. Fig. 20. Left part: imposed direction of travel. Right part: multi-floor environment. Fig. 21. Block diagram of multi-constraints-based foot-mounted PDR algorithm of WHU-GNSS team system. In the WHU-GNSS team system, an improved adaptive threshold is used instead of the fixed threshold method to detect the stance-phase in each gait cycle. The adaptive threshold method is adaptable to different gait frequencies in dynamic motion. Once the stance phase is detected, a zero velocity vector is used to estimate and correct the navigation error [41]. The heading angle error and the z-axis gyroscope bias of the ZUPT algorithm are unobservable. Thus, the following methods are used to constrain the error divergence of the heading angle. First, the Zero angular rate update (ZARU) algorithm is employed to estimate the gyroscope bias and heading angle error [45]. Compared with the stance phase detection algorithm, a stricter fixed threshold is used and a more extended continuous period to determine the update chance of ZARU. Second, when a pedestrian is determined to be walking in a straight-line path or the corridor’s primary orientation, the Improved heuristic drift elimination (iHDE) and the straight-line constraint algorithms are applied to estimate the heading angle and the z-axis gyroscope bias [46], [47]. These algorithms can effectively improve the performance and reliability of pedestrian navigation. The height error divergence is also a significant problem in Foot-PDR, especially for multi-floor navigation and positioning applications. In the absence of a barometer, an effective height constraint algorithm is adopted to reduce the error drift along the vertical channel. When pedestrians go up and downstairs, the slope angle can be considered constant in most cases [48]. In the WHU-GNSS solution, the stride length and the slope angle between adjacent footsteps are used to determine whether the pedestrian is walking on a plane or going up and downstairs. Then the slope-based or plane-based height constraint algorithm is used to improve the estimated height accuracy in Foot-PDR. The other extreme scenario is the escalator or lift. Usually, escalators run at a constant speed. When a pedestrian stands relatively static on the escalator, the specific forces measured by the foot-mounted IMU are almost all derived from local gravity. The gravity information can be fused in a tightly coupled manner in the WHU-GNSS solution, so the drifting error can be constrained even when a pedestrian stands still on an escalator. Moreover, when the pedestrian takes a lift, the specific forces (i.e., the accelerations) will exhibit clear acceleration motion and deceleration motion process. The vertical (up or down) velocity information of the pedestrian can be estimated using acceleration and deceleration motions. Thus, the vertical velocity can be as observation information to improve the performance and stability of the Foot-PDR. Many ferromagnetic materials exist in indoor building structures. So, magnetometers cannot be used to determine the heading angle in Foot-PDR directly. Yet, combined with a rough position, the magnetic field signals can recognize similar areas when the pedestrians return to places they have walked before. This meaningful information can help improve the robustness of Foot-PDR in practical application. The Foot-PDR is integrated with GNSS signals in a loosely-coupled manner [42]. Satellites with small elevations should be discarded to avoid the gross error as much as possible. Besides, some measurements with low quality judged 5028 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Fig. 22. The scheme of foot-mounted PDR system based on multi-constraint algorithms proposed by AIR team. by the innovation vector’s magnitude and covariance need to be rejected in the Kalman Filter (KF). Furthermore, the adaptively robust filtering algorithm is used to control the effects of inaccurate measurements in the WHU-GNSS solution and improve system accuracy. The optimal inertial sensor parameters (i.e., the bias instability of gyroscopes and accelerometers, the angular random walk, and the velocity random walk) are determined through the provided long-term static data. The magnetometer is also calibrated through the classic ellipsoid fitting method. 2) Team AIR: The pedestrian foot-mounted PDR system proposed by AIR team is shown in Fig. 22. In the above framework, five constraint algorithms are included in the middle modules: Stance & Still Phase Detection, the Heuristic Drift Elimination (HDE), the Height Update Algorithm (HUPT), the Zero-velocity update (ZUPT), and the Earth Magnetic Yaw. Meanwhile, the Stance & Still Phase Detection includes two components: the Generalized Likelihood Ratio Test (GLRT) detector algorithm used under the condition of the slow and normal pedestrian gait speed, and the Hidden Markov Model (HMM) detector algorithm used under the condition of the dynamic and fast pedestrian gait speed. After that, using the improved HDE and HUPT method to estimate current position errors, ZUPT is used to estimate the velocity error, while Earth Magnetic Yaw based on quasiStatic Magnetic Field (QSMF) method is used to estimate the heading error. a) The multi-constraint algorithms: A gait or a walk cycle consists of two phases: the swing and stance phase. In the swing phase, the foot is not in contact with the ground. In contrast, the foot contacts the ground in the stance phase. GLRT algorithm has obvious advantages for zero speed detection of stable pedestrian gait velocity, while HMM algorithm has a good effect for zero speed detection of dynamic and fast pedestrian gait speed. Thus, the two methods are combined to achieve the dynamic human stance & still phase detection [49]. Fig. 23. Revise the current step’s inertial recursive position with the position calculated from the stride heading in the AIR team system. When the Stance & Still Phase Detection detects the stance and swing phases of human foot gait from the data from IMUs, ZUPT method is used to constraint the velocity divergence [50]. HDE algorithm is a very useful method to constraint the system’s heading drift, if the indoor reference heading can be known in advance. In the AIR team method, the initial heading is used to calculate several possible reference directions of pedestrian walking [48]. Then, unlike the existing HDE method, which mainly corrects inertia recursive heading, the closest reference direction is used to calculate the estimate position at the current footstep, then uses the position error between the estimate position and the inertia recursive position to restrain the position divergence. The procedure is shown in Fig. 23. Height divergence is a major problem in Inertial Navigation System (INS)-based foot-mounted PDR system in multi-story positioning. If a pedestrian is walking on a plane, the slope of the current stride is approximately zero degree, if that, keep the height always unchanged. While walking on a staircase, the method proposed uses the actual slope of the stairs (usually 20 45 degrees) to calculate the height change of the current stride, which can be used to constrain the height divergence of the current stride [48]. If pedestrian is on an elevator or escalator, it mainly can be effectively determined by analyzing the characteristics of acceleration, especially the acceleration in the vertical direction. The magnetic field is very useful to estimate the heading of the system, but the magnetic disturbance has a severely effect on the estimation. In AIR team system, an improved QSMF method combined with a compass filter is used to estimate the heading in the perturbed magnetic field [51]. In addition, in areas where pedestrians repeatedly walk, a series of magnetic sequence information is used for pedestrian trajectory matching to improve the effect of heading constraint. 3) Team Free-Walking: The positioning system proposed by team Free-Walking is shown in Fig. 24. The Free-Walking system combines data pre-processing, motion mode recognition, INS mechanization, adaptive zero velocity detection, ZUPT-aided Kalman Filter (KF) and altitude constraint. The data pre-processing includes sensor calibration, filtering and Coordinate system transformation. After POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5029 Fig. 24. System architecture of proposed pedestrian inertial navigation based on motion mode recognition proposed by Free-Walking team. pre-processing, motion mode recognition algorithms are used to help adaptive threshold ZUPT detection. Then a ZUPTbased KF is used to get position information. Meanwhile, motion mode results is also used to constraint height error. a) Error-constraint method based on walking mode: For pedestrian positioning, the human motion modes describe the overall movement of pedestrians. The pedestrian motion modes are particularly important for pedestrian navigation, while the pedestrian motion modes are variable during the procedure of pedestrian navigation. Therefore, a walking mode classifier is designed (see Fig. 25) based on the stacked denoising autoencoder [52] and temporal Convolutional neural network (CNN) with attention to recognize eight pedestrian motion modes [53], [54]. ZUPT-aided INS has ability to suppress navigation errors. Free-Walking team uses the periodic gait-cycle window to divide the pedestrian movement into discrete gait cycles; then, the minimum value in each gait cycle is taken as the zero-speed state point. The time length of the gait cycle is different under different motions. The gait-cycle duration is adaptively adjusted based on the classification result of walking mode to adapt to various pedestrian motions [55]. Compared to the existing methods, the proposed method does not need to set the zero-speed detection threshold, and performs well for zero-speed interval detection under various pedestrian movements. The stationary state of the foot during the stance phase is taken and feeds the zero-velocity information (pseudomeasurement) into KF to compensate for the velocity, the position and the attitude errors. The height errors in Strapdown Inertial Navigation System (SINS) solution will grow without boundary and cannot be eliminated by ZUPT measurements. When a user walks on the same floor, the altitude does not change. The altitude changes only when the user goes up and down stairs. Therefore, the vertical displacement of pedestrian is constrained by two factors: stair height and motion mode. If the height of each stair in a multi-floor building is fixed, the height of each gait cycle is determined by the number of walking stairs in that gait cycle. Therefore, the classification result of walking mode is used to constrain the height error. 4) Team BHSNIP: The Pedestrian Navigation System (PNS) based on Inertial navigation system–extended Kalman filter– zero velocity update (IEZ) –also referred as INS-EKF-ZUPT– is widely used in complex environments without external infrastructure owing to its characteristics of autonomy and continuity. However, due to the poor observability of heading errors to ZUPT and the instability of vertical inertial channels, further corrections of the estimated trajectories under the IEZ framework are still needed to obtain higher positioning accuracy. In order to achieve high performance for PNS in terms of accuracy and robustness, BHSNIP team integrates the Micro-Electro-Mechanical Systems–Inertial Measurement Unit (MEMS-IMU) and Global Positioning System (GPS) as shown in Fig. 26. In this scheme, MEMS-IMU provides the 3-axis accelerometer, 3-axis magnetometer, and 3-axis gyroscope readings which are [ fxfyfz], [magxmagymagz], and [ωxωyωz] in the body frame, respectively. The main work has the following features: 1) Aiming at the weakly observability of heading drift for MEMS-IMU, the iHDE algorithm is proposed. The algorithm has the following three steps: First, heading information is extracted from pedestrian’s straight-line motion track, which is used to construct four or eight datum directions of the building; second, building heading information is utilized to estimate yaw errors of trajectories that satisfy specified rules; and third, these yaw errors are utilized as the EKF observation to estimate the state error of the navigation parameters. 2) In order to deal with the problem that the inertial vertical channel is unstable under the traditional IEZ framework, which makes it impossible to locate the floor by SINS solutions, the improved step height equidistant (ISHE) is exploited. At the beginning, the adaptive network-based fuzzy inference system (ANFIS) is used to identify different vertical modes including elevator, escalator and staircase (walking upstairs, horizontal movement, and walking downstairs). Then, the floor information or altitude is estimated by ISHE. 3) To detect the stance phase accurately, adaptive-ZUPT algorithm is used based on backward neural network. In conventional researches, positioning performance is easily affected by the ZUPT with fixed threshold, because it is difficult to determine ZUPT conditions for jump, fast walking, running. 4) GPS is fused with MEMS-IMUMEMS-IMU through Robust Extended Kalman Filter (REKF), which can remove the contaminated points of GPS signal. What is more, GPS can provide global coordinates. Fig. 27 shows the horizontal trajectory. The estimated track starts from the red circle and the blue line represents the moving trail of the pedestrian based on the proposed method. The positive direction of abscissa and longitudinal represents 5030 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Fig. 25. Pedestrian walking mode recognition based on the stacked denoising autoencoder and temporal convolutional network with attention in the Free-Walking team system. Fig. 26. Scheme of BHSNIP team positioning system. Fig. 27. Estimated horizontal trajectory by Team BHSNIP for Track 4 of IPIN Competition 2020. East and north respectively. The track in the figure is shown in relative coordinates that will be transformed into the WGS84 coordinate system. Fig. 28 illustrates the three-dimensional trajectory. The estimated track also starts from the red circle and the blue line represents the moving trail of the pedestrian based on the proposed method. The x-axis, y-axis and z-axis of the Fig. 28. Estimated 3D trajectory by Team BHSNIP for Track 4 of IPIN Competition 2020. coordinate system represent east, north and up respectively. The relative coordinates representing the track in Fig. 28 will be transformed to the WGS84 coordinate system. VI. TRACK 5: XDR CHALLENGE IN MANUFACTURING 2020 A. Track Description The purpose of Track 5 is to evaluate the practical performance of indoor localisation methods under realistic industrial scenarios. Indoor localisation competitions have been held, named “PDR Challenge” or “xDR Challenge” as the official competitions or the relevant event in past IPIN conferences. Track 5 is a sequel of the PDR/xDR Challenge series competition, which is named as “xDR Challenge in Manufacturing 2020”. In this year’s competition, the competitors are asked to estimate the trajectory of employees working in the factory and forklifts driven in the factory. As specific industrial scenarios, the target for PDR Challenge 2017 and xDR Challenge 2018 were picking operation in a warehouse [56], while for xDR Challenge 2019 it was POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5031 serving in a restaurant and manufacturing operations in a factory. The scenario of the competition for 2020 was manufacturing operations in a factory. Competitors were required to estimate operators’ trajectory and forklifts’ trajectory in the factory by utilising indoor localisation methods based on dead reckoning algorithm, positional correction methods with Bluetooth Low Energy (BLE) beacons and other information provided. Characteristics of Track 5 can be summarised as follows: 1) Utilisingthe DataActuallyUsedinthe Operation: Similar to other Tracks, Track 5 aims to compare practical performance of indoor localisation methods or systems under realistic industrial scenarios. Its most remarkable characteristic is that the data provided to competitors is obtained from by an analysing system for manufacturing operation which was used during real operation [57], after approval of provision of the data actually used. The operators are carrying Android devices which measure sensor data for the analysis based on indoor localisation. This means that in Track 5 data is not provided by an actor following a predetermined path, but by real operators doing their daily job. This adds significant difficulty in estimating the trajectory with respect to other Tracks, mostly because the target movements include various types of motion during the manufacturing operations, rather than simply walking at constant speed and staying still for a while. As the data set, we provided measured sensor data that include angular velocity, acceleration, magnetism, atmospheric pressure, and RSSI of BLE beacons. Also, partial ground truth positions are provided for correcting the position. These ground truth data are assumed to be available from the record of the operations and required for long-term estimation by indoor localisation. The lengths of the data are in units of working hours. The lengths per data are about 2 hours to 7 hours. 2) Evaluating Dead Reckoning Methods for Various Types of Moving Objects: The PDR/xDR Challenge series competitions deal with indoor localisation methods based on various types of the dead reckoning methods. Dead reckoning for vehicle is called Vehicle Dead-Reckoning (VDR). The term “xDR” is used to indicate various types of dead reckoning. The target of the Track 5 is not only operators working in the factory, but also forklifts driven in the factory. Dead reckoning of the vehicle such as the forklifts is a quite challenging topic. Thus, there are two separated sub-Tracks for PDR and VDR. 3) Multi-Faceted Evaluation of Performance for Indoor Localisation Methods: In order to evaluate practical performance under industrial scenarios, multi-faceted evaluation metrics has been used. The evaluation metrics in the PDR/xDR Challenges has been revised. As the evaluation metrics for this year’s competition, a three-evaluation indicators and three-negative check criteria were adopted as follows: Evaluation indicators about error •Absolute error – Circular Error (CE): absolute 2D positional error compared with ground truth position. •Error distribution bias – Circular Accuracy (CA): evaluating degree of bias of error distribution in 2D error space. •Error accumulation gradient (EAG): evaluating speed of error accumulation caused by relative tracking with dead reckoning. Negative Checks •Requirement of moving velocity: checking if local moving speeds in the trajectory are less than a defined threshold. •Requirement of validity of trajectory: checking the incursion of the trajectory into un-walkable area. •Coverage ratio: check if each evaluation point has corresponding submitted results. Each evaluation indicator and criterion are converted into evaluation indexes up to 100 and weighted summed for calculating the integrated index which determines the winner of the competition. We adopted median of CEs (CE50) as an indicator of the absolute error. The error accumulation is the one of main concerns in relative tracking method such as xDR. In order to evaluate the error accumulation, BLE signals in the data set have been intentionally and partially deleted [56]. Partial ground-truth position is provided for error correction and for evaluating the speed of error accumulation from the correction points where the ground truth position is provided. Competitors are required to deal with these unique characteristics of the data set. CE75 has not been used for determining the winner, but only for comparison according to the EvAAL framework. However, CE75 can be easily calculated by using our evaluation script for calculating evaluation indicators and negative checks. Please refer to the script shared on the GitHub for further details [58]. B. Competition Area The target field for the PDR subtrack is shown in Fig. 29. The target field of the VDR subtrack is shown in Fig. 30. We provided some examples as sample data sets. In the figures, examples of the movements of an operator and a forklift are shown in blue dots. The yellow dots represent examples of the partial ground truth data for correcting the positional errors. The black coloured areas represents the un-walkable areas. Competitors are able to avoid the incursion into the un-walkable area by using map matching techniques. BLE beacons are arranged in the target area for absolute localisation and positional correction. According to the demands of the factory for maintenance, solar-powered BLE beacons, Fujitsu’s PulsarGum, are used. The interval of signal emission is 1.26 s at minimum, but it is not guaranteed and varies in proportion to the amount of generated electricity. Competitors are required to deal with this characteristic of the beacon. C. Indoor Positioning Solutions Provided by Competitors 1) Team KawaguchiLab: Team KawaguchiLab has studied IMU-based indoor localization using smartphone. In the 2020 competition, the challenge was to integrate KawaguchiLab IMU-based research with non-IMU sensor based system (BLE, map information), and to build a robust indoor positioning system. KawaguchiLab system is simple because it makes no complex assumptions. Therefore, even in a Track 5 environment where there are few movement constraints, it works 5032 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Fig. 29. The target area of PDR subtrack in Track 5. Fig. 30. The target area of VDR subtrack in Track 5. Fig. 31. The scheme of three steps indoor localization system of KawaguchiLab team system. robustly, although there is a trade-off for some loss of accuracy. Fig. 31 shows overview of KawaguchiLab team system. It consists of three phases: denoising, dead reckoning, and compensation. a) Denoising phase: Gyroscopes have an offset that depends on the inherent characteristics of the sensor and temperature. It causes a serious cumulative errors in dead reckoning. Hence, they are removed using real-time offset removal algorithm: First, whether the sensor is stationary or moving is obtained with an Fast Fourier Transform (FFT) based method; second, the offset by averaging the angular velocity while stationary is calculated. Finally, the angular velocityis calibrated using the latest updated offset. b) Dead reckoning phase: In speed estimation, Deep Neural Network (DNN) base method is used [59]–[61]. Deep neural network architecture consist of Long Short-Term Memory (LSTM) and full-connected layer LSTM extract time series features of 3-axis acceleration by sliding window and full-connected layer converts time series features to speed. This approach gains robustness to noisy data and work with various gaits. In heading estimation, first, gravity direction ˆgDCS is estimated using Multiplicative Extended Kalman Filter (MEKF) [62]. DCS represents the device coordinate system. Secon, angular velocity ωDCS is projected to gravity to get the horizontal angular velocity ˆωGCS z. GCS represents the global coordinate system. Projection process is as follows: ˆωGCS z=−ωDCS ·ˆgDCS ˆgDCS(8) Finally, the heading is calculated by integrating time-series horizontal angular velocity. Integration process is as follows: ˆ h=ˆωGCS zdt (9) c) Compensation phase: Pseudo reference position from BLE signal is generated to compensate trajectory. BLE signals are searched using sliding window for about 10 s. Then, the distance from BLE beacon to subject is estimated using three or more BLE signals. A pseudo reference position by using these distance. Similarity transformation model [63] is used to compensate the trajectory using the true reference position and pseudo reference position. The parameter of this model is updated using similitude ratio. The similitude ratio sis calculated by using actual moving distance dand estimated moving distance de. s=d de (10) The parameter alpha is updated by multiplying similitude ratio (α0=1). αk=sαk−1(11) Finally, the αis multipied to the estimated position change. The path is generated using map image as physical constraints to avoid obstacles. The shortest path from one reference point to the next one is calculated and with astar algorithm the next reference point is searched. 2) TeamYONAYONA: YONAYONA team indoor positioning technology is implemented in two stages: absolute position determination using BLE signals and map matching using map information. Using the acceleration and angular velocity measured by the IMU is a relative positioning approach, which often causes drifting errors. Therefore, YONAYONA system first efficiently estimates the location based on the RSSI, position, and signal strength parameters, and then corrects for the natural behavior of the person’s walking speed and direction. A major challenge for this algorithm is to deal with the situation when the number of observed BLE beacons is not enough or when there is a wall between the previous predicted position and the next predicted position. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5033 a) Absolute position determination: Since the number of BLE beacons observed is not constant, absolute positioning is calculated by selecting three beacons with high RSSI at a certain time (every 0.5 s in this implementation). The distance between the observer and the beacons can be computed by RSSI and Ptx (measured RSSI 0.1 m away from the beacon). As a result of trying various approaches to estimate the position based on the distance data, such as trilateration, position averaging, and position averaging with power value weighting, the average-weighted method, which has the least error, is applied in this algorithm. b) Map matching: In this section, an algorithm is build to predict realistic human movement based on the map data provided by the competition organizers. In cases where a line connecting two points estimated by absolute surveying would encroach into a wall, an inaccessible point with a nearby accessible one is replaced. Then, the estimated points are connected with each other in a smooth trajectory so that the walking speed can be kept within a sensible range. c) Problem: This algorithm relies on absolute position estimations, which makes it difficult to deal with situations where there is a large error in the value of the signal received from the beacons, or where the number of signals received is not sufficient. In this implementation, the estimation accuracy within the Absolute Localization Inapplicable Period (ALIP) time set at a specific time was reduced, resulting in a larger error. In fact, there were not enough time to build an algorithm that also implemented PDR and VDR by the competition deadline, so it is not possible to refer to relative positioning. A possible improvement to this technique is to design a robust system using the Kalman Filter (KF) from two estimates, one for absolute positioning and one for relative positioning. VII. TRACK 6: SMARTPHONE-BASED VEHICLE POSITIONING WITHOUT ADDITIONAL EQUIPMENT A. Track Description The goal of Track 6 is to evaluate the performance of different integrated navigation solutions based on the sensors of vehicle-mounted smartphone, such as GNSS, MEMS and magnetometer, etc. A Huawei mate20 smartphone was used to record raw multi-sensor data in the vehicle scene and a reference system based on Differential Global Navigation Satellite System (DGNSS) and Fiber Optic Gyro Inertial Navigation System (FOG-INS) with an expected accuracy of 5 cm at 1 Hz provided the ground truth. Two data sets were provided. The first one containing the ground-truth reference was used for sensor and algorithm calibration. The second one was for the calculation of the coordinates and accuracy evaluation. B. Competition Area The test route of Track 6 (see Fig. 32) includes an outdoor scenario with unobstructed satellite view, an attenuation scenario with partially obstructed view and an indoor scenario without satellite view. In the test process (see Fig. 33), there were several long interruptions of GNSS signal and an irregular test route was adopted. Besides the navigation measurements derived from the sensors installed in smartphone, there Fig. 32. The test route and GNSS condition of Track 6. were no external aid information and no prior knowledge of the test route. The competitors could only rely on smartphone to calculate the vehicle position. The test area of Track 6 was selected in Haidian airport and surrounding areas, Beijing. The whole test route was about 19 km and consisted of two phases: the initial alignment phase and the final evaluation phase. The initial alignment phase was carried out in an open sky scene. It can be specifically divided into the sensor calibration stage (traverse the posture states, about 3 minutes), the static initial alignment (about 5 minutes), and the dynamic alignment (several running, stop and turn around, about 15 minutes). The evaluation stage was carried out in the scene of GNSS signal obstruction and simulated interruption. It can be specifically divided into three stages: 1) frequent GNSS signal attenuation stage: obstructed buildings, tree shades, etc. – about 25 minutes; 2) simulated GNSS absent signal stage: completely interrupted, simulated by turning off the Mobile phone GNSS positioning function; 3) indoor parking stage – about 3 minutes. Following the EvAAL evaluation criteria, the 75% horizontal positioning error of competitors output points was evaluated. C. Indoor Positioning Solutions Provided by Competitors 1) Team WHU & AutoNavi: WHU&Autonavi Team system uses GNSS/INS integrated positioning as the basic algorithm and focus on making full use of vehicle motion constraint information and magnetometer observations to provide stable positioning services. Fig. 34 shows the flowchart of the vehicle integrated positioning algorithm based on smartphone built-in sensors. And the algorithm can be divided into 3 parts: 1) GNSS/INS integrated positioning algorithm (the red dotted part), 2) the vehicle motion model constraints (the orange part), and 3) magnetic heading constraint (the green part). a) GNSS/INS integrated positioning algorithm: GNSS/ INSintegrated positioning is the most basic and backbone algorithm in vehicle positioning scenarios. INS is used as a bridge to correlate all available observations, and GNSS, as the only available absolute positioning method in the offline mode of the smartphone, determines the positioning performance of the system. INS mechanization is employed to integrate the gyros and accelerometer output. Due to the low performance of the 5034 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Fig. 33. The test process of Track 6. Fig. 34. Flowchart of the vehicle integrated positioning algorithm based on smartphone built-in sensors of the WHU&Autonavi Team system. smartphone built-in sensors, the influence of the angular rate and sculling effect caused by the rotation of Earth and motion speed can be ignore [64], [65]. Therefore, the rigorous INS mechanization can be simplified to achieve more efficient calculations. An Extended Kalman Filter (EKF) is employed to fuse GNSS and INS for reducing the error caused by non-linearity. And the 20-dimensional navigation error state includes position, velocity, attitude, gyroscope bias, accelerometer bias, misalignment angle (the angular difference between the smartphone built-in sensor and the vehicle coordinate system), and the lever arm parameters (the offset of the sensor measurement center to the center of the vehicle coordinate system). To maximize the navigation performance of the sensor, the performance parameters of the gyroscope and accelerometer are adjusted according to the three sets of training data given by the competition. For smartphones, the distance between the GNSS antenna and the IMU measurement center is very close (e.g., several centimeters), and the GNSS position accuracy in single-point positioning mode is at the meter level, so the GNSS antenna and the IMU measurement center can be considered to overlap. Besides, since the standard deviation cannot accurately determine the true positioning accuracy of the GNSS position, the chi-square test is used to eliminate the gross errors in the GNSS position to ensure the reliability of the filtering [66]. b) Vehicle motion model constraints: To deal with scenarios where GNSS signals are interfered in a complex environment, the vehicle motion constraint model is fully used to improve the relative positioning capability of the system. WHU&Autonavi ystem simply divides the vehicle motion state into stationary and moving by using the raw output of gyroscope and accelerometer. Stationary state: When the vehicle is judged to be stationary, it can be considered that the speed of the vehicle is zero, that is, Zero-velocity update (ZUPT). ZUPT is an effective means to control the accumulation of velocity error. At the same time, the heading of the vehicle should remain unchanged, and all heading errors can be considered to be caused by sensor errors. The WHU&Autonavi system stores the heading angle at the initial moment of the stationary period and constructs a virtual heading angle observation value, so as to achieve the purpose of effectively controlling the accumulation of heading angle error, called Zero Integrated Heading Rate (ZIHR) [67]. Motion state: For the normal driving behavior of ordinary users, the vehicle will only move forward or backward. Based on such objective facts, it can be assumed that the lateral and vertical speeds in the vehicle coordinate system (that is, the v system) are always zero [67]. However, the forward speed of the vehicle still cannot be accurately obtained. WHU&Autonavi Team system uses rticl supervised learning method to train the vehicle forward speed prediction model [68], and the error can be controlled within 0.5 ms−1. Due to the random disassembly and reinstallation of the smartphone, the problem of the installation angle and lever arm parameters is not fixed. At this time, traditional direct setting or pre-calibration methods do not have the conditions for implementation. Automatic calibration of the installation angle and lever arm parameters can make the vehicle motion constraint algorithm more applicable. c) Magnetic heading constraints: The magnetic interference caused by the vehicle shell can be equivalent to the magnetometer bias. So, the heading angle calculated based on the magnetometer observations can still accurately reflect the true heading angle change after the calibration and deduction of the magnetometer bias. Besides, the quasi-Static Magnetic Field (QSMF) is employed for avoiding environmental magnetic interference [69]. 2) SZU-MellivoraCapensis: The data collection of Track 6 is located near the Beijing Haidian Airport. Its goal is to evaluate POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5035 the performance of vehicle navigation solutions based on the integration of different sensors such as GNSS, MEMS, and magnetometers on in-vehicle smartphones. This test is under typical urban road conditions. The smartphone is fixed inside the vehicle, and data is collected through the phone sensor. A single test process lasts about 1 hour and the test route consists of static initial alignment phase (about 5 minutes), open environment phase (about 20 minutes), obstructed environment phase where the GNSS signal is attenuated or blocked by the surrounding buildings or trees (about 25 minutes, during which the GNSS positioning results will be frequently interrupted) and no GNSS signal phase (underground parking lots about 10 minutes, with no GNSS positioning results). The driving process of the test vehicle includes going straight, left/right turning, reversing and parking. To get the update of the vehicle’s position, SZU-Mellivora Capensis team system obtain its velocity and heading. As for the velocity update, the system uses accelerometer and gyroscope, extract their data and align the coordinates, and then train them through proposed Deep Neural Network (DNN) to get the predicted velocity. The same is true for the heading prediction, but raw data used comes from the gyroscope, the magnetometer and the AHRS. Based on the prediction of velocity and heading, the relative displacement of the vehicle can be inferred. Then, the federated filter is used for data fusion. The weight factor is modified through observability to improve the filter and achieve high-precision localization. Finally, a smoothing filter is applied in this method. The traditional inertial dead-reckoning mentioned above to estimate the motion of the vehicle is a challenging problem. To reduce this unavoidable inertial drift, a data-driven approach is used to inertial tracking. Referring to the network structure on IONet, the motion state of the vehicle is predicted by a trained deep Recurrent Neural Network (RNN). The RNN maintains the local hidden state within a time window, and then extracts the potential features of the time series. These features affect the state output at the next moment, thus enabling an effective recovery of the potential connection between data features and vehicle motion. The time window size is chosen as 1 s (50 frames). The data within the window are (n×3×50)dimensional long-term dependent feature vectors constructed by stacking aligned nsensors. The changes of _v and _h in 1 s can be predicted by Equation 12: (v, h)=RNN((ai,w i,mi,gi)T t)(12) Unlike previous data-driven-inertial tracking work, the regression of the displacement vector is split into two separate parts: velocity estimation and heading estimation. The division of the regression task reduces the impact of extraneous sensors on prediction accuracy. In the velocity estimation part, input data are the 3-axis accelerometer and 3-axis gravity sensor data for a one-second period, which are corrected for the coordinate system alignment described above. The output is the average velocity over this time period, based in the two-dimensional plane. In the heading estimation section, input data are the 3-axis gyroscope and 3-axis magnetometer data during the time period, and the output is the sum of the heading changes in one second. The Fig. 35. The RNN framework of the proposed method by ZU-Mellivora Capensis team. above input data is the best combination of sensors after the experiments performed. Fig. 35 shows the RNN framework proposed in this system. A two-layer Long Short-Term Memory (LSTM) is used as the core module to solve the gradient explosion and vanishing problem of traditional RNNs, and it can effectively exploit the long-term dependence of time series. Each LSTM layer has 256 hidden nodes well above the dimensionality of the input data. This is in order to give enough inputs to the LSTM units so that the LSTM can fully utilize its function of selecting useful information. To avoid the overfitting problem, a dropout layer is added after each LSTM layer to increase the orthogonality between the features in each layer. Finally, a fully connected layer is placed to regress the velocity and heading changes, respectively. The loss function is defined in terms of the mean square error between the motion parameters and the ground truth. The ADAM optimizer is chosen to minimize this loss value and learn to obtain the best parameters within the RNN. After obtaining the velocity and heading, the trajectory points can be expressed as: x=x0+vdt ·cos(h0+h) y=y0+vdt ·sin(h0+h)(13) 3) Team YAI: YAI Team system uses three types of sensor data in this competition, namely ACCE,AHRS,GNSS.Inthe data pre-processing part, the ACCE and AHRS data were averaged per second to obtain data with a frequency of 1 Hz, while the missing GNSS were marked. First, the displacement of the vehicle per second is obtained by adding the initial velocity of the original GNSS to the ACCE data. Then, theYAWangledataofAHRS is initialized. After setting the initial direction angle, the angle ranges from minus 180 to 180 degrees. The proposed framework used Kalman Filter (KF) for tracking. Fig. 36 shows the flow chart of the proposed tracking framework. The prepossessed data was introduced into the Kalman filter and GNSS to get KF gain to correct the error. Because the KF relied on the previous path to calculate, it does not work well during the missing section and may 5042 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Technology through the Center for Analytics-DataApplications (ADA-Center) within the framework of “BAYERN DIGITAL II.” Team UMinho (Track 3) was supported by FCT—Fundação para a Ciência e Tecnologia within the R&D Units Project Scope under Grant UIDB/00319/2020, and the Ph.D. Fellowship under Grant PD/BD/137401/2018. Team YAI (Track 3) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 109-2221-E-197-026. Team Indora (Track 3) was supported in part by the Slovak Grant Agency, Ministry of Education and Academy of Science, Slovakia, under Grant 1/0177/21, and in part by the Slovak Research and Development Agency under Contract APVV15-0091. Team TJU (Track 3) was supported in part by the National Natural Science Foundation of China under Grant 61771338 and in part by the Tianjin Research Funding under Grant 18ZXRHSY00190. Team Next-Newbie Reckoners (Track 3) were supported by the Singapore Government through the Industry Alignment Fund—Industry Collaboration Projects Grant. This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is a collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU). Team KawaguchiLab (Track 5) was supported by JSPS KAKENHI under Grant JP17H01762. Team WHU&AutoNavi (Track 6) was supported by the National Key Research and Development Program of China under Grant 2016YFB0502202. Team YAI (Tracks 6 and 7) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 110-2634-F-155-001. Francesco Potortì, Antonino Crivello, Paolo Barsocchi, Michele Girolami, and Filippo Palumbo are with the Information Science and Technologies Institute, National Research Council, 56124 Pisa, Italy (e-mail: antonino.criv[email protected].it). Joaquín Torres-Sospedra is with UBIK Geospatial Solutions S.L., 12006 Castellón, Spain (e-mail: [email protected]). Darwin Quezada-Gaibor is with the Institute of New Imaging Technologies, Unviersitat Jaume I, 12006 Castellón, Spain, and also with the Electrical Engineering Unit, Tampere University, 33100 Tampere, Finland. Antonio Ramón Jiménez and Fernando Seco are with the Centre for Automation and Robotics (CSIC-UPM), 28500 Arganda del Rey, Spain. Antoni Pérez-Navarro is with the Faculty of Computer Sciences, Multimedia and Telecommunication, Universitat Oberta de Catalunya, 08860 Barcelona, Spain, and also with the Internet Interdisciplinary Institute (IN3), Universitat Oberta deCatalunya, 08860Barcelona, Spain. Miguel Ortiz, Ni Zhu, and Valerie Renaudin are with the AMEGEOLOC, Université Gustave Eiffel, IFSTTAR, 44344 Bouguenais, France. Ryosuke Ichikari is with the National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, Japan. Ryo Shimomura and Nozomu Ohta are with the National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 3058560, Japan, and also with the Department of Intelligent Interaction Technologies, University of Tsukuba, Tsukuba 305-8573, Japan. Takeshi Kurata is with the National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8560, Japan, and also with the Faculty of Engineering, Information and Systems, University of Tsukuba, Tsukuba 305-8573, Japan. Satsuki Nagae is with the Department of Intelligent Interaction Technologies, University of Tsukuba, Tsukuba 305-8573, Japan. Dongyan Wei, Xinchun Ji, and Wenchao Zhang are with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China. Sebastian Kram, Maximilian Stahlke, and Christopher Mutschler are with the Fraunhofer Institute for Integrated Circuits IIS, 90411 Nürnberg, Germany. Ruizhi Chen, Yuan Wu, Wei Li, Yue Yu, Shihao Xu, and Lixiong Huang are with the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University, Wuhan 430072, China. Tao Liu, Jian Kuang, and Wenfei Ge are with the GNSS Research Center, Wuhan University, Wuhan 430072, China. Xiaoji Niu is with the GNSS Research Center, Wuhan University, Wuhan 430072, China, and also with the Collaborative Innovation Center of Geospatial Technology, Wuhan University, Wuhan 430072, China. Takuto Yoshida, Yoshiteru Nagata, Yuto Fukushima, Nobuya Fukatani, Nozomi Hayashida, Yusuke Asai, and Kenta Urano are with the Department of Information and Communication Engineering, Graduate School of Engineering, Nagoya University, Nagoya 464-8601, Japan. Nien-Ting Lee, Shih-Hau Fang, You-Cheng Jie, Shawn-Rong Young, Chia-An Yu, and Chun-Hao Hung are with the Department of Electrical Engineering, Yuan Ze University, Taoyuan 32003, Taiwan, and also with the MOST Joint Research Center for AI Technology and All Vista Healthcare, Taipei 10617, Taiwan. Ying-Ren Chien and Chih-Chieh Yu are with the Department of Electrical Engineering, National Ilan University, Yilan 26047, Taiwan. Chengqi Ma and David R. Selviah are with the Electronic and Electrical Engineering Department, University College London, London WC1E 7JE, U.K. Bang Wu, Yonglei Fan, and Stefan Poslad are with the Electronic Engineering and Computer Science Department, Queen Mary University of London, London E1 4NS, U.K. Wei Zhang, Yankun Wang, and Weixi Wang are with the Department of Research Institute for Smart Cities, Shenzhen University, Shenzhen 518060, China. Hong Yuan is with the Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China. Yoshitomo Yonamoto and Tomoya Kaichi are with the Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan. Masahiro Yamaguchi is with the Nippon Electric Company Ltd., (NEC), Kawasaki 211-8666, Japan. Baoding Zhou and Xu Liu are with College of Civil and Transportation Engineering, Shenzhen University, Shenzhen 518060, China. Zhining Gu is with the School of Architecture and Urban Planning, Shenzhen University, Shenzhen 518060, China. Chengjing Yang, ZhiqianWu, Doudou Xie, and Can Huang are with the College of Civil and Transportation Engineering, Shenzhen University, Shenzhen 518060, China. Lingxiang Zheng, Ao Peng, and Ge Jin are with the Department of Informatics and Communication Engineering,Xiamen University, Xiamen 361005, China. Qu Wang is with the School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China. Haiyong Luo and Linfeng Bao are with the Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China. Hao Xiong, Pushuo Zhang, and Fang Zhao are with the School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876, China. Leonid Antsfeld and Boris Chidlovskii are with Naver Labs Europe, 38240 Meylan, France. Haitao Jiang, Ming Xia, Dayu Yan, Yuhang Li, and Yitong Dong are with the School of Electronic and Information Engineering, Beihang University, Beijing 100083, China. Ivo Silva, Cristiano Pendão, Filipe Meneses, Maria João Nicolau, António Costa, and Adriano Moreira are with the ALGORITMI Research Center, University of Minho, 4800-058 Guimarães, Portugal. Cedric De Cock and David Plets are with the imec-WAVES Group, Department of Information Technology, Ghent University, 9052 Ghent, Belgium. Miroslav Opiela and Jakub Džama are with the Faculty of Science, Institute of Computer Science, Pavol Jozef Šafárik University in Košice, 04001 Košice, Slovakia. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5043 Liqiang Zhang, Hu Li, Boxuan Chen, and Yu Liu are with the School of Microelectronics, Tianjin University, Tianjin 300072, China. Seanglidet Yean, Bo Zhi Lim, Wei Jie Teo, Bu Sung Lee, and Hong Lye Oh are with the School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798. REFERENCES [1] V. Renaudin et al., “Evaluating indoor positioning systems in a shopping mall: The lessons learned from the IPIN 2018 competition,” IEEE Access, vol. 7, pp. 148594–148628, 2019. [2] F. Potortì et al., “The IPIN 2019 indoor localisation competition— Description and results,” IEEE Access, vol. 8, pp. 206674–206718, 2020. [3] F. Potortì et al., “Comparing the performance of indoor localization systems through the EvAAL framework,” Sensors, vol. 17, no. 10, p. 2327, Oct. 2017. [4] P. Barsocchi, S. Chessa, F. Furfari, and F. Potortì, “Evaluating ambient assisted living solutions: The localization competition,” IEEE Pervasive Comput., vol. 12, no. 4, pp. 72–79, Oct./Dec. 2013. [5] F. Potortì, A. Crivello, and F. Palumbo, “The EvAAL evaluation framework and the IPIN competitions,” in Geographical and Fingerprinting Data to Create Systems for Indoor Positioning and Indoor/Outdoor Navigation. Amsterdam, The Netherlands: Elsevier, 2019, pp. 209–224. [6] Information Technology—Real Time Locating Systems–Test and Evaluation of Localization and Tracking Systems, Standard ISO/IEC 18305:2016, 2016. [7] F. Potortì, A. Crivello, P. Barsocchi, and F. Palumbo, “Evaluation of indoor localisation systems: Comments on the ISO/IEC 18305 standard,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN), Sep. 2018, pp. 1–7. [8]G.M.Mendoza-Silvaet al., “Beyond Euclidean distance for error measurement in pedestrian indoor location,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–11, 2021. [Online]. Available: https://ieeexplore. ieee.org/document/9186638 [9] F. Furfari et al., “Discovering location based services: A unified approach for heterogeneous indoor localization systems,” Internet Things, vol. 13, Mar. 2021, Art. no. 100334. [10] J. Torres-Sospedra et al., “The smartphone-based offline indoor location competition at IPIN 2016: Analysis and future work,” Sensors, vol. 17, no. 3, pp. 1–17, 2017. [11] J. Torres-Sospedra et al., “Off-line evaluation of mobile-centric indoor positioning systems: The experiences from the 2017 IPIN competition,” Sensors, vol. 18, no. 2, p. 487, Feb. 2018. [12] A. R. Jiménez, F. Seco, and J. Torres-Sospedra, “Tools for smartphone multi-sensor data registration and GT mapping for positioning applications,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN), Sep. 2019, pp. 1–8. [13] J. Torres-Sospedra, D. Quezada-Gaibor, A. R. Jiménez, F. Seco, and A. Perez-Navarro, “Datasets and supporting materials for the IPIN 2020 competition track 3 (smartphone-based, off-site),” Zenodo, 2020, doi: 10.5281/zenodo.4314992. [14] A. R. Jiménez et al., “Datasets and supporting materials for the IPIN 2019 competition track 3 (smartphone-based, off-site),” Zenodo, 2019, doi: 10.5281/zenodo.3606765. [15] A. R. Jiménez et al., “Datasets and supporting materials for the IPIN 2018 competition track 3 (smartphone-based, off-site),” Zenodo, 2018, doi: 10.5281/zenodo.2823964. [16] A. R. Jimenez, G. M. Mendoza-Silva, F. Seco, and J. Torres-Sospedra, “Datasets and supporting materials for the IPIN 2017 competition track 3 (smartphone-based, off-site),” Zenodo, 2017, doi: 10.5281/zenodo.2823924. [17] A. R. Jimenez, G. M. Mendoza-Silva, R. Montoliu, F. Seco, and J. Torres-Sospedra, “Datasets and supporting materials for the IPIN 2016 competition track 3 (smartphone-based, off-site),” Zenodo, 2016, doi: 10.5281/zenodo.2791530. [18] H.-J. Jang, J. Kim, and D.-H. Hwang, “Robust step detection method for pedestrian navigation systems,” Electron. Lett., vol. 43, no. 14, pp. 749–751, 2007. [Online]. Available: https://digitallibrary.theiet.org/content/journals/10.1049/el_20070478 [19] H. Weinberg, “Using the ADXL202 in pedometer and personal navigation applications,” Analog Devices, Norwood, MA, USA, Appl. Note AN-602, Feb. 2002, vol. 2. [20] E. R. Bachmann, R. B. McGhee, X. Yun, and M. J. Zyda, “Inertial and magnetic posture tracking for inserting humans into networked virtual environments,” in Proc. ACM Symp. Virtual Reality Softw. Technol. (VRST). New York, NY, USA: Association for Computing Machinery, 2001, pp. 9–16, doi: 10.1145/505008.505011. [21] L. Zheng, Z. Wu, W. Zhou, S. Weng, and H. Zheng, “A smartphone based hand-held indoor positioning system,” in Frontier Computing, J. C. Hung, N. Y. Yen, and K.-C. Li, Eds. Singapore: Springer, 2016, pp. 639–650. [22] B. Chidlovskii and L. Antsfeld, “Semi-supervised variational autoencoder for WiFi indoor localization,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN), Sep. 2019, pp. 1–8. [23] N. Lee, S. Ahn, and D. Han, “AMID: Accurate magnetic indoor localization using deep learning,” Sensors, vol. 18, no. 5, p. 1598, May 2018. [24] L. Antsfeld, B. Chidlovskii, and D. Borisov, “Magnetic sensor based indoor positioning by multi-channel deep regression,” in Proc. Conf. Embedded Netw. Sensor Syst. (SenSys), Nov. 2020, pp. 707–708. [25] I. Silva, A. Moreira, M. J. Nicolau, and C. Pendão, “Floor planfree particle filter for indoor positioning of industrial vehicles,” in Proc. Int. Conf. Localization (ICL-GNSS), Tampere, Finland, 2020, pp. 14–30. [Online]. Available: http://ceur-ws.org/Vol-2626/paper2.pdf [26] W. Kang and Y. Han, “SmartPDR: Smartphone-based pedestrian dead reckoning for indoor localization,” IEEE Sensors J., vol. 15, no. 5, pp. 2906–2916, May 2015. [27] N.-H. Ho, P. Truong, and G.-M. Jeong, “Step-detection and adaptive step-length estimation for pedestrian dead-reckoning at various walking speeds using a smartphone,” Sensors, vol. 16, no. 9, p. 1423, Sep. 2016. [28] B. Li, B. Harvey, and T. Gallagher, “Using barometers to determine the height for indoor positioning,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat., Oct. 2013, pp. 1–7. [29] J. Trogh, D. Plets, L. Martens, and W. Joseph, “Advanced real-time indoor tracking based on the Viterbi algorithm and semantic data,” Int. J. Distrib. Sensor Netw., vol. 2015, pp. 1–11, Oct. 2015. [30] WiCa Heuristic Indoor Propagation Prediction Tool (WHIPP). Accessed: Jan. 1, 2021. [Online]. Available: https://www.waves.intec. ugent.be/exposure-tool/expert-edition [31] D. Plets, W. Joseph, K. Vanhecke, E. Tanghe, and L. Martens, “Coverage prediction and optimization algorithms for indoor environments,” EURASIP J. Wireless Commun. Netw., vol. 2012, no. 1, pp. 1–23, Dec. 2012. [32] Widyawan, M. Klepal, and S. Beauregard, “A novel backtracking particle filter for pattern matching indoor localization,” in Proc. 1st ACM Int. Workshop Mobile Entity Localization Tracking GPS-Less Environ., 2008, pp. 79–84. [33] M. Opiela and F. Galˇcík, “Grid-based Bayesian filtering methods for pedestrian dead reckoning indoor positioning using smartphones,” Sensors, vol. 20, no. 18, p. 5343, Sep. 2020. [34] W. H. K. de Vries, H. E. J. Veeger, C. T. M. Baten, and F. C. T. van der Helm, “Magnetic distortion in motion labs, implications for validating inertial magnetic sensors,” Gait Posture, vol. 29, no. 4, pp. 535–541, Jun. 2009. [35] L. Zhang, Y. Liu, and J. Sun, “A hybrid framework for mitigating heading drift for a wearable pedestrian navigation system through adaptive fusion of inertial and magnetic measurements,” Appl. Sci., vol. 11, no. 4, p. 1902, Feb. 2021. [Online]. Available: https://www.mdpi.com/20763417/11/4/1902 [36] S. O. Madgwick, A. J. Harrison, and R. Vaidyanathan, “Estimation of IMU and MARG orientation using a gradient descent algorithm,” in Proc. IEEE Int. Conf. Rehabil. Robot., Jun./Jul. 2011, pp. 1–7. [37] R. S. Sinha, S.-M. Lee, M. Rim, and S.-H. Hwang, “Data augmentation schemes for deep learning in an indoor positioning application,” Electronics, vol. 8, no. 5, p. 554, May 2019. [38] M. Ortiz, A. Ramesh, N. Zhu, and V. Renaudin, “Datasets and supporting materials for the IPIN 2020 competition track 4 (foot-mounted IMU based positioning, off-site),” Zenodo, 2021, doi: 10.5281/zenodo.4668618. [39] M. Ortiz, J. Perul, J. Torres-Sospedra, and V. Renaudin, “Datasets and supporting materials for the IPIN 2019 competition track 4 (foot-mounted IMU based positioning, off-site),” Zenodo, 2020, doi: 10.5281/zenodo.3937220. [40] M. Ortiz, J. Perul, J. Torres-Sospedra, and V. Renaudin, “Datasets and supporting materials for the IPIN 2018 competition track 4 (foot-mounted IMU based positioning, off-site),” Zenodo, 2019, doi: 10.5281/zenodo.3228012. [41] E. Foxlin, “Pedestrian tracking with shoe-mounted inertial sensors,” IEEE Comput. Graph. Appl., vol. 25, no. 6, pp. 38–46, Nov. 2005. [42] S. Godha and G. Lachapelle, “Foot mounted inertial system for pedestrian navigation,” Meas. Sci. Technol., vol. 19, no. 7, Jul. 2008, Art. no. 075202. [43] I. Skog, P. Handel, J. O. Nilsson, and J. Rantakokko, “Zero-velocity detection—An algorithm evaluation,” IEEE Trans. Biomed. Eng., vol. 57, no. 11, pp. 2657–2666, Nov. 2010. 5044 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 [44] Z. Wang, H. Zhao, S. Qiu, and Q. Gao, “Stance-phase detection for ZUPT-aided foot-mounted pedestrian navigation system,” IEEE/ASME Trans. Mechatronics, vol. 20, no. 6, pp. 3170–3181, Dec. 2015. [45] A. R. Jiménez, F. Seco, J. C. Prieto, and J. Guevara, “Indoor pedestrian navigation using an INS/EKF framework for yaw drift reduction and a foot-mounted IMU,” in Proc. 7th Workshop Positioning, Navigat. Commun., Mar. 2010, pp. 135–143. [46] J. Borenstein and L. Ojeda, “Heuristic drift elimination for personnel tracking systems,” J. Navigat., vol. 63, no. 4, p. 591, 2010. [47] A. R. Jiménez, F. Seco, F. Zampella, J. C. Prieto, and J. Guevara, “Improved heuristic drift elimination (iHDE) for pedestrian navigation in complex buildings,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat., Sep. 2011, pp. 1–8. [48] W. Zhang, D. Wei, and H. Yuan, “The improved constraint methods for foot-mounted PDR system,” IEEE Access, vol. 8, pp. 31764–31779, 2020. [49] W. Zhang, X. Li, D. Wei, X. Ji, and H. Yuan, “A foot-mounted PDR system based on IMU/EKF+HMM+ZUPT+ZARU+HDR+compass algorithm,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN), Sep. 2017, pp. 1–5. [50] W. Zhang, D. Wei, and H. Yuan, “Novel drift reduction methods in footmounted PDR system,” Sensors, vol. 19, no. 18, p. 3962, Sep. 2019. [51] W. Zhang, D. Wei, P. Gong, and H. Yuan, “The PDR system based on improved QSF+map matching algorithm,” in Proc. China Satell. Navigat. Conf. Singapore: Springer, 2018, pp. 753–764. [52] Q. Wang et al., “Personalized stride-length estimation based on active online learning,” IEEE Internet Things J., vol. 7, no. 6, pp. 4885–4897, Jun. 2020. [53] Q. Wang et al., “Pedestrian dead reckoning based on walking pattern recognition and online magnetic fingerprint trajectory calibration,” IEEE Internet Things J., vol. 8, no. 3, pp. 2011–2026, Feb. 2021. [54] X. Gao, H. Luo, Q. Wang, F. Zhao, L. Ye, and Y. Zhang, “A human activity recognition algorithm based on stacking denoising autoencoder and LightGBM,” Sensors, vol. 19, no. 4, p. 947, Feb. 2019. [55] C. Yu, L. Haiyong, Z. Fang, W. Qu, and S. Wenhua, “Adaptive Kalman filtering-based pedestrian navigation algorithm for smartphones,” Int. J. Adv. Robot. Syst., vol. 17, no. 3, May 2020, Art. no. 172988142093093. [56] R. Ichikari, K. Kaji, R. Shimomura, M. Kourogi, T. Okuma, and T. Kurata, “Off-site indoor localization competitions based on measured data in a warehouse,” Sensors, vol. 19, no. 4, p. 763, Feb. 2019, doi: 10.3390/s19040763. [57] T. Kurata et al., “IoH technologies into indoor manufacturing sites,” in Proc. Int. Conf. Adv. Prod. Manage. Syst., vol. 567. Cham, Switzerland: Springer, 2019, pp. 372–380, doi: 10.1007/978-3-030-29996-5_43. [58] xDR-Challenge-2020-Evaluation. Accessed: Mar. 28, 2021. [Online]. Available: https://github.com/PDR-benchmark-standardizationcommittee/xDR-Challenge-2020-evaluation [59] T. Yoshida, J. Nozaki, K. Urano, K. Hiroi, T. Yonezawa, and N. Kawaguchi, “Gait dependency of smartphone walking speed estimation using deep learning (poster),” in Proc. 17th Annu. Int. Conf. Mobile Syst., Appl., Services (MobiSys). New York, NY, USA: Association for Computing Machinery, Jun. 2019, pp. 641–642, doi: 10.1145/3307334.3328667. [60] T. Yoshida et al., “Sampling rate dependency in pedestrian walking speed estimation using dualCNN-LSTM,” in Proc. UbiComp/ISWC. New York, NY, USA: Association for Computing Machinery, Sep. 2019, pp. 862–868, doi: 10.1145/3341162.3343765. [61] N. Kawaguchi, J. Nozaki, T. Yoshida, K. Hiroi, T. Yonezawa, and K. Kaji, “End-to-end walking speed estimation method for smartphone PDR using dualCNN-LSTM,” in Proc. CEUR Workshop, vol. 2498, F. Potortì, V. Renaudin, K. O’Keefe, and F. Palumbo, Eds., 2019, pp. 463–470. [Online]. Available: http://ceurws.org/Vol-2498/short60.pdf [62] A. Manos, I. Klein, and T. Hazan, “Gravity direction estimation and heading determination for pedestrian navigation,” in Proc. Int. Conf. Indoor Positioning Indoor Navigat. (IPIN), Sep. 2018, pp. 206–212. [63] J. Nozaki, K. Hiroi, K. Kaji, and N. Kawaguchi, “Compensation scheme for PDR using sparse location and error model,” in Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput., ACM Int. Symp. Wearable Comput. New York, NY, USA: Association for Computing Machinery, Sep. 2017, pp. 587–596, doi: 10.1145/3123024.3124406. [64] J. Kuang, X. Niu, and X. Chen, “Robust pedestrian dead reckoning based on MEMS-IMU for smartphones,” Sensors, vol. 18, no. 5, p. 1391, May 2018. [65] Q. Zhang, X. Niu, H. Zhang, and C. Shi, “Algorithm improvement of the low-end GNSS/INS systems for land vehicles navigation,” Math. Problem Eng, vol. 2013, Jun. 2013, Art. no. 435286. [66] B. Li et al., “Robust Kalman filtering based on chi-square increment and its application,” Remote Sens., vol. 12, no. 4, p. 732, Feb. 2020. [67] E. H. Shin, “Estimation techniques for low-cost inertial navigation,” Univ. Calgary, Calgary, AB, Canada, UCGE Rep. 20219, 2005. [68] M. A. Esfahani, H. Wang, K. Wu, and S. Yuan, “AbolDeepIO: A novel deep inertial odometry network for autonomous vehicles,” IEEE Trans. Intell. Transp. Syst., vol. 21, no. 5, pp. 1941–1950, May 2020. [69] Y. Li, “Integration of MEMS sensors, WiFi, and magnetic features for indoor pedestrian navigation with consumer portable devices,” Ph.D. dissertation, Univ. Calgary, Calgary, AB, Canada, 2016, doi: 10.11575/PRISM/26584. [70] M. Stahlke, S. Kram, C. Mutschler, and T. Mahr, “NLOS detection using UWB channel impulse responses and convolutional neural networks,” in Proc. Int. Conf. Localization GNSS (ICL-GNSS), Jun. 2020, pp. 1–6. [71] K. Bregar and M. Mohorcic, “Improving indoor localization using convolutional neural networks on computationally restricted devices,” IEEE Access, vol. 6, pp. 17429–17441, 2018. [72] A. Niitsoo, T. Edelhäußer, E. Eberlein, N. Hadaschik, and C. Mutschler, “A deep learning approach to position estimation from channel impulse responses,” Sensors, vol. 19, no. 5, p. 1064, Mar. 2019. [73] S. Kram, M. Stahlke, T. Feigl, J. Seitz, and J. Thielecke, “UWB channel impulse responses for positioning in complex environments: A detailed feature analysis,” Sensors, vol. 19, no. 24, p. 5547, Dec. 2019. [74] E. Leitinger, F. Meyer, F. Hlawatsch, K. Witrisal, F. Tufvesson, and M. Z. Win, “A belief propagation algorithm for multipathbased SLAM,” IEEE Trans. Wireless Commun., vol. 18, no. 12, pp. 5613–5629, Dec. 2019. [75] M. Ulmschneider, C. Gentner, T. Jost, and A. Dammann, “Raoblackwellized Gaussian sum particle filtering for multipath assisted positioning,” J. Electr. Comput. Eng., vol. 2018, pp. 1–15, Apr. 2018. [76] R. Karásek and C. Gentner, “Stochastic data association for multipath assisted positioning using a single transmitter,” IEEE Access,vol.8, pp. 46735–46752, 2020. [77] E. Leitinger, P. Meissner, C. Rüdisser, G. Dumphart, and K. Witrisal, “Evaluation of position-related information in multipath components for indoor positioning,” IEEE J. Sel. Areas Commun., vol. 33, no. 11, pp. 2313–2328, Nov. 2015. [78] S.-H. Fang, W.-H. Chang, Y. Tsao, H.-C. Shih, and C. Wang, “Channel state reconstruction using multilevel discrete wavelet transform for improved fingerprinting-based indoor localization,” IEEE Sensors J., vol. 16, no. 21, pp. 7784–7791, Nov. 2016. [79] S.-H. Fang, C.-C. Li, W.-C. Lu, Z. Xu, and Y.-R. Chien, “Enhanced device-free human detection: Efficient learning from phase and amplitude of channel state information,” IEEE Trans. Veh. Technol., vol. 68, no. 3, pp. 3048–3051, Mar. 2019. Francesco Potortì (Member, IEEE) has been working in satellite and terrestrial communications with the Information Science and Technologies Institute, National Research Council, Pisa, Italy, since 1989, where he isa Senior Researcher. He hasorganizedthe 2011–2013 EvAAL competitions;defined the EvAAL framework; organizedthe IPIN competitions from 2014 to 2017; and chaired the tenth edition of the IPIN conference and the sixth edition of the IPIN Competition in 2019. He has coauthored more than 80 peer-reviewed scientific articles. His current research interests include RSS-based indoor localization, interoperability, and evaluation of indoor localization systems. He is a member of the IPIN Steering Board. Joaquín Torres-Sospedra was born in Castelló, Spain, in 1979. He received the Ph.D. degree in ensembles of neural networks and machine learning from the Universitat Jaume I in 2011. He joined the Institute of New Imaging Technologies (INIT), Universitat Jaume I, in April 2013, where he led indoor positioning projects. Since January 2020, he has been the Scientific Coordinator of UBIK Geospatial Solutions and still collaborates with the INIT, as well as other international research institutions. He has supervised five master’s and two Ph.D. students. He is supervising six Ph.D. students. He has authored more than 120 articles in journals and conferences. His current research interests include indoor positioning solutions based on Wi-Fi and BLE, machine learning, and evaluation. He is the Chair of the IPIN International Standards Committee and the IPIN Smartphone-Based Off-Site Competition. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5045 Darwin Quezada-Gaibor received the bachelor’s degree in mechatronic engineering from the Universidad Tecnológica América, Ecuador, in 2013, and the master’s degree in radio engineering—GNSS receivers: hardware and software from Samara National Research University, Russia, in 2017. He is currently pursuing the Ph.D. degree with the Universitat Jaume I, Spain, and Tampere University, Finland. He is an Early Stage Researcher (ESR) with the Universitat Jaume I and Tampere University. His main research interests include VoIP, cloud computing, networking, servers, and open-source software. Antonio Ramón Jiménez was born in Santander, Spain, in 1968. He received the degree in physics and computer science and the Ph.D. degree in physics from the Universidad Complutense de Madrid, Madrid, Spain, in 1991 and 1998, respectively. Since 1993, he has been with the Centre for Automation and Robotics, Spanish Council for Scientific Research, Madrid, where he holds a research position. He has authored more than 100 articles in journals and conference proceedings. His current research interests include local positioning solutions for indoor/ GPS-denied localization and navigation of persons and robots, signal processing, Bayesian estimation, and inertial-ultrasonic-RFID sensor fusion. He is a reviewer for many international journals and projects in the field. Fernando Seco was born in Madrid, Spain, in 1972. He received the degree in physics from the Universidad Complutense of Madrid, Madrid, in 1996, and the Ph.D. degree in physics from the Universidad Nacional de Educación a Distancia, Madrid, in 2002, with a dissertation about the magnetostrictive generation of ultrasonic waves applied to a linear position sensor. Since 1997, he has been with the Center for Automation and Robotics, Spanish Council for Scientific Research, Arganda del Rey, Madrid, where he holds a research position. His current research interests include the design and development of indoor local positioning systems, especially those based on ultrasonic and radio-frequency technologies in signal processing for CDMA-based localization systems and Bayesian estimation. Antoni Pérez-Navarro (Member, IEEE) received the bachelor’s and Ph.D. degrees in physics from the Universitat Autònoma de Barcelona (UAB), in 1995 and 2000, respectively. He is currently the Director of the Technological Observatory, EIMT department, the Director of the technical collection of books ofeditorial Ediuoc, the Deputy Director of Research with the eLearn Center, Universitat Oberta de Catalunya (UOC), and a Lecturer with the Computer Science, Multimedia and Telecommunication Department (EIMT department). He also works with the Escola Universitària Salesiana de Sarrià (EUSS) where he gives classes in industrial engineering grades. He has published several articles in international journals on all these topics. His main research interests include indoor positioning and the prevention of diseases via smartphones. He is a part of the Technical Program Committee of the IPIN. He is one of the organizers of the symposium “Challenges of Fingerprinting in Indoor Positioning and Navigation,” Barcelona, in 2015 and the main Chair of the IPIN Conference in 2021. He acts as a reviewer of several journals. Miguel Ortiz received the M.Sc. degree in mechanics, automation, and engineering from the Ecole Nationale Supérieure d’Arts et Métier in 2001. He joined the lab after spending six years in a company where he managed systems architecture for automotive applications. He has 12 years of experience in the GNSS domain as a research engineer. Since 2019, he has been the Head Deputy of the GEOLOC Laboratory, Université Gustave Eiffel, Nantes Campus. He is a Research Engineer with the GEOLOC Laboratory, Université Gustave Eiffel (ex-IFSTTAR). As an Expert in embedded electronic systems, his research interests focus on software and hardware developments for both intelligent transport systems (ITS) and pedestrian navigation research field. Since 2017, he has been the Convenor of the CEN/CENELEC TC5-WG1 named “Navigation and Positioning Receivers for Road Applications.” Ni Zhu received the degree in aeronautic telecommunications engineering from the Ecole Nationale de l’Aviation Civile (ENAC) in 2015, and the Ph.D. degree in science of information and communication from the University of Lille in 2018. She is a Research Fellow with the GEOLOC Laboratory, Université Gustave Eiffel (ex-IFSTTAR). Her research interests include GNSS channel propagation modeling in urban environments, integrity monitoring for terrestrial applications, and multi-sensory fusion techniques for indoor/outdoor pedestrian positioning. Valerie Renaudin (Member, IEEE) received the M.Sc. degree in geomatics engineering from ESGT in 1999, and the Ph.D. degree in computer, communication, and information sciences from the EPFL in 2009. She was a Technical Director with Swissat Company, Samstagern, Switzerland, developing real-time geopositioning solutions based on a permanent global navigation satellite system (GNSS) network, and a Senior Research Associate with the PLAN Group, University of Calgary, Canada. She is a Research Director (eq. Full Professor) with Université Gustave Eiffel (ex-IFSTTAR), France, where she is also leading the Geopositioning Laboratory (GEOLOC), where she built a team specializing in positioning and navigation for travelers in multimodal transport. Her research interests include outdoor/indoor navigation using GNSS, and inertial and magnetic data, particularly for pedestrians to improve sustainable personal mobility. Since 2013, she has been a member of the IEEE Society. She is the Steering Committee of the International Conference on Indoor Positioning and Indoor Navigation. She was a recipient of the European Marie Curie Career Integration Grant for her project smartWALK. Ryosuke Ichikari received the Ph.D. degree in engineering from Ritsumeikan University in 2010. He is a Senior Researcher with the Human Augmentation Research Center, National Institute of Advanced Industrial Science and Technology (AIST), Japan. His research interests include indoor localization, virtual/mixed reality, and assistive technology for people with disabilities. Ryo Shimomura received the master’s degree from the University of Tsukuba, Japan, in 2018, where he is currently pursuing the Ph.D. degree. 5046 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Nozomu Ohta received the master’s degree from the University of Tsukuba, Japan, in 2020, where he is currently pursuing the Ph.D. degree. Satsuki Nagae is currently pursuing the master’s degree with the University of Tsukuba, Japan. Takeshi Kurata received the B.E., M.E., and D.E. degrees from the University of Tsukuba, Japan. Since 1996, he has been working as a Researcher with AIST, where he is currently the Deputy Director of the Human Augmentation Research Center. From 2003 to 2005, he was a Visiting Scholar with the HIT Laboratory, University of Washington, USA. From 2011 to 2014, he was a Ph.D. Co-Supervisor with Joseph Fourier University (UJF), Grenoble, France. Since 2020, he has been with the ISO/IEC JTC 1/SC 24 HoD, Japan. From 2018 to 2020, he was the Group Leader of the IoT R&D Center, Sumitomo Electric Industries, Ltd. He is also a Professor with the Faculty of Engineering, Information and Systems, University of Tsukuba (Cooperative Graduate School Program). His current research interests include indoor positioning, service research, assistive technology, the IoH, and XR. He received the FY2016 AIST President Award (research). From 2014 to 2017, he was the PDR Benchmark Standardization Committee Chair. Dongyan Wei (Member, IEEE) received the B.S. degree in communication engineering from the University of Electronic Science and Technology of China (UESTC) in 2006, and the Ph.D. degree in signal and information processing from the Beijing University of Post and Telecommunication (BUPT) in 2011. He is currently a Research Fellow of the Aerospace Information Research (AIR) Institute, Chinese Academy of Sciences (CAS). He is the author of one book, more than 30 articles, and more than 20 inventions. His research interests include indoor position, multi-sensor fusion, and positioning in wireless networks. He is a TPC Member of the IPIN 2019 and the Deputy Chair of the IPIN 2022. Xinchun Ji received the B.S. and M.S. degrees in guidance navigation and control (GNC) from the Beijing University of Aeronautics and Astronautics (BUAA) in 2010 and 2013, respectively. He is currently pursuing the Ph.D. degree in electronic information with Northwestern Polytechnical University (NWPU). He is currently a Senior Engineer with the Aerospace Information Research (AIR) Institute, Chinese Academy of Sciences (CAS). His research interests include multi-sensor fusion and geomagnetic matching for vehicle navigation applications. Wenchao Zhang received the B.S. degree in surveying engineering from the China University of Mining and Technology (CUMT) in 2013, the M.S. degree in surveying engineering from Information Engineering University in 2016, and the Ph.D. degree in signal and information processing from the University of Chinese Academy of Sciences (UCAS) in 2020. He is currently an Assistant Researcher with the Aero Information Research (AIR) Institute, Chinese Academy of Sciences (CAS). His research interests include the multi-information fusion method, integrated navigation algorithm, and pedestrian autonomous positioning algorithm. Sebastian Kram received the master’s degree in electrical and communication engineering from the Friedrich Alexander University of Erlangen-Nuremberg (FAU), Germany, in 2017. He joined the Locating and Communication Systems Department, Fraunhofer Institute for Integrated Circuits IIS in 2017. Since 2020, he has been working with the Navigation Group at the Chair for Information Technology (Communication Electronics), FAU. His research interests include tracking algorithms, machine learning, and sensor data fusion. He focuses on radio signal-based adaptive cooperative positioning in adverse environments using both modeland data-driven methods. Maximilian Stahlke received the master’s degree in electronic and mechatronic systems from the Institute of Technology Georg Simon Ohm, Germany, in 2020. Since 2020, he has been working with the Machine Learning and Information Fusion Group, Locating and Communication Systems Department, Fraunhofer Institute for Integrated Circuits IIS. His research interest includes hybrid positioning for radio-based localization systems with a focus on modeland data-driven sensor fusion. Christopher Mutschler received the Diploma and Ph.D. degrees in computer science from Friedrich-Alexander-University Erlangen-Nuremberg (FAU) in 2010 and 2014, respectively. From 2017 to 2019, he was the Head of the Machine Learning and Information Fusion Group, where he was the Chief Scientist. He is the Head of the Precise Positioning and Analytics Department, Fraunhofer Institute for Integrated Circuits IIS, Nüremberg, Germany. He is a part-time Scientific Staff with FriedrichAlexander-University Erlangen-Nuremberg (FAU), offering courses on machine learning. His research interests include machine learning and hybrid sensor fusion for radio-based locating systems. Antonino Crivello received the Ph.D. degree in information engineering and science from the University of Siena, Italy, in 2018. He is a Researcher with the Information Science and Technology Institute, National Research Council (ISTI-CNR), Pisa, Italy. His research interests include indoor positioning and ambient assisted living. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5047 Paolo Barsocchi received the M.Sc. and Ph.D. degrees in information engineering from the University of Pisa, Italy, in 2003 and 2007, respectively. He works as a Researcher with the Information Science and Technologies Institute, National Research Council. He has coauthored more than 100 articles published in international journals and conference proceedings. His research interests include wireless mobile systems and architectures, cyber-physical systems, indoor localization, and wireless sensor networks. He has been a member of numerous program committees and the program chair of several conferences. He is a part of the editorial board of international journals. Michele Girolami received the Ph.D. degree from the Department of Computer Science, University of Pisa in 2015. He is a Researcher with the Information Science and Technology Institute, National Research Council (ISTI-CNR). He contributes to EU and national research projects and supports the organization of conferences and workshops. His research interests are focused on mobile crowdsensing, edge and pervasive computing, and ambient intelligence systems. Filippo Palumbo received the M.Sc. (Hons.) degree incomputer science engineeringfrom the Polytechnic University of Bari, Italy, in 2010, and the Ph.D. degree in computer science from the University of Pisa, Italy, in 2016. He is with the Information Science and Technologies Institute, National Research Council. He has participated in several EUand national-funded research actions in the areas of ambient intelligence. His research interests include the application of AI to wireless sensor networks for intelligent system design and software development in distributed systems. Ruizhi Chen was an Endowed Chair Professor with Texas A&M University-Corpus Christi, USA, the Head and a Professor with the Department of Navigation and Positioning, Finnish Geodetic Institute, Finland, and the Engineering Manager of Nokia, Finland. He is currently a Professor and the Director of the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University. He has publishedtwo books andmore than 200 scientific articles. His current research interests include indoor positioning, satellite navigation, and location-based services. Yuan Wu received the B.S. degree in computer science and technology from Southwest University, China, in 2016, and the M.S. degree in computer application technology from the Institute of Automation, Chinese Academy of Sciences, China, in 2019. He is currently pursuing the Ph.D. degree in geodesy and survey engineering with the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China. His research interests include indoor positioning and navigation, information fusion, and location-based services. Wei Li received the B.S. and M.S. degrees in surveying and mappingfrom the Beijing University of Civil Engineering and Architecture in 2016. He is currently pursuing the Ph.D. degree in geodesy and survey engineering with Wuhan University, Wuhan, China. His research interests include indoor positioning and navigation, and sensor fusion. Yue Yu received the B.S. and M.S. degrees from the Chongqing University of Posts and Telecommunications. He is currently pursuing the Ph.D. degree in geodesy and survey engineering with Wuhan University, Wuhan, China. His research interests include inertial positioning and navigation technology, indoor positioning, and navigation technology-based on chance signal, signal processing, and fusion technology. Shihao Xu received the B.S. degree in surveying and mapping engineering from the China University of Mining and Technology, Jiangsu, China, in 2019. He is currently pursuing the M.S. degree in geodesy and survey engineering with Wuhan University, Wuhan, China. His research interests include the development of location-based services, indoor positioning and navigation technology, and information fusion. Lixiong Huang received the B.S. degree in surveying and mapping engineering from the China University of Mining and Technology, Jiangsu, China, in 2019. He is currently pursuing the M.S. degree in geodesy and survey engineering with Wuhan University, Wuhan, China. His research interests include the development of location-based services, indoor positioning and navigation technology, and information fusion. Tao Liu received the B.S. degree in geographic information system and the M.S. degree in surveying andmapping from LiaoningTechnical University, Fuxin, China, in 2015 and 2018, respectively. He is currently pursuing the Ph.D. degree with the GNSS Research Center, Wuhan University, Wuhan, China. His research interests focus on inertial navigation, multi-sensor fusion algorithm, IMU-based body sensor networks, pedestrian navigation, and indoor positioning. Jian Kuang received the B.Eng. and Ph.D. degrees in geodesy and survey engineering from Wuhan University, Wuhan, China, in 2013 and 2019, respectively. He is currently a Postdoctoral Fellow with the GNSS Research Center, Wuhan University. His research interests focus on inertial navigation, pedestrian navigation, and indoor positioning. 5048 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Xiaoji Niu received the B.Eng. (Hons.) degree in mechanical and electrical engineering and the Ph.D. degree from Tsinghua University, Beijing, China, in 1997 and 2002, respectively. From 2003 to 2007, he was a Postdoctoral Fellow with the Mobile Multi Sensor Systems (MMSS) Research Group, Department of Geomatics Engineering, University of Calgary. From 2007 to 2009, he was a Senior Scientist with SiRF Technology, Inc. He is a Professor with the GNSS Research Center, Collaborative Innovation Center of Geospatial Technology, Wuhan University, Wuhan, China. His research interests focus on INS and GNSS/INS integration for land vehicle navigation and pedestrian navigation. Takuto Yoshida received the B.E. degree in engineering from Nagoya University, Japan, in 2020, where he is currently pursuing the master’s degree with the Graduate School of Engineering. His research interests include indoor positioning, human activity recognition, and machine intelligence. Yoshiteru Nagata received the B.E. degree in engineering from Nagoya University, Japan, in 2021, where he is currently pursuing the master’s degree with the Graduate School of Engineering. Since 2021, he has been a Rinnai Scholarship Foundation Scholarship Student. His research interests include indoor positioning, person-flow estimation, and autonomous driving. Yuto Fukushima received the B.E. degree in engineering from Nagoya University, Japan, in 2021, where he is currently pursuing the master’s degree with the Graduate School of Engineering. His research interests include autonomous mobile robots and human–robot collaboration. Nobuya Fukatani received the B.E. degree in engineering from Nagoya University, Japan in 2020, where he is currently pursuing the master’s degree with the Graduate School of Engineering. His research interests include human activity recognition and power consumption analysis. Nozomi Hayashida received the B.E. degree in engineering from Osaka Prefecture University, Japan, in 2020. He is currently pursuing the master’s degree with the Graduate School of Engineering, Nagoya University. His research interests include AR/MR and human–computer interaction. Yusuke Asai received the B.E. degree in engineering from Nagoya University, Japan, in 2019, where he is currently pursuing the master’s degree with the Graduate School of Engineering. His research interests focus on collaborative autonomous mobile robots in an indoor environment and infrastructural sensor cooperation with mobile robots. Kenta Urano (Graduate Student Member, IEEE) received the B.E., M.E., and Ph.D. degrees in engineering from Nagoya University, Japan, in 2016, 2018, and 2021, respectively. Since 2021, he has been an Assistant Professor with the Graduate School of Engineering, Nagoya University. His research interests include location-based systems, human activity recognition, and biosignal entertainment computing. Wenfei Ge was born in Henan, China, in 1996. She received the bachelor’s degree in surveying and mapping engineering from Wuhan University, China, in 2018, where she is currently pursuing the master’s degree with the GNSS Research Center. Her research interests include aided inertial navigation and multisource sensor integrated navigation. Nien-Ting Lee received the B.S. degree in computer science and information engineering from Chung Hua University, Hsinchu, Taiwan, in 2020. He is currently pursuing the M.S. degree in electrical engineering with the Wireless Mobile Computing Laboratory, Yuan Ze University, Taoyuan, Taiwan. His research interests include AI technology and data analysis. Shih-Hau Fang (Senior Member, IEEE) received the B.S. degree in communication engineering from National Chiao Tung University in 1999, and the M.S. and Ph.D. degrees in communication engineering from National Taiwan University, Taiwan, in 2001 and 2009, respectively. From 2001 to 2007, he was a Software Architect with Chung-Hwa Telecom Ltd. He joined Yuan Ze University (YZU) in 2009. He is currently a Full Professor with the Department of Electrical Engineering, YZU, and also with the MOST Joint Research Center for AI Technology and All Vista Healthcare, Taiwan. He is also a Technical Advisor with HyXen and PTCom Technology Company Ltd. His research interests include artificial intelligence,mobile computing, machine learning, and signal processing. He has received several awards for his research work, including the Young Scholar Research Award from YZU in 2012, the Project for Excellent Junior Research Investigators from MOST in 2013, the Outstanding Young Electrical Engineer Award from the Chinese Institute of Electrical Engineering in 2017,the Outstanding Research Award fromYZU in2018,andthe Best Synergy Award from the Far Eastern Group in 2018. His team received the third place of IEEE Multimedia Big Data (BigMM) HTC Challenge in 2016 and the third place of Indoor Positioning and Indoor Navigation (IPIN) in 2017. He serves as the YZU President’s Special Assistant. He was also an Associate Editor of IEICE Transactions on Information and Systems . POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5049 You-Cheng Jie received the B.S. degree in communications engineering from Yuan Ze University, Taoyuan, Taiwan, in 2020, where he is currently pursuing the M.S. degree in electrical engineering with the Wireless Mobile Computing Laboratory. His research interest includes millimeter-wave radar. Shawn-Rong Young is currently pursuing the B.S. degree in electrical engineering with Yuan Ze University, Taoyuan, Taiwan. Ying-Ren Chien (Senior Member, IEEE) received the B.S. degree in electronic engineering from the National Yunlin University of Science and Technology, Douliu, Taiwan, in 1999, and the M.S. degree in electrical engineering and the Ph.D. degree in communication engineering from National Taiwan University, Taipei, Taiwan, in 2001 and 2009, respectively. Since 2012, he has been with the Department of Electrical Engineering, National Ilan University, Yilan, Taiwan, where he is currently a Full Professor. His research interests include adaptive signal processing theory, machine learning, the Internet of Things, and interference cancellation. Chih-Chieh Yu is currently pursuing the M.S. degree with the Department of Electrical Engineering, National Ilan University, Yilan, Taiwan. His research interests include indoor positioning, machine learning, and signal processing. Chengqi Ma received the B.S. degree in communication engineering from the Harbin Institute of Technology (HIT), Harbin, China, in 2012, and the M.S. degree in wireless communication from Lund University, Lund, Sweden, in 2015. He is currently pursuing the Ph.D. degree with the Electrical Engineering Department, University College London (UCL), London, U.K. His research interests include the development of indoor positioning systems, human activity recognition, and the Internet of Things. Bang Wu received the B.S. and M.S. degrees from the School of Geodesy and Geomatics, Wuhan University (WHU), Wuhan, China, in 2014 and 2016, respectively. He is currently pursuing the Ph.D. degree with the School of Electronic Engineering and Computer Science, Queen Mary University of London (QMUL), London, U.K. His research interests include indoor positioning and indoor navigation, human activity recognition, the Internet of Things, and artificial intelligence. Wei Zhang received the Ph.D. degree from the School of Geodesy and Geomatics, Wuhan University, Wuhan, China, in 2018. He is currently holding a postdoctoral position with the Department of Research Institute for Smart Cities, Shenzhen University, Shenzhen, China. His current research interest focuses on indoor positioning and indoor navigation. Yankun Wang received the Ph.D. degree in cartography from the Wuhan University of China in 2018. He joined Shenzhen University in 2018, where he is currently holding a postdoctoral position with the Department of Research Institute for Smart Cities, School of Architecture and Urban Planning. His main research interests include spatial modeling, indoor positioning, and spatial cognition. Yonglei Fan received the B.S. degree from South China Normal University in 2017, and the M.S. degree from the University of Chinese Academic and Sciences, Beijing, in 2020. He is currently pursuing the Ph.D. degree with the School of Electric Engineering and Computer Science, Queen Mary University of London, London, U.K. His research interests include the IoT, positioning estimation, geo-information data mining, and other location-related analysis. Stefan Poslad received the Ph.D. degree from Newcastle University. He is currently an Associate Professor with the School of Electronic Engineering and Computer Science, Queen Mary University of London (QMUL), London, U.K. He is the Head of the IoT Laboratory. His research interests include indoor positioning and indoor navigation, human activity recognition, the Internet of Things, ubiquitous computing, semantic web, distributed system management, and artificial intelligence. David R. Selviah (Member, IEEE) received the Ph.D. degree in photonic engineering from the Trinity College, Cambridge University, Cambridge, U.K. He is currently a Reader in optical devices, interconnects, algorithms, and systems with the Electronic and Electrical Engineering Department, University College London. His current research interests include machine learning, feature recognition, 3-D point cloud processing, indoor positioning and navigation, and quantum dot material characterization. Weixi Wang received the Ph.D. degree in geodesy and survey engineering from Liaoning Technical University, Liaoning, China, in 2007. He completed a postdoctoral fellowship with the School of Resource and Environmental Sciences, Wuhan University, China, in 2013. He is currently an Associate Professor with the School of Architecture and Urban Planning, Shenzhen University, Shenzhen, China. He is also the Deputy Head of the Department of Urban Spatial Information Engineering, School of Architecture and Urban Planning. He is the author or coauthor of over 30 refereed journal articles and conference papers, and one book. His current research interests include real-time positioning and navigation, target feature extraction and matching, and 3-D model reconstruction. 5050 IEEE SENSORS JOURNAL, VOL. 22, NO. 6, MARCH 15, 2022 Hong Yuan received the Ph.D. degree from the Shanxi Observatory of the Chinese Academy of Sciences in 1995. He is currently a Research Fellow of the Aerospace Information Research (AIR) Institute, Chinese Academy of Sciences. He has been engaged in the research on ionospheric radio wave propagation, GPS/Beidou satellite navigation system construction, manned space applications, ionospheric physics, and ionospheric detection. He has hosted or participated in 13 national and provincial-level projects, eight provincial andministerial scienceand technology awards, and more than 30 invention patents. He has published more than 60 articles. He engaged in software and hardware design and algorithm research related to satellite navigation, multi-source fusion navigation, and ionospheric detection. Yoshitomo Yonamoto received the B.E. degree from Keio University in 2020, where he is currently pursuing the M.S. degree in computer science. His research interests include computer vision and indoor localization. Masahiro Yamaguchi received the B.S., M.S., and Ph.D. degrees in engineering from Keio University, Japan, in 2016, 2017, and 2021, respectively. From 2018 to 2019, he was a University Project Assistant with the Graz University of Technology. Since 2021, he has been working with NEC. His research interests include SLAM, 3D reconstruction, and computer vision. Tomoya Kaichi received the B.E. and M.Sc.Eng. degrees in information and computer science from Keio University, Japan, in 2017 and 2018, respectively, where he is currently pursuing the Ph.D. degree in science and technology. His main research interests include human motion estimation, optical–inertial sensor fusion, and hyperspectral analysis. Baoding Zhou received the Ph.D. degree in photogrammetry and remote sensing from Wuhan University, Wuhan, China, in 2015. He is currently an Assistant Professor with the College of Civil and Transportation Engineering, Shenzhen University, Shenzhen, China. His research interests include indoor localization and mapping, mobile computing, and intelligent transportation. Xu Liu received the B.E. degree in geomatics engineering from the Chengdu University of Technology, China, in 2016, and the M.E. degree in geodesy and geomatics engineering from the Kunming University of Science and Technology, China, in 2019. He is currently pursuing the Ph.D. degree with the School of Civil and Transportation Engineering, Shenzhen University. His main research interests include indoor localization and navigation, and the Internet of Things. Zhining Gu received the B.S. degree in geographic information science from Harbin Normal University, China, in 2019. He is currently pursuing the M.E. degree with Shenzhen University. His main research interests include the design and implementation of indoor tracking and navigation systems, deep learning algorithm design, and intelligent transportation. Chengjing Yang received the B.S. degree in traffic transportation from Dalian Maritime University, China, in 2019. He is currently pursuing the M.E. degree with Shenzhen University. His main research interests include indoor positioning and intelligent transportation. Zhiqian Wu received the B.S. degree in traffic transportation from Dalian Maritime University, China, in 2020. He is currently pursuing the M.E. degree with Shenzhen University. His main research interests include pedestrian indoor positioning and intelligent transportation. Doudou Xie received the B.S. degree in traffic engineering from Shandong Jiao Tong University, China, in 2019. He is currently pursuing the M.E. degree with Shenzhen University. His main research interests include the design and implementation of indoor tracking and navigation systems, robot mapping and self-localization, and intelligent transportation. Can Huang is currently pursuing the B.E. degree with Shenzhen University. His main research interests include indoor localization and intelligent transportation. Lingxiang Zheng received the Ph.D. degree in artificial intelligence from Xiamen University, China. He is currentlya Professor with the School of Informatics, Xiamen University. His research interests include indoor positioning, mobile computing, and smart devices. POTORTÌ et al. : OFF-LINE EVALUATION OF INDOOR POSITIONING SYSTEMS IN DIFFERENT SCENARIOS 5051 Ao Peng (Member, IEEE) received the M.Sc. and Ph.D. degrees in communication and information systems from Xiamen University, Fujian, China, in 2011 and 2014, respectively. He joined the School of Informatics, Xiamen University, in 2015, where he is currently an Assistant Professor. His research interests include satellite navigation and multi-source positioning and navigation. Ge Jin received the B.S. degree in communication engineering from Hohai University (HHU), Changzhou, China, in 2015. He is currently pursuing the M.S. degree in electronic and communication engineering with Xiamen University (XMU), Xiamen, China. His research interest includes indoor positioning systems. Qu Wang received the B.S. degree from the School of Software Engineering, Beijing University of Posts and Telecommunication, China, in 2013, and the M.S. degree from the University of Chinese Academy of Sciences, Beijing, China, in 2017. He is currently pursuing the Ph.D. degree with the School of Information and Communication Engineering, Beijing University of Posts and Telecommunications. His current interests include location-based services, pervasive computing, computer vision, and machine learning. Haiyong Luo (Member, IEEE) received the B.S. degree from the Department of Electronics and Information Engineering, Huazhong University of Science and Technology, Wuhan, China,in 1989, the M.S. degree from the School of Information and Communication Engineering, Beijing University of Posts and Telecommunication, China, in 2002, and the Ph.D. degree in computer science from the University of Chines Academy of Sciences, Beijing, China, in 2008. He is currently an Associate Professor with the Institute of Computer Technology, Chinese Academy of Sciences, China. His main research interests include location-based services, pervasive computing, mobile computing, and the Internet of Things. Hao Xiong received the B.S. degree from the School of Software Engineering, Beijing University of Posts and Telecommunications, Beijing, China, in 2019, where he is currently pursuing the M.S. degree. His current research interests include location-based services, pervasive computing, and machine learning. Linfeng Bao received the B.S. degree from the School of Electronic Information, Wuhan University, China. He is currently pursuing the graduate degree with the Research Center for Ubiquitous Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences. His research interests include embedded systems and GNSS aided inertial navigation systems. Pushuo Zhang received the B.S. degree from the School of Geological Engineering, Chang’an University, China, in 2019. He is currently pursuing the M.S. degree with the School of Software Engineering, Beijing University of Posts and Telecommunications, China. His current research interests include location-based services, pervasive computing, inertial navigation, and machine learning. Fang Zhao received the B.S. degree from the School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China, in 1990, the M.S. and Ph.D. degrees in computer science and technology from the Beijing University of Posts and Telecommunication, Beijing, China, in 2004 and 2009, respectively. She is currently a Professor with the School of Software Engineering, Beijing University of Posts and Telecommunication. Her research interests include mobile computing, location-based services, and computer networks. Chia-An Yu is currently pursuing the B.S. degree in electrical engineering with the Wireless Mobile Computing Laboratory, Yuan Ze University, Taoyuan, Taiwan. His research interests include sensor data calculation and transformer model in speech enhancement. Chun-Hao Hung received the B.S. degree in computer science and information engineering from Chung Hua University, Hsinchu, Taiwan, in 2020. He is currently pursuing the M.S. degree in electrical engineering with the Wireless Mobile Computing Laboratory, Yuan Ze University, Taoyuan, Taiwan. His research interests include sensor data calculation and database analysis. Leonid Antsfeld received the M.Sc. degree in applied mathematics and computer science from Technion, Israel Institute of Technology, in 2005, and the Ph.D. degree in computer science from the University of New South Wales, Sydney, NSW, Australia, in 2014. He has extensive industrial experience solving real-world complex problems by applying his research at Rafael, Intel, NICTA (Australia’s Information and Communications Technology Research Centre of Excellence), and Xerox Innovation Group. Since 2017, he has been a Senior Researcher with Naver Labs Europe, Grenoble, France, the biggest industrial research lab in AI. He is the author of several scientific articles and patents. His current research interest includes sensor fusion for indoor positioning and navigation.