Analysis of the Accuracy of Ten Algorithms for Orientation Estimation Using Inertial and Magnetic Sensing under Optimal Conditions: One Size Does Not Fit All [original]

sensors

Article

Analysis of the Accuracy of Ten Algorithms for Orientation

Estimation Using Inertial and Magnetic Sensing under Optimal

Conditions: One Size Does Not Fit All

Marco Caruso 1,* , Angelo Maria Sabatini 2, Daniel Laidig 3, Thomas Seel 3, Marco Knaflitz 1,

Ugo Della Croce 4and Andrea Cereatti 1





Citation: Caruso, M.; Sabatini, A.M.;

Laidig, D.; Seel, T.; Knaflitz, M.; Della

Croce, U.; Cereatti, A. Analysis of the

Accuracy of Ten Algorithms for

Orientation Estimation Using Inertial

and Magnetic Sensing under Optimal

Conditions: One Size Does Not Fit

All. Sensors 2021,21, 2543. https://

doi.org/10.3390/s21072543

Academic Editor: Alberto J. Palma

Received: 31 January 2021

Accepted: 24 March 2021

Published: 5 April 2021

Publisher’s Note: MDPI stays neutral

with regard to jurisdictional claims in

published maps and institutional affil-

iations.

Licensee MDPI, Basel, Switzerland.

This article is an open access article

distributed under the terms and

conditions of the Creative Commons

Attribution (CC BY) license (https://

creativecommons.org/licenses/by/

4.0/).

1PolitoBIOMed Lab—Biomedical Engineering Lab and Department of Electronics and Telecommunications,

Politecnico di Torino, 10129 Torino, Italy; mar[email protected] (M.K.); andrea.cer[email protected] (A.C.)

2Department of Excellence in Robotics & AI, The BioRobotics Institute, Scuola Superiore Sant’Anna,

56127 Pisa, Italy; [email protected]

3Control Systems Group, Technische Universität Berlin, 10623 Berlin, Germany;

[email protected] (D.L.); seel@control.tu-berlin.de (T.S.)

4Department of Biomedical Sciences, University of Sassari, 07100 Sassari, Italy; [email protected]

*Correspondence: mar[email protected]

Abstract:

The orientation of a magneto and inertial measurement unit (MIMU) is estimated by means

of sensor fusion algorithms (SFAs) thus enabling human motion tracking. However, despite several

SFAs implementations proposed over the last decades, there is still a lack of consensus about the best

performing SFAs and their accuracy. As suggested by recent literature, the filter parameters play a

central role in determining the orientation errors. The aim of this work is to analyze the accuracy

of ten SFAs while running under the best possible conditions (i.e., their parameter values are set

using the orientation reference) in nine experimental scenarios including three rotation rates and

three commercial products. The main finding is that parameter values must be specific for each SFA

according to the experimental scenario to avoid errors comparable to those obtained when the default

parameter values are used. Overall, when optimally tuned, no statistically significant differences are

observed among the different SFAs in all tested experimental scenarios and the absolute errors are

included between 3.8 deg and 7.1 deg. Increasing the rotation rate generally leads to a significant

performance worsening. Errors are also influenced by the MIMU commercial model. SFA MATLAB

implementations have been made available online.

Keywords:

MIMU; orientation estimation; filter parameters; filter comparison; wearable sensors;

sensor fusion; human motion; Kalman filters; complementary filters; optimal parameters

1. Introduction

The accurate estimation of the orientation of a rigid body from the recordings of minia-

turized low-cost magneto-inertial measurement units (MIMUs) is still an open challenge

for the human movement analysis community. Errors affecting the orientation estimates

have a direct negative impact on the quality of estimated quantities in both angular and

linear kinematics, therefore limiting the full exploitation of inertial sensing in monitoring

daily-life physical activities, as well as in clinical and sports applications [1].

In its full configuration, a MIMU embeds a triaxial accelerometer which measures the

specific force (i.e., the vector difference between the coordinate and the gravity accelera-

tions), a triaxial gyroscope which measures the angular rate, and a triaxial magnetometer

which senses the local magnetic field (i.e., the vector sum between the Earth’s magnetic field

and the external magnetic fields created by ferromagnetic disturbances). The sensor fusion

approach aims at estimating the absolute orientation of the MIMU with respect to a global

coordinate system (GCS), usually defined to have a vertical axis aligned with the gravity

direction and one horizontal axis direction aligned with the Earth’s magnetic north, by

Sensors 2021,21, 2543. https://doi.org/10.3390/s21072543 https://www.mdpi.com/journal/sensors

Sensors 2021,21, 2543 2 of 25

exploiting the complementary characteristics of the signals recorded by the MIMU. The first

step consists in integrating the kinematics equation which links the angular rate with the

orientation change over time to obtain a first approximation of the orientation estimate. The

initial conditions for the integration can be obtained by an absolute orientation estimate by

using only the accelerometer and the magnetometer measurements in absence of motion [

However, the orientation estimated this way is prone to drift due to the integration of the

slow-varying bias affecting the gyroscope measurements [

]. To cope with this problem the

accelerometer and the magnetometer measurements are employed to correct the drift in

both roll and pitch angles (also jointly known as inclination) and yaw angle (also known as

declination or heading). Nonetheless, such correction shows some limitations. In fact, the

inclination estimated by the accelerometer is highly reliable only during static conditions

(i.e., when the coordinate acceleration is negligible and only the gravity direction is sensed).

Moreover, the heading resulting from the magnetometer measurements needs to be dealt

with care due to the presence of external magnetic fields. Researchers from different fields

such as navigation and biomechanics have proposed several sensor fusion implementa-

tions over the years, including machine and deep learning approaches, to provide accurate

orientation estimates using MIMUs [

–

]. The large majority of the published sensor

fusion algorithms (SFAs) can be grouped in two main classes: Kalman filters (KF) [

]

and complementary filters (CF). In the last decades, several formulations of both classes

have been proposed including different mathematical orientation representations (e.g.,

quaternion, rotation matrix, Euler angles etc.), different Kalman filter formulations (direct

or indirect, linear, extended, unscented, etc.), and different strategies to fuse the signal

information (algebraic or optimization) [

]. Despite the large number of studies aimed

at comparatively evaluating different sensor fusion algorithms [

–

] and

type of sensors (SFA inter-consistency) [

], contradictory results have been observed and

the literature is still inconclusive about the expected level of accuracy associated to the

MIMU orientation estimation.

Based on the existing body of literature, it is difficult to draw conclusions regarding the

“best” algorithm and filtering approach (e.g., CF or KF). Furthermore, errors appear to be

highly variable depending on the experimental scenario, commercial device, and algorithm,

thus making the generalization of the results impossible [

]. A way of looking at the body

of literature is to group studies proposing novel algorithms (“original algorithm studies”)

([

–

]) and studies focusing on the comparison of existing algorithms/software

packages without proposing new ones (“comparative studies”) ([

–

]). A summary

of the results in “comparative studies” and “original algorithm studies” are reported in

the Appendix A(Tables A1 and A2). As a general observation, the magnitude of the

errors reported in “comparative studies” are usually higher than those reported in the

“original algorithm studies”, up to one order of magnitude. For example, the errors

for Madgwick’s filter reported in the original study [

] amounted to about 1 deg, while

in [

], which involved more challenging experimental conditions, errors were greater

than 13 deg. Furthermore, the filter proposed by Guo et al., in [

] was compared with

the KF by Valenti et al. [

] for which the errors were twice as large than those reported

in the original article. These differences may be due to a number of reasons. First, in the

“comparative studies” the experimental conditions under which SFAs are tested are often

different from those employed in the original paper in terms of hardware, sensor noise,

rotation rates, accelerations magnitude, ferromagnetic disturbances, type of motion, etc.

In addition, when a new SFA is presented and its performance evaluated, the proponents

often know how to optimally tune the SFA for the specific operating conditions based

either on ground-truth knowledge or following a trial-and-error approach [

]. However,

when non-experts apply SFAs to different experimental datasets, or in specific human

movement applications, the SFA optimization can be difficult resulting in a performance

deterioration. It has often been observed that feeding the proper parameter values to

any SFA is crucial [

]. Several intrinsic and extrinsic factors affect the choice of the

parameter values, among them the most influencing are amplitude of motion, sensors

Sensors 2021,21, 2543 3 of 25

noise specification, time required by the algorithm to reach convergence, and the amount

of ferromagnetic disturbances [31,33].

Based on the considerations above, it is evident that carrying out any “fair”, meaning-

ful and generalizable comparative evaluation among SFAs requires the SFAs parameters

to be properly tuned and used under identical experimental conditions (i.e., using the

same dataset).

The primary aim of this work is to perform a thorough comparative evaluation of the

accuracy of ten of the most popular SFAs proposed in the literature by considering experi-

mental data recorded by three commercial products (Xsens-MTx (Xsens, Enschede, The

Netherlands), APDM-Opal (APDM INC., Portland, OR, U.S.A.), and Shimmer-Shimmer3

(Shimmer Sensing, Dublin, Ireland)) and for three rotation rates (slow, medium, and fast

motions) using the orientation provided by a multi-camera stereo-photogrammetric system

(SP) as ground truth. For the sake of analysis generalizability, the SFAs performance is

assessed under optimal and default parameters tuning. In the optimal tuning configura-

tion, parameters are determined by minimizing the absolute orientation error with respect

to the gold standard for each experimental scenario, allowing for the assessment of the

filter performance under its best possible conditions. In addition, the errors obtained

using the default parameter values as defined by the SFA proponents were also computed

to highlight the impact of using non-tuned and generic parameter values for different

experimental scenarios. Computation time of the different SFAs was also evaluated.

To the best of our knowledge, this study is the most comprehensive study evaluating

a considerable number of SFAs under optimal filter parameter tuning condition and under

various experimental scenarios.

2. Materials and Methods

2.1. Optimal Working Conditions

To work properly, each SFA requires the tuning of a variable number of parameters [

In the present context, optimal working conditions refer to the parameter values providing

the lowest absolute average orientation error for a given experimental data recording

(i.e., they are specialized for each dataset) and hence the best achievable performance

(best case scenario). In other words, each parameter value of each SFA was optimally

tuned on each of the nine experimental scenarios (three rotation rates for three commercial

products). The selection of the optimal parameter values is performed relying on the gold

standard orientation. This strategy is implemented exclusively for comparative purposes

and may be replicated only with the aid of a reference orientation (e.g., SP system with

sub-millimeter accuracy).

2.2. Selected Algorithms

A total of ten SFAs, including five complementary filters and five Kalman filters were

selected among the most popular and performing ones: Mahony et al. 2008 [

] (MAH),

Madgiwck et al., 2011 [

] (MAD), Sabatini 2011 [

] (SAB), Valenti et al., 2015 [

] (VAC),

Ligorio and Sabatini 2015 [

] (LIG), Valenti et al., 2016 [

] (VAK), Seel et al., 2017 [

]

(SEL), Guo et al., 2017 [

] (GUO), MATLAB complementary filter R2020a (MCF) which is

the MathWorks implementation of VAC but with only two parameters, MATLAB Kalman

filter R2020a (MKF), which is the MathWorks implementation of the filter by Roetenberg

et al., 2005 [

]. The details for each SFA are reported in Table 1including the total

number of parameters exposed.

As stated in the introduction, all the SFAs are based on the angular velocity integration

to obtain a first approximation of the orientation estimate. The differences are related to

how the accelerometer and magnetometer measurements are used to compensate for the

drift caused by the time integration of the angular velocity and to additional strategies

designed to deal with the linear acceleration and the ferromagnetic disturbances. In the

following, a short description of each filter is given.

Sensors 2021,21, 2543 4 of 25

Table 1.

Details of each Sensor Fusion Algorithm (SFA) considered. The # Params column reports the total number of

parameters of each SFA. The p

and p

report the description of the parameter tuned to detect the optimal values. a.u. =

arbitrary units.

CF # Params p1Default p2Default

MAH 2

—inverse gyroscope weight

1 rad/s ki—weight for online bias

estimation 0.3 rad/s

MAD 1 β—inverse gyroscope weight 0.1 rad/s / /

VAC 9 gmag—magnetometer weight 0.01 a.u. ath2—threshold for

accelerometer vector selection

0.2 a.u.

SEL 4 τacc—accelerometer time

constant 1 s τmag—magnetometer time

constant 3 s

MCF 2 gmag—magnetometer weight 0.01 a.u. / /

KF # Params p1Default p2Default

SAB 6 σgyr—inverse gyroscope

weight 0.007 rad/s ath—threshold for

accelerometer vector selection

40 mg

LIG 6 σgyr—inverse gyroscope

weight 1 rad/s

—Gauss-Markov parameter

of the prediction model to set

the variance of external

acceleration and

ferromagnetic disturbances

1 a.u.

VAK 3 σgyr—inverse gyroscope

weight 0.004 rad/s σacc—inverse accelerometer

weight 0.014 m/s2

GUO 3 σgyr—inverse gyroscope

weight 0.001 rad/s / /

MKF 8 σ2gyr—inverse gyroscope

weight

9.14

−5

(rad/s)

2/ /

MAD is a CF in which the accelerometer and the magnetometer measurements are

fused by means of a gradient descent algorithm. For the magnetic readings, only the

horizontal projection is used to correct the orientation. The fusion process is governed by a

unique parameter. A low value of it gives more weight to the gyroscope measurements.

MAH is a CF which considers the discrepancy between the measured Earth’s fixed vector

(gravity and magnetic field) and their estimates obtained using the previous orientation.

This discrepancy (called error) is then weighted by a parameter and subtracted from

the gyroscope signal before its integration. As opposite to MAD the magnetic readings

influence both the attitude and heading. In neither filter is it possible to weigh differently

the accelerometer and the magnetometer contributions and no strategy is implemented

to compensate for the linear acceleration or the magnetic disturbances. SEL is a CF with

independent accelerometer-based inclination correction and magnetometer-based heading

correction. The latter is purely horizontal, which ensures that magnetic disturbances cannot

affect the inclination. The algorithm is parameterized via two correction constants for the

inclination and heading disagreements, one optional bias estimation parameter and an

adaptation factor that reduces the weight of the accelerometer readings during dynamic

motions. VAC is a CF which employs the accelerometer readings to correct the inclination

by comparing the actual and the observed gravity direction. The magnetometer readings

are then projected onto the horizontal plane and the angle between the observed magnetic

North and the estimated one is used to correct the heading. The two correction processes are

governed by two independent gains. VAC implements a linear two-thresholds method to

progressively reject the measurements whereas their magnitude exceeds the expected value

(i.e., 9.81 m/s

and the local magnetic norm, respectively). MCF is the implementation of

VAC by MathWorks from Sensor Fusion and Tracking Toolbox.

VAK, LIG GUO, SAB, and MCF belong to the class of the KFs. As a general rule,

the weight given to the information provided by each of the three sensors is governed by

dedicated parameters. In particular, the higher is the value of these parameters the less

the information provided is trusted. For this reason, they are called “inverse weight”. A

Sensors 2021,21, 2543 5 of 25

typical feature of the KF is the possibility to track the disturbances with the “state-vector

augmentation technique”. While on one hand it represents an advantage, on the other hand

each quantity tracked in the state vector must be weighted with dedicated parameters to

account for the uncertainty in their modelling and, above all, a large state vector dimension

may result in observability problem (i.e., the information contained in the output variables

is no longer sufficient to completely describe the system behavior).

VAK employs the same algebraic approach of VAC to correct the orientation, but

as opposite to VAC no thresholds are used to reject linear accelerations or ferromagnetic

disturbances which are instead employed in SAB. In addition, SAB allows the modelling

of the ferromagnetic disturbances, seen as a time-variant bias superimposed to the mag-

netometer readings. LIG consists of two KFs which separately estimate the inclination

and the heading, using the information provided by the gyroscope/accelerometer and

gyroscope/magnetometer, independently. The two pieces of information are then merged

by using an algebraic method. Linear accelerations and ferromagnetic disturbances are

modelled as a first order Gauss-Markov model. GUO is a KF explicitly designed to perform

fast. To this end, an algebraic approach which fuses the accelerometer and magnetometer

measurements for the orientation correction is adopted and no additional strategy to filter

out the linear accelerations and ferromagnetic disturbances is implemented. MKF is the

MathWorks implementation (Sensor Fusion and Tracking Toolbox) of the filter originally

proposed by Luinge et al. [

] and extended by Roetenberg et al. [

] which is also embedded

in the Xsens software. Differently from the other four KFs described, MKF is an indirect KF,

which means that it minimizes the uncertainty of the orientation error rather than of the

absolute orientation (direct formulation). In this filter, the inclination and heading errors are

separately computed by comparing the actual and the estimated directions of the gravity

and global magnetic field using the information provided by gyroscope/accelerometer and

gyroscope/magnetometer, respectively. These two orientation errors are then included in

the state vector to be minimized. MKF augments its state vector with the gyroscope bias,

acceleration errors (seen as the linear acceleration component in the accelerometer output)

and the ferromagnetic disturbances.

For each SFA, the optimal tuning involved a heuristic space search of the two most

important parameters (namely

and

) when relevant. The decision to tune at most

two parameters was a compromise between the search space dimension and the related

computational time. As a general rule, considering that the gyroscope is the main source of

information in a sensor fusion framework, the parameter related to the weight given to

it was always tuned, when exposed by the SFA. All the remaining parameter values are

set to default. In fact, it has to be said that the accelerometer and magnetometer related

parameters should not be set based on the sensor noise only (i.e., electrical noise) because

in the strict sense it is not representative of real problems affecting the two sensors: the

linear accelerations and the ferromagnetic disturbances.

All the SFA codes have been made available on GitHub website (the link is reported

in “Data Availability Statement” section). All the implementations are written in MATLAB

code. All the details for each implementation are given in the code headers.

2.3. Experimental Setup

Considering that the performance of the SFAs could greatly vary due to the hardware

characteristics, in this study, three pairs of commercial MIMUs were considered: Xsens-

MTx, APDM-Opal, Shimmer-Shimmer3. The specifications for each model are reported

in Tables A3–A5 of Appendix B. A wooden board was used and the three MIMU pairs

were aligned as depicted in Figure 1. A T-square was employed to draw the lines which

ensured the accurate positioning of the MIMUs and the markers. The orthogonal tolerance

of the instrument is reflected in an alignment error lower than 0.2 deg. The MIMUs were

positioned at a relative distance of 50 mm. A total of eight reflective spherical markers

(diameter equal to 14 mm, minimum inter-distance of 85 mm) were located on the board

to provide the orientation reference whose trajectories were acquired by 12 infrared SP

Loading more pages...