data
Article
BROAD—A Benchmark for Robust Inertial
Orientation Estimation
Daniel Laidig 1,* , Marco Caruso 2,3 , Andrea Cereatti 3and Thomas Seel 4
Citation: Laidig, D.; Caruso, M.;
Cereatti, A.; Seel, T. BROAD—A
Benchmark for Robust Inertial
Orientation Estimation. Data 2021,6,
72. https://doi.org/10.3390/
data6070072
Academic Editor: Joaquín
Torres-Sospedra
Received: 12 May 2021
Accepted: 9 June 2021
Published: 27 June 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1Control Systems Group, Technische Universität Berlin, 10623 Berlin, Germany
2PolitoBIOMed Lab—Biomedical Engineering Lab, Politecnico di Torino, 10129 Torino, Italy;
marco.car[email protected]
3Department of Electronics and Telecommunications, Politecnico di Torino, 10129 Torino, Italy;
andrea.cer[email protected]
4Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität
*Correspondence: laidig@control.tu-berlin.de
Abstract:
Inertial measurement units (IMUs) enable orientation, velocity, and position estimation
in several application domains ranging from robotics and autonomous vehicles to human motion
capture and rehabilitation engineering. Errors in orientation estimation greatly affect any of those
motion parameters. The present work explains the main challenges in inertial orientation estimation
(IOE) and presents an extensive benchmark dataset that includes 3D inertial and magnetic data
with synchronized optical marker-based ground truth measurements, the Berlin Robust Orientation
Estimation Assessment Dataset (BROAD). The BROAD dataset consists of 39 trials that are conducted
at different speeds and include various types of movement. Thereof, 23 trials are performed in
an undisturbed indoor environment, and 16 trials are recorded with deliberate magnetometer and
accelerometer disturbances. We furthermore propose error metrics that allow for IOE accuracy
evaluation while separating the heading and inclination portions of the error and introduce well-
defined benchmark metrics. Based on the proposed benchmark, we perform an exemplary case
study on two widely used openly available IOE algorithms. Due to the broad range of motion and
disturbance scenarios, the proposed benchmark is expected to provide valuable insight and useful
tools for the assessment, selection, and further development of inertial sensor fusion methods and
IMU-based application systems.
Keywords:
inertial sensor; inertial measurement unit; orientation estimation; attitude estimation;
magnetic disturbances; benchmark dataset
1. Introduction
Inertial measurement units (IMUs) have become small and lightweight and are there-
fore used in an increasing number of application domains. They are integrated into various
types of consumer electronics, used in autonomous drones and vehicles, and facilitate
non-restrictive human motion tracking in various health care and sporting applications [
1
].
Examples of the latter include rehabilitation robotics [
2
], feedback-controlled neuroprosthe-
ses [3,4], and rehabilitation monitoring [5,6].
An IMU measures angular rates, specific force (also called proper acceleration), and
magnetic field strength. The measurements are 3D vectors in a local coordinate system
that rotates with the object of interest to which the IMU is attached. Motion analysis using
inertial sensors usually involves the derivation of motion parameters like the orientation of
the object to which the sensor is attached and its velocity and position with respect to an
inertial frame of reference [
7
,
8
]. In order to determine those motion parameters, one must
first determine the orientation of the IMU with respect to an inertial frame of reference. In
the following, we call this step inertial orientation estimation (IOE).
Data 2021,6, 72. https://doi.org/10.3390/data6070072 https://www.mdpi.com/journal/data
Data 2021,6, 72 2 of 20
The inertial frame of reference is commonly defined by a vertical coordinate axis,
defined by the direction of gravity, and a horizontal coordinate axis that is aligned with
the horizontal component of Earth’s magnetic field. Sensor fusion methods are employed
to combine the accelerometer and magnetometer readings with the angular velocity mea-
surements of the gyroscope that is strapdown integrated to track changes of orientation.
This fundamental problem of inertial sensor fusion has been solved by a large number
of previously proposed IOE algorithms. A good overview of existing methods is found
in [
9
,
10
]. The majority of methods represent the orientation in terms of unit quaternions.
Commonly used filter structures are complementary filters and (extended/unscented)
Kalman filters.
It is a well-known fact that the amount of information and certainty that is contained
in each measurement signal varies depending on the performed motion and environmental
factors such as vibrations and magnetic disturbances [
11
,
12
]. In a fast and jerky motion, the
accelerometer must be used much more carefully than during a smooth and slow motion.
The magnetometer measurements are known to be highly susceptible to the presence of
ferromagnetic material and electronic devices [
13
]. Previous research has led to adaptive
algorithms that try to compensate such variations and disturbances [14,15].
As illustrated in Figure 1, there is a general need for robust IOE algorithms that
provide accurate orientation estimates and perform well for a broad range of motions
without the need to manually adjust tuning parameters for each type of motion [
16
]. When
it comes to assessing the performance of IOE algorithms, a vast number of contributions
are found that evaluate specific algorithms in specific application contexts, but few papers
investigate the ability of IOE algorithms to perform across different types of motion and
environmental conditions. To the best of our knowledge, only Caruso et al.
[9]
provides
a systematic evaluation of multiple algorithms with respect to three different movement
speeds, and Fan et al.
[17]
investigates the influence of magnetic disturbances on the
attitude and heading estimates. However, there are no studies providing a systematic and
comprehensive evaluation of the impact of different magnetic disturbances, the difference
between translational and rotational motions, and different movement speeds on various
IOE algorithms.
IOE algorithm 1, parametrization 1
IOE algorithm 1, parametrization 2
IOE algorithm 2, default parametrization
hypothetical ideal IOE algorithm
37 7 7 ?
7 7 3 3 ?
73 3 7?
3 3 3 3 3
3
7
?
good performance
poor performance
uncertain
limited rotations,
fast translations
rapid rotations,
limited translations
fast periodic
translations
slow with magnetic
disturbances
?
unknown
conditions
Figure 1.
The accuracy of inertial orientation estimation (IOE) depends on the employed algorithm,
the chosen algorithm parametrization, and the specific application scenario. There is a lack of datasets
and methods for systematic evaluation of IOE algorithm performance across a broad range of motion
characteristics and environmental conditions.
As detailed in Section 2, a thorough algorithm comparison is limited by a lack of an
extensive and openly available benchmark that includes a large number of trials comprising
a diverse set of movement types and environmental conditions and therefore allows for a
truly comprehensive evaluation and assessment of IOE solutions. Such a heterogeneous
set of trials with either only rotation, only translation, or combined movements at different
speeds and with different durations is important for two reasons. First, in order to assess the
Data 2021,6, 72 3 of 20
robustness of an IOE algorithm for a wide variety of motions and environmental conditions,
those motions and conditions must be included in the data set. Second, comparing the
errors for different trials yields insight into how algorithm performance or the choice of
optimal parameters depends on the characteristics of the motion. As magnetic disturbances
represent a major challenge in orientation estimation, it is crucial to not only consider
homogeneous magnetic fields but to also include a broad range of magnetic disturbances.
The present contribution aims at filling this gap by providing a benchmark dataset
that is particularly useful for the objective assessment and further development of IOE
algorithms. To the best of our knowledge, this the first publicly available benchmark
dataset that
• includes a broad range of different motions at various speeds
• contains separate trials with various deliberate magnetic disturbances
• contains separate trials with disturbances that affect the measured accelerations
•
is already time-synchronized and contains ground truth data that requires no further
preprocessing.
We further introduce error metrics that separately consider heading, inclination, and
the total orientation error, specify well-defined benchmark metrics that can be used to
assess and compare IOE algorithm performance, and provide example code to calculate
those metrics.
The remainder of the article is structured as follows. In Section 2we review a number
of openly available datasets that are suitable for objective performance assessment of IOE
algorithms. In Section 3we present the new benchmark dataset and describe the contents
and file structure. In Section 4we describe the measurement setup, the performed data
preprocessing, and introduce error metrics useful for quantifying the orientation estimation
accuracy and define reproducible benchmark metrics. Section 5is dedicated to applying
some existing orientation estimation algorithms to the proposed benchmark. Conclusions
are presented in Section 6.
2. Brief Review of Existing Datasets for IOE Validation
Objective assessment of the accuracy of an IOE algorithm requires a highly reliable
and accurate ground truth measurement. The most widely accepted gold standard mea-
surement for the orientation of a moving object is to derive its orientation from the position
measurements of active or reflective optical markers that are tracked by a set of cameras, a
technique that is known as stereophotogrammetry or optical motion capture (OMC). In the
past two decades, a number of studies have been performed in which an IMU and optical
markers are attached to moving objects, including human body segments, aerial vehicles,
and robotic systems. While many of the datasets in these studies would be suitable for
accuracy evaluation of IOE algorithms, the datasets are often not openly available or only
available upon request to the authors, often due to privacy or ethical concerns. Further-
more, there is a lack of systematic benchmarking approaches for IOE accuracy evaluation.
Despite this general lack of datasets and methods for evaluation, a few datasets have been
made publicly available and are briefly reviewed in the following. Some of these datasets
are created for general IOE validation and some are provided for evaluation in specific
application contexts.
In total, we found five publicly available datasets that contain optical and inertial data
from a moving object in a way that it allows for accuracy evaluation of an IOE algorithm.
An overview of key features of the found datasets and the proposed dataset is given in
Table 1.
Data 2021,6, 72 4 of 20
Table 1. Overview of key features of available datasets and the proposed benchmark dataset.
Dataset Sensor
Sampling Rate
# of Trials
Isolated Translations
Isolated Rotations
Different Velocities
Short and Long Motion Phases
Accelerometer Disturbances
Disturbed Magnetic Fields
Initial Rest Phase
GT Includes Position
GT Is Synchronized
GT Coordinate Frames Aligned
Reproducible Error Metrics Defined
RepoIMU T-stick [18] Xsens MTi 100 Hz 29 3337 77 7737 7
RepoIMU Pendulum [18] custom [19] 90–166 Hz a22 7 7 37 77 7737 7
Sassari [16] 3 models 100 Hz 3 73 3 777333 7b3
OxIOD [20] iPhone 100 Hz 132 7 7 3777333 7 7
EuRoC MAV [21] ADIS16448 200 Hz 6 c7 7 3737 7 3 3 3 7
TUM VI [22] BMI160 200 Hz 6 d7777 77 73 3 3 7
BROAD myon aktos-t 286 Hz 39 3 3 3 3 3 3 3 3 3 3 3
a
effective rate much lower, see description.
b
local frames aligned by precise placement, reference frames not aligned, see description.
conly counting the Vicon room trials with full OMC ground truth. donly counting the room trials with full OMC ground truth.
In order to allow for evaluation of different aspects of an IOE algorithm, a useful univer-
sal benchmark dataset should fulfill a number of requirements. First and foremost, it should
contain a large number of trials and a wide range of movements—including isolated trans-
lation and rotation movements—conducted at different speeds. To evaluate the robustness
against magnetic disturbances, it is crucial to include both data recorded in a magnetically
undisturbed environment and recordings with deliberate magnetic disturbances.
Furthermore, the quality of both the recorded IMU data as well as the ground truth
OMC data is essential. In order to evaluate the performance in state-of-the-art applications,
a state-of-the-art IMU with a sufficiently high sampling rate should be employed. Addition-
ally, care should be taken to avoid artifacts due to errors in the reference system or in the
recording of the IMU data. Figure 2shows four examples of artifacts found in the publicly
available datasets. The effects of such issues in the recorded data can often dominate the
overall estimation error. In the best case, this makes the resulting observations less distinct
and in the worst case, it could lead to wrong conclusions. Therefore, measurement data
should be carefully checked before it is used for evaluation.
As can be seen in Table 1, each of the previously published datasets covers some of
the mentioned aspects but none of them fulfill all previously mentioned requirements
for a universal IOE benchmarking dataset. In the following, we will discuss each dataset
in detail.
2.1. RepoIMU Dataset (T-Stick Trials)
To the best of our knowledge, the dataset RepoIMU [
18
] is, up to now, the only dataset
aimed at IOE evaluation with a dedicated publication. The dataset consists of two distinct
sets of trials, recorded with a T-stick and a pendulum.
The T-stick data consists of 29 trials with a duration of approximately
90 s
each. As
the name implies, the IMU is attached to a T-shaped stick equipped with six reflective
markers. Each trial consists of either slow or fast rotation around one primary sensor axis
or translation along one primary sensor axis. Data from an XSens MTi IMU and a Vicon
Nexus OMC system is synchronized and provided at 100 Hz.
Data 2021,6, 72 5 of 20
The authors explicitly state that the coordinate system of IMU and ground truth
are not aligned and propose a method to compensate one of the two required rotations
(
cf. Section 4.2
) by a method based on quaternion averaging. Unfortunately, some of the
trials contain gyroscope clipping (Figure 2a) and artifacts in the ground truth orientation
(Figure 2c) that have a significant effect on the obtained errors. Therefore, careful prepro-
cessing and exclusion of some trials should be considered when using the dataset for IOE
accuracy evaluation.
0 2 46
time [s]
−400
−200
0
200
400
angular rate [°/s]
(a) gyroscope clipping
x
y
z
0.0 0.1 0.2 0.3
time [s]
−10
−5
0
5
10
acceleration [m/s2]
(b) repeated samples
x
y
z
0246 8 10
time [s]
0
50
100
150
Euler angles [°]
(c) artifacts in ground truth (1)
z
y
x
0.0 0.2 0.4 0.6 0.8 1.0
time [s]
−100
0
100
Euler angles [°]
(d) artifacts in ground truth (2)
z
y
x
Figure 2.
Examples of artifacts found in existing datasets. (
a
) Gyroscope clipping leading to large
errors in angular rate strapdown integration. (
b
) Repeated samples in IMU data, leading to a very low
effective sampling rate. (
c
,
d
) Two examples of artifacts found in the OMC ground truth orientations,
potentially caused by interpolation of gaps and swapped markers.
2.2. RepoIMU Dataset (Pendulum Trials)
The second part of the RepoIMU datasets consists of data from a triple pendulum
on which IMUs are mounted. The measurement data is provided at
90 Hz
or
166 Hz
.
However, the IMU data contains frequently repeated samples, as shown in Figure 2b. This
is typically a result of artificial upsampling or transmission problems where lost samples
get replaced by copying the last received sample and effectively reduces the sampling rate.
The sampling rate that is obtained when repeated samples are discarded is around
25 Hz
and
48 Hz
for the accelerometer and gyroscope, respectively. Due to this fact, we cannot
recommend using the pendulum trials for high precision IOE accuracy evaluation.
2.3. Sassari Dataset
The dataset published in [
16
] is targeted to the validation of a parameter-tuning
approach based on the orientation difference of two IMUs of the same model. To facilitate
this, six IMUs from three manufacturers (Xsens, APDM, Shimmer) are placed on one
wooden board. Rotation around specific axes and free rotation around all axes are repeated
at three different speeds. The data is synchronized and provided at
100 Hz
. The local
coordinate frames are aligned by precise manual placement. The authors clearly describe
how they calculate the obtained error metrics, including a method of using the initial
orientation to align the reference frames.
Loading more pages...