EXPLORING DIMENSIONALITY REDUCTION EFFECTS IN MIXED REALITY FOR

ANALYZING TINNITUS PATIENT DATA

Burkhard Hoppenstedt(a), Manfred Reichert(b), Christian Schneider(c), Klaus Kammerer(d), Winfried Schlee(e),

Thomas Probst(f), Berthold Langguth(g), Rüdiger Pryss(h)

(a), (b), (d), (h) Institute of Databases and Information System, Ulm University

(c), (e), (g) Department of Psychiatry and Psychotherapy, University of Regensburg

ABSTRACT

In the context of big data analytics, gaining insights into

high-dimensional data sets can be properly achieved,

inter alia, by the use of visual analytics. Current

developments in the field of immersive analytics,

mainly driven by the improvements of smart glasses

and virtual reality headsets, are one enabler to enhance

user-friendly and interactive ways for data analytics.

Along this trend, more and more fields in the medical

patient data set by the use of dimensionality reduction

effects. The latter is represented by resulting clusters,

which are identified through the density of particles,

while information loss is denoted as the remaining

covered variance. Technically, the graphical interface of

the prototype includes a correlation coefficient graph, a

plot for the information loss, and a 3D particle system.

Furthermore, the prototype provides a voice recognition

feature to select or deselect relevant data variables by its

users. Moreover, based on a machine learning library,

in a better way. Altogether, the presented tool may

constitute a promising direction for the medical as well

as other domains.

Keywords: immersive analytics, dimensionality

reduction, mixed reality, covariance graph

1. INTRODUCTION

Recent developments of smart glasses offer new

perspectives in the field of immersive analytics. The

latter is a research field that investigates new display

technologies for analytical reasoning (Chandler 2015).

In many cases, augmented reality approaches use a

three-dimensional representation of data, which enables

the user to recognize spatial contexts of data more

easily.

In this context, Figure 1 presents our categorization of

different approaches in the field of augmented reality

for smart glasses. Note that there exists a variety of

from the real world by the use of a headset that

simulates an environment that is similar to the real

world. Second, assisted reality (ASR) constitutes the

concept of appliances (e.g., again headsets) for which

the augmented information is not directly in the user’s

field of view. Consequently, the augmented information

must be actively focused on to obtain further insights.

For example, an industrial maintainer is repairing a

machine and needs a clear field of vision. Though, he

should be able to check the current machine state with a

real world by using the concept of spatial mapping, also

denoted as 3D reconstruction (Izadi 2011). Hereby, a

room is scanned, usually by the use of depth-sensors,

and the resulting, generated model can be used as an

interface between holograms and the real world. Note

that this concept enables new interaction possibilities in

the context of immersive analytics as diagrams to be

analyzed can be placed nearly anywhere in the real

world. In this context, we discuss the following research

question along a high-dimensional data set of tinnitus

(Wold 1987). This Euclidean distance-based technique,

in turn, is often used for classification purposes in

combination with other approaches, such as neural

networks. Thereby, the PCA transfers all values into a

subdimension, which allows for displaying a three-

dimensional plot for data sets of arbitrary size.

However, since information can be lost in this

transformation process, our approach particularly

addresses this issue during the dimensionality reduction.

In addition, we focus on two other major aspects:

HoloLens, a head-mounted display for mixed-reality,

and the unity game engine (Technologies 2015). The

data set that is used for the prototype stems from the

TrackYourTinnitus platform (TYT). Note that the latter

and Android users to gather everyday life data with

their own smartphones to understand their individual

tinnitus situation better. Tinnitus can be described as the

phantom perception of sound. Note that symptoms for

tinnitus are subjective and vary over time. Therefore,

general treatment is challenging as well as costly.

Especially in the context of chronic disorders, a

comprehensive and quick access to patient data is of

utmost importance. On one hand, clinicians and

researchers want to obtain the required patient

information (e.g., what are the characteristic variables

of an individual patient) as quick as possible in order to

conduct studies with promising hypotheses or to start a

proper patient treatment. On the other, by sharing

information on patient data in a proper way,

efficiently. Therefore, the presented approach and the

developed prototype shall support clinicians and

researchers in this context.

The remainder of the paper is structured as

follows: Section 2 discusses related work, while Section

3 introduces the mathematical background for the

pursued dimensionality reduction. In Section 4, the

developed prototype is presented, in which the data set,

the Graphical User Interface (GUI), and the backend

are presented. Threats to validity are presented in

loss of quality quantification (Gracia 2016), the authors

found that three-dimensional visualizations are superior

compared to two-dimensional representations. The

authors compared the tasks point classification, distance

perception, and outlier identification in two ways. First,

they evaluated a visual approach and, second, they

applied an analytical counterpart. Furthermore, they

conducted a user study and compared 2D and 3D

scenarios on a display. However, they did not use smart

glasses to evaluate their models. A second user study

theory of improved performance in a three-dimensional

space (Arms 1999), compared 2D and 3D visualizations

by using interaction (i.e., time measurement) and

visualization tests (i.e., correct identification). The

subjects were asked to identify clusters, to determine

the dimension of a dataset, and to classify the radial

sparseness of data. Similar to our work, a prototype for

dimensionality reduced scatterplots was developed and

examined in (Wagner Filho 2017). The subjects had to

identify the closest party, party outliers, and the closest

dimensionality reduction, was tested in an exceptional

project called Be the Data (Chen 2016). Persons were

embodying by data points, while the floor represents a

2D projection. This idea relies on findings, where

bodily experiences, such as gesturing, body orientation,

and distance perception support the cognitive process

(Bakker 2011). Note that the concepts of bodily

experiences are an important part of mixed reality and,

hence, can be associated with our work.

The Microsoft HoloLens was profoundly evaluated in

visualization for high-dimensional data is described as

“a cognitive bottleneck on the path between data and

discovery”.

Altogether, the introduced literature shows the potential

of immersive analytics, though indicate potential

weaknesses in our pursued context.

3. PRINCIPAL COMPONENT ANALYSIS

The principal component analysis (PCA) is a technique

to find patterns in high-dimensional data. Common use

, (1.2)

follows:

at a certain point in time. In a first preprocessing step,

the data set was cleaned from missing values, which

might occur if the data is stored incompletely due to

missing user inputs or errors caused by the used smart

mobile devices (cf. Table 1). Next, each column is

normalized to ensure comparability between the

dimensions. However, we lose information about the

absolute values of each dimension on one hand. On the

other, a uniform representation for three dimensions

becomes possible (cf. Figure 2).

REQ1: High-dimensional data needs to be displayed

and for existing clusters it should be easily possible to

identify them.

REQ2: A simple data representation is essential since

the application users are not necessarily data science

experts.

REQ3: The relation between the data sets dimensions is

a core function and needs to be displayed using a quick

overview feature.

REQ6: Due to the complexity of the data set, the user

needs precise application feedback and easy input

possibilities during the data analysis workflow.

4.2. The HoloLens

The HoloLens offers a variety of sensors to improve the

user interaction and user feedback (cf. Table 2). The

Inertial Measurement Unit (IMU) contains a

combination of accelerometers and gyroscopes, which

stabilize the visualization of holograms by providing the

user commands and ambient noise. Furthermore, due to

the microphone array’s positioning, the identification of

the direction of external sounds is easily possible.

from different directions, based on the user’s relative

position to a virtual object. This can be used to guide

the user through a room and direct his field of view to

relevant diagrams or information.

infinite projections are not possible, neither to the actual

distance nor to the actual proximity. Moreover, the

HoloLens offers gesture- and gaze recognition. In our

work, we solely utilize the tap-to-place-interaction via

gestures (cf. Figure 3).

With a weight of 579g, the HoloLens usually needs a

longer period for getting familiar with the appliance.

The first introduced graphical component is a particle

system as shown in Figure 4. Most importantly here is

the increasing brightness for particles in the same

position as configured by a shader. This effect

simplifies the detection of clusters as regions with a

high particle density appear brighter than those with

only few contained data points (cf. REQ1).

Furthermore, the particle system is labeled with the

corresponding dimension name on each axis. It is

possible to plot the same variable on several axes. When

Voice commands allow for the plot manipulation, where

a hologram scalation by predefined values can be

realized using the keywords plus and minus. As

introduced in Section 4.2, a natural zoom by

approaching the hologram is only possible to 60cm,

therefore the scalation of the hologram replaces this use

Furthermore, the variable assignment to each axis can

be edited using voice commands and the resulting

changes in the plot are animated, so that the user can

voice input. Still, the voice commands are intuitive and

fulfill REQ6.

The number of variables to be displayed is further on

denoted as variables collection. All items in the

variables collection are shown next to the particle

system, as well as in a correlation coefficient graph (cf.

Figure 5). The latter is a variant of an existing approach

(Peña 2013) using the concept of color coding. Negative

of each color, where the covariance intervals [0,1] and

[-1,0] are mapped to the new opacity value in the range

[0,100%]. In Table 3, therefore, a sample correlation

matrix for five variables is shown. Note that, although

the covariance is used for the PCA calculation, we

visualize the correlation as a normalized form of the

The correlation is denoted as

As can be obtained from Eq. 1.4, covariance and

correlation depend on each other.

Figure 5 shows the resulting graphs based on Table 3.

The covariance plot is solely shown to underline the

difference to the graphical variant. The covariance

graph marks only the strongest edges, while the

introduced correlation graph fades irrelevant values.

The user of the prototype can obtain this information

from the graph to improve the dimensionality reduction

by removing variables that don’t fit well into the graph;

caused by the Principal Component Analysis. A bar plot

shows the percentage of the three most important

components for the overall variance. Figure 6 presents

the variance of each component in a stacked bar. Due to

the transparency, the user is able to recognize the

importance of each component, while the red cube

represents the discarded information.

Figure 6: Information Loss Component

features. Second, the particle system allows for the

visualization of high-dimensional data and a simplified

detection of clusters through the brightness. Finally, the

stacked bar of the PCA components variance allow for a

quick estimation of each component’s importance and

the information loss.

4.4. Backend

The core concept of this application is to separate the

algorithm implementation from the visualization to

using the Web Server Gateway Interface (WSGI).

Moreover, the PCA implementation is realized by the

free machine learning library scikit-learn (Pedregosa

2011), and the numerical and scientific library NumPy

(Walt, Colbert, and Varoquaux 2011). The pursued

workflow, in turn, is as follows: Via voice commands,

variables can be selected or deselected from the data set,

which is stored on the server.

Based on the number of selected variables n, and the

number of entries in the data set m, a matrix is

generated. This matrix serves as the input for the PCA.

components. First, to receive a three-dimensional

reduced data set, only the three components

representing the highest variances are used. The original

data set can now be transformed into the new subspace.

Second, the PCA is computed with the maximal number

of components to show the distribution of components

concerning their variance. The mixed reality application

is then able to access - via a REST call - the computed

variance ratio vector and the transformed data set.

installation and therefore intuitiveness of the

application. On one hand, the presented application

shall enable simplified insights into methods of

dimensionality reduction, which could be technically

shown. On the other, the developed application design

for the backend and its required installation procedure

are currently inappropriately designed for large-scale

practical scenarios. Moreover, the need for an Internet

connection disqualifies the current approach for local

working environments like the ones that can be found in

values in the particle system are normalized, which

distorts the impression of the real range of values and

instead only represents a relative view on the data set.

Finally, the prototype has not been evaluated in a

psychologic study yet. Therefore, amongst others,

insights on the cognitive load for users when using this

application in practice is currently unexplored and must

be evaluated in an empirical study.

function overview with respect to the major goal

pursued by a PCA and can therefore be used for

immersive analytics in the context of large-scale

healthcare data like the one shown for tinnitus patients.

We regard such technical opportunities especially in the

context of healthcare scenarios as an enabler to analyze

patient data more efficiently. However, many other

domains crave for such appliances that can be used to

perform immersive analytics (e.g., in the context of

predictive maintenance).

study, in which the users need to solve cluster-based

tasks, such as the assignment of occurring data patterns

to the correct cluster and interpreting the effect on a

cluster when removing a dimension. The study is

compression using PCA and backpropagation

learning. pattern recognition, 33(9), 1555-1560.

Donalek, C., Djorgovski, S. G., Cioc, A., Wang, A.,

Zhang, J., Lawler, E., Yeh, S., Mahabal, A.,

Graham, M., Drake, A., Davidoff, S., Norris, J. S.,

& Longo, G., 2014. Immersive and collaborative

data visualization using virtual reality platforms.

Big Data (Big Data), 2014 IEEE International

Conference on, 609-614.

Evans, G., Miller, J., Pena, M. I., MacAllister, A., &

Von Landesberger, T., 2016. New insights into the

suitability of the third dimension for visualizing

multivariate/multidimensional data: A study based

on loss of quality quantification. Information

Visualization, 15(1): 3-30.

Grinberg, M., 2018. Flask web development:

developing web applications with python. O'Reilly

Media, Inc.

Hoffman, K., & Kunze, R., 1971. Characteristic values

in linear algebra. Prentice-Hall, New Jersey.

on User interface software and technology, 559-

568.

LaValle, S., 2013. Sensor Fusion: Keeping It Simple.

Available from:

https://developer.oculus.com/blog/sensor-fusion-

keeping-it-simple/ [accessed 12 May 2018]

Milgram, P., & Kishino, F., 1994. A taxonomy of

mixed reality visual displays. IEICE

TRANSACTIONS on Information and Systems,

77(12), 1321-1329.

Raja, D., Bowman, D., Lucas, J., & North, C., 2004.

Schlee, W., Pryss, R. C., Probst, T., Schobel, J.,

Technologies, U., 2015. Unity - Manual: Unity Manual.

Van Der Maaten, L., Postma, E., & Van den Herik, J.,

Wagner Filho, J. A., Rey, M. F., Freitas, C. M., &

Wold, S., Esbensen, K., & Geladi, P., 1987. Principal

Yang, J., Zhang, D., Frangi, A. F., & Yang, J. Y., 2004.

AUTHORS BIOGRAPHY

Burkhard Hoppenstedt holds a M. Sc. in Computer

Science from Ulm University and his research addresses

aspects on big data analytics, especially with the focus

on predictive maintenance in an industrial context.

Klaus Kammerer holds a M. Sc. in Computer Science

from Ulm University. Next to his research on process-

Regensburg and also Head of the multidisciplinary

Tinnitus Clinic of the University of Regensburg.

Thomas Probst holds a PhD in psychology from the

Humboldt University Berlin. His research focuses on

the development and evaluation of IT-based systems for

the diagnostic-therapeutic process. In 2017, he has been

appointed as a full professor at the Danube University

Krems.

Rüdiger Pryss holds a Diploma in Computer Science.

After his studies, Rüdiger Pryss worked as consultant

appointed as full professor at the University of Ulm,

where he is director of the Institute of Databases and

Information Systems (DBIS).

Winfried Schlee holds a PhD in clinical

neuropsychology, where he introduced the concept of

the Global Model of Tinnitus Perception. In 2013, he

work focuses on discovering new methods for the

treatment and measurement of chronic tinnitus.

Christian Schneider holds a M.Sc. from the University