Wha You See Is Wha You Can Change:
Human-Cen e ed Machine Lea ning By In e ac i e Visualiza ion
Dominik Sachaa,∗
, Michael Sedlmai b, Leishi Zhangc, John A. Leed, Jaakko Pel onene, Daniel
Weiskop , S ephen C. No hg, Daniel A. Keima
aUni e si y o Kons anz, Ge many
bUni e si y o Vienna, Aus ia
cMiddlesex Uni e si y, UK
dUni e si ´e ca holique de Lou ain, Belgium
eAal o Uni e si y and Uni e si y o Tampe e, Finland
Uni e si y o S u ga , Ge many
gIn o isible, Oldwick NJ, USA
Abs ac
Visual analy ics (VA) sys ems help da a analys s sol e complex p oblems in e ac i ely, by in e-
g a ing au oma ed da a analysis and mining, such as machine lea ning (ML) based me hods, wi h
in e ac i e isualiza ions. We p opose a concep ual amewo k ha models human in e ac ions
wi h ML componen s in he VA p ocess, and ha pu s he cen al ela ionship be ween au oma ed
algo i hms and in e ac i e isualiza ions in o sha p ocus. The amewo k is illus a ed wi h se e al
examples and we u he elabo a e on he in e ac i e ML p ocess by iden i ying key scena ios whe e
ML me hods a e combined wi h human eedback h ough in e ac i e isualiza ion. We de i e i e
open esea ch challenges a he in e sec ion o ML and isualiza ion esea ch, whose solu ion should
lead o mo e e ec i e da a analysis.
Keywo ds: Machine Lea ning, In o ma ion Visualiza ion, In e ac ion, Visual Analy ics
1. In oduc ion
Real-wo ld da a analysis usually elies hea ily on bo h au oma ic p ocessing and human expe -
ise. Da a size and complexi y o en p eclude simply looking a all he da a, and make machine
lea ning (ML) and o he algo i hmic app oaches a ac i e, and e en ine i able. Howe e , he
powe o ML canno be ully exploi ed wi hou human guidance. I emains a challenge o ansla e5
∗Co esponding au ho
Email add ess: [email p o ec ed] (Dominik Sacha)
P ep in submi ed o Neu ocompu ing. To appea unde DOI: 10.1016/j.neucom.2017.01.105 Ap il 28, 2017
This is an accep ed manusc ip . The o iginal a icle has been published in Neu ocompu ing. 2017,
ol. 268, pp. 164-175. DOI:h ps://dx.doi.o g/10.1016/j.neucom.2017.01.105.
eal-wo ld phenomena and analysis asks, which a e o en unde -speci ied, in o ML p oblems. I is
di icul o choose and apply app op ia e me hods in di e se applica ion domains and asks. Mo e
impo an ly, i is c ucial o be able o inco po a e he knowledge, insigh , and eedback o human
expe s in o he analy ic p ocess, so ha models can be uned and hypo heses e ined.
In a ypical se ing, domain expe s use ML and isualiza ion me hods p o ided by common10
so wa e ools (e.g., SPSS, R, Tableau) “ou o he box”. Realis ically, he domain expe s’ p o-
iciency in ML may be limi ed, and he unde lying compu a ions may no be anspa en and
comp ehensi e enough o p o ide he eedback needed o guide model e inemen . Visualiza ions
a e o en used o display he ML model esul s wi hou o e ing in e ac ions ha igge ecalcula-
ions. This esul s in a e y s anda dized con igu a ion o he ML and isualiza ion pipelines based15
on de aul pa ame e s ha domain expe s may no know how o adap . The si ua ion may be
imp o ed by ha ing domain expe s collabo a e wi h da a scien is s, imp o ing he e ec i eness o
analysis, bu also leading o a much mo e cos ly i e a i e design p ocess. ML esea che s usually
know how o une models di ec ly in ML pla o ms (e.g., Ma lab, R, Py hon) and p o ide esul s o
domain expe s. Howe e , domain expe s gene ally ind i necessa y o lea n how models beha e20
and how o e alua e esul s o p o ide use ul eedback.
By in eg a ing ML algo i hms wi h in e ac i e isualiza ion, isual analy ics (VA) aims a p o-
iding isual pla o ms o analys s o in e ac di ec ly wi h da a and models [1]. Tam e . al [2]
illus a ed in case s udies ha human-cen ic ML can p oduce be e esul s han pu ely machine-
cen ic me hods. In such cases, an analys is enabled o s ee he compu a ion and in e ac wi h25
he model and da a h ough an in e ac i e isual in e ace. Despi e much e o o da e, hough,
solu ions om ML and VA a e s ill no in e wo en closely enough o sa is y he needs o many
eal-wo ld applica ions [3, 4]. Fo example, in exis ing oolki s (such as WEKA, Elki, o ja aML),
igh in eg a ion be ween in e ac i e isualiza ion and ML p ocess is missing. Mos o hese ools
p esen modeling esul s as s a ic isualiza ions; in e ac ions a e o en limi ed o command line in-30
e aces o use in e ace con ols ha a e no in ui i e and accessible o end-use s. Towa d be e
in eg a ion o ML and VA, in ecen yea s concep ual amewo ks ha cha ac e ize he in e play
be ween hem ha e been p oposed [1, 3, 4, 5]. I appea s mos amewo ks we e designed om he
pe spec i e o in e ac i e isualiza ion, ocusing on he ole o he “human in he loop”. A close
connec ion be ween isualiza ion and common ML pa adigms (such as unsupe ised and (semi-)35
supe ised lea ning; classi ica ion, eg ession, clus e ing, e c.) including speci ics o hese me hods
2
(e.g., SVM s. andom o es s in classi ica ion) and hei implemen a ions is needed. In his pape ,
we pu a sha pe ocus on scena ios in which complemen a y ML and VA me hods a e combined,
and p opose a amewo k o a igh e ela ionship be ween ML and VA. To do so, we iden i y
aspec s o au oma ed ML echniques ha a e amenable o in e ac i e con ol, and illus a e hese40
wi h examples. We u he desc ibe human ac o s wi hin his p ocess ha should be conside ed
ca e ully in he design o in e ac i e isual ML sys ems, and enume a e analysis scena ios. The
p oposed concep ual amewo k opens pe spec i es on new ways o combining au oma ed and in-
e ac i e me hods, which will lead o be e in eg a ed, and, ul ima ely, mo e e ec i e da a analysis
sys ems.45
Resea che s in bo h ML and isualiza ion ha e ealized o some ime ha close collabo a ion
could help o sol e his p oblem. An in e disciplina y eam wi h expe s om he ML and isualiza-
ion communi ies was o med a a Dags uhl Semina on “B idging In o ma ion Visualiza ion wi h
Machine Lea ning” [6]. The amewo k p oposed in his s udy is he ou come o se e al i e a ions
o discussions, eedback, and amewo k e inemen s made by his eam. The ini ial e sion o his50
amewo k [6] was based on a su ey o se e al ea lie amewo ks and sys ems combining ML and
in e ac i e isualiza ion. Subsequen ly, he amewo k was e ined by applying i o a la ge se o
example applica ions (iden i ied in he isualiza ion, ML, and HCI li e a u e) and by inco po a ing
ex e nal eedback om expe s, such as con e ence submission e iews. This led us o a p ocess
o amewo k e inemen , ca ied ou o e 1.5 yea s, including ex ensions and simpli ica ions, ali-55
da ion, and e alua ion by analyzing exis ing VA sys ems and ML echniques. This pape ex ends
an ini ial epo in ESANN 2016 [7] o include an ex ended e iew o p io wo k, a mo e de ailed
amewo k, examples o p o iding au oma ed suppo o each s age, iden i ica ion and desc ip ion
o scena ios whe e analysis and eedback ake place, and addi ional discussion h oughou .
The es o his pape is s uc u ed as ollows. Sec ion 2 discusses ela ed wo k on he in e play60
o machine lea ning and human eedback. Sec ion 3 in oduces ou concep ual amewo k and
he key s ages in i s in e ac i e pipeline, and hey a e illus a ed wi h examples in Sec ion 4.
Sec ion 5 examines he human in e ac ion loop in mo e de ail, desc ibing he s ages o ac ion
and analysis scena ios whe e in e ac ion occu s. Sec ion 6 iden i ies i e challenges and associa ed
oppo uni ies in c ea ing sys ems ha ully use he amewo k. Sec ion 7 ga he s conclusions and65
inal discussions.
3
Figu e 1: Visual Analy ics F amewo k by Keim e al. [1]. ML in e ac ions a e ela ed o model building and pa ame e
e inemen .
2. Rela ed Wo k
The li e a u e desc ibes ela ed models ha cap u e he in e play be ween ML sys em com-
ponen s and human eedback loops. We will discuss se e al di e en pe spec i es on his opic,
di ided in o VA models, in e ac ion axonomies, in e ac i e ML, and human-cen e ed design. This70
sec ion concludes wi h a high-le el summa y o in e es ed eade s wi hou ML expe ise.
Visual Analy ics Models.Pipeline-based models such as he Re e ence Model o In o ma ion
Visualiza ion [8] o he Knowledge Disco e y P ocess in Da abases (KDD) [9] usually con ain eed-
back loops ha co e all he subcomponen s wi h he po en ial o use in e ac ion. In he s anda d
VA model [1], he analysis p ocess is cha ac e ized by in e ac ions be ween da a, isualiza ions,75
models o da a, and use s, o knowledge disco e y (see Figu e 1). ML in e ac ion in his ame-
wo k is aimed a model building and pa ame e e inemen . Sacha e al. ex ended his model [4]
o encompass he p ocess o human knowledge gene a ion. This ex ended model cla i ies he ole
o humans in knowledge gene a ion, and highligh s he impo ance o suppo ing igh e in eg a-
ion o human and machine. Se e al o he models ocus on a clea depic ion o he human da a80
analysis p ocess, including Pi olli and Ca d’s sensemaking p ocess [10], and Pike e al.’s science o
in e ac ion [11]. Ende e al. cha ac e ized he in e ac ion p ocess be ween a human analys and
4
au oma ed analysis echniques as he “human is he loop” [12] and p oposed a model o coupling
cogni ion and compu a ion [3]. Mo e ecen ly, Chen and Golan [13] p o ided an abs ac model
o desc ibe six classes o human-machine wo k lows in combina ion wi h an in o ma ion- heo e ic85
measu e o cos -bene i . Thei model allows one o analyze wo k lows composed o machine compu-
a ions and human in e ac ions suppo ed by di e en “le els” o isualiza ions. All hese models
e lec a high-le el unde s anding o sys em and human concep s.
In e ac ion & Task Taxonomies.Ano he se o models ela ed o ou endea o seek o cha -
ac e ize and o ganize he asks and in e ac ions in a isual da a analysis p ocess. Fo example,90
B ehme and Munzne [14] p opose a comp ehensi e isualiza ion ask axonomy. Howe e , model
in e ac ions only a ise in asks hey e e o as “agg ega e” o “de i e” asks. Landesbe ge e
al. [15] de ine a axonomy ha includes in e ac ion and da a p ocessing. Thei axonomy p o-
ides wo ypes o da a p ocessing in e ac ions: da a changes, such as edi ing o selec ing da a,
and p ocessing changes, such as scheme o pa ame e changes. They inco po a e Be ini and95
Lalanne’s [16] dis inc ion o human in e en ion le els, ha dis inguishes, o example, be ween
scheme uning (e.g., pa ame e e inemen ) and scheme changing (e.g., changing he model) in e -
ac ions. M¨uhlbache e al. [17] in es iga e and ca ego ize se e al ypes o use in ol emen o black
box algo i hms wi h di e en cha ac e is ics. The cha ac e iza ion o in e ac ions in ou amewo k
is o hogonal o hese axonomies and ex ends hem wi h a dedica ed iew on in e ac ion wi h ML100
componen s.
In e ac i e Machine Lea ning.While he abo e models we e s ongly amed om he iew-
poin o isualiza ion and VA, he e is also g owing in e es in he ML communi y o inco po a e
human in e ac ion mo e ully in he analysis and lea ning p ocesses. A ypical scena io would be
ha a human obse es o explo es he cu en s a e o a lea ning sys em, and explici ly o implic-105
i ly guides an ongoing aining p ocess. This is o en e e ed o as human-in- he-loop machine
lea ning, o mo e b oadly, in e ac i e machine lea ning. The classical example is ecommende
sys ems ha in e use s’ pe sonal p e e ences om p e ious choices. Use can p o ide con inuous
eedback, such as by eco ding addi ional choices, o by explici ly sco ing (liking/disliking) indi-
idual i ems [18, 19, 20]. In simila scena ios, he use p o ides class labels, o which a machine110
lea ning classi ie is (con inuously) ained. This concep is s ongly linked o he opic o ac i e
lea ning [21, 22], which aims a e icien choices o samples du ing aining epochs o achie e as
5

Figu e 2: S ages o in e ac ion [24].
con e gence o a lea ning algo i hm. An inc eased demand o in ol ing use eedback is unde sco ed
by ecen wo k in es iga ing mo e elabo a e use models o mo e in ica e o ms o in e ac ion. Fo
example, Ame shi e al. [5] p opose a se o high-le el pa adigms by which use in ol emen in ML115
may be cha ac e ized, simila o models ha ha e been discussed in he VA communi y [1]. They
also s ess he impo ance o accoun ing o use beha io , and he po en ial bene i s o collabo a i e
esea ch be ween ML expe s and he human-compu e in e ac ion communi y. Simila ly, G oce
and colleagues in es iga e sample selec ion s a egies o es classi ie s e ec i ely ia sys ema ic
eedback eques s o end use s [23].120
Human-Cen e ed Design.Ano he pe spec i e on he analysis p ocess is p o ided by he human-
compu e in e ac ion (HCI) domain. We a e able o adop commonly known concep s and e ms
in ou in e ac i e ML se ing, conside ing he in e play be ween human and machine. A amous
example is No man’s S ages o Ac ion cycle [24], shown in Figu e 2. A he cen e o he cycle
a e he Goals ha a human analys wan s o achie e. No man dis inguishes be ween wo majo 125
s ages o an in e ac ion: Execu ion and E alua ion. In he Execu ion phase, he human (1) o ms
an in en ion o ac and (2) speci ies a sequence o ac ions ha is (3) inally execu ed o he wo ld.
Subsequen ly, in he E alua ion phase, he s a e o he wo ld has o be (1) obse ed, (2) in e p e ed,
and (3) inally compa ed and e alua ed wi h espec o he ini ial goals. No man u he desc ibes
6
he dis ances o “gul s” be ween he human goals and he wo ld ha need o be b idged when130
humans in e ac wi h (digi al) in e aces. The Gul o Execu ion desc ibes he p oblem when he
human does no know how o pe o m an ac ion, whe eas he Gul o E alua ion indica es ha
humans a e no able o e alua e he esul o an ac ion. Human-cen e ed use in e ace design a -
emp s o b idge hese gaps. Use in e ace ea u es need o be isible and o e A o dances o he
end use . These (pe cei ed) a o dances a e ela ionships be ween a pe son and a physical/digi al135
objec , and sugges how he objec migh be used [24]. Visual Cues (e.g., isual elemen s, icons, o
anima ions ha a ac a en ion) may guide he end use du ing he analysis p ocess. On he one
hand, a sys em should communica e he p og ess o ongoing compu a ions o he quali y o esul s
o he analys . On he o he hand, isual cues may guide he analys o “handles” o objec s ha
can be manipula ed wi hin he in e ace. In his espec , he concep o di ec manipula ion [25]140
has been demons a ed o enable in ui i e ope a ions o end use s. In e ac i e isualiza ions o ML
model s uc u es and da a i ems allow di ec in e ac ion ha is mo e con enien and mo e easily
in e p e ed han ex commands.
Machine Lea ning O e iew.A wide ange o ML algo i hms and me hods ha e been p o-
posed and employed in p ac ice. One way o dis inguish hese me hods is based on hei lea ning145
pa adigm, which is ei he supe ised (examples o sys em inpu s and desi ed ou pu s a e bo h
p o ided), unsupe ised (no desi ed ou pu s a e speci ied), o semi-supe ised (no all ou pu s a e
a ailable, ypically only a ew). While supe ised me hods aim o lea n he inpu -ou pu ela ion-
ship om he p o ided examples, unsupe ised me hods a emp o ex ac hidden s uc u es om
hem. These lea ning pa adigms can be ins an ia ed in o speci ic ca ego ies o ML asks. Reg es-150
sion aims o bes p edic any o m o con inuous ou pu s as a unc ion o he inpu s. Classi ica ion
aims o p edic class labels o membe ships associa ed wi h he inpu s. On he unsupe ised side,
clus e ing aims o iden i y g oups o hie a chies p esen in da a. Simila ly, dimensionali y educ ion
and mani old lea ning bo h aim o iden i y linea o nonlinea ela ionships be ween he obse ed
a iables and o ep esen he subspace whe e mos o he da a a ia ion happens wi h ewe la en 155
a iables. These a e a ew examples o emblema ic ML asks, among many o he s, like associa-
ion ule lea ning, missing alue impu a ion, ime se ies p edic ion and ou lie de ec ion, no el y
de ec ion. In p ac ice, se e al o hese abs ac asks a e combined in a da a low o sol e eal-
wo ld p oblems and analysis. Fo example, one migh apply dimensionali y educ ion ( o mi iga e
7
he compu a ional impac o wo king wi h high dimensional da a) be o e applying eg ession o 160
ime se ies p edic ion. Such combina ions o me hods, hough use ul in p ac ice, lead o composi e
models wi h he e ogeneous pa ame e iza ion, which a e di icul o ain, and ime-consuming o
alida e.
In summa y, bo h VA and ML communi ies ha e no iced he gaps be ween au oma ic ML165
and human in e ac ions in da a analy ics sys ems, which limi hei e ec i eness in sol ing eal
wo ld applica ion p oblems. Va ious models o concep ualize he po en ial in eg a ion o ML and
in e ac i e isualiza ions ha e been p oposed. These models, howe e , s ill ha e ei he a s ong
human/ isualiza ion ocus, o a s ong algo i hmic ocus. In his pape we p opose a new concep ual
amewo k ha co e s bo h aspec s wi h he objec i e o p o iding a mo e sys ema ic iew o how170
in e ac i e isualiza ion and ML algo i hms can be in eg a ed in p ac ice.
3. Human-Cen e ed Machine Lea ning F amewo k
As shown in Figu e 3, ou amewo k uni ies, embeds, and ex ends exis ing heo ies on in e ac i e
ML and VA by in eg a ing and gene alizing obse a ions om eme gen case s udies and examples.
The amewo k combines ypical ML and VA pipeline componen s (A–D) wi h an analys s’ i e a i e175
e alua ion and e inemen p ocess (E). An analys can in e ac wi h he indi idual s ages in his
pipeline h ough a isual in e ace (D), which ac s as a media o o “lens” be ween he human
and he ML componen s (dashed a ows). Changes a e hen sen back o he isual in e ace and
p esen ed o he analys (solid a ows). The da k blue boxes in he igu e deno e examples o
au oma ed me hods ha suppo he analys in pe o ming speci ic in e ac ions. Ou amewo k180
illus a es ha a mul idisciplina y pe spec i e combining ML and VA is needed o p o ide usable
and accessible access o end-use s (domain expe s). Fo example, da a ope a ions, isualiza ion
echniques, and human-compu e in e ac ion (blocks A,D, and Ein Figu e 3) a e add essed in he
isualiza ion communi y, whe eas ML algo i hms, se ups, and op imiza ion (blocks Band C) a e
co e o ML esea ch. Nex , we de ail he in e ac ions in ol ed in each s age o he analysis p ocess,185
and discuss possible au oma ic suppo o acili a e hese in e ac ions.
Edi s & En ichmen (A).In ML, da a is o en seen as ixed o immu able, bu many VA ools
suppo da a cleaning, w angling, edi ing, and en ichmen , which is essen ial in many applica ions
8
Figu e 3: P oposed concep ual amewo k: A e e ence in e ac i e VA/ML pipeline is shown on he le (A–D),
complemen ed by se e al in e ac ion op ions (ligh blue boxes) and exempla y au oma ed me hods o suppo in-
e ac ion (da k blue boxes). In e ac ions de i e changes o be obse ed, in e p e ed, alida ed, and e ined by he
analys (E). Visual in e aces (D) a e he “lens” be ween ML models and he analys . Dashed a ows indica e whe e
di ec in e ac ions wi h isualiza ions mus be ansla ed o ML pipeline adap a ions. The colo s o he pipeline
componen s e e o he ones de ined in he VA p ocess model [1] shown in Figu e 1.
[26]. Fo example, in a ypical ac i e lea ning scena io in ML, a domain expe may wan o
inc emen ally add labels o da a while aining a classi ie in o de o injec domain knowledge190
and imp o e he quali y o he classi ie . Ano he example o Edi s & En ichmen in e ac ion is
he es ing o “wha -i scena ios” on he da a. The analys migh wan o change o emo e some
da a poin s and see he e ec o es ce ain assump ions abou he da a. Da a edi ing is o en
ollowed by a “wa m es a ” o he ML pipeline, i e a i ely p opaga ing esul s o he analys .
F om an ML pe spec i e, da a edi ing combined wi h use eedback can be seen as a o m o c oss-195
alida ion/boo s ap. In hese echniques, he ML model is e- ained wi h “modi ied” da a, ei he
da a held ou in c oss- alida ion and lea e-one-ou , o changed in o some o he obse ed ins ance
ha is hen ein oduced in he boo s ap sample. Howe e , adi ional c oss- alida ion/boo s ap
a e pe o med wi h s ic ules abou how da a a e held ou o modi ied, o pu sue s a is ical goals
abou gene aliza ion pe o mance, whe eas he edi ing and eedback discussed he e a e pe o med200
by he use o ca y ou a ask, no cons ained o a speci ic ma hema ical o maliza ion o he
ask. Hence da a edi ing migh be seen as a kind o “me a” c oss- alida ion, equi ing p ope
quali y assessmen o he use ’s ask.
Au oma ic Suppo : Va ious s a is ical and ML echniques exis o p econdi ioning and p o-
9
and ba cha ), he app oach gi es analys s a con enien way o explo ing al e na i e con igu a ions
o p ep ocessing s eps (E).
360
The pa i ion-based amewo k by M¨uhlbache and Pi inge [35] (Figu e 4-b) p o ides he
analys wi h di e en p e-buil eg ession model a ian s ha can be selec ed and e ined (C). The
sys em isualizes hese eg ession models in a ma ix iew in combina ion wi h u he ea u es
and quali y in o ma ion (D). The analys can apply ea u e- ans o ma ions and o e s p epa a ion
pa ame e s ha a e used o ea u e pa i ioning (B). All in e ac ions igge ecalcula ions o he365
eg ession model a ian s and quali y me ics enable he analys o e alua e hese changes (E).
BoababView [37] (Figu e 4-d) isualizes he s uc u e o a decision ee in combina ion wi h
da a, ea u e, and quali y in o ma ion (D). The analys can pe o m ee ope a ions (e.g., spli o
me ge nodes) and adap speci ic pa ame e s, such as spli poin s alues (C). The analys can inspec 370
and ollow changes o he da a low wi hin he ee, and e alua e he p ecision o he classi ie a
any s age (E).
EnsembleMa ix [42, 46] le s analys s build classi ie ensembles by disco e ing se e al com-
bina ion s a egies (C). The sys em o e s se e al con usion ma ix isualiza ions o each classi ie 375
wi h a combined main ma ix, as well as a linea classi ie combina ion iew (D). An ex ension o
he ool u he measu es he accu acy o an ensemble classi ie be o e and a e use in e ac ion o
suppo analys s in hei e alua ion p ocess (E).
Voyage [47] p o ides he analys wi h isual ecommenda ions o ace ed b owsing h ough a380
se ies o au oma ically gene a ed isualiza ions (D) ha ma ch he unde lying da a’s cha ac e is ics
wi h use p e e ences.
DimS ille [48] allows analys s o design and alida e se e al s eps o a dimensionali y e-
duc ion wo k low (A–D). This app oach makes i possible o compa e al e na i e wo k lows and385
alida e each s ep o he ML pipeline o iden i y which phases can be imp o ed (E).
16

Table 1: Examples g ouped by amewo k componen s (A–E).
Pa Examples
(A) Adding class labels [43], adjus ing & emo ing da a [34], adding ex ual anno-
a ions [36], sugges ing ans o ma ions [44]
(B) Re-a ange da a poin s [45], dimension loadings [34], e m weigh ings [36],
p epa a ion pa ame e s [35]
(C)
Pa ame e uning [43], making model selec ions [35], building ensemble clas-
si ie s [46], de ining cons ain s in a o ce-di ec ed layou [36], ee ope a ions
[37]
(D)
Mul iple linked iews [34, 43], 2D-spa ializa ion wi h di ec in e ac ions [36, 45],
p e-buil ML a ian s [35], ee- isualiza ions [37], (con usion) ma ix [34, 37,
46], b owsing isualiza ions [47]
(E) Responsi e isualiza ion upda es [34, 35, 37, 36, 45], ML wo k low design [48],
measu ing in e ac ion quali y [42]
5. Human-Cen e ed Machine Lea ning Loop
In his sec ion, we ocus mo e closely on he analys ’s eedback loop, desc ibed as an i e a i e
cycle o Execu ions and E alua ions (shown in Figu e 3-E and in mo e de ail in Figu e 5). This
loop “connec s” an analysis sys em and an analys whose abili y o p o ide eedback depends on390
indi idual ac o s, as well as he isual in e aces o he sys em. We will ou line indi idual human
ac o s o he analys , and elabo a e on po en ial s ages o ac ion in mo e de ail (Figu e 5-le ).
Subsequen ly, we enume a e se e al analysis scena ios o illus a e ypes o eedback ha can be
p o ided by an analys (Figu e 5- igh ).
5.1. S ages o Ac ion395
We adop No man’s S ages o Ac ion [24] o dis inguish wo phases wi hin he human-in- he-
loop model, shown in Figu e 5. This model desc ibes he in e play and collabo a ion be ween he
sys em and he analys . While he p e ious sec ion desc ibed de ails o he analysis sys em, we
now shi ou ocus o he analys , and desc ibe he phases o Execu ion and E alua ion in mo e
de ails.400
17
Figu e 5: Human-cen e ed ML p ocess loop shown in mo e de ail. The loop e eals impo an cha ac e is ics o
analy ic ac i i y; he igh hand side shows di e en analysis scena ios.
The Analys .The analys o ms goals based on indi idual p io knowledge, and assump ions
abou he da a/ isual in e ace. The abili y o analys s o p o ide eedback depends on ac o s
such as echnical compe ence (e.g., expe ise in da a analysis, ML, ma hema ics, s a is ics, o isu-
aliza ion) and applica ion domain knowledge (e.g., biology, business, digi al humani ies, o spo s).
Analys s may ha e highly di e se backg ounds and he e o e a ying le el o skills and ela ed ca-405
pabili ies. Consequen ly, hey may p o ide di e en kinds o eedback and “ ake on di e en oles”.
Typically, da a scien is s, such as ML expe s wi h s ong ma hema ical skills, can ain specialized
ML models and echniques, bu may miss signi ican anomalies in da a gene a ion o collec ion.
Con e sely, domain expe s may be e y awa e o hese de ails, bu o e look impo an p ope ies
o models hey migh app oach as o - he-shel black boxes. To o e come his, in in e disciplina y410
esea ch, ML expe s, isualiza ion expe s, and da a owne s usually wo k in collabo a i e eams.
In his case, isualiza ion p o ides a common pla o m o communica ion. These di e ences and
gaps be ween di e en ypes o use s can be add essed by a Liaison, a pe son sha ing language
and knowledge om he applica ion domain and om he isualiza ion domain, wi h he goal o
media ing communica ion issues [49].415
Execu ion.Applying No man’s S ages o Execu ion o ou in e ac i e ML se ing (1) he in en-
ions o ac may be based on assump ions abou he ML model o he da a a hand, (2) he sequence
o ac ions desc ibes he di e en ML pipeline adap ions, and (3) he ac ual execu ion is ealized
h ough he isual in e ace (o isualiza ion) and hea ily depends on i s usabili y. As poin ed
18
ou in he p e ious pa ag aph, analys s usually need use in e aces ailo ed o hei indi idual420
capabili ies. Visual me apho s and ac ions (e.g., mo ing poin s, o p o iding labels) need o be ac-
cessible and amilia . Command line in e aces and speci ic pa ame e s a e o en ope able o da a
scien is s, howe e applica ion domain expe s o en expec simple and in ui i e use in e aces o
p o ide eedback. No e ha he isual in e ac ions should ai h ully e lec and ansla e he ana-
lys s assump ions o ML pipeline adap ions. I he analys is no able o pe o m a desi ed ac ion425
(e.g., because o poo usabili y o he use in e ace) he e is a gap be ween human and machine,
also known as “gul o execu ion” [24].
E alua ion.Be o e he analys is enabled o p o ide ( u he ) eedback, he/she has o E alua e
he cu en s a e o he sys em. In ou desc ibed VA/ML pipeline, changes made by he analys
o eedback gi en by he analys (should) cause (1) obse able eac ions in he analysis sys em.430
These obse a ions—in ou con ex usually ep esen ed as isual pa e ns (e.g., g oups, sequences,
ou lie s)—ha e o be (2) in e p e ed by he analys who can le e age his/he domain knowledge.
Finally, he analys has o (3) alida e and e i y he de i ed insigh s acco ding o p e ious goals
and assump ions. Visual in e aces should he e o e allow he analys o compa e di e en s a es o
he analysis sys em, by swi ching be ween isualiza ion esul s be o e and a e he compu a ions.435
Anima ions and ansi ions be ween s a es o p og essi e/in e media e ML esul s may enhance
he in e p e abili y o complex ML models. Design s udies ha e o be conduc ed in o de o “b ing
he en i e ML pipeline close ” o he domain expe s men al models, language, and me apho s
[50]. In e p e abili y is essen ial o e alua ing he ob ained esul s and also o p o iding u he
eedback in subsequen loops/i e a ions [51]. No e ha misin e p e a ions may cause poo eedback440
and he e o e impai he ML pipeline con igu a ion. This gap be ween machine and he human is
known as “gul o e alua ion” [24]. Especially in ML when he analys is p esen ed wi h a inal
esul , i is o en a challenge o ind ou “why” he esul is no good enough. Se e al me hods may
be combined in o complex pipelines, making i ha d o assess he quali y o he indi idual blocks.
5.2. Analysis Scena ios445
This sec ion enume a es six analysis scena ios illus a ing a a ie y o s a egies and eedback
ha can be inco po a ed in a isual in e ac i e ML se up. No ice ha some scena ios o e lap,
and can be combined o swi ched du ing an analysis session. Speci ically, he i s wo scena ios
19
(con i ma o y analysis and hypo hesis o ming) can be seen as highe le el analysis goals, in con as
wi h he la e ou scena ios.450
Con i ma o y Analysis.An ML model is buil on assump ions abou he domain and da a a
hand. In an in e ac i e ML session, analys s may make use o di e en ML ypes o model and
con i m hypo heses. In his ac i i y, hey o en co ec and e ine model pa ame e s o mo e closely
ma ch hei assump ions; hey also may need o gene a e and collec e idence o ei he e i y o
alsi y hypo heses [4]. Such e idence may be p o ided by s a is ical es s, o by inspec ing isual455
pa e ns gene a ed by a mo e complex ML algo i hm. Fo example, a g ouping o simila obse -
a ions can be compu ed by clus e ing, o classi ica ion. Howe e , analys s also ha e expec a ions
abou g oupings and may need o check whe he hei assump ions a e consis en wi h he ML
esul s. In many cases, echniques do no i “ou o he box” and need o be e ined by an expe .
Hypo hesis Fo ming.Ano he analysis goal is o gene a e, o m and e ine hypo heses. In his460
case, he analysis is mo e explo a o y, and ML models can be in oked and isualized o ge b oad
o e iews. Se e al unsupe ised ML me hods a e e ec i e o e ealing ce ain s uc u es ha a e
o he wise hidden in da a (e.g., ea u e selec ion, dimensionali y educ ion, clus e ing, ou lie o
no el y de ec ion). Visualiza ions suppo he analys in spo ing pa e ns ha can be in es iga ed
in mo e de ail. Such pa e ns can be, o example, mani olds, ou lie s, sequences, clus e s, o ends.465
No e ha spo ing pa e ns may be he esul o pu e se endipi y du ing analysis. Howe e , once
a pa e n has been spo ed, an analys gene ally needs o disco e “why” he pa e n exis s, and
consequen ly o ms mo e conc e e hypo heses and may swi ch o con i ma o y analysis.
Con on ing ML Resul s o S uc u es.In an i e a i e ML p ocess, he analys p o ides eed-
back abou esul s o he ML pipeline. Domain expe s, who a e able o exploi domain knowledge,470
can e ec i ely adap ML esul s i hey spo e o s wi hin isualiza ions ha do no ma ch hei
assump ions o p io knowledge. Fo example, hey can e-o ganize au oma ically gene a ed g oups
[52] o adjus class labels [53]. Fu he mo e, pa ame e s o weigh s can be uned o adap ML
s uc u es o ocus he analysis on speci ic ea u es, o o de e mine he g anula i y o he ML
algo i hms (e.g., he numbe o clus e s desi ed). Reo ganizing poin s (“decla ing dis ances”) can475
co ec dis ances be ween speci ic obse a ions when hey a e known o he analys (e.g., [45]).
20
Adap ing ML Pipelines.Depending on he da a and analysis ask a hand, he ML pipeline
may no be “comp ehensi e” enough and may equi e econ igu a ion o accommoda e addi ional
ML compu a ions o ea u es o he da ase . ML models can be hough o as a omic componen s
ha can be combined and hen equi e some me a-assessmen , wi h he di icul y ha alida ion480
aces combina o ial g ow h and can become in ac able. An ML algo i hm could, o example,
equi e addi ional p e-p ocessing, speci ic ea u e selec ions and ans o ma ions. I ML models
become “ oo complex” some pa s o he ML pipeline may need o be simpli ied o e en emo ed
( o example, in case o model o e i ing).
“Wha -I ”-Analysis.In e ac ions enable he analys o expe imen wi h an ML pipeline and485
obse e how i eac s o changes o eedback. This may con ibu e o be e unde s anding o how
he model beha es, e en wi hou ML expe ise. An example is o in es iga e how he inal ML
esul s a e a ec ed by adap ing dimension loadings in a p oblem o dimensionali y educ ion (such
as in iPCA [34]). In such a case, he analys can iden i y which da a i ems a e a ec ed and ela ed
o speci ic ea u es. The same can be done wi h manipula ing da a obse a ions. The analys may490
examine wha happens i a pa icula obse a ion is p esen o absen in he da a.
Expe Ve i ica ion.ML models aim o de ec s uc u e and pa e ns in da a, such as ends.
Howe e , in a eal wo ld use case, pa e ns ha e o be c oss-checked based on “ex e nal” knowledge.
One possibili y is o apply he ML model o o he ex e nal da a sou ces o es whe he he pa e n
ecu s as in a kind o manual alida ion o es p ocedu e. These da a sou ces may be ob ained495
om ano he da abase o eposi o y. Howe e , i such da a is no a ailable, he domain expe
has o judge whe he obse ed s uc u es o pa e ns a e plausible and use ul. In his case, se e al
domain expe s may collabo a i ely discuss he ou come, o design u he expe imen s.
6. Challenges & Oppo uni ies
On he pa h owa d sys ems ha ully implemen he p oposed amewo k, we encoun e se e al500
impo an esea ch challenges ha mus be o e come. We iden i ied i e ele an challenges a he
in e sec ion o ML and isualiza ion esea ch. We will desc ibe how join esea ch in hese a eas
opens up no el oppo uni ies o ad ance p ac ical da a analysis.
21

Designing In e ac ion o ML Adap ion.A a ie y o ML algo i hms, o e ing a b oad se
o design op ions and pa ame e s, ha e been desc ibed in he li e a u e. Ye , we ind no gene ic505
way o in e ace ML wi h isualiza ion. Cu en isual analy ics sys ems a e o en es ic ed o
wo king wi h a small se o ML echniques and pa ame e s. Fu he mo e, wi hin cu en in e -
aces, swi ching be ween ML models is likely o dis up a human’s sense o con ex in he analysis
p ocess. To add ess, his, new app oaches a e needed ha suppo analys s in making sense o
such model changes. In addi ion, exis ing examples such as Fo ceSpi e and iPCA nicely illus a e510
how unde s andable di ec in e ac ions can be combined wi h model changes in a s aigh o wa d
manne . Di ec manipula ion has p o en e ec i e and easy- o-lea n o accessing compu a ional
ools [25]. I has, howe e , no been ex ensi ely explo ed in he con ex o ML so a . O en, ML
models a e designed o unique, s a ic con igu a ions, whe eas in VA i e a i e e inemen is needed.
Mapping use inpu s o mo e complex algo i hmic ac ions along he en i e ML pipeline emains515
an open p oblem. One key ques ion is how o ansla e “simple” in e ac ions wi hin he isual
in e ace o da a manipula ion, p ep ocessing, o ML model-adap ion ope a ions and combina ions
he eo . —Oppo uni ies: Cen al o ou concep ual amewo k is he idea ha he unde lying ML
design op ions and me a-pa ame e s, which usually canno be op imized au oma ically, can o en
ins ead be s ee ed by con enien , i e a i e use in e ac ions. Accessible in e ac ions and smoo h520
ansi ions be ween di e en ML models will help analys s o de elop in ui ion o o m men al
models [54] abou he unde lying da a, as well as abou he unc ion o beha io o complex ML
me hods. Conside he case o swi ching be ween di e en ML models: a wha poin does he sys-
em ealize— om use eedback— ha he cu en ML model migh no be he mos app op ia e
one anymo e? I could hen sugges an al e na i e model, and smoo hly ansi ion o i . Ins ead525
o linea p ojec ion wi h PCA, i migh , o ins ance, sugges a mo e complex nonlinea dimension-
ali y educ ion me hod like mul idimensional scaling o -SNE. Con inuous model spaces [55, 56]
con ibu e some p elimina y ideas owa ds such solu ions, which a e dependen on he ML models’
me a- o hype -pa ame e s and hei in e p e abili y. Fu he , mo e gene al ways o apply and
adap ML h ough expe eedback (e.g., labeling o a ing) would allow us o ake ad an age o a530
la ge , mo e powe ul se o ML me hods. The p e ious examples demons a e ha he e is as
space o u u e esea ch, gi en he g ea a ie y o a ailable ML echniques and hei associa ed
pa ame e spaces. A join e o om bo h he ML and VA communi ies is needed o ace his
esea ch challenge.
22
Guidance.Ano he majo challenge is o adequa ely suppo applica ion domain expe s in s ee -535
ing he ML pipeline. Analys s can be o e whelmed by he wide ange o ML models and pa ame e s,
along wi h he challenges o wo king wi h la ge da a se s. Mo eo e , analysis p oblems a e o en
incomple ely de ined, and change o e ime, esul ing in a complex analysis p ocess wi h much
ial-and-e o . Consequen ly, analys s may change, adap , o swi ch asks equen ly. While ana-
lys s may b ing c ucial domain-dependen in o ma ion o p oblem-sol ing, hey o en lack ad anced540
p og amming skills and s a is ical expe ise, and he e o e equi e assis ance and guidance (e.g., by
p o iding ecommenda ions abou ope a ions on da a and al e na i e models.) —Oppo uni ies:
I is impo an o be e unde s and he asks, p ac ices, and s umbling blocks o domain expe s
(which likely di e om hose o isualiza ion o ML expe s). Adop ing a design s udy me hodol-
ogy is a iable app oach owa ds gaining be e unde s anding o such use cha ac e is ics [50] and545
p o iding app op ia e guidance. Fu he mo e, enhanced measu es and ools could poin analys s o
in e es ing da a, pa ame e iza ions, and ML models h ough au oma ic ecommenda ions. While
many measu es exis , bo h depic ing da a and pe cep ual cha ac e is ics (e.g., [57]), cu en ly i
is no well unde s ood how hey can be e ec i ely exploi ed in in e ac i e analy ical p ocesses.
Conside a ele ance eedback app oach whe e he lea ning sys em e ie es a se o in e es ing550
isualiza ions based on i e a i e use eedback. In each i e a ion he analys ma ks he p esen ed
esul s ei he as posi i e (in e es ing) o nega i e (unin e es ing) [53]. How could he sys em de ec
i a pa e n was spo ed and he analysis ask changes om o e iew o de ail? The e o e, we
en ision he usage o analy ic p o enance which “cap u es he in e ac i e explo a ion p ocess and
he accompanied human easoning p ocess du ing sensemaking” [41]. This in o ma ion could guide555
he analysis p ocess o mee he analys s’ needs, which migh be de i ed om hei beha io . In
he VA communi y, esea ch has been ca ied ou on cap u ing, isualizing, and eusing analy ic
p o enance. Howe e , mo e wo k is needed on modeling such in o ma ion o shape o e ine he
o e all analysis as well as speci ic ML me hods. Doing his is an in e es ing esea ch p oblem ha
will equi e expe ise om he ML communi y.560
Measu ing Quali y & Consis ency.In he ich human-in- he-loop analysis p ocess we en ision,
i is c ucial o ensu e bo h ML model quali y and isualiza ion quali y. Ye , hese wo ypes o quali y
assu ance do no always align well. Fo example, in a isual ep esen a ion o a da a embedding,
a e dimensionali y educ ion, he e migh be a ade-o be ween p ese a ion o he o iginal da a
23
s uc u e, and eadabili y o pa e ns, due o in insically high dimensionali y. While measu es ha e565
been de eloped ha desc ibe each o hese aspec s o quali y in da a analysis (e.g., [55], [56], [57],
and [58]), he challenge is o help analys s o ind he igh balance be ween hem, so ha meaning ul
analysis is enhanced. Beyond measu ing ML and isualiza ion quali y, ou amewo k sugges s a
hi d ype o quali y assessmen , which is he le el o consis ency be ween he ML model and he
analys ’s expec a ions. The goal o da a analysis is o ex ac eliable and ele an knowledge om570
da a. Assuming ha he e exis s some “g ound u h” o back up such knowledge, i is he goal
o ML and isualiza ion o e eal i wi h high ideli y. A he same ime, he use will ha e a
p io i knowledge and expec a ions, which in he ideal case should closely ma ch wha he analysis
e eals. While an ML model will su ely seek o accu a ely desc ibe he da a, essen ial pieces o
in o ma ion o con ex known by analys s may be una ailable o an algo i hm. In his case, he se o 575
ML assump ions may be incomple e, a challenge o en encoun e ed in explo a o y da a analysis. —
Oppo uni ies: To ex e nalize his missing in o ma ion, i is impo an o check consis ency be ween
wha he ML model p esen s and wha he analys expec s. I inconsis en , he analys should ei he
suspec a p oblem wi h he ML model and p o ide eedback abou missing in o ma ion, o accep
ha he expec ed pa e ns we e no ound in he da a. I consis en , analys s usually conclude he e580
is a con i ma ion o hei expec a ion. No e, al hough consis ency be ween human and machine is
desi able, i does no gua an ee co ec e lec ion o he unde lying g ound u h in he da a pe
se. Cu en ly, consis ency checks a e o en done manually. Au oma ic me hods ha sys ema ically
check consis ency, highligh inconsis encies and ecommend any needed emedia ion could help.
Join e o om bo h ML and VA communi ies is needed o enhance hese measu es, especially by585
combining and b idging hem.
Handling Unce ain y.The e a e se e al s ages in ou amewo k whe e unce ain y migh be
deal wi h explici ly. Unce ain y may a ise om se e al sou ces o un eliabili y o agueness,
such as da a desc ibed by p obabili y densi y desc ip ions, missing da a, o e en sys ema ic e o s
in modeling. Visualiza ions hemsel es can also in oduce unce ain y, o example, due o eso-590
lu ion o con as e ec s [59]. In ou amewo k, unce ain y implici ly p opaga es h ough he
pipeline and e en ually may a ec he analys ’s us -building p ocess [60]. P ope ly desc ibing,
quan i ying, and o mally p opaga ing unce ain y in all pipeline s ages will be a majo challenge
in de eloping obus , e ec i e ools. —Oppo uni ies: Al e na i e isualiza ions can be gene -
24
a ed and in es iga ed o aise he analys ’s awa eness o he sou ces o unce ain y wi hin he ML595
pipeline. Fu he mo e, analys s can be suppo ed in (in e ac i ely) explo ing, unde s anding and
educing unce ain y [61]. Be e in eg a ion o unce ain y measu es om da a, p ep ocessing, ML
models, and isualiza ion can be expec ed o p o ide a holis ic pe spec i e and unde s anding o
unce ain y. The e is much p e ious wo k on isualiza ion echniques o display da a unce ain y
o spa ial da a, such as olume o low isualiza ion [62, 63]. We ind less wo k on unce ain y600
isualiza ion o abs ac da a, such as high-dimensional da a isualiza ion [64]. As abs ac da a
is ypical in ML applica ions, he e is a need o imp o ed unce ain y isualiza ion along he an-
aly ical pipeline ou lined in ou amewo k. A join e o by ML and isualiza ion esea che s is
needed o handle unce ain y wi hin he en i e pipeline.
ML-Vis In e ope abili y.A inal challenge a ises because mos exis ing ML algo i hms, oolki s605
and lib a ies we e no designed o suppo in e ac i e isualiza ion. Scalabili y p oblems in compu-
a ion may cause long delays ha impede in e ac ion; pa ame e s may no be adap able o isible;
and ele an in o ma ion (e.g., quali y measu es o in e nal ML s uc u es) may be inaccessible.
The abili y o communica e hese ypes o algo i hmic in o ma ion and o ake ad an age o hem
o cons uc be e use in e aces is o en desc ibed as “opening he black box” [17]. Especially610
in he case o di ec human in e ac ions, i is o en di icul o speed up he ML compu a ions
enough o p o ide he desi ed esponsi e beha io o he isualiza ion. Ano he challenge is o
ain ML echniques om in e ac i e inpu s, which ypically a e ew in numbe . —Oppo uni ies:
The isualiza ion communi y could bene i om ML algo i hms and lib a ies ha mee speci ic
equi emen s, such as exposing in e media e o p og essi e esul s, and p o iding meaning ul and615
in e p e able pa ame e s o handles o in eg a e hem wi h in e ac i e isualiza ions. Addi ional
in o ma ion, such as model s uc u es, p ep ocessings, and quali y in o ma ion can be isualized.
Recen ly, no el VA sys ems ha e been desc ibed ha p o ide app oxima ed o p og essi e com-
pu a ions wi h in e ac i e and s ee able isualiza ions (e.g., p og essi e -SNE [65] o p og essi e
PCA [66]). These examples sugges conside able po en ial o an expanded, gene alized in eg a-620
ion o ML wi h isualiza ion. In summa y, bo h communi ies could gain much om lib a y and
amewo k designs in o med by he equi emen s o bo h in e ac i e isualiza ion and ML.
25
[53] M. Beh isch, F. Ko kmaz, L. Shao, T. Sch eck, Feedback-d i en in e ac i e explo a ion o
la ge mul idimensional da a suppo ed by isual classi ie , in: IEEE Con . Visual Analy ics in
Science and Technology (VAST), 2014, pp. 43–52. doi:10.1109/VAST.2014.7042480.790
[54] Z. Liu, J. S asko, Men al models, isual easoning and in e ac ion in in o ma ion isualiza ion:
A op-down pe spec i e, IEEE T ans. Visualiza ion and Compu e G aphics (TVCG) 16 (6)
(2010) 999–1008. doi:10.1109/TVCG.2010.177.
[55] J. Venna, J. Pel onen, K. Nybo, H. Aidos, S. Kaski, In o ma ion e ie al pe spec i e o non-
linea dimensionali y educ ion o da a isualiza ion, Jou nal o Machine Lea ning Resea ch795
11 (2010) 451–490.
[56] J. A. Lee, E. Rena d, G. Be na d, P. Dupon , M. Ve leysen, Type 1 and 2 mix u es o Kullback-
Leible di e gences as cos unc ions in dimensionali y educ ion based on simila i y p ese a-
ion, Neu ocompu ing 112 (2013) 92–108. doi:10.1016/j.neucom.2012.12.036.
[57] E. Be ini, A. Ta u, D. A. Keim, Quali y me ics in high-dimensional da a isualiza ion: An800
o e iew and sys ema iza ion, IEEE T ans. Visualiza ion and Compu e G aphics (TVCG)
17 (12) (2011) 2203–2212.
[58] M. Sips, B. Neube , J. P. Lewis, P. Han ahan, Selec ing good iews o high-dimensional da a
using class consis ency, Compu e G aphics Fo um 28 (3) (2009) 831–838.
[59] T. Zuk, S. Ca pendale, Visualiza ion o unce ain y and easoning, in: P oceedings o he805
8 h In e na ional Symposium on Sma G aphics, SG ’07, Sp inge -Ve lag, Be lin, Heidelbe g,
2007, pp. 164–177. doi:10.1007/978-3-540-73214-3_15.
[60] D. Sacha, H. Sena a ne, B. C. Kwon, G. P. Ellis, D. A. Keim, The ole o unce ain y, awa eness,
and us in isual analy ics, IEEE T ans. Visualiza ion and Compu e G aphics (TVCG) 22 (1)
(2016) 240–249. doi:10.1109/TVCG.2015.2467591.810
[61] C. D. Co ea, Y. Chan, K. Ma, A amewo k o unce ain y-awa e isual analy ics, in: IEEE
Con . Visual Analy ics in Science and Technology (VAST), 2009, pp. 51–58. doi:10.1109/
VAST.2009.5332611.
[62] A. Pang, C. M. Wi enb ink, S. K. Lodha, App oaches o unce ain y isualiza ion, The Visual
Compu e 13 (8) (1997) 370–390.815
32

[63] K. B odlie, R. A. Oso io, A. Lopes, A e iew o unce ain y in da a isualiza ion, in: J. Dill,
R. Ea nshaw, D. Kasik, J. Vince, P. C. Wong (Eds.), Expanding he F on ie s o Visual
Analy ics and Visualiza ion, Sp inge , 2012, pp. 81–109.
[64] M. Skeels, B. Lee, G. Smi h, G. G. Robe son, Re ealing unce ain y o in o ma ion isual-
iza ion, In o ma ion Visualiza ion 9 (1) (2010) 70–81.820
[65] N. Pezzo i, B. Lelie eld , L. an de Maa en, T. Holl , E. Eisemann, A. Vilano a, App oxi-
ma ed and use s ee able sne o p og essi e isual analy ics, IEEE T ans. Visualiza ion and
Compu e G aphics (TVCG) PP (99) (2016) 1–1. doi:10.1109/TVCG.2016.2570755.
[66] C. Tu kay, E. Kaya, S. Balcisoy, H. Hause , Designing p og essi e and in e ac i e analy ics p o-
cesses o high-dimensional da a analysis, IEEE T ans. Visualiza ion and Compu e G aphics825
(TVCG) 23 (1) (2017) 131–140. doi:10.1109/TVCG.2016.2598470.
[67] K. Poppe , The logic o scien i ic disco e y, Rou ledge, 2005.
[68] A. Dix, Theo e ical analysis and heo y c ea ion, Resea ch Me hods in Human-Compu e
In e ac ion (2008) 175–195.
33