Kelan, Elisabe h
Book
Pa e ns o Inclusion: How Gende Ma e s o
Au oma ion, A i icial In elligence and he Fu u e o Wo k
P o ided in Coope a ion wi h:
Taylo & F ancis G oup
Sugges ed Ci a ion: Kelan, Elisabe h (2025) : Pa e ns o Inclusion: How Gende Ma e s o
Au oma ion, A i icial In elligence and he Fu u e o Wo k, ISBN 978-1-040-13004-9, Rou ledge,
Ox o d,
h ps://doi.o g/10.4324/9781003427100
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/312728
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/legalcode
PATTERNS OF INCLUSION
I is widely p esumed ha digi alisa ion, au oma ion and a i icial in el-
ligence (AI) shape he u u e o wo k; ye , gende is a ely conside ed in
hose deba es. This g ound- b eaking book, w i en by a leading hinke
on gende , inclusion and o ganisa ions, is based on in- dep h esea ch o
show which pa e ns o gende and digi alisa ion eme ge. By wea ing hese
di e en pa e ns oge he , is i possible o unde s and he dynamic and
complex ways gende and digi alisa ion in e wine in he wo k con ex ?
The book highligh s how u u es o wo k a e imagined be ween au o-
ma ion and augmen a ion: i shows which asks a e expec ed o be done by
machines, and whe e humans a e expec ed o ha e a compe i i e ad an age.
The book showcases how algo i hmic bias is cons uc ed as ul ima ely ix-
able, and analyses in/ isibili ies in AI p oduc ion p ocesses. Abo e all, he
book shows how pa e ns ela ing o gende and inclusion a e shaped and
could be eshaped.
This inno a i e book p o ides a s imula ing and p o oca i e ead o
hose who a e in e es ed in how au oma ion and AI shape he u u e o
wo k in ega d o gende and wha his means o inclusion.
Elisabe h Kelan is P o esso o Leade ship and O ganisa ion a Essex
Business School, Uni e si y o Essex, Uni ed Kingdom. Kelan is an expe
on gende and digi alisa ion, women’s leade ship, men as change agen s o
gende equali y, gene a ions a wo k, and di e si y and inclusion.
“A g ound- b eaking book ha ills a c i ical gap by p o iding a much-
needed ho ough analysis h ough a gende lens o pe spec i es on he
po en ial impac s o au oma ion and AI on he u u e o wo k. I enables
us o imagine mo e inclusi e and equi able scena ios ha be e equip
us o o ge a ai e digi al u u e o all.”
U sula Wynho en, Di ec o and Rep esen a i e o he Uni ed Na ions, In e na ional
Telecommunica ion Union
“An impo an wo k ha p o ides unique, imely and excep ional
insigh s in o he digi alisa ion o gende . Th ough igo ous esea ch,
Elisabe h Kelan illumina es he gende ed alues, decisions and biases
ha shape digi alisa ion, au oma ion and a i icial in elligence sys ems,
and why hey mus be g ounded in equi y and inclusion.”
Melissa Suzanne Fishe , New Yo k Uni e si y Ins i u e o Public Knowledge and
School o P o essional S udies
PATTERNS OF INCLUSION
How Gende Ma e s o Au oma ion,
A i icial In elligence and he Fu u e
o Wo k
Elisabe h Kelan
Designed co e image: Ge y Images/anna
Fi s published 2025
by Rou ledge
4 Pa k Squa e, Mil on Pa k, Abingdon, Oxon OX14 4RN
and by Rou ledge
605 Thi d A enue, New Yo k, NY 10158
Rou ledge is an imp in o he Taylo & F ancis G oup, an in o ma business
© 2025 Elisabe h Kelan
The igh o Elisabe h Kelan o be iden i ied as au ho o his wo k has been
asse ed in acco dance wi h sec ions 77 and 78 o he Copy igh , Designs
and Pa en s Ac 1988.
The Open Access e sion o his book, a ailable a www. aylo ancis.com, has been
made a ailable unde a C ea i e Commons A ibu ion (CC-BY) 4.0 license.
Any hi d pa y ma e ial in his book is no included in he OA C ea i e Commons
license, unless indica ed o he wise in a c edi line o he ma e ial. Please di ec any
pe missions enqui ies o he o iginal igh sholde .
T adema k no ice: P oduc o co po a e names may be adema ks o egis e ed
adema ks, and a e used only o iden i ica ion and explana ion wi hou in en o
in inge.
This book is based on a Le e hulme T us Majo Resea ch Fellowship [MRF-2019-069]
and epo s indings om a p ojec unded by he B i ish Academy [SRG20 200195].
This publica ion was suppo ed by he Uni e si y o Essex’s Open Access Fund.
Da a A ailabili y S a emen
Due o he na u e o his esea ch, pa icipan s o his book did no ag ee o hei
da a o be sha ed publicly, so suppo ing da a is no a ailable.
B i ish Lib a y Ca aloguing- in- Publica ion Da a
A ca alogue eco d o his book is a ailable om he B i ish Lib a y
ISBN: 9781032731728 (hbk)
ISBN: 9781032669892 (pbk)
ISBN: 9781003427100 (ebk)
DOI: 10.4324/9781003427100
Typese in Joanna
by Newgen
To Michael, who always encou ages me o be cu ious abou echnologies.
CONTENTS
P e ace ix
Acknowledgemen s xiii
1 In oduc ion: Human- Like 1
2 Imaging Fu u es be ween Au oma ion and Augmen a ion 24
3 Uniquely Human and he Au oma abili y o
Socio- Emo ional Skills 47
4 Algo i hmic Bias as Ul ima ely Fixable 71
5 In/ Visibili y by Design 95
6 Conclusion: Unw i en Rules 122
Appendix 142
Re e ences 146
Index 162
ACKNOWLEDGEMENTS
xi
I has been an as ic o wo k wi h he eam a Rou ledge on his publi-
ca ion. In pa icula , Rebecca Ma sh no only belie ed in his book bu also
p o ided in aluable eedback.
Essex Business School has p o ided an in ellec ually s imula ing home
du ing he esea ch o and w i ing o he book. I could no ha e wished
o mo e suppo i e colleagues.
I am deeply g a e ul o my amily o gi ing me he ime and space o
w i e his book.
Some o he ma e ial co e ed in his book has been published in aca-
demic jou nals wi h Wiley:
Kelan, E. K. (2024). Algo i hmic inclusion: Shaping he p edic i e
algo i hms o a i icial in elligence in hi ing. Human Resou ce Managemen
Jou nal, online ea ly.
Kelan, E. K. (2023). Au oma ion anxie y and augmen a ion aspi -
a ion: Sub ex s o he u u e o wo k. B i ish Jou nal o Managemen , 34(4),
2057– 2074.
Finally, I am ex emely g a e ul o hose indi iduals who kindly sha ed
hei expe iences and e lec ions wi h me du ing he in e iews. I would
also like o hank hose who ha e p o ided access o echnology which
I could explo e o his esea ch. This book would no ha e been possible
wi hou his suppo .
newgenp epd
This chap e has been made a ailable unde a CC BY – A ibu ion license.
DOI: 10.4324/9781003427100-1
1
INTRODUCTION
HUMAN- LIKE
In oduc ion
The u u e o wo k and digi alisa ion is a opic ha is domina ing discussions
in he media and in many o ganisa ions. While he u u e o wo k is egu-
la ly in oked, i has been sugges ed ha he e m ‘ u u e o wo k’ is poly-
semous wi h a ious meanings a ached o i . While how gende ma e s in
hese ans o ma ion p ocesses is egula ly igno ed, how gende is ele an
o digi alisa ion a wo k akes cen e s age in his book. Based on de ailed
empi ical esea ch, I sugges in his book ha we need o look a pa e ns
a ound gende and echnology ha eme ge in di e en se ings. Howe e ,
only by wea ing oge he hese di e en pa e ns is i possible o unde -
s and he dynamic and complex ways in which gende and digi alisa ion
in he wo k con ex a e in e wined. This in oduc o y chap e discusses
why ocusing on how he u u e o wo k is imagined is cen al o which
u u es a e being made possible and impossible. I also sugges ha ech-
nologies a e o en imbued wi h magical and my hical quali ies in e e yday
con e sa ions. I hen u n o discussing how he gende – echnology
dynamic is unde s ood be o e explaining he unde lying esea ch o and
he s uc u e o his book. The book a gues ha we need o unde s and he
dynamics be ween gende and digi alisa ion in he wo k con ex o c ea e
mo e equi able u u es.
INTRODUCTION: HUMAN-LIKE
2
Imagining Fu u es o Wo k and Human- Like In elligence
The u u e o wo k is a opic ha enjoys a cons an in e es – mul iple
epo s and books a e au ho ed e e y yea ha ace he ques ion o wha
he u u e o wo k migh hold. Wajcman (2017) obse es ha p edic ing
he u u e o wo k has become ‘big business’. This ex ends o con e ences
on ha opic. She inds ha such con e ences ollow a amilia and p e-
dic able pa e n: echnologis s ma el a he la es echnological p og ess,
economis s pain a conce ning pic u e o he u u e o jobs and u u is s
p edic he nex ends (Wajcman, 2017). The e m ‘ u u e o wo k’ in i sel
has been desc ibed as a ‘ loa ing signi ie ’ o which a ious meanings a e
a ached by di e en g oups (Schlogl e al., 2021). A common ea u e o
discussions on he u u e o wo k is ha hey play wi h he dicho omy
be ween u opia and dys opia (Schlogl e al., 2021; Howc o & Taylo , 2023).
One could a gue ha how he u u e is imagined has limi ed consequences
because mos o such p edic ions end o be w ong anyway. Howe e , i
would be a mis ake o dismiss hese p edic ions o he u u e as i ele an
o meaningless. Such isions o he u u e ha e p o ound consequences
o socie ies because hey c ea e ideas o wha migh be possible (U y,
2016). They also c ea e eali ies (Schlogl e al., 2021). U y (2016) wa ns
agains seeing u u es simply based on a p ede e mined pa h o de elop-
men o echnologies o as seeing u u es as comple ely open and emp y.
How u u es a e imagined, o ins ance, in discou ses on he u u e o wo k,
shapes wha migh o migh no be possible.
Science ic ion can p o ide a bluep in o how po en ial desi able o
undesi able u u es a e imagined, which hen in luences wha is seen as
possible (Jasano , 2015). Ins ead, Jasano (2015) ollows he idea ha
socie y and echnology a e co- cons uc ed. Such a co- cons uc ion becomes
mani es in socio echnical imagina ies (Jasano , 2015). Socio echnical
imagina ies a e ‘collec i ely held, ins i u ionally s abilized, and publicly
pe o med isions o desi able u u es, anima ed by sha ed unde s andings
o o ms o social li e and social o de a ainable h ough, and suppo i e
o , ad ances in science and echnology’ (Jasano , 2015, p. 4). In o he
wo ds, socio echnical imagina ies a e a way o analyse how indi iduals
a e engaging in collec i e p ac ices when imaging he u u e. Di e ging
socio echnical imagina ies can coexis and shape one ano he , and a e
e alua ed and deba ed h ough socie al discou ses. E en hough mos
INTRODUCTION: HUMAN-LIKE 3
socio echnical imagina ies a e ul ima ely in e es ed in shaping u u es
ha a e desi able, hey o en do so by ou lining scena ios ha ough o be
a oided; he ension be ween a desi able u opia and an undesi able dys opia
is ac i ely used in socio echnical imagina ies (Jasano , 2015).
Much o he li e a u e on he u u e o wo k a emp s o wa n o undesi -
able dys opias, pa icula ly h ough job losses associa ed wi h echnologies.
Many o he widely ci ed epo s on he u u e o wo k engage in p edic ions
on wha he u u e o wo k migh en ail and how many jobs a e going o
be los due o au oma ion ha is d i en by echnologies (Manyika, Chui, &
Mi emadi, 2017; Hawkswo h, Be iman, & Goel, 2018; O ganisa ion o
Economic Coope a ion and De elopmen [OECD], 2016; Wo ld Economic
Fo um, 2020; Ballies e & Elsheikhi, 2018; Ha zius e al., 2023). I is also
common o con ex ualise cu en changes in ega d o di e en indus ial
e olu ions. Schwab (2018) sugges s ha he Fi s Indus ial Re olu ion was
ma ked by mechanisa ion o he ex ile indus y in B i ain; he second was
associa ed wi h elec ici y, he elephone and he au omobile; he hi d
was ela ed o changes in digi al compu ing in he 1950s; and he ou h
is ma ked by a ange o new echnologies, including a i icial in elligence
(AI), dis ibu ed ledge s (blockchain), ad anced ma e ials and i ual and
augmen ed eali ies. Schwab (2018) p edic s ha exponen ial g ow h in
hese new echnologies will lead o apid change, and ha au oma ion may
accele a e job losses. In con as , McA ee and B ynjol sson (2014) coined
he e m ‘second machine age’. The i s machine age is equa ed wi h he
Indus ial Re olu ion when machines imp o ed human labou ; he second
machine age is said o ha e begun in he mid- 1990s wi h digi alisa ion
and is cha ac e ised by machines no simply ollowing ules bu sol ing
p oblems on hei own. Thus, machines now pe o m cogni i e asks p e-
iously ese ed o humans. This leads o ea s ha humans a e eplaced
by machines.
Fea s o humans being eplaced by machines a e o cou se no new.
Fo ins ance, du ing wha Schwab (2018) would call he Fi s Indus ial
Re olu ion, he ela ionship be ween machines and humans was ed awn
and a common pe cep ion was ha machines a e going o eplace phys-
ical powe ha people had exe ed be o e (S andage, 2002). Howe e , he
a i al o he Mechanical Tu k, he 18 h- cen u y li e- size chess- playing
au oma on (see Chap e 5), challenged his idea, because he Mechanical
Tu k seemed o ou pe o m humans men ally (S andage, 2002). Al hough
INTRODUCTION: HUMAN-LIKE
4
wha appea ed magical o audiences a he ime u ned ou o be a hoax, he
possibili y ha machines could ou do humans men ally was ce ainly pa
o he ascina ion wi h he Mechanical Tu k.
This blu ing o wha cons i u es human and wha cons i u es machine
in elligence is also isible in how Joseph Weizenbaum’s cha bo ELIZA was
ecei ed. ELIZA was a na u al language p ocessing compu e p og amme
de eloped by Joseph Weizenbaum in he 1960s o explo e how humans and
machines communica e. The name ELIZA was chosen as a e e ence o Eliza
Dooli le in Geo ge Be na d Shaw’s play Pygmalion (Na ale, 2019; Dillon,
2020). This choice o name ca ies s ong gende and class conno a ions
(Dillon, 2020). Weizenbaum’s aim wi h ELIZA was o show humans ha AI
is an illusion (Na ale, 2019). Howe e , Weizenbaum was shocked by he ac
ha a he han ecognising he di e ence be ween human in elligence and
AI, humans engaged emo ionally wi h he machine and an h opomo phised
i (T eusch, 2017). In o he wo ds, use s pe cei ed ELIZA’s answe s as
human- like. E en when use s, such as Weizenbaum’s sec e a y, knew ha
ELIZA was no engaging on an emo ional le el, hey asc ibed emo ional
compe ence o he cha bo (Dillon, 2020; T eusch, 2017; Rhee, 2023).
The ‘ELIZA e ec ’ desc ibes how humans p esume mo e in elligence in
a machine han eally exis s (Ho s ad e , 1995; Dillon, 2020). While we
e u n o he gende ing o cu en i ual pe sonal assis an s (VPAs) (see
Chap e 5), Dillon obse es ha ‘when a human being is con e sing wi h
a VPA, he b ain is p ocessing ha con e sa ion as i would a con e sa ion
wi h ano he human being. The Eliza (sic) e ec is he e embedded in he
neu al esponse o he oice’ (Dillon, 2020, p. 11). Humans hus engage
wi h machine- gene a ed oices in he same way as wi h human oices. This
suppo s he illusions o AI as human- like in elligence a he han exposing
i as an illusion, as Weizenbaum had hoped. When Cha GPT eached he
mains eam in la e 2022, many commen a o s simila ly ma elled a he
human- like answe s he cha bo was able o p o ide (Ha zius e al., 2023).
I was exac ly his human- likeness ha p omo ed conce ns ha human
men al capaci ies could be eplaced wi h machines (Ha zius e al., 2023).
This in many ways echoes Weizenbaum’s own conce n abou AI, which he
wan ed o expose wi h ELIZA (T eusch, 2017), bu also he wide conce ns
ha humans will be eplaced by machines.
Al hough p edic ions abou humans being eplaced by machines
appea as dys opian and a e usually ollowed wi h calls o a uni e sal
INTRODUCTION: HUMAN-LIKE 5
basic income, he e a e also socio echnical imagina ies ha a e u opian
and as such mo e hope ul. He e, u opias a e imagined as desi able, whe e
humans can ocus on speci ic asks: hose whe e humans ha e a com-
pe i i e ad an age o whe e hey collabo a e wi h machines (Ha zius
e al., 2023; Daughe y & Wilson, 2018). I is a gued ha hese ech -
nologies also mean ha new jobs eme ge, and while some jobs disappea ,
new ones will be c ea ed (Ha zius e al., 2023). Exis ing jobs migh also
be enhanced by echnologies h ough new human– machine collabo -
a ion (Daughe y & Wilson, 2018). These u opian discou ses a e closely
associa ed wi h wha has been called augmen a ion, whe e humans
and machines augmen each o he ’s skills (Raisch & K akowski, 2021).
While au oma ion is la gely associa ed wi h dys opian ideas o jobs being
eplaced by machines, augmen a ion ep esen s he u opian idea ha
humans can ei he ocus on ac i i ies whe e hey ou pe o m machines
o collabo a e wi h machines.
Al hough au oma ion and augmen a ion a e o en p esen ed as opposing,
mos wo kplaces will expe ience bo h au oma ion and augmen a ion o
di e en deg ees. This in i sel is no a new phenomenon. When Senne
(1998) e u ned o a bake y ha he had isi ed many yea s be o e, he
no iced how he p ocess o baking b ead has been compu e ised; he bake s
no longe made psychical con ac wi h he ing edien s and moni o ed he
b ead- making p ocess ia sc eens. Senne (1998) a gues ha his leads o
a deskilling and aliena ion o bake s wi h no hands- on knowledge o how
o p oduce b ead. Wo k has become wha Senne (1998) calls ‘illegible’ o
he bake s. This p e- emp ed Senne ’s (2008) la e a gumen ha c a wo k
is a way h ough which people comp ehend hei wo lds. O cou se, many
people would a gue ha he digi alisa ion o b ead making is enhancing
bake s’ skills – hey need o know abou echnology and how o engage
wi h his echnology o achie e op imal esul s. In ac , Senne (1998)
desc ibes ha bake s manipula e he machines i some hing goes w ong
and hus de elop addi ional knowledge, bu he s ill main ains ha bake s
ha e los he abili y o bake b ead in he adi ional sense. As digi alisa-
ion changes he skills equi ed o jobs, such a gumen s sugges ha ce -
ain skills will no longe be equi ed and will be los because people do
no in es ime in honing hem. This means ha while some jobs will be
au oma ed and migh disappea o e ime, digi alisa ion will also change
he skills equi ed o do exis ing jobs.
INTRODUCTION: HUMAN-LIKE
6
The e is also an impo an change o which jobs a e expec ed o dis-
appea . While in he pas , blue- colla wo k was p esumed o be au oma ed,
i has mo e ecen ly been sugges ed ha he ocus o job eplacemen due
o au oma ion has mo ed o whi e- colla p o essional jobs (c . Wajcman,
2017; Howc o & Rube y, 2019; Ca e, 2020). This ma ks a shi om
machines eplacing physical human powe , o machines h ea ening o
eplace human minds. They emula e human in elligence. While some jobs
migh be eplaced due o echnologies in p o essional se ices, i can also
be expec ed ha he asks p o essionals do will change. Digi alisa ion will
a ec empo ali ies in hose p o essions. Fo ins ance, when sp eadshee s
we e i s in oduced, accoun an s we e able o comple e asks quicke due
o au oma ion, bu clien s also s a ed o expec a quicke u na ound and
also mo e analy ical insigh (O’Conno , 2023). Simila ly, he in oduc ion
o new communica ion ools like online ideocon e encing ia Zoom and
sha ed calenda s has led o mo e wo k a he han less wi h e e - dec easing
inc emen s o ime immedia ely illed by demands, leading o he conclu-
sion ha ‘whi e- colla wo k always seems o expand o ill he ime a ail-
able’ (O’Conno , 2023). This chimes wi h esea ch ha has shown ha
he idea ha digi al calenda s op imise one’s ime is a alse belie because
such echnologies a e no gi ing indi iduals mo e ime (Wajcman, 2019).
Ins ead, digi al echnologies con ibu e o an accele a ion o e e yday li e
(Wajcman, 2015). Simila ly, echnologies ha a e expec ed o be ime
sa ing o en do no deli e on he expec ed e ec s. Cowan (1983) shows
how household echnologies, which should sa e e o and ime on house-
hold asks, mean ha mo he s and wi es inc easingly ook on household
ac i i ies ha had p e iously been comple ed by a he s, husbands, chil-
d en and se an s. As such, household echnologies did con ibu e o an
in ensi ica ion o wha mo he s a e expec ed o do wi hou eeing up ime
(Cowan, 1983).
Talking abou household echnologies also d aws a en ion o how he
u u e o wo k is commonly amed: mos discussions on he u u e o wo k
ocus on paid wo k a he han unpaid wo k (Lehdon i a e al., 2023). Wo k
done in he home such as ca ing o child en o elde ly ela i es, p epa ing
meals o w i ing bi hday ca ds is i ually ne e discussed in ela ion o he
u u e o wo k. Feminis s ha e long made he a gumen ha wo k should
encompass wo k done in he household (Oakley, 2018; England & Lawson,
2005). This can en ail unpaid and paid ca e wo k whe e a speci ic ocus in
INTRODUCTION: HUMAN-LIKE 7
ega d o unpaid wo k is on how his wo k is o en acialised (Eh en eich
& Hochschild, 2003; Gu ié ez- Rod íguez, 2014). Unpaid wo k a home is
also expe iencing digi alisa ion, o e ing he oppo uni y o explo e wha
and how au oma ion and augmen a ion happen and in e ac wi h gende
(S enge s & Kennedy, 2020; Lehdon i a e al., 2023).
While i has o be acknowledged ha he de ini ion o wo k employed
in discussions o he u u e o wo k is na ow and ha his is p oblem-
a ic in i sel , in his book, I ha e decided o ocus on how such speci ic
pe spec i es o he wo ld ha e consequences o how he u u e o wo k is
imagined. I a gue o conside ing he complex in e play o in e sec ional
inequali ies, which is discussed mo e ully la e . Such an analysis allows o
ques ioning wha speci ic iews o he wo ld allow us o see and wha hey
obscu e. The acknowledgemen ha wo k la gely ocuses on paid wo k is
a necessa y bu no su icien analy ical ool o make he dynamic in e play
o in e sec ional inequali ies isible. The book he e o e ocuses on wo k as
paid wo k and acknowledges ha such a aming o wo k is exclusiona y
because i excludes unpaid ca e wo k. Howe e , how he u u e o (paid)
wo k is imagined is in i sel a subjec wo hy o s udy because in hese
socio echnical imagina ies, pa e ns o inclusion and exclusion migh be
pe pe ua ed o challenged.
The Magic o Human- Like Technology
In e e yday con e sa ions, e ms used o desc ibe cu en echnologies such
as digi alisa ion, AI and algo i hms ha e o en a my hical and magical quali y
o hem. Finn (2017) sugges s ha people ha e de eloped a s ong belie
in algo i hms ha is ai h- like; wi h li le unde s anding how such ech-
nologies ope a e, Finn (2017) sugges s ha people belie e in hem. Finn
(2017) a gues ha machines occupy simila spaces o magical and my hical
hinking in p imi i e socie ies. Building on Malinowski’s in luen ial wo k,
Finn (2017) sugges s ha echnology akes he place o i uals and belie s
in mode n socie ies. Finn (2017) p oposes ha engaging in i uals such as
summoning a ide ia Ube ul ils he same unc ion as i uals in p imi i e
socie ies: i helps humans o deal wi h dange and unce ain y. In many
e e yday con e sa ions, e ms like algo i hm ake on a my hical and magical
s a us whe e hese e ms a e used wi hou ully unde s anding wha hey
en ail and how hese echnologies wo k. This my hical and magical na u e
INTRODUCTION: HUMAN-LIKE
8
o e ms like algo i hms and AI can be exposed by looking a de ini ions o
hose e ms, which o mos indi iduals wo king ou side a na ow academic
ield will appea as abs ac . Howe e , in o de o weaken he my hical and
magical powe o hese e ms, i is impo an o de elop an unde s anding
o wha hese e ms e e o wi hou ge ing los in he echnical de ini ions
o hese e ms.
A i s e m one hea s egula ly is digi alisa ion. Digi alisa ion di e s i s
o all om digi isa ion. Digi isa ion en ails a mo e om analogue o digi al
(Ox o d English Dic iona y, 2023d). Fo ins ance, i migh be decided o
scan all pape eco ds o c ea e a digi al eplica, which is digi isa ion. In
ega d o hi ing, i was long common o conduc pen- and- pape psycho-
me ic es s, and i he same es is simply o e ed ia a compu e , his
would also be digi isa ion. Essen ial o digi isa ion is ha a digi al eplica
o an objec is c ea ed. As such, i a ain company decides o c ea e digi al
wins o es , o example, how di e en olling s ock pe o ms on acks,
his is a o m o digi isa ion. Digi alisa ion by con as is a b oade con-
cep and e e s o he adop ion o echnology by an o ganisa ion, coun y
o indus y (Ox o d English Dic iona y, 2023c). Fo example, i one
ans e s a pen- and- pape psychome ic es o an online e sion wi hou
any changes, hen he deli e y o he es changes bu he es i sel does
no change. Howe e , i one uses AI o p edic he bes candida e o a job,
he p ocess i sel changes. This is wha is commonly mean wi h digi alisa-
ion. Digi alisa ion is hus a e m ha desc ibes how p ocesses hemsel es
change due o he applica ion o a digi al echnology. Digi alisa ion is also
o en e e ed o as digi al ans o ma ion. Digi alisa ion i sel is a b oad
e m ha can en ail a my iad o o he e ms and echnologies.
One way in which p ocesses change h ough digi alisa ion is au oma-
ion. Au oma ion means ha a p ocess ha was p e iously done by a human
is done by a machine, o in o he wo ds, ‘[ ] he ac ion o p ocess o in odu-
cing au oma ic equipmen o de ices in o a manu ac u ing o o he p ocess
o acili y; (also) he ac o making some hing (as a sys em, de ice, e c.)
au oma ic’ (Ox o d English Dic iona y, 2023b). While some au oma ion
migh use AI, no all au oma ion will necessa ily use AI. An example o
au oma ion is he ca ac o y: adi ionally, humans would ha e assembled a
ca bu now obo s do he same job. This is o en con as ed wi h augmen-
a ion, whe e humans and machines collabo a e and augmen each o he ’s
skills (Raisch & K akowski, 2021).
INTRODUCTION: HUMAN-LIKE 9
One echnology ha is cen al o digi alisa ion is AI. The OECD has
de ined an AI sys em as ‘a machine- based sys em ha can, o a gi en se
o human- de ined objec i es, make p edic ions, ecommenda ions, o
decisions in luencing eal o i ual en i onmen s. AI sys ems a e designed
o ope a e wi h a ying le els o au onomy’ (OECD, 2019). The Eu opean
Union de ines an AI sys em in simila e ms as
a machine- based sys em ha is designed o ope a e wi h a ying
le els o au onomy and ha may exhibi adap i eness a e deploy-
men , and ha , o explici o implici objec i es, in e s, om he inpu
i ecei es, how o gene a e ou pu s such as p edic ions, con en ,
ecommenda ions, o decisions ha can in luence physical o i ual
en i onmen s.’
(Eu opean Pa liamen , 2024, p. 165)
Ano he de ini ion o AI is ha o AI as ‘in elligen agen s’ ha pe cei e hei
en i onmen and pe o m ac ions (Russell & No ig, 2021). In elligence
he e means ha a machine appea s o display human- like in elligence
(Russell & No ig, 2021).
Much o wha is commonly called AI is echnically machine lea ning.
Machine lea ning is a subse o AI. Machine lea ning means ha ‘a com-
pu e obse es some da a, builds a model based on he da a, and uses he
model as bo h a hypo hesis abou he wo ld and a piece o so wa e ha
can sol e p oblems’ (Russell & No ig, 2021, p. 669). Da a is cen al in
he machine lea ning p ocess; commonly, machine lea ning in ol es he
machine sea ching o pa e ns in he da a o de elop a model o wha
he machine pe cei es as ue in his con ex (B oussa d, 2018). This is
he aining o lea ning pa and he model is hen es ed wi h new da a
o see how accu a e p edic ions a e (B oussa d, 2018). I we wan o
de ine AI, a use ul de ini ion en ails he ways o analyse, de i e lea ning
om and make p edic ions based on da a (Kelan, 2024). Based on his
de ini ion, we can see ha da a is cen al o AI. We see ha exp essed
o en in he o m o Big Da a, which is hen analysed, lea ned om and
used as he basis o making p edic ions. Many w i e s on he u u e o
wo k he e o e liken da a o he ‘new oil’ (De C eme , 2020; F ey, 2019;
Schwab, 2018) o indica e ha da a is he new na u al esou ce ha has
o be mined.
INTRODUCTION: HUMAN-LIKE
16
in e wined. I he g owing impo ance o AI is discussed, i is hus cen al
o also explo e he gende dynamics en ailed in AI (Toupin, 2023).
Dis inguishing be ween gende by associa ion and by design is heo -
e ically use ul, bu in p ac ice, i is mo e di icul o dis inguish be ween
hese wo ideas. This is commonly ph ased in ela ion o conce ns ha
he lack o ep esen a ion o women and o he g oups in he design o
new echnologies leads o echnologies being biased (see also Chap e 6).
While he lack o women among he designe s o new echnologies could
be seen as an exclusion om po en ially luc a i e ca ee s, he idea ha
women as designe s au oma ically leads o gende - inclusi e design is
p oblema ic in i sel . Fi s , i igno es he social- shaping dynamics a ound
how echnologies a e designed, which migh , o ins ance, ollow com-
me cial conce ns. So e en i women ha e unique insigh s in o he wo ld,
which is o en exp essed as si ua ed knowledge (Ha away, 1991), hey
migh be unable o b ing his knowledge o bea on he design p ocess.
Second, i p esumes ha women designe s speak o all women. This
igno es ha a whi e, middle- class woman migh ha e di e en iews on
he wo ld han a Black, wo king- class woman. Women a e posi ioned
di e en ly by in e sec ing inequali ies (Acke , 2006; McCall, 2005;
Nash, 2008; C enshaw, 1989) and whose oices a e hea d and lis ened is
in luenced by complex powe dynamics (Spi ak, 1988; Mohan y, 1986).
Conside ing gende by associa ion and gende by design is a use ul ana-
ly ical de ice o unde s and he social shaping o gende and echnology
i such analyses conside how women migh be posi ioned di e en ly in
ela ion o echnologies.
In o de o illus a e e lec ions ha seeing gende and echnology as
mu ually cons i u i e can lead o, I wan o u n o a eading o he Tu ing
Tes ha is in o med by gende . The Tu ing Tes has become emblema ic
o seeing echnology as displaying human- like in elligence. The Tu ing Tes
is a hough expe imen by Alan Tu ing, he ma hema ician and compu e
scien is , o en associa ed wi h his wo k on code b eaking a Ble chley Pa k
du ing he Second Wo ld Wa . Tu ing’s hough expe imen (1950) discusses
he possibili ies a ound machines being ‘in elligen ’. In his w i ing, Tu ing
(1950) sugges s ha an in elligen machine should be able o pe o m a
game – e e ed o as he imi a ion game – ha was a popula pa y game
a he ime. The co e idea o his game is ha a human is unable o disce n
i he answe s gi en o igina e om a machine o a human (Su ko, 2020).
INTRODUCTION: HUMAN-LIKE 17
I a human canno dis inguish he answe om a human and a machine,
he machine passes he Tu ing Tes by displaying human- like in elligence.
Wha is commonly igno ed in he popula pe cep ion o he Tu ing Tes
is he gende dimension o he o iginal Tu ing Tes (Shah & Wa wick, 2016;
Su ko, 2020; Geno a, 1994; D age & F abe i, 2023). In o de o unde -
s and how he Tu ing Tes migh ela e o gende , i is necessa y o explain
he imi a ion game in mo e de ail. The imi a ion game has h ee playe s: a
man (A), a woman (B) and an in e oga o o ei he sex1 (C) (Tu ing, 1950;
Saygin e al., 2000). The aim o he game is o he in e oga o (C) o
de e mine who is a woman by asking ques ions such as abou hai leng h.
Bo h A and B mus con ince C ha hey a e indeed a woman. The in e o-
ga o (C) canno see ei he playe s A and B. The playe s should no use hei
oice o communica e because his migh gi e away gende . Ins ead, hey
communica e ia no es ha should ideally be yped up ia a elep in e
(Tu ing, 1950) o no allow conclusion abou gende om handw i ing.
Tu ing akes his game a s ep u he whe e a machine akes he place o
A. The mos common in e p e a ion o he Tu ing Tes is ha C now needs
o ind ou who is he human: A o B (Saygin e al., 2000)? Such an in e -
p e a ion en ails ha he aim o he game is no longe o con ince C who
he woman is bu ins ead who he human is. One migh also ead his scen-
a io as a way in which A and B a emp o appea as a human woman. Tu ing
himsel does no men ion ha he game has been al e ed om passing as
a woman o passing as a human. Two a gumen s a e used o suppo his
idea. Fi s , in ying o pass as a human woman, nei he he man no he
machine has an ad an age (Saygin e al., 2000). They bo h ha e o impe -
sona e a woman because in he modi ied e sion, A is no a man bu a
machine, whe eas B he woman is now B he human. A second a gumen
is ha Tu ing, as a gay man, migh ha e picked gende pu pose ully o
d aw a en ion o gende (Geno a, 1994; Hayes & Fo d, 1995). While hose
a gumen s ha e some pu chase, i seems mo e likely ha Tu ing in ac did
no mean o he game o be abou guessing gende , whe e he human and
he machine pass as a woman, bu a he abou i a machine can con ince a
human in o belie ing ha he machine is indeed human.
Howe e , he idea ha gende migh be cen al o he Tu ing Tes
p o ides in i sel an in e es ing hough expe imen . I he Tu ing Tes is
unde s ood as a man and a machine passing as a woman, his allows o
connec ions wi h gende heo ies whe e gende is o en seen as some hing
INTRODUCTION: HUMAN-LIKE
18
ha is done, achie ed and pe o med (Wes & Zimme man, 1987; Bu le ,
1990).2 I he Tu ing Tes is seen as a way o pass as a woman, hen he man
and he machine ha e o emula e adi ional ma ke s o eminini y and dis-
play hem. Fo example, in he con ex o he game we ha e seen ques ions
abou leng h o hai . A he ime, long hai was ese ed o women and as
such could be seen as a ma ke o eminini y. I bo h he machine and he
man p e end o ha e long hai o pass as a woman, hey ha e unde s ood
hese ma ke s o eminini y and a e able o use hem o pass as a woman. Fo
a machine o unde s and he ma ke s o eminini y, i would ha e needed
o lea n om da a ha women end o ha e long hai and men sho hai
o hen make he p edic ion ha ha ing long hai means ha he pe son is
mo e likely o be a woman. We will del e in o ques ions o how a machine
knows who is a woman in Chap e 5 in mo e de ail. Fo he ime being, i
su ices o say ha eading he Tu ing Tes om a gende and echnology
pe spec i e opens up no el esea ch ques ions o in es iga e in he u u e.
Resea ch Pa e ns
In his book, I wea e oge he ma e ials om a ange o se ings ha
oge he o m pa e ns ha can in o m us abou how gende and digi al-
isa ion a e mu ually shaped. I de ail my esea ch app oach in he appendix.
The i s esea ch con ex is books on he u u e o wo k ha a e w i en o
he popula ma ke . I selec ed books ha ocused on wo k and echnology.
I scanned he business p ess on book e iews, sea ched Amazon’s ecom-
mende sys em and asked o book ecommenda ions in my p o essional
ne wo k. I isi ed physical books o es o see wha was shel ed in sec ions
on he u u e o wo k. I also in e iewed hough leade s on he u u e o
wo k. I app oached indi iduals who had a isible p esence in discussions
abou he u u e o wo k such as by gi ing keyno es, being on a panel,
gi ing media in e iews o publishing epo s. Those indi iduals wo ked
in in e na ional o ganisa ions, policy and lea ned socie ies. I also spoke o
indi iduals who we e wo king in p o essional se ice i ms ad ising clien s
on he u u e o wo k and he ield o AI e hics. The indings om he ana-
lysis o books and he in e iews wi h hough leade s in o m Chap e 2 o
illumina e deba es on au oma ion and augmen a ion.
Building on he in e iews and he books, I hen ocused on explo ing
hose a eas ha we e seen as pa icula ly h ea ened by eme ging
INTRODUCTION: HUMAN-LIKE 19
echnologies: he p o essions. I spoke wi h indi iduals wo king in a ange
o p o essions such as audi , ax, legal, consul ing, inancial echnology and
a chi ec u e. I also spoke o a ange o indi iduals who we e expe s on
so- called new wo k p ac ices such as holac acy (Robe son, 2015), who
could alk abou how p o essional wo k can be o ganised in di e en ways.
I was in e es ed o explo e which skills a e changing in ela ion o ech-
nology and can be ained h ough echnologies such as VR. I was able o
conduc a o m o au o- e hnog aphy (Hine, 2020; Spa kes, 2003) by using
a VR headse (Oculus Ques 2). I unde ook aining in a ange o con ex s
and se ings such as counselling, heal h and sa e y, onboa ding o people
wo king in g oce y s o es and leading o inclusion. I analyse his ma e ial
in Chap e 3 o show which skills a e cons uc ed as uniquely human.
Ano he a ea ha was egula ly men ioned pa icula ly in he con ex o
algo i hmic bias was using AI in hi ing. In his book, I d aw on in e iews
wi h indi iduals who wo ked in a eas associa ed wi h hi ing echnologies.
These included indi iduals wo king in di e en unc ions such as hose
who p o ide he echnology o hi ing, hose who wo k in HR unc ions,
and ec ui e s and hi ing consul an s. In addi ion, I es ed some o he
hi ing echnologies mysel . This included VR ec ui men en i onmen s,
online ap i ude and pe sonali y es s and asynch onous ideo in e iews.
I la gely d aw on his ma e ial in Chap e 4.
The inal con ex om which I d aw ma e ial ela es o how da a and
gende a e in e wined in p oduc ion p ocesses a ound AI. My main in e es
he e was on how AI and gende a e mu ually shaping h ough p ocesses
such as da a labelling o da a anno a ion. I spoke o a ange o indi iduals
who we e in ol ed in he AI p oduc ion p ocess. Those indi iduals we e,
o ins ance, linguis s wo king in au oma ic speech ecogni ion, indi iduals
managing da a labelle s o expe s on how AI is c ea ed. Some in e iewees
showed me examples o how da a labelling was done and how gende
ma e s in he p ocess, which was help ul o unde s and hese p ac ices.
This ma e ial is la gely co e ed in Chap e 5.
While all o hese con ex s a e dis inc , he e was signi ican o e lap
be ween he opics ha came o he o e when he ma e ial was analysed.
As such, he di e en con ex s combine in speci ic ways o show pa e ns
o gende and digi alisa ion. These pa e ns we e signi ican ly in luenced by
being conduc ed a a speci ic poin in ime. This in luences which echnolo-
gies we e men ioned as examples. The pandemic i sel changed digi alisa ion
INTRODUCTION: HUMAN-LIKE
20
subs an ially and is said o ha e accele a ed digi alisa ion (Amankwah-
Amoah e al., 2021; Schlogl e al., 2021; McKinsey, 2020). Video con e -
encing mo ed om a a ely used echnology o he s anda d medium o
how wo k was done du ing mos o he pandemic. Wo k i sel has changed
due o he pandemic. Selec ing candida es using digi al means became a
necessi y o e nigh and has, as a consequence, e ol ed. A he same ime,
he pandemic mean ha he esea ch I was conduc ing was a he di e en
han I had planned (see he appendix). By wea ing di e en pa e ns ha
eme ge in hese con ex s oge he , his book shows how gende and digi -
alisa ion a wo k ope a e in di e en se ings and i also shows how hese
di e en pa e ns o m a la ge pic u e on gende and digi alisa ion in he
wo k con ex .
S uc u e o he Book
The book is s uc u ed in o wo b oad sec ions. The i s sec ion ocuses
on he discou se o he u u e o wo k. In his pa o he book, I analyse
common opes used in he books on he u u e o wo k and by in e iewees
such as he man- e sus- machine idea and how emo ions a e cons uc ed
as uniquely human. The second pa o he book ocuses on da a and da a
p ac ices. I analyse how algo i hmic bias mani es s in ela ion o hi ing and
wha his means o inclusion. I also show how da a p ac ices cons uc and
econs uc wo ld iews such as a ound gende and di e si y. Following his
in oduc ion, he book is s uc u ed in o ou chap e s and a conclusion.
How he u u e o wo k is imagined is a he cen e o Chap e 2. The
chap e analyses how dys opian iews eme ge in popula books on he
u u e o wo k, which pi ch humans agains machines in an epic ba le.
I discuss his as he man- agains - machine ope whe e machines eplace
humans. I also show ha hose who hold a - isk jobs a e imagined as men,
pa icula ly middle- class men in whi e- colla p o essions. The chap e
shows how an al e na i e, possibly mo e u opian pe spec i e akes hold
whe e humans and machines a e no enemies bu ac ually wo k oge he .
This human– machine collabo a ion is he alded as a new o m o di e si y.
In his scena io, humans engage in enjoyable and c ea i e asks whe eas
machines do he epe i i e and mundane wo k. The books I analysed main-
ain ha socio- emo ional skills a e ou o each o machines. Howe e ,
he chap e shows how cons uc ing socio- emo ional skills as sa e om
INTRODUCTION: HUMAN-LIKE 21
au oma ion lea es ou a wide conside a ion o how gende , ace and class
s uc u e cu en and u u e inequali ies.
In Chap e 3, I con inue his line o inqui y by ques ioning he idea
ha socio- emo ional skills a e indeed ou side o he ealm o machines.
This chap e d aws on in e iews wi h hough leade s and p o essionals.
I show how d udgewo k is widely assumed o be au oma able because i
ollows epea able pa e ns. I show how he au oma ion o d udgewo k
shi s asks in p o essional wo k and changes s uc u es in p o essional
i ms. The chap e shows how socio- emo ional skills a e cons uc ed as
uniquely human because hey a e seen as ou o each o machines. Looking
pa icula ly a examples whe e machines could be said o engage in socio-
emo ional skills, I show how machines a e ained o ecognise human
emo ions bu also how machines ain humans in showing app op ia e
emo ions. The chap e a gues ha socio- emo ional skills a e ollowing
pa e ns ha can be au oma ed and as such do no cons i u e a compe i i e
ad an age o humans o e machines. Howe e , i hose socio- emo ional
skills a e pe o med by machines will depend o a la ge deg ee on he social
desi abili y o ha ing hese asks pe o med by machines.
Chap e 4 hen engages wi h he ques ion o how echnologies a e used
in hi ing p ac ices. Hi ing is a cen al unc ion o he o ganisa ion: i is
impo an o ha e he igh people in place o ensu e ha he o ganisa-
ion can ul il i s pu pose. Ye hi ing is also a p ocess augh wi h human
bias. This is whe e echnology comes in because i p omises o make hese
p ocesses no only mo e e icien bu also mo e objec i e. Howe e , ech-
nologies ha e been shown o epea and ampli y exac ly hese human biases.
The chap e aces how a echno- op imis ’s pe spec i e ha echnology
imp o es business p ocesses can be econciled wi h algo i hmic bias. This
chap e d aws on in e iews wi h expe s on he u u e o wo k, hose
who design hi ing echnologies, as well as my own expe iences wi h hose
echnologies. Fi s , I show ha a echno- op imis ’s s ance is he dominan
pe spec i e in mos o he in e iews and ha his s ance is main ained
by cons uc ing algo i hmic bias as ul ima ely ixable. The chap e de ails
how i is sugges ed ha bias eme ges om people and which p ocesses
and p ac ices a e cons uc ed as being able o ix algo i hmic bias. I also
show ha some in e iewees displayed wha I call echno- hesi a ion, a
wai - and- see s ance, which acknowledged ha AI- suppo ed hi ing will
become no malised o e ime. O e all, he chap e makes he a gumen
INTRODUCTION: HUMAN-LIKE
22
ha algo i hmic bias has o be cons uc ed as ul ima ely ixable o main ain
a echno- op imis ic s ance.
Chap e 5 aces he ques ion o how a compu e knows abou he
gende o he pe son. I show ha da a labelling is cen al in his p ocess. The
chap e cen es on wo aspec s. Fi s , i ocuses on who is doing da a label-
ling wo k. The chap e discusses AI’s hidden wo k o ce – hose who label
da a in he AI supply chain. Second, he chap e discusses he cons uc ions
o eali y ha eme ge om da a labelling. I discuss p ac ices such as how
classi ica ions used in da a labelling ep esen speci ic wo ld iews. The
chap e sugges s ha wha is p esen ed as objec i e and uni e sal is in ac
subjec i e and pa ial and as such po en ially exclusiona y. The chap e
sugges s ha in o de o c ea e mo e inclusiona y app oaches, i is cen al
o make he unde lying p ocesses o cons uc ions ha happen in ela ion
o da a labelling isible.
Chap e 6 o e s a conclusion by wea ing di e en h eads ha he book
unco e ed oge he . I show which u u e- shaping pa e ns a ound digi al-
isa ion and gende eme ge. The chap e illus a es how seemingly isola ed
issues o m pa e ns ha ei he hinde o os e inclusion. I specula e how
al e na i e u u es migh be c ea ed. The inal chap e also sugges s ha da a
is inhe en ly social and hus ine i ably biased. Howe e , while his migh
lead o exis ing gende pa e ns being epea ed, I sugges ha al e na i e
pa e ns a e possible ha can be mo e inclusi e.
Conclusion
In his opening chap e o he book, I show how gende , digi alisa ion and
he u u e o wo k can be concei ed as pa e ns. Pa e ns a e cen al o
echnologies such as machine lea ning ha ead exis ing pa e ns o p e-
dic po en ial u u es. This book ocuses on di e en indi idual pa e ns
ha i b ough oge he show mo e complex and dynamic pa e ns o how
gende and echnology in e ac in he wo k con ex . The chap e s a ed by
a guing ha how u u es o (paid) wo k a e imagined shapes how po en ial
u u es migh un old. I hen sugges ed ha many new echnologies appea
magical and my hical o use s. I de ined key e ms o echnologies I use in
his book and how I employ hese e ms. In he nex sec ion, I explained
how I see gende and echnology as co- cons uc ed and how seeing gende
as pe o med in and h ough echnologies is a use ul lens o his esea ch.
INTRODUCTION: HUMAN-LIKE 23
Following his, I ou lined how he esea ch was cons uc ed be o e p o-
iding de ails on how his book is s uc u ed. The book wea es oge he
di e en pa e ns o gende and echnology ha eme ge in speci ic se ings.
By wea ing hese pa e ns oge he , a mo e nuanced, complex and dynamic
pa e n o how gende and digi alisa ion in e ac in he u u e o wo k
eme ges. The book hus a gues ha pa e ns o gende and digi alisa ion a e
dynamic and complex bu can be used o c ea ing inclusion.
No es
1 Sex is used in he o iginal.
2 I ha e discussed di e ences and simila i ies o hese app oaches o how gende is
done and pe o med in de ail elsewhe e (Kelan, 2009, 2010).
This chap e has been made a ailable unde a CC BY – A ibu ion license.
DOI: 10.4324/9781003427100-2
2
IMAGING FUTURES
BETWEEN AUTOMATION
AND AUGMENTATION
In oduc ion
Discou ses on he u u e p o ide empla es o how po en ial u u es migh
un old. These empla es ep esen pa e ns based on which hinking o he
u u e is s uc u ed. Such discou ses o he u u e egula ly play wi h he
con as be ween u opian and dys opian isions o he u u e (Jasano ,
2015; Schlogl e al., 2021; Howc o & Taylo , 2023). Fo discou ses on
echnology a wo k, hese isions mani es in ela ion o au oma ion and
augmen a ion (Kelan, 2023b; Raisch & K akowski, 2021). Au oma ion
p esen s a ca as ophic iew o he u u e o a dys opia whe e machines ha e
aken- away jobs ha humans used o do, leading o mass unemploymen
and social un es . Au oma ion cons uc s humans and machines as enemies,
wi h bo h compe ing o he same wo k. Augmen a ion, in con as , means
ha humans and machines collabo a e o achie e wo k oge he by playing
o each o he ’s s eng hs. In his ision o he u u e, humans and machines
collabo a e seemingly ha moniously. Humans a e said o pi o o skills ha
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 25
machines a e p esumed o s uggle wi h: socio- emo ional skills. Socio-
emo ional skills a e o en seen as some hing ha only humans can do.
In his chap e , we will see how au oma ion and augmen a ion a e no
only egula ly in oked concep s in he books on he u u e o wo k ha
I analysed, bu hey also esona e s ongly wi h gende (Kelan, 2023b).
Discussions o au oma ion egula ly e e ed he man- e sus- machine
ope. Fu he mo e, he jobs ha we e a isk o au oma ion o en belonged
o men. The p o essions we e p edic ed o disappea , ye he ole women
play in he p o essions was neglec ed. Augmen a ion in con as was
associa ed wi h he ise o socio- emo ional skills, which we e cons uc ed
as a co e human skill ha bo h women and men can display. Technologies
we e discussed as gende ed, pa icula ly in he con ex o algo i hmic bias.
This chap e aces how books on he u u e o wo k in ela ion o au oma-
ion and augmen a ion ela e o gende . I show how discou ses o he u u e
o wo k eplica e some gende pa e ns bu also c ea e new pa e ns. This
chap e discusses he con ou s o some o he pa e ns ha will be discussed
in g ea e de ail h oughou he book.
Visions o he Fu u e o Wo k
P io o he Co id- 19 pandemic, one o he mos p essing ques ions egu-
la ly asked in he media was ‘will a obo ake you job?’. The idea o obo s
s ealing jobs is an es ablished ope in discussions abou echnology. Such
discussions a e o en ollowed by a men ion o Luddi es des oying ex ile
machine y. These images pi ch humans agains machines. As machines a e
deployed, humans lose hei jobs. Al hough esea ch has shown ha au o-
ma ion eplaces and c ea es new wo k (Au o , 2015; Au o e al., 2023),
he idea ha humans a e eplaced by he la es echnologies associa ed
wi h digi alisa ion pe mea es popula imagina ies such as he mul i ude
o epo s on he u u e o wo k (OECD, 2016; F ey & Osbo ne, 2017;
Manyika, Chui, e al., 2017; Manyika, Lund, e al., 2017). These wo ying
scena ios abou he u u e o wo k seem o sugges ha mass unemploymen
will des abilise economies and socie ies unless u gen policy in e en ions
a e aken. The Co id- 19 pandemic has, a leas empo a ily, called in o
ques ion such scena ios. Ins ead, ideas such as he ‘g ea esigna ion’ and
he ‘g ea a i ion’ sugges ha people no longe wan o wo k like be o e
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION
32
e e enced by Baldwin, Benana , de C eme , F ey, McA ee and B ynjol sson,
and Schwab, and o en discussed in he con ex o he human being bea en
by he machine ha has become in elligen enough o do so.
The man- e sus- machine ope also inds i s exp ession in au oma ion
anxie y, which de C eme , Schwab and Susskind use o e e o he ea ha
a obo migh ake you job. The idea ha he e is less wo k le o humans
is also cen al in Me iso is’ book. De C eme concludes ha he hinking
o man- e sus- machine, oge he wi h scien i ic e idence on au oma ion
anxie y, cen es mos discussions on AI’s po en ial o eplace people’s jobs.
In a simila ein, Susskind sugges s ha machines con inually imp o e
hei pe o mance, which limi s ac i i ies whe e humans ha e he edge. Fo
Susskind, his challenges he idea ha humans a e supe io and could no
be eplaced. Ins ead, he sugges s ha an ‘in e io i y assump ion’ migh be
mo e accu a e in ha machines a he han humans become he no m o
pe o ming asks.
A a ia ion o au oma ion anxie y ha Daughe y and Wilson and de
C eme e e ence is algo i hm a e sion. Daughe y and Wilson desc ibe
algo i hm a e sion as he phenomenon ha people us humans mo e
han machines, whe eas de C eme no es ha i means ha people a oid
aking ad ice om algo i hms. De C eme explains he suspicion owa ds
algo i hms wi h he me apho o he black box. Gi en he ac ha
algo i hms unc ion like black boxes, people a e scep ical abou algo i hms
making au onomous decisions. In ac , he black box idea is ano he
common me apho used ha appea s in i e books (Baldwin, Daughe y
and Wilson, Schwab, Wes , and o cou se, de C eme , as discussed p e i-
ously). I commonly desc ibes ha he ecommenda ions ha algo i hms
make a e opaque, e en o hose who design hose sys ems. The lack o
explainabili y is a p oblem ha Daughe y and Wilson, and Baldwin
men ion. Wes , in con as , e e s o how he Eu opean Union’s Gene al
Da a P o ec ion Regula ion is add essing he black box by gi ing indi-
iduals insigh in o how he black box ope a es. The black box me apho ,
oge he wi h a discussion o algo i hm a e sion, is mobilised o explain
why au oma ion anxie y is a conce n.
Howe e , while he man- e sus- machine ope is egula ly ound in he
books I analysed o exp ess au oma ion anxie y, I also ound ha augmen-
a ion as humans and machines collabo a ing was discussed in he books.
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 33
Daughe y and Wilson a e possibly he mos explici in ha hey ans o m
he man- e sus- machine ope in o human + machine o a gue o he
alue o augmen a ion. I is also no able ha in he book i le, Daughe y and
Wilson use human + machine, which challenges he a he exclusiona y
no ion o he man- e sus- machine ope. In some ins ances, hey alk abou
man + machine as a di ec con as o man e sus machine. Daughe y and
Wilson explain ha i is common o see humans and machines as i als
whe e machines s eal humans’ jobs. Howe e , in hei book, hey s ess ha
machines and humans collabo a e, which ep esen s augmen a ion a he
han au oma ion and eplacemen . Simila ly, de C eme p esumes ha
humans and machines will collabo a e, and he calls his he new di e si y.
De C eme w i es an en i e sec ion abou his new di e si y. No ably, di e -
si y is no e e ing o di e si y among humans, bu his new di e si y ha
de C eme desc ibes lies be ween humans and machines. Like wi h human–
human di e si y, de C eme cau ions ha human– machine di e si y will
appea alien o many humans. This is due o he ac ha we ha e been
condi ioned o hink abou humans and machines as an agonis s a he han
as symbio ic. As such, his new o m o di e si y is in ac esona ing wi h
augmen a ion.
Gi en he popula i y o he man- e sus- machine ope in popula cul-
u e, i is no su p ising ha he ope was egula ly used in he books
I analysed. The man- e sus- machine ope was in oked di ec ly bu also
indi ec ly; o ins ance, when he black box o AI and algo i hmic a e sion
was discussed. Howe e , i is no able ha augmen a ion was p esen ed as
an al e na i e o au oma ion, whe e machines eplace humans. In ac , his
augmen a ion was alked abou as new di e si y, whe e di e si y e e s
o humans and machines collabo a ing. Ye , e en hese collabo a ions
be ween humans and machines we e ma ed by he po en ial nega i e
impac o humans seeing machines as enemy. The ob ious c i icism o he
man- e sus- machine ope is ha i could be ead as gende exclusiona y.
While clea ly, some au ho s make a emp s o be mo e gende inclusi e by
using human + machine, he man- e sus- machine ope is la gely used
o a icula e au oma ion anxie y and o show how augmen a ion migh
be hampe ed by humans eeling hos ile o echnology. I will now u n o
unde s anding in a mo e g anula way who is expec ed o lose ou in his
epic ba le be ween humans and machines.
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION
34
Men’s Jobs
In he p e ious sec ion, we ha e seen how he man- e sus- machine ope is
commonly used o exp ess au oma ion anxie ies. I will now show ha i is
men who a e cons uc ed as a isk o seeing hei jobs disappea . The isk
ha men migh lose hei jobs is leading o a heigh ened conce n o he
u u e o wo k. Fo example, F ey s a es ha i is men who a e mo e likely
o be eplaced by obo s han women. The e is also conce n ha men in
hei p ime will ace edundancy bu lack he lexibili y in hei iden i ies
o mo e o al e na i e jobs, as Susskind says. This a ec s men in whi e-
and blue- colla jobs and hus spans di e en class backg ounds. As such,
he a gumen ha men who lose hei jobs a e unable o ind new wo k,
which a ec s hei sense o sel , has o be ead in he con ex o men’s
adi ional ole in socie y. Men’s adi ional ole in Wes e n socie ies en ails
being a b eadwinne . This is, o ins ance, add essed by Benana . Benana
sugges s ha he concep o he male b eadwinne as he head o he house-
hold, and women as he main ca egi e s ea ning supplemen al incomes, is
deeply ensh ined in he sociocul u al ab ic o many economies. He ci es
mini- jobs in Ge many, which he sugges s a e e ec i ely designed o be
done by s ay- a - home wi es, whose incomes supplemen hose o he main
male b eadwinne s. This adi ional a angemen is ewa ded by he s a e
h ough ax incen i es.
The gende - seg ega ed na u e o he wo k o ce, wi h men and women
being clus e ed in di e en jobs, o e e ences o how women and men
migh be posi ioned in hese new u u es o wo k, is a bes ma ginal
in he books. Howe e , F ey and Susskind discuss pink- colla wo k.
Pink- colla wo k has adi ionally been done by women and is he e-
o e associa ed wi h he colou pink. Susskind alludes o he ac ha he
naming o pink- colla wo k is un o una e. He goes on o explain ha
men who miss ou on blue- colla wo k a e o en unwilling o ake on
pink- colla wo k. He desc ibes his as p oblema ic because many pink-
colla jobs a e a he momen ou o each o machines. F ey in u n
discusses how pink- colla wo k became mo e impo an wi h he in o-
duc ion o he ypew i e . He he e links job g ow h o mechanisa ion.
A he same ime, he acknowledges ha he g ow h o he pink- colla
wo k o ce came o an end in he 2000s as compu e s became ubiqui ous.
Howe e , he does no see hese ends as a ec ing women nega i ely
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 35
because he s a es ha women made in oads in o well- paid jobs, so much
so ha , as he s a es, younge women in he Uni ed S a es now ou - ea n
hei male coun e pa s. O e all, hese cons uc ions lea e he eade
wi h he imp ession ha women a e doing well, in spi e o au oma ion
h ea s. Ye , he e is a conce n o men who lose hei jobs bu a e no
lexible enough in hei iden i ies o ake on o he ypes o wo k ha a e
seeing g ow h because such wo k is associa ed wi h women.
In his sec ion, we ha e seen ha men’s jobs, bo h in blue- and whi e-
colla jobs, a e cons uc ed as pa icula ly a h ea o au oma ion. Women’s
jobs in con as a e no seen as unde h ea o au oma ion. Pink- colla
wo k, o ins ance, is cons uc ed as u u e- p oo and i is also s a ed ha
women in gene ally do well p o essionally. This lea es he eade wi h he
imp ession ha we need o ocus on he demise o men’s jobs. While blue-
colla jobs a e a ec ed by au oma ion and ha e been a ec ed by i o a long
ime, a new h ea is iden i ied in he books: he h ea o whi e- colla jobs.
Disappea ing P o essions
Al hough bo h blue- and whi e- colla jobs a e cons uc ed as a isk, i is
pa icula ly whi e- colla wo k ha many o he au ho s o he u u e o
wo k books ocus on. Whi e- colla wo k is la gely seen as desi able wo k
in he books. Baldwin s a es, his o ically, blue- colla jobs we e a ec ed by
au oma ion, which means ha whi e- colla and p o essional jobs we e
shel e ed om obo s and globalisa ion un il now. He ocuses speci ically
on whi e- colla obo s ha a e eplacing middle- class jobs. Al hough whi e-
colla obo s a e no ye as good as whi e- colla humans, obo s a e simply
mo e cos e ec i e: Baldwin s a es ha a whi e- colla obo cos s a i h o
a wo ke in he de eloped wo ld and a hi d o a wo ke in less de eloped
a eas. I should be no ed ha whi e- colla obo s will no ake o e en i e
occupa ions. Howe e , hey migh well ake o e speci ic asks, which o e
ime can educe he need o human whi e- colla wo ke s o e all.
Whi e- colla and p o essional jobs a e also hose ha equi e an up on
in es men in o educa ion ha is hen paying o wi h highe li e ime
ea nings. Benana desc ibes his as he idea ha a good educa ion will lead
o and ensu e a good middle- class job. Howe e , he idea ha a good edu-
ca ion will lead o a good middle- class job is now changing, acco ding o
he books I analysed. Simila ly, Susskind alks abou he signalling en ailed
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION
36
in ha ing a good college deg ee. Whe eas in he pas , a good college deg ee
would be impo an o ind a job and climb he occupa ional ladde , he
good college deg ee has los i s signi icance. F ey p o ides a a ionale why
whi e- colla wo k is well paid: he a gues ha in he Second Indus ial
Re olu ion, he oppo uni y cos s o educa ion dec eased because highe -
le el skills we e in demand; as a esul , whi e- colla wo ke s we e paid well
o hei educa ion. A d ama ic change is now ha a good educa ion no
longe gua an ees ha indi iduals can achie e and sus ain a middle- class
li es yle. I middle- class wo ke s a e being eplaced by echnology, his is
expec ed o ha e wide- anging e ec s. F ey, o ins ance, p esumes ha he
eplacemen o middle- class wo ke s by machines will dec ease demand o
local se ices. Ea lie edundancies a e cons uc ed as ha ing had limi ed
e ec s, as i was possible o ind o he p o essional jobs elsewhe e. A cen al
conce n o u u e- o - wo k w i e s is hus ha secu ing a p ospe ous u u e
by in es ing in educa ional c eden ials no longe applies.
The link be ween educa ion and ca ee is nowhe e mo e isible han in he
p o essions. Baldwin de ails ha p o essional jobs we e shel e ed om glo-
balisa ion and obo s because hey equi ed ace- o- ace con ac , which no
longe is he case. Benana ci es Robe Reich’s con en ion ha echnology
is eplacing p o essional jobs, and Schwab sugges s ha au oma ion is now
eplacing p o essional wo ke s such as accoun an s and lawye s a he han
ac o y wo ke s. The books men ion p o essional wo k in inance (Baldwin,
de C eme and Schwab) and accoun ing (F ey, Schwab and Me iso is).
Howe e , he p o ession a ac ing he mos a en ion is legal wo k, which
is discussed in se en books. Baldwin p o ides a ious examples o how
legal wo k is being au oma ed using so wa e, such as Lex Machina and
Ra el Law, which helps o so h ough in o ma ion and e en sugges s legal
s a egies. This means ha many asks ha junio lawye s would ha e ad-
i ionally done a e now au oma ed. Baldwin summa ises ha whe eas a law
deg ee was a secu e way o ensu e middle- class p ospe i y, whi e- colla
obo s a e now compe ing wi h junio lawye s. Acco ding o de C eme ,
in he legal wo ld, au oma ed ad iso s a e used o con es pa king icke s,
and Susskind obse es ha au oma ed documen e iew sys ems can scan
ma e ial mo e swi ly and o en mo e accu a ely. Susskind men ions how
a law i m uses so wa e o educe he ime human lawye s ha e o spend
on asks. Wes men ions an AI- d i en bank up cy legal assis an . Mos o he
books sugges ha legal wo k is a high isk o au oma ion, and only F ey
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 37
oices a dissen ing iew. Al hough F ey acknowledges ha legal lib a ies a e
a ailable online, he ci es a s udy calcula ing ha only 13% o legal asks can
ac ually be au oma ed, suppo ing his own p edic ion ha legal wo k is a
low isk o being au oma ed.
The common eno in he books is ha p o essional jobs whe e educa-
ion is ewa ded a e disappea ing. These p o essional jobs a e p esumed o
be held by men. This is no o say ha hese books a e unawa e ha women
ake p o essional jobs. In ac , he books s ongly signal gende awa eness.
Fo ins ance, au ho s a e egula ly ci ing s o ies o an enginee , a p o esso ,
a so wa e de elope o a su geon who u n ou o use he p onoun ‘she’.
By using he p onoun ‘she’, he au ho s b eak he implici assump ion ha
hese p o essional jobs a e held by men only. Howe e , wha is missing om
he discussions is an acknowledgemen ha women ha e made in- oads
in o p o essional wo k. O cou se, hese p o essions, as much esea ch has
shown, a e a om gende inclusi e (Ely, 1995; Walsh, 2012; Lupu, 2012;
Koko - Blamey, 2021, 2023). Howe e , mos books ail o discuss he ac
ha he p o essions ha e become mo e di e se o e he yea s and women,
in pa icula , a e inc easingly p esen in he p o essions. By no discussing
how he gende p esen a ion has changed he p o essions in g ea e de ail,
eade s will be le wi h he imp ession ha hese jobs a e held by men. This
is pa icula ly he case because men in gene al a e singled ou as pa icu-
la ly a ec ed by he au oma ion, as we ha e seen in he p e ious sec ion.
A second aspec ha is a ely discussed is how he p o essions a e chan-
ging. While some au ho s p edic he dea h o he p o essions, a leas F ey
seemed mo e scep ical in ega d o wha he u u e o he legal p o ession
holds. Howe e , in eali y, he p o essions migh be ans o med wi h some
aspec s being done wi h he help o echnology, while o he asks migh
gain mo e p ominence. As such, he con en o p o essional wo k migh
change, equi ing subs an ial changes in ega d o how p o essional wo k
is o ganised and how aining is s uc u ed in hose i ms. While mos o
hese ques ions a e beyond he scope o he books analysed, I will e isi
some o hose ques ions in Chap e 3.
Socio- Emo ional Skills as Human Ad an age
I is e iden ha he dys opian images used in he u u e o wo k books
I analysed ela e o au oma ion and pa icula ly he au oma ion o
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION
38
p o essional jobs. Ye , some o he books also ske ch he con ou s o a mo e
u opian u u e o wo k. This en ails ha humans and machine collabo a e
and enhance each o he ’s skills. In o he wo ds, he books alk abou aug-
men a ion (Kelan, 2023b; Raisch & K akowski, 2021). Ano he a ea ha
he books add ess a e hose jobs ha a e ou o each o machines. These
jobs a e cons uc ed as u u e- p oo . Bo h he jobs ha a e augmen ed by
echnologies and hose ha a e sa e om au oma ion sha e in common
ha hey ely on a speci ic se o human skills. I call hese skills socio-
emo ional skills and will discuss in his sec ion how hese socio- emo ional
skills a e cons uc ed as he key compe i i e ad an age ha humans ha e
o e machines.
By and la ge, he skills ha a e said o be u u e- p oo a e hose ha a e
seen as di icul o machines o accomplish. These include leade ship (de
C eme , McA ee and B ynjol sson), eamwo k (McA ee and B ynjol sson),
c ea i i y (De C eme , Me iso is and Schwab), coaching (McA ee and
B ynjol sson) and, in gene al, any hing ha equi es socio- emo ional
skills. I use socio- emo ional skills because he au ho s o he book ha e
di e en names o such skills. Fo example, Me iso is ci es Anne- Ma ie
Slaugh e saying ha he economic o ma ion has mo ed om hi ing
hands o e hi ing heads o hi ing hea s, whe e hea s a e used o p e-
sen socio- emo ional skills. McA ee and B ynjol sson use he example ha
when ecei ing a medical diagnosis, people p e e o ecei e hose om
compassiona e people a he han machines. This means ha compassion
is cons uc ed as a desi able skill in humans. Susskind uses he e m ‘social
in elligence’ and s a es ha echnologies canno deal wi h asks equi ing
social in elligence well, such as p o iding empa hy o ace- o- ace in e -
ac ion. De C eme a gues ha algo i hms lack wha he calls social skills.
Baldwin, Me iso is, McA ee and B ynjol sson, and Schwab alk abou social
and in e pe sonal skills, Baldwin and Susskind abou social in elligence, De
C eme abou emo ional in elligence, and De C eme , Me iso is, McA ee
and Baldwin also use empa hy. Gi en he ac ha au ho s use a ange o
e ms o desc ibe such skills, I use socio- emo ional skills as an umb ella
e m o hese skills. I is socio- emo ional skills ha a e cons uc ed as
uniquely human and as ha d o eplace by machines.
The books cons uc jobs ha en ail socio- emo ional skills as sa e
om au oma ion. These include medicine (F ey), eaching (Baldwin and
Susskind) and social wo k (Susskind). Ano he a ea o g ow h a e pe sonal
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 39
se ices, which en ails beach body coaches, yoga ins uc o s and Zumba
ins uc o s, which a e men ioned in a ious combina ions by Susskind,
Wes , McA ee and B ynjol sson, and F ey. Common o hese a eas o wo k
is mo i a ing people h ough socio- emo ional suppo , and he assump ion
is ha machines could no p o ide his socio- emo ional suppo . I is
in e es ing ha hese pa icula a eas o wo k a e used o illus a e he
impo ance o socio- emo ional skills, while changes in he p o essions
owa ds mo e socio- emo ional skills we e a ely discussed.
The ise o socio- emo ional skills is o cou se no new. As a ma e o
ac , I explo ed his in ea lie esea ch, whe e I analysed he books on he
u u e o wo k ha exis ed a he ime (Kelan, 2008b). In hose books, i
was no able ha socio- emo ional skills we e cons uc ed as eminine and
p esumed o eside in women. Ye , his was no he case in he cu en
ba ch o books ha I analysed o his esea ch. The closes o such asso-
cia ion was in ela ion o paid ca e wo k. Susskind and Wes , o ins ance,
men ion ha women o en wo k in ca ing p o essions, and Susskind e en
poin s ou ha ca ing wo k is o en unde alued. One could now p e-
sume ha ca e wo k is sa e om au oma ion, which is indeed alluded o.
Howe e , Wes cas s some doub on whe he such jobs a e indeed u u e-
p oo . Wes e e ences ha echnology is changing ca e gi ing. This migh
mean ha socio- emo ional skills migh e en ually also cease o be cen al
o ca e- gi ing oles, bu his was an opinion ha was no p edominan .
O e all, he image ha eme ges in he books is ha socio- emo ional skills
a e uniquely human. As a consequence, jobs ha equi e socio- emo ional
skills a e cons uc ed as sa e om au oma ion. Whe eas ea lie esea ch has
shown how socio- emo ional skills a e o en p esumed bu no ewa ded in
women, in hese books, socio- emo ional skills a e p esen ed as gende neu-
al and as i e e yone could engage in hem. I will e u n o he ques ion o
i socio- emo ional skills a e indeed uniquely human in Chap e 3.
C ea ing he Human– Machine In e ace
A e discussing he p esumed impo ance o socio- emo ional skills, I would
now like o discuss how machines and humans collabo a e o augmen one
ano he . I will ocus pa icula ly on echnical p o essions he e because hey
a e egula ly men ioned in he books. The au ho s call his he human–
machine in e ace (Baldwin, Naugh y and De C eme ), which includes
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION
40
machine lea ning enginee s, da a scien is s and big da a a chi ec s (F ey).
Commonly, hese a eas o wo k appea a emo ed om socio- emo ional
skills. Howe e , as we will see, he way hese a eas o wo k a e desc ibed
indica es ha hey ely on a o m o socio- emo ional skills. Daughe y and
Wilson, in pa icula , desc ibe how machine lea ning p og amme s a e
becoming mo e like eache s who ain he algo i hms. This is di e en
o how p og amming was adi ionally pe cei ed – as w i ing code ha a
machine execu es. Now, a p og amme seems akin o a eache , who eaches
no child en bu machines.
Mos o he books a e conce ned wi h high- end AI wo k. The books dis-
cuss he skills sho age in jobs ela ed o AI, which is said o limi he de el-
opmen in he ield (Baldwin). This means ha oo ew humans a e able o
do hese highly skilled jobs ha a e equi ed o make AI unc ion. Howe e ,
a he han aining mo e indi iduals, he solu ion he sugges s ecu s o
echnology. Baldwin sugges s ha Google’s au oma ed machine lea ning,
whe e machines ain o he machines, migh be a solu ion o his skills
sho age.
Howe e , Daughe y and Wilson also discuss how new echnolo-
gies equi e o he expe s ha ha e speci ic skill se s ha one migh no
adi ionally associa e wi h echnology wo k. Such expe s know abou
human con e sa ion, humou and empa hy. Those expe s can each he
echnology o emula e socio- emo ional skills. Daughe y and Wilson p o-
ide a ange o examples o such oles: a poe , no elis and playw igh
a Mic oso ’s Co ana and a ehicle design an h opologis a Nissan. This
socio- emo ional wo k o machines appea s o be needed in o de o
design echnology wi h which humans enjoy engaging and ha is i o
pu pose. To achie e his, echnology mus emula e humans, and acco ding
o Daughe y and Wilson, humans a e bes placed o enable his. This
means ha many eme ging jobs will p o ide a so o a ec i e labou o
machines.
The e is compa a i ely li le discussion abou he demog aphic back-
g ound o hose who design AI in he books. Schwab men ions ha women
hold less han 25% o IT jobs. This is p oblema ic o him because i means
ha many ideas a e no conside ed, which in u n is cons uc ed as a hin-
d ance o he de elopmen o wha he calls he Fou h Indus ial Re olu ion.
Howe e , apa om ha , he lack o di e si y in ega d o gende and
o he dimensions o di e ence is a ely discussed. This is no able because
IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 41
i he e is a skills sho age, i is commonly claimed ha women should be
mobilised o ake hese oles, pa icula ly i hey in ol e a socio- emo ional
skills componen . Howe e , such sugges ions a e no commonly made in
he books.
Ano he a ea ha seems less well discussed in he books is he wo k in
AI supply chains. In con as o he much in- demand designe s o AI, jobs
ha in ol e p epa ing da a o machine lea ning h ough, o ins ance, da a
labelling o da a anno a ion, a e a ely discussed. The no able excep ion is
he book by Daughe y and Wilson, which e e ences he wo k ha can
easily be ou sou ced o c owd- sou ced, such as aining AI. We will e u n
o how such ypes o wo k in e sec wi h gende in Chap e 5.
Al hough he skills sho age in ela ion o AI wo k is egula ly in oked
in he books, he e a e also signs ha he adi ional jobs associa ed wi h
c ea ing eme ging echnologies a e changing. I is sugges ed ha pa s o
p og amming jobs can be au oma ed by machines p og amming hem-
sel es – o a he using ools ha gene a e code. As men ioned be o e,
p og amme s a e p esumed o become like eache s o aine s. Addi ionally,
expe s on socio- emo ional skills will be equi ed o each machines how
o communica e wi h humans. Howe e , we ha e also seen ha he gende
composi ion o hese jobs is o en neglec ed. Simila ly, less high- end wo k
such as ha equi ed o p epa e da a o machine lea ning is less egula ly
he ocus o a en ion.
Gende ed by Design
Gende and echnology a e no a he co e o any o he books, bu some
o he books do p o ide in e es ing illus a ions o how gende is en e ed
in o he deba e. Schwab, in pa icula , discusses wha is akin o echnology
being gende ed by design. Schwab sugges s ha how machines a e p o-
g ammed and how hey in e ac is impac ed by sexism and acism. In his
iew, obo s, pa icula ly humanoid obo s, a e no longe bound by ace
and gende when hey a e designed. Ye , cus ome se ice obo s o en dis-
play emale cha ac e is ics and indus ial obo s ea u e male cha ac e is ics.
In Schwab’s iew, a he han challenging old s e eo ypes, he same s e eo-
ypes a e epea ed. He a gues ha he well- being o all indi iduals would
be inc eased i mo e conscious choices a e made du ing he de elopmen
o echnologies, which would hen a oid epea ing he same s e eo ypes.
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
48
chap e , I discuss how echnologies allow o such aining o be conduc ed
in new o ma s such as in VR.
Whe eas Chap e 2 ocused on books ha discuss he u u e o wo k,
his chap e is based on he in e iews ( u he de ails can be ound in
he appendix). In he in e iews, I asked in e iewees abou changes ha
hey pe cei e in ela ion o he u u e o wo k and echnology and how
hey saw issues such as a ound au oma ion and augmen a ion. The unde -
lying ma e ial o he chap e is in na u e e y di e en o he books.
Whe eas he books p esen ed ca e ully a gued and e idenced ideas ha had
unde gone no mal book publishing p ocesses, he in e iews ep esen
occasioned answe s. While he answe s we e na u ally less polished han
ex published in a book, hey p o ided a pa icula ly ich can as o ana-
lyse discu si e pa e ns ha a e egula ly mobilised a ound he u u e o
wo k, echnology and gende . The answe s o en e lec ed he expe iences
o in e iewees, including e lec ions on changes ha hey had pe cei ed
in he wo kplace. Some alked abou new ways o wo king, whe eas o he s
discussed echnologies hey had de eloped. In some cases, I was able o
y some o hose echnologies mysel . I was he eby able o add an add-
i ional ace o he discussions h ough my own expe ience wi h hese
echnologies.
This chap e he eby ques ions in how a socio- emo ional skills a e
uniquely human and ou o each o machines. I i s show how d udgewo k
is expec ed o be au oma ed because i ollows epea able pa e ns. I hen
ou line which consequences his migh ha e o ask p o iles and s uc u es
in p o essional wo k, be o e showing how socio- emo ional skills a e
cons uc ed as a compe i i e ad an age o humans o e machines. I hen
que y i socio- emo ional skills a e uniquely human by showing how socio-
emo ional skills can be unde s ood as pa e ns ha can be au oma ed. While
i is echnically possible o au oma e socio- emo ional skills o a deg ee,
such an au oma ion o socio- emo ional skills migh no be deemed socially
desi able.
D udgewo k as Repea able Pa e ns
A common heme in he in e iews was o sugges ha machines can do
manual, epe i i e and mundane asks, which will ee humans om he
d udge y o wo k. In e es ingly, d udge y ha ks back o one o he language
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 49
oo s o he wo d obo (Capek, 1920; Ox o d English Dic iona y, 2023e)
(see Chap e 1), al hough none o he in e iewees made his connec ion.
In many ways, in e iewees we e less conce ned abou physical asks
being eplaced. This migh in pa be due o he ac ha physical labou
being eplaced is a common ea u e o discou ses since he Indus ial
Re olu ion. The e was a clea expec a ion ha physical asks could be
easily done by machines. Lucy asse s ha machines can easily do asks
ha a e pu ely based on physical powe a he han human hough . I was
discussed ha echnology could displace manual labou in assembly line
wo k o ac o y ype wo k. In con as , in e iewees we e less conce ned
abou wo k in wa ehouses being au oma ed. Felicia, o ins ance, s a ed ha
hese a e wo kplaces whe e humans a e ea ed like machines. Simila ly,
My a alked abou how wa ehouse wo ke s a e old by a b acele wha o
do. Fo he , his means being ea ed as a human obo . Wha Felicia and
My a a icula e is he ac ha jobs being au oma ed is no necessa ily p ob-
lema ic because in many jobs, people a e al eady ea ed like machines. In a
sense, i is a con inua ion o au oma ion ha has been a ec ing la gely jobs
in manu ac u ing and wa ehouses. Howe e , due o he quali y o some o
hose jobs, pa icula ly in wa ehouses, i was no conside ed as pa icula ly
p oblema ic ha hese jobs migh all away.
Since I was mainly in e es ed in p o essional wo k, mos in e iewees
e lec ed on how p o essional wo k will change. In e iewees egula ly
e e enced asks ha could be au oma ed. In pa icula , in e iewees ocused
on epe i i e asks ha we e singled ou as ipe o au oma ion in p o-
essional wo k. P ime a ge s o such au oma ion a e wha William, who
wo ks in a law i m, called d udgewo k. He saw his d udgewo k as eady
o au oma ion. In e iewees like Yoshi o and Howa d alked abou how
manual, epe i i e and p edic able asks would be au oma ed. Ral men ions
ha hese asks migh no be simple bu can also be sophis ica ed, as long as
hey a e based on epea able pa e ns. I d udgewo k is disappea ing, William
sugges ed ha humans can ocus on highe - end and mo e in e es ing wo k.
Pe e desc ibes his as people being able o ocus on cogni i e asks ha a e
mo e ‘challenging, in e es ing and de elopmen al’.
Ano he way o exp ess his sen imen was o sugges ha d udgewo k –
o wha Anas asia desc ibes as bo ing wo k – akes away om c ea i e asks.
A common example in a chi ec u e e e ed o how ba h ooms a e placed.
Yoshi o ecalls how in his ea ly days in a chi ec u e, he was copying and
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
50
pas ing ba h ooms in o esiden ial layou s – a ask ha he ound dull and
bo ing. This is no longe needed oday. Anas asia simila ly s a es ha i is
no longe equi ed o d aw mundane hings like ba h ooms and ca pa ks
because he e is al eady a my iad o layou s a ailable ha one can use and
adap . In such con ex s, i makes li le sense o c ea e 30 di e en layou s
a esh. Acco ding o Yoshi o, pa e ns in a chi ec u e a e well es ablished.
I is equally well es ablished how a one- bed oom apa men should be
s uc u ed, which means ha he e is no need o ein en such spaces.
The eason why such wo k is no longe needed is because as Yoshi o s a es
a chi ec u e is based on pa e ns ha a e epea ed o e and o e again.
In o de o au oma e asks, hese pa e ns ha e o be made explici by
ans o ming he knowledge om a chi ec s in o ules ha can be used o
c ea e algo i hms. Caleb alks abou he idea ha a chi ec u e is abou codi-
ying aes he ics by c ea ing a ule- based model ha is p edic ing design
op ions. These algo i hms hen encapsula e he knowledge, expe ience and
he signa u e s yles o a chi ec s, and he p edic ions such a sys em makes
eplica e a speci ic a chi ec u al s yle.
I is no able ha o he p o essional con ex s b ough up simila
illus a ions o au oma ion. Fo ins ance, in e iewees o en e lec ed on
how legal wo k changed h ough he yea s. Be e ly commen s ha ea ly
in he ca ee , she spen hou s going h ough olde s o documen s wi h a
pen, which is no longe done anymo e. William simila ly commen s ha
lawye s no longe ha e o go h ough books because he e a e online legal
da abases hey can use. Tessa alked abou how lawye s use empla es and
ha e echnology ha changes clause numbe s and de ini ions o a wo d
au oma ically. In a bank con ex , Zac alks abou he manual labou o pu -
ing numbe s in o sp eadshee s, which is done au oma ically oday. In
audi ing, Pe e men ions ha d ones a e checking s ock in wa ehouses,
meaning ha audi o s no longe ha e o a el o a loca ion o make hese
checks physically. Duncan ecalls how his o ganisa ion s a ed o au oma e
adding ax codes o long sp eadshee s. These ac i i ies we e no mally done
by junio accoun an s who wen h ough wo- housand lines o a sp ead-
shee , adding ax codes manually. One could imagine ha his wo k is
a he edious. When Duncan’s o ganisa ion implemen ed echnology o
his ask, he expec ed an up oa because he echnology was aking away
a huge chunk o he jobs ha junio accoun an s we e doing. Howe e ,
Duncan ecoun s how he junio accoun an s we e deligh ed ha hey no
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 51
longe had o go h ough lines and lines o sp eadshee due o he edious-
ness o his ask. D udgewo k had been au oma ed.
Howe e , many o he in e iewees also s a ed ha such change in
ega d o echnology is no new bu o ms pa o changes ha hey ha e
expe ienced in hei li e ime. Vic o , a consul an , said ha consul an s
ha e been augmen ed by echnology o a long ime. He ecalls how in he
1990s, Lo us No es augmen ed he in elligence o consul an s. Anas asia
commen s ha in a chi ec u e, e e y new echnology such as compu e -
aided design led o sugges ions ha a chi ec s will become edundan , and
she sees he same a gumen s in ela ion o AI and machine lea ning. Be e ly,
a lawye , men ioned ha he 17 yea s o expe ience ha she buil up in he
a ea o expe ise ha e been eplaced by gene a i e AI. Be e ly ames his as
he expe ise disappea ing. Al hough one migh p esume ha his is sca y,
Be e ly seems less conce ned abou i . O e all, he e was less panic abou
he changes in p o essional wo k in he in e iews compa ed o he books.
A common pe spec i e was ha while some jobs will disappea , o he s will
appea . Anas asia a icula es his as o e e y edundan job, he e a e 20
o he new jobs eme ging.
Shi ing Tasks and S uc u es
I was widely acknowledged ha asks and s uc u es o p o essional wo k
migh change. A common heme in he in e iews was o alk abou he
changing asks and s uc u es in p o essional wo k. Fo ins ance, Yoshi o
imagines ha a chi ec s o he u u e will be ‘shephe ds o algo i hms’,
which means ha pa o he p o essional ask o he u u e will be o
de elop pa e ns ha a e coded. This cons i u es a majo change in ega d o
how a chi ec s do hei wo k because a key pa o he job will be o codi y
aes he ics in o echnology, o echo Caleb’s ea lie s a emen .
O he changes in he ask p o ile e e , o ins ance, o how ime is spen .
Xe xes uses he example o how a p og amme would adi ionally need
wo hou s o w i e a piece o code, bu i he p og amme uses Cha GPT,
he same ask can be done in i e minu es. Vic o alked abou how man-
agemen consul an s p oduce a lo o Powe Poin slides o communica e. In
he pas , he said ha a consul an migh spend 15 minu es o ga he in o -
ma ion and 45 minu es o ‘massage’ he slide o de elop he igh cha o
g aphic. Tha has now changed because consul an s now spend he majo i y
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
52
o ime on deep hinking and he s o y ha hey wan o ell; he c ea ion o
he slide i sel is swi because he e a e au oma ed ools ha can be used.
He main ains ha he au oma ion o slides will no make consul an s edun-
dan because he human is augmen ed bu no eplaced by echnology.
Ye , Vic o also acknowledges ha he e migh be a ime- sa ing com-
ponen , which migh in he long un a ec he hou ly pay s uc u e ha
many p o essions employ. While Vic o migh ha e spen one hou o c ea e
a slide in he pas , he would only equi e 30 minu es oday and migh
c ea e wo slides ins ead o one in an hou . Yoshi o obse es a simila ime-
sa ing componen o echnologies in a chi ec u e, whe e wo k ha would
ha e equi ed ou hou s in he pas can now be done in ou minu es. Ye ,
clien s a e s ill cha ged o ou hou s. Be e ly d aws on a s ikingly simila
example, saying ha wo k ha would ha e aken a lawye ou hou s o
comple e can now be comple ed in 30 seconds o less. She acknowledges
ha his will be a challenge o he no mal cha ging s uc u e by he hou
ha many p o essional i ms use. I a ask now only akes minu es, why
would clien s pay he high hou ly a es o p o essionals? This ques ions he
adi ional hou ly pay model employed in many p o essional i ms.
New echnologies also open up he possibili y o changing how wo k
is done. In pa icula , gene a i e AI is seen as a game change o p o es-
sional wo k. Vic o alks abou how junio consul an s use Cha GPT o b ing
hemsel es up o speed on speci ic clien con ex s swi ly. In he pas , a con-
sul an who is un amilia wi h how a bank ope a es migh ead Wikipedia
and a icles on inancial blogs. Today, hey will ask Cha GPT. Vic o s a es ha
his will no make hem an expe bu i helps hem o unde s and wha a
easu y depa men a a bank does. I he echnology is a bi w ong abou
such basic in o ma ion o e en ‘hallucina es’, i does no ma e g ea ly
because i would no go in o a epo o he clien bu a he p o ides back-
g ound knowledge o allow he consul an s o unde s and a con ex swi ly.
Vic o is less conce ned ha managemen consul ing could be eplaced by
gene a i e AI because he likens he ou pu o Cha GPT o how his eenage
son would answe a ques ion. I is a basic answe bu lacks he le el o
sophis ica ion and igou ha a clien would pay ‘big money’ o .
In gene al, he e we e many examples o how Cha GPT and gene a-
i e AI mo e gene ally a e used in he p o essions. Anas asia alked abou a
ma ke ing company ha uses Cha GPT o gene a e pi ch ideas and hey hen
y o bea he sys em by coming up wi h be e pi ches. Be e ly men ioned
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 53
ha in he law i m, hey spen a lo o ime on ‘p omp enginee ing’
o ensu e ha he esponses om gene a i e AI ma ch wha is equi ed.
William desc ibes he use o gene a i e AI as a c ea i e spa ing pa ne o
a oil o c ea i e hinking; i is used o de elop ideas wi h a machine bu i
does no eplace he human. Al hough gene a i e AI migh au oma e asks
in p o essional wo k, gene a i e AI is mainly seen as a ool o condense
in o ma ion and o p o ide ideas, which augmen s p o essional wo k.
The ime- sa ing componen o echnology can no only upend he cha -
ging s uc u e in he p o essions bu i migh also equi e di e en o gan-
isa ional s uc u es. Many p o essional i ms a e o ganised in a py amid
s uc u e, whe e la ge numbe s o junio p o essionals a e hi ed and
hen compe e o become one o he ew pa ne s. Vic o alks abou how
he e a e housands o young people in hei 20s manually going h ough
documen s and ci cling i ems in Big Fou i ms – he la ges p o essional
se ice ne wo ks, namely Deloi e, EY, KPMG and PwC – when p epa ing
axes. Vic o sugges s ha hese housands o young people migh no be
equi ed in he u u e. Simila ly, Be e ly men ions ha he pool o junio
lawye s is going o be sh inking as a consequence o d udgewo k no longe
being equi ed o he same deg ee.
The e was also a sen imen ha i junio p o essionals no longe ha e o
do d udgewo k, hey miss ou on lea ning. Vic o alks abou how ea ly in
his ca ee , a lo o lea ning came h ough doing bo ing s u , which allowed
him o build up men al ‘muscle memo y’. He desc ibes ‘muscle memo y’
as a e lex o know wha migh be w ong in a si ua ion. Caleb alks abou a
simila momen in ega d o a chi ec u e, whe e he expe ience o ha ing
con on ed a p oblem in a p e ious si ua ion helps a chi ec s o sol e a
cu en p oblem. Yoshi o also men ions how a chi ec u al pa e ns became
o him second na u e h ough epe i ion and wo king closely wi h o he
a chi ec s. While i was common o see d udgewo k as a o m o lea ning,
which migh no longe be possible, Anas asia p esumed ha echnology
can in ac help junio a chi ec s o de elop knowledge o do he job well.
She a gues ha echnology can assis junio a chi ec s in building wha she
calls an in ui ion. Such an in ui ion would in he iew ake yea s o cul i-
a e o he wise. In a sense, she a gues ha he in ui ion de eloped h ough
d udgewo k can be eplaced by he coded expe ience in echnology.
P o essional wo k is o en based on an app en iceship model whe e
junio people lea n by obse ing o he s. William calls his osmo ic lea ning.
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
54
Acco ding o Vic o , i is no jus doing manual asks o e and o e again
ha cons i u es lea ning in he p o essions, bu also si ing in housand con-
e ence ooms o unde s and wha makes di e en execu i es ick. Va ious
in e iewees alked abou how much ime junio lawye s spen obse ing
mo e senio lawye s o see how hey deal wi h clien s and how o ackle
speci ic issues. A adi ional a gumen is ha i mo e asks a e au oma ed,
educing he need o junio lawye s, hey will ha e less chance o obse e
how o do hei jobs once hey become mo e senio . In many ways, he
pandemic p o ided a glimpse a some o he p oblems ha he lack o
shadowing senio colleagues migh ha e in he u u e. Due o he ac ha
wo k was done emo ely, many junio lawye s missed ou on he oppo -
uni y o obse e he day- o- day equi emen s o wha i means o be a
senio lawye . Tessa explained his as ollows: whe eas be o e, junio lawye s
would si close o a senio lawye and o e hea how he senio lawye akes
u gen clien calls, du ing he pandemic, he junio lawye s we e o en no
included in ad hoc clien mee ings because he clien ei he called on a
mobile o he ideo mee ing link was no ci cula ed o he junio lawye s.
Be o e, junio lawye s would soak up knowledge h ough obse a ion; hey
we e now excluded om such p ocesses because hey we e no longe phys-
ically p esen in he space. William alks abou how e en a anging a deb ie
a e an online mee ing wi h a clien has he addi ional hu dle o se ing
up ano he online mee ing. I , s uc u ally, much o he epea able asks a e
done by machines, leading o ewe junio p o essionals being equi ed,
his could mean ha he p og ession om junio o senio p o essional and
he en i e s uc u e o p o essional i ms migh need o change.
The e a e also wide changes o he p o essions ha he in e iewees
imagined. Fo ins ance, Be e ly likened he changes in he legal p o ession
o he e o ma ion whe e people ge access o some hing ha was p e iously
closely gua ded. She desc ibes his as a democ a isa ion o legal knowledge,
which in he iew is a posi i e de elopmen . Yoshi o echoes his poin when
he sugges s ha due o p o essionalisa ion, knowledge became p o ec ed
and a cha ge is due o access his knowledge; o ins ance, consul an s a e
cha ging o hei knowledge and knowledge is si ing behind paywalls.
He a gues ha echnology means ha such knowledge could be accessed
di e en ly. He sugges s ha making such knowledge – which in his iew
lies in pa e ns in a chi ec u e – mo e widely accessible would be bene-
icial. The e migh also be bene i s o smalle i ms, as Anas asia hin ed
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 55
when sugges ing ha echnology can le el he playing ield o allow smalle
a chi ec u al p ac ices o compe e wi h la ge ones. O e all, hese changes
in asks and s uc u es sugges ha wha humans migh do in he u u e o
wo k changes. I will ace his in he nex sec ion.
The Human Ad an age
We ha e seen so a ha in p o essional wo k, d udgewo k is seen as
au oma able, and while ha b ings ce ain challenges o asks and
s uc u es, he o e all sen imen in he in e iews was ha echnology
is augmen ing wo k. Viewing hese changes mo e holis ically, some
in e iewees commen ed how au oma ing d udgewo k would allow people
o spend hei ime di e en ly. Je ey likened his eme ging u u e as akin
o S a T ek, he sci- i anchise; Je ey sugges s ha in S a T ek, humans can
dedica e hemsel es o explo a ion and c ea ion, which in a sense would
ee indi iduals o engage in o he ac i i ies. Gab ielle said au oma ion will
ee up ime o humans o do o he hings, which, as she poin s ou , migh
no only ela e o he public sphe e o wo k bu also en ail o he ac i i ies
like ca ing o amily membe s.
Mos hinking on wha humans migh do cen ed on he idea wha
machines cu en ly canno do, o whe e he use o machines would be
undesi able. Those a eas la gely equi ed social in e ac ion. Fo ins ance,
Ral sugges ed ha any hing ha wo ks based on ela ionships, such as sales
o cus ome se ice, will equi e humans. Bank elle s a e a p o ession ha
is o en men ioned in ela ion o echnological change. Fo ins ance, Bessen
(2015) discusses he assump ion ha wi h he in oduc ion o ATMs, bank
elle s migh disappea . Howe e , a he han disappea ing, he asks ha
bank elle s engaged in changed (Bessen, 2015). Simila ly, Osca , who
wo ks in a bank, e e ences bank elle s because hei jobs a e seen as unde
h ea by digi alisa ion. Howe e , Osca was o he opinion ha much o he
wo k ha bank elle s would ha e adi ionally done in a b anch would in
he u u e be mo ing o he back o ice. Bank elle s would likely be dealing
wi h simila issues as oday, bu he in e ac ion is acili a ed by digi al ech-
nologies. This speaks o he poin ha pe sonal ela ionships will equi e a
human ouch. Je ey alked abou o he a eas whe e human inpu would
be equi ed, such as in be ea emen counselling. Ra he han HR sending an
au oma ed message, he would expec ha a human con e sa ion is equi ed
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
56
in such si ua ions. Kenne h also alks abou empa hy ha is, o ins ance,
shown in heal hca e as being ha d o eplace. In a simila ein, Ben sugges s
ha since asks ha can be done by a machine will be done by a machine in
he u u e, his lea es so skills o humans. This includes unde s anding and
managing he sel and o he s. Ben p esumes ha humans will be in demand
o hose so skills.
I is pe haps no su p ising ha socio- emo ional skills a e singled ou as
pa icula ly impo an in ega d o p o essional wo k. Fo ins ance, Vic o
alks abou he impo ance o ‘ eading he oom’ o a consul an . A con-
sul an p esen ing o a clien will no ice i he CEO is nodding bu o he
execu i es a e scep ical. A consul an would hen know o schedule ex a
mee ings wi h he scep ical execu i es o b ing hem on boa d. He sugges s
ha echnology canno do his in e - human wo k. Simila ly, Vic o alks
abou he impo ance o execu i e assis an s who he desc ibes as he social
glue. While many o he asks ha execu i e assis an s do, such as sched-
uling mee ings, could be au oma ed, execu i e assis an s do much mo e
han his: hey know wha is happening in he o ice, hey know he poli ics
and hey ead e e y email, which gi es hem addi ional social knowledge
ha a machine could no eplica e, acco ding o Vic o .
This also has consequences on how p o essional i ms a e likely o hi e
in he u u e. Be e ly e lec ed abou wha ype o people a u u e law
i m migh need o ec ui . She sugges s ha in he u u e, law i ms need
people who a e be e a lis ening o clien s. Be e ly acknowledges ha
gene a i e AI can p o ide many answe s bu ha a lawye has o unde -
s and he clien . The lawye needs o unde s and i he clien is mo e isk-
a e se o open o isk, which will in o m he legal s a egy sugges ed.
She says ha as a managing pa ne in a legal i m, she would be looking
o ec ui indi iduals wi h lis ening skills, o wha she calls an ‘empha ic
lawye ’. The no ion o he empa hic lawye encapsula es he idea ha
socio- emo ional skills a e cons uc ed as i al in he u u e o wo k. In
a simila ein, William alks abou how wo king wi h clien s equi es an
unde s anding o he cul u e and he powe dynamics in he o ganisa ion.
He alks abou how an expe ienced lawye has o unde s and he cul u e
and he powe dynamics ha happen in his con ex , which is used o
sol e p oblems and ind a consensus. This is no legal knowledge as such
bu wha he desc ibes as aci knowledge ha people ha e akin o a gu
eeling o an in ui ion.
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 57
The o e all idea ha eme ges om he in e iews is ha i pa s o p o-
essional wo k a e au oma ed, hen socio- emo ional skills become a com-
pe i i e ad an age o humans. This is la gely o he ac ha socio- emo ional
skills a e cons uc ed as uniquely human. In o he wo ds, hose skills ei he
canno be pe o med by machines o i is socially no desi able ha hose
skills a e pe o med by machines. By ocusing on socio- emo ional skills
as uniquely human, he in e iewees also sugges ha machines will no
eplace human labou comple ely. The in e iew accoun s we e, by and
la ge, hope ul. D udgewo k is handed o e o machines and humans can
ocus on in e es ing wo k, and wo k ha equi es socio- emo ional skills.
I is he e ha humans shine because hey can do hings ha machines a e
pe cei ed as unable o do o whe e he use o machines is undesi able.
The Gende ing o Socio- Emo ional Skills
T adi ionally, discussions abou socio- emo ional skills we e gende ed.
S e eo ypically, i is p esumed ha women a e pa icula ly good a and well-
sui ed o displaying socio- emo ional skills. Such ideas a e, o ins ance,
d awn upon by CEOs, who jus i y gende equali y by e e ing o such
gende essen ialised skills (Kelan & W a il, 2021). These s e eo ypes ha e
consequences in he wo kplace in ha women displaying socio- emo ional
skills a e o en no ewa ded o hem; he assump ion is ha women
simply do wha comes na u ally o hem (Fle che , 1999; Kelan, 2008a).
Ye , in he in e iews like in he books, hose socio- emo ional skills we e
a ely gende ed. The only example o a pe son linking socio- emo ional
skills wi h gende came om Duncan. Duncan a gues ha women should
ind i easie o s ay employed when au oma ion akes hold. He admi s
ha he has no speci ic da a o his assump ion bu in his obse a ion o
20 yea s, he no iced ha women a e be e han men a pe suading people
o adop hei ideas and a building eams and communi ies, which is cen-
al in p o essional se ices jobs. Duncan he e es ablishes a link be ween
gende and skills in ha he a gues ha women a e be e a building eams
and pe suading o he s. He sugges s ha hese socio- emo ional skills migh
gi e women he edge when i comes o skills equi ed in he wo kplace.
Socio- emo ional skills we e loosely linked o gende in ela ion o ca e
wo k. Gab ielle alked abou how wi h an ageing popula ion, he e is
inc eased demand o heal hca e wo ke s. Since women a e o e - p esen ed
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
64
icky one. I ound looking in o he eyes o many o he a a a s was a a he
s ange expe ience. One clea ly no es ha he a a a s a e no eal, and i
eels odd o look hem in he eyes. Based on ha eedback o a lack o eye
con ac , I decided o s a e a he a a a s’ eyes. In a eal- li e in e ac ion I am
su e my s a ing eye con ac would ha e been pe cei ed nega i ely. Ye when
I s a ed, I s ill go he eedback ha I do no make enough eye con ac .
I wonde ed in how a he echnology acking me migh no be as accu -
a ely calib a ed as i should be.
Ra he han analysing he con ex o alk, he eedback I ecei ed in
VR was pu ely on he deli e y o he alk. In ano he si ua ion, I go add-
i ional eedback ha I need o look mo e o he igh o he le . One o my
a ou i es was a unique sco e ha I go in some o he exe cises. This unique
sco e, as he app old me, was measu ing i an 11- yea - old would unde -
s and me. I ound his o be a a he odd me ic because all he ainings
we e designed o be in p o essional wo k con ex s a he han schools, and
as such, he in e ac ion wi h 11- yea - olds migh be a he limi ed. I also
wonde ed how he sco e came abou . Had people been eco ded be o e
and an 11- yea - old had lis ened o he eco ding o check comp ehension?
As such, he eedback ha such apps p o ide was in my expe ience
a he mixed. Some o i was help ul, whe eas o he hings kep coming up
e en when I did speci ically ha in he nex ei e a ion. Much o his ech-
nology is s ill in in ancy and eedback will likely ge be e o e he yea s.
I also wonde ed in how a such echnologies could be eliably chosen,
o ins ance, o selec candida es. I he echnology does no unde s and
me, I am su e ha o he people wi h accen s would also s uggle. I he
echnology could no ack my eye mo emen accu a ely, I am no su e i
I would be selec ed o a job ha equi es such a connec ion.
Hands as Liminal in VR
VR aining is designed o anspo he lea ne s in o a new space whe e hey
can p ac ice new skills. Imme sion is hus key. One common assump ion is
ha imme sion equi es a highly ealis ic scena io whe e simula ed people
appea eal. Fo all VR apps I used, ha was a om being he case. As
I men ioned be o e, i was clea o me ha he a a a s I in e ac ed wi h
a e no eal people. Thei eyes would o en look pa icula ly wei d and
hey lacked any acial exp ession ha I alued, o ins ance, wi h Viola, he
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 65
echnology om Soul Machines ha I ied. Viola only appea ed on my
compu e sc een a he han in VR. Howe e , i was no able ha in VR en i -
onmen s, e o s had been made o gi e he a a a s di e en oices by hose
who c ea ed hem, and I app ecia ed ha some cha ac e s had accen s ha
sounded ei he F ench o A ican. Al hough a emp s had been made o
make he scena ios ealis ic, ini ially, I hough i would be ha d o imagine
ha one can imme se onesel in he en i onmen .
O he s I spoke o had simila conce ns. F anklin, o ins ance, ecoun s a
s o y o how one o he pa icipan s o a VR aining was highly esis an o
unde go heal h and sa e y aining in his new o ma . He ied o con ince
his colleagues ha hey should all e use his aining. Howe e , he could be
pe suaded o gi e i a go. He pu he headse on and s a ed he simula ion.
A one poin , he linched – in he VR scena io, a nail had gone h ough
his hand. E en hough his pe son had been scep ical abou he aining,
ha ing he expe ience o a nail going h ough his hand shocked him. He
ealises ha wha he aining o e s is some hing mo e han a heo e ical
unde s anding o wha can happen i you do no adhe e o heal h and sa e y
s anda ds. F anklin desc ibes his an ‘emo ional impac ’ in he in e iew
whe e he nail- h ough- he- hand scena io c ea ed addi ional lea ning ha
a leas acco ding o F anklin is di icul o achie e in ano he way.
While doing VR aining mysel , I was egula ly in i ed o pick an a a a .
On mos pla o ms, I could selec om a se o p ese cha ac e s. The e a e
commonly less modi ica ion op ions han I had expe ienced building an
a a a o one o he i ual ec ui men e en s I a ended, whe e one had a
lo o choice in how one p esen s onesel , including he abili y o wea lip-
lops in ec ui men simula ions. In VR, howe e , i seemed mo e common
o pick an a a a ha was p e- es ablished. The e was less o a emp a ion
o build an a a a ha migh co espond mo e closely o how you migh
appea in eal li e. Ins ead, I was o e ed he op ion o appea as a La ino o
a Black woman. In mos cases, I o go which a a a I had picked because
i was ai ly inconsequen ial o mos o he VR in e ac ions I engaged in.
Ye , a se e al imes du ing he expe ience, I was shocked when my hands
looked di e en in VR han in eal li e. I no ed in my ield no es ha I was
su p ised when my inge nails had b igh ed nail polish on.
Toge he wi h he example ha F anklin ecoun ed ea lie , i appea s ha
hands occupy a liminal space be ween he physical and he i ual wo ld.
Du ing a VR expe ience, you no only ha e a headse on ha blocks ou
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
66
he physical en i onmen in which you a e si ua ed and eplaces i wi h a
i ual en i onmen , bu you also hold con olle s in you hands, which
allow you o d i e he ac ion by clicking on ce ain elemen s. As a ma e o
ac , du ing he ini ial amilia isa ion wi h VR, I lea ned how o manipula e
i ual objec s by using he con olle . Fo ins ance, I would lea n o li an
objec and hen h ow i . You hand mo emen is also used in some me ics
o analyse, o example, you body language. I you c oss you a ms, he
con olle s will also be c ossed, which is ead as a closed body posi ion.
While in o he VR en i onmen s, he body mo es a ound mo e dynam-
ically, in mos o he aining in VR I comple ed, I was s a iona y, ei he
s anding o si ing, bu a ely mo ing h ough space. Glancing a my hands
in VR was hus one o he ew ins ances whe e i became clea ha my
embodimen as a i ual a a a was di e en . This o en led o a momen o
su p ise akin o wha he pe son who had a nail go h ough his i ual hand
mus ha e expe ienced.
My VR expe iences did no include a nail- h ough- he- hand scena io
like F anklin desc ibes, bu i is no ha d o imagine based on my own
expe iences ha he e is a momen o shock and su p ise. The hands seem
o be liminal bounda y be ween a sense o sel as a pe son and he i ual
en i onmen in which one inds onesel .
Pe spec i e Taking
Apa om heal h and sa e y applica ions, VR was p esen ed as pa icula ly
use ul o help humans o p ac ise socio- emo ional skills ha a e cons uc ed
as impo an o he u u e o wo k. Ma exp essed his by e e encing wha
he calls an ‘old adage’, ha one mus walk a mile in someone else’s shoes o
comp ehend hei li e si ua ion. He explains ha his is pa icula ly use ul
o di e si y and inclusion because VR allows you o y on someone else’s
shoes. VR allows you o ake someone else’s place and make an expe ience
ha you migh no mally no ha e. Ma p o ides he example o aking on
he posi ion o a woman colleague and play h ough a scena io om he
pe spec i e. Ma a gues ha his helps wi h aking pe spec i e and is mo e
impac ul han wa ching a ideo. He says ha by being in VR in his si u-
a ion, you migh ‘ eel you blood boil’ because you a e ea ed un ai ly. This
sugges s ha VR allows access o emo ions ha one no mally migh no
expe ience, which in u n can help o de elop empa hy o o he s.
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 67
F anklin o e ed ano he s ikingly simila explana ion as o why VR wo ks
well in ega d o di e si y and inclusion aining. F anklin desc ibes how VR
aining allows you o expe ience wha i eels like o be ma ginalised. He
a gues ha o many people who did no ha e such expe iences be o e, i
is icky o unde s and wha ma ginalisa ion eels like. This is said o limi
hei po en ial o ake ac ion on i . F anklin sugges s ha VR aining ge s
people o e he hu dle o unde s and wha i eels like o be in such a si u-
a ion and how o espond o ha . He likens i o he di e ence be ween
an academic explana ion and he p ac ical expe ience. Fo F anklin, i you
expe ience being he odd pe son ou o ha you ideas a e no app ecia ed,
people de elop wha he calls hei so skills. Wha we see he e is ha ech-
nology is used o allow people o make di e en expe iences o de elop
socio- emo ional skills.
In ega d o how such aining is deli e ed, I had he oppo uni y o pa -
icipa e in se e al di e si y and inclusion ainings in VR. In one scena io,
I me S e e, who ea ed Sand a in sexis ways. One o he exe cises in ol ed
p essing he con olle in my le hand o inclusi e beha iou s and he con-
olle in he igh o exclusi e beha iou s. Following his, I was p esen ed
wi h a ange o sen ences I could pick om o ad ance he con e sa ion.
Depending on my choice, I was gi en eedback i ha was a good o bad
choice. One elemen I no iced is ha when I alked wi h S e e, he seemed o
mi o my beha iou – i I was agg essi e, he also became agg essi e. This
mi o ing is wha ac o s who a e engaged o pe o m di e si y and inclu-
sion scena ios in pe son ha e also been asked o do. I did no expec his o
be a ea u e o he VR aining. The key lea ning o he aining was o allow
S e e o come up wi h why his beha iou is p oblema ic himsel a he han
elling him wha was w ong.
The inal pa o he scena io in ol ed me eco ding a closing s a emen
in which I add essed S e e di ec ly and ou lined wha we had ag eed in
e ms o he way o wa d. Then I changed a a a and was in S e e’s pos-
i ion. I saw he a a a ha I had chosen be o e deli e he speech ha had
jus eco ded back o me. As S e e, I would no ice how i el o be a he
ecei ing end o such messages. I could hen shi back and e- eco d he
message. The idea was ha I no ice mysel ha some hing was oo ha sh o
no clea and ha I could hen co ec ha in he nex ei e a ion. This o m
o sel - eedback was a he use ul. Such e akes a e no possible in eal li e
and a e edious in physical aining, bu hey a e a key ea u e o VR. I can
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
68
p ac ise as much as I wan . I ob iously elies on he ainee o spo hings
ha do no wo k well.
Role plays and o he hea e- based me hods a e egula ly employed
in di e si y and inclusion aining, o en making use o ac o s o pu ing
employees in di e en oles. They can be a he e ec i e o pe spec i e
aking. Howe e , VR allows o his o be ele a ed. Ra he han enac ing a
si ua ion wi h a colleague o an ac o , which is a i icial in i sel , VR allows
pa icipan s o engage in such exe cises on hei own. Seeing he wo ld om
a di e en an age poin can be acili a ed by VR. This is essen ial o de elop
empa hy and he socio- emo ional skills ha a e cons uc ed as he human
co e ad an age. Ye such aining is deli e ed by a machine. O cou se,
humans design he echnologies, bu he ac ual aining o socio- emo ional
skills is machine- acili a ed. Machines a e hus able o ain socio- emo ional
skills in humans, complica ing he idea ha socio- emo ional skills a e ou
o each o machines.
Socio- Emo ional Skills as Pa e ns
So a , I ha e shown ha machines a e able o mimic and ain humans
in socio- emo ional skills. Since socio- emo ional skills ollow epea able
pa e ns, socio- emo ional skills can be au oma ed. Like many o he epea -
able pa e ns ha can be au oma ed in sp eadshee s, legal documen s o
p esen a ion slides, socio- emo ional skills ollow pa e ns ha can be ans-
e ed in o code. Such p ocesses o ans e ing emo ions in o compu e
pa e ns a e subjec i e, and how subjec i e assessmen s a e ans o med in o
objec i e and uni e sal assessmen s will be a he cen e o Chap e 5. While
hese p ocesses a e deeply p oblema ic, o he pu pose o his chap e ,
he ac ha socio- emo ional pa e ns can be au oma ed cas s doub on he
cons uc ion o socio- emo ional skills as uniquely human, and as such, as a
co e compe i i e ad an age o humans o e machines.
I and o wha deg ee socio- emo ional skills will be au oma ed depends
also on wha is deemed socially accep able. Fo example, i has been shown
ha ha ing machines in ol ed in childca e is echnically possible bu is
deemed by expe s in he ield as p oblema ic o social easons due o
implica ions o child en’s de elopmen and p i acy conce ns (Lehdon i a
e al., 2023). The e o e, i is possible ha socio- emo ional skills a wo k will
con inue o be comple ed by humans. Howe e , his migh no be due o
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 69
he ac ha socio- emo ional skills canno be pe o med by machines bu
a he due o a desi e o ha e humans pe o m hose skills.
Howe e , he e is po en ial o use echnology o ain indi iduals. Fo
ins ance, he empa he ic p o essional, o pa aph ase Be e ly, migh well
be ained by machines o unde s and how o be e suppo colleagues
and clien s. I he pa e ns ha cons i u e d udgewo k can be embedded
in machines o gi e p o essionals a subs a e o he expe ience o many
yea s o p o essional p ac ice, hen his is ce ainly possible in ela ion o
socio- emo ional skills as well. When Anas asia alked abou how echnology
migh p o ide junio p o essionals wi h an in ui ion ha is ained on p e-
ious pa e ns and hus p o ides access o p o essional expe ience, his
in ui ion migh also ex end o how o display socio- emo ional skills.
Conclusion
In his chap e , I ques ioned in how a socio- emo ional skills a e a co e
human ad an age o humans o e machines. I illus a ed ha mos
in e iewees expec ed d udgewo k ha ollows epea able pa e ns o be
comple ed by machines in he u u e. In p o essional wo k, his mean ,
o ins ance, ha indi iduals would no longe go manually h ough long
documen s bu ha echnology could do ha wo k. I also in ol ed using
echnology o lea n and o c ea e ou pu s. I pa s o p o essional wo k
a e au oma ed, his changes he ask p o ile o p o essionals and is likely
ha ing an impac on how p o essional i ms a e s uc u ed. In pa icula ,
his migh a ec he need o la ge g oups o junio people who engage in
much o he d udgewo k. Howe e , a he han seeing he p o essions as
disappea ing as a consequence, in e iewees alked abou how his migh
democ a ise p o essional knowledge by making i mo e widely a ailable. I
was also s essed ha p o essionals will equi e a di e en skill se , such as
being empa he ic. I is in e es ing o no e ha like mos o he books, he
in e iewees la gely esis ed he idea o cons uc socio- emo ional skills as
some hing women a e good a . This cons i u es a depa u e o how socio-
emo ional skills a e o en alked abou .
Empa hy, as well as o he socio- emo ional skills, we e egula ly
cons uc ed as uniquely human and hus ou o each o machines. In his
chap e , I ques ioned in how a socio- emo ional skills a e indeed ou side
o he each o machines and hus cons i u e a human ad an age. I i s
UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS
70
looked a how in e iewees alked abou ca e wo k and au oma ion o hen
ocus on how machines emula e emo ions o ain emo ional esponses,
including in VR. I ha e sugges ed ha e en hough he expe iences in VR
appea a i icial, hey migh be well- sui ed o de elop socio- emo ional
skills in pa icipan s. The u u e empa he ic p o essional migh well be
ained in VR. The chap e ques ioned i socio- emo ional skills a e uniquely
human and as such cons i u e he co e compe i i e ad an age o humans
o e machines. Howe e , while socio- emo ional skills a e au oma able o a
deg ee, i is ques ionable i his is socially desi able.
No es
1 As a side no e, when Je ey and I spoke, i was du ing one o he Co id lockdowns
and schools we e closed. While Je ey was speaking wi h me, his daugh e was
si ing a he same able doing he wo k he school had gi en he bu also lis ening in
on he in e iew. Je ey could no emembe he name o he seal and only a e his
daugh e sea ched o i online, she eminds him ha i is called PARO.
2 I use shopping ca and cookies he e because he VR expe ience was si ua ed in he
Uni ed S a es.
This chap e has been made a ailable unde a CC BY – A ibu ion license.
DOI: 10.4324/9781003427100-4
4
ALGORITHMIC BIAS
AS ULTIMATELY FIXABLE
In oduc ion
Finding people wi h he igh skills o do jobs is cen ally impo an o
o ganisa ions o unc ion well. Hi ing also o en ollows speci ic pa e ns
such as iden i ying skills ha a e needed in di e en unc ions in he o gan-
isa ion. Ye hi ing is complex and ime- consuming, making hi ing an ideal
a ea o deploy echnology. One canno ail o see how o an o ganisa ion, i
mus be appealing o use echnology o de e mine which candida e is bes
sui ed o a ole. In many ways, using echnology o ec ui people ollows
he idea ha some hing as pe cei ed subjec i e as hi ing can be ans o med
in o an objec i e p ocess (Kang, 2023). Howe e , he p ocesses h ough
which his p esumed objec i i y is achie ed a e a om unp oblema ic
(Kang, 2023). Technology also has he po en ial o help wi h ano he aspec
o he hi ing p ocess: he bias o hose who selec he candida e. Fo a long
ime, ec ui men has been impac ed by human bias and echnology o e s
he possibili y o educe his bias (Feloni, 2017; McIl aine, 2018; Riley,
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE
72
2018). The e o e, AI in hi ing is a echno- op imis ’s d eam in ha p ocesses
o an o ganisa ion a e imp o ed h ough echnology.
Howe e , his echno- op imism is in many ways dampened by he ac
ha AI has been shown o epea and ampli y bias in hi ing (Dalenbe g,
2018; Vassilopoulou e al., 2024; Kelan, 2024). The media is egula ly
poin ing o dange s associa ed wi h AI in hi ing. Amazon’s ailed a emp o
use AI in hi ing unc ions is he s anda d example used in he media (BBC
News, 2018; Das in, 2018) and beyond. In his case, he pa e n ha he
AI iden i ied and epea ed h ough p edic ions was exclusiona y. I AI is
ampli ying biases hen his aises se ious ques ions i using AI in hi ing can
indeed ul il his echno- op imis ic d eam. In his chap e , I ques ion how
a echno- op imis ’s s ance ha en ails ha AI imp o es business p ocesses
can be econciled wi h he exis ence o algo i hmic bias in hi ing. I sugges
ha his is achie ed by cons uc ing algo i hmic bias as ul ima ely ixable.
Techno- op imism as a s ance was supplemen ed wi h he pe spec i e o
echno- hesi a ion. Techno- hesi a ion is no a ejec ion o echnology as
such. A ejec ion o echnology o an acknowledgemen ha echnology
can lead o mo e ha m han good would be akin o he s ance o echno-
pessimism. Ins ead, his s ance p esen s a hesi a ion ha could be dissol ed
once he e is mo e socie al accep ance o and con idence in he use o AI
in hi ing.
Hi ing he Candida es ha Bes Mee he P o ile
Al hough all pa s o human esou ces can po en ially be impac ed by digi -
alisa ion (Cheng & Hacke , 2021; Tambe e al., 2019), hi ing has been
an a ea a he o e on o being ans o med h ough digi al p ocesses
(Eubanks, B., 2018). Reasons why hi ing is a p ime candida e o digi -
alisa ion a e due o he epe i i e na u e o he p ocess and he po en ial
educ ion o mis akes ha digi alisa ion o e s (Eubanks, B., 2018). In add-
i ion, digi alisa ion can b oaden he candida e pool, make he hi ing p o-
cess mo e e icien , lead o highe job enu e, and make he p ocess quicke
and educe cos s (Black & an Esch, 2020; Ho man e al., 2015; Johnson
e al., 2021; Tippins e al., 2021).
Digi alisa ion can be used in a ange o p ocesses in he hi ing unnel
(Sanchez- Monede o & Dencik, 2019; Sánchez- Monede o e al. 2020).
Howe e , he e a e ce ain aspec s in he hi ing p ocess whe e he use o
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 73
digi al echnologies is mo e common, such as sc eening candida es (Albe ,
2019). Fo ins ance, a candida e migh join a i ual ca ee s ai o mee an
employe . Be o e submi ing an applica ion, he candida e can be in i ed
o engage wi h a cha bo o check i basic equi emen s o he job a e
me . I so, he candida e migh be in i ed o submi a CV, which is hen
checked based on ce ain keywo ds. I has been no ed ha some candida es
a emp o game he sys em by, o ins ance, submi ing a CV ha includes
e e ences o eli e uni e si ies like Ox o d and Camb idge in whi e ex
ha is in isible o he human eye bu ha would be picked up i an AI
sea ched o keywo ds ha include such eli e uni e si ies (Bu anyi, 2018).
I a candida e’s CV is selec ed, candida es migh hen be in i ed o pa ici-
pa e in a numbe o simula ions and games o es hei skills and abili ies
(Tippins, 2015). In case candida es p oceed u he , hey a e commonly
asked o pa icipa e in asynch onous online in e iews whe e candida es
migh use a mobile phone o compu e o eco d hemsel es answe ing a
se ies o ques ions designed o see i he candida es a e a good i o he
posi ion (Köchling & Wehne , 2020; Albe , 2019).
The hi ing p ocess is expec ed o be changed signi ican ly by digi alisa-
ion, bu many o he unde lying p inciples o hi ing will apply o digi alised
o ms o hi ing as much as hey do o non- digi alised o ms o hi ing.
Human esou ce p ac i ione s o en ely on guidelines such as he ‘Uni o m
Guidelines On Employee Selec ion P ocedu es’ (Biddle Consul ing G oup,
2023) o guide he hi ing p ocess. Be o e s a ing a hi ing p ocess, a job
analysis is commonly conduc ed in which knowledge, skills, abili ies and
o he cha ac e is ics (KSAOs) a e iden i ied ha a e equi ed o do he job
well. Then assessmen s a e de eloped ha assess he candida es agains he
KSAOs. This is ensu ed h ough a a ie y o alidi y es s ha a e conduc ed.
These en ail c i e ion- alidi y o , in o he wo ds, ha he selec ion p o-
cedu e p edic s job pe o mance; con en alidi y, which means ha wha is
being assessed is indica i e o doing he job well; inally, cons uc alidi y,
which shows ha he da a collec ion is indica i e o how well he KSAOs a e
ma ched by he candida e (Biddle Consul ing G oup, 2023).
I hi ing is digi alised, a simila p ocess should be ollowed, apa om
he ac ha echnology is mo e cen al in i . His o ically, a candida e
migh ha e been in i ed o comple e a pen- and- pape assessmen , which is
e alua ed by humans o i wi h KSAOs. Now, candida es migh be in i ed
o comple e a ious games online o assess o wha deg ee candida es
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE
80
which inequali ies migh po en ially inc ease, and as such, i he educ ion
o inequali ies is seen as o posi i e alue, hen echno- op imism would no
be necessa ily a s ance one can uphold. As such, my in e es was on how
people nego ia e being echno- op imis s wi h algo i hmic bias.
Humans as Making AI Biased
In e iewees would commonly e e ence he ac ha algo i hmic bias only
exis s because humans ha e biases and AI lea ns such biases om humans.
Lucy sugges s ha algo i hms hemsel es a e no biased bu ha he da a
ha algo i hms a e ained wi h is. Fo Lucy, algo i hms a e jus ways o un
numbe s, and hey ein o ce bias because hey ha e been ed biased da a.
F anklin s a ed ha humans a e biased and ha AI is ampli ying his. In o he
wo ds, he bias is al eady he e and AI jus makes i isible. An on desc ibes
algo i hmic bias as a human p oblem. He jus i ies his iew by s a ing ha
human bias becomes mani es in da a se s and in he machine lea ning p o-
cess, hose biases a e lea ned by a machine. An on sugges s ha AI simply
copies biases om humans. One migh p esume ha such algo i hmic bias
hus dampens he op imism o AI, bu An on in his nex discu si e mo e
sugges s ha AI p o ides he unique oppo uni y o make human biases
isible in a sys ema ic way, and o en o he i s ime. He acknowledges
ha biases exis ed be o e bu hey we e hidden and AI makes hem isible,
angible and, mos impo an ly o An on, add essable. The eby, he s ance
o echno- op imism is e ained by a guing ha algo i hmic bias is bene i-
cial in so a as i makes human bias isible h ough echnology. Howa d
made a simila a gumen . Howa d a gues ha machine lea ning picks up
he pa e ns om human beha iou and ampli ies hem, bu i can also
pu a spo ligh on hose biases. Howa d sugges s ha companies can use
AI and machine lea ning o show hem whe e hey ha e disc imina ed in
he pas by analysing his o ic da a. As such, pa e ns o disc imina ion can
be made isible, which by consequence would be good o o ganisa ions.
The discu si e mo es ha Howa d and An on display a e s ikingly simila
in ha nega i e conno a ions o algo i hmic bias a e u ned in o a bene i
o using AI.
A a ia ion o his s ance is ha humans a e biased. Hen y alked abou
how humans a e biased and he e o e p oduce biased da a. He s a es ha
ec ui e s and hi ing manage s o en belie e ha hey a e excellen a
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 81
hi ing, bu Hen y is o he opinion ha once you look a hei a ing, i
becomes ob ious ha hey a e no e y good a a ing people, and by ex en-
sion, iden i ying he mos sui able pe son o he job. Hen y, o ins ance,
main ains ha machines a e less biased han human e alua o s by gi ing
he example o a s udy ha looks a how hi ing manage s e alua e com-
pe ence in emale- and male- domina ed occupa ions (he ci es nu ses and
cons uc ion wo ke s). He uses his s udy o sugges ha he machine sco e
was less biased han he human sco e. He concludes ha i an algo i hm is
co ec ly designed, i is less biased han un ained human e alua o s. Hen y
displays echno- op imism in so a as a well- designed and es ed algo i hm
is cons uc ed as supe io o humans, no only in selec ing he bes - sui ed
pe son bu also in a oiding bias.
Humans gene ally appea as he bea e s o bias in he decision- making
p ocess. Jasmine said, o ins ance, ha AI- suppo ed hi ing is o en used
o make p ocesses easie , bu o he , he eal bene i is ha i s anda dises
p ocesses. S anda dising p ocesses inc eases ai ness up un il he poin when
human decision- make s ha e a say. This can happen, acco ding o Jasmine,
o example, in ace- o- ace in e iews o when a human decision- make is
gi en he sco es o he indi iduals ha bes i he job bu hen only picks
he men o ake hem u he . Fo Jasmine, AI can be used o make hi ing
ai e and o educe disc imina ion, bu she sugges s ha once humans
in luence he decision- making p ocess bias c eeps back in. Like wi h Hen y,
Jasmine cons uc s AI as ai e han humans. She says ha ega dless o how
much one ies o educe bias in AI, because once human decision- make s
en e he pic u e, bias can e- eme ge. Such an idea is indeed suppo ed
by esea ch, which has shown ha using AI in ec ui men can lead o a
educ ion in di e si y, bu ha his is no due o algo i hmic bias bu a he
due o he ec ui men manage : he ec ui men manage s, o ins ance,
migh no ollow he machine- gene a ed anking and he eby in oduce
bias (Bu sell & Roumbanis, 2024).
Fixing he Da a and he Ra e
In o de o main ain he s ance o echno- op imism, i was common o
a gue ha algo i hmic bias is ixable. The discou se o sugges ha algo-
i hmic bias is ixable i s o all ela ed o da a. This is no su p ising because
mos people see da a as he oo cause o algo i hmic bias, as we ha e seen
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE
82
ea lie in his chap e . Ginny alked abou ha checking he aining da a
ha models lea n om is pa amoun o a oid ha he da a is eplica ing
biases. Kenne h sugges ed ha aining da a needs o be di e si ied o c ea e
ep esen a i e da a se s, and he opines ha big companies a e al eady doing
his. He u he speci ies how his is done: i you ha e a aining da a se
based on 99% da a om men and 1% da a om women hen i is neces-
sa y o balance he da a inpu o 50% women and 50% men and o p o-
ceed acco dingly in ega d o ace by including da a based on 25% Black
popula ions. An on desc ibes i as me ely a ‘ echnical p oblem’ o build da a
se s ha do no su e om bias.
Isabel uses a simila a gumen bu is mo e scep ical ha changing he
inpu will be enough. She main ains ha i he e a e ew da a poin s o
women and e en ewe o Black women, hen any p edic ions ha an AI
sys em eaches will be weake o hese g oups. Fo he , he p oblem lies
in his o ic da a, which will always eplica e he same ou comes. Ins ead,
she sugges s ha comple ely new da a se s need o be c ea ed, which is
simila o wha Kenne h men ioned. Howe e , he app oach o ix he da a
is di e en : in he hi ing echnology company whe e she wo ks, hey ha e
decided no o use his o ic da a a all. They do no use CVs and hey do no
use wha ‘good’ looked like in he pas . Ins ead, hey a e ying o build new
da a se s ha do no su e om his o ic bias. Howe e , much o wha she
discusses ela es o u u e de elopmen s whe e she is op imis ic ha one
day, hi ing decisions can be made wi hou bias. This s ong belie in he
po en ial o echnology and ha algo i hmic bias is ixable is a ypical dis-
cu si e mo e o main ain he pe spec i e o echno- op imisms, in spi e o
he challenges associa ed wi h algo i hmic bias.
Apa om ixing ep esen a ion in da a, ixing he human who
p oduces inpu da a was also lagged as impo an by many people I spoke
wi h. While he ixing he da a idea discussed be o e la gely cen ed on
changing ep esen a ions, hese a gumen s ocused mo e on how da a
eme ges. The classic example is ha a manage migh display gende bias
and hus a e women lowe in pe o mance e alua ions, which a e hen
ed in o HR sys ems (Edwa ds & Edwa ds, 2019). In some cases, he man-
age does no e en ha e o be biased, bu gende ed da a inpu can esul
om inpu ha is well- in ended bu has he opposi e e ec . We migh
he e hink abou he women ha we e anked h ee ou o i e while on
ma e ni y lea e, as men ioned ea lie in his chap e . Al hough none o he
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 83
in e iewees used his speci ic example, human a e s we e men ioned in
ega d o how eco ded in e iews we e sco ed. Such eco ded in e iews
a e o en used in hi ing, whe e on a hi ing pla o m, a candida e is
eco ding answe s o speci ic ques ions ia ideo. In his in e ac ion, he
in e ac ion pa ne is ac ually he machine ha poses ques ions and he
in e iewee answe s. A common assump ion is ha hose in e iews a e
sco ed by AI. Howe e , Ki s y and Ginny cla i ied ha his is no he case.
In Ki s y’s o ganisa ions, such eco ded in e iews a e sco ed by humans
who a e ained assesso s who assess based on a ma ix o ub ic. The
same is ue o Ginny, who s a es ha only 20% o eco ded in e iews
a e sco ed by machines, which means ha 80% a e sco ed by humans.
Again, he humans a e ained indus ial/o ganisa ional psychologis s
who e alua e based on ub ics. These ub ics a e de eloped by he indus-
ial/o ganisa ional psychology eam a e conduc ing a job analysis o
ensu e ha he co ec compe encies a e measu ed. These compe encies
hemsel es a e assessed h ough models ha p edic hose compe encies,
and hose models we e de eloped in he company based on ained expe
a e s.
Ginny goes on o p o ide an example o how his wo ks when a candi-
da e is assessed in ela ion o cus ome se ice. In he eco ded in e iew,
candida es a e asked o desc ibe a ime when hey had o deal wi h a di icul
cus ome . The o ganisa ion has housands o examples o how o he people
answe ed his ques ion, based on which he model, and he ub ics we e
de eloped and de ined. The ained e alua o s hen e alua e he candida es’
answe s as good, medium o low, o example. Ginny desc ibes how hey
o iginally hough ha humans a e biased and ha he e o e, he sco ing
would be biased oo. Howe e , hey disco e ed ha wi h his s anda disa-
ion h ough ub ics and ained assesso s, human bias is educed. Ginny’s
a gumen sugges s ha he human bias in c ea ing da a can be minimised
h ough aining humans and assessing in ub ics. I ixes he messy and
un uly human pa , which Ginny desc ibes as people going on hei gu
eeling based on uns uc u ed in e iews o make hi ing decisions. This
gu eeling Ginny sugges s has impac ed he hi ing p ocess in he pas . I
a gu eeling is used o hi e candida es and his da a is used in machine
lea ning, he esul ing hi ing p ocess will be sa u a ed wi h human biases.
By educing his gu eeling and he human biases, he po en ial algo i hmic
bias in hi ing is also educed.
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE
84
Hen y also alked abou imp o ing algo i hms. Hen y who wo ks in a
hi ing echnology company insis ed ha 90% o his company’s algo i hms
a e based on human a e s. In o de o illus a e why his is impo an , he
ci ed he saying ha ‘an algo i hm is only as good as he da a i ’s being
modelled upon’, which in machine lea ning is o en desc ibed as ‘ga bage
in, ga bage ou ’ (Weye e & Lange , 2019). Hen y explains ha his means
ha i you ha e bias in he da a, you a e modelling bias in you algo-
i hm. As such, i is cen al o a oid his bias in da a inpu . To build hei
models, he company has mul iple ained human e alua o s e alua ing
each in e iew. They hen check he ag eemen be ween he model and a
human e alua o . As such, he p edic i e powe o he models is measu ed
agains expe human p edic ions. Hen y a gues ha by educing he bias
in human- a ed da a, he p edic i e models o AI can be imp o ed. Like
in Ginny’s example, he idea ha human a ia ion in assessmen makes
algo i hms biased, o mo e b oadly ha socie al bias is shaping algo i hmic
bias, is being employed o ad ance he s ance ha human bias has o be
educed in da a inpu o a oid algo i hmic bias. This means ha i human
a e s a e ‘ ixed’, algo i hmic bias can be ixed oo.
Fixing he Algo i hm
Apa om ixing he da a and he (human) a e , in e iewees also
sugges ed o ix he algo i hm. This can ake di e en o ms. F anklin alked
abou a p o ide ha emo ed any pe sonal in o ma ion om applica ions
o ensu e ha candida es a e e alua ed based on skills. Among he in o -
ma ion ha is excluded is whe e he candida e wo ked be o e. F anklin
sugges s ha big echnology companies a e domina ed by men and he
p o ide is emo ing his in o ma ion o allow hi ing manage s o ocus
on skills a he han imp essi e sounding company names. The candida es’
names, gende and ace/e hnici y a e also emo ed. Al hough i could be
a gued ha emo ing such in o ma ion om candida es’ p o iles educes
he abili y o he AI o lea n any bias and o in luence he hi ing man-
age , in mos cases, much o such in o ma ion would be e ealed a la e
in e iew s ages, a which poin i can s ill in luence hi ing manage s.
B adley simila ly alked abou ways in which algo i hms a e blinded by, o
example, excluding i a candida e iden i ies as emale o male, which a e
he wo gende op ions B adley men ions. Acco ding o B adley, he idea is
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 85
ha algo i hms hen canno dis inguish be ween who is emale and who
is male and as such, bias migh be educed. B adley is scep ical ha such an
app oach ul ima ely wo ks bu he is awa e ha i exis s.
Ano he app oach o dealing wi h algo i hmic bias h ough blinding
he algo i hm is p o ided by Hen y. Hen y alks abou how his company
has collec ed millions o eco ded ideo in e iews and hey we e able o
de ec gende and acial di e ences in how ques ions a e answe ed and
which wo ds a e used. He explains ha men would use di e en wo ds
o women o desc ibe ce ain compe ences. His company’s app oach is o
‘blacklis ’ hose wo ds o a oid ha hey a e included in he algo i hm.
The eby, Hen y sugges s ha he algo i hm would no be in luenced by
gende di e ences in ega d o which wo ds a e used o exp ess compe-
encies. Ginny sugges s ha he ‘beau y o algo i hms’ is ha e en i he
aining da a is biased, his can be mi iga ed in algo i hms o a oid ha
his bias is ep oduced. She acknowledges ha you need o know wha you
a e doing o a oid ha . Howe e , in he iew, i is possible o con ol he
algo i hmic much mo e han he human mind. Ginny also sugges s ha
in o ma ion migh be in e ed by speech: women migh alk mo e abou
childca e, whe eas men alk abou ugby. He solu ion, simila o wha
Hen y sugges ed, is o ‘block’ he wo ds ‘ ugby’ and ‘childca e’ o blind he
algo i hm o such gende di e ences. Such app oaches sugges ha i is
possible o blind algo i hm by ei he no including ce ain in o ma ion o
by blocking ce ain wo ds. The idea is ha i hose inpu s a e no included
in he p edic ions he algo i hm makes, i is possible o con ol bias. Again,
we see ha he a gumen ha i is easie o con ol algo i hm han i is o
con ol human minds is made, con ibu ing o he iew ha i he algo-
i hm is amed by excluding human bias, i is possible o use algo i hms o
make ec ui men ai e .
I was also common o alk abou quali y con ols ha a e implemen ed
in ega d o algo i hms. Lucy explained ha in he company, a second
algo i hm was de eloped ha checks he i s algo i hm o ai ness. She
exp esses ha his gi es people con idence ha he algo i hm is no buil
on bias. Simila ly, Ki s y acknowledges ha no algo i hm is pe ec , like no
selec ion ool is pe ec . She, he e o e, says ha in he company, he algo-
i hm is checked o ad e se impac o di e en p o iles e e y yea , which
allows ‘ weaking’ he algo i hms i need be. Ad e se impac is used in he
Uni ed S a es o e e o ‘ he nega i e e ec an un ai and biased selec ion
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE
86
p ocedu e has on a p o ec ed class’ (Mond agon, 2018), which can include,
in he Uni ed S a es, sex, ace, age and disabili y, among o he s. Duncan
exp esses a simila sen imen in ega d o he ac ha any algo i hm ha
makes impo an decisions, such as who ge s a job, should unde go quali y
assu ance o ensu e ha he e is cla i y o wha he algo i hm measu es.
Kenne h also sugges s ha one should engage an AI audi ing company
o assess he algo i hms ha a e used, which is some hing ha was done
by many o he p o ide s I spoke o. Kenne h, in pa icula , alks abou
A/ B es ing as impo an when audi ing an algo i hm. A/ B es ing is a
andomised expe imen ha in ol es an A e sion and a B e sion o see
i hey pe o m di e en ly. Kenne h gi es he ollowing example o wha
would happen i one looks o a op p og amme . He sugges s eeding he
AI sys em wi h in o ma ion om Black candida es and ew whi e candida es
and hen compa ing ha o a di e en composi ion o candida es o see i
he e a e di e ences in he esul . I he AI sys em picks one o he ew
whi e men, he a gues, and igno es he Black candida es, he e migh be bias
in he algo i hm.
Hen y desc ibes ha once a model is buil , his o ganisa ion ollows up
wi h an ad e se impac analysis, which is wha Ki s y men ioned ea lie . This
includes, as he says, o un s a is ics o see how men sco e e sus women
and e alua e he mean di e ences, he s anda d de ia ion di e ences and
so on. Ideally, men and women should sco e as equally as possible and i
he e is a di e ence, one has o ask why ha is and go back in o ind why
women and men sco e di e en ly. Ginny p o ides u he de ail by a guing
ha he e migh be easons why g oups sco e di e en ly, which would be
ine om a legal pe spec i e in he Uni ed S a es. This can include li ing
hea y loads as a job equi emen , which hen would mean ha mo e men
han women migh quali y. She desc ibes his as a bona ide equi emen .
Ginny also alks abou he ou - i h ule ha is o en used in ega d o
ad e se impac in he Uni ed S a es. The ou - i h ule is a way o assess
ad e se impac h ough inding ha he selec ion a e o one g oup is less
han 80% o he g oup wi h he highes selec ion a e, as ou lined in he
Uni o m Guidelines On Employee Selec ion P ocedu es (Biddle Consul ing
G oup, 2023). Howe e , Ginny asse s ha while he ou - i h ule is
o en ci ed, one could clea his hu dle bu migh s ill ind ha he e is
un ai ness. Ginny pa icula ly poin s o subg oups ha could be e alua ed
un ai ly, such as op- pe o ming women being e alua ed di e en ly han
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 87
lowe - pe o ming women. As such, i is necessa y o di ide g oups u he
o check in a mo e g anula way ha bias does no a ec he model by
including, o ins ance, in o ma ion on pe o mance. Al hough he ou -
i h ule is egula ly men ioned, Ginny, as well as o he s such as Alisha,
s ess ha he ou - i h ule is no he sine qua non o assess equali y and u -
he es s a e equi ed o es ablish ai ness.
O e all, i was o en a gued ha a de ailed assessmen is equi ed o
ensu e ha algo i hms do no p oduce bias. Mos in e iewees ag eed ha
his is no an easy ea bu ha i could be used o ensu e ha algo i hms a e
no p oducing biased p edic ions. In ha way, he iew ha algo i hms can
be ixed suppo s he s ance ha machines can be de- biased. This in u n
suppo s a echno- op imis ’s pe spec i e.
Machines Do No Make Hi ing Decisions, Humans Do
Howe e , he e was also a s ance ha cau ioned agains he use o AI in
hi ing. Those pe spec i es we e no echno- pessimis ic in he sense ha
echnology was seen as leading o poo e ou comes. This s ance was
cha ac e ised by being hesi an o use AI in hi ing. This s ance was no an
ou igh ejec ion o AI in gene al bu a hesi a ion o using AI a his poin
in ime.
Mos o he in e iewees sugges ed ha algo i hms could be ixed, bu
Alisha goes u he in ag eeing ha some imes an algo i hmic ix can be
ound, o ins ance, in a hi ing algo i hm whe e i is possible o ix he
da a o he model. Ye , his was o Alisha jus a s opgap measu e un il one
is able o ind a be e solu ion. She speci ies ha by be e , in his con-
ex , she means mo e equi able. Howe e , o he , echnologies a e o en
used o igno e p oblems a ound inequali ies. She goes on o explain ha
people o en hide behind he enee o he algo i hm. Alisha speci ically
s a es ha algo i hms a e said o be based on da a and ha implies ha
hi ing becomes mo e objec i e i algo i hms a e used. This in ac chimes
wi h a pe spec i e ha I ou lined ea lie in his chap e in ega d o echno-
op imism. Howe e , Alisha a gues ha some imes algo i hms a e simply
ools o ‘igno e di icul con e sa ions’. She sugges s ha hese a e di i-
cul con e sa ions abou inequali ies. As such, she concludes ha people
simply hide behind algo i hms and hei p esumed objec i eness wi hou
add essing inequali ies in a mo e p o ound way.
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE
88
Howe e , Alisha was he only pe son mobilising he idea ha algo i hms
a e a enee o some hing o hide behind. I was a mo e common o
sugges ha algo i hms imp o e human decision- making. Lucy s essed
ha algo i hms should no make all decisions bu ins ead, hey make
ecommenda ions ha can hen be aken u he wi h human knowledge.
As such, she poin s o he impo ance ha humans a e in he d i e ’s sea
when i comes o making decisions. Lucy quali ies he s a emen by saying
ha humans ob iously need o unde s and how he algo i hm a i es a a
sugges ion. In a simila ein, Jasmine suppo s he idea ha AI- suppo ed
hi ing is ai e bu she also says ha human o e sigh o e decisions is
impo an . Hen y was mo e speci ic in ha he sugges ed ha a machine
allows people o make be e decisions. He ag eed ha a machine should
no make he ac ual decision bu a machine can assis humans in imp o ing
decisions. Hen y jus i ies his poin o iew by saying ha he machine will
use s anda dised and consis en da a, which in u n can help humans o be
less biased. As such, hese ideas o how machines and humans collabo a e
s ill ollow he idea o echno- op imism in so a as hey a e used o sugges
ha human decision- making will imp o e, leading o be e decision.
Again, Alisha was sligh ly mo e hesi an abou his machine– human col-
labo a ion. Alisha con es s he idea ha hi ing manage s a e all powe ul.
She ag ees ha hey ha e some powe in o ganisa ions bu she poin s ou
ha he AI sys em has o be designed in such a way ha allows he hi ing
manage o ques ion he algo i hm o o e en o e u n he ecommen-
da ion. As such, she insis s ha humans would be able o con adic he
p edic ions a machine makes. Howe e , Alisha is no su e i humans will
ac ually do ha . Fo ins ance, i a candida e is sugges ed by a machine,
would a human ques ion his judgemen o would he human pick he
pa h o leas esis ance and ollow wha he machine ecommended. Fo
Alisha, ha is a ques ion o human na u e, whe e picking he pa h o leas
esis ance is common. Fu he mo e, she acknowledges ha sys ems o en
do no allow o dissen . A hi ing manage migh ha e less oppo uni ies
o ques ion he p edic ion o an ideal candida e o a machine because he
sys em has no been buil wi h his in mind.
When I spoke wi h F anklin abou he isks o using only candida es
ha a machine has sugges ed, F anklin ecalls a con e sa ion wi h one o
his clien s who his o ically has ec ui ed he op people by going wi h
who is on op o he s ack. Howe e , his clien said ha people u he
ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 89
down he lis a e o en less in demand and hey show g ea de elopmen al
po en ial. F anklin sugges s ha candida es in he middle o he pile would
ul ima ely be be e people o hi e in he long un. This isk o only hi ing
he bes - sui ed pe son o he job a he hi ing mid- ange people who
migh be able o de elop is howe e no a unc ion o using AI in hi ing.
As a ma e o ac , much o how ec ui men is done is ocusing on he
pe son who bes i s he speci ica ion one has se ou . This is codi ied in he
Uni o m Guidelines On Employee Selec ion P ocedu es (Biddle Consul ing
G oup, 2023), which, as I men ioned be o e, is egula ly used when hi ing.
Algo i hms mechanise his p ocess and he e o e in ensi y he ocus on
hose who bes i he c i e ia se ou .
Ye , i has been a gued ha he e is a bene i in going o a ‘wildca d’ hi e
om ime o ime o b eak he pa e ns ha ha e been es ablished o e ime
o wha skills a e equi ed o do a job (Tambe e al., 2019). A mo e de el-
opmen al pe spec i e o hi ing would depa om much o how hi ing
looks like a his poin in ime. The e is a isk ha using machines o mech-
anise hi ing leads o an e en na owe ocus on hose who ma ch he skills
equi ed bes . While much conce n is cu en ly on e adica ing bias om AI
hi ing p ocesses by, o example, blocking language ha migh gi e away
gende o ace, he e is a wide ques ion in how a skills ha a e assessed
migh disad an age hose who ha e no de eloped hose skills ye bu who
could do so in he u u e. A he momen , skills ha a e assessed in hi ing a e
mo e baseline skills and o en no some hing ha canno be changed much.
Fo ins ance, he ‘Big Fi e’ pe sonali y assessmen , on which much hi ing is
based, assesses ex a e sion o i one is ou going. I i has been es ablished
ha a salespe son should be ou going and hus high on ex a e sion, his
i s he cu en model o a salespe son. One could p esume ha he use o
echnology is leading o his p o ile o he salespe son as ex a e is ge ing
mo e and mo e e ined. Howe e , o machine lea ning, b eaking hose
pa e ns wi h a ‘wildca d’ migh be as use ul. As such, s anda dising ec ui -
men migh be bene icial o educe bias in he hi ing p ocess, bu picking
someone unusual migh be help ul o in oduce a ia ions in pa e ns.
Howe e , such wide e lec ions, which could be desc ibed as echno-
hesi a ion, we e a ely b ough up in he in e iews. While some people
e lec ed on how humans and machines collabo a e in decision- making,
i was also clea ha i is no he case ha an AI makes a hi ing decision
wi hou human in luence. AI il e s people ou ha do no i he skills
IN/VISIBILITY BY DESIGN
96
a pic u e o wha wo ds a e linked o speci ic sounds. This happens h ough
labels ha connec an image o sound wi h a wo d ha he compu e can
unde s and. Designe s o AI commonly de elop classi ica ions o labels ha
hen need o be connec ed o da a. These labels a e o en assigned o da a
by wo ke s in he Global Sou h ei he as c owd wo k o in mo e adi ional
employmen o ms. While he wo king condi ions o hese wo ke s a e
egula ly he ocus o a en ion, hese wo ke s also play a cen al ole in c e-
a ing da a se s ha a e cons uc ed as uni e sal and objec i e, e en hough
hey a e based on complex p ocesses o meaning making en ailed in such
assessmen s. The chap e shows how hese subjec i e p ocesses o knowing
a e ha monised and s anda dised as an objec i e u h in da a labelling. I
is also discussed ha many o he classi ica ions used a e exclusiona y. Fo
ins ance, gende labels a e commonly concei ed as a bina y. This chap e
hus aces how da a labelling cons uc s a knowable wo ld. The chap e
ocuses on he mechanisms o cons uc ion and he in e na ional di ision
o labou ha hese p ocesses en ail.
AI’s Hidden Wo k o ce
I people pic u e someone who wo ks in AI, he images ha a e conju ed
up include highly paid da a scien is s, p obably whi e, a man and based in
Silicon Valley. Howe e , much o he wo k ha allows AI o lea n is done by
da a anno a o s who label da a, which in u n allows o machine lea ning
o happen. When Cha GPT, a cha bo de eloped by OpenAI, was publicly
launched in No embe 2022, many people ma elled a Cha GPT’s abili y
o gene a e ex ha is ha d o dis inguish om wha a human migh w i e
(Mollick, 2022; Abdullah e al., 2022). Howe e , unlike p e ious cha bo s
such as Mic oso ’s Tay (Vincen , 2016), Cha GPT did no p oduce acis and
sexis alk. Tha is no a coincidence. The p e ious e sion o he echnology
had in ac a endency o p oduce acis and sexis alk (Pe igo, 2023). In
o de o a oid ha , OpenAI ensu ed ha any hing ha could be seen as
acis and sexis would be il e ed ou in Cha GPT’s answe s (Pe igo, 2023).
Howe e , o de e mine wha is acis and sexis , he AI sys em needed o
lea n wha acism and sexism look like. The da a o Cha GPT was sc aped
om he in e ne . This ine i ably included da a ha could be seen as acis
and sexis . Since OpenAI wan ed o a oid ha Cha GPT ep oduces acis
and sexis language, he sys em had o lea n wha such language looks like.
IN/VISIBILITY BY DESIGN 97
In o de o iden i y acis and sexis language, OpenAI ollowed a simila
app oach ha is used by, o ins ance, Me a’s Facebook o il e ou oxic
language. This app oach equi es humans o label any da a ha is sexis and
acis o ensu e ha i can be excluded om he AI ou pu ha a cha bo
like Cha GPT migh p oduce (Pe igo, 2023). This equi ed human da a
anno a o s o label da a ha could be seen as acis o sexis (Pe igo, 2023).
Such human- in- he- loop app oaches a e o en ou sou ced o o ganisa ions
in he Global Sou h. This human inpu is o en in isible and le ou o ocus
when he p esumed achie emen s o AI a e ma elled a .
Hiding he human inpu in echnology is in ac no a new phenomenon.
Amazon’s MTu k is a case in poin . MTu k s ands o Mechanical Tu k and
is hus a e e ence o a 18 h- cen u y li e- size chess- playing au oma on ha
was d essed in O oman clo hing (S ephens, 2023; Geoghegan, 2020; G ay
& Su i, 2019; S andage, 2002) (see also Chap e 1). Wol gang on Kempelen
de eloped he au oma on and p esen ed i o he i s ime in 1770 a
he Habsbu g cou , and i hen was exhibi ed in Eu ope and he Uni ed
S a es (S ephens, 2023; Geoghegan, 2020). The chess- playing au oma on
p e ended o be a machine ha played and o en won agains humans.
Howe e , ins ead o being an au oma ed chess machine, he au oma on
equi ed a human hiding in he machine who pe o med he chess mo es
ha we e hen ansla ed ia mechanics o he chess boa d (I ani, 2015;
S ephens, 2023; Geoghegan, 2020; S andage, 2002). As many people a he
ime al eady p esumed, he Tu k u ned ou o be a hoax (S ephens, 2023).
Howe e , as S andage sugges s, he a i al o he au oma on
coincided wi h he beginnings o he indus ial e olu ion, when
machines i s began o displace human wo ke s, and he ela ionship
be ween people and machines was being ede ined. The chess playe
posed a challenge o anyone who ook e uge in he idea ha machine
migh be able o ou pe o m humans physically bu could no ou do
hem men ally.
(S andage, 2002, p. xi )
I was pa icula ly he au oma on’s p esumed abili y o in e ac wi h i s
opponen s du ing he chess game ha was deemed implausible because
i equi ed machine in elligence (S andage, 2002). No su p isingly, he
ac i i y o playing chess is s ill seen as one way o e alua ing machine
IN/VISIBILITY BY DESIGN
98
in elligence agains ha o humans (S andage, 2002). The Mechanical Tu k
hus aised he spec e ha machines can eplace human in elligence and
no jus he physical powe o humans (S andage, 2002). As we ha e seen
in Chap e 3, a simila ques ion is aised oday in ega d o wha skills a e
uniquely human and i machines a e able o emula e emo ions, which up
o now ha e been seen as a human ad an age.
Bu why was he au oma on called he Tu k? Al hough on Kempelen
ne e named he au oma on a Tu k, he au oma on wo e O oman d ess,
which led o he name Tu k. The eason why he au oma on was d essed in
O oman d ess exp esses a o m o O ien alism bu also e lec s he long-
s anding i al y be ween he O oman and he Habsbu g empi es including
he Tu kish siege o Vienna (Geoghegan, 2020). I has also been sugges ed
ha he Tu kish s yle was popula in Vienna a he ime (S andage, 2002).
Ano he eason ha he name Tu k was adop ed migh in ac ela e o
Ge man language, whe e he e b ‘ ü ken’ ansla es as ‘ o u k’ and means
‘ o ake’. The e b has s ong pejo a i e conno a ions, which is why i is
seen as disc imina o y and should be a oided (Duden, 2023; Geoghegan,
2020). The e ymology o he e b ‘ ü ken’ is unclea (Duden, 2023;
Geoghegan, 2020) and one po en ial o igin o he e b in ac goes back
o he Mechanical Tu k being a ake (Geoghegan, 2020). Ye , nei he he
o igins o he e b no he o igins o he au oma on ha e been conclu-
si ely shown.
The name MTu k, which Amazon has chosen o i s se ices, is emin-
iscen o on Kempelen’s au oma on. The MTu k se ice de eloped ou o
Amazon’s a emp s o educe he numbe o duplica e lis ings (S ephens,
2023; I ani, 2015). Amazon ied o au oma e his ac ion bu ailed because
‘[ ] he ask equi ed a ce ain ype o pa e n ecogni ion – he abili y o
de ec sub le di e ences and simila i ies be ween pic u es and ex – which
we e easy o a human b ain bu could no be eplica ed by compu e (sic)’
(S ephens, 2023, p. 66). In o he wo ds, he e a e ce ain pa e ns ha
only humans can ecognise. The e o e, Amazon decided o gi e small, indi-
idual asks o wo ke s who could comple e he asks in piecemeal wo k
(S ephens, 2023). Amazon hen o e ed he se ice o o he clien s, leading
o he de elopmen o he pla o m MTu k, whe e clien s’, o eques e s’,
asks we e ma ched wi h people willing o do hese mic o asks o o en
small amoun s o money (S ephens, 2023). As men ioned be o e, such
app oaches a e o en called ‘human- in- he- loop’. A human- in- he- loop
IN/VISIBILITY BY DESIGN 99
app oach is equi ed when human in elligence is needed o comple e a ask.
Je Bezos calls MTu k ‘a i icial a i icial in elligence’ (S ephens, 2023).2
MTu k became emblema ic o c owdsou ced pla o m wo k (I ani,
2015; Howc o & Be g all- Kå ebo n, 2019). As Howc o and Be g all-
Kå ebo n s a e, ‘[o] nline ask c owdwo k o e s paid wo k (some imes sub-
jec o eques e sa is ac ion) o speci ied asks and he ini ia ing ac o is
he eques e ’ (Howc o & Be g all- Kå ebo n, 2019, p. 26). Resea ch on
MTu k and o he pla o ms has egula ly s essed he exploi a i e na u e o
hese ypes o wo k (I ani, 2015; Howc o & Be g all- Kå ebo n, 2019).
Apa om being o en seen as economically p eca ious wo k, he wo k can
also lea e indi iduals men ally sca ed; i has been a gued ha men al heal h
issues a ise in many people who mode a e social media con en (Bui, 2020;
I ani, 2016). In hei g ound- b eaking s udy, G ay and Su i (2019) desc ibe
such wo k as ‘ghos wo k’ because we o en p esume ha he wo k is done
by a machine bu in ac he wo k is comple ed by humans. An example is
secu i y backg ound checks o Ube d i e s whe e he d i e has g own
a bea d and as such no longe ma ches he image on ile; a human is hen
asked o de e mine i he pe son signing in as a d i e is indeed he same
as he pe son on ile (G ay & Su i, 2019). Th ough hei de ailed s udy o
pla o m wo k, G ay and Su i (2019) show how ‘algo i hmic c uel y’ is
a ec ing indi iduals engaging in his wo k. They de ail he s uggle o ind
wo k and ge paid, and he isola ion en ailed in hese ypes o wo kplaces
(G ay & Su i, 2019).
Due o he c i icism o pla o m wo k as being exploi a i e, many com-
panies ha e s a ed o engage in p ac ices o sus ainable sou cing by ou -
sou cing such wo k o p o ide s ha o e s able employmen condi ions
o hei wo ke s (G ay & Su i, 2019). In a emp s o make supply chains
mo e sus ainable, o ganisa ions ha equi e mode a ion o social media
da a o da a anno a ion ha e s a ed o p e e supplie s like Sama o iMe i ,
whose mission is o o e wo k and hus a li elihood o indi iduals in
he Global Sou h (Pe igo, 2022,2023; Mu gia, 2019). Al hough hese
wo kplaces ha e been lauded as he angua d o AI (Mu gia, 2019), i has
also been epo ed ha hose wo kplaces can be exploi a i e in hei own
igh (Pe igo, 2022, 2023; Pilling & Mu gia, 2023, 2023, 2023; Pilling,
2024). As men ioned be o e, such wo k o mode a ing social media o
labelling o ensi e language o en lea es employees in such i ms psycho-
logically sca ed (Pe igo, 2022; Pilling & Mu gia, 2023; Pilling, 2024). I
IN/VISIBILITY BY DESIGN
100
is sugges ed ha o AI o wo k in he Global No h, wo ke s in he Global
Sou h ha e o pu hei men al heal h on he line3 (Pe igo, 2022). As such,
e o s o ensu e ha echnologies a e ee om sexis , acis and ha m ul
language and image y a e o en me by he challenges o global supply
chains.
The Need o Da a Anno a ion
Al hough wo king condi ions a e igh ly a he cen e o many discussions
on da a anno a ion, a his poin , i is use ul o explo e why da a anno-
a ion o da a labelling is needed in AI in he i s ins ance. In a nu shell,
machine lea ning, a subse o AI, is o en desc ibed as a pa e n ecog-
nise – a pa e n is spo ed and his is used o make p edic ions (Caliskan
e al., 2017). No all machine lea ning equi es labelled da a. In unsupe -
ised machine lea ning, da a labels a e no equi ed, bu o supe ised
machine lea ning, da a needs o be labelled (Bechmann & Bowke , 2019).
Fo supe ised machine lea ning, some da a migh al eady be labelled in
he da a se , which a e hen used o build models. In o he cases, he da a
migh equi e labelling. Fo ins ance, o sel - d i ing ca s, i is necessa y o
a machine o be able o ead how a s op sign looks, wha a bus looks like
o how humans o di e en shapes and sizes appea . Such image da a hus
has o be labelled o ell he machine exac ly wha a s op sign looks like.
A simila p ocess is ollowed o language ha is, o ins ance, equi ed o
oice ecogni ion so wa e. As such, da a anno a o s ha e o label images o
language in da a se s ha can hen be used o machine lea ning. Mos o us
engage in da a labelling ee o cha ge: when asked o p o e ha we a e no
a obo in online in e ac ions, we ha e o, o example, selec all pic u es
ha ha e a mo o cycle in i . The ac ha we need o iden i y images and
ex s o show ou humani y illus a es ha machines s uggle wi h his
ac i i y, c ea ing he need o humans o label da a in he i s place.
Da a has o be labelled o es ablish some hing ha is called ‘g ound
u h’. G ound u h is a e m used in compu e science and da a science
and is cen al o how da a is used in algo i hms (Ja on 2017, 2021). Ja on
(2017, 2021) desc ibes how g ound u h is ele an in supe ised machine
lea ning: one s a s wi h a da a se and his da a se is labelled by humans
wi h clea a ge s (e.g. oad signs, ca s, humans), which he algo i hm will
ha e o iden i y. I he e is disag eemen among da a labelle s on how o
IN/VISIBILITY BY DESIGN 101
label da a, o en, he majo i y o e is used o es ablish wha should coun as
g ound u h (McCluskey e al., 2021). The labelled da a and he unlabelled
da a o m a da abase ha is called g ound u h (Ja on, 2017). Mo e spe-
ci ically, ‘g ound u h e e s o in o ma ion ha is assumed o be ue o
an (sic) ML [machine lea ning] sys em’ (Kang, 2023, p. 1). The da a se
is hen spli in o a aining se and an e alua ion se (Ja on, 2017, 2021).
The aining se allows he designe s o ‘ex ac o mal in o ma ion abou
he a ge s and ansla e hem in o ma hema ical exp essions’ (Ja on, 2017,
p. 815). These ma hema ical exp essions a e hen ans o med in o code
and he algo i hm is es ed on he o he se , which is called he e alua ion
se (Ja on, 2017, 2021). I is hen assessed i he algo i hm unc ioned as
expec ed by compa ing he esul wi h he da a labels assigned by humans
(Ja on, 2017, 2021). The g ound u h is hus based on he labels humans
ha e assigned o da a and allows o check o he co ec ness o he algo-
i hm (G osman & Reigelu h, 2019). The g ound u h is as such a way o
compa e wha a machine lea ned wi h wha human labelle s judged o be
he case. The human labou ha goes in o de eloping an algo i hm and
seeing how well i is pe o ming is cen al. In o he wo ds, he human
labelling da a helps machines o know wha is ue.
Al hough he name – g ound u h – sugges s objec i i y, es ablishing
g ound u h is an in e p e i e p ac ice (Miceli e al., 2020; Hen iksen &
Bechmann, 2020; Paullada e al., 2021). Da a labelle s commonly ecei e
ins uc ions on how o label a ex o an image om he designe s o AI,
which has been desc ibed as a way in which powe om he designe s o
AI on o he people who label da a is exe ed (Miceli e al., 2020). How
he designe s o AI es ablish hose classi ica ions has been desc ibed as
subjec i e, and in some cases, a bi a y, and hese classi ica ions a e hen
c ea ed and pe pe ua ed h ough AI sys ems (Miceli e al., 2020; Noble,
2018; Eubanks, V., 2018). Wi hin he con ines o he desc ip ions p o ided
by he designe s o AI, he da a labelle s o en ha e o make subjec i e
decisions (Miceli e al., 2020). This means ha da a anno a ion is a sense-
making p ac ice (Miceli e al., 2020). The e o e, da a labelle s migh dis-
ag ee on how o label da a. As p e iously men ioned, he majo i y o e
is egula ly used in such cases (McCluskey e al., 2021). Howe e , using
a majo i y o e obscu es ins ances whe e da a anno a o s sys ema ically
disag ee which is pa icula ly impo an in subjec i e asks, like assessing
ha e speech o a ec (Da ani e al., 2022). I he disag eemen is aken
IN/VISIBILITY BY DESIGN
102
in o conside a ion when models a e being buil , he esul ing models a e
sugges ed o pe o m be e (Da ani e al., 2022). In consequence, aking
he subjec i i y o decisions ha da a anno a o s migh make in o conside -
a ion is impo an . Howe e , in he p ocess o machine lea ning, hese sub-
jec i e decisions a e o en obscu ed behind a p esumed objec i i y de i ed
h ough majo i y o es. E en he name, g ound u h implies depic ing an
objec i e eali y. Ye , esea ch has shown ha p esumed objec i e ca ego ies
a e egula ly based on in e p e a ion (Bowke & S a , 2000). Al hough he
issue o da a anno a ion and subjec i e decision- making is o en aised in
ega d o supe ised lea ning, i has been shown ha unsupe ised lea ning
also elies on human supe ision in ega d o, o example, da a cleaning
o se ing he numbe o opics (Bechmann & Bowke , 2019). E en he
inclusion o exclusion o da a in a da a se can be seen as a way in which
con ex ual ac o s in luence and shape wha he machine can lea n (Den on
e al., 2020; Miceli e al., 2020).
Since da a se a e pi o al o machine lea ning, i has been sugges ed
ha da a se s should come wi h da ashee s ha desc ibe why he da a was
collec ed, wha he da a is composed o , how he da a was collec ed, and
wha he da a should be used o , among o he issues (Geb u e al., 2021).
A simila app oach is also ollowed in he elec onics indus y, whe e
da ashee s a e c ea ed o each componen ha de ails es esul s, usage
and ope a ing cha ac e is ics (Geb u e al., 2021). O he simila app oaches
a e ollowed o d ugs, which a e accompanied by in o ma ion on how
o use and wha he side e ec s migh be. Including da ashee s o da a
se s could, o ins ance, en ail in o ma ion on which subpopula ions a e
included and how hey a e dis ibu ed in he da a se (Geb u e al., 2021).
In ega d o da a collec ion, i should be conside ed which c owd wo ke s
we e used and how hey we e compensa ed (Geb u e al., 2021). Simila ly,
in hei six h p inciple o da a eminism, D’Ignazio and Klein (2020) dis-
cuss how con ex is ele an o da a collec ion. They a gue ha da a is no
neu al and ha he con ex in which he da a is collec ed, analysed and
communica ed is impo an o make powe dynamics isible (D’Ignazio &
Klein, 2020). In an empi ical applica ion o hose concep s, Miceli and co-
au ho s (2021) ask how he con ex o p oduc ion in da a image se s can be
made isible. They show ha clien s a e gene ally esponsible o de ining
classi ica ions based on which, o example, ace should be labelled in he
da a. Clien s a e in he d i ing sea in ega d o de ining hese classi ica ions
IN/VISIBILITY BY DESIGN 103
and ca ego ies. Gi en he ac ha clien s a e o en based in he Global
No h, whe eas he labelle s a e o en based in he Global Sou h, his also
in oduces a powe dynamic (Miceli e al., 2021). The esea ch also s esses
ha o ganisa ions migh be hesi an o ex ensi ely documen he con ex in
which da a se s a e c ea ed due o he ime, e o and complexi y in ol ed
in doing so (Miceli e al., 2021). As such, i is i al o de elop e ec i e ways
o including he con ex in which da a se s a e c ea ed, including he powe
dynamics a play in any da ashee s o da a se s p o ided.
Humans a e cen ally impo an o helping machines lea n by
ecognising and labelling pa e ns. This p ocess needs o be unde s ood as
a subjec i e one. As such, any subjec i e decisions, o example, associa ed
wi h de ining ca ego ies need o be ecognised as such by documen ing
hose decisions. Addi ionally, da a se s need no only include in o ma ion
on how subjec i e decisions we e made bu also on he wo ke s who label
he da a and as such in oduce hei own subjec i e decision- making in
he p ocess. Da a se s need o include de ails on he con ex in which hose
decisions a e being aken, such as he labou condi ions o hose who label
da a as well as he powe dynamics be ween he clien in he Global No h
and he p o ide o en in he Global Sou h. The eby, i would be possible o
make he human labou and he subjec i e decision- making ha goes in o
da a se s isible.
Classi ying he Wo ld
Classi ica ion is a cen al ac i i y o machine lea ning bu he poli ical
dimension o classi ica ion is o en igno ed (C aw o d, 2021). Bowke
and S a (2000) desc ibe he ac o classi ying o as hey call i ‘so ing
hings ou ’ as deeply human. In o he wo ds, classi ica ions a e a o m
o o ganising he wo ld. A classi ica ion is de ined as a ‘spa ial, empo al
o spa io- empo al segmen a ion o he wo ld’ (Bowke & S a , 2000,
p. 10). Ideal classi ica ions ollow unique and consis en p inciples such
as a empo al o de (Bowke & S a , 2000). Ca ego ies a e mu ually exclu-
si e and each ins ance i s in o jus one ca ego y (Bowke & S a , 2000).
Ideally a classi ica ion sys em co e s all po en ial ins ances, bu his ideal
is ne e ully achie ed in eali y (Bowke & S a , 2000). ‘[C] lassi ica ions
a e powe ul echnologies’ (Bowke & S a , 2000, p. 319), which by being
embedded in in as uc u es become in isible. This is cen al o how hey
IN/VISIBILITY BY DESIGN
104
un old hei powe . As such, Bowke and S a (2000) a gue o ecognising
he a chi ec u es o classi ica ions as poli ical and o challenging he aken
o g an ed s a us ha classi ica ions o en ha e. C aw o d (2021) d aws
a en ion o he ac ha AI is based on classi ica ions ha a e embedded in
in as uc u e and ha a e poli ical. Howe e , aining da a se s and AI in a-
s uc u e a e egula ly seen as ‘pu ely echnical’, e en hough ‘ hey na u -
alize a pa icula o de ing o he wo ld which p oduces e ec s ha a e seen
o jus i y hei o iginal o de ing’ (C aw o d, 2021, p. 139).
In classi ica ions a ound gende and ace, he p esumed ideal is ha ca -
ego ies a e clea ly de inable, clea cu and mu ually exclusi e. Gende and
ace a e ea ed as au oma ically de ec able and as some hing ha can be
p edic ed by AI sys ems (C aw o d, 2021). I is common o da a se s o
ollow a bina y classi ica ion o gende such as using one o emale and
ze o o male and an equally unde complex classi ica ion o ace o maybe
i e g oups (C aw o d, 2021). This is highly p oblema ic as he example o
how IBM ied o deal wi h algo i hmic bias shows: IBM aimed o inc ease
he di e si y in da a on acial ecogni ion and hey asked c owd wo ke s o
label aces as ei he male o emale on a bina y classi ica ion; ye anyone
who was no nea ly i ing in o his bina y was excluded om he da a se
(C aw o d, 2021). When acing how classi ica ions a e used in ImageNe ,
an image da abase, C aw o d (2021) shows how a ailable classi ica ions
unde ‘adul body’ con ain ‘adul male body’ and ‘adul emale body’,
whe e male and emale a e na u alised. He e, gende is classi ied in bio-
logical e ms and as a bina y. The e is an op ion o ‘he maph odi e’ bu his
is classi ied unde bisexual (C aw o d, 2021). C aw o d (2021) concluded
ha non- bina y indi iduals a e ei he igno ed o placed in a ca ego y ela ed
o sexuali y. This means, as C aw o d (2021, p. 146) sugges s, ‘[m] achine
lea ning sys ems a e (…) cons uc ing ace and gende : hey a e de ining he
wo ld wi hin he e ms hey ha e se ’ (i alics in o iginal). Ye , hese sys ems
hide he poli ics en ailed in hei cons uc ion, which p i ileges clea -
cu ca ego ies o e he complexi ies o e e yday li e. These classi ica ions
a e no only o de ing he p esen bu hey a e, h ough AI sys ems, also
pe pe ua ed in he u u e s uc u ing o how he wo ld is classi ied in he
yea s o come.
Following he idea ha machines a e cons uc ing gende , i is e i-
den ha o machine lea ning o happen, gende as a ca ego y has o be
cons uc ed. This en ails o de ine wha gende is and o ope a ionalise
IN/VISIBILITY BY DESIGN 105
gende o allow a machine o ecognise gende (Keyes, 2018). How gende
is de ined and ope a ionalised in machine lea ning has an e ec on how
gende is p edic ed (Keyes, 2018). This is happening, o ins ance, in
Au oma ic Gende Recogni ion, whe e gende is ‘ ead’ om pho og aphs
(Keyes, 2018). The gende bina y has long been ques ioned by esea ch
(Bu le , 1990; Faus o- S e ling, 2000) bu an analysis o pape s on human–
compu e in e ac ion has shown ha gende is ea ed as a bina y 94.8%
o he ime (Keyes, 2018). The esea ch also ound ha gende is no only
ope a ionalised as bina y bu also as physiological and immu able (Keyes,
2018). Keyes (2018) sugges s ha his can be ele an , o ins ance, in
ega d o billboa ds ha should show d esses o women and ca s o men;
i a ansman passes such a billboa d and is shown d esses, he billboa d
will ha e concluded ha he ansman is a woman. A consequence o his is
ha ansgende indi iduals a e likely o be misclassi ied, misgende ed and
ul ima ely e ased (Keyes, 2018). I is he e o e impo an o explo e how AI
p oduces a speci ic e sion o eali y by explo ing, o example, who is seen
as a woman in acial ecogni ion sys ems (D age & F abe i, 2023). As such,
a machine is eading he gende o a pe son h ough a bina y classi ica ion
sys em ha is hen ep oduced in he p edic ions made.
Helping AI o Unde s and he Wo ld
In o de o illus a e how da a p ac ices help AI unde s and he wo ld,
I would now like o u n o how people I in e iewed spoke abou such
p ac ices. Fi s , many in e iewees add essed why da a labelling is needed
in he i s place. Kenne h sugges ed ha only abou 30% o da a labelling
is au oma ed oday. Kenne h explained ha AI is unable o ecognise moun-
ains in a pic u e o a middle- aged man. Fo machine lea ning o happen, a
pic u e needs o be labelled wi h such in o ma ion o be ained. Kenne h
s a es ha AI needs o ha e da a ha is labelled wi h in o ma ion such as
wha is a ca and wha is a dog, and o his, da a labelling is impo an .
Howa d men ions how machines lea n by example and can only do ha i
da a is labelled, and like Kenne h, he e e ences ha an AI needs o know
wha a ca is and wha a dog is. Simila ly, Nicole used ca s and dogs o
explain why da a has o be labelled: i has o be de ined wha dogs look like
and wha ca s look like, and images ha e o be labelled as such o allow a
compu e ision sys em o make accu a e p edic ions i an image con ains
IN/VISIBILITY BY DESIGN
112
on o explain ha we hink o da a se s as ‘objec i e sou ces o u h’, which
she inds conce ning. She explains ha ca ego isa ions and classi ica ions
ha e been cus om- made o a con ex and a e hus no uni e sally applic-
able. Selena s esses ha he con ex in which decisions on da a se s a e
being made is impo an , and she sugges s ha he e should be a eco d
o hese decisions. Selena goes u he by s a ing ha we also need o ask
who is bene i ing om his wo k. She a gues ha powe dynamics ha a e
embedded in da a labelling need o be conside ed alongside he limi a ions
o such da a p ac ices.
Da yl ollows a simila line o hough when a icula ing how a machine
lea ning p ac i ione migh go abou doing an image classi ica ion ask.
This p ocess s a s by concep ualising and aming wha he ask is and
wha labels migh be used in he sys em, which is simila o wha was
discussed in he p e ious sec ion. Howe e , Da yl s esses ha his in ol es
deciding on a ca ego ical schema based on which he millions o images
ha ha e been collec ed can be o ganised. Da yl sugges s ha he e a e a
on o design decisions ha go in o ha , down o which wo ds one uses
o desc ibe he wo ld and in which language ha is going o be. You migh
decide on English, which is hen a Global No h bias. Then you decide
o pick a housand wo ds, bu ca ing up he wo ld in o nea ca ego ies
wi h limi ed wo ds is no easy. Da yl s a es ha pe spec i es shape his
igh down o which images show up in he da a se in he i s place. Fo
ins ance, one migh use a web sea ch o he di e en ca ego ies such as
doc o . Then he e migh be a human- in- he- loop, a da a labelle , who says
i his image shows a doc o . This collec ion o da a is no pe spec i eless,
as Da yl s a es, because a pe son migh ha e a speci ic concep ualisa ion o
how a doc o , a nu se o , o he sake o he a gumen , a baske ball, looks.
The pe spec i e ha is aken, Da yl explains, is o en a whi e male, Wes e n
pe spec i e o he wo ld.
Da yl goes on o s ess ha he choice o ca ego ies has a p o ound
impac on machine lea ning and wha ca ego ies machine lea ning is p o-
ducing. These ca ego ies ha e o be linked o he ‘signals’ in he image. Fo
example, i he da a se only con ains whi e, male doc o s bu no one in a
su geon’s uni o m o sc ubs because ha has no been labelled in he da a,
his has implica ions o wha he sys em can ‘see’. These da a se s a e, as
Da yl s esses, no only used o ain he AI sys em bu also o assess i s pe -
o mance in he eal wo ld. This leads o a ci cula logic, as Da yl explains: i
IN/VISIBILITY BY DESIGN 113
one uses a speci ic concep ualisa ion o wha a doc o is and wha a nu se
is, his concep ualisa ion is used o measu e how well he sys em wo ks.
Addi ionally, he e is a speci ic concep ualisa ion o an image classi ica ion
embedded in he da a se ; wha is con ained in an a i icially ixed ca ego y
is only one in e p e a ion o he image. Da yl s a es ha human ision is
con ex ual in ha how humans unde s and and desc ibe he wo ld depends
on social iden i ies and cul u al con ex s. Quenna aised a simila conce n
when she alked abou how meaning is con ex dependen and will a y
globally. In o he wo ds, no e e yone is eading an image in he same way,
bu Da yl says ha o machine lea ning, he human ways o in e p e ing
he isual wo ld a e bounded and limi ed in ega d o da a se s. Da yl s a es
ha his has b oad implica ions because he assump ion is ha compu e s
can see and e eal he u h abou an image. Wha Da yl is sugges ing
is ha only speci ic ways o seeing a e embedded in classi ica ions and
ca ego isa ions h ough which AI sys ems see he wo ld.
Da yl expands on his poin by alking abou epis emology, which is
unde lying he cons uc ion o da a se s. Da yl s a es ha he unde lying
epis emology is ha da a labelling is abou ecognising a sel - e iden u h
in an image. This esona es wi h how Callum desc ibed g ound u h: an
‘a omic bi o u h om he eal wo ld’. The assump ion he e is ha he
label assigned is a ue ep esen a ion o how he wo ld is. Raymond, in
con as , alks abou he g ound u h o a da a se as a be e exp ession
because wha is desc ibed is wha is co ec wi hin he da a se . He s esses
ha his is no a gene al o gene ic u h o wha migh be ue o one
pe son, bu a he a ela ionship wi hin da a.
Da yl, howe e , s a es ha how his u h is es ablished d aws on
p ocesses, which a e deemed o iden i y he ob ious and sel - e iden .
C owd wo ke s a e expec ed o label images, and his p ocess en ails iden i-
ying some hing ha is clea and ob ious in he wo ld and ha c owd wo k
is an accep able way o sol ing ha ask. This has consequences, acco ding
o Da yl, o how his wo k is done, and ha a aceless and nameless c ew
o wo ke s who label images wi h a e age sco es is an accep able way o
ge ing o his sel - e iden u h.
Da yl acknowledges ha he e a e con ex ual di e ences in how people
label. Da yl s a es ha he acelessness and namelessness o his p ocess
also con ibu es o he imp ession o he inal da a se s being uni e sal.
Howe e , Da yl ques ions i his uni e sali y is eally ue because i migh
IN/VISIBILITY BY DESIGN
114
be based on a single label a ached o a single da a ins ance. Howe e , whe e
his label came om and who ended up doing his wo k is los . Da yl
explains ha his claim o uni e sali y is e lec ed in ha he people doing
he labelling a e seen as no ma e ing. Those people a e no seen, he e a e
nameless, and whe e hose people come om does no ma e . I is accep ed
ha a human has o do his wo k, bu Da yl s a es ha hese in as uc u es
used o comple e his wo k make wo ke s in isible. To sum up hus a ,
Da yl connec s he claim o uni e sali y o da a ha is made di ec ly o he
in isibili y o wo ke s. Only i he wo ke s and he wo k a e made in is-
ible is i possible o claim ha da a labelling c ea es uni e sal u hs abou
he wo ld.
Da yl hen goes on o a icula e how people ha e s a ed o hink abou
how di e en anno a o s b ing di e en pe spec i es o bea , which needs
o be cap u ed. This can be a ia ion in da a labelling, which is a sign ha
people do a ask di e en ly. Howe e , ha is no he no m in da a anno a ion
because mos da a- labelling p ojec s ea labels as sel - e iden , wi hou he
need o in e p e a ion. Such a ia ions in seeing need o be made in isible
o ensu e ha da a labels a e e icien , scalable and cos - e ec i e. In o de
o a oid ha , Da yl sugges s ha i is necessa y o ecognise ha ways o
seeing he wo ld di e among people.
The cons uc ion o ways o seeing and desc ibing he wo ld ha bo h
Selena and Da yl alk abou is meaning ul, no only o he wo king
condi ions in which such wo k is done, bu i also igno es he ac ha how
people pe cei e and desc ibe he wo ld a ies. While A a ecognised his
poin , Selena and Da yl expand on his and Da yl links i o epis emology.
Ways o knowing di e and o en, he pe spec i es ha claim uni e sali y
a e in ac no hing bu a god ick, as Ha away (1991) migh say. The e
a e speci ic pe spec i es embedded in how classi ica ions a e designed,
how labels a e desc ibed and how labels a e being applied. Howe e , mos
da a se s seem o p e end ha hey o e a iew om nowhe e o claim
ha he in o ma ion hey en ail is uni e sally ue. This in u n ende s he
mechanisms o p oduc ion o hese da a se s in isible.
I a pe spec i e is aken o ep esen he uni e sal u h, his is possibly
he pe spec i e o designe s o AI who de elop he classi ica ions and w i e
he labels desc ip ions. The da a anno a o s ollow hose ins uc ions and, i
hey do no apply he labels co ec ly, a e old how o label he wo ld in ways
ha is desc ibed in he labels. Howe e , da a anno a ion companies ha e a
IN/VISIBILITY BY DESIGN 115
c ucial media ing unc ion he e. This is o en conside ed in ega d o wha
wo king condi ions hey o e . Howe e , he da a anno a ion companies
also nego ia e meaning be ween he da a labelle s and he AI designe s in
he clien companies. This impo an media ing unc ion o how know-
ledge is c ea ed emains o en simila ly unacknowledged. In o he wo ds,
da a- labelling companies a e he o ganisa ional link in he epis emological
chain – hey media e wha knowing abou he wo ld is embedded in da a
labelling.
Cons uc ing Gende
The ac ha gende is commonly labelled as a bina y was egula ly
discussed by hose who we e in ol ed in and amilia wi h da a- labelling
p ocesses. A common conce n aised by, o ins ance, Pa ke , Da yl and
A a is ha gende in AI is ypically bina y. A a asks wha ha would mean
o a pe son who does no iden i y as one o hose bina y gende s. A a said
ha decisions a e being made based on he ca ego ies ha a e included in
an AI sys em. She hus alludes o he ac ha hose who do no iden i y
based on he wo op ions o gende o e ed, migh no be included and
hus become in isible. Selena simila ly poin s o he p oblem o ei ying
gende h ough ca ego ising people along a gende bina y while also
e asing ans and quee iden i ies. Addi ionally, Selena is conce ned ha
in e sec ional expe iences o gende a e made in isible. She explains ha
he expe iences o a whi e woman a e di e en om a Black woman and
jus lumping women in o a ca ego y o women is making his di e ence
in isible.
B enda men ioned ha mo e clien s a e conce ned abou bias in he
da a bu ha mos o he da a s ill ollows a bina y app oach in ega d o
gende . Da yl also spoke abou he assump ion ha gende is commonly
concep ualised as a bina y and ha gende is ea ed as knowable om an
image. Da yl explains ha gende appea s as a ixed and a na u al ca ego y
in machine lea ning, e en hough i is cons uc ed, si ua ed and shi ing.
This is p oblema ic o he de elopmen o compu e ision sys ems, as
Da yl s a es. Howe e , like B enda, Da yl has obse ed a shi in ecen
yea s whe e i has been ecognised ha gende is no bina y and ha i is
no possible o know someone’s gende by looking a an image. Ins ead,
i da a se s a e labelled wi h gende , he da a is labelled wi h ‘pe cei ed
IN/VISIBILITY BY DESIGN
116
gende ’. Da yl asse s ha his is a s ep in he igh di ec ion because mos
da a se s ha e an as e isk nex o gende wi h he s a emen ha gende is
no bina y. Da yl says ha his acknowledges he idea ha gende is no a
bina y bu she complains ha he same da a se s hen go on o use gende
as a bina y. The same phenomenon o s a ing ha gende is non- bina y o
p oceed wi h gende concep ualised as a bina y was obse ed by Pa ke .
Such a discu si e mo e shows ha he e is an awa eness o gende as a
non- bina y, which, howe e , does no lead o any changes in p ac ices o
how gende is concep ualised.
Da yl s a es ha e en i da a se s a e labelled wi h pe cei ed gende ,
hese pe cep ions a e s ill cul u ally and socially si ua ed pe cep ions o
masculine and eminine p esen a ions. Da yl s esses ha such pe cep ions
shi geog aphically be ween cul u es bu also in ega d o age and ace.
In Da yl’s iew, classi ying people in o gende ca ego ies is p oblema ic
because hose ca ego isa ions a e o en acialised and could be seen as an
exp ession o wha Da yl calls a colonial p ojec . Da yl says ha who
de ines hose ca ego ies o pe cei ed male and pe cei ed emale is cen al
because hese a e no objec i e o sel - e iden .
This aises wide ques ions o Da yl in ega d o wha needs o be
measu ed a all. So, o ins ance, i you need a sys em ha wo ks o di e en
gende ca ego ies, i migh be bes o go wi h sel - iden i ica ion o indi-
iduals. I one needs o know how a sys em pe o ms o people wi h sho
o long hai , acial hai o no , hen his could be used o analysis. I you
need a gende label and canno ely on sel - iden i ica ion, Da yl sugges s
ha aming hose labels as no sel - e iden is cen al and ha going wi h
he pe cep ion o labelle s migh be possible i his would be amed as a
pe cep ion a he han a ac .
Da yl goes on o explain ha he labelle s could gi e some e idence
why hey come o a judgmen , such as wha a pe son is wea ing, he pe -
cep ion o seconda y sex cha ac e is ics, g ooming s yles o p esen a ion.
Fo Da yl, he a icula ion o how one a i es a a judgemen is key and
would allow con ex ualising he esul ing labels. I has o be clea ha hese
a e judgemen s, no an objec i e measu e. Acco ding o Da yl, pa o he
p oblem is how ques ions o da a labelle s a e o mula ed. Fu he mo e,
as Da yl s a es, one has o collec in o ma ion abou he da a labelle s
hemsel es because people who ha e di e en ela ionships wi h gende ,
such as being quee , ans o gende - di e se olk,4 migh come o di e en
IN/VISIBILITY BY DESIGN 117
judgemen s in ega d o gende labels han a cisgende pe son who has
ne e hough abou he socially cons uc ed na u e o gende . A simila
poin was aised by Hayden. Addi ionally, people om di e en cul u es
migh unde s and gende ca ego ies di e en ly, as Da yl poin s ou . Da yl
s a es ha i is he e o e impo an wha in o ms hei eading o gende
and why hey ead gende in ce ain ways. Such addi ional in o ma ion
abou da a labelle s alongside aming ques ions in a p ecise ashion will
allow o anno a ions ha a e con ex ualised.
Ano he eason why labelling gende migh be impo an is p o ided by
Callum. Callum alked abou he isk o ca ego ies being in e ed, which he
cons uc s as mo e p oblema ic han ha ing an explici label. He p o ides
he example o someone who migh ha e a LGBT5 ini ia i e on hei CV.
Then, acco ding o Callum, i is no clea i you iden i y as LGBT o i
you a e an ally. Howe e , o he model, ha does no ma e because i
migh s ill in e om he da a ha you a e a less good candida e. Callum
implies ha i LGBT s a us would be explici ly labelled, he e migh be
ways o mi iga e o bias, which is mo e di icul i he e is no label bu
he in o ma ion is in e ed. Simila ly, Sabine s a ed ha e en when gende
is no explici ly s a ed, he e will be a ple ho a o p oxies o gende ha
a e in e ed om da a. This conce n goes back o some issues ha we e
discussed in Chap e 4.
B enda p o ides a simila example when e e ing o an academic pape .
The pape used language da a om T us pilo o analyse gende and wo d
choice. B enda is scep ical abou he me hodology used in he pape : no
only was gende ega ded as bina y bu names we e used o deduc i a
pe son is a woman o a man. The pape ound ha women and men use
di e en wo ds, wi h women being mo e desc ip i e, such as using an-
as ic, wonde ul, awesome, happy, and men using wo ds ocusing on p ice
and quali y, like inexpensi e, economic, cheap, bes quali y and so on. When
I asked wha his migh mean, ha an AI could conclude ha by using such
wo ds you a e a woman, B enda p o ides a use case whe e a pe son w i es
a e iew abou a p oduc , he language is analysed, he pe son is classi ied
as woman o man, acco ding o he language used, and hen he pe son is
shown ad e isemen s a ge ed a women. B enda la e expands on ha by
saying ha language is pa o gende socialisa ion and non- bina y indi-
iduals migh also use language in di e en ways and migh hus no be
a ge ed co ec ly by hose ads.
IN/VISIBILITY BY DESIGN
118
Gende was also discussed in ela ion o language ansla ions. B enda
alked abou an example om Google T ansla e, whe e Hunga ian is
ansla ed in o English. B enda sha es an example wi h me whe e in
Hunga ian, he p onouns a e gende neu al, bu he English ansla ion
ans o ms his gende neu ali y in o some hing ha is s e eo ypically gen-
de ed such as ‘she is beau i ul’ and ‘he is cle e ’. Wha is in e es ing is ha
he o iginal language was no gende ed bu gende ing is in oduced when
he ex is ansla ed. He e, he machine ansla ion is doing he gende ing.
B enda uses a speci ic example om de eloping a cha bo whe e he
cha bo au oma ically was e e enced as a he and he in e io designe was
e e enced as a she.
Ano he issue in ela ion o gende is co- e e ence agging. Co- e e ence
agging means, as B enda explains, o ag he e e ence be ween names and
p onouns. She uses he example, ‘Alex wen o he conce ; he said i was
amazing’, which means ha ‘Alex’ and ‘he’ a e co- e e en and ‘conce ’ and
‘i ’ a e co- e e en . Bu wha happens i Alex is a woman? Then he sys em
needs o be able o unde s and ha Alex can be a ‘she’. O i Alex is non-
bina y hen he sys em should say ‘ hey’. B enda says ha i a sys em is no
ained o ha e his lexibili y, i is likely o exclude and he sys em is biased.
Howe e , B enda acknowledges ha she has ne e seen such a p ojec ha
was designed wi h inclusion in mind.
Speech ecogni ion migh also pose speci ic challenges om a di e -
si y pe spec i e. A a alked abou how in speech da a collec ion, one migh
a emp o ind examples o egional dialec s and hen ha e men and
women speak in ha dialec , bu he mo e g anula his in e sec ionali y
becomes, he less examples he e will be, which is also some hing Geo gia
men ioned. This a ec s he aining da a, which A a says will be less obus .
Speech ecogni ion also has a no ma i e aspec o i , as B enda elabo a es.
She s a es ha au oma ic speech ecogni ion oices like Si i and Alexa
speak s anda d oices, emula ing wha is called s able linguis ic pe iods
o people. This pe iod is de ined as people be ween he ages o 20 and 55.
Da a se s migh con ain 10%– 15% o people o e 65, which migh mean
ha people o e 65 a e less well unde s ood. Equally younge people, who
migh be mo e inno a i e wi h language o who migh o migh no go
h ough pube y, migh also no be unde s ood. The speech da a will also
be collec ed om men and women, and as such, ans pe sons migh be
less well unde s ood. Simila ly, wha B enda desc ibes as ‘s e eo ypical gay
IN/VISIBILITY BY DESIGN 119
male speech’ is some hing ha an au oma ion speech ecogni ion engine
needs o be ained on.
Finally, ano he issue ela ing o gende ha was men ioned was ha
Si i and Alexa and o he VPAs had de aul eminine- sounding oices (see
also Chap e 1). This is a opic o egula academic, policy and media con-
ce n (Equals & UNESCO, 2019; Dillon, 2020; S enge s & Kennedy, 2020;
Su ko, 2020), which has led p o ide s o o e mo e di e si y in VPA oices
(Ba aniuk, 2022). As such, i is no su p ising ha he opic was men ioned
in he in e iews as well. A a sugges ed ha he de aul eminine oices
e lec gende s e eo ypes ha designe s had, bu she also alked abou how
he e a e now a emp s o de elop non- bina y oices, o ins ance, in P ojec
Q – an a emp o c ea e a gende less oice (P ojec Q, 2023).6 While A a
hough ha his is an in e es ing de elopmen , she wonde ed in how a
his will emain niche and he s anda d is going o be eminine oices o
assis an s. Simila ly, La a alked abou how many o he clien s gi e i ual
assis an s o cha bo s a gende ed and o en eminine name. She desc ibes
how she is pushing back agains clien s who pick gende ed names and she
p oudly s a es ha mos o he i ual assis an o cha bo s she wo ked on
did no end up wi h gende ed names.
O e all, many o he in e iewees a icula e how gende is ele an
o da a labelling and, by ex ension, machine lea ning. The in e iewees
showed an awa eness o he ac ha cu en p ac ices a ound gende and
da a o en mean ha bina y gende is ei ied. While many desc ibed hese
p ocesses as p oblema ic, hey also acknowledged how di icul i is o
change hose p ac ices owa ds mo e inclusion. Mo eo e , i is e iden ha
p ac ices a ound da a labelling a e cons uc ing gende . The gende pa e ns
used in machine lea ning shape which gende pa e ns a e p edic ed.
Conclusion
The chap e ocused on pa e ns ha humans ha e o ecognise o help AI
lea n. This chap e s a ed wi h he ques ion o how machines ecognise
gende pa e ns, o in o he wo ds, how a machine knows who is a woman,
a man o non- bina y. The chap e sugges ed ha machines pe cei e he
wo ld h ough labels ha a e assigned by humans o de eloped du ing
machine lea ning. I humans add hese labels, his o en happens h ough
AI’s hidden wo k o ce – hose who label da a as a c owd wo k ask o by
IN/VISIBILITY BY DESIGN
120
indi iduals adding labels wo king in he AI supply chain. This wo k is o en
done in he Global Sou h. While much esea ch has igh ly ocused on he
wo king condi ions o hese wo ke s, his chap e has pa icula ly s essed
ha such wo ke s in e p e he wo ld bu ha hese in e p e a ions a e
made in isible. The chap e explains ha da a labelling is necessa y o allow
machine lea ning sys ems o ecognise pa e ns, which hen o m pa
o he ou pu s he machine p oduces, o in o he wo ds, he p edic ions.
The e o e, da a labelling is needed o AI o help machines see, hea and
unde s and he wo ld. Fo his o happen, a g ound u h – labelled da a
ha is assumed o be ue in machine lea ning – needs o be es ablished o
allow machine lea ning and o check he quali y o machine lea ning.
Labels a e classi ica ions and he chap e discusses how classi ica ions a e
ways in which he wo ld is o de ed. I was shown ha once his o ganisa-
ion o knowledge has happened, he classi ica ions o en become accep ed
o how he wo ld is. As such, hese ways o o ganising he wo ld h ough
classi ica ions such as in machine lea ning labels a e poli ical bu he
p ocesses o cons uc ion a e made in isible. In da a labelling, i is seen as
impo an o c ea e a consensus among human labelle s and mode a o s
abou which labels o apply o bes ep esen he wo ld. These p ocesses en ail
u ning subjec i e decisions in o a seemingly objec i e and uni e sal u h.
Howe e , he chap e has shown ha his uni e sali y is ca e ully nego ia ed
be ween di e en ac o s in da a labelling who embed wha is knowable
abou he wo ld in labelled da a se s. Wha is knowable abou gende in da a
labelling gene ally seems o ollow an unde s anding o gende as a bina y,
wi h limi ed scope o concei e gende beyond a bina y. The wo ld ha is
being cons uc ed h ough machine lea ning is in many ways a simpli ied
unde s anding o he wo ld, which is p esen ed as objec i e and uni e sal.
The eby, classi ica ions ha a e concep ualised and ope a ionalised h ough
da a labels cons uc a eali y. Ye , his cons uc ion o eali y is a po en ially
exclusiona y one. Building mo e inclusiona y app oaches in ega d o da a
labelling is cen al o make hese cons uc ion p ocesses isible and an-
gible. As such, he chap e has a gued ha he e a e some pa e ns ha only
humans can ecognise bu ha he subjec i e p ocesses based on which
his ecogni ion happens a e egula ly made in isible o sugges ha hese
pa e ns a e objec i e and uni e sal.
IN/VISIBILITY BY DESIGN 121
No es
1 Da a labelling and da a anno a ion a e used in e changeably in his chap e .
2 I should be no ed ha academic esea ch o en elies on MTu k as well (Aguinis
e al., 2021). Fo example, academic esea ch egula ly d aws on MTu k o ind
pa icipan s who can comple e su eys.
3 Sama says i o e s p emium pay and psychological suppo o such ype o wo k
(Pe igo, 2023). Ye , i has been claimed ha such psychological suppo is di icul
o access (Pe igo, 2023).
4 This is he e minology Da yl used.
5 LGBT is he e m Callum used.
6 P ojec Q aims o c ea e a gende less oice and should no be con used wi h OpenAI’s
Q* p ojec (Lee, 2023).
CONCLUSION: UNWRITTEN RULES
128
F om Magic o Making Rules Visible
Fo many people, e ms like AI, algo i hms o digi alisa ion appea my h-
ical and magical because he wo kings o hose echnologies a e hidden
om sigh (Finn, 2017) (see also Chap e 1). Simila ly, he e m ‘black box’
desc ibes he opaqueness o echnologies whe e e en hose who design
hese new echnologies can o en no ully explain why hey wo k in speci ic
ways (Pasquale, 2015). In Chap e 5, I ha e a gued ha in isibili ies a ound
how da a is p epa ed o AI a e cen al o make AI appea as objec i e and
uni e sal. Howe e , hese in isibili ies o such p ocesses a e also impo an
o see AI as magical and my hical. This magical and my hical na u e o ech-
nologies in i es us o engage wi h hese echnologies in i ualis ic ways ha
can unc ion o mi iga e he isks and unce ain ies o mode n li e (Finn,
2017). While hose echnologies migh p o ide a kind o i ualis ic com-
o , many o hose echnologies ha e consequences o people’s li es and
as such, he e is an u gen need o unde s and and explain how hese ech-
nologies wo k. In a sense, i is impo an o make echnologies less magical
and my hical o s a engaging wi h hem in a mo e enligh ened way.
Th oughou he book, I ha e sugges ed ha echnology is shaped in
design and use by socie y and ice e sa. This pe spec i e eme ges om he
social shaping o echnology app oach (MacKenzie & Wajcman, 1999)
(see Chap e 1). Resea ch in his ein would no mally show how speci ic
echnologies a e shaped by socie y, such as how he elec ic e sion o he
e ige a o became he no m (Cowan, 1999). Ano he example comes
om he gende ed meanings associa ed wi h mic owa e o ens. When he
mic owa e was i s in oduced in o homes, i was imagined and ma ke ed
as a way o men o ehea ood bu i s usage o en led o unexpec ed esul s
such as women cus omising indi idual meals o amily membe s wi h
he aid o a mic owa e (Cockbu n & O m od, 1993). This classical s udy
abou gende and echnology highligh s he luc ua ing and in e ela ed
ways in which meaning a ound new echnologies in e sec s wi h gende
(Cockbu n & O m od, 1993). Such s udies abo e all show how social
ela ions en e echnology; social no ms en e he design and use o ech-
nologies and hese echnologies shape social no ms in u n. Technologies
such as AI lea n social no ms h ough da a bu also associa ed p ocesses and
decisions (see Chap e s 4 and 5). As such, hese echnologies a e lea ning
he unw i en ules o socie y. Bu hese echnologies will also be able
CONCLUSION: UNWRITTEN RULES 129
o pick up changes and inconsis encies in ela ion o such pa e ns. Fo
ins ance, he esea ch on mic owa e o ens shows how gende ma e ialises
in echnology in expec ed bu also unexpec ed ways, leading o pa e ns
ha show con inui y bu also change (Cockbu n & O m od, 1993). So a ,
esea ch has la gely ocused on he pa e ns ha epea exclusion, such as by
c ea ing algo i hmic biases and as such, au oma ing inequali ies (Eubanks,
V., 2018). Howe e , one can expec ha he e is possibly mo e a ie y in
he pa e ns o inclusion and exclusion ha eme ge in ela ion o gende
and digi alisa ion.
Al hough AI and ela ed echnologies a e igh ly seen as c ea ing isks
o exclusion (Eubanks, V., 2018; Benjamin, 2019), I ha e discussed aces
o how AI- suppo ed hi ing can be unde s ood as making he unde lying
ules o socie y isible and hus angible (see Chap e 4). In Chap e 4,
I showed how hose who design and use hi ing echnologies cons uc
algo i hmic bias as ul ima ely ixable. I ha e c i iqued his pe spec i e as an
exp ession o echno- op imism ha concei es socie y as some hing ha is
ixable h ough echnological means. Such a pe spec i e pe mea es he ech
indus y. This echno- op imism sugges s ha he black box o echnology
can be opened and co ec i e measu es can be aken, leading o explain-
able AI. I ques ioned i such biases a e ixable because hey a e a unc ion
o socie y. Howe e , while he idea o a echnological ix o socie al issues
should be ques ioned, echnologies such as AI a o d us wi h he possibili y
o showing how he unw i en ules in socie y unc ion o sys ema ically
exclude g oups o people. AI as a pa e n eade and epea e is cen al o
unde s anding he unw i en ules o how socie ies ope a e. In o he wo ds,
AI epea s and ampli ies human biases bu i also makes inequali ies ha
exis in socie y isible. Howe e , he e is no hing me e abou his because
c ys allising he unw i en ules o socie y is impo an and meaning ul.
Making he unw i en ules o socie y isible is meaning ul, no in he
sense ha i allows us o apply a echnical ix, as many o he in e iewees
sugges ed. Ins ead, i migh allow us o change he unw i en ules o socie y
in gene al and a scale. Fo example, p e ious in e en ions in ega d o dis-
c imina ion in he wo kplace ha e ocused on ‘ ixing’ indi idual decision-
make s. I a whi e male hi ing manage ends up hi ing a younge e sion
o himsel , we ied o change he p ac ices o he indi idual manage by
calling ou his pa e n.2 Hypo he ically, AI- suppo ed hi ing echnolo-
gies migh now show us ha whi e, young men migh be gene ally he
CONCLUSION: UNWRITTEN RULES
130
p e e ed candida e o be hi ed. Fo o he posi ions such as in ca e wo k,
he implici assump ion migh be ha such wo k is done by women om
he Global Sou h. This is o cou se some hing ha esea che s ha e shown
o a long ime (Acke , 1990). In ac , we ha e seen in Chap e 2 ha many
discou ses on he u u e o wo k con ain a conce n o a speci ic ype o
pe son: male, whi e- colla p o essionals. This was he implici ideal wo ke
imagined in he books on he u u e o wo k. AI has he po en ial o make
such unw i en ules isible and angible in a sys ema ic way. Al hough
I would cau ion agains a emp ing o ix socie y by ixing echnology,
being able o make he unw i en ules angible migh allow o changing
hem as socie ies.
Such a pe spec i e also opens he possibili y o seeing such pa e ns
as mo e complex and dynamic. The e migh be dominan and less dom-
inan pa e ns bu he e migh also be pa e ns ha con adic each o he .
This complexi y and dynamic na u e o pa e n is cen al o how socie ies
unc ion, and s essing such con adic ions and complica ions in pa e ns
would be a no el way o hink abou digi alisa ion and gende . The e a e
always mul iple pa e ns compe ing o a en ion. Equally, o new pa e ns
o o m, a new componen needs o be in oduced o emphasised. We ha e
seen a momen o such a ecogni ion in ela ion o he wild ca d hi es in
Chap e 4. I we hi e people who migh no i he pa e n in he mos
pe ec way, we open up he oppo uni y o new pa e ns o eme ge. As
such, pa e ns should no be seen as de e minis . Such pe spec i es dom-
ina e hinking on algo i hmic bias whe e he isk o epea ing pas pa e ns
is leading o exclusion. While such isks ha e o be aken se iously, i is also
impo an o emembe ha new pa e ns can be c ea ed, which migh lead
o g ea e inclusion.
As a ma e o ac , he common ix o algo i hmic bias, ixing he da a, is
in a sense in oducing a new pa e n. Since da a is o en cons uc ed as he
basis o algo i hmic bias, as we ha e seen in Chap e 4, many app oaches
o deal wi h algo i hmic bias a e cen ed on ixing he da a. This migh
include ensu ing ha da a se s ep esen socie y. I AI ails o ecognise Black
aces, he solu ion is o include mo e Black aces om which AI can lea n
(Buolamwini & Geb u, 2018). I AI sugges ed only male candida es because
he unde lying da a se included la gely men’s CVs, hen he solu ion is o
include CVs by women (Das in, 2018). Chap e 4 de ails many o hose
‘ ixes’ o ensu e ha algo i hmic bias is educed. We also need o look a
CONCLUSION: UNWRITTEN RULES 131
how da a is being p oduced and p ocessed, which was a he cen e o
Chap e 5. I has been sugges ed ha ways o wa d a e da ashee s o da a
se s (Geb u e al., 2021). Da ashee s should include, o ins ance, in o ma-
ion abou he mo i a ion o collec his da a, he composi ion o he da a
se , he collec ion p ocess o he da a, he p e- p ocessing/ cleaning/ label-
ling o he da a se , uses, dis ibu ion and main enance (Geb u e al., 2021).
By p o iding de ailed ques ions ha should be conside ed in da ashee s
o da a se s, Geb u and co- au ho s (2021) p o ide admi able guidance o
imp o e da a se s.
While hese app oaches o imp o ed da a a e necessa y, i is ques ion-
able i hey a e su icien . Da a will be imp o ed by being mo e ep esen-
a i e, mo e e hical and mo e anspa en . Howe e , he unde lying issue
ha da a is e lec ing socie y will emain. Ou s anding da a p ac ices migh
help o educe algo i hmic bias, bu he social pa e ning o da a migh
come h ough in ano he way. Fo example, in AI- suppo ed hi ing, a p o-
ide migh d op acial ecogni ion o a oid ha he selec ion o sugges ed
candida es is no in luenced by he echnology being less able o ecog-
nise Black women. Ye , he same Black women migh use ce ain lan-
guage cons uc ions, which migh be judged as less sui able o a ole and
hus il e ed ou . O cou se, good da a p ac ices could educe hese isks,
bu he e is a dange ha algo i hmic biases eme ge in o he shapes and
o ms because da a is inhe en ly social. Socie al ela ions imp in on da a.
Howe e , as I ha e a gued, i is also possible o use his as an oppo uni y
o c ea e al e na i e pa e ns.
Rei ica ion Machines
Rei ica ion is a cha ge egula ly le ied agains esea ch on gende . Fo
ins ance, esea ch migh s a e ha gende is seen as socially cons uc ed,
ye hen p oceeds in he empi ical pa o ope a e based on a ixed gende
bina y (Nen wich & Kelan, 2014). Fo much esea ch on gende , a s anda d
c i icism is ha such esea ch is e- es ablishing gende bina ies a he han
challenging hem. Seeing gende as luid and lexible o coun e ac he
concep ion o gende as a ixed bina y has been discussed in academia, a
leas since Bu le ’s seminal book, Gende T ouble (Bu le , 1990). The idea o
gende beyond a bina y has ecei ed pu chase in wide socie y.3 While such
concep ions o gende a e o en de ided as being pa o ‘gende ideology’
CONCLUSION: UNWRITTEN RULES
132
(Kuha & Pa e no e, 2017), i should be no ed ha , o example, Ge many
legally ecognises a hi d gende and hus mo es beyond a gende bina y
(An i- Disk iminie ungss elle des Bundes, 2023). One could hus a gue ha
mo ing beyond a gende bina y, which has been cen al o gende s udies
o a while, is inc easingly some hing ha is ecognised in o ganisa ions
and wide socie y.
While mo ing beyond gende bina ies has eached he mains eam, he e
is a s ong endency in AI o e- es ablish he gende bina y. Fo example,
in HR, a emp s ha e been made o o e mo e han wo op ions o signi y
gende , ye in AI- suppo ed hi ing, gende is la gely ea ed as a bina y (see
Chap e 4). As I ha e shown in Chap e 5, he gende bina y as unde lying
classi ica ion is a ely ques ioned in machine lea ning. Simila o esea ch
in gende s udies, esea ch in machine lea ning o en s a es ha gende is
no a bina y ye p oceeds o ea gende as an unchangeable bina y (Keyes,
2018) (see also Chap e 5). Machines ead gende h ough he da a in a
a ie y o ways. Gende migh be sel - iden i ied o someone else is picking
a gende label. Fu he mo e, gende is embedded in da a h ough, o
ins ance, ce ain wo ds ha people commonly ead as wo ds women use.
E en hough popula pe cep ions o gende a e mo ing beyond a gende
bina y, AI is o en ei ying gende as a bina y.
I AI is a gende ei ica ion machine, his a o ds he abili y o s udy
he unw i en ules h ough which gende is es ablished. App oaches ha
see gende as a doing (Wes & Zimme man, 1987) o pe o med (Bu le ,
1990) o en seek o unde s and such unw i en ules o how gende is
es ablished in in e ac ions. Fo example, Go man’s (1979) wo k on ad e -
ising showed he i ualisa ion o gende by showing how ela i e size is
a ma ke o gende . Ga inkel’s (1984) wo k engaged wi h iden i ying
ma ke s o eminini y. I AI is making ules o socie al in e ac ions isible,
hen AI is an oppo uni y o unde s and gende in socie y.
The eme ging wo lds o AI ei y gende as a bina y o , in o he wo ds,
AI c ea es wo lds ha a e by and la ge based on gende bina ies. I gende
is ea ed as a bina y in daily li e and da a e lec s his, machines will lea n
gende bina ies by de aul . I has been igh ly poin ed ou ha his is p ob-
lema ic i i leads o indi iduals being misgende ed (Keyes, 2018). I also
es ic s which u u es can be de eloped. I we ely on AI eplica ing a
ce ain e sion o socie y, his limi s which u u es can be c ea ed. In his
case, u u es whe e gende mo ed beyond a bina y a e less likely. As such,
CONCLUSION: UNWRITTEN RULES 133
ei ying gende in and h ough AI means ha wha u u es a e possible is
cu ailed.
Poli ics o Visibili ies
An unde lying conce n o his book we e p ocesses o how gende is made
isible and in isible in discou ses on he u u e o wo k (Chap e s 2 and
3) bu also in ela ion o how AI cons uc s a speci ic eali y (Chap e 5). As
discussed in Chap e 2, in books on he u u e o wo k, he main conce n
was o male, middle- class b eadwinne s whose jobs migh be au oma ed.
The common conce n was ha i machines ake o e jobs, people will no
be able o ea n an income. Howe e , someone migh s ill p o i om he
labou o machines: hose who own he machines. Fo ins ance, hese migh
be ounde s o o sha eholde s in Silicon Valley companies. We ha e also
seen in Chap e 5 ha he wo king p ac ices o da a labelle s and how hey
con ibu e o cons uc ing AI wo lds a e made in isible. One could e en go
as a o alk abou epis emic e asu e (Mahalingam & Sel a aj, 2023) in his
con ex . The concep o epis emic e asu e deno es a p ocess o how he li ed
expe iences o hose om disad an aged backg ounds a e delegi imised
h ough cul u al p ac ices shaped by p i ileged g oups (Mahalingam &
Sel a aj, 2023).4 Mo eo e , mos o hese da a labelle s wo k indi ec ly o
o ganisa ions ha , h ough hei labou , u n a p o i . Howe e , owne ship
s uc u es and who p o i s om unning he machines o om he e o s
o da a labelle s a e hidden om sigh in mos accoun s on he u u e o
wo k. The likelihood is ha hose who p o i om he labou o machines
belong o ai ly small g oups o indi iduals.
Conce ns a ound he u u e o wo k mani es in opes such as he epic
ba le be ween man and machine. Du ing he Fi s Indus ial Re olu ion,
physical powe was eplaced by machines and he assump ion was ha
humans we e supe io in ellec ually (S andage, 2002). Howe e , AI ha
appea s o display human- like in elligence is now a majo unde lying con-
ce n d i ing he hinking on he u u e o wo k. Hence, we see hose who
ha e adi ionally used hei in ellec o ea n a li ing, such as p o essional
wo ke s, being he main ocal poin o conce ns in ega d o he u u e o
jobs. These conce ns a e mi iga ed h ough wo discu si e s a egies: i s ,
o poin owa ds augmen a ion o sugges ha humans and machines
collabo a e and, as such, humans a e equi ed in he u u e; second, o
CONCLUSION: UNWRITTEN RULES
134
sugges ha ce ain abili ies a e beyond he ealm o machines such as
socio- emo ional skills. Ye , as I ha e shown in Chap e 3, socio- emo ional
skills a e wi hin he ealm o machines because hey ollow au oma able
pa e ns. This does no mean ha machines ha e emo ions bu ha hey
can ead and espond o emo ions ha humans display. Like wi h o he
social in e ac ions, machines can disce n he unw i en ules o emo ions
om da a and epea hese pa e ns o, o ins ance, ain ideal emo ional
esponses in humans. I appea s ha like du ing he Indus ial Re olu ion,
when physical powe was eplaced by machines bu in elligence was
pe cei ed as uniquely human, echnologies like AI seem o h ea en human
in elligence, ye imply ha socio- emo ional skills a e uniquely human.
Howe e , his assump ion migh no hold ue i machines also appea
human- like in ega d o emo ions.
The e a e ob ious ensions be ween becoming isible and being no is-
ible. In some ins ances, gaining isibili y is cen al o ha ing one’s iden-
i y ecognised, ye in o he cases, ha ing pe sonal da a e ealed can be
ha m ul. Da a is o en sc aped om he in e ne such as om social media,
which can be used, o ins ance, in hi ing. Sc aping da a om social media
o hi ing is p oblema ic because i iola es he p i acy o candida es and
migh also e eal membe ship a ilia ions, such as in ela ion o age o ace
(Black e al., 2015; Jeske e al., 2019). Simila ly, belonging o an LGBTQ
g oup could lead o indi iduals being anked lowe in hi ing p ocesses
(Tomase e al., 2021). I has also been sugges ed ha some da a migh be
oo sensi i e o include. An example would be how G ind passed use s’
HIV s a us o hi d pa ies; while his in o ma ion is p o ided olun a ily
and wi h consen , i has been sugges ed ha his in o ma ion is oo sensi-
i e o be held by such pla o ms (Rzepka, 2023). In such cases, isibili y
migh be highly p oblema ic because i can be used o exclude indi iduals.
Ano he way o deal wi h he in isibili y o da a is syn he ic da a (Eldan &
Li, 2023; Gunaseka e al., 2023; Jacobsen, 2023). Syn he ic da a p omises
o p o ide unbiased and labelled da a by including a ia ion on, o
ins ance, age, ace and gende (Jacobsen, 2023). Gi en he complexi y o
eal- li e da a o acial ecogni ion, he Mixed Reali y & AI Labs a Mic oso
Camb idge de eloped he model Con igNe o gene a e pho o ealis ic syn-
he ic aces, while allowing a modi ica ion o hese ou pu s, o ins ance, by
adop ing di e en poses o including di e en skin ones (Jacobsen, 2023).
As we ha e seen, o ins ance, in Chap e 4, he unde lying da a is o en
CONCLUSION: UNWRITTEN RULES 135
blamed o algo i hmic bias, and syn he ic da a seems a ac i e because
i e adica es issues a ound ep esen a ion in da a se s; in o he wo ds, i a
g oup is unde ep esen ed, he missing g oup is simply gene a ed and hen
included in machine lea ning. Jacobsen (2023) wa ns o he endency o
solely see algo i hmic bias as a ‘ aining da ase p oblem’ ha can be ixed
using syn he ic da a. Algo i hmic bias can, as we ha e seen in Chap e 4,
also eme ge h ough designing models poo ly (see also Jacobsen, 2023).
Jacobsen (2023) a gues ha syn he ic da a is p esen ed as a way o de-
isk da a whe e syn he ic da a is cons uc ed as isk- ee. Howe e , such
a mo e also means ha wide c i icism in ega d o esis ing and chal-
lenging machine lea ning is silenced (Jacobsen, 2023). Syn he ic da a can
be seen as ano he way o ‘ ix’ da a and hus algo i hm wi hou paying
a en ion o wide social implica ions o echnologies (Jacobsen, 2023). As
I ha e ou lined ea lie in his chap e , ying o exclude he social om da a
is highly p oblema ic. Al hough syn he ic da a p omises o esol e many
issues in ega d o di e si y and inclusion, unless echnology is unde s ood
as social, such ixes will emain pa ial.
Accoun abili y and Responsibili y
The book also o e s pe spec i es on accoun abili y and esponsibili y in
ela ion o digi alisa ion. Th oughou he book, I ha e s essed ha ech-
nologies a e shaped by social ela ions and ice e sa, d awing on he social
shaping o echnology app oach (MacKenzie & Wajcman, 1999). The book
has p o ided coun less examples o how his shaping happens, om how
AI is used in hi ing (Chap e 4) o how da a is labelled o AI o lea n
(Chap e 5). The cen al idea eme ging om ma e ial is ha echnology
does no appea ou o nowhe e. I is c ea ed by people, and how hese
people hink and beha e in luences wha echnology is c ea ed. Which
echnology is c ea ed is also in luenced by hose who inance he de elop-
men o hose echnologies. As I ha e ou lined in Chap e 1, he e a e es ed
in e es s behind os e ing one echnology o e ano he . Such an abili y o
shape echnologies also comes wi h esponsibili y and accoun abili y.
While mos esea ch ends o show us how adi ional social pa e ns a e
epea ed, I sugges ed in his book ha he po en ial o he social shaping
o echnology can be u ilised o c ea e, de elop and os e echnology ha
is c ea ing mo e inclusi e u u es. This is no an au oma ic p ocess bu one
CONCLUSION: UNWRITTEN RULES
136
ha equi es ha ca e is aken o and conside a ion is gi en o how gende
as well as o he o ms o di e si y a ec echnology in design and use and
ice e sa. This is a cons an p ocess because di e si ies hemsel es a e chan-
ging alongside echnologies. I is also no an easy p ocess because i could
be easonably p esumed ha wha is bene icial o one g oup o people
migh no be bene icial o ano he . Such complexi ies need o be e lec ed
on and conside ed. Di e en in e es s and consequences ha e o be ca e-
ully analysed and weigh ed.
This also equi es us o s ep away om he idea ha he e a e simple ixes
ha could be applied o echnology o ensu e ha i is inclusi e. As I ha e
demons a ed in Chap e 4, algo i hmic bias is o en seen as some hing ha
can be ixed h ough echnical means. The e is also a endency o apply a
checkbox men ali y o di e si y o show ha one has conside ed di e si y.
O en, such checks only go as a as legally equi ed. Howe e , ins ead o
seeing his o m o di e si y- p oo ing as a one- o p ocess, how a speci ic
echnology ela es o di e si y will ha e o be ques ioned con inuously. I
is easy o see how his can lead o ‘analysis pa alysis’ due o he shee com-
plexi y ha is en ailed in such a p ocess. This is pa icula ly he case once
in e sec ionali y is aken in o accoun . While his app oach is challenging,
i could be a way o ensu e ha echnologies a e mo e inclusi e in design
and use. We a e hus able o design a u u e ha is po en ially mo e inclu-
si e han he pas .
Embedding esponsibili y, accoun abili y and go e nance in he AI supply
chain has, howe e , p o en o be challenging. I ha e men ioned ea lie ha
i has been sugges ed ha da a se s should come wi h da ashee s (Geb u
e al., 2021). O he s ha e sugges ed ha da a se s should ha e some hing
simila o nu i ional labels (Chmielinski e al., 2022). Such addi ional
de ail would include in o ma ion on how da a was collec ed and p ocessed.
Howe e , a key conce n wi h such app oaches is he ac ha AI supply
chains consis o many ac o s; AI sys ems a e assembled using an a ay o
p e- exis ing so wa e o which a mul i ude o people con ibu e (Widde
& Na us, 2023). Widde and Na us (2023) c i ique exis ing app oaches o
manage esponsibili y and accoun abili y as equi ing ‘panop ical isibili y
in o he echnology’ (Widde & Na us, 2023, p. 8), alongside a con ol o e
his echnology. Widde and Na us (2023) a gue his is a ely he case in
he AI supply chain. Since much o he AI supply chain is based on modu-
la i y, i has been sugges ed ha accoun abili y should be loca ed wi hin
CONCLUSION: UNWRITTEN RULES 137
he indi idual modules (Widde & Na us, 2023). This echoes he eminis
concep o ‘loca ed accoun abili ies’ (Suchman, 2002). Addi ionally, he
in e sec ions be ween modules need o be s eng hened, which can include,
o ins ance, ha cus ome s check ha da a labelle s a e emune a ed
adequa ely (Widde & Na us, 2023). Finally, Widde and Na us (2023)
sugges ha modula i y migh be eplaced comple ely, wi h a new sys em
such as one based on p inciples o design jus ice (Cos anza- Chock, 2020).
Ano he aspec o accoun abili y ela es o he ac ha we o en an h opo-
mo phise echnologies. One meaning o an h opomo phism e e s o he
p ocess o a ibu ing human cha ac e is ics o pe sonali y o a non- human
en i y such as an objec o an animal (Ox o d English Dic iona y, 2023a).
Fo ins ance, in Chap e 1, we saw how Weizenbaum was bewilde ed by he
ac ha people an h opomo phised he cha bo ELIZA (T eusch, 2017). In
he books on he u u e o wo k ha I analysed o his esea ch, a simila
endency o an h opomo phise echnologies could also be obse ed. In
Chap e 2, I discussed how Baldwin (2019) desc ibes how cus ome s
wan ed o da e o buy oses o Ti any, a i ual assis an in a ca deale -
ship in Texas. An h opomo phising is p esuming human- like in elligence in
machines. Gi en he ac ha AI o en is said o emula e human in elligence,
his endency o an h opomo phising is p obably no su p ising. Al hough
academic esea ch is o en c i ical abou an h opomo phising while ying
o a oid i , i opens an a enue o hink abou unde wha condi ions a
machine migh ca y esponsibili y and accoun abili y. We no mally si ua e
accoun abili y and esponsibili y in hose who design new echnologies.
Howe e , he e is possibly scope o e lec on in how a machines could also
ca y esponsibili y o ce ain ou comes. Beyond ha , he e is also scope
o hink abou accoun abili y and esponsibili y as sha ed be ween a ious
ac o s. This would hen call o de eloping a mo e complex unde s anding
o how he social and he echnical a e mu ually cons i u i e and wha his
means o accoun abili y and esponsibili y.
Si ua ing he Resea ch and Fu u e Resea ch A enues
Finally, I wan o e lec on how much he speci ic pe iod o ime du ing
which he esea ch was conduc ed shaped his book and wha u u e esea ch
migh explo e. As I ha e ou lined in Chap e 1, he esea ch was de eloped
p io o he Co id- 19 pandemic and he p ima y ma e ial was colla ed
