Patterns of Inclusion: How Gender Matters for Automation, Artificial Intelligence and the Future of Work

Kelan, Elisabeth Book Patterns of Inclusion: How Gender Matters for Automation, Artificial Intelligence and the Future of Work Provided in Cooperation with: Taylor & Francis Group Suggested Citation: Kelan, Elisabeth (2025) : Patterns of Inclusion: How Gender Matters for Automation, Artificial Intelligence and the Future of Work, ISBN 978-1-040-13004-9, Routledge, Oxford, https://doi.org/10.4324/9781003427100 This Version is available at: https://hdl.handle.net/10419/312728 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/legalcode PATTERNS OF INCLUSION It is widely presumed that digitalisation, automation and artificial intelligence (AI) shape the future of work; yet, gender is rarely considered in those debates. This groundbreaking book, written by a leading thinker on gender, inclusion and organisations, is based on indepth research to show which patterns of gender and digitalisation emerge. By weaving these different patterns together, is it possible to understand the dynamic and complex ways gender and digitalisation intertwine in the work context? The book highlights how futures of work are imagined between automation and augmentation: it shows which tasks are expected to be done by machines, and where humans are expected to have a competitive advantage. The book showcases how algorithmic bias is constructed as ultimately fixable, and analyses in/ visibilities in AI production processes. Above all, the book shows how patterns relating to gender and inclusion are shaped and could be reshaped. This innovative book provides a stimulating and provocative read for those who are interested in how automation and AI shape the future of work in regard to gender and what this means for inclusion. Elisabeth Kelan is Professor of Leadership and Organisation at Essex Business School, University of Essex, United Kingdom. Kelan is an expert on gender and digitalisation, women’s leadership, men as change agents for gender equality, generations at work, and diversity and inclusion. “A groundbreaking book that fills a critical gap by providing a muchneeded thorough analysis through a gender lens of perspectives on the potential impacts of automation and AI on the future of work. It enables us to imagine more inclusive and equitable scenarios that better equip us to forge a fairer digital future for all.” Ursula Wynhoven, Director and Representative to the United Nations, International Telecommunication Union “An important work that provides unique, timely and exceptional insights into the digitalisation of gender. Through rigorous research, Elisabeth Kelan illuminates the gendered values, decisions and biases that shape digitalisation, automation and artificial intelligence systems, and why they must be grounded in equity and inclusion.” Melissa Suzanne Fisher, New York University Institute for Public Knowledge and School of Professional Studies PATTERNS OF INCLUSION How Gender Matters for Automation, Artificial Intelligence and the Future of Work Elisabeth Kelan Designed cover image: Getty Images/anna First published 2025 by Routledge 4 Park Square, Milton Park, Abingdon, Oxon OX14 4RN and by Routledge 605 Third Avenue, New York, NY 10158 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2025 Elisabeth Kelan The right of Elisabeth Kelan to be identified as author of this work has been asserted in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. The Open Access version of this book, available at www.taylorfrancis.com, has been made available under a Creative Commons Attribution (CC-BY) 4.0 license. Any third party material in this book is not included in the OA Creative Commons license, unless indicated otherwise in a credit line to the material. Please direct any permissions enquiries to the original rightsholder. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. This book is based on a Leverhulme Trust Major Research Fellowship [MRF-2019-069] and reports findings from a project funded by the British Academy [SRG20\200195]. This publication was supported by the University of Essex’s Open Access Fund. Data Availability Statement Due to the nature of this research, participants of this book did not agree for their data to be shared publicly, so supporting data is not available. British Library CataloguinginPublication Data A catalogue record for this book is available from the British Library ISBN: 9781032731728 (hbk) ISBN: 9781032669892 (pbk) ISBN: 9781003427100 (ebk) DOI: 10.4324/9781003427100 Typeset in Joanna by Newgen To Michael, who always encourages me to be curious about technologies. CONTENTS Preface ix Acknowledgements xiii 1 Introduction: HumanLike 1 2 Imaging Futures between Automation and Augmentation 24 3 Uniquely Human and the Automatability of SocioEmotional Skills 47 4 Algorithmic Bias as Ultimately Fixable 71 5 In/ Visibility by Design 95 6 Conclusion: Unwritten Rules 122 Appendix 142 References 146 Index 162 ACKNOWLEDGEMENTS xiv It has been fantastic to work with the team at Routledge on this publication. In particular, Rebecca Marsh not only believed in this book but also provided invaluable feedback. Essex Business School has provided an intellectually stimulating home during the research for and writing of the book. I could not have wished for more supportive colleagues. I am deeply grateful to my family for giving me the time and space to write this book. Some of the material covered in this book has been published in academic journals with Wiley: Kelan, E. K. (2024). Algorithmic inclusion: Shaping the predictive algorithms of artificial intelligence in hiring. Human Resource Management Journal, online early. Kelan, E. K. (2023). Automation anxiety and augmentation aspiration: Subtexts of the future of work. British Journal of Management, 34(4), 2057– 2074. Finally, I am extremely grateful to those individuals who kindly shared their experiences and reflections with me during the interviews. I would also like to thank those who have provided access to technology which I could explore for this research. This book would not have been possible without this support. newgenprepdf This chapter has been made available under a CC BY – Attribution license. DOI: 10.4324/9781003427100-1 1 INTRODUCTION HUMANLIKE Introduction The future of work and digitalisation is a topic that is dominating discussions in the media and in many organisations. While the future of work is regularly invoked, it has been suggested that the term ‘future of work’ is polysemous with various meanings attached to it. While how gender matters in these transformation processes is regularly ignored, how gender is relevant for digitalisation at work takes centre stage in this book. Based on detailed empirical research, I suggest in this book that we need to look at patterns around gender and technology that emerge in different settings. However, only by weaving together these different patterns is it possible to understand the dynamic and complex ways in which gender and digitalisation in the work context are intertwined. This introductory chapter discusses why focusing on how the future of work is imagined is central for which futures are being made possible and impossible. I also suggest that technologies are often imbued with magical and mythical qualities in everyday conversations. I then turn to discussing how the gender– technology dynamic is understood before explaining the underlying research for and the structure of this book. The book argues that we need to understand the dynamics between gender and digitalisation in the work context to create more equitable futures. INTRODUCTION: HUMAN-LIKE 2 Imagining Futures of Work and HumanLike Intelligence The future of work is a topic that enjoys a constant interest – multiple reports and books are authored every year that trace the question of what the future of work might hold. Wajcman (2017) observes that predicting the future of work has become ‘big business’. This extends to conferences on that topic. She finds that such conferences follow a familiar and predictable pattern: technologists marvel at the latest technological progress, economists paint a concerning picture of the future of jobs and futurists predict the next trends (Wajcman, 2017). The term ‘future of work’ in itself has been described as a ‘floating signifier’ to which various meanings are attached by different groups (Schlogl et al., 2021). A common feature for discussions on the future of work is that they play with the dichotomy between utopia and dystopia (Schlogl et al., 2021; Howcroft & Taylor, 2023). One could argue that how the future is imagined has limited consequences because most of such predictions tend to be wrong anyway. However, it would be a mistake to dismiss these predictions of the future as irrelevant or meaningless. Such visions for the future have profound consequences for societies because they create ideas of what might be possible (U r ry, 2016). They also create realities (Schlogl et al., 2021). Urry (2016) warns against seeing futures simply based on a predetermined path of development of technologies or as seeing futures as completely open and empty. How futures are imagined, for instance, in discourses on the future of work, shapes what might or might not be possible. Science fiction can provide a blueprint of how potential desirable or undesirable futures are imagined, which then influences what is seen as possible (Jasanoff, 2015). Instead, Jasanoff (2015) follows the idea that society and technology are coconstructed. Such a coconstruction becomes manifest in sociotechnical imaginaries (Jasanoff, 2015). Sociotechnical imaginaries are ‘collectively held, institutionally stabilized, and publicly performed visions of desirable futures, animated by shared understandings of forms of social life and social order attainable through, and supportive of, advances in science and technology’ (Jasanoff, 2015, p. 4). In other words, sociotechnical imaginaries are a way to analyse how individuals are engaging in collective practices when imaging the future. Diverging sociotechnical imaginaries can coexist and shape one another, and are evaluated and debated through societal discourses. Even though most INTRODUCTION: HUMAN-LIKE 3 sociotechnical imaginaries are ultimately interested in shaping futures that are desirable, they often do so by outlining scenarios that ought to be avoided; the tension between a desirable utopia and an undesirable dystopia is actively used in sociotechnical imaginaries (Jasanoff, 2015). Much of the literature on the future of work attempts to warn of undesirable dystopias, particularly through job losses associated with technologies. Many of the widely cited reports on the future of work engage in predictions on what the future of work might entail and how many jobs are going to be lost due to automation that is driven by technologies (Manyika, Chui, & Miremadi, 2017; Hawksworth, Berriman, & Goel, 2018; Organisation for Economic Cooperation and Development [OECD], 2016; World Economic Forum, 2020; Balliester & Elsheikhi, 2018; Hatzius et al., 2023). It is also common to contextualise current changes in regard to different industrial revolutions. Schwab (2018) suggests that the First Industrial Revolution was marked by mechanisation of the textile industry in Britain; the second was associated with electricity, the telephone and the automobile; the third was related to changes in digital computing in the 1950s; and the fourth is marked by a range of new technologies, including artificial intelligence (AI), distributed ledgers (blockchain), advanced materials and virtual and augmented realities. Schwab (2018) predicts that exponential growth in these new technologies will lead to rapid change, and that automation may accelerate job losses. In contrast, McAfee and Brynjolfsson (2014) coined the term ‘second machine age’. The first machine age is equated with the Industrial Revolution when machines improved human labour; the second machine age is said to have begun in the mid1990s with digitalisation and is characterised by machines not simply following rules but solving problems on their own. Thus, machines now perform cognitive tasks previously reserved for humans. This leads to fears that humans are replaced by machines. Fears of humans being replaced by machines are of course not new. For instance, during what Schwab (2018) would call the First Industrial Revolution, the relationship between machines and humans was redrawn and a common perception was that machines are going to replace physical power that people had exerted before (Standage, 2002). However, the arrival of the Mechanical Turk, the 18thcentury lifesize chessplaying automaton (see Chapter 5), challenged this idea, because the Mechanical Turk seemed to outperform humans mentally (Standage, 2002). Although INTRODUCTION: HUMAN-LIKE 4 what appeared magical to audiences at the time turned out to be a hoax, the possibility that machines could outdo humans mentally was certainly part of the fascination with the Mechanical Turk. This blurring of what constitutes human and what constitutes machine intelligence is also visible in how Joseph Weizenbaum’s chatbot ELIZA was received. ELIZA was a natural language processing computer programme developed by Joseph Weizenbaum in the 1960s to explore how humans and machines communicate. The name ELIZA was chosen as a reference to Eliza Doolittle in George Bernard Shaw’s play Pygmalion (Natale, 2019; Dillon, 2020). This choice of name carries strong gender and class connotations (Dillon, 2020). Weizenbaum’s aim with ELIZA was to show humans that AI is an illusion (Natale, 2019). However, Weizenbaum was shocked by the fact that rather than recognising the difference between human intelligence and AI, humans engaged emotionally with the machine and anthropomorphised it (Treusch, 2017). In other words, users perceived ELIZA’s answers as humanlike. Even when users, such as Weizenbaum’s secretary, knew that ELIZA was not engaging on an emotional level, they ascribed emotional competence to the chatbot (Dillon, 2020; Treusch, 2017; Rhee, 2023). The ‘ELIZA effect’ describes how humans presume more intelligence in a machine than really exists (Hofstadter, 1995; Dillon, 2020). While we return to the gendering of current virtual personal assistants (VPAs) (see Chapter 5), Dillon observes that ‘when a human being is conversing with a VPA, the brain is processing that conversation as it would a conversation with another human being. The Eliza (sic) effect is here embedded in the neural response to the voice’ (Dillon, 2020, p. 11). Humans thus engage with machinegenerated voices in the same way as with human voices. This supports the illusions of AI as humanlike intelligence rather than exposing it as an illusion, as Weizenbaum had hoped. When ChatGPT reached the mainstream in late 2022, many commentators similarly marvelled at the humanlike answers the chatbot was able to provide (Hatzius et al., 2023). It was exactly this humanlikeness that promoted concerns that human mental capacities could be replaced with machines (Hatzius et al., 2023). This in many ways echoes Weizenbaum’s own concern about AI, which he wanted to expose with ELIZA (Treusch, 2017), but also the wider concerns that humans will be replaced by machines. Although predictions about humans being replaced by machines appear as dystopian and are usually followed with calls for a universal INTRODUCTION: HUMAN-LIKE 5 basic income, there are also sociotechnical imaginaries that are utopian and as such more hopeful. Here, utopias are imagined as desirable, where humans can focus on specific tasks: those where humans have a competitive advantage or where they collaborate with machines (Hatzius et al., 2023; Daugherty & Wilson, 2018). It is argued that these tech - nologies also mean that new jobs emerge, and while some jobs disappear, new ones will be created (Hatzius et al., 2023). Existing jobs might also be enhanced by technologies through new human– machine collaboration (Daugherty & Wilson, 2018). These utopian discourses are closely associated with what has been called augmentation, where humans and machines augment each other’s skills (Raisch & Krakowski, 2021). While automation is largely associated with dystopian ideas of jobs being replaced by machines, augmentation represents the utopian idea that humans can either focus on activities where they outperform machines or collaborate with machines. Although automation and augmentation are often presented as opposing, most workplaces will experience both automation and augmentation to different degrees. This in itself is not a new phenomenon. When Sennett (1998) returned to a bakery that he had visited many years before, he noticed how the process of baking bread has been computerised; the bakers no longer made psychical contact with the ingredients and monitored the breadmaking process via screens. Sennett (1998) argues that this leads to a deskilling and alienation of bakers with no handson knowledge of how to produce bread. Work has become what Sennett (1998) calls ‘illegible’ to the bakers. This preempted Sennett’s (2008) later argument that craftwork is a way through which people comprehend their worlds. Of course, many people would argue that the digitalisation of bread making is enhancing bakers’ skills – they need to know about technology and how to engage with this technology to achieve optimal results. In fact, Sennett (1998) describes that bakers manipulate the machines if something goes wrong and thus develop additional knowledge, but he still maintains that bakers have lost the ability to bake bread in the traditional sense. As digitalisation changes the skills required for jobs, such arguments suggest that certain skills will no longer be required and will be lost because people do not invest time in honing them. This means that while some jobs will be automated and might disappear over time, digitalisation will also change the skills required to do existing jobs. INTRODUCTION: HUMAN-LIKE 6 There is also an important change to which jobs are expected to disappear. While in the past, bluecollar work was presumed to be automated, it has more recently been suggested that the focus of job replacement due to automation has moved to whitecollar professional jobs (cf. Wajcman, 2017; Howcroft & Rubery, 2019; Cave, 2020). This marks a shift from machines replacing physical human power, to machines threatening to replace human minds. They emulate human intelligence. While some jobs might be replaced due to technologies in professional services, it can also be expected that the tasks professionals do will change. Digitalisation will affect temporalities in those professions. For instance, when spreadsheets were first introduced, accountants were able to complete tasks quicker due to automation, but clients also started to expect a quicker turnaround and also more analytical insight (O’Connor, 2023). Similarly, the introduction of new communication tools like online videoconferencing via Zoom and shared calendars has led to more work rather than less with everdecreasing increments of time immediately filled by demands, leading to the conclusion that ‘whitecollar work always seems to expand to fill the time available’ (O’Connor, 2023). This chimes with research that has shown that the idea that digital calendars optimise one’s time is a false belief because such technologies are not giving individuals more time (Wajcman, 2019). Instead, digital technologies contribute to an acceleration of everyday life (Wajcman, 2015). Similarly, technologies that are expected to be time saving often do not deliver on the expected effects. Cowan (1983) shows how household technologies, which should save effort and time on household tasks, meant that mothers and wives increasingly took on household activities that had previously been completed by fathers, husbands, children and servants. As such, household technologies did contribute to an intensification of what mothers are expected to do without freeing up time (Cowan, 1983). Talking about household technologies also draws attention to how the future of work is commonly framed: most discussions on the future of work focus on paid work rather than unpaid work (Lehdonvirta et al., 2023). Work done in the home such as caring for children or elderly relatives, preparing meals or writing birthday cards is virtually never discussed in relation to the future of work. Feminists have long made the argument that work should encompass work done in the household (Oakley, 2018; England & Lawson, 2005). This can entail unpaid and paid care work where a specific focus in INTRODUCTION: HUMAN-LIKE 7 regard to unpaid work is on how this work is often racialised (Ehrenreich & Hochschild, 2003; GutiérrezRodríguez, 2014). Unpaid work at home is also experiencing digitalisation, offering the opportunity to explore what and how automation and augmentation happen and interact with gender (Strengers & Kennedy, 2020; Lehdonvirta et al., 2023). While it has to be acknowledged that the definition of work employed in discussions of the future of work is narrow and that this is problematic in itself, in this book, I have decided to focus on how such specific perspectives of the world have consequences for how the future of work is imagined. I argue for considering the complex interplay of intersectional inequalities, which is discussed more fully later. Such an analysis allows for questioning what specific views of the world allow us to see and what they obscure. The acknowledgement that work largely focuses on paid work is a necessary but not sufficient analytical tool to make the dynamic interplay of intersectional inequalities visible. The book therefore focuses on work as paid work and acknowledges that such a framing of work is exclusionary because it excludes unpaid care work. However, how the future of (paid) work is imagined is in itself a subject worthy of study because in these sociotechnical imaginaries, patterns of inclusion and exclusion might be perpetuated or challenged. The Magic of HumanLike Technology In everyday conversations, terms used to describe current technologies such as digitalisation, AI and algorithms have often a mythical and magical quality to them. Finn (2017) suggests that people have developed a strong belief in algorithms that is faithlike; with little understanding how such technologies operate, Finn (2017) suggests that people believe in them. Finn (2017) argues that machines occupy similar spaces to magical and mythical thinking in primitive societies. Building on Malinowski’s influential work, Finn (2017) suggests that technology takes the place of rituals and beliefs in modern societies. Finn (2017) proposes that engaging in rituals such as summoning a ride via Uber fulfils the same function as rituals in primitive societies: it helps humans to deal with danger and uncertainty. In many everyday conversations, terms like algorithm take on a mythical and magical status where these terms are used without fully understanding what they entail and how these technologies work. This mythical and magical nature INTRODUCTION: HUMAN-LIKE 8 of terms like algorithms and AI can be exposed by looking at definitions of those terms, which to most individuals working outside a narrow academic field will appear as abstract. However, in order to weaken the mythical and magical power of these terms, it is important to develop an understanding of what these terms refer to without getting lost in the technical definitions of these terms. A first term one hears regularly is digitalisation. Digitalisation differs first of all from digitisation. Digitisation entails a move from analogue to digital (Oxford English Dictionary, 2023d). For instance, it might be decided to scan all paper records to create a digital replica, which is digitisation. In regard to hiring, it was long common to conduct penandpaper psychometric tests, and if the same test is simply offered via a computer, this would also be digitisation. Essential for digitisation is that a digital replica of an object is created. As such, if a train company decides to create digital twins to test, for example, how different rolling stock performs on tracks, this is a form of digitisation. Digitalisation by contrast is a broader concept and refers to the adoption of technology by an organisation, country or industry (Oxford English Dictionary, 2023c). For example, if one transfers a penandpaper psychometric test to an online version without any changes, then the delivery of the test changes but the test itself does not change. However, if one uses AI to predict the best candidate for a job, the process itself changes. This is what is commonly meant with digitalisation. Digitalisation is thus a term that describes how processes themselves change due to the application of a digital technology. Digitalisation is also often referred to as digital transformation. Digitalisation itself is a broad term that can entail a myriad of other terms and technologies. One way in which processes change through digitalisation is automation. Automation means that a process that was previously done by a human is done by a machine, or in other words, ‘[t] he action or process of introducing automatic equipment or devices into a manufacturing or other process or facility; (also) the fact of making something (as a system, device, etc.) automatic’ (Oxford English Dictionary, 2023b). While some automation might use AI, not all automation will necessarily use AI. An example of automation is the car factory: traditionally, humans would have assembled a car but now robots do the same job. This is often contrasted with augmentation, where humans and machines collaborate and augment each other’s skills (Raisch & Krakowski, 2021). INTRODUCTION: HUMAN-LIKE 9 One technology that is central for digitalisation is AI. The OECD has defined an AI system as ‘a machinebased system that can, for a given set of humandefined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy’ (OECD, 2019). The European Union defines an AI system in similar terms as a machinebased system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.’ (European Parliament, 2024, p. 165) Another definition of AI is that of AI as ‘intelligent agents’ that perceive their environment and perform actions (Russell & Norvig, 2021). Intelligence here means that a machine appears to display humanlike intelligence (Russell & Norvig, 2021). Much of what is commonly called AI is technically machine learning. Machine learning is a subset of AI. Machine learning means that ‘a computer observes some data, builds a model based on the data, and uses the model as both a hypothesis about the world and a piece of software that can solve problems’ (Russell & Norvig, 2021, p. 669). Data is central in the machine learning process; commonly, machine learning involves the machine searching for patterns in the data to develop a model of what the machine perceives as true in this context (Broussard, 2018). This is the training or learning part and the model is then tested with new data to see how accurate predictions are (Broussard, 2018). If we want to define AI, a useful definition entails the ways to analyse, derive learning from and make predictions based on data (Kelan, 2024). Based on this definition, we can see that data is central for AI. We see that expressed often in the form of Big Data, which is then analysed, learned from and used as the basis for making predictions. Many writers on the future of work therefore liken data to the ‘new oil’ (De Cremer, 2020; Frey, 2019; Schwab, 2018) to indicate that data is the new natural resource that has to be mined. INTRODUCTION: HUMAN-LIKE 16 intertwined. If the growing importance of AI is discussed, it is thus central to also explore the gender dynamics entailed in AI (Toupin, 2023). Distinguishing between gender by association and by design is theoretically useful, but in practice, it is more difficult to distinguish between these two ideas. This is commonly phrased in relation to concerns that the lack of representation of women and other groups in the design of new technologies leads to technologies being biased (see also Chapter 6). While the lack of women among the designers of new technologies could be seen as an exclusion from potentially lucrative careers, the idea that women as designers automatically leads to genderinclusive design is problematic in itself. First, it ignores the socialshaping dynamics around how technologies are designed, which might, for instance, follow commercial concerns. So even if women have unique insights into the world, which is often expressed as situated knowledge (Haraway, 1991), they might be unable to bring this knowledge to bear on the design process. Second, it presumes that women designers speak for all women. This ignores that a white, middleclass woman might have different views on the world than a Black, workingclass woman. Women are positioned differently by intersecting inequalities (Acker, 2006; McCall, 2005; Nash, 2008; Crenshaw, 1989) and whose voices are heard and listened is influenced by complex power dynamics (Spivak, 1988; Mohanty, 1986). Considering gender by association and gender by design is a useful analytical device to understand the social shaping of gender and technology if such analyses consider how women might be positioned differently in relation to technologies. In order to illustrate reflections that seeing gender and technology as mutually constitutive can lead to, I want to turn to a reading of the Turing Test that is informed by gender. The Turing Test has become emblematic for seeing technology as displaying humanlike intelligence. The Turing Test is a thought experiment by Alan Turing, the mathematician and computer scientist, often associated with his work on code breaking at Bletchley Park during the Second World War. Turing’s thought experiment (1950) discusses the possibilities around machines being ‘intelligent’. In his writing, Turing (1950) suggests that an intelligent machine should be able to perform a game – referred to as the imitation game – that was a popular party game at the time. The core idea of this game is that a human is unable to discern if the answers given originate from a machine or a human (Sutko, 2020). INTRODUCTION: HUMAN-LIKE 17 If a human cannot distinguish the answer from a human and a machine, the machine passes the Turing Test by displaying humanlike intelligence. What is commonly ignored in the popular perception of the Turing Test is the gender dimension of the original Turing Test (Shah & Warwick, 2016; Sutko, 2020; Genova, 1994; Drage & Frabetti, 2023). In order to under - stand how the Turing Test might relate to gender, it is necessary to explain the imitation game in more detail. The imitation game has three players: a man (A), a woman (B) and an interrogator of either sex1 (C) (Turing, 1950; Saygin et al., 2000). The aim of the game is for the interrogator (C) to determine who is a woman by asking questions such as about hair length. Both A and B must convince C that they are indeed a woman. The interrogator (C) cannot see either players A and B. The players should not use their voice to communicate because this might give away gender. Instead, they communicate via notes that should ideally be typed up via a teleprinter (Turing, 1950) to not allow conclusion about gender from handwriting. Turing takes this game a step further where a machine takes the place of A. The most common interpretation of the Turing Test is that C now needs to find out who is the human: A or B (Saygin et al., 2000)? Such an interpretation entails that the aim of the game is no longer to convince C who the woman is but instead who the human is. One might also read this scenario as a way in which A and B attempt to appear as a human woman. Turing himself does not mention that the game has been altered from passing as a woman to passing as a human. Two arguments are used to support this idea. First, in trying to pass as a human woman, neither the man nor the machine has an advantage (Saygin et al., 2000). They both have to impersonate a woman because in the modified version, A is not a man but a machine, whereas B the woman is now B the human. A second argument is that Turing, as a gay man, might have picked gender purposefully to draw attention to gender (Genova, 1994; Hayes & Ford, 1995). While those arguments have some purchase, it seems more likely that Turing in fact did not mean for the game to be about guessing gender, where the human and the machine pass as a woman, but rather about if a machine can convince a human into believing that the machine is indeed human. However, the idea that gender might be central for the Turing Test provides in itself an interesting thought experiment. If the Turing Test is understood as a man and a machine passing as a woman, this allows for connections with gender theories where gender is often seen as something INTRODUCTION: HUMAN-LIKE 18 that is done, achieved and performed (West & Zimmerman, 1987; Butler, 1990).2 If the Turing Test is seen as a way to pass as a woman, then the man and the machine have to emulate traditional markers of femininity and display them. For example, in the context of the game we have seen questions about length of hair. At the time, long hair was reserved for women and as such could be seen as a marker of femininity. If both the machine and the man pretend to have long hair to pass as a woman, they have understood these markers of femininity and are able to use them to pass as a woman. For a machine to understand the markers of femininity, it would have needed to learn from data that women tend to have long hair and men short hair to then make the prediction that having long hair means that the person is more likely to be a woman. We will delve into questions of how a machine knows who is a woman in Chapter 5 in more detail. For the time being, it suffices to say that reading the Turing Test from a gender and technology perspective opens up novel research questions to investigate in the future. Research Patterns In this book, I weave together materials from a range of settings that together form patterns that can inform us about how gender and digitalisation are mutually shaped. I detail my research approach in the appendix. The first research context is books on the future of work that are written for the popular market. I selected books that focused on work and technology. I scanned the business press on book reviews, searched Amazon’s recommender system and asked for book recommendations in my professional network. I visited physical bookstores to see what was shelved in sections on the future of work. I also interviewed thought leaders on the future of work. I approached individuals who had a visible presence in discussions about the future of work such as by giving keynotes, being on a panel, giving media interviews or publishing reports. Those individuals worked in international organisations, policy and learned societies. I also spoke to individuals who were working in professional service firms advising clients on the future of work and the field of AI ethics. The findings from the analysis of books and the interviews with thought leaders inform Chapter 2 to illuminate debates on automation and augmentation. Building on the interviews and the books, I then focused on exploring those areas that were seen as particularly threatened by emerging INTRODUCTION: HUMAN-LIKE 19 technologies: the professions. I spoke with individuals working in a range of professions such as audit, tax, legal, consulting, financial technology and architecture. I also spoke to a range of individuals who were experts on socalled new work practices such as holacracy (Robertson, 2015), who could talk about how professional work can be organised in different ways. I was interested to explore which skills are changing in relation to technology and can be trained through technologies such as VR. I was able to conduct a form of autoethnography (Hine, 2020; Sparkes, 2003) by using a VR headset (Oculus Quest 2). I undertook training in a range of contexts and settings such as counselling, health and safety, onboarding for people working in grocery stores and leading for inclusion. I analyse this material in Chapter 3 to show which skills are constructed as uniquely human. Another area that was regularly mentioned particularly in the context of algorithmic bias was using AI in hiring. In this book, I draw on interviews with individuals who worked in areas associated with hiring technologies. These included individuals working in different functions such as those who provide the technology for hiring, those who work in HR functions, and recruiters and hiring consultants. In addition, I tested some of the hiring technologies myself. This included VR recruitment environments, online aptitude and personality tests and asynchronous video interviews. I largely draw on this material in Chapter 4. The final context from which I draw material relates to how data and gender are intertwined in production processes around AI. My main interest here was on how AI and gender are mutually shaping through processes such as data labelling or data annotation. I spoke to a range of individuals who were involved in the AI production process. Those individuals were, for instance, linguists working in automatic speech recognition, individuals managing data labellers or experts on how AI is created. Some interviewees showed me examples of how data labelling was done and how gender matters in the process, which was helpful to understand these practices. This material is largely covered in Chapter 5. While all of these contexts are distinct, there was significant overlap between the topics that came to the fore when the material was analysed. As such, the different contexts combine in specific ways to show patterns of gender and digitalisation. These patterns were significantly influenced by being conducted at a specific point in time. This influences which technologies were mentioned as examples. The pandemic itself changed digitalisation INTRODUCTION: HUMAN-LIKE 20 substantially and is said to have accelerated digitalisation (AmankwahAmoah et al., 2021; Schlogl et al., 2021; McKinsey, 2020). Video confer - encing moved from a rarely used technology to the standard medium of how work was done during most of the pandemic. Work itself has changed due to the pandemic. Selecting candidates using digital means became a necessity overnight and has, as a consequence, evolved. At the same time, the pandemic meant that the research I was conducting was rather different than I had planned (see the appendix). By weaving different patterns that emerge in these contexts together, this book shows how gender and digitalisation at work operate in different settings and it also shows how these different patterns form a larger picture on gender and digitalisation in the work context. Structure of the Book The book is structured into two broad sections. The first section focuses on the discourse of the future of work. In this part of the book, I analyse common tropes used in the books on the future of work and by interviewees such as the manversusmachine idea and how emotions are constructed as uniquely human. The second part of the book focuses on data and data practices. I analyse how algorithmic bias manifests in relation to hiring and what this means for inclusion. I also show how data practices construct and reconstruct worldviews such as around gender and diversity. Following this introduction, the book is structured into four chapters and a conclusion. How the future of work is imagined is at the centre of Chapter 2. The chapter analyses how dystopian views emerge in popular books on the future of work, which pitch humans against machines in an epic battle. I discuss this as the managainstmachine trope where machines replace humans. I also show that those who hold atrisk jobs are imagined as men, particularly middleclass men in whitecollar professions. The chapter shows how an alternative, possibly more utopian perspective takes hold where humans and machines are not enemies but actually work together. This human– machine collaboration is heralded as a new form of diversity. In this scenario, humans engage in enjoyable and creative tasks whereas machines do the repetitive and mundane work. The books I analysed maintain that socioemotional skills are out of reach for machines. However, the chapter shows how constructing socioemotional skills as safe from INTRODUCTION: HUMAN-LIKE 21 automation leaves out a wider consideration of how gender, race and class structure current and future inequalities. In Chapter 3, I continue this line of inquiry by questioning the idea that socioemotional skills are indeed outside of the realm of machines. This chapter draws on interviews with thought leaders and professionals. I show how drudgework is widely assumed to be automatable because it follows repeatable patterns. I show how the automation of drudgework shifts tasks in professional work and changes structures in professional firms. The chapter shows how socioemotional skills are constructed as uniquely human because they are seen as out of reach of machines. Looking particularly at examples where machines could be said to engage in socioemotional skills, I show how machines are trained to recognise human emotions but also how machines train humans in showing appropriate emotions. The chapter argues that socioemotional skills are following patterns that can be automated and as such do not constitute a competitive advantage of humans over machines. However, if those socioemotional skills are performed by machines will depend to a large degree on the social desirability of having these tasks performed by machines. Chapter 4 then engages with the question of how technologies are used in hiring practices. Hiring is a central function of the organisation: it is important to have the right people in place to ensure that the organisation can fulfil its purpose. Yet hiring is also a process fraught with human bias. This is where technology comes in because it promises to make these processes not only more efficient but also more objective. However, technologies have been shown to repeat and amplify exactly these human biases. The chapter traces how a technooptimist’s perspective that technology improves business processes can be reconciled with algorithmic bias. This chapter draws on interviews with experts on the future of work, those who design hiring technologies, as well as my own experiences with those technologies. First, I show that a technooptimist’s stance is the dominant perspective in most of the interviews and that this stance is maintained by constructing algorithmic bias as ultimately fixable. The chapter details how it is suggested that bias emerges from people and which processes and practices are constructed as being able to fix algorithmic bias. I also show that some interviewees displayed what I call technohesitation, a waitandsee stance, which acknowledged that AIsupported hiring will become normalised over time. Overall, the chapter makes the argument INTRODUCTION: HUMAN-LIKE 22 that algorithmic bias has to be constructed as ultimately fixable to maintain a technooptimistic stance. Chapter 5 traces the question of how a computer knows about the gender of the person. I show that data labelling is central in this process. The chapter centres on two aspects. First, it focuses on who is doing data labelling work. The chapter discusses AI’s hidden workforce – those who label data in the AI supply chain. Second, the chapter discusses the constructions of reality that emerge from data labelling. I discuss practices such as how classifications used in data labelling represent specific worldviews. The chapter suggests that what is presented as objective and universal is in fact subjective and partial and as such potentially exclusionary. The chapter suggests that in order to create more inclusionary approaches, it is central to make the underlying processes of constructions that happen in relation to data labelling visible. Chapter 6 offers a conclusion by weaving different threads that the book uncovered together. I show which futureshaping patterns around digitalisation and gender emerge. The chapter illustrates how seemingly isolated issues form patterns that either hinder or foster inclusion. I speculate how alternative futures might be created. The final chapter also suggests that data is inherently social and thus inevitably biased. However, while this might lead to existing gender patterns being repeated, I suggest that alternative patterns are possible that can be more inclusive. Conclusion In this opening chapter of the book, I show how gender, digitalisation and the future of work can be conceived as patterns. Patterns are central for technologies such as machine learning that read existing patterns to predict potential futures. This book focuses on different individual patterns that if brought together show more complex and dynamic patterns of how gender and technology interact in the work context. The chapter started by arguing that how futures of (paid) work are imagined shapes how potential futures might unfold. I then suggested that many new technologies appear magical and mythical to users. I defined key terms for technologies I use in this book and how I employ these terms. In the next section, I explained how I see gender and technology as coconstructed and how seeing gender as performed in and through technologies is a useful lens for this research. INTRODUCTION: HUMAN-LIKE 23 Following this, I outlined how the research was constructed before providing details on how this book is structured. The book weaves together different patterns of gender and technology that emerge in specific settings. By weaving these patterns together, a more nuanced, complex and dynamic pattern of how gender and digitalisation interact in the future of work emerges. The book thus argues that patterns of gender and digitalisation are dynamic and complex but can be used for creating inclusion. Notes 1 Sex is used in the original. 2 I have discussed differences and similarities of these approaches of how gender is done and performed in detail elsewhere (Kelan, 2009, 2010). This chapter has been made available under a CC BY – Attribution license. DOI: 10.4324/9781003427100-2 2 IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION Introduction Discourses on the future provide templates for how potential futures might unfold. These templates represent patterns based on which thinking of the future is structured. Such discourses of the future regularly play with the contrast between utopian and dystopian visions of the future (Jasanoff, 2015; Schlogl et al., 2021; Howcroft & Taylor, 2023). For discourses on technology at work, these visions manifest in relation to automation and augmentation (Kelan, 2023b; Raisch & Krakowski, 2021). Automation presents a catastrophic view of the future or a dystopia where machines have takenaway jobs that humans used to do, leading to mass unemployment and social unrest. Automation constructs humans and machines as enemies, with both competing for the same work. Augmentation, in contrast, means that humans and machines collaborate to achieve work together by playing to each other’s strengths. In this vision of the future, humans and machines collaborate seemingly harmoniously. Humans are said to pivot to skills that IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 25 machines are presumed to struggle with: socioemotional skills. Socioemotional skills are often seen as something that only humans can do. In this chapter, we will see how automation and augmentation are not only regularly invoked concepts in the books on the future of work that I analysed, but they also resonate strongly with gender (Kelan, 2023b). Discussions of automation regularly referred the manversusmachine trope. Furthermore, the jobs that were at risk of automation often belonged to men. The professions were predicted to disappear, yet the role women play in the professions was neglected. Augmentation in contrast was associated with the rise of socioemotional skills, which were constructed as a core human skill that both women and men can display. Technologies were discussed as gendered, particularly in the context of algorithmic bias. This chapter traces how books on the future of work in relation to automation and augmentation relate to gender. I show how discourses of the future of work replicate some gender patterns but also create new patterns. This chapter discusses the contours of some of the patterns that will be discussed in greater detail throughout the book. Visions of the Future of Work Prior to the Covid19 pandemic, one of the most pressing questions regularly asked in the media was ‘will a robot take your job?’. The idea of robots stealing jobs is an established trope in discussions about technology. Such discussions are often followed by a mention of Luddites destroying textile machinery. These images pitch humans against machines. As machines are deployed, humans lose their jobs. Although research has shown that automation replaces and creates new work (Autor, 2015; Autor et al., 2023), the idea that humans are replaced by the latest technologies associated with digitalisation permeates popular imaginaries such as the multitude of reports on the future of work (OECD, 2016; Frey & Osborne, 2017; Manyika, Chui, et al., 2017; Manyika, Lund, et al., 2017). These worrying scenarios about the future of work seem to suggest that mass unemployment will destabilise economies and societies unless urgent policy interventions are taken. The Covid19 pandemic has, at least temporarily, called into question such scenarios. Instead, ideas such as the ‘great resignation’ and the ‘great attrition’ suggest that people no longer want to work like before IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 32 referenced by Baldwin, Benanav, de Cremer, Frey, McAfee and Brynjolfsson, and Schwab, and often discussed in the context of the human being beaten by the machine that has become intelligent enough to do so. The manversusmachine trope also finds its expression in automation anxiety, which de Cremer, Schwab and Susskind use to refer to the fear that a robot might take your job. The idea that there is less work left for humans is also central in Merisotis’ book. De Cremer concludes that the thinking of manversusmachine, together with scientific evidence on automation anxiety, centres most discussions on AI’s potential to replace people’s jobs. In a similar vein, Susskind suggests that machines continually improve their performance, which limits activities where humans have the edge. For Susskind, this challenges the idea that humans are superior and could not be replaced. Instead, he suggests that an ‘inferiority assumption’ might be more accurate in that machines rather than humans become the norm for performing tasks. A variation of automation anxiety that Daugherty and Wilson and de Cremer reference is algorithm aversion. Daugherty and Wilson describe algorithm aversion as the phenomenon that people trust humans more than machines, whereas de Cremer notes that it means that people avoid taking advice from algorithms. De Cremer explains the suspicion towards algorithms with the metaphor of the black box. Given the fact that algorithms function like black boxes, people are sceptical about algorithms making autonomous decisions. In fact, the black box idea is another common metaphor used that appears in five books (Baldwin, Daugherty and Wilson, Schwab, West, and of course, de Cremer, as discussed previously). It commonly describes that the recommendations that algorithms make are opaque, even to those who design those systems. The lack of explainability is a problem that Daugherty and Wilson, and Baldwin mention. West, in contrast, refers to how the European Union’s General Data Protection Regulation is addressing the black box by giving individuals insight into how the black box operates. The black box metaphor, together with a discussion of algorithm aversion, is mobilised to explain why automation anxiety is a concern. However, while the manversusmachine trope is regularly found in the books I analysed to express automation anxiety, I also found that augmentation as humans and machines collaborating was discussed in the books. IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 33 Daugherty and Wilson are possibly the most explicit in that they transform the manversusmachine trope into human + machine to argue for the value of augmentation. It is also notable that in the book title, Daugherty and Wilson use human + machine, which challenges the rather exclusionary notion of the manversusmachine trope. In some instances, they talk about man + machine as a direct contrast to man versus machine. Daugherty and Wilson explain that it is common to see humans and machines as rivals where machines steal humans’ jobs. However, in their book, they stress that machines and humans collaborate, which represents augmentation rather than automation and replacement. Similarly, de Cremer presumes that humans and machines will collaborate, and he calls this the new diversity. De Cremer writes an entire section about this new diversity. Notably, diversity is not referring to diversity among humans, but this new diversity that de Cremer describes lies between humans and machines. Like with human– human diversity, de Cremer cautions that human– machine diversity will appear alien to many humans. This is due to the fact that we have been conditioned to think about humans and machines as antagonists rather than as symbiotic. As such, this new form of diversity is in fact resonating with augmentation. Given the popularity of the manversusmachine trope in popular culture, it is not surprising that the trope was regularly used in the books I analysed. The manversusmachine trope was invoked directly but also indirectly; for instance, when the black box of AI and algorithmic aversion was discussed. However, it is notable that augmentation was presented as an alternative to automation, where machines replace humans. In fact, this augmentation was talked about as new diversity, where diversity refers to humans and machines collaborating. Yet, even these collaborations between humans and machines were marred by the potential negative impact of humans seeing machines as enemy. The obvious criticism of the manversusmachine trope is that it could be read as gender exclusionary. While clearly, some authors make attempts to be more gender inclusive by using human + machine, the manversusmachine trope is largely used to articulate automation anxiety and to show how augmentation might be hampered by humans feeling hostile to technology. I will now turn to understanding in a more granular way who is expected to lose out in this epic battle between humans and machines. IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 34 Men’s Jobs In the previous section, we have seen how the manversusmachine trope is commonly used to express automation anxieties. I will now show that it is men who are constructed as at risk of seeing their jobs disappear. The risk that men might lose their jobs is leading to a heightened concern for the future of work. For example, Frey states that it is men who are more likely to be replaced by robots than women. There is also concern that men in their prime will face redundancy but lack the flexibility in their identities to move to alternative jobs, as Susskind says. This affects men in whiteand bluecollar jobs and thus spans different class backgrounds. As such, the argument that men who lose their jobs are unable to find new work, which affects their sense of self, has to be read in the context of men’s traditional role in society. Men’s traditional role in Western societies entails being a breadwinner. This is, for instance, addressed by Benanav. Benanav suggests that the concept of the male breadwinner as the head of the household, and women as the main caregivers earning supplemental incomes, is deeply enshrined in the sociocultural fabric of many economies. He cites minijobs in Germany, which he suggests are effectively designed to be done by stayathome wives, whose incomes supplement those of the main male breadwinners. This traditional arrangement is rewarded by the state through tax incentives. The gendersegregated nature of the workforce, with men and women being clustered in different jobs, or references to how women and men might be positioned in these new futures of work, is at best marginal in the books. However, Frey and Susskind discuss pinkcollar work. Pinkcollar work has traditionally been done by women and is therefore associated with the colour pink. Susskind alludes to the fact that the naming of pinkcollar work is unfortunate. He goes on to explain that men who miss out on bluecollar work are often unwilling to take on pinkcollar work. He describes this as problematic because many pinkcollar jobs are at the moment out of reach of machines. Frey in turn discusses how pinkcollar work became more important with the introduction of the typewriter. He here links job growth to mechanisation. At the same time, he acknowledges that the growth of the pinkcollar workforce came to an end in the 2000s as computers became ubiquitous. However, he does not see these trends as affecting women negatively IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 35 because he states that women made inroads into wellpaid jobs, so much so that, as he states, younger women in the United States now outearn their male counterparts. Overall, these constructions leave the reader with the impression that women are doing well, in spite of automation threats. Yet, there is a concern for men who lose their jobs but are not flexible enough in their identities to take on other types of work that are seeing growth because such work is associated with women. In this section, we have seen that men’s jobs, both in blueand whitecollar jobs, are constructed as particularly at threat of automation. Women’s jobs in contrast are not seen as under threat of automation. Pinkcollar work, for instance, is constructed as futureproof and it is also stated that women in generally do well professionally. This leaves the reader with the impression that we need to focus on the demise of men’s jobs. While bluecollar jobs are affected by automation and have been affected by it for a long time, a new threat is identified in the books: the threat to whitecollar jobs. Disappearing Professions Although both blueand whitecollar jobs are constructed as at risk, it is particularly whitecollar work that many of the authors of the future of work books focus on. Whitecollar work is largely seen as desirable work in the books. Baldwin states, historically, bluecollar jobs were affected by automation, which means that whitecollar and professional jobs were sheltered from robots and globalisation until now. He focuses specifically on whitecollar robots that are replacing middleclass jobs. Although whitecollar robots are not yet as good as whitecollar humans, robots are simply more cost effective: Baldwin states that a whitecollar robot costs a fifth of a worker in the developed world and a third of a worker in less developed areas. It should be noted that whitecollar robots will not take over entire occupations. However, they might well take over specific tasks, which over time can reduce the need for human whitecollar workers overall. Whitecollar and professional jobs are also those that require an upfront investment into education that is then paying off with higher lifetime earnings. Benanav describes this as the idea that a good education will lead to and ensure a good middleclass job. However, the idea that a good education will lead to a good middleclass job is now changing, according to the books I analysed. Similarly, Susskind talks about the signalling entailed IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 36 in having a good college degree. Whereas in the past, a good college degree would be important to find a job and climb the occupational ladder, the good college degree has lost its significance. Frey provides a rationale why whitecollar work is well paid: he argues that in the Second Industrial Revolution, the opportunity costs for education decreased because higherlevel skills were in demand; as a result, whitecollar workers were paid well for their education. A dramatic change is now that a good education no longer guarantees that individuals can achieve and sustain a middleclass lifestyle. If middleclass workers are being replaced by technology, this is expected to have wideranging effects. Frey, for instance, presumes that the replacement of middleclass workers by machines will decrease demand for local services. Earlier redundancies are constructed as having had limited effects, as it was possible to find other professional jobs elsewhere. A central concern of futureofwork writers is thus that securing a prosperous future by investing in educational credentials no longer applies. The link between education and career is nowhere more visible than in the professions. Baldwin details that professional jobs were sheltered from globalisation and robots because they required facetoface contact, which no longer is the case. Benanav cites Robert Reich’s contention that technology is replacing professional jobs, and Schwab suggests that automation is now replacing professional workers such as accountants and lawyers rather than factory workers. The books mention professional work in finance (Baldwin, de Cremer and Schwab) and accounting (Frey, Schwab and Merisotis). However, the profession attracting the most attention is legal work, which is discussed in seven books. Baldwin provides various examples of how legal work is being automated using software, such as Lex Machina and Ravel Law, which helps to sort through information and even suggests legal strategies. This means that many tasks that junior lawyers would have traditionally done are now automated. Baldwin summarises that whereas a law degree was a secure way to ensure middleclass prosperity, whitecollar robots are now competing with junior lawyers. According to de Cremer, in the legal world, automated advisors are used to contest parking tickets, and Susskind observes that automated document review systems can scan material more swiftly and often more accurately. Susskind mentions how a law firm uses software to reduce the time human lawyers have to spend on tasks. West mentions an AIdriven bankruptcy legal assistant. Most of the books suggest that legal work is at high risk of automation, and only Frey IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 37 voices a dissenting view. Although Frey acknowledges that legal libraries are available online, he cites a study calculating that only 13% of legal tasks can actually be automated, supporting his own prediction that legal work is at low risk of being automated. The common tenor in the books is that professional jobs where education is rewarded are disappearing. These professional jobs are presumed to be held by men. This is not to say that these books are unaware that women take professional jobs. In fact, the books strongly signal gender awareness. For instance, authors are regularly citing stories of an engineer, a professor, a software developer or a surgeon who turn out to use the pronoun ‘she’. By using the pronoun ‘she’, the authors break the implicit assumption that these professional jobs are held by men only. However, what is missing from the discussions is an acknowledgement that women have made inroads into professional work. Of course, these professions, as much research has shown, are far from gender inclusive (Ely, 1995; Walsh, 2012; Lupu, 2012; KokotBlamey, 2021, 2023). However, most books fail to discuss the fact that the professions have become more diverse over the years and women, in particular, are increasingly present in the professions. By not discussing how the gender presentation has changed the professions in greater detail, readers will be left with the impression that these jobs are held by men. This is particularly the case because men in general are singled out as particularly affected by the automation, as we have seen in the previous section. A second aspect that is rarely discussed is how the professions are changing. While some authors predict the death of the professions, at least Frey seemed more sceptical in regard to what the future of the legal profession holds. However, in reality, the professions might be transformed with some aspects being done with the help of technology, while other tasks might gain more prominence. As such, the content of professional work might change, requiring substantial changes in regard to how professional work is organised and how training is structured in those firms. While most of these questions are beyond the scope of the books analysed, I will revisit some of those questions in Chapter 3. SocioEmotional Skills as Human Advantage It is evident that the dystopian images used in the future of work books I analysed relate to automation and particularly the automation of IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 38 professional jobs. Yet, some of the books also sketch the contours of a more utopian future of work. This entails that humans and machine collaborate and enhance each other’s skills. In other words, the books talk about augmentation (Kelan, 2023b; Raisch & Krakowski, 2021). Another area that the books address are those jobs that are out of reach of machines. These jobs are constructed as futureproof. Both the jobs that are augmented by technologies and those that are safe from automation share in common that they rely on a specific set of human skills. I call these skills socioemotional skills and will discuss in this section how these socioemotional skills are constructed as the key competitive advantage that humans have over machines. By and large, the skills that are said to be futureproof are those that are seen as difficult for machines to accomplish. These include leadership (de Cremer, McAfee and Brynjolfsson), teamwork (McAfee and Brynjolfsson), creativity (De Cremer, Merisotis and Schwab), coaching (McAfee and Brynjolfsson) and, in general, anything that requires socioemotional skills. I use socioemotional skills because the authors of the book have different names for such skills. For example, Merisotis cites AnneMarie Slaughter saying that the economic formation has moved from hiring hands over hiring heads to hiring hearts, where hearts are used to present socioemotional skills. McAfee and Brynjolfsson use the example that when receiving a medical diagnosis, people prefer to receive those from compassionate people rather than machines. This means that compassion is constructed as a desirable skill in humans. Susskind uses the term ‘social intelligence’ and states that technologies cannot deal with tasks requiring social intelligence well, such as providing empathy or facetoface interaction. De Cremer argues that algorithms lack what he calls social skills. Baldwin, Merisotis, McAfee and Brynjolfsson, and Schwab talk about social and interpersonal skills, Baldwin and Susskind about social intelligence, De Cremer about emotional intelligence, and De Cremer, Merisotis, McAfee and Baldwin also use empathy. Given the fact that authors use a range of terms to describe such skills, I use socioemotional skills as an umbrella term for these skills. It is socioemotional skills that are constructed as uniquely human and as hard to replace by machines. The books construct jobs that entail socioemotional skills as safe from automation. These include medicine (Frey), teaching (Baldwin and Susskind) and social work (Susskind). Another area of growth are personal IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 39 services, which entails beach body coaches, yoga instructors and Zumba instructors, which are mentioned in various combinations by Susskind, West, McAfee and Brynjolfsson, and Frey. Common to these areas of work is motivating people through socioemotional support, and the assumption is that machines could not provide this socioemotional support. It is interesting that these particular areas of work are used to illustrate the importance of socioemotional skills, while changes in the professions towards more socioemotional skills were rarely discussed. The rise of socioemotional skills is of course not new. As a matter of fact, I explored this in earlier research, where I analysed the books on the future of work that existed at the time (Kelan, 2008b). In those books, it was notable that socioemotional skills were constructed as feminine and presumed to reside in women. Yet, this was not the case in the current batch of books that I analysed for this research. The closest of such association was in relation to paid care work. Susskind and West, for instance, mention that women often work in caring professions, and Susskind even points out that caring work is often undervalued. One could now presume that care work is safe from automation, which is indeed alluded to. However, West casts some doubt on whether such jobs are indeed futureproof. West references that technology is changing care giving. This might mean that socioemotional skills might eventually also cease to be central for caregiving roles, but this was an opinion that was not predominant. Overall, the image that emerges in the books is that socioemotional skills are uniquely human. As a consequence, jobs that require socioemotional skills are constructed as safe from automation. Whereas earlier research has shown how socioemotional skills are often presumed but not rewarded in women, in these books, socioemotional skills are presented as gender neutral and as if everyone could engage in them. I will return to the question of if socioemotional skills are indeed uniquely human in Chapter 3. Creating the Human– Machine Interface After discussing the presumed importance of socioemotional skills, I would now like to discuss how machines and humans collaborate or augment one another. I will focus particularly on technical professions here because they are regularly mentioned in the books. The authors call this the human– machine interface (Baldwin, Naughty and De Cremer), which includes IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 40 machine learning engineers, data scientists and big data architects (Frey). Commonly, these areas of work appear far removed from socioemotional skills. However, as we will see, the way these areas of work are described indicates that they rely on a form of socioemotional skills. Daugherty and Wilson, in particular, describe how machine learning programmers are becoming more like teachers who train the algorithms. This is different to how programming was traditionally perceived – as writing code that a machine executes. Now, a programmer seems akin to a teacher, who teaches not children but machines. Most of the books are concerned with highend AI work. The books discuss the skills shortage in jobs related to AI, which is said to limit the development in the field (Baldwin). This means that too few humans are able to do these highly skilled jobs that are required to make AI function. However, rather than training more individuals, the solution he suggests recurs to technology. Baldwin suggests that Google’s automated machine learning, where machines train other machines, might be a solution for this skills shortage. However, Daugherty and Wilson also discuss how new technologies require other experts that have specific skill sets that one might not traditionally associate with technology work. Such experts know about human conversation, humour and empathy. Those experts can teach the technology to emulate socioemotional skills. Daugherty and Wilson provide a range of examples of such roles: a poet, novelist and playwright at Microsoft’s Cortana and a vehicle design anthropologist at Nissan. This socioemotional work for machines appears to be needed in order to design technology with which humans enjoy engaging and that is fit for purpose. To achieve this, technology must emulate humans, and according to Daugherty and Wilson, humans are best placed to enable this. This means that many emerging jobs will provide a sort of affective labour for machines. There is comparatively little discussion about the demographic background of those who design AI in the books. Schwab mentions that women hold less than 25% of IT jobs. This is problematic for him because it means that many ideas are not considered, which in turn is constructed as a hindrance for the development of what he calls the Fourth Industrial Revolution. However, apart from that, the lack of diversity in regard to gender and other dimensions of difference is rarely discussed. This is notable because IMAGING FUTURES BETWEEN AUTOMATION AND AUGMENTATION 41 if there is a skills shortage, it is commonly claimed that women should be mobilised to take these roles, particularly if they involve a socioemotional skills component. However, such suggestions are not commonly made in the books. Another area that seems less well discussed in the books is the work in AI supply chains. In contrast to the much indemand designers of AI, jobs that involve preparing data for machine learning through, for instance, data labelling or data annotation, are rarely discussed. The notable exception is the book by Daugherty and Wilson, which references the work that can easily be outsourced or crowdsourced, such as training AI. We will return to how such types of work intersect with gender in Chapter 5. Although the skills shortage in relation to AI work is regularly invoked in the books, there are also signs that the traditional jobs associated with creating emerging technologies are changing. It is suggested that parts of programming jobs can be automated by machines programming themselves – or rather using tools that generate code. As mentioned before, programmers are presumed to become like teachers or trainers. Additionally, experts on socioemotional skills will be required to teach machines how to communicate with humans. However, we have also seen that the gender composition of these jobs is often neglected. Similarly, less highend work such as that required to prepare data for machine learning is less regularly the focus of attention. Gendered by Design Gender and technology are not at the core of any of the books, but some of the books do provide interesting illustrations of how gender is entered into the debate. Schwab, in particular, discusses what is akin to technology being gendered by design. Schwab suggests that how machines are programmed and how they interact is impacted by sexism and racism. In his view, robots, particularly humanoid robots, are no longer bound by race and gender when they are designed. Yet, customer service robots often display female characteristics and industrial robots feature male characteristics. In Schwab’s view, rather than challenging old stereotypes, the same stereotypes are repeated. He argues that the wellbeing of all individuals would be increased if more conscious choices are made during the development of technologies, which would then avoid repeating the same stereotypes. UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 48 chapter, I discuss how technologies allow for such training to be conducted in new formats such as in VR. Whereas Chapter 2 focused on books that discuss the future of work, this chapter is based on the interviews (further details can be found in the appendix). In the interviews, I asked interviewees about changes that they perceive in relation to the future of work and technology and how they saw issues such as around automation and augmentation. The underlying material for the chapter is in nature very different to the books. Whereas the books presented carefully argued and evidenced ideas that had undergone normal book publishing processes, the interviews represent occasioned answers. While the answers were naturally less polished than text published in a book, they provided a particularly rich canvas to analyse discursive patterns that are regularly mobilised around the future of work, technology and gender. The answers often reflected the experiences of interviewees, including reflections on changes that they had perceived in the workplace. Some talked about new ways of working, whereas others discussed technologies they had developed. In some cases, I was able to try some of those technologies myself. I was thereby able to add an additional facet to the discussions through my own experience with these technologies. This chapter thereby questions in how far socioemotional skills are uniquely human and out of reach for machines. I first show how drudgework is expected to be automated because it follows repeatable patterns. I then outline which consequences this might have for task profiles and structures in professional work, before showing how socioemotional skills are constructed as a competitive advantage for humans over machines. I then query if socioemotional skills are uniquely human by showing how socioemotional skills can be understood as patterns that can be automated. While it is technically possible to automate socioemotional skills to a degree, such an automation of socioemotional skills might not be deemed socially desirable. Drudgework as Repeatable Patterns A common theme in the interviews was to suggest that machines can do manual, repetitive and mundane tasks, which will free humans from the drudgery of work. Interestingly, drudgery harks back to one of the language UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 49 roots of the word robot (Capek, 1920; Oxford English Dictionary, 2023e) (see Chapter 1), although none of the interviewees made this connection. In many ways, interviewees were less concerned about physical tasks being replaced. This might in part be due to the fact that physical labour being replaced is a common feature of discourses since the Industrial Revolution. There was a clear expectation that physical tasks could be easily done by machines. Lucy asserts that machines can easily do tasks that are purely based on physical power rather than human thought. It was discussed that technology could displace manual labour in assembly line work or factory type work. In contrast, interviewees were less concerned about work in warehouses being automated. Felicia, for instance, stated that these are workplaces where humans are treated like machines. Similarly, Myra talked about how warehouse workers are told by a bracelet what to do. For her, this means being treated as a human robot. What Felicia and Myra articulate is the fact that jobs being automated is not necessarily problematic because in many jobs, people are already treated like machines. In a sense, it is a continuation of automation that has been affecting largely jobs in manufacturing and warehouses. However, due to the quality of some of those jobs, particularly in warehouses, it was not considered as particularly problematic that these jobs might fall away. Since I was mainly interested in professional work, most interviewees reflected on how professional work will change. Interviewees regularly referenced tasks that could be automated. In particular, interviewees focused on repetitive tasks that were singled out as ripe for automation in professional work. Prime targets for such automation are what William, who works in a law firm, called drudgework. He saw this drudgework as ready for automation. Interviewees like Yoshiro and Howard talked about how manual, repetitive and predictable tasks would be automated. Ralf mentions that these tasks might not be simple but can also be sophisticated, as long as they are based on repeatable patterns. If drudgework is disappearing, William suggested that humans can focus on higherend and more interesting work. Peter describes this as people being able to focus on cognitive tasks that are more ‘challenging, interesting and developmental’. Another way to express this sentiment was to suggest that drudgework – or what Anastasia describes as boring work – takes away from creative tasks. A common example in architecture referred to how bathrooms are placed. Yoshiro recalls how in his early days in architecture, he was copying and UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 50 pasting bathrooms into residential layouts – a task that he found dull and boring. This is no longer needed today. Anastasia similarly states that it is no longer required to draw mundane things like bathrooms and car parks because there is already a myriad of layouts available that one can use and adapt. In such contexts, it makes little sense to create 30 different layouts afresh. According to Yoshiro, patterns in architecture are well established. It is equally well established how a onebedroom apartment should be structured, which means that there is no need to reinvent such spaces. The reason why such work is no longer needed is because as Yoshiro states architecture is based on patterns that are repeated over and over again. In order to automate tasks, these patterns have to be made explicit by transforming the knowledge from architects into rules that can be used to create algorithms. Caleb talks about the idea that architecture is about codifying aesthetics by creating a rulebased model that is predicting design options. These algorithms then encapsulate the knowledge, experience and the signature styles of architects, and the predictions such a system makes replicate a specific architectural style. It is notable that other professional contexts brought up similar illustrations of automation. For instance, interviewees often reflected on how legal work changed through the years. Beverly comments that early in her career, she spent hours going through folders of documents with a pen, which is no longer done anymore. William similarly comments that lawyers no longer have to go through books because there are online legal databases they can use. Tessa talked about how lawyers use templates and have technology that changes clause numbers and definitions of a word automatically. In a bank context, Zac talks about the manual labour of putting numbers into spreadsheets, which is done automatically today. In auditing, Peter mentions that drones are checking stock in warehouses, meaning that auditors no longer have to travel to a location to make these checks physically. Duncan recalls how his organisation started to automate adding tax codes to long spreadsheets. These activities were normally done by junior accountants who went through twothousand lines of a spreadsheet, adding tax codes manually. One could imagine that this work is rather tedious. When Duncan’s organisation implemented technology for this task, he expected an uproar because the technology was taking away a huge chunk of the jobs that junior accountants were doing. However, Duncan recounts how the junior accountants were delighted that they no UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 51 longer had to go through lines and lines of spreadsheet due to the tediousness of this task. Drudgework had been automated. However, many of the interviewees also stated that such change in regard to technology is not new but forms part of changes that they have experienced in their lifetime. Victor, a consultant, said that consultants have been augmented by technology for a long time. He recalls how in the 1990s, Lotus Notes augmented the intelligence of consultants. Anastasia comments that in architecture, every new technology such as computeraided design led to suggestions that architects will become redundant, and she sees the same arguments in relation to AI and machine learning. Beverly, a lawyer, mentioned that the 17 years of experience that she built up in her area of expertise have been replaced by generative AI. Beverly frames this as her expertise disappearing. Although one might presume that this is scary, Beverly seems less concerned about it. Overall, there was less panic about the changes in professional work in the interviews compared to the books. A common perspective was that while some jobs will disappear, others will appear. Anastasia articulates this as for every redundant job, there are 20 other new jobs emerging. Shifting Tasks and Structures It was widely acknowledged that tasks and structures of professional work might change. A common theme in the interviews was to talk about the changing tasks and structures in professional work. For instance, Yoshiro imagines that architects of the future will be ‘shepherds of algorithms’, which means that part of the professional task of the future will be to develop patterns that are coded. This constitutes a major change in regard to how architects do their work because a key part of the job will be to codify aesthetics into technology, to echo Caleb’s earlier statement. Other changes in the task profile refer, for instance, to how time is spent. Xerxes uses the example of how a programmer would traditionally need two hours to write a piece of code, but if the programmer uses ChatGPT, the same task can be done in five minutes. Victor talked about how management consultants produce a lot of PowerPoint slides to communicate. In the past, he said that a consultant might spend 15 minutes to gather information and 45 minutes to ‘massage’ the slide to develop the right chart or graphic. That has now changed because consultants now spend the majority UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 52 of time on deep thinking and the story that they want to tell; the creation of the slide itself is swift because there are automated tools that can be used. He maintains that the automation of slides will not make consultants redundant because the human is augmented but not replaced by technology. Yet, Victor also acknowledges that there might be a timesaving component, which might in the long run affect the hourly pay structure that many professions employ. While Victor might have spent one hour to create a slide in the past, he would only require 30 minutes today and might create two slides instead of one in an hour. Yoshiro observes a similar timesaving component of technologies in architecture, where work that would have required four hours in the past can now be done in four minutes. Yet, clients are still charged for four hours. Beverly draws on a strikingly similar example, saying that work that would have taken a lawyer four hours to complete can now be completed in 30 seconds or less. She acknowledges that this will be a challenge to the normal charging structure by the hour that many professional firms use. If a task now only takes minutes, why would clients pay the high hourly rates of professionals? This questions the traditional hourly pay model employed in many professional firms. New technologies also open up the possibility of changing how work is done. In particular, generative AI is seen as a game changer for professional work. Victor talks about how junior consultants use ChatGPT to bring themselves up to speed on specific client contexts swiftly. In the past, a consultant who is unfamiliar with how a bank operates might read Wikipedia and articles on financial blogs. Today, they will ask ChatGPT. Victor states that this will not make them an expert but it helps them to understand what a treasury department at a bank does. If the technology is a bit wrong about such basic information or even ‘hallucinates’, it does not matter greatly because it would not go into a report to the client but rather provides background knowledge to allow the consultants to understand a context swiftly. Victor is less concerned that management consulting could be replaced by generative AI because he likens the output of ChatGPT to how his teenage son would answer a question. It is a basic answer but lacks the level of sophistication and rigour that a client would pay ‘big money’ for. In general, there were many examples of how ChatGPT and generative AI more generally are used in the professions. Anastasia talked about a marketing company that uses ChatGPT to generate pitch ideas and they then try to beat the system by coming up with better pitches. Beverly mentioned UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 53 that in her law firm, they spent a lot of time on ‘prompt engineering’ to ensure that the responses from generative AI match what is required. William describes the use of generative AI as a creative sparring partner or a foil for creative thinking; it is used to develop ideas with a machine but it does not replace the human. Although generative AI might automate tasks in professional work, generative AI is mainly seen as a tool to condense information and to provide ideas, which augments professional work. The timesaving component of technology can not only upend the charging structure in the professions but it might also require different organisational structures. Many professional firms are organised in a pyramid structure, where large numbers of junior professionals are hired and then compete to become one of the few partners. Victor talks about how there are thousands of young people in their 20s manually going through documents and circling items in Big Four firms – the largest professional service networks, namely Deloitte, EY, KPMG and PwC – when preparing taxes. Victor suggests that these thousands of young people might not be required in the future. Similarly, Beverly mentions that the pool of junior lawyers is going to be shrinking as a consequence of drudgework no longer being required to the same degree. There was also a sentiment that if junior professionals no longer have to do drudgework, they miss out on learning. Victor talks about how early in his career, a lot of learning came through doing boring stuff, which allowed him to build up mental ‘muscle memory’. He describes ‘muscle memory’ as a reflex to know what might be wrong in a situation. Caleb talks about a similar moment in regard to architecture, where the experience of having confronted a problem in a previous situation helps architects to solve a current problem. Yoshiro also mentions how architectural patterns became for him second nature through repetition and working closely with other architects. While it was common to see drudgework as a form of learning, which might no longer be possible, Anastasia presumed that technology can in fact help junior architects to develop knowledge to do the job well. She argues that technology can assist junior architects in building what she calls an intuition. Such an intuition would in her view take years to cultivate otherwise. In a sense, she argues that the intuition developed through drudgework can be replaced by the coded experience in technology. Professional work is often based on an apprenticeship model where junior people learn by observing others. William calls this osmotic learning. UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 54 According to Victor, it is not just doing manual tasks over and over again that constitutes learning in the professions, but also sitting in thousand conference rooms to understand what makes different executives tick. Various interviewees talked about how much time junior lawyers spent observing more senior lawyers to see how they deal with clients and how to tackle specific issues. A traditional argument is that if more tasks are automated, reducing the need for junior lawyers, they will have less chance to observe how to do their jobs once they become more senior. In many ways, the pandemic provided a glimpse at some of the problems that the lack of shadowing senior colleagues might have in the future. Due to the fact that work was done remotely, many junior lawyers missed out on the opportunity to observe the daytoday requirements of what it means to be a senior lawyer. Tessa explained this as follows: whereas before, junior lawyers would sit close to a senior lawyer and overhear how the senior lawyer takes urgent client calls, during the pandemic, the junior lawyers were often not included in ad hoc client meetings because the client either called on a mobile or the video meeting link was not circulated to the junior lawyers. Before, junior lawyers would soak up knowledge through observation; they were now excluded from such processes because they were no longer physically present in the space. William talks about how even arranging a debrief after an online meeting with a client has the additional hurdle of setting up another online meeting. If, structurally, much of the repeatable tasks are done by machines, leading to fewer junior professionals being required, this could mean that the progression from junior to senior professional and the entire structure of professional firms might need to change. There are also wider changes to the professions that the interviewees imagined. For instance, Beverly likened the changes in the legal profession to the reformation where people get access to something that was previously closely guarded. She describes this as a democratisation of legal knowledge, which in her view is a positive development. Yoshiro echoes this point when he suggests that due to professionalisation, knowledge became protected and a charge is due to access this knowledge; for instance, consultants are charging for their knowledge and knowledge is sitting behind paywalls. He argues that technology means that such knowledge could be accessed differently. He suggests that making such knowledge – which in his view lies in patterns in architecture – more widely accessible would be beneficial. There might also be benefits for smaller firms, as Anastasia hinted UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 55 when suggesting that technology can level the playing field to allow smaller architectural practices to compete with larger ones. Overall, these changes in tasks and structures suggest that what humans might do in the future of work changes. I will trace this in the next section. The Human Advantage We have seen so far that in professional work, drudgework is seen as automatable, and while that brings certain challenges for tasks and structures, the overall sentiment in the interviews was that technology is augmenting work. Viewing these changes more holistically, some interviewees commented how automating drudgework would allow people to spend their time differently. Jeffrey likened this emerging future as akin to Star Trek, the scifi franchise; Jeffrey suggests that in Star Trek, humans can dedicate themselves to exploration and creation, which in a sense would free individuals to engage in other activities. Gabrielle said automation will free up time for humans to do other things, which, as she points out, might not only relate to the public sphere of work but also entail other activities like caring for family members. Most thinking on what humans might do centred on the idea what machines currently cannot do, or where the use of machines would be undesirable. Those areas largely required social interaction. For instance, Ralf suggested that anything that works based on relationships, such as sales or customer service, will require humans. Bank tellers are a profession that is often mentioned in relation to technological change. For instance, Bessen (2015) discusses the assumption that with the introduction of ATMs, bank tellers might disappear. However, rather than disappearing, the tasks that bank tellers engaged in changed (Bessen, 2015). Similarly, Oscar, who works in a bank, references bank tellers because their jobs are seen as under threat by digitalisation. However, Oscar was of the opinion that much of the work that bank tellers would have traditionally done in a branch would in the future be moving to the back office. Bank tellers would likely be dealing with similar issues as today, but the interaction is facilitated by digital technologies. This speaks to the point that personal relationships will require a human touch. Jeffrey talked about other areas where human input would be required, such as in bereavement counselling. Rather than HR sending an automated message, he would expect that a human conversation is required UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 56 in such situations. Kenneth also talks about empathy that is, for instance, shown in healthcare as being hard to replace. In a similar vein, Ben suggests that since tasks that can be done by a machine will be done by a machine in the future, this leaves soft skills to humans. This includes understanding and managing the self and others. Ben presumes that humans will be in demand for those soft skills. It is perhaps not surprising that socioemotional skills are singled out as particularly important in regard to professional work. For instance, Victor talks about the importance of ‘reading the room’ for a consultant. A consultant presenting to a client will notice if the CEO is nodding but other executives are sceptical. A consultant would then know to schedule extra meetings with the sceptical executives to bring them on board. He suggests that technology cannot do this interhuman work. Similarly, Victor talks about the importance of executive assistants who he describes as the social glue. While many of the tasks that executive assistants do, such as scheduling meetings, could be automated, executive assistants do much more than this: they know what is happening in the office, they know the politics and they read every email, which gives them additional social knowledge that a machine could not replicate, according to Victor. This also has consequences on how professional firms are likely to hire in the future. Beverly reflected about what type of people a future law firm might need to recruit. She suggests that in the future, law firms need people who are better at listening to clients. Beverly acknowledges that generative AI can provide many answers but that a lawyer has to understand the client. The lawyer needs to understand if the client is more riskaverse or open to risk, which will inform the legal strategy suggested. She says that as a managing partner in a legal firm, she would be looking to recruit individuals with listening skills, or what she calls an ‘emphatic lawyer’. The notion of the empathic lawyer encapsulates the idea that socioemotional skills are constructed as vital in the future of work. In a similar vein, William talks about how working with clients requires an understanding of the culture and the power dynamics in the organisation. He talks about how an experienced lawyer has to understand the culture and the power dynamics that happen in this context, which is used to solve problems and find a consensus. This is not legal knowledge as such but what he describes as tacit knowledge that people have akin to a gut feeling or an intuition. UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 57 The overall idea that emerges from the interviews is that if parts of professional work are automated, then socioemotional skills become a competitive advantage of humans. This is largely to the fact that socioemotional skills are constructed as uniquely human. In other words, those skills either cannot be performed by machines or it is socially not desirable that those skills are performed by machines. By focusing on socioemotional skills as uniquely human, the interviewees also suggest that machines will not replace human labour completely. The interview accounts were, by and large, hopeful. Drudgework is handed over to machines and humans can focus on interesting work, and work that requires socioemotional skills. It is here that humans shine because they can do things that machines are perceived as unable to do or where the use of machines is undesirable. The Gendering of SocioEmotional Skills Traditionally, discussions about socioemotional skills were gendered. Stereotypically, it is presumed that women are particularly good at and wellsuited for displaying socioemotional skills. Such ideas are, for instance, drawn upon by CEOs, who justify gender equality by referring to such gender essentialised skills (Kelan & Wratil, 2021). These stereotypes have consequences in the workplace in that women displaying socioemotional skills are often not rewarded for them; the assumption is that women simply do what comes naturally to them (Fletcher, 1999; Kelan, 2008a). Yet, in the interviews like in the books, those socioemotional skills were rarely gendered. The only example of a person linking socioemotional skills with gender came from Duncan. Duncan argues that women should find it easier to stay employed when automation takes hold. He admits that he has no specific data for this assumption but in his observation of 20 years, he noticed that women are better than men at persuading people to adopt their ideas and at building teams and communities, which is central in professional services jobs. Duncan here establishes a link between gender and skills in that he argues that women are better at building teams and persuading others. He suggests that these socioemotional skills might give women the edge when it comes to skills required in the workplace. Socioemotional skills were loosely linked to gender in relation to care work. Gabrielle talked about how with an ageing population, there is increased demand for healthcare workers. Since women are overpresented UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 64 tricky one. I found looking into the eyes of many of the avatars was a rather strange experience. One clearly notes that the avatars are not real, and it feels odd to look them in the eyes. Based on that feedback of a lack of eye contact, I decided to stare at the avatars’ eyes. In a reallife interaction I am sure my staring eye contact would have been perceived negatively. Yet when I stared, I still got the feedback that I do not make enough eye contact. I wondered in how far the technology tracking me might not be as accurately calibrated as it should be. Rather than analysing the context of talk, the feedback I received in VR was purely on the delivery of the talk. In another situation, I got additional feedback that I need to look more to the right or the left. One of my favourites was a unique score that I got in some of the exercises. This unique score, as the app told me, was measuring if an 11yearold would understand me. I found this to be a rather odd metric because all the trainings were designed to be in professional work contexts rather than schools, and as such, the interaction with 11yearolds might be rather limited. I also wondered how the score came about. Had people been recorded before and an 11yearold had listened to the recording to check comprehension? As such, the feedback that such apps provide was in my experience rather mixed. Some of it was helpful, whereas other things kept coming up even when I did specifically that in the next reiteration. Much of this technology is still in infancy and feedback will likely get better over the years. I also wondered in how far such technologies could be reliably chosen, for instance, to select candidates. If the technology does not understand me, I am sure that other people with accents would also struggle. If the technology could not track my eye movement accurately, I am not sure if I would be selected for a job that requires such a connection. Hands as Liminal in VR VR training is designed to transport the learners into a new space where they can practice new skills. Immersion is thus key. One common assumption is that immersion requires a highly realistic scenario where simulated people appear real. For all VR apps I used, that was far from being the case. As I mentioned before, it was clear to me that the avatars I interacted with are not real people. Their eyes would often look particularly weird and they lacked any facial expression that I valued, for instance, with Viola, the UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 65 technology from Soul Machines that I tried. Viola only appeared on my computer screen rather than in VR. However, it was notable that in VR environments, efforts had been made to give the avatars different voices by those who created them, and I appreciated that some characters had accents that sounded either French or African. Although attempts had been made to make the scenarios realistic, initially, I thought it would be hard to imagine that one can immerse oneself in the environment. Others I spoke to had similar concerns. Franklin, for instance, recounts a story of how one of the participants of a VR training was highly resistant to undergo health and safety training in this new format. He tried to convince his colleagues that they should all refuse this training. However, he could be persuaded to give it a go. He put the headset on and started the simulation. At one point, he flinched – in the VR scenario, a nail had gone through his hand. Even though this person had been sceptical about the training, having the experience of a nail going through his hand shocked him. He realises that what the training offers is something more than a theoretical understanding of what can happen if you do not adhere to health and safety standards. Franklin describes this an ‘emotional impact’ in the interview where the nailthroughthehand scenario created additional learning that at least according to Franklin is difficult to achieve in another way. While doing VR training myself, I was regularly invited to pick an avatar. On most platforms, I could select from a set of preset characters. There are commonly less modification options than I had experienced building an avatar for one of the virtual recruitment events I attended, where one had a lot of choice in how one presents oneself, including the ability to wear flipflops in recruitment simulations. In VR, however, it seemed more common to pick an avatar that was preestablished. There was less of a temptation to build an avatar that might correspond more closely to how you might appear in real life. Instead, I was offered the option to appear as a Latino or a Black woman. In most cases, I forgot which avatar I had picked because it was fairly inconsequential for most of the VR interactions I engaged in. Yet, at several times during the experience, I was shocked when my hands looked different in VR than in real life. I noted in my field notes that I was surprised when my finger nails had bright red nail polish on. Together with the example that Franklin recounted earlier, it appears that hands occupy a liminal space between the physical and the virtual world. During a VR experience, you not only have a headset on that blocks out UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 66 the physical environment in which you are situated and replaces it with a virtual environment, but you also hold controllers in your hands, which allow you to drive the action by clicking on certain elements. As a matter of fact, during the initial familiarisation with VR, I learned how to manipulate virtual objects by using the controller. For instance, I would learn to lift an object and then throw it. Your hand movement is also used in some metrics to analyse, for example, your body language. If you cross your arms, the controllers will also be crossed, which is read as a closed body position. While in other VR environments, the body moves around more dynamically, in most of the training in VR I completed, I was stationary, either standing or sitting, but rarely moving through space. Glancing at my hands in VR was thus one of the few instances where it became clear that my embodiment as a virtual avatar was different. This often led to a moment of surprise akin to what the person who had a nail go through his virtual hand must have experienced. My VR experiences did not include a nailthroughthehand scenario like Franklin describes, but it is not hard to imagine based on my own experiences that there is a moment of shock and surprise. The hands seem to be liminal boundary between a sense of self as a person and the virtual environment in which one finds oneself. Perspective Taking Apart from health and safety applications, VR was presented as particularly useful to help humans to practise socioemotional skills that are constructed as important for the future of work. Matt expressed this by referencing what he calls an ‘old adage’, that one must walk a mile in someone else’s shoes to comprehend their life situation. He explains that this is particularly useful for diversity and inclusion because VR allows you to try on someone else’s shoes. VR allows you to take someone else’s place and make an experience that you might normally not have. Matt provides the example of taking on the position of a woman colleague and play through a scenario from her perspective. Matt argues that this helps with taking perspective and is more impactful than watching a video. He says that by being in VR in this situation, you might ‘feel your blood boil’ because you are treated unfairly. This suggests that VR allows access to emotions that one normally might not experience, which in turn can help to develop empathy for others. UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 67 Franklin offered another strikingly similar explanation as to why VR works well in regard to diversity and inclusion training. Franklin describes how VR training allows you to experience what it feels like to be marginalised. He argues that for many people who did not have such experiences before, it is tricky to understand what marginalisation feels like. This is said to limit their potential to take action on it. Franklin suggests that VR training gets people over the hurdle to understand what it feels like to be in such a situation and how to respond to that. He likens it to the difference between an academic explanation and the practical experience. For Franklin, if you experience being the odd person out or that your ideas are not appreciated, people develop what he calls their soft skills. What we see here is that technology is used to allow people to make different experiences to develop socioemotional skills. In regard to how such training is delivered, I had the opportunity to participate in several diversity and inclusion trainings in VR. In one scenario, I met Steve, who treated Sandra in sexist ways. One of the exercises involved pressing the controller in my left hand for inclusive behaviours and the controller in the right for exclusive behaviours. Following this, I was presented with a range of sentences I could pick from to advance the conversation. Depending on my choice, I was given feedback if that was a good or bad choice. One element I noticed is that when I talked with Steve, he seemed to mirror my behaviour – if I was aggressive, he also became aggressive. This mirroring is what actors who are engaged to perform diversity and inclusion scenarios in person have also been asked to do. I did not expect this to be a feature of the VR training. The key learning of the training was to allow Steve to come up with why his behaviour is problematic himself rather than telling him what was wrong. The final part of the scenario involved me recording a closing statement in which I addressed Steve directly and outlined what we had agreed in terms of the way forward. Then I changed avatar and was in Steve’s position. I saw the avatar that I had chosen before deliver the speech that had just recorded back to me. As Steve, I would notice how it felt to be at the receiving end of such messages. I could then shift back and rerecord the message. The idea was that I notice myself that something was too harsh or not clear and that I could then correct that in the next reiteration. This form of selffeedback was rather useful. Such retakes are not possible in real life and are tedious in physical training, but they are a key feature of VR. I can UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 68 practise as much as I want. It obviously relies on the trainee to spot things that do not work well. Role plays and other theatrebased methods are regularly employed in diversity and inclusion training, often making use of actors or putting employees in different roles. They can be rather effective for perspective taking. However, VR allows for this to be elevated. Rather than enacting a situation with a colleague or an actor, which is artificial in itself, VR allows participants to engage in such exercises on their own. Seeing the world from a different vantage point can be facilitated by VR. This is essential to develop empathy and the socioemotional skills that are constructed as the human core advantage. Yet such training is delivered by a machine. Of course, humans design the technologies, but the actual training of socioemotional skills is machinefacilitated. Machines are thus able to train socioemotional skills in humans, complicating the idea that socioemotional skills are out of reach of machines. SocioEmotional Skills as Patterns So far, I have shown that machines are able to mimic and train humans in socioemotional skills. Since socioemotional skills follow repeatable patterns, socioemotional skills can be automated. Like many of the repeatable patterns that can be automated in spreadsheets, legal documents or presentation slides, socioemotional skills follow patterns that can be transferred into code. Such processes of transferring emotions into computer patterns are subjective, and how subjective assessments are transformed into objective and universal assessments will be at the centre of Chapter 5. While these processes are deeply problematic, for the purpose of this chapter, the fact that socioemotional patterns can be automated casts doubt on the construction of socioemotional skills as uniquely human, and as such, as a core competitive advantage of humans over machines. If and to what degree socioemotional skills will be automated depends also on what is deemed socially acceptable. For example, it has been shown that having machines involved in childcare is technically possible but is deemed by experts in the field as problematic for social reasons due to implications for children’s development and privacy concerns (Lehdonvirta et al., 2023). Therefore, it is possible that socioemotional skills at work will continue to be completed by humans. However, this might not be due to UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 69 the fact that socioemotional skills cannot be performed by machines but rather due to a desire to have humans perform those skills. However, there is potential to use technology to train individuals. For instance, the empathetic professional, to paraphrase Beverly, might well be trained by machines to understand how to better support colleagues and clients. If the patterns that constitute drudgework can be embedded in machines to give professionals a substrate of the experience of many years of professional practice, then this is certainly possible in relation to socioemotional skills as well. When Anastasia talked about how technology might provide junior professionals with an intuition that is trained on previous patterns and thus provides access to professional experience, this intuition might also extend to how to display socioemotional skills. Conclusion In this chapter, I questioned in how far socioemotional skills are a core human advantage for humans over machines. I illustrated that most interviewees expected drudgework that follows repeatable patterns to be completed by machines in the future. In professional work, this meant, for instance, that individuals would no longer go manually through long documents but that technology could do that work. It also involved using technology to learn and to create outputs. If parts of professional work are automated, this changes the task profile of professionals and is likely having an impact on how professional firms are structured. In particular, this might affect the need for large groups of junior people who engage in much of the drudgework. However, rather than seeing the professions as disappearing as a consequence, interviewees talked about how this might democratise professional knowledge by making it more widely available. It was also stressed that professionals will require a different skill set, such as being empathetic. It is interesting to note that like most of the books, the interviewees largely resisted the idea to construct socioemotional skills as something women are good at. This constitutes a departure to how socioemotional skills are often talked about. Empathy, as well as other socioemotional skills, were regularly constructed as uniquely human and thus out of reach of machines. In this chapter, I questioned in how far socioemotional skills are indeed outside of the reach of machines and thus constitute a human advantage. I first UNIqUELY HUMAN AND AUTOMATABILITY OF SOCIO-EMOTIONAL SKILLS 70 looked at how interviewees talked about care work and automation to then focus on how machines emulate emotions or train emotional responses, including in VR. I have suggested that even though the experiences in VR appear artificial, they might be wellsuited to develop socioemotional skills in participants. The future empathetic professional might well be trained in VR. The chapter questioned if socioemotional skills are uniquely human and as such constitute the core competitive advantage of humans over machines. However, while socioemotional skills are automatable to a degree, it is questionable if this is socially desirable. Notes 1 As a side note, when Jeffrey and I spoke, it was during one of the Covid lockdowns and schools were closed. While Jeffrey was speaking with me, his daughter was sitting at the same table doing the work the school had given her but also listening in on the interview. Jeffrey could not remember the name of the seal and only after his daughter searched for it online, she reminds him that it is called PARO. 2 I use shopping cart and cookies here because the VR experience was situated in the United States. This chapter has been made available under a CC BY – Attribution license. DOI: 10.4324/9781003427100-4 4 ALGORITHMIC BIAS AS ULTIMATELY FIXABLE Introduction Finding people with the right skills to do jobs is centrally important for organisations to function well. Hiring also often follows specific patterns such as identifying skills that are needed in different functions in the organisation. Yet hiring is complex and timeconsuming, making hiring an ideal area to deploy technology. One cannot fail to see how for an organisation, it must be appealing to use technology to determine which candidate is best suited for a role. In many ways, using technology to recruit people follows the idea that something as perceived subjective as hiring can be transformed into an objective process (Kang, 2023). However, the processes through which this presumed objectivity is achieved are far from unproblematic (Kang, 2023). Technology also has the potential to help with another aspect of the hiring process: the bias of those who select the candidate. For a long time, recruitment has been impacted by human bias and technology offers the possibility to reduce this bias (Feloni, 2017; McIlvaine, 2018; Riley, ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 72 2018). Therefore, AI in hiring is a technooptimist’s dream in that processes of an organisation are improved through technology. However, this technooptimism is in many ways dampened by the fact that AI has been shown to repeat and amplify bias in hiring (Dalenberg, 2018; Vassilopoulou et al., 2024; Kelan, 2024). The media is regularly pointing to dangers associated with AI in hiring. Amazon’s failed attempt to use AI in hiring functions is the standard example used in the media (BBC News, 2018; Dastin, 2018) and beyond. In this case, the pattern that the AI identified and repeated through predictions was exclusionary. If AI is amplifying biases then this raises serious questions if using AI in hiring can indeed fulfil this technooptimistic dream. In this chapter, I question how a technooptimist’s stance that entails that AI improves business processes can be reconciled with the existence of algorithmic bias in hiring. I suggest that this is achieved by constructing algorithmic bias as ultimately fixable. Technooptimism as a stance was supplemented with the perspective of technohesitation. Technohesitation is not a rejection of technology as such. A rejection of technology or an acknowledgement that technology can lead to more harm than good would be akin to the stance of technopessimism. Instead, this stance presents a hesitation that could be dissolved once there is more societal acceptance of and confidence in the use of AI in hiring. Hiring the Candidates that Best Meet the Profile Although all parts of human resources can potentially be impacted by digitalisation (Cheng & Hackett, 2021; Tambe et al., 2019), hiring has been an area at the forefront of being transformed through digital processes (Eubanks, B., 2018). Reasons why hiring is a prime candidate for digitalisation are due to the repetitive nature of the process and the potential reduction of mistakes that digitalisation offers (Eubanks, B., 2018). In addition, digitalisation can broaden the candidate pool, make the hiring process more efficient, lead to higher job tenure, and make the process quicker and reduce costs (Black & van Esch, 2020; Hoffman et al., 2015; Johnson et al., 2021; Tippins et al., 2021). Digitalisation can be used in a range of processes in the hiring funnel (SanchezMonedero & Dencik, 2019; SánchezMonedero et al. 2020). However, there are certain aspects in the hiring process where the use of ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 73 digital technologies is more common, such as screening candidates (Albert, 2019). For instance, a candidate might join a virtual careers fair to meet an employer. Before submitting an application, the candidate can be invited to engage with a chatbot to check if basic requirements for the job are met. If so, the candidate might be invited to submit a CV, which is then checked based on certain keywords. It has been noted that some candidates attempt to game the system by, for instance, submitting a CV that includes references to elite universities like Oxford and Cambridge in white text that is invisible to the human eye but that would be picked up if an AI searched for keywords that include such elite universities (Buranyi, 2018). If a candidate’s CV is selected, candidates might then be invited to participate in a number of simulations and games to test their skills and abilities (Tippins, 2015). In case candidates proceed further, they are commonly asked to participate in asynchronous online interviews where candidates might use a mobile phone or computer to record themselves answering a series of questions designed to see if the candidates are a good fit for the position (Köchling & Wehner, 2020; Albert, 2019). The hiring process is expected to be changed significantly by digitalisation, but many of the underlying principles of hiring will apply to digitalised forms of hiring as much as they do to nondigitalised forms of hiring. Human resource practitioners often rely on guidelines such as the ‘Uniform Guidelines On Employee Selection Procedures’ (Biddle Consulting Group, 2023) to guide the hiring process. Before starting a hiring process, a job analysis is commonly conducted in which knowledge, skills, abilities and other characteristics (KSAOs) are identified that are required to do the job well. Then assessments are developed that assess the candidates against the KSAOs. This is ensured through a variety of validity tests that are conducted. These entail criterionvalidity or, in other words, that the selection procedure predicts job performance; content validity, which means that what is being assessed is indicative of doing the job well; finally, construct validity, which shows that the data collection is indicative of how well the KSAOs are matched by the candidate (Biddle Consulting Group, 2023). If hiring is digitalised, a similar process should be followed, apart from the fact that technology is more central in it. Historically, a candidate might have been invited to complete a penandpaper assessment, which is evaluated by humans for fit with KSAOs. Now, candidates might be invited to complete various games online to assess to what degree candidates ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 80 which inequalities might potentially increase, and as such, if the reduction of inequalities is seen as of positive value, then technooptimism would not be necessarily a stance one can uphold. As such, my interest was on how people negotiate being technooptimists with algorithmic bias. Humans as Making AI Biased Interviewees would commonly reference the fact that algorithmic bias only exists because humans have biases and AI learns such biases from humans. Lucy suggests that algorithms themselves are not biased but that the data that algorithms are trained with is. For Lucy, algorithms are just ways to run numbers, and they reinforce bias because they have been fed biased data. Franklin stated that humans are biased and that AI is amplifying this. In other words, the bias is already there and AI just makes it visible. Anton describes algorithmic bias as a human problem. He justifies his view by stating that human bias becomes manifest in data sets and in the machine learning process, those biases are learned by a machine. Anton suggests that AI simply copies biases from humans. One might presume that such algorithmic bias thus dampens the optimism for AI, but Anton in his next discursive move suggests that AI provides the unique opportunity to make human biases visible in a systematic way, and often for the first time. He acknowledges that biases existed before but they were hidden and AI makes them visible, tangible and, most importantly for Anton, addressable. Thereby, the stance of technooptimism is retained by arguing that algorithmic bias is beneficial in so far as it makes human bias visible through technology. Howard made a similar argument. Howard argues that machine learning picks up the patterns from human behaviour and amplifies them, but it can also put a spotlight on those biases. Howard suggests that companies can use AI and machine learning to show them where they have discriminated in the past by analysing historic data. As such, patterns of discrimination can be made visible, which by consequence would be good for organisations. The discursive moves that Howard and Anton display are strikingly similar in that negative connotations of algorithmic bias are turned into a benefit of using AI. A variation of this stance is that humans are biased. Henry talked about how humans are biased and therefore produce biased data. He states that recruiters and hiring managers often believe that they are excellent at ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 81 hiring, but Henry is of the opinion that once you look at their rating, it becomes obvious that they are not very good at rating people, and by extension, identifying the most suitable person for the job. Henry, for instance, maintains that machines are less biased than human evaluators by giving the example of a study that looks at how hiring managers evaluate competence in femaleand maledominated occupations (he cites nurses and construction workers). He uses this study to suggest that the machine score was less biased than the human score. He concludes that if an algorithm is correctly designed, it is less biased than untrained human evaluators. Henry displays technooptimism in so far as a welldesigned and tested algorithm is constructed as superior to humans, not only in selecting the bestsuited person but also in avoiding bias. Humans generally appear as the bearers of bias in the decisionmaking process. Jasmine said, for instance, that AIsupported hiring is often used to make processes easier, but for her, the real benefit is that it standardises processes. Standardising processes increases fairness up until the point when human decisionmakers have a say. This can happen, according to Jasmine, for example, in facetoface interviews or when a human decisionmaker is given the scores of the individuals that best fit the job but then only picks the men to take them further. For Jasmine, AI can be used to make hiring fairer and to reduce discrimination, but she suggests that once humans influence the decisionmaking process bias creeps back in. Like with Henry, Jasmine constructs AI as fairer than humans. She says that regardless of how much one tries to reduce bias in AI, because once human decisionmakers enter the picture, bias can reemerge. Such an idea is indeed supported by research, which has shown that using AI in recruitment can lead to a reduction in diversity, but that this is not due to algorithmic bias but rather due to the recruitment manager: the recruitment managers, for instance, might not follow the machinegenerated ranking and thereby introduce bias (Bursell & Roumbanis, 2024). Fixing the Data and the Rater In order to maintain the stance of technooptimism, it was common to argue that algorithmic bias is fixable. The discourse to suggest that algorithmic bias is fixable first of all related to data. This is not surprising because most people see data as the root cause of algorithmic bias, as we have seen ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 82 earlier in this chapter. Ginny talked about that checking the training data that models learn from is paramount to avoid that the data is replicating biases. Kenneth suggested that training data needs to be diversified to create representative data sets, and he opines that big companies are already doing this. He further specifies how this is done: if you have a training data set based on 99% data from men and 1% data from women then it is necessary to balance the data input to 50% women and 50% men and to proceed accordingly in regard to race by including data based on 25% Black populations. Anton describes it as merely a ‘technical problem’ to build data sets that do not suffer from bias. Isabel uses a similar argument but is more sceptical that changing the input will be enough. She maintains that if there are few data points for women and even fewer for Black women, then any predictions that an AI system reaches will be weaker for these groups. For her, the problem lies in historic data, which will always replicate the same outcomes. Instead, she suggests that completely new data sets need to be created, which is similar to what Kenneth mentioned. However, her approach to fix the data is different: in the hiring technology company where she works, they have decided not to use historic data at all. They do not use CVs and they do not use what ‘good’ looked like in the past. Instead, they are trying to build new data sets that do not suffer from historic bias. However, much of what she discusses relates to future developments where she is optimistic that one day, hiring decisions can be made without bias. This strong belief in the potential of technology and that algorithmic bias is fixable is a typical discursive move to maintain the perspective of technooptimisms, in spite of the challenges associated with algorithmic bias. Apart from fixing representation in data, fixing the human who produces input data was also flagged as important by many people I spoke with. While the fixing the data idea discussed before largely centred on changing representations, these arguments focused more on how data emerges. The classic example is that a manager might display gender bias and thus rate women lower in performance evaluations, which are then fed into HR systems (Edwards & Edwards, 2019). In some cases, the manager does not even have to be biased, but gendered data input can result from input that is wellintended but has the opposite effect. We might here think about the women that were ranked three out of five while on maternity leave, as mentioned earlier in this chapter. Although none of the ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 83 interviewees used this specific example, human raters were mentioned in regard to how recorded interviews were scored. Such recorded interviews are often used in hiring, where on a hiring platform, a candidate is recording answers to specific questions via video. In this interaction, the interaction partner is actually the machine that poses questions and the interviewee answers. A common assumption is that those interviews are scored by AI. However, Kirsty and Ginny clarified that this is not the case. In Kirsty’s organisations, such recorded interviews are scored by humans who are trained assessors who assess based on a matrix or rubric. The same is true for Ginny, who states that only 20% of recorded interviews are scored by machines, which means that 80% are scored by humans. Again, the humans are trained industrial/organisational psychologists who evaluate based on rubrics. These rubrics are developed by the industrial/organisational psychology team after conducting a job analysis to ensure that the correct competencies are measured. These competencies themselves are assessed through models that predict those competencies, and those models were developed in the company based on trained expert raters. Ginny goes on to provide an example of how this works when a candidate is assessed in relation to customer service. In the recorded interview, candidates are asked to describe a time when they had to deal with a difficult customer. The organisation has thousands of examples of how other people answered this question, based on which the model, and the rubrics were developed and defined. The trained evaluators then evaluate the candidates’ answers as good, medium or low, for example. Ginny describes how they originally thought that humans are biased and that therefore, the scoring would be biased too. However, they discovered that with this standardisation through rubrics and trained assessors, human bias is reduced. Ginny’s argument suggests that the human bias in creating data can be minimised through training humans and assessing in rubrics. It fixes the messy and unruly human part, which Ginny describes as people going on their gut feeling based on unstructured interviews to make hiring decisions. This gut feeling Ginny suggests has impacted the hiring process in the past. If a gut feeling is used to hire candidates and this data is used in machine learning, the resulting hiring process will be saturated with human biases. By reducing this gut feeling and the human biases, the potential algorithmic bias in hiring is also reduced. ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 84 Henry also talked about improving algorithms. Henry who works in a hiring technology company insisted that 90% of his company’s algorithms are based on human raters. In order to illustrate why this is important, he cited the saying that ‘an algorithm is only as good as the data it’s being modelled upon’, which in machine learning is often described as ‘garbage in, garbage out’ (Weyerer & Langer, 2019). Henry explains that this means that if you have bias in the data, you are modelling bias in your algorithm. As such, it is central to avoid this bias in data input. To build their models, the company has multiple trained human evaluators evaluating each interview. They then check the agreement between the model and a human evaluator. As such, the predictive power of the models is measured against expert human predictions. Henry argues that by reducing the bias in humanrated data, the predictive models of AI can be improved. Like in Ginny’s example, the idea that human variation in assessment makes algorithms biased, or more broadly that societal bias is shaping algorithmic bias, is being employed to advance the stance that human bias has to be reduced in data input to avoid algorithmic bias. This means that if human raters are ‘fixed’, algorithmic bias can be fixed too. Fixing the Algorithm Apart from fixing the data and the (human) rater, interviewees also suggested to fix the algorithm. This can take different forms. Franklin talked about a provider that removed any personal information from applications to ensure that candidates are evaluated based on skills. Among the information that is excluded is where the candidate worked before. Franklin suggests that big technology companies are dominated by men and the provider is removing this information to allow hiring managers to focus on skills rather than impressive sounding company names. The candidates’ names, gender and race/ethnicity are also removed. Although it could be argued that removing such information from candidates’ profiles reduces the ability of the AI to learn any bias and to influence the hiring manager, in most cases, much of such information would be revealed at later interview stages, at which point it can still influence hiring managers. Bradley similarly talked about ways in which algorithms are blinded by, for example, excluding if a candidate identifies as female or male, which are the two gender options Bradley mentions. According to Bradley, the idea is ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 85 that algorithms then cannot distinguish between who is female and who is male and as such, bias might be reduced. Bradley is sceptical that such an approach ultimately works but he is aware that it exists. Another approach of dealing with algorithmic bias through blinding the algorithm is provided by Henry. Henry talks about how his company has collected millions of recorded video interviews and they were able to detect gender and racial differences in how questions are answered and which words are used. He explains that men would use different words to women to describe certain competences. His company’s approach is to ‘blacklist’ those words to avoid that they are included in the algorithm. Thereby, Henry suggests that the algorithm would not be influenced by gender differences in regard to which words are used to express competencies. Ginny suggests that the ‘beauty of algorithms’ is that even if the training data is biased, this can be mitigated in algorithms to avoid that this bias is reproduced. She acknowledges that you need to know what you are doing to avoid that. However, in her view, it is possible to control the algorithmic much more than the human mind. Ginny also suggests that information might be inferred by speech: women might talk more about childcare, whereas men talk about rugby. Her solution, similar to what Henry suggested, is to ‘block’ the words ‘rugby’ and ‘childcare’ to blind the algorithm for such gender differences. Such approaches suggest that it is possible to blind algorithm by either not including certain information or by blocking certain words. The idea is that if those inputs are not included in the predictions the algorithm makes, it is possible to control bias. Again, we see that the argument that it is easier to control algorithm than it is to control human minds is made, contributing to the view that if the algorithm is tamed by excluding human bias, it is possible to use algorithms to make recruitment fairer. It was also common to talk about quality controls that are implemented in regard to algorithms. Lucy explained that in her company, a second algorithm was developed that checks the first algorithm for fairness. She expresses that this gives people confidence that the algorithm is not built on bias. Similarly, Kirsty acknowledges that no algorithm is perfect, like no selection tool is perfect. She, therefore, says that in her company, the algorithm is checked for adverse impact for different profiles every year, which allows ‘tweaking’ the algorithms if need be. Adverse impact is used in the United States to refer to ‘the negative effect an unfair and biased selection ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 86 procedure has on a protected class’ (Mondragon, 2018), which can include, in the United States, sex, race, age and disability, among others. Duncan expresses a similar sentiment in regard to the fact that any algorithm that makes important decisions, such as who gets a job, should undergo quality assurance to ensure that there is clarity of what the algorithm measures. Kenneth also suggests that one should engage an AI auditing company to assess the algorithms that are used, which is something that was done by many of the providers I spoke to. Kenneth, in particular, talks about A/ B testing as important when auditing an algorithm. A/ B testing is a randomised experiment that involves an A version and a B version to see if they perform differently. Kenneth gives the following example of what would happen if one looks for a top programmer. He suggests feeding the AI system with information from Black candidates and few white candidates and then comparing that to a different composition of candidates to see if there are differences in the result. If the AI system picks one of the few white men, he argues, and ignores the Black candidates, there might be bias in the algorithm. Henry describes that once a model is built, his organisation follows up with an adverse impact analysis, which is what Kirsty mentioned earlier. This includes, as he says, to run statistics to see how men score versus women and evaluate the mean differences, the standard deviation differences and so on. Ideally, men and women should score as equally as possible and if there is a difference, one has to ask why that is and go back in to find why women and men score differently. Ginny provides further detail by arguing that there might be reasons why groups score differently, which would be fine from a legal perspective in the United States. This can include lifting heavy loads as a job requirement, which then would mean that more men than women might qualify. She describes this as a bona fide requirement. Ginny also talks about the fourfifth rule that is often used in regard to adverse impact in the United States. The fourfifth rule is a way to assess adverse impact through finding that the selection rate of one group is less than 80% of the group with the highest selection rate, as outlined in the Uniform Guidelines On Employee Selection Procedures (Biddle Consulting Group, 2023). However, Ginny asserts that while the fourfifth rule is often cited, one could clear this hurdle but might still find that there is unfairness. Ginny particularly points to subgroups that could be evaluated unfairly, such as topperforming women being evaluated differently than ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 87 lowerperforming women. As such, it is necessary to divide groups further to check in a more granular way that bias does not affect the model by including, for instance, information on performance. Although the fourfifth rule is regularly mentioned, Ginny, as well as others such as Alisha, stress that the fourfifth rule is not the sine qua non to assess equality and further tests are required to establish fairness. Overall, it was often argued that a detailed assessment is required to ensure that algorithms do not produce bias. Most interviewees agreed that this is not an easy feat but that it could be used to ensure that algorithms are not producing biased predictions. In that way, the view that algorithms can be fixed supports the stance that machines can be debiased. This in turn supports a technooptimist’s perspective. Machines Do Not Make Hiring Decisions, Humans Do However, there was also a stance that cautioned against the use of AI in hiring. Those perspectives were not technopessimistic in the sense that technology was seen as leading to poorer outcomes. This stance was characterised by being hesitant to use AI in hiring. This stance was not an outright rejection of AI in general but a hesitation of using AI at this point in time. Most of the interviewees suggested that algorithms could be fixed, but Alisha goes further in agreeing that sometimes an algorithmic fix can be found, for instance, in a hiring algorithm where it is possible to fix the data or the model. Yet, this was for Alisha just a stopgap measure until one is able to find a better solution. She specifies that by better, in this context, she means more equitable. However, for her, technologies are often used to ignore problems around inequalities. She goes on to explain that people often hide behind the veneer of the algorithm. Alisha specifically states that algorithms are said to be based on data and that implies that hiring becomes more objective if algorithms are used. This in fact chimes with a perspective that I outlined earlier in this chapter in regard to technooptimism. However, Alisha argues that sometimes algorithms are simply tools to ‘ignore difficult conversations’. She suggests that these are difficult conversations about inequalities. As such, she concludes that people simply hide behind algorithms and their presumed objectiveness without addressing inequalities in a more profound way. ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 88 However, Alisha was the only person mobilising the idea that algorithms are a veneer or something to hide behind. It was far more common to suggest that algorithms improve human decisionmaking. Lucy stressed that algorithms should not make all decisions but instead, they make recommendations that can then be taken further with human knowledge. As such, she points to the importance that humans are in the driver’s seat when it comes to making decisions. Lucy qualifies her statement by saying that humans obviously need to understand how the algorithm arrives at a suggestion. In a similar vein, Jasmine supports the idea that AIsupported hiring is fairer but she also says that human oversight over decisions is important. Henry was more specific in that he suggested that a machine allows people to make better decisions. He agreed that a machine should not make the actual decision but a machine can assist humans in improving decisions. Henry justifies this point of view by saying that the machine will use standardised and consistent data, which in turn can help humans to be less biased. As such, these ideas of how machines and humans collaborate still follow the idea of technooptimism in so far as they are used to suggest that human decisionmaking will improve, leading to better decision. Again, Alisha was slightly more hesitant about this machine– human collaboration. Alisha contests the idea that hiring managers are all powerful. She agrees that they have some power in organisations but she points out that the AI system has to be designed in such a way that allows the hiring manager to question the algorithm or to even overturn the recommendation. As such, she insists that humans would be able to contradict the predictions a machine makes. However, Alisha is not sure if humans will actually do that. For instance, if a candidate is suggested by a machine, would a human question this judgement or would the human pick the path of least resistance and follow what the machine recommended. For Alisha, that is a question of human nature, where picking the path of least resistance is common. Furthermore, she acknowledges that systems often do not allow for dissent. A hiring manager might have less opportunities to question the prediction of an ideal candidate of a machine because the system has not been built with this in mind. When I spoke with Franklin about the risks of using only candidates that a machine has suggested, Franklin recalls a conversation with one of his clients who historically has recruited the top people by going with who is on top of the stack. However, this client said that people further ALGORITHMIC BIAS AS ULTIMATELY FIXABLE 89 down the list are often less in demand and they show great developmental potential. Franklin suggests that candidates in the middle of the pile would ultimately be better people to hire in the long run. This risk of only hiring the bestsuited person for the job rather hiring midrange people who might be able to develop is however not a function of using AI in hiring. As a matter of fact, much of how recruitment is done is focusing on the person who best fits the specification one has set out. This is codified in the Uniform Guidelines On Employee Selection Procedures (Biddle Consulting Group, 2023), which, as I mentioned before, is regularly used when hiring. Algorithms mechanise this process and therefore intensify the focus on those who best fit the criteria set out. Yet, it has been argued that there is a benefit in going for a ‘wildcard’ hire from time to time to break the patterns that have been established over time of what skills are required to do a job (Tambe et al., 2019). A more developmental perspective of hiring would depart from much of how hiring looks like at this point in time. There is a risk that using machines to mechanise hiring leads to an even narrower focus on those who match the skills required best. While much concern is currently on eradicating bias from AI hiring processes by, for example, blocking language that might give away gender or race, there is a wider question in how far skills that are assessed might disadvantage those who have not developed those skills yet but who could do so in the future. At the moment, skills that are assessed in hiring are more baseline skills and often not something that cannot be changed much. For instance, the ‘Big Five’ personality assessment, on which much hiring is based, assesses extraversion or if one is outgoing. If it has been established that a salesperson should be outgoing and thus high on extraversion, this fits the current model of a salesperson. One could presume that the use of technology is leading to this profile of the salesperson as extravert is getting more and more refined. However, for machine learning, breaking those patterns with a ‘wildcard’ might be as useful. As such, standardising recruitment might be beneficial to reduce bias in the hiring process, but picking someone unusual might be helpful to introduce variations in patterns. However, such wider reflections, which could be described as technohesitation, were rarely brought up in the interviews. While some people reflected on how humans and machines collaborate in decisionmaking, it was also clear that it is not the case that an AI makes a hiring decision without human influence. AI filters people out that do not fit the skills IN/VISIBILITY BY DESIGN 96 a picture or what words are linked to specific sounds. This happens through labels that connect an image or sound with a word that the computer can understand. Designers of AI commonly develop classifications of labels that then need to be connected to data. These labels are often assigned to data by workers in the Global South either as crowd work or in more traditional employment forms. While the working conditions of these workers are regularly the focus of attention, these workers also play a central role in creating data sets that are constructed as universal and objective, even though they are based on complex processes of meaning making entailed in such assessments. The chapter shows how these subjective processes of knowing are harmonised and standardised as an objective truth in data labelling. It is also discussed that many of the classifications used are exclusionary. For instance, gender labels are commonly conceived as a binary. This chapter thus traces how data labelling constructs a knowable world. The chapter focuses on the mechanisms of construction and the international division of labour that these processes entail. AI’s Hidden Workforce If people picture someone who works in AI, the images that are conjured up include highly paid data scientists, probably white, a man and based in Silicon Valley. However, much of the work that allows AI to learn is done by data annotators who label data, which in turn allows for machine learning to happen. When ChatGPT, a chatbot developed by OpenAI, was publicly launched in November 2022, many people marvelled at ChatGPT’s ability to generate text that is hard to distinguish from what a human might write (Mollick, 2022; Abdullah et al., 2022). However, unlike previous chatbots such as Microsoft’s Tay (Vincent, 2016), ChatGPT did not produce racist and sexist talk. That is not a coincidence. The previous version of the technology had in fact a tendency to produce racist and sexist talk (Perrigo, 2023). In order to avoid that, OpenAI ensured that anything that could be seen as racist and sexist would be filtered out in ChatGPT’s answers (Perrigo, 2023). However, to determine what is racist and sexist, the AI system needed to learn what racism and sexism look like. The data for ChatGPT was scraped from the internet. This inevitably included data that could be seen as racist and sexist. Since OpenAI wanted to avoid that ChatGPT reproduces racist and sexist language, the system had to learn what such language looks like. IN/VISIBILITY BY DESIGN 97 In order to identify racist and sexist language, OpenAI followed a similar approach that is used by, for instance, Meta’s Facebook to filter out toxic language. This approach requires humans to label any data that is sexist and racist to ensure that it can be excluded from the AI output that a chatbot like ChatGPT might produce (Perrigo, 2023). This required human data annotators to label data that could be seen as racist or sexist (Perrigo, 2023). Such humanintheloop approaches are often outsourced to organisations in the Global South. This human input is often invisible and left out of focus when the presumed achievements of AI are marvelled at. Hiding the human input in technology is in fact not a new phenomenon. Amazon’s MTurk is a case in point. MTurk stands for Mechanical Turk and is thus a reference to a 18thcentury lifesize chessplaying automaton that was dressed in Ottoman clothing (Stephens, 2023; Geoghegan, 2020; Gray & Suri, 2019; Standage, 2002) (see also Chapter 1). Wolfgang von Kempelen developed the automaton and presented it for the first time in 1770 at the Habsburg court, and it then was exhibited in Europe and the United States (Stephens, 2023; Geoghegan, 2020). The chessplaying automaton pretended to be a machine that played and often won against humans. However, instead of being an automated chess machine, the automaton required a human hiding in the machine who performed the chess moves that were then translated via mechanics to the chess board (Irani, 2015; Stephens, 2023; Geoghegan, 2020; Standage, 2002). As many people at the time already presumed, the Turk turned out to be a hoax (Stephens, 2023). However, as Standage suggests, the arrival of the automaton coincided with the beginnings of the industrial revolution, when machines first began to displace human workers, and the relationship between people and machines was being redefined. The chess player posed a challenge to anyone who took refuge in the idea that machine might be able to outperform humans physically but could not outdo them mentally. (Standage, 2002, p. xiv) It was particularly the automaton’s presumed ability to interact with its opponents during the chess game that was deemed implausible because it required machine intelligence (Standage, 2002). Not surprisingly, the activity of playing chess is still seen as one way of evaluating machine IN/VISIBILITY BY DESIGN 98 intelligence against that of humans (Standage, 2002). The Mechanical Turk thus raised the spectre that machines can replace human intelligence and not just the physical power of humans (Standage, 2002). As we have seen in Chapter 3, a similar question is raised today in regard to what skills are uniquely human and if machines are able to emulate emotions, which up to now have been seen as a human advantage. But why was the automaton called the Turk? Although von Kempelen never named the automaton a Turk, the automaton wore Ottoman dress, which led to the name Turk. The reason why the automaton was dressed in Ottoman dress expresses a form of Orientalism but also reflects the longstanding rivalry between the Ottoman and the Habsburg empires including the Turkish siege of Vienna (Geoghegan, 2020). It has also been suggested that the Turkish style was popular in Vienna at the time (Standage, 2002). Another reason that the name Turk was adopted might in fact relate to German language, where the verb ‘türken’ translates as ‘to turk’ and means ‘to fake’. The verb has strong pejorative connotations, which is why it is seen as discriminatory and should be avoided (Duden, 2023; Geoghegan, 2020). The etymology of the verb ‘türken’ is unclear (Duden, 2023; Geoghegan, 2020) and one potential origin of the verb in fact goes back to the Mechanical Turk being a fake (Geoghegan, 2020). Yet, neither the origins of the verb nor the origins of the automaton have been conclusively shown. The name MTurk, which Amazon has chosen for its services, is reminiscent of von Kempelen’s automaton. The MTurk service developed out of Amazon’s attempts to reduce the number of duplicate listings (Stephens, 2023; Irani, 2015). Amazon tried to automate this action but failed because ‘[t] he task required a certain type of pattern recognition – the ability to detect subtle differences and similarities between pictures and text – which were easy for a human brain but could not be replicated by computer (sic)’ (Stephens, 2023, p. 66). In other words, there are certain patterns that only humans can recognise. Therefore, Amazon decided to give small, individual tasks to workers who could complete the tasks in piecemeal work (Stephens, 2023). Amazon then offered the service to other clients, leading to the development of the platform MTurk, where clients’, or requesters’, tasks were matched with people willing to do these micro tasks for often small amounts of money (Stephens, 2023). As mentioned before, such approaches are often called ‘humanintheloop’. A humanintheloop IN/VISIBILITY BY DESIGN 99 approach is required when human intelligence is needed to complete a task. Jeff Bezos calls MTurk ‘artificial artificial intelligence’ (Stephens, 2023).2 MTurk became emblematic for crowdsourced platform work (Irani, 2015; Howcroft & BergvallKåreborn, 2019). As Howcroft and BergvallKåreborn state, ‘[o] nline task crowdwork offers paid work (sometimes subject to requester satisfaction) for specified tasks and the initiating actor is the requester’ (Howcroft & BergvallKåreborn, 2019, p. 26). Research on MTurk and other platforms has regularly stressed the exploitative nature of these types of work (Irani, 2015; Howcroft & BergvallKåreborn, 2019). Apart from being often seen as economically precarious work, the work can also leave individuals mentally scarred; it has been argued that mental health issues arise in many people who moderate social media content (Bui, 2020; Irani, 2016). In their groundbreaking study, Gray and Suri (2019) describe such work as ‘ghost work’ because we often presume that the work is done by a machine but in fact the work is completed by humans. An example is security background checks for Uber drivers where the driver has grown a beard and as such no longer matches the image on file; a human is then tasked to determine if the person signing in as a driver is indeed the same as the person on file (Gray & Suri, 2019). Through their detailed study of platform work, Gray and Suri (2019) show how ‘algorithmic cruelty’ is affecting individuals engaging in this work. They detail the struggle to find work and get paid, and the isolation entailed in these types of workplaces (Gray & Suri, 2019). Due to the criticism of platform work as being exploitative, many companies have started to engage in practices of sustainable sourcing by outsourcing such work to providers that offer stable employment conditions for their workers (Gray & Suri, 2019). In attempts to make supply chains more sustainable, organisations that require moderation of social media data or data annotation have started to prefer suppliers like Sama or iMerit, whose mission is to offer work and thus a livelihood to individuals in the Global South (Perrigo, 2022,2023; Murgia, 2019). Although these workplaces have been lauded as the vanguard of AI (Murgia, 2019), it has also been reported that those workplaces can be exploitative in their own right (Perrigo, 2022, 2023; Pilling & Murgia, 2023, 2023, 2023; Pilling, 2024). As mentioned before, such work of moderating social media or labelling offensive language often leaves employees in such firms psychologically scarred (Perrigo, 2022; Pilling & Murgia, 2023; Pilling, 2024). It IN/VISIBILITY BY DESIGN 100 is suggested that for AI to work in the Global North, workers in the Global South have to put their mental health on the line3 (Perrigo, 2022). As such, efforts to ensure that technologies are free from sexist, racist and harmful language and imagery are often met by the challenges of global supply chains. The Need for Data Annotation Although working conditions are rightly at the centre of many discussions on data annotation, at this point, it is useful to explore why data annotation or data labelling is needed in AI in the first instance. In a nutshell, machine learning, a subset of AI, is often described as a pattern recogniser – a pattern is spotted and this is used to make predictions (Caliskan et al., 2017). Not all machine learning requires labelled data. In unsupervised machine learning, data labels are not required, but for supervised machine learning, data needs to be labelled (Bechmann & Bowker, 2019). For supervised machine learning, some data might already be labelled in the data set, which are then used to build models. In other cases, the data might require labelling. For instance, for selfdriving cars, it is necessary for a machine to be able to read how a stop sign looks, what a bus looks like or how humans of different shapes and sizes appear. Such image data thus has to be labelled to tell the machine exactly what a stop sign looks like. A similar process is followed for language that is, for instance, required for voice recognition software. As such, data annotators have to label images or language in data sets that can then be used for machine learning. Most of us engage in data labelling free of charge: when asked to prove that we are not a robot in online interactions, we have to, for example, select all pictures that have a motorcycle in it. The fact that we need to identify images and texts to show our humanity illustrates that machines struggle with this activity, creating the need for humans to label data in the first place. Data has to be labelled to establish something that is called ‘ground truth’. Ground truth is a term used in computer science and data science and is central for how data is used in algorithms (Jaton 2017, 2021). Jaton (2017, 2021) describes how ground truth is relevant in supervised machine learning: one starts with a data set and this data set is labelled by humans with clear targets (e.g. road signs, cars, humans), which the algorithm will have to identify. If there is disagreement among data labellers on how to IN/VISIBILITY BY DESIGN 101 label data, often, the majority vote is used to establish what should count as ground truth (McCluskey et al., 2021). The labelled data and the unlabelled data form a database that is called ground truth (Jaton, 2017). More specifically, ‘ground truth refers to information that is assumed to be true for an (sic) ML [machine learning] system’ (Kang, 2023, p. 1). The data set is then split into a training set and an evaluation set (Jaton, 2017, 2021). The training set allows the designers to ‘extract formal information about the targets and translate them into mathematical expressions’ (Jaton, 2017, p. 815). These mathematical expressions are then transformed into code and the algorithm is tested on the other set, which is called the evaluation set (Jaton, 2017, 2021). It is then assessed if the algorithm functioned as expected by comparing the result with the data labels assigned by humans (Jaton, 2017, 2021). The ground truth is thus based on the labels humans have assigned to data and allows to check for the correctness of the algorithm (Grosman & Reigeluth, 2019). The ground truth is as such a way to compare what a machine learned with what human labellers judged to be the case. The human labour that goes into developing an algorithm and seeing how well it is performing is central. In other words, the human labelling data helps machines to know what is true. Although the name – ground truth – suggests objectivity, establishing ground truth is an interpretive practice (Miceli et al., 2020; Henriksen & Bechmann, 2020; Paullada et al., 2021). Data labellers commonly receive instructions on how to label a text or an image from the designers of AI, which has been described as a way in which power from the designers of AI onto the people who label data is exerted (Miceli et al., 2020). How the designers of AI establish those classifications has been described as subjective, and in some cases, arbitrary, and these classifications are then created and perpetuated through AI systems (Miceli et al., 2020; Noble, 2018; Eubanks, V., 2018). Within the confines of the descriptions provided by the designers of AI, the data labellers often have to make subjective decisions (Miceli et al., 2020). This means that data annotation is a sensemaking practice (Miceli et al., 2020). Therefore, data labellers might disagree on how to label data. As previously mentioned, the majority vote is regularly used in such cases (McCluskey et al., 2021). However, using a majority vote obscures instances where data annotators systematically disagree which is particularly important in subjective tasks, like assessing hate speech or affect (Davani et al., 2022). If the disagreement is taken IN/VISIBILITY BY DESIGN 102 into consideration when models are being built, the resulting models are suggested to perform better (Davani et al., 2022). In consequence, taking the subjectivity of decisions that data annotators might make into consideration is important. However, in the process of machine learning, these subjective decisions are often obscured behind a presumed objectivity derived through majority votes. Even the name, ground truth implies depicting an objective reality. Yet, research has shown that presumed objective categories are regularly based on interpretation (Bowker & Star, 2000). Although the issue of data annotation and subjective decisionmaking is often raised in regard to supervised learning, it has been shown that unsupervised learning also relies on human supervision in regard to, for example, data cleaning or setting the number of topics (Bechmann & Bowker, 2019). Even the inclusion or exclusion of data in a data set can be seen as a way in which contextual factors influence and shape what the machine can learn (Denton et al., 2020; Miceli et al., 2020). Since data set are pivotal for machine learning, it has been suggested that data sets should come with datasheets that describe why the data was collected, what the data is composed of, how the data was collected, and what the data should be used for, among other issues (Gebru et al., 2021). A similar approach is also followed in the electronics industry, where datasheets are created for each component that details test results, usage and operating characteristics (Gebru et al., 2021). Other similar approaches are followed for drugs, which are accompanied by information on how to use and what the side effects might be. Including datasheets for data sets could, for instance, entail information on which subpopulations are included and how they are distributed in the data set (Gebru et al., 2021). In regard to data collection, it should be considered which crowd workers were used and how they were compensated (Gebru et al., 2021). Similarly, in their sixth principle of data feminism, D’Ignazio and Klein (2020) discuss how context is relevant for data collection. They argue that data is not neutral and that the context in which the data is collected, analysed and communicated is important to make power dynamics visible (D’Ignazio & Klein, 2020). In an empirical application of those concepts, Miceli and coauthors (2021) ask how the context of production in data image sets can be made visible. They show that clients are generally responsible for defining classifications based on which, for example, race should be labelled in the data. Clients are in the driving seat in regard to defining these classifications IN/VISIBILITY BY DESIGN 103 and categories. Given the fact that clients are often based in the Global North, whereas the labellers are often based in the Global South, this also introduces a power dynamic (Miceli et al., 2021). The research also stresses that organisations might be hesitant to extensively document the context in which data sets are created due to the time, effort and complexity involved in doing so (Miceli et al., 2021). As such, it is vital to develop effective ways of including the context in which data sets are created, including the power dynamics at play in any datasheets for data sets provided. Humans are centrally important for helping machines learn by recognising and labelling patterns. This process needs to be understood as a subjective one. As such, any subjective decisions, for example, associated with defining categories need to be recognised as such by documenting those decisions. Additionally, data sets need not only include information on how subjective decisions were made but also on the workers who label the data and as such introduce their own subjective decisionmaking in the process. Data sets need to include details on the context in which those decisions are being taken, such as the labour conditions of those who label data as well as the power dynamics between the client in the Global North and the provider often in the Global South. Thereby, it would be possible to make the human labour and the subjective decisionmaking that goes into data sets visible. Classifying the World Classification is a central activity for machine learning but the political dimension of classification is often ignored (Crawford, 2021). Bowker and Star (2000) describe the act of classifying or as they call it ‘sorting things out’ as deeply human. In other words, classifications are a form of organising the world. A classification is defined as a ‘spatial, temporal or spatiotemporal segmentation of the world’ (Bowker & Star, 2000, p. 10). Ideal classifications follow unique and consistent principles such as a temporal order (Bowker & Star, 2000). Categories are mutually exclusive and each instance fits into just one category (Bowker & Star, 2000). Ideally a classification system covers all potential instances, but this ideal is never fully achieved in reality (Bowker & Star, 2000). ‘[C] lassifications are powerful technologies’ (Bowker & Star, 2000, p. 319), which by being embedded in infrastructures become invisible. This is central to how they IN/VISIBILITY BY DESIGN 104 unfold their power. As such, Bowker and Star (2000) argue for recognising the architectures of classifications as political and for challenging the taken for granted status that classifications often have. Crawford (2021) draws attention to the fact that AI is based on classifications that are embedded in infrastructure and that are political. However, training data sets and AI infrastructure are regularly seen as ‘purely technical’, even though ‘they naturalize a particular ordering of the world which produces effects that are seen to justify their original ordering’ (Crawford, 2021, p. 139). In classifications around gender and race, the presumed ideal is that categories are clearly definable, clear cut and mutually exclusive. Gender and race are treated as automatically detectable and as something that can be predicted by AI systems (Crawford, 2021). It is common for data sets to follow a binary classification of gender such as using one for female and zero for male and an equally under complex classification for race of maybe five groups (Crawford, 2021). This is highly problematic as the example of how IBM tried to deal with algorithmic bias shows: IBM aimed to increase the diversity in data on facial recognition and they asked crowd workers to label faces as either male or female on a binary classification; yet anyone who was not neatly fitting into this binary was excluded from the data set (Crawford, 2021). When tracing how classifications are used in ImageNet, an image database, Crawford (2021) shows how available classifications under ‘adult body’ contain ‘adult male body’ and ‘adult female body’, where male and female are naturalised. Here, gender is classified in biological terms and as a binary. There is an option for ‘hermaphrodite’ but this is classified under bisexual (Crawford, 2021). Crawford (2021) concluded that nonbinary individuals are either ignored or placed in a category related to sexuality. This means, as Crawford (2021, p. 146) suggests, ‘[m] achine learning systems are (…) constructing race and gender: they are defining the world within the terms they have set’ (italics in original). Yet, these systems hide the politics entailed in their construction, which privileges clearcut categories over the complexities of everyday life. These classifications are not only ordering the present but they are, through AI systems, also perpetuated in the future structuring of how the world is classified in the years to come. Following the idea that machines are constructing gender, it is evident that for machine learning to happen, gender as a category has to be constructed. This entails to define what gender is and to operationalise IN/VISIBILITY BY DESIGN 105 gender to allow a machine to recognise gender (Keyes, 2018). How gender is defined and operationalised in machine learning has an effect on how gender is predicted (Keyes, 2018). This is happening, for instance, in Automatic Gender Recognition, where gender is ‘read’ from photographs (Keyes, 2018). The gender binary has long been questioned by research (Butler, 1990; FaustoSterling, 2000) but an analysis of papers on human– computer interaction has shown that gender is treated as a binary 94.8% of the time (Keyes, 2018). The research also found that gender is not only operationalised as binary but also as physiological and immutable (Keyes, 2018). Keyes (2018) suggests that this can be relevant, for instance, in regard to billboards that should show dresses to women and cars to men; if a transman passes such a billboard and is shown dresses, the billboard will have concluded that the transman is a woman. A consequence of this is that transgender individuals are likely to be misclassified, misgendered and ultimately erased (Keyes, 2018). It is therefore important to explore how AI produces a specific version of reality by exploring, for example, who is seen as a woman in facial recognition systems (Drage & Frabetti, 2023). As such, a machine is reading the gender of a person through a binary classification system that is then reproduced in the predictions made. Helping AI to Understand the World In order to illustrate how data practices help AI understand the world, I would now like to turn to how people I interviewed spoke about such practices. First, many interviewees addressed why data labelling is needed in the first place. Kenneth suggested that only about 30% of data labelling is automated today. Kenneth explained that AI is unable to recognise mountains in a picture or a middleaged man. For machine learning to happen, a picture needs to be labelled with such information to be trained. Kenneth states that AI needs to have data that is labelled with information such as what is a cat and what is a dog, and for this, data labelling is important. Howard mentions how machines learn by example and can only do that if data is labelled, and like Kenneth, he references that an AI needs to know what a cat is and what a dog is. Similarly, Nicole used cats and dogs to explain why data has to be labelled: it has to be defined what dogs look like and what cats look like, and images have to be labelled as such to allow a computer vision system to make accurate predictions if an image contains IN/VISIBILITY BY DESIGN 112 on to explain that we think of data sets as ‘objective sources of truth’, which she finds concerning. She explains that categorisations and classifications have been custommade for a context and are thus not universally applicable. Selena stresses that the context in which decisions on data sets are being made is important, and she suggests that there should be a record of these decisions. Selena goes further by stating that we also need to ask who is benefiting from this work. She argues that power dynamics that are embedded in data labelling need to be considered alongside the limitations of such data practices. Darryl follows a similar line of thought when articulating how a machine learning practitioner might go about doing an image classification task. This process starts by conceptualising and framing what the task is and what labels might be used in the system, which is similar to what was discussed in the previous section. However, Darryl stresses that this involves deciding on a categorical schema based on which the millions of images that have been collected can be organised. Darryl suggests that there are a ton of design decisions that go into that, down to which words one uses to describe the world and in which language that is going to be. You might decide on English, which is then a Global North bias. Then you decide to pick a thousand words, but carving up the world into neat categories with limited words is not easy. Darryl states that perspectives shape this right down to which images show up in the data set in the first place. For instance, one might use a web search for the different categories such as doctor. Then there might be a humanintheloop, a data labeller, who says if this image shows a doctor. This collection of data is not perspectiveless, as Darryl states, because a person might have a specific conceptualisation of how a doctor, a nurse or, for the sake of the argument, a basketball, looks. The perspective that is taken, Darryl explains, is often a white male, Western perspective of the world. Darryl goes on to stress that the choice of categories has a profound impact on machine learning and what categories machine learning is producing. These categories have to be linked to the ‘signals’ in the image. For example, if the data set only contains white, male doctors but no one in a surgeon’s uniform or scrubs because that has not been labelled in the data, this has implications for what the system can ‘see’. These data sets are, as Darryl stresses, not only used to train the AI system but also to assess its performance in the real world. This leads to a circular logic, as Darryl explains: if IN/VISIBILITY BY DESIGN 113 one uses a specific conceptualisation of what a doctor is and what a nurse is, this conceptualisation is used to measure how well the system works. Additionally, there is a specific conceptualisation of an image classification embedded in the data set; what is contained in an artificially fixed category is only one interpretation of the image. Darryl states that human vision is contextual in that how humans understand and describe the world depends on social identities and cultural contexts. Quenna raised a similar concern when she talked about how meaning is context dependent and will vary globally. In other words, not everyone is reading an image in the same way, but Darryl says that for machine learning, the human ways of interpreting the visual world are bounded and limited in regard to data sets. Darryl states that this has broad implications because the assumption is that computers can see and reveal the truth about an image. What Darryl is suggesting is that only specific ways of seeing are embedded in classifications and categorisations through which AI systems see the world. Darryl expands on this point by talking about epistemology, which is underlying the construction of data sets. Darryl states that the underlying epistemology is that data labelling is about recognising a selfevident truth in an image. This resonates with how Callum described ground truth: an ‘atomic bit of truth from the real world’. The assumption here is that the label assigned is a true representation of how the world is. Raymond, in contrast, talks about the ground truth of a data set as a better expression because what is described is what is correct within the data set. He stresses that this is not a general or generic truth or what might be true for one person, but rather a relationship within data. Darryl, however, states that how this truth is established draws on processes, which are deemed to identify the obvious and selfevident. Crowd workers are expected to label images, and this process entails identifying something that is clear and obvious in the world and that crowd work is an acceptable way of solving that task. This has consequences, according to Darryl, for how this work is done, and that a faceless and nameless crew of workers who label images with average scores is an acceptable way of getting to this selfevident truth. Darryl acknowledges that there are contextual differences in how people label. Darryl states that the facelessness and namelessness of this process also contributes to the impression of the final data sets being universal. However, Darryl questions if this universality is really true because it might IN/VISIBILITY BY DESIGN 114 be based on a single label attached to a single data instance. However, where this label came from and who ended up doing this work is lost. Darryl explains that this claim to universality is reflected in that the people doing the labelling are seen as not mattering. Those people are not seen, there are nameless, and where those people come from does not matter. It is accepted that a human has to do this work, but Darryl states that these infrastructures used to complete this work make workers invisible. To sum up thus far, Darryl connects the claim to universality of data that is made directly to the invisibility of workers. Only if the workers and the work are made invisible is it possible to claim that data labelling creates universal truths about the world. Darryl then goes on to articulate how people have started to think about how different annotators bring different perspectives to bear, which needs to be captured. This can be variation in data labelling, which is a sign that people do a task differently. However, that is not the norm in data annotation because most datalabelling projects treat labels as selfevident, without the need of interpretation. Such variations in seeing need to be made invisible to ensure that data labels are efficient, scalable and costeffective. In order to avoid that, Darryl suggests that it is necessary to recognise that ways of seeing the world differ among people. The construction of ways of seeing and describing the world that both Selena and Darryl talk about is meaningful, not only for the working conditions in which such work is done, but it also ignores the fact that how people perceive and describe the world varies. While Ava recognised this point, Selena and Darryl expand on this and Darryl links it to epistemology. Ways of knowing differ and often, the perspectives that claim universality are in fact nothing but a god trick, as Haraway (1991) might say. There are specific perspectives embedded in how classifications are designed, how labels are described and how labels are being applied. However, most data sets seem to pretend that they offer a view from nowhere to claim that the information they entail is universally true. This in turn renders the mechanisms of production of these data sets invisible. If a perspective is taken to represent the universal truth, this is possibly the perspective of designers of AI who develop the classifications and write the labels descriptions. The data annotators follow those instructions and, if they do not apply the labels correctly, are told how to label the world in ways that is described in the labels. However, data annotation companies have a IN/VISIBILITY BY DESIGN 115 crucial mediating function here. This is often considered in regard to what working conditions they offer. However, the data annotation companies also negotiate meaning between the data labellers and the AI designers in the client companies. This important mediating function of how knowledge is created remains often similarly unacknowledged. In other words, datalabelling companies are the organisational link in the epistemological chain – they mediate what knowing about the world is embedded in data labelling. Constructing Gender The fact that gender is commonly labelled as a binary was regularly discussed by those who were involved in and familiar with datalabelling processes. A common concern raised by, for instance, Parker, Darryl and Ava is that gender in AI is typically binary. Ava asks what that would mean for a person who does not identify as one of those binary genders. Ava said that decisions are being made based on the categories that are included in an AI system. She thus alludes to the fact that those who do not identify based on the two options of gender offered, might not be included and thus become invisible. Selena similarly points to the problem of reifying gender through categorising people along a gender binary while also erasing trans and queer identities. Additionally, Selena is concerned that intersectional experiences of gender are made invisible. She explains that the experiences of a white woman are different from a Black woman and just lumping women into a category of women is making this difference invisible. Brenda mentioned that more clients are concerned about bias in the data but that most of the data still follows a binary approach in regard to gender. Darryl also spoke about the assumption that gender is commonly conceptualised as a binary and that gender is treated as knowable from an image. Darryl explains that gender appears as a fixed and a natural category in machine learning, even though it is constructed, situated and shifting. This is problematic for the development of computer vision systems, as Darryl states. However, like Brenda, Darryl has observed a shift in recent years where it has been recognised that gender is not binary and that it is not possible to know someone’s gender by looking at an image. Instead, if data sets are labelled with gender, the data is labelled with ‘perceived IN/VISIBILITY BY DESIGN 116 gender’. Darryl asserts that this is a step in the right direction because most data sets have an asterisk next to gender with the statement that gender is not binary. Darryl says that this acknowledges the idea that gender is not a binary but she complains that the same data sets then go on to use gender as a binary. The same phenomenon of stating that gender is nonbinary to proceed with gender conceptualised as a binary was observed by Parker. Such a discursive move shows that there is an awareness for gender as a nonbinary, which, however, does not lead to any changes in practices of how gender is conceptualised. Darryl states that even if data sets are labelled with perceived gender, these perceptions are still culturally and socially situated perceptions of masculine and feminine presentations. Darryl stresses that such perceptions shift geographically between cultures but also in regard to age and race. In Darryl’s view, classifying people into gender categories is problematic because those categorisations are often racialised and could be seen as an expression of what Darryl calls a colonial project. Darryl says that who defines those categories of perceived male and perceived female is central because these are not objective or selfevident. This raises wider questions for Darryl in regard to what needs to be measured at all. So, for instance, if you need a system that works for different gender categories, it might be best to go with selfidentification of individuals. If one needs to know how a system performs for people with short or long hair, facial hair or not, then this could be used for analysis. If you need a gender label and cannot rely on selfidentification, Darryl suggests that framing those labels as not selfevident is central and that going with the perception of labellers might be possible if this would be framed as a perception rather than a fact. Darryl goes on to explain that the labellers could give some evidence why they come to a judgment, such as what a person is wearing, the perception of secondary sex characteristics, grooming styles or presentation. For Darryl, the articulation of how one arrives at a judgement is key and would allow contextualising the resulting labels. It has to be clear that these are judgements, not an objective measure. According to Darryl, part of the problem is how questions to data labellers are formulated. Furthermore, as Darryl states, one has to collect information about the data labellers themselves because people who have different relationships with gender, such as being queer, trans or genderdiverse folk,4 might come to different IN/VISIBILITY BY DESIGN 117 judgements in regard to gender labels than a cisgender person who has never thought about the socially constructed nature of gender. A similar point was raised by Hayden. Additionally, people from different cultures might understand gender categories differently, as Darryl points out. Darryl states that it is therefore important what informs their reading of gender and why they read gender in certain ways. Such additional information about data labellers alongside framing questions in a precise fashion will allow for annotations that are contextualised. Another reason why labelling gender might be important is provided by Callum. Callum talked about the risk of categories being inferred, which he constructs as more problematic than having an explicit label. He provides the example of someone who might have a LGBT5 initiative on their CV. Then, according to Callum, it is not clear if you identify as LGBT or if you are an ally. However, for the model, that does not matter because it might still infer from the data that you are a less good candidate. Callum implies that if LGBT status would be explicitly labelled, there might be ways to mitigate for bias, which is more difficult if there is no label but the information is inferred. Similarly, Sabine stated that even when gender is not explicitly stated, there will be a plethora of proxies for gender that are inferred from data. This concern goes back to some issues that were discussed in Chapter 4. Brenda provides a similar example when referring to an academic paper. The paper used language data from Trustpilot to analyse gender and word choice. Brenda is sceptical about the methodology used in the paper: not only was gender regarded as binary but names were used to deduct if a person is a woman or a man. The paper found that women and men use different words, with women being more descriptive, such as using fantastic, wonderful, awesome, happy, and men using words focusing on price and quality, like inexpensive, economic, cheap, best quality and so on. When I asked what this might mean, that an AI could conclude that by using such words you are a woman, Brenda provides a use case where a person writes a review about a product, the language is analysed, the person is classified as woman or man, according to the language used, and then the person is shown advertisements targeted at women. Brenda later expands on that by saying that language is part of gender socialisation and nonbinary individuals might also use language in different ways and might thus not be targeted correctly by those ads. IN/VISIBILITY BY DESIGN 118 Gender was also discussed in relation to language translations. Brenda talked about an example from Google Translate, where Hungarian is translated into English. Brenda shares an example with me where in Hungarian, the pronouns are gender neutral, but the English translation transforms this gender neutrality into something that is stereotypically gendered such as ‘she is beautiful’ and ‘he is clever’. What is interesting is that the original language was not gendered but gendering is introduced when the text is translated. Here, the machine translation is doing the gendering. Brenda uses a specific example from developing a chatbot where the chatbot automatically was referenced as a he and the interior designer was referenced as a she. Another issue in relation to gender is coreference tagging. Coreference tagging means, as Brenda explains, to tag the reference between names and pronouns. She uses the example, ‘Alex went to the concert; he said it was amazing’, which means that ‘Alex’ and ‘he’ are coreferent and ‘concert’ and ‘it’ are coreferent. But what happens if Alex is a woman? Then the system needs to be able to understand that Alex can be a ‘she’. Or if Alex is nonbinary then the system should say ‘they’. Brenda says that if a system is not trained to have this flexibility, it is likely to exclude and the system is biased. However, Brenda acknowledges that she has never seen such a project that was designed with inclusion in mind. Speech recognition might also pose specific challenges from a diversity perspective. Ava talked about how in speech data collection, one might attempt to find examples of regional dialects and then have men and women speak in that dialect, but the more granular this intersectionality becomes, the less examples there will be, which is also something Georgia mentioned. This affects the training data, which Ava says will be less robust. Speech recognition also has a normative aspect to it, as Brenda elaborates. She states that automatic speech recognition voices like Siri and Alexa speak standard voices, emulating what is called stable linguistic periods of people. This period is defined as people between the ages of 20 and 55. Data sets might contain 10%– 15% of people over 65, which might mean that people over 65 are less well understood. Equally younger people, who might be more innovative with language or who might or might not go through puberty, might also not be understood. The speech data will also be collected from men and women, and as such, trans persons might be less well understood. Similarly, what Brenda describes as ‘stereotypical gay IN/VISIBILITY BY DESIGN 119 male speech’ is something that an automation speech recognition engine needs to be trained on. Finally, another issue relating to gender that was mentioned was that Siri and Alexa and other VPAs had default femininesounding voices (see also Chapter 1). This is a topic of regular academic, policy and media concern (Equals & UNESCO, 2019; Dillon, 2020; Strengers & Kennedy, 2020; Sutko, 2020), which has led providers to offer more diversity in VPA voices (Baraniuk, 2022). As such, it is not surprising that the topic was mentioned in the interviews as well. Ava suggested that the default feminine voices reflect gender stereotypes that designers had, but she also talked about how there are now attempts to develop nonbinary voices, for instance, in Project Q – an attempt to create a genderless voice (Project Q, 2023).6 While Ava thought that this is an interesting development, she wondered in how far this will remain niche and the standard is going to be feminine voices for assistants. Similarly, Larra talked about how many of her clients give virtual assistants or chatbots a gendered and often feminine name. She describes how she is pushing back against clients who pick gendered names and she proudly states that most of the virtual assistant or chatbots she worked on did not end up with gendered names. Overall, many of the interviewees articulate how gender is relevant for data labelling and, by extension, machine learning. The interviewees showed an awareness for the fact that current practices around gender and data often mean that binary gender is reified. While many described these processes as problematic, they also acknowledged how difficult it is to change those practices towards more inclusion. Moreover, it is evident that practices around data labelling are constructing gender. The gender patterns used in machine learning shape which gender patterns are predicted. Conclusion The chapter focused on patterns that humans have to recognise to help AI learn. This chapter started with the question of how machines recognise gender patterns, or in other words, how a machine knows who is a woman, a man or nonbinary. The chapter suggested that machines perceive the world through labels that are assigned by humans or developed during machine learning. If humans add these labels, this often happens through AI’s hidden workforce – those who label data as a crowd work task or by IN/VISIBILITY BY DESIGN 120 individuals adding labels working in the AI supply chain. This work is often done in the Global South. While much research has rightly focused on the working conditions of these workers, this chapter has particularly stressed that such workers interpret the world but that these interpretations are made invisible. The chapter explains that data labelling is necessary to allow machine learning systems to recognise patterns, which then form part of the outputs the machine produces, or in other words, the predictions. Therefore, data labelling is needed for AI to help machines see, hear and understand the world. For this to happen, a ground truth – labelled data that is assumed to be true in machine learning – needs to be established to allow machine learning and to check the quality of machine learning. Labels are classifications and the chapter discusses how classifications are ways in which the world is ordered. It was shown that once this organisation of knowledge has happened, the classifications often become accepted for how the world is. As such, these ways of organising the world through classifications such as in machine learning labels are political but the processes of construction are made invisible. In data labelling, it is seen as important to create a consensus among human labellers and moderators about which labels to apply to best represent the world. These processes entail turning subjective decisions into a seemingly objective and universal truth. However, the chapter has shown that this universality is carefully negotiated between different actors in data labelling who embed what is knowable about the world in labelled data sets. What is knowable about gender in data labelling generally seems to follow an understanding of gender as a binary, with limited scope to conceive gender beyond a binary. The world that is being constructed through machine learning is in many ways a simplified understanding of the world, which is presented as objective and universal. Thereby, classifications that are conceptualised and operationalised through data labels construct a reality. Yet, this construction of reality is a potentially exclusionary one. Building more inclusionary approaches in regard to data labelling is central to make these construction processes visible and tangible. As such, the chapter has argued that there are some patterns that only humans can recognise but that the subjective processes based on which this recognition happens are regularly made invisible to suggest that these patterns are objective and universal. IN/VISIBILITY BY DESIGN 121 Notes 1 Data labelling and data annotation are used interchangeably in this chapter. 2 It should be noted that academic research often relies on MTurk as well (Aguinis et al., 2021). For example, academic research regularly draws on MTurk to find participants who can complete surveys. 3 Sama says it offers premium pay and psychological support for such type of work (Perrigo, 2023). Yet, it has been claimed that such psychological support is difficult to access (Perrigo, 2023). 4 This is the terminology Darryl used. 5 LGBT is the term Callum used. 6 Project Q aims to create a genderless voice and should not be confused with OpenAI’s Q* project (Lee, 2023). CONCLUSION: UNWRITTEN RULES 128 From Magic to Making Rules Visible For many people, terms like AI, algorithms or digitalisation appear mythical and magical because the workings of those technologies are hidden from sight (Finn, 2017) (see also Chapter 1). Similarly, the term ‘black box’ describes the opaqueness of technologies where even those who design these new technologies can often not fully explain why they work in specific ways (Pasquale, 2015). In Chapter 5, I have argued that invisibilities around how data is prepared for AI are central to make AI appear as objective and universal. However, these invisibilities of such processes are also important to see AI as magical and mythical. This magical and mythical nature of technologies invites us to engage with these technologies in ritualistic ways that can function to mitigate the risks and uncertainties of modern life (Finn, 2017). While those technologies might provide a kind of ritualistic comfort, many of those technologies have consequences for people’s lives and as such, there is an urgent need to understand and explain how these technologies work. In a sense, it is important to make technologies less magical and mythical to start engaging with them in a more enlightened way. Throughout the book, I have suggested that technology is shaped in design and use by society and vice versa. This perspective emerges from the social shaping of technology approach (MacKenzie & Wajcman, 1999) (see Chapter 1). Research in this vein would normally show how specific technologies are shaped by society, such as how the electric version of the refrigerator became the norm (Cowan, 1999). Another example comes from the gendered meanings associated with microwave ovens. When the microwave was first introduced into homes, it was imagined and marketed as a way for men to reheat food but its usage often led to unexpected results such as women customising individual meals for family members with the aid of a microwave (Cockburn & Ormrod, 1993). This classical study about gender and technology highlights the fluctuating and interrelated ways in which meaning around new technologies intersects with gender (Cockburn & Ormrod, 1993). Such studies above all show how social relations enter technology; social norms enter the design and use of technologies and these technologies shape social norms in turn. Technologies such as AI learn social norms through data but also associated processes and decisions (see Chapters 4 and 5). As such, these technologies are learning the unwritten rules of society. But these technologies will also be able CONCLUSION: UNWRITTEN RULES 129 to pick up changes and inconsistencies in relation to such patterns. For instance, the research on microwave ovens shows how gender materialises in technology in expected but also unexpected ways, leading to patterns that show continuity but also change (Cockburn & Ormrod, 1993). So far, research has largely focused on the patterns that repeat exclusion, such as by creating algorithmic biases and as such, automating inequalities (Eubanks, V., 2018). However, one can expect that there is possibly more variety in the patterns of inclusion and exclusion that emerge in relation to gender and digitalisation. Although AI and related technologies are rightly seen as creating risks of exclusion (Eubanks, V., 2018; Benjamin, 2019), I have discussed traces of how AIsupported hiring can be understood as making the underlying rules of society visible and thus tangible (see Chapter 4). In Chapter 4, I showed how those who design and use hiring technologies construct algorithmic bias as ultimately fixable. I have critiqued this perspective as an expression of technooptimism that conceives society as something that is fixable through technological means. Such a perspective permeates the tech industry. This technooptimism suggests that the black box of technology can be opened and corrective measures can be taken, leading to explainable AI. I questioned if such biases are fixable because they are a function of society. However, while the idea of a technological fix of societal issues should be questioned, technologies such as AI afford us with the possibility of showing how the unwritten rules in society function to systematically exclude groups of people. AI as a pattern reader and repeater is central to understanding the unwritten rules of how societies operate. In other words, AI repeats and amplifies human biases but it also makes inequalities that exist in society visible. However, there is nothing mere about this because crystallising the unwritten rules of society is important and meaningful. Making the unwritten rules of society visible is meaningful, not in the sense that it allows us to apply a technical fix, as many of the interviewees suggested. Instead, it might allow us to change the unwritten rules of society in general and at scale. For example, previous interventions in regard to discrimination in the workplace have focused on ‘fixing’ individual decisionmakers. If a white male hiring manager ends up hiring a younger version of himself, we tried to change the practices of the individual manager by calling out this pattern.2 Hypothetically, AIsupported hiring technologies might now show us that white, young men might be generally the CONCLUSION: UNWRITTEN RULES 130 preferred candidate to be hired. For other positions such as in care work, the implicit assumption might be that such work is done by women from the Global South. This is of course something that researchers have shown for a long time (Acker, 1990). In fact, we have seen in Chapter 2 that many discourses on the future of work contain a concern for a specific type of person: male, whitecollar professionals. This was the implicit ideal worker imagined in the books on the future of work. AI has the potential to make such unwritten rules visible and tangible in a systematic way. Although I would caution against attempting to fix society by fixing technology, being able to make the unwritten rules tangible might allow for changing them as societies. Such a perspective also opens the possibility of seeing such patterns as more complex and dynamic. There might be dominant and less dominant patterns but there might also be patterns that contradict each other. This complexity and dynamic nature of pattern is central to how societies function, and stressing such contradictions and complications in patterns would be a novel way to think about digitalisation and gender. There are always multiple patterns competing for attention. Equally, for new patterns to form, a new component needs to be introduced or emphasised. We have seen a moment of such a recognition in relation to the wild card hires in Chapter 4. If we hire people who might not fit the pattern in the most perfect way, we open up the opportunity for new patterns to emerge. As such, patterns should not be seen as determinist. Such perspectives dominate thinking on algorithmic bias where the risk of repeating past patterns is leading to exclusion. While such risks have to be taken seriously, it is also important to remember that new patterns can be created, which might lead to greater inclusion. As a matter of fact, the common fix to algorithmic bias, fixing the data, is in a sense introducing a new pattern. Since data is often constructed as the basis for algorithmic bias, as we have seen in Chapter 4, many approaches to deal with algorithmic bias are centred on fixing the data. This might include ensuring that data sets represent society. If AI fails to recognise Black faces, the solution is to include more Black faces from which AI can learn (Buolamwini & Gebru, 2018). If AI suggested only male candidates because the underlying data set included largely men’s CVs, then the solution is to include CVs by women (Dastin, 2018). Chapter 4 details many of those ‘fixes’ to ensure that algorithmic bias is reduced. We also need to look at CONCLUSION: UNWRITTEN RULES 131 how data is being produced and processed, which was at the centre of Chapter 5. It has been suggested that ways forward are datasheets for data sets (Gebru et al., 2021). Datasheets should include, for instance, information about the motivation to collect this data, the composition of the data set, the collection process of the data, the preprocessing/ cleaning/ labelling of the data set, uses, distribution and maintenance (Gebru et al., 2021). By providing detailed questions that should be considered in datasheets for data sets, Gebru and coauthors (2021) provide admirable guidance to improve data sets. While these approaches to improved data are necessary, it is questionable if they are sufficient. Data will be improved by being more representative, more ethical and more transparent. However, the underlying issue that data is reflecting society will remain. Outstanding data practices might help to reduce algorithmic bias, but the social patterning of data might come through in another way. For example, in AIsupported hiring, a provider might drop facial recognition to avoid that the selection of suggested candidates is not influenced by the technology being less able to recognise Black women. Yet, the same Black women might use certain language constructions, which might be judged as less suitable for a role and thus filtered out. Of course, good data practices could reduce these risks, but there is a danger that algorithmic biases emerge in other shapes and forms because data is inherently social. Societal relations imprint on data. However, as I have argued, it is also possible to use this as an opportunity to create alternative patterns. Reification Machines Reification is a charge regularly levied against research on gender. For instance, research might state that gender is seen as socially constructed, yet then proceeds in the empirical part to operate based on a fixed gender binary (Nentwich & Kelan, 2014). For much research on gender, a standard criticism is that such research is reestablishing gender binaries rather than challenging them. Seeing gender as fluid and flexible to counteract the conception of gender as a fixed binary has been discussed in academia, at least since Butler’s seminal book, Gender Trouble (Butler, 1990). The idea of gender beyond a binary has received purchase in wider society.3 While such conceptions of gender are often derided as being part of ‘gender ideology’ CONCLUSION: UNWRITTEN RULES 132 (Kuhar & Paternotte, 2017), it should be noted that, for example, Germany legally recognises a third gender and thus moves beyond a gender binary (AntiDiskriminierungsstelle des Bundes, 2023). One could thus argue that moving beyond a gender binary, which has been central for gender studies for a while, is increasingly something that is recognised in organisations and wider society. While moving beyond gender binaries has reached the mainstream, there is a strong tendency in AI to reestablish the gender binary. For example, in HR, attempts have been made to offer more than two options to signify gender, yet in AIsupported hiring, gender is largely treated as a binary (see Chapter 4). As I have shown in Chapter 5, the gender binary as underlying classification is rarely questioned in machine learning. Similar to research in gender studies, research in machine learning often states that gender is not a binary yet proceeds to treat gender as an unchangeable binary (Keyes, 2018) (see also Chapter 5). Machines read gender through the data in a variety of ways. Gender might be selfidentified or someone else is picking a gender label. Furthermore, gender is embedded in data through, for instance, certain words that people commonly read as words women use. Even though popular perceptions of gender are moving beyond a gender binary, AI is often reifying gender as a binary. If AI is a gender reification machine, this affords the ability to study the unwritten rules through which gender is established. Approaches that see gender as a doing (West & Zimmerman, 1987) or performed (Butler, 1990) often seek to understand such unwritten rules of how gender is established in interactions. For example, Goffman’s (1979) work on advertising showed the ritualisation of gender by showing how relative size is a marker of gender. Garfinkel’s (1984) work engaged with identifying markers of femininity. If AI is making rules of societal interactions visible, then AI is an opportunity to understand gender in society. The emerging worlds of AI reify gender as a binary or, in other words, AI creates worlds that are by and large based on gender binaries. If gender is treated as a binary in daily life and data reflects this, machines will learn gender binaries by default. It has been rightly pointed out that this is problematic if it leads to individuals being misgendered (Keyes, 2018). It also restricts which futures can be developed. If we rely on AI replicating a certain version of society, this limits which futures can be created. In this case, futures where gender moved beyond a binary are less likely. As such, CONCLUSION: UNWRITTEN RULES 133 reifying gender in and through AI means that what futures are possible is curtailed. Politics of Visibilities An underlying concern for this book were processes of how gender is made visible and invisible in discourses on the future of work (Chapters 2 and 3) but also in relation to how AI constructs a specific reality (Chapter 5). As discussed in Chapter 2, in books on the future of work, the main concern was for male, middleclass breadwinners whose jobs might be automated. The common concern was that if machines take over jobs, people will not be able to earn an income. However, someone might still profit from the labour of machines: those who own the machines. For instance, these might be founders of or shareholders in Silicon Valley companies. We have also seen in Chapter 5 that the working practices of data labellers and how they contribute to constructing AI worlds are made invisible. One could even go as far to talk about epistemic erasure (Mahalingam & Selvaraj, 2023) in this context. The concept of epistemic erasure denotes a process of how the lived experiences of those from disadvantaged backgrounds are delegitimised through cultural practices shaped by privileged groups (Mahalingam & Selvaraj, 2023).4 Moreover, most of these data labellers work indirectly for organisations that, through their labour, turn a profit. However, ownership structures and who profits from running the machines or from the efforts of data labellers are hidden from sight in most accounts on the future of work. The likelihood is that those who profit from the labour of machines belong to fairly small groups of individuals. Concerns around the future of work manifest in tropes such as the epic battle between man and machine. During the First Industrial Revolution, physical power was replaced by machines and the assumption was that humans were superior intellectually (Standage, 2002). However, AI that appears to display humanlike intelligence is now a major underlying concern driving the thinking on the future of work. Hence, we see those who have traditionally used their intellect to earn a living, such as professional workers, being the main focal point for concerns in regard to the future of jobs. These concerns are mitigated through two discursive strategies: first, to point towards augmentation to suggest that humans and machines collaborate and, as such, humans are required in the future; second, to CONCLUSION: UNWRITTEN RULES 134 suggest that certain abilities are beyond the realm of machines such as socioemotional skills. Yet, as I have shown in Chapter 3, socioemotional skills are within the realm of machines because they follow automatable patterns. This does not mean that machines have emotions but that they can read and respond to emotions that humans display. Like with other social interactions, machines can discern the unwritten rules of emotions from data and repeat these patterns to, for instance, train ideal emotional responses in humans. It appears that like during the Industrial Revolution, when physical power was replaced by machines but intelligence was perceived as uniquely human, technologies like AI seem to threaten human intelligence, yet imply that socioemotional skills are uniquely human. However, this assumption might not hold true if machines also appear humanlike in regard to emotions. There are obvious tensions between becoming visible and being not visible. In some instances, gaining visibility is central for having one’s identity recognised, yet in other cases, having personal data revealed can be harmful. Data is often scraped from the internet such as from social media, which can be used, for instance, in hiring. Scraping data from social media for hiring is problematic because it violates the privacy of candidates and might also reveal membership affiliations, such as in relation to age or race (Black et al., 2015; Jeske et al., 2019). Similarly, belonging to an LGBTQ group could lead to individuals being ranked lower in hiring processes (Tomasev et al., 2021). It has also been suggested that some data might be too sensitive to include. An example would be how Grindr passed users’ HIV status to third parties; while this information is provided voluntarily and with consent, it has been suggested that this information is too sensitive to be held by such platforms (Rzepka, 2023). In such cases, visibility might be highly problematic because it can be used to exclude individuals. Another way to deal with the invisibility of data is synthetic data (Eldan & Li, 2023; Gunasekar et al., 2023; Jacobsen, 2023). Synthetic data promises to provide unbiased and labelled data by including variation on, for instance, age, race and gender (Jacobsen, 2023). Given the complexity of reallife data for facial recognition, the Mixed Reality & AI Labs at Microsoft Cambridge developed the model ConfigNet to generate photorealistic synthetic faces, while allowing a modification of these outputs, for instance, by adopting different poses or including different skin tones (Jacobsen, 2023). As we have seen, for instance, in Chapter 4, the underlying data is often CONCLUSION: UNWRITTEN RULES 135 blamed for algorithmic bias, and synthetic data seems attractive because it eradicates issues around representation in data sets; in other words, if a group is underrepresented, the missing group is simply generated and then included in machine learning. Jacobsen (2023) warns of the tendency to solely see algorithmic bias as a ‘training dataset problem’ that can be fixed using synthetic data. Algorithmic bias can, as we have seen in Chapter 4, also emerge through designing models poorly (see also Jacobsen, 2023). Jacobsen (2023) argues that synthetic data is presented as a way to derisk data where synthetic data is constructed as riskfree. However, such a move also means that wider criticism in regard to resisting and challenging machine learning is silenced (Jacobsen, 2023). Synthetic data can be seen as another way to ‘fix’ data and thus algorithm without paying attention to wider social implications of technologies (Jacobsen, 2023). As I have outlined earlier in this chapter, trying to exclude the social from data is highly problematic. Although synthetic data promises to resolve many issues in regard to diversity and inclusion, unless technology is understood as social, such fixes will remain partial. Accountability and Responsibility The book also offers perspectives on accountability and responsibility in relation to digitalisation. Throughout the book, I have stressed that technologies are shaped by social relations and vice versa, drawing on the social shaping of technology approach (MacKenzie & Wajcman, 1999). The book has provided countless examples of how this shaping happens, from how AI is used in hiring (Chapter 4) to how data is labelled for AI to learn (Chapter 5). The central idea emerging from material is that technology does not appear out of nowhere. It is created by people, and how these people think and behave influences what technology is created. Which technology is created is also influenced by those who finance the development of those technologies. As I have outlined in Chapter 1, there are vested interests behind fostering one technology over another. Such an ability to shape technologies also comes with responsibility and accountability. While most research tends to show us how traditional social patterns are repeated, I suggested in this book that the potential of the social shaping of technology can be utilised to create, develop and foster technology that is creating more inclusive futures. This is not an automatic process but one CONCLUSION: UNWRITTEN RULES 136 that requires that care is taken of and consideration is given to how gender as well as other forms of diversity affect technology in design and use and vice versa. This is a constant process because diversities themselves are changing alongside technologies. It is also not an easy process because it could be reasonably presumed that what is beneficial for one group of people might not be beneficial for another. Such complexities need to be reflected on and considered. Different interests and consequences have to be carefully analysed and weighted. This also requires us to step away from the idea that there are simple fixes that could be applied to technology to ensure that it is inclusive. As I have demonstrated in Chapter 4, algorithmic bias is often seen as something that can be fixed through technical means. There is also a tendency to apply a checkbox mentality to diversity to show that one has considered diversity. Often, such checks only go as far as legally required. However, instead of seeing this form of diversityproofing as a oneoff process, how a specific technology relates to diversity will have to be questioned continuously. It is easy to see how this can lead to ‘analysis paralysis’ due to the sheer complexity that is entailed in such a process. This is particularly the case once intersectionality is taken into account. While this approach is challenging, it could be a way to ensure that technologies are more inclusive in design and use. We are thus able to design a future that is potentially more inclusive than the past. Embedding responsibility, accountability and governance in the AI supply chain has, however, proven to be challenging. I have mentioned earlier that it has been suggested that data sets should come with datasheets (Gebru et al., 2021). Others have suggested that data sets should have something similar to nutritional labels (Chmielinski et al., 2022). Such additional detail would include information on how data was collected and processed. However, a key concern with such approaches is the fact that AI supply chains consist of many actors; AI systems are assembled using an array of preexisting software to which a multitude of people contribute (Widder & Nafus, 2023). Widder and Nafus (2023) critique existing approaches to manage responsibility and accountability as requiring ‘panoptical visibility into the technology’ (Widder & Nafus, 2023, p. 8), alongside a control over this technology. Widder and Nafus (2023) argue this is rarely the case in the AI supply chain. Since much of the AI supply chain is based on modularity, it has been suggested that accountability should be located within CONCLUSION: UNWRITTEN RULES 137 the individual modules (Widder & Nafus, 2023). This echoes the feminist concept of ‘located accountabilities’ (Suchman, 2002). Additionally, the intersections between modules need to be strengthened, which can include, for instance, that customers check that data labellers are remunerated adequately (Widder & Nafus, 2023). Finally, Widder and Nafus (2023) suggest that modularity might be replaced completely, with a new system such as one based on principles of design justice (CostanzaChock, 2020). Another aspect of accountability relates to the fact that we often anthropomorphise technologies. One meaning of anthropomorphism refers to the process of attributing human characteristics or personality to a nonhuman entity such as an object or an animal (Oxford English Dictionary, 2023a). For instance, in Chapter 1, we saw how Weizenbaum was bewildered by the fact that people anthropomorphised the chatbot ELIZA (Treusch, 2017). In the books on the future of work that I analysed for this research, a similar tendency to anthropomorphise technologies could also be observed. In Chapter 2, I discussed how Baldwin (2019) describes how customers wanted to date or buy roses for Tiffany, a virtual assistant in a car dealership in Texas. Anthropomorphising is presuming humanlike intelligence in machines. Given the fact that AI often is said to emulate human intelligence, this tendency of anthropomorphising is probably not surprising. Although academic research is often critical about anthropomorphising while trying to avoid it, it opens an avenue to think about under what conditions a machine might carry responsibility and accountability. We normally situate accountability and responsibility in those who design new technologies. However, there is possibly scope to reflect on in how far machines could also carry responsibility for certain outcomes. Beyond that, there is also scope to think about accountability and responsibility as shared between various actors. This would then call for developing a more complex understanding of how the social and the technical are mutually constitutive and what this means for accountability and responsibility. Situating the Research and Future Research Avenues Finally, I want to reflect on how much the specific period of time during which the research was conducted shaped this book and what future research might explore. As I have outlined in Chapter 1, the research was developed prior to the Covid19 pandemic and the primary material was collated