Algorithmic social ordering: Towards a conceptual framework

Eyert, Florian; Irgmaier, Florian; Ulbricht, Lena Book Part — Published Version Algorithmic social ordering: Towards a conceptual framework Provided in Cooperation with: WZB Berlin Social Science Center Suggested Citation: Eyert, Florian; Irgmaier, Florian; Ulbricht, Lena (2018) : Algorithmic social ordering: Towards a conceptual framework, In: Getzinger, Günter (Ed.): Critical issues in science, technology and society studies: Conference proceedings of the 17th STS Conference Graz 2018, 7th-8th May 2018, ISBN 978-3-85125-625-3, Verlag der Technischen Universität Graz, Graz, pp. 48-57, https://doi.org/10.3217/978-3-85125-625-3 This Version is available at: https://hdl.handle.net/10419/191592 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by-nc-nd/4.0/ Algorithmic social ordering. Towards a conceptual framework EYERT, Florian, IRGMAIER, Florian, ULBRICHT, Lena Weizenbaum Institute for the Networked Society, Germany Introduction As the possibilities of digital technology continue to expand, its advance into more and more parts of everyday life and societal organization seems to assume a new quality. Increasingly, automatic information and decision-making systems are used to structure social processes, replace human judgement and generate order, as captured in the concepts of “algorithmic management” (Lee et al. 2015), “algorithmic regulation” (Yeung 2017a), “algocracy” (Aneesh 2009) and “governance by algorithms” (Just & Latzer 2017). It seems that governance and management is becoming more granular, subtle, responsive, encompassing and networked. Given the enormous possible consequences, we aim to contribute to a systematic understanding of these developments by analysing them from a particular conceptual perspective: the intersection of regulation and quantification. Our general approach is shaped by the question of social order and the mechanisms of its establishment and perpetuation. In the social sciences this foundational problem usually refers to societies in general, as in Hobbes’s political thought and its sociological interpretation by Talcott Parsons (1966), but it is also central for processes at the micro- and meso-level. Following Berger and Luckmann, we stress that social order is “an ongoing human production” (Berger & Luckmann 1991, 69), and propose to analyse the current societal transformations from the perspective of regulation as a specific form of such social ordering. With Julia Black, we understand regulation as “the sustained and focused attempt to alter the behaviour of others according to defined standards or purposes with the intention of producing a broadly identified outcome [...].” (Black 2002, 26) It is set apart from other forms of social ordering in four main dimensions. Unlike tradition or the habitus, it is intentional. Unlike spontaneous acts of violence, its perspective is one of permanence. Unlike government, it includes a variety of actors beyond the state. And unlike Weber’s domination, it is analytically independent of legitimacy. We believe that by developing a clear conceptual understanding of regulation and its technological components, we will be able to track the implications of the digital transformation for regulation and social ordering more generally. Informed by cybernetics, Christopher Hood, Henry Rothstein and Robert Baldwin (2001) have developed such an analytical framework for regulation, dissecting it into three components: information gathering, standard setting, and behaviour modification. In order to address the problems at hand, we draw on and aim to contribute to three main strands of literature. The sociology of quantification, valuation and classification has recently gained importance due to a an expansion of rankings, ratings and scores. It discusses the causes and consequences of the growing reliance on numbers in all parts of society. By transforming a continuous reality into a discrete representation, quantification promises to render human affairs increasingly commensurable and calculable, thereby increasing possibilities for control (Mau 2017). But while some authors have emphasized the role of digital technology in this process, until now we lack an understanding both of the concrete technological logic and the broader social implications of these processes. Supported by the spread of ever more powerful computer 48 technology, quantification increasingly assumes the form of an immense datafication. “Big data” and the “data deluge” are looming over an increasing number of segments of modern society. We believe that a conceptual understanding of algorithmic regulation could make a contribution here. Another crucial point of reference is the literature on regulation. While a large part of regulatory studies has pursued a “command and control” (Black 2002) view of regulation with a strong focus on the state and its legal tools, more recently a part of the literature has turned towards a wider and more decentred understanding of regulation. Furthermore, notions such as “code is law” (Lessig 2006) or “regulation by design” (Yeung 2008) have disclosed the role of technology in regulation. These analyses provide a valuable starting point, but especially regarding the technological dimension of regulation a lot of work still remains to be done. Finally, science, technology and society studies provide our primary access point to understanding the interaction between technological artefacts and social processes. Bruno Latour and others have proposed a shift towards a “sociology of associations” (Latour 2005) that cultivates an interest in the complex entanglements and assemblings of human and non-human actants. It is in such a way that we are enabled to study the myriad of small changes that add up to the rise of new digital forms of regulation and regulatory regimes. Bridging the three literatures promises a fruitful and original perspective from which digitalization becomes visible as a non-linear, socially constructed and inherently political process. In the following we will sketch a conceptual framework that can guide such research. A framework for studying algorithmic regulation Our starting point for the analysis of regulation is a classic framework from regulatory studies, first spelled out by Hood, Rothstein and Baldwin (2001). This perspective adopts a cybernetic angle by analyzing regulation along the three components of information gathering, standard setting and behaviour modification. Karen Yeung (2017a) has drawn on this approach in her recent analysis of what she, adopting the term from O’Reilly (2013), refers to as algorithmic regulation. Her systematization is a valuable step towards a conceptual framework that we would like to build on by extending it with a number of further analytical distinctions. The first component is that of information gathering, which is any form of collecting knowledge about some segment of reality that is to be regulated. It involves an epistemic process of construction based on information as well as their interpretation and modelling (Mahr 2003). The increasing datafication of both processes has led to the current duality of big data and machine learning. Karen Yeung has distinguished between reactive and preemptive information gathering, where the former uses given data to detect violations and the latter uses it to predict future behaviour (Yeung 2017a). For a further analysis, Bruno Latour’s notion of “immutable mobiles” provides a fruitful starting point. Immutable mobiles for him are inscriptions that are “mobile but also immutable, presentable, readable and combinable with one another” (Latour 1990a) and thereby allow control at a distance. In an iconic way, this is illustrated by the map that makes it possible to control the territory. With Latour we can analyze modern digital information systems as an environment for new and more effective forms of immutable mobiles, which crucially rely on what Adrian MacKenzie, in his ethnography of machine learning, has called “vectorization”: the “drawing together” of a variety of heterogeneous aspects of reality into a mathematically well described feature space in which 49 every concrete object can be represented as a vector (MacKenzie 2015). But every such vector is characterized by a complexity gap towards the reality it describes, which is why Susan Star concludes that no mobiles are completely immutable (Star 1995, 91). The local specificities undermine the modular and standardized descriptions of immutable mobiles. James Scott has developed the concept of “legibilization” (Scott 1998) to describe such simplifications aimed at control for the pre-digital world and we think it is highly relevant for the digital one as well. In order to analyze the strategic implications of such immutable mobiles as tools of legibilization, we propose to look at different kinds of algorithmic tools that can be employed in epistemic models for regulation. Examples would be simple deductive models along an IF-THEN logic, statistical models gained from training sets through supervised learning or mathematical optimization procedures that recommend analytically optimal behaviour. These tools, along with Slota’s and Bowker’s recent remark (2016) that knowledge is temporalized in new ways through algorithms, suggest that regulation might become more and more adaptive. The second, primarily normative component is that of standard setting. It refers to the goals that are to be obtained by regulation. Karen Yeung distinguishes between fixed and adaptive standards, i.e., those that are determined once and for all and those that can change over time (Yeung 2017a, 3). In order to become social realities, goals and standards of regulation undergo a process of definition, translation and calculation. No goal or standard ‘speaks for itself’, it needs indicators to be accessible and assessable. Goals and standards can take various forms: they can be shaped by a social domain or by a discipline, as in the case of the conflict between technical and legal standards; they can be area-specific, as in public health or commercial services; they can be abstract, like democratic values or customer satisfaction, or specific, like a 5% increase in users who are over 40 or a 1 pound weight loss within a month. The measurement can be quantitative or qualitative, direct or indirect, objective or subjective and absolute or relational. Whereas rules are always subject to interpretation due to the indeterminacy of language, code is binary and, even if not accurate, inherently determined. In order to be processed by algorithms, developers need to define their goals and standards in a way that can be applied to the data available. This requires the operationalization of broader goals that breaks them down into a variety of measurable entities. A specificity of algorithmic regulation is then the easy combination of more stable general goals and adaptive sub-standards. Behaviour modification, finally, encompasses the effective dimension of regulation: the actions through which a decision is enforced. It is the attempt to move the regulated entity closer towards a desirable state by deploying appropriate means. Yeung distinguishes between such systems that automatically administer specific sanctions or decisions and those that keep a human “on the loop” (ibid., 4), but while being useful, this distinction reveals little about how behaviour is actually modified. From the perspective of behaviour modification, regulation is successful to the extent that undesired forms of behaviour are rendered improbable. It is therefore dependent on a relation of strength and resistance or, in Latour’s words, on a relation between “programs” and “antiprograms” (Latour 1990b, 105ff.). “Programs” are those factors that contribute to steering the behaviour of a particular entity — be it human or not — in a direction deemed desirable. Modifying behaviour thus means mobilizing all available programs from which desired effects can be expected. “Anti-programs”, in contrast, refer to all the factors moving a particular entity in 50 a direction other than the one desired (Latour 1992; 1990b, 105). This category encompasses everything that increases the likelihood a regulatory attempt fails (from the viewpoint of the regulator). If a program is to be successful, it thus has to neutralize all anti-programs, i.e., all counteracting factors, so that the entity in question behaves in the desired way. In his famous vignette about a hotel manager trying to ensure that guests leave their room keys at the counter before going for a walk, Latour points out two strategies for developing successful programs: acting on the customers and their interpretation of the world or acting on the key itself by attaching a weight to it that makes it uncomfortable to carry around (Latour 1990b). For Latour, this is the distinction between incorporation and excorporation. Elaborating on these two notions is a useful way of discerning types of behaviour modification. Incorporation operates through the modification of the inner working of the regulated entity. In Foucauldian terms, we could speak of processes of subjectivation (Foucault 1990) and discipline (Foucault 1995, 135ff.): of internalizing subjective norms and values and creating bodily or mental routines and conditioning. An example of the former can be found in “lifelogging” and related practices of data-based self-monitoring that affect individuals’ subjectivities and, thus, their individual behaviour, e.g., by increasing sporting activity (Schaupp 2016). The latter form of incorporation is employed, for instance, in what Fogg calls “conditioning technology”, i.e., “[c]omputing technology [using] positive reinforcement to shape complex behaviour or transform existing behaviors into habits” (Fogg 2003, 53; see also Berlin Script Collective 2017, 24). Excorporation, in contrast, focusses on environments and situations rather than on individuals themselves. Madeleine Akrich’s (1992) analysis of how innovators inscribe normative assumptions into technology is instructive here. Building on a recent paper by the Berlin Script Collective (2017), we can distinguish three types of such behaviour modification through technological environments: coercion,inducement and the initiation of re-interpretations through the provision of knowledge. In each of these three cases, behaviour modification operates by (re)arranging the environment — e.g., through laws, taxes, or information campaigns — in such a way that the desired behaviour is rendered the most (instrumentally or normatively) rational option. To these three types of excorporate behaviour modification that make use of the rational capacities of individuals we add a fourth one which we term influence through non-rational properties: This type of influence makes use of the non-rational aspects of human behaviour and has been popularized by the debate about “nudges” and their mechanisms that exploit “heuristics and biases” (Thaler & Sunstein 2008; for digital environments, this kind of influence has also been explored by Yeung 2017b). The “status quo bias” for instance can be utilized through purposeful setting of technological defaults. As the fields of Persuasive Computing and Human-Computer Interaction show, digital technologies can be applied in all four of these types. Concluding this brief sketch of the framework, it is important to note that the three components should not be understood as successive phases. In reality, they are always intertwined, as the following case illustrates. The example of Uber As an illustration, the framework is now applied to the well-researched case of the personal transportation platform Uber and its regulation of drivers. Uber drivers usually register with the app and then indicate that they are currently available for work. While they are, they will be 51 matched with a nearby waiting customer, which they can accept or decline. If they accept, they are expected to pick up the customer and take them to their destination along a route suggested by the app. After the ride, the driver receives a rating from the customer. For the following discussion we apply our framework to the empirical descriptions by Lee et al. (2015), Rosenblat and Stark (2016) and Scheiber (2017). Information gathering: Firstly, our framework focuses on the ways in which the regulatees and the environment are modelled by Uber, i.e., which aspects are digitally represented, which information is gathered and how that information is then used to shape the whole work process. Uber records and processes various kinds of information through its app and the sensors integrated in smartphones. This includes the locations of drivers and customers, but also aspects of the drivers’ driving behaviour, for instance, their braking and accelerating. Furthermore, map data, traffic information and the location of other drivers are taken into account. However, other aspects such as the altruistic and non-economic motivations of drivers, the condition of the car or the road, the emotional state of the customer, the traffic policy of the city or the current weather are, to our knowledge, not taken into account. It has to be stressed at this point that it is Uber alone who defines what kinds of information are gathered — not the drivers, nor the customers. Standard setting: Uber’s general goal of maximizing revenue is broken down into a number of secondary standards which Uber deems fitting to achieve that goal: Optimally matching drivers with passengers for short pickup times, suggesting to take Uber-chosen routes, reaching a smooth driving behaviour, realizing a maximum price for the ride and improving the customers’ experience are among them. Through a universal rating system, standard setting procedures are partly delegated from the company to customers. After each ride, passengers evaluate their drivers through a five-star rating system, without being restrained in their choice of criteria. This feature renders the regulatory process decentral and dynamic because behaviour that got a driver five stars last month may not get them five stars today. Here, too, it is insightful to look at those standards that were seemingly not deemed relevant by Uber, such as the drivers’ health and happiness or ecological aspects. Behaviour modification: For the sake of brevity this analysis is limited to the excorporate behaviour modification of drivers. Interestingly, all of the four types described above can be found in the case of Uber. 1) Fear of coercion is used in the mechanism that drivers whose ratings fall below a certain level, or who repeatedly decline ride requests or cancel rides, lose access to their accounts and are excluded from the Uber market. Before this happens, the drivers receive warnings, which are — of course — directly intended to change their behaviour. 2) Uber uses monetary inducement in a highly fine-grained and dynamic manner in order to facilitate a favourable allocation between supply and demand. This is achieved through so called “surge pricing”, a temporary and local rise in fares after the system has identified a high demand in a certain area. This area will, for a limited time, be highlighted in red on the interactive map of the app. 3) The driving assistance by the Uber system corresponds to the initiation of re-interpretations. The most obvious means for this is the navigation function of the Uber driver app that ensures 52 drivers go in the right direction, which is essential for an enterprise specialized in transportation. Hence, local knowledge of streets and routes is not required for drivers. Preferably, drivers go along an Uber-chosen route. However, the criteria for suggesting a given route are not disclosed to the driver. It could be the shortest route or the route with maximum (or minimum) public visibility of the Uber car itself. 4) Fourth and last, sophisticated influence is exerted by exploiting the non-rational characteristics of drivers. For instance, Uber exploits people’s “loss aversion“, a phenomenon well-documented in behavioural economics, in a specific way: When drivers are about to log off, they receive a message reminding them of the money they would ‘lose’ by stopping now. For instance, one message reads “Are you sure you want to go offline? Demand is very high in your area. Make more money, don’t stop now!“ (Rosenblat & Stark 2016, 3768). Further applications The framework presented here is not limited to investigating the case of Uber. It can also shed light on other cases in which technological artefacts are used for regulation. On the micro-level, we can analyse forms of technologically mediated self-regulation. Fitbits and other kinds of wearable fitness trackers which are supposed to improve their users’ health are striking examples. Sleep trackers like WakeMate or energy use trackers like Wattvision enable similar kinds of self-regulation. A conceptual framework of regulation can add a fruitful perspective to the ongoing debate about the “quantified self” (Lupton 2016). On the meso-level, we can analyse organizational regulation that is supported by artefacts or software. For instance, Raffetseder, Schaupp and Staab (2017) have investigated the software Salesforce that is able to automatically assign incoming tasks to employees, according to parameters such as current workload or experience. And recently, Amazon developed a wristband that vibrates when employees in warehouses place goods in the wrong shelves (Yeginsu 2018) – a form of excorporate behaviour modification that affords itself to quick incorporation. Our framework also allows us to analyse how whole populations or sub-populations are regulated with the help of technology. Examples are computer-assisted forms of organizing the criminal justice system (such as the ambitions of predictive policing products like PredPol or sentencing software like Northpointe’s COMPAS), the automated curation of media content practiced by Facebook through its Newsfeed algorithm or the spectacular vision of a unified citizen score that currently haunts the newspapers with regards to the plans and actions of the Chinese government. Politicization Another advantage of this framework is that it sheds light not only on the ways in which algorithmic regulation operates, but also on the ways in which it is contested and politicized. Many of the conflicts around algorithmic regulation can be attributed to one of the three components. Some conflicts mainly focus on the gathering and the modeling of data. For instance, many privacy debates centre on the question who has the right to use which information, or which categories of information are really necessary for providing a service (for the case of Uber, see Zakrzewski 2015). In other cases, users request not less but more categories, for example 53 concerning gender identities on Facebook (Bivens 2017). In a third group of cases, users dispute the validity of inferred categories, for example the prediction of sexual orientation from Facebook posts. Standard setting is also a major site of contestation. Often, there are conflicts about the purposes that are being served and about the indicators against which success is measured. Many of debates also focus on the transparency and accountability of algorithmic decisions (Pasquale 2015; Ananny & Crawford 2018), for instance when people demand knowing how their creditworthiness is determined. When it comes to behaviour modification, conflicts arise about what instruments are legitimate and adequate. The current debates over the legitimacy of nudges (White 2013; Sunstein 2016) or about political microtargeting (Zuiderveen Borgesius et al. 2018) provide vivid examples. In essence, these are debates about whether certain forms of exerting influence can be reconciled with human dignity or democracy. The conflicts surrounding each of the three components also point towards what different forms of algorithmic regulation might look like. They show that there are other possible ways of gathering and processing information, other ways of defining goals, and other ways of influencing people. By criticizing the status quo, and by articulating other possibilities, the persons involved insist that there is a choice between different forms of establishing social order. In this sense, each of the three components, or all of them, can become politicized. Conclusion In this contribution, we have, in due brevity, sketched a framework for analysing regulation in different spheres of society. As an illustration of its applicability, we have shown cursorily how the regulation of drivers by Uber can be made sense of. Additionally, we have pointed towards other examples of information gathering, standard setting, and behaviour modification. We think that the framework presented above can make at least four contributions to science, technology and society studies. Firstly, it further differentiates the ways in which behaviour can be influenced through technology by taking into account a number of different theoretical angles. Secondly, focusing not only on influence, but also on data, models, and standards allows us to trace how different kinds of regulation are composed by establishing links between different building blocks. Thirdly, this framework helps us make sense of the digital transformation precisely because it is not limited to it; rather, we can compare digital and non-digital ways of regulation, and can thus determine what is and what isn’t unique about the digital transformation. Lastly, it allows us to distinguish various forms of political struggles around algorithmic regulation by directing scholarly attention to the regulatory components that are being politicized. Of course, the framework is no end in itself, but has to prove itself in empirical work. Future research will therefore be dedicated to refining it and using it to conduct comparative research. As a result of this, we hope to identify typical combinations of attributes across the three components. Such clusters would then allow a more detailed and contextualized understanding of algorithmic regulation and might point to strategies typical for specific spheres of society or levels of organization. Finally, such an analytical approach therefore promises a better informed diagnosis about how social order is established and maintained in the digital age. 54 Literature Akrich, Madeleine (1992), ‘On the De-Scription of Technical Objects’, in Bijker, Wiebe E. and John Law (Eds.), Shaping Technology / Building Society, Inside Technology. Cambridge, MA: MIT Press, 205–224. Ananny, Mike and Kate Crawford (2018), ‘Seeing without Knowing: Limitations of the Transparency Ideal and Its Application to Algorithmic Accountability’, New Media & Society 20 (3): 973–989. Aneesh, Aneesh (2009), ‘Global Labor: Algocratic Modes of Organization’, Sociological Theory 27 (4): 347–370. Berger, Peter L. and Thomas Luckmann (1991), The Social Construction of Reality: A Treatise in the Sociology of Knowledge, London: Penguin. Berlin Script Collective (2017), ‘Comparing Scripts and Scripting Comparisons. Toward a Systematic Analysis of Technologically Mediated Influence’, Technical University Technology Studies Working Papers, Berlin: Technische Universität Berlin, http://www.ts.tu-berlin.de/fileadmin/ i62_tstypo3/TUTS-WP-1-2017_ComparingScripts.pdf [downloaded 22 July 2018]. Bivens, Rena (2017), ‘The Gender Binary Will Not Be Deprogrammed: Ten Years of Coding Gender on Facebook’, New Media & Society 19 (6): 880–898. Black, Julia (2002), ‘Critical Reflections on Regulation’, Australian Journal of Legal Philosophy 27: 1– 37. Fogg, B. J. (2003), Persuasive Technology: Using Computers to Change What We Think and Do, Amsterdam/Boston, MA: Interactive Technologies. Foucault, Michel (1990), The History of Sexuality: The Use of Pleasure, New York: Vintage Books. Foucault, Michel (1995), Discipline and Punish: The Birth of the Prison, New York: Vintage Books. Hood, Christopher, Henry Rothstein, and Robert Baldwin (2001), The Government of Risk: Understanding Risk Regulation Regimes, Oxford/New York: Oxford University Press. Just, Natascha and Michael Latzer (2017), ‘Governance by Algorithms: Reality Construction by Algorithmic Selection on the Internet’, Media, Culture & Society 39 (2): 238–258. Latour, Bruno (1990a), ‘Drawing Things Together’, in Lynch, Michael and Steve Woolgar (Eds.), Representation in Scientific Practice, Cambridge, MA: MIT Press, 19–68. Latour, Bruno (1990b), ‘Technology Is Society Made Durable’, The Sociological Review 38 (1 (Suppl.)): 103–131. Latour, Bruno (1992), ‘Where Are the Missing Masses? The Sociology of a Few Mundane Artifacts’, in in Bijker, Wiebe E. and John Law (Eds.), Shaping Technology / Building Society, Inside Technology. Cambridge, MA: MIT Press, 225–258. Latour, Bruno (2005), Reassembling the Social: An Introduction to Actor-Network-Theory, Oxford: Oxford University Press. Lee, Min Kyung, Daniel Kusbit, Evan Metsky, and Laura Dabbish (2015), ‘Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers’, in Begole, Bo and Jinwoo Kim (Eds.), Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, New York: ACM, 1603–1612. 55