Paddling in the Shallows: A Manifesto for Doing Better Research with Games

Epigram #2 — Paddling in the Shallows: A Manifesto for Doing Better Research with Games Michael Heron; Michael Crabb Haunted Playground Epigrams (No. 2) Haunted Playground Press Abstract. Academic publication often does not treat popular culture as an equal partner when it comes to setting the parameters of research work. More than anything else, academic inquiry into popular culture outside of specialist gaming venues is defined by a lack of appropriate context setting. The result is that irrelevant cultural touchstones receive undue prominence with the concomitant negative impact on broader relevance to the community. Examples include board game papers that never stray from Ludo, Monopoly, Catan and Scrabble and video game projects that act as if the past two decades of game development didn’t happen. The costs to our discipline are considerable, particularly when it comes to the reputation of academia as a first-order source of insight and commentary into popular culture - our own work may draw from the best of what contemporary gaming culture has to offer, but it is in other fields that games are most commonly encountered within research. In this paper we outline this problem with reference to the guiding philosophy ‘nothing about us without us’ and provide some actionable suggestions for how we can do better going forward. Cite as: Heron, M., and Crabb, M. (2025). Paddling in the Shadows: A Manifesto for Doing Better Research with Games.Haunted Playground Epigrams (No. 2). Haunted Playground Press. Available online at: https://haunted-playground.com Haunted Playground Epigrams #3 1 Introduction One of the most effective ways for academic research to demonstrate real world relevance is through linking to the interests of a mainstream audience Bastow et al., 2014; Morrell and Duncan-Andrade, 2002; Watermeyer, 2016. One of the most interesting and effective unions where this is most easily observable occurs in the joining of academic inquiry to popular culture Hill et al., 2017; Schudson, 1987. This is separate and distinct from specialist academic research in these areaa - in this circumstance researchers are expected to be conversant with the depth and breadth of the material as a fundamental prerequisite of engagement with cutting edge work. Instead, we are discussing a kind of ’impact appropriation’ that takes the form of a looser association where a complex technique is applied to a product of popular culture to create a primarily academic research output. One might apply a distributed artificial intelligence technique to a particular video game to learn more about how agents can collaborate in a formal system Koeman et al., 2018; use a television show such as Game of Thrones or Battlestar Galactica to discuss issues of international relations Dyson, 2015; or use a board game like Settlers of Catan to explore issues of constrained game theory Guhe and Lascarides, 2014. Typically these research outputs are primarily focused on the academic technique and exploring its implications. The cultural artefact represents little more than an innovative application to a mainstream touchstone. The end goal of such work is inevitably to sharpen or refine a complex technique. This is outwith the scope of our discussion. Thereexistsasecondaryforminwhichpopularcultureisintegratedintoacademia, andthat’sthrough the application of a discipline lens with the expectation that the subject of the integration is improved through that application. Examples of papers in this domain include Emmen and Lampropoulos, 2014; Hardy, 2019; Heron et al., 2018b. When there is a good match between cultural artefact and academic discipline, this can produce excellent and impactful work where the expertise of specialist researchers can offer insights into the production of popular culture. However there is a recurring problem in how this work is created and reported. In many cases it addresses issues of peripheral importance, or applies important techniques to products of marginal value. The end result is that much good academic work is squandered on solving problems that don’t need to be solved, for contexts that simply don’t matter. For pure academic research outputs, this would be caught at peer review. For novel syntheses with popular culture, these outputs are much more difficult to evaluate for reasons we will outline later in this paper. We do not present here an empirical evaluation of an issue. We do not wish to name and shame authors, or undermine the often high-quality academic work that is being done. Instead we offer this as a form of substantiated academic essay drawing from personal insight and peer review experience. We appreciate there are counterexamples to every point we raise in this paper, and we do not argue that these problems are universal. Instead we present this as the beginning of a discussion on the ways in which academia can often dismiss the value of fandom expertise, and the consequent deleterious effect this has on academic impact and popular discourse of academic work. Academia has a reputation problem amongst many fandoms Larsen and Zubernis, 2011; Suomela et al., 2019, and the fact that this problem is not well discussed in the literature is a symptom of the larger problem. We reflect upon why this might be, and whether there are remedies we can, and should, employ - both in our role as games academics and as academics attached to other disciplines. Our focus here, given the enthusiasms of the authors, will be on popular gaming culture. The observations however are applicable to a wide variety of topics. In conversation with the wider community about this issue, we have sometimes received push-back Paddling in the Shallows 1 Haunted Playground Epigrams #3 on the grounds it is not for academics to critique what other academics choose to publish. However, we view this paper as a contribution with regards to the self-correcting nature of science. We all have a responsibility to identify what we consider to be research morbidities, and do what we can to respectfully correct them. 2 Paddling in the Shallows To avoid unfairly focusing on individual research outputs, we will illustrate this issue with an extreme hypothetical example. Imagine in your capacity as an academic peer reviewer you are sent a paper that has been submitted to a popular, high ranking conference in the field of human-computer interaction. The paper is on techniques to improve player satisfaction in a game of Pacman. The work is excellent - academically robust and rigorously explored. The hypothesis is set out clearly, defended well, and the results are both quantitatively and qualitatively convincing. The work, in other words, is first-rate and represents undoubtedly years of difficult effort on the part of at least one PhD student and their academic supervisors. The authors are right - they absolutely can improve the experience people have playing Pacman. But only Pacman. The work is so tied into its context that while it has some insights that can feed forward, they are sufficiently general that they lack any real concrete weight. The problem is that Pacman is a game that has long ceased to have anything other than niche relevance to the topic of computer games as a cutting edge professional discipline. The game still has something left to offer - the competitive retro gamer scene continues to thrive on the periphery of the hobby Diah et al., 2015; Reisner, 2016. However improving the satisfaction of people playing the game reflects at best a marginal contribution to the popular context that the work has addressed. The paper in other words does something great within an application domain that is largely irrelevant. In and of itself this represents little more than a squandering of individual time - an indulgence that can be largely ignored in the wider economy of academic attention. However, let’s imagine that set against this backdrop the authors of this hypothetical research paper argue in the conclusion that their work represents an important contribution to the literature. They have to after all, it’s one of the important conditions under which papers are inevitably accepted to conferences ACM, 2018. In these arguments it becomes clear that Pacman was chosen not because it was an ideal test-case for the work, but rather that the authors didn’t know enough about computer games to understand that it was no longer broadly relevant in popular gaming culture. Their expertise, justifiably, was in the skilful and effective exploration of an academic discipline. They did not consider games to be of the same academic importance. This represents an extreme and fictional example of a problem that is genuinely threaded through modern academic discourse. Computer games are sufficiently integrated into society that it is clear to most readers that attempting to draw modern conclusions about gaming based on a 1980 release would lack credibility. Within a less conspicuous example it becomes a much more difficult process for a peer-reviewer, or an academic researcher, to evaluate validity. We have personal experience for example of peer reviewing an analysis of board games that used the ’best sellers’ list of Amazon to curate its selection 1and an accessibility paper that looked to make Ludo more accessible through 1At the time of writing, this would be Connect 4, Candy Land, Tapple, Zingo Bingo, Codenames, Sorry, Sequence, Checkers, Trouble, and a foam dart board Paddling in the Shallows 2 Haunted Playground Epigrams #3 the use of felt for tactility. We have seen literature reviews on gaming that ignore much of what is common state-of-the-art within Game Studies. We know to what extent computer games have advanced over four decades, but it’s harder to assess this for other, more obscure gaming genres. The many and varied sub-fields of gaming make this difficult for specialists within the larger parent discipline. Researchers may have written extensively on video games and board games, but would be lost at sea were they to turn their attention to the topic of role-playing games or LARPs without an extensive effort to acclimatise. It’s very clear, when reading non-gamer papers that intersect with our other areas of personal enthusiasm, that many research outputs are developed without taking the critical step to situate them within their popular culture context. Their authors have not peered through the ‘Johari Window’: researchers are unaware of what is known to others, but unknown to the self Luft and Ingham, 1961. Consider the role that Settlers of Catan (1995) has in board-game related studies. A small sampling of recent papers that have substantially incorporated it or its design include Austin et al., 2019; Boda, 2018; Chappin et al., 2017; Dobre and Lascarides, 2018; Illingworth and Wake, 2018. As above, we make no judgement in this paper on whether these represent valuable contributions to the discipline or the literature. Some of them for example are novel extensions of Catan designed to leverage its wide recognition towards more socially-positive goals. We merely note that Catan is almost thirty-years old now. It does not seem unreasonable to assume that considerable advancements may have been made in the intervening time with regards to board game design. No discipline ever holds still - innovation is a constant even in fields where the physical form of a modern artefact looks very similar to ones of earlier eras. Indeed, given recent work in hobbyist board gaming Heron et al., 2018a, 2018b, we can with confidence state that Catan is no longer a game that can be considered a suitable proxy for modern board game sensibilities. It is still played regularly, but it is a reflection of a design ethos that has been left considerably in the past. For many research outputs, this lack of recency or relevance leads to two interlinking and related problems: 1. The research outputs address research questions through the use of cultural properties that are often decades out of date. 2. High quality academic work is performed, often at great cost to an institution, to produce solutions or insights that no longer have real world relevance. The end result is a perpetual paddling in the shallows, where progress on real world issues is rarely made because the frame of contribution is anchored too heavily in the past. The result of focusing on Catan (1995) means that similar effort is never invested on more modern games that reflect the payload of the intervening decades of game design innovation. Attention in this scenario represents a kind of Matthew Effect Merton, 1968 where academic interest is proportional to awareness. Catan is a game that even those with only a passing knowledge of the hobby will recognize. That Catan is such a common touchstone in the application of academic techniques to board-games means that inevitably others are drawn to it for no deeper reason than its sheer ubiquity. At exactly the time it should be fading out of relevance, it attains an unshakeable permanence simply as a consequence of being a fixture of the academic literature. Instead of academics moving on to richer and newer games, the field becomes ossified to the point of ritual, with the appropriate observances regularly invoked as a kind of shibboleth woven through the bibliography of a thousand papers and dissertations. This speaks to a concern with how much of this research is conducted. It’s a byproduct of a kind of intellectual insularity that is particular to many of us working in academia. Our basis for reference in Paddling in the Shallows 3 Haunted Playground Epigrams #3 academic work is the academic literature and we are correct to be anchored in the work that has come before. However, many of the most meaningful observations and insights relating to wider popular culture are to be found nowhere in the scholarly records. Or, at best, they are reflected second hand through a process of academic ethnography (c.f. Kim, 2020; McCardle, 2003; Paris, 2016). Much of the cutting edge game design and critical discussion does not take place within academic journals. Game developers and designers rarely consult with academia in the production of real-world outputs. Substantial perspectives that would enrich a background analysis are missing from research outputs because their most significant proponents have never gone through an academic publishing process. Inevitably this is reflected in a paper as an area being described as ‘unexplored’, as opposed to being recognised as ‘undiscovered’. Discounting of anything outside of the academic ivory tower is detrimental in understanding the broader community and concepts that are required within niche and emerging subject areas. If one weretolookforthebestboardgamesintheworld,lookingatspecialistresourcessuchBoardGameGeek (BGG) would be a far better proxy of crowd-sourced assessed quality than simply picking the top selling board games from Amazon Zalewski et al., 2019. And yet BGG has a presence in the scholarly literature that is notable primarily for its absence. We have discussed why this is a problem for academia in terms of squandered effort, but it’s also a problem for the enthusiast groups to which this academic work is (mis)applied. There are immensely vibrant online communities that discuss and dissect popular culture and they pay attention to academic work when it comes their way. Often the response to substantial publications is extremely negative. Even a cursory researcher investigation into these areas of enthusiast attention would have shown how misaligned the academic goals were with those of the wider community. This has a massively negative impact on the perception of the work. Just as hobbyist communities are ill-served by academia, academic practices are not well understood by hobbyists. Were this simply a case of a small number of objections raised by a small number of people, that would be a significant enough downside given the disproportionate power such influencers have to shape modern opinion Holland, 2019; Jin et al., 2019. Coupled to this though, Reddit, Facebook and Twitter have greater search engine prominence than most academic outlets. A search for a paper may very well find that many of the results reflect not sober analysis but instead raw online anger. This comes with an overall reputation cost to the researcher. Consider Heron et al., 2014 as a specific example. This was generally well received as an academic output, but also came under intense criticism on Twitter, Reddit and other platforms. A search of the academic literature only shows the paper cited in appropriate contexts, with appropriate academic commentary. A search on the wider Internet demonstrates a much more significantly negative response. The summary of all of this is that we often see other disciplines, when they make use of games, are sometimes solving problems that no longer exist in problem domains that are no longer relevant. Instead of gaining positive impactful attention we find research will often rile up those that should be the most enthusiastic about the work. We see great things that didn’t need to be done, and we are seeing that too often and at too great a cost to the reputation and effectiveness of academia as a whole. That specialist game researchers do not make these mistakes so often is largely irrelevant - we are driven by the whorls and eddies of much stronger and stranger tides. Paddling in the Shallows 4 Haunted Playground Epigrams #3 3 Nothing About Us Without Us Core to this problem is what seems to be a form of assumption that pop cultural fields are not seen to require the same level of dedication to investigation as other academic disciplines. Simply, these domains of application ‘look easy’ and as a result many researchers assume that a shallow, and often distantly second-hand, appreciation of the topic is enough. And in many cases, they’re right insofar as academic acceptability is concerned. The academic discipline is mostly seen as pre-eminent and important, the domain to which it is applied is not. We can agree perhaps that pop culture may not have a particular depth of academic literature associated with it. Or at least not within the body of work considered to be of sufficient scholarly prestige to merit the honour of citation. It is a misapprehension though to assume that academic skill translates automatically into popular culture expertise. When performing an operation, one would not ask the surgeon to step in for the anesthetist simply because of their mastery of one domain. What looks simple from the outside, given the dearth of scholarly analysis, sometimes masks a deeper complexity. That depth often has a vibrancy and energy that is precisely because it is divorced from the formality of the scholarly process. Low hanging fruit sometimes only appears this way from a sufficient distance, and when viewed from close up will provide a greater (and often more rewarding) challenge. Academic expertise being assumed to act as an effective proxy for popular-cultural domain knowledge is a manifestation of the Dunning-Kruger effect Dunning, 2011. A lack of immersion in the complicated and often fan-driven canon of a hobby heron2017pacman results in the perception that one, without exerting any effort, is qualified to identify problems and comment upon their solutions. It is broadly equivalent to seeing a cinematic trailer for an Avengers movie and then assuming one is qualified to talk about the complex interconnectivity of the Batman franchise. One might be able to mimic some of the language and conceptual frameworks, but only enough to convince others that have similarly a shallow understanding of a complex transmedial property that represents in itself a nascent modern mythology Reynolds, 1994. Pop culture commentary is a form of collaborative myth-making Bacon-Smith, 1992 and assigning a lesser intellectual burden simply because it is a form of folk wisdom is sorely myth-taken. We don’t expect that all academics become subject matter experts in the popular culture domains to which they apply their expertise. Instead we propose that we begin to treat popular culture expertise as a distinct, specialist discipline of equal importance to those more broadly accepted as academic. In multi-disciplinary work, we don’t expect everyone to be equally conversant with each discipline - instead we expect experts to collaborate and combine their strengths with appropriate give and take and respect on all sides. We expect all partners to be part of scoping, evaluation, and in the planning of impact. The infrastructure of desk checks, expert peer review and post-publication commentary has no formal spaces in which these enthusiast perspectives can be included in anything other than an incidental and accidental basis. Where pop-culture expertise is involved in review, it is usually as a happy accident of convergence - where an academically qualified peer also happened to have subject matter expertise in the topic of inquiry. That is a role that many of us reading this paper will have performed - to be the sole voice in the peer-review process that questions fundamental pop-cultural assumptions about game-related papers. ‘Nothing About Us, Without Us’ is a popular slogan in many disciplines, particularly those that focus on political representation Ryder et al., 2014, disability studies Charlton, 2000 and the wider Paddling in the Shallows 5 Haunted Playground Epigrams #3 inclusion agenda Sherrill, 2006. The core idea is simple - nothing should be decided for a particular group without the meaningful participation of that group in setting policy and evaluating outcomes. Some academic disciplines, for example the accessibility community, have wholeheartedly adopted this concept in principle if not necessarily always in practice. While this makes work more logistically difficult to carry out Gilbert, 2004; Rimmer et al., 2004 it lends a genuine authenticity to the results that would otherwise be lacking Radermacher et al., 2010. What we argue here is that ‘nothing about us, without us’ is a principle that also applies to academic commentary on popular culture. In many ways this represents the only true way to ensure acceptance and real-world impact of the results. Importantly it also ensures that good work is not wasted on irrelevant outcomes. It ensures that we take modern culture as seriously as we do any other discipline, with the expectation that the quality and impact of relevant work is enriched as a consequence. What we currently observe is an expression of silo thinking Shaw and Frost, 2015 where communities of enthusiasm, communities of practice, and communities of research are not meaningfully collaborating towards shared aims. ‘No taxation without representation’ was a common slogan during the American Revolutionary war, and a fundamental principle upon which the Founders case was built Dorfman, 2007. It is applicable here - public money, in the form of scholarly grants and academic time, is often being spent in the production of these outputs. It is reasonable, in our view, that we work to ensure that the taxes that support such outlays are properly representative. The discontinuity and erracity of the interface between academia and industry is a well assessed and discussedproblem(c.f. Ankrah andOmar, 2015; Ankrah etal., 2013; Kingand Persily, 2018), and itis one of critical importance. However, here weargue that the interface between academia and fandoms is equally important and equally worthy of being formally addressed. Increasingly we expect public and private partnership to be at the core of grant seeking behaviour. We have as yet no equivalent expectation of enthusiasts being given a seat at the table. We posit then a public-private-fandom triad that offers a potential Triforce Cirilla, 2020 of academic impact if only wielded correctly. 4 A Philosophy of Inclusion for Fandom It is not our intention here to simply complain about a problem. Instead, drawing from best practise in user-focused disciplines, we present a series of aims that would go a considerable distance towards resolving these systemic flaws in the academic process. 4.1 Google Search as well as Google Scholar Academia has long understood the importance of context - the background section in an academic paper often represents its most dedicated scholarship Booth et al., 2016. The literature review stage of a PhD is often expected to last six months to a year with that time being invested simply in developing mastery over a field of study. Even in degree level projects at universities, we expect students to understand the value of the academic literature even if most will never fully integrate it into a professional working life. This though represents a form of academic arrogance, that the quality of academic work is so far in excess of more popular outlets that it occupies a rarefied strata that requires deference. So robust is Paddling in the Shallows 6 Haunted Playground Epigrams #3 this view that it has seeped into public consciousness, where ‘peer reviewed’ has become a kind of synonym for ‘correct’ when discussing primary contributions to the literature Heron and Belford, 2015. This despite the many structural flaws in peer review Beall, 2018; Stahel and Moore, 2014 and the limitations in what it can realistically validate Easley et al., 2013; Ferguson and Heene, 2012 . Some resources aimed at helping non-specialists do research work will even rank reference sources in tiers, with ‘peer reviewed academic journals’ obviously occupying the highest of these. The result of this is a dismissal of, and occasionally disdain for, rich sources of insight - particularly whenthe locusof enthusiasm fora topicis notfoundwithin academia. Weargue as ageneral principle here that literature reviews should not just focus on the academic literature. This is especially critical when addressing issues of popular culture because the vast majority of discourse on the topic is never conducted academically. Given the energy of an average fandom subreddit or forum, it’s likely that any ’novel’ observation has already undergone a much more rigorous, and adversarial, peer review than any scholarly contribution to the academic literature can expect. The work that the authors of this paper have done has been peer-reviewed in academic outlets and publicly reviewed on Reddit threads. We can assure you, through personal experience, that academic peer reviewing is far less febrile than the review conducted by an enthusiast fandom. We see abundant examples of work in the literature where a simple search engine query of key terms would have yielded dramatic improvements in the selection criteria of research outputs. As accessibility researchers, we can usethe work ofMeeple LikeUs 2asanexample. Wehave seenseveralinteresting papers on the topic of board game accessibility where the impact of the work was limited as a result of poor choice of games for analysis. There is little evidence of focusing academic inquiry on difficult issues or ‘wicked problems’ Buchanan, 1992 - arguably the ones that academia is best positioned to address. Instead effort has gone into making marginal improvements for games that were already broadly accessible. The work would have been better, the impact greater, and the research more fun to do if the authors had simply typed ‘board game accessibility’ into Google and then clicked the first link. In this we don’t make the claim that the non-academic work is necessarily superior to the work done by other authors. Rather we argue that a standard search of the non-academic literature can show that the territory for a planned expedition is often better explored than may have been considered. In the ideal circumstance, pre-existing work may have already mapped out points of special interest for the consideration of future explorers. For those of us acting as peer reviewers for gaming-related papers within other academic disciplines, we are uniquely placed to be able to bring a degree of enforcement here. We should always be asking, as a first-order inquiry, where a literature review demonstrates its non-academic scholarship and ensuring the case is made within the text for why a gaming property was selected as the subject of deeper investigation. 4.2 Interest of Enthusiasts as well as Researchers One easy way to ensure that the work about to be undertaken is likely to be relevant to a hobbyist audience is find an enthusiast and see how excited they are when you give them the elevator pitch. There are many sites where fans gather, and simply dropping a post saying ‘We’re planning to do this, what do you think?’ will likely result in dozens of responses exploring the possibility space around the idea. If you’re planning to look into economic behaviour seeking in Monopoly, head over to 2https://meeplelikeus.co.uk Paddling in the Shallows 7 Haunted Playground Epigrams #3 /r/boardgames on Reddit and say so. Give it a day and you’ll find a hundred replies telling you why you should be looking at Chinatown instead, or Lords of Vegas. Perhaps they’ll explain why Sidereal Confluence is a better choice for the behaviour you’re looking to explore, or why the tightly constrained auctions tied to the cumulative and emergent value propositions of Modern Art are more appropriate. If a researcher doesn’t know any of those game exist, that’s shows exactly why it’s important to discuss thoughts with those that do. If researchers don’t wish to expose their early ideas to the crucible of public criticism, they’ll find enthusiasts in their extended social circle easily enough. People are out there, and they’ll be delighted to be taken seriously. They represent stakeholders in the hobby, but also potential advocates for the work. Morethanthisthoughtheyrepresentbrakesonthewastefulexpenditureoftimeandeffortonmarginal or trivial outcomes. One cannot expect to become an expert in all topics. Leaning on the expertise of enthusiasts means that potential work can be refined into a form that actually captures the attention of those that it is presumably intended to serve in at least some capacity. Measuring the enthusiasm of enthusiasts as a precursor check before conducting costly work is invaluable, and the changes they suggest in terms of subject matter relevance can be priceless. They know the subject, they know its canon and its texts, and they are an invaluable source that research work will be immeasurably better for tapping. Again, those of us acting as peer-reviewers can provide this suggestion as part of the feed-forward component of our contributions. 4.3 Popular Cultural Artefacts as well as Academic Literature As an ongoing part of research, we advocate that academics treat popular cultural artefacts as primary contributions to the literature. An academic working in games as a discipline for example should play games the same way they read papers - focusing on key titles, ensuring they dip into interesting genres while tightly and rigorously controlling for quality, relevance, and recency. Games are the primary text of our discipline. Instead of thinking of the bibliography versus the filmography/ludography we should require the scholarship to be integrated and treated equivalently. When publishing work on the topic of popular cultural artefacts, the literature review about the primary texts of that discipline should demonstrate the same confidence in the material that we’d expect from the academic element. It’s important that reviewers are able to assess, without expertise, the likely effort that went into identifying appropriate cultural artefacts. Discussion of the background should permit a non-specialist to evaluate the timelieness and appropriateness of the choices made. One could even go further and argue that when outlining a contribution to the literature, it should include both the popular and the academic contributions equally. Again, advocacy here within our roles as peer-reviewers is important, but so too is workingwithin thebureaucraticstructures oforganizations such asthe IEEEandthe ACM to change the formula by which academic work is assessed and published. 4.4 Include Enthusiasts as Co-Authors It is also important to include enthusiasts as meaningful contributors to academic production if at all possible. In this, tempering enthusiasm with academic rigor may add a complexity to collaboration Hyland, 2004 - by its very nature, the (often idealised) dispassion of scholarship does not always sit Paddling in the Shallows 8