Autonomation, Not Automation: Activities and Needs of European Fact-checkers as a Basis for Designing Human-Centered AI Systems

Autonomation, Not Automation: Activities and Needs of European Fact-checkers as a Basis for Designing Human-Centered AI Systems ANDREA HRCKOVA∗,Kempelen Institute of Intelligent Technologies, Slovakia ROBERT MORO,Kempelen Institute of Intelligent Technologies, Slovakia IVAN SRBA,Kempelen Institute of Intelligent Technologies, Slovakia JAKUB SIMKO,Kempelen Institute of Intelligent Technologies, Slovakia MARIA BIELIKOVA,Kempelen Institute of Intelligent Technologies, Slovakia To mitigate the negative effects of false information more effectively, the development of Artificial Intelligence (AI) systems to assist fact-checkers is needed. Nevertheless, the lack of focus on the needs of these stakeholders results in their limited acceptance and skepticism toward automating the whole fact-checking process. In this study, we conducted semi-structured in-depth interviews with Central European fact-checkers. Their activities and problems were analyzed using iterative content analysis. The most significant problems were validated with a survey of European fact-checkers, in which we collected 24 responses from 20 countries, i.e., 62% of active European signatories of the International Fact-Checking Network (IFCN). Our contributions include an in-depth examination of the variability of fact-checking work in non-English-speaking regions, which still remained largely uncovered. By aligning them with the knowledge from prior studies, we created conceptual models that help to understand the fact-checking processes. In addition, we mapped our findings on the fact-checkers’ activities and needs to the relevant tasks for AI research, while providing a discussion on three AI tasks that were not covered by previous similar studies. The new opportunities identified for AI researchers and developers have implications for the focus of AI research in this domain. CCS Concepts: •Human-centered computing → Empirical studies in collaborative and social computing;Empirical studies in HCI; Additional Key Words and Phrases: fact-checkers, misinformation, disinformation, human-centered artificial intelligence, human-information interaction ACM Reference Format: Andrea Hrckova, Robert Moro, Ivan Srba, Jakub Simko, and Maria Bielikova. 2025. Autonomation, Not Automation: Activities and Needs of European Fact-checkers as a Basis for Designing Human-Centered AI Systems. ACM J. Responsib. Comput. 1, 1, Article 1 (January 2025), 44 pages. https://doi.org/10.1145/3764592 1 Introduction A fact-checker is “a person whose job is to make sure that the facts are correct, especially in something published” [ 1 ]. These professionals work either in larger newspaper agencies (e.g., in AFP, Deutsche Welle, Washington Post) or small or medium-sized NGOs focused just on factchecking (e.g., Full Fact, PolitiFact.com, FactCheck.org, etc.). Many aspects of the fact-checker’s work, as well as the problems this profession faces, differ across the positions (e.g., the required education) or remain unclear (e.g., stages in the fact-checking process; see Fig. 3and geo-cultural Authors’ Contact Information: Andrea Hrckova, andrea.hr[email protected], Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia; Robert Moro, [email protected], Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia; Ivan Srba, [email protected], Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia; Jakub Simko, jakub. [email protected], Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia; Maria Bielikova, maria.bielikova@ kinit.sk, Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia. ©2025 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ACM Journal on Responsible Computing,https://doi.org/10.1145/3764592. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:2 Hrckova, et al. differences – see Fig. 1). Nevertheless, research shows [ 46 ] that fact-checkers define their role and work as a public service and develop activities to instill a culture of factuality that they value. Exemptions include fringe state-supported fact-checkers associated with authoritarian governance structures that weaponize fact-checking practices and exploit or mimic the social standing of accredited fact-checkers to fact-check for national interests [56]. Research literature reports varying findings on the extent to which fact-checking reduces misinformation, with estimates ranging from minor effects [ 21 ], to moderate [ 68 ], to substantial [ 17 ] reductions in belief in misinformation after exposure to a fact-checking label. In these studies, participants assessed the accuracy of various real and fabricated statements, either with or without a fact-checking tag. Such discrepancies may stem from contextual differences in how misinformation is evaluated, including the type of misinformation, the platform and format of the fact-check, the source of the debunking information, and the metrics used to measure effectiveness (ranging from a 3-point Likert scale on the falsity of the news [ 21 ] to a 7-point Likert scale on agreement with the news [ 17 ]), as well as prior beliefs or personality characteristics of the participants. For example, knowledgeable individuals with partisan views tend to scrutinize fact-checking more [ 98 ]. Also, who the fact-checker is matters when assessing the success in debunking misperceptions [ 93 ]. However, in general, fact-checking has been shown to be effective in combination with information literacy interventions in reducing political misinformation in terms of lowering issue agreement and perceived accuracy [ 35 ]. Fact-checkers also successfully lowered the agreement with attitudinally congruent political misinformation and can help overcome political polarization [36]. Despite these positive outcomes of fact-checking, active cooperation with social media platforms such as X or Facebook has been ceased recently or is in danger of being ceased despite previous statements that ‘it works’ 1 . Without the cooperation with social media, the dissemination of factchecks will be even less effective. Also, there is currently a trend of replacing the professional fact-checks by community notes, i.e., the consensus of regular users. On one hand, community notes might be an effective approach to mitigate trust issues with simple misinformation flags [ 28 ]. On the other hand, it may result in the fact-checking process being more subjective and susceptible to the bias of community members. More specifically, previous studies showed that community notes tend to be biased towards fact-checking posts from large accounts [ 65 ]. Also, contextual features – in particular, the partisanship of the users – are far more predictive of judgments than the content of the posts and evaluations themselves (e.g., users preferentially challenge content from those with whom they disagree politically) [ 5 ]. Finally, a recent study [ 16 ] showed that community notes are more of a complement than a replacement for traditional fact-checking since community moderation relies on professional fact-checking – more specifically, community notes on posts linked to broader narratives are twice as likely to reference fact-checking sources compared to other sources. Although growing, the number of professional fact-checkers remains low in comparison to the vast amount of misinformation. The latest census by the Duke Reporters’ Lab 2 identified 378 active fact-checking projects worldwide. Only 88 organizations worldwide (34 in Europe) actively cooperate with social media platforms through the International Fact-checking network 3 , and in some cases, there is merely one fact-checker per country. Imbalance between such scarcity of fact-checkers and misinformation overload causes just a fraction of potential false content is being checked. Furthermore, the work of fact-checkers is laborious and sometimes repetitive. However, 1 https://web.archive.org/web/20250110130651/https://www.facebook.com/formedia/blog/third-party-fact-checking-howit-works 2https://reporterslab.org/fact-checking 3https://ifcncodeofprinciples.poynter.org/signatories ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:3 there is an opportunity to improve balance by providing appropriate technological support for challenging or repetitive tasks. While there is an increasing body of research on automation of the fact-checking process, especially with the utilization of Artificial Intelligence (AI) technologies, full automation of the whole fact-checking process is still viewed with skepticism by the fact-checkers as well as researchers [ 2 , 27 , 44 , 53 ]. The main reason is that some parts of the fact-checking process require human judgments or actions. Tasks such as assessing the credibility of the sources of evidence are currently not fully automated with sufficient trustworthiness (e.g., by providing appropriate explanations), accuracy, or generability (e.g., checking complex recent claims where evidence may still be missing) [ 58 ]. The real potential of AI-based tools, therefore, currently lies in assisted fact-checking instead of trying to automate the whole process [31]. With this in mind, we reinvoke a philosophy of machine design called “autonomation”, developed in the Toyota Production System [ 62 ]. Autonomation is a blend of “autonomous” and “automation” and may be described as “intelligent automation” or “automation with a human touch”. Compared to (semi-)automation that is typically orchestrated by a centralized computer controller, autonomation separates the work of humans from machines and thus enables the quick correction of the mistakes made by machines. Yet, autonomation relieves humans of the need to continuously judge whether the operation of the machine is right. In the “autonomated” systems, the workers are self-inspecting their work and can source-inspect the work of the machines. This is a difference from an “automatic” system (that operates without human intervention) or an “automated” system (that requires human input or monitoring but is controlled by technology). Autonomation liberates people from automatable tasks, whereas technology has to serve people and processes, not vice versa. In the context of fact-checking, we use the term metaphorically, as it is more commonly used in the engineering industry. Although the HCI and information science community has significantly helped to better understand the gap between the social and technical, the existing research works on AI-assisted fact-checking are still often detached from real fact-checkers and thus do not optimally comply with their actual needs and expectations. As Nakov et al. point out, there is a “lack of collaboration between researchers and practitioners in defining tasks and developing datasets” for automated factchecking [ 58 ]. This contributes to the skepticism of fact-checkers towards automating the entire fact-checking process, especially processes that require human judgments [ 2 , 53 ]. On the contrary, some studies, e.g., [60] mention too much trust in the system’s predictions. In this study, our objective is to investigate the fact-checking procedures to design the effective and appropriate human-centered AI tools to counter false information. The role of humans is central here: the tools should empower humans instead of replacing them. This work offers the following contributions: (1) We unify the various categorizations of the fact-checking process across different computer and social science literature [2,10,53,58]. (2) We investigate the activities, problems, resources, and tools of under-researched European fact-checkers and examine the specifics of fact-checking work in the non-English speakingregions. We align them with the knowledge from existing studies, namely [ 2 , 27 , 53 , 58 ], and visualize them jointly as conceptual models. (3) We identify new opportunities for research and development of AI tools designed to support fact-checkers’ daily work in line with human-centered AI principles. In this direction, we discuss three AI tasks that previous studies of fact-checkers’ needs and activities have not addressed yet. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:4 Hrckova, et al. Fig. 1. The regions covered in existing research on fact-checkers (when specified) 2 Background and related work 2.1 Practices and challenges of fact-checkers in Europe and beyond Although the research on fact-checkers’ practices and problems is growing (see, e.g., [ 2 , 46 , 53 , 102 ]), the European region as well as non-American and non-English speaking fact-checkers remain still under-explored as can be seen on the map (Fig. 1). At the same time, there are specific challenges that the European fact-checkers face. These stem from two main sources. First, the European fact-checkers need to operate in a highly multinational and multilingual environment, since many disinformation campaigns easily spread across the borders. To maximize the negative impact in each affected country, disinformation narratives and content itself is often translated and adjusted to specific societal and cultural aspects. Second, the problem of online disinformation and foreign information manipulations and inference has been recognized in official policy documents at the level of the European Union (EU) and specific legislation has been drafted impacting how the social media platforms should respond to disinformation. Specifically, a European approach to tackle these phenomena emphasizes the importance of a transparent, trustworthy, and accountable online ecosystem while promoting quality journalism and media literacy 4 . Fact-checking has been recognized as one of the important ways to tackle false information and it was supported by various EU initiatives such as the Social Observatory for Disinformation and Social Media Analysis (SOMA) 5 or the European Digital Media Observatory (EDMO) 6 together with its regional hubs. Most recently, with the adoption of the Digital Service Act (DSA) 7 , the previously voluntary Code of Practice on Disinformation 8 has become the official Code of Conduct, which will be used to determine DSA compliance of the online platforms operating in the EU digital space regarding disinformation risks9. 4https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:52018DC0236 5https://www.disinfobservatory.org/ 6https://edmo.eu/ 7https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/europe-fit-digital-age/digital-services-act_en 8https://digital-strategy.ec.europa.eu/en/library/2022-strengthened-code-practice-disinformation 9https://digital-strategy.ec.europa.eu/en/library/code-conduct-disinformation ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:5 Regarding the existing studies on fact-checkers’ practices and problems, the Full Fact report [ 2 ] was the first one (published in 2020) to globally examine them. Following the semi-structured interviews with fact-checkers from 19 organizations, the report lays out the main challenges factcheckers face, such as large amounts of potential claims to check, managing tips and suggestions from readers, or the accessibility of information from relevant authorities to check information credibility. Another study published in 2022 consisted of semi-structured interviews conducted with 21 fact-checkers from 19 countries [ 53 ]. The results highlight the fact-checkers’ motivations, which are driven by a sense of social responsibility, despite encountering numerous challenges throughout the process. This was further reinforced in [ 46 ], which confirms that fact-checkers commonly believe in their ability to determine objective truth, relying on evidence and a transparent process that ensures reproducibility. Regarding the most common challenges of fact-checkers, these include – based on [ 53 ] – the enormous volume of misinformation, the shortcomings in available tools (such as algorithms that favor virality), confirmation bias of information consumers, the existence of echo chambers in online spaces, and insufficient data and resources for fact-checking efforts. Besides that, the authors point out the reasons for the limited uptake of computational tools by these professionals, such as the limited scope to a specific function within the fact-checking process. Furthermore, the integration of the tools with each other proved to be lacking, as the tools are developed by different entities. The authors suggest developing a unified platform with humans-in-the-loop, or, alternatively, establishing a set of standards to turn the fact-checking process into an efficient and streamlined operation. Five key problematic areas regarding misinformation fact-checking were also identified in [ 102 ] to be: 1) the limited affordances of digital technologies, 2) limited agency on platform infrastructures, 3) limited expertise and human resources, 4) hostility toward fact-checking actors, and 5) fact-checks fueling misinformation. Additional studies focused on broadening the understanding of various stakeholders (editors, external fact-checkers, in-house fact-checkers, investigators, and researchers, as well as social media managers and advocates) [ 44 ] or on the identification of common features of fact-checking methodology [74]. However, none of these studies focused specifically on the European context. The notable exception is the research on fact-checkers’ practices done by NORDIS (NORdic observatory for digital media and information DISorders) [ 27 ], which was, however, limited in scope to Nordic countries, i.e., Norway, Sweden, Finland, and Denmark. Semi-structured interviews with 14 respondents from these countries, supplemented with 5 non-Nordic professional journalists and fact-checkers, were conducted. The findings revealed four types of tools that are needed to automate the parts of the fact-checking process considered “boring” for human fact-checkers (social network monitoring, political debate monitoring, claim collection and detection, and context-dependent verification, especially multimedia content) as well as their expected characteristics (e.g., adaptation of tools to Nordic languages). The authors underline the necessity to incorporate the context into the designing process of AI tools, especially 1) the ethical principles of journalism; 2) human values (such as social responsibility to deliver accurate information, creativity, and intuitiveness of the fact-checking process); 3) human expertise with technology; and 4) human relationship to the information resources. In the follow-up study [ 26 ], the authors suggest that the focus needs to be moved from a technological point of view toward a social one, provided that a confidence relationship is established between the communities, developers, and fact-checkers/journalists involved from either side of the tool. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:6 Hrckova, et al. In our study, we increase the scope to organizations from 20 European countries and investigate the differences as well as similarities in their problems and needs across various dimensions (including geography, language, or organizational type). 2.2 AI support of fact-checkers A lot of AI research is devoted to assisting fact-checkers by automating the individual stages of factcheckers’ work [ 58 , 105 ]. Reflecting (at least partially) the common steps of the fact-checking process identified in studies above, it focuses on tasks such as finding claims worth fact-checking [ 13 , 29 , 47 ], searching for previously fact-checked claims [ 37 , 45 , 77 , 80 , 94 ], or claim verification and evidence retrieval [ 48 , 67 , 107 ]. Another group of research focuses on the development of the necessary datasets [ 37 , 59 , 76 , 84 ] and end-to-end systems or monitoring platforms [ 41 , 86 ]. Developing AI-based methods has also been addressed through competitions and data challenges, especially as a part of CheckThat! Labs and SemEval workshops, where tasks such as check-worthiness estimation [78] or detecting previously fact-checked claims [63,77] have been proposed. In parallel with research and development of various AI methods and models, several tools aiming to support the fact-checking process have been created. Some of them directly apply AI in order to provide fact-checkers with more advanced features, including automation of some of the fact-checking steps. In the following overview (see also Fig. 13), we mention especially those that have been explicitly mentioned by the fact-checkers during our study. At first, there are several tools that allow social media monitoring. Typically, these tools are developed for purposes different from fact-checking (e.g., marketing), such as NewsWhip 10 ,StoryBoard 11 or BuzzSumo 12 . Fact-checkers, however, found their way to use these tools also for fact-checking purposes. Some tools are also provided directly by social media platforms, such as Meta Content Library 13 (a replacement for the former and widely used CrowdTangle tool) dedicated to monitoring Facebook, Instagram, and Threads, or X Pro 14 (a successor of the former TweetDeck tool) dedicated to monitoring X. For searching within existing fact-checks, fact-checkers can also use dedicated tools. To this end, Google Fact Check Explorer tool 15 allows to search within fact-checking articles that use the so-called ClaimReview Schema 16 . The ClaimReview Schema allows fact-checkers to annotate their fact-checks by the most important metadata, such as the fact-checked claim, the resulting veracity label, or the link to fact-checked content. Finally, there are tools that aim to perform end-to-end fact-checking (automating all its steps) or at least cover multiple steps at the same time. At first, Full Fact AI 17 provides a set of AI-based tools for collecting and monitoring the data, identifying and labeling claims, matching identified claims against a database of past fact-checks, and evidence retrieval to speed up verification of new so-far non-fact-checked claims. Similarly, ClaimBuster 18 provides support from monitoring and selecting claims up to their verification against the selected knowledge-bases [ 41 ]. Lastly, the Verification plugin19 developed and maintained by the EU-funded vera.ai project (while originally 10https://www.newswhip.com/ 11https://storyboard.news/ 12https://buzzsumo.com/ 13https://transparency.meta.com/en-gb/researchtools/meta-content-library/ 14https://pro.x.com/ 15https://toolbox.google.com/factcheck/explorer 16https://schema.org/ClaimReview 17https://fullfact.org/ai/about/ 18https://idir.uta.edu/claimbuster/ 19https://www.veraai.eu/posts/verification-plugin ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:7 created in the InVID and WeVerify projects), provides a set of tools for detecting various types of false information not only in textual content but also in images and videos. Despite existing support and analyses of fact-checkers’ work, there is a partial disconnection between the fact-checkers’ needs and the efforts to create (AI) tools to support their work. Their design is treated as a technical solution to a technological problem and ignores the complex social and situational context [ 44 ]. There is also a lack of collaboration between AI researchers and developers with fact-checkers [ 58 ]. For example, capabilities of many of the existing fact-checking tools may be limited because various AI-based tools (e.g., detection methods, search engines) typically achieve a worse performance when applied on non-English or low-resource languages. Similarly, the stage monitoring the online space is insufficiently covered by AI research in the context of fact-checkers, although it is carried out routinely by these professionals and in some works [ 2 , 44 ] even recognized as “the hardest part of the fact-checking process”. A notable exception is the recent study [ 51 ], which indicates a change in trends by using a co-design approach to develop NLP tools for fact-checkers. This problem is not common only to fact-checkers. Although the conceptual foundations of HCAI are extensively discussed in recent literature [ 81 , 103 ] and guidelines for building human-centered AI products exist (e.g. [ 30 ]), the industry practices and methods appear to lag behind [ 15 , 38 ]. The lack of end-user viewpoint in the early design-related activities is well known from the Humancentered Design (HCD) practice [38]. Specifically, while tech-savvy fact-checkers are often called for testing of developed AI tools, these often turn out to be “irrelevant” for their work [ 27 ]. This can be prevented by early inclusion of fact-checkers in the tool design process. The challenge is that the capabilities of AI are unclear to users who set the end-user requirements [ 38 ]. This disproportion calls for an interdisciplinary approach, where the practices and problems of users (fact-checkers) are studied by social science and humanities (SSH) researchers and at the same time discussed with AI researchers to design powerful AI tools. 3 Methodology In this study, we aim to address the identified gap in the field – our aim is to investigate the activities and needs of fact-checkers to support research and development of the effective and appropriate human-centered AI tools to counter false information. We specifically focus on the European context, which (as shown in Section 2.1) has multiple historical, cultural, language, and legislative specifics; and remains under-studied in this area. Furthermore, the analysis of existing works as well as practical deployment of AI-based tools for fact-checkers showed a need for more human-centered AI approaches. Human-centered AI (HCAI) is based on processes that extend user experience design methods such as stakeholder engagement [ 81 ]. The goal is to create tools that augment and enhance human performance. HCAI systems emphasize a high level of human control while embedding high levels of automation that can be achieved by a good design [81]. Human-Centered Design (HCD) approaches are mentioned to be capable of contributing to the field of HCAI as well [ 7 ]. Specifically, design thinking, previously named Need-Design Response (NDR), focuses on the design and development of any tools or systems for the physical, intellectual, and emotional needs of people. It allows identifying human needs in the early phases of the design project through practices such as need finding. User needs, especially information needs and user interaction with information, focused primarily on the cognitive viewpoint are studied also in information science, specifically in information behavior studies [11]. Human-Centered Interaction (HCI), as an interdisciplinary field, adopts a “human-centered design” approach to develop computing products that meet user needs, which makes this field a potentially strong contributor to HCAI. Nevertheless, the development of AI systems is still ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:8 Hrckova, et al. Fig. 2. Flow chart of the research design (research process, methods, and terminology) mainly driven by a “technology-centered design” approach [ 81 , 103 ]. To respond to the challenge, a human-centered AI approach was proposed that places humans at the center of AI design as the ultimate decision makers [81,103]. The HCAI framework [ 103 ] includes in addition to “ethically aligned design” and “technology” also “human factors design” to ensure that AI solutions are explainable, comprehensible, useful and usable. To accomplish this, they start from the needs of humans and implement the human-centered design approach advocated by the HCI community (e.g., user research and modeling) in the research and development of AI systems. However, developers are usually less concerned on what people need in their lives or the social impact of AI [ 15 ]. These are the defining features of users’ positive experiences and need to be addressed for HCAI to be truly human-centered. Based on the principles of HCAI and HCD, we make our research study the first stage of the HCAI design process, in which end users are constantly involved in shaping and evaluating the supporting AI tools. This is complementary to the existing human-in-the-loop approaches to fact-checking, where the human workforce is used only to train and validate models in continuous ways, usually to annotate [ 23 ] and evaluate models [ 104 ] or provide feedback [ 79 ] in order to reduce bias, increase accuracy, etc. 3.1 Research design To analyze the activities and needs of fact-checkers, we first engaged these stakeholders individually in semi-structured in-depth interviews. Using the iterative content analysis, we identified the activities – repeating routines performed by fact-checkers in the individual stages of the fact-checking process. These activities were further validated with previous research work on fact-checker practices and visualized by conceptual maps of the fact-checking process. By proceeding from the content analysis, we also identified particular fact-checkers’ needs. Nevertheless, the needs are to some extent implicit, since they are inner motivational states to reach goals, and it is possible to be unaware of one’ s true needs [ 18 ]. As not all respondents were tech-savvy, they referred more often to problems instead of directly formulating their needs for specific technological support. Therefore, in this paper, we use problems of fact-checkers as a substitution for their needs (i.e., a corresponding need refers to finding a solution that can solve the problem expressed by a fact-checker). The most significant problems (and resulting needs), connected to the activities they relate to, were consequently verified by a quantitative validation survey. Finally, the implications for AI tasks and tools were derived. We illustrate this methodology (research process, methods, and terminology) in Fig. 2. In our research, we address the following research questions: ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:9 • RQ1: Which information resources, procedures, and information technologies do factcheckers use, and how do they depend on local specifics? • RQ2: What are the biggest problems of European fact-checkers that can be addressed by the support of artificial intelligence? • RQ3: Which parts of the fact-checking processes are suitable and needed for potential autonomation? To answer RQ1, we conducted semi-structured in-depth interviews, in which we were interested in how Central European fact-checkers: 1) monitor the online space to identify potential misinformation; 2) select potential false claims/narratives; 3) communicate and avoid potential duplication; 4) verify content credibility and veracity; and finally, 5) disseminate fact-checks. We compared our findings with fact-checkers worldwide. To answer RQ2, we quantitatively surveyed fact-checkers across Europe to assess the weight of the findings of the interviews. Based on the findings with respect to RQ1 and RQ2 and based on current state-of-the-art research and technical possibilities in the field of AI, we identified the implications for AI development in different stages of the fact-checking process to answer RQ3. 3.2 Selection criteria for respondents The in-depth interviews were conducted with nine fact-checkers from five major Central European (Slovak, Czech, and Polish) fact-checking organizations. All of the respondents performed factchecking professionally as part of their full-time job; none of them was employed in social media. We involved both external and in-house professionals in management positions (editors-in-chief and project coordinator) and fact-checking-only positions, but also fluid and overlapping fact-checking roles with the roles of journalist, PR manager, editorial manager, and senior research fellow. Our sample represents one of the low-resource language groups, under-explored in AI-research. The sample covers a variety of sizes and types of organizations – from small NGOs to large news agencies. The majority of organizations also collaborated with social media providers (e.g., Meta). Most of them were part of the IFCN network, which prohibits any kind of political connection. Two organizations were not members of the IFCN (and did not collaborate with social media). These organizations focused on the fact-checking of political discussions in mass media (mainly TV) and partisan news. The product of these two organizations was an article summarizing the main arguments in a misleading topic of interest (not a structured fact-check of a claim). For this work, we define Central Europe as the countries of the Visegrád Group [ 89 ]. While Central Europe geographically belongs to the European region, its historical and linguistic context differs from Western Europe (Eastern Block legacy, Slavic languages). This has many consequences on its current social and cultural setting and related issues (e.g., the level of Russian influence). Contextual factors affect the choices of action and the use of sources and channels in the online environment [ 3 ]. We can thus assume that some needs of Central European fact-checkers can differ from the fact-checkers elsewhere in Europe/the world. In particular, we can expect differences when compared with high-resource language areas. Nevertheless, Central European fact-checkers form a sample too small to derive conclusions about the problems of fact-checkers (not covered in existing literature). Therefore, we validated the results of the interviews with a validation survey. The survey was responded to by 24 representatives (N = 24) of 21 European fact-checking organizations, covering 20 countries. Albeit a nominally small sample, our survey respondents represent approximately 62% of active European fact-checking organizations that are IFCN signatories. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:16 Hrckova, et al. Fig. 6. Conceptual model of the communication part of the fact-checking process 4.4 Verification of content credibility and veracity Content evaluation is widely regarded as an activity that requires human judgment. Both in our region and globally [2,44,53], this process depends heavily on communication with people. Factcheckers view domain experts as valuable sources of information, and at times, they contact the authors of disputed content to request evidence or responses to fact-checks. They also engage with government officials and press agencies to request or clarify data—a step that can significantly delay the process, as such officials tend to be highly cautious in their statements. This reliance on interpersonal communication and expert input underscores the complexity of content evaluation and represents a key obstacle to its automation. Nevertheless, while evaluation itself resists full automation, fact-checkers view evidence retrieval—a timeand labor-intensive stage—as a promising candidate for technological support and partial automation. Evidence retrieval involves the acquisition of primary sources to achieve the maximum possible objectivity (Fig. 7). Our focus on human-information interaction enabled us to go deeper into the specific resources that the fact-checkers use for evidence retrieval. The textual sources involve mostly official data (as statistics), official information on the websites of governments, private companies, and NGOs, or original research studies. A fact-checker (P3) describes the process: “We have to find the right source for the data, such as in the Polish statistical bureau, central controller’s office, Eurostat... If the claim is not about data, we try to find facts or proof since as journalists, we can be sued for everything we write... It is not just debunking whether [the claim] is true or not... Lots of claims are manipulative or partially true, ... Very often we use experts (such as in finance, energy, or climate) to evaluate information ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:17 Fig. 7. Conceptual model of the verification part of the fact-checking process in statements – including implicit ones... The media is not our sources, they are good for background or if someone said something in the media.” The official data are hard to retrieve as they are not usually in easily accessible formats or are not publicly available. Fact-checkers in all regions experience different difficulties in reaching primary and official data sources, depending on the level of development of a country [ 53 ]. However, not being able to verify a claim is one of the biggest frustrations of fact-checkers [27]. Fact-checkers often work with audiovisual evidence, such as parliamentary speeches or recorded interviews to check whether a person mentioned the claim that someone else accuses him or her of saying. The fact-checkers frequently require watching hours of videos, many times without any result. Therefore, most of the Central European respondents would appreciate having searchable transcripts of the videos in local languages. Secondary sources such as mass media are rarely used for verification of claims. They were mentioned (if at all) rather as illustrative than reliable. Sources like Wikipedia were never used or considered credible for verification during fact-checking, as confirmed also by [ 27 ]. Nevertheless, according to [ 32 ], textual sources, such as news articles, academic papers, and Wikipedia documents, have been one of the most commonly used types of evidence for automated fact-checking. As the above mentioned sources are not utilized by fact-checkers, verification tools trained or tested just on Wikipedia and mass media datasets will never be sufficient for the work of these professionals. The largest exception to the non-usage of secondary sources are existing fact-checks of other fact-checking organizations. These are seen as the most credible secondary source of evidence for Central European fact-checkers. Existing fact-checks within the same organization are considered ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:18 Hrckova, et al. good sources of information, as politicians and other actors often repeat themselves. However, previously existing fact-checks (regardless of their relative provenience) are often hard to retrieve as P3 continues: “The most cumbersome is to find, if a claim or statement has already been fact-checked or if fake news was verified by someone else. Most of the fact-checks come from abroad. We have to check if English, French, Italian, German fact-checkers did it... If yes, we sometimes use it so that we can prove the history of spreading, then we quote these fact-checking websites.“ Another reason is, that many fact-checks are not created with the necessary structure or metadata and are thus difficult to search through services such as Google Fact-Check Explorer. Adhering to the most common fact-check schema, the ClaimReview, is rather cumbersome for fact-checkers under their present technological conditions. According to our respondents (Fig. 7), few search engines are helpful for fact-checkers. They mention mostly the Google Reverse Image Search that can identify similar images. Ordinary Google Search is not sufficient for the needs of Central European fact-checkers as it does not retrieve the most credible and hard-to-search textual sources of evidence. This is in accordance with [ 53 ] as well as [ 39 ], who studied the effectiveness of commercial web search engines for fact-checking. They found out that the engine’s performance in retrieving relevant evidence was weakly correlated with the retrieval of topically relevant pages. InVid tool in the Verification plugin was seen as very relevant for our respondents as it decomposes the videos into keyframes (images) that are possible to be reverse-searched. Nevertheless, as some misinformation is very complex, fact-checkers need to do more than reverse-search the images. Interestingly, participants in the study of [ 53 ], who reside mostly outside of Europe, do not report being aware of these tools. Content verification is seen by our respondents as an “intuitive process” that results from the longterm practice of fact-checkers, often journalists. Fact-checkers prevailingly stated that they notice questionable sources “at first sight”. However, one respondent shared his credibility indicators for the evaluation of sources, namely: non-existent author, problematic source (in the past), factuality, distortions, biases, objectivity, out-of-the-context use, sentiment and argumentation fallacies. In previous research [ 53 ] a “contextualization stage” was mentioned, where most fact-checkers looked into the evolution of the claim from its origin to the present state to help readers understand their conclusions. This process is very challenging, especially in foreign languages. One of our interviewees mentioned the problematic tracing of such a history, particularly on social media, where the search capabilities are very limited. Fact-checkers’ verification processes are generally very transparent, as the websites of factchecking organizations often explain how the judgment was reached, attaching the sources as well (often their archived versions). Besides that, it is very rigorous. As reported, none of the fact-checks is published without consulting it with at least one editor. Participants from Africa and Oceania have even longer peer review processes than Western countries [53]. 4.5 Dissemination of fact-checks Because of capacity reasons, dissemination of fact-checks mostly takes place where the misinformation spreads – on social media, mostly Facebook (Fig. 8). Fact-checks are published on the websites of the organizations. In connection to web publishing, Central European fact-checkers frequently report limited technological capabilities in their organizations (unlike some fact-checkers throughout the world [ 2 , 53 ]). These prevent them from full exploitation of search engine optimization (SEO), ClaimReview, appearing in Google News, or paid ads on Google as noted by P5: ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:19 Fig. 8. Conceptual model of the dissemination part of the fact-checking process “We thought of using the ClaimReview format, we tried it on fifteen claims in the Google Search Console. But as there are many fact-checks, we would need a process that would fill it automatically. It was very time consuming to fill all those fields.” Our respondents from small NGOs also struggle with personal (marketing and design) capacities to be able to make their fact-checks more appealing and popular by including visuals (e.g., infographics, videos, comic stripes). This is a common practice for copy editors in some fact-checking organizations [44]. Nevertheless, fact-checkers communicate with the media, as P6 claimed: “Sometimes the media notice that we fact-checked something, sometimes we reach the media ourselves, sometimes we have a project with them to fact-check something.” Fact-checkers around the world pointed out that they communicate with their audience through instant messaging services, particularly WhatsApp [ 2 , 44 , 53 ]. Nevertheless, this communication is more cumbersome, and Central European fact-checkers do not utilize these channels. Some of our fact-checkers, as well as the respondents in the study of [ 53 ] and [ 27 ] revealed concerns about the limited reach and potential of their outcomes. The collaboration with social media platforms achieved some success. Nonetheless, fact-checking is a long process, and carrying out all its stages allows misinformation to spread in the meantime. Unlike our respondents, some fact-checking institutions also reach out to policy makers and civil organizations or organize literacy campaigns to strengthen the dissemination of facts. 5 Results and findings: Validation survey 5.1 The problems of European fact-checkers In general, we can conclude that the validation survey supports and extends the findings of the in-depth interviews. Regarding the differences, the survey showed a higher urgency to autonomate alerts and user tips from instant messaging services and filtering the monitored outlets. The coordination with other fact-checkers and the analysis of the impact of the fact-checks was perceived ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:20 Hrckova, et al. Fig. 9. The biggest problems of European fact-checkers identified by the proposed coefficient of the perceived importance of the problem (scale of the coefficient ranges from 1 to 30) as an issue with a lower urgency than it seemed from the interviews. Thus, these problems are considered specific to the region and need to be addressed locally. Addressing RQ2, our results (Fig. 9) indicate that the biggest problems of European fact-checkers lie in the monitoring as well as in the verification parts of the process. The interviewed factcheckers are overloaded with potentially false content, which requires better filtering (beyond virality measures). According to the results from interviews, the verification of the truthfullness of the claim is not perceived as an issue. However, some parts of the verification process take most of the valuable time of fact-checkers and would urgently need some AI support. The validation survey confirmed that one of the biggest problems relates to searching for the sources of evidence for verification, particularly in hard-to-find official documents, videos, and statistics. This is especially true for cases where data needs to be integrated, as one fact-checker (P10) noted: “[I] hardly believe that machine recognition of misinterpretation of scientific studies/statistics is possible.” Even when misinformation is verified, its numerous versions exist and are shared on the internet. Therefore, the tools that would be able to identify other versions of the same misinformation would be crucial for misinformation mitigation. One fact-checker (P8) illustrated some parts of this process as follows: “The content on Facebook is very hard to find. We have to use some tricks, but still, there is a lot of manual work. It is very hard to identify, given a text query, all the places and URL addresses where the content exists (external pages, Facebook posts... images, and videos are even harder). No tool would do this and this consumes about half of our time.... I think the technology to automate searching the source of misinformation already exist and with them, we would be able to do much more fact-checking.” Additionally, two ideas were mentioned frequently by our respondents (10 times each): •a tool to monitor trending YouTube claims and/or topics, and •a tool connected to WhatsApp’s API which collects and prioritizes reader tips. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:21 Fig. 10. Differences in the coefficients of the most important problems depending on the location of the fact-checkers (Eastern vs. Western Europe) We further investigated the differences between the fact-checkers’ problems according to the location, language group affiliation, and size of their organization. 5.2 Differences between the fact-checkers’ problems The fact-checking organizations that were respondents of our validation survey have been split into two groups: 12 Eastern (including Central) European organizations (from these countries: Czech Republic, Slovakia, Hungary, Poland, Lithuania, Latvia, Estonia, Croatia, Slovenia), and 10 Western European organizations (from these countries: Belgium, Italy, Ireland, France, Spain, Germany, Switzerland, Austria, Denmark). We further distinguish 12 organizations from countries using low-resource languages (from Slavic and Ugro-Finnish language groups), and 10 organizations from countries with high-resource languages (from Germanic and Romance language groups). Finally, the size of organizations varied as follows: 3 large (250 and more employees), 3 medium (50 to 249 employees), 6 small (10 to 49 employees) and 9 micro-sized (less than 10 employees). 5.2.1 Differences between the problems of Eastern and Western European fact-checkers. The data do not show many differences between the problems of Eastern and Western European fact-checkers (Fig. 10). However, searching within the existing fact-checks (both their own and from other factchecking organizations and in potentially different languages) is considered a more urgent problem to solve by Eastern European fact-checkers. This recognized need is relevant to address, as the existing fact-checks are an important source of evidence for these professionals. The need for a tool that would help to autonomate this part of the process was confirmed by a fact-checker (P9) in a comment of the survey: “Would totally be a great help. Compiling all the IFCN signatories’ outputs into one big (keyword-driven) database? A fact-checker’s dream. Although, it is fair to say that Google Search supplements a lot of this proposition – although it’s always difficult with other languages than English – relevant now for Ukrainian.“ The first explanation of different perceptions of this difficulty is a better coverage of fact-checks in bigger language groups by Google Fact-Check Explorer. The lack of experience and/or resources ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:22 Hrckova, et al. Fig. 11. Differences in the coefficients of the most important problems depending on fact-checkers’ language group affiliation to invest in ClaimReview markup was mentioned both by Eastern as well as some Western European fact-checkers. The second explanation of this difference is that Eastern European fact-checkers need to check first the Western European fact-checks (which seems not to be the other way around). For example, a fact-checker (P7) revealed in the interview that: “Russian trolls know that in order to be successful, they have to go through the West (Germany, for example), because Poles do not like Russian sources... Also, sometimes what is popular abroad – mostly in Czech or English, such as vaccines – is popular in Poland as well.” In contrast, Western European fact-checkers focus more on misinformation modalities beyond text and platforms beyond Facebook. This is demonstrated by their pressing need to autonomate alerts from instant messaging services, such as WhatsApp, possibly Telegram, and a more urgent need to search for image manipulations. The less perceived urgency of these needs by Eastern European respondents can be explained by the lack of capacity that allows them to focus mainly on text and Facebook, as mentioned during some of the interviews. 5.2.2 Differences between the problems of European fact-checkers according to their language affiliation. Regarding the various language groups of the fact-checkers, we can see (Fig. 11) that there are some differences in the perceived problems, but they are not always language-related. Nevertheless, the perceived urgency level stands out in the Ugro-Finnish group that is the smallest language group of participants. According to the results of the survey, these professionals are much less involved in the coordination of actions of other fact-checkers. Generally, fact-checkers from the low resource languages, such as Hungarian, Slovak or Czech perceive more urgently the language-related problems in searching for the other versions of misinformation or in searching within the existing fact-checks. This is also in line with the NORDIS report [27]. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:23 Fig. 12. Differences in the coefficients of the most important problems depending on fact-checking organization size We can hypothesize that misinformation in dialects would be even harder to detect by machines than small language groups. This puts the people communicating in dialects in a more vulnerable position. A fact-checker (P10) supports this hypothesis: “Machine does not recognise the dialect...A tool which recognizes the dialect would help to simplify my work.” 5.2.3 Differences between the problems of European fact-checkers according to the size of their organization. If we divide the results according to the size of the fact-checking organizations (Fig. 12), we can see that large organizations perceive the tasks connected with: 1) image and video analysis; 2) monitoring of instant messaging services; as well as 3) searching for the other versions of misinformation as more difficult than smaller organizations do. The reason lies again in the higher capacities of these organizations to focus on more tasks that require more time, as confirmed during interviews. This finding points out that these tasks are not of little importance, but of little personal capacities to involve in such responsibilities. The autonomation of such tasks by artificial intelligence would help bigger organizations in the first place, but in the end also the smaller organizations, as the process of disinformation detection would be much easier and possible to complete with less capacity. The other problems that were identified in our research include marketing issues (insights analysis, ClaimReview) that are just partially relevant for AI research. The reasons for these problems of both types of organizations are different: the small ones face capacity issues or lack of knowledge in terms of technology or marketing support; the fact-checkers of the largest organizations face complicated processes of the big media concerns that prevent fact-checkers to edit or monitor any content on the website that is common for all parts of the organization. Nevertheless, filling out ClaimReview would help AI researchers to collect better datasets of the previous fact-checks and insights analysis would provide more information about the topics that are important for users. 6 Implications for AI research and AI-based tools To answer RQ3, we identified implications and opportunities for research and development of AI tools that would support fact-checkers in fulfilling their tasks. For clarity, we mapped the stages of ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:24 Hrckova, et al. Fig. 13. Mapping of the top 10 most important problems of European fact-checkers with the corresponding stages of the fact-checking process, selected existing tools, and implications for AI support fact-checking process and the most urgent problems of fact-checkers (resulting from our validation survey) with the corresponding implications for AI support (Fig. 13). Our proposed implications for AI support come from the current state-of-the-art research and technical possibilities in the area of AI – machine learning (ML) and NLP (including the latest development of generative large language models), and also from available datasets and tools and are consistent with the work by Nakov et al . [58] . Compared to Nakov et al . [58] , we add three more AI tasks: 1) check-worthy document detection, 2) mapping of the existing fact-checks to additional (newly appearing) online content, and 3) fact-check summarization and personalization. When discussing these implications, we aim to emphasize the need for human-centered AI systems (and a role that the current generative AI can play there) as well as reflect on the specific challenges and needs of the European fact-checkers identified in the previous sections. At the same time, we recognize that some of the most important problems do not require AI-based solutions. In such cases, the implementation of suitable tools represents mainly an engineering challenge ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:25 (e.g., how to implement a tool that fact-checkers can use to insert ClaimReview schema into their fact-checks), which is out of the scope of this work. 6.1 Implications for Check-worthy document detection The first AI task aims to support filtering the check-worthy documents (that is, news articles, blog posts, or even posts on social media) from monitored sources, which addresses the European fact-checkers’ problem of filtering relevant content for fact-checking from the monitored outlets. It can be defined as follows: given an input set of documents, detect such documents that are worth fact-checking, and thus they should receive fact-checkers’ attention (e.g., they contain factual claims, they are potentially impactful and harmful to society, etc.). This task can be addressed either as a classification (a document is or is not check-worthy) or a ranking problem (prioritizing the most check-worthy documents). It should be addressed with information available at document creation time (i.e., without relying on user feedback that appears later). Given that false information spreads faster than true information [ 95 ], the impact of a fact-check can be limited once the post has already become viral. On the other hand, early fact-checks can limit the spread of false information and can thus serve as a way of pre-bunking, which has already been shown to be effective in existing studies [71]. We argue that this task should generally precede the task of check-worthy claims detection (extraction) to limit and prioritize the amount of potential misinformation content the fact-checkers need to examine. Current systems (e.g., a fact-checking tool used within Meta’s Third-Party FactChecking Program) often prioritize the virality of a post as noted by the fact-checkers in our as well as in the previous studies [ 2 , 27 , 53 ]. In research work, check-worthiness detection is originally related to claims (i.e., identification of particular sentences, typically in political debates). Nevertheless, starting in 2020, the CheckThat! Lab introduced the detection of check-worthy tweets [ 39 ]. Although tweets are natively short texts containing just a few sentences, we would like to emphasize the distinction with check-worthy claim detection (see below), and we consider this as the first step towards check-worthy document detection. At the moment, we are not aware of any works on check-worthiness detection for longer pieces of text (e.g., for social media posts from other platforms that do not pose such strict length limitations, or even for whole news articles/blogs) or for multimodal content. Such extension of current research works and the construction of a suitable dataset represents the possible next step. For an AI solution to assist the fact-checkers, it needs to target sources relevant for them, consider criteria that are typically used by fact-checkers when they decide what to fact-check, and provide a means of justification/explanation of the selection of the check-worthy documents. The last two points can be facilitated by a related, more generic task of credibility assessment. Credibility signals (e.g., ones proposed by W3C Credible Web Community Group 24 ) would help fact-checkers (as well as other media professionals) to pre-screen the document and decide whether to proceed with the manual in-depth investigation to determine the necessity to fact-check it. Such a tool would speed up fact-checkers’ comprehension of the online content. To detect such indicators, a wide variety of techniques may be used, from a simple lookup in whitelists/blacklists, through the automatic check of predefined credibility criteria (e.g., presence of an author’s name, the known editorial board of a newspaper, etc.) up to advanced ML/NLP models. These would, for example, classify typographical and stylistic characteristics reflecting psychological features influencing the reader’s sentiment, detect the use of logical fallacies, or classify the leaning (bias) of the text [ 22 , 25 ]. With the recent advancements of generative AI, namely large language models (LLMs), and their uptake for evaluation of various aspects of texts, such as text quality [ 34 ] or level of personalization [ 100 , 110 ], 24https://credweb.org/ ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:32 Hrckova, et al. 2021), 100113. doi:10.1016/j.osnem.2020.100113 [14] Md Momen Bhuiyan, Amy X. Zhang, Connie Moon Sehat, and Tanushree Mitra. 2020. Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (Oct. 2020), 93:1–93:26. doi:10.1145/3415164 [15] William J. Bingley, Caitlin Curtis, Steven Lockey, Alina Bialkowski, Nicole Gillespie, S. Alexander Haslam, Ryan KL Ko, Niklas Steffens, Janet Wiles, and Peter Worthy. 2023. Where is the human in human-centered AI? Insights from developer priorities and user experiences. Computers in human behavior 141 (2023), 107617–. doi:10.1016/j.chb.2022. 107617 Publisher: Elsevier Ltd. [16] Nadav Borenstein, Greta Warren, Desmond Elliott, and Isabelle Augenstein. 2025. Can Community Notes Replace Professional Fact-Checkers? arXiv:2502.14132 [cs.CL] https://arxiv.org/abs/2502.14132 [17] Mark Boukes and Michael Hameleers. 2023. Fighting lies with facts or humor: Comparing the effectiveness of satirical and regular fact-checks in response to misinformation and disinformation. Communication Monographs 90, 1 (Jan. 2023), 69–91. doi:10.1080/03637751.2022.2097284 [18] Donald O. Case and Lisa M. Given. 2016. Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior. Emerald Group Publishing. [19] Gullal Singh Cheema, Sherzod Hakimov, Abdul Sittar, Eric Müller-Budack, Christian Otto, and Ralph Ewerth. 2022. MM-Claims: A Dataset for Multimodal Claim Detection in Social Media. In Findings of the Association for Computational Linguistics: NAACL 2022, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 962–979. doi:10.18653/v1/2022.findings-naacl.72 [20] Eun Cheol Choi and Emilio Ferrara. 2023. Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation. arXiv:2310.09223 [cs.CL] https://arxiv.org/abs/2310.09223 [21] Katherine Clayton, Spencer Blair, Jonathan A. Busam, Samuel Forstner, John Glance, Guy Green, Anna Kawata, Akhila Kovvuri, Jonathan Martin, Evan Morgan, Morgan Sandhu, Rachel Sang, Rachel Scholz-Bright, Austin T. Welch, Andrew G. Wolff, Amanda Zhou, and Brendan Nyhan. 2020. Real Solutions for Fake News? Measuring the Effectiveness of General Warnings and Fact-Check Tags in Reducing Belief in False Stories on Social Media. Political Behavior 42, 4 (Dec. 2020), 1073–1095. doi:10.1007/s11109-019-09533-0 [22] Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1377–1414. doi:10.18653/v1/2020.semeval-1.186 [23] Gianluca Demartini, Stefano Mizzaro, and Damiano Spina. 2020. Human-in-the-loop Artificial Intelligence for Fighting Online Misinformation: Challenges and Opportunities. IEEE Data Eng. Bull. 43, 3 (2020), 65–74. [24] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. doi:10.18653/v1/N19-1423 [25] Sameer Dhoju, Md Main Uddin Rony, Muhammad Ashad Kabir, and Naeemul Hassan. 2019. Differences in Health News from Reliable and Unreliable Media. In Companion Proceedings of The 2019 World Wide Web Conference (WWW ’19). Association for Computing Machinery, New York, NY, USA, 981–987. doi:10.1145/3308560.3316741 [26] Laurence Dierickx, Carl-Gustav Lindén, and Andreas Lothe Opdahl. 2023. Automated fact-checking to support professional practices: systematic literature review and meta-analysis. International Journal of Communication 17 (2023), 21–21. [27] L. et al. Dierickx. 2022. Report on the user needs of fact-checkers. Technical Report. NORDIS. https://datalab.au.dk/ fileadmin/Datalab/NORDIS_reports/Report_task_4.2_Fact-checkers_user_needs.pdf [28] Chiara Patricia Drolsbach, Kirill Solovev, and Nicolas Pröllochs. 2024. Community notes increase trust in fact-checking on social media. PNAS Nexus 3, 7 (05 2024), pgae217. doi:10.1093/pnasnexus/pgae217 arXiv:https://academic.oup.com/pnasnexus/article-pdf/3/7/pgae217/58698734/pgae217.pdf [29] Pepa Gencheva, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017. A Context-Aware Approach for Detecting Worth-Checking Claims in Political Debates. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017. INCOMA Ltd., Varna, Bulgaria, 267–276. doi:10.26615/ 978-954-452-049-6_037 [30] Pair Google. 2021. People + AI Guidebook. https://design.google/ai-guidebook [31] Lucas Graves, Brendan Nyhan, and Jason Reifler. 2016. Understanding Innovations in Journalistic Practice: A Field Experiment Examining Motivations for Fact-Checking. Journal of Communication 66, 1 (2016), 102–138. doi:10.1111/jcom.12198 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcom.12198. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:33 [32] Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A Survey on Automated Fact-Checking. Transactions of the Association for Computational Linguistics 10 (Feb. 2022), 178–206. doi:10.1162/tacl_a_00454 [33] Ashim Gupta and Vivek Srikumar. 2021. X-Fact: A New Benchmark Dataset for Multilingual Fact Checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, Online, 675–682. doi:10.18653/v1/2021.acl-short.86 [34] Rishav Hada, Varun Gumma, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2024. METAL: Towards Multilingual Meta-Evaluation. In Findings of the Association for Computational Linguistics: NAACL 2024, Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 2280–2298. doi:10.18653/v1/2024.findings-naacl.148 [35] Michael Hameleers. 2022. Separating truth from lies: comparing the effects of news media literacy interventions and fact-checkers in response to political misinformation in the US and Netherlands. Information, Communication & Society 25, 1 (Jan. 2022), 110–126. doi:10.1080/1369118X.2020.1764603 Publisher: Routledge _eprint: https://doi.org/10.1080/1369118X.2020.1764603. [36] Michael Hameleers and Toni G. L. A. van der Meer. 2020. Misinformation and Polarization in a High-Choice Media Environment: How Effective Are Political Fact-Checkers? Communication Research 47, 2 (March 2020), 227–250. doi:10.1177/0093650218819671 Publisher: SAGE Publications Inc. [37] Momchil Hardalov, Anton Chernyavskiy, Ivan Koychev, Dmitry Ilvovsky, and Preslav Nakov. 2022. CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (Eds.). Association for Computational Linguistics, Online only, 266–285. doi:10.18653/v1/2022.aacl-main.22 [38] Maria Hartikainen, Kaisa Väänänen, Anu Lehtiö, Saara Ala-Luopa, and Thomas Olsson. 2022. Human-Centered AI Design in Reality: A Study of Developer Companies’ Practices : A study of Developer Companies’ Practices. In Nordic Human-Computer Interaction Conference (NordiCHI ’22). Association for Computing Machinery, New York, NY, USA, 1–11. doi:10.1145/3546155.3546677 [39] Maram Hasanain and Tamer Elsayed. 2022. Studying effectiveness of Web search for fact checking. Journal of the Association for Information Science and Technology 73, 5 (2022), 738–751. doi:10.1002/asi.24577 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.24577. [40] Maram Hasanain, Reem Suwaileh, Sanne Weering, Chengkai Li, Tommaso Caselli, Wajdi Zaghouani, Alberto BarrónCedeño, Preslav Nakov, and Firoj Alam. 2024. Overview of the CLEF-2024 CheckThat! Lab Task 1 on Check-Worthiness Estimation of Multigenre Content. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024) (CEUR Workshop Proceedings), Guglielmo Faggioli, Nicola Ferro, Petra Galuščáková, and Alba García Seco de Herrera (Eds.). CEUR Workshop Proceedings, 276–286. Publisher Copyright: ©2024 Copyright for this paper by its authors.; 25th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF 2024 ; Conference date: 09-09-2024 Through 12-09-2024. [41] Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017. Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17). Association for Computing Machinery, New York, NY, USA, 1803–1812. doi:10.1145/3097983.3098131 [42] Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh. 2020. COVIDLies: Detecting COVID-19 Misinformation on Social Media. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. Association for Computational Linguistics, Online. doi:10.18653/v1/2020.nlpcovid192.11 [43] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. Inf. Syst. 43, 2, Article 42 (Jan. 2025), 55 pages. doi:10.1145/3703155 [44] Prerna Juneja and Tanushree Mitra. 2022. Human and Technological Infrastructures of Fact-checking. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (Nov. 2022), 418:1–418:36. doi:10.1145/3555143 [45] Ashkan Kazemi, Kiran Garimella, Devin Gaffney, and Scott A. Hale. 2021. Claim Matching Beyond English to Scale Global Fact-Checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 4504–4517. doi:10.18653/v1/2021.acl-long.347 [46] Michael Koliska and Jessica Roberts. 2024. Epistemology of Fact Checking: An Examination of Practices and Beliefs of Fact Checkers Around the World. Digital Journalism (2024), 1–21. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:34 Hrckova, et al. [47] Lev Konstantinovskiy, Oliver Price, Mevan Babakar, and Arkaitz Zubiaga. 2021. Toward Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection. Digital Threats: Research and Practice 2, 2 (April 2021), 14:1–14:16. doi:10.1145/3412869 [48] Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language Models as Fact Checkers?. In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics, Online, 36–41. doi:10.18653/v1/2020.fever-1.5 [49] João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, and Carolina Scarton. 2024. Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals. arXiv:2309.07601 [cs.CL] https://arxiv.org/abs/2309.07601 [50] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages. [51] Houjiang Liu, Anubrata Das, Alexander Boltz, Didi Zhou, Daisy Pinaroc, Matthew Lease, and Min Kyung Lee. 2024. Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1–44. [52] Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2022. Did I See It Before? Detecting Previously-Checked Claims over Twitter. In Advances in Information Retrieval (Lecture Notes in Computer Science), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer International Publishing, Cham, 367–381. doi:10.1007/978-3-030-99736-6_25 [53] Nicholas Micallef, Vivienne Armacost, Nasir Memon, and Sameer Patil. 2022. True or False: Studying the Work Practices of Professional Fact-Checkers. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (April 2022), 127:1–127:44. doi:10.1145/3512974 [54] Sebastião Miranda, David Nogueira, Afonso Mendes, Andreas Vlachos, Andrew Secker, Rebecca Garrett, Jeff Mitchel, and Zita Marinho. 2019. Automated Fact Checking in the News Room. In The World Wide Web Conference (WWW ’19). Association for Computing Machinery, New York, NY, USA, 3579–3583. doi:10.1145/3308558.3314135 [55] Yuki Momii, Tetsuya Takiguchi, and Yasuo Ariki. 2024. RAG-Fusion Based Information Retrieval for Fact-Checking. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 47–54. doi:10.18653/v1/2024.fever-1.4 [56] Silvia Montaña-Niño, Victoria Vziatysheva, Ehsan Dehghan, Anand Badola, Guangnan Zhu, Otávio Vinhas, Michelle Riedlinger, and Sofya Glazunova. 2024. Fact-Checkers on the Fringe: Investigating Methods and Practices Associated With Contested Areas of Fact-Checking. Media and Communication 12 (2024), Article–number. [57] Preslav Nakov, Alberto Barrón-Cedeño, Giovanni Da San Martino, Firoj Alam, Mucahid Kutlu, Wajdi Zaghouani, Mucahid Kutlu, Wajdi Zaghouani, Chengkai Li, Shaden Shaar, Hamdy Mubarak, and Alex Nikolov. 2022. Overview of the CLEF-2022 CheckThat! Lab Task 1 on Identifying Relevant Claims in Tweets. CEUR Workshop Proceedings 3180 (Sept. 2022), 368–392. https://dclibrary.mbzuai.ac.ae/nlpfp/16 [58] Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. Automated Fact-Checking for Assisting Human Fact-Checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Zhi-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4551–4558. doi:10.24963/ijcai.2021/619 Survey Track. [59] Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. 2021. The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News. In Advances in Information Retrieval (Lecture Notes in Computer Science), Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (Eds.). Springer International Publishing, Cham, 639–649. doi:10.1007/978-3-030-72240-1_75 [60] An T. Nguyen, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Elizabeth Tate, Byron C. Wallace, and Matthew Lease. 2018. Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 189–199. doi:10.1145/3242587.3242666 [61] Jeppe Nørregaard and Leon Derczynski. 2021. DanFEVER: claim verification dataset for Danish. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press, Sweden, Reykjavik, Iceland (Online), 422–428. https://aclanthology.org/2021.nodalida-main.47 [62] Taiichi Ohno and Norman Bodek. 1988. Toyota Production System: Beyond Large-Scale Production (1st edition ed.). Productivity Press, Cambridge, Mass. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:35 [63] Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podrouzek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. 2025. SemEval-2025 Task 7: Multilingual and Crosslingual FactChecked Claim Retrieval. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Sara Rosenthal, Aiala Rosá, Debanjan Ghosh, and Marcos Zampieri (Eds.). Association for Computational Linguistics, Vienna, Austria, 2498–2511. https://aclanthology.org/2025.semeval-1.323/ [64] Matúš Pikuliak, Ivan Srba, Robert Moro, Timo Hromadka, Timotej Smoleň, Martin Melišek, Ivan Vykopal, Jakub Simko, Juraj Podroužek, and Maria Bielikova. 2023. Multilingual Previously Fact-Checked Claim Retrieval. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 16477–16500. doi:10.18653/v1/2023.emnlp-main.1027 [65] Moritz Pilarski, Kirill Olegovich Solovev, and Nicolas Pröllochs. 2024. Community Notes vs. Snoping: How the Crowd Selects Fact-Checking Targets on Social Media. Proceedings of the International AAAI Conference on Web and Social Media 18, 1 (May 2024), 1262–1275. doi:10.1609/icwsm.v18i1.31387 [66] Jakub Piskorski, Tarek Mahmoud, Nikolaos Nikolaidis, Ricardo Campos, Alipio Mario Jorge, Dimitar Dimitrov, Purificação Silvano, Roman Yangarber, Shivam Sharma, Tanmoy Chakraborty, Nuno Guimaraes, Elisa Sartori, Nicolas Stefanovitch, Zhuohan Xie, Preslav Nakov, and Giovanni Da San Martino. 2025. SemEval 2025 Task 10: Multilingual Characterization and Extraction of Narratives from Online News. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Sara Rosenthal, Aiala Rosá, Debanjan Ghosh, and Marcos Zampieri (Eds.). Association for Computational Linguistics, Vienna, Austria, 2610–2643. https://aclanthology.org/2025.semeval-1.331/ [67] Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2017. Where the Truth Lies: Explaining the Credibility of Emerging Claims on the Web and Social Media. In Proceedings of the 26th International Conference on World Wide Web Companion (WWW ’17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1003–1012. doi:10.1145/3041021.3055133 [68] Ethan Porter and Thomas J. Wood. 2021. The global effectiveness of fact-checking: Evidence from simultaneous experiments in Argentina, Nigeria, South Africa, and the United Kingdom. Proceedings of the National Academy of Sciences 118, 37 (Sept. 2021), e2104235118. doi:10.1073/pnas.2104235118 Publisher: Proceedings of the National Academy of Sciences. [69] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML’23). JMLR.org, Article 1182, 27 pages. [70] Alan Ramponi, Marco Rovera, Robert Moro, and Sara Tonelli. 2025. Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches. arXiv:2505.22118 [cs.CL] https://arxiv.org/abs/2505.22118 [71] Jon Roozenbeek, Sander van der Linden, Beth Goldberg, Steve Rathje, and Stephan Lewandowsky. 2022. Psychological inoculation improves resilience against misinformation on social media. Science Advances 8, 34 (Aug. 2022), eabo6254. doi:10.1126/sciadv.abo6254 Publisher: American Association for the Advancement of Science. [72] Daniel Russo, Shane Kaszefski-Yaschuk, Jacopo Staiano, and Marco Guerini. 2023. Countering Misinformation via Emotional Response Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 11476–11492. doi:10.18653/v1/2023.emnlp-main.703 [73] Daniel Russo, Serra Sinem Tekiroğlu, and Marco Guerini. 2023. Benchmarking the Generation of Fact Checking Explanations. Transactions of the Association for Computational Linguistics 11 (10 2023), 1250–1264. doi:10.1162/tacl_ a_00601 arXiv:https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00601/2163489/tacl_a_00601.pdf [74] Hada M Sánchez Gonzales, María Sánchez González, and Javier Martos Moreno. 2024. The Methodology Used by Fact-Checkers. An In-Depth Analysis of Commonly Used Strategies. Journalism Practice (2024), 1–24. [75] Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos (Eds.). 2024. Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER). Association for Computational Linguistics, Miami, Florida, USA. doi:10.18653/v1/2024.fever-1.0 [76] Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a Known Lie: Detecting Previously Fact-Checked Claims. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 3607–3618. doi:10.18653/v1/2020.acl-main.332 [77] S. Shaar, F. Haouari, W. Mansour, M. Hasanain, N. Babulkov, F. Alam, G. da San Martino, T. Elsayed, and P. Nakov. 2021. Overview of the CLEF-2021 CheckThat! Lab Task 2 on detecting previously fact-checked claims in tweets and political debates. 2021 Working Notes of CLEF - Conference and Labs of the Evaluation Forum, CLEF-WN 2021 (2021), 393–405. [78] S. Shaar, M. Hasanain, B. Hamdan, Z. S. Ali, F. Haouari, A. Nikolov, and M. Kutlu. 2021. Overview of the CLEF2021 CheckThat! Lab Task 1 on check-worthiness estimation in tweets and political debates. CEUR-WS. https: //gcris.etu.edu.tr/handle/20.500.11851/8345 Accepted: 2022-01-15T13:02:35Z ISSN: 1613-0073. ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:36 Hrckova, et al. [79] Shaban Shabani, Zarina Charlesworth, Maria Sokhn, and Heiko Schuldt. 2021. SAMS: human-in-the-loop approach to combat the sharing of digital misinformation. In Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering. CEUR workshop proceedings. [80] Qiang Sheng, Juan Cao, Xueyao Zhang, Xirong Li, and Lei Zhong. 2021. Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 5468–5481. doi:10.18653/v1/2021.acllong.425 [81] B. Shneiderman. 2022. Human-Centered Artificial Intelligence – HCIL. Oxford University Press. [82] Iknoor Singh, Carolina Scarton, and Kalina Bontcheva. 2023. UTDRM: unsupervised method for training debunkednarrative retrieval models. EPJ Data Science 12, 1 (2023), 59. [83] Iknoor Singh, Carolina Scarton, Xingyi Song, and Kalina Bontcheva. 2024. Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checking. arXiv:2308.05680 [cs.CL] https://arxiv. org/abs/2308.05680 [84] Ivan Srba, Branislav Pecher, Matus Tomlein, Robert Moro, Elena Stefancova, Jakub Simko, and Maria Bielikova. 2022. Monant Medical Misinformation Dataset: Mapping Articles to Fact-Checked Claims. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22). Association for Computing Machinery, New York, NY, USA, 2949–2959. doi:10.1145/3477495.3531726 [85] Ivan Srba, Olesya Razuvayevskaya, João A. Leite, Robert Moro, Ipek Baris Schlicht, Sara Tonelli, Francisco Moreno García, Santiago Barrio Lottmann, Denis Teyssou, Valentin Porcellini, Carolina Scarton, Kalina Bontcheva, and Maria Bielikova. 2024. A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models. arXiv:2410.21360 [cs.CL] https://arxiv.org/abs/2410.21360 [86] I. et al Srba. [n. d.]. Monant: universal and extensible platform for monitoring, detection and mitigation of antisocial behaviour.. In Proceedings of Workshop on Reducing Online Misinformation Exposure. Rome, 1–7. https://rome2019. github.io/papers/Srba_etal_ROME2019.pdf [87] Megha Sundriyal, Tanmoy Chakraborty, and Preslav Nakov. 2023. From Chaos to Clarity: Claim Normalization to Empower Fact-Checking. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6594–6609. doi:10.18653/v1/ 2023.findings-emnlp.439 [88] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 809–819. doi:10.18653/v1/N18-1074 [89] Ronald Tiersky, Karen Alter, Eric S. Einhorn, Mark Gilbert, Robert Graham, Erik Jones, Sean Kay, John Logue, Jochen Lorentzen, Michael Mannin, Patrick McCarthy, John Van Oudenaren, Emanuela Poli, Jeffrey Simon, Ray Taras, and Helga Welsh. 2004. Europe Today: National Politics, European Integration, and European Security (2nd edition ed.). Rowman & Littlefield Publishers, Lanham. [90] Nafis Irtiza Tripto, Saranya Venkatraman, Dominik Macko, Robert Moro, Ivan Srba, Adaku Uchendu, Thai Le, and Dongwon Lee. 2024. A Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Texts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 6608–6625. doi:10.18653/v1/2024.acl-long.357 [91] Herbert Ullrich, Jan Drchal, Martin Rýpar, Hana Vincourová, and Václav Moravec. 2023. CsFEVER and CTKFacts: acquiring Czech data for fact verification. Lang. Resour. Eval. 57, 4 (May 2023), 1571–1605. doi:10.1007/s10579-02309654-3 [92] Herbert Ullrich, Tomáš Mlynář, and Jan Drchal. 2024. AIC CTU system at AVeriTeC: Re-framing automated factchecking as a simple RAG task. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 137–150. doi:10.18653/v1/2024.fever-1.16 [93] Patrick FA Van Erkel, Peter Van Aelst, Claes H de Vreese, David N Hopmann, Jörg Matthes, James Stanyer, and Nicoleta Corbu. 2024. When are fact-checks effective? An experimental study on the inclusion of the misinformation source and the source of fact-checks in 16 European countries. Mass Communication and Society (2024), 1–26. [94] Nguyen Vo and Kyumin Lee. 2020. Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7717–7731. doi:10.18653/v1/2020.emnlp-main.621 ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:37 [95] Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science 359, 6380 (March 2018), 1146–1151. doi:10.1126/science.aap9559 Publisher: American Association for the Advancement of Science. [96] Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, Tatiana Anikina, Michal Gregor, and Marián Šimko. 2025. Large Language Models for Multilingual Previously Fact-Checked Claim Detection. arXiv:2503.02737 [cs.CL] https: //arxiv.org/abs/2503.02737 [97] Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, and Marián Šimko. 2024. Generative Large Language Models in Automated Fact-Checking: A Survey. arXiv:2407.02351 [cs.CL] https://arxiv.org/abs/2407.02351 [98] Nathan Walter, Jonathan Cohen, R. Lance Holbert, and Yasmin Morag. 2020. Fact-Checking: A Meta-Analysis of What Works and for Whom. Political Communication 37, 3 (May 2020), 350–375. doi:10.1080/10584609.2019.1668894 Publisher: Routledge _eprint: https://doi.org/10.1080/10584609.2019.1668894. [99] Xuezhi Wang, Cong Yu, Simon Baumgartner, and Flip Korn. 2018. Relevant Document Discovery for Fact-Checking Articles. In Companion Proceedings of the The Web Conference 2018 (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 525–533. doi:10.1145/3184558.3188723 [100] Yaqing Wang, Jiepu Jiang, Mingyang Zhang, Cheng Li, Yi Liang, Qiaozhu Mei, and Michael Bendersky. 2023. Automated Evaluation of Personalized Text Generation using Large Language Models. arXiv:2310.11593 [cs.CL] https://arxiv. org/abs/2310.11593 [101] Greta Warren, Irina Shklovski, and Isabelle Augenstein. 2025. Show Me the Work: Fact-Checkers’ Requirements for Explainable Automated Fact-Checking. Association for Computing Machinery, New York, NY, USA. https: //doi.org/10.1145/3706598.3713277 [102] Oscar Westlund, Valerie Belair-Gagnon, Lucas Graves, Rebekah Larsen, and Steen Steensen. 2024. What Is the Problem with Misinformation? Fact-checking as a Sociotechnical and Problem-Solving Practice. Journalism Studies 25, 8 (2024), 898–918. [103] Wei Xu. 2019. Toward human-centered AI: a perspective from human-computer interaction. Interactions 26, 4 (June 2019), 42–46. doi:10.1145/3328485 [104] Jing Yang, Didier Vega-Oliveros, Tais Seibt, and Anderson Rocha. 2021. Scalable Fact-checking with Human-in-theLoop. In 2021 IEEE International Workshop on Information Forensics and Security (WIFS). IEEE, 1–6. [105] Xia Zeng, Amani S. Abumansour, and Arkaitz Zubiaga. 2021. Automated fact-checking: A survey. Language and Linguistics Compass 15, 10 (2021), e12438. doi:10.1111/lnc3.12438 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/lnc3.12438. [106] Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking Large Language Models for News Summarization. arXiv:2301.13848 [cs.CL] [107] Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul N. Bennett, and Saurabh Tiwary. 2020. TransformerXH: Multi-Evidence Reasoning with eXtra Hop Attention. In International Conference on Learning Representations. https://api.semanticscholar.org/CorpusID:212745235 [108] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. arXiv:2304.04675 [cs.CL] [109] Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-Checking Meets Fauxtography: Verifying Claims About Images. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 2099–2108. doi:10.18653/v1/D19-1216 [110] Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopál, Katarína Marcinčinová, and Matúš Mesarčík. 2025. Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 780–797. doi:10.18653/v1/2025.acl-long.38 ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:38 Hrckova, et al. A Semi-structured in-depth interview questions Shortened introduction We would like to talk with you about the process of your fact-checking and about the problems it brings to you. Our aim is to identify how the appropriate AI support to fact-checkers can be provided. Table 1. Semi-structured in-depth interview questions Category Question Basic information Institution Position What kind of false information do you check? Problem and needs to autonomate What do you miss the most when fact-checking? What is the most hard/ cumbersome when fact-checking? What are the most repetitive works when fact-checking that could be automatized? What are the most time consuming issues? What and how would you suggest improving the information systems that you use? Monitoring and selection of potential misinformation How do you spot news that need to be checked? Do you also check the popularity of the claims before fact-checking? How do you check it? When is the right time for you to fact-check? Where do you spot false information? Do you use any kind of resource management tools? Would it be beneficial to you if you had the (possibly most popular) check-worthy claims prepared by an AI system? Verifying the content credibility and veracity How do you verify whether the claim/ news are false or manipulated? What kind of resources do you use to fact-check the news? Do you mention them in the fact-check? Which criteria do you use to verify the credibility of content? Who verifies your fact-checks? What does your evaluation look like? Is it a scale or textual evaluation? Would it be beneficial to you, if AI identifies some credibility criteria for you in the news (like e.g., missing author or sources, hateful sentiment, spell check errors etc.?) What else would help you in verifying the content? Communication and avoiding duplication Which channels do you use for communication with other fact-checkers? Do you have a platform for communication? What type of communication is there? Do you also exchange some know-how there? How do you organize your work between your colleagues? Do you have any kind of system that you use in your organization to organize your workflow? How do you organize your work across fact-checkers in other organizations? Do you check whether the claim is already fact-checked? Does it happen, there are duplications in fact-checking? Would it be beneficial to you, if the information technology that you use checks whether the content is already fact-checked? Dissemination of fact-checks Do you publish your fact-checks just on your website or do you communicate them more widely? continues on next page ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:39 Do you also contact the person who made the misleading claim and ask them to correct or withdraw the claim? How is your fact-check structured? Do you also use semantics (like tags) to help search engines identify the claims? Which tools do you use to make the semantic markup? Do you cooperate with social media? How? Which languages do you cover? What would be helpful in dissemination of your fact-checks? Other Unstructured discussion B Quantitative validation survey questions Introduction Dear fact-checker, dear editor, We have collected the most serious problems that were mentioned during our interviews with Central European fact-checkers, operating in Slavic languages. This survey is meant to collect the answers of the factcheckers of the rest of Europe to become a more complex picture of the needs and problems of fact-checkers. As we plan to publish the results as a research paper, your answers may serve as important inputs for the AI research community to research and develop better solutions for you and to help your processes be smarter and smoother. Therefore, please, indicate the level and frequency of problem felt during your fact-checking process, as well as the perceived priority of support needed by a tool / tech. assistant for your work tasks that take you the most of the time or are most repetitive. Thank you very much for your valuable answers as well as for your important work. Table 2. Quantitative validation survey questions # Question Answer options I. Institution Free text answer II. How big is your institution? 1) micro (fewer than 10 employees) 2) small (10 to 49 employees) 3) medium-sized (50 to 249 employees) 4) large (250 and more employees) III. How often do you need to filter from monitored outlets manually to decide what to focus on? Example: You are overloaded with potential disinformation from e.g., Crowd Tangle. You need to filter them to see just the results „this looks suspicious“, „this might be important“...) 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never IV. If it was the case, how seriously do you suffer from manual filtering from monitored outlets? Would you appreciate some tech. support (tools) for this work? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer continues on next page ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. 1:40 Hrckova, et al. V. How often would you need to have any alerts about potential disinformation from instant messaging services? Example: You noticed that there are many screenshots from Telegram that are shared on Facebook and you would like to have a quicker alert before it is shared heavily on Facebook 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never VI. If it was the case, how seriously do you suffer from the late alerts about potential disinformation from instant messaging services? Would you appreciate some tech. support (tools) for this work? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer VII. How often do you need to seek factual claims suitable for fact-checking in a selected article? Example: You are overloaded by potential disinformation and you need to filter out just the factual claims that need/can be fact-checked 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never VIII. If it was the case, how seriously do you suffer from seeking factual claims suitable for factchecking in a selected article? Would you appreciate some tech. support (tools) for this work? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer IX. How often would you need to coordinate with other fact-checking organizations to avoid duplicates of your work? Example: You want to be aware, who is doing what, not to do the duplicate fact-checks. Better coordination with the other fact-checking organizations is needed 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never X. If it was the case, how seriously do you suffer from duplicates of your work with other factchecking organizations? Would you appreciate some tech. support for better coordination with the other fact-checking organizations? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer continues on next page ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025. Activities and Needs of European Fact-checkers as a Basis for Designing HCAI Systems 1:41 XI. How often do you need to search for the source of evidence for verification of the potential disinformation? Example: You need to find the relevant proof that the information you are fact-checking is manipulated, not true etc. You need a better search in the official statistics, media etc...to fulfill this task. 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never XII. If it was the case, how seriously do you suffer from searching for the source of evidence for verification of the potential disinformation? Would you appreciate some tech. support (tools) for this work? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer XIII. How often do you need to search for the source of evidence in videos? Example: You need to verify a very toxic rumor about a politician and you know, you will find the proof in parliamentary speeches. But it is very hard to search within these materials (videos without appropriate metadata). You would need a searchable textual transcript of the video. 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never XIV. If it was the case, how seriously do you suffer from searching for the source of evidence in videos? Would you appreciate some tech. support (tools) for this work? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer XV. How often do you need to seek the debunking comments of common users under videos? Example: Some comments under manipulative Youtube videos (e.g., with links) debunk false information that the video contains. You need to quickly find the potential debunking comments to be able to assess the videos faster. 1) More times a day 2) About once a day 3) About once a week 4) About once a month 5) Less than once a month 6) Never XVI. If it was the case, how seriously do you suffer from seeking the debunking comments of common users under videos? Would you appreciate some tech. support (tools) for this work? 5 point Likert scale: 1 = No problem, I like to do it; 5 = I perceive it as a big problem. I really need a help with this Please, provide us comments, if you have any Free text answer continues on next page ACM J. Responsib. Comput., Vol. 1, No. 1, Article 1. Publication date: January 2025.