Student evaluations of the credibility and argumentation of online sources

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC-ND 4.0 https://creativecommons.org/licenses/by-nc-nd/4.0/ Student evaluations of the credibility and argumentation of online sources © 2021 The Author(s). Published with license by Taylor & Francis Group, LLC Published version Marttunen, Miika; Salminen, Timo; Utriainen, Jukka Marttunen, M., Salminen, T., & Utriainen, J. (2021). Student evaluations of the credibility and argumentation of online sources. Journal of Educational Research, 114(3), 294-305. https://doi.org/10.1080/00220671.2021.1929052 2021 Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=vjer20 The Journal of Educational Research ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/vjer20 Student evaluations of the credibility and argumentation of online sources Miika Marttunen, Timo Salminen & Jukka Utriainen To cite this article: Miika Marttunen, Timo Salminen & Jukka Utriainen (2021): Student evaluations of the credibility and argumentation of online sources, The Journal of Educational Research, DOI: 10.1080/00220671.2021.1929052 To link to this article: https://doi.org/10.1080/00220671.2021.1929052 © 2021 The Author(s). Published with license by Taylor & Francis Group, LLC Published online: 31 May 2021. Submit your article to this journal View related articles View Crossmark data Student evaluations of the credibility and argumentation of online sources Miika Marttunen a , Timo Salminen b , and Jukka Utriainen b a Department of Education, University of Jyv€ askyl€ a, Jyv€ askyl€ a, Finland; b Finnish Institute for Educational Research, University of Jyv€ askyl€ a, Jyv€ askyl€ a, Finland ABSTRACT This study investigated upper secondary school students’skills in evaluating the credibility and argumentative content of a blog text and a YouTube video. Both sources concerned child vaccination, the blog text opposing and the YouTube video supporting it. Students rated each source as credible, fairly credible or non-credible, justified their ratings, and analyzed the argumentation of both sources. Their justifications were analyzed for trustworthiness and expertise and their argument analyses for identification of the main position of the source and the reasons supporting it. Students’justification skills proved fairly weak, and they also struggled with recognizing unbalanced argumentation. Students’skill in analyzing the argumentation used in the sources also proved inadequate, especially in the blog text task. Overall academic achievement significantly predicted students’credibility evaluation and argument analysis skills. The results suggest that greater emphasis should be placed on tasks involving the interpretation and analysis of online information. ARTICLE HISTORY Received 10 June 2020 Revised 31 March 2021 Accepted 3 May 2021 KEYWORDS Online sources; multimodality; upper secondary school; argumentation; critical literacy; credibility evaluation Introduction The internet offers a favorable environment for disseminating inaccurate and biased information, including misinformation and disinformation (Walsh, 2010). This is possible as anyone can publish freely on the internet without subjection to any quality control mechanisms (Zubiaga et al., 2018) or traditional gatekeeper models (Metzger et al., 2010) that have been instituted to uphold the credibility standards of online information. When defining bias Walton (1991) focuses particularly on the argumentative content of the source. According to him, the basic idea of bias is a lack of appropriate balance or neutrality in argumentation, that is, a tendency to consistently and unduly favor one side over the other, ignoring the arguments on both sides. Further, Walton stresses that bias also involves an absence of critical doubt, which appears in a failure to push the viewpoint one supports, thereby implying a departure from reasoned argumentation and critical discussion. Another reason for bias may be that an arguer has some ulterior intentions or something to gain based, e.g., on some ideological or commercial interest (Kammerer et al., 2015). With the rapid growth of electronic commerce, for example, where purchasing decisions are increasingly made in online environments, lack of objectivity in the information given about merchandise can be misleading (Xiao & Benbasat, 2015), exposing shoppers to commercial manipulation. Inaccurate or biased information is also harmful to citizens and society as it threatens democratic political processes and the democratic values that shape public policies in such life domains as health, science, and finance (Broniatowski et al., 2018; Kahne & Bowyer, 2017). For example, internet searches are commonly made for healthrelated information (Chen et al., 2018) despite the fact that many health and nutrition websites are inaccurate or commercially biased (see Kammerer et al., 2016) and may even contain distorted information (Chen et al., 2018). Thus, important decisions pertaining to health issues may be more strongly affected by biased and flawed argumentation advanced by manufacturers than by rational argumentation and objective comparison of alternatives. Because biased online information may have harmful effects, protecting people from deceitful information is important. As digitalization increasingly penetrates all areas of life, the role of schools in promoting the capacity of young people to use and identify accurate and useful information becomes increasingly important (Kahne & Bowyer, 2017). Promoting students’skills in evaluating the credibility of online information and analyzing the quality of its argumentation is of particular importance in helping students discriminate between reliable and unreliable online sources. Namely, research has shown analytical argumentation and thinking to be important for the critical evaluation of online information. Pennycook and Rand (2019) found that the ability to discern partisan fake news from real news was associated with the propensity to engage in analytical reasoning. Further, Bronstein et al. (2019) found that people over 18 with reduced analytic thinking were prone to believe in fake news. The present study investigated upper secondary school students’ability to evaluate the credibility and argumentative CONTACT Miika Marttunen [email protected] Department of Education, University of Jyv€ askyl€ a, PO Box 35, Jyv€ askyl€ a FI-40014, Finland ß2021 The Author(s). Published with license by Taylor & Francis Group, LLC This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4. 0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. THE JOURNAL OF EDUCATIONAL RESEARCH https://doi.org/10.1080/00220671.2021.1929052 content of two online social media resources familiar to young people, i.e., a blog text, and a YouTube video. The two sources represented opposite viewpoints on a current health-related topic, child vaccination. To find out to what extent students paid attention to source bias, both sources were selected to include flawed and unbalanced argumentation favoring only one side of the question of whether it is wise to vaccinate children or not. Literature review Evaluation of credibility and argumentation Credibility can be defined as the believability of a source or message and has two primary dimensions: trustworthiness and expertise (Flanagin & Metzger, 2008). Tseng and Fogg (1999) define expertise as knowledgeable, experienced, and competent and trustworthiness as well-intentioned, truthful, and unbiased. When evaluating the credibility of online sources for expertise, various source information and source features (Bråten et al., 2018) are often of interest, including document type, when and why the document was published, whether the author’s or organization’s contact information is given, and the author’s affiliation and qualifications (Metzger, 2007; Paul et al., 2017). When evaluating trustworthiness, various content features of the web site are important, e.g., how objective, up to date and in-depth is the information provided (Metzger, 2007) and whether there are indications of source bias. Source bias may appear in unbalanced argumentation when only one side of the issue is covered and favored over the other, or in some hidden intentions or motives of the writer related to e.g. political or commercial interests (Kiili et al., 2018; Walton, 1991). Evaluating the argumentative quality of a text requires the reader to pay attention to its argumentative structure. Toulmin (1958) defines the following main argumentative elements for analyzing the structure of argumentative texts: aclaim is the argument’s conclusion, data are the facts cited as premises for the claim, and a warrant is a general operating principle bridging the data and the claim (Fulkerson, 1996, p. 18; Nussbaum, 2011). In an argumentative text, a claim or a conclusion presents the writer’s main position or message. In a high-quality argumentative text, the main claim is clearly stated and supported by relevant and sufficient premises (Fulkerson, 1996). Argumentation plays an important role when evaluating the trustworthiness of online sources, as one should be able to assess whether the premises underlying the views presented are relevant and sufficient (Fulkerson, 1996), whether the text harbors hidden motives (e.g., economic or ideological), and whether the premises for and against the claim receive equal weight in the source. The terms grounds (Osborne et al., 2004), evidence (Forzani, 2016) and reason (Means & Voss, 1996) are also used when referring to elements of an argument justifying a claim or a conclusion. In this study, the term “reason”is used to refer to elements of arguments supporting the claim or conclusion, and “argumentation perspective”to refer to categories of reasons representing a specific perspective, such as a financial, ethical or health-related perspective. Xu and Yao (2015) state that the justifications or reasons representing various argumentation perspectives indicate the breadth of argumentation, a further necessary criterion for high-quality argumentation. Breadth of argumentation, or space of debate as van Amelsvoort et al. (2007) put it, is defined as the number of topics and subtopics mentioned that relate to the issue discussed: The higher the number of topics and subtopics related to the arguments used in the discussion, the higher the argumentative quality of the discussion. Previous research on evaluation of online information It is easily assumed that because young people are fluent in using the internet and social media, they are equally savvy about what they find there. However, recent research results show the opposite. Coiro et al. (2015) found that 7 th graders were unable to justify whether the author of an online source was an expert in the field the source represented. Further, the Stanford History Education Group (2016) showed that students at different educational levels had poor skills in evaluating information delivered through social media channels, and thus were at risk for being duped by disinformation, including flawed argumentation. For example, Miller and Bartlett (2012) in their survey (n¼509) of primary and secondary school teachers reported that students commonly struggle to recognize biased information and may use information that has been deliberately produced to be misleading. Similarly, Kiili et al. (2018) found that only about half of their sample of 6 th grade elementary school students questioned the credibility of the commercial online source and only about a fifth fully recognized commercial bias. Similarly, Fogg et al. (2003) asked somewhat older regular web users (n¼2 684; average age 39.9) to comment on the credibility of preselected websites representing various content categories. Only 11.6% of the participants’credibility comments referred to information bias. Kiili, Laurinen, and Marttunen (2008) found that upper secondary school students seldom evaluated the credibility of online information and that when they did, they mainly attended to such aspects as the publisher, author or expert interviewed in the source. Cognitively demanding evaluation strategies, such as the evaluation of argumentation or comparison of texts, were rarely used. At college level, McClure and Clink (2009) examined English composition students’ research essays to determine the amount of attention students gave to analyzing and crediting online sources, particularly their timeliness, authority, and bias. They found that students not only struggled with understanding bias but also ignored it when it hampered their writing. Previous research has shown that students’critical thinking has a positive association with their academic achievement (e.g., Fong et al., 2017; Ghanizadeh, 2017; Marin & Halpern, 2011). Similarly, university applicants’argumentative writing skills predicted their academic achievement in university (Preiss et al., 2013). These findings stress the 2 M. MARTTUNEN ET AL. importance of academic achievement in terms of students’ argumentation and critical thinking. In this study we explored this issue further by studying whether upper secondary school students’academic achievement is associated with their credibility evaluation and argument analysis skills. Prior studies also suggest that girls perform better than boys in critical evaluation of online sources (Forzani, 2016; Taylor & Dalal, 2017), prior knowledge on a topic enhances students’critical evaluation of sources on similar topics (Forzani, 2016), and that offline and online reading skills are intertwined but their relationship remains unclear (Leu et al., 2013; Kiili et al., 2018). Further, prior knowledge occupies an important role in readers’activities during offline reading comprehension (e.g., Means & Voss, 1985) but in online reading contexts its role is more complex (Coiro, 2011). To explore these issues further, this study investigated to what extent students’performance in the critical and argumentative evaluation of online sources can be explained by their gender, prior knowledge on the topic, and offline reading fluency. Aim and research questions Many studies have shown that students commonly struggle in evaluating the credibility of online sources and recognizing bias. Deficiency in credibility evaluation is a serious problem as, e.g., bloggers and video bloggers have a lot of influence particularly on young people who often adopt their ideas despite the absence of guarantees on the trustworthiness, objectivity, and unbiased nature of information disseminated in social media. Thus, further knowledge is needed on how students evaluate online sources low in credibility. Students’recognition of bias in online information has been widely investigated. However, to our knowledge, no studies exist in which students have evaluated the credibility of argumentatively unbalanced online sources representing different modalities (text, video) and addressing the same topic from opposite viewpoints (for and against). Such a comparative study may further understanding of whether young people differ in how they deal with unreliable information presented through different social media and whether they are more likely to be misled by erroneous and unobjective information presented via one type of social media rather than another. Further, this information is needed in developing pedagogy to help future citizens recognize inaccurate information and fake news (Vosoughi et al., 2018). This study investigated how students deal with two online social media sources frequently used by young people: a written blog and a multimodal video. Upper secondary school students were given a blog text and a YouTube video, one supporting and the other opposing the same currently contested topic (child vaccination). These online sources were deliberately selected for their unbalanced argumentation on the topic and hence questionable credibility. Both sources lacked credibility also in that the author of both sources was not an expert in the field (vaccination) and that they did not contain any references to support the presented arguments. Students’skills in evaluating these online sources for argumentative content and credibility were explored. Also examined was whether students’skills can be explained by gender, prior knowledge on the topic, reading fluency, and previous academic achievement. The research questions were: 1. How skilled are upper secondary school students in evaluating the credibility of unreliable online sources? 1.1 How do students rate the credibility of unreliable sources? 1.2 How do students justify the lack of credibility of unreliable sources? 1.3 Do students identify unbalanced argumentation? 2. How skilled are upper secondary school students in analyzing the argumentative content of unreliable online sources? 3. To what extent are students’skills explained by gender, prior knowledge on the topic, reading fluency, and previous academic achievement? Method Subjects Participants were 404 upper secondary school students (94.6% aged 17) from seven schools in Finland. There were 60.4% female students and 38.9% male students (missing information 0.7%). Most (79.2%) of the participants were students in three large urban upper secondary schools and the rest (20.8%) students in four small and middle-sized schools in semi-urban and rural areas. A written consent to participate in the study was obtained from the parents of underage students. Our research was approved by all the participating upper secondary schools and included only minimum risk for participants. According to the Finnish standard for institutional review board approvals, a statement by the Ethical Board of the University was not required. Study context This study is a part of a larger research project on Finnish upper secondary school students’argumentative online inquiry competencies. In the project, students completed an online task in which they composed an argumentative letter to imaginary Finnish members of parliament on the advantages and disadvantages of child vaccination aimed at informing their legislative work on this issue. The online task followed an online research and comprehension framework (Leu et al., 2013), including sub-tasks on the information search, evaluating, synthesis and communication. This article reports the results of the analyses of a sub-task on students’evaluations of the credibility and argumentative content of two online sources utilizing the evaluation component of the framework by Leu et al. (2013). THE JOURNAL OF EDUCATIONAL RESEARCH 3 The two sources, a blog text and a multimodal YouTube video, viewed child vaccination from opposite standpoints. The Finnish blog text (3 pages, 967 words) presented several potential disadvantages of vaccination by appealing to various side effects of vaccines, the financial benefits of vaccination for the pharmaceutical industry, and emotional arguments by describing situations in which parents have reported harmful effects of vaccines on their children. The blog text concluded that children should not be vaccinated. In contrast, the YouTube video presented arguments for vaccination, concluding that it is wise to vaccinate children. In the video, two US comedians (Penn and Teller) argued for vaccination by means of humor, music, sound effects, and visual demonstrations such as gestures and motion. The comedians also cited statistics on both the effectiveness of vaccinations and their minimal injurious effects and described the potential negative effects of not vaccinating children. The video, in English with Finnish subtitles, lasted for 1.5 minutes. Data collection Prior knowledge and reading fluency In this study, students’prior knowledge on the child vaccination was assessed with ten statements, three of which were correct and seven false. The statements were selected based on both current scientific knowledge and common misinformation disseminated by anti-vaccination movements. To strengthen content validity, two university lecturers in health sciences and a medical expert reviewed the statements. Students were asked to select the three statements they thought were correct. Students’prior knowledge scores varied from 0 to 3 depending on the number of correct statements selected. Reading fluency was measured by a word chain test (Holopainen et al., 2004). The task was to separate as many of a series of 100 words, written in four-word clusters without spaces in between, as possible in 90 seconds (Kiuru et al., 2011). Credibility evaluation and argument analysis tasks To evaluate the credibility of the online sources the students completed two tasks, one for the blog text and one for the YouTube video. In both tasks, the students first rated the credibility of the source by answering a multiple choice question (How credible do you think the text/video is?) by selecting one of three responses (credible, fairly credible, not credible), and then justifying their credibility evaluation in writing by answering the question “Justify your choice, why do you think this?.” Students also completed two tasks on the argumentative content of the online sources, one for the blog and one for the video. In both tasks, the students wrote down the main position on child vaccination they identified in the source and the most important justifications for that position given in the source. Data analysis Students’credibility evaluation skills First, the frequencies of the students’responses (n¼404) to the multiple-choice questions on the credibility of each online source were calculated. Next, the students’justifications for their credibility ratings (n¼403) were analyzed for trustworthiness and expertise. On trustworthiness, students scored points when they appealed to content-related issues: use of citations, research-based information, and unbalanced argumentation (Tseng & Fogg, 1999; Metzger, 2007; Walton, 1991). On expertise, the students scored points when they cited external characteristics of the source, e.g., author attributes, form of the web source, and organization(s) affiliated with the web page (Metzger, 2007). As both sources included unbalanced argumentation on the topic their credibility was principally deemed questionable. Consequently, students scored 0 points if they had not questioned the credibility of the source, i.e., selected the response option “credible”. When they had questioned credibility, i.e., chosen the options “fairly credible”or “not credible”, the response was scored from 1 to 4 depending on how well the lack of credibility had been justified. No justifications scored 1 point and one or more justifications, relating either to expertise or to trustworthiness, scored 2 points. Identification of unbalanced argumentation was emphasized in the scoring. Thus, 3 points was scored when lack of credibility had been justified by appealing to unbalanced argumentation. Identification of unbalanced argumentation is illustrated in the following two examples of students’justifications (see Table 1): The text is based only on the disadvantages of vaccination and totally ignores its advantages (Student 428, Blog text); The issue was treated only from one side and the attitude for not using vaccines was initially negative (Student 174, YouTube video). Justifications including both expertise and trustworthiness, although without reference to unbalanced argumentation, also earned 3 points. The highest score (4 points) was given when the justifications included, at the least, both expertise and reference to unbalanced argumentation (Table 1). To test the reliability of the analysis criteria of students’ justifications, 15% of the data were selected at random and analyzed by two independent raters, reaching 82% agreement (Cohen’s kappa, Œ¼.74). Students’argument analysis skills Analysis of students’skills to analyze the argumentation of the blog text and the YouTube video proceeded in two phases. First, the researchers analyzed the argumentation of these two online sources. Next, the students’responses to the argument analysis tasks were compared with the argumentation in the sources. Argumentation of the online sources. First, all the reasons against vaccination in the blog text (42 reasons) and all those for vaccination in the video (5 reasons) were identified. The identified reasons were then classified into 4 M. MARTTUNEN ET AL. argumentation perspectives on a data-driven basis (Hsieh & Shannon, 2005). For the blog text, the argumentation perspectives (an example of a reason in parenthesis) were: 1) Consequences of vaccination (Vaccination results in serious, even fatal injury); 2) Effectiveness of vaccination (Going through things notices that vaccinations do not protect from diseases); 3) Consistency of vaccination (Viral vaccines can never be safe); and 4) Support in problematic situations (Children falling ill from vaccinations are given help by official bodies). Similarly, for the video, the argumentation perspectives were: 1) Consequences of vaccination (Unvaccinated people are more likely to contract diseases); 2) Effectiveness of vaccination (Before Salk’s vaccine there were approximately 58,000 cases of polio). Analyzing students’responses. Students’skills in analyzing the argumentation in the source were scored based on their responses to each task (397 for the blog and 403 for the video). Students who failed to correctly identify the position taken in the source scored 0 points. Students who correctly identified the source position scored from 1 to 4 depending on the quality of their identification (position quality) and how many argumentation perspectives they found to support that position. Position quality was analyzed by investigating how well the position identified by a student corresponded to the source position, i.e., against vaccination in the blog and for vaccination in the video. Position quality indicated how well a student had understood (partially vs. fully) the main message of the source. In responses indicating partial understanding the source position was correctly identified but either only implicitly embedded in the response or expressed in the form of a descriptive claim by referring, for example, to specific harms or benefits of vaccines. Examples of descriptive claims were “Vaccines cause autism”(Student 416, Blog text) and “Vaccines protect people from severe diseases”(Student 95, YouTube video). Full understanding of the main message of the source was indicated by responses in which the source position was expressed in a form of a correct conclusion expressed in the source. Examples of correct conclusions were “Children should not be vaccinated” (Student 335, Blog text) and “It is wise to vaccinate children” (Student 217, YouTube video). The minimum requirement for a score of 1 in the blog task was that the student’s response included a position on child vaccination corresponding to that presented in the source (i.e., against vaccination, Table 2). Students scored 2 when the source position (SP) was correctly identified and justified from two argumentation perspectives (AP), as illustrated in the following response by student 296 (analysis markings of the authors in parentheses): “The author is clearly against vaccinations (SP) and presents many justifications for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vaccinated children (AP 1, Effectiveness of vaccination). She also extensively describes the harm caused by vaccinations”(AP 2, Consequences of vaccination). Students scored 2 also when Table 1. Examples of scoring students’responses to the credibility evaluation tasks. Examples of student responses Score Scoring criteria Blog text task YouTube video task 0 Credibility of source not questioned The facts prove that the claims of the author are credible. (St. 390) The video was convincing. (St. 263) 1 Credibility questioned without relevant justifications for the lack of credibility Lots of facts. (St. 417) Everybody can find such things on the internet because it is so wide (St. 151) The video is quite credible because it includes facts and exact figures. (St. 37) 2 Credibility questioned, and justification(s) relating either to expertise (E) or trustworthiness (T) presented Because this is a blog (E) and the writer is necessarily not an expert (E) in this field (St. 219) No references (T) listed at the end of the text, and the studies were somewhat vague (St. 290) Penn and Teller are magicians, not vaccination guys (E) (St. 422) There are no references (T) presented on the video. (St. 199) 3 Credibility questioned, and justifications relating both to expertise (E) and trustworthiness (T) presented, or unbalanced argumentation (UA) identified The text is based only on the disadvantages of vaccination and totally ignores its advantages (UA) (St. 428) Because the text is a blog written by one person (E), an opinion without reliable references (T) (St. 409) The issue was treated only from one side (UA) and the attitude for not using vaccines was initially negative (St. 174) 4 Credibility questioned, and unbalanced argumentation (UA) identified, and justification(s) relating to expertise (E) presented The source is a website of a natural products company, and the text is ablog (E) which makes is primarily rather subjective. Discussion on the issue is very one-sided (UA), and different approaches to the issue are not presented. (St. 436) I think like this because the video does not tell whether the persons are professionals or not (E), or whether they are only expressing their own opinion. Although the video also includes facts, no references (T) are used. The video is solely about the advantages of vaccines, nothing about disadvantages (UA), although they must certainly also exist. (St. 171) Note. St. ¼Student; T ¼Justification relating to trustworthiness; E ¼Justification relating to expertise; UA ¼Justification relating to unbalanced argumentation; The core words guiding the scoring are italicized. THE JOURNAL OF EDUCATIONAL RESEARCH 5 the response included a correct conclusion justified from one argumentation perspective. Score 3 was given when the source position was correct and justified from three argumentation perspectives, or a correct conclusion was justified from two argumentation perspectives. Score 4 was given when the response included a correct conclusion justified from at least three argumentation perspectives (Table 2). Students scored 1 on the YouTube video task when they correctly identified the source position taken on child vaccination (i.e., for vaccination, see Table 3). Students scored 2 when either the source position was correctly identified and justified from one argumentation perspective, or the conclusion was correct but lacked justifications. Score 3 was given when the source position was correct and justified from two argumentation perspectives, or the conclusion (CO) was correct and justified from one argumentation perspective (AP). The latter option is illustrated in the response by student 354 (analysis markings of the authors in parentheses): “The main message is that vaccination is not useless (CO). The most important justification is the survival of vaccinated children”(AP, Effectiveness of vaccination). Students scored 4 when their response included a correct conclusion justified from both argumentation perspectives (Table 3). To test the reliability of the analysis criteria for both the blog and the video, two independent raters classified 15% of the data selected at random, reaching 87% agreement (Cohen’s kappa, Œ¼.79). Variables used in the study The dependent variables used in this study were formed based on the students’credibility ratings, credibility evaluation, identification of unbalanced argumentation, and argument analysis for each source (Table 4). Table 2. Examples of scoring of students’responses to the blog text argumentation analysis task. Score Student response Interpretation 0 The author thinks vaccination is useful but not necessary. She justifies her opinion by facts. (St. 122) Source position (Vaccination is useful) not correctly identified 1 The author finds vaccines dangerous and that one should take them with a pinch of salt. (St. 303) Source position (against vaccination) correctly identified (Vaccines are dangerous). 2 The author is clearly against vaccinations and presents many justifications for her stand. She states that more children vaccinated against whooping cough contracted the disease than non-vaccinated children. She also extensively the describes the harm caused by vaccinations (St. 296) Source position correctly identified (against vaccination) and justified from two argumentation perspectives: 1. Effectiveness of vaccination (Vaccinated children contracted the disease they were vaccinated against) 2. Consequences of vaccination (Vaccinations cause harm) 3 The author thinks vaccines are completely useless and might even be dangerous. She justifies her claim by examples on how vaccines have several injurious effects and that they do not always protect people from diseases they are vaccinated against. (St. 337) Correct conclusion (Vaccines should not be used) justified with two argumentation perspectives: 1. Consequences of vaccination (Vaccines have many different harmful effects) 2. Effectiveness of vaccination (Vaccines do not always protect people from diseases they are vaccinated against) 4 The author thinks vaccines are harmful and should not be used. She justifies her claim by giving examples of people who have contracted the disease they were vaccinated against. She explains why she thinks that viral vaccines are dangerous by describing how they are produced. At the end she also tells about a child who became autistic after vaccination, and about a child who had narcolepsy after vaccination. (St. 385) Correct conclusion (Vaccines should not be used) justified with three argumentation perspectives: 1. Effectiveness of vaccination (Vaccines are not effective, because vaccinated people contracted the disease) 2. Consistency of vaccination (Viral vaccinations are dangerous because of the production process) 3. Consequences of vaccination (A child became autistic after vaccination and another child had narcolepsy) Note. St. ¼Student; ¼Authentic reason included in the blog text. Table 3. Examples of scoring of students’responses to the YouTube video argumentation analysis task. Score Student response Interpretation 0 The key point is that vaccines are complete bullshit :) (St. 134) Source position not correctly identified (Vaccines are bullshit) 1 Vaccines protect against many diseases and deaths (St. 115) Source position (for vaccination) correctly identified (Vaccines protect against diseases). 2 Vaccines do not cause autism and vaccines save people from bad diseases. The most important justifications were the use of visuals showing how vaccinated people can protect themselves from diseases and demonstrating the number of deaths from these diseases when not vaccinated. (St. 95) A descriptive claim (Vaccines save people from bad diseases) justified from one argumentation perspective: 1. Effectiveness of vaccination (Vaccines are effective, because they protect people from diseases) 3 The main message is that vaccination is not useless. The most important justification is the survival of vaccinated children. (St. 354) Correct conclusion (Vaccination is not useless) justified with one argumentation perspective: 1. Effectiveness of vaccination (The most important justification is the survival of vaccinated children) 4 The main point is that it is stupid not to vaccinate a child. Therefore, the video supports vaccines. The opinion was justified by listing many diseases against which vaccinations are used and saying that one can get the disease if not vaccinated. However, the video mentions that research exists claiming that vaccinations cause autism. However, this claim was completely refuted in the video. (St. 17) Correct conclusion (It is stupid to not vaccinate a child) justified with two argumentation perspectives: 1. Effectiveness of vaccination (Vaccinations protect children from many diseases) 2. Consequences of vaccination (Vaccinations are claimed to cause autism, which is refuted in the video) Note. St. ¼Student; ¼Authentic reason included in the video. 6 M. MARTTUNEN ET AL. Students’most recent grades in various upper secondary school subjects, performance in the word chain test, prior knowledge on vaccination, and gender were used as independent variables (Table 4). In Finland, academic performance in various subjects in upper secondary schools is rated from 4 to 10. Moreover, in the Finnish course-based school system at upper secondary level (European Commission, 2019), both schools and students have considerable freedom in organizing studies. Hence, the number of courses taken in different subjects may vary considerably between students, as was the case here. Moreover, some students did not consent to the use of their school grades for research purposes, which increased the amount of missing data. The proportion of missing data for the grades ranged from 15% to 32%. Statistical analyses To answer research questions (RQ) one and two, descriptive statistics on the level of students’credibility evaluation and argument analysis skills were used. Additionally, Wilcoxon’s signed ranks test was applied to compare students’credibility ratings for the blog text and video, and McNemar’s test was used to compare how students identified unbalanced argumentation in the sources (Davis, 2013). Furthermore, paired samples t-test was applied to compare differences between the blog text and the video in students’overall performance in the credibility assessments and argument analysis skills (Davis, 2013). To answer RQ 3, Structural Equation Modeling (SEM) was applied (Kline, 2011). All SEM analyses were conducted using Mplus 7.4. Maximum likelihood estimation with robust standard errors (MLR) was used as an estimation method, as the variables showed slight non-normality (skewness values from .68 to .45 and kurtosis values from .80 to 1.20). In the analysis, Overall academic achievement, Prior knowledge of the topic, Reading fluency and Gender were independent variables, and Argument analysis and Credibility evaluation skills dependent variables. In addition, school grades were indicators for the bi-factor model: Overall academic achievement was modeled as a generic factor for which all the individual grades were indicators. To complement the generic factor, specific factors for each individual grade were estimated to model variance not accounted for by the general factor. The following fit indices and cutoff values for acceptable fit of the model were used: v 2 -test, ns, p>.05, Root Mean Square Error of Approximation (RMSEA) value under .06, and Tucker-Lewis Index (TLI) and Comparative Fit Index (CFI) values over .95 (Hu & Bentler, 1999). Effect sizes used for the repeated measures tests were reported as Cohen’sd (Cooligan, 2017). For the non-parametric tests, the effect sizes were transformed to Cohen’sd. For the SEM-analyses, squared r’s were calculated. Results Students’credibility ratings of unreliable online sources More than a half of the students (59.1%) considered the blog text fairly credible, slightly above one-third (37.5%) not credible, and only a few (3.5%) credible. Similarly, the majority (53.1%) judged the video to be fairly credible and 38.2% non-credible. In contrast to the blog, nearly a tenth (8.7%) of the students considered the video credible. The difference between the students’credibility ratings for the blog and the video was not statistically significant (Z¼ 1.20, p¼.232). Students’skills in evaluating the credibility and analyzing the argumentative content of unreliable online sources Students’skill in evaluating the credibility of the online sources proved rather weak. Although the students commonly questioned the credibility of both sources, they justified the non-credible nature of the sources poorly. More than a half of the students, 50.3% for the blog and 58.8% for the video, offered no justifications or appealed only to either authority or trustworthiness (1-2 points, Table 5). Students’performance on the blog (M¼2.4) and video (M¼2.1) tasks differed significantly (t¼4.68, df ¼402, p< .001, d¼.29). As in the credibility evaluation, students’skill in analyzing the argumentation in the online sources was also fairly weak. This appeared especially in the blog text task. The majority (76.3%; points 0-2 in total) failed to identify the correct position taken in the text (0 points), or identified it but with either no or one justification (1 point), or found at most two of the four argumentation perspectives (2 points; Table 5). Further, only 2.8% of students correctly identified the main conclusion and several justifications for it (4 points; Table 5). Students performed somewhat better in the video task, as the corresponding proportions were 53.3% (0 2 points) and 10.1% (4 points). The difference between Table 4. Variables used in the study. Variable NScoring range Dependent skill variables Credibility rating for the blog 403 1–3 Credibility rating for the video 403 1–3 Credibility evaluation for the blog 403 0–4 Credibility evaluation for the video 403 0–4 Identifying unbalanced argumentation in the blog 403 0 ¼No; 1 ¼Yes Identifying unbalanced argumentation in the video 403 0 ¼No; 1 ¼Yes Argument analysis for the blog 397 0–4 Argument analysis for the video 403 0–4 Independent variables Grades Philosophy 273 4–10 Psychology 337 4–10 Health education 311 4–10 Mother tongue and literature 344 4–10 Mathematics 331 4–10 History 342 4–10 Other Reading fluency 404 0–100 Prior knowledge 404 0–3 Gender 401 Note. Females, n¼244 (60.6%); Males, n¼157 (39.2%). THE JOURNAL OF EDUCATIONAL RESEARCH 7