Validation of item pool for early adolescents’ emotional skills assessment in music therapy

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Validation of item pool for early adolescents’ emotional skills assessment in music therapy © 2023 the Authors Published version Salokivi, Maija; Salanterä, Sanna; Saarikallio, Suvi; Ala-Ruona, Esa Salokivi, M., Salanterä, S., Saarikallio, S., & Ala-Ruona, E. (2024). Validation of item pool for early adolescents’ emotional skills assessment in music therapy. Nordic Journal of Music Therapy, 33(2), 108-121. https://doi.org/10.1080/08098131.2023.2245433 2024 Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=rnjm20 Nordic Journal of Music Therapy ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rnjm20 Validation of item pool for early adolescents’ emotional skills assessment in music therapy Maija Salokivi, Sanna Salanterä, Suvi Saarikallio & Esa Ala-Ruona To cite this article: Maija Salokivi, Sanna Salanterä, Suvi Saarikallio & Esa Ala-Ruona (2023): Validation of item pool for early adolescents’ emotional skills assessment in music therapy, Nordic Journal of Music Therapy, DOI: 10.1080/08098131.2023.2245433 To link to this article: https://doi.org/10.1080/08098131.2023.2245433 © 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. View supplementary material Published online: 04 Sep 2023. Submit your article to this journal View related articles View Crossmark data ORIGINAL RESEARCH ARTICLE Validation of item pool for early adolescents’ emotional skills assessment in music therapy Maija Salokivi a , Sanna Salanterä b , Suvi Saarikallio c and Esa Ala-Ruona d a Department of Music, Art and Culture Studies, University of Jyväskylä, Jyväskylä, Finland; b Clinical Nursing Science, University of Turku, Turku, Finland; c Music Education, University of Jyväskylä, Jyväskylä, Finland; d Music Therapy, University of Jyväskylä, Jyväskylä, Finland ABSTRACT Introduction: Evaluating the validity of the content is an essential step in developing an assessment tool, including an analysis of the quality of the items within the tool. This study describes the content validation of items in the early adolescents’ emotional skills assessment tool in a music therapy context. Methods: Content validity was evaluated based on relevance scores provided by two expert panels. Psychometric scores were obtained by calculating item-specific content validity index (I-CVI), scale-specific content validity index (S-CVI) and modified kappa score. In addition, the coverage and understandability of the items were evaluated. Results: The validation process identified 60 valid items distributed across six components of emotional skills: expressing, monitoring, identifying, understanding, regulating and the ability to use emotional information. Item I-CVI scores ranged from 0.80 to 1.00, the scale content validity index (S-CVI) was 0.95, the modified kappa score ranged from 0.65 to 1.00, item coverage at scale level was 1.00, and item understandability was 0.92. Discussion: The items developed in the study have high validity and are scientifically grounded. The items can be a first step towards a validated assessment tool to evaluate emotional skills in early adolescents. The added value of this study is that the set of items developed is the first to cover all the components of emotional skills identified in the literature. Therefore, music therapists can use the items to observe in more detail the different dimensions of emotional skills in early adolescents. ARTICLE HISTORY Received 9 September 2022; Accepted 18 July 2023 KEYWORDS Emotional skills; early adolescents; content validity; item analysis; music therapy assessment Introduction Early adolescents (around 11–13) undergo many biological, externally visible changes, but their inner emotional development is also progressing. These changes in emotional development include greater emotional reactivity; an increased ability to reflect on CONTACT Maija Salokivi [email protected] Department of Music, Art and Culture Studies, University of Jyväskylä, PL 35, Jyväskylä FI-40014, Finland Supplemental data for this article can be accessed online at https://doi.org/10.1080/08098131.2023.2245433. NORDIC JOURNAL OF MUSIC THERAPY https://doi.org/10.1080/08098131.2023.2245433 © 2023 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons. org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent. emotions and to evaluate the acceptability and expression of emotions; and new strategies to manage emotions (Davey et al., 2008; Steinberg, 2005). Unfortunately, many early adolescents have a range of mental health conditions that complicate their years of adolescence. Mental health conditions such as depression, anxiety, behavioural disorders, and developmental disability are the leading causes of illness and disability among young people (10–19 years old). Unfortunately, half of these conditions begin already by the time a young person reaches 14 years of age (World Health Organization, 2021). Children and young people with mental health conditions face a wide range of risks, including poor school performance, poor subjective emotional well-being, behavioural problems and adverse life events such as dropping out of school (Aviles et al., 2006; McLeod et al., 2012; Tempelaar et al., 2014). In addition, these mental health conditions are often accompanied by deficits in emotional and interpersonal skills (Gonçalves et al., 2019; Parker et al., 2006). In Finland, where the authors of this study come from, early adolescents with mental health conditions are the largest group of music therapy clients. Early adolescents account for 60% of all children (0–15 years old) using music therapy, and 85% of the early adolescents in music therapy have mental health problems (Social Insurance Institution of Finland, 2017). The significant size of the client group, the importance of emotional skills in the lives of children and adolescents, and the practical benefits of music therapy with children and adolescents with mental health conditions (Geipel et al., 2018; Gold et al., 2004, 2007) are a justifiable reason to seek a validated assessment tool for the progress of early adolescents' emotional skills in a music therapy context. In addition, the requirement for evidence-based practice, the expectations of funders, and the desire of music therapists to communicate the impact and effectiveness of their work make the development of an assessment tool meaningful to the field (Cripps et al., 2016). There is currently no validated and reliable assessment tool in music therapy for assessing emotional skills in early adolescents (Cripps et al., 2016). Moreover, existing assessment tools only consider a limited number of emotional skills and their different components, not all of which have been defined in the scientific literature (Salokivi et al., 2022). Existing music therapy assessment tools that are available for children and adolescents assess different aspects of emotional skills, such as emotional expression (Langan, 2009; Mackeith et al., 2011), emotional responsiveness (Layman et al., 2002), emotional constriction (Wells, 1988), social-emotional functioning or behaviour (Douglas, 2006; Goodman, 1989), affect range (Loewy, 2000), and differentiation, expression, regulation and self-awareness (Baxter et al., 2007). This study uses the term “early adolescents’ emotional skills” even though the English term “emotion” does not have an unambiguous meaning, and there is no scientific consensus on the definition of the term (Frijda, 2016; Izard, 2010; Lakoff, 2016). The term “emotion” is a multi-component and multi-level concept (Zachar, 2010), and it can be viewed as an everyday concept or as a scientific concept (Widen & Russell, 2010). The difference between an everyday and scientific concept can be examined by separating the descriptive and prescriptive definitions. The descriptive definition refers to the definition of a word in everyday life, and the prescriptive definition refers to the scientific concept of a word, which includes a set of events that scientific theory aims to explain (Widen & Russell, 2010). In this study, we use the descriptive definition of The APA Dictionary of Psychology of the word “emotion”: “a complex pattern of reactions, including experiential, behavioural and physiological 2M. SALOKIVI ET AL. elements, through which an individual attempts to process a personally significant issue or event” (American Psychological Association, 2023). Validating the items of the assessment tool through an operationalisation process The validity of an assessment tool means the extent to which it measures the characteristics of the issue being studied (Devon et al., 2007). The validity of the assessment tool includes content, construct and criterion validity (Lynn, 1986). Content validity is a prerequisite for other types of validity and should therefore be a priority for assessment developers (Zamanzadeh et al., 2015). Content validity assessment examines how well a sample of items captures the operational definition of a concept (Polit & Beck, 2006) and to what extent the items adequately represent the content domain (Carmines & Zeller, 1979; Yaghmaie, 2003). This study presents the content validation process of an item pool used to assess emotional skills in early adolescents with mental health conditions in the context of music therapy. The research questions for this study are: What kind of item pool will be generated based on the results of previous studies (Salokivi et al., 2022, 2023)? Does the item pool have adequate content validity? The progression of content validation of the item pool is presented through a fivestep operationalisation process: (1) developing a theoretical definition; (2) specifying variables derived from the theoretical definition; (3) identifying observable indicators; (4) selecting means of measuring the indicators; and (5) evaluating the adequacy of the resulting operational definition (Waltz et al., 2016). The starting point (step 1) for the validation process is carefully determining the concept (Salkind, 2012). We reported on the process of concept definition in our previous study (Salokivi et al., 2022), in which the components of emotional skills in early adolescents were theoretically defined through a scoping review and concept analysis. The second (2) and third (3) steps of the operationalisation we reported in the study by Salokivi et al. (2023). In those steps, we applied the components of early adolescents’ emotional skills to the music therapy clinical practice. We used focus group interviews and content analysis to explore how music therapists worked with the different components of emotional skills in early adolescents, what methods they use, and where they can see the progress of the emotional skills during the therapy process. This knowledge was a foundation for the next steps presented in this study: (4) selecting means of measuring the indicators and (5) evaluating the adequacy of the resulting operational definition. In step four, the item pool is formulated, and in step five, the adequacy of the created items is evaluated based on their psychometric values. Figure 1 shows the validation process for the items. Methods The item pool initially included 62 items, which fell within the six primary components of the conceptual model of emotional skills: expressing emotions; monitoring emotions; identifying emotions; understanding emotions; regulating emotions; and the ability to use emotional information for self-management and establishing social relationships (Salokivi et al., 2022). Each component contained several items. The number of items in the subcomponents ranged from four to 14. The first author of this study formulated the item pool based on the results of two previous studies NORDIC JOURNAL OF MUSIC THERAPY 3 (Salokivi et al., 2022, 2023). The senior researcher, also the second author of this article, commented on the items and suggested changes. After the items were reedited, a final set of items was formed, and the content validity of these items was analysed. Analysing items The relevance of the items was studied based on the answers of expert panels through the Content Validity Index (CVI) and Modified Kappa. Psychometric scores were also calculated for the understandability of the items and coverage of the scale. In addition, qualitative written comments were collected from expert panellists to help further develop the questions where necessary. Participants of the expert panels Two groups of panellists were involved in the first round of the item evaluation for relevance, understandability, and coverage. One group comprised health researchers (n = 8), and the other comprised music therapy clinicians (n = 8). Health researchers represented theoretical expertise, and music therapists represented clinical expertise. The health researchers were doctoral researchers who volunteered to participate. Instrument development is a part of their education, and they are familiar with evaluating items, especially from the perspective of written form and understandability. Music therapists were volunteer clinicians who participated in an earlier phase of the research project and consented to participate in this phase of the study. They were specialists in the content and context. The responses of the music therapists and health researchers were Figure 1. Validation of the items through a five-step operationalisation process. 4M. SALOKIVI ET AL. examined separately for possible divergences. The research team were secondround panellists (n = 3) and evaluated the understandability of the reformulated items when needed. The panellists were senior researchers with instrument development and music therapy expertise. Content validity index Content validity was calculated at the item level using the Item Content Validity Index (I-CVI) and the scale level with the Scale Content Validity Index (S-CVI) (Lynn, 1986). In this study, the I-CVI and S-CVI were examined as follows: The panels of experts were asked to rate each item on the scale regarding its relevance to the underlying concept (Polit & Beck, 2006). The developed questionnaire had a 4-point ordinal scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant) (Lynn, 1986; Waltz & Bausell, 1981). The I-CVI was calculated as follows: the number of experts who gave the question a score of 3 or 4 divided by the total number of experts (Polit & Beck, 2006). The I-CVI had to be at least 0.78 for an item to be accepted. The recommended threshold for a panel of six to ten experts is 0.78 (Lynn, 1986). If the I-CVI of an item was less than 0.78 when the panellist groups’ responses were calculated separately, both groups’ average I-CVI determined whether the item was accepted. The S-CVI in this study was an S-CVI/Ave (average), which is defined as the average I-CVI value at the scale level and is a recommendation for the S-CVI calculation when there are several members in the expert panel (Polit & Beck, 2006; Waltz & Bausell, 1981). S-CVI/Ave was calculated by summing the I-CVI values of the 3- and 4-classified items, divided by the total number of items. The value had to be at least .90 for the scale to be accepted (Polit & Beck, 2006). Modified kappa The modified kappa (k*) is the consensus among evaluators that the item is relevant and considers the possibility of chance agreement (Polit et al., 2007). The kappa values were evaluated to determine whether they were fair, good, or excellent (Cicchetti & Sparrow, 1981; Fleiss, 1971). Polit et al. provided scale developers with a ready-made table for estimating k* values based on the number of experts and their agreement. Scale developers can compare their survey I-CVI values with this table without having to calculate the modified kappa. This study used a pre-established table, which is presented in Table 1. However, if the researcher wants to calculate the k* value, the first information needed is the subject’s random agreement. The random agreement can be examined using a binomial random variable formula that calculates how often a given event occurs in a fixed number of trials or attempts (Polit et al., 2007; Ross, 2017): Pc¼N! A!NAð Þ! � �:5N Pc refers to the probability of chance agreement, N is the number of experts, and A is the number agreeing on good relevance. Sign !, after N and A, means that the number is a factorial function; in other words, one must multiply all whole numbers from the chosen number to 1 (for example, 3! = 3 × 2 × 1 = 6). The second step is NORDIC JOURNAL OF MUSIC THERAPY 5 calculating the I-CVI (the proportion of agreements on relevance) and the probability of chance agreement (Polit et al., 2007). Understandability and the coverage of the items The panel of experts was asked to rate each item for understandability. The scale used was the same as in the relevance evaluation: a 4-point ordinal scale (1 = not clear, 2 = somewhat clear, 3 = quite clear, and 4 = highly clear). If the understandability of the item was < 0.78 (the same threshold value as in the relevance calculation), the item was reformulated and developed further with the second round’s smaller group of senior researchers (n = 3) until the understandability of the item was at an acceptable level. Because of the small size of this group, a universal agreement was required before the item was accepted. In this context, the universal agreement meant that all panellists gave a score of 3 or 4 for the item. The expert panellists had the opportunity to write a comment on each item if they wanted to clarify why the item was difficult to understand or if they wanted to give their recommendations for rewording the item. These open-ended qualitative comments were used to develop better items if the items still needed to receive an appropriate validity score in the first round. The panels of experts were also asked to evaluate the items’ coverage of the concept (emotional skills of early adolescents). The panellists received a short semi-structured questionnaire about coverage. The scale for the evaluation was a 4-point ordinal scale Table 1. Evaluation of I-CVIs with different numbers of experts and agreement. (1) (2) (3)a (4)b (5)c (6)d Number of Experts Number Giving Rating of 3 or 4 1-CVI Pc k* Evaluation 3 3 1.00 .125 l.00 Excellent 3 2 .67 .375 .47 Fair 4 4 1.00 .063 l.00 Excellent 4 3 .75 .25 .67 Good 5 5 1.00 .041 l.00 Excellent 5 4 .80 .156 .76 Excellent 6 6 1.00 .016 l.00 Excellent 6 5 .83 .094 .81 Excellent 6 4 .67 .234 .57 Fair 7 7 1.00 .008 l.00 Excellent 7 6 .86 .055 .85 Excellent 7 5 .71 .164 .65 Good 8 8 1.00 .004 l.00 Excellent 8 7 .88 .031 .88 Excellent 8 6 .75 .109 .72 Good 9 9 1.00 .002 l.00 Excellent 9 8 .89 .014 .89 Excellent 9 7 .78 .070 .76 Excellent a I-CVI, item level content validity index. b Pc (probability of a chance occurrence) was computed using the formula for a binomial random variable, with one specific outcome: Pc = [N!/A! (N - A) !]*.5 N where N = number of experts and A = number agreeing on good relevance. c k* = kappa designating agreement on relevance: k* = (1-CVI – pc)/(1 – pc). d Evaluation criteria for kappa, using guidelines described in Cicchetti and Sparrow (1981) and Fleiss (1971): Fair = k of .40 to .59; Good = k of .60–.74; and Excellent = k > .74. From “Polit et al. (2007), Is the CVI an acceptable indicator of content validity? Appraisal and recommendations“. Copyright 2007 by John Wiley & Sons. 6M. SALOKIVI ET AL. (1 = strongly disagree, 2 = somewhat disagree, 3 = somewhat agree, and 4 = strongly agree) (David, 1992). They could also comment if they wanted to add or remove items or comment on something else concerning the item. Translation of the items The panellists were Finnish, and the original items were in Finnish. The final item pool’s Finnish items were translated into English using DeepL Translator, a neural machine translator that uses an artificial neural network (DeepLGmbH, 2017). After translating the AI, a native speaker of both Finnish and English translated the English items back into Finnish. The result of the back-translation was compared to the original Finnish items, and if some of the items needed to be reworded to improve understandability, changes were made. However, the back-translation process did not significantly alter the English translation of the Deep L translator. The meanings of the items remained the same. The AI translation used “emotion” and “feeling” as parallel terms in English, while only one term was used in the original Finnish version. In the translated English items, the term “emotion” was retained. Thus, the final English language set of items was created. Ethical considerations The study received ethical approval from the Human Sciences Ethics Committee of the University of Jyväskylä (Number: 746/13.00.04.00/2020). Participants in expert panels received the study’s fact sheet and data protection notification. The panellists were aware that they had the right to refuse to participate in the study or to discontinue participation if they wished, without penalty. Data were saved and stored in a secure data environment. The participants in the expert panel were not identifiable in the research report. Results The results of this study describe the content validity of the items in the item pool to be used to assess emotional skills in early adolescents. The items in the item pool were categorised according to the components of the emotional skills of early adolescents (Salokivi et al., 2022). The emotional skills components were: expressing, monitoring, identifying, understanding, and regulating emotions; and the ability to use emotional information both on an individual level to develop and establish positive selfmanagement and on a social level to develop and establish positive relationships. Content validity Item content validity index, scale content validity index and modified kappa After averaging the I-CVI values of the original 62 items between the groups, only one item had an I-CVI value below 0.78. This item was deleted before the full-scale content validity index, and modified kappa was calculated. When calculating the average of both panellists’ groups, item-specific I-CVI values ranged from 0.80 to 1.00 for 61 items, which is an excellent result (Cicchetti & Sparrow, 1981; Fleiss, 1971). The more detailed results of the expert panels in terms of I-CVI scores for all 62 items were as NORDIC JOURNAL OF MUSIC THERAPY 7 Parker, J. G., Rubin, K. H., Erath, S. A., Wojslawowicz, J. C., & Buskirk, A. A. (2006). Peer relationships, child development, and adjustment: A developmental psychopathology perspective. In D. Cicchetti & D. J. Cohen (Eds.), Developmental psychopathology: Theory and method (pp. 419–493). John Wiley & Sons, Inc. https://doi.org/10.1002/9780470939383.ch12 Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing and Health, 29(5), 489–497. https:// doi.org/10.1002/nur.20147 Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Focus on research methods: Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 469–467. https://doi.org/10.1002/nur.20199 Ross, S. M. (2017). Discrete random variables. In S. M. Ross (Ed.), Introductory statistics (pp. 203–262). Academic Press. https://doi.org/10.1016/b978-0-12-804317-2.00005-9 Salkind, N. (2012). Content validity. In N. Salkind (Ed.), Encyclopedia of research design. SAGE Publications, Inc. https://doi.org/10.4135/9781412961288 Salokivi, M., Salanterä, S., & Ala-Ruona, E. (2022). Scoping review and concept analysis of early adolescents’ emotional skills: Towards development of a music therapy assessment tool. Nordic Journal of Music Therapy, 31(1), 63–88. https://doi.org/10.1080/08098131.2021.1903977 Salokivi, M., Salanterä, S., Saarikallio, S., & Ala-Ruona, E. (2023). Promoting emotional skills in early adolescents with mental health conditions in music therapy – A content analysis of focus group interviews. Nordic Journal of Music Therapy 32 (5), 423–444. https://doi.org/10.1080/08098131. 2023.2169336 Social Insurance Institution of Finland. (2017). Kelan kuntoutustilasto. https://helda.helsinki.fi/han dle/10138/234527 Spiro, N., Tsiris, G., & Cripps, C. (2020). Sounds good, but . . . what is it? An introduction to outcome measurement from a music therapy perspective. Approaches: An Interdisciplinary Journal of Music Therapy, 12(1), 8–29. https://approaches.gr/spiro-a20180627/ Steinberg, L. (2005). Cognitive and affective development in adolescence. Trends in Cognitive Sciences, 9(2), 69–74. https://doi.org/10.1016/j.tics.2004.12.005 Tempelaar, W. M., Otjes, C. P., Bun, C. J., Plevier, C. M., van Gastel, W. A., MacCabe, J. H., Kahn, R. S., & Boks, M. P. M. (2014). Delayed school progression and mental health problems in adolescence: A population-based study in 10,803 adolescents. BMC Psychiatry, 14(244). https://doi. org/10.1186/s12888-014-0244-5 Waldon, E. G., Lindahl Jacobsen, S., & Gattino, G. (2018). Assessment in music therapy: Psychometric and theoretical considerations. In S. L. Jacobsen, E. G. Waldon, & G. Gattino (Eds.), Music therapy assessment: Theory, research and application (pp. 47–80). Jessica Kingsley Publishers. Waltz, C. F., & Bausell, R. B. (1981). Nursing research: Design, statistics, and computer analysis. F.A. Davis. Waltz, C. F., Strickland, O. L., & Lenz, E. R. (2016). Measurement in nursing and health research. Springer Publishing Company. https://doi.org/10.1891/9780826170620 Wells, N. (1988). An individual music therapy assessment procedure for emotionally disturbed young adolescents. The Arts in Psychotherapy, 15(1), 47–54. https://doi.org/10.1016/0197-4556(88)90051-2 Widen, S. C., & Russell, J. A. (2010). Descriptive and prescriptive definitions of emotion. Emotion Review, 2(4), 377–378. https://doi.org/10.1177/1754073910374667 World Health Organization. (2021, August 9). Adolescent mental health. https://www.who.int/newsroom/fact-sheets/detail/adolescent-mental-health Yaghmaie, F. (2003). Content validity and its estimation. Journal of Medical Education, 3(1), 25–27. https://brieflands.com/articles/jme-105015.html Zachar, P. (2010). Has there been conceptual progress in the science of emotion? Emotion Review, 2 (4), 381–382. https://doi.org/10.1177/1754073910374668 Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A.-R. (2015). Design and implementation content validity study: Development of an instrument for measuring patient-centred communication. Tabriz University of Medical Sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017 14 M. SALOKIVI ET AL.