Examining the structure of ideas of reference in clinical and community samples

Examining the structure of ideas of reference in clinical and community samples Juan F. Rodríguez-Testal a , Pedro Bendala-Rodríguez a , Salvador Perona-Garcelán a,b , Cristina Senín-Calderón c, ⁎ a Personality, Evaluation and Psychological Treatment Department, University of Seville, Seville, Spain. P St. Camilo José Cela s/n, 41018 Seville, Spain b Virgen del Rocío Outpatient Mental Hospital, University Hospital Virgen del Rocío, Seville, Spain St. Manuel Siurot s/n, 41013 Seville, Spain c Department of Psychology, University of Cadiz, Ave. República Árabe Saharaui S/N, 11510 Puerto Real, Cádiz, Spain abstractarticle info Aims: This study addresses the psychometric properties of a Spanish validation of the REF scale of ideas of reference (IRs) in detecting and following at-risk mental states and psychosis. Methods: A total of 9447 participants were distributed in three groups: 676 patients with various diagnoses—154 with psychotic disorders, 6291 youths aged 11 to 20, and 2480 adult participants aged 21 to 84. Results: Youths had higher scores than adults on IRs, observing a progressive decrease and stabilization in the twenties. Exploratory factor analysis provided a structure for the overall IRs score, with five first-order dimensions and one second-order dimension. Confirmatory factor analysis supported the structure with excellent fit. The REF scale was invariant across sex and samples. The internal consistency of the complete scale was excellent and acceptable across the five first-order factors. Strong relationships were found with the positive dimension of the community assessment of psychic experience-42, as well as with aberrant salience. Low and moderate relationships were found with public self-consciousness, anxiety, and depression. Youths and patients diagnosed with schizophrenia and other psychotic disorders had a high mean IRs frequency. Male sex, greater age (among the adults), and the “causal explanations”,“Songs, newspapers, books”and “laughing and commenting” REF subscales showed predictive power in the diagnostic categories of schizophrenia and other psychotic disorders. Conclusions: The results provide satisfactory that the REF scale could be used to study psychosis. © 2019 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Keywords: Referential thinking Ideas of reference REF scale Proneness to psychosis Schizophrenia 1. Introduction Ideas of reference (IRs) are a form of self-referential processing [1] that concern the self and the distinction between self-generated and other-generated stimuli [2,3]. As IRs make inferences about the mental state and behavior of others [4,5], they could be considered a form of social cognition, which is a cognitive process that develops from adolescence onwards [6]. IRs, whether delusional or not, are frequent manifestations in various pathologies of the psychotic spectrum, as described in many classical studies [7–10]. However, since delusions are by nature selfreferential, even when they do not involve delusional IRs, it may be that self-referential processing has been given insufficient attention as regards understanding the onset and maintenance of delusional activity in general and delusional IRs in particular [11]. From this point of view, IRs could be understood as one part of a process that leads to delusions, anomalous experiences that can culminate in the assignment of meaning as a positive symptom [12]. Clinical research often describes IRs as psychotic-like experiences (PLEs), which are studied in light of the broad psychotic phenotype, a psychometric continuum of interest for analyzing the risk and onset of psychosis [13]. Although PLEs are relatively frequent in the general population (about 8%), they have a low incidence of transition to threshold psychosis, which requires that the PLEs be persistent [14,15], that various preclinical expressions interact [16], and finally, loss of insight about the experience. Nonetheless, the concept of PLEs itself has often been criticized as not having any uniform definition, with imprecise boundaries between what is normal and pathological, as well as variable characteristics and clinical manifestation [17]. On the other hand, PLEs reliably increase the risk of mental health service use [18], even in cases that were originally considered false positives [19]. Many studies have suggested that IRs are of particular clinical importance among PLEs, and that their frequency and severity are critical to analyze the psychotic process [20]. For example, they are significant predictors of physical anhedonia [21], persecutory thinking, paranoia stability, distrust [22–24] and transition to psychosis in at-risk subjects [25]. IRs appear in all types of responses within the hierarchical structure of paranoia in the general population, they are the basis of persecutory Comprehensive Psychiatry 93 (2019) 48–55 ⁎Corresponding author. E-mail address: [email protected] (C. Senín-Calderón). https://doi.org/10.1016/j.comppsych.2019.06.006 0010-440X/© 2019 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Contents lists available at ScienceDirect Comprehensive Psychiatry journal homepage: www.elsevier.com/locate/comppsych ideas [26], and can even be the basis of grandiose thinking [27]. With regards to prodromal indicators, IRs are among the basic symptoms as unstable IRs, as it happens with the Bonn scale for the assessment of basic symptoms (BSABS [28]) or the adult versions and children and adolescent de la schizophrenia proneness instrument (SPI-A/SPI-CY [29,30]). IRs are considered prodromal among the criteria of ultrahigh clinical risk [31], such as attenuated psychotic symptoms (stable IRs) and in Trait plus State Risk Factor (related to schizotypal disorder). IRs are also considered from the perspective of basic self-disturbance (e.g., examination of anomalous self-experience [32]; examination of anomalous world experience [33]), or, more generally, as a characteristic of schizotypy [34]. Lenzenweger, Bennett and Lilenfeld [35] developed the REF referential thinking scale to evaluate schizotypy, emphasizing that referential thinking is not necessarily linked to schizotypy and that it is not exclusively pathological. Later authors have attempted to differentiate socalled classical or simple IRs (they're looking at me, they're laughing at me, etc.) from guilt IRs [36]. Still others suggested IRs of communication (paralinguistic, non-verbal, mass media, inanimate objects) and of observation (related to paranoia [37]), and Cicero and Kerns [38]categorized IRs as either positive or negative. The REF scale [35] includes items on the ascription of causality to common coincidences [39], which is an attributional process related to the onset of delusional activity [40]. In summary, although research has tended to consider PLEs together, it may be of interest to analyze them separately [41]. IRs can be considered a PLE that is observable in the general population, and their relevance as a clinical sign probably depends on their intensity, or on the concern or distress they cause [42]. Furthermore, IRs may be responsible for the impaired reflective component of social cognition (social cue perception and mentalizing) seen in schizophrenia [4]. The REF scale is extensively used in research. The present study proposes a validation of the REF scale in Spanish, exploring its fitatdifferent ages and in different diagnostic categories. As mentioned above, the scale presumes that IRs are prominent in adolescence, during which there is intense development of the self and of social information-pro- cessing [43]. To make IRs more useful for the study of psychometrics and the clinical risk of psychosis, their psychometric characteristics relevance as clinical indicators must be specified. In brief, IRs as a PLE can occur naturally in the general population, but their continuity over time may indicate the approach of psychotic activity [44]. The present study aimed to (1) elucidate the average prevalence of IRs by age and diagnostic group, (2) study the internal structure of the REF scale, (3) analyze the internal consistency of the scale and the evidence for convergent and divergent validity with other scales, and (4) study the predictive capacity of the REF subscales for detecting the psychometric risk of psychosis. 2. Methods 2.1. Participants The final sample of this study comprised 9447 participants, after eliminating 659. The sample was distributed in three groups (see Table 1): the first consisted of youths from the general population without psychopathology and aged from 11 to 20 (healthy youth controls), the second comprised subjects from the general population without psychopathology and aged 21 to 84 (healthy adult controls), and the third was made up of patients with psychopathology aged 18 to 79. A total of 310 youths and 338 adults were eliminated from the sample because (1) they reported some psychopathology during the period of study (16%), (2) were taking psychopharmacological treatment (36.6%), (3) data were missing on the REF scale (28.7%), or (4) they did not answer all the questions (27%). Eleven patients were eliminated because their primary diagnosis comprised too few cases for analysis. 2.2. Procedure The data were collected across several studies using various procedures. The youths took the tests in groups at their schools during class time. Of the adults, 44.9% were university students who filled out the tests in class, while the rest were found through snowball sampling. The patient sample was collected by incidental sampling of patients at one private psychology clinic and various public hospitals. The diagnoses were made by psychologists with over 20 years' clinical experience. To facilitate later statistical analysis, primary diagnoses were grouped together into categories, following the DSM-IV-TR [45]. All participants were informed of the research objectives and gave their written consent. Parents or guardians of the minors signed their consent. The study followed the precepts of the Declaration of Helsinki and was approved by the ethical committee. 2.3. Materials Before the evaluation tests were administered, the participants completed a questionnaire about psychopathology, psychoactive drug prescription, sex, and age. 2.3.1. REF referential thinking scale [35] This is a self-report that indicates whether the frequency of the subject's IRs suggests a clinical condition or is relatively normal within the general population. It consists of 34 items with “true/false”answer choices. The scale's authors contrived five factors: (1) laughing/ commenting—the belief that others are laughing at the subject or commenting on the subject's behavior, (2) attention/appearance—the interpretation that other persons are paying special attention to the subject's appearance, (3) guilt/shame—related to the guilt or shame the subject feels, (4) songs, newspapers, and books—beliefs that the communications media, books or stories, contain messages for the subject, (5) reaction—changes or reactions of the subject. The total score of the REF scale is calculated as the sum of its items. The scale's authors found an internal consistency of 0.83 to 0.85 for the overall scale, as well as a strong relationship with measures evaluating perceptual aberration and magical ideation. Test-retest reliability was 0.86 after 4 weeks. Lezenweger's original article and the REF scale were translated into Spanish by a team comprising a native English speaker and one of the authors, both of whom were experts in its clinical contents. They Table 1 Key characteristics of the sample distributed in three groups. Characteristics Healthy control youth (n= 6291) Healthy control adults (n = 2480) Patients (n= 676) Mean age in years (SD) 15.54 (2.53) 31.56 (12.38) 34.63 (12.44) Female % 58.7 63.2 59 Diagnosis (n) Depressive disorders 136 Adjustment disorders 64 Anxiety disorders 152 Somatoform disorders 44 Bipolar disorders 34 Schizophrenia and other psychotic disorders (SZOP) (3.89% residual stage) 154 Eating disorders 23 Others applicable to Axis 1 17 Personality disorders cluster “A”11 Schizotypal personality disorder 5 Paranoid personality disorder 5 Schizoid personality disorder 1 Personality disorders cluster “B”15 Personality disorders cluster “C”8 Unspecified personality disorders 18 49J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55 were then translated back into English and analyzed for unintended changes. The Spanish translation was then adjusted accordingly. Several clinicians reviewed the final version in Spanish (Appendix A in Supplementary Material). This version was applied to the university students and patients, with α-values of 0.83 and 0.91, respectively, and test-re- test reliability values of 0.66 and 0.76 for students and patients [46]. 2.3.2. Community assessment of psychic experiences-42 (CAPE-42) [47,48] This self-report evaluates attenuated positive and negative psychotic experiences and depressive symptoms. It has 42 items with Likert-type answer choices from (1) “almost never”to (4) “almost always.”The internal consistency of the Spanish validation was appropriate (α-value = 0.78 to 0.93) and was related to delusional ideation as well as to trait and state anxiety. The positive dimension was used in the present study. The α-value for the internal consistency of this dimension with the study sample was 0.75. 2.3.3. Aberrant salience inventory [49,50] This scale consists of 29 items with “yes/no”answer choices. It evaluates the attribution of significance to stimuli that are usually irrelevant. The complete scale showed high internal consistency (α= 0.89) and evidence of concurrent validity with other measures of proneness to psychosis. In the present study, the scale had an α-value of 0.86 for the entire sample. 2.3.4. General health questionnaire (GHQ-28) [51,52] This screening scale evaluates symptoms of anxiety, depression, somatization, and social dysfunction. It has adequate reliability (test–retest value = 0.90) and validity (sensitivity: 44%–100%, specificity: 74%–93%). In the present study, only the anxiety (α= 0.77) and depression (α= 0.79) subscales were used. 2.3.5. Revised self-consciousness scale [53,54] This scale has 22 items with five answer choices from “not at all characteristic”to “very characteristic.”It evaluates private and public self-consciousness and social anxiety. The Spanish version has adequate reliability (α-value for private self-consciousness = 0.92, α-value for public self-consciousness = 0.75, α-value for social anxiety = 0.81), and validity (content and construct) indicators. In the present study, the public self-consciousness subscale was used, with an α-value of 0.78 in the study sample. 2.4. Statistical analyses The healthy population sample (youths and adults) was divided into age groups by decades. A one-way ANOVA of the total REF score was then performed on these age groups. Next, a one-way ANOVA of the average REF diagnostic categories was performed. The total sample (patients, youths and adults healthy) was divided at random into two groups. Exploratory factor analysis (EFA) was used with Sample 1 to find evidence of construct validity, and the structure found was subjected to confirmatory factor analysis (CFA) with Sample 2. Internal consistency was found with the ordinal α-value. The structure found was subjected to confirmatory factor analysis (CFA) by applying the “robust diagonally weighted least squares”method to the asymptotic covariance matrix. It was then compared with other models (null model, unidimensional model, and original model by Lenzenweger [35]). The models' fit was evaluated using the following goodness-of-fit criteria: Satorra-Bentler chi-square (SB-χ 2 ), comparative fit index (CFI), nonnormed fit index (NNFI), incremental fit index (IFI), which had to be over 0.90 [55], root mean square error of approximation (RMSEA) and its 90% confidence interval and standardized root mean square residual (SRMR), which must be below 0.05 to be considered adequate, and between 0.05 and 0.08 to be considered acceptable [56]. In addition, Akaike's information criterion (AIC) were used to find the most parsimonious model. Multi-group confirmatory factor analyses (MGCFA) were conducted to test the invariance of the REF scale across samples and sex, following the steps recommended by Byrne [57], in which each new model is nested in the previous one and parameters are subjected to increasingly restrictive tests of equality. Firstly, the unconstrained baseline model was estimated separately for each group. A MGCFA was then carried out to evaluate configural invariance (M0), whereby the model parameters were estimated freely across groups. Next the model was tested to ensure that the constraining factor loadings were equivalent in the two groups (M1; factor loading invariance or metric invariance). The previous invariance of the factor loadings was then maintained, and all factor variances and covariances were constrained to ensure they were the same across groups (M2, structural invariance). The model fit was assessed in terms of ΔCFI and ΔRMSEA. Invariance was identified when (1) ΔCFI was ≤0.010 and (2) ΔRMSEA was ≤0.015 [58]. Convergent and divergent validity was assessed using Pearson's correlation coefficients. The size of the correlations was interpreted using the method described by Cohen [59] wherein rvalues ≥0.10, 0.30, and 0.50 are used as benchmarks for small, medium, and large effects, respectively. A binary logistic regression analysis was performed on the participants in the adult group, with schizophrenia and other psychotic disorders (SZOP), sex, and age in the first block, constituting the dependent variables, and REF factors in the second block, constituting the independent variables. This same analysis was repeated with the youths, whereby the age variable was eliminated (only three participants with SZOP were under 20 years of age). Statistical analyses were performed using Factor 10.4.01, SPSS 24, and Lisrel 8.7 software. 3. Results 3.1. Preliminary analyses One-way ANOVA of the total REF score between the youth and adult age groups in the general population showed statistically significant differences: F(5, 8765) = 267.97, pb.001, η 2 = 0.13. The post-hoc Dunnet's C-test showed that the two youth groups had a higher average REF score than the rest of the age groups. A decrease in IRs average with age was observed (Fig. 1; Table S1 in the supplementary material shows an analysis by age). This was fitted to a logarithmic trend of negative tendency and explained almost 76% of the variance (y=−1.98 ln[x] + 10.381). One-way ANOVA of the average REF score between the diagnostic categories found significant differences: F(11, 664) = 6.98, pb.001, η 2 = 0.10 (Table S2 in the supplementary material). The subjects with Schizophrenia and other psychotic disorders (SZOT) had a higher REF average (M= 12.53, SD = 8.44). The Dunnet's C-test showed significant differences between subjects with SZOT and those with depressive disorders (M= 7.37, SD = 6.40), anxiety (M= 7.74, SD = 6.22), adjustment disorders (M= 6.19, SD = 5.16), other disorders applicable to Axis I (M= 7.64, SD = 6.90), Group C personality disorders (M= 4.63, SD = 4.37), and unspecified personality disorders (M=7.33,SD = 4.81). 3.2. Evidence of validity based on the scale's internal structure The total sample was divided at random into two groups. The sociodemographic variables (sex and age, pN.05), sample (patients, youths and adults healthy) and REF overall measurements (pN.05) were equivalent in both groups (Table S4 in the supplementary material). 3.2.1. Exploratory factor analyses The EFA of the REF scale done with Sample 1 (n= 4781) yielded five first-order factors and one second-order factor, explaining 54% of the variance. The EFA was estimated using the “robust diagonally weighted least squares”method on the tetrachoric correlation matrix, with 50 J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55 promin rotation. Both the Kaiser–Meyer–Olkin test (0.92) and Bartlett's statistic (X 2 (561) = 26,470.3, pb.001) indicated that these data had adequate factorability. Schwarz's Bayesian information criterion recommended extraction of five factors that were similar to those found by the authors of the scale, with some differences, as mentioned below. The first factor corresponds to what the authors call “attention and appearance”and is loaded with Items 8, 9, 18, 21, 24 to 26 and 32, but only the last three itemsand Item8 coincide with those foundby the authors of the scale. However, the content of the rest of the items fits perfectly with the “attention and appearance”IRs. The second factor concerns “laughing and commenting”and is loaded with items 1 to 4, 6, and 14, all of which corroborate the factor proposed by the scale's authors. The third factor corresponds to “songs, newspapers, and books” and is loaded with items 10 to 13. These items coincide with the original factor by Lenzeweger et al. 1997, however, in the validation by the authors, four additional items loaded which in this EFA saturated on Factor 5. Items 28 to 31 and 33 load on the third factor, which relates to “guilt and shame”IRs. All these items coincide with the factor found by the scale's authors. Items 5, 7, 15, 16, 17, 19, 20, 22, 23, 27, and 34 are grouped together in a fifth factor. This grouping does not correspond to any of the factors found by the authors of the REF scale. The content of these items reflects self-referential interpretations of causality in random or irrelevant events. As such, this factor has been called “causal explanations”. Items 16 and 19 have very low communalities (0.175 and 0.070 respectively) and low factor loadings. They can also problematic. Specifically, item 16 refers to driving, which not all respondents do. As such, it is not very discriminative. Similarly, item 19 is the only item on the scale that phrases the question negatively, which may affect the respondents' understanding. Table 2 shows the completely standardized factor loadings. 3.2.2. Reliability The ordinal αcoefficient was estimated for each of the factors in the three samples of participants (see Table 2). All the REF subscales had acceptable indicators in all three samples, although the lowest values were found in the sample of youths, where the αin the “songs, newspapers, and books”, not reach 0.70. The ordinal α-value of the total scale across all three samples was 0.97 (patients), 0.94 (youths), 0.97 (adults). 3.2.3. Confirmatory factor analysis Several CFAs were performed with Sample 2 (n= 4666) to test the following models: a null model (Model 1), a unidimensional model (Model 2), the internal structure of the REF scale with in the EFA (Model 3), and the original structure found by Lenzenweger [35] (Model 4). The results of all the CFAs showed excellent fit, but the model with five first-order factors and one second-order factor had the highest goodness-of-fit indicators and the lowest AICs (Table 3). 3.2.4. Measurement invariance across samples and sex Firstly, individual CFAs were carried out for all cases in each group to find the baseline model, and the goodness-of-fit indicators were found to be adequate. Several multi-group CFAs were then performed on the REF scale across both sex and samples. The unconstrained model (M0) showed adequate fit in both comparisons across samples and across sexes (CFI and RMSEA b0.05), indicating that the number of underlying factors was equivalent across samples and sex (see Table S5 in the Fig. 1. Average and logarithmic trend of ideas of reference. Table 2 Standardized item-factor loadings for the REF scale with sample 1 (n = 4781). Variable Attention, appearance Laughing, commenting Songs, newspapers, books Guilt, shame Causal explanations Item 8 0.871 0.054 −0.046 −0.006 −0.171 Item 24 0.741 0.165 −0.071 0.115 −0.183 Item 25 0.685 −0.145 0.134 −0.121 0.108 Item 32 0.530 0.003 0.065 −0.103 0.328 Item 21 0.514 −0.062 0.009 −0.063 0.249 Item 9 0.500 0.302 −0.009 0.101 −0.042 Item 26 0.477 −0.121 0.084 0.040 0.214 Item 18 0.436 0.219 −0.001 0.032 0.097 Item 3 −0.131 0.864 0.083 0.081 −0.116 Item 2 0.141 0.814 0.032 −0.155 −0.025 Item 14 0.042 0.788 −0.081 0.081 0.043 Item 1 −0.051 0.743 0.118 0.103 −0.135 Item 4 0.143 0.583 −0.085 −0.022 0.161 Item 6 −0.189 0.564 −0.076 −0.018 0.385 Item 10 0.032 −0.030 0.760 −0.017 −0.142 Item 11 0.034 −0.060 0.706 0.038 −0.137 Item 13 −0.001 −0.055 0.596 −0.072 0.217 Item 12 0.111 −0.018 0.461 0.060 0.103 Item 31 −0.036 0.074 0.016 0.757 0.049 Item 30 −0.115 0.038 −0.042 0.699 0.144 Item 28 −0.012 0.011 0.220 0.531 −0.019 Item 29 0.021 0.119 0.204 0.399 −0.095 Item 33 0.234 0.201 −0.001 0.344 0.044 Item 15 −0.239 0.360 0.249 −0.425 0.887 Item 17 0.007 −0.128 −0.204 0.197 0.606 Item 22 0.146 −0.168 0.070 0.121 0.569 Item 5 0.119 0.261 −0.134 −0.024 0.474 Item 34 0.297 −0.050 0.082 −0.061 0.474 Item 27 −0.034 −0.031 −0.048 0.255 0.409 Item 23 0.151 0.364 −0.047 −0.030 0.402 Item 20 0.118 −0.014 0.001 0.126 0.355 Item 16 −0.038 −0.137 0.059 0.252 0.262 Item 7 0.153 0.098 −0.033 0.136 0.248 Item 19 0.034 0.152 0.061 −0.118 0.165 Ordinal αreliability analysis Patients 0.90 0.94 0.87 0.85 0.90 Adults 0.89 0.92 0.81 0.83 0.83 Youths 0.84 0.86 0.69 0.77 0.72 Primary loadings for each observed variable are in bold. 51J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55 supplementary material). Next, the MGCFA was performed to test the factor loading invariant (M1). The comparisons between the M1 and M0 models were not significant (ΔCFI b0.01 and ΔRMSEA b0.015), indicating that the factor loading pattern was equivalent across groups and thus that the content of each item was perceived and interpreted in the same way across the samples. Finally, structural invariance was tested (M2). When M2 was compared with M1, no significant increase was observed in the CFI and RMSEA indices, indicating that the dimensionality of the constructs was maintained across groups. These results show that the REF structure, the relationship between the indicators of each variable and their respective latent factor, and the relationship among the latent variables were equivalent across samples and sex. 3.3. Evidence of validity with respect to other scales In all samples, the total REF score was strongly related to the positive dimension of the CAPE scale, as well as to aberrant salience, indicating convergent validity. Conversely, divergent validity showed low and moderate relationships with public self-consciousness, anxiety, and depression in all three study samples. These results are shown in Table 4, while the correlations are shown with the REF subscales in Table S6 of the supplementary material. 3.4. Evidence of discriminative validity Binary logistic regression analysis revealed that male sex, age, and high score in “causal explanations”,“Songs, newspapers, books”and “laughing and commenting”were clinically useful variables for differentiatingpatients diagnosed with SZOP from adults with no psychopathology, explaining 40.6% of the variance. In the analysis of the group of youths and the participants diagnosed with SZOP, the percentage of explained variance was somewhat lower (15.6%). All the variables entered were statistically significant. However, as in the analysis of the adults, the “causal explanations”and “laughing and commenting”factors were the best predictors of this diagnostic category (Table 5). 4. Discussion The main objective of this study was to explore the fit of the REF scale at different ages in the healthy population, as well as in diagnostic categories with patients. IRs, like PLEs in the general population, may provide insight into the onset and maintenance of psychosis. However, the study of PLEs has a drawback because it tends to either under- or overestimate the tendency towards transition to psychosis [15,60]. Nevertheless, its usefulness has also been emphasized [61,62]. Specifically, PLEs span different mental processes that are changeable or unstable until finally becoming delusions [63]. One key to this transition from initial IRs to delusional activity is probably IRs stability, frequency, distress, worry, etc. [42]. However, it should be borne in mind that this study is cross-sectional, and so any allusion to the temporal dimension of IRs must always be tentative and merely made as a proposal. IRs, like PLEs, are a prodromal indicator or manifestation of different disorders. To illustrate, they have been included among the criteria for body dysmorphic disorder [64]. They are involved in both normal and pathological evolutionary development, but are a major factor in psychotic conditions [6,65]. In the present study, comparisons of IRs as a PLE among age groups showed that youths had the highest scores on the REF scale (large effect size). This result is consistent with the negative relationship between the REF score and age found by Startup, Sakrouge, and Mason [66]. More generally, it corroborates studies involving measures related to paranoia [67] or delusions [68]. In a more detailed analysis of IRs evolution in the present study (Fig. 1 and Table S1 in the supplementary material), the frequency of IRs became stabilized at ages in the twenties. One precedent in the literature showed similar distribution by age and sex, although only one item was related to IRs [69]. This distribution by age may indicate that interpersonal sensitivity develops and adjusts from adolescence to young adulthood [43]. The mean IRs frequency after 21 years of age in the present study corroborated that found in subjects with no psychopathology by the authors of the scale. However, Startup et al. [66] reported an extremely low mean (1.52, SD = 1.99). Patients diagnosed with SZOP should have a higher mean than others, although patients with Group A and B personality disorders and bipolar disorder also have higher means (N10 points). However, the difference is too small to draw any definite conclusions. Nonetheless, when diagnoses are compared to each other or to normal functioning, IRs are generally viewed as static. When IRs evolution is analyzed at different moments in time, change can be observed in such parameters as frequency, conviction, and distress [70], indicating that longitudinal designs must be carried out to verify the stabilization mentioned above, as well as its possible relationship to psychosis onset in specific at-risk populations. As has already been done with other PLEs, such as aberrant salience [44], a follow-up of IRs should be developed using experience sampling or time series. These techniques Table 3 Fit indices of confirmatory factor analysis for the four models with sample 2 (n = 4666). Models a SB-X 2 df CFI NNFI IFI RMSEA [90% CI] SRMR AIC Model 1 266,370.491 561 266,438.735 Model 2 4121.171 527 0.986 0.986 0.986 0.038 [0.037, 0.039] 0.078 4763.644 Model 3 1914.823 522 0.995 0.994 0.995 0.024 [0.022, 0.025] 0.061 2060.323 Model 4 3092.462 517 0.990 0.989 0.990 0.033 [0.032, 0.034] 0.073 3829.411 a Model 1: Null model; Model 2: Unidimensional model; Model 3: The internal structure of the REF scale found in the EFA; Model 4: Original structure of the REF scale obtained by Lenzenweger et al. [35]. Table 4 Pearson's correlations for the REF total and other variables. Patient group CAPE Positive (n= 33) Aberrant salience (n= 271) Public self-consciousness (n= 190) GHQ-Anxiety (n= 455) GHQ-Depression (n= 455) REF total 0.656⁎⁎ 0.557⁎⁎ 0.225⁎⁎ 0.125⁎⁎ 0.199⁎⁎ Healthy control youth CAPE positive (n= 307) Aberrant salience (n= 4788) Public self-consciousness (n= 866) GHQ- Anxiety (n= 726) GHQ- Depression (n= 726) REF total 0.528⁎⁎ 0.586⁎⁎ 0.342⁎⁎ 0.330⁎⁎ 0.354⁎⁎ Healthy control adults Cape positive (n= 178) Aberrant salience (n= 775) Public self-consciousness (n= 799) GHQ- Anxiety (n= 1189) GHQ- Depression (n= 1189) REF total 0.564⁎⁎ 0.460⁎⁎ 0.274⁎⁎ 0.300⁎⁎ 0.289⁎⁎ ⁎⁎ pb.001. 52 J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55 are a useful precedent for following up IRs in the residual phase of schizophrenia [71]. Concerning the internal structure of the REF scale, the EFA supported five first-order factors and one second-order factor, also recommending the total score of the scale. The structure differs slightly from the one found by Lenzenweger et al. [35]inuniversity students, particularly in the factor known as “causal explanations” in the present study. With only three items, the original subscale (reaction) may have had problems with internal consistency, although the scale's authors did not report on its reliability. Both the overall REF scale and the five first-order factors showed adequate internal consistency, especially in the sample of patients and adults. The analysis of invariance showed that males and females had an equivalent interpretation of the items on the REF scale, as did patients and participants from the general population. No other investigations regarding validation of the REF scale could be found for comparison in the literature, other than the original paper introducing the instrument. Therefore, the present study complements other procedures and samples to forward the understanding of IRs as a mental process related to normal and pathological development. Compared to other measures, the REF scale showed strong correlation with the positive dimension of the CAPE-42 scale and with aberrant salience, corroborating the results of the scale's authors and of Meyer and Lenzenweger [34] with regards to positive schizotypy and measures of proneness to psychosis (“perceptual aberration”and “magical ideation”). Other studies related to positive symptoms have shown that the relationship of IRs with inner speech is mediated by dissociation [72], as well as by memories of childhood threat and submission [73]. The relationship between IRs and public selfconsciousness was low in the group of patients and adults, but moderate in youths. These results suggest, as found by Lenzenweger et al. [35], that referential thinking is a different construct from awareness of the self as a social object. Finally, the correlation between IRs and anxiety and depressive symptoms was low in the group of patients and adults, while it was moderate in the group of youths, corroborating the reports of Lenzenweger et al. [35] and Meyer & Lenzenweger [34]. The regression analyses showed that male sex, in both youths and adults, predicted the SZOP diagnostic category [74]. Among the REF scale factors, the best predictors for comparing adults were “causal explanations”,“Songs, newspapers, books”and “laughing and commenting,”and all the factors were good predictors in youths. It follows that the different content of the analyzed IRs is significant. The present study had some limitations. It was a cross-sectional study that used the data sets of several research projects, so many participants did not fill out all the tests. Furthermore, the retest reliability of the current sample is questionable. The section on instruments provides the previous results. However, in the present study, the retest could not be applied to the entire sample. In the patient group, the diversity of the primary diagnoses necessitated grouping by diagnostic category, so the possibility to study IRs within each specific disorder was limited. The diagnoses made by the authors did not follow any structured interviews, and no reliability indicators were set for them; this could limit some of the results. To specify the effect of age on prediction in youths, a larger group of youths with SZOP should have been used. The present study emphasized the importance of the REF scale to differentiate IRs from other PLEs. However, associated factors should also be considered, such as functional decline, distress, deficient coping, and self-distur- bance [3,75], although a previous study has investigated concern about IRs [42]. Nonetheless, the present study had some strengths. It was thefirst to validate the REF since the scale was developed, and it involved a large representative sample of all ages. This enabled the study of IRs from adolescence to old age, providing a scale for their interpretation and comparison (Tables S1 and S3 in the supplementary information), as well as evidence for their invariance across sex, healthy populations, and clinical subjects. In conclusion, the data provided in the present study suggest that the REF scale could be used to detect IRs with psychometric guarantees, from what is considered normal according to the developmental stage of the subject to what is considered clinically relevant. Future studies should revise the prospective evaluation of this measure, with patient follow-ups and subgroups of at-risk participants. They should also establish the relationship of the REF scale with clinical risk, as determined by diagnostic interview, and with other indicators, such as anxiety, depression, and negative symptoms [76]. Funding The present study was supported by the Ministry of Economy and Competitiveness (Spain) and co-funded by the European Regional Development Fund (grant No.: PSI2015-68659-P). Declaration of Competing Interest The authors declare that they have no conflicts of interest related to the present study. Table 5 Results of binary logistic regression analysis of REF total factors associated with presence of the schizophrenia and other psychotic disorders diagnostic class. Outcome variables Model Predictor variables OR a 95% CI pNagelkerke's R 2 Healthy control adults vs SZOP b Step 1 Sex 1.09 [2.107, 4.225] b0.001 0.083 Age 0.032 [1.021, 1.044] b0.001 Step 2 Sex 2.720 [1.770, 4.179] b0.001 0.406 Age 1.172 [1.029, 1.060] b0.001 F1 c 0.076 [0.949, 1.227] 0.247 F2 0.531 [1.481, 1.953] b0.001 F3 0.220 [1.033, 1.503] 0.021 F4 0.103 [0.953, 1.290] 0.182 F5 0.300 [1.188, 1.532] b0.001 Healthy control youth vs SZOP Step 1 Sex 1.025 [1.988, 3.906] b0.001 0.029 Step 2 Sex 1.085 [2.056, 4.257] b0.001 0.156 F1 0.174 [0.758, 0.932] b0.001 F2 0.486 [1.458, 1.811] b0.001 F3 0.225 [1.073, 1.463] 0.004 F4 0.152 [0.750, 0.984] 0.028 F5 0.300 [1.225, 1.487] b0.001 a OR: odds ratio. b SZOP: Schizophrenia and other psychotic disorders. c F1: Attention, appearance; F2: Laughing, commenting; F3: Songs, newspapers, books; F4: Guilt, shame; F5: Causal explanations. 53J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55 Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi. org/10.1016/j.comppsych.2019.06.006. References [1] Iijima Y, Takano K, Boddez Y, Raes F, Tanno Y. Stuttering thoughts: negative self-ref- erent thinking is less sensitive to aversive outcomes in people with higher levels of depressive symptoms. Front Psychol 2017;8:1–15. https://doi.org/10.3389/fpsyg. 2017.01333. [2] Brent BK, Seidman LJ, Thermenos HW, Holt DJ, Keshavan MS. Self-disturbances as a possible premorbid indicator of schizophrenia risk: a neurodevelopmental perspective. Schizophr Res 2014;152:73–80. https://doi.org/10.1016/j.schres.2013.07.038. [3] Nelson B, Yung AR. Psychotic-like experiences as overdetermined phenomena: when do they increase risk for psychotic disorder? Schizophr Res 2009;108: 303–4. https://doi.org/10.1016/j.schres.2008.10.006. [4] Green MF, Horan WP, Lee J. Social cognition in schizophrenia. Nat Rev Neurosci 2015;16:620–31. https://doi.org/10.1038/nrn4005. [5] Wastler HM, Lenzenweger MF. Cone of gaze in positive schizotypy: relationship to referential thinking and social functioning. Personal Disord Theory Res Treat 2018; 9:324–32https://doi.org/10.1037/per0000258. [6] Senín-Calderón C, Perona-Garcelán S, Fuentes-Márquez S, Rodríguez-Testal JJF. A mediation model for ideas of reference: the role of the gray model, self-conscious- ness, and emotional symptoms. Psychol Rep 2017;120:443–59. https://doi.org/10. 1177/0033294117693593. [7] La Conrad K. Esquizofrenia incipiente [Incipient schizophrenia]. Madrid: Fundación Archivos de Neurobiología; 1958. [8] Jaspers K. Psicopatología general [General Psychopathology]. Buenos Aires: Editorial Beta; 1977. [9] Kretschmer E. El delirio sensitivo de referencia [Sensitive delusion of reference] Triacastel. Madrid ; 1966. [10] Schneider K. Psicopatología Clínica. Madrid: Fundación Archivos de Neurobiología; 1997. [11] Wong GHY, Hui CLM, Tang JYM, Chiu CPY, Lam MML, Chan SKW, et al. Screening and assessing ideas and delusions of reference using a semi-structured interview scale: a validation study of the Ideas of Reference Interview Scale (IRIS) in early psychosis patients. Schizophr Res 2012;135:158–63. https://doi.org/10.1016/j.schres.2011.12. 006. [12] Maher BA, Spitzer M. Delusions. In: Sutker PB, Adams HE, editors. Compr. Handb. Psychopathol. New York: Plenum Press; 2001. p. 294. [13] Van Os J, Linscott RJ. Introduction: the extended psychosis phenotype - relationship with schizophrenia and with ultrahigh risk status for psychosis. Schizophr Bull 2012; 38:227–30. https://doi.org/10.1093/schbul/sbr188. [14] Van Nierop M, Van Os J, Gunther N, Myin-Germeys I, De Graaf R, Ten Have M, et al. Phenotypically continuous with clinical psychosis, discontinuous in need for care: evidence for an extended psychosis phenotype. Schizophr Bull 2012;38:231–8. https://doi.org/10.1093/schbul/sbr129. [15] Zammit S, Kounali D, Cannon M, David AS, Gunnell D, Heron J, et al. Psychotic experiences and psychotic disorders at age 18 in relation to psychotic experiences at age 12 in a longitudinal population-based cohort study. Am J Psychiatry 2013;170:742– 50. doi:10/vxr. [16] Wigman JTW, De Vos S, Wichers M, Van Os J, Bartels-Velthuis AA. A transdiagnostic network approach to psychosis. Schizophr Bull 2017;43:122–32. https://doi.org/10. 1093/schbul/sbw095. [17] Lee KW, Chan KW, Chang WC, Lee EHM, Hui CLM, Chen EYH. A systematic review on definitions and assessments of psychotic-like experiences. Early Interv Psychiatry 2016;10:3–16. https://doi.org/10.1111/eip.12228. [18] Bhavsar V, Maccabe JH, Hatch SL, Hotopf M, Boydell J, McGuire P. Subclinical psychotic experiences and subsequent contact with mental health services. BJPsych Open 2017;3:64–70. https://doi.org/10.1192/bjpo.bp.117.004689. [19] van der Steen Y, Myin-Germeys I, van Nierop M, ten Have M, de Graaf R, van Dorsselaer S, et al. ‘False-positive’self-reported psychotic experiences in the general population: an investigation of outcome, predictive factors and clinical relevance. Epidemiol Psychiatr Sci 2018:1–12. https://doi.org/10.1017/S2045796018000197. [20] Peters E, Ward T, Jackson M, Morgan C, Charalambides M, Mcguire P, et al. Clinical, socio-demographic and psychological characteristics in individuals with persistent psychotic experiences with and without a “need for care”. World Psychiatry 2016; 15:41–52. https://doi.org/10.1002/wps.20301. [21] Unterrassner L, Wyss TA, Wotruba D, Haker H, Rössler W. The intricate relationship between psychotic-like experiences and associated subclinical symptoms in healthy individuals. Front Psychol 2017;8. https://doi.org/10.3389/fpsyg.2017.01537. [22] Bell V, O'Driscoll C. The network structure of paranoia in the general population. Soc Psychiatry Psychiatr Epidemiol 2018:1–8. https://doi.org/10.1007/s00127-018- 1487-0. [23] Freeman D, McManus S, Brugha T, Meltzer H, Jenkins R, Bebbington P. Concomitants of paranoia in the general population. Psychol Med 2011;41:923–36. https://doi.org/ 10.1017/S0033291710001546. [24] Morrison SC, Cohen AS. The moderating effects of perceived intentionality: exploring the relationships between ideas of reference, paranoia and social anxiety in schizotypy. Cogn Neuropsychiatry 2014;19:527–39. https://doi.org/10.1080/ 13546805.2014.931839. [25] Salokangas RKR, Dingemans P, Heinimaa M, Svirskis T, Luutonen S, Hietala J, et al. Prediction of psychosis in clinical high-risk patients by the Schizotypal Personality Questionnaire. Results of the EPOS project. Eur Psychiatry 2013;28:469–75. https://doi.org/10.1016/j.eurpsy.2013.01.001. [26] Bebbington PE, McBride O, Steel C, Kuipers E, RadovanovičM, Brugha T, et al. The structure of paranoia in the general population. Br J Psychiatry 2013;202:419–27. https://doi.org/10.1192/bjp.bp.112.119032. [27] Barragan M, Laurens KR, Navarro JB, Obiols JE. Psychotic-like experiences and depressive symptoms in a community sample of adolescents. Eur Psychiatry 2011 (6):396–401. https://doi.org/10.1016/j.eurpsy.2010.12.007. [28] Gross G, Huber G, Klosterkötter J, Linz M. BSABS. Bonn scale for the assessment of basic symptoms. Shaker Verlag: Aachen; 2008. [29] Schultze-Lutter F, Addington J, Ruhrmann S, Klosterkötter J. Schizophrenia proneness instrument, adult version (SPI-A). Roma: Giovanni Fioriti Editore s.r.l; 2007. [30] Marshall M, Schultze-Lutter F, Marshall M, Koch E. Schizophrenia proneness instrument child and youth version (SPI-CY). Geneva: Giovanni Fioriti; 2012. [31] Yung AR, Phillips LJ, Yuen HP, McGorry PD. Risk factors for psychosis in an ultra high-risk group: psychopathology and clinical features. Schizophr Res 2004;67: 131–42. https://doi.org/10.1016/S0920-9964(03)00192-0. [32] Parnas J, Møller P, Kircher T, Thalbitzer J, Jansson L, Handest P, et al. EASE: examination of anomalous self-experience. Psychopathology 2005;38:236–58. https://doi. org/10.1159/000088441. [33] Sass L, Pienkos E, Skodlar B, Stanghellini G, Fuchs T, Parnas J, et al. EAWE: examination of anomalous world experience. Psychopathology 2017;50:10–54. doi:10.1159/ 000454928. [34] Meyer EC, Lenzenweger MF. The specificity of referential thinking: a comparison of schizotypy and social anxiety. Psychiatry Res 2009;165:78–87. https://doi.org/10. 1016/j.psychres.2007.10.015. [35] Lenzenweger MF, Bennett ME, Lilenfeld LR. The referential thinking scale as a measure of schizotypy: scale development and initial construct validation. Psychol Assess 1997;9:452–63. https://doi.org/10.1037/1040-3590.9.4.452. [36] Wing J, Cooper J, Sartorius N. The description and classification of psychiatric symptoms: an instruction manual for the PSE and CATEGO system. London: University Press; 1974. [37] Startup M, Startup S. On two kinds of delusion of reference. Psychiatry Res 2005; 137:87–92. https://doi.org/10.1016/j.psychres.2005.07.007. [38] Cicero DC, Kerns JG. Unpleasant and pleasant referential thinking: relations with self-processing, paranoia, and other schizotypal traits. J Res Pers 2011;45:208–18. https://doi.org/10.1016/j.jrp.2011.02.002. [39] Juston Osborne K, Mittal VA. Every-day coincidences and referential thinking: differentiating normative experiences from symptoms in psychosis. Schizophr Res 2018; 197:570–1. https://doi.org/10.1016/j.schres.2017.11.008. [40] Moritz S, Bentall RP, Kolbeck K, Roesch-Ely D. Monocausal attribution and its relationship with reasoning biases in schizophrenia. Schizophr Res 2018;193:77–82. https://doi.org/10.1016/j.schres.2017.06.057. [41] Elahi A, Perez Algorta G, Varese F, McIntyre JC, Bentall RP. Do paranoid delusions exist on a continuum with subclinical paranoia? A multi-method taxometric study. Schizophr Res 2017;190:77–81. https://doi.org/10.1016/j.schres.2017. 03.022. [42] Senín-Calderón C, Rodríguez-Testal JF. Perona- Garcelán S, Senín-Calderón, C., Rodríguez-Testal JF, Perona-Garcelán S. Las ideas de referencia y la preocupación por su presencia: estudio sobre su relevancia para la caracterización de las psicosis [ideas of reference and the concern for their presence: study about their relevance for the characterization of psychosis]. Rev Iberoam Psicol y Salud 2016;7:1–8. https://doi.org/10.1016/j.rips.2015.10.004. [43] Steinberg L. Cognitive and affective development in adolescence. Trends Cogn Sci 2005;9:69–74. https://doi.org/10.1016/j.tics.2004.12.005. [44] Reininghaus U, Kempton MJ, Valmaggia L, Craig TKJ, Garety P, Onyejiaka A, et al. Stress sensitivity, aberrant salience, and threat anticipation in early psychosis: an experience sampling study. Schizophr Bull 2016;42:712–22. https://doi.org/10.1093/ schbul/sbv190. [45] American Psychiatric Association. Diagnostic and statistical manual of mental disorders 4th edition, text revision (DSM-IV-TR). Washington DC: APA; 2000. [46] Senín-Calderón C, Rodríguez-Testal JF, Fernández-Jiménez E, Valdés-Diaz M, Benítez-Hernández MM, Fuentes-Márquez S, et al. Reliability and validity of the ref scale for referential thinking. Eur Psychiatry 2010; Suppl. 1:758. doi:https://doi. org/10.1016/S0924-9338(10)70755-0. [47] Brenner K, Schmitz N, Pawliuk N, Fathalli F, Joober R, Ciampi A, et al. Validation of the English and French versions of the Community Assessment of Psychic Experiences (CAPE) with a Montreal community sample. Schizophr Res 2007;95:86–95. https://doi.org/10.1016/j.schres.2007.06.017. [48] Fonseca-Pedrero E, Paino M, Lemos-Giráldez S, Muñiz J. Validación de la Escala para la Evaluación Comunitaria de las Experiencias Psíquicas-42 (CAPE-42) en universitarios y pacientes con psicosis [Validation of the Community Assessment Psychic Experiences -42 (CAPE-42) in Spanish college students and patients with psychosis]. Actas Españolas Psiquiatr 2012;40:169–76. [49] Cicero DC, Kerns JG, McCarthy DM. The Aberrant Salience Inventory: a new measure of psychosis proneness. Psychol Assess 2010;22:688–701. https://doi.org/10.1037/ a0019913. [50] Fernández-León S, Senín-Calderón C, Gutiérrez-López ML, Rodríguez-Testal JF. Spanish validation of the aberrant salience inventory in a general adolescent population. Psicothema 2019;31:210–7. [51] Goldberg D, Hillier VF. A scaled version of the General Health Questionnaire. Psychol Med 1979;9:139–45. [52] Lobo A, Perez-Echeverria MJ, Artal J. Validity of the scaled version of the general health questionnaire (GHQ-28) in a Spanish population. Psychol Med 1986;16: 135–40. 54 J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55 [53] Scheier MF, Carver CS. The self-consciousness scale: a revised version for use with general Populations1. J Appl Soc Psychol 1985;15:687–99. https://doi.org/10.1111/ j.1559-1816.1985.tb02268.x. [54] Baños R, Belloch A, Perpiña C. Self-consciousness scale a study of Spanish housewives. Psychol Rep 1990;67:771–4. https://doi.org/10.2466/pr0.1990.66.3.771. [55] Baumgartner H, Homburg C. Applications of structural equation modeling in marketing and consumer research: a review. Int J Res Mark 1996;13:139–61. https:// doi.org/10.1016/0167-8116(95)00038-0. [56] Schermelleh-Engel K, Moosbrugger H, Müller H. Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods Psychol Res Online 2003;8:23–74. https://doi.org/10.1002/0470010940. [57] Byrne B. Structural equation modeling with Lisrel, Prelis, and Simplis. Basic concepts, applications, and programming. London: Lawrence Erlbaum; 1998. [58] Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct Equ Model 2007;14:464–504. https://doi.org/10.1080/10705510701301834. [59] Cohen J. Statistical power analysis for the behavioral sciences. vol. 2. Hillsdate, NJ: LEA; 1988. doi:10.1234/12345678. [60] Fusar-Poli P, Cappucciati M, Rutigliano G, Schultze-Lutter F, Bonoldi I, Borgwardt S, et al. At risk or not at risk? A meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction. World Psychiatry 2015;14:322–32. [61] Kaymaz N, Drukker M, Lieb R, Wittchen HU, Werbeloff N, Weiser M, et al. Do subthreshold psychotic experiences predict clinical outcomes in unselected non-help- seeking population-based samples? A systematic review and meta-analysis, enriched with new results. Psychol Med 2012;42:2239–53. https://doi.org/10. 1017/S0033291711002911. [62] Linscott RJ, van Os J. An updated and conservative systematic review and metaanalysis of epidemiological evidence on psychotic experiences in children and adults: on the pathway from proneness to persistence to dimensional expression across mental disorders. Psychol Med 2013;43:1133–49. https://doi.org/ 10.1017/S0033291712001626. [63] Henriksen MG, Parnas J. Delusional Mood. In: Stanghellini G, Raballo A, Broome M, Vincent Fernandez A, Fusar-poli S, Rosfort R, editors. Oxford Handb. Phenomenol. Psychopathol., vol. 1, Oxford handbooks online; 2018, p. 1–14. doi:10.1093/ oxfordhb/9780198803157.013.72. [64] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th edition (DSM-5). Arlington: VA; 2013. [65] Senín-Calderón C, Rodríguez-Testal JF, Perona-Garcelán S, Perpiñá C. Body image and adolescence: a behavioral impairment model. Psychiatry Res 2017;248:121–6. https://doi.org/10.1016/j.psychres.2016.12.003. [66] Startup M, Sakrouge R, Mason OJ. The criterion and discriminant validity of the Referential Thinking (REF) Scale. Psychol Assess 2010;22:65–9. https://doi.org/10. 1037/a0016584. [67] van Os J, Linscott RJ, Myin-Germeys I, Delespaul P, Krabbendam L. A systematic review and meta-analysis of the psychosis continuum: evidence for a psychosis proneness–persistence–impairment model of psychotic disorder. Psychol Med 2009;39: 179. https://doi.org/10.1017/S0033291708003814. [68] McGrath JJ, Saha S, Al-Hamzawi A, Andrade L, Benjet C, Bromet EJ, et al. The bidirectional associations between psychotic experiences and DSM-IV mental disorders. Am J Psychiatry 2016;173:997–1006. https://doi.org/10.1176/appi.ajp.2016.15101293. [69] Abe K, Suzuki T. Prevalence of some symptoms in adolescence and maturity: social phobias, anxiety symptoms, episodic illusions and idea of reference. Psychopathology 1986;19:200–5. https://doi.org/10.1159/000284448. [70] Wang Y, song Shi H, hua Liu W, jie Xie D, lei Geng F, Yan C, et al. Trajectories of schizotypy and their emotional and social functioning: an 18-month follow-up study. Schizophr Res 2018;193:384–90. https://doi.org/10.1016/j.schres.2017.07. 038. [71] Rodríguez-Testal J, Valdés-Díaz M, Benítez-Hernández, M Fuentes-Márquez, S Fernández-Jiménez E, Senín Calderón C. Stability and Reliability of the Assessment of Referential Thinking by the REF Scale. Wordl Psychiatry 2009;8:297. [72] Bellido-Zanin G, Perona-Garcelán S, Senín-Calderón C, López-Jiménez AMAM, Rodríguez-Testal JFJF. Relationship between inner dialog and ideas of reference and the mediating role of dissociation. Scand J Psychol 2017;58:100–6. https://doi. org/10.1111/sjop.12344. [73] Bellido-Zanin G, Perona-Garcelán S, Senín-Calderón C, López-Jiménez AMAM, Ruiz- Veguilla M, Rodríguez-Testal JFJF. Childhood memories of threatening experiences and submissiveness and its relationship to hallucination proneness and ideas of reference: the mediating role of dissociation. Scand J Psychol 2018;59:407–13. https:// doi.org/10.1111/sjop.12455. [74] Fusar-Poli P, Rutigliano G, Stahl D, Davies C, Bonoldi I, Reilly T, et al. Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis. JAMA Psychiat 2017;74:493. https://doi.org/10.1001/jamapsychiatry. 2017.0284. [75] Preti A, Cella M, Raballo A, Vellante M. Psychotic-like or unusual subjective experiences? The role of certainty in the appraisal of the subclinical psychotic phenotype. Psychiatry Res 2012;200:669–73. https://doi.org/10.1016/j.psychres.2012.07.014. [76] Jones HJ, Stergiakouli E, Tansey KE, Hubbard L, Heron J, Cannon M, et al. Phenotypic manifestation of genetic risk for schizophrenia during adolescence in the general population. JAMA Psychiat 2016;73:221–8. doi:10.1001/jamapsychiatry.2015.3058. 55J.F. Rodríguez-Testal et al. / Comprehensive Psychiatry 93 (2019) 48–55