Conners 4 Manual Chapter 11: Development

Development

view all chapter tables | print this section

• Samples
• Analyses & Results

The goal of the Conners 4–Short is to assess the core and associated symptoms of, and impairment related to, ADHD. Therefore, four of the six Content Scales were included—Inattention/Executive Dysfunction, Hyperactivity, Impulsivity, and Emotional Dysregulation—as well as all of the Impairment & Functional Outcome Scales: Schoolwork, Peer Interactions, and Family Life (on Parent and Self-Report only). Please see Table 1.1 in chapter 1, Introduction, for a comparison of the content included in the full-length Conners 4 and Conners 4–Short; note that scales related to co-occurring symptoms were not included, specifically the Depressed Mood, Anxious Thoughts, DSM Oppositional Defiant Disorder Symptoms, and DSM Conduct Disorder Symptoms scales, as these scales extend beyond the goal of a brief and focused shortened form. To mitigate the risks to measurement precision, reliability, and validity that can occur with short forms, recommended practices in developing short forms were followed for these scales (Emons et al., 2007; Kruyen et al., 2013; Smith et al., 2000; Ziegler et al., 2014).

Samples

The Conners 4–Short was derived and validated using the Total Samples (see Tables 6.5 and 6.6 in chapter 6, Development). The Total Samples included youth from the general population and clinical diagnostic groups, yielding a sample of 3,251 parents and 2,877 teachers who rated youth aged 6 to 18 years, as well as 1,586 youth aged 8 to 18 who provided self-report ratings. The Total Samples were used to select and validate the items for the shortened Content Scales. Note that six youth from the Parent Total Sample, four from the Teacher Total Sample, and three from the Self-Report Total Sample were excluded from analyses as they had omitted items that affected analyses relevant to the creation of a shortened form. The samples were split into calibration and validation subsamples (see Tables 11.1 [a to c] and 11.2 [a to b] for demographic characteristics of the rated youth and the raters, respectively).

Analyses & Results

Identical procedures were used to develop the Conners 4–Short for the Parent, Teacher, and Self-Report (Clark et al., 2020). Consistent with recommended practice for developing shortened forms, both statistical methods and expert judgment were employed to ensure breadth of coverage of the construct was retained in the shortened forms (Kruyen et al., 2013; Smith et al., 2000; Ziegler et al., 2014).

The item selection and subsequent validation of the shortened forms were carried out according to the following steps:

Step 1—Core items selected. Five external experts in child and adolescent ADHD (see Acknowledgements) were asked to identify items from the full-length Conners 4 that best represent the core construct for each scale. Experts were asked to identify core items for Parent, Teacher, and Self separately. The number of experts who endorsed each item as core was summed, producing a score that ranged from 0 to 5. Expert consensus on a core item was defined as receiving a minimum score of 4, which represents agreement from at least 4 out of the 5 experts. All core items were initially included in the shortened scales, though a small subset of core items were later excluded due to statistical considerations outlined in Step 2.

Step 2—Items excluded due to statistical considerations. Items were examined for local dependence (LD), as well as differential item functioning (DIF) across demographic groups (i.e., gender, race/ethnicity, and PEL) in the Total Sample. If there was evidence for significant and meaningful DIF or LD, the item was excluded from consideration on the Conners 4–Short. Including the items could mean the scale score would be affected by factors other than the construct being measured (that is, significant DIF would indicate item responses are unduly influenced by group characteristics, while meaningful LD would suggest that responses are influenced by item similarity such that the items may be related for a reason beyond their shared latent construct). While DIF and LD had negligible impact on the full-length Conners 4 (see chapter 6, Development, for item selection procedures that evaluated these same statistics and found little evidence for the meaningful influence of either statistic in the full-length Conners 4 items), DIF and LD can have a larger influence on shorter scales and therefore more stringent criteria were set for the Conners 4–Short.

Items with a medium DIF effect size in terms of the tested demographic groups were excluded from consideration for the short form. Using this criterion, no items were excluded from the Parent or Teacher report, and only one item was excluded from the Self-Report (“It’s hard for me to get back to work after I’ve been interrupted”) as there was a moderate effect size difference between Black and White youth.

LD was assessed using (a) residual correlations among items greater than .15, (b) modification indices for 1-factor confirmatory factor analysis (CFA) models for each of the scales to assess residual correlation pairs, and (c) the presence of a significant χ2 test (Chen & Thissen, 1997). When LD was detected for an item pair, the item with the better measurement properties overall was considered for inclusion on the shortened forms.

Step 3—Remaining items selected. Item selection was done using the calibration sample by systematically adding items, one at a time, to the core set of items for each scale based on the following considerations:

• Statistical Properties. Additional items were added based on item discrimination, precision of measurement, and the ability to discriminate between the General Population and ADHD samples.

  • Item discrimination assesses an item’s ability to distinguish youth at low versus high levels of the trait. Item discrimination was measured using the slope parameter of each item from an IRT model. Higher values (e.g., > .75) were favored, as they indicate higher discrimination (Embretson & Reise, 2000).

  • Precision of measurement is inversely related to the amount of error, so that an item with low measurement error has high precision (more details can be found in Test Information in chapter 8, Reliability). Precision of measurement for items was assessed using item information curves (IICs). An IIC graphically shows precision of measurement across the range of theta. Precision at or above 1.5 standard deviations (SD) from the average level of the trait was targeted, to best capture both subclinical and clinical levels of the trait.

  • Cliff’s delta (Cliff’s d; Romano et al., 2006) was employed to examine how well each item distinguished between the General Population and the ADHD samples. Cliff’s d is a measure of effect size used for non-parametric data. Items with higher effect sizes were preferred, as they indicate better discrimination between groups.

• Expert ratings. When items had similar statistics, the item with the higher expert rating was retained.

• Content represented. Many scales assess different content areas or facets within the construct. For example, for the Conners 4 Hyperactivity scale, 64% of the items assess behavioral aspects of hyperactivity, about 9% assess verbal hyperactivity, and about 27% assess both behavioral and verbal aspects. A similar ratio of items was retained for the shortened form, and across all raters, to ensure proportional coverage of all facets of the construct measured.

Step 4—Alternate shortened versions compared. The development team set a minimum and maximum length for each scale on the short form (see Table 11.3). The minimum was the fewest number of items that would allow for reasonable breadth of coverage; the maximum was approximately two-thirds of the full-length scale. For example, Inattention/Executive Dysfunction is the longest scale with the most content to cover. Therefore, it required more items than other scales (as seen in Table 11.3). Starting with the minimum length, alternate-length short forms were created sequentially by adding one item at a time; therefore, for example, a 6-item version and a 7-item version only differed by one additional item. This method allowed for the ability to test the ideal length that balanced efficiency with reliability and validity (Smith et al., 2000).

The following criteria were used to assess reliability and validity:

• Measurement precision of the scale focusing at and above 1.5 SD was assessed using test information curves (TICs). While 1.5 SDs above the mean typically reflects the lower boundary of elevated scores, priority was given to shortened versions of the scales that could provide information at or above 2 SDs above the mean, as that is more distinctly within a typical clinical range. A TIC is interpreted in the same manner as an IIC (described in Step 3), but it measures information for an entire test or scale, rather than at the item level. Information values greater than 10 indicate high precision, values below 10 are moderately precise, and values near 5 are considered adequate (Flannery et al., 1995; Reeve & Fayers, 2005).

• Goodness-of-fit statistics were explored to ensure consistency in the factor structure between shortened and full-length scales. This comparison is helpful for ensuring that construct validity is retained (Rammstedt & Beierlein, 2014) and that all dimensions of the construct are proportionally represented in the short form (Maloney et al., 2011). Multiple fit statistics were considered; a more detailed discussion of each of these fit indices is provided in Internal Structure in chapter 9, Validity.

• Internal consistency, as measured by alpha and omega, was evaluated (see Internal Consistency in chapter 8, Reliability, for a detailed discussion of these metrics). Higher values were favored, and no values were so high that they indicated item possible redundancy (i.e., above .95).

• Correlations (as measured via Kendall’s tau coefficient, given the non-normality of the distribution of the scales) between raw scores on the shortened scales and the full-length scales were assessed. High correlation coefficients provide evidence that the scales are measuring the same construct. Reliability, validity, and construct coverage were prioritized over correlation.

The statistical properties for each of the versions were evaluated, and results were compared against the full-length Conners 4 as a reference point. In instances where a shorter version performed as well statistically as a version with more items, the version that included the fewest items was favored. The process is illustrated with the Conners 4–Short Parent Family Life scale as an example. As seen in Table 11.4, 4- and 5-item versions of this scale were compared, and the analyses revealed acceptable and similar results for both versions in terms of correlations to the full-length scale, internal consistency, and model fit. In addition, Table 11.5 displays mean differences between the General Population and each ADHD presentation, in which all groups show similarly sized mean differences for all scale lengths (i.e., no compromise is made when shortening the scale). The 4-item version showed good statistics across all criteria. The precision of measurement, as seen in Figure 11.1, showed that both versions surpassed a value of 10. Based on these results, the 4-item version was selected for the Conners 4–Short Parent Family Life scale. This process of comparing various scale lengths for each scale on the Conners 4–Short was repeated until a final set of items was selected for all scales.

Figure 11.1. Comparison of Short Form Options: Conners 4 Parent Family Life

Step 5—Final short form tested. The final set of items selected for the Conners 4–Short Content Scales and Impairment & Functional Outcome Scales included 32 items for Teacher and 36 items for Parent and Self-Report. Once the final versions for each scale were selected, items were recalibrated using IRT analyses for both the calibration and validation samples. The items selected for the Conners 4–Short scales, along with their slope (a parameter) and location parameters (b parameters) of the recalibration, can be found in Tables 11.6 to 11.8. The Conners 4–Short demonstrated strong item discrimination, with a minimum slope greater than 1.0 for all except one item on Self-Report, suggesting the selected items discriminate very well between low and high levels of the construct being measured by each scale.

The same criteria used in selecting the items for the Conners 4–Short scales was used to evaluate the shortened forms with the validation sample. Correlations were computed with Kendall’s tau to evaluate the relationship between the full-length and shortened forms. The full-length Conners 4 and Conners 4–Short showed very strong, positive, and statistically significant (p < .001) correlations for all scales, ranging from .77 to .90, across all forms (see Table 11.9).

Estimates of internal consistency for the Conners 4–Short scale scores within the validation sample all demonstrated high reliability, with alpha and omega values at or above .85 on the Parent and Teacher forms (see Table 11.10). Values are a bit lower on the Self-Report, but only drop by .01 from the full-length Conners 4 (see Table 11.10), indicating minimal compromises in reliability when using the shortened version.

Test information was also explored in the validation samples across all forms, as seen in Figure 11.2. Nearly all scales on the Parent and Teacher forms had test information values above 10 at 2 SD above the mean; the Self-Report showed moderate test information, with values greater than 5. These results are similar to the full-length Conners 4, in which the Self-Report had lower test information (see Test Information in chapter 8, Reliability). The test information of the Conners 4–Short shows minimal loss in reliability compared to the full-length Conners 4 (or, in the instance of the Inattention/Executive Dysfunction scale, some loss occurred but the scale remains highly informative), providing strong evidence to validate the selected shortened scales.

Figure 11.2. Test Information Curves: Conners 4–Short (Validation Sample)

(a) Inattention/Executive Dysfunction	(b) Hyperactivity
(c) Impulsivity	(d) Emotional Dysregulation
(e) Schoolwork	(f) Peer Interactions
(g) Family Life

The ability of the shortened scales to distinguish between the General Population and youth diagnosed with ADHD was explored in the validation sample, with the inclusion all three presentations of ADHD: Predominantly Inattentive Presentation (ADHD Inattentive), Predominantly Hyperactive/Impulsive Presentation (ADHD Hyperactive/Impulsive), and Combined Presentation (ADHD Combined). Results, as measured by Cliff’s d effect sizes of group differences, are presented in Table 11.11 and show that the differentiation ability of the test is comparable between the full-length Conners 4 and the Conners 4–Short. Effect sizes are only marginally different between the two versions across Parent, Teacher, and Self-Report, and the overlapping confidence intervals indicate that differences between the form lengths are not significant. Replicating the discriminatory ability of the Conners 4 scales with the Conners 4–Short scales provides additional evidence that the selected items for the Conners 4–Short perform well.

Mirroring what was tested in the full-length Conners 4 (see Internal Structure in chapter 9, Validity), the Conners 4–Short scales were subjected to CFA using the validation sample to determine whether the structure of the full-length Conners 4 was retained in the Conners 4–Short. Goodness-of-fit statistics are used to evaluate the competing models (see Internal Structure in chapter 9, Validity, for interpretation guidelines). The shortened forms capture the symptoms and impairments associated with ADHD, but not associated and commonly co-occurring symptoms of depression and anxiety. When comparing the full-length Conners 4 and the Conners 4–Short Content Scales, only the scales included on the Conners 4–Short form were used. That is, the Anxious Thoughts and Depressed Mood scales were not included in the comparison between the full-length Conners 4 and Conners 4–Short, and such, a 4-factor, rather than 6-factor, model was tested for both test lengths for these analyses. This change in structure is because the goal of this analysis was to ensure that the loss of items does not impede the factor structure; comparing models with a different number of factors would confound this comparison. The fit statistics for the full-length and shortened versions support the replicated factor structure between the two lengths. The 4-factor correlated-factors model with robust estimation for ordinal items for the Content Scale items and a 3-factor (2-factor for Teacher) model for the Impairment & Functional Outcome Scale items showed good fit across all fit statistics.

The development phase goal was to create a shortened version of the Conners 4 that measured symptoms and impairments associated with ADHD efficiently and with minimal reduction of empirical psychometric properties. Overall, the results from this validation sample demonstrated similar properties as the full-length Conners 4, in terms of correspondence between the short and full versions, internal consistency, test information, ability to distinguish between General Population and ADHD groups, and internal structure. These results serve to address the stated goal and provide evidence for the successful creation of shortened forms.

< Back

Next >