• Samples
• Analyses and Results

The goal of the CAARS 2–Short is to assess core and associated symptoms of ADHD. Therefore, all five CAARS 2 Content Scales were included–Inattention/Executive Dysfunction, Hyperactivity, Impulsivity, Emotional Dysregulation, and Negative Self-Concept. Note that the CAARS 2–Short also includes the Response Style Analysis, ADHD Index, and Additional Questions; these components are parallel to the full-length form and are not analyzed or discussed separately in this chapter. Please see Table 1.1 in chapter 1, Introduction, for a comparison of the content included in the full-length CAARS 2 and CAARS 2–Short. In order to mitigate the risks to measurement precision, reliability, and validity that can occur with abbreviated versions of scales, recommended practices for developing short forms were followed for the CAARS 2–Short (Emons et al., 2007; Kruyen et al., 2013; Smith et al., 2000; Ziegler et al., 2014).

Samples

The CAARS 2–Short was derived and validated using the CAARS 2 Total Sample (see Table 6.4 in chapter 6, Development). The Total Sample, which included individuals from the general population and from clinical groups, comprised a sample of 2,232 individuals aged 18 or older who completed the CAARS 2 Self-Report and 2,150 observers who rated adults aged 18 or older. The Total Sample was used to select and validate the items for the shortened Content Scales. Note that two individuals from the Observer Total Sample were excluded from analyses as they had omitted items that affected analyses relevant to the creation of a shortened form. The samples were split into calibration and validation subsamples (see Tables 11.1a and 11.1b for demographic characteristics of the rated individuals and the raters, respectively).

Analyses and Results

Identical procedures were used to develop the CAARS 2–Short for the Self-Report and Observer forms. Consistent with recommended practice in developing shortened forms, both statistical methods and expert judgment were employed to ensure breadth of coverage of the target construct was retained in the shortened forms (Kruyen et al., 2013; Smith et al., 2000; Ziegler et al., 2014). The steps involved in item selection and subsequent validation of the shortened forms were as follows:

Step 1–Core items selected. Five experts in adult ADHD (see Acknowledgements) were asked to identify items from the full-length CAARS 2 that best represented the core construct for each scale. Experts were asked to identify core items for Self-Report and Observer separately. The number of experts who endorsed each item as core was summed, producing a score that ranged from 0 to 5 for each item. Expert consensus on a core item was defined as an item score of 4 or higher, which represented agreement from at least 4 out of 5 experts. All core items were initially included in the shortened scales, though a small subset of core items were later excluded due to statistical considerations as outlined in Step 2.

Step 2–Items excluded due to statistical considerations. Items were examined for local dependence (LD) as well as differential item functioning (DIF) across demographic groups (i.e., gender, race/ethnicity, language(s)¹ spoken, and education level [EL]) in the Total Sample. LD refers to the assumption of an IRT model that the items in a scale share variance only due to a common factor and are not related to one another in other ways (e.g., a response to one question depends on a response from an earlier question; Embretson & Reise, 2000). DIF refers to the assumption of an IRT model that there is no statistical item bias in terms of group differences (Embreton & Reise, 2000). If there was evidence for significant and meaningful LD or DIF, the item was excluded from consideration on the CAARS 2–Short. Including the items could mean the scale score would be affected by factors other than the construct being measured (that is, significant DIF would indicate item responses are unduly influenced by group characteristics, while meaningful LD would suggest that responses are influenced by item similarity such that the items may be related for a reason beyond their shared latent construct). While LD and DIF had negligible impact on the full-length CAARS 2 (see chapter 6, Development, for item selection procedures that evaluated these same statistics and found little evidence for the meaningful influence of either statistic in the full-length CAARS 2 items), LD and DIF can have a larger influence on shorter scales and therefore more stringent criteria were set for the CAARS 2–Short.

Items with a medium DIF effect size in terms of the tested demographic groups were excluded from consideration for the short form. Using this criterion, no items were excluded from the Self-Report form and only one item was excluded from the Observer form (as there was a moderate effect size between Hispanic and White individuals for Observer).

LD was assessed using (a) residual correlations among items greater than .15, (b) modification indices for 1-factor confirmatory factor analysis (CFA) models for each of the scales to assess residual correlation pairs, and (c) the presence of a significant χ² test (Chen & Thissen, 1997). When LD was detected for an item pair, the item with the better measurement properties overall was considered for inclusion on the shortened forms.

Step 3–Remaining items selected. Item selection was done using the calibration sample, by systematically adding items one at a time to the core set of items for each scale, based on the following considerations:

• Statistical Properties. Additional items were added based on item discrimination, precision of measurement, and ability to discriminate between the General Population and ADHD samples.
  
  
  • Item discrimination assesses an item’s ability to distinguish individuals at low versus high levels of the trait. Item discrimination was measured using the slope parameter of each item from an IRT model. Higher values (e.g., > .75) were favored, as they indicate better discrimination (Embretson & Reise, 2000).
    
    
  • Precision of measurement is inversely related to the amount of error, so that an item with low error has high precision. Precision of measurement for items was assessed using item information curves (IICs). An IIC graphically shows precision of measurement across the range of the construct being measured, also known as theta. Precision at or above 1.5 SD from the average level of the construct was targeted, to best capture both subclinical and clinical levels of the construct. Greater amounts of information indicate higher precision of measurement and lower standard error (more details can be found in Test Information in chapter 8, Reliability).
    
    
  • Cliff’s delta (Cliff’s d; Cliff, 1993) was employed to examine how well each item distinguished between the General Population and ADHD samples. Cliff’s d is a measure of effect size used for non-parametric data. Items with higher effect sizes were preferred as they indicate better discrimination between groups.
    
    
• Expert ratings. When items had similar statistics, the item with the higher expert rating was retained
  
  
• Content represented. Many scales assess different content areas or facets within the construct. For example, for the full-length CAARS 2 Hyperactivity scale, 61% of the items assess behavioral aspects of hyperactivity, 31% assess verbal hyperactivity, and 8% assess both behavioral and verbal aspects. A similar ratio of items was retained for the shortened form, and across both raters, to ensure proportional coverage of all facets of the construct measured.
  
  

Note that the CAARS 2–Short Self-Report and Observer were developed separately; while they both cover the same core content areas, they differ at the item-level. Experts identified different items as core for the different rater types, and statistical analysis dictated empirical selection of certain items for the Self-Report and other items for the Observer form. As a result, the CAARS 2–Short Self-Report and Observer items are overlapping, but not completely aligned.

Step 4–Alternate shortened versions compared. The development team set a minimum and maximum length for each scale on the short form (see Table 11.2). The minimum was the fewest number of items that would still allow for reasonable breadth of coverage; the maximum was approximately two-thirds of the full-length scale. For example, the Inattention/Executive Dysfunction is the longest scale with the most content to cover. Therefore, it required more items than other scales (as seen in Table 11.3). Starting with the minimum length, alternate-length short forms were created sequentially by adding one item at a time; therefore, for example, a 7-item version was compared to an 8-item version, which only differed by one additional item. This approach enabled testing for the ideal length that balanced efficiency with reliability and validity (Smith et al., 2000).

The following criteria were used to assess reliability and validity:

• Measurement precision of the scale, with an emphasis on peak precision at 1.5 or 2 standard deviations above the mean for a given construct. Ensuring precision at this range was the focus, as that is typically understood to capture the clinical range of the constructs measured (see also Test Information in chapter 8, Reliability). Information values greater than 10 indicate high precision, values below 10 are moderately precise, and values near 5 are considered adequate (Flannery et al., 1995; Reeve & Fayers, 2005).
  
  
• Goodness-of-fit statistics were explored to ensure consistency in the factor structure between shortened and full-length scales. This comparison is helpful for ensuring that construct validity is retained (Rammstedt & Beierlein, 2014) and that all dimensions of the construct are proportionally represented in the short form (Maloney et al., 2011). A detailed discussion of the multiple fit indices considered is provided in Internal Structure in chapter 9, Validity.
  
  
• Internal consistency, as measured by alpha and omega, was evaluated (see Internal Consistency in chapter 8, Reliability, for a detailed discussion of these metrics).
  
  
• Correlations between raw scores on the shortened scales and the full-length scales were assessed (via Kendall’s tau coefficient, given the non-normality of the distribution of the scales). High correlation coefficients provide evidence that the scales are measuring the same construct. Reliability, validity, and construct coverage were prioritized over correlation between form lengths.
  
  

The statistical properties for each of the alternate versions were evaluated, and results for each were compared against the full-length CAARS 2 as a reference point. In instances where a shorter version performed as well statistically as a version with more items, the version that included the fewest items was favored. The process is illustrated with the CAARS 2–Short Observer Impulsivity scale as an example. As seen in Table 11.3, 4-, 5-, 6-, and 7-item versions of this scale were compared, and the analyses revealed acceptable and similar results for all versions in terms of correlations to the full-length scale, internal consistency, and model fit. However, compared to the other versions, the 6-item version had slightly less desirable fit statistics (higher RMSEA and SRMR and lower CFI and TLI), and the 4-item version had slightly lower internal consistency estimates. The precision of measurement, as seen in Figure 11.1, showed that the 7-item version was the only one to surpass a value of 10. Based on these results, the 7-item version was selected for the CAARS 2–Short Impulsivity scale. This process of comparing various scale lengths for each scale on the CAARS 2–Short was repeated until a final set of items was selected for all scales.

Click to expand

Figure 11.1. Comparison of Short Form Options: CAARS 2–Short Observer Impulsivity Scale

Step 5–Final short form tested. The final set of items selected for the CAARS 2–Short Content Scales included 37 items each for Self-Report and Observer. Once the final versions for each scale were selected, items were recalibrated using IRT analyses for both the calibration and validation samples. The items selected for the CAARS 2–Short Content Scales, along with the slope (a) and location (b) parameters of the recalibration, can be found in Tables 11.4 and 11.5. Overall, the CAARS 2–Short demonstrated strong item discrimination, with a minimum slope greater than 1.0 for all samples tested. These results suggest that the selected items distinguish well between low and high levels of the construct being measured by each scale.

The same criteria used in selecting the items for the CAARS 2–Short scales was used to evaluate the shortened forms with the validation sample. Correlations were computed with Kendall’s tau to evaluate the relationship between the full-length and shortened forms. The full-length CAARS 2 and CAARS 2–Short showed very strong, positive, and statistically significant (p < .001) correlations for all scales, ranging from .83 to .93 across forms (see Table 11.6).

Estimates of internal consistency for the CAARS 2–Short scale scores within the validation sample all demonstrated high reliability, with alpha and omega values at or above .85 for Self-Report and Observer (see Table 11.7). The maximum decrease in internal consistency from the CAARS 2 to the CAARS 2–Short was .05 (see Table 11.7), indicating minimal compromises when using the shortened version.

Test information was also explored in the validation samples for Self-Report and Observer, as seen in Figure 11.2. Nearly all scales on both forms had test information values above 10 at 2 SD above the mean; for Self-Report, Impulsivity showed moderate test information, with a peak value greater than 5. These results are similar to the full-length CAARS 2 (see Test Information in chapter 8, Reliability). The test information of the CAARS 2–Short shows minimal loss in reliability compared to the full-length CAARS 2, providing strong evidence for the validity of the shortened scales.

Click to expand

Figure 11.2. Test Information for CAARS 2–Short: Validation Sample

a) Inattention/Executive Dysfunction

b) Hyperactivity

c) Impulsivity

d) Emotional Dysregulation

e) Negative Self-Concept

The ability of the shortened scales to distinguish between the General Population and individuals diagnosed with ADHD (Predominantly Inattentive or Combined Presentation) was explored in the validation sample. Results, as measured by Cliff’s d effect sizes of group differences, are presented in Table 11.8, and show that the full-length CAARS 2 and the CAARS 2–Short are comparable with respect to how well they differentiate between General Population and ADHD groups. For both Self-Report and Observer, effect sizes are only marginally different between the two versions, and the overlapping confidence intervals indicate that differences between the form lengths are not significant. Replicating the discriminating ability of the CAARS 2 scales with the CAARS 2–Short scales provides additional evidence that the selected items for the CAARS 2–Short perform well.

Mirroring what was tested in the full-length CAARS 2 (see Internal Structure in chapter 9, Validity), a CFA was used to determine whether the structure of the full-length CAARS 2 was retained in the CAARS 2–Short. Results from the CFA, presented in Table 11.9, indicate that the 5-factor model is an excellent fit for both the full-length and shortened forms of the CAARS 2. The fit statistics for the full-length and shortened versions support the replicated factor structure across the two lengths.

The development phase goal was to create a shortened version of the CAARS 2 that measured symptoms associated with ADHD efficiently and with minimal reduction of empirical psychometric properties. Overall, the results from this validation sample demonstrated that the CAARS 2–Short has psychometric properties that are comparable to the full-length CAARS 2, in terms of correspondence, internal consistency, test information, ability to distinguish between General Population and ADHD groups, and internal structure.