• Clinical Group Differences
  • ADHD and General Population Comparisons
  • ADHD, Depression/Anxiety, and General Population Comparisons
• Classification Accuracy

Validity refers to the accuracy of measurement of the intended construct, or the degree to which evidence supports the interpretation of test scores for an intended use (AERA, APA, & NCME, 2014). Multiple sources of validity are considered when designing and evaluating a test. For test scores from the CAARS 2–Short, validity evidence based on relationships between tests scores and criterion variables was investigated (see Analyses and Results earlier in this chapter for details regarding the investigation of factor structure and correlations between the full-length and shortened forms for additional validity evidence). Please see chapter 9, Validity, for detailed descriptions of each type of analysis discussed in this section.

Clinical Group Differences

To provide evidence of the criterion-related validity of the scores from the CAARS 2–Short, mean test score differences were compared within the following groups:

1. Individuals diagnosed with ADHD Inattentive Presentation (ADHD Inattentive), ADHD Combined Presentation (ADHD Combined), and individuals from the general population.
   
   
2. Individuals diagnosed with ADHD (ADHD Inattentive, and ADHD Combined), individuals with Depression or Anxiety (diagnostic groups include Major Depressive Disorder, Persistent Depressive Disorder, Generalized Anxiety Disorder, Panic Disorder, Separation Anxiety, and Social Anxiety Disorder), and individuals from the general population.
   

All analyses were conducted via a series of analysis of variance tests (ANOVA; conducted in R via the stats package, version 3.6.1; R Core Team, 2013). Given the large number of comparisons conducted, a conservative significance level (p < .01) was adopted to determine statistical significance. Effect sizes, as measured by eta-squared (η²) and by Cohen’s d, are provided for all analyses.

ADHD and General Population Comparisons

To provide additional evidence of the criterion-related validity of the scores from the CAARS 2–Short, mean score differences between individuals from the general population and those with ADHD (including those diagnosed with ADHD Inattentive or ADHD Combined) were compared across CAARS 2–Short Content Scale scores. ADHD Predominantly Hyperactive/Impulsive Presentation was excluded from the analyses, as the sample size was too small to provide stable estimates (consistent with population estimates for this presentation type in adults; APA, 2013). To facilitate comparisons between these groups, a subsample of the general population sample was selected to match the demographics of the ADHD sample (i.e., unifying ADHD Inattentive and ADHD Combined). The demographic characteristics of the General Population and ADHD groups and their raters are presented in appendix J.

Individuals in the ADHD Inattentive and ADHD Combined groups were expected, on average, to have higher scores on all CAARS 2–Short Content Scales than individuals in the General Population group, as the CAARS 2–Short is designed to capture symptoms related to adult ADHD. Individuals in the ADHD Combined group were expected to score higher on Hyperactivity, Impulsivity, and Emotional Dysregulation than individuals in the ADHD Inattentive group, as these scales capture features of ADHD that are not as prominent for the Inattentive Presentation. Comparisons between the General Population and ADHD groups were analyzed with a series of ANOVAs, and significant omnibus F-tests were followed up with Tukey’s honestly significant difference (HSD) post-hoc tests for pairwise comparisons. Results are displayed in Tables 11.19 to 11.20 and also depicted graphically in Figures 11.4 to 11.5.

As expected, significant differences were observed for all scale-level comparisons between the General Population and ADHD groups, for both the CAARS 2 Self-Report and Observer. The size of these differences, as measured by eta-squared (η²; see Cohen, 1973, for detailed interpretation; guidelines are provided as notes below each table), were large (Self-Report η² = .26 to .61 and Observer η² = .14 to .43). Across both rater types, the largest effects were observed on the Inattention/Executive Dysfunction scale.

Pairwise comparisons between the individual ADHD groups and the General Population group yielded moderate to large effect sizes, as measured by Cohen’s d. Specifically, large effect sizes were found when looking at comparisons between the ADHD Inattentive and General Population groups for Self-Report (median Cohen’s d = 1.10), and moderate to large effects were found for Observer (median Cohen’s d = 0.85). Similarly, large effect sizes were found when comparing the ADHD Combined and General Population groups (median Cohen’s d = 2.47 for Self-Report and 1.17 for Observer). These results provide strong evidence to support the validity of the CAARS 2–Short, as they demonstrate that the CAARS 2–Short Content scale scores successfully distinguish the profiles of individuals with and without ADHD.

Furthermore, significant differences between presentations of ADHD (i.e., Inattentive vs. Combined Presentations) were also observed for the following Self-Report scales: Hyperactivity, Impulsivity, and Emotional Dysregulation, with higher scores observed for ADHD Combined than for ADHD Inattentive in all instances (see Figure 11.4). A similar pattern of results was noted on the Observer, although the results were not statistically significant (see Figure 11.5). These findings were as expected, given the presence of more hyperactive and impulsive symptoms among those diagnosed with ADHD Combined than those diagnosed with ADHD Inattentive. These clear, statistically significant, and expected group differences provide strong evidence for the validity of the CAARS 2–Short scores included in these analyses.

Click to expand

Figure 11.4. Profiles for General Population and ADHD Groups: CAARS 2–Short Self-Report

Click to expand

Figure 11.5. Profiles for General Population and ADHD Groups: CAARS–Short 2 Observer

ADHD, Depression/Anxiety, and General Population Comparisons

Additional group comparisons investigated differences between the scores of individuals with ADHD (all presentations combined into a single group; Self-Report N = 122; Observer N = 79), individuals with Depression or Anxiety (Self-Report N = 123; Observer N = 98), and a subset of the General Population selected to correspond demographically to the combination of clinical cases in the other two groups (Self-Report N = 245; Observer N = 177). The Depression/Anxiety sample included individuals with a confirmed diagnosis of one or more of Major Depressive Episode, Major Depressive Disorder, Persistent Depressive Disorder, Generalized Anxiety Disorder, Separation Anxiety Disorder, Social Anxiety Disorder, or Panic Disorder. For these analyses, individuals were excluded from the Depression/Anxiety sample if they had a co-occurring diagnosis of ADHD and, similarly, from the ADHD sample if they had any of the aforementioned depressive or anxiety disorders. For the General Population sample, a subsample of the full General Population sample was selected that corresponded to the combined ADHD and Depression/Anxiety samples in terms of gender, age group, race/ethnicity, and education level. The demographic characteristics of the ADHD, Depression/Anxiety, and General Population groups and their raters are presented in appendix J.

Individuals with Depression and/or Anxiety were expected to score higher than the General Population but lower than those with ADHD on most CAARS 2–Short Content Scales. Specifically, individuals with Depression and/or Anxiety were anticipated to have elevated scale scores where there is symptom overlap between these internalizing disorders and ADHD (i.e., Inattention/Executive Dysfunction, Negative Self-Concept, and Emotional Dysregulation). However, individuals with ADHD were expected to score significantly higher than individuals with Depression and Anxiety on scales that reflect features more unique to ADHD (i.e., Hyperactivity and Impulsivity).

The results of ANOVAs conducted to explore the differences between the ADHD, Depression/Anxiety, and correspondingly sized General Population groups are presented in Tables 11.21 and 11.22. Significant differences were observed between the groups for all scales (p < .001), with effect sizes that ranged from moderate to very large (η² ranged from .19 to .48 for Self-Report, and .11 to .38 for Observer). Sizeable differences were observed for nearly all pairwise comparisons, highlighting the clearly distinct profiles of scores across CAARS 2–Short Content Scales for these three groups (see Figure 11.6 and Figure 11.7 for a graphical depiction of mean scale scores by group for Self-Report and Observer, respectively).

The ADHD group scored significantly higher than the General Population across all scales for Self-Report and Observer, with large effect sizes for all contrasts (median Cohen’s d = 1.41 for Self-Report, 1.08 for Observer). The Depression/Anxiety group also scored significantly higher than the General Population sample for most scales (median Cohen’s d = 0.72 for Self-Report, 0.55 for Observer). Moreover, differences between the ADHD group and Depression/Anxiety group revealed moderate to large effect sizes for the Inattention/Executive Dysfunction, Hyperactivity, and Impulsivity scales (median Cohen’s d = 1.05 for Self-Report, 0.92 for Observer), while there were no significant differences between the groups for Emotional Dysregulation (Cohen’s d = 0.20 for Self-Report and 0.25 for Observer) and Negative Self-Concept (Cohen’s d = -0.14 for Self-Report and -0.20 for Observer). The size and direction of the effects matched expectations. Overall, for scales containing features and symptoms of ADHD that do not typically overlap with those of internalizing disorders (e.g., hyperactivity, impulsivity), the differences were significant and moderate to large in size, whereas for scales such as Emotional Dysregulation and Negative Self-Concept that capture features common to both ADHD and internalizing disorders, the effects were not significant and were negligible to small in size. These results provide strong evidence for the validity of these CAARS 2–Short scores, as there were clear differences between groups that were hypothesized to score differently, resulting in clearly distinct profiles of scores.

Click to expand

Figure 11.6. Profiles of ADHD, Depression/Anxiety, and General Population Groups: CAARS 2–Short Self-Report

Note. Dep/Anx = individuals with a diagnosis of a Depressive Disorder and/or Anxiety Disorder.

Click to expand

Figure 11.7. Profiles of ADHD, Depression/Anxiety, and General Population Groups: CAARS 2–Short Observer

Note. Dep/Anx = individuals with a diagnosis of a Depressive Disorder and/or Anxiety Disorder.

Classification Accuracy

Classification accuracy statistics were derived for the CAARS 2–Short to consider the extent to which scores from each form correctly classify individuals into their respective groups (i.e., General Population vs. ADHD). For an operational definition of classification accuracy and detailed description of methods used, see chapter 9, Validity. For a description of the classification accuracy statistics that were examined and how each is derived, see chapter 6, Development.

For the CAARS 2–Short, the binary classification modelling consisted of two sets of analyses: (a) binomial logistic regression, followed by (b) the creation of confusion matrices and the derivation of classification accuracy statistics. The logistic regressions were used to predict how well the CAARS 2–Short Content Scales’ T-scores can identify individuals from the General Population versus those diagnosed with ADHD (all presentations).

To facilitate unbiased predictions, matched samples were created to ensure that each of the General Population and ADHD groups used for binary classification modelling were nearly equivalent in terms of their demographic composition. The matched General Population and ADHD samples used to examine the classification accuracy of the CAARS 2–Short were the same as those used to examine the classification accuracy of the full-length forms. The demographic characteristics of each sample are presented in appendix J.

Predicting diagnostic status depends on the prevalence of ADHD in the population. The prevalence (or base rate of ADHD in a relevant population) can vary widely depending on the purpose of the evaluation and the setting. For example, in a screening setting you might expect the prevalence of ADHD to be around 10% or less; whereas in a clinically referred sample a prevalence of approximately 50% may be more likely, or 60%-80% in an ADHD-specific clinical practice. Accordingly, the classification accuracy statistics of the CAARS 2–Short scales, assuming a 50% base rate, are summarized in Table 11.23, and the Positive and Negative Predictive Values based on varying base rates are provided in Table 11.24. The overall correct classification rate was high (89.7% for Self-Report and 84.1% for Observer), with a desired balance between sensitivity and specificity. These results support the use of the CAARS 2–Short for effective classification of individuals with and without ADHD.