Conners 4 Manual Chapter 11: Validity

Validity

view all chapter tables | print this section

• Clinical Group Differences
• ADHD and General Population Comparisons
• ADHD and Depression/Anxiety Comparisons
• Classification Accuracy

Validity refers to the accuracy of measurement of the intended construct, or the degree to which evidence supports the interpretation of test scores for an intended use (AERA, APA, & NCME, 2014). Multiple sources of validity are considered when designing and evaluating psychological tests. For the Conners 4–Short, validity evidence based on relations to criterion variables was investigated (see also Analyses and Results earlier in this chapter for details regarding the investigation of factor structure and correlations between the full-length and shortened forms for additional validity evidence). Please see chapter 9, Validity, for detailed descriptions of each type of validity analysis discussed in this section.

Clinical Group Differences

To provide evidence of the criterion-related validity of the scores from the Conners 4–Short, differences among the following groups were explored:

1. Youth diagnosed with ADHD Inattentive Presentation (ADHD Inattentive), ADHD Hyperactive/Impulsive Presentation (ADHD Hyperactive/Impulsive), ADHD Combined Presentation (ADHD Combined), and youth from the general population.

2. Youth diagnosed with ADHD (all presentations) and youth diagnosed with Depression or Anxiety (diagnostic groups include Major Depressive Disorder, Persistent Depressive Disorder, Generalized Anxiety Disorder, Panic Disorder, Separation Anxiety, and Social Anxiety Disorder).

All analyses were conducted via a series of analysis of variance tests (ANOVA; conducted in R via the stats package, version 3.6.1; R Core Team, 2013). Given the large number of comparisons conducted, a conservative significance level (p < .01) was adopted to determine statistical significance. Effect sizes as measured by eta-squared (η²) and by Cohen’s d are provided for all analyses.

ADHD and General Population Comparisons

Mean score differences between youth from the general population and those with ADHD (including those diagnosed with ADHD Inattentive, ADHD Hyperactive/Impulsive, and ADHD Combined) were compared across Conners 4–Short Content Scale scores and Impairment & Functional Outcome Scale scores. To facilitate comparisons between these groups, a subsample of the General Population group was created to match the demographics of the broader ADHD sample (i.e., unifying ADHD Inattentive, ADHD Hyperactive/Impulsive, and ADHD Combined). The demographic characteristics of the General Population and ADHD groups and their raters are presented in appendix F (see Tables F.12 to F.16).

It was expected that large differences between the ADHD groups and the General Population group would be observed across all scales on the Conners 4–Short. While all ADHD presentations were expected to display higher scores across scales than the General Population, it was also expected that when comparing scores between ADHD presentations, the ADHD Hyperactive/Impulsive and ADHD Combined groups would exhibit comparatively higher scores on scales indexing hyperactive and/or impulsive symptoms than the ADHD Inattentive group.

Comparisons between these four groups were analyzed with a series of ANOVAs, and significant omnibus F-tests were followed up with Tukey’s Honestly Significant Difference (HSD) post-hoc tests for pairwise comparisons. Results are displayed in Tables 11.37 to 11.39. Mean scale score profiles are also depicted graphically in Figures 11.4 to 11.6.

As expected, significant differences were observed for nearly all scale level comparisons across the Parent, Teacher, and Self-Report forms. The size of these differences varied, as measured by η² and Cohen’s d effect size values.

For the Conners 4–Short Content Scales, effect sizes for the omnibus F test were all large on the Conners 4–Short Parent (η² ranged from .27 to .52) and were medium to large for the Conners 4–Short Teacher (η² ranged from .09 to .22) and Conners 4–Short Self-Report (η² ranged from .11 to .25). Across all three rater types, the largest effects were observed on the Inattention/Executive Dysfunction scale. In looking at specific pairwise comparisons, results of the ADHD to General Population comparisons were explored first, followed by an exploration of the differences between the ADHD groups.

For the Conners 4–Short Parent, all three ADHD groups had significantly higher scores than the General Population group on all scales and effect sizes for all significant effects were large (Parent Cohen’s d ranged from 0.97 to 2.35). For the Conners 4–Short Teacher, the ADHD Hyperactive/Impulsive and ADHD Combined groups had significantly higher scores than the General Population group on all scales; however, the ADHD Inattention group only scored significantly higher than the General Population group on the Inattention/Executive Dysfunction Scale. All statistically significant effects were medium to large in size (Teacher Cohen’s d ranged from 0.68 to 1.13). For the Conners 4–Short Self-Report, all three ADHD groups scored significantly higher than the General Population group on all Content Scales, with medium to large effect sizes (Self-Report Cohen’s d ranged from 0.64 to 1.37).

Differences between presentations of ADHD (i.e., Inattentive vs. Hyperactive/Impulsive vs. Combined) were also observed. As expected, across all three forms, the ADHD Combined and ADHD Hyperactive/Impulsive groups did not differ significantly from one another. However, also as expected, the ADHD Combined and ADHD Hyperactive/Impulsive groups scored higher than the ADHD Inattentive group on the Hyperactivity and Impulsivity scales for both Parent and Teacher, with medium to large effect sizes for Parent and small to medium effect sizes for Teacher (Parent Cohen’s d ranged from 0.65 to 1.08; Teacher Cohen’s d ranged from 0.43 to 0.73). No such pattern was observed on the Conners 4–Short Self-Report. Overall, the Parent and Teacher findings were expected, given the likelihood that youth diagnosed with ADHD Hyperactive/Impulsive or Combined presentation would have more hyperactive and impulsive symptoms than those diagnosed with ADHD Inattentive. While the modest sample size for Self-Report may have precluded the observation of statistically significant differences between presentations, it is important to consider that small to medium effect sizes were observed on the Hyperactivity and Impulsivity scales (Cohen’s d = 0.60 and 0.36, respectively) when comparing the ADHD Inattentive and ADHD Hyperactive/Impulsive groups. These results provide strong evidence to support the validity of the Conners 4–Short scores, as clear differences between ADHD groups emerged and were in line with expectations.

Results for the shortened Impairment & Functional Outcome Scales revealed that all ANOVAs were statistically significant; effect sizes were large on the Conners 4–Short Parent (η² was .33 for each scale), medium to large for the Conners 4–Short Teacher (η² ranged from .13 to .16), and medium for the Conners 4–Short Self-Report (η² ranged from .11 to .13).

Specific pairwise comparisons for the shortened Impairment & Functional Outcome Scales revealed that for the Schoolwork scale, all ADHD groups had significantly higher scores than the General Population group for Parent, Teacher, and Self-Report, with medium to large effect sizes. (Parent Cohen’s d = 1.30 to 1.60, Teacher Cohen’s d = 0.82 to 1.01, Self-Report Cohen’s d = 0.71 to 0.78). Similarly, for the Peer Interactions Scale, all ADHD groups had higher scores than the General Population group for Parent, Teacher, and Self-Report, with small to large effect sizes (Parent Cohen’s d = 1.15 to 1.83, Teacher Cohen’s d = 0.46 to 1.06, Self-Report Cohen’s d = 0.64 to 0.78). Finally, for the Family Life scale, all ADHD groups had higher scores than the General Population group for Parent, with large effect sizes (Cohen’s d = 1.01 to 1.82). For the Self-Report, the ADHD Hyperactive/Impulsive and ADHD Combined groups both had significantly higher scores than the General Population group on the Family Life scale, with medium to large effect sizes for the significant effects (Cohen’s d = 0.79 and 0.86, respectively).

No significant differences between presentations of ADHD (i.e., Inattentive vs. Hyperactive/Impulsive vs. Combined) were observed on the Schoolwork scale for the three rater types. However, Parent raters indicated higher levels of impairment in Peer Interactions and Family Life for the ADHD Hyperactive/Impulsive and ADHD Combined groups than the ADHD Inattentive group, with small to medium effect sizes (d = 0.46 to 0.57). Teachers reported significantly more impairment in Peer Interactions for the ADHD Hyperactive/Impulsive group than the ADHD Inattentive group, with a medium effect size (d = 0.51). There were no significant differences between presentations of ADHD on the Self-Report Peer Interactions or Family Life scales for youth.

Taken together, these results are consistent with what was found on the full-length Conners 4 and provide strong evidence for the validity of the Conners 4–Short to not only distinguish between youth with ADHD and those from the general population, but also to distinguish between ADHD presentations. It is worth noting that the effect sizes for Conners 4–Short Parent exceed those for the Conners 4–Short Teacher and Self-Report (that is, there are larger differences between youth with and without ADHD when examining parent-reported ratings, as opposed to teacher or self-reported ratings). This trend is consistent with patterns from the results of different rater types presented in previous literature (e.g., Narad et al., 2015).

Figure 11.4. Profiles for General Population and ADHD Groups: Conners 4–Short Parent

Figure 11.5. Profiles for General Population and ADHD Groups: Conners 4–Short Teacher

Figure 11.6. Profiles for General Population and ADHD Groups: Conners 4–Short Self-Report

ADHD and Depression/Anxiety Comparisons

To consider how well the Conners 4–Short is able to facilitate a differential diagnosis or to distinguish youth with ADHD from youth with other clinical disorders, the Conners 4–Short scale scores were compared between youth with ADHD but no co-occurring depression (i.e., Major Depressive Disorder or Persistent Depressive Disorder) or anxiety (i.e., Generalized Anxiety Disorder, Panic Disorder, Social Anxiety, or Specific Phobias) and youth with either depression or anxiety but no co-occurring ADHD. To facilitate comparisons between the ADHD group and the Depression/Anxiety group, all available ADHD and Depression/Anxiety cases were included; however, the groups were not matched based on demographic characteristics, as matching would have resulted in too few cases to make meaningful comparisons between groups. The demographic characteristics of the ADHD and Depression/Anxiety groups are presented in appendix F (see Table F.23 and Table F.24).

It was expected that the ADHD group and the Depression/Anxiety groups would display distinct symptom profiles. For the Inattention/Executive Dysfunction scale, it was expected that both groups would exhibit high scores due to symptom overlap with Inattention/Executive Dysfunction (see chapter 9, Validity). However, because Inattention/Executive Dysfunction is a core feature of ADHD, it was expected that youth in the ADHD group would exhibit higher ratings than youth in the Depression/Anxiety group. The ADHD group was expected to score higher than the Depression/Anxiety group on the Conners 4 Content Scales that are highly specific to the symptoms of ADHD (e.g., Hyperactivity, Impulsivity), whereas the Depression/Anxiety group was expected to approach the scores of the ADHD group on scales where there is symptom overlap with ADHD (e.g., Emotional Dysregulation). In terms of the Impairment & Functional Outcome Scales, it was expected that the ADHD group and the Depression/Anxiety group would exhibit similar scores, given that all groups are associated with functional impairment.

Results from the ANOVAs are found in Tables 11.40 to 11.42. As expected, ratings of youth in the ADHD group were higher than ratings of youth in the Depression/Anxiety group on the Inattention/Executive Dysfunction (Parent Cohen’s d = 0.96, Teacher Cohen’s d = 0.44, and Self-Report Cohen’s d = 0.62), Hyperactivity (Parent Cohen’s d = 0.90, Teacher Cohen’s d = 0.72, and Self-Report Cohen’s d = 0.53), and Impulsivity scales (Parent Cohen’s d = 0.79, Teacher Cohen’s d = 0.47, and Self-Report Cohen’s d = 0.55). Also as expected, the ADHD group and the Depression/Anxiety group scored similarly on Emotional Dysregulation with no significant differences between groups. In terms of the Impairment & Functional Outcome Scales, both groups showed similar levels of impairment across scales, but small to medium differences emerged on the Schoolwork scale (Parent Cohen’s d = 0.73, Teacher Cohen’s d = 0.37, and Self-Report Cohen’s d = 0.42), with the ADHD group exhibiting more impairment than the Depression/Anxiety group. For the Peer Interactions scale, a small difference emerged on Self-Report (Cohen’s d = 0.39) with the Depression/Anxiety group scoring higher than the ADHD group. Conversely, for this same scale, a small difference emerged on Parent (Cohen’s d = 0.35) with the ADHD group scoring higher than the Depression/Anxiety group. For the Family Life scale, a small difference also emerged on Parent (Cohen’s d = 0.34) with the ADHD group scoring higher than the Depression/Anxiety group.

Classification Accuracy

Classification accuracy statistics were derived for the Conners 4–Short to consider the extent to which scores from each form correctly classify individuals into their respective groups (i.e., General Population vs. ADHD). For an operational definition of classification accuracy and a detailed description of methods used, see chapter 9, Validity. For a description of the classification accuracy statistics that were examined and how they are each derived, see chapter 6, Development.

For the Conners 4–Short, the binary classification modelling consisted of two sets of analyses: (a) binomial logistic regression, followed by (b) the creation of confusion matrices and the derivation of classification accuracy statistics. The logistic regression was used to predict how well the Conners 4–Short scales’ T-scores distinguished between individuals from the General Population and those diagnosed with ADHD (all presentations).

To facilitate unbiased predictions, matched samples were created to ensure that each of the General Population and ADHD groups that were submitted to binary classification modelling were nearly equivalent in terms of their demographic composition. The matched General Population and ADHD samples used to examine the classification accuracy of the Conners 4–Short were the same as those used to examine the classification accuracy of the full-length forms. The demographic characteristics of each sample are presented in appendix F (see Table F.25 to Table F.29). Group membership (ADHD group and General Population group) was predicted using the set of Content Scales and Impairment & Functional Outcome Scales that make up the Conners 4–Short, in tandem, as diagnostic decisions ought to be made accounting for both symptoms and impairment.

Predicting diagnostic status depends on the prevalence of ADHD in the population. The prevalence (or base rate of ADHD in the clinician’s referral population) can vary widely depending on the purpose of the evaluation and the setting. For example, in a screening setting one might expect the prevalence of ADHD to be around 10% or less, whereas in a clinically referred sample, a prevalence of approximately 50% may be more likely, rising to a prevalence of 60%–80% in an ADHD-specific clinical practice. Accordingly, the classification accuracy statistics of the Conners 4–Short scales, assuming a 50% base rate, are summarized in Table 11.43, and the Positive and Negative Predictive Values based on varying base rates are provided in Table 11.44. The overall correct classification rate was moderate to high (84.9% for Parent, 73.0% for Teacher, and 72.8% for Self-Report), with a balance between sensitivity and specificity. These results support the use of the Conners 4–Short for effective classification of individuals with and without ADHD.

< Back

Next >