• Sample
• Reliability
  • Table K.1. Internal Consistency: 2019 Samples for the CAARS 2
• Validity
  • Content and DSM Symptom Scales
    • Table K.2. Measurement Invariance (2019 vs. 2022): CAARS 2 Self-Report
    • Table K.3. Measurement Invariance (2019 vs. 2022): CAARS 2 Observer
    • Table K.4. Differential Test Functioning Effect Sizes (2019 vs. 2022)
    • Table K.5. Mean Differences (2019 vs. 2022): CAARS 2 Self-Report
    • Table K.6. Mean Differences (2019 vs. 2022): CAARS 2 Observer
  • CAARS 2–ADHD Index
  • Associated Clinical Concern Items and Impairment & Functional Outcome Items
    • Table K.7. Proportion of Associated Clinical Concern and Impairment & Functional Outcome Items Endorsed/Elevated for the 2019 and 2022 Samples
• Summary

The Conners Adult ADHD Rating Scales 2nd Edition (CAARS™ 2) released norms based on samples collected in 2019 (see chapter 7, Standardization, for details about the Normative Samples). However, since the collection of that data, the COVID-19 pandemic occurred and disrupted life for many around the world. Part of the disruption included public health measures (e.g., physical distancing, working from home), which helped control the spread of COVID-19. On the other hand, these measures also had unintended consequences, such as adversely affecting mental health. For example, 4 in 10 adults in the U.S. reported symptoms of anxiety or depression during the pandemic, compared to 1 in 10 before it in 2019 (National Center for Health Statistics, 2022).

Although adverse mental health symptoms appeared to increase during this time, it is important to investigate whether a fundamental shift occurred in the normative reports of ADHD symptoms and features, as measured by the CAARS 2. If the pandemic has affected how ADHD manifests, updates to the CAARS 2 may be needed to prevent over- or under-identification of ADHD and to maintain the CAARS 2’s strong psychometric properties.

To investigate this, a study was conducted to compare a general population sample collected post-pandemic outbreak (termed the “2022 sample” throughout this section) with a demographically matched subset of the existing Normative Sample established before the pandemic outbreak (termed “2019 sample” throughout this section; see chapter 7, Standardization). In this study, the psychometric properties of the two samples were compared to determine whether a meaningful change had occurred. Evidence of reliability and validity were investigated for the CAARS 2 scales and item-level content, and findings are reported in this appendix. Overall, minimal differences were found, supporting the continued use of the Normative Samples.

Sample

Data collection for the 2022 sample took place from late May to early June when responses to the CAARS 2 were collected digitally via online recruitment. Participants accessed the assessment via an email link that was shared with them, and they completed either the Self-Report or Observer form for a small monetary compensation.

Three hundred and sixteen adults completed the CAARS 2 Self-Report form, and 314 adults completed the CAARS 2 Observer form. Eligible individuals for this study were from the U.S., aged 18 years or older, read English “very well,” and did not have a clinical diagnosis. Data were cleaned prior to analysis. Based on data quality metrics (e.g., indicators of careless/random responding), 51 and 64 individuals were removed from the Self-Report and Observer samples respectively due to violations of one or more rigorous data quality metrics. To compare these samples with the 2019 Normative Samples, individuals were matched on gender, age (which was coarsened to five ordered levels), race/ethnicity, and education level. This process resulted in a further exclusion of 16 cases for both the Self-Report and Observer samples, due to a lack of matching demographic characteristics in the Normative Sample.

The final sample size for the 2022 sample was 249 individuals for the Self-Report (with a corresponding 249 matched individuals from the 2019 Normative Sample; see chapter 7, Standardization, for information about the full Normative Sample) and 234 individuals for Observer (with a corresponding 234 matched individuals from the 2019 Normative Sample). The demographic characteristics for both the 2019 and 2022 samples for the Self-Report and Observer forms can be seen in Appendix J, Tables J.36 and J.37.

Reliability

The reliability of the CAARS 2 scales for the 2022 samples was assessed via internal consistency estimates (alpha and omega) and test information functions (see Internal Consistency and Test Information in chapter 8, Reliability, for more details). Internal consistency estimates were found to be excellent (see Table K.1) and in line with the full normative sample (see Tables 8.1a, 8.1b, 8.2a, and 8.2b in chapter 8, Reliability); the median omega for both the Self-Report and Observer was .96.

Test information functions for the CAARS 2 Self-Report and Observer were also examined for the 2019 and 2022 samples (see Figure K.1 and Figure K.2 for the Self-Report and Observer, respectively). All test information functions had high information across the relevant range of the ability scale, with peaks exceeding values of 10, indicating very high measurement precision. Test information functions were similarly above recommended guidelines for both the 2019 and 2022 samples.

Taken together, results from the internal consistency analysis and test information functions revealed that scale reliability for the CAARS 2 Self-Report and Observer has remained sufficiently precise since the outbreak of the COVID-19 pandemic.

Click to expand

Figure K.1. Test Information Functions for the CAARS 2 Self-Report: 2019 and 2022 Samples

a) Inattentive/Executive Dysfunction

b) Hyperactivity

c) Impulsivity

d) Emotional Dysregulation

e) Negative Self-Concept

f) DSM ADHD Inattentive Symptoms

g) DSM Hyperactive/Impulsive Symptoms

h) DSM ADHD Total Symptoms

Click to expand

Figure K.2. Test Information Functions for the CAARS 2 Observer: 2019 and 2022 Samples

a) Inattentive/Executive Dysfunction

b) Hyperactivity

c) Impulsivity

d) Emotional Dysregulation

e) Negative Self-Concept

f) DSM ADHD Inattentive Symptoms

g) DSM Hyperactive/Impulsive Symptoms

h) DSM ADHD Total Symptoms

Validity

Content and DSM Symptom Scales

CAARS 2 Content Scales were analyzed for invariance of the factor structure and latent traits (using measurement invariance [MI] and differential test functioning [DTF]; see appendix M for details about these methodologies; note also that DSM symptom scales are not analyzed in this way, as the items are subsumed by the Content Scales and not factor-derived). Mean scale score differences between the two samples were also computed. Results of the MI analyses are presented in Tables K.2 and K.3, and examples of results for the DTF approach are presented graphically in Figure K.3, with effect sizes to quantify the differences seen in Table K.4.

Overall, the CAARS 2 Content Scales were found to be invariant across the 2019 and 2022 samples and no DTF was detected. These results provide strong evidence that the structure of the CAARS 2 Content Scales did not change as a result of the COVID-19 pandemic.

Click to expand

Figure K.3. Differential Test Functioning (2019 vs. 2022)

a1) Self–Report - Inattention

a2) Self–Report - Hyperactivity

b1) Observer - Inattention

b2) Observer - Hyperactivity

Mean scale scores for the 2019 and 2022 samples were compared, and results are presented in Tables K.5 and K.6. No statistically significant differences were observed between Content Scale or DSM Symptom Scale scores for the 2019 and 2022 samples (all p > .01), and all effect sizes were small or negligible (median d = 0.12 for both the Self-Report and Observer). Thus, mean observed scale scores did not appear to differ across the 2019 and 2022 samples.

CAARS™ 2–ADHD Index

The CAARS 2–ADHD Index was examined to confirm that it performed similarly across the 2019 and 2022 samples (see chapter 12, CAARS 2–ADHD Index, for more information on the development, scores, and psychometric properties of the CAARS 2–ADHD Index). The probability scores for each sample were compared using the Wilcoxon Rank Sum Test (note that this non-parametric approach was favored, as the probability score does not meet assumptions of normality; Wilcoxon, 1945); an effect size, r (Rosenthal, 1991), is also provided, which can be interpreted using the correlation guidelines provided in Test-Retest Reliability in chapter 8, Reliability.

There was no statistically significant difference in the probability scores between the 2019 and 2022 samples for both Self-Report (V = 32,864; p = .221; r = -0.05), and Observer (V = 27,438; p = .966; r = -0.002), and the magnitude of effects was very weak. Thus, it appears the CAARS 2-ADHD Index operated similarly in both the 2019 and 2022 samples.

Associated Clinical Concern Items and Impairment & Functional Outcome Items

The Associated Clinical Concern Items and Impairment & Functional Outcome Items of the CAARS 2 were examined in both the 2019 and 2022 samples. For each item, the proportion of individuals endorsing an item or providing an elevated item response in each sample was determined (see the Associated Clinical Concerns: Item Selection and Scoring section in chapter 6, Development for more information on how endorsed and elevated responses were determined for items in these scales). A chi-square independence test was performed to see whether these proportions were statistically significantly different in the 2019 and 2022 samples.

Results of the item-level analyses are presented in Table K.7. Overall, it was found that there were some statistically significant differences between the Associated Clinical Concern Items (for Observer) and the Impairment & Functional Outcomes Items (for both Self-Report and Observer). For these significant differences, proportions were higher in the 2022 sample as opposed to the 2019 sample. Given that the Content Scale scores appear to be unchanged during this same period, as described earlier in this appendix, but significant differences were noted for some of the item-level content, it is possible that the pandemic and its associated struggles are responsible for the increase in frequency and/or severity of these impairment items, as opposed to changes in the nature of ADHD symptoms and features itself. Thus, it would be prudent to monitor the changes observed here over the coming years.

Summary

The COVID-19 pandemic and its associated public health response led to worse mental health outcomes for individuals (NCHS, 2022). To ensure that the CAARS 2 structure and its normative scores did not change, a study was conducted to compare responses collected in 2022 to demographically matched subsets of the CAARS 2 Normative Samples. Overall, it was found that the CAARS 2 had comparable reliability in 2022 as compared to 2019. Further, it was found that the validity of the CAARS 2 was not compromised, as evidenced by the invariance of its Content Scales, a lack of statistical and practical differences in observed mean scale scores, and a lack of difference in probability scores associated with the CAARS 2-ADHD Index. Although it was found that there were some differences for select items in the Associated Clinical Concern Items and Impairment & Functional Outcome Items, the fact that ADHD Content scales did not significantly change indicates that these impairments may have been influenced by the pandemic itself. These elevations in the Associated Clinical Concern Items and Impairment & Functional Outcome Items may return to their 2019 levels as we proceed past the pandemic, or they may remain elevated if impacts continue to persist.