• Gender
• Race/Ethnicity
• Country of Residence
• Education Level

To provide evidence that the CAARS 2–ADHD Index provides a fair and unbiased measurement for diverse populations, differences were examined between demographic groups. Measurement invariance was explored via differential test functioning (DTF), with a visual inspection of test characteristic curves for each group (see appendix M for details about the methodology). A summary of corresponding effect sizes for the differences in test characteristic curves is provided throughout this section, presented as the estimated test score standardized difference (ETSSD). Given the multidimensional nature of ADHD (as evidenced by the factor structure presented in Internal Structure in chapter 9, Validity) and that the CAARS 2–ADHD Index comprises items from multiple content domains, measurement invariance testing with nested confirmatory factor analyses (CFAs) was not practical, as the CAARS 2–ADHD Index was not designed to mirror the full multidimensional structure of the CAARS 2. The items on the CAARS 2–ADHD Index are indicators that together produce a unidimensional measure of ADHD symptoms and a corresponding score that can reflect one’s group membership (i.e., likelihood of resembling scores from the ADHD Reference Sample); therefore, DTF alone was a sufficient method of evaluating invariance. Prior to conducting the DTF analyses, CFAs were conducted to ensure the statistical requirement for unidimensionality was met. Results supported a 1-factor model for both the Self-Report (CFI = .986, TLI = .983, RMSEA = .075, SRMR = .037) and Observer (CFI = .964, TLI = .957, RMSEA = .109, SRMR = .058). Because invariance testing relies on modeled data (i.e., estimating the population, rather than describing the sample), larger sample sizes are required and a greater range of responses is desirable. Therefore, the Total Sample (as described in the Standardization Phase in chapter 6, Development) is used for these invariance analyses, because it includes a considerable number of individuals from the general population, as well as individuals with clinical diagnoses (which will extend the variability of responses).

To examine the generalizability of the obtained scores, the effects of demographic group membership were analyzed by a comparison of effect sizes between response distributions of the groups. Group differences were calculated on subsets of the Normative Sample (see appendix J for demographic characteristics of the samples used in this section).

Gender

The invariance between males and females for the CAARS 2–ADHD Index was explored via DTF analyses using the Total Sample for each rater form. In these comparisons, a positive estimated test score standardized difference (ETSSD) value denotes that females would receive a higher score when males and females were equal with respect to ADHD-related symptoms, while negative values indicate females would receive a lower score than males. The effect size of the DTF statistic measured by the ETSSD was -0.01 for both Self-Report and Observer. Effects of this magnitude are all trivial in size. These results from the DTF analyses indicate that the CAARS 2–ADHD Index demonstrates psychometric equivalence across males and females, as there was no evidence for meaningful differences in test functioning between the two groups.

To further explore potential gender differences, observed group differences between scores on the CAARS 2–ADHD Index were also investigated. These group differences were analyzed using a matched sample (matched by age, race/ethnicity, education level, language[s] spoken, and clinical status; refer to appendix J for demographic characteristics of the matched samples). Differences are presented as Cliff’s d values, comparing the frequency of each probability score category within each gender group. Results of these analyses are presented in Table 12.18 for both forms. Effect sizes were negligible for both comparisons (Cliff’s d = -.04 for Self-Report and Cliff’s d = .00 for Observer).

Overall, the very small differences observed between gender groups add support for the generalizable use of the CAARS 2–ADHD Index across males and females. These findings, together with the DTF results, provide evidence for the equitable measurement by gender when using the CAARS 2–ADHD Index.

Race/Ethnicity

Invariance between Hispanic and White individuals and between Black and White individuals for the CAARS 2–ADHD Index was explored via DTF analyses using the Total Sample for each rater form (the sample size for Asian individuals was too small to permit DTF). When comparing Hispanic and White individuals, the ETSSD effect size was -0.04 for Self-Report and 0.00 for Observer. When comparing Black and White individuals, effect sizes were 0.05 for Self-Report and 0.06 for Observer. All effects were trivial in size, indicating no evidence of DTF by race/ethnicity. These results demonstrate a lack of race/ethnicity-based measurement bias between White, Black, and Hispanic groups, reinforcing the generalizability of the CAARS 2–ADHD Index by demonstrating psychometric equivalence for White, Hispanic, and Black individuals.

To further explore potential race/ethnicity differences, observed group differences between scores on the CAARS 2–ADHD Index were also investigated. These group differences were analyzed using matched samples from the Normative Sample (matched on gender, language[s] spoken, clinical status, and EL; refer to appendix J for demographic characteristics of the matched samples). Results of these analyses for all forms are presented in Table 12.19. Effects were negligible when comparing ratings of White and Black, White and Hispanic, and White and Asian individuals (Cliff’s d ranging from -.10 to -.14 for Self-Report and .00 to .08 for Observer).

Overall, the very small differences observed between race/ethnicity groups add support for the fair and generalizable use of the CAARS 2–ADHD Index across these populations, and, together with the DTF results, provide evidence for equitable measurement when using the CAARS 2–ADHD Index.

Country of Residence

To address equivalence of scores across countries, individuals in the U.S. and Canada were compared on the CAARS 2–ADHD Index. Cross-cultural differences were expected to be minimal, and the lack of meaningful differences would support the generalizability and utility of the CAARS 2–ADHD Index for use in both the U.S. and Canada.

The invariance by country of residence for the CAARS 2–ADHD Index was explored via DTF analyses using the Total Sample for each rater form. The effect size of the DTF statistic was negligible for both forms (ETSSD = -0.01 for Self-Report and 0.00 for Observer; negative ETSSD values indicate higher expected scores for Americans than Canadians even when matched on the construct being measured). Trivial differences were found (effect sizes at or below |.01|), which demonstrates the invariance of the CAARS 2–ADHD Index across countries and supports its generalizability to U.S. and Canadian populations.

To further explore potential country of residence differences, the CAARS 2–ADHD Index was also investigated in terms of observed group differences between scores. These group differences were analyzed using a matched sample (matched on gender, language[s] spoken, clinical status, and EL; refer to appendix J for demographic characteristics of the matched samples). Group differences were analyzed via Cliff’s d effect size values, which compared the two countries in terms of the percentage of the sample scoring within each category of the CAARS 2–ADHD Index probability score. Results of these analyses are presented in Table 12.20 for both forms. Effect sizes were negligible for all comparisons (Cliff’s d = .05 for Self-Report and Cliff’s d = .05 for Observer).

Overall, the very small differences observed between the country of residence groups add support for the fair and generalizable use of the CAARS 2–ADHD Index with individuals from the U.S. and Canada, and together with the DTF results, provide evidence for equitable measurement of individuals from the U.S. and Canada, using the CAARS 2-ADHD Index.

Education Level

Education level (EL) can sometimes be considered a proxy for, or a contributing factor to, one’s socioeconomic status (SES), which is among the sociocultural variables that may influence the fairness of a test. It was expected that the constructs measured on the CAARS 2 would be independent of influence from EL. To test this hypothesis and ensure generalizability of scores from the CAARS 2–ADHD Index, individuals in the Self-Report and Observer samples reported the EL of the rated individual from one of five options: No high school diploma (EL 1), High school diploma/GED (EL 2), Some college or associate degree (EL 3), Bachelor’s degree (EL 4), and Graduate or professional degree (EL 5). Within the normative samples, the proportion of individuals in each of these groups matched recent U.S. and Canadian census values (more information can be found in Education Level in chapter 7, Standardization).

Equivalence across the EL groups was investigated with DTF analyses using the Total Sample for the Self-Report and Observer. For the sake of DTF analyses, which require binary variables, EL was re-categorized into two groups: individuals without post-secondary education (EL 1 and EL 2; N = 505 for Self-Report and N = 515 for Observer) and individuals with post-secondary education (EL 3, EL 4, and EL 5; N = 815 for Self-Report and N = 804 for Observer). The DTF analysis revealed trivial effect sizes (ETSSD = 0.02 for Self-Report and 0.06 for Observer). The very small values observed in this analysis support the invariance of the CAARS 2–ADHD Index with regard to the educational background of the individual being rated.

To further explore potential EL differences, the CAARS 2–ADHD Index was also investigated for potential group differences in terms of the frequency distribution of scores for each group. These group differences were analyzed using the entire Normative Sample. Although previous sections of this chapter looked at group differences based on matched subsamples of the Normative Sample, it was not possible to create demographically matched groups for the EL analysis due to small sample sizes of the EL groups. Given the lack of differences found for other demographic variables (as evidenced by results earlier in this section), results for the EL analysis are unlikely to be influenced by the inclusion of covariates. Differences are presented as Cliff’s d values, which compare the percentage of the sample scoring within each category of the CAARS 2–ADHD Index probability score between the EL groups. Results of this analysis are presented in Table 12.21a and Table 12.21b for Self-Report and Observer. Effect sizes were negligible for all of the comparisons (Cliff’s d ranges from -.06 to .05 for Self-Report and -.02 to .08 for Observer).

Overall, the very small differences observed between EL groups demonstrate a lack of influence of education level on the CAARS 2–ADHD Index. This finding adds support for the fair and generalizable use of the CAARS 2–ADHD Index, and together with the DTF results, provides evidence for equitable measurement across EL groups when using the CAARS 2–ADHD Index.