• Gender
• Race/Ethnicity
• Country of Residence
• Education Level

As with the CAARS 2, a goal for the CAARS 2–Short was to provide fair and unbiased measurement for diverse populations. In order to establish that this goal was met, differences between demographic groups (e.g., gender, race/ethnicity, country of residence, and education level) were examined to ensure that background characteristics of the individual being rated do not affect their test scores. Two main methods of evaluating bias were employed: invariance tests and mean group difference tests. Analyses related to fairness included measurement invariance (MI), differential test functioning (DTF), and comparisons of mean group differences; more information about the methodologies used can be found in appendix M. Visual inspections of DTF results were carried out for all scales (see Figure 10.1 in chapter 10, Fairness, for an example); however, for ease of presentation, effect size statistics are provided within this chapter to summarize the results. Because invariance testing relies on modeled data (i.e., estimating the population, rather than describing the sample), larger sample sizes are required, and a greater range of responses is desired. Therefore, the Total Sample (as described in the Standardization Phase in chapter 6, Development) was used for these invariance analyses, because it includes a considerable number of individuals from the general population, as well as individuals with clinical diagnoses (which will extend the variability of responses). Mean group differences were calculated on subsets of the Normative Sample (see the Fairness section in appendix J for demographic characteristics of the samples).

Gender

Gender, for the purposes of these fairness-related analyses, is defined as the rated individual’s gender identity. Analyses were conducted to compare males and females on the CAARS 2–Short in terms of MI, DTF, and mean group differences. The very small sample size for individuals who are non-binary or indicated “Other” for gender (Self-Report N = 11 and Observer N = 5) did not allow for meaningful testing. Therefore, when assessing invariance by gender, only males (Self-Report N = 1,028; Observer N = 1,021) and females (Self-Report N = 1,186; Observer N = 1,123) were included.

Invariance between males and females for the CAARS 2–Short was first explored via MI analyses (see Table 11.25). For Self-Report and Observer, there are some models with a significant Satorra-Bentler χ² test (e.g., strict models); however, the absence of any decline in many other fit statistics, such as CFI and SRMR, suggest invariance was upheld. As more constraints were added throughout the process of testing MI, model fit did not change in a meaningful way, indicating that the factor model is invariant between males and females

In addition to MI results, DTF analyses were conducted to explore the invariance of the CAARS 2–Short for males and females through a different framework. The effect sizes of the DTF statistics, as measured by the expected test score standardized difference (ETSSD), are summarized in Table 11.26 for the CAARS 2–Short. Negligible differences (i.e., ETTSD ≤ |.06|) between males and females were observed across all scales and both forms, demonstrating invariance by gender.

To examine observed group differences between genders, a matched sample of male individuals was selected at random from the Normative Samples. Individuals were matched by education level (EL), language(s) spoken, clinical status, race/ethnicity, and age. The demographic characteristics of the rated individuals in the matched samples (and their raters, where applicable) are presented in appendix J.

The paired samples of males and females were then compared for significant differences across mean scores. Results of the ANOVAs and descriptive statistics for each scale are presented in Tables 11.27 and 11.28. When comparing ratings of male and female individuals, for Self-Report, there were statistically significant differences observed for Inattention/Executive Dysfunction, Hyperactivity, and Impulsivity scales, wherein males yielded slightly higher scores than females; however, the effect sizes were small (Cohen’s d = |0.23 to 0.29|). For the Observer results, the only statistically significant effect observed was on the Negative Self-Concept scale, wherein ratings of females yielded slightly higher scores than males; however, the effect size was small (Cohen’s d = |0.26|). The effect of gender on the remainder of the scale scores was not statistically significant.

Overall, these results support the absence of meaningful gender differences. Taken together, results from the MI, DTF, and mean group difference analyses indicate psychometric equivalence between males and females for the CAARS 2–Short Content Scales. There was no strong evidence for meaningful differences in terms of latent structure nor in terms of test functioning between the two gender groups, and scores for males and females were not meaningfully different. Although results of these analyses suggest that use of Combined Gender as the primary reference group is appropriate, there may be instances where specific comparisons to a gender group are desired. Gender Specific and Combined Gender normative scoring options are available; please see Scoring and Report Options in chapter 3, Administration and Scoring, for details.

Race/Ethnicity

Establishing that the CAARS 2 operates equivalently and produces equally valid scores that can be interpreted in the same way across racial and ethnic groups is a critical component of ensuring that psychometric fairness standards are met. For the U.S. portion of the CAARS 2 Normative Sample, race and ethnicity were categorized according to the U.S. Census Bureau classifications into the following five groups: Hispanic (regardless of race), Black, Asian, White, and Other (which includes Native, Multiracial, and other racial identities not otherwise listed). Individuals whose race/ethnicity was classified as Other were excluded from MI and DTF analyses due to small sample sizes (meaningful interpretation of group-level scores is challenging given that this category includes a multitude of race groups). Additionally, race/ethnicity analyses in this section are limited to individuals who live in the U.S., as the Canadian sample sizes were too small to permit meaningful analyses. More details about the correspondence to the Census classifications can be found in Race/Ethnicity in chapter 7, Standardization.

Differences among the U.S. racial and ethnic groups were explored with regard to the CAARS 2–Short structure and scores. It was expected that there would be negligible differences in terms of race/ethnicity, as the test was designed to minimize the impact of an individual’s background, with the goal of generalizing to diverse populations.

First, MI was explored within the U.S. subsamples of the CAARS 2–Short Total Sample. Due to the smaller sample sizes for Black individuals on the CAARS 2–Short forms (Self-Report N = 186; Observer N = 199), Hispanic individuals (Self-Report N = 229; Observer N = 232) and Black individuals were combined into a larger group (Self-Report N = 415; Observer N = 431), and the combined group was compared to White individuals (Self-Report N = 1,311; Observer N = 1,021) for MI analysis. It should be acknowledged that combining the groups in this way, while necessitated by small samples sizes, may nonetheless limit interpretability of results; It is important for future analyses to explore White vs. Hispanic and White vs. Black comparisons in larger samples to confirm the pattern of results presented within this section.

Table 11.29 presents the MI results for Self-Report and Observer, respectively. For Self-Report, there were no significant decreases in model fit (in particular, the Satorra-Bentler χ² test) when comparing the increasingly strict models tested at each subsequent level. For Observer, although the comparison between strong (third level of testing) and strict models (fourth level of testing) was significantly different when examined with the Satorra-Bentler χ² test, no other fit statistics showed any meaningful decreases in model fit, indicating that strong invariance is met. This evidence supports the CAARS 2–Short’s ability to measure the construct with the same structure for White individuals as it does for a combined sample of Black and Hispanic individuals.

DTF analyses were examined next, to explore the invariance of the CAARS 2–Short across race/ethnicity, specifically between Hispanic and White individuals and between Black and White individuals (note that Black and Hispanic groups were not directly compared, as the investigation here was more concerned with comparisons of historically marginalized groups to a majority group; instead, analyses permitted separating Black and Hispanic groups apart for DTF, rather than combining them as had been done for MI). The effect size of the DTF statistics measured by the ETSSD are summarized in Table 11.30. The largest effect size across Self-Report and Observer, when comparing either Black or Hispanic to White individuals, was ETSSD = |0.12|. The differences between groups are negligible, demonstrating a lack of measurement bias between White, Black, and Hispanic groups and reinforcing the generalizability of the CAARS 2.

To investigate racial/ethnic groups in terms of their mean score differences, subsamples of the U.S. portion of the Normative Sample of Hispanic, Black, and Asian individuals, respectively, were compared, respectively, to a corresponding subsample of White individuals from the Normative Sample that were matched on gender, education level (EL), language(s) spoken, clinical status, and age. The demographic characteristics of the rated individuals in the matched samples (and their raters, where applicable) are presented in appendix J.

Comparisons between the matched Hispanic and White samples, Black and White samples, and Asian and White samples were analyzed via a series of ANOVAs, and results are presented in Tables 11.31 to 11.36. These tables present the means and standard deviations of the CAARS 2 scale scores, along with the significance tests and effect sizes.

When comparing ratings of Hispanic and White individuals, Observer results indicated there were no statistically significant differences observed across all scales, with negligible to small effect sizes between groups (median Cohen’s d = |0.08|). For Self-Report, a statistically significant difference was observed between Hispanic and White individuals on the Negative Self-Concept scale, with a small effect size (Cohen’s d = 0.43), wherein White individuals endorsed slightly more symptoms or slightly greater severity than Hispanic individuals.

When comparing ratings of Black and White individuals, Observer results indicated there were no statistically significant differences across all scales, with negligible to small effects (median Cohen’s d = |0.07|). For Self-Report, there were statistically significant differences for the Hyperactivity, Impulsivity, and Negative Self-Concept scales. Lower scores were observed for Black individuals than White individuals, with small to medium effect sizes for the significant differences (median Cohen’s d = |0.32|), indicating that White individuals endorsed more symptoms or greater severity than Black individuals.

When comparing ratings of Asian and White individuals across both forms, results indicated that there were no statistically significant effects observed across all scales. Cohen’s d effect sizes, capturing the size of the difference between group means, demonstrated negligible to small effects (Cohen’s d = |0.04| to |0.41|).

Overall, there were small differences, the majority of which were not significant, observed between White and Hispanic, Black, and Asian individuals on the CAARS 2–Short. Taken together, results from the MI, DTF, and mean group difference analyses indicate psychometric equivalence for Hispanic and White individuals, Black and White individuals, and Asian and White individuals. Together with the absence of evidence for measurement bias, there is support for equity in terms of race/ethnic groups for the CAARS 2–Short and its appropriate use in racially and ethnically diverse populations.

Country of Residence

To address equivalence of scores across different countries, ratings of individuals in the U.S. and Canada were compared on the CAARS 2–Short. Cross-cultural differences were expected to be minimal, and the lack of meaningful differences would support the generalizability and utility of the CAARS 2–Short in both countries.

The invariance of the factor structure was compared on the individual’s country of residence. Results examining MI between the U.S. (Self-Report N = 1,881; Observer N = 1,821 for Observer) and Canada (Self-Report N = 344; Observer N = 329) are found in Table 11.37.

Across the CAARS 2–Short Self-Report and Observer scales, there were no statistically significant reductions in the Satorra-Bentler χ² test or any declines in model fit statistics when comparing different levels of invariance. Because the CAARS 2–Short meets the most stringent level of invariance tested in terms of country of residence, these results support generalizability of the CAARS 2–Short Self-Report and Observer forms to individuals who live in the U.S. or Canadal. Next, DTF was evaluated with regard to country of residence. Effect sizes of the DTF analyses for both CAARS 2–Short forms are presented in Table 11.38. Negligible differences between countries were found (i.e., ETSSD ≤ |.06|), which further supports the generalizability of the CAARS 2–Short to U.S. and Canadian populations alike.

To examine group differences between countries, a subsample of the individuals from the U.S. portion of the Normative Sample was randomly selected to match a sample of Canadian individuals from the Normative Sample in terms of gender, education level (EL), language(s) spoken, clinical status, and age (Self-Report N = 172 and Observer N = 162). Refer to appendix J for the demographic characteristics of the samples.

The paired U.S. and Canadian samples were then compared for significant differences across mean scores. Results of the ANOVAs and descriptive statistics for each scale are presented in Table 11.39 and Table 11.40. Across both rater forms and all scales, there were no statistically significant effects of country of residence. Ratings of individuals from the U.S. and Canada resulted in very similar mean scores as described by Cohen’s d; all effect sizes were negligible to small (median Cohen’s d = 0.19 and 0.07 for Self-Report and Observer, respectively). These results indicate that country of residence (specifically, U.S. vs. Canada) had no significant effect on the CAARS 2–Short scales scores.

Taken together, results from the MI, DTF, and mean group difference analyses indicate psychometric equivalence between individuals from the U.S. or Canada alike on the CAARS 2–Short scales. There was no evidence for meaningful differences in terms of latent structure nor in terms of test functioning between the groups, and scores were not meaningfully different.

Education Level

An individual’s education level (EL) can sometimes be considered a proxy for or a contributing factor to one’s socioeconomic status (SES), which is among the sociocultural variables that may influence the fairness of a test. It was expected that the constructs measured on the CAARS 2–Short would be independent of influence from EL. To test this hypothesis and ensure generalizability of scores from the CAARS 2–Short Content Scales, individuals in the Self-Report and Observer samples reported the EL of the rated individual using one of five options: No high school diploma (EL 1), High school diploma/GED (EL 2), Some college/university or associate degree (EL 3), Bachelor’s degree (EL 4), or Graduate or professional degree (EL 5; more information about the representativeness of these groups can be found in Education Level in chapter 7, Standardization). For the sake of invariance analyses, EL was re-categorized into two groups comprising individuals with and without post-secondary education (i.e., Group 1 consists of EL 1 and EL 2: N = 1,515 for Self-Report and N = 1,134 for Observer; Group 2 consists of EL 3, EL 4, and EL 5: N = 710 for Self-Report and N = 796 for Observer).

First, differences in the factor structure based on EL groups were evaluated with MI. With more stringent models tested at each level, neither the CAARS 2–Short Self-Report nor the CAARS 2–Short Observer displayed meaningful deterioration in model fit (see Table 11.41). For Self-Report, some comparisons were significant using the Satorra-Bentler χ² test (e.g., the loading versus intercept model comparison; p < .001); however, the indicators must be considered together, and no other model fit statistics indicated meaningful change. Therefore, the observed change in model fit is minor and not meaningful, such that invariance between the EL groups on the construct assessed by the CAARS 2–Short can reasonably be assumed. These results support the invariance of the CAARS 2–Short across factor structure, thresholds, loadings, and intercepts between individuals with and without post-secondary education, meeting the first-step criteria for establishing its unbiased and generalizable use across these populations.

Next, differences in the CAARS 2–Short scales’ functioning for the two broad EL groups were explored with DTF. The effect sizes from DTF analyses are summarized in Table 11.42. Results from both Self-Report and Observer show negligible differences between EL groups (e.g., maximum ETSSD = |.07|). The lack of differential functioning of the CAARS 2–Short scales between the different EL groups is further evidence of the test’s equivalence across demographic subgroups.

Next, the five EL groups were compared in terms of the observed mean score group differences. These group differences were analyzed using the entire Normative Sample. EL was compared via a series of ANCOVAs, with covariates to statistically control for the effects of other demographic factors (i.e., gender, age, race/ethnicity, primary language spoken, clinical status). Significant ANCOVA results (i.e., p < .01) were followed up with Tukey’s Honestly Significant Difference (HSD) post-hoc test to evaluate pairwise comparisons, alongside estimates of effect sizes for the omnibus test and pairwise differences.

Results of the ANCOVAs for Self-Report are provided in Table 11.43a, with effect sizes of the pairwise comparisons between group means provided in Table 11.43b. Corresponding results for Observer are provided in Tables 11.44a and 11.44b. For both forms, significant differences between EL groups were only observed for the CAARS 2–Short Emotional Dysregulation scale, but the size of these effects was negligible (partial η² = .01 for both forms). The post-hoc analysis and pairwise comparisons revealed that ratings of individuals from the EL 1 and EL 4 groups resulted in statistically significantly different mean scores, but effect sizes were small (Cohen’s d = 0.32 for Self-Report and 0.27 for Observer). This effect manifests as a difference of approximately 3 T-score points higher for individuals in the EL 1 group on the CAARS 2–Short Emotional Dysregulation scale.

Overall, these results support the absence of meaningful differences in the measurement properties of the test across low and high EL groups. Taken together, results from the MI, DTF, and mean group difference analyses indicate that the CAARS 2–Short can generalize across EL groups for the Content Scales. There was no strong evidence for meaningful differences in terms of latent structure nor in terms of test functioning between the two groups, and scores were not meaningfully different, supporting the unbiased use of the CAARS 2–Short for individuals with a range of educational backgrounds and levels.