Conners 4 Manual Chapter 11: Fairness

Fairness

view all chapter tables | print this section

• Gender
• Race/Ethnicity
• Country of Residence
• Parental Education Level

To provide evidence that the Conners 4–Short provides fair and unbiased measurement for diverse populations, differences between demographic groups were examined (i.e., gender, race/ethnicity, country of residence, and PEL; PEL was applicable to Parent and Self-Report only) to ensure that background characteristics of raters do not affect their test scores. Two main methods of evaluating bias were employed: invariance tests and mean group difference tests.

Invariance testing broadly refers to the degree that a test generalizes across groups. When a test is invariant across groups, the test measures the same construct in the same way (Vandenberg & Lance, 2000), regardless of demographic group membership. Establishing measurement invariance (MI) is a necessary step in applying the test scores to specific groups, as without it, interpretation becomes extremely difficult since the same score does not necessarily mean the same thing for different groups. MI can be established using different analyses, including confirmatory factor analysis (CFA; from a classical test theory [CTT] framework) and differential test functioning (DTF; from an item-response theory [IRT] framework). CFA involves statistical comparison between fit of increasingly strict models, while DTF involves a visual inspection of test characteristic curves for each group, which can be summarized with an effect size statistic. Visual inspection was carried out for all scales (see Figure 10.1 in chapter 10, Fairness, for an example); however, for ease of presentation, only the effect size statistics are provided within this chapter to summarize the results. The results of each type of analysis provide slightly different information (see chapter 10, Fairness, for a detailed explanation of these methods). Because invariance testing relies on modeled data (i.e., estimating the population, rather than describing the sample), larger sample sizes are required, and a greater range of responses is desired. Therefore, the Total Sample (as described in the Standardization Phase in chapter 6, Development) is used for these invariance analyses, because it includes a considerable number of youth from the general population, as well as youth with clinical diagnoses (which will extend the variability of responses).

To examine the generalizability of the obtained scores, the effects of demographic group membership were analyzed by a comparison of mean differences. Note that mean differences observed for subgroups of a population do not directly address measurement bias in the way that MI and DTF analyses do; however, mean differences do offer insight into the applied use of the test and whether adverse effects may occur through the intended use. If a test is invariant across groups, confidence can be greater that observed differences in scores between groups reflect population differences, rather than measurement bias. That is, the test may be consistent in its measurement of the construct across demographic groups, and sensitive enough to detect real-world group differences that occur independently of the test or that are due to an unmeasured variable. Mean group differences were calculated based on subsets of the Normative Samples (see appendix F for demographic characteristics of the samples).

Gender

Gender, for the purposes of fairness-related analyses, is defined as the rated youth’s gender identity, with analyses conducted using the Total Samples to compare males and females on the Conners 4–Short in terms of MI and DTF. The sample size for youth who are non-binary or indicated “Other” for gender (N = 6 for Parent, N = 1 for Teacher, and N = 3, for Self-Report) did not allow for meaningful testing. Therefore, when assessing invariance by gender, only males (N = 1,706 for Parent; N = 1,473 for Teacher; and N = 787 for Self-Report) and females (N = 1,539 for Parent; N = 1,404 for Teacher; and N = 796 for Self-Report) were included.

Invariance between males and females for the Conners 4–Short was first explored via MI analyses (see Tables 11.45 through 11.47). The Satorra-Bentler χ² (Satorra, 2000) was statistically significant for some comparisons (e.g., the intercept comparisons). The other fit statistics did not show any decline in model fit for any of the comparisons, indicating there was no meaningful change overall and that invariance was upheld. This pattern of results is consistent across all forms. Therefore, as more constraints are added throughout the process of testing MI, the model fit does not change in a meaningful way, indicating that the factor structure, loadings, thresholds, and intercepts are invariant between males and females.

In addition to MI results, DTF analyses were conducted to explore the invariance of the Conners 4–Short for males and females. The effect sizes of the DTF analyses, as measured by the expected test score standardized difference (ETSSD), are summarized in Table 11.48 for Conners 4–Short Parent, Teacher, and Self-Report. Negligible differences (i.e., effect sizes lower than |.08|) between males and females were observed across all forms and all scales.

Taken together, results from both MI and DTF analyses indicate that the Conners 4–Short scale scores demonstrate psychometric equivalence across males and females, as there was no evidence for meaningful differences between the two groups in terms of their latent structure, nor in their test functioning.

To compare males and females in terms of their mean score differences, subsamples of the Conners 4–Short Normative Samples of males were compared to a corresponding subsample of females from the Normative Sample that were matched on PEL (for Parent and Self-Report only), age, race/ethnicity, language(s) spoken, and clinical status. Refer to Table F.36 and Table F.37 in appendix F for the demographic characteristics of the samples. Results, including descriptive statistics and estimates of significance and effect sizes for the ANOVA tests, are presented in Tables 11.49 to 11.51.

For Conners 4–Short Parent, there were no statistically significant effects across all scales, with negligible effect sizes (Cohen’s d = |0.01| to |0.12|). For Conners 4–Short Teacher, there was a statistically significant effect of gender for all scales other than Emotional Dysregulation, with ratings of males resulting in slightly higher scores than ratings of females; however, the effect sizes were negligible to small (Cohen’s d = .03 to .41). For Conners 4–Short Self-Report, there were no statistically significant effects across all scales, with negligible effect sizes (Cohen’s d = 0.00 to |0.14|).

Overall, these results support the absence of meaningful gender differences, and together with the MI and DTF results, provide evidence for equitable measurement for males and females when using the Conners 4–Short.

Race/Ethnicity

The historical and continuing marginalization of certain racial and ethnic groups has the potential to disadvantage certain groups (e.g., yield higher scores) or introduce unintended bias during the test construction. Therefore, it was important to examine how the Conners 4–Short operates for these groups to ensure equity standards are met. For the U.S. portion of the Conners 4 Normative Samples, race and ethnicity were categorized according to the U.S. Census classifications into the following five groups: Hispanic (regardless of race), Black, Asian, White, and Other (includes Native American, Multiracial, and other racial identities not otherwise listed). Youth whose race/ethnicity was classified as Asian or Other were excluded from MI and DTF analyses due to small sample sizes (for Other in particular, meaningful interpretation of group-level scores is challenging given this category includes a multitude of race groups). Additionally, race/ethnicity analyses in this section are limited to youth who live in the U.S., as the Canadian sample sizes were too small to permit meaningful analyses. More details about the correspondence to the U.S. Census classifications can be found in Race/Ethnicity in chapter 7, Standardization. Differences among the U.S. racial and ethnic groups were explored with regard to the Conners 4–Short structure and scores. It was expected that there would be negligible differences in terms of race/ethnicity as the test was designed to minimize the influence of cultural background, with the goal of generalizing to diverse populations.

When assessing invariance with MI and DTF, comparisons were conducted for groups of sufficient size; that is, White youth were compared with Hispanic youth and Black youth, but comparisons could not be made with Asian youth due to the limited sample size. Note that for the full-length Conners 4, analyses of MI were conducted with groups of White youth and a combined group of Black and Hispanic youth, while for the Conners 4–Short the analyses could be conducted for the groups separately, as the complexity of the factor model is reduced for the shortened forms and the number of ordinal items included is considerably lower than for the full-length forms. When assessing mean group differences by race/ethnicity, comparisons were made between White youth and Hispanic, Black, and Asian youth.

First, MI was explored within the U.S. subsamples of the Conners 4–Short Total Samples. Tables 11.52 to 11.54 present the MI results for the comparison of Hispanic (N = 520 for Parent, N = 411 for Teacher, and N = 286 for Self-Report) and White youth (N = 1,728 for Parent; N = 1,545 for Teacher; and N = 791 for Self-Report). There were no meaningful decreases in model fit across Parent, Teacher, and Self-Report. The Satorra-Bentler χ² test was significant for many of the comparisons, but other fit statistics did not show a decline in fit, indicating no evidence for a lack of invariance.

Tables 11.55 to 11.57 present the MI results for the comparison of Black (N = 312 for Parent, N = 376 for Teacher and N = 171 for Self-Report) and White youth (N = 1,728 for Parent; N = 1,545 for Teacher; and N = 791 for Self-Report). Across all rater forms, there was no evidence of a meaningful decrease in model fit when comparing these two racial groups. While the Satorra-Bentler χ² test was significant for some comparisons (e.g., the intercept model when testing Content Scales), other fit statistics did not show a decline in fit, supporting the overall invariance of the Conners 4–Short scales. Note that there were several improvements in model fit, including a CFI change greater than .01 on the Impairment & Functional Outcome Scales for White and Black groups for Self-Report. However, improvements to the goodness-of-fit indices do not indicate differences related to invariance in the same way that decreases or deteriorations in fit indices do. Therefore, the MI analysis supports measurement invariance.

DTF analyses were examined next, to explore the invariance of the Conners 4–Short scales between Hispanic and White youth and between Black and White youth. The effect size of the DTF statistics measured by the ETSSD are summarized in Table 11.58. The largest effect size across Parent, Teacher, and Self-Report, when comparing either Black or Hispanic to White youth, was ESSTD = |0.08|, representing a trivial effect size. The differences between groups are negligible. Taken together, both MI and DTF analyses indicate that the Conners 4–Short demonstrates psychometric equivalence for White and Hispanic youth, and for White and Black youth, supporting the fair use of the test.

To investigate race/ethnic groups in terms of their mean score differences, subsamples of the U.S. portion of the Normative Samples of Hispanic, Black, and Asian youth, respectively, were compared to a corresponding subsample of White youth from the Normative Samples that were matched on age, gender, language(s) spoken, clinical status, and PEL (PEL matched for parent and self-report only). Refer to Tables F.38 to F.42 in appendix F for the demographic characteristics of the samples.

Comparisons between the matched Hispanic and White samples, Black and White samples, and Asian and White samples were analyzed via a series of ANOVAs, and results are presented in Tables 11.59 to 11.67.

When comparing ratings of Hispanic and White youth across all three forms, results indicated that there were no statistically significant effects observed across all scales. Cohen’s d effect sizes, capturing the size of the difference between group means, demonstrated negligible to small effects (Cohen’s d = 0.00 to |0.27|).

When comparing ratings of Black and White youth, the Parent and Self-Report results showed no statistically significant effects across all scales. Cohen’s d effect sizes were negligible to small (Cohen’s d = |0.01| to |0.28| across both Parent and Self-Report). The effect of student race on the Teacher form was statistically significant for Inattention/Executive Dysfunction, Hyperactivity, Impulsivity, and Schoolwork, wherein ratings resulted in slightly higher scores for Black students than White students (Cohen’s d = |0.21| to |0.40|). While race/ethnicity did not have a significant effect on the scale scores for Parent and Self-Report, the White and Black comparisons for the Conners 4–Short Teacher showed several small but statistically significant effects. This effect manifests as a difference of up to 4 points more for Black students on some the Conners 4–Short scales.

When comparing ratings of Asian and White youth across all forms, results indicated there were no statistically significant effects observed across all scales. The effect of race was minimal for Parent and Teacher, as evidenced by negligible to small effect sizes (Cohen’s d = |0.02| to |0.42|). Although not statistically significant, a small effect size was observed when comparing teacher ratings of Asian and White youth on Impulsivity; scores of White students were approximately 3 points higher on average, with a Cohen’s d of 0.41. For Self-Report, no significant effects were found; however, small effect sizes were observed for three scales (Cohen’s d ranging from 0.33 to 0.38).

Overall, there were small differences (the majority of which were not significant), observed between White and Hispanic, Black, and Asian youth on the Conners 4–Short. While the results presented in this section provide evidence for a lack of measurement bias in the Conners 4–Short, when considering obtained scores (as described in chapter 4, Interpretation), it is important to note that there was a small but statistically significant trend for Black students to be rated higher on some scales than White students on the Teacher form. This trend of ratings may be important to keep in mind when initiating contact with teacher raters, to encourage mindfulness of potential threats to providing unbiased ratings, or when comparing results for Black youth from the Parent and Self-Report form to the Teacher form, as difficulties may appear to vary across domains. This trend also reflects what has been documented in previous literature, such that teachers may perceive Black students’ behavior as more problematic than their White peers (Neal et al., 2003; Rowley et al., 2014).

Together with the absence of evidence for measurement bias, there is support for equity in terms of race/ethnic groups for the Conners 4–Short and its appropriate use in racially and ethnically diverse populations.

Country of Residence

To address equivalence of scores across different countries, ratings of youth in the U.S. and Canada were compared on the Conners 4–Short. Cross-cultural differences were expected to be minimal, and the lack of meaningful differences would support the generalizability and utility of the Conners 4–Short in both countries.

First, the invariance of the factor structure was compared based on the youth’s country of residence. Note that for the full-length Conners 4, comparisons between the U.S. and Canada on the Self-Report form could not be reliably calculated due to the complexity of the model. The fewer ordinal items on the Conners 4–Short resulted in reduced model complexity, which allowed the comparisons to be tested. Results examining MI across the U.S. (N = 2,838 for Parent; N = 2,563 for Teacher; and N = 1,357 for Self-Report) and Canada (N = 376 for Parent, N = 301 for Teacher, and N = 199 for Self-Report) are found in Tables 11.68 to 11.70. Across all forms, there was no statistically significant reduction in fit indices (e.g., Satorra-Bentler χ² test p > .05). The CFI increased by greater than .01 for the threshold versus loading comparison in the Impairment & Functional Outcome Scales models for each form. This change was not a decrease in model fit, which might indicate differences in groups, but rather an improvement in fit. Therefore, the MI analysis supports measurement invariance. These results were consistent across Content Scales and Impairment & Functional Outcome Scales. As the Conners 4–Short meets the most stringent level of invariance tested in terms of country of residence, these results support generalizability of the Conners 4–Short between U.S. and Canadian youth regarding factor structure, loadings, thresholds, and intercepts.

Next, differential test functioning was evaluated with respect to country of residence. Results of the DTF analyses for all Conners 4–Short forms are presented in Table 11.71. The results revealed trivial effect sizes, as measured by ETSSD. Trivial differences were found (i.e., effect sizes at or below |.06| across all forms), lending further support to the generalizability of the Conners 4–Short across U.S. and Canadian populations.

To examine group differences between countries, a subsample of the youth from the U.S. portion of the Normative Samples were selected at random to match a sample of Canadian youth from the Normative Sample in terms of gender, PEL (for Parent and Self-Report only), language(s) spoken, clinical status, race/ethnicity, and age (N = 148 for Parent, N = 238 for Teacher, and N = 160 for Self-Report). Refer to Table F.43 and Table F.44 in appendix F for the demographic characteristics of the samples. The paired samples of Americans and Canadians were then compared for significant differences between mean scores on the Conners 4–Short.

Results of the ANOVAs and descriptive statistics for each scale are presented in Tables 11.72 to 11.74. Across all rater forms and all scales, there were no statistically significant effects of country of residence. Ratings of youth from the U.S. and Canada resulted in very similar mean scores as described by Cohen’s d; all effect sizes were negligible to small (Cohen’s d = 0.00 to |0.23|). These results indicate that country of residence (specifically, U.S. vs. Canada) had no significant effect on the Conners 4–Short scale scores, and along with the results of MI and DTF analyses demonstrating equivalence, there is evidence that the Conners 4–Short is appropriate for use with American and Canadian populations alike.

Parental Education Level

Parental education level (PEL) can sometimes be considered a proxy for or a contributing factor to one’s socioeconomic status (SES), with SES being a characteristic upon which fairness can unduly vary. It was expected that the constructs measured on the Conners 4–Short would be independent of influence from PEL. To test this hypothesis and ensure generalizability of scores from the Conners 4–Short scales, individuals in the Parent and Self-Report samples reported the PEL of the rated youth from one of five options: No high school diploma (PEL 1), High school diploma/GED (PEL 2), Some college or associate’s degree (PEL 3), Bachelor’s degree (PEL 4), or Graduate or professional degree (PEL 5). Within the Normative Samples, the proportion of individuals in each of these groups matched recent U.S. and Canadian census values (more information can be found in Parental Education Level in chapter 7, Standardization). MI and DTF analyses require binary variables, therefore PEL was re-categorized into two groups comprising parents with and without post-secondary education (i.e., Group 1 consists of PEL 1 and PEL 2: N = 855 for Parent and N = 530 for Self-Report; Group 2 consists of PEL 3, PEL 4, and PEL 5: N = 2,385 for Parent and N = 1,057 for Self-Report).

First, differences in the factor structure based on PEL were evaluated with MI. With more stringent models tested at each level, neither the Conners 4–Short Parent nor the Conners 4–Short Self-Report displayed meaningful deterioration in model fit (see Tables 11.75 to 11.76). For Parent, some comparisons were significant using the Satorra-Bentler χ² test (e.g., the loading versus intercept model comparison of the Content Scales; p < .001); however, the indicators must be considered together, and no other model fit statistics indicated meaningful change. Therefore, the observed change in model fit is minor and not meaningful, such that invariance between the PEL groups on the construct assessed by the Conners 4–Short can reasonably be assumed. These results support the invariance of the Conners 4–Short across factor structure, thresholds, loadings, and intercepts between youth whose parents do not have post-secondary education and youth whose parents do have post-secondary education, meeting the first-step criteria for establishing its unbiased and generalizable use with these populations.

Next, differences in scale functioning for the two broad PEL groups were explored with DTF. The effect sizes from DTF analyses are summarized in Table 11.77. Results from both Parent and Self-Report show trivial differences between PEL groups (e.g., the largest effect size was |.06|, classified as a negligible effect). The lack of differential functioning of the Conners 4–Short scales between the different PEL groups is further evidence of the test’s equivalence across demographic subgroups. Taken together, both MI and DTF analyses indicate that the Conners 4–Short can generalize across levels of parental education, as there are no meaningful differences between PEL groups in the measurement properties of the test.

Next, the five PEL groups were compared in terms of the observed mean score group differences. These group differences were analyzed using the entire Normative Samples. PEL was compared via a series of ANCOVAs, with covariates to statistically control for the effects of other demographic factors (i.e., language(s) spoken, race/ethnicity, clinical status, age group, and gender).

Results of the ANCOVAs for Parent are provided in Table 11.78a, with effect sizes of the pairwise comparisons between group estimated marginal means provided in Table 11.78b. Corresponding results for Self-Report are provided in Tables 11.79a and 11.79b. No significant differences between PEL groups were observed for either form, and effect sizes were negligible across both forms (partial η² = .00 to .01).

In line with expectations, PEL does not appear to have a meaningful influence on Conners 4–Short scores, as evidenced by the lack of statistically and practically significant group differences in mean scores. Therefore, these results, along with the MI and DTF findings that upheld the scales’ invariance with respect to PEL, support the unbiased use of the Conners 4–Short for youth with parents with varied educational backgrounds.

< Back

Next >