Conners 4 Manual Chapter 11: Reliability

Reliability

view all chapter tables | print this section

• Internal Consistency
• Standard Error of Measurement
• Test Information
• Test-Retest Reliability
• Inter-Rater Reliability

Reliability refers to the amount of measurement error, which is determined by examining the consistency of measurements obtained across different administrations or parts of the instrument (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014). Multiple indicators of reliability are provided for scores from the Conners 4–Short, including internal consistency, test information, test-retest reliability, and inter-rater reliability. See chapter 8, Reliability, for a detailed description of each type of reliability discussed in this section.

Internal Consistency

Internal consistency estimates for the Conners 4–Short are presented in Tables 11.13 to 11.18 for the Normative and ADHD Reference Samples (see chapter 7, Standardization, for a description of the Normative and ADHD Reference Samples; see Internal Consistency in chapter 8, Reliability, for a more in-depth description of the coefficients in the analyses).

The reliability coefficients presented in Tables 11.13 to 11.18 indicate that the Conners 4–Short meets or exceeds guidelines for internal consistency for all age groups. Across all age groups and genders in the Normative Samples, the median omega of the Conners 4–Short scales was .92 (ranging from .81 to .97) for Parent, .93 (ranging from .87 to .97) for Teacher, and .85 (ranging from .74 to .93) for Self-Report. For the ADHD Reference Samples, the median omega was .90 (ranging from .84 to .97) for Parent, .94 (ranging from .82 to .97) for Teacher, and .83 (ranging from .65 to .89) for Self-Report. In summary, multiple metrics indicate that the Conners 4–Short scale scores provide consistent and reliable estimates of the constructs being measured.

Standard Error of Measurement

Internal consistency values provide a measure of a test’s reliability, but these values also have practical applications through the calculation of the standard error of measurement (SEM), a statistic that quantifies the amount of error in the obtained scores. SEM values were calculated for the Conners 4–Short using the SD of the reference sample’s T-scores (i.e., SD = 10), along with the omega reliability coefficient. Overall, the median SEM was 2.79 for the Parent, 2.58 for Teacher, and 3.89 for Self-Report in the Normative Samples, and 3.11 for Parent, 2.54 for Teacher, and 4.15 for Self-Report in the ADHD Reference Samples (see Table 11.19 to 11.20). The low values reported here indicate a very small standard error of measurement, or very little error in the estimated true scores.

Practical applications of SEM values for the Conners 4–Short include the calculation of confidence intervals (see chapter 4, Interpretation, for a description of how confidence intervals can be used in the interpretation of results). For the Conners 4–Short, 90% and 95% confidence interval levels were calculated based on the SEM for each scale (note that confidence intervals for the Normative Samples are automatically included in the digital reports; see Scoring Options in chapter 3, Scoring and Reports; see also appendix B for confidence intervals based on SEM for the Normative and ADHD Reference Samples).

Test Information

An analysis of test information was conducted on the Total Sample (i.e., all General Population and all clinical cases combined; see Tables 6.5 and 6.6 in chapter 6, Development, for demographic characteristics of this sample), to provide maximum information in estimating these functions via the mirt package in R (Chalmers, 2012). As seen in Figure 11.3, the peak of the curve for most scales is at approximately 2 SD above the mean; the area under the test information functions (or curves) is wide, such that precision is fairly consistent from average levels of the construct (i.e., when theta, on the x-axis, is at 0) to 2 to 3 SD above the mean, depending on the scale. For Parent and Teacher forms, most scales are at or above 10, which indicates high precision of measurement, or reliability, for each scale. Results on the Self-Report form were more moderate, consistent with the full-length Conners 4 (see Test Information in chapter 8, Reliability). The scales for Self-Report had values at about 5 for most scales, which represents adequate reliability, with higher precision on the Inattention/Executive Dysfunction scale.

Figure 11.3. Test Information Functions by Scale: Conners 4–Short

(a) Inattention/Executive Dysfunction	(b) Hyperactivity
(c) Impulsivity	(d) Emotional Dysregulation
(e) Schoolwork	(f) Peer Interactions
(g) Family Life

Test-Retest Reliability

The test-retest reliability of the Conners 4–Short was assessed by computing the correlation of scores obtained on two separate administrations over a 2- to 4-week interval (14 to 30 days) within a subset of youth from the General Population sample (N = 81 for Parent, N = 61 for Teacher, and N = 68 for Self-Report; refer to Table F.1 and Table F.2 in appendix F for demographic characteristics of the test-retest samples). Measures with stable scores are expected to have high correlations, indicating little change in scores from one administration to the next.

The obtained correlations, as well as those corrected for variability, and descriptive statistics are provided in Tables 11.21 to 11.23. Corrected correlations ranged from .83 to .94 for Parent, .78 to .90 for Teacher, and .52 to .78 for Self-Report (all p < .001). Overall, the results provide evidence that the Conners 4–Short scale scores have moderate to very strong test-retest reliability. The stable nature of the scores, as demonstrated by the test-retest reliability coefficients, provide confidence that changes observed in a youth’s score over time are more likely due to a true change in the symptoms or impairments, as opposed to imprecise measurement.

The stability of the Conners 4–Short standardized scores was further evaluated by calculating the differences between each individual’s Time 1 and Time 2 ratings. If scores increased or decreased by greater than, or equal to, 10 points (i.e., 1 SD or greater), this change is considered notable. Tables 11.24 to 11.26 present the percentage of the sample with increases and decreases in scores, with most comparisons showing differences of fewer than 10 points. Tables 11.24 to 11.26 also present the mean differences, as well as the differences in SDs, between the ratings from Time 1 and Time 2 (positive differences indicate that scores increased at Time 2, while negative differences indicate that scores decreased at Time 2). The differences from Time 1 to Time 2 showed only slight increases or decreases in scores for Parent, Teacher, and Self-Report (mean differences ranged from -2.1 to 0.4 points across all forms), indicating consistency in responses across the time interval. Additionally, the differences between the SDs were quite small (ranging from -1.7 to 1.6 across all forms), showing a similar dispersion of scores from Time 1 to Time 2. Taken together, the strong correlations between administrations, the proportion of the sample with minimal change in their scores, and the small mean differences in scores across a short time interval demonstrate the reliability of scale scores for the Conners 4–Short across administrations.

Inter-Rater Reliability

Inter-rater reliability refers to the degree of agreement between two raters who are describing the same youth. Two inter-rater studies were conducted with the Conners 4–Short scale scores: (a) two raters of the same rater type rated the same youth (i.e., two parents or two teachers), and (b) two raters of different rater types rated the same youth (i.e., parent/teacher, parent/self-report, or teacher/self-report).

Inter-Rater Reliability Study 1. In the first inter-rater study for the Conners 4–Short scale scores, the dyads were comprised of two Parents (N = 68) and two Teachers (N = 34) rating the same youth (refer to Table F.3 and Table F.4 in appendix F for the demographic characteristics of the inter-rater samples).

The obtained inter-rater reliability coefficients, as well as those corrected for range restriction, are provided in Tables 11.27 and 11.28. All Conners 4–Short Parent scale scores showed strong to very strong inter-rater agreement (corrected correlations ranged from r = .68 to .81, all p < .001), and all Teacher scales showed weak-to-moderate inter-rater agreement (corrected correlations ranged from r  = .21 to .54). Tables 11.27 and 11.28 also shows the means, medians, and SDs for each rater, highlighting the alignment of average scores between the two raters. The pattern of results for Parent are certainly indicative of agreement and consistency; however, the modest association for Teacher is a reminder of the value of multiple informants and multiple sources of information when conducting a comprehensive assessment.

The consistency between raters was further evaluated by calculating the difference between scores for parent and teacher dyads in the inter-rater samples. Tables 11.29 and 11.30 present the percentage of the sample displaying differences greater than, or equal to, 10 points (i.e., 1 SD or greater), as well as the absolute mean differences (i.e., the difference between ratings from pairs of parents or teachers averaged across the samples). The results suggest that scores on the Conners 4–Short have high levels of consistency for parent dyads (scores were less than 1 SD apart for 77.9% to 86.8% of the sample). Scores demonstrated moderate consistency for teacher dyads (scores were less than 1 SD apart for 55.9% to 70.6% of this sample). The absolute mean and SD differences were also small (i.e., on average, there were small differences in mean scores between the raters, and there was similar dispersion in scores), providing further evidence for inter-rater consistency, given the high degree of agreement between the same types of raters.

Inter-Rater Reliability Study 2. In the second inter-rater study, comparisons were made across different types of raters. As the Conners 4–Short Parent, Teacher and Self-Report all measure similar constructs, similarity in scores across different types of raters, as well as between observer ratings and self-reported ratings, would provide evidence of the reliability of the test scores. Although some degree of similarity was expected between informants, it was nonetheless expected that a certain degree of incongruence would exist between the ratings from different informants (i.e., the correlations should be small to moderate in size), because various informants may have different opinions about, or different experiences with, the youth’s behavior, and because they see the youth in different contexts.

For the Conners 4–Short, correlation coefficients (Pearson’s r; LeBreton & Senter, 2008) were calculated between the following pairs of raters: (a) parent and teacher, (b) parent and self-report, and (c) teacher and self-report. To examine these relationships, parents, teachers, and youth provided ratings of the same youth (N = 62; all ratings completed within a 30-day period). All youth in this sample had a clinical diagnosis (refer to Table F.5 in appendix F for demographic characteristics of the rated youth and raters). The obtained correlation coefficients between different rater types, as well as those corrected for range restrictions (Bryant & Gokhale, 1972), are provided in Table 11.31 (Parent/Teacher), Table 11.32, (Parent/Self-Report), and Table 11.33 (Teacher/Self-Report).

Results were similar to the results found on the full-length Conners 4. The Parent/Teacher sample demonstrated weak to moderate agreement (corrected correlations ranged from r = .24 to .53). For comparisons between Parent and Self-Report ratings, corrected correlations demonstrated very weak to moderate agreement (corrected correlations ranged from r = .04 to .35), and very weak to weak agreement for comparisons between the Teacher and Self-Report ratings (corrected correlations ranged from r = .03 to .21). This range in agreement between raters may be due to a variety of reasons, one of which is setting differences. Youth can report on their own behaviors across multiple settings, whereas parents and teachers can only report on those behaviors observed in specific environments, such as home and school.

In addition, the congruence of scores was analyzed by examining the percentage of the samples with similar ratings. These results are summarized in Tables 11.34 to 11.36. As seen in Table 11.34, it was a frequent occurrence in the Parent/Teacher sample for ratings of the youth to differ by less than 1 SD (ranging from 53.2% to 71.0% of the sample, across all scales). Notably, parents provided slightly higher ratings than teachers (as evidenced by the small negative mean differences across all scales).

A similar pattern can be seen in the comparisons between Parent and Self-Report ratings, as presented in Table 11.35. Parents and youth tend to provide moderately similar ratings, yielding scores that are within 1 SD of each other for 48.4% to 61.3% of this sample. Additionally, the differences between their scores are modest, ranging from -0.5 to -4.9 points apart, with higher ratings coming from the parent raters. Similarly, for the Teacher/Self-Report comparisons, aligned ratings that are within 1 SD of each other were observed for 50.0% to 62.9% of this sample (see Table 11.36), with slightly higher scores coming from the self-reported ratings.

Overall, these findings demonstrate differences between the ratings of parents, teachers, and youth, with parents typically yielding the highest scores of the three rater types. Parents may have a unique opportunity for insight into their child that teachers may lack, thereby providing a different perspective. These findings demonstrate the varying viewpoints of raters with various types of relationships to the youth being rated. The raters may be drawing upon their experiences with the individual being rated in dissimilar contexts, which may lead to different responses between rater types. Results serve to highlight the importance of examining information from multiple sources when conducting a comprehensive assessment.

< Back

Next >