Conners 4 Manual Chapter 12: Reliability

Reliability

view all chapter tables | print this section

• Internal Consistency
• Standard Error of Measurement
• Test Information
• Test-Retest Reliability
• Inter-Rater Reliability

Multiple indicators of reliability are provided for scores from the Conners 4–ADHD Index, including internal consistency, test information, test-retest reliability, and inter-rater reliability. A thorough explanation of reliability, as well as the reliability evidence for the full-length Conners 4, can be found in chapter 8, Reliability. Note that the Conners 4–ADHD Index raw scores are used primarily in this section to permit correlational analyses.

Internal Consistency

Internal consistency estimates for the Conners 4–ADHD Index are presented in Tables 12.15 and 12.16 for the Normative and ADHD Reference Samples (see chapter 7, Standardization, for a description of these samples; see chapter 8, Reliability, for an in-depth description of internal consistency and the coefficients reported in this section).

The reliability coefficients presented in Tables 12.15 and 12.16 exceed guidelines for high reliability and indicate that the Conners 4–ADHD Index has excellent internal consistency on all forms across all age groups. Across all age groups and genders in the Normative Samples, the median omega of the Conners 4–ADHD Index for Parent was .93 (ranging from .91 to .94), .90 for Teacher (ranging from .86 to .92), and .87 for Self-Report (ranging from .85 to .90). For the ADHD Reference Samples, the median omega for Parent was .89 (ranging from .87 to .91), .87 for Teacher (ranging from .85 to .89), and .83 for Self-Report (ranging from .81 to .85). In summary, multiple metrics indicate that the Conners 4–ADHD Index provides cohesive, consistent, and reliable estimates of key predictive ADHD symptoms.

Standard Error of Measurement

The standard error of measurement (SEM) is a statistic that quantifies the amount of error in the obtained raw scores (more details are provided in chapter 8, Reliability). Overall, the median SEM was 2.67 for Parent, 3.20 for Teacher, and 3.57 for Self-Report in the Conners 4 Normative Samples, and 3.28 for Parent, 3.61 for Teacher, and 4.12 for Self-Report in the ADHD Reference Samples (see Table 12.17 for SEM values for the Normative and ADHD Reference Samples). The low values reported here indicate a small standard error of measurement, or very little error in the estimated true scores for the Conners 4–ADHD Index, and therefore a high degree of reliability.

Test Information

Test information was explored in the Total Samples for Parent, Teacher, and Self-Report (for details about the Total Samples, see Tables 6.5 and 6.6 in chapter 6, Development; for details about test information, see chapter 8, Reliability). As seen in Figure 12.1, precision of measurement for the Conners 4–ADHD Index is excellent for both the Parent and Teacher forms. Values exceeded 10 for average levels of the trait (i.e., denoted as 0 along the x-axis) to past 2 standard deviations (SD) above the mean, which reflect clinical levels of the trait. For Self-Report, values stay at about 10 for average levels of the trait to 2 SD above the mean. Values of this magnitude meet guidelines for test information, suggesting sufficient precision of measurement for the Conners 4–ADHD Index across all rater forms and supporting the reliability of this index.

Figure 12.1. Test Information: Conners 4–ADHD Index

Test-Retest Reliability

The test-retest reliability of the Conners 4–ADHD Index was assessed by computing the correlation of raw scores from individuals who completed the test on two separate occasions. It was expected that the index would have stable scores that do not vary significantly upon re-administration. Ratings of youth from the general population (i.e., youth who did not have a clinical diagnosis) completed the Conners 4–ADHD Index with a 2- to 4-week time interval in between administrations (N = 81 for Parent; N = 61 for Teacher; N = 68 for Self-Report; refer to Table F.1 and Table F.2 in appendix F for demographic characteristics of the test-retest samples).

The correlation coefficients and descriptive statistics of the raw scores from the test-retest samples are provided in Table 12.18. Results demonstrated that the Conners 4–ADHD Index has strong (r = .66 for Self-Report, p < .001) to very strong (r = .91 for Parent, r = .89 for Teacher, p < .001) evidence of test-retest reliability. As further evidence of score stability over the retest period, mean scores from each administration were closely aligned, as seen in Table 12.19. The stability of the Conners 4–ADHD Index probability score was further evaluated by calculating the difference between the percentage of the samples within each probability score category range from Time 1 to Time 2. Overall, fewer than 10%, 7%, and 11% of the Parent, Teacher, and Self-Report samples, respectively, showed a shift in probability score category; that is, scores that were classified as “Low” at Time 1 were overwhelmingly likely to also be classified as “Low” at Time 2. The stable nature of the scores, as demonstrated by the test-retest reliability coefficient, provides confidence that any change observed in scores over time is more likely due to a true change, as opposed to imprecise measurement.

Inter-Rater Reliability

Inter-rater reliability refers to the degree of agreement between two raters who are rating the same youth. Estimates of inter-rater reliability help describe levels of consistency between raters, typically indexed by a correlation coefficient (in this case, Pearson’s r; LeBreton & Senter, 2008). Consistency between raw scores from various raters was explored for the Conners–ADHD Index Parent, Teacher, and Self-Report. Two inter-rater studies were conducted with the Conners 4–ADHD Index: (a) two raters of the same rater type rated the same youth (i.e., two parents or two teachers), and (b) two raters of different rater types rated the same youth (i.e., parent/teacher, parent/self-report, or teacher/self-report).

Inter-Rater Reliability Study 1. In the first inter-rater study for the Conners 4–ADHD Index, the dyads were comprised of two Parents (N = 68) and two Teachers (N = 34) rating the same youth (refer to Table F.3 and Table F.4 in appendix F for the demographic characteristics of the inter-rater samples).

The obtained inter-rater reliability coefficients are provided in Table 12.20. Strong (Teacher r = .62, p < .001) to very strong (Parent r = .83, p < .001) inter-rater agreement was found for the Conners 4–ADHD Index. Table 12.20 also shows the descriptive statistics for each rater, highlighting the similarity of average scores between the two raters through the very small differences between mean raw scores. The consistency of the Conners 4–ADHD Index probability score was further evaluated by calculating the difference between the percentage of the sample within each Probability Score category range from Parent 1 to Parent 2 or Teacher 1 to Teacher 2. There was a negligible effect size of the difference between raters (Cliff’s d = -.01 for Parent and .03 for Teacher; see Table 12.21), indicating that different raters typically provided ratings that led to scores within the same category, as seen through the similar percentages of each sample with scores in each category. These findings support the reliability of the Conners 4–ADHD Index.

Inter-Rater Reliability Study 2. In the second inter-rater study, comparisons were made across different types of raters. As the Conners 4–ADHD Index Parent, Teacher and Self-Report all measure similar constructs, similarity in scores across different types of raters, as well as between observer ratings and self-reported ratings, would provide evidence of the reliability of the test scores. Although some degree of similarity was expected between informants, it was nonetheless expected that a certain degree of incongruence would exist between the ratings from different informants (i.e., the correlations should be small to moderate in size), because various informants may have different opinions about, or different experiences with, the youth’s behavior, and because they see the youth in different contexts.

For the Conners 4–ADHD Index, correlation coefficients were calculated between the following pairs of raters: (a) parent and teacher, (b) parent and self-report, and (c) teacher and self-report. To examine these relationships, parents, teachers, and youth provided ratings of the same youth (N = 62; all ratings completed within a 30-day period). All youth in this sample had a clinical diagnosis (refer to Table F.5 in appendix F for demographic characteristics of the rated youth and raters). The obtained correlation coefficients between different rater types are provided in Table 12.22.

Results were similar to the results found on the full-length Conners 4 and the Conners 4–Short. The correlations between the Conners 4–ADHD Index raw probability scores ranged from very weak (Teacher/Self-Report r = .11) to weak (Parent/Self-Report r = .21) to moderate (Parent/Teacher r = .48, p < .001). In addition, there was a small effect size of the difference between raters (Cliff’s d ranges from -.10 to .27; see Table 12.23). These results indicate that although raters largely provided ratings that led to similar categories of the probability score, some differences may arise, as evidenced by the slight differences seen in the proportions of each rater sample in each score category in Table 12.23. This range in agreement between raters may be due to a variety of reasons, one of which is setting differences. Youth can report on their own behaviors across multiple settings, whereas parents and teachers can only report on those behaviors observed in specific environments, such as home and school. There is clearly immense value in obtaining ratings from multiple sources when administering the Conners 4–ADHD Index.

< Back

Next >