• Internal Consistency and Standard Error of Measurement
• Test Information
• Test-Retest Reliability
• Inter-Rater Reliability

Reliability refers to the amount of measurement error, which is determined by examining the consistency of measurements obtained across different administrations or parts of the instrument (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014). Multiple indicators of reliability are provided for scores from the CAARS 2–Short, including internal consistency, test information, test-retest reliability, and inter-rater reliability. See chapter 8, Reliability, for a detailed description of each type of reliability discussed in this section.

Internal Consistency and Standard Error of Measurement

Internal consistency estimates for the CAARS 2–Short are presented in Tables 11.10 to 11.11 for the Normative and ADHD Reference Samples (see chapter 7, Standardization, for a description of the Normative and ADHD Reference Samples; see chapter 8, Reliability, for a more in-depth description of the coefficients in this section).

The reliability coefficients presented in Tables 11.10 to 11.11 indicate that the CAARS 2–Short meets or exceeds guidelines for internal consistency for all age groups. Across all age groups and genders in the Normative Sample, the median omega of the CAARS 2–Short scale scores was .91 (ranging from .80 to .97) for Self-Report, and .93 (ranging from .78 to .98) for Observer. For the ADHD Reference Sample, the median omega was .89 (ranging from .84 to .92) for Self-Report, and .92 (ranging from .87 to .95) for the Observer. In summary, multiple metrics indicate that the CAARS 2–Short scale scores provide consistent and reliable estimates of the constructs being measured.

Internal consistency values provide a measure of a test’s reliability, but these values also have practical applications through the calculation of the standard error of measurement (SEM). SEM values were calculated for the CAARS 2–Short using the SD of the reference sample’s T-scores (i.e., SD = 10), along with the omega reliability coefficient. Overall, the median SEM was 3.07 for Self-Report and 2.69 for Observer in the Normative Sample, and 3.28 for Self-Report and 2.76 for Observer in the ADHD Reference Sample (see Tables 11.10 and 11.11). The low values reported here indicate a very small standard error of measurement, or very little error in the estimated true scores.

Practical applications of SEM for the CAARS 2–Short include the calculation of confidence intervals (see chapter 4, Interpretation, for a description of how confidence intervals can be used in the interpretation of results). For the CAARS 2–Short, 90% and 95% confidence intervals were calculated based on the SEM for each scale. (Note that confidence intervals for the Principal Normative Sample are automatically included in the digital reports; see Scoring Options in chapter 3, Administration and Scoring. See appendix C for confidence intervals based on SEM for the ADHD Reference Samples.)

Test Information

An analysis of test information was conducted on the Total Sample (i.e., all General Population and all clinical cases combined; see chapter 6, Development, for a description of this sample), to provide maximum information in estimating these functions via the mirt package in R (Chalmers, 2012). As seen in Figure 11.3, the Self-Report and Observer forms of the CAARS 2–Short demonstrate high precision across the range of the trait being measured for the Inattention/Executive Dysfunction scale (note that this scale is presented as an example; figures for all the other scales are provided in appendix L). The peak of the curve for the scales is approximately 2 SD above the mean, and the area under the test information curve is wide, such that precision is sufficiently high for a wide range of scores along the continuum of each construct. The high degree of precision and small degree of error provide strong evidence for the reliability of the CAARS 2–Short scales.

Click to expand

Figure 11.3. Test Information Functions for Inattention/Executive Dysfunction: CAARS 2–Short

Test-Retest Reliability

The test-retest reliability of the CAARS 2–Short was assessed by computing the correlation of scores obtained on two separate administrations over a 2- to 4-week interval (14 to 30 days) within a subset of individuals from the General Population sample (N = 88 for Self-Report and N = 61 for Observer; refer to appendix J for demographic characteristics of the test-retest samples). Measures of relatively stable constructs (like ADHD) are expected to have high correlations, indicating little change in scores from one administration to the next.

The obtained correlations, as well as those corrected for variation (Bryant & Gokhale, 1972), are provided in Tables 11.14 and 11.15. Corrected correlations ranged from .78 to .95 for Self-Report and .76 to .90 for Observer (all p < .001). As further evidence of score stability over the course of the retest period, mean scores from each time point are closely aligned (as seen in the very small Cohen’s d values). The stable nature of the scores, as demonstrated by the test-retest reliability coefficients, provides assurance that changes observed in CAARS 2–Short scores over time are due to true changes, as opposed to imprecise measurement.

Inter-Rater Reliability

Inter-rater reliability refers to the degree of agreement between two raters who are rating the same individual. Two inter-rater studies were conducted with the CAARS 2: (a) two Observers of the same rater type rated the same individual (e.g., two friends), and (b) two different types of raters rated the same individual (e.g., self-report/observer, or parent/friend).

Study 1: Two Observers, Same Rater Type. In the first inter-rater study, dyads of Observers (N = 29) completed the CAARS 2–Short about the same individual (refer to appendix J for the demographic characteristics of the inter-rater samples).

The obtained inter-rater correlation coefficients, as well as those corrected for range restriction, are provided in Table 11.16. Results of the inter-rater study were indicative of moderate to strong levels of consistency within rater dyads across all CAARS 2–Short scales (corrected r ranged from .40 to .69, all p < .001). Table 11.16 also shows the means, medians, and standard deviations for each rater, highlighting the alignment of average scores between the two raters. This pattern of results indicates that different raters can provide different perspectives and is a reminder of the value of obtaining ratings from multiple raters and evaluating multiple sources of information when conducting a comprehensive assessment.

Study 2: Different Type of Raters. In the second inter-rater study, comparisons were made across different types of raters, including comparison of Self-Report and Observer report, as well as comparison of different types of Observers (e.g., spouse and friend). As the CAARS 2–Short Self-Report and Observer both measure similar constructs, similarity in scores across the different types of Observers, as well as between Observer ratings and self-reported ratings, would provide additional evidence of the reliability of the test scores. Although some degree of similarity in ratings is expected between informants, given that they are rating the same individual on the same constructs, it is nonetheless expected that there will be a certain degree of incongruence between their ratings, because different informants see the individual in different contexts and may have different perceptions of or experiences with the individual’s behavior.

For the CAARS 2–Short, correlation coefficients (with corrections for range restriction) were calculated between scores for the following pairs of raters: (a) Self-Report and Observer (N = 211), and (b) two observers with differing relations to the individual (N = 47; all ratings completed within a 30-day period). Dyads with two observers were required to represent different relationship types for Study 2 (e.g., a spouse/romantic partner and a friend, but not two friends). Refer to appendix J for the demographic characteristics of the raters and the individual being rated.

Results were similar to the results found in the full-length CAARS 2 (see Inter-Rater Reliability in chapter 8, Reliability) and are provided in Table 11.17 (Self-Report/Observer) and Table 11.18 (Observer/Observer). The corrected correlations were weak to moderate in the Self-Report/Observer dyads (ranging from r = .35 to .44, p < .001), as well in the Observer/Observer dyads (ranging from r = .25 to .48, p < .001).

Overall, these findings suggest moderate differences between the ratings of Self-Report and Observers, with Self-Report typically yielding the highest scores. A number of factors may reduce the level of agreement between different raters, including setting differences, level of insight, and nature of the relationship. Individuals also have unique insight into their own experiences and behaviors that may not be readily evident to observers, especially for behaviors that may be more inwardly felt than outwardly expressed. Results serve to highlight the importance of examining information from multiple sources when conducting a comprehensive assessment.