Inter-rater reliability refers to the degree of agreement between two raters who are rating the same individual. Estimates of inter-rater reliability help describe levels of consistency between raters, typically indexed by a correlation coefficient like Pearson’s r (LeBreton & Senter, 2008). Two inter-rater studies were conducted with the CAARS 2: (a) two Observers of the same rater type (e.g., two friends) rated the same individual; and (b) two different types of raters (e.g., self-report/observer, or parent/friend) rated the same individual.

Study 1: Two Observers, Same Rater Type. In the first inter-rater study, dyads of Observers (N = 29) completed the CAARS 2 about the same individual. The dyads were comprised of either two relatives (79.3%; most frequently included a parent and a sibling, two different parents, or two different children) or two friends (20.7%). Raters were paired based on similarity in the setting in which they could observe the rated individual (e.g., siblings, parents, and children may have similar exposure to home life, while friends may see the individual in social settings primarily). Ratings were obtained from the dyads on two separate administrations (all ratings completed within a 30-day period). Demographic characteristics of these samples are provided in appendix J.

Scores on the CAARS 2 scales for the paired raters were compared via Pearson’s correlations, ranging from -1 to 1, with higher values indicating greater consistency or agreement between raters. The obtained inter-rater reliability coefficients and those corrected for range variation (Bryant & Gokhale, 1972), as well as the means, medians and standard deviations for each dyad, are provided in Table 8.7. Results of the inter-rater study were indicative of moderate to strong levels of consistency within rater dyads across all CAARS 2 scales (i.e., corrected r ranged from .40 to .70). This pattern of results indicates that different raters can provide different perspectives and is a reminder of the value of obtaining ratings from multiple raters and evaluating multiple sources of information when conducting a comprehensive assessment.

Study 2: Different Types of Raters. In the second inter-rater study, comparisons were made across different types of raters, including comparison of Self-Report and Observer, as well as comparison of different types of Observers (e.g., spouse and friend). As the CAARS 2 Self-Report and Observer both measure the same constructs, similarity in scores across the different types of Observer raters, as well as between Observer ratings and self-reported ratings, would provide additional evidence of the reliability of the test scores. Although some degree of similarity in ratings is expected between informants, given that they are rating the same individual on the same constructs, it is nonetheless expected that there will be a certain degree of incongruence between their ratings, because different informants see the individual in different contexts and may have different perceptions of or experiences with the individual’s behavior.

For the CAARS 2, correlation coefficients (Pearson’s r, LeBreton & Senter, 2008) were calculated between scores for the following pairs of raters: (a) Observer and Self-Report (N = 211), and (b) two Observers with differing relations to the individual (N = 47; all ratings completed within a 30-day period). Dyads with two observers were required to represent different relationship types for Study 2 (e.g., a spouse/romantic partner and a friend, but not two friends). Refer to appendix J for the demographic characteristics of the raters and the individual being rated.

The obtained correlation coefficients between different rater types, as well as those corrected for range variation (Bryant & Gokhale, 1972), are provided in Table 8.8 (Self-Report/Observer) and Table 8.9 (Observer/Observer). The corrected correlations were weak to moderate in the Self-Report/Observer dyads (median r = .44, ranging from r = .37 to .46, p < .001), as well in the Observer/Observer dyads (median r = .45, ranging from r = .26 to .58, p < .001).

Overall, these findings suggest modest differences between the ratings of Self-Report and Observer dyads, with self-reported ratings typically yielding higher scores. Additionally, Observers of different types had a similar pattern of results in that relationships were modest, such that there was some agreement while also providing unique insight. The low-to-moderate agreement observed between Observer and Self-Report results may be due to a variety of factors, such as differences in setting (e.g., consistency of behaviors can vary), in level of insight (e.g., ability to observe internal processes, and/or one’s own self-awareness), and in the nature of the relationship (e.g., willingness to disclose may vary for one’s parent as opposed to one’s spouse). Discrepancies between raters’ scores, as seen in these studies, emphasize the importance of consulting multiple sources to capture unique information that can reveal relevant differences. Self-report captures an individual’s unique insight into their own experiences and behaviors that may not be readily evident to observers, especially for behaviors that may be more inwardly felt than outwardly expressed (e.g., subjective feelings of restlessness). Two observers with different roles in the individual’s life may be drawing upon their experiences with the individual being rated in dissimilar contexts or from different vantage points, which may lead to slightly different responses between rater types and thereby reduce the similarity of their ratings. Results serve to highlight the importance of examining information from multiple sources when conducting a comprehensive assessment.