• Samples
• Analyses and Results

The creation of the CAARS 2–ADHD Index began with the identification of items from the full-length CAARS 2 that best differentiated between individuals from the general population (i.e., without a clinical diagnosis) and those with ADHD. The ADHD Index was designed to comprise a limited number of the most discriminating items, so as to provide a brief and efficient tool that could be used independently (e.g., for screening purposes) or in conjunction with other CAARS 2 scores. Development of the CAARS 2–ADHD Index began upon the completion of the CAARS 2 standardization phase (see Item Development in chapter 6, Development) and relied on the same samples and item pool to identify the best-performing items.

Samples

To begin the item selection phase, the Total Samples of the CAARS 2 (see Standardization Phase in chapter 6, Development) were divided into two subsets: individuals without a clinical diagnosis (non-clinical General Population samples) and the ADHD Reference Samples. Note that individuals diagnosed with ADHD-Hyperactive/Impulsive Presentation were excluded from the analyses due to their underrepresentation in the ADHD Reference Sample (N = 10 for Self-Report; N = 9 for Observer). The General Population samples were substantially larger than the ADHD samples; however, unbalanced samples can result in problems when classifying groups (as prediction will naturally favor the larger group). Therefore, samples were balanced by matching individuals in the ADHD Reference sample with a corresponding individual from the General Population in terms of age, gender, race/ethnicity, and education level (EL). Coarsened matching was used to facilitate the creation of comparable samples (e.g., EL groups were collapsed such that No High School Diploma [EL 1] and High School Diploma [EL 2] were combined when seeking matching samples). The matched pairs of individuals from the General Population and individuals diagnosed with ADHD were then split into a training sample and a validation sample, at a 70/30 ratio, respectively. The demographic characteristics for the rated individuals and the raters are found in Tables 12.1 to 12.3.

Analyses and Results

A gradient boosting machine learning model (GBM; Friedman, 2001) was employed to select items from the CAARS 2 Content Scales and Impairment & Functional Outcome Items that were best able to classify individuals who belong to the General Population versus ADHD group. Machine learning models are increasingly used to develop and test psychological assessment tools (Bleidorn & Hopwood, 2019; Dwyer et al., 2018). GBM creates a series of decision trees using the variables in the model. A decision tree organizes the variables (i.e., items) into a series of steps that best classify individuals in the sample. In models created by GBM, trees are built sequentially so that each new tree learns from errors in the previous tree to improve classification accuracy. GBM models have some advantages over regression models and other more traditional approaches to classification and prediction, including the ability to handle a large number of items.

Once the item pool was selected and the samples were created, the first step in the analysis was to tune the model in the training sample. Parameters that maximized the performance of the model while striking a balance between complexity and accuracy were selected. A model that is overly complex will have good prediction in the sample but will not generalize well to other samples; alternately, a model that is too simple will have poor accuracy (Miller et al., 2016). K-fold cross-validation was employed in the model-tuning step to select the model with the optimal complexity. K-fold cross-validation was conducted in a series of five steps: (a) the training sample was split into five subsets; (b) the model was trained on all but one subset; (c) the model was evaluated in the final subset and the prediction error was calculated; (d) the previous steps were repeated five times (k = 5), rotating which subset of the sample was treated as a final subset in each round; and (e) the prediction error across the five iterations was averaged. Following these steps, the model with the lowest cross-validation prediction error value was selected as the final model. Model tuning was completed via the caret package in R v.6.0-86 (Kuhn, 2008), and the final tuning parameters were used for the GBM, via the gbm package v.2.1.7 in R (Ridgeway, 2006).

The results from GBM analysis with the training sample were used to select variables to be considered for the ADHD Index. The following classification accuracy statistics were used to help decide between models (see chapter 6, Development, for a description of each statistic): overall correct classification, sensitivity, specificity, positive predictive value, negative predictive value, and kappa. As a global construct, classification accuracy assesses the ability of the model to accurately determine whether the individual’s scores more closely resemble those from the ADHD or General Population group. Results from the model are compared to the criterion, which, in this case, was an ADHD diagnosis.

The top 10 best discriminating items were selected using the relative importance statistic from the GBM. The relative importance statistic indicates how important variables are, relative to others in the model (note that there are no absolute guidelines when interpreting this statistic; instead, it is used to compare items to one another). A 12-item solution was also tested by expanding the 10-item solution. In instances where the 10-item solution did not include content from Hyperactivity or Impulsivity, then an item from those Content Scales with the highest relative importance statistic was selected to extend content coverage. Selected items for the 10- and 12-item versions are presented in Tables 12.4 and 12.5, respectively. Note that because the analyses were conducted independently for Self-Report and Observer ratings, the items selected are not identical (although they share overlapping content).

Model tuning and the GBM analyses were repeated for the 10-item and 12-item solutions with the training sample. Classification accuracy results from the potential item subsets were compared to each other and were compared with results from a version that included all candidate items. Results (as presented in Table 12.6) revealed that the item subsets performed well relative to all items. The 12-item solutions for both rater forms were preferred over the 10-item solutions as they had slightly better classification statistics.

The next step in the development process was to test the 10-item and 12-item solutions using the validation samples. Results of the classification accuracy for this analysis are presented in Table 12.7. It is notable that many of the items that emerged as top candidates for the ADHD Index (9 of 12 for Self-Report; 7 of 12 for Observer) were items that originated from the Inattention/Executive Dysfunction scale, as well as the fact that some of the same items were candidates across the two rater forms. The item content of the 12-item solution included an item from either the Hyperactivity (for Observer) or Impulsivity (for Self-Report) that was not found in the 10-item version. Considering most items identified by the GBM were Inattentive/Executive Dysfunction items, this additional breadth of content was deemed important, and the 12-item solution was retained.

Once the final CAARS 2–ADHD Index items were selected, the performance of the new ADHD Index was compared to the performance of the original CAARS ADHD Index to ensure that the new Index performed as well or better than the previous iteration. The raw summed scores of each Index were compared using Receiver Operating Characteristic (ROC) curves and area under the ROC curve (AUC). A ROC curve plots the performance of a measure at each possible cut-off score, or threshold. AUC is a widely used measurement of accuracy. It represents the total area under the AUC curve and is interpreted as the probability that an individual diagnosed with ADHD selected at random will have a higher score than a randomly selected individual from the general population. Values from .50 to .69 reflect poor accuracy, values between .70 to .90 reflect moderate accuracy, and values above .90 are highly accurate (Fischer et al., 2003). The ROC curves are plotted in Figures 12.1 and 12.2 below. Visual inspection of the ROC curves shows that the CAARS 2–ADHD Index covers a greater area under the curve than the original ADHD Index, thereby improving the likelihood of correct classification. Results between the two Indexes were contrasted using DeLong’s test (DeLong et al., 1988) with 5,000 bootstrapped samples using the validation sample. Results show the CAARS 2–ADHD Index performed statistically significantly better than the original CAARS ADHD Index for both Self-Report (Z = -3.55, p < 0.01; original ADHD Index AUC = .89, CAARS 2–ADHD Index AUC = .95) and Observer (Z = -2.86, p < 0.01, original ADHD Index AUC = .84, CAARS 2–ADHD Index AUC = .94). While both the original and new ADHD Index exceed guidelines for acceptable accuracy, these results show that the CAARS 2–ADHD Index has high classification accuracy, validate that the items selected are performing well, and demonstrate that the current Index represents an improvement over the original.

Click to expand

Figure 12.1. ROC Curve: Comparison between Original and CAARS-2 ADHD Index Self-Report

Click to expand

Figure 12.2. ROC Curve: Comparison between Original and CAARS-2 ADHD Index Observer