Conners 4 Manual Chapter 12: Development

Development

view all chapter tables | print this section

• Samples
• Analyses & Results

The goal of the development of the Conners 4–ADHD Index was to identify items from the full-length Conners 4 that best differentiate between youth from the general population (i.e., without a clinical diagnosis) and those with ADHD. This index is designed to be brief and efficient, and accordingly, the number of items on the scale was targeted to be quite small. An ADHD Index was included in the Conners 3rd Edition™ (Conners 3™; Conners, 2008); however, given the new item pool developed for the Conners 4 (see Item Development in chapter 6, Development), a new index was needed to identify the best-performing items in the revised item pool.

Samples

To begin the item selection phase, the non-clinical General Population samples and the ADHD Reference Samples of the Conners 4 Parent, Teacher, and Self-Report were included for analyses (see Standardization in chapter 6, Development, for the Total Samples from which these subsets were drawn). The General Population samples were substantially larger than the ADHD Reference Samples; however, unbalanced samples can result in problems when classifying groups because prediction will naturally favor the larger group. Therefore, samples were balanced by matching each youth in the ADHD Reference Sample with a corresponding youth from the General Population matched in terms of age, gender, race/ethnicity, and parental education level (PEL; matched for Parent and Self-Report samples only). Coarsened matching was used to facilitate the creation of comparable samples (e.g., PEL groups were collapsed such that No High School Diploma [PEL 1] and High School Diploma [PEL 2] were combined when seeking matching samples). The matched pairs of youth from the General Population and youth diagnosed with ADHD were then split into a training sample and a validation sample, at a 70/30 ratio, respectively. The demographic characteristics of the rated youth and the parent and teacher raters are found in Tables 12.1 to 12.5.

Analyses and Results

A gradient-boosting machine learning model (GBM; Friedman, 2001) was employed to select the items that were best able to classify General Population and ADHD groups. Machine learning models are increasingly used in psychology in developing and testing assessment tools (Bleidorn & Hopwood, 2019; Dwyer et al., 2018). GBM creates a series of decision trees using the variables in the model. A decision tree organizes the variables (i.e., items) into a series of steps that best classify individuals in the sample. In models created by GBM, trees are built sequentially so that each new tree learns from errors in the previous tree to improve classification accuracy. GBM models have some advantages over more traditional approaches to classification and prediction such as regression models, including the ability to handle a large number of items. This advantage was important for the Conners 4–ADHD Index, as there were over 60 items included in each candidate item pool for Parent, Teacher, and Self-Report (i.e., items from the Conners 4 Content Scales related to ADHD Symptoms, including Inattention/Executive Dysfunction, Hyperactivity, Impulsivity, and Emotional Dysregulation, as well as the Impairment & Functional Outcome Scales, were considered for inclusion when creating this index).

Once the items for inclusion were selected and the samples were created, the first step in the analysis was to tune the model in the training sample. In model tuning, the parameters that maximized the performance of the model and the balance between complexity and accuracy were selected. A model that is overly complex will have good prediction in the sample but will not generalize well to other samples; alternatively, a model that is too simple will have poor accuracy (Miller et al., 2016). K-fold cross-validation was employed in the model tuning step to select the model with the optimal complexity. K-fold cross-validation was conducted in a series of five steps: (a) the training sample was split into five subsets; (b) the model was trained on all but one subset; (c) the model was evaluated in the final subset and the prediction error was calculated; (d) the previous steps were repeated five times (k = 5), rotating which subset of the sample was treated as a final subset in each round; and (e) the prediction error across the five iterations was averaged. Following these steps, the model with the lowest cross-validation prediction error value was selected as the final model. Model tuning was completed via the caret package in R v.6.0-86 (Kuhn, 2008), and the final tuning parameters were used for the GBM, via the gbm package v.2.1.7 in R (Ridgeway, 2006).

The results from the GBM analysis with the training sample were used to select the variables to be considered for the index. During the initial stages of development, two separate models were employed to select the items for the Conners 4–ADHD Index: (a) an unweighted model to compare differentiation between the General Population and ADHD groups, and (b) a weighted model in which each ADHD presentation was weighted to be equally represented in the sample (e.g., although ADHD Combined was more prevalent in the sample, the proportion of individuals with ADHD Inattentive and ADHD Hyperactive/Impulsive presentations were statistically weighted to be equal to one another). The following classification accuracy statistics were used to help decide between models (see chapter 6, Development, for a description of each statistic): overall correct classification, sensitivity, specificity, positive predictive value, negative predictive value, and kappa. Classification accuracy assesses the ability of the model to accurately determine whether the youth’s scores more closely resemble the ADHD or General Population group. Results from the model are compared to the criterion, which, in this case, refers to an ADHD diagnosis. The weighted models were more successful in accurately classifying youth; therefore, the results presented in this section refer to the weighted models.

The top 10 best discriminating items were selected using the relative importance statistic from the GBM. The relative importance statistic indicates how important variables are, relative to others in the model (note that there are no absolute guidelines when interpreting this statistic; instead, it is used to compare items to one another). A 12-item solution was also tested by adding 2 additional items to the 10-item solution. In instances where the 10-item solution did not include content from Hyperactivity or Impulsivity, then an item from those Content Scales with the highest relative importance statistic was selected to extend content coverage. Selected items for the 10- and 12-item versions created for Parent, Teacher, and Self Report are presented in Tables 12.6, 12.7, and 12.8, respectively.

Model tuning and the GBM analyses were repeated for the 10-item and 12-item solutions with the training sample. Classification accuracy results from the potential item subsets were compared to each other and were compared with results from a version that included all candidate items. Results (as presented in Table 12.9) revealed that the item subsets performed well relative to all items. The 12-item solutions for all rater forms were selected as they had better content coverage than the 10-item solutions and generally had slightly better classification statistics.

The next step in the development process was to test the 12-item solutions using the validation samples. Results of the classification accuracy for this analysis are presented in Table 12.10.

Finally, two different scoring approaches were compared: a sum score and a boosted score that doubled the weight of the most important items identified in the GBM. The top items were selected based on the variable importance score. As this importance score is relative (rather than a set criterion or cut-off value), the relative size of the values, and therefore the decrease from each item’s importance to the next item, was considered. Considering these results, a boosted score was tested where the weight of the most important items was doubled and then summed with the rest of the items. This scoring approach was compared to a standard raw sum score. The score that maximized the best balance between sensitivity and specificity was identified from the training sample and applied to the validation sample. Classification results of the different scoring approaches are presented in Table 12.11. Results revealed that for Parent, there was not enough of an improvement to warrant this method, as there were considerable trade-offs in the classification accuracy statistics. Therefore, a sum score was selected to compute the raw score, in which all items contribute equally. In contrast, for Teacher and Self-Report, results suggested that doubling the weight of the most important items (i.e., the top three items for Teacher, and top two items for Self-Report) resulted in an improvement in classification accuracy. This increase was most pronounced for the Self-Report, where the boosted score improved statistics to a larger degree than other forms. The final items selected are presented in Tables 12.12, 12.13, and 12.14 for Parent, Teacher, and Self-Report, respectively.

< Back

Next >