Manual

Conners 4 Manual

Chapter 12: Development


Development

view all chapter tables | print this section

The goal of the development of the Conners 4–ADHD Index was to identify items from the full-length Conners 4 that best differentiate between youth from the general population (i.e., without a clinical diagnosis) and those with ADHD. This index is designed to be brief and efficient, and accordingly, the number of items on the scale was targeted to be quite small. An ADHD Index was included in the Conners 3rd Edition™ (Conners 3™; Conners, 2008); however, given the new item pool developed for the Conners 4 (see Item Development in chapter 6, Development), a new index was needed to identify the best-performing items in the revised item pool.


Samples

To begin the item selection phase, the non-clinical General Population samples and the ADHD Reference Samples of the Conners 4 Parent, Teacher, and Self-Report were included for analyses (see Standardization in chapter 6, Development, for the Total Samples from which these subsets were drawn). The General Population samples were substantially larger than the ADHD Reference Samples; however, unbalanced samples can result in problems when classifying groups because prediction will naturally favor the larger group. Therefore, samples were balanced by matching each youth in the ADHD Reference Sample with a corresponding youth from the General Population matched in terms of age, gender, race/ethnicity, and parental education level (PEL; matched for Parent and Self-Report samples only). Coarsened matching was used to facilitate the creation of comparable samples (e.g., PEL groups were collapsed such that No High School Diploma [PEL 1] and High School Diploma [PEL 2] were combined when seeking matching samples). The matched pairs of youth from the General Population and youth diagnosed with ADHD were then split into a training sample and a validation sample, at a 70/30 ratio, respectively. The demographic characteristics of the rated youth and the parent and teacher raters are found in Tables 12.1 to 12.5.


Click to expand

Table 12.1. Demographic Characteristics of Rated Youth: Conners 4–ADHD Index Parent Training and Validation Samples

Demographic

Training

Validation

General Population

ADHD

General Population

ADHD

N

%

N

%

N

%

N

%

Gender

Male

266

68.0

265

67.8

96

60.4

98

61.3

Female

125

32.0

126

32.2

63

39.6

62

38.8

U.S. Race/Ethnicity

Hispanic

36

9.2

33

8.4

13

8.2

12

7.5

Asian

4

1.0

3

0.8

2

1.3

2

1.3

Black

14

3.6

13

3.3

4

2.5

4

2.5

White

277

70.8

269

68.8

114

71.7

113

70.6

Other

16

4.1

23

5.9

4

2.5

7

4.4

Canadian Race/Ethnicity

Not visible minority

37

9.5

47

12.0

18

11.3

20

12.5

Visible minority

7

1.8

3

0.8

4

2.5

2

1.3

Parental Education Level

No high school diploma

8

2.0

7

1.8

1

0.6

0

0.0

High school diploma/GED

35

9.0

32

8.2

18

11.3

17

10.6

Some college or associate’s degree

92

23.5

89

22.8

46

28.9

46

28.8

Bachelor’s degree

149

38.1

135

34.5

46

28.9

43

26.9

Graduate or professional degree

107

27.4

128

32.7

48

30.2

54

33.8

U.S. Region

Northeast

67

17.1

60

15.3

26

16.4

22

13.8

Midwest

85

21.7

97

24.8

24

15.1

40

25.0

South

136

34.8

142

36.3

65

40.9

55

34.4

West

59

15.1

42

10.7

22

13.8

21

13.1

Canadian Region

Central

27

6.9

27

6.9

12

7.5

13

8.1

East

3

0.8

7

1.8

3

1.9

2

1.3

West

14

3.6

16

4.1

7

4.4

7

4.4

Language(s) Spoken

English only

391

100.0

391

100.0

159

100.0

160

100.0

English & Non-English

0

0.0

0

0.0

0

0.0

0

0.0

Diagnosis

ADHD Inattentive

0

0.0

120

30.7

0

0.0

49

30.6

ADHD Combined

0

0.0

209

53.5

0

0.0

86

53.8

ADHD Hyperactive/Impulsive

0

0.0

62

15.9

0

0.0

25

15.6

Anxiety

0

0.0

82

21.0

0

0.0

29

18.1

Depression

0

0.0

19

4.9

0

0.0

7

4.4

Other Diagnoses

0

0.0

130

33.2

0

0.0

58

36.3

No Diagnosis

391

100.0

0

0.0

159

100.0

0

0.0

Age in years M (SD)

11.4 (3.0)

11.4 (3.0)

11.4 (3.0)

11.4 (3.0)

Total

391

100.0

391

100.0

159

100.0

160

100.0

Note. Anxiety includes Generalized Anxiety Disorder, Panic Disorder, and Social Anxiety Disorder. Depression includes Major Depressive Disorder, Major Depressive Episode, and Persistent Depressive Disorder. Other Diagnoses include less frequently reported co-occurring diagnoses, such as Autism Spectrum Disorder and Learning Disorders. The sum of diagnoses is greater than the total N because youth with co-occurring diagnoses count towards more than one diagnostic group.



Click to expand

Table 12.3. Demographic Characteristics of Rated Youth: Conners 4–ADHD Index Teacher Training and Validation Samples

Demographic

Training

Validation

General Population

ADHD

General Population

ADHD

N

%

N

%

N

%

N

%

Gender

Male

62

26.5

62

26.5

55

63.2

55

63.2

Female

172

73.5

172

73.5

32

36.8

32

36.8

U.S. Race/Ethnicity

Hispanic

14

6.0

14

6.0

5

5.7

5

5.7

Asian

0

0.0

0

0.0

1

1.1

1

1.1

Black

22

9.4

22

9.4

14

16.1

14

16.1

White

156

66.7

173

73.9

52

59.8

60

69.0

Other

9

3.8

13

5.6

4

4.6

3

3.4

Canadian Race/Ethnicity

Not visible minority

28

12.0

12

5.1

10

11.5

2

2.3

Visible minority

5

2.1

0

0.0

1

1.1

2

2.3

U.S. Region

Northeast

44

18.8

84

35.9

18

20.7

26

29.9

Midwest

43

18.4

44

18.8

18

20.7

21

24.1

South

92

39.3

81

34.6

27

31.0

29

33.3

West

22

9.4

13

5.6

13

14.9

7

8.0

Canadian Region

East

4

1.7

0

0.0

3

3.4

0

0.0

Central

21

9.0

9

3.8

5

5.7

3

3.4

West

8

3.4

3

1.3

3

3.4

1

1.1

Language(s) Spoken

English only

234

100.0

234

100.0

87

100.0

87

100.0

English & Non-English

0

0.0

0

0.0

0

0.0

0

0.0

Diagnosis

ADHD Inattentive

0

0.0

71

30.3

0

0.0

26

29.9

ADHD Combined

0

0.0

114

48.7

0

0.0

45

51.7

ADHD Hyperactive/Impulsive

0

0.0

49

20.9

0

0.0

16

18.4

Anxiety

0

0.0

38

16.2

0

0.0

13

14.9

Depression

0

0.0

11

4.7

0

0.0

5

5.7

Other Diagnoses

0

0.0

93

39.7

0

0.0

36

41.4

No Diagnosis

234

100.0

0

0.0

87

100.0

0

0.0

Age in years M (SD)

11.8 (3.2)

11.8 (3.1)

11.6 (3.1)

11.6 (3.1)

Total

234

100.0

234

100.0

87

100.0

87

100.0

Note. Anxiety includes Generalized Anxiety Disorder, Panic Disorder, and Social Anxiety Disorder. Depression includes Major Depressive Disorder, Major Depressive Episode, and Persistent Depressive Disorder. Other Diagnoses include less frequently reported co-occurring diagnoses, such as Autism Spectrum Disorder and Learning Disorders. The sum of diagnoses is greater than the total N because youth with co-occurring diagnoses count towards more than one diagnostic group.



Click to expand

Table 12.5. Demographic Characteristics: Conners 4–ADHD Index Self-Report Training and Validation Samples

Demographic

Training

Validation

General Population

ADHD

General Population

ADHD

N

%

N

%

N

%

N

%

Gender

Male

107

62.9

106

62.4

37

62.7

37

62.7

Female

63

37.1

63

37.1

22

37.3

22

37.3

Other

0

0.0

1

0.6

0

0.0

0

0.0

U.S. Race/Ethnicity

Hispanic

20

11.8

11

6.5

7

11.9

5

8.5

Asian

1

0.6

1

0.6

0

0.0

0

0.0

Black

10

5.9

12

7.1

1

1.7

3

5.1

White

109

64.1

123

72.4

37

62.7

43

72.9

Other

6

3.5

12

7.1

0

0.0

4

6.8

Canadian Race/Ethnicity

Not visible minority

23

13.5

11

6.5

10

16.9

4

6.8

Visible minority

1

0.6

0

0.0

4

6.8

0

0.0

Parental Education Level

No high school diploma

5

2.9

5

2.9

0

0.0

0

0.0

High school diploma/GED

29

17.1

30

17.6

11

18.6

11

18.6

Some college or associate’s degree

43

25.3

34

20.0

15

25.4

13

22.0

Bachelor’s degree

52

30.6

55

32.4

18

30.5

17

28.8

Graduate or professional degree

41

24.1

46

27.1

15

25.4

18

30.5

U.S. Region

Northeast

29

17.1

35

20.6

7

11.9

7

11.9

Midwest

32

18.8

46

27.1

16

27.1

13

22.0

South

50

29.4

69

40.6

14

23.7

30

50.8

West

35

20.6

9

5.3

8

13.6

5

8.5

Canadian Region

Central

17

10.0

8

4.7

12

20.3

1

1.7

East

3

1.8

0

0.0

1

1.7

1

1.7

West

4

2.4

3

1.8

1

1.7

2

3.4

Language(s) Spoken

English only

170

100.0

170

100.0

59

100.0

59

100.0

English & Non-English

0

0.0

0

0.0

0

0.0

0

0.0

Diagnosis

ADHD Inattentive

0

0.0

63

37.1

0

0.0

22

37.3

ADHD Combined

0

0.0

79

46.5

0

0.0

29

49.2

ADHD Hyperactive/Impulsive

0

0.0

28

16.5

0

0.0

8

13.6

Anxiety

0

0.0

31

18.2

0

0.0

5

8.5

Depression

0

0.0

10

5.9

0

0.0

1

1.7

Other Diagnoses

0

0.0

51

30.0

0

0.0

19

32.2

No Diagnosis

170

100.0

0

0.0

59

100.0

0

0.0

Age in years M (SD)

12.6 (2.8)

12.6 (2.8)

12.6 (2.7)

12.5 (2.7)

Total

170

100.0

170

100.0

59

100.0

59

100.0

Note. Anxiety includes Generalized Anxiety Disorder, Panic Disorder, and Social Anxiety Disorder. Depression includes Major Depressive Disorder, Major Depressive Episode, and Persistent Depressive Disorder. Other Diagnoses include less frequently reported co-occurring diagnoses, such as Autism Spectrum Disorder and Learning Disorders. The sum of diagnoses is greater than the total N because youth with co-occurring diagnoses count towards more than one diagnostic group.


Analyses and Results

A gradient-boosting machine learning model (GBM; Friedman, 2001) was employed to select the items that were best able to classify General Population and ADHD groups. Machine learning models are increasingly used in psychology in developing and testing assessment tools (Bleidorn & Hopwood, 2019; Dwyer et al., 2018). GBM creates a series of decision trees using the variables in the model. A decision tree organizes the variables (i.e., items) into a series of steps that best classify individuals in the sample. In models created by GBM, trees are built sequentially so that each new tree learns from errors in the previous tree to improve classification accuracy. GBM models have some advantages over more traditional approaches to classification and prediction such as regression models, including the ability to handle a large number of items. This advantage was important for the Conners 4–ADHD Index, as there were over 60 items included in each candidate item pool for Parent, Teacher, and Self-Report (i.e., items from the Conners 4 Content Scales related to ADHD Symptoms, including Inattention/Executive Dysfunction, Hyperactivity, Impulsivity, and Emotional Dysregulation, as well as the Impairment & Functional Outcome Scales, were considered for inclusion when creating this index).

Once the items for inclusion were selected and the samples were created, the first step in the analysis was to tune the model in the training sample. In model tuning, the parameters that maximized the performance of the model and the balance between complexity and accuracy were selected. A model that is overly complex will have good prediction in the sample but will not generalize well to other samples; alternatively, a model that is too simple will have poor accuracy (Miller et al., 2016). K-fold cross-validation was employed in the model tuning step to select the model with the optimal complexity. K-fold cross-validation was conducted in a series of five steps: (a) the training sample was split into five subsets; (b) the model was trained on all but one subset; (c) the model was evaluated in the final subset and the prediction error was calculated; (d) the previous steps were repeated five times (k = 5), rotating which subset of the sample was treated as a final subset in each round; and (e) the prediction error across the five iterations was averaged. Following these steps, the model with the lowest cross-validation prediction error value was selected as the final model. Model tuning was completed via the caret package in R v.6.0-86 (Kuhn, 2008), and the final tuning parameters were used for the GBM, via the gbm package v.2.1.7 in R (Ridgeway, 2006).

The results from the GBM analysis with the training sample were used to select the variables to be considered for the index. During the initial stages of development, two separate models were employed to select the items for the Conners 4–ADHD Index: (a) an unweighted model to compare differentiation between the General Population and ADHD groups, and (b) a weighted model in which each ADHD presentation was weighted to be equally represented in the sample (e.g., although ADHD Combined was more prevalent in the sample, the proportion of individuals with ADHD Inattentive and ADHD Hyperactive/Impulsive presentations were statistically weighted to be equal to one another). The following classification accuracy statistics were used to help decide between models (see chapter 6, Development, for a description of each statistic): overall correct classification, sensitivity, specificity, positive predictive value, negative predictive value, and kappa. Classification accuracy assesses the ability of the model to accurately determine whether the youth’s scores more closely resemble the ADHD or General Population group. Results from the model are compared to the criterion, which, in this case, refers to an ADHD diagnosis. The weighted models were more successful in accurately classifying youth; therefore, the results presented in this section refer to the weighted models.

The top 10 best discriminating items were selected using the relative importance statistic from the GBM. The relative importance statistic indicates how important variables are, relative to others in the model (note that there are no absolute guidelines when interpreting this statistic; instead, it is used to compare items to one another). A 12-item solution was also tested by adding 2 additional items to the 10-item solution. In instances where the 10-item solution did not include content from Hyperactivity or Impulsivity, then an item from those Content Scales with the highest relative importance statistic was selected to extend content coverage. Selected items for the 10- and 12-item versions created for Parent, Teacher, and Self Report are presented in Tables 12.6, 12.7, and 12.8, respectively.





Model tuning and the GBM analyses were repeated for the 10-item and 12-item solutions with the training sample. Classification accuracy results from the potential item subsets were compared to each other and were compared with results from a version that included all candidate items. Results (as presented in Table 12.9) revealed that the item subsets performed well relative to all items. The 12-item solutions for all rater forms were selected as they had better content coverage than the 10-item solutions and generally had slightly better classification statistics.



The next step in the development process was to test the 12-item solutions using the validation samples. Results of the classification accuracy for this analysis are presented in Table 12.10.



Finally, two different scoring approaches were compared: a sum score and a boosted score that doubled the weight of the most important items identified in the GBM. The top items were selected based on the variable importance score. As this importance score is relative (rather than a set criterion or cut-off value), the relative size of the values, and therefore the decrease from each item’s importance to the next item, was considered. Considering these results, a boosted score was tested where the weight of the most important items was doubled and then summed with the rest of the items. This scoring approach was compared to a standard raw sum score. The score that maximized the best balance between sensitivity and specificity was identified from the training sample and applied to the validation sample. Classification results of the different scoring approaches are presented in Table 12.11. Results revealed that for Parent, there was not enough of an improvement to warrant this method, as there were considerable trade-offs in the classification accuracy statistics. Therefore, a sum score was selected to compute the raw score, in which all items contribute equally. In contrast, for Teacher and Self-Report, results suggested that doubling the weight of the most important items (i.e., the top three items for Teacher, and top two items for Self-Report) resulted in an improvement in classification accuracy. This increase was most pronounced for the Self-Report, where the boosted score improved statistics to a larger degree than other forms. The final items selected are presented in Tables 12.12, 12.13, and 12.14 for Parent, Teacher, and Self-Report, respectively.







< Back Next >