• Samples
• Analyses & Results

To address the goals defined during the conceptualization phase, a pilot study was conducted. Items for extended and novel content, as well as improved alignment to the DSM Symptom Criteria, were evaluated in the pilot study samples. The pilot study was administered digitally, either through local or remote administration. Parent/guardian and teacher raters completed the test from an emailed link. Youth raters completed the test digitally, in person, and with supervision from a trained administrator to provide support for the youth if needed. The parent/guardian, teacher, and youth (who were 18 years old) raters provided informed consent to participate in this data collection effort. For youth raters who were 17 years of age or younger, informed consent was obtained from their parent/guardian, and assent was obtained from the youths themselves. All participants received small monetary compensation for their participation. The items and scales were then subjected to empirical evaluation.

Samples

The pilot version of the Conners 4 was administered to the parents and teachers of youth (and to the youth themselves) where the youth did not have a mental health diagnosis (described herein as the General Population samples). The pilot version was also administered to raters of youth (and youth themselves) where the youth did have a mental health diagnosis (described herein as the Clinical samples).

The General Population samples consisted of roughly equal representation by age and by gender of the youth being rated, and representation of broad categories of race/ethnicity, geographic region, and parental educational level (PEL) that closely matched U.S. census proportions at the time of recruitment (within approximately 2% of the proportions derived from the American Community Survey 2016; U.S. Census Bureau, 2017). A description of the rated youth in the pilot samples is presented in Table 6.1 and a description of the raters is provided in Table 6.2. These samples (N = 518 for Parent, N = 490 for Teacher, N = 388 for Self-Report) were drawn from the general population and individuals were eligible to participate if the youth being rated had never been diagnosed with a psychological or psychiatric disorder. For the Parent sample, 56.2% reported being the child’s biological mother, 35.7% reported being the child’s biological father, and the remaining 8.1% identified as non-biological parents, legal guardians, and other relatives. For the Teacher sample, all of the raters reported that they knew the student for longer than one month, 97.0% interacted with them weekly or more often, and all teachers knew the student well (either moderately or very well). Teachers reported providing instruction for a wide range of classes, including math, English, history, and music.

The Clinical samples (N = 349 for Parent, N = 242 for Teacher, N = 245 for Self-Report) gathered ratings of youth with a confirmed mental health disorder (see Table 6.3 for a detailed description of the youth being rated, and Table 6.4 for information about the raters). Clinical cases were recruited directly through clinicians, who provided details of the diagnoses to confirm their status. In particular, the sample predominantly featured youth with a diagnosis of one of the three presentations of ADHD (45.9% for Parent, 44.7% for Teacher, and 40.7% for Self-Report). Other clinical disorders, such as Oppositional Defiant Disorder, Conduct Disorder, Generalized Anxiety Disorder, and Major Depressive Disorder, were also represented in this sample. Many youth in the Clinical sample reported having more than one mental health diagnosis. Youth in the Clinical sample varied with respect to whether they were currently taking medication to treat their disorders. For those who reported taking medication, they were not asked to stop taking their medication. For the Parent sample, 81.9% were biological mothers, 8.0% were biological fathers, and the remaining 9.8% included non-biological parents, legal guardians, and other relatives. For the Teacher sample, all raters reported that they knew the student for longer than one month, 97.9% interacted with them weekly or more, and 96.3% knew the student well (either moderately or very well). Teachers reported providing instruction for a wide range of classes, including math, English, history, and music, as well as special education and resource room (i.e., a special education program in a separate setting or a smaller designated room).

Analyses & Results

The items from the Content Scales, Impairment & Functional Outcome Scales, and DSM Symptom Scales were evaluated using both classical test theory (CTT) and item-response theory (IRT) frameworks. Items were first reviewed via (a) response frequencies by clinical and demographic group, to ensure all response options were endorsed and to understand the distribution of responses; (b) inter-item correlations to ensure relatedness of content; and (c) group differences to ensure the items appear to behave in hypothesized ways (e.g., individuals diagnosed with ADHD Combined Presentation were expected to endorse the Hyperactivity and Impulsivity items more so than individuals with ADHD Inattentive Presentation).

Items were then subjected to exploratory factor analyses (EFA) to understand the latent structure of the item pool across all three rater forms (i.e., Parent, Teacher, and Self-Report) and for purposes of dimension and item reduction. The tentative factor structure was incorporated into further analyses for the inspection of internal consistency, inter-scale correlations, and item-to-scale correlations.

The results, including the structure of the scales and item-level analyses, were weighed alongside clinical relevance to determine the set of best performing items that would be carried through to the next phase of development. Items were flagged for removal if

• non-negligible demographic differences were observed;
• items did not relate well to their intended factor;
• expected differences between clinical groups were not notable, or did not adhere to hypotheses (e.g., if an item written to measure inattentive behavior was rated similarly both by individuals with ADHD and those with no clinical diagnosis); or
• either item-to-scale-total correlations or item communalities were low (i.e., an indicator of the unique contribution of an individual item to its scale; low values defined as < .20; Kline, 1986).

Additionally, IRT analyses were used to ensure items were kept that provided sufficient precision of measurement across the latent trait continuum with emphasis and preference given to items that provided measurement precision at higher trait levels. Poorly performing items were either removed or revised in instances where a small change could be applied. In addition, the Critical & Indicator Items, along with items from the different validity metrics (i.e., positive impression, negative impression, and ADHD SVT) were all retained without changes for further investigation at the next stage of development. At the completion of the pilot phase, there were 269 items on the Conners 4 Parent, 256 items on the Conners 4 Teacher, and 290 items on the Conners 4 Self-Report.