Predicting Health Utilities for Children With AutismSpectrum DisordersNalin Payakachat, J. Mick Tilford, Karen A. Kuhlthau, N. Job van Exel, Erica Kovacs, Jayne Bellando,Jeffrey M. Pyne, and Werner B.F. Brouwer

Comparative effectiveness of interventions for children with autism spectrum disorders (ASDs) that incorporates costs islacking due to the scarcity of information on health utility scores or preference-weighted outcomes typically used forcalculating quality-adjusted life years (QALYs). This study created algorithms for mapping clinical and behavioralmeasures for children with ASDs to health utility scores. The algorithms could be useful for estimating the value ofdifferent interventions and treatments used in the care of children with ASDs. Participants were recruited from twoAutism Treatment Network sites. Health utility data based on the Health Utilities Index Mark 3 (HUI3) for the child wereobtained from the primary caregiver (proxy-reported) through a survey (N = 224). During the initial clinic visit,proxy-reported measures of the Child Behavior Checklist, Vineland II Adaptive Behavior Scales, and the Pediatric Qualityof Life Inventory 4.0 (start measures) were obtained and then merged with the survey data. Nine mapping algorithmswere developed using the HUI3 scores as dependent variables in ordinary least squares regressions along with the startmeasures, the Autism Diagnostic Observation Schedule, to measure severity, child age, and cognitive ability as indepen-dent predictors. In-sample cross-validation was conducted to evaluate predictive accuracy. Multiple imputation tech-niques were used for missing data. The average age for children with ASDs in this study was 8.4 (standard deviation = 3.5)years. Almost half of the children (47%) had cognitive impairment (IQ ≤ 70). Total scores for all of the outcome measureswere significantly associated with the HUI3 score. The algorithms can be applied to clinical studies containing startmeasures of children with ASDs to predict QALYs gained from interventions. Autism Res 2014, ••: ••–••. © 2014International Society for Autism Research, Wiley Periodicals, Inc.

Keywords: mapping; predictive algorithms; equating measure; autism; health utilities; clinical measure; behavioral measure;quality of life measure

Introduction

The prevalence of children with autism spectrum disor-ders (ASDs) has increased significantly, with recent esti-mates suggesting that 1 in 88 to 1 in 50 children are nowaffected [Blumberg et al., 2013; Centers for DiseaseControl and Prevention, 2012]. ASDs are neuro-developmental disorders beginning in childhood andaffecting outcomes into adulthood [Howlin, Goode,Hutton, & Rutter, 2004]. ASDs are characterized, invarying degrees, by social interaction difficulties, commu-nication challenges, and a tendency to engage in repeti-tive behaviors [Lord & Bishop, 2010; Lord & McGee,2001]. With the publication of the Diagnostic and Statis-tical Manual of Mental Disorders, 5th edition, the three

categories of ASD symptoms (social difficulties, commu-nication impairments, and repetitive/restricted behav-iors) have been merged into two categories (social-communication impairment and repetitive/restrictedbehaviors), and the former diagnostic subtypes have beenmerged into one umbrella diagnosis of ASD [AmericanPsychiatric Association, 2013].

While ASD is usually a lifelong condition, both chil-dren and adults benefit from interventions that reduceASD symptoms and improve skills and abilities (such aslanguage, cognitive, and adaptive skills) [Dawson et al.,2010; Howlin, Magiati, & Charman, 2009; Warren et al.,2011]. Because of the increased prevalence of childrenwith ASDs, resources for understanding the comparativeeffectiveness of alternative interventions have increased

From the Division of Pharmaceutical Evaluation and Policy, University of Arkansas for Medical Sciences, Little Rock, Arkansas (N.P., J.M.T.); Departmentof Health Policy and Management, University of Arkansas for Medical Sciences, Little Rock, Arkansas (J.M.T.); Department of Pediatrics, University ofArkansas for Medical Sciences, Little Rock, Arkansas (J.M.T., J.B.); Department of Pediatrics, Harvard Medical School, Boston, Massachusetts (K.A.K.);Center for Adolescent Health Policy, Massachusetts General Hospital, Boston, Massachusetts (K.A.K.); Department of Health Policy and Management,Erasmus University, Rotterdam, The Netherlands (N.J.v.E., W.B.F.B.); Department of Psychiatry, Columbia University Medical Center, New York, NewYork (E.K.); Center for Mental Healthcare and Outcomes Research, Central Arkansas Veterans Healthcare System, Little Rock, Arkansas (J.M.P.); PsychiatricResearch Institute, University of Arkansas for Medical Sciences, Little Rock, Arkansas (J.M.P.)

Received December 18, 2013; accepted for publication August 1, 2014Address for correspondence and reprints: Nalin Payakachat, Division of Pharmaceutical Evaluation and Policy, University of Arkansas for Medical

Sciences, 4301 W. Markham St., #522, Little Rock, AR 72223. E-mail: npayakachat@uams.eduPublished online in Wiley Online Library (wileyonlinelibrary.com)DOI: 10.1002/aur.1409© 2014 International Society for Autism Research, Wiley Periodicals, Inc.

RESEARCH ARTICLE

INSAR 1Autism Research ••: ••–••, 2014

commensurate with the burden of the condition on thepopulation. Understanding the value of interventions forchildren with ASDs, however, has the potential to trans-late clinical comparative effectiveness research findingsinto sustained practice [Glasgow & Steiner, 2012;Glasgow et al., 2012, 2013].

For example, there is clear evidence that children withASDs benefit from intensive behavioral services [Dawsonet al., 2010; Rothenberg & Samson, 2009; Warren et al.,2011]. The National Research Council guidelines fortreating children with autism recommend providing25 hr a week of intensive therapy [Lord & Bishop, 2010;Lord & McGee, 2001; Myers et al., 2007; The NationalAutism Center, 2009; Scottish Intercollegiate GuidelinesNetwork, 2007]. Yet payers raise concerns about the costsof intensive behavioral interventions and who shouldpay for them [Rothenberg & Samson, 2009]. Informationon the value of intensive behavioral services based onformal cost-effectiveness analyses could inform guide-lines for providing services to children with ASDs, but arelacking. Indeed, implementation scientists recognize thatone of the “greatest opportunities” to translate clinicalcomparative effectiveness evidence into practice or policyis an understanding of costs in relation to outcomesbecause it provides a clear rationale for decision makers toact [Glasgow et al., 2013].

The lack of information on cost-effectiveness seemspartly due to the scarcity of information in this popula-tion on health utility scores (alternatively referred to aspreference-weighted outcomes), which are necessary forcalculating quality-adjusted life years (QALYs). The QALYis a measure that combines both health-related qualityand quantity of life into a single index [Neumann,Goldie, & Weinstein, 2000]. Economic evaluations thatuse the cost per QALY metric theoretically permit com-parison of interventions for ASD with other physical andmental health interventions in both child and adultpopulations [Greenberg & Neumann, 2011; Neumann &Greenberg, 2009; Neumann et al., 2000; Weinstein,Siegel, Gold, Kamlet, & Russell, 1996].

To generate QALY estimates for economic evaluation,researchers need evidence on health utilities. Typically,utilities are expressed on a scale ranging from 0 (the valueattached to the state “dead”) to 1 (representing the valueof the state of full health). Negative health utility scoresare possible since there may be health states that peopleconsider to be worse than dead (e.g., vegetative states).However, health utility scores cannot exceed 1 as there isno health state better than perfect health [Gold, Siegel,Russell, & Weinstein, 1996]. Health utility scores can beestimated directly using methods to express the value ofa health state like a visual analog scale, standard gamble,or a time trade-off method [Brazier, Ratcliffe, Salomon, &Tsuchiya, 2007]. Often, however, they are estimated indi-rectly using existing multi-attribute health classification

systems [Brazier et al., 2007], such as the Health UtilitiesIndex Mark 3 (HUI3) [Feeny et al., 2002], QWB-SA [Seiber,Groessl, Ganiats, & Kaplan, 2008], EQ-5D [The EuroQolGroup, 1990], or SF-6D [Brazier & Roberts, 2004]. Thesesystems are based on instruments to classify health statesdescribed by a number of health domains with pre-defined health utility scores that enable the translation ofall possible health states measured within the instrumentinto a health utility score. The basic idea behind theseinstruments is that they can be applied generally acrossconditions and populations to facilitate standardizedcomparisons, although much has been written about thecomparability of scores generated by different instru-ments [Lipscomb, Drummond, Fryback, Gold, & Revicki,2009]. To date, only a few studies have estimated healthutilities for children with ASDs [Petrou & Kupek, 2009;Petrou et al., 2010; Tilford et al., 2012], and no study hasestimated health utility gains associated with specifictreatments, such as intensive behavioral services. Thus,formal cost-effectiveness of services for children withASDs according to the cost per QALY metric as recom-mended by guidelines for conducting such evaluations inboth the United States and the United Kingdom [TheNational Institute for Health and Care Excellence, 2013;National Institute for Health and Clinical Excellence(NICE), 2008; Weinstein et al., 1996] is lacking despitetheir potential to inform discussions of value. FollowingBailey [2009], we believe that more discussion of thevalue of services for children with ASDs appears war-ranted [Bailey, 2009].

Because estimates of health utility gains associated withinterventions for children with ASDs are central todescribing cost-effectiveness, and data are lacking, thereis a need to generate estimates of health utility gains withnew methods. In the absence of health utility scores inclinical studies, determining value using cost-effectiveness analysis can be accomplished with an indi-rect approach, which is called “mapping” [Longworth &Rowen, 2011]. Mapping can be used to predict healthutilities based on clinical or behavioral data to estimateQALY gains [Longworth & Rowen, 2011]. Algorithms formapping clinical outcomes in terms of health utilitieshave been reported for a number of conditions [Crott &Briggs, 2010; Dakin, Petrou, Haggard, Benge, &Williamson, 2010; Goldsmith, Dyer, Buxton, & Sharples,2010; Longworth, Buxton, Sculpher, & Smith, 2005;Payakachat et al., 2009; Versteegh et al., 2012], but nosuch research has been developed specifically to predicthealth utilities for children with ASDs.

Mapping is defined as the development and use of analgorithm to predict health utility scores using data onother indicators or measures of health [Longworth &Rowen, 2011]. In the absence of direct elicitation ofhealth utility scores in clinical trials, a mathematicalmapping approach that explains relationships between

INSAR2 Payakachat et al./Mapping outcome measures to health utilities

health utility scores and clinical and behavioral measuresor other quality of life measures is a useful (althoughnecessarily second-best) alternative [The NationalInstitute for Health and Care Excellence, 2013; NICE,2008]. Predictive algorithms from the mapping approachcan also help identify the aspects of behavioral function-ing among children with ASDs that particularly impacthealth utility scores and the magnitude of this impact.The models can then be applied to data from existingclinical trials or other studies containing the predictivemeasures necessary to predict health utility scores. Thepredicted health utility scores can then be linked back todata collected within the original study and can be usedto calculate QALYs for use in economic evaluations oftreatments or interventions.

This study provides algorithms to map clinical, behav-ioral, and health-related quality of life (HRQL) measures,including the Child Behavior Checklist (CBCL),Vineland-II Adaptive Behavior Scales (Vineland-II), andPediatric Quality of Life Inventory (PedsQL) version 4.0,which are typically used in clinical studies of ASD treat-ments and interventions onto a health utility measure(the HUI3) using statistical association. The predictivealgorithms provided can be applied to clinical data gen-erated in comparative effectiveness evaluations of ASDtreatments to generate health utility scores. The findingsthus could be helpful in estimating value in terms ofcost-effectiveness of different ASD treatments, and tocompare estimates with treatments for other populations.

MethodsSample and Data Collection

The sampling frame for this study was children aged 4–17years old diagnosed with ASDs and their parents whoparticipated in two Autism Treatment Network sites (adevelopmental center in Little Rock, Arkansas, and anoutpatient psychiatric clinic at Columbia UniversityMedical Center in New York). Families of diagnosed chil-dren who had been seen at the clinics and previouslyagreed to be contacted about future studies as part of theirparticipation in the Autism Treatment Network registrywere identified as potential study subjects. A surveypackage was sent by mail to eligible families with a self-addressed envelope for return. Families that returned thesurvey, signed Health Insurance Portability and Account-ability Act forms, and consent/assent forms were pro-vided a $25 gift card for participating in the research.Approximately 10% of the families in the Little Rockclinic and 5% of the families in the Columbia clinic didnot agree to be contacted about future research studies.The study protocol was approved by the institutionalreview boards at Columbia University/New York StatePsychiatric Institute and the University of Arkansas forMedical Sciences.

The survey package contained instruments to measurehealth utility scores for the child with ASD and theprimary caregiver. An information sheet specified thatthe primary caregiver of the child should complete thesurvey marked “to be completed by the caregiver aboutthe child with ASD.” Data from the survey were thenlinked to the clinical information of children with ASDsobtained from the Autism Treatment Network to createthe final data file.

Outcome Measures

There are several generic, preference-based HQRL instru-ments that can be used to produce health utility scores;however, not all of them have been used nor are designedspecifically for children [Payakachat, Tilford, Kovacs, &Kuhlthau, 2012]. The HUI3 [Feeny et al., 2002] wasselected to serve as our primary outcome for measuringhealth utilities. We decided to use the primary caregiveras a proxy to report health for their child with ASDbecause the comprehension level for the HUI3 was fairlyhigh for children with ASDs; importantly, a significantproportion of children with ASDs have cognitive disabil-ity, in addition to developmental disability. The HUI3 hasbeen used previously in studies involving children in thesame age ranges with our current study [Buysse et al.,2008; Petrou & Kupek, 2009; Prince et al., 2010]. Previousresearch showed that HUI3 was able to differentiatehealth status between children with ASDs and unaffectedchildren. At the time this survey research was conducted,the EQ-5Y (EQ-5D Youth version) [Wille et al., 2010] andCHU9D (Child Health Utility 9D) [Stevens, 2010] werenot available for public use.

The HUI3 measures responses on eight domains ofhealth, namely vision, hearing, speech, ambulation, dex-terity, emotion, cognition, and pain. Not all of the healthdomains may be affected by ASDs, since the instrument isintended to be generic, that is to be able to capture healthutility associated with different conditions, in order to beable to produce comparable health utility scores in avariety of diseases. The HUI3 includes a multi-attributescoring algorithm associated with the domains to gener-ate a utility score based on subject responses. The result-ing health utility scores range from −0.36 (worst possiblehealth) to 1.0 (best possible health). Subject responses onthe eight domains of HUI3 were used to calculate HUI3utility scores using a multiplicative, multi-attribute utilityfunction [Feeny et al., 2002]. Differences or changes onHUI3 utility scores of ≥ 0.03 are considered to be clini-cally important [Luo, Johnson, & Coons, 2010].

Detailed information on patient and family demo-graphics, patient diagnosis, medical history, physical andneurological exam, clinical and behavioral measures,quality of life measures, and experience with care wereobtained from the Autism Treatment Network registry.

3Payakachat et al./Mapping outcome measures to health utilitiesINSAR

From the available measures, we selected relevant out-comes that could be mapped onto the HUI3 health utilityscores. The clinical and behavioral outcome measuresincluded the CBCL, Vineland-II, and PedsQL version 4.0,all of which were administered at the Autism TreatmentNetwork clinic baseline visit and continuing throughfollow-up. As many children had both baseline andfollow-up data, all outcomes used for analysis werechosen based on the closest time to survey completion.

CBCL. The CBCL is a standardized behavioral inventorythat covers a wide range of psychiatric symptoms andundesirable behaviors, and is based on parent report[Achenbach & Ruffle, 2000]. The CBCL assesses both inter-nalizing (i.e., anxious, depressive, and overcontrolled) andexternalizing (i.e., aggressive, hyperactive, noncompliant,and undercontrolled) behavior symptoms. The CBCL totalproblems score is a combined measure of externalizingand internalizing problems. CBCL scores are expressed asT scores (mean of 50, standard deviation [SD] = 10). Tscores of 60–69 are considered to be in a borderline clinicalrange, while T scores of 70 or above reflect clinicallysignificant behavioral problems in comparison to sameage/gender peers). The CBCL has been shown to be validand reliable in children with ASDs [Dekker, Nunn, & Koot,2002; Wallander, Dekker, & Koot, 2006].

Vineland-II. The Vineland-II, is a semi-structured,clinician-administered interview conducted with a care-giver that is used to assess the performance of everydayactivities required for personal and social sufficiencyacross a broad range of conditions [Sparrow, Cicchetti, &Balla, 2005]. The Vineland-II assesses overall adaptivefunctioning as well as adaptive behavior in specificdomains of communication, socialization, daily livingskills, and motor skills (the latter for fine and gross move-ments in children aged < 7 years). The adaptive compos-ite and domain scores are expressed as age-based stand-ard score (mean of 100, SD = 15), with higher scoresreflecting better adaptive functioning. The cut-off pointof ≤ 70 is considered to reflect functioning within a lowadaptive level.

PedsQL. The PedsQL version 4.0 generic module isdesigned for use in children aged 2–18 years and has 23items grouped into four domains: physical, emotional,social, and school functioning [Varni, Seid, & Kurtin,2001]. Each item is converted into a 0–100 scale, withhigher scores indicating better quality of life. The PedsQLtotal score is the unweighted average of all item scores.Domain scores are calculated by averaging all items inthat domain. This instrument is able to differentiatequality of life outcomes between children with ASDs andunaffected children [Kuhlthau et al., 2010]. A cut-offpoint for at-risk status for impaired HRQL (parent proxyreport) is 65.4 [Varni, Burwinkle, Seid, & Skarr, 2003].

Statistical Analyses

The HUI3 health utility score was used as the “target”measure. The CBCL, Vineland-II, and PedsQL were usedas the “start” measures. We also included child’s age,cognitive ability (IQ), and Autism Diagnostic ObservationSchedule (ADOS) calibrated severity scores as other pre-dictor variables in each model. Spearman’s correlationswere performed to examine the strength of the associa-tions between the target and each of the start measures.

The aim of the statistical analysis is to establishmapping algorithms to predict HUI3 health utility scoresfrom different start variables and other control variables.Because health utility scores in general have non-normaldistributions with negative skew, as well as ceiling effects(a large spike at the upper bound) that violate theassumptions of ordinary least squares (OLS) regression,various statistical modeling approaches have been con-sidered in previous studies [Brazier, Yang, Tsuchiya, &Rowen, 2010; Longworth & Rowen, 2011]. A number ofmodels, including the censored least absolute deviation(CLAD) model, the Tobit model, generalized linearmodels, latent class models (LCMs), two-part or two-stepmodels (TPM), and a random effects censored mixturemodel have been evaluated. A recent simulation studyfound that OLS is still superior to many of the alternativeapproaches, including CLAD, Tobit, TPMs, and LCMs[Pullenayegum et al., 2010]. The beta regression approachwas also used in comparison to OLS to address the chal-lenges of the presence of a large spike at 1 in the healthutility score distribution [Basu & Manca, 2012]. Theresults indicated that the beta regression is useful whencovariate effects are large and there are large spikes at theupper bound of the distribution. OLS, however, still pro-vides unbiased estimates when covariate effects are lessthan 0.03 following an SD change in the independentvariable and there are small spikes at the upper end of thedistribution.

The distribution of HUI3 scores in our sample is nega-tively skewed (skewness of −0.9), but only a small per-centage of respondent HUI3 scores reach the ceiling of 1(4.1%, Fig. 1). Thus, the distribution of HUI3 scores forthis study does not correspond to the distribution pat-terns that benefit from beta regression [Basu & Manca,2012]. For this reason, we decided to use OLS for allmapping algorithms in this study.

Model specifications. Seven models were specified inthis study based on either domain scores or a total scoreof each instrument as a start measure (Table 1). The ratio-nale for model specification was based on finding startmeasures that are normally reported as domain or totalscores in ASD research. Mapping algorithms could thenbe created to produce health utility scores based on theavailability of start measures (CBCL, Vineland-II, PedsQL)

INSAR4 Payakachat et al./Mapping outcome measures to health utilities

in clinical studies. Two models and resulting algorithmswere created for the CBCL and PedsQL, while threemodels were generated for the Vineland-II. The first CBCLmodel used the internalizing and externalizing problem Tscores, which are the two subdomains of the CBCL. Thesecond CBCL model was estimated only with the total

problem T scores, which is the total composite score ofthe CBCL. The first PedsQL model consisted of all fourdomains, while the second PedsQL model included onlythe total score. The first Vineland-II model comprised allfour Vineland-II domain scores (communication, dailyliving skills, socialization, and motor skills), while the

Figure 1. Distribution of observed Health Utilities Index Mark 3 (HUI3) scores.

Table 1. Model Specifications

Models Dependent variables Independent variables Covariatesa

Child Behavior Checklist (CBCL) Model 1 HUI3 Internalizing problems T score YesExternalizing problems T score

Child Behavior Checklist (CBCL) Model 2 HUI3 CBCL total problems T score YesVineland-II Adaptive Behavior Scales Model 1 HUI3 Communication Yes

Daily living skillsSocializationMotor skills

Vineland-II Adaptive Behavior Scales Model 2 HUI3 Communication YesDaily living skillsSocialization

Vineland-II Adaptive Behavior Scales Model 3 HUI3 Vineland-II composite score YesPediatrics Quality of Life Inventory (PedsQL) Model 1 HUI3 Physical functioning Yes

Emotional functioningSocial functioningSchool functioning

Pediatrics Quality of Life Inventory (PedsQL) Model 2 HUI3 PedsQL total score YesCombined Model 1 HUI3 CBCL total problems T score Yes

Vineland-II composite scorePedsQL total score

Combined Model 2 HUI3 CBCL total problems T score NoVineland-II composite scorePedsQL total score

HUI3, Health Utilities Index Mark 3.aCovariates are child age, squared child age, log cognitive ability (IQ), Autism Diagnostic Observation Schedule severity score.

5Payakachat et al./Mapping outcome measures to health utilitiesINSAR

second Vineland-II model had only three domain scores(communication, daily living skills, and socialization)because the motor skills domain is used only for children< 7 years old. The third Vineland-II model was estimatedwith only the adaptive composite score. Two additionalmodels were proposed (combined Models 1 and 2,Table 1) that included all three start measures (the totalscores from CBCL, Vineland-II, and PedsQL).

Clinical information on children with ASDs for modelbuilding in the models included the ADOS severity scoreand cognitive ability scores. The ADOS, a clinician-administered interview, considered the “gold standard”observational assessment for diagnosing ASD. The ADOScontains four assessment modules designed to elicitbehaviors directly relevant to the diagnosis of ASD atdifferent developmental levels and chronological ages[Lord et al., 2000]. The ADOS severity score is a calibratedmetric used to facilitate comparison of scores across thefour different ADOS modules and offers a method toquantify relative severity of ASD symptoms that is inde-pendent of age and language level [Gotham, Pickles, &Lord, 2009; Shumway et al., 2012]. ADOS severity scoresrange from 1 to 10, with scores of 1–3 representingnonspectrum classification, 4–5 corresponding to an“autism spectrum” classification, and 6–10 indicating an“autism” classification on the ADOS. Cognitive abilitytests used in the Autism Treatment Network registryincluded the Stanford-Binet Intelligence Scales, 5thedition, Abbreviated Battery (SB-5), an individuallyadministered, standardized cognitive assessment that canbe used with individuals age 2 years and older. However,the Mullen Scales of Early Learning, American GuidanceService Edition [Mullen, 1997] or the Bayley Scales ofInfant Development, 3rd edition [Bayley, 2005] were alsoused in the Autism Treatment Network registry to assesscognitive functioning when the child could not obtain abasal score on the SB-5 during the initial baseline evalua-tion. In our sample, the majority of children were assessedusing the SB-5 (84.3%). The SB-5 nonverbal and verbal IQand the other two measures of cognitive functioning arecomparable for the purposes of the Autism TreatmentNetwork registry as well as within our study, with correla-tions ranging from 0.78 to 0.84 [Roid, 2003]. All threecognitive measures are in comparison to age-based normsand provide a standard score with a mean of 100 and SD of15 to describe a child’s cognitive ability. A standard scoreless than 70 indicates impaired cognitive functioning.

The ADOS severity score and cognitive ability tests wereadministered only at the baseline visit. ADOS severityscores were added in each model to control for the sever-ity of the condition across the different modules of theADOS instrument. Child age (both linear and squaredterms) and logarithm of IQ were included in the model.No interaction terms were included in any models inorder to keep the models as simple as possible.

We recognize that different combinations of modelscould be specified to reach the most predictive model.However, the ability to use such a model in real-worldsettings may be limited if the clinical research project didnot have all of the various start measures. Thus, we reportthe results of parsimonious models that can be used byresearchers with only limited start measures. Models thatare more complicated, and thus more predictive, may beconsidered in future research studies or on an ad hocbasis.

Model specification was tested using the Ramsey RESETtest [Ramsey, 1969] and Link test [Pregibon, 1980]. TheOLS estimates were generated with robust standarderrors to address heteroscedasticity [Long & Ervin, 2000].All statistical significance levels were set at a P-valueof 0.05.

Model comparisons. We conducted an in-sample,cross-validation analysis to assess predictive accuracy ofthe mapping algorithms using a k-fold technique wherethe dataset was randomly partitioned into k subsamples(k = 5) with 1,000 replications [Kohavi, 1995]. Onesubsample was retained as the validation data for testingthe predictive accuracy of the model and the remainingfour subsamples were used as training data. The processwas then repeated five times, with each of the fivesubsamples used exactly once as the validation data. Ifany health utility scores were predicted to be greater than1.0, they were then truncated at 1.0 to remain consistentwith the health utility scale bounds.

The model predictive accuracy was determined usingthe individual mean absolute prediction error (MAPE)and root mean squared error (RMSE), which provided thedeviation between the predicted and observed healthutility scores [Longworth & Rowen, 2011]. The averageMAPE and RMSE from the fivefold, in-sample cross-validation analysis were then calculated. The mean errorfor an OLS model is usually near zero due to the estima-tion technique. Therefore, we did not include the meanerrors in the results. Error patterns across the scale of thehealth utility scores were reported to provide detailedinformation on how each mapping algorithm performedacross the range of scores. Finally, the intraclass correla-tion coefficients (ICC), a measure of agreement betweenpredicted and observed HUI3 scores, were calculated andreported for each algorithm using a two-way mixedmodel analysis of variance.

Missing data. A multiple imputation approach wasused to handle missing data. Missing observations werereplaced with a set of plausible values that account for theuncertainty about the right value to impute using PROCMI. The imputed datasets are then analyzed for each OLSmodel. The results from each analysis were then com-bined using PROC MIANALYZE. Both procedures wereconducted in SAS software (SAS 9.3, SAS Institute Inc.).

INSAR6 Payakachat et al./Mapping outcome measures to health utilities

Results

There were 224 surveys returned from two Autism Treat-ment Network sites (a response rate of 54.6%). Demo-graphic characteristics of the sample are presented inTable 2. The average age for children with ASDs inthis study was 8.4 (SD = 3.5) years. The sample was

predominantly Caucasian children (75.2%) relative toAfrican American (9.5%) and Hispanic (9%) children.Most children were in preschool or kindergarten (45%)or elementary school (33%). Of these children, 98.6%lived at home, with the primary caregivers being bio-logical parents. Almost half of the children (47%) hadcognitive impairment, scoring within the second per-centile rank or below (cognitive ability score (IQ) ≤ 70).The average IQ was 75.7 (SD = 24.3), which was posit-ively correlated with the HUI3 scores (ρ = 0.34,P < 0.001). The ADOS severity score ranged from 2 to 10with a mean of 7.2 (SD = 1.8). The severity score had mildbut significant correlation with HUI3 scores (ρ = −0.14,P = 0.04).

Average health utility scores as derived from the HUI3for the full sample of children with ASDs was 0.66(SD = 0.23). Children with autistic disorder had thelowest health utility scores on the HUI3 (mean of 0.63,SD = 0.24). The average HUI3 scores were the highest inthe children with Asperger’s disorder (mean of 0.77,SD = 0.15). The common problem of a spike in scores at 1(perfect health) was not apparent with the HUI3 scores asonly nine observations (4.1%) reached the upper boundof the distribution (Fig. 1).

Outcome measures from the CBCL, Vineland-II, andPedsQL instruments are presented in Table 3. The averageof the CBCL total problem T scores was 63.7 (SD = 9.4),which was higher than norm populations (mean of 50).Of the affected children, 31% had the CBCL totalproblem T score ≥ 70, which is a cut-off point for clini-cally significant behavioral problems. The CBCL totalproblem T scores were negatively associated with theHUI3 scores as expected (ρ = −0.22, P = 0.002); the morebehavior problems, the lower the health utility scores.Examination of the adaptive behavior domains indicatedthat the lowest scores were achieved on the socializationdomain (average score = 68.1, SD = 11.3). The Vineland-IIadaptive composite scores averaged 68.4 (SD = 11.1) and58% had scores ≤ 70 (low adaptive behavior skills). TheVineland-II composite scores were positively correlatedwith the HUI3 health utility score (ρ = 0.45, P < 0.001).The PedsQL 4.0 also had the lowest scores on the socialfunctioning (average score = 49.5, SD = 24.0) domain, fol-lowed by school functioning (average score = 63.1,SD = 20.3). The total PedsQL score averaged 63.0(SD = 15.7) and 51% had impaired HRQL (score ≤ 64.5).All PedsQL domain and total scores were positively asso-ciated with the HUI3 scores (P < 0.05). Overall, the utilitymeasures had weak to moderate correlations with theoutcome or “start” measures.

The Ramsey RESET and Link tests did not indicate aproblem with model specification as P-values for bothtests were greater than 0.05 in all models. Missing datapatterns appeared to be arbitrary. Percentage of missingdata ranges from 2.7 (HUI3 scores) to 16.5% (the CBCL

Table 2. Demographic Characteristics of Children WithAutism Spectrum Disorders From Two Sites of the AutismTreatment Network

N = 224

Age, mean ± standard deviation (range) 8.4 ± 3.5 (4.0–17.9)Gender

Male 86.6%Female 13.4%

Race/ethnicityCaucasian 75.2%African American 9.5%Hispanic 9.0%Asian 2.3%Other 4.1%

Education levelPreschool/kindergarten 45.0%1st–2nd grade 18.8%3rd–5th grade 14.2%6th–8th grade 7.8%9th–12th grade 5.0%Other (e.g., home-based, special education) 8.3%

Birth order1st child 50.7%2nd child 29.2%3rd child 13.0%Others 7.2%

Child living arrangementAt home 98.6%In an institution or developmental center 1.4%

Type of school the child attendNonspecialized public school 44.6%Specialized public school 8.2%Vocational public school 0.5%Private school 7.3%Home school 2.3%Special education school 26.8%Others (e.g., special education program,

self-contained class)10.5%

Primary caregiverBiological parents 94.6%Adoptive parents 3.6%Guardian 0.5%Others 1.3%

DiagnosisAutistic disorder 73.4%Asperger’s disorder 8.7%Pervasive developmental disorder-not

otherwise specified17.9%

Cognitive abilitya (IQ) ≤ 70 47%

aCognition scores are based on the Stanford-Binet Intelligence Scales,5th edition, Abbreviated Battery, or either the Mullen scales or the Bayleyscales.

7Payakachat et al./Mapping outcome measures to health utilitiesINSAR

and PedsQL Social domain) (Table 3). We specified fiveimputation sets for the multiple imputation process sincethe amount of missing data was less than 20%, to achieveat least 95% relative efficiencies [SAS Institute Inc, 2011].A Markov chain Monte Carlo method with theexpectation-maximization (EM) algorithm was used toimpute missing values before entering into the fivefoldin-sample, cross-validation procedure for each model.The EM algorithm (posterior mode) converged and rela-tive efficiency ranged from 95.7 (the Vineland-II MotorSkills) to 99.7% (HUI3 and IQ). Model prediction errors(MAPE and RMSE) were reported in Table A1. ModelMAPEs ranged from 0.1591 (the PedsQL Model 1) to0.1819 (the CBCL Model 1). Model RMSEs ranged from0.2001 (the PedsQL Model 1) to 0.2308 (the CBCL Model1). Agreement between the predicted and observed HUI3scores was the lowest for the CBCL Model 1 (ICC = 0.455)and highest for the combined Model 1 (ICC = 0.681)(Table A1). Absolute prediction errors (APEs) were alsoreported by size of error for each model (Table A2).The combined Model 2 and PedsQL Models 1 have thelowest percentage of APEs > 0.20 at 31.7% and 31.9%,respectively.

The in-sample cross-validation prediction errors(MAPEs) for the observed HUI3 score range were reportedto examine degree of prediction errors in different healthstates (Table A3). In our sample, only 7.8% (n = 17)reported HUI3 scores lower than 0.25 and 10.1% (n = 22)

reported HUI3 score between 0.9 and 1.0 (Table A3). Allpredicted models performed poorly at the lower end ofthe HUI3 scores (HUI3 < 0.25) and performed best at theHUI3 range of 0.5–0.9. Analysis of scatter plots betweenpredicted and observed HUI3 scores indicated that therewas a tendency for the OLS models to overpredict at thelower end and underpredict at the upper end of the HUI3(results not shown).

Algorithms presented as mathematical equations topredict HUI3 scores for each start measure, as well asrobust standard errors, are provided in Table 4. The CBCLmodels, the internalizing problems T score (Model 1) andthe total problem T score (Model 2), were significantlyassociated with the HUI3 scores (P = 0.003 and P < 0.001,respectively). The communication and motor skilldomains of the Vineland-II were significantly related tothe HUI3 scores in the Vineland-II model 1 (P = 0.008 andP = 0.049, respectively). For the Vineland-II Model 2,which excludes the motor skill domain, the communica-tion and daily living skill domains are significantlyrelated with the HUI3 scores (P = 0.041 and P = 0.016,respectively). The Vineland-II composite score is also sig-nificantly associated with the HUI3 scores (P < 0.001).The PedsQL Model 1 found significant relationshipsbetween the physical and social functioning domains andthe HUI3 (P = 0.001 and P = 0.003, respectively). ThePedsQL total score was statistically significant in Model 2(P < 0.001). The Vineland-II composite score and PedsQL

Table 3. Outcome Measures and Spearman Correlations With HUI3 Scores (N = 224)

Outcome measures N Missing data (%) Mean (SD), range

Spearman correlations

HUI3

HUI3 scores 218 2.7 0.66 (0.23), −0.098–1.0 1.000ADOS severity scores 205 8.4 7.2 (1.8), 2–10 −0.143*Cognitive ability 197 12.1 75.7 (24.3), 41–148 0.310*Child Behavior Checklist

Externalizing problems T score 187 16.5 58.3 (11.0), 33–85 −0.110Internalizing problems T score 187 16.5 60.9 (10.3), 33–83 −0.203**Total problems T score 187 16.5 63.7 (9.4), 39–92 −0.224**

Vineland-II Adaptive Behavior ScalesCommunication 197 12.1 71.7 (15.2), 33–135 0.385***Daily living skills 197 12.1 71.2 (13.3), 33–107 0.392***Socialization 196 12.5 68.1 (11.3), 40–103 0.334***Motor skillsa 122 0 76.3 (12.4), 40–114 0.483***Composite score 194 13.4 68.4 (11.1), 36–105 0.445***

Pediatrics Quality of Life Inventory 4.0Physical functioning 194 13.4 71.0 (20.0), 3.1–100 0.333***Emotional functioning 190 15.2 64.8 (20.8), 5.0–100 0.152*Social functioning 192 14.3 49.5 (24.0), 0–100 0.343***School functioning 187 16.5 63.1 (20.3), 20–100 0.189*Total score 194 13.4 63.0 (15.7), 11.8–100 0.375***

ADOS, Autism Diagnostic Observation Schedule; HUI3, Health Utilities Index Mark 3; SD, standard deviation.aThe Vineland-II Motor Skills domain was used only in children age < 7 years.*P < 0.05; **P < 0.01; ***P < 0.001.

INSAR8 Payakachat et al./Mapping outcome measures to health utilities

Table 4. Algorithms for Estimating HUI3 Scores by Model

(a) HUI3 and Child Behavior Checklist Models

Parameter

Model 1 Model 2

Estimate Robust STDERR P Estimate Robust STDERR P

Externalizing problems T score −0.0015 0.0014 0.435Internalizing problems T score −0.0043 0.0016 0.003Total problems T score −0.0068 0.0015 < 0.001ADOS severity −0.0124 0.0085 0.165 −0.0136 0.0082 0.107Child age 0.0078 0.0192 0.709 0.0116 0.0189 0.569Squared child age −0.0001 0.0011 0.924 −0.0002 0.0011 0.832Log cognitive ability (IQ) 0.2282 0.0511 0.033 0.2217 0.0503 0.025Intercept 0.0744 0.2567 0.843 0.1726 0.2505 0.614

(b) HUI3 and Vineland-II models

Parameter

Model 1 Model 2 Model 3

Estimate Robust STDERR P Estimate Robust STDERR P Estimate Robust STDERR P

Communication 0.0033 0.0015 0.008 0.0037 0.0015 0.041Daily living skills 0.0032 0.0018 0.152 0.0046 0.0017 0.016Socialization 0.0002 0.0019 0.923 0.0010 0.0019 0.651Motor skills 0.0033 0.0017 0.049Composite score 0.0103 0.0016 < 0.001ADOS severity −0.0050 0.0079 0.553 −0.0051 0.0079 0.539 −0.0063 0.0078 0.437Child age 0.0097 0.0193 0.644 0.0105 0.0192 0.619 0.0119 0.0186 0.553Squared child age 0.0004 0.0010 0.746 0.0003 0.0010 0.788 0.0003 0.0010 0.805Log cognitive ability (IQ) 0.0252 0.0485 0.761 0.0238 0.0494 0.771 0.0304 0.0478 0.705Intercept −0.2367 0.2032 0.393 −0.1630 0.2008 0.525 −0.2438 0.2015 0.342

(c) HUI3 and Pediatrics Quality of Life Inventory models

Parameter

Model 1 Model 2

Estimate Robust STDERR P Estimate Robust STDERR P

Physical functioning 0.0031 0.0008 0.001Emotional functioning 0.0004 0.0008 0.656Social functioning 0.0021 0.0007 0.003School functioning −0.0001 0.0008 0.957Total score 0.0059 0.0008 < 0.001ADOS severity −0.0104 0.0075 0.196 −0.0100 0.0077 0.224Child age 0.0016 0.0183 0.465 0.0177 0.0184 0.407Squared child age −0.0003 0.0010 0.785 −0.0039 0.0010 0.738Log cognitive ability (IQ) 0.1875 0.0445 0.019 0.1971 0.0454 0.023Intercept −0.5172 0.2262 0.160 −0.5374 0.2298 0.136

(d) HUI3 and combined measures models

Parameter

Model 1 Model 2

Estimate Robust STDERR P Estimate Robust STDERR P

CBCL total problems T score −0.0028 0.0017 0.176 −0.0023 0.0019 0.244Vineland II composite score 0.0067 0.0015 < 0.001 0.0076 0.0013 < 0.001PedsQL total score 0.0041 0.0009 0.001 0.0034 0.0011 0.013ADOS severity −0.0120 0.0072 0.122Child age 0.0273 0.0197 0.192Squared child age −0.0006 0.0010 0.558Log cognitive ability (IQ) 0.1204 0.0511 0.100Intercept −0.4850 0.2345 0.079 0.0667 0.1886 0.730

ADOS, Autism Diagnostic Observation Schedule; CBCL, Chile Behavior Checklist; HUI3, Health Utilities Index Mark 3; PedsQL, Pediatrics Quality of LifeInventory; STDERR, standard error.

9Payakachat et al./Mapping outcome measures to health utilitiesINSAR

total score were statistically significantly associated withHUI3 in both the combined models.

Discussion

The increased prevalence of autism has created renewedinterest in ensuring that children with ASDs receive ser-vices to achieve optimal outcomes [Interagency AutismCoordinating Committee, 2011]. Evaluations of the effec-tiveness of services for children with ASDs typicallyinclude clinical, behavioral, and HRQL outcome mea-sures that cannot be translated into the cost per QALY forinforming health policy decision makers despite thepotential for information on cost-effectiveness to trans-late effective treatments into sustained practice [Glasgow& Steiner, 2012]. Cost-effectiveness evaluations rely onQALYs as the recommended outcome measure because itcan facilitate comparisons across different type of treat-ments or interventions, health conditions, and patientgroups [Weinstein et al., 1996]. The lack of trials andother clinical studies that prospectively measure QALYshas led to limited information on the cost-effectiveness ofservices for children with ASDs. Indeed, we are unawareof any studies that use the cost per QALY metric to evalu-ate services for children with ASDs.

This study seeks to increase the amount of informationavailable on QALY gains associated with services for chil-dren with ASDs by creating mapping algorithms that canbe used with clinical outcome measures. In this study, wecombined information on outcome measures typicallyused in clinical trials of services for children with ASDswith health utility data as measured by the HUI3. Withthese data, the investigators developed several mappingalgorithms that can be used to predict health utility scoresbased on clinical, behavioral, and HRQL outcome mea-sures. Investigators can use the algorithms along with datafrom successful trials of alternative treatments for childrenwith autism to predict the QALYs gained from the inter-ventions studied. If investigators have only one of the“start” measures, they can choose an algorithm with thesame subdomain scores used in their original studies topredict HUI3 scores based on the lowest MAPE (Table A1).Similarly, when investigators have more than one of the“start” measures, they may choose an algorithm that hasthe lowest MAPE. The suggestions are based solely onauthors’ opinions. The QALY information could then becombined with cost data to calculate incremental cost-effectiveness ratios for use in resource allocation decisions[Weinstein et al., 1996]. In short, investigators will have agreater opportunity to ask whether an effective interven-tion is “worth it” relative to the cost. Asking such aquestion has the potential to translate interventions intosustained practice if decision makers can see that thecost-effectiveness profiles for autism services are similar to

other medical or public health interventions, or if someservices achieve similar outcomes at lower cost.

To illustrate, the investigators used information fromthe Vineland-II Model 1 mapping algorithm derived inthis paper (Table 4b) to evaluate the QALYs gained fromearly intervention services: HUI3 utility scores =−0.2367 + 0.0033×communication + 0.0032× daily livingskills + 0.0002× socialization + 0.0033× motor skills +(−0.0050)×ADOS severity scores + 0.0252×log(IQ). Find-ings from the Early Start Denver model (ESDM) showedimprovements or less regression on the Vineland-IIdomains and clinical outcomes (ADOS severity score andIQ) when compared with the standard intervention(assess-and-monitor, A/M) [Dawson et al., 2010]. Thechanges in Vineland-II domains from the ESDM and A/Mwere the communication (+13.7 vs. −0.7), daily livingskills (−6.1 vs. −14.5), socialization (−4.6 vs. −8.9), andmotor skills (−9.9 vs. −23.1). The ADOS severity amongthe cohort who received the ESDM was improved (−0.2)when compared with the A/M group (+0.3). The IQamong children who received the ESDM increased 17.6points when compared with 7.0 points in the A/M group.While the improved outcomes clearly suggest benefitsassociated with the intervention, it is unclear whether theESDM is cost-effective relative to other medical andpublic health services. Application of the mapping algo-rithm suggests average QALY gains of 0.13 at 2 yearspost-baseline. This summary measure of health gainspoints to large increases in HRQL that has a high prob-ability of being a cost-effective investment of societalresources. While our findings are intended for illustrativepurposes as we lack access to the data, the evidenceclearly suggests that a full-scale investigation of the cost-effectiveness of the ESDM for children with ASDs is war-ranted to provide evidence for health system decisionmaking.

Limitations

There are a number of limitations that need to beaddressed in this study. The mapping algorithms pro-vided in this study were modeled based on children fromtwo locations with ADOS severity scores ranging frommild to severe. The average ADOS severity scores were notstatistically significantly different between the twoAutism Treatment Network locations. Additionally, eachAutism Treatment Network site had only 9% of theaffected children who had ADOS severity scores ≤ 5. Twothirds of the sample from both Autism TreatmentNetwork sites had ADOS severity scores in the range of 6and 8. Generalizability of these algorithms may belimited, and additional research on applying them to datafrom clinical trials is needed. Researchers who desire touse these mapping algorithms in different populationsshould recognize these limitations. The sample size for

INSAR10 Payakachat et al./Mapping outcome measures to health utilities

this mapping exercise was adequate but nonethelesslimited. Naturally, a larger sample size is always desirable,particularly if one would like to include additional vari-ables or interaction terms in the model.

Although the mapping approach is gaining popularity,as it permits researchers to predict health utility scoreswhen original studies did not include such a measure,mapping should be considered a second-best solution todirectly collected health utility values [Longworth &Rowen, 2011]. Hence, the use of mapping algorithms willlead to increased uncertainty and error around the esti-mated health utility values. In this regard, using themapping algorithms to predict HUI3 scores for cost-effectiveness analyses should incorporate sensitivityanalyses to test the robustness of the QALY estimates[Chapman et al., 2004]. Mapping algorithms also greatlyexpand the potential information set from prior com-parative effectiveness research studies. It will take muchlonger to accumulate similar information based onprimary data collection of health utility scores in prospec-tive clinical studies.

The generic instruments for measuring health utilitiesare applicable for a wide range of health conditions andtreatments, although there are special issues in applica-tions involving children [Grosse, Prosser, Asakawa, &Feeny, 2010; Payakachat et al., 2012; Prosser, Grosse,Kemper, Tarini, & Perrin, 2012; Prosser, Hammitt, &Keren, 2007; Ungar, 2007, 2010, 2011]. In particular,evaluation of health utilities in young children (< 5) isespecially difficult as instruments are not designed forthis age group and typically requires out-of-sample pre-diction. Most of the children in this sample were above 5years old (87.9%). Importantly, we only map outcomes ofthe child onto health utilities. It is now recognized thatinterventions affecting children have a high probabilityof affecting the health of caregivers and other familymembers, and these effects should be incorporated ineconomic evaluations [Meltzer & Smith, 2012]. However,mapping family “spillover” effects for economic evalua-tion is beyond the scope of this paper.

The recognition that children represent a special popu-lation for economic evaluation has led to the creation ofnew utility measures (such as the CHU9D [Stevens, 2010]and the EQ-5D Youth [Wille et al., 2010]) that mayimprove prediction. We could not include these measuresat the time this study began, as they were not availablefor public use. Initial evidence suggests sufficient con-struct validity with the HUI3 to warrant its use inmapping exercises predicting QALY gains [Horsman,Fulong, Feeny, & Torrance, 2003]. However, it should alsobe pointed out that QALYs pertain to health-relatedutility only. One may consider conditions, such as ASDs,to have an impact on broader well-being also. Hence,using restrictive outcome measures could result in ignor-ing relevant outcomes. This needs to be investigated

further, since cost-effectiveness, defined broadly, allowsthe use of broader outcome measures than QALYs[Brouwer, Culyer, van Exel, & Rutten, 2008]. While thisdebate is directly relevant for how resources should beallocated to the treatment and prevention of autism, itremains beyond the scope of this study.

The predictive accuracy of the mapping algorithms wasexamined using the difference between predicted andobserved values (i.e., MAPE and RMSE) from in-samplecross-validation to provide an indication of the size of theprediction errors. Unfortunately, we cannot report out-of-sample predictive accuracy because there is currently noother study in children with ASDs that contains HUI3scores and the “start” measure. The predictive errorsreported from our mapping exercises are comparable tothose reported in the literature [Brazier et al., 2010;Longworth & Rowen, 2011]. However, the predictionerrors are often larger for models mapping a condition-specific measure onto a generic utility score than mappinggeneric health outcome measures onto generic utilitymeasures [Longworth & Rowen, 2011]. One reason may bethe limited conceptual overlap between the start measuresand the HUI3 utility measure. We are aware of only twostudies that mapped generic preference-based scores fromother measures used in child health conditions. Dakinet al. [2010] mapped OM8-30 measure for children withotitis media to HUI2 and HUI3 [Dakin et al., 2010]. Ouralgorithms performed relatively similar to Dakin’s algo-rithms for the HUI3, with prediction errors in the sameranges. Khan, Petrou, Rivero-Arias, Walters, and Boyle(2014) mapped the PedsQL to EQ-5D in a relativelyhealthy population of school-aged children. Khan et al.’salgorithms have similar prediction errors in terms ofRMSEs when compared with ours, but their mean predic-tion errors are relatively smaller. However, Khan et al.’salgorithms are robust only to healthy children aged 11–15years and EQ-5D scores ≥ 0.6. We did not report explana-tory power using R-squared since it is less informative forevaluating mapping performance relative to predictionerrors [Brazier et al., 2010; Longworth & Rowen, 2011].

When predicting the HUI3 scores using the mappingalgorithms, it is important to be aware that predictedscores greater than 1.0 are possible. Since any predictedHUI3 scores above 1.0 is an artifact of the statisticalmodel, researchers should truncate any predicted scoreof greater than 1.0 to 1.0 to be consistent with the origi-nal health utility scale. In addition, sensitivity analysesof point estimates mapped from any start measuresshould be incorporated into cost-effectiveness analyses[Briggs, 2000]. In probabilistic sensitivity analyses, robuststandard errors of each start measure should be used.The pattern of errors in this study is similar to othermapping studies that used condition-specific start mea-sures [Goldsmith et al., 2010; Payakachat et al., 2009].The algorithms do not perform well in the lower end of

11Payakachat et al./Mapping outcome measures to health utilitiesINSAR

the health utility scores (i.e., HUI3 < 0.25). The problemof overpredicted health utility scores for patients inpoor health was also reported in the literature [Brazieret al., 2010; Versteegh et al., 2012]. A separate mappingalgorithm to predict health utility for children with ASDswho are in poor health may reduce overpredictionproblem [Versteegh, Rowen, Brazier, & Stolk, 2010], butwe feel that on average the algorithms provided in thisstudy will permit new estimates of the cost per QALYgained for services provided to children with ASDs. Ourcurrent sample contained only 17 children (7.8%) whohad HUI3 scores of < 0.25, and only two (0.9%) out of theentire sample had HUI3 scores lower than 0. Futureresearch is needed to test these mapping algorithms indifferent datasets containing health outcome measuresfor children with ASDs. Lastly, the mapping algorithmsare generated from imputed data, which may influencethe parameters of the estimated algorithms, as well asunderestimate underlying relationships between theHUI3 scores and the “start” measures.

Conclusions

The mapping algorithms provided in this study can beapplied to clinical trials and other studies containing oneof the start measures (CBCL, Vineland-II, PedsQL) forchildren with ASDs in order to predict health utilityscores when direct elicitation of health utility scores werenot possible. The predicted health utilities can be incor-porated into economic evaluations in terms of QALYs.Evidence from economic evaluations can provide usefulinformation to decision-makers, including healthcareproviders, policy makers, and families, as cost-effectiveness information has the potential to justify theprovision of services that are underprovided.

Acknowledgments

The project was supported by Grant No. R01MH089466from the National Institute of Mental Health with JMTand KAK serving as principal investigators, and Grant No.R03MH102495 with NP as the principal investigator. JMTalso was supported by the Translational Research Insti-tute, Grant No. UL1TR000039 through the NIH NationalCenter for Research Resources and National Center forAdvancing Translational Sciences. The content of thisarticle is solely the responsibility of the authors and doesnot necessarily represent the official views of the NationalInstitute of Mental Health or the National Institutes ofHealth. The authors acknowledge the members of theAutism Treatment Network for use of the data. Thedata for the study were collected as part of the AutismTreatment Network, a program of Autism Speaks.Further support came from a cooperative agreement

(UA3MC11054) from the U.S. Department of Health andHuman Services, Health Resources and Services Adminis-tration, and Maternal and Child Health ResearchProgram, to the Massachusetts General Hospital. Thework described in this article represents the independentefforts of the authors with no restrictions from thefunding source or the Autism Treatment Network. Noneof the authors of this study reported a conflict of inter-est associated with the preparation of the manuscript.Maria Melguizo, Nupur Chowdhury, Rebecca Rieger, andLatunja Sockwell provided excellent research assistance.

References

Achenbach, T.M., & Ruffle, T.M. (2000). The Child BehaviorChecklist and related forms for assessing behavioral/emotional problems and competencies. Pediatrics in Review,21, 265–271.

American Psychiatric Association. (2013). Diagnostic and Statis-tical Manual of Mental Disorders, 5th edition (DSM-V).Arlinton, VA: American Psychiatric Publishing.

Bailey, A.J. (2009). Where are the autism economists? AutismResearch, 2, 245.

Basu, A., & Manca, A. (2012). Regression estimators for generichealth-related quality of life and quality-adjusted life years.Medical Decision Making, 32, 56–69.

Bayley, N. (2005). Bayley scales of infant and toddler develop-ment (3rd ed.). San Antonio, TX: Pearson Education, Inc.

Blumberg, S.J., Bramlett, M.D., Kogan, M.D., Schieve, L.A., Jones,J.R., & Lu, M.C. (2013). Changes in prevalence of parent-reported autism spectrum disorder in school-aged U.S. chil-dren: 2007 to 2011-2012 (Rep. No. 65). National Center forHealth Statistics, Hyattsville, MD.

Brazier, J.E., Ratcliffe, J., Salomon, J., & Tsuchiya, A. (2007).Valuing health. In J.E. Brazier, J. Ratcliffe, J. Salomon, & A.Tsuchiya (Eds.), Measuring and valuing health benefits foreconomic evaluation (pp. 83–138). New York, NY: OxfordUniversity Press Inc.

Brazier, J.E., & Roberts, J. (2004). The estimation of a preference-based measure of health from the SF-12. Medical Care, 42,851–859.

Brazier, J.E., Yang, Y., Tsuchiya, A., & Rowen, D.L. (2010). Areview of studies mapping (or cross walking) non-preferencebased measures of health to generic preference-based mea-sures. The European Journal of Health Economics, 11, 215–225.

Briggs, A.H. (2000). Handling uncertainty in cost-effectivenessmodels. Pharmacoeconomics, 17, 479–500.

Brouwer, W.B.F., Culyer, A.J., van Exel, N.J., & Rutten, F.F.H.(2008). Welfarism vs. extra-welfarism. Journal of Health Eco-nomics, 27, 325–338.

Buysse, C.M., Raat, H., Hazelzet, J.A., Hulst, J.M., Cransberg, K.,et al. (2008). Long-term health status in childhood survivorsof meningococcal septic shock. Archives of Pediatrics & Ado-lescent Medicine, 162, 1036–1041.

Centers for Disease Control and Prevention. (2012). Prevalenceof autism spectrum disorders—Autism and DevelopmentalDisabilities Monitoring Network, 14 Sites, United States,

INSAR12 Payakachat et al./Mapping outcome measures to health utilities

2008. MMWR. Surveillance Summaries: Morbidity and Mor-tality Weekly Report. Surveillance Summaries, 61, 1–19.

Chapman, R.H., Berger, M., Weinstein, M.C., Weeks, J.C., Goldie,S., & Neumann, P.J. (2004). When does quality-adjustinglife-years matter in cost-effectiveness analysis? Health Eco-nomics, 13, 429–436.

Crott, R., & Briggs, A. (2010). Mapping the QLQ-C30 quality oflife cancer questionnaire to EQ-5D patient preferences. Euro-pean Journal of Health Economics, 11, 427–434.

Dakin, H., Petrou, S., Haggard, M., Benge, S., & Williamson, I.(2010). Mapping analyses to estimate health utilities based onresponses to the OM8-30 otitis media questionnaire. Qualityof Life Research, 19, 65–80.

Dawson, G., Rogers, S., Munson, J., Smith, M., Winter, J., et al.(2010). Randomized, controlled trial of an intervention fortoddlers with autism: The Early Start Denver Model. Pediat-rics, 125, e17–e23.

Dekker, M.C., Nunn, R., & Koot, H.M. (2002). Psychometricproperties of the revised Developmental Behaviour Checklistscales in Dutch children with intellectual disability. Journalof Intellectual Disability Research, 46, 61–75.

The EuroQol Group. (1990). EuroQol—A new facility for themeasurement of health-related quality of life. Health Policy,16, 199–208.

Feeny, D., Furlong, W., Torrance, G.W., Goldsmith, C.H., Zhu, Z.,et al. (2002). Multiattribute and single-attribute utility func-tions for the Health Utilities Index Mark 3 system. MedicalCare, 40, 113–128.

Glasgow, R.E., Doria-Rose, V.P., Khoury, M.J., Elzarrad, M.,Brown, M.L., & Stange, K.C. (2013). Comparative effective-ness research in cancer: What has been funded and whatknowledge gaps remain? Journal of the National Cancer Insti-tute, 105, 766–773.

Glasgow, R.E., & Steiner, J.F. (2012). Comparative effectivenessresearch to accelerate translation: Recommendations for anemerging field of science. In R.C. Brownson, G.A. Colditz, &E.K. Proctor (Eds.), Dissemination and implementationresearch in health (pp. 72–92). New York, NY: Oxford Uni-versity Press.

Glasgow, R.E., Vinson, C., Chambers, D., Khoury, M.J., Kaplan,R.M., & Hunter, C. (2012). National Institutes of Healthapproaches to dissemination and implementation science:Current and future directions. American Journal of PublicHealth, 102, 1274–1281.

Gold, M., Siegel, J., Russell, L., & Weinstein, M. (1996). Costeffectiveness in health and medicine. New York, NY: OxfordUniversity Press.

Goldsmith, K., Dyer, M., Buxton, M., & Sharples, L. (2010).Mapping of the EQ-5D index from clinical outcome measuresand demographic variables in patients with coronary heartdisease. Health and Quality of Life Outcomes, 8, 54.

Gotham, K., Pickles, A., & Lord, C. (2009). StandardizingADOS scores for a measure of severity in autism spectrumdisorders. Journal of Autism and Developmental Disorder, 39,693–705.

Greenberg, D., & Neumann, P.J. (2011). Does adjusting forhealth-related quality of life matter in economic evaluationsof cancer-related interventions? Expert Review ofPharmacoeconomics & Outcomes Research, 11, 113–119.

Grosse, S.D., Prosser, L.A., Asakawa, K., & Feeny, D. (2010).QALY weights for neurosensory impairments in pediatriceconomic evaluations: Case studies and a critique. ExpertReview of Pharmacoeconomics & Outcomes Research, 10,293–308.

Horsman, J., Fulong, W., Feeny, D., & Torrance, G. (2003). TheHealth Utilities Index (HUI®): Concepts, measurement prop-erties and applications. Health and Quality of Life Outcomes,1, 54.

Howlin, P., Goode, S., Hutton, J., & Rutter, M. (2004). Adultoutcome for children with autism. Journal of Child Psychol-ogy and Psychiatry, 45, 212–229.

Howlin, P., Magiati, I., & Charman, T. (2009). Systematic reviewof early intensive behavioral interventions for children withautism. American Journal on Intellectual and DevelopmentalDisabilities, 114, 23–41.

Interagency Autism Coordinating Committee. (2011). 2011IACC strategic plan for autism spectrum disorder research.Retrieved October 3, 2012, from http://iacc.hhs.gov/strategic-plan/2011/IACC_2011_Strategic_Plan.pdf

Khan, K.A., Petrou, S., Rivero-Arias, O., Walters, S.J., & Boyle, S.E.(2014). Mapping EQ-5D utility scores from the PedsQLgeneric core scales. Pharmacoeconomics, 32, 693–706.doi:10.1007/s40273-014-0153-y; April 9.

Kohavi, R. (1995). A study of cross-validation and bootstrap foraccuracy estimation and model selection. Proceedings of the14th International Joint Conference on Artificial Intelligence,2, 1137-1143.

Kuhlthau, K., Orlich, F., Hall, T., Sikora, D., Kovacs, E., et al.(2010). Health-related quality of life in children with autismspectrum disorders: Results from the autism treatmentnetwork. Journal of Autism and Developmental Disorders,40, 721–729.

Lipscomb, J., Drummond, M., Fryback, D., Gold, M., & Revicki,D. (2009). Retaining, and enhancing, the QALY. Value inHealth, 12(Suppl. 1), S18–S26.

Long, S., & Ervin, L. (2000). Using heteroscedasticity consistentstandard errors in the linear regression model. The AmericanStatistician, 54, 217–224.

Longworth, L., Buxton, M., Sculpher, M., & Smith, D. (2005).Estimating utility data from clinical indicators for patientswith stable angina. European Journal of Health Economics, 6,347–353.

Longworth, L.J., & Rowen, D. (2011). NICE DSU technicalsupport document 10: The use of mapping methods to esti-mate health state utility values. Decision Support Unit,ScHARR, University of Sheffield, Sheffield, UK.

Lord, C., & Bishop, S.L. (2010). Autism spectrum disorders: Diag-nosis, prevalence, and services for children and families.Society for Research in Child Development. Sharing Childand Youth Development Knowledge. Social Policy Report, 24,1–25.

Lord, C., & McGee, J. (2001). Educating children with autism.Washington, DC: National Academies Press.

Lord, C., Risi, S., Lambrecht, L., Cook, E.H. Jr., Leventhal,B.L., et al. (2000). The Autism Diagnostic ObservationSchedule-Generic: A standard measure of social and commu-nication deficits associated with the spectrum of autism.Journal of Autism and Developmental Disorders, 30, 205–223.

13Payakachat et al./Mapping outcome measures to health utilitiesINSAR

Luo, N., Johnson, J.A., & Coons, S.J. (2010). Using instrument-defined health state transitions to estimate minimally impor-tant differences for four preference-based health-relatedquality of life instruments. Medical Care, 48, 365–371.

Meltzer, D.O., & Smith, P.C. (2012). Theoretical issues relevant tothe economic evaluation of health technologies. In V.P. Mark,T.G. McGuire, & P.P. Barros (Eds.), Handbook of health eco-nomics (Vol. 2, pp. 433–469). Amsterdam: Elsevier ScienceLtd.

Mullen, E. (1997). Mullen Scales of Early Learning. Los Angeles,CA: Western Psychological Services.

Myers, S.M., Johnson, C.P., & the Council on Children WithDisabilities. (2007). Management of children with autismspectrum disorders. Pediatrics, 120, 1162–1182.

The National Autism Center. (2009). National standards report.Randolph, MA: National Autism Center.

The National Institute for Health and Care Excellence. (2013).Guide to the methods of technology appraisal 2013. London,England: The National Institute for Health and CareExcellence.

National Institute for Health and Clinical Excellence (NICE).(2008). Guide to the methods of technology appraisal.London: NICE.

Neumann, P.J., Goldie, S.J., & Weinstein, M.C. (2000).Preference-based measures in economic evaluation in healthcare. Annual Review of Public Health, 21, 587–611.

Neumann, P.J., & Greenberg, D. (2009). Is the United States readyfor QALYs? Health Affairs (Project Hope), 28, 1366–1371.

Payakachat, N., Summers, K.H., Pleil, A.M., Murawski, M.M.,Thomas, J. III, et al. (2009). Predicting EQ-5D utility scoresfrom the 25-item National Eye Institute Vision FunctionQuestionnaire (NEI-VFQ 25) in patients with age-relatedmacular degeneration. Quality of Life Research, 18, 801–813.

Payakachat, N., Tilford, J.M., Kovacs, E., & Kuhlthau, K. (2012).Autism spectrum disorders: A review of measures for clinical,health services and cost-effectiveness applications. ExpertReview of Pharmacoeconomics & Outcomes Research, 12,485–503.

Petrou, S., Johnson, S., Wolke, D., Hollis, C., Kochhar, P., &Marlow, N. (2010). Economic costs and preference-basedhealth-related quality of life outcomes associated with child-hood psychiatric disorders. The British Journal of Psychiatry,197, 395–404.

Petrou, S., & Kupek, E. (2009). Estimating preference-basedHealth Utilities Index Mark 3 utility scores for childhoodconditions in England and Scotland. Medical DecisionMaking, 29, 291–303.

Pregibon, D. (1980). Goodness of link tests for generalized linearmodels. Applied Statistics, 29, 15–24.

Prince, F.H.M., Geerdink, L.M., Borsboom, G.J.J.M., Twilt, M.,van Rossum, M.A.J., et al. (2010). Major improvements inhealth-related quality of life during the use of etanercept inpatients with previously refractory juvenile idiopathic arthri-tis. Annals of the Rheumatic Diseases, 69, 138–142.

Prosser, L.A., Grosse, S.D., Kemper, A.R., Tarini, B.A., & Perrin,J.M. (2012). Decision analysis, economic evaluation, andnewborn screening: Challenges and opportunities. Geneticsin Medicine, 14, 703–712.

Prosser, L.A., Hammitt, J.K., & Keren, R. (2007). Measuringhealth preferences for use in cost-utility and cost-benefitanalyses of interventions in children: Theoretical and meth-odological considerations. Pharmacoeconomics, 25, 713–726.

Pullenayegum, E.M., Tarride, J.E., Xie, F., Goeree, R., Gerstein,H.C., & O’Reilly, D. (2010). Analysis of health utility datawhen some subjects attain the upper bound of 1: Are Tobitand CLAD models appropriate? Value in Health, 13, 487–494.

Ramsey, J. (1969). Tests for specification error in classical linearleast squares regression analysis. Journal of the Royal Statis-tical Society Series B, 31, 350–371.

Roid, G. (2003). Stanford-Binet Intelligence Scales (5th ed.).Itasca, IL: Riverside Publishing.

Rothenberg, B., & Samson, D. (2009). Early intensive behavioralintervention based on applied behavior analysis amongchildren with autism spectrum disorders. Technology Evalu-ation Center Assessment Program. Executive Summary, 23,1–61.

SAS Institute Inc. (2011). The MI procedure. In SAS/STAT 9.3user’s guide, SAS Institute Inc, Cary, NC, p. 4610.

Scottish Intercollegiate Guidelines Network. (2007). Assessment,diagnosis and clinical interventions for children and youngpeople with autism spectrum disorders. A national clinicalguideline. (Rep. No. SIGN Publication No. 98). Scottish Inter-collegiate Guidelines Network (SIN), Edinburgh, Scotland.

Seiber, W.J., Groessl, E.J., Ganiats, T.G., & Kaplan, R.M. (2008).Quality of Well Being Self-Administered (QWB-SA) Scale:User’s manual. Health Services Research Center, University ofCalifornia, San Diego, San Diego, CA.

Shumway, S., Farmer, C., Thurm, A., Joseph, L., Black, D., &Golden, C. (2012). The ADOS calibrated severity score: Rela-tionship to phenotypic variables and stability over time.Autism Research, 5, 267–276.

Sparrow, S.S., Cicchetti, D.V., & Balla, D.A. (2005). VinelandAdaptive Behavior Scales (2nd ed.). Circle Pines, MN: Ameri-can Guidance Services, Inc.

Stevens, K. (2010). Valuation of the Child Health Utility Index9D (CHU9D). Pharmacoeconomics, 30, 729–747.

Tilford, J.M., Payakachat, N., Kovacs, E., Pyne, J.M., Brouwer, W.,et al. (2012). Preference-based health-related quality of lifeoutcomes in children with autism spectrum disorders: A com-parison of generic instruments. Pharmacoeconomics, 30,1–19.

Ungar, W.J. (2007). Paediatric health economic evaluations: Aworld view. Healthcare Quarterly, 10, 134–135.

Ungar, W.J. (2010). Economic evaluation in child health. NewYork, NY: Oxford Publisher Press.

Ungar, W.J. (2011). Challenges in health state valuation in pae-diatric economic evaluation: Are QALYs contraindicated?Pharmacoeconomics, 29, 641–652.

Varni, J.W., Burwinkle, T.M., Seid, M., & Skarr, D. (2003). ThePedsQLTM 4.0 as a pediatric population health measure: Fea-sibility, reliability, and validity. Ambulatory Pediatrics, 3,329–341.

Varni, J.W., Seid, M., & Kurtin, P.S. (2001). The PedsQL 4.0:Reliability and validity of the Pediatric Quality of LifeInventoryTM version 4.0 generic core scales in healthy andpatient populations. Medical Care, 39, 800–812.

INSAR14 Payakachat et al./Mapping outcome measures to health utilities

Versteegh, M.M., Leunis, A., Luime, J.J., Boggild, M.,Uyl-de Groot, C.A., & Stolk, E.A. (2012). Mapping QLQ-C30,HAQ, and MSIS-29 on EQ-5D. Medical Decision Making, 32,554–568.

Versteegh, M.M., Rowen, D., Brazier, J.E., & Stolk, E.A. (2010).Mapping onto EQ-5D for patients in poor health. Health andQuality of Life Outcomes, 8, 141–148.

Wallander, J.L., Dekker, M.C., & Koot, H.M. (2006). Risk factorsfor psychopathology in children with intellectual disability:A prospective longitudinal population-based study. Journalof Intellectual and Disability Research, 50, 259–268.

Warren, Z., McPheeters, M.L., Sathe, N., Foss-Feig, J.H., Glasser,A., & Veenstra-VanderWeele, J. (2011). A systematic review ofearly intensive intervention for autism spectrum disorders.Pediatrics, 127, e1303–e1311.

Weinstein, M.C., Siegel, J.E., Gold, M.R., Kamlet, M.S., & Russell,L.B. (1996). Recommendations of the panel on cost-effectiveness in health and medicine. JAMA: The Journal ofthe American Medical Association, 276, 1253–1258.

Wille, N., Badia, X., Bonsel, G., Burström, K., Cavrini, G., et al.(2010). Development of the EQ-5D-Y: A child-friendly versionof the EQ-5D. Quality of Life Research, 19, 875–886.

Appendix A

Table A1. Prediction Errors From OLS Regression Models, In-Sample Cross-Validation

MAPE RMSE ICC

CBCL Model 1 0.1819 0.2308 0.455CBCL Model 2 0.1777 0.2266 0.474Vineland-II Model 1 0.1681 0.2064 0.616Vineland-II Model 2 0.1674 0.2111 0.595Vineland-II Model 3 0.1666 0.2095 0.607PedsQL Model 1 0.1591 0.2001 0.587PedsQL Model 2 0.1618 0.2053 0.644Combined Model 1 0.1592 0.2025 0.681Combined Model 2 0.1610 0.2026 0.636

ICC, intraclass correlation coefficient; MAPE, mean absolute prediction error; OLS, ordinary least squares; RMSE, root mean square error.

Table A2. In-Sample Cross-Validation Absolute Prediction Errors (APE) by Size of Error

OLS %APE ≤ 0.05 0.05 < %APE ≤ 0.10 0.10 < %APE ≤ 0.15 0.15 < %APE ≤ 0.20 %APE > 0.20

CBCL Model 1 17.0 18.3 15.3 12.8 36.5CBCL Model 2 17.2 18.2 17.1 12.6 34.8Vineland-II Model 1 16.5 15.3 18.2 16.9 33.1Vineland-II Model 2 19.4 18.3 14.7 12.2 35.4Vineland-II Model 3 19.7 16.1 14.9 16.4 32.8PedsQL Model 1 19.8 20.5 16.2 11.6 31.9PedsQL Model 2 19.7 21.8 15.1 10.0 33.5Combined Model 1 22.2 16.3 16.4 12.8 32.3Combined Model 2 19.1 17.2 21.0 11.0 31.7

Note. Rows sum to 100%.

Table A3. In-Sample Cross-Validation Prediction Errors for HUI-3 Score Range by Estimation Model

HUI3 scorerange

Distributionof observedHUI3 scores

(%)

CBCL Model 1 CBCL Model 2Vineland-II

Model 1Vineland-II

Model 2Vineland-II

Model 3 PedsQL Model 1 PedsQL Model 2CombinedModel 1

CombinedModel 2

MAPE (SE) MAPE (SE) MAPE (SE) MAPE (SE) MAPE (SE) MAPE (SE) MAPE (SE) MAPE (SE) MAPE (SE)

< 0 0.9 0.5950 (0.0012) 0.6021 (0.0012) 0.5790 (0.0018) 0.5571 (0.0012) 0.5853 (0.0010) 0.5221 (0.0026) 0.5253 (0.0041) 0.5108 (0.0019) 0.5274 (0.0040)0–0.249 6.9 0.4210 (0.0023) 0.4076 (0.0026) 0.2791 (0.0026) 0.3113 (0.0025) 0.3118 (0.0024) 0.3473 (0.0022) 0.3364 (0.0025) 0.2952 (0.0030) 0.3042 (0.0035)0.25–0.499 13.3 0.2251 (0.0021) 0.2168 (0.0020) 0.2027 (0.0022) 0.2245 (0.0018) 0.2197 (0.0017) 0.2011 (0.0020) 0.1931 (0.0020) 0.1787 (0.0022) 0.2044 (0.0016)0.5–0.699 28.0 0.0976 (0.0009) 0.0936 (0.0009) 0.1260 (0.0010) 0.1015 (0.0008) 0.1020 (0.0009) 0.1157 (0.0010) 0.1075 (0.0009) 0.1200 (0.0010) 0.1115 (0.0009)0.7–0.799 17.4 0.1284 (0.0012) 0.1365 (0.0012) 0.1126 (0.0014) 0.1009 (0.0010) 0.1052 (0.0009) 0.1244 (0.0012) 0.1164 (0.0012) 0.1157 (0.0010) 0.1041 (0.0009)0.8–0.899 23.4 0.1600 (0.0012) 0.1574 (0.0013) 0.1591 (0.0017) 0.1441 (0.0013) 0.1382 (0.0012) 0.1334 (0.0012) 0.1418 (0.0011) 0.1363 (0.0014) 0.1454 (0.0013)0.9–1.0 10.1 0.2659 (0.0019) 0.2433 (0.0019) 0.2567 (0.0022) 0.2631 (0.0016) 0.2607 (0.0017) 0.2024 (0.0021) 0.2285 (0.0022) 0.2235 (0.0019) 0.2356 (0.0019)Full index 100 0.1819 (0.0008) 0.1777 (0.0008) 0.1681 (0.0008) 0.1674 (0.0007) 0.1666 (0.0007) 0.1591 (0.0007) 0.1618 (0.0007) 0.1592 (0.0007) 0.1610 (0.0007)

MAPE, mean absolute prediction error.

15Payakachat et al./Mapping outcome measures to health utilitiesINSAR