Embed Size (px)
Citation preview
Priority Setting in Pediatric Preventive Care Research
by
Mikael Jacob Katz Lavigne
A thesis submitted in conformity with the requirements for the degree of Master of Science
Institute of Health Policy, Management and Evaluation Dalla Lana School of Public Health
University of Toronto
© Copyright by Mikael Jacob Katz Lavigne, 2018
ii
Priority Setting in Pediatric Preventive Care Research
Mikael Jacob Katz Lavigne
Master of Science
Institute of Health Policy, Management and Evaluation
Dalla Lana School of Public Health
University of Toronto
2018
Abstract
Objectives: To identify unanswered research questions in pediatric preventive care that are most
important to parents and clinicians and to assess differences in priorities between the two groups.
Methods: Unanswered research questions were collected from parents of children aged 0-5 years
and clinicians via an online questionnaire using a James Lind Alliance-developed methodology.
Similar submissions were combined and ranked. Parents and clinicians selected the 10 most
important unanswered research questions at a consensus workshop. The categories of questions
from parents and clinicians were compared.
Results: 1046 submissions were combined into 79 indicative questions. The top 10 unanswered
research questions related to mental health, parental stress, physical activity, obesity, childhood
development, behaviour management and screen time. Parents were more likely to ask questions
about screen time and environmental toxins.
Conclusions: The top 10 most important unanswered research questions in pediatric preventive
care from the perspective of parents and clinicians were identified.
iii
Acknowledgments
I have had the good fortune to work on this project with many accomplished individuals. Firstly,
I would like to thank my thesis supervisor, Dr. Andreas Laupacis, for his invaluable guidance
and for the countless hours spent supporting me through this process. To Dr. Catherine Birken, I
appreciate the support you have given me since I first worked with you as a resident and
continuing through to this project. I would like to thank Dr. Jonathon Maguire for his sage advice
and Dr. Sharon Straus for her excellent feedback and suggestions. I would also like to thank Dr.
Patricia Parkin for her support and encouragement.
Katherine Cowan and Sally Crowe from the James Lind Alliance provided feedback at the onset
of the project, which was much appreciated. I also appreciate the assistance of several present
and former members of the TARGet Kids! research network including Dr. Cory Borkhoff, Dr.
Kawsari Abdullah, Matthew D’Ascanio and Victoria Latimer.
I would like to thank the members of the Steering Group of this study for their hard work
through the multiple steps of this priority setting process, as well as the parents, clinicians and
group facilitators who attended the final workshop. Yang Chen and Dr. Gerald Lebovic,
biostatisticians at the Applied Health Research Centre at St. Michael’s Hospital, provided
assistance with the statistical methods. The Pediatricians’ Alliance of Ontario (PAO) provided
invaluable support during the priority setting process and final workshop.
Thank you to my mother Linda and father Jean-François for the encouragement and motivation
to help me get to where I am today, and to my parents-in-law Brian and Sarah for their support.
Thank you to my sister Sarah for showing me how far hard work and dedication can get you.
Most importantly I would like to thank my wife Mollie, whose incredible energy, love and
support motivated me every day, and my son Benjamin, who makes every day better than the
last. Ben, I dedicate this work to you with all my love.
iv
Table of Contents
Contents
Acknowledgments .......................................................................................................................... iii
Table of Contents ........................................................................................................................... iv
List of Tables ............................................................................................................................... viii
List of Figures ................................................................................................................................ ix
List of Appendices .......................................................................................................................... x
Chapter 1 Introduction .................................................................................................................... 1
Background ................................................................................................................................ 1
1.1 Preventive care in pediatrics ............................................................................................... 1
1.1.1 Existing pediatric preventive care guidelines ......................................................... 2
1.1.2 Gaps in pediatric preventive care recommendations .............................................. 3
1.2 Involving healthcare consumers and providers in research priority setting ....................... 4
1.2.1 Differences in the priorities of researchers and healthcare consumers ................... 4
1.2.2 The James Lind Alliance ........................................................................................ 6
1.2.3 Differences between parents and clinicians ............................................................ 6
1.3 Primary and Secondary Objectives ..................................................................................... 7
Chapter 2 Review of the Literature ................................................................................................. 8
Priority Setting Methodology ..................................................................................................... 8
2.1 Clinical burden and cost effectiveness ................................................................................ 9
2.2 Types of Priority Setting Methodologies .......................................................................... 10
2.2.1 Citizens’ juries ...................................................................................................... 11
2.2.2 Delphi method ....................................................................................................... 11
2.2.3 Nominal group technique ...................................................................................... 12
v
2.2.4 Scoping Reviews and Global Evidence Mapping ................................................. 13
2.2.5 Dialogue Method .................................................................................................. 14
2.2.6 The James Lind Alliance Protocol ........................................................................ 15
2.3 Priority Setting in Pediatric Medicine and Primary Care Settings ................................... 17
2.3.1 Priority Topics for Parents and Clinicians in Primary Care ................................. 17
2.4 Research Priority Setting in Pediatrics ............................................................................. 21
2.5 Funded PSP research questions ........................................................................................ 22
2.6 Barriers in patient involvement in research ...................................................................... 23
Chapter 3 Research Methodology ................................................................................................. 26
Methods .................................................................................................................................... 26
3.1 Study Design ..................................................................................................................... 26
3.1.1 Study Setting: The TARGet Kids! Research Network ......................................... 26
3.1.2 Steering Group ...................................................................................................... 26
3.1.3 Questionnaire Design ............................................................................................ 27
3.1.4 Online Questionnaire Implementation .................................................................. 28
3.1.5 Questionnaire Face Validity ................................................................................. 28
3.2 Data Collection and Analysis ............................................................................................ 28
3.2.1 Steps Involved in Interim and Final Research Priority List Generation ............... 28
3.2.2 Final Workshop: Nominal Group Technique ....................................................... 30
3.2.3 Data Security and Protection of Personal Information ......................................... 31
3.2.4 Secondary Outcome: Statistical Analysis ............................................................. 31
3.3 Research Ethics Board Approval ...................................................................................... 31
Chapter 4 Results .......................................................................................................................... 33
Results ...................................................................................................................................... 33
4.1 Demographic Characteristics ............................................................................................ 33
4.1.1 Questionnaire Respondents: Parents and Clinicians ............................................. 33
vi
4.1.2 Workshop Participants .......................................................................................... 34
4.2 Identification of Unanswered Research Questions ........................................................... 34
4.2.1 Questionnaire submissions .................................................................................... 34
4.2.2 Analysis of questionnaire submissions ................................................................. 34
4.2.3 Generation of Interim List .................................................................................... 35
4.2.4 Workshop .............................................................................................................. 37
4.2.5 Primary Outcome: Final List of Unanswered Research Questions ...................... 37
4.3 Secondary Outcome: Comparison between Parents and Clinicians ................................. 38
4.4 Participant Experience during Workshop ......................................................................... 39
Chapter 5 Discussion and Conclusion .......................................................................................... 40
Discussion ................................................................................................................................ 40
5.1 Top Research Priorities in Pediatric Preventive Care ....................................................... 40
5.2 Priority Setting Process: Lessons Learned ........................................................................ 41
5.2.1 Steering Group ...................................................................................................... 41
5.2.2 Generation of the Top 10 Priorities ...................................................................... 42
5.2.3 Participant Perceptions .......................................................................................... 42
5.3 Strengths and Limitations ................................................................................................. 43
5.4 Impact of the Top 10 List .................................................................................................. 44
5.5 Future Research ................................................................................................................ 45
5.6 Conclusion ........................................................................................................................ 45
References ..................................................................................................................................... 46
Appendices .................................................................................................................................... 63
Appendix 1: Modified JLA methodology flow chart ............................................................... 64
Appendix 2: Questionnaire ...................................................................................................... 65
Appendix 3: Master List of Indicative Questions .................................................................... 67
Appendix 4: Interim Ranked List of Unanswered Research Questions ................................... 70
vii
Appendix 5: Final Top 20 Ranked List of Unanswered Research Questions .......................... 72
Appendix 6: Composition of the Steering Group .................................................................... 73
Appendix 7: The TARGet Kids! Primary Care Research Network ......................................... 74
Copyright Acknowledgements ...................................................................................................... 75
viii
List of Tables
Table 1: Demographic information for questionnaire respondents
Table 2: Interim Ranked List of Unanswered Research Questions
Table 3: Final ranked list of the Top 20 unanswered research questions
Table 4: Questions submitted by parents and clinicians by category
ix
List of Figures
Figure 1: The James Lind Alliance Priority Setting Partnership (PSP) process.
Figure 2: Flow chart of research priority setting process.
x
List of Appendices
Appendix 1: Modified JLA methodology flow chart
Appendix 2: Questionnaire
Appendix 3: Master List of Indicative Questions
Appendix 4: Interim Ranked List of Unanswered Research Questions
Appendix 5: Final Top 10 Ranked List of Unanswered Research Questions
Appendix 6: Composition of the Steering Group
Appendix 7: The TARGet Kids! Primary Care Research Network
1
Chapter 1 Introduction
Background
1.1 Preventive care in pediatrics
Preventable disease exerts a profound toll on the health of children and adults. In the United
States, the Centers for Disease Control (CDC) state that “Chronic diseases, such as heart disease,
cancer, and diabetes, are responsible for 7 of every 10 deaths among Americans each year and
account for 75% of the nation’s health spending” (Centers for Disease Control, 2017). In a study
of three cohorts of children from the National Longitudinal Survey of Youth-Child Cohort
between 1988 and 2006, chronic diseases such as obesity were present in 12.8% of cohort 1
(followed from 1988 to 1994), 25.1% in cohort 2 (from 1994 to 2000) and 26.6% of cohort 3
(followed from 2000 to 2006) (Van Cleave, Gortmaker, & Perrin, 2010). For both chronic and
acute illness, preventive care may help to “modify important disease-defining risk factors”
(Boyle, Perrin, & Moyer, 2014). Disease prevention efforts may focus on “health outcomes,
which represent physical and emotional well-being and optimal functioning at home, in school
and in the community” (Bright Futures, 4th edition).
There are many examples of preventive care measures that are implemented during childhood
and adolescence and aimed at maintaining optimal health (Bright Futures, 3rd Edition, 2007),
Melnyk et al., 2012). Immunization decreases the burden of vaccine-preventable illnesses such
as measles, which caused the death of more than 500,000 children worldwide in 2003 (UNICEF,
2017). Iron deficiency anemia screening is recommended for selected patient populations such as
premature infants or those with a low birthweight (Bright Futures, 3rd Edition, 2007). Hearing
screening is mandated for newborns across Canada and in most states in the USA because there
is high quality evidence that such screening helps babies with hearing loss receive appropriate
assessment and intervention more quickly (Bright Futures, 3rd Edition, 2007). Although many
preventive care efforts are aimed at adults, prevention starts before conception (Moos et al.,
2008). For example, women of childbearing age who are considering pregnancy are encouraged
to take folic acid to prevent neural tube defects (Moos et al., 2008).
2
Childhood provides an important window for families and healthcare providers to implement
evidence-based preventive healthcare measures in order to set a foundation for good health
throughout the individual’s life (Center on the Developing Child at Harvard University, 2010).
For example, changes in an individual’s BMI in childhood and adolescence may affect their BMI
and the amount of subcutaneous fat in adulthood, providing a potential target to modify a disease
course later in life (Kindblom et al., 2009). A particular focus is placed on prevention during
well-child visits as this is the setting in which most screening guidelines are implemented (Van
Cleave et al., 2012).
1.1.1 Existing pediatric preventive care guidelines
Several organizations provide preventive care recommendations for individuals under the age of
18 years. In Canada, the Canadian Pediatric Society (CPS) publishes position statements aimed
at promoting evidence-based pediatric care, among which are preventive care guidelines
(Canadian Pediatric Society, 2017). The CPS also distributes the Rourke Baby Record (Rourke,
Leduc, & Rourke, 2014) and the Greig Health Record (Greig, Constantin, Carsley, & Cummings,
2010), which are guidelines that can be used for well-child visits throughout a child’s journey to
adulthood. Another organization, the Canadian Task Force on Preventive Health Care
(CTFPHC), was created in 1976 and develops “clinical practice guidelines that support primary
care providers in delivery preventive health care” (Canadian Task Force on Preventive Health
Care, 2017a). The CTFPHC has been a pioneer in promoting evidence-based preventive care
guidelines (Moyer & Butler, 2004). In the United States, the American Academy of Pediatrics
(AAP) publishes the Bright Futures guidelines, which informs screening and prevention during
pediatric assessments (Bright Futures, 3rd Edition, 2007). The U.S. Preventive Services Task
Force (USPSTF) also issues specific recommendations for preventive care for children and
adolescents (U.S. Preventive Services Task Force, 2014). In the United Kingdom, the National
Institute for Health and Care Excellence (NICE) publishes guidelines that review such topics as
promoting healthy lifestyle in children under 5 years of age (National Institute for Health and
Care Excellence, 2016), and contraception and sexually transmitted infection (STI) prevention
(National Institute for Health and Care Excellence, 2007).
3
1.1.2 Gaps in pediatric preventive care recommendations
Many preventive care recommendations for children lack high quality supportive evidence
(Kistin, 2011; Melnyk et al., 2012; Moyer & Butler, 2004). A review of the recommendations
from 7 healthcare organizations in the U.S. and Canada found that 42 interventions were
recommended by at least 2 of these organizations but a randomized controlled trial had only been
conducted for 2 of these interventions (Moyer & Butler, 2004). The pediatric guidelines
currently available from the CTFPHC, about developmental delay, obesity in children and
tobacco smoking in children and adolescents all comment on a mixed evidence base with
multiple recommendations informed by low quality evidence (Canadian Task Force on
Preventive Health Care, 2015, 2016, 2017b). Of 54 recommendations by the UPSTF for child or
adolescent preventive care, 20 (37%) were based on poor quality evidence (U.S. Preventive
Services Task Force, 2014). The Bright Futures Guidelines published by the American Academy
of Pediatrics have also acknowledged numerous gaps in the evidence for pediatric preventive
care (Bright Futures, 3rd Edition, 2007). In a report to the U.S. Congress in 2014, the USPSTF
identified 7 areas in child and adolescent health that should be priorities for further research on
preventive care (U.S. Preventive Services Task Force, 2015). These areas were identified in large
part because of insufficient evidence to provide guidance around the utility of related
interventions. The USPSTF commented that there is a “lack of data from clinical research trials”
and it is difficult to assess “health benefits in children and adolescents, especially for those
outcomes that might not occur for years” (U.S. Preventive Services Task Force, 2015). The
priority areas for research were identified as follows:
1. “Mental health conditions and substance abuse,
2. Obesity and Cardiovascular Health
3. Behaviour and Development
4. Infectious Diseases
5. Cancer Prevention
6. Injury and Child Maltreatment
7. Vision Disorders.” (U.S. Preventive Services Task Force, 2015)
Given the extent of the evidence gaps in pediatric screening and prevention, and the limited
availability of research funding, it is important to set priorities about which important topics
should be the subject of health research. How should preventive care research topics for children
be prioritized?
4
1.2 Involving healthcare consumers and providers in research priority setting
There are many reasons to involve patients and families in the identification of research
priorities. Doing so gives a voice to healthcare consumers who have become more and more
involved in planning the care they receive and in developing appropriate tools and services for
patients (Oliver et al., 2004). Involving the public (patients, caregivers and other stakeholders) in
setting research priorities may lead to the funding of research that is of higher relevance to
patients (Brett et al., 2014; Chalmers et al., 2014). Involving patients and families in the design,
planning and execution of the research process has been thought to lead to greater buy-in and a
greater focus on issues of relevance to healthcare consumers (Stewart & Oliver, 2008). Indeed,
research involving the public may improve the “credibility of the research with patient/parent
groups and professionals” (Shen et al., 2017). It may also produce research that “has a higher
methodological or ethical quality; produces findings which are more relevant to practical
decisions made by consumers and those caring for them; is presented in more accessible and
widely disseminated reports; or more appropriately influences policy and practice” (Boote,
Wong, & Booth, 2015; Oliver et al., 2004). It has been argued that public involvement in health
research is important because “health research can benefit from the experiential knowledge and
personal insights of patients, carers and service users … the public have a right to be involved in
any publicly funded research that may impact on their health status or the services they receive”
(Boote et al., 2015).
1.2.1 Differences in the priorities of researchers and healthcare consumers
It has been well documented that the priorities of researchers may not align well with the
priorities of patients and providers (Crowe, Fenton, Hall, Cowan, & Chalmers, 2015; Jun et al.,
2015; Murad et al., 2011; Tallon, Chard, & Dieppe, 2000). Although patients and providers are
increasingly recognized as important research partners, a systematic map of priority setting
studies found that only 19% of 258 studies involved both patients and clinicians in the process
(Stewart & Oliver, 2008).
Differences between the priorities of osteoarthritis patients and researchers include that patients
prioritize research into lifestyle modification and physical therapy, as compared to a
5
preponderance of studies on medications in the literature (Tallon et al., 2000). In the field of
organ transplantation, a top 10 list of research priorities identified at a workshop by 10 patients
and 5 researchers were compared to the research topics from 1658 articles published in two
major transplantation-related journals between 2012 and 2014 (Allard et al., 2017). There were
significant differences between the literature and the top 10 list, as only approximately 25% of
the published research studies related to the identified priorities and 2 priorities accounted for
82% of the related articles (Allard et al., 2017). Crowe et al. extracted the treatment-related
questions identified by patients and families during research priority setting projects conducted
between 2007 and 2012 and compared them to treatments that were studied in clinical trials in
the UK during that time period (Crowe et al., 2015). The priority setting project questions
relating to treatment were less likely to be about medications (23/126, 13%) than the treatments
studied in industry-sponsored clinical trials (689/798, 86%) and clinical trials sponsored by non-
commercial organizations (397/1069, 37%) (Crowe et al., 2015).
In type 1 diabetes, which is often diagnosed in childhood, a review of public priorities for
research compared to the topics of funded studies showed that there were areas of similarity but
also notable areas of divergence (Boddy, Cowan, Gibson, & Britten, 2017). Patients with type 1
diabetes and their caregivers provided 859 research questions that were compared to 172
research projects on type 1 diabetes funded between January 2010 and December 2011 in the
United Kingdom (Boddy et al., 2017). These projects were funded by charities (75/172, 43.6%),
the UK government (35/172, 20.3%), industry (5/172, 2.9%) or mixed funding sources (57/172,
33.1%) (Boddy et al., 2017). Although agreement was found between the public perspective and
funded research on the importance of blood sugar control, complications of diabetes and research
into cures for the condition, notable disagreements occurred in several areas. Patients and their
caregivers were more concerned about variations in care delivery based on geographic location
and local health policy, whereas funded research more often focused on factors leading to low or
non-adherence with diabetes screening and treatment. In addition, differences were noted related
to means of insulin administration other than injection, psychosocial issues and the relationship
of diabetes with women’s health (Boddy et al., 2017). The authors suggested that research may
not be focused sufficiently on questions about the day to day struggles of patients with type 1
diabetes, especially related to access to care and quality of life.
6
1.2.2 The James Lind Alliance
The James Lind Alliance (JLA) was developed to prioritize health research on topics that matter
to patients, caregivers and clinicians (Partridge & Scadding, 2004). The JLA became part of the
Evaluation, Trials and Studies Coordinating Centre of the NIHR in the United Kingdom in 2013
(VisionUK, 2013). James Lind was a British Navy sailor and physician who conducted what is
considered to be the first clinical trial which assessed the efficacy of different types of treatments
for scurvy in sailors (Magiorkinis, Beloukas, & Diamantis, 2011).
The JLA aims to address the “mismatch between the research being carried out and the research
evidence needed by patients and clinicians every day” (Crowe et al., 2015). This mismatch leads
to a suboptimal use of research funds, and potential waste (Chalmers & Glasziou, 2009). The
JLA developed a Priority-Setting Partnership (PSP) process that brings these patients, caregivers
and clinicians together to prioritize health research that is of value to the consumer (Cowan &
Oliver, 2013). The PSP process starts with the identification of potentially unanswered research
topics and culminates with an in-person workshop to finalize a top 10 list of the most important
research questions (James Lind Alliance, 2016). A number of PSPs related to child health have
been conducted and have identified lists of important research questions related to asthma,
childhood disability, pediatric mental health, lower limb surgery in children and other topics
(James Lind Alliance, 2018). These PSPs as well as the JLA methodology are discussed in
greater detail in Chapter 2.
1.2.3 Differences between parents and clinicians
Although the James Lind Alliance process promotes consensus-building among patients,
caregivers and clinicians, these different groups may not have the same viewpoints on research
topics of importance. Patient and caregiver groups may prioritize questions that improve the
quality of healthcare that they or their family members receive, while clinicians may focus on
questions that fill evidence gaps that affect the care provided to patients (Boney, Nathanson,
Grocott, Metcalf, & Steering Group for the National Institute of Academic Anaesthesia/James
Lind Alliance Anaesthesia and Peri-operative Care Priority Setting Partnership, 2017).
A secondary analysis of 50 research priorities from a PSP related to anesthesia and peri-operative
care compared the priorities of patients with those of clinicians; some of the clinicians had also
7
received anesthesia or peri-operative care themselves (Boney et al., 2017). Patients and clinicians
had identified important research priorities in three categories: clinical effectiveness (25 out of
50 total questions analyzed), patient experience (19 out of 50 questions) and patient safety (6 out
of 50 questions). Clinicians – including those who had been patients themselves - were
significantly more likely to prioritize questions about clinical effectiveness and patient safety
than patients, who considered the patient experience questions more important (Boney et al.,
2017).
1.3 Primary and Secondary Objectives
The primary objective of this study was to identify the 10 most important unanswered research
questions in pediatric preventive care from the perspective of parents and clinicians using the
JLA approach. The secondary objective was to identify any similarities and differences between
the research questions submitted by parents and clinicians.
8
Chapter 2 Review of the Literature
Priority Setting Methodology
In health research, there are many different methodologies used for priority setting. Dubois and
Graff assessed relevant published literature and the priority setting methods used by nine
organizations including the Agency for Healthcare Research and Quality in the United States, the
NICE in the United Kingdom, and the World Health Organization (Dubois & Graff, 2011). They
proposed a framework for priority setting in comparative effectiveness research that involves the
following steps:
• Defining priority research areas more specifically
• Engaging all stakeholders
• Setting clear criteria for prioritization
• Assessing the evidence related to each priority topic
• Reaching consensus with a transparent and rigorous methodology
• Communicating the methods and results in a clear manner
• Update research priorities with newly available evidence
• Perform an evaluation of the priority setting process (Dubois & Graff, 2011)
When setting criteria for prioritization, Dubois and Graff suggested considering criteria related to
the condition being studied, which are “Prevalence, incidence; clinical burden of illness; cost
(per patient, in aggregate); current variation in care/outcomes; current level of evidence gaps;
addresses special population” as well as criteria related to the research itself, including “Cost to
conduct research; time frame to complete research; likelihood of research success; likelihood the
research will influence care; likelihood that others would fund research” (Dubois & Graff, 2011).
Before discussing priority setting methodologies that were used in the generation of the James
Lind Alliance protocol and fit within the framework proposed by Dubois and Graff, it is
important to consider the impact of the burden and cost of disease as factors in research
prioritization.
9
2.1 Clinical burden and cost effectiveness
In order to identify the medical conditions that cause the highest resource use in pediatric
hospitals, Keren et al. examined data from the Pediatric Health Information System (PHIS)
database, which contains information from 43 children’s hospitals in North America (Keren et
al., 2012). The goal of the study was to identify the most important topics for comparative
effectiveness research (CER) based on “prevalence, cost, and variation in care, measured in
terms of the variation in resource utilization for children hospitalized with specific conditions”
(Keren et al., 2012). They identified the top 50 most costly and most prevalent conditions, with a
total of 77 conditions identified. The top ten conditions were responsible for 36% of standardized
costs; these include such diseases as respiratory distress syndrome, pneumonia, idiopathic
scoliosis, hypoplastic left heart syndrome, and bronchiolitis. These conditions were generally
either highly prevalent or had a high per-patient cost, which explains the overall expense. For
example, although hypoplastic left heart syndrome was 63rd out of 77 conditions on the list
identified by Keren et al., the cost per patient for this condition is $104,037. Conversely,
pneumonia is 5th out of 77 on the list of most expensive conditions, but its cost per patient is
$8293. Interestingly, conditions such as anorexia nervosa, which have high per-patient costs,
were not listed. A Canadian study in 2015 showed that mean hospital costs per admission in a
tertiary care centre for patients with AN were $51,349 in Canadian dollars (Toulany et al., 2015).
Though expensive, and despite anorexia nervosa being “third most common chronic condition
affecting adolescent girls, with an incidence of about 5%”, the total number of patients admitted
to hospital is likely not as high as other costly conditions (Toulany et al., 2015).
Keren et al. also calculated an intraclass correlation coefficient (ICC), which determined what
proportion of variability in costs for each condition was accounted for by differences within the
various hospitals in the PHIS database. In this case, “the ICC approaches 0 if variation across
hospitals is small, and the ICC approaches 1 as hospitals begin to account for all variation of
costs” (Keren et al., 2012). Conditions with high ICCs included idiopathic scoliosis (0.37),
hypertrophy of the tonsils and adenoids (0.30), dental caries (0.22), inguinal hernias (0.25) and
hypospadias (0.37).
The list of the most expensive conditions in the PHIS database was recommended by the
research team in this study as a starting point for research priority setting, not only because the
10
conditions cost so much and utilize so many resources, but also because for some of them, there
is substantial variation between hospitals that could represent opportunities for research (Keren
et al., 2012). However, basing a priority setting methodology primarily on expense implies that
other less common or costly conditions may be less important. As described, anorexia nervosa is
costly per patient but does not appear on this list despite being highly prevalent and having a
profound impact on patients and families (Toulany et al., 2015). Conditions primarily managed
in an outpatient setting, such as anorexia nervosa, may not be accounted for, and this
methodology also does not directly involve patients, parents and clinicians in setting research
opportunities. This information may be useful in the context of an overall analysis of research
priorities using multiple factors including the input of healthcare consumers and providers.
2.2 Types of Priority Setting Methodologies
A systematic review by a group from the United Kingdom’s National Institute for Health
Research Health Technology Assessment (HTA) Programme identified techniques that are used
to elicit the preferences of members of the public for healthcare (Ryan et al., 2001). Both
quantitative and qualitative methods were described. Quantitative methodologies included scales
such as the Likert and Guttman scales, which are useful in many settings but are flawed as they
“do not consider strength of preference or the importance of different components within a total
score” (Ryan et al., 2001). Qualitative techniques included such methods as case study analyses,
the Delphi technique, citizens’ juries and the nominal group technique. The review assessed the
various techniques in terms of their “validity; reliability; generalizability; objectivity;
acceptability to respondents; and cost” (Ryan et al., 2001) The HTA review recommended the
Delphi method and citizens’ juries for “eliciting public views on the provision of healthcare”
because of their ability to involve the public and to assess complex questions (Ryan et al., 2001).
Other methods that have been used for research priority setting include the Nominal Group
Technique, Global Evidence Mapping and the Dialogue Method. These methods, along with the
Delphi method and the James Lind Alliance protocol, are considered deliberative as they go
beyond consulting stakeholders through focus groups or surveys and involve discussion and
individual or group decision-making (Mitton, Smith, Peacock, Evoy, & Abelson, 2009). These
five techniques share many similar elements that help to achieve consensus.
11
2.2.1 Citizens’ juries
Citizens’ juries involve presenting expert evidence about a topic to a group of 12 to 16 members
of the public, much like during a judicial trial. Participants are randomly selected, work with one
or two moderators and have the opportunity to cross-examine witnesses called to back up the
points of view presented during the process as well as to discuss their decisions as a group (Ryan
et al., 2001). This process can provide rich information and its format is suited to addressing
complex questions; however, it is time-intensive and costly and may not be generalizable due to
the small number of participants (Ryan et al., 2001). Although they can be used in research
prioritization, citizens’ juries have primarily been used to make health policy decisions (Street,
2014).
2.2.2 Delphi method
The RAND corporation originally created the Delphi method in the 1950s to “synthesize expert
opinion, mainly on the emergence of new technologies” (Murphy et al., 1998). Participants in a
process using the Delphi method do not meet in person, but rather are asked to complete a
questionnaire about their opinion on the issue being discussed. The responses are collected and
summarized and the results sent back to participants, who may change their responses based on
the perspective of the group. This may occur multiple times, and the final product is analyzed
statistically and the extent to which consensus has been obtained is reported to participants (Ryan
et al., 2001).
A research priority setting exercise related to pediatric palliative care and utilizing the Delphi
method was performed in 2008 by a team from several Canadian and American universities.
Steele et al. identified priorities in pediatric palliative care from the point of view of both
researchers and frontline clinicians (Steele et al., 2008). Communication occurred via e-mail.
Participants first individually identified five priority research topics related to pediatric palliative
care; submitted topics were reviewed and categorized. The list of submitted questions was
emailed back to participants who ranked them on a Likert scale from 1 (not important) to 5 (very
important), and 14 questions were retained for a third ranking step as they were considered
important by two thirds of the participants. In the third ranking step, participants were sent the
list of 14 questions via email and were asked to choose the 5 most important questions. Four
12
questions were ranked as important by more than half of the participants. The four topics
identified were:
1) “What matters most for patients and parents receiving pediatric palliative services?
2) What are the bereavement needs of families in pediatric palliative care?
3) What are the best practice standards in pain and symptom management?
4) What are effective strategies to alleviate suffering at the end of life?” (Steele et al., 2008)
The Delphi method was felt to be helpful in this study because it allowed individuals across
Canada to participate. However this study did not involve patients or families in setting priorities
and did not benefit from a face to face meeting for final ranking of priorities (Steele et al., 2008).
When participants complete questionnaires or rank submissions, they are doing so primarily
based on their personal experiences and understanding of the topic of discussion, whereas a face
to face meeting allows participants to integrate a variety of information, experiences and
viewpoints into the decision-making process (James Lind Alliance, 2016). A 2009 scoping
review of 175 studies involving the public in health care priority setting found that studies with a
face-to-face element reported successful outcomes more than those studies without a face-to-face
meeting (Mitton et al., 2009). Of the priority setting studies with face-to-face interaction, 75%
were felt to be successful by each study’s authors, as compared to 49% of studies without face-
to-face interaction, although the criteria for success were not formally defined (Mitton et al.,
2009).
2.2.3 Nominal group technique
The Nominal Group Technique (NGT) is an iterative process that was originally developed by
Van de Ven and Delbecq in 1968 (Totkidis, 2010). The NGT involves a face-to-face meeting
during which the topic of discussion is introduced, followed by each participant individually
recording ideas on the topic (Totkidis, 2010). These ideas are then read out and discussed by the
group, followed by a clarification step which may include combining or eliminating ideas, and a
voting step wherein each participant contributes their point of view on the final ranking of ideas.
As the NGT includes a face-to-face meeting, it cannot be an anonymous process, unlike the
Delphi method (Campbell, 2010). In some applications of the NGT, background information
such as a review of the literature has been provided to participants to improve their ability to
participate in the process (Jones & Hunter, 1995)
13
The Nominal Group Technique is felt to be effective for building consensus and setting
priorities; it prevents individuals from dominating the discussion and is “particularly helpful in
generating ideas and priorities in situations where participants are likely to have diverse views on
a subject where little is known” (Corner et al., 2007). The ranking process bears similarities to
the Delphi method; however, the NGT may provide a greater number of ideas given the real-time
nature of the discussion and may have a “greater potential for creative decision making”
(Totkidis, 2010).
The NGT has also been described as a method that can be modified to take into account “the
available research and participant time, or the level of clarification, consensus or generalizability
required for the topic” (McMillan, King, & Tully, 2016). For example, a panel of clinicians and
family members used the nominal group technique to identify “important early morbidities
associated with cardiac surgery suitable for routine monitoring” in a pediatric population (Pagel
et al., 2017). During two panel meetings attended by 3 family members and 12 clinicians, an
initial 66 questions were proposed, which were narrowed down to 10 questions by the end of the
second meeting, following the NGT method. However, the study team added a step in between
the two panel meetings wherein a group of clinicians decided which morbidities could in reality
be monitored (Pagel et al., 2017). The authors felt that the presence of family members brought
ideas to the table that would not otherwise have been seriously considered, and that the benefit of
the NGT was to “reduce the influence of power differentials and of dominant personalities on
group decision making while retaining the benefit of discussion absent from other systematic
approaches such as Delphi” (Pagel et al., 2017).
2.2.4 Scoping Reviews and Global Evidence Mapping
Scoping reviews are used in “the examination of a broader area to identify gaps in the research
knowledge base, clarify key concepts, and report on the types of evidence that address and
inform practice in the field” (The Joanna Briggs Institute, 2015). They do not examine a specific
question in the way a systematic review would but rather provide a subject overview that can
help identify topics for future systematic reviews (The Joanna Briggs Institute, 2015).
The Global Evidence Mapping method combines a scoping review with a priority setting step to
create evidence maps that identify priority knowledge gaps requiring further study (Bragge et al.,
2011). The three key steps are “Setting the boundaries and context of the map”, “searching for
14
and selection of relevant studies”, and “Reporting on yield and study characteristics” (Bragge et
al., 2011). In the boundary setting step, potential research questions are identified from expert
consultation, literature review, workshops in which questions were identified using the nominal
group technique, and from questionnaires completed by members of the public. These potential
questions are transformed into research questions in the PICO format (Problem, Intervention,
Comparison, Outcome) and then a prioritization step asking participants to rank the identified
questions online then takes place. No face to face prioritization step occurs. Subsequently, the
literature is searched to identify relevant studies to the potential research questions, the results of
the included studies are described in the evidence map, and high priority questions for study are
described based on the prioritization step (Bragge et al., 2011). One study focusing specifically
on “rehabilitation and long-term care in traumatic brain injury (TBI) patients” used the Global
Evidence Mapping method but also held a forum attended by 116 members at the end of the
process to identify priority research themes in this area (Clavisi, Bragge, Tavender, Turner, &
Gruen, 2013). Challenges with this process included “stakeholder engagement, coordination,
communication, capacity building, databases, and infrastructure needs” (Clavisi et al., 2013).
2.2.5 Dialogue Method
The Dialogue Method is “grounded in the notion of participation as dialogue” and consists of six
phases: “exploration; consultation; prioritization; integration; programming; and
implementation” (Abma & Broerse, 2010). First, researchers involve stakeholders and the public
in the assessment of research needs and identification of relevant research questions and then
prioritize these questions through various methods including questionnaires, focus groups and
the Delphi method (Abma & Broerse, 2010). A “dialogue meeting” then takes place, which is a
face-to-face meeting at which various stakeholders integrate the research questions into one list
and prioritize them by importance (low, medium or high); attempts are made to balance the
number of researchers/clinicians and patients but this is noted as a limitation as it has been
challenging to strike that balance (Abma & Broerse, 2010). Finally, a research program is
developed and implemented based on the identified priorities, occurring in conjunction with
funding bodies who are sometimes involved from the start of the project (such as in a study
about intellectual disability using the Dialogue Method). This method has been used for a variety
of studies including burns, diabetes, spinal cord injury, asthma/COPD, congenital heart disease
in children, and others (Abma & Broerse, 2010; Abma, Pittens, Visse, Elberse, & Broerse, 2015;
15
Caron-Flinterman, Broerse, Teerling, & Bunders, 2005; Elberse, Caron‐Flinterman, & Broerse,
2011). Other challenges include difficulty with recruitment, giving the public and researchers an
equal say especially in face-to-face workshops, and the complexity of the project.
2.2.6 The James Lind Alliance Protocol
The James Lind Alliance priority setting protocol was originally designed to “bring patients,
their caregivers and clinicians together to identify and prioritize unanswered questions about the
effects of treatments (‘treatment uncertainties’) in specific conditions or areas of healthcare for
research” (James Lind Alliance, 2016). A “treatment uncertainty” occurs when there are no
recent systematic reviews about the impact of a specific treatment, or if research shows that a
treatment’s effect is not clear (James Lind Alliance, 2016). Many recent priority setting
partnerships have moved away from the original model of focusing exclusively on treatment
uncertainty and have included “other health care interventions such as prevention, diagnosis,
rehabilitation, care and service organization and delivery” (James Lind Alliance, 2016).
The JLA method is intended to give patients and their families and clinicians a voice in
healthcare research (James Lind Alliance, 2016). This method uses an iterative process, shown in
Figure 1, that involves a modified version of the Nominal Group Technique. In this modified
NGT, rather than identifying potential research questions during a face-to-face session, data is
first collected and processed from participants, often in an online questionnaire; potentially
unanswered research questions are then brought to patients, family members and clinicians in a
group session to complete the steps of the NGT such as discussion, clarification and voting that
have been previously described (James Lind Alliance, 2016).
16
Figure 1: The James Lind Alliance Priority Setting Partnership (PSP) process (James Lind
Alliance, 2016)
2.2.6.1 JLA Protocol Steps
Comprehensive details about the JLA protocol can be found in the James Lind Alliance
Guidebook, available on the JLA’s website (James Lind Alliance, 2016). The JLA protocol
closely mirrors the prioritization framework proposed by Dubois and Graff (Dubois & Graff,
2011). The first step is the establishment of a Steering Group that is composed of organizations
and individuals involved with the topic at hand, from charities and government organizations to
patients and families. Duties of the Steering Group can include questionnaire development, data
analysis and involvement in the interim and final priority setting stages (James Lind Alliance,
2016).
Next, potential uncertainties are gathered, often by a questionnaire distributed to patients,
caregivers, clinicians and others. Potential uncertainties may also be obtained from the literature
and clinical practice guidelines. Uncertainties that are submitted are combined where applicable
Establishing Steering Group
Gathering Uncertainties
Data Processing and Verifying Uncertainties
Interim Priority Setting
Final Priority Setting
Dissemination of top 10 priorities
17
and assessed to see if they are answered by the health literature. Those that are not considered
answered (i.e. “true uncertainties”) are ranked by study participants during an interim ranking
step. This ranked list is taken to a final ranking workshop where participants work with
facilitators and follow a modified Nominal Group Technique to reach consensus. The workshop
consists of multiple small group discussion sessions, each followed by a ranking step,
culminating in a final group ranking of the most important research uncertainties (James Lind
Alliance, 2016).
2.2.6.2 Why the JLA protocol?
The James Lind Alliance Priority Setting Protocol was chosen for this study for several reasons.
Firstly, the JLA protocol has been described as the “best-researched public-clinician partnership
approach for research topic generation” and is commonly used to identify important research
priorities from the perspective of patients, family members and clinicians, especially in the
United Kingdom (as discussed in section 2.4) (Nass, Levine, & Yancy, 2012). It combines the
benefits of in-person discussion and voting provided by the Nominal Group Technique with the
advantage of being able to obtain research priorities from a variety of stakeholders (James Lind
Alliance, 2016). Identifying unanswered research questions prior to the in-person portion of the
protocol helps to ensure that workshop participants have the right information to accomplish
their priority setting goals. In addition, one team member within our group has experience with
this methodology.
2.3 Priority Setting in Pediatric Medicine and Primary Care Settings
When seeking to identify the most important unanswered research questions in pediatric
preventive care, it is important to gain an understanding of the topics that have been identified as
important by patients, parents and caregivers, especially in a primary care setting, where
preventive care is often provided.
2.3.1 Priority Topics for Parents and Clinicians in Primary Care
A number of studies have examined the content of well-child visits including the topics that
parents and clinicians consider important to address during primary care visits for children. A
literature review was performed to identify studies that focused on this question. We performed a
18
search of Medline, Embase and CINAHL for broad preventive care topics identified in the
literature as important to parents and clinicians. In particular, we sought to identify studies that
discussed questions that parents ask their doctors, especially in a primary care setting and with
regards to keeping children healthy as they grow up, as well as studies that examined the reasons
for which parents bring their children to see their doctor – both in terms of scheduled and
unscheduled visits. Ten studies were identified and used to inform the study questionnaire’s
questions about specific topics (Burklow, Vaughn, Valerius, & Schultz, 2001; Garbutt et al.,
2012; Hickson, Altemeier, & O’Connor, 1983; Norlin, Crawford, Bell, Sheng, & Stein, 2011;
Norlin, Sharp, & Firth, 2007; Nunes & Ayala, 2011; Olson et al., 2004; Radecki, Olson, Frintner,
Tanner, & Stein, 2009; Stickler, 1996; Van Cleave, Heisler, Devries, Joiner, & Davis, 2007).
In 2001, Burklow et al. asked 143 parents enrolled in primary care practices in urban and
suburban settings to identify which topics described as “psychosocial concerns” affected their
children and which were actually discussed during visits with the physicians (Burklow et al.,
2001). The topics included discipline and school performance, among others, and were discussed
less than 50% of the time they actually occurred. The topics felt to be of greatest importance to
parents out of the 9 originally described were “discipline problems (n=104, 72.7%), school
struggles (n=77, 53.8%), sibling difficulties (n=74, 51.7%) and the effects of witnessing violence
(n=72, 50.3%)” (Burklow et al., 2001).
A research team from the Washington University School of Medicine published a study in 2012
which identified “current health-related issues of concern” by asking 1119 parents to rate a list of
30 items by importance (Garbutt et al., 2012). The list of 30 items was modified from the
University of Michigan C S Mott Children’s Hospital National Poll on Children’s Health, a
nationally representative survey that “measures parental attitudes, experiences and priorities
regarding health-related issues and trends for US children” (C.S. Mott Children’s Hospital,
2017). Ten extra items were added on top of the original 20 based on expert opinion (Garbutt et
al., 2012). Problems listed as important to parents, namely identified as being of “large” or
“medium” importance on a scale including “small” or “no problem”, included “allergies (69%),
lack of exercise (68%), asthma (65%), attention deficit hyperactivity disorder (65%), internet
safety (63%), obesity (59%), smoking (58%), and bullying (57%)” (Garbutt et al., 2012). Parents
also provided their own concerns which included “healthy nutrition; obesity; lack of exercise,
19
healthy growth and development; safety and injury prevention; and mental health issues”
(Garbutt et al., 2012).
The study by Garbutt et al. also identified that the concerns elicited from parents varied between
age groups. Of note, the primary concern by far for children from 0 to 5 years of age was
“diseases” (99.8% selected by parents for children under 2 years of age, 86.7% selected by
parents of children aged 2 to 5 years), particularly acute infectious diseases under 2 years of age
and allergies for children from 2 to 5 years of age (23% and 26% respectively). Other important
topics for children from 0 to 5 years of age included food/activity, healthy nutrition, safety and
development (Garbutt et al., 2012).
To assess what proportion of Bright Futures guidelines were being implemented in primary care
well-child visits, as well as to examine what percentage of clinicians asked open-ended questions
to identify concerns from children or parents, Norlin et al. observed 483 interactions between
patients and clinicians (43 pediatricians, 6 nurse practitioners and 3 physician assistants) (Norlin
et al., 2011). This study found that the clinician asked the parent or child an open-ended question
about concerns to begin the visit less than 40% of the time; overall, however, an open-ended
question was asked at least once in 76.8% of the observed visits. When an open-ended question
was not asked by the clinician, the child or parent brought up their own questions without
prompting in 47.3% of the relevant visits. In addition, 58% of recommendations in the Bright
Futures guidelines were not discussed during these visits with children aged 0 to 19 years of age.
Topics that were not discussed as often as specified in the guidelines included “family support,
parental well-being, behavior/discipline, physical activity, media screen time, risk
reduction/substance use, puberty/sex, social-peer interactions, and violence” (Norlin et al., 2011).
It was also interesting to note that Norlin et al. identified 3 topics in each of four pediatric age
groups that were least addressed in the observed primary care visits: in infancy, family support;
in early childhood, parental well-being; in middle childhood, risk reduction/substance use; and in
adolescence, violence. In fact, violence was only discussed in one visit, whereas the Bright
Futures guidelines recommend addressing this topic at 67% of visits.
Another study published by Norlin, Sharp and Firth in 2007 studied 890 well-child and sick
visits with 35 pediatricians and identified questions posed by patients & families that could not
be answered by the clinician (Norlin et al., 2007). The relative importance of these questions was
20
also assessed from the perspective of the 35 pediatricians. Questions that could not be answered
by the clinician were asked in 19.1% of visits (17.3% of well child visits and 19.7% of sick child
visits); these questions were judged to be important in 60.1% of cases. A concern that there is
insufficient evidence played a role in physicians only seeking to answer 27.5% of these questions
(Norlin et al., 2007).
Nunes and Ayala assessed discussion topics during clinical assessments with pediatricians in a
study published in 2011. During 49 visits to 5 pediatricians, 14.3 topics were discussed at each
visit, primarily “feeding (100%), diseases (71%), psychomotor development (67%) and growth
(33%)” (Nunes & Ayala, 2011). In addition, the education level of the parent attending the visit
did not influence the content of the discussion during the visit.
In 2004, data was reported from the National Survey of Early Childhood Health performed in the
United States from the parents of 2068 children between 4 and 35 months of age (Olson et al.,
2004). Data from the Periodic Survey of Fellows from the American Academy of Pediatrics was
used to compare parental and pediatrician rankings of the frequency of discussion of topics
during well-child (Olson et al., 2004). The topics most frequently discussed were
“immunizations, feeding issues and sleep patterns”; other topics addressed less often included
those related to development and to the family, such as “reading (discussed for 61% of children
19-35 months) and child care (discussed for 26% of children 19-35 months)” (Olson et al.,
2004). However, for 36% of children between 4 and 9 months old, and 56% of children between
10 and 35 months old, the parent reported that an important topic was not discussed during the
visit. The primary topics of importance that were not discussed were discipline and toilet
training, and other topics “reported by at least 15% of parents are burn prevention, child care,
reading, vocabulary development, and social development” (Olson et al., 2004). Several factors
made it more likely that topics of discussion would not be addressed during a visit; for example,
families with lower socioeconomic status, those with a shorter visit length and those seeing
pediatric residents discussed fewer topics (Olson et al., 2004).
Other studies focusing on well-child visits include one published in 2009 which consisted of 20
focus groups with parents who valued “an opportunity to discuss patient priorities” such as
“promotion of well-child care, greater emphasis on development and behavior, and expanded
options for information exchange” (Radecki et al., 2009). A survey of 173 parents and 115
21
children in Minnesota identified concerns of importance to the study population, which included
topics such as “discipline, car accidents, eating right, … exposure to environmental poisons and
head injury” among others (Stickler, 1996). Finally, in 2007, a questionnaire was completed by
500 parents in two community-based pediatric practices in Michigan which identified that
parents of children with special healthcare needs discussed 3.2 illness-related topics per visit and
79% perceived illness as one of the 3 most important topics of discussion with their primary care
physician, vs 2.2 illness-related topics and 53% perceived top 3 importance of illness in other
parents (Van Cleave et al., 2007).
Although the topics identified in these studies provide an overview of the topics parents consider
important, the studies were not identified through a systematic review. Therefore, the list of
topics cannot be considered comprehensive. However, these studies identified a variety of topics
about the health and growth of children that are of importance to parents, many of which may be
under-represented in healthcare research, and which create a framework to collect potentially
unanswered research questions from members of the public. In addition, the literature shows that
there are a large number of concerns to be addressed at well-child visits. Many parents feel that
their concerns are not appropriately discussed, and clinicians frequently perceive that they do not
have the evidence to answer questions identified by parents.
2.4 Research Priority Setting in Pediatrics
Pediatric research priority setting studies have focused on a variety of topics and typically tend to
be related to a specific medical condition. The James Lind Alliance website lists 89 Priority
Setting Partnerships, of which at least 23 (25.8%) are related to conditions that can affect
children or involve children in setting research priorities (James Lind Alliance, 2018). A 2015
study sought to identify research priorities related to children with neurodisability and involved
13 charities and 8 professional organizations as partners (Morris et al., 2015). This study
followed the James Lind Alliance protocol and received 809 submissions from 369 respondents
to an online survey, which were narrowed down to 57 potential research questions after merging
similar items. Interestingly, this study directly sought the opinion of children and young people
during the data collection portion, facilitated in part by their partners such as the UK Council for
Disabled Children; 11 individuals under the age of 24 with neurodisability participated, although
the number of people under age 18 in that group was not provided (Morris et al., 2015). The
22
interim ranking step generated a list of 25 questions that were studied at the final ranking
workshop, which was attended by 22 patients, caregivers, charity members or health
professionals. Participants prioritized certain questions over others due to factors including “the
number of children likely to benefit and the intensity of the impact of the symptom, broader
consequences for families, likelihood of research informing clinical, policy or family decisions”
(Morris et al., 2015). The top 3 questions in the final top 10 were about physical therapies,
communication strategies, and ways to improve the attitudes of children towards disabilities.
A PSP about acne identified research uncertainties from 2310 respondents including 844
individuals 16 to 24 years of age; the 30 most common questions were ranked by 2807
participants and subsequently prioritized at a workshop where the top 10 list was “largely
focused on management strategies, optimum use of prescription medications and the role of non-
drug based interventions” (Layton et al., 2015). Priorities related to cleft lip and palate were
generated from 300 individuals providing 600 responses to a questionnaire and the step-wise
creation of a top 10 list which addressed such topics as psychological interventions, relationship
outcomes, best repair techniques and speech therapy (James Lind Alliance, 2012). Other studies
related to conditions present in pediatric patients such as asthma, cystic fibrosis, type 1 diabetes,
eczema, preterm birth and stillbirth produced top 10 lists with questions relating to treatment,
education, psychological effects and support, investigations and other complementary therapies
(Batchelor et al., 2013; Duley, Uhm, Oliver, & Preterm Birth Priority Setting Partnership
Steering Group, 2014; Elwyn et al., 2010; Gadsby et al., 2012; Heazell et al., 2015; Rowbotham
et al., 2017). These PSPs focus on a variety of medical conditions and there are no PSPs related
to the more general concept of preventive care in children (James Lind Alliance, 2018).
2.5 Funded PSP research questions
There has not been a systematic tally of funding provided to studies based on prioritized research
questions, and it may be challenging to ascertain to what extent the top ten lists of research
priorities influence the decisions of funding bodies or the kinds of research questions researchers
decide to address. Top ten lists may exert influence by affecting which studies are supported by
funding bodies such as government agencies or charities, or may encourage researchers to focus
on specific topics; however, formal study of this question is required. Despite this, there are
instances where lists of top ten priorities have influenced research funding decisions. Results
23
from the first priority-setting partnership organized by the James Lind Alliance were published
in 2007 and identified the top 10 research priorities related to asthma (Elwyn et al., 2010; Petit-
Zeman, 2007, p.). The Health Technology Assessment Programme from the NIHR funded a
study related to one of the top 10 priorities: “What are the benefits of breathing exercises as a
form of physical therapy for asthma?” This study, which became the first PSP funded by the
HTA to report its results, was published by Bruton et al. in the Lancet Respiratory Medicine in
December of 2017, and described a randomized controlled trial assessing the impact of a “digital
self-guided breathing retraining intervention” (Bruton et al., 2017). This intervention consisted of
“an illustration of, and training in, diaphragmatic breathing, nasal breathing, slow breathing,
controlled breath holds, and simple relaxation exercises” (Bruton et al., 2017) for patients with
asthma. The primary outcome studied was the score on the Asthma Quality of Life (AQLQ)
questionnaire at the 12-month time point in the trial; intention-to-treat analysis showed that
AQLQ scores were significantly higher in the group performing the breathing training
intervention as compared to the standard care group.
The NHS HTA unit also provided funding for studies focused on four research priorities
identified by patients, families and clinicians during a priority setting project about schizophrenia
(Lloyd, White, & Chalmers, 2012). These priorities include managing the side effects of
medication for schizophrenia (in particular, sexual dysfunction and weight gain), managing
schizophrenia that is refractory to treatment and identifying techniques to catch relapses early
(Lloyd et al., 2012).
2.6 Barriers in patient involvement in research
Several perceived barriers related to patient involvement in research have been addressed in the
literature. Bruni et al. discussed a range of concerns with public engagement such as objectivity,
knowledge level and generalizability of results (Bruni, Laupacis, Martin, & University of
Toronto Priority Setting in Health Care Research Group, 2008). Members of the public involved
in research may not be perceived as objective; however, all members of a research team may
have their own biases (Bruni et al., 2008). Objectivity should not necessarily be expected from
the public who may instead participate in “value-based deliberations”, providing insight about
the “values and priorities of the community, which should lead to higher quality, or at least
greater acceptance of, priority-setting decisions” (Bruni et al., 2008). From a knowledge
24
standpoint, members of the public provide valuable information about the “beliefs of the public
at large” and the “lived experience of using the healthcare system” (Bruni et al., 2008). Although
individuals participating in priority setting may not be representative of the general public, Bruni
et al. argue that other stakeholders participating in decision-making cannot possibly consider all
of the views of healthcare providers and consumers, and that “Fair-minded people from all
relevant constituencies need to be at the table, participate in deliberations and articulate relevant
values. They need not represent all sectors of their communities” (Bruni et al., 2008).
In 2014, Domecq et al. described barriers to patient engagement in research in a systematic
review of 142 studies involving the public in designing and conducting research (Domecq et al.,
2014). These challenges were primarily related to the complexity of the process of engaging the
public in research. Public involvement in research requires a time commitment that may be
challenging for some participants; it may also require transportation to research-related events
and participation in training (Domecq et al., 2014). These commitments may be challenging for
families caring for young children or elderly or sick family members. Financial challenges may
also limit the ability of members of the public to participate in research, although an increasing
number of research projects provide stipends to compensate for the time of participants. Another
concern is that studies could reduce the value of public participation by only making it seem that
patient involvement was inclusive of minority groups (i.e. tokenism) (Domecq et al., 2014).
Finally, there is a worry among some people that public involvement may increase the likelihood
of identifying research questions that are not relevant or cannot be studied (Domecq et al., 2014).
A few studies assessed in the systematic review discussed solutions to these problems, including
“spending adequate time to build reciprocal relationships (between patients and researcher),
fostering mutual respect and developing clear expectations that are explicitly described and
documented in study protocols” (Domecq et al., 2014).
Researchers may have a negative attitude towards patient engagement in research or may face
structural barriers such as high workload and insufficient resources (Burchell, 2015). Assessing
the impact of public engagement is challenging because reporting of this involvement is not
always of high quality (Brett et al., 2014). While public engagement in research is not a perfect
process, patients, families and caregivers provide a unique viewpoint that is different from that of
researchers and clinicians. They have first-hand experience with illness and can be both
motivated and productive. Although they are not generally research content experts, the public
25
can provide an important complement to the opinions of those who currently set the research
agenda so that topics and outcomes of importance to healthcare consumers are prioritized.
26
Chapter 3 Research Methodology
Methods
3.1 Study Design
This research priority setting study was based upon the JLA Priority Setting Partnership protocol
(see Appendix 1) (Cowan & Oliver, 2013). The results have been reported in a paper published
in Archives of Disease in Childhood in 2017 (Lavigne, Birken, Maguire, Straus, & Laupacis,
2017).
3.1.1 Study Setting: The TARGet Kids! Research Network
TARGet Kids is a primary care research network primarily situated in Toronto, Ontario, Canada.
This university-affiliated network’s aim is to advance the scientific basis for chronic disease
prevention and develop solutions for common children’s health problems (Carsley et al., 2015).
TARGet Kids! enrolls children under the age of 6 years being followed by physicians in primary
care clinics (pediatrics or family medicine). This primary care research network was ideal for
this study given the target age group and setting in which they receive care.
3.1.2 Steering Group
The Steering Group was established to coordinate and participate in several key steps of the
priority setting process. A James Lind Alliance adviser was not formally involved in the process.
The research team met with two members of the JLA to discuss the project and to get feedback
prior to the creation of the research protocol.
Five parents and 5 pediatricians were recruited to be members of the Steering Group. The
pediatricians were identified from the list of clinicians working in the various TARGet Kids!-
affiliated clinics, who were contacted via email and asked whether they would be interested in
participating in the study’s Steering Group. When recruiting parents, physicians from several
TARGet Kids!-affiliated clinics were contacted via email to see whether they could suggest
parents who might be interested in being a Steering Group member. These parents were then
contacted via email to ascertain their interest in joining the Steering Group. Potential clinician
27
and parent members were provided information about the role. Once the 5 parents and 5
clinicians had agreed to join the Steering Group, they then participated in a teleconference to
discuss the project and their role going forward. Subsequent communication took place via
email.
The Steering Group played a role in two steps of this priority setting process. Firstly, the Steering
Group members participated in the development of the study questionnaire by reviewing the
draft questionnaire and providing feedback as well as assessing the face validity of the
questionnaire. The members also participated in the interim priority ranking process.
3.1.3 Questionnaire Design
An online questionnaire was developed to obtain research priorities in pediatric preventive care
from parents and clinicians. The questionnaire was implemented online due to the convenience
and accessibility of this method and to ensure that data was collected securely.
The questionnaire was designed to avoid bias; that is, if the questions only focused on specific
areas of interest, important unanswered research questions might be missed. We first used an
open-ended question that asked participants to identify questions about keeping children healthy
as they grow up: “Think about the most important ways that you can help your child stay as
healthy as possible growing up. Please share with us one or several questions that you have about
keeping children healthy” (see Appendix 2).
Participants were also asked to submit questions about specific preventive care topics. As
described above, these questions were derived from ten relevant studies that suggested eight
categories of questions of concern to parents (Burklow et al., 2001; Garbutt et al., 2012; Hickson
et al., 1983; Norlin et al., 2011, 2007; Nunes & Ayala, 2011; Olson et al., 2004; Radecki et al.,
2009; Stickler, 1996; Van Cleave et al., 2007): physical activity, nutrition, growth, parenting,
behaviour, mental health, development, and preventing childhood illness. For example,
respondents were asked “Do you have questions related to child development and learning
(including speaking, reading and writing, motor skills like walking or using a pencil, social
skills, or others)?”
28
3.1.4 Online Questionnaire Implementation
The questionnaire was distributed online using the REDCap platform hosted by the Hospital for
Sick Children Research Institute (Harris et al., 2009). Questions were collected from parents of
children aged 0 to 5 years through the TARGet Kids! research network, as well as from
pediatricians and family physicians through TARGet Kids! and the Ontario Medical Association
(OMA), the organization representing physicians in the province of Ontario, Canada. Eight
hundred parents whose email addresses were provided to TARGet Kids! were asked to complete
the study questionnaire online. Eighty clinicians either within the TARGet Kids! network or
working in Toronto, Ontario, were contacted; in addition, the questionnaire link was distributed
via email to members of the Ontario Medical Association Pediatrics Section. Demographic
information was collected from participants, and clinicians were asked about their specialty and
whether they work in primary care. Responses were submitted anonymously and no personally
identifiable health information was collected.
3.1.5 Questionnaire Face Validity
The face validity of the questionnaire was reviewed by the Steering Group members. After
reviewing the text of the questionnaire, all Steering Group members agreed that from their point
of view, the questionnaire should accomplish its aims of obtaining potentially unanswered
research questions from respondents. The questionnaire was then implemented online and data
was collected from February to November 2014.
3.2 Data Collection and Analysis
3.2.1 Steps Involved in Interim and Final Research Priority List Generation
3.2.1.1 Indicative Question Generation for Master List
Submissions that were not research questions, were unrelated to prevention or were considered to
already be answered by high-quality evidence (such as a systematic review or randomized
controlled trial) were removed. The removal of questions meeting these criteria was decided
upon by consensus during an in-person meeting with three members of the research team, who
reviewed the list of submissions from the questionnaire. Agreement was not calculated because
the research team members participated in a collaborative process and did not have
29
disagreements of note with regards to the categorization of questions. The remaining questions
were grouped by similarity and were used to generate indicative questions that represented the
theme of the original submissions. For example, the question “How much screen time is
appropriate for children?” was generated by combining questions such as “How much time
should a child spend in front of the TV?”, “How much TV / screen time is considered too
much?” and “What is the maximum daily amount of time a 3 year old should be in front of a
computer/TV/iPad?” Parent and clinician questions were kept separate at this stage.
A review was performed of relevant guidelines and position statements, and studies in Medline,
EMBASE and CINAHL (in English and related to children aged 0 to 17 years of age, between
1946 and 2014), to assess the evidence related to each indicative question. Indicative questions
were considered unanswered if there was no systematic review on the topic, if there was a
systematic review which indicated insufficient evidence or if there was a grade I (Insufficient)
level of evidence in one or more preventive health guidelines (including Bright Futures (Bright
Futures, 3rd Edition, 2007), the USPSTF recommendations(U.S. Preventive Services Task
Force, 2014), the Rourke Baby Record (Rourke et al., 2014), the Greig Health Record (Greig et
al., 2010), and position statements from the Canadian Paediatric Society and American Academy
of Pediatrics). Questions were reviewed by 3 team members and unanswered indicative
questions were retained.
A Master List of unanswered indicative questions was generated by combining similar
unanswered questions submitted by parents and clinicians. Master List questions were ranked by
the number of respondents whose questionnaire submissions fell within the scope of each
indicative question (the “parent and clinician score”). For example, if a question about a certain
topic was submitted by 15 parents and 10 clinicians, the indicative question was given a parent
and clinician score of 25. The top ranked question was the one that had the highest number of
submissions from parents and clinicians (the maximum score).
3.2.1.2 Generation of Interim List
Steering Group members were given the Master List, containing 79 questions. They were asked
to individually rank the top 30 questions in the Master List from most to least important (with 1
being the most important). The questions were ranked based on the Steering Group members’
perception of their importance to pediatric preventive care research. A “Steering Group score”
30
was generated from the sum of these rankings and used to reorder the Master List questions by
importance. The Steering Group was only asked to rank the top 30 questions as it is difficult to
rank a list of 79 questions from 1 to 79 in order of importance. In addition, the goal was to obtain
a shorter list to be used as the Interim List that would be carried forward to the workshop.
3.2.2 Final Workshop: Nominal Group Technique
An in-person priority setting workshop was held in December 2014 based on the James Lind
Alliance process (Cowan & Oliver, 2013). Consensus was achieved between parents and
clinicians on the top 10 most important unanswered research questions using a modified
Nominal Group Technique (James Lind Alliance, 2016). Parents and clinicians were recruited
through the TARGet Kids! research network and the Ontario Medical Association. This process
used the same email address lists described in section 3.1.4. One week before the workshop, the
Interim List was distributed to attendees, who individually ranked the unanswered research
questions in order of importance prior to attending the workshop.
Parents and clinicians were evenly divided into four groups, each with a facilitator. During 2
morning sessions, the four groups ranked the 39 questions on the Interim List. Each group
discussed the Interim List and proposed changes to the ranking based on the perceived
importance of questions; through multiple rounds of discussion allowing time for each
participant to state their viewpoint, each group reached a consensus between members as to the
revised order of questions in order of importance. If a question was felt to be thematically
similar to another, the groups were allowed to “combine” the two questions, in which case the
new broader question was assigned a ranking, but the original question was not. The mean
ranking for each question was calculated across the four small groups and the top 20 questions
were retained for the afternoon sessions. During the first afternoon session, participants were
assigned to four different small groups, and each group discussed and ranked the top 20
questions. The mean ranking for each question was again calculated across the four small groups
and a final session which involved all participants was used to reach consensus on the top 10
most important unanswered research questions.
After the workshop, attendees were asked to rate on a scale of 1 (poor) to 5 (excellent) their
overall experience and how effectively the modified Nominal Group Technique allowed the
31
group to achieve consensus on the top 10 unanswered research questions. An overall average
rating was calculated for both questions.
3.2.3 Data Security and Protection of Personal Information
Data collected from parents and clinicians was stored electronically at the Hospital for Sick
Children Research Institute. No personally identifiable health information was collected.
3.2.4 Secondary Outcome: Statistical Analysis
The secondary analysis was to compare the questions submitted by parents and clinicians to
assess whether one group was more likely than the other to submit questions in certain
categories. Descriptive statistics were calculated for parents and clinicians who responded to the
online questionnaire. Questions in the Master List were organized into 11 thematic categories by
3 team members by consensus. To identify similarities and differences in response between
categories of unanswered research questions in the Master List, we used the Chi-Square test of
independence or Fisher’s exact test. Associations were evaluated between the type of respondent
(parent or clinician) and the submission of at least one unanswered research question for that
category (yes or no). The False Discovery Rate method was used to account for multiple testing
by adjusting the raw p values obtained from the Chi-Square tests (Benjamini & Hochberg, 1995).
In addition, the workshop evaluation data was used to assess participants’ satisfaction by
generating an average score for each question. Data analysis for this study was performed using
SAS™ software, version 9.3 of the SAS System for Windows.
3.3 Research Ethics Board Approval
The study protocol was approved by the Research Ethics Board of the Hospital for Sick Children
as an amendment to the TARGet Kids! study protocol. Administrative approval was obtained
from the University of Toronto. Ethics approval was not obtained at St. Michael’s Hospital as the
identification of important unanswered research questions to be used to inform research in
TARGet Kids! was felt to qualify as a quality improvement project and thus not be subject to
REB review.
32
Figure 2: Flow chart of research priority setting process, reproduced with permission from
Lavigne et al., 2017.
Questionnaire respondents
•800 parents were contacted via email; 115 parents responded
•80 clinicians were contacted via email, and the questionnaire link was distributed via the OMA Pediatrics Section mailing list; 42 clinicians responded
Submitted questions
•Parents submitted 791 questions; 397 were unanswered, 183 answered, 211 not research questions. Unanswered questions were combined into 57 indicative questions
•Clinicians submitted 255 questions; 151 were unanswered, 18 answered, 99 not research questions. Unanswered questions were combined into 45 indicative questions
Master List of Indicative Questions
•The indicative questions from parents and clinicians were combined into 79 indicative questions
•These were first ranked by the number of total submissions used to generate each indicative question
•The questions were then ranked in order of importance from 1 to 79 by Steering Group members
Interim Ranked List
•39 questions from the Master List of indicative questions were retained for the interim list; the top 30 questions as ranked by Steering Committee members, and 9 additional questions that were in the top 30 by total number of submissions
Final Ranked List of Unanswered Research
Questions
•During the final workshop, consensus was reached on the top 10 unanswered research questions in pediatric preventive care
33
Chapter 4 Results
Results
4.1 Demographic Characteristics
4.1.1 Questionnaire Respondents: Parents and Clinicians
Respondents to the online questionnaire included 115 parents and 42 clinicians (see Table 1).
The response rate was 20% for parents. The clinician response rate could not be calculated
because the total number of clinicians who received the invitation to the survey was not known.
Parents who responded to the questionnaire were overwhelmingly female (98%), mostly aged 30
to 39 years (68%) and most had one (42%) or two (51%) children. Most parents were university
educated (94%) and 76% reported a household income between above $100,000. Clinicians were
predominantly male (58.6%), older than 50 years of age (65.5%) and worked in primary care
settings (65.5%). In comparison, based on data from the 2011 Canadian census, 55% of Ontario
residents aged 30 to 39 years of age were female (Statistics Canada, 2016). Of the more than 2.2
million families in Ontario with children in the same year, approximately 44% had 1 child and
40% had 2 children (Government of Canada, 2017). Only 5.5% of Ontario residents had
household income between $100,000 and $199,999 in 2011 (Government of Canada, 2017).
N Percent
Parents Gender Female 82 97.6% Male 2 2.4% Age 30 to 39 years 57 67.9% 40 to 49 years 26 31.0% 50 years and older 1 1.2% Number of
children
One (1) 35 41.7% Two (2) 43 51.2% Three (3) 5 6.0% Four (4) 1 1.2% Level of
education
High school, college or
other non-university
training
5 6.0%
University certificate,
diploma or degree
79 94.0%
34
Household
income
Under $50,000 5 6.9% $50,000 to under
$100,000
12 16.7% $100,000 and above 55 76.4% Clinicians Gender Female 12 41.4% Male 17 58.6% Age 30 to 39 years 5 17.2% 40 to 49 years 5 17.2% 50 years and older 19 65.5% Role Pediatrician 29 100% Work in primary
care setting
Yes 19 65.5% No 9 32.1% 31 parents did not respond to demographic questions, except for household income, which had 43 non-responders 13 clinicians did not respond to demographic questions, except for work in primary care setting, which had 14 non-responders
Table 1: Demographic information for questionnaire respondents
4.1.2 Workshop Participants
10 parents (8 mothers and 2 fathers) and eighteen clinicians (12 pediatricians, 5 family
physicians and 1 nurse) participated in the final priority setting workshop. Two of the clinicians
were also members of the Steering Group.
4.2 Identification of Unanswered Research Questions
4.2.1 Questionnaire submissions
Parents and clinicians submitted a total of 791 and 255 questions about pediatric preventive care
research, respectively, for a total of 1046 questions. Most submissions pertained to nutrition and
obesity (20% of clinician questions, 18% of parent questions), illness prevention (19% and 17%)
and parenting and behaviour (16% and 15%).
4.2.2 Analysis of questionnaire submissions
Of the questions submitted, 397 parent questions (50%) and 151 clinician questions (59%) were
found to be unanswered. As some questions were similar in both groups, fifty-seven unique
questions from parents and 45 from clinicians were combined into the 79 indicative questions of
the Master List (see Appendix 3). The top ranked indicative question based on the total number
of submissions was “What are effective strategies for parents to discipline children?” which had
been submitted by 10 clinicians and 40 parents. Other commonly submitted unanswered research
35
questions included the impact of electronic devices on child development (32 parents and 5
clinicians), and effective interventions to promote healthy sleep (19 parents and 8 clinicians) (see
table 3).
4.2.3 Generation of Interim List
The Master List of 79 questions was sent to the project’s Steering Group and each member
assigned a rank of 1 to 79 to the questions, with 1 being most important and 79 being least
important. These rankings were compiled and used to generate the Interim Ranked List of
questions. Where questions were tied, they were ordered based on their ranking in the Master
List (i.e. if two questions were tied and one was originally ranked 12th in the Master List and the
other was ranked 29th, the question ranked 12th would be higher in the Interim List).
Thirty-nine questions of the original 79 questions on the Master list were retained as the Interim
List. These 39 questions were felt to be most important to the parent and clinician questionnaire
respondents and to the Steering Group. The first 30 questions were the top 30 ranked by the
Steering Group members. Nine other questions that had been ranked in the top 30 of the Master
List but were not ranked in the top 30 of the interim list by the Steering Group were retained to
preserve the most frequently submitted questions by parents and clinicians (see Appendix 4). As
well, nine questions that were originally not in the top 30 in the Master List moved into the top
30 of the Interim List (in bold on table 2): Questions 31, 32, 33, 37, 38, 41, 42, 50 and 58 from
the Master List were retained in the Interim List as questions 29, 30, 16, 14, 25, 26, 8, 6 and 17
respectively.
Interim List Rank
Master List Rank
Steering Group Score
Question
1 5 168 What are effective interventions for obesity prevention in young children?
2 2 140 How do electronic devices (such as computers, cell phones and tablets) impact development?
3 8 128 What are effective strategies for screening and prevention of mental health problems?
4 16 124 What strategies optimize immunization rates?
5 10 123 What are effective methods for screening for developmental delay in children?
6 50 116 Is there a relationship between nutrition and behaviour?
7 21 108 What interventions promote social skill development?
36
8 42 105 What is the impact of parental stress on children?
9 4 104 What interventions increase physical activity in children?
10 29 103 What are the predictors of obesity in children?
11 25 101 How can autism spectrum disorder be identified early?
12 20 86 What are the health and developmental effects of pesticides?
13* 23 86 What is the health and developmental impact of consumption of organic food vs. non-organic food?
14 37 82 What is the optimal timing of the introduction of solid food in babies, in order to prevent disease?
15 15 79 What strategies prevent allergies?
16 33 72 What is the impact of outdoor play on developmental outcomes?
17 58 69 What impact does the provision of primary care by pediatricians have on health outcomes?
18 9 65 What interventions prevent dental caries?
19 6 64 How much screen time is appropriate for children?
20 14 64 What strategies prevent asthma?
21 1 60 What are effective strategies for parents to discipline children?
22 3 59 What are effective interventions to promote healthy sleep habits?
23 26 58 What is the impact of daycare attendance on child health and development?
24 24 56 What are effective strategies for early identification of language difficulties?
25 38 55 What strategies can prevent ADHD?
26* 41 55 How does parental mental health affect the mental health of children?
27 11 51 What are the health benefits of vitamin supplementation?
28 18 47 How can viral respiratory illnesses be prevented?
29* 31 47 How can iron deficiency anemia be prevented in infants and toddlers?
30* 32 47 What is the impact of daily exercise on health outcomes in young children?
31 12 What are effective strategies for vision screening in children?
32 17 How much sleep optimizes development?
33 22 What are the health and developmental effects of electromagnetic waves from electronic devices?
34 28 What are predisposing factors and interventions to prevent picky eating?
35 13 Do children who are picky eaters require micronutrient screening?
36 30 Does screen time (exposure to TV, tablets, computers, etc.) cause vision problems?
37 19 What are effective strategies for screening for obstructive sleep apnea?
38 27 What is the optimal timing to introduce electronic devices to children?
39 7 What are effective interventions to prevent and manage temper tantrums?
Bold: Not in the original top 30 ranked questions from the Master List * Received the same Steering Group score
Table 2: Interim Ranked List of Unanswered Research Questions
37
4.2.4 Workshop
During the first two small group sessions of the workshop, 3 out of 4 groups felt that certain
thematically similar questions from the Interim List should be combined. One question was
suggested for combination with another question by all 3 of those groups: “How do electronic
devices (such as computers, cell phones and tablets) impact development?” The groups
suggested combining this question with the following questions: “How much screen time is
appropriate for children?” and “What is the optimal timing to introduce electronic devices to
children?” One group did not combine any questions, and did not assign numerical rankings
beyond number 30 as they did not want to use any of the time available to discuss questions that
they knew were going to be in their 10 least important.
In addition, the wording of question 21 in the Interim List was altered by full-group consensus
from “What are effective strategies for parents to discipline children?” to “What are effective
strategies for behaviour management in children?” This change was put in place because
workshop attendees felt that it should reflect the broader concept of children’s behaviour which
was thought to encompass the original question. As well, although consensus was the goal of the
workshop, there was some controversy around the question “What strategies optimize
immunization rates?”. Workshop participants agreed that this is an important question but felt
that mental health and development needed greater attention from the research community as
compared to immunization, a topic perceived by many participants to be relatively well studied.
One workshop attendee noted surprise that immunization was not higher on the final ranked list
given its importance to child health.
Participants reached consensus on the top ten unanswered questions (see table 3). The top
question was “What are effective strategies for screening and prevention of mental health
problems?” The other top five questions were about interventions to increase physical activity,
the impact of day care attendance on health and development, interventions for obesity
prevention in young children and methods to promote social skill development.
4.2.5 Primary Outcome: Final List of Unanswered Research Questions
Rank Question
38
1 What are effective strategies for screening and prevention of mental health problems? 2 What interventions increase physical activity in children? 3 What is the impact of daycare attendance on child health and development? 4 What are effective interventions for obesity prevention in young children? 5 What interventions promote social skill development? 6 What is the impact of parental stress on children? 7 What are effective methods for screening for developmental delay in children? 8 What are effective strategies for behaviour management in children? 9 What nutritional factors affect child behaviour? 10 How much screen time is appropriate for children? 11 What are effective interventions to promote healthy sleep habits? 12 What strategies optimize immunization rates? 13 What are predisposing factors and interventions to prevent picky eating? 14 How can iron deficiency anemia be prevented in infants and toddlers? 15 How can viral respiratory illnesses be prevented? 16 What interventions prevent dental caries? 17 How much sleep optimizes development? 18 What strategies prevent allergies? 19 What are the predictors of obesity in children? 20 What strategies can prevent ADHD?
Table 3: Final ranked list of the Top 20 unanswered research questions, reproduced with
permission from Lavigne et al., 2017.
4.3 Secondary Outcome: Comparison between Parents and Clinicians
Parents were more likely than clinicians to submit questions about screen time and media
exposure (49/115 parents vs 8/42 clinicians, p < 0.05) as well as environmental toxins (18/115
parents vs 0/42 clinicians, p < 0.05) (see table 4). Clinicians were more likely to submit
questions in the category of poverty and access to care (4/42 clinicians vs 2/115 parents) but the
differences were not statistically significant after adjusting for multiple testing.
Question Category Parents (n=115) Clinicians (n =42) p value Adj. p value1 Odds Ratio (95% CI) Growth, nutrition and physical activity
94 (81.7%) 32 (76.2%) 0.44 0.60 1.40 (0.60 – 3.28)
Parenting and behaviour 65 (56.5%) 17 (40.5%) 0.07 0.16 1.91 (0.93 – 3.92) Child development and learning
55 (47.8%) 20 (47.6%) 0.98 0.98 1.01 (0.50 – 2.05)
Preventing childhood infections and other diseases
70 (60.9%) 21 (50.0%) 0.22 0.35 1.56 (0.76 – 3.17)
Mental health 29 (25.2%) 11 (26.2%) 0.90 0.98 0.95 (0.42 – 2.13) Oral health, vision and 49 (42.6%) 11 (26.2%) 0.06 0.16 2.09 (0.96 – 4.57)
39
hearing Safety and injury prevention
12 (10.4%) 5 (11.9%) 0.74 0.90 0.86 (0.29 – 2.61)
Sleeping or night-time waking
44 (38.3%) 11 (26.2%) 0.16 0.29 1.75 (0.80 – 3.83)
Screen time and media exposure
49 (42.6%) 8 (19.0%) 0.01 0.04 3.16 (1.34 – 7.41)
Environmental toxins 18 (15.7%) 0 (0.00%) 0.004 0.04 N/A2 Poverty and access to care
2 (1.74%) 4 (9.5%) 0.04 0.16 0.17 (0.03 – 0.95)
1 Adjusted for multiple testing using the False Discovery Rate method (37)
2 Unable to calculate due to zero value for clinicians
Table 4: Questions submitted by parents and clinicians by category, reproduced with permission
from Lavigne et al., 2017.
Parental concerns about environmental toxins ranged from pesticides in food to the impact of
wireless communication devices. Environmental toxins are seldom mentioned in preventive care
guidelines, with the exception of Bright Futures recommendations about lead in paint (Bright
Futures, 3rd Edition, 2007).
4.4 Participant Experience during Workshop
In the post-workshop questionnaire (57% response frequency), respondents ranked their overall
experience 4.3 out of 5. When asked how well they felt the Nominal Group Technique allowed
the priority-setting goal to be reached, the average score was 3.7 out of 5. Six respondents
provided comments. One parent respondent noted that parents and clinicians may understand the
research questions differently and suggested providing a brief paragraph on what is known about
each question. Another respondent remarked that as a parent, they deferred to clinicians during
parts of the group session due to a perceived lack of knowledge about certain questions. Two
other respondents felt the process efficiently accomplished the project objectives, and that
parents were a crucial part of the workshop. Multiple participants commented that their
experience was a positive one and they felt engaged with and satisfied by the process.
40
Chapter 5 Discussion and Conclusion
Discussion
5.1 Top Research Priorities in Pediatric Preventive Care
This study identified the top 10 unanswered research questions in pediatric preventive care from
the perspective of parents and clinicians in Ontario, Canada using a process informed by the JLA
PSP (Cowan & Oliver, 2013). Parents were more likely to identify questions about screen time,
media exposure and environmental toxins than clinicians. Little information about environmental
toxins is contained in preventive care guidelines, and this topic did not make the top ten list at
the workshop.
The questions in the top 10 list included topics such as mental health, child development, obesity
prevention and physical activity. The USPSTF’s panel of experts identified these topics among
seven “high priority evidence gaps” related to children and adolescents in a report to the US
Congress in 2014 (U.S. Preventive Services Task Force, 2015). Two categories in the USPSTF
report were not identified by the respondents in our study, namely cancer prevention and child
maltreatment. This study identified topics that were not addressed by the USPSTF report such as
environmental toxins and poverty. Recommendations for priority evidence gaps by organizations
such as the USPSTF may benefit from the input of parents and clinicians to identify such
important topics that might otherwise be missed. The absence of cancer prevention and child
maltreatment in our list of research priorities may be due to the nature of these topics. Cancer
prevention may not be a major concern for parents of children under the age of 5, at least in the
short term. It is unclear whether the absence of child maltreatment as a priority topic resulted
from the topic being perceived as less related to child health, at least in the sense of “keeping
children healthy as they grow up” as phrased in the study questionnaire.
The C. S. Mott Children’s Hospital National Poll on Children Health identified the 10 most
important health concerns from the perspective of parents in 2016 (C.S. Mott Children’s
Hospital, 2016). The top 10 health concerns in 2016 were:
“1. Bullying/cyberbullying (61%)
2. Not enough exercise (60%)
41
3. Unhealthy eating (57%)
4. Drug abuse (56%)
5. Internet safety (55%)
6. Child abuse and neglect (53%)
7. Suicide (45%)
8. Depression (44%)
9. Teen pregnancy (43%)
10. Stress (43%)” (C.S. Mott Children’s Hospital, 2016)
There are many common elements between the National Poll on Children’s Health’s results and
the top 10 list of unanswered research questions that we identified. Mental health, exercise and
nutrition were the most common topics in both lists. The National Poll had items such as internet
safety and teen pregnancy that were not addressed in our top 10 list, likely because parents
enrolled in this study have children aged 0 to 5 years and may not have been exposed to these
concerns at this stage. Although parents who reported health concerns were likely not
considering them in the context of possible research topics, and were not specifically asked about
prevention, these similarities point to the importance of mental health support in children, and a
need for further knowledge translation and research on related patient-centered outcomes.
Physical activity and nutrition is also an important concern for parents and was identified as an
important research priority by both groups. Strikingly, parents did identify that child abuse and
neglect is an important health concern; however, to obtain these results, the Poll asked parents to
rank 24 health-related problems affecting children and teenagers, whereas in our study, topic lists
were not provided ahead of time (with the exception of the broader categories described in the
questionnaire) (C.S. Mott Children’s Hospital, 2016).
5.2 Priority Setting Process: Lessons Learned
5.2.1 Steering Group
The Steering Group in this study played an important albeit small role. However, it was
challenging to keep individuals connected during the priority setting process to the iterative
nature of this research. In this study, the Steering Group was primarily focused on assessing the
validity of the questionnaire and generating the Interim List. There appears to be a range of
involvement in other PSPs as well, as well as variation in the size and composition of the
Steering Group, and this needs to be customized to the specific project at hand. Given the time
42
commitment and workload involved with this type of study, incentives such as financial support
could promote continued engagement with the project over the longer term.
5.2.2 Generation of the Top 10 Priorities
The iterative process as described in Chapter 3 provided rich data but was also time-consuming
and labour intensive. Given that this PSP was performed within the context of a Master’s Degree
project, there was no administrative support or research assistant attached to the project, which is
something that many PSPs coordinated by charities or other stakeholder organizations have the
benefit of obtaining. As a consequence, it was challenging to maintain the tight timelines
required to maintain momentum in a project of this scope, especially considering the many
individuals who donated their time not just to filling out the questionnaire but also to subsequent
steps and to the in-person final workshop. Despite this, the project was successful in large part
due to the dedication of the research team to completing the objectives and ensuring that parents
and clinicians were given a voice when choosing health research priorities.
5.2.3 Participant Perceptions
In general, feedback during the multiple steps of the study and culminating with the final
workshop was very positive. Parents were pleased to have the opportunity to have their voices
heard and to promote research related to the health of children as they grow up. There was a
perception as described in certain participants that parents might not have been as prepared as
clinicians to rank the questions in the Interim List. This is not due to a lack of motivation or
opinion on these topics, but rather there was concern that parents and caregivers may not have a
sufficient knowledge base to truly assess whether one topic is more important than another.
Although clinicians likely have the medical expertise to understand whether a problem can
seriously affect a child or not, this is not the only way that patients and families approach
research priority setting, as has been previously discussed. Indeed, it is the very fact that patients
and families may have certain priorities that clinicians or researchers would not even think of
that makes this process so important. Even if parents were to have the same knowledge base as
clinicians, it does not imply that their research priorities would be the same. Future research
could examine the difference in research priorities between clinicians with children and those
without children to study the extent to which child rearing influences their prioritization around
important health research topics.
43
5.3 Strengths and Limitations
Strengths of our study include the number of submissions as well as the methodology used to
identify unanswered research questions. While many PSPs have involved stakeholder groups
related to a specific condition such as asthma, this study involved the parents of healthy young
children being cared for in primary care settings, and clinicians who provide this care. The focus
on preventive care in general as opposed to focusing on a particular disease was a strength in the
sense that it allowed study participants to think more broadly about keeping children healthy as
they grow up, which may have elicited priorities that would not appear elsewhere especially
when discussing a specific disease in any given study.
Participants submitted 1046 potentially unanswered research questions, reflecting the diversity of
topics in pediatric preventive care. During the workshop, the priority setting methodology
attempted to balance viewpoints from stakeholders and to prevent the strongest voices from
carrying the most weight. This was accomplished through multiple sessions with different
groups, and by having moderators with considerable focus group and priority setting experience
to ensure that all participants’ viewpoints were heard. There was also a good questionnaire
response rate for parents; other PSPs have had similar response frequencies such as PSPs on
vitiligo and tinnitus (18 and 19% respectively) (Eleftheriadou et al., 2011; Hall, Mohamad,
Firkins, Fenton, & Group), 2013).
A limitation of this study was that parent respondents were primarily well-educated mothers
living in high-income households, and it is possible that their priorities may be different than
fathers or parents of lower socioeconomic status. A similar study should be performed with more
varied sample populations to assess generalizability. While the JLA methodology provided
representation from parents and clinicians, recruitment was challenging, possibly because we
studied preventive care in general. In PSPs which were focused on one medical condition or
disease, the motivation to participate may have been higher because patients or caregivers were
directly affected by the disease or because many participants were members of organizations and
charities focused on that issue. In addition, parents of a child with a chronic disease may have a
stronger interest in participating in this type of study. It may have been difficult for participants
to devise questions about keeping children healthy as they grow up given that there are so many
possible answers, but we had more than 1000 questions submitted and we tried to provide
44
guidelines by generating the list of specific questions in the questionnaire based on the literature
review discussed in Chapter 2.
PSPs focused on a specific disease benefit from the knowledge and experiences of parents of
children with that disease. However, there are many ways that parents obtain and synthesize
information about preventive care – not only from their child’s healthcare providers, but also
from media sources including television and online sources ranging from Facebook groups to
message boards. In addition, they may draw on their own experiences and knowledge of the
experiences of family members when considering what keeping a child healthy entails. As such,
parents are still prepared to think about keeping their children healthy and indeed, this is a major
preoccupation of parents as evidenced by the data from the National Children’s Poll mentioned
above, among others (C.S. Mott Children’s Hospital, 2016).
Pediatricians contacted through the Ontario Medical Association practice in many different
subspecialties which may have adversely affected the response rate to a questionnaire about
topics more commonly addressed in primary care settings. In addition, many of the unanswered
research questions that we identified are broadly worded. An important next step may be to
identify topics within these questions that may be the subject of more specific health research
questions.
In addition, given that this study was focused on children aged 0 to 5 years, we did not ask
children and teenagers to participate in the priority setting process. While children under the age
of 5 would likely not be able to contribute, older children and teenagers may have different
perspectives and could be assessed in future research. Some parents felt that the extent of their
knowledge may not have allowed them to make fully informed decisions on the relative
importance of unanswered questions during the workshop. PSPs in the future might consider
providing information on the existing evidence base for each question to facilitate decision
making. For example, a brief summary of existing knowledge could be provided during the
workshop on the cards on which each unanswered research question is printed.
5.4 Impact of the Top 10 List
The top 10 list of questions have already been used to inform research within the TARGet Kids!
Research Network. The “Preventing and Addressing At-obesity-Risk Early Years iNtervention
45
With Home Visits in Toronto” (PARENT) trial seeks to assess the impact of a “primary-care
based intervention that combines group based parenting skills training and public health nurse
home visits” on the weight of children aged 18 months to 3 years of age, compared to children
receiving regular preventive care (Birken, 2017). As the most important unanswered research
question in the top 10 list pertains to the screening and prevention of mental health problems, the
PARENT trial has been designed to incorporate questionnaires to assess the impact of the
intervention on the mental health of the parent, which is known to affect the health and
development of the child (Manning & Gregoire, 2006).
5.5 Future Research
Future research may be focused on several facets of this work, including assessing the
generalizability of these results across other study populations and providing a platform for older
children and teenagers to inform preventive care priorities. Now that the top 10 list has been
created, specific questions within these general research priorities may be examined in future
research with a view to producing a more comprehensive list to be considered by funding
agencies and preventive care organizations.
5.6 Conclusion
To our knowledge, no other study has brought parents and clinicians together to identify the most
important priorities for preventive care research for children. Engaging patients, families and
caregivers in research prioritization may focus research on outcomes of importance to knowledge
users and promote fair use of limited research funding. This process is particularly important in
pediatric research, as children are an understudied and vulnerable population (Diekema, 2006;
Klassen, Hartling, Craig, & Offringa, 2008; Martinez-Castaldi, Silverstein, & Bauchner, 2008).
We hope that the research priorities identified through this study will help researchers, funders
and professional organizations advance preventive healthcare for children.
46
References
Abma, T. A., & Broerse, J. E. W. (2010). Patient participation as dialogue: setting research
agendas. Health Expectations: An International Journal of Public Participation in Health
Care and Health Policy, 13(2), 160–173. https://doi.org/10.1111/j.1369-
7625.2009.00549.x
Abma, T. A., Pittens, C. A. C. M., Visse, M., Elberse, J. E., & Broerse, J. E. W. (2015). Patient
involvement in research programming and implementation. Health Expectations, 18(6),
2449–2464. https://doi.org/10.1111/hex.12213
Allard, J., Durand, C., Anthony, S. J., Dumez, V., Hartell, D., Hébert, M.-J., … Fortin, M.-C.
(2017). Perspectives of Patients, Caregivers and Researchers on Research Priorities in
Donation and Transplantation in Canada: A Pilot Workshop. Transplantation Direct,
3(2), e127. https://doi.org/10.1097/TXD.0000000000000639
Batchelor, J. M., Ridd, M. J., Clarke, T., Ahmed, A., Cox, M., Crowe, S., … Thomas, K. S.
(2013). The Eczema Priority Setting Partnership: a collaboration between patients, carers,
clinicians and researchers to identify and prioritize important research questions for the
treatment of eczema. The British Journal of Dermatology, 168(3), 577–582.
https://doi.org/10.1111/bjd.12040
Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and
Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B
(Methodological), 57(1), 289–300.
47
Birken, C. (2017). Preventing and Addressing At-obesity-Risk Early Years iNtervention With
Home Visits in Toronto (PARENT): A Pragmatic RCT. Retrieved March 5, 2018, from
https://clinicaltrials.gov/ct2/show/NCT03219697
Boddy, K., Cowan, K., Gibson, A., & Britten, N. (2017). Does funded research reflect the
priorities of people living with type 1 diabetes? A secondary analysis of research
questions. BMJ Open, 7(9), e016540. https://doi.org/10.1136/bmjopen-2017-016540
Boney, O., Nathanson, M. H., Grocott, M. P. W., Metcalf, L., & Steering Group for the National
Institute of Academic Anaesthesia/James Lind Alliance Anaesthesia and Peri-operative
Care Priority Setting Partnership. (2017). Differences between patients’ and clinicians’
research priorities from the Anaesthesia and Peri-operative Care Priority Setting
Partnership. Anaesthesia, 72(9), 1134–1138. https://doi.org/10.1111/anae.13936
Boote, J., Wong, R., & Booth, A. (2015). “Talking the talk or walking the walk?” A bibliometric
review of the literature on public involvement in health research published between 1995
and 2009. Health Expectations: An International Journal of Public Participation in
Health Care and Health Policy, 18(1), 44–57. https://doi.org/10.1111/hex.12007
Boyle, C. A., Perrin, J. M., & Moyer, V. A. (2014). Use of Clinical Preventive Services in
Infants, Children, and Adolescents. JAMA, 312(15), 1509–1510.
https://doi.org/10.1001/jama.2014.12890
Bragge, P., Clavisi, O., Turner, T., Tavender, E., Collie, A., & Gruen, R. L. (2011). The Global
Evidence Mapping Initiative: Scoping research in broad topic areas. BMC Medical
Research Methodology, 11(1), 92. https://doi.org/10.1186/1471-2288-11-92
48
Brett, J., Staniszewska, S., Mockford, C., Herron-Marx, S., Hughes, J., Tysall, C., & Suleman, R.
(2014). Mapping the impact of patient and public involvement on health and social care
research: a systematic review. Health Expectations: An International Journal of Public
Participation in Health Care and Health Policy, 17(5), 637–650.
https://doi.org/10.1111/j.1369-7625.2012.00795.x
Bright Futures, 3rd Edition. (2007). American Academy of Pediatrics. Retrieved from
http://ebooks.aappublications.org/content/9781581102239/9781581102239
Bruni, R. A., Laupacis, A., Martin, D. K., & University of Toronto Priority Setting in Health
Care Research Group. (2008). Public engagement in setting priorities in health care.
CMAJ: Canadian Medical Association Journal = Journal de l’Association Medicale
Canadienne, 179(1), 15–18. https://doi.org/10.1503/cmaj.071656
Bruton, A., Lee, A., Yardley, L., Raftery, J., Arden-Close, E., Kirby, S., … Thomas, M. (2017).
Physiotherapy breathing retraining for asthma: a randomised controlled trial. The Lancet.
Respiratory Medicine. https://doi.org/10.1016/S2213-2600(17)30474-5
Burchell, K. (2015). Factors affecting public engagement by researchers: literature review.
Burklow, K. A., Vaughn, L. M., Valerius, K. S., & Schultz, J. R. (2001). Parental expectations
regarding discussions on psychosocial topics during pediatric office visits. Clinical
Pediatrics, 40(10), 555–562. https://doi.org/10.1177/000992280104001004
Campbell, S. (2010, June). Deliberative Priority Setting – a CIHR KT module. Retrieved March
5, 2018, from http://www.cihr-irsc.gc.ca/e/43533.html#s33
49
Canadian Pediatric Society. (2017). Position statements and practice points. Retrieved November
13, 2017, from https://www.cps.ca/en/documents
Canadian Task Force on Preventive Health Care. (2015). Obesity in Children. Retrieved
November 19, 2017, from https://canadiantaskforce.ca/guidelines/published-
guidelines/obesity-in-children/
Canadian Task Force on Preventive Health Care. (2016). Developmental Delay. Retrieved
November 19, 2017, from https://canadiantaskforce.ca/guidelines/published-
guidelines/developmental-delay/
Canadian Task Force on Preventive Health Care. (2017a). About Us. Retrieved November 19,
2017, from https://canadiantaskforce.ca/about/
Canadian Task Force on Preventive Health Care. (2017b). Tobacco Smoking in Children and
Adolescents. Retrieved November 19, 2017, from
https://canadiantaskforce.ca/guidelines/published-guidelines/tobacco-smoking-in-
children-and-adolescents/
Caron-Flinterman, F. J., Broerse, J. E. W., Teerling, J., & Bunders, J. F. G. (2005). Patients’
priorities concerning health research: the case of asthma and COPD research in the
Netherlands. Health Expectations, 8(3), 253–263. https://doi.org/10.1111/j.1369-
7625.2005.00337.x
Carsley, S., Borkhoff, C. M., Maguire, J. L., Birken, C. S., Khovratovich, M., McCrindle, B., …
TARGet Kids! Collaboration. (2015). Cohort Profile: The Applied Research Group for
50
Kids (TARGet Kids!). International Journal of Epidemiology, 44(3), 776–788.
https://doi.org/10.1093/ije/dyu123
Center on the Developing Child at Harvard University. (2010). The Foundations of Lifelong
Health Are Built in Early Childhood. Retrieved from
http://developingchild.harvard.edu/wp-content/uploads/2010/05/Foundations-of-
Lifelong-Health.pdf
Centers for Disease Control. (2017). Preventive Health Care, Gateway to Health
Communication. Retrieved November 13, 2017, from
https://www.cdc.gov/healthcommunication/toolstemplates/entertainmented/tips/Preventiv
eHealth.html
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., …
Oliver, S. (2014). How to increase value and reduce waste when research priorities are
set. The Lancet, 383(9912), 156–165. https://doi.org/10.1016/S0140-6736(13)62229-1
Chalmers, I., & Glasziou, P. (2009). Avoidable waste in the production and reporting of research
evidence. The Lancet, 374(9683), 86–89. https://doi.org/10.1016/S0140-6736(09)60329-
9
Clavisi, O., Bragge, P., Tavender, E., Turner, T., & Gruen, R. L. (2013). Effective stakeholder
participation in setting research priorities using a Global Evidence Mapping approach.
Journal of Clinical Epidemiology, 66(5), 496-502.e2.
https://doi.org/10.1016/j.jclinepi.2012.04.002
51
Corner, J., Wright, D., Hopkinson, J., Gunaratnam, Y., McDonald, J. W., & Foster, C. (2007).
The research priorities of patients attending UK cancer treatment centres: findings from a
modified nominal group study. British Journal of Cancer, 96(6), 875–881.
https://doi.org/10.1038/sj.bjc.6603662
Cowan, K., & Oliver, S. (2013). The James Lind Alliance Guidebook, Version 5. Retrieved from
http://www.jlaguidebook.org/pdfguidebook/guidebook.pdf
Crowe, S., Fenton, M., Hall, M., Cowan, K., & Chalmers, I. (2015). Patients’, clinicians’ and the
research communities’ priorities for treatment research: there is an important mismatch.
Research Involvement and Engagement, 1, 2. https://doi.org/10.1186/s40900-015-0003-x
C.S. Mott Children’s Hospital. (2016). National Poll on Children’s Health. Retrieved December
20, 2017, from https://mottpoll.org/sites/default/files/documents/081516_top10.pdf
C.S. Mott Children’s Hospital. (2017). About the Mott Poll. Retrieved December 20, 2017, from
https://mottpoll.org/about
Diekema, D. S. (2006). Conducting ethical research in pediatrics: a brief historical overview and
review of pediatric regulations. The Journal of Pediatrics, 149(1 Suppl), S3-11.
https://doi.org/10.1016/j.jpeds.2006.04.043
Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., … Murad, M. H.
(2014). Patient engagement in research: a systematic review. BMC Health Services
Research, 14, 89. https://doi.org/10.1186/1472-6963-14-89
52
Dubois, R. W., & Graff, J. S. (2011). Setting priorities for comparative effectiveness research:
from assessing public health benefits to being open with the public. Health Affairs
(Project Hope), 30(12), 2235–2242. https://doi.org/10.1377/hlthaff.2011.0136
Duley, L., Uhm, S., Oliver, S., & Preterm Birth Priority Setting Partnership Steering Group.
(2014). Top 15 UK research priorities for preterm birth. Lancet (London, England),
383(9934), 2041–2042. https://doi.org/10.1016/S0140-6736(14)60989-2
Elberse, J. E., Caron‐Flinterman, J. F., & Broerse, J. E. W. (2011). Patient–expert partnerships in
research: how to stimulate inclusion of patient perspectives. Health Expectations : An
International Journal of Public Participation in Health Care and Health Policy, 14(3),
225–239. https://doi.org/10.1111/j.1369-7625.2010.00647.x
Eleftheriadou, V., Whitton, M. E., Gawkrodger, D. J., Batchelor, J., Corne, J., Lamb, B., …
vitiligo priority setting partnership. (2011). Future research into the treatment of vitiligo:
where should our priorities lie? Results of the vitiligo priority setting partnership. The
British Journal of Dermatology, 164(3), 530–536. https://doi.org/10.1111/j.1365-
2133.2010.10160.x
Elwyn, G., Crowe, S., Fenton, M., Firkins, L., Versnel, J., Walker, S., … Gelder, C. (2010).
Identifying and prioritizing uncertainties: patient and clinician engagement in the
identification of research questions. Journal of Evaluation in Clinical Practice, 16(3),
627–631. https://doi.org/10.1111/j.1365-2753.2009.01262.x
Gadsby, R., Snow, R., Daly, A. C., Crowe, S., Matyka, K., Hall, B., & Petrie, J. (2012). Setting
research priorities for Type 1 diabetes. Diabetic Medicine, 29(10), 1321–1326.
https://doi.org/10.1111/j.1464-5491.2012.03755.x
53
Garbutt, J. M., Leege, E., Sterkel, R., Gentry, S., Wallendorf, M., & Strunk, R. C. (2012). What
are parents worried about? Health problems and health concerns for children. Clinical
Pediatrics, 51(9), 840–847. https://doi.org/10.1177/0009922812455093
Government of Canada, S. C. (2017, October 31). CANSIM - Canadian socioeconomic database
from Statistics Canada. Retrieved December 20, 2017, from
http://www5.statcan.gc.ca/cansim/home-accueil?lang=eng
Greig, A., Constantin, E., Carsley, S., & Cummings, C. (2010). Preventive health care visits for
children and adolescents aged six to 17 years: The Greig Health Record – Executive
Summary. Paediatrics & Child Health, 15(3), 157–159.
Hall, D. A., Mohamad, N., Firkins, L., Fenton, M., & Group), D. S. behalf of the S. (2013).
Identifying and prioritizing unmet research questions for people with tinnitus: the James
Lind Alliance Tinnitus Priority Setting Partnership. Clinical Investigation, 3(1).
Retrieved from http://www.openaccessjournals.com/abstract/identifying-and-prioritizing-
unmet-research-questions-for-people-with-tinnitus-the-james-lind-alliance-tinnitus-
priority-7883.html
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research
Electronic Data Capture (REDCap) - A metadata-driven methodology and workflow
process for providing translational research informatics support. Journal of Biomedical
Informatics, 42(2), 377–381. https://doi.org/10.1016/j.jbi.2008.08.010
Heazell, A. E. P., Whitworth, M. K., Whitcombe, J., Glover, S. W., Bevan, C., Brewin, J., …
Metcalf, L. (2015). Research priorities for stillbirth: process overview and results from
UK Stillbirth Priority Setting Partnership. Ultrasound in Obstetrics & Gynecology: The
54
Official Journal of the International Society of Ultrasound in Obstetrics and Gynecology,
46(6), 641–647. https://doi.org/10.1002/uog.15738
Hickson, G. B., Altemeier, W. A., & O’Connor, S. (1983). Concerns of mothers seeking care in
private pediatric offices: opportunities for expanding services. Pediatrics, 72(5), 619–
624.
James Lind Alliance. (2012). Cleft Lip and Palate. Retrieved December 20, 2017, from
http://www.jla.nihr.ac.uk/priority-setting-partnerships/cleft-lip-and-palate/
James Lind Alliance. (2016). The James Lind Alliance Guidebook, Version 6. Retrieved
December 19, 2017, from http://www.jla.nihr.ac.uk/jla-guidebook/downloads/JLA-
Guidebook-Version-6-February-2016.pdf
James Lind Alliance. (2018). The PSPs. Retrieved February 16, 2018, from
http://www.jla.nihr.ac.uk/priority-setting-partnerships/
Jones, J., & Hunter, D. (1995). Consensus methods for medical and health services research.
BMJ (Clinical Research Ed.), 311(7001), 376–380.
Jun, M., Manns, B., Laupacis, A., Manns, L., Rehal, B., Crowe, S., & Hemmelgarn, B. R.
(2015). Assessing the extent to which current clinical research is consistent with patient
priorities: a scoping review using a case study in patients on or nearing dialysis.
Canadian Journal of Kidney Health and Disease, 2, 35. https://doi.org/10.1186/s40697-
015-0070-9
Keren, R., Luan, X., Localio, R., Hall, M., McLeod, L., Dai, D., … Pediatric Research in
Inpatient Settings (PRIS) Network. (2012). Prioritization of comparative effectiveness
55
research topics in hospital pediatrics. Archives of Pediatrics & Adolescent Medicine,
166(12), 1155–1164. https://doi.org/10.1001/archpediatrics.2012.1266
Kindblom, J. M., Lorentzon, M., Hellqvist, A., Lönn, L., Brandberg, J., Nilsson, S., … Ohlsson,
C. (2009). BMI changes during childhood and adolescence as predictors of amount of
adult subcutaneous and visceral adipose tissue in men: the GOOD Study. Diabetes, 58(4),
867–874. https://doi.org/10.2337/db08-0606
Kistin, C. J. (2011). US healthcare reform and the evidence gap: the need for high-quality
research in paediatric preventive care. Evidence-Based Medicine, 16(2), 34–35.
https://doi.org/10.1136/ebm20006
Klassen, T. P., Hartling, L., Craig, J. C., & Offringa, M. (2008). Children are not just small
adults: the urgent need for high-quality trial evidence in children. PLoS Medicine, 5(8),
e172. https://doi.org/10.1371/journal.pmed.0050172
Lavigne, M., Birken, C. S., Maguire, J. L., Straus, S., & Laupacis, A. (2017). Priority setting in
paediatric preventive care research. Archives of Disease in Childhood, 102(8), 748–753.
https://doi.org/10.1136/archdischild-2016-312284
Layton, A., Eady, E. A., Peat, M., Whitehouse, H., Levell, N., Ridd, M., … Firkins, L. (2015).
Identifying acne treatment uncertainties via a James Lind Alliance Priority Setting
Partnership. BMJ Open, 5(7), e008085. https://doi.org/10.1136/bmjopen-2015-008085
Lloyd, K., White, J., & Chalmers, I. (2012). Schizophrenia: Patients’ research priorities get
funded. Nature, 487(7408), 432. https://doi.org/10.1038/487432b
56
Magiorkinis, E., Beloukas, A., & Diamantis, A. (2011). Scurvy: past, present and future.
European Journal of Internal Medicine, 22(2), 147–152.
https://doi.org/10.1016/j.ejim.2010.10.006
Manning, C., & Gregoire, A. (2006). Effects of parental mental illness on children. Psychiatry,
5(1), 10–12. https://doi.org/10.1383/psyt.2006.5.1.10
Martinez-Castaldi, C., Silverstein, M., & Bauchner, H. (2008). Child versus adult research: the
gap in high-quality study design. Pediatrics, 122(1), 52–57.
https://doi.org/10.1542/peds.2007-2849
McMillan, S. S., King, M., & Tully, M. P. (2016). How to use the nominal group and Delphi
techniques. International Journal of Clinical Pharmacy, 38, 655–662.
https://doi.org/10.1007/s11096-016-0257-x
Melnyk, B. M., Grossman, D. C., Chou, R., Mabry-Hernandez, I., Nicholson, W., DeWitt, T. G.,
… US Preventive Services Task Force. (2012). USPSTF perspective on evidence-based
preventive recommendations for children. Pediatrics, 130(2), e399-407.
https://doi.org/10.1542/peds.2011-2087
Mitton, C., Smith, N., Peacock, S., Evoy, B., & Abelson, J. (2009). Public participation in health
care priority setting: A scoping review. Health Policy, 91(3), 219–228.
https://doi.org/10.1016/j.healthpol.2009.01.005
Moos, M.-K., Dunlop, A. L., Jack, B. W., Nelson, L., Coonrod, D. V., Long, R., … Gardiner, P.
M. (2008). Healthier women, healthier reproductive outcomes: recommendations for the
57
routine care of all women of reproductive age. American Journal of Obstetrics and
Gynecology, 199(6 Suppl 2), S280-289. https://doi.org/10.1016/j.ajog.2008.08.060
Morris, C., Simkiss, D., Busk, M., Morris, M., Allard, A., Denness, J., … Cowan, K. (2015).
Setting research priorities to improve the health of children and young people with
neurodisability: a British Academy of Childhood Disability-James Lind Alliance
Research Priority Setting Partnership. BMJ Open, 5(1), e006233.
https://doi.org/10.1136/bmjopen-2014-006233
Moyer, V. A., & Butler, M. (2004). Gaps in the Evidence for Well-Child Care: A Challenge to
Our Profession. Pediatrics, 114(6), 1511–1521. https://doi.org/10.1542/peds.2004-1076
Murad, M. H., Shah, N. D., Van Houten, H. K., Ziegenfuss, J. Y., Deming, J. R., Beebe, T. J., …
Montori, V. M. (2011). Individuals with diabetes preferred that future trials use patient-
important outcomes and provide pragmatic inferences. Journal of Clinical Epidemiology,
64(7), 743–748. https://doi.org/10.1016/j.jclinepi.2010.08.005
Murphy, M. K., Black, N. A., Lamping, D. L., McKee, C. M., Sanderson, C. F., Askham, J., &
Marteau, T. (1998). Consensus development methods, and their use in clinical guideline
development. Health Technology Assessment (Winchester, England), 2(3), i–iv, 1–88.
Nass, P., Levine, S., & Yancy, C. (2012). Methods for involving patients in topic generation for
patient-centered comparative effectiveness research - an international perspective.
National Institute for Health and Care Excellence. (2007). Sexually transmitted infections and
under-18 conceptions: prevention. Retrieved April 17, 2017, from
https://www.nice.org.uk/guidance/ph3
58
National Institute for Health and Care Excellence. (2016). Early years: promoting health and
wellbeing in under 5s. Retrieved April 17, 2017, from
https://www.nice.org.uk/guidance/qs128
Norlin, C., Crawford, M. A., Bell, C. T., Sheng, X., & Stein, M. T. (2011). Delivery of well-
child care: a look inside the door. Academic Pediatrics, 11(1), 18–26.
https://doi.org/10.1016/j.acap.2010.12.008
Norlin, C., Sharp, A. L., & Firth, S. D. (2007). Unanswered questions prompted during pediatric
primary care visits. Ambulatory Pediatrics: The Official Journal of the Ambulatory
Pediatric Association, 7(5), 396–400. https://doi.org/10.1016/j.ambp.2007.05.008
Nunes, C., & Ayala, M. (2011). What do paediatricians and mothers talk about in the well-child
program visits? Anales de Pediatria, 75(4), 239–246.
Oliver, S., Clarke-Jones, L., Rees, R., Milne, R., Buchanan, P., Gabbay, J., … Stein, K. (2004).
Involving consumers in research and development agenda setting for the NHS:
developing an evidence-based approach. Health Technology Assessment (Winchester,
England), 8(15), 1–148, III–IV.
Olson, L. M., Inkelas, M., Halfon, N., Schuster, M. A., O’Connor, K. G., & Mistry, R. (2004).
Overview of the Content of Health Supervision for Young Children: Reports From
Parents and Pediatricians. Pediatrics, 113(Supplement 5), 1907–1916.
Pagel, C., Brown, K. L., McLeod, I., Jepps, H., Wray, J., Chigaru, L., … Utley, M. (2017).
Selection by a panel of clinicians and family representatives of important early
morbidities associated with paediatric cardiac surgery suitable for routine monitoring
59
using the nominal group technique and a robust voting process. BMJ Open, 7(5),
e014743. https://doi.org/10.1136/bmjopen-2016-014743
Partridge, N., & Scadding, J. (2004). The James Lind Alliance: patients and clinicians should
jointly identify their priorities for clinical trials. The Lancet, 364(9449), 1923–1924.
https://doi.org/10.1016/S0140-6736(04)17494-1
Petit-Zeman, S. (2007, May 21). Turning the Tables. Retrieved February 16, 2018, from
http://www.theguardian.com/society/2007/may/21/health
Radecki, L., Olson, L. M., Frintner, M. P., Tanner, J. L., & Stein, M. T. (2009). What Do
Families Want From Well-Child Care? Including Parents in the Rethinking Discussion.
Pediatrics, 124(3), 858–865. https://doi.org/10.1542/peds.2008-2352
Rourke, L., Leduc, D., & Rourke, J. (2014). The Rourke Baby Record. Retrieved April 1, 2014,
from http://www.rourkebabyrecord.ca/pdf/RBR2014Nat_Eng.pdf
Rowbotham, N. J., Smith, S., Leighton, P. A., Rayner, O. C., Gathercole, K., Elliott, Z. C., …
Smyth, A. R. (2017). The top 10 research priorities in cystic fibrosis developed by a
partnership between people with CF and healthcare providers. Thorax.
https://doi.org/10.1136/thoraxjnl-2017-210473
Ryan, M., Scott, D. A., Reeves, C., Bate, A., van Teijlingen, E. R., Russell, E. M., … Robb, C.
M. (2001). Eliciting public preferences for healthcare: a systematic review of techniques.
Health Technology Assessment (Winchester, England), 5(5), 1–186.
Shen, S., Doyle-Thomas, K. A. R., Beesley, L., Karmali, A., Williams, L., Tanel, N., &
McPherson, A. C. (2017). How and why should we engage parents as co-researchers in
60
health research? A scoping review of current practices. Health Expectations: An
International Journal of Public Participation in Health Care and Health Policy, 20(4),
543–554. https://doi.org/10.1111/hex.12490
Statistics Canada. (2016). Population by broad age groups and sex, 2011 counts for females, for
Canada, provinces and territories. Retrieved December 20, 2017, from
http://www12.statcan.gc.ca/census-recensement/2011/dp-pd/hlt-fst/as-
sa/Pages/highlight.cfm?TabID=1&Lang=E&Asc=1&PRCode=01&OrderBy=999&Sex=
3&View=1&tableID=21&queryID=1
Steele, R., Bosma, H., Johnston, M. F., Cadell, S., Davies, B., Siden, H., & Straatman, L. (2008).
Research priorities in pediatric palliative care: a Delphi study. Journal of Palliative Care,
24(4), 229–239.
Stewart, R., & Oliver, S. (2008). A systematic map of studies of patients’ and clinicians’
research priorities (Report). London: The James Lind Alliance. Retrieved from
http://eprints.ioe.ac.uk/2549/
Stickler, G. B. (1996). Worries of Parents and Their Children. Clinical Pediatrics, 35(2), 84–90.
https://doi.org/10.1177/000992289603500206
Tallon, D., Chard, J., & Dieppe, P. (2000). Relation between agendas of the research community
and the research consumer. Lancet (London, England), 355(9220), 2037–2040.
https://doi.org/10.1016/S0140-6736(00)02351-5
The Joanna Briggs Institute. (2015). Joanna Briggs Institute Reviewers’ Manual: 2015 edition /
Supplement. Retrieved February 28, 2018, from
61
https://joannabriggs.org/assets/docs/sumari/Reviewers-Manual_Methodology-for-JBI-
Scoping-Reviews_2015_v2.pdf
Totkidis, V. (2010). Applying the Nominal Group Technique (NGT) in Community Based
Action Research for Health Promotion and Disease Prevention. The Australian
Community Psychologist, 22(1), 18–29.
Toulany, A., Wong, M., Katzman, D. K., Akseer, N., Steinegger, C., Hancock-Howard, R. L., &
Coyte, P. C. (2015). Cost analysis of inpatient treatment of anorexia nervosa in
adolescents: hospital and caregiver perspectives. CMAJ Open, 3(2), E192–E197.
https://doi.org/10.9778/cmajo.20140086
UNICEF. (2017). Immunization - Why are children dying? Retrieved November 13, 2017, from
https://www.unicef.org/immunization/index_why.html
U.S. Preventive Services Task Force. (2014, May 19). Guide to Clinical Preventive Services,
2014. Section 3. Recommendations for Children and Adolescents [Text]. Retrieved April
17, 2017, from /professionals/clinicians-providers/guidelines-
recommendations/guide/section3.html
U.S. Preventive Services Task Force. (2015, November). Fourth Annual Report to Congress on
High-Priority Evidence Gaps for Clinical Preventive Services. Retrieved April 17, 2017,
from https://www.uspreventiveservicestaskforce.org/Page/Name/fourth-annual-report-to-
congress-on-high-priority-evidence-gaps-for-clinical-preventive-services
62
Van Cleave, J., Gortmaker, S. L., & Perrin, J. M. (2010). Dynamics of Obesity and Chronic
Health Conditions Among Children and Youth. JAMA, 303(7), 623–630.
https://doi.org/10.1001/jama.2010.104
Van Cleave, J., Heisler, M., Devries, J. M., Joiner, T. A., & Davis, M. M. (2007). Discussion of
Illness During Well-Child Care Visits With Parents of Children With and Without
Special Health Care Needs. Archives of Pediatrics & Adolescent Medicine, 161(12),
1170–1175. https://doi.org/10.1001/archpedi.161.12.1170
Van Cleave, J., Kuhlthau, K. A., Bloom, S., Newacheck, P. W., Nozzolillo, A. A., Homer, C. J.,
& Perrin, J. M. (2012). Interventions to improve screening and follow-up in primary care:
a systematic review of the evidence. Academic Pediatrics, 12(4), 269–282.
https://doi.org/10.1016/j.acap.2012.02.004
VisionUK. (2013, April 11). NETSCC becomes the new home for the JLA PSPs. Retrieved
December 21, 2017, from https://www.visionuk.org.uk/netscc-becomes-the-new-home-
for-the-jla-psps/
63
Appendices
Appendix 1: Modified JLA methodology flow chart
Appendix 2: Questionnaire
Appendix 3: Master List of Indicative Questions
Appendix 4: Interim Ranked List of Unanswered Research Questions
Appendix 5: Final Top 10 Ranked List of Unanswered Research Questions
Appendix 6: Composition of the Steering Group
Appendix 7: The TARGet Kids! Primary Care Research Network
64
Appendix 1: Modified JLA methodology flow chart
Step 1:Establish Steering
Group
Step 2: Create Online Questionnaire
Step 3:Collect Research
Questions
Step 4:Identify Unanswered Research Questions
Step 5:Categorize Unanswered
Research Questions
Step 6:
Interim Ranking Step
Step 7:
Final Ranking Workshop
Step 8: Disseminate Findings
65
Appendix 2: Questionnaire
Questionnaire
Thank you for participating in TARGet Kids!. In order to conduct research that is relevant to you
and your child, we are interested in understanding the most important unanswered questions that
parents have about children’s preventive care. Preventive care helps parents keep their children
as healthy as possible by promoting healthy lifestyles and behaviour as well as screening to
prevent children’s illness and injury.
Unanswered questions may be general or may be focused on an issue of interest to you or your
child. Please answer the following:
1) Think about the most important ways that you can help your child stay as healthy
as possible growing up. Please share with us one or several questions that you have
about keeping children healthy.
These can be questions that you would like to ask your child’s health care provider (such as
doctor, nurse practitioner, nurse, dietitian, or other) or questions that your child’s health care
provider cannot answer and you think should be the subject of future health research. They can
be about preventing your child from becoming sick or injured, screening for illness, or ways to
maintain a healthy lifestyle and supportive family relationships, for example.
It is easy to overlook questions that may be important to you. In the next part of this
questionnaire, we have provided some questions to help you think of different ways to keep your
child healthy as he or she grows up. We are asking you to think about health problems that some
children develop as they grow up, and ways to prevent these from happening. You can submit as
many questions as you would like for each of these issues:
1) Do you have questions related to child development and learning (including speaking,
reading and writing, motor skills like walking or using a pencil, social skills, or others)?
2) Do you have questions about your child’s growth, nutrition and physical activity
(including healthy eating, starting solid foods, milk or juice intake, obesity, time in front
of the TV or computer, active play, or others)?
66
3) Do you have questions related to preventing childhood infections or other diseases
(including vaccines for prevention, colds and ear infections, asthma, allergies, or others)?
4) Do you have questions related to behaviour and discipline (including crying or temper
tantrums, setting limits, toilet training, bullying, or others)?
5) Do you have questions related to oral (mouth) health (including prevention of cavities),
vision and hearing?
6) Do you have questions related to safety and injury prevention (including car seats, safe
sleeping environments, or preventing injuries such as broken bones, poisoning, or burns,
or others)?
7) Do you have questions relating to parenting and family support (including parental
stress, depression, creating a supportive home environment, parenting skills, or others)?
8) Do you have questions about your child’s sleeping or night-time waking (such as sleep
habits, snoring or breathing problems, night terrors, feeding at night, sleeping in his or
her own bed, or others)?
9) Other questions: If you have questions that do not fall under one of the topics listed
above, please list them here.
67
Appendix 3: Master List of Indicative Questions
Rank Question
1 What are effective strategies for parents to discipline children?
2 How do electronic devices (such as computers, cell phones and tablets) impact
development?
3 What are effective interventions to promote healthy sleep habits?
4 What interventions increase physical activity in children?
5 What are effective interventions for childhood obesity prevention?
6 How much screen time is appropriate for children?
7 What are effective interventions to prevent and manage temper tantrums?
8 What are effective strategies for screening and prevention of mental health problems?
9 What interventions prevent dental caries?
10 What are effective methods for screening for developmental delay in children?
11 What are the health benefits of vitamin supplementation?
12 What are effective strategies for vision screening in children?
13 Do children who are picky eaters require micronutrient screening?
14 What strategies prevent asthma?
15 What strategies prevent allergies?
16 What strategies optimize immunization?
17 How much sleep optimizes development?
18 How can viral respiratory illnesses be prevented?
19 What are effective strategies for screening for obstructive sleep apnea?
20 What are the health and developmental effects of pesticides?
21 What interventions promote social skill development?
22 What are the health and developmental effects of electromagnetic waves from
electronic devices?
23 What is the health and developmental impact of consumption of organic food vs. non-
organic food?
24 What are effective strategies for early identification of language difficulties?
25 How can autism spectrum disorder be identified early?
26 What is the impact of daycare attendance on child health and development?
27 What is the optimal timing to introduce electronic devices to children?
28 What are predisposing factors and interventions to prevent picky eating?
29 What are the predictors of obesity in children?
30 Does screen time (exposure to TV, tablets, computers, etc.) cause vision problems?
31 How can iron deficiency anemia be prevented in infants and toddlers?
32 What is the impact of daily exercise on health outcomes?
33 What is the impact of outdoor play on developmental outcomes?
34 What strategies improve the provision of anticipatory guidance by clinicians?
35 What is the impact of air pollution on children in urban centres?
36 When should hearing tests be performed in children?
37 What is the optimal timing of the introduction of solid food in babies, in order to
prevent disease?
38 What strategies can prevent ADHD?
68
39 What is the impact of alternative schools on child development?
40 What are effective methods to promote hand hygiene?
41 How does parental mental health affect the mental health of children?
42 What is the impact of parental stress on children?
43 Can anticipatory guidance decrease the rate of childhood obesity?
44 When should parents start flossing their children’s teeth?
45 When should children first see an eye doctor?
46 What is the most effective way to provide dental and vision care for children living in
poverty?
47 What are the predictors of influenza immunization among children and parents?
48 What can be done to prevent immunized children from being exposed to disease from
unimmunized children?
49 What is the impact of breastfeeding beyond 1 year of age on child health and
development?
50 Is there a relationship between nutrition and behaviour?
51 What steps can be taken to prevent the development of inflammatory bowel disease?
52 What steps can be taken to prevent the development of diabetes?
53 What is the impact of anticipatory guidance around injury prevention on rates of
childhood injury?
54 What strategies lead to improved medication compliance?
55 What school policies exist to prevent obesity?
56 What is the effect of primary healthcare anticipatory guidelines on health outcomes and
health care cost?
57 What are the predictors of ADHD?
58 What impact does the provision of primary care by pediatricians have on health
outcomes?
59 What is the role of complementary medicine and probiotics in the prevention of
common childhood illnesses?
60 How does being raised in a single parent household affect child health and
development?
61 What interventions can decrease the risk of cancer in children?
62 What is the impact of drinking non-dairy milk (such as soy or almond) on health
outcomes in children?
63 What modifiable factors influence the amount of screen time?
64 Do products such as shampoo and soap used for children contain chemicals that are
known to cause disease in humans?
65 What are the predictors and outcomes of temper tantrums?
66 What are the impacts of universal parenting support programs such as Ontario Early
Years Centres on child health and development?
67 What is the impact of divorce on the mental health of children?
68 How does conflict in the home affect the mental health of children?
69 What is the effect of growth monitoring for obesity prevention?
70 What perinatal strategies work to prevent hospitalization or ER visits in the first month
of life?
71 How can the impact of poverty on health be moderated by physicians?
69
72 What are the main barriers to accessing primary health care for urban children?
73 What are the health effects of processed foods on children?
74 How can learning disability be prevented?
75 Does being cared for by an employed caregiver have any impacts on a child’s mental
health?
76 Does consumption of meat or dairy products from animals treated with hormones
impact the growth and development of children?
77 Do children receiving antibiotics benefit from probiotics?
78 How do antibiotics impact long-term health?
79 How can bullying be prevented?
70
Appendix 4: Interim Ranked List of Unanswered Research Questions
Interim Ranked List of Unanswered Research Questions
Rank Question
1 What are effective interventions for obesity prevention in young children?
2 How do electronic devices (such as computers, cell phones and tablets) impact
development?
3 What are effective strategies for screening and prevention of mental health problems?
4 What strategies optimize immunization rates?
5 What are effective methods for screening for developmental delay in children?
6 Is there a relationship between nutrition and behaviour?
7 What interventions promote social skill development?
8 What is the impact of parental stress on children?
9 What interventions increase physical activity in children?
10 What are the predictors of obesity in children?
11 How can autism spectrum disorder be identified early?
12 What are the health and developmental effects of pesticides?
13 What is the health and developmental impact of consumption of organic food vs. non-
organic food?
14 What is the optimal timing of the introduction of solid food in babies, in order to prevent
disease?
15 What strategies prevent allergies?
16 What is the impact of outdoor play on developmental outcomes?
17 What impact does the provision of primary care by pediatricians have on health
outcomes?
18 What interventions prevent dental caries?
19 How much screen time is appropriate for children?
20 What strategies prevent asthma?
21 What are effective strategies for parents to discipline children?
22 What are effective interventions to promote healthy sleep habits?
23 What is the impact of daycare attendance on child health and development?
24 What are effective strategies for early identification of language difficulties?
25 What strategies can prevent ADHD?
26 How does parental mental health affect the mental health of children?
27 What are the health benefits of vitamin supplementation?
28 How can viral respiratory illnesses be prevented?
29 How can iron deficiency anemia be prevented in infants and toddlers?
30 What is the impact of daily exercise on health outcomes in young children?
31 What are effective strategies for vision screening in children?
32 How much sleep optimizes development?
33 What are the health and developmental effects of electromagnetic waves from electronic
devices?
34 What are predisposing factors and interventions to prevent picky eating?
35 Do children who are picky eaters require micronutrient screening?
71
36 Does screen time (exposure to TV, tablets, computers, etc.) cause vision problems?
37 What are effective strategies for screening for obstructive sleep apnea?
38 What is the optimal timing to introduce electronic devices to children?
39 What are effective interventions to prevent and manage temper tantrums?
72
Appendix 5: Final Top 20 Ranked List of Unanswered Research Questions
Rank Question
1 What are effective strategies for screening and prevention of mental health problems?
problems?problems? 2 What interventions increase physical activity in children?
3 What is the impact of daycare attendance on child health and development?
4 What are effective interventions for obesity prevention in young children?
5 What interventions promote social skill development?
6 What is the impact of parental stress on children?
7 What are effective methods for screening for developmental delay in children?
8 What are effective strategies for behaviour management in children?
9 What nutritional factors affect child behaviour?
10 How much screen time is appropriate for children?
11 What are effective interventions to promote healthy sleep habits?
12 What strategies optimize immunization rates?
13 What are predisposing factors and interventions to prevent picky eating?
14 How can iron deficiency anemia be prevented in infants and toddlers?
15 How can viral respiratory illnesses be prevented?
16 What interventions prevent dental caries?
17 How much sleep optimizes development?
18 What strategies prevent allergies?
19 What are the predictors of obesity in children?
20 What strategies can prevent ADHD?
73
Appendix 6: Composition of the Steering Group
Parents:
Caroline Alphonso
Jennifer Alton
Mia Brown
Jessica Morrison Smith
Fenella Petrie
Clinicians:
Tony Barozzino, MD – Pediatrician
Bernadette Becker, RN
Anh Do, MD – Pediatrician
Jonathon Maguire, MD - Pediatrician
Sheila Jacobson, MD - Pediatrician
74
Appendix 7: The TARGet Kids! Primary Care Research Network
TARGet Kids! is a primary care research network affiliated with the University of Toronto, in
Toronto, Ontario, Canada. Its aim is to advance the scientific basis for chronic disease prevention
and to develop innovative solutions for common children’s health problems. TARGet Kids!
enrolls children between 0 and 5 years of age attending primary care visits at clinics across the
Greater Toronto Area. More information about the TARGet Kids! network can be found at
http://www.targetkids.ca.
75
Copyright Acknowledgements
Certain portions of text were reproduced from Lavigne et al., 2017, with permission from the
British Medical Journal Permissions Team.
Figure 2, table 2, table 3 and table 4 were reproduced or adapted with permission from Lavigne
et al., 2017.
