Issues in Selecting Measurement Issues in Selecting Measurement Tools in Assessing Eligibility, Tools in Assessing Eligibility,

Process and Outcomes in Social Process and Outcomes in Social and Behavioral Intervention and Behavioral Intervention

ResearchResearch

Patrick R. Clifford, Ph.D.Patrick R. Clifford, Ph.D.Professor and Associate Dean for Professor and Associate Dean for

Research Research University of Medicine and Dentistry University of Medicine and Dentistry of New Jersey School of Public Healthof New Jersey School of Public Health

NIH Summer Institute on Social and NIH Summer Institute on Social and Behavioral Intervention ResearchBehavioral Intervention Research

July 9 – 13, 2012July 9 – 13, 2012

A Prefatory NoteA Prefatory NoteGiven the allocated timeframe combined Given the allocated timeframe combined with the breadth of material specific to with the breadth of material specific to measurement issues relevant to social and measurement issues relevant to social and behavioral intervention research (not to behavioral intervention research (not to mention the diversity of participant mention the diversity of participant academic training and professional academic training and professional experience), the information presented experience), the information presented must be viewed as an overview of issues must be viewed as an overview of issues related to measurement tool selection related to measurement tool selection (i.e., this presentation centers on relatively (i.e., this presentation centers on relatively few issues thought to be of particular few issues thought to be of particular importance). importance).

Intervention ResearchIntervention Research• Whether conducting program evaluation or Whether conducting program evaluation or

intervention outcomes research, it is intervention outcomes research, it is important to be clear about the purpose(s) important to be clear about the purpose(s) for which the data will be used. for which the data will be used.

• Clarity of purpose will markedly affect what Clarity of purpose will markedly affect what data are to be collected, how they will be data are to be collected, how they will be collected, and how they will be presented. collected, and how they will be presented.

• Most intervention research/evaluation Most intervention research/evaluation decisions regarding outcomes have their decisions regarding outcomes have their pros and cons. pros and cons.

Important Initial Important Initial Questions Questions

• What is/are the What is/are the Research QuestionResearch Question(s) of interest?(s) of interest?• What are the related What are the related Research HypothesesResearch Hypotheses? ? • What variables (e.g., independent, dependent, What variables (e.g., independent, dependent,

covariate) are needed to investigate the research covariate) are needed to investigate the research hypotheses and answer the research question? hypotheses and answer the research question?

• How will the data be collected (e.g., RI, IVR, How will the data be collected (e.g., RI, IVR, online)?online)?

• How do said variables need to be operationalized? How do said variables need to be operationalized? • Do relevant psychometrically sound (with respect Do relevant psychometrically sound (with respect

to the population of interest) to the population of interest) measures/procedures exist? measures/procedures exist?

Screening for EligibilityScreening for EligibilityFactors to Consider: Factors to Consider: – Participant recruitment (e.g., media, internet, clinical Participant recruitment (e.g., media, internet, clinical

prog.) prog.) – Participant characteristics (e.g., homeless, underage)Participant characteristics (e.g., homeless, underage)– Study inclusion/exclusion criteriaStudy inclusion/exclusion criteria– Invasiveness of data collection tool (e.g., blood, tissue) Invasiveness of data collection tool (e.g., blood, tissue) – Subject burden (e.g., travel, assessment time, scheduling) Subject burden (e.g., travel, assessment time, scheduling) – Sensitivity of questions (e.g., illegal or sexual behavior) Sensitivity of questions (e.g., illegal or sexual behavior) – Research Interviewer training (e.g., client engagement) Research Interviewer training (e.g., client engagement) – Laboratory services (e.g., blood analysis services) Laboratory services (e.g., blood analysis services) – Yield (i.e., total screened/eligible & provide consent) Yield (i.e., total screened/eligible & provide consent) – Costs (e.g., instrument development and administration) Costs (e.g., instrument development and administration)

Potential Consequences of Potential Consequences of ScreeningScreening

• Burdensome, invasive, and/or sensitive Burdensome, invasive, and/or sensitive screening activities can:screening activities can:

•diminish recruitment efforts diminish recruitment efforts •result in an underpowered study result in an underpowered study •contribute to sample bias contribute to sample bias

• Overly restrictive inclusion/exclusion Overly restrictive inclusion/exclusion criteria reduces external validity (i.e., criteria reduces external validity (i.e., generalizability) generalizability)

• Electronic (e.g., IVR) and internet methods Electronic (e.g., IVR) and internet methods of data collection can be efficient but raise of data collection can be efficient but raise data integrity and respondent concerns data integrity and respondent concerns (e.g., erroneous and missing data, (e.g., erroneous and missing data, respondent identity and state of being).respondent identity and state of being).

Screening Criteria, Tools Screening Criteria, Tools and Methodsand Methods

To summarize, considerations To summarize, considerations pertaining to screening criteria, the pertaining to screening criteria, the selection of measurement tools, and the selection of measurement tools, and the method of screening for participant method of screening for participant eligibility are critical, and if not eligibility are critical, and if not performed adequately can undermine performed adequately can undermine the value of the best designed studies.the value of the best designed studies.

Process EvaluationProcess Evaluation• Process evaluation centers on efforts to Process evaluation centers on efforts to

assess the extent to which an intervention is assess the extent to which an intervention is implemented as planned and treatments implemented as planned and treatments delivered as intended (i.e., the extent to delivered as intended (i.e., the extent to which the study protocol is followed) which the study protocol is followed)

• Process evaluation is important for Process evaluation is important for understanding and interpreting study understanding and interpreting study findings (e.g., it would be a mistake to findings (e.g., it would be a mistake to interpret negative findings as the result of an interpret negative findings as the result of an ineffective treatment if the treatment was not ineffective treatment if the treatment was not delivered appropriately; sometimes referred delivered appropriately; sometimes referred to as a Type III error) to as a Type III error)

Assessing Program Assessing Program ImplementationImplementation

Factors to consider: Factors to consider: – Resources (e.g., office space, equipment)Resources (e.g., office space, equipment)

– Interventionist training (e.g., performance criterion)Interventionist training (e.g., performance criterion)– Interventionist supervision (e.g., protocol adherence) Interventionist supervision (e.g., protocol adherence) – Treatment group assignment (e.g., randomization) Treatment group assignment (e.g., randomization) – Participant participation (e.g., attendance, Participant participation (e.g., attendance,

engagement) engagement) – Data collection activities (e.g., at relevant time points) Data collection activities (e.g., at relevant time points) – Research Interviewer training (e.g., performance Research Interviewer training (e.g., performance

criteria) criteria) – Research Interviewer supervision (e.g., protocol Research Interviewer supervision (e.g., protocol

adherence) adherence)

Inadequate Inadequate Implementation ProcessImplementation Process

• Exposure to ineffective Exposure to ineffective treatments/interventionstreatments/interventions

• Exposure to sub-threshold levels of the Exposure to sub-threshold levels of the intervention intervention

• Reduced design sensitivity and power Reduced design sensitivity and power • Increased probability of type II error Increased probability of type II error • Increased likelihood of confoundingIncreased likelihood of confounding• Interpretation of study findings severely Interpretation of study findings severely

hampered hampered • Costly (i.e., inefficient use of research Costly (i.e., inefficient use of research

resources to include money, effort, and time)resources to include money, effort, and time)

Assessing Intervention Assessing Intervention OutcomesOutcomes

• To maximize social and behavioral To maximize social and behavioral intervention outcomes research, it will be intervention outcomes research, it will be necessary to assess the participants and necessary to assess the participants and their relevant behaviors of concern (e.g., their relevant behaviors of concern (e.g., smoking, exercise, alcohol use) prior to smoking, exercise, alcohol use) prior to the intervention (i.e., baseline), during the the intervention (i.e., baseline), during the intervention (e.g., mediator variables), intervention (e.g., mediator variables), end-of-intervention, and post-intervention end-of-intervention, and post-intervention completion, which may have unexpected completion, which may have unexpected results (e.g., enhanced intervention results (e.g., enhanced intervention effects and reduced design sensitivity). effects and reduced design sensitivity).

Measurement Tool Measurement Tool ConsiderationsConsiderations• Complexity of the relevant constructs (e.g., Complexity of the relevant constructs (e.g.,

dimensionality of measures, abstract vs. tangible) dimensionality of measures, abstract vs. tangible) • Sensitivity of the information to be collected (e.g., Sensitivity of the information to be collected (e.g.,

illegal behavior, consequences of honest reporting)illegal behavior, consequences of honest reporting)• Method of data collection (e.g., in-person interview, Method of data collection (e.g., in-person interview,

IVR, internet) and potential benefits and IVR, internet) and potential benefits and disadvantages disadvantages

• Invasiveness of the data collection tool (e.g., blood Invasiveness of the data collection tool (e.g., blood sample vs. questionnaire) as it can affect attrition sample vs. questionnaire) as it can affect attrition

• Instrument psychometric properties (i.e., reliability, Instrument psychometric properties (i.e., reliability, validity) validity)

• Data analysis plan (e.g., continuous vs. categorical) Data analysis plan (e.g., continuous vs. categorical) • Correlations among variables (e.g., repeated Correlations among variables (e.g., repeated

measurements, multicollinearity)measurements, multicollinearity)

Instrument Selection Instrument Selection Considerations Considerations

• The measure should be sufficiently sensitive The measure should be sufficiently sensitive to identify meaningful changeto identify meaningful change

• The measure should be standardized (i.e., The measure should be standardized (i.e., possess acceptable psychometric properties possess acceptable psychometric properties and provide a measure of the behavior of and provide a measure of the behavior of interest) and minimal (e.g., SIP vs. DrInC, interest) and minimal (e.g., SIP vs. DrInC, Procidano Support Measures)Procidano Support Measures)

• The resources allocated to the measurement The resources allocated to the measurement of a construct should be proportional to its of a construct should be proportional to its importance importance

• Selected measures should be the most cost Selected measures should be the most cost effective available to measure the constructs effective available to measure the constructs of interest of interest

Additional Additional ConsiderationsConsiderations

• Data collection via the internet Data collection via the internet

• Assessment reactivity effectsAssessment reactivity effects– Frequency of assessment Frequency of assessment – Comprehensiveness of the assessment Comprehensiveness of the assessment

batterybattery

• Participant tracking systems to Participant tracking systems to minimize attritionminimize attrition

Electronic Data Electronic Data CollectionCollection

• AdvantagesAdvantages– Does not require available research staff to collect Does not require available research staff to collect

datadata– Participant convenience regarding assessment Participant convenience regarding assessment

completion completion – Eliminates research staff data entry errors and costs Eliminates research staff data entry errors and costs – Access to broad populations of interest Access to broad populations of interest

• Disadvantages Disadvantages – Data integrity concerns (i.e., missing and erroneous Data integrity concerns (i.e., missing and erroneous

data) data) – Respondent identification Respondent identification – Respondent state of beingRespondent state of being

Electronic Data Electronic Data Collection BotsCollection Bots

• Originally, referred to computer Originally, referred to computer programs that generate automated programs that generate automated responses to Web survey formsresponses to Web survey forms

• ““Human Spam Bots” refers to people Human Spam Bots” refers to people who repeatedly complete and submit who repeatedly complete and submit surveys on behalf of someone elsesurveys on behalf of someone else

Procedures to Enhance Procedures to Enhance the Integrity of the Integrity of Electronic DataElectronic Data• Record time spent in activityRecord time spent in activity

• Check for items with suspect responsesCheck for items with suspect responses• Include impossible item response choices Include impossible item response choices

(e.g., ask where person heard about the (e.g., ask where person heard about the study and include responses that are study and include responses that are incorrect) incorrect)

• Check response patterns (e.g., same Check response patterns (e.g., same response choice across close-ended response choice across close-ended questions or minimal responses to open-questions or minimal responses to open-ended questions ended questions

• Check for suspicious email addresses (e.g., Check for suspicious email addresses (e.g., abcd@gmail.comabcd@gmail.com

Research Assessment Research Assessment EffectsEffects

• Both the frequency and comprehensiveness Both the frequency and comprehensiveness of the assessment battery contribute to of the assessment battery contribute to improved clinical outcomesimproved clinical outcomes

• Reduced research design sensitivityReduced research design sensitivity• Increased type II errorIncreased type II error• Methodological confoundMethodological confound• Transfer of less effective, or possibly even Transfer of less effective, or possibly even

ineffective, treatments to community ineffective, treatments to community settingssettings

The More Salient Aspects The More Salient Aspects of the Research Protocolof the Research Protocol

• Interacting with a Research InterviewerInteracting with a Research Interviewer• Frequent assessment (e.g., quarterly) Frequent assessment (e.g., quarterly) • Comprehensive assessment Comprehensive assessment • Extended post-treatment follow-up Extended post-treatment follow-up

periodperiod• Location: (i.e., treatment and research Location: (i.e., treatment and research

typically occur in the same setting typically occur in the same setting • Inclusion of secondary data sources (e.g., Inclusion of secondary data sources (e.g.,

collateral involvement, laboratory tests)collateral involvement, laboratory tests)

Minimize and/or Control for Minimize and/or Control for Potential Assessment Potential Assessment

Reactivity Reactivity EffectsEffects• Reduce assessment frequency (e.g., semi-Reduce assessment frequency (e.g., semi-

annual assessments as opposed to quarterly) annual assessments as opposed to quarterly) • Reduce the comprehensiveness of the Reduce the comprehensiveness of the

assessment battery (e.g., limit to domains assessment battery (e.g., limit to domains necessary for answering research hypotheses) necessary for answering research hypotheses)

• Use separate settings for treatment and Use separate settings for treatment and research related activities research related activities

• Include a minimal or no assessment control Include a minimal or no assessment control group group

• Consider using missing data study Consider using missing data study designs/analyses (i.e., often referred to as designs/analyses (i.e., often referred to as efficiency designs) efficiency designs)

Data Collection Data Collection ConsiderationsConsiderations• Reduce study participant-research staff Reduce study participant-research staff

interactions (e.g., use of computerized interactions (e.g., use of computerized assessments, telephone vs. in-person assessments, telephone vs. in-person interviews)interviews)

• Limit data collection to essential measures Limit data collection to essential measures • Consider data instruments that contain fewer Consider data instruments that contain fewer

items items • Avoid using highly correlated measures of Avoid using highly correlated measures of

the same construct the same construct • To the extent possible, use existing To the extent possible, use existing

information (e.g., medical record data) as information (e.g., medical record data) as opposed to participant self-report data opposed to participant self-report data

• Exclude non-essential collateral reports and Exclude non-essential collateral reports and laboratory tests laboratory tests

Participant Tracking Participant Tracking SystemsSystems

• Contribute to enhanced participant retention Contribute to enhanced participant retention rates and mitigate attrition rates and mitigate attrition

• Obtain contact information from at least two Obtain contact information from at least two persons who always know how to get in-touch persons who always know how to get in-touch with the participant (e.g., parents, relatives, with the participant (e.g., parents, relatives, friends)friends)

• Record participant’s full name, SS #, date and Record participant’s full name, SS #, date and place of birth, driver’s license # and state place of birth, driver’s license # and state issued, employment contact information, full issued, employment contact information, full names and addresses of parents (if living) names and addresses of parents (if living)

• Most importantly, obtain consent to use all Most importantly, obtain consent to use all available means to locate the participant for available means to locate the participant for follow-up follow-up