““You Can Observe a Lot by Just Watching”: You Can Observe a Lot by Just Watching”: Methods for Assessing ParentingMethods for Assessing Parenting

Carla MatiasCarla Matias

Stephen B.C. ScottStephen B.C. Scott

Thomas G. O’ConnorThomas G. O’Connor

Acknowledgements: This work was made possible with funding from the ESRC Methods Programme grant, the Joseph Rowntree Foundation, The Jacobs Foundation, and the PsychiatryResearch Trust. Additional funding ( CM PhD research) from Ministry of Science & Technology,Lisbon, Portugal

Aims & OutlookAims & Outlook

1.1. To review the context in which the To review the context in which the measures component of the ESRC measures component of the ESRC programme grant was conceivedprogramme grant was conceived

2.2. Describe some current (and probably Describe some current (and probably accurate) thinking on the measurement accurate) thinking on the measurement of parent-child relationshipsof parent-child relationships

3.3. Describe in some detail a study that used Describe in some detail a study that used an intervention design to tackle a major an intervention design to tackle a major conceptual measurement questionconceptual measurement question

Research Questions for Research Questions for Parenting/Family StudiesParenting/Family Studies

1. What are the evidence-based 1. What are the evidence-based assessments, and is there a assessments, and is there a methodological “best practice?”methodological “best practice?”

2. How good is the fit between constructs 2. How good is the fit between constructs and their measurement?and their measurement?

3. Is there convergence among theories re: 3. Is there convergence among theories re: measurement?measurement?

4. Are there exemplar measures within each 4. Are there exemplar measures within each measurement approach?measurement approach?

Research Questions for Research Questions for Parenting/Family Studies, 1Parenting/Family Studies, 1

Evidence-based assessments and “best Evidence-based assessments and “best practice”practice”

1.1. There is robust evidence that observational, There is robust evidence that observational, questionnaire, and interview methods questionnaire, and interview methods allall have a place at the ‘research table’have a place at the ‘research table’

2.2. A A Multi-methodMulti-method approach is a central take- approach is a central take-home messagehome message

3.3. Valid & reliable measures exist, so there is Valid & reliable measures exist, so there is no need to ‘re-invent the wheel’ (although no need to ‘re-invent the wheel’ (although tweaking is normal)tweaking is normal)

Research Questions for Research Questions for Parenting/Family Studies, 2Parenting/Family Studies, 2

Are measures “good enough” at capturing central Are measures “good enough” at capturing central constructsconstructs

1.1. There is general consensus re: the dimensions of There is general consensus re: the dimensions of the parent-child relationship that require attentionthe parent-child relationship that require attention

2.2. There is notably modest application of research There is notably modest application of research measures in clinical contextsmeasures in clinical contexts

3.3. There is a trade-off between detail/depth and There is a trade-off between detail/depth and time/moneytime/money

4.4. There is no There is no singlesingle best approach; neither is there a best approach; neither is there a satisfactory “quick & dirty” way to assess the parent-satisfactory “quick & dirty” way to assess the parent-child relationshipchild relationship

Research Questions for Research Questions for Parenting/Family Studies, 3Parenting/Family Studies, 3

Theory unto measurementTheory unto measurement1.1. Different theories emphasize different Different theories emphasize different

measures and settings, to some extentmeasures and settings, to some extent

2.2. There are theory-based differences in level There are theory-based differences in level of analysis of analysis (cf. parental commands v. mutuality)(cf. parental commands v. mutuality)

3.3. A single observational paradigm might serve A single observational paradigm might serve several models and measurement several models and measurement approachesapproaches

Research Questions for Research Questions for Parenting/Family Studies, 4Parenting/Family Studies, 4

Exemplars and examples of measurementExemplars and examples of measurement1.1. Social learning theory: Social learning theory: observationsobservations of of

coercive (harsh, negative control) coercive (harsh, negative control) interactions patterns are centralinteractions patterns are central

2.2. Attachment theory: Attachment theory: observationsobservations of sensitive of sensitive parenting, secure base behaviorparenting, secure base behavior

3.3. Parenting typologies: q’naire Parenting typologies: q’naire oror observations observations of authoritative parentingof authoritative parenting

Inevitable Challenges for Inevitable Challenges for Parenting/Family StudiesParenting/Family Studies

Measures selection is a more Measures selection is a more confounded process than theory confounded process than theory selectionselection

Considerable time & money are needed Considerable time & money are needed to meet the best-practice standard of to meet the best-practice standard of multi-method assessmentmulti-method assessment

Researcher prerogative to fiddle with and Researcher prerogative to fiddle with and make up measures goes under-make up measures goes under-challengedchallenged

A Case StudyA Case Study

Primary Age Learning Skills Primary Age Learning Skills (PALS Project)(PALS Project) (Scott, O’Connor & Futh, 2005Scott, O’Connor & Futh, 2005

Primary Age Learning Skills Primary Age Learning Skills (PALS Project)(PALS Project) (Scott, O’Connor & Futh, 2005)Scott, O’Connor & Futh, 2005)

In the context of an intervention study using a In the context of an intervention study using a cognitive-behavioral approach, how should we cognitive-behavioral approach, how should we assess the parent-child relationship?assess the parent-child relationship? Should we stick to those measures used in Should we stick to those measures used in

standard assessments of this sort of standard assessments of this sort of intervention intervention (i.e., behavioral counts)(i.e., behavioral counts)??

Which sorts of measures of relationship quality Which sorts of measures of relationship quality would be most sensitive to change induced by would be most sensitive to change induced by this intervention?this intervention?

Two Major Classes of Measurement Two Major Classes of Measurement of Parent-Child Relationshipof Parent-Child Relationship

RESPONDENT-Based Measures - RESPONDENT-Based Measures - ‘Subjective’‘Subjective’ Questionnaires (parent, child)Questionnaires (parent, child) Interviews (parent, child)Interviews (parent, child)

DIRECT OBSERVATION - ‘Objective’DIRECT OBSERVATION - ‘Objective’ Coding of interactions by independent ratersCoding of interactions by independent raters

Usefulness of Report MethodsUsefulness of Report Methods

AdvantagesAdvantages

Lower CostLower Cost

Accessibility & Flexibility - Accessibility & Flexibility - Quick ResultsQuick Results

Breadth of Assessment: 1) Breadth of Assessment: 1) behaviour over longer time behaviour over longer time span & in different span & in different contexts; 2) Information on contexts; 2) Information on behaviour, attitudes, and behaviour, attitudes, and other aspects of family lifeother aspects of family life

DisadvantagesDisadvantages

Informant Bias (e.g. Informant Bias (e.g. distorted recall; social-distorted recall; social-desirability; interpretation desirability; interpretation issues, etc)issues, etc)

Method Error (e.g. item-Method Error (e.g. item-wording and scaling wording and scaling interpretation; not interpretation; not

context-specific) context-specific)

Usefulness of Observational Usefulness of Observational MethodsMethods

AdvantagesAdvantages

Greater Objectivity - Greater Objectivity - Assessing actual rather Assessing actual rather than reported behaviourthan reported behaviour

Contextual sensitivity; Contextual sensitivity; intensity & frequency of intensity & frequency of behaviour; behaviour;

Predictive Validity & Predictive Validity & Sensitive to change is Sensitive to change is well-documentedwell-documented

DisadvantagesDisadvantages

Subject(s) Bias: reactivity Subject(s) Bias: reactivity effects, e.g., to the effects, e.g., to the camera or researchercamera or researcherMethod Error: 1) Context-Method Error: 1) Context-specificity (behaviour specificity (behaviour sampled at one point in sampled at one point in time); 2) Observer Biastime); 2) Observer BiasExpensive and time-Expensive and time-consumingconsuming

To Observe or Not To Observe?To Observe or Not To Observe?

Using a single-source approach is not Using a single-source approach is not now regarded as scientifically acceptable now regarded as scientifically acceptable because any method is affected by sizable because any method is affected by sizable errorerrorThe study of complex constructs such as The study of complex constructs such as parenting necessarily require adoption of a parenting necessarily require adoption of a multi-method multi-informantmulti-method multi-informant approach approachOn the other hand, the overlap between On the other hand, the overlap between methods is usually modest, at bestmethods is usually modest, at best

Primary Age Learning Skills Primary Age Learning Skills (PALS Project)(PALS Project) (Scott, O’Connor & Futh, 2005)Scott, O’Connor & Futh, 2005)

Evaluates effectiveness of a behavioural Evaluates effectiveness of a behavioural parenting programme for an at-risk population of parenting programme for an at-risk population of school-age childrenschool-age children

Multi-method assessmentMulti-method assessment at pre, post and follow- at pre, post and follow-upup 1) Respondent-based methods (Questionnaire 1) Respondent-based methods (Questionnaire

& Interview) - evaluate ‘subjective’ change & Interview) - evaluate ‘subjective’ change 2) Direct observation (development of new 2) Direct observation (development of new

coding scheme) - evaluates ‘objective’ changecoding scheme) - evaluates ‘objective’ change

A Focus on Attachment Measurement A Focus on Attachment Measurement using Direct Observation:using Direct Observation:

The CARP The CARP (Matias et al, 2005)(Matias et al, 2005)

What do we mean by attachment-related What do we mean by attachment-related parenting?parenting? Core concept of Core concept of Sensitive RespondingSensitive Responding - - Operationalisation:Operationalisation:

Responsiveness to childResponsiveness to childChild Mindedness Child Mindedness FacilitationFacilitationResponsive EngagementResponsive EngagementPromoting Autonomy/EncouragingPromoting Autonomy/EncouragingWarmthWarmth

The PALS Parenting BatteryThe PALS Parenting Battery

Observation of parent-child interactions in semi-Observation of parent-child interactions in semi-structured play, structured task, and clean-up structured play, structured task, and clean-up 25’+25’+

Parent questionnaires of instrumental and Parent questionnaires of instrumental and expressive warmth, conflict, coercion, punitiveness expressive warmth, conflict, coercion, punitiveness 15’+15’+

Parent interviews of discipline strategies and Parent interviews of discipline strategies and normal routines normal routines 20’+20’+

Parent interviews to assess the parent’s ‘working Parent interviews to assess the parent’s ‘working model of the child’ model of the child’ 25’+25’+

Attachment story stem narratives with the child Attachment story stem narratives with the child 25’+25’+

NB: length of assessment in small print; for observations, add 40’+ for coding; for interviews add 30’+ for coding; for attachment narratives add 30’+ for coding

What is Novel about PALS?What is Novel about PALS?MultiMulti-method, -method, multimulti-theory prevention -theory prevention

study in a study in a multimulti-ethnic sample-ethnic sampleMulti-methodMulti-method

Observation, interview & questionnaire of parent-Observation, interview & questionnaire of parent-child relationship and outcomechild relationship and outcome

Multi-theoryMulti-theoryAttachment and social learning theory approach to Attachment and social learning theory approach to studying parenting in everyday situationsstudying parenting in everyday situations Attachment is a neglected area of research in interventions with Attachment is a neglected area of research in interventions with

school-age childrenschool-age children Will improvements in one measure or theory translate to improvements Will improvements in one measure or theory translate to improvements

in another?in another?

Multi-ethnicMulti-ethnicA majority of parents (Southwark) were minority and A majority of parents (Southwark) were minority and poorpoor

PALS Summary of Findings from PALS Summary of Findings from ReportReport Methods Methods

Changes in Parenting Changes in Parenting

No change on questionnaireNo change on questionnaire

Interview - Increases in reported use of ‘calm discipline’ and decreased use of ‘criticism’ but no change in reported use of praise

Changes in Child BehaviourChanges in Child Behaviour

Questionnaire & Interview: No improvement Questionnaire & Interview: No improvement in child antisocial behaviour, Hyperactivity in child antisocial behaviour, Hyperactivity /ADHD & child emotional symptoms /ADHD & child emotional symptoms

Is that the end of the story?Is that the end of the story?

DIRECT OBSERVATION: Parenting Outcome (i)DIRECT OBSERVATION: Parenting Outcome (i)

1. Change in Sensitive Responding 1. Change in Sensitive Responding Intention to Treat AnalysisIntention to Treat Analysis

Graph 1 Graph 1 Effect size(Effect size(es)es) = = 0.370.37

10

10.5

11

11.5

12

Pre Follow up

Intervention n =39Control n = 39

DIRECT OBSERVATION: DIRECT OBSERVATION: Parenting Outcome (ii)Parenting Outcome (ii)

2. Change in Sensitive Responding 2. Change in Sensitive Responding Per ProtocolPer Protocol Analysis Analysis

Graph 2 Effect Size ( Graph 2 Effect Size ( es)es) = = 0.670.67

10

10.5

11

11.5

12

12.5

13

Pre Follow up

5+ sessionsattended n = 15Control n = 39

DIRECT OBSERVATION - Child Outcome (i)DIRECT OBSERVATION - Child Outcome (i)

3. Change in Child Attention on Task3. Change in Child Attention on TaskIntention to Treat AnalysisIntention to Treat Analysis

Graph 3Graph 3 Effect Size (Effect Size (es)es)

== . .32 32

16

17

18

19

Pre Follow up

Intervention n=39Control n = 39

DIRECT OBSERVATION - Child Outcome (ii)DIRECT OBSERVATION - Child Outcome (ii)

4. Change in Child Attention on Task 4. Change in Child Attention on Task Per Protocol AnalysisPer Protocol Analysis

Graph 4Graph 4 Effect Size (Effect Size (es)es) = .78 = .78

15

16

17

18

19

20

Pre Follow up

5+ sessions n =15Control n = 39

Direct ObservationDirect Observation 5. Ethnicity Effects (i) - Mean Differences at Baseline 5. Ethnicity Effects (i) - Mean Differences at Baseline

02468

101214

BlackAfrican(n=43)

WhiteBritish(n=17)

Black Afro-Caribbean

(n=16)

Sensitive Responding Mean Values at Time 1

ANOVA ANOVA showed that showed that West African West African group scored group scored significantly significantly lower in SR lower in SR compared to compared to White British White British (p<.01) & (p<.01) & Black Afro-Black Afro-Caribbean Caribbean groups groups

(p<.05)(p<.05)

Direct ObservationDirect ObservationEthnicity Effects (ii)Ethnicity Effects (ii) - Intervention Outcome - Intervention Outcome

6. Despite ethnic differences at baseline, intervention 6. Despite ethnic differences at baseline, intervention

was equally successful in increasing Sensitive was equally successful in increasing Sensitive Responding across all ethnic groupsResponding across all ethnic groups

PALS Study Summary:PALS Study Summary:Value of Observational MethodsValue of Observational Methods

A Social-Learning based Intervention increased Sensitive A Social-Learning based Intervention increased Sensitive Responding, an Attachment derived conceptResponding, an Attachment derived concept

Interventions effects may generalize across method & theoryInterventions effects may generalize across method & theory

Ethnic differences in observed Sensitive Responding were Ethnic differences in observed Sensitive Responding were not found in self-report measuresnot found in self-report measures

How families present and represent themselves requires sensitive How families present and represent themselves requires sensitive assessmentassessment

Group differences prior to treatment does not imply group Group differences prior to treatment does not imply group moderation of treatment effectsmoderation of treatment effects

Observed Attention on Task increased, but no change in Observed Attention on Task increased, but no change in child behaviour was detected in respondent-based child behaviour was detected in respondent-based measuresmeasures

Observed data imply ‘real’ observable change, but the definitions of Observed data imply ‘real’ observable change, but the definitions of robust change robust change maymay imply more than mono-method evidence imply more than mono-method evidence

Main ConclusionsMain Conclusions

Studies relying on a single method are suspectStudies relying on a single method are suspectMulti-method assessments of parent-child Multi-method assessments of parent-child relationship quality and outcomes are scientifically relationship quality and outcomes are scientifically required, but difficult to attainrequired, but difficult to attainThe call for a multi-method approach implies the lack The call for a multi-method approach implies the lack of a single ‘gold standard’:of a single ‘gold standard’: for detecting change and a ‘successful’ interventionfor detecting change and a ‘successful’ intervention for assessing cultural variationsfor assessing cultural variations for defining a meaningful effect sizefor defining a meaningful effect size

The application of findings from multi-method The application of findings from multi-method investigations to a clinical or social policy context will investigations to a clinical or social policy context will be neither direct nor easybe neither direct nor easy