Logic models and qualitative synthesis Ideas and issues

Logic models and qualitative synthesis

Ideas and issues

18/04/23 © The University of Sheffield

2

• Why logic models?

• What are logic models?

• How do I construct a logic model?

• Qualitative synthesis

• Where does and does it not get you?

Outline


3

Link practice to outcomes

Explore complex relationships

Examine processes and systems

Include multiple sources of evidence

Identify further

research needs

Healthcare research

Logic models/conceptual models/impact models

A systematic and visual way to present relationships and outcomes

Read from left to right following a chain of reasoning


4

Health promotion literature


5

Conveying connections between a programme activities and outcomes

(the logic of the programme)

Lando et al., 2006


6

Inputs Short-term outcomes

Mid-term outcomes

Long-term outcomesActivitie

s

Lando et al., 2006

Programme evaluation


7

Relationships among programme resources planned activities, and anticipated changes or

results

The Kellog Foundation, 2004


8

Resources/inputs

Activities

Outputs

Outcomes

Impact

Planned work Intended results

W.K. Kellog Foundation, 2004

Public health


9

Explore ways in which intervention impacts on different levels of outcomesJoly et al., 2007


10

Inputs

Outputs

Long-term outcomes

Short-term out-comes

Strategies

Inter-mediate outcomes

Joly et al., 2007

Worked example• We are examining interventions to improve

the happiness of systematic reviewers

• We can find little published evidence reporting happiness outcomes

• We are wanting to model how an intervention might help reviewers


11


12

Intervention

Short term outcomes

Medium term Longer term impacts

Social support

Increased happiness

Reduce adverse stress

Improvedmental healthProg-

ramme of lunch time group walks Improved

physical health

Improved fitness

Reduction of pain

Improved diet

Heart rate

Biscuit intake

Caffeine

Headaches/back/neck

Cortisol

Personal contact

Building on the method


13

NICE Centre for Public Health Excellence

What interventions are effective in enhancing mental wellbeing in the workplace?


14

• Previous systematic review (Graveling et al. 2008) lack of evidence for effectiveness of interventions

• BUT was this lack of evidence or lack of study designs meeting inclusion criteria?


15


16

2883 de-duplicated retrieved citations

Full paper reviewed299

Reject title/abstract 2584

Reject 297Language 2, population 20, not relevant 43, Not peer reviewed 4, discussion/review 102, study design 101

Included papers 2

Are we missing something?


17

Expert reference group identified literature in addition to papers from systematic review

Proposal to use logic modelling to explore relationships


18Review paper 45

Discussion paper 35

Survey reporting associations 31

Survey reporting prevalence 30

Policy document/report 29

Cohort study 19

Book 10

Meta analysis 8

Cluster randomised controlled trial 5

Case study 4

Book chapter 3

Qualitative study 2

Randomised controlled trial 1

Controlled before and after study 1

Case control study 1

Workproductivity/competitiveness

Sickness absence behaviour

Work (stress) related Conditions

Health-relatedbehaviours& disease sequela

Health inequalities

Individual psychological‘adjustment’/wellbeing

•Job satisfaction•Commitment•Empowerment•Perceived jobstress

National policies &practices

•Economic policy•Employment -rights/ H&S

•Education /training•Social welfare

Economic & social Economic & social trendstrends•• Economic Economic --employment employment levels, wages, securitylevels, wages, security

••Changes in nature of Changes in nature of work work

••Labour marketLabour market

••Distribution of Distribution of incomeincome

Competitivenesslinked of trends inheath & wellbeing

of workforce(burden of stress-related absence) Job design:

•Workload•Control/autonomy•Role clarity

Job development:•Security•Pay/promotion equity•Learning•Flexibility

Individual characteristics•Gender, ethnicityage•Social circumstances

OrganisationalAttributes•Values & beliefs•Policies & Practices•Health & safety

Social relationships•Org support•Co-worker support•Participation•commitment

Social stratification

Life time workexperience

(risk/resilience)

EnvironmentalHazards& assets


19

Logic model creation

Little detail regarding the method

• “was drafted by the authors”

• “was developed in conjunction with an expert panel”

• “A group with a facilitator identified activities, outcomes and resources needed, initial model circulated, then refined and clarified”

• A group reviewed existing models, identified principles of new model, the model was refined following feedback


20

A systematic way to build a model?


21


22

Method

Data extractionAnalysis


23

Synthesis

Method

Extraction

Description of elements impacting on wellbeing and associations between elements extracted on developed form

Analysis


24

Synthesis


25

WARNINGReviewers of a

nervous disposition look

away now

Treating all data equallyTreating all data as qualitative

Method

Extraction

Analysis

Line by line reading of extractions,

coding of data, use of

CAQDA software (Nvivo) for storage and retrieval


26

Synthesis

Qualitative synthesis

• Combining of primary qualitative data

• Emerging approaches (Meta-synthesis, thematic synthesis, meta-ethnography etc)

• Seeking and developing recurring patterns or themes across the data


27

Thematic synthesis (Thomas & Harden 2008), examine themes across set, and by process of comparison and contrast establish meta-themes (integrate and/or create)


28

For example

Primary analysis – a paper describes outcomes following a buddy system intervention on employee wellbeing, elements of data coded as peer interaction. Another paper describes the link between working at home and reduced wellbeing, elements of data coded as social interaction.

Secondary analysis – peer interaction and social interaction data combined into social support theme, later when other data added a category of social resources is created.


29


30

Health

Well being

Job satisfaction

Work contextHealth and safetyManagement priorities/valuesSupervisor behaviourFeedback & appraisalOrganisational climateOrganisational justiceWork-place supportEmployee participationCommunication systemsIndustry type

Individual factorsIndividual response to workPersonality traitsSelf esteemSelf efficacyPsychological flexibilityPerson-environment fitValuesSocial resourcesIndividual responses to management style or working practicesOptimal workload thresholdCoping response

EconomicAnd social trends

Fixed term contractsFlexible employmentHealth inequalitiesCultural diversityDemographic changesNew technologyICTGlobalisationChange from production to service economy

Individual characteristicsGenderAge Social circumstancesEducationEthnicity Marital statusPredispositions

National polices and practicesEqual opportunities Anti-discrimination policiesFamily-friendly policiesMaximum working hours legislation

Behavioural outcomesHealth-related behaviourSickness absenceStaff turnoverPresenteeismBurnoutCitizenship

Attitudinal outcomesCommitmentMotivationEngagementEmployee expectationsPerceptions of fair treatment

Business outcomesAbsence/turnover costsPerformanceProductivityCustomer satisfactionProfitabilityOptimal staffingSatisfy statutory regulationsSafety/avoidance of litigationCorporate social responsibilityCorporate imageRecruitment cost savingsAttracting and retaining skilled workforceCompetitivenessInsurance premiums/healthcare costs

Work contentWork demandsJob control/decision latitudeEffort requiredRewardsRole Working schedulesOpportunity for learning/developmentMonotonySkills utilisationWorthwhile/fulfillingStability/job futureOccupational group

Intervention points OutcomesWider influences

Where could logic modelling get you?

• May act as a common frame of reference, defining poorly developed concepts

• Used as a participatory and communication tool,

user involvement?


31

Provides a systems perspective

• Increasing recognition of systems perspective in healthcare practice eg pt safety

• Recognition of complexity eg MRC Developing and evaluating complex interventions


32

Implementation Research

eg Eccles et al. 2009, Cooksey Report

Transfer of research findings into practice

Complexity of context

Fidelity of interventions


33

• Used as a research development tool, consider issues that a research agenda needs to address

• Takes an inclusive approach to evidence, areas where little trial work

• Pilot work, Mapping review?


34

Links between advice services for patients, and health outcomes


35

Financial outcomes

Health outcomes

Advice

Where does it not get you?• Effectiveness

• Issues with identification of material – selection bias

• Issues with hierarchy of evidence – representation of strength?

• Hierarchy of qual designs? Frequency? Shows where research has been done rather than evidence?


36

More issues• Critical appraisal especially qualitative

sources

• Issues with integration of diff types of data

• Removal of context

• Use of primary material in secondary synthesis

• Integration versus interpretation


37

Logic is the

beginning of wisdom,

not the end.

Leonard Nimoy

Logic will get you from A to

B.

Imagination will take you

everywhere.

Albert Einstein


38


39

39

Promise improvement,Exact retribution

Check with Clients

Logical Model for Creating Achievable and Sustainable Change Modalities over Time

Put paper in printer

Engage 1st gear

Realize that you look important if it’s really really

confusing.

Sta

rt t

hin

kin

g a

bout

creati

ng a

really

com

fusi

ng c

hart

.

Realize that even you don’t understand any

of this.

Roll down hill

Capture Vision

Engage bullshit generator

ReportOutcomes

WaxEloquent

The world is a

better place

because they

actually paid you

to do this.

Analyze everything you can think of in

depth

Deta

iled

cau

se a

nd

eff

ect

follo

wsBuild

conceptual engine

Engage graphics generator

Realize that you look important if you’re the only

one who understands it.

InternalizeConceptualize

Target audience Create message Engage audienceRealize/Engage Actualize/Energize Promote/Enhance Expose/Repose

Hirechimp Let Vision

Go

Re-captureVision

Jump start engine

AchieveGoals

FinalizeResults

WaxCar

Firechimp

Put orange box here.

Keep close watch on vision.

Unknown author

References

Baxter R, Killoran, A, Kelly MP, Goyder E. Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. Public Health, 2010; 124: 99-106.

Barbour R, Barbour M. Evaluating and synthesising qualitative research: the need to develop a distinctive approach. J Eval Clin Prac 2003;9:179–86.

Bunton R, Macdonald G, editors. Health promotion: disciplines, diversity and developments. 2nd edn. London: Routledge; 2002

Cooksey D, A Review of UK health research funding. Norwich: HMSO; 2006.

Eccles M, Armstrong D, Baker R et al. An Implementation Research Agenda. Implementation Science, 2009: 4: 18 doi: 10.1186/1748-590804018.

Graveling R, Crawford J, Cowie H, Amati C, Vohra S. A review of workplace interventions that promote mental wellbeing in the workplace. Edinburgh: Institute of Occupational Medicine; 2008.

Joly B, Polyak G, Davis M, Brewster J, Tremain B, Raevsky C, et al. Linking accreditation and public health outcomes: a logic model approach. J Public Health Manag Prac 2007;13:349–56.

Kelly M, Stewart E, Morgan A, Killoran A, Fischer A, Threlfall A, Bonnefoy J. A conceptual framework for public health: NICE’s emerging approach. Public Health 2009;123:14–20.

Kellog. Foundation, The logic model development guide. Michigan: Kellog Foundation; 2004.

Lando J, Williams S, Williams B, Sturgis S. A logic model for the integration of mental health into chronic disease prevention and health promotion. Prev Chron Disease Pub Health Res Prac t Policy 2006: 3; 1-4.

Schmitz C. Everything you wanted to know about logic models but were afraid to ask. Michigan: Kellog Foundation; 1999.


40


41

Tannihill A. Beyond evidence to ethics: a decision-making framework for health promotion, public health and health improvement. Health Promot Int 2008;23:380–90.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Meth 2008;8:45.Tones K, Green J. Health promotion: planning and strategies. London: Sage; 2004.

Further reference lists/resources:http://www.cdc.gov/eval/resources.htm#logic model

Documents

Logic models and qualitative synthesis Ideas and issues