Local Exercise Action PilotsLocal Exercise Action PilotsThe National EvaluationThe National Evaluation

General Lessons Learnt So General Lessons Learnt So Far!Far!

27-28 May 2004: 27-28 May 2004: Andy PringleAndy Pringle

leeds metropolitan universityleeds metropolitan university

Who Am I?Who Am I?

Andy PringleAndy Pringle Carnegie Faculty of Sport & Education.Carnegie Faculty of Sport & Education. Leeds Metropolitan University.Leeds Metropolitan University. Senior Lecturer in Active Lifestyles Senior Lecturer in Active Lifestyles

and Health.and Health. Project Manager for the National Project Manager for the National

Evaluation of LEAP: Local Exercise Evaluation of LEAP: Local Exercise Action Pilots.Action Pilots.

OutlineOutline What is LEAP?What is LEAP?

General overview of the National Evaluation General overview of the National Evaluation Framework for LEAP.Framework for LEAP.

General lessonsGeneral lessons that have been learnt in that have been learnt in developing and managing the National developing and managing the National Evaluation Framework.Evaluation Framework.– These are generic and apply to whatever These are generic and apply to whatever

population physical activity interventions are population physical activity interventions are targeted including older people.targeted including older people.

Some suggestions of what we would do Some suggestions of what we would do differently next time around.differently next time around.

What is LEAP?What is LEAP? Local Exercise Action Pilots:Local Exercise Action Pilots: This is a £2.6 million contract awarded by This is a £2.6 million contract awarded by

Department of Health, Sport England and Department of Health, Sport England and The Countryside Agency in 2003 to 10 The Countryside Agency in 2003 to 10 Primary Care Trusts (PCTs/sites) to Primary Care Trusts (PCTs/sites) to deliver and evaluate physical activity deliver and evaluate physical activity interventions. interventions.

There are 10 pilot sites from the North There are 10 pilot sites from the North East to the South West of England using a East to the South West of England using a variety of physical activity interventions variety of physical activity interventions which are being evaluated.which are being evaluated.

LEAP will run for two years.LEAP will run for two years.

The InterventionsThe Interventions The interventions follow 7 themes:The interventions follow 7 themes: 1. Exercise Referral1. Exercise Referral 2. Classes and Groups2. Classes and Groups 3. Peer Mentoring3. Peer Mentoring 4. Campaigns and Directories4. Campaigns and Directories 5. Motivational Interviews5. Motivational Interviews 6. Outdoors and Transport6. Outdoors and Transport 7. Training Leaders and Coordinators7. Training Leaders and Coordinators with a range of physical activity target with a range of physical activity target

groups.groups. Including older peopleIncluding older people..

What is the Role of leeds What is the Role of leeds metropolitan university?metropolitan university?

What is the National What is the National Evaluation Framework?Evaluation Framework? Leeds Met has the contract to evaluate Leeds Met has the contract to evaluate

LEAP working in partnership with other LEAP working in partnership with other organizations and consultants.organizations and consultants.

The basic aim of the National Evaluation The basic aim of the National Evaluation Framework is to establish:Framework is to establish:

Which community physical activity Which community physical activity interventions are effective in getting interventions are effective in getting people physically active?people physically active?

Also looking to find out other information Also looking to find out other information as well.as well.

The National Evaluation The National Evaluation FrameworkFramework

5 LEAP PCTs

Dudley

Great Yarmouth

North Kirklees

Wandsworth

West Cornwall

Comparison Group

LEAP SURVEY

MORI + BHF

Survey Pre (Nov 2003)

and Post (Nov 2005)

LEAP Interventions

NATIONAL EVALUATION

Leeds Metropolitan University + Matrix

MACRO EVALUATION

MESO EVALUATION

MICRO EVALUATION

GENERIC QUESTIONS

1. Physical Activity

2. Cost Effectiveness

3. What works, why and how: providers perspective

4. Participants Perspective

INTERVENTION THEMES

1. Exercise Referral

2. Classes and Groups

3. Peer Mentoring

4. Campaigns and Directories

5. Motivational Interviews

6. Outdoors and Transport

7. Training Leaders and Coordinators

LOCAL EVALUATIONS

CASE STUDIES

EACH INTERVENTION

Run by 10 PCTs

Supported by Leeds Met Team

Three LevelsThree Levels 1.1. MACRO MACRO 2.2. MESO MESO 3.3. MICRO MICRO 1. The 1. The MACROMACRO evaluation comprises two evaluation comprises two

elements: elements: – 1. 1. MORI Survey MORI Survey – 2. 2. Case Study data collection that addresses a set of Case Study data collection that addresses a set of

generic questions common to the evaluation of all generic questions common to the evaluation of all interventions.interventions.

2. The 2. The MESOMESO evaluation comprises seven evaluation comprises seven intervention themes where interventions across intervention themes where interventions across the 10 participating PCTs can be compared using the 10 participating PCTs can be compared using data generated by addressing theme specific data generated by addressing theme specific evaluation questions.evaluation questions.

3. The 3. The MICROMICRO evaluation comprises the 10 evaluation comprises the 10 separate local evaluations incorporating a case separate local evaluations incorporating a case study of each intervention.study of each intervention.

Macro 1. The MORI SurveyMacro 1. The MORI Survey Macro Level 1. An independent MORI Macro Level 1. An independent MORI

survey. survey. Telephone survey of 800 participants Telephone survey of 800 participants

conducted in five of the PCTs before conducted in five of the PCTs before (November and December 2003), (November and December 2003),

and after (November and December and after (November and December 2005), the two years of physical 2005), the two years of physical activity intervention. activity intervention.

The MORI SurveyThe MORI Survey 10 minute telephone questionnaire.10 minute telephone questionnaire. The survey instrument covers The survey instrument covers

questions on:questions on: Physical activity category.Physical activity category. Awareness of local physical activity Awareness of local physical activity

campaigns. campaigns. Knowledge and attitudes to physical Knowledge and attitudes to physical

activity.activity. Social climate, intention to be active. Social climate, intention to be active. Demographics.Demographics.

Macro 2. A Case Study in the 10 Macro 2. A Case Study in the 10 PCTS/SitesPCTS/Sites

The majority of the information for the N.E.F is The majority of the information for the N.E.F is being collected at the PCT level by PCTS and being collected at the PCT level by PCTS and associates.associates.

A common set of questions across all interventions A common set of questions across all interventions where appropriate.where appropriate.

1.1. Baseline and change in physical activity status. Baseline and change in physical activity status.

2.2. Effect of the intervention on participant’s FITT of Effect of the intervention on participant’s FITT of participation.participation.

3.3. Who the participants are and where are they Who the participants are and where are they are from? Contact details & demographics.are from? Contact details & demographics.

4.4. Is the activity Is the activity new or a replacement physicalnew or a replacement physical activity.activity.

The National Evaluation The National Evaluation FrameworkFramework

5.5. Cost analysis of planning, designing and Cost analysis of planning, designing and delivering the LEAP interventions delivering the LEAP interventions

Data collection on Data collection on inputsinputs (e.g.. personnel), (e.g.. personnel), outputsoutputs (e.g.. classes) and (e.g.. classes) and outcomesoutcomes (e.g.. change in physical activity behavior).(e.g.. change in physical activity behavior).

– Working in collaboration with MATRIX CR.Working in collaboration with MATRIX CR.

6.6. Qualitative evaluation of planning, Qualitative evaluation of planning, delivery and evaluation of the interventions: delivery and evaluation of the interventions:

– What works, why and how?What works, why and how?

– Undertaken by the Leeds Met Field Researchers.Undertaken by the Leeds Met Field Researchers.

What are the lessons that What are the lessons that have been learnt….so far?have been learnt….so far?

What action could be taken to What action could be taken to prevent and manage these prevent and manage these

issues in the future? issues in the future?

CollaborationCollaboration It is clear that there are some It is clear that there are some “first rate”“first rate”

examples of collaborative working between examples of collaborative working between the PCTs/Pilot Sites and Leeds Met in the the PCTs/Pilot Sites and Leeds Met in the development of the Evaluation Framework.development of the Evaluation Framework.

However there are examples of less However there are examples of less enthusiasm for the evaluation (enthusiasm for the evaluation (in its in its current contextcurrent context).).

The perception of what evaluation The perception of what evaluation constitutes and how much work is needed:constitutes and how much work is needed:

Resource base available.Resource base available. Skills available for evaluation.Skills available for evaluation.

Specific information Specific information There are examples of where the PCTS There are examples of where the PCTS

have under-estimated/been unclear of have under-estimated/been unclear of the size of the work commitment of the the size of the work commitment of the evaluation component for LEAP.evaluation component for LEAP.

Action:Action: Provision of clear and specific Provision of clear and specific information of what the PCTs information of what the PCTs commitment is at the outset is needed.commitment is at the outset is needed.

SpecificallySpecifically what the magnitude of the what the magnitude of the task is?task is?

Clarity on Roles and Clarity on Roles and

ResponsibilitiesResponsibilities

The The specificspecific types of interventions to be types of interventions to be evaluated.evaluated.

Finalizing the design of the evaluation Finalizing the design of the evaluation framework (to Inc tools, dates of data framework (to Inc tools, dates of data capture.capture.

Leeds Met to be involved in the Leeds Met to be involved in the site selection. site selection.

Resources: Money & TimeResources: Money & Time Resources will always be a contentious Resources will always be a contentious

issue.issue. Health Promotion writers talk of the Health Promotion writers talk of the

need to ensure that evaluation is need to ensure that evaluation is supported with the necessary resources supported with the necessary resources (Green & Tones, 2004, Naidoo & (Green & Tones, 2004, Naidoo & Wills, 2000).Wills, 2000).

It is important that evaluation It is important that evaluation frameworks are well supported with frameworks are well supported with resources so that the the best possible resources so that the the best possible job can be done. job can be done.

Other IssuesOther Issues Choice of tools: Choice of tools:

– Particularly for older measuring physical Particularly for older measuring physical activity levels in people.activity levels in people.

IPAQ: Recall of information.IPAQ: Recall of information.

– Where English is not spoken.Where English is not spoken.– Objective measures of physical activity:Objective measures of physical activity:

Anxiety of some of the sites.Anxiety of some of the sites.

Local Ethical Clearance.Local Ethical Clearance.

Tensions between the job of evaluation Tensions between the job of evaluation and delivery of physical activity and delivery of physical activity interventions: Clear Information. interventions: Clear Information.

For further information:For further information: Andy PringleAndy Pringle Senior Lecturer: Active Lifestyles & Senior Lecturer: Active Lifestyles &

HealthHealth Carnegie Faculty of Sport & EducationCarnegie Faculty of Sport & Education Leeds Metropolitan UniversityLeeds Metropolitan University Fairfax 212, Beckett Park CampusFairfax 212, Beckett Park Campus LEEDS LS6 3QS.LEEDS LS6 3QS. A.Pringle@LEEDSMET.AC.UKA.Pringle@LEEDSMET.AC.UK 0113 283 2600 Ext.7409.0113 283 2600 Ext.7409. WWW.LEEDSMET.AC.UKWWW.LEEDSMET.AC.UK

leeds metropolitan universityAndy Pringle

Senior Lecturer: Active Lifestyles & Health