Transition

Evaluation Toolkit

1

Evaluation Tools Guide

Strategic Partnership

1. SP1: Partnership Meeting Observation Form Page (16)

2. SP2: Baseline Data Evaluation – Partners Annex (2)

3. SP3: Baseline Data Evaluation – Project Team, WLL Annex (3)

4. SP4: Health as a Cross Cutting Theme Questionnaire Available via WLL

5. SP5: Project Case Study Page (9)

Delivery Model – Trainer/IAG Adviser

1. SP2: Baseline Data Evaluation Annex (2)

2. DM1 Trainer/IAG Adviser Reflective Log Page (10)

3. DM2: Trainer/IAG Adviser Case Study Page (14)

4. DM3: Peer Observations of One to One meeting with Learner Page (17)

5. DM4: Observation of One to One meeting with Learner Page (18)

6. DM5: Quality Monitoring Visit Record (Partners) Page (19)

7. E1: Employer Survey

Participants

1. P1: Participant IAG & Session Evaluation Page (11)

2. P2: Participant Final Reflective Log Page (12)

3. P3: Participant Case Study Page (13)

Photographs

PC1: Consent Form and Guidance Page (31)

Annex (1)

2

Why evaluate the Transition project Annex (1)

3

Evaluation Tool Person(s) Responsible Frequency Deadline for Submission

SP1: Partnership Observation Form

Rinova, Julie Parish or representative

Every meeting Within 3 days of every meeting

SP2: Baseline Data Form Evaluation: Partners

All partners Once By 18th July 2012

SP3: Baseline Data Evaluation: WLL

Wandsworth Lifelong Learning (WLL), project management team

Once By 18th July 2012

SP4: Health and Wellbeing Questionnaire

Tim Brunt Once By August 2012

SP5: Project Case Study Wandsworth Lifelong Learning

3 over the lifetime of the project

1. November, 20122. September, 20133. December, 2014

DM1: Trainer/IAG Adviser Reflective Log

All partners Each partner to produce two, annually

By end of Nov 2012By end of April 2013By end of August 2014

DM2: Trainer/IAG Adviser Case Study

All partners Each partner to produce one, annually

By end of April 2013By end of August 2014

DM3: Peer Observation of One to One meeting with Learner

All partners One to be produced for each partner, annually

By end of April 2013By end of April 2014

DM4: Observation of One to One with Learner

WLLAll partners

One, for each partner, annually

By end of March 2013By end of March 2014

DM5: Quality Monitoring Visit

WLLEach partner to be visited at least once, annually

By March 2013By March 2014

E1: Employer Survey(Optional)

All partners, as appropriate

Once each year By end of April 2013By end of July 2014

P1: Participant IAG & Learning Evaluation

All partners On completion of IAG support or learning intervention

Reported Quarterly, by given reporting deadline

P2: Participant Reflective Log

All partners On exit from the project Reported Quarterly, by given reporting deadline

P3: Client Case Study All Partners 1 produced Quarterly, min. two from each partner

Reported Quarterly, by given reporting deadline

4

How to use the Toolkit

The Toolkit has been designed for use by all of the Transition partners. It has been based on examples of surveys and questionnaires used from within the partnership, and from elsewhere.

We have adopted a colour guide to help you navigate the Toolkit and to select the right guidance and tool for the job.

Purple Guides: provides an overview of how to prepare for and write up a Case Study, and guides for the production of Reflective Logs.

Green Tools: are the templates that all project partners should use to capture feedback and effective practice.

Orange Observations: are the templates that will be used by peers and managers to capture effective practice.

Important Note: do customize these tools to fit with the project by using the Wandsworth, Skills Funding Agency and the ESF logos, and your own logo of course, if there is space.

You will also need to customize the surveys/questionnaires with the title of the service, activity or support that you have provided, but please do not change any of the questions. We will review questions used through our partnership meetings.

Thank-you!

5

Purple GuidesUsing the Guides and Tools - creatively: please use the following guides to help you decide how you will use the Toolkit, for example, you might choose not to use for the forms with participants, but use the questions asked to capture their feedback on video or audio. You might want to ask a group to respond to the questions, rather than an individual, so you could create a ‘talking wall’ or a more creative level, a ‘mind map’ – to explore thoughts and opinions. Basically, you can be as creative as you like, you just need to use the questions provided.

You might choose to provide a Feedback Box – located in reception, for example, where the questionnaires are placed, or just simply provide envelopes to place them in – making the process of providing feedback confidential.

If you are feeling brave, you might want to introduce evaluation during inductions, exploring with tutors/IAG advisers and participants exactly what is going to be evaluated, why, and what else they would like to evaluate.

Working with the group, you might want to capture a group or individual ‘historic timeline’ to capture impact at key stages of the project.

6

Transition ProjectSuggested Structure for Case Study Stories for Trainer/IAG Adviser

Essentially, for Trainer/IAG Adviser the case studies need to demonstrate their awareness of transferable skills, personal development and employability skills, how they acquired their own and how they weave such skills into their own teaching, and examples of successful outcomes with some of their participants. Successful outcomes for them might include:

Planning transferable skills, personal development and employability skills into their teaching sessions

Helping participants move on to more education/training Helping participants move into apprenticeships and/or sustainable employment

At least two of the delivery team should produce a Case Study Story for each delivery year of the TRANSITION project. The Case Study does not necessarily be written, it can be captured on video or audio, but these will need to be high quality, and should follow the proposed format.

Transition ProjectSuggested Structure for Case Study Stories for Participants

For participants the case studies need to demonstrate their awareness of transferable skills, personal development and employability skills and why these are useful, which ones they have acquired and how, how they have been able to evidence them, and examples of successful outcomes: e.g. where they have been useful in helping them get work, or move on to more training and education, improving self confidence, and the like. Successful outcomes for them might include:

Improvement in softer life skills Moving into more education/training Getting a job or an apprenticeship Understanding the value of various types of learning

At least two Case Study Stories should be produced for each delivery year of the TRANSITION project at key transition points within the delivery calendar.

Transition ProjectWriting up the Case Study Story

Suggestion for pitch, tone and styleIn writing up the case study we suggest that it should be:

Written in simple, plain English Have a strong narrative thread to the story – particularly around transferable skills, personal

development, and employability skills and successful examples thereof Written in a style familiar to the target audience Supported with a photograph, where possible

Structure

Try and give the case study a good headline.

If it helps to clarify the story, give subsequent paragraphs sub headlines.

7

Like many newspaper and magazine articles try and get the flavour of the key story element in the first paragraph and then expand in subsequent paragraphs.

We would suggest a word count of somewhere between 300 and 500 words. As with Trainer/IAG Case Studies, the Participant Case Studies can be captured on video or audio, and should be high quality, and follow the same format.

Transition ProjectReflective Practice

Trainer/Adviser Reflective Log for Delivery of Sessions with participants* and Final Reflective Log

Reflective practice in the Lifelong Learning sector is now widely recognized as an integral element of continuing professional development. It also provides essential information for evaluators – to capture and promote demonstrable evidence of change and adaptation, also improvements in practice that can be shared with others. In a professional setting, reflection is:

Deliberate Purposeful Structured About liking theory and practice To do with learning About change and development

It is recommended that the delivery team complete at least one reflective log for each cohort of participants, or a minimum of twice each year, for the duration of the TRANSITION project.

Trainers/IAG Advisers are encouraged to use the Tools provided in this Toolkit. At least two reflective logs should be produced each year of TRANSITION project delivery, and each member of the delivery team will produce one final reflective log, each delivery year. Reflective logs can be produced in video or audio formats, be high quality, and follow the proposed format.

*Sessions can be either group or individual sessions e.g. one to one

Transition ProjectReflective and Inclusive Practice

Participant Final Reflective Workshop

Most of us find it quite a challenge to think about our own skills. It is hard enough to think about what areas we need to focus on developing, but it is even more difficult to acknowledge what we are really good at. The key features of effective reflection are identifying, analysing and exploring what we have done, understanding what we have done well and seeing how we can be more effective.

The Final Reflective Workshop has been designed to support the capture of wide-ranging aspects of the journey experienced by the young person participating in the Transition Project, along with Trainers and Advisers.

In an ideal world, each participant leaving the TRANSITION project should engage in a final reflective workshop, however, this is not always appropriate or possible. Therefore, we suggest that these workshops are embedded into delivery. It is advised that at least 30 per cent of those leaving TRANSITION will take part in a workshop. The Rinova team will facilitate at least 4 workshops during the lifetime of the project.

8

Green Tools

9

Transition ProjectSP5 Project Case Study

Name of the project Enter the name of the project and any “branding”

Address of the project Enter the address of the project, and list the partners and their address, if appropriate

Name of contact Enter name(s) of people within your organisation who will be responsible for handling any external communications for the project

Contact details Please provide telephone contact details, along with time(s) of availability, email address, and website address

Project aims and objectives

Use about 50 words to do this

Funding details E.G. this project receives £xxx,xxx from the London Councils European Social Fund Co-financing programme. Funding is provided over 2 years and payments are made on the results achieved

Main beneficiaries Describe, in brief, the main beneficiaries

What do you hope to achieve for the local community

Describe, in brief, what you hope to achieve for the community – is the project tackling a particular issue or addressing a problem, or does more positively, what solutions does it bring? Quotes help bring the narrative to life.

Outcomes Please give a description of the specific outcomes for the project. What impact has been achieved, based on outcomes or experience from running similar projects? What are your future plans?

Please confirm who needs to approve the draft case study for your organisation:Name:Telephone:Email address:Signature:

Case study approval / confirmation for internal/external use:Name:Position:Email:Signature:

10

Transition ProjectDM1 Trainer/IAG Adviser Reflective Log

Session and Venue Enter Details here

Brief Description of what was delivered

Enter Details here

Number and type of Participant

If a 1-2-1 meeting, give example of the stage of support e.g. 2nd session out of 5, with a young person who has recently left school with no qualifications.

If a workshop or class based session, enter size of group, the stage of the programme they are at, if relevant, their general level, and a profile of the group.

Plans and Objectives If a 1-2-1 give details of your plan(s) for the session and what the key objectives/goals of the meeting were.If a workshop/session give outlines of the Lesson Plan and the Learning Objectives, noting if any of these were changed to meet with the needs of the individual or group.

Positive points of the session

Reflect on what went really well and why you thought it did, taking into consideration the group, etc.

Negative points of the session

Reflect on what did not go so well and why you thought it did not, taking into consideration the group, etc. Give a description of what happened. How can you modify your practice?

How did you include Transferable skills and was this successful?

Give examples and evaluate success of approach and outcome for participant

How did you include personal development/Soft Skills and was this successful?

Give examples and evaluate success of approach and outcome for participant

How did you include Employability Skills and was this successful?

Give examples and evaluate success of approach and outcome for participant

Outcomes Achieved Were all learning objectives achieved and if not, what were the reasons? Were there additional objectives set, and were these achieved? If not, what were the reasons?

Conclusions: Key areas of development and/or improvement based on your own and participant evaluations

On reflection, what would you change based on your own analysis of the session and the feedback from participants?

11

Transition ProjectP1 Participant IAG or Session Evaluation

Your Name:Date:Name of the College/Organisation:

Course/Project Title: IAG Adviser/Trainer:

Pease mark your answer with an X

Did you find the Tutor/Trainer well prepared and easy to understand?

Do you think the Tutor/Adviser treated you/others fairly and with respect?

Has the session helped review and set your goals?

Has the session helped you set realistic plans for the future?

Has the session has helped improve your self-confidence?

Has the session helped you feel more positive about the opportunities that you have in life?

Yes

Yes

Yes

Yes

Yes

Yes

No

No

No

No

No

No

Would you recommend the session to someone you know? Yes No

What did you find most useful?

12

What did you find least useful?

Signature:

Thank you very much for completing this form!

13

Transition ProjectP2 Participant Final Reflective Log

Your Name:Date: Name of the College/Organisation:

Please place an X in the box that most closely reflects your opinion.

As a result of your tailored programme of support:

Q1 Have your chances of becoming employed been improved? Yes No

Q2 Has the project helped you get onto another course? Yes No

Q3 Have you made progress in securing an apprenticeship? Yes No

Q5 Do you have a clear idea of what you are going to do next? Yes No

Q6 Do you feel motivated? Yes No

Now, please answer the following 3 Quick Questions below:

1. What was the best part of the project and why?

2. How could we improve the project?

3. Finally, how did the information, advice and guidance that you received help you?

14

Thank you very much for completing this form!

15

Transition ProjectP3 Participant Case Study

Participant Name: (can be anonymous)

Name of the College/Organisation:

The Story Of Enter real name or pseudonym if required to protect privacy herePlease provide a photograph in the project setting

Enter photo here … ensuring that permission has been received from the participant and any others featured

Trainer/Adviser and Project Partner

Enter name(s) / details here

Background and experience

Ask about the relevant background information to provide a short pen picture: e.g. experience of school, previous difficulties with formal learning situations, employment experience (or absence of it), an social issues they are willing to share (previous anti-social behaviour, homelessness, etc)

Their qualifications What qualifications and accreditations to they have (both formal and non-formal learning)? If none, or very low level you could ask they if they think that has been a problem or constitutes an obstacle to progress?

Their career and work aspirations

What particular job type or industry they are interested in, and why, and what course(s) they are on at the moment.

Their employment/education portfolio

Have they worked before, and if so what has their experience been. If they are in work, what is their experience of this employment and any in-work support/training?

Description of their current learning environment/situation

Ask what they like and/or dislike about the IAG, learning and the environment offered through this project.

Transferable Skills Do they know what transferable skills are? Do they know why employers are interested in them? If they do know, what transferable skills have they gained over the lifetime of the programme?

Personal Development/Soft Skills

Do they know what soft skills are? Do they know why employers are interested in them? If they do know, what soft skills have they gained over the lifetime of the programme?

Employability Skills Do they know what employability skills are? Do they know why employers are interested in them? If they do know, what employability skills have they gained over the lifetime of the programme?

The outcomes Has the programme helped them in any of the following areas:- improved soft/personal skills: e.g. self confidence, working with others,

etc- getting work or securing an apprenticeship- getting into formal education or training

Their ambitions What is it that they want or would like to do in the future?Where do they see themselves in 5 years time?How do they see the current learning they are doing helping them to achieve what they want?

16

17

Transition ProjectDM2 Trainer/IAG Adviser Case StudyTrainer/Adviser Name: (can be anonymous)Name of the College/Organisation:

The Story Of Enter real name or pseudonym if required to protect privacy here

Please provide a photograph in the project setting

Enter photo here … ensuring that permission has been received from the participant and any others featured

Trainer/Adviser and Project Partner

Enter name(s) / details here

Their specialist disciple: background and experience

Your background and experience in the specific / specialist area that you deliver on the programme, and any personal experiences of the education system that are interesting and relevant to the project and the participants.

Their employment portfolio

Have you always worked in the education/learning sector? If not, what other types of employment have influenced their work choice / teaching practice: e.g. you now teach a vocational/technical job they held in the past e.g. bricklayer or IT specialist?

Description of the learning environment and situation

Please describe the IAG and/or learning situations/environments you have experience of in the past and how it compares with the programme you are now involved with.

What about your experience of working with young people who are NEET? How has this influenced the way you approach your training/IAG provision?

When comparing and contrasting different IAG learning environments experienced throughout the project, what are the strengths and weaknesses, and how have these impacted on the training/advice experience.

Delivering transferable skills

Do you actively work transferable skills activities into your training/IAG?Can you provide examples?

Delivering soft skills Do you actively work soft skills activities into your training/IAG provision?Can you provide examples?

Delivery of health and wellbeing

Do you actively work health and wellbeing activities into your training/IAG provision?Can you provide examples?

Delivering employability skills

Do you actively work employability skills activities into their training/IAG provision?Can you provide examples?

Demonstrable impact

Can you please provide an example of where a transferable/soft/employability skill delivered during your work has really benefitted a particular participant?

18

Orange Observations

19

Transition ProjectSP1 Partnership Meeting Observation

Name:Date:

How is the partnership working strategically to achieve outcomes and sustainability?How is the partnership maximizing their networks?

How is the delivery model adapting to reflect identified changes?

How is the delivery partnership working to address issues and take collective corrective actions?How is the partnership reflecting and acting upon knowledge of the ‘bigger picture’ and external influences?

20

Transition ProjectDM3 Peer Observation of One to One Meeting with Learner

Adviser Name:Adviser’s Organisation:Observer Name:Observer’s Organisation:Date:

In terms of what you observed, what worked?

What might you use in your own practice?

21

Transition ProjectDM4 Observation of One to One Meeting with Learner

Partner Name:

Adviser Name:

WLL Observer Name:

Date:

Length of Observation:

Notes: highlight good practice

Suggestions for improvement

22

Transition ProjectDM5 Quality Monitoring Visit: Record Form

Partner Organisation:

Partner Organisation Representative Name:

WLL Manager’s Name:

Date:

Purpose of Visit: Documentation Check Premises Check Learner Meeting Delivery Discussion/Guidance

Discussion Topics and Notes

Actions Agreed

Learner Meeting NotesDiscussion Topics and Notes

Actions Agreed

23

Employer Survey (Optional)

Transition ProjectE1 Employer Survey

Name:Date:Name of organisation:

Please circle the score that most closely reflects your opinion.Agree Disagree

Communications with the organisation arranging visits to the place of work were excellent.

4 3 2 1

The young person benefitted immensely from the in-work support that they received.

4 3 2 1

We as employers benefitted immensely from the in-work guidance given which helped the young person settle well and be productive.

4 3 2 1

The number of in-work support sessions given hindered further development opportunities in the workplace.

4 3 2 1

Support provided by the organisation from recruitment, interview and induction helped us choose the right young person for the job.

4 3 2 1

We took part in the programme as we are an important part of the community and should offer opportunities to young people.

4 3 2 1

We want to engage and work with fresh young minds.

4 3 2 1

We want to help young people gain a better insight into and knowledge of our industry/type of business.

4 3 2 1

We took part in the programme to motivate existing employees and help with

4 3 2 1

24

their personal development.The in-work support provided to the young person helped with retention and progression in the workplace.

4 3 2 1

We took part as we want to contribute to the development of the future workforce.

4 3 2 1

Thank you for answering these questions your answers help us to improve!

25

Annex (2)

Transition Evaluation Baseline Data Form

Organisation Name:

Person completing the form:

Partner organisationbullet points are enough but feel free to write more

What is the context you are working in?In relation to:

Geographical focus

What area / sector is the

main focus of your

delivery?

What do you think are the

key issues facing the

young people you work

with that affect their

ability to engage with

EET?

Are there other

programmes locally that

are offering similar

support for young

people?

Target Groups:Can you tell us about the young people you work with (other than being aged 16-19)?For example:

Are they from particular

ethnic groups?

Are they migrants or

refugees?

Do they have disabilities?

Other?

What do you think are their specific support needs in relation to this project?Why did your organisation choose to be involved?How does this programme fit with your existing aims, strategies and/or projects?Are you currently working with any other partners?Who are you working with and why?

Promotion and Outreach: How are you reaching the

27

target groups?

How will your delivery

meet the particular

learning & support needs

of the target groups?

How have these been

identified?

What do you think are the main challenges for your organisation in delivering this project?During the initial months of the project have you needed to change your Schemes of Work or Delivery Plan?For example have you needed to shorten or lengthen your delivery, or change the focus of your delivery to ensure participants are recruited, engaged and/or retained?How are you addressing the ESF Cross Cutting Themes within your Delivery?For example are you delivering health related workshops, or capturing improved health through participant self-evaluations?What do you hope to get from the partnership? What do you bring to

28

the partnership?Can you tell us a bit about your experience of the setting up stage (pre-contracting stage) of the project?For example was it straightforward / complex / difficult / new to you. Did partners give you support and was this useful? Is there anything that could have improved this stage for you?

Do you think that the way in which the setting up stage was managed was positive for the project partnership and subsequent stages? If yes, in what ways?

29

Annex (3)

Transition Evaluation

Project ManagementBaseline Data Form

Organisation Name:

Person completing the form:

Partner organisationbullet points are enough but feel free to write more

What is the context you are working in?In relation to:

The Project Management

team, staffing numbers,

expertise and line management

arrangements

What do you think are the key

issues facing the young people

you work with that affect their

ability to engage with project

management requirements?

In your opinion, what was the

30

end result of producing a

Project Handbook and

delivering a Project

Management induction

workshop for partners?

Have any template forms or

documents required a review

during the set up phase? If yes,

please give details.

As a project lead, did you provide template forms to capture aspects of evaluation e.g. participant satisfaction, Trainer observations, and the like, or are partners using their own?

Partners:Can you tell us about the partners that you are working with in terms of the following:

Ability to respond to given

project reporting deadlines?

Evidence of delivery against

profile to date and

performance?

Revision of schemes of work to

31

address performance (+) or (-)?

Other?

What do you think are their specific support needs in relation to this project?Project Delivery AdditionalityWhat evidence do you have to prove that delivery is additional to partners’ core offer and does not either i) double count or ii) duplicate what is being delivered as their core offer?Partnership / Steering Group MeetingsHow many meetings have you had to date? What is your overall impression from these meetings e.g. could they be run differently or perhaps encourage more contributions to the partners?Promotion and Outreach:

What aspects of the project’s

outreach and promotions

activities would you describe as

innovative or imaginative?

So far, has there been a need

to review and/or adopt a

different Promotions/Outreach

strategy for the project? If so,

please give details.

Do you have a written

32

Promotions/Outreach Strategy

for the project? If so, please

attach. If no, please explain

why not.

What do you think are the main Project Management and/or Partnership Management challenges for your organisation in delivering this project?During the initial months of the project have you needed to change your Project Management approach, or delivery profile?For example have you needed to spend more or less time with partners than anticipated? Have you needed to re-profile the project? If so, please give details.How are you addressing the ESF Cross Cutting Themes with partners?For example, have you created an Equalities and Diversity/Equal Opportunities Statement/Policy/Strategy for the project? Have you done similar for the Health Cross Cutting Theme? Are you collecting evidence of effective practice, especially in terms of the Health Cross Cutting theme? And finally, what about a Sustainability Strategy?

Can you tell us a bit about your quality

33

assurance processes and protocols and how these have helped set up the project?For example, is there requirement for an Equalities Impact Assessment? Is there a Risk Analysis for the project, and if so, how are these risks communicated to the team involved in managing/delivering the Transition project?

Do you have a Safeguarding Policy for the project? If not for the whole project, have you sought copies of Safeguarding Policies from all partners?

What about external inspection reports, SARs or QIPs from partners – are you seeking to gather any of these?

What aspects of your project management would you describe as ‘effective practice’?

How will you monitor and/or measure the quality of provision across the partnership?What measures are currently in place to monitor and measure? How can the Quality and Evaluation process assist you with this requirement?

34

Annex (1)

Using Photographic Images with Case Studies

Photographs may be an excellent accompaniment to your case study, increasing its visual appeal and making it stand out and attract attention. If you would like to arrange photography for your case study, here are some guidelines to help you get the right image.

1. Do not rely on newspapers to send a photographer. It may be best to book your own.2. Always remember to ask for a quote in advance to ensure that you are getting good value for money.

Also, ask whether extras such as, mileage and a CD with the photographs on are included.3. You should request that the photographer take digital images and brief them on your exact

requirements in advance of the day.4. Make the pictures as dynamic as possible. Shots in action are always better than static posed shots.5. Get people into the pictures. Journalists like lively people images showing them doing something

interesting. Three is the idea number of people, five is the maximum.6. Get permission. Under the Data Protection Act you must gain permission from the people in the

photo to use the image in the public domain. Therefore, you must use a consent form.7. If you send your photo to a newspaper, make sure it is fully captioned – papers need all names and

titles

PC1: Photography Consent Form

We are committed to the protection of individual’s privacy, and under the Data Protection Act 1998 require your consent to use your image(s).

Please read the terms and conditions of use printed below, before completing it. It is intended that the photos taken today will be used as part of publicity for (insert the name of the project, the providers name, etc).

The photos may be used on the (insert the organisation’s name) websites, printed publications, electronic presentations, and submission to newsletters, newspapers or exhibition packs.

Terms and Conditions

1. (insert organisation/providers name) … cannot accept any liability for any loss, damage or personal injury, howsoever caused, through use of the images through third parties.

2. Websites can be seen globally and no warranty can be given that adequate data protection legislation will apply.

3. Your full name will be supplied with photographs where necessary. However, we will not include any personal e-mail or personal address, telephone or fax numbers in association with the images.

4. Your image(s) will be retained indefinitely on our files. You may withdraw your consent at any time in writing to (enter details of who should be contacted, organisation/provider name, address). You must however, acknowledge that images that have previously been used with your consent may still be in circulation and not withdrawn.

If you agree to the above usage then please sing the boxes below.

Signed:

Date:

Name: IN BLOCK CAPITALS

Important! If the person being photographed is under 18 years of age, a parent or legal guardian must also sign:

Signed:

Date:

Name: IN BLOCK CAPITALS

36

Annex (2)

Why Quality and Evaluation?

This Evaluation Toolkit has been designed especially for partners working collaboratively on the Transition project, which is ESF/SFA funded for the period 1st November 2011 through to 31st March 2015. It is designed to cover all of the key stages of the project, provide demonstrable evidence of effective practice, and blend well with the requirements for partners to meet Matrix, Ofsted and other quality assurance requirements.

1.1 Why have a Toolkit on Monitoring and Evaluation?

Monitoring and evaluation enable you to assess the quality and impact of your work, against your action plans and your strategic plan. The information resulting from monitoring and evaluation is essential for those concerned about the efficiency, effectiveness and impact of the project.

Some of the benefits can be: Improve the provision of learning opportunities for young people by being more effective and

efficient in capturing their ‘voice’ – needs, wants, desires, and the like; Improve the provision of individual advice and guidance (IAG) by providing an opportunity for young

people to be involved more actively in planning and reflecting on their individual journeys and distance travelled;

Improve the development and delivery of programmes for young people; and Improve the planning and implementation of evaluation as a means of achieving continuous

improvement.

The Toolkit will be useful when: You are setting up systems for data collection during the set-up phase of your project; You want to analyse data collected through the monitoring process; You are concerned about how efficiently or effectively you are working; You reach a stage in your project when you think it would be useful to evaluate what impact the

project is having on the lives of young people and other key stakeholders connected to the project; You want to capture and demonstrate to others the effective practice that is emerging from your

delivery.

1.2 The Monitoring and Evaluation Cycle

37

Monitoring and Evaluation can: Help you identify problems and their causes Suggest possible solutions to problems Raise questions about assumptions and strategy Push you to reflect on where you are going and how you are getting there Provide you with information and insight Encourage you to act on the information and insight Increase the likelihood that you will make a positive development difference

1.3 The Transition Project: What do we want to know?

In the Framework document, the three inter-connecting levels of (a) Strategic Partnership, (b) Delivery Model and (c) Participant, were specified. Within this context we need to focus on the following:

Efficiency: is the input appropriate for the intended output? Effectiveness: did we change, improve, and/or succeed? Impact: did we make a difference to the situation we are trying to address?

So, some of the questions we will be asking as part of the evaluation include the following:

Who is benefitting from what we do? How much are they benefitting? Are the participants passive recipients or does the process enable them to have some control over

what they do whilst engaged on the project, and elsewhere in their lives? Are there lessons in what we are doing that have a broader impact than just what is happening on

our project? Can we do what we are doing in some way for the long-term, or will the impact of our work cease

when the project comes to an end? Are we getting the optimum outputs for the least possible amount of inputs?

1.4 How we will monitor and evaluate

We will be using a variety of methods across the partnership including the following:

Quantitative approach

Evaluate, Learn, Decide

Plan

ImplementMonitor

Reflect, Learn, Decide

Adjust

38

Monitoring Data: accessing monitoring data produced by the partnership to chart performance and assist the partnership in taking corrective action, if needed.

Qualitative approach Internal-evaluation: this will involve the project partners holding up a mirror to itself and assessing

how it is doing, as a way of learning and improving practice. It takes a very self-reflective and honest organisation to do this effectively, but it can be an important learning experience;

Participatory evaluation: this is about empowerment and will involve staff and project participants working together on evaluation through reflection, discussion and learning from each other.

2. Strategic Partnership

The observations and case studies are being done in order to illuminate and capture effective practice and lessons learned from the partnership. In particular, we are keen to learn and present information for dissemination on how the partnership has worked strategically to achieve project success including collaborating with others to achieve participants progression, while at the same time, capture evidence of wider partnership working, and how sustainability for the most successful aspects of the project can be achieved.

3. Delivery Model

Throughout the lifetime of the project it will be important to capture how the delivery model has changed to meet local dynamics, findings from individual partner experiences captured through evaluation – especially how the delivery model has been changed to meet needs. Initially, the evaluation process will look at the Specification Model as given by the ESF/SFA and how this was interpreted by the partnership, will capture any innovative or unique features, and will capture effective practice – through observations and reviews with partners.

The matrix developed will feature approaches adopted by each partner at all six stages of delivery:

Stage (1) Identify and recruit participantStage (2) Diagnostic needs assessmentStage (3) Design and deliver bespoke interventionStage (4) EET opportunity identified and accessedStage (5) Support participant and host organisation to sustain EET

In the ESF/SFA Specification, it is stated that all projects must incorporate the following: A person-centred action plan drawing on an existing recent assessment of the young

person’s needs by the school, college or by a Connexions personal adviser, and on in-depth discussions with the young person. The action plan should be:

o Provided or supervised by a qualified advisero Developed in partnership and in consultation with the young person and in

consultationo Set out an agreed definition of the young person’s pathway to EETo Set out clear steps towards sustained re-engagement for 26 weeks or two termso Used to monitor the young person’s progress on the programme

Partners will be encouraged to use these action plans to monitor progress or the ‘distance travelled’ by the young person, and as a tool to review the approach of the adviser assigned.

39

4. Participants

What we want to show impact via Case Study Stories

The case studies are being done in order to illuminate and provide some personal stories for the Transition Project which will be provided to Trainers/Trainers/Advisers who work in both formal and non-formal learning situations as part of the ESF/SFA funded programme, being led by Wandsworth Lifelong Learning, Wandsworth Council.

We are interested in case stories from both Trainers/Advisers and participants in each of the areas that the Transition project will focus on. So the ideal case study will be one Trainer/Trainer or Adviser and one participant from each of the following programme stages:

Stage (1) Identify and recruit participantStage (2) Diagnostic needs assessmentStage (3) Design and deliver bespoke interventionStage (4) EET opportunity identified and accessedStage (5) Support participant and host organisation to sustain EET

5. The Importance of Reflective Practice for Trainers and Advisers

For Trainers/Trainers/Advisers we would like to capture and demonstrate how reflective practice informs the continuous improvement of both the programme structure and session content. The session based Reflective Log will assist with us with this. In terms of beneficiaries, we will encourage all partners to provide us with an appropriate sample of Participants’ Action Plans or at the very least a Snap shot of their individual Distance Travelled. When a participant comes to the end of their time on the project, we are recommending that all participants be asked to complete the Participant Final Reflective Log.

We also recommend that partners follow-up participants three, six and twelve months after they have left the project.

Why Reflective Practice?Reflective practice involves thoughtful consideration of an experience, situation, or topic, both positive and negative, which results in an outcome of changed perspective (Spalding 2004).

40

When reflecting we need to consider some of the following:

Values and beliefs: our own and those of others Actions: what we did and the skills we used Concepts: how related literature or reading has informed and supported our learning and

actions Context: how what we have learnt and experienced relates to our learning environment,

home life, social network, and the like Social and emotional skills: it is important to demonstrate how we use social and emotional

skills in our approach to learning, life and others, because these underpin who we are Social skills: how we relate to others Self-awareness: understanding of what we are good at and aspects of our skills that need to

be developed Empathy: understanding other points of view Managing our feelings: recognising our feelings and expressing them appropriately Motivation: how we support ourselves and others

Essentially, for the Trainer/Trainer/Adviser case studies we need to demonstrate the Trainer/trainers awareness of transferable skills, personal development and employability skills, how they acquired their own expertise, how they weave such skills into their own teaching, and examples of successful outcomes with some of their participants. Successful outcomes for them might include:

Planning transferable skills, personal development and employability skills into their teaching sessions

Helping participants move on to more education/training Helping participants move into apprenticeships and/or sustainable employment

Description. What Happened?

Feelings. What were you thinking

and feeling?

Evaluation. What was good and bad

about the experience?

Analysis. What sense can you

make of the situation?

Conclusion, what else could you

have done?

Action Plan. If it arises again what

else would you do?

41

42