Using action learning to develop leaders in the justice sector

Evaluating the effectiveness of the action learning pilot programme in Wales

Simon LeckieMarch 2007

Evaluating the effectiveness of the action learning programme in Wales 1

Contents

Executive summary..........................................................................................2

1. Background and overview.........................................................................31.1 A new approach......................................................................................31.2 Action learning........................................................................................41.3 Aim of the justice sector action learning pilot..........................................51.4 Objectives of a justice sector action learning programme.......................5

2. Research objectives...................................................................................62.1 Organisations involved............................................................................62.2 Why this research is important................................................................62.3 Purpose of research................................................................................7

3. Methodology...............................................................................................73.1 Primary data...........................................................................................73.2 Secondary data.......................................................................................83.3 Constraints..............................................................................................8

4. Analysis.......................................................................................................84.1 Kirkpatrick model....................................................................................84.2 Reaction of set members........................................................................94.3 Learning..................................................................................................94.4 Behaviour................................................................................................94.4 Results....................................................................................................9

5. Structure of final report.............................................................................95.1 Executive summary...............................................................................105.2 Introduction...........................................................................................105.3 Section 1: Action learning theory..........................................................105.4 Section 2: Evaluation methodology.......................................................105.5 Section 3: Issues addressed by the pilot..............................................105.6 Section 4: Results achieved..................................................................105.7 Conclusions..........................................................................................105.8 Recommendations for future action learning programmes...................115.9 Reflections............................................................................................115.10 Appendices.........................................................................................115.11 References.........................................................................................11

6. Time schedule..........................................................................................116.1 Collecting data......................................................................................116.2 Analysing data......................................................................................116.3 Writing the report..................................................................................126.4 Constraints............................................................................................126.5 Time schedule......................................................................................12

Bibliography.................................................................................................14

Evaluating the effectiveness of the action learning programme in Wales 1

Executive summary

The first section provides an overview of the action learning pilot that is taking place in Wales. The characteristics of action learning and the objectives for the pilot are also introduced here.

The second section sets out the research objectives and the issues that the evaluation of the pilot project will address. The question to be answered is: Can action learning be applied within the justice sector to enhance leadership skills and improve the performance of organisations across the justice system?

The next section explains the methodology that will be followed. Research will be based mainly on studying primary data gathered directly from the pilot group members. Some secondary data will be reviewed and this will be obtained from existing operational performance management systems within the organisations involved in the pilot.

The fourth section describes how the data will be evaluated through the use of the Kirkpatrick model. The impact of the project will be measured against:

reaction of student - what they thought and felt about the training learning - the resulting increase in knowledge or capability behaviour - extent of behaviour and capability improvement and

implementation/application results - the effects on the business or environment resulting from the

trainee's performance

Depending on the results the overall effectiveness of action learning within a justice sector context will be determined here.

The next section outlines the likely structure of the project and outlines the contents of the final report. The final section of the proposal explains the key activities required to undertake the evaluation of the pilot project and provides a detailed schedule in the form of a Gantt chart.

Evaluating the effectiveness of the action learning programme in Wales 2

1. Background and overview

1Skills for Justice is managing the delivery of an action learning pilot project in Wales. This pilot is being undertaken for employers in the Justice sector to demonstrate the potential of action learning for use in management and leadership development.

This pilot builds on the action learning project undertaken by the Construction Industry Training Board (CITB) on behalf of the Sector Skills Development Agency (SSDA) Management and Leadership Board, which developed a 20 point model for effective action learning to address the key leadership challenges facing Small to Medium sized Enterprises (SMEs) owner/managers. The key challenges were identified as:

Leading change Implementing change initiatives Communicating change to staff

These challenges mirror the key leadership challenges identified by justice sector employers.

This pilot also builds on the mapping of management and leadership current provision carried out by Skills for Justice as part of the development of the 2Sector Skills Agreement (SSA). The key findings of the mapping were:

Broad range of management and leadership development already taking place but management and leadership still seen as a key area for development

Considerable investment but limited evaluation of impact on performance at either an individual or organisational level

Little cross-organisational development activities taking place despite a clear willingness for this to happen

1.1 A new approachThere is a clear need to consider new approaches to management and leadership development within the sector. There is also a need, and a willingness, to open up learning across organisational boundaries and this pilot aims to demonstrate how action learning can be used to support multi-agency learning and address some of the challenges facing the Justice sector.

1 Skills for Justice is the Sector Skills Council (SSC) for the justice sector. SSCs are independent, UK-wide, employer-led organisations developed to tackle the skills and productivity needs of their sectors. A network of 25 SSCs allow employers to provide leadership for strategic action to meet their sector’s skills needs.2 Sector Skills Agreements are compacts between employers, learning providers and funders of learning. Facilitated by Sector Skills Councils, they are designed to deliver action to meet priority skills needs that will drive improvement and performance.

Evaluating the effectiveness of the action learning programme in Wales 3

This pilot also aims to show how action learning could be used to build on the management and leadership development programmes currently in use by providing an opportunity for experienced managers to apply their learning and share experiences to resolve real problems and improve their organisation’s performance.

1.2 Action learningThere is some confusion about what action learning actually is. The phrase ‘action learning’ has become one of the current management ‘buzzwords’ and is used (often incorrectly) to describe a wide range of management development activities. The following definitions, taken from the International Foundation for Action Learning http://www.ifal.org.uk, provide clarity and help to explain what action learning is:

A powerful form of problem solving combined with intentional learning in order to bring about change in individuals and the organisation.

Essential elements of action learning

-tackling real tasks in the real world and the real role

-learning with and through each other

-taking individual responsibility and actually implementing solutions and plans.

At the heart of the process is the 'action learning set'. This is a group of individuals who meet at regular intervals for each member to explore a challenging open-ended problem or opportunity. Every member in turn works on their 'task' and the others as ' friends' provide support and challenge. The aim is to help each member both to tackle the task and to learn from this.

A basic premise of action learning is: 'there is no learning without action and no (sober and deliberate) action without learning' (Reg Revans)

By using the knowledge and experience of a small group of people combined with skilled questioning, individuals are enabled to re-interpret old and familiar concepts and produce fresh ideas - often without needing new knowledge.

From these definitions the key features of action learning include learning through sharing the knowledge and experience of a small group of individuals to address real world problems, often supported by the use of formal training inputs. A current feature of action learning is that the process is usually supported by a facilitator who guides and supports the action learning set.

These definitions have been used to provide a framework of reference during the development of the pilot.

Evaluating the effectiveness of the action learning programme in Wales 4

Management and leadership skills development within the justice sector is largely based on formal programmes, delivered through in-house training provision. There is very limited use of experiential learning and very few examples of any multi-agency or external programmes. Action learning is very different to the majority of existing programmes currently in use, as can be seen by table 1:

Action learning Existing approachesAction on real tasks or problems at work Learning not directly linked to real

tasks or problems and difficult to establish link between programme and individual’s jobs/role

Learning is from reflections on action taken

Learning is from trainer/course material

Tasks/problems are individual rather than collective

Typically designed to deliver collective learning experience, usually on a generic management and leadership theme

Tasks/problems are chosen independently by individuals

Aims and objectives set for the group

Questioning as the main way to help participants proceed with their problems

Telling as the main way to help participants proceed with their problems

Facilitators are used Trainers are usedTable 1 - Comparing the features of action learning3 with existing approaches to management and leadership development

1.3 Aim of the justice sector action learning pilotTo increase the leadership capability of leaders in the justice sector by addressing specific strategic issues that they face and enabling them to implement an appropriate programme of change to resolve those issues within their own business.

1.4 Objectives of a justice sector action learning programmeBy the end of the pilot participants will have:

1. Identified a specific, real life, strategic business issue facing them and their organisation that needs immediate resolution.

2. Worked through a process with a small group, facilitated by an expert, to analyse the problem, develop a plan to tackle the problem and implement a programme of change to resolve the problem.

3. Evaluated the impact of the process on themselves and their business to measure the effectiveness of the programme.

4. Identified a personal development plan to enhance their business and leadership skills.

3 What has action learning learned to become?, Mike Pedler, John Burgoyne and Cheryl Brook

Evaluating the effectiveness of the action learning programme in Wales 5

2. Research objectives

The action learning project undertaken by the SSDA referred to in section 1 was aimed at SME owner/managers. Whilst the key leadership issues reported were similar in nature to those identified by employers in the justice sector and the results of the SSDA project are seen as desirable there is no evidence available to show the effectiveness of action learning within a justice sector context. The question to be addressed is therefore: Can action learning be applied within the justice sector to enhance leadership skills and improve the performance of organisations across the justice system? This question will be addressed through this project.

2.1 Organisations involved A range of organisations are involved in this pilot:

Skills for Justice – have promoted action learning as a potential solution to many of the management and leadership skills issues identified across the sector and in addressing the challenges raised in building effective networks across the justice system. Skills for Justice are project managing the pilot on behalf of employers and want to identify tangible benefits of action learning to promote its use more widely across the justice sector.

Welsh Assembly Government (WAG) – have provided funding to meet 100% of the costs for managers from the voluntary sector and 75% of the costs for managers from non-voluntary sector organisations. As part of the contractual arrangements between WAG and Skills for Justice the pilot needs to demonstrate the delivery of the aims and objectives specified for the pilot (see sections 1.3 and 1.4).

North Wales Police, Dyfed-Powys Police, Her Majesty’s Prison Service, Group 4 Security, The Court Service, The Probation Service and The Department for Constitutional Affairs – senior managers from each of these organisations are taking part in the pilot. Each of these organisations will want to be able to determine the impact of the pilot in terms of the personal leadership skills development of each participant and on the operational problem that is being addressed through the action learning set.

Furthermore, the pilot is of interest to all employers in the justice sector, across the UK.

2.2 Why this research is important Research undertaken as part of the SSA process has identified the development of management and leadership skills as a priority. Justice sector employers already have in place a wide range of management and leadership development programmes, primarily delivered in house with limited use of higher education or external training providers. The vast majority of this provision is delivered within each strand in the sector and tends to focus on theory.

Evaluating the effectiveness of the action learning programme in Wales 6

This pilot programme is being undertaken to prove the value of action learning and show the benefits of undertaking development programmes based on experiential learning that bring together leaders from different parts of the justice sector to share their experiences. The added benefit of an action learning approach being the creation of networks between set members that will support the delivery of a joined up justice system.

Skills for Justice believes that action learning provides a solution to many of the management and leadership skills issues reported by employers. The other key point is that action learning will compliment many of the existing management and leadership development programmes currently in use across the sector by providing a structured process that will enable leaders to apply their learning and share their experiences with other leaders from across the sector. The evaluation of the pilot programme will establish the validity of these beliefs.

2.3 Purpose of researchThe findings of this evaluation will be used to determine the effectiveness of action learning for leadership development within the justice sector. If proven to be successful the findings of this research will be used to inform the delivery and promotion of future action learning sets.

3. Methodology

The research for this evaluation will be based largely on primary data. Where relevant, secondary data related to the issues being addressed by set members, will also be used.

3.1 Primary dataPrimary data will be gathered by:

Reviewing progress against the outcomes specified by the set members (in terms of both the issue to be resolved and their personal development)

Interviews with all members of the action learning set. Using the Kouzes and Posner Leadership Practices Inventory and

reviewing progress made against each set member’s baseline score Obtaining feedback from people who work for the set members Obtaining feedback from set members line managers (where

appropriate) Obtaining feedback from the action learning set facilitators

Evaluating the effectiveness of the action learning programme in Wales 7

3.2 Secondary dataSecondary data will be obtained from each set members operational performance measures. Only the measures relevant to the issue being addressed will be reviewed, as agreed between each set member and the set facilitator.

3.3 ConstraintsAs set members are senior managers within the justice sector the biggest constraint will be their availability for interview. To minimise the risk of set members not being available interviews will be booked well in advance and carried out in whatever way is preferred by set members, either face-to-face or by telephone. As part of the agreement to be a set member each person has agreed to be available to provide information to enable an evaluation of the pilot to be carried out.

4. AnalysisIn order to carry out a structured and robust evaluation of the pilot the Kirkpatrick four-level model will be used.

4.1 Kirkpatrick model4Kirkpatrick's book Evaluating Training Programs defined his originally published ideas of 1959, thereby further increasing awareness of them, so that his theory has now become arguably the most widely used and popular model for the evaluation of training and learning. Kirkpatrick's four-level model is now considered an industry standard across the HR and training communities. The four levels of training evaluation model was later redefined and updated in Kirkpatrick's 1998 book, called 'Evaluating Training Programs: The Four Levels'.

The four levels of Kirkpatrick's evaluation model essentially measure:

reaction of student - what they thought and felt about the training learning - the resulting increase in knowledge or capability behaviour - extent of behaviour and capability improvement and

implementation/application results - the effects on the business or environment resulting from the

trainee's performance

4 Source: Alan Chapman, www.businessballs.com

Evaluating the effectiveness of the action learning programme in Wales 8

4.2 Reaction of set membersThis will evaluate what set members felt about action learning pilot. Feedback will be gathered via evaluation forms and group work within some workshops. This information can be easily obtained and analysed and will be used to identify opportunities for improvement to the format and delivery of subsequent workshops.

4.3 LearningThis will measure the increase in knowledge delivered by the pilot. As each set members requirements will be different and related to their personal and organisational development needs this information will be gathered through interviews with individual set members.

4.4 BehaviourThis will evaluate the extent of applied learning back on the job – actual implementation in the workplace. Information will be gathered from set member’s line managers and the people being led by the set members. A key aspect of this form of evaluation is that observation and interview over time are required to assess change, relevance of change, and sustainability of change.

4.4 ResultsThis will examine the effect on the operational environment by the set member. A range of measures are likely to already be in place via normal management systems and reporting - the challenge is to relate to the measures to the issue that the set member brings to the pilot. The measures to be used and other critical success factors were defined and agreed between the set member, their line manager (where appropriate) and the facilitator and it is these measures that will be analysed to determine the overall Return on Investment (ROI) delivered by the pilot.

5. Structure of the evaluation reportThis chapter provides an indication of the likely structure of the evaluation report along with some initial ideas about the material that will be included. The structure will be as follows:

Executive Summary Introduction Section 1: Action learning theory Section 2: Evaluation methodology Section 3: Issues addressed by the pilot Section 4 : Results achieved Conclusions Recommendations for future action learning programmes Reflections

Evaluating the effectiveness of the action learning programme in Wales 9

Appendices References

5.1 Executive summaryThe executive summary provides the reader with the ‘headline’ messages from the project. This will include the key findings from the research and outline the results achieved by the pilot.

5.2 IntroductionThis will give the reader an overview of the project. It will introduce action learning and outline the aims and objectives of the pilot programme. The purpose of the project will be detailed here and the methodology followed to determine the impact of the pilot will be briefly described. All the remaining sections within the project report will be introduced here.

5.3 Section 1: Action learning theoryThis section will provide an understanding of action learning and how it was applied within the justice sector. The twenty point good practice model developed by the SSDA will be explained here and any key issues related to the application of action learning will be drawn to the attention of the reader.

5.4 Section 2: Evaluation methodologyThe whole research process used will be explained here. As the evaluation of the pilot is based predominantly on primary data the approaches through which information was obtained will be described.

5.5 Section 3: Issues addressed by the pilotThis section will list the organisations involved in the pilot and describe the issues that set members brought to the pilot. The critical success factors for each issue and each set members personal development objectives will also be included here.

5.6 Section 4: Results achievedThis section provides the reader with the findings arising from the analysis that will have been carried out. It will describe the results achieved in relation to the issues that set members brought to the pilot and in relation to each set members personal development objectives. Any problems that may have arisen during the delivery of the pilot will be presented here and this will inform recommendations made in that part of the report.

5.7 ConclusionsThis section brings together the key points from the previous sections. Any key problems identified during the analysis of results will again be presented

Evaluating the effectiveness of the action learning programme in Wales 10

here and will be used to draw out key recommendations and suggested solutions for future action learning programmes.

5.8 Recommendations for future action learning programmesThis section will suggest improvements that could be made to help improve the effectiveness of future action learning programmes. Key marketing messages will be included here as well as any noteworthy examples of success.

5.9 ReflectionsThis section will provide a review of the evaluation that has been carried out. The entire process will be analysed and evaluated with particular regard to the methodology followed and the appropriateness of the Kirkpatrick model.

5.10 AppendicesAny additional material relevant to the evaluation will be included here in support of the report, e.g. blank evaluation forms.

5.11 ReferencesA list of reference materials including books, magazine articles, reports, websites etc will be included here.

6. Time scheduleBroadly speaking, there are three main activities that will need to carried out to undertake this project.

6.1 Collecting dataThis part of the evaluation includes collecting data about the impact that the action learning pilot is having on the set members and their organisations. The data to be collected will be in the form of evaluation forms, feedback from interviews with set members, their line managers and team members and operational performance data related to the issues being addressed via the pilot. Theory regarding action learning and the Kirkpatrick model will also be collected. The deadline for collection is the 30th June 2007.

6.2 Analysing dataThis part will be made up of a number of activities:

Collating and evaluating comments from evaluation forms Collating and evaluating feedback from interviews

Evaluating the effectiveness of the action learning programme in Wales 11

Reviewing operational performance data related to the issues being addressed by set members

Reviewing set members progress against personal development objectives

Selecting and reviewing appropriate literature.

This activity will take place at the same time as the collection of data. The deadline for completion of all analysing activities is 31st October 2007.

6.3 Writing the report This is the final part of the evaluation and focuses on the writing of the report. This activity will be driven by the previous activities and actual writing will begin at the same time as the collection and analysing is carried out. A draft report will be produced and submitted to distance learning team for comment. The final report will be based on the draft and any comments and suggestions made by the project tutors. The deadline for the final report is 31st December 2007.

6.4 ConstraintsThe main barriers are:

The time in relation to the deadline for collection of data. Data relating to the outcomes delivered by the pilot cannot be collected until after each workshop and at other milestones within the pilot. Any delays in the delivery of the pilot could impact on the completion of its evaluation. To manage this the dates for each workshop have been agreed well in advance and committed to set members diaries.

Accessibility of people for interviews. This is another key factor as much of the data will be gathered through interviews. To manage this set members have agreed to provide information and the most convenient method for carrying out the interviews will be agreed with all those involved.

6.5 Time schedule The Gantt Chart on the next page shows a detailed time schedule for the management of the evaluation project.

Evaluating the effectiveness of the action learning programme in Wales 12

Month June July August September October November DecemberActivityCollect feedback on workshops Carry out interviews with set membersCarry out interviews with set members’ line managers (where relevant)Obtain feedback from set members’ teamsReview relevant operational performance measuresReview progress against set members’ personal development objectives Collect literature about action learning and Kirkpatrick modelSelect appropriate literatureAnalyse dataWrite theory sectionWrite section 1Write section 2Write section 3Write section 4Decide on conclusions and recommendationsWrite conclusionsWrite recommendationsWrite introductionReview whole processWrite reflectionsWrite executive summaryHand in draft to distance learning centreAlter report based on comments receivedSubmit final report

Evaluating the effectiveness of the action learning programme in Wales 13

Bibliography

Henley Management College 2004: The Development of the SME Leadership Model Results of Research Study Final Report, http://www.ssda-mandl.org.uk/leadermodel.htm

Ian McGill and Anne Brockbank: The Action Learning Handbook, Routledge Farmer, 2004

International Foundation for Action Learning – http://www.ifal.org.uk

Mike Pedler, John Burgoyne and Cheryl Brook: What has Action Learning Learned to Become?, Action Learning Research and Practice, April 2005

Mike Pedler: Action Learning for Managers, Lemos and Crane, 1996

Reg Revans: ABC of Action Learning, Lemos and Crane, 1998

Skills for Business Network: How to Develop Leadership Capability And Improve Business Performance – a Proven Case, http://www.ssda-mandl.org.uk/leadermodel.htm, March 2006

Skills for Justice: Sector Skills Agreement Phase One Report, www.skillsforjustice.com

Skills for Justice: Sector Skills Agreement Phase Two Report, www.skillsforjustice.com

Evaluating the effectiveness of the action learning programme in Wales 14