Evaluation of the Team Manager Development Programme ... · Evaluation of the Team Manager Development Programme: Final Report i 10 April 2013 Executive summary This is the final

Evaluation of the Team Manager Development Programme: Final Report Social Services Improvement Agency

10 April 2013

This page is intentionally blank

Evaluation of the Team Manager Development Programme: Final Report

10 April 2013

Evaluation of the Team Manager Development Programme: Final Report Social Services Improvement Agency

A report submitted by ICF GHK

Date: 10 April 2013

Job Number J9052

Rakhee Patel

ICF GHK 2nd Floor, Clerkenwell House 67 Clerkenwell Road London EC1R 5BL

T +44 (0)20 7611 1107 F +44 (0)20 3368 6960

www.ghkint.com


10 April 2013

Document Control

Document Title Evaluation of the Team Manager Development Programme: Final Report

Job number J9052

Prepared by Rakhee Patel

Checked by Shane Beadle

Date 10 April 2013


10 April 2013

Contents

Executive summary ............................................................................................................ i

1 Introduction .........................................................................................................1 1.1 Overview of the TMDP ............................................................................................................. 1 1.2 Aim of the study ....................................................................................................................... 1 1.3 Evaluation methodology ........................................................................................................... 1 1.4 Structure of this report .............................................................................................................. 5

2 Overview of the TMDP ..........................................................................................6 2.1 Development, rationale, aims and objectives of the TMDP ..................................................... 6 2.2 Target group ............................................................................................................................. 6 2.3 Delivery of the TMDP ............................................................................................................... 7 2.4 Management and partnership working..................................................................................... 9 2.5 Charges .................................................................................................................................... 9 2.6 Programme expectations ....................................................................................................... 10

3 Design, delivery and performance of the TMDP .................................................. 12 3.1 Development, rationale, aims and objectives ........................................................................ 12 3.2 Profile of TMDP participants .................................................................................................. 13 3.3 Delivery .................................................................................................................................. 16 3.4 Management and partnership working................................................................................... 24 3.5 Fees ....................................................................................................................................... 25 3.6 Key messages ........................................................................................................................ 26

4 Outcomes and impacts ....................................................................................... 28 4.1 Immediate and medium-term outcomes ................................................................................ 28 How participants disseminated learning to colleagues ......................................................................... 36 4.2 Longer term outcomes ........................................................................................................... 38 4.3 Enablers ................................................................................................................................. 39 4.4 Assessment of outcomes for different groups........................................................................ 40 4.5 Additionality of outcomes ....................................................................................................... 41 4.6 Key messages ........................................................................................................................ 41

5 Benchmarking analysis ....................................................................................... 43

6 Conclusions and recommendations ..................................................................... 49 6.1 Is the rationale that underpins the TMDP valid? .................................................................... 49 6.2 How effectively have local authorities identified eligible participants and has the intended

target group been reached? ................................................................................................... 49 6.3 Has the course been delivered appropriately and effectively? .............................................. 50 6.4 Is there evidence of outcomes and impacts for participants and their organisations? What

is the additionality of outcomes? Does the course offer value for money? ........................... 52 6.5 What is the potential for the programme to deliver its expected achievements? .................. 53

Annex 1 Evaluation Framework ............................................................................ 55


i 10 April 2013

Executive summary

This is the final report of the evaluation of the Team Manager Development Programme (TMDP),

undertaken by ICF GHK on behalf of the Social Services Improvement Agency (SSIA). It draws on two

rounds of qualitative and quantitative research tasks: the first undertaken between February 2012 and

May 2012; and the second between October 2012 and December 2012.

Aim of the study

This final report builds on an interim report submitted in May 2012, which provided a formative

evaluation of the operation and quality of the TMDP, identified emerging outcomes and impacts and

made recommendations to shape future provision. This report is intended to provide a summative

assessment of the impacts of the programme of training commissioned and a final assessment of

whether the rationale for the course is valid, how effectively the intended target group has been

reached and whether the course has been delivered appropriately and effectively.

Method

The research in the first stage included:

■ Interviews with 27 programme participants: a mix of participants from the first eight TMDP cohorts

(two to three per cohort), 16 local authorities, a mix of current and aspiring team managers (19 and

eight respectively), and a combination of those working in adult and children’s services. Around

half of the interviewees had completed the programme, while half were mid-way through;

■ Interviews with line managers, peers and supervisees: for 13 interviewees who had completed the

programme, interviews of five line managers and 16 colleagues; for the 14 participants who had

not completed, interviews of ten line managers;

■ Interviews with five tutors who deliver the programme; and

■ An online survey of all TMDP participants to explore their views on the different components of the

course, what they have learnt and whether they have applied their learning. A total of 99

responses were received, which represented a response rate of 54%. Of these, 45% had

completed the programme, 53% were still participating and 2% did not make it clear.

In the final stage it included:

■ Follow-up interviews with 18 participants from the sample to explore whether learning from the

course has been used in their day to day management practice and to identify what has enabled

this or acted as a barrier;

■ Interviews of two line managers and six peers and supervisees of participants who had completed

the course, which had to be supplemented by a short survey because it was difficult to engage

them in interviews. The survey had 20 responses. This means that it has not been possible to

draw out as much as would have been expected about the impact of the course on teams and

organisations;

■ An online survey to the participants who were part-way through the course during the earlier

research to ask them questions about how they had applied learning from the course and wider

impacts, to which 33 out of 52 responded (63%). The survey was also sent to an additional 128

participants who had either joined the programme on cohorts 11 and 12 or who had not responded

to the earlier survey. 51 responses were received (40%). The overall response rate, taking

account of responses for the interim and final stage survey was 65% for completed participants

and 54% for participants part way through, which allows a relatively high level of confidence in the

results.

■ Research about comparator courses. Desk research was carried out about a comparator

programme, Leading to Deliver (Scottish Government and Taylor Clarke Partnership). Information

was collected about the aims and objectives of the course; target groups; level and volume of

credits; delivery model; marketing and promotion activity; fees; development and delivery costs;


ii 10 April 2013

outputs; and outcomes. Information on programme websites, the published evaluation report, and

telephone interviews with staff responsible for the programme within Taylor Clarke was used.

■ Analysis of MI data: participant information; course schedules; summary of feedback

questionnaires for each module of each course; and the results of the 360 degree exercise that

participants complete at the start and end of the course.

Conclusions

Is the rationale that underpins the TMDP valid?

There is significant evidence that the TMDP’s rationale is valid and that the programme continues to

be relevant. Stakeholders, delivery staff, line managers and participants believe there is a clear need

for a bespoke social work team manager programme because generic management courses do not

address key aspects of social work. Stakeholders noted the importance of the programme because of

developments to the Continuing Professional Education and Learning (CPEL) framework that has

identified two key career pathways for social workers, a practice and a management route. The TMDP

is considered to be an important feature of the management route.

The research suggests that an effective and well-run scoping and development phase that sought the

views of key stakeholders has been a key success factor in ensuring the programme’s content

addresses workforce development needs effectively. It also suggests that having a national

programme has increased expectations about the knowledge, skills and competences that team

managers need. The TMDP is being seen as a required qualification for all aspiring and current team

managers and as a stepping stone for further study.

As a consequence, we recommend that the course should continue to be offered in the future.

How effectively have local authorities identified eligible participants and has the intended target group been reached?

The evidence suggests that the intended target group of current social work team managers has been

effectively reached throughout Wales although there is also evidence that practice in identifying and

recruiting participants differs between local authority areas and that some participants are

inadequately informed about the requirements of the programme by their employer. As a

consequence, we recommend the continued use of the pre-programme seminar to inform both

participants and their line managers.

The target group has been extended to include aspiring managers and professionals from other

professional backgrounds that are related to social work, such as occupational therapy and community

nursing. The emphasis of national policy on delivering integrated services suggests that it is

appropriate to bring together practitioners from different service areas and there is evidence that

these individuals and their organisations are benefitting from the programme as much as those from

social work backgrounds. However, if the programme continues to expand its offer to these groups, it

will be important to assess whether it continues to meet their needs. The commissioners of the

programme are keen to closely monitor the programme as it is extended. As a consequence, we

recommend representatives from the relevant professional bodies should also be included in

governance arrangements. If the number of professionals from other backgrounds increases, it is

also likely to have implications for local authority training managers who will be responsible for funding

and allocating places.

A care home manager was accepted onto the programme during the early stages of the programme

Given that that the roles and responsibilities of this position are significantly different to those of social

work or occupational therapy (OT) team managers, it is not clear that this is appropriate. There have

been no further care home managers accepted onto the course. We recommend continued

monitoring of requests from local authorities to place care home managers on the programme.

There is no evidence to suggest that achievements and impacts for participants in different types of

job are different. We have found that aspiring and current team managers experience similar

outcomes and have similar views about its relevance. For those who have been in managerial roles

for a longer time, the programme consolidates and refreshes knowledge, while for newer managers,

the programme has been successful in providing understanding about the skills team managers


iii 10 April 2013

require and supporting managers to use and apply these skills to improve their services and teams’

performance. There is also evidence to suggest that participants from OT backgrounds experience

similar outcomes to professionals from a social work background. As a consequence, we

recommend that the current mix of participants is maintained.

Has the course been delivered appropriately and effectively?

Regional delivery model

The regional delivery model is effective and provides flexibility for participants. It has also contributed

to learning that would not have taken place if the programme had been delivered at local authority

level. We recommend that it should, therefore, be maintained.

Pre-programme seminar

The evidence demonstrates that the pre-programme seminar has been delivered effectively, although

difficulties in ensuring all line managers attend persist. As the programme continues, it will become

increasingly likely that line managers will have already attended for one participant and may therefore

be reluctant to attend the same seminar again. As a consequence, we recommend that other ways

of engaging and staying in touch with line managers are explored and implemented.

360 degree assessment and Personal Development Plan (PDP)

The 360 degree assessment has been a valued component of the programme, with around two thirds

of survey respondents reporting it to be very useful or useful, even though there have been some

practical difficulties in relation to the same staff not being available to complete the ‘before’ and ‘after’

and colleagues not being able to comment on the skills of newly appointed managers. As a

consequence, we recommend the use of the 360 degree tool as a key component of the

programme should continue.

Use of the PDP appears to have been varied, with some participants reporting that although they

thought it was beneficial to develop the plan at the start of the programme, they had used it to a limited

extent subsequently. This may be in part because of a lack of support from line managers (only a fifth

of survey respondents indicated they had considerable support or the level of support required to

develop and review the PDP). As a consequence, we recommend the management committee

and delivery agent should consider how they can influence local authority senior managers to

ensure PDPs are used and appraised more regularly. They should also monitor whether the

recent introduction of the requirement for line managers to sign off PDPs improves how they

are used to support participants’ development.

Taught days and action learning sets

Delivery of the taught days and action learning sets has also been appropriate and effective with four

fifths of survey respondents describing them as either very useful or useful. Most participants praised

the practical work-based focus of the programme and valued the theoretical frameworks to which they

were introduced. A few aspiring team managers reported that they found the first teaching days

difficult because they were learning new terms and concepts at a relatively fast pace. It is to be

expected that undertaking postgraduate study may pose challenges and that aspiring managers take

slightly longer to adapt. This is perhaps an indication of the level of up-skilling that is taking place.

The action learning sets have for many participants been a highlight of the programme. There is

evidence that TMDP tutors have effectively facilitated discussion to support critical examination of

work-based problems. The range of backgrounds from which participants come has enriched this.

Around a quarter of participants interviewed suggested that continuing the action learning sets after

the end of the course would be helpful; meeting with peers to share learning, experiences and

problems would be a good form of post-programme support. As a consequence, we recommend

that the action learning set approach should be maintained and the possibility of offering the

opportunities for follow up action learning sets post-completion should be explored.

Individual mentoring or support

Nearly three quarters of survey respondents (74%) received individual mentoring or support while

undertaking the TMDP. Of these, over four fifths found the support useful (38%) or very useful (47%).

Participants have highlighted that constructive feedback has been provided and that good professional


iv 10 April 2013

relationships have been built. They reported that tutors have been available and helpful. As a

consequence, we recommend the delivery agent should continue to emphasise that tutors are

available to provide additional guidance, should continue to monitor attendance and amend

student feedback forms to include a section on tutorials. The delivery agent should also ensure

that participants are aware that tutorials can take place over the telephone to overcome any

potential barriers related to travel times and distance.

Assignments

The challenge of completing assignments alongside a pressurised and busy job is evident.

Nonetheless, participants have found the assignments helpful and, in particular, the assignments for

modules 2 and 3. A few participants noted that starting a module before submitting the assignment for

the previous models creates additional pressures. As a consequence, the assignments appear to be

working well and we recommend that assignments should be maintained in a similar format,

although the possibility of re-scheduling deadlines should be explored.

Tri-partite agreement

The tri-partite agreement sets out employer commitment to provide participants with a minimum of one

study day per module. According to the survey, around a fifth of participants have had little or no study

leave and under a third (28%) considered they had the level of support needed. In this context, the

agreement is a useful tool for negotiating with employers to enable participants to take their allocated

study leave. Regardless of whether they were able to take study leave or not, most participants

interviewed reported that the allocation of one day was insufficient and they had to use their own time

to complete assignments. A few local authorities have offered their staff additional study days and this

appears to have helped participants. As a consequence, we recommend that changing the tri-

partite agreement to include a commitment to two study days should be considered. In addition,

we recommend that the SSIA and delivery agent should continue their efforts to build strong

relationships with local authority training managers to ensure that they act as advocates for

the TMDP participants and support them to take their allocated study leave.

Line manager involvement

There is evidence that not all line managers have provided the level of support required during the

programme. Only 20% of respondents to the survey reported that employers had provided the level of

support needed or considerable support in terms of advice and support with assignments and 23%

reported support to develop and review the PDP. The mechanisms in place to encourage line

manager involvement, such as the tri-partite agreement, pre-programme seminar, the sign off of the

PDP and a statement in the work-based project, are valued by participants but consideration should

be given to other actions, such as a mid-term meeting between delivery staff, line managers and

participants and involving line managers at the end of the programme to support participants’

progression after completing. A mid-term review between students, line managers and tutors is a key

component of the undergraduate social work degree so staff are generally familiar with this approach

and understand its value. As a consequence, we recommend that mid-term and end of programme

seminars, whether face to face or virtually, either at regional or local authority level, should be

included in the programme.

Has the programme been effectively monitored and reviewed?

There are effective processes in place for monitoring and reviewing the programme, which have

resulted in modifications and improvements to the programme over time to better respond to students’

needs. These include regular meetings between delivery staff to discuss student feedback,

achievements and their experiences of teaching. As a consequence, we recommend these

meetings should continue to take place. Furthermore, stakeholders have indicated that the

programme has been effectively managed by the national programme management committee. There

is clear evidence that the committee is functioning well and has brought together the right

stakeholders. As a consequence, we recommend that as the programme moves into its next

phase, strong governance arrangements should continue to be in place.


v 10 April 2013

Is there evidence of outcomes and impacts for participants and their organisations? What is the additionality of outcomes? Does the course offer value for money?

The majority of participants have achieved many of the expected immediate and medium term

outcomes of the TMDP. All of the participants interviewed have applied learning to effect change in at

least one area of their management practice as a result of the course. In most cases, participants

have reported that new processes introduced as a result of the work-based project have continued to

be used after completion of the TMDP.

There is evidence for around half the participants that after completing the course they have made

other changes to the way they manage their team beyond the requirements of the work-based project

component of the course. In particular, there is evidence of sustained change in relation to ensuring

the national policy agenda shapes the team’s work, managing change, improving service user

involvement, managing the performance of individuals, and using different management tools and

techniques. There is less evidence of sustained changes to processes for audit and inspection, the

use of research and evidence to shape work, and managing demand and capacity.

Generally, this suggests that the TMDP has achieved its aims and objectives and that it has:

■ Supported managers to develop an in-depth understanding of the key drivers influencing social

care and policy and practice in Wales;

■ Supported the development of critical thinking skills to improve case management decisions;

■ Provided managers with a range of tools and techniques to improve the quality of practice within

their teams have been met; and

■ Contributed substantially to helping team managers address the challenge of increasing demand

for services and increasing user expectations at a time of declining resources.

There is evidence that outcomes have arisen largely a result of the course although a few participants

considered that over time some of the outcomes they reported (relating to improved processes for

assessing and reviewing quality, managing demand and capacity and service user involvement, for

example) would have been achieved even if they had not taken part in the TMDP. However, they

consider that as a result of the TMDP, these have happened much more quickly and systematically. A

few participants, including those working in youth offending teams, reported that other training courses

had influenced their understanding, knowledge and behaviour as well as the TMDP. They attributed

changes in practice to both sources of learning. There is no evidence that the TMDP resulted in

displacement or substitution of other learning about management practice.

Having the support of line managers and other colleagues who have completed the TMDP enables

learning to be applied over a sustained period of time. Therefore, to enhance outcomes and ensure

they are sustained over time, we recommend that efforts should continue to roll out the TMDP

to more team managers and to provide complementary training to senior managers.

The inputs for the design and delivery of the TMDP seem appropriate to the level of outputs and

outcomes achieved and the unit cost of £3,145 per participant compares favourably to another similar

course, with comparable outcomes for participants.

What is the potential for the programme to deliver its expected achievements?

As yet there is little evidence of impact on service efficiency and effectiveness though there is

evidence of this in many cases on a small scale within areas that the participants manage. There is

emerging evidence that the TMDP is supporting greater career progression and, in particular,

providing an improved transition for aspiring managers. The survey suggests that a third (33%) of

participants have changed jobs since starting the TMDP and most have gone on to higher level jobs.

There are also examples of participants interviewed who have been promoted while doing or after

completing the TMDP, who believe that the programme has contributed significantly to their success in

securing a new position. As yet there is little evidence of greater satisfaction among service users as

a result of the TMDP but there is evidence of learning in relation to how to involve service users in

quality improvement projects and examples of applying this learning from participant and line manager

interviews. As a consequence, it would appear that there is significant potential for the

programme to deliver its expected impacts on the development of the workforce.


1 10 April 2013

1 Introduction

This is the final report of the evaluation of the Team Manager Development Programme

(TMDP), undertaken by ICF GHK on behalf of the Social Services Improvement Agency

(SSIA). It draws on two rounds of qualitative and quantitative research tasks: the first

undertaken between February 2012 and May 2012; and the second between October 2012

and December 2012.

1.1 Overview of the TMDP

The TMDP (also known as the Post Graduate Certificate in Managing Practice Quality in

Social Care) is a national programme delivered across Wales and aimed at frontline social

work team managers and senior practitioners or those who aspire to those positions. The

programme was commissioned by the SSIA and is delivered by the Institute of Public Care

at Oxford Brookes University in partnership with the University of Glamorgan and Research

in Practice. The programme aims to enable team managers to improve service quality and

outcomes.

1.2 Aim of the study

This final report builds on an interim report submitted in May 2012, which provided a

formative evaluation of the operation and quality of the TMDP and identified emerging

outcomes and impacts. This report is intended to update the previous report and provide a

final assessment of whether the rationale for the course is valid, how effectively the intended

target group has been reached and whether the course has been delivered appropriately

and effectively.

In addition, this report is intended to update previous findings about emerging outcomes and

impacts and provide a summative assessment of the impacts of the programme of training

commissioned. It therefore has the following key objectives:

■ To assess outcomes and impacts achieved to date for individual participants, their teams

and organisations, and to highlight specific examples of improved practice resulting from

participation;

■ To assess to what extent any outcomes and impacts have arisen as a result of the

course;

■ To identify the enablers and barriers faced by participants to applying learning from the

course to every day professional practice; and

■ To assess whether the course offers value for money, drawing on a comparative

analysis with similar social work leadership and management courses.

1.3 Evaluation methodology

1.3.1 Scoping stage – March 2012

Based on interviews with key stakeholders, a review of documentary material and

management information, a logic model was produced for the programme which sets out

the theory underpinning the TMDP i.e. how it is intended to bring about benefits for

participants, members of their team and their organisations. The logic model drew on the

Kirkpatrick model1 . The Kirkpatrick model provides a framework for considering the ‘causal

chain’ between learning and impact. The immediate, short term, intermediate and longer

term outcomes and impacts expected from a training programme are examined as a

chronological sequence. Researching the stages of the ‘journey’ that an individual can make

after training and support, provides evidence from which inferences can be drawn about

1 See http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm for more information on the

Kirkpatrick model.

http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm


2 10 April 2013

changes to their individual capacity and the organisational capacity of the place where they

work (Figure 1.1).

Figure 1.1 The Kirkpatrick model for the evaluation of training programmes

The logic model and the framework are presented in Figure 2.2 and Annex 1 respectively.

1.3.2 Stage 2: Interim report – March 2012-May 2012

To assess the programme against the logic model and evaluation framework, we undertook

the following research tasks, in addition to the analysis of MI:

■ 27 semi-structured telephone interviews with a sample of programme participants

(against a target of 25). Our target was to interview at least three participants from each

of the first eight cohorts and we achieved this except for cohorts two, three and six, for

which we interviewed two participants (see Error! Reference source not found. below).

This provided a good spread of participants across the cohorts.

Table 1.1 Achieved sample of interviewees - cohort

Cohort Achieved number of interviews

1 5

2 2

3 2

4 4

5 5

6 2

7 4

8 3

TOTAL 27


3 10 April 2013

We were also aiming for 17 participants who had team management responsibilities and

eight who did not. We exceeded the target for current team managers (19) and achieved

the target of eight aspiring managers. It should be noted that two of the ‘aspiring’

managers we interviewed moved into the role of team manager while on the TMDP.

We also intended to get a good spread of local authorities across the sample, with

representatives from at least 14 local authorities, and a mix of participants who work with

children and adults. We also achieved these targets, with participants from 16 local

authorities included in the sample and around half of participants working in adult

services and half in children’s services (14 and 13 respectively).

The interviews explored their views of the TMDP programme and its different

components, learning outcomes, actions taken as a result of the programme and, for

those interviewees who had completed the programme, perceived outcomes and

impacts.

■ Interviews with line managers, peers and supervisees. We intended to interview the

line manager or training manager of each of the 25 participants we interviewed. In

addition, for those participants who had completed the programme we aimed to interview

two colleagues, ideally one peer and one supervisee in their team. In relation to the 13

participants who had completed, we have interviewed five line managers and 16

colleagues. For the 14 participants who had not completed, we interviewed ten line

managers. Five participants did not provide contact details for line managers and

colleagues. Five line managers declined to participate or were not available at the

agreed time for the interview.

■ Five semi-structured telephone interviews with key stakeholders. We interviewed

wider stakeholders with an interest in the quality and content of the programme,

including representatives from the Care Council for Wales, Welsh Government, the

Association of Directors of Social Services Cymru and two local authority training

managers. The interviews explored views on whether the rationale for the TMDP is

considered to be valid; whether the programme complements existing training provision

for team managers; whether there are any gaps in the programme; the management and

delivery of the programme; and the outcomes and impacts to date.

■ Interviews with the five tutors who deliver the programme. These interviews

explored tutors’ experiences of teaching on the course and its effectiveness to date.

■ Survey. We carried out an online survey of all TMDP participants to explore their views

on the different components of the course, what they had learnt and whether they had

applied their learning. An email with a link to the survey and three subsequent reminder

emails was sent to 184 participants. A total of 99 responses were received, which

represents a response rate of 54% for this survey2. Of these, 45% of respondents had

completed the programme, 53% were still participating and 2% did not make clear what

stage they were at. Participants who had completed the programme were asked

questions about how they have applied learning and wider impacts, while those still

doing the programme were asked about increases to knowledge and understanding so

far.

The findings of these research tasks were used to produce the interim evaluation report

submitted in May 2012.

1.3.3 Stage 3: Final report – October 2012-December 2012

The final stage of the study has included the following research tasks:

■ Follow-up interviews with the same sample of participants. We intended to re-

interview 25 participants we had interviewed during Stage 2 of the research and were

2 For a 95% confidence level this gives a confidence interval of 6.7.


4 10 April 2013

able to complete 18 of these interviews. For those participants who had completed the

TMDP last time, the aim of the interviews was to explore whether learning from the

course has been used in their day to day management practice and to identify what has

enabled this or acted as a barrier. We explored whether plans and ideas mentioned in

the previous interview had been implemented. For participants who were part-way

through the course last time and had now finished, we explored views on the value and

effectiveness of the programme, and whether upon completion it has led to changes in

approaches to management ‘on the ground’ within their organisation.

■ Interviews with line managers, peers and supervisees. For the participants who were

part-way through the course last time but have now finished, we intended to re-interview

their line manager and, in addition, conduct interviews with two colleagues. We achieved

interviews with two line managers and six colleagues. Two line managers declined to

participate in the research because of staff shortages and other demands on their time.

Team managers were reluctant to provide contact details of peers and colleagues

because they were concerned about the pressures on their staff’s time. Where we were

provided with contact details, it was difficult to secure interviews. Some colleagues and

peers felt that they did not know enough about the TMDP and their manager’s

involvement and could not be persuaded to take part. In most cases, we were not able to

make email or telephone contact with line managers, peers or supervisees. Typically,

three emails were sent and three or four telephone messages were left.

Because of the difficulties in engaging line managers in interviews, we sent out a short,

supplementary survey to line managers, which received 20 responses.

Therefore, this report mainly relies on interviews conducted for the interim report with line

managers, peers and supervisees to assess the impact of the TMDP on participants’

teams and organisations. This means that we have not been able to draw firm

conclusions about the impact of the course on wider teams and organisations.

■ Survey. During Stage 2, participants who were part-way through the course were not

asked questions about how they had applied learning from the course and wider

impacts. We therefore re-sent questions relating to the application of learning and

impacts to these 52 participants, to seek reflections now that they had completed the

course; 33 responded (63%). We also sent the survey to 43 additional participants on

cohorts eleven and twelve, who had not been surveyed previously, and the 85

participants from cohorts one to ten who did not respond last time (a total of 128

participants); we received 51 responses in total (40%). 15 of these were from

participants who had completed the course, 31 from those who were part way through

and five did not state whether they had completed or not. This means that overall the

survey was sent to 142 participants who completed the course (cohorts 1-8) and 93

responses were received; and it was sent to 94 participants who were part way through

(cohorts 9-12) and 51 responses were received. This represents a response rate of 65%

for completed participants and 54% for participants part way through. This means we

can have a relatively high level of confidence in the results3.

■ Research about comparator courses. We carried out desk research about the two

comparator programmes identified during the scoping stage, Leading to Deliver (Scottish

Government and Taylor Francis Partnership) and the Front Line Leaders programme

(National Skills Academy for Social Care- NSA). We looked for information about the

aims and objectives of the courses; target groups; level and volume of credits; delivery

model; marketing and promotion activity; fees; development and delivery costs; outputs;

and outcomes. We drew on publicly available information (such as information on

programme websites) and also conducted short telephone interviews with staff

responsible for the programmes within the National Skills Academy for social Care in

England and the Taylor Clarke Partnership in Scotland.

Our desk research in March 2012, during the scoping stage of the study, identified the

National Skills Academy for Social Care Front Line Leaders programme as a programme

3 For a 95% confidence level, this gives us a confidence interval of 5.71 for completed participants.


5 10 April 2013

targeted at supervisors and managers working in front line social care settings in

England. The website for the programme noted that on successful completion of the

programme, participants gain a Postgraduate Certificate in Advanced Business Practice

(30 credits at Level 7). The website also reported that the programme had been

evaluated. In November 2012, when we tried to access information on the public

website, the relevant page was no longer available. Staff at the NSA were not able to

provide further information about this course. They directed us instead to a Level 4

programme targeted at supervising care staff in domiciliary and residential care or to a

leadership programme targeted at senior managers, which are not appropriate

comparators. The main training offered to front line managers by the NSA appears to be

a series of five day-long workshops. Although on topics comparable to those addressed

by the TMDP (Leading and Managing Cultural Development; Successful Behaviour for

Managers and Leaders; Supervision and Appraisal; Leading and Managing the Process

of Change; and Leading and Managing the Inspection Process), the workshops do not

have university accreditation and do not lead to a Postgraduate Certificate. We have

therefore not been able to identify and conduct research about a suitable comparator

course delivered by the NSA. This research task has therefore focused on obtaining

information about the Leading to Deliver programme in Scotland.

■ Analysis of MI data. We updated our analysis of programme MI data so that it included

the following data to the end of October 2012: participant information; course schedules;

summary of feedback questionnaires for each module of each course; and the results of

the 360 degree exercise that participants complete at the start and end of the course.

1.4 Structure of this report

The remainder of the report is structured as follows:

■ Chapter 2 presents an overview of the TMDP’s rationale, delivery model, content and

intended outcomes and impacts;

■ Chapter 3 assesses the design, delivery and performance of the TMDP;

■ Chapter 4 assesses outcomes and impacts achieved;

■ Chapter 5 presents findings from the benchmark analysis; and

■ Chapter 6 presents conclusions and recommendations.


6 10 April 2013

2 Overview of the TMDP

In this section, we provide a narrative overview of the programme and also reproduce the

logic model we have developed that summarises the logic implicit between activities,

outputs/outcomes and longer term impacts.

2.1 Development, rationale, aims and objectives of the TMDP

The TMDP was developed in response to significant challenges facing front line managers in

social care in Wales. Key among these was the need to deliver high quality services in a

context of rising demand, increased user expectation and declining resources, which

requires the reshaping of core services, adoption of collaborative approaches to

commissioning and delivery and the development of an outcomes-focused performance

culture. These challenges were set out clearly in the ‘Fulfilled Lives, Supportive

Communities’ ten year strategy for Social Services in Wales (WG, 2006) and have been

endorsed in a range of subsequent policy statements, notably ‘Sustainable Social Services

for Wales: A Framework for Action’ (WG, 2011).

In 2009, the SSIA commissioned the Institute of Public Care (IPC) to design the programme.

IPC brought significant relevant experience, having previously designed and delivered a 20

credit module about ‘Leading Quality Improvement and Change in Public Care’. IPC carried

out extensive scoping research to explore the training needs of current and aspiring team

managers and to understand what other provision was available. The scoping exercise drew

on a survey of training managers, operational managers, senior managers and senior

practitioners and also regional workshops that brought these stakeholders together. It

established that:

■ There was consensus among stakeholders that building the skills of frontline managers

is crucial to ensuring that strategic priorities and imperatives are translated into effective

front-line practice and building a positive organisational culture that focuses on delivery;

■ Existing local authority management programmes tend to bring managers together from

different service areas rather than being tailored to the needs of social work staff;

■ Managing poor performance, change and case management were areas where there

was a need for knowledge, understanding and skills to be developed; and

■ There were around 400 team managers and 500 senior practitioners in Wales.

Furthermore, the scoping study found that although some local authorities had worked in

collaboration with Higher Education Institutions to develop training specifically for social work

managers, the content and format varied across Wales.

The scoping research therefore confirmed the need for a national training programme for

current and aspiring social work team managers.

The overall aims of the TMDP are to:

■ Help managers develop an in-depth understanding of the key drivers influencing social

care policy and practice in Wales;

■ Support the development of critical thinking skills to improve case management

decisions; and

■ Provide managers with a range of tools and techniques to improve the quality of practice

within their teams by building and sustaining positive culture and professional behaviour.

2.2 Target group

As well as current team managers, the TMDP is targeted at aspiring team managers,

typically in a senior practitioner or senior social worker role. To be eligible to register on the

programme, potential participants must have:


7 10 April 2013

■ A recognised social work professional qualification and registration with the Care Council

for Wales, the General Social Care Council, the Scottish Social Services Council or the

Northern Ireland Social Care Council (or equivalent body for non-social work staff where

applicable);

■ Three years post-qualifying experience;

■ A current position as a team manager or a senior practitioner, or who have been

identified as an emergent leader; and

■ The ability to study at postgraduate level.

The course also considers applications from occupational therapist team managers and

other health professionals.

The intention was to make the programme available to around 750 team managers and

senior practitioners working in Wales from its start in 2010.

2.3 Delivery of the TMDP

Following an open tender process, IPC, in partnership with the University of Glamorgan and

Research in Practice, was commissioned to deliver the programme. The TMDP is formally

accredited by Oxford Brookes University and provides participants with 60 Credit

Accumulation and Transfer (CAT) points towards a Masters degree. The programme is

included within the Care Council for Wales’s Continuing Professional Education and

Learning Framework (the CPEL Framework) as a recommended qualification for appropriate

practitioners.

The programme is delivered by a team of five tutors with experience of managing front line

health or social services. All tutors have completed the Oxford Brookes associate teacher

certificate and the team is overseen by the Assistant Director of the IPC.

2.3.1 Identification and recruitment of participants

Training managers of each local authority are asked to nominate potential participants. It is

intended that they work with service managers to disseminate information and promote the

programme through internal channels such as intranets and email distribution lists. Local

authorities are required to submit an application form to the IPC for each candidate, which

provides basic information (name, job title contact details, Care Council for Wales or

equivalent body registration number, line manager contact details and whether the full cost

of the programme will be met by the applicant or whether the place applied for is subsidised

by the SSIA). IPC then assess whether the applicant is eligible. In cases where there are

questions in relation to the eligibility of the candidate, IPC present the application and explain

the issues to the management committee.

2.3.2 Delivery location and model

The TMDP is delivered through regional cohorts with courses available in North Wales,

Cardiff, Carmarthen, Cardiff and Swansea. There have been 13 courses between May 2010

and October 2013 and nine of these are complete. It was intended that there would be no

more than 20 participants per cohort. The programme comprises three modules, usually

delivered over a period of 12 months, which include taught days, facilitated action learning

sets and assessed work.

2.3.3 Key programme components

The course begins with a pre-programme seminar for participants and their line managers to

inform them about what to expect from the programme in terms of content and time

commitment.

Before the first module begins, participants are required to complete a 360 degree

assessment tool. This is an electronic questionnaire which the participant, their line manager

and a selection of colleagues complete in order to identify areas for personal development.

The assessment is, therefore, intended to be the basis for a personal development plan

(PDP), which participants use and refine throughout the programme. Although integral


8 10 April 2013

components of the programme, neither the 360 degree assessment or PDP are assessed or

contribute to the final grade.

For each of the three modules there are two taught days, which ‘combine theoretical input

with case studies and practical application exercises (Student Handbook). This is followed by

three support days which include facilitated action learning sets, scenario exercises and

individual support in the form of a tutorial meeting.

At the end of each module, participants undertake an assignment which is assessed. It is

intended that these are tailored to the specific work contexts of individual participants and so

are agreed between participants and tutors. At the end of the first module there is a written

assignment, at the end of the second module a reflective work-based project, and at the end

of the final module a work-based case study and presentation.

Participants can submit assignments in Welsh but there is currently no Welsh-language

delivery.

Figure 2.1 Structure of the TMDP

Source: SSIA

2.3.4 Monitoring and review

A process has been defined for monitoring and review of the programme. During the early

stages of the programme (spring and autumn 2010), the delivery team met on a monthly

basis in order to ensure that they had a shared understanding of the course, that delivery

was consistent and to allow the team to refine the course based on experiences of initial

delivery. Subsequently, the delivery team has had quarterly review meetings. These

meetings provide a formal opportunity for ongoing quality assurance and also to make any

updates to course content that may be necessary because of policy changes. Because of

the pace at which policy is changing, IPC recently requested regular formal updates on the

policy agenda from the management committee. In addition, feedback is systematically


9 10 April 2013

collected from students. Each participant completes a module feedback form and the results

are collated and analysed for each course.

2.3.5 Line manager involvement

As stated in the Student Handbook, ‘the ethos of the programme is to support the learning of

students through a working partnership involving the student, their employing agency and

IPC’. There is, therefore, an expectation that line managers provide ongoing support to

participants throughout the programme by offering ‘formal and informal advice, support and

review of progress on assignments”. This includes ensuring participants have time to attend

taught days and undertake self-study (a minimum of one day per module). A tri-partite

agreement has been established which sets out the responsibilities of line managers,

participants and the IPC and must be signed by all parties.

Since its inception, the programme has evolved to build in more systematic involvement of

line managers. In addition to the tri-partite agreement, an additional line manager

observation has been included in the assessment, there is greater description in course

materials about the role of participants’ line managers and line managers are now required

to sign off the PDP.

2.4 Management and partnership working

The development and implementation of the programme is overseen by a National

Programme Management Committee comprising representatives of the Association of

Directors of Social Services in Wales (ADSS Cymru), course participants, Care Council

Wales and the All-Wales Social Services Training Managers Group. There are no formal

performance management targets for IPC.

2.5 Charges

Course fees are £2750. Fees are currently paid for jointly by participants’ sponsoring local

authorities and the SSIA. In 2009-2010, the SSIA contributed £1,250 per student for 103

students and in 2010-11, £1,000 per participants for 106 students, resulting in a total of 209

subsidised places being offered. Participants must be employed by the local authority to be

eligible for a subsidised place.

Table 2.1 Number of subsidised places offered to each local authority

Local authority No. of subsidised places

Flintshire

(only joined the programme in 2010-11) 4

Blaenau Gwent 6

Ceredigion 6

Merthyr 6

Monmouthshire 6

Torfaen 6

Ynys Mon 7

Bridgend 8

Denbighshire 8

NPT 8

Pembrokeshire 8

Vale of Glamorgan 8

Gwynedd 9

Conwy 10

Wrexham 10

Caerphilly 12


10 10 April 2013

Local authority No. of subsidised places

Newport 12

Powys 12

Cardiff 13

Carmarthen 14

RCT 18

Swansea 18

209

SSIA Management Information

2.6 Programme expectations

The logic model below (Figure 2.2) provides a summary of the programme’s rationale,

inputs, target group, delivery and management activities and intended outputs and

outcomes.

In Chapter 3 we assess how effective the design, delivery and management of the TMDP, as

set out in the logic model, has been and in Chapter 4 we assess whether there is any

evidence that the outcomes presented in the logic model have been achieved.


11 10 April 2013

Figure 2.2 Logic model of the TMDP

Inputs:

Course fees of £2750:• SSIA financial

contribution: £1,250 in 2009/10 for 103 students; £1,000 in 2010/11 for 106 students

• Employer (local authority) financial contributions and any remission of working hours to attend taught days and action learning sets and for study leave

In kind contributions:• Time,

expertise and knowledge of SSIA and National Programme Management Committee members

• Expertise of delivery team and stakeholders to enhance the training offer

• Time of participants outside of working hours

• Time of any work-based mentors

• Time of line managers

Activities:National Programme Management Committee

Commissioning of TMDP

Course design and production of teaching materials

Partnership working between the SSIA, course provider and nominating local authorities

Marketing and promotion

Local authorities identify eligible participants and provide informal and formal support and review throughout the course

Learning agreements between provider, employer and student

Delivery of an accredited course comprising three modules delivered on a regional basis across Wales, which includes:• 360 degree assessment

at start and end• development of a

Personal Development Plan

• pre-programme seminar for participants and line managers

• two taught days per module (three modules)

• formative and summative assignments

• action learning sets-three days

• Individual mentoring or coaching support (telephone or face to face)

Ongoing review and monitoring, including student feedback

Outputs:

Production of a student handbook

No. of participants accessing the programme

No. of cohorts

No. of modules successfully completed

No. of participants achieving the PG Certificate

No. of 360 degree assessments and Personal Development Plans completed at the start and end of the programme

Immediate Outcomes:ParticipantsConsider the course to be relevant to their roles and needs

Have increased confidence and are better able to identify support for themselves

Have increased understanding of the national agenda for change

Have increased understanding of the audit, inspection and review processes in Wales and effective responses

Have increased understanding of how to manage partnerships across disciplines

Believe they have acquired new knowledge and understanding about the factors influencing quality, believe they are able to improve quality within their team;, see opportunities to apply learning and are planning to do so. This includes understanding:• How to manage effective demand and

capacity• How to manage casework• How to include service users in quality

improvement projects• How to manage individual

performance within teams, particularly new staff and those performing poorly

• How to use evidence-based practice, how to interpret research and embed in service provision

• The relative benefits of different management tools

• How to review, monitor and analyse quality arrangements

Have increased understanding of how to evaluate and reflect on their own personal competences

Share learning with colleagues

Employers recognise the learning acquired and value the training

Medium-termOutcomes:

ParticipantsImproved transition for aspiring managers to the role of team manager

Reflect the national agenda in their local practice

Improved planning and management of national audit, inspection and review processes

Improved management of partnerships across disciplines

Apply knowledge and understanding to lead quality improvements:• Improved processes for assessing and

reviewing quality arrangements• Improved management of demand and

capacity• Improved management of casework• Improved involvement of service users in

quality improvement projects• Improved management of the performance

of individuals, including improved identification of development needs, better supervision and support for new staff and staff performing poorly

• Increased use of a range of management tools

• Improved use of research and evidence in practice

Have increased commitment to ongoingprofessional development

Believe they have increased opportunity for career progression

Team members feel more motivated and supported

Employers recognise increased capacity to manage change and improve service quality

Long-termOutcomes:

Improved efficiency and effectiveness of services managed directly by participants, including improved budget management and value for money

Fewer management resources elsewhere in the organisation spent resolving problems in services managed by participants

Greater career progression for social work team managers

Greater satisfaction among users of those services managed by participants that services are focussed on their needs

.

Target group:

Team managers and aspiring managers with:

• a recognised social work professional qualification and registration with the Care Council for Wales the General Social Care Council, the Scottish Social Services Council or the Northern Ireland Social Care Council (or equivalent body for non-social work staff where applicable);

• three years post-qualifying experience;

• a current position as a team manager or a senior practitioner, or who have been identified as an emergent leader

• the ability to study at postgraduate level

.

The Team Manager Development ProgrammeContext and rationale: The Welsh Government’s Strategy for Social Services – ‘Sustainable Social Services for Wales: A Framework for Action’ (February 2011) sets clear expectations for services that are citizen-focused, deliver positive outcomes that contribute to the well-being of users, are provided in collaboration across local authorities and with other sectors and provide value for money. The key role of front line workers in making this a reality, supported by visionary leadership, is recognised. This reinforces the founding principles of the Team Manager Development Programme, which has sought to support team managers in improving the consistency of practice quality. It has sought to provide focus where previous local authority training tended to be generalist in approach, and provide a national framework to facilitate greater consistency across the country. The driving aim of the programme has been to equip team managers with the right skills to lead and sustain quality improvements’


12 10 April 2013

3 Design, delivery and performance of the TMDP

In this section we present evidence from:

■ Interviews with participants, their line managers and colleagues, delivery staff and

stakeholders;

■ The survey of TMDP participants about the effectiveness of the design, delivery and

performance of the TMDP; and

■ MI data to the end of October 2012.

We interviewed participants twice: once in March 2012 and once in November 2012.

Similarly, the survey was sent out in two phases.

In particular we synthesise evidence about the development, rationale, aims and objectives

of the programme; the profile of TMDP participants and how this compares to the intended

target group; identification and recruitment of participants; the regional delivery model;

effectiveness of the key components of the programme; monitoring and review processes;

line manager involvement; and management of the programme.

3.1 Development, rationale, aims and objectives

Stakeholders, delivery staff, participants and their line managers expressed the view that the

rationale underpinning the TMDP is appropriate. They considered that the gaps in

knowledge and skills of team managers identified in the development phase of the

programme were accurate and welcomed an all-Wales programme that ensures consistency

of training provision. There was agreement that the focus on current and aspiring team

managers was appropriate and that investment in the skills of future team managers is just

as important as upskilling existing team managers.

Some stakeholders reported that the ongoing development of the CPEL Framework meant

“the rationale for having a dedicated TMDP is even more important now than before”. The

framework identifies two key career pathways for social workers, a practice and

management route. Their view was that the TMDP is important for the management route.

Stakeholders also consider that the TMDP fits well with existing provision. For example, it

was reported that some local authorities4 require their staff to complete an Institute of

Leadership and Management (ILM) Level 5 qualification prior to starting the TMDP. The

TMDP is considered to offer good progression and to build effectively on the principles

introduced in the ILM qualification. Stakeholders also reported that having a management

course specifically for social workers is crucial and addresses gaps in relation to case

management, for example, that more generic leadership and management courses do not

cover. Tutors’ reflections on participants’ lack of knowledge of policy context and drivers and

of management tools also suggest the aims and objectives of the programme are

appropriate. One stakeholder noted that an area for future development could be the

development of an accredited continuing professional development module.

This support for the rationale, aims and objectives of the TMDP among stakeholders is

closely linked to the success of the consultation exercise that took place in 2009 during the

development phase of the programme. Stakeholders who had been involved in

commissioning this exercise as well as those who participated in it, commented that the

process succeeded in engaging key stakeholders and helped to secure an early commitment

to the programme. They also considered that staff at IPC facilitated discussions effectively

and that IPC staff’s experience of social services was important. One stakeholder noted,

“this was one of the best consultation events I’ve been to. People felt real ownership. IPC

appreciated people’s experience, understood the demands of juggling social work tasks and

had a good understanding of social care”. One stakeholder commented though that the

4 It is not possible to ascertain from the evidence whether this practice takes place in local authorities in a

particular region of Wales.


13 10 April 2013

involvement of service users in designing the programme had been limited and that, in

general, service users’ involvement in the programme could be improved.

A few participants highlighted that they had very limited formal management training and

education since they qualified, in some cases over 15 years ago. They therefore welcomed

the opportunity to participate in the TMDP.

Those who had more recently undertaken management training generally felt that the TMDP

added value because it was focused on the Welsh context and also because it was focussed

on social workers. For example, one current team manager who had previously participated

in generic management training reported that when she first heard about the programme,

she: “hoped for once we might have some management training that is relevant to our role”.

Another felt that the “NVQs are less relevant because they are not focussed on improving

quality in social care”. Another team manager noted particular gaps in the Level 4

management qualification she had completed in relation to theories underpinning leadership

and management, guidance on how to run projects and HR issues. Participants also noted

that the TMDP differed from other management training in that it offers an opportunity to

come together with peers.

Many participants recognised that “team managers are thrown in at the deep end” without

the skills and experience they need to be a confident and effective team manager. One

participant felt that “it should be a requirement for anyone doing a management role. You

should have the qualification to become a manager rather than doing it in post”.

Line managers similarly acknowledged that many team managers have “moved up through

the ranks without necessarily having management training” and that the TMDP helped to

address this gap. Several considered the programme was particularly important for newly

appointed team managers. They welcomed the focus on quality improvement, particularly

case management and managing team performance and supported the broad aims and

objectives of the programme.

3.2 Profile of TMDP participants

3.2.1 Job titles and management responsibilities

The TMDP’s target group is current and aspiring social work team managers working in adult

or children’s services or team managers in other related professions. Management

information and interviews with delivery staff, line managers and participants suggest that

the intended target group has been effectively reached.

At the time of the interim evaluation, 181 participants had registered on the TMDP.

According to the programme’s MI, an additional 85 students have been recruited between

the end of February and the end of October 2012. This means that in total, 266 participants

have registered on the TMDP, which is around a third of the 750 participants the programme

has a long-term ambition of supporting.

The recorded job titles of participants are wide-ranging but can be clustered into six broad

categories of role: acting or assistant team managers; team managers; senior practitioners;

principal practitioners; service managers; and social workers5. Distribution of participants

across these roles is shown in Figure 3.1 below. Around half of participants are team

managers (47%) and approximately a quarter (23%) are senior practitioners. The proportion

5 Participants who do not have the specific job title of ‘Team manager’ and have been included in this category

are: Team Leader, Duty Manager, Manager, Family Support Manager, Practitioner Manager, Home Manager, Flying Start Parenting Coordinator, Project Manager, POVA Coordinator, QA Officer, PLO Court Care Proceedings Coordinator, On track Manager, Adult Protection Coordinator.

Participants who do not have the specific job title of ‘Service manager’ and have been included in this category are: Acting Operational Manager, Operational Manager, Locality Manager, Assistant Manager (Operations), and Occupational Therapy Manager

In addition, the ‘Senior Practitioner’ category includes ‘lead professionals’.


14 10 April 2013

of team managers and senior practitioners has remained roughly the same between the

interim and final evaluation reports.

Figure 3.1 Participants by categories of job role

Management information

In the interim evaluation report we noted that three occupational therapy team managers,

one community nurse team manager and one care home manager had participated in the

programme. During the research for the interim report, a few stakeholders and one member

of the delivery staff expressed concern that if further participants who are not within the

direct target group of social work team managers were accepted on the course, this would

dilute benefits for all participants. For these stakeholders, a key aspect of the added value of

the TMDP is that it is a bespoke programme for social work team managers and they

consider widening the target group will undermine the objective of providing tailored training.

Other stakeholders and delivery staff, however, considered that the mix of participants

worked well and was important in the context of providing integrated services.

Since then, an additional two occupational therapy professionals have been recruited to the

programme, but no additional community nurse managers or care home managers. This

suggests that concerns about the target group expanding have largely been addressed and

a focus has been maintained on the core target group.

TMDP tutors, interviewed during the interim stage of research, considered that the pre-

programme seminar has helped to ensure that those with relevant experience, as well as the

willingness to commit to the course, have participated. Similarly, all of the participants we

interviewed and their line managers considered that it was appropriate for them to have

taken part in the programme, because they were either current or aspiring team managers

who had not had similar training previously.

3.2.2 Other characteristics

At the final stage of the evaluation, according to the participant survey around four fifths of

participants are female (77%); 6% have a disability; two thirds are White Welsh (66%) and

nearly a quarter White English (23%); nearly half (45%) are between 45 and 54 years of age;

almost all (94%) are in full time employment; just under half work with children and young

people (47%); just under half (45%) with adults and 6% with both groups of service users.


15 10 April 2013

Figure 3.2 Ethnicity of TMDP participants

Welsh64%

English23%

Any other white background

9%

White and Black Caribbean

1%

Any other mixed/multiple

background (please specify)

1%

Indian1%

Any other ethnic group

1%

Participant survey

Figure 3.3 Age of TMDP participants

0

10

20

30

40

50

60

70

80

26-34 35-44 45-54 55-64 No response

Prefer not to say

Participant survey


16 10 April 2013

3.3 Delivery

3.3.1 Identification and recruitment of participants

Interviews with participants, line managers and delivery agents suggested that practice

differs between local authorities in relation to identifying and registering participants on the

programme. In some local authorities where there is strong senior manager commitment to

TMDP, a planned and strategic approach has been taken to identifying which members of

staff should undertake the programme. Workforce needs and skills gaps have been

considered in the round and factors, such as which staff were facing particular pressures

because of changes in their service areas, as well as their previous skills and experiences,

were used as the basis for selecting participants. Typically, in these cases, the decision to

take part in the programme was jointly made by participants, line managers, training

managers and senior managers. As a result, participants felt well-informed before they

started.

In other cases, participants reported that they were keen to do the course as a result of

internal communications (flyers, intranet notices, mention in team meetings) and approached

their line managers to request a place. They obtained further information from their line

managers and also discussed the course with their training managers. In a few cases,

participants reported that they had heard about the programme through colleagues who had

participated in the TMDP and subsequently recommended it.

In a few cases, participants reported that they were nominated by their employer and

registered on the programme without adequate information and involvement in the decision-

making process. Tutors confirmed this and reported that they noticed that some participants

arrived at the pre-programme seminar with little information about the course content and

format and the required time commitment. They considered that the pre-programme seminar

was important to address this and to ensure that all potential participants have clear

expectations about the course before they begin.

3.3.2 Regional delivery and local authority participation

At the time of the interim report, there had been ten TMDP courses delivered in four

locations: Cardiff (5 courses), North Wales (2 courses), Carmarthen (2 courses) and

Swansea (1 course). Since then, an additional three courses have been established. Two of

these are in South Wales (Swansea and Cardiff) and one is in North Wales.

As Table 3.1 shows, across the 13 courses, an average of 20 participants started each

course, ranging from 14 to 24. A total of 12 participants withdrew from the programme during

its delivery. An additional 13 participants signed up to take part in the TMDP but withdrew

before it began. There were also 24 participants who deferred to a later programme during

the process of studying (for a variety of reasons, including maternity leave, sickness or failing

modules).

Table 3.1 Number of participants per course

Course No. of starters

Course 1 – North Wales: May 2010 24

Course 2 – Cardiff: September 2010 20

Course 3 – Carmarthen; October 2010 21

Course 4 – Cardiff: December 2010 21

Course 5 – Swansea: January 2011 16

Course 6– Cardiff; April 2011 14

Course 7– Carmarthen: June 2011 20


17 10 April 2013

Course No. of starters

Course 8 –North Wales: September 2011 23

Course 9 –Cardiff: October 2011 20

Course 10 –Cardiff: December 2011 22

Course 11 – Swansea

April 2012

21

Course 12 –North Wales: September

2012

22

Course 13 – Cardiff: November 2012 22

Figure 3.4 below shows that there are representatives from each of the 22 local

authorities in Wales, although the number of participants per local authority varies.

Figure 3.4 Participants by local authority

At the interim evaluation stage, a comparison of the number of subsidised places allocated

to each local authority with the number of participants on the programme showed that:

■ Two local authorities (Vale of Glamorgan and Caerphilly) had used all of their allocated

places;

■ Six local authorities had ‘bought’ additional places at full cost (Blaenau Gwent,

Ceredigion, Merthyr, Pembrokeshire, Powys and Swansea); and

■ The remaining local authorities had not yet used all of their allocated subsidised place

and, in most of these cases, around a fifth of allocated places had not yet been used and

in five local authorities (Flintshire, Isle of Anglesey, Bridgend, Newport and Cardiff) over

40% of allocated places remained unused.

Since then, the five local authorities that had more than 40% of allocated places remaining

have all taken up additional places: Flintshire has now used all allocated places; Cardiff has

used all the allocated places and bought another at full cost; Bridgend and Newport have

only 10% of allocated places unused; and the Isle of Anglesey has around 30% of places

unused.

This means that overall:


18 10 April 2013

■ Almost three quarters of local authorities have used their allocation of subsidised places

and, of these, two thirds have gone on to purchase full cost places;

■ In the remaining quarter of local authorities (6), between 10% and 20% of the subsidised

places have not been taken up6.

In general, stakeholders, delivery staff, participants and their line managers agreed that a

regional delivery model that brought together participants from different local authorities was

appropriate. Although they recognised that this meant some participants had to spend time

travelling to taught days and supported days, they considered the benefits gained from

sharing learning with staff from other areas outweighed this potential inconvenience. One

participant noted, “It’s a far better way to learn than within the boundaries of your own local

authority. Having the chance to learn from other areas is important”. Another considered that

he would not have been able to be “as relaxed and reflective” if he had been solely with staff

from his local authority. Only a couple of participants disagreed and would have preferred

delivery to take place at local authority level. A few of the participants we interviewed

recommended that there should be more courses available in North Wales and mid-Wales to

cut down on their travel times.

3.3.3 Key programme components

3.3.3.1 Pre-programme seminar

Both the survey and interview data suggest that the pre-programme seminar was valued.

Just over three fifths of survey respondents (61%) rated the seminar as either very useful or

useful and nearly one fifth (18%) as moderately useful.

Most participants interviewed felt well-informed about what the course would entail and the

time commitment required after attending. Participants reported that it was important for line

managers to hear first- hand about the course and particularly to understand that it is an

intensive programme that requires significant time commitment. Participants also valued the

opportunity to meet others before the official start of the programme. This helped ease

nerves and was an enjoyable way to begin the programme. A few participants suggested

that the pre-programme seminar could have been improved with a shorter presentation. For

example one senior practitioner highlighted that “when they started talking about the quality

agenda and government policy, I was completely lost”. He considered that the hour-long

presentation could usefully have been condensed into a 20 minute overview of the

programme, with more detailed coverage of wider policy issues left for the first taught day.

The participant, line manager and delivery staff interviews suggested attendance by line

managers varied. In some cases, line managers did not attend the pre-programme seminar

because they had participated in the TMDP themselves. In these cases, participants tended

to still feel well-supported and did not view lack of attendance as a barrier. In other cases,

line managers did not attend the seminar because they had already been once for another

member of staff and so did not see the value in attending again. In a few cases participants

reported that their line managers had not attended because “they were not interested” or

“didn’t have the time”.

3.3.3.2 360 degree assessment and PDP

Over two thirds of survey respondents considered that the 360 degree assessment was very

useful (34%) or useful (35%) and a similar proportion found the PDP very useful (33%) or

useful (35%).

This was corroborated by the participants we interviewed who reported that the 360 degree

assessment and the subsequent development and use of the PDP had been beneficial and

for a few it was “quite enlightening and an absolutely essential start”. Participants

commented that the 360 degree exercise was helpful to identify personal development areas

and that the before and after exercise was useful for “understanding distance travelled’.

Some participants also suggested that this component of the programme had helped to

6 We have assumed that 209 subsidised places have been allocated to local authorities.


19 10 April 2013

increase their self-confidence because peers had given them higher ratings than the ratings

they had given themselves in the self-assessment. This was encouraging and offered a

boost to their confidence.

Although many participants agreed that the task of developing the PDP at the start of the

programme was useful, they reported using the PDP to differing extents throughout the

course. A few participants used the PDP as a key tool to shape and inform personal

development and reviewed and updated it on an ongoing basis throughout the programme.

For example, one participant reported that she had learnt that “for a PDP to be useful it must

be seen as a dynamic tool”, and she therefore reviewed it after each module. She

particularly valued the emphasis on linking the PDP to the work-based project undertaken for

the second module. Other participants reported that they revisited the PDP at the end of the

programme but used it very little in between. They suggested there could be greater

emphasis in the programme about the need to re-visit the PDP and that it could be

integrated more within other components of the programme. A few line managers noted that

they had supported participants to develop their PDPs and thought this was a useful

exercise.

Around a fifth of the participants interviewed reported they did not find the 360 degree

assessment useful. The main reasons provided for this were that they were relatively new in

post and so colleagues were not able to provide feedback on their performance or that

colleagues moved into different roles and so were not available for the ‘after’ assessment. In

other cases, participants felt that they did not work closely enough with colleagues for them

to be able to make an accurate assessment of their strengths and weaknesses.

TMDP tutors noted that several participants commented that they were not used to regular

appraisals at work and therefore the 360 degree assessment and PDP offered an

opportunity for constructive feedback that was generally valued. One tutor also noted that for

some participants, a barrier to completing the online 360 degree assessment was lack of

adequate ICT skills.

3.3.3.3 Taught days

Over four fifths of survey respondents found the taught days either very useful (55%) or

useful (27%).

Most participants we interviewed were extremely positive about the taught days. They valued

time away from the demands of the workplace to learn and reflect on their practice. They

found tutors were knowledgeable and had relevant professional experience, presentations

were generally interesting and different learning styles were taken account of through a

range of different types of activities. A few participants reported they found the taught days to

be the least valuable component of the course and suggested that less PowerPoint

presentations would make the taught days more engaging. They generally recognised that

this was their individual learning preference and felt that others had benefitted from and

enjoyed the taught days. It should also be noted that in response to feedback from

participants, delivery staff reported that they have changed the format of the taught days to

include fewer presentations.

The pace of teaching was also generally considered to be appropriate, although a few

participants reported that they found the first taught day of the first module confusing and

would have preferred more time to cover topics. They were being introduced quite quickly to

a range of new areas they knew little about and found this challenging and disorienting. For

example, one senior practitioner noted: “I was there in a room full of already established

team managers and I felt slightly alien to the discussion. I can liken it to being in a room full

of people speaking a foreign language”. He reported, though, adjusting to new terminology

and concepts relatively quickly: “The language became more familiar and I became more

fluent in it and was able to take part in discussions more”.

A few participants in earlier cohorts acknowledged that the programme was new and that

“inevitably tutors were working things out as they were going along a bit”. As such, they

identified small changes that could be made to improve the programme (for example, the

order in which topics are covered and some tweaks to learning materials). They reported that


20 10 April 2013

they had given feedback on these issues to tutors and understood that their experiences and

views informed subsequent changes to the programme. Interviews with delivery staff also

highlight that the early months of the programme in particular were crucial for “fine-tuning”

elements of the programme.

Most participants interviewed considered that two taught days per module is appropriate. In

recognition of the demands the TMDP places on participants, one stakeholder suggested

that the possibility of offering the course in a more unitised way that allows students to

spread learning over time so “it is more phased or chunked and therefore reduced the

burden” should be explored.

Interviews with participants and delivery staff and attendance records relating to cohorts 8-12

suggest that attendance levels at taught days are generally high although typically one or

two people do not attend each session.

A few stakeholders and members of delivery staff noted that the TMDP is currently not

delivered in Welsh, although participants can choose to submit assignments in Welsh. They

suggested that this might be a potential area for development.

3.3.3.4 Action learning sets

According to the survey, nearly four fifths of survey respondents found the action learning

sets either very useful (53%) or useful (25%).

Interviews with participants, line managers, delivery staff, colleagues and feedback

questionnaires collected throughout the programme indicate that the action learning sets

were for most participants a highlight of the programme. They were considered, alongside

the work-based project, to be one of the most useful components of the programme.

Participants valued the practical problem-solving approach and many commented that their

roles can be at times isolating, so they appreciated an opportunity for dialogue with peers.

Typical comments about the action learning sets include:

“They were excellent, it was so helpful to discuss issues with other managers in a safe

environment”

“I gained a lot from them. It was good to know there are other people who deal with

similar situations and to be able to learn from them”

“Often team managers work in isolation and the peer group within the local authority is

small- to be with others and to see how other local authorities manage is important”.

“For me it was about being able to test ideas in a safe environment with people that

understand where you’re coming from”.

Participants commented that having participants working in a wide range of service areas

was particularly helpful for the action learning sets and enriched the discussions. One line

manager noted that there are few other opportunities for staff to build professional

relationships with peers from other local authorities and that exchanging ideas through these

relationships is important.

A few participants, although they found the action learning sets helpful, considered that there

was too much time dedicated to them and found “analysing several problem situations” was

repetitive.

Around a quarter of participants suggested that continuing the action learning set approach

after the end of the course would be helpful and that meeting with peers to share learning,

experiences and problems would be a good form of post-programme support. Participants

recognised that lack of time was a potential barrier and were generally unsure about what

the format or regularity of this type of potential post-programme support should be.

3.3.3.5 Individual tutorials

Nearly three quarters of survey respondents (74%) reported that they had received individual

support or tutorials while undertaking the TMDP and attendance records suggests that the


21 10 April 2013

proportion was even higher. Of these, over four fifths found the support useful (38%) or very

useful (47%).

This was supported by the views expressed in interviews with participants. Most praised the

supportive and friendly approach of tutors in general and reported that the one-to-one

tutorials were an important component of the programme that contributed substantially to

learning. They found that feedback was “constructive and useful” and reported building

“strong professional relationships”. In addition to the dedicated face to face contact, many

participants reported that they had contacted their tutors for advice and guidance via email

and telephone. There was a widely-held view that tutors were “very available” and

responded promptly to requests for help. Those participants who had experienced difficulties

because of illness and other personal circumstances found tutors to be flexible and

responsive in supporting them to re-arrange submission dates for assignments. A few

interviewees considered that tutors could have been “more dynamic”, and “more proactive in

offering support” and did not find the one to one tutorials particular helpful.

3.3.3.6 Assignments

Nearly four fifths of survey respondents found the assignments either very useful (45%) or

useful (33%).

Although all participants interviewed commented that completing assignments alongside the

demands of their jobs was extremely challenging, many reported that they found the

assignments helpful and well-designed. They reported that the work-based project and final

presentation were of particular value in developing their understanding of quality and

management issues (“they really allowed me to reflect on my practice”).

Many participants had not undertaken intensive higher education learning for several years

and therefore struggled to get to grips with academic conventions such as referencing and

writing in an academic style. Nonetheless, they valued “having to go back to theory, learn

and read”.

A few participants commented that the overlapping deadlines (ie starting the next module

before completing the assignment for the previous module) was difficult and recommended

amending timings to avoid this. A few participants also suggested that it would be helpful to

provide more information about the work-based project that is undertaken for module 2

earlier in the programme so participants could plan ahead. Generally, though, participants

considered the timeframe for completing assignments was appropriate.

3.3.3.7 Study leave

Nearly half of survey respondents (46%) thought that they had the level of support needed or

considerable support from their employer in the form of study leave for assignments. Almost

a fifth considered that they had no support at all (4%) or little support (13%).


22 10 April 2013

Figure 3.5 Participant views on the support received from employers in relation to study leave

Please indicate how much support for the following you have received from your employer

during the course: study leave to do the assignment

A little support

13%

BLANK6%

Considerable support

18%

No support at all4%

Some support31%

The level of support needed

28%

Participant survey

Participants interviewed valued the allocation of one study day per module and that the local

authority’s obligation to offer them was formalised in the tri-partite agreement. However,

around a third of participants reported that they were not able to take the days. In most

cases this was because participants themselves felt they could not afford the time. One

participant noted: “if you take time off, you just have more to do when you come back”.

Unsurprisingly, those participants who were able to use their allocated study days

highlighted the need to schedule the days well in advance.

A few participants interviewed highlighted that their local authorities were very supportive of

the programme and noted that they were allocated additional study days above the minimum

level and were supported to take them.

Most participants reported that the allocation of one day per module was not sufficient and

they had to undertake significant additional work in their own time at weekends and during

annual leave. This is to be expected given that the TMDP is a level 7 qualification that is

intended to stretch and challenge learners.

3.3.4 Programme content

The survey evidence suggests that most participants who have completed the TMDP found

the content of modules either very useful or useful. Around half of participants rated each

module as very useful (52% for modules 1 and 2 and 47% for module 3), while around a

third rated the content of each module as useful (34% for module 1 and 29% for modules 2

and 3). In addition, around half rated all modules as very useful or useful. The evidence also

suggests that modules were viewed as equally useful.


23 10 April 2013

Figure 3.6 Participant views on the usefulness of the content of modules

For each module you have completed, please give your opinion about how useful the

content of each module has been.

0.0

10.0

20.0

30.0

40.0

50.0

60.0

Module 1 Module 2 Module 3

Very useful

Useful

Moderately useful

Slightly useful

Not at all useful

Don't know

Participant survey

Most participants reported that they found it appropriate to begin with an overview of current

policy drivers and found this a helpful “framework of reference”. A few suggested an

increased focus on supervision and on the management of integrated service teams would

have been helpful.

One stakeholder also expressed the view that there is a need to review the programme to

ensure it reflects the policy shift towards the delivery of more integrated services (for

example through Team Around the Family and Integrated Family Support Services). “We

need to be careful we’re not preparing managers for yesterday’s world”.

3.3.5 Line manager involvement

Evidence from the survey suggests that participants consider their employers have not

provided the level of support required during the programme. Only a fifth reported that their

employers had provided the level of support needed or considerable support in the form of

advice and support with assignments. Nearly half of respondents thought that they had only

a little support (25%) or no support at all (22%) of this nature. Nearly two fifths of survey

respondents thought that they had only a little support (24%) or no support at all (14%) in

developing and reviewing their Personal Development Plan.

Support to attend taught days and action learning sets was far better, with four fifths (78%)

reporting the level of support needed or considerable support.

A few of the line managers interviewed welcomed the pre-programme seminar as a way of

being involved in the programme and would have liked more contact, perhaps in a similar

format, with the delivery staff throughout the course. Several of the line managers, for

example, suggested that a mid-term meeting between delivery staff, participants and line

managers would be helpful. One TMDP tutor also reflected that there is potential to develop

line manager involvement at the end of the course. S/he suggested that consideration could

be given to the delivery staff “handing back” to the line manager at the end of the programme

to support improved continuation and progression.

Participant and line manager interviews suggested that some line managers have used

individual supervisions to discuss the TMDP, helped with defining the work-based project

and offered more ad hoc, informal support and guidance. In line with the survey, the

interviews suggest though that other managers have been less engaged with the programme

and have provided more limited support.


24 10 April 2013

Figure 3.7 Usefulness of the learning contract

Please give your opinion about how useful different parts of the course have been to ensure

the training has been relevant to your job: learning contract

Participant survey

Over half (53%) of survey respondents considered that the learning contract between

participants, managers and the IPC has been very useful or useful. Participants, line

managers and delivery staff largely agreed that the tri-partite agreement is an important

feature of the programme and that it should be maintained as a component of the

programme in future. It was acknowledged though that the agreement is more symbolic than

functional in that supportive line managers would engage with the programme regardless of

whether an agreement is in place and similarly, even with the agreement in place, some line

managers have not engaged. Nonetheless, it is considered to formalise line manager

commitment and makes it easier for TMDP tutors to contact line managers if there are

perceived to be any organisational barriers to participation. Tutors reported that they have

contacted line managers, with participants’ consent, on a few occasions.

3.4 Management and partnership working

There was consensus among stakeholders and delivery staff that the programme’s steering

committee had functioned effectively and been one of the programme’s key success factors.

There was a view that it has successfully brought together stakeholders and developed a

sense of ownership and commitment among them. One stakeholder commented, “It’s owned

and endorsed by all local authorities it’s not just a working group”, while another noted, “it’s

an example of good collaboration and effective partnership working. The committee has

been a key success of the programme- they all bring a different set of skills”. Stakeholders

reported that attendance has been good and praised the committee’s chair.

The representative of the delivery agent who attends steering meetings also found the input

of the committee “enormously helpful”. As well as being able to draw on the expertise of

members, she reported that the meetings provided a useful opportunity to feedback

intelligence tutors were gaining from their teaching, such as the lack of consistent appraisal

processes for team managers in the sector.

Both delivery staff and stakeholders commented on the value of a member of the committee

attending the final celebration day of each cohort, which offered them first hand opportunity

to hear about participants’ experience of doing the programme. They also noted the effective

project management support provided by the SSIA.


25 10 April 2013

Most stakeholders agreed that the IPC-led consortium “has been an excellent partner” and

highlighted in particular the flexibility and responsiveness of staff. One stakeholder

commented “they have a knack of listening to what customers want and translating into

deliverables that really make a difference".

3.5 Fees

A few stakeholders considered that the SSIA subsidy has been crucial to the success of the

programme to date. Their view is that without the subsidy take up would have been far lower.


26 10 April 2013

3.6 Key messages

Development, rationale, aims and objectives

■ All stakeholders, participants, line managers and delivery staff believe the rationale underpinning

the TMDP is appropriate and continues to be valid.

■ Stakeholders consider the TMDP complements existing training provision, building effectively on

Level 4 and 5 qualifications on the Credit and Qualifications Framework for Wales.

■ The development phase of the TMDP was widely considered to have been carried out

effectively, with the right stakeholders consulted in a skilled and constructive manner. This has

helped to foster commitment to the programme over time.

■ One stakeholder commented though that the involvement of service users in designing the

programme had been limited and that, in general, service users’ involvement in the programme

could be improved

Target group

■ There is evidence that the intended target group has successfully been engaged in the

programme. During the interim research there were emerging concerns among a few

stakeholders that widening eligibility to include care home managers, professionals from other

backgrounds and non-operational managers is not appropriate. It appears that since then, the

focus has been on social work team managers, aspiring managers and occupational therapy

managers and that professionals from other professions and contexts have not been included.

Identifying participants

■ There has been differing practice across local authorities in identifying participants who would

benefit from the programme. Some local authorities have taken a strategic and inclusive

approach while in other local authorities, participants have been nominated or self-nominated in

a more ad hoc manner without adequate information about the requirements of the programme.

Regional delivery model

■ Most stakeholders, line managers and participants valued the regional delivery model because it

offers the opportunity to share learning across local authority boundaries.

■ The interim evaluation found that participation in the programme had varied across local

authorities. Since then, take up among local authorities appears to have become more

consistent. Almost three quarters of local authorities have used their allocation of subsidised

places and, of these, two thirds have gone on to purchase full cost places; and in the remaining

quarter of local authorities (6), between 10% and 20% of the subsidised places have not been

taken up .

Key components of the programme

■ Almost two thirds of participants (61%) consider the pre-programme seminar to be very useful or

useful and most interviewees reported that they felt well-informed after attending. Attendance of

line managers at the seminar has varied. Reasons for not attending include having completed

the programme themselves, lack of time, or having attended the seminar previously.

■ The 360 degree assessment has been a valued component of the programme, with around two

thirds of survey respondents reporting it to be very useful or useful. This is notwithstanding some

practical difficulties in relation to the same staff not being available to complete the ‘before’ and

‘after’ and colleagues not being able to comment on the skills of newly appointed managers.

■ Although many participants agreed that the task of developing the PDP at the start of the

programme was useful, they reported using the PDP to differing extents throughout the course.

Some participants, for example reported that they revisited the PDP at the end of the programme

but used it very little in between.

■ Taught days and action learning sets are considered to be very useful components of the

programme. Most participants found the content appropriate, tutors helpful and approachable

and many valued the opportunity to share problems and experiences with peers from other local

authorities.

■ Around a quarter of participants interviewed suggested that continuing the action learning set

approach after the end of the course would be helpful and that meeting with peers to share

learning, experiences and problems would be a good form of post-programme support.

Participants recognised that lack of time was a potential barrier and were generally unsure about

what the format or regularity of this type of potential post-programme support should be.

■ Nearly three quarters of survey respondents (74%) received individual mentoring or support

while undertaking the TMDP. Of these, over four fifths found the support useful (38%) or very

useful (47%). Although interviews with participants highlighted the challenge of completing

assignments alongside work responsibilities, the interviews and survey suggest that


27 10 April 2013

assignments were also highly valued by participants, with around half (45%) stating they were

very useful and a third (33%) that they were useful. A few participants recommended that timings

should be reviewed so the assignment for one module is completed before starting the next

module.

■ Most participants have found that the allocation of one study day per module is not sufficient and

have used annual leave and weekends to undertake self-study and complete assignments. This

is to be expected for a Level 7 programme. A few local authorities have been very supportive

and allocated additional study days. In addition, around a third of participants interviewed

reported that they had been unable to use the allocated study days because of work pressures.

■ Stakeholders, delivery staff and participants considered the monitoring and review processes

that have been established to be responsive and effective in addressing the need for any

changes.

■ Although a few line managers appear to have had relatively high levels of engagement with the

programme, a few suggested it would be helpful to have a mid-term meeting with delivery staff

and participants.

■ Participants, delivery staff and line managers agree that the tri-partite learning contract is an

important feature of the programme because it formalises responsibilities and in the case that

one party does not comply, offers leverage in any subsequent negotiations.

■ Delivery staff considered that their quarterly review meetings as well as more informal review

processes had been crucial to support the development of the programme. Some participants

commented on the ample opportunity they had to provide feedback and to make suggestions for

improvements.

Management and partnership working ■ There was consensus among stakeholders and delivery staff that the steering committee had

functioned effectively and been one of the programme’s key success factors. Delivery staff and

stakeholders also commented on the effective project management support provided by the

SSIA.


28 10 April 2013

4 Outcomes and impacts

In this section, we begin by updating the survey findings so they take account of additional

responses received during the final round of research. We then present the findings of the

interim report and draw on follow-up interviews with participants from October-December

2012, in order to evaluate the extent to which outcomes reported last time have been

sustained over time and additional outcomes and longer term impacts have emerged.

We also present findings about what has enabled outcomes to be achieved and sustained

and what has acted as a barrier, and assess the additionality of outcomes.

4.1 Immediate and medium-term outcomes

4.1.1 Relevance

Over three fifths of survey respondents (61%) believed the programme had met their

objectives to a great extent or completely, while only 5% of respondents thought it had not

met their objectives at all or only a little. The same proportions reported that the course had

been completely relevant to the challenges they face in the workplace or relevant to a

considerable degree.

Around a third of participants interviewed noted that the programme was very relevant to the

current context in which they work which is characterised by significant changes to services

because of budget cuts and a move to integrated service provision. The emphasis on

change management was therefore considered to be particularly pertinent. Participants also

commented on the fact that the programme covered current policy and one participant

reported that “this really makes it stand out from other postgraduate courses”.

There is some emerging evidence to suggest that the course is less relevant to the needs of

those in youth offending teams (YOTs). Interviewees working in YOTs noted “that the course

does not really reflect what is happening in my area” and highlighted the need to take better

account of other service areas, particularly in the first module about policy context.

Nonetheless, the YOTs staff still reported that they gained benefits from other aspects of the

programme and were able to apply the management tools introduced in the programme.

4.1.2 Increased confidence

Interviews with participants, delivery staff, line managers, peers and supervisees suggested

that participants have increased confidence to manage and lead, as well as to seek support

if required, as a result of the programme.

Around a third of participants interviewed highlighted that they had gained confidence

through the course. For example, one participant commented that as a result of participating

in the TMDP “I am certainly more confident and assertive”. A few participants noted in

particular that they felt more confident in endeavouring to influence senior managers within

their organisations. For example, one participant reported: “I have gained confidence in

presenting evidence to my managers and making a case for something”, while another

stated “I am now confident in ‘selling’ ideas to senior managers across the local authority”.

Line managers too noticed a change in the confidence of their staff. For example, one line

manager commented: “having gained a merit on the programme, she is now more confident

in taking the lead, and also in delegating to members of her team. She appears to trust her

own management judgements more”.

Interviews with participants highlighted that the taught days and, in particular, the action

learning sets have contributed significantly to developing confidence. For example, one

participant noted “doing the course, listening to others about how they do things, has helped

me to be more assertive and confident”.

Analysis of before and after scores for the 360 degree assessment for participants who have

completed the course demonstrates an average increase from 4.6 to 5.6 in self-scoring,

which also suggested an increase in confidence and skills as a result of the TMDP.


29 10 April 2013

There is evidence that participants’ confidence has continued to influence management

practice beyond completion of the course:

■ One participant reported that he had been piloting a new model of preventative practice.

He reported that he had presented the results of his work to a board of senior managers

and key partners over the summer. The aim of the meeting was to explore whether the

local authority and partners should invest in the new model. He thought the presentation

went well and that this was attributable to the TMDP: “It made me realise I couldn’t just

stand up there with an idea and that I needed to have data and statistics. It’s made me

more savvy”.

■ Several participants (who had completed the course at the time of the first interview)

reported that they remained more confident to hold discussions with their team members

about issues that were creating conflict and disagreement. They attributed this to

learning from the TMDP.

Several participants commented that the opportunity to interact with peers, as well as

increasing confidence of many participants, had supported longer term professional

relationships that were likely to contribute to improved service development. Two participants

for example had met with peers from other local authorities after they completed the

programme. In one case, the meetings were relatively informal and were seen as an

opportunity to catch up on developments in each other’s teams and service areas. This was

valued as a form of ongoing peer support. In another case, after hearing about a particular

way of working in the action learning sets, a participant went to seek more information and

has subsequently implemented a similar model in her local authority. There is little evidence

though that these relationships are sustained over longer periods of time.

4.1.3 The national agenda for change

Around two thirds (63%) of survey respondents thought that their knowledge and

competency in relation to the national agenda for change in social work had increased (39%

considerably and 24% greatly). Around a third (31%) considered that as a result there was

an increased alignment between their work and the national policy agenda.

This was corroborated by interviews with participants and delivery staff. Almost two thirds of

the participants we interviewed highlighted this as an area in which they felt they had learnt a

lot and which was particularly relevant. In some cases, participants (and in particular senior

practitioners) had very little prior knowledge of the wider policy agenda. One participant

commented, “I didn’t realise how many Welsh Government publications there are. After

reading through all the research material, I felt we are far from delivering that”. In other

cases, participants reported that the programme had helped to consolidate and deepen

previous knowledge and understanding about the policy agenda and to give an “increased

focus”. For example, one participant noted: “Before this, I would read local authority

guidance, but would just go along without fully understanding, we weren’t really questioning”.

For another participant, “studying the national agenda has given me the opportunity to look

into things I don’t normally, it’s spurred me to look into things further”

For many participants, learning about the national agenda for change had informed their

work in that they felt that they now understood the “bigger picture” to which they contribute.

According to the survey, four fifths of participants (80%) considered the programme had led

to increased alignment between their work and the national policy agenda to some degree or

to a considerable degree.

Around a quarter of participants interviewed reported that they had shared learning about the

national policy agenda with their teams by presenting an overview of key policy documents

in team meetings or explaining important features of policies in individual supervision

sessions, and discussing the implications for their work. For example, one participant who

manages a multidisciplinary team of over 30 professionals including social workers,

occupational therapists, community nurses and building surveyors reported that the course


30 10 April 2013

had “reminded her to explain to someone why we are doing something”. She considers this

to be particular important because of the diverse range of professionals she oversees.

There is some evidence that participants have continued to ensure they and their teams are

well informed about current policy beyond completion of the course. Some participants have

continued to read key documents themselves and also to share them with their teams, “even

though we don’t have to anymore”:

■ One participant noted that national policy issues are “discussed on a regular basis” in

team meetings and that assessing how up to date the team members are on policy

initiatives has now been incorporated into annual appraisals.

■ Another participant stated that her team has merged with another team over the last six

months, leading to substantial change and re-organisation. She reported that she has

drawn on national policy changes to explain changes at the team level. She has found

that this approach allows her to root explanations about why changes are taking place to

staff roles in national policy which helps to reduce frustration and resistance among staff.

In general, line managers tended to report some limited impact in this area but did not notice

substantially increased alignment between their organisation’s work and the national policy

agenda.

4.1.4 Audit, inspection and review processes

Just over two fifths (41%) of respondents to the survey thought their knowledge and

competency in relation to the audit, inspection and review processes in Wales had improved

considerably (30%) or greatly (11%) while a further 37% thought that it had improved to

some extent. Only five percent of survey respondents thought that it had not improved at all.

Only a couple of participants interviewed highlighted audit, inspection and review processes

as a particular area of learning or an area where they had applied learning to make changes

to practice. This may reflect the cyclical nature of inspections. One participant whose

department was undergoing inspection commented that the TMDP has helped her “to

prepare and plan effectively”.

Line managers generally reported limited improvements to systems to plan for and manage

audit, inspection and review processes in Wales as a result of the TMDP.

4.1.5 Managing partnerships across disciplines

Around a third (32%) of survey respondents thought that their knowledge of managing

partnerships across disciplines had improved considerably (23%) or greatly (9%) and a

similar proportion that that the training had improved their management of partnerships. Ten

per cent of respondents thought that their knowledge in this area had not improved at all.

Interviews with participants indicated that the TMDP had had some impact on knowledge

and competency in relation to managing partnerships. A few participants highlighted this as

an important area of learning. For example, one participant responsible for implementing

more integrated and joined up service models within her local authority noted that the TMDP

had “given her an overview of different services necessary for making strategic decisions

about better joint working across departments and local authorities”.

There is emerging evidence that learning about effective partnership working has continued

to be applied after completion of the course and embedded in practice:

■ One participant considered that “working relationships have improved between us and

voluntary agencies” because she is now ensuring “social workers take it forward

themselves and are responsible for driving it forward”. She considers that the TMDP’s

focus on delegating and ensuring team members take responsibility for their work

informed her approach.

■ Another participant who works in children’s services noted that she has been building

relationships with the Youth Offending Service.


31 10 April 2013

■ A senior practitioner from one of the earlier cohorts, described developing a leaflet that

lists local voluntary and community sector organisations that social services can refer to.

He identified the need to build better links with the voluntary sector and felt this would be

a helpful resource to colleagues.

Line managers also recognised improved management of relationships with partners in other

disciplines as a result of staff’s participation in the TMDP, with some reporting considerable

or great impact in this area.

4.1.6 Improving processes for assessing and reviewing quality arrangements

A majority (58%) of survey respondents felt that their knowledge of how to review and

analyse quality arrangements had improved considerably (37%) or greatly (21%) and a

similar proportion (54%) agreed that this has led to improved processes for assessing and

reviewing quality arrangements.

This is supported by interviews with participants and their peers and supervisees, with

several examples provided of learning from the course leading to changes in quality review

arrangements:

■ A few participants who had completed the programme highlighted that they had

introduced new performance indicators for their teams. One participant manages a

fostering team (for which there are no national level indicators) and after doing the TMDP

she considered it necessary to introduce her own performance indicators for the team.

She reported that she had been working with colleagues within the local authority’s

central performance team to develop and implement these.

■ A few participants also reported that they had been drawing on their team’s views of

strengths and weaknesses of services more to assess and review quality. For example,

one participant facilitated a discussion in a team meeting based on a ‘SWOT’ analysis.

Several participants also reported that they were collecting more comprehensive data to help

them to make judgements about the quality of the service they provide:

■ One manager of a youth offending team who had been developing a new model of

preventative practice in partnership with schools had ensured that quantitative and

qualitative data was being gathered so that the effectiveness of the intervention could be

assessed. “I wouldn’t have even asked those questions prior to doing the course”.

■ Another participant commented that she had “been making more effective use of

performance indicators and ensuring qualitative and quantitative data was collected

more systematically” to assess progress against performance indicators.

■ Another manager whose team works with vulnerable families had introduced new

assessment tools that staff complete with families at the start and end of each support

session to reflect on progress and also to increase opportunities for service users to

provide feedback. A member of the participant’s team reported that “families feel really

empowered and we know if our service is working or not”.

■ A senior social worker in an out of hours service reported that she had initiated a process

to collect evidence about what the service does and its outcomes. “I’ve shown that I can

produce evidence to support the best model. It’s meant that I can offer suggestions to

senior managers and that’s all down to the TMDP”.

Participants who introduced new quality review arrangements as part of the work-based

project component of the course reported that they have maintained the new approach after

completion of the course and that the new processes are generally now well-embedded in

their team’s work.

Some line managers also highlighted that their staff’s participation in the TMDP had led to

improved processes for assessing and reviewing quality arrangements.


32 10 April 2013

4.1.7 Managing demand and capacity and managing casework

Evidence from the survey suggests that the TMDP is contributing to increased knowledge

and competency in relation to managing demand and capacity. Around half of survey

respondents (46%) reported that their knowledge and competency in this area had increased

considerably or greatly. A further third (37%) of participants reported that their knowledge of

managing case work had improved considerably or greatly. However, about a fifth (15%) of

participants reported there had been little or no increase in knowledge of how to manage

demand and capacity and a tenth that their knowledge of how to manage casework had not

improved. Just over two fifths of respondents (66%) who had completed the programme felt

that as a result of the TMDP they have improved management of demand and capacity to a

considerable (30%) or great degree (8%).

Again, interviews with participants corroborated findings from the survey. A few participants

interviewed reported that they had introduced or were developing new caseload

management processes.

■ One manager reported that she had introduced a traffic light system, under which

caseload numbers are graded as green, amber or red to highlight which members of

staff are available to take on new cases or may require additional support.

■ For his work-based project, one participant undertook a workload audit to inform

potential new workload management processes. He developed a template that members

of his team were required to complete which allowed him to gather key information about

the complexity of cases his team were working on. This had helped him to ensure case

allocation and distribution was more even and did not rely on a simple measure of

number of cases.

A few participants also highlighted that as a result of the TMDP they had established new

duty worker systems. For example, one senior practitioner reported that for her work-based

project she reviewed the duty officer system and collected information about the amount of

time being spent by social workers on relatively basic administrative tasks such as

administering small amounts of money to parents to use during their contact times. On the

basis of the information collected the participant and her line manager were able to identify

that resources were not being used efficiently. They have therefore made changes to the

process so that an administrator supports the team, “which has really helped to free up the

time of social workers”.

Follow up interviews of participants have indicated that many participants who introduced

new case allocation and management systems while on the course consider they are now

well established and reported that they have continued to use them. Those who were

considering introducing new systems of allocating cases have generally not done so, mainly

because “other priorities have got in the way”. This suggests that there is greater long term

impact on practice where change has been introduced through the work-based project.

4.1.8 Service user involvement

Around half of survey respondents (49%) reported their knowledge and competency in

relation to including service users in quality improvement has increased considerably or

greatly. Of those who have completed the programme, one third (40%) agreed that as a

result of changes to practice attributable to the TMDP, there is improved involvement of

service users in quality improvement projects to a considerable (20%) or great degree (13%)

The interviews with participants, peers and supervisees suggested that the TMDP had

influenced practice in this area, with around a quarter of interviewees describing work that

had been undertaken to involve service users in quality improvement projects:

■ One participant and her line manager described a consultation with disabled children and

young people in the local authority, which included collecting feedback and views via a

website. As a result of the consultation, a range of new information materials have been

developed and there may also be changes to delivery of services, for example

independent review officers allocated to each child. The participant who led this work


33 10 April 2013

reported that the programme has equipped her with the skills she needed to take this

work forward.

■ Another participant, who is a senior practitioner in a looked after children’s team,

reported: “If I hadn’t done the course, I’d have been more superficial about how I go

about service user involvement”. She described her work-based project for which she

carried out in-depth interviews with service users about how the recent addition of a

contact supervisor to the team would affect service users. She felt this went “beyond a

tick box exercise for once”, and “without the course I wouldn’t have understood how

useful this type of consultation can be”. She felt that because of the course she carried

out the consultation in a more formal and professional manner (“not just a chat”) than

she might otherwise have. It generated recommendations from parents which she

shared with her manager. These were about the room where contact meetings take

place and the wish for a kitchen so parents could eat and drink with their children. Since

the consultation, contact meetings take place at a different venue with these

arrangements. The participant noted that the in-depth consultation would not have

happened previously. The participant is keen to maintain high quality service user

involvement: “Now I’ve seen it in practice, I really believe in it. It is important and can

help. It’s another new and external view”.

■ Another participant described working with a service user group to improve the way that

staff record information about service users. She shared anonymous case notes with a

group of service users, who “provided very negative feedback about the poor quality of

the notes”. This exercise in part led her to initiate a project to rationalise the information

collection tools used to record information on service users and to facilitate access

between both social service and health professionals. She is confident the new system

she introduced is having a direct impact on the standard of care by reducing

inaccuracies. The same participant also described collecting feedback from service

users about their experiences of the current care pathway on discharge from hospital.

The exercise suggested that many service users find the process disorienting. The

TMDP participant is therefore planning to redesign the care pathway for hospital

discharge so that all users are discharged to a single intermediate care provider who will

then set up a care plan for them.

Line managers tended to notice a slight improvement in relation to involving service users in

quality improvement projects.

A few participants reported that they had continued to improve service user participation in

their services after completing the programme and had taken forward work-based projects

they had started on the course. For example, a participant from one of the earlier cohorts

who had started to develop a website for service users reported that she has continued to

progress the site and its content. She had, for example, led an exercise to film young people

in care about their views on the service and what they required from it so their contributions

could be posted on the website.

4.1.9 Managing performance of individuals

There is evidence to suggest that the TMDP is supporting improved knowledge and

competency in relation to managing the performance of individuals in teams. Nearly half of

all survey respondents (45%) reported that they considered their knowledge and

competency in this area had increased considerably (30%) or greatly(15%). A similar

proportion of participants who have completed the programme reported that changes in

practice as a result of the TMDP have led to improved management of the performance of

individuals within the team to a considerable or great degree (42%). Eight percent of

respondents felt that there had been no improvement in their knowledge in this area at all.

Around a quarter of participants we interviewed stated that they felt better equipped to

manage performance as a result of participating in the TMDP. For example:

■ One participant reported that “my individual practice in terms of managing capability

issues has probably significantly changed. I am able to react sooner to people failing in

their role”.


34 10 April 2013

■ Another participant reported that she was more able to challenge and address

underperformance: “I’m a bit more assertive now. Before, I would try and be more

supportive- when you come from a social work background, that’s your instinct. But, now

I realise I can’t always be that. I feel much more confident to challenge people”. She

provided an example of supporting a member of her team who “has capability issues”,

who “could come across as a bit intimidating” and who prior to the doing the course she

“struggled with”. Since doing the course she has realised that “there are policies and

procedures you can use to help you”. As a result, she is currently taking the member of

staff through capability processes. He is responding well to this.

Other participants have made changes to the way they run their supervision sessions and

team meetings in order to manage the performance of individuals more effectively:

■ One participant reported that the supervision sessions she leads are more structured as

a result of the TMDP and that she also uses problem solving approaches that were

developed during the action learning sets.

■ Another participant reported that he “adapted and changed our team meetings based on

the experiences of others” to allow greater involvement of partners from other disciplines.

Previously, team meetings were a forum for sharing information about changes and

processes. Now, he has shortened the meetings by sharing this information via email

and allowing people to read in their own time. “I’m giving people more autonomy and not

spoon feeding them”. He considers that having shorter meetings is an improvement.

With the time saved he has been able to arrange for partners to come and do

presentations on their work, which aids partnership working. For example, colleagues

from other social work teams, safeguarding teams and mental health have attended

team meetings. He considers that this is “opening up his team’s understanding of other

roles and services. It enables people to reflect on how we work with other agencies and

has created openness to other agencies”.

In general, changes made by participants in relation to managing the performance of

individuals have been sustained. One participant from one of the earlier cohorts reported that

“supervisions have continued to take place in a more structured way and that now everyone

knows more what is expected from them”. She noted that her approach is underpinned by an

understanding that the performance of individuals needs to be managed in relation to the

KPIs of the organisation.

Some line managers also reported that staff who had participated in the TMPD

demonstrated improved management of the performance of individuals within the team.

4.1.10 Management tools

Over half of survey respondents (55%) reported that their knowledge and competency in

relation to understanding the relative benefits of different management tools has increased

considerably (33%) or greatly (22%). The survey suggests this learning has been applied,

with over a third of participants (42%) indicating that the TMDP has led to increased use of

different management tools to some extent, and a similar proportion (41%) reporting

increased use to a considerable or great degree. However, almost a fifth of participants

(17%) reported that their knowledge and competency in this area had increased not at all or

a little.

Participants mentioned a range of management tools that they have applied in their roles,

including: the fish bone diagram (a tool that supports an analysis of the cause and effect of

problems); development days (a forum for teams to identify and analyse issues that affect

their work); and the use of problem solving techniques in team meetings. For example,

around a third of participants noted that they had introduced an approach similar to action

learning sets in their own team meetings or intend to do so because they found the

opportunity to analyse problems and difficult situations with peers so helpful. One participant

who was part-way through the programme noted: “I have found that there is a blame culture

in my local authority, so I am going to use action learning sets to say ‘what can we do

differently?’. Other participants reported that they had changed the structure of team

meetings so that time could be dedicated to action learning sets on case work. They now set


35 10 April 2013

aside time for members of the team to discuss their cases and seek the views of peers. Both

participants, their line managers and supervisees reported that this had been useful and

improved the quality and efficiency of their work.

Some participants also highlighted that they had been able to manage their own workloads

and time more effectively as a result of participating in the TMDP. A key aspect of this has

been improving their ability to delegate tasks and to facilitate others to solve problems:

■ One participant reported that before completing the TMDP, if she delegated a task to one

of her colleagues but found it had not been completed adequately, her “first instinct was

to say, ‘I’ll just do it myself”. However, when this happened recently, “because of the

course, I went back to the course materials and reflected. I realised that I could have

clarified more what needed to be done. I went back to her, went through it again, and

she did it well”. Another participant noted the programme had made him reflect on “how if

you’re going to ask something of someone, you need to put things into bitesize tasks and

be clear about how they will know if they have achieved what you intended”. He reported

that he would not have had these thought processes before doing the course and is now

striving to achieve this in his interactions with his team.

■ Another manager reported that she had changed her management style as a result of

the TMDP. She noted that if a member of staff comes to her with a problem, she no

longer “takes responsibility for all the solutions. I now turn it on its head and ask ‘what do

you think we should do’”. She considered this was effective in finding solutions and

thought the team valued that “she shows respect for their skills”. The change in approach

was noted by supervisees who commented: “she now puts it back to the staff and makes

people take responsibility. She asks them, ‘what do you think could be done to improve

this’”.

Some participants emphasised that they had applied new management tools within their

organisations as they planned and implemented changes to services:

■ One participant commented that as a result of changes in her local authority to deliver

integrated services at locality level, she will have greater line management

responsibilities as well as greater responsibility for service delivery. She reported that the

TMDP has provided her with the tools to plan resources and budgets in advance of the

change.

■ Another participant reported that before the TMDP he “relied primarily on a democratic

style of management”, which involved lengthy discussions with staff and extensive

consultation. He feels he is more directive now and that the TMDP has given him the

confidence to do this. Although workload pressures have meant there is less time for

lengthy discussion, he was clear that the TMDP has had an important role in changing

his behaviour.

Another participant reported that he had introduced learning sessions every month after

team meetings and feels that this was effective in offering a “learning opportunity without

threat”. For example, he had arranged sessions on strength-based interviewing. He

considered that he “wouldn’t have run the sessions if it weren’t for the TMDP” and although

he recognised that “maintaining momentum” would be a challenge because of pressures on

his team’s time, he intended to endeavour to continue to use them.

There is evidence that the use of different management tools has become well-embedded in

practice and that participants have continued to use different techniques in their work after

completion of the course. Some line managers too reported considerably increased use of

different management tools as a result of participation in the TMDP.

4.1.11 Use of research and evidence in practice

According to survey evidence, around half of participants (51%) reported that their

knowledge and competency in relation to using research and evidence to inform their work

had increased considerably or greatly. In addition, around a half (45%) of participants who

have completed the programme reported that the programme has led to improved use of

research to inform evidence to a considerable or great degree.


36 10 April 2013

There was limited evidence that participants are using research and evidence to a greater

extent in their work. Some noted that they occasionally share interesting pieces of research

in the team meetings and a few participants gave examples of using evidence about the

outcomes of their own services to develop their work (see ‘Improving processes for

assessing and reviewing quality arrangements’). Line managers also commented on

considerably improved use of research to inform practice.

4.1.12 Increased understanding of how to evaluate and reflect on personal competence

Over half of survey participants (52%) reported that their knowledge of how to evaluate and

reflect on their personal competences had increased considerably (29%) or greatly (21%).

Only 1% of respondents felt that there had been no improvement in their knowledge in this

area.

Greater reflection on personal competences, skills and practice was also an important theme

in interviews with some participants. One participant stated that “one of the main differences

the programme has made is asking myself questions” while another commented, “it’s given

me the opportunity to reflect on myself, the characteristics of me as a person. It’s been very

helpful to understand how I address and work with people”. Others reflected on specific

personal ways of working: “I have learnt on a personal level that I try and do everything. As a

manager you need to delegate more”. Another participant noted: “The course has made me

address gaps in my knowledge and skills. I now know where I want to be”.

4.1.13 Sharing learning with colleagues

As illustrated in Figure 4.1, the most popular method of disseminating learning from the

TDMP is through informal discussions with around 70% of survey respondents undertaking

this sort of dissemination. Over a third of respondents (37%) said that they distributed

learning materials to colleagues. Less than one tenth of respondents (8%) said that they did

not undertake any dissemination activities.

Figure 4.1 How participants disseminated learning to colleagues

0

20

40

60

80

100

120

Nothing Dedicated meetings/briefings

Dedicated presentations

during other meetings

Produced notes/reports

Informal discussions

Distribution of learning materials

received

Evidence from interviews with line managers and participants suggested that some local

authorities regularly bring team managers from across services together. Where this forum

exists, TMDP participants had been able to share ideas and learning with other team

managers who had not yet completed the TMDP and this was considered to have been

beneficial.


37 10 April 2013

4.1.14 Increased commitment to continuing professional development

Around two thirds (67%) of respondents who had completed the programme indicated that

they felt more committed to continuing professional development to a great or considerable

degree and very few (just 4%) reported no or just a slight increase in their commitment to

CPD.

TMDP tutors also reported that some participants had raised with them the possibility of

continuing to study to achieve a full Masters qualification or had enquired about other

professional development opportunities.

A couple of participants reported that they had started investigating Masters courses and the

possibility of doing additional courses to convert their postgraduate certificate into a full

Masters qualification. One participant noted, “there’s no way I’d have thought about doing

this before the TMDP”. Most participants noted that they have always been committed to the

principle of CPD, which is why they opted to do the TMDP. Therefore, the TMDP has

reinforced their existing commitment to CPD but has not necessarily increased it.

4.1.15 Employers value training and recognise increased capacity to manage change and improve quality

Interviews with line managers suggested that employers valued the training and that they

recognised increased capacity to manage change and improve quality among participants as

a result. All line managers that we interviewed noticed an increase in staff skills and

confidence, some reported the introduction of new management systems and processes that

they attributed to the TMDP, and some mentioned a change in management style among

participants. For example:

■ One line manager highlighted how a participant handled a “difficult member of staff”

differently as a result of the TMDP. “She came away with ideas from the course and so

handled and managed the person differently”. In her view, this change of approach

would have been unlikely to have taken place without the TMDP.

■ Another manager who supervises three TMDP participants reported that the programme

had exceeded her expectations in terms of the understanding, knowledge, attitudes,

skills and competences gained by participants. All three of these participants had been

promoted since completing the programme and the line manager noted that during their

job interviews they were all able to illustrate their learning from the programme. As well

as increased confidence she has noticed improved performance monitoring of their own

teams and improved ability to critically reflect on decisions.

■ One manager felt his supervisee had managed a merger of two teams more effectively

than she would have previously as a result of the TMDP. He described that she

organised information sessions and away days for staff. On these days, she “has let

them lead the sessions, because she thinks the impetus for change has to come from

them rather than me or her telling people what changes will happen. The team have

come up with an action plan”. This approach has worked well in his view and has meant

the team have been more productive because they “have spent last time whinging about

the changes and actually getting on and doing the job”. He noted that a few years

previously, she had managed an internal change very differently and “in a much more

top down way” which had meant that the change had been met with greater resistance

from staff and therefore it had been more difficult to implement.

A few line managers reported that they already considered the TMDP had been beneficial for

teams as well as individuals because improved skills meant they were able to better support

colleagues. For example, one line manager reflected that his member of staff has improved

capacity to make decisions for the benefit of the team and the organisation: “Before, she

would take actions in relation to an individual’s circumstances. Now, she thinks about the

impact on the team and the organisation more widely”. Other managers expect to see

impacts on other team members in the longer term.


38 10 April 2013

A TMDP tutor commented that a local authority senior manager who is several steps

removed from front-line managers had commented that he had started to see changes to

management processes as result of his colleagues’ participation in the course.

4.1.16 Impact of training on team members

Figure 4.2 To what extent has the training changed your team’s work

Not at all6%

A little13%

To some degree

53%

A great deal25%

Completely changed

3%

Figure 4.2 shows that over half of those survey respondents who had completed the

programme thought that the training had changed their team’s work to some degree (53%),

while almost one fifth thought that it had not changed their team’s work at all (6%) or very

little (13%). Over one quarter thought that it had changed their team’s work a great deal

(25%) or completely (3%).

Staff supervised by TMDP participants feel more motivated as a result of their team

manager’s participation in the programme and cited several changes in their teams:

■ One supervisee noted that morale in her team had been very low previously and that her

manager had used different tools to address this. She reported that team members had

greater opportunity to express their views and to contribute to shaping change, which

improved morale.

■ Another supervisee reported that “team spirit had improved” and attributed this to the

TMDP.

■ Another social worker noted that their manager was delegating tasks more than before,

which made staff “feel freer” and more likely “to take on responsibility and think for

themselves”.

■ One supervised reported that “team meetings now include circle time” which allows team

members to talk about how they are feeling about their work, as well as to talk about

more practical concerns. Supervisions have also changed to include a greater focus on

pastoral issues and the welfare of team members. As a result of these changes, the

supervisee reported feeling more supported and suggested that there has been reduced

conflict among team members because of the greater opportunity to talk and resolve

issues. This is likely to have had a positive impact on the efficiency of the team.

4.2 Longer term outcomes

4.2.1 Improved efficiency and effectiveness of services

Just under one third of survey respondents who had completed the TMDP thought that the

training had a considerable (28%) or great (4%) impact on an improved efficiency and

effectiveness of the services they manage and around a quarter considered the training had

either a considerable (18%) or great impact (6%) on developing services that offer value for

money.


39 10 April 2013

There was no further evidence from interviews with participants, line managers or

stakeholders that the TMDP has achieved intended longer term outcomes of improved

efficiency and effectiveness of services or fewer management resources elsewhere being

spent resolving problems in services managed by participants. It is not clear whether this is

because the outcomes have not been achieved or because of the limited number of

interviews we have been able to conduct with line managers.

4.2.2 Greater career progression

There is emerging evidence that the TMDP is supporting greater career progression and, in

particular, providing an improved transition for aspiring managers.

One third of survey respondents who had completed the TMDP thought that the training had

either a considerable (23%) or great impact (10%) on improved career progression for team

managers. Nearly one quarter (23%) thought that the training had no impact.

According to the survey, a third of participants (33%) have changed jobs since starting the

TMDP and most have gone on to higher level roles. Those that were not previously team

managers have either moved into team manager or acting team manager jobs. Those that

were already team managers have moved on to principal manager roles, service manager

roles, moved from ‘acting team manager roles’ to permanent roles, or taken on strategic

management responsibilities. In a few cases, the new role was a move across into another

team manager role.

Interviews with participants also indicated improved career progression:

■ Two of the participants that we interviewed had been promoted during their time on the

TMDP from senior practitioner to team manager. Both commented that being able to

draw on their learning from the TMDP had helped significantly during the interview.

■ One participant had applied for a team manager role before starting the TMDP but had

been unsuccessful in his application. The feedback he received highlighted the need for

greater awareness and knowledge of the strategic aims of the local authorities and the

national policy context. As a result of this, he was selected as a candidate for the TMDP.

Half way through the course he secured a temporary team manager position and shortly

after completing it he was appointed permanent team manager. He considered that “the

TMDP gave me the confidence and the vocabulary and knowledge I needed” to be able

to address the broader strategic agenda in his interview.

A couple of participants expressed frustration that despite completing the course, gaining

new skills and competences and feeling as though they had applied these to become more

effective managers, there were few opportunities in the sector in the current climate to

progress.

4.2.3 Greater satisfaction among service users

Just under one fifth of survey respondents who had completed the TMDP thought that the

training had a considerable (18%) impact satisfaction among service users. One fifth of

respondents did not know (10%) or thought that the programme had no impact (10%) on

service users. The fact that there is evidence of learning in relation to how to involve service

users in quality improvement projects and examples of applying this learning from participant

and line manager interviews, suggests that longer term outcomes in relation to greater

satisfaction among service users may be achieved. The interviews with participants however

did not provide any concrete examples or evidence of this.

4.3 Enablers

Interviews with participants, line managers and delivery staff highlighted that the following

factors enable the translation of learning from the programme into improved practice:

■ Engagement of senior managers. Interviews with participants and line managers

suggest that if line managers have participated in the programme themselves, this

makes it easier for participants to apply their learning and improve quality within their


40 10 April 2013

teams. One line manager we interviewed reported that she has been able to have very

constructive discussions and “interesting debates in supervision sessions” with the

members of her team who are currently participating and she attributed this largely to the

fact that she had covered the material herself as a TMDP participant. In addition,

because she understands the time commitment and also the benefits the programme

brings she ensures that her staff are able to attend taught days, action learning sets and

take study leave. She noted that even though her line manager was supportive, “I was

called out of taught days- it’s helpful if your manager has done the course”. A TMDP

tutor also commented “students often to say to me, I wish my manager could do this

course” and this was also mentioned by many of the participants we interviewed. For

example, one participant stated: “They should get senior managers to do the course,

when the hierarchy above hasn’t done it, it’s very hard”.

■ Internal forums for team managers. There is emerging evidence that if team managers

have the opportunity to meet with other team managers within their service areas,

learning and the application of learning is easier. One participant, for example, noted: “I

am involved in team manager meetings at division level and also at locality level, which

involve myself and between three and five other team managers. We have been able to

discuss management issues and strategic aims and raise ideas from the TMDP. The

TMDP has highlighted the importance of communication and these meetings are very

important”.

In a few cases, participants reported that their plans to make improvements to quality and to

modify management arrangements have “fallen by the wayside”. In some instances,

participants reported that staff shortages were the main reasons for this. In a few cases, line

managers’ lack of understanding about what participants were trying to achieve was cited as

a barrier. This suggests that for outcomes to be sustained over time, senior managers’ need

to receive further training.

As more team managers complete the programme and progress into senior manager

positions this should enable future team managers to translate learning into improved

practice and to continue to make positive changes over time. The SSIA and partners are

also looking to draw on the experience of the TMDP and evaluation to start scoping a

complementary programme for senior managers, as required under Sustainable Social

Services.

4.4 Assessment of outcomes for different groups

Evidence from the quantitative and qualitative research suggests that aspiring and current

team managers experience similar outcomes and that the programme is equally appropriate

and relevant to both these groups.

The survey evidence indicates similar increases in knowledge and understanding and

application of learning among both new and more experienced managers. Evidence from the

interviews with line managers and participants supports this and suggests that for those who

have been in managerial roles for a longer time, the programme consolidates and refreshes

knowledge. This is exemplified by comments from experienced managers such as:

■ “The course affirmed my existing skills and knowledge. I think I’m not a bad manager –

it’s given me acknowledgement- I’m already doing those things and thinking in that way. I

feel more confident now”.

■ “The course has really re-energised me”

■ “It’s consolidated what I knew”.

Importantly, it has also supported more experienced managers who have typically had little

opportunity to interact with peers to become part of a professional community of practice.

This has been important for building confidence.

For newer managers, the programme has been successful in promoting understanding about

the skills the role of team manager requires and supporting managers to use and apply

these skills to improve their services.


41 10 April 2013

Although participants working in YOTs highlighted that they felt that aspects of the first

module that dealt with the policy agenda were less relevant to their service area, evidence

from interviews and the survey suggests that these staff have still experienced many positive

outcomes as a result of the TMDP. There was also evidence that participants from

occupational therapy backgrounds experience similar outcomes to those from a social work

background.

There is no evidence that any one group (current or aspiring managers/ new or more

experienced managers) is more likely than the other to continue to apply learning over time

and achieve sustained changes to professional practice.

4.5 Additionality of outcomes

In most cases participants considered that the changes to management practice they

reported were entirely or largely attributable to the TMDP. Some participants considered that

over time some of the outcomes they reported (relating to improved processes for assessing

and reviewing quality, managing demand and capacity and service user involvement, for

example) would have been achieved even if they had not taken part in the TMDP. However,

they consider that as a result of the TMDP, these have happened much more quickly. A few

participants, including those working in youth offending teams, reported that other training

courses had influenced their understanding, knowledge and behaviour as well as the TMDP.

They attributed changes in practice to both sources of learning. None of the participants we

interviewed reported that the changes to management practice undertaken as a result of the

TMDP have replaced any other actions they used to undertake and that they consider good

practice.

4.6 Key messages

Immediate and medium term outcomes

The survey and interviews demonstrate that the majority of participants have achieved many of the

expected immediate and medium term outcomes of the TMDP:

All of the participants interviewed have applied learning to effect change in at least one area of their

management practice as a result of the course.

In most cases, changes initiated through the TMDP’s work-based project, have been sustained over

time.

In around half of cases, there is evidence that after completing the course participants have made

additional changes to the way they manage their team beyond the requirements of the work-based

project component of the course. In particular, there is evidence of sustained change in relation to

ensuring the national policy agenda shapes the team’s work, improving service user involvement,

managing the performance of individuals and using different management tools. The two areas in

which there is less evidence of sustained changes to practice are processes for audit and inspection

and the use of research and evidence to shape work.

■ Two thirds (61%) have found the course relevant and meeting their objectives to a great or

considerable extent.

■ Around a third of participants interviewed have increased confidence to act as a manager, many

pointing to the action learning sets in enabling this as well as the tools and projects. There is

evidence that this confidence to act as a manager and leader has remained after completion of

the course and continued to influence managers’ behaviour.

■ Around two thirds (63%) consider their knowledge and understanding of the national agenda of

change in social work has increased to a great or considerable extent with many interviewees

able to provide evidence of the value of this to their work.

■ Just over two fifths (41%) consider their knowledge and competency in audit, review and

inspection processes has increased to a great or considerable extent though relatively few have

indicated use in their work which may reflect the cyclical nature of inspection.

■ Around a third of participants (32%) consider their knowledge and competency of managing

partnerships has increased to a great or considerable extent with several interviewees applying

this learning to partnership working.

■ Nearly a half (46%) consider their knowledge and competency in managing demand and

capacity and managing casework has increased to a great or considerable extent with around a


42 10 April 2013

fifth of interviewees having introduced a change to practice.

■ Around half (49%) consider their knowledge and competency in service user involvement in

quality improvement has increased to a great or considerable extent with around a quarter of

interviewees doing this to improve service user experiences.

■ Nearly a half (45%) consider their knowledge and competency in managing the performance of

individuals has increased to a great or considerable extent with many interviewees feeling they

are better equipped and some providing examples of tackling performance.

■ Over half (55%) consider their knowledge and competency in different management tools has

increased to a great or considerable extent with some interviewees able to give examples of

their use to improve their own and team’s performance.

■ Around half (51%) consider their knowledge and competency in research and evidence has

increased to a great or considerable extent though few have used this.

■ Both aspiring and existing team managers have found the course to be relevant and useful and

applicable in their jobs.

■ There is emerging evidence that staff supervised by TMDP participants feel more motivated as a

result of their team manager’s participation in the programme.

Longer term outcomes

■ As yet there is little evidence of impact on service efficiency and effectiveness though there is

evidence of this in many cases on a small scale within areas that the participants manage.

Other outcomes

■ Almost all participants have shared some learning and this has mostly taken place through in

informal meetings and discussions with colleagues.

■ Many line managers value the training that has been provided and can see the difference. Those

line managers with whom we were able to conduct follow up interviews considered that changes

to management practice had been sustained over time.

■ Many aspiring managers have found the course useful for their progression and promotion.

■ Several participants have been promoted.

Additionality

■ In most cases participants considered that the changes to management practice they reported

were entirely or largely attributable to the TMDP. There is no evidence that there has been

displacement of other management activity considered to be good practice as a result of the

TMDP.


43 10 April 2013

5 Benchmarking analysis

In this section we compare the TMDP to the Leading to Deliver programme, which was

funded by Scottish Government and delivered by the Taylor Clarke Partnership between

2003 and 2008.

In the table below we draw on information from an evaluation of the Leading to Deliver

programme7 , a telephone interview with the programme manager at the Taylor Clarke

Partnership and information from the SSIA, to summarise information about:

▪ Programme aims and objectivities

▪ Target groups (eligibility criteria)

▪ Allocation of postgraduate credits

▪ Delivery model

▪ Marketing and promotion activities

▪ Costs to local authorities

▪ Expenditure and outputs

▪ Benefits accruing to participants and their organisations

We then assess the relative achievements and cost effectiveness of the two programmes.

7 7 Evaluation of Leading to Deliver, York Consulting, 2008:

http://www.scotland.gov.uk/Resource/Doc/239960/0066214.pdf/1

http://www.scotland.gov.uk/Publications/2008/10/02133248/1

http://www.scotland.gov.uk/Publications/2008/10/02133248/1


44 10 April 2013

TMDP Leading to Deliver

Programme

aims and

objectives

Help managers develop an in-depth understanding of the key drivers

influencing social care policy and practice in Wales

Support the development of critical thinking skills to improve case

management decisions

Provide managers with a range of tools and techniques to improve the

quality of practice within their teams by building and sustaining positive

culture and professional behaviour

To build leadership skills and confidence

Develop skills in change management

Promote working across boundaries

Create strengths in strategic planning, implementation and performance

management

Target groups

(eligibility

criteria)

Frontline social work team managers and senior practitioners or those

who aspire to those positions.

In order to be eligible, participants must have a recognised social work

professional qualification and registration with the Care Council for Wales,

the General Social Care Council, the Scottish Social Services Council or

the Northern Ireland Social Care Council (or equivalent body for non-

social work staff where applicable)

Three years post-qualifying experience

The ability to study at postgraduate level.

First line and middle managers in local authority and voluntary sector

organisations

Level and

volume of credits

Post Graduate Certificate in Managing Practice Quality in Social Care (60

credits)

Postgraduate Certificate in Social Services Leadership (60 credits)

Delivery model Delivered by IPC, University of Glamorgan and Research in Practice

Regional courses – approximately 20 participants per cohort

Three modules completed over 12 months: 1) An Introduction to

Managing Practice Quality 2) Evidencing Performance and Quality 3)

Leading and Managing for Quality

Per module there are two taught days and three support days including

action learning sets, scenario exercises and individual support = 15

contact days in total

Pre-programme seminar for participants and their line managers

Delivered by Taylor Clarke Partnership Ltd plus external speakers.

Centralised delivery location (Edinburgh)

Five modules: 1) Changing to Lead 2) Leading and Influencing Change 3)

Leading Change – service delivery 4) Leading delivery through effective

relationships 5) Practice analysis

Each module taught over a three day residential course = 15 contact

days in total

Use of an online forum to support knowledge exchange in between the

residential weekends


45 10 April 2013

Before and after 360 degree assessment

Assignments: one written assignment; reflective work-based project; work-

based case study and presentation

Assignment: one assignment per module, with a work-based ‘change’

project for the final module.

Marketing and

promotion

activities

Promoted through training managers and directors of social services in

each of the local authorities.

Promoted principally via local authorities. The Scottish Leadership

Foundation had already compiled a list of heads of social services, heads

of voluntary and community sector organisations and government

departments. Letters went out to all of them asking them to nominate

candidates.

Costs to

purchasers

Course fees are £2750.

The costs of the programme are currently met jointly by participants’

sponsoring local authorities and the SSIA. In 2009-2010, the SSIA

contributed £1,250 per student for 103 students and in 2010-11, £1,000

per participant for 106 students, resulting in a total of 209 subsidised

places.

Fully funded by Scottish Government so no cost to either learners or

employers.

Plan for cohort 6 – which didn’t run – was to split the cost 50:50 with

employers so government would pay for training while employer would

pay for accreditation costs.

Take up

266 participants between May 2010 and October 2012 = approximately

106 participants per year

500 participants completed the programme between 2003 and 2008 = 100

participants per year

Expenditure

Development

Stakeholder consultation and development of an agreed specification for

the programme: £38,000:

Programme development costs after awarding of the contract to IPC:

£50,000

TOTAL DEVELOPMENT COSTS: £88,000

Delivery costs May 2010 – October 2012

Local authority payments: £486,250

SSIA payment to IPC for delivery and accreditation: £310,250

Staff time at SSIA: Year 1: £12,000; Year 2: £8350; Year 3: £7720 =

£28,070

Steering group and partner in-kind contributions:

Development

Information not available

Delivery costs Sept 2003-Sept 2008

Accreditation, training events, accommodation and materials: £450,000

per cohort + a one off cost of £5,000 for university validation

TOTAL DELIVERY COSTS: £2.25 million


46 10 April 2013

2 x LA directors- 7 days – estimated day rate of £450 = £6300

1 x training manager – 7 days – estimated day rate of £225 = £1575

CCW director– 2 days – estimated day rate of £320 = £640

CCW officer – 6 days – estimated day rate of £160 = £960

All Wales Training Managers, Principal Training Officer – estimated day

rate of £250 – 10 days = £2500

TOTAL DELIVERY COSTS: £836,545

Source: SSIA data

Outputs

91 completed participants at the end of July 2012 (over 26 months); with

another 175 participants part-way through the course.

Approximately 425 social work managers completed the qualification

certificate (over approximately 60 months)

Evidence of

benefits accrued

to individuals

and

organisations

The survey and interviews demonstrate that the majority of participants

have already achieved some if not many of the expected immediate and

medium term outcomes of the TMDP. This is generally supported by the

staff they work with and examples of knowledge and competencies gained

being applied in the workplace.

■ Over three fifths (61%) have found the course relevant and meeting

their objectives to a great or considerable extent.

■ Around a third of participants interviewed have increased confidence

to act as a manager many pointing to the action learning sets in

enabling this as well as the tools and projects.

■ Around two thirds (63%) consider their knowledge and understanding

of the national agenda of change in social work has increased to a

great or considerable extent with many interviewees able to provide

evidence of the value of this to their work.

■ Just over two fifths (41%) consider their knowledge and competency

in audit, review and inspection processes has increased to a great or

considerable extent though relatively few have indicated use in their

work which may reflect the cyclical nature of inspection.

■ Just under a third (32%) consider their knowledge and competency of

managing partnerships has increased to a great or considerable

extent with several interviewees applying this learning to partnership

■ 75% of participants and 61% of line managers found the course

relevant to social work services.

■ 62% of participants rated the course as better than other similar

courses undertaken previously.

■ The majority of participants agreed that attending the programme has

benefited both their employer (91%) and their work (90%)

■ Both participants and line managers reported that participants’ skills

to being ‘a leader in charge’ have improved a lot (57% and 53%

respectively).

■ Participants reported that taking part in the programme had affected

their working practice in the following ways: managing change more

effectively (21%), increased confidence (18%), being a more

confident manager (17%) improved strategic management (15%) and

increased ability to delegate 911%)

■ Over half of line managers (58%) and over four fifths of participants

(84%) agreed that the programme has had a lasting effect on working

behaviour.

■ Around three quarters of line managers (77%) reported that

employees had used the skills learned on the programme.

■ Less than half of participants (43%) considered that the programme

has given them a better appreciation of service user needs, although


47 10 April 2013

working.

■ Nearly a half (46%) consider their knowledge and competency in

managing demand and capacity has increased to a great or

considerable extent with around a fifth of interviewees having

introduced a change to practice.

■ Nearly half (49%) consider their knowledge and competency in

service user involvement in quality improvement has increased to a

great or considerable extent with around a quarter of interviewees

doing this to improve service user experiences.

■ Nearly a half (45%) consider their knowledge and competency in

managing the performance of individuals has increased to a great or

considerable extent with many interviewees feeling they are better

equipped and some providing examples of tackling performance.

■ Over half (55%) consider their knowledge and competency in different

management tools has increased to a great or considerable extent

with some interviewees able to give examples of their use to improve

their own and team’s performance.

■ Around a half (51%) consider their knowledge and competency in

research and evidence has increased to a great or considerable

extent though few have used this.

■ Many have indicated that they have adopted practices as a direct

result of the course.

■ Both aspiring and existing team managers have found the course to

be relevant and useful and applicable in their jobs.

■ There is emerging evidence that staff supervised by TMDP

participants feel more motivated as a result of their team manager’s

participation in the programme.

■ There is evidence that participants have continued to apply learning

after completing the course, with some examples of sustained

changes to practice over time

eight in ten (79%) agreed that their learning will provide sustainable

benefits for service users.

Source: A survey of participants an (survey sent to 385 participants; 196

respondents, response rate of 51%) and a survey of line managers

(survey sent to 123 line managers, 43 responses received, 35%)


48 10 April 2013

This table demonstrates that:

■ The TMDP and Leading to Deliver have similar aims and objectives, target groups and

level and volume of credits.

■ Although the delivery model adopted for Leading to Deliver (five three-day residential

weekends with online support in between), is significantly different to that of TMDP, the

number of face to face contact days is the same (15).

■ Both programmes have achieved positive outcomes, with evidence of increased

knowledge and understanding in key areas of management and leadership and evidence

that this has been applied in day to day professional practice.

■ Based on delivery costs alone8, the cost per participant for the TMDP is £3145 and the

cost per participant for Leading to Deliver is £5295.

This suggests that the TMDP’s regional delivery model is more cost effective than the national

residential training model adopted by Leading to Deliver and has comparable outcomes.

8 This includes estimates of in-kind contributions as set out in the table, excludes development costs and

assumes that all TMDP participants currently part-way through the course will complete and achieve the qualification.


49 10 April 2013

6 Conclusions and recommendations

In this section we draw together the evidence presented in the preceding sections to offer a

series of conclusions about:

■ Whether the rationale for the course is valid;

■ How effectively the intended target group has been reached;

■ Whether the course has been delivered appropriately and effectively;

■ Whether there is evidence of outcomes and impacts for participants and their

organisations, the additionality of outcomes and whether the course offers value for

money; and

■ What the potential for the programme is to deliver its expected achievements

We highlight key recommendations for the future of the course.

6.1 Is the rationale that underpins the TMDP valid?

There is significant evidence that the TMDP’s rationale is valid and that the programme continues to

be relevant. There is agreement among stakeholders, delivery staff, line managers and participants

that there is a clear need for a bespoke social work team manager programme because generic

management courses do not address key aspects of social work nor respond to specific

management issues faced by social care managers. Some participants reported that they had been

a team manager for some time without any formal management training. Stakeholders noted the

importance of the programme in light of ongoing developments to the CPEL framework that has

identified two key career pathways for social workers, a practice and a management route. The

TMDP is considered to be an important feature of the management route.

Our research suggests that an effective and well-run scoping and development phase that sought

the views of key stakeholders has been a key success factor in ensuring the programme’s content

addresses workforce development needs effectively. In addition, the approach of commissioning

design and delivery of the programme separately appears to have been appropriate and effective.

There is evidence that having a national programme has increased expectations about the

knowledge, skills and competences that team managers need and that the TMDP is coming to be

seen as a required qualification for all aspiring and current team managers and as a stepping stone

for further study.

As a consequence, we recommend that the course should continue to be offered in the future.

6.2 How effectively have local authorities identified eligible participants and has the intended target group been reached?

The evidence suggests that the intended target group of current social work team managers has

been effectively reached throughout Wales although there is also evidence that practice in identifying

and recruiting participants differs between local authority areas and that some participants are

inadequately informed about the requirements of the programme by their employer. As a

consequence, we recommend the continued use of the pre-programme seminar to inform

both participants and their line managers.

The target group has been extended to include aspiring managers and professionals from other

professional backgrounds that are related to social work, such as occupational therapy and

community nursing. The emphasis of national policy on delivering integrated services suggests that it

is appropriate to bring together practitioners from different service areas and there is evidence that

these individuals and their organisations are benefitting from the programme as much as those from

social work backgrounds. However, if the programme continues to expand its offer to these groups, it

will be important to assess whether it continues to meet their needs. The commissioners of the

programme are keen to closely monitor the programme as it is extended. As a consequence, we

recommend representatives from the relevant professional bodies should also be included in

governance arrangements. If the number of professionals from other backgrounds increases, it is

also likely to have implications for local authority training managers who will be responsible for

funding and allocating places.


50 10 April 2013

A care home manager was accepted onto the programme during the early stages of the programme

Given that that the roles and responsibilities of this position are significantly different to those of

social work team managers or occupational therapy team managers, it is not clear that this is

appropriate. Our interim evaluation recommended that the management committee should monitor

closely any further requests from local authorities to place care home managers on the programme

and since then, there have been no further care home managers accepted onto the course. We

recommend continued monitoring of requests from local authorities to place care home

managers on the programme.

There is no evidence to suggest that achievements and impacts for participants in different job types

are different. We have found that aspiring and current team managers experience similar outcomes

and that the programme is equally appropriate and relevant to both these groups. Similarly, the

programme brings benefits to new and more experienced team managers. For those who have been

in managerial roles for a longer time, the programme consolidates and refreshes knowledge, while

for newer managers, the programme has been successful in promoting understanding about the

skills team managers require and supporting managers to use and apply these skills to improve their

services and the performance of their teams. There is also evidence to suggest that participants from

occupational therapy backgrounds experience similar outcomes to professional from a social work

background. As a consequence, we recommend that the current mix of participants is

maintained.

6.3 Has the course been delivered appropriately and effectively?

The regional delivery model is effective and has contributed to learning that would not have taken

place if the programme had been delivered at local authority level. We recommend that it should,

therefore, be maintained.

The evidence demonstrates that the pre-programme seminar has been delivered effectively,

although difficulties in ensuring all line managers attend persist. As the programme continues, it will

become increasingly likely that line managers will have already attended for one participant and may

therefore be reluctant to attend the same seminar again. As a consequence, we recommend that

other ways of engaging and staying in touch with line managers are explored and

implemented.

The 360 degree assessment has been a valued component of the programme, with around two

thirds of survey respondents reporting it to be very useful or useful, even though there have been

some practical difficulties in relation to the same staff not being available to complete the ‘before’ and

‘after’ and colleagues not being able to comment on the skills of newly appointed managers. As a

consequence, we recommend the use of the 360 degree tool as a key component of the

programme should continue.

Use of the PDP appears to have been varied, with some participants reporting that although they

thought it was beneficial to develop the plan at the start of the programme, they had used it to a

limited extent subsequently. This may be in part because of a lack of support from line managers

(only a fifth of survey respondents indicated they had considerable support or the level of support

required to develop and review the PDP). As a consequence, we recommend the management

committee and delivery agent should consider how they can influence local authority senior

managers to ensure PDPs are used and appraised more regularly. They should also monitor

whether the recent introduction of the requirement for line managers to sign off PDPs

improves how they are used to support participants’ development.

Delivery of the taught days and action learning sets has also been appropriate and effective with four

fifths of survey respondents describing them as either very useful or useful. Most participants praised

the practical work-based nature of the programme and also valued the theoretical frameworks to

which they were introduced. A few aspiring team managers reported that they found the first teaching

days difficult because they were learning new terminology and concepts at a relatively fast pace. It is

to be expected that undertaking postgraduate study for the first time may pose challenges and that

aspiring managers take slightly longer to adapt. This is all an indication of the level of upskilling that

is taking place.

The action learning sets have been effective and for many participants a highlight of the programme.

There is evidence that TMDP tutors have effectively facilitated discussion between peers to support

critical examination of work-based problems. The range of backgrounds from which participants

come has enriched discussions and learning. Around a quarter of participants interviewed


51 10 April 2013

suggested that continuing the action learning set approach after the end of the course would be

helpful and that meeting with peers to share learning, experiences and problems would be a good

form of post-programme support. As a consequence, we recommend that the action learning

set approach should be maintained and the possibility of offering the opportunities for follow

up action learning sets post-completion should be explored.

Nearly three quarters of survey respondents (74%) received individual mentoring or support while

undertaking the TMDP. Of these, over four fifths found the support useful (38%) or very useful

(47%). Participants have highlighted that constructive feedback has been provided and that good

professional relationships have been built. They reported that tutors have been available and helpful.

As a consequence, we recommend the delivery agent should continue to emphasise that

tutors are available to provide additional guidance , should continue to monitor attendance

and amend student feedback forms to include a section on tutorials. The delivery agent

should also ensure that participants are aware that tutorials can take place over the telephone

to overcome any potential barriers related to travel times and distance.

The challenge of completing assignments alongside a pressurised and busy job is evident.

Nonetheless, participants have found the assignments helpful and, in particular, the assignments for

modules 2 and 3. A few participants noted that starting a module before submitting the assignment

for the previous models creates additional pressures. As a consequence, the assignments appear to

be working well and we recommend that assignments should be maintained in a similar format,

although the possibility of re-scheduling deadlines should be explored.

The tri-partite agreement sets out employer commitment to provide participants with a minimum of

one study day per module. However, according to the survey, around a fifth of participants have had

little or no support in relation to study leave and under a third (28%) considered they had the level of

support needed. In this context, the agreement is a useful tool for negotiating with employers who do

not ensure participants take their allocated study leave. Regardless of whether they were able to

take study leave or not, most participants interviewed reported that the allocation of one day was

insufficient and they had to use their own time to complete assignments. A few local authorities have

offered their staff additional study days and this appears to have helped participants. As a

consequence, we recommend that changing the tri-partite agreement to include a commitment

to two study days should be considered. In addition, we recommend that the SSIA and delivery

agent should continue their efforts to build strong relationships with local authority training

managers to ensure that they act as advocates for the TMDP and participants and support

them to take their allocated study leave.

There is evidence that the content of the programme is broadly appropriate and relevant, with around

half of participants rating each module as very useful and a third rating the content of each module

as useful. However, interviews with participants working in YOTs suggested that the first module

about policy context is less applicable to their service area. As a consequence, we recommend

that when teaching programme to staff working in YOTS, efforts should be made to ensure

greater relevance.

Stakeholders and delivery staff recognise the need to continually update content so it reflects current

policies and practice and there is evidence that this has been happening so far. One stakeholder and

a few participants suggested that future updating should include a focus on providing integrated

services. As a consequence, we recommend the committee should review the programme to

ensure that it equips participants with a sound understanding of management and quality

issues related to integrated service provision and the skills to implement integrated services.

There is evidence that not all line managers have provided the level of support required during the

programme. Only a fifth of respondents to the survey reported that employers had provided the level

of support needed or considerable support in terms of advice and support with assignments (20%)

and support to develop and review the PDP (23%). The mechanisms in place to encourage line

manager involvement, such as the tri-partite agreement, pre-programme seminar, the requirement to

sign off the PDP and to provide a statement as part of the work-based project, are valued by

participants, managers and delivery staff. There is evidence, though, that there is scope to expand

on these even further. For example, some line managers suggested a mid-term meeting between

delivery staff, line managers and participants and one TMDP tutor suggested involving line managers

at the end of the programme would support participants’ progression after completing. A mid-term

review between students, line managers and university tutors is a key component of the

undergraduate social work degree and so staff are generally familiar with this approach and

understand its value. As a consequence, we recommend that mid-term and end of programme

seminars, whether face to face or virtually, either at regional or local authority level, should


52 10 April 2013

be included in the programme.

There are effective processes in place for monitoring and reviewing the programme, which have

resulted in modifications and improvements to the programme over time to better respond to

students’ needs. These include regular meetings between delivery staff to discuss student feedback,

achievements and their experiences of teaching. As a consequence, we recommend these

meetings should continue to take place.

Furthermore, the programme has been effectively managed by the national programme management

committee. There is clear evidence that the committee is functioning well and has brought together

the right stakeholders. As a consequence, we recommend that as the programme moves into

its next phase, strong governance arrangements should continue to be in place.

6.4 Is there evidence of outcomes and impacts for participants and their organisations? What is the additionality of outcomes? Does the course offer value for money?

The majority of participants have achieved many of the expected immediate and medium term

outcomes of the TMDP. All of the participants interviewed have applied learning to effect change in

at least one area of their management practice as a result of the course. In most cases, participants

have reported that new processes introduced as a result of the work-based project have continued to

be used after completion of the TMDP.

In around half of cases, there is evidence that after completing the course participants have made

additional changes to the way they manage their team beyond the requirements of the work-based

project component of the course. In particular, there is evidence of sustained change in relation to

ensuring the national policy agenda shapes the team’s work, managing change, improving service

user involvement, managing the performance of individuals and using different management tools

and techniques. Areas in which there is less evidence of sustained changes to practice are

processes for audit and inspection, the use of research and evidence to shape work, and managing

demand and capacity.

Generally, this suggests that the TMDP has achieved its aims and objectives and that it has:

■ Supported managers to develop an in-depth understanding of the key drivers influencing social

care and policy and practice in Wales

■ Supported the development of critical thinking skills to improve case management decisions;

■ Provided managers with a range of tools and techniques to improve the quality of practice within

their teams have been met,

■ Contributed substantially to helping team managers address the challenge of increasing demand

for services and increasing user expectations at a time of declining resources

There is evidence that outcomes have arisen largely a result of the course although a few

participants considered that over time some of the outcomes they reported (relating to improved

processes for assessing and reviewing quality, managing demand and capacity and service user

involvement, for example) would have been achieved even if they had not taken part in the TMDP.

However, they consider that as a result of the TMDP, these have happened much more quickly and

systematically. A few participants, including those working in youth offending teams, reported that

other training courses had influenced their understanding, knowledge and behaviour as well as the

TMDP. They attributed changes in practice to both sources of learning. There is no evidence that the

TMDP resulted in displacement or substitution of other good management practice.

Having the support of line managers and other colleagues who have completed the TMDP enables

learning to be applied over a sustained period of time. Therefore, to enhance outcomes and

ensure they are sustained over time, efforts should continue to roll out the TMDP to more

team managers and to provide complementary training to senior managers.

The inputs for the design and delivery of the TMDP seem appropriate to the level of outputs and

outcomes achieved and the unit cost of £3145 per participant compares favourably to another similar

course, with comparable outcomes for participants.


53 10 April 2013

6.5 What is the potential for the programme to deliver its expected achievements?

As yet there is little evidence of impact on service efficiency and effectiveness though there is

evidence of this in many cases on a small scale within areas that the participants manage. There is

emerging evidence that the TMDP is supporting greater career progression and, in particular,

providing an improved transition for aspiring managers. The survey suggests that a third (33%) of

participants have changed jobs since starting the TMDP and most have gone on to higher level jobs.

There are also examples of participants interviewed who have been promoted while doing or after

completing the TMDP, who believe that the programme has contributed significantly to their success

in securing a new position. As yet there is little evidence of greater satisfaction among service users

as a result of the TMDP but there is evidence of learning in relation to how to involve service users in

quality improvement projects and examples of applying this learning from participant and line

manager interviews. As a consequence, it would appear that there is significant potential for

the programme to deliver its expected impacts.


54 10 April 2013

ANNEXES


55 10 April 2013

Annex 1 Evaluation Framework


56 10 April 2013

Element of logic model Indicator Tool / method for evidence collection

Inputs Course fees and in-kind contributions -

Rationale The rationale that underpins TMDP is valid Interviews with line managers, participants and stakeholders

Activities / process Local authorities identify eligible participants and provide effective informal and formal support and review throughout the course

Interviews with participants, line managers, delivery agents and stakeholders; participant survey

Learning agreements between provider, employer and student are used appropriately and effectively

Interviews with participants, line managers, delivery agents

The course is delivered appropriately and effectively Interviews with participants, line managers, delivery agents, and stakeholders Participant survey

The course if effectively monitored and reviewed Interviews with participants, line managers, delivery agents

Target group The intended target group is reached Interviews with participants, line managers, delivery agents and stakeholders; participant survey; participant MI data

Outputs Production of a student handbook Student handbook

No. of participants accessing the programme Participant MI data

No. of cohorts Participant MI data

No. of modules successfully completed Participant MI data

No. of participants achieving the PG Certificate Participant MI data

No. of 360 degree assessments and Personal Development Plans completed at the start and end of the programme

360 degree assessment data; Participant survey

Immediate outcomes The course is relevant to the roles and needs of team manager participants Interviews with participants, line managers and delivery agents; Participant survey

Participants have increased confidence and are better able to identify support for themselves

Interviews with participants, line managers and delivery agents; Participant survey


57 10 April 2013

Participants have increased understanding of the national agenda for change Interviews with participants, line managers and delivery agents; Participant survey

Participants have increased understanding of the audit, inspection and review processes in Wales and effective responses


Participants have increased understanding of how to manage partnerships across disciplines


Participants have increased understanding of how to manage effective demand and capacity


Participants have increased understanding of how to manage casework Interviews with participants, line managers and delivery agents; Participant survey

Participants have increased understanding of how to include service users in quality improvement projects


Participants have increased understanding of how to manage individual performance within teams, particularly new staff and those performing poorly


Participants have increased understanding of how to use evidence-based practice, how to interpret research and embed in service provision


Participants have increased understanding of the relative benefits of different management tools


Participants have increased understanding of how to review, monitor and analyse quality arrangements


Participants have increased understanding of how to evaluate and reflect on their own personal competences


Participants share learning with colleagues Interviews with colleagues, participants and line managers; Participant survey

Employers recognise the learning acquired and value the training Line manager interviews

Medium-term outcomes Improved transition for aspiring managers to the role of team manager Interviews with participants, line managers and delivery agents; Participant survey

Participants reflect the national agenda in their local practice Interviews with participants, line managers, colleagues, delivery agents; Participant survey


58 10 April 2013

Improved planning and management of national audit, inspection and review processes

Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved management of partnerships across discipline Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved processes for assessing and reviewing quality arrangements Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved management of demand and capacity Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved management of casework Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved involvement of service users in quality improvement projects Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved management of the performance of individuals, including improved identification of development needs, better supervision and support for new staff and staff performing poorly

Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Increased use of a range of management tools Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Improved use of research and evidence in practice Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Have increased commitment to ongoing professional development Interviews with participants, line managers, colleagues, delivery agents; Participant survey

Participants believe they have increased opportunity for career progression Interviews with participants, line managers, colleagues, delivery agents; Participant


59 10 April 2013

survey

Team members feel more motivated and supported Interviews with participants, line managers, colleagues, delivery agents

Employers recognise increased capacity to manage change and improve service quality

Line manager interviews

Impacts Improved efficiency and effectiveness of services managed directly by participants, leading to value for money services

Interviews with participants, line managers, colleagues, delivery agents and stakeholders; Participant survey

Fewer management resources elsewhere in the organisation spent resolving problems in services managed by participants

Interviews with participants, line managers, colleagues, delivery agents and stakeholders Participant survey

Greater satisfaction among users of those services managed by participants that services are focussed on their needs

Interviews with participants, line managers, colleagues, delivery agents and stakeholders Participant survey

Documents

Evaluation of the Team Manager Development Programme ... · Evaluation of the Team Manager Development Programme: Final Report i 10 April 2013 Executive summary This is the final