Learning about Learning – Evaluation of a National Child

Protection Training Programme

Kate SkinnerInstitute Lead: Research Development and Application

In this Presentation I will:

• Tell you about the training programme

• Describe our evaluation methods

• Identify what we learned from it

• Invite your thoughts and views

The Programme:

• Followed child death where knowledge of child protection (CP) found lacking in social work staff who worked with adults

• Funded by Scottish Government (formerly Scottish Executive)

• Available to social workers from 32 local authorities in Scotland

The Programme (ii)

• Aimed to include learning about substance misuse, domestic violence and mental health

• Comprised 4 days for adult services staff (2 days on CP and 2 days with staff from children’s services on joint working)

• 2 days for children’s services staff

• Delivered locally by project staff

Programme (iii)

• Programme accredited for 20 credits (200 hours study) at SCQF level 9 (3rd year of a 4 year degree) with written assignment

• Hard copy materials not provided for participants tho’ virtual learning environment arranged for participants to access materials

The Evaluation (i)

• Commissioned via successful competitive tender

• Funded by the Project

• Commissioned in 2005 and completed by independent team from Universities of Stirling and Kingston in March 2007

The Evaluation (ii)

• Based on Kirkpatrick’s (1994) four levels of evaluation

• Used a multi-modal approach

Aims of the Study - To Evaluate impact on :

• Practitioners’ knowledge

• Intra-agency cooperation

• Intra-agency communication

• Initial assessments

• Ability to identify children at risk of harm

• Practitioners’ confidence re roles and responsibilities

Study Design

• Classroom Observation • Scrutiny of Programme Materials • Scrutiny of Participants’ Feedback • Knowledge tests • Short vignettes in which participants applied learning• Trainers’ views on programme• Scrutiny of assessment grades • External Examiner’s Reports • Participants’ views on changes to practice • Survey of Managers • Interviews with participants• Interviews with service users• Examination of service users’ files

What does the Literature tell us? (i)

• Evaluation must be systematic & include transfer of learning in the workplace (Baginsky and MacPherson, 2005; Ogilvie-Whyte, 2006)

• Collaborative working is difficult(Cooper et al, 2003; Huxham & Vangan, 2005)

• There is a knowledge base to be learned (Shardlow et al, 2004)

• Learning needs to connect to what people do (Rogers, 1974; Gardner, 2006)

What does the Literature tell us? (ii)

• Learning needs systematic preparation and support (Cherniss, 1998;Skinner & Whyte, 2004)

• Learning is shared responsibility of commissioners, learners, managers and trainers (Curry et al, 1994)

• Without involvement of all above, retention of learning and implementation will not occur systematically (Woodhouse and Pengelly; 1991 Fineman, 1997)

Findings (i)

• Little/no preparation of participants by managers or trainers

• Participants had v low expectations of programme as trigger for practice change

• Significant differences in delivery between project team members

• Disappointing changes in level of knowledge• Major discrepancies between feedback and

transfer of learning

Findings (ii)

• Some resistance to thorough evaluation of training as legitimate use of staff time

• Assessment of learning given v low priority by participants (3% of whole population)

• Self-report limited as measure for retention of learning

• Little attention given to retention of learning by staff, managers and trainers

Findings (iii)

• Intra- and inter-agency communication and collaboration is difficult and requires dedicated learning programmes to both raise their profile and enable learning of techniques

Concerns (i)

• Rhetoric of measurement, effectiveness and value for money not backed up in practice

• Self-reports viewed as sufficient proof of worth of training

• Absence of reliable objective data on impact

Concerns (ii)

• Suspicion that very little practice change resulted despite expensive, competent training arrangements

• Concern that government believe that training offers a speedy, reliable and productive response to a practice problem

Questions:

• Would it be better to do less training and focus more on retention?

• Are we using research on how people learn?

• Is it OK to go on a course and not expect to have to change what we do?

• Do we need to do more evaluation of this type to understand more about what kind of learning we should be offering?