The Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP): development and psychometric properties Article

Accepted Version

Stallard, P., Myles, P. and Branson, A. (2014) The Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP): development and psychometric properties. Behavioural and Cognitive Psychotherapy, 42 (3). pp. 269-282.ISSN 1352-4658 doi: https://doi.org/10.1017/S135246581300115X Available at https://centaur.reading.ac.uk/39440/

It is advisable to refer to the publisher’s version if you intend to cite from the work. See Guidance on citing .

To link to this article DOI: http://dx.doi.org/10.1017/S135246581300115X

Publisher: Cambridge University Press

All outputs in CentAUR are protected by Intellectual Property Rights law, including copyright law. Copyright and IPR is retained by the creators or other copyright holders. Terms and conditions for use of this material are defined in the End User Agreement .

www.reading.ac.uk/centaur

CentAUR

Central Archive at the University of Reading Reading’s research outputs online

1

The Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP):

development and psychometric properties

Paul Stallard1, Pam Myles2 & Amanda Branson2

1 Department of Health

22/23 Eastwood

University of Bath

Bath

BA2 7AY

2 Charlie Waller Institute

School of Psychology & CLS

University of Reading

Earley Gate

Reading RG6 6AL

Address for correspondence: Professor Paul Stallard, Department of Health, University of

Bath and Head of Psychological Therapies Oxford Health NHS Foundation Trust,

Bath BA2 7AY, UK; P.Stallard@bath.ac.uk

2

Abstract

Background: There is increased interest in developing training in cognitive behaviour therapy (CBT)

with children and young people. However the assessment of clinical competence has relied upon the

use of measures such as the Cognitive Therapy Scale-Revised (CTS-R: Blackburn et al 2001) which has

been validated to assess competence with adults. The appropriateness of this measure to assess

competence when working with children and young people has been questioned.

Aim: This paper describes the development and initial evaluation of the Cognitive Behaviour Therapy

Scale for Children and Young People (CBTS-CYP) developed specifically to assess competence in CBT

with children and young people.

Methods: A cross section of child CBT practitioners (n=61) were consulted to establish face validity.

Internal reliability, convergent validity and discriminative ability were assessed in two studies. In the

first, 12 assessors independently rated a single video using both the Cognitive Behaviour Therapy

Scale for Children and Young People (CBTS-CYP) and Cognitive Therapy Scale–Revised (CTS-Revised:

Blackburn et al 2001). In the second, 48 different recordings of CBT undertaken with children and

young people were rated on both the CBTS-CYP and CTS-R.

Results: Face validity and internal reliability of the CBTS-CYP were high, and convergent validity with

the CTS-R was good. The CBTS-CYP compared well with the CTS-R in discriminative ability.

Conclusion: The CBTS–CYP provides an appropriate way of assessing competence in using CBT with

children and young people. Further work is required to assess robustness with younger children and

the impact of group training in reducing inter-rater variations.

3

Introduction

In October 2011 the Improving Access to Psychological Therapies (IAPT) programme for working age

adults was extended to children and young people (CYP-IAPT; Department of Health, 2011). The

primary aim of CYP-IAPT is to increase the availability of evidence-based psychological interventions

for children and young people supported with ongoing clinical supervision. A notable difference

from the adult IAPT programme is that training is focused upon clinicians working within existing

child and adolescent mental health services (CAMHS) rather than creating separate IAPT services. By

so doing, CYP-IAPT aims to transform current services by embedding the CYP-IAPT philosophy of

evidence-based practice, child/young person centred services and outcome informed practice

throughout CAMHS.

The development of evidence-based practice requires the implementation of the knowledge, skills

and attitudes (KSAs) from successful randomised controlled trials into every day practice. For the

adult IAPT programme, Roth and Pilling extracted these KSAs and developed a competency

framework that subsequently underpinned IAPT training (Roth & Pilling, 2008). This framework has

been extended to CAMHS, and was further developed to define the competences required to deliver

CBT for children and young people suffering with anxiety and depressive disorders, and provide the

backbone for the national CYP-IAPT training curriculum (Roth, Calder & Pilling, 2011). However, in

order to determine if a sufficient level of competence in evidence-based practice has been acquired,

an appropriate method is required to determine whether or not competence has been achieved.

The most widely used tool for measuring CBT competence with adults is the Cognitive Therapy

Scale–Revised (CTS-R) by Blackburn, James, Milne, Baker, Standart, Garland and Reichelt (2001)

which was a revised version of the original Cognitive Therapy Scale developed by Young and Beck

(1980; 1988). The CTS-R consists of 12 items which assess important generic CBT skills. These include

four general skills (feedback: collaboration; pacing and efficient use of time, and interpersonal

effectiveness) and seven specific CBT skills (eliciting appropriate emotional expression; eliciting key

4

cognitions; eliciting behaviours; guided discovery; conceptual integration; application of change

methods and homework setting). The remaining item, agenda setting, overlaps both sets of items

and is included in the general and specific CBT skills sub scales. Whole video-tapes are rated on a 7-

point scale reflecting different levels of competence (0 = incompetent through to 6 = expert) with a

score of less than 2 on any individual item or a total score of less than 50% (a score of 36) reflecting

unsatisfactory performance.

The applicability of the CTS-R to assess competence in the use of CBT with children and young

people has been questioned (Fuggle, Dunsmuir & Curry, 2012). In particular, the discriminative

ability of the total score to determine competence and the absence of developmentally appropriate

criteria were identified as limitations. This led Fuggle and colleagues to develop the CBT Session

Competency Framework (CBTSCF) to assess competence in the use of CBT with children and young

people (Fuggle et al, 2012). The CBTSCF is a 24-item scale which includes all 12 items from the CTS-R

but includes additional items more specific to working with children such as child-centred practice

and creativity. The authors propose that 15 domains of the scale should be evident in every session

(e.g. therapeutic alliance, collaborative practice) with the remaining nine being demonstrated in

some sessions depending upon the stage of therapy (e.g. discovering cognitions, recognising

emotions, developing a shared formulation). Each item is rated as ‘competent’, ‘partially

competent’ or ‘failed’; there is no total score, rather an overall recommendation of pass or fail. No

psychometric evaluations of the scale have been reported and the robustness and sensitivity of the

three-point scoring system is unknown. Similarly, the inclusion of optional items to assess

competence is problematic. It is unclear whether their absence appropriately indicates that the

items were not required or that the trainee inappropriately failed to recognise their importance.

The absence of a psychometrically robust scale developed specifically to assess CBT competence

with children and young people led to the development of the Cognitive Behaviour Therapy Scale for

Children and Young People (CBTS-CYP). The scale was modelled on the widely used CTS-R; all skills

5

were expected to be demonstrated in each session and were rated on a 7-point scale of

competence. The purpose of this paper is to describe the development and preliminary

psychometric analysis of the Cognitive Behaviour Therapy Scale for Children and Young People

(CBTS-CYP). The development of the scale items will be described, psychometric properties

evaluated and comparison with the CTS-R examined.

Cognitive Behaviour Therapy Scale for Children and Young People (CBTS- CYP)

1. Guiding Principles

Although the reliability, validity, item overlap and thresholds of the CTS-R have been questioned, it is

nonetheless widely used and considered to provide a comprehensive overview of the generic skills

required to competently practice CBT with adults (Kazantzis, 2003; Fairburn & Cooper, 2011; Keen &

Freeston, 2008). It was therefore decided that, in as far as possible, any measure developed for use

with children and young people should build upon and be consistent with the CTS-R. Firstly, in terms

of content, the CTS-R assesses the specific use of CBT methods as well as general skills that facilitate

their effective delivery. It was therefore decided that the CBTS-CYP would contain items that

assessed both competence in the application of specific methods and in the process of using CBT

with children and young people. Secondly, it was decided that all 12 items of the CTS-R should be

covered by the CBTS-CYP although these might need to be modified and others added to reflect the

use of CBT with children and young people. Thirdly, the framework for defining the development of

competence proposed by Dreyfus (1986) and adapted into a 7-point Likert scale on the CTS-R would

form the basis of the CBTS-CYP assessment of competence. Fourthly, by using the same rating scale

the thresholds for assessing competence would be the same as those on the CTS-R (i.e. 2 or more

on each item and a total score of 50% or more). Finally, it was decided that the scale would be

developed to assess both verbal and non-verbal behaviours and so could be used like the CTS-R to

assess both audio and video recordings of whole clinical sessions.

6

It was anticipated that specific items would not necessarily be mutually exclusive. For example, a

formulation requires the development of a shared conceptualisation in which important cognitions,

emotions and behaviours are bound together within the CBT model. The elicitation and

identification of key cognitions and processes would therefore be expected to be associated with the

formulation. Similarly, CBT typically involves developing an understanding of the links between

cognitions, emotions and behaviours and as such there will inevitably be overlap between these

different aspects of the cognitive behavioural model.

2. CBTS-CYP Development

2.1. Process items

A modest but consistent relationship has been found in child psychotherapy between the

therapeutic alliance and treatment outcomes (Shirk & Karver, 2003; Karver, Shirk, Handelsman,

Fields, Crisp, Gudmundsen & McMakin 2006; Mcleod, 2011). Research into which specific aspects of

the alliance are particularly important is generally lacking and hindered by the absence of consistent

terminology and frameworks (Elvins & Green, 2008; Fjermestad, Haugland, Heiervang & Ost, 2009).

Despite these limitations Creed and Kendall (2005) identified collaboration as an important predictor

of alliance in CBT with children and young people. This was defined by behaviours suggesting a

partnership with the child and therapist working together as a team, agreeing shared goals with the

therapist actively inviting the young person’s participation and involvement. Excessive formality

resulting in the therapist talking to the child in an aloof or patronising way and pushing the young

person to speak about uncomfortable emotions had a negative effect on the relationship whilst

Russell, Shirk and Jungbluth (2008) found that therapists’ responsiveness characterised by warm,

positive and empathic behaviours had a positive effect. Similarly, flexibility or creativity whereby the

therapist makes sessions more active, uses different methods to explain ideas such as games, role-

plays, involving others and attempts to match the concepts to the child’s interests was positively

related to child engagement (Chu & Kendall, 2009). Engagement was subsequently associated with

7

positive outcomes leading the authors to suggest that particular attention may need to be paid to

securing and maintaining engagement with children by making sessions interesting and enjoyable.

Children are typically referred to psychological services because of concerns identified by others and

may not themselves recognise or acknowledge any problems or the need to do anything different

(Mcleod & Weisz, 2005). It is therefore unsurprising that the development of self-efficacy through

motivational techniques highlighting positive treatment expectancies, challenging pessimism and

providing praise have been identified as important (Russell et al, 2008). A further aspect of the

therapeutic process which requires specific attention is that of collaborative inquiry which requires

people to “become scientific investigators of their own thinking” (Beck & Dozios, 2011, p400). The

promotion of a reflective and investigative approach requires careful attention when working with

children who are typically use to being provided with information and answers. Finally, the need to

adapt CBT to the developmental level of the child or young person has been emphasised by many

writers (Stallard, 2002; Friedberg & McClure, 2002) and has been reflected in different versions of

CBT programmes for children and adolescents (Barrett, Lowry-Webster & Turner, 1999). This

requires the therapist to ensure that CBT is pitched at the right level and is consistent with the

child’s cognitive, emotional, verbal and reasoning ability.

These key aspects of the process of CBT with children and young people have been encapsulated by

the acronym PRECISE (Stallard, 2005). The therapeutic process involves the child and their family

working in a partnership (P) with the therapist. The partnership is based upon collaborative

empiricism and highlights the active role of the child and their parents/carers in securing change.

The therapist needs to pitch the intervention at the right developmental level (R) to ensure that it is

consistent with the child’s cognitive, linguistic, memory and perspective taking ability. The therapist

adopts a warm, caring, respectful and empathic (E) relationship and creatively (C) and flexibly

conveys the concepts of CBT in a way that matches the child’s interests and understanding. A key

8

concept of CBT is that of guided discovery and investigation (I) which is developed through the

adoption of a curious and reflective stance. Self-efficacy (S) is promoted as the therapist helps the

child or young person to discover and build upon their strengths, skills and ideas. Finally, therapy

sessions should be enjoyable (E) and engaging to maintain the child’s motivation and commitment to

change.

The above seven items form the basis of the process sub-scale of the CBTS-CYP. Table 1

demonstrates how these are assumed to relate to items from the CTS-R.

Insert Table 1 here

2.2. CBT method items

The CTS-R includes items that assess the eliciting of emotions, cognitions and behaviours as well as

an item assessing application of change methods. These four items are subsumed within three items

on the CBTS-CYP relating to the awareness, identification and management of emotions (E)

cognitions (C) and behaviours (B). Formulation (F) relates to the conceptual integration dimension of

the CTS-R. Unlike the CTS-R, there is no specific homework setting dimension on the CBTS-CYP.

Instead homework is subsumed within the discovery (D) dimension which also includes the

development of self-discovery and understanding. Similarly, there is no separate item assessing

agenda-setting which is incorporated within a wider item assessing general session planning and

organisational skills (G). Finally, a core characteristic of CYP-IAPT is the use of routine outcome and

goal planning measures resulting in an additional item of assessment (A) being included.

The CBTS-CYP and CTS-R equivalent method items are summarised in Table 2.

Insert Table 2 here

2.3. CBTS-CYP framework

The 14 items that constitute the CBTS-CYP are summarised in Figure 1. The assessment (A) informs

the formulation (F) which encapsulates key cognitions (C), emotions (E) and behaviours (B). These

9

are evaluated and more functional ones developed through the process of discovery (D) with general

organisational skills (G) facilitating the effective use of CBT methods. This occurs within a therapeutic

process based on partnerships (P) working with methods being pitched at the right (R)

developmental level. The relationship is empathic (E) with the therapist being flexible and creative

(C) in the way CBT is adapted to reflect the child’s interests. A curious stance encourages self-

discovery through investigation (I) and reflection with a positive and hopeful approach promoting

self-efficacy (S). Finally, engagement and interest is maximised by ensuring that sessions are

enjoyable (E).

The 14 items are each rated using the same 7-point Dreyfus Likert Scale (Dreyfus, 1986) as used in

the CTS-R (Blackburn et al, 2001). By using the 0-6 measure of competence across the 14 items, the

total score can range between 0-84 and is converted to an overall percentage by dividing the total

score by 84 and multiplying by 100.

Insert Figure 1 here

Method

The internal reliability, convergent validity and discriminative ability were assessed in two studies. In

the first, 12 markers for the University of Reading CYP-IAPT programme independently rated a single

video recording of a clinical CBT session with a young male adolescent (aged 12 years) with an

anxiety disorder using both the CBTS-CYP and CTS-R. In the second, 48 audio recordings submitted

to the University of Reading by 18 trainees in fulfilment of the CYP-IAPT CBT training requirements

were rated by University assessors on both the CBTS-CYP and CTS-R. Assessments and recordings

were submitted in relation to the three, national curriculum led modules covering core CYP-IAPT

principles, anxiety and depression (see www.iapt.nhs.uk/cyp-iapt). The primary problems in the

submissions were depression (n=20); separation anxiety (n=9); social anxiety (n=6); OCD (n=5); panic

(n=3); generalised anxiety (n=3), specific phobia (n=1) and PTSD (n=1). Young people in the

10

recordings were predominantly female (n=31, 64.6%) and ranged in age from 9-17 years (M=14.44,

sd=2.31).

Face validity

An overarching description of each item was developed by the author (PS) and some examples of

specific behaviours that evidence the competence were generated. The conceptual framework,

headline description and detailed descriptors were then discussed with a cross-section of child CBT

therapists (n=61) ranging in expertise and experience.

Firstly, training in using the CBTS-CYP was delivered to seven experienced child CBT supervisors

(applying for accreditation as CBT therapists by the British Association for Behavioural and Cognitive

Psychotherapies - BABCP) and four experienced child mental health specialists who were markers on

the CYP-IAPT course at the University of Reading. Following feedback, a second iteration was

developed which included greater specification of each competence level. A second group of four

experienced CBT supervisors (applying for BABCP accreditation) and 12 markers were provided

training in how to use the CBTS-CYP and gave feedback on the framework which was considered to

comprehensively capture both the process and skills involved in using CBT with children and young

people. In total 27 people were involved in the initial discussions and training. The group were

predominantly Clinical Psychologists (89%), aged 25-44 (63%) with over 5 years of experience

working with children and young people (81%). Finally, the framework was discussed with 18 CBT

trainees on the Reading CYP-IAPT course and with a group of 16 clinical psychologists undertaking

their clinical training. Feedback was unanimously positive with the CBTS-CYP being viewed as

comprehensive and more accurately reflecting the process of CBT with children and young people

than the CTS-R.

Internal reliability

11

Intraclass correlations (ICC) were calculated for the 12 ratings of the single video. A two way mixed

effects approach ICC with absolute agreement is appropriate where several raters assess the same

target (Shrout & Fleiss, 1979). For the seven process items ICC was 0.95 (95%CI: 0.90-0.98), for the

seven method items, 0.90 (95%CI: 0.79-0.97) and the ICC for the total 14 items (0.96) of the CBTS-

CYP (95%CI: 0.91–0.99) were substantial demonstrating good reliability.

These compared well with the CTS-R where the equivalents were 0.93 (95%CI: 0.84-0.98) for the five

general items, 0.91 (95%CI: 0.80-0.97) for the eight specific items and 0.95 (95%CI: 0.90-0.98) for the

total 12 items. Although the total score sums all 12 items, the agenda setting item is included in

both the general and specific item sub-scales.

Convergent Validity

Correlations between CBTS-CYP and CTS-R sub-scale and total scores

Convergent validity was determined by comparing the CBTS-CYP against the CTS-R. In the first study

(n=12 raters) Pearson correlation coefficients between the CBTS-CYP process and CTS-R general

scores (r=.98, p<.001), CBTS-CYP method and CTS-R specific scores (r=.91, p<.0001) and total scores

(r=.98, p<.0001) were very high. Similarly, in the second study (n=48 session recordings) correlations

between process and general scores (r=.79, p<.0001), method and specific skills (r=.91, p<.0001) and

total scores (r=.93, p<.0001) were again very high, suggesting that the two measures are highly

correlated.

Correlations between individual items on the CTS-R and CBTS-CYP

To investigate the a-priori assumed relationship between individual items on the CBTS–CYP and CTS-

R, correlational analysis (Pearsons R) was undertaken of the 48 session recordings.

Insert Table 3 here

12

Results, summarised in Table 3, indicate that the predicted correlations were all significant, ranging

from r=.93 between formulation and conceptual integration to r=.31 between emotional techniques

and application of change methods.

Discriminative Ability

Results from the 12 raters using the two pass and fail criteria on the CBTS-CYP and CTS-R (i.e. no

single item is scored less than 2 and that the total score exceeds 42, i.e. 50%) are shown in Table 4.

Insert Table 4 here

In study one there was agreement between the CBTS-CYP and CTS-R on whether the session

recording was passed or failed by 11/12 (91.7%) raters using the 50% criteria and by 11/12 using the

single item criteria. In terms of the total percentage criteria, 8/12 failed the tape on both the CBTS-

CYP and the CTS-R. There was one instance (rater 6) where the case failed on the CTS-R but passed

on the CBTS-CYP. In terms of single items, 9/12 raters failed the tape on both the CBTS-CYP and CTS-

R with one (rater 4) failing the tape on the CBTS-CYP only. Using both criteria, 10/12 raters failed the

tape on both the CBTS-CYP and the CTS-R.

All 12 raters met to discuss individual rater scores, and a group rating agreed for each item. The

agreed total CBTS-CYP percentage indicated that the session should be failed; eight raters

consequently lowered their scores and four increased them. Using the individual item criteria, the

consensus was that three items on the CBTS-CYP were failed (creative, assessment and discovery).

The discriminative ability of the CBTS-CYP and CTS-R was also explored by examining ratings of the

48 session recordings (Study 2). Agreement between the CBTS-CYP and CTS-R on whether a case had

passed or failed (on either or both of the individual items or total score criteria) was found in 37

(77%) cases. Of the remainder, eight recordings were failed only on the CTS-R and three only on the

13

CBTS-CYP. An analysis of the individual items which were failed on either measure are summarised

in Table 5.

Insert Table 5 here

Session recordings were failed on all 12 individual items of the CTS-R with homework setting and

conceptual integration being more frequently failed. For the CBTS-CYP, all items were failed except

for assessing the right developmental level. The most frequently failed items were formulation,

discovery experiments and behavioural techniques.

Finally, changes in competence of the 18 trainees as they progressed through training were

examined by comparing total scores on the CTS-R and CBTS-CYP for each of the three assessed

modules. The results highlight that competence increased by approximately eight points on both

measures from the start to the end of training. Average total scores on the CTS-R and CBTS-CYP were

respectively module one (m=47.17, sd=7.11; m=50.83, sd=8.18), module two (m=52.22, sd=9, 37;

m=56.06, sd=10.36) and module three (m=55.42, sd=8.31; m=58.42, sd=8.45).

Discussion

This study reports on the development and initial evaluation of the Cognitive Behaviour Therapy

Scale for Children and Young People (CBTS-CYP) specifically developed to assess competence in

undertaking CBT with children and young people. The results indicate that the CBTS-CYP has good

reliability and face validity and compares well, using the total score and single item criteria, with the

CTS-R in discriminative ability. This is encouraging and suggests that the CBTS-CYP is an appropriate

measure to determine competence with children. However, in reaching this conclusion it is

recognised that there are a number of issues which need to be highlighted.

First, whilst the internal consistency of the CBTS-CYP is high, this also suggests that there is

considerable overlap between individual items. This interconnectedness was also found with the

14

CTS-R in this study and in previous studies (Blackburn et al, 2001). From a theoretical perspective

this is not surprising since it would not be expected that items would be mutually exclusive. Whilst

the emphasis of individual sessions might vary in their primary focus, CBT is an integrated therapy

which will typically draw upon many aspects of the CBT framework in each session. This point was

highlighted by Blackburn et al (2001) who noted that CBT training does not typically focus on one

specific skill/aspect at a time but involves training in the global model. The high internal consistency

of the CBTS-CYP and CTS-R may therefore indicate considerable overlap between items but may also

reflect the way that CBT skills are acquired.

Secondly, whilst the CTS-R has been widely used to assess CBT competence our use of this measure

to validate the CBTS-CYP raises issues which need to be acknowledged. First the CBTS-CYP includes

all the items (except homework) from the CTS-R. The strong correlation found in our study between

these measures may therefore reflect the considerable overlap in content between them. Second,

despite specific scoring guidelines there continues to be wide variations on the CTS-R between

individual raters, a finding that has been noted in this and previous studies (Blackburn et al, 2001;

Keen & Freeston, 2008). Equally, the criteria for determining competence have not been

demonstrated and so it is unclear whether someone who scores 48% is actually “incompetent”

compared to another who scores 55% or whether this perhaps reflects differences in client co-

operation or engagement. Issues such as these raise questions about whether the CTS-R is an

appropriate comparator against which to assess the CBTS-CYP and whether the pass criteria are

valid. These limitations are acknowledged but given that the CTS-R is the most widely used scale for

assessing competence in CBT with adults, this comparison using the same pass criteria does seem

appropriate.

Thirdly, an examination of the specific item failure criteria revealed that in the 48 session recordings

assessed, the right ‘developmental level’ item on the CBTS-CYP was never failed. Similarly, other

15

CBTS-CYP process items such as ‘partnership working’, ‘creative and enjoyable’, and ‘engaging’ were

seldom endorsed thereby questioning whether they are important determinants of competence. In

considering this question there are two particular issues that should be noted. First, the selection

process for the trainees whose recordings were used in this study was rigorous. One criteria

requiring a minimum of two years of relevant experience of working with children and young people

results in the average number of relevant years of experience of the 18 trainees being just over 10

years (range 2-31 years). Our cohort was therefore already very experienced in working with

children and young people and would be expected to have already acquired the process skills to

work therapeutically with this age group. The low frequency of failure on process items may

therefore reflect established competence compared to the higher rate of failure on the CBT method

items where the trainees are inexperienced and acquiring specific skills. Further studies using the

CBTS-CYP to assess therapists with more limited experience of working with children should be

undertaken to clarify this issue. The second possibility which needs to be considered relates to the

age of the young people in the recordings assessed since there was a tendency towards submitting

sessions involving adolescents as opposed to children. The average age of the young people in the

recordings was 14.44 years with only four (8.3%) submissions involving children under the age of 11.

Undertaking CBT with younger children will require greater attention to the process issues defined

by the CBTS-CYP. For example, creativity is more important with younger children in order to

maintain their interest. Similarly, pitching the intervention at the right developmental level is more

challenging when working with younger children who are still acquiring skills. Process issues are

therefore likely to be more important with this younger age group and it is likely that trainees will be

more likely to fail to demonstrate competence.

This study has highlighted considerable pre-training variability between raters with total scores of

the 12 raters assessing the same video ranging from 19-68% on the CBTS-CYP and 24-70% on the

CTS-R. Although the CBTS-CYP and CTS-R include detailed descriptions of individual items and

16

specify differing competence levels, simply reading the scoring instructions does not appear

sufficient to ensure good consistency between raters. In this study, after completing the CBTS-CYP

and CTS-R independently, the 12 markers were convened as a group to discuss their marks and to

agree a consensual rating for each item. Eight markers lowered their initial ratings on the CBTS-CYP

following the discussion by up to 37 points with the lowest four raters increasing their scores by up

to 14 points. A similar pattern emerged with the CTS-R where nine raters lowered their scores by up

to 41 points and three increased their scores by up to 5 points. During the consensual discussion it

emerged that whilst the highest scorers had noted evidence of each item, the lower raters had also

identified missed opportunities and had considered the extent to which the competence was

displayed across the whole of the recording. Future studies should explore the effect of consensual

group discussions on reducing variations between raters.

Despite these limitations, this study has demonstrated that the CBTS-CYP, developed specifically for

assessing CBT with children and adolescents, has good face and convergent validity and

discriminative ability. These results are promising and suggest that the CBTS-CYP provides a useful

way of assessing CBT competence for therapists working with this younger age group.

Acknowledgements

The authors are grateful to everyone who has contributed to the development and evaluation of the

CBTS-CYP. Particular thanks go to the University of Reading CYP-IAPT supervisors who provided

many helpful suggestions during early drafts of the questionnaire. The authors would also like to

thank the first year CYP-IAPT trainees from the University of Reading for trialling the CBTS-CYP.

References

17

Barrett, P.M., Lowry-Webster, H., & Turner, C.M. (1999). Friends for Children participant workbook.

Brisbane, Australia: Australian Academic Press.

Beck, A.T. & Dozois, D.J.A. (2011). Cognitive Therapy: Current Status and Future Directions. Annual

Review of Medicine. 62, 397–409.

Blackburn, I.M., James, I.A., Milne, D.L., Baker, C., Standart, S., Garland, A. & Reichelt, F.K. (2001).

The revised cognitive therapy scale (CTS-R): psychometric properties. Behavioural and Cognitive

Psychotherapy, 29,431-446.

Chu, B.C. & Kendall, P.C. (2009). Therapist responsiveness to child engagement: flexibility within

manual based CBT for anxious children. Journal of Clinical Psychology, 65 (7), 736-754.

Creed, T.A. & Kendall, P.C. (2005). Therapist alliance-building behavior within a cognitive-behavioural

treatment for anxiety in youth. Journal of Consulting and Clinical Psychology, 73, 498-505.

Department of Health (2011). Children and teenagers to benefit from successful adult mental health

therapy. http://mediacentre.dh.gov.uk/2011/10/25/

Dreyfus, H.L. (1986). The Dreyfus model of skill acquisition. In J Burke (ED). Competency based

education and training. London: Falmer Press.

Elvins, R. & Green, J. (2008). The conceptualisation and measurement of the therapeutic alliance: an

empirical review. Clinical Psychology Review, 28, 1167-1187.

18

Fairburn, C.G & Cooper, Z. (2011). Therapist competence, therapist quality, and therapist training.

Behaviour Research and Therapy, 49, 373-378.

Fjermestad, K.W., Haugland, B.S.M., Heiervang, E & Ost, L.G. (2009). Relationship factors and

outcome in child anxiety treatment studies. Clinical Child Psychology and Psychiatry, 14 (2), 195-214.

Friedberg, R.D & McClure, J.A. (2002). Clinical practice of cognitive therapy with children and

adolescents; the nuts and bolts. New York: Guilford Press.

Fuggle, P., Dunsmuir, S & Curry, V. (2012). CBT with children, young people and families. London,

Sage Publications Ltd.

Karver, M.S., Handelsman, J.B., Fields, S. & Bickman, L. (2006). Meta-analysis of therapeutic

relationship variables in youth and family therapy: the evidence for different relationship variables in

the child and adolescent outcome treatment literature. Clinical Psychology Review, 26, 50-65.

Karver, M., Shirk, S., Handelsman, J.B., Fields, S., Crisp, H., Gudmundsen, G & McMakin, D. (2008).

Relationship processes in youth psychotherapy: measuring alliance, alliance-building behaviours and

client involvement. Journal of Emotional and Behavioural Disorders, 16, 15-28.

Kazantzis, N. (2003). Therapist Competence in Cognitive-behavioural Therapies: Review of the

Contemporary Empirical Evidence. Behaviour Change, 20, 1-12.

Keen, A.J.A & Freeston, M.H. (2008). Assessing competence in cognitive behaviour therapy. British

Journal of Psychiatry, 193, 60-64.

19

McLeod, B.D. (2011). Relation of the alliance with outcomes in youth psychotherapy: a meta-

analysis. Clinical Psychology Review, 31, 603-616.

McLeod, B.D & Weisz, J.R. (2005). The therapy process observational coding system – alliance scale:

measure characteristics and prediction of outcome in usual clinical practice. Journal of Consulting

and Clinical Psychology, 73, 323-333.

Roth, A. D. & Pilling, S. (2008). Using an evidence-based methodology to identify the competencies

required to deliver effective cognitive and behavioural therapy for depression and anxiety disorders.

Behavioural and Cognitive Psychotherapy, 36, 129–147.

Roth, A.D., Calder, F & Pilling, S. (2011). A competence framework for child and adolescent mental

health services. Edinburgh. NHS Education for Scotland.

Russell, R., Shirk, S. & Jungbluth, N. (2008). First-session pathways to the working alliance in

cognitive-behavioural therapy for adolescent depression. Psychotherapy Research, 18, 15-27.

Sburlati, E. S., Schniering, C.A., Lyneham, H.J., & Rapee, R. M. (2011). A model of therapist

competencies for the empirically supported cognitive behavioral treatment of child and adolescent

anxiety and depressive disorders. Clinical Child and Family Psychology Review, 14, 89-109.

Shirk, S.R., & Karver, M. (2003). Prediction of treatment outcome form relationship variables in child

and adolescent therapy: a meta-analytic review. Journal of Consulting and Clinical Psychology, 71,

452-464.

Shrout, P.E. & Fleiss, J.L. (1979). Intraclass correlations: uses in assessing rater reliability.

Psychological Bulletin, 86, 420-428.

20

Stallard, P. (2002). Think Good Feel Good: a cognitive behaviour therapy workbook for children and

young people. Winchester, John Wiley.

Stallard, P. (2005). A Clinicians Guide to Think Good Feel Good: using CBT with children and young

people. Winchester, John Wiley.

Young, J.E., & Beck, A.T. (1980) Cognitive therapy scale: rating manual. Unpublished Manuscript,

University of Pennsylvania, Philadelphia, PA.

Young, J.E. & Beck, A.T. (1988) Cognitive therapy scale: rating manual. Unpublished Manuscript,

University of Pennsylvania, Philadelphia, PA.

21

Table 1: CBTS-CYP Process and CTS-R equivalent items.

CBTS-CYP – Process Item Equivalent CTS-R item

Partnership working (P) Collaboration and learning together The therapist establishes a collaborative and respectful partnership with the child/young person (and as appropriate their carers) in which they are actively engaged in working together towards a set of joint goals and targets

Collaboration

Right Developmental Level (R) Pitch, methods and family involvement The therapist engages with the child/young person and family in a way, level and manner that is consistent with their developmental level and understanding

None

Empathy (E) Acknowledge, genuine, warm, understanding The therapist is able to empathise with the child/young person and their carers/family through the development of a genuine, warm and respectful relationship

Interpersonal effectiveness

Creative (C) The therapist is able to creatively adapt the ideas and concepts of CBT to facilitate the child/young person and their parents/carer understanding and engagement in therapy

None

Investigation (I) Discovery and reflection The therapist adopts an open and curious stance that facilitates guided discovery and reflection

Feedback

Self-efficacy (S) Builds on strengths and ideas The therapist adopts an empowering and enabling approach in which self-efficacy and positive attempts at change are promoted

None

Enjoyable and Engaging (E) Interesting and engaging The therapist is able to make therapy sessions appropriately interesting and engaging

None

22

Table 2: CBTS-CYP Method and CTS-R equivalent items.

CBTS-CYP – Method Item Equivalent CTS-R item

Assessment & Goals (A) Ratings and diaries The therapist establishes clear goals for the intervention and appropriately uses diaries, questionnaires and rating scales for assessment and to monitor progress.

None

Behavioural techniques (B) Awareness, triggers, techniques of change The therapist demonstrates appropriate use of a variety of behavioural techniques to facilitate understanding and therapeutic change

Eliciting behaviours Application of change methods

Cognitive techniques (C) Awareness, identification, challenge, cognitive reframe The therapist demonstrates appropriate use of a variety of cognitive techniques to facilitate understanding and therapeutic change.

Eliciting key cognitions Application of change methods

Discovery (D) Experiments and homework The therapist appropriately uses a variety of methods to facilitate self-discovery and understanding

Guided discovery Homework setting

Emotional (E) Awareness, identification and management The therapist appropriately uses a variety of emotional techniques to facilitate understanding and therapeutic change

Eliciting appropriate emotional expression Application of Change methods

Formulation (F) Integration of CBT model The therapist facilitates the development of a coherent understanding which highlights the relationships between events, cognitions, emotions, physiological responses and behaviours.

Conceptual integration

General skills (G) Session planning, organization and pacing Sessions have a clear agenda; the therapist is well prepared and conducts sessions in a calm and organized way.

Agenda setting and adherence Pacing and efficient use of time

23

Table 3: Correlations (Pearson R) between CBTS-CYP and equivalent CTS-R items (n=48)

Assumed CBTS-CYP item CTS-R Item Correlation Significance

General Skills (G) Agenda setting and adherence .601 .000

Investigation (I) Feedback .693 .000

Partnership (P) Collaboration .897 .000

General Skills (G) Pacing and efficient use of time .676 .000

Empathy (E) Interpersonal effectiveness .677 .000

Emotional techniques (E) Eliciting appropriate emotional expression .819 .000

Cognitive techniques (C) Eliciting key cognitions .918 .000

Behavioural techniques (B) Eliciting behaviours .877 .000

Discovery experiments (D) Guided discovery .625 .000

Formulation (F) Conceptual integration .934 .000

Behavioural techniques (B)

Cognitive techniques (C)

Emotional techniques (E)

Application of change methods .538

.705

.308

.000

.000

.033

Discovery experiments (D) Homework setting .609 .000

24

Table 4: Comparison of raters scores (total percentages and number of items scoring <2) on the

CBTS-CYP and CTS-R (n=12 raters)

Rater Total CBTS-CYP (%) Total CTS-R (%) CBTS-CYP number item < 2

CTS-R number item <2

1 28 35 1 1

2 47 49 1 1

3 44 40 1 2

4 43 43 1 0

5 66 63 0 0

6 54 47 1 1

7 26 24 8 7

8 31 28 5 5

9 19 24 12 9

10 54 54 0 0

11 68 70 1 1

12 40 41 3 3

Consensual rating 33 29 3 5

25

Table 5: Summary of individual failed items on the CBTS-CYP and CTS-R

C–BTS-CYP CTS-R

Number of times failed Number of times failed

CBTS-CYP – Item Study 1 N=12 raters

Study 2 N=48

recordings

CTS-R Item Study 1 N=12 raters

Study 2 N=48

recordings

Partnership Working 2 0 Agenda setting & adherence 2 3

Right Developmental level 0 0 Feedback 2 2

Empathic 2 1 Collaboration 2 1

Creative 3 0 Pacing & use of time 0 2

Investigative 3 2 Interpersonal effectiveness 1 1

Self-efficacy 1 2 Elicit emotional expression 2 2

Enjoyable and engaging 2 0 Elicit key cognitions 0 1

Assessment and Goals 5 1 Elicit behaviours 1 1

Behavioural Techniques 4 4 Guided discovery 4 1

Cognitive Techniques 3 3 Conceptual integration 3 6

Discovery Experiments 8 3 Apply change methods 8 3

Emotional Techniques 1 2 Homework setting 5 8

Formulation 1 5

General Skills 1 1

26

Figure 1: CBTS-CYP Framework

Assessment (A)

Formulation (F)

Cognitive (C)

Emotional (E) Behavioural (B)

Discovery (D)

General skills (G)