Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
1
Evaluating the Wider Outcomes of Schooling:
The Oasis ECHO Project
Technical Report
Dr Ruth Deakin Crick Professor Howard Green Steven Barr Adeela Shafi Dr Wen-Jung Peng
2
CONTENTS
Acknowledgements ......................................................................................................................................... 5
Introduction .................................................................................................................................................... 6
Background and Rationale ............................................................................................................................... 6
A Review of the Literature ............................................................................................................................. 10
Evaluating the outcomes of schooling .................................................................................................................. 11
Contextual Value-Added ....................................................................................................................................... 11
Multi-level modelling ............................................................................................................................................ 12
School effectiveness and school improvement (SESI) .......................................................................................... 14
The use of quantitative and qualitative data ........................................................................................................ 14
Wider outcomes of school .................................................................................................................................... 16
Effective leadership .............................................................................................................................................. 18
Moving forward .................................................................................................................................................... 19
Reframing Schools as Complex Living Systems .............................................................................................. 20
Understanding schools as complex systems ......................................................................................................... 22
Viewpoint One: Leaders leading learning and change in the community ............................................................ 23
Viewpoint Two: Teachers as learners ................................................................................................................... 24
Viewpoint Three: Students as learners - deep learning as a complex psycho-social system ............................... 26
The challenge of complex data collection and representation for schools ..................................................... 28
CHOICE OF APPROACH TO RESOLVING COMPLEXITY IN LEARNING COMMUNITIES ........................ 29
SYSTEMS THINKING FOR RESOLVING COMPLEXITY AND MANAGING UNCERTAINTY ...................... 30
RESEARCH DESIGN AND METHODOLOGY .............................................................................................. 31
SYSTEMS DESIGNING AS IMPROVEMENT RESEARCH DESIGN ................................................................ 31
CASE STUDY SAMPLE ....................................................................................................................................... 32
PHASING OF RESEARCH ................................................................................................................................... 32
STEP ONE: DESIGN MODEL AND MEASUREMENT PARAMETERS ........................................................... 33
STEP TWO: COLLECT AND ANALYSE ALL DATA .......................................................................................... 36
3
MAPPING THE DATA STRANDS ONTO THE CRITICAL SUCCESS INDICATORS ...................................... 40
STEP THREE: GENERATE AND ANALYSE PERIMETA MODEL .................................................................... 45
STEP FOUR: FEEDBACK TO LEADERS AND EXPLORE SOLUTIONS .......................................................... 50
STEP FIVE: EMBED NEW PRACTICES ............................................................................................................ 51
SUMMARY ............................................................................................................. Error! Bookmark not defined.
Summary ............................................................................................................................................................... 51
Data Analysis: Social science.......................................................................................................................... 52
Strand One - Development of students as learners in Year 7 ..................................................................................... 52
Strand Two - Student perceptions of the impact of the Education Charter ............................................................... 59
Strand Three - Stories of transformation .................................................................................................................... 70
Strand Four - Post-16 transition and progress to adulthood ...................................................................................... 72
Strand Five - Quality of relationships between parents/carers and the Academy ..................................................... 84
Strand Six - Learning of teachers: impact of CPD on the quality of classroom practice ............................................. 90
Strand Seven - Impact CPD on the quality of leadership and management and classroom practice ......................... 99
Strand Eight & Nine - Key Performance Indicators & Key outcome for Year 11 students – GCSE results ................ 108
Hiearchical process model .................................................................................................................................. 134
Strand One - Development of students as learners in Year 7 ................................................................................... 134
Strand Two - Student perceptions of the impact of the Education Charter ........................................................... 138
Strand Three - Stories of transformation .................................................................................................................. 142
Strand Four: Post-16 transition and progress to adulthood ..................................................................................... 143
Strand Five: Quality of relationships between parents/carers and the Academy .................................................... 146
Strand Six: Learning of teachers: impact of CPD on the quality of classroom practice ............................................ 149
Strand Seven: Impact of CPD on the quality of leadership and management and classroom practice .................... 155
Strand Eight and Nine: Context for student learning - Key Performance Indicators and GSCE results .................... 158
Triangulation: HPM and Social Science ............................................................................................................... 158
Discussion of Findings .................................................................................................................................. 159
ECHO Strand 1 - Development of students as learners in Year 7 ............................................................................. 159
ECHO Strand 2 - Student perceptions of the impact of the Education Charter ........................................................ 161
4
ECHO Strand 3 – Stories of transformation .............................................................................................................. 162
ECHO Strand 4 - Post-16 transition and progress to adulthood ............................................................................... 162
ECHO Strand 5 – Quality of relationships between parents/carers and the Academy............................................. 164
ECHO Strand 6 - Learning of teachers: impact of CPD on the quality of classroom practice ................................... 165
ECHO Strand 7 - Impact of Continued Professional Development (CPD) on the quality of leadership and
management and classroom practice ....................................................................................................................... 166
Usefulness of Perimeta data for leadership decisioning ......................................... Error! Bookmark not defined.
Practical and Technical Issues for Development ................................................................................................ 167
Conclusions and Next Steps ......................................................................................................................... 169
Bibliography and References ....................................................................................................................... 172
APPENDIX .......................................................................................................... Error! Bookmark not defined.
APPENDIX 1 GLOSSARY OF TERMS ............................................................................................................. 179
APPENDIX 2 OASIS CHARTER ....................................................................................................................... 179
APPENDIX 3 CRITICAL SUCCESS INDICATORS (CSI) ................................................................................ 181
APPENDIX 4 QUESTIONAIRES ...................................................................................................................... 182
STRAND ONE – STUDENTS SELF-REPORT QUESTIONNAIRE ON LEARNING POWER .............................. 182
STRAND TWO - STUDENTS PERCEPTIONS OF EDUCATION CHARTER ....................................................... 182
STRAND THREE – STUDENT NARRATIVE IN TERVIEWS ................................................................................. 184
STRAND FOUR – YEAR 11 LEAVERS PERCEPTIONS ........................................................................................ 185
STRAND FIVE - PERCEPTIONS OF PAREN TS AND CARERS OF THEIR CHILDREN'S ACADEMIES .......... 186
STRAND SIX - LEARNING OF TEACHERS ........................................................................................................... 188
STRAND SEVEN - SENIOR LEADERSHIP ............................................................................................................ 191
APPENDIX 5 FULL PERIMETA RESULTS ................................................................................................................ 195
APPENDIX 6 ......................................................................................................................................................... 211
5
ACKNOWLEDGEMENTS
The authors of this report would like to thank the following for their valuable contributions to the
ECHO Project:
The University of Bristol
Oasis Community Learning
Jerusalem Trust
Dr Helen Jelfs who undertook analysis of the teacher and student interview data
Claire Wilson who coordinated data gathering from the Academies at Oasis Community
Learning
John Fuller, Director of Education at OCL, for his support throughout the Project
The Principals and Staff of Oasis Academies, Enfield Academy, Lords Hill Academy and
Wintringham Academy for taking the time to be a part of this project. Academies have been
anonymised within this report.
6
INTRODUCTION
A continuing challenge for the education system is how to evaluate the wider outcomes of schools.
Academic results are important but other, less easily quantifiable measures of success make for a
complete education. For example, the development of students as life-long learners, employability
skills, citizenship, self-confidence, teamwork and emotional wellbeing are widely recognised as
essential qualities for individual success in adult life and for social cohesion. Unless methods are
found to evaluate these broader outcomes, the education system will continue to focus on a single
measure of school effectiveness: test/exam results. This report describes the rationale,
methodology and findings of a pilot research project that applied hierarchical process modelling to
schools as complex living systems, using software developed by engineers at the University of
Bristol, called Perimeta. The aim was to create a systems design which accounted for the full range
of outcomes valued by each school, collect evidence of success – in the form of quantitative,
qualitative and narrative data – and to model this using Perimeta software which returns visual
analytic feedback against each outcome in the form of the Italian flag. Red represents what is not
working, green represents what is successful and white represents what is not known, and is
therefore an area for organisational learning and development. The project involves three
Academies in the UK. The systems design which was developed to underpin this pilot study, is one
which recognises that the purpose of the school is to facilitate the learning and achievement of all
students and the core processes which are essential for fulfilling this are: leadership learning,
teacher learning and student learning.
BACKGROUND AND RATIONALE
Ten years ago Macbeath and McGlynn (Macbeath and Mcglynn, 2002) reviewed thirty years of
school effectiveness research and described the shift that had occurred from evaluating schools as
whole units to a more specific focus on what is happening in individual classrooms. They argued
that this shift should be complemented by a wider focus on school culture:
‘It is not a matter of either/or: school or classroom, management or teachers, teaching or learning. Measuring effectiveness means sharpening our thinking as to where we should give most attention and invest our energies at any given time and in the light of the priorities we pursue. And as we get better at it we recognise that in good schools the boundaries between different levels become so blurred that they defy even the most inventive of statistical techniques’. (2002:6)
They then go on to describe a model of evaluation that puts pupil learning at the centre, but set in
the context of a school culture that sustains staff learning, leadership that creates and maintains
the culture and an outward-facing dimension involving home and community. They continue:
7
‘In deciding what to evaluate there is an irresistible temptation to measure what is easiest
and most accessible to measurement. Measurement of pupil attainment is unambiguously
concrete and appealing because over a century and more we have honed the instruments for
assessing attainment (and used them) for monitoring and comparing teacher effectiveness’.
(2002:7)
School self-evaluation (SSE) involves schools evaluating their own performance in order to
propel improvement and, according to Ritchie (2007), to maintain a level of control on external
evaluations, within a target driven culture (particularly in the UK). School self-evaluation tends
to focus on leadership and its impact on school evaluation. When done rigorously it can provide
the tools to set objectives for school improvement so that all schools can be effective. The
distinction between assessment and evaluation is that the latter demands taking a step back
from a specific piece of work or programme of study and asking questions like: was the
experience worthwhile? What was learnt from the process? What might be done next time to
improve? How is the school developing effective learners? Assessment, on the other hand, is
more about measuring a more quantifiable and narrower set of school outcomes, most often in
the form of test or exam results (Sammons, 1999).
The early application of information technology as a tool to support data-gathering and analysis
for the evaluation of school performance, combined with a target-driven culture of school
improvement, has resulted in an ever more single-minded focus on quantifiable measures of
success in schools. Whilst the development of sophisticated statistical techniques such as multi-
level modeling (Goldstein, 1986) have attempted to address the complexity of the range of
variables operating at different levels within the system and the interactions between them, so
as to tease out a more nuanced account of what makes a school effective, there remains
however, a great reliance on assessment scores. Despite this recognition of the complexity of
evaluating school outcomes school evaluation has continued towards a reductionist focus on
test performance at the expense of a wider, more balanced range of outcomes. This movement
has been fuelled significantly by political imperatives. Davies (Davies, 2011) raises similar
concerns about the current restricted view of ‘success’ from the perspective of strategic
leadership. He suggests that, by focusing on too narrow a range of school performance
measures, the upward trajectory of pupil attainment might plateau:
‘Success can be seen in how children achieve academically, socially, spiritually, physically and emotionally; it is enabling children to be all they can be. The difficult question is how do you know that you have been successful? Standardized test scores, even when adjusted for value-added dimensions, tell only part of the story. Two challenges emerge. One is that measuring success by easily quantifiable measures is to ignore that some aspects of success are recognised by indicators which point to success, but do not by any means fully explain or measure that success. Secondly, approaches that make schools successful initially may not be the ones that are necessary to take them on to higher levels of performance, so that isolating what approaches lead to sustainable success is difficult. A good example of this is the difference between shallow and deep learning. Coaching children for standard
8
assessment tasks (SATS) tests may increase short term results and the school would be considered successful. However, instead of putting in enormous efforts every year to boost results, a longer-term and more sustainable approach would be to involve ‘deep learning’ approaches that develop a learning culture in individuals and the school’. (2006:12)
However as technology has continued to develop there are new opportunities emerging for the re-
presentation of complex data and the development of learning analytics which offer new ways of
responding to complexity in learning communities (Buckingham Shum and Deakin Crick, 2012).
Combined with insights into systems thinking, systems design and systems modelling developed in
the corporate sector and engineering in particular, new technologies offer a potentially richer
approach to evaluating the wider purposes of education, taking a broader view of both processes
and outcomes which is a better representation of the ‘messy reality’ of life in learning
organisations.
The ultimate aim of this project is to develop useful ways of assessing complex processes in
learning communities which encourage powerful learning and feedback at all levels of the learning
system. What we are seeking for is a richer approach to evidence based self-evaluation which will
enable a holistic approach to learning and performance, a participatory culture of high aspirations
and deep learning and a context of continual improvement where standards of pupil attainment
continue to improve. School self-evaluation can provide the tools to set objectives for school
improvement so that all schools can be effective. Leadership has been shown to be a significant
component of school improvement and research has been continuing to support this notion
through the decades (see Hallinger & Heck, 2010). It reinforces the growing support over how
effective leadership supporting teaching and learning and promoting professional development and
change can lead to improving school outcomes.
However, this is a particular challenge for schools and groups of schools which have a broader view
of education and a desire to extend the measures of school performance beyond the easily
quantifiable. These include schools underpinned by alternative philosophies such as Co-operative
schools or Humanscale schools as well as many faith based schools. Writing from the perspective of
Church schools, for example, (Grace, 2002) refers to this challenge when he writes:
‘Such religious and transcendent purposes, a sense of mission rather than simply of provision, gives…‘depth to a schooling process that is otherwise dominated by a rhetoric of test scores, performance standards and professional accountability’.
Bryk et al (Bryk et al., 1993) also point to the need for more studies of the inspirational ideology
which animates many Catholic schools. They admit the scepticism which this idea produces,
especially in a research culture strongly influenced by secular and positivistic assumptions, but
make a powerful case for such research:
‘Some may question our claim of a causal role for this inspirational ideology ….unlike the effects
of academic organization or school structure, which can be largely captured in regression
9
analysis and effect sizes, estimating the influence of ideology is a more complex and less certain
endeavour. Ironically, these effects are harder to study and yet also more pervasive … To ignore
the importance of ideology because it cannot be easily captured in statistical analysis or
summarised with numbers would be a serious mistake. Statistical analysis can help us to see
some things but they can also blind us to the influence of factors that are beyond their current
horizons’. (1993, 303-4)
This pilot research project takes on the challenge of evaluating the wider intentions and outcomes
of schools more systematically, drawing on quantitative, qualitative and narrative data. Good
schools have always sought the views of students and parents in taking a broader view about how
well the school is doing and planning for change. In the UK, OFSTED inspections continue to
monitor broader aspects of education like spiritual, moral, social and cultural development but this
evidence typically becomes secondary, rather than integral, when overall and final judgements are
made about performance. The fundamental questions behind this project are these: if we
acknowledge the importance of student attainment but also have the ambition to educate our
students for a set of broader outcomes, how can we know how well we are doing and what we
might need to do to improve? How can we do this in systematic, sustainable and convincing ways?
10
A REVIEW OF THE LITERATURE
Evaluating outcomes of schools, known as school effectiveness research (SER), originates from the
phenomenon of the ineffective school (Edmonds, 1979). It has been a specific field of study only in
the last 40 years, but in that time has been emerging with a growing international profile. With
education high on the agenda for most governments, evaluating the outcomes of schools has been
the subject of political debate not least for those countries where education is state-funded, with a
focus on improving school standards and political accountability (Teddlie and Reynolds 2000).
Although discussions around schooling and its effectiveness have raged for some time, research
began to be more focussed in the 1970s, particularly in the UK and US with the works of Coleman
(1966), Jencks et al (1972) and Rutter et al (1979). These studies argued that whilst schooling
generally makes a difference, the particular school attended has little impact on outcomes. These
early studies reinforced the sociological determinism argument that factors such as IQ, gender and
social background are more important for school outcomes than what actually happens in school.
They further argued that whilst some schools do seem to do better than others, it was more due to
their student intake than the effectiveness of the school. Unease and at times hostility (eg
Reynolds 1997) at claims made by these studies led to the development of the now established
field of school effectiveness research (also recently called educational effectiveness research EER to
encompass wider education services such as pre-school, further and higher education), with a
dedicated journal School Effectiveness and School Improvement established in 1990.
Different schools had differing levels of effectiveness and in order to address the issues brought to
the fore by the earlier studies, it was important to disentangle the complexity of factors which
contribute to school outcomes. These factors include ability, prior attainment, personal
dispositions, age, gender and family background amongst others, which combined with educational
experiences in school impact on later attainment progress and development (Sammons, 1999). The
multitude of factors and their interactions have meant that school effectiveness research has had
its fair share of issues, both on philosophical and methodological grounds. Some of these will be
explored in the next sections.
School effectiveness research is not to be confused with school improvement (SIR) research which
presents itself as a separate and distinct domain, specifically exploring how schools can be
improved. How schools improve is a ‘process and not an event’ (Reynolds et al., 1996 p.14),
therefore, cannot be measured in the same ways as school effectiveness. It is because school
effectiveness research and school improvement research are conceptually different in several ways
(Reynolds et al., 1996) that they have maintained a parallel existence as domains of enquiry despite
being essentially components of the same thing: how to make schools better. However, if they are
two sides of the same coin then there must be more connecting the two fields of research. In
recognition of this there is growing trend of bringing both these fields together and this will be
explored later in this literature review. But first it would be useful to examine the issues in school
11
effectiveness research, as it is this which has traditionally been most interested in evaluating the
outcomes of schooling.
EVALUATING THE OUTCOMES OF SCHOOLING
Whilst the research in evaluating outcomes of schools has grown in stature, it has been subject to
much debate on philosophical and methodological grounds. The philosophical and political debate
is focussed around (i) policy issues, where raising standards and increased accountability became
the main discourse and (ii) the philosophical debate which centred around epistemological issues:
values in education, the purpose of schooling, what is a ‘good school and so on. These are justified
and on-going debates (White & Barber 1997) and influence a third domain of enquiry: (iii) the
methodological debate. It is the methodological debate which is the main focus of this review.
Mainly led by statisticians, the debate surrounding methodology in school effectiveness research
centred on the analyses of data, stemming largely from the controversial Rutter et al (1979) 15000
Hours study. This study was influential, primarily because of the claim that pupil attainment was
due to social background and that school made little difference to life chances. However, the study
was severely criticised for its small sample size and, importantly, because it did not fully take
account of the complex nature of schools and the various impacts on outcomes in its statistical
analysis (Heath & Clifford, 1980; Sammons 1999). This sparked a growth in sophisticated statistical
analyses of school effectiveness data, such as multi-level modelling, comparing various levels of
school and student data, both between and within schools. It has become an essential and integral
part of the research on school effectiveness (Goldstein 1987).
CONTEXTUAL VALUE-ADDED
School effectiveness can be defined as a school where students progress more than might be
expected, given the school's student intake. In that sense, an ineffective school is where students
make less than expected progress, given its intake (Mortimore 1991a). This definition emphasises
the importance of ‘intake’ and refers to the now familiar term ‘value-added’.
It is fairly well established that raw-score indicators of pupil attainment are not a reflection of the
school effect, but are more an indicator of the school intake (Gorard, 2010). It has been
acknowledged by governments and policy makers that to compare schools based on these raw
scores is unfair and an inaccurate reflection of a school’s effectiveness. Consequently contextual
value-added (CVA) is a regular feature in the discussions of school effectiveness for researchers,
governments, policy makers and to a certain degree parents. The CVA scores take account of the
different starting points of a pupil and include indicators such as prior attainment, socioeconomic
status (SES) which is usually assessed by eligibility of students for free school meals (FSM), school
size and pupil mobility amongst others. School effectiveness research will include such data or
variations of it in their analysis of school effects and as claimed by the DCSF (2007) provided a much
fairer and more valid way to compare school effectiveness.
12
Contextual value added (CVA) is calculated using a complex statistical analysis (see Gorard 2010 for
a more detailed description) and has become the accepted way in which to ensure schools are
compared on a more like-for-like basis - indeed it is infinitely better than using raw scores as
indicators. However Gorard (2010) has argued in his recent paper that it is time to move on from
school effectiveness research which relies solely on CVA, primarily on the basis that CVA
calculations assume ‘perfect datasets’. In reality, data is often missing or data from different
datasets cannot be matched appropriately. For example, data from Pupil Level Annual School
Census (PLASC) and National Pupil Database (NPD) do not always map neatly on to one another.
Whilst, data analysts can claim that these are random and that they can statistically account for or
‘clean’ such data, it does not remove the issue that the data is not complete or of good quality. If
data at this early stage is incomplete or inaccurate, then the value of the CVA needs to be
questioned (Rowe, 2009).
Furthermore, Gorard (2010) advocated that such data limits what education is all about. He argues
for a refocus in enquiry into school effectiveness onto school processes, fostering outcomes other
than the tradition attainment score. These are often referred to as the non-cognitive outcomes of
schools, referring to pupil development in the area of social justice, aspirations and citizenship – all
of which are largely ignored in school effectiveness research. This could be because such a focus
would rely more on qualitative data which might be difficult for the quantitative researcher to
genuinely embrace. The complexity and vastness of the factors which affect school outcomes has
pointed towards the need to collect data in the form of ‘facts and figures’ from a wide range of
schools which has driven the movement towards large scale quantitative data collection.
Additionally, different value-added models themselves can yield different results. Thomas &
Mortimore's (1996) study compared different multi-level models and argued that if the data is rich
and wide-ranging then the school-context is not significant in predicting pupil outcomes. But
otherwise, even CVA seems to explain pupil intake only if the data is not of a good quality,
reinforcing Gorard’s (2010) point about the quality of data. Their analysis also emphasised school
level effects in terms of the effectiveness of different departments, especially for pupils with
differing levels of prior attainment. This reflected the growing plethora of criticisms of SER on the
use of aggregated data at the school level because an aggregate comparison of schools risks the
under estimation and potential misrepresentation of school effects (eg Willms & Raudenbush 1989;
Goldstein 1986). The development of multi-level modelling has provided a tool to overcome some
of the criticisms levelled at the methodology in SER.
MULTI-LEVEL MODELLING
The traditional statistical analysis used in school effectiveness research was OLS regression and
whilst it enabled sufficient analysis of data, it was criticised that it did not truly enable analysis of
the different levels of effect on school outcomes, producing over or under-estimates of what was
really reflected in a dataset. The development of multi-level modelling in the 90s as a statistical
13
package available commercially allowed data to be analysed at several levels (Fitz-Gibbon 1996).
Multi-level modelling enabled analysis to be conducted at sector, district, school and student level
across different time points with up to 15 levels available. Many considered it to be revolutionary
in the way it has provided a solution to many of the methodological criticisms of school
effectiveness research (Teddlie, Reynolds & Sammon 2000). Described as the answer to the issue
of units of analysis in school effectiveness research and that of comparing data across different
time points, it appeared to be the solution to many methodological problems. It also meant that
longitudinal studies could be assessed appropriately using this multi-level technique, resulting in a
growth in longitudinal studies (eg Mortimore et al, 1988) which addressed another criticism of
school effectiveness research (Teddlie and Reynolds, 2000). Furthermore, there has been a
growing tendency to focus, not just on schools as a unit of analysis, but on individual classrooms
(MacBeath and Glynn, 2002) and the role leadership plays in influencing what happens in a
classroom (Sammons, 2011), demonstrating how school effectiveness research has become more
sophisticated and detailed.
Whilst multilevel modelling was considered ground-breaking in terms of school effectiveness
research, more recently the ‘honeymoon period’ has been wearing off. Revisiting many of the pre-
multilevel analysis data, several studies have revealed similar results, whichever the mode of
analysis, especially at the school level (eg Fitz-Gibbon 1991a, 1995a, 1996; Gray et al 1995),
particularly if the size of the unit of analysis got close to 30. This seems to suggest that although
multilevel data has provided greater opportunities for analysis, more traditional methods such OLS
regression analysis (admittedly a more simpler analysis) are still useful. Consequently, Teddlie &
Reynolds (2000) argue that older studies which were disregarded because they used simpler
analyses should be reconsidered so that a decade or so of research is not lost.
The development of sophisticated methods of data analysis, such as multilevel modelling, structural
equation modelling, growth curve modelling, regression-continuity modelling, mean there is still
great emphasis on quantitative data collection analysis. There has been concern about the
excessive interest in the statistical analysis of data (Scheerens, 1992) with comparatively very little
discussion of the use of qualitative data and its analysis which reflects a general methods debate in
educational research (eg Lincoln and Guba, 1985; Tashakkori and Teddlie, 1988, 2003; Patton,
1990). This debate has fuelled and advocated the use of more diverse research methods in school
effectiveness research.
Consequently, school effectiveness research (SER) is often criticised for measuring a very narrow
set of school outcomes: namely attainment test scores, adjusted for the contextual value added
(Sammons, 1999). But as already discussed, these are fraught with issues. A further issue in SER is
that schools are complex communities in their own right where the student experience is far more
diverse than what can be reflected in the measurement of attainment scores (Levin, 2012). In that
sense, attainment in the form of test scores cannot be the whole picture of school effectiveness.
(Teddlie & Reynolds, 2000; Sammons, 1999). What needs to happen is for a deeper exploration of
14
what happens in schools in terms of the non-cognitive outcomes of schools, such as citizenship,
social cognition and well-being in addition to academic outcomes (Van de Wal, 2007). These
outcomes are often best explored using qualitative data alongside the quantitative. However,
these issues have usually been the domain of school improvement research (SI) rather than SER,
serving to explain the apparent professional polarisation of the two genres of research (Gray et al
1995).
SCHOOL EFFECTIVENESS AND SCHOOL IMPROVEMENT (SESI)
The rise of school effectiveness research (SER) stimulated the growth of school improvement (SI)
research but attempts to combine them have not always been successful (Creemers, Kyriakides and
Sammons 2010). SI primarily uses qualitative data often in the form of case histories and narratives
with an emphasis on practical knowledge and engagement with practitioners. Chapman (2011)
emphasised that the SI community advocate that improvement comes from within an organisation
not beyond it.
Until fairly recently, the field of school effectiveness research (SER) and school improvement have
generally been treated as distinct and separate with their own histories and traditions. This is
somewhat surprising because school effectiveness research can inform school improvement
practices which can then propel further research based on what is revealed at school improvement
level (Day et al, 2008). Thus, they do appear to be two sides of the same coin, although tensions
between the two have meant that they have often been treated distinctly where SER tends to have
its focus on exploring differences between schools and their outcomes, adopting a primarily
positivist stance (Sammons, 1999). The main form of data collection is quantitative and the results
are more about a picture of ‘what is’ which can be compared to other similar schools. In other
words, SER presents a picture of the effective or ineffective school and what characterises it as
such. School improvement research, on the other hand, is more focussed on a school and its
process of improvement (Chapman et al., 2011). The research tends to use more qualitative forms
of data with a concentration on teachers, school processes and the journey of school improvement
rather than the outcomes. In short, SI research is more focussed on making schools a ‘better place’
(Reynolds et al, 1996). Because of their different epistemologies and methodologies, they have
tended to remain distinct and have perhaps become a victim of academic ‘paradigm wars’ (Gage,
1989). However, as mixed methods research is becoming more widely used and described as the
third methodological community (Johnson et al 2007), minimising the weaknesses of each method
alone, educational researchers are increasing using mixed methods research designs, attempting to
combine school effectiveness and school improvement research (Teddlie & Sammons, 2010).
THE USE OF QUANTITATIVE AND QUALITATIVE DATA
As school effectiveness research seeks to identify and measure school effectiveness, the vast
majority of its studies have focussed on large scale longitudinal or cross-sectional data from a range
of schools in order to draw conclusions. However, as discussed, it is becoming more appropriate to
15
consider schools as organisations which are continually changing and evolving and are thus dynamic
places (Mulford, 2013) in which constant change is the norm. If this is the case, then taking a
‘snapshot’ of a school as is done in SER in order to judge effectiveness, is at best limited and at
worst simplistic. However it also represents a more serious failing to address issues of equity by
valorising a narrow view of educational achievement (Chapman and Gunter, 2009).
A study by Day, Sammons & Gu (2008) combined qualitative and quantitative methodologies to
explore teacher lives’ and their impact on teacher effectiveness in school. They claim to go beyond
simple integration of qualitative and quantitative methods towards synergy, pointing out that
simple integration is not enough and that the findings from the qualitative and quantitative strands
of research need to be synergised, creating deeper understanding and meaning which would not be
possible by simple integration or the sum of the two strands. Through their study they explored the
impact of early/mid-career teacher and end of career teacher commitments, using both
quantitative and qualitative data, initially by integrating the two types of data throughout where
one informed the other. Their conclusions about life phase and relative effectiveness were more
informed through the synergy of the quantitative and qualitative elements. They point to using
mixed methodologies in a synergistic as well as integrative way to truly benefit from the two
methods and create greater and deeper understandings. However, their discussion did
acknowledge the difficulties, particularly because often the qualitative and quantitative aspects of a
study are separate until the integration stages. This can be due to practical difficulties, for example
collecting quantitative data and then trying to integrate it with the corresponding qualitative data
may mean there is often a time lag, preventing truly concurrent dialogue. Nevertheless, the
engagement with the data leading to new synergies was greater than the individual contributions
of qualitative or quantitative data and therefore, important in understanding teacher effectiveness
based on career phase. This study is an example of how school effectiveness research is making
tentative moves towards mixed methods.
However, whilst this paints a picture of a growing sense of harmony between quantitative and
qualitative methods, it would be naïve to assume that both methods are being given equal
weighting when it comes to comes to reporting and drawing conclusions (Creswell, Shope, Plano
Clark, & Green, 2006). This may be because traditionally researchers fall into either the
quantitative or qualitative camp, usually with differing ontological, epistemological and micro-
political commitments, resulting in the polarisation of these methods, or the incompatibility thesis
(Guba & Lincoln 1994). Because many school effectiveness researchers come from a quantitative
tradition and are attempting to include qualitative data, the end analysis often still ends up skewed
towards the quantitative (Creswell et al, 2006).
This could also be due to the perception of uncertainty in the appropriate analysis of qualitative
data. Quantitative data collection has its origins in the positivist tradition involving the collection of
numerical data (Reynolds, Bollen, Creemers, Hopkins, Stoll & Lagerweij, 1996), whereas qualitative
data is laden with meanings, interpretations, explanations of language, behaviour and culture
16
which are open to inconsistencies, uncertainties and alternative perceptions. This of course is a part
of the lure of an interpretivist epistemology, serving as a source of deeper understanding of
complex social phenomena (Mathison, 1988). Thus, in this way qualitative data often get
‘relegated’ to ‘soft data’ (Creswell et al., 2006) by a primarily quantitative researcher. Because of
this and the school effectivenss research community’s focus on identifying effective schools,
qualitative data has become the domain of the school improvement field. Whilst facts and figures
are also open to interpretation (Scott 2000), many researchers believed that relevant stakeholders
would only take notice of quantitative research (Reynolds et al 1996). For example, exam grades
are a statement of what a student has achieved in a particular school at a particular point in time.
This grade is assessed irrespective of the process or whether the student had a good day, whether
the teacher was effective and all the other issues that may have influenced that exam grade. By
not considering the processes which led to this grade, interpretation is limited and offers little in
the way of how a school might improve this exam grade. These factors could be quantified, for
example, in the number of revision sessions or the number of hours a school dedicated to
delivering the subject. But school effectiveness research tends not to explore the processes
involved in improving grades. In this way, school effectiveness research needs to move beyond
what makes an effective school to concurrently exploring what processes within a school make it
effective in order to drive school improvement practices (Stoll & Fink 1992). In other words, it is
time for school effectiveness research and school improvement research to be part of the same
story. But in order for this to be the case, there needs to be greater acceptance of the methods
traditionally used by each of the research communities (Morgan, 2007).
This should not be impossible as adopting mixed methods in SER and SI research is compatible with
multilevel modelling which focusses in on different levels, such as the school, classroom, teacher,
student and which are inter-connected and/or nested within one another. Qualitative methods can
be employed to explore these inter-connected relationships in a deeper way, allowing more diverse
research questions to be addressed and which the quantitative method alone stops short of
(Reynolds et al 1996). In order to explore the complexity of SER and the range of levels which
multilevel modelling aimed to address, it would be appropriate to use qualitative data but which
would have to be fully integrated into the research design creating synergy in the findings which
would be greater than the sum of the qualitative and quantitative data analysis. It would also
bridge the gap between SER and SI making them more integral and reciprocal. The result could be
more effective and more improving schools (Mortimore 1991).
WIDER OUTCOMES OF SCHOOL
NON COGNITIVE OUTCOMES
Wider outcomes of schooling have become more interesting to researchers as the agenda for
education (in the UK at least) now includes, for example, citizenship as part of the National
Curriculum since 2002. Personal, social, health and economic education (PSHE) is also a non-
17
statutory subject, but which many schools choose to incorporate because the Office for Standards
in Education (OFSTED) assesses schools on this as one of their criteria. These are not academic
outcomes, but ones which schools and other educational establishments are expected to
incorporate into their education programmes. The 2010 Education White Paper emphasised the
importance of PSHE in helping young people develop the skills needed to make important decisions
in life including a healthy lifestyle and financial choices. Non-cognitive outcomes, such as
citizenships, social cognitions, well-being are also seen as important element of education (eg
Reynolds, 1992 ; (Isac, Maslowski, Creemers, & van der Werf, 2013) and as advocated by Teddlie
and Reynolds (2000), school effectiveness research needs to be multidimensional incorporating
both cognitive and non-cognitive outcomes (Opdenakker & Van Damme, 2000), especially as claims
are being made that effective social skills are more likely to determine life chances in the 21st
century (Mulford, 2013; Holmlund & Silva, 2009) This reinforces the call for using quantitative and
qualitative data in order to explore these phenomena more deeply.
However, wider outcomes of school have not featured in school effectiveness research for several
reasons. Firstly, they are not easily quantified or measured (Knuver & Brandsma, 1993), secondly,
because politically school systems have focused on academic attainment rather than the personal
and social development of the individual and so questions as to whether schools should be held
accountable for these are often raised (Knuver and Brandsma (1993). Thirdly, because such
research would require qualitative data which is not always considered easy to collect or analyse
and stakeholders do not always consider the findings seriously (Reynolds, 1996). Nevertheless,
recent research into non-cognitive outcomes has been growing, although has often been in relation
to how they may or may not be correlated with cognitive outcomes i.e. academic attainment,
rather than as outcomes in themselves. For example, the Van Landeghem, Van Damme,
Opdenakker, De Frairie, & Onghena (2002) explored the effect of school and class on non-cognitive
outcomes and reiterated the findings of other studies, that an effective school does not necessarily
equate to a school with effective non-cognitive outcomes (Opdenakker & Van Damme, 2000). They
used multilevel modelling on a range of survey data, but Van der Wal & Waslander (2007) discuss
how neither these nor their own study were able to draw firm conclusions on the trade-off when
focussing on either cognitive of non-cognitive outcomes. This reflects the notion that the
relationship between cognitive and non-cognitive outcomes is yet to be fully established. Van der
Wal & Waslander's (2007) research also used survey data and multivariate multilevel analysis. It is
noteworthy that few studies on school effectiveness research, even those which focus on non-
cognitive outcomes, make use of more qualitative data.
Furthermore, if at the school level, data is collected via interviews or case stories (qualitative), it is
not always used for setting objectives for improving outcomes. Rather the data tend to be used to
inform school improvement processes (Harris, 2001). This does not necessarily translate into a basis
from which to set objectives which are still largely determined by school effectiveness studies. This
once again points towards the need to integrate school effectiveness and school improvement
research so that their relative strengths are combined, and both contribute to setting objectives for
18
school improvement. Armstrong et al (2012) argue that the field of school effectiveness and
improvement research requires a dramatic change and should be renamed educational
effectiveness and improvement (EEI) encompassing all educational establishments, so that
effectiveness and improvement go hand in hand.
EFFECTIVE LEADERSHIP
A further development in school effectiveness and school improvement studies is a focus on school
leadership stimulated by the study of academically improved and effective schools. It has been
shown that effective and improved schools have a clear vision of their goals and which actions to
take to reach them (Hodges 2000). Leadership studies have been moving towards the centre of
school effectiveness and school improvement research (Teddlie and Reynolds 2000). A mixed
methods study by Sammons (2011) (which was part of a larger 3 year project commissioned by the
DCSF and summarised by Day et al (2010), found that leadership effects influence student
outcomes by their influence on teachers and on the creation of a favourable school climate. The
wider project, of which this paper was a part, reported on 20 case studies of qualitative data which
were thematically analysed and resulted in a matrix of the main themes from across the studies.
This was a study which made an attempt to integrate and create a synergy of the qualitative and
quantitative strands. It is worth noting that reports or research commissioned by policymakers is
more likely to be mixed methodologies simply because there is likely less commitment to particular
ontological positions than in academic circles. Nevertheless, in the case of this research, the lead
researchers are academics which demonstrate a genuine move towards mixed methods research.
The focus on leadership and its impact on schools is associated with school self-evaluation (SSE)
which involves schools evaluating their own performance in order to propel improvement and
maintain a level of control of evaluation, within an externally imposed target driven culture
(particularly in the UK) (Ritchie, 2007). When done rigorously SSE can provide the tools to set
objectives for school improvement so that all schools can become more effective. Leadership has
been shown to be a significant component of school improvement and research has been
continuing to support this notion through the decades (see Hallinger & Heck, 2010). It reinforces
the growing support for effective leadership focusing on teaching and learning and promoting
professional development as a process driver for school effectiveness and improvement.
Hallinger and Heck (2010), however, also point out that many leadership studies rely on case
studies and cross-sectional surveys. They rightfully advocate the need for longitudinal studies in
order to document process and change. But whilst this is important, they fail to recognise the value
of the case study data which have informed leadership studies and have enabled the current
conclusions of leadership to be made. Nevertheless school self evaluation has also been shown to
be an effective way in which school leadership can be been improved. This is despite SSE often
being criticised for lacking validity and reliability (see Kyriakides & Campbell, 2004 for a detailed
19
critical analysis), emphasising that external evaluation has to be a part of an overall approach to
school self-evaluation (Kyriakides & Campbell, 2004).
Alternatively, other studies of leadership have continued to rely on quantitative data to study its
effects on classroom practice and ultimately student outcomes. For example, Leithwood & Jantzi
(2006) used survey data from 2,290 teachers from 655 primary schools to analyse transformational
leadership against student achievement as measured by SATs scored at the end of Key Stage 2.
They found that leadership did affect teachers’ classroom practice but not student achievement.
But this could be because student outcome was measured by the narrow focus of academic
attainment (SATs) scores which as discussed cannot present a complete picture of what are
essentially qualitative qualities of a school.
This further advocates the use of more qualitative and perhaps longitudinal mixed method studies
(Kyriakides & Campbell, 2004) which could help produce a model of lasting impact on school
improvement helping to set targets and objectives as a whole school approach.
MOVING FORWARD
This literature review has aimed to untangle the research on the outcomes of schools as there are
various layers which overlap and are mutually re-inforcing. These moved from school effectiveness
research which explores differences between schools (largely using quantitative data) to school
improvement research (which uses primarily qualitative data) to a focus on the study of leadership
and its impact on school improvement to the more recent study of school self-evaluation, which is
a still developing field. What appears evident is that research on how schools operate has
proceeded on parallel but often separate tracks: effectiveness, improvement, leadership or self-
evaluation. However, schools are learning communities and organisations with complex structures
and processes and while it is tempting to break down these components and study each part
individually, it is important to bring the parts back into a whole in order to fully understand
schooling and its wider outcomes in the fullest sense.
One of the reasons for the distinctions between the fields lies in the methodologies available and
throughout the studies in this review there has been an extensive reliance on either quantitative or
qualitative data. Their ontological and epistemological foundations seem to have encouraged this
divide to continue. This is despite the regular calls for employing more mixed methods designs
(Charles Teddlie & Reynolds, 2000) across the genres. Influential researchers such as Muijs (2006,
2012) do make efforts to point out the ‘risk’ of, for example, attributional bias, in qualitative data,
which refers to an interviewee attributing success to themselves or internal factors and setbacks
down to external factors to do with the organisation, policy or management. However, this
emphasises further that it is important that to move on from the polarisation of these
methodologies to work towards developing more sophisticated analysis tools for qualitative and
narrative data as well as for quantitative data. It is not sufficient to brush off qualitative data on
the pretext that it is difficult to analyse or that it cannot be ‘trusted’, rather it is an opportunity to
20
explore qualitative and narrative data analysis as a field of research in itself. This is the point at
which qualitative and mixed methods designs will truly flourish, where qualitative data will
genuinely inform how schools are evaluated in a wider sense, informing how schools set their
objectives for improvement.
Developments in computer packages for analysing qualitative data (QDA), such as NVivo or
MAXQDA have been extremely useful in helping manage large amounts of data and taking on the
burden of some of the labour intensive aspect. However, computer-assisted qualitative data
analysis software (CAQDAS) cannot ‘analyse’ the data or fully explore the intricacies of qualitative
data (Bazely, 2007), which is primarily a human activity because it involves meaning making and
interpretation. Others criticisms, particularly made by positivists, have been levelled at the
proximity the researcher has to qualitative data. This could be minimised by the use of CAQDAS
with their extensive search, access and coding facilities potentially reducing a reliance on ‘first
impressions’ when reading qualitative data (Garcia-Horta and Guerra-Ramos, 2007) and make for
more robust interpretations (Bergin, 2011). This makes the process of the analysis of qualitative
data more open to scrutiny and thereby greater transparency – addressing another criticism
levelled at qualitative data analysis (Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005).
However these packages are still being developed and are comparatively some way from the
sophisticated statistical packages designed for quantitative data. They, therefore, deserve further
exploration and development.
Evaluating wider outcomes of schooling is a complex challenge because of the complexity of factors
which are inter-connected and inter-related, and the differing 'truth claims' of different genres of
research. This means that it is essential to develop research designs and methodologies which truly
explore all facets of this complicated process of schooling.
REFRAMING SCHOOLS AS COMPLEX LIVING SYSTEMS
One of the big challenges emerging from this literature review is how to harness the synergy that
might be created through the inter-action of different approaches to knowledge generation and
use and their differing approaches to truth claims. Such synergy might better inform leadership
decisioning in schooling than a focus on only one method at the expense of others, because it
would reflect the complex reality of schools. However, not only does this challenge include the
knowledge generated by those studying schools (i.e. researchers) but, significantly, it includes the
experiential professional knowledge generated by leaders and even the connected intelligence of
the whole community, including students. This is surely a complex challenge which is impelling in
21C conditions of risk, uncertainty and diversity, and the ubiquitous use of technology and data
offers new opportunities to engage with it.
21
In order to address the main aims of this project it is thus important to consider schools in all their
complexity. Such complexity is a fundamental issue for education and schooling and, as has been
demonstrated in the review of the literature, is represents a challenge in educational research.
Schools have multiple stakeholders and multiple outcomes. A school’s core processes of student
learning and achievement are themselves complex and dynamic and cannot be reduced to, or
described by, a single variable. As Wheatly and McCombs (Wheatley, 1999, Wheatley and Kellner-
Rogers, 1998); (Mccombs and Whisler, 1997) argue, a school is a living system and in order to
understand improvement and change in schooling we need to take this into account. A complex
living system is one which is self-organising, purposeful, layered, interdependent and operating 'far
from equilibrium' (Davis and Sumara, 2006); (Checkland and Scholes, 1999). This means that there is
no single blue print for improvement, or single measure of success for school effectiveness,
because each school operates in a unique context, with unique individuals – what works in one may
literally not work in another. Leaders therefore need to be able to respond appropriately to their
context as ‘designers of learning’ rather than ‘deliverers of pre-determined curricula’. In other
words they need to be able to lead by the spirit, not the letter of leadership for learning (Marshall
and Drummond, 2006).
Research into complex systems offers some signposts for this enquiry. There are properties of
complex systems that form a set of principles which underpin learning design. Blockley (Blockley,
2010) identifies 'layers, feedback loops and processes' as key properties and emergence as the
unpredictable outcomes of the relationships and interactions of key processes within a system.
Emergence is at the heart of complex systems thinking - thus the challenge of dealing with
uncertainty and risk is a challenge at the heart of leadership. Fundamentally this requires us to
recognise that there are limits to what we can know and therefore predict. Knowing what we don’t
know and acting accordingly is 'humility', a core virtue for both leaders and learners, for without
humility there can be little new learning or change. The ability to purposefully adapt and change
throughout a lifecycle is what makes an organisation or an individual resilient and sustainable. At
the heart of this resilience and sustainability is learning - self-aware, purposeful, conceptual
reorganisation - at all levels of the system: students, teachers, leaders and parents/carers.
In terms of developing evaluation models for schools, systems thinking also demonstrates that a
reductionist focus on the measurement and improvement of a single variable (for example a test
result) distorts both the process and the outcome of the system (James and Gipps, 1998); (James et
al., 2007);(James et al., 2007, Assessment Reform Group, 1999, Reay, 1999). As Mason (Mason,
2008) argues
'trying to isolate and quantify the salience of any particular factor is not only impossible, but also wrongheaded'. Isolate, even hypothetically, any one factor and not only is the whole complex web of connections among the constituent factors altered - so is the influence of (probably) every other factor too'. (2008:41)
22
We know for example, that an over focus on high stakes summative testing and assessment not
only distorts how teachers teach, but it also distorts student learning and creates an 'own goal'
since it depresses student motivation for learning (Harlen and Deakin Crick, 2003b, Harlen and
Deakin Crick, 2003a). It does more than simply relegating wider outcomes to second class goals – it
actually scores an own goal.
Not only is it important to focus on a range of processes and variables in schools which are deemed
to fulfill a particular purpose, but it is also important from a systems perspective to understand the
whole, the parts and how they interact. These are described by Goldstone (Goldstone, 2006) as
'contextualised' and 'decontextualised' aspects of a system. In developing contextualised accounts
learners and their environments (students, teachers, leaders and organisations) are seen as parts of
a single whole. How someone learns depends in part on the larger system in which they learn.
Elements of the system (both individual learners and other system elements) cannot be understood
independently. Rather, the interactions of the elements give rise to emergent behaviours that
would not arise through their independence.
A key concern with a singular focus on contextualisation is its inability to lead to generalization
(Goldstone, 2006). But complexity theorists have also identified some principles of complexity that
can be applied to different cases from seemingly unrelated domains - for example, Blockley's ideas
of layers, feedback loops and processes, and the ideas of emergence and uncertainty. Thus, while
learning, teaching and leadership are contextualised, we can also understand them as
decontextualised and identify patterns which can be generalized across highly contextualised
instances. So these two apparently contradictory ideas – contextualisation and decontextualsiation
– are two aspects of a common process of conceptual reorganisation - i.e, learning (Goldstone,
2006, p.37). In other words, students, teachers and leaders must recursively consider general
principles and specific contexts in order to learn.1.
UNDERSTANDING SCHOOLS AS COMPLEX SYSTEMS
We now turn to the model of a school as complex learning system which has informed this pilot
study. A system is defined by its purpose (Blockley, 2010, Blockley and Godfrey, 2000) and the
primary purpose of a school is the learning and achievement of its students. As we have argued,
learning is a core property of a resilient system. Thus for schools, learning is both a core process
and a product (desired outcome). This makes the application of systems thinking particularly salient
for schools - in contrast for example, to a supermarket in which learning is a core process but
providing food services and generating profits are the desired outcomes. In understanding the
defining importance of purpose in a system, we also foreground the concept of direction, of
journey of dynamic change and of lifecycle. For an individual learner, a team or an organisation, this
1 For a further glossary of terms from systems thinking which are relevant to schools see Appendix 1
23
implies an intelligent shared direction - and thus leadership. An individual student who is taking
responsibility for their own learning and life story is exercising personal leadership towards a
chosen purpose. A team or an organisation which is moving towards a shared purpose is exercising
leadership through individual, team and organisational learning and change. For a school as a
complex living system, leadership and learning are thus core processes as well as desired outcomes.
Drawing on (Goldspink, 2007), (Bryk et al., 2010), (Deakin Crick et al., 2011, Deakin Crick et al.,
2010, Deakin Crick, 2009), we have identified three key processes in schools as learning
communities, which constitute sub-systems or layers of learning and change processes which
provide 'viewpoints' from which to understand the system as a whole. These are (i) leadership -
including both community and school (ii) teacher professional learning and (iii) student engagement
in learning and achievement. In the next section we present the rationale for selecting these three
viewpoints.
Table 1 Three sub-systems or viewpoints for schools as complex systems
VIEWPOINT ONE: LEADERS LEADING LEARNING AND CHANGE IN THE COMMUNITY
In their conclusions to an extensive international survey of educational leadership and management
Davies and West-Burnham (2003) highlight several challenges for school leadership in the future,
including equity and entitlement, social trends, policy and innovation, funding and the nature of
‘schooling’ itself, how and where it happens and what it comprises. They go on to argue for new
models of leadership that are fundamentally concerned with strategy, values and learning. Beare
(2001), looking to the future, suggests that schools will need a focus on learning as the prime
mission and professional leadership where the leader and leadership teams give highest priority to
the professional purpose of the school, personally and frequently monitor the learning
programmes, put time and energy into school improvement, give support to the staff involved with
learning programme (and) put tangible emphasis on instructional leadership.
The relationship between school leadership and learning has been explored in several studies (e.g.
National College for School Leadership 2004; 2010) which all concluded that the most successful
Teachers
Leaders Students
24
systems, based on measures of student engagement and attainment, prioritised staff motivation
and commitment, teaching and learning practices and developing teachers’ capacities for
leadership (Gunter 2001; Bottery 2004). Cochran-Smith (2003) and Darling-Hammond and
Bransford (2005) and several other studies have established the importance of effective teaching
for supporting enhanced student achievements, and there is now a great deal of evidence behind
the claim that leadership that focuses on the quality of teaching is crucial for maintaining and
supporting improvement in the quality of learning in schools (Robinson, Lloyd and Rowe, 2008). At
the heart of leadership for learning (e.g., MacBeath and Cheng, 2008) is the concern with making
schools learning organisations supporting teachers’ and students’ learning and with greater
outreach to the communities they serve. And, as Silins and Mulford (2002) found in their
comprehensive study of leadership effects on student and organisational learning, student
outcomes are more likely to improve when leadership is distributed throughout the school
community and when teachers are empowered in their spheres of interest and expertise. The
McKinsey Report (2007), derived from an international survey of the most successful education
systems, found that a focus on teacher recruitment and professional learning were more important
as determinants of success than funding, social background of students, regularity of external
inspection or class sizes. However, PricewaterhouseCoopers’ (2007) study into school leadership
sounded a warning note when it found evidence that many school leaders, as a result of external
demands, felt that they did not have sufficient time to focus on teaching and learning.
In Bryk et al’s (2010) research, the most effective school leaders were catalytic agents for systemic
improvement, synchronously and tenaciously focusing on new relationships with parents and
community; building teachers’ professional capacity; creating a student-centred learning
environment and providing guidance about pedagogy and supports for teaching and learning.
Goldspink's (2007) research identified that the leadership qualities required for complexity are not
among the typical selection criteria for principals. They include: a level of modesty and
circumspection and a capacity to question one’s own deepest assumptions while inviting others to
participate in critical enquiry. These personal qualities and the assumptions about leadership as a
core systems process which underpin them have not been widely applied in education and few
school leaders are familiar with the relevant investigative, dispositional and analytical processes
(Zohar 1997).
Viewpoint Two: Teachers as learners
Collaborative, classroom-based, research-informed professional learning and enquiry in schools
represents an important facet of the ‘deep structures’ of learning in organizations and their
communities. Professional learning is a vital pre-condition for school improvement through its
positive influence on teachers’ classroom practices and their students’ learning (e.g., Birman, et al,
2000; Cohen and Hill, 2001; Day and Leith, 2007; Desimone et al., 2002; Garet et al., 2001; Loxley et
al., 2007). Schools’ promotion of teachers’ participation in school-to-school and other networking
activity can support improvement-related activity in schools and classrooms through enabling
25
teachers to access and engage with an expanded pool of practice ideas, resources, and sources of
support, increased opportunities for mutual problem-solving, knowledge creation and transfer, and
a heightened sense of valuing professional achievement and accomplishment (e.g., Little and
Veugelers, 2005; Jackson and Temperley, 2007; Stoll et al., 2007; Katz and Earl, 2010). Thus, an
excavation of a school’s deep learning structures is likely to reveal that at the heart of those
structures is the teacher as learner and scholar (Shulman, 2011).
There has been a great deal of international interest in understanding how professional learning
can fulfil its potential for supporting school improvement. For example this question has been
addressed by Meiers and Ingvarson (2005) in Australia; Piesanen, Kiviniemi, and Valkonen (2007) in
Finland; Timperley and Alton-Lee (2008) in New Zealand; and Garet, Porter, Desimone, Birman, and
Yoon (2001) in the USA. However, despite the importance of teachers’ learning, a great deal of
teachers’ professional learning activities are ineffective (Sykes, 1996; Hanushek, 2005). Borko
(2004), Clarke and Hollingsworth (2001), and Timperley and Alton-Lee (2008) have all argued that
the problem stems from simplistic conceptualisations of teachers’ professional learning that fail to
consider how learning is embedded in teachers’ professional work and the complex institutional
and community contexts in which that work develops. An important purpose of the proposed
research is to develop theoretical and empirical understandings of teachers’ learning and practice
as a complex system (Clarke and Collins, 2007; Collins and Clarke, 2008; Curtis and Stollar, 2002;
Davis and Sumara, 2006; Weaver, 1948) and to help schools develop appropriate strategies for
supporting its emergence and scaling up.
Misunderstanding the nature of teacher learning by underplaying its complexity leads to focus on
the micro-context (individual teachers or individual activities or programmes) to the exclusion of
influences from meso (institutional) and macro (school system) contexts (e.g., Bore and Wright,
2009; Bottery and Wright, 1996). Adopting a complexity thinking perspective, we assume that
teacher learning does not emerge as a series of isolated events but simultaneously in the activity of
autonomous entities (teachers), collectives (school phase and subject groups) and subsystems
within grander unities (schools within school systems within socio-political educational contexts).
These nested systems and subsystems associated with teacher learning are interdependent and
reciprocally influential. As a result, to explain teacher professional learning, one must consider what
sort of local knowledge, problems, routines, and aspirations shape and are shaped by individual
practices and beliefs. How are these then framed by the other systems involved? Thus we construe
teacher learning as a complex system representing recursive interactions between systems and
elements that coalesce in ways that are unpredictable but also highly patterned (Clarke and Collins,
2007). Therefore, identifying emergent patterns of interaction within and between levels of activity
that would constitute an explanatory theory, here, of teacher learning as a complex system
requires variable-inclusive (as opposed to control) strategies for research, development, planning
and evaluation.
26
Complex systems need to be off balance in order to move forward. Wheatley (1999) in her analysis
of large institutions as complex systems, notes that organizational equilibrium is “a sure path to
institutional death”. Helping schools and groups of teachers become aware of the full range of
dissonance between their values and practices in relation to teachers’ and school learning and
leadership was an effective intervention for promoting change, growth and deep learning as part of
the Learning how to Learn project (Pedder et al., 2005; Pedder, 2006; 2007; Pedder and MacBeath,
2008). Further mapping of patterns of dissonance in schools nationwide was an important feature
of the State of the Nation CPD study (Pedder et al., 2010; Opfer and Pedder, 2011). Dissonance
between what teachers consider important for enhancing the quality of their students’ learning
opportunities and perceptions of current practice may result in what Woolfolk et al (2009) refer to
as ‘change-provoking disequilibrium’, further underlining the practical significance of attending to
relationships between values and practices. Argyris and Schön’s (1996) work on ‘theories of action’
illustrate that dissonance serves as a catalyst for schools to attempt to change their environment in
ways that better support learning.
Viewpoint Three: Students as learners - deep learning as a complex psycho-social system
Deep learning occurs when students choose to invest in processes of learning that are authentic,
personally owned and enable agency in processes of knowledge construction. Learning Futures
research (2010, 2011) identified 'authenticity, agency and identity' as key elements of pedagogy
which lead to engagement and depth in learning, rather than superficial recall or performance
orientation. Where these occurred, students described their learning as transformative for them as
individuals - they were authors of their own learning journey, in a process of 'becoming' (Seely
Brown 2009). Engagement in learning is necessary for depth of outcome, but it is a complex
construct. Fredricks et al's (2004) review identified the components of engagement most
commonly identified by researchers as behaviour and participation, (Fullarton, 2002; Willms, 2003),
emotion - a sense of belonging and value (Willms, 2003) and cognitive beliefs about learning and
achievement (Munns and McFadden, 2000). A range of studies provided evidence about how
these variables contribute to engagement including: involvement and wellbeing (Zyngier, 2004;
Shernoff et al., 2003; Goldspink, 2008); interest (Hidi and Renninger, 2006; Eccles et al., 1998)
student epistemic assumptions, including assumptions about fixed/variable intelligence (self-
theories) (Dweck, 2000; Baxter Magolda, 2004; Cano, 2005; Hofer, 2001; Schommer, 1990); meta-
cognitive skill which has been found to have a greater impact on outcomes than intelligence
(Veenman et al., 2006); and students’ learning power (Deakin Crick et al 2010; Deakin Crick & Ren
2011; Deakin Crick 2011). In addition to factors internal to the student, several studies identify the
influence of pedagogy and school climate on learner engagement: school culture and climate
(Anderson, 1982) and quality pedagogy (Newmann and Wehlage, 1995; Ladwig and Gore, 200;
Ladwig et al., 2007; Hattie, 1999; Hattie, 2003). Some aspects of pedagogy are particularly
important to the engagement of low SES and Indigenous learners in Australia (Gale et al., 2008;
Amosa et al., 2007; Zevenbergen et al., 2004) as are aspects of learning power and self reflection
(Deakin Crick and Grushka, 2010; Deakin Crick, 2009c; Goodson and Deakin Crick, 2009). Fredricks
27
et al. summarised their review by suggesting that the individual types of engagement (behavioural,
cognitive, emotional) have 'not been studied in combination, either as results of antecedents nor as
influences on outcomes' and that research has tended to use variable-centred rather than pattern-
centred techniques, cross-sectional rather than longitudinal (2004, p.87). The result is that we have
little information about the interactions between different aspects of learner engagement and little
information about the development and malleability of engagement over time.
Central to engagement is the idea that the learners themselves need to want to learn, and to
become aware of themselves as learners. Black and colleagues (2006) argued that a focus on
learning to learn and assessment for learning in schools is important particularly when it leads to
the promotion of ‘autonomous learning’ or ‘intentional learning’ (Bereiter and Scardamaila 1989).
Intentional learning implies agency and choice on the part of the learner and goes beyond the
acquisition of study skills and strategies, requiring practices which invoke the learner to take
responsibility for their own learning in a relational context. Hautamaki et al., (2002) also emphasise
the importance of learner agency and self-regulation.
Learning power is a central concept in engagement. It is a term used to describe that complex mix
of personal qualities that characterise effective learners and is defined as ‘a form of consciousness
characterised by particular dispositions, values and attitudes, with a lateral and a temporal
connectivity’ (Deakin Crick 2007). Learning power operates through relationships, where trust,
affirmation and challenge play a constitutive role and it is ‘storied’ in the trajectory a person brings
to their learning and in their hopes and aspirations. Successive empirical studies identified seven
dimensions of learning power: changing and learning, critical curiosity, meaning making, creativity,
learning relationships, strategic awareness and resilience and a self-assessment tool based on these
scales, which provides a framework for a mentored conversation which moves between the
learner, their story and a negotiated learning outcome (Deakin Crick, 2007; 2005, Deakin Crick et
al., 2004).
What has also emerged from these empirical studies is that the self-assessment of learning power
is a necessary but not sufficient aspect of a learning ecology. Other key factors include (i) the
quality of learning/assessment relationships; (ii) the creation of a locally owned language for
learning, including dialogue; (iii) the extensive use of metaphor and modelling; (iv) the use of
learning power dimensions as scaffolding for enquiry, (v) the re-sequencing of the content of the
curriculum, and (vii) the relationship between the personal/autopoietic and the public in relation to
learning. Learning power is thus a part of a ‘learning journey’ set in a complex living system. Seely
Brown and Thomas (2009) suggest that this journey is about ‘learning to become’ set in a
participatory framework, which embraces embodied and tacit knowledge as well as explicit
knowledge (Polayni 1967; Heron and Reason (1997).
These are some of the complex conditions necessary for deep learning, embedded in a critical
socio-cultural context (Habermas 1972; Freire 1972) in which students identify value and purpose
and are authors of their learning, embodied in a particular context in place and over time (Goodson
28
and Beista, 2010, Goodson and Deakin Crick, 2009, Goodson, 2009). Deep learning is described by
Bateson (1972) as third level learning, which involves personal transformation – rather than only
repetition (primary learning) or learning to learn (secondary learning). Deep learning is best
understood as a complex psycho-social system, with subsystems which include identity, experience
and story. The learner arrives at a learning opportunity already possessing a way of knowing and
being in the world which is the sum of their experience to date. Vygotsky (1978, 1962/1934)
described this as ‘perezhivanija’, the term used for accumulated lived emotional experience,
including values, attitudes, beliefs, schemas and affect. For Vygotsky, perezhivanija is the process
through which interactions in the ‘zone of proximal development’ are perceived by the learner,
described by Mahn and John-Steiner (2002) as a complex whole, a system of systems which
includes the inter-related and interdependent elements of participants, environments, artefacts
(such as computers, or tools) and context. Sfard and Prusak, (2005) suggest that the notion of
identity - ‘collections of stories about persons that are "reifying, endorsable by others and
significant" is the missing link between learning and its socio-cultural context. The challenge in this
project is to develop forms of pedagogy which attend to the complexity of student engagement in
learning and the concomitant depth of process and outcome, including (i) the selfhood, identity
and story of the student in community and tradition (ii) the personal qualities necessary to engage
with new learning opportunities (iii) the co- construction of knowledge, skills and understanding
and (iv) the competence to engage with authentic learning in the real world (Deakin Crick, 2011,
Deakin Crick, 2009b, Deakin Crick, 2009a).
THE CHALLENGE OF COMPLEX DATA COLLECTION AND REPRESENTATION FOR SCHOOLS
These three core processes inform the design of this pilot study. The challenge is to generate a
systems design for a school and then evaluate and improve its core processes and their outcomes
through data collection, analysis and rapid feedback in a way which enables leaders to make
decisions based on a wide and complex range of evidence and provide performance data for
accountability purposes. This involves complex data collection, representation and interpretation
which offers a challenge to traditional approaches in which a ‘post positive scientific method’
encourages a reductionist focus on a part, not the whole. Leadership requires the harnessing of
collective intelligence – and the speed and complexity with which data can now be manipulated
and represented presents unique learning challenges for leaders. Understanding the weight of
evidence which such data offers is a further challenge.
Complexity in learning communities can be understood in different ways. A problem can be
complicated (i.e predictable) but not complex. Complexity can be highly complex (unpredictable
and uncertain) evidentially complex (obviously so) and causally complex (one thing leads to
another). Figure 2 reproduces Barr's (2013) typology of complexity, and overlays these types of
complexity onto some of the problems of leading learning communities. These problems are not
29
confined to any one part of this diagram. Whilst they seem to be more ‘complex’ than ‘complicated’
in nature, the distinction is still an important one and relates to both axes of complexity – that is,
the problems of learning communities are a mixture of ‘highly complex’, ‘evidentially complex’,
‘causally complex’ and (merely) complicated.
Table 2 A typology of complexity in learning communities
CHOICE OF APPROACH TO RESOLVING COMPLEXITY IN LEARNING COMMUNITIES
The debate in the domain of school improvement and effectiveness studies has already been
shown to be influenced by significantly different worldviews within educational research. Often the
post positivist world view is described as 'objective' whilst an interpretive worldview is described as
'subjective' and these are set in opposition to each other. Morgan and Smircich (1980) set out a
‘network of basic assumptions characterizing the subjective-objective debate within social science’.
Set out against a linear six-point scale of approaches to social science from ‘subjectivist’ to
‘objectivist’ these assumptions concern ontology, human nature and basic epistemological stances.
From this network, the assumptions that most closely match the complexities of learning
communities described previously are selected and presented in Table 1.
Table 3 Selection of assumptions appropriate to complex problems of learning communities
(with assumptions after Morgan and Smircich, 1980)
Type of
assumption
Descriptions appropriate to complex
problems of learning communities
Position on subjective-
objective spectrum
30
Core ontological
assumptions
‘Reality as a contextual field of information’
‘Each school operates in a unique context’
(Davis and Sumara, 2006)
‘Reality as a social construction’
‘Complex web of connections among the
constituent factors’ (Mason, 2008)
‘Deep learning as a complex psycho-social
system’ (Learning Futures, 2010)
4: Borderline objective/
subjective
2: Somewhat subjective
Assumptions about
human nature
‘Man as social constructor’
Complex mix of personal qualities that
characterise effective learners (Deakin
Crick, 2012)
2: Somewhat subjective
Basic
epistemological
stance
‘To map contexts’
In Bronfenbrenner (Bronfenbrenner, 1989)
terms ‘System of systems which includes
the inter-related and interdependent
elements’
‘To understand how social reality is created’ ‘Helping schools and groups of teachers
become aware of the full range of
dissonance between their values and
practices’ (Pedder and MacBeath, 2008)
4: Borderline objective/
subjective
2: Somewhat subjective
1 describes the position of each assumption against Morgan and Smircich’s (1980) six-point scale
from ‘subjective’ to ‘objective’. The pattern of these choices across the spectrum determine that a
multi-methodology (Mingers and Brocklesby, 1997) is needed for dealing with the diversely
complex problems of learning communities. Morgan and Smircich’s network of assumptions aligns
with the following research methods, which are subsequently considered in more detail for
inclusion in the research methodology:
Hermeneutic analysis of results of qualitative and narrative surveys of the experiences of
stakeholders including students, parents/carers, teachers and principals;
Historical quantitative analyses of schools performance data for the same students; and
Interpretive contextual analysis of experiences and performance data.
SYSTEMS THINKING FOR RESOLVING COMPLEXITY AND MANAGING UNCERTAINTY
Systems thinking offers a way of both getting to grips with complexity and a forum for engagement
and participation with the leaders, students and teachers. Systems thinking aims to overcome
traditional barriers to such development by seeking to make complex ideas and data accessible in
such a way that innovation and progress are possible. It does this by attending to systems design –
31
of an organisation or a process – and modelling that complexity through a visual analytic. This then
provides the salient parameters for a measurement model which informs intelligent leadership. `
Of the many systems thinking approaches available (Jackson, 2000; Rosenhead and Mingers, 2001),
the most appropriate for supporting collaborative development and leadership decisioning in
complex systems similar to learning communities, is hierarchical process modelling (eg Davis et al,
2010) which has three important characteristics:
Visual/ effective reporting of complex ideas and information is enhanced using hierarchical
mapping of processes and an ‘Italian Flag’ model of evidence;
Assimilating all forms of evidence – data, prediction and opinion; and
Facilitating access to key information required for informed discussion, innovation and
agreement.
The Perimeta software was developed by University of Bristol as a research tool for systems
thinking. It supports collaborative development of solutions to complex problems by providing a
highly visual interface for understanding complex cause-and-effect and complex evidence.
Commercial applications of Perimeta have been developed. Perimeta can be described as:
A learning analytic designed to model diverse and complex processes
Driven by stakeholder purpose
Capable of dealing with hard, soft and narrative data in evidence of success, failure and
‘what we don’t know’
A visual environment for sense-making in complexity
A framework for self-evaluation and dialogue
RESEARCH DESIGN AND METHODOLOGY
SYSTEMS DESIGNING AS IMPROVEMENT RESEARCH DESIGN
The approach to the research design and methodology for this pilot project followed the process of
systems designing developed in the Systems Centre at the University of Bristol (Blockley, 2010,
Blockley and Godfrey, 2000). The sequence of events for the pilot project are as follows: First, the
system boundaries of the OCL schools were defined according to their locally defined purpose.
Next a rich picture was elicited about the system, including identifying stakeholders and the core
processes for achieving the system’s purpose, together with their unique measurement
parameters. From here, key outcomes of the system which were deemed to fulfil its purpose
formed criteria against which the evidence gathering process was defined. These were designed for
entry into the Perimeta software for modelling. The software then took a range of types of data and
provided evidence about what is working well – what is working against desired outcomes and
32
what is not known. This is returned to the stakeholders as a rich, visual analytic to be used for
decision-making, celebration and improvement. What this approach offers for the challenge we
are addressing in this pilot project, is the ability to take the outcomes of traditional quantitative
data and qualitative data (including narrative) and combine it in a bespoke systems design to
visually re-present an overall picture of development and achievement against several key target
outcomes. The computation of traditional outcomes into the 'Italian flag' model of
certainty/uncertainty about success or failure and its visualisation allows leaders to apprehend a
large amount of complex data, encouraging holisitic thinking. It also permits the interrogation of
high level indicators of failure, or uncertainty in any core outcome through 'drilling down' into
much greater detail in order to aid leadership decisioning.
CASE STUDY SAMPLE
The sample chosen for this case study was a group of three Academies in the UK. Oasis Academies
are secondary schools in the United Kingdom, governed by Oasis Community Learning which is a
charitable trust. The vision of Oasis Academies ‘is to create both outstanding schools and
community hubs’. As described in its Education Charter Oasis (See Appendix 2) has a commitment
to ‘transforming lives, learning and communities to achieve stated outcomes for students, staff,
parents/ carers and community members’. A continuing challenge for Oasis is twofold: firstly, how
to evaluate these wider outcomes of schools; and secondly where to target improvements to best
effect. This project is referred to within the Academy chain as ECHO - Evaluating Charter
Outcomes.
Academic results are important but other, less easily quantifiable measures of success are relevant
to transforming lives, learning and communities. For example, the development of students as life-
long learners, employability skills, citizenship, self-confidence, teamwork and emotional wellbeing
are widely recognised as essential qualities for individual success in adult life and for social
cohesion. In the UK, OFSTED2 inspections continue to monitor broader aspects of education like
spiritual, moral, social and cultural development but this evidence typically becomes secondary,
rather than integral, when overall and final judgements are made about performance.
This sample was chosen because of the shared commitment of the Trust to develop useful ways of
assessing complex processes in schools and their communities which encourage powerful learning
and feedback at all levels.
PHASING OF RESEARCH
A phased approach to the research project was adopted, illustrated in 3, with the twin aim of
testing the model and of providing evidence for improvement for Academy leaders and the Trust.
The overall project proceeded in five key steps. First, the designing process, which entailed
2 Office for Standards in Education, Children’s Services and Skills
33
identifying the vision, core processes and key outcomes, specific to this group of Academies. It also
included the identification of the measurement model. The second step was the collection of data
to meet the measurement model designed to represent progress towards the key outcomes. Step
three was the construction of the Perimeta HPM Model. The aim of these first three steps was to
build understanding about the complexities of processes and evidences relating to learning
communities and to collect and analyse data so as to test the efficacy of the model. Step four was
feedback to users for leadership decisioning and exploration of solutions and, for the Oasis
Community Learning Acadamies, step five involves embedding the new practices.
Since this was a pilot project the generation of the Perimeta model was a lengthy process which
took place over a school year. This report focuses on this part of the project, with key areas for
development reported in the findings section. Because it was a pilot project the data were
analysed traditionally and within Perimeta and comparison and triangulation of the results was an
important part of the process. This report is available for the professional communities concerned
to evaluate as part of the overall pilot.
Table 4 Phasing of research
STEP ONE: DESIGN MODEL AND MEASUREMENT PARAMETERS
UNDERSTANDING STAKEHOLDER OBJECTIVES AND CONSTRAINTS
Four main groups of stakeholders in Oasis Academies are recognised in the context of a learning
community:
Students (with Year subgroups);
Parents/ carers of students;
Teachers; and
Senior leaders.
The objectives of stakeholders are complex at an individual/ personal level and also at the collective
level (for example Year groups, the teaching staff). In addition to the standard measures of schools
performance in the form of KPIs and GCSE results, the interwoven Strands of the ECHO research
were designed to address a number of critical success factors founded on validated research. These
critical success factors are summarised in Table 2 below.
Design Model & Measurement
parameters
Collect and analyse all data
Generate and analyse Perimeta
model
Feedback to leaders & explore
solutions
Embed new practices
34
Table 5: Critical success factors for stakeholders
Critical success factors Data Collection Strands Staff
Students and staff feel valued and included in a community that is focused on learning and achievement
Strand 6 Teacher questionnaire Strand 2 Students Questionnaire Strand 3 student narrative interviews
Students and staff are effective learners and believe they are being helped to reach their full potential
Strand 1 ELLI questionnaire
Students enjoy a wide range of opportunities for learning and believe that broad aspects of their development are being catered for
Strand 2 Students Questionnaire Strand 3 student narrative interviews
Students maintain a good rate of progress at 3 key transitions - 6/7, 10/11 and 13/HEI
Strand 9 Attainment data for Year 11 Strand 4 Y11 Questionnaire
Students and staff achieve results that meet aspirations and expectations and feel that their successes are rewarded
Strand 6 Teacher questionnaire Strand 8 KPIs student progress
Students and staff contribute actively to a community that focuses on service to others and where there is shared leadership
Strand 6 Teacher questionnaire Strand 3 student narrative interviews
Staff can provide evidence that training and development opportunities have helped them to improve their classroom practice and/or their effectiveness as leaders and managers
Strand 6 Teacher questionnaire
Staff contribute actively to a community that learns together, shares what works best and knows what to do to achieve
Strand 6 Teacher questionnaire
Parent's/carers feel included and involved in their children’s education at the Academy and know there is an open door for contact when they need it
Strand 5 Parents/carers questionnaire
Senior leaders are effective role models as 'leaders of learning'
Strand 7 Leaders questionnaire
Constraints in the achievement of stakeholder objectives were not pre-supposed but rather tested
through analysis of the responses to questionnaires and interviews. A number of constraints
became evident in the responses and are discussed in more detail in Section Error! Reference
source not found.. In subsequent Phases of the ECHO project the learning about constraints will
enable objectives to be refined so that they are more specific to the stakeholders of Oasis
Academies.
UNDERSTANDING STAKEHOLDER EXPECTATIONS
As with the treatment of constraints, expectations of stakeholders regarding their performance
were not pre-supposed but rather tested through analysis of responses. Each questionnaire
included a number of statements designed to elicit self-perceptions of the adequacy of individual
performance, for example ‘I did as well as I had hope in my GCSEs’. Expectation of performance was
35
therefore treated as an output of the Integrated Approach, that is as a property of the learning of
stakeholders.
The selection of process performance measures for the analysis of responses (see (d) below) was
therefore separated from any consideration of constrains or expectations.
MAPPING CAUSE AND EFFECT
Figure 4 illustrates the alignment of high-level processes contributing to a learning community.
Although incomplete, this provides a systems overview of how an Academy operates in pursuit of
stakeholder objectives. The highest level represents the aims of society in general, and of
communities, regarding the education of children. The next level represents the outcomes achieved
through education by individual stakeholders including students but also their parents/carers,
teachers and leaders. The third level in this illustration represents the outputs of education in the
form of self-perceptions of learning by individuals in each of these groups. In the fourth level the
outputs of specific tests are related to third-level perceptions of performance.
Table 6 Illustrative performance hierarchy for a learning community
The Oasis Charter provided the majority of the first three levels including the vision of a learning
community and critical success factors. From the Transforming Learning section of the Charter,
thirteen statements were articulated to describe the intended behaviours (inputs) and experiences
of students, teachers, leaders and parents/carers, creating a set of critical success indicators for
Oasis Academies. Table 7 develops the illustrative systems view of Figure 4 into a hierarchy of
processes for a learning community based on the Oasis Charter.
Table 7 Hierarchy of specific requirements for success of a learning community based on the Oasis Charter
Level 1:
Vision for
learning
community
1.1 Establishing and sustaining a group of high achieving learning communities that enables
everyone to realise their full potential and refuses to put limits on achievement
Level 2:
Stakeholder
2.1 Developing the
learning of students so
2.2 Developing the
learning of teachers so
2.3 Developing the
learning of leaders so
2.4 Engaging
parents/carers
36
outcomes that they realise their
full potential
that they realise their
full potential and lead
the learning of
students effectively
that they lead the
learning of teachers
and students
effectively
effectively in the
learning activities of
the Academy and in
supporting the
learning of their
children
Level 3:
Outputs of
key
processes
3.1
Fee
ling
va
lue
d a
nd
in
clu
ded
in o
ur
com
mu
nity
3.2
Fee
ling
pa
rt o
f a
co
mm
unity
tha
t fo
cuses its
activitie
s o
n
lea
rnin
g a
nd
achie
ve
me
nt
3.3
Kn
ow
ing
th
at I
am
becom
ing
an
effe
ctive le
arn
er
3.4
Kn
ow
ing
th
at
we
are
be
ing
he
lped
to r
each
ou
r fu
ll po
tentia
l
3.5
Ma
inta
inin
g a
goo
d r
ate
of
pro
gre
ss a
t ke
y tra
nsitio
ns,
for
exam
ple
, p
rim
ary
to s
eco
nd
ary
scho
ol
3.6
Ach
ievin
g r
esu
lts th
at m
eet
my a
spir
atio
ns a
nd e
xp
ecta
tion
s
3.7
Fee
ling
tha
t m
y s
uccesse
s
are
reco
gnis
ed
3.8
Co
ntr
ibutin
g a
ctive
ly t
o a
co
mm
unity w
hic
h f
ocuse
s o
n
se
rvic
e t
o o
the
rs
3.9
Co
ntr
ibutin
g a
ctive
ly t
o a
co
mm
unity w
he
re s
hare
d
lea
de
rship
is p
rom
ote
d
3.1
0 P
rovid
ing e
vid
en
ce t
ha
t
tra
inin
g a
nd
de
ve
lopm
en
t
op
po
rtu
nitie
s h
ave
he
lpe
d m
e to
imp
rove
my c
lassro
om
pra
ctice
an
d/o
r m
y e
ffe
ctiven
ess a
s a
lea
de
r a
nd
man
age
r
3.1
1 C
on
trib
uting
active
ly t
o a
co
mm
unity t
ha
t le
arn
s to
geth
er,
sh
are
s w
hat
wo
rks b
est a
nd
kn
ow
s w
ha
t it n
ee
ds t
o d
o to
imp
rove
3.1
2 F
ee
ling
in
clu
de
d a
nd
involv
ed
with
my c
hild
ren’s
ed
uca
tion
at th
e A
ca
de
my a
nd
kn
ow
ing
tha
t th
ere
is a
n o
pe
n
do
or
for
con
tact
wh
en
I n
ee
d it
3.1
3 B
ein
g e
ffective
ro
le m
od
els
as le
ad
ers
of
lea
rnin
g
Level 4:
Key
processes
Student experiences of learning
Teacher experiences of learning
Leader experiences of learning
Parent/ carer experiences of learning
STEP TWO: COLLECT AND ANALYSE ALL DATA
Data collection methods were then designed to collect evidence in each of these nines Strands
which mapped onto the level 3 outputs. These included closed questionnaires which combined
research-validated scales from previous studies plus bespoke questions designed to elicit evidence
about some of the specific outcomes of the Charter. There were semi-structured interviews and
narrative interviews with teachers, students and leaders. Additional standard key performance
indicators (KPIs) were collected from each Academy to cover attainment and attendance, behaviour
etc. These data collection methods were designed specifically to provide evidence about one or
more of the twelve input statements in the model. For an example of this mapping process see
Table 5 below. For a full report on all questionnaires and interview transcripts see Appendix Two.
During the pilot project, three Oasis Academies were involved in providing data:
From students in different Year Groups: questionnaires and individual interviews
From teachers: questionnaire and individual interviews
From senior leaders: questionnaire and recorded group discussion
From parents/carers: questionnaire
Other aggregated data that is publically available like attendance figures and exam results
4 summarises the nine Strands and sources of evidence used in the ECHO project.
37
Table 8 ECHO sources of evidence
ECHO
Strand
Stakeholder group Source of evidence (and planned number of respondents)
1 Students in Year 7 Research-validated questionnaire completed at the beginning and end of Year 7 (N =
450)
2 Students in Years 8
and 10
Questionnaire with some research-validated questions plus questions relating to the
Charter (N=600)
3 Students in Year 9 Recorded interview based on a few questions designed to get under the surface of
learning (N= 30)
4 Students in Year 11 Questionnaire designed to provide evidence about the impact of education at an Oasis
Academy on the transition to FE, training and/or work and on into later life (N=300)
5 Parents/ carers Questionnaire about engagement with and support from the Academy for children’s
learning (N=100)
6 Teachers Questionnaire and recorded interview about the impact of CPD on classroom practice
and experience of the Oasis ethos (rhetoric or reality?) (N=30)
7 Leaders Questionnaire for senior leaders initially, followed by discussion at a senior leadership
team meeting about their role as ‘leaders of learning’, their impact on the culture and
climate of the Academy and the learning of teachers and students (N=25)
8 - Data from Key Performance Indicators, e.g. attendance, exclusions, student progress
9 Students in Year 11 GCSE results (N=300)
STRANDS OF DATA
Each strand of data related to one or more of 12 critical success indicators (CSIs See Table 5.). In
identifying the following strands of data, it was important to be conscious of the overall context of
the Academy as a complex system and learning organisation. Boundaries of the ‘system’ are drawn
and defined by its purpose and its complexity is understood through the concept of layers and
feedback loops. The layers identified were the students, teachers, leaders, parents/community and
the feedback loops constitute learning as a core process. A key concept informing the design is all
individuals as effective learners.
STRAND ONE: DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7
This consisted of data obtained from the Year 7 cohort from all three Academies (N=450), using the
research-validated questionnaire Effective Lifelong Learning Inventory (ELLI), completed at the
beginning and end of Year 7. ELLI is a 72 item questionnaire measuring seven dimensions of
learning power. Available online, this instantly produces a profile of each learner, in the form of a
7-spoked spider diagram. This feedback then becomes the starting point for interventions such as
mentoring conversations and strategies for developing learning power, individually and collectively.
38
The data for input, therefore, was either 72 variables with values ranging from 1-4 (where 1 = a
little like me and 4 = not at all like me), or seven variables as a percentage score. The latter was a
more powerful indicator in social science statistical terms. In addition, the degree of change (from
pre-test to post-test) as a new variable could also be identified.
STRAND TWO: STUDENT QUESTIONNAIRE AND PROPOSED ANALYSIS
Data from a 40 item questionnaire with some research-validated items plus additional questions
relating to the Oasis Charter, answered on a scale of 1-4 was used with students from Years 8 and
10 (N=600). These were input into the Perimeta model as 40 variables with values ranging from 1-
4, (where 4 = very good and 1= poor). Additionally, a basic statistical computation was conducted
on the items to explore whether there were overarching themes in the data. This would reduce the
data to a set of more powerful variables as percentage.
STRAND THREE: STORIES OF TRANSFORMATION
This Strand involved interviews with students in Year 9 (N=30) based on questions designed to elicit
the quality of their learning. Video and audio recordings of students’ stories formed the data sets
and were facilitated by a teacher with four or five key questions relating to one or more of the CSIs.
The video recordings were thematically analysed and rated on a score of 1-4 (with 4 = very positive
and 1 = very negative).
STRAND FOUR: POST-16 TRANSITION AND PROGRESS TO ADULTHOOD
This Strand included a questionnaire for Year 11 from two of the Academies (N=300) which was
designed to provide evidence about the impact of education at an Oasis Academy on the transition
to further education, training and/or work and on into later life. The questionnaire contains 40
items, focusing on experiences at the Academy and the transition to the next stage. The aim was to
continue to track the cohort over a number of years.
These data were input as 40 variables with values ranging from 1-4 (with 4 = very good and 1=
poor). Additionally, basic statistical computations were conducted on the items to explore whether
there are overarching themes in the data – i.e. to reduce the data to a set of more powerful
variables as a percentage
STRAND FIVE: LEARNING IN A WIDER CONTEXT – QUALITY OF RELATIONSHIPS WITH
PARENTS/CARERS
This Strand involved a 40 item questionnaire for parents and carers (N=100) from one Academy
about their engagement with and support from the Academy for their children’s learning. As with
Strand Two, this questionnaire contained additional open-ended questions relating to the Oasis
Charter, answered on a scale of 1-4.
39
These were input as 40 variables with values ranging from 1-4, (where 4 = very good and 1= poor).
Additionally, a basic statistical computation was conducted on the items to explore whether there
were overarching themes in the data – i.e. to reduce the data to a set of more powerful variables as
percentage.
STRAND SIX: THE LEARNING OF TEACHERS - IMPACT OF CPD ON THE QUALITY OF CLASSROOM
PRACTICE
This Strand has a focus on teachers (N=30) with data collected via a three part 66 item
questionnaire about the impact of continued professional development on classroom practice and
experience of the Oasis ethos (rhetoric or reality) against the Oasis Charter on a four point Likert
type scale.
As with Strand Two and Five these were input as 66 variables with values ranging from 1-4, (where
4 = agree strongly and 1= disagree strongly). Additionally, a basic statistical computation was
conducted on the items to explore whether there were overarching themes in the data – i.e. to
reduce the data to a set of more powerful variables as percentage.
STRAND SEVEN: THE LEARNING OF PRINICIPALS – STRATEGIC APPROACHES TO THE
TRANSFORMATION OF LEARNING
The data for this strand came from questionnaires for senior leaders initially about their role as
‘leaders of learning’, their impact on the culture and climate of the Academy and the learning of
teachers and students (N=25 for senior leadership teams)
The questionnaire contains 55 items, including one open-ended question. The 54 closed items were
structured with a four-point Likert type scale (where 1 = strongly disagree and 4 = strongly agree).
These were inputted as 54 variables with values ranging from 1-4, as above. Additionally a basic
statistical computation was conducted on the items to explore whether there were any overarching
themes in the data – i.e. to reduce the data to a set of more powerful variables as a percentage.
STRAND EIGHT: CONTEXT FOR STUDENT LEARNING - KEY PERFORMANCE INDICATORS
Data from Key Performance Indicators, for example, attendance, free school meals (FSM)
exclusions, special educational needs (SEN) was used for this strand. There were up to 10
aggregated variables for each year group (N=5) and data was explored for differences within and
between Academies with regards to gender, FSM and SEN.
STRAND NINE: KEY OUTCOME FOR YEAR 11 STUDENTS – GCSE/EXAM RESULTS
Data for this Strand came from Year 11 GCSE results. Five aggregated variables will be used for
including 5+ higher GCSEs including English and Mathematics and Science. The data will be
40
explored for differences within and between Academies with regards to gender, FSM and SEN
against GCSE exam results in the key subjects.
MAPPING THE DATA STRANDS ONTO THE CRITICAL SUCCESS INDICATORS
The table in the next section presents the data map of each Strand of data by item, to each critical
success indicator in the model. This mapping was created and moderated by two researchers
forming and moderating judgements about data structure. This details how all the CSIs are
addressed within the nine Strands of data collection, although only examples are used because of
the size of the document.
41
Table 9 Critical Success Indicators mapped onto data collection strands
CRITICAL SUCCESS FACTORS
Data Collection Strands Staff
Data Collection Strands Students
Items or scales Data Structure
Focus of items or interviews for each CSI - assumptions and evidence for each CSI
Key issues or questions
Students and staff feel valued and included in a community that is focused on learning and achievement
Strand 6 Teacher questionnaire
Strand 2 Students Questionnaire Strand 3 student narrative interviews
Involvement in learning I was concentrating and hard to distract I took a lot of care with what I was doing I was working hard on the learning I was very focused on the learning Negative affect I was nervous I told myself 'I can’t do this' and felt unhappy I was afraid in case I got things wrong I was unhappy with what I did
Likert type scale
Students and staff are effective learners and believe they are being helped to reach their full potential
Strand 1 ELLI Strand 1 ELLI Changing and learning, meaning making critical curiosity, creativity, strategic awareness, learning relationships, resilience 72 items Openness to learning I did more than I was asked to do I came up with new ideas on my own When I found something hard I tried another way I was excited to explore new things Interest The subject we were doing is very interesting to me I have always been curious about what we were learning I found the subject boring (inverted) I could never be bored with this subject.
Percentage scores from scales of items Likert type scale
The quality and strength of each individual’s orientation towards learning - how open are they to engaging with new learning opportunities within the Academy? How well do they understand themselves as learners? How strong is their learning identity? Are they open to new experiences? Does what they learn hold their interest?
A two-concept CSI
42
CRITICAL SUCCESS FACTORS
Data Collection Strands Staff
Data Collection Strands Students
Items or scales Data Structure
Focus of items or interviews for each CSI - assumptions and evidence for each CSI
Key issues or questions
Students enjoy a wide range of opportunities for learning and believe that broad aspects of their development are being catered for
Strand 2 Students Questionnaire Strand 3 student narrative interviews
Positive Affect I felt proud of what I achieved I was very happy with what I did I felt content with my learning
Likert type scale Students feel good about being in school
A two-concept CSI
Students maintain a good rate of progress at 3 key transitions - 6/7, 10/11 and 13/HEI
Strand 9 Attainment data for Year 11 Strand 4 Y11 Questionnaire
Schools data sets Predetermined How can we measure progress at Years 6/7 and Year 13/HEI transition?
Students and staff achieve results that meet aspirations and expectations and feel that their successes are rewarded
Strand 6 Teacher questionnaire
Strand 8 KPIs student progress
A value added score - a regression co-efficient
A two-concept CSI
Students and staff contribute actively to a community that focuses on service to others and where there is shared leadership
Strand 6 Teacher questionnaire
Strand 3 student narrative interviews
Negative social functioning What I did upset others I played around instead of learning I did not do what was asked I did not want to work with others Positive social functioning I offered to help others I was included by other students I worked with others wherever I could
Likert type scale Students feel good about being in school
A two-concept CSI
Staff can provide evidence that training and development opportunities have helped them to improve their classroom practice and/or their effectiveness as leaders and managers
Strand 6 Teacher questionnaire
Can you give me an example of a time when…..
Narrative analysis as rating
43
CRITICAL SUCCESS FACTORS
Data Collection Strands Staff
Data Collection Strands Students
Items or scales Data Structure
Focus of items or interviews for each CSI - assumptions and evidence for each CSI
Key issues or questions
Staff contribute actively to a community that learns together, shares what works best and knows what to do to achieve
Strand 6 Teacher questionnaire
Teacher Enquiry Staff draw on good practice from other schools as a means to further their own professional development Staff read research reports as one source of useful ideas for improving their practice Staff use the web as one source of useful ideas for improving their practice Students are consulted about how they learn most effectively Staff relate what works in their own practice to research findings Staff modify their practice in the light of published research evidence Staff carry out joint research/evaluation with one or more colleagues as a way of improving practice Building social capital Staff regularly collaborate to plan their teaching If staff have a problem with their teaching they usually turn to colleagues for help Teachers suggest ideas or approaches for colleagues to try out in class Teachers make collective agreements to test out new ideas Teachers discuss openly with colleagues what and how they are learning Staff frequently use informal opportunities to discuss how students learn Staff offer one another reassurance and support Teacher Critical and responsive learning Staff are able to see how practices that work in one context might be adapted to other contexts Staff reflect on their practice as a way of identifying professional learning needs Staff experiment with their practice as a conscious strategy for improving classroom teaching and learning Staff modify their practice in the light of feedback from their students Staff modify their practice in the light of evidence from self-evaluations of their classroom practice Staff modify their practice in the light of evidence from evaluations of their classroom practice by managers or other colleagues
Likert type scale 1. Enquiry: using and responding to different sources of evidence, carrying out joint research and evaluation with colleagues 2. Teacher Critical and responsive learning, through experimentation, responding to feedback self-evaluation and reflection 3. Building social capital, learning, working and supporting each other 4.Teachers value learning
44
CRITICAL SUCCESS FACTORS
Data Collection Strands Staff
Data Collection Strands Students
Items or scales Data Structure
Focus of items or interviews for each CSI - assumptions and evidence for each CSI
Key issues or questions
Parent's/carers feel included and involved in their children’s education at the Academy and know there is an open door for contact when they need it
Strand 5 Parents/carers questionnaire Likert type scales A two-concept CSI
Senior leaders are effective role models as 'leaders of learning'
Strand 7 Leaders questionnaire
Likert type scales
45
STEP THREE: GENERATE AND ANALYSE PERIMETA MODEL
DEFINING PROCESS PERFORMANCE MEASURES
Performance measures were established for each of five types of processes of a learning
community. The five types were selected to allow a breakdown by some of the main areas of
interest regarding learning community performance, that is, by gender and by Academy. Other
areas such as the influence of social factors on performance will be considered for inclusion in later
Phases of the ECHO project. Table 6 summarises the objectives, measures and target performance
levels of the five types.
Table 10 Process performance measures
Type of process Objective Measure Target
Outputs:
Overall All stakeholders strongly agree with all
statements
Percentage
score
100% strongly agree
Gender All stakeholders of given Gender
strongly agree with all statements
Percentage
score
100% strongly agree
Academy All stakeholders of given Academy
strongly agree with all statements
Percentage
score
100% strongly agree
Question All stakeholders strongly agree with
given statement
Percentage
score
100% strongly agree
Inputs:
Responses to
statements in
questionnaire
Stakeholder strongly agrees with given
statement
Percentage
score
100% strongly agree
COMMUNICATING EVIDENCE
Two aspects of the performance of a learning community need to be communicated effectively in
order to make sense of the evidence and to support improvement. The first aspect relates to the
higher-level questions about how well the system is working, and the second relates to the qualities
of evidence derived from responses to questionnaires and interviews.
A simple dashboard was devised to assist in communicating evidence of performance in respect of
the many and diverse outcomes and outputs of a learning community. shows the four main
components of the dashboard, reflecting the following hierarchy of dimensions of performance of a
learning community:
By Question {Q1..Qj} answered by each Respondent {N1..Ni};
By Gender {G1, G2} for all Questions answered by Respondents of that Gender;
46
By Academy {A1, A2, A3} for all Questions answered by Respondents belonging to that Academy; and
For all Questions answered by all Respondents.
Table 11 Components of performance dashboard for a learning community
Performance by ACADEMY {A1, A2, A3} Performance OVERALL
Inputs by RESPONDENT and QUESTION {N1xQ1 .. Ni x Qj}
Performance by QUESTION {Q1 .. Qj}
Performance by GENDER {M, F}
Recognising the uncertainty inherent in responses to a questionnaire, the ECHO project adopted
the ‘Italian Flag’ graph to represent the quality of all of the evidence and consisting of:
‘Green’ representing the strength of evidence of positive self-perception of learning;
‘Red’ representing the strength of evidence of negative self-perception of learning; and
‘White’ representing lack of evidence, or uncertainty in self-perception of learning.
Table 12 illustrates Italian Flags for some examples of self-perception of learning.
Table 12: Illustration of Italian Flag graphs in context of a learning community
Categories of evidence in Italian Flag Evidence of success
Lack of evidence
Evidence of failure
Application to learning community Evidence of positive self-perception of learning
Uncertainty in self-perception of learning
Evidence of negative self-perception of learning
Example: High self-perception of learning from high proportion of ‘strongly agree’ responses
Example: Medium self-perception of learning from high proportion of ‘agree’ responses
Example: Medium self-perception of learning from high proportion of ‘disagree’ responses
Example: Low self-perception of learning indicated by high proportion of ‘strongly disagree’ responses
ELICITING EVIDENCE
The design of questionnaires and interviews was based on the systems view of a learning
community represented by Table 5. Nine Strands of evidence of performance relating to self-
perceptions of the experiences of learning of students, teachers, leaders and parents/carers was
47
then gathered for input to the model using questionnaires, semi-structured interviews combined
research-validated questions with additional questions about the specific outcomes of the Charter
and standard KPIs. The evidence provided by responses to each question or by themes from
qualitative or narrative interview data was specifically designed to provide evidence about one or
more of the thirteen input statements in the model.
Uncertainty is introduced in the evidence from questionnaires and interviews by answers that are
not absolutely positive or absolutely negative. Given a scale from ‘strongly disagree’ to ‘strongly
agree’, the intermediate points of ‘disagree’ and ‘agree’ entail some contrary evidence and some
uncertainty, as indicated by the examples given in Table 6.
Another factor in making sense of the evidence is the degree of confidence we have in the
individual respondent. A number of considerations are relevant, for example the respondent’s
understanding of the question and experience to answer it. The respondent may be considered to
be biased or in some other way misleading or misled.
Hall et al (2004) describe a technique for mapping from linguistic descriptions of ‘performance’ and
‘confidence in judgement of performance’ to interval values of performance compatible with Italian
Flag figures of merit. Table 13 reproduces their illustration of the mapping and presents an
interpretation of Hall et al’s mapping for a set of 25 discrete combinations of performance and
confidence scales. The performance scale is based on a four-point Likert scale from ‘strongly
disagree’ to ‘strongly agree’, corresponding to scores from 1 to 4 recorded by respondents to ECHO
project questionnaires.
Table 13 Hall et al’s (2004) mapping from linguistic descriptions of ‘performance’ and ‘confidence in judgement of performance’ to interval values
of performance
48
Table 14 Conversion of scores from Likert scale to Sp and Sn values of evidence
Confidence
judgment
Rating
Very high 100% (0.00,0.00) (0.25,0.25) (0.50,0.50) (0.75,0.75) (1.00,1.00)
High 75% (0.00,0.05) (0.20,0.30) (0.42,0.58) (0.70,0.80) (0.95,1.00)
Medium 50% (0.00,0.10) (0.10,0.40) (0.38,0.62) (0.60,0.90) (0.90,1.00)
Low 25% (0.00,0.25) (0.05,0.50) (0.30,0.70) (0.50,0.95) (0.75,1.00)
Very low 0% (0.00,0.60) (0.01,0.80) (0.05,0.95) (0.20,1,00) (0.40,1.00)
Rating 0% 25% 50% 75% 100%
Performance judgment 0 1 – strongly
disagree
2 - disagree 3 - agree 4 – strongly agree
MODELLING PROCESS PERFORMANCE
Process performance functions were created by combining the Likert conversion of Table 147 with
the process performance measurement scales of Figure 5. At the input level, direct evidence of
respondents’ raw scores (from 1 to 4) for each question were converted into Italian Flag figures of
merit where the best possible (100%) performance was full Green (respondent strongly agrees, and
very high confidence in the respondent) and the worst possible performance was mostly Red
(respondent strongly disagrees, but very low confidence in the respondent). For output processes
(collated by Question, by Gender, by Academy and Overall) the definitions of best and worst
performance and all points in between were judged on a similar scale.
MODELLING THE SIGNIFICANCE OF INTER-PROCESS RELATIONSHIPS
As established earlier, the significance of pairwise relationships or links between ‘parent’ and ‘child’
parameters/ processes is modelled in the Integrated Approach using three detailed attributes of
Sufficiency, Necessity and Dependency defined by Davis & Fletcher (2000):
‘The Sufficiency or relevance of the evidence to its parent process is judged as a single
number in the [0,1] range;
A sub-process is a Necessity if the parent process cannot succeed without it. Consequently, in
the event of failure of the sub-process, the parent process fails’; and
Dependency is the degree of overlap between sub-processes and describes the degree of
commonality in the sources of evidence’.
49
The significance of pairwise relationships or links between processes defining ‘causes’ and ‘effects’
was modelled using three detailed attributes of Sufficiency, Necessity and Dependency defined
earlier. On the basis of experience in modelling ‘many to one’ performance relationships, the values
indicated in Table 13 were assigned.
Table 15 Sufficiency, Necessity and Dependency of relationships between processes
‘Effect’ process ‘Cause’ process Sufficiency Necessity Dependency
Overall performance Responses to statements in
questionnaires
0.3 0.4 1.0
Gender performance Responses to statements in
questionnaires
0.3 0.4 1.0
Academy performance Responses to statements in
questionnaires
0.3 0.4 1.0
Question performance Responses to statements in
questionnaires
0.3 0.4 1.0
ESTIMATING SYSTEM PERFORMANCE
Full system models of the features of each of Strands 1 to 7 were developed in the Perimeta toolkit,
combining the following features defined above:
A hierarchy linking the responses to questionnaire statements (input processes) in turn to
output processes representing the performance by Question, by participating Academies, by
respondent Gender (where given) and Overall;
Responses to questions using Likert ratings from 1 (‘poor’ or ‘strongly disagree’) to 4 (‘very
good’ or ‘strongly agree’);
Process performance functions using linguistic measures related to Likert rating scales; and
Sufficiency, Necessity and Dependency ratings for each cause and effect relationship
The Perimeta models used the ‘Juniper’ algorithm to propagate the evidence and provide estimates
of output performance by Question, by Gender, by Academy and Overall. A full explanation of the
Juniper algorithm is given in the Appendix 6, and is based on the description given by Davis and
Fletcher (2000).
The Perimeta models were each configured to produce a dashboard summary view as well as
tabulated results for each Question, each Academy, each Gender and Overall. A sample datasheet
from a Perimeta model is reproduced in Table 16 below.
50
Table 16 Sample of Perimeta model datasheet
VALIDATING INSIGHTS
The parallel analysis by the ECHO project team of the response data using standard statistical
techniques allowed the results of the Integrated Approach to be validated. There following
comparisons were made between the two sets of results, for each Strand where appropriate:
Positive self-perception of learning, versus mean scores;
Strength of positive or negative bias, versus range of scores;
Uncertainty in self-perception of learning, versus standard deviation of scores;
Comparison of results between Academies; and
Comparison of results between Genders.
STEP FOUR: FEEDBACK TO LEADERS AND EXPLORE SOLUTIONS
Stakeholder validation was limited in Phase 1 of the ECHO project by lack of direct engagement
with stakeholders. However, detailed reviews were carried out twice during Phase1 with senior
leaders, providing valuable feedback and endorsement of the approach.
Step Four of the ECHO project would develop the approach to creating learning communities,
focusing on:
Deepening understanding of the issues of evaluation and refining questionnaires;
51
Developing the Integrated Approach and its application to the complex problem of creating
learning communities; and
Engaging stakeholders in developing solutions.
STEP FIVE: EMBED NEW PRACTICES
Step Five of the ECHO project would be a continuous process of creating learning communities,
transferring capability to the stakeholders and focusing on:
Supporting stakeholder-led improvements in learning with appropriate access to
appropriate tools and techniques;
Developing the Integrated Approach to measure the success of improvements and support
continuous improvement.
SUMMARY
In this chapter, the approach to the research design and the methodology through which the
purpose can be achieved is presented. First, the system boundaries of the Academies were
identified to elicit a rich picture of the system, including identifying stakeholders and the core
processes for achieving the system’s purpose, together with their unique measurement
parameters. From here, key outcomes of the system which were deemed to fulfil its purpose
formed criteria against which the evidence gathering process was defined. The data were collected
through questionnaires, interviews and from the Academies' own performance data. The values
were converted into the Perimeta Italian Flag model through a process which mapped confidence
against performance. All data from all cases were entered into the Perimeta model which was
based on the systems design. A visual report was then produced which provided evidence about
what was working well – what was working against desired outcomes and what was not known.
52
DATA ANALYSIS: SOCIAL SCIENCE
The data were analysed strand by strand using traditional social science analyses methods,
including descriptive statistics, t-tests and ANOVA as well as factor analyses. The findings from each
Strand are presented below.
STRAND ONE - DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7
Strand One was designed to assess Year 7 students’ self-reporting of their learning dispositions,
using the Effective Lifelong Learning Inventory's seven scales.
QUESTIONNAIRE AND FEEDBACK
The Effective Lifelong Learning Inventory (ELLI) questionnaire is an online survey consisting of seven
scales, totalling 72 items. These are research validated scales which measure the following student
learning dispositions: changing and learning, meaning making, creativity, critical curiosity, strategic
awareness, learning relationships and resilience.5 The items were structured with a four point likert
type scale labelled strongly disagree, disagree, agree, strongly agree. These were treated as
numerical scores at the stage of analysis in order to generate immediate feedback to students in
the form of a spider diagram (see Figure 8), feedback to teachers in the form of histograms and
mean scores and to provide raw data for analysis. The student feedback provided a framework for a
coaching conversation with the teacher or peers about learning. The histograms provided
diagnostic data for teachers about their class’s orientation to learning.
Table 17 Feedback to users in the form of a spider diagram representing the 7 dimensions of learning power
5 For reliability and validity statistics on these scales see Deakin Crick & Yu 2008, and Ren and Deakin Crick, 2013.
53
INTERVENTIONS
All of the Academies introduced changes in their approach to teaching and learning with the year
groups in this sample. These interventions were focused on coaching for learning power and
enquiry based learning.
THE SAMPLE
Students in Year 7 in each of the three Academies completed the Effective Lifelong Learning
Inventory (ELLI) between October and December of the academic year 2010/11. A sample of
teachers from each Academy were trained in the interpretation of the ELLI survey, coaching for
learning with students based on the data and the design of teaching and learning interventions to
stimulate students’ engagement in learning. At the end of the year the Academies completed a
post-test survey.
For technical and logistical reasons, from the total sample of 450 students, only 274 were robust
enough to contribute to data analysis, after careful screening and analysis. This was due to some
technical issues in retrieving data from a new provider and also because some of the Academies
had difficulty in students completing the post-tests within the time frame. Table 10 summarises the
sample for Strand 1.
Table 18 Strand One sample by Academy and gender
Sample by Gender and Academy
Academy Total
1 2 3
Gender Male 48 53 50 151
Female 66 24 33 123
Total 114 77 83 274
FINDINGS
In order to assess whether there was any difference between the first and second time of
administration of the ELLI, a paired samples t-test was computed with all complete data, combining
the three Academies. As can be seen from Tables 11 and 12, there were significant increases in
student scores on all seven learning power dimensions between Time 1 and Time 2, except for
Resilience where the mean score was significantly less in Time 2.
54
Table 19 Mean scores on each learning power scale Time 1 and Time 2
ELLI Scale Time one and Time Two Mean N Std. Deviation Std. Error Mean
Changing & Learning Time 1 68.64 139 20.27 1.71
Time 2 76.97 139 19.41 1.64
Learning Relationships Time 1 61.13 139 16.98 1.44
Time 2 69.34 139 16.46 1.39
Strategic Awareness Time 1 58.69 139 18.98 1.61
Time 2 68.10 139 18.88 1.60
Resilience Time 1 49.11 139 18.82 1.59
Time 2 44.73 139 18.95 1.60
Creativity Time 1 59.40 139 20.85 1.70
Time 2 69.32 139 17.91 1.51
Meaning Making Time 1 63.30 139 19.90 1.68
Time 2 74.75 139 18.84 1.59
Critical Curiosity Time 1 55.68 139 18.90 1.60
Time 2 71.16 139 18.48 1.56
Table 20 Paired t-tests
SD StE t df Sig(2 tailed)
Changing and Learning 18.98 1.61 -5.17 138 .000
Learning Relationships 17.24 1.46 -5.61 138 .000
Strategic Awareness 14.41 1.22 -7.69 138 .000
Resilience 16.97 1.43 3.04 138 .003
Creativity 18.01 1.52 -6.49 138 .000
Meaning Making 18.81 1.59 -7.17 138 .000
Critical Curiosity 12.54 1.06 -14.55 138 .000
DIFFERENCES BETWEEN ACADEMIES
In order to compare the differences between Academies an Analysis of Variance (ANOVA) was
computed at Time 1 and Time 2. This indicated that there were no significant differences between
the Academies at Time 1. However, the ANOVA computation for Time 2 shows significant
differences. Table 13 shows the descriptive statistics for each Academy at Time 2 against each
scale. Table 14 shows the ANOVA table with tests of significance for each scale also at Time 2. This
shows there to be significant differences between Academies in changing and learning, learning
relationships, strategic awareness and meaning making. However, there were no significant
differences in the other learning scales between Academies.
55
Table 21 Descriptive statistics for each Academy at Time 2
Table 22 ANOVA
between
Academies Time 2
ANOVA
Sum of df Mean Square F Sig.
N Mean Std. Deviation
CL.2: CL
Changing &
Learning
1 78 82.37 15.19
2 28 65.17 20.54
3 33 74.24 22.66
Total 139 76.97 19.41
LR.2: LR
Learning
Relationships
1 78 72.68 15.73
2 28 59.62 14.80
3 33 69.69 16.67
Total 139 69.34 16.46
SA.2: SA
Strategic Awareness
1 78 70.51 18.49
2 28 59.61 18.99
3 33 69.61 18.20
Total 139 68.10 18.88
RS.2: RS
Resilience
1 78 44.77 20.40
2 28 45.44 15.41
3 33 44.02 18.62
Total 139 44.73 18.95
CR.2: CR
Creativity
1 78 69.31 16.88
2 28 64.76 19.38
3 33 73.23 18.62
Total 139 69.32 17.91
MM.2: MM
Meaning Making
1 78 77.71 17.10
2 28 67.17 19.58
3 33 74.17 20.79
Total 139 74.75 18.84
CC.2: CC
Critical Curiosity
1 78 71.74 17.75
2 28 68.78 18.21
3 33 71.82 20.70
Total 139 71.18 18.48
56
Squares
CL.2: CL
Changing &
Learning
Between Groups 6414.550 2 3207.2 9.56 .000
Within Groups 45610.830 136 335.37
Total 52025.380 138
LR.2: LR
Learning
Relationships
Between Groups 3520.797 2 1760.39 7.06 .001
Within Groups 33877.507 136 249.09
Total 37398.304 138
SA.2: SA
Strategic
Awareness
Between Groups 2545.942 2 1272.97 3.70 .027
Within Groups 46673.191 136 343.18
Total 49219.133 138
RS.2: RS
Resilience
Between Groups 30.813 2 15.40 .04 .959
Within Groups 49554.488 136 364.37
Total 49585.300 138
CR.2: CR
Creativity
Between Groups 1086.832 2 543.41 1.71 .185
Within Groups 43206.053 136 317.69
Total 44292.886 138
MM.2: MM
Meaning
Making
Between Groups 2303.469 2 1151.73 3.35 .038
Within Groups 46705.813 136 343.42
Total 49009.282 138
CC.2: CC
Critical
Curiosity
Between Groups 199.881 2 99.94 .28 .749
Within Groups 46954.100 136 345.25
Total 47153.980 138
DIFFERENCES IN PRE-POST CHANGE BETWEEN ACADEMIES
In order to explore these differences further paired t-tests were conducted for the three Academies
separately. Although care should be taken in their interpretation as the sample size is not complete
and is different for each Academy, it appears that the contribution to the overall positive change
was not evenly distributed between the Academies.
Academy 1
Table 23 Descriptive Statistics Academy 1
Mean N Std. Deviation Std. Error Mean
Pair 1 CL.1: CL 71.26 78 19.86 2.25
CL.2: CL 82.37 78 15.19 1.72
Pair 2 LR.1: LR 64.67 78 15.11 1.71
LR.2: LR 72.69 78 15.74 1.78
Pair 3 SA.1: SA 61.54 78 18.38 2.08
SA.2: SA 70.51 78 18.49 2.09
Pair 4 RS.1: RS 48.57 78 19.36 2.19
57
RS.2: RS 44.77 78 20.40 2.31
Pair 5 CR.1: CR 60.90 78 19.46 2.20
CR.2: CR 69.32 78 16.89 1.91
Pair 6 MM.1: MM 66.00 78 18.89 2.14
MM.2: MM 77.72 78 17.10 1.94
Pair 7 CC.1: CC 57.88 78 18.05 2.04
CC.2: CC 71.75 78 17.76 2.01
Table 24 Paired t-test Academy 1
Mean Std. Deviation
Std. Error Mean
t df Sig. (2-tailed)
Pair 1 CL.1: CL - CL.2: CL -11.11
18.54 2.10 -5.29 77 0.00
Pair 2 LR.1: LR - LR.2: LR -8.01 15.72 1.78 -4.50 77 0.00
Pair 3 SA.1: SA - SA.2: SA -8.97 12.99 1.47 -6.10 77 0.00
Pair 4 RS.1: RS - RS.2: RS 3.80 17.13 1.94 1.96 77 0.05
Pair 5 CR.1: CR - CR.2: CR -8.42 16.58 1.88 -4.48 77 0.00
Pair 6 MM.1: MM - MM.2: MM -11.72
17.85 2.02 -5.80 77 0.00
Pair 7 CC.1: CC - CC.2: CC -13.87
10.43 1.18 -11.74 77 0.00
Academy 2
Table 25 Descriptive Statistics Academy 2
Paired Samples Statistics
Mean N Std. Deviation Std. Error Mean
Pair 1 CL.1: CL 59.52 28 22.07 4.17
CL.2: CL 65.18 28 20.54 3.88
Pair 2 LR.1: LR 54.17 28 17.12 3.23
LR.2: LR 59.62 28 14.80 2.80
Pair 3 SA.1: SA 49.45 28 15.87 3.00
SA.2: SA 59.62 28 18.99 3.59
Pair 4 RS.1: RS 49.86 28 19.23 3.63
RS.2: RS 45.45 28 15.41 2.91
Pair 5 CR.1: CR 53.33 28 22.88 4.32
CR.2: CR 64.76 28 19.38 3.66
Pair 6 MM.1: MM 55.10 28 21.23 4.01
MM.2: MM 67.18 28 19.58 3.70
Pair 7 CC.1: CC 50.13 28 18.98 3.59
CC.2: CC 68.78 28 18.21 3.44
58
Table 26 Paired t-tests Academy 2
Mean Std.
Deviation
Std. Error
Mean
t df Sig. (2-
tailed)
Pair 1 CL.1: CL - CL.2: CL -5.65 21.28 4.02 -1.41 27 0.17
Pair 2 LR.1: LR - LR.2: LR -5.46 17.72 3.35 -1.63 27 0.12
Pair 3 SA.1: SA - SA.2: SA -10.16 17.70 3.35 -3.04 27 0.01
Pair 4 RS.1: RS - RS.2: RS 4.41 18.65 3.53 1.25 27 0.22
Pair 5 CR.1: CR - CR.2: CR -11.43 22.54 4.26 -2.68 27 0.01
Pair 6 MM.1: MM - MM.2: MM
-12.07 22.31 4.22 -2.86 27 0.01
Pair 7 CC.1: CC - CC.2: CC -18.65 16.08 3.04 -6.14 27 0.00
Academy 3
Table 27 Descriptive Statistics Academy 3
Mean N Std.
Deviation
Std. Error
Mean
Pair 1 CL.1: CL 70.20 33 17.93 3.12
CL.2: CL 74.24 33 22.67 3.95
Pair 2 LR.1: LR 58.67 33 19.18 3.34
LR.2: LR 69.70 33 16.67 2.90
Pair 3 SA.1: SA 59.83 33 20.82 3.62
SA.2: SA 69.62 33 18.20 3.17
Pair 4 RS.1: RS 49.79 33 17.67 3.08
RS.2: RS 44.03 33 18.62 3.24
Pair 5 CR.1: CR 61.01 33 21.96 3.82
CR.2: CR 73.23 33 18.62 3.24
Pair 6 MM.1: MM 63.93 33 19.81 3.45
MM.2: MM 74.17 33 20.79 3.62
Pair 7 CC.1: CC 55.22 33 20.38 3.55
CC.2: CC 71.83 33 20.70 3.60
Table 28 Paired t-test Academy 3
Mean Std.
Deviation
Std. Error
Mean
t df Sig. (2-tailed)
CL.1: CL - CL.2: CL -4.04 17.32 3.02 -1.34 32 0.19
LR.1: LR - LR.2: LR -11.03 20.20 3.52 -3.14 32 0.00
SA.1: SA - SA.2: SA -9.79 14.97 2.61 -3.76 32 0.00
RS.1: RS - RS.2: RS 5.76 15.49 2.70 2.14 32 0.04
CR.1: CR - CR.2: CR -12.22 17.21 3.00 -4.08 32 0.00
MM.1: MM - MM.2: MM -10.25 18.37 3.20 -3.20 32 0.00
CC.1: CC - CC.2: CC -16.61 13.49 2.35 -7.08 32 0.00
59
SUMMARY AND FINDINGS FROM STRAND ONE
Strand One was designed to assess Year 7 students’ self-reporting of their learning dispositions,
using the Effective Lifelong Learning Inventory's seven scales.
The findings demonstrated there was a significant increase in mean score between Time 1 and Time
2 on six of the seven dimensions of learning power, based on the Effective Lifelong Learning
Inventory, which means that something other than chance must account for this increase. The
increase was not uniform across all three Academies where Academy 1 and 3 were more similar in
terms of the statistical significance in many of the dimensions of learning power.
This seems to suggest that whilst there are significant increases in scores for all the Academies
overall (except for Resilience), there are other factors which contributed to the differing levels of
significance in the seven dimensions between Academies. This could be due to a range of factors
such as the students themselves, the delivery of the interventions or the overall leadership of the
Academy and are worth exploring further.
STRAND TWO - STUDENT PERCEPTIONS OF THE IMPACT OF THE EDUCATION CHARTER
Strand Two was a questionnaire designed to assess students' perceptions about their learning,
teaching, progress and achievement in an Oasis Academy.
QUESTIONNAIRE
The Learning of Students questionnaire consisted of 42 items structured with a four-point Likert-
scale labeled, strongly disagree, disagree, agree, and strongly agree. The total number of student
respondents was 377. The students were selected from Years 8 and 10.
Table 29 Data Description for Strand Two
Variable Defined value
Missing
Academy 1, 3
Gender 1, 2 0.3% (1 record)
S21 – S242
1-4 0.0-4.0% (some records with value=0 which are treated as missing leading to 322 records without any missing on these variables)
STUDENT RESPONSES BY QUESTION
The following Table 22 shows students' responses by question, with the number of students
responding to each question and the percentage of the total.
Table 30 Strand 2 Student questionnaire responses by item
60
Item 1 strongly
disagree
2 disagree 3 agree 4 strongly
agree
missing
N % N % N % N % Total %
S21 1. I feel included in the activities that are available to me in the Academy
27 7.2 75 19.9 230 61.0 45 11.9 377 0.0
S22 2. I feel safe in the Academy 22 5.8 43 11.4 204 54.1 108 28.6 377 0.0
S23 3. I feel valued and cared for as an individual in the Academy
32 8.5 84 22.3 204 54.1 54 14.3 374 0.8
S24 4. I like learning new things 8 2.1 37 9.8 203 53.8 128 34.0 376 0.3
S25 5. I usually concentrate on my learning
9 2.4 44 11.7 224 59.4 98 26.0 375 0.5
S26 6. I am hard to distract 74 19.6 184 48.8 91 24.1 27 7.2 376 0.3
S27 7. I like working with other students to help my learning
20 5.3 63 16.7 180 47.7 111 29.4 374 0.8
S28 9. I am confident in my learning 40 10.6 96 25.5 167 44.3 74 19.6 377 0.0
S29 8. I don't distract other students 19 5.0 49 13.0 206 54.6 102 27.1 376 0.3
S210 10. I take a lot of care with my work 8 2.1 46 12.2 233 61.8 90 23.9 377 0.0
S211 11. I work hard 9 2.4 39 10.3 217 57.6 111 29.4 376 0.3
S212 12. I do more than I am asked to do in class
31 8.2 156 41.4 155 41.1 33 8.8 375 0.5
S213 13. I don't mind making mistakes because I learn from them
23 6.1 84 22.3 182 48.3 87 23.1 376 0.3
S214 14. I come up with new ideas to help my learning
23 6.1 102 27.1 189 50.1 62 16.4 376 0.3
S215 15. I enjoy my learning 32 8.5 87 23.1 194 51.5 63 16.7 376 0.3
S216 16. My teachers teach well so that I learn successfully
27 7.2 78 20.7 198 52.5 70 18.6 373 1.1
S217 17. I have regular opportunities to express my opinion to my teachers about my learning
42 11.1 126 33.4 158 41.9 45 11.9 371 1.6
S218 18. I often have conversations with my teachers which help me to make better progress
52 13.8 140 37.1 138 36.6 42 11.1 372 1.3
S219 19. My teachers seem to enjoy teaching my classes
36 9.5 116 30.8 176 46.7 46 12.2 374 0.8
S220 20. My teachers often share things that they have learned themselves
25 6.6 83 22.0 202 53.6 63 16.7 373 1.1
S221 21. I get on well with my teachers 21 5.6 71 18.8 191 50.7 91 24.1 374 0.8
S222 22. I get extra support to help my learning when I need it
60 15.9 95 25.2 173 45.9 46 12.2 374 0.8
S223 23. I know that the senior teachers want us to learn successfully
17 4.5 53 14.1 214 56.8 90 23.9 374 0.8
S224 24. Everyone works very hard to make the Academy a great place to learn and to do well
23 6.1 101 26.8 176 46.7 74 19.6 374 0.8
S225 25. My teachers constantly expect me to improve on my personal best
9 2.4 53 14.1 184 48.8 128 34.0 374 0.8
S226 26. Right now I am achieving the best I possibly can
23 6.1 75 19.9 178 47.2 93 24.7 369 2.1
61
S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy
16 4.2 56 14.9 189 50.1 108 28.6 369 2.1
S228 28. I have kept up a good rate of progress ever since my first year at the Academy
14 3.7 73 19.4 190 50.4 90 23.9 367 2.7
S229 29. I think that my results at the moment are as good as I can do
25 6.6 117 31.0 153 40.6 72 19.1 367 2.7
S230 30. I feel that my successes at the Academy are recognised
43 11.4 115 30.5 172 45.6 38 10.1 368 2.4
S231 31. I feel that my successes with activities outside the Academy are recognised
70 18.6 112 29.7 142 37.7 43 11.4 367 2.7
S232 32. I feel proud about what I have achieved so far
14 3.7 82 21.8 176 46.7 96 25.5 368 2.4
S233 33. We are taught about the responsibilities of students as well as their rights
21 5.6 75 19.9 193 51.2 76 20.2 365 3.2
S234 34. We care about helping each other in the Academy community
28 7.4 83 22.0 195 51.7 61 16.2 367 2.7
S235 35. We are encouraged to care about the needs of other people in the local community and around the world
17 4.5 80 21.2 193 51.2 77 20.4 367 2.7
S236 36. I regularly offer to help others 25 6.6 89 23.6 179 47.5 75 19.9 368 2.4
S237 37. My parents or carers feel that they are always welcome in the Academy
27 7.2 56 14.9 179 47.5 100 26.5 362 4.0
S238 38. My parents or carers feel involved with my learning and my progress
28 7.4 63 16.7 180 47.7 95 25.2 366 2.9
S239 39. My parents or carers are pleased with my progress
21 5.6 50 13.3 181 48.0 113 30.0 365 3.2
S240 40. My parents or carers have always been able to sort out any problems that I have had
27 7.2 58 15.4 154 40.8 127 33.7 366 2.9
S241 41. I am encouraged to lead activities in the classroom
57 15.1 114 30.2 156 41.4 42 11.1 369 2.1
S242 42. I have had opportunities to lead other activities at the Academy
85 22.5 102 27.1 125 33.2 57 15.1 369 2.1
DIFFERENCES BETWEEN SCHOOLS BY QUESTIONS
Individually, independent t-tests were carried out on each item to examine whether there were
statistically significant mean differences in individual items between the Academies.
It was found that, on average, there were no mean differences in agreement or disagreement
between Academy 1 and Academy 3 student respondents to 10 out of 42 items. These include the
statements about ‘feel safe’ in the school (S22), ‘like work with other students’ (S27)or ‘take a lot of
care’ (S210) to help their learning or work, the expectation from teachers to improve on ‘personal
best’ (S225), achieving their best as they ‘possibly can’ (S226), ‘feel proud’ of what they have
62
achieved, their results ‘are as good as’ they can do (S229), parents or carers pleased with their
‘learning and progress’ (S232), they being encouraged to ‘lead activities in the classroom (S239),
and being offered opportunities to ‘lead other activities’ in the school (S241) (see shaded areas in
the Table 23 below).
However, Academy 1 student respondents reported averagely higher mean responses than their
counterparts of Academy 3 to the remaining 32 items at α = 0.05 level of statistical significance (see
non-shaded areas in the Table 26. below). In other words, to some degree, students in Academy 1
tend to agree more7 than Academy 3 students to the statements of, for example, ‘like learning new
things’ (S24),’concentrate on my learning (S25), ‘work hard’ (SS211), not minding ‘making mistakes’
(S213), ‘get on well’ with their teachers (S221), knowing that ‘the senior teachers want us to
learning successfully’ (S223), keeping up ‘a good rate of progress’ (S227), and their parents or carers
being involved with their learning and progress (S238) as well as being able to ‘sort out any
problems’ that they have had (S240). On the other hand, Academy 1 student respondents are also
likely to disagree less8 than Academy 3 student respondents on ‘feel valued and cared’ (S23), being
‘confident in my learning’ (S28), their teachers ‘teach well’ (S216), ‘everyone works very hard to
make’ their school a great place to learn (S224), and being ‘taught about the responsibilities of
students as well as their rights’ (S233).
Table 31 t-test for Equality of Means for each item (purple shaded = non significance at α = 0.05)
t-test for Equality of Means
(purple shaded = non significance
at α = 0.05)
t df Sig.
(2-
taile
d)
Mean Diff.
Ac Mn N SD Total Mean N SD
S21 1. I feel included in the activities that are available to me in the Academy
5.94 375.00 0.00 0.46 1 2.94 244 0.61
5.36 203.08 0.00 0.46 3 2.48 133 0.88 Total 2.78 377 0.75
S22 2. I feel safe in the Academy
1.96 375.00 0.05 0.17 1 3.11 244 0.66
1.75 198.36 0.08 0.17 3 2.95 133 0.99 Total 3.06 377 0.80
S23 3. I feel valued and cared for as an individual in the Academy
3.28 372.00 0.00 0.28 1 2.85 243 0.64
2.87 186.27 0.00 0.28 3 2.56 131 1.02 Total 2.75 374 0.81
S24 4. I like learning new things
3.68 374.00 0.00 0.27 1 3.30 244 0.55
3.21 186.31 0.00 0.27 3 3.02 132 0.89 Total 3.20 376 0.70
S25 5. I usually concentrate on my learning
2.82 373.00 0.01 0.21 1 3.17 243 0.54
2.47 185.83 0.01 0.21 3 2.96 132 0.88 Total 3.10 375 0.68
S26 6. I am hard to distract
2.87 374.00 0.00 0.26 1 2.28 244 0.73
2.64 211.47 0.01 0.26 3 2.02 132 0.98 Total 2.19 376 0.83
S27 7. I like working with other students to help my
1.55 372.00 0.12 0.14 1 3.07 243 0.72
1.41 204.20 0.16 0.14 3 2.93 131 0.99 Total 3.02 374 0.82
7 Referring to Academy 1 students reporting a mean of greater than 3.00 to the exampled statements
8 Referring to Academy 1 students reporting a mean of equal to or less than 3.00 to the exampled statements
63
learning
S28 9. I am confident in my learning
5.49 375.00 0.00 0.51 1 2.91 244 0.71
4.88 195.62 0.00 0.51 3 2.40 133 1.09 Total 2.73 377 0.90
S29 8. I don't distract other students
2.84 374.00 0.00 0.24 1 3.12 244 0.70
2.65 219.91 0.01 0.24 3 2.89 132 0.89 Total 3.04 376 0.78
S210 10. I take a lot of care with my work
1.28 375.00 0.20 0.09 1 3.11 244 0.59
1.18 216.73 0.24 0.09 3 3.02 133 0.78 Total 3.07 377 0.66
S211 11. I work hard
3.00 374.00 0.00 0.22 1 3.22 244 0.57
2.69 197.58 0.01 0.22 3 3.00 132 0.85 Total 3.14 376 0.69
S212 12. I do more than I am asked to do in class
4.82 373.00 0.00 0.39 1 2.64 242 0.66
4.41 211.47 0.00 0.39 3 2.26 133 0.89 Total 2.51 375 0.77
S213 13. I don't mind making mistakes because I learn from them
4.50 374.00 0.00 0.39 1 3.02 243 0.69
4.06 202.00 0.00 0.39 3 2.63 133 1.00 Total 2.89 376 0.83
S214 14. I come up with new ideas to help my learning
4.39 374.00 0.00 0.37 1 2.90 243 0.67
4.00 208.55 0.00 0.37 3 2.53 133 0.93 Total 2.77 376 0.79
S215 15. I enjoy my learning
6.42 374.00 0.00 0.55 1 2.96 243 0.68
5.83 206.62 0.00 0.55 3 2.41 133 0.95 Total 2.77 376 0.83
S216 16. My teachers teach well so that I learn successfully
4.01 371.00 0.00 0.35 1 2.95 243 0.69
3.63 200.25 0.00 0.35 3 2.61 130 0.97 Total 2.83 373 0.81
S217 17. I have regular opportunities to express my opinion to my teachers about my learning
3.82 369.00 0.00 0.35 1 2.68 241 0.76
3.57 219.21 0.00 0.35 3 2.33 130 0.95 Total 2.56 371 0.85
S218 18. I often have conversations with my teachers which help me to make better progress
3.69 370.00 0.00 0.34 1 2.58 243 0.75
3.37 203.91 0.00 0.34 3 2.23 129 1.02 Total 2.46 372 0.87
S219 19. My teachers seem to enjoy teaching my classes
5.28 372.00 0.00 0.46 1 2.78 244 0.67
4.71 192.92 0.00 0.46 3 2.32 130 0.99 Total 2.62 374 0.82
S220 20. My teachers often share things that they have learned themselves
4.27 371.00 0.00 0.36 1 2.94 242 0.67
3.87 203.58 0.00 0.36 3 2.58 131 0.94 Total 2.81 373 0.79
S221 21. I get on well with my teachers
3.15 372.00 0.00 0.27 1 3.04 243 0.69
2.86 203.79 0.00 0.27 3 2.76 131 0.97 Total 2.94 374 0.81
S222 22. I get extra support to help my learning when I need i
6.41 372.00 0.00 0.60 1 2.76 243 0.76
5.87 208.56 0.00 0.60 3 2.16 131 1.02 Total 2.55 374 0.90
S223 23. I know that the senior teachers want us to learn successfully
3.98 372.00 0.00 0.32 1 3.12 243 0.62
3.56 196.23 0.00 0.32 3 2.80 131 0.92 Total 3.01 374 0.75
S224 24. Everyone works very hard to make the Academy a great place to learn and to do well
3.29 372.00 0.00 0.29 1 2.91 244 0.76
3.11 225.17 0.00 0.29 3 2.62 130 0.91 Total 2.80 374 0.82
S225 25. My teachers constantly expect me to improve on my personal best
1.57 372.00 0.12 0.13 1 3.20 244 0.66
1.45 208.13 0.15 0.13 3 3.07 130 0.88 Total 3.15 374 0.75
S226 26. Right no w I am achieving the best I possibly can
1.19 367.00 0.23 0.11 1 2.96 239 0.74
1.09 206.80 0.28 0.11 3 2.85 130 1.00 Total 2.92 369 0.84
S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy
3.53 367.00 0.00 0.30 1 3.16 239 0.70
3.28 215.54 0.00 0.30 3 2.86 130 0.90 Total 3.05 369 0.79
64
S228 28. I have kept up a good rate of progress ever since my first year at the Academy
3.46 365.00 0.00 0.29 1 3.07 238 0.66
3.15 201.58 0.00 0.29 3 2.78 129 0.92 Total 2.97 367 0.77
S229 29. I think that my results at the moment are as good as I can do
-0.95 365.00 0.34 -0.09 1 2.71 238 0.83
-0.93 247.91 0.35 -0.09 3 2.80 129 0.89 Total 2.74 367 0.85
S230 30. I feel that my successes at the Academy are recognised
2.84 366.00 0.00 0.25 1 2.65 238 0.71
2.57 200.84 0.01 0.25 3 2.39 130 1.00 Total 2.56 368 0.83
S231 31. I feel that my successes with activities outside the Academy are recognised
2.31 365.00 0.02 0.23 1 2.51 238 0.78
2.08 195.17 0.04 0.23 3 2.28 129 1.14 Total 2.43 367 0.93
S232 32. I feel proud about what I have achieved so far
1.11 366.00 0.27 0.10 1 3.00 239 0.70
1.01 204.70 0.31 0.10 3 2.90 129 0.95 Total 2.96 368 0.80
S233 33. We are taught about the responsibilities of students as well as their rights
3.86 363.00 0.00 0.33 1 3.00 236 0.68
3.52 203.40 0.00 0.33 3 2.67 129 0.94 Total 2.89 365 0.80
S234 34. We care about helping each other in the Academy community
4.05 365.00 0.00 0.35 1 2.91 237 0.65
3.60 191.48 0.00 0.35 3 2.56 130 1.00 Total 2.79 367 0.81
S235 35. We are encouraged to care about the needs of other people in the local community and around the world
2.41 365.00 0.02 0.20 1 2.97 238 0.68
2.20 204.29 0.03 0.20 3 2.77 129 0.92 Total 2.90 367 0.78
S236 36. I regularly offer to help others
4.50 366.00 0.00 0.40 1 2.97 238 0.72
4.14 209.62 0.00 0.40 3 2.57 130 0.96 Total 2.83 368 0.83
S237 37. My parents or carers feel that they are always welcome in the Academy
3.86 360.00 0.00 0.36 1 3.10 234 0.71
3.48 195.29 0.00 0.36 3 2.74 128 1.03 Total 2.97 362 0.85
S238 38. My parents or carers feel involved with my learning and my progress
5.22 364.00 0.00 0.47 1 3.10 237 0.72
4.75 201.18 0.00 0.47 3 2.63 129 1.00 Total 2.93 366 0.86
S239 39. My parents or carers are pleased with my progress
1.93 363.00 0.05 0.17 1 3.12 236 0.71
1.76 202.03 0.08 0.17 3 2.95 129 0.99 Total 3.06 365 0.82
S240 40. My parents or carers have always been able to sort out any problems that I have had
3.90 364.00 0.00 0.37 1 3.17 237 0.75
3.51 196.48 0.00 0.37 3 2.80 129 1.08 Total 3.04 366 0.90
S241 41. I am encouraged to lead activities in the classroom
1.97 367.00 0.05 0.19 1 2.56 240 0.77
1.79 200.72 0.07 0.19 3 2.37 129 1.07 Total 2.50 369 0.89
S242 42. I have had opportunities to lead other activities at the Academy
0.42 367.00 0.68 0.05 1 2.43 240 0.92
0.39 217.03 0.70 0.05 3 2.39 129 1.15 Total 2.42 369 1.01
GENDER DIFFERENCE IN ITEM MEAN
Individual independent t-tests were carried out on each item to examine whether there were
statistically significant mean differences between genders.
65
The statistical results indicate that male and female student respondents on average provided
similar responses to 20 out of 42 items such as ‘feel included’ in available activities (S221), ‘like
learning new things’ (S224), ‘don’t distract other students’ (S229), ‘feel’ their successes at the
Academy recognised (S231), also about their teachers ‘teach well’ (S216) and ‘seem to enjoy
teaching’ their classes (S219) as well as ‘expect me to improve on my personal best’ (S225), in
addition to the statements about their parents or carers feeling that they are ‘always welcome in
the Academy’ (SS237), ‘involved with my learning and my progress (S238), and ‘pleased with my
progress’ (S240) (see shaded areas in the Table 24 below).
Interestingly, across the Academies female student respondents reported higher means than male
student respondents on the remaining 22 items at α = 0.05 level of statistical significance. For
example, female students generally agree more 9 than male counterparts that they ‘feel safe in the
Academy’ (S22), ‘concentrate on my learning’ (S25), ‘work with other students’ (S27), ‘care with my
work’ (S210) and ‘work hard’ (SS211), ‘get on well with my teachers’ (S221), keeping up ‘a good rate
of progress’ (S227) and ‘feel proud’ about what they have achieved (S232). And female students
also on average disagree less10 than male students on ‘confident in my learning’ (S28), ‘do more’
than being asked to do in the class (S212), ‘enjoying my learning’ (S215), ‘get extra support’ to help
their learning (S222), and their results ‘are as good as’ they can do (S229).
Table 32t-test for Equality of Means for each item by gender (purple shaded = non significance at α = 0.05)
Item t df Sig. (2-
tailed)
Mean
Diff.
Gender N Mean SD
S21 1. I feel included in the activities that are available to me in the Academy
-0.94 374.00 0.35 -0.07 1 Male 198 2.74 0.82
-0.95 369.67 0.34 -0.07 2 Female 178 2.81 0.66
S22 2. I feel safe in the Academy -2.62 374.00 0.01 -0.21 1 Male 198 2.95 0.87
-2.66 367.22 0.01 -0.21 2 Female 178 3.17 0.68
S23 3. I feel valued and cared for as an individual in the Academy
-0.37 371.00 0.71 -0.03 1 Male 197 2.74 0.82
-0.37 367.90 0.71 -0.03 2 Female 176 2.77 0.80
S24 4. I like learning new things 0.24 373.00 0.81 0.02 1 Male 197 3.21 0.72
0.24 372.75 0.81 0.02 2 Female 178 3.19 0.67
S25 5. I usually concentrate on my learning -3.80 372.00 0.00 -0.26 1 Male 198
2.97 0.69
-3.82 371.43 0.00 -0.26 2 Female 176 3.24 0.64
S26 6. I am hard to distract -1.68 373.00 0.09 -0.14 1 Male 198 2.12 0.85
-1.69 372.10 0.09 -0.14 2 Female 177 2.26 0.80
S27 7. I like working with other students to help my learning
-2.11 371.00 0.04 -0.18 1 Male 196 2.94 0.83
-2.12 368.25 0.04 -0.18 2 Female 177 3.12 0.81
S28 9. I am confident in my learning -3.78 374.00 0.00 -0.34 1 Male 198 2.57 0.94
-3.81 373.72 0.00 -0.34 2 Female 178 2.91 0.82
S29 8. I don't distract other students -1.05 373.00 0.29 -0.08 1 Male 198 3.00 0.80
-1.06 372.00 0.29 -0.08 2 Female 177 3.08 0.75
S210 10. I take a lot of care with my work -4.92 374.00 0.00 -0.33 1 Male 198 2.92 0.71
-4.98 366.93 0.00 -0.33 2 Female 178 3.25 0.56
9 Referring to female students reporting a mean of greater than 3.00 to the exampled statements
10 Referring to female students reporting a mean of equal to or less than 3.00 to the exampled statements
66
S211 11. I work hard -3.24 373.00 0.00 -0.23 1 Male 197 3.04 0.77
-3.29 361.36 0.00 -0.23 2 Female 178 3.26 0.58
S212 12. I do more than I am asked to do in class -3.07 372.00 0.00 -0.24 1 Male 196 2.39 0.79
-3.08 371.98 0.00 -0.24 2 Female 178 2.63 0.73
S213 13. I don't mind making mistakes because I learn from them
-2.04 373.00 0.04 -0.17 1 Male 198 2.80 0.84
-2.04 370.81 0.04 -0.17 2 Female 177 2.98 0.81
S214 14. I come up with new ideas to help my learning
-0.28 373.00 0.78 -0.02 1 Male 198 2.76 0.79
-0.28 367.89 0.78 -0.02 2 Female 177 2.79 0.80
S215 15. I enjoy my learning -2.68 373.00 0.01 -0.23 1 Male 197 2.66 0.87
-2.70 372.76 0.01 -0.23 2 Female 178 2.89 0.77
S216 16. My teachers teach well so that I learn successfully
-0.96 370.00 0.34 -0.08 1 Male 195 2.79 0.86
-0.96 369.68 0.34 -0.08 2 Female 177 2.88 0.76
S217 17. I have regular opportunities to express my opinion to my teachers about my learning
-1.78 368.00 0.08 -0.16 1 Male 194 2.48 0.89
-1.79 367.94 0.07 -0.16 2 Female 176 2.64 0.80
S218 18. I often have conversations with my teachers which help me to make better progress
-1.18 369.00 0.24 -0.11 1 Male 195 2.41 0.92
-1.18 368.91 0.24 -0.11 2 Female 176 2.51 0.81
S219 19. My teachers seem to enjoy teaching my classes
-1.81 371.00 0.07 -0.15 1 Male 195 2.55 0.83
-1.81 368.82 0.07 -0.15 2 Female 178 2.70 0.81
S220 20. My teachers often share things that they have learned themselves
-0.80 370.00 0.42 -0.07 1 Male 196 2.78 0.83
-0.81 369.95 0.42 -0.07 2 Female 176 2.85 0.75
S221 21. I get on well with my teachers -3.11 371.00 0.00 -0.26 1 Male 195 2.82 0.85
-3.13 370.63 0.00 -0.26 2 Female 178 3.08 0.75
S222 22. I get extra support to help my learning when I need it
-3.22 371.00 0.00 -0.30 1 Male 196 2.41 0.93
-3.23 370.97 0.00 -0.30 2 Female 177 2.71 0.85
S223 23. I know that the senior teachers want us to learn successfully
-1.46 371.00 0.15 -0.11 1 Male 195 2.95 0.83
-1.47 362.69 0.14 -0.11 2 Female 178 3.07 0.65
S224 24. Everyone works very hard to make the Academy a great place to learn and to do well
-2.90 371.00 0.00 -0.25 1 Male 195 2.69 0.87
-2.92 370.23 0.00 -0.25 2 Female 178 2.93 0.76
S225 25. My teachers constantly expect me to improve on my personal best
-1.92 371.00 0.06 -0.15 1 Male 195 3.08 0.78
-1.93 370.92 0.05 -0.15 2 Female 178 3.23 0.70
S226 26. Right now I am achieving the best I possibly can
-3.85 366.00 0.00 -0.33 1 Male 193 2.77 0.84
-3.85 364.61 0.00 -0.33 2 Female 175 3.10 0.81
S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy
-2.41 366.00 0.02 -0.20 1 Male 193 2.96 0.83
-2.43 365.11 0.02 -0.20 2 Female 175 3.16 0.72
S228 28. I have kept up a good rate of progress ever since my first year at the Academy
-2.34 364.00 0.02 -0.19 1 Male 192 2.88 0.82
-2.36 363.28 0.02 -0.19 2 Female 174 3.07 0.71
S229 29. I think that my results at the moment are as good as I can do
-2.69 364.00 0.01 -0.24 1 Male 192 2.63 0.84
-2.69 359.92 0.01 -0.24 2 Female 174 2.87 0.85
S230 30. I feel that my successes at the Academy are recognised
-1.60 365.00 0.11 -0.14 1 Male 192 2.49 0.89
-1.62 362.96 0.11 -0.14 2 Female 175 2.63 0.75
S231 31. I feel that my successes with activities outside the Academy are recognised
-1.39 364.00 0.16 -0.14 1 Male 192 2.36 0.99
-1.40 363.37 0.16 -0.14 2 Female 174 2.50 0.86
67
S232 32. I feel proud about what I have achieved so far
-3.40 365.00 0.00 -0.28 1 Male 193 2.83 0.85
-3.43 363.65 0.00 -0.28 2 Female 174 3.11 0.72
S233 33. We are taught about the responsibilities of students as well as their rights
-2.12 362.00 0.03 -0.18 1 Male 191 2.81 0.84
-2.14 361.60 0.03 -0.18 2 Female 173 2.98 0.74
S234 34. We care about helping each other in the Academy community
-4.27 364.00 0.00 -0.35 1 Male 191 2.62 0.89
-4.32 349.81 0.00 -0.35 2 Female 175 2.97 0.66
S235 35. We are encouraged to care about the needs of other people in the local community and around the world
-3.30 364.00 0.00 -0.27 1 Male 191 2.77 0.79
-3.31 363.90 0.00 -0.27 2 Female 175 3.04 0.74
S236 36. I regularly offer to help others -3.12 365.00 0.00 -0.27 1 Male 192 2.70 0.88
-3.14 363.28 0.00 -0.27 2 Female 175 2.97 0.75
S237 37. My parents or carers feel that they are always welcome in the Academy
-1.58 359.00 0.12 -0.14 1 Male 188 2.90 0.88
-1.58 358.98 0.11 -0.14 2 Female 173 3.05 0.82
S238 38. My parents or carers feel involved with my learning and my progress
-1.15 363.00 0.25 -0.10 1 Male 191 2.88 0.89
-1.16 362.98 0.25 -0.10 2 Female 174 2.99 0.82
S239 39. My parents or carers are pleased with my progress
-1.84 362.00 0.07 -0.16 1 Male 189 2.98 0.86
-1.85 361.83 0.07 -0.16 2 Female 175 3.14 0.78
S240 40. My parents or carers have always been able to sort out any problems that I have had
-1.44 363.00 0.15 -0.13 1 Male 190 2.97 0.92
-1.44 362.73 0.15 -0.13 2 Female 175 3.11 0.87
S241 41. I am encouraged to lead activities in the classroom
-1.53 366.00 0.13 -0.14 1 Male 191 2.43 0.87
-1.52 361.66 0.13 -0.14 2 Female 177 2.57 0.90
S242 42. I have had opportunities to lead other activities at the Academy
-0.77 366.00 0.44 -0.08 1 Male 191 2.38 1.07
-0.77 365.08 0.44 -0.08 2 Female 177 2.46 0.94
UNDERLYING THEMES IN THE DATA
An exploratory factor analysis was conducted to investigate whether there are any potential
themes underlying the items. The following rotated component matrix deleted six items (S21, S22,
S26, S27, S28, S211, S214) and produced the following six themes or factors. These factors could
form the basis for a more powerful scale for future surveys and reduction of data in a future
Perimeta model. They require further interpretation, however they represent key themes through
which the students understood or construed their experience at their Academy.
Table 33 Underlying themes: rotated component matrix from exploratory factor analysis with alpha reliability co-efficient for each resulting scale
1 2 3 4 5 6 α S222 22. I get extra support to help my learning when I need it .645 .203 .023 .125 .119 .110
S217 17. I have regular opportunities to express my opinion to my teachers about my learning
.639 .196 .173 .105 .036 .277
S216 16. My teachers teach well so that I learn successfully .634 .239 .285 .253 .017 -.062 S23 3. I feel valued and cared for as an individual in the Academy .573 -.066 .352 .229 .202 .105
S218 18. I often have conversations with my teachers which help me to make better progress
.565 .184 .023 .161 .072 .379
S233 33. We are taught about the responsibilities of students as well as their rights
.510 .112 .087 .375 .289 .141
68
S230 30. I feel that my successes at the Academy are recognised .474 -.040 .349 .021 .263 .391 S219 19. My teachers seem to enjoy teaching my classes .427 .140 .232 .412 .130 .094
S220 20. My teachers often share things that they have learned themselves
.411 .237 .020 .216 .268 -.017 0.84
S24 4. I like learning new things .112 .661 .078 -.063 .146 -.041 S215 15. I enjoy my learning .294 .646 .155 .104 .285 .030
S25 5. I usually concentrate on my learning .085 .607 .329 .293 .097 .058 S236 36. I regularly offer to help others .104 .530 .130 .132 .273 .244 S29 8. I don't distract other students .167 .527 .359 .085 .145 .079
S210 10. I take a lot of care with my work -.004 .517 .428 .267 -.036 .139 S213 13. I don't mind making mistakes because I learn from them .222 .514 .026 .308 .129 -.007
S212 12. I do more than I am asked to do in class .165 .501 .235 .134 -.077 .433 0.82 S226 26. Right now I am achieving the best I possibly can .058 .222 .703 .229 .124 .166
S228 28. I have kept up a good rate of progress ever since my first year at the Academy
.236 .289 .655 .046 .221 .032
S227 27. From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the
Academy
.313 .184 .654 .019 .231 -.138
S229 29. I think that my results at the moment are as good as I can do .110 .106 .585 .149 .100 .288 S232 32. I feel proud about what I have achieved so far .112 .220 .539 .317 .248 .219 0.81
S224 24. Everyone works very hard to make the Academy a great place to learn and to do well
.238 .048 .064 .685 .104 .220
S234 34. We care about helping each other in the Academy community
.278 .106 .117 .609 .262 .254
S235 35. We are encouraged to care about the needs of other people in the local community and around the world
.226 .154 .146 .605 .320 .110
S221 21. I get on well with my teachers .335 .348 .329 .458 -.109 -.005 S223 23. I know that the senior teachers want us to learn successfully .356 .239 .180 .408 .284 -.061
S225 25. My teachers constantly expect me to improve on my personal best
.028 .292 .246 .381 .060 -.008 0.76
S240 40. My parents or carers have always been able to sort out any problems that I have had
.070 .218 .151 .190 .706 .044
S238 38. My parents or carers feel involved with my learning and my progress
.281 .133 .128 .164 .680 .161
S237 37. My parents or carers feel that they are always welcome in the Academy
.144 .135 .292 .309 .569 .169
S239 39. My parents or carers are pleased with my progress .133 .272 .424 .026 .541 .033 0.78 S241 41. I am encouraged to lead activities in the classroom .013 .205 .157 .108 .126 .764
S242 42. I have had opportunities to lead other activities at the Academy
.246 -.085 .032 .141 .044 .668
S231 31. I feel that my successes with activities outside the Academy are recognised
.401 .043 .101 .074 .342 .445 0.64
Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.a. Rotation converged in 10 iterations.
These factors suggest the presence of underlying themes in the data. These could be computed into
new variables which would provide a more economical and interpretable measure, with more
statistical power. Table 26 summarises these themes.
69
Table 34 Themes from the factor analysis
SUMMARY AND FINDINGS FROM STRAND TWO
For Strand Two the questionnaire was designed to assess students' perceptions about their
learning, teaching, progress and achievement in an Oasis Academy.
The findings showed there to be no mean differences in agreement or disagreement for 10 of the
42 items assessing student perceptions on their learning, teaching, progress and achievement in an
Oasis Academy. However, Academy 1 student respondents reported averagely higher mean
responses than their counterparts of Academy 3 to the remaining 32 items which suggests that, to
some degree, students in Academy 1 tend to agree more than Academy 3 students.
With regards to gender differences, the statistical results indicated that male and female student
respondents on average provided similar responses to 20 out of 42 items, although, across the
Academies female student respondents reported significantly higher means than male student
respondents on the remaining 22 items. Thus, there appear to be gender differences on over half
of the items, possibly suggesting a difference in how males and females perceive at least some of
their learning, teaching, progress and achievement.
There were six underlying themes in the data which could be used for a more economical
evaluation and presentation of data. They also suggest that how students perceive their schooling is
consistent with the values of the Oasis Community Learning Charter.
Theme one I have good relationships for learning in my classroom and community
Theme two I enjoy learning and I take responsibility for it
Theme three I am doing as well as I can
Theme four Everybody in my Academy cares about my learning
Theme five My family is involved in my learning
Theme six I have opportunities to exercise leadership inside and outside the Academy
70
STRAND THREE - STORIES OF TRANSFORMATION
Strand Three was a narrative interview with a small sample of students which aimed to explore
whether students were deeply engaged in their learning and whether they were able to recount
stories of significant personal change.
THE SAMPLE
The sample consisted of 22 students in Year 9 from two of the Academies. Thirteen were from
Academy 1 and nine were from Academy 3. Academy 2 data was not available for this analysis.
Table 35 Strand 3 Sample of students by Academy and gender
Gender Total
Male Female
Academy Lords Hill 4 9 13
Wintringham 4 5 9
Total 8 14 22
NARRATIVE INTERVIEW SCHEDULE
Each student was invited to talk with a teacher in response to the following questions.
1. Tell me about a time when you learned something really successfully in the last two or three
weeks
Why did you learn successfully?
What did it feel like
What happened as a result?
2. Tell me about your worst experience of learning in the last two or three weeks?
Why was it a bad experience of learning?
What did it feel like?
What happened as a result?
3. How different are you now as a learner compared with a year ago?
How would you describe the differences?
What has happened to make these differences?
How do you feel now about the future as a learner?
What changes would help you to learn better?
71
4. How would you describe yourself as a learner now? If you could choose an animal (or a car)
which is most like you as a learner, which animal (or car) would it be? Why did you choose
that particular animal (or car)? [Probe further for reasons, if necessary]
5. What have been the best things about your time at the Academy so far?
6. What have been the worst things about your time at the Academy so far?
7. How do you feel about your future, after your time at the Academy?
ANALYSIS
The interviews were recorded and transcribed. The analysis was thematic, based on the criteria for
deep engagement in learning developed in the Learning Futures Projects 2010/11. The key question
was ‘Is deep transformative learning taking place?’ Judgements were based on the identification of
the following themes:
Authenticity –self-authorship and a genuine and meaningful learning process
Identity – a rich language of learning described in terms of learning as a journey or active
process
Agency – taking responsibility for own learning and purposeful application
Stories of significant learning
Each student was rated on a scale of 1 to 4 as follows:
4 = very high evidence of deep transformative learning taking place
3 = high evidence of deep transformative learning taking place
2 = low evidence of deep transformative learning taking place
1 = very low evidence of deep transformative learning taking place
FINDINGS
The following tables show the scores for each student in Academy 1. There was no significant
difference between genders.
Table 36 Rating score for students in Academy 1
Frequency Percent Valid Percent Cumulative Percent
Evidence of deep
transformative
learning taking
place
very low 1 7.7 7.7 7.7
low 5 38.5 38.5 46.2
high 6 46.2 46.2 92.3
very high 1 7.7 7.7 100.0
72
Frequency Percent Valid Percent Cumulative Percent
Evidence of deep
transformative
learning taking
place
very low 1 7.7 7.7 7.7
low 5 38.5 38.5 46.2
high 6 46.2 46.2 92.3
very high 1 7.7 7.7 100.0
Total 13 100.0 100.0
Table 37Histogram of rating score for students in Academy 1
SUMMARY
From this sample we there was only one student who demonstrated very little evidence of
transformative learning in his or her narrative. Overall there was greater evidence of the presence
of transformative learning in the sample with one student ranking very high.
STRAND FOUR - POST-16 TRANSITION AND PROGRESS TO ADULTHOOD
This questionnaire was designed to explore the experiences of leavers from two Academies in
terms of the Oasis Charter, and their expectations of their transition to the next stage of education.
73
SAMPLE
The questionnaire contained 30 items and was administrated in two Academies and in total
received responses from 138 student participants in Year 11.
FINDINGS
The following Table 38 shows the frequency of responses to each question as a number and as a
percentage. The final column shows the missing values.
Table 38 Strand Four frequency of value response per question for Y11 leavers
1
Strongly
Disagree
2
Disagree
3
Agree
4
Strongly
Agree
missing
Item N % N % N % N % N %
S41: I was valued and included during my time at the Academy
0 0.0 7 5.1 99 71.7 32 23.2 0 0.0
S42: I felt part of a community that was focused on learning
2 1.4 9 6.5 101 73.2 26 18.8 0 0.0
S43: I felt safe at the Academy 1 0.7 7 5.1 88 63.8 42 30.4 0 0.0
S44: I enjoyed my time at the Academy 1 0.7 10 7.2 88 63.8 38 27.5 1 0.7
S45: I did as well as I had hoped in my GCSEs
1 0.7 25 18.1 84 60.9 22 15.9 6 4.3
S46: My teachers accurately predicted my GCSE results
3 2.2 27 19.6 80 58.0 22 15.9 6 4.3
S47: I am looking forward to the move to Sixth Form, Further Education, training or work
3 2.2 14 10.1 72 52.2 46 33.3 3 2.2
S48: I have been well prepared for the next stage of my life
1 0.7 14 10.1 90 65.2 29 21.0 4 2.9
S49: My successes were recognised during my time at the Academy
3 2.2 8 5.8 97 70.3 26 18.8 4 2.9
S410: I was helped and supported whenever I struggled with my learning
2 1.4 15 10.9 85 61.6 32 23.2 4 2.9
S411: I developed a sense of responsibility towards other people during my time at the Academy
1 0.7 13 9.4 90 65.2 29 21.0 5 3.6
S412: The Academy helped to make me more understanding about different cultures and types of people
4 2.9 13 9.4 94 68.1 24 17.4 3 2.2
S413: I supported activities that would improve things for other people
2 1.4 20 14.5 96 69.6 17 12.3 3 2.2
S414: I was offered opportunities for leadership during my time at the Academy
5 3.6 23 16.7 84 60.9 23 16.7 3 2.2
S415: I led activities during my time at the Academy
4 2.9 39 28.3 71 51.4 21 15.2 3 2.2
S416: My teachers taught me well at the Academy
2 1.4 10 7.2 77 55.8 45 32.6 4 2.9
S417: My teachers were always trying to improve their teaching
1 0.7 18 13.0 84 60.9 32 23.2 3 2.2
S418: I was regularly challenged to do 1 0.7 16 11.6 89 64.5 28 20.3 4 2.9
74
better so that I could reach my full potential
S419: My parents/carers felt involved with my education at the Academy
7 5.1 23 16.7 80 58.0 22 15.9 6 4.3
S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
5 3.6 14 10.1 81 58.7 34 24.6 4 2.9
S421: Senior staff at the Academy wanted us to learn well and achieve our best
0 0.0 7 5.1 92 66.7 33 23.9 6 4.3
S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
1 0.7 15 10.9 87 63.0 26 18.8 9 6.5
S423: The Academy was well run 1 0.7 13 9.4 84 60.9 30 21.7 10 7.2
S424: There were great facilities for learning
0 0.0 9 6.5 83 60.1 36 26.1 10 7.2
S425: There was a broad curriculum that helped me to develop as a whole person
0 0.0 17 12.3 84 60.9 25 18.1 12 8.7
S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
2 1.4 14 10.1 88 63.8 24 17.4 10 7.2
S427: The Academy made good use of new technologies for learning
1 0.7 9 6.5 87 63.0 32 23.2 9 6.5
S428: I never felt that anyone was putting limits on what I could achieve
2 1.4 15 10.9 84 60.9 28 20.3 9 6.5
S429: I feel proud about what I achieved at the Academy
2 1.4 16 11.6 84 60.9 25 18.1 11 8.0
S430: I could not have gone to a better place for my secondary education
6 4.3 12 8.7 75 54.3 35 25.4 10 7.2
Interestingly, student respondents provided overall positive responses to the majority of the
statements. Four in five (at least 85%) respondents reported agree or strongly agree to 24 out of
30 items (see Table 38). Particularly, 30.4%, 33.6% and 34.1% of the students expressed their strong
agreement on feeling safe when they were in the Academy (S43), their Academy teachers teaching
them well (S416) and looking forward to their future life (S47) respectively. The 6 items that
received the responses of disagree or strongly disagree from 16.3% up to 31.9% of student
respondents (see Table 38) with a mean just below 3.00 (see Table 40) were about lending their
support to activities that ‘would improve things for other people’ (S413), meeting self-expectation
in GCSE (S45), being offered opportunities for leadership (S414), teachers having precisely predicted
what they could achieve in GCSE (S46), parents/carers’ feeling in involvement with ‘my education at
the Academy’ (S419) and having led activates during their time in the Academy (S415).
Table 39 ranks the questions in order of the strength of agreement, combining strongly agree and
agree into one score and strongly disagree and disagree into one score.
Table 39 Rank order of agreement to each question
Item number and description 1+2 Disagree / Strongly Disagree
1 Strongly Disagree
2 Disagree
3 Agree
4 Strongly
Agree
3+4 Agree/
Strongly Agree
Item N % N % N % N % N % N %
S41: I was valued and included during my time at the Academy
7 5.1 0 0.0 7 5.1 99 71.7 32 23.2 131 94.9
75
S421: Senior staff at the Academy wanted us to learn well and achieve our best
7 5.3 0 0.0 7 5.3 92 69.7 33 25.0 125 94.7
S43: I felt safe at the Academy 8 5.8 1 0.7 7 5.1 88 63.8 42 30.4 130 94.2
S424: There were great facilities for learning
9 7.0 0 0.0 9 7.0 83 64.8 36 28.1 119 93.0
S427: The Academy made good use of new technologies for learning
10 7.8 1 0.8 9 7.0 87 67.4 32 24.8 119 92.2
S42: I felt part of a community that was focused on learning
11 8.0 2 1.4 9 6.5 101 73.2 26 18.8 127 92.0
S44: I enjoyed my time at the Academy
11 8.0 1 0.7 10 7.3 88 64.2 38 27.7 126 92.0
S49: My successes were recognised during my time at the Academy
11 8.2 3 2.2 8 6.0 97 72.4 26 19.4 123 91.8
S416: My teachers taught me well at the Academy
12 9.0 2 1.5 10 7.5 77 57.5 45 33.6 122 91.0
S411: I developed a sense of responsibility towards other people during my time at the Academy
14 10.5 1 0.8 13 9.8 90 67.7 29 21.8 119 89.5
S423: The Academy was well run 14 10.9 1 0.8 13 10.2 84 65.6 30 23.4 114 89.1
S48: I have been well prepared for the next stage of my life
15 11.2 1 0.7 14 10.4 90 67.2 29 21.6 119 88.8
S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
16 12.4 1 0.8 15 11.6 87 67.4 26 20.2 113 87.6
S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
16 12.5 2 1.6 14 10.9 88 68.8 24 18.8 112 87.5
S47: I am looking forward to the move to Sixth Form, Further Education, training or work
17 12.6 3 2.2 14 10.4 72 53.3 46 34.1 118 87.4
S412: The Academy helped to make me more understanding about different cultures and types of people
17 12.6 4 3.0 13 9.6 94 69.6 24 17.8 118 87.4
S410: I was helped and supported whenever I struggled with my learning
17 12.7 2 1.5 15 11.2 85 63.4 32 23.9 117 87.3
S418: I was regularly challenged to do better so that I could reach my full potential
17 12.7 1 0.7 16 11.9 89 66.4 28 20.9 117 87.3
S428: I never felt that anyone was putting limits on what I could achieve
17 13.2 2 1.6 15 11.6 84 65.1 28 21.7 112 86.8
S425: There was a broad curriculum that helped me to develop as a whole person
17 13.5 0 0.0 17 13.5 84 66.7 25 19.8 109 86.5
S417: My teachers were always trying to improve their teaching
19 14.1 1 0.7 18 13.3 84 62.2 32 23.7 110 85.9
S430: I could not have gone to a better place for my secondary education
18 14.1 6 4.7 12 9.4 75 58.6 35 27.3 116 85.9
S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
19 14.2 5 3.7 14 10.4 81 60.4 34 25.4 109 85.8
S429: I feel proud about what I achieved at the Academy
18 14.2 2 1.6 16 12.6 84 66.1 25 19.7 115 85.8
S413: I supported activities that would improve things for other
22 16.3 2 1.5 20 14.8 96 71.1 17 12.6 113 83.7
76
people
S45: I did as well as I had hoped in my GCSEs
26 19.7 1 0.8 25 18.9 84 63.6 22 16.7 106 80.3
S414: I was offered opportunities for leadership during my time at the Academy
28 20.7 5 3.7 23 17.0 84 62.2 23 17.0 107 79.3
S46: My teachers accurately predicted my GCSE results
30 22.7 3 2.3 27 20.5 80 60.6 22 16.7 102 77.3
S419: My parents/carers felt involved with my education at the Academy
30 22.7 7 5.3 23 17.4 80 60.6 22 16.7 102 77.3
S415: I led activities during my time at the Academy
43 31.9 4 3.0 39 28.9 71 52.6 21 15.6 92 68.1
The following Table 40 shows the mean score and standard deviation for each question, ranked
from the lowest score. Table 40 Mean score per item, ranked from lowest to highest
Item N Min Max Mean SD
S415: I led activities during my time at the Academy 135 1 4 2.81 0.73
S419: My parents/carers felt involved with my education at the Academy 132 1 4 2.89 0.74
S46: My teachers accurately predicted my GCSE results 132 1 4 2.92 0.68
S414: I was offered opportunities for leadership during my time at the Academy 135 1 4 2.93 0.70
S413: I supported activities that would improve things for other people 135 1 4 2.95 0.58
S45: I did as well as I had hoped in my GCSEs 132 1 4 2.96 0.62
S412: The Academy helped to make me more understanding about different cultures and types of people
135 1 4 3.02 0.63
S429: I feel proud about what I achieved at the Academy 127 1 4 3.04 0.62
S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
128 1 4 3.05 0.60
S425: There was a broad curriculum that helped me to develop as a whole person 126 2 4 3.06 0.58
S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
129 1 4 3.07 0.59
S428: I never felt that anyone was putting limits on what I could achieve 129 1 4 3.07 0.63
S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
134 1 4 3.07 0.71
S418: I was regularly challenged to do better so that I could reach my full potential
134 1 4 3.07 0.60
S430: I could not have gone to a better place for my secondary education 128 1 4 3.09 0.74
S417: My teachers were always trying to improve their teaching 135 1 4 3.09 0.63
S49: My successes were recognised during my time at the Academy 134 1 4 3.09 0.58
S42: I felt part of a community that was focused on learning 138 1 4 3.09 0.55
S48: I have been well prepared for the next stage of my life 134 1 4 3.10 0.59
S410: I was helped and supported whenever I struggled with my learning 134 1 4 3.10 0.64
S411: I developed a sense of responsibility towards other people during my time at the Academy
133 1 4 3.11 0.58
S423: The Academy was well run 128 1 4 3.12 0.60
S427: The Academy made good use of new technologies for learning 129 1 4 3.16 0.57
S41: I was valued and included during my time at the Academy 138 2 4 3.18 0.50
S44: I enjoyed my time at the Academy 137 1 4 3.19 0.59
S47: I am looking forward to the move to Sixth Form, Further Education, training or work
135 1 4 3.19 0.71
S421: Senior staff at the Academy wanted us to learn well and achieve our best 132 2 4 3.20 0.52
S424: There were great facilities for learning 128 2 4 3.21 0.56
S416: My teachers taught me well at the Academy 134 1 4 3.23 0.65
S43: I felt safe at the Academy 138 1 4 3.24 0.57
77
COMPARISONS BETWEEN ACADEMIES
Individual independent t-tests were carried out on each item to examine whether there were
statistically significant mean differences in individual items between the Academies. In general,
there were no statistical mean differences at α = 0.05 level between the two Academies in 24 out of
30 items indicating the student respondents from the Academies had similar views on these
statements. However, Academy 1 had lower means at α = 0.05 level of statistical significance in the
remaining six items than Academy 3 (see purple shaded in the Table 41 below). On average,
Academy 3 student respondents tended to report a greater degree of agreement on enjoying the
time at the Academy (S44), doing well along with GCSE self-expectation (S45), not ever feeling
‘anyone was putting limits on what I could achieve’ (S428), staff wanting them to achieve their best
(S421), their Academy being well managed (S423), and considering their Academy being best place
for their education (S430).
Table 41 Comparison between Academies on each question
t-test for Equality of Means
(purple shaded = significance at α
= 0.05)
t df Sig. (2-tailed)
Mean Diff.
Academy N Mean SD Total N Mean SD
S41 S41: I was valued and included during my time at the Academy
-0.50 136.00 0.62 -0.04 1 52 3.15 0.50 138 3.18 0.50
-0.50 108.40 0.62 -0.04 3 86 3.20 0.50 Total
S42 S42: I felt part of a community that was focused on learning
-0.92 136.00 0.36 -0.09 1 52 3.04 0.48 138 3.09 0.55
-0.97 123.91 0.34 -0.09 3 86 3.13 0.59 Total
S43 S43: I felt safe at the Academy
-1.67 136.00 0.10 -0.17 1 52 3.13 0.53 138 3.24 0.57
-1.73 118.26 0.09 -0.17 3 86 3.30 0.60 Total S44 S44: I enjoyed my time at the Academy
-2.34 135.00 0.02 -0.24 1 51 3.04 0.56 137 3.19 0.59
-2.37 108.50 0.02 -0.24 3 86 3.28 0.59 Total S45 S45: I did as well as I had hoped in my GCSEs
-2.16 130.00 0.03 -0.24 1 46 2.80 0.69 132 2.96 0.62
-2.04 78.82 0.04 -0.24 3 86 3.05 0.57 Total S46 S46: My teachers accurately predicted my GCSE results
-1.37 130.00 0.17 -0.17 1 47 2.81 0.68 132 2.92 0.68
-1.36 94.08 0.18 -0.17 3 85 2.98 0.67 Total
S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work
-1.21 133.00 0.23 -0.15 1 51 3.10 0.64 135 3.19 0.71
-1.26 117.68 0.21 -0.15 3 84 3.25 0.74 Total
S48 S48: I have been well prepared for the next stage of my life
-0.87 132.00 0.39 -0.09 1 50 3.04 0.57 134 3.10 0.59
-0.88 106.93 0.38 -0.09 3 84 3.13 0.60 Total
S49 S49: My successes were recognised during my time at the Academy
-1.09 132.00 0.28 -0.11 1 51 3.02 0.62 134 3.09 0.58
-1.07 97.95 0.29 -0.11 3 83 3.13 0.56 Total
S410 S410: I was helped and supported whenever I struggled with my learning
-1.37 132.00 0.17 -0.15 1 50 3.00 0.67 134 3.10 0.64
-1.34 95.67 0.18 -0.15 3 84 3.15 0.61 Total
S411 S411: I developed a sense of responsibility towards other people during my time at the
0.19 131.00 0.85 0.02 1 51 3.12 0.55 133 3.11 0.58
0.20 112.79 0.84 0.02 3 82 3.10 0.60 Total
78
Academy
S412 S412: The Academy helped to make me more understanding about different cultures and types of people
-0.88 133.00 0.38 -0.10 1 51 2.96 0.56 135 3.02 0.63
-0.92 118.93 0.36 -0.10 3 84 3.06 0.66 Total
S413 S413: I supported activities that would improve things for other people
-1.03 133.00 0.30 -0.11 1 51 2.88 0.55 135 2.95 0.58
-1.05 111.17 0.30 -0.11 3 84 2.99 0.59 Total
S414 S414: I was offered opportunities for leadership during my time at the Academy
-0.31 133.00 0.76 -0.04 1 51 2.90 0.57 135 2.93 0.70
-0.33 127.02 0.74 -0.04 3 84 2.94 0.77 Total
S415 S415: I led activities during my time at the Academy
-0.77 133.00 0.44 -0.10 1 51 2.75 0.66 135 2.81 0.73
-0.80 118.16 0.42 -0.10 3 84 2.85 0.77 Total
S416 S416: My teachers taught me well at the Academy
-1.32 132.00 0.19 -0.15 1 51 3.14 0.53 134 3.23 0.65
-1.41 126.79 0.16 -0.15 3 83 3.29 0.71 Total
S417 S417: My teachers were always trying to improve their teaching
-1.00 133.00 0.32 -0.11 1 51 3.02 0.65 135 3.09 0.63
-0.99 101.62 0.33 -0.11 3 84 3.13 0.62 Total
S418 S418: I was regularly challenged to do better so that I could reach my full potential
-0.24 132.00 0.81 -0.03 1 51 3.06 0.54 134 3.07 0.60
-0.25 117.42 0.80 -0.03 3 83 3.08 0.63 Total
S419 S419: My parents/carers felt involved with my education at the Academy
0.68 130.00 0.50 0.09 1 51 2.94 0.68 132 2.89 0.74
0.70 117.11 0.49 0.09 3 81 2.85 0.78 Total
S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
1.05 132.00 0.30 0.13 1 51 3.16 0.61 134 3.07 0.71
1.11 122.92 0.27 0.13 3 83 3.02 0.76 Total
S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best
-3.25 130.00 0.00 -0.29 1 51 3.02 0.47 132 3.20 0.52
-3.32 113.94 0.00 -0.29 3 81 3.31 0.52 Total
S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
-1.07 127.00 0.29 -0.11 1 50 3.00 0.49 129 3.07 0.59
-1.13 121.88 0.26 -0.11 3 79 3.11 0.64 Total
S423 S423: The Academy was well run
-2.11 126.00 0.04 -0.23 1 50 2.98 0.59 128 3.12 0.60
-2.11 104.65 0.04 -0.23 3 78 3.21 0.59 Total S424 S424: There were great facilities for learning
-1.82 126.00 0.07 -0.18 1 50 3.10 0.46 128 3.21 0.56
-1.93 121.67 0.06 -0.18 3 78 3.28 0.60 Total S425 S425: There was a broad curriculum that helped me to develop as a whole person
-0.67 124.00 0.51 -0.07 1 49 3.02 0.52 126 3.06 0.58
-0.69 113.87 0.49 -0.07 3 77 3.09 0.61 Total
S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
-0.40 126.00 0.69 -0.04 1 50 3.02 0.47 128 3.05 0.60
-0.44 124.79 0.66 -0.04 3 78 3.06 0.67 Total
S427 S427: The Academy made good use of new technologies for learning
-1.00 127.00 0.32 -0.10 1 50 3.10 0.61 129 3.16 0.57
-0.97 94.46 0.34 -0.10 3 79 3.20 0.54 Total
S428 S428: I never felt that anyone was putting limits on what I could achieve
-2.19 127.00 0.03 -0.24 1 50 2.92 0.63 129 3.07 0.63
-2.17 101.19 0.03 -0.24 3 79 3.16 0.61 Total
S429 S429: I feel proud about what I achieved at the Academy
-1.45 125.00 0.15 -0.16 1 49 2.94 0.63 127 3.04 0.62
-1.44 100.82 0.15 -0.16 3 78 3.10 0.62 Total
S430 S430: I could not have -2.29 126.00 0.02 -0.30 1 49 2.90 0.71 128 3.09 0.74
79
gone to a better place for my secondary education
-2.31 104.68 0.02 -0.30 3 79 3.20 0.74 Total N Mean Std. Deviat
ion
COMPARISON IN ITEM MEAN BETWEEN GENDERS
Individual independent t-test were carried out on each item to examine whether there were
statistically significant mean differences in individual items between genders. Regardless of
Academies, statistical results found no significant gender differences in all statements at α = 0.05
level. In other words, male and female student respondents provided similar responses to all
statements (see Table 42 below).
Table 42 Comparison between genders on each item
t-test for Equality of Means
(purple shaded = significance at α =
0.05)
t df Sig.
(2-
tailed)
Mean
Diff.
GENDER N Mean SD
S41 S41: I was valued and included during my time at the Academy
-0.60 102.00 0.55 -0.06 1 Male 52 3.17 0.51
-0.60 101.19 0.55 -0.06 2 Female 52 3.23 0.47
S42 S42: I felt part of a community that was focused on learning
0.70 102.00 0.49 0.08 1 Male 52 3.17 0.55
0.70 101.88 0.49 0.08 2 Female 52 3.10 0.57
S43 S43: I felt safe at the Academy 0.85 102.00 0.40 0.10 1 Male 52 3.37 0.53
0.85 98.80 0.40 0.10 2 Female 52 3.27 0.63
S44 S44: I enjoyed my time at the Academy 0.78 101.00 0.44 0.09 1 Male 52 3.29 0.50
0.78 90.69 0.44 0.09 2 Female 51 3.20 0.69
S45 S45: I did as well as I had hoped in my GCSEs
0.79 101.00 0.43 0.10 1 Male 52 3.08 0.55
0.79 96.45 0.43 0.10 2 Female 51 2.98 0.68
S46 S46: My teachers accurately predicted my GCSE results
1.64 101.00 0.10 0.21 1 Male 52 3.12 0.62
1.64 98.84 0.10 0.21 2 Female 51 2.90 0.70
S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work
-0.18 99.00 0.86 -0.03 1 Male 51 3.25 0.74
-0.18 97.36 0.86 -0.03 2 Female 50 3.28 0.64
S48 S48: I have been well prepared for the next stage of my life
1.06 98.00 0.29 0.12 1 Male 50 3.20 0.49
1.06 92.56 0.29 0.12 2 Female 50 3.08 0.63
S49 S49: My successes were recognised during my time at the Academy
2.00 98.00 0.05 0.21 1 Male 51 3.24 0.47
1.99 91.57 0.05 0.21 2 Female 49 3.02 0.59
S410 S410: I was helped and supported whenever I struggled with my learning
0.61 98.00 0.54 0.08 1 Male 50 3.18 0.60
0.61 95.25 0.54 0.08 2 Female 50 3.10 0.71
S411 S411: I developed a sense of responsibility towards other people during my time at the Academy
1.07 97.00 0.29 0.12 1 Male 50 3.22 0.55
1.07 96.93 0.29 0.12 2 Female 49 3.10 0.55
S412 S412: The Academy helped to make me more understanding about different cultures and types of people
-1.09 99.00 0.28 -0.14 1 Male 51 2.96 0.63
-1.09 98.81 0.28 -0.14 2 Female 50 3.10 0.65
S413 S413: I supported activities that would improve things for other people
1.05 99.00 0.29 0.12 1 Male 51 3.08 0.59
1.05 98.22 0.29 0.12 2 Female 50 2.96 0.53
S414 S414: I was offered opportunities for 0.73 99.00 0.47 0.10 1 Male 51 3.02 0.71
80
leadership during my time at the Academy 0.73 98.84 0.47 0.10 2 Female 50 2.92 0.67
S415 S415: I led activities during my time at the Academy
-1.05 99.00 0.30 -0.16 1 Male 51 2.76 0.79
-1.05 97.88 0.30 -0.16 2 Female 50 2.92 0.70
S416 S416: My teachers taught me well at the Academy
1.23 99.00 0.22 0.15 1 Male 51 3.37 0.60
1.23 98.04 0.22 0.15 2 Female 50 3.22 0.65
S417 S417: My teachers were always trying to improve their teaching
0.94 99.00 0.35 0.12 1 Male 51 3.18 0.62
0.94 98.98 0.35 0.12 2 Female 50 3.06 0.62
S418 S418: I was regularly challenged to do better so that I could reach my full potential
0.47 98.00 0.64 0.06 1 Male 50 3.12 0.66
0.47 97.63 0.64 0.06 2 Female 50 3.06 0.62
S419 S419: My parents/carers felt involved with my education at the Academy
0.17 96.00 0.87 0.02 1 Male 50 2.92 0.75
0.17 95.83 0.87 0.02 2 Female 48 2.90 0.69
S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
-0.27 98.00 0.79 -0.04 1 Male 50 3.08 0.70
-0.27 96.92 0.79 -0.04 2 Female 50 3.12 0.77
S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best
1.75 96.00 0.08 0.18 1 Male 49 3.35 0.52
1.75 95.98 0.08 0.18 2 Female 49 3.16 0.51
S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
-1.19 96.00 0.24 -0.15 1 Male 50 3.02 0.65
-1.19 94.72 0.24 -0.15 2 Female 48 3.17 0.56
S423 S423: The Academy was well run 0.48 95.00 0.63 0.06 1 Male 49 3.22 0.62
0.48 94.31 0.63 0.06 2 Female 48 3.17 0.56
S424 S424: There were great facilities for learning
1.04 95.00 0.30 0.12 1 Male 49 3.35 0.56
1.04 94.99 0.30 0.12 2 Female 48 3.23 0.56
S425 S425: There was a broad curriculum that helped me to develop as a whole person
-0.59 94.00 0.56 -0.07 1 Male 50 3.06 0.62
-0.59 93.77 0.55 -0.07 2 Female 46 3.13 0.54
S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
-0.04 95.00 0.97 -0.01 1 Male 50 3.08 0.67
-0.04 94.56 0.97 -0.01 2 Female 47 3.09 0.58
S427 S427: The Academy made good use of new technologies for learning
1.42 96.00 0.16 0.15 1 Male 50 3.32 0.59
1.42 93.45 0.16 0.15 2 Female 48 3.17 0.48
S428 S428: I never felt that anyone was putting limits on what I could achieve
0.28 96.00 0.78 0.04 1 Male 50 3.10 0.74
0.28 91.42 0.78 0.04 2 Female 48 3.06 0.56
S429 S429: I feel proud about what I achieved at the Academy
-0.23 94.00 0.82 -0.03 1 Male 49 3.10 0.55
-0.23 93.97 0.82 -0.03 2 Female 47 3.13 0.54
S430 S430: I could not have gone to a better place for my secondary education
1.75 95.00 0.08 0.26 1 Male 49 3.29 0.79
1.75 94.03 0.08 0.26 2 Female 48 3.02 0.70
COMPARISON IN ITEM MEAN BETWEEN STUDENTS WITH FREE SCHOOL MEALS AND NON-FREE
SCHOOL MEALS
An individual independent t-test was carried out on each item to examine whether there were
statistically significant mean differences in individual items between FSM and non FSM receivers.
However, not all Academy 1 student respondents provided FSM information. Statistical results
found no significant FSM/non FSM differences in all statements at α = 0.05 level. In other words,
FSM and non FSM student respondents in Academy 3 provided similar responses to all statements
(see the Table 43 below).
Table 43 Comparison between students with and without Free School Meals on each item
t df Sig. (2-
tailed)
Mean
Diff.
FSM N Mean SD
S41 S41: I was valued and included during my time at the Academy
0.87 74.00 0.39 0.11 1 FSM 26 3.27 0.60
0.81 41.00 0.43 0.11 2 No FSM 50 3.16 30.47
S42 S42: I felt part of a community that was -1.28 74.00 0.21 -0.18 1 FSM 26 3.04 0.77
81
focused on learning -1.10 34.64 0.28 -0.18 2 No FSM 50 3.22 0.46
S43 S43: I felt safe at the Academy 0.99 74.00 0.33 0.14 1 FSM 26 3.42 0.58
1.01 53.05 0.32 0.14 2 No FSM 50 3.28 0.61
S44 S44: I enjoyed my time at the Academy -1.60 74.00 0.11 -0.23 1 FSM 26 3.15 0.73
-1.42 37.01 0.16 -0.23 2 No FSM 50 3.38 0.49
S45 S45: I did as well as I had hoped in my GCSEs
-0.17 74.00 0.86 -0.02 1 FSM 26 3.08 0.56
-0.17 49.46 0.86 -0.02 2 No FSM 50 3.10 0.54
S46 S46: My teachers accurately predicted my GCSE results
-0.14 74.00 0.89 -0.02 1 FSM 26 3.04 0.77
-0.13 38.51 0.90 -0.02 2 No FSM 50 3.06 0.55
S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work
0.09 72.00 0.93 0.02 1 FSM 26 3.31 0.84
0.09 40.04 0.93 0.02 2 No FSM 48 3.29 0.62
S48 S48: I have been well prepared for the next stage of my life
-0.39 72.00 0.70 -0.05 1 FSM 26 3.15 0.67
-0.36 40.40 0.72 -0.05 2 No FSM 48 3.21 0.50
S49 S49: My successes were recognised during my time at the Academy
1.16 71.00 0.25 0.16 1 FSM 26 3.27 0.72
1.03 37.27 0.31 0.16 2 No FSM 47 3.11 0.48
S410 S410: I was helped and supported whenever I struggled with my learning
-0.33 72.00 0.74 -0.05 1 FSM 26 3.12 0.77
-0.30 39.74 0.77 -0.05 2 No FSM 48 3.17 0.56
S411 S411: I developed a sense of responsibility towards other people during my time at the Academy
-0.35 70.00 0.73 -0.05 1 FSM 25 3.12 0.73
-0.31 35.51 0.76 -0.05 2 No FSM 47 3.17 0.48
S412 S412: The Academy helped to make me more understanding about different cultures and types of people
-0.38 72.00 0.70 -0.06 1 FSM 26 3.00 0.69
-0.38 49.64 0.71 -0.06 2 No FSM 48 3.06 0.67
S413 S413: I supported activities that would improve things for other people
0.27 72.00 0.79 0.04 1 FSM 26 3.04 0.77
0.23 34.87 0.82 0.04 2 No FSM 48 3.00 0.46
S414 S414: I was offered opportunities for leadership during my time at the Academy
-0.21 72.00 0.83 -0.04 1 FSM 26 2.96 0.87
-0.20 40.51 0.84 -0.04 2 No FSM 48 3.00 0.65
S415 S415: I led activities during my time at the Academy
-0.27 72.00 0.79 -0.05 1 FSM 26 2.85 0.78
-0.26 49.47 0.79 -0.05 2 No FSM 48 2.90 0.75
S416 S416: My teachers taught me well at the Academy
0.45 72.00 0.65 0.08 1 FSM 26 3.35 0.80
0.42 41.16 0.68 0.08 2 No FSM 48 3.27 0.61
S417 S417: My teachers were always trying to improve their teaching
0.97 72.00 0.34 0.15 1 FSM 26 3.23 0.71
0.91 43.15 0.37 0.15 2 No FSM 48 3.08 0.58
S418 S418: I was regularly challenged to do better so that I could reach my full potential
-0.02 71.00 0.98 0.00 1 FSM 25 3.08 0.81
-0.02 36.98 0.99 0.00 2 No FSM 48 3.08 0.58
S419 S419: My parents/carers felt involved with my education at the Academy
0.93 69.00 0.36 0.17 1 FSM 24 3.00 0.88
0.84 35.53 0.41 0.17 2 No FSM 47 2.83 0.64
S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
1.07 71.00 0.29 0.20 1 FSM 25 3.16 0.90
0.98 38.83 0.33 0.20 2 No FSM 48 2.96 0.68
S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best
1.37 69.00 0.17 0.18 1 FSM 25 3.44 0.65
1.23 36.44 0.23 0.18 2 No FSM 46 3.26 0.44
S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
1.94 69.00 0.06 0.31 1 FSM 24 3.33 0.64
1.95 46.78 0.06 0.31 2 No FSM 47 3.02 0.64
S423 S423: The Academy was well run 1.46 68.00 0.15 0.22 1 FSM 23 3.39 0.58
1.47 45.04 0.15 0.22 2 No FSM 47 3.17 0.60
S424 S424: There were great facilities for learning
0.19 68.00 0.85 0.03 1 FSM 24 3.33 0.70
0.18 38.23 0.86 0.03 2 No FSM 46 3.30 0.55
S425 S425: There was a broad curriculum that helped me to develop as a whole person
-0.99 67.00 0.33 -0.16 1 FSM 24 3.00 0.72
-0.92 38.18 0.36 -0.16 2 No FSM 45 3.16 0.56
S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
-0.26 68.00 0.80 -0.05 1 FSM 24 3.04 0.69
-0.26 46.96 0.80 -0.05 2 No FSM 46 3.09 0.69
S427 S427: The Academy made good use of new technologies for learning
0.26 69.00 0.80 0.04 1 FSM 24 3.25 0.68
0.24 36.67 0.81 0.04 2 No FSM 47 3.21 0.51
S428 S428: I never felt that anyone was putting limits on what I could achieve
0.11 69.00 0.91 0.02 1 FSM 24 3.17 0.64
0.11 45.64 0.91 0.02 2 No FSM 47 3.15 0.62
S429 S429: I feel proud about what I achieved 1.57 68.00 0.12 0.22 1 FSM 23 3.30 0.56
82
at the Academy 1.55 42.81 0.13 0.22 2 No FSM 47 3.09 0.54
S430 S430: I could not have gone to a better place for my secondary education
0.62 69.00 0.54 0.12 1 FSM 24 3.29 0.75
0.63 48.55 0.53 0.12 2 No FSM 47 3.17 0.79
PRELIMINARY DATA REDUCTION: EXPLORING UNDERLYING THEMES IN THE WAYS STUDENTS
ANSWER THE QUESTIONS
An exploratory factor analysis was computed to identify any underlying themes in the ways in
which students perceived their time at the Academies. Four themes (see Table 45) were identified
and four items were deleted from the process (Deleted Items: S45, S46, S49, S424).
Table 44 Exploratory Factor Analysis showing item loadings and reliability co-efficients
Component
1 2 3 4 α
S422 S422: Everyone at the Academy worked very hard to make it a great place to learn and to do well
.757 .354 .212 .164
S428 S428: I never felt that anyone was putting limits on what I could achieve
.741 .186 .197 -.228
S410 S410: I was helped and supported whenever I struggled with my learning
.716 .223 .072 .213
S417 S417: My teachers were always trying to improve their teaching
.712 .215 .248 .107
S43 S43: I felt safe at the Academy .665 .199 .099 .294
S423 S423: The Academy was well run .610 .410 .223 .016
S419 S419: My parents/carers felt involved with my education at the Academy
.561 .128 .459 .125
S429 S429: I feel proud about what I achieved at the Academy
.557 .365 .181 .396
S44 S44: I enjoyed my time at the Academy .500 .362 .162 .252
S412 S412: The Academy helped to make me more understanding about different cultures and types of people
.469 .275 .245 .364
S42 S42: I felt part of a community that was focused on learning
.459 .386 .207 .228 0.90
S430 S430: I could not have gone to a better place for my secondary education
.267 .767 .031 .066
S427 S427: The Academy made good use of new technologies for learning
.158 .757 .114 .085
S418 S418: I was regularly challenged to do better so that I could reach my full potential
.215 .645 .239 .174
S411 S411: I developed a sense of responsibility towards other people during my time at the Academy
.191 .643 .364 .286
S420 S420: My parents felt that they could find the right person to talk to at the Academy if they had any concerns about my education
.382 .613 -.019 .264
S425 S425: There was a broad curriculum that helped me to develop as a whole person
.262 .604 .472 .134
S41 S41: I was valued and included during my time at the Academy
.411 .592 .105 .211
S421 S421: Senior staff at the Academy wanted us to learn well and achieve our best
.461 .491 .110 .027 0.88
S414 S414: I was offered opportunities for leadership during my time at the Academy
.320 .096 .753 .085
83
S415 S415: I led activities during my time at the Academy .164 .044 .733 .045
S413 S413: I supported activities that would improve things for other people
.016 .213 .704 .382
S426 S426: I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
.332 .458 .610 -.008 0.76
S48 S48: I have been well prepared for the next stage of my life
.055 .042 .254 .710
S47 S47: I am looking forward to the move to Sixth Form, Further Education, training or work
.150 .223 .028 .673
S416 S416: My teachers taught me well at the Academy .401 .459 .014 .518 0.66
Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 7 iterations.
UNDERLYING THEMES IN THE DATA
These four underlying themes (see Table 45) demonstrate how the year eleven leavers perceived
their time at the Academies. It does not necessarily demonstrate the degree to which students
experienced this - for example they could have been not at all proud, or seen their Academy as not
a learning community. These could be computed into new variables for greater economy of data
and statistical power. They are theoretically consistent with the Oasis Charter.
Table 45 Themes from factor analysis
Theme One I was proud to be part of a positive learning community
Theme Two I was continually encouraged to do my best for myself and others
Theme Three I was offered the chance to lead and serve others
Theme Four I am positive about the next stage
SUMMARY
This Strand sought to explore the experiences of leavers from two of the Academies and their
expectations of the transition to the next stage of their education.
When comparing Academies, there were no statistical differences between the two Academies in
24 out of 30 items indicating the student respondents had similar views on these statements.
However, Academy 1 had lower means in the remaining six items than Academy 3, where on
average, Academy 3 respondents tended to report a greater degree of agreement on enjoying their
time at the Academy, suggesting that Academy 3 students overall rating their Academy more highly
than Academy 1 students, although this was just on 6 of the 30 items.
There were no statistically significant gender differences in any of the Academies on how students
reported their experiences of the Academy or their expectations of the transition. There were also
no statistically significant differences between students receiving FSM compared to those with no
FSM, suggesting that neither gender nor socioeconomic background was a particular factor
impacting on students’ experiences of their Academies.
84
STRAND FIVE - QUALITY OF RELATIONSHIPS BETWEEN PARENTS/CARERS AND THE
ACADEMY
This Strand was designed to explore how well parents and carers understand and experience the
ECHO charter’s values in practice.
QUESTIONNAIRE
The parents/carers individual questionnaire was structured in two parts. Part I, was general
questions, in addition to one open-ended question, with 24 items on a four-point Likert-scale
labeled, strongly agree, agree, disagree and strongly disagree, which were coded as 4, 3, 2, and 1
respectively. Similarly, Part II consisted of one open-ended question and another set of six items
using the same Likert-scale and the same coding system as employed in Part I, and was for
parents/carers whose first language was not English to fill in. The total number of parent/carer
respondents was 72 from Academy 1.
FINDINGS
The following Table 46 shows the number of respondents, and the percentage for each value on
each question. It also shows the number and percentage of missing values.
Table 46 Strand Five Individual Item Values
1 Strongly
Disagree
2 Disagree 3 Agree 4 Strongly
Agree
0 missing
Item N % N % N % N % N %
Part I
S51: My child is known and valued as an individual
1 1.4 1 1.4 41 56.9 29 40.3 0 .0
S52: The Academy is providing for my child's particular needs
4 5.6 1 1.4 47 65.3 20 27.8 0 .0
S53: My child feels included in the life of the Academy, in classrooms and in other activities
4 5.6 1 1.4 39 54.2 28 38.9 0 .0
S54: The Academy focuses its work on providing the best opportunities for learning for all students
8 11.1 0 .0 37 51.4 27 37.5 0 .0
S55: My child is taught well 6 8.3 2 2.8 41 56.9 23 31.9 0 .0
S56: My child is being challenged to improve on previous best performance in all areas
5 6.9 1 1.4 40 55.6 24 33.3 2 2.8
S57: My child's progress at the Academy has exceeded our expectations
16 22.2 6 8.3 35 48.6 14 19.4 1 1.4
S58: My child's successes are recognised
9 12.5 3 4.2 33 45.8 26 36.1 1 1.4
85
S59: The Academy ensures my child is well looked after
8 11.1 3 4.2 39 54.2 21 29.2 1 1.4
S510: The Academy helps my child to have a healthy lifestyle
8 11.1 2 2.8 54 75.0 8 11.1 0 .0
S511: There is a good standard of behaviour at the Academy
12 16.7 6 8.3 39 54.2 14 19.4 1 1.4
S512: My child is developing an attitude of helping others
19 26.4 2 2.8 40 55.6 8 11.1 3 4.2
S513: My child is taking opportunities to lead activities
22 30.6 0 .0 36 50.0 12 16.7 2 2.8
S514: My child has kept up a good rate of progress at key transitions
7 9.7 1 1.4 43 59.7 20 27.8 1 1.4
gS515: I get regular and helpful information about my child's progress
14 19.4 3 4.2 31 43.1 24 33.3 0 .0
S516: The Academy responds well to my concerns
5 6.9 6 8.3 37 51.4 22 30.6 2 2.8
S517: I always find staff knowledgeable about my child
15 20.8 2 2.8 29 40.3 26 36.1 0 .0
S518: I always find staff helpful when discussing my child's progress
5 6.9 3 4.2 33 45.8 29 40.3 2 2.8
S519: The Academy helps me to support my child's learning
8 11.1 3 4.2 42 58.3 17 23.6 2 2.8
S520: The Academy is led and managed effectively
5 6.9 1 1.4 42 58.3 21 29.2 3 4.2
S521: My impression is that senior staff focus their work on making the Academy a good place to learn
5 6.9 2 2.8 38 52.8 24 33.3 3 4.2
S522: My impression is that senior staff expect all students to achieve well
4 5.6 1 1.4 38 52.8 28 38.9 1 1.4
S523: My child enjoys their education at the Academy
7 9.7 3 4.2 35 48.6 26 36.1 1 1.4
S524: I would recommend the Academy to another Parent/Carer
7 9.7 3 4.2 30 41.7 30 41.7 2 2.8
Part II
S526: The Academy has been helpful in ensuring that I understand the general information sent to parents/carers
0 .0 0 .0 48 66.7 3 4.2 21 29.2
S527: The Academy has been helpful in ensuring that I understand the information about my child's progress
1 1.4 0 .0 6 8.3 6 8.3 59 81.9
S528: I feel that the Academy understand and takes proper account of the particular issues and sensitivities of my culture
1 1.4 0 .0 4 5.6 5 6.9 62 86.1
S529: If I have a concern I feel confident that I can make contact with relevant staff
1 1.4 0 .0 5 6.9 3 4.2 63 87.5
S530: If I have a concern I feel confident that I can communicate with staff effectively
1 1.4 0 .0 3 4.2 5 6.9 63 87.5
86
S531: I would recommend the Academy to another parent/carer from my community
0 .0 0 .0 4 5.6 4 5.6 64 88.9
PARENT/CARER PARTICIPANTS’ TENDENCY TO INDIVIDUAL ITEMS IN PART I
This questionnaire was conducted in one Academy. No information about gender or whether or not
the parent/carer was a native English speaker was provided. Individual item descriptive means,
calculated based on numeric codes of the responses, were used to examine parent/carer
participants’ tendency to disagree or agree with individual statement items in Part I.
Overall, there were one-third of items with a mean below 3.00 indicating descriptively parent/carer
participants’ general disagreement on these items (see Table 39 below). Among those, 'the five
most disagreed with' statements by the respondents were that parents/carers in general did not
think that their child ‘is developing an attitude of helping others’ (S512), ‘is taking opportunities to
lead activities’ (S513), the progress of their child ‘has exceeded’ their expectations (S57), the
Academy having ‘a good standard of behaviour’ (S511) and the Academy helping them to ‘have a
healthy lifestyle’ (S510). Similar findings were also found in terms of frequency of each of the four-
point Likert scale for individual items. As shown in Table 40, 22.5 to 31.4 percentages of
parent/carer respondents strongly disagreed that their child ‘is developing an attitude of helping
others’ (S512), ‘is taking opportunities to lead activities’ (S513), and the progress of their child ‘has
exceeded’ their expectations (S57). Nevertheless, the findings also indicated that the three
statements were among the five items with least consensus between the parents/carers’ responses
(i.e. highest SD (standard deviation), see Table 47).
In contrast to the findings stated above, the five items which parent/carer respondents reported
highest means (above 3.00) were the agreement on their child ‘is known and valued as an
individual’ (S51), senior staff expecting ‘all students to achieve well’ (S522), their child ‘feels
included’ in the school (S53), finding ‘staff helpful’ in discussing their child’s progress (S518), and
‘would recommend the Academy to another parent/carer’ (S524). The five items also received
relatively greatest percentage in ‘strongly agree’ response (40.3%, 39.4%, 38.9%, 41.4% and 42.9%
respectively, see Table 48). Also the respondents reported most consensuses in agreement on the
first three statements in terms of least SD (see Table 48).
Interestingly, all findings mentioned above seemingly indicate that parents/carers think the
Academy is doing its work well in terms of paying attention to individual students, having high
expectation of all students, and being helpful in discussing about their child’s progress. However,
somehow, the work done by the Academy did not lead to students’ development in helping people,
making progress of expectation (see also Table 48 for at least 91% of the respondents agree or
strongly agree on ‘My child is being challenged to improve on previous best performance in all
areas’ (S56)), good behaviour and healthy lifestyle suggesting an issue of effectiveness.
Nevertheless, the majority of the respondents would recommend the Academy to another
parent/carer.
87
Table 47 below, shows the mean and standard deviation for each question, ranked from the lowest
to the highest.
Table 47 Parent/carer participants’ responses to Part I individual items listed by mean in ascending order
Item N Min Max Mean SD
S512: My child is developing an attitude of helping others 69 1 4 2.54 1.02
S513: My child is taking opportunities to lead activities 70 1 4 2.54 1.11
S57 : My child's progress at the Academy has exceeded our expectations 71 1 4 2.66 1.04
S511 1: There is a good standard of behaviour at the Academy 71 1 4 2.77 0.96
S510: The Academy helps my child to have a healthy lifestyle 72 1 4 2.86 0.76
gS515: I get regular and helpful information about my child's progress 72 1 4 2.90 1.08
S517: I always find staff knowledgeable about my child 72 1 4 2.92 1.11
S519: The Academy helps me to support my child's learning 70 1 4 2.97 0.87
S59: The Academy ensures my child is well looked after 71 1 4 3.03 0.89
S514: My child has kept up a good rate of progress at key transitions 71 1 4 3.07 0.83
S58: My child's successes are recognised 71 1 4 3.07 0.96
S516: The Academy responds well to my concerns 70 1 4 3.09 0.83
S55: My child is taught well 72 1 4 3.13 0.82
S523 : My child enjoys their education at the Academy 71 1 4 3.13 0.89
S520: The Academy is led and managed effectively 69 1 4 3.14 0.77
S52: The Academy is providing for my child's particular needs 72 1 4 3.15 0.71
S54: The Academy focuses its work on providing the best opportunities for learning for all students
72 1 4 3.15 0.90
S521: My impression is that senior staff focus their work on making the Academy a good place to learn
69 1 4 3.17 0.80
S56: My child is being challenged to improve on previous best performance in all areas
70 1 4 3.19 0.79
S524: I would recommend the Academy to another Parent/Carer 70 1 4 3.19 0.92
S518: I always find staff helpful when discussing my child's progress 70 1 4 3.23 0.84
S53: My child feels included in the life of the Academy, in classrooms and in other activities
72 1 4 3.26 0.75
S522: My impression is that senior staff expect all students to achieve well 71 1 4 3.27 0.76
S51: My child is known and valued as an individual 72 1 4 3.36 0.59
The following table presents the frequency of each score as a number and a percentage for each
item in Part I of the questionnaire.
Table 48 Frequency of the parent/carer participants to Part I individual items
1 Strongly
Disagree
2 Disagree 3 Agree 4 Strongly
Agree
N % N % N % N %
S510: The Academy helps my child to have a healthy lifestyle
8 11.1 2 2.8 54 75.0 8 11.1
S512: My child is developing an attitude of helping others
19 27.5 2 2.9 40 58.0 8 11.6
S513: My child is taking opportunities to lead activities 22 31.4 0 .0 36 51.4 12 17.1
88
S511: There is a good standard of behaviour at the Academy
12 16.9 6 8.5 39 54.9 14 19.7
S57: My child's progress at the Academy has exceeded our expectations
16 22.5 6 8.5 35 49.3 14 19.7
S519 : The Academy helps me to support my child's learning
8 11.4 3 4.3 42 60.0 17 24.3
S52: The Academy is providing for my child's particular needs
4 5.6 1 1.4 47 65.3 20 27.8
S514: My child has kept up a good rate of progress at key transitions
7 9.9 1 1.4 43 60.6 20 28.2
S59: The Academy ensures my child is well looked after 8 11.3 3 4.2 39 54.9 21 29.6
S520: The Academy is led and managed effectively 5 7.2 1 1.4 42 60.9 21 30.4
S516 : The Academy responds well to my concerns 5 7.1 6 8.6 37 52.9 22 31.4
S55: My child is taught well 6 8.3 2 2.8 41 56.9 23 31.9
gS515: I get regular and helpful information about my child's progress
14 19.4 3 4.2 31 43.1 24 33.3
S56: My child is being challenged to improve on previous best performance in all areas
5 7.1 1 1.4 40 57.1 24 34.3
S521: My impression is that senior staff focus their work on making the Academy a good place to learn
5 7.2 2 2.9 38 55.1 24 34.8
S517: I always find staff knowledgeable about my child 15 20.8 2 2.8 29 40.3 26 36.1
S523: My child enjoys their education at the Academy 7 9.9 3 4.2 35 49.3 26 36.6
S58: My child's successes are recognised 9 12.7 3 4.2 33 46.5 26 36.6
S54: The Academy focuses its work on providing the best opportunities for learning for all students
8 11.1 0 .0 37 51.4 27 37.5
S53: My child feels included in the life of the Academy, in classrooms and in other activities
4 5.6 1 1.4 39 54.2 28 38.9
S522: My impression is that senior staff expect all students to achieve well
4 5.6 1 1.4 38 53.5 28 39.4
S51: My child is known and valued as an individual 1 1.4 1 1.4 41 56.9 29 40.3
S518 : I always find staff helpful when discussing my child's progress
5 7.1 3 4.3 33 47.1 29 41.4
S524 : I would recommend the Academy to another Parent/Carer
7 10.0 3 4.3 30 42.9 30 42.9
EXPLORING PATTERNS IN PARENT/CARER PARTICIPANTS’ RESPONSES
An exploratory factor analysis was computed in order to explore whether there were any
underlying themes in the data. The factors, items and alpha reliability co-efficient of the resulting
scale is presented in Table 49 below. One item was deleted: S524.
Table 49 Rotated Component Matrix
Component α
Items 1 2 3 4
S522: My impression is that senior staff expect all students to achieve well
.844 .044 .214 .196
S521: My impression is that senior staff focus their work on .751 .220 .379 .086
89
making the Academy a good place to learn
S518: I always find staff helpful when discussing my child's progress
.724 .445 .101 .115
S54: The Academy focuses its work on providing the best opportunities for learning for all students
.706 .307 .186 .316
S56: My child is being challenged to improve on previous best performance in all areas
.693 .059 .393 .201
S55: My child is taught well .638 .490 .169 .139
S517: I always find staff knowledgeable about my child .635 .466 .112 .349
S510: The Academy helps my child to have a healthy lifestyle .587 .234 .228 .408
gS515: I get regular and helpful information about my child's progress
.466 .349 .320 .444
S519: The Academy helps me to support my child's learning .437 .375 .355 .209 0.93
S51: My child is known and valued as an individual .211 .773 .308 .133
S520: The Academy is led and managed effectively .510 .617 .253 .061
S512: My child is developing an attitude of helping others .198 .608 .117 .270
S53: My child feels included in the life of the Academy, in classrooms and in other activities
.225 .571 .510 .264
S523: My child enjoys their education at the Academy .464 .470 .398 .362 0.85
S511: There is a good standard of behaviour at the Academy .285 .108 .702 .094
S59 : The Academy ensures my child is well looked after .201 .234 .658 .430
S52: The Academy is providing for my child's particular needs .197 .511 .655 .231
S58: My child's successes are recognised .540 .425 .541 -.099 0.82
S514: My child has kept up a good rate of progress at key transitions
.247 .131 .314 .772
S513: My child is taking opportunities to lead activities -.064 .482 -.054 .710
S57: My child's progress at the Academy has exceeded our expectations
.430 -.025 .238 .657
S516: The Academy responds well to my concerns .483 .361 .085 .510 0.82
Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.Rotation converged in 13 iterations.
UNDERLYING THEMES IN THE DATA
There were four themes that emerged from the parents and carers data (see Table 50). These
suggest that the ways in which parents understood their relationship with the Academy were
similar conceptually. The themes could be computed into new variables for greater statistical
power and economy of data.
Table 50 Themes from factor analysis for Strand 5
Theme One The Academy provides an environment that challenges my child and
enables me to help
Theme Two My child enjoys learning as a valued individual in a well run, inclusive
90
Academy
Theme Three The Academy ensures that my child is valued as an individual
Theme Four My child is making good progress in learning and leadership in a responsive
Academy
SUMMARY AND FINDINGS FROM STRAND FIVE
This Strand explored how well parents and carers understood and experienced the ECHO charter’s
values in practice.
The findings showed that a third of the statements within the 30 item 2-part questionnaire were
generally disagreed with. Many of these statements seemed to refer to issues of developing
attitudes to help others or to take opportunities to lead activities or referred to behaviour or a
healthy lifestyle. However, these findings also showed these same statements to be among those
in which the parents/carers had least consensus. This suggests that whilst there are general
disagreements about certain aspects of how parents/carers experienced the Oasis Charter, these
disagreements do not all concord with one another and could reflect the diversity of experiences
due to the parents’ expectations or perceptions. These could be affected by other factors such as
culture or language, however data on English as an additional language was missing so such
assumptions could be problematic.
The underlying themes in the data suggest key ways in which parents approach the Academy, in a
generally positive, inclusive way, where they feel their child is valued and challenged in learning.
STRAND SIX - LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF
CLASSROOM PRACTICE
Strand Six was designed to explore teacher’s perceptions of their Academy and the impact of their
professional learning on classroom practice.
QUESTIONNAIRE
The questionnaire was administered in one Academy and received responses from 10 teacher
participants. The questionnaire was in three parts, with questions answered on a four-point Likert-
scale: 28 (Part A), 26 (Part B), 12 (Part C). The following Tables 51, 52 and 53 show the distribution
of scores for each question, as a number and a percentage, as well as missing values.
Table 51 Strand Five Part A Item Values
91
Part A 1 2 3 4 Missing
Item N % N % N % N % N %
S6A1 S6 A1: Staff as well as students learn in this Academy
2 20.0 1 10.0 4 40.0 3 30.0 0 0.0
S6A2 S6 A2: I draw on good practice from other schools as a means to further my own professional development
4 40.0 4 40.0 2 20.0 0 0.0 0 0.0
S6A3 S6 A3: I read research reports as one source of useful ideas for improving my practice
1 10.0 7 70.0 2 20.0 0 0.0 0 0.0
S6A4 S6 A4: I use the web as one source of useful ideas for improving my practice
0 0.0 3 30.0 5 50.0 2 20.0 0 0.0
S6A5 S6 A5: I consult my students about how they learn most effectively
0 0.0 5 50.0 2 20.0 3 30.0 0 0.0
S6A6 S6 A6: I relate what works in my own practice to research findings
0 0.0 6 60.0 3 30.0 1 10.0 0 0.0
S6A7 S6 A7: I am able to see how practices that work in one context might be adapted to other contexts
0 0.0 2 20.0 5 50.0 3 30.0 0 0.0
S6A8 S6 A8: I use insights from my own professional learning to feed into Academy policy development
3 30.0 3 30.0 0 0.0 3 30.0 1 10.0
S6A9 S6 A9: I reflect on my practice as a way of identifying my professional learning needs
0 0.0 0 0.0 7 70.0 3 30.0 0 0.0
S6A10 S6 A10: I experiment with my practice as a conscious strategy for improving classroom teaching and learning
0 0.0 0 0.0 4 40.0 6 60.0 0 0.0
S6A11 S6 A11: I modify my practice in the light of feedback from my students
0 0.0 0 0.0 7 70.0 3 30.0 0 0.0
S6A12 S6 A12: I modify my practice in the light of published research evidence
3 30.0 3 30.0 4 40.0 0 0.0 0 0.0
S6A13 S6 A13: I modify my practice in the light of evidence from self-evaluations of my classroom practice
0 0.0 0 0.0 7 70.0 3 30.0 0 0.0
S6A14 S6 A14: I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues
0 0.0 0 0.0 4 40.0 5 50.0 1 10.0
S6A15 S6 A15: I carry out joint research/evaluation with one or more colleagues as a way of improving my practice
5 50.0 1 10.0 3 30.0 0 0.0 1 10.0
S6A16 S6 A16: I collaborate to plan my teaching
0 0.0 4 40.0 3 30.0 3 30.0 0 0.0
S6A17 S6 A17: I observe other colleagues in the classroom and we give each other feedback
0 0.0 5 50.0 3 30.0 2 20.0 0 0.0
S6A18 S6 A18: I engage in team teaching as a way of improving practice
3 30.0 4 40.0 1 10.0 2 20.0 0 0.0
S6A19 S6 A19: If I have a problem with my teaching I turn to colleagues for help
0 0.0 0 0.0 8 80.0 2 20.0 0 0.0
S6A20 S6 A20: I suggest ideas or approaches for colleagues to try in class
2 20.0 0 0.0 6 60.0 2 20.0 0 0.0
S6A21 S6 A21: I make collective agreements to test out new ideas
0 0.0 5 50.0 2 20.0 3 30.0 0 0.0
S6A22 S6 A22: I discuss openly with colleagues what and how they are learning
0 0.0 1 10.0 4 40.0 4 40.0 1 10.0
S6A23 S6 A23: I use informal opportunities to discuss how children learn
0 0.0 5 50.0 2 20.0 3 30.0 0 0.0
92
S6A24 S6 A24: I offer reassurance and support to colleagues
0 0.0 2 20.0 2 20.0 6 60.0 0 0.0
S6A25 S6 A25: My practice is based on the belief that all students are capable of learning
0 0.0 0 0.0 2 20.0 8 80.0 0 0.0
S6A26 S6 A26: My experience is that students in this Academy enjoy learning
0 0.0 3 30.0 5 50.0 1 10.0 1 10.0
S6A27 S6 A27: My experience is that student success is celebrated
0 0.0 0 0.0 5 50.0 2 20.0 3 30.0
S6A28 S6 A28: I discuss with colleagues how students might be helped to learn how to learn
0 0.0 0 0.0 6 60.0 3 30.0 1 10.0
Table 52 Strand 6 Part B Item values
Part B 1 2 3 4 Missing
Item N % N % N % N % N %
S6B1 S6 B1: Senior leaders communicate a clear vision of where the Academy is going
3 30.0 2 20.0 2 20.0 3 30.0 0 0.0
S6B2 S6 B2: Staff have a commitment to the whole Academy as well as to their department, key stage and/or year group
0 0.0 0 0.0 4 40.0 6 60.0 0 0.0
S6B3 S6 B3: Senior leaders promote commitment among staff to the whole Academy as well as to the department, key stage and/or year group
0 0.0 2 20.0 5 50.0 3 30.0 0 0.0
S6B4 S6 B4: There is effective communication between senior leaders and teachers
0 0.0 8 80.0 1 10.0 0 0.0 1 10.0
S6B5 S6 B5: There are processes for involving all staff in decision-making
2 20.0 5 50.0 1 10.0 1 10.0 1 10.0
S6B6 S6 B6: Teachers’ professional know-how is used in the formulation of Academy policy and goals
0 0.0 7 70.0 2 20.0 0 0.0 1 10.0
S6B7 S6 B7: Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices
0 0.0 5 50.0 3 30.0 2 20.0 0 0.0
S6B8 S6 B8: Opportunities are provided for teachers to critically evaluate Academy policy
2 20.0 5 50.0 1 10.0 1 10.0 1 10.0
S6B9 S6 B9: Staff are involved in evaluating Academy policy
3 30.0 4 40.0 0 0.0 3 30.0 0 0.0
S6B10 S6 B10: Staff participate in important decision-making
4 40.0 3 30.0 1 10.0 1 10.0 1 10.0
S6B11 S6 B11: There are processes for involving students in decision-making
0 0.0 8 80.0 1 10.0 0 0.0 1 10.0
S6B12 S6 B12: Staff have a good working knowledge of the Academy Development Plan (or your Academy’s equivalent document)
3 30.0 2 20.0 4 40.0 1 10.0 0 0.0
S6B13 S6 B13: Staff see the Academy Development Plan as relevant and useful to learning and teaching
0 0.0 3 30.0 5 50.0 2 20.0 0 0.0
S6B14 S6 B14: Staff development time is used effectively to realise Academy Development Plan priorities
0 0.0 5 50.0 4 40.0 1 10.0 0 0.0
S6B15 S6 B15: Staff development time is used effectively in the Academy
0 0.0 5 50.0 5 50.0 0 0.0 0 0.0
93
S6B16 S6 B16: The Academy provides cover to allow staff joint planning time
9 90.0 1 10.0 0 0.0 0 0.0 0 0.0
S6B17 S6 B17: Teachers are encouraged to experiment with new ideas as a way of promoting professional growth
0 0.0 0 0.0 5 50.0 5 50.0 0 0.0
S6B18 S6 B18: Formal training provides opportunities for teachers to develop professionally
0 0.0 2 20.0 7 70.0 1 10.0 0 0.0
S6B19 S6 B19: Teachers are helped to develop skills to assess students’ work in ways that move students on in their learning
0 0.0 3 30.0 5 50.0 2 20.0 0 0.0
S6B20 S6 B20: Teachers are helped to develop skills to observe learning as it happens in the classroom
0 0.0 3 30.0 6 60.0 0 0.0 1 10.0
S6B21 S6 B21: Senior leaders support teachers in sharing practice with other schools through networking
3 30.0 2 20.0 5 50.0 0 0.0 0 0.0
S6B22 S6 B22: Information is collected from teachers on those aspects of their work that they themselves think they do effectively
2 20.0 5 50.0 3 30.0 0 0.0 0 0.0
S6B23 S6 B23: Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students
2 20.0 0 0.0 6 60.0 2 20.0 0 0.0
S6B24 S6 B24: Information is collected from teachers on informal teacher networking in which they play an active role
5 50.0 5 50.0 0 0.0 0 0.0 0 0.0
S6B25 S6 B25: Teacher-initiated networking is an integral element of staff development
5 50.0 5 50.0 0 0.0 0 0.0 0 0.0
S6B26 S6 B26: Learning how to learn is an issue discussed in staff development time
2 20.0 4 40.0 3 30.0 0 0.0 1 10.0
Table 53 Strand 6 Part C Item Values
Part C 1 2 3 4 missing
Item N % N % N % N % N %
S6C1 S6 C1: I feel included in a community that values what I can offer to the staff team and to students
0 0.0 2 20.0 4 40.0 4 40.0 0 0.0
S6C2 S6 C2: I believe that I am working in a community that focuses our attention and activities on learning to improve teaching and student outcomes
0 0.0 3 30.0 5 50.0 2 20.0 0 0.0
S6C3 S6 C3: My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes
0 0.0 5 50.0 4 40.0 1 10.0 0 0.0
S6C4 S6 C4: I have had opportunities to work alongside and/or train with colleagues in other phases to improve students’ progress
3 30.0 0 0.0 0 0.0 7 70.0 0 0.0
S6C5 S6 C5: My experience is that senior leaders are role models as ‘leaders of learning’ in the Academy
0 0.0 5 50.0 0 0.0 5 50.0 0 0.0
S6C6 S6 C6: Leadership is shared in this Academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience
0 0.0 1 10.0 8 80.0 0 0.0 1 10.0
94
S6C7 S6 C7: My experience is that the Academy promotes a balance between the rights and the responsibilities of all members of the community
0 0.0 6 60.0 4 40.0 0 0.0 0 0.0
S6C8 S6 C8: There is an emphasis among students and staff on caring about and helping other people in the Academy community
0 0.0 6 60.0 1 10.0 3 30.0 0 0.0
S6C9 S6 C9: There is an emphasis among students and staff on caring about and helping other people in the wider community
0 0.0 5 50.0 2 20.0 3 30.0 0 0.0
S6C10 S6 C10: The Academy is successful in helping parents/carers to support the learning of their children
0 0.0 7 70.0 3 30.0 0 0.0 0 0.0
S6C11 S6 C11: The achievements of staff are recognised and celebrated
0 0.0 4 40.0 5 50.0 0 0.0 1 10.0
S6C12 S6 C12: I believe that I am realising my full potential as a teacher
0 0.0 0 0.0 5 50.0 5 50.0 0 0.0
PART A – ITEM MEAN AND STANDARD DEVIATIONS
There were 16 out of 28 items with a mean of below 3.00 suggesting that the teacher respondents
in the Academy had a tendency to disagree with these statements concerning their professional
learning.
On average, the five least frequently reported processes were concerned with active use of
educational evidence to advance practice and enhance their professional development in term of
‘carry out joint research/evaluation’ to improve practice (S6A15), ‘draw on good practice from
other schools’ for further professional development (S6A2), ‘read research reports’ for practice
improvement (S6A3), ‘modify practice in the light of published research evidence’ (S6A12) and
‘engage in team teaching’ for improvement of practice (S6A18).
In contrast, among the 12 out of 28 items with a mean of above 3.00 the five most regular
processes engaged in by the teacher participants were in relation to improving their teaching and
learning practice based on ‘the belief that all students are capable of learning’ (S6A25), ‘experiment
with my practice as a conscious strategy’ (S6A10), ‘evidence from evaluations of my classroom
practice by managers or other colleagues’ (S6A14), offering ‘reassurance and support to colleagues’
(S6A24), and discussing with colleagues ‘what and how they are learning’ (S6A22) as well as ‘how
students might be helped to learn how to learn’ (S6A28).
Further, there was generally high variation between the teacher respondents’ responses to the
frequency with which they engaged in the following processes: using insights of their professional
learning to ‘feed into Academy policy development’ (S6A8), believing that ‘staff as well as students
learn in this Academy’ (S6A1), being involved in ‘team teaching as a way of improving practice’
(S6A18), suggesting ‘ideas or approaches for colleagues to try in class’ (S6A20) and conducting joint
research/evaluation to improve practice (S6A15). On the other hand, there was a common
agreement by the teacher respondents about themselves, believing ‘all students are capable of
learning’ (S6A25), seeking colleagues’ help when facing ‘a problem with my teaching’ (S6A19),
95
reflecting on their practice for ‘identifying my professional learning needs’ (S6A9), modifying their
practice taking account of ‘evidence from self-evaluation’ (S6A13) as well as ‘feedback from my
students’ (S6A11).
The following Table 54 shows the mean score and standard deviation on each question ranked from
the lowest score to the highest.
Table 54 Strand Six Part A, Means and Standard Deviations, ranked from lowest to highest
N Min Max Mean SD
Part A
S6A15 S6 A15: I carry out joint research/evaluation with one or more colleagues as a way of improving my practice
9 1 3 1.78 0.97
S6A2 S6 A2: I draw on good practice from other schools as a means to further my own professional development
10 1 3 1.80 0.79
S6A3 S6 A3: I read research reports as one source of useful ideas for improving my practice
10 1 3 2.10 0.57
S6A12 S6 A12: I modify my practice in the light of published research evidence
10 1 3 2.10 0.88
S6A18 S6 A18: I engage in team teaching as a way of improving practice 10 1 4 2.20 1.14
S6A8 S6 A8: I use insights from my own professional learning to feed into Academy policy development
9 1 4 2.33 1.32
S6A6 S6 A6: I relate what works in my own practice to research findings 10 2 4 2.50 0.71
S6A17 S6 A17: I observe other colleagues in the classroom and we give each other feedback
10 2 4 2.70 0.82
S6A26 S6 A26: My experience is that students in this Academy enjoy learning
9 2 4 2.78 0.67
S6A5 S6 A5: I consult my students about how they learn most effectively 10 2 4 2.80 0.92
S6A21 S6 A21: I make collective agreements to test out new ideas 10 2 4 2.80 0.92
S6A23 S6 A23: I use informal opportunities to discuss how children learn 10 2 4 2.80 0.92
S6A20 S6 A20: I suggest ideas or approaches for colleagues to try in class 10 1 4 2.80 1.03
S6A1 S6 A1: Staff as well as students learn in this Academy 10 1 4 2.80 1.14
S6A4 S6 A4: I use the web as one source of useful ideas for improving my practice
10 2 4 2.90 0.74
S6A16 S6 A16: I collaborate to plan my teaching 10 2 4 2.90 0.88
S6A7 S6 A7: I am able to see how practices that work in one context might be adapted to other contexts
10 2 4 3.10 0.74
S6A19 S6 A19: If I have a problem with my teaching I turn to colleagues for help
10 3 4 3.20 0.42
S6A27 S6 A27: My experience is that student success is celebrated 7 3 4 3.29 0.49
S6A9 S6 A9: I reflect on my practice as a way of identifying my professional learning needs
10 3 4 3.30 0.48
S6A13 S6 A13: I modify my practice in the light of evidence from self-evaluations of my classroom practice
10 3 4 3.30 0.48
S6A11 S6 A11: I modify my practice in the light of feedback from my students
10 3 4 3.30 0.48
S6A22 S6 A22: I discuss openly with colleagues what and how they are learning
9 2 4 3.33 0.71
S6A28 S6 A28: I discuss with colleagues how students might be helped to learn how to learn
9 3 4 3.33 0.50
96
S6A24 S6 A24: I offer reassurance and support to colleagues 10 2 4 3.40 0.84
S6A14 S6 A14: I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues
9 3 4 3.56 0.53
S6A10 S6 A10: I experiment with my practice as a conscious strategy for improving classroom teaching and learning
10 3 4 3.60 0.52
S6A25 S6 A25: My practice is based on the belief that all students are capable of learning
10 3 4 3.80 0.42
PART B – ITEM MEAN SCORES AND STANDARD DEVIATIONS
Only three (11.5%) out of 26 items have a mean of above 3.00 indicating that the teacher
respondents considered that the majority of the activities (88.5%) stated in Part B regarding
leadership practices and systems in the Academy were not frequently happening. The three
generally most common processes were about staff having ‘a commitment to the whole Academy
as well as to their department, key stage and/or year group’ (S6B2) along with senior leaders’
promotion of this perspective (S6B3) and teachers being encouraged to ‘experiment with new ideas
as a way of promoting professional growth’ (S6B17). On the contrary, there were four items with a
mean of below 2.00 implying that the most infrequent processes were perceived by these teachers
to be the Academy providing ‘cover to allow staff joint planning time’ (S6B16), teachers collecting
information from informal networking where ‘they play an active role’ (S6B24), teacher-initiated
networking playing an vital role for staff development (S6B25), staff being involved in ‘important
decision-making’ (S6B10) and teachers searching for information about how to do work effectively
(S6B22).
Additionally, the teacher respondents had five greatest disagreement, or variation in response
about how often senior leaders communicate clearly about the vision of the Academy (S6B1), staff
being engaged in evaluating the Academy policy (S6B9), staff having enough understanding of the
Academy plan (S6B12), staff taking part in important decision-making (S6B10) and teachers
collecting information about the effective learning to learn of their students (S6B23), although
these activities were generally reported to tend to be less frequent. Comparatively, the five
processes that received greatest agreement on the responses from the teacher respondents
included how often the Academy buys out staff time for joint planning (S6B16), an effective
communication between senior leaders and teachers taking place (S6B4), students being involved
in the Academy decision-making (S6B11), the Academy using teachers’ profession know-how to
establish relevant policy and goals (S6B6) and helping teachers to develop skills of observing
classroom learning (S6B20).
Table 55 below shows the means and standard deviations for each item in Part B, ranked from
lowest to highest.
97
Table 55 Strand 6, part B, means and standard deviations ranked from lowest to highest
N Min Max Mean SD
Part B
S6B16: The Academy provides cover to allow staff joint planning time 10 1 2 1.10 0.32
S6B24: Information is collected from teachers on informal teacher networking in which they play an active role
10 1 2 1.50 0.53
S6B25: Teacher-initiated networking is an integral element of staff development
10 1 2 1.50 0.53
S6B10: Staff participate in important decision-making 9 1 4 1.89 1.05
S6B22: Information is collected from teachers on those aspects of their work that they themselves think they do effectively
10 1 3 2.10 0.74
S6B4: There is effective communication between senior leaders and teachers
9 2 3 2.11 0.33
S6B26: Learning how to learn is an issue discussed in staff development time
9 1 3 2.11 0.78
S6B8: Opportunities are provided for teachers to critically evaluate Academy policy
9 1 4 2.11 0.93
S6B5: There are processes for involving all staff in decision-making 9 1 4 2.11 0.93
S6B11: There are processes for involving students in decision-making 9 2 3 2.11 0.33
S6B21: Senior leaders support teachers in sharing practice with other schools through networking
10 1 3 2.20 0.92
S6B6: Teachers’ professional know-how is used in the formulation of Academy policy and goals
9 2 3 2.22 0.44
S6B9 : Staff are involved in evaluating Academy policy 10 1 4 2.30 1.25
S6B12: Staff have a good working knowledge of the Academy Development Plan (or your Academy’s equivalent document)
10 1 4 2.30 1.06
S6B15: Staff development time is used effectively in the Academy 10 2 3 2.50 0.53
S6B1 : Senior leaders communicate a clear vision of where the Academy is going
10 1 4 2.50 1.27
S6B14: Staff development time is used effectively to realise Academy Development Plan priorities
10 2 4 2.60 0.70
S6B20: Teachers are helped to develop skills to observe learning as it happens in the classroom
9 2 3 2.67 0.50
S6B7: Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices
10 2 4 2.70 0.82
S6B23: Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students
10 1 4 2.80 1.03
S6B13: Staff see the Academy Development Plan as relevant and useful to learning and teaching
10 2 4 2.90 0.74
S6B19: Teachers are helped to develop skills to assess students’ work in ways that move students on in their learning
10 2 4 2.90 0.74
S6B18: Formal training provides opportunities for teachers to develop professionally
10 2 4 2.90 0.57
S6B3: Senior leaders promote commitment among staff to the whole Academy as well as to the department, key stage and/or year group
10 2 4 3.10 0.74
S6B17: Teachers are encouraged to experiment with new ideas as a way of promoting professional growth
10 3 4 3.50 0.53
S6B2: Staff have a commitment to the whole Academy as well as to their 10 3 4 3.60 0.52
98
department, key stage and/or year group
PART C – ITEM MEANS AND STANDARD DEVIATIONS
There were four out of 12 items with a mean of equal to or above 3.00 indicating that teachers
agree strongly with the belief that they are realizing their ‘full potential as a teacher’ (S6C12),
feeling ‘included in a community that values’ what they can offer (S6C1), being given ‘opportunities
to work alongside and/or train with colleagues in other phases to improve students’ progress’
(S6C4) and experiencing that senior teachers behave as role models ‘as leaders of learning’ (S6C5).
However, the last two questions also demonstrated greatest variation from the teacher
respondents. In contrast, the five items most strongly disagreed with by the teacher respondents
were about the Academy productively assisting ‘parents/carers to support the learning of their
children’ (S6C10), the Academy promoting ‘a balance between the rights and the responsibilities’
within the community (S6C7), staff achievement being ‘recognised and celebrated’ (S6C11), other
staff members ‘involved in training and development activities to improve their classroom practice
and student outcomes’ (S6C3) and ‘students and staff on caring about and helping other people in
the Academy community’ being emphasised (S6C8) where the first three also obtained highest
agreements in the teacher respondents’ responses.
The following Table 56 shows the mean scores and standard deviations, ranked from lowest to
highest.
Table 56 Strand 6 Part C, mean score and standard deviation ranked from lowest to highest
N Min Max Mean SD
Part C
S6C10 S6 C10: The Academy is successful in helping parents/carers to support the learning of their children
10 2 3 2.30 0.48
S6C7 S6 C7: My experience is that the Academy promotes a balance between the rights and the responsibilities of all members of the community
10 2 3 2.40 0.52
S6C11 S6 C11: The achievements of staff are recognised and celebrated 9 2 3 2.56 0.53
S6C3 S6 C3: My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes
10 2 4 2.60 0.70
S6C8 S6 C8: There is an emphasis among students and staff on caring about and helping other people in the Academy community
10 2 4 2.70 0.95
S6C9 S6 C9: There is an emphasis among students and staff on caring about and helping other people in the wider community
10 2 4 2.80 0.92
S6C6 S6 C6: Leadership is shared in this Academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience
9 2 3 2.89 0.33
S6C2 S6 C2: I believe that I am working in a community that focuses our attention and activities on learning to improve teaching and student outcomes
10 2 4 2.90 0.74
99
S6C5 S6 C5: My experience is that senior leaders are role models as ‘leaders of learning’ in the Academy
10 2 4 3.00 1.05
S6C4 S6 C4: I have had opportunities to work alongside and/or train with colleagues in other phases to improve students’ progress
10 1 4 3.10 1.45
S6C1 S6 C1: I feel included in a community that values what I can offer to the staff team and to students
10 2 4 3.20 0.79
S6C12 S6 C12: I believe that I am realising my full potential as a teacher 10 3 4 3.50 0.53
SUMMARY AND FINDINGS FROM STRAND SIX
This Strand was a three part questionnaire designed to explore teachers’ perceptions of their
Academy and the impact of their professional learning on their classroom practice.
The first part of the questionnaire revealed a range of findings with regards to professional learning
processes, particularly research and evaluation and or applying findings of published research to
the teachers’ practice. However, teachers agreed on the five most regular processes engaged in by
the teacher participants which referred to improving their teaching and learning practice.
There was, however, a higher degree of variation with regard to the extent to which they felt
involved or engaged in practices which promoted the policy or professional development of the
Academy. But interestingly, teachers tended to show agreement on items which referred to self-
reflection and evaluation or for seeking help for teaching related issues.
The part of the questionnaire which contained items on leadership practices found that the
majority of statements did not frequently apply to practices at the Academy. Much of the findings
from this section seem to suggest that teachers did not feel especially involved in practices which
might promote the staff as part of a whole team, however, staff reported strong commitment to
their departments, key stage or year group.
The findings, nevertheless, did show that teachers felt they were reaching their potential as
teachers and felt presented with opportunities for development with good role models of senior
teachers. What is also interesting is that teachers generally disagreed with the extent to which the
Academy supports parents in the learning of their children and the balancing of
rights/responsibilities within the community. This seems to concur with the findings of Strand Five
where parents/carers responded in a similar way on items which explored how the Academy
performed as part of a community.
STRAND SEVEN - IMPACT OF CONTINUED PROFESSIONAL DEVELOPMENT (CPD) ON THE
QUALITY OF LEADERSHIP AND MANAGEMENT AND CLASSROOM PRACTICE
This Strand explored senior leaders’ perceptions of how the Academy is fulfilling the OCL Charter.
100
QUESTIONNAIRE
The Leadership questionnaire consisted of 55 items including one open-ended question. The 54
closed items were structured with a four-point Likert type scale labeled, strongly disagree, disagree,
agree, and strongly agree, which were subsequently coded as 1, 2, 3 and 4 respectively and treated
as numerical scores at the stage of analysis in order to investigate respondents’ disagreement or
agreement on individual statements.
SAMPLE AND DATA COLLECTION
The questionnaire was administered in the three Academies as an online survey constructed on the
Academies intranet. In total, 25 questionnaires filled in by SMT staff were received: nine from
Academy 1, ten from Academy 2 and six from Academy 3. The percentages of missing responses
from the 25 respondents ranged from 0.0 to 36.0 across all Likert-scale items. Referring to the
highest missing response rate, there were 9 out of 25 SMT staff (36.0% of the respondents) missing
out reporting whether or not they agree on their lessons being rated as good or outstanding (S73).
Noticeably, of these, 7 were from the same Academy indicating possibly that in the Academy either
SMT staff did not undertake teaching or there was no mechanism of appraisal of SMT staff’s
teaching. Overall SMT participants responded more positively than negatively to two-third of the
items. (Details of SMT participants’ responses to individual items are provided in Appendix 4)
FINDINGS AND DISCUSSION
Table 57 below presents the responses for each question as a numerical value and a percentage. It
also identifies the number of missing values.
Table 57 SMT participants’ responses to 54 Likert-scale items
Item 1
strongly
disagree
2
disagree
3
agree
4 strongly
agree
%
missing
N % N % N % N % total
S71 1. I have a good understanding of my own and other people’s learning
0 0.0 0 0.0 13 52.0 12 48.0 25 0.0
S72 2. I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date
0 0.0 1 4.3 9 39.1 13 56.5 23 8.0
S73 3. I teach regularly and my lessons are rated good or outstanding
0 0.0 0 0.0 5 31.3 11 68.8 16 36.0
S74 4. I undertake regular and substantial professional development for myself
0 0.0 5 20.0 11 44.0 9 36.0 25 0.0
S75 5. I understand the differences between 0 0.0 0 0.0 9 36.0 16 64.0 25 0.0
101
good and outstanding lessons
S76 6. I regularly work with staff to improve their teaching practice
0 0.0 1 4.5 13 59.1 8 36.4 22 12.0
S77 7. I can design, monitor and assess programmes for effective teaching and learning
0 0.0 0 0.0 17 77.3 5 22.7 22 12.0
S78 8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the Academy
1 4.2 4 16.7 15 62.5 4 16.7 24 4.0
S79 9. I understand what an Academy needs to do to become good and outstanding
0 0.0 0 0.0 12 48.0 13 52.0 25 0.0
S710 10. I regularly work with staff to help them improve their leadership and management
0 0.0 1 4.3 14 60.9 8 34.8 23 8.0
S711 11. I am an effective coach and use my coaching skills regularly to help others to improve their performance
0 0.0 0 0.0 18 78.3 5 21.7 23 8.0
S712 12. I regularly interact with staff in other schools, Academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our Academy
2 8.7 4 17.4 10 43.5 7 30.4 23 8.0
S713 13. I make regular and effective contributions to educational dialogue that shapes whole Academy policy and informs practice
0 0.0 0 0.0 13 54.2 11 45.8 24 4.0
S714 14. I contribute effectively to the critical development of Academy systems and structures in order to ensure quality teaching focused on student progress in learning
0 0.0 2 8.7 9 39.1 12 52.2 23 8.0
S715 15. The senior leadership team communicates a clear vision of where the Academy is going so that there is a shared sense of purpose
0 0.0 0 0.0 12 50.0 12 50.0 24 4.0
S716 16. Staff have a commitment to the whole Academy as well as to their department, key stage and/or year group
0 0.0 0 0.0 18 72.0 7 28.0 25 0.0
S717 17. The senior leadership team effectively promotes commitment among staff to the future development of the whole Academy
0 0.0 3 12.0 11 44.0 11 44.0 25 0.0
S718 18. There is effective communication between senior leaders and teachers
0 0.0 2 8.0 17 68.0 6 24.0 25 0.0
S719 19. There are processes for involving all staff in decision-making
0 0.0 2 8.0 16 64.0 7 28.0 25 0.0
S720 20. Teachers’ professional know-how is used in the formulation of Academy policy and goals
0 0.0 4 16.0 19 76.0 2 8.0 25 0.0
S721 21. Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices
0 0.0 5 21.7 14 60.9 4 17.4 23 8.0
S722 22. Opportunities are provided for teachers to critically evaluate Academy policy
0 0.0 4 16.7 18 75.0 2 8.3 24 4.0
S723 23. Staff participate in important 0 0.0 5 20.0 17 68.0 3 12.0 25 0.0
102
decision-making
S724 24. There are processes for involving students in decision-making, including how best to improve classroom practice
0 0.0 5 20.8 12 50.0 7 29.2 24 4.0
S725 25. Staff have a good working knowledge of the Academy Development Plan
0 0.0 7 29.2 13 54.2 4 16.7 24 4.0
S726 26. Staff see the Academy Development Plan as relevant and useful to learning and teaching
0 0.0 6 25.0 16 66.7 2 8.3 24 4.0
S727 27. Staff development time is used effectively to realise Academy Development Plan priorities
0 0.0 1 4.2 19 79.2 4 16.7 24 4.0
S728 28. Staff development time is used effectively in the Academy, particularly through the explicit modelling of good practice
0 0.0 2 8.0 18 72.0 5 20.0 25 0.0
S729 29. The Academy provides support to allow staff joint planning time
0 0.0 6 25.0 17 70.8 1 4.2 24 4.0
S730 30. Teachers are encouraged to experiment with new ideas as a way of promoting professional growth and improving classroom practice
0 0.0 1 4.2 17 70.8 6 25.0 24 4.0
S731 31. Formal training provides opportunities for teachers to develop professionally
0 0.0 0 0.0 14 56.0 11 44.0 25 0.0
S732 32. Teachers are helped to develop skills to assess students’ work in ways that move their students on in their learning
0 0.0 1 4.2 18 75.0 5 20.8 24 4.0
S733 33. Teachers are helped to develop skills to observe learning as it happens in the classroom
0 0.0 1 4.2 18 75.0 5 20.8 24 4.0
S734 34. Senior leaders support teachers in sharing practice with other schools and Academies through networking
1 4.2 1 4.2 18 75.0 4 16.7 24 4.0
S735 35. Information is collected from teachers on those aspects of their work that they themselves think they do effectively
0 0.0 4 16.7 16 66.7 4 16.7 24 4.0
S736 36. Information is collected from teachers on effective ways they promote ‘learning to learn’ skills and knowledge among their students
0 0.0 6 25.0 17 70.8 1 4.2 24 4.0
S737 37. Information is collected from teachers on informal teacher networking in which they play an active role
0 0.0 9 39.1 14 60.9 0 0.0 23 8.0
S738 38. Teacher-initiated networking is an integral element of staff development
0 0.0 8 34.8 15 65.2 0 0.0 23 8.0
S739 39. ‘Learning how to learn’ is an issue discussed in staff development time
0 0.0 5 22.7 16 72.7 1 4.5 22 12.0
S740 40. There is a well-developed shared language and a rich dialogue about teaching and learning across the Academy
0 0.0 1 4.2 18 75.0 5 20.8 24 4.0
S741 41. We are good at systematic monitoring, use of data and feedback across the Academy
0 0.0 4 16.0 9 36.0 12 48.0 25 0.0
103
S742 42. We have efficient and effective support systems and structures across the Academy
0 0.0 0 0.0 13 52.0 12 48.0 25 0.0
S743 43. The senior leadership team are effective role models as ‘leaders of learning’,
0 0.0 1 4.0 14 56.0 10 40.0 25 0.0
S744 44. We have developed successful approaches to distributed leadership across the Academy where the pathways of accountability are working well
0 0.0 0 0.0 18 72.0 7 28.0 25 0.0
S745 45. The Academy is successful in helping parents/carers to support the learning of their children
0 0.0 9 36.0 16 64.0 0 0.0 25 0.0
S746 46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes
2 9.1 5 22.7 15 68.2 0 0.0 22 12.0
S747 47. The Academy involves a wide range of other stakeholders directly and powerfully in the education of students
0 0.0 2 8.3 18 75.0 4 16.7 24 4.0
S748 48. The Academy is good at securing the broadest possible impact of successful innovations in teaching and learning
0 0.0 2 8.3 21 87.5 1 4.2 24 4.0
S749 49. There is a widespread and well-established climate of trust in the Academy
0 0.0 1 4.0 20 80.0 4 16.0 25 0.0
S750 50. There is an appetite to face weaknesses and bring about changes for improvement in the Academy
0 0.0 1 4.2 14 58.3 9 37.5 24 4.0
S751 51. Good personal relationships and strong teams help us to face challenges with confidence
0 0.0 0 0.0 14 56.0 11 44.0 25 0.0
S752 52. The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing
0 0.0 0 0.0 8 32.0 17 68.0 25 0.0
S753 53. The senior leadership team is good at long term planning for capacity building
0 0.0 3 12.5 12 50.0 9 37.5 24 4.0
S754 54. As an Academy, we are good at celebrating success
0 0.0 1 4.2 16 66.7 7 29.2 24 4.0
LOWEST AND HIGHEST SCORES
Table 58 below lists the items with the five lowest and five highest mean responses. On average,
SMT respondents seemed to be less positive on the government fulfilling its role in developing
strategy to enhance teaching and learning leading to improvement of student outcomes (S746) and
the Academies being of assistance to parents/carers in their children’s learning (S745). Also the
respondents tended to disagree on information being collected from informal teacher networking
(S737), teacher initiated networking being used as an important platform for staff development
(S738) and specific time being allocated for staff jointly planning (S729). Interestingly, there was
tendency of agreement between SMT themselves, for example, knowing what needs to be done for
an Academy to be good and outstanding (S79), maintaining their professional development (S72) as
well as possessing skills and resilience to overcome challenges (S752) in addition to understanding
104
the difference between good and outstanding lessons (S755), and their lessons being rated as good
or outstanding (S73). These findings suggest that Academy SMT staff may perceive an unfriendly
macro working environment in terms of relevant governmental and Academy based policy
measures in supporting teaching and learning that focuses on student achievement, while they take
up the responsibility for their own professional development.
Table 58 Items of the five highest and least mean scores
N Min Max Mean SD
Items of the five lowest mean responses
S746 46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes
22 1 3 2.59 0.67
S737 37. Information is collected from teachers on informal teacher networking in which they play an active role
23 2 3 2.61 0.50
S745 45. The Academy is successful in helping parents/carers to support the learning of their children
25 2 3 2.64 0.49
S738 38. Teacher-initiated networking is an integral element of staff development
23 2 3 2.65 0.49
S729 29. The Academy provides support to allow staff joint planning time
24 2 4 2.79 0.51
Items of the five highest mean responses
S79 9. I understand what an Academy needs to do to become good and outstanding
25 3 4 3.52 0.51
S72 2. I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date
23 2 4 3.52 0.59
S75 5. I understand the differences between good and outstanding lessons
25 3 4 3.64 0.49
S752 52. The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing
25 3 4 3.68 0.48
S73 3. I teach regularly and my lessons are rated good or outstanding
16 3 4 3.69 0.48
(1=strongly disagree to 4=strongly agree)
Table 59 presents items with most and least consensus in the respondents’ responses ranging from
0.36 to 0.93 in terms of standard deviation, a measure employed to demonstrate the dispersion of
the SMT participants’ responses from their mean responses. In general, SMT staff reported the
least differences in whether or not they agree on their Academy being good at securing the impact
of innovation on teaching and learning (S748), effectively helping others’ improvement (S711),
being able to plan and evaluate effective teaching and learning programmes (S77), sound trust
existed within their Academy (S749) and staff development time well used to implement the
Academy development plan (S727). On the other hand, the least concurrences of SMT staff
responses were about their disagreement or agreement on regularly taking part in reflective
enquiry (S78), students being involved in the Academy’s decision making particularly relating to
classroom practice (S724), regularly attending professional development activities (S74), being good
at using evidence based data (S741) and frequently networking outside the Academy (S712) for
improvement.
105
These findings imply the issues of fairness and democracy across Academies in relation to
opportunity being created for improvement of teaching and learning through self-reflection,
participation in relevant professional development activities, use and analysis of data, engagement
in external partnerships and involvement of students in decision making.
Table 59 Items of the five most and five least consensuses in responses (ascending rank by standard deviation (SD)
N Min Max Mean SD
Items of the five most consensuses in responses
S748 48. The Academy is good at securing the broadest possible impact of successful innovations in teaching and learning
24 2 4 2.96 0.36
S711 11. I am an effective coach and use my coaching skills regularly to help others to improve their performance
23 3 4 3.22 0.42
S77 7. I can design, monitor and assess programmes for effective teaching and learning
22 3 4 3.23 0.43
S749 49. There is a widespread and well-established climate of trust in the Academy
25 2 4 3.12 0.44
S727 27. Staff development time is used effectively to realise Academy Development Plan priorities
24 2 4 3.13 0.45
Items of the five least consensuses in responses
S78 8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the Academy
24 1 4 2.92 0.72
S724 24. There are processes for involving students in decision-making, including how best to improve classroom practice
24 2 4 3.08 0.72
S74 4. I undertake regular and substantial professional development for myself
25 2 4 3.16 0.75
S741 41. We are good at systematic monitoring, use of data and feedback across the Academy
25 2 4 3.32 0.75
S712 12. I regularly interact with staff in other schools, Academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our Academy
23 1 4 2.96 0.93
(1=strongly disagree to 4=strongly agree)
EXPLORING PATTERNS IN SENIOR LEADER PARTICIPANTS’ RESPONSES
In combination with varimax rotation to hypothesise potential components being uncorrelated to
each other, Principal Components Analysis (PCA) was performed to explore possible underlying
patterns in SMT participants’ responses to the Likert-scale items by reducing the number of items
and dimensionality. As compared to the number of items in the questionnaire, the sample size
obtained in the study was relatively small. Hence, missing values of individual items were replaced
with the mean during the procedure of principal component analysis. However, the item “I teach
regularly and my lessons are rated good or outstanding” (S73) was excluded from the analysis
because nearly one-third of the participants reported with missing responses.
Multiple criteria were taken into account to determine and retain items/components for
interpretation including i) an eigenvalue greater than 1.0; ii) the scree test; iii) proportion of
variance accounted for; and iv) interpretability (e.g., item loadings greater than 0.4 (and on only
106
one given component to be expected), items of a given component sharing similar conceptual
meaning. PCA was repeatedly conducted. After removing 33 items, four potentially meaningful
components with eigenvalues greater than 1 emerged that explained 73.6% of the total variance.
All the remaining 20 items had communalities greater than 0.5 (rounded). Bartlett’s test was
statistically significant at 0.00 level and the value of overall Kaiser Meyer Olker measure of sampling
adequacy was 0.55 indicating a mild satisfactory analysis.
Table 60 shows the rotated component matrix for the items with the estimated loadings. These
factors were found to be reliable with Cronbach's Alpha value ranging from 0.84 to 0.88. Since this
study is exploratory in nature some items with loadings of greater than 0.400 on more than two
components were retained for the purpose of identifying underlying dimensions to be tested in a
larger sample size in a further study.
Table 60 Rotated Component Matrix Strand Seven
Item Component
1 2 3 4
S734 34. Senior leaders support teachers in sharing practice with other schools and Academies through networking
.890
S749 49. There is a widespread and well-established climate of trust in the Academy
.771
S712 12. I regularly interact with staff in other schools, Academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our Academy
.764
S714 14. I contribute effectively to the critical development of Academy systems and structures in order to ensure quality teaching focused on student progress in learning
.648 .503
S746 46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes
.636
S713 13. I make regular and effective contributions to educational dialogue that shapes whole Academy policy and informs practice
.573 .471
S78 8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the Academy
.459 .455 .448
S727 27. Staff development time is used effectively to realise Academy Development Plan priorities
.813
S750 50. There is an appetite to face weaknesses and bring about changes for improvement in the Academy
.788
S728 28. Staff development time is used effectively in the Academy, particularly through the explicit modelling of good practice
.781
S741 41. We are good at systematic monitoring, use of data and feedback across the Academy
.711
S753 53. The senior leadership team is good at long term planning for capacity building
.686 .565
S742 42. We have efficient and effective support systems and structures across the Academy
.423 .634
S726 26. Staff see the Academy Development Plan as relevant and useful to learning and teaching
.872
107
S725 25. Staff have a good working knowledge of the Academy Development Plan .803
S723 23. Staff participate in important decision-making .625
S720 20. Teachers’ professional know-how is used in the formulation of Academy policy and goals
.866
S719 19. There are processes for involving all staff in decision-making .838
S721 21. Teachers’ professional know-how is used in the formulation of Academy policy, even where this leads to a questioning of established rules, procedures and practices
.659
S718 18. There is effective communication between senior leaders and teachers .528
Reliability (alpha) 0.88 0.88 0.85 0.84
Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 6 iterations.
FOUR UNDERLYING THEMES
The four underlying themes are outlined in Table 61 below.
Table 61 Underlying Themes for staff questionnaire
Theme one explains 41.3% of the variance with a
reliability of Cronbach’s Alpha value of 0.88 including
seven items
I engage in learning and dialogue about the leadership of our Academy with a wide range of colleagues in a positive climate of trust
Theme two has a Cronbach’s Alpha value of 0.88
explaining 13.7% of the variance and consists of six items
We monitor progress carefully, face up to challenges and systematically support and encourage teacher progress
Theme three with a reliability of Cronbach’s Alpha value of
0.85 explaining 10.6% of the variance contains three items
The Academy Development Plan is relevant to learning and teaching and is a focus for shared decision making
Theme four has a Cronbach’s Alph value of 0.84 explaining
6.5% of the variance and includes four items
All staff are involved in Academy decision making
COMPUTING NEW VARIABLES
Because the exploratory factor analysis reveals four clear themes, for this Strand, new variables
were computed for each individual and these were then used for the Perimeta model.
SUMMARY AND FINDINGS FROM STRAND SEVEN
This Strand explored senior leaders’ perceptions from all three Academies and how they are viewed
to be fulfilling the OCL Charter.
The findings from the 55 item questionnaire seems to suggest that senior management teams
(SMT) perceive an unfriendly macro working environment in terms of relevant governmental and
Academy based policy measures in supporting teaching and learning that focuses on student
108
achievement. However, SMT seem to demonstrate a confidence in taking up the responsibility for
their professional development.
The findings also imply the issues of fairness and democracy across Academies in relation to
opportunity being created for improvement of teaching and learning through self-reflection,
participation in relevant professional development activities, use and analysis of data, engagement
in external partnerships and involvement of students in decision making.
Principal Components Analysis (PCA) was performed to explore possible underlying patterns in SMT
participants’ responses to the Likert-scale items by reducing the number of items and
dimensionality. Four underlying themes were revealed which were used to reduce and strengthen
the data for further analysis (by Perimeta).
STRAND EIGHT & NINE – CONTEXT FOR STUDENT LEARNING - KEY PERFORMANCE
INDICATORS & KEY OUTCOME FOR YEAR 11 STUDENTS – GCSE/EXAM RESULTS
Data from Key Performance Indicators, such as attendance, free school meals (FSM, exclusions,
special educational needs (SEND), gender and GCSE results.
SAMPLE AND DATA COLLECTION
KPI data and the GCSE results from Academy 2 (153 records) and Academy 3 (166 records) were
explored for differences between and within Academies with regards to gender, FSM and SEN in
relation to GCSE results in English, Maths and Science.
DIFFERENCES WITHIN ACADEMIES
Academy 2
Note that only the results of i) GCSE Mathematics, ii) GCSE English Language & GCSE English
Literature, and iii) GCSE Science Double Award/GCSE Biology & GCSCE Chemistry & GCSE Physics
that were taken by the majority of students were analysed by gender, FSM and SEN. Table 54
indicates the number of students by gender, FSM and SEN where 55.6% of students were male.
There were 20.3% of students entitled to FSM. Notably, 47.7% of students have received the
support of either SEN school action or SEN school action plus.
Table 62 Frequency of Academy 2 students by gender, FSM and SEN
Students by gender
Students by FSM
Students by SEN
109
Gender Frequency %
male 85 55.6
female 68 44.4
Total 153 100.0
FSM Frequency %
not
Eligible
122 79.7
Eligible 31 20.3
Total 153 100.0
SEN Frequency %
No 80 52.3
School
action
38 24.8
School
action
plus
35 22.9
Total 153 100.0
GCSE Mathematics: All 153 students at Academy 2 took GCSE Maths. One student (0.7%) was
graded as ‘X’ for completing only part of the course. Over half the students (83, 54.2%) received a C
grade (see Table 63). Students graded at either AA to B totaled 29, (19%) and D to F totaled 26,
(17%).
Table 63 Academy 2 Frequency of GCSE Maths grades
Grade Frequency % %
AA 6 3.9
19.0 A 9 5.9
B 14 9.2
C 83 54.2 54.2
D 9 5.9
17.0 E 6 3.9
F 11 7.2
G 12 7.8 7.8
U 2 1.3 1.3
110
X 1 0.7 0.7
Total 153 100.0 100.0
GCSE Mathematics by Gender: Of the 153 students in Academy 2 who took GCSE Maths, 85
(55.6%) were male. As shown on the chart below (Figure 9), more male than female students were
awarded grades A to C. A higher percentage of female students were among the D to G grades. The
results reflect the existing pattern of Maths attainment between the genders where male students
generally outperform female students.
Table 64 GCSE Maths results by gender
GCSE Mathematics by FSM: Of 153 students in Academy 2 who took the GCSE Maths exam, 31
(20.3%) were FSM eligible students. Table 65 below shows that a large percentage of FSM and Non
FSM students achieved grade C. However, there was a marked difference in the percentage of FSM
students at 22.6% achieving grade G as compared with 4.1% of Non FSM students.
Table65 GCSE Maths results by FSM
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
GCSE Mathematics (sch_2)
Male
Female
-10.0%
10.0%
30.0%
50.0%
70.0%
AA A B C D E F G UPer
cen
tage
wit
hin
FS
M
GCSE Mathematics (Sch_2)
No…
111
GCSE Mathematics by SEN: There were a high proportion of students in Academy 2 identified as
having learning difficulties or disabilities. Of 153 students in Academy 2 who took GCSE Maths, 38
(24.8%) received specific additional help (i.e. SEN status = school action), and 35 (22.9%) who
received further external help (i.e. SEN status = action_plus). As expected, and shown in Figure 11
below, a greater percentage of No_SEN students were rewarded AA to C grades than their Yes_SEN
counterparts. Additionally, a much higher percentages of students with action_SEN (5.3% / 10.5%)
or action_plus_SEN (22.9% / 20.0%) than those students without SEN (1.3% / 1.3%) had a grade of
F/G. Noticeably, there were 2 (5.3%) action_SEN students that had a grade of A indicating some
impact of the interventions made by the Academy for those SEN students. However, these two
students were labelled as gifted or talented.
Table 66 GCSE Mathematics by SEN
GCSE Mathematics by EAL: Only 12 (7.8%) students reported English as an Additional Language.
GCSE Mathematics by Ethnicity: Of the 153 students, 135 or 88.2% were white English.
GCSE English/GCSE English Language & GCSE English Literature: Students can take GCSE English or
both GCSE English Language and GCSE English Literature. In Academy 2, there were 80 students
(52.3%) who took GCSE English while 71 students (46.4%) sat both English Language and English
Literature exams (note that two students (1.3%) only took GCSE English Language but not English
Literature). Table 67 shows the percentage of students taking each of the GCSE English exams and
those who did not.
Table 67 Academy 2 Frequency of GCSE English / English Language / English Literature results
GCSE English
GCSE English Language
GCSE English Literature
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
SEN
GCSE Mathematics (Sch_2)
No_SEN
action_SEN
action_plus_SEN
112
Grade Frequency %
NoSitIn 73 47.7
C 7 4.6
D 38 24.8
E 21 13.7
F 11 7.2
G 2 1.3
U 1 .7
Total 153 100.0
Grade Frequency %
NoSitIn 80 52.3
AA 1 .7
A 5 3.3
B 12 7.8
C 46 30.1
D 9 5.9
Total 153 100.0
Grade Frequency %
NoSitIn 82 53.6
A 2 1.3
B 17 11.1
C 18 11.8
D 27 17.6
E 7 4.6
Total 153 100.0
Moreover, among those who took both English Language and English Literature, a total of 47.8% of
students achieved the same grade for both the GCSEs (see Table 68 below). The remaining majority
of students, however, scored a higher grade in English Language than in English Literature, where
31% of students achieved grade at C in English Language and grade of D in English Literature.
Table 68 GCSE English Language and English Literature results
113
GCSE English/GCSE English Language and GCSE English Literature by Gender: Among those
students that took GCSE English, no students achieved grade B or higher (see Table 69 below). More
females than males achieved grades C, E, F and G. However, more males than females achieved a
grade D - 54.2% compared to the females at 31.6%.
Table 69 GCSE English by Gender
Interestingly, and on the contrary, those students who took GCSE English Language all had a grade
of D or above, where the majority scored grade C, 64.8% for male and 61.1% for female (see Table
70 below). The percentages of male and female students with a grade of A or B or C were similar;
but there were more females than male students who were awarded a D grade in the GCSE.
Table 70 GCSE English Language by Gender
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
AA A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
GCSE English (Sch_2)
Male
Female
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
GCSE English Language (Sch_2)
Male
Female
114
As to those students who took GCSE English Literature, in general, slightly higher percentages of
male students than those of female students achieved a B, D or E grade. However, the percentage
of female student with a grade C was twice as high as those of male students.
Table 71 GCSE English Literature by Gender
GCSE English/GCSE English Language and GCSE English Literature by FSM: There was little
difference between FSM and non-FSM at grade C in GCSE English. However, the number of FSM
students achieving grade D was half that of non-FSM. The percentage of FSM students achieving
grades E, F or G was greater than non-FSM students.
Table 72 GCSE English by FSM
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
GCSE English Literature (Sch_2)
Male
Female
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
AA A B C D E F G U
Per
cen
tage
wit
hin
FSM
GCSE English (Sch_2)
No_FSM
115
In GCSE English Language, no FSM students achieved grade B or higher compared to 28.2% of non-
FSM students who achieved grade B or higher. Although at grade C there was little difference
between non-FSM and FSM students. However, the difference at grade D was more marked where
three times more FSM students achieved grade D compared to non-FSM students.
Table 73 GCSE English Language by FSM
Similarly, with regarding to GCSE English Literature, a lower percentage of FSM students achieved
grade C or higher compared to non-FSM. In contrast, there was a marked difference in the
percentage of FSM students achieving the grades D or E (see Table 74 below) compared with non-
FSM.
Table 74 GCSE English Literature by FSM
GCSE English/GCSE English Language and GCSE English Literature by SEN: The percentage of No
SEN students achieving grades C or D in GCSE English were greater than those students identified as
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
FSM
GCSE English Language (Sch_2)
No_FSM
Yes_FSM
-10.0%
10.0%
30.0%
50.0%
70.0%
AA A B C D E F G UPer
cen
tage
wit
hin
FS
M
GCSE English Literature (Sch_2)
No_F…
116
SEN. Also, a relatively higher percentage of action plus SEN students achieved an E or F grade than
action SEN students.
Table 75 GCSE English Literature by SEN
Only students identified as No SEN achieved grade B or above in GCSE English Language (see Table
76 below) with the majority of SEN students achieving a grade C.
Table 76 GCSE English Language by SEN
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
AA A B C D E F G U
Per
cen
tage
wit
hin
SEN
GCSE English (Sch_2)
No_SEN
action_SEN
action_plus_SEN
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
SEN
GCSE English Language (Sch_2)
No_SEN
action_SEN
action_plus_SEN
117
As shown in Table 77 below, No action plus SEN students scored grade B or above in GCSE English
Literature. However, the differences between the groups at grade C and D were less marked and a
greater percentage of action plus SEN than action only SEN achieved grade C.
Table 77 GCSE English Literature by SEN
GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics: Students can
opt to take the GCSE Science Double Award or three separate sciences - GCSE Biology, Chemistry or
Physics. In Academy 2, 20 (13.1%) did not take any of the science exams, while 108 (70.6%)
students took the GCSE Science Double Award and 25 (16.3%) students took the three science
exams. Table 78 shows the percentage of students sitting either the Science Double Award, the
separate sciences as well as those not taking either.
Table 78 Academy 2 Frequency of GCSE Science Double Award / Biology / Chemistry / Physics results
GCSE Science Double
Award
Grade Frequency %
NoSitIn 45 29.4
AA 2 1.3
GCSE Biology
Grade Frequency %
NoSitIn 128 83.7
AA 1 .7
A 9 5.9
GCSE Chemistry
Grade Frequency %
NoSitIn 128 83.7
A 5 3.3
AA 3 2.0
GCSE Physics
Grade Frequency %
NoSitIn 128 83.7
A 3 2.0
AA 1 .7
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
SEN
GCSE English Literature (Sch_2)
No_SEN
action_SEN
action_plus_SEN
118
AB 1 .7
BB 5 3.3
BC 16 10.5
CC 45 29.4
CD 10 6.5
DD 7 4.6
DE 5 3.3
EE 7 4.6
EF 2 1.3
FF 1 .7
FG 1 .7
GG 3 2.0
U 3 2.0
Total 153 100.0
B 6 3.9
C 6 3.9
D 3 2.0
Total 153 100.0
B 8 5.2
C 5 3.3
D 4 2.6
Total 153 100.0
B 10 6.5
C 8 5.2
D 3 2.0
Total 153 100.0
GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics by Gender:
There were 46 female and 62 male students taking the GCSE Science Double Award exam with a
slightly higher percentage (72.9%) of male than female students (67.6%). A higher percentage of
female than male students (76.1% and 54.8%, respectively) achieved grade CC or above.
119
Table 79 GCSE Double Science Award by gender
With regard to GCSE Biology results, whilst no female students achieved grade AA (nor B grade), a
considerable higher percentages of females achieved grade A than male students.
Table 80 GCSE Biology results by gender
The results for GCSE Chemistry were similar for female and male students at grade AA or B (see
Table 81 below). However, relatively lower percentages of females achieved grade A or grade D in
contrast to a higher percentages of females achieving grade C.
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA AB BB BC CC CD DD DE EE EF FF FG GG U
Per
cen
tage
wit
hin
Ge
nd
er
GCSE Science Double Award (Sch_2)
Male
Female
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G UPer
cen
tage
wit
hin
Ge
nd
er
GCSE Biology (sch_2)
Male
Female
120
Table 81 GCSE Chemistry results by gender
In general, there was little difference between female and male students at grade AA or B the in
GCSE Physics exam.
Table 82 GCSE Physics results by gender
GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics by FSM: Of the
108 students who took the GCSE Science Double Award exam, 21 were FSM entitled students. A
higher percentage of non-FSM students than FSM students had a double grade at CD or above
(78.2% and 52.4% respectively). In contrast, the percentage of FSM students was three times as
much as the non-FSM who achieved a double grade of DD or below or ungraded (33.3% and 11.5%
respectively).
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G UPer
cen
tage
wit
hin
Ge
nd
er
GCSE Chemistry (sch_2)
Male
Female
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
GCSE Physics (sch_2)
Male
Female
121
Table 83 GCSE Double Science Award by FSM
Among the 25 students that took the three science-related subject exams, 2 were FSM entitled
both achieving grade C in GCSE Biology as well as in GCSE Physics; however, one achieved grade C
and the other grade D in the GCSE Chemistry exam. The remaining 23 non-FSM students achieved
grade B or higher in GCSE Biology, GCSE Chemistry and GCSE Physics respectively.
Table 84 GCSE Biology results by FSM
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA AB BB BC CC CD DD DE EE EF FF FG GG U
Per
cen
tage
wit
hin
FSM
GCSE Science Double Award (Sch_2)
No_FSM
10
122
Table 85 GCSE Chemistry results by FSM
Table 86 GCSE Physics results by FSM
GCSE Science Double Award/GCSE Biology and GCSE Chemistry and GCSE Physics by SEN: Of the
108 students who took the GCSE Science Double Award exam, 28 were defined as SEN students.
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
FSM
GCSE Chemistry (Sch_2)
No_FSM
Yes_FSM
100%
123
Table 87 GCSE Science Double Award results by SEN
Table 88 GCSE Biology res ults by SEN
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA AB BB BC CC CD DD DE EE EF FF FG GG U
Per
cen
tage
wit
hin
SEN
GCSE Science Double Award (Sch_2)
No_SEN
action_SEN
action_plus_SEN
100%
100%
124
Table 89 GCSE Chemistry results by SEN
Table 90 GCSE Physics results by SEN
100%
100%
100% 100%
125
Academy 3
Only the results of i) uncapped average point score (APS) / capped average point score (APS) / total
points, ii) [GCSE] English, iii) [GCSE] Maths taken by the majority of students were analysed by
gender, FSM and SEN.
Table 91 summarises student composition by gender, FSM and SEN in Academy 3. Overall, 53.6% of
students were male, which is similar to Academy 2 (see Table 75.). There were 34.3% of students
entitled to FSM, and in Academy 2 only 20.3% of students entitled to FSM. Considerable 54.2% of
students, a 6.5 percentage points higher than that of in Academy 2, have received the support of
either SEN school action or SEN school action plus or been identified as statement of SEN.
Table 91 Academy 3 summary of students by gender, FSM and SEN
Students by gender
Gender Frequency %
male 89 53.6
female 77 46.4
Total 166 100.0
Students by FSM
FSM Frequency %
not
Eligible
109 65.7
Eligible 57 34.3
Total 166 100.0
Students by SEN
SEN Frequency %
No 75 45.2
School
action
48 28.9
School
action plus
34 20.5
Statement
of SEN
9 5.4
Total 166 100.0
Uncapped average point score (APS) and capped average point score (APS): Note: After cross
checking between capped APS and total points as well as records of individual exams provided in
the dataset, it is unclear whether the uncapped APS, capped APS and total points of a pupil
provided in the dataset were calculated based on i) average total point scores of all of the pupil
results at KS4, ii) average total point scores of the pupil’s best 8 results including English and Maths
at KS 4, and iii) total point scores of all of the pupil results at KS4, respectively. Take the minimum
capped APS (see Table 77.) as an example, which were 23.3 suggesting a minimum total points of
186.4 obtained from multiplying 23.3 by 8. However, as shown in Table 77., the minimum total
points were 35.0.
126
Eventually, in addition to the English and Maths exam results, only capped APS was analysed and
reported below.
Table 92 Mean and SD of uncapped and capped average point score (APS)
N Minimum Maximum Mean SD
Uncapped average point score (APS) 164 23.3 53.2 40.1 6.0
Capped average point score (APS) 164 23.3 58.0 43.5 6.8
Total Points 164 35.0 876.0 509.7 167.3
Capped Average Point Score by Gender: An independent t-test was carried to explore whether
there was difference between genders in mean in average point score (APS). The results showed
that there was statistically significant difference in mean APS between genders at α = 0.05 level. On
average males have a lower mean in capped APS (42.22 versus 44.98 in mean) than female
students. However this varied much less around the mean (5.27 versus 8.09 in SD) (see Table 93).
Table 93 Mean and SD of capped APS between gender
Gender N Mean SD
Male 88 42.22 5.27
Female 76 44.98 8.09
t(162)=-2.62, p < 0.05
Capped Average Point Score by FSM: An independent t-test was conducted to compare the mean
APS scores of non-FSM and FSM student groups. There was a statistically significant difference in
mean APS between non-FSM and FSM student groups at α = 0.05 level. On average students who
were not entitled to FSM scored about 3.47 APS higher than students who were entitled to FSM
(44.68 versus 41.21 in mean) (see Table 94).
127
Table 94 Mean and SD of capped APS between non-FSM and FSM student groups
FSM N Mean SD
No_FSM 108 44.68 6.70
Yes_FSM 56 41.21 6.59
t(162)=3.47, p < 0.01
Capped Average Point Score by SEN: An independent t-test was carried to compare the mean APSs
of non-SEN and SEN student groups. There was a statistically significant difference in mean APS
between non-SEN and SEN student groups at α = 0.05 level. The average mean of 6.38 of APS
scored more by non-SEN (46.96 in mean) than SEN (40.58 in mean) groups (see Table 95).
Table 95 Mean and SD of capped APS between No_SEN and Yes_SEN student groups
SEN N Mean SD
No_SEN 75 46.96 6.49
Yes_SEN 89 40.58 5.71
t(162)=6.70, p < 0.01
English: Only 3% of students did not the take English exam (see Table 96).
Table 96 Frequency of English results of
Academy 3 students
Grade Frequency %
NoSitIn 5 3.0
A 3 1.8
B 20 12.0
128
C 54 32.5
D 26 15.7
E 33 19.9
F 20 12.0
G 5 3.0
Total 166 100.0
GCSE English by Gender: A small proportion (3.4%) of male, but no female students achieved grade
A. As shown in Figure below, a greater percentage of male than female students achieved grade C
or lower. In contrast, there was considerably higher percentage of female students who achieved
grade B than male students. Further, it is not clear whether English exams taken by Academy 2 and
Academy 3 were the same GCSE exam. However, based on both scoring systems provided (which
were ‘grading’), it seems that a greater percentage of Academy 3 than Academy 2 students
achieved grade C or above.
Table 97 Academy 3 English results
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
English (sch_3)
Male
Female
129
English by FSM: Among the students that took the English exam, a higher percentages of non-FSM
than FSM students achieved grade A, B or C. In contrast, more FSM than non-FSM students
achieved grade E, F, or G.
Table 98 Academy 3 English results by FSM
Maths: In Academy 3, 97.6% students took the Maths exam (see Table 99).
Table 99 Academy 3 Frequency of grades
Grade Frequency %
NoSitIn 4 2.4
A* 3 1.8
A 11 6.6
B 31 18.7
C 58 34.9
D 12 7.2
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA A B C D E F G U
Per
cen
tage
wit
hin
FSM
English (Sch_3)
No_FSM
Yes_FSM
130
E 10 6.0
F 16 9.6
G 21 12.7
Total 166 100.0
Maths by Gender
Table 100 Academy 3 Maths results
Maths by FSM
Table 101 Academy 3 Maths results by FSM
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
A* A B C D E F G U
Per
cen
tage
wit
hin
Ge
nd
er
Maths (sch_3)
Male
Female
.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
A* A B C D E F G U
Per
cen
tage
wit
hin
FSM
Maths (Sch_3)
No_FSM
Yes_FSM
131
DIFFERENCES BETWEEN ACADEMIES
Note that only the results of English and Maths exams were compared between the two
Academies.
Table 102 Academy 2 and 3 GCSE English/English results
Table 103 Academy 2 and 3 GCSE Maths/Maths results
SUMMARY AND FINDINGS FROM STRANDS EIGHT & NINE
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
AA/A* A B C D E F G U
Per
cen
tage
GCSE English / English
Sch_2
Sch_3
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
AA/A* A B C D E F G U
Per
cen
tage
GCSE Mathematics / Maths
Sch_2
Sch_3
132
These Strands used data from Key Performance Indicators, such as attendance, free school meals
(FSM, exclusions, special educational needs (SEND), gender and GCSE from Academy 2 and
Academy 3 for differences between and within Academies with regards to gender, FSM and SEN in
relation to GCSE results in English, Maths and Science.
Generally, in Academy 2, male students performed poorer in GCSE English and Double Science but
outperformed female students in terms of percentages at Grade C or above in Maths. A similar
trend applied to Academy 3 male and female students in GCSE English and Math exams. However,
overall in Academy 3, female students had higher mean scores in Capped APS than male students.
In both Academies, disadvantaged (ie FSM students) and SEN students were underperforming than
their counterpart students.
Comparing the two Academies in English and Maths exam results, Academy 2 students had
performed slightly better in Maths but underperformed in English in comparison to Academy 3
students. (Note: it needs to confirm that the marking systems used in those English Exams taken by
the two Academy students were the same.) Regarding gender difference, a higher percentage of
male students than female students in Academy 2, but similar percentages of male and female
students in Academy 3 received Grade A- C in the Maths exam.
133
134
HIEARCHICAL PROCESS MODEL
Hierarchical Process Modelling (HPM) was used in addition to the traditional social science methods
of data analysis to each of the Strands, using the Perimeta software as described earlier. The
findings are presented below beginning with a summary of the findings for that Strand from the
social science methods.
STRAND ONE - DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7
SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS
Strand One was designed to assess Year 7 students’ self-reporting of their learning dispositions,
using the Effective Lifelong Learning Inventory's seven scales.
The findings from the social science method demonstrated there was a significant increase in mean
score between Time 1 and Time 2 on six of the seven dimensions of learning power, based on the
Effective Lifelong Learning Inventory, which means that something other than chance must account
for this increase. The increase was not uniform across all three Academies where Academy 1 and 3
were more similar in terms of their statistical significance in many of the dimensions of learning.
This seems to suggest that whilst there are significant increases in scores for all the Academies
overall, except for Resilience, there are other factors which contributed to the differing levels of
significance in the seven dimensions between Academies. This could be due to a range of factors
such as the students themselves, the delivery of the interventions or the overall leadership of the
Academy and are worth exploring further.
PERIMETA ANALYSIS
A total of 1918 data points (for 274 respondents x 7 statements) were entered in the Strand 1 Perimeta model and verified through statistical comparison with the source data.
Performance ratings representing evidence of self-perception of learning power were calculated
from the performance scores assuming a relatively high level of confidence of 75% in the responses,
on the basis that ELLI is a research-validated questionnaire.
The Perimeta model of Strand 1 performance was used to propagate the evidence of Responses
and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender
and by Question). For ease of computation the Strand 1 model was configured in four parts, for
Male and Female students and Times 1 and 2.
135
Table 145 presents a full list of results for Strand 1 covering performance Overall, by Gender, by
Academy and by Question. Results for male and female students are tabulated separately.
Table 104 presents dashboard summaries of the full results of the Perimeta analysis of the
experience of male and female students at the beginning (Time 1) and end (Time 2) of Year 7.
Table 104 Strand 1 - Results dashboards
Male students, Time 1 Male students, Time 2
Female students, Time 1 Female students, Time 2
Strand 1 performance analysis by Question
Table 105 illustrates the distribution of all responses at both Time 1 and Time 2 for Strand 1
according to the strength of evidence of self-perception of learning power. The horizontal axis
represents positive evidence (POS) and the vertical axis represents negative evidence (NEG), so that
the more uncertain (UNC) responses would appear towards the lower left of the graph. This
indicates both the very wide range of strengths of perception in dimensions of learning (10 to 80
percent positive) and the relatively steady degree of uncertainty (20 to 30 percent). The results
show a bias towards positive perceptions, with average positive evidence (57%) outweighing
average uncertainty (27%) and average negative evidence (16%).
136
Table 105 Strand 1 – Distribution of responses
The strongest responses in terms of strength of evidence in the statements are identified in Table
106.
Table 106 Strand 1 – Strongest responses
Gender Strongest positive evidence Strongest negative evidence
Time 1 Time 2 Time 1 Time 2
Male students S1 6 Meaning Making
67%
(+ 2 over 50%)
S1 7 Critical Curiosity
80%
(+ 5 over 50%)
S1 5 Creativity 29%
(+ 2 over 25%)
S1 4 Resilience 26%
(+ no others > 25%)
Female students S1 1 Changing &
Learning 82%
(+2 others > 50%)
S1 1 Changing &
Learning 85%
(+5 others > 50%)
S1 4 Resilience 50%
(+ no others > 25%)
S1 4 Resilience 62%
(+ no others > 25%)
Table 107 summarises the change in student perceptions between the start and end of Year 7. This
shows that students’ positive self-perceptions of their learning power increased (and/or negative
perceptions decreased) during Year 7 for five of the seven ELLI dimensions, but did not change for
Changing & Learning and actually fell for Resilience. The generally modest changes in uncertainty
mirrored this pattern, decreasing or remaining the same over the year but increasing for Resilience.
137
Table 107 Strand 1 – Changes in performance over Year 7 by Question and Gender
Q Statement Males Females
POS UNC NEG POS UNC NEG
All All + -- -- ++ -- --
S1 1 Changing & Learning l l l l l l
S1 2 Learning Relationships + l . + - -
S1 3 Strategic Awareness ++ - -- + l -
S1 4 Resilience - + l - - +
S1 5 Creativity ++ - - - ++ - - - -
S1 6 Meaning Making + - - ++ - - - -
S1 7 Critical Curiosity + - -- ++ - - - -
Key: ++ substantial increase; + increase; l similar; – decrease; - - substantial decrease
Strand 1 performance analysis by Gender
Differences between the perceptions of male and female students are also evident in Table 66. In
general female students were more positive than males that their learning power increased (and
less negative). With regard to Resilience, where both genders perceive a worsening of learning
power, females were less uncertain while males are more uncertain, and females were more
negative while males were unchanged.
Strand 1 performance analysis by Academy
Performance by Academy differed substantially at Time 1 in terms of evidence of self-perception of
learning power, with Academy 3 (53% positive for males and 44% for females) outperforming
Academy 1 (32% and 36%) and Academy 2 (24% and 26%). At Time 2 the Academies are much
closer in performance but with Academy 1 on top (87% and 76%) of Academy 3 (80% and 80%) and
Academy 2 (80% and 63%). Therefore there were substantial improvements in positive self-
perceptions of learning power at each Academy, with the most marked improvements at Academy
1 (55% and 40%) and Academy 2 (56% and 37%) ahead of Academy 3 (27% and 36%).
Results obtained using the Perimeta model of Strand 1 are compared in Table 108 with the results
of standard statistical tests provided by the ECHO project team.
Table 108 Strand 1 – Comparison of results from Perimeta and statistical tests
Perimeta results Statistical test results Notes
Positive self-perception of learning
power:
increased for five dimensions;
did not change for Changing &
Learning;
decreased for Resilience
Learning power (using T-test):
mean scores increased significantly
for six dimensions;
decreased for Resilience
Perimeta results broadly consistent
with T-test means
Uncertainty in learning power
decreased or remained the same
for six dimensions
increased for Resilience
Standard deviation of scores
decreased a little or no change
Perimeta uncertainties and
statistical standard deviations not
related
Positive self-perception of learning Performance by Academy (using Perimeta results apparently the
reverse of the ANOVA results,
138
power by Academy
substantial differences between
Academies at Time 1
much less difference at Time 2
ANOVA):
no statistically significant
differences between
Academies at Time 1
significant differences at Time 2.
revealing deeper insight into
positive perceptions
Academy 1 improved the most out
of the three Academies
Academy 1 improved the most out
of the three Academies
Full agreement between Perimeta
and social science analyses
STRAND TWO - STUDENT PERCEPTIONS OF THE IMPACT OF THE EDUCATION CHARTER
SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS
For Strand Two the questionnaire was designed to assess students' perceptions about their
learning, teaching, progress and achievement in an Oasis Academy.
The findings analysed with traditional social science methods showed there to be no mean
differences in agreement or disagreement for 10 of the 42 items assessing student perceptions on
their learning, teaching, progress and achievement in an Oasis Academy. However, Academy 1
student respondents reported averagely higher mean responses than their counterparts of
Academy 3 to the remaining 32 items which suggests that, to some degree, students in Academy 1
tend to agree more than Academy 3 students.
With regards to gender differences, the statistical results indicated that male and female student
respondents on average provided similar responses to 20 out of 42 items, although, across the
Academies female student respondents reported significantly higher means than male student
respondents on the remaining 22 items. Thus, there appear to be gender differences on over half
of the items, possibly suggesting a difference in how males and females perceive at least some of
their learning, teaching, progress and achievement.
PERIMETA ANALYSIS
A total of 15792 data points (for 376 respondents x 42 statements) were entered in the Perimeta model for Strand 1 and verified through statistical comparison with the source data.
Performance ratings representing evidence of strength of self-perceptions of learning were
calculated from the performance scores assuming a level of confidence of 50% in the responses.
The Perimeta model of Strand 2 performance was used to propagate the evidence of Responses
and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender
and by Question).
139
Table 146 (Appendix 5) Strand 2 presents a full list of results covering performance Overall, by
Gender, by Academy and by Question. Results for male and female students are tabulated
separately.
Table 109 presents dashboard summaries of the full results of the Perimeta analysis of the
experience of male and female students.
Table 109 Strand 2 - Results dashboards
Male students Female students
Strand 2 performance analysis by Question
Table 110 illustrates the distribution of all responses for Strand 2 according to the strength of
evidence of self-perception of learning. The horizontal axis represents positive evidence (POS) and
the vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses
would appear towards the lower left of the graph. This indicates both the very wide range of
strengths of perceptions (10% to 100%) and in the degree of uncertainty (zero to 60%). The results
show a very strong bias towards positive perceptions, with average positive evidence (73%)
outweighing average uncertainty (20%) and average negative evidence (7%).
Table 110 Strand 2 – Distribution of responses
140
Strand 2 performance analysis by Gender
Table 111 shows a substantial difference overall between the perceptions of male and female
students. For all questions, the combined evidence of positive self-perceptions for males was 86%
and for females only 39%. Both genders show similarly low levels of evidence of negative self-
perceptions of learning. They differ greatly in their uncertainty with males at 13% and females at
55%.
Table 112 compares the ‘top ten’ responses of male and female students, showing in bold font the
common ground between them. Whilst male and female students agreed strongly with many
statements (with almost complete agreement on some), there were only two statements in
common in their top ten positive ratings. Uncertainty in responses was also similar between male
and female students, but again there were few (three) statements in common.
It is in regard to disagreement with statements that there is the greatest difference between male
and female students in terms of scores, and at the same time the greatest similarity in their top ten
statements with six of them in common. It is noted that few statements in either list were
especially strongly disagreed with.
A high degree of correlation between the top ten lists for positive evidence, uncertainty and
negative evidence of self-perceptions of learning. For male students there were seven statements
(six for females) to which responses were both uncertain and negative.
Table 111 Strand 2- Comparison of most substantial responses by male and female students
Top 10 statements by evidence of positive self-perceptions of learning (POS) Male students Female students
Q Statement POS Q Statement POS
S2 9 I am confident in my learning 0.99 S2 4 I like learning new things 1.00
S2 13 I don’t mind making mistakes because I learn from them 0.96
S2 35 We are encouraged to care about the needs of other people in the local community and around the world
1.00
S2 25 My teachers constantly expect me to improve on my personal best 0.96
S2 9 I am confident in my learning 0.99
S2 21 I get on well with my teachers 0.96 S2 2 I feel safe in the Academy 0.99
S2 32 I feel proud about what I have achieved so far 0.96
S2 40 My parents or carers have always been able to sort out any problems that I have had
0.98
S2 16 My teachers teach well so that I learn successfully 0.96
S2 23 I know that the senior teachers want us to learn successfully
0.97
S2 17 I have regular opportunities to express my opinion to my teachers about my learning 0.96
S2 37 My parents or carers feel that they are always welcome in the Academy
0.95
S2 38 My parents or carers feel involved with my learning and my progress 0.96
S2 25 My teachers constantly expect me to improve on my personal best
0.94
S2 18 I often have conversations with my teachers which help me to make better progress 0.96
S2 33 We are taught about the responsibilities of students as well as their rights
0.93
S2 31 I feel that my successes with activities outside the Academy are recognised 0.91
S2 20 My teachers often share things that they have learned themselves
0.91
Table 112 Strand 2 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)
Male students Female students
Q Statement UNC Q Statement UNC
S2 2 I feel safe in the Academy 0.60 S2 31 I feel that my successes with activities outside
the Academy are recognised 0.60
S2 36 I regularly offer to help others 0.37 S2 7 I like working with other students to help my
learning 0.56
141
S2 41 I am encouraged to lead activities in the classroom 0.35
S2 42 I have had opportunities to lead other activities at the Academy
0.55
S2 8 I don’t distract other students 0.34 S2 14 I come up with new ideas to help my learning 0.47
S2 14 I come up with new ideas to help my learning 0.33
S2 41 I am encouraged to lead activities in the classroom
0.36
S2 22 I get extra support to help my learninAcademy 1g when I need it 0.33
S2 19 My teachers seem to enjoy teaching my classes 0.36
S2 30 I feel that my successes at the Academy are recognised 0.33
S2 18 I often have conversations with my teachers which help me to make better progress
0.35
S2 6 I am hard to distract 0.33 S2 8 I don’t distract other students 0.35
S2 40
My parents or carers have always been able to sort out any problems that I have had 0.33
S2 17 I have regular opportunities to express my opinion to my teachers about my learning
0.34
S2 24
Everyone works very hard to make the Academy a great place to learn and to do well 0.33
S2 3 I feel valued and cared for as an individual in the Academy
0.33
Table 113 Strand 2 - Top 10 statements by evidence of negative self-perceptions of learning (NEG)
Male students Female students
Q Statement NEG Q Statement NEG
S2 8 I don’t distract other students 0.28 S2 22 I get extra support to help my learning when
I need it 0.70
S2 14 I come up with new ideas to help my learning 0.17
S2 6 I am hard to distract 0.52
S2 22 I get extra support to help my learning when I need it 0.17
S2 18 I often have conversations with my teachers which help me to make better progress
0.48
S2 30 I feel that my successes at the Academy are recognised 0.16
S2 3 I feel valued and cared for as an individual in the Academy
0.42
S2 6 I am hard to distract 0.15 S2 14 I come up with new ideas to help my
learning 0.26
S2 40
My parents or carers have always been able to sort out any problems that I have had 0.15
S2 17 I have regular opportunities to express my opinion to my teachers about my learning
0.20
S2 24
Everyone works very hard to make the Academy a great place to learn and to do well 0.15
S2 41 I am encouraged to lead activities in the classroom
0.19
S2 1 I feel included in the activities that are available to me in the Academy 0.15
S2 19 My teachers seem to enjoy teaching my classes 0.18
S2 29 I think that my results at the moment are as good as I can do 0.15
S2 8 I don’t distract other students 0.17
S2 41 I am encouraged to lead activities in the classroom 0.10
S2 1 I feel included in the activities that are available to me in the Academy
0.13
Strand 2 performance analysis by Academy
Table 146 (Appendix 2) shows substantial differences between the two Academies taking part in
Strand 2 in terms of the positive outlook of for female students (Academy 1 84% positive, Academy
3 39%) and the uncertainty of females students (Academy 1 14% uncertain, Academy 3 55%). For
male students the difference between Academies is less marked.
Results obtained using the Perimeta model of Strand 2 are compared in Table 114 with the results
of standard statistical tests provided by the ECHO project team.
Table 114 Strand 2 – Comparison of results from Perimeta and statistical tests
Perimeta results Statistical test results Notes
Academy 1 similar to Academy 3 in
evidence of positive self-perception
of learning (male students)
10 of 42 statements with
significantly higher mean responses
for Academy 1, with remaining 32
statements similar in response
Perimeta analysis broadly
consistent with statistical analysis
142
Academy 1 substantially more
positive than Academy 3 in self-
perception of learning (female
students)
22 of 42 statements with
significantly higher mean responses
for female students
Perimeta analysis broadly
consistent with statistical analysis
Evidence of strong bias towards
positive perceptions
Means of scores – none above 3.5
and only one third above 3
Perimeta analysis reveals deeper
insight into positive perceptions
Evidence of negative self-
perception is generally low but with
some anomalies
Means of scores all but one above
2.5
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
negative perceptions
Evidence of substantial uncertainty
in many areas of questionnaire
Standard deviation of scores from
questionnaire not excessive
Perimeta uncertainties statistical
standard deviations not related
STRAND THREE - STORIES OF TRANSFORMATION
Strand three was a narrative analysis of a sample of ten students' stories about their experience of
the Academy. It was analysed thematically for evidence of deep, transformative learning and each
case was ranked on a scale of one (low) to four (high).
A full analysis of this small dataset using Perimeta was not meaningful, for the equivalent of one
Question and one Academy and with few students of either Gender. The following Table illustrates
these results in the form of Italian Flag graphics.
Table 115 Strand 3 – Results in Italian flag format
Student Gender Combined Score Italian Flag representation
1 Male 4
2 Female 3
3 Male 3
4 Female 2
5 Female 3
6 Female 2
7 Female 3
8 Male 3
9 Female 2
10 Male 1
11 Female 2
12 Female 2
13 Female 3
143
STRAND FOUR: POST-16 TRANSITION AND PROGRESS TO ADULTHOOD
SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS
This Strand sought to explore the experiences of leavers from two of the Academies and their
expectations of the transition to the next stage of their education.
When comparing Academies using the traditional social science methods, there were no statistical
differences between the two Academies in 24 out of 30 items indicating the student respondents
had similar views on these statements. However, Academy 1 had lower means in the remaining six
items than Academy 3, where on average, Academy 3 respondents tended to report a greater
degree of agreement on enjoying their time at the Academy, suggesting that Academy 3 students
overall rating their Academy more highly than Academy 1 students, although this was just on 6 of
the 30 items.
There were no statistically significant gender differences in any of the Academies on how students
reported their experiences of the Academy or their expectations of the transition. There were also
no statistically significant differences between students receiving FSM compared to those with no
FSM, suggesting that neither gender nor socioeconomic background was a particular factor
impacting on students’ experiences of their Academies.
PERIMETA ANALYSIS
A total of 4140 data points (for 138 respondents x 30 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.
Performance ratings representing evidence of self-perception of learning were calculated from the
performance scores assuming a level of confidence of 50% in the responses.
The Perimeta model of Strand 4 performance was used to propagate the evidence of Responses
and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender
and by Question).
Table 146 (Appendix 5) presents a full list of results for Strand 2 covering performance Overall, by
Gender, by Academy and by Question.
Table 115 presents dashboard summaries of the full results of the Perimeta analysis of the
experience of students.
144
Table 116 Strand 4 - Results dashboard
Strand 4 performance analysis by Question
Table 116 illustrates the distribution of all responses for Strand 4 according to strength of evidence
of self-perception of learning. The horizontal axis represents positive evidence (POS) and the
vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses
would appear towards the lower left of the graph. This indicates both the wide range of strengths
of perceptions (50% to 100%) and in the degree of uncertainty (zero to 40%). The results show a
very strong bias towards positive perceptions, with average positive evidence (84%) outweighing
average uncertainty (14%) and average negative evidence (2%).
Table 117 Strand 4 – Distribution of responses
145
Table 118 Strand 4 - Comparison of most substantial responses
Top 10 statements by evidence of positive self-perceptions of learning (POS)
Q Statement POS
S4 10 I was helped and supported whenever I struggled with my learning 0.98
S4 16 My teachers taught me well at the Academy 0.90
S4 27 The Academy made good use of new technologies for learning 0.90
S4 1 I was valued and included during my time at the Academy 0.89
S4 18 I was regularly challenged to do better so that I could reach my full potential 0.89
S4 24 There were great facilities for learning 0.89
S4 2 I felt part of a community that was focused on learning 0.89
S4 11 I developed a sense of responsibility towards other people during my time at the Academy 0.89
S4 26 I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum 0.89
S4 3 I felt safe at the Academy 0.89
Table 119 Strand 4 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)
Q Statement UNC
S4 13 I supported activities that would improve things for other people 0.38
S4 15 I led activities during my time at the Academy 0.34
S4 7 I am looking forward to the move to Sixth Form, Further Education, training or work 0.33
S4 19 My parents/carers felt involved with my education at the Academy 0.15
S4 22 Everyone at the Academy worked very hard to make it a great place to learn and to do well 0.15
S4 4 I enjoyed my time at the Academy 0.15
S4 30 I could not have gone to a better place for my secondary education 0.14
S4 14 I was offered opportunities for leadership during my time at the Academy 0.14
S4 28 I never felt that anyone was putting limits on what I could achieve 0.13
S4 6 My teachers accurately predicted my GCSE results 0.13
Table 120 Strand 4 - Top 10 statements by evidence of negative self-perceptions of learning (NEG)
Q Statement NEG
S4 7 I am looking forward to the move to Sixth Form, Further Education, training or work 0.18
S4 15 I led activities during my time at the Academy 0.17
No others above 10%
Strand 4 performance analysis by Gender
Table 147 (Appendix 5) shows a substantial difference between genders. Male students were much
more positive (89%) than females (36%) in their self-perceptions of learning. However, females
were not so much more negative but rather more uncertain (56%) than males (11%).
Strand 4 performance analysis by Academy
Table 147 shows that there was no substantial difference between the results obtained from each
of the two participating Academies. Academy 3 students were slightly more positive (94%) than
those of Academy 1 (85%) and slightly less uncertain overall.
Results obtained using the Perimeta model of Strand 4 are compared in Table 120 with the results
of standard statistical tests provided by the ECHO project team.
Table 121 Strand 4 – Comparison of results from Perimeta and statistical tests
Perimeta results Statistical test results Notes
No substantial difference between
Academies in evidence of positive
self-perception of learning
6 of 30 statements with significantly
higher responses for Academy 3,
with remaining 24 statements
Perimeta analysis broadly
consistent with statistical analysis
146
similar in response
Substantial difference between
male and female students in
evidence of positive self-perception
of learning
No significant difference between
responses of male and female
students
Perimeta analysis different from
statistical analysis, and reveals
deeper insight into positive
perceptions
Evidence of strong bias towards
positive perceptions
Means of scores – none above 3.5
but 80% above 3
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
positive perceptions
Evidence of negative self-
perception is low
Means of scores above 2.5 for all
statements
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
negative perceptions
Evidence of substantial uncertainty
in many areas of questionnaire
Standard deviation of scores from
questionnaire not excessive
Perimeta uncertainties statistical
standard deviations not related
STRAND FIVE: QUALITY OF RELATIONSHIPS BETWEEN PARENTS/CARERS AND THE
ACADEMY
SUMMARY FROM SOCIAL SCIENCE DATA ANALYSIS
This Strand explored how well parents and carers understood and experienced the ECHO charter’s
values in practice.
The findings analysed using traditional social science methods showed that a third of the
statements within the 30 item 2-part questionnaire were generally disagreed with. Many of these
statements seemed to refer to issues of developing attitudes to help others or to take opportunities
to lead activities or referred to behaviour or a healthy lifestyle. However, these findings also
showed these same statements to be among those in which the parents/carers had least
consensus. This suggests that whilst there are general disagreements about certain aspects of how
parents/carers experienced the Oasis Charter, these disagreements do not all concord with one
another and could reflect the diversity of experiences due to the parents’ expectations or
perceptions. These could be affected by other factors such as culture or language, however data on
English as an additional language was missing so such assumptions could be problematic.
PERIMETA ANALYSIS
A total of 1728 data points (72 respondents x 24 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.
147
Performance ratings representing evidence of evidence of self-perception of learning power were
calculated from the performance scores assuming a level of confidence of 50% in the responses.
The Perimeta model of Strand 5 performance was used to propagate the evidence of Responses
and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender
and by Question).
Table 148 (Appendix 5) presents a full list of results for Strand 5 covering performance Overall, by
Academy and by Question. Table 121 presents dashboard summaries of the full results of the
Perimeta analysis of the experience of students.
Table 122 Strand 5 - Results dashboard
Strand 5 performance analysis by Question
Table 122 illustrates the distribution of all responses for Strand 5 according to the strength of
evidence of self-perception of learning. The horizontal axis represents positive evidence (POS) and
the vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses
would appear towards the lower left of the graph. This indicates both the very wide range of
strengths of perceptions (20% to 100%) and in the degree of uncertainty (10% to 60%). The results
show a bias towards positive perceptions, with average positive evidence (52%) outweighing
average uncertainty (40%) which is a substantial factor in these responses.
148
Table 123 Strand 5 – Distribution of responses
Table 124 Strand 5 - Comparison of most substantial responses
Strand 5 - Top 10 statements by evidence of positive self-perceptions of learning (POS)
Q Statement BiS
S5 1 My child is known and valued as an individual 0.87
S5 5 My child is taught well 0.59
S5 6 My child is being challenged to improve on previous best performance in all areas 0.59
S5 8 My child’s successes are recognised 0.59
S5 9 The Academy ensures my child is well looked after 0.59
S5 15 I get regular and helpful information about my child’s progress 0.59
S5 20 The Academy is led and managed effectively 0.59
S5 21 My impression is that senior staff focus their work on making the Academy a good place to learn 0.59
S5 22 My impression is that senior staff expect all students to achieve well 0.59
S5 19 The Academy helps me to support my child’s learning 0.59
Table 125 Strand 5 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)
Q Statement UiS
S5 23 My child enjoys their education at the Academy 0.59
S5 2 The Academy is providing for my child’s particular needs 0.57
S5 4 The Academy focuses its work on providing the best opportunities for learning for all students 0.57
S5 10 The Academy helps my child to have a healthy lifestyle 0.57
S5 3 My child feels included in the life of the Academy, in classrooms and in other activities 0.55
S5 14 My child has kept up a good rate of progress at key transitions (for example, from primary to secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth Form/College; from Year 13 to further education/higher education/employment)
0.55
S5 13 My child is taking opportunities to lead activities (for example, leading group work in the classroom, leading a sports team, giving feedback to teachers about their learning, being a representative on a Student Council, taking a lead role in Academy performances of music, drama or dance, being a coach, mentor or ‘buddy’)
0.43
S5 6 My child is being challenged to improve on previous best performance in all areas 0.37
S5 8 My child’s successes are recognised 0.37
149
S5 20 The Academy is led and managed effectively 0.37
Table 126 Strand 5 - Top 10 statements by evidence of negative self-perceptions of learning (NEG)
Q Statement DiS
S5 13 My child is taking opportunities to lead activities (for example, leading group work in the classroom, leading a sports team, giving feedback to teachers about their learning, being a representative on a Student Council, taking a lead role in Academy performances of music, drama or dance, being a coach, mentor or ‘buddy’)
0.33
S5 7 My child’s progress at the Academy has exceeded our expectations 0.27
S5 11 There is a good standard of behaviour at the Academy 0.15
S5 14 My child has kept up a good rate of progress at key transitions (for example, from primary to secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth Form/College; from Year 13 to further education/higher education/employment)
0.10
S5 3 My child feels included in the life of the Academy, in classrooms and in other activities 0.10
S5 24 I would recommend the Academy to another parent/carer 0.08
S5 10 The Academy helps my child to have a healthy lifestyle 0.07
S5 2 The Academy is providing for my child’s particular needs 0.06
S5 4 The Academy focuses its work on providing the best opportunities for learning for all students 0.06
S5 12 My child is developing an attitude of helping others (for example, by doing practical activities to improve life at the Academy, by supporting people in need, getting involved with community projects, fund-raising)
0.06
Results obtained using the Perimeta model of Strand 5 are compared in Table 126 with the results
of standard statistical
Table 127 Strand 5 – Comparison of results from Perimeta and statistical tests
Perimeta results Statistical test results Notes
Evidence of bias towards positive
perceptions
Means of scores – none above 3.5
but majority above 3
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
positive perceptions
Evidence of negative self-
perception of child’s learning is low,
but including uncertainty amounts
to approximately half of evidence
Means of scores above 2.5 for all
statements
One third of statements disagreed
with
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
negative perceptions
Evidence of substantial uncertainty
in many areas of questionnaire
Standard deviation of scores from
questionnaire not excessive
Perimeta uncertainties statistical
standard deviations not related
STRAND SIX: LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF
CLASSROOM PRACTICE
SUMMARY FROM SOCIAL SCIENCE ANALYSIS
This Strand was a three part questionnaire designed to explore teachers’ perceptions of their
Academy and the impact of their professional learning on their classroom practice.
Employing traditional social science methods of data analysis, the first part of the questionnaire
revealed a range of findings with regards to professional learning processes, particularly research
and evaluation and or applying findings of published research to the teachers’ practice. However,
150
teachers agreed on the five most regular processes engaged in by the teacher participants which
referred to improving their teaching and learning practice.
There was, however, a higher degree of variation with regards to the extent with which they felt
involved or engaged in practices which promoted the policy or professional development of the
Academy. But interestingly, teachers tended to show agreement on items which referred to self-
reflection and evaluation or for seeking help for teaching related issues.
The part of the questionnaire which contained items on leadership practices found that the
majority of statements did not frequently apply to practices at the Academy. Much of the findings
from this section seem to suggest that teachers did not feel especially involved in practices which
might promote the staff as part of a whole team, however, staff reported strong commitment to
their departments, key stage or year group.
The findings, nevertheless, did show that teachers felt they were reaching their potential as
teachers and felt presented with opportunities for development with good role models of senior
teachers. What is also interesting is that teachers generally disagreed with the extent to which the
Academy supports parents in the learning of their children and the balancing of
rights/responsibilities within the community. This seemed to concur with the findings of Strand
Five where parents/carers responded in a similar way on items which explored how the Academy
performed as part of a community.
PERIMETA ANALYSIS
A total of 650 data points (for 10 respondents x 65 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.
Performance ratings representing evidence of self-perception of learning were calculated from the
performance scores assuming a level of confidence of 50% in the responses.
The Perimeta model of Strand 6 performance was used to propagate the evidence of Responses
and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender
and by Question).
Table 149 (Appendix 5) presents a full list of results for Strand 6 covering performance Overall, by
Academy and by Question.
Table 127 presents dashboard summaries of the full results of the Perimeta analysis of the
experience of students.
151
Table 128 Strand 6 - Results dashboard
Strand 6 performance analysis by Question
Table 128 illustrates the distribution of all responses for Strand 6 according to the strength of
evidence of self-perception of learning. The horizontal axis represents positive evidence (POS) and
the vertical axis represents negative evidence (NEG), so that the more uncertain (UNC) responses
would appear towards the lower left of the graph. This indicates both the very wide range of
strengths of perceptions (5% to 100%) and in the degree of uncertainty (zero to 60%). The results
show a very strong bias towards positive perceptions, with average positive evidence (66%)
outweighing average uncertainty (23%) and average negative evidence (12%).
152
Table 129 Strand 6 – Distribution of responses
Table 130 Strand 6 - Comparison of most substantial responses
Strand 6 Parts A, B and C - Top 10 statements by positive self-perceptions of learning (POS)
Q Statement POS
S6 A25 My practice is based on the belief that all students are capable of learning 1.00
S6 A10 I experiment with my practice as a conscious strategy for improving classroom teaching and learning 1.00
S6 A1 Staff as well as students learn in this academy 0.92
S6 A11 I modify my practice in the light of feedback from my students 0.89
S6 A14 I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues
0.87
S6 A27 My experience is that student success is celebrated 0.87
S6 A9 I reflect on my practice as a way of identifying my professional learning needs 0.86
S6 A13 I modify my practice in the light of evidence from self-evaluations of my classroom practice 0.86
S6 A7 I am able to see how practices that work in one context might be adapted to other contexts 0.85
S6 A19 If I have a problem with my teaching I turn to colleagues for help 0.85
Q Statement POS
S6 B1 Senior leaders communicate a clear vision of where the academy is going 0.99
S6 B2 Staff have a commitment to the whole academy as well as to their department, key stage and/or year group 0.92
S6 B16 The academy provides cover to allow staff joint planning time 0.90
S6 B6 Teachers’ professional know-how is used in the formulation of academy policy and goals 0.87
S6 B22 Information is collected from teachers on those aspects of their work that they themselves think they do effectively 0.87
S6 B12 Staff have a good working knowledge of the Academy Development Plan (or your academy’s equivalent document) 0.85
153
S6 B18 Formal training provides opportunities for teachers to develop professionally 0.85
S6 B17 Teachers are encouraged to experiment with new ideas as a way of promoting professional growth 0.84
S6 B8 Opportunities are provided for teachers to critically evaluate academy policy 0.64
S6 B14 Staff development time is used effectively to realise Academy Development Plan priorities 0.62
Q Statement POS
S6 C1 I feel included in a community that values what I can offer to the staff team and to students 0.99
S6 C4 I have had opportunities to work alongside and/or train with colleagues in dother phases to improve students’ progress 0.99
S6 C5 My experience is that senior leaders are role models as ‘leaders of learning’ in the academy 0.98
S6 C2 I believe that I am working in a community that focuses our attention and activities on learning to improve teaching and student outcomes 0.95
S6 C9 There is an emphasis among students and staff on caring about and helping other people in the wider community 0.91
S6 C8 There is an emphasis among students and staff on caring about and helping other people in the academy community 0.90
S6 C12 I believe that I am realising my full potential as a teacher 0.90
S6 C6 Leadership is shared in this academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience 0.81
S6 C7 My experience is that the academy promotes a balance between the rights and the responsibilities of all members of the community 0.72
S6 C3 My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes 0.65
Table 131 Strand 6 Parts A, B and C - Top 10 statements by uncertainty in self-perceptions of learning (UNC)
Q Statement UNC
S6 A20 I suggest ideas or approaches for colleagues to try in class 0.55
S6 A18 I engage in team teaching as a way of improving practice 0.49
S6 A2 I draw on good practice from other schools as a means to further my own professional development 0.43
S6 A28 I discuss with colleagues how students might be helped to learn how to learn 0.38
S6 A5 I consult my students about how they learn most effectively 0.37
S6 A22 I discuss openly with colleagues what and how they are learning 0.36
S6 A17 I observe other colleagues in the classroom and we give each other feedback 0.36
S6 A8 I use insights from my own professional learning to feed into academy policy development 0.33
S6 A12 I modify my practice in the light of published research evidence 0.33
S6 A21 I make collective agreements to test out new ideas 0.33
Q Statement UNC
S6 B9 Staff are involved in evaluating academy policy 0.43
S6 B25 Teacher-initiated networking is an integral element of staff development 0.40
S6 B20 Teachers are helped to develop skills to observe learning as it happens in the classroom 0.37
S6 B3 Senior leaders promote commitment among staff to the whole academy as well as to the department, key stage and/or year group 0.37
S6 B13 Staff see the Academy Development Plan as relevant and useful to learning and teaching 0.35
S6 B19 Teachers are helped to develop skills to assess students’ work in ways that move students on in their learning 0.35
S6 B23 Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students 0.35
S6 B10 Staff participate in important decision-making 0.33
S6 B24 Information is collected from teachers on informal teacher networking in which they play an active role 0.33
S6 B8 Opportunities are provided for teachers to critically evaluate academy policy 0.33
Q Statement UNC
S6 C11 The achievements of staff are recognised and celebrated 0.35
S6 C10 The academy is successful in helping parents/carers to support the learning of their children 0.33
S6 C3 My experience suggests that other members of staff are involved in training and development activities to improve their classroom practice and student outcomes 0.25
S6 C7 My experience is that the academy promotes a balance between the rights and the responsibilities of all members of the community 0.21
154
S6 C6 Leadership is shared in this academy and opportunities to take on leadership roles are available to all those that want them and have the relevant knowledge and experience 0.16
No others above 10%
Table 132Strand 6 Parts A, B and C - Top 10 statements by negative self-perceptions of learning (NEG)
Q Statement NEG
S6 A3 I read research reports as one source of useful ideas for improving my practice 0.37
S6 A15 I carry out joint research/evaluation with one or more colleagues as a way of improving my practice 0.36
S6 A2 I draw on good practice from other schools as a means to further my own professional development 0.24
S6 A18 I engage in team teaching as a way of improving practice 0.22
S6 A21 I make collective agreements to test out new ideas 0.16
S6 A23 I use informal opportunities to discuss how children learn 0.16
S6 A12 I modify my practice in the light of published research evidence 0.16
S6 A8 I use insights from my own professional learning to feed into academy policy development 0.14
No others above 10%
Q Statement NEG
S6 B15 Staff development time is used effectively in the academy 0.81
S6 B23 Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students 0.51
S6 B24 Information is collected from teachers on informal teacher networking in which they play an active role 0.45
S6 B3 Senior leaders promote commitment among staff to the whole academy as well as to the department, key stage and/or year group 0.44
S6 B25 Teacher-initiated networking is an integral element of staff development 0.38
S6 B10 Staff participate in important decision-making 0.32
S6 B9 Staff are involved in evaluating academy policy 0.25
S6 B20 Teachers are helped to develop skills to observe learning as it happens in the classroom 0.19
S6 B7 Teachers’ professional know-how is used in the formulation of academy policy, even where this leads to a questioning of established rules, procedures and practices 0.16
S6 B21 Senior leaders support teachers in sharing practice with other schools through networking 0.16
Q Statement NEG
S6 C10 The academy is successful in helping parents/carers to support the learning of their children 0.26
S6 C11 The achievements of staff are recognised and celebrated 0.14
No others above 10%
Results obtained using the Perimeta model of Strand 6 are compared in Table 132 with the results
of standard statistical tests provided by the ECHO project team.
Table 133 Strand 6 – Comparison of results from Perimeta and statistical tests
Perimeta results Statistical test results Notes
Evidence of strong bias towards
positive perceptions
Means of scores above 3.5 for
some statements and above 3 for
over 40% of statements
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
positive perceptions
Evidence of negative perceptions Means of scores below 2.5 for
nearly half of statements
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
negative perceptions
Evidence of substantial uncertainty
in many areas of questionnaire
Standard deviation of scores from
questionnaire not excessive
Perimeta uncertainties statistical
standard deviations not related
155
STRAND SEVEN: IMPACT OF CONTINUED PROFESSIONAL DEVELOPMENT (CPD) ON THE
QUALITY OF LEADERSHIP AND MANAGEMENT AND CLASSROOM PRACTICE
This Strand explored senior leaders’ perceptions from all three Academies and how they are viewed
to be fulfilling the OCL Charter.
The findings from traditional social science analysis methods of the 55 item questionnaire seemed
to suggest that senior management teams (SMT) perceive an unfriendly macro working
environment in terms of relevant governmental and Academy based policy measures in supporting
teaching and learning that focuses on student achievement. However, SMT seemed to
demonstrate a confidence in taking up the responsibility for their professional development.
The findings also implied the issues of fairness and democracy across Academies in relation to
opportunity being created for improvement of teaching and learning through self-reflection,
participation in relevant professional development activities, use and analysis of data, engagement
in external partnerships and involvement of students in decision making.
Principal Components Analysis (PCA) was performed to explore possible underlying patterns in SMT
participants’ responses to the Likert-scale items by reducing the number of items and
dimensionality, where four underlying themes were revealed.
PERIMETA ANALYSIS
A total of 1350 data points (for 25 respondents x 54 statements) were entered in the Perimeta models and verified through statistical comparison with the source data.
Performance ratings representing evidence of self-perception of learning were calculated from the
performance scores assuming a level of confidence of 50% in the responses.
The Perimeta model of Strand 7 performance was used to propagate the evidence of Responses
and to estimate the performance of the system of defined outputs (Overall, by Academy, by Gender
and by Question).
Table 150 presents a full list of results for Strand 7 covering performance Overall, by Academy and
by Question.
Table 133 presents dashboard summaries of the full results of the Perimeta analysis of the
experience of students.
156
Table 134 Strand 7 - Results dashboard
Strand 7 performance analysis by Question
Table 134 illustrates the distribution of all responses for Strand 7 according to evidence of self-
perception of learning. The horizontal axis represents positive evidence (POS) and the vertical axis
represents negative evidence (NEG), so that the more uncertain (UNC) responses would appear
towards the lower left of the graph. This indicates both the wide range of strengths of perceptions
(25% to 100%) and in the degree of uncertainty (zero to 60%). The results show a very strong bias
towards positive perceptions, with average positive evidence (75%) outweighing average
uncertainty (23%) and average negative evidence (3%).
Table 135 Strand 7 – Distribution of responses
157
Table 136 Strand 7 - Comparison of most substantial responses
Strand 7 - Top 10 statements by positive self-perceptions of learning (POS)
Q Statement POS
S7 1 I have a good understanding of my own and other people’s learning 1.00
S7 5 I understand the differences between good and outstanding lessons 1.00
S7 2 I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date 1.00
S7 51 Good personal relationships and strong teams help us to face challenges with confidence 0.98
S7 15 The senior leadership team communicates a clear vision of where the academy is going so that there is a shared sense of purpose
0.90
S7 17 The senior leadership team effectively promotes commitment among staff to the future development of the whole academy
0.90
S7 24 There are processes for involving students in decision-making, including how best to improve classroom practice 0.90
S7 42 We have efficient and effective support systems and structures across the academy 0.90
S7 52 The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing
0.90
S7 53 The senior leadership team is good at long term planning for capacity building (for example, successfully addressing issues like student progress across key transitions; improving the ‘tail’ of student underachievement and/or significant achievement gaps; succession planning)
0.90
Table 137 Strand 7 - Top 10 statements by uncertainty in self-perceptions of learning (UNC)
Q Statement UNC
S7 8 I regularly participate in critical and reflective enquiry to develop teaching and learning across the academy 0.57
S7 12 I regularly interact with staff in other schools, academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our academy
0.48
S7 18 There is effective communication between senior leaders and teachers 0.38
S7 19 There are processes for involving all staff in decision-making 0.38
S7 25 Staff have a good working knowledge of the Academy Development Plan 0.38
S7 47 The academy involves a wide range of other stakeholders directly and powerfully in the education of students 0.38
S7 20 Teachers’ professional know-how is used in the formulation of academy policy and goals 0.38
S7 23 Staff participate in important decision-making 0.38
S7 26 Staff see the Academy Development Plan as relevant and useful to learning and teaching 0.38
S7 28 Staff development time is used effectively in the academy, particularly through the explicit modelling of good practice
0.36
Table 138 Strand 7 - Top 10 statements by negative self-perceptions of learning (NEG)
Q Statement DIS
S7 12 I regularly interact with staff in other schools, academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our academy
0.24
S7 39 ‘Learning how to learn’ is an issue discussed in staff development time 0.13
No others above 10%
Strand 7 performance analysis by Academy
158
Table 150 (Appendix 5) shows only minor differences between the responses of leaders of the three
Academies.
Results obtained using the Perimeta model of Strand 7 are compared in Error! Reference source
not found. 138 with the results of standard statistical tests provided by the ECHO project team.
Table 139 Strand 7 – Comparison of results from Perimeta and statistical tests
Perimeta results Statistical test results Notes
Evidence of self-perception of
learning
Means of scores from questionnaire Perimeta analysis broadly
consistent with statistical analysis
Evidence of strong bias towards
positive perceptions
Means of scores above 3.5 for
relatively high proportion of
statements
Perimeta analysis broadly
consistent with statistical analysis
and reveals deeper insight into
positive perceptions
Some evidence of negative
perceptions
Means of scores all above 2.5
suggesting no disagreement with
statements
Perimeta analysis reveals deeper
insight into negative perceptions
Evidence of substantial uncertainty
in many areas of questionnaire
Standard deviation of scores from
questionnaire not excessive
Perimeta uncertainties statistical
standard deviations not related
STRAND EIGHT AND NINE: CONTEXT FOR STUDENT LEARNING - KEY PERFORMANCE
INDICATORS AND GCSE EXAM RESULTS
Perimeta analysis was not appropriate for Strands 8 and 9 because the data relating to schools and
GCSE results are purely quantitative, with no qualitative element.
Step Four of the ECHO project will explore how Academies can use KPI and GCSE results to deepen
the insights from Perimeta analyses of student self-perceptions of learning in Strands 1, 2, 3 and 4.
TRIANGULATION: HPM AND SOCIAL SCIENCE
The same data were used for analysis by traditional social science methods and for Perimeta
modelling. After both analyses had been completed a process of triangulation was undertaken in
order to check the validity of the findings across the different types of analysis. To do this the
highest scored items from the Perimeta analysis and the descriptive statistics were compared.
Because Perimeta accounted for certainty and uncertainty while the descriptive statistics ranked
the absolute scores, an exact mapping was not excepted. Rather a broad correlation was expected
because the same evidence was used in a radically different measurement model.
For example, Strand 5, item 15 N=135 the Mean = 2.81 Standard Deviation = 0.75, while in the
Perimeta model Positive Certainty =0.272 Uncertainty = 0.348 and Negative Certainty = 0.38
159
Perimeta makes assumptions about uncertainty based on a performance judgment and a
confidence judgment. One assumption we made was that a large degree of uncertainty would
correlate with a large standard deviation. A second assumption was that a ranking of the most
agreed with or disagreed with items would be broadly similar.
Examples from each of the outcomes from each of the Strands was compared on these
assumptions. There was an expected degree of correlation between what was certain/high in all the
data, but less correlation between what was uncertain and the measures of standard deviation.
WHAT CAN WE LEARN FROM THESE TWO APPROACHES
The Perimeta model shows that 'there are a number of people in the middle that we need to get to
know better' and thus it flags up areas for development and further exploration. The Social Science
model presents a more 'black and white' analysis - 39% disagree and 69% agree, whereas the
Perimeta model shows that there are more people we should be concerned about than the 39%.
The real benefits of Perimeta come in the next stage of the process because it is designed to draw
people into exploring and self-managing their own data. It is essentially a decision making tool, not
only a measurement tool. It measures for the purpose of stimulating positive change, according to a
locally defined purpose.
DISCUSSION OF FINDINGS
ECHO STRAND 1 - DEVELOPMENT OF STUDENTS AS LEARNERS IN YEAR 7
The Integrated Approach applied to Strand 1 has revealed key insights into the strength of self-
perception of the learning power of students in Year 7. There were clear differences between the
performances of Academies in terms of the changes their students perceived over the course of the
year. Differences in perceptions between male and female students were insubstantial. The
Resilience dimension of student learning power decreased whilst increases were seen in the other
six ELLI dimensions.
Resilience was consistently the lowest rated dimension, or in the lowest rated group, for male and
female students and Time 1 and Time 2. Resilience is known through experience in the use of the
ELLI framework to be a complex measure of learning power, not directly related to other ELLI
measures. It would not be appropriate for students to be resilient in their attitude to negative
behaviours, hence low or worsening Resilience in Year 7 is not necessarily a bad sign. This aspect of
the design of the Strand 1 research will be reassessed in Phase 2 of the ECHO project.
160
Results representing strength of self-perception of learning power are broadly consistent between
the Integrated Approach and statistical methods. This comparison has helped to build confidence in
both sets of analyses. However, validation of results in the conventional sense proved to be
challenging because the Integrated Approach is not a substitute for more familiar statistical
techniques but rather a complement to them. Detailed reviews with key stakeholders including
members of the ECHO project team and senior Oasis Academy leaders were necessary to build
confidence in the approach and its results, focusing on:
Understanding the theoretical difference between uncertainty and statistical variation;
Recognising the existence of uncertainty in the evidence provided by student responses;
and
Appreciating that insights into uncertainty as well as positive and negative evidence of self-
perception of learning power could enhance the development of improvements in student
experiences in Year 7.
Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative
responses to the Strand 1 questionnaire. Even assuming a relative high level of confidence (75%) in
the evidence, uncertainty was demonstrated to be a substantial factor accounting for around one
third of the evidence obtained, and was not greatly changed over the course of the year.
The Integrated Approach revealed a different perspective from the statistical analysis of the
performance of Academies, showing that they converged in performance terms from substantially
different starting points (whilst the statistical analysis suggested the Academies had diverged from
similar starting points). The finding, backed up by the statistical analysis, that Academy 1 improved
the most of the three, is consistent with Academy 1’s history of involvement with learning
frameworks similar to ELLI. This hypothesis will be tested in Phase 2 of the ECHO project.
Key insights established in Phase 1 of the research (and described above) would provide the basis
for deeper engagement and solution development with students in Phase 2, exploring the reasons
behind responses to the ELLI questionnaire that are exceptionally positive, or negative, or
uncertain. Error! Reference source not found. identifies a number of areas for further investigation
in Phase 2.
In Phase 1 the Integrated Approach has established for Strand 1 the value of additional insights,
complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2
the Perimeta model for Strand 1 will be further enhanced by ‘drilling down’ to the underlying
causes of positive and negative perceptions. This enhancement will be guided by the priority areas
identified in Table 139 and will help to analyse the more detailed underlying data that will be
obtained from the ELLI learning framework.
Table 140 Strand 1 – Areas for further investigation in Phase 2
Q Statement Time Gender Academy
161
Evidence of positive self-perception of learning > 75%
S1 1 Changing & Learning T1, T2 G2 A1, A2, A3
S1 6 Meaning Making T2 G2 A1, A2, A3
S1 5 Creativity T2 G2 A1, A2, A3
S1 7 Critical Curiosity T2 G1, G2 A1, A2, A3
Evidence of negative self-perception of learning > 25%
S1 4 Resilience T1, T2 G1, G2 A1, A2, A3
S1 5 Creativity T1 G1 A1, A2, A3
S1 3 Strategic Awareness T1 G1 A1, A2, A3
Uncertainty in self-perception of learning > 25%
All T1, T2 G1, G2 A1, A2, A3
High degree change from Time 1 to Time 2
S1 3 Strategic Awareness G1 A1, A2, A3
S1 5 Creativity G1, G2 A1, A2, A3
S1 6 Meaning Making G2 A1, A2, A3
S1 7 Critical Curiosity G2 A1, A2, A3
ECHO STRAND 2 - STUDENT PERCEPTIONS OF THE IMPACT OF THE EDUCATION CHARTER
The Integrated Approach applied to Strand 2 has revealed key insights into the self-perception of
students in Years 8 and 10 of their development as learners. There were clear differences between
the two participating Academies and between the genders in the strength of self-perceptions of
learning. The particular statements that students most readily agreed with, or were uncertain
about, were substantially different between males and females.
Results representing strength of self-perception of learning power are broadly consistent between
the Integrated Approach and statistical methods. This comparison has helped to build confidence in
both sets of analyses. However, validation of results in the conventional sense proved to be
challenging because the Integrated Approach is not a substitute for more familiar statistical
techniques but rather a complement to them. Detailed reviews with key stakeholders including
members of the ECHO project team and senior Oasis Academy leaders were necessary to build
confidence in the approach and its results, focusing on:
Understanding the theoretical difference between uncertainty and statistical variation;
Recognising the existence of uncertainty in the evidence provided by student responses;
and
Appreciating that insights into uncertainty as well as positive and negative evidence of self-
perception of learning could enhance the development of improvements in student
experiences in Year 7.
Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative
responses to the Strand 2 questionnaire. Uncertainty was demonstrated to be a substantial factor
accounting for around one fifth of the evidence obtained.
Key insights established in Phase 1 of the research (and described above) would provide the basis
for deeper engagement and solution development with students in Phase 2, exploring the reasons
162
behind responses to the ELLI questionnaire that are exceptionally positive, or negative, or
uncertain. Table 89 identifies a number of areas for further investigation in Phase 2.
In Phase 1 the Integrated Approach has established for Strand 2 the value of additional insights,
complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2
the Perimeta model for Strand 2 will be further enhanced by ‘drilling down’ to the underlying
causes of positive and negative perceptions. This enhancement will be guided by the priority areas
and themes identified in Table 140, and by themes revealed by statistical factor analysis, and will
help to analyse the more detailed underlying data.
Table 141 Strand 2 – Statements for further investigation in Phase 2
Q Statement Gender Academy
Evidence of positive self-perception of learning > 75%
S2 4 I like learning new things G2 A1, A2, A3
S2 35 We are encouraged to care about the needs of other people in the local community and around the world
G2 A1, A2, A3
S2 9 I am confident in my learning G2 A1, A2, A3
S2 2 I feel safe in the Academy G2 A1, A2, A3
S2 9 I am confident in my learning G1 A1, A2, A3
Uncertainty in self-perception of learning > 50%
S2 2 I feel safe in the Academy G1 A1, A2, A3
S2 31 I feel that my successes with activities outside the Academy are recognised G2 A1, A2, A3
S2 7 I like working with other students to help my learning G2 A1, A2, A3
S2 42 I have had opportunities to lead other activities at the Academy G2 A1, A2, A3
Evidence of negative self-perception of learning > 25%
S2 22 I get extra support to help my learning when I need it G2 A1, A2, A3
S2 6 I am hard to distract G2 A1, A2, A3
S2 18 I often have conversations with my teachers which help me to make better progress G2 A1, A2, A3
S2 3 I feel valued and cared for as an individual in the Academy G2 A1, A2, A3
S2 8 I don’t distract other students G1 A1, A2, A3
S2 14 I come up with new ideas to help my learning G2 A1, A2, A3
Strong/ common themes in responses
Student: confidence, learning from mistakes, learning new things, pride, responsibility
Teacher: quality of teaching, expectations, sharing learning
School/ leader: feeling safe
Community/ parent: recognition of activities outside
ECHO STRAND 3 – STORIES OF TRANSFORMATION
A full analysis of this small dataset using Perimeta was not meaningful, for the equivalent of one
Question and one Academy and with few students of either Gender. In Step Four of the ECHO
project, when the pilot study will be widened to many more Academies, the Perimeta analysis is
expected to be highly appropriate as an enabler of insights into the reasons behind the diversity of
students’ stories.
ECHO STRAND 4 - POST-16 TRANSITION AND PROGRESS TO ADULTHOOD
The Integrated Approach applied to Strand 4 has revealed key insights into the self-perception of
learning of students in the post-16 transition and progression to adulthood. There were clear
163
differences between male and female students in the strength of self-perceptions of learning. The
particular statements that students most readily agreed with, or were uncertain about, were
substantially different between males and females.
Results representing strength of self-perception of learning power are broadly consistent between
the Integrated Approach and statistical methods in comparing Academies, but different in
comparing genders. Validation of results in the conventional sense proved to be challenging
because the Integrated Approach is not a substitute for more familiar statistical techniques but
rather a complement to them. Detailed reviews with key stakeholders including members of the
ECHO project team and senior Oasis Academy leaders were necessary to build confidence in the
approach and its results, focusing on:
Understanding the theoretical difference between uncertainty and statistical variation;
Recognising the existence of uncertainty in the evidence provided by student responses;
and
Appreciating that insights into uncertainty as well as positive and negative evidence of self-
perception of learning could enhance the development of improvements in student
experiences in Year 11.
Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative
responses to the Strand 4 questionnaire. Uncertainty was demonstrated to be a substantial factor
accounting for around one seventh of the evidence obtained.
Key insights established in Phase 1 of the research (and described above) would provide the basis
for deeper engagement and solution development with students in Phase 2, exploring the reasons
behind responses to the questionnaire that are exceptionally positive, or negative, or uncertain.
Table 90 identifies a number of areas for further investigation in Phase 2.
In Phase 1 the Integrated Approach has established for Strand 4 the value of additional insights,
complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2
the Perimeta model for Strand 4 will be further enhanced by ‘drilling down’ to the underlying
causes of positive and negative perceptions. This enhancement will be guided by the priority areas
and themes identified in Table 141, and by themes revealed by statistical factor analysis, and will
help to analyse the more detailed underlying data.
Table 142 Strand 4 – Statements for further investigation in Phase 2
Q Statement Gender Academy
Evidence of positive self-perception > 98%
S4 10 I was helped and supported whenever I struggled with my learning G1 & G2 A1, A2, A3
Uncertainty in self-perception of learning > 50%
None None
Evidence of negative self-perception > 25%
none None
Strong/ common themes in responses
Student:
164
Teacher: support, quality of teaching, challenging
School/ leader: valued and included, community, technology, facilities
Community/ parent:
ECHO STRAND 5 – QUALITY OF RELATIONSHIPS BETWEEN PARENTS/CARERS AND THE
ACADEMY
The Integrated Approach applied to Strand 5 has revealed key insights into the self-perception of
parents/ carers of the learning of their children.
Results representing strength of self-perception of learning are broadly consistent between the
Integrated Approach and statistical methods. Validation of results in the conventional sense proved
to be challenging because the Integrated Approach is not a substitute for more familiar statistical
techniques but rather a complement to them. Detailed reviews with key stakeholders including
members of the ECHO project team and senior Oasis Academy leaders were necessary to build
confidence in the approach and its results, focusing on:
Understanding the theoretical difference between uncertainty and statistical variation;
Recognising the existence of uncertainty in the evidence provided by parent/carer
responses; and
Appreciating that insights into uncertainty as well as positive and negative evidence of self-
perception of learning power could enhance the development of improvements in
parent/carer experiences of their children’s learning.
Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative
responses to the Strand 5 questionnaire. Uncertainty was demonstrated to be a substantial factor
accounting for around 40 percent of the evidence obtained.
Key insights established in Phase 1 of the research (and described above) would provide the basis
for deeper engagement and solution development with parents/ carers in Phase 2, exploring the
reasons behind responses to the questionnaire that are exceptionally positive, or negative, or
uncertain. Error! Reference source not found.le 91 identifies a number of areas for further
investigation in Phase 2.
In Phase 1 the Integrated Approach has established for Strand 4 the value of additional insights,
complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2
the Perimeta model for Strand 5 will be further enhanced by ‘drilling down’ to the underlying
causes of positive and negative perceptions. This enhancement will be guided by the priority areas
and themes identified in Table 142 and by themes revealed by statistical factor analysis, and will
help to analyse the more detailed underlying data.
165
Table 143 Strand 5 – Statements for further investigation in Phase 2
Q Statement Gender Academy
Evidence of positive self-perception of learning > 98%
None
Uncertainty in self-perception of learning >50%
S5 23 My child enjoys their education at the Academy
S5 2 The Academy is providing for my child’s particular needs
S5 4 The Academy focuses its work on providing the best opportunities for learning for all students
S5 10 The Academy helps my child to have a healthy lifestyle
S5 3 My child feels included in the life of the Academy, in classrooms and in other activities
S5 14 My child has kept up a good rate of progress at key transitions (for example, from primary to
secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth
Form/College; from Year 13 to further education/higher education/employment)
Evidence of negative self-perception of learning > 25%
S5 13 My child is taking opportunities to lead activities (for example, leading group work in the
classroom, leading a sports team, giving feedback to teachers about their learning, being a
representative on a Student Council, taking a lead role in Academy performances of music,
drama or dance, being a coach, mentor or ‘buddy’)
S5 7 My child’s progress at the Academy has exceeded our expectations
Strong/ common themes in responses
Student:
Teacher: quality of teaching, challenging
School/ leader: valued, recognised, looked after, informed, effective leadership
Community/ parent:
ECHO STRAND 6 - LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF
CLASSROOM PRACTICE
The Integrated Approach applied to Strand 6 has revealed key insights into the self-perception of
learning of teachers.
Validation of results in the conventional sense proved to be challenging because the Integrated
Approach is not a substitute for more familiar statistical techniques but rather a complement to
them. Detailed reviews with key stakeholders including members of the ECHO project team and
senior Oasis Academy leaders were necessary to build confidence in the approach and its results,
focusing on:
Understanding the theoretical difference between uncertainty and statistical variation;
Recognising the existence of uncertainty in the evidence provided by student responses;
and
Appreciating that insights into uncertainty as well as positive and negative evidence of self-
perception of learning could enhance the development of improvements in teacher
experiences.
166
Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative
responses to the Strand 6 questionnaire. Uncertainty was demonstrated to be a substantial factor
accounting for around one quarter of the evidence obtained.
Key insights established in Phase 1 of the research (and described above) would provide the basis
for deeper engagement and solution development with students in Phase 2, exploring the reasons
behind responses to the questionnaire that are exceptionally positive, or negative, or uncertain.
Table 92 identifies a number of areas for further investigation in Phase 2.
In Phase 1 the Integrated Approach has established for Strand 6 the value of additional insights,
complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2
the Perimeta model for Strand 6 will be further enhanced by ‘drilling down’ to the underlying
causes of positive and negative perceptions. This enhancement will be guided by the priority areas
and themes identified in Table 143, and by themes revealed by statistical factor analysis, and will
help to analyse the more detailed underlying data.
Table 144 Strand 6 – Statements for further investigation in Phase 2
Q Statement
Evidence of positive self-perception of learning >98%
S6 A25 My practice is based on the belief that all students are capable of learning
S6 A10 I experiment with my practice as a conscious strategy for improving classroom teaching and learning
S6 B1 Senior leaders communicate a clear vision of where the academy is going
S6 C1 I feel included in a community that values what I can offer to the staff team and to students
S6 C4 I have had opportunities to work alongside and/or train with colleagues in dother phases to improve students’ progress
S6 C5 My experience is that senior leaders are role models as ‘leaders of learning’ in the academy
Uncertainty in self-perception of learning > 25%
S6 A20 I suggest ideas or approaches for colleagues to try in class
Evidence of negative self-perception of learning >25%
S6 B15 Staff development time is used effectively in the academy
S6 B23 Information is collected from teachers on effective ways they promote learning to learn skills and knowledge among their students
Strong/ common themes in responses
Student: teacher learning, reflection, collaboration, belief in student capability,
Teacher: innovation,
School/ leader: celebrating success, support, communication, commitment, engagement, training
Community/ parent:
ECHO STRAND 7 - IMPACT OF CONTINUED PROFESSIONAL DEVELOPMENT (CPD) ON THE
QUALITY OF LEADERSHIP AND MANAGEMENT AND CLASSROOM PRACTICE
The Integrated Approach applied to Strand 7 has revealed key insights into the self-perception of
learning of leaders.
Validation of results in the conventional sense proved to be challenging because the Integrated
Approach is not a substitute for more familiar statistical techniques but rather a complement to
them. Detailed reviews with key stakeholders including members of the ECHO project team and
167
senior Oasis Academy leaders were necessary to build confidence in the approach and its results,
focusing on:
Understanding the theoretical difference between uncertainty and statistical variation;
Recognising the existence of uncertainty in the evidence provided by student responses;
and
Appreciating that insights into uncertainty as well as positive and negative evidence of self-
perception of learning could enhance the development of improvements in teacher
experiences.
Using the Integrated Approach, uncertainty was recognised clearly and explicitly in the qualitative
responses to the Strand 7 questionnaire. Uncertainty was demonstrated to be a substantial factor
accounting for around one quarter of the evidence obtained.
Key insights established in Phase 1 of the research (and described above) would provide the basis
for deeper engagement and solution development with students in Phase 2, exploring the reasons
behind responses to the questionnaire that are exceptionally positive, or negative, or uncertain.
Table 93 identifies a number of areas for further investigation in Phase 2.
In Phase 1 the Integrated Approach has established for Strand 7 the value of additional insights,
complementary and sometimes challenging to those of conventional statistical analysis. In Phase 2
the Perimeta model for Strand 7 will be further enhanced by ‘drilling down’ to the underlying
causes of positive and negative perceptions. This enhancement will be guided by the priority areas
and themes identified in Table 144, and by themes revealed by statistical factor analysis, and will
help to analyse the more detailed underlying data.
Table 145 Strand 7 – Statements for further investigation in Phase 2
Q Statement
Evidence of positive self-perception of learning >98%
S7 1 I have a good understanding of my own and other people’s learning
S7 5 I understand the differences between good and outstanding lessons
S7 2 I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date
S7 51 Good personal relationships and strong teams help us to face challenges with confidence
Uncertainty in self-perception of learning >50%
S7 8 I regularly participate in critical and reflective enquiry to develop teaching and learning across the academy
Evidence of negative self-perception of learning >25%
None
Strong/ common themes in responses
Student: engagement with students
Teacher: engagement with teachers
School/ leader: insight into people and lesson quality, good relationships, dealing with complex challenges, long-term planning
Community/ parents:
PRACTICAL AND TECHNICAL ISSUES FOR DEVELOPMENT
168
The project proved to be more demanding in terms of resource than was anticipated for three
reasons. First it was essentially a proof of concept. The task was to explore whether hierarchical
process modelling and the Perimeta tool in particular would be useful in supporting the evaluation
of the wider outcomes of schooling. For this reason, and secondly, we chose to compare traditional
methods and Perimeta modelling in analysis of the data, and this was productive but time
consuming. Thirdly, the bespoke surveys were designed to fit the HPM model, and data for these
surveys was collected by each school on line on their own intranet service, and then compiled by
the research team into the core data sets. These data were then entered manually into the
Perimeta model. There was a significant attrition rate on data collection because of this.
Any development of this approach to school self evaluation would require significantly more
automation than was possible in this pilot, as well as agreement around a systems design which is
shared with a group of schools, with the possibility of the addition of a bespoke line of enquiry to
suit local school needs. Therefore further applications of this model would not require the
selection and construction of surveys which was necessary in the pilot.
This pilot study has produced surveys which can be significantly and productively reduced in size,
with new variables being computed from scales which have already been validated. This should
reduce the amount of data to be entered into the Perimeta model whilst also ensuring that it has
more power as a measurement tool, rated with a higher confidence score in the translation into the
Italian Flag model.
With advances in technology and learning analytics it is within our power now to create an online
learning environment which both provides survey tools, and knowledge structuring tools, such as
Perimeta. Outputs of surveys could be undertaken by the school community on a planned basis
throughout the year and be automatically entered into the Perimeta model. This then becomes a
leadership decisioning tool available to leaders throughout the academic year, and particularly at
key stages of development planning cycle.
However, this proof of concept has simply indicated that the research question which we set out
with is valuable and worth pursuing. Beyond what is reported in this report is a second phase
where the data and findings are made available to the schools concerned and the Multi-Academy
Trust to explore their value in networked improvement community processes. If, for example,
there is a key issue which emerges about the need for a holistic and system wide approach to
curriculum and pedagogy in order to achieve the purpose of all students taking responsibility for
their own learning, then this complex issue can become a focus for school improvement. Consistent
with the philosophy underlying Perimeta, of responsible self-management, diversity, participation
and responding to emergence, these issues could become a focus for authentic professional
enquiry. A promising model to use for this, which stimulates professional learning and research, is
Design Educational Engineering and Development which has been developed by the Carnegie
Foundation for the Advancement of Teaching (Bryk et al., 2010, Bryk, 2009, Bryk and Gomez, 2008).
In this model a school or community of schools identifies a shared, complex problem and proceeds
169
through a process of rapid prototyping research and evaluation, led by teams of teachers in 24
week cycles. Whilst teachers undertake their own evaluation and analysis of their interventions
which are designed to address the chosen complex issue, the whole system commits to a shared
disciplined evaluation framework. Meanwhile the collective intelligence which emerges from these
prototypes is harnessed by school leadership - and academic critical friends - and shared across the
network and beyond. Clearly this model is consistent with the purpose of Perimeta. A Perimeta
model may also provide a really useful disciplined evaluation framework which feeds into the
overall school self evaluation and performance management.
These ideals are a long way beyond the completion of this proof of concept pilot study. However,
conceptually, practically, professionally and technically there is real promise in this approach.
CONCLUSIONS AND NEXT STEPS
The introduction to this report describes the challenge taken up by the ECHO Project and the
participating academies in trying to find systematic, reliable and sustainable ways of evaluating the
wider outcomes of schools. Although the separate components of the Project, for example student
and parent questionnaires, are not new tools for school self-evaluation, the pilot phase of the ECHO
Project has been an attempt to weave all the evaluation Strands together in order to create a whole
and more balanced snapshot of the school’s performance for improvement planning.
The pilot phase of the ECHO Project between 2010 and 2013 was designed as a ‘proof of concept’
exercise. It is encouraging to note from the previous section of the report that initial analyses
suggest that ‘conceptually, professionally, practically and technically there is real promise in this
approach’. The use of the Perimeta model has been an important feature of the Project as it
enables links to be established between extended self-evaluation, performance management and
school improvement planning.
The need for further research has been highlighted throughout the report and, in particular, the
relationship between quantitative and qualitative perspectives on performance would benefit from
deeper investigation. For example, what might be the relationships between Year 11 leaver
responses to the Strand 4 questionnaire and GCSE results (Strand 9)? Is there a link between the
responses of students to the Strand 1, Strand 2 and Strand 3 questions and attendance figures for
the same students (Strand 8)?
The pilot phase has helped to test, through traditional statistical techniques, the quality of
questions used in the various questionnaires and interview schedules. As a result, it is suggested
that these can be refined and the amount of data that they generate reduced. It has also prompted
the need to develop further, and possibly build a 360-degree perspective into, the evaluation of
teacher learning and leadership learning. These are both relatively new areas for the evaluation of
the impact of learning on the effectiveness of teachers and leaders.
170
The more systematic and broadly based evaluation developed through the ECHO Project has the
potential to enhance the depth and range of information available to senior leadership teams as
they undertake school improvement planning. The ECHO evaluation processes can be tailored to
the context, purposes and needs of individual schools. The results could also generate issues arising
from and shared by a group of schools rather than a single school. This would benefit from a
coordinated approach to improvement by, for example, a multi-academy group or a local authority
and create opportunities to share knowledge across a group of schools through a networked
improvement community.
A unifying idea behind the ECHO Project has been the quality of the relationships between the
learning of leaders, the learning of teachers and the learning of students as, working together, they
seek to secure outstanding outcomes for students and continuous improvement of their schools.
Insights from parents/carers about the learning of their children and the effectiveness of the school
were added to the data that was analysed by statistical methods and the use of Perimeta. Oasis is
currently developing two additional questionnaires for governors and the wider community in
order to extend the base of evaluation evidence and involve all key stakeholders.
Perimeta is a novel data analysis programme in that it enables leaders to identify areas of
uncertainty during the process of self-evaluation as well as aspects of performance that are more
black and white, or red and green to use the Italian flag metaphor. Clearly weaknesses (the red
indicators) need to be addressed but schools will also benefit from exploring areas of uncertainty in
more depth. To quote from a section earlier in the report Perimeta is designed to draw people into
exploring and managing their own data. It is essentially a decision-making tool, not only a
measurement tool. It measures for the purpose of stimulating positive change according to locally
defined need’.
The results of the pilot phase of the ECHO Project have led the core team to the conclusion that
both the concept and the approach are worth pursuing further. The following are
recommendations for the next steps:
1. The results will be shared with the leadership teams and staff of the three academies that
participated in the pilot project; their feedback will help to frame the next phase of research
and development.
2. The questionnaires and interview schedules used in the pilot phase will be refined, put
online and gradually rolled out across the Oasis group of academies; an additional
questionnaire for governors is being piloted during 2013/14 and a questionnaire for the
wider community developed for initial use in 2014.
3. Senior leadership teams in schools and academies, multi-academy groups, local authorities
and all those involved with raising standards in schools will be encouraged to explore the
171
potential of this more systematic and rigorous approach to evaluating the wider outcomes
of schools for strategic planning and improving student outcomes.
4. Further development will be undertaken on the Perimeta model with the intention of
making it directly available to schools.
5. Further research will be commissioned on a number of issues arising from the pilot phase of
the ECHO Project.
6. The ECHO Project is already creating international interest and there will be wider
dissemination of the results.
7. It is expected that additional suggestions for Phase 2 of the ECHO Project will arise as the
results of the pilot phase are shared more widely.
172
BIBLIOGRAPHY AND REFERENCES
Amosa, W., Ladwig, J., Griffiths, T. & Gore, J. 2007. Equity effects of Quality Teaching: Closing the gap. Australian Association for Research in Education conference. Fremantle.
Anderson, C. S. 1982. The Search for School Climate: A Review of the Research. Review of Educational Research, 52, 368-420.
Argyris, C., & Schön, D. A. (1996). Organizational learning. 2. Theory, method, and practice. Addison-Wesley
Assessment Reform Group 1999. Assessment for Learning: Beyond the Black Box, Cambridge, University of Cambridge School of Education.
Bateson, G. 1972. Steps to an Ecology of Mind, San Fancisco,, Chandler.
Baxter Magolda, M. B. 2004. Evolution of a constructivist conceptualization of epistemological reflection. Educational Psychologist, 39, 31-42.
Bazeley, P. (2007). Qualitative analysis with NVivo. London: Sage.
Beare, H. 2001. Creating the Futures School, London, Routledge.
Bereiter, C. & Scardamalia, M. 1989. Intentional learning as a goal of instruction. In: Resnick, L. (ed.) Knowing, Learning and INstruction. Essarys in honour of Robert Glaser. Hillsdale NJ: Lawrence Erlbaum Associates.
Bergin, M. (2011). NVivo 8 and consistency in data analysis: reflecting on the use of a qualitative data analysis program.
Nurse researcher, 18(3), 6–12.
Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional development that works. Educational leadership, 57(8), 28–33.
Black, P., Mccormick, R., James, M. & Pedder, D. 2006. Assessment for learning and learning how to learn: a theoretical enquiry. Research Papers in Education, 21, 119-132.
Blockley, D. 2010. The importance of being process. Civil engineering and environmental systems, 27, 189-199.
Blockley, D. & Godfrey, P. 2000. Doing it Differently, systems for rethinking construction, London, Telford.
Bottery, M. 1990. The Morality of the School, London, Cassell Educational Ltd.
Bottery, M. 1992. The Ethics of Educational Management, London, Cassell Educational Ltd.
Bottery, M. 2004. The Challenges of Educational Leadership, London, Paul Chapman.
Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional development that works. Educational leadership, 57(8), 28–33.
Bore, A., & Wright, N. (2009). The wicked and complex in education: developing a transdisciplinary perspective for policy formulation, implementation and professional practice. Journal of Education for Teaching, 35(3), 241–256.
Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational researcher, 33(8),
173
3–15.
Blockley, D. 2010. The importance of being process. Civil engineering and environmental systems, 27, 189-199.
Blockley, D. & Godfrey, P. 2000. Doing it Differently, systems for rethinking construction, London, Telford.
Bronfenbrenner, U. 1989. Ecological System Theory. Annals of Child Development, 6, 187-249.
Bryk, A. S., Lee, V. E., & Peter, B. (1993). Holland, Catholic Schools and the Common Good. Also, Coulson, Market Education.
Bryk, A. S., Sebring, P. B., Allensworth, E., Easton, J. Q., & Luppescu, S. (2010). Organizing schools for improvement: Lessons from Chicago. University of Chicago Press.
Buckingham Shum & Deakin Crick, R. 2012. Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, BC: ACM Press: NY.
Cano, F. 2005. Epistemological Beliefs and approaches to learning: Their change through secondary school and their influence on academic performance. British Journal of Educational Psychology, 75, 203-221.
Checkland, P., & Scholes, J. (1999). Soft systems methodology: a 30-year retrospective. Soft systems methodology in action. Wiley Chichester.
Clarke, A., & Collins, S. (2007). Complexity science and student teacher supervision. Teaching and Teacher Education, 23(2), 160–172.
Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teaching and teacher education, 18(8), 947–967.
Cochran-Smith, M. (2003). Learning and unlearning: The education of teacher educators. Teaching and teacher education, 19(1), 5–28.
Cohen, D. K., & Hill, H. C. (2001). Learning policy: When state education reform works. Yale University Press.
Collins, S., & Clarke, A. (2008). Activity frames and complexity thinking: Honoring both public and personal agendas in an emergent curriculum. Teaching and Teacher Education, 24(4), 1003–1014.
Crick, R. D. (2010). Integrating the personal with the public: Values, virtues and learning and the challenges of assessment. In International research handbook on values education and student wellbeing (pp. 883–896). Springer.
Crick, R. D. (2012). Deep engagement as a complex system: Identity, learning power and authentic enquiry. In Handbook of research on student engagement (pp. 675–694). Springer.
Cui, W.& Blockley, D.I. Interval Probability Theory for Evidential Support. Int.J.Intelligent Systems 5 183-192. 1990
Curtis, M. J., & Stollar, S. A. (2002). Best Practices in System-Level Change. Retrieved from http://psycnet.apa.org/psycinfo/2006-03715-016
Davies, B. (2011). Leading the strategically focused school: Success and sustainability. Sage Publications Limited.
Davis, J P & Fletcher A. Managing Assets under Uncertainty. Society of Petroleum Engineers Asia Pacific Conference on
Integrated Modelling for Asset Management. Yokohama, Japan. 2000
Darling-Hammond, L. & Bransford, J. 2005. Preparing Teachers for a Changing World: What Teachers Should Learn and be Able to Do, San Francisco, Jossey-Bass.
Davis, B., & Sumara, D. J. (2006). Complexity and education: Inquiries into learning, teaching, and research. Psychology
174
Press.
Davies, B., & West-Burnham, J. (2003). The handbook of educational leadership and management. Pearson Education.
Day, C. and Leith, R. 2007. “The continuing professional development of teachers: Issues of coherence, cohesion and effectiveness”. In International handbook of school effectiveness and improvement, Edited by: Townsend, T. 468–83. Dordrecht, , The Netherlands: Springer.
Deakin Crick, R., Jelfs, H., Ren, K., & Symonds, J. (2010). Learning futures. London: Paul Hamlyn Foundation.
Deakin Crick, R., Grushka, K., Heitmeyer, D., & Nicholson, M. (2010). Learning, Place and Identity: An exploration of the affordances of a pedagogy of place amongst Indignous Australian students. Bristol, UK: University of Bristol.
Deakin Crick, R., Jelfs, H., Huang, S. & Wang, Q. 2011. Learning Futures Final Report. London: Paul Hamlyn Foundation.
Deakin Crick, Ruth. (2009). Pedagogical challenges for personalisation: integrating the personal with the public through context-driven enquiry. The Curriculum Journal, 20(3), 185–189Deakin Crick, R. 2009. Inquiry-based learning: reconciling the personal with the public in a democratic and archaeological pedagogy. Curriculum Journal, 20, 73 - 92.
Dempster, A.P. Upper and lower probability inference for families of hypotheses with monotone density ratios. Annals
of Math. Statist. Vol.40, No.3, pp. 953-969, 1969.
Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational evaluation and policy analysis, 24(2), 81–112.
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of health services research & policy, 10(1), 45–53B.
Dweck, C. S. 2000. Self-Theories: Their role in Motivation, Personality, and Development, New York, Psychology Press.
Eccles, J., Wigfield, A. & Schiefele, U. 1998. Handbook of Child Psychology,.
Fredricks, J., Blumenfeld, P. & Paris, A. 2004. School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research74, 59-109.
Freire, P. 1972. Pedagogy of the Oppressed, Harmondsworth, Penguin.
Fullarton, S. 2002. Student engagement with school: individual and school-level influences. LSAY Research Report Melbourne: Australian Council for Educational Research.
Gale, T., Tranter, D., Bills, D., Hattam, R. & Comber, B. 2008. Interventions early in school as a means to improve higher education outcomes for disadvantaged (particularly low SES) students: review of the Australian and international literature. Canberra: The Department of Education, Employment and Workplace Relations.
Guerra-Ramos, M. T., García-Horta, J. B., López-Valentín, D., Gómez-Galindo, A., & Balderas-Robledo, R. (2007). SCIENCE TEACHERS’ATTITUDES AND PERCEPCTIONS RELATED TO PRACTICAL WORK: A SELF REPORT QUESTIONNAIRE.
Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American educational research journal, 38(4), 915–945.
Goldspink, C. (2007). Rethinking Educational Reform A Loosely Coupled and Complex Systems Perspective. Educational Management Administration & Leadership, 35(1), 27–50.
Goldstein, H. (1987). Multilevel models in education and social research. Oxford University Press.
Goldstone, R. 2006. The Complex Systems See-Change in Education. The Journal of the Learning Sciences, 15, 35-43.
175
Goodson, I. & Beista, G. 2010. Narrative Learning, , London and New York, Routledge.
Goodson, I. & Deakin Crick, R. 2009. Curriculum as narration: tales from the children of the colonised. Curriculum Journal, 20, 225 - 236.
Gunter, H. 2001. Leaders and Leadership in Education, London, Paul Chapman.
Grace, G. (2002). Catholic schools: Mission, markets, and morality. Routledge.
Habermas, J. (1972). Knowledge and human Interests (paperback). Beacon Press, USA. Retrieved from http://www.lavoisier.fr/livre/notice.asp?id=OLKWAXAOKA6OWO
Hall, J. K. (2004). Language Learning as an Interactional Achievement. The Modern Language Journal, 88(4), 607–612. doi:10.2307/3588591
Hall J W, Blockley D I, Davis J P Uncertain Inference Using Interval Probability Theory. Int. J. Approximate Reasoning. 19
247-264, 1998.
Hallinger, P., & Heck, R. H. (2010). Collaborative leadership and school improvement: understanding the impact on school capacity and student learning. School Leadership & Management, 30(2), 95–110. doi:10.1080/13632431003663214
Hanushek, E. & Woessmann, V. 2011. The economics of international differences in educational achievement. In: Hanushek, E., Machin, S. & Woessmann, L. (eds.) Handbook of the Economics of Education. Amsterdam: North Holland.
Hattie, J. 1999. Influences on Student Learning. Auckland: University of Auckland.
Hattie, J. 2009. Visible Learing A Synthesis of over 800 Meta-Analyses relating to learning, London, Routledge.
Hautamaki, J., Arinen, P., Eronen, S., Hautamaki, A., Kupiainen, S., Lindblom, B., Niemivirta, M., Pakaslahti, L., Rantanen, P. & Scheinin, P. 2002. Assessing Learning to Learn: a framework, Helsinki, National Board of Education.
Heron, J. & Reason, P. 1997. A Participatory Inquiry Paradigm. Qualitative Inquiry, 3, 274-294.
Harlen, W. & Deakin Crick, R. 2003a. A systematic review of the impact of summative assessment and testing on pupils' motivation for learning. London: Evidence for Policy and Practice Co-ordinating Centre Department for Education and Skills.
Harlen, W. & Deakin Crick, R. 2003b. Testing and Motivation for Learning. Assessment in Education, 10, 169-207.
Harlen, Wynne, & Crick, R. D. (2002b). Review: What is the evidence of the impact of summative assessment and tests on students’ motivation for learning.
Hidi, S. & Renninger, K. A. 2006. The Four-phase Model of Interest Development. Educational Psychologist, 41, 111-127.
Hofer, B. K. 2001. Personal Epistemology Research: Implications for Learning and Teaching. Journal of Educational Psychology Review, 13, 353-383.
Jackson, M. C. (2000). Systems approaches to management. Springer. James, M., & Gipps, C. (1998). Broadening the basis of assessment to prevent the narrowing of learning. Curriculum journal, 9(3), 285–297.
Jackson, D., & Temperley, J. (2007). From professional learning community to networked learning community. Professional learning communities: Divergence, depth and dilemmas, 45–62.
James, M. & Gipps, C. 1998. Broadening the basis of assessment to prevent the narrowing of learning. The Curriculum Journal, 9, 285-297.
James, M., Mccormick, R., Black, P., Carmichael, P., Drummond, M., Fox, A., Macbeath, J., Marshall, B., Pedder, D.,
176
Procter, R., Swaffield, S., Swann, J. & Wiliam, D. 2007. Improving Learning How to Learn: Classrooms, Schools and Networks, London, Routledge.
Katz, S., & Earl, L. (2010). Learning about networked learning communities. School Effectiveness and School Improvement, 21(1), 27–51.
Kyriakides, L., & Campbell, R. J. (2004). School self-evaluation and school improvement: A critique of values and procedures. Studies in educational evaluation, 30(1), 23–36.
Ladwig, J., Smith, M., Gore, J. M., Amosa, W. & Griffiths, T. 2007. Quality of Pedagogy and Student Acheivement: Multi-level replication of Authentic Pedagogy. ANARE Annual Conference. Freemantle, Western Australia.
Leithwood, K. & Jantzi, D. 1999. The relative effects of principal and teacher sources of leadership on student engagement with school. Educational Administration Quarterly, 35, 27.
Little, A. D., & Veugelers, R. (2005). Internationalisation of R&D in the UK. Report to the office of Science and Technology.
Loxley, A., Johnston, K., Murchan, D., Fitzgerald, H., & Quinn, M. (2007). The role of whole-school contexts in shaping the experiences and outcomes associated with professional development. Journal of In-Service Education, 33(3), 265–285.
Mahn, H. & John-Steiner, V. 2002. Developing the affective zone of proximal development. In: Wells, G. & Claxton, C. (eds.) Learning for Life in the 21st Century, Socio-cultural perspectives on the future of education. Oxford: Blackwell.
Macbeath, J., & CHENG, Y. C. (2008). Leadership for learning: International perspectives. Retrieved from http://repository.ied.edu.hk/dspace/handle/2260.2/8940
MacBeath, J., & McGlynn, A. (2002). Self-evaluation: What’s in it for Schools? Routledge.
Marshall, B. & Drummond, M. 2006. How teachers engage with Assessment for Learning: lessons from the classroom. Research papers in Education, 21, 133-149.
Mason, M. (2008). What is complexity theory and what are its implications for educational change? Educational Philosophy and Theory, 40(1), 35–49.
Mason, M. (ed.) 2008. Complexity Theory and the Philosophy of Education, Chichester: Wiley-Blackwell.
McCombs, B. L., & Whisler, J. S. (1997). The Learner-Centered Classroom and School: Strategies for Increasing Student Motivation and Achievement. The Jossey-Bass Education Series. ERIC.
Mckinsey Report 2007. How the world's best performing school systems come out on top. London: McKInsey.
Munns, G. & Mcfadden, M. 2000. First Chance, Second Chance or Last Chance? Resistence and Response to education. British Journal of Sociology of Education, 21, 59-75.
Newmann, F. & Wehlage, G. 1995. Successful school restructuring, Madison WI, Center on Organisation and Restructuring of Schools.
Opfer, V. & Pedder, D. 2011. The lost promise of teacher professional development in England. European Journal of Teacher Education, 34, 3-24.
Meiers, M., & Ingvarson, L. (2005). Investigating the links between teacher professional development and student learning outcomes. Department of Education, Science and Training. Mingers, J., & Brocklesby, J. (1997). Multimethodology: towards a framework for mixing methodologies. Omega, 25(5), 489–509.
Morgan, G., & Smircich, L. (1980). The case for qualitative research. Academy of management review, 5(4), 491–500.
177
Muijs, D. (2010). Doing quantitative research in education with SPSS. Sage.
PriceWaterhouseCoopers. 2007. The economic benefits of a degree. Research report. London: Universities UK.
Pedder, D. 2006. Organizational conditions that foster successful classroom promotion of learning how to learn. Research Papers in Education,, 21, 171-200.
Pedder, D. 2007. Profiling teachers' professional learning practices and values: differences between and within schools. The Curriculum Journal 18, 231-252.
Pedder, D., James, M. & Macbeath, J. 2005. How teachers value and practise professional learning. Research Papers in Education, 20, 209-243.
Pedder, D., & MacBeath, J. (2008). Organisational learning approaches to school leadership and management: teachers’ values and perceptions of practice. School Effectiveness and School Improvement, 19(2), 207–224.
Piesanen, E., Kiviniemi, U., & Valkonen, S. (2007). Follow-up and Evaluation of the Teacher Education Development Programme. Continuing Teacher Education in 2005 and its Follow-up 1998-2005 by Fields and Teaching Subjects in Different Types of Educational Institutions. Occasional papers, 38.
Polanyi, M. (1967). The tacit dimension. 1967. Garden City, New York, 217.
Reay, D., & Wiliam, D. (1999). ’I’ll be a nothing’: structure, agency and the construction of identity through assessment [1]. British Educational Research Journal, 25(3), 343–354.
Ren, K., & Deakin Crick, R. (2013). Empowering underachieving adolescents: an emancipatory learning perspective on underachievement. Pedagogies: An International Journal, (ahead-of-print), 1–20.
Ritchie, R. (2007). School Self-Evaluation. Advances in Program Evaluation, 10, 85–101. doi:10.1016/S1474-7863(07)10006-5
Robinson, V. M., Lloyd, C. A., & Rowe, K. J. (2008). The impact of leadership on student outcomes: An analysis of the differential effects of leadership types. Educational administration quarterly, 44(5), 635–674.
Rosenhead, J., & Mingers, J. (2001). Rational analysis for a problematic world revisited. John Wiley and Sons.
Sammons, P. (1999). School effectiveness: Coming of age in the twenty-first century (Vol. 6). Taylor & Francis.
Seely Brown, J. & Thomas, D. 2009. Learning for a World of Constant Change. 7th Gillon Colloquium. University of Southern California.
Sfard, A. & Prusak, A. 2005. Telling Identities: In Search of an Analytic Tool for Investigating Learning as a Culturally Shaped Activity. Educaitonal Researcher 34, 14-22.
Shernoff, D. J., Csikszentmihalyi, M., Schneider, B. & Steele , E. 2003. Student Engagement in High School classrooms from the perspective of flow. . School Psychology Quarterly, 18, 18.
Silins, H. M., Bill 2001. Reframing Schools: The Case for system, teacher and student learning. AARE.
Shafer, G. A Mathematical Theory of Evidence, Princetown: Princetown University Press, 1976.
Shulman, L. (2011). The skills of helping individuals, families, groups, and communities. Cengage Learning.
Shum, S. B., & Crick, R. D. (2012). Learning dispositions and transferable competencies: pedagogy, modelling and learning analytics. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 92–101).
Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of educational psychology, 82(3), 498.
178
Stoll, L., & Louis, K. S. (2007). Professional learning communities: Divergence, depth and dilemmas. McGraw-Hill International.
Sykes, G. (1996). Reform OF and AS Professional Development. Phi Delta Kappan, 77(7), 464–67.
Teddlie, Charles, & Reynolds, D. (2000). The International Handbook of School Effectiveness Research. Routledge.
Timperley, H., & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of research in education, 32(1), 328–369.
Wheatley, M. 1999. Leadership and the new science: Discovering order in a chaotic world (2nd ed.), San Francisco, Berrett-Koehler Publishers.
Wheatley, M. & Kellner-Rogers, M. 1998. Bringing Life to Organizational Change. Journal for Stategic Performance Management, April/May.
Wheatley, M. 1999. Leadership and the new science: Discovering order in a chaotic world (2nd ed.), San Francisco, Berrett-Koehler Publishers.
Wheatley, M. & Kellner-Rogers, M. 1998. Bringing Life to Organizational Change. Journal for Stategic Performance Management, April/May.
Willms, J. 2003. Student engagement at school: a sense of belonging and participation: results from PISA 2000. Paris: OECD.
Zevenbergen, R., Mousley, J. & Sullivan, P. 2004. Making the pedagogic relay inclusive for indigenous Australian students in mathematics classrooms. International Journal of Inclusive Education 8, 391-405.
Zohar, D. 1997. Rewiring the Corporate Brain: using the new science to rethink how we structure and lead organisations, San Francisco, Berrett-Koehler.
Zyngier, D. 2004. Doing Education not Doing Time: Engaging Paedagogies and Pedagogues - what does student engagement look like in action? International Education Research Conference Melbourne.
Weaver, W. (1948). Science and complexity. American scientist, 36(4), 536–544.
Woolfolk Hoy, A., Hoy, W. K., & Davis, H. A. (2009). Teachers’ self-efficacy beliefs. Handbook of motivation at school, 627–654.
Veenman, M. V., Van Hout-Wolters, B. H., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and learning, 1(1), 3–14.
Vygotsky, L. 1978. Mind and society: the development of higher psychological processes, MA, Harvard University Press.
179
APPENDIX
APPENDIX 1 GLOSSARY OF TERMS
Research into the properties of social systems suggests that there are some key properties which are particularly relevant in understanding schools. These provide the project with key reference points for testing, critique and revision as we learn to what extent they help researchers and school leaders make sense of the dilemmas they face.
Autopoiesis: the process whereby a system reproduces itself. System components generate recursively the same network of processes which produced them - this is relevant to the emergence of learner identity in co-constructed domains of meaning (De Jaegher and Di Paolo, 2007; Kay, 2002; Goldspink and Kay, 2009; Goldspink, 2009) and hence for the way we approach learning as well as school change.
Dissonance: lack of agreement, alignment, balance or conflict between people, processes or things - this creates a space for deep learning through identifying and formulating problems, and finding solutions, or resolutions, as a key element of enquiry.
Emergence: the way complex systems and patterns arise out of a multiplicity of relatively simple interactions - the creation of something new from the relationships between processes is a space for creativity and learning – creative leadership is required to manage complex organisations (Sawyer, 2001; Goldspink, 2010; Stacey, 1996).
Resilience: the ability of a system or person to continue functioning when stressed and where necessary, adapt and change in response to challenge - this is a core aspect of learning power - for individuals, organisations and communities.
Synergy: the whole is more than the sum of its parts - a learning community has a culture and synergy which cannot be reduced to a single process. In addition to these we include trust.
Trust: relationships of such quality that they can withstand the challenges of risk, uncertainty, inequality and difference (Bryk et al 2010; Deakin Crick 2007; Bond 2004) and identity.
Identity: learning that enables the identification and location of ‘self’ in personal, organisational and community narratives (Varela, 1997; Bruner, 1990, 1996; Sfaard & Pusak 2005; Deakin Crick, 2011).
APPENDIX 2 OASIS CHARTER
Oasis Community Learning’s Education Charter was agreed by the Board of Trustees in November 2009. It is based on the views of all the main stakeholders in Oasis Academies and serves as a statement of the organisation’s overall values, vision and purposes. At the time, the Trustees suggested that wider measures of the performance of Oasis Academies might be developed, hence the ECHO Project. Between 2010 and 2013, ECHO has focused on a pilot project to evaluate the
180
‘Transforming Learning’ section of the Charter. The other two sections are ‘Transforming Lives’ and ‘Transforming Communities’.
TRANSFORMING LEARNING - We have a passion for learning and we want everyone to achieve
their full, God-given potential
Creating environments in all our Academies where learning is fun
Providing all our students with excellent opportunities for learning and assessment to support learning
Ensuring that students maintain good progress between the primary and secondary phases and other key transitions
Aiming for ‘good’ to ‘outstanding’ standards of learning and teaching in all our Academies and, where necessary, getting the basics right to raise standards of attainment quickly
Understanding and thinking critically about different worldviews as frameworks for learning
Providing a forward-looking, broad and balanced curriculum which develops the gifts and talents of all students and those that teach and lead them
Ensuring that Oasis Academies become centres of excellence for their specialisms
Ensuring that students are accessing a wide range of extended opportunities for learning within and beyond the Academy
Ensuring that learning balances knowledge and understanding with skills for life and caters for the whole person: academically, vocationally, socially, morally, spiritually, physically, emotionally and environmentally
Providing progression routes for all students in the secondary phase that are attractive and accessible, helping them to meet challenging personal targets and achieve excellent results
Enabling students to make a successful transition to interdependent living through employment, further/higher education or other beneficial activities; becoming confident, competent and fulfilled adults
Providing learning and teaching in an environment that is healthy and safe and where there are excellent services for care, guidance and support, particularly for the most vulnerable students
Developing a culture of mutual accountability and rigorous evaluation; being well prepared for external review and inspection
Managing the resources that we are given in a responsible way to ensure maximum impact on the quality of education provided
Developing our staff and sharing our successes between the Oasis Academies and more widely
Measuring performance based on the aspirations and outcomes of the Education Charter; committing to continuous improvement
Celebrating success widely and regularly; supporting others with their learning and being pleased when others do well
181
APPENDIX 3 CRITICAL SUCCESS INDICATORS (CSI)
The ECHO Project will focus on CSIs relating to the learning outcomes of the Education Charter,
particular the section ‘Transforming Learning’. This will complement the Academy/Hub evaluation
projects that focus on spiritual, moral, social and cultural development and on the Oasis ethos.
The sets of questions that form the basis of the evaluation strands will be designed to provide
evidence for one or more of the twelve 3rd Layer CSIs. The subsequent Perimeta analysis will
provide developmental feedback to the Academies about quality of performance on the CSIs
according to the colours of the Italian flag: red = not being achieved; green = being achieved; white
= insufficient evidence to make a judgement.
Critical Success Indicators for an Oasis Academy
1st Level CSI:
1.1 Establishing and sustaining a group of high achieving learning communities that enables
everyone to realise their full potential and refuses to put limits on achievement
2nd Level CSIs:
2.1 Developing the learning of students so that they realise their full potential
2.2 Developing the learning of teachers so that they realise their full potential
2.3 Developing the learning of leaders so that they lead the learning of teachers and students effectively
2.4 Engaging parents/carers effectively in the learning activities of the Academy and in supporting the learning of their children
3rd Level CSIs:
3.1 Feeling valued and included in our community (evidence from students & staff)
3.2 Feeling part of a community that focuses its activities on learning (students & staff)
3.3 Knowing that we are being helped to reach our full potential and achieve our personal best (students & staff)
3.4 Maintaining a good rate of progress at key transitions, for example, primary to secondary school (students)
3.5 Achieving results that meet my best expectations (students & staff)
3.6 Feeling that my successes are recognised (students & staff)
182
3.7 Contributing actively to a community that focuses on service to others (students & staff) 3.8 Contributing actively to a community where shared leadership is promoted (students & staff)
3.9 Providing evidence that training and development opportunities have helped me to improve my
classroom practice (staff)
3.10 Contributing actively to a community that learns together, shares what works best and knows what it needs to do to improve (staff)
3.11 Feeling included and involved with my children’s education at the Academy and knowing that there is an open door for contact when I need it (parents/carers)
3.12 Being effective role models as leaders of learning (senior/middle leaders)
APPENDIX 4 QUESTIONAIRES
STRAND ONE – STUDENTS SELF-REPORT QUESTIONNAIRE ON LEARNING POWER
A 72 item research validated online questionnaire, the Effective Lifelong Learning Inventory (ELLI).
STRAND TWO - STUDENTS PERCEPTIONS OF EDUCATION CHARTER
Student Questionnaire
This questionnaire will help us to find out more about what our students think and feel about their
learning, teaching, progress and achievement in an Oasis Academy. By getting your views we can
celebrate what is going well and try to make improvements where things are not going as well as
you would like.
When we analyse the results of the questionnaire, we shall be combining all the students’
responses and so what you say as an individual student will be confidential.
Please answer the questions by marking the response that best describes what you think or feel
about each statement.
Questions
1. I feel included in the activities that are available to me in the Academy
*Strongly agree *Agree *Disagree *Strongly disagree
(Repeat after each question)
2. I feel safe in the Academy
3. I feel valued and cared for as an individual in the Academy
4. I like learning new things
5. I usually concentrate on my learning
183
6. I am hard to distract
7. I like working with other students to help my learning
8. I don’t distract other students
9. I am confident in my learning
10. I take a lot of care with my work
11. I work hard
12. I do more than I am asked to do in class
13. I don’t mind making mistakes because I learn from them
14. I come up with new ideas to help my learning
15. I enjoy my learning
16. My teachers teach well so that I learn successfully
17. I have regular opportunities to express my opinion to my teachers about my learning
18. I often have conversations with my teachers which help me to make better progress
19. My teachers seem to enjoy teaching my classes
20. My teachers often share things that they have learned themselves
21. I get on well with my teachers
22. I get extra support to help my learning when I need it
23. I know that the senior teachers want us to learn successfully
24. Everyone works very hard to make the Academy a great place to learn and to do well
25. My teachers constantly expect me to improve on my personal best
26. Right now I am achieving the best I possibly can
27. From what I remember, I kept up a good rate of progress between my last year at primary
school and my first year at the Academy
28. I have kept up a good rate of progress ever since my first year at the Academy
29. I think that my results at the moment are as good as I can do
30. I feel that my successes at the Academy are recognised
31. I feel that my successes with activities outside the Academy are recognised
32. I feel proud about what I have achieved so far
33. We are taught about the responsibilities of students as well as their rights
34. We care about helping each other in the Academy community
35. We are encouraged to care about the needs of other people in the local community and
around the world
36. I regularly offer to help others
37. My parents or carers feel that they are always welcome in the Academy
38. My parents or carers feel involved with my learning and my progress
39. My parents or carers are pleased with my progress
40. My parents or carers have always been able to sort out any problems that I have had
41. I am encouraged to lead activities in the classroom
42. I have had opportunities to lead other activities at the Academy
(For example, sports teams, fund-raising, student council activities)
184
STRAND THREE – STUDENT NARRATIVE INTERVIEWS
Student Interview Schedule
This is a semi-structured interview schedule, designed to invite students to tell stories about their
learning. We are interested in how they understand and experience success and struggle in
learning, how they understand themselves as learners and whether they see themselves as
changing and improving over time. These questions are not designed to be read verbatim – rather
they provide a structured framework for a natural conversation. If a question doesn’t work –
reframe it, and feel free to add open-ended prompts.
It is important that you tell the students why we are asking these questions and what will happen
to the recording we make. Please see the ECHO Consent Guidance and stress that:
The data that we gather will be anonymous and not traceable to a particular student.
There is no right answer to each question.
We want to know how people are learning in our Academy so we can improve what we do
and make it a better place to learn.
At the start of each recording, please state the student’s unique reference number.
Questions
8. Tell me about a time when you learned something really successfully in the last two or three
weeks
Why did you learn successfully?
What did it feel like
What happened as a result?
9. Tell me about your worst experience of learning in the last two or three weeks?
Why was it a bad experience of learning?
What did it feel like?
What happened as a result?
10. How different are you now as a learner compared with a year ago?
How would you describe the differences?
What has happened to make these differences?
How do you feel now about the future as a learner?
What changes would help you to learn better?
11. How would you describe yourself as a learner now? If you could choose an animal (or a car)
which is most like you as a learner, which animal (or car) would it be? Why did you choose
that particular animal (or car)? [Probe further for reasons, if necessary]
12. What have been the best things about your time at the Academy so far?
13. What have been the worst things about your time at the Academy so far?
14. How do you feel about your future, after your time at the Academy?
185
STRAND FOUR – YEAR 11 LEAVERS PERCEPTIONS OF THEIR EXPERIENCE AT THEIR OASIS
ACADEMY
1. I was valued and included during my time at the Academy
*Strongly agree *Agree *Disagree *Strongly disagree
(Repeat after each question)
2. I felt part of a community that was focused on learning
3. I felt safe at the Academy 4. 5.
6. I enjoyed my time at the Academy 7.
8. I did as well as I had hoped in my GCSEs 9.
10. My teachers accurately predicted my GCSE results
11. I am looking forward to the move to Sixth Form, Further Education, training or work
12. I have been well prepared for the next stage of my life
13. My successes were recognised during my time at the Academy
14. I was helped and supported whenever I struggled with my learning
15. I developed a sense of responsibility towards other people during my time at the
Academy
16. The Academy helped to make me more understanding about different cultures and
types of people
17. I supported activities that would improve things for other people
18. I was offered opportunities for leadership during my time at the Academy
19. I led activities during my time at the Academy
20. My teachers taught me well at the Academy 21.
22. My teachers were always trying to improve their teaching
23. I was regularly challenged to do better so that I could reach my full potential
24. My parents/carers felt involved with my education at the Academy
25. My parents felt that they could find the right person to talk to at the Academy if they
had any concerns about my education
26. Senior staff at the Academy wanted us to learn well and achieve our best
27. Everyone at the Academy worked very hard to make it a great place to learn and to do
well
28. The Academy was well run 29. 30.
31. There were great facilities for learning 32.
33. There was a broad curriculum that helped me to develop as a whole person
34. I enjoyed a wide range of opportunities for learning beyond the timetabled curriculum
35. The Academy made good use of new technologies for learning
36. I never felt that anyone was putting limits on what I could achieve
37. I feel proud about what I achieved at the Academy
186
38. I could not have gone to a better place for my secondary education
STRAND FIVE - PERCEPTIONS OF PARENTS AND CARERS OF THEIR CHILDREN'S
ACADEMIES
This questionnaire has been designed to gather evidence about parent/carer experiences of the
education of their children at the Academy. Your views will help us to plan further improvements
that will provide an even better education for all our students.
Please answer the questions in Part I by marking the most appropriate response for each question
based on the experience of your child at the Academy. If you have more than one child at the
Academy and their experiences have been different, please complete an additional questionnaire.
Part II provides some additional questions for parents/carers in families where the first language is
not English.
Your responses will be treated with complete confidence. We shall only be reporting on the
combined views of all the parents/carers who complete the questionnaire. If we do want to follow
up individual responses, we shall ask for your permission.
Thank you for completing the questionnaire.
Part I: General Questions
1. My child is known and valued as an individual
*Strongly agree *Agree *Disagree *Strongly disagree
2. The Academy is providing for my child’s particular needs
3. My child feels included in the life of the Academy, in classrooms and in other activities
4. The Academy focuses its work on providing the best opportunities for learning for all
students
5. My child is taught well
6. My child is being challenged to improve on previous best performance in all areas
7. My child’s progress at the Academy has exceeded our expectations
8. My child’s successes are recognised
9. The Academy ensures my child is well looked after
10. The Academy helps my child to have a healthy lifestyle
11. There is a good standard of behaviour at the Academy
12. My child is developing an attitude of helping others
(for example, by doing practical activities to improve life at the Academy, by supporting
people in need, getting involved with community projects, fund-raising)
13. My child is taking opportunities to lead activities (for example, leading group work in the
classroom, leading a sports team, giving feedback to teachers about their learning, being a
representative on a Student Council, taking a lead role in Academy performances of music,
drama or dance, being a coach, mentor or ‘buddy’)
187
14. My child has kept up a good rate of progress at key transition (for example, from primary to
secondary school; from Year 9 to GCSE courses in Years 10 & 11; from Year 11 to Sixth
Form/College; from Year 13 to further education/higher education/employment)
15. I get regular and helpful information about my child’s progress
16. The Academy responds well to my concerns
17. I always find staff knowledgeable about my child
18. I always find staff helpful when discussing my child’s progress
19. The Academy helps me to support my child’s learning
20. The Academy is led and managed effectively
21. My impression is that senior staff focus their work on making the Academy a good place to
learn
22. My impression is that senior staff expect all students to achieve well
23. My child enjoys their education at the Academy
24. I would recommend the Academy to another parent/carer
25. If you want to explain any of your answers, or if there is anything else you want us to be
aware of about your child’s experience of the Academy, please give details here:
Part II: Additional questions for parents/carers whose first language is not English
26. The Academy has been helpful in ensuring that I understand the general information sent to
parents/carers
27. The Academy has been helpful in ensuring that I understand the information about my
child’s progress
28. I feel that the Academy understands and takes proper account of the particular issues and
sensitivities of my culture
29. If I have a concern, I feel confident that I can make contact with relevant staff
30. If I have a concern, I feel confident that I can communicate with staff effectively
31. I would recommend the Academy to another parent/carer from my community
32. If you want to explain any of your answers, or if there is anything else you want us to be
aware of about your child’s experience of the Academy, please give details here:
188
STRAND SIX - LEARNING OF TEACHERS: IMPACT OF CPD ON THE QUALITY OF
CLASSROOM PRACTICE
Teacher Questionnaire This questionnaire is based on two sets of research-validated questions about teachers’ learning from the national Learning How to Learn project, based at the Department of Education, Cambridge University, (Parts A and B). We are grateful for permission to use these questions. There are six additional questions in a final section that focus on aspects of the Oasis Education Charter, (Part C). Your responses will be treated with complete confidence. We shall only be reporting on the combined views of all the teachers completing the questionnaire and the teacher interview. If we do want to follow up individual responses, we shall ask your permission beforehand. We are very grateful for the time that you have given to these aspects of the ECHO Project. Please complete the questionnaire by marking the most appropriate response to each question based on your experience as a teacher in this academy.
PART A – PROFESSIONAL LEARNING
1. Staff as well as students learn in this academy
* Very often *Often *Sometimes *Rarely 2. I draw on good practice from other schools as a means to further my own professional
development 3. I read research reports as one source of useful ideas for improving my practice 4. I use the web as one source of useful ideas for improving my practice 5. I consult my students about how they learn most effectively 6. I relate what works in my own practice to research findings 7. I am able to see how practices that work in one context might be adapted to other contexts 8. I use insights from my own professional learning to feed into academy policy development 9. I reflect on my practice as a way of identifying my professional learning needs 10. I experiment with my practice as a conscious strategy for improving classroom teaching and
learning 11. I modify my practice in the light of feedback from my students 12. I modify my practice in the light of published research evidence 13. I modify my practice in the light of evidence from self-evaluations of my classroom practice
189
14. I modify my practice in the light of evidence from evaluations of my classroom practice by managers or other colleagues
15. I carry out joint research/evaluation with one or more colleagues as a way of improving my practice
16. I collaborate to plan my teaching 17. I observe other colleagues in the classroom and we give each other feedback 18. I engage in team teaching as a way of improving practice 19. If I have a problem with my teaching I turn to colleagues for help 20. I suggest ideas or approaches for colleagues to try in class 21. I make collective agreements to test out new ideas 22. I discuss openly with colleagues what and how they are learning 23. I use informal opportunities to discuss how children learn 24. I offer reassurance and support to colleagues 25. My practice is based on the belief that all students are capable of learning 26. My experience is that students in this academy enjoy learning 27. My experience is that student success is celebrated 28. I discuss with colleagues how students might be helped to learn how to learn
PART B – ACADEMY LEADERSHIP PRACTICES AND SYSTEMS
1. Senior leaders communicate a clear vision of where the academy is going
*Very often *Often *Sometimes *Rarely 2. Staff have a commitment to the whole academy as well as to their department, key stage
and/or year group 3. Senior leaders promote commitment among staff to the whole academy as well as to the
department, key stage and/or year group 4. There is effective communication between senior leaders and teachers 5. There are processes for involving all staff in decision-making 6. Teachers’ professional know-how is used in the formulation of academy policy and goals 7. Teachers’ professional know-how is used in the formulation of academy policy, even where
this leads to a questioning of established rules, procedures and practices 8. Opportunities are provided for teachers to critically evaluate academy policy 9. Staff are involved in evaluating academy policy 10. Staff participate in important decision-making 11. There are processes for involving students in decision-making 12. Staff have a good working knowledge of the Academy Development Plan (or your academy’s
equivalent document) 13. Staff see the Academy Development Plan as relevant and useful to learning and teaching 14. Staff development time is used effectively to realise Academy Development Plan priorities 15. Staff development time is used effectively in the academy 16. The academy provides cover to allow staff joint planning time 17. Teachers are encouraged to experiment with new ideas as a way of promoting professional
growth 18. Formal training provides opportunities for teachers to develop professionally 19. Teachers are helped to develop skills to assess students’ work in ways that move students
on in their learning
190
20. Teachers are helped to develop skills to observe learning as it happens in the classroom 21. Senior leaders support teachers in sharing practice with other schools through networking 22. Information is collected from teachers on those aspects of their work that they themselves
think they do effectively 23. Information is collected from teachers on effective ways they promote learning to learn
skills and knowledge among their students 24. Information is collected from teachers on informal teacher networking in which they play an
active role 25. Teacher-initiated networking is an integral element of staff development 26. Learning how to learn is an issue discussed in staff development time
PART C – ASPECTS OF THE OASIS EDUCATION CHARTER
1. I feel included in a community that values what I can offer to the staff team and to students
*Very often *Often *Sometimes *Rarely 2. I believe that I am working in a community that focuses our attention and activities on
learning to improve teaching and student outcomes 3. My experience suggests that other members of staff are involved in training and
development activities to improve their classroom practice and student outcomes 4. I have had opportunities to work alongside and/or train with colleagues in other phases to
improve students’ progress, e.g. secondary with primary teachers or secondary with tertiary phase teachers
5. My experience is that senior leaders are role models as ‘leaders of learning’ in the academy 6. Leadership is shared in this academy and opportunities to take on leadership roles are
available to all those that want them and have the relevant knowledge and experience 7. My experience is that the academy promotes a balance between the rights and the
responsibilities of all members of the community 8. There is an emphasis among students and staff on caring about and helping other people in
the academy community 9. There is an emphasis among students and staff on caring about and helping other people in
the wider community 10. The academy is successful in helping parents/carers to support the learning of their children 11. The achievements of staff are recognised and celebrated 12. I believe that I am realising my full potential as a teacher
PART D – BACKGROUND INFORMATION
Please tick or fill in the appropriate boxes: 1. Female --- Male --- 2. Name of your academy -------------------------------- 3. Years of teaching experience --- 4. Years at this academy --- 5. Post and responsibility (if relevant) --------------------------------- 6. Please identify your main area of responsibility ----------------------------- 7. Please identify the main subjects that you teach -----------------------------
191
STRAND SEVEN - SENIOR LEADERSHIP
Individual questionnaire for senior leaders, followed by a team discussion
Introduction
This questionnaire is part of the ECHO Project which is exploring approaches to broaden the range
of evidence used to evaluate Oasis Academies. The Project, based on a partnership between Oasis
Community Learning, University of Bristol and PA Consulting, is being undertaken as a pilot study in
three Oasis Academies. We are using Perimeta software to analyse the data generated by the
evidence-gathering processes. It is hoped that the analysis will then help to inform strategic
thinking and development planning in each Academy, so that student outcomes continue to
improve across a broader range of objectives.
The ECHO Project is focusing on evaluation of the ‘Transforming Learning’ dimension of OCL’s
Education Charter, (ECHO stands for Evaluating Charter Outcomes). We are using a model that
proposes the effective relationship between student learning, teacher learning, the leadership of
learning and parental support as a key to creating Oasis Academies as high-achieving learning
communities. Our aim is to develop processes for evaluating the broader aspects of education in
Oasis Academies that are both affordable and sustainable.
For the pilot project in 2011/12, we shall only be investigating the role of senior leaders in the
leadership of learning. If the pilot is successful, it will be possible to extend this aspect of evidence-
gathering to other leaders in subsequent years.
In devising this questionnaire, we have drawn from the following international sources about the
relationship between leadership, learning, teaching, student attainment and overall school
improvement, including several research-validated questionnaires:
Questionnaires from the Learning How to Learn research project (2002, University of
Cambridge)
Resources for Learning-centred Leadership (2004, National College)
Framework for Teaching for Effective Learning: Unleashing learning potential (2010,
Government of South Australia)
Essential supports for improvement from Organizing Schools for Improvement: Lessons from
Chicago (2010, Anthony S Bryk, et al., University of Chicago Press)
Core behaviours from Resourceful leadership: how directors of children’s services improve
outcomes for children (2011, National College)
The questionnaire will be completed by individual members of the senior team at your Academy
and returned to Howard Green a few days before a meeting of the Academy Leadership Team. The
192
aggregated responses will then be used to identify common threads, anomalies and emerging
issues and to create an agenda for discussion at the ALT meeting. The confidentiality of individual
responses is guaranteed. On this basis, please would you put your name in the space below:
Your name:
Leadership of Learning Questionnaire
Please answer the following questions as honestly and objectively as possible, either underlining
the appropriate option on the four-point scale or (for the last question only) by providing a brief
written response. If you are not a teacher/teaching member of the senior leadership team, please
leave the questions that are not relevant (for example, numbers 2 and 3).
1. I have a good understanding of my own and other people’s learning
*Strongly agree *Agree *Disagree *Strongly disagree
2. I keep my knowledge about teaching and learning and about my phase and/or subject specialism up to date
3. I teach regularly and my lessons are rated good or outstanding
4. I undertake regular and substantial professional development for myself
5. I understand the differences between good and outstanding lessons
6. I regularly work with staff to improve their teaching practice
7. I can design, monitor and assess programmes for effective teaching and learning
8. I regularly participate in critical and reflective enquiry to develop teaching and learning across the academy
9. I understand what an academy needs to do to become good and outstanding
10. I regularly work with staff to help them improve their leadership and management
11. I am an effective coach and use my coaching skills regularly to help others to improve their performance
12. I regularly interact with staff in other schools, academies and/or organisations to build learning partnerships which connect staff and students to learning opportunities beyond our academy
13. I make regular and effective contributions to educational dialogue that shapes whole academy policy and informs practice
14. I contribute effectively to the critical development of academy systems and structures in order to ensure quality teaching focused on student progress in learning
193
15. The senior leadership team communicates a clear vision of where the academy is going so that there is a shared sense of purpose
16. Staff have a commitment to the whole academy as well as to their department, key stage and/or year group
17. The senior leadership team effectively promotes commitment among staff to the future development of the whole academy
18. There is effective communication between senior leaders and teachers
19. There are processes for involving all staff in decision-making
20. Teachers’ professional know-how is used in the formulation of academy policy and goals
21. Teachers’ professional know-how is used in the formulation of academy policy, even where this leads to a questioning of established rules, procedures and practices
22. Opportunities are provided for teachers to critically evaluate academy policy
23. Staff participate in important decision-making
24. There are processes for involving students in decision-making, including how best to improve classroom practice
25. Staff have a good working knowledge of the Academy Development Plan
26. Staff see the Academy Development Plan as relevant and useful to learning and teaching
27. Staff development time is used effectively to realise Academy Development Plan priorities
28. Staff development time is used effectively in the academy, particularly through the explicit modelling of good practice
29. The academy provides support to allow staff joint planning time
30. Teachers are encouraged to experiment with new ideas as a way of promoting professional growth and improving classroom practice
31. Formal training provides opportunities for teachers to develop professionally
32. Teachers are helped to develop skills to assess students’ work in ways that move their students on in their learning
33. Teachers are helped to develop skills to observe learning as it happens in the classroom
34. Senior leaders support teachers in sharing practice with other schools and academies through networking
194
35. Information is collected from teachers on those aspects of their work that they themselves think they do effectively
36. Information is collected from teachers on effective ways they promote ‘learning tolearn’ skills and knowledge among their students
37. Information is collected from teachers on informal teacher networking in which they play an active role
38. Teacher-initiated networking is an integral element of staff development
39. ‘Learning how to learn’ is an issue discussed in staff development time
40. There is a well-developed shared language and a rich dialogue about teaching and learning across the academy
41. We are good at systematic monitoring, use of data and feedback across the academy
42. We have efficient and effective support systems and structures across the academy
43. The senior leadership team are effective role models as ‘leaders of learning’, (for example, by securing broad commitment to learning for all that is focused on improving the quality of teaching and student outcomes)
44. We have developed successful approaches to distributed leadership across the academy where the pathways of accountability are working well
45. The academy is successful in helping parents/carers to support the learning of their children
46. The governors play a full and appropriate part in strategic planning to improve teaching and learning and student outcomes
47. The academy involves a wide range of other stakeholders directly and powerfully in the education of students
48. The academy is good at securing the broadest possible impact of successful innovations in teaching and learning
49. There is a widespread and well-established climate of trust in the academy
50. There is an appetite to face weaknesses and bring about changes for improvement in the academy
51. Good personal relationships and strong teams help us to face challenges with confidence
52. The senior leadership team has the skills and resilience to successfully work through the more complex challenges that we are facing
53. The senior leadership team is good at long term planning for capacity building (for example, successfully addressing issues like student progress across key transitions; improving the ‘tail’ of student underachievement and/or significant achievement gaps; succession planning)
195
54. As an academy, we are good at celebrating success
55. If there is one thing that you think would improve the leadership of learning in the academy, what would it be?
APPENDIX 5 FULL PERIMETA RESULTS
TABLE 146 STRAND 1 FULL RESULTS
Male students
Q Statement Time 1 Time 2
POS UNC NEG POS UNC NE
G
The learning of students - development of students as learners in Yr 7 0.53 0.30 0.17 0.80 0.16 0.04
ACADEMY 1 Enfield 0.32 0.39 0.28 0.87 0.11 0.02
ACADEMY 2 Lords Hill 0.24 0.32 0.44 0.80 0.17 0.03
ACADEMY 3 Wintringham 0.53 0.30 0.17 0.80 0.16 0.04
GENDER 1 Male 0.53 0.30 0.17 0.80 0.16 0.04
GENDER 2 Female – not included
S1 1 Changing & Learning 0.61 0.26 0.13 0.60 0.27 0.13
S1 2 Learning Relationships 0.49 0.31 0.20 0.56 0.29 0.15
S1 3 Strategic Awareness 0.40 0.32 0.28 0.74 0.20 0.06
S1 4 Resilience 0.41 0.33 0.27 0.34 0.39 0.26
S1 5 Creativity 0.38 0.33 0.29 0.70 0.22 0.08
S1 6 Meaning Making 0.67 0.24 0.08 0.57 0.28 0.15
S1 7 Critical Curiosity 0.53 0.30 0.17 0.80 0.16 0.04
Female students
Q Statement Time 1 Time 2
POS UNC NEG POS UN
C
NE
G
The learning of students - development of students as learners in Yr 7 0.44 0.33 0.24 0.80 0.17 0.03
ACADEMY 1 Enfield 0.36 0.33 0.31 0.76 0.19 0.06
ACADEMY 2 Lords Hill 0.26 0.37 0.36 0.63 0.27 0.10
ACADEMY 3 Wintringham 0.44 0.33 0.24 0.80 0.17 0.03
GENDER 1 Male – not included
GENDER 2 Female 0.44 0.33 0.24 0.80 0.17 0.03
S1 1 Changing & Learning 0.82 0.16 0.03 0.85 0.14 0.01
S1 2 Learning Relationships 0.57 0.32 0.10 0.66 0.29 0.05
S1 3 Strategic Awareness 0.60 0.27 0.13 0.71 0.25 0.04
S1 4 Resilience 0.19 0.31 0.50 0.12 0.26 0.62
S1 5 Creativity 0.39 0.42 0.19 0.82 0.15 0.03
S1 6 Meaning Making 0.47 0.42 0.11 0.83 0.16 0.02
S1 7 Critical Curiosity 0.44 0.33 0.24 0.80 0.17 0.03
TABLE 147 STRAND 2 FULL RESULTS
196
Male students
Quest Statement PO
S
UN
C
NEG
The learning of students 0.86 0.13 0.01
ACADEMY 1 Enfield 0.72 0.22 0.07
ACADEMY 2 Lords Hill 0.00 1.00 0.00
ACADEMY 3 Wintringham 0.86 0.13 0.01
GENDER 1 Male 0.86 0.13 0.01
S2 1 I feel included in the activities that are available to me in the Academy 0.53 0.33 0.15
S2 2 I feel safe in the Academy 0.40 0.60 0.00
S2 3 I feel valued and cared for as an individual in the Academy 0.79 0.18 0.04
S2 4 I like learning new things 0.79 0.18 0.04
S2 5 I usually concentrate on my learning 0.79 0.18 0.04
S2 6 I am hard to distract 0.52 0.33 0.15
S2 7 I like working with other students to help my learning 0.87 0.12 0.01
S2 8 I don’t distract other students 0.38 0.34 0.28
S2 9 I am confident in my learning 0.99 0.01 0.00
S2 10 I take a lot of care with my work 0.87 0.12 0.01
S2 11 I work hard 0.79 0.18 0.04
S2 12 I do more than I am asked to do in class 0.78 0.18 0.04
S2 13 I don’t mind making mistakes because I learn from them 0.96 0.04 0.00
S2 14 I come up with new ideas to help my learning 0.50 0.33 0.17
S2 15 I enjoy my learning 0.78 0.18 0.04
S2 16 My teachers teach well so that I learn successfully 0.96 0.04 0.00
S2 17 I have regular opportunities to express my opinion to my teachers about my learning 0.96 0.04 0.00
S2 18 I often have conversations with my teachers which help me to make better progress 0.96 0.04 0.00
S2 19 My teachers seem to enjoy teaching my classes 0.78 0.18 0.04
S2 20 My teachers often share things that they have learned themselves 0.77 0.18 0.04
S2 21 I get on well with my teachers 0.96 0.04 0.00
S2 22 I get extra support to help my learning when I need it 0.50 0.33 0.17
S2 23 I know that the senior teachers want us to learn successfully 0.79 0.18 0.04
S2 24 Everyone works very hard to make the Academy a great place to learn and to do well 0.52 0.33 0.15
S2 25 My teachers constantly expect me to improve on my personal best 0.96 0.04 0.00
S2 26 Right now I am achieving the best I possibly can 0.90 0.10 0.00
S2 27 From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy
0.87 0.12 0.01
S2 28 I have kept up a good rate of progress ever since my first year at the Academy 0.78 0.18 0.04
S2 29 I think that my results at the moment are as good as I can do 0.53 0.33 0.15
S2 30 I feel that my successes at the Academy are recognised 0.51 0.33 0.16
S2 31 I feel that my successes with activities outside the Academy are recognised 0.91 0.08 0.01
S2 32 I feel proud about what I have achieved so far 0.96 0.04 0.00
S2 33 We are taught about the responsibilities of students as well as their rights 0.78 0.18 0.04
S2 34 We care about helping each other in the Academy community 0.78 0.18 0.04
S2 35 We are encouraged to care about the needs of other people in the local community and around the world
0.79 0.18 0.04
S2 36 I regularly offer to help others 0.59 0.37 0.04
S2 37 My parents or carers feel that they are always welcome in the Academy 0.87 0.12 0.01
S2 38 My parents or carers feel involved with my learning and my progress 0.96 0.04 0.00
S2 39 My parents or carers are pleased with my progress 0.79 0.18 0.04
S2 40 My parents or carers have always been able to sort out any problems that I have had 0.52 0.33 0.15
S2 41 I am encouraged to lead activities in the classroom 0.55 0.35 0.10
S2 42 I have had opportunities to lead other activities at the Academy 0.86 0.13 0.01
Female students
Q Statement PO
S
UN
C
NEG
The learning of students 0.39 0.55 0.06
ACADEMY 1 Enfield 0.84 0.14 0.02
ACADEMY 2 Lords Hill 0.00 1.00 0.00
ACADEMY 3 Wintringham 0.39 0.55 0.06
GENDER 2 Female 0.39 0.55 0.06
S2 1 I feel included in the activities that are available to me in the Academy 0.58 0.29 0.13
S2 2 I feel safe in the Academy 0.99 0.01 0.00
197
S2 3 I feel valued and cared for as an individual in the Academy 0.25 0.33 0.42
S2 4 I like learning new things 1.00 0.00 0.00
S2 5 I usually concentrate on my learning 0.87 0.12 0.01
S2 6 I am hard to distract 0.16 0.32 0.52
S2 7 I like working with other students to help my learning 0.36 0.56 0.07
S2 8 I don’t distract other students 0.48 0.35 0.17
S2 9 I am confident in my learning 0.99 0.01 0.00
S2 10 I take a lot of care with my work 0.90 0.10 0.00
S2 11 I work hard 0.88 0.12 0.01
S2 12 I do more than I am asked to do in class 0.63 0.26 0.11
S2 13 I don’t mind making mistakes because I learn from them 0.63 0.26 0.11
S2 14 I come up with new ideas to help my learning 0.27 0.47 0.26
S2 15 I enjoy my learning 0.82 0.15 0.03
S2 16 My teachers teach well so that I learn successfully 0.70 0.22 0.08
S2 17 I have regular opportunities to express my opinion to my teachers about my learning 0.47 0.34 0.20
S2 18 I often have conversations with my teachers which help me to make better progress 0.17 0.35 0.48
S2 19 My teachers seem to enjoy teaching my classes 0.46 0.36 0.18
S2 20 My teachers often share things that they have learned themselves 0.91 0.09 0.01
S2 21 I get on well with my teachers 0.79 0.17 0.04
S2 22 I get extra support to help my learning when I need it 0.08 0.22 0.70
S2 23 I know that the senior teachers want us to learn successfully 0.97 0.03 0.00
S2 24 Everyone works very hard to make the Academy a great place to learn and to do well 0.74 0.20 0.06
S2 25 My teachers constantly expect me to improve on my personal best 0.94 0.05 0.00
S2 26 Right now I am achieving the best I possibly can 0.88 0.12 0.01
S2 27 From what I remember, I kept up a good rate of progress between my last year at primary school and my first year at the Academy
0.84 0.14 0.02
S2 28 I have kept up a good rate of progress ever since my first year at the Academy 0.81 0.15 0.03
S2 29 I think that my results at the moment are as good as I can do 0.83 0.15 0.03
S2 30 I feel that my successes at the Academy are recognised 0.66 0.24 0.09
S2 31 I feel that my successes with activities outside the Academy are recognised 0.34 0.60 0.06
S2 32 I feel proud about what I have achieved so far 0.89 0.11 0.01
S2 33 We are taught about the responsibilities of students as well as their rights 0.93 0.07 0.01
S2 34 We care about helping each other in the Academy community 0.88 0.11 0.01
S2 35 We are encouraged to care about the needs of other people in the local community and around the world
1.00 0.00 0.00
S2 36 I regularly offer to help others 0.82 0.15 0.03
S2 37 My parents or carers feel that they are always welcome in the Academy 0.95 0.05 0.00
S2 38 My parents or carers feel involved with my learning and my progress 0.73 0.20 0.07
S2 39 My parents or carers are pleased with my progress 0.89 0.11 0.00
S2 40 My parents or carers have always been able to sort out any problems that I have had 0.98 0.02 0.00
S2 41 I am encouraged to lead activities in the classroom 0.45 0.36 0.19
S2 42 I have had opportunities to lead other activities at the Academy 0.39 0.55 0.06
TABLE 148 STRAND 4 FULL RESULTS
Quest Statement POS UNC NEG
S4 Learning of students post 16 transition and progress to adulthood 0.85 0.14 0.02
ACADEMY 1 Enfield 0.85 0.14 0.02
ACADEMY 2 Lords Hill 0.00 1.00 0.00
ACADEMY 3 Wintringham 0.94 0.05 0.00
GENDER 1 Male 0.89 0.11 0.00
198
GENDER 2 Female 0.36 0.56 0.09
S4 1 1. I was valued and included during my time at the Academy 0.89 0.11 0.00
S4 2 2. I felt part of a community that was focused on learning 0.89 0.11 0.00
S4 3 3. I felt safe at the Academy 0.89 0.11 0.00
S4 4 4. I enjoyed my time at the Academy 0.83 0.15 0.03
S4 5 5. I did as well as I had hoped in my GCSEs 0.88 0.11 0.01
S4 6 6. My teachers accurately predicted my GCSE results 0.86 0.13 0.01
S4 7 7. I am looking forward to the move to Sixth Form, Further Education,
training or work
0.50 0.33 0.18
S4 8 8. I have been well prepared for the next stage of my life 0.87 0.12 0.01
S4 9 9. My successes were recognised during my time at the Academy 0.89 0.11 0.00
S4 10 10. I was helped and supported whenever I struggled with my learning 0.98 0.02 0.00
S4 11 11. I developed a sense of responsibility towards other people during
my time at the Academy
0.89 0.11 0.00
S4 12 12. The Academy helped to make me more understanding about
different cultures and types of people
0.87 0.12 0.01
S4 13 13. I supported activities that would improve things for other people 0.61 0.38 0.01
S4 14 14. I was offered opportunities for leadership during my time at the
Academy
0.85 0.14 0.02
S4 15 15. I led activities during my time at the Academy 0.49 0.34 0.17
S4 16 16. My teachers taught me well at the Academy 0.90 0.10 0.00
S4 17 17. My teachers were always trying to improve their teaching 0.87 0.12 0.01
199
S4 18 18. I was regularly challenged to do better so that I could reach my full
potential
0.89 0.11 0.00
S4 19 19. My parents/carers felt involved with my education at the Academy 0.83 0.15 0.03
S4 20 20. My parents/carers felt that they could find the right person to talk
to at the Academy if they had any concerns about my education
0.87 0.12 0.01
S4 21 21. Senior staff at the Academy wanted us to learn well and achieve
our best
0.87 0.12 0.01
S4 22 22. Everyone at the Academy worked very hard to make it a great place
to learn and to do well
0.83 0.15 0.03
S4 23 23. The Academy was well run 0.87 0.12 0.01
S4 24 24. There were great facilities for learning 0.89 0.11 0.00
S4 25 25. There was a broad curriculum that helped me to develop as a whole
person
0.89 0.11 0.01
S4 26 26. I enjoyed a wide range of opportunities for learning beyond the
timetabled curriculum
0.89 0.11 0.00
S4 27 27. The Academy made good use of new technologies for learning 0.90 0.10 0.00
S4 28 28. I never felt that anyone was putting limits on what I could achieve 0.85 0.13 0.02
S4 29 29. I feel proud about what I achieved at the Academy 0.87 0.12 0.01
S4 30 30. I could not have gone to a better place for my secondary education 0.85 0.14 0.02
TABLE 149 STRAND 5 FULL RESULTS
Quest Statement POS UNC NEG
Parents/ Carers 2013 (STRAND 5) 0.56 0.36 0.08
200
Acad1 2013 0.56 0.36 0.08
Acad2 2013 0.00 1.00 0.00
Acad3 2013 0.00 1.00 0.00
Gen1 2013 0.00 1.00 0.00
Gen2 2013 0.00 1.00 0.00
S5 1 1. My child is known and valued as an individual 0.87 0.12 0.01
S5 2 2. The Academy is providing for my child’s particular needs 0.37 0.57 0.06
S5 3 3. My child feels included in the life of the Academy, in classrooms and
in other activities
0.35 0.55 0.10
S5 4 4. The Academy focuses its work on providing the best opportunities
for learning for all students
0.37 0.57 0.06
S5 5 5. My child is taught well 0.59 0.37 0.04
S5 6 6. My child is being challenged to improve on previous best
performance in all areas
0.59 0.37 0.04
S5 7 7. My child’s progress at the Academy has exceeded our expectations 0.43 0.30 0.27
S5 8 8. My child’s successes are recognised 0.59 0.37 0.04
S5 9 9. The Academy ensures my child is well looked after 0.59 0.37 0.04
S5 10 10. The Academy helps my child to have a healthy lifestyle 0.37 0.57 0.07
S5 11 11. There is a good standard of behaviour at the Academy 0.50 0.34 0.15
S5 12 12. My child is developing an attitude of helping others (for example,
by doing practical activities to improve life at the Academy, by
supporting people in need, getting involved with community projects,
fund-raising)
0.58 0.36 0.06
201
S5 13 13. My child is taking opportunities to lead activities (for example,
leading group work in the classroom, leading a sports team, giving
feedback to teachers about their learning, being a representative on a
Student Council, taking a lead role in Academy performances of music,
drama or dance, being a coach, mentor or ‘buddy’)
0.24 0.43 0.33
S5 14 14. My child has kept up a good rate of progress at key transitions (for
example, from primary to secondary school; from Year 9 to GCSE
courses in Years 10 & 11; from Year 11 to Sixth Form/College; from Year
13 to further education/higher education/employment)
0.35 0.55 0.10
S5 15 15. I get regular and helpful information about my child’s progress 0.59 0.37 0.04
S5 16 16. The Academy responds well to my concerns 0.59 0.36 0.05
S5 17 17. I always find staff knowledgeable about my child 0.59 0.36 0.05
S5 18 18. I always find staff helpful when discussing my child’s progress 0.59 0.36 0.05
S5 19 19. The Academy helps me to support my child’s learning 0.59 0.37 0.04
S5 20 20. The Academy is led and managed effectively 0.59 0.37 0.04
S5 21 21. My impression is that senior staff focus their work on making the
Academy a good place to learn
0.59 0.37 0.04
S5 22 22. My impression is that senior staff expect all students to achieve
well
0.59 0.37 0.04
S5 23 23. My child enjoys their education at the Academy 0.39 0.59 0.02
S5 24 24. I would recommend the Academy to another parent/carer 0.56 0.36 0.08
S5 25 (25)26. The Academy has been helpful in ensuring that I understand the
general information sent to parents/carers
0.00 1.00 0.00
S5 26 (26)27. The Academy has been helpful in ensuring that I understand the
information about my child’s progress
0.00 1.00 0.00
S5 27 (27)28. I feel that the Academy understands and takes proper account 0.00 1.00 0.00
202
of the particular issues and sensitivities of my culture
S5 28 (28)29. If I have a concern, I feel confident that I can make contact with
relevant staff
0.00 1.00 0.00
S5 29 (29)30. If I have a concern, I feel confident that I can communicate with
staff effectively
0.00 1.00 0.00
S5 30 (30)31. I would recommend the Academy to another parent/carer from
my community
0.00 1.00 0.00
TABLE 150 STRAND 6 FULL RESULTS
Quest Statement POS UNC NEG
ECHO Project Strand 6 – The Learning of Teachers 0.90 0.10 0.00
ACADEMY 1 Enfield 0.90 0.10 0.00
ACADEMY 2 Lords Hill 0.00 1.00 0.00
ACADEMY 3 Wintringham 0.00 1.00 0.00
GENDER 1 Male 0.94 0.05 0.00
GENDER 2 Female 0.89 0.11 0.00
S6 A1 Staff as well as students learn in this academy 0.92 0.07 0.01
S6 A2 I draw on good practice from other schools as a means to further my
own professional development
0.34 0.43 0.24
S6 A3 I read research reports as one source of useful ideas for improving my
practice
0.31 0.32 0.37
S6 A4 I use the web as one source of useful ideas for improving my practice 0.77 0.19 0.05
S6 A5 I consult my students about how they learn most effectively 0.59 0.37 0.04
203
S6 A6 I relate what works in my own practice to research findings 0.78 0.18 0.04
S6 A7 I am able to see how practices that work in one context might be
adapted to other contexts
0.85 0.13 0.02
S6 A8 I use insights from my own professional learning to feed into academy
policy development
0.52 0.33 0.14
S6 A9 I reflect on my practice as a way of identifying my professional
learning needs
0.86 0.13 0.02
S6 A10 I experiment with my practice as a conscious strategy for improving
classroom teaching and learning
1.00 0.00 0.00
S6 A11 I modify my practice in the light of feedback from my students 0.89 0.11 0.00
S6 A12 I modify my practice in the light of published research evidence 0.52 0.33 0.16
S6 A13 I modify my practice in the light of evidence from self-evaluations of
my classroom practice
0.86 0.13 0.02
S6 A14 I modify my practice in the light of evidence from evaluations of my
classroom practice by managers or other colleagues
0.87 0.12 0.01
S6 A15 I carry out joint research/evaluation with one or more colleagues as a
way of improving my practice
0.33 0.31 0.36
S6 A16 I collaborate to plan my teaching 0.83 0.15 0.03
S6 A17 I observe other colleagues in the classroom and we give each other
feedback
0.58 0.36 0.06
S6 A18 I engage in team teaching as a way of improving practice 0.29 0.49 0.22
S6 A19 If I have a problem with my teaching I turn to colleagues for help 0.85 0.14 0.02
S6 A20 I suggest ideas or approaches for colleagues to try in class 0.35 0.55 0.10
S6 A21 I make collective agreements to test out new ideas 0.51 0.33 0.16
204
S6 A22 I discuss openly with colleagues what and how they are learning 0.59 0.36 0.05
S6 A23 I use informal opportunities to discuss how children learn 0.51 0.33 0.16
S6 A24 I offer reassurance and support to colleagues 0.79 0.18 0.04
S6 A25 My practice is based on the belief that all students are capable of
learning
1.00 0.00 0.00
S6 A26 My experience is that students in this academy enjoy learning 0.77 0.18 0.05
S6 A27 My experience is that student success is celebrated 0.87 0.12 0.01
S6 A28 I discuss with colleagues how students might be helped to learn how
to learn
0.62 0.38 0.01
S6 B1 Senior leaders communicate a clear vision of where the academy is
going
0.99 0.01 0.00
S6 B2 Staff have a commitment to the whole academy as well as to their
department, key stage and/or year group
0.92 0.08 0.00
S6 B3 Senior leaders promote commitment among staff to the whole
academy as well as to the department, key stage and/or year group
0.19 0.37 0.44
S6 B4 There is effective communication between senior leaders and
teachers
0.58 0.29 0.13
S6 B5 There are processes for involving all staff in decision-making 0.57 0.29 0.14
S6 B6 Teachers’ professional know-how is used in the formulation of
academy policy and goals
0.87 0.12 0.01
S6 B7 Teachers’ professional know-how is used in the formulation of
academy policy, even where this leads to a questioning of established
rules, procedures and practices
0.52 0.32 0.16
S6 B8 Opportunities are provided for teachers to critically evaluate academy
policy
0.64 0.33 0.03
205
S6 B9 Staff are involved in evaluating academy policy 0.32 0.43 0.25
S6 B10 Staff participate in important decision-making 0.34 0.33 0.32
S6 B11 There are processes for involving students in decision-making 0.56 0.30 0.14
S6 B12 Staff have a good working knowledge of the Academy Development
Plan (or your academy’s equivalent document)
0.85 0.13 0.02
S6 B13 Staff see the Academy Development Plan as relevant and useful to
learning and teaching
0.53 0.35 0.12
S6 B14 Staff development time is used effectively to realise Academy
Development Plan priorities
0.62 0.27 0.12
S6 B15 Staff development time is used effectively in the academy 0.04 0.16 0.81
S6 B16 The academy provides cover to allow staff joint planning time 0.90 0.10 0.00
S6 B17 Teachers are encouraged to experiment with new ideas as a way of
promoting professional growth
0.84 0.14 0.02
S6 B18 Formal training provides opportunities for teachers to develop
professionally
0.85 0.13 0.02
S6 B19 Teachers are helped to develop skills to assess students’ work in ways
that move students on in their learning
0.52 0.35 0.13
S6 B20 Teachers are helped to develop skills to observe learning as it happens
in the classroom
0.44 0.37 0.19
S6 B21 Senior leaders support teachers in sharing practice with other schools
through networking
0.53 0.32 0.16
S6 B22 Information is collected from teachers on those aspects of their work
that they themselves think they do effectively
0.87 0.12 0.01
S6 B23 Information is collected from teachers on effective ways they promote
learning to learn skills and knowledge among their students
0.14 0.35 0.51
206
S6 B24 Information is collected from teachers on informal teacher networking
in which they play an active role
0.22 0.33 0.45
S6 B25 Teacher-initiated networking is an integral element of staff
development
0.22 0.40 0.38
S6 C1 I feel included in a community that values what I can offer to the staff
team and to students
0.99 0.01 0.00
S6 C2 I believe that I am working in a community that focuses our attention
and activities on learning to improve teaching and student outcomes
0.95 0.05 0.00
S6 C3 My experience suggests that other members of staff are involved in
training and development activities to improve their classroom
practice and student outcomes
0.65 0.25 0.10
S6 C4 I have had opportunities to work alongside and/or train with
colleagues in dother phases to improve students’ progress
0.99 0.01 0.00
S6 C5 My experience is that senior leaders are role models as ‘leaders of
learning’ in the academy
0.98 0.02 0.00
S6 C6 Leadership is shared in this academy and opportunities to take on
leadership roles are available to all those that want them and have the
relevant knowledge and experience
0.81 0.16 0.04
S6 C7 My experience is that the academy promotes a balance between the
rights and the responsibilities of all members of the community
0.72 0.21 0.07
S6 C8 There is an emphasis among students and staff on caring about and
helping other people in the academy community
0.90 0.09 0.01
S6 C9 There is an emphasis among students and staff on caring about and
helping other people in the wider community
0.91 0.09 0.01
S6 C10 The academy is successful in helping parents/carers to support the
learning of their children
0.40 0.33 0.26
S6 C11 The achievements of staff are recognised and celebrated 0.52 0.35 0.14
S6 C12 I believe that I am realising my full potential as a teacher 0.90 0.10 0.00
207
TABLE 151 STRAND 7 FULL RESULTS
Quest Statement POS UNC NEG
ECHO Strand 7: Leadership of learning 0.90 0.10 0.00
ACADEMY 1 Enfield 0.98 0.02 0.00
ACADEMY 2 Lords Hill 0.90 0.10 0.00
ACADEMY 3 Wintringham 0.83 0.14 0.03
GENDER 1 Male 0.00 1.00 0.00
GENDER 2 Female 0.00 1.00 0.00
S7 1 1. I have a good understanding of my own and other people’s
learning
1.00 0.00 0.00
S7 2 2. I keep my knowledge about teaching and learning and about my
phase and/or subject specialism up to date
1.00 0.01 0.00
S7 3 3. I teach regularly and my lessons are rated good or outstanding 0.81 0.18 0.01
S7 4 4. I undertake regular and substantial professional development for
myself
0.89 0.11 0.01
S7 5 5. I understand the differences between good and outstanding
lessons
1.00 0.00 0.00
S7 6 6. I regularly work with staff to improve their teaching practice 0.77 0.18 0.05
S7 7 7. I can design, monitor and assess programmes for effective
teaching and learning
0.89 0.11 0.01
S7 8 8. I regularly participate in critical and reflective enquiry to develop
teaching and learning across the academy
0.37 0.57 0.07
S7 9 9. I understand what an academy needs to do to become good and 0.90 0.10 0.00
208
outstanding
S7 10 10. I regularly work with staff to help them improve their leadership
and management
0.89 0.11 0.00
S7 11 11. I am an effective coach and use my coaching skills regularly to
help others to improve their performance
0.89 0.10 0.00
S7 12 12. I regularly interact with staff in other schools, academies and/or
organisations to build learning partnerships which connect staff and
students to learning opportunities beyond our academy
0.29 0.48 0.24
S7 13 13. I make regular and effective contributions to educational dialogue
that shapes whole academy policy and informs practice
0.87 0.12 0.01
S7 14 14. I contribute effectively to the critical development of academy
systems and structures in order to ensure quality teaching focused on
student progress in learning
0.89 0.11 0.00
S7 15 15. The senior leadership team communicates a clear vision of where
the academy is going so that there is a shared sense of purpose
0.90 0.10 0.00
S7 16 16. Staff have a commitment to the whole academy as well as to
their department, key stage and/or year group
0.89 0.11 0.00
S7 17 17. The senior leadership team effectively promotes commitment
among staff to the future development of the whole academy
0.90 0.10 0.00
S7 18 18. There is effective communication between senior leaders and
teachers
0.61 0.38 0.01
S7 19 19. There are processes for involving all staff in decision-making 0.61 0.38 0.01
S7 20 20. Teachers’ professional know-how is used in the formulation of
academy policy and goals
0.61 0.38 0.02
S7 21 21. Teachers’ professional know-how is used in the formulation of
academy policy, even where this leads to a questioning of established
rules, procedures and practices
0.57 0.36 0.07
S7 22 22. Opportunities are provided for teachers to critically evaluate 0.58 0.36 0.06
209
academy policy
S7 23 23. Staff participate in important decision-making 0.61 0.38 0.02
S7 24 24. There are processes for involving students in decision-making,
including how best to improve classroom practice
0.90 0.10 0.00
S7 25 25. Staff have a good working knowledge of the Academy
Development Plan
0.61 0.38 0.01
S7 26 26. Staff see the Academy Development Plan as relevant and
useful to learning and teaching
0.61 0.38 0.02
S7 27 27. Staff development time is used effectively to realise Academy
Development Plan priorities
0.73 0.20 0.06
S7 28 28. Staff development time is used effectively in the academy,
particularly through the explicit modelling of good practice
0.59 0.36 0.05
S7 29 29. The academy provides support to allow staff joint planning
time
0.57 0.36 0.07
S7 30 30. Teachers are encouraged to experiment with new ideas as a
way of promoting professional growth and improving classroom
practice
0.85 0.13 0.02
S7 31 31. Formal training provides opportunities for teachers to develop
professionally
0.86 0.13 0.02
S7 32 32. Teachers are helped to develop skills to assess students’ work
in ways that move their students on in their learning
0.58 0.36 0.06
S7 33 33. Teachers are helped to develop skills to observe learning as it
happens in the classroom
0.57 0.36 0.07
S7 34 34. Senior leaders support teachers in sharing practice with other
schools and academies through networking
0.57 0.36 0.07
S7 35 35. Information is collected from teachers on those aspects of
their work that they themselves think they do effectively
0.87 0.12 0.01
210
S7 36 36. Information is collected from teachers on effective ways they
promote ‘learning to learn’ skills and knowledge among their students
0.80 0.16 0.04
S7 37 37. Information is collected from teachers on informal teacher
networking in which they play an active role
0.79 0.17 0.04
S7 38 38. Teacher-initiated networking is an integral element of staff
development
0.79 0.17 0.04
S7 39 39. ‘Learning how to learn’ is an issue discussed in staff
development time
0.52 0.35 0.13
S7 40 40. There is a well-developed shared language and a rich dialogue
about teaching and learning across the academy
0.89 0.11 0.00
S7 41 41. We are good at systematic monitoring, use of data and
feedback across the academy
0.59 0.36 0.05
S7 42 42. We have efficient and effective support systems and
structures across the academy
0.90 0.10 0.00
S7 43 43. The senior leadership team are effective role models as
‘leaders of learning’, (for example, by securing broad commitment to
learning for all that is focused on improving the quality of teaching
and student outcomes)
0.58 0.36 0.06
S7 44 44. We have developed successful approaches to distributed
leadership across the academy where the pathways of accountability
are working well
0.86 0.13 0.02
S7 45 45. The academy is successful in helping parents/carers to support
the learning of their children
0.57 0.36 0.08
S7 46 46. The governors play a full and appropriate part in strategic
planning to improve teaching and learning and student outcomes
0.57 0.36 0.07
S7 47 47. The academy involves a wide range of other stakeholders
directly and powerfully in the education of students
0.61 0.38 0.01
S7 48 48. The academy is good at securing the broadest possible impact
of successful innovations in teaching and learning
0.57 0.36 0.07
211
S7 49 49. There is a widespread and well-established climate of trust in
the academy
0.85 0.13 0.02
S7 50 50. There is an appetite to face weaknesses and bring about
changes for improvement in the academy
0.85 0.13 0.02
S7 51 51. Good personal relationships and strong teams help us to face
challenges with confidence
0.98 0.02 0.00
S7 52 52. The senior leadership team has the skills and resilience to
successfully work through the more complex challenges that we are
facing
0.90 0.10 0.00
S7 53 53. The senior leadership team is good at long term planning for
capacity building (for example, successfully addressing issues like
student progress across key transitions; improving the ‘tail’ of student
underachievement and/or significant achievement gaps; succession
planning)
0.90 0.10 0.00
S7 54 54. As an academy, we are good at celebrating success 0.90 0.10 0.00
APPENDIX 6
EXTRACT FROM DAVIS AND FLETCHER (2000)
The method [of propagation of evidence] is based on the Interval Probability Theory of Cui & Blockley (1990). This itself
is a development of earlier ideas of using intervals by Dempster (1969) and Shafer (1976).
Consider the conjecture, H, that a process is going to be successful. In the ideal situation the dependability of a process
to deliver its objectives is certain. However in reality the certainty depends on the dependability of its sub-processes
about which we need evidence. Suppose then also that there is evidence from a sub-process E. We are concerned with
the probability P(H) that H will be successful. The situation is illustrated using the Venn diagrams in Figure A.
Interestingly, the last condition (c) is the most general and can be considered as a generalisation of (b) since the
evidence is now only partially sufficient. The computation of the contribution of the evidence E to the conjecture H has
been implemented in the method of Hall et al (1998), considering only the accumulation of evidence from the sub-
processes adding to the knowledge about H.
P(H) = P(H | E).P(E)
It is necessary to look at what happens when two or more pieces of evidence from two or more sub-processes are
accumulated. It can be seen from Figure B that there are a number of variables that need to be known to deduce the
212
probability of the success of the process (conjecture H). Firstly, the probability of the evidence itself being dependable
is required. This may be elicited directly or it may come from its own daughter processes. This is P(E). Then the
relevance or level of ‘sufficiency’ of E1 and E2 to H, or P(H | E)is needed.
When there is more than one sub-process, the dependency or overlap between the two is needed. This is so that
common evidence is not counted twice. In the case of the total probability method of Hall et al (1998), further
judgements are required to specify the P(H | E)terms.
The necessary condition expressed in Figure A comes out naturally as part of the total probability approach in that if for
example both E1 and E2 are necessary conditions, then:
where the value [0,0] represents the interval probability with meaning no support for the conjecture and full support
against, i.e. since E1 and E2 are necessary, anything not E1 and not E2 has no contribution to make.
The method implemented in the software has no way of expressing this, and so a heuristic has been introduced to
express the weight of the necessary condition (Figure A) as no P(H | E)terms are used. The significance of the
heuristic in practice is to recognise strong evidence against the success of a process that may lead ultimately to the
failure of the parent process.
The algorithms used in the software uses the heuristic described above and only single probability numbers for the
expressions of ‘sufficiency’ and ‘dependency ‘and a Boolean operator for necessity. The advantages [over other
algorithms] are that process models can be built and explored very quickly and have been found in practice to be
powerful tools for illuminating the decision making process.
Figure A: Venn diagrams of the logical conditions (a) E necessary for H, (b) E sufficient for E, and (c) E relevant to H respectively.
Figure B: The contributions of two sub-processes E1 & E2 to the conjecture H.