Evaluation of InCAS Computer-Adaptive Assessments December … · 2019. 6. 26. · 83% (n = 847) of teachers administered InCAS to pupils with Special Educational Needs (SEN). Of

Evaluation of InCASComputer-Adaptive Assessments

Primary Pupil Profile

December 2007

Scope of the Report

The aim of this phase of the research was to collect information about Year 5 teachers’

experiences of InCAS computer-based assessment and to evaluate support for the

programme.

Outcomes of this evaluation will be used to inform future planning and opportunities for

development.

This report does not include a full evaluation of the use of InCAS with Special

Educational Needs or Irish Medium pupils. However, separate evaluations will be

taking place for these areas. A pupil evaluation is currently being collated and will be

published by the end of June 2008.

Prepared by Research & Statistics KW/2011/0708 1

CONTENTS

Section Page

1.0

Executive Summary

3

2.0

Introduction

6

3.0

Method

8

4.0

Results

10

4.1 Previous experience of delivering the Pupil Profile using InCAS 10

4.2 Evaluation of training on InCAS assessment data 25

5.0

Conclusion

34

6.0

Appendices

6.1 Method of consultation 37


1.0 Executive Summary

Rationale

This report presents the findings from the second phase of the implementation of the

Primary Pupil Profile in Year 5. The aim of this phase of the research was to collect

information on Year 5 teachers experiences with InCAS, and conduct a thorough

evaluation of the 36 training sessions on the InCAS assessment data (facilitated

throughout October and November 2007).

Information gathered from this phase of the evaluation will help inform developments of

the InCAS assessments and future training facilitated by the Pupil Profile

Implementation Team.

Key Findings

Findings from this evaluation have been very mixed and even conflicting at times.

While teachers rated most aspects of the InCAS assessments highly, they also stressed

the need for adaptations and made recommendations. Below is a summary of the

findings. This focuses on respondents’ previous experiences of delivering the Pupil

Profile using InCAS and their evaluation of the training on interpreting the InCAS

assessment data.

To date, 80.8% of teachers (n = 743) found implementing the Pupil Profile a positive

experience. Over 82% of teachers (82.5%, n = 696) also felt that the feedback from the

InCAS assessments was consistent with their professional judgement. 59.3% (n = 512)

believed that the feedback provided them with supplementary information about their

pupils. The majority of respondents rated all the Pupil Profile resources as useful.

Over 95% of teachers deemed the InCAS assessments to be user-friendly for pupils

(95.5%, n = 961). Many said that pupils found the assessments enjoyable.


83% (n = 847) of teachers administered InCAS to pupils with Special Educational Needs

(SEN). Of these respondents, 86.3% (n = 643) believed that such pupils found the

assessments manageable. 20% of respondents (n = 177) also administered InCAS to

pupils with English as an additional language. Of these, 72.6% (n = 127) felt that such

pupils found the assessments manageable.

Teachers rated the training on the InCAS assessment data very highly. The majority of

respondents rated the training in the use of Standard Feedback (97.5%, n = 1113) and

Custom Feedback (97.7%, n = 1116) as useful and are now confident at interpreting

both forms of feedback (>96%). Furthermore, since taking part in the training, a large

number of respondents indicated that they intend to make use of the five additional

modules offered by InCAS. The three most popular modules were: Mental Arithmetic

(32.2%, n = 382); Spelling (30.2%, n = 337); and Developed Ability (19.8%, n = 228).

Despite such positive feedback, some respondents highlighted areas of concern and

dissatisfaction. A large number of respondents (48.8%, n = 500) experienced difficulty

accessing InCAS on their C2K networked computers. The main problems concerned:

the installation of InCAS; pupil’s passwords; wireless access; and difficulty adjusting the

volume.

65.8% of respondents (n = 506) indicated that they would like additional support or

resources. The most requested support and resources includes: preparation time

(n = 124); additional computers/laptops (n = 93); further guidance on InCAS (n = 67);

and mice for laptops (n = 67).

Despite 86.3% of teachers indicating that InCAS was manageable for pupils with

Special Educational Needs, 47.6% of respondents (n = 271) felt adaptations were

required for the assessments to be suitable for these pupils. Suggested adaptations

include: simplifying questions (n = 46); additional time (n = 37); and larger font (n = 33).


The Irish Medium Maths version of InCAS was not used by any teachers. This was due

to the software not being available on the managed service until after the last training

event. CCEA had completed the translation by early September. The lead-in was a

consequence of testing the software on the C2K managed service.

Following the training, 81% of teachers indicated that they felt confident reporting

assessment feedback to parents. However many of the comments made throughout

both questionnaires suggest that this aspect of the Pupil Profile process is the one

teachers are most apprehensive about.

Recommendations

1. Continue to work in partnership with CEM and C2K to determine how technical

difficulties identified within this report can be prevented from occurring in the

future.

2. Provide schools with mice for laptops.

3. Keep under review any adaptations which may be required for pupils with SEN,

focusing on the six main categories of Special Educational Needs.

4. Inform the Department of Education of teachers’ request for a guidance resource

on sharing InCAS feedback with parents and requests for additional support and

resources (for example computers, mice and preparation time).


2.0 Introduction

The implementation of the Pupil Profile process commenced in September 2007 with

the InCAS computer-based diagnostic assessments made available to all primary

schools in Northern Ireland. InCAS was developed by the Curriculum, Evaluation and

Management Centre at Durham University and is an adaptive computer-based

assessment. Computer-adaptive assessments are tailored to match a pupil’s ability.

The assessment programme generates items depending on the pupil’s response to a

previous number of items. The difficulty of subsequent assessment items presented to

the pupil depends upon their responses as they proceed through the assessment. The

adaptability of this assessment method means that learners are not faced with the de-

motivating experience of being unable to answer many of the items (which is often the

case with pencil and paper tests).

The InCAS computer-based assessment consists of 11 modules. By the end of Autumn

term 2007, all primary schools in Northern Ireland were required to administer four

modules to Year 5 pupils to produce both a Reading age and General Maths age. The

Reading age is achieved on the completion of the Word Recognition, Word Decoding

and Comprehension modules. The General Maths module generates the General

Maths Age. Teachers can also make full use of the other seven modules within the

InCAS assessment tool (Spelling, Mental Arithmetic, Picture Vocabulary, Non-Verbal

Ability, Attitude to Maths, Attitude to Reading and Attitude to School).

The Department of Education has determined that 2007/2008 is a transition year for

new assessment and reporting arrangements. During this time outcomes of the InCAS

assessments will only be used internally in schools. The information from these

assessments does not have to be formally reported to parents in the first year of use.

All Year 5 teachers were provided with one day’s training on interpreting the

assessment data produced from the InCAS assessments. This training ran throughout

October and November 2007. The training was scheduled to allow teachers the


opportunity to be trained before meeting with parents (if the school policy was to have

this parent meeting in the transition year). Teachers were encouraged to bring their

class assessment outcomes to the training to analyse and interpret.

CCEA’s Research & Statistics Unit is currently undertaking a full evaluation of the

implementation of the Primary Pupil Profile. This phase of the evaluation focuses on

the implementation and support of the InCAS assessments.

Information collected from this evaluation will be used to assess if:

• InCAS software and support materials are manageable for schools.

• InCAS feedback provides teachers with useful information about their pupils/classes.

• Teachers are suitably trained and have a clear understanding of the assessment

software.

• Teachers can interpret the results and are confident to provide feedback to parents.

• InCAS assessments are suitable for pupils with Special Educational Needs (SEN) or

English as an additional language.


3.0 Method

3.1 Design

The purpose of this study was to gather the views and opinions of all teachers currently

implementing InCAS assessments to Year 5 pupils. Information was collected via

two questionnaires, both of which contained qualitative and quantitative questions.

SPSS 14.0 was used to analyse the quantitative data. Qualitative data from

open-ended questions were categorised using thematic content coding.

3.2 Participants

All Year 5 teachers in Northern Ireland were invited to attend training on InCAS

assessment data. A total of 1425 teachers attended the training. 80.2% (n = 1143)

completed a questionnaire on their experiences of InCAS prior to the training. At the

end of the training day, 80.8% (n = 1151) completed an evaluation questionnaire on the

InCAS assessment data training.

The breakdown of respondents can be seen in Table 1 below.

Table 1: Breakdown of Respondents

Responses Sector Number %

Teacher 915 80.0 Principal 112 9.8 Vice Principal 71 6.2 Assessment Coordinator 21 1.8 SENCo 24 2.1 Other 1 0.1 Total 1143 100.0

88 respondents indicated that they had previously participated in an InCAS trial with

CCEA and 59 had taken part in a Pupil Profile trial.


3.3 Breakdown of teachers’ classes by year group

40.3% of respondents teach a composite class (n = 441). The majority of these

teachers have the following year groups in their class:

• Year 4 and 5 (35.3%, n = 151).

• Year 5 and 6 (28%, n = 120).

• Year 5 to 7 (14.5%, n = 62).

• Year 3 to 5 (10.7%, n = 46).

Within the majority of respondents’ schools there is either one or two Year 5 teachers

(79%, n = 854). 18.6% (n = 201) of respondents’ schools have between two and four

Year 5 teachers, and 2.4% (n = 26) have between five and nine Year 5 teachers.

3.4 Organisation of school computers

71.8% (n = 821) of respondents have a computer in their classroom. 57.6% (n = 658)

have a computer suite and 12% of teachers have access to a computer cluster.


4.0 Results

4.1 Previous Experience of Delivering the Pupil Profile Using InCAS This section presents the results for each question from the questionnaire. Results are

based on questionnaires from 1143 respondents. Percentages relate to the number of

teachers who answered each question, which may be fewer than 1143. The main

qualitative issues to support the responses given are also highlighted. A number of

respondents reiterated comments throughout their questionnaire; as a result you will

find some comments repeated throughout this section of the report.

Pupil Profile 80.8% of teachers (n = 743) found implementing the Pupil Profile a positive experience.

A large number of teachers commented that pupils enjoyed completing the InCAS

assessments (153 comments). However teachers identified a number of aspects with

the first stage of the Pupil Profile process (InCAS assessments) with which they felt

dissatisfied. These are listed below:

• InCAS was very time consuming to set up (71 comments).

• School does not have enough available computers (26 comments).

• No wireless access on laptops (22 comments).

• Computers not commissioned by C2K (21 comments).

• Heavy workload for the start of the academic year (14 comments).

• Sub cover required (14 comments).

• Sound on the computers is difficult to adjust (14 comments).

• Laptops did not work (13 comments).

• Classroom management difficult in a composite class (11 comments).

• Pupils could not use touch pad on laptops and require a mouse (10 comments).

• Difficulties uploading results (9 comments).

• Computers froze during some pupils assessments (8 comments).

• Laptops were delivered too late (8 comments).


Using InCAS Just under half of the respondents (48.8%, n = 500) indicated that they had experienced

difficulty accessing InCAS on their school’s C2K networked computers. Many teachers

commented on the various technical problems they encountered. These are

summarised below:

• Experienced difficulties installing InCAS onto computers (84 comments).

• Pupils were unable to log on using their passwords (76 comments).

• Wireless access was poor on the laptops (37 comments).

• Experienced difficulty adjusting sound (35 comments).

• Laptops did not function (33 comments).

• Computers crashed during assessments (28 comments).

• Experienced difficulty uploading data (21 comments).

• InCAS icon disappeared from desktop (20 comments).

• Not aware teachers had to exit ADMIN to ensure pupils logged onto system

(18 comments).

• Computer froze during assessments (16 comments).

• Downloading SIMS data was troublesome (11 comments).

• No mice with laptops (9 comments).

• School computers were not on C2K network (4 comments).

• Logged call with C2K but still have not had problem rectified (4 comments).

Respondents were asked if they had received the additional hardware being rolled out

by C2K before administering the assessments. The majority of teachers indicated that

they had received additional computers/laptops (90.9%, n = 949) and headphones

(90.8%, n = 943) before administering the assessments. However 42.7% of

respondents (n = 349) who answered this question had not received ‘Lot 6 refresh’ and

over a third of respondents (33.6%, n = 323) had not received wireless access before

administering the InCAS assessments.


InCAS Feedback

Figure 1 indicates that over half of the

respondents felt either very confident or

confident interpreting the InCAS

feedback (57%, n = 534). A number of

the respondents commented that the

feedback was easy to interpret

Figure 1: Confidence Interpreting InCAS Feedback

6.9

33.2

9.8

50.1

0

20

40

60

80

Veryconfident

Confident Notconfident

Not at allconfident

% o

f Res

pond

ents

(75 comments). However 33.2%

(n = 311) did not feel confident at

interpreting the feedback and 9.8% were

not at all confident (n = 92). A large number of teachers stated that they had not yet interpreted the data

(129 comments) and this is likely to have had an impact on their level of confidence.

Comments also showed that teachers would like further training on interpreting the data

and understanding the terminology (93 comments).

82.5% of teachers (n = 696) felt the feedback from the InCAS assessments was

consistent with their professional judgement. Many teachers stated that there were “no

surprises” with the feedback as it was “similar to what [they] would have predicted”

(121 comments). Nevertheless a large number of comments suggest that there were

discrepancies with the InCAS feedback; “some of the results were surprising and gave

food for thought on the pupil’s present situation academically”. 101 respondents

commented that some pupils received lower scores than they expected and 89 stated

that some of their pupils received higher scores than they would have expected.


Nearly 60% of teachers believed that the InCAS feedback had provided them with

supplementary information about their pupils (59.3%, n = 512). Such teachers

commented that the feedback provided them with the following additional information:

• Breakdown of literacy scores (135 comments).

• Highlighted areas pupils are having difficulty with (48 comments).

• Word Recognition and Decoding identified pupils with poor phonic skills

(40 comments).

• Pupils mental mathematics ability (29 comments).

• Information to compare with pupils standardised score (6 comments).

Over 75% of respondents (75.6%, n = 534) felt InCAS’s diagnostic information was

helpful in informing learning and teaching. Many of these teachers commented that it:

identified pupils’ strengths and weaknesses (98 comments); confirmed what teachers

already know (35 comments); and provided a detailed breakdown of information in the

Reading section (32 comments).

However a sizeable number of teachers did not find this information helpful

(24.4%, n = 172). Many of these teachers made the following comments:

• Would like a more detailed breakdown for Mathematics, separated into topic areas

(43 comments).

• Unsure of the reliability of the test (18 comments).

• Require further training on InCAS feedback (16 comments).

• Require more information on pupils specific areas of weakness (13 comments).


Resources & Support Respondents were asked to rate the Pupil Profile resources they had used. The

resources that the greatest number of respondents had used included the Overview of

the Pupil Profile (39.8%, n = 455) and the Pupil Profile Helpdesk

(35.3%, n = 404). Resources least used by respondents included the Classroom

Management Video Case Studies (15.4%, n = 176), the SIMS Wizard (17.5%, n = 200)

and the Frequently Asked Questions (FAQ’s) (19.8%. 226).

Figure 2: % of Respondents Rating Resources as Useful

94.586.5 85.8

68.2

92.895.293.3

0

20

40

60

80

100

PP Helpdesk Overview ofthe PP

InCAS onlinetutorials

Classroommanagementvideo case

studies

Pupil Demo SIMS Wizard FAQ

% o

f Res

pond

ents

Figure 2 illustrates that the majority of respondents rated the resources for the Pupil

Profile as useful. The resource respondents deemed to be least useful was the

Classroom Management Video Case Studies, with 31.9% of teachers rating this

resource as not useful or not at all useful (n = 56). However it is important to note that

this resource was the least used support material, accessed by only 176 teachers.

The majority of comments written about the Pupil Profile and InCAS resources were

positive. Teachers commended the usefulness of the: Pupil Profile Helpdesk (38

comments); Pupil Demo (31 comments); Pupil Profile/InCAS booklet (28 comments);

and the online resources (14 comments). The only criticism was from teachers who


were not aware of the existence of these resources. A small number of teachers made

the following recommendations:

• Pupil demo should contain more questions for each example (12 comments).

• Training should take place closer to September (4 comments).

• “It would be useful to have a problem checklist or guideline to help if problems

occur, e.g. children hitting escape button” (1 comment).

Respondents were also asked to rate the support they have received from CCEA to

date. 88.2% of teachers indicated that they were satisfied with the support they have

received from CCEA. Many teachers praised the support they have received from the

Pupil Profile training (43 comments), resources (65 comments) and CCEA staff, mostly

through the Helpdesk (65 comments). Nevertheless a number of respondents had the

following concerns:

• Timing of training was inappropriate (24 comments).

• Have not received any training to date (18 comments).

• Require further training on interpreting results (15 comments).

• Difficulty getting through to the Helpdesk (13 comments).

• Require training on providing feedback/information to parents (10 comments).

• Sub cover is required for teachers to implement InCAS in school (9 comments).


Nearly two thirds of respondents said they would like additional support or resources

(65.8%, n = 506). A summary of the support and resources teachers requested is listed

below:

• Time out of class - sub cover (124 comments).

• Additional computers/laptops (93 comments).

• Further guidance on InCAS (67 comments)

o Interpreting data/results (n = 49)

o Reporting feedback to parents (n = 20)

o How to help children with problems (n = 5)

o What to do with the information (n = 5).

• Mice for laptops (67 comments).

• More time to administer InCAS and interpret results (39 comments).

• InCAS to be more developed (30 comments)

o More detailed breakdown of results, especially for Numeracy/Maths (n = 17)

o Option to print graphs/tables on one page (n = 3)

o A sort/filter option applied to results (n = 2).

• Classroom assistant required for classroom management (27 comments).

• Sample test for pupils (24 comments).

• Headphone/earphones that can adjust sound (24 comments).

• Continued ICT support (20 comments).


Parental Involvement Over a third of respondents (37%, n = 388) informed parents about the InCAS

assessments before administering them to pupils. A large number of teachers

communicated this information in the following ways:

• Discussed at a parent teacher evening / revised curriculum evening

(182 comments).

• In writing (61 comments).

• Informally - mentioned briefly in passing (41 comments).

• Held a specific evening about InCAS (18 comments).

• Demos/examples were shown to parents (7 comments).

A sizeable number of teachers (n = 35) highlighted that some parents were concerned

about the InCAS assessments. Teachers stated that parents were particularly

concerned about:

• How the scores will be used (9 comments).

• If the results will be used as a transfer procedure (8 comments).

• The focus on the children’s weaknesses (5 comments).

• Their child’s lack of computer skills (3 comments).

A number of schools (n = 18) made a decision not to tell parents about the InCAS

assessments. Other teachers did not feel confident enough to discuss InCAS with

parents (n = 7).


Only 13% of respondents (n = 106) shared InCAS feedback with parents. Of these

teachers, 40 discussed the feedback at parent evenings and three provided parents

with the results in writing. 71.1% of these teachers (n = 54) believed that the

assessment feedback they provided to parents was meaningful.

Many other respondents indicated that they intend to share InCAS feedback with

parents (n = 246). Other respondents said that they did not plan to inform parents of the

InCAS feedback as it is not compulsory (n = 32) or because they did not feel confident

enough to do so (n = 15).

Figure 3: Confidence Reporting Assessment Feedback to Parents

Respondents were asked to rate how

confident they felt at reporting

assessment feedback to parents.

Figure 3 illustrates that only 43.9% of

respondents (n = 112) felt very confident

or confident informing parents of their

child’s results.

6.3

37.6

15.7

40.4

0

20

40

60

Veryconfident


Not at allconfident

% o

f Res

pond

ents

Many respondents indicated that they were unable to answer this question as they had

not yet received or interpreted their feedback. A small number of teachers felt that they

required further training (n = 26) and others were hopeful that the training they were

about to receive would increase their confidence in reporting feedback to parents.

Some teachers indicated that they had concerns about parents reactions to the

assessments (n = 13) and about the assessments reliability (n = 11).



Pupils’ Experiences The majority of respondents believed that the InCAS assessments were user-friendly for

pupils (95.5%, n = 961). Many teachers stated that pupils enjoyed completing the

assessments and many did so with confidence (209 comments). Nevertheless the

following concerns were made by teachers about pupils’ experiences:

• Pupils would prefer to use a mouse on the laptops (60 comments).

• Pupils were unable to adjust the volume during the test (46 comments).

• Comprehension section was very difficult (37 comments).

• The text in the Comprehension section was too small (33 comments).

• Some pupils did not take the assessment seriously and some treated it like a game

(29 comments).

• Graphics were dull and not very child-friendly (24 comments).

• Manageability of the assessment depended on the pupil’s ability (20 comments).

• Assessments were not user-friendly for SEN pupils (10 comments).

• The score should not have been shown at the end of assessments as pupils

compared results (9 comments).

• The comprehension section timed out on some pupils (9 comments).

• Teachers had difficulty logging pupils on to the system (7 comments).

• The language and certain parts of the Mathematics assessment were too complex

for Year 5 pupils (7 comments).

• There should be a facility to leave/return to a question (6 comments).

• The test should have a pause facility (6 comments).

• Some pupils were not aware that they had to press ‘continue’ after each section in

the Reading assessment (6 comments).

• Pupils found the clock off-putting (5 comments).

Pupils with Special Educational Needs 83% of teachers (n = 847) administered InCAS to pupils with Special Educational Needs

(SEN). 61.4% (n = 520) of these respondents had a pupil with a Statement of Special

Educational Need. Categories of pupils’ Statement of Educational Need included:

• Cognitive and learning (n = 446).

• Social, emotional and behavioural (n = 237).

• Communications and interaction (n = 162).

• Medical conditions/syndromes (n = 162).

• Physical (n = 60).

• Sensory (n = 46).

Figure 4: Manageability of Assessments for Pupils with Special Educational Needs

17

3.210.5

69.3

0

20

40

60

80

Verymanageable

Manageable Notmanageable

Not at allmanageable

% o

f Res

pond

ents

Figure 4 indicates that 86.3 % (n=643) of

respondents that administered InCAS to

pupils with SEN believed pupils found

the assessments manageable. However

13.7% (n= 102) felt the assessments

were not manageable for SEN pupils.


Respondents were asked to comment on their answer. The majority of comments

highlighted problems pupils experienced when completing the assessments:

• Pupils required support to stay on track (52 comments).

• Pupils guessed and treated the InCAS assessments as a game (44 comments).

• Pupils found it enjoyable (29 comments).

• Pupils found it hard to concentrate (23 comments).

• The Comprehension section was too difficult for SEN pupils (19 comments).

• The test was pitched at too high a level (13 comments).

• Visually impaired children found it difficult to read from the screen (10 comments).

Just under half of the responding teachers (47.6%, n = 271) said adaptations to the

InCAS assessments are required to make it accessible to pupils with SEN. These

teachers made the following recommendations:

• Simplify questions for SEN pupils level of ability (46 comments).

• Additional time for SEN pupils to complete the assessment (37 comments).

• Larger font (33 comments).

• Brighter colours on the computer screen (19 comments).

• A shorter assessment (15 comments).

• Classroom support when pupils sit the assessment (15 comments).


Pupils with English as an Additional Language 20% of respondents (n = 177) administered InCAS to pupils with English as an

additional language (EAL). Of these, 72.6% (n=127) felt the assessments were

manageable for pupils with English as an additional language. A small number of

teachers said that the manageability of the test depended on the pupils competency

with the English language (n = 7). The following comments were also made:

• Part of the test should be in the pupils first language (5 comments).

• This type of assessment is not suitable for pupils with EAL (5 comments).

• There was a lot of guess work (5 comments).

• Pupils had a poor grasp of English (5 comments).

• Some pupils did not understand the instructions (4 comments).

• Reading assessments were particularly difficult (2 comments).

• Teacher intervention was required (2 comments).

Pupils from the Irish Medium Sector The Irish Medium Maths version of InCAS was not used by any teachers. A small

number of teachers said that their school was awaiting installation of Irish Medium

Mathematics (n = 6) and two teachers said they did not know that this version was

available.

Eight respondents felt that the translation for the Irish Medium Maths version of InCAS

was not appropriate for their pupils.

Four teachers from the Irish Medium sector administered the English version of Reading

to their pupils. Three of these teachers felt the pupils’ performance in InCAS reflected

their ability. However one teacher believed that some pupils found the Reading and

Spelling tasks difficult.



Additional Comments

Respondents were given the opportunity to make additional comments at the end of the

questionnaire. Many of these responses have already been incorporated into the main

body of the report but the majority of responses are also summarised below.

A number of concerns were raised. Some respondents (n=37) found InCAS extremely

time consuming and teachers requested sub cover to complete the assessments

(n=55). 20 respondents were frustrated with the difficulties of gaining computer access

for their class and felt InCAS is best administered in school computer suites. A

significant number of teachers experienced technical difficulties whilst carrying out

InCAS. These included:

• Difficulties adjusting sound (33 comments).

• Problems with laptop wireless connection (19 comments).

• Difficulties logging pupils on to the system (14 comments).

• Computer screens freezing (8 comments).

• Computers crashing (7 comments).

• Difficulties with uploading results (6 comments).

Respondents believed that the Mathematics (n=25) and Comprehension (n=30) content

was pitched at too high a level for some Year 5 pupils and that the multiple choice

format provided some pupils the opportunity to guess the answers (n=29). A large

number of respondents (n=122) believed that the Mathematics module should be

broken down into specific areas, for example, shape and space or data handling.

Pupils had a visible raw score at the end of their assessment and this resulted in some

pupils comparing marks. Teachers felt this information should be removed from the

assessment (n=20).

The following recommendations were also made:

• A mouse function is needed for laptops (51 comments).

• Teachers require more information on reporting back to parents (24 comments).

• Schools require more computers/laptops (23 comments).

• Teachers require time out of class to analyse information (21 comments).

• Pupils should be able to spend time practising before starting the test

(16 comments).

• Pupils should be able to repeat the assessment at the end of the year to compare

scores and see if progress has been made (13 comments).

• All tables and charts should be presented on one page (13 comments).

• A sort filter would allow teachers to view the results of each class separately

(10 comments).


4.2 Evaluation of training on InCAS Assessment Data

This section presents the results for each question from the questionnaire. Results are

based on questionnaires from 1151 respondents. Percentages relate to the number of

teachers who answered each question which may be fewer than 1151. The main

qualitative issues to support the responses given are also highlighted. A number of

respondents reiterated comments throughout their questionnaire; as a result you will

find some comments repeated throughout this section of the report.

Standard Feedback

Figure 5: Training in the Use of Standard Feedback

38.4

59.2

2.1 0.40

20

40

60

80

Very useful Useful Not useful Not at alluseful

% o

f Res

pond

ents

The majority of respondents

(97.5%, n = 1113) felt the training in the

use of Standard Feedback (Scores

tables) was useful or very useful. There

were many positive comments

highlighting that this aspect of the training

was very clear and informative (n = 58).

A small number of teachers said the training has provided them with a better

understanding of how to interpret Standard Feedback (n = 8) and that the handouts and

booklet covering the key processes in InCAS will be useful references for schools

(7 comments).

2.5% (n = 28) of respondents did not find the training on Standard Feedback useful. A

number of these teachers believed that the training was unnecessary as the

feedback was “self explanatory” (n = 6) and that all information was covered in the

booklet (n = 5). One teacher stated that:


“the money spent on the training course [would have been] better spent on

teacher sub cover to analyse results”.

Following the training, 97.5% (n = 1108) of respondents said they felt confident about

interpreting Standard Feedback on their pupils. A number of teachers commented that

the data was straightforward to interpret (n = 17) and that the training had enhanced

their confidence in interpreting the data (n = 13). Nine teachers also believed that the

InCAS resources (booklets, handouts and online material) would be useful sources of

information when interpreting feedback in school.

Nevertheless a number of teachers (n = 23) were concerned about the time that it will

take them to interpret and understand the results. A small number of teachers were

also apprehensive about sharing Standard Feedback with parents.

Custom Feedback

Figure 6: Training in the Use of Custom Feedback

97.7% (n = 1116) of respondents rated

the training in the use of Custom

Feedback as either very useful or

useful. A number of teachers

commented that this aspect of the

training provided them with an

informative overview of the methods of

displaying feedback (n = 37).

31.1

66.6

2 0.30

20

40

60

80

Veryuseful

Useful Not useful Not at alluseful

% o

f Res

pond

ents


16 teachers felt a practical session using computers would have been more valuable.

The remaining comments indicate that teachers have a variety of views and opinions

about Custom Feedback, some of which are conflicting. These comments include:

• A lot of information to assimilate (17 comments).

• Reluctant to show parents this feedback (16 comments).

• Useful illustration when reporting to parents (12 comments).

• Dislike longitudinal charts (8 comments).

• Prefer tables to graphs as they are more specific (7 comments).

• Standardised age scores will be useful (6 comments).

• Graphs are an immediate visual aid (4 comments).

A small number of teachers recommended that the following changes are made to

Custom Feedback:

• Individual pupil reports of all modules presented on one page (1 comment).

• Include confidence intervals on assessment tables (1 comment).

• “It needs to be made clearer that the green line in the longitudinal chart is the child’s

chronological age” (1 comment).

Following the training, just over 96% of respondents (96.3%, n = 1094) indicated that

they felt confident at the prospect of interpreting the Custom Feedback about their

pupils. A number of qualitative comments were made. Teachers said they gained

confidence from the training (n = 24) and they will be confident provided they are given

adequate time to interpret feedback (n = 18). Six respondents stated that they will feel

more confident after further reading and practise. A small number of teachers

reinforced the usefulness of the InCAS booklet (n = 4).


Age Comparison Chart

Figure 7: Age Comparison ChartTeachers were asked to rate the

usefulness of the Age Comparison

Chart. Figure 7 indicates that the

majority of teachers believed that the

Age Comparison Chart was either very

useful or useful (93%, n = 1045). The

remainder of respondents (7%, n = 79)

rated this chart as not useful or not at all

useful.

26.2

66.8

6.30.7

0.0

20.0

40.0

60.0

80.0

Veryuseful

Useful Notuseful

Not at alluseful

% o

f Res

pond

ents

The majority of comments about the age comparison were positive. A large number of

teachers believed this information to be very useful as it presents the range of ability

within a class (n = 67). Teachers also felt this information will be useful for:

• School management and informing school development (n = 14).

• Grouping pupils (n = 12).

• Providing information to parents (n = 7).

• Detecting pupils encountering problems (n = 4).

However a number of teachers highlighted that they found this information confusing

(n = 18) and seven teachers felt this chart was not suitable for pupils with lower abilities.


Non Statutory Assessments

Figure 8: Non Statutory AssessmentsFigure 8 indicates that a large number of

teachers intend to make use of the five

other modules InCAS offers. The three

most popular modules were: Mental

Arithmetic (33.2%, n = 382); Spelling

(30.2%, n = 337); and Developed Ability

(19.8%, n = 228). 3.7% of respondents

(n = 43) intend to use all of the InCAS

modules.

3.74.88.6

19.8

30.233.2

0

10

20

30

40

50

Mental

Arithmeti

c

Spellin

g

Develo

ped A

bility

Attitud

e Mod

ules

Non-V

erbal

All modu

les

% o

f Res

pond

ents

A large number of respondents indicated that they will continue to use other

assessment tools. The most common assessment tools teachers plan to use are:

• NFER (n = 315).

• Suffolk (n = 33).

• NRIT (n = 18).

• End of year assessments (n = 16).

• Nelson (n = 15).

• Alta Maths (n = 10).

• Richmond (n = 6).


45 teachers stated that they only intend to use other assessment tools until InCAS is

fully implemented. A further 45 teachers stated that they are unsure if their school will

continue using other assessments as this will be determined by their school

management.

Sharing InCAS Feedback with Parents

The vast majority of respondents (91.4%, n = 1026) felt that the training informed

teachers how the outcomes of the InCAS assessments can be shared with parents. A

number of teachers felt that the tables and charts will be useful when sharing outcomes

of InCAS with parents (n = 24) and that the training on the use of appropriate language

was helpful. Nevertheless a number of teachers had concerns about reporting InCAS

results to parents. These concerns are summarised below:

• Apprehensive at sharing this information with parents (31 comments).

• Require further guidance on sharing information with parents (25 comments).

• Need more time to become familiar with feedback (5 comments).

• Not confident at sharing feedback with parents of pupils with lower abilities

(5 comments).

Figure 9: Confidence at Sharing Assessment Feedback with Parents

Respondents were asked to indicate

how confident they felt at reporting the

InCAS assessment feedback to parents.

Figure 9 illustrates that just over four-

fifths of respondents (81%, n = 909) felt

either very confident or confident.

17.3% (n = 194) were not confident and

1.7% (n = 19) were not at all confident.

9.2

71.8

17.3

1.70

20

40

60

80

Veryconfident


Not at allconfident

% o

f Res

pond

ents


Teachers were asked to comment on this question. The majority of comments received

expressed concerns about reporting assessment feedback to parents. The comments

are summarised below:

• Teachers are concerned about how parents may interpret the information

(25 comments).

• Teachers are anxious about discussing low scores with parents (20 comments).

• Ability of parent may affect their interpretation of results (19 comments).

• Teachers require more time to analyse results (17 comments).

• Schools will have to decide on a co-ordinated approach (17 comments).

• Discrepancies with pupils’ scores that do not reflect their ability (8 comments).

• Teachers require further training on reporting assessment feedback to parents

(7 comments).

• Will not feedback using graphs or charts. Some of this information is too stark to

present to parents (6 comments).

Figure 10: Format of Sharing Information

with ParentsFigure 10 illustrates that over two thirds

of teachers (67.5%, n = 777) felt that the

Scores Table would be the most

meaningful format to share InCAS

assessment outcomes with parents.

37.5% of respondents (n = 432)

believed that the Scores Chart would be

a useful format to share feedback with

19.4

67.5

37.5

9.20

20406080

100

Scores

Table

Scores

Chart

Long

itudina

l Chart

Differen

ce Cha

rt

% o

f Res

pond

ents

parents and just under 20%

(19.4%, n = 223) indicated the Difference Chart was their preferred format. 9.2% of

teachers felt the Longitudinal Chart would be a meaningful format of sharing information

assessment outcomes with parents.


A large number of comments were provided. Many teachers stated that they would use

a combination of the tables and charts to share InCAS Feedback with parents

(51 comments). Of the remaining comments, the vast majority were about concerns

teachers had about providing parents with the results of the InCAS assessments. Some

of these concerns are listed below:

• Format of sharing feedback will depend on parents ability to understand information

(37 comments).

• Unsure of which format to use (15 comments).

• Format of sharing feedback will depend on the ability of the pupil (13 comments).

• Would not use Longitudinal chart to share outcomes with parents (12 comments).

• More time required to discuss this at a whole school level (9 comments).

• Charts require too much explanation (7 comments).

A small number of teachers made the following recommendations:

• It would be useful to have both the chart and graphic information on one page when

reporting to parents (7 comments).

• “There needs to be more information provided on the assessment outcomes shared

with parents” (1 comment).


Additional Points for Consideration Respondents were given the opportunity to make additional comments at the end of the

questionnaire. A lot of these responses have already been captured in the main body of

the report, however a further summary is given below.

A large number of respondents took this as an opportunity to commend the training

stating that it was very useful and informative (73 comments). Teachers also praised

the presenters delivery of the training (12 comments).

A number of concerns were also made. Firstly, a large number of teachers stated that

they required time to analyse the assessment feedback (44 comments). A small

number of teachers requested sub cover to administer the InCAS assessments. A

small number of respondents also said their school required additional computers

(n = 6) and laptops (n = 3).

The following recommendations were also made:

• Mathematics should be broken down into specific areas for example shape and

space or data handling (18 comments).

• All tables and charts should be presented on one page (10 comments).

• InCAS should be customised to suit the needs of SEN and MLD pupils

(7 comments).


5.0 Conclusion

Results from this evaluation highlighted a number of areas respondents were

dissatisfied with. 48.8% of teachers (n = 500) experienced various technical difficulties

accessing InCAS on their C2K networked computers. It is imperative that CCEA in

conjunction with its partners (DE, C2K & CEM) work through such technical difficulties

to determine how they can prevent these problems occurring in the future.

Over two thirds of respondents (68.8%, n = 506) indicated that they would like additional

support or resources. Requests included: additional preparation time (n = 124);

additional computers/laptops (n = 93); further guidance on InCAS (n = 67); and mice for

laptops (n = 67). A small number of teachers (n = 12) felt the pupil demo should contain

more questions for each example. It is important that CCEA prioritises teachers’

requests as some will have a large impact on teachers and pupils.

Results contained within this report suggest that adaptations and special arrangements

have to be made to InCAS in order for it to be user-friendly for pupils with SEN. This

evaluation has touched on some of the changes that are required. However it is

imperative that further research is conducted on the manageability of InCAS for pupils

with SEN. Such research should focus on the adaptations required for the six main

categories of Special Needs. Information should be collected from observations and

case studies that can then be used as a resource for Special Educational Needs

teachers.

Throughout both questionnaires many teachers expressed concern at the prospect of

sharing InCAS feedback with parents. Respondents requested a guidance resource

outlining how teachers can share feedback with parents. This will also be a useful

source of information for future implementation years.

Despite the concerns highlighted above, there were aspects of InCAS that teachers

rated highly. Overall, 80.8% of teachers (n = 743) have found implementing the Pupil


Profile a positive experience. Over 95% of teachers deemed the InCAS assessments to

be user-friendly for pupils and many stated that pupils enjoyed completing them. 82.5%

of teachers (n = 696) found the InCAS feedback consistent with their professional

judgement. Furthermore, nearly 60% of teachers (59.3%, n = 512) believed that InCAS

provided them with additional information about their pupils.

The findings suggest that although many teachers believe that a number of adaptations

are required to make the assessments more manageable and user-friendly for both

schools and pupils, many also recognise the value of InCAS.

Recommendations 1. Continue to work in partnership with CEM and C2K to determine how technical

difficulties identified within this report can be prevented from occurring in the future.

2. Provide schools with mice for laptops.

3. Keep under review any adaptations which may be required for pupils with SEN,

focusing on the six main categories of Special Educational Needs.

4. Inform the Department of Education of teachers request for a guidance resource on

sharing InCAS feedback with parents and requests for additional support and

resources (for example computers, mice and preparation time).



APPENDICES

METHOD OF CONSULTATION The roll out of the Pupil Profile Process commenced this academic year with Years 1

and 5. CCEA’s Research & Statistics Unit will be undertaking a full evaluation of the

implementation of the Primary Pupil Profile. This years evaluation involves consultation

with Principals, Year 1 and 5 teachers and pupils, and parents with children in Year 1

and 5. Consultation will be undertaken at various phases of this academic year and a

range of research methods will be employed to collect feedback. Details of the

consultation are provided below.

Phase 1 Consultation: May 2007 All Year 5 teachers across Northern Ireland

(approximately 1500) were invited to attend awareness

raising training on delivering the Pupil Profile and using

InCAS computer-based assessments. A questionnaire

was used to collect Year 5 teachers’ views and opinions

of this training. A total of 1395 questionnaires were

completed and returned.

Phase 2 Consultation

October - November 2007

Before administering InCAS to pupils, all Year 5 teachers

across Northern Ireland were invited to attend training on

understanding InCAS assessment data. A total of 1425

teachers attended this training. Before the training

commenced 80.2% of teachers (n = 1143) completed a

questionnaire on their previous experiences with InCAS. The

aim of this evaluation was to obtain teachers views and

opinions of InCAS which would be used to inform the future

development of the software for 2008/2009.



On completion of the training 80.8% (n = 1151) of teachers

also completed an evaluation questionnaire based on the

days training on ‘Understanding InCAS Assessment Data’.

November - December

2007 Following the teacher training on ‘Understanding InCAS

Assessment Data’, teachers were required to administer the

InCAS assessments to their pupils by 31st December 2007.

Questionnaires were posted to all Year 5 teachers

(approximately 1500) and pupils (approximately 25,000) in

Northern Ireland. The aim of the questionnaires was to

evaluate the success of the implementation of the Pupil

Profile Process within Year 5 to date and to inform the

development of the InCAS computer-based assessments.

Phase 3 Consultation

June 2008 Schools are required to produce an Annual Pupil Profile

Report for all Year 1 and 5 pupils by the end of June 2008.

Throughout June, CCEA’s Research & Statistics Unit will be

consulting will all key stakeholders involved in this process:

Principals, teachers, pupils and parents. Questionnaires will

be the main method of collecting data; however a range of

qualitative methods will also be used. A representative

sample of participants will be selected to take part in research

seminars and focus groups. Details of these methods are

documented below.

Seminars & Focus Groups Throughout June 2008, CCEA’s Research & Statistics Unit

will be facilitating; two Principal seminars, ten teacher

seminars and two parent focus groups. The purpose of the

seminars and focus groups is to collect rich in-depth

information that will supplement data collected from

questionnaires. Two members of the Research & Statistics

Unit will facilitate discussions at these consultations and a

representative from the Pupil Profile Implementation Team

will be present to answer any questions that may arise during

the course of the day.


www.nicurriculum.org.uk

Documents

Evaluation of InCAS Computer-Adaptive Assessments December … · 2019. 6. 26. · 83% (n = 847) of teachers administered InCAS to pupils with Special Educational Needs (SEN). Of