Technical Report Ch 08

8/9/2019 Technical Report Ch 08

1/10

192

First European Survey on Language Competences: Technical Report

!"#$%&' )*

+$&'#%,-./ 0

12#3,%4 5-.,%-',.6


2/10

193


8 Operations - Quality monitoring

This chapter provides an overview of the quality procedures employed for the ESLC.Note, the discussion focuses on the Main Study processes only, Field Trial processes

are not discussed unless relevant.

8.1 An introduction

It is essential that users of the ESLC data have confidence that the data collection

activities were undertaken to a high standard and for the purpose of creating an

international dataset of a quality that will enable valid comparisons across participating

educational systems. There were various methods, detailed further below, used to

ensure this confidence.

The ESLC Technical Standards (which can be seen in Appendix 4) provided the set of

standards upon which the data collection activities were based and were fundamental

to the quality control methods employed by SurveyLang for the ESLC. There are three

types of standards; each with a specific purpose. Data Standards ensure that all

collected data can be added to the final ESLC 2011 dataset that will be released by

the Commission. Management Standardsensure that all ESLC operational objectives

are met in a timely and coordinated manner. National Involvement Standardsensure

that the internationally developed instruments meet the highest standards of cross-

national, cross-cultural and cross-linguistic validity and equivalence and that the ESLC

results have the greatest possible meaning for national stakeholders.

The Data Standardsoutlined the standards for the following areas:

target population and sampling

translation

test administration

security of materials

quality monitoring

printing of material

marking, coding and data entry

data submission

All SurveyLang procedures were carefully developed and documented to ensure data

of the desired quality. Quality monitoring played an important role and the

implementation of the operational procedures documented in the guidelines and

manuals made available to NRCs were continually monitored. In any cases where the

documented operational processes were not fully implemented, these were logged and

discussed with the NRC to understand the likely impact for the data. Quality monitoring

was, therefore, the process of systematically observing and recording the extent to

which data were collected and stored according to the procedures described in the


3/10

194


ESLC field operations documentation. Quality monitoring was a continuous process

and was a shared responsibility between the NRC and SurveyLang.

The main elements of the quality monitoring procedures were:

project milestones and signed off documentation were managed

Central Training: NRC attendance at central SurveyLang training

Credential sheets: SurveyLang provided role definitions and criteria for the

appointment of Test Administrators, Quality Monitors, Data Entry staff,

Markers of Writing, and Coders for the open responses in the Student

Questionnaire

SurveyLang team: SurveyLang assisted NRCs in the planning and

implementation of key processes. SurveyLang systematically monitored the

key processes of sampling, translation, field operations and data entry, the

marking of writing and the coding of the open responses in the Student

Questionnaire

Central Issues Log: all risks and issues relating to the implementation of

operational processes were stored on a central register and regularly

reviewed

Quality Monitors: Quality Monitors observed the implementation of ESLC field

operations at the educational system level, NRCs appointed a Quality Monitor

to make unannounced visits to, typically, 10 schools in each educational

system, interviewing both the Test Administrator and School Coordinator. The

Quality Monitor also visited each National Research Centre and interviewed

the NRC and five data entry staff. The lead Quality Monitor for eacheducational system wrote a full report on the results of their visits and

interviews. Further details about their role are available below in section 8.4.

ESLC Administration Report forms: ESLC Test Administrators completed a

report after each ESLC test administration, thus providing an overview of the

test administration across participating educational systems.

ESLC Administration Issues Log: ESLC Test Administrators completed a log

of all administration issues, thus providing an overview of any arising issues

across participating educational systems.

NRC report: SurveyLang developed a template report which allowed NRCs to

systematically self-report on the implementation of key processes in their

educational system.

8.2 Support for NRCs in quality monitoring

The documentation that formed the basis for the quality monitoring procedures were:

ESLC Technical Standards

NRC Field Operations Manual


4/10

195


Test Administrator Manuals (paper-based and computer-based including a

script for test administration)

School Coordinator Guidelines (paper-based and computer-based)

Sampling Manual Translation Guidelines and Manuals

Data Entry Guidelines (Listening, Reading and Writing and Student

Questionnaires)

Marking of Writing documentation

Coding Guidelines

Quality Plan for Quality Monitors and Quality Plan Report.

The quality monitoring instruments developed from these manuals and guidelines

included role credential sheets, a range of sampling forms, WebTrans for the

translation and verification work, a Test Administrator interview protocol for the QualityMonitor, a School Coordinator interview protocol for the Quality Monitor, a Data Entry

interview protocol for the Quality Monitor, NRC interview protocol for the Quality

Monitor, NRC feedback report templates, an ESLC Administration Report Form and an

ESLC Administration Issues Log.

Credential sheets: as outlined above, SurveyLang provided role definitions and criteria

for the appointment of Test Administrators, Quality Monitors, Data Entry staff, Markers

of Writing, and Coders for the open responses in the Student Questionnaire

Sampling forms: SurveyLang developed a series of forms for collecting key data and

for monitoring school and student sampling outcomes. The NRC and SurveyLang

experts negotiated agreement on sampling plans and outcomes (see chapter 4).

WebTrans for translation and verification: this system managed the quality monitoring

activities for translation of all documentation at the national level (see Chapter 5).

Test Administrator quality monitor interview protocol: a standard schedule was

prepared by SurveyLang to systematically record the outcomes of the Quality

See Task 1 below in section 8.4 for further information.

School Coordinator quality monitor interview protocol: a standard schedule was

prepared by SurveyLang to systematically record the outcomes of the Quality


Data Entry quality monitor interview protocol: a standard schedule was prepared by

.


NRC quality monitor interview protocol: a standard schedule was prepared by

te visit.

The interview protocol recorded information on:

the general organisation of the ESLC in that educational system


5/10


6/10


7/10

198


SurveyLang provided a full credential sheet for Quality Monitors. In brief, the Quality

Monitor:

should have past experience of acting as a Quality Monitor on a similar project

or in a similar role should have fluency in English and the questionnaire language

should not be an employee at the same organisation as the NRC

must not be an immediate relative of an employee at the NRC

must not be line managed by the NRC

must send their report directly to SurveyLang in electronic format

must be able to and have the capacity to independently and effectively

communicate with SurveyLang using email and telephone.

However any feedback that could be used by the NRC to correct the way tasks were

managed duringthe Main Study were discussed with the NRC so the NRC could takequick action. Such instances were to be documented in the report.

The NRC had to meet with the Quality Monitor to:

Train the Quality Monitor in the background of the ESLC.

Make all operational documentation available particularly the School

Coordinator Guidelines, Test Administrator Manual and the Data Entry

Guidelines.

Make background information about the project available, e.g. the Inception

and Interim Reports.

Make a plan of schools and data entry staff for the Quality Monitor to visit. Inform the Quality Monitor of variations agreed to standard SurveyLang

procedures.

Be available to respond to questions raised by the Quality Monitor. Where

there are several Quality Monitors, one person should be appointed as the

lead person and should be the key liaison with the NRC.

Inform the Quality Monitor that they can ask questions directly of SurveyLang

if they wish. Where there are several Quality Monitors, the lead person should

be the contact with SurveyLang.

SurveyLang suggested that the Quality Monitor attended -country Test

Administrator training.

The appointed Quality Monitor (or Quality Monitors in the case of Task 1 below) made

unannounced visits to assess the quality of the implementation of SurveyLang

processes and procedures carried out within the educational system.

Task 1: Test Administration: This task required the Quality Monitor to visit 10 schools;

a mix across administration modes (computer and paper-based where both were

used), regions and Test Administrators was preferred. The Quality Monitor was

required to:


8/10

199


Be at the school from 1 hour before the start of the administration to the end of

administration when the Test Administrator and School Coordinator had

packaged completed test materials up and completed the student tracking

form.

Interview the Test Administrator and School Coordinator separately for

approximately 15 minutes. A template list of questions was provided for this

purpose covering all aspects before, during and after the test administration

focusing particularly on the logistical arrangements, materials management

and clarity of documentation and processes.

Write all responses down from the Test Administrator and School Coordinator

interviews.

Summarise the key findings from the interview in terms of what worked well

and what did not work well for the Test Administrators and the School

Coordinators. Were the procedures followed as specified? What problems

were encountered? How were these resolved?

Quality Monitors may also have wished to review some of the Administration

Report Forms for the Test Administrations that they observed.

Task 2: Data Entry staff: This task required a quality check of the data entry work. The

Quality Monitor had to review a sample of the data entry work from each data entry

person employed by the NRC. The Quality Monitor was required to:

Review a sample of five booklets per skill (Reading, Listening, Writing and the

questionnaires) from the work each data entry person had performed.

Check that the data was entered correctly for each data entry person. In the

report, the number of mistakes found had to be entered. The data had to be

corrected and the NRC informed so that they could take corrective measures

if necessary.

Interview each data entry person for approximately 5 minutes. A template list

of questions was provided for this purpose, covering aspects such as the

confidence of their accuracy.

Write all responses down and document in the report what worked well and

well for each data entry staff member. Were the procedures

followed as specified? What problems were encountered? How were these

resolved?

Task 3: NRC report: The Quality Monitor was required to:

Talk through each step in the period from mid January (including printing and

the receipt of materials) to data submission for approximately 30 minutes. A

template list of questions was provided for this purpose.

Write all responses down and document in the report what worked well and

well for the NRC. Were the procedures followed as specified?

What problems were encountered? How were these resolved?


9/10

200


8.5 Quality monitoring data

The quality monitoring data collected from all of the documents and mechanisms in

paragraph 0 were carefully reviewed and analysed both after the Field Trial and theMain Study.

Feedback was assessed by SurveyLang in order to improve all central and educational

system processes and documentation after the Field Trial. All feedback and data was

assessed again after the Main Study.

ESLC Quality monitor reports: each of the 16 educational systems submitted a Quality

Monitor report on the conduct of testing sessions. The report consisted of a summary

of their general observations together with a summary of the main findings from each

of the different types of interviews conducted.

ESLC NRC Reports: each of the 16 NRCs submitted feedback reports on the overall

processes and documentation for the Main Study.

In general, the quality monitoring reports and NRC feedback reports suggested a

strong organisational base existed within educational systems for the conduct of the

ESLC. The Quality Monitor reports indicated that, overall, NRC staff had a very good

understanding of the operational aspects of the ESLC. The reports indicated that the

ESLC administrations were conducted in a manner that was largely consistent with the

documented procedures in the ESLC operations manuals.

Issues that were reported from the Main Study include:

The required school sample size was not reached for one or both test

languages (England, Greece). Note: to minimise the impact of this issue,

additional students were drawn in the sample across participating schools.

Some schools did not reach the required participation rate but did not inform

the NRC and did not organise follow-up sessions (France).

Some schools were unable to administer the Listening test in the required

classroom setting because there were not enough rooms or test

administrators available (German Community of Belgium).

Data entry deadlines could not be met due to staff shortages. The team leader

monitoring data input had to undertake data entry himself and therefore could

not supervise others properly (Malta).

Data entry deadlines could not be met due to staff shortages and financial

restrictions which prevented the NRC from employing experienced staff

(Greece).

Data entry files became corrupted and could not be recovered requiring re-

entry of data which resulted in delays (German Community of Belgium).

There were complaints about the lack of information to parents before the

administration (German Community of Belgium).


10/10

Documents

Technical Report Ch 08