Embed Size (px)
Citation preview
ICAEA Test Design Guidelines
Session 1
Achievements and challenges with the LPRs
INTERNATIONAL CIVIL AVIATION ENGLISH ASSOCIATIONsupporting the use of English for aviation safety
Michael Kay
ICAEA President
22 years in Aviation English Testing and Training
MA Applied Linguistics, University of Melbourne, Australia
Thai ANSP (Aeronautical Radio of Thailand), since 2011
Neil Bullock
ICAEA Vice President
20 years in ATC (UK), 15 years in Aviation English Testing and Training
MA Applied Linguistics, University of Birmingham, UK
Swiss FOCA- LPR, since 2007
Ana Monteiro
ICAEA Board Member
15 years in Aviation English Testing
MA Applied Linguistics, Federal University of Rio de Janeiro, PhD Applied Linguistics and Discourse Studies, Carleton University, Ottawa, Canada
ANAC, Brazil since 2003
Development Team
Terence Gerighty
ICAEA Vice President
25 years in aviation English training
BA Hons, Dip.Ed., London University
Author of training materials for pilots, controllers and cabin crew
Daniel Gorbold
ICAEA Vice President
17 years in aviation English training and testingBA Hons, MA (City of London University, UK)Qatar Airways since 2017
Support Team
Norma BravoICAEA Vice President14 years in aviation English training and testingBA LinguisticsANAC (Argentina) and EANA SE since 2005
Cecilia DenisICAEA Board Member20 years in aviation English training and testingBA English Teaching, BA EducationCIPE and ANAC (Argentina) since 1999
Support Team -
South America
Test Design Guidelines: Workshops
For:• Civil Aviation Authorities• Test Service Providers (test developers)Objectives:• Develop awareness of key issues at stake and risks with poor testing standards• Develop awareness of essential design features that affect the quality and
effectiveness of all aspects of LPR testingFormat:• Input sessions and presentations • Hands-on interactive activities for participants to apply the TDGs and explore
issues of good/bad ICAO LPR test instrument design• Participants sharing experiences in supportive exchanges
Discussion
1. What happens when more than one LPR test is accepted by a CAA for pilots/ATCOs?
2. Do you have confidence to evaluate LPR tests you need to approve?
3. What criteria do you use to evaluate the quality of LPR tests you approve?
4. Are the tests used of the same standard?
The Test Design Criteria1. Test instruments need to include appropriate tasks that directly assess how test takers use
language in radiotelephony communication contexts.
2. Separate test instruments need to be designed for pilots and air traffic controllers.
3. Test instruments need to contain tasks dedicated to assessing listening comprehension, separate from tasks designed to assess speaking performance.
4. Test instruments need to comprise distinct sections with a range of appropriate test task types.
5. Test instruments need to include test tasks that allow test takers to engage in interactive and extended communication.
6. Test instruments need to include tasks and items which allow the assessment to differentiate between ICAO language proficiency levels.
7. Test instruments need to contain appropriate tasks that assess test takers’ abilities to understand and communicate in real-world contexts.
8. Test instruments need to have a sufficient number of equivalent versions, with each version of the test representing the test instrument in the same way.
Does the test instrument include appropriate tasks that directly assess communication skills in aeronautical radiotelephony contexts?
Are the speaking tasks and content specifically developed to assess the language needs of either pilots or controllers?
Does the test instrument contain parts specifically designed to assess listening comprehension?
Are there distinct sections with a range and variety of appropriate task types?
Are there tasks requiring test takers to participate in extended and interactive communication?
Are there test tasks, items or sections to assess different proficiency levels?
Are there tasks that assess communication in real-world contexts?
Are there a sufficient number of equivalent versions of the test?
ICAO LPR Test Instrument Evaluation Toolkit
Including tasks and content in non-routine radiotelephony contexts in both speaking and listening parts of the test.
Including appropriate tasks and content to assess listening comprehension in radiotelephony contexts
Including tasks and content in radiotelephony contexts specifically designed for pilots or controllers in both listening and speaking parts of the test
And, the tasks are appropriate and effective in both the speaking and listening parts of the test
Including in radiotelephony contexts in the speaking part of the test
In both the speaking and listening comprehension parts of the test instrument
In both the speaking and listening parts of the test, aligned to pilot or controller roles.
For both the speaking and listening comprehension parts of the test
YES
YES
YES
YES
YES
YES
YES
YES
NO
NO
NO
NO
NO
NO
NO
NO
The test instrument is
likely to be effective.
It is likely to be suitable for
ICAO LPRs
The test instrument
needs further development.
It is not yet suitable for ICAO LPRs
The test instrument is ineffective.
It is not suitable for ICAO LPRs
2008-2011 Wait-and-see approach widespread
Testing gathers pace
2005-2008 Information
gathering, test development, gradual implementation
2009-2015Wide variation in testing standards emerges
2011-2019Varying testing standards emerge with some testing practices dominating
2013ICAO test
endorsement
scheme
launched
2003ICAO LPRs
announced
2008Standard
becomes
effective for
new
licences
2011Standard
becomes
effective for
existing
licences
Country A Country B Country C County D
Level
4
Level 4
Level 4
Level 4
Safe
ty im
plic
ations
6
5
4
3
2
1
6
5
4
3
2
1
Level 4
Test B
Level
4
Level 4
Test B
Level 4
Test Y
Test Q
6
5
4
3
2
1
Country A Country B Country C County D
Safe
ty im
plic
ations
6
5
4
3
2
1
Country A Country B
Level 4
Level 4
ATCOs
PILOTS
?
LPR compliance
?Language proficiency
Safety
Trust and confidence
