1

ANNUAL REPORT

2018

CETAPCentre for EducationalTesting for Access and

2

1. Executive Summary

2018 was yet another remarkable year for CETAP. Our dedicated staff have done an incredible job at expanding our collaboration and outreach while strengthening our service delivery and increasing our research and publications output.

We have also been able to increase the value of our services. This was enhanced by the Boston Consulting Group (BCG) project funded by the Michael and Susan Dell Foundation, which was completed earlier this year. It gave CETAP insight into the diagnostic use of the NBTs as well as the growth opportunities for the department.

A few highlights from this year include:

• Writers survey

• Diagnostic Mathematics Information for Student Retention and Success (DMISRS) Project and symposium

• Partnership with the Joint Education Trust (JET) on teacher assessments

• Collaboration with Educational Testing Services (ETS) & ACT in the USA

• Workshops with higher education institutions

• Hosted an international symposium with GAPS in August – Creating an ecosystem for Student Success

• Outreach initiatives and partnerships (MSDF; SAGEA; KZN DoE)

Our ongoing goal is to ensure the tests are accessible and within reach of all learners by constantly reviewing our test sites, venues and the online queries from students. Thank you for wholeheartedly being a part of our very important work towards student learning and success. The first roll-out of the NBT in 2009, this year 2019, marks 10 years for the National Benchmark Test (NBT).

We are looking forward to much more in 2019 and beyond by contributing to student success!- Naziema Jappie - Director (CETAP)

3

CETAP ANNUAL REPORT 2018

2. Mission, aims and objectives

The Centre for Educational Testing for Access and Placement (CETAP) is a department in the Centre for Higher Educational Development (CHED) at the University of Cape Town. CETAP is the largest post-school educational testing centre in South Africa and uses the most up-to-date test processes, practices and measurement theories. CETAP has developed numerous assessment instruments and engages continuously with the post-school sector to understand the needs and develop further tests.

CETAP’s mission is to be the nationally preferred provider of meaningful research and assessment that contributes to access and success in higher and further education and training. In order to realise this vision, CETAP strives to:

• Develop tests in Academic Literacy, Quantitative Literacy, Scientific Reasoning and Mathematics that are valid predictors of performance in educational contexts – especially for students from educationally disadvantaged backgrounds. These tests are developed nationally by discipline experts and tap competencies that are required by these educational contexts.

• Do research into the reliability and validity of these tests; explore different methodologies investigating this capacity; look into the factors that predict student success; build extensive databases on student throughput; and to subject the tests to scrutiny (reliability; validity; bias; coherence and internal consistency).

• Provide excellent client services; to organize, arrange, develop and create tests and testing infrastructure (national and international); to provide resulting and reporting information in the areas of access, placement, diagnosis and prediction; and to be efficient, professional, and reliable and offer quick turnarounds.

CETAP has a number of tests available, such as the Placement Test in English for Educational Purposes (PTEEP), the Mathematics Achievement Test (MACH), the Scientific Reasoning Test (SRT), the Quantitative Literacy Test (QLT), the Academic Literacy Test of Academic Potential (ALTAP), the Mathematics Comprehension Test (MCOM), and the Advanced Academic and Quantitative Literacy Test (AQL+). Most significantly, however, it is home to the National Benchmark Tests Project (NBTP). The National Benchmark Test comprises three tests: the National Benchmark Test in Academic Literacy (NBT AL), the National Benchmark Test in Quantitative Literacy (NBT QL) and the National Benchmark Test in Mathematics (NBT MAT).

The main objective of the NBTs is to assess the entry level academic skills of students in Academic Literacy (AL), Quantitative Literacy (QL) and Mathematics (MAT). In addition, the project also provides a service to Higher Education

Institutions requiring additional information to assist in selection and placement of students in appropriate curricular routes. The project has also assisted with curriculum development through first year teaching and learning forums and in relation to foundation, extended and augmented courses.

The NBTs are designed to provide complementary criterion-referenced information to supplement norm-referenced school-leaving results such as those provided by the National Senior Certificate (NSC).

Essentially, the purpose of NBT is the following:

• To identify talented but educationally disadvantaged students whose schooling might not have allowed them to demonstrate their full ability.

• To contribute to ensuring that students who are identified as having potential in terms of the tests also receive offers of academic places and support (such as academic development opportunities, financial aid, residence places, and so on).

• To contribute to the diagnosis of all incoming students’ academic strengths and weaknesses, in order to feed into the design of learning and teaching support and curriculum development.

• To act as a complementary assessment measure to the school-leaving examination – especially given that the school-leaving certificates, such as that provided through the National Senior Certificate (NSC) and other such bodies, are made up of different school leaving subjects and that scores on these subjects are norm-referenced and often difficult to interpret by the post school education systems.

DMSRIS Symposium

4

5

CETAP ANNUAL REPORT 2018

3. CETAP ORGANOGRAM 2018

Director

Operational Manager

Logistics Manager

Data Administrators

Project Assistants

Store Clerk

Test Development Cordinator

Maths Research Lead

QL Research Lead

AL Research Lead

Test Admistrator

Statistician

Communications & Marketing Manager

Data Manager

Secretary

7

4. Overview of activities in 2018 and departmental / project level measures of success

Engagement with potential funders

DHETDiagnostic Mathematics Information for Student Retention and Success (DMISRS) Project.The project seeks to use diagnostic NSC and NBT data to inform how best to serve first-year Mathematics students and to research the effect of blended learning and curriculum-integrated support initiatives for mathematics in the first year. This information should inform the nature of foundation courses and curriculum responsiveness, reduce university (especially first year) drop out, and increase throughput and graduation rates. Budget of the project is R4,421,596.

Workshops

INSTITUTIONAL WORKSHOPS:

CPUT • 13 March 2018 – Forum / Workshop on Academic Literacy in Higher Education

VUT • 18 June 2018 - Best Practice and Benefits of the NBTs Workshop for Health Sciences staff• 19 June 2018 - Best Practice and Benefits of the NBTs Workshop for Accounting staff

WSU • 6-7 August 2018 – T&L Workshops ADP and Mathematics staff (East London)

TEST DEVELOPMENT WORKSHOPS:

• Item Development Workshop: 7-9 May 2018• Item Review Workshop: 1-3 August 2018• MAT Test Assembly Workshop: 11-13 December• AQL Test Assembly Workshops: Ongoing throughout 2018• MCOM Test Development Workshop: 25-26 January 2019

7

CETAP ANNUAL REPORT 2018

CONSULTATIVE FORUMS AND MEETINGS:

• USAf CoP Teaching and Learning of Mathematics - 13 February 2018• USAf Consultative Forum - 16 May 2018• USAf Workshop - 20 August 2018, Gauteng• USAf CoP Teaching and Learning of Mathematics - 29 August 2018• USAf Consultative Forum - 5 November 2018

5. Environmental scan

With the austerity measures imposed on CETAP in 2017 and the loss of all GOB funding, CETAP has been experiencing some financial strain which requires the NBT project to carry the cost of all its current staffing and operating budget.

CETAP has been required to seek innovative ways of reducing costs. Operating costs have been achieved through dramatically reducing the number of test sessions (and venues in some cases) in the national test cycle in alignment with outcomes of the BCG business review. This has had the desired effect of reducing costs but allied effects of reduced writer numbers in the cycle may be attributed to this action. Storage space is still at a premium and there is a need to locate extra space on campus in 2019. CETAP has experienced significant attrition of staff in 2018 for various reasons. However, this may in part be attributed to uncertainty created by the loss of UCT funding, and the concomitant effect on CETAP’s perceived sustainability.

UCT has recently seen a radical change in senior executive management which resulted in loss of institutional knowledge pertaining to CETAP’s current and historical contributions to UCT. It must be noted that the NBT project no longer falls within the ambit of USAf but still has wide a reach to all universities in the sector.

6. Unit Research and Conference Presentations• Frith, V., Prince, R. The National Benchmark Quantitative Literacy Test for Applicants to South African Higher

Education. Numeracy, 11(2)

• Jappie, N. B., (2018) ‘The Role of University Leadership in Advocating Social Justice in Higher Education’ in Jeffries, R., Diversity, Equity, and Inclusivity in Contemporary Higher Education, Pensylvannia, USA. IGI Global Publishers

• Jappie, N. B., Mutakwa, D. The preparedness of the African home language speaking students for university: insights from the national benchmark tests. 11th Conference of the International Test Commission from July 2-5, 2018 in Montréal

• Jappie, N. B., Mutakwa, D. Institutional Effectiveness in Fostering Student Retention and Success in 1st Year. 11th Conference of the International Test Commission from July 2-5, 2018 in Montréal

• Jappie, N.B., Reimagining Global Access to Postsecondary Education (GAPS) Perspectives of GAPS Partners Symposium. WERA Conference. Cape Town. August 3-5, 2018

• Nel, B.P. Investigating students’ perception of a blended classroom approach used in a post-graduate module at an urban university in South Africa 26th Annual Conference of Southern African Association for Research in Mathematics, Science And Technology Education (Saarmste) from 16-19 January 2018 in Botswana.

ITC Conference in Montreal, Canada

8

CETAP ANNUAL REPORT 2018

• Nel, B. What are the implications of the NBTP quantitative literacy test results for teachers? 24th The Association for Mathematics Education of South Africa Annual National Congress, 25-29 June 2018.

• Padayachee, P. & Steyn, S. 2018. Decoding the language in and the language of mathematics : implications for assessment. Paper presented at the International Conference on Research in Education and Science, Marmaris, Turkey, May 2018.

• Prince, R., Conradie, J., Mutakwa, D. Assessment of mathematics learning potential: the mathematics comprehension test. 11th Conference of the International Test Commission from July 2-5, 2018 in Montréal.

• Prince, R. 2018. Assessment of Multimodal Academic Numeracy Practices.9th International Conference on Multimodality (9ICOM), University of Southern Denmark. Odense. 15 – 17 August.

• Simpson, Z., & Prince, R. 2018. Teaching, Learning, and Employing Analytical Frameworks as Performance: Analysis of a Quantitative Literacy Event in Applied Mechanics. Designs for Learning, 10(1), 76–87.

• DOI: http://doi.org/10.16993/dfl.95

ACT (USA) visit

9

7. Staff Development in 2018

• Naziema Jappie – registered for PhD

• Sanet Steyn – submitted dissertation for Master’s Degree Artium (English)

• Ashley Niekerk – Registered Advance Diploma – Application Development

• Juandre Alard completed the Basic Communication in isiXhosa

• Lynia Fortuin – Regsitered for BA Administration

• Estelle Murray attended management training (“Building a Team”) through the Cape Higher Education Consortium’s Regional Middle Management Development Programme

• Darlington Mutakwa received a URC conference grant of R9700 for SAAIR Conference 2018

• Shaunda Swarts is studying towards the National Diploma in Business Management

8. Contact with outside bodies/ social responsiveness

• Michael & Susan Dell Foundation – collaboration

• Boston Consulting Group ( 13 weeks)

• IBM Student Systems project for UCT (5 weeks)

• Educational Testing Services – ETS

• ACT (USA)

• Global Access for Postsecondary Education (GAPS)

• University of Denver, Colarado (USA)

10

CETAP ANNUAL REPORT 2018

Boston Consulting Group (BCG) Review of NBT & CETAP

The primary output from the 13-week BCG project included the following:

I. Current accessibility and uses of the NBT assessment data, strength and weaknesses of the NBT operations, a quantitative analysis that validates the diagnostic value of the NBT, and a compelling business case for the NBT to be used by institutions operating in the post-secondary school sector.

II. Summary of interviews conducted with key stakeholders, with emphasis on desired needs and opportunities to improve on the NBT.

III. Recommendations regarding opportunities for the CETAP, MSDF and other funders, based on the current post-secondary school landscape, and summary of strengths and gaps of the NBT project.

IV. A proposed business plan and roadmap for implementation of recommendations for the growth areas for CETAP

IBM CSC Project at University of Cape TownThe main task of this 5-week long project was to design a process model to support and integrate the Student Management System for the University of Cape Town’s key processes relating to Admissions, Offer Management and Registration resulting in an improved Student experience.

I. Perform a current state analysis of the Student Application, Admissions and Registration process and systems. Document a high-level Process map identifying gaps and challenges.

II. Identify the risks relating to the project scope that can impact the long-term sustainability of the University.

III. Recommend actions to address the gaps and challenges and improve efficiency.

IBM CSC Initiative with UCTInitiative with UCT

11

13

9. Distinguished visitors to unit

Guests that attended the CETAP Seminar in August 2018:

• Catherine Millet from Educational Testing Service (ETS) & Global Access to Postsecondary Education (GAPS) • Mary Tupan and Watte Zijlsra from Amsterdam• Frank Tuitt from University of Denver• Saran Stewart from Kingston, Jamaica• Prof. Michael Nettles from ETS visited the department in November 2018 when he attended and

presented the keynote at the NBT Consultative Workshop.

10. Summary statistics for the NBT Project 2019 Intake Cycle

Top 10 Venues – National Sessions 2019 Intake

Venue Number of Writers

UCT Upper Campus 6 353

Pretoria Main Campus Thuto Bld 5 885

UJ Auckland Park 3 608

UL Turfloop Campus OCH 3 261

University of Stellenbosch 2 870

Thohoyandou 2 299

UFS Bloemfontein Campus 2 153

UFH East London Campus 1 885

JHB Vega Bordeaux 1 810

Monash SA 1 756

13

CETAP ANNUAL REPORT 2018

Venues and Testing in 2019 Intake Cycle

14

Writer Numbers 2019 Intake (up to January 2019)

15

CETAP ANNUAL REPORT 2018

Session Type Number of Writers

National 71 048

Special 3 525

Disability 249

Remote 106

Writer Numbers 2019 Intake (up to January 2019)

11. Plans for 2019• New assessments for schools in collaboration with ETS • Continuation of the teacher assessment project with JET• To launch the sub domain analysis report for writers of the NBT • Standard setting workshop • NBT Consultative Forum• Invigilator training • Institutional workshop

16

CETAPCentre for EducationalTesting for Access and

CONTACT

CETAP Director: Naziema JappieNaziema.Jappie@uct.ac.za

+27 21 650 2076

Test Development Coordinator: Robert PrinceRobert.Prince@uct.ac.za

+27 21 650 5833

Operations Manager: Emlyn BalarinEmlyn.Balarin.@uct.ac.za

+27 21 650 5509

Senior Secetary: Lynia FortuinLynia.Fortuin@uct.ac.za

+27 21 650 5462