Ontario’s experience in AHELO’s Civil Engineering Strand

Presented by: Mary Catharine LennonSenior Research AnalystHigher Education Quality Council of Ontario

Presented to: State Higher Education Executive AuthorityAugust 9, 2012Chicago, Illinois

Ontario’s experience in AHELO’s Civil Engineering Strand

Informing the Future of Higher Education

2

Canada is 1 of 9 countries participating in EngineeringAustraliaCanadaColumbiaEgyptJapanMexicoRussiaSlovakiaUnited Arab Emirates


3

HEQCO is implementing AHELO on behalf of MTCU


Council of Ministers of Education (CMEC) is the Canadian representative to OECD’s Education

Branch

Ministry of Training, Colleges and Universities (MTCU) is a member of

CMEC

HEQCO is an agency of MTCU

What is HEQCO?

“The objective of the Council is to assist the Minister in improving all aspects of the postsecondary sector, including improving the quality of education provided in the sector, access to post-secondary education and accountability of post-secondary institutions.”

5

Ontario activities timeline


July 2011

HEQCO joined project

October

Recruited institutions

November

Hired lead scorer

December

Vetted and adapted the documents

January 2012

IC training and ethics

February

Recruitment, IT

preparations and TA training

March

Student testing

April

Faculty and institutional

survey collection

May

Scoring

6

AHELO’s Civil Engineering strand• Pilot managed by the Australian Council on

Educational Research (ACER)• Expected learning outcomes were developed

through Tuning process• Assessment tool developed by ACER, Japan’s

National Institute for Educational Policy Research (NIER), the European and Global Engineering Education network (EUGENE), in collaboration with international Engineering education experts


7

The Engineering Test frameworkUnderstanding and application of design methodologies to meet requirements

Practical competencies required to solve problems, conducting investigations, and designing engineering devices and processes. Covers non-technical elements of civil engineering practice like professional ethics, responsibilities and the impact of engineering solutions in a global economic, societal and environmental context

Using analytical methods to identify, formulate and solve problems

Effective communication and awareness of the wider civil engineering context

Knowledge and understanding of underlying scientific and mathematical principles – general sciences; materials and construction; structural engineering; geotechnical engineering; hydraulic engineering and urban and rural planning


Engineering design

Engineering practice

Engineering analysis

Generic engineering

skills

Basic engineering

skills

8

The test• Students are provided 2 of 3 performance

tasks, each requiring 30 minutes to complete• Students are provided 20 of 40 multiple choice

questions, together requiring 30 minutes to complete

• Students complete the context survey


9

How it works

• Online tests and contextual questionnaires are completed by students submitted directly to ACER for evaluation

• Short written responses are graded by trained Ontario-based Civil Engineering faculty and submitted to ACER

• All analysis of Civil Engineering strand occurs at ACER

• ACER provides result of Engineering strand to OECD


10

9 of 10 Ontario universities participated


http://www.mcmaster.ca/home.cfm

http://www.uottawa.ca/welcome.html

http://www.ryerson.ca/index.html

http://www.utoronto.ca/

http://www.uwindsor.ca/

http://carleton.ca/

11

Student recruitment• Having specific and small

population simplified recruitment

Recruitment examples: emails, websites, classroom announcements, meetings of class and civil engineering student society representatives


http://www1.carleton.ca/cee/oecd-ahelo-test/

12

Student incentivesLimited funds and lots of creativity!

Incentive examples: class pictures, voucher for formal end of term dinner, donation to the engineering societies, individual gift cards, increased amount on gift card if certain percentage of participation achieved, food after exam, raffles for ipad3


13

The resultsOntario Institutions Students who wrote

test/populationFaculty who completed survey/population

Institution 1 61/77 = 79.2% 2/18 = 11.1%

Institution 2 23/34 = 67.6% 18/20 = 90%

Institution 3 87/137 = 63.5% 17/18 = 94.4%

Institution 4 57/100 = 57% 16/18 = 88.9%

Institution 5 65/114 = 57% 25/34 = 73.5%

Institution 6 43/90 = 47.8% 27/40 = 67.5%

Institution 7 45/77 = 58.4% 19/32 = 59.4%

Institution 8 26/44 = 59.1% 15/18 = 83.3%

Institution 9 35/56 = 62.5% * 16/17 = 94.1%

Total 442/729 = 60.6 155/215 = 72%


*Note: Institution 9 had 62.5% of their students write the test, but only 5.4% were electronically captured, altering the final numbers for analysis to 56%*

14

AHELO Transmission of Results

• ‘Institution Report’: Institutions receive a report detailing the performance of the cohort of students from their individual institution

• ‘Jurisdictional report’: will outline the performance of Ontario’s Civil Engineering programs

• ‘Inter-jurisdictional report’: Australia and Canada have agreed to share data for comparative purposes

• ‘International Report’: The OECD will provide a final report reviewing the feasibility study


15

Challenges and successesChallenges Successes


• Institutional Research Ethics Board approval made it data linking impossible.

• Technical issues at one institution

• Considerable ‘buy-in’ from all levels

• Discipline-strand simplified sampling and recruitment

• Weekly Institutional Coordinator teleconferences were extremely useful

Lessons Learned• Get to institutions early in the academic year• Allow time for Ethics approval• Ensure adequate IT support

16

Find out more! Visit heqco.ca


http://heqco.ca/

Documents

Ontario’s experience in AHELO’s Civil Engineering Strand