Assessing the Assessment: The Transition from In-Person to

Online Engineering Classes

Alternate Approaches to Assessment in QR Courses

William TsaiAssociate Professor, Mechanical Engineering

8 September 2020

Agenda• Spring 2020 Transition to Online Exams• Exam Assessment and Response• Helpful LMS Tools

Spring 2020 Transition to Online Exams

Before the COVID-19 Shutdown• Two classes

• Thermodynamics (2nd year): 2 sections, ~40 students• Heat Transfer (3rd year): 2 sections,~40 students• Experiences in both courses were similar

• Completed Exam 1 approximately 2 weeks before• 50-minute, in-class exam• Results were nominal

• Planning on Exam 2 in 2-3 weeks afterwards

Exam 2: Modifying an In-Person Exam• Based on 1 hour in-person exam• 2-hour online exam offered during a 9-hour window• Open book, open notes

• Reduce motivation for cheating• Handwritten work submitted after the exam

• Credit for “minor” errors with corrected problem and review of work within 24 hours

• Gave sample exam to familiarize students• Collected feedback about the sample• Generally students thought it was about right in terms of length

… Well that didn’t go according to plan

0

5

10

15

20

25

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100

Exam 1 Exam 2

Mean: 45.2Avg. Change: - 41.9

Exam Assessment and Response

Poll: What did students say about the exam?Choose your top 2 guesses

• Questions were too hard (conceptually difficult)• Questions were too long (lots of algebra or too many steps)• Questions were unclear and unlike examples• Exam was too long (too many questions)• Other concerns

Student Feedback on the Exam

• Problems were average difficulty and met expectations

• By far the biggest concern was time

• Needed additional 10 minutes to 10 hours

• Problems took more time than the sample exam

• Workflow issues

• Going back and fourth between LMS and written work took more time

• Hard to come back to problems, esp. those with long solutions

• Some students did not have a dedicated two-hour block of time

More Formative

Assessments

Better Utilize LMS Tools

Reevaluate Time Limits

Improvements Based on Feedback

Focus on Learning

Outcomes

Changes for the Final Exam• Basic exam structure was like Exam 2• Larger, more comprehensive practice exams provided• Questions more directly targeted at learning outcomes

• Sum of questions yielded similar complexity to in-person exam• Exam occurred over a 3-day duration

• Students could start exam and return at any time in that period• Student required to submit their handwritten work

0

5

10

15

20

25

0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100

# St

uden

ts

Exam 1 Exam 2 Final

Final Exam Results

Mean: 79.4Avg. Change: - 7.4

Summary of Lessons Learned• Online exam is not an in-person exam you post online

• Workflow and time is not what you expect• Carefully consider your questions

• Does the question align with the learning outcome being assessed?• Use more formative ”quizzes” in LMS

• Use low stakes, graded activities familiarizes you and students • Provide feedback on unexpected challenges

• Pay attention to the little details• Problem appearance, interpretation can affect pacing

Helpful LMS Tools

Why use LMS Quizzes?• More to LMS quizzes than multiple choice or simple numerical

solutions

• Automated feedback means less time grading• Does require more work upfront

• Can create variation in problems• Mitigates basic cheating concerns

• Facilitates building more formative assessments• More practice before summative assessment

Helpful LMS Tools• Formula Based Questions• Questions Banks• Repeated Attempts and Regulated Feedback

Formula Based Questions• Allows for a question with variable values• Allows for variable answers

• However solution process remains the same• Requires rethink of problems to take advantage of variations

• Demonstration

• References• Instructure Canvas Help for Formula Questions• D2L Brightspace Help for Arithmetic Questions

Question Banks• Central location for you to store you quiz questions• Allows variation of questions for the same assessment• Also good for question organization

• Demonstration

• References• Instructure Canvas Guide for Question Banks• D2L Brightspace Documentation on Question Libraries

Multiple Quiz Attempts• Easily setup multiple attempts at a quiz

• Useful if you want them to demonstrate mastery • Can be based on performance• Very useful for formative assessments (e.g. homework, sample exams)

• Can control what student sees after quiz

• Demonstration

• References• Instructure Canvas Guide for Quiz Options• D2L Brightspace Guide to Creating and Managing Quizzes• D2L Brightspace Guide to Customizing Submission Views

Application

https://docs.google.com/document/d/1KhvkfhqPvDVyzgQUkG9tE

KqVPR1phoyJE2fsJnS09H0/edit?usp=sharing

• If you brought a sample quiz or exam, consider how you might

apply ideas from this presentation or other presentations in the

series

• Breakout into small groups, discuss, and share your ideas on

the Google Doc

Thank You• Hope you found some element of this discussion helpful• Welcome your feedback, thoughts, and suggestions• Happy to share materials

• Please feel free to contact me with questions• wtsai@csum.edu