A system for enhancement A system for enhancement of manual exam reviewing of manual exam reviewing processprocess

Ivan BudišćakBoris Milašinović

Faculty of Electrical Engineering and Computing

University of Zagreb, Croatia

EnvironmentEnvironment1st year of studyProgramming and Software

EngineeringData Structures and Algorithms

◦>=700 students on each course◦Continuous assessment

Using software for automatic evaluation of assignments (Ahyco) Homeworks Quizzes (Multiple choice tests)

Manually reviewing 2 mid-term exams and final exam Class activity

Traditional manual exam Traditional manual exam reviewing (with pencil and reviewing (with pencil and paper)paper)Pros:

◦“Natural” way to review exams◦Can be done everywhere (in case you

have access to exams )Cons:

◦Difficult to split exams on more than one reviewer (lot of merging and sorting)

◦Unable to have dislocated reviewers◦Students have to be able to see reviewed

exams (additional time and place)◦Needs place for storing exams

Digital exams archivesDigital exams archivesRetain manually reviewing process Keep students exams and reviewing

remarks together in digitalised archiveMotivation:

◦ Potentially useful resource for further research i.e. finding common mistakes

◦ No need to keep thousands of papers 3 x 700 students x more that 4 papers per each

student => huge paper towers ◦ Enable dislocated reviewers◦ Enable dividing particular exam on 2 or more

reviewers◦ Avoid long queues of students just want to see

their exams

Former attempt (few Former attempt (few years ago)years ago) Students were writing code in labs under

assistants supervision and uploaded final code Code was manually reviewed using HTML editor Pros:

◦ No problems with recognizing handwriting ◦ Code indentation was able to be solved if program

had been syntactically correct Cons:

◦ Time consuming on both sides of examination process: Small lab’s capacity Many of students in the first year of study still write faster

than they type◦ Technology drawback:

Reviewing extremely time consuming using rudimental HTML editor

Limited functionality of reviewing process

New technology – new New technology – new chancechanceSilverlight

◦Cross platform and cross browser◦Rich (and fast) user interface inside

browser 2D graphics Animations

◦HTML DOM access from managed code

◦Web services

How it works (1)How it works (1)

How it works (2)How it works (2)

Main toolbar◦Browse assessments◦Browse questions◦Enter score◦Save remarks

How it works (3)How it works (3)

Exam toolbarBrowse exam papers (pages)Enter text, draw line and

polylineSelect remarkZoomSave remark template

Reviewers opinionsReviewers opinionsTakes little bit longer than “normal”

reviewing with paper and pencilNot suitable for small numbers of exams

◦Time needed to get a grip on tool’s interface and mouse movement

Opposite situation: more exams – more same errors◦Once written, comments of common errors

can be just dragged and dropped on the exam

Saves time for exams insights◦Still has to be hold, but last significantly less

Students opinions - Students opinions - questionnairequestionnaire

Question Total answers 1(%) 2(%) 3(%) 4(%) 5(%) -(%)

How often do you attend exam insights (for all courses): 1. Almost always 2. Often (cca 75%) 3. 50% 4. Rarely (cca 25%) 5. Almost never

133 6,02 8,27 17,3 34,6 33,1 0,75

When and why do you attend exam insights for manually reviewed exams: 1. Never, because there is always a large number of other students 2. I attend just to see what I did wrong but I achieved expected score 3. I attend because I want to try to get some extra points but I know what I had done wrong and I have achieved expected score 4. I usually attend because I like to know what I had done wrong and my score is less than I expected 5. I will attend because I don't understand official answers and would like reviewer to explain me difference between my answers and official answers

133 18,8 6,77 4,51 58,7 6,02 5,26

With score from 1 to 5 rate your satisfaction after exam insight (1 - not satisfied, 5 - very satisfied)

133 3,76 12 33,8 27,8 4,51 18,1

Do you attend exam insight for online reviewed exams: 1. Never, because I saw what I had done wrong and I achieved adequate score 2. I attend because I know what I had done wrong but I think I should achieve greater score 3. I attend because I don't understand reviewer's comments 4. I attend to try to get some extra points 5. I never attend exam insight

133 57,1 16,5 5,26 4,51 11,3 5,26

With score from 1 to 5 rate usefulness of online exam reviewing (1 - not useful, 5 - very useful) 133 0,75 1,5 1,5 23,3 66,9 6,02

Very useful because it is now possible to see my own mistakes, and the reviewer comments, and thus reducing the number of students who go on insights which are less crowded and everything is quickly resolved.Most of us are also curious and eager, we want to see why we do not have the expected number of points and that is why online access to the exams is very useful to us. Also, I think we are in most cases too lazy to go to the insights to see our mistakes and in this way, from pure curiosity, we all (or at least most of us) will see where we made a mistake and what official answer should look like. In this way we also learn a thing or two.

Just one question... WHY THIS THING WAS NOT INTRODUCED FOR THE ENTIRE FACULTY A LONG TIME AGO! : P

Students opinionsStudents opinions – – comments (1)comments (1)

This is an excellent idea and should be used on as many courses as possible. On the Web, we can see where we made a mistake and there is no need to go to exam insight, which is usually crowded. Also, thanks to this system, the reviewing criteria are much clearer and more uniform. [Otherwise, this is the only questionnaire that I filled because I believe that it really can achieve positive change.] Once again, every honour for this initiative and I hope that more students will complete the questionnaire, and that this method of assessment will be used for all students on this course, and in the (near) future on other courses.

Online insights are a great thing, because it allows all of us direct access to our exams. Thus saving us time that would have otherwise been lost (often in vain), on the insights and waiting in line. Praise the idea, performance and assistants in a detailed and well performed job even though this is additional burden for them!

Students opinionsStudents opinions – – comments (2)comments (2)

Might be useful if there is an official communication channel that would allow online access (something different from the e-mail). It would be very helpful for fellow colleagues that are traveling to the faculty. Distance from the faculty is one of my reasons why I don’t attend the insights and consultation.

Only 2 “negative” comments:◦ Online insights are not useful to me because I don’t

understand official exam solutions and I want assistant to explain me the difference between official and my solution.

◦ Some assistants are to harsh Note: Probably was comparing his/her and friends exams

Students opinionsStudents opinions – – comments (3)comments (3)

ConclusionConclusionStudents are very satisfiedAdditional attention needed

◦Reviews are now public (assistants should have almost unique criteria)

Reviewing lasts longer (depending on the number of exams)◦Possible improvements

Touch screen monitors Tablet PC