Embed Size (px)
Citation preview
The Unbearable Heaviness of Marking
What are the challenges of marking coursework?
– Give a numerical value as a quantification of student attainment
– Clarifying what constitutes attainment– Ensuring consistency– In a way that is publicly defensible– Under time pressure
But why do we do it?
• To give feedback• But feedback is a vast topic• Depends on the person and the task
Open University
• Feedback is, put simply; additional tutoring that is tailored to the learner’s current needs. In the simplest case, this means that there is a mismatch between students’ and the tutors’ conceptual models, and the feedback is reducing or correcting this mismatch, very much as feedback is used in cybernetic systems.
Laurillard’s Conversational Model
The Principles of Amplification
• Explain the student’s error• In a way suitable to this student
The Difficulties
• We have to imagine this unknowing• We have to dismantle our own “taking-for-
granted” knowledge• It needs to be rhetorically appropriate to the
receiver
Then we have to give it a mark
• In a way comparable to the other work we have received
We can be flexible with mistakes
• But not too flexible
Time and Volume
• 2 case studies– Marking >200 pieces of coursework for level 1
emedia design course– Marking 88 pieces of coursework for level M
multimedia specification course
>200 1st Year CourseworksNESSWF Pages bgcolorh&wbtns btn_nav2imgs filesizetweeningbtn_pos CONTENT superbtnsupercontent FINAL SCOREAds.08 5 0 5 5 0 0 5 5 5 0 2 0 0 0 0 0 29.375 np lp
Adg.08 5 5 5 5 3 3 5 5 0 0 5 0 0 0 0 0 40.5 np
Adp.08 5 0 5 0 2 5 5 0 5 5 10 10 10 0 5 5 CHECK IF MORE C0NTENT WAS THERE49.5 plag
AS0 5 5 5 5 5 5 5 5 5 5 5 10 5 0 0 10 80 alrightAdf.08 0 3 5 5 5 2 2 5 5 2 0 0 0 0 0 0 31.725 hb lp
Aha.08 0 cant see
Aha.08 0 0 0 0
Aa0 5 5 5 5 5 0 5 0 5 5 2 0 10 5 5 5 57.75 alright – check scoreAw0 5 5 5 5 2 2 5 0 0 0 2 0 0 0 0 0 30.45 npAa0 5 5 5 5 5 5 5 0 0 5 5 2 0 0 0 0 45.5 alrightAo0 5 0 5 5 5 0 5 5 5 5 0 0 0 0 0 0 37.5 hb hp bsw rt?AF0 0 0 2 0 2 0 5 0 2 0 0 0 0 0 12.925Amm.08 5 5 5 5 2 5 5 5 5 5 5 10 0 0 0 0 60.125 hb not good buttons
Anj.08 5 5 5 2 2 5 2 0 5 2 5 0 0 0 0 2 35.1 np
And.08 5 5 5 5 5 5 5 5 0 2 5 2 0 0 0 0 49 alright
Anr.08 5 5 5 5 2 5 5 5 5 0 2 0 0 0 0 0 43.475 np
Ase.08 5 5 5 5 5 2 0 5 0 0 0 0 0 0 2 33.6 np
Ac0 5 5 5 5 0 0 5 0 5 0 5 0 3 0 0 0 33.125 hp rtAs0 0Aye.08 5 5 2 0 5 5 5 5 0 5 0 0 3 0 0 0 38.4 bja
Bar.08 5 5 5 5 2 2 5 4 5 0 0 0 0 0 0 0 37.125 np rt
Bm0 5 3 5 5 5 5 5 5 5 0 5 0 10 0 0 0 57 alrightBad.00 0
Bj0 5 5 5 5 5 5 5 5 5 0 5 0 0 5 10 0 65 alrightBas.08 5 5 5 0 2 2 5 5 5 0 5 0 0 0 0 0 36.25
BaA.05 0
Ben.08 5 5 5 5 5 5 5 5 5 5 10 10 5 0 0 10 85 alright
Beo.08 5 5 5 5 0 2 5 2 5 0 0 10 0 0 0 0 39.875 btns bad
Blr.08 5 5 5 5 5 5 5 5 5 5 10 10 10 5 5 10 100
br0 0Bra.08 5 5 5 5 2 2 5 5 0 0 0 0 0 0 0 5 38.25 np - not full
BuR.07 5 5 5 5 5 5 5 5 5 5 5 0 5 5 0 10 75 alright
Bk0 5 5 5 5 0 0 5 5 5 5 0 0 0 0 0 0 no mark for buttons b ecause they did not behave like butto0ns37.5
BTN_SNDSCOOL_ANIM TOGGLE
CANTSEE
7
DOESNTSHOW
Marking Strategies
• Rating Scale– Simple numerical value for each criterion
• Checklist– Binary yes/no scores for properties of the
submission• Rubric
– A set of criteria with 3-5 attainment descriptors for each
Emedia Marksheet• Marking Scheme• BASICS• • Publish an swf file and upload it to studynet (5)• • Has correct number of pages with correct headings on each (5)• • Correct background colour (5)• • Correct width and height of the flash file (5)• • Correct number of buttons with correct colours for them (5)• • Make buttons navigate to correct frames using simple action script (5)• • Contains at least two images of you (5)• • Small filesize (5)• INTERMEDIATE• • Motion Tweening Of Position of Things in the animation welcome screen OR Motion• Tweening of Visibility of Things (for fade ins and fade outs) in the animation welcome• screen (5)• • Correct and Nice Positioning of Buttons and Content (5)• • Good easy on the eye content (text and image) not too little, not to much and all‐• relevant (10)• • Button clicks have small sounds associated with them (10)• ADVANCED• • Some very clever and visually elegant animation (using Shape Tweens, or Motion Guides• or Masking) in the animation welcome screen (10)• • Extremely well positioned and pleasant looking buttons (5)• • Extremely well judged content (5)• • An immediately visible and functioning background music toggle
• Final Score = Σbasics + (Σbasics/40) * (Σintermediate+Σadvanced)
Evolution of criteria• In some categories the the marking was binary yes/no values - and in some cases "sort of". Explanation below:• SWF - Binary - If student submitted correctly 5 if not 0• Pages - if student's work had correct number of pages, and the presentation stopped on those pages, student got 5 marks. If there was the odd one missing, or the odd one not
stopping then sort of " mark was given - e.g 2.
The background color was either right or wrong - 5 or 0.
Width and height - usually 5 or 0.
Button score varied: 5 if all buttons were of same style and were captioned according to which screen they went to then the score was 5. If the presentation used "next and forward" buttons rather than buttons going from every location to every other location (as specified by the brief) then deducationwas made.
Button navigation: If all buttons went to the right screen and stopped there then student scored 5. If any misdirection then student scored less.
If two images of the student were present, student scored 5 - if only 1 - then the score was 2.
If the file was only marginally over the filesize limit then the student had a small reduction from 5.
The presence of the tween was marked 5 or zero.
If the buttons were fixed in the same position throughout the whole app, then the student scored 5. If they juddered a little, the student scored 2. If button positions moved randomly about then the scrore was 0.
The student scored 10 if the content was well done in style and substance. If there was anything wrong with either the student scored only 5. If the content was severely found wanting - the score was 0.
If the student had uniform sounds for all buttons, the score was 10. If there was uneven sound coverage for the buttons then 5 (and potentially lower if only one button say had sound). If no sounds 0.
10 points for a cool animation which used advanced animation features of flash - with some complexity. If the animation, even if it used advanced features, but was just too trivial, then the score went down to 5
If buttons were extremely elegant then an extra 5 points were given - and similarly if the content was extremely well judged.
With the music toggle - 10 pts if it worked perfectly (the toggling and the the playing and muting of the music) - 5 points if the toggling didnt work perfectly.
Rubric
SSAM Level 2 and 3 Marksheet (a)
Assessment Criteria Typical Qualities of Grade A Work Typical Qualities of Grade B Work Typical Qualities of Grade C Work Typical Qualities of Grade D Work Typical Qualities of Grade E Work
Conceptual Development – Engagement with the processes of invention, ideas development, and creative and critical reflection and evaluation. The use of strategies for identifying and solving problems individually and through team work
Solutions are innovative and reframe the task within its context. Considerable depth of engagement and successful media experimentation clearly documented. Detailed planning and clear rationale for decisions and account of design processes.Strong group work skills.
Solutions relate directly to task and may step beyond conventionsStrong engagement with visualisation media and processes, evaluation of alternatives, solutions come from processWell-documented and detailed experimentation and planningEffective member of group
Solutions are appropriate to task, work well within conventionsGood use of visualisation media and planning processes, range of alternatives put forward and evaluatedExperimentation to support implementation within conventions
Solutions limited to task and address conventions Solutions found or adoptedSome planning but completion is rushedSome experimentation and limited alternatives Works within group ethos
Solutions reframe task inappropriately and do not address conventions.Basic use of visualisation strategies, few alternatives, limited evaluation.Little experimentation
Contextual Awareness –An informed response to instances of (i) digital media practice and its artistic and cultural contexts (ii) current and emergent technologies and technical contexts
Well informed and developed personal response to digital media and their broader contexts.Clear and effective discussion of examples.Contextual awareness integral to practical outcomes.Clear awareness of personal relationship to tradition and practice.
Detailed discussion of appropriate examples from many sourcesClear interrelation of examples and broader contextsContextual awareness informs task
Examples used well, some discussion and links to broader context.Good grasp of contexts, but sometimes restricted to immediate concerns.
Examples used to illustrate contextsReference to broader contexts based on given sources Limited interrelation ofexamples and their broader contexts
A few examples, drawn from given or obvious sources.Very limited reference to broader contexts.Little engagement with new areas or knowledge.
Assessment Criteria Typical Qualities of Grade A Work Typical Qualities of Grade B Work Typical Qualities of Grade C Work Typical Qualities of Grade D Work Typical Qualities of Grade E Work
Enquiry and Communication – Breadth and depth of information gathering and evaluation; synthesis of ideas from diverse sources; the use of written, visual, interactive, oral and sonic communication skills.
Synthesis and analysis which draws on many sources. Original responses to those sources articulated clearly in appropriate forms and conventions.High level of achievement in use of media skills.
Synthesis, analysis, and descriptions used appropriatelyEffective use of information gathering and recording of sourcesSelection from many sources steps beyond the obvious, Consistently accurate communication, well expressed and accurate
Some synthesis: tends to analysis and description.Steps beyond given sources.Uses information gathering strategies well.Expression is generally accurate but may not flow well.Uses conventions and structure well.
Analysis and descriptionRelies on given sources or limited to obvious sourcesA few inaccuracies in communication, some difficulty with expression of ideasClear overall structure and approach
Some Analysis, tends towards descriptionDoes not make full use of given sourcesSome inaccuracies in vocabulary and syntax, ideas not expressed or developed wellFragmented text and structure
Practice –Using appropriate processes, techniques and technologies for the realisation of individual, or collaborative, artefacts that address agreed or given briefs
High level of attainment in aesthetic and technical domains.Work has a wholeness and personal style, and may be ambitious in scope and intention.New and complex processes engaged with positively and successfully.
Strong achievement in the aesthetic and technical domainsConsistency, clear purpose and wholeness of the workEffective use of appropriate processesConfident engagement with new and/or complex processes
Good achievement in the aesthetic and technical domains.Work has consistency
Average achievement in aesthetic and technical domains – some inconsistencies in the work and it may lack resolutionEstablished Skills used well, some new skills employed, Range of processes limited to immediate solution to task
Inconsistent achievement in aesthetic and technical domains – work lacks resolution and depthLimited engagement with appropriate processes – may rely on previously established skills and processes handled with some competence
SSAM Level 2 and 3 Marksheet (a)
The value of a rubric
• helps faculty score students' work accurately, fairly, a reliably and quickly
• requires faculty to be much more precise about criteria for evaluating student work • can lead to shared standards among faculty about what make a good essay or project • gives students a better idea about the qualities their work should exhibit
• sets standards for .student achievement and make the standards explicit
• allows students to self-assess work prior to submission and to offer feedback to peers
• helps students understand the meaning of a grade;
• improves student performance, particularly among lower achieving students
• can lead to substantive conversations between faculty and students about the requirements for excellence within the field or discipline. Involving students in developing the rubric can be very beneficial
The drawbacks of rubrics
• developing rubrics tends to be hard, time-consuming work • it is difficult to find just the right language to describe qualities of student performance that
distinguish between mediocre and excellent work. If a student can achieve a high score on all the criteria and still not perform well at the task, you may have imprecise or inappropriate criteria
• it is difficult to find the right language to describe differences among student performance levels for each important criteria or quality. It should be more than just that poor work has less of something than good work.
• rubrics may require numerous revisions before they work "just right.“
Sometimes it is difficult to mark with rubric if the student work shows “incoherent attainment”. For instance very wise statements amid very absurd ones. In these cases both a high value descriptor and a low value one have elements of truth in them.
Write a Rubric for Participation in a Discussion Forum
1
2
3
Model Answer!
From: Blended learning in higher education By D. Randy Garrison, Norman D. Vaughan
Issues with model answer
My View
• As much as possible, the statements in rubrics should be expressed in subjective terms
• Keep objective measures for checklists
Multimedia Specification Assignment Briefing
You will be given a minimal content prototype developed by another group. You will evaluate it by producing a five minute long slide show containing screen captures of the application and an audio commentary by yourself. You can also produce slides using text and graphics in order to illustrate concepts and to emphasize points. The slide show should be a sequence of audio and graphics and be runnable in •Powerpoint•SmirkThe slide show should be watchable in its entirety as a sequential presentation not requiring user interaction. However you should provide some navigational functionality.
Category CriteriaTechnical Correctness A fully functional slide show is created of at least
10 slides of visual and audio material. Analysis should be made primarily through audio. There should be no broken links, audio not playing or other problems.
20%
5 Audio quality perfect, many slides, lasts between 5-8 minutes, slide visual quality extremely good – screen grabs are clear and annotations perfectly readable. Specific titles for slides.4 High quality audio and plentiful slides. Potentially some minor flaws.3 Audio clear but not necessarily good. May have too few slides, may lack titles for slides.2 Audio quality is poor, or too few slides, or too short.1 Audio very bad and difficult to listen to without editing by the tutor[audio quality is understood as: not too soft, not too loud, not distorted, without background noise]Holistic Evaluation Clarity of presentation. References to similar sites
on the internet.
20%
5 Insightful personal response given to a number of aspects of the site (can include colour choice, visual balance on pages, appropriateness of font, location of bugs and spelling mistakes). Generates some extremely clever and insightful points.4 Very good response to the site. Makes a number of interesting points.3 Makes a number of good but very obvious points.2 A very small and fleeting personal response to the site. Makes some untenable or ill-considered propositions inappropriate to the site under consideration1 Very lame and brief comments on these aspects of the site
Heuristic Evaluation using Nielsen: Diagnosis and Recommendations
The correctness with which you connect usability principles and elements of the site you are evaluating
Your ability to propose a solution to a usability problem encountered. The correctness of any praise of a good implementation of a usability principle.30%
5 A brilliant, detailed and very shrewd application of heuristic analysis to the site. At least 3 heuristics are covered and very correct instances of violation or upholding found. Good alternative implementation strategies advanced. Some very original insights generated.4 A very good application of heuristic analysis to the site covering at least 3 heuristics– with good alternative strategies advanced. Alternative implementation strategies are valid.3 (a) Makes a number of acute comments about the site without correctly associating them with a Nielsen heuristic (b) makes some correct use of Nielsen heuristics but not enough2 Does not actually use Nielsen, or if you do, it is misapplied with various problems going under the wrong headings. Very few actionable points made. Alternative implementation strategies not necessarily credible.1 Completely misapplies Nielsen wherever his heuristics are used – or does not use the heuristics but says nothing which might fall under one of the heuristic headingsInformation Architecture: Diagnosis and Recommendations
The correctness with which you connect the stages and practices of the Information Architecture methodology to the site you are evaluating (particularly in the areas of audience, and grouping and labelling of content).
Your ability to propose solutions to problems encountered during analysis. The correctness of any praise of a good implementation of the Information
Architecture method.30%
5 Explicitly and shrewdly applies concepts like grouping and labelling of content and scenarios and goal and mission to the site and evaluates the choices made and offers alternative solutions to problems. Looks at alternative grouping and labelling on other sites and compares them in a very specific and detailed fashion4 Looks at the sight from the point of view of the information architecture method- correctly applies various phenomena in the site to stages in the IA method.3 Makes a number of good points which could fall under the heading of IA but are not explicitly stated as such – or misses obvious things which should be clear from an understanding of the IA methodology.2 Makes some but often poor points which impinge somewhat on the IA categories but not very well1 Makes a number of flawed conclusions relating to IA and doesn’t explicitly reference those categories
But then you have to mark
• Technological aids: Googledocs allows you to create forms which will write to spreadsheets
• Studynet too – next academic year will allow the use of departmentally approved rubrics
Marking
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 7512:00:00 AM
12:28:48 AM
12:57:36 AM
01:26:24 AM
01:55:12 AM
02:24:00 AM
02:52:48 AM
03:21:36 AM
Series1
Averages
• Overall Time Taken: approx 31 hours• Breaks: 17 hours
– Break understood as any gap longer than one hour between consecutive markings
• Average time to mark (without breaks being factored in) approx 39 minutes
• After factoring in breaks 11 minutes
Breaks
• Sometimes a break represents genuine recovery time between markings
• Sometimes leisure• Mentally I experienced it as necessary after
every 6 markings
Very different from checklist
• Between 20-30 a day• Did not feel tired after marking numbers• Easier to mark• More enjoyable artefacts• Did not require free text feedback
Quality of Feedback
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 880
50
100
150
200
250
300
Series1
Number of Words of Freetext Feedback Given – average 79
Top Two• This was difficult to mark in that it was done in a way totally contrary to
the spirit of the assignment - which states:
"five minute long slide show containing screen captures of the application ...text and graphics ...The slide show should be a sequence of audio and graphics...should be watchable in its entirety as a sequential presentation not requiring user interaction."
unfortunately there were no graphics, the presentation did require my intervention continually - and while there seemed to be a lot of audio files - only a subset of them seemed to play in powerpoint. Therefore I have decided to give onefor the technical correctness section - in that none of the descriptors really relate to what you have produced - and will judge the other categories on their merits.
The point about the bars in the holistic evaluation section was well made - as was the positioning of about. However, you did not mention "aesthetic and minimalist design" which is the heuristic which most screams to me the problems of this site. You make a good point about the grouping and labelling of content in the information architecture section - and earlier in the holistic section you talked about the intended audience but this needed to be taken further - with comparison to other websites - and some deliberation on how the grouping and labelling of content might be better adapted to its presumed audience.
• Overall not bad - you make a number of good points - the font and button background contrast for instance - also in your navigation through the site you expose some issues like the difficult to find rooms "gallery" button. You also make a number of good points in the heuristic section - but often under the wrong heading. For instance the non-linking "Taj" logo (which should go back to the homepage but doesnt). This, most clearly is a violation of "user control and freedom" (we cannot get out of unwanted states) - and more tangentially violates consistency and standards (typically logos positioned there takes us back to the homepage). However, you associated it correctly with the latter - but also with "flexibility and efficiency of use" - which is much less relevant (this heuristic is more for things like shortcut keys - i.e. doing the same thing in a variety of ways).
Overall (this is the feedback I would like you to take forward into other work you do on this msc - particularly your final project report) - the points you made didn't really fit into an organized narrative. You said some valid things in the information architecture section - but you did so in a meandering way. Try in future to be much more organized in the categorization and sequencing of what you want to say in order that it is clearer and has more impact. You do say interesting things - so make sure their order is right!
Bottom Four
Rather poor. The only real valid observation is the suggestion of moving the image which obscures the title. Unfortunately you did not really say anything beyond that.
Not bad - some good comments about the information architecture - the size of buttons - the badly functioning search box - good on consistency and standards and the terminology of "tactics" and "skills". Not bad
A very nice review. Shrewd comments about the information architecture- particularly problems with the site title, and your reflections about who the true intended audience are - are extremely good. Your heuristic comments and holistic also very good.
Some good points on button labelling - the left button - and problems with abbreviation. More effort needed to be made on the information architecture section - maybe viewing other websites - delving more into the grouping and labelling.
Same Assignment Distance Learning 6 Students
1 2 3 4 5 60
50
100
150
200
250
300
Series1
The Lowest number of words
xxxx - absolutely superb analysis. The information architecture section particularly good. Your description of the issue with "forms" appearing under "services" and the confusion with "downloads" was truly magisterial. And your alternative IA through a diagram was very well done. At times I thought you were maybe a little too critical - I found the horizontal menus a nice piece of innovation rather than a transgression of consistency and standards. However the level of detail in your critique was amazing - the part on permit and permits an example of this. Overall an extremely good analysis.
HE Academy Recommendations• setting realistic targets for achievable turnaround time on assessed work• making the schedule of feedback on assignments clearer to students• timetabling of assignments more evenly through the academic year to avoid bottlenecks• instituting an effective monitoring system for the timing and placing of assessments• providing more timely feedback through activities such as assessment in class time• increasing the number of staff to ensure better staff-student contact on feedback issues• providing feedback in alternative forms• involving students in feedback processes, for example, using them as coordinators• introducing standardised feedback systems, such as standardised forms, hand-in and return
procedures• ensuring feedback is given promptly and not delayed by external moderation• auditing practice within an institution and encouraging the spread of effective practice• ensuring that assessed work provides a basis for future improvement by• (a) indicating how future improvement might be achieved (b) ensuring• learning structures (such as semesterised modules) do not foreclose on• the possibility of improvement• using one-to-one tutorials for feedback on assessed work.
Assessment and feedback (Questions in the National Student Survey)
Q5 The criteria used in marking have been clear in advance
Q6 Assessment arrangements and marking have been fair
Q7 Feedback on my work have been prompt.
Q8 I have received detailed comments on my work
Q9 Feedback on my work has helped me clarify things I did not understand
My recommendations
• If marking <100 students use rubrics, if more, use checklists
• Recognize the flight structure of marking– Take off, cruising, landing
• Make rubrics contain primarily subjective criteria requiring interpretation
• Make rubrics simple (no more than 4 criteria per artefact)
• Revisit rubrics when writing module evaluation!
