Upload
maude-wilkerson
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Katharina Stöttinger ISBE Berlin 2013
Measuring Vocational Competence
Written Performance Assessments beyond Multiple Choice
Katharina Stöttinger
Department for Business Education and DevelopmentKarl-Franzens-Universität Graz
Berlin, August 2013
Katharina Stöttinger ISBE Berlin 2013
How Students Feel About Tests
* Probably not an actual
Einstein quote
Katharina Stöttinger ISBE Berlin 2013
Objectives for this Session
• Gain insight into the challenges involved in developing an instrument to measure occupational skills
• Get inspiration for designing competence-based learning opportunities for students of commercial subjects/students undergoing vocational training
• Gain insights into ensuring a satisfactory level of consistency when it comes to the rating of open-ended test items.
Katharina Stöttinger ISBE Berlin 2013
Outline
• General requirements for competence based assessments
• Construction of suitable items• Construction of rating guidelines• Implication for teachers
Katharina Stöttinger ISBE Berlin 2013
General Reqirements for Competence Based Assessments
Items should • represent authentic challenges grounded in real-world working
situations• be open-ended• activate complex cognitive functions, requiring students to
– reproduce factual knowledge– contextualize it– apply it to a specific situation – and potentially transfer it to new problems with a similar structure.
• trigger responses that can be rated with a satisfactory level of consistency among various experts.
Katharina Stöttinger ISBE Berlin 2013
Case Study: A Comparison of Offers
You work for FunInTheSun, a manufacturer of sporting equipment who sponsors the regional beach-volleyball tourney.
Your boss Mrs. Kofler wants to use this occasion to distribute goody-bags with promotional freebies to V.I.P guests. Each goody-bag should contain a memory stick with the name of your company printed on it.
Now Mrs. Kofler has asked you to compare offers from four manufacturers of promotional freebies, choose the best offer and write her a memo explaining your choice.
Katharina Stöttinger ISBE Berlin 2013
Case Study – A Comparison of Offers
Consider the following context:
• Today is July, 18th. The beach volleyball tourney will take place on July, 27th. Mrs. Kofler has asked you to hurry with your decision, since she doesn’t want to assemble the goody-bags at the last minute.
• The guest list for the event contains 46 V.I.P guests.
• Your budget for the memory sticks amounts to EUR 300,-.
Katharina Stöttinger ISBE Berlin 2013
Case Study – A Comparison of Offers
Price per unit
Production Time
Minimum order quantity
Bulk Discount
Cash Discount
Expiration Date
Katharina Stöttinger ISBE Berlin 2013
Case Study - Potential Solutions
Cheapest option is also supplier with longest production time
• Prioritize cost minimization and trust that there won’t be any delays • Choose the second cheapest supplier, who is only marginally
more expensive than the cheapest option and could deliver a lot faster
• Contact a third supplier, whose offer has already expired but who would be cheaper and faster than both options and ask whether the offer could be extended
Answer should ideally demonstrate an awareness of trade-offs
Katharina Stöttinger ISBE Berlin 2013
Case Study - Competences
Competence Revealed through
Reading Comprehension
& Diligence
•picking out the relevant details from a mass of materials•noticing expiration dates and mininum order requirements•noticing differences in what is covered in the price per unit (VAT; costs of imprinting the company‘s logo)
Pertinent factual knowledge
•Applicaton of technical terms and business conventions•knowing the order of discounts•knowing the meaning of VAT
Katharina Stöttinger ISBE Berlin 2013
Case Study - Competences
Competence Revealed through
Mathematical Skills
•calculating total costs•accounting for tax and discounts
Rhetorical Skills •Verbalizing a thought process•Presenting a convincing argument
Critical Thinking Skills
•awareness of trade-offs•considering the wider context
Katharina Stöttinger ISBE Berlin 2013
Case Study – Common Misapprehensions
• Students only compare the price per item
• Minimum order requirements as a knock-out criterion
• Failure to take advantage of discounts
• General lack of insight into the interplay of quantity, price per item, discounts and percentages
Katharina Stöttinger ISBE Berlin 2013
Case Study – Learning Objectives
• Make sure that things are compareable before you compare them. (Some prices contain VAT and costs for personalizations, some don‘t).
• Price per unit is not the only factor that determines total cost.
• Order quantity required is not the only factor that determines total cost.
• Take advantage of discounts
Katharina Stöttinger ISBE Berlin 2013
Case Study – Learning Objectives
• Know the conventions of business (eg. Bulk discounts are subtracted before cash discounts)
• Be aware that certain conditions might be subject to negotiation (eg. Sometimes expired offers might be extended on request.)
• Cost is not the only criterion – time and quality matter as well.
• Be aware of trade-offs.
Katharina Stöttinger ISBE Berlin 2013
Case Study – Learning Objectives
Transferable skills:
Before making any decision• Make sure you know all the factors that
influence the potential outcome.
• Make sure you know how those factors influence each other (eg. the relationship between quantity and discounts)
• Never look at a single factor in isolation.
Katharina Stöttinger ISBE Berlin 2013
Case Study – Learning Objectives
Transferable skills
When trying to convince others of your choice• Make your choice as transparent as possible• List all the factors that have influenced your
choice• Point out any trade-offs involved• Explain your priorities with reference to the
situation at hand.
(=> Respect the complexity of a problem. Demonstrate awareness of multicausality, interdependencies, trade-offs and wider context.)
Katharina Stöttinger ISBE Berlin 2013
General Reqirements for Competence Based Assessments
Items should • represent authentic challenges grounded
in real-world working situations
• be open-ended
• activate complex cognitive functions
• trigger responses that can be rated with a satisfactory level of consistency among various experts.
Katharina Stöttinger ISBE Berlin 2013
Case-Study – Rating Guideline
Rating Guideline based on an analysis of • competences involved in a useful approach to the
problem• common misconceptions and mistakes inherent in
less competent approaches• learning objectives derived from these differences.
Katharina Stöttinger ISBE Berlin 2013
Case Study – Rating Guideline
Item The Student Rating
Calculation Checks whether any given offer is covered by the budget
0 …no
1 … yes
Takes VAT into account 0 … no
1 … yes
Takes minimum order
requirement into account
0 … no
1 … yes
Takes advantage of bulk discount 0 … no
1 …yes, with errors
2 .. .yes, no errors
Takes advantage of cash discount 0 … no
1 …yes, with errors
2 .. .yes, no errors
Katharina Stöttinger ISBE Berlin 2013
Case Study – Rating Guideline
Item The Student Rating
Choice Makes a choice and provides an explanation
0 …no
1 … yes, justification is questionable
2 … yes, justifaction is consistent
Considers expiration date of offers
0 … no
1 … yes
Considers costs 0 … no
1 … yes
Considers production time 0 … no
1 …yes
Considers possibility of re-negotiation
0 … no
1 … yes
Katharina Stöttinger ISBE Berlin 2013
Personal Mission Statement as a Teacher of Commercial Subjects
• I help people learn how to rationalize their choices.• I help people learn how to appreciate complexity.
Guiding Principles:• There is usually more than one valid approach to anything. • Still, some approaches might make more sense than others. • The best approach is the one that makes the most sense to all the people
involved in its implementation.• In order to help others make sense of your choices, make them transparent.• The goal is not objectivity, because objectivity is an illusion as soon as trade-
offs are involved. • The goal is inter-subjectivity.• There is always more than one factor to consider.• Context matters.
Katharina Stöttinger ISBE Berlin 2013
Rating Guidelines as a Tool for Teaching
Rating Guidelines• Require teachers to thouroughly reflect upon the
challenges inherent in any given learning opportunity and students‘ potential missapprehensions in those regards
• Make assessments more transparent and promote a perception of fairness
• Provide a good framework for face-to-face feedback for students, promoting students‘ self reflection
• Facilitate team-teaching
Katharina Stöttinger ISBE Berlin 2013