Upload
vubao
View
226
Download
2
Embed Size (px)
Citation preview
BENGKEL PENILAIAN
OUTCOME BASED EDUCATION
8 Julai 2013 Dewan Delima, DeTAR Putra
Penceramah: Dr Aishah bt Abu Bakar
Pengarah, Bahagian Pengurusan Pembangunan Akademik
J abatan Pengaj ian Tinggi
1. Reflection - Current assessment practice and the need to change
2. Application of Constructive Alignment at subject level 3. CQI and closing the loop 4. Developmental Framework in programme design
ACTION
ADJUSTMENT
Bengkel Peniiaian Outcome Based Education
Te chIng for Quality Learning
How are they related
• Constructive Alignment
• Outcome-based Education
• Malaysian Qualification Framework
3
Towards MQF - basis for quality assurance of
higher education
In line with internationally recognized good
practices (like Washington Accord for
Engineering Education)
Areas of Evaluation in MQF
1. Vision, Mission, Educational Goals and Learning Outcomes; 2. Curriculum Design and Delivery; 3. Assessment of Students; ------,---------, 4. Student Selection and Support Services;
HEARTofQA5. Academic Staff; 6. Educational Resources; 7. Programme Monitoring and Review; 8. Leadership, Governance and Administration; and
P"""~-"'!!"'"~----~--~" 9. Total Continual Quality Improvement.
Bengkel Penilaian Outcome Based Education
6
Bengkel Penilaian Outcome Based Education
Teaching Assessment Curriculum
Lecture ExaminationList of topics to be covered
Tutorial Test
Laboratory Assignment
Teacher's and Student's Perspective
Intended LO -+ [T&LActivities I AssessmentJII
LearningOutcomes Assessment Activities
I Institutional Mission I Stakeholders Needs
c:; Program Educational Objectives (PEO) (desired skills of alumni)
•Program Outcomes (PO) (desired knowledge, skills and attitude of graduating seniors)
(considers attribute as specified by Accreditation body)
• --Outcome Indicators (Assessment Instruments & Methods)~~l Performance Targets (Criteria for acceptable performance)
I
Course Outcomes (CO) Course Learning Objectives
Bengkel Penilaian Outcome Based Education
Reasons for writing LO • Help to guide students in their learning as it explain what is
expected of them in turn helping them to succeed in their
studies
• Help staff to focus on exactly what they want students to
achieve in terms of both knowledge and skills
• Provide a useful guide to inform potential candidates and
employers about the general knowledge and understanding
that a graduate will possess.
What are LO • Specific intentions of a programme or module written in
specific terms
• In the Malaysian Quality Framework (MQF), LO is a
statement on what students should know, understand and can
do upon completion of a period of study (bearing in mind the
MQF qualification level: Advanced Diploma, Bachelors,
masters, Doctoral)
10
Bengkel Penilaian Outcome Based Education
9
ASSESSMENT theOBEway
Aishah Abu Bakar
Bengkel Penilaian Outcome Based Education
Continuous assessment -lack of systematic feedback
SOIL MECHANIC S I
SEMESTER 2 I SES12008-2009
.l. Find~.
I No Malric No Laboratory/)O Cant a98mnl 25
I f, IT1 [,lT2 Int aubllll T1 Cw Tllt
I 1 KEA060002 7.0 7 .0 5.0 19.0 1.5 7.5 3.0
2 KEA060011 10.0 8 .0 10.0 28.0 6.0 10.0 . 3. 5
3 KEA.OG OO 12 10.0 8.0 10.0 28.0 2.5 6.5 4.0
, 4 I<EA060014 10.0 8.0 10.0 28.0 8 .5 1 10.0 5.0
~ KE.A0600 15 10.0 8.0 1
10. 0 28.0 5.0 10.0 4.5
6 KEA06001 7 7.0 7 .0 110.0 24.0 9 .0 6.5 i 5.0
7 KEA060030 7.0 7 .0 10.0 24.0 4 .5 6.5 1 4.5
I 8 KEA060034 7.0 7.0 10.0 24.0 6.0 6.5 4.5
9 KEA060040 7 .0 7 .0 6 .0 20.0 7.0 I
8.5 4.5
sublot
13.0
19.5
13.0
23.5
19.5
20.5
15.5
17.0
20.0
Limited feature in the current system to incorporate assessment data on the other two domains - affective and psychomotor
SOIL 1EC HANIC S I
SE ~E STER 1 I SE SI 2 008 · 2009
I No M.trIoNo LaborOI(or,130 COM ..ssmnt 125 EJonU45 Gu.nd r--
NlTl MT2 NlT3 ..b<ot T1 c 'W Tut . "btot $ 'IIbtot Tot~
" K E AOBOO 3 6 6.0 6 .0 6 . 5 1 ••5 6 . 5 5.5 0 . 0 1 2.0 8 .0 38.5 I
2 K EAOSOO42 6 . 0 6.0 6.5 18. 5 5.0 5.5 0.0 10.5 19.0 48.0
:3 KEA070004 6 .0 6 .0 7 .0 19.0 6 .0 7.5 2.5 16.0 14. 5 49.5
I 4 K EA070010 6 . 0 6.0 7.0 19.0 7.5 8.5 5.0 21..0 38.0 78.0
I 1> K EA070 011 I 6.0 6.0 6.5 1 ••5 5 . 5 10.0 4 .0 19.5 45.0 83. 0
6 KEA070D 12 7.0 7.25. 7 . 5 21.8 5 . 5 10.0 2 . 0 17.5 19. 5 58.. 8
I ' K E ."'-070 01:3 1
23.3 6.0 I 1 6.0 73.3? 7.0 7 .25 9.0 6.0 1 L4 . 0 34.0
8 K E A 070016 7 .0 7.25 6.5 20•• 3.0 6 .0 4 .5 13. 5 18. 0 52.3
9 KEA070018 7.0 7.25 6 ,,5 20.8 5 . 5 I 6.0 5.0 16.5 39. 0 76.3
10 K EA070019 7 .0 7 .25 6 . 5 20•• 8 .Q 6.0 3 .5 17.5 36. 5 II 74.8
I 11 K EA0700 22 7.0 7.25 6.5 20.8 7.0. 6.0 2.0 15.0 25.0 60.8
12 K EA0 70024 I 7 .0 17 . 25 6.5 20.. 1 . 0 10.0 2.0 13.0 28.0 61.8
13 K EA070 0 25 I 7 .0 7 .25 9.0 23.3 5 . 5 6 .0 5 . 0 1 6.5 37.0 76.8
I 14 K EA070026 I 7.0 7.25 7.0 2L3 1 . 5 8 . 5 2.0 ,~ 29.5 62.8
Bengkei Penifaian Outcome Based Education
ICOURSE LEARNING OUTCOMES (CO)
PROGRAMME LEARNING OUTCOMES (PO) METHOD OF
ASSESSMENT PO PO PO PO PO PO PO PO PO PO
DELIVERY
1 2 3 4 5 6 7 8 9 10
Define soil as in engineering context and Lectures and Examination short relate problems associated with the I coursework reportdefinition with Ilocal' soil condition " (PBL) CW Interview
Identify and differentiate the different types of soil and their properties and
Lectures and Short test and
classify soil using British and I or Unified I I laboratory'i \'
laboratory work Soil Classification System. work/report
Conduct laboratory tests for Demonstration Direct observation determination of soil index and
, ,,' I and laboratory on laboratory" " compaction, work, work/report
Solve calculation problem using mechanics involving physical properties, -.; -.; Lecture and Examination & compaction, seepage and effective tutorials graded tutorial. stress. Showthe use of soil mechanics concepts
Lecture,in engineering works ,j 'i , i coursevvork. Short report &
(PBL) group interview
Bengkel Penilaian Outcome Based Education
Course V.AEA2132 SOIL MECH ANIC I
legends;
LO Learni ng Outccme Cede Af2 INTEREST
LO Dese Le arnir.g Outcome Desc ription C02 ANAL '(TICAL & CRITI CAL
AM Assessm ent P.lethod COL KNOW LEDGE
CO Cog nitive PS~ OeSER ~nON
AF Affect ive PS2 DYNAMISM
PS $Tfehomotor PS 3 ~lA.NlpuLAnON
17
-~~ .... ....................--,..,-~-....,.- ......_................................-- ......
~-- .........- ...... ......... (\IlOWM\/- Refer to list of assessment
methods extracts from 5 Bloxham and P. Boyd" Developing Effective Assessment in Higher Education A Practical Guide"
18
Bengkel Penilaian Outcome Based Education
Possible L.T.As Common I LOs Possible Assessment Tools
I Set reading, lecture, report on field .. Describe .. Assignment, essay question exam trip, write essay
-
Tutorial, activities, write essay, ... Explain .. Assignment, essay question exam, oral presentation
Project, assignment ... Integrate ... Project, assignment
Project, case study ... Apply .. Project, case study, experiment
PBl, case study ... SoJve problem .. Case study, project, experiment
Project, poster ... Design, create .. Project, experiment, poster
Reflective diary ... Reflect .. Reflective diary, portfolio, self-assessment
presentation, debate, role play, A range of oral, writing or listening tasks addressing the IlOs, e.g .
... Communicate ... presentation, debate, role play, reporting, assignment, precis,
paraphrasing, answering questions
The point is not how you are go"ng to teach but how and what you want your students to learn.
NOTE! Many o/these TLAs can be assessments tasks as well. Then you have excellent alignment.
Knowledge Ability to observe and remember previously learned Information; knowledge of specific facts, terms, concepts, principtes, ideas, events., p_laces etc; mastery of subject material
Recall or Recognize information
multiple-choice test, recount facts or statistics, recall a process, rules, definitions; quote law or procedure
arrange, define, describe, label, list, memorize, recognize, relate, reproduce, select, state
2
3
Com prehension Ability to understand Information and grasp material, lTanalating knowledge from one form to another; Interpreting, comparing a nd contrasting materia I; predicting consequences and future trends
Application Ability to use information, learned material, methods, concepts, theories, principles, laws and theories in new situations, problem solving using required knowledge or skills
Understand meaning, restate data in one's own wordS, interpret, extrapolate, translate
Use or apply knowledge, put theory into practice, use knowledge in response to real circumstances
explain or interpret meaning from a given scenario or statement, suggest treatment, reaction or solution to given problem, create examples or metaphors
a theory into practical effect, demonstrate, solve a problem, manage an activity
explain , reiterate, reword , critique, classify, summarize, illustrate, translate, review, report, discuss, re-write, estimate, interpret, theorise, paraphrase, reference, example
use, apply, discover, manage, execute, solve, produce, implement, construct, change, prepare, conduct, perforrJl, react, respond, rolE2play
Bengke/ Peni/aian Outcome Based Education
4
5
6
Analysis Ability to breakdown material and recognition of organization structure; identlrlCation of components and relationships between components, recognition of patterns and hidden meetings
Synthesis (Create/build) Ability to combine parts or apply prior skills and knowledge to produce a new whole; integrate ideas Into a solution; generalize from a given facts; propose a plan action; fonnulate new classification methods.
Evaluation Ability to judge and alsess the value of theories and p ....... ntstion, based on their value, logic or adequacy, for a given purpose; compare and make choices based on reasoned argument, verify the value of evld.nce, recognize BubJectivlty
Interpret elements, organizational principles, structure, construction, internal relationships, quality, reliability or individual components
Develop new unique structures, systems, models, approaches, ideas, creative thinking, operations
Assess effectiveness of whole concepts, in relation to values, outputs efficacy, viability; critical thinking, strategic comparison and review, judgment of relating to external
identify constituent parts and functions of a process or concept, or de-construct a methodology or process, making qualitative assessment of elements, relationships, values and effects; measure requirements or needs
develop plans or procedures, design solutions, integrate methods, resources, ideas, parts; create teams or new approaches, write protocols or contingencies
review strategic options or plans in terms of efficacy, return on investment or cost-effectiveness, practicability; assess sustainability; perform a SWOT analysis in relation to alternatives; produce a financial justification for a proposition or venture, calculate the effects of a plan
analyse, break down, catalogue, compare, quantify, measure, test, examine, experiment, relate, graph, diagram, plot, extrapolate, value, divide
develop, plan, build, create, design, organise, revise, formulate, propose, establish , assemble, integrate, rearrange, modify
review, justify, assess, present a case for, defend, report on, investigate, direct, appraise, argue, project-manage
21
~~,~-;--.'~I",",~~~~ 'L:, ....... ~ "~ '" ~-- .-. ",
~r~ ~= ". .......,. -\ :::;. : ~..;-"; ,.,. ,1$. ··i ' ., ; ..I~. '. .., -" ' . ... ...
~~~~----~--.---:----:-1 Imitation copy action of watch teacher or trainer
another; observe and repeat action, and replicate process or activity
2 reproduce activity carry out task from from instruction or
Manipulation written or verbal
memory instruction
3 Precision perform a task or activity reliably, execute skill
with expertise and to independent of high quality without help assistance or
instruction; able to demonstrate an activity to other learners
-~~ - ..~ .~~ "._._ .1.
copy, follow, replicate, repeat, adhere
re-create, build, perform, execute, implement
demonstrate, complete, show, perfect, calibrate, control,
22
Bengkel Penilaian Outcome Based Education
4 Articulation adapt and relate and combine construct, solve, integrate expertise associated activities to combine, coordinate, to satisfy a non- develop methods to integrate, adapt, standard objective meet varying, novel develop, formulate,
req u i rements modify, master
5 Naturalization automated, define aim, approach design, specify, unconscious and strategy for use of manage, invent, mastery of activity activities to meet project-manage and related skills strategic need at strategic level
23
open toReceive listen to teacher or ask, listen, focus, Willingness to experience, willing trainer, take interest in attend, take part, partk:ipate in an activity to attend to a stimulus; to hear session or learning discuss, acknowledge, getting and holding the experience, take notes, hear, be open to, attention of students
turn up, make time for retain, follow, learning experience, concentrate, read, do, participate passively feel
2 Respond react and participate actively in react, respond, seek Actively participates; participate actively group discussion, clarification , interpret, demonstrates interest in an object, activity or active participation in clarify, provide other phenomena, seeks or activity, interest in references and pursues this object, activity or phenomena outcomes, enthusiasm examples, contribute ,
for action, question and question, present, cite, probe ideas, suggest become animated or interpretation excited, help team ,
write , perform
24
Bengkel Penilaian Outcome Based Education
3 Value attach values and decide worth and argue, challenge, Value or worth attached express personal relevance of ideas, debate, refute, confront, to an object, actlvlty or phenomena; varies from opinions experiences; accept or justify, persuade, simple acceptance to commit to particular criticise,commitment
stance or action
4 Organise or reconcile internal qualify and quantify build, develop, formulate, Conceptualize conflicts; develop personal views, state defend, modify, relate, values value system personal position and prioritise, reconcile, Compare and contrast, reasons, state beliefs contrast, arrange, and resolve conflict to build a consistent value compare system, emphasis on comparing and synthesizing values
5 Internalize or adopt belief system self-reliant; behave act, display, influence, characterise and philosophy consistently with solve, practice, values personal value set Adopt a vlue system for a length of time that contributes to a particular ' lifestyle' (I.e direct 25behavior)
Things to decide in assessment plan
• Getting the level right
• Types of assessment method to assess learning outcome
• How many assessment
• Using formative assignment
• Scheduling the assessment across the curriculum
• How can plagiarism be reduced
• Building in flexibility 26
More example of assessment plan P9 14/52 HEacademy UK
Bengkel Penilaian Outcome Based Education
Any more Quality Assurance requirements that we missed?
3.2 Assessment Methods
3.2.1 Benchmarked Standards
• Assessment must be summative and formative.
• A variety of methods and tools must be used appropriately to assess the learning
outcomes and competendes.
• There must be mechanisms to ensure the validity, reliability, consistency,
currency and fairness of the assessment methods.
Formative and Summative Assessment. What are they really?
Are assessment and evaluation the same?
Continuous Assessment = Formative assessment?
Summative Assessment =Final Examination?
Bengkel Penilaian Outcome Based Education
Is it another TEST?
CCSSL definition of Formative assessment is
• a process use by teachers and students Not to eva luate or grading
On-going and• During instruction not after class
• That provide feedback Where I am now, where am I going and how to get there
Using data • To adjust ongoing teaching and learning gathered
• To improved students achievement of Lea rn ing is achieved after
intended learning outcome action taken
Reflection
CURRENT PRACTICE
.-r-' I-
Test I FinalCWgp CW ~ Examinterview report
MULTI POINT SUMMATNE + SUMMATIVE (Developing Effective Assessment In Higher Education, Sue Bloxham and Pete Boyd, Open University Press 2007)
Bengkel Penilaian Outcome Based Education
Marking scheme / criteria - how well they are align with LO?
Marker's skills?
What about giving feedback?
Marking Practice - Skills _
C03 - Ability to Conduct laboratory tests for determination of soil index and compaction.
manipulation observation
32 e.g. ASSESSING BY DIRECT OBSERVATION - have you got the skill to assess
Bengkel Penilaian Outcome Based Education
l\.1anipulation: 5 - CXJtstarding Shows higl.:bility il m~ulating laboratory or demorstration tools! eq.Jipment! measuring ~paratus e:;pedally on settilg up Q(l>L.QllffuJJ:P.;j experimental! demonstration procedLres. Also shows hg, ability in ~lating materials accorcr.g to av3ability, needs and skills
4 - Excellent Shows ability to ~late welilaxlratory or dernorstration tools! eq..Jprnent! rreasLrilg apparatus especially on settilg up ~..w,fu)Ji:lg experimental! dernorstratioll procedLres . Able to ~late well materials accordi1g to avail.:bility, needs 31d skills.
3 - Gocd Able to ~Iate laboratory or dernoost:ration tods! eqjpment ! rreBSLri'lg apparatus espeaaly on settilg up and perforrTi'lg experimental! demonstration procedres. Able to manipulate materials accordilg to avatability, needs and skills .
2 - Satisfactory AverC¥je ma~ulatio n of laboratory or dernorstration tods! eq..ipment ! rneastrng ~paratus especially on setting up m..w....f.QwjJg experimental! demonstration proceru-es . Average rT1BIi:lulation of materials according to availability, needs and skills.
1 - Needs help DiffICult to ~late laboratory or demorstration tools! eq.Jiprnent! rneasumg ~atus especially on setting up and perforrTi'lg experimental! demonstration procedres. Difficult to rnaripulate materials accordilq to availability, needs and skills.
Observation 5 - CXJtstandlng Shows high .:bility n observng payilg more attention n irrportant procerues, .:DIe to spot changes or LnJSUBI occurrence,ibehavior. Record ·observation correctly and does extra notation on what has been observed.
4 - Excellent Shows ability in observing payilg attention on important procedures, able to spot changes or unusual occurrence/behavior . Record observation and make small notation on what has been observed.
3 - Gocd Shows averC¥je ability il observing paying attention on important proceciJres, averC¥je cblity to spot changes' or l.J!"U>UBI occurrence/behavior . Do minimal record on observation .
2 - Satisfactory Shows little .:bility in observng i~rtant procedures . Do mimal record on observation .
1 - Needs help Does not pay attention on important procerues neither make any record on i,/\,O;Jl; !.Y~
NORM -REFERENCE
Aims to discriminate between students and is designed so that student performance is distributed across a range, so that those who do better on the assessment task receive higher grades than those who do less well
10% - Grade A
30% - Grade B
33
CRITERION -REFERENCE
Students being judge based on the quality of their work on set criteria rather than the performance of other members of their particular cohort.
aBE
Desirable nowadays
Give protection to markers to justify their judgment
Implication ??
Refer to Orthogonal Assessment - UK HE Academy Example
Bengkel Penilaian Outcome Based Education
CGPA? What does it tells about students achievement - case study
Should affective and psychomotor be report using numbers?
What will the report card look like in future?
How will it be used.
Establishing Developmental Framework Critical Thinking
Observation Interpretation Judgment Planning
Fouv\'clClt~Ov\'
KV\,owLecl g e f
'cleV\,t~ftJ The
pvobLeVlA f
expLove IV\,tey-pvetClhoV\,s § GoV\,V\,ec,hov\,s
fpv~ov~hze
A LtevVl.,Clhves r.
ev\'vLs~ov\'
stvClteg ~c, IV\,V\,ovCl hoV\,
GOv\'fuseclPerformance FClC,t
Patterns -g,~Cl, secl
~ -- JUVlA-pev
pe y-petuCl L PVCl g VlAClt~C, pevfOVVlAevAv\'ClLtJzev
F~v\'clev 1 D~st~vcguLsV1
Pr~ov~t~zeveLevl1 V\,t § ReLl1te
~ssues ~vveLevl1 V\,t l1SSUIIVl:pt~Ov\'S §
l1 v\'cl Iv\'fovVVtl1tLov\' b~l1ses
~V\,fOVVVtl1t~ov\'
Rel1cl AV\,l1 LtJze Just~ftJ pvos § cov\,scOv\'fl~ct~v\'g
l1ssuVVtptLoV\,sOP~v\'~ov\'s
Interventions
Steps in Critical
Thinking
I stvClteg~c,- Re-V~S~O~V
ArHwLl1te v~s~ov\'
Re~V\,te¥'flvet
~V\,fOVVVtl1t~ov\'
Step 1 Step 2 Step 3 Step 4
Steps for Better Thinking Performance Patterns, http://www.wolcottlynch.cOffi
Bengkel Penilaian Outcome Based Education
Where will the Program Learning Outcomes be infused across the Curriculum?
Course #1
Course #2
...
Course #N
Industrial Training
Capstone Course
Communicate orally and in writing
Introduce
Emphasize
Use
Use
Assess
Analyze, interpret and evaluate results
Introduce
Emphasize
Assess
Assess
Listen to diverse populations
Introduce
Emphasize
Emphasizel Use
Use
Assess
Demonstrate teaching effectiveness
Introduce
Emphasize Use
Assess
Assess g
7
Table 2 - The changing focus ofa programme as it moves through the different levels of study
Level 1 2 3 4
Subject Consolidation Development Authority Mastery
Skills Leaming Key Independence Research
Intellectual capacities
Knowledge Understanding Manipulation
Application Analysis Synthesis
Evaluation Judgment Reasoning
Formulation Conceptualization Reflection
Ref: UK HE Academy 38
Bengkel Penilaian Outcome Based Education
Programme Learning Development Plan For Laboratory Work
Level Aims Materials Method Answer
Demonstration 0 Given Given Given Given
Exercise 1 Given Given Given Open
Structured enquiry 2 Given Given pert or whole
Open or part given
Given
Open Enquiry 3 Given Open Open Open
Project 4 Open? Open Open Open
As one moves up the levels one moves further towards the development of active deep learning
39
Programme Learning Development Plan For Laboratory Work
Demonstration
Usually done to demonstrate theoretical principles. Demonstrator is usually a lecturer or postgraduate student
Exercise
Tightly structures experiments designed to yield well-known results. Students learn to follow precise instructions and in so doing learn specific techniques of observation and manipulation. Careful reading of instruction c,an often reveal the answers required
Struc;tured Enquiry
Lightly structured experiments which require students to select materials and to develop procedures. Students develop problemsolving and interpretative skills as well as manual and observation skills 40
Bengke/ Peni/aian Outcome Based Education
Programme Learning Development Plan For Laboratory Work
Open-ended Enquiry
Students identify a problem, formulate the problem clearly, hoose and design experimental procedures, interpret results and consider their implications. The constraints on the student may be time and the range of equipment and materials available. Open-ended enquiries used in miniaturized form the skills of the research scientist. They can be useful for a preliminary for project work .. Students develop problem- solving and interpretative skills as well as manual and observation skills
41
Programme Learning Development Plan For Laboratory Work
Projects
Based on long experiments of a series of experiments or field studies. The project may be selected by a student, offered by a supervisor, or by local industry or the community. The end products may be a dissertation, design plan, model, computer programme or a simulation. They enable students to explore a field deeply, they develop initiative and resourcefulness, they may stimulate a student's intellectual curiosity and they also develop project and time management skills. Guidelines for project are essentials
42
Bengkel Penilaian Outcome Based Education
LO - Soft Skill: Communication
Year 1 : can communicate effectively in a format appropriate to the discipline(s) and report practical procedures in a clear and concise manner
Year 2 : can communicate effectively in a manner appropriate to the discipline(s) and report practical procedures in a clear and concise manner in a variety of formats
Year 3 (honours) : can engage effectively in debate in a professional manner and produce detailed and coherent project report
Master's : can engage confidently in academic and professional communication with others, reporting on action, autonomously and competently
43
8engkel Penilaian Outcome Based Education
DEPARTMENT OF CIVIL ENGINEERING, FACULTY OF ENGINEERING, UNIVERSITI MALAYA
EVALUATION OF COURSE OUTCOME ACHIEVEMENT SEMESTER 1 SESSION 2009/2010
Course Code KAEA2132 Course Title Soil Mechanics I Course Type Departmental Core Course
1. Course Learning Outcome Assessment Plan :
I :
COURSE LEARNING OUTCOMES (CO) METHOD OF DELIVERY
i
METHOD OF ASSESSMENT
I
Examination question
addressing the CO
C01 : Ability to define soil as in engineering context and relate problems associated with the definition with local soil condition
Lectures and coursework
I Examination and short report
Q1
C02: Ability to identify and differentiate the different types of soil and their properties and classify soil using British and I or Unified Soil Classification System.
Lectures and laboratory work
Short test and laboratory work/report
none
C03: Ability to conduct laboratory tests for determination of soil index and soil compaction.
Demonstration and laboratory work.
Direct observation on laboratory work/report
none
C04: Ability to solve calculation problem using mechanics involving physical properties, compaction, seepage and effective stress.
Lecture and tutorials
Examination & graded tutorial.
Q2, Q3, Q4
COS: Ability to show the use of soil mechanics concepts in engineering works
Lecture, active session and coursework.
Short report & group interview
Q1
Formative Assessment (55%)
Assessment method 1 CO addressed Weightage Short Test (soil Classification) CO2 10% Coursework report & interview C01, COS 20% Laboratory (MT1 & MT2) Direct observation and lab report)
C02, C03 I
20%
Graded Tutorial (Flownet) C04 5%
Summative Assessment (45%) (Students to answer 3 ques~ions out of 4 questions given) Assessment method CO addressed Weightage Soil definition and related problem C01 15% Calculation and solving problem related to Compaction Permeability & seepage Effective stress
C04 15% 15% 15%
Bengkel Penilaian Outcome Based Education
2. Course Learning Outcome Achievement (CO Achievement)
100
90
- 80
--0 ~ 70
I: cu 60E cu 50>.!!! .I: 40 (.)
'" 300 0 20
10
0
---'8-8.8--------------- ---,;;--81.2
C01 C02 C03 C04 C05
Cou rse Outcome
(a) Indirect measurement via student perception through end of semester survey (* refer to attachment 1: end of semester student survey on Co- average)
100
90
80
- 70 ~ ~-c Q) 60 E Q)
> 50 .~ .s::. u 40« 0 0 30
20
10
0 C01 C02 C03 C04 C05
Course Outcome
(b) Direct measurement via course assessment methods (* refer to attachment 2: CO achievement - PI)
Note: PT= 80% students achieved more than 50% of each CO
Figure 1 : Course Learning Outcome achievement measured through indirect and direct measurements
Bengkel Penilaian Outcome Based Education
3. Contribution of Course Learning Outcome (CO) to Programme Outcome (PO) :
I
OJ C))
~ ~
~ :J
~ iii :J
o c: o o :3 C))
IOJ Q;) '(/I (* from direct measurement- course assessment) 'c)) 'Ct n, ,0. c: g g':J
COURSE LEARNING OUTCOMES (CO) PROGRAMME LEARNING OUTCOMES (PO)
ASSESSMENT I
P01 P02 P03 P04 POS POG P07 POB POg P010
C01: Define soil as in engineering context and relate problems associated with the definition with local soil condition
72.2 Examination and I
short report
C02: Identify and differentiate the different types of soil and their properties and classify soil using British and / or Unified Soil Classification System.
88.9 88.9 , Short test and I
laboratory work/report
C03: Conduct laboratory tests for determination of soil index and compaction .
94.4 94.4 94.4 Laboratory work/report
I
C04: : Solve calculation problem using mechanics involving physical properties, compaction, seepage and effective stress.
22.2 22.2 Examination,
I laboratory report & graded tutorial.
COS: Show the use of soil mechanics concepts in engineering works
83.3 83.3 83.3 Short report and
I
group interview I
Percentage carried to PO 72.2 68.5 83.3 83.3 I
4.Students Performance According to University Grading System
MlSSM" 11 SEMRAI BlLAJIGU CALOW JllDGIKlJ'l' S UBJU 11120
SESI I 200t/2010 SD1I:S'11:R; 1
Koo S UB.JJ:K I DEA2132
BIL CALOW : 18
MD.. 55.47
STDDEV 9.55
8
• 6
~~.~~~.....--------.-------------------
2
J • .
•1. o
A- B-t B- C-t c c- D
Figure 2 Distribution of students performance to university grading system
Table 1 Students performance to the various bands
Failed Moderate Good
Band (0 - 49%) (50 - 69%) (>70%)
[C- and below] [C to 8] [8+ to A]
Nos. Students 3 12 3
Percentage (%) 17 66 17
(* total number of students IS 18)
Bengkel Penilaian Outcome Based Education
5. Instructor's Comment
a. From Figure 1 (a) students believed that they have achieved all fives course learning outcome.
b. However, actual students performance via direct measurement (Figure 1 (b )), shows there are 2 eo's (Le. C01 and C04) not being achieved in accordance to Performance Target (PT : 80% students in class achieved more than 50% in each CO).
c. C04 was found to be the lowest (22.2%). Analysis shows that this is due to large percentage (30-45% from calculation question and 5% from graded tutorial) of C04 being measured through high stake assessment; that is final examination which may be disadvantage in this case.
d. Achievement of C03 being the highest perhaps due to direct observation using rubric being used.
e. C04 does not seems to cause dramatic negative effect on PO. f. Better assessment method such as interview and critical review report in PBL
assessment activity proved to enable in-depth learning. Implementation of PBL and non-traditional assessment method did not impaired achievement of CO.
6. Future Action Plan For CQI :
a. It is important for C04 in Engineering field to achieved its performance target (PT) . The fact that it is well below PT, assessment method for this CO need to be refined. This may be improved either by not confining large percentage of this CO to final examination or more effort need to be given to prepare students to such assessment method.
b. Keeping C04 assessment method, more calculation exercises are to be given as classroom activities in the next semester (change in classroom instruction I delivery method).
c. The overall design of this course is to be refined further for its constructive alignment. Better examination question setting is to be considered.(Le. make 01 - compulsory and answer 2 calculation questions from 02,03 and 04).
d. Assessment rubrics need to be refined from content-based to outcome-based for several assessment methods employed.
e. Assessing laboratory work via direct observation using rubric need to be improved.
7.1nstructor's Detail :
Name : Dr. Aishah binti Abu Bakar Room : L7-10, Block L, Engineering Tower, UM Tel. No. : 03 - 7967 5302 E-mail : [email protected]
Bengkel Penilaian Outcome Based Education
Attachment 1 :
DEPARTMENT OF CIVIL ENGINEERING, FACULTY OF ENGINEERING, UNIVERSITI MALAYA
COURSE OUTCOMES SURVEY FORM SEMESTER 2 SESSION 2007/08
Name
Matrix No.
Su~ect/Code : ____~KA~E=A~2~1=3=2~:~S~0~il~M~e=ch=a=n=ic=s~I~______________________
Please rate your level of achievement of Course Outcomes in the scale from 1 to 5 as described below.
[ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ]
Poor Fair Avera~e Good Excellent
The Course outcomes fior t h· IS sub·IJect are:
No. Course Outcomes (CO) Achievement
1 2 3 4 5
1 Ability to define soil as in engineering context and relate problems associated with the definition with local soil condition
2
Ability to identify and differentiate the different types of soil and their properties and classify soil using British and / or Unified Soil Classification System.
3 Ability to conduct laboratory tests for determination of soil index and soil compaction.
4 Ability to solve calculation problem usmg mechanics involving physical properties, compaction, seepage and effective stress.
5 Ability to show the use of soil mechanics concepts in engineering works
If your rating for any of the outcomes is less than 3, please give comments and suggestions how it can be improved.
Thank you for your cooperation.
Bengke/ Peni/aian Outcome Based Education
Attachment 1 - Student Perception on CO achievement Result
C01 Define soil as in engineering context and relate problems associated with local soil condition
O.OX
S1.3X
1S.Sx
O.OX
O.Ox
O.OX 20.0X 40.0x SO.OX SO.OX
C02 Identify and differentiate the different types of soil and their properties and classify soil using British and I or Unified Soil Classification System
I
100.0X
SS.SX
' S.3X
O.OX
O.OX
O.OX 20.0X 40.0X SO.OX SO.OX
C03 Conduct laboratory tests for determination of soil index and compaction
.S.3X
SS.Sx
25.0X
O.OX
I
O.OX 20.0X 40.0x SO.OX SO.OX
Bengkel Penilaian Outcome Based Education
C04 ISol..,e calculation problem using mechanics in..,ol..,ing ph~sical properties. compaction. seepage
I land effecti..,e stress
0 O.OX [1] f--
0 O.OX [2] _S.3X
[3] SS.SX r--4 25.0X
[4] r--
11 68.8x 25.0x .-1511 S.3X O.Ox
,-
; ~
O.OX i--6 ••_.",..._ 3.81 O.Ox 20.0x 40.0x SO.OX SO.Ox r--
Answer rate 16
f---o lS
C05 Show the use of soil mechanics concepts in engineering works
0 O.OX [1] r- r--[2) 31.3x0 O.OX [3)
r- r--4 25.0X 43.S:.-: r- r---7 43_8x [4] 25.0 -:: r- r---
[5]5 31.3x O.OX r-I-
f-f-6 .._.",...... 4.06 O.Ox
f-f-
Answer rate 16 O.OX 10.0x 20.0X lS
30.0X 40.0x 50_OX f- r--I-r-
I j I I I 1 j
O u.rall F indino o f C O (S OIL --............. ""NIC S 1 1
4.30 4.19
4.20
4.10 40S
4.00
3.90 3.81 3.81 3.81
3.S0 .'
3.70
3.60
Bengkel Penilaian Outcome Based Education
Attachment 2 : CO Achievement
OJ CIl
~ ~ ~ ::J
~ iii' ::J
oc:
:3 CIl
OJ til en CIl Q
rn Q c: ~ g. ::J
SOIL MECHANICS I-SEMESTER 1 I SESI 2009·2010
f _. T T -
~ ~T -r---,' t ~.=r -r· -[ t -+ -f r-r-~ , - -I I- .
10 10 10 10 10 5 15 15 15 15 No Matrie, No Labor atory/20 Cont assmnt 135 EHam/45 Grand
I MTI MT2 ".....,.t Tl ,Olin CW'R Tut I'll"".' QI Q2 Q3 Q4 S'll~t.t Total
1 KEA06003 6.3 0.0 8.3 14.5 5.0 9.0 0.0 23.5 5.0 1.5 0.0 8.5 36.3
2 KEA07004 6.3 5.0 11.3 7.0 7.0 9.0 4.0 32.0 12.0 8.5 7.0 27.5 70.8
3 KEA08000 6.3 5.0 lU 5.0 ' 8.0 4.5 29.5 1.0 6.0 0.0 7.0 47.8
4 KEAOBOOO 6.3 5.0 11.3 5.0 8.0 7.5 5.0 30.5 1.0 10.0 0.0 11.0 52.8
5 KEAOBOOO 6.3 5.0 lU 5.0 8.0 7.5 4.5 30.0 1.0 5.5 3.5 10.0 51.3 -
6 KEA08000 6.3 5.0 lU 6.0 8.0 7.5 4.0 30.5 10.5 1.0 1.5 13.0 54.8
7 KEA08001 6.8 7.5 14.3 9.0 8.0' 7 .5 5_0 34.5 4.5 8.0 14.0 28.5 75.3
:3 KEAO:3001 6.8 7.5 14.3 7.5 8.0 7.0 4.0 31.5 2.0 3.0 8.0 13.0 58.8
9 KEA08001 6.8 7.5 14.3 8.0 6.0 9.0 4.0 32.0 4.5 3.5 1.5 9_5 55.8
10 KEA08002 6.8 7.5 14.3 6.0 5.0 9.0 4.5 29.5 12.0 8.5 7.0 27.5 71.3
11 KEA013002 6.8 7.5 14.3 5.0 8.0 7.0 4.5 29.5 3.0 6.5 7.0 16.5 60.3
12 KEA08003 6.8 7.5 14.3 5.5 5.0 6.0 5.0 21.5 6.0 7.5 2.0 15.5 56.3
13 KEA08003 7.5 4.0 11.5 5.5 6.0 3.5 4.0 25.0 6.0 7.0 5.0 18.0 54.5
14 KEA08003 7.5 4.0 11.5 4.0 6.0 3.5 5.0 23.5 5.5 6.0 9.5 21.0 56.0
15 KEA08003 7.5 4.0 11.5 5.5 6.0 3.5 4.5 24.5 5.0 5.0 4.0 14.0 50.0
16 KES06002 7.5 4.0 11.5 7.5 8.0 4.0 31.5 2.0 2.0 3.0 7 .• 50.0
17 KES08002 7.5 4.0 11.5 0.0 5.0 6.0 5.0 21.0 7.0 6.5 5.0 18.5 51.0
18 KES08002 7.5 4.0 11.5 2.5 5.0 6.0 3.0 21.5 8.5 2.0 1.5 12.0 45.0
74.5 52.5 69.0 78.0 55.4
CO Achie...",ment : %of ~tudents in class that achie... ed more than 50% of a particular CO
PT =CO Achie... ement : 80 % of students: in class that achie... ed more than 50% of a particular CO
I . t + ,. j I
jI !- I
- .. --+- - -4. CW'lnt MTI MTl Tut CW'lnt J~
.t Q2 r -
CW'R MT2 I MT2 CW'Rl iQl , Tl Q314
Indi ...idual Attainment of CO
COl C01(:,q CO2 C02(%J C03 C03(:'-;) C04 C04(%) C05 C06(%)
0.5 54.3 0.4 35.8 0.3 31.3 0.0 4.3 0.7 70.0
0.8 80.0 0.6 60.8 0.6 56.3 0.6 55.7 0.8 80.0 * 0.8 75.0 0.5 54.2 0.6 56.3 0.2 23.0 0.8 75.0
0.8 17.5 0.5 54.2 0.6 56.3 0.3 32.0 0.8 17.5
0.8 17.5 0.5 54.2 0.6 56.3 0.3 29.0 0.8 17.5
0.7 74.3 0.6 57.5 0.6 56.3 0.1 14.3 0.8 17.5
0.8 17.5 0.8 17.5 0.7 71.3 0.6 63.0 0.8 17.5 P
0.8 75.0 0.7 12.5 0.7 71.3 0 .3 34.0 0.8 75.0
0.8 75.0 0.7 74.2 0.7 71.3 0.3 27.0 0.8 75.0
0.7 74.3 0.7 67.5 0.7 71.3 0.6 57.1 0.7 70.0 P
0.8 75.0 0.6 64.2 0.7 71.3 0.4 42.0 0.8 75.0
0.5 48.6 0.7 65.8 0.7 71.3 0.4 41.4 0.6 55.0
0.4 44.3 0.6 60.0 0.6 57.5 0.5 45.7 0.5 47.5
0.4 42.9 0.5 51.7 0.6 57.5 0.6 58.6 0.5 47.5
0.5 47.5 0.6 56.7 0.6 57.5 0.4 37.0 0.5 47.5
0.5 48.6 0.6 63.3 0.6 57.5 0.3 25.7 0.8 75.0
0.5 51.4 0.4 38.3 0.6 57.5 0.5 47.1 0.6 55.0 e 0.6 55.7 0.5 46.7 0.6 57.5 0.2 18.6 0.6 55.0 P
64.1 58.6 60.3 36.4 67.4
72.2 88.9 94.4 22.2 83.3
NA A A NA A - - - - - - - ---- -
8
Orthogonal assessment This is probably the most radical change in the philosophy of assessment practice that has come as a result of the move to an outcomes model of programme design and assessment (such as described in the UK.SPEC).
Traditional assessment practice involves a number of assignments which togeti1er produce an average mark or grade for the module. Consider the assessment strategy as described in the table below. For the purposes of this analysis, let us assume that there are two assessments, each having four criteria of equal weight (25%). Cmn represents the nth criterion of assignment m. We will assume also that the assessors are using a percentage marking scheme and averaging numerically. The pass mark is 40%. The same argument v%uld apply usi'ng grades, in which case an algorithm would be used for averaging.
Assessment 1 Assessment 2 Criteria used for judgments
C11 C21
C12 C22 C13 C23 C14 C24
A student receives marks as shown in the next table:
Assessment 1 Assessment 2 Average mark (%)
Criteria used 12 9 for jUdgments 12 11
15 10 8 7
Mark for assignment (%)
47 37 42
The student's average mark for the module is 42%. They have failed the second component (37%), but the rules will allow compensation for marks of 35% or above in one component. Therefore compensation will be applied to the second assignment, and the student passes the module.
However, with an outcomes model, the process is different. Consider the model below. In this outcomes based model, the learning outcomes for the module have been established in the module specification template (assume there are four). These are used consistently across the assignments. That is, the judgments for each assignment are made against the
3 Grade drift is a term used to describe the tendency for a marker to unconsciously revise their judgment of performance. If the first assignment marked is of a high standard , then those that follow will suffer by comparison. Alternatively , if the first few scripts are of a lower quality and receive marks or grades, and then a high qual1ity script is encountered , those that follow will receive 100ver grades or marks even if they are of the same quality as the first few. Explicit criteria tend to 'tie down' judgments and make them more consistent
Section 2, page 16 Assessment of Leaming Outcomes Engineering Subject Centre
Bengkel Penilaian Outcome Based Education
(relevant) learning outcomes, using the cnteria for each learning outcome. This analysis uses the same marks matrix as in the analysis above, but this need not be the case.
The marks are now averaged across the learning outcomes (not the assignments) to give an average mark (or grade) for each learning outcome.
Assessment 1
Assessment 2
Average mark for each outcome
]%) L01 12 9 42 L02 12 11 46 L03 15 10 50 L04 8 7 30 Averaqe mark (%) 42
The implications for the student are very different in this case. The average mark for the module is still 42%. However, the student has 'failed' on learning outcome 4. That is, they have not demonstrated that they have attained this outcome. There is no compensation for this; students must demonstrate each and every outcome, so this student will have failed the module. Any attempt at recovery through a retaken assignment needs only to forus on learning outcome 4.
Note: 1. In effect, the student has been given two attempts to demonstrate L04 and has failed
on both occasions. 2. Even if the average mark is a fail mark for a learning outcome (30% as in L04), if any
attempt, in any assignment, results in a pass mark (10 or greater in this example), then the student has, at some time during the module, demonstrated this learning outcome, and should IpaSS. For example, marks of 10 and 5 would still result in an average mark of 30% for this learning outcome. This would traditionally be seen as a fail mark, however, the student has demonstrated the outcome in the first assignment, so the module is 'passed'.
Implications for recording and decision -making For the assessor
1. For each assessment, judgments are made (and marks are awarded or grades are assigned) against the learning outcomes being assessed (using the relevant criteria for the LOs).
2. Orthogonal assessment removes any decision-making in regard to compensation, as it is not necessary to compensate for individual components of assessment.
3. There can be no compensation for a 'failed' learning outcome. 4 . It is possible to 'pass' each assignment, but to fail the module. For example, using the
grid above, the student has achieved 47% for assignment 1 (pass). If they achieved 15 for L01 in assignment 2, then they would achieve 43% for this assignment (pass) and an average of 45% for the module (pass). However, they still have not achieved a pass for L04 on either occasion. This is what matters, so they will fail the module.
5. Feedback to the student on each assignment will need to focus on their attainment against the Los as well as the assignment itself. The student will need to be advised that, even though they may have achieved a 'pass' mark for the assignment, they have failed to demonstrate a learning outcome, and they must demonstrate this LO in the remaining assignment
Bengkel Penilaian Outcome Based Education
Grade Descriptors for final honours undergraduate
OJ Cb
~ [ ~ ::I
~ iii' ::I
o
i c:
~ g: Q.
~
80-100 AA
70-79 A
60-69 B
50-59 C
40-49 D
35-39 F
30-34 F
0-29 F
Students meet all the requirements for an a grade. There is evidence of exceptional scholarship, including critical evaluation and synthesis of issues and information that generates originality and challenges existing approaches. Accurate and detailed use of range of evidence. Comprehensive knowledge and understanding of theories, principles and concepts. Student has met learning outcomes (LOs) of the assessment with evidence of comprehensive and up-to-date knowledge and understanding of concepts and theories and their relationship. the work shows a detailed appreciation of how aspects of the subject are uncertain, contradictory or limited. The work adopts a well-sustained critical approach using a breadth of evidence, reasoning and reflection. Work shows evidence of a mature and independent approach to problem solving, the student can create appropriate hypothesis and select, justify and use imaginative and innovative approaches in their investigations Student has met the LOs of the assessment with evidence of comprehensive and up-to-date knowledge and understanding of concepts and theories and their interrelationship with an awareness of how aspect s of the subject are uncertain, and contradictory or limited. The work adopts a critical approach using breadth of evidence, reasoning and reflection Work shows evidence that the student can confidently and autonomously in the identification and definition of complex problems and select, justify and use approaches aimed at their resolution. Student has met the LOs of the assessment with evidence of detailed knowledge and understanding of key concepts and theories, including an awareness of the provisional nature of knowledge. The work shows evidence of a general critical approach using individual judgment and reflection, although there is some limitation in the ability to conceptualize and / or apply theory Work shows evidence that student can act without guidance in the identification of complex problems and can apply knowledge and skills to their resolution Student has met the LOs of the assessment, with evidence of knowledge and understanding of key concepts and theories including basic recognition of the complexity of the subject. The work is for the most part descriptive rather than based on argument and logical reasoning. Work shows evidence that the student can apply appropriate learning accurately to complex problems and / or practical contexts. Student has not met all the LOs of the assessment, with only basic knowledge of key concepts and theories and weaknesses in understanding. there is little or no recognition in the complexity of the subject. Work is largely descriptive with some unsubstantiated assertion. Analysis is minimal or contradictory. Unable to always apply learning accurately to complex problems and / or practical contexts. For profesional courses any work which contains evidence of, or reference to, unsafe or dangerous practice should be deemed to fail. Student has not met all the LOs, with inadequate knowledge or understanding of key concepts and theories. there is no recognition of complexity of the subject. The work is descriptive and uncritical, with unsubstantiated assertion and a lack of analysis. Insufficient understanding of given tools/methods to apply learning accurately or safely to complex problems and / or practical contexts. Students has failed the majority of the LOs.
~ source: extract from grade descriptors for levels 0-5, University of Cumbriag: