Ensuring the Alignment of Assessment

with

Learning Outcomes

Working Smarter not Harder

Dr. Michele Pinnock

From which angle are we viewing the outcome?

• Are your learning outcomes attainable?• What are we valuing process or product?• Are you compromising the validity and

reliability of the assessment • Is your expert knowledge; prior learning

experience [when I was in College] syndrome; your knowledge of the standards and expectations of the real world all clouding your role as assessor?

CONCERNS

• Students complain that they are being over-assessed

• Students complain about not receiving meaningful feedback

• Teachers complain about the ‘ marking’; we continue to over-assess and overworked ourselves

Valuing vs Crediting

• What are you valuing and what is being credited by your course?

• Are we valuing high order thinking?• We say we value completing assignments of

time but do we credit such?

How valid are our assessments?

• Are we testing what we say we would test?• Is our test mirroring the emphasis of our instruction?• Are we asking enough questions to make informed

judgments about student performance?• Are we able to accurately predict future

performance of our students accurately?

What are we prepared to do about this situation?

The Message from the Assessment

• Assessment sends clear messages to students regarding what is valued in a course.

• Students who perceive that the assessment will test memorization are more likely to adopt surface approach to learning (Scouller, 1998)

• Ensure that the message from assessment is in accordance with what is valued by the lecturer and the course.

Owning Intended Learning Outcomes

• We must clearly communicate to our students the intended learning outcomes so they can own them and take responsibility for achieving them

• Students are rarely concerned about learning outcomes they are interested in what is being valued by the assessment.

• Assessment defines what students regard as important, how they spend their time and how they come to see themselves as students and then as graduates.

• Students take their cues from what is assessed rather than from what lecturers assert is important

George Brown et al ,

How do we get students to learn what we intend for them to learn?

• Constructive Alignment is the answer.

?

Constructive Alignment

• This is a conscious effort to provide the learner with a clearly specified goal, a well designed learning activity or activities that are appropriate for the task, and well designed assessment criteria for giving feedback to the learner.

Aligning learning outcomes, learning and teaching activities and the assessment.

Adapted from Biggs(1999) p 27

Constructive Alignment• Encourages clarity in the design of the

curriculum, • Ensures that both students and teachers are

cognizant of what is being valued and the weighting being ascribed to such

• Offers transparency in the links between learning and assessment.

• Encourages institutions to strive towards quality assurance and enhancement.

Think about Assessment at the beginning of the Learning Process

According to Biggs(1995) in aligned courses, • objectives are usually clear, functioning at the

high order level; • teaching methods usually elicit from students

those learning activities that are likely to achieve those objectives; and

• our assessment confirms that students are in fact learning what our objectives say they should learn (p.11)

Unaligned Course

Teacher Intentions

Student

Activity

Exam

Assessment

e.g.- explain- relate- prove- apply

e.g.- memorize- describe

"Dealing with the test"

Aligned Course

Teacher Intentions

Student

Activity

Exam

Assessmente.g.- explain- relate- prove- apply

e.g.- explain- relate- prove- apply

Assessment influences Learning

• Assessment methods and requirements probably have a greater influence on how and what students learn than any other single factor (Boud,1995).

• How we go about assessing students can make significant impact on how well they achieve in their studies

• Poor assessment design will lead students to ascribe to behaviours that are counter-productive to learning

Steps Towards Constructive Alignment

Consider assessment at the beginning of the course • Teachers need to be clear on attainment targets

for their learners• Understand, explore the assessment possibilities

for measuring these targets and then selecting the most relevant

• Understand the process that will cause students to attain these targets

Designing Intended Learning Outcomes (ILO)

• Bloom’s Taxonomy• Structure of Observed Learning Outcomes

(SOLO) Taxonomy helps to map levels of understanding that can be built into the intended learning outcomes and to create the assessment rubrics.

Valuing competences

Have the student do something,and then measure product and/or process

'SOLO' = Structure of the Observed Learning Outcome

Objective !

To learn to: analyze systems for... explain cause/effects... prove properties of... compare methods of... ...

[ Competence := knowledge + capacity to act upon it ]

'SOLO'

SOLO 1 no understanding irrelevant information misses point

SOLO 2

SOLO 3

SOLO 4

SOLO 5

"pre-structural"

to identify to do procedure to recite

"uni-structural"

to classify to combine to enumerate

"multi-structural"

to relate to compare to analyze

"relational"

to generalize to hypothesize to theorize

"extended abstract"

dept

h(q

ualit

ative

leve

ls)

surf

ace

(qua

ntita

tive

leve

ls)

SOLO • to theorize• to generalize• to hypothesize• to predict• to analyze• to relate• to compare• to explain causes• to describe• to combine• to classify• to perform algorithm• to do simple procedure• to define• to identify• to recite

extendedabstract

relational

multi structural

SOLO 5

SOLO 4

SOLO 3

R

R2

R3

R1

R

xx

Graphic Illustration Legend

immediately relevant aspects – given! related or hypothetical – not given! irrellevant or inappropriate student responsexR

R

SOLO 2

uni structural

R

x R'

R''

x

RX

Designing Student Activities• It is imperative that congruence exists between

the – intended learning outcomes; intended learning outcomes; – the learning activities and the learning activities and – the assessment taskthe assessment task

• If we value level 2 & 3 of the SOLO - Surface Understanding will be attained

• Valuing level 4 & 5 will result in Deep Understanding

SOLO (verbs)

"extended abstract"

"relational"

"multi structural" &"uni structural"

to theorize to hypothesize to generalize to critize to predict

to apply theory (to 'near' problems) to reason about (reach conclusion) to explain (cause-effect) to explain (similarities-differences) to explain (strengths-weaknesses)

to describe to structure at collate to combine to classify to perform algorithm

to apply theory (to 'distant' problems) to put-into-perspective to reflect to judge to discuss

SOLO 2+3

SOLO 4

SOLO 5

to analyze to argue to relate to compare to integrate

to enumerate to paraphrase to do simple procedure to define to identify / name to recite

Copied from http://www.cmu.edu/teaching/assessment/basics/alignment.html

TYPE OF LEARNING OBJECTIVE EXAMPLES OF APPROPRIATE ASSESSMENTS

RecallRecognizeIdentify

Objective test items such as fill-in-the-blank, matching, labeling, or multiple-choice questions that require students to:•recall or recognize terms, facts, and concepts

InterpretExemplifyClassifySummarizeInferCompareExplain

Activities such as papers, exams, problem sets, class discussions, or concept maps that require students to:•summarize readings, films, or speeches•compare and contrast two or more theories, events, or processes•classify or categorize cases, elements, or events using established criteria•paraphrase documents or speeches•find or identify examples or illustrations of a concept or principle

ApplyExecuteImplement

Activities such as problem sets, performances, labs, prototyping, or simulations that require students to:•use procedures to solve or complete familiar or unfamiliar tasks•determine which procedure(s) are most appropriate for a given task

TYPE OF LEARNING OBJECTIVE EXAMPLES OF APPROPRIATE ASSESSMENTS

AnalyzeDifferentiateOrganizeAttribute

Activities such as case studies, critiques, labs, papers, projects, debates, or concept maps that require students to:•discriminate or select relevant and irrelevant parts•determine how elements function together•determine bias, values, or underlying intent in presented material

EvaluateCheckCritiqueAssess

Activities such as journals, diaries, critiques, problem sets, product reviews, or studies that require students to:•test, monitor, judge, or critique readings, performances, or products against established criteria or standards

CreateGeneratePlanProduceDesign

Activities such as research projects, musical compositions, performances, essays, business plans, website designs, or set designs that require students to:•make, build, design or generate something new

This table does not list all possible examples of appropriate assessments. 

Authentic Assessment• Authentic assessments require students to be

effective performers with acquired knowledge.• Traditional tests tend to reveal only whether the

student can recognize, recall or "plug in" what was learned most times out of context.

• Authentic assessments present the student with the full array of tasks that mirror the priorities and challenges found in reality.

Assessing Learning in the Information Era

• With this information age – students need to learn how not just to access, but evaluate and use appropriate information to solve real problems/issues

• Let’s get them to make value judgments about the validity and reliability of the information

• Remember that student learning does not just depend on what we teach

• Spend more time and resources on assessment-

"The Learning Pyramid"Average

retention rate

5%

10%

20%

30%

50%

75%

80%

Lecture

Reading

Audiovisual

Demonstration

Discussion group

Practice by doing

Teaching others

[Kilde: NTL Institute for Applied Behavioral Science, Bethel, Maine]

passive

activestudent

level

of

enga

gem

ent

Biggs (2003) Concept Map showing relationships within the Curriculum Design Process.

Concerns

• Incidental learning outcomes need to be identified and if of value incorporated as intended learning outcomes for the next course offering

• Constructive alignment cannot be achieved or maintained in an institutional system that does not allow frequent review and modification by teachers

Putting it all TogetherThere are four stages in a constructively aligned curriculum

(Biggs & Tang, 2007, pp. 54-55).

1. Describe the intended learning outcomes in the form of a verb (learning activity), its object (the content), and specify the context and a standard the students are to attain.

2. Create a learning environment using teaching/learning activities that address that verb and therefore are likely to bring about the intended outcomes.

3. Use assessment tasks that also contain that verb, thus enabling you to judge with the help of rubrics if and how well students’ performances meet the criteria.

4. Transform these judgments into standard grading criteria.

Examine carefully Goals/ Rationale

of your course

Operationalize the goals into

intended learning outcomes

(ILO)

Choose carefully the Assessment

That measures these Outcomes

Choose carefully the forms of teaching

That focus on these outcomes

Do we possess a vision of Evaluation

Unless the purpose is perceived to be significant, the procedures are clearly understood, and the results are perceived to be useful and relevant, the individuals whose performances are being assessed will not do their best and will not facilitate the assessment process.

Learners can resist even high stake test if they are deemed meaningless and unmanageable.

MEANINGFUL ASSESSMENT

• Achieves specific goals or purposes that are significant to all especially students

• Clear procedure, criteria, rubrics that are understood by ALL stakeholders

• Produce results that provide clear directions for improved learning and instruction

Manageable Assessment

Describes assessment that provides useful information on performance given time and other resources.

• Planning and organizing the assessment• Collecting / analyzing assessment data• Recording and communicating the results of

the assessment to stakeholders

Aligning Coursework

• Coursework as grades or part of the learning process?

• Are we engaged in formative assessment?

Proportions of Formative Assessment in Classrooms

No Formative

assessment

Token Formative

Assessment

Moderate Formative

Assessment

Near-Total Formative

Assessment

Class Activities Formative Assessment

Formative Assessment

• Ongoing• Aids learning– Teachers feed information back to students in ways that

enable the student to learn better, or

– Students can engage in a similar, self-reflective process. • Helps students understand the rules of the game • Helps in the continuous monitoring of the quality of

the work as both students and teacher strive towards attaining desired learning outcomes

Formative Assessment

• Aids instruction – using evidence of student’s mastery status to make adjustments to instruction if the evidence suggests these adjustments are warranted

Formative assessment make teachers teach better and learners learn better

Formative assessment represents evidence-based instructional decision making

Steps for Establishing Formative Assessment That Solicits Classroom Climate Shifts

1. Distribute classroom climate guidelines2. Seek trust constantly and nurture it seriously3. Model and reinforce appropriate conduct4. Solicit students’ advice on classroom climate5. Assess students’ relevant affective status

Concerns/ Dissonance

• Formative assessment though it elicits deeper understanding among learners it will not necessarily improve student’s scores on the standardized examinations

How well do teachers manage assessment ?

• Many assessment procedures are labour intensive and time consuming on the part of teachers

• Teachers spend 15 -20 minutes outside of school grading essay assignments.

Swain & Swain (1999)

VALIDITYThe accuracy of a test to test what it is suppose to

test.• Number of items• The types of items• The time allotted for the completion of the

assessment• Quality of task instructions• The weighing of each assessment in relation to the

course objective and the course in its entirety

RELIABILITY

This speaks to the consistency of the assessment in determining student performance.

If assessment is not reliable then decisions made based on such will lead to problems.

Issues associated with some frequently used test formats

GROUP ASSESSMENT• Deciding on group members• Assessing the input of each members process vs. product• Assessing whether all members possess the

knowledge/ skill/ attitude being examined

Formative assessment aids learning by generating feedback information that is of benefit to students and to teachers.

Feedback on performance, in class or on assignments, enables students

• to restructure their understanding /skills and • build more powerful ideas and capabilities.

Conditions necessary for students to benefit from feedback

According to Sadler (1989):

a) possess a concept of the goal/standard or reference level being aimed for

b) compare the actual (or current) level of performance with that goal or standard

c) engage in appropriate action which leads to some closure of the gap

Issues with Feedback

• Students not clear on learning outcomes• Students not sure what standards/

expectations look like• Students not sure about what is actually

necessary to help students close the gap. Example of a comment -‘this essay is not

sufficiently analytical’)

Good feedback practice:

1. Facilitates the development of self-assessment (reflection) in learning. 2. Encourages teacher and peer dialogue around learning. 3. Helps clarify what good performance is (goals, criteria, expected

standards). 4. Provides opportunities to close the gap between current and desired

performance. 5. Delivers high quality information to students about their learning. 6. Encourages positive motivational beliefs and self-esteem . 7. Provides information to teachers that can be used to help shape the

teaching.

FEEDBACK now Feed-forward

• Feedback is the most significant factor in student progress

• For feedback to be effective students must be able to apply the comments they receive to improve their chances of success with the following assessment

• High level of feedback results in high quality student outcomes and so expectations are achieved

Feedback

• Comparing students is not of importance because our objectives/ criterion are written in terms of individual mastery of a course

• Forms of feedback- checklists/ rubrics/ codes/ regular conferencing

feedback

• How can we get students to read the feedback and do something with it, react to it, respond to it, use it as a feed-forward – something that will make the next assignment better.

• We need to design ways of giving feedback much faster, maybe emailing

ASSESSMENT INSTRUMENTS

Checklists Rating Scales Rubrics

• Tools used to measure performance on assessments.

RUBRICS

Rubrics are a scoring scale consisting of

a set of criteria that describe what expectations are

being assessed/evaluated and

descriptions of levels of quality used to evaluate

students work or to guide students to desired

performance levels.

Rubrics

• Communicate expectations and aid instruction• Indicates quality and quantity of student

learning • Should be given with the assessment task • allow students the ability to assess their own

work

Sample of a Maths Rubric4 3 2 1

Demonstrates a thorough understanding of the main concepts.

Demonstrates an understanding of the main concepts.

Demonstrates a partial understanding of the main concepts.

Demonstrates little understanding of the main concepts.

Very capably and independently selects the most efficient algorithm and solutions are accurate.

Performs algorithms accurately and is usually accurate.

With some assistance, performs algorithms with some accuracy present.

Uses and performs simple algorithms with some accuracy present. Assistance is usually required.

Independently applies the steps required with a very high degree of accuracy.

Applies the steps of a problem and is usually accurate.

Some effectiveness is evident when following and applying the steps of a problem.

Often forgets the steps in a problem, some accuracy noted some of the time.

Thorough analysis of the problem with accurate solutions.

Analysis of the problem is evident, considerable accuracy.

Analyzes the problem with some success, accuracy needs to improve.

Very little evidence of analysis. Some educated guesses. Accuracy is weak.

Assessment Instruments List of

Assessment tools

http://course1.winona.edu/shatfield/air/rubrics.htm

http://academic.scranton.edu/department/assessment/ksom/

http://flightline.highline.edu/socc/ToolsResources/Tools/samplerubrics.htm

Written Papers http://www.fordham.edu/halsall/med/rubric.html

Class Participation

Critical thinking

Class Discussion http://www.landmark-project.com/classweb/tools/printable.php?rbrc_id=64683

Journal Entry http://course1.winona.edu/shatfield/air/Journal%20Entry%20Rubric.pdf

Class Debate http://www.winona.edu/AIR/documents/classdebate.pdf

Oral Communication

Problem solving

Presentation Checklist

http://www.sasked.gov.sk.ca/docs/comm20/pg061.pdf

Selecting Assessment Strategies

• Ensure that the strategy does mirror the objectives• Are competencies and skills being adequately

measured?• At the tertiary level we rely heavily on three main

assessment strategies – class presentations, essays and examination. Recently group work has been added to the list

Alternate Assessment• Portfolios where a complexity of meaningful tasks

geared towards enhancing learning are included. Please note that for portfolios assessment is the secondary purpose

• Simulations; • Work experience • Projects – Product? Process? Or both product and

process• Debates• Displays

• Concern – external examiners – need to validate grades and thus need proof of assessment.

Answer • design your assessment framework prior

to the course; videotape; and even invite them to play a passive role while you assess

Designing Assessments that Capture Student Interest and

Promote Learning • Using the Document Camera• Discussion Board• Weblogs• Chatrooms• Making movies/digital stories• Podcasts

Consider all the factors prior to…

• Emailing of assignments- implications- who will print; format etc

• Digital form of assignment – copy and paste etc

• Availability of resources/ materials• Guarding against plagiarism

Enhancing Reliability ..• Frequency and duration of assignments• More assessment tasks lead to increase in reliability

– greater consistency• More and thus shorter assessment items – students

will get more guidance on their performance which often leads to less anxiety HOWEVER In the real world students will have to face problems that are multifaceted and so the assessment would not be accurately measuring their ability to solve such problems

• Sometimes in a bid to ensuring reliability validity is sacrificed

• Using more than one scorer increases reliability• Defining marking scheme

Good Assessment Principles

• Use a range of assessment tasks to ensure balance of coverage and depth

• Validity achieved when assessment items measure the kind of knowledge desired – Relevance

• Emphasis of instruction is in tandem with the emphasis of assessment - Balance

• Assessment tasks are pitched at the levels outlined in the learning outcomes

Evaluation as a Tool for Empowering our Students

• Ensure that all students have an equal opportunity to achieve to the best of their ability– Consider the Learner - Test Anxiety; workload– Assessment of, as and for Learning

Integrating Assessment• Look at the assessment • How do assessment pieces relate to other pieces

over the semester• Are students being asked to do a similar piece of

work in another course – integration within disciplines (departments) and with other disciplines (departments)

• Look at the weightings• Who will be marking – set your schedule• What type of feedback will be given and when

Managing the measures

• Using excel

Online Assessment

• http://quizstar.4teachers.org

• http://quizstar.4teachers.org/instructor/class_mgr_ov.jsp?pl=cm&cl=cm_ov&qsts=1345707814404

Moving Forward…

• Constructive alignment cannot be achieved or maintained f we are not actively engaged in systematic and frequent reviews and modification of course and programme offerings

• We need to ensure that we use a modest number of extraordinarily important curricular aims

• We must be committed to analyzing and compiling reports for each outcome / course

Conclusion• Let’s stop marking • Let’s move towards ensuring that there is a

seamless transition between assessment and learning

• Ensure that assessment is a meaningfully integrated into the learning process

• Let’s ensure that opportunities for meaningful and timely feedback is intertwined with instruction

Let’s Work Smarter not Harder

• I thank you

References• Biggs, J. (1999). Teaching for Quality Learning at University,

(SRHE and Open University Press, Buckingham)• Biggs, J. (2003). Aligning Teaching and Assessment to

Curriculum Objectives, (Imaginative Curriculum Project, LTSN Generic Centre)

• Biggs, J and Tang C. (2007). Teaching for Quality Learning at University, (McGraw-Hill and Open University Press, Maidenhead)

• Sadler, D. (1989) Formative assessment and the design of instructional systems Instructional Science 18, 119-144.