Planning Formative Assessment Workshop FGSR 2012 · Planning Formative Assessment Workshop 2012 References and Further Reading Some of the resources used today for the “Outcomes”

Planning Formative Assessment Workshop 2012

Planning Formative Assessment Workshop 3:30‐4:30pm, CCIS L2‐200, Jan. 12, 2012

Based on a follow up workshop to a 2008 TLEF Award:

“Formative Assessment Technique Development in Engineering”

Planning Formative Assessment Workshop 2012 Speaker contact info: John A. Nychka (Assoc. Prof., CME); [email protected]; 492‐4796; ECERF 7‐040

This workshop is based on a follow up workshop to an initial University of Alberta Teaching Learning and Enhancement Fund (TLEF) Award: “Formative Assessment Technique Development in Engineering” (2008)

J.A. Nychka, S. Kresta, U. Sundararaj, Q. Liu, F. Forbes, Chemical and Materials Engineering, Faculty of Engineering; $20,000

Excellence in education cannot be achieved or continue without assessment. Assessment is not just grade reporting. Assessment involves the measure of progress to help students and instructors learn how to improve. We will use TLEF funds to: a. hold a classroom assessment techniques workshop by an expert leader in the field; b. assist University of Alberta faculty become formally trained in the techniques of formative assessment; (through workshops like this one!) c. promote culture change from teacher‐centered instruction to learner‐centered instruction; d. give instructors the tools and forum necessary to develop their own formative assessment techniques and incorporate them in course re‐design; e. encourage and facilitate instructors to share their research on formative assessment in scholarly venues (i.e. journals).

AGENDA 3:30 to 3:35pm Welcome and Introduction 3:35 to 3:42pm Part 1: Brief overview of “Learner Outcomes” and “Formative Assessment” 3:42 to 4:25pm Part 2: “Planning Formative Assessment” 4:25 to 4:30pm Feedback and closing remarks Not all the activities in this handout will be covered during the workshop, but working through them with some colleagues (especially those outside of your department or faculty) will be very useful.

IMPORTANT! The workshop is meant to be interactive. In preparation for the workshop, having one particular course in mind (a course you have taught or will teach in the upcoming academic year):

1) Take the online teaching goals inventory (with respect to the chosen course) and bring the results with you to the workshop. The teaching goals inventory can be accessed at: http://fm.iowa.uiowa.edu/fmi/xsl/tgi/data_entry.xsl?‐db=tgi_data&‐lay=Layout01&‐view

2) Bring the course syllabus. 3) Bring an exam from the course. 4) Bring a homework or lab assignment from the course. 5) Identify a topic or unit within the course which you can easily assess, and another which you

have difficulty assessing.


References and Further Reading Some of the resources used today for the “Outcomes” and “Assessment” sessions were modified versions which were originally prepared and supplied by the Centre for Teaching and Learning, and can be found at: http://www.ctl.ualberta.ca/Teaching_Services/TES_Docs/TES_Resources.html Prior TLEF efforts on Formative Assessment:

http://www.ctl.ualberta.ca/Symposiums/LessTeachingMoreLearning.html (Including Sadler keynote podcast, papers, and workshop materials; Angelo keynote podcast, workshops I‐III, and reference materials).

Outcomes Teaching Goals Inventory – online system http://fm.iowa.uiowa.edu/fmi/xsl/tgi/data_entry.xsl?‐db=tgi_data&‐lay=Layout01&‐view Portland State University – “Assessment Step‐by‐Step” http://www.pdx.edu/cae/assessing‐teaching‐learning http://www.pdx.edu/cae/assessment‐step‐step Lincoln Land Community College – “Bloom’s Taxonomy of Measurable Verbs” http://www.llcc.edu/LinkClick.aspx?fileticket=%2F0BA4qlDaAE%3D&tabid=3938 Kansas State University – “How to Write Student Learning Outcomes” (has a good verb listing) http://www.k‐state.edu/assessment/slo/how.htm National Center for Geographic Information and Analysis, University of California Santa Barbara (condensed verb listing) http://www.ncgia.ucsb.edu/education/curricula/giscc/units/format/outcomes.html Prairie South School Division, Moose Jaw, SK – Student Learning Outcomes http://www.prairiesouth.ca/staff/curriculum‐a‐learning/assessment/formative‐assessment‐afl.html American Psychological Association – The Assessment Cyberguide for Learning Goals and Outcomes, 2009: http://www.apa.org/ed/governance/bea/assessment‐cyberguide‐v2.pdf

Formative Assessment/Assessment/Classroom Assessment Thomas A. Angelo and K. Patricia Cross (1993), “Classroom Assessment Techniques: A Handbook for College Teachers”, 2nd Ed., Jossey –Bass, San Francisco, pp. 427.

Carol Boston (2002) “The Concept of Formative Assessment” Practical Assessment, Research & Evaluation, 8(9) http://pareonline.net/getvn.asp?v=8&n=9


Prairie South School Division, Moose Jaw, SK – Classroom Formative Assessment http://www.prairiesouth.ca/staff/curriculum‐a‐learning/assessment/formative‐assessment‐afl.html Valerie J. Shute (2007), “Focus on Formative Feedback” Educational Testing Service Research Report http://www.ets.org/Media/Research/pdf/RR‐07‐11.pdf McGill University Office of Student Teaching: Formative Assessment Rubrics Guide for evaluation of student teaching: http://www.mcgill.ca/files/ost/RUBRICS_GUIDE.pdf Lincoln University ‐ “Avoiding Assessment Pitfalls” http://www.lincoln.edu/assessment/planning/pitfalls.pdf University of Kentucky – Classroom Assessment Techniques (a variety of examples of created CATs) http://www.uky.edu/IRPE/assessment/classroomtech.html “9 Principles of Good Practice for Assessing Student Learning” American Association for Higher Education (2003): http://www.cnu.edu/assessment/pdf/aahe_9_assessment_principles.pdf Peter T. Ewell (2006) “Improving Undergraduate Education through the Assessment of Student Learning (Presentation)” National Center for Higher Education Management Systems: http://www.nchems.org/pubs/detail.php?id=84 University of Illinois, Urbana‐Champagne – “Unit Plan for Assessing and Improving Student Learning in Degree Programs”. Multiple discipline plans are available: http://cte.illinois.edu/outcomes/unit_assess.html


Activities

Planning Formative Assessment


Activity 1: Background Knowledge Probe Circle the answer that most represents your current state of knowledge:

1) Classroom Assessment a) Have never heard of this b) Have heard of it, but don’t really know what it means c) Have some idea what this means, but not too clear d) Have a clear idea of what this means and can explain it

2) Teaching Goals Inventory

a) Have never heard of this b) Have heard of it, but don’t really know what it means c) Have some idea what this means, but not too clear d) Have a clear idea of what this means and can explain it

Activity 2: Focused Listing On the lines below, please list 5‐7 words or short phrases that describe/define what the phrase “Classroom Assessment” means to you.

________________________________ ________________________________ ________________________________ ________________________________ ________________________________ ________________________________

Activity 3: Pro and Con Grid Directions: Considering everything you know about CATs at this point, what do you see as the most important pros (cons), or costs (benefits) of using this approach. List at least 3 important cons (costs) and at 3 pros (benefits) below.

Advantages of CATS Disadvantages of CATS

Adapted fro

P

om Angelo: http:

Planning

Ac

://www.ctl.ualbe

g Format

tivity 4: A

erta.ca/docume

tive Asse

Applicatio

ents/WorkshopI.

essment

ons Card

.pdf

t Workshhop 2012

Angelo

P

o and Cross (199

Planning

Samp

3) pg 43‐44.

g Format

ple Asses

tive Asse

ssment Pr

essment

roject Pla

t Worksh

an

hop 2012

Angelo

Anaylattent

P

o and Cross (199

ze the propotion to determ

Planning

T

3) p 49.

Activ

sed assessmemining answe

g Format

The Asses

vity 5: W

ent plans in thers for questi

tive Asse

ssment P

hat woul

he sample caons 5‐10 from

essment

Planner

ld you do

lculus exampm the above A

t Worksh

o?

ple above, payAssessment P

hop 20

ying close Planner.

12

Angelo: http

5.

P

p://www.ctl.ualb

Planning

berta.ca/docum

g Format

ents/Workshop

tive Asse

I.pdf

essmentt Workshhop 2012


Sample CAT: Customized Content, Form, and Function Outlines University of Kentucky MSE 201 Fall 2006 CAT 11: 7 minutes October 13, 2006

This is a Classroom Assessment Technique, not a quiz!

You are supposed to tell your boss how long to process some flat 1018 steel plate in a carburization furnace. He tells you that the plate will be in a furnace at a temperature of 893K in an atmosphere of carbon monoxide/carbon dioxide (equivalent carbon concentration of 0.8wt%). He would like the treatment to achieve a carbon content of 0.34wt% at a depth of 200 microns. The diffusivity of carbon in steel at the temperature in question is 2.44x10‐11 m2/s. Circle the equation(s) you can use to solve this problem.

A. ⎟⎠⎞

⎜⎝⎛∂∂

−=xCDJ

B. ⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

=⎟⎠⎞

⎜⎝⎛∂∂

2

2

xCD

tC

C. Dtxeff γ≅

D. ⎟⎟⎠

⎞⎜⎜⎝

⎛−=⎟⎟

⎠

⎞⎜⎜⎝

⎛−−

Dtxerf

CCCtxC

os

o

21),(

Using the information you have in the problem statement fill in the grid below:

Nychka: http://www.uky.edu/IRPE/assessment/classroomtech.html

Variable Definition Units Value from problem x

t

Cs

Co

C(x,t)

D γ


Activity 6: CAT Analysis

For the previous sample CAT, the following higher order learning skills were primarily targeted:

1. develop ability to synthesize and integrate information and ideas 2. problem solving skills 3. analytic skills

along with the following teaching goals:

4. improve ability to follow directions, instructions and plans 5. learn to evaluate methods and materials in this subject

Notes: Three of the four equations listed can be used to solve the problem.

The data collected from the CAT is listed below: 16 students handed in the CAT (out of 22 registered students)

5 students circled only 1 equation (100% chose 1 of the 3 correct answers) 7 students circled 2 equations (86% chose 2 of the three correct answers) 4 students circled 3 equations (75% chose 3 of 3 correct answers)

10 students had 4 or less blank spaces out of 21 6 students did not attempt to fill in 11 or more blank spaces out of a total possible 21

Figure 1. a) Number of students as a function of unanswered spaces; b) Mean number of correct answers, Nc, versus the number of spaces left unanswered. Error bars are one standard deviation.

How would you interpret the collected data with respect to the teaching goals? Is the data meaningful? What would you do to change the CAT to make it more effective?


Angelo and Cross (1993) pg 59.


50 CATS by Angelo and Cross (1993) Prepared by D. Moore & K. Cunningham, September 2006

University of Kentucky, Lexington Source: http://www.uky.edu/IRPE/assessment/classroomtech.html

Techniques for Assessing Course‐Related Knowledge & Skills

I. Assessing Prior Knowledge, Recall, and Understanding The CATS in this group are recommended to assess declarative learning, the content of a particular subject. 1. Background Knowledge Probe: short, simple questionnaires prepared by instructors for use at the

beginning of a course or at the start of new units or topics; can serve as a pretest; typically elicits more detailed information than CAT2. .

2. Focused Listing: focuses students’ attention on a single important term, name, or concept from a lesson or class session and directs students to list ideas related to the “focus.”

3. Misconception/Preconception Check: focus is on uncovering prior knowledge or beliefs that hinder or block new learning; can be designed to uncover incorrect or incomplete knowledge, attitudes, or values

4. Empty Outlines: in a limited amount of time students complete an empty or partially completed outline of an in‐class presentation or homework assignment

5. Memory Matrix: students complete a table about course content in which row and column headings are complete but cells are empty

6. Minute Paper: perhaps the most frequently used CAT; students answer 2 questions (What was the most important thing you learned during this class? And What important question remains unanswered?)

7. Muddiest Point: considered my many as the simplest CAT; students respond to 1 question (What was the muddiest point in _________ ?); well suited to large, lower division courses but not to those which emphasize integration, synthesis and evaluation

II. Assessing Skill in analysis and Critical Thinking The CATS in this group focus on analysis—the breaking down of information, questions, or problems to facilitate understanding and problem solving 8. Categorizing Grid: student complete a grid containing 2 or 3 overarching concepts and a variety of

related subordinate elements associated with the larger concepts 9. Defining Features Matrix: students categorize concepts according to presence or absence of

important defining features 10. Pro and Con Grid: students list pros/cons, costs/benefits, advantages/disadvantages of an issue,

question or value of competing claims 11. Content, Form, and Function Outlines: in an outline form, students analyze the “what” (content),

“how” (form), and “why” (function) of a particular message (e.g. poem, newspaper story, billboard, critical essay); also called “What, How, & Why Outlines

12. Analytic Memos: students write a one‐ or two‐page analysis of a specific problem or issue to help inform a decision‐maker

Planning Formative Assessment Workshop 2012 III. Assessing Skill in Synthesis and Creative Thinking The CATS in this group focus on synthesis—each stimulate the student to create, and allow the faculty to assess, original intellectual products that result from a synthesis of course content and the students’ intelligence, judgment, knowledge, and skills. 13. One‐Sentence Summary: students answer the questions “Who does what to whom, when, where,

how, and why?” (WDWWWWHW) about a given topic and then creates a single informative, grammatical, and long summary sentence

14. Word Journal: involves a 2 part response; 1st the student summarizes a short text in a single word and 2nd the student writes 1‐2 paragraphs explaining the word choice

15. Approximate Analogies: students simply complete the 2nd half of an analogy—a is to b as x is to y; described as approximate because rigor of formal logic is not required

16. Concept Maps: students draw or diagram the mental connections they make between a major concept and other concepts they have learned

17. Invented Dialogues: students synthesize their knowledge of issues, personalities, and historical periods into the form of a carefully structured illustrative conversation; 2 levels of invention (select and weave quotes from primary sources or invent reasonable quotes that fit characters and context)

18. Annotated Portfolios: students assemble a very limited number of examples of creative work and supplement with own commentary on significance of examples

IV. Assessing Skill in Problem Solving The CATS in this group focus on problem solving skills of various kinds—recognition of types of problems, determining principles and techniques to solve, perceiving similarities of problem features and ability to reflect and then alter solution strategies. 19. Problem Recognition Tasks: students recognize and identify particular problem types 20. What’s the Principle?: students identify principle or principles to solve problems of various types 21. Documented Problem Solutions: students track in a written format the steps they take to solve

problems as if for a “show & tell” 22. Audio‐ and Videotaped Protocols: students work through a problem solving process and it is

captured to allow instructors to assess metacognition (learner’s awareness of and control of thinking)

V. Assessing Skill in Application and Performance The CATS in this group focus on students’ abilities to apply important—sometimes referenced as conditional knowledge—knowing when and where to apply what know and can do. 23. Directed Paraphrasing: students paraphrase part of a lesson for a specific audience demonstrating

ability to translate highly specialized information into language the clients or customers can understand

24. Application Cards: students generate examples of real‐work applications for important principles, generalizations, theories or procedures

25. Student‐Generated Test Questions: students generate test questions and model answers for critical areas of learning

26. Human Tableau or Class Modeling: Students transform and apply their learning into doing by physically modeling a process or representing an image.

Planning Formative Assessment Workshop 2012 27. Paper or Project Prospectus: Students create a brief plan for a paper or project based on your

guiding questions.

Techniques for Assessing Learner Attitudes, Values, and Self‐Awareness

VI. Assessing Students’ Awareness of Their Attitudes and Values The CATS in this group are designed to assist teachers in developing students’ attitudes, opinions, values, and self‐awareness within the course curriculum. 28. Classroom Opinion Polls: Students indicate degree of agreement or disagreement with a

statement or prompt. 29. Double‐entry Journals: Students record and respond to significant passages of text 30. Profiles of Admiral Individuals: Students write a brief description of the characteristics of a person

they admire in a field related to the course 31. Everyday Ethical Dilemma: Students respond to a case study that poses a discipline‐related ethical

dilemma 32. Course‐related Self‐Confidence Surveys: Students complete an anonymous survey indicating their

level of confidence in mastering the course material

VII. Assessing Students’ Self‐Awareness as Learners The CATS in this group are recommended to help students express personal goals and clarify self‐concept in order to make a connection between the articulated goals and those of the course. 33. Focused Autobiographical Sketches: Students write a brief description of a successful learning

experience they had relevant to the course material. 34. Interest/Knowledge/Skills Checklists: Students complete a checklist survey to indicate their

knowledge, skills and interest in various course topics. 35. Goal Ranking and Matching: Students list and prioritize 3 to 5 goals they have for their own

learning in the course. 36. Self‐Assessment Ways of Learning: Students compare themselves with several different “learning

styles” profiles to find the most likely match.

VIII. Assessing Course‐Related Learning and Study Skills, Strategies, and Behaviors The CATS in this group focus both student and teacher attention on the behaviors the student actually engages in when trying to learn. 37. Productive Study‐Time Logs: Students complete a study log to record the quantity and quality of

time spent studying for a specific course. 38. Punctuated Lectures: Students briefly reflect then create a written record of their listening level of

a lecture. Repeat twice in the same lecture and 2‐ 3 times over 2 to 3 weeks. 39. Process Analysis: Students outline the process they take in completing a specified assignment.

Planning Formative Assessment Workshop 2012 40. Diagnostic Learning Logs: Students write to learn by identifying, diagnosing, and prescribing

solutions to their own learning problems.

Techniques for Assessing Learner Reactions to Instruction

IX. Assessing Learner Reactions to Teachers and Teaching The CATS in this group are designed to provide context‐specific feedback that can improve teaching within a particular course. 41. Chain Notes: On an index card that is distributed in advance, each student responds to an open‐

ended prompt about his or her mental activity that is answered in less than a minute. 42. Electronic Survey Feedback: Students respond to a question or short series of questions about the

effectiveness of the course. 43. Teacher‐designed Feedback Forms: Students respond to specific questions through a focused

feedback form about the effectiveness of a particular class session. 44. Group Instructional Feedback Technique: Students respond to three questions related to the

student’s learning in the course. 45. Classroom Assessment Quality Circles: A group or groups of students provide the instructor with

ongoing assessment of the course through structured interactions.

X. Assessing Learner Reactions to Class Activities, Assignments, and Materials The CATS in this group are designed to give teachers information that will help them improve their course materials and assignments. 46. RSQC2 (Recall, Summarize, Question, Connect and Comment): Students write brief statements

that recall, summarize, question, connect and comment on meaningful points from previous class. 47. Group‐Work Evaluation: Students complete a brief survey about how their group is functioning

and make suggestions for improving the group process. 48. Reading Rating Sheets: Students complete a form that rates the effectiveness of the assigned

readings. 49. Assignment Assessments: Students respond to 2 or 3 open‐ended questions about the value of an

assignment to their learning. 50. Exam Evaluations: Students provide feedback about an exam’s learning value and/or format.


Why are you here?

Your progress on the intended workshop learner outcomes.

All your learning, knowledge, and ideas summed up in one place!


Why are you here?

FROM THE BACKGROUND READING

1. a) Define learner outcomes, being sure to capture at least 3 of the 5 main qualities as indicated by educational researchers. b) Define formative assessment, being sure to make a comparison to summative assessment whilst also distinguishing between assessment of and for learning.

TODAY

2. Write a short description (intended for your colleagues who did not come today) of 2 CATs you plan to try in your course(s) this coming academic term.

a. (name of technique you should really try!)____________________________________

Here is how it works!

b. (name of technique you should really try!)____________________________________

Here is how it works!


Why are you here? (Part 2)

3. Keeping in mind Outcome 2, chose one of the CATs. Re‐write your intended learning outcome to the descriptions of formative assessment techniques and provide 1 or 2 short sentences about why you chose these techniques to assess learning of the chosen outcomes. Be sure to include statements of what you plan to learn about student learning. (You may want to consider an outcome which you currently have difficulty assessing ). Use the checklist (Exhibit 4.1) to avoid common problems.

Course name and #_____________. The main goal is:______________________________________

My intended learning outcome :

My reasoning for using ___________________________________ (assessment technique) to assess the above listed outcome is:

4. Through observation of formative assessment, and reaction to it, in today’s workshop, write down 2 strategies for how you plan to organize formative assessment in order to make analysis simple, fast, and informative (e.g., aligning method with cognitive level). Also write 2 strategies describing how you plan to share feedback with the class (e.g., time span, communication method, outcome oriented, etc.) My strategies for organizing formative assessment implementation for simple, fast, and informative analysis (what must be present so that you are most efficient and learn the most about student learning):

a.

b.

My Feedback sharing strategies (what must be present for students to learn about their own learning):

a.

b.

Angelo

P

o: http://www.ct

Planning

tl.ualberta.ca/do

g Format

ocuments/Work

tive Asse

kshopI.pdf

essmentt Workshhop 2012


Feedback Form

1. Please rate the overall quality of this session on the scale below: 1 2 3 4 5

Very poor Poor Acceptable Good Excellent

2. Please rate the overall usefulness of the session below:

1 2 3 4 5 Useless Not very Somewhat Very Extremely

3. Please rate the effectiveness of the presenter below:

1 2 3 4 5

Not at all Not very Somewhat Very Extremely

4. What did you learn that you can apply to your own teaching? (Please be as specific as possible)

5. How could the session have been more useful to you? (Please be as specific as possible)

6. What was the muddiest point from the entire workshop? (what is still unclear to you?)

Documents

Planning Formative Assessment Workshop FGSR 2012 · Planning Formative Assessment Workshop 2012 References and Further Reading Some of the resources used today for the “Outcomes”